nicolas-georger/deepseek-kubernetes-sandbox
1
0
Fork 0
mirror of https://github.com/sredevopsorg/deepseek-kubernetes-sandbox.git synced 2026-11-03 01:13:24 +00:00
Code Projects Releases Activity

Make DSH agents-as-a-service browser-usable; fix probe crash-loop (#3)

Browse Source 
* Rewrite DSH gateway for browser sessions, HTTP and WebSocket

- Replace the Flask proxy with a Starlette ASGI app (uvicorn factory),
  keeping the control plane (claims, wake/suspend, idle sweeper).
- Add HttpOnly gateway sessions and a server-side DSH launch-token
  handshake: read the token from pod logs, exchange it for a dsh-auth
  cookie, cache per (user, pod UID, Host), refresh once on 401.
- Proxy HTTP/SSE and WebSocket (/api/remote.mux) under /u/<user>/, with
  header stripping/injection, Location rewriting and wake-on-connect.
- Add a landing/signup/login page, health endpoint, pinned dependencies,
  a read-only-rootfs gateway image, 87 cluster-free tests and docs.

* Fix DSH probe crash-loop and harden gateway manifests

- Probe /favicon.svg instead of /: DSH gates / behind its launch-token
  cookie and returns 401, which made warm-pool pods crash-loop.
- Pass DSH_TRUSTED_HOSTS through the entrypoint as --trusted-host args
  for public/ingress hosts; keep localhost trusted.
- Deploy the published :gateway image, add pods/pods-log RBAC, /healthz
  probes, read-only rootfs with a /tmp emptyDir and non-root security.
- Add optional Traefik Ingress and document the prerequisites.

* Update CI, smoke tests, build tooling and docs

- Run gateway pytest/ruff and build/smoke-test the image in CI.
- Parameterize the DSH build version and pnpm version inputs.
- Exercise cookie signup, HTML proxy, WebSocket auth, suspend/wake and
  delete in run-test-kind.sh.
- Add Makefile/build.sh targets and document the architecture, handshake,
  probes, trusted hosts, troubleshooting and limitations.

---------

Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
This commit is contained in:
SREDevOps.org Agent
2026-10-21 12:10:03 -03:00
committed by GitHub
parent c9be929212
commit 6301c21bea
23 changed files with 3700 additions and 660 deletions
Download Patch File Download Diff File Expand all files Collapse all files

144
README.md
View File

@@ -3,34 +3,110 @@
Packages [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) Web (`dsh web`) into
a multi-arch, Kubernetes-ready container image, and ships the
[agents-as-a-service](examples/dsh-agents-as-a-service/) platform pattern — running a personal
DSH Web instance per user on [kubernetes-sigs/agent-sandbox](https://github.com/kubernetes-sigs/agent-sandbox).
DSH Web instance per user on [kubernetes-sigs/agent-sandbox](https://github.com/kubernetes-sigs/agent-sandbox)
behind a small browser-usable gateway.
## What's in this repo
| Path | Purpose |
| --- | --- |
| [`Dockerfile`](Dockerfile) | Multi-stage build that clones DSH, `pnpm pack`s the CLI, and produces a slim runtime image |
| [`build.sh`](build.sh) | Local build + run (builds `dsh-web:<version>` and runs it) |
| [`.github/workflows/multi-build.yaml`](.github/workflows/multi-build.yaml) | CI build (amd64 + arm64) and push to GHCR |
| [`deploy/k8s/sandbox-template.yaml`](deploy/k8s/sandbox-template.yaml) | Minimal single-tenant `SandboxTemplate` |
| [`Dockerfile`](Dockerfile) | Multi-stage build that clones DSH, `pnpm pack`s the CLI, and produces a slim runtime image (`:3080`) |
| [`build.sh`](build.sh), [`Makefile`](Makefile) | Local image builds: DSH Web and gateway, plus test/lint targets |
| [`examples/dsh-agents-as-a-service/`](examples/dsh-agents-as-a-service/) | The full multi-tenant platform pattern (warm pool, per-user claims, gateway, smoke test) |
| [`examples/dsh-agents-as-a-service/gateway/`](examples/dsh-agents-as-a-service/gateway/) | Starlette ASGI gateway: signup, cookie sessions, `/u/{user}/**` HTTP + WebSocket proxy, idle sweeper |
| [`deploy/k8s/sandbox-template.yaml`](deploy/k8s/sandbox-template.yaml) | Minimal single-tenant `SandboxTemplate` |
| [`docs/architecture.md`](docs/architecture.md) | Container and sequence diagrams, design decisions, security notes |
| [`.github/workflows/`](.github/workflows/) | CI: manifests dry-run, gateway tests/lint, multi-arch image builds |
## Quick start (image)
## Published images
The image is published to GitHub Container Registry as
`ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:<tag>`, where `<tag>` is `main`,
`latest`, or a DSH version tag such as `dsh-v0.2.1-rc.2`.
Everything is published to GitHub Container Registry as
`ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox`:
| Tag | Contents | Port |
| --- | --- | --- |
| `dsh-v0.2.1-rc.2` (also `main`, `latest`) | DSH Web image | `3080` |
| `gateway` | Starlette gateway image | `8080` |
The DSH Web tag matches the upstream DSH git tag it was built from.
## Quick start (DSH Web image)
```bash
docker run --rm -it \
docker run -d --name dsh-web \
-p 3080:3080 \
-v dsh-home:/home/node/.dsh \
-e DEEPSEEK_API_KEY \
ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.2
```
> **Local port note:** `dsh web` listens on `3080` inside the container. If `3080` is
> already in use on your workstation, map a different host port (e.g. `-p 18081:3080`).
**DSH Web is not usable at a bare `/`.** Without its launch-token cookie it answers `401`, and
the token is only printed in the startup logs:
```bash
docker logs dsh-web 2>&1 | grep -m1 '?token='
# dsh web: http://localhost:3080/?token=...
```
Open the printed `?token=...` URL once — DSH sets its auth cookie and the UI loads normally
afterwards. Hitting `/` directly (or after the cookie expires) returns `401` by design; that is
why Kubernetes probes use `/favicon.svg` (see below) and why the gateway performs this
handshake for the browser.
> **Different host port?** If `3080` is already taken on your workstation, map another port and
> set the public URL so the printed token URL matches: `-p 18081:3080 -e
> DSH_PUBLIC_URL=http://localhost:18081`, then open `http://localhost:18081/?token=...`.
## Quick start (Kubernetes)
```bash
kubectl create namespace dsh-demo --dry-run=client -o yaml | kubectl apply -f -
kubectl -n dsh-demo create secret generic dsh-provider-keys \
--from-literal=DEEPSEEK_API_KEY="sk-..."
kubectl apply -f deploy/k8s/sandbox-template.yaml
# A template needs a warm pool and a claim to become a running Sandbox.
kubectl apply -f - <<'EOF'
apiVersion: extensions.agents.x-k8s.io/v1beta1
kind: SandboxWarmPool
metadata:
name: dsh-pool
namespace: dsh-demo
spec:
replicas: 1
sandboxTemplateRef:
name: dsh-web-template
---
apiVersion: extensions.agents.x-k8s.io/v1beta1
kind: SandboxClaim
metadata:
name: dsh
namespace: dsh-demo
spec:
warmPoolRef:
name: dsh-pool
EOF
kubectl -n dsh-demo wait sandboxclaim/dsh --for=condition=Ready --timeout=180s
```
Then reach the instance through its launch-token URL:
```bash
SB=$(kubectl -n dsh-demo get sandboxclaim dsh -o jsonpath='{.status.sandbox.name}')
kubectl -n dsh-demo logs "$SB" | grep -m1 '?token='
kubectl -n dsh-demo port-forward "pod/$SB" 18081:3080
# open http://localhost:18081/?token=... from the log line
```
Two things bite first-time setups:
- **Probes must not hit `/`.** DSH returns `401` there without the launch token, so a
readiness/liveness probe on `/` crash-loops the pod. The shipped templates use
`/favicon.svg`, which is unauthenticated; keep that path in custom templates.
- **Host allow-listing.** DSH rejects unknown `Host` headers. `localhost` and `127.0.0.1` are
always trusted; for any other hostname add it to the template's `DSH_TRUSTED_HOSTS` (comma or
space separated). The gateway forwards the browser's `Host`, so the public gateway hostname
must be listed there.
## The multi-user platform pattern
@@ -38,15 +114,23 @@ docker run --rm -it \
[hermes-agents-as-a-service](https://github.com/kubernetes-sigs/agent-sandbox/tree/main/examples/hermes-agents-as-a-service)
example to DSH Web. It demonstrates the platform shape — one `SandboxTemplate`, one warm pool,
one tiny `SandboxClaim` per user, and `operatingMode` as the cost dial — plus a gateway that
turns it into a usable service (signup, per-user tokens, `/u/{user}/**` proxy with
wake-on-connect, idle suspend). See its [README](examples/dsh-agents-as-a-service/README.md) for
the full walkthrough.
turns it into a usable service.
**Browser flow:** open the gateway landing page, sign up, and get an `HttpOnly` session cookie.
The browser then browses `http://<gateway>/u/<user>/`, which proxies HTTP **and WebSocket** to
that user's DSH Web. The gateway reads the DSH launch token from the pod logs, performs the
token handshake server-side, and injects the resulting DSH cookie on every proxied request — the
browser never sees the token. Suspended instances are woken transparently on the next request;
idle ones are suspended again after `IDLE_TIMEOUT`.
See the [example README](examples/dsh-agents-as-a-service/README.md) for the full walkthrough
and [docs/architecture.md](docs/architecture.md) for the diagrams and trade-offs.
### Mechanism mapping: Hermes → DSH
| Hermes example mechanism | Role | Adaptation for DSH |
| :--- | :--- | :--- |
| `SandboxTemplate` | Reusable Pod blueprint | Image `dsh-web`, port `3080` |
| `SandboxTemplate` | Reusable Pod blueprint | Image `dsh-web`, port `3080`, probes on `/favicon.svg` |
| `volumeClaimTemplates` | Auto-create a PVC per user | Mount `/home/node/.dsh` (DSH state) |
| `SandboxWarmPool` | Pre-warmed instances | Keep as-is; size for expected concurrency |
| `SandboxClaim` + `warmPoolRef` | "Registration" adopts a pooled instance | Keep exactly as-is; one claim per user |
@@ -60,19 +144,37 @@ the full walkthrough.
- **Single port** — DSH Web serves everything on `3080`; no API/dashboard port split.
- **One secret** — only `DEEPSEEK_API_KEY`; no dashboard/session/api-server credentials.
- **Host forwarding** — the gateway forwards `Host` + `X-Forwarded-*` (and keeps
`DSH_PUBLIC_URL` empty) so DSH generates reachable links.
- **WebSocket caveat** — the requests-based proxy handles SSE but not WebSocket upgrades; use a
path-based Ingress per user for full web-GUI interactivity.
`DSH_PUBLIC_URL` empty) so DSH generates reachable links; add the public hostname to
`DSH_TRUSTED_HOSTS`.
- **Browser sessions + WebSocket** — the gateway is Starlette ASGI, so it proxies WebSocket
upgrades as well as HTTP/SSE, and authenticates browsers with an `HttpOnly` cookie instead of
exposing per-user bearer tokens in URLs.
## Build it yourself
```bash
./build.sh # docker build + run (dsh-web:0.2.1-rc.2)
# or
./build.sh # build the DSH Web image, then run it interactively
./build.sh web # build the DSH Web image only
./build.sh gateway # build the gateway image
make image # same as ./build.sh web
make gateway # gateway image
make test # gateway unit tests (pytest)
make lint # gateway lint (ruff)
```
Or directly:
```bash
docker build \
--build-arg DSH_VERSION=dsh-v0.2.1-rc.2 \
--build-arg PNPM_VERSION=11.28.5 \
-t dsh-web:0.2.1-rc.2 .
docker build -t ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway \
examples/dsh-agents-as-a-service/gateway/
```
See [`Dockerfile`](Dockerfile) for build args (`DSH_REPO`, `DSH_VERSION`, `PNPM_VERSION`) and
the runtime image contract (non-root `node` user, `DSH_HOME=/home/node/.dsh`, `EXPOSE 3080`).
`.env.sample` documents the runtime variables for both images.