nicolas-georger/deepseek-kubernetes-sandbox
1
0
Fork 0
mirror of https://github.com/sredevopsorg/deepseek-kubernetes-sandbox.git synced 2026-11-03 01:13:24 +00:00
Code Projects Releases Activity
Files
6301c21beafaccd78f569ed1c1726b70ae150fc7
deepseek-kubernetes-sandbox/examples/dsh-agents-as-a-service/gateway/README.md
SREDevOps.org Agent 6301c21bea Make DSH agents-as-a-service browser-usable; fix probe crash-loop (#3) 
* Rewrite DSH gateway for browser sessions, HTTP and WebSocket

- Replace the Flask proxy with a Starlette ASGI app (uvicorn factory),
  keeping the control plane (claims, wake/suspend, idle sweeper).
- Add HttpOnly gateway sessions and a server-side DSH launch-token
  handshake: read the token from pod logs, exchange it for a dsh-auth
  cookie, cache per (user, pod UID, Host), refresh once on 401.
- Proxy HTTP/SSE and WebSocket (/api/remote.mux) under /u/<user>/, with
  header stripping/injection, Location rewriting and wake-on-connect.
- Add a landing/signup/login page, health endpoint, pinned dependencies,
  a read-only-rootfs gateway image, 87 cluster-free tests and docs.

* Fix DSH probe crash-loop and harden gateway manifests

- Probe /favicon.svg instead of /: DSH gates / behind its launch-token
  cookie and returns 401, which made warm-pool pods crash-loop.
- Pass DSH_TRUSTED_HOSTS through the entrypoint as --trusted-host args
  for public/ingress hosts; keep localhost trusted.
- Deploy the published :gateway image, add pods/pods-log RBAC, /healthz
  probes, read-only rootfs with a /tmp emptyDir and non-root security.
- Add optional Traefik Ingress and document the prerequisites.

* Update CI, smoke tests, build tooling and docs

- Run gateway pytest/ruff and build/smoke-test the image in CI.
- Parameterize the DSH build version and pnpm version inputs.
- Exercise cookie signup, HTML proxy, WebSocket auth, suspend/wake and
  delete in run-test-kind.sh.
- Add Makefile/build.sh targets and document the architecture, handshake,
  probes, trusted hosts, troubleshooting and limitations.

---------

Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
2026-10-21 12:10:03 -03:00

185 lines
9.0 KiB
Markdown
Raw Blame History

# DSH agents-as-a-service gateway
A single-file, browser-usable control + data plane for
[DeepSeek Harness Web (DSH)](https://github.com/deepseek-ai/deepseek-harness)
sandboxes running behind the `agent-sandbox` CRDs.
It does three things:
1. **Users** — creates a `SandboxClaim` from a warm pool, mints a bearer token
(only its SHA-256 is stored as a claim annotation), reports derived state,
and deletes the claim (cascading the sandbox and PVC).
2. **Browser sessions** — exchanges a user token for an HttpOnly session
cookie so the DSH UI works in a browser at `/u/<user>/`. Automation can
still use `Authorization: Bearer <token>`.
3. **Reverse proxy** — proxies HTTP (including SSE) and WebSocket traffic to
the sandbox's DSH Web server on `:3080`, including DSH's own cookie
handshake.
DSH is pinned to `dsh-v0.2.1-rc.2` in the example manifests. It binds `:3080`,
serves `/`, gates `/api` behind a `dsh-auth-*` cookie and requires a WebSocket
at `/api/remote.mux`.
## Flow
```
browser / curl gateway cluster
| | |
| POST /users {"user":"alice"} | |
|------------------------------>| create SandboxClaim ---------->|
| | wait Ready |
| 201 {token} + session cookie | |
|<------------------------------| |
| GET /u/alice/ (cookie) | |
|------------------------------>| claim -> sandbox -> pod |
| | read pod logs (pods/log) |
| | find "dsh web: ...?token=..." |
| | GET http://<fqdn>:3080/?token= |
| | capture dsh-auth-* Set-Cookie |
| | proxy with Host/Origin kept |
| HTML / SSE / WebSocket |<----------------------------->|
|<------------------------------| |
```
The launch token is printed by DSH to its pod stdout:
```
dsh web: http://<podIP>:3080/?token=<base64url>
```
The gateway reads the tail of the pod logs, parses the **last** such line and
performs the `?token=` handshake itself with the browser's `Host` header. The
resulting `dsh-auth-*` cookie is cached per `(user, pod UID, Host)` and injected
into proxied requests. The browser never sees the launch token or the bearer
token. A failed handshake returns `503` with `Retry-After: 10`.
## API
| Method | Path | Auth | Purpose |
| --- | --- | --- | --- |
| `GET` | `/healthz` | none | Liveness/readiness. Never touches the cluster. |
| `GET` | `/` | none | Landing page with signup + token login forms. |
| `POST` | `/users` | none | Signup. JSON body `{"user": "<dns-1123 label>"}` or form field `user`. JSON returns `201` when Ready, `202` when accepted but still Waking; form returns `303` to `/u/<user>/`. Both set the session cookie. `409` if the user exists, `400` if invalid, `413` if the body is too large. |
| `POST` | `/login` | none | Form (`user`, `token`) returns `303` to `/u/<user>/`; JSON returns `200`. Sets the session cookie. `401` on a bad token, `404` for an unknown user. |
| `POST` | `/logout` | none | Clears the session cookie and `303`s to `/`. |
| `GET` | `/users/{user}` | bearer or cookie | Derived state: `Provisioning`, `Waking`, `Ready`, `Suspending`, `Suspended`. |
| `DELETE` | `/users/{user}` | bearer or cookie | Delete the claim (sandbox + PVC cascade). Cookie-auth mutations require a same-origin `Origin`/`Referer`. |
| `GET` | `/u/{user}` | — | `308` redirect to `/u/{user}/`. |
| `*` | `/u/{user}/{path}` | bearer or cookie | HTTP proxy to DSH. Waits for a suspended sandbox to wake. Wakes on connect. |
| `WS` | `/u/{user}/{path}` | bearer or cookie | WebSocket proxy (e.g. `/api/remote.mux`). |
### Auth
* `Authorization: Bearer <token>` is checked with a timing-safe SHA-256
comparison against the `aaas.example.com/token-sha256` claim annotation.
* The browser session cookie is
`dsh_gw_session=<user>:<token>` (HttpOnly, SameSite=Lax, Path=/, optional
Secure). A session for user A never authorizes user B.
* Cookie-authenticated **mutations** (`POST`, `PUT`, `PATCH`, `DELETE`) and
cookie-authenticated **WebSocket upgrades** require the request `Origin`
(or `Referer` fallback) to match the `Host` header. Missing or mismatched
cross-origin metadata is rejected (`403` HTTP, close code `4401` for WS).
* The gateway's bearer token, the browser `Cookie` header and any `?token=`
query parameter are never forwarded upstream. Only the cached `dsh-auth-*`
cookie is injected.
* Upstream `Set-Cookie` headers are dropped. `Location` headers that are
absolute paths are rewritten under `/u/<user>/`, and any `token` query
parameter is stripped.
### Proxy failure semantics
| Situation | Result |
| --- | --- |
| Unknown user / claim | `404` |
| Missing or invalid credentials | `401` |
| Cross-origin cookie mutation | `403` |
| Request body over `MAX_BODY_MB` | `413` |
| Sandbox not provisioned, wake timeout, handshake failure | `503` + `Retry-After: 10` |
| Upstream connection/protocol failure | `502` |
| WS auth failure (unknown user, bad token, cross-origin) | accept, then close `4401` |
| WS upstream/handshake failure | accept, then close `1011` |
A close code can only be delivered after the WebSocket upgrade is accepted;
Starlette would turn an early close into an HTTP 403, which is why WS auth
failures are `101`, then close `4401`.
## Configuration
All values are read from the environment by `Settings.from_env()`; every one
can also be injected into `create_app` for tests.
| Variable | Default | Meaning |
| --- | --- | --- |
| `NAMESPACE` | `dsh-demo` | Namespace holding the claims, sandboxes and pods. |
| `POOL` | `dsh-pool` | `warmPoolRef.name` for new claims. |
| `IDLE_TIMEOUT` | `60` | Seconds of inactivity before the sweeper suspends a sandbox. |
| `WAKE_TIMEOUT` | `120` | Seconds to wait for a sandbox to become Ready (signup and wake-on-connect). |
| `DSH_PORT` | `3080` | Upstream DSH port. |
| `DSH_CONTAINER` | `dsh-web` | Container whose logs carry the launch token. |
| `DSH_LOG_TAIL_LINES` | `2000` | How many log lines to scan for the launch token. |
| `COOKIE_NAME` | `dsh_gw_session` | Browser session cookie name. |
| `COOKIE_SECURE` | `false` | Add `Secure` to the session cookie when served over TLS. |
| `COOKIE_MAX_AGE` | `604800` | Session cookie lifetime in seconds. |
| `MAX_BODY_MB` | `32` | Maximum buffered request body (and `413` threshold). |
| `HTTP_TIMEOUT` | `300` | Upstream connect/read timeout in seconds. |
| `SWEEP_INTERVAL` | `15` | Seconds between idle sweeps. |
### Required RBAC
The gateway's ServiceAccount needs, in its namespace:
* `sandboxclaims` (`extensions.agents.x-k8s.io`): get, list, create, delete
* `sandboxes` (`agents.x-k8s.io`): get, patch
* `pods`: get
* `pods/log`: get
## Running locally
The module has no import-time cluster access and the Kubernetes adapter is
lazy, so `/healthz` works without a kubeconfig:
```bash
python -m pip install -r requirements-dev.txt
python -m pytest -q
python -m ruff check .
# Uvicorn factory (loads in-cluster credentials only on the first claim call)
uvicorn gateway:create_app --factory --host 0.0.0.0 --port 8080
```
Container image:
```bash
docker build -t dsh-gateway examples/dsh-agents-as-a-service/gateway
```
The image runs as UID `10000`, sets `PYTHONDONTWRITEBYTECODE=1` and
`HOME=/tmp` (read-only-rootfs compatible), listens on `8080`, and disables
uvicorn's access log so bearer tokens in URLs are not logged.
## Known limitations
* **Log scraping is fragile.** The launch token is discovered from pod logs.
Log rotation, a very chatty container, or DSH changing its startup message
breaks the handshake. Prefer a first-class token/credential API when DSH
offers one.
* **Single replica, in-memory state.** The handshake cookie cache, idle
activity clocks and in-flight counters live in one process. Running more
than one replica would need shared state (or sticky routing), otherwise each
replica handshakes independently and the sweeper may race another replica.
* **In-memory token cache has no TTL.** It is keyed by pod UID, so pod
restarts invalidate it naturally; otherwise entries survive until the pod or
claim is deleted.
* **No TLS termination.** Run behind an ingress/load balancer; set
`COOKIE_SECURE=true` there.
* **Origin-based CSRF protection only.** Cookie-auth mutations require a
same-origin `Origin`/`Referer`; there is no per-form CSRF token, rate
limiting or login throttling.
* **HTTP request bodies are buffered** up to `MAX_BODY_MB`; uploads are not
streamed. Responses (including SSE) are streamed.
* **Upstream cookies are not forwarded.** DSH cannot rotate its own
`dsh-auth-*` cookie through the gateway; on an upstream `401` the gateway
re-runs the handshake once and replays the request.
* **Only DSH on `:3080` is proxied.** There is no dashboard/API-port split and
no platform API key injection.
View Git Blame Copy Permalink