mirror of
https://github.com/sredevopsorg/deepseek-kubernetes-sandbox.git
synced 2026-11-03 01:13:24 +00:00
Make DSH agents-as-a-service browser-usable; fix probe crash-loop (#3)
* Rewrite DSH gateway for browser sessions, HTTP and WebSocket - Replace the Flask proxy with a Starlette ASGI app (uvicorn factory), keeping the control plane (claims, wake/suspend, idle sweeper). - Add HttpOnly gateway sessions and a server-side DSH launch-token handshake: read the token from pod logs, exchange it for a dsh-auth cookie, cache per (user, pod UID, Host), refresh once on 401. - Proxy HTTP/SSE and WebSocket (/api/remote.mux) under /u/<user>/, with header stripping/injection, Location rewriting and wake-on-connect. - Add a landing/signup/login page, health endpoint, pinned dependencies, a read-only-rootfs gateway image, 87 cluster-free tests and docs. * Fix DSH probe crash-loop and harden gateway manifests - Probe /favicon.svg instead of /: DSH gates / behind its launch-token cookie and returns 401, which made warm-pool pods crash-loop. - Pass DSH_TRUSTED_HOSTS through the entrypoint as --trusted-host args for public/ingress hosts; keep localhost trusted. - Deploy the published :gateway image, add pods/pods-log RBAC, /healthz probes, read-only rootfs with a /tmp emptyDir and non-root security. - Add optional Traefik Ingress and document the prerequisites. * Update CI, smoke tests, build tooling and docs - Run gateway pytest/ruff and build/smoke-test the image in CI. - Parameterize the DSH build version and pnpm version inputs. - Exercise cookie signup, HTML proxy, WebSocket auth, suspend/wake and delete in run-test-kind.sh. - Add Makefile/build.sh targets and document the architecture, handshake, probes, trusted hosts, troubleshooting and limitations. --------- Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
This commit is contained in:
committed by
GitHub
parent
c9be929212
commit
6301c21bea
@@ -1,7 +1,18 @@
|
||||
FROM python:3.12-slim
|
||||
|
||||
# Read-only-rootfs friendly: no .pyc writes, no home directory writes.
|
||||
ENV PYTHONDONTWRITEBYTECODE=1 \
|
||||
PYTHONUNBUFFERED=1 \
|
||||
HOME=/tmp
|
||||
|
||||
WORKDIR /app
|
||||
COPY requirements.txt .
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
COPY gateway.py .
|
||||
|
||||
EXPOSE 8080
|
||||
# Run as a fixed non-root UID; HOME=/tmp keeps caches out of the image.
|
||||
USER 10000
|
||||
CMD ["python", "gateway.py"]
|
||||
|
||||
CMD ["uvicorn", "gateway:create_app", "--factory", \
|
||||
"--host", "0.0.0.0", "--port", "8080", "--no-access-log"]
|
||||
|
||||
184
examples/dsh-agents-as-a-service/gateway/README.md
Normal file
184
examples/dsh-agents-as-a-service/gateway/README.md
Normal file
@@ -0,0 +1,184 @@
|
||||
# DSH agents-as-a-service gateway
|
||||
|
||||
A single-file, browser-usable control + data plane for
|
||||
[DeepSeek Harness Web (DSH)](https://github.com/deepseek-ai/deepseek-harness)
|
||||
sandboxes running behind the `agent-sandbox` CRDs.
|
||||
|
||||
It does three things:
|
||||
|
||||
1. **Users** — creates a `SandboxClaim` from a warm pool, mints a bearer token
|
||||
(only its SHA-256 is stored as a claim annotation), reports derived state,
|
||||
and deletes the claim (cascading the sandbox and PVC).
|
||||
2. **Browser sessions** — exchanges a user token for an HttpOnly session
|
||||
cookie so the DSH UI works in a browser at `/u/<user>/`. Automation can
|
||||
still use `Authorization: Bearer <token>`.
|
||||
3. **Reverse proxy** — proxies HTTP (including SSE) and WebSocket traffic to
|
||||
the sandbox's DSH Web server on `:3080`, including DSH's own cookie
|
||||
handshake.
|
||||
|
||||
DSH is pinned to `dsh-v0.2.1-rc.2` in the example manifests. It binds `:3080`,
|
||||
serves `/`, gates `/api` behind a `dsh-auth-*` cookie and requires a WebSocket
|
||||
at `/api/remote.mux`.
|
||||
|
||||
## Flow
|
||||
|
||||
```
|
||||
browser / curl gateway cluster
|
||||
| | |
|
||||
| POST /users {"user":"alice"} | |
|
||||
|------------------------------>| create SandboxClaim ---------->|
|
||||
| | wait Ready |
|
||||
| 201 {token} + session cookie | |
|
||||
|<------------------------------| |
|
||||
| GET /u/alice/ (cookie) | |
|
||||
|------------------------------>| claim -> sandbox -> pod |
|
||||
| | read pod logs (pods/log) |
|
||||
| | find "dsh web: ...?token=..." |
|
||||
| | GET http://<fqdn>:3080/?token= |
|
||||
| | capture dsh-auth-* Set-Cookie |
|
||||
| | proxy with Host/Origin kept |
|
||||
| HTML / SSE / WebSocket |<----------------------------->|
|
||||
|<------------------------------| |
|
||||
```
|
||||
|
||||
The launch token is printed by DSH to its pod stdout:
|
||||
|
||||
```
|
||||
dsh web: http://<podIP>:3080/?token=<base64url>
|
||||
```
|
||||
|
||||
The gateway reads the tail of the pod logs, parses the **last** such line and
|
||||
performs the `?token=` handshake itself with the browser's `Host` header. The
|
||||
resulting `dsh-auth-*` cookie is cached per `(user, pod UID, Host)` and injected
|
||||
into proxied requests. The browser never sees the launch token or the bearer
|
||||
token. A failed handshake returns `503` with `Retry-After: 10`.
|
||||
|
||||
## API
|
||||
|
||||
| Method | Path | Auth | Purpose |
|
||||
| --- | --- | --- | --- |
|
||||
| `GET` | `/healthz` | none | Liveness/readiness. Never touches the cluster. |
|
||||
| `GET` | `/` | none | Landing page with signup + token login forms. |
|
||||
| `POST` | `/users` | none | Signup. JSON body `{"user": "<dns-1123 label>"}` or form field `user`. JSON returns `201` when Ready, `202` when accepted but still Waking; form returns `303` to `/u/<user>/`. Both set the session cookie. `409` if the user exists, `400` if invalid, `413` if the body is too large. |
|
||||
| `POST` | `/login` | none | Form (`user`, `token`) returns `303` to `/u/<user>/`; JSON returns `200`. Sets the session cookie. `401` on a bad token, `404` for an unknown user. |
|
||||
| `POST` | `/logout` | none | Clears the session cookie and `303`s to `/`. |
|
||||
| `GET` | `/users/{user}` | bearer or cookie | Derived state: `Provisioning`, `Waking`, `Ready`, `Suspending`, `Suspended`. |
|
||||
| `DELETE` | `/users/{user}` | bearer or cookie | Delete the claim (sandbox + PVC cascade). Cookie-auth mutations require a same-origin `Origin`/`Referer`. |
|
||||
| `GET` | `/u/{user}` | — | `308` redirect to `/u/{user}/`. |
|
||||
| `*` | `/u/{user}/{path}` | bearer or cookie | HTTP proxy to DSH. Waits for a suspended sandbox to wake. Wakes on connect. |
|
||||
| `WS` | `/u/{user}/{path}` | bearer or cookie | WebSocket proxy (e.g. `/api/remote.mux`). |
|
||||
|
||||
### Auth
|
||||
|
||||
* `Authorization: Bearer <token>` is checked with a timing-safe SHA-256
|
||||
comparison against the `aaas.example.com/token-sha256` claim annotation.
|
||||
* The browser session cookie is
|
||||
`dsh_gw_session=<user>:<token>` (HttpOnly, SameSite=Lax, Path=/, optional
|
||||
Secure). A session for user A never authorizes user B.
|
||||
* Cookie-authenticated **mutations** (`POST`, `PUT`, `PATCH`, `DELETE`) and
|
||||
cookie-authenticated **WebSocket upgrades** require the request `Origin`
|
||||
(or `Referer` fallback) to match the `Host` header. Missing or mismatched
|
||||
cross-origin metadata is rejected (`403` HTTP, close code `4401` for WS).
|
||||
* The gateway's bearer token, the browser `Cookie` header and any `?token=`
|
||||
query parameter are never forwarded upstream. Only the cached `dsh-auth-*`
|
||||
cookie is injected.
|
||||
* Upstream `Set-Cookie` headers are dropped. `Location` headers that are
|
||||
absolute paths are rewritten under `/u/<user>/`, and any `token` query
|
||||
parameter is stripped.
|
||||
|
||||
### Proxy failure semantics
|
||||
|
||||
| Situation | Result |
|
||||
| --- | --- |
|
||||
| Unknown user / claim | `404` |
|
||||
| Missing or invalid credentials | `401` |
|
||||
| Cross-origin cookie mutation | `403` |
|
||||
| Request body over `MAX_BODY_MB` | `413` |
|
||||
| Sandbox not provisioned, wake timeout, handshake failure | `503` + `Retry-After: 10` |
|
||||
| Upstream connection/protocol failure | `502` |
|
||||
| WS auth failure (unknown user, bad token, cross-origin) | accept, then close `4401` |
|
||||
| WS upstream/handshake failure | accept, then close `1011` |
|
||||
|
||||
A close code can only be delivered after the WebSocket upgrade is accepted;
|
||||
Starlette would turn an early close into an HTTP 403, which is why WS auth
|
||||
failures are `101`, then close `4401`.
|
||||
|
||||
## Configuration
|
||||
|
||||
All values are read from the environment by `Settings.from_env()`; every one
|
||||
can also be injected into `create_app` for tests.
|
||||
|
||||
| Variable | Default | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `NAMESPACE` | `dsh-demo` | Namespace holding the claims, sandboxes and pods. |
|
||||
| `POOL` | `dsh-pool` | `warmPoolRef.name` for new claims. |
|
||||
| `IDLE_TIMEOUT` | `60` | Seconds of inactivity before the sweeper suspends a sandbox. |
|
||||
| `WAKE_TIMEOUT` | `120` | Seconds to wait for a sandbox to become Ready (signup and wake-on-connect). |
|
||||
| `DSH_PORT` | `3080` | Upstream DSH port. |
|
||||
| `DSH_CONTAINER` | `dsh-web` | Container whose logs carry the launch token. |
|
||||
| `DSH_LOG_TAIL_LINES` | `2000` | How many log lines to scan for the launch token. |
|
||||
| `COOKIE_NAME` | `dsh_gw_session` | Browser session cookie name. |
|
||||
| `COOKIE_SECURE` | `false` | Add `Secure` to the session cookie when served over TLS. |
|
||||
| `COOKIE_MAX_AGE` | `604800` | Session cookie lifetime in seconds. |
|
||||
| `MAX_BODY_MB` | `32` | Maximum buffered request body (and `413` threshold). |
|
||||
| `HTTP_TIMEOUT` | `300` | Upstream connect/read timeout in seconds. |
|
||||
| `SWEEP_INTERVAL` | `15` | Seconds between idle sweeps. |
|
||||
|
||||
### Required RBAC
|
||||
|
||||
The gateway's ServiceAccount needs, in its namespace:
|
||||
|
||||
* `sandboxclaims` (`extensions.agents.x-k8s.io`): get, list, create, delete
|
||||
* `sandboxes` (`agents.x-k8s.io`): get, patch
|
||||
* `pods`: get
|
||||
* `pods/log`: get
|
||||
|
||||
## Running locally
|
||||
|
||||
The module has no import-time cluster access and the Kubernetes adapter is
|
||||
lazy, so `/healthz` works without a kubeconfig:
|
||||
|
||||
```bash
|
||||
python -m pip install -r requirements-dev.txt
|
||||
python -m pytest -q
|
||||
python -m ruff check .
|
||||
|
||||
# Uvicorn factory (loads in-cluster credentials only on the first claim call)
|
||||
uvicorn gateway:create_app --factory --host 0.0.0.0 --port 8080
|
||||
```
|
||||
|
||||
Container image:
|
||||
|
||||
```bash
|
||||
docker build -t dsh-gateway examples/dsh-agents-as-a-service/gateway
|
||||
```
|
||||
|
||||
The image runs as UID `10000`, sets `PYTHONDONTWRITEBYTECODE=1` and
|
||||
`HOME=/tmp` (read-only-rootfs compatible), listens on `8080`, and disables
|
||||
uvicorn's access log so bearer tokens in URLs are not logged.
|
||||
|
||||
## Known limitations
|
||||
|
||||
* **Log scraping is fragile.** The launch token is discovered from pod logs.
|
||||
Log rotation, a very chatty container, or DSH changing its startup message
|
||||
breaks the handshake. Prefer a first-class token/credential API when DSH
|
||||
offers one.
|
||||
* **Single replica, in-memory state.** The handshake cookie cache, idle
|
||||
activity clocks and in-flight counters live in one process. Running more
|
||||
than one replica would need shared state (or sticky routing), otherwise each
|
||||
replica handshakes independently and the sweeper may race another replica.
|
||||
* **In-memory token cache has no TTL.** It is keyed by pod UID, so pod
|
||||
restarts invalidate it naturally; otherwise entries survive until the pod or
|
||||
claim is deleted.
|
||||
* **No TLS termination.** Run behind an ingress/load balancer; set
|
||||
`COOKIE_SECURE=true` there.
|
||||
* **Origin-based CSRF protection only.** Cookie-auth mutations require a
|
||||
same-origin `Origin`/`Referer`; there is no per-form CSRF token, rate
|
||||
limiting or login throttling.
|
||||
* **HTTP request bodies are buffered** up to `MAX_BODY_MB`; uploads are not
|
||||
streamed. Responses (including SSE) are streamed.
|
||||
* **Upstream cookies are not forwarded.** DSH cannot rotate its own
|
||||
`dsh-auth-*` cookie through the gateway; on an upstream `401` the gateway
|
||||
re-runs the handshake once and replays the request.
|
||||
* **Only DSH on `:3080` is proxied.** There is no dashboard/API-port split and
|
||||
no platform API key injection.
|
||||
File diff suppressed because it is too large
Load Diff
14
examples/dsh-agents-as-a-service/gateway/pyproject.toml
Normal file
14
examples/dsh-agents-as-a-service/gateway/pyproject.toml
Normal file
@@ -0,0 +1,14 @@
|
||||
[tool.pytest.ini_options]
|
||||
asyncio_mode = "auto"
|
||||
testpaths = ["."]
|
||||
|
||||
[tool.ruff]
|
||||
target-version = "py312"
|
||||
line-length = 100
|
||||
|
||||
[tool.ruff.lint]
|
||||
select = ["E", "F", "I", "B", "UP", "PIE", "RUF"]
|
||||
ignore = ["RUF001", "RUF002", "RUF003"]
|
||||
|
||||
[tool.ruff.lint.per-file-ignores]
|
||||
"test_gateway.py" = ["E501"]
|
||||
@@ -0,0 +1,4 @@
|
||||
-r requirements.txt
|
||||
pytest==9.1.1
|
||||
pytest-asyncio==1.4.0
|
||||
ruff==0.17.1
|
||||
@@ -1,3 +1,6 @@
|
||||
flask==3.1.*
|
||||
requests==2.32.*
|
||||
kubernetes==31.0.*
|
||||
starlette==1.9.0
|
||||
uvicorn[standard]==0.54.0
|
||||
httpx==0.28.1
|
||||
websockets==17.3
|
||||
kubernetes==37.0.1
|
||||
python-multipart==0.0.32
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user