nicolas-georger/deepseek-kubernetes-sandbox
1
0
Fork 0
mirror of https://github.com/sredevopsorg/deepseek-kubernetes-sandbox.git synced 2026-11-03 01:13:24 +00:00
Code Projects Releases Activity

feat(repo)!: repo reorganization (#16)

Browse Source 
* feat(repo)!: restructure around the dsh-harness image and one example per implementation

- the root Dockerfile is the primary artifact; versions.env is the single source of truth
- merge gateway-api/dsh-demo into examples/dsh-agents-as-a-service (manifests/, exposure variants)
- move the single-tenant template into examples/minimal-single-tenant
- publish the gateway as a nested package under the repository image
- mirror the DSH ref for release identities (first release dsh-v0.2.1-rc.2-r2)
- CI: lint/python/manifests/gateway-image, reusable multi-arch build, tag-driven release, weekly e2e
- docs: README, docs/{main-image,examples,gitflow,versioning}, example READMEs, AGENTS.md, governance
- SSO is opt-in again; gateway pull policy and cookie defaults match the documented demo

BREAKING CHANGE: example manifest paths, the gateway image name, and the removal
of deploy/ and gateway-api/ affect existing deployments; see the migration table
in README.md.

* docs(changelog): generate initial changelog from history

* fix(scripts): make set-version update Dockerfile and sample revision pins

* build(hadolint): document intentional lint exceptions for the main image

---------

Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
This commit is contained in:
Nicolás Georger
2026-10-25 08:37:43 -03:00
committed by GitHub
parent 53e55fc851
commit 3b70f179ac
69 changed files with 3774 additions and 1643 deletions
Download Patch File Download Diff File Expand all files Collapse all files

487
examples/dsh-agents-as-a-service/README.md
View File

@@ -2,20 +2,41 @@
This example distills the upstream
[hermes-agents-as-a-service](https://github.com/kubernetes-sigs/agent-sandbox/tree/main/examples/hermes-agents-as-a-service)
platform pattern into pure agent-sandbox resources + kubectl, running **DeepSeek Harness Web
(DSH)** instead of Hermes. Every user of such a platform gets a personal, long-lived DSH Web
instance that is **provisioned in ~2 seconds from a warm pool, suspended when idle to save
cost, and woken transparently when the user returns** — with all state (conversations,
sessions, config) surviving on a per-user PVC at `/home/node/.dsh`.
platform pattern into pure agent-sandbox resources + kubectl, running **DeepSeek
Harness Web (DSH)** instead of Hermes. Every user of such a platform gets a
personal, long-lived DSH Web instance that is **provisioned in ~2 seconds from a
warm pool, suspended when idle to save cost, and woken transparently when the
user returns** — with all state (conversations, sessions, config) surviving on a
per-user PVC at `/home/node/.dsh`.
It ships in two layers:
It ships in three layers:
1. **The resource layer** (steps 1–7): pure manifests + kubectl, so every mechanism is visible.
2. **The service layer** (step 8): a small included **gateway** (`gateway/`) that makes the
platform browser-usable — a signup page/API, `HttpOnly` cookie sessions, a `/u/{user}/**`
proxy for HTTP **and WebSocket** that transparently **wakes suspended instances on
connect**, and an idle sweeper that suspends them again. The gateway performs the DSH
launch-token handshake server-side, so the browser never sees the token.
1. **The resource layer** (steps 1–7): pure manifests + kubectl, so every
mechanism is visible.
2. **The service layer** (step 8): a small included **gateway** (`gateway/`) that
makes the platform browser-usable — a signup page/API, `HttpOnly` cookie
sessions, a `/u/{user}/**` proxy for HTTP **and WebSocket** that transparently
**wakes suspended instances on connect**, and an idle sweeper that suspends
them again. The gateway performs the DSH launch-token handshake server-side,
so the browser never sees the token.
3. **The exposure layer** (step 9): choose exactly one way in — a plain Traefik
Ingress, or Gateway API + oauth2-proxy (GitHub SSO).
All manifests live under [`manifests/`](manifests/):
```
manifests/
+-- 00-prereqs.yaml # namespace + empty provider-key Secret
+-- 10-sandboxtemplate.yaml # the DSH Web blueprint
+-- 20-sandboxwarmpool.yaml # 2 pre-warmed spares
+-- 30-claim-alice.yaml # one user
+-- 40-claim-rejected.yaml # negative demo: env injection rejected
+-- 50-gateway.yaml # Deployment/RBAC/Service for the gateway
+-- exposure/
+-- README.md # the two exposure variants, when to use which
+-- ingress/60-ingress.yaml
+-- gateway-api-sso/70-gateway.yaml ... 74-tls-secret.yaml.example
```
What it demonstrates, feature by feature:
@@ -34,20 +55,21 @@ What it demonstrates, feature by feature:
- **Multi-arch**: the DSH image is built for amd64 and arm64, so there is no
`kubernetes.io/arch: amd64` nodeSelector.
- **Single port**: DSH Web serves everything on `3080`; the gateway proxies to one upstream
instead of splitting `/v1/*` (API) from the dashboard.
- **One shared secret**: only `DEEPSEEK_API_KEY` is injected (via `envFrom`); there is no
dashboard basic-auth, session-cookie secret, or API-server key.
- **Launch-token handshake**: DSH gates `/` behind a token it prints only in its logs. The
gateway reads it, performs the handshake, and keeps the resulting auth cookie server-side;
the browser authenticates to the gateway with its own cookie. See
[How the DSH handshake works](#how-the-dsh-handshake-works).
- **Host allow-list**: DSH rejects unknown `Host` headers. The gateway forwards the browser's
`Host`, so the public hostname must be added to `DSH_TRUSTED_HOSTS` in the template.
- **Probes target `/favicon.svg`**: `/` returns `401` without the launch token, so a probe on
`/` crash-loops the pod. `/favicon.svg` is unauthenticated, and a `startupProbe` gates
liveness/readiness so a slow cold start (plugin activation, or a resumed sandbox) is not
restarted mid-boot.
- **Single port**: DSH Web serves everything on `3080`; the gateway proxies to one
upstream instead of splitting `/v1/*` (API) from the dashboard.
- **One shared secret**: only `DEEPSEEK_API_KEY` is injected (via `envFrom`); there
is no dashboard basic-auth, session-cookie secret, or API-server key.
- **Launch-token handshake**: DSH gates `/` behind a token it prints only in its
logs. The gateway reads it, performs the handshake, and keeps the resulting
auth cookie server-side; the browser authenticates to the gateway with its own
cookie. See [How the DSH handshake works](#how-the-dsh-handshake-works).
- **Host allow-list**: DSH rejects unknown `Host` headers. The gateway forwards
the browser's `Host`, so the public hostname must be added to
`DSH_TRUSTED_HOSTS` in the template.
- **Probes target `/favicon.svg`**: `/` returns `401` without the launch token, so
a probe on `/` crash-loops the pod. `/favicon.svg` is unauthenticated, and a
`startupProbe` gates liveness/readiness so a slow cold start (plugin
activation, or a resumed sandbox) is not restarted mid-boot.
## Prerequisites
@@ -55,43 +77,49 @@ What it demonstrates, feature by feature:
- agent-sandbox with extensions installed:
```bash
VERSION=v1.0.7
kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/download/${VERSION}/sandbox-with-extensions.yaml
VERSION=v1.0.7 # AGENT_SANDBOX_VERSION in versions.env
kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/download/$VERSION/sandbox-with-extensions.yaml
kubectl -n agent-sandbox-system rollout status deployment --timeout=180s
```
Or run everything below in one shot with `./run-test-kind.sh` (expects a kind cluster + the
manifest above already applied; it creates and deletes the namespace itself).
For step 9's SSO variant you additionally need the Gateway API CRDs and a
Gateway implementation (`gatewayClassName: traefik` in the manifests); see the
[exposure README](manifests/exposure/README.md).
- Optional: run the whole resource + gateway walkthrough in one shot with
[`scripts/e2e-kind.sh`](scripts/e2e-kind.sh). It expects a kind cluster with
agent-sandbox installed, then creates and deletes the `dsh-demo` namespace
itself. It stops after the gateway layer (step 8); exposure is not applied.
## 1. Install the platform resources
First the namespace, then the shared provider key. The key reaches **every** tenant sandbox
(a warm-pool constraint: pod env is stamped before any user exists), so it is **never
committed** — generate a fresh value per install:
First the namespace, then the shared provider key. The key reaches **every**
tenant sandbox (a warm-pool constraint: pod env is stamped before any user
exists), so it is **never committed** — generate a fresh value per install:
```bash
kubectl apply -f 00-prereqs.yaml
kubectl apply -f manifests/00-prereqs.yaml
kubectl -n dsh-demo create secret generic dsh-provider-keys \
--from-literal=DEEPSEEK_API_KEY="sk-..."
kubectl apply -f 10-sandboxtemplate.yaml -f 20-sandboxwarmpool.yaml
kubectl apply -f manifests/10-sandboxtemplate.yaml -f manifests/20-sandboxwarmpool.yaml
```
Watch the two warm spares boot (first time pulls the image — subsequent operations never pay
that again; that is the point of the pool):
Watch the two warm spares boot (first time pulls the image — subsequent operations
never pay that again; that is the point of the pool):
```bash
kubectl -n dsh-demo get sandboxes -l agents.x-k8s.io/warm-pool-sandbox -w
```
Wait until both show `READY: True`. Each spare already has its PVC bound and DSH Web serving
on port `3080`.
Wait until both show `READY: True`. Each spare already has its PVC bound and DSH
Web serving on port `3080`.
## 2. "Signup": claim an instance (~2s)
A user signing up is one small claim — `warmPoolRef` plus a pod label:
```bash
kubectl apply -f 30-claim-alice.yaml
kubectl apply -f manifests/30-claim-alice.yaml
kubectl -n dsh-demo wait sandboxclaim dsh-alice --for=condition=Ready --timeout=60s
```
@@ -105,20 +133,21 @@ kubectl -n dsh-demo get sandboxes --show-labels
Three things worth noticing:
- **The sandbox keeps its pool-generated name** (`dsh-pool-…`), not the claim's name —
adoption re-labels an existing warm sandbox rather than creating one. A platform must always
resolve user → sandbox through `claim.status.sandbox.name`.
- The `agents.x-k8s.io/warm-pool-sandbox` label is **gone** from the adopted sandbox (it left
the pool's inventory), and the pool has already started replenishing back to 2 spares.
- **The sandbox keeps its pool-generated name** (`dsh-pool-…`), not the claim's
name — adoption re-labels an existing warm sandbox rather than creating one. A
platform must always resolve user → sandbox through `claim.status.sandbox.name`.
- The `agents.x-k8s.io/warm-pool-sandbox` label is **gone** from the adopted
sandbox (it left the pool's inventory), and the pool has already started
replenishing back to 2 spares.
- The pod carries the claim's `additionalPodMetadata` label
(`sandbox.users.io/dsh-user=alice`), and
`kubectl -n dsh-demo get sandbox $SB -o jsonpath='{.status.serviceFQDN}'` shows the stable
DNS name the gateway proxies to.
`kubectl -n dsh-demo get sandbox $SB -o jsonpath='{.status.serviceFQDN}'` shows
the stable DNS name the gateway proxies to.
## 3. Talk to the instance
DSH Web needs its launch token on first contact — a bare `/` returns `401`. The token is in the
pod logs:
DSH Web needs its launch token on first contact — a bare `/` returns `401`. The
token is in the pod logs:
```bash
kubectl -n dsh-demo logs "$SB" | grep -m1 '?token='
@@ -127,16 +156,16 @@ kubectl -n dsh-demo port-forward "pod/$SB" 18081:3080
# open http://localhost:18081/?token=... (from the log line)
```
> **Local port note:** DSH Web listens on `3080` **in-cluster**. If `3080` is already in use
> on your workstation (e.g. by a local DSH), forward to a different local port (here `18081`),
> never `3080`. `localhost` is always in DSH's trusted-host list, so the forwarded request is
> accepted.
> **Local port note:** DSH Web listens on `3080` **in-cluster**. If `3080` is
> already in use on your workstation (e.g. by a local DSH), forward to a
> different local port (here `18081`), never `3080`. `localhost` is always in
> DSH's trusted-host list, so the forwarded request is accepted.
> **Why the pod stays up:** this claim was applied by hand, so it carries no gateway ownership
> annotation (`aaas.example.com/managed-by`). The gateway's idle sweeper only suspends claims it
> created through `POST /users`; the resource-layer claims from these manifests are never
> touched. You can deploy the gateway (step 8) and still keep a sandbox you are port-forwarding
> to.
> **Why the pod stays up:** this claim was applied by hand, so it carries no
> gateway ownership annotation (`aaas.example.com/managed-by`). The gateway's idle
> sweeper only suspends claims it created through `POST /users`; the resource-layer
> claims from these manifests are never touched. You can deploy the gateway
> (step 8) and still keep a sandbox you are port-forwarding to.
## 4. Write state on the PVC
@@ -161,20 +190,20 @@ kubectl -n dsh-demo wait sandbox "$SB" --for=condition=Ready --timeout=180s
kubectl -n dsh-demo exec "$SB" -- cat /home/node/.dsh/marker.txt # "remember me"
```
> **Why resume works:** the pod template sets `fsGroupChangePolicy: OnRootMismatch`. DSH requires
> its `.credentials.yaml` to be `0600`, while the kubelet's default fsGroup policy widens every
> file to `0660` on each pod start — a resumed sandbox would crash-loop. See
> [Troubleshooting](#troubleshooting) for the one-time fix on sandboxes created before this
> setting existed.
> **Why resume works:** the pod template sets `fsGroupChangePolicy: OnRootMismatch`.
> DSH requires its `.credentials.yaml` to be `0600`, while the kubelet's default
> fsGroup policy widens every file to `0660` on each pod start — a resumed sandbox
> would crash-loop. See [Troubleshooting](#troubleshooting) for the one-time fix
> on sandboxes created before this setting existed.
## 7. Injection policy rejects env-carrying claims
```bash
kubectl apply -f 40-claim-rejected.yaml
kubectl apply -f manifests/40-claim-rejected.yaml
kubectl -n dsh-demo get sandboxclaim dsh-mallory \
-o jsonpath='{.status.conditions[?(@.type=="Ready")].reason}'
# EnvVarsInjectionRejected — the claim never bound a sandbox
kubectl delete -f 40-claim-rejected.yaml
kubectl delete -f manifests/40-claim-rejected.yaml
```
## 8. The gateway: signup, browser proxy, idle-suspend, wake-on-connect
@@ -182,59 +211,66 @@ kubectl delete -f 40-claim-rejected.yaml
### Deploy it
```bash
# Published image (on kind, load it into the cluster; on a real cluster it is
# pulled from GHCR):
docker pull ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway
kind load docker-image ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway --name <kind-cluster>
# Published image (on kind, load it into the cluster; on a real cluster the
# kubelet pulls it from GHCR):
GW=ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway:dsh-v0.2.1-rc.2-r2
docker pull "$GW"
kind load docker-image "$GW" --name <kind-cluster>
# ... or build it locally and tag it exactly as 50-gateway.yaml expects:
docker build -t ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway gateway/
kind load docker-image ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway --name <kind-cluster>
# ... or build it locally and tag it exactly as manifests/50-gateway.yaml
# expects (make gateway builds dsh-aaas-gateway:dev):
make gateway
docker tag dsh-aaas-gateway:dev "$GW"
kind load docker-image "$GW" --name <kind-cluster>
kubectl apply -f 50-gateway.yaml
kubectl apply -f manifests/50-gateway.yaml
kubectl -n dsh-demo rollout status deploy/dsh-gateway --timeout=120s
kubectl -n dsh-demo port-forward svc/dsh-gateway 18080:8080
```
> The gateway binds `8080`; the local port-forward uses `18080` and never `3080`.
>
> **Re-deploying after a new image?** The workflow publishes the mutable `:gateway` tag and the
> Deployment uses `imagePullPolicy: IfNotPresent`, so a node that already cached the tag keeps
> the old image across `kubectl apply`/`rollout restart`. Pin the digest for a reproducible
> rollout (the workflow's `Inspect image` step and `docker buildx imagetools inspect` both show
> it):
> **Image pull policy.** `manifests/50-gateway.yaml` uses
> `imagePullPolicy: IfNotPresent` with the release identity, so a locally built
> image must be loaded into the cluster under exactly that name (kind:
> `kind load`). A node that already cached a different tag is the usual cause of
> "my new build is not running"; pin the digest for a reproducible rollout:
>
> ```bash
> kubectl -n dsh-demo set image deploy/dsh-gateway \
> gateway=ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox@sha256:<digest>
> gateway=ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway@sha256:<digest>
> kubectl -n dsh-demo rollout status deploy/dsh-gateway --timeout=120s
> ```
>
> **Applying over an existing install?** Run `kubectl diff -f 10-sandboxtemplate.yaml`
> first. Cluster-specific fields such as `nodeAffinity` and the warm-pool replica count are
> intentionally not in the repo manifests; preserve them with a targeted patch instead of a
> **Applying over an existing install?** Run
> `kubectl diff -f manifests/10-sandboxtemplate.yaml` first. Cluster-specific
> fields such as `nodeAffinity` and the warm-pool replica count are intentionally
> not in the repo manifests; preserve them with a targeted patch instead of a
> blind apply. Moving the probes to `/favicon.svg` is the crash-loop fix.
>
> The template also sets `fsGroupChangePolicy: OnRootMismatch` (see the note in step 6).
> Existing `Sandbox` objects snapshot their pod template, so a template apply does not update
> them: patch each sandbox's `spec.podTemplate.spec.securityContext.fsGroupChangePolicy` and
> The template also sets `fsGroupChangePolicy: OnRootMismatch` (see the note in
> step 6). Existing `Sandbox` objects snapshot their pod template, so a template
> apply does not update them: patch each sandbox's
> `spec.podTemplate.spec.securityContext.fsGroupChangePolicy` and
> `chmod 600` its `.credentials.yaml`, or recreate it.
### Browser flow
Open <http://localhost:18080>. The landing page has a one-field signup form. Enter `bob`, and
the gateway:
Open <http://localhost:18080>. The landing page has a one-field signup form.
Enter `bob`, and the gateway:
1. creates the `SandboxClaim` over the warm pool,
2. waits for it to become `Ready`,
3. sets an `HttpOnly`, `SameSite=Lax` cookie `dsh_gw_session=<user>:<token>` on the browser,
3. sets an `HttpOnly`, `SameSite=Lax` cookie `dsh_gw_session=<user>:<token>` on
the browser,
4. redirects to `/u/bob/`, where the DSH Web UI loads through the gateway.
From then on every request under `/u/bob/` (assets, API calls, SSE, WebSocket) is proxied to
bob's sandbox. The DSH launch token and DSH's own auth cookie stay in the gateway; the browser
only holds the gateway cookie.
From then on every request under `/u/bob/` (assets, API calls, SSE, WebSocket) is
proxied to bob's sandbox. The DSH launch token and DSH's own auth cookie stay in
the gateway; the browser only holds the gateway cookie.
The same flow with `curl`, no JavaScript (the session cookie is the only credential):
The same flow with `curl`, no JavaScript (the session cookie is the only
credential):
```bash
JAR=$(mktemp)
@@ -252,48 +288,53 @@ curl -s -b "$JAR" -H 'Origin: http://localhost:18080' -X DELETE http://localhost
```
API clients that prefer bearer tokens can use `POST /users` with a JSON body
(`{"user":"bob"}`) and the returned one-time token, `POST /login` to exchange a token for a
cookie, and `Authorization: Bearer <token>` on subsequent calls.
(`{"user":"bob"}`) and the returned one-time token, `POST /login` to exchange a
token for a cookie, and `Authorization: Bearer <token>` on subsequent calls.
### Gateway API
| Route | Purpose |
| --- | --- |
| `GET /healthz` | Liveness/readiness; no cluster access |
| `GET /` | Landing page: signup, token login, logout |
| `GET /` | Landing page: signup, token login, logout. With SSO it auto-provisions and connects instead. |
| `POST /users` | Signup. JSON or form; `201` (JSON), `202` (waking), `303` (browser redirect), `409` if user exists; sets the session cookie |
| `POST /login` | Exchange a bearer token for the session cookie |
| `POST /logout` | Clear the session cookie |
| `POST /recreate` | SSO only: delete the authenticated user's claim (wiping workspace data) and provision a fresh one; `303` to `/u/{user}/`. Cookie-authenticated calls must be same-origin. |
| `GET /users/{user}` | Derived state (`Ready`/`Waking`/`Suspended`) |
| `DELETE /users/{user}` | Delete claim (cascades sandbox + PVC) |
| `GET /u/{user}` | `308` redirect to `/u/{user}/` |
| `GET /u/{user}/**` | HTTP/SSE proxy to the user's DSH Web |
| `WS /u/{user}/**` | WebSocket proxy to the user's DSH Web |
The full API reference, configuration table, auth semantics and failure matrix
live in the [gateway README](gateway/README.md).
### How the DSH handshake works
DSH does not expose an API for its launch token; it prints a URL containing `?token=…` in the
pod logs. For each `(user, pod UID, Host)` the gateway:
DSH does not expose an API for its launch token; it prints a URL containing
`?token=…` in the pod logs. For each `(user, pod UID, Host)` the gateway:
1. resolves the claim → sandbox → pod (annotation `agents.x-k8s.io/pod-name`, or the sandbox
name as pod name),
2. reads the pod's logs (last `DSH_LOG_TAIL_LINES` lines) and parses the most recent
`dsh web: …?token=…` line,
1. resolves the claim → sandbox → pod (annotation `agents.x-k8s.io/pod-name`, or
the sandbox name as pod name),
2. reads the pod's logs (last `DSH_LOG_TAIL_LINES` lines) and parses the most
recent `dsh web: …?token=…` line,
3. requests `http://<serviceFQDN>:3080/?token=…` with the browser's `Host`,
4. captures DSH's `dsh-auth-*` `Set-Cookie` response and caches it,
5. injects that cookie (and strips the browser's `Cookie`/`Authorization` and any `?token=`)
on every proxied HTTP and WebSocket request,
6. refreshes the handshake once if DSH answers `401`, and drops the cached cookie when the
sandbox suspends or is deleted.
5. injects that cookie (and strips the browser's `Cookie`/`Authorization` and any
`?token=`) on every proxied HTTP and WebSocket request,
6. refreshes the handshake once if DSH answers `401`, and drops the cached cookie
when the sandbox suspends or is deleted.
This is why the gateway's RBAC includes `pods` `get`/`list` and `pods/log` `get`. It is also
the most fragile part of the design; see [Limitations](#limitations).
This is why the gateway's RBAC includes `pods` `get`/`list` and `pods/log`
`get`. It is also the most fragile part of the design; see
[Limitations](#limitations).
### DSH_TRUSTED_HOSTS
DSH rejects requests whose `Host` is not allow-listed (`localhost` and `127.0.0.1` are always
trusted). Because the gateway forwards the browser's `Host`, set the public hostname in the
template:
DSH rejects requests whose `Host` is not allow-listed (`localhost` and
`127.0.0.1` are always trusted). Because the gateway forwards the browser's
`Host`, set the public hostname in the template:
```yaml
env:
@@ -301,36 +342,151 @@ env:
value: "dsh.example.com" # comma or space separated; empty for port-forward only
```
The same applies to a local `docker run`: the printed token URL uses `DSH_PUBLIC_URL` if set,
otherwise `http://localhost:${DSH_PORT}`.
The same applies to a local `docker run`: the printed token URL uses
`DSH_PUBLIC_URL` if set, otherwise `http://localhost:<DSH_PORT>`.
### WebSocket
The gateway is a Starlette ASGI app and proxies WebSocket upgrades with a bidirectional pump, so
the DSH web GUI is fully interactive. Requirements:
The gateway is a Starlette ASGI app and proxies WebSocket upgrades with a
bidirectional pump, so the DSH web GUI is fully interactive. Requirements:
- the ingress/load balancer in front of the gateway must pass `Upgrade`/`Connection` headers;
`60-ingress.yaml` is a starting point (Traefik handles upgrades by default),
- idle suspension is disabled for users with in-flight proxied requests or open WebSockets, so
a long-running chat stream is not cut off mid-turn,
- the upstream upgrade dials the sandbox **pod IP**, forwards no `Host`/`Origin`, and performs
the launch-token handshake under that same `pod-IP:3080` authority. DSH fences on `Host`
(only its bind address, loopback, or an allow-listed host passes) and binds its auth cookie to
the request authority (`payload.authority === authority`), while `websockets` derives `Host`
from the URI and cannot be overridden. A browser-host cookie or forwarded origin makes the
upgrade fail (`401`/`403`, surfaced as close `1011`), which the client retries forever as
`[connection] connection lost, retry #N`,
- missing/invalid gateway auth is rejected right after the upgrade with close code `4401`
(WebSocket auth has no HTTP `401` path); cross-origin or Origin-less upgrades with the cookie
are rejected the same way (CSWSH protection), so clients must send a matching `Origin`;
upstream bridging failures close with `1011`.
- the ingress/load balancer in front of the gateway must pass `Upgrade`/`Connection`
headers; [`manifests/exposure/ingress/60-ingress.yaml`](manifests/exposure/ingress/60-ingress.yaml)
is a starting point (Traefik handles upgrades by default),
- idle suspension is disabled for users with in-flight proxied requests or open
WebSockets, so a long-running chat stream is not cut off mid-turn,
- the upstream upgrade dials the sandbox **pod IP**, forwards no `Host`/`Origin`,
and performs the launch-token handshake under that same `pod-IP:3080` authority.
DSH fences on `Host` (only its bind address, loopback, or an allow-listed host
passes) and binds its auth cookie to the request authority
(`payload.authority === authority`), while `websockets` derives `Host` from the
URI and cannot be overridden. A browser-host cookie or forwarded origin makes
the upgrade fail (`401`/`403`, surfaced as close `1011`), which the client
retries forever as `[connection] connection lost, retry #N`,
- missing/invalid gateway auth is rejected right after the upgrade with close code
`4401` (WebSocket auth has no HTTP `401` path); cross-origin or Origin-less
upgrades with the cookie are rejected the same way (CSWSH protection), so
clients must send a matching `Origin`; upstream bridging failures close with
`1011`.
### Path routing
Each user's DSH Web is served under `/u/<user>/`. DSH injects `<base href="./">` and uses
relative asset/API URLs, so the UI works under a path prefix without a per-user hostname or
wildcard DNS. The gateway also rewrites absolute-path `Location` headers into `/u/<user>/…` and
redirects `/u/<user>` to the trailing-slash form with `308` so relative URLs resolve correctly.
Each user's DSH Web is served under `/u/<user>/`. DSH injects `<base href="./">`
and uses relative asset/API URLs, so the UI works under a path prefix without a
per-user hostname or wildcard DNS. The gateway also rewrites absolute-path
`Location` headers into `/u/<user>/…` and redirects `/u/<user>` to the
trailing-slash form with `308` so relative URLs resolve correctly.
## 9. Expose it: choose exactly one variant
The base manifests expose nothing outside the cluster (`dsh-gateway` is a
ClusterIP Service on port `8080`). Pick **one** variant — they are alternatives,
not a stack. Details, commands and the switch procedure are in the
[exposure README](manifests/exposure/README.md).
> Apply the base resources first, then the variant **by path**. Do not use
> `kubectl apply -R`: it would apply both variants and create two paths to the
> same gateway, which is unsafe once SSO is enabled.
### Option A — plain Ingress (gateway's own session auth)
```bash
# Replace the dsh.example.com host and TLS Secret (or drop the tls: block for
# plain HTTP), then:
kubectl apply -f manifests/exposure/ingress/
```
1. Set `DSH_TRUSTED_HOSTS` in `manifests/10-sandboxtemplate.yaml` to the same
public host (the gateway forwards the browser's `Host`) and re-apply the
template.
2. Set `COOKIE_SECURE=true` on the gateway once the Ingress serves HTTPS:
`kubectl -n dsh-demo set env deploy/dsh-gateway COOKIE_SECURE=true`.
(The base manifest keeps `false` so the plain-HTTP port-forward demo works.)
3. Verify: `curl -sI https://<host>/healthz` returns `200`; sign up in a browser
at `https://<host>/`.
Keep the router free of buffering/compression middlewares: they break WebSocket
upgrades and streaming.
### Option B — Gateway API + GitHub SSO (oauth2-proxy)
`manifests/exposure/gateway-api-sso/` puts a TLS `Gateway` + `HTTPRoute` in
front of **oauth2-proxy**, which authenticates the browser against GitHub
(restricted to a configured organization) and forwards to the gateway with
`X-Forwarded-User` set. The gateway trusts that header only when
`AUTH_PROXY_HEADER` is set (SSO is **opt-in**).
Prerequisites:
- Gateway API CRDs and a Gateway implementation; the manifests assume
`gatewayClassName: traefik` (the CRD version is pinned as
`GATEWAY_API_VERSION` in `versions.env`).
- A GitHub OAuth App whose homepage/callback match the public FQDN
(`https://<host>/` and `https://<host>/oauth2/callback`).
- The oauth2-proxy credentials Secret (template:
`73-oauth2-proxy-secret.yaml.example`) and a TLS Secret for the Gateway
listener (template: `74-tls-secret.yaml.example`).
Steps:
```bash
# 1. Real Secrets; the committed *.yaml.example files are templates only.
kubectl -n dsh-demo create secret generic oauth2-proxy \
--from-literal=client-id='<GitHub OAuth App client ID>' \
--from-literal=client-secret='<GitHub OAuth App client secret>' \
--from-literal=cookie-secret="$(python3 -c 'import secrets; print(secrets.token_urlsafe(32))')"
kubectl -n dsh-demo create secret tls dsh-demo-tls --cert=tls.crt --key=tls.key
# 2. Replace dsh-demo.example.com (70/71/72) and your-github-org (72).
# 3. Enable SSO on the gateway and trust the public host in the template.
kubectl -n dsh-demo set env deploy/dsh-gateway \
AUTH_PROXY_HEADER=X-Forwarded-User COOKIE_SECURE=true
# ... and set DSH_TRUSTED_HOSTS=dsh-demo.example.com in
# manifests/10-sandboxtemplate.yaml, then re-apply the template.
# 4. Apply the variant.
kubectl apply -f manifests/exposure/gateway-api-sso/
```
Verification:
```bash
kubectl get gateway dsh-gateway -n dsh-demo -o yaml # Programmed=True
kubectl get httproute -n dsh-demo -o wide # backend oauth2-proxy:4180, Accepted
kubectl get deploy oauth2-proxy -n dsh-demo # 1/1 Ready
kubectl get endpoints oauth2-proxy -n dsh-demo # backend present
kubectl -n dsh-demo logs deploy/oauth2-proxy --tail=50 # no config errors
```
Browser check: open `https://<host>/` — you are redirected to GitHub, and after
signing in as a member of the allowed organization the gateway auto-provisions
your sandbox (the first page can show `Provisioning`) and connects you. A
non-member is denied before reaching the gateway. GitHub logins are lowercase
DNS-1123 labels, so they map 1:1 onto the gateway's `user` validation. The SSO
landing page also offers **Recreate sandbox** (`POST /recreate`), which deletes
the claim — wiping the workspace PVC — and provisions a fresh one.
Sign-out: the gateway `/logout` clears `dsh_gw_session`; oauth2-proxy's session
can be cleared at `/oauth2/sign_out`.
> **SSO trust boundary — read this before enabling it.** After SSO is on, the
> `dsh-gateway` Service must be reachable **only** through oauth2-proxy (it is
> ClusterIP). Anyone who can reach `dsh-gateway:8080` directly can send
> `X-Forwarded-User: <victim>` and be treated as that user. Never apply the
> Ingress variant (or any other route to the Service) while SSO is enabled, and
> do not expose the gateway Service with `kubectl port-forward` on a shared
> machine while `AUTH_PROXY_HEADER` is set. To turn SSO off, remove the variable
> (`kubectl -n dsh-demo set env deploy/dsh-gateway AUTH_PROXY_HEADER-`) and
> delete the SSO variant; the gateway falls back to bearer-token/session auth.
### Switching variants
Delete the variant you had before applying the other one
(`kubectl delete -f manifests/exposure/ingress/`), revert
`AUTH_PROXY_HEADER`/`COOKIE_SECURE`, and verify with
`kubectl -n dsh-demo get ingress,gateway,httproute` that exactly one path to the
gateway remains.
## Troubleshooting
@@ -346,36 +502,65 @@ redirects `/u/<user>` to the trailing-slash form with `308` so relative URLs res
| Console spams `[connection] connection lost, retry #N` | The upstream WS upgrade is being rejected, surfaced as close `1011`. The gateway dials the sandbox **pod IP**, forwards no `Host`/`Origin`, and handshakes under the same `pod-IP:3080` authority DSH binds its cookie to; an older gateway image, or a sandbox with an empty `status.podIPs`, hits this loop |
| `manifest.webmanifest` returns `401` in the console | Browsers fetch the Web App Manifest without credentials, so the gateway's session cookie is absent and the request fails closed. The UI is unaffected; only the PWA install prompt is lost |
| Signup is slow / `202` then `Waking` | Warm pool exhausted (`kubectl get sandboxes -l agents.x-k8s.io/warm-pool-sandbox`); scale the pool |
| `403` from the Kubernetes API in gateway logs | The gateway ServiceAccount is missing claim/sandbox/pod-log verbs (see `50-gateway.yaml`) |
| Step 3: `pod "$SB" not found` after deploying the gateway | The claim was suspended by the idle sweeper. Claims created by the gateway carry `aaas.example.com/managed-by=dsh-gateway`; hand-applied claims (steps 1–7) do not and are never swept. Upgrade the gateway to a build that honors the marker, or resume the sandbox: `kubectl -n dsh-demo patch sandbox "$SB" --type merge -p '{"spec":{"operatingMode":"Running"}}'` |
| Resume crash-loops: `credentials-local: … is readable beyond its owner (mode 660)` | The pod template lacks `fsGroupChangePolicy: OnRootMismatch`, so the kubelet's default fsGroup policy rewrote `.credentials.yaml` to `0660`; DSH requires `0600` and refuses to start. New installs get the policy from the template. For an existing sandbox: suspend it, run a throwaway pod on `dsh-home-$SB` to `chmod 600 /home/node/.dsh/.credentials.yaml`, patch the sandbox's `podTemplate` with the policy, then resume. |
| `403` from the Kubernetes API in gateway logs | The gateway ServiceAccount is missing claim/sandbox/pod-log verbs (see `manifests/50-gateway.yaml`) |
| Gateway pod `ErrImagePull` / `ImagePullBackOff` (especially on kind) | The manifest uses `imagePullPolicy: IfNotPresent` with the release identity. Build with `make gateway`, retag to exactly the manifest's image, `kind load docker-image <that-ref>`, or set the Deployment image to your local tag / pinned digest. On a real cluster, make sure the GHCR package is readable (public or a pull secret) |
| Step 3: `pod "$SB" not found` after deploying the gateway | The claim was suspended by the idle sweeper. Claims created by the gateway carry `aaas.example.com/managed-by=dsh-gateway`; hand-applied claims (steps 1–7) do not and are never swept. Resume the sandbox: `kubectl -n dsh-demo patch sandbox "$SB" --type merge -p '{"spec":{"operatingMode":"Running"}}'` |
| Resume crash-loops: `credentials-local: … is readable beyond its owner (mode 660)` | The pod template lacks `fsGroupChangePolicy: OnRootMismatch`, so the kubelet's default fsGroup policy rewrote `.credentials.yaml` to `0660`; DSH requires `0600` and refuses to start. New installs get the policy from the template. For an existing sandbox: suspend it, run a throwaway pod on `dsh-home-$SB` to `chmod 600 /home/node/.dsh/.credentials.yaml`, patch the sandbox's `podTemplate` with the policy, then resume |
| SSO: GitHub login succeeds but the gateway asks for a token / returns `401` to `/recreate` | `AUTH_PROXY_HEADER` is not set on `deploy/dsh-gateway` (SSO is opt-in). Set `AUTH_PROXY_HEADER=X-Forwarded-User` and confirm the request actually reaches the gateway through oauth2-proxy |
| SSO: gateway treats a caller as another user / bad `X-Forwarded-User` accepted | Trust boundary violation: something other than oauth2-proxy can reach `dsh-gateway:8080`. Remove the second path (Ingress/Gateway/port-forward), keep the Service ClusterIP, and only re-enable SSO once the proxy is the sole entry point. Never put a trusted identity header in front of a service that is directly reachable |
| SSO: oauth2-proxy `cookie_secret must be 16, 24, or 32 bytes ... but is 44 bytes` | The cookie secret was generated as standard base64. Use URL-safe unpadded base64 (see `73-oauth2-proxy-secret.yaml.example`) |
| SSO: redirect loop (`ERR_TOO_MANY_REDIRECTS`) | oauth2-proxy `--redirect-url` / callback does not match the public FQDN, or TLS is not terminating at the Gateway listener; fix the OAuth App URLs and the manifest placeholders |
| SSO: DSH rejects the public host behind the proxy | Add the public hostname to `DSH_TRUSTED_HOSTS` in `manifests/10-sandboxtemplate.yaml`; the gateway forwards the browser's `Host` |
## Limitations
Deliberately **not** production code:
- **Single replica.** Idle clocks, in-flight tracking, and the handshake cache are in memory.
Running two gateway replicas would double-suspend or lose wake tracking; scale up only after
moving that state to a shared store.
- **Log scraping.** The launch token is recovered from pod logs. Log rotation, a restarted pod,
or a changed DSH log format can break the handshake until the next `401` refresh.
- **Shared provider key.** `DEEPSEEK_API_KEY` is shared by all tenants via the template. Real
multi-tenancy needs per-tenant credentials or an egress proxy.
- **No TLS in the gateway.** Terminate TLS at an Ingress/reverse proxy and set
- **Single replica.** Idle clocks, in-flight tracking, and the handshake cache
are in memory. Running two gateway replicas would double-suspend or lose wake
tracking; scale up only after moving that state to a shared store.
- **Log scraping.** The launch token is recovered from pod logs. Log rotation, a
restarted pod, or a changed DSH log format can break the handshake until the
next `401` refresh.
- **Shared provider key.** `DEEPSEEK_API_KEY` is shared by all tenants via the
template. Real multi-tenancy needs per-tenant credentials or an egress proxy.
- **No TLS in the gateway.** Terminate TLS at an Ingress or Gateway and set
`COOKIE_SECURE=true`.
- **In-memory user directory.** Users are claims + annotations; there is no database, password
reset, or admin API. Tokens are stored only as SHA-256 hashes in claim annotations.
- **Bearer tokens in URLs.** Avoid putting the signup token in a URL/log; the browser flow uses
the `HttpOnly` cookie instead.
- **SSO trust is hop-based.** `AUTH_PROXY_HEADER` trusts whoever sets the header;
the only guarantee is that the gateway Service is not directly reachable.
There is no signature, mTLS, or NetworkPolicy enforcing that in this example.
- **OAuth org is the only authorization.** Any GitHub user in the configured
organization gets an auto-provisioned sandbox; there is no per-user allow-list,
quota, or admin API.
- **In-memory user directory.** Users are claims + annotations; there is no
database, password reset, or admin API. Tokens are stored only as SHA-256
hashes in claim annotations.
- **Bearer tokens in URLs.** Avoid putting the signup token in a URL/log; the
browser flow uses the `HttpOnly` cookie instead.
## Cost model
The same economics as the upstream pattern: suspend-when-idle plus warm pools turn an
always-on instance into a sub-second cold-start pause per return-after-absence, at a fraction
of the cost.
The same economics as the upstream pattern: suspend-when-idle plus warm pools turn
an always-on instance into a sub-second cold-start pause per return-after-absence,
at a fraction of the cost.
## Cleanup
```bash
# Delete an exposure variant first if you applied one (optional; namespace
# deletion below removes everything):
kubectl delete -f manifests/exposure/ingress/ # or gateway-api-sso/
# Then the whole example:
kubectl delete namespace dsh-demo
```
If you created the oauth2-proxy/TLS Secrets or the GitHub OAuth App for step 9,
delete the Secrets with the namespace and revoke the OAuth App in GitHub.
## More
- [Gateway README](gateway/README.md) — API, configuration, auth/SSO, failure
semantics, limitations.
- [Architecture](docs/architecture.md) — container/sequence diagrams, ADRs,
security and RBAC, failure modes, evolution path.
- [Exposure README](manifests/exposure/README.md) — the two variants in detail.