deepseek-kubernetes-sandbox/examples/dsh-agents-as-a-service/README.md

# DeepSeek Harness Web as a Service: the multi-user platform pattern

This example distills the upstream
[hermes-agents-as-a-service](https://github.com/kubernetes-sigs/agent-sandbox/tree/main/examples/hermes-agents-as-a-service)
platform pattern into pure agent-sandbox resources + kubectl, running **DeepSeek
Harness Web (DSH)** instead of Hermes. Every user of such a platform gets a
personal, long-lived DSH Web instance that is **provisioned in ~2 seconds from a
warm pool, suspended when idle to save cost, and woken transparently when the
user returns** — with all state (conversations, sessions, config, and the agent
workspace) surviving on the per-user PVC at `/home/node/.dsh` + `/workspace`.
Suspend-when-idle plus the warm pool keep cost proportional to use: a return
after absence pays a short cold-start pause, not an always-on instance.

It ships in three layers:

1. **The resource layer** (steps 1–7): pure manifests + kubectl, so every
   mechanism is visible.
2. **The service layer** (step 8): a small included **gateway** (`gateway/`) that
   makes the platform browser-usable — a signup page/API, `HttpOnly` cookie
   sessions, a `/u/{user}/**` proxy for HTTP **and WebSocket** that transparently
   **wakes suspended instances on connect**, and an idle sweeper that suspends
   them again. The gateway performs the DSH launch-token handshake server-side,
   so the browser never sees the token.
3. **The exposure layer** (step 9): choose exactly one way in — a plain Traefik
   Ingress, or Gateway API + oauth2-proxy (GitHub SSO).

All manifests live under [`manifests/`](manifests/):

```
manifests/
+-- 00-prereqs.yaml              # namespace + empty provider-key Secret
+-- 10-sandboxtemplate.yaml      # the DSH Web blueprint
+-- 20-sandboxwarmpool.yaml      # 2 pre-warmed spares
+-- 30-claim-alice.yaml          # one user
+-- 40-claim-rejected.yaml       # negative demo: env injection rejected
+-- 50-gateway.yaml              # Deployment/RBAC/Service for the gateway
+-- optional/
|   +-- README.md                # opt-in manifests, applied by path
|   +-- network-policy/10-sandbox-networkpolicy.yaml
+-- exposure/
    +-- README.md                 # the two exposure variants, when to use which
    +-- ingress/60-ingress.yaml
    +-- ingress/61-networkpolicy.yaml
    +-- gateway-api-sso/70-gateway.yaml ... 75-networkpolicies.yaml
```

What it demonstrates, feature by feature:

| agent-sandbox feature | Role in the platform |
| --- | --- |
| `SandboxTemplate` with `volumeClaimTemplates` | Identical DSH pods; per-user state at `/home/node/.dsh` and the agent workspace at `/workspace`, on one PVC (the workspace is a `subPath` mount, so DSH state stays private) |
| `envVarsInjectionPolicy` / `volumeClaimTemplatesPolicy: Disallowed` | Claims that would bypass the warm pool are **rejected loudly**, not cold-started silently |
| `service: true` | Stable per-sandbox DNS (`status.serviceFQDN`) for the gateway to route on |
| `SandboxWarmPool` | Signup latency is pre-paid: image pull, PVC provision, DSH boot all happen before any user exists |
| `SandboxClaim` + `warmPoolRef` | "Signup" = create a ~10-line claim; adoption is a label flip, Ready in ~2s |
| `additionalPodMetadata` | The one warm-compatible per-claim customization (user label on the pod) |
| `Sandbox.spec.operatingMode` | Suspend = delete only the pod (PVC + Service survive); resume = recreate pod, reattach the same PVC |
| Cascade delete | Deleting the claim garbage-collects sandbox **and** PVC — account deletion in one call |

## Differences from the Hermes example

- **Multi-arch**: the DSH image is built for amd64 and arm64, so there is no
  `kubernetes.io/arch: amd64` nodeSelector.
- **Single port**: DSH Web serves everything on `3080`; the gateway proxies to one
  upstream instead of splitting `/v1/*` (API) from the dashboard.
- **Shared provider keys**: `DEEPSEEK_API_KEY` plus any keys in the optional
  `dsh-provider-keys-extra` Secret are injected via `envFrom` into every
  sandbox. There is no per-tenant key, dashboard basic-auth, session-cookie
  secret, or API-server key.
- **Launch-token handshake**: DSH gates `/` behind a token it prints only in its
  logs. The gateway reads it, performs the handshake, and keeps the resulting
  auth cookie server-side; the browser authenticates to the gateway with its own
  cookie. See [How the DSH handshake works](#how-the-dsh-handshake-works).
- **Host allow-list**: DSH rejects unknown `Host` headers. The gateway forwards
  the browser's `Host`, so the public hostname must be added to
  `DSH_TRUSTED_HOSTS` in the template.
- **Probes target `/favicon.svg`**: `/` returns `401` without the launch token, so
  a probe on `/` crash-loops the pod. `/favicon.svg` is unauthenticated, and a
  `startupProbe` gates liveness/readiness so a slow cold start (plugin
  activation, or a resumed sandbox) is not restarted mid-boot.

## Prerequisites

Run the walkthrough from this example's directory (`examples/dsh-agents-as-a-service/`).

- A Kubernetes cluster with a default StorageClass (kind and GKE both work as-is).
- agent-sandbox with extensions installed:

  ```bash
  . ../../versions.env
  kubectl apply -f "https://github.com/kubernetes-sigs/agent-sandbox/releases/download/${AGENT_SANDBOX_VERSION}/sandbox-with-extensions.yaml"
  kubectl -n agent-sandbox-system rollout status deployment --timeout=180s
  ```

  For step 9's SSO variant you additionally need the Gateway API CRDs and a
  Gateway implementation (`gatewayClassName: traefik` in the manifests); see the
  [exposure README](manifests/exposure/README.md).

- Optional: run the whole resource + gateway walkthrough in one shot with
  [`scripts/e2e-kind.sh`](scripts/e2e-kind.sh). It expects a kind cluster with
  agent-sandbox installed, then creates and deletes the `dsh-demo` namespace
  itself. It stops after the gateway layer (step 8); exposure is not applied.

## 1. Install the platform resources

First the namespace, then the shared provider keys. The keys reach **every**
tenant sandbox (a warm-pool constraint: pod env is stamped before any user
exists), so they are **never committed** — generate fresh values per install.
`00-prereqs.yaml` already creates the empty `dsh-provider-keys` Secret, so use
an upsert:

```bash
kubectl apply -f manifests/00-prereqs.yaml
kubectl -n dsh-demo create secret generic dsh-provider-keys \
  --from-literal=DEEPSEEK_API_KEY="sk-..." \
  --dry-run=client -o yaml | kubectl apply -f -
```

Optional: additional providers (OpenRouter, Gemini, NVIDIA, OpenAI, Anthropic,
...) go in a second Secret. It is referenced with `optional: true`, so the
sandboxes start without it, and every key becomes an env var:

```bash
kubectl -n dsh-demo create secret generic dsh-provider-keys-extra \
  --from-literal=OPENROUTER_API_KEY="sk-or-..." \
  --from-literal=GEMINI_API_KEY="..." \
  --from-literal=NVIDIA_API_KEY="nvapi-..." \
  --dry-run=client -o yaml | kubectl apply -f -
```

Create both Secrets **before** the template/warm pool. A later Secret change
does not update running pods: delete the warm-pool spares and recreate claimed
pods to pick up new keys. Keys are shared by all tenants; a key saved in DSH's
credential store wins over env. See
[optional hardening](docs/optional-hardening.md) for the full env-name list and
the refresh procedure.

```bash
kubectl apply -f manifests/10-sandboxtemplate.yaml -f manifests/20-sandboxwarmpool.yaml
```

Watch the two warm spares boot (first time pulls the image — subsequent operations
never pay that again; that is the point of the pool):

```bash
kubectl -n dsh-demo get sandboxes -l agents.x-k8s.io/warm-pool-sandbox -w
```

Wait until both show `READY: True`. Each spare already has its PVC bound and DSH
Web serving on port `3080`.

## 2. "Signup": claim an instance (~2s)

A user signing up is one small claim — `warmPoolRef` plus a pod label:

```bash
kubectl apply -f manifests/30-claim-alice.yaml
kubectl -n dsh-demo wait sandboxclaim dsh-alice --for=condition=Ready --timeout=60s
```

Inspect what happened:

```bash
SB=$(kubectl -n dsh-demo get sandboxclaim dsh-alice -o jsonpath='{.status.sandbox.name}')
echo "alice's sandbox: $SB"
kubectl -n dsh-demo get sandboxes --show-labels
```

Three things worth noticing:

- **The sandbox keeps its pool-generated name** (`dsh-pool-…`), not the claim's
  name — adoption re-labels an existing warm sandbox rather than creating one. A
  platform must always resolve user → sandbox through `claim.status.sandbox.name`.
- The `agents.x-k8s.io/warm-pool-sandbox` label is **gone** from the adopted
  sandbox (it left the pool's inventory), and the pool has already started
  replenishing back to 2 spares.
- The pod carries the claim's `additionalPodMetadata` label
  (`sandbox.users.io/dsh-user=alice`), and
  `kubectl -n dsh-demo get sandbox $SB -o jsonpath='{.status.serviceFQDN}'` shows
  the stable DNS name the gateway proxies to.

## 3. Talk to the instance

DSH Web needs its launch token on first contact — a bare `/` returns `401`. The
token is in the pod logs:

```bash
kubectl -n dsh-demo logs "$SB" | grep -m1 '?token='
# dsh web: http://<pod-ip>:3080/?token=...
kubectl -n dsh-demo port-forward "pod/$SB" 18081:3080
# open http://localhost:18081/?token=... (from the log line)
```

> **Local port note:** DSH Web listens on `3080` **in-cluster**. If `3080` is
> already in use on your workstation (e.g. by a local DSH), forward to a
> different local port (here `18081`), never `3080`. `localhost` is always in
> DSH's trusted-host list, so the forwarded request is accepted.

> **Why the pod stays up:** this claim was applied by hand, so it carries no
> gateway ownership annotation (`aaas.example.com/managed-by`). The gateway's idle
> sweeper only suspends claims it created through `POST /users`; the resource-layer
> claims from these manifests are never touched. You can deploy the gateway
> (step 8) and still keep a sandbox you are port-forwarding to.

## 4. Write state on the PVC

```bash
kubectl -n dsh-demo exec "$SB" -- printenv HOME                        # /workspace
kubectl -n dsh-demo exec "$SB" -- sh -c 'echo "remember me" > /home/node/.dsh/marker.txt'
kubectl -n dsh-demo exec "$SB" -- sh -c 'echo "remember me too" > /workspace/marker.txt'
```

`/workspace` (the DSH image's `WORKDIR`, where agents keep files) is the same
`dsh-home` PVC mounted again with `subPath: workspace`. The two trees stay
separate, so `.credentials.yaml` and the session store are never part of the
agent-visible workspace.

The template also sets `HOME=/workspace`. DSH's directory picker (the folder
explorer used to choose a workspace) has no start-path setting: when it opens,
its browse backend lists `$HOME`, so with the image default (`/home/node`, which
is not on a volume) it would start outside the persisted tree. Pointing `HOME`
at the workspace makes the picker open on it (labelled *Home* in its breadcrumbs)
and makes `~` in agent commands mean `/workspace`. The trade-off is that tools
keeping caches or config under `$HOME` (`npm`/`pnpm`, `uv`, `git`, `gh`, and
DSH's global `~/.agents` instructions and skills) write dotfiles onto the PVC
inside the agent workspace, where the agent and the file explorer can see them.
`DSH_HOME` is explicit (`/home/node/.dsh`), so sessions, config and credentials
are unaffected. Pod env is stamped when a sandbox is created: a sandbox from
before this setting keeps `HOME=/home/node` until it is recreated (see
[Troubleshooting](#troubleshooting)).

## 5. Suspend: delete the pod, keep the PVC + Service

```bash
kubectl -n dsh-demo patch sandbox "$SB" --type merge -p '{"spec":{"operatingMode":"Suspended"}}'
kubectl -n dsh-demo wait --for=delete "pod/$SB" --timeout=120s
kubectl -n dsh-demo get pvc "dsh-home-$SB"   # still here
kubectl -n dsh-demo get svc "$SB"            # still here
```

## 6. Resume and verify state survived

```bash
kubectl -n dsh-demo patch sandbox "$SB" --type merge -p '{"spec":{"operatingMode":"Running"}}'
kubectl -n dsh-demo wait sandbox "$SB" --for=condition=Ready --timeout=180s
kubectl -n dsh-demo exec "$SB" -- cat /home/node/.dsh/marker.txt   # "remember me"
kubectl -n dsh-demo exec "$SB" -- cat /workspace/marker.txt        # "remember me too"
```

> **Why resume works:** the pod template sets `fsGroupChangePolicy: OnRootMismatch`;
> without it the kubelet widens `.credentials.yaml` to `0660` and DSH crash-loops.
> See [Troubleshooting](#troubleshooting) for the one-time fix on older sandboxes.

## 7. Injection policy rejects env-carrying claims

```bash
kubectl apply -f manifests/40-claim-rejected.yaml
kubectl -n dsh-demo get sandboxclaim dsh-mallory \
  -o jsonpath='{.status.conditions[?(@.type=="Ready")].reason}'
# EnvVarsInjectionRejected — the claim never bound a sandbox
kubectl delete -f manifests/40-claim-rejected.yaml
```

## 8. The gateway: signup, browser proxy, idle-suspend, wake-on-connect

### Deploy it

```bash
# Published image (on kind, load it into the cluster; on a real cluster the
# kubelet pulls it from GHCR):
GW=ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway:dsh-v0.2.1-rc.5
docker pull "$GW"
kind load docker-image "$GW" --name <kind-cluster>

# ... or build it locally and tag it exactly as manifests/50-gateway.yaml
# expects (make gateway builds dsh-aaas-gateway:dev):
make -C ../.. gateway
docker tag dsh-aaas-gateway:dev "$GW"
kind load docker-image "$GW" --name <kind-cluster>

kubectl apply -f manifests/50-gateway.yaml
kubectl -n dsh-demo rollout status deploy/dsh-gateway --timeout=120s
kubectl -n dsh-demo port-forward svc/dsh-gateway 18080:8080
```

> The gateway binds `8080`; the local port-forward uses `18080` and never `3080`.
>
> **Image pull policy.** `manifests/50-gateway.yaml` uses
> `imagePullPolicy: IfNotPresent` with the release identity, so a locally built
> image must be loaded into the cluster under exactly that name (kind:
> `kind load`). A node that already cached a different tag is the usual cause of
> "my new build is not running"; pin the digest for a reproducible rollout:
>
> ```bash
> kubectl -n dsh-demo set image deploy/dsh-gateway \
>   gateway=ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway@sha256:<digest>
> kubectl -n dsh-demo rollout status deploy/dsh-gateway --timeout=120s
> ```
>
> **Applying over an existing install?** Run
> `kubectl diff -f manifests/10-sandboxtemplate.yaml` first. Cluster-specific
> fields such as `nodeAffinity` and the warm-pool replica count are intentionally
> not in the repo manifests; preserve them with a targeted patch instead of a
> blind apply. Moving the probes to `/favicon.svg` is the crash-loop fix.
>
> The template also sets `fsGroupChangePolicy: OnRootMismatch` (see the note in
> step 6). Existing `Sandbox` objects snapshot their pod template, so a template
> apply does not update them: patch each sandbox's
> `spec.podTemplate.spec.securityContext.fsGroupChangePolicy` and
> `chmod 600` its `.credentials.yaml`, or recreate it.
>
> The optional gVisor/`nodeSelector` fields are pod-spec changes too: existing
> Sandboxes snapshot them, and warm-pool spares are not replaced by a template
> edit. See [optional hardening](docs/optional-hardening.md) for the patch and
> recreation commands. A provider-key Secret change likewise needs pod
> recreation to take effect.

### Browser flow

Open <http://localhost:18080>. The landing page has a one-field signup form.
Enter `bob`, and the gateway:

1. creates the `SandboxClaim` over the warm pool,
2. waits for it to become `Ready`,
3. sets an `HttpOnly`, `SameSite=Lax` cookie `dsh_gw_session=<user>:<token>` on
   the browser,
4. redirects to `/u/bob/`, where the DSH Web UI loads through the gateway.

From then on every request under `/u/bob/` (assets, API calls, SSE, WebSocket) is
proxied to bob's sandbox. The DSH launch token and DSH's own auth cookie stay in
the gateway; the browser only holds the gateway cookie.

The same flow with `curl`, no JavaScript (the session cookie is the only
credential):

```bash
JAR=$(mktemp)
# signup (form POST, exactly what the landing page sends)
curl -s -i -c "$JAR" -H 'Origin: http://localhost:18080' \
  -d 'user=bob' http://localhost:18080/users
# proxy: bob's DSH Web through the gateway
curl -s -b "$JAR" http://localhost:18080/u/bob/ | head -5
# without the cookie the gateway fails closed
curl -s -o /dev/null -w '%{http_code}\n' http://localhost:18080/u/bob/   # 401
# derived state: Ready / Waking / Suspended / ...
curl -s -b "$JAR" http://localhost:18080/users/bob
# delete bob (cascades sandbox + PVC)
curl -s -b "$JAR" -H 'Origin: http://localhost:18080' -X DELETE http://localhost:18080/users/bob
```

API clients that prefer bearer tokens can use `POST /users` with a JSON body
(`{"user":"bob"}`) and the returned one-time token, `POST /login` to exchange a
token for a cookie, and `Authorization: Bearer <token>` on subsequent calls.

### Gateway routes

The browser flow above uses the gateway's main routes. The full route table,
status codes, configuration and failure matrix live in the
[gateway README](gateway/README.md).

### How the DSH handshake works

DSH prints its launch token only in pod logs. The gateway resolves
claim → sandbox → pod, reads the pod-log tail, performs the `?token=` handshake
under the browser's `Host`, caches `dsh-auth-*` per `(user, pod UID, Host)`, and
refreshes once on a `401`. That is why its RBAC includes `pods` `get`/`list` and
`pods/log` `get`; it is also the most fragile part of the design (see
[Limitations](#limitations) and the
[architecture ADR](docs/architecture.md#adr-3-server-side-launch-token-handshake-instead-of-exposing-the-token)).

### DSH_TRUSTED_HOSTS

DSH rejects requests whose `Host` is not allow-listed (`localhost` and
`127.0.0.1` are always trusted). Because the gateway forwards the browser's
`Host`, set the public hostname in the template:

```yaml
env:
  - name: DSH_TRUSTED_HOSTS
    value: "dsh.example.com"   # comma or space separated; empty for port-forward only
```

The same applies to a local `docker run`: the printed token URL uses
`DSH_PUBLIC_URL` if set, otherwise `http://localhost:<DSH_PORT>`.

### WebSocket

The gateway proxies WebSocket upgrades with a bidirectional pump, so the DSH web
GUI is fully interactive. The entry point must pass `Upgrade`/`Connection`
headers; [`manifests/exposure/ingress/60-ingress.yaml`](manifests/exposure/ingress/60-ingress.yaml)
is a starting point (Traefik handles upgrades by default). Idle suspension is
disabled while a proxied request or socket is in flight, so a long-running turn is
not cut off.

The upstream upgrade dials the sandbox **pod IP**, forwards no `Host`/`Origin`,
and performs the launch-token handshake under the same `pod-IP:3080` authority.
Missing or cross-origin auth closes with `4401`; upstream bridging failures close
with `1011`. See the [gateway README](gateway/README.md#proxy-failure-semantics)
for the full matrix.

### Path routing

Each user's DSH Web is served under `/u/<user>/`. DSH injects `<base href="./">`
and uses relative asset/API URLs, so the UI works under a path prefix without a
per-user hostname or wildcard DNS. The gateway also rewrites absolute-path
`Location` headers into `/u/<user>/…` and redirects `/u/<user>` to the
trailing-slash form with `308` so relative URLs resolve correctly.

## 9. Expose it: choose exactly one variant

The base manifests expose nothing outside the cluster (`dsh-gateway` is a
ClusterIP Service on port `8080`). Pick **one** variant — they are alternatives,
not a stack. Details, commands and the switch procedure are in the
[exposure README](manifests/exposure/README.md).

> Apply the base resources first, then the variant **by path**. Do not use
> `kubectl apply -R`: it would apply both variants and create two paths to the
> same gateway, which is unsafe once SSO is enabled.

Both variants also ship NetworkPolicies: `ingress/61-networkpolicy.yaml` limits
gateway ingress to Traefik pods, and `gateway-api-sso/75-networkpolicies.yaml`
limits the gateway to oauth2-proxy and oauth2-proxy egress to DNS, public HTTPS
and the gateway. They are part of the variant; see the
[exposure README](manifests/exposure/README.md) and
[optional hardening](docs/optional-hardening.md).

### Option A — plain Ingress (gateway's own session auth)

1. Set the Ingress host/TLS (or drop the `tls:` block), and set
   `DSH_TRUSTED_HOSTS` in `manifests/10-sandboxtemplate.yaml` to the same public
   host; re-apply the template.
2. Set `COOKIE_SECURE=true` once the Ingress serves HTTPS:
   `kubectl -n dsh-demo set env deploy/dsh-gateway COOKIE_SECURE=true`.
3. `kubectl apply -f manifests/exposure/ingress/`; verify
   `curl -sI https://<host>/healthz` returns `200`.

Keep the router free of buffering/compression middlewares: they break WebSocket
upgrades and streaming. The variant's `61-networkpolicy.yaml` allows only Traefik
pods to reach the gateway; see the [exposure README](manifests/exposure/README.md)
for selector and probe caveats.

### Option B — Gateway API + GitHub SSO (oauth2-proxy)

`manifests/exposure/gateway-api-sso/` puts a TLS `Gateway` + `HTTPRoute` in
front of **oauth2-proxy**, which authenticates the browser against GitHub
(restricted to a configured organization) and forwards to the gateway with
`X-Forwarded-User` set. The gateway trusts that header only when
`AUTH_PROXY_HEADER` is set (SSO is **opt-in**).

1. Install Gateway API CRDs and a Gateway implementation; the manifests assume
   `gatewayClassName: traefik` (the CRD version is `GATEWAY_API_VERSION` in
   `versions.env`).
2. Create a GitHub OAuth App whose homepage/callback match the public FQDN, and
   create the oauth2-proxy and TLS Secrets from
   `73-oauth2-proxy-secret.yaml.example` and `74-tls-secret.yaml.example`.
3. Replace `dsh-demo.example.com` and `your-github-org`, set
   `DSH_TRUSTED_HOSTS` in the template, and enable SSO:
   `kubectl -n dsh-demo set env deploy/dsh-gateway AUTH_PROXY_HEADER=X-Forwarded-User COOKIE_SECURE=true`.
4. `kubectl apply -f manifests/exposure/gateway-api-sso/`; verify the Gateway,
   HTTPRoute, oauth2-proxy Deployment and endpoints as shown in the
   [exposure README](manifests/exposure/README.md).

> **SSO trust boundary — read this before enabling it.** After SSO is on, the
> `dsh-gateway` Service must be reachable **only** through oauth2-proxy (it is
> ClusterIP). The variant's `75-networkpolicies.yaml` restricts pod traffic
> accordingly: the gateway accepts traffic only from oauth2-proxy pods,
> oauth2-proxy only from the Gateway implementation, and its egress only to DNS,
> public HTTPS and the gateway. Anyone who can reach `dsh-gateway:8080` directly
> can send `X-Forwarded-User: <victim>`, so never add a second path. To turn SSO
> off, remove the variable
> (`kubectl -n dsh-demo set env deploy/dsh-gateway AUTH_PROXY_HEADER-`) and
> delete the SSO variant.

### Switching variants

Delete the variant you had before applying the other one, revert
`AUTH_PROXY_HEADER`/`COOKIE_SECURE`, and verify with
`kubectl -n dsh-demo get ingress,gateway,httproute` that exactly one path to the
gateway remains. Full commands and verification are in the
[exposure README](manifests/exposure/README.md).

## Optional hardening

The base walkthrough is deliberately portable. Five opt-in settings are
documented in [optional hardening](docs/optional-hardening.md):

- **gVisor** — uncomment `runtimeClassName: gvisor` before applying, or patch an
  existing install. Requires a `gvisor` RuntimeClass and capable nodes.
  `kubectl port-forward` does not work under gVisor; the gateway path does.
- **nodeSelector** — uncomment and replace the placeholder node label to pin
  sandboxes and the gateway to a dedicated pool; add a toleration for tainted
  pools.
- **Sandbox NetworkPolicy** —
  `kubectl apply -f manifests/optional/network-policy/` allows only
  gateway→sandbox on `3080` and sandbox→DNS/public `443`.
- **Provider keys** — create `dsh-provider-keys-extra` as shown in step 1.
- **Failed-Pod recovery** — uncomment `failedPodPolicy: Recreate` so the
  controller replaces a `Failed` sandbox Pod (node eviction) instead of parking
  the sandbox at `Finished=True`. Requires agent-sandbox v1.0.8+.

Apply each independently and re-run the walkthrough. `kubectl apply -f manifests/`
is non-recursive, so the optional manifests are not applied by the base step.

## Troubleshooting

| Symptom | Cause / fix |
| --- | --- |
| Pod `CrashLoopBackOff`, probes failing with `401` | Probe path is `/`; change readiness/liveness to `/favicon.svg` |
| Browser shows `401` from DSH through the gateway | Gateway handshake failed — check `kubectl -n dsh-demo logs deploy/dsh-gateway` and that RBAC grants `pods/log` |
| `502` / `upstream unavailable` | Sandbox pod is not Ready (wake timed out) or the Service DNS is stale; check `kubectl get sandbox`, `WAKE_TIMEOUT` |
| `503` with `Retry-After` | The launch token was not found in the pod logs yet; the gateway retries after a short delay. Check the pod logs and `DSH_LOG_TAIL_LINES` |
| `409` on signup | User already has a claim; use `POST /login` with the existing token or delete the user first |
| Cookie set but browser keeps getting `401` | `COOKIE_SECURE=true` while serving over plain HTTP (secure cookies are not sent); use HTTPS or set `false` for local demos |
| WebSocket closes immediately (`4401`) | Missing/expired gateway cookie, or the upgrade's `Origin` does not match `Host` (CSWSH check); DSH rejecting `Host` instead surfaces as close `1011` — add the host to `DSH_TRUSTED_HOSTS` |
| Console spams `[connection] connection lost, retry #N` | The upstream WS upgrade is being rejected, surfaced as close `1011`. The gateway dials the sandbox **pod IP**, forwards no `Host`/`Origin`, and handshakes under the same `pod-IP:3080` authority DSH binds its cookie to; an older gateway image, or a sandbox with an empty `status.podIPs`, hits this loop |
| `manifest.webmanifest` returns `401` in the console | Browsers fetch the Web App Manifest without credentials, so the gateway's session cookie is absent and the request fails closed. The UI is unaffected; only the PWA install prompt is lost |
| Signup is slow / `202` then `Waking` | Warm pool exhausted (`kubectl get sandboxes -l agents.x-k8s.io/warm-pool-sandbox`); scale the pool |
| `403` from the Kubernetes API in gateway logs | The gateway ServiceAccount is missing claim/sandbox/pod-log verbs (see `manifests/50-gateway.yaml`) |
| Gateway pod `ErrImagePull` / `ImagePullBackOff` (especially on kind) | The manifest uses `imagePullPolicy: IfNotPresent` with the release identity. Build with `make -C ../.. gateway`, retag to exactly the manifest's image, `kind load docker-image <that-ref>`, or set the Deployment image to your local tag / pinned digest. On a real cluster, make sure the GHCR package is readable (public or a pull secret) |
| Step 3: `pod "$SB" not found` after deploying the gateway | The claim was suspended by the idle sweeper. Claims created by the gateway carry `aaas.example.com/managed-by=dsh-gateway`; hand-applied claims (steps 1–7) do not and are never swept. Resume the sandbox: `kubectl -n dsh-demo patch sandbox "$SB" --type merge -p '{"spec":{"operatingMode":"Running"}}'` |
| Resume crash-loops: `credentials-local: … is readable beyond its owner (mode 660)` | The pod template lacks `fsGroupChangePolicy: OnRootMismatch`, so the kubelet's default fsGroup policy rewrote `.credentials.yaml` to `0660`; DSH requires `0600` and refuses to start. New installs get the policy from the template. For an existing sandbox: suspend it, run a throwaway pod on `dsh-home-$SB` to `chmod 600 /home/node/.dsh/.credentials.yaml`, patch the sandbox's `podTemplate` with the policy, then resume |
| Agents get `Permission denied` writing to `/workspace` on a fresh or resumed pod | The `/workspace` mount is the `dsh-home` PVC's `workspace/` subPath, and the kubelet is known to leave a `subPath` directory at `0755` if it is interrupted between creating and chmodding it ([kubernetes#142464](https://github.com/kubernetes/kubernetes/issues/142464)); later mounts never correct it. Suspend the sandbox, run a throwaway pod on `dsh-home-$SB` mounted at `/data`, `chmod 2775 /data/workspace` (do **not** chown/chmod recursively — that rewrites `.credentials.yaml`), then resume. If you recreate the sandbox instead, its current `/workspace` contents go with the PVC |
| Every bash command fails: `SANDBOX_UNAVAILABLE`, `sandbox mode "workspace-write" is requested but no sandbox backend is usable` | DSH's own command sandbox found neither backend: `bubblewrap` ships in the image but is unsetuid, so it needs an unprivileged user namespace this hardened pod does not grant, and no Landlock runner exists on this kernel/runtime — the confined executor fails closed. The template sets `DSH_PERMISSION_MODE=danger-full-access` to keep agents usable inside the pod boundary. To restore confinement, confirm a backend inside the pod (`bwrap --ro-bind / / true`; `grep -o landlock /sys/kernel/security/lsm`), set `DSH_PERMISSION_MODE=workspace-write` (or `read-only`), and recreate the pod — the value is stamped at creation. A session that explicitly chose a mode keeps that choice |
| SSO: GitHub login succeeds but the gateway asks for a token / returns `401` to `/recreate` | `AUTH_PROXY_HEADER` is not set on `deploy/dsh-gateway` (SSO is opt-in). Set `AUTH_PROXY_HEADER=X-Forwarded-User` and confirm the request actually reaches the gateway through oauth2-proxy |
| SSO: gateway treats a caller as another user / bad `X-Forwarded-User` accepted | Trust boundary violation: something other than oauth2-proxy can reach `dsh-gateway:8080`. Remove the second path (Ingress/Gateway/port-forward), keep the Service ClusterIP, and only re-enable SSO once the proxy is the sole entry point. Never put a trusted identity header in front of a service that is directly reachable |
| SSO: oauth2-proxy `cookie_secret must be 16, 24, or 32 bytes ... but is 44 bytes` | The cookie secret was generated as standard base64. Use URL-safe unpadded base64 (see `73-oauth2-proxy-secret.yaml.example`) |
| SSO: redirect loop (`ERR_TOO_MANY_REDIRECTS`) | oauth2-proxy `--redirect-url` / callback does not match the public FQDN, or TLS is not terminating at the Gateway listener; fix the OAuth App URLs and the manifest placeholders |
| SSO: DSH rejects the public host behind the proxy | Add the public hostname to `DSH_TRUSTED_HOSTS` in `manifests/10-sandboxtemplate.yaml`; the gateway forwards the browser's `Host` |
| Pods `Pending` after setting `nodeSelector` | No node matches the label, the pool is full, or a tainted pool needs a toleration; inspect `kubectl describe pod` and the node labels |
| gVisor: `kubectl port-forward` fails or the connection is refused | Port-forward is incompatible with gVisor; reach DSH through the gateway (or a reachable Service/Sandbox Router) |
| Sandbox cannot reach a model API after applying the optional NetworkPolicy | The policy allows DNS and public TCP `443` only; a proxy, custom base URL, port `80` or non-443 endpoint needs an explicit egress rule |
| Gateway cannot reach a sandbox after applying an exposure NetworkPolicy | Check that the CNI enforces NetworkPolicy and that the Traefik/oauth2-proxy pod labels match; kubelet probes may also need the node CIDR |
| New provider key not visible in a sandbox | Pod env is stamped at creation: delete the warm-pool spares and recreate claimed pods. A key saved in DSH's credential store also wins over env |
| Folder picker opens on `/home/node` instead of the workspace | The sandbox was created before the template set `HOME=/workspace`, and pod env is stamped at creation. Verify with `kubectl -n dsh-demo exec "$SB" -- printenv HOME`; patch the sandbox's `podTemplate` with the env (or recreate the sandbox — the PVC keeps DSH state and workspace) and resume |
| File explorer and `ls /workspace` show `.cache`, `.config`, `.local`, `.npm`, `.agents` | Expected with `HOME=/workspace`: `~`-relative tool caches and DSH's global agent instructions/skills live on the PVC inside the workspace. Remove the `HOME` env from the template (and recreate the sandbox) if you would rather keep the picker on the ephemeral image home; point caches elsewhere per tool if only they are unwanted |
| Sandbox reports `Finished=True` and every gateway request fails | The sandbox Pod reached phase `Failed` (node eviction, spot reclaim) and the default `Retain` keeps it. Delete the Pod so the controller recreates it, or set `failedPodPolicy: Recreate` (agent-sandbox v1.0.8+; see optional hardening) |

## Limitations

Deliberately **not** production code:

- **Single replica.** Idle clocks, in-flight tracking, and the handshake cache
  are in memory. Running two gateway replicas would double-suspend or lose wake
  tracking; scale up only after moving that state to a shared store.
- **Log scraping.** The launch token is recovered from pod logs. Log rotation, a
  restarted pod, or a changed DSH log format can break the handshake until the
  next `401` refresh.
- **Shared provider keys.** `DEEPSEEK_API_KEY` and any keys in the optional
  `dsh-provider-keys-extra` Secret are shared by all tenants via the template.
  Tenants can override a key through DSH's stored credential file on their PVC,
  but the platform does not manage that. Real multi-tenancy needs per-tenant
  credentials or an egress proxy.
- **No TLS in the gateway.** Terminate TLS at an Ingress or Gateway and set
  `COOKIE_SECURE=true`.
- **SSO trust is hop-based.** `AUTH_PROXY_HEADER` trusts whoever sets the header.
  The SSO variant ships `75-networkpolicies.yaml`, which restricts the gateway
  to oauth2-proxy and oauth2-proxy egress to the gateway/DNS/public HTTPS; a
  custom exposure without that policy has no signature, mTLS or network
  enforcement.
- **DSH's inner file sandbox is off here.** The template sets
  `DSH_PERMISSION_MODE=danger-full-access` because neither backend of DSH's own
  command sandbox is usable in this pod: `bubblewrap` is installed in the image
  but is unsetuid, so it needs an unprivileged user namespace that the pod's
  seccomp/capability surface does not grant (`capabilities.drop: ["ALL"]`), and
  the kernel/runtime exposes no Landlock runner. The executor therefore fails
  closed with `SANDBOX_UNAVAILABLE` and every bash command fails. The per-user
  pod remains the boundary — non-root,
  capabilities dropped, `seccompProfile: RuntimeDefault`, its own PVC and Service
  — but an agent command can now read its own `DSH_HOME` (including
  `.credentials.yaml`) and write anywhere in the container. On hosts where a
  backend works, set `workspace-write` (or `read-only`) and verify before relying
  on the inner policy.
- **Isolation is opt-in.** gVisor, the sandbox NetworkPolicy and the exposure
  policies are opt-in; without them, sandboxes share the host kernel and can
  reach the cluster network. The base example is a demo, not a hardened
  multi-tenant platform.
- **OAuth org is the only authorization.** Any GitHub user in the configured
  organization gets an auto-provisioned sandbox; there is no per-user allow-list,
  quota, or admin API.
- **In-memory user directory.** Users are claims + annotations; there is no
  database, password reset, or admin API. Tokens are stored only as SHA-256
  hashes in claim annotations.
- **Bearer tokens in URLs.** Avoid putting the signup token in a URL/log; the
  browser flow uses the `HttpOnly` cookie instead.

## Cleanup

```bash
# Delete an exposure variant first if you applied one (optional; namespace
# deletion below removes everything):
kubectl delete -f manifests/exposure/ingress/          # or gateway-api-sso/
# Then the whole example:
kubectl delete namespace dsh-demo
```

If you created the oauth2-proxy/TLS Secrets or the GitHub OAuth App for step 9,
delete the Secrets with the namespace and revoke the OAuth App in GitHub.

## More

- [Gateway README](gateway/README.md) — API, configuration, auth/SSO, failure
  semantics, limitations.
- [Architecture](docs/architecture.md) — container/sequence diagrams, ADRs,
  security and RBAC, failure modes, evolution path.
- [Exposure README](manifests/exposure/README.md) — the two variants in detail.
- [Optional hardening](docs/optional-hardening.md) — gVisor, nodeSelector,
  NetworkPolicy, extra provider keys and failed-Pod recovery, with verification
  and rollback.