nicolas-georger/deepseek-kubernetes-sandbox
1
0
Fork 0
mirror of https://github.com/sredevopsorg/deepseek-kubernetes-sandbox.git synced 2026-11-03 01:13:24 +00:00
Code Projects Releases Activity

feat(repo)!: repo reorganization (#16)

Browse Source 
* feat(repo)!: restructure around the dsh-harness image and one example per implementation

- the root Dockerfile is the primary artifact; versions.env is the single source of truth
- merge gateway-api/dsh-demo into examples/dsh-agents-as-a-service (manifests/, exposure variants)
- move the single-tenant template into examples/minimal-single-tenant
- publish the gateway as a nested package under the repository image
- mirror the DSH ref for release identities (first release dsh-v0.2.1-rc.2-r2)
- CI: lint/python/manifests/gateway-image, reusable multi-arch build, tag-driven release, weekly e2e
- docs: README, docs/{main-image,examples,gitflow,versioning}, example READMEs, AGENTS.md, governance
- SSO is opt-in again; gateway pull policy and cookie defaults match the documented demo

BREAKING CHANGE: example manifest paths, the gateway image name, and the removal
of deploy/ and gateway-api/ affect existing deployments; see the migration table
in README.md.

* docs(changelog): generate initial changelog from history

* fix(scripts): make set-version update Dockerfile and sample revision pins

* build(hadolint): document intentional lint exceptions for the main image

---------

Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
This commit is contained in:
Nicolás Georger
2026-10-25 08:37:43 -03:00
committed by GitHub
parent 53e55fc851
commit 3b70f179ac
69 changed files with 3774 additions and 1643 deletions
Download Patch File Download Diff File Expand all files Collapse all files

487
examples/dsh-agents-as-a-service/README.md
View File

@@ -2,20 +2,41 @@
This example distills the upstream
[hermes-agents-as-a-service](https://github.com/kubernetes-sigs/agent-sandbox/tree/main/examples/hermes-agents-as-a-service)
platform pattern into pure agent-sandbox resources + kubectl, running **DeepSeek Harness Web
(DSH)** instead of Hermes. Every user of such a platform gets a personal, long-lived DSH Web
instance that is **provisioned in ~2 seconds from a warm pool, suspended when idle to save
cost, and woken transparently when the user returns** — with all state (conversations,
sessions, config) surviving on a per-user PVC at `/home/node/.dsh`.
platform pattern into pure agent-sandbox resources + kubectl, running **DeepSeek
Harness Web (DSH)** instead of Hermes. Every user of such a platform gets a
personal, long-lived DSH Web instance that is **provisioned in ~2 seconds from a
warm pool, suspended when idle to save cost, and woken transparently when the
user returns** — with all state (conversations, sessions, config) surviving on a
per-user PVC at `/home/node/.dsh`.
It ships in two layers:
It ships in three layers:
1. **The resource layer** (steps 1–7): pure manifests + kubectl, so every mechanism is visible.
2. **The service layer** (step 8): a small included **gateway** (`gateway/`) that makes the
platform browser-usable — a signup page/API, `HttpOnly` cookie sessions, a `/u/{user}/**`
proxy for HTTP **and WebSocket** that transparently **wakes suspended instances on
connect**, and an idle sweeper that suspends them again. The gateway performs the DSH
launch-token handshake server-side, so the browser never sees the token.
1. **The resource layer** (steps 1–7): pure manifests + kubectl, so every
mechanism is visible.
2. **The service layer** (step 8): a small included **gateway** (`gateway/`) that
makes the platform browser-usable — a signup page/API, `HttpOnly` cookie
sessions, a `/u/{user}/**` proxy for HTTP **and WebSocket** that transparently
**wakes suspended instances on connect**, and an idle sweeper that suspends
them again. The gateway performs the DSH launch-token handshake server-side,
so the browser never sees the token.
3. **The exposure layer** (step 9): choose exactly one way in — a plain Traefik
Ingress, or Gateway API + oauth2-proxy (GitHub SSO).
All manifests live under [`manifests/`](manifests/):
```
manifests/
+-- 00-prereqs.yaml # namespace + empty provider-key Secret
+-- 10-sandboxtemplate.yaml # the DSH Web blueprint
+-- 20-sandboxwarmpool.yaml # 2 pre-warmed spares
+-- 30-claim-alice.yaml # one user
+-- 40-claim-rejected.yaml # negative demo: env injection rejected
+-- 50-gateway.yaml # Deployment/RBAC/Service for the gateway
+-- exposure/
+-- README.md # the two exposure variants, when to use which
+-- ingress/60-ingress.yaml
+-- gateway-api-sso/70-gateway.yaml ... 74-tls-secret.yaml.example
```
What it demonstrates, feature by feature:
@@ -34,20 +55,21 @@ What it demonstrates, feature by feature:
- **Multi-arch**: the DSH image is built for amd64 and arm64, so there is no
`kubernetes.io/arch: amd64` nodeSelector.
- **Single port**: DSH Web serves everything on `3080`; the gateway proxies to one upstream
instead of splitting `/v1/*` (API) from the dashboard.
- **One shared secret**: only `DEEPSEEK_API_KEY` is injected (via `envFrom`); there is no
dashboard basic-auth, session-cookie secret, or API-server key.
- **Launch-token handshake**: DSH gates `/` behind a token it prints only in its logs. The
gateway reads it, performs the handshake, and keeps the resulting auth cookie server-side;
the browser authenticates to the gateway with its own cookie. See
[How the DSH handshake works](#how-the-dsh-handshake-works).
- **Host allow-list**: DSH rejects unknown `Host` headers. The gateway forwards the browser's
`Host`, so the public hostname must be added to `DSH_TRUSTED_HOSTS` in the template.
- **Probes target `/favicon.svg`**: `/` returns `401` without the launch token, so a probe on
`/` crash-loops the pod. `/favicon.svg` is unauthenticated, and a `startupProbe` gates
liveness/readiness so a slow cold start (plugin activation, or a resumed sandbox) is not
restarted mid-boot.
- **Single port**: DSH Web serves everything on `3080`; the gateway proxies to one
upstream instead of splitting `/v1/*` (API) from the dashboard.
- **One shared secret**: only `DEEPSEEK_API_KEY` is injected (via `envFrom`); there
is no dashboard basic-auth, session-cookie secret, or API-server key.
- **Launch-token handshake**: DSH gates `/` behind a token it prints only in its
logs. The gateway reads it, performs the handshake, and keeps the resulting
auth cookie server-side; the browser authenticates to the gateway with its own
cookie. See [How the DSH handshake works](#how-the-dsh-handshake-works).
- **Host allow-list**: DSH rejects unknown `Host` headers. The gateway forwards
the browser's `Host`, so the public hostname must be added to
`DSH_TRUSTED_HOSTS` in the template.
- **Probes target `/favicon.svg`**: `/` returns `401` without the launch token, so
a probe on `/` crash-loops the pod. `/favicon.svg` is unauthenticated, and a
`startupProbe` gates liveness/readiness so a slow cold start (plugin
activation, or a resumed sandbox) is not restarted mid-boot.
## Prerequisites
@@ -55,43 +77,49 @@ What it demonstrates, feature by feature:
- agent-sandbox with extensions installed:
```bash
VERSION=v1.0.7
kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/download/${VERSION}/sandbox-with-extensions.yaml
VERSION=v1.0.7 # AGENT_SANDBOX_VERSION in versions.env
kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/download/$VERSION/sandbox-with-extensions.yaml
kubectl -n agent-sandbox-system rollout status deployment --timeout=180s
```
Or run everything below in one shot with `./run-test-kind.sh` (expects a kind cluster + the
manifest above already applied; it creates and deletes the namespace itself).
For step 9's SSO variant you additionally need the Gateway API CRDs and a
Gateway implementation (`gatewayClassName: traefik` in the manifests); see the
[exposure README](manifests/exposure/README.md).
- Optional: run the whole resource + gateway walkthrough in one shot with
[`scripts/e2e-kind.sh`](scripts/e2e-kind.sh). It expects a kind cluster with
agent-sandbox installed, then creates and deletes the `dsh-demo` namespace
itself. It stops after the gateway layer (step 8); exposure is not applied.
## 1. Install the platform resources
First the namespace, then the shared provider key. The key reaches **every** tenant sandbox
(a warm-pool constraint: pod env is stamped before any user exists), so it is **never
committed** — generate a fresh value per install:
First the namespace, then the shared provider key. The key reaches **every**
tenant sandbox (a warm-pool constraint: pod env is stamped before any user
exists), so it is **never committed** — generate a fresh value per install:
```bash
kubectl apply -f 00-prereqs.yaml
kubectl apply -f manifests/00-prereqs.yaml
kubectl -n dsh-demo create secret generic dsh-provider-keys \
--from-literal=DEEPSEEK_API_KEY="sk-..."
kubectl apply -f 10-sandboxtemplate.yaml -f 20-sandboxwarmpool.yaml
kubectl apply -f manifests/10-sandboxtemplate.yaml -f manifests/20-sandboxwarmpool.yaml
```
Watch the two warm spares boot (first time pulls the image — subsequent operations never pay
that again; that is the point of the pool):
Watch the two warm spares boot (first time pulls the image — subsequent operations
never pay that again; that is the point of the pool):
```bash
kubectl -n dsh-demo get sandboxes -l agents.x-k8s.io/warm-pool-sandbox -w
```
Wait until both show `READY: True`. Each spare already has its PVC bound and DSH Web serving
on port `3080`.
Wait until both show `READY: True`. Each spare already has its PVC bound and DSH
Web serving on port `3080`.
## 2. "Signup": claim an instance (~2s)
A user signing up is one small claim — `warmPoolRef` plus a pod label:
```bash
kubectl apply -f 30-claim-alice.yaml
kubectl apply -f manifests/30-claim-alice.yaml
kubectl -n dsh-demo wait sandboxclaim dsh-alice --for=condition=Ready --timeout=60s
```
@@ -105,20 +133,21 @@ kubectl -n dsh-demo get sandboxes --show-labels
Three things worth noticing:
- **The sandbox keeps its pool-generated name** (`dsh-pool-…`), not the claim's name —
adoption re-labels an existing warm sandbox rather than creating one. A platform must always
resolve user → sandbox through `claim.status.sandbox.name`.
- The `agents.x-k8s.io/warm-pool-sandbox` label is **gone** from the adopted sandbox (it left
the pool's inventory), and the pool has already started replenishing back to 2 spares.
- **The sandbox keeps its pool-generated name** (`dsh-pool-…`), not the claim's
name — adoption re-labels an existing warm sandbox rather than creating one. A
platform must always resolve user → sandbox through `claim.status.sandbox.name`.
- The `agents.x-k8s.io/warm-pool-sandbox` label is **gone** from the adopted
sandbox (it left the pool's inventory), and the pool has already started
replenishing back to 2 spares.
- The pod carries the claim's `additionalPodMetadata` label
(`sandbox.users.io/dsh-user=alice`), and
`kubectl -n dsh-demo get sandbox $SB -o jsonpath='{.status.serviceFQDN}'` shows the stable
DNS name the gateway proxies to.
`kubectl -n dsh-demo get sandbox $SB -o jsonpath='{.status.serviceFQDN}'` shows
the stable DNS name the gateway proxies to.
## 3. Talk to the instance
DSH Web needs its launch token on first contact — a bare `/` returns `401`. The token is in the
pod logs:
DSH Web needs its launch token on first contact — a bare `/` returns `401`. The
token is in the pod logs:
```bash
kubectl -n dsh-demo logs "$SB" | grep -m1 '?token='
@@ -127,16 +156,16 @@ kubectl -n dsh-demo port-forward "pod/$SB" 18081:3080
# open http://localhost:18081/?token=... (from the log line)
```
> **Local port note:** DSH Web listens on `3080` **in-cluster**. If `3080` is already in use
> on your workstation (e.g. by a local DSH), forward to a different local port (here `18081`),
> never `3080`. `localhost` is always in DSH's trusted-host list, so the forwarded request is
> accepted.
> **Local port note:** DSH Web listens on `3080` **in-cluster**. If `3080` is
> already in use on your workstation (e.g. by a local DSH), forward to a
> different local port (here `18081`), never `3080`. `localhost` is always in
> DSH's trusted-host list, so the forwarded request is accepted.
> **Why the pod stays up:** this claim was applied by hand, so it carries no gateway ownership
> annotation (`aaas.example.com/managed-by`). The gateway's idle sweeper only suspends claims it
> created through `POST /users`; the resource-layer claims from these manifests are never
> touched. You can deploy the gateway (step 8) and still keep a sandbox you are port-forwarding
> to.
> **Why the pod stays up:** this claim was applied by hand, so it carries no
> gateway ownership annotation (`aaas.example.com/managed-by`). The gateway's idle
> sweeper only suspends claims it created through `POST /users`; the resource-layer
> claims from these manifests are never touched. You can deploy the gateway
> (step 8) and still keep a sandbox you are port-forwarding to.
## 4. Write state on the PVC
@@ -161,20 +190,20 @@ kubectl -n dsh-demo wait sandbox "$SB" --for=condition=Ready --timeout=180s
kubectl -n dsh-demo exec "$SB" -- cat /home/node/.dsh/marker.txt # "remember me"
```
> **Why resume works:** the pod template sets `fsGroupChangePolicy: OnRootMismatch`. DSH requires
> its `.credentials.yaml` to be `0600`, while the kubelet's default fsGroup policy widens every
> file to `0660` on each pod start — a resumed sandbox would crash-loop. See
> [Troubleshooting](#troubleshooting) for the one-time fix on sandboxes created before this
> setting existed.
> **Why resume works:** the pod template sets `fsGroupChangePolicy: OnRootMismatch`.
> DSH requires its `.credentials.yaml` to be `0600`, while the kubelet's default
> fsGroup policy widens every file to `0660` on each pod start — a resumed sandbox
> would crash-loop. See [Troubleshooting](#troubleshooting) for the one-time fix
> on sandboxes created before this setting existed.
## 7. Injection policy rejects env-carrying claims
```bash
kubectl apply -f 40-claim-rejected.yaml
kubectl apply -f manifests/40-claim-rejected.yaml
kubectl -n dsh-demo get sandboxclaim dsh-mallory \
-o jsonpath='{.status.conditions[?(@.type=="Ready")].reason}'
# EnvVarsInjectionRejected — the claim never bound a sandbox
kubectl delete -f 40-claim-rejected.yaml
kubectl delete -f manifests/40-claim-rejected.yaml
```
## 8. The gateway: signup, browser proxy, idle-suspend, wake-on-connect
@@ -182,59 +211,66 @@ kubectl delete -f 40-claim-rejected.yaml
### Deploy it
```bash
# Published image (on kind, load it into the cluster; on a real cluster it is
# pulled from GHCR):
docker pull ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway
kind load docker-image ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway --name <kind-cluster>
# Published image (on kind, load it into the cluster; on a real cluster the
# kubelet pulls it from GHCR):
GW=ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway:dsh-v0.2.1-rc.2-r2
docker pull "$GW"
kind load docker-image "$GW" --name <kind-cluster>
# ... or build it locally and tag it exactly as 50-gateway.yaml expects:
docker build -t ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway gateway/
kind load docker-image ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway --name <kind-cluster>
# ... or build it locally and tag it exactly as manifests/50-gateway.yaml
# expects (make gateway builds dsh-aaas-gateway:dev):
make gateway
docker tag dsh-aaas-gateway:dev "$GW"
kind load docker-image "$GW" --name <kind-cluster>
kubectl apply -f 50-gateway.yaml
kubectl apply -f manifests/50-gateway.yaml
kubectl -n dsh-demo rollout status deploy/dsh-gateway --timeout=120s
kubectl -n dsh-demo port-forward svc/dsh-gateway 18080:8080
```
> The gateway binds `8080`; the local port-forward uses `18080` and never `3080`.
>
> **Re-deploying after a new image?** The workflow publishes the mutable `:gateway` tag and the
> Deployment uses `imagePullPolicy: IfNotPresent`, so a node that already cached the tag keeps
> the old image across `kubectl apply`/`rollout restart`. Pin the digest for a reproducible
> rollout (the workflow's `Inspect image` step and `docker buildx imagetools inspect` both show
> it):
> **Image pull policy.** `manifests/50-gateway.yaml` uses
> `imagePullPolicy: IfNotPresent` with the release identity, so a locally built
> image must be loaded into the cluster under exactly that name (kind:
> `kind load`). A node that already cached a different tag is the usual cause of
> "my new build is not running"; pin the digest for a reproducible rollout:
>
> ```bash
> kubectl -n dsh-demo set image deploy/dsh-gateway \
> gateway=ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox@sha256:<digest>
> gateway=ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway@sha256:<digest>
> kubectl -n dsh-demo rollout status deploy/dsh-gateway --timeout=120s
> ```
>
> **Applying over an existing install?** Run `kubectl diff -f 10-sandboxtemplate.yaml`
> first. Cluster-specific fields such as `nodeAffinity` and the warm-pool replica count are
> intentionally not in the repo manifests; preserve them with a targeted patch instead of a
> **Applying over an existing install?** Run
> `kubectl diff -f manifests/10-sandboxtemplate.yaml` first. Cluster-specific
> fields such as `nodeAffinity` and the warm-pool replica count are intentionally
> not in the repo manifests; preserve them with a targeted patch instead of a
> blind apply. Moving the probes to `/favicon.svg` is the crash-loop fix.
>
> The template also sets `fsGroupChangePolicy: OnRootMismatch` (see the note in step 6).
> Existing `Sandbox` objects snapshot their pod template, so a template apply does not update
> them: patch each sandbox's `spec.podTemplate.spec.securityContext.fsGroupChangePolicy` and
> The template also sets `fsGroupChangePolicy: OnRootMismatch` (see the note in
> step 6). Existing `Sandbox` objects snapshot their pod template, so a template
> apply does not update them: patch each sandbox's
> `spec.podTemplate.spec.securityContext.fsGroupChangePolicy` and
> `chmod 600` its `.credentials.yaml`, or recreate it.
### Browser flow
Open <http://localhost:18080>. The landing page has a one-field signup form. Enter `bob`, and
the gateway:
Open <http://localhost:18080>. The landing page has a one-field signup form.
Enter `bob`, and the gateway:
1. creates the `SandboxClaim` over the warm pool,
2. waits for it to become `Ready`,
3. sets an `HttpOnly`, `SameSite=Lax` cookie `dsh_gw_session=<user>:<token>` on the browser,
3. sets an `HttpOnly`, `SameSite=Lax` cookie `dsh_gw_session=<user>:<token>` on
the browser,
4. redirects to `/u/bob/`, where the DSH Web UI loads through the gateway.
From then on every request under `/u/bob/` (assets, API calls, SSE, WebSocket) is proxied to
bob's sandbox. The DSH launch token and DSH's own auth cookie stay in the gateway; the browser
only holds the gateway cookie.
From then on every request under `/u/bob/` (assets, API calls, SSE, WebSocket) is
proxied to bob's sandbox. The DSH launch token and DSH's own auth cookie stay in
the gateway; the browser only holds the gateway cookie.
The same flow with `curl`, no JavaScript (the session cookie is the only credential):
The same flow with `curl`, no JavaScript (the session cookie is the only
credential):
```bash
JAR=$(mktemp)
@@ -252,48 +288,53 @@ curl -s -b "$JAR" -H 'Origin: http://localhost:18080' -X DELETE http://localhost
```
API clients that prefer bearer tokens can use `POST /users` with a JSON body
(`{"user":"bob"}`) and the returned one-time token, `POST /login` to exchange a token for a
cookie, and `Authorization: Bearer <token>` on subsequent calls.
(`{"user":"bob"}`) and the returned one-time token, `POST /login` to exchange a
token for a cookie, and `Authorization: Bearer <token>` on subsequent calls.
### Gateway API
| Route | Purpose |
| --- | --- |
| `GET /healthz` | Liveness/readiness; no cluster access |
| `GET /` | Landing page: signup, token login, logout |
| `GET /` | Landing page: signup, token login, logout. With SSO it auto-provisions and connects instead. |
| `POST /users` | Signup. JSON or form; `201` (JSON), `202` (waking), `303` (browser redirect), `409` if user exists; sets the session cookie |
| `POST /login` | Exchange a bearer token for the session cookie |
| `POST /logout` | Clear the session cookie |
| `POST /recreate` | SSO only: delete the authenticated user's claim (wiping workspace data) and provision a fresh one; `303` to `/u/{user}/`. Cookie-authenticated calls must be same-origin. |
| `GET /users/{user}` | Derived state (`Ready`/`Waking`/`Suspended`) |
| `DELETE /users/{user}` | Delete claim (cascades sandbox + PVC) |
| `GET /u/{user}` | `308` redirect to `/u/{user}/` |
| `GET /u/{user}/**` | HTTP/SSE proxy to the user's DSH Web |
| `WS /u/{user}/**` | WebSocket proxy to the user's DSH Web |
The full API reference, configuration table, auth semantics and failure matrix
live in the [gateway README](gateway/README.md).
### How the DSH handshake works
DSH does not expose an API for its launch token; it prints a URL containing `?token=…` in the
pod logs. For each `(user, pod UID, Host)` the gateway:
DSH does not expose an API for its launch token; it prints a URL containing
`?token=…` in the pod logs. For each `(user, pod UID, Host)` the gateway:
1. resolves the claim → sandbox → pod (annotation `agents.x-k8s.io/pod-name`, or the sandbox
name as pod name),
2. reads the pod's logs (last `DSH_LOG_TAIL_LINES` lines) and parses the most recent
`dsh web: …?token=…` line,
1. resolves the claim → sandbox → pod (annotation `agents.x-k8s.io/pod-name`, or
the sandbox name as pod name),
2. reads the pod's logs (last `DSH_LOG_TAIL_LINES` lines) and parses the most
recent `dsh web: …?token=…` line,
3. requests `http://<serviceFQDN>:3080/?token=…` with the browser's `Host`,
4. captures DSH's `dsh-auth-*` `Set-Cookie` response and caches it,
5. injects that cookie (and strips the browser's `Cookie`/`Authorization` and any `?token=`)
on every proxied HTTP and WebSocket request,
6. refreshes the handshake once if DSH answers `401`, and drops the cached cookie when the
sandbox suspends or is deleted.
5. injects that cookie (and strips the browser's `Cookie`/`Authorization` and any
`?token=`) on every proxied HTTP and WebSocket request,
6. refreshes the handshake once if DSH answers `401`, and drops the cached cookie
when the sandbox suspends or is deleted.
This is why the gateway's RBAC includes `pods` `get`/`list` and `pods/log` `get`. It is also
the most fragile part of the design; see [Limitations](#limitations).
This is why the gateway's RBAC includes `pods` `get`/`list` and `pods/log`
`get`. It is also the most fragile part of the design; see
[Limitations](#limitations).
### DSH_TRUSTED_HOSTS
DSH rejects requests whose `Host` is not allow-listed (`localhost` and `127.0.0.1` are always
trusted). Because the gateway forwards the browser's `Host`, set the public hostname in the
template:
DSH rejects requests whose `Host` is not allow-listed (`localhost` and
`127.0.0.1` are always trusted). Because the gateway forwards the browser's
`Host`, set the public hostname in the template:
```yaml
env:
@@ -301,36 +342,151 @@ env:
value: "dsh.example.com" # comma or space separated; empty for port-forward only
```
The same applies to a local `docker run`: the printed token URL uses `DSH_PUBLIC_URL` if set,
otherwise `http://localhost:${DSH_PORT}`.
The same applies to a local `docker run`: the printed token URL uses
`DSH_PUBLIC_URL` if set, otherwise `http://localhost:<DSH_PORT>`.
### WebSocket
The gateway is a Starlette ASGI app and proxies WebSocket upgrades with a bidirectional pump, so
the DSH web GUI is fully interactive. Requirements:
The gateway is a Starlette ASGI app and proxies WebSocket upgrades with a
bidirectional pump, so the DSH web GUI is fully interactive. Requirements:
- the ingress/load balancer in front of the gateway must pass `Upgrade`/`Connection` headers;
`60-ingress.yaml` is a starting point (Traefik handles upgrades by default),
- idle suspension is disabled for users with in-flight proxied requests or open WebSockets, so
a long-running chat stream is not cut off mid-turn,
- the upstream upgrade dials the sandbox **pod IP**, forwards no `Host`/`Origin`, and performs
the launch-token handshake under that same `pod-IP:3080` authority. DSH fences on `Host`
(only its bind address, loopback, or an allow-listed host passes) and binds its auth cookie to
the request authority (`payload.authority === authority`), while `websockets` derives `Host`
from the URI and cannot be overridden. A browser-host cookie or forwarded origin makes the
upgrade fail (`401`/`403`, surfaced as close `1011`), which the client retries forever as
`[connection] connection lost, retry #N`,
- missing/invalid gateway auth is rejected right after the upgrade with close code `4401`
(WebSocket auth has no HTTP `401` path); cross-origin or Origin-less upgrades with the cookie
are rejected the same way (CSWSH protection), so clients must send a matching `Origin`;
upstream bridging failures close with `1011`.
- the ingress/load balancer in front of the gateway must pass `Upgrade`/`Connection`
headers; [`manifests/exposure/ingress/60-ingress.yaml`](manifests/exposure/ingress/60-ingress.yaml)
is a starting point (Traefik handles upgrades by default),
- idle suspension is disabled for users with in-flight proxied requests or open
WebSockets, so a long-running chat stream is not cut off mid-turn,
- the upstream upgrade dials the sandbox **pod IP**, forwards no `Host`/`Origin`,
and performs the launch-token handshake under that same `pod-IP:3080` authority.
DSH fences on `Host` (only its bind address, loopback, or an allow-listed host
passes) and binds its auth cookie to the request authority
(`payload.authority === authority`), while `websockets` derives `Host` from the
URI and cannot be overridden. A browser-host cookie or forwarded origin makes
the upgrade fail (`401`/`403`, surfaced as close `1011`), which the client
retries forever as `[connection] connection lost, retry #N`,
- missing/invalid gateway auth is rejected right after the upgrade with close code
`4401` (WebSocket auth has no HTTP `401` path); cross-origin or Origin-less
upgrades with the cookie are rejected the same way (CSWSH protection), so
clients must send a matching `Origin`; upstream bridging failures close with
`1011`.
### Path routing
Each user's DSH Web is served under `/u/<user>/`. DSH injects `<base href="./">` and uses
relative asset/API URLs, so the UI works under a path prefix without a per-user hostname or
wildcard DNS. The gateway also rewrites absolute-path `Location` headers into `/u/<user>/…` and
redirects `/u/<user>` to the trailing-slash form with `308` so relative URLs resolve correctly.
Each user's DSH Web is served under `/u/<user>/`. DSH injects `<base href="./">`
and uses relative asset/API URLs, so the UI works under a path prefix without a
per-user hostname or wildcard DNS. The gateway also rewrites absolute-path
`Location` headers into `/u/<user>/…` and redirects `/u/<user>` to the
trailing-slash form with `308` so relative URLs resolve correctly.
## 9. Expose it: choose exactly one variant
The base manifests expose nothing outside the cluster (`dsh-gateway` is a
ClusterIP Service on port `8080`). Pick **one** variant — they are alternatives,
not a stack. Details, commands and the switch procedure are in the
[exposure README](manifests/exposure/README.md).
> Apply the base resources first, then the variant **by path**. Do not use
> `kubectl apply -R`: it would apply both variants and create two paths to the
> same gateway, which is unsafe once SSO is enabled.
### Option A — plain Ingress (gateway's own session auth)
```bash
# Replace the dsh.example.com host and TLS Secret (or drop the tls: block for
# plain HTTP), then:
kubectl apply -f manifests/exposure/ingress/
```
1. Set `DSH_TRUSTED_HOSTS` in `manifests/10-sandboxtemplate.yaml` to the same
public host (the gateway forwards the browser's `Host`) and re-apply the
template.
2. Set `COOKIE_SECURE=true` on the gateway once the Ingress serves HTTPS:
`kubectl -n dsh-demo set env deploy/dsh-gateway COOKIE_SECURE=true`.
(The base manifest keeps `false` so the plain-HTTP port-forward demo works.)
3. Verify: `curl -sI https://<host>/healthz` returns `200`; sign up in a browser
at `https://<host>/`.
Keep the router free of buffering/compression middlewares: they break WebSocket
upgrades and streaming.
### Option B — Gateway API + GitHub SSO (oauth2-proxy)
`manifests/exposure/gateway-api-sso/` puts a TLS `Gateway` + `HTTPRoute` in
front of **oauth2-proxy**, which authenticates the browser against GitHub
(restricted to a configured organization) and forwards to the gateway with
`X-Forwarded-User` set. The gateway trusts that header only when
`AUTH_PROXY_HEADER` is set (SSO is **opt-in**).
Prerequisites:
- Gateway API CRDs and a Gateway implementation; the manifests assume
`gatewayClassName: traefik` (the CRD version is pinned as
`GATEWAY_API_VERSION` in `versions.env`).
- A GitHub OAuth App whose homepage/callback match the public FQDN
(`https://<host>/` and `https://<host>/oauth2/callback`).
- The oauth2-proxy credentials Secret (template:
`73-oauth2-proxy-secret.yaml.example`) and a TLS Secret for the Gateway
listener (template: `74-tls-secret.yaml.example`).
Steps:
```bash
# 1. Real Secrets; the committed *.yaml.example files are templates only.
kubectl -n dsh-demo create secret generic oauth2-proxy \
--from-literal=client-id='<GitHub OAuth App client ID>' \
--from-literal=client-secret='<GitHub OAuth App client secret>' \
--from-literal=cookie-secret="$(python3 -c 'import secrets; print(secrets.token_urlsafe(32))')"
kubectl -n dsh-demo create secret tls dsh-demo-tls --cert=tls.crt --key=tls.key
# 2. Replace dsh-demo.example.com (70/71/72) and your-github-org (72).
# 3. Enable SSO on the gateway and trust the public host in the template.
kubectl -n dsh-demo set env deploy/dsh-gateway \
AUTH_PROXY_HEADER=X-Forwarded-User COOKIE_SECURE=true
# ... and set DSH_TRUSTED_HOSTS=dsh-demo.example.com in
# manifests/10-sandboxtemplate.yaml, then re-apply the template.
# 4. Apply the variant.
kubectl apply -f manifests/exposure/gateway-api-sso/
```
Verification:
```bash
kubectl get gateway dsh-gateway -n dsh-demo -o yaml # Programmed=True
kubectl get httproute -n dsh-demo -o wide # backend oauth2-proxy:4180, Accepted
kubectl get deploy oauth2-proxy -n dsh-demo # 1/1 Ready
kubectl get endpoints oauth2-proxy -n dsh-demo # backend present
kubectl -n dsh-demo logs deploy/oauth2-proxy --tail=50 # no config errors
```
Browser check: open `https://<host>/` — you are redirected to GitHub, and after
signing in as a member of the allowed organization the gateway auto-provisions
your sandbox (the first page can show `Provisioning`) and connects you. A
non-member is denied before reaching the gateway. GitHub logins are lowercase
DNS-1123 labels, so they map 1:1 onto the gateway's `user` validation. The SSO
landing page also offers **Recreate sandbox** (`POST /recreate`), which deletes
the claim — wiping the workspace PVC — and provisions a fresh one.
Sign-out: the gateway `/logout` clears `dsh_gw_session`; oauth2-proxy's session
can be cleared at `/oauth2/sign_out`.
> **SSO trust boundary — read this before enabling it.** After SSO is on, the
> `dsh-gateway` Service must be reachable **only** through oauth2-proxy (it is
> ClusterIP). Anyone who can reach `dsh-gateway:8080` directly can send
> `X-Forwarded-User: <victim>` and be treated as that user. Never apply the
> Ingress variant (or any other route to the Service) while SSO is enabled, and
> do not expose the gateway Service with `kubectl port-forward` on a shared
> machine while `AUTH_PROXY_HEADER` is set. To turn SSO off, remove the variable
> (`kubectl -n dsh-demo set env deploy/dsh-gateway AUTH_PROXY_HEADER-`) and
> delete the SSO variant; the gateway falls back to bearer-token/session auth.
### Switching variants
Delete the variant you had before applying the other one
(`kubectl delete -f manifests/exposure/ingress/`), revert
`AUTH_PROXY_HEADER`/`COOKIE_SECURE`, and verify with
`kubectl -n dsh-demo get ingress,gateway,httproute` that exactly one path to the
gateway remains.
## Troubleshooting
@@ -346,36 +502,65 @@ redirects `/u/<user>` to the trailing-slash form with `308` so relative URLs res
| Console spams `[connection] connection lost, retry #N` | The upstream WS upgrade is being rejected, surfaced as close `1011`. The gateway dials the sandbox **pod IP**, forwards no `Host`/`Origin`, and handshakes under the same `pod-IP:3080` authority DSH binds its cookie to; an older gateway image, or a sandbox with an empty `status.podIPs`, hits this loop |
| `manifest.webmanifest` returns `401` in the console | Browsers fetch the Web App Manifest without credentials, so the gateway's session cookie is absent and the request fails closed. The UI is unaffected; only the PWA install prompt is lost |
| Signup is slow / `202` then `Waking` | Warm pool exhausted (`kubectl get sandboxes -l agents.x-k8s.io/warm-pool-sandbox`); scale the pool |
| `403` from the Kubernetes API in gateway logs | The gateway ServiceAccount is missing claim/sandbox/pod-log verbs (see `50-gateway.yaml`) |
| Step 3: `pod "$SB" not found` after deploying the gateway | The claim was suspended by the idle sweeper. Claims created by the gateway carry `aaas.example.com/managed-by=dsh-gateway`; hand-applied claims (steps 1–7) do not and are never swept. Upgrade the gateway to a build that honors the marker, or resume the sandbox: `kubectl -n dsh-demo patch sandbox "$SB" --type merge -p '{"spec":{"operatingMode":"Running"}}'` |
| Resume crash-loops: `credentials-local: … is readable beyond its owner (mode 660)` | The pod template lacks `fsGroupChangePolicy: OnRootMismatch`, so the kubelet's default fsGroup policy rewrote `.credentials.yaml` to `0660`; DSH requires `0600` and refuses to start. New installs get the policy from the template. For an existing sandbox: suspend it, run a throwaway pod on `dsh-home-$SB` to `chmod 600 /home/node/.dsh/.credentials.yaml`, patch the sandbox's `podTemplate` with the policy, then resume. |
| `403` from the Kubernetes API in gateway logs | The gateway ServiceAccount is missing claim/sandbox/pod-log verbs (see `manifests/50-gateway.yaml`) |
| Gateway pod `ErrImagePull` / `ImagePullBackOff` (especially on kind) | The manifest uses `imagePullPolicy: IfNotPresent` with the release identity. Build with `make gateway`, retag to exactly the manifest's image, `kind load docker-image <that-ref>`, or set the Deployment image to your local tag / pinned digest. On a real cluster, make sure the GHCR package is readable (public or a pull secret) |
| Step 3: `pod "$SB" not found` after deploying the gateway | The claim was suspended by the idle sweeper. Claims created by the gateway carry `aaas.example.com/managed-by=dsh-gateway`; hand-applied claims (steps 1–7) do not and are never swept. Resume the sandbox: `kubectl -n dsh-demo patch sandbox "$SB" --type merge -p '{"spec":{"operatingMode":"Running"}}'` |
| Resume crash-loops: `credentials-local: … is readable beyond its owner (mode 660)` | The pod template lacks `fsGroupChangePolicy: OnRootMismatch`, so the kubelet's default fsGroup policy rewrote `.credentials.yaml` to `0660`; DSH requires `0600` and refuses to start. New installs get the policy from the template. For an existing sandbox: suspend it, run a throwaway pod on `dsh-home-$SB` to `chmod 600 /home/node/.dsh/.credentials.yaml`, patch the sandbox's `podTemplate` with the policy, then resume |
| SSO: GitHub login succeeds but the gateway asks for a token / returns `401` to `/recreate` | `AUTH_PROXY_HEADER` is not set on `deploy/dsh-gateway` (SSO is opt-in). Set `AUTH_PROXY_HEADER=X-Forwarded-User` and confirm the request actually reaches the gateway through oauth2-proxy |
| SSO: gateway treats a caller as another user / bad `X-Forwarded-User` accepted | Trust boundary violation: something other than oauth2-proxy can reach `dsh-gateway:8080`. Remove the second path (Ingress/Gateway/port-forward), keep the Service ClusterIP, and only re-enable SSO once the proxy is the sole entry point. Never put a trusted identity header in front of a service that is directly reachable |
| SSO: oauth2-proxy `cookie_secret must be 16, 24, or 32 bytes ... but is 44 bytes` | The cookie secret was generated as standard base64. Use URL-safe unpadded base64 (see `73-oauth2-proxy-secret.yaml.example`) |
| SSO: redirect loop (`ERR_TOO_MANY_REDIRECTS`) | oauth2-proxy `--redirect-url` / callback does not match the public FQDN, or TLS is not terminating at the Gateway listener; fix the OAuth App URLs and the manifest placeholders |
| SSO: DSH rejects the public host behind the proxy | Add the public hostname to `DSH_TRUSTED_HOSTS` in `manifests/10-sandboxtemplate.yaml`; the gateway forwards the browser's `Host` |
## Limitations
Deliberately **not** production code:
- **Single replica.** Idle clocks, in-flight tracking, and the handshake cache are in memory.
Running two gateway replicas would double-suspend or lose wake tracking; scale up only after
moving that state to a shared store.
- **Log scraping.** The launch token is recovered from pod logs. Log rotation, a restarted pod,
or a changed DSH log format can break the handshake until the next `401` refresh.
- **Shared provider key.** `DEEPSEEK_API_KEY` is shared by all tenants via the template. Real
multi-tenancy needs per-tenant credentials or an egress proxy.
- **No TLS in the gateway.** Terminate TLS at an Ingress/reverse proxy and set
- **Single replica.** Idle clocks, in-flight tracking, and the handshake cache
are in memory. Running two gateway replicas would double-suspend or lose wake
tracking; scale up only after moving that state to a shared store.
- **Log scraping.** The launch token is recovered from pod logs. Log rotation, a
restarted pod, or a changed DSH log format can break the handshake until the
next `401` refresh.
- **Shared provider key.** `DEEPSEEK_API_KEY` is shared by all tenants via the
template. Real multi-tenancy needs per-tenant credentials or an egress proxy.
- **No TLS in the gateway.** Terminate TLS at an Ingress or Gateway and set
`COOKIE_SECURE=true`.
- **In-memory user directory.** Users are claims + annotations; there is no database, password
reset, or admin API. Tokens are stored only as SHA-256 hashes in claim annotations.
- **Bearer tokens in URLs.** Avoid putting the signup token in a URL/log; the browser flow uses
the `HttpOnly` cookie instead.
- **SSO trust is hop-based.** `AUTH_PROXY_HEADER` trusts whoever sets the header;
the only guarantee is that the gateway Service is not directly reachable.
There is no signature, mTLS, or NetworkPolicy enforcing that in this example.
- **OAuth org is the only authorization.** Any GitHub user in the configured
organization gets an auto-provisioned sandbox; there is no per-user allow-list,
quota, or admin API.
- **In-memory user directory.** Users are claims + annotations; there is no
database, password reset, or admin API. Tokens are stored only as SHA-256
hashes in claim annotations.
- **Bearer tokens in URLs.** Avoid putting the signup token in a URL/log; the
browser flow uses the `HttpOnly` cookie instead.
## Cost model
The same economics as the upstream pattern: suspend-when-idle plus warm pools turn an
always-on instance into a sub-second cold-start pause per return-after-absence, at a fraction
of the cost.
The same economics as the upstream pattern: suspend-when-idle plus warm pools turn
an always-on instance into a sub-second cold-start pause per return-after-absence,
at a fraction of the cost.
## Cleanup
```bash
# Delete an exposure variant first if you applied one (optional; namespace
# deletion below removes everything):
kubectl delete -f manifests/exposure/ingress/ # or gateway-api-sso/
# Then the whole example:
kubectl delete namespace dsh-demo
```
If you created the oauth2-proxy/TLS Secrets or the GitHub OAuth App for step 9,
delete the Secrets with the namespace and revoke the OAuth App in GitHub.
## More
- [Gateway README](gateway/README.md) — API, configuration, auth/SSO, failure
semantics, limitations.
- [Architecture](docs/architecture.md) — container/sequence diagrams, ADRs,
security and RBAC, failure modes, evolution path.
- [Exposure README](manifests/exposure/README.md) — the two variants in detail.

337
examples/dsh-agents-as-a-service/docs/architecture.md Normal file
View File

@@ -0,0 +1,337 @@
# Architecture
This document describes the browser-usable **agents-as-a-service** platform in this example
([README](../README.md), [manifests](../manifests/), [gateway](../gateway/)): how the gateway,
Kubernetes, and DSH Web fit together, and why the design looks the way it does.
Scope: one Kubernetes namespace (`dsh-demo`), one shared DSH Web image, one gateway deployment,
and one DSH Web pod + PVC per user. The cluster is assumed to run
[agent-sandbox](https://github.com/kubernetes-sigs/agent-sandbox) v1.0.7 with the extensions
(`SandboxTemplate`, `SandboxWarmPool`, `SandboxClaim`).
## Container view
```mermaid
flowchart LR
browser["Browser"]
subgraph cluster["Kubernetes cluster - namespace dsh-demo"]
gw["Gateway Deployment<br/>Starlette ASGI on uvicorn :8080<br/>landing page + auth + proxy + sweeper"]
api["Kubernetes API Server"]
ctrl["agent-sandbox controller"]
pool["SandboxWarmPool dsh-pool<br/>2 pre-warmed spares"]
claim["SandboxClaim dsh-USER<br/>one per user"]
sb["Sandbox dsh-pool-xxxx<br/>spec.operatingMode"]
svc["Headless Service<br/>status.serviceFQDN"]
pod["DSH Web pod<br/>node process :3080"]
pvc[("PVC dsh-home-SANDBOX<br/>/home/node/.dsh")]
end
browser -->|"HTTP + WebSocket<br/>/u/USER/**"| gw
gw -->|"claims, sandboxes, pods, pod logs"| api
api --> ctrl
ctrl -->|"adopt / suspend / resume"| claim
claim --> sb
pool -.->|"replenishes"| sb
sb --> svc
svc --> pod
pod --> pvc
```
Responsibilities:
| Component | Responsibility |
| --- | --- |
| **Browser** | Holds only the gateway session cookie (`dsh_gw_session`). Talks exclusively to the gateway. |
| **Gateway** | Signup/login, session auth, `SandboxClaim` lifecycle, wake-on-connect, idle sweeper, HTTP/SSE/WebSocket reverse proxy, DSH launch-token handshake. |
| **Kubernetes API** | Source of truth for claims, sandboxes, and pods; also the log source for the DSH launch token. |
| **agent-sandbox controller** | Adopts warm spares on claim, flips `operatingMode` (Running/Suspended), garbage-collects sandbox + PVC on claim delete. |
| **DSH Web pod** | One `dsh web` process per user, state on the per-user PVC, listening on `3080`. |
| **Warm pool** | Keeps `N` Ready sandboxes so signup is a label flip instead of a cold start. |
## Exposure variants
The base manifests deploy the gateway as a ClusterIP Service (`dsh-gateway:8080`) and expose
nothing outside the cluster. Two **mutually exclusive** variants live under
[`manifests/exposure/`](../manifests/exposure/):
| Variant | Entry point | Authentication | Notes |
| --- | --- | --- | --- |
| [Ingress](../manifests/exposure/ingress/60-ingress.yaml) | Traefik `Ingress` + optional TLS | the gateway's own signup/session cookie | simplest; no external IdP; set `COOKIE_SECURE=true` with HTTPS |
| [Gateway API + SSO](../manifests/exposure/gateway-api-sso/70-gateway.yaml) | Gateway API `Gateway`/`HTTPRoute` terminating TLS | GitHub OAuth via [oauth2-proxy](../manifests/exposure/gateway-api-sso/72-oauth2-proxy.yaml), identity forwarded as `X-Forwarded-User` | requires `AUTH_PROXY_HEADER` on the gateway; the trust boundary applies |
They are alternatives, not a stack: applying both creates a second path to the same gateway
and defeats SSO. When SSO is enabled, the Service must stay ClusterIP and be reachable only
through oauth2-proxy — otherwise any client can send `X-Forwarded-User` itself and be treated
as that user. The gateway ignores the header unless `AUTH_PROXY_HEADER` is set. See the
[exposure README](../manifests/exposure/README.md) and the
[gateway README](../gateway/README.md#sso-opt-in).
## Flows
### Signup (browser)
```mermaid
sequenceDiagram
autonumber
actor B as Browser
participant G as Gateway
participant K as Kubernetes API
participant C as agent-sandbox controller
B->>G: POST /users (form: user=bob, Origin)
G->>G: validate user name and body size
G->>K: create SandboxClaim dsh-bob (warmPoolRef)
K->>C: reconcile claim
C-->>K: adopt spare, Ready, status.sandbox.name = dsh-pool-xxxx
G->>K: read claim (status) + sandbox + pod
G-->>B: 303 Location /u/bob/ + Set-Cookie dsh_gw_session
Note over G: JSON clients instead get 201 (Ready),<br/>202 (accepted, still waking), or 409 (conflict)
```
Form signup waits up to `WAKE_TIMEOUT` for the claim to become `Ready`, then redirects; the
first proxied request also waits for `Ready` and performs the DSH handshake (below). A `409`
means the user already has a claim, in which case `POST /login` (bearer token) or the existing
cookie is the right path.
### First request / proxy (with handshake)
```mermaid
sequenceDiagram
autonumber
actor B as Browser
participant G as Gateway
participant K as Kubernetes API
participant D as DSH Web pod
B->>G: GET /u/bob/ (Cookie dsh_gw_session)
G->>G: authenticate cookie, user = bob
G->>K: resolve claim -> sandbox -> pod
alt sandbox Suspended
G->>K: patch operatingMode=Running (wake)
loop until Ready or WAKE_TIMEOUT
G->>K: read sandbox Ready condition
end
end
G->>K: read pod logs (tail)
K-->>G: "dsh web: ...?token=..." line
G->>D: GET /?token=... (browser Host, X-Forwarded-*)
D-->>G: 303 + Set-Cookie dsh-auth-*
G->>G: cache cookie for (user, pod UID, Host)
G->>G: strip Cookie/Authorization/hop-by-hop headers
G->>D: replay request with dsh-auth cookie
D-->>G: 200 HTML / SSE stream
G-->>B: 200 (upstream Set-Cookie dropped, Location rewritten)
```
Handshake cache behavior: one entry per `(user, pod UID, Host)`; refreshed once when DSH answers
`401`; dropped when the sandbox is suspended or deleted. The launch token and the DSH auth
cookie never reach the browser.
### Wake-on-connect and idle suspend
```mermaid
sequenceDiagram
autonumber
participant S as Sweeper - lifespan task
participant G as Gateway
participant K as Kubernetes API
actor B as Browser
S->>G: no proxied request for IDLE_TIMEOUT
G->>K: patch sandbox operatingMode=Suspended
K-->>G: pod deleted, PVC + Service retained
Note over S,G: in-flight requests/WebSockets are tracked,<br/>so a busy turn is never suspended
B->>G: GET /u/bob/ after a while
G->>K: patch operatingMode=Running
G->>K: wait for Ready (WAKE_TIMEOUT)
G-->>B: request held, then served
```
Suspend deletes only the pod. The PVC keeps conversations, sessions, and config, so resume
reattaches exactly the same state.
**Ownership boundary.** The gateway shares its namespace with claims created outside it — the
example's resource-layer walkthrough applies `dsh-alice` by hand, and every claim is named
`dsh-<user>`. Claims the gateway creates carry `aaas.example.com/managed-by=dsh-gateway`; the
sweeper skips everything else, so it can never suspend a sandbox an operator is using. Without
that marker the sweeper would suspend a hand-applied claim 60s after first sighting and break
`kubectl logs`/`port-forward` against its pod.
**Resume and file ownership.** The pod template sets `fsGroupChangePolicy: OnRootMismatch` on
purpose. DSH's credentials plugin refuses to boot when `.credentials.yaml` has any group/other
permission bit (it creates the file `0600`). The kubelet's default fsGroup policy (`Always`)
recursively ORs `0660` into every file on each pod start, so the first resume of a sandbox would
crash-loop with `mode 660`. `OnRootMismatch` only rewrites ownership/permissions when the volume
root does not already match the fsGroup, which it does after the first mount — so suspend/resume
keeps working. Sandboxes snapshot their pod template, so an updated `SandboxTemplate` does not
retroactively change existing ones.
### WebSocket
```mermaid
sequenceDiagram
autonumber
actor B as Browser
participant G as Gateway
participant D as DSH Web pod
B->>G: GET /u/bob/... Upgrade: websocket (Cookie dsh_gw_session)
G->>G: authenticate + mark user in-flight
alt cookie missing/invalid
G-->>B: 101 upgrade, then close 4401
else authenticated
G->>D: websockets.connect(path, Host, Cookie dsh-auth)
alt upstream handshake fails
G-->>B: 101 upgrade, then close 1011
else bridged
G-->>B: 101 Switching Protocols
loop frames
B->>D: frames proxied through G
D->>B: frames proxied through G
end
G->>G: clear in-flight on close
end
end
```
## Decisions and trade-offs
### ADR-1: Starlette ASGI instead of a sync Flask proxy
- **Decision:** implement the gateway as a Starlette app served by `uvicorn
gateway:create_app --factory`.
- **Why:** the browser needs WebSocket upgrades, streaming SSE responses, and concurrent
long-poll wake requests. A requests-based Flask proxy cannot upgrade connections and blocks
workers while a sandbox wakes.
- **Cost:** async code paths and explicit bridging for the sync `kubernetes` client (offloaded
to threads). Accepted: one small service, one framework.
### ADR-2: Path-based routing (`/u/{user}/`) instead of per-user subdomains
- **Decision:** serve each user under `https://gateway/u/{user}/`.
- **Why:** one hostname, one TLS certificate, no wildcard DNS or per-user Ingress objects. DSH
uses relative asset/API URLs and injects `<base href="./">`, so a path prefix works; the
gateway rewrites absolute-path `Location` headers and redirects `/u/{user}` to
`/u/{user}/`.
- **Cost:** any upstream code that emits root-absolute URLs would need rewriting; cookie
domains are shared (mitigated by stripping DSH `Set-Cookie`). If DSH ever changes its base
handling, per-user subdomains are the fallback.
### ADR-3: Server-side launch-token handshake instead of exposing the token
- **Decision:** the gateway reads DSH's launch token from pod logs, performs the token
handshake itself, caches DSH's auth cookie, and authenticates the browser with its own
`HttpOnly` cookie.
- **Why:** DSH has no token API and returns `401` on `/` without the launch-token cookie.
Handing the token to the browser would put it in URLs, history, and logs, and would let users
bypass gateway auth.
- **Cost:** log reading permission and a brittle parser; see [Upstream feature
request](#upstream-feature-request-remove-log-scraping). This is the least durable part of the
design and the first thing to replace.
### ADR-4: In-memory state, single replica
- **Decision:** idle clocks, handshake cache, and in-flight tracking live in gateway memory;
the Deployment runs one replica.
- **Why:** the simplest thing that works, with no extra datastore for a demo/reference
platform.
- **Cost:** a restart forgets idle timers (sandboxes stay Running until their next request) and
the handshake cache; two replicas would double-suspend and race. Scaling out requires moving
this state to a shared store (Redis/Postgres) or leader election. Documented instead of
hidden.
### ADR-5: Claims as the user record
- **Decision:** a user is a `SandboxClaim` named `dsh-<user>`; the bearer-token SHA-256 lives in
a claim annotation, and the gateway marks its own claims with
`aaas.example.com/managed-by=dsh-gateway`.
- **Why:** no database, no separate account service; deleting the claim is account deletion and
cascades everything. The ownership annotation keeps a namespaced claim list safe to share with
objects created by hand or by other controllers.
- **Cost:** no password reset, no admin API, no audit log of its own; Kubernetes API load grows
with users. Claims created before the ownership marker existed must be annotated
(`kubectl annotate sandboxclaim ...`) to keep being swept. Fine for a reference pattern, not
for a product.
### ADR-6: Gateway cookie session instead of bearer tokens in the browser
- **Decision:** browsers get `dsh_gw_session=<user>:<token>`, `HttpOnly`, `SameSite=Lax`
(`Secure` when enabled); API clients may still use `Authorization: Bearer`.
- **Why:** tokens never appear in URLs or JS-readable storage, and `SameSite=Lax` plus a strict
`Origin`/`Referer` check on cookie-authenticated mutations blocks basic CSRF.
- **Cost:** cookie-based auth needs TLS and correct `COOKIE_SECURE`; there is no signed-cookie
layer, so the gateway trusts the token hash comparison for validation.
## Security and RBAC
The gateway runs with a namespace-scoped ServiceAccount (`manifests/50-gateway.yaml`):
| API group | Resource | Verbs | Why |
| --- | --- | --- | --- |
| `extensions.agents.x-k8s.io` | `sandboxclaims` | `get`, `list`, `create`, `delete` | signup, state, account deletion |
| `agents.x-k8s.io` | `sandboxes` | `get`, `list`, `patch` | resolve DNS/pod, wake/suspend |
| `""` (core) | `pods` | `get`, `list` | resolve the pod behind a sandbox |
| `""` (core) | `pods/log` | `get` | launch-token handshake |
Notes and limits:
- **Pod logs are sensitive.** They contain the DSH launch token and possibly model output. The
permission is scoped to one namespace, but anyone who can read gateway memory or logs can
impersonate users. Prefer the upstream token API when available.
- **No cluster-wide access, no secret reads.** The gateway does not read `dsh-provider-keys`.
- **Browser-facing hardening:** `HttpOnly`/`SameSite=Lax`/optional `Secure` cookies,
timing-safe SHA-256 token comparison, strict same-origin check on cookie-auth mutations,
request-body limit (`MAX_BODY_MB`), hop-by-hop header stripping, upstream `Set-Cookie`
dropped, DSH `Location` headers rewritten under `/u/<user>/`.
- **Host allow-list:** DSH rejects unknown `Host` headers. The gateway forwards the browser's
`Host`, so every public hostname must be in `DSH_TRUSTED_HOSTS`; `localhost`/`127.0.0.1` are
always trusted.
- **TLS is external.** The gateway serves plain HTTP; terminate TLS at the Ingress and set
`COOKIE_SECURE=true`. Without TLS, session cookies are replayable on the network.
- **Tenant isolation** is pod + PVC separation; all tenants share one namespace, one DSH binary,
and one `DEEPSEEK_API_KEY`. Add NetworkPolicies and per-tenant credentials before calling it
multi-tenant-grade.
- **No gateway rate limiting.** Signup can create claims until the cluster or warm pool is
exhausted; put limits at the ingress for real deployments.
## Failure modes
| Failure | Symptom | Behavior / mitigation |
| --- | --- | --- |
| DSH log line changed or rotated away | Proxy returns `503` (missing launch token) | Fix the parser or restart the pod so the token line is re-emitted |
| Probe on `/` | Pod `CrashLoopBackOff`, `401` in probe logs | Use `/favicon.svg` for readiness/liveness (shipped templates do) |
| Warm pool exhausted | JSON signup returns `202` (state `Waking`); form signup is slow | Increase `replicas` in the warm pool |
| Gateway restart | Idle sandboxes stop suspending; handshake cache lost | Rebuilt lazily on next request; consider a shared store for production |
| DSH auth cookie expired mid-session | Proxy `401` once, then recovers | Gateway refreshes the handshake once |
| Upstream pod name changed after resume | Cached cookie keyed by old pod UID is useless | Cache is invalidated on suspend; next request re-handshakes |
| Two gateway replicas | Double suspend/wake races | Keep `replicas: 1` until state is externalized |
## Upstream feature request: remove log scraping
DSH should expose the launch token as a first-class, machine-readable interface so platforms do
not need `pods/log` access. Proposed shapes (any one suffices):
1. **Token file:** `dsh web --token-file /run/dsh/token` writes the current launch token (mode
`0600`) and rewrites it on rotation.
2. **Environment input:** accept `DSH_LAUNCH_TOKEN` (platform-generated) so the token never
has to be discovered.
3. **Loopback/trusted-host endpoint:** `GET /api/auth/launch-token` that only answers on the
pod's loopback or a trusted host and returns the token plus expiry.
4. **Token in a well-known file** under `DSH_HOME` with the same semantics as (1).
Why this matters: log scraping is brittle (rotation, restart, format changes), requires a broad
`pods/log` permission that also exposes application output, and can leak tokens into log
pipelines. Until an upstream API exists, the gateway's parser, `DSH_LOG_TAIL_LINES`, and the
single-retry refresh are the supported workaround.
## Evolution path
1. Replace log scraping with the upstream token API (above), deleting the `pods/log` permission.
2. Move idle clocks, in-flight tracking, and handshake cache to a shared store; then scale the
gateway horizontally (sticky routing not required if state is shared).
3. Add per-tenant API keys or an egress proxy so a single key is not shared by all sandboxes.
4. Add NetworkPolicies (gateway → sandboxes only), TLS everywhere, rate limits, and audit
logging at the ingress.
5. If DSH grows absolute URLs or per-user cookies that cannot be rewritten, switch to
per-user subdomains with wildcard DNS/TLS.

11
examples/dsh-agents-as-a-service/gateway/.dockerignore Normal file
View File

@@ -0,0 +1,11 @@
# Runtime image only needs the app and its pinned runtime requirements.
__pycache__/
*.py[cod]
.pytest_cache/
.ruff_cache/
.venv/
test_gateway.py
requirements-dev.txt
pyproject.toml
README.md
coverage.xml

66
examples/dsh-agents-as-a-service/gateway/README.md
View File

@@ -4,7 +4,7 @@ A single-file, browser-usable control + data plane for
[DeepSeek Harness Web (DSH)](https://github.com/deepseek-ai/deepseek-harness)
sandboxes running behind the `agent-sandbox` CRDs.
It does three things:
It does four things:
1. **Users** — creates a `SandboxClaim` from a warm pool, mints a bearer token
(only its SHA-256 is stored as a claim annotation), reports derived state,
@@ -15,10 +15,15 @@ It does three things:
3. **Reverse proxy** — proxies HTTP (including SSE) and WebSocket traffic to
the sandbox's DSH Web server on `:3080`, including DSH's own cookie
handshake.
4. **Optional SSO** — when `AUTH_PROXY_HEADER` is set, trusts the identity
header an authenticating reverse proxy (for example oauth2-proxy) forwards,
auto-provisions that user, and exposes `POST /recreate` for a clean restart.
DSH is pinned to `dsh-v0.2.1-rc.2` in the example manifests. It binds `:3080`,
serves `/`, gates `/api` behind a `dsh-auth-*` cookie and requires a WebSocket
at `/api/remote.mux`.
The DSH version pin lives in [`versions.env`](../../../versions.env)
(`DSH_VERSION` plus the optional repository `RELEASE_REVISION`); the example
manifests and published images use the release identity built from it (currently
`dsh-v0.2.1-rc.2-r2`). DSH binds `:3080`, serves `/`, gates `/api` behind a
`dsh-auth-*` cookie and requires a WebSocket at `/api/remote.mux`.
## Flow
@@ -58,10 +63,11 @@ token. A failed handshake returns `503` with `Retry-After: 10`.
| Method | Path | Auth | Purpose |
| --- | --- | --- | --- |
| `GET` | `/healthz` | none | Liveness/readiness. Never touches the cluster. |
| `GET` | `/` | none | Landing page with signup + token login forms. |
| `GET` | `/` | none | Landing page with signup + token login forms. With SSO it auto-provisions the proxy-identified user and redirects to their workspace. |
| `POST` | `/users` | none | Signup. JSON body `{"user": "<dns-1123 label>"}` or form field `user`. JSON returns `201` when Ready, `202` when accepted but still Waking; form returns `303` to `/u/<user>/`. Both set the session cookie. `409` if the user exists, `400` if invalid, `413` if the body is too large. |
| `POST` | `/login` | none | Form (`user`, `token`) returns `303` to `/u/<user>/`; JSON returns `200`. Sets the session cookie. `401` on a bad token, `404` for an unknown user. |
| `POST` | `/logout` | none | Clears the session cookie and `303`s to `/`. |
| `POST` | `/recreate` | trusted proxy header | **SSO only.** Deletes the authenticated user's claim (cascading sandbox + PVC, so workspace data is wiped), waits out finalizers, provisions a fresh claim and `303`s to `/u/<user>/`. `401` without a valid identity header; cookie-authenticated calls must be same-origin. |
| `GET` | `/users/{user}` | bearer or cookie | Derived state: `Provisioning`, `Waking`, `Ready`, `Suspending`, `Suspended`. |
| `DELETE` | `/users/{user}` | bearer or cookie | Delete the claim (sandbox + PVC cascade). Cookie-auth mutations require a same-origin `Origin`/`Referer`. |
| `GET` | `/u/{user}` | — | `308` redirect to `/u/{user}/`. |
@@ -86,6 +92,37 @@ token. A failed handshake returns `503` with `Retry-After: 10`.
absolute paths are rewritten under `/u/<user>/`, and any `token` query
parameter is stripped.
### SSO (opt-in)
By default the gateway is the only authenticator: users sign up with a token or
the session cookie, and the gateway ignores all identity headers. Set
`AUTH_PROXY_HEADER` to enable SSO behind an authenticating reverse proxy such as
oauth2-proxy:
```bash
kubectl -n dsh-demo set env deploy/dsh-gateway AUTH_PROXY_HEADER=X-Forwarded-User
```
When set, the gateway treats a request carrying that header as authenticated as
that user, provided the value is a valid DNS-1123 label; the header is never
trusted when the variable is unset. The landing page then auto-provisions the
user (creates their claim idempotently, mints a rotated session token) and
redirects to `/u/<user>/`, and `POST /recreate` becomes available for a clean
restart. The base manifests leave it unset on purpose; the
[SSO exposure variant](../manifests/exposure/gateway-api-sso/) documents how to
enable it on the Deployment (`kubectl set env`).
> **Trust boundary — this is the security-critical part.** A trusted identity
> header is only safe when **nothing else can reach the gateway**. The gateway
> Service must stay ClusterIP and be reachable exclusively through the
> authenticating proxy; if a client can reach `dsh-gateway:8080` directly (a
> second Ingress/Gateway/Route, a port-forward on a shared host, a broad
> NetworkPolicy), it can send `X-Forwarded-User` itself and impersonate any user.
> Treat `AUTH_PROXY_HEADER` as "the proxy already authenticated this request",
> not as an authentication mechanism of its own. Disable SSO by removing the
> variable: the header is then ignored and bearer-token/session auth applies
> again.
### Proxy failure semantics
| Situation | Result |
@@ -123,6 +160,7 @@ can also be injected into `create_app` for tests.
| `MAX_BODY_MB` | `32` | Maximum buffered request body (and `413` threshold). |
| `HTTP_TIMEOUT` | `300` | Upstream connect/read timeout in seconds. Proxied streams (SSE, long responses) keep the connect/write budget but have **no read timeout**, so an idle event stream is not cut every five minutes. |
| `SWEEP_INTERVAL` | `15` | Seconds between idle sweeps. |
| `AUTH_PROXY_HEADER` | unset | Opt-in SSO: name of the trusted identity header set by an authenticating reverse proxy (e.g. `X-Forwarded-User`). Unset disables SSO and the header is never trusted. Enable only when the Service is reachable exclusively through that proxy (see [SSO](#sso-opt-in)). |
### Required RBAC
@@ -147,12 +185,21 @@ python -m ruff check .
uvicorn gateway:create_app --factory --host 0.0.0.0 --port 8080
```
Container image:
Container image, from the repository root:
```bash
docker build -t dsh-gateway examples/dsh-agents-as-a-service/gateway
make gateway # builds dsh-aaas-gateway:dev from this directory
```
The equivalent direct build is
`docker build -t dsh-aaas-gateway:dev examples/dsh-agents-as-a-service/gateway`.
For kind, retag it to the image reference in
`examples/dsh-agents-as-a-service/manifests/50-gateway.yaml` and `kind load` it.
Published images are nested below the main image as
`<GHCR_IMAGE>/dsh-agents-as-a-service/gateway:<identity>`, where the identity
comes from [`versions.env`](../../../versions.env) — see
[`docs/versioning.md`](../../../docs/versioning.md).
The image runs as UID `10000`, sets `PYTHONDONTWRITEBYTECODE=1` and
`HOME=/tmp` (read-only-rootfs compatible), listens on `8080`, and disables
uvicorn's access log so bearer tokens in URLs are not logged.
@@ -182,3 +229,8 @@ uvicorn's access log so bearer tokens in URLs are not logged.
re-runs the handshake once and replays the request.
* **Only DSH on `:3080` is proxied.** There is no dashboard/API-port split and
no platform API key injection.
* **SSO trusts a header.** With `AUTH_PROXY_HEADER` set, any client that can
reach the gateway Service directly can spoof the identity header. The gateway
cannot enforce the trust boundary itself: keep the Service ClusterIP and
reachable only through the authenticating proxy. There is no mTLS or signed
identity assertion.

147
examples/dsh-agents-as-a-service/gateway/gateway.py
View File

@@ -195,9 +195,7 @@ def derive_state(sandbox: dict | None) -> str:
def token_matches(claim: dict, token: str | None) -> bool:
"""Timing-safe check of a user token against the stored SHA-256."""
want = (claim.get("metadata", {}).get("annotations") or {}).get(
TOKEN_ANNOTATION, ""
)
want = (claim.get("metadata", {}).get("annotations") or {}).get(TOKEN_ANNOTATION, "")
if not token or not want:
return False
got = hashlib.sha256(token.encode()).hexdigest()
@@ -518,9 +516,7 @@ class Kubernetes:
async def get_pod(self, name: str) -> dict | None:
try:
pod = await self._call(
"core", "read_namespaced_pod", name, self.namespace
)
pod = await self._call("core", "read_namespaced_pod", name, self.namespace)
except ApiException as exc:
if exc.status == 404:
return None
@@ -530,9 +526,7 @@ class Kubernetes:
# so callers can keep using ``pod.get(...)``.
return k8s_client.ApiClient().sanitize_for_serialization(pod)
async def read_pod_logs(
self, name: str, container: str, tail_lines: int
) -> str:
async def read_pod_logs(self, name: str, container: str, tail_lines: int) -> str:
result = await self._call(
"core",
"read_namespaced_pod_log",
@@ -616,9 +610,7 @@ class Gateway:
return False # a session for another user never authorizes this one
return token_matches(claim, token)
async def _sso_session(
self, remote: str, headers, cookies, response: Response
) -> dict | None:
async def _sso_session(self, remote: str, headers, cookies, response: Response) -> dict | None:
"""Provision ``remote`` idempotently and mint/refresh a session token.
The authenticating proxy has already verified ``remote``, so it is
@@ -682,9 +674,7 @@ class Gateway:
while True:
async with self.state.lock:
if not self.state.maintenance.get(user):
self.state.in_flight[user] = (
self.state.in_flight.get(user, 0) + 1
)
self.state.in_flight[user] = self.state.in_flight.get(user, 0) + 1
self.state.last_activity[user] = time.time()
return True
if time.monotonic() > deadline:
@@ -693,9 +683,7 @@ class Gateway:
async def _release(self, user: str) -> None:
async with self.state.lock:
self.state.in_flight[user] = max(
self.state.in_flight.get(user, 1) - 1, 0
)
self.state.in_flight[user] = max(self.state.in_flight.get(user, 1) - 1, 0)
self.state.last_activity[user] = time.time()
# -- sandbox / handshake helpers --------------------------------------
@@ -724,9 +712,7 @@ class Gateway:
# Wake-on-connect: safe in ANY state, including mid-suspension.
self._clear_handshake(user)
try:
await self.k8s.patch_operating_mode(
sandbox["metadata"]["name"], "Running"
)
await self.k8s.patch_operating_mode(sandbox["metadata"]["name"], "Running")
except Exception as exc:
raise SandboxUnavailable("could not wake the sandbox") from exc
sandbox = await self.wait_ready(user, self.settings.wake_timeout)
@@ -740,10 +726,7 @@ class Gateway:
def _pod_name(self, sandbox: dict) -> str:
annotations = sandbox.get("metadata", {}).get("annotations") or {}
return (
annotations.get(POD_NAME_ANNOTATION)
or sandbox["metadata"]["name"]
)
return annotations.get(POD_NAME_ANNOTATION) or sandbox["metadata"]["name"]
@staticmethod
def _pod_ip(sandbox: dict) -> str | None:
@@ -787,16 +770,11 @@ class Gateway:
self.settings.dsh_log_tail_lines,
)
except ApiException as exc:
raise HandshakeError(
f"could not read logs for {pod_name}: {exc}"
) from exc
raise HandshakeError(f"could not read logs for {pod_name}: {exc}") from exc
launch_token = parse_dsh_launch_token(logs)
if not launch_token:
raise HandshakeError("DSH launch token not found in pod logs")
url = (
f"http://{fqdn}:{self.settings.dsh_port}/"
f"?token={quote(launch_token, safe='')}"
)
url = f"http://{fqdn}:{self.settings.dsh_port}/?token={quote(launch_token, safe='')}"
headers = {
"Host": host,
"X-Forwarded-Host": host,
@@ -812,9 +790,7 @@ class Gateway:
except httpx.HTTPError as exc:
raise HandshakeError(f"DSH handshake failed: {exc}") from exc
if response.status_code >= 400:
raise HandshakeError(
f"DSH handshake returned {response.status_code}"
)
raise HandshakeError(f"DSH handshake returned {response.status_code}")
cookie = extract_dsh_cookies(response.headers.get_list("set-cookie"))
if not cookie:
raise HandshakeError("DSH handshake set no dsh-auth cookie")
@@ -856,7 +832,7 @@ class Gateway:
else:
status = html.escape(state)
confirm = (
'onsubmit="return confirm(\'Delete this sandbox and start fresh? '
"onsubmit=\"return confirm('Delete this sandbox and start fresh? "
"Your workspace data will be wiped.')\""
)
body = (
@@ -887,9 +863,7 @@ class Gateway:
# session cookie minted by _sso_session; the real page is rendered
# once the sandbox state is known.
response = HTMLResponse()
claim = await self._sso_session(
remote, request.headers, request.cookies, response
)
claim = await self._sso_session(remote, request.headers, request.cookies, response)
sandbox = await self._sandbox_of(claim) if claim else None
page = HTMLResponse(self._sso_page(remote, derive_state(sandbox)))
if "set-cookie" in response.headers:
@@ -953,14 +927,10 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
@staticmethod
def _wants_form(request: Request) -> bool:
ctype = request.headers.get("content-type", "").lower()
return ctype.startswith(
("application/x-www-form-urlencoded", "multipart/form-data")
)
return ctype.startswith(("application/x-www-form-urlencoded", "multipart/form-data"))
async def _read_payload(self, request: Request) -> dict:
if request.headers.get("content-type", "").lower().startswith(
"application/json"
):
if request.headers.get("content-type", "").lower().startswith("application/json"):
data = await self._read_body(request)
try:
payload = json.loads(data)
@@ -997,9 +967,7 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
return self._error(413, "request body too large")
user = payload.get("user", "")
if not valid_user(user):
return self._error(
400, "body must be {'user': '<dns-1123 label>'}"
)
return self._error(400, "body must be {'user': '<dns-1123 label>'}")
if await self.k8s.get_claim(user) is not None:
return self._error(409, "user exists")
token = secrets.token_urlsafe(32)
@@ -1059,9 +1027,7 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
if claim is None:
return self._error(404, "not found")
token, cookie_user = self.credentials(request.headers, request.cookies)
if not self._authorized(
claim, token, cookie_user, user, self.remote_user(request.headers)
):
if not self._authorized(claim, token, cookie_user, user, self.remote_user(request.headers)):
return self._error(401, "unauthorized")
sandbox = await self._sandbox_of(claim)
return JSONResponse(
@@ -1080,9 +1046,7 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
if claim is None:
return self._error(404, "not found")
token, cookie_user = self.credentials(request.headers, request.cookies)
if not self._authorized(
claim, token, cookie_user, user, self.remote_user(request.headers)
):
if not self._authorized(claim, token, cookie_user, user, self.remote_user(request.headers)):
return self._error(401, "unauthorized")
if self._mutation_from_cookie(request.headers, "DELETE", cookie_user):
return self._error(403, "cross-origin request rejected")
@@ -1091,9 +1055,7 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
async with self.state.lock:
self.state.last_activity.pop(user, None)
self.state.in_flight.pop(user, None)
return JSONResponse(
{"user": user, "note": "claim deleted; sandbox and PVC cascade"}
)
return JSONResponse({"user": user, "note": "claim deleted; sandbox and PVC cascade"})
async def recreate(self, request: Request) -> Response:
"""Delete and re-provision the SSO user's sandbox for a clean start.
@@ -1122,9 +1084,7 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
break
await asyncio.sleep(1)
response = RedirectResponse(f"/u/{remote}/", status_code=303)
await self._sso_session(
remote, request.headers, request.cookies, response
)
await self._sso_session(remote, request.headers, request.cookies, response)
return response
# -- HTTP proxy --------------------------------------------------------
@@ -1141,9 +1101,7 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
return RedirectResponse("/", status_code=303)
return self._error(404, "unknown user")
token, cookie_user = self.credentials(request.headers, request.cookies)
if remote == user and not (
cookie_user == user and token_matches(claim, token)
):
if remote == user and not (cookie_user == user and token_matches(claim, token)):
# SSO user without a valid session cookie: bootstrap, then continue.
return RedirectResponse("/", status_code=303)
if not self._authorized(claim, token, cookie_user, user, remote):
@@ -1152,31 +1110,21 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
return self._error(403, "cross-origin request rejected")
if not await self._acquire(user):
return self._error(
503, "gateway busy, retry shortly", {"Retry-After": "5"}
)
return self._error(503, "gateway busy, retry shortly", {"Retry-After": "5"})
streaming = False
try:
sandbox, fqdn = await self._resolve_sandbox(user, claim)
host = request.headers.get("host", "")
cookie, _, was_cached = await self.dsh_cookie(
user, sandbox, fqdn, host
)
cookie, _, was_cached = await self.dsh_cookie(user, sandbox, fqdn, host)
body = await self._read_body(request)
upstream = await self._send_upstream(
request, user, fqdn, cookie, body
)
upstream = await self._send_upstream(request, user, fqdn, cookie, body)
if upstream.status_code == 401 and was_cached:
# The cached DSH cookie expired (pod restart, logout, ...):
# handshake once more and replay the (already buffered) request.
await upstream.aclose()
cookie, _, _ = await self.dsh_cookie(
user, sandbox, fqdn, host, force=True
)
upstream = await self._send_upstream(
request, user, fqdn, cookie, body
)
cookie, _, _ = await self.dsh_cookie(user, sandbox, fqdn, host, force=True)
upstream = await self._send_upstream(request, user, fqdn, cookie, body)
response = StreamingResponse(
self._relay(user, upstream),
@@ -1216,8 +1164,7 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
headers = {
key: value
for key, value in request.headers.items()
if key.lower() not in OWNED_REQUEST_HEADERS
and key.lower() != "content-length"
if key.lower() not in OWNED_REQUEST_HEADERS and key.lower() != "content-length"
}
host = request.headers.get("host", "")
headers["host"] = host
@@ -1229,9 +1176,7 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
else:
headers["x-forwarded-for"] = ""
port = host.rsplit(":", 1)[1] if ":" in host else ""
headers["x-forwarded-port"] = port or (
"443" if request.url.scheme == "https" else "80"
)
headers["x-forwarded-port"] = port or ("443" if request.url.scheme == "https" else "80")
last_error: Exception | None = None
for attempt in range(self.settings.upstream_attempts):
@@ -1341,9 +1286,7 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
if ":" in upstream_host and not upstream_host.startswith("["):
upstream_host = f"[{upstream_host}]" # IPv6 literal
authority = f"{upstream_host}:{self.settings.dsh_port}"
cookie, _, _ = await self.dsh_cookie(
user, sandbox, fqdn, authority
)
cookie, _, _ = await self.dsh_cookie(user, sandbox, fqdn, authority)
url = f"ws://{authority}/{path}"
query = filtered_query(websocket.url.query)
if query:
@@ -1362,9 +1305,7 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
headers["x-forwarded-for"] = websocket.client.host
subprotocols = [
part.strip()
for part in websocket.headers.get(
"sec-websocket-protocol", ""
).split(",")
for part in websocket.headers.get("sec-websocket-protocol", "").split(",")
if part.strip()
]
upstream_cm = self.ws_connect(
@@ -1380,16 +1321,12 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
return await self._ws_reject(websocket, 1011)
try:
await websocket.accept(
subprotocol=getattr(upstream, "subprotocol", None)
)
await websocket.accept(subprotocol=getattr(upstream, "subprotocol", None))
except RuntimeError:
pass
await self._pump(websocket, upstream)
with contextlib.suppress(Exception):
await websocket.close(
code=self._upstream_close_code(upstream)
)
await websocket.close(code=self._upstream_close_code(upstream))
finally:
if upstream_cm is not None:
with contextlib.suppress(Exception):
@@ -1429,9 +1366,7 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
asyncio.create_task(client_to_upstream()),
asyncio.create_task(upstream_to_client()),
]
done, pending = await asyncio.wait(
tasks, return_when=asyncio.FIRST_COMPLETED
)
done, pending = await asyncio.wait(tasks, return_when=asyncio.FIRST_COMPLETED)
for task in pending:
task.cancel()
await asyncio.gather(*done, *pending, return_exceptions=True)
@@ -1458,17 +1393,13 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
continue
if self.state.in_flight.get(user, 0) > 0:
continue # requests in flight => the user is active
idle = time.time() - self.state.last_activity.setdefault(
user, time.time()
)
idle = time.time() - self.state.last_activity.setdefault(user, time.time())
if idle < self.settings.idle_timeout:
continue
self.state.maintenance[user] = True
try:
print(f"suspending {user} (idle {int(idle)}s)", flush=True)
await self.k8s.patch_operating_mode(
sandbox["metadata"]["name"], "Suspended"
)
await self.k8s.patch_operating_mode(sandbox["metadata"]["name"], "Suspended")
self._clear_handshake(user)
finally:
async with self.state.lock:
@@ -1519,12 +1450,8 @@ def create_app(
k8s = k8s if k8s is not None else Kubernetes(settings.namespace)
owns_http_client = http_client is None
if http_client is None:
http_client = httpx.AsyncClient(
follow_redirects=False, timeout=settings.http_timeout
)
gateway = Gateway(
settings, k8s, http_client, ws_connect or websockets.connect
)
http_client = httpx.AsyncClient(follow_redirects=False, timeout=settings.http_timeout)
gateway = Gateway(settings, k8s, http_client, ws_connect or websockets.connect)
routes = [
Route("/healthz", gateway.healthz),
Route("/", gateway.landing),

1
examples/dsh-agents-as-a-service/gateway/requirements-dev.txt
View File

@@ -1,4 +1,5 @@
-r requirements.txt
pytest==9.1.1
pytest-asyncio==1.4.0
pytest-cov==7.1.0
ruff==0.17.1

140
examples/dsh-agents-as-a-service/gateway/test_gateway.py
View File

@@ -45,9 +45,7 @@ FQDN = "dsh-alice-sbx.test-ns.svc.cluster.local"
def dsh_response(status=200, content=b"", headers=None):
"""A MockTransport response that is a real stream (``aiter_raw`` works)."""
return httpx.Response(
status, headers=headers, stream=httpx.ByteStream(content)
)
return httpx.Response(status, headers=headers, stream=httpx.ByteStream(content))
# --- fakes -----------------------------------------------------------------
@@ -99,9 +97,7 @@ class FakeK8s:
self.token_patches.append((user, token_hash))
claim = self.claims.get(user)
if claim is not None:
annotations = claim.setdefault("metadata", {}).setdefault(
"annotations", {}
)
annotations = claim.setdefault("metadata", {}).setdefault("annotations", {})
annotations["aaas.example.com/token-sha256"] = token_hash
async def get_pod(self, name):
@@ -118,9 +114,7 @@ class FakeUpstream:
self.requests = []
self.handshake_requests = []
self.handshake_status = 303
self.handshake_cookies = [
"dsh-auth-a=one; Path=/; HttpOnly; SameSite=Strict"
]
self.handshake_cookies = ["dsh-auth-a=one; Path=/; HttpOnly; SameSite=Strict"]
self.proxy_handler = None
self.proxy_requests = []
@@ -140,9 +134,7 @@ class FakeUpstream:
self.proxy_requests.append(request)
if self.proxy_handler is not None:
return self.proxy_handler(request)
return dsh_response(
200, b"hello from DSH", {"content-type": "text/plain"}
)
return dsh_response(200, b"hello from DSH", {"content-type": "text/plain"})
class FakeWS:
@@ -207,9 +199,7 @@ def add_user(k8s, user="alice", token=TOKEN, ready=True, pod_name=None, suffix="
"metadata": {
"name": claim_name(user),
"annotations": {
"aaas.example.com/token-sha256": hashlib.sha256(
token.encode()
).hexdigest()
"aaas.example.com/token-sha256": hashlib.sha256(token.encode()).hexdigest()
},
},
"status": {"sandbox": {"name": sandbox_name}},
@@ -224,10 +214,7 @@ def add_user(k8s, user="alice", token=TOKEN, ready=True, pod_name=None, suffix="
},
}
k8s.pods[pod_name] = {"metadata": {"name": pod_name, "uid": POD_UID}}
k8s.logs[pod_name] = (
"booting\n"
f"dsh web: http://10.0.0.7:3080/?token={TOKEN}-launch\n"
)
k8s.logs[pod_name] = f"booting\ndsh web: http://10.0.0.7:3080/?token={TOKEN}-launch\n"
return sandbox_name, pod_name
@@ -243,9 +230,7 @@ def env():
)
k8s = FakeK8s()
upstream = FakeUpstream()
http_client = httpx.AsyncClient(
transport=httpx.MockTransport(upstream), timeout=5
)
http_client = httpx.AsyncClient(transport=httpx.MockTransport(upstream), timeout=5)
ws_connect = FakeWSConnect()
app = create_app(
settings=settings,
@@ -275,9 +260,7 @@ def asgi_client(env, cookies=None):
def stale_handshake(env):
env.gateway.state.handshake[("alice", POD_UID, "gateway.test")] = (
"dsh-auth-a=stale"
)
env.gateway.state.handshake[("alice", POD_UID, "gateway.test")] = "dsh-auth-a=stale"
# --- pure helpers ----------------------------------------------------------
@@ -419,9 +402,7 @@ def test_token_matches_is_timing_safe_hash_check():
claim = {
"metadata": {
"annotations": {
"aaas.example.com/token-sha256": hashlib.sha256(
TOKEN.encode()
).hexdigest()
"aaas.example.com/token-sha256": hashlib.sha256(TOKEN.encode()).hexdigest()
}
}
}
@@ -486,14 +467,9 @@ async def test_kubernetes_create_claim_body(monkeypatch):
body = args[4]
assert body["metadata"]["name"] == "dsh-alice"
assert body["metadata"]["annotations"]["aaas.example.com/token-sha256"] == "hash"
assert (
body["metadata"]["annotations"]["aaas.example.com/managed-by"]
== "dsh-gateway"
)
assert body["metadata"]["annotations"]["aaas.example.com/managed-by"] == "dsh-gateway"
assert body["spec"]["warmPoolRef"]["name"] == "test-pool"
assert body["spec"]["additionalPodMetadata"]["labels"][
"sandbox.users.io/dsh-user"
] == "alice"
assert body["spec"]["additionalPodMetadata"]["labels"]["sandbox.users.io/dsh-user"] == "alice"
@pytest.mark.asyncio
@@ -625,9 +601,7 @@ async def test_landing_page_has_signup_and_login_forms(env):
@pytest.mark.asyncio
async def test_landing_page_shows_session_when_cookie_present(env):
async with asgi_client(
env, cookies={"dsh_gw_session": f"alice:{TOKEN}"}
) as client:
async with asgi_client(env, cookies={"dsh_gw_session": f"alice:{TOKEN}"}) as client:
response = await client.get("/")
assert "Signed in as" in response.text
assert "/u/alice/" in response.text
@@ -645,9 +619,7 @@ async def test_workspace_redirect_adds_trailing_slash(env):
@pytest.mark.asyncio
@pytest.mark.parametrize(
"bad_user", ["Alice", "-alice", "alice-", "", "a_b", 123, None, "a" * 60]
)
@pytest.mark.parametrize("bad_user", ["Alice", "-alice", "alice-", "", "a_b", 123, None, "a" * 60])
async def test_signup_rejects_invalid_dns1123_label(env, bad_user):
async with asgi_client(env) as client:
response = await client.post("/users", json={"user": bad_user})
@@ -781,9 +753,7 @@ async def test_login_json_and_failures(env):
@pytest.mark.asyncio
async def test_logout_clears_cookie(env):
async with asgi_client(
env, cookies={"dsh_gw_session": f"alice:{TOKEN}"}
) as client:
async with asgi_client(env, cookies={"dsh_gw_session": f"alice:{TOKEN}"}) as client:
response = await client.post("/logout")
assert response.status_code == 303
assert response.headers["location"] == "/"
@@ -798,12 +768,8 @@ async def test_logout_clears_cookie(env):
async def test_get_user_requires_valid_bearer_or_cookie(env):
add_user(env.k8s)
async with asgi_client(env) as client:
forbidden = await client.get(
"/users/alice", headers={"Authorization": "Bearer wrong"}
)
bearer = await client.get(
"/users/alice", headers={"Authorization": f"Bearer {TOKEN}"}
)
forbidden = await client.get("/users/alice", headers={"Authorization": "Bearer wrong"})
bearer = await client.get("/users/alice", headers={"Authorization": f"Bearer {TOKEN}"})
cookie = await client.get(
"/users/alice",
headers={"Cookie": f"dsh_gw_session=alice:{TOKEN}"},
@@ -839,9 +805,7 @@ async def test_invalid_user_in_path_is_404(env, path):
async def test_delete_user_requires_auth(env):
add_user(env.k8s)
async with asgi_client(env) as client:
response = await client.delete(
"/users/alice", headers={"Authorization": "Bearer wrong"}
)
response = await client.delete("/users/alice", headers={"Authorization": "Bearer wrong"})
assert response.status_code == 401
assert env.k8s.deleted == []
@@ -853,12 +817,8 @@ async def test_delete_with_cookie_requires_same_origin(env):
cookies = {"dsh_gw_session": f"alice:{TOKEN}"}
async with asgi_client(env, cookies=cookies) as client:
missing = await client.delete("/users/alice")
cross = await client.delete(
"/users/alice", headers={"Origin": "http://evil.test"}
)
good = await client.delete(
"/users/alice", headers={"Origin": "http://gateway.test"}
)
cross = await client.delete("/users/alice", headers={"Origin": "http://evil.test"})
good = await client.delete("/users/alice", headers={"Origin": "http://gateway.test"})
assert missing.status_code == 403
assert cross.status_code == 403
assert good.status_code == 200
@@ -870,9 +830,7 @@ async def test_delete_with_cookie_requires_same_origin(env):
async def test_delete_with_bearer_does_not_need_origin(env):
add_user(env.k8s)
async with asgi_client(env) as client:
response = await client.delete(
"/users/alice", headers={"Authorization": f"Bearer {TOKEN}"}
)
response = await client.delete("/users/alice", headers={"Authorization": f"Bearer {TOKEN}"})
assert response.status_code == 200
@@ -881,9 +839,7 @@ async def test_delete_missing_claim_is_404(env):
add_user(env.k8s)
env.k8s.claims.clear()
async with asgi_client(env) as client:
response = await client.delete(
"/users/alice", headers={"Authorization": f"Bearer {TOKEN}"}
)
response = await client.delete("/users/alice", headers={"Authorization": f"Bearer {TOKEN}"})
assert response.status_code == 404
@@ -958,9 +914,9 @@ async def test_missing_cookie_in_handshake_is_503(env):
@pytest.mark.asyncio
async def test_pod_annotation_overrides_sandbox_name(env):
sandbox_name, pod_name = add_user(env.k8s)
env.k8s.sandboxes[sandbox_name]["metadata"]["annotations"][
"agents.x-k8s.io/pod-name"
] = "custom-pod"
env.k8s.sandboxes[sandbox_name]["metadata"]["annotations"]["agents.x-k8s.io/pod-name"] = (
"custom-pod"
)
env.k8s.pods["custom-pod"] = {"metadata": {"uid": "other-uid"}}
env.k8s.logs["custom-pod"] = env.k8s.logs[pod_name]
async with asgi_client(env, cookies={"dsh_gw_session": f"alice:{TOKEN}"}) as client:
@@ -1082,9 +1038,7 @@ async def test_proxy_upstream_connection_error_is_502(env):
@pytest.mark.asyncio
async def test_proxy_body_limit_returns_413(env):
add_user(env.k8s)
env.gateway.settings = Settings(
**{**env.settings.__dict__, "max_body_bytes": 4}
)
env.gateway.settings = Settings(**{**env.settings.__dict__, "max_body_bytes": 4})
async with asgi_client(env) as client:
response = await client.post(
"/u/alice/api/echo",
@@ -1194,9 +1148,7 @@ async def test_proxied_requests_use_the_stream_timeout(env, monkeypatch):
"server": ("gateway.test", 80),
"path_params": {"user": "alice", "path": "events"},
}
response = await env.gateway._send_upstream(
Request(scope), "alice", FQDN, "dsh-auth-a=1", b""
)
response = await env.gateway._send_upstream(Request(scope), "alice", FQDN, "dsh-auth-a=1", b"")
await response.aclose()
assert captured["timeout"].read is None
@@ -1359,9 +1311,7 @@ async def test_sweeper_skips_claim_it_does_not_own(env):
assert "alice" not in env.gateway.state.last_activity
# ...and does not suspend it even when it looks long-idle.
env.gateway.state.last_activity["alice"] = (
time.time() - env.settings.idle_timeout - 10
)
env.gateway.state.last_activity["alice"] = time.time() - env.settings.idle_timeout - 10
await env.gateway.sweep_once()
assert env.k8s.patches == []
@@ -1370,9 +1320,7 @@ async def test_sweeper_skips_claim_it_does_not_own(env):
async def test_sweeper_suspends_idle_ready_sandbox(env):
env.k8s.claims["alice"] = ready_claim()
env.k8s.sandboxes["dsh-alice-sbx"] = ready_sandbox()
env.gateway.state.last_activity["alice"] = (
time.time() - env.settings.idle_timeout - 10
)
env.gateway.state.last_activity["alice"] = time.time() - env.settings.idle_timeout - 10
env.gateway.state.handshake[("alice", POD_UID, "gateway.test")] = "dsh-auth-a=x"
await env.gateway.sweep_once()
@@ -1478,9 +1426,7 @@ async def test_sso_landing_new_user_shows_provisioning(env):
async def test_sso_landing_ready_connects_and_offers_recreate_only(env):
enable_sso(env)
add_user(env.k8s)
async with asgi_client(
env, cookies={"dsh_gw_session": f"alice:{TOKEN}"}
) as client:
async with asgi_client(env, cookies={"dsh_gw_session": f"alice:{TOKEN}"}) as client:
response = await client.get("/", headers={"X-Forwarded-User": "alice"})
assert response.status_code == 200
assert "Signed in as <strong>alice</strong>" in response.text
@@ -1512,9 +1458,7 @@ async def test_recreate_deletes_and_reprovisions(env):
enable_sso(env)
add_user(env.k8s)
async with asgi_client(env) as client:
response = await client.post(
"/recreate", headers={"X-Forwarded-User": "alice"}
)
response = await client.post("/recreate", headers={"X-Forwarded-User": "alice"})
assert response.status_code == 303
assert response.headers["location"] == "/u/alice/"
assert env.k8s.deleted == ["alice"]
@@ -1526,9 +1470,7 @@ async def test_recreate_deletes_and_reprovisions(env):
async def test_recreate_requires_sso(env):
add_user(env.k8s)
async with asgi_client(env) as client:
response = await client.post(
"/recreate", headers={"X-Forwarded-User": "alice"}
)
response = await client.post("/recreate", headers={"X-Forwarded-User": "alice"})
assert response.status_code == 401
assert env.k8s.deleted == []
@@ -1537,9 +1479,7 @@ async def test_recreate_requires_sso(env):
async def test_recreate_rejects_cross_origin(env):
enable_sso(env)
add_user(env.k8s)
async with asgi_client(
env, cookies={"dsh_gw_session": f"alice:{TOKEN}"}
) as client:
async with asgi_client(env, cookies={"dsh_gw_session": f"alice:{TOKEN}"}) as client:
response = await client.post(
"/recreate",
headers={"X-Forwarded-User": "alice", "Origin": "http://evil.test"},
@@ -1553,9 +1493,7 @@ async def test_sso_get_user_authorizes_via_header(env):
enable_sso(env)
add_user(env.k8s)
async with asgi_client(env) as client:
response = await client.get(
"/users/alice", headers={"X-Forwarded-User": "alice"}
)
response = await client.get("/users/alice", headers={"X-Forwarded-User": "alice"})
assert response.status_code == 200
assert response.json()["user"] == "alice"
@@ -1565,9 +1503,7 @@ async def test_sso_proxy_redirects_to_bootstrap_without_session(env):
enable_sso(env)
add_user(env.k8s)
async with asgi_client(env) as client:
response = await client.get(
"/u/alice/", headers={"X-Forwarded-User": "alice"}
)
response = await client.get("/u/alice/", headers={"X-Forwarded-User": "alice"})
assert response.status_code == 303
assert response.headers["location"] == "/"
@@ -1576,9 +1512,7 @@ async def test_sso_proxy_redirects_to_bootstrap_without_session(env):
async def test_sso_proxy_redirects_to_bootstrap_when_claim_missing(env):
enable_sso(env)
async with asgi_client(env) as client:
response = await client.get(
"/u/alice/", headers={"X-Forwarded-User": "alice"}
)
response = await client.get("/u/alice/", headers={"X-Forwarded-User": "alice"})
assert response.status_code == 303
assert response.headers["location"] == "/"
@@ -1587,7 +1521,5 @@ async def test_sso_proxy_redirects_to_bootstrap_when_claim_missing(env):
async def test_spoofed_header_ignored_when_sso_disabled(env):
add_user(env.k8s)
async with asgi_client(env) as client:
response = await client.get(
"/u/alice/", headers={"X-Forwarded-User": "alice"}
)
response = await client.get("/u/alice/", headers={"X-Forwarded-User": "alice"})
assert response.status_code == 401

2
examples/dsh-agents-as-a-service/00-prereqs.yaml → examples/dsh-agents-as-a-service/manifests/00-prereqs.yaml
View File

@@ -5,7 +5,7 @@
# on the SandboxTemplate, stamped before any user exists — a warm-pool
# constraint), so a real key committed here would be shared by every install
# of this example. Generate a fresh value per install (README step 1;
# run-test-kind.sh uses a throwaway):
# scripts/e2e-kind.sh uses a throwaway):
#
# kubectl -n dsh-demo create secret generic dsh-provider-keys \
# --from-literal=DEEPSEEK_API_KEY="sk-..."

18
examples/dsh-agents-as-a-service/10-sandboxtemplate.yaml → examples/dsh-agents-as-a-service/manifests/10-sandboxtemplate.yaml
View File

@@ -2,6 +2,9 @@
# exists, so per-user secrets cannot live in pod env; the shared provider key
# is injected via envFrom, and claims that would bypass the warm pool are
# rejected outright (see 40-claim-rejected.yaml).
#
# A stripped-down single-tenant variant of this template (no warm pool, no
# policy enforcement) lives in examples/minimal-single-tenant/.
apiVersion: extensions.agents.x-k8s.io/v1beta1
kind: SandboxTemplate
metadata:
@@ -18,8 +21,8 @@ spec:
# The gateway routes each user to their DSH Web instance by this name.
service: true
# No gateway pod scopes ingress in this standalone demo, so the
# NetworkPolicy is left unmanaged. A production platform would use
# The example gateway does not manage NetworkPolicy, so it is left
# unmanaged here. A production platform would use
# networkPolicyManagement: Managed and restrict ingress to its gateway on
# :3080 — with the shared provider key, that NetworkPolicy is the per-user
# isolation boundary.
@@ -41,11 +44,12 @@ spec:
# with 0660 and crash-loops before the gateway can proxy to it.
# OnRootMismatch skips that walk once the volume root already carries
# the fsGroup ownership/mode applied on the first mount.
# Keep in sync with deploy/k8s/sandbox-template.yaml.
# Keep in sync with the minimal variant:
# examples/minimal-single-tenant/manifests/sandbox-template.yaml.
fsGroupChangePolicy: OnRootMismatch
containers:
- name: dsh-web
image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.2
image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.2-r2
imagePullPolicy: IfNotPresent
ports:
- name: web
@@ -61,8 +65,10 @@ spec:
value: ""
# DSH Host/Origin-fences /api. The gateway forwards the browser's
# Host, so DSH must trust the public gateway host. Empty is right
# for port-forward/localhost; for the optional Ingress set this to
# the same host as 60-ingress.yaml (e.g. dsh.example.com).
# for port-forward/localhost; for either exposure variant set this
# to the public host (e.g. dsh.example.com for
# manifests/exposure/ingress/60-ingress.yaml, or
# dsh-demo.example.com for the gateway-api-sso/ HTTPRoute).
- name: DSH_TRUSTED_HOSTS
value: ""
# Probes target /favicon.svg: DSH gates / behind its launch-token

0
examples/dsh-agents-as-a-service/20-sandboxwarmpool.yaml → examples/dsh-agents-as-a-service/manifests/20-sandboxwarmpool.yaml
View File

0
examples/dsh-agents-as-a-service/30-claim-alice.yaml → examples/dsh-agents-as-a-service/manifests/30-claim-alice.yaml
View File

0
examples/dsh-agents-as-a-service/40-claim-rejected.yaml → examples/dsh-agents-as-a-service/manifests/40-claim-rejected.yaml
View File

55
examples/dsh-agents-as-a-service/50-gateway.yaml → examples/dsh-agents-as-a-service/manifests/50-gateway.yaml
View File

@@ -6,11 +6,19 @@
# POST /users, POST /login, ... session cookie auth
# /u/<user>/... HTTP + WebSocket proxy to the user's DSH
#
# Image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway
# For local kind development you can build and load a local tag instead, then
# change `image` below to it:
# docker build -t dsh-aaas-gateway:demo gateway/
# kind load docker-image dsh-aaas-gateway:demo
# Image: the release image published for this example, pinned by tag.
# imagePullPolicy: IfNotPresent lets a local kind build loaded under the same
# tag be used without a registry round-trip (scripts/e2e-kind.sh builds
# dsh-aaas-gateway:e2e and overrides this image).
#
# SSO IS OPT-IN: this manifest runs the gateway with its own bearer/session
# auth only, so AUTH_PROXY_HEADER is intentionally NOT set (see the env
# block). To put GitHub OAuth in front of it, use
# manifests/exposure/gateway-api-sso/.
#
# This Service is ClusterIP and is the only intended entry point: expose it
# through exactly ONE of the documented variants in manifests/exposure/
# (ingress/ or gateway-api-sso/), never both and never directly.
apiVersion: v1
kind: ServiceAccount
metadata:
@@ -84,8 +92,8 @@ spec:
type: RuntimeDefault
containers:
- name: gateway
image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway
imagePullPolicy: Always
image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway:dsh-v0.2.1-rc.2-r2
imagePullPolicy: IfNotPresent
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
@@ -105,6 +113,10 @@ spec:
# gateway created are swept: a claim applied by hand from the
# resource-layer manifests carries no aaas.example.com/managed-by
# annotation and is never suspended.
#
# These are the demo values (360/240). scripts/e2e-kind.sh
# overrides both to 60/120 with `kubectl set env` so the smoke
# test does not wait six minutes for the sweeper.
- name: IDLE_TIMEOUT
value: "360"
- name: WAKE_TIMEOUT
@@ -113,19 +125,24 @@ spec:
# the DSH launch token.
- name: DSH_CONTAINER
value: dsh-web
# Cookie dsh_gw_session is HttpOnly + SameSite=Lax. Set to "true"
# when this Service is reached over HTTPS (e.g. the optional
# 60-ingress.yaml with TLS); "false" keeps the plain-HTTP demo
# usable via port-forward.
# Cookie dsh_gw_session is HttpOnly + SameSite=Lax. "false" keeps
# the plain-HTTP demo usable through port-forward: a Secure cookie
# is never sent over HTTP, so login would silently fail.
# HTTPS deployments (the ingress/ variant with TLS, or
# gateway-api-sso/, which terminates TLS at the Gateway) MUST set
# this to "true".
- name: COOKIE_SECURE
value: "true"
# Trusted reverse-proxy identity header for SSO. oauth2-proxy sets
# X-Forwarded-User to the authenticated GitHub login on every request;
# the gateway trusts it (and auto-provisions the user) only when this
# is set. Remove/unset to disable SSO and fall back to bearer/session
# auth only.
- name: AUTH_PROXY_HEADER
value: X-Forwarded-User
value: "false"
# SSO IS OPT-IN AND DISABLED HERE. With AUTH_PROXY_HEADER unset the
# gateway trusts only its own bearer/session auth and ignores every
# identity header. Enable it ONLY behind
# manifests/exposure/gateway-api-sso/, where oauth2-proxy is the
# sole route to this Service: set it to X-Forwarded-User, keep the
# Service ClusterIP and reachable only through the proxy, and
# terminate TLS in front of the gateway. Otherwise anyone who can
# reach :8080 can impersonate any user by sending that header.
# - name: AUTH_PROXY_HEADER
# value: X-Forwarded-User
# Reject proxied request bodies larger than this (413).
- name: MAX_BODY_MB
value: "32"

135
examples/dsh-agents-as-a-service/manifests/exposure/README.md Normal file
View File

@@ -0,0 +1,135 @@
# Exposing the gateway
The base manifests deploy the gateway as a ClusterIP Service
(`dsh-gateway:8080`) and expose nothing outside the cluster. WebSocket
upgrades on `/u/<user>/**` are part of the contract, so the entry point must
forward `Upgrade`/`Connection` headers (Traefik does by default).
There are two **mutually exclusive** exposure variants. Pick exactly one:
| Variant | Entry point | Authentication | TLS |
| --- | --- | --- | --- |
| [`ingress/60-ingress.yaml`](ingress/60-ingress.yaml) | Traefik `Ingress` | the gateway's own signup/session cookie | optional, recommended |
| [`gateway-api-sso/`](gateway-api-sso/72-oauth2-proxy.yaml) | Gateway API `Gateway` + `HTTPRoute` | GitHub OAuth via oauth2-proxy, then the gateway cookie | required (the Gateway listener terminates it) |
Applying both gives the gateway two entry points, and the SSO variant is only
safe when the gateway is reachable **exclusively** through oauth2-proxy. Do not
mix them.
## Applying a variant
Run these commands from the example root
(`examples/dsh-agents-as-a-service/`), the same working directory the example
README uses.
`kubectl apply -f manifests/` is **not recursive**: it applies only the
top-level numbered files and does not descend into `exposure/`. Apply the base
resources first, then the chosen variant explicitly:
```bash
# Base resources: namespace, template, warm pool, claims, gateway.
kubectl apply -f manifests/
# Plain Ingress exposure:
kubectl apply -f manifests/exposure/ingress/
# ... or SSO exposure (create the two Secrets and replace the example.com
# placeholders first — see below):
kubectl apply -f manifests/exposure/gateway-api-sso/
```
**Never use `kubectl apply -R`** (recursive): it would apply both variants at
once and hand out a second, unauthenticated path to the same gateway. Apply the
variant directory on purpose, or the individual files by name.
`kubectl` only reads `.yaml`, `.yml` and `.json` files from a directory, so
the `*.yaml.example` secret templates are skipped automatically; create the real
Secrets imperatively (commands below) before applying the directory.
## `ingress/`: plain Ingress, no SSO
`60-ingress.yaml` routes a host through the cluster's Traefik
(`ingressClassName: traefik`; k3s ships one) to the gateway Service.
1. Replace the `dsh.example.com` host and the `dsh-example-tls` Secret name
with real values, or delete the `tls:` block to serve plain HTTP.
2. Set `DSH_TRUSTED_HOSTS` in the SandboxTemplate to the same public host:
the gateway forwards the browser's `Host`, and DSH rejects unknown hosts on
`/api` (see `../10-sandboxtemplate.yaml`).
3. Set `COOKIE_SECURE=true` on the gateway once the Ingress serves HTTPS:
`kubectl -n dsh-demo set env deploy/dsh-gateway COOKIE_SECURE=true`. The
base manifest keeps `false` so the plain-HTTP port-forward demo works; a
`Secure` cookie is never sent over HTTP, which silently breaks login.
4. Do not attach buffering or compression middlewares to this router: they
break WebSocket upgrades and streaming.
Verify: `curl -sI https://<host>/healthz` returns `200`.
## `gateway-api-sso/`: TLS + GitHub OAuth
`70-gateway.yaml` defines the TLS `Gateway`, `71-httproute.yaml` sends every
request to oauth2-proxy, and `72-oauth2-proxy.yaml` authenticates the browser
against GitHub and proxies to the gateway with `X-Forwarded-User` set.
Prerequisites:
- Gateway API CRDs and a Gateway implementation installed;
`gatewayClassName: traefik` matches the Traefik Gateway provider. The CRD
version is pinned in `versions.env` (`GATEWAY_API_VERSION`).
- The OAuth App credentials and a cookie secret, from
`73-oauth2-proxy-secret.yaml.example`.
- A TLS certificate/key for the public host, from
`74-tls-secret.yaml.example`.
- oauth2-proxy is pinned in the manifest to
`quay.io/oauth2-proxy/oauth2-proxy:v7.15.5` (`versions.env`:
`OAUTH2_PROXY_IMAGE`).
Steps:
```bash
# 1. Real Secrets (the committed files are *.yaml.example templates).
kubectl -n dsh-demo create secret generic oauth2-proxy \
--from-literal=client-id='<GitHub OAuth App client ID>' \
--from-literal=client-secret='<GitHub OAuth App client secret>' \
--from-literal=cookie-secret="$(python3 -c 'import secrets; print(secrets.token_urlsafe(32))')"
kubectl -n dsh-demo create secret tls dsh-demo-tls --cert=tls.crt --key=tls.key
# 2. Replace dsh-demo.example.com (70/71/72) and your-github-org (72) with
# the real public host and allowed GitHub organization.
# 3. Enable SSO on the gateway and trust the public host in the template.
kubectl -n dsh-demo set env deploy/dsh-gateway \
AUTH_PROXY_HEADER=X-Forwarded-User COOKIE_SECURE=true
# then set DSH_TRUSTED_HOSTS=dsh-demo.example.com in the SandboxTemplate
# (see ../10-sandboxtemplate.yaml) and re-apply it.
# 4. Apply the variant.
kubectl apply -f manifests/exposure/gateway-api-sso/
```
### SSO trust boundary
`AUTH_PROXY_HEADER` is opt-in and deliberately unset in the base gateway
manifest: the gateway ignores identity headers until it is set. Enabling SSO
takes **both** sides:
1. `AUTH_PROXY_HEADER=X-Forwarded-User` on `deploy/dsh-gateway` (step 3
above). Without it, oauth2-proxy's login has no effect on the gateway and
users are still asked for the gateway's own token.
2. The gateway Service must be reachable **only through oauth2-proxy**: keep it
ClusterIP, route all browser traffic through the `HTTPRoute` →
oauth2-proxy → `dsh-gateway` chain, and never apply `ingress/` (or any
other path to the Service) while SSO is enabled. A client that can reach
`dsh-gateway:8080` directly can set `X-Forwarded-User` itself and
impersonate any user.
Also terminate TLS in front of the gateway (the Gateway listener does) and
leave `COOKIE_SECURE=true` so the session cookie never travels in clear text.
## Switching variants
Delete the variant you had before applying the other one, e.g.
`kubectl delete -f manifests/exposure/ingress/`, and revert
`AUTH_PROXY_HEADER`/`COOKIE_SECURE` on the gateway. Verify with
`kubectl -n dsh-demo get ingress,gateway,httproute` that exactly one path to
the gateway remains.

25
examples/dsh-agents-as-a-service/manifests/exposure/gateway-api-sso/70-gateway.yaml Normal file
View File

@@ -0,0 +1,25 @@
# Browser entry point (TLS) for the gateway-api-sso variant: terminates TLS
# for dsh-demo.example.com and routes through 71-httproute.yaml to
# oauth2-proxy, which authenticates the browser before the gateway.
# Mutually exclusive with ../../ingress/ (see ../../README.md). The
# "dsh-demo-tls" Secret comes from 74-tls-secret.yaml.example.
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: dsh-gateway
namespace: dsh-demo
labels:
app.kubernetes.io/part-of: dsh-demo
spec:
gatewayClassName: traefik
listeners:
- name: https
port: 8443
protocol: HTTPS
tls:
mode: Terminate
certificateRefs:
- name: dsh-demo-tls
allowedRoutes:
namespaces:
from: Same

23
examples/dsh-agents-as-a-service/manifests/exposure/gateway-api-sso/71-httproute.yaml Normal file
View File

@@ -0,0 +1,23 @@
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: dsh-gateway
namespace: dsh-demo
labels:
app.kubernetes.io/part-of: dsh-demo
spec:
parentRefs:
- name: dsh-gateway
namespace: dsh-demo
sectionName: https
hostnames:
# TODO: replace with the actual public FQDN before applying.
- dsh-demo.example.com
rules:
- backendRefs:
# Route through oauth2-proxy (GitHub OAuth) first; it authenticates the
# browser and forwards to dsh-gateway:8080 with X-Forwarded-User set.
# The gateway only trusts that header when AUTH_PROXY_HEADER is set on
# its Deployment — see 72-oauth2-proxy.yaml and ../README.md.
- name: oauth2-proxy
port: 4180

134
examples/dsh-agents-as-a-service/manifests/exposure/gateway-api-sso/72-oauth2-proxy.yaml Normal file
View File

@@ -0,0 +1,134 @@
# oauth2-proxy — GitHub OAuth login in front of the dsh-gateway.
#
# Runs as a reverse proxy: it authenticates the browser against GitHub
# (restricted to a configured organization), then forwards the request to
# the dsh-gateway Service with the authenticated identity in X-Forwarded-User
# (plus X-Auth-Request-* via --set-xauthrequest). The gateway trusts that
# header only when AUTH_PROXY_HEADER is set on its Deployment.
#
# ENABLING SSO TAKES BOTH SIDES (see ../../../exposure/README.md):
# 1. Set AUTH_PROXY_HEADER=X-Forwarded-User on deploy/dsh-gateway, otherwise
# the gateway ignores the header and the login is cosmetic:
# kubectl -n dsh-demo set env deploy/dsh-gateway \
# AUTH_PROXY_HEADER=X-Forwarded-User COOKIE_SECURE=true
# 2. Make dsh-gateway reachable ONLY through this proxy: keep the Service
# ClusterIP, route browser traffic through 71-httproute.yaml, and do not
# apply ../ingress/ (or anything else that reaches the gateway). A
# client that can reach dsh-gateway:8080 can spoof X-Forwarded-User and
# impersonate any user.
# TLS terminates at the Gateway listener (70-gateway.yaml); keep
# --cookie-secure=true and COOKIE_SECURE=true on the gateway.
#
# The image is pinned to quay.io/oauth2-proxy/oauth2-proxy:v7.15.5
# (versions.env: OAUTH2_PROXY_IMAGE). Credentials come from the
# "oauth2-proxy" Secret (see 73-oauth2-proxy-secret.yaml.example); this
# manifest never carries secrets.
apiVersion: apps/v1
kind: Deployment
metadata:
name: oauth2-proxy
namespace: dsh-demo
labels:
app: oauth2-proxy
app.kubernetes.io/part-of: dsh-demo
spec:
replicas: 1
selector:
matchLabels:
app: oauth2-proxy
template:
metadata:
labels:
app: oauth2-proxy
app.kubernetes.io/part-of: dsh-demo
spec:
securityContext:
runAsNonRoot: true
runAsUser: 2000
runAsGroup: 2000
seccompProfile:
type: RuntimeDefault
containers:
- name: oauth2-proxy
image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.5
imagePullPolicy: IfNotPresent
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: ["ALL"]
args:
- --http-address=0.0.0.0:4180
- --provider=github
# TODO: replace with the GitHub organization allowed to sign in.
- --github-org=your-github-org
- --email-domain=*
# Reverse-proxy target: the gateway backend Service.
- --upstream=http://dsh-gateway:8080
# Public callback (TLS terminates at the Gateway listener).
# TODO: replace with the actual public FQDN.
- --redirect-url=https://dsh-demo.example.com/oauth2/callback
- --cookie-secure=true
- --cookie-name=_oauth2_proxy
# Also expose X-Auth-Request-User/Email/Groups upstream.
- --set-xauthrequest
# Single provider: skip the provider-selection page.
- --skip-provider-button
env:
- name: OAUTH2_PROXY_CLIENT_ID
valueFrom:
secretKeyRef:
name: oauth2-proxy
key: client-id
- name: OAUTH2_PROXY_CLIENT_SECRET
valueFrom:
secretKeyRef:
name: oauth2-proxy
key: client-secret
- name: OAUTH2_PROXY_COOKIE_SECRET
valueFrom:
secretKeyRef:
name: oauth2-proxy
key: cookie-secret
ports:
- name: http
containerPort: 4180
livenessProbe:
httpGet:
path: /ping
port: http
initialDelaySeconds: 5
periodSeconds: 10
timeoutSeconds: 1
failureThreshold: 3
readinessProbe:
httpGet:
path: /ready
port: http
initialDelaySeconds: 5
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
resources:
requests:
cpu: 10m
memory: 32Mi
limits:
cpu: 200m
memory: 128Mi
---
apiVersion: v1
kind: Service
metadata:
name: oauth2-proxy
namespace: dsh-demo
labels:
app: oauth2-proxy
app.kubernetes.io/part-of: dsh-demo
spec:
selector:
app: oauth2-proxy
ports:
- name: http
port: 4180
targetPort: http

39
examples/dsh-agents-as-a-service/manifests/exposure/gateway-api-sso/73-oauth2-proxy-secret.yaml.example Normal file
View File

@@ -0,0 +1,39 @@
# EXAMPLE ONLY — do not apply as-is.
#
# oauth2-proxy reads its GitHub OAuth credentials and cookie-signing secret from
# this Secret. Populate real values and apply:
#
# kubectl -n dsh-demo create secret generic oauth2-proxy \
# --from-literal=client-id='<GitHub OAuth App client ID>' \
# --from-literal=client-secret='<GitHub OAuth App client secret>' \
# --from-literal=cookie-secret="$(python3 -c 'import secrets; print(secrets.token_urlsafe(32))')"
#
# The cookie secret must decode (base64) to exactly 16, 24, or 32 bytes.
# oauth2-proxy base64-decodes it, so it must be URL-safe, unpadded base64.
# Generate it with either of:
#
# python3 -c 'import secrets; print(secrets.token_urlsafe(32))'
# openssl rand -base64 32 | tr -- '+/' '-_' | tr -d '='
#
# Do NOT use plain `openssl rand -base64 32`: that yields 44 characters of
# standard base64 (with "=" and possibly "+"/"/"), which oauth2-proxy cannot
# decode and therefore treats as 44 raw bytes, failing with "cookie_secret must
# be 16, 24, or 32 bytes ... but is 44 bytes".
#
# GitHub OAuth App (https://github.com/settings/developers):
# Homepage URL https://dsh-demo.example.com
# Authorization callback https://dsh-demo.example.com/oauth2/callback
# (replace dsh-demo.example.com with the actual public FQDN)
#
# NOTE: pass the base64 value as-is via --from-literal (kubectl stores the
# literal; it does not base64-encode it a second time).
apiVersion: v1
kind: Secret
metadata:
name: oauth2-proxy
namespace: dsh-demo
type: Opaque
stringData:
client-id: "<github-oauth-app-client-id>"
client-secret: "<github-oauth-app-client-secret>"
cookie-secret: "<urlsafe-unpadded-base64-of-32-random-bytes>"

17
examples/dsh-agents-as-a-service/manifests/exposure/gateway-api-sso/74-tls-secret.yaml.example Normal file
View File

@@ -0,0 +1,17 @@
# EXAMPLE ONLY — do not apply as-is.
#
# Copy to tls-secret.yaml and populate with real base64-encoded PEM material,
# then apply. The secret must live in the dsh-demo namespace and be named
# "dsh-demo-tls" (referenced by 70-gateway.yaml).
#
# kubectl -n dsh-demo create secret tls dsh-demo-tls \
# --cert=tls.crt --key=tls.key --dry-run=client -o yaml > tls-secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: dsh-demo-tls
namespace: dsh-demo
type: kubernetes.io/tls
data:
tls.crt: <base64-encoded-certificate>
tls.key: <base64-encoded-private-key>

22
examples/dsh-agents-as-a-service/60-ingress.yaml → examples/dsh-agents-as-a-service/manifests/exposure/ingress/60-ingress.yaml
View File

@@ -1,15 +1,21 @@
# OPTIONAL browser entry point: expose the gateway through Traefik.
#
# Requires 50-gateway.yaml (Service dsh-gateway:8080) and a Traefik install
# with an IngressClass named "traefik" (k3s ships one). Replace the placeholder
# host below with a name that resolves to your ingress address, and either
# supply a real TLS Secret or delete the `tls:` block for plain HTTP.
# This is the "no SSO" exposure variant and is MUTUALLY EXCLUSIVE with
# ../gateway-api-sso/ — apply exactly one of them (see ../README.md).
#
# Requires the base manifests (Service dsh-gateway:8080 from
# manifests/50-gateway.yaml) and a Traefik install with an IngressClass named
# "traefik" (k3s ships one). Replace the placeholder host below with a name
# that resolves to your ingress address, and either supply a real TLS Secret or
# delete the `tls:` block for plain HTTP.
#
# Two things must match this host:
# 1. SandboxTemplate (10-sandboxtemplate.yaml): set DSH_TRUSTED_HOSTS to
# this host so DSH accepts the forwarded Host/Origin on /api.
# 2. Gateway Deployment (50-gateway.yaml): set COOKIE_SECURE to "true"
# once this Ingress serves HTTPS, so the session cookie is Secure.
# 1. SandboxTemplate (manifests/10-sandboxtemplate.yaml): set
# DSH_TRUSTED_HOSTS to this host so DSH accepts the forwarded Host/Origin
# on /api.
# 2. Gateway Deployment (manifests/50-gateway.yaml): set COOKIE_SECURE to
# "true" once this Ingress serves HTTPS, so the session cookie is Secure.
# The base manifest sets "false" for the plain-HTTP port-forward demo.
#
# WebSocket support is REQUIRED: the browser talks to /u/<user>/... and DSH
# uses WebSockets for live session traffic. Traefik upgrades WS/SSE

62
examples/dsh-agents-as-a-service/run-test-kind.sh → examples/dsh-agents-as-a-service/scripts/e2e-kind.sh
View File

@@ -1,13 +1,37 @@
#!/usr/bin/env bash
# Smoke test for the dsh-agents-as-a-service example: runs the full README
# walkthrough non-interactively against the current kubectl context.
# Expects agent-sandbox (with extensions) already installed.
# End-to-end smoke test for the dsh-agents-as-a-service example: runs the
# README walkthrough non-interactively against the current kubectl context.
#
# Prerequisites:
# - A kind cluster whose name matches the current kubectl context, or set
# KIND_CLUSTER=<name>.
# - agent-sandbox with extensions installed (API v1beta1). The release
# manifest is pinned by AGENT_SANDBOX_VERSION in versions.env:
# . ../../../versions.env
# kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/download/${AGENT_SANDBOX_VERSION}/sandbox-with-extensions.yaml
# - docker, to build the gateway image (skipped with SKIP_GATEWAY=1).
#
# Environment:
# SKIP_GATEWAY=1 run only the resource-layer checks (no gateway build/load,
# no signup/proxy/WebSocket/idle/wake checks)
# KIND_CLUSTER kind cluster name for "kind load docker-image"
# (default: derived from the current kubectl context)
# GW_LOCAL_IMAGE local gateway image tag
# (default: dsh-aaas-gateway:e2e)
#
# The gateway is built from the working tree (gateway/) and loaded into kind,
# so this tests the example as checked out, not the published image.
#
# The script is destructive only inside the dsh-demo namespace: it creates the
# namespace (and a throwaway provider key) and DELETES it on exit via the
# cleanup trap.
#
# NOTE: the only local port bound is the gateway port-forward on 18080.
# The DSH Web container port (3080) is in-cluster only and is never bound
# on the local host.
set -euo pipefail
cd "$(dirname "$0")"
# Example root: paths below are manifests/... and gateway/... from here.
cd "$(dirname "$0")/.."
NS=dsh-demo
pass() { echo "PASS: $1"; }
@@ -20,12 +44,12 @@ cleanup() {
trap cleanup EXIT
echo "=== 1. install prereqs (throwaway key) + template + warm pool"
kubectl apply -f 00-prereqs.yaml
kubectl apply -f manifests/00-prereqs.yaml
# The provider key is never committed; generate a throwaway value.
kubectl -n "$NS" delete secret dsh-provider-keys --ignore-not-found >/dev/null
kubectl -n "$NS" create secret generic dsh-provider-keys \
--from-literal=DEEPSEEK_API_KEY="sk-test-$(openssl rand -hex 16)"
kubectl apply -f 10-sandboxtemplate.yaml -f 20-sandboxwarmpool.yaml
kubectl apply -f manifests/10-sandboxtemplate.yaml -f manifests/20-sandboxwarmpool.yaml
for _ in $(seq 1 120); do
READY=$(kubectl -n "$NS" get sandboxes -l agents.x-k8s.io/warm-pool-sandbox \
-o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' | grep -c True || true)
@@ -37,7 +61,7 @@ pass "warm pool: 2 spares Ready"
echo "=== 2. signup via claim (warm adoption)"
T0=$(date +%s)
kubectl apply -f 30-claim-alice.yaml
kubectl apply -f manifests/30-claim-alice.yaml
kubectl -n "$NS" wait sandboxclaim dsh-alice --for=condition=Ready --timeout=60s
ELAPSED=$(( $(date +%s) - T0 ))
SB=$(kubectl -n "$NS" get sandboxclaim dsh-alice -o jsonpath='{.status.sandbox.name}')
@@ -65,7 +89,7 @@ MARKER=$(kubectl -n "$NS" exec "$POD" -- cat /home/node/.dsh/marker.txt)
pass "resumed: state survived on the reattached PVC"
echo "=== 6. injection policy rejects env-carrying claims"
kubectl apply -f 40-claim-rejected.yaml
kubectl apply -f manifests/40-claim-rejected.yaml
# Wait for the POSITIVE rejection signal, not merely "not bound yet".
REASON=""
for _ in $(seq 1 12); do
@@ -78,19 +102,25 @@ done
|| fail "claim was not rejected (Ready reason='$REASON')"
BOUND=$(kubectl -n "$NS" get sandboxclaim dsh-mallory -o jsonpath='{.status.sandbox.name}' 2>/dev/null || true)
[ -z "$BOUND" ] || fail "env-carrying claim unexpectedly bound sandbox '$BOUND'"
kubectl delete -f 40-claim-rejected.yaml >/dev/null
kubectl delete -f manifests/40-claim-rejected.yaml >/dev/null
pass "claim rejected with EnvVarsInjectionRejected and never bound"
echo "=== 7. gateway: browser signup, proxy, idle-suspend, wake-on-connect"
if command -v docker >/dev/null && command -v kind >/dev/null; then
KIND_CLUSTER="${KIND_CLUSTER:-$(kubectl config current-context | sed 's/^kind-//')}"
# Tag exactly what 50-gateway.yaml references so the smoke test exercises
# the shipped manifest unchanged (imagePullPolicy: IfNotPresent uses the
# image loaded into kind).
GW_IMAGE=ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway
docker build -q -t "$GW_IMAGE" gateway/ >/dev/null
kind load docker-image "$GW_IMAGE" --name "$KIND_CLUSTER" >/dev/null
kubectl apply -f 50-gateway.yaml
GW_LOCAL_IMAGE="${GW_LOCAL_IMAGE:-dsh-aaas-gateway:e2e}"
# Test the working tree, not the published image: build the gateway from
# gateway/, load it into kind, then point the Deployment at that tag. The
# shipped manifest pins the release image; imagePullPolicy: IfNotPresent
# makes the node use the loaded one.
docker build -q -t "$GW_LOCAL_IMAGE" gateway/ >/dev/null
kind load docker-image "$GW_LOCAL_IMAGE" --name "$KIND_CLUSTER" >/dev/null
kubectl apply -f manifests/50-gateway.yaml
kubectl -n "$NS" set image deploy/dsh-gateway gateway="$GW_LOCAL_IMAGE"
# Override the demo timeouts from manifests/50-gateway.yaml (360/240) with the
# gateway's working-tree defaults, so idle-suspend and wake-on-connect are
# exercised in seconds instead of minutes.
kubectl -n "$NS" set env deploy/dsh-gateway IDLE_TIMEOUT=60 WAKE_TIMEOUT=120
kubectl -n "$NS" rollout status deploy/dsh-gateway --timeout=120s
# Local port-forward uses 18080 (NOT 3080, which stays in-cluster).
kubectl -n "$NS" port-forward svc/dsh-gateway 18080:8080 >/dev/null 2>&1 &