mirror of
https://github.com/sredevopsorg/deepseek-kubernetes-sandbox.git
synced 2026-11-03 01:13:24 +00:00
feat(repo)!: repo reorganization (#16)
* feat(repo)!: restructure around the dsh-harness image and one example per implementation
- the root Dockerfile is the primary artifact; versions.env is the single source of truth
- merge gateway-api/dsh-demo into examples/dsh-agents-as-a-service (manifests/, exposure variants)
- move the single-tenant template into examples/minimal-single-tenant
- publish the gateway as a nested package under the repository image
- mirror the DSH ref for release identities (first release dsh-v0.2.1-rc.2-r2)
- CI: lint/python/manifests/gateway-image, reusable multi-arch build, tag-driven release, weekly e2e
- docs: README, docs/{main-image,examples,gitflow,versioning}, example READMEs, AGENTS.md, governance
- SSO is opt-in again; gateway pull policy and cookie defaults match the documented demo
BREAKING CHANGE: example manifest paths, the gateway image name, and the removal
of deploy/ and gateway-api/ affect existing deployments; see the migration table
in README.md.
* docs(changelog): generate initial changelog from history
* fix(scripts): make set-version update Dockerfile and sample revision pins
* build(hadolint): document intentional lint exceptions for the main image
---------
Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
This commit is contained in:
Nicolás Georger
47
examples/README.md
Normal file
47
examples/README.md
Normal file
@@ -0,0 +1,47 @@
|
||||
# Examples
|
||||
|
||||
Every implementation in this repository is a self-contained example: one
|
||||
directory, plain `kubectl`, an end-to-end README, and a cleanup command.
|
||||
Examples consume the [main DSH Web image](../docs/main-image.md); they are
|
||||
reference material, not libraries.
|
||||
|
||||
| Example | What it demonstrates | Needs |
|
||||
| --- | --- | --- |
|
||||
| [`examples/dsh-agents-as-a-service/`](dsh-agents-as-a-service/) | The full multi-tenant platform pattern: one `SandboxTemplate` with per-user PVCs, a warm pool, one ~10-line `SandboxClaim` per user, suspend-when-idle / wake-on-return, and a browser gateway with signup, `HttpOnly` sessions, an HTTP + WebSocket proxy and two exposure variants (Ingress, or Gateway API + GitHub SSO). | agent-sandbox with extensions; kind for the e2e script. |
|
||||
| [`examples/minimal-single-tenant/`](minimal-single-tenant/) | The smallest useful setup: a single `SandboxTemplate` and one `SandboxClaim`, no gateway — reach DSH Web with a launch-token URL through `kubectl port-forward`. | agent-sandbox with extensions. |
|
||||
|
||||
## The example contract (short version)
|
||||
|
||||
Full rules: [docs/examples.md](../docs/examples.md). `scripts/check-examples.sh`
|
||||
enforces the layout, and `make check` runs it.
|
||||
|
||||
```
|
||||
examples/<name>/
|
||||
+-- README.md # walkthrough: prerequisites, steps, troubleshooting, cleanup
|
||||
+-- manifests/ # NN-*.yaml, applied with plain kubectl
|
||||
| +-- exposure/<variant>/ # optional per-variant manifests...
|
||||
| +-- exposure/README.md # ...required whenever exposure/ exists
|
||||
+-- scripts/ # optional executable, idempotent, self-cleaning *.sh
|
||||
+-- docs/ # optional architecture/ADR material
|
||||
+-- <component>/ # optional component with Dockerfile + README + tests
|
||||
```
|
||||
|
||||
- Secrets never ship populated: use `*.yaml.example` files with placeholders.
|
||||
- Manifests are numbered (`00-`, `10-`, ...) and read as an ordered walkthrough.
|
||||
- Component images are published nested below the main image:
|
||||
`<registry>/<owner>/<repo>/<example>/<component>`.
|
||||
- Every example must be listed above with its literal `examples/<name>/` path.
|
||||
- Scripts under `scripts/` must be executable, idempotent and clean up after
|
||||
themselves.
|
||||
|
||||
## Running an example
|
||||
|
||||
```bash
|
||||
# e.g. the multi-tenant platform example
|
||||
kubectl apply -f examples/dsh-agents-as-a-service/manifests/
|
||||
|
||||
# or run the whole walkthrough (kind + agent-sandbox required) and let it clean up
|
||||
examples/dsh-agents-as-a-service/scripts/e2e-kind.sh
|
||||
```
|
||||
|
||||
Each example README has the exact prerequisites, exposure options and teardown.
|
||||
@@ -2,20 +2,41 @@
|
||||
|
||||
This example distills the upstream
|
||||
[hermes-agents-as-a-service](https://github.com/kubernetes-sigs/agent-sandbox/tree/main/examples/hermes-agents-as-a-service)
|
||||
platform pattern into pure agent-sandbox resources + kubectl, running **DeepSeek Harness Web
|
||||
(DSH)** instead of Hermes. Every user of such a platform gets a personal, long-lived DSH Web
|
||||
instance that is **provisioned in ~2 seconds from a warm pool, suspended when idle to save
|
||||
cost, and woken transparently when the user returns** — with all state (conversations,
|
||||
sessions, config) surviving on a per-user PVC at `/home/node/.dsh`.
|
||||
platform pattern into pure agent-sandbox resources + kubectl, running **DeepSeek
|
||||
Harness Web (DSH)** instead of Hermes. Every user of such a platform gets a
|
||||
personal, long-lived DSH Web instance that is **provisioned in ~2 seconds from a
|
||||
warm pool, suspended when idle to save cost, and woken transparently when the
|
||||
user returns** — with all state (conversations, sessions, config) surviving on a
|
||||
per-user PVC at `/home/node/.dsh`.
|
||||
|
||||
It ships in two layers:
|
||||
It ships in three layers:
|
||||
|
||||
1. **The resource layer** (steps 1–7): pure manifests + kubectl, so every mechanism is visible.
|
||||
2. **The service layer** (step 8): a small included **gateway** (`gateway/`) that makes the
|
||||
platform browser-usable — a signup page/API, `HttpOnly` cookie sessions, a `/u/{user}/**`
|
||||
proxy for HTTP **and WebSocket** that transparently **wakes suspended instances on
|
||||
connect**, and an idle sweeper that suspends them again. The gateway performs the DSH
|
||||
launch-token handshake server-side, so the browser never sees the token.
|
||||
1. **The resource layer** (steps 1–7): pure manifests + kubectl, so every
|
||||
mechanism is visible.
|
||||
2. **The service layer** (step 8): a small included **gateway** (`gateway/`) that
|
||||
makes the platform browser-usable — a signup page/API, `HttpOnly` cookie
|
||||
sessions, a `/u/{user}/**` proxy for HTTP **and WebSocket** that transparently
|
||||
**wakes suspended instances on connect**, and an idle sweeper that suspends
|
||||
them again. The gateway performs the DSH launch-token handshake server-side,
|
||||
so the browser never sees the token.
|
||||
3. **The exposure layer** (step 9): choose exactly one way in — a plain Traefik
|
||||
Ingress, or Gateway API + oauth2-proxy (GitHub SSO).
|
||||
|
||||
All manifests live under [`manifests/`](manifests/):
|
||||
|
||||
```
|
||||
manifests/
|
||||
+-- 00-prereqs.yaml # namespace + empty provider-key Secret
|
||||
+-- 10-sandboxtemplate.yaml # the DSH Web blueprint
|
||||
+-- 20-sandboxwarmpool.yaml # 2 pre-warmed spares
|
||||
+-- 30-claim-alice.yaml # one user
|
||||
+-- 40-claim-rejected.yaml # negative demo: env injection rejected
|
||||
+-- 50-gateway.yaml # Deployment/RBAC/Service for the gateway
|
||||
+-- exposure/
|
||||
+-- README.md # the two exposure variants, when to use which
|
||||
+-- ingress/60-ingress.yaml
|
||||
+-- gateway-api-sso/70-gateway.yaml ... 74-tls-secret.yaml.example
|
||||
```
|
||||
|
||||
What it demonstrates, feature by feature:
|
||||
|
||||
@@ -34,20 +55,21 @@ What it demonstrates, feature by feature:
|
||||
|
||||
- **Multi-arch**: the DSH image is built for amd64 and arm64, so there is no
|
||||
`kubernetes.io/arch: amd64` nodeSelector.
|
||||
- **Single port**: DSH Web serves everything on `3080`; the gateway proxies to one upstream
|
||||
instead of splitting `/v1/*` (API) from the dashboard.
|
||||
- **One shared secret**: only `DEEPSEEK_API_KEY` is injected (via `envFrom`); there is no
|
||||
dashboard basic-auth, session-cookie secret, or API-server key.
|
||||
- **Launch-token handshake**: DSH gates `/` behind a token it prints only in its logs. The
|
||||
gateway reads it, performs the handshake, and keeps the resulting auth cookie server-side;
|
||||
the browser authenticates to the gateway with its own cookie. See
|
||||
[How the DSH handshake works](#how-the-dsh-handshake-works).
|
||||
- **Host allow-list**: DSH rejects unknown `Host` headers. The gateway forwards the browser's
|
||||
`Host`, so the public hostname must be added to `DSH_TRUSTED_HOSTS` in the template.
|
||||
- **Probes target `/favicon.svg`**: `/` returns `401` without the launch token, so a probe on
|
||||
`/` crash-loops the pod. `/favicon.svg` is unauthenticated, and a `startupProbe` gates
|
||||
liveness/readiness so a slow cold start (plugin activation, or a resumed sandbox) is not
|
||||
restarted mid-boot.
|
||||
- **Single port**: DSH Web serves everything on `3080`; the gateway proxies to one
|
||||
upstream instead of splitting `/v1/*` (API) from the dashboard.
|
||||
- **One shared secret**: only `DEEPSEEK_API_KEY` is injected (via `envFrom`); there
|
||||
is no dashboard basic-auth, session-cookie secret, or API-server key.
|
||||
- **Launch-token handshake**: DSH gates `/` behind a token it prints only in its
|
||||
logs. The gateway reads it, performs the handshake, and keeps the resulting
|
||||
auth cookie server-side; the browser authenticates to the gateway with its own
|
||||
cookie. See [How the DSH handshake works](#how-the-dsh-handshake-works).
|
||||
- **Host allow-list**: DSH rejects unknown `Host` headers. The gateway forwards
|
||||
the browser's `Host`, so the public hostname must be added to
|
||||
`DSH_TRUSTED_HOSTS` in the template.
|
||||
- **Probes target `/favicon.svg`**: `/` returns `401` without the launch token, so
|
||||
a probe on `/` crash-loops the pod. `/favicon.svg` is unauthenticated, and a
|
||||
`startupProbe` gates liveness/readiness so a slow cold start (plugin
|
||||
activation, or a resumed sandbox) is not restarted mid-boot.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
@@ -55,43 +77,49 @@ What it demonstrates, feature by feature:
|
||||
- agent-sandbox with extensions installed:
|
||||
|
||||
```bash
|
||||
VERSION=v1.0.7
|
||||
kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/download/${VERSION}/sandbox-with-extensions.yaml
|
||||
VERSION=v1.0.7 # AGENT_SANDBOX_VERSION in versions.env
|
||||
kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/download/$VERSION/sandbox-with-extensions.yaml
|
||||
kubectl -n agent-sandbox-system rollout status deployment --timeout=180s
|
||||
```
|
||||
|
||||
Or run everything below in one shot with `./run-test-kind.sh` (expects a kind cluster + the
|
||||
manifest above already applied; it creates and deletes the namespace itself).
|
||||
For step 9's SSO variant you additionally need the Gateway API CRDs and a
|
||||
Gateway implementation (`gatewayClassName: traefik` in the manifests); see the
|
||||
[exposure README](manifests/exposure/README.md).
|
||||
|
||||
- Optional: run the whole resource + gateway walkthrough in one shot with
|
||||
[`scripts/e2e-kind.sh`](scripts/e2e-kind.sh). It expects a kind cluster with
|
||||
agent-sandbox installed, then creates and deletes the `dsh-demo` namespace
|
||||
itself. It stops after the gateway layer (step 8); exposure is not applied.
|
||||
|
||||
## 1. Install the platform resources
|
||||
|
||||
First the namespace, then the shared provider key. The key reaches **every** tenant sandbox
|
||||
(a warm-pool constraint: pod env is stamped before any user exists), so it is **never
|
||||
committed** — generate a fresh value per install:
|
||||
First the namespace, then the shared provider key. The key reaches **every**
|
||||
tenant sandbox (a warm-pool constraint: pod env is stamped before any user
|
||||
exists), so it is **never committed** — generate a fresh value per install:
|
||||
|
||||
```bash
|
||||
kubectl apply -f 00-prereqs.yaml
|
||||
kubectl apply -f manifests/00-prereqs.yaml
|
||||
kubectl -n dsh-demo create secret generic dsh-provider-keys \
|
||||
--from-literal=DEEPSEEK_API_KEY="sk-..."
|
||||
kubectl apply -f 10-sandboxtemplate.yaml -f 20-sandboxwarmpool.yaml
|
||||
kubectl apply -f manifests/10-sandboxtemplate.yaml -f manifests/20-sandboxwarmpool.yaml
|
||||
```
|
||||
|
||||
Watch the two warm spares boot (first time pulls the image — subsequent operations never pay
|
||||
that again; that is the point of the pool):
|
||||
Watch the two warm spares boot (first time pulls the image — subsequent operations
|
||||
never pay that again; that is the point of the pool):
|
||||
|
||||
```bash
|
||||
kubectl -n dsh-demo get sandboxes -l agents.x-k8s.io/warm-pool-sandbox -w
|
||||
```
|
||||
|
||||
Wait until both show `READY: True`. Each spare already has its PVC bound and DSH Web serving
|
||||
on port `3080`.
|
||||
Wait until both show `READY: True`. Each spare already has its PVC bound and DSH
|
||||
Web serving on port `3080`.
|
||||
|
||||
## 2. "Signup": claim an instance (~2s)
|
||||
|
||||
A user signing up is one small claim — `warmPoolRef` plus a pod label:
|
||||
|
||||
```bash
|
||||
kubectl apply -f 30-claim-alice.yaml
|
||||
kubectl apply -f manifests/30-claim-alice.yaml
|
||||
kubectl -n dsh-demo wait sandboxclaim dsh-alice --for=condition=Ready --timeout=60s
|
||||
```
|
||||
|
||||
@@ -105,20 +133,21 @@ kubectl -n dsh-demo get sandboxes --show-labels
|
||||
|
||||
Three things worth noticing:
|
||||
|
||||
- **The sandbox keeps its pool-generated name** (`dsh-pool-…`), not the claim's name —
|
||||
adoption re-labels an existing warm sandbox rather than creating one. A platform must always
|
||||
resolve user → sandbox through `claim.status.sandbox.name`.
|
||||
- The `agents.x-k8s.io/warm-pool-sandbox` label is **gone** from the adopted sandbox (it left
|
||||
the pool's inventory), and the pool has already started replenishing back to 2 spares.
|
||||
- **The sandbox keeps its pool-generated name** (`dsh-pool-…`), not the claim's
|
||||
name — adoption re-labels an existing warm sandbox rather than creating one. A
|
||||
platform must always resolve user → sandbox through `claim.status.sandbox.name`.
|
||||
- The `agents.x-k8s.io/warm-pool-sandbox` label is **gone** from the adopted
|
||||
sandbox (it left the pool's inventory), and the pool has already started
|
||||
replenishing back to 2 spares.
|
||||
- The pod carries the claim's `additionalPodMetadata` label
|
||||
(`sandbox.users.io/dsh-user=alice`), and
|
||||
`kubectl -n dsh-demo get sandbox $SB -o jsonpath='{.status.serviceFQDN}'` shows the stable
|
||||
DNS name the gateway proxies to.
|
||||
`kubectl -n dsh-demo get sandbox $SB -o jsonpath='{.status.serviceFQDN}'` shows
|
||||
the stable DNS name the gateway proxies to.
|
||||
|
||||
## 3. Talk to the instance
|
||||
|
||||
DSH Web needs its launch token on first contact — a bare `/` returns `401`. The token is in the
|
||||
pod logs:
|
||||
DSH Web needs its launch token on first contact — a bare `/` returns `401`. The
|
||||
token is in the pod logs:
|
||||
|
||||
```bash
|
||||
kubectl -n dsh-demo logs "$SB" | grep -m1 '?token='
|
||||
@@ -127,16 +156,16 @@ kubectl -n dsh-demo port-forward "pod/$SB" 18081:3080
|
||||
# open http://localhost:18081/?token=... (from the log line)
|
||||
```
|
||||
|
||||
> **Local port note:** DSH Web listens on `3080` **in-cluster**. If `3080` is already in use
|
||||
> on your workstation (e.g. by a local DSH), forward to a different local port (here `18081`),
|
||||
> never `3080`. `localhost` is always in DSH's trusted-host list, so the forwarded request is
|
||||
> accepted.
|
||||
> **Local port note:** DSH Web listens on `3080` **in-cluster**. If `3080` is
|
||||
> already in use on your workstation (e.g. by a local DSH), forward to a
|
||||
> different local port (here `18081`), never `3080`. `localhost` is always in
|
||||
> DSH's trusted-host list, so the forwarded request is accepted.
|
||||
|
||||
> **Why the pod stays up:** this claim was applied by hand, so it carries no gateway ownership
|
||||
> annotation (`aaas.example.com/managed-by`). The gateway's idle sweeper only suspends claims it
|
||||
> created through `POST /users`; the resource-layer claims from these manifests are never
|
||||
> touched. You can deploy the gateway (step 8) and still keep a sandbox you are port-forwarding
|
||||
> to.
|
||||
> **Why the pod stays up:** this claim was applied by hand, so it carries no
|
||||
> gateway ownership annotation (`aaas.example.com/managed-by`). The gateway's idle
|
||||
> sweeper only suspends claims it created through `POST /users`; the resource-layer
|
||||
> claims from these manifests are never touched. You can deploy the gateway
|
||||
> (step 8) and still keep a sandbox you are port-forwarding to.
|
||||
|
||||
## 4. Write state on the PVC
|
||||
|
||||
@@ -161,20 +190,20 @@ kubectl -n dsh-demo wait sandbox "$SB" --for=condition=Ready --timeout=180s
|
||||
kubectl -n dsh-demo exec "$SB" -- cat /home/node/.dsh/marker.txt # "remember me"
|
||||
```
|
||||
|
||||
> **Why resume works:** the pod template sets `fsGroupChangePolicy: OnRootMismatch`. DSH requires
|
||||
> its `.credentials.yaml` to be `0600`, while the kubelet's default fsGroup policy widens every
|
||||
> file to `0660` on each pod start — a resumed sandbox would crash-loop. See
|
||||
> [Troubleshooting](#troubleshooting) for the one-time fix on sandboxes created before this
|
||||
> setting existed.
|
||||
> **Why resume works:** the pod template sets `fsGroupChangePolicy: OnRootMismatch`.
|
||||
> DSH requires its `.credentials.yaml` to be `0600`, while the kubelet's default
|
||||
> fsGroup policy widens every file to `0660` on each pod start — a resumed sandbox
|
||||
> would crash-loop. See [Troubleshooting](#troubleshooting) for the one-time fix
|
||||
> on sandboxes created before this setting existed.
|
||||
|
||||
## 7. Injection policy rejects env-carrying claims
|
||||
|
||||
```bash
|
||||
kubectl apply -f 40-claim-rejected.yaml
|
||||
kubectl apply -f manifests/40-claim-rejected.yaml
|
||||
kubectl -n dsh-demo get sandboxclaim dsh-mallory \
|
||||
-o jsonpath='{.status.conditions[?(@.type=="Ready")].reason}'
|
||||
# EnvVarsInjectionRejected — the claim never bound a sandbox
|
||||
kubectl delete -f 40-claim-rejected.yaml
|
||||
kubectl delete -f manifests/40-claim-rejected.yaml
|
||||
```
|
||||
|
||||
## 8. The gateway: signup, browser proxy, idle-suspend, wake-on-connect
|
||||
@@ -182,59 +211,66 @@ kubectl delete -f 40-claim-rejected.yaml
|
||||
### Deploy it
|
||||
|
||||
```bash
|
||||
# Published image (on kind, load it into the cluster; on a real cluster it is
|
||||
# pulled from GHCR):
|
||||
docker pull ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway
|
||||
kind load docker-image ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway --name <kind-cluster>
|
||||
# Published image (on kind, load it into the cluster; on a real cluster the
|
||||
# kubelet pulls it from GHCR):
|
||||
GW=ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway:dsh-v0.2.1-rc.2-r2
|
||||
docker pull "$GW"
|
||||
kind load docker-image "$GW" --name <kind-cluster>
|
||||
|
||||
# ... or build it locally and tag it exactly as 50-gateway.yaml expects:
|
||||
docker build -t ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway gateway/
|
||||
kind load docker-image ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway --name <kind-cluster>
|
||||
# ... or build it locally and tag it exactly as manifests/50-gateway.yaml
|
||||
# expects (make gateway builds dsh-aaas-gateway:dev):
|
||||
make gateway
|
||||
docker tag dsh-aaas-gateway:dev "$GW"
|
||||
kind load docker-image "$GW" --name <kind-cluster>
|
||||
|
||||
kubectl apply -f 50-gateway.yaml
|
||||
kubectl apply -f manifests/50-gateway.yaml
|
||||
kubectl -n dsh-demo rollout status deploy/dsh-gateway --timeout=120s
|
||||
kubectl -n dsh-demo port-forward svc/dsh-gateway 18080:8080
|
||||
```
|
||||
|
||||
> The gateway binds `8080`; the local port-forward uses `18080` and never `3080`.
|
||||
>
|
||||
> **Re-deploying after a new image?** The workflow publishes the mutable `:gateway` tag and the
|
||||
> Deployment uses `imagePullPolicy: IfNotPresent`, so a node that already cached the tag keeps
|
||||
> the old image across `kubectl apply`/`rollout restart`. Pin the digest for a reproducible
|
||||
> rollout (the workflow's `Inspect image` step and `docker buildx imagetools inspect` both show
|
||||
> it):
|
||||
> **Image pull policy.** `manifests/50-gateway.yaml` uses
|
||||
> `imagePullPolicy: IfNotPresent` with the release identity, so a locally built
|
||||
> image must be loaded into the cluster under exactly that name (kind:
|
||||
> `kind load`). A node that already cached a different tag is the usual cause of
|
||||
> "my new build is not running"; pin the digest for a reproducible rollout:
|
||||
>
|
||||
> ```bash
|
||||
> kubectl -n dsh-demo set image deploy/dsh-gateway \
|
||||
> gateway=ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox@sha256:<digest>
|
||||
> gateway=ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway@sha256:<digest>
|
||||
> kubectl -n dsh-demo rollout status deploy/dsh-gateway --timeout=120s
|
||||
> ```
|
||||
>
|
||||
> **Applying over an existing install?** Run `kubectl diff -f 10-sandboxtemplate.yaml`
|
||||
> first. Cluster-specific fields such as `nodeAffinity` and the warm-pool replica count are
|
||||
> intentionally not in the repo manifests; preserve them with a targeted patch instead of a
|
||||
> **Applying over an existing install?** Run
|
||||
> `kubectl diff -f manifests/10-sandboxtemplate.yaml` first. Cluster-specific
|
||||
> fields such as `nodeAffinity` and the warm-pool replica count are intentionally
|
||||
> not in the repo manifests; preserve them with a targeted patch instead of a
|
||||
> blind apply. Moving the probes to `/favicon.svg` is the crash-loop fix.
|
||||
>
|
||||
> The template also sets `fsGroupChangePolicy: OnRootMismatch` (see the note in step 6).
|
||||
> Existing `Sandbox` objects snapshot their pod template, so a template apply does not update
|
||||
> them: patch each sandbox's `spec.podTemplate.spec.securityContext.fsGroupChangePolicy` and
|
||||
> The template also sets `fsGroupChangePolicy: OnRootMismatch` (see the note in
|
||||
> step 6). Existing `Sandbox` objects snapshot their pod template, so a template
|
||||
> apply does not update them: patch each sandbox's
|
||||
> `spec.podTemplate.spec.securityContext.fsGroupChangePolicy` and
|
||||
> `chmod 600` its `.credentials.yaml`, or recreate it.
|
||||
|
||||
### Browser flow
|
||||
|
||||
Open <http://localhost:18080>. The landing page has a one-field signup form. Enter `bob`, and
|
||||
the gateway:
|
||||
Open <http://localhost:18080>. The landing page has a one-field signup form.
|
||||
Enter `bob`, and the gateway:
|
||||
|
||||
1. creates the `SandboxClaim` over the warm pool,
|
||||
2. waits for it to become `Ready`,
|
||||
3. sets an `HttpOnly`, `SameSite=Lax` cookie `dsh_gw_session=<user>:<token>` on the browser,
|
||||
3. sets an `HttpOnly`, `SameSite=Lax` cookie `dsh_gw_session=<user>:<token>` on
|
||||
the browser,
|
||||
4. redirects to `/u/bob/`, where the DSH Web UI loads through the gateway.
|
||||
|
||||
From then on every request under `/u/bob/` (assets, API calls, SSE, WebSocket) is proxied to
|
||||
bob's sandbox. The DSH launch token and DSH's own auth cookie stay in the gateway; the browser
|
||||
only holds the gateway cookie.
|
||||
From then on every request under `/u/bob/` (assets, API calls, SSE, WebSocket) is
|
||||
proxied to bob's sandbox. The DSH launch token and DSH's own auth cookie stay in
|
||||
the gateway; the browser only holds the gateway cookie.
|
||||
|
||||
The same flow with `curl`, no JavaScript (the session cookie is the only credential):
|
||||
The same flow with `curl`, no JavaScript (the session cookie is the only
|
||||
credential):
|
||||
|
||||
```bash
|
||||
JAR=$(mktemp)
|
||||
@@ -252,48 +288,53 @@ curl -s -b "$JAR" -H 'Origin: http://localhost:18080' -X DELETE http://localhost
|
||||
```
|
||||
|
||||
API clients that prefer bearer tokens can use `POST /users` with a JSON body
|
||||
(`{"user":"bob"}`) and the returned one-time token, `POST /login` to exchange a token for a
|
||||
cookie, and `Authorization: Bearer <token>` on subsequent calls.
|
||||
(`{"user":"bob"}`) and the returned one-time token, `POST /login` to exchange a
|
||||
token for a cookie, and `Authorization: Bearer <token>` on subsequent calls.
|
||||
|
||||
### Gateway API
|
||||
|
||||
| Route | Purpose |
|
||||
| --- | --- |
|
||||
| `GET /healthz` | Liveness/readiness; no cluster access |
|
||||
| `GET /` | Landing page: signup, token login, logout |
|
||||
| `GET /` | Landing page: signup, token login, logout. With SSO it auto-provisions and connects instead. |
|
||||
| `POST /users` | Signup. JSON or form; `201` (JSON), `202` (waking), `303` (browser redirect), `409` if user exists; sets the session cookie |
|
||||
| `POST /login` | Exchange a bearer token for the session cookie |
|
||||
| `POST /logout` | Clear the session cookie |
|
||||
| `POST /recreate` | SSO only: delete the authenticated user's claim (wiping workspace data) and provision a fresh one; `303` to `/u/{user}/`. Cookie-authenticated calls must be same-origin. |
|
||||
| `GET /users/{user}` | Derived state (`Ready`/`Waking`/`Suspended`) |
|
||||
| `DELETE /users/{user}` | Delete claim (cascades sandbox + PVC) |
|
||||
| `GET /u/{user}` | `308` redirect to `/u/{user}/` |
|
||||
| `GET /u/{user}/**` | HTTP/SSE proxy to the user's DSH Web |
|
||||
| `WS /u/{user}/**` | WebSocket proxy to the user's DSH Web |
|
||||
|
||||
The full API reference, configuration table, auth semantics and failure matrix
|
||||
live in the [gateway README](gateway/README.md).
|
||||
|
||||
### How the DSH handshake works
|
||||
|
||||
DSH does not expose an API for its launch token; it prints a URL containing `?token=…` in the
|
||||
pod logs. For each `(user, pod UID, Host)` the gateway:
|
||||
DSH does not expose an API for its launch token; it prints a URL containing
|
||||
`?token=…` in the pod logs. For each `(user, pod UID, Host)` the gateway:
|
||||
|
||||
1. resolves the claim → sandbox → pod (annotation `agents.x-k8s.io/pod-name`, or the sandbox
|
||||
name as pod name),
|
||||
2. reads the pod's logs (last `DSH_LOG_TAIL_LINES` lines) and parses the most recent
|
||||
`dsh web: …?token=…` line,
|
||||
1. resolves the claim → sandbox → pod (annotation `agents.x-k8s.io/pod-name`, or
|
||||
the sandbox name as pod name),
|
||||
2. reads the pod's logs (last `DSH_LOG_TAIL_LINES` lines) and parses the most
|
||||
recent `dsh web: …?token=…` line,
|
||||
3. requests `http://<serviceFQDN>:3080/?token=…` with the browser's `Host`,
|
||||
4. captures DSH's `dsh-auth-*` `Set-Cookie` response and caches it,
|
||||
5. injects that cookie (and strips the browser's `Cookie`/`Authorization` and any `?token=`)
|
||||
on every proxied HTTP and WebSocket request,
|
||||
6. refreshes the handshake once if DSH answers `401`, and drops the cached cookie when the
|
||||
sandbox suspends or is deleted.
|
||||
5. injects that cookie (and strips the browser's `Cookie`/`Authorization` and any
|
||||
`?token=`) on every proxied HTTP and WebSocket request,
|
||||
6. refreshes the handshake once if DSH answers `401`, and drops the cached cookie
|
||||
when the sandbox suspends or is deleted.
|
||||
|
||||
This is why the gateway's RBAC includes `pods` `get`/`list` and `pods/log` `get`. It is also
|
||||
the most fragile part of the design; see [Limitations](#limitations).
|
||||
This is why the gateway's RBAC includes `pods` `get`/`list` and `pods/log`
|
||||
`get`. It is also the most fragile part of the design; see
|
||||
[Limitations](#limitations).
|
||||
|
||||
### DSH_TRUSTED_HOSTS
|
||||
|
||||
DSH rejects requests whose `Host` is not allow-listed (`localhost` and `127.0.0.1` are always
|
||||
trusted). Because the gateway forwards the browser's `Host`, set the public hostname in the
|
||||
template:
|
||||
DSH rejects requests whose `Host` is not allow-listed (`localhost` and
|
||||
`127.0.0.1` are always trusted). Because the gateway forwards the browser's
|
||||
`Host`, set the public hostname in the template:
|
||||
|
||||
```yaml
|
||||
env:
|
||||
@@ -301,36 +342,151 @@ env:
|
||||
value: "dsh.example.com" # comma or space separated; empty for port-forward only
|
||||
```
|
||||
|
||||
The same applies to a local `docker run`: the printed token URL uses `DSH_PUBLIC_URL` if set,
|
||||
otherwise `http://localhost:${DSH_PORT}`.
|
||||
The same applies to a local `docker run`: the printed token URL uses
|
||||
`DSH_PUBLIC_URL` if set, otherwise `http://localhost:<DSH_PORT>`.
|
||||
|
||||
### WebSocket
|
||||
|
||||
The gateway is a Starlette ASGI app and proxies WebSocket upgrades with a bidirectional pump, so
|
||||
the DSH web GUI is fully interactive. Requirements:
|
||||
The gateway is a Starlette ASGI app and proxies WebSocket upgrades with a
|
||||
bidirectional pump, so the DSH web GUI is fully interactive. Requirements:
|
||||
|
||||
- the ingress/load balancer in front of the gateway must pass `Upgrade`/`Connection` headers;
|
||||
`60-ingress.yaml` is a starting point (Traefik handles upgrades by default),
|
||||
- idle suspension is disabled for users with in-flight proxied requests or open WebSockets, so
|
||||
a long-running chat stream is not cut off mid-turn,
|
||||
- the upstream upgrade dials the sandbox **pod IP**, forwards no `Host`/`Origin`, and performs
|
||||
the launch-token handshake under that same `pod-IP:3080` authority. DSH fences on `Host`
|
||||
(only its bind address, loopback, or an allow-listed host passes) and binds its auth cookie to
|
||||
the request authority (`payload.authority === authority`), while `websockets` derives `Host`
|
||||
from the URI and cannot be overridden. A browser-host cookie or forwarded origin makes the
|
||||
upgrade fail (`401`/`403`, surfaced as close `1011`), which the client retries forever as
|
||||
`[connection] connection lost, retry #N`,
|
||||
- missing/invalid gateway auth is rejected right after the upgrade with close code `4401`
|
||||
(WebSocket auth has no HTTP `401` path); cross-origin or Origin-less upgrades with the cookie
|
||||
are rejected the same way (CSWSH protection), so clients must send a matching `Origin`;
|
||||
upstream bridging failures close with `1011`.
|
||||
- the ingress/load balancer in front of the gateway must pass `Upgrade`/`Connection`
|
||||
headers; [`manifests/exposure/ingress/60-ingress.yaml`](manifests/exposure/ingress/60-ingress.yaml)
|
||||
is a starting point (Traefik handles upgrades by default),
|
||||
- idle suspension is disabled for users with in-flight proxied requests or open
|
||||
WebSockets, so a long-running chat stream is not cut off mid-turn,
|
||||
- the upstream upgrade dials the sandbox **pod IP**, forwards no `Host`/`Origin`,
|
||||
and performs the launch-token handshake under that same `pod-IP:3080` authority.
|
||||
DSH fences on `Host` (only its bind address, loopback, or an allow-listed host
|
||||
passes) and binds its auth cookie to the request authority
|
||||
(`payload.authority === authority`), while `websockets` derives `Host` from the
|
||||
URI and cannot be overridden. A browser-host cookie or forwarded origin makes
|
||||
the upgrade fail (`401`/`403`, surfaced as close `1011`), which the client
|
||||
retries forever as `[connection] connection lost, retry #N`,
|
||||
- missing/invalid gateway auth is rejected right after the upgrade with close code
|
||||
`4401` (WebSocket auth has no HTTP `401` path); cross-origin or Origin-less
|
||||
upgrades with the cookie are rejected the same way (CSWSH protection), so
|
||||
clients must send a matching `Origin`; upstream bridging failures close with
|
||||
`1011`.
|
||||
|
||||
### Path routing
|
||||
|
||||
Each user's DSH Web is served under `/u/<user>/`. DSH injects `<base href="./">` and uses
|
||||
relative asset/API URLs, so the UI works under a path prefix without a per-user hostname or
|
||||
wildcard DNS. The gateway also rewrites absolute-path `Location` headers into `/u/<user>/…` and
|
||||
redirects `/u/<user>` to the trailing-slash form with `308` so relative URLs resolve correctly.
|
||||
Each user's DSH Web is served under `/u/<user>/`. DSH injects `<base href="./">`
|
||||
and uses relative asset/API URLs, so the UI works under a path prefix without a
|
||||
per-user hostname or wildcard DNS. The gateway also rewrites absolute-path
|
||||
`Location` headers into `/u/<user>/…` and redirects `/u/<user>` to the
|
||||
trailing-slash form with `308` so relative URLs resolve correctly.
|
||||
|
||||
## 9. Expose it: choose exactly one variant
|
||||
|
||||
The base manifests expose nothing outside the cluster (`dsh-gateway` is a
|
||||
ClusterIP Service on port `8080`). Pick **one** variant — they are alternatives,
|
||||
not a stack. Details, commands and the switch procedure are in the
|
||||
[exposure README](manifests/exposure/README.md).
|
||||
|
||||
> Apply the base resources first, then the variant **by path**. Do not use
|
||||
> `kubectl apply -R`: it would apply both variants and create two paths to the
|
||||
> same gateway, which is unsafe once SSO is enabled.
|
||||
|
||||
### Option A — plain Ingress (gateway's own session auth)
|
||||
|
||||
```bash
|
||||
# Replace the dsh.example.com host and TLS Secret (or drop the tls: block for
|
||||
# plain HTTP), then:
|
||||
kubectl apply -f manifests/exposure/ingress/
|
||||
```
|
||||
|
||||
1. Set `DSH_TRUSTED_HOSTS` in `manifests/10-sandboxtemplate.yaml` to the same
|
||||
public host (the gateway forwards the browser's `Host`) and re-apply the
|
||||
template.
|
||||
2. Set `COOKIE_SECURE=true` on the gateway once the Ingress serves HTTPS:
|
||||
`kubectl -n dsh-demo set env deploy/dsh-gateway COOKIE_SECURE=true`.
|
||||
(The base manifest keeps `false` so the plain-HTTP port-forward demo works.)
|
||||
3. Verify: `curl -sI https://<host>/healthz` returns `200`; sign up in a browser
|
||||
at `https://<host>/`.
|
||||
|
||||
Keep the router free of buffering/compression middlewares: they break WebSocket
|
||||
upgrades and streaming.
|
||||
|
||||
### Option B — Gateway API + GitHub SSO (oauth2-proxy)
|
||||
|
||||
`manifests/exposure/gateway-api-sso/` puts a TLS `Gateway` + `HTTPRoute` in
|
||||
front of **oauth2-proxy**, which authenticates the browser against GitHub
|
||||
(restricted to a configured organization) and forwards to the gateway with
|
||||
`X-Forwarded-User` set. The gateway trusts that header only when
|
||||
`AUTH_PROXY_HEADER` is set (SSO is **opt-in**).
|
||||
|
||||
Prerequisites:
|
||||
|
||||
- Gateway API CRDs and a Gateway implementation; the manifests assume
|
||||
`gatewayClassName: traefik` (the CRD version is pinned as
|
||||
`GATEWAY_API_VERSION` in `versions.env`).
|
||||
- A GitHub OAuth App whose homepage/callback match the public FQDN
|
||||
(`https://<host>/` and `https://<host>/oauth2/callback`).
|
||||
- The oauth2-proxy credentials Secret (template:
|
||||
`73-oauth2-proxy-secret.yaml.example`) and a TLS Secret for the Gateway
|
||||
listener (template: `74-tls-secret.yaml.example`).
|
||||
|
||||
Steps:
|
||||
|
||||
```bash
|
||||
# 1. Real Secrets; the committed *.yaml.example files are templates only.
|
||||
kubectl -n dsh-demo create secret generic oauth2-proxy \
|
||||
--from-literal=client-id='<GitHub OAuth App client ID>' \
|
||||
--from-literal=client-secret='<GitHub OAuth App client secret>' \
|
||||
--from-literal=cookie-secret="$(python3 -c 'import secrets; print(secrets.token_urlsafe(32))')"
|
||||
kubectl -n dsh-demo create secret tls dsh-demo-tls --cert=tls.crt --key=tls.key
|
||||
|
||||
# 2. Replace dsh-demo.example.com (70/71/72) and your-github-org (72).
|
||||
|
||||
# 3. Enable SSO on the gateway and trust the public host in the template.
|
||||
kubectl -n dsh-demo set env deploy/dsh-gateway \
|
||||
AUTH_PROXY_HEADER=X-Forwarded-User COOKIE_SECURE=true
|
||||
# ... and set DSH_TRUSTED_HOSTS=dsh-demo.example.com in
|
||||
# manifests/10-sandboxtemplate.yaml, then re-apply the template.
|
||||
|
||||
# 4. Apply the variant.
|
||||
kubectl apply -f manifests/exposure/gateway-api-sso/
|
||||
```
|
||||
|
||||
Verification:
|
||||
|
||||
```bash
|
||||
kubectl get gateway dsh-gateway -n dsh-demo -o yaml # Programmed=True
|
||||
kubectl get httproute -n dsh-demo -o wide # backend oauth2-proxy:4180, Accepted
|
||||
kubectl get deploy oauth2-proxy -n dsh-demo # 1/1 Ready
|
||||
kubectl get endpoints oauth2-proxy -n dsh-demo # backend present
|
||||
kubectl -n dsh-demo logs deploy/oauth2-proxy --tail=50 # no config errors
|
||||
```
|
||||
|
||||
Browser check: open `https://<host>/` — you are redirected to GitHub, and after
|
||||
signing in as a member of the allowed organization the gateway auto-provisions
|
||||
your sandbox (the first page can show `Provisioning`) and connects you. A
|
||||
non-member is denied before reaching the gateway. GitHub logins are lowercase
|
||||
DNS-1123 labels, so they map 1:1 onto the gateway's `user` validation. The SSO
|
||||
landing page also offers **Recreate sandbox** (`POST /recreate`), which deletes
|
||||
the claim — wiping the workspace PVC — and provisions a fresh one.
|
||||
|
||||
Sign-out: the gateway `/logout` clears `dsh_gw_session`; oauth2-proxy's session
|
||||
can be cleared at `/oauth2/sign_out`.
|
||||
|
||||
> **SSO trust boundary — read this before enabling it.** After SSO is on, the
|
||||
> `dsh-gateway` Service must be reachable **only** through oauth2-proxy (it is
|
||||
> ClusterIP). Anyone who can reach `dsh-gateway:8080` directly can send
|
||||
> `X-Forwarded-User: <victim>` and be treated as that user. Never apply the
|
||||
> Ingress variant (or any other route to the Service) while SSO is enabled, and
|
||||
> do not expose the gateway Service with `kubectl port-forward` on a shared
|
||||
> machine while `AUTH_PROXY_HEADER` is set. To turn SSO off, remove the variable
|
||||
> (`kubectl -n dsh-demo set env deploy/dsh-gateway AUTH_PROXY_HEADER-`) and
|
||||
> delete the SSO variant; the gateway falls back to bearer-token/session auth.
|
||||
|
||||
### Switching variants
|
||||
|
||||
Delete the variant you had before applying the other one
|
||||
(`kubectl delete -f manifests/exposure/ingress/`), revert
|
||||
`AUTH_PROXY_HEADER`/`COOKIE_SECURE`, and verify with
|
||||
`kubectl -n dsh-demo get ingress,gateway,httproute` that exactly one path to the
|
||||
gateway remains.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
@@ -346,36 +502,65 @@ redirects `/u/<user>` to the trailing-slash form with `308` so relative URLs res
|
||||
| Console spams `[connection] connection lost, retry #N` | The upstream WS upgrade is being rejected, surfaced as close `1011`. The gateway dials the sandbox **pod IP**, forwards no `Host`/`Origin`, and handshakes under the same `pod-IP:3080` authority DSH binds its cookie to; an older gateway image, or a sandbox with an empty `status.podIPs`, hits this loop |
|
||||
| `manifest.webmanifest` returns `401` in the console | Browsers fetch the Web App Manifest without credentials, so the gateway's session cookie is absent and the request fails closed. The UI is unaffected; only the PWA install prompt is lost |
|
||||
| Signup is slow / `202` then `Waking` | Warm pool exhausted (`kubectl get sandboxes -l agents.x-k8s.io/warm-pool-sandbox`); scale the pool |
|
||||
| `403` from the Kubernetes API in gateway logs | The gateway ServiceAccount is missing claim/sandbox/pod-log verbs (see `50-gateway.yaml`) |
|
||||
| Step 3: `pod "$SB" not found` after deploying the gateway | The claim was suspended by the idle sweeper. Claims created by the gateway carry `aaas.example.com/managed-by=dsh-gateway`; hand-applied claims (steps 1–7) do not and are never swept. Upgrade the gateway to a build that honors the marker, or resume the sandbox: `kubectl -n dsh-demo patch sandbox "$SB" --type merge -p '{"spec":{"operatingMode":"Running"}}'` |
|
||||
| Resume crash-loops: `credentials-local: … is readable beyond its owner (mode 660)` | The pod template lacks `fsGroupChangePolicy: OnRootMismatch`, so the kubelet's default fsGroup policy rewrote `.credentials.yaml` to `0660`; DSH requires `0600` and refuses to start. New installs get the policy from the template. For an existing sandbox: suspend it, run a throwaway pod on `dsh-home-$SB` to `chmod 600 /home/node/.dsh/.credentials.yaml`, patch the sandbox's `podTemplate` with the policy, then resume. |
|
||||
| `403` from the Kubernetes API in gateway logs | The gateway ServiceAccount is missing claim/sandbox/pod-log verbs (see `manifests/50-gateway.yaml`) |
|
||||
| Gateway pod `ErrImagePull` / `ImagePullBackOff` (especially on kind) | The manifest uses `imagePullPolicy: IfNotPresent` with the release identity. Build with `make gateway`, retag to exactly the manifest's image, `kind load docker-image <that-ref>`, or set the Deployment image to your local tag / pinned digest. On a real cluster, make sure the GHCR package is readable (public or a pull secret) |
|
||||
| Step 3: `pod "$SB" not found` after deploying the gateway | The claim was suspended by the idle sweeper. Claims created by the gateway carry `aaas.example.com/managed-by=dsh-gateway`; hand-applied claims (steps 1–7) do not and are never swept. Resume the sandbox: `kubectl -n dsh-demo patch sandbox "$SB" --type merge -p '{"spec":{"operatingMode":"Running"}}'` |
|
||||
| Resume crash-loops: `credentials-local: … is readable beyond its owner (mode 660)` | The pod template lacks `fsGroupChangePolicy: OnRootMismatch`, so the kubelet's default fsGroup policy rewrote `.credentials.yaml` to `0660`; DSH requires `0600` and refuses to start. New installs get the policy from the template. For an existing sandbox: suspend it, run a throwaway pod on `dsh-home-$SB` to `chmod 600 /home/node/.dsh/.credentials.yaml`, patch the sandbox's `podTemplate` with the policy, then resume |
|
||||
| SSO: GitHub login succeeds but the gateway asks for a token / returns `401` to `/recreate` | `AUTH_PROXY_HEADER` is not set on `deploy/dsh-gateway` (SSO is opt-in). Set `AUTH_PROXY_HEADER=X-Forwarded-User` and confirm the request actually reaches the gateway through oauth2-proxy |
|
||||
| SSO: gateway treats a caller as another user / bad `X-Forwarded-User` accepted | Trust boundary violation: something other than oauth2-proxy can reach `dsh-gateway:8080`. Remove the second path (Ingress/Gateway/port-forward), keep the Service ClusterIP, and only re-enable SSO once the proxy is the sole entry point. Never put a trusted identity header in front of a service that is directly reachable |
|
||||
| SSO: oauth2-proxy `cookie_secret must be 16, 24, or 32 bytes ... but is 44 bytes` | The cookie secret was generated as standard base64. Use URL-safe unpadded base64 (see `73-oauth2-proxy-secret.yaml.example`) |
|
||||
| SSO: redirect loop (`ERR_TOO_MANY_REDIRECTS`) | oauth2-proxy `--redirect-url` / callback does not match the public FQDN, or TLS is not terminating at the Gateway listener; fix the OAuth App URLs and the manifest placeholders |
|
||||
| SSO: DSH rejects the public host behind the proxy | Add the public hostname to `DSH_TRUSTED_HOSTS` in `manifests/10-sandboxtemplate.yaml`; the gateway forwards the browser's `Host` |
|
||||
|
||||
## Limitations
|
||||
|
||||
Deliberately **not** production code:
|
||||
|
||||
- **Single replica.** Idle clocks, in-flight tracking, and the handshake cache are in memory.
|
||||
Running two gateway replicas would double-suspend or lose wake tracking; scale up only after
|
||||
moving that state to a shared store.
|
||||
- **Log scraping.** The launch token is recovered from pod logs. Log rotation, a restarted pod,
|
||||
or a changed DSH log format can break the handshake until the next `401` refresh.
|
||||
- **Shared provider key.** `DEEPSEEK_API_KEY` is shared by all tenants via the template. Real
|
||||
multi-tenancy needs per-tenant credentials or an egress proxy.
|
||||
- **No TLS in the gateway.** Terminate TLS at an Ingress/reverse proxy and set
|
||||
- **Single replica.** Idle clocks, in-flight tracking, and the handshake cache
|
||||
are in memory. Running two gateway replicas would double-suspend or lose wake
|
||||
tracking; scale up only after moving that state to a shared store.
|
||||
- **Log scraping.** The launch token is recovered from pod logs. Log rotation, a
|
||||
restarted pod, or a changed DSH log format can break the handshake until the
|
||||
next `401` refresh.
|
||||
- **Shared provider key.** `DEEPSEEK_API_KEY` is shared by all tenants via the
|
||||
template. Real multi-tenancy needs per-tenant credentials or an egress proxy.
|
||||
- **No TLS in the gateway.** Terminate TLS at an Ingress or Gateway and set
|
||||
`COOKIE_SECURE=true`.
|
||||
- **In-memory user directory.** Users are claims + annotations; there is no database, password
|
||||
reset, or admin API. Tokens are stored only as SHA-256 hashes in claim annotations.
|
||||
- **Bearer tokens in URLs.** Avoid putting the signup token in a URL/log; the browser flow uses
|
||||
the `HttpOnly` cookie instead.
|
||||
- **SSO trust is hop-based.** `AUTH_PROXY_HEADER` trusts whoever sets the header;
|
||||
the only guarantee is that the gateway Service is not directly reachable.
|
||||
There is no signature, mTLS, or NetworkPolicy enforcing that in this example.
|
||||
- **OAuth org is the only authorization.** Any GitHub user in the configured
|
||||
organization gets an auto-provisioned sandbox; there is no per-user allow-list,
|
||||
quota, or admin API.
|
||||
- **In-memory user directory.** Users are claims + annotations; there is no
|
||||
database, password reset, or admin API. Tokens are stored only as SHA-256
|
||||
hashes in claim annotations.
|
||||
- **Bearer tokens in URLs.** Avoid putting the signup token in a URL/log; the
|
||||
browser flow uses the `HttpOnly` cookie instead.
|
||||
|
||||
## Cost model
|
||||
|
||||
The same economics as the upstream pattern: suspend-when-idle plus warm pools turn an
|
||||
always-on instance into a sub-second cold-start pause per return-after-absence, at a fraction
|
||||
of the cost.
|
||||
The same economics as the upstream pattern: suspend-when-idle plus warm pools turn
|
||||
an always-on instance into a sub-second cold-start pause per return-after-absence,
|
||||
at a fraction of the cost.
|
||||
|
||||
## Cleanup
|
||||
|
||||
```bash
|
||||
# Delete an exposure variant first if you applied one (optional; namespace
|
||||
# deletion below removes everything):
|
||||
kubectl delete -f manifests/exposure/ingress/ # or gateway-api-sso/
|
||||
# Then the whole example:
|
||||
kubectl delete namespace dsh-demo
|
||||
```
|
||||
|
||||
If you created the oauth2-proxy/TLS Secrets or the GitHub OAuth App for step 9,
|
||||
delete the Secrets with the namespace and revoke the OAuth App in GitHub.
|
||||
|
||||
## More
|
||||
|
||||
- [Gateway README](gateway/README.md) — API, configuration, auth/SSO, failure
|
||||
semantics, limitations.
|
||||
- [Architecture](docs/architecture.md) — container/sequence diagrams, ADRs,
|
||||
security and RBAC, failure modes, evolution path.
|
||||
- [Exposure README](manifests/exposure/README.md) — the two variants in detail.
|
||||
|
||||
337
examples/dsh-agents-as-a-service/docs/architecture.md
Normal file
337
examples/dsh-agents-as-a-service/docs/architecture.md
Normal file
@@ -0,0 +1,337 @@
|
||||
# Architecture
|
||||
|
||||
This document describes the browser-usable **agents-as-a-service** platform in this example
|
||||
([README](../README.md), [manifests](../manifests/), [gateway](../gateway/)): how the gateway,
|
||||
Kubernetes, and DSH Web fit together, and why the design looks the way it does.
|
||||
|
||||
Scope: one Kubernetes namespace (`dsh-demo`), one shared DSH Web image, one gateway deployment,
|
||||
and one DSH Web pod + PVC per user. The cluster is assumed to run
|
||||
[agent-sandbox](https://github.com/kubernetes-sigs/agent-sandbox) v1.0.7 with the extensions
|
||||
(`SandboxTemplate`, `SandboxWarmPool`, `SandboxClaim`).
|
||||
|
||||
## Container view
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
browser["Browser"]
|
||||
|
||||
subgraph cluster["Kubernetes cluster - namespace dsh-demo"]
|
||||
gw["Gateway Deployment<br/>Starlette ASGI on uvicorn :8080<br/>landing page + auth + proxy + sweeper"]
|
||||
api["Kubernetes API Server"]
|
||||
ctrl["agent-sandbox controller"]
|
||||
pool["SandboxWarmPool dsh-pool<br/>2 pre-warmed spares"]
|
||||
claim["SandboxClaim dsh-USER<br/>one per user"]
|
||||
sb["Sandbox dsh-pool-xxxx<br/>spec.operatingMode"]
|
||||
svc["Headless Service<br/>status.serviceFQDN"]
|
||||
pod["DSH Web pod<br/>node process :3080"]
|
||||
pvc[("PVC dsh-home-SANDBOX<br/>/home/node/.dsh")]
|
||||
end
|
||||
|
||||
browser -->|"HTTP + WebSocket<br/>/u/USER/**"| gw
|
||||
gw -->|"claims, sandboxes, pods, pod logs"| api
|
||||
api --> ctrl
|
||||
ctrl -->|"adopt / suspend / resume"| claim
|
||||
claim --> sb
|
||||
pool -.->|"replenishes"| sb
|
||||
sb --> svc
|
||||
svc --> pod
|
||||
pod --> pvc
|
||||
```
|
||||
|
||||
Responsibilities:
|
||||
|
||||
| Component | Responsibility |
|
||||
| --- | --- |
|
||||
| **Browser** | Holds only the gateway session cookie (`dsh_gw_session`). Talks exclusively to the gateway. |
|
||||
| **Gateway** | Signup/login, session auth, `SandboxClaim` lifecycle, wake-on-connect, idle sweeper, HTTP/SSE/WebSocket reverse proxy, DSH launch-token handshake. |
|
||||
| **Kubernetes API** | Source of truth for claims, sandboxes, and pods; also the log source for the DSH launch token. |
|
||||
| **agent-sandbox controller** | Adopts warm spares on claim, flips `operatingMode` (Running/Suspended), garbage-collects sandbox + PVC on claim delete. |
|
||||
| **DSH Web pod** | One `dsh web` process per user, state on the per-user PVC, listening on `3080`. |
|
||||
| **Warm pool** | Keeps `N` Ready sandboxes so signup is a label flip instead of a cold start. |
|
||||
|
||||
## Exposure variants
|
||||
|
||||
The base manifests deploy the gateway as a ClusterIP Service (`dsh-gateway:8080`) and expose
|
||||
nothing outside the cluster. Two **mutually exclusive** variants live under
|
||||
[`manifests/exposure/`](../manifests/exposure/):
|
||||
|
||||
| Variant | Entry point | Authentication | Notes |
|
||||
| --- | --- | --- | --- |
|
||||
| [Ingress](../manifests/exposure/ingress/60-ingress.yaml) | Traefik `Ingress` + optional TLS | the gateway's own signup/session cookie | simplest; no external IdP; set `COOKIE_SECURE=true` with HTTPS |
|
||||
| [Gateway API + SSO](../manifests/exposure/gateway-api-sso/70-gateway.yaml) | Gateway API `Gateway`/`HTTPRoute` terminating TLS | GitHub OAuth via [oauth2-proxy](../manifests/exposure/gateway-api-sso/72-oauth2-proxy.yaml), identity forwarded as `X-Forwarded-User` | requires `AUTH_PROXY_HEADER` on the gateway; the trust boundary applies |
|
||||
|
||||
They are alternatives, not a stack: applying both creates a second path to the same gateway
|
||||
and defeats SSO. When SSO is enabled, the Service must stay ClusterIP and be reachable only
|
||||
through oauth2-proxy — otherwise any client can send `X-Forwarded-User` itself and be treated
|
||||
as that user. The gateway ignores the header unless `AUTH_PROXY_HEADER` is set. See the
|
||||
[exposure README](../manifests/exposure/README.md) and the
|
||||
[gateway README](../gateway/README.md#sso-opt-in).
|
||||
|
||||
## Flows
|
||||
|
||||
### Signup (browser)
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
actor B as Browser
|
||||
participant G as Gateway
|
||||
participant K as Kubernetes API
|
||||
participant C as agent-sandbox controller
|
||||
|
||||
B->>G: POST /users (form: user=bob, Origin)
|
||||
G->>G: validate user name and body size
|
||||
G->>K: create SandboxClaim dsh-bob (warmPoolRef)
|
||||
K->>C: reconcile claim
|
||||
C-->>K: adopt spare, Ready, status.sandbox.name = dsh-pool-xxxx
|
||||
G->>K: read claim (status) + sandbox + pod
|
||||
G-->>B: 303 Location /u/bob/ + Set-Cookie dsh_gw_session
|
||||
Note over G: JSON clients instead get 201 (Ready),<br/>202 (accepted, still waking), or 409 (conflict)
|
||||
```
|
||||
|
||||
Form signup waits up to `WAKE_TIMEOUT` for the claim to become `Ready`, then redirects; the
|
||||
first proxied request also waits for `Ready` and performs the DSH handshake (below). A `409`
|
||||
means the user already has a claim, in which case `POST /login` (bearer token) or the existing
|
||||
cookie is the right path.
|
||||
|
||||
### First request / proxy (with handshake)
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
actor B as Browser
|
||||
participant G as Gateway
|
||||
participant K as Kubernetes API
|
||||
participant D as DSH Web pod
|
||||
|
||||
B->>G: GET /u/bob/ (Cookie dsh_gw_session)
|
||||
G->>G: authenticate cookie, user = bob
|
||||
G->>K: resolve claim -> sandbox -> pod
|
||||
alt sandbox Suspended
|
||||
G->>K: patch operatingMode=Running (wake)
|
||||
loop until Ready or WAKE_TIMEOUT
|
||||
G->>K: read sandbox Ready condition
|
||||
end
|
||||
end
|
||||
G->>K: read pod logs (tail)
|
||||
K-->>G: "dsh web: ...?token=..." line
|
||||
G->>D: GET /?token=... (browser Host, X-Forwarded-*)
|
||||
D-->>G: 303 + Set-Cookie dsh-auth-*
|
||||
G->>G: cache cookie for (user, pod UID, Host)
|
||||
G->>G: strip Cookie/Authorization/hop-by-hop headers
|
||||
G->>D: replay request with dsh-auth cookie
|
||||
D-->>G: 200 HTML / SSE stream
|
||||
G-->>B: 200 (upstream Set-Cookie dropped, Location rewritten)
|
||||
```
|
||||
|
||||
Handshake cache behavior: one entry per `(user, pod UID, Host)`; refreshed once when DSH answers
|
||||
`401`; dropped when the sandbox is suspended or deleted. The launch token and the DSH auth
|
||||
cookie never reach the browser.
|
||||
|
||||
### Wake-on-connect and idle suspend
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant S as Sweeper - lifespan task
|
||||
participant G as Gateway
|
||||
participant K as Kubernetes API
|
||||
actor B as Browser
|
||||
|
||||
S->>G: no proxied request for IDLE_TIMEOUT
|
||||
G->>K: patch sandbox operatingMode=Suspended
|
||||
K-->>G: pod deleted, PVC + Service retained
|
||||
Note over S,G: in-flight requests/WebSockets are tracked,<br/>so a busy turn is never suspended
|
||||
B->>G: GET /u/bob/ after a while
|
||||
G->>K: patch operatingMode=Running
|
||||
G->>K: wait for Ready (WAKE_TIMEOUT)
|
||||
G-->>B: request held, then served
|
||||
```
|
||||
|
||||
Suspend deletes only the pod. The PVC keeps conversations, sessions, and config, so resume
|
||||
reattaches exactly the same state.
|
||||
|
||||
**Ownership boundary.** The gateway shares its namespace with claims created outside it — the
|
||||
example's resource-layer walkthrough applies `dsh-alice` by hand, and every claim is named
|
||||
`dsh-<user>`. Claims the gateway creates carry `aaas.example.com/managed-by=dsh-gateway`; the
|
||||
sweeper skips everything else, so it can never suspend a sandbox an operator is using. Without
|
||||
that marker the sweeper would suspend a hand-applied claim 60s after first sighting and break
|
||||
`kubectl logs`/`port-forward` against its pod.
|
||||
|
||||
**Resume and file ownership.** The pod template sets `fsGroupChangePolicy: OnRootMismatch` on
|
||||
purpose. DSH's credentials plugin refuses to boot when `.credentials.yaml` has any group/other
|
||||
permission bit (it creates the file `0600`). The kubelet's default fsGroup policy (`Always`)
|
||||
recursively ORs `0660` into every file on each pod start, so the first resume of a sandbox would
|
||||
crash-loop with `mode 660`. `OnRootMismatch` only rewrites ownership/permissions when the volume
|
||||
root does not already match the fsGroup, which it does after the first mount — so suspend/resume
|
||||
keeps working. Sandboxes snapshot their pod template, so an updated `SandboxTemplate` does not
|
||||
retroactively change existing ones.
|
||||
|
||||
### WebSocket
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
actor B as Browser
|
||||
participant G as Gateway
|
||||
participant D as DSH Web pod
|
||||
|
||||
B->>G: GET /u/bob/... Upgrade: websocket (Cookie dsh_gw_session)
|
||||
G->>G: authenticate + mark user in-flight
|
||||
alt cookie missing/invalid
|
||||
G-->>B: 101 upgrade, then close 4401
|
||||
else authenticated
|
||||
G->>D: websockets.connect(path, Host, Cookie dsh-auth)
|
||||
alt upstream handshake fails
|
||||
G-->>B: 101 upgrade, then close 1011
|
||||
else bridged
|
||||
G-->>B: 101 Switching Protocols
|
||||
loop frames
|
||||
B->>D: frames proxied through G
|
||||
D->>B: frames proxied through G
|
||||
end
|
||||
G->>G: clear in-flight on close
|
||||
end
|
||||
end
|
||||
```
|
||||
|
||||
## Decisions and trade-offs
|
||||
|
||||
### ADR-1: Starlette ASGI instead of a sync Flask proxy
|
||||
|
||||
- **Decision:** implement the gateway as a Starlette app served by `uvicorn
|
||||
gateway:create_app --factory`.
|
||||
- **Why:** the browser needs WebSocket upgrades, streaming SSE responses, and concurrent
|
||||
long-poll wake requests. A requests-based Flask proxy cannot upgrade connections and blocks
|
||||
workers while a sandbox wakes.
|
||||
- **Cost:** async code paths and explicit bridging for the sync `kubernetes` client (offloaded
|
||||
to threads). Accepted: one small service, one framework.
|
||||
|
||||
### ADR-2: Path-based routing (`/u/{user}/`) instead of per-user subdomains
|
||||
|
||||
- **Decision:** serve each user under `https://gateway/u/{user}/`.
|
||||
- **Why:** one hostname, one TLS certificate, no wildcard DNS or per-user Ingress objects. DSH
|
||||
uses relative asset/API URLs and injects `<base href="./">`, so a path prefix works; the
|
||||
gateway rewrites absolute-path `Location` headers and redirects `/u/{user}` to
|
||||
`/u/{user}/`.
|
||||
- **Cost:** any upstream code that emits root-absolute URLs would need rewriting; cookie
|
||||
domains are shared (mitigated by stripping DSH `Set-Cookie`). If DSH ever changes its base
|
||||
handling, per-user subdomains are the fallback.
|
||||
|
||||
### ADR-3: Server-side launch-token handshake instead of exposing the token
|
||||
|
||||
- **Decision:** the gateway reads DSH's launch token from pod logs, performs the token
|
||||
handshake itself, caches DSH's auth cookie, and authenticates the browser with its own
|
||||
`HttpOnly` cookie.
|
||||
- **Why:** DSH has no token API and returns `401` on `/` without the launch-token cookie.
|
||||
Handing the token to the browser would put it in URLs, history, and logs, and would let users
|
||||
bypass gateway auth.
|
||||
- **Cost:** log reading permission and a brittle parser; see [Upstream feature
|
||||
request](#upstream-feature-request-remove-log-scraping). This is the least durable part of the
|
||||
design and the first thing to replace.
|
||||
|
||||
### ADR-4: In-memory state, single replica
|
||||
|
||||
- **Decision:** idle clocks, handshake cache, and in-flight tracking live in gateway memory;
|
||||
the Deployment runs one replica.
|
||||
- **Why:** the simplest thing that works, with no extra datastore for a demo/reference
|
||||
platform.
|
||||
- **Cost:** a restart forgets idle timers (sandboxes stay Running until their next request) and
|
||||
the handshake cache; two replicas would double-suspend and race. Scaling out requires moving
|
||||
this state to a shared store (Redis/Postgres) or leader election. Documented instead of
|
||||
hidden.
|
||||
|
||||
### ADR-5: Claims as the user record
|
||||
|
||||
- **Decision:** a user is a `SandboxClaim` named `dsh-<user>`; the bearer-token SHA-256 lives in
|
||||
a claim annotation, and the gateway marks its own claims with
|
||||
`aaas.example.com/managed-by=dsh-gateway`.
|
||||
- **Why:** no database, no separate account service; deleting the claim is account deletion and
|
||||
cascades everything. The ownership annotation keeps a namespaced claim list safe to share with
|
||||
objects created by hand or by other controllers.
|
||||
- **Cost:** no password reset, no admin API, no audit log of its own; Kubernetes API load grows
|
||||
with users. Claims created before the ownership marker existed must be annotated
|
||||
(`kubectl annotate sandboxclaim ...`) to keep being swept. Fine for a reference pattern, not
|
||||
for a product.
|
||||
|
||||
### ADR-6: Gateway cookie session instead of bearer tokens in the browser
|
||||
|
||||
- **Decision:** browsers get `dsh_gw_session=<user>:<token>`, `HttpOnly`, `SameSite=Lax`
|
||||
(`Secure` when enabled); API clients may still use `Authorization: Bearer`.
|
||||
- **Why:** tokens never appear in URLs or JS-readable storage, and `SameSite=Lax` plus a strict
|
||||
`Origin`/`Referer` check on cookie-authenticated mutations blocks basic CSRF.
|
||||
- **Cost:** cookie-based auth needs TLS and correct `COOKIE_SECURE`; there is no signed-cookie
|
||||
layer, so the gateway trusts the token hash comparison for validation.
|
||||
|
||||
## Security and RBAC
|
||||
|
||||
The gateway runs with a namespace-scoped ServiceAccount (`manifests/50-gateway.yaml`):
|
||||
|
||||
| API group | Resource | Verbs | Why |
|
||||
| --- | --- | --- | --- |
|
||||
| `extensions.agents.x-k8s.io` | `sandboxclaims` | `get`, `list`, `create`, `delete` | signup, state, account deletion |
|
||||
| `agents.x-k8s.io` | `sandboxes` | `get`, `list`, `patch` | resolve DNS/pod, wake/suspend |
|
||||
| `""` (core) | `pods` | `get`, `list` | resolve the pod behind a sandbox |
|
||||
| `""` (core) | `pods/log` | `get` | launch-token handshake |
|
||||
|
||||
Notes and limits:
|
||||
|
||||
- **Pod logs are sensitive.** They contain the DSH launch token and possibly model output. The
|
||||
permission is scoped to one namespace, but anyone who can read gateway memory or logs can
|
||||
impersonate users. Prefer the upstream token API when available.
|
||||
- **No cluster-wide access, no secret reads.** The gateway does not read `dsh-provider-keys`.
|
||||
- **Browser-facing hardening:** `HttpOnly`/`SameSite=Lax`/optional `Secure` cookies,
|
||||
timing-safe SHA-256 token comparison, strict same-origin check on cookie-auth mutations,
|
||||
request-body limit (`MAX_BODY_MB`), hop-by-hop header stripping, upstream `Set-Cookie`
|
||||
dropped, DSH `Location` headers rewritten under `/u/<user>/`.
|
||||
- **Host allow-list:** DSH rejects unknown `Host` headers. The gateway forwards the browser's
|
||||
`Host`, so every public hostname must be in `DSH_TRUSTED_HOSTS`; `localhost`/`127.0.0.1` are
|
||||
always trusted.
|
||||
- **TLS is external.** The gateway serves plain HTTP; terminate TLS at the Ingress and set
|
||||
`COOKIE_SECURE=true`. Without TLS, session cookies are replayable on the network.
|
||||
- **Tenant isolation** is pod + PVC separation; all tenants share one namespace, one DSH binary,
|
||||
and one `DEEPSEEK_API_KEY`. Add NetworkPolicies and per-tenant credentials before calling it
|
||||
multi-tenant-grade.
|
||||
- **No gateway rate limiting.** Signup can create claims until the cluster or warm pool is
|
||||
exhausted; put limits at the ingress for real deployments.
|
||||
|
||||
## Failure modes
|
||||
|
||||
| Failure | Symptom | Behavior / mitigation |
|
||||
| --- | --- | --- |
|
||||
| DSH log line changed or rotated away | Proxy returns `503` (missing launch token) | Fix the parser or restart the pod so the token line is re-emitted |
|
||||
| Probe on `/` | Pod `CrashLoopBackOff`, `401` in probe logs | Use `/favicon.svg` for readiness/liveness (shipped templates do) |
|
||||
| Warm pool exhausted | JSON signup returns `202` (state `Waking`); form signup is slow | Increase `replicas` in the warm pool |
|
||||
| Gateway restart | Idle sandboxes stop suspending; handshake cache lost | Rebuilt lazily on next request; consider a shared store for production |
|
||||
| DSH auth cookie expired mid-session | Proxy `401` once, then recovers | Gateway refreshes the handshake once |
|
||||
| Upstream pod name changed after resume | Cached cookie keyed by old pod UID is useless | Cache is invalidated on suspend; next request re-handshakes |
|
||||
| Two gateway replicas | Double suspend/wake races | Keep `replicas: 1` until state is externalized |
|
||||
|
||||
## Upstream feature request: remove log scraping
|
||||
|
||||
DSH should expose the launch token as a first-class, machine-readable interface so platforms do
|
||||
not need `pods/log` access. Proposed shapes (any one suffices):
|
||||
|
||||
1. **Token file:** `dsh web --token-file /run/dsh/token` writes the current launch token (mode
|
||||
`0600`) and rewrites it on rotation.
|
||||
2. **Environment input:** accept `DSH_LAUNCH_TOKEN` (platform-generated) so the token never
|
||||
has to be discovered.
|
||||
3. **Loopback/trusted-host endpoint:** `GET /api/auth/launch-token` that only answers on the
|
||||
pod's loopback or a trusted host and returns the token plus expiry.
|
||||
4. **Token in a well-known file** under `DSH_HOME` with the same semantics as (1).
|
||||
|
||||
Why this matters: log scraping is brittle (rotation, restart, format changes), requires a broad
|
||||
`pods/log` permission that also exposes application output, and can leak tokens into log
|
||||
pipelines. Until an upstream API exists, the gateway's parser, `DSH_LOG_TAIL_LINES`, and the
|
||||
single-retry refresh are the supported workaround.
|
||||
|
||||
## Evolution path
|
||||
|
||||
1. Replace log scraping with the upstream token API (above), deleting the `pods/log` permission.
|
||||
2. Move idle clocks, in-flight tracking, and handshake cache to a shared store; then scale the
|
||||
gateway horizontally (sticky routing not required if state is shared).
|
||||
3. Add per-tenant API keys or an egress proxy so a single key is not shared by all sandboxes.
|
||||
4. Add NetworkPolicies (gateway → sandboxes only), TLS everywhere, rate limits, and audit
|
||||
logging at the ingress.
|
||||
5. If DSH grows absolute URLs or per-user cookies that cannot be rewritten, switch to
|
||||
per-user subdomains with wildcard DNS/TLS.
|
||||
11
examples/dsh-agents-as-a-service/gateway/.dockerignore
Normal file
11
examples/dsh-agents-as-a-service/gateway/.dockerignore
Normal file
@@ -0,0 +1,11 @@
|
||||
# Runtime image only needs the app and its pinned runtime requirements.
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
.pytest_cache/
|
||||
.ruff_cache/
|
||||
.venv/
|
||||
test_gateway.py
|
||||
requirements-dev.txt
|
||||
pyproject.toml
|
||||
README.md
|
||||
coverage.xml
|
||||
@@ -4,7 +4,7 @@ A single-file, browser-usable control + data plane for
|
||||
[DeepSeek Harness Web (DSH)](https://github.com/deepseek-ai/deepseek-harness)
|
||||
sandboxes running behind the `agent-sandbox` CRDs.
|
||||
|
||||
It does three things:
|
||||
It does four things:
|
||||
|
||||
1. **Users** — creates a `SandboxClaim` from a warm pool, mints a bearer token
|
||||
(only its SHA-256 is stored as a claim annotation), reports derived state,
|
||||
@@ -15,10 +15,15 @@ It does three things:
|
||||
3. **Reverse proxy** — proxies HTTP (including SSE) and WebSocket traffic to
|
||||
the sandbox's DSH Web server on `:3080`, including DSH's own cookie
|
||||
handshake.
|
||||
4. **Optional SSO** — when `AUTH_PROXY_HEADER` is set, trusts the identity
|
||||
header an authenticating reverse proxy (for example oauth2-proxy) forwards,
|
||||
auto-provisions that user, and exposes `POST /recreate` for a clean restart.
|
||||
|
||||
DSH is pinned to `dsh-v0.2.1-rc.2` in the example manifests. It binds `:3080`,
|
||||
serves `/`, gates `/api` behind a `dsh-auth-*` cookie and requires a WebSocket
|
||||
at `/api/remote.mux`.
|
||||
The DSH version pin lives in [`versions.env`](../../../versions.env)
|
||||
(`DSH_VERSION` plus the optional repository `RELEASE_REVISION`); the example
|
||||
manifests and published images use the release identity built from it (currently
|
||||
`dsh-v0.2.1-rc.2-r2`). DSH binds `:3080`, serves `/`, gates `/api` behind a
|
||||
`dsh-auth-*` cookie and requires a WebSocket at `/api/remote.mux`.
|
||||
|
||||
## Flow
|
||||
|
||||
@@ -58,10 +63,11 @@ token. A failed handshake returns `503` with `Retry-After: 10`.
|
||||
| Method | Path | Auth | Purpose |
|
||||
| --- | --- | --- | --- |
|
||||
| `GET` | `/healthz` | none | Liveness/readiness. Never touches the cluster. |
|
||||
| `GET` | `/` | none | Landing page with signup + token login forms. |
|
||||
| `GET` | `/` | none | Landing page with signup + token login forms. With SSO it auto-provisions the proxy-identified user and redirects to their workspace. |
|
||||
| `POST` | `/users` | none | Signup. JSON body `{"user": "<dns-1123 label>"}` or form field `user`. JSON returns `201` when Ready, `202` when accepted but still Waking; form returns `303` to `/u/<user>/`. Both set the session cookie. `409` if the user exists, `400` if invalid, `413` if the body is too large. |
|
||||
| `POST` | `/login` | none | Form (`user`, `token`) returns `303` to `/u/<user>/`; JSON returns `200`. Sets the session cookie. `401` on a bad token, `404` for an unknown user. |
|
||||
| `POST` | `/logout` | none | Clears the session cookie and `303`s to `/`. |
|
||||
| `POST` | `/recreate` | trusted proxy header | **SSO only.** Deletes the authenticated user's claim (cascading sandbox + PVC, so workspace data is wiped), waits out finalizers, provisions a fresh claim and `303`s to `/u/<user>/`. `401` without a valid identity header; cookie-authenticated calls must be same-origin. |
|
||||
| `GET` | `/users/{user}` | bearer or cookie | Derived state: `Provisioning`, `Waking`, `Ready`, `Suspending`, `Suspended`. |
|
||||
| `DELETE` | `/users/{user}` | bearer or cookie | Delete the claim (sandbox + PVC cascade). Cookie-auth mutations require a same-origin `Origin`/`Referer`. |
|
||||
| `GET` | `/u/{user}` | — | `308` redirect to `/u/{user}/`. |
|
||||
@@ -86,6 +92,37 @@ token. A failed handshake returns `503` with `Retry-After: 10`.
|
||||
absolute paths are rewritten under `/u/<user>/`, and any `token` query
|
||||
parameter is stripped.
|
||||
|
||||
### SSO (opt-in)
|
||||
|
||||
By default the gateway is the only authenticator: users sign up with a token or
|
||||
the session cookie, and the gateway ignores all identity headers. Set
|
||||
`AUTH_PROXY_HEADER` to enable SSO behind an authenticating reverse proxy such as
|
||||
oauth2-proxy:
|
||||
|
||||
```bash
|
||||
kubectl -n dsh-demo set env deploy/dsh-gateway AUTH_PROXY_HEADER=X-Forwarded-User
|
||||
```
|
||||
|
||||
When set, the gateway treats a request carrying that header as authenticated as
|
||||
that user, provided the value is a valid DNS-1123 label; the header is never
|
||||
trusted when the variable is unset. The landing page then auto-provisions the
|
||||
user (creates their claim idempotently, mints a rotated session token) and
|
||||
redirects to `/u/<user>/`, and `POST /recreate` becomes available for a clean
|
||||
restart. The base manifests leave it unset on purpose; the
|
||||
[SSO exposure variant](../manifests/exposure/gateway-api-sso/) documents how to
|
||||
enable it on the Deployment (`kubectl set env`).
|
||||
|
||||
> **Trust boundary — this is the security-critical part.** A trusted identity
|
||||
> header is only safe when **nothing else can reach the gateway**. The gateway
|
||||
> Service must stay ClusterIP and be reachable exclusively through the
|
||||
> authenticating proxy; if a client can reach `dsh-gateway:8080` directly (a
|
||||
> second Ingress/Gateway/Route, a port-forward on a shared host, a broad
|
||||
> NetworkPolicy), it can send `X-Forwarded-User` itself and impersonate any user.
|
||||
> Treat `AUTH_PROXY_HEADER` as "the proxy already authenticated this request",
|
||||
> not as an authentication mechanism of its own. Disable SSO by removing the
|
||||
> variable: the header is then ignored and bearer-token/session auth applies
|
||||
> again.
|
||||
|
||||
### Proxy failure semantics
|
||||
|
||||
| Situation | Result |
|
||||
@@ -123,6 +160,7 @@ can also be injected into `create_app` for tests.
|
||||
| `MAX_BODY_MB` | `32` | Maximum buffered request body (and `413` threshold). |
|
||||
| `HTTP_TIMEOUT` | `300` | Upstream connect/read timeout in seconds. Proxied streams (SSE, long responses) keep the connect/write budget but have **no read timeout**, so an idle event stream is not cut every five minutes. |
|
||||
| `SWEEP_INTERVAL` | `15` | Seconds between idle sweeps. |
|
||||
| `AUTH_PROXY_HEADER` | unset | Opt-in SSO: name of the trusted identity header set by an authenticating reverse proxy (e.g. `X-Forwarded-User`). Unset disables SSO and the header is never trusted. Enable only when the Service is reachable exclusively through that proxy (see [SSO](#sso-opt-in)). |
|
||||
|
||||
### Required RBAC
|
||||
|
||||
@@ -147,12 +185,21 @@ python -m ruff check .
|
||||
uvicorn gateway:create_app --factory --host 0.0.0.0 --port 8080
|
||||
```
|
||||
|
||||
Container image:
|
||||
Container image, from the repository root:
|
||||
|
||||
```bash
|
||||
docker build -t dsh-gateway examples/dsh-agents-as-a-service/gateway
|
||||
make gateway # builds dsh-aaas-gateway:dev from this directory
|
||||
```
|
||||
|
||||
The equivalent direct build is
|
||||
`docker build -t dsh-aaas-gateway:dev examples/dsh-agents-as-a-service/gateway`.
|
||||
For kind, retag it to the image reference in
|
||||
`examples/dsh-agents-as-a-service/manifests/50-gateway.yaml` and `kind load` it.
|
||||
Published images are nested below the main image as
|
||||
`<GHCR_IMAGE>/dsh-agents-as-a-service/gateway:<identity>`, where the identity
|
||||
comes from [`versions.env`](../../../versions.env) — see
|
||||
[`docs/versioning.md`](../../../docs/versioning.md).
|
||||
|
||||
The image runs as UID `10000`, sets `PYTHONDONTWRITEBYTECODE=1` and
|
||||
`HOME=/tmp` (read-only-rootfs compatible), listens on `8080`, and disables
|
||||
uvicorn's access log so bearer tokens in URLs are not logged.
|
||||
@@ -182,3 +229,8 @@ uvicorn's access log so bearer tokens in URLs are not logged.
|
||||
re-runs the handshake once and replays the request.
|
||||
* **Only DSH on `:3080` is proxied.** There is no dashboard/API-port split and
|
||||
no platform API key injection.
|
||||
* **SSO trusts a header.** With `AUTH_PROXY_HEADER` set, any client that can
|
||||
reach the gateway Service directly can spoof the identity header. The gateway
|
||||
cannot enforce the trust boundary itself: keep the Service ClusterIP and
|
||||
reachable only through the authenticating proxy. There is no mTLS or signed
|
||||
identity assertion.
|
||||
|
||||
@@ -195,9 +195,7 @@ def derive_state(sandbox: dict | None) -> str:
|
||||
|
||||
def token_matches(claim: dict, token: str | None) -> bool:
|
||||
"""Timing-safe check of a user token against the stored SHA-256."""
|
||||
want = (claim.get("metadata", {}).get("annotations") or {}).get(
|
||||
TOKEN_ANNOTATION, ""
|
||||
)
|
||||
want = (claim.get("metadata", {}).get("annotations") or {}).get(TOKEN_ANNOTATION, "")
|
||||
if not token or not want:
|
||||
return False
|
||||
got = hashlib.sha256(token.encode()).hexdigest()
|
||||
@@ -518,9 +516,7 @@ class Kubernetes:
|
||||
|
||||
async def get_pod(self, name: str) -> dict | None:
|
||||
try:
|
||||
pod = await self._call(
|
||||
"core", "read_namespaced_pod", name, self.namespace
|
||||
)
|
||||
pod = await self._call("core", "read_namespaced_pod", name, self.namespace)
|
||||
except ApiException as exc:
|
||||
if exc.status == 404:
|
||||
return None
|
||||
@@ -530,9 +526,7 @@ class Kubernetes:
|
||||
# so callers can keep using ``pod.get(...)``.
|
||||
return k8s_client.ApiClient().sanitize_for_serialization(pod)
|
||||
|
||||
async def read_pod_logs(
|
||||
self, name: str, container: str, tail_lines: int
|
||||
) -> str:
|
||||
async def read_pod_logs(self, name: str, container: str, tail_lines: int) -> str:
|
||||
result = await self._call(
|
||||
"core",
|
||||
"read_namespaced_pod_log",
|
||||
@@ -616,9 +610,7 @@ class Gateway:
|
||||
return False # a session for another user never authorizes this one
|
||||
return token_matches(claim, token)
|
||||
|
||||
async def _sso_session(
|
||||
self, remote: str, headers, cookies, response: Response
|
||||
) -> dict | None:
|
||||
async def _sso_session(self, remote: str, headers, cookies, response: Response) -> dict | None:
|
||||
"""Provision ``remote`` idempotently and mint/refresh a session token.
|
||||
|
||||
The authenticating proxy has already verified ``remote``, so it is
|
||||
@@ -682,9 +674,7 @@ class Gateway:
|
||||
while True:
|
||||
async with self.state.lock:
|
||||
if not self.state.maintenance.get(user):
|
||||
self.state.in_flight[user] = (
|
||||
self.state.in_flight.get(user, 0) + 1
|
||||
)
|
||||
self.state.in_flight[user] = self.state.in_flight.get(user, 0) + 1
|
||||
self.state.last_activity[user] = time.time()
|
||||
return True
|
||||
if time.monotonic() > deadline:
|
||||
@@ -693,9 +683,7 @@ class Gateway:
|
||||
|
||||
async def _release(self, user: str) -> None:
|
||||
async with self.state.lock:
|
||||
self.state.in_flight[user] = max(
|
||||
self.state.in_flight.get(user, 1) - 1, 0
|
||||
)
|
||||
self.state.in_flight[user] = max(self.state.in_flight.get(user, 1) - 1, 0)
|
||||
self.state.last_activity[user] = time.time()
|
||||
|
||||
# -- sandbox / handshake helpers --------------------------------------
|
||||
@@ -724,9 +712,7 @@ class Gateway:
|
||||
# Wake-on-connect: safe in ANY state, including mid-suspension.
|
||||
self._clear_handshake(user)
|
||||
try:
|
||||
await self.k8s.patch_operating_mode(
|
||||
sandbox["metadata"]["name"], "Running"
|
||||
)
|
||||
await self.k8s.patch_operating_mode(sandbox["metadata"]["name"], "Running")
|
||||
except Exception as exc:
|
||||
raise SandboxUnavailable("could not wake the sandbox") from exc
|
||||
sandbox = await self.wait_ready(user, self.settings.wake_timeout)
|
||||
@@ -740,10 +726,7 @@ class Gateway:
|
||||
|
||||
def _pod_name(self, sandbox: dict) -> str:
|
||||
annotations = sandbox.get("metadata", {}).get("annotations") or {}
|
||||
return (
|
||||
annotations.get(POD_NAME_ANNOTATION)
|
||||
or sandbox["metadata"]["name"]
|
||||
)
|
||||
return annotations.get(POD_NAME_ANNOTATION) or sandbox["metadata"]["name"]
|
||||
|
||||
@staticmethod
|
||||
def _pod_ip(sandbox: dict) -> str | None:
|
||||
@@ -787,16 +770,11 @@ class Gateway:
|
||||
self.settings.dsh_log_tail_lines,
|
||||
)
|
||||
except ApiException as exc:
|
||||
raise HandshakeError(
|
||||
f"could not read logs for {pod_name}: {exc}"
|
||||
) from exc
|
||||
raise HandshakeError(f"could not read logs for {pod_name}: {exc}") from exc
|
||||
launch_token = parse_dsh_launch_token(logs)
|
||||
if not launch_token:
|
||||
raise HandshakeError("DSH launch token not found in pod logs")
|
||||
url = (
|
||||
f"http://{fqdn}:{self.settings.dsh_port}/"
|
||||
f"?token={quote(launch_token, safe='')}"
|
||||
)
|
||||
url = f"http://{fqdn}:{self.settings.dsh_port}/?token={quote(launch_token, safe='')}"
|
||||
headers = {
|
||||
"Host": host,
|
||||
"X-Forwarded-Host": host,
|
||||
@@ -812,9 +790,7 @@ class Gateway:
|
||||
except httpx.HTTPError as exc:
|
||||
raise HandshakeError(f"DSH handshake failed: {exc}") from exc
|
||||
if response.status_code >= 400:
|
||||
raise HandshakeError(
|
||||
f"DSH handshake returned {response.status_code}"
|
||||
)
|
||||
raise HandshakeError(f"DSH handshake returned {response.status_code}")
|
||||
cookie = extract_dsh_cookies(response.headers.get_list("set-cookie"))
|
||||
if not cookie:
|
||||
raise HandshakeError("DSH handshake set no dsh-auth cookie")
|
||||
@@ -856,7 +832,7 @@ class Gateway:
|
||||
else:
|
||||
status = html.escape(state)
|
||||
confirm = (
|
||||
'onsubmit="return confirm(\'Delete this sandbox and start fresh? '
|
||||
"onsubmit=\"return confirm('Delete this sandbox and start fresh? "
|
||||
"Your workspace data will be wiped.')\""
|
||||
)
|
||||
body = (
|
||||
@@ -887,9 +863,7 @@ class Gateway:
|
||||
# session cookie minted by _sso_session; the real page is rendered
|
||||
# once the sandbox state is known.
|
||||
response = HTMLResponse()
|
||||
claim = await self._sso_session(
|
||||
remote, request.headers, request.cookies, response
|
||||
)
|
||||
claim = await self._sso_session(remote, request.headers, request.cookies, response)
|
||||
sandbox = await self._sandbox_of(claim) if claim else None
|
||||
page = HTMLResponse(self._sso_page(remote, derive_state(sandbox)))
|
||||
if "set-cookie" in response.headers:
|
||||
@@ -953,14 +927,10 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
|
||||
@staticmethod
|
||||
def _wants_form(request: Request) -> bool:
|
||||
ctype = request.headers.get("content-type", "").lower()
|
||||
return ctype.startswith(
|
||||
("application/x-www-form-urlencoded", "multipart/form-data")
|
||||
)
|
||||
return ctype.startswith(("application/x-www-form-urlencoded", "multipart/form-data"))
|
||||
|
||||
async def _read_payload(self, request: Request) -> dict:
|
||||
if request.headers.get("content-type", "").lower().startswith(
|
||||
"application/json"
|
||||
):
|
||||
if request.headers.get("content-type", "").lower().startswith("application/json"):
|
||||
data = await self._read_body(request)
|
||||
try:
|
||||
payload = json.loads(data)
|
||||
@@ -997,9 +967,7 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
|
||||
return self._error(413, "request body too large")
|
||||
user = payload.get("user", "")
|
||||
if not valid_user(user):
|
||||
return self._error(
|
||||
400, "body must be {'user': '<dns-1123 label>'}"
|
||||
)
|
||||
return self._error(400, "body must be {'user': '<dns-1123 label>'}")
|
||||
if await self.k8s.get_claim(user) is not None:
|
||||
return self._error(409, "user exists")
|
||||
token = secrets.token_urlsafe(32)
|
||||
@@ -1059,9 +1027,7 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
|
||||
if claim is None:
|
||||
return self._error(404, "not found")
|
||||
token, cookie_user = self.credentials(request.headers, request.cookies)
|
||||
if not self._authorized(
|
||||
claim, token, cookie_user, user, self.remote_user(request.headers)
|
||||
):
|
||||
if not self._authorized(claim, token, cookie_user, user, self.remote_user(request.headers)):
|
||||
return self._error(401, "unauthorized")
|
||||
sandbox = await self._sandbox_of(claim)
|
||||
return JSONResponse(
|
||||
@@ -1080,9 +1046,7 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
|
||||
if claim is None:
|
||||
return self._error(404, "not found")
|
||||
token, cookie_user = self.credentials(request.headers, request.cookies)
|
||||
if not self._authorized(
|
||||
claim, token, cookie_user, user, self.remote_user(request.headers)
|
||||
):
|
||||
if not self._authorized(claim, token, cookie_user, user, self.remote_user(request.headers)):
|
||||
return self._error(401, "unauthorized")
|
||||
if self._mutation_from_cookie(request.headers, "DELETE", cookie_user):
|
||||
return self._error(403, "cross-origin request rejected")
|
||||
@@ -1091,9 +1055,7 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
|
||||
async with self.state.lock:
|
||||
self.state.last_activity.pop(user, None)
|
||||
self.state.in_flight.pop(user, None)
|
||||
return JSONResponse(
|
||||
{"user": user, "note": "claim deleted; sandbox and PVC cascade"}
|
||||
)
|
||||
return JSONResponse({"user": user, "note": "claim deleted; sandbox and PVC cascade"})
|
||||
|
||||
async def recreate(self, request: Request) -> Response:
|
||||
"""Delete and re-provision the SSO user's sandbox for a clean start.
|
||||
@@ -1122,9 +1084,7 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
|
||||
break
|
||||
await asyncio.sleep(1)
|
||||
response = RedirectResponse(f"/u/{remote}/", status_code=303)
|
||||
await self._sso_session(
|
||||
remote, request.headers, request.cookies, response
|
||||
)
|
||||
await self._sso_session(remote, request.headers, request.cookies, response)
|
||||
return response
|
||||
|
||||
# -- HTTP proxy --------------------------------------------------------
|
||||
@@ -1141,9 +1101,7 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
|
||||
return RedirectResponse("/", status_code=303)
|
||||
return self._error(404, "unknown user")
|
||||
token, cookie_user = self.credentials(request.headers, request.cookies)
|
||||
if remote == user and not (
|
||||
cookie_user == user and token_matches(claim, token)
|
||||
):
|
||||
if remote == user and not (cookie_user == user and token_matches(claim, token)):
|
||||
# SSO user without a valid session cookie: bootstrap, then continue.
|
||||
return RedirectResponse("/", status_code=303)
|
||||
if not self._authorized(claim, token, cookie_user, user, remote):
|
||||
@@ -1152,31 +1110,21 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
|
||||
return self._error(403, "cross-origin request rejected")
|
||||
|
||||
if not await self._acquire(user):
|
||||
return self._error(
|
||||
503, "gateway busy, retry shortly", {"Retry-After": "5"}
|
||||
)
|
||||
return self._error(503, "gateway busy, retry shortly", {"Retry-After": "5"})
|
||||
streaming = False
|
||||
try:
|
||||
sandbox, fqdn = await self._resolve_sandbox(user, claim)
|
||||
host = request.headers.get("host", "")
|
||||
cookie, _, was_cached = await self.dsh_cookie(
|
||||
user, sandbox, fqdn, host
|
||||
)
|
||||
cookie, _, was_cached = await self.dsh_cookie(user, sandbox, fqdn, host)
|
||||
body = await self._read_body(request)
|
||||
|
||||
upstream = await self._send_upstream(
|
||||
request, user, fqdn, cookie, body
|
||||
)
|
||||
upstream = await self._send_upstream(request, user, fqdn, cookie, body)
|
||||
if upstream.status_code == 401 and was_cached:
|
||||
# The cached DSH cookie expired (pod restart, logout, ...):
|
||||
# handshake once more and replay the (already buffered) request.
|
||||
await upstream.aclose()
|
||||
cookie, _, _ = await self.dsh_cookie(
|
||||
user, sandbox, fqdn, host, force=True
|
||||
)
|
||||
upstream = await self._send_upstream(
|
||||
request, user, fqdn, cookie, body
|
||||
)
|
||||
cookie, _, _ = await self.dsh_cookie(user, sandbox, fqdn, host, force=True)
|
||||
upstream = await self._send_upstream(request, user, fqdn, cookie, body)
|
||||
|
||||
response = StreamingResponse(
|
||||
self._relay(user, upstream),
|
||||
@@ -1216,8 +1164,7 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
|
||||
headers = {
|
||||
key: value
|
||||
for key, value in request.headers.items()
|
||||
if key.lower() not in OWNED_REQUEST_HEADERS
|
||||
and key.lower() != "content-length"
|
||||
if key.lower() not in OWNED_REQUEST_HEADERS and key.lower() != "content-length"
|
||||
}
|
||||
host = request.headers.get("host", "")
|
||||
headers["host"] = host
|
||||
@@ -1229,9 +1176,7 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
|
||||
else:
|
||||
headers["x-forwarded-for"] = ""
|
||||
port = host.rsplit(":", 1)[1] if ":" in host else ""
|
||||
headers["x-forwarded-port"] = port or (
|
||||
"443" if request.url.scheme == "https" else "80"
|
||||
)
|
||||
headers["x-forwarded-port"] = port or ("443" if request.url.scheme == "https" else "80")
|
||||
|
||||
last_error: Exception | None = None
|
||||
for attempt in range(self.settings.upstream_attempts):
|
||||
@@ -1341,9 +1286,7 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
|
||||
if ":" in upstream_host and not upstream_host.startswith("["):
|
||||
upstream_host = f"[{upstream_host}]" # IPv6 literal
|
||||
authority = f"{upstream_host}:{self.settings.dsh_port}"
|
||||
cookie, _, _ = await self.dsh_cookie(
|
||||
user, sandbox, fqdn, authority
|
||||
)
|
||||
cookie, _, _ = await self.dsh_cookie(user, sandbox, fqdn, authority)
|
||||
url = f"ws://{authority}/{path}"
|
||||
query = filtered_query(websocket.url.query)
|
||||
if query:
|
||||
@@ -1362,9 +1305,7 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
|
||||
headers["x-forwarded-for"] = websocket.client.host
|
||||
subprotocols = [
|
||||
part.strip()
|
||||
for part in websocket.headers.get(
|
||||
"sec-websocket-protocol", ""
|
||||
).split(",")
|
||||
for part in websocket.headers.get("sec-websocket-protocol", "").split(",")
|
||||
if part.strip()
|
||||
]
|
||||
upstream_cm = self.ws_connect(
|
||||
@@ -1380,16 +1321,12 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
|
||||
return await self._ws_reject(websocket, 1011)
|
||||
|
||||
try:
|
||||
await websocket.accept(
|
||||
subprotocol=getattr(upstream, "subprotocol", None)
|
||||
)
|
||||
await websocket.accept(subprotocol=getattr(upstream, "subprotocol", None))
|
||||
except RuntimeError:
|
||||
pass
|
||||
await self._pump(websocket, upstream)
|
||||
with contextlib.suppress(Exception):
|
||||
await websocket.close(
|
||||
code=self._upstream_close_code(upstream)
|
||||
)
|
||||
await websocket.close(code=self._upstream_close_code(upstream))
|
||||
finally:
|
||||
if upstream_cm is not None:
|
||||
with contextlib.suppress(Exception):
|
||||
@@ -1429,9 +1366,7 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
|
||||
asyncio.create_task(client_to_upstream()),
|
||||
asyncio.create_task(upstream_to_client()),
|
||||
]
|
||||
done, pending = await asyncio.wait(
|
||||
tasks, return_when=asyncio.FIRST_COMPLETED
|
||||
)
|
||||
done, pending = await asyncio.wait(tasks, return_when=asyncio.FIRST_COMPLETED)
|
||||
for task in pending:
|
||||
task.cancel()
|
||||
await asyncio.gather(*done, *pending, return_exceptions=True)
|
||||
@@ -1458,17 +1393,13 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
|
||||
continue
|
||||
if self.state.in_flight.get(user, 0) > 0:
|
||||
continue # requests in flight => the user is active
|
||||
idle = time.time() - self.state.last_activity.setdefault(
|
||||
user, time.time()
|
||||
)
|
||||
idle = time.time() - self.state.last_activity.setdefault(user, time.time())
|
||||
if idle < self.settings.idle_timeout:
|
||||
continue
|
||||
self.state.maintenance[user] = True
|
||||
try:
|
||||
print(f"suspending {user} (idle {int(idle)}s)", flush=True)
|
||||
await self.k8s.patch_operating_mode(
|
||||
sandbox["metadata"]["name"], "Suspended"
|
||||
)
|
||||
await self.k8s.patch_operating_mode(sandbox["metadata"]["name"], "Suspended")
|
||||
self._clear_handshake(user)
|
||||
finally:
|
||||
async with self.state.lock:
|
||||
@@ -1519,12 +1450,8 @@ def create_app(
|
||||
k8s = k8s if k8s is not None else Kubernetes(settings.namespace)
|
||||
owns_http_client = http_client is None
|
||||
if http_client is None:
|
||||
http_client = httpx.AsyncClient(
|
||||
follow_redirects=False, timeout=settings.http_timeout
|
||||
)
|
||||
gateway = Gateway(
|
||||
settings, k8s, http_client, ws_connect or websockets.connect
|
||||
)
|
||||
http_client = httpx.AsyncClient(follow_redirects=False, timeout=settings.http_timeout)
|
||||
gateway = Gateway(settings, k8s, http_client, ws_connect or websockets.connect)
|
||||
routes = [
|
||||
Route("/healthz", gateway.healthz),
|
||||
Route("/", gateway.landing),
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
-r requirements.txt
|
||||
pytest==9.1.1
|
||||
pytest-asyncio==1.4.0
|
||||
pytest-cov==7.1.0
|
||||
ruff==0.17.1
|
||||
|
||||
@@ -45,9 +45,7 @@ FQDN = "dsh-alice-sbx.test-ns.svc.cluster.local"
|
||||
|
||||
def dsh_response(status=200, content=b"", headers=None):
|
||||
"""A MockTransport response that is a real stream (``aiter_raw`` works)."""
|
||||
return httpx.Response(
|
||||
status, headers=headers, stream=httpx.ByteStream(content)
|
||||
)
|
||||
return httpx.Response(status, headers=headers, stream=httpx.ByteStream(content))
|
||||
|
||||
|
||||
# --- fakes -----------------------------------------------------------------
|
||||
@@ -99,9 +97,7 @@ class FakeK8s:
|
||||
self.token_patches.append((user, token_hash))
|
||||
claim = self.claims.get(user)
|
||||
if claim is not None:
|
||||
annotations = claim.setdefault("metadata", {}).setdefault(
|
||||
"annotations", {}
|
||||
)
|
||||
annotations = claim.setdefault("metadata", {}).setdefault("annotations", {})
|
||||
annotations["aaas.example.com/token-sha256"] = token_hash
|
||||
|
||||
async def get_pod(self, name):
|
||||
@@ -118,9 +114,7 @@ class FakeUpstream:
|
||||
self.requests = []
|
||||
self.handshake_requests = []
|
||||
self.handshake_status = 303
|
||||
self.handshake_cookies = [
|
||||
"dsh-auth-a=one; Path=/; HttpOnly; SameSite=Strict"
|
||||
]
|
||||
self.handshake_cookies = ["dsh-auth-a=one; Path=/; HttpOnly; SameSite=Strict"]
|
||||
self.proxy_handler = None
|
||||
self.proxy_requests = []
|
||||
|
||||
@@ -140,9 +134,7 @@ class FakeUpstream:
|
||||
self.proxy_requests.append(request)
|
||||
if self.proxy_handler is not None:
|
||||
return self.proxy_handler(request)
|
||||
return dsh_response(
|
||||
200, b"hello from DSH", {"content-type": "text/plain"}
|
||||
)
|
||||
return dsh_response(200, b"hello from DSH", {"content-type": "text/plain"})
|
||||
|
||||
|
||||
class FakeWS:
|
||||
@@ -207,9 +199,7 @@ def add_user(k8s, user="alice", token=TOKEN, ready=True, pod_name=None, suffix="
|
||||
"metadata": {
|
||||
"name": claim_name(user),
|
||||
"annotations": {
|
||||
"aaas.example.com/token-sha256": hashlib.sha256(
|
||||
token.encode()
|
||||
).hexdigest()
|
||||
"aaas.example.com/token-sha256": hashlib.sha256(token.encode()).hexdigest()
|
||||
},
|
||||
},
|
||||
"status": {"sandbox": {"name": sandbox_name}},
|
||||
@@ -224,10 +214,7 @@ def add_user(k8s, user="alice", token=TOKEN, ready=True, pod_name=None, suffix="
|
||||
},
|
||||
}
|
||||
k8s.pods[pod_name] = {"metadata": {"name": pod_name, "uid": POD_UID}}
|
||||
k8s.logs[pod_name] = (
|
||||
"booting\n"
|
||||
f"dsh web: http://10.0.0.7:3080/?token={TOKEN}-launch\n"
|
||||
)
|
||||
k8s.logs[pod_name] = f"booting\ndsh web: http://10.0.0.7:3080/?token={TOKEN}-launch\n"
|
||||
return sandbox_name, pod_name
|
||||
|
||||
|
||||
@@ -243,9 +230,7 @@ def env():
|
||||
)
|
||||
k8s = FakeK8s()
|
||||
upstream = FakeUpstream()
|
||||
http_client = httpx.AsyncClient(
|
||||
transport=httpx.MockTransport(upstream), timeout=5
|
||||
)
|
||||
http_client = httpx.AsyncClient(transport=httpx.MockTransport(upstream), timeout=5)
|
||||
ws_connect = FakeWSConnect()
|
||||
app = create_app(
|
||||
settings=settings,
|
||||
@@ -275,9 +260,7 @@ def asgi_client(env, cookies=None):
|
||||
|
||||
|
||||
def stale_handshake(env):
|
||||
env.gateway.state.handshake[("alice", POD_UID, "gateway.test")] = (
|
||||
"dsh-auth-a=stale"
|
||||
)
|
||||
env.gateway.state.handshake[("alice", POD_UID, "gateway.test")] = "dsh-auth-a=stale"
|
||||
|
||||
|
||||
# --- pure helpers ----------------------------------------------------------
|
||||
@@ -419,9 +402,7 @@ def test_token_matches_is_timing_safe_hash_check():
|
||||
claim = {
|
||||
"metadata": {
|
||||
"annotations": {
|
||||
"aaas.example.com/token-sha256": hashlib.sha256(
|
||||
TOKEN.encode()
|
||||
).hexdigest()
|
||||
"aaas.example.com/token-sha256": hashlib.sha256(TOKEN.encode()).hexdigest()
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -486,14 +467,9 @@ async def test_kubernetes_create_claim_body(monkeypatch):
|
||||
body = args[4]
|
||||
assert body["metadata"]["name"] == "dsh-alice"
|
||||
assert body["metadata"]["annotations"]["aaas.example.com/token-sha256"] == "hash"
|
||||
assert (
|
||||
body["metadata"]["annotations"]["aaas.example.com/managed-by"]
|
||||
== "dsh-gateway"
|
||||
)
|
||||
assert body["metadata"]["annotations"]["aaas.example.com/managed-by"] == "dsh-gateway"
|
||||
assert body["spec"]["warmPoolRef"]["name"] == "test-pool"
|
||||
assert body["spec"]["additionalPodMetadata"]["labels"][
|
||||
"sandbox.users.io/dsh-user"
|
||||
] == "alice"
|
||||
assert body["spec"]["additionalPodMetadata"]["labels"]["sandbox.users.io/dsh-user"] == "alice"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@@ -625,9 +601,7 @@ async def test_landing_page_has_signup_and_login_forms(env):
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_landing_page_shows_session_when_cookie_present(env):
|
||||
async with asgi_client(
|
||||
env, cookies={"dsh_gw_session": f"alice:{TOKEN}"}
|
||||
) as client:
|
||||
async with asgi_client(env, cookies={"dsh_gw_session": f"alice:{TOKEN}"}) as client:
|
||||
response = await client.get("/")
|
||||
assert "Signed in as" in response.text
|
||||
assert "/u/alice/" in response.text
|
||||
@@ -645,9 +619,7 @@ async def test_workspace_redirect_adds_trailing_slash(env):
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
"bad_user", ["Alice", "-alice", "alice-", "", "a_b", 123, None, "a" * 60]
|
||||
)
|
||||
@pytest.mark.parametrize("bad_user", ["Alice", "-alice", "alice-", "", "a_b", 123, None, "a" * 60])
|
||||
async def test_signup_rejects_invalid_dns1123_label(env, bad_user):
|
||||
async with asgi_client(env) as client:
|
||||
response = await client.post("/users", json={"user": bad_user})
|
||||
@@ -781,9 +753,7 @@ async def test_login_json_and_failures(env):
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_logout_clears_cookie(env):
|
||||
async with asgi_client(
|
||||
env, cookies={"dsh_gw_session": f"alice:{TOKEN}"}
|
||||
) as client:
|
||||
async with asgi_client(env, cookies={"dsh_gw_session": f"alice:{TOKEN}"}) as client:
|
||||
response = await client.post("/logout")
|
||||
assert response.status_code == 303
|
||||
assert response.headers["location"] == "/"
|
||||
@@ -798,12 +768,8 @@ async def test_logout_clears_cookie(env):
|
||||
async def test_get_user_requires_valid_bearer_or_cookie(env):
|
||||
add_user(env.k8s)
|
||||
async with asgi_client(env) as client:
|
||||
forbidden = await client.get(
|
||||
"/users/alice", headers={"Authorization": "Bearer wrong"}
|
||||
)
|
||||
bearer = await client.get(
|
||||
"/users/alice", headers={"Authorization": f"Bearer {TOKEN}"}
|
||||
)
|
||||
forbidden = await client.get("/users/alice", headers={"Authorization": "Bearer wrong"})
|
||||
bearer = await client.get("/users/alice", headers={"Authorization": f"Bearer {TOKEN}"})
|
||||
cookie = await client.get(
|
||||
"/users/alice",
|
||||
headers={"Cookie": f"dsh_gw_session=alice:{TOKEN}"},
|
||||
@@ -839,9 +805,7 @@ async def test_invalid_user_in_path_is_404(env, path):
|
||||
async def test_delete_user_requires_auth(env):
|
||||
add_user(env.k8s)
|
||||
async with asgi_client(env) as client:
|
||||
response = await client.delete(
|
||||
"/users/alice", headers={"Authorization": "Bearer wrong"}
|
||||
)
|
||||
response = await client.delete("/users/alice", headers={"Authorization": "Bearer wrong"})
|
||||
assert response.status_code == 401
|
||||
assert env.k8s.deleted == []
|
||||
|
||||
@@ -853,12 +817,8 @@ async def test_delete_with_cookie_requires_same_origin(env):
|
||||
cookies = {"dsh_gw_session": f"alice:{TOKEN}"}
|
||||
async with asgi_client(env, cookies=cookies) as client:
|
||||
missing = await client.delete("/users/alice")
|
||||
cross = await client.delete(
|
||||
"/users/alice", headers={"Origin": "http://evil.test"}
|
||||
)
|
||||
good = await client.delete(
|
||||
"/users/alice", headers={"Origin": "http://gateway.test"}
|
||||
)
|
||||
cross = await client.delete("/users/alice", headers={"Origin": "http://evil.test"})
|
||||
good = await client.delete("/users/alice", headers={"Origin": "http://gateway.test"})
|
||||
assert missing.status_code == 403
|
||||
assert cross.status_code == 403
|
||||
assert good.status_code == 200
|
||||
@@ -870,9 +830,7 @@ async def test_delete_with_cookie_requires_same_origin(env):
|
||||
async def test_delete_with_bearer_does_not_need_origin(env):
|
||||
add_user(env.k8s)
|
||||
async with asgi_client(env) as client:
|
||||
response = await client.delete(
|
||||
"/users/alice", headers={"Authorization": f"Bearer {TOKEN}"}
|
||||
)
|
||||
response = await client.delete("/users/alice", headers={"Authorization": f"Bearer {TOKEN}"})
|
||||
assert response.status_code == 200
|
||||
|
||||
|
||||
@@ -881,9 +839,7 @@ async def test_delete_missing_claim_is_404(env):
|
||||
add_user(env.k8s)
|
||||
env.k8s.claims.clear()
|
||||
async with asgi_client(env) as client:
|
||||
response = await client.delete(
|
||||
"/users/alice", headers={"Authorization": f"Bearer {TOKEN}"}
|
||||
)
|
||||
response = await client.delete("/users/alice", headers={"Authorization": f"Bearer {TOKEN}"})
|
||||
assert response.status_code == 404
|
||||
|
||||
|
||||
@@ -958,9 +914,9 @@ async def test_missing_cookie_in_handshake_is_503(env):
|
||||
@pytest.mark.asyncio
|
||||
async def test_pod_annotation_overrides_sandbox_name(env):
|
||||
sandbox_name, pod_name = add_user(env.k8s)
|
||||
env.k8s.sandboxes[sandbox_name]["metadata"]["annotations"][
|
||||
"agents.x-k8s.io/pod-name"
|
||||
] = "custom-pod"
|
||||
env.k8s.sandboxes[sandbox_name]["metadata"]["annotations"]["agents.x-k8s.io/pod-name"] = (
|
||||
"custom-pod"
|
||||
)
|
||||
env.k8s.pods["custom-pod"] = {"metadata": {"uid": "other-uid"}}
|
||||
env.k8s.logs["custom-pod"] = env.k8s.logs[pod_name]
|
||||
async with asgi_client(env, cookies={"dsh_gw_session": f"alice:{TOKEN}"}) as client:
|
||||
@@ -1082,9 +1038,7 @@ async def test_proxy_upstream_connection_error_is_502(env):
|
||||
@pytest.mark.asyncio
|
||||
async def test_proxy_body_limit_returns_413(env):
|
||||
add_user(env.k8s)
|
||||
env.gateway.settings = Settings(
|
||||
**{**env.settings.__dict__, "max_body_bytes": 4}
|
||||
)
|
||||
env.gateway.settings = Settings(**{**env.settings.__dict__, "max_body_bytes": 4})
|
||||
async with asgi_client(env) as client:
|
||||
response = await client.post(
|
||||
"/u/alice/api/echo",
|
||||
@@ -1194,9 +1148,7 @@ async def test_proxied_requests_use_the_stream_timeout(env, monkeypatch):
|
||||
"server": ("gateway.test", 80),
|
||||
"path_params": {"user": "alice", "path": "events"},
|
||||
}
|
||||
response = await env.gateway._send_upstream(
|
||||
Request(scope), "alice", FQDN, "dsh-auth-a=1", b""
|
||||
)
|
||||
response = await env.gateway._send_upstream(Request(scope), "alice", FQDN, "dsh-auth-a=1", b"")
|
||||
await response.aclose()
|
||||
|
||||
assert captured["timeout"].read is None
|
||||
@@ -1359,9 +1311,7 @@ async def test_sweeper_skips_claim_it_does_not_own(env):
|
||||
assert "alice" not in env.gateway.state.last_activity
|
||||
|
||||
# ...and does not suspend it even when it looks long-idle.
|
||||
env.gateway.state.last_activity["alice"] = (
|
||||
time.time() - env.settings.idle_timeout - 10
|
||||
)
|
||||
env.gateway.state.last_activity["alice"] = time.time() - env.settings.idle_timeout - 10
|
||||
await env.gateway.sweep_once()
|
||||
assert env.k8s.patches == []
|
||||
|
||||
@@ -1370,9 +1320,7 @@ async def test_sweeper_skips_claim_it_does_not_own(env):
|
||||
async def test_sweeper_suspends_idle_ready_sandbox(env):
|
||||
env.k8s.claims["alice"] = ready_claim()
|
||||
env.k8s.sandboxes["dsh-alice-sbx"] = ready_sandbox()
|
||||
env.gateway.state.last_activity["alice"] = (
|
||||
time.time() - env.settings.idle_timeout - 10
|
||||
)
|
||||
env.gateway.state.last_activity["alice"] = time.time() - env.settings.idle_timeout - 10
|
||||
env.gateway.state.handshake[("alice", POD_UID, "gateway.test")] = "dsh-auth-a=x"
|
||||
|
||||
await env.gateway.sweep_once()
|
||||
@@ -1478,9 +1426,7 @@ async def test_sso_landing_new_user_shows_provisioning(env):
|
||||
async def test_sso_landing_ready_connects_and_offers_recreate_only(env):
|
||||
enable_sso(env)
|
||||
add_user(env.k8s)
|
||||
async with asgi_client(
|
||||
env, cookies={"dsh_gw_session": f"alice:{TOKEN}"}
|
||||
) as client:
|
||||
async with asgi_client(env, cookies={"dsh_gw_session": f"alice:{TOKEN}"}) as client:
|
||||
response = await client.get("/", headers={"X-Forwarded-User": "alice"})
|
||||
assert response.status_code == 200
|
||||
assert "Signed in as <strong>alice</strong>" in response.text
|
||||
@@ -1512,9 +1458,7 @@ async def test_recreate_deletes_and_reprovisions(env):
|
||||
enable_sso(env)
|
||||
add_user(env.k8s)
|
||||
async with asgi_client(env) as client:
|
||||
response = await client.post(
|
||||
"/recreate", headers={"X-Forwarded-User": "alice"}
|
||||
)
|
||||
response = await client.post("/recreate", headers={"X-Forwarded-User": "alice"})
|
||||
assert response.status_code == 303
|
||||
assert response.headers["location"] == "/u/alice/"
|
||||
assert env.k8s.deleted == ["alice"]
|
||||
@@ -1526,9 +1470,7 @@ async def test_recreate_deletes_and_reprovisions(env):
|
||||
async def test_recreate_requires_sso(env):
|
||||
add_user(env.k8s)
|
||||
async with asgi_client(env) as client:
|
||||
response = await client.post(
|
||||
"/recreate", headers={"X-Forwarded-User": "alice"}
|
||||
)
|
||||
response = await client.post("/recreate", headers={"X-Forwarded-User": "alice"})
|
||||
assert response.status_code == 401
|
||||
assert env.k8s.deleted == []
|
||||
|
||||
@@ -1537,9 +1479,7 @@ async def test_recreate_requires_sso(env):
|
||||
async def test_recreate_rejects_cross_origin(env):
|
||||
enable_sso(env)
|
||||
add_user(env.k8s)
|
||||
async with asgi_client(
|
||||
env, cookies={"dsh_gw_session": f"alice:{TOKEN}"}
|
||||
) as client:
|
||||
async with asgi_client(env, cookies={"dsh_gw_session": f"alice:{TOKEN}"}) as client:
|
||||
response = await client.post(
|
||||
"/recreate",
|
||||
headers={"X-Forwarded-User": "alice", "Origin": "http://evil.test"},
|
||||
@@ -1553,9 +1493,7 @@ async def test_sso_get_user_authorizes_via_header(env):
|
||||
enable_sso(env)
|
||||
add_user(env.k8s)
|
||||
async with asgi_client(env) as client:
|
||||
response = await client.get(
|
||||
"/users/alice", headers={"X-Forwarded-User": "alice"}
|
||||
)
|
||||
response = await client.get("/users/alice", headers={"X-Forwarded-User": "alice"})
|
||||
assert response.status_code == 200
|
||||
assert response.json()["user"] == "alice"
|
||||
|
||||
@@ -1565,9 +1503,7 @@ async def test_sso_proxy_redirects_to_bootstrap_without_session(env):
|
||||
enable_sso(env)
|
||||
add_user(env.k8s)
|
||||
async with asgi_client(env) as client:
|
||||
response = await client.get(
|
||||
"/u/alice/", headers={"X-Forwarded-User": "alice"}
|
||||
)
|
||||
response = await client.get("/u/alice/", headers={"X-Forwarded-User": "alice"})
|
||||
assert response.status_code == 303
|
||||
assert response.headers["location"] == "/"
|
||||
|
||||
@@ -1576,9 +1512,7 @@ async def test_sso_proxy_redirects_to_bootstrap_without_session(env):
|
||||
async def test_sso_proxy_redirects_to_bootstrap_when_claim_missing(env):
|
||||
enable_sso(env)
|
||||
async with asgi_client(env) as client:
|
||||
response = await client.get(
|
||||
"/u/alice/", headers={"X-Forwarded-User": "alice"}
|
||||
)
|
||||
response = await client.get("/u/alice/", headers={"X-Forwarded-User": "alice"})
|
||||
assert response.status_code == 303
|
||||
assert response.headers["location"] == "/"
|
||||
|
||||
@@ -1587,7 +1521,5 @@ async def test_sso_proxy_redirects_to_bootstrap_when_claim_missing(env):
|
||||
async def test_spoofed_header_ignored_when_sso_disabled(env):
|
||||
add_user(env.k8s)
|
||||
async with asgi_client(env) as client:
|
||||
response = await client.get(
|
||||
"/u/alice/", headers={"X-Forwarded-User": "alice"}
|
||||
)
|
||||
response = await client.get("/u/alice/", headers={"X-Forwarded-User": "alice"})
|
||||
assert response.status_code == 401
|
||||
|
||||
@@ -5,7 +5,7 @@
|
||||
# on the SandboxTemplate, stamped before any user exists — a warm-pool
|
||||
# constraint), so a real key committed here would be shared by every install
|
||||
# of this example. Generate a fresh value per install (README step 1;
|
||||
# run-test-kind.sh uses a throwaway):
|
||||
# scripts/e2e-kind.sh uses a throwaway):
|
||||
#
|
||||
# kubectl -n dsh-demo create secret generic dsh-provider-keys \
|
||||
# --from-literal=DEEPSEEK_API_KEY="sk-..."
|
||||
@@ -2,6 +2,9 @@
|
||||
# exists, so per-user secrets cannot live in pod env; the shared provider key
|
||||
# is injected via envFrom, and claims that would bypass the warm pool are
|
||||
# rejected outright (see 40-claim-rejected.yaml).
|
||||
#
|
||||
# A stripped-down single-tenant variant of this template (no warm pool, no
|
||||
# policy enforcement) lives in examples/minimal-single-tenant/.
|
||||
apiVersion: extensions.agents.x-k8s.io/v1beta1
|
||||
kind: SandboxTemplate
|
||||
metadata:
|
||||
@@ -18,8 +21,8 @@ spec:
|
||||
# The gateway routes each user to their DSH Web instance by this name.
|
||||
service: true
|
||||
|
||||
# No gateway pod scopes ingress in this standalone demo, so the
|
||||
# NetworkPolicy is left unmanaged. A production platform would use
|
||||
# The example gateway does not manage NetworkPolicy, so it is left
|
||||
# unmanaged here. A production platform would use
|
||||
# networkPolicyManagement: Managed and restrict ingress to its gateway on
|
||||
# :3080 — with the shared provider key, that NetworkPolicy is the per-user
|
||||
# isolation boundary.
|
||||
@@ -41,11 +44,12 @@ spec:
|
||||
# with 0660 and crash-loops before the gateway can proxy to it.
|
||||
# OnRootMismatch skips that walk once the volume root already carries
|
||||
# the fsGroup ownership/mode applied on the first mount.
|
||||
# Keep in sync with deploy/k8s/sandbox-template.yaml.
|
||||
# Keep in sync with the minimal variant:
|
||||
# examples/minimal-single-tenant/manifests/sandbox-template.yaml.
|
||||
fsGroupChangePolicy: OnRootMismatch
|
||||
containers:
|
||||
- name: dsh-web
|
||||
image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.2
|
||||
image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.2-r2
|
||||
imagePullPolicy: IfNotPresent
|
||||
ports:
|
||||
- name: web
|
||||
@@ -61,8 +65,10 @@ spec:
|
||||
value: ""
|
||||
# DSH Host/Origin-fences /api. The gateway forwards the browser's
|
||||
# Host, so DSH must trust the public gateway host. Empty is right
|
||||
# for port-forward/localhost; for the optional Ingress set this to
|
||||
# the same host as 60-ingress.yaml (e.g. dsh.example.com).
|
||||
# for port-forward/localhost; for either exposure variant set this
|
||||
# to the public host (e.g. dsh.example.com for
|
||||
# manifests/exposure/ingress/60-ingress.yaml, or
|
||||
# dsh-demo.example.com for the gateway-api-sso/ HTTPRoute).
|
||||
- name: DSH_TRUSTED_HOSTS
|
||||
value: ""
|
||||
# Probes target /favicon.svg: DSH gates / behind its launch-token
|
||||
@@ -6,11 +6,19 @@
|
||||
# POST /users, POST /login, ... session cookie auth
|
||||
# /u/<user>/... HTTP + WebSocket proxy to the user's DSH
|
||||
#
|
||||
# Image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway
|
||||
# For local kind development you can build and load a local tag instead, then
|
||||
# change `image` below to it:
|
||||
# docker build -t dsh-aaas-gateway:demo gateway/
|
||||
# kind load docker-image dsh-aaas-gateway:demo
|
||||
# Image: the release image published for this example, pinned by tag.
|
||||
# imagePullPolicy: IfNotPresent lets a local kind build loaded under the same
|
||||
# tag be used without a registry round-trip (scripts/e2e-kind.sh builds
|
||||
# dsh-aaas-gateway:e2e and overrides this image).
|
||||
#
|
||||
# SSO IS OPT-IN: this manifest runs the gateway with its own bearer/session
|
||||
# auth only, so AUTH_PROXY_HEADER is intentionally NOT set (see the env
|
||||
# block). To put GitHub OAuth in front of it, use
|
||||
# manifests/exposure/gateway-api-sso/.
|
||||
#
|
||||
# This Service is ClusterIP and is the only intended entry point: expose it
|
||||
# through exactly ONE of the documented variants in manifests/exposure/
|
||||
# (ingress/ or gateway-api-sso/), never both and never directly.
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
@@ -84,8 +92,8 @@ spec:
|
||||
type: RuntimeDefault
|
||||
containers:
|
||||
- name: gateway
|
||||
image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway
|
||||
imagePullPolicy: Always
|
||||
image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway:dsh-v0.2.1-rc.2-r2
|
||||
imagePullPolicy: IfNotPresent
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
@@ -105,6 +113,10 @@ spec:
|
||||
# gateway created are swept: a claim applied by hand from the
|
||||
# resource-layer manifests carries no aaas.example.com/managed-by
|
||||
# annotation and is never suspended.
|
||||
#
|
||||
# These are the demo values (360/240). scripts/e2e-kind.sh
|
||||
# overrides both to 60/120 with `kubectl set env` so the smoke
|
||||
# test does not wait six minutes for the sweeper.
|
||||
- name: IDLE_TIMEOUT
|
||||
value: "360"
|
||||
- name: WAKE_TIMEOUT
|
||||
@@ -113,19 +125,24 @@ spec:
|
||||
# the DSH launch token.
|
||||
- name: DSH_CONTAINER
|
||||
value: dsh-web
|
||||
# Cookie dsh_gw_session is HttpOnly + SameSite=Lax. Set to "true"
|
||||
# when this Service is reached over HTTPS (e.g. the optional
|
||||
# 60-ingress.yaml with TLS); "false" keeps the plain-HTTP demo
|
||||
# usable via port-forward.
|
||||
# Cookie dsh_gw_session is HttpOnly + SameSite=Lax. "false" keeps
|
||||
# the plain-HTTP demo usable through port-forward: a Secure cookie
|
||||
# is never sent over HTTP, so login would silently fail.
|
||||
# HTTPS deployments (the ingress/ variant with TLS, or
|
||||
# gateway-api-sso/, which terminates TLS at the Gateway) MUST set
|
||||
# this to "true".
|
||||
- name: COOKIE_SECURE
|
||||
value: "true"
|
||||
# Trusted reverse-proxy identity header for SSO. oauth2-proxy sets
|
||||
# X-Forwarded-User to the authenticated GitHub login on every request;
|
||||
# the gateway trusts it (and auto-provisions the user) only when this
|
||||
# is set. Remove/unset to disable SSO and fall back to bearer/session
|
||||
# auth only.
|
||||
- name: AUTH_PROXY_HEADER
|
||||
value: X-Forwarded-User
|
||||
value: "false"
|
||||
# SSO IS OPT-IN AND DISABLED HERE. With AUTH_PROXY_HEADER unset the
|
||||
# gateway trusts only its own bearer/session auth and ignores every
|
||||
# identity header. Enable it ONLY behind
|
||||
# manifests/exposure/gateway-api-sso/, where oauth2-proxy is the
|
||||
# sole route to this Service: set it to X-Forwarded-User, keep the
|
||||
# Service ClusterIP and reachable only through the proxy, and
|
||||
# terminate TLS in front of the gateway. Otherwise anyone who can
|
||||
# reach :8080 can impersonate any user by sending that header.
|
||||
# - name: AUTH_PROXY_HEADER
|
||||
# value: X-Forwarded-User
|
||||
# Reject proxied request bodies larger than this (413).
|
||||
- name: MAX_BODY_MB
|
||||
value: "32"
|
||||
135
examples/dsh-agents-as-a-service/manifests/exposure/README.md
Normal file
135
examples/dsh-agents-as-a-service/manifests/exposure/README.md
Normal file
@@ -0,0 +1,135 @@
|
||||
# Exposing the gateway
|
||||
|
||||
The base manifests deploy the gateway as a ClusterIP Service
|
||||
(`dsh-gateway:8080`) and expose nothing outside the cluster. WebSocket
|
||||
upgrades on `/u/<user>/**` are part of the contract, so the entry point must
|
||||
forward `Upgrade`/`Connection` headers (Traefik does by default).
|
||||
|
||||
There are two **mutually exclusive** exposure variants. Pick exactly one:
|
||||
|
||||
| Variant | Entry point | Authentication | TLS |
|
||||
| --- | --- | --- | --- |
|
||||
| [`ingress/60-ingress.yaml`](ingress/60-ingress.yaml) | Traefik `Ingress` | the gateway's own signup/session cookie | optional, recommended |
|
||||
| [`gateway-api-sso/`](gateway-api-sso/72-oauth2-proxy.yaml) | Gateway API `Gateway` + `HTTPRoute` | GitHub OAuth via oauth2-proxy, then the gateway cookie | required (the Gateway listener terminates it) |
|
||||
|
||||
Applying both gives the gateway two entry points, and the SSO variant is only
|
||||
safe when the gateway is reachable **exclusively** through oauth2-proxy. Do not
|
||||
mix them.
|
||||
|
||||
## Applying a variant
|
||||
|
||||
Run these commands from the example root
|
||||
(`examples/dsh-agents-as-a-service/`), the same working directory the example
|
||||
README uses.
|
||||
|
||||
`kubectl apply -f manifests/` is **not recursive**: it applies only the
|
||||
top-level numbered files and does not descend into `exposure/`. Apply the base
|
||||
resources first, then the chosen variant explicitly:
|
||||
|
||||
```bash
|
||||
# Base resources: namespace, template, warm pool, claims, gateway.
|
||||
kubectl apply -f manifests/
|
||||
|
||||
# Plain Ingress exposure:
|
||||
kubectl apply -f manifests/exposure/ingress/
|
||||
|
||||
# ... or SSO exposure (create the two Secrets and replace the example.com
|
||||
# placeholders first — see below):
|
||||
kubectl apply -f manifests/exposure/gateway-api-sso/
|
||||
```
|
||||
|
||||
**Never use `kubectl apply -R`** (recursive): it would apply both variants at
|
||||
once and hand out a second, unauthenticated path to the same gateway. Apply the
|
||||
variant directory on purpose, or the individual files by name.
|
||||
|
||||
`kubectl` only reads `.yaml`, `.yml` and `.json` files from a directory, so
|
||||
the `*.yaml.example` secret templates are skipped automatically; create the real
|
||||
Secrets imperatively (commands below) before applying the directory.
|
||||
|
||||
## `ingress/`: plain Ingress, no SSO
|
||||
|
||||
`60-ingress.yaml` routes a host through the cluster's Traefik
|
||||
(`ingressClassName: traefik`; k3s ships one) to the gateway Service.
|
||||
|
||||
1. Replace the `dsh.example.com` host and the `dsh-example-tls` Secret name
|
||||
with real values, or delete the `tls:` block to serve plain HTTP.
|
||||
2. Set `DSH_TRUSTED_HOSTS` in the SandboxTemplate to the same public host:
|
||||
the gateway forwards the browser's `Host`, and DSH rejects unknown hosts on
|
||||
`/api` (see `../10-sandboxtemplate.yaml`).
|
||||
3. Set `COOKIE_SECURE=true` on the gateway once the Ingress serves HTTPS:
|
||||
`kubectl -n dsh-demo set env deploy/dsh-gateway COOKIE_SECURE=true`. The
|
||||
base manifest keeps `false` so the plain-HTTP port-forward demo works; a
|
||||
`Secure` cookie is never sent over HTTP, which silently breaks login.
|
||||
4. Do not attach buffering or compression middlewares to this router: they
|
||||
break WebSocket upgrades and streaming.
|
||||
|
||||
Verify: `curl -sI https://<host>/healthz` returns `200`.
|
||||
|
||||
## `gateway-api-sso/`: TLS + GitHub OAuth
|
||||
|
||||
`70-gateway.yaml` defines the TLS `Gateway`, `71-httproute.yaml` sends every
|
||||
request to oauth2-proxy, and `72-oauth2-proxy.yaml` authenticates the browser
|
||||
against GitHub and proxies to the gateway with `X-Forwarded-User` set.
|
||||
|
||||
Prerequisites:
|
||||
|
||||
- Gateway API CRDs and a Gateway implementation installed;
|
||||
`gatewayClassName: traefik` matches the Traefik Gateway provider. The CRD
|
||||
version is pinned in `versions.env` (`GATEWAY_API_VERSION`).
|
||||
- The OAuth App credentials and a cookie secret, from
|
||||
`73-oauth2-proxy-secret.yaml.example`.
|
||||
- A TLS certificate/key for the public host, from
|
||||
`74-tls-secret.yaml.example`.
|
||||
- oauth2-proxy is pinned in the manifest to
|
||||
`quay.io/oauth2-proxy/oauth2-proxy:v7.15.5` (`versions.env`:
|
||||
`OAUTH2_PROXY_IMAGE`).
|
||||
|
||||
Steps:
|
||||
|
||||
```bash
|
||||
# 1. Real Secrets (the committed files are *.yaml.example templates).
|
||||
kubectl -n dsh-demo create secret generic oauth2-proxy \
|
||||
--from-literal=client-id='<GitHub OAuth App client ID>' \
|
||||
--from-literal=client-secret='<GitHub OAuth App client secret>' \
|
||||
--from-literal=cookie-secret="$(python3 -c 'import secrets; print(secrets.token_urlsafe(32))')"
|
||||
kubectl -n dsh-demo create secret tls dsh-demo-tls --cert=tls.crt --key=tls.key
|
||||
|
||||
# 2. Replace dsh-demo.example.com (70/71/72) and your-github-org (72) with
|
||||
# the real public host and allowed GitHub organization.
|
||||
|
||||
# 3. Enable SSO on the gateway and trust the public host in the template.
|
||||
kubectl -n dsh-demo set env deploy/dsh-gateway \
|
||||
AUTH_PROXY_HEADER=X-Forwarded-User COOKIE_SECURE=true
|
||||
# then set DSH_TRUSTED_HOSTS=dsh-demo.example.com in the SandboxTemplate
|
||||
# (see ../10-sandboxtemplate.yaml) and re-apply it.
|
||||
|
||||
# 4. Apply the variant.
|
||||
kubectl apply -f manifests/exposure/gateway-api-sso/
|
||||
```
|
||||
|
||||
### SSO trust boundary
|
||||
|
||||
`AUTH_PROXY_HEADER` is opt-in and deliberately unset in the base gateway
|
||||
manifest: the gateway ignores identity headers until it is set. Enabling SSO
|
||||
takes **both** sides:
|
||||
|
||||
1. `AUTH_PROXY_HEADER=X-Forwarded-User` on `deploy/dsh-gateway` (step 3
|
||||
above). Without it, oauth2-proxy's login has no effect on the gateway and
|
||||
users are still asked for the gateway's own token.
|
||||
2. The gateway Service must be reachable **only through oauth2-proxy**: keep it
|
||||
ClusterIP, route all browser traffic through the `HTTPRoute` →
|
||||
oauth2-proxy → `dsh-gateway` chain, and never apply `ingress/` (or any
|
||||
other path to the Service) while SSO is enabled. A client that can reach
|
||||
`dsh-gateway:8080` directly can set `X-Forwarded-User` itself and
|
||||
impersonate any user.
|
||||
|
||||
Also terminate TLS in front of the gateway (the Gateway listener does) and
|
||||
leave `COOKIE_SECURE=true` so the session cookie never travels in clear text.
|
||||
|
||||
## Switching variants
|
||||
|
||||
Delete the variant you had before applying the other one, e.g.
|
||||
`kubectl delete -f manifests/exposure/ingress/`, and revert
|
||||
`AUTH_PROXY_HEADER`/`COOKIE_SECURE` on the gateway. Verify with
|
||||
`kubectl -n dsh-demo get ingress,gateway,httproute` that exactly one path to
|
||||
the gateway remains.
|
||||
@@ -0,0 +1,25 @@
|
||||
# Browser entry point (TLS) for the gateway-api-sso variant: terminates TLS
|
||||
# for dsh-demo.example.com and routes through 71-httproute.yaml to
|
||||
# oauth2-proxy, which authenticates the browser before the gateway.
|
||||
# Mutually exclusive with ../../ingress/ (see ../../README.md). The
|
||||
# "dsh-demo-tls" Secret comes from 74-tls-secret.yaml.example.
|
||||
apiVersion: gateway.networking.k8s.io/v1
|
||||
kind: Gateway
|
||||
metadata:
|
||||
name: dsh-gateway
|
||||
namespace: dsh-demo
|
||||
labels:
|
||||
app.kubernetes.io/part-of: dsh-demo
|
||||
spec:
|
||||
gatewayClassName: traefik
|
||||
listeners:
|
||||
- name: https
|
||||
port: 8443
|
||||
protocol: HTTPS
|
||||
tls:
|
||||
mode: Terminate
|
||||
certificateRefs:
|
||||
- name: dsh-demo-tls
|
||||
allowedRoutes:
|
||||
namespaces:
|
||||
from: Same
|
||||
@@ -0,0 +1,23 @@
|
||||
apiVersion: gateway.networking.k8s.io/v1
|
||||
kind: HTTPRoute
|
||||
metadata:
|
||||
name: dsh-gateway
|
||||
namespace: dsh-demo
|
||||
labels:
|
||||
app.kubernetes.io/part-of: dsh-demo
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: dsh-gateway
|
||||
namespace: dsh-demo
|
||||
sectionName: https
|
||||
hostnames:
|
||||
# TODO: replace with the actual public FQDN before applying.
|
||||
- dsh-demo.example.com
|
||||
rules:
|
||||
- backendRefs:
|
||||
# Route through oauth2-proxy (GitHub OAuth) first; it authenticates the
|
||||
# browser and forwards to dsh-gateway:8080 with X-Forwarded-User set.
|
||||
# The gateway only trusts that header when AUTH_PROXY_HEADER is set on
|
||||
# its Deployment — see 72-oauth2-proxy.yaml and ../README.md.
|
||||
- name: oauth2-proxy
|
||||
port: 4180
|
||||
@@ -0,0 +1,134 @@
|
||||
# oauth2-proxy — GitHub OAuth login in front of the dsh-gateway.
|
||||
#
|
||||
# Runs as a reverse proxy: it authenticates the browser against GitHub
|
||||
# (restricted to a configured organization), then forwards the request to
|
||||
# the dsh-gateway Service with the authenticated identity in X-Forwarded-User
|
||||
# (plus X-Auth-Request-* via --set-xauthrequest). The gateway trusts that
|
||||
# header only when AUTH_PROXY_HEADER is set on its Deployment.
|
||||
#
|
||||
# ENABLING SSO TAKES BOTH SIDES (see ../../../exposure/README.md):
|
||||
# 1. Set AUTH_PROXY_HEADER=X-Forwarded-User on deploy/dsh-gateway, otherwise
|
||||
# the gateway ignores the header and the login is cosmetic:
|
||||
# kubectl -n dsh-demo set env deploy/dsh-gateway \
|
||||
# AUTH_PROXY_HEADER=X-Forwarded-User COOKIE_SECURE=true
|
||||
# 2. Make dsh-gateway reachable ONLY through this proxy: keep the Service
|
||||
# ClusterIP, route browser traffic through 71-httproute.yaml, and do not
|
||||
# apply ../ingress/ (or anything else that reaches the gateway). A
|
||||
# client that can reach dsh-gateway:8080 can spoof X-Forwarded-User and
|
||||
# impersonate any user.
|
||||
# TLS terminates at the Gateway listener (70-gateway.yaml); keep
|
||||
# --cookie-secure=true and COOKIE_SECURE=true on the gateway.
|
||||
#
|
||||
# The image is pinned to quay.io/oauth2-proxy/oauth2-proxy:v7.15.5
|
||||
# (versions.env: OAUTH2_PROXY_IMAGE). Credentials come from the
|
||||
# "oauth2-proxy" Secret (see 73-oauth2-proxy-secret.yaml.example); this
|
||||
# manifest never carries secrets.
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: oauth2-proxy
|
||||
namespace: dsh-demo
|
||||
labels:
|
||||
app: oauth2-proxy
|
||||
app.kubernetes.io/part-of: dsh-demo
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: oauth2-proxy
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: oauth2-proxy
|
||||
app.kubernetes.io/part-of: dsh-demo
|
||||
spec:
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 2000
|
||||
runAsGroup: 2000
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
containers:
|
||||
- name: oauth2-proxy
|
||||
image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.5
|
||||
imagePullPolicy: IfNotPresent
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop: ["ALL"]
|
||||
args:
|
||||
- --http-address=0.0.0.0:4180
|
||||
- --provider=github
|
||||
# TODO: replace with the GitHub organization allowed to sign in.
|
||||
- --github-org=your-github-org
|
||||
- --email-domain=*
|
||||
# Reverse-proxy target: the gateway backend Service.
|
||||
- --upstream=http://dsh-gateway:8080
|
||||
# Public callback (TLS terminates at the Gateway listener).
|
||||
# TODO: replace with the actual public FQDN.
|
||||
- --redirect-url=https://dsh-demo.example.com/oauth2/callback
|
||||
- --cookie-secure=true
|
||||
- --cookie-name=_oauth2_proxy
|
||||
# Also expose X-Auth-Request-User/Email/Groups upstream.
|
||||
- --set-xauthrequest
|
||||
# Single provider: skip the provider-selection page.
|
||||
- --skip-provider-button
|
||||
env:
|
||||
- name: OAUTH2_PROXY_CLIENT_ID
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: oauth2-proxy
|
||||
key: client-id
|
||||
- name: OAUTH2_PROXY_CLIENT_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: oauth2-proxy
|
||||
key: client-secret
|
||||
- name: OAUTH2_PROXY_COOKIE_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: oauth2-proxy
|
||||
key: cookie-secret
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 4180
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /ping
|
||||
port: http
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 1
|
||||
failureThreshold: 3
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /ready
|
||||
port: http
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
failureThreshold: 3
|
||||
resources:
|
||||
requests:
|
||||
cpu: 10m
|
||||
memory: 32Mi
|
||||
limits:
|
||||
cpu: 200m
|
||||
memory: 128Mi
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: oauth2-proxy
|
||||
namespace: dsh-demo
|
||||
labels:
|
||||
app: oauth2-proxy
|
||||
app.kubernetes.io/part-of: dsh-demo
|
||||
spec:
|
||||
selector:
|
||||
app: oauth2-proxy
|
||||
ports:
|
||||
- name: http
|
||||
port: 4180
|
||||
targetPort: http
|
||||
@@ -0,0 +1,39 @@
|
||||
# EXAMPLE ONLY — do not apply as-is.
|
||||
#
|
||||
# oauth2-proxy reads its GitHub OAuth credentials and cookie-signing secret from
|
||||
# this Secret. Populate real values and apply:
|
||||
#
|
||||
# kubectl -n dsh-demo create secret generic oauth2-proxy \
|
||||
# --from-literal=client-id='<GitHub OAuth App client ID>' \
|
||||
# --from-literal=client-secret='<GitHub OAuth App client secret>' \
|
||||
# --from-literal=cookie-secret="$(python3 -c 'import secrets; print(secrets.token_urlsafe(32))')"
|
||||
#
|
||||
# The cookie secret must decode (base64) to exactly 16, 24, or 32 bytes.
|
||||
# oauth2-proxy base64-decodes it, so it must be URL-safe, unpadded base64.
|
||||
# Generate it with either of:
|
||||
#
|
||||
# python3 -c 'import secrets; print(secrets.token_urlsafe(32))'
|
||||
# openssl rand -base64 32 | tr -- '+/' '-_' | tr -d '='
|
||||
#
|
||||
# Do NOT use plain `openssl rand -base64 32`: that yields 44 characters of
|
||||
# standard base64 (with "=" and possibly "+"/"/"), which oauth2-proxy cannot
|
||||
# decode and therefore treats as 44 raw bytes, failing with "cookie_secret must
|
||||
# be 16, 24, or 32 bytes ... but is 44 bytes".
|
||||
#
|
||||
# GitHub OAuth App (https://github.com/settings/developers):
|
||||
# Homepage URL https://dsh-demo.example.com
|
||||
# Authorization callback https://dsh-demo.example.com/oauth2/callback
|
||||
# (replace dsh-demo.example.com with the actual public FQDN)
|
||||
#
|
||||
# NOTE: pass the base64 value as-is via --from-literal (kubectl stores the
|
||||
# literal; it does not base64-encode it a second time).
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: oauth2-proxy
|
||||
namespace: dsh-demo
|
||||
type: Opaque
|
||||
stringData:
|
||||
client-id: "<github-oauth-app-client-id>"
|
||||
client-secret: "<github-oauth-app-client-secret>"
|
||||
cookie-secret: "<urlsafe-unpadded-base64-of-32-random-bytes>"
|
||||
@@ -0,0 +1,17 @@
|
||||
# EXAMPLE ONLY — do not apply as-is.
|
||||
#
|
||||
# Copy to tls-secret.yaml and populate with real base64-encoded PEM material,
|
||||
# then apply. The secret must live in the dsh-demo namespace and be named
|
||||
# "dsh-demo-tls" (referenced by 70-gateway.yaml).
|
||||
#
|
||||
# kubectl -n dsh-demo create secret tls dsh-demo-tls \
|
||||
# --cert=tls.crt --key=tls.key --dry-run=client -o yaml > tls-secret.yaml
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: dsh-demo-tls
|
||||
namespace: dsh-demo
|
||||
type: kubernetes.io/tls
|
||||
data:
|
||||
tls.crt: <base64-encoded-certificate>
|
||||
tls.key: <base64-encoded-private-key>
|
||||
@@ -1,15 +1,21 @@
|
||||
# OPTIONAL browser entry point: expose the gateway through Traefik.
|
||||
#
|
||||
# Requires 50-gateway.yaml (Service dsh-gateway:8080) and a Traefik install
|
||||
# with an IngressClass named "traefik" (k3s ships one). Replace the placeholder
|
||||
# host below with a name that resolves to your ingress address, and either
|
||||
# supply a real TLS Secret or delete the `tls:` block for plain HTTP.
|
||||
# This is the "no SSO" exposure variant and is MUTUALLY EXCLUSIVE with
|
||||
# ../gateway-api-sso/ — apply exactly one of them (see ../README.md).
|
||||
#
|
||||
# Requires the base manifests (Service dsh-gateway:8080 from
|
||||
# manifests/50-gateway.yaml) and a Traefik install with an IngressClass named
|
||||
# "traefik" (k3s ships one). Replace the placeholder host below with a name
|
||||
# that resolves to your ingress address, and either supply a real TLS Secret or
|
||||
# delete the `tls:` block for plain HTTP.
|
||||
#
|
||||
# Two things must match this host:
|
||||
# 1. SandboxTemplate (10-sandboxtemplate.yaml): set DSH_TRUSTED_HOSTS to
|
||||
# this host so DSH accepts the forwarded Host/Origin on /api.
|
||||
# 2. Gateway Deployment (50-gateway.yaml): set COOKIE_SECURE to "true"
|
||||
# once this Ingress serves HTTPS, so the session cookie is Secure.
|
||||
# 1. SandboxTemplate (manifests/10-sandboxtemplate.yaml): set
|
||||
# DSH_TRUSTED_HOSTS to this host so DSH accepts the forwarded Host/Origin
|
||||
# on /api.
|
||||
# 2. Gateway Deployment (manifests/50-gateway.yaml): set COOKIE_SECURE to
|
||||
# "true" once this Ingress serves HTTPS, so the session cookie is Secure.
|
||||
# The base manifest sets "false" for the plain-HTTP port-forward demo.
|
||||
#
|
||||
# WebSocket support is REQUIRED: the browser talks to /u/<user>/... and DSH
|
||||
# uses WebSockets for live session traffic. Traefik upgrades WS/SSE
|
||||
@@ -1,13 +1,37 @@
|
||||
#!/usr/bin/env bash
|
||||
# Smoke test for the dsh-agents-as-a-service example: runs the full README
|
||||
# walkthrough non-interactively against the current kubectl context.
|
||||
# Expects agent-sandbox (with extensions) already installed.
|
||||
# End-to-end smoke test for the dsh-agents-as-a-service example: runs the
|
||||
# README walkthrough non-interactively against the current kubectl context.
|
||||
#
|
||||
# Prerequisites:
|
||||
# - A kind cluster whose name matches the current kubectl context, or set
|
||||
# KIND_CLUSTER=<name>.
|
||||
# - agent-sandbox with extensions installed (API v1beta1). The release
|
||||
# manifest is pinned by AGENT_SANDBOX_VERSION in versions.env:
|
||||
# . ../../../versions.env
|
||||
# kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/download/${AGENT_SANDBOX_VERSION}/sandbox-with-extensions.yaml
|
||||
# - docker, to build the gateway image (skipped with SKIP_GATEWAY=1).
|
||||
#
|
||||
# Environment:
|
||||
# SKIP_GATEWAY=1 run only the resource-layer checks (no gateway build/load,
|
||||
# no signup/proxy/WebSocket/idle/wake checks)
|
||||
# KIND_CLUSTER kind cluster name for "kind load docker-image"
|
||||
# (default: derived from the current kubectl context)
|
||||
# GW_LOCAL_IMAGE local gateway image tag
|
||||
# (default: dsh-aaas-gateway:e2e)
|
||||
#
|
||||
# The gateway is built from the working tree (gateway/) and loaded into kind,
|
||||
# so this tests the example as checked out, not the published image.
|
||||
#
|
||||
# The script is destructive only inside the dsh-demo namespace: it creates the
|
||||
# namespace (and a throwaway provider key) and DELETES it on exit via the
|
||||
# cleanup trap.
|
||||
#
|
||||
# NOTE: the only local port bound is the gateway port-forward on 18080.
|
||||
# The DSH Web container port (3080) is in-cluster only and is never bound
|
||||
# on the local host.
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")"
|
||||
# Example root: paths below are manifests/... and gateway/... from here.
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
NS=dsh-demo
|
||||
pass() { echo "PASS: $1"; }
|
||||
@@ -20,12 +44,12 @@ cleanup() {
|
||||
trap cleanup EXIT
|
||||
|
||||
echo "=== 1. install prereqs (throwaway key) + template + warm pool"
|
||||
kubectl apply -f 00-prereqs.yaml
|
||||
kubectl apply -f manifests/00-prereqs.yaml
|
||||
# The provider key is never committed; generate a throwaway value.
|
||||
kubectl -n "$NS" delete secret dsh-provider-keys --ignore-not-found >/dev/null
|
||||
kubectl -n "$NS" create secret generic dsh-provider-keys \
|
||||
--from-literal=DEEPSEEK_API_KEY="sk-test-$(openssl rand -hex 16)"
|
||||
kubectl apply -f 10-sandboxtemplate.yaml -f 20-sandboxwarmpool.yaml
|
||||
kubectl apply -f manifests/10-sandboxtemplate.yaml -f manifests/20-sandboxwarmpool.yaml
|
||||
for _ in $(seq 1 120); do
|
||||
READY=$(kubectl -n "$NS" get sandboxes -l agents.x-k8s.io/warm-pool-sandbox \
|
||||
-o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' | grep -c True || true)
|
||||
@@ -37,7 +61,7 @@ pass "warm pool: 2 spares Ready"
|
||||
|
||||
echo "=== 2. signup via claim (warm adoption)"
|
||||
T0=$(date +%s)
|
||||
kubectl apply -f 30-claim-alice.yaml
|
||||
kubectl apply -f manifests/30-claim-alice.yaml
|
||||
kubectl -n "$NS" wait sandboxclaim dsh-alice --for=condition=Ready --timeout=60s
|
||||
ELAPSED=$(( $(date +%s) - T0 ))
|
||||
SB=$(kubectl -n "$NS" get sandboxclaim dsh-alice -o jsonpath='{.status.sandbox.name}')
|
||||
@@ -65,7 +89,7 @@ MARKER=$(kubectl -n "$NS" exec "$POD" -- cat /home/node/.dsh/marker.txt)
|
||||
pass "resumed: state survived on the reattached PVC"
|
||||
|
||||
echo "=== 6. injection policy rejects env-carrying claims"
|
||||
kubectl apply -f 40-claim-rejected.yaml
|
||||
kubectl apply -f manifests/40-claim-rejected.yaml
|
||||
# Wait for the POSITIVE rejection signal, not merely "not bound yet".
|
||||
REASON=""
|
||||
for _ in $(seq 1 12); do
|
||||
@@ -78,19 +102,25 @@ done
|
||||
|| fail "claim was not rejected (Ready reason='$REASON')"
|
||||
BOUND=$(kubectl -n "$NS" get sandboxclaim dsh-mallory -o jsonpath='{.status.sandbox.name}' 2>/dev/null || true)
|
||||
[ -z "$BOUND" ] || fail "env-carrying claim unexpectedly bound sandbox '$BOUND'"
|
||||
kubectl delete -f 40-claim-rejected.yaml >/dev/null
|
||||
kubectl delete -f manifests/40-claim-rejected.yaml >/dev/null
|
||||
pass "claim rejected with EnvVarsInjectionRejected and never bound"
|
||||
|
||||
echo "=== 7. gateway: browser signup, proxy, idle-suspend, wake-on-connect"
|
||||
if command -v docker >/dev/null && command -v kind >/dev/null; then
|
||||
KIND_CLUSTER="${KIND_CLUSTER:-$(kubectl config current-context | sed 's/^kind-//')}"
|
||||
# Tag exactly what 50-gateway.yaml references so the smoke test exercises
|
||||
# the shipped manifest unchanged (imagePullPolicy: IfNotPresent uses the
|
||||
# image loaded into kind).
|
||||
GW_IMAGE=ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway
|
||||
docker build -q -t "$GW_IMAGE" gateway/ >/dev/null
|
||||
kind load docker-image "$GW_IMAGE" --name "$KIND_CLUSTER" >/dev/null
|
||||
kubectl apply -f 50-gateway.yaml
|
||||
GW_LOCAL_IMAGE="${GW_LOCAL_IMAGE:-dsh-aaas-gateway:e2e}"
|
||||
# Test the working tree, not the published image: build the gateway from
|
||||
# gateway/, load it into kind, then point the Deployment at that tag. The
|
||||
# shipped manifest pins the release image; imagePullPolicy: IfNotPresent
|
||||
# makes the node use the loaded one.
|
||||
docker build -q -t "$GW_LOCAL_IMAGE" gateway/ >/dev/null
|
||||
kind load docker-image "$GW_LOCAL_IMAGE" --name "$KIND_CLUSTER" >/dev/null
|
||||
kubectl apply -f manifests/50-gateway.yaml
|
||||
kubectl -n "$NS" set image deploy/dsh-gateway gateway="$GW_LOCAL_IMAGE"
|
||||
# Override the demo timeouts from manifests/50-gateway.yaml (360/240) with the
|
||||
# gateway's working-tree defaults, so idle-suspend and wake-on-connect are
|
||||
# exercised in seconds instead of minutes.
|
||||
kubectl -n "$NS" set env deploy/dsh-gateway IDLE_TIMEOUT=60 WAKE_TIMEOUT=120
|
||||
kubectl -n "$NS" rollout status deploy/dsh-gateway --timeout=120s
|
||||
# Local port-forward uses 18080 (NOT 3080, which stays in-cluster).
|
||||
kubectl -n "$NS" port-forward svc/dsh-gateway 18080:8080 >/dev/null 2>&1 &
|
||||
127
examples/minimal-single-tenant/README.md
Normal file
127
examples/minimal-single-tenant/README.md
Normal file
@@ -0,0 +1,127 @@
|
||||
# Minimal single-tenant DSH Web
|
||||
|
||||
The smallest Kubernetes deployment of DeepSeek Harness (DSH) Web: one
|
||||
`SandboxTemplate`, a one-replica warm pool and one claim. It is the "just one
|
||||
DSH Web instance" quickstart.
|
||||
|
||||
For the full platform — warm pool sizing, per-user claims, a browser-facing
|
||||
gateway with cookie sessions, idle-suspend and wake-on-connect — see
|
||||
[`examples/dsh-agents-as-a-service/`](../dsh-agents-as-a-service/README.md).
|
||||
This example deliberately has no gateway: you reach DSH through a
|
||||
`kubectl port-forward` and its launch-token URL.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- A Kubernetes cluster with a default StorageClass (kind and GKE both work).
|
||||
- The agent-sandbox controller with extensions (API `v1beta1`) installed.
|
||||
The version is pinned as `AGENT_SANDBOX_VERSION` in
|
||||
[`versions.env`](../../versions.env):
|
||||
|
||||
```bash
|
||||
# from the repository root:
|
||||
. versions.env
|
||||
kubectl apply -f "https://github.com/kubernetes-sigs/agent-sandbox/releases/download/${AGENT_SANDBOX_VERSION}/sandbox-with-extensions.yaml"
|
||||
kubectl -n agent-sandbox-system rollout status deployment --timeout=180s
|
||||
```
|
||||
|
||||
- A DeepSeek API key for real model calls. The pod starts without one; only
|
||||
model requests fail.
|
||||
|
||||
## Quickstart
|
||||
|
||||
Run these from this example's directory (`examples/minimal-single-tenant/`).
|
||||
|
||||
```bash
|
||||
# 1. Namespace + shared provider key. The key is injected into the sandbox via
|
||||
# envFrom and is never committed.
|
||||
kubectl create namespace dsh-demo
|
||||
kubectl -n dsh-demo create secret generic dsh-provider-keys \
|
||||
--from-literal=DEEPSEEK_API_KEY="sk-..."
|
||||
|
||||
# 2. The template itself (image pinned in the manifest).
|
||||
kubectl apply -f manifests/sandbox-template.yaml
|
||||
|
||||
# 3. A template only describes pods; a warm pool turns it into running
|
||||
# Sandboxes and a claim adopts one. One replica is enough here.
|
||||
kubectl apply -f - <<'EOF'
|
||||
apiVersion: extensions.agents.x-k8s.io/v1beta1
|
||||
kind: SandboxWarmPool
|
||||
metadata:
|
||||
name: dsh-pool
|
||||
namespace: dsh-demo
|
||||
spec:
|
||||
replicas: 1
|
||||
sandboxTemplateRef:
|
||||
name: dsh-web-template
|
||||
---
|
||||
apiVersion: extensions.agents.x-k8s.io/v1beta1
|
||||
kind: SandboxClaim
|
||||
metadata:
|
||||
name: dsh
|
||||
namespace: dsh-demo
|
||||
spec:
|
||||
warmPoolRef:
|
||||
name: dsh-pool
|
||||
EOF
|
||||
|
||||
kubectl -n dsh-demo wait sandboxclaim/dsh --for=condition=Ready --timeout=180s
|
||||
```
|
||||
|
||||
## Talk to the instance
|
||||
|
||||
DSH gates `/` behind a launch token it prints only in its startup logs. Open
|
||||
the printed URL once; DSH then sets its own auth cookie and the UI loads
|
||||
normally.
|
||||
|
||||
```bash
|
||||
SB=$(kubectl -n dsh-demo get sandboxclaim dsh -o jsonpath='{.status.sandbox.name}')
|
||||
kubectl -n dsh-demo logs "$SB" | grep -m1 '?token='
|
||||
# dsh web: http://<pod-ip>:3080/?token=...
|
||||
|
||||
kubectl -n dsh-demo port-forward "pod/$SB" 18081:3080
|
||||
# open http://localhost:18081/?token=... from the log line
|
||||
```
|
||||
|
||||
> **Port note:** DSH Web listens on `3080` in-cluster. If `3080` is already in
|
||||
> use on your workstation, forward to another local port (here `18081`), never
|
||||
> `3080`. `localhost` is always in DSH's trusted-host list, so the forwarded
|
||||
> request is accepted.
|
||||
|
||||
## Probes and trusted hosts
|
||||
|
||||
Two settings in `manifests/sandbox-template.yaml` are easy to get wrong:
|
||||
|
||||
- **Probe path.** `/` returns `401` without the launch-token cookie, so a probe
|
||||
there crash-loops the pod. The template probes `/favicon.svg` (served
|
||||
unauthenticated) and gates liveness/readiness behind a `startupProbe`, which
|
||||
gives a cold start (plugin activation, or the first resume) up to 120s before
|
||||
a restart.
|
||||
- **Host allow-list.** DSH rejects unknown `Host` headers. `localhost` and
|
||||
`127.0.0.1` are always trusted; if you later front the sandbox with an
|
||||
Ingress or proxy, set `DSH_TRUSTED_HOSTS` to that public host (comma or space
|
||||
separated). The full example explains both exposure variants in
|
||||
[`manifests/exposure/README.md`](../dsh-agents-as-a-service/manifests/exposure/README.md).
|
||||
|
||||
The template also sets `fsGroupChangePolicy: OnRootMismatch`. DSH's
|
||||
credentials plugin requires `.credentials.yaml` to be `0600`, while the
|
||||
kubelet's default fsGroup policy rewrites it to `0660` on every pod start —
|
||||
without this policy a recreated or resumed sandbox crash-loops. The rationale
|
||||
is spelled out in
|
||||
[`10-sandboxtemplate.yaml`](../dsh-agents-as-a-service/manifests/10-sandboxtemplate.yaml).
|
||||
|
||||
## Suspend, resume and clean up
|
||||
|
||||
```bash
|
||||
# Suspend: the pod is deleted, the PVC (state) and Service are kept.
|
||||
kubectl -n dsh-demo patch sandbox "$SB" --type merge -p '{"spec":{"operatingMode":"Suspended"}}'
|
||||
|
||||
# Resume: a new pod reattaches the same PVC; conversations survive.
|
||||
kubectl -n dsh-demo patch sandbox "$SB" --type merge -p '{"spec":{"operatingMode":"Running"}}'
|
||||
kubectl -n dsh-demo wait sandbox "$SB" --for=condition=Ready --timeout=180s
|
||||
|
||||
# Delete everything this quickstart created.
|
||||
kubectl -n dsh-demo delete sandboxclaim dsh
|
||||
kubectl delete namespace dsh-demo
|
||||
```
|
||||
|
||||
Deleting the claim cascades to the sandbox and its PVC.
|
||||
@@ -0,0 +1,98 @@
|
||||
# Minimal single-tenant SandboxTemplate for DeepSeek Harness Web.
|
||||
#
|
||||
# This is the "just one DSH Web" quick-start template. For the full
|
||||
# multi-tenant platform (warm pool, per-user claims, gateway, idle-suspend),
|
||||
# see examples/dsh-agents-as-a-service/.
|
||||
#
|
||||
# Claims reference this template directly (spec.sandboxTemplateRef): there is
|
||||
# no warm pool and no per-user platform here. See the README for the full
|
||||
# quickstart; in short the prerequisites are:
|
||||
# - Namespace dsh-demo.
|
||||
# - Secret dsh-provider-keys in dsh-demo with key DEEPSEEK_API_KEY
|
||||
# (injected via envFrom; the pod starts without it, only model calls fail).
|
||||
# The agent-sandbox controller (v1.0.7, API v1beta1) must be installed.
|
||||
apiVersion: extensions.agents.x-k8s.io/v1beta1
|
||||
kind: SandboxTemplate
|
||||
metadata:
|
||||
name: dsh-web-template
|
||||
namespace: dsh-demo
|
||||
spec:
|
||||
service: true # headless Service per Sandbox -> stable DNS (status.serviceFQDN)
|
||||
podTemplate:
|
||||
spec:
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 1000 # Matches the UID of the 'node' user
|
||||
runAsGroup: 1000
|
||||
fsGroup: 1000
|
||||
# DSH's credentials plugin refuses to start when .credentials.yaml has
|
||||
# any group/other permission bit (it creates the file 0600). The
|
||||
# default kubelet fsGroup policy ("Always") rewrites it to 0660 on every
|
||||
# pod start, so a recreated or resumed sandbox crash-loops; OnRootMismatch
|
||||
# avoids that walk once the volume root already carries the ownership.
|
||||
# Rationale in full at
|
||||
# examples/dsh-agents-as-a-service/manifests/10-sandboxtemplate.yaml.
|
||||
fsGroupChangePolicy: OnRootMismatch
|
||||
containers:
|
||||
- name: dsh-web
|
||||
image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.2-r2
|
||||
imagePullPolicy: IfNotPresent
|
||||
ports:
|
||||
- name: web
|
||||
containerPort: 3080
|
||||
envFrom:
|
||||
- secretRef:
|
||||
name: dsh-provider-keys # Secret with key DEEPSEEK_API_KEY
|
||||
env:
|
||||
# Empty => DSH derives links from the request Host. Keep it empty
|
||||
# when fronting the sandbox with a proxy/gateway.
|
||||
- name: DSH_PUBLIC_URL
|
||||
value: ""
|
||||
# DSH Host/Origin-fences /api. Empty trusts only localhost, which
|
||||
# is right for `kubectl port-forward`. Set this to the public host
|
||||
# (e.g. dsh.example.com) if you front DSH with a proxy or Ingress,
|
||||
# so requests forwarded with the browser's Host are accepted. The
|
||||
# full example documents both exposure variants in
|
||||
# examples/dsh-agents-as-a-service/manifests/exposure/README.md.
|
||||
- name: DSH_TRUSTED_HOSTS
|
||||
value: ""
|
||||
volumeMounts:
|
||||
- name: dsh-home
|
||||
mountPath: /home/node/.dsh # DSH_HOME
|
||||
# / is behind DSH's launch-token cookie (401 without it); use an
|
||||
# unauthenticated path for probes. The startup probe gives a cold
|
||||
# start (plugin activation on a small node, or the first resume of a
|
||||
# suspended sandbox) up to 120s before liveness can restart it.
|
||||
startupProbe:
|
||||
httpGet:
|
||||
path: /favicon.svg
|
||||
port: 3080
|
||||
periodSeconds: 5
|
||||
failureThreshold: 24
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /favicon.svg
|
||||
port: 3080
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /favicon.svg
|
||||
port: 3080
|
||||
initialDelaySeconds: 15
|
||||
periodSeconds: 15
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
capabilities:
|
||||
drop: ["ALL"]
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
volumeClaimTemplates:
|
||||
- metadata:
|
||||
name: dsh-home
|
||||
spec:
|
||||
accessModes: ["ReadWriteOnce"]
|
||||
resources:
|
||||
requests:
|
||||
storage: 5Gi
|
||||
# storageClassName omitted => cluster default
|
||||
Reference in New Issue
Block a user