nicolas-georger/deepseek-kubernetes-sandbox
1
0
Fork 0
mirror of https://github.com/sredevopsorg/deepseek-kubernetes-sandbox.git synced 2026-11-03 01:13:24 +00:00
Code Projects Releases Activity

Make DSH agents-as-a-service browser-usable; fix probe crash-loop (#3)

Browse Source 
* Rewrite DSH gateway for browser sessions, HTTP and WebSocket

- Replace the Flask proxy with a Starlette ASGI app (uvicorn factory),
  keeping the control plane (claims, wake/suspend, idle sweeper).
- Add HttpOnly gateway sessions and a server-side DSH launch-token
  handshake: read the token from pod logs, exchange it for a dsh-auth
  cookie, cache per (user, pod UID, Host), refresh once on 401.
- Proxy HTTP/SSE and WebSocket (/api/remote.mux) under /u/<user>/, with
  header stripping/injection, Location rewriting and wake-on-connect.
- Add a landing/signup/login page, health endpoint, pinned dependencies,
  a read-only-rootfs gateway image, 87 cluster-free tests and docs.

* Fix DSH probe crash-loop and harden gateway manifests

- Probe /favicon.svg instead of /: DSH gates / behind its launch-token
  cookie and returns 401, which made warm-pool pods crash-loop.
- Pass DSH_TRUSTED_HOSTS through the entrypoint as --trusted-host args
  for public/ingress hosts; keep localhost trusted.
- Deploy the published :gateway image, add pods/pods-log RBAC, /healthz
  probes, read-only rootfs with a /tmp emptyDir and non-root security.
- Add optional Traefik Ingress and document the prerequisites.

* Update CI, smoke tests, build tooling and docs

- Run gateway pytest/ruff and build/smoke-test the image in CI.
- Parameterize the DSH build version and pnpm version inputs.
- Exercise cookie signup, HTML proxy, WebSocket auth, suspend/wake and
  delete in run-test-kind.sh.
- Add Makefile/build.sh targets and document the architecture, handshake,
  probes, trusted hosts, troubleshooting and limitations.

---------

Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
This commit is contained in:
SREDevOps.org Agent
2026-10-21 12:10:03 -03:00
committed by GitHub
parent c9be929212
commit 6301c21bea
23 changed files with 3700 additions and 660 deletions
Download Patch File Download Diff File Expand all files Collapse all files

25
.env.sample
View File

@@ -1,4 +1,29 @@
# Copy to .env and fill in. Never commit a real key.
# --- DSH Web image (Dockerfile / build.sh) ---
DEEPSEEK_API_KEY= DEEPSEEK_API_KEY=
DSH_VERSION=dsh-v0.2.1-rc.2 DSH_VERSION=dsh-v0.2.1-rc.2
PNPM_VERSION=11.28.5 PNPM_VERSION=11.28.5
# DSH Web port inside the container, and the local port published by
# build.sh / docker run. The gateway also uses it as the upstream port.
DSH_PORT=3080 DSH_PORT=3080
# Comma/space separated extra Host allow-list entries for DSH (for an
# Ingress/public hostname). localhost and 127.0.0.1 are always trusted.
DSH_TRUSTED_HOSTS=
# --- Gateway (examples/dsh-agents-as-a-service) ---
GATEWAY_IMAGE=ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway
NAMESPACE=dsh-demo
POOL=dsh-pool
# Seconds a sandbox may stay idle before the sweeper suspends it.
IDLE_TIMEOUT=60
# Seconds a waking request may wait for the sandbox to become Ready.
WAKE_TIMEOUT=120
DSH_CONTAINER=dsh-web
DSH_LOG_TAIL_LINES=2000
COOKIE_NAME=dsh_gw_session
# Set to true only behind HTTPS; secure cookies are not sent over plain HTTP.
COOKIE_SECURE=false
COOKIE_MAX_AGE=604800
MAX_BODY_MB=32
HTTP_TIMEOUT=300

10
.github/workflows/multi-build-gateway.yaml vendored
View File

@@ -1,5 +1,7 @@
# This workflow builds a multi-arch Docker image using GitHub Actions and separated Github Runners with native support for ARM64 and AMD64 architectures, without using QEMU emulation. # This workflow builds the multi-arch gateway image (Starlette ASGI, served by
# It uses Docker Buildx to build and push the image to GitHub Container Registry (GHCR). # uvicorn on :8080) and pushes it to GHCR as
# ghcr.io/<owner>/<repo>:gateway. The DSH Web image is built by multi-build.yaml.
# It uses Docker Buildx with native AMD64 and ARM64 runners, without QEMU.
name: Gateway Build multi arch Docker Image with separate Github Runners name: Gateway Build multi arch Docker Image with separate Github Runners
on: on:
@@ -11,9 +13,7 @@ on:
- 'examples/dsh-agents-as-a-service/gateway/**' - 'examples/dsh-agents-as-a-service/gateway/**'
- '.github/workflows/multi-build-gateway.yaml' - '.github/workflows/multi-build-gateway.yaml'
env: env:
# The name of the Docker image to be built and pushed to GHCR # ghcr.io/<owner>/<repo>; this workflow always publishes the `:gateway` tag.
# The image name is derived from the GitHub repository name and the GitHub Container Registry (GHCR) URL.
# The image name will be in the format: ghcr.io/<owner>/<repo>
GHCR_IMAGE: ghcr.io/${{ github.repository }} GHCR_IMAGE: ghcr.io/${{ github.repository }}
permissions: permissions:
# Global permissions for the workflow, which can be overridden at the job level # Global permissions for the workflow, which can be overridden at the job level

26
.github/workflows/multi-build.yaml vendored
View File

@@ -4,6 +4,17 @@ name: Build multi arch Docker Image with separate Github Runners
on: on:
workflow_dispatch: workflow_dispatch:
inputs:
dsh_version:
description: "DSH git ref to build (tag or branch)"
required: true
default: dsh-v0.2.1-rc.2
type: string
pnpm_version:
description: "pnpm version used to build DSH"
required: true
default: "11.28.5"
type: string
push: push:
branches: branches:
- main - main
@@ -15,7 +26,10 @@ env:
# The image name is derived from the GitHub repository name and the GitHub Container Registry (GHCR) URL. # The image name is derived from the GitHub repository name and the GitHub Container Registry (GHCR) URL.
# The image name will be in the format: ghcr.io/<owner>/<repo> # The image name will be in the format: ghcr.io/<owner>/<repo>
GHCR_IMAGE: ghcr.io/${{ github.repository }} GHCR_IMAGE: ghcr.io/${{ github.repository }}
DSH_VERSION: dsh-v0.2.1-rc.2 #TODO: Make this configurable from workflow inputs and/or an custom Github Action # workflow_dispatch inputs win; pushes to main fall back to the pinned
# defaults. Both values flow into the Docker build args and the image tags.
DSH_VERSION: ${{ inputs.dsh_version || 'dsh-v0.2.1-rc.2' }}
PNPM_VERSION: ${{ inputs.pnpm_version || '11.28.5' }}
permissions: permissions:
# Global permissions for the workflow, which can be overridden at the job level # Global permissions for the workflow, which can be overridden at the job level
contents: read contents: read
@@ -130,14 +144,14 @@ jobs:
uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7.4.0 uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7.4.0
env: env:
DOCKER_BUILDKIT: 1 DOCKER_BUILDKIT: 1
DSH_VERSION: dsh-v0.2.1-rc.2 #TODO: Make this configurable from workflow inputs and/or an custom Github Action
PNPM_VERSION: 11.28.5
DSH_PORT: 3080 DSH_PORT: 3080
with: with:
context: . context: .
platforms: ${{ matrix.platform }} platforms: ${{ matrix.platform }}
labels: ${{ steps.meta.outputs.labels }} labels: ${{ steps.meta.outputs.labels }}
build-args: DSH_VERSION=${{ env.DSH_VERSION }} build-args: |
DSH_VERSION=${{ env.DSH_VERSION }}
PNPM_VERSION=${{ env.PNPM_VERSION }}
annotations: ${{ steps.meta.outputs.annotations }} annotations: ${{ steps.meta.outputs.annotations }}
outputs: type=image,name=${{ env.GHCR_IMAGE }},push-by-digest=true,name-canonical=true,push=true,oci-mediatypes=true outputs: type=image,name=${{ env.GHCR_IMAGE }},push-by-digest=true,name-canonical=true,push=true,oci-mediatypes=true
cache-from: type=gha,scope=${{ github.repository }}-${{ github.ref_name }}-${{ matrix.platform }} cache-from: type=gha,scope=${{ github.repository }}-${{ github.ref_name }}-${{ matrix.platform }}
@@ -209,8 +223,6 @@ jobs:
# The metadata includes information such as the image name, tags, and labels. # The metadata includes information such as the image name, tags, and labels.
id: meta id: meta
uses: docker/metadata-action@dc802804100637a589fabce1cb79ff13a1411302 # v6.2.0 uses: docker/metadata-action@dc802804100637a589fabce1cb79ff13a1411302 # v6.2.0
env:
DSH_VERSION: ${{ env.DSH_VERSION }}
with: with:
images: ${{ env.GHCR_IMAGE }} images: ${{ env.GHCR_IMAGE }}
annotations: | annotations: |
@@ -218,7 +230,7 @@ jobs:
tags: | tags: |
type=raw,value=main,enable=${{ github.ref_name == 'main' }} type=raw,value=main,enable=${{ github.ref_name == 'main' }}
type=raw,value=latest,enable=${{ github.ref_name == 'main' }} type=raw,value=latest,enable=${{ github.ref_name == 'main' }}
type=raw,value=${{ env.DSH_VERSION }},enable=${{ github.ref_name == 'main' }} type=raw,value=${{ env.DSH_VERSION }},enable=${{ github.ref_name == 'main' || github.event_name == 'workflow_dispatch' }}
- name: Set up Docker Buildx - name: Set up Docker Buildx
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1 uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1

80
.github/workflows/validate.yaml vendored
View File

@@ -1,8 +1,9 @@
name: Validate name: Validate
# Validates the Kubernetes manifests against the real agent-sandbox CRD # Validates the Kubernetes manifests against the real agent-sandbox CRD
# schema (server-side dry-run on a throwaway kind cluster) and runs the # schema (server-side dry-run on a throwaway kind cluster), runs the gateway
# gateway unit tests. Runs on every PR and on pushes to main. # unit tests + lint, and builds/smoke-tests the gateway image without a
# cluster. Runs on every PR and on pushes to main.
on: on:
pull_request: pull_request:
push: push:
@@ -45,6 +46,10 @@ jobs:
echo "all manifests validated" echo "all manifests validated"
gateway-tests: gateway-tests:
# The gateway lives in examples/dsh-agents-as-a-service/gateway/. Its
# runtime pins and dev pins are kept separate so the image only ships
# requirements.txt while CI installs both files.
name: Gateway unit tests and lint
runs-on: ubuntu-latest runs-on: ubuntu-latest
defaults: defaults:
run: run:
@@ -57,9 +62,76 @@ jobs:
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with: with:
python-version: "3.12" python-version: "3.12"
cache: pip
cache-dependency-path: |
examples/dsh-agents-as-a-service/gateway/requirements.txt
examples/dsh-agents-as-a-service/gateway/requirements-dev.txt
- name: Install dependencies - name: Install dependencies
run: pip install -r requirements.txt pytest run: |
python -m pip install --upgrade pip
pip install -r requirements.txt -r requirements-dev.txt
- name: Lint (ruff)
run: python -m ruff check .
- name: Run gateway unit tests - name: Run gateway unit tests
run: python -m pytest test_gateway.py -v run: python -m pytest -v
gateway-image:
# No cluster and no real credentials: the gateway must start and answer
# /healthz with a dummy kubeconfig, which catches broken imports, missing
# runtime deps, and an unresponsive ASGI entrypoint.
name: Build and smoke-test gateway image
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Build gateway image
run: |
docker build \
-t dsh-aaas-gateway:ci \
examples/dsh-agents-as-a-service/gateway/
- name: Smoke-test /healthz without a cluster
run: |
set -euo pipefail
cat > /tmp/kubeconfig <<'EOF'
apiVersion: v1
kind: Config
clusters:
- name: dummy
cluster:
server: https://127.0.0.1:6443
insecure-skip-tls-verify: true
contexts:
- name: dummy
context:
cluster: dummy
user: dummy
users:
- name: dummy
user: {}
current-context: dummy
EOF
docker run -d --name gateway-smoke \
-p 18080:8080 \
-e NAMESPACE=dsh-demo \
-e POOL=dsh-pool \
-e KUBECONFIG=/tmp/kubeconfig \
-v /tmp/kubeconfig:/tmp/kubeconfig:ro \
dsh-aaas-gateway:ci
trap 'docker logs gateway-smoke || true; docker rm -f gateway-smoke || true' EXIT
code=""
for _ in $(seq 1 30); do
code=$(curl -s -o /dev/null -w '%{http_code}' http://localhost:18080/healthz || true)
[ "$code" = "200" ] && break
sleep 1
done
if [ "$code" != "200" ]; then
echo "gateway /healthz returned '$code' instead of 200" >&2
exit 1
fi
echo "gateway image healthy: GET /healthz -> 200"

12
.gitignore vendored
View File

@@ -68,7 +68,7 @@ web_modules/
# dotenv environment variable files # dotenv environment variable files
.env .env
.env.* .env.*
!.env.example !.env.sample
# parcel-bundler cache (https://parceljs.org/) # parcel-bundler cache (https://parceljs.org/)
.cache .cache
@@ -148,8 +148,14 @@ vite.config.ts.timestamp-*
# Turbo Repo # Turbo Repo
.turbo .turbo
!.env.sample
# Python # Python
__pycache__/ __pycache__/
*.py[cod] *.py[cod]
.pytest_cache/
.ruff_cache/
.mypy_cache/
.coverage
.coverage.*
htmlcov/
.venv/
venv/

13
Dockerfile
View File

@@ -76,6 +76,7 @@ set -eu
: "${DSH_PORT:=3080}" : "${DSH_PORT:=3080}"
: "${DSH_HOST:=}" : "${DSH_HOST:=}"
: "${DSH_TRUSTED_HOSTS:=}"
DSH_PUBLIC_URL="${DSH_PUBLIC_URL-http://localhost:${DSH_PORT}}" DSH_PUBLIC_URL="${DSH_PUBLIC_URL-http://localhost:${DSH_PORT}}"
# DSH rejects wildcard hosts. Pick the container/pod's first non-loopback IPv4. # DSH rejects wildcard hosts. Pick the container/pod's first non-loopback IPv4.
@@ -87,6 +88,18 @@ if [ -n "$DSH_PUBLIC_URL" ]; then
set -- --public-url "$DSH_PUBLIC_URL" "$@" set -- --public-url "$DSH_PUBLIC_URL" "$@"
fi fi
# Additional trusted hosts (e.g. the public gateway host behind an Ingress).
# DSH Host/Origin-fences /api: without this, requests forwarded with the
# browser's Host header are rejected. Accepts comma- or space-separated names;
# localhost/127.0.0.1 are always trusted below.
old_ifs="$IFS"
IFS=', '
for trusted_host in $DSH_TRUSTED_HOSTS; do
[ -n "$trusted_host" ] || continue
set -- "$@" --trusted-host "$trusted_host"
done
IFS="$old_ifs"
exec node /app/lib/bin.js web \ exec node /app/lib/bin.js web \
--no-open \ --no-open \
--host "$DSH_HOST" \ --host "$DSH_HOST" \

37
Makefile Normal file
View File

@@ -0,0 +1,37 @@
# Local build and test helpers. Every variable can be overridden, e.g.
# make image DSH_VERSION=dsh-v0.2.1-rc.2 IMAGE=dsh-web:dev
# make test
DSH_VERSION ?= dsh-v0.2.1-rc.2
PNPM_VERSION ?= 11.28.5
IMAGE ?= dsh-web:0.2.1-rc.2
GATEWAY_IMAGE ?= ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway
GATEWAY_DIR := examples/dsh-agents-as-a-service/gateway
PYTHON ?= python3
DSH_PORT ?= 3080
.PHONY: help image gateway run test lint
help: ## Show available targets
@grep -E '^[a-z-]+:.*?## ' $(MAKEFILE_LIST) | sed 's/:.*## /\t/'
image: ## Build the DSH Web image
docker build \
--build-arg DSH_VERSION=$(DSH_VERSION) \
--build-arg PNPM_VERSION=$(PNPM_VERSION) \
-t $(IMAGE) .
gateway: ## Build the gateway image
docker build -t $(GATEWAY_IMAGE) $(GATEWAY_DIR)/
run: image ## Build and run the DSH Web image
docker run --rm -it \
-p $(DSH_PORT):3080 \
-v dsh-home:/home/node/.dsh \
-e DEEPSEEK_API_KEY \
$(IMAGE)
test: ## Run the gateway unit tests (needs requirements + requirements-dev installed)
cd $(GATEWAY_DIR) && $(PYTHON) -m pytest -q
lint: ## Lint the gateway with ruff
cd $(GATEWAY_DIR) && ruff check .

144
README.md
View File

@@ -3,34 +3,110 @@
Packages [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) Web (`dsh web`) into Packages [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) Web (`dsh web`) into
a multi-arch, Kubernetes-ready container image, and ships the a multi-arch, Kubernetes-ready container image, and ships the
[agents-as-a-service](examples/dsh-agents-as-a-service/) platform pattern — running a personal [agents-as-a-service](examples/dsh-agents-as-a-service/) platform pattern — running a personal
DSH Web instance per user on [kubernetes-sigs/agent-sandbox](https://github.com/kubernetes-sigs/agent-sandbox). DSH Web instance per user on [kubernetes-sigs/agent-sandbox](https://github.com/kubernetes-sigs/agent-sandbox)
behind a small browser-usable gateway.
## What's in this repo ## What's in this repo
| Path | Purpose | | Path | Purpose |
| --- | --- | | --- | --- |
| [`Dockerfile`](Dockerfile) | Multi-stage build that clones DSH, `pnpm pack`s the CLI, and produces a slim runtime image | | [`Dockerfile`](Dockerfile) | Multi-stage build that clones DSH, `pnpm pack`s the CLI, and produces a slim runtime image (`:3080`) |
| [`build.sh`](build.sh) | Local build + run (builds `dsh-web:<version>` and runs it) | | [`build.sh`](build.sh), [`Makefile`](Makefile) | Local image builds: DSH Web and gateway, plus test/lint targets |
| [`.github/workflows/multi-build.yaml`](.github/workflows/multi-build.yaml) | CI build (amd64 + arm64) and push to GHCR |
| [`deploy/k8s/sandbox-template.yaml`](deploy/k8s/sandbox-template.yaml) | Minimal single-tenant `SandboxTemplate` |
| [`examples/dsh-agents-as-a-service/`](examples/dsh-agents-as-a-service/) | The full multi-tenant platform pattern (warm pool, per-user claims, gateway, smoke test) | | [`examples/dsh-agents-as-a-service/`](examples/dsh-agents-as-a-service/) | The full multi-tenant platform pattern (warm pool, per-user claims, gateway, smoke test) |
| [`examples/dsh-agents-as-a-service/gateway/`](examples/dsh-agents-as-a-service/gateway/) | Starlette ASGI gateway: signup, cookie sessions, `/u/{user}/**` HTTP + WebSocket proxy, idle sweeper |
| [`deploy/k8s/sandbox-template.yaml`](deploy/k8s/sandbox-template.yaml) | Minimal single-tenant `SandboxTemplate` |
| [`docs/architecture.md`](docs/architecture.md) | Container and sequence diagrams, design decisions, security notes |
| [`.github/workflows/`](.github/workflows/) | CI: manifests dry-run, gateway tests/lint, multi-arch image builds |
## Quick start (image) ## Published images
The image is published to GitHub Container Registry as Everything is published to GitHub Container Registry as
`ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:<tag>`, where `<tag>` is `main`, `ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox`:
`latest`, or a DSH version tag such as `dsh-v0.2.1-rc.2`.
| Tag | Contents | Port |
| --- | --- | --- |
| `dsh-v0.2.1-rc.2` (also `main`, `latest`) | DSH Web image | `3080` |
| `gateway` | Starlette gateway image | `8080` |
The DSH Web tag matches the upstream DSH git tag it was built from.
## Quick start (DSH Web image)
```bash ```bash
docker run --rm -it \ docker run -d --name dsh-web \
-p 3080:3080 \ -p 3080:3080 \
-v dsh-home:/home/node/.dsh \ -v dsh-home:/home/node/.dsh \
-e DEEPSEEK_API_KEY \ -e DEEPSEEK_API_KEY \
ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.2 ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.2
``` ```
> **Local port note:** `dsh web` listens on `3080` inside the container. If `3080` is **DSH Web is not usable at a bare `/`.** Without its launch-token cookie it answers `401`, and
> already in use on your workstation, map a different host port (e.g. `-p 18081:3080`). the token is only printed in the startup logs:
```bash
docker logs dsh-web 2>&1 | grep -m1 '?token='
# dsh web: http://localhost:3080/?token=...
```
Open the printed `?token=...` URL once — DSH sets its auth cookie and the UI loads normally
afterwards. Hitting `/` directly (or after the cookie expires) returns `401` by design; that is
why Kubernetes probes use `/favicon.svg` (see below) and why the gateway performs this
handshake for the browser.
> **Different host port?** If `3080` is already taken on your workstation, map another port and
> set the public URL so the printed token URL matches: `-p 18081:3080 -e
> DSH_PUBLIC_URL=http://localhost:18081`, then open `http://localhost:18081/?token=...`.
## Quick start (Kubernetes)
```bash
kubectl create namespace dsh-demo --dry-run=client -o yaml | kubectl apply -f -
kubectl -n dsh-demo create secret generic dsh-provider-keys \
--from-literal=DEEPSEEK_API_KEY="sk-..."
kubectl apply -f deploy/k8s/sandbox-template.yaml
# A template needs a warm pool and a claim to become a running Sandbox.
kubectl apply -f - <<'EOF'
apiVersion: extensions.agents.x-k8s.io/v1beta1
kind: SandboxWarmPool
metadata:
name: dsh-pool
namespace: dsh-demo
spec:
replicas: 1
sandboxTemplateRef:
name: dsh-web-template
---
apiVersion: extensions.agents.x-k8s.io/v1beta1
kind: SandboxClaim
metadata:
name: dsh
namespace: dsh-demo
spec:
warmPoolRef:
name: dsh-pool
EOF
kubectl -n dsh-demo wait sandboxclaim/dsh --for=condition=Ready --timeout=180s
```
Then reach the instance through its launch-token URL:
```bash
SB=$(kubectl -n dsh-demo get sandboxclaim dsh -o jsonpath='{.status.sandbox.name}')
kubectl -n dsh-demo logs "$SB" | grep -m1 '?token='
kubectl -n dsh-demo port-forward "pod/$SB" 18081:3080
# open http://localhost:18081/?token=... from the log line
```
Two things bite first-time setups:
- **Probes must not hit `/`.** DSH returns `401` there without the launch token, so a
readiness/liveness probe on `/` crash-loops the pod. The shipped templates use
`/favicon.svg`, which is unauthenticated; keep that path in custom templates.
- **Host allow-listing.** DSH rejects unknown `Host` headers. `localhost` and `127.0.0.1` are
always trusted; for any other hostname add it to the template's `DSH_TRUSTED_HOSTS` (comma or
space separated). The gateway forwards the browser's `Host`, so the public gateway hostname
must be listed there.
## The multi-user platform pattern ## The multi-user platform pattern
@@ -38,15 +114,23 @@ docker run --rm -it \
[hermes-agents-as-a-service](https://github.com/kubernetes-sigs/agent-sandbox/tree/main/examples/hermes-agents-as-a-service) [hermes-agents-as-a-service](https://github.com/kubernetes-sigs/agent-sandbox/tree/main/examples/hermes-agents-as-a-service)
example to DSH Web. It demonstrates the platform shape — one `SandboxTemplate`, one warm pool, example to DSH Web. It demonstrates the platform shape — one `SandboxTemplate`, one warm pool,
one tiny `SandboxClaim` per user, and `operatingMode` as the cost dial — plus a gateway that one tiny `SandboxClaim` per user, and `operatingMode` as the cost dial — plus a gateway that
turns it into a usable service (signup, per-user tokens, `/u/{user}/**` proxy with turns it into a usable service.
wake-on-connect, idle suspend). See its [README](examples/dsh-agents-as-a-service/README.md) for
the full walkthrough. **Browser flow:** open the gateway landing page, sign up, and get an `HttpOnly` session cookie.
The browser then browses `http://<gateway>/u/<user>/`, which proxies HTTP **and WebSocket** to
that user's DSH Web. The gateway reads the DSH launch token from the pod logs, performs the
token handshake server-side, and injects the resulting DSH cookie on every proxied request — the
browser never sees the token. Suspended instances are woken transparently on the next request;
idle ones are suspended again after `IDLE_TIMEOUT`.
See the [example README](examples/dsh-agents-as-a-service/README.md) for the full walkthrough
and [docs/architecture.md](docs/architecture.md) for the diagrams and trade-offs.
### Mechanism mapping: Hermes → DSH ### Mechanism mapping: Hermes → DSH
| Hermes example mechanism | Role | Adaptation for DSH | | Hermes example mechanism | Role | Adaptation for DSH |
| :--- | :--- | :--- | | :--- | :--- | :--- |
| `SandboxTemplate` | Reusable Pod blueprint | Image `dsh-web`, port `3080` | | `SandboxTemplate` | Reusable Pod blueprint | Image `dsh-web`, port `3080`, probes on `/favicon.svg` |
| `volumeClaimTemplates` | Auto-create a PVC per user | Mount `/home/node/.dsh` (DSH state) | | `volumeClaimTemplates` | Auto-create a PVC per user | Mount `/home/node/.dsh` (DSH state) |
| `SandboxWarmPool` | Pre-warmed instances | Keep as-is; size for expected concurrency | | `SandboxWarmPool` | Pre-warmed instances | Keep as-is; size for expected concurrency |
| `SandboxClaim` + `warmPoolRef` | "Registration" adopts a pooled instance | Keep exactly as-is; one claim per user | | `SandboxClaim` + `warmPoolRef` | "Registration" adopts a pooled instance | Keep exactly as-is; one claim per user |
@@ -60,19 +144,37 @@ the full walkthrough.
- **Single port** — DSH Web serves everything on `3080`; no API/dashboard port split. - **Single port** — DSH Web serves everything on `3080`; no API/dashboard port split.
- **One secret** — only `DEEPSEEK_API_KEY`; no dashboard/session/api-server credentials. - **One secret** — only `DEEPSEEK_API_KEY`; no dashboard/session/api-server credentials.
- **Host forwarding** — the gateway forwards `Host` + `X-Forwarded-*` (and keeps - **Host forwarding** — the gateway forwards `Host` + `X-Forwarded-*` (and keeps
`DSH_PUBLIC_URL` empty) so DSH generates reachable links. `DSH_PUBLIC_URL` empty) so DSH generates reachable links; add the public hostname to
- **WebSocket caveat** — the requests-based proxy handles SSE but not WebSocket upgrades; use a `DSH_TRUSTED_HOSTS`.
path-based Ingress per user for full web-GUI interactivity. - **Browser sessions + WebSocket** — the gateway is Starlette ASGI, so it proxies WebSocket
upgrades as well as HTTP/SSE, and authenticates browsers with an `HttpOnly` cookie instead of
exposing per-user bearer tokens in URLs.
## Build it yourself ## Build it yourself
```bash ```bash
./build.sh # docker build + run (dsh-web:0.2.1-rc.2) ./build.sh # build the DSH Web image, then run it interactively
# or ./build.sh web # build the DSH Web image only
./build.sh gateway # build the gateway image
make image # same as ./build.sh web
make gateway # gateway image
make test # gateway unit tests (pytest)
make lint # gateway lint (ruff)
```
Or directly:
```bash
docker build \ docker build \
--build-arg DSH_VERSION=dsh-v0.2.1-rc.2 \ --build-arg DSH_VERSION=dsh-v0.2.1-rc.2 \
--build-arg PNPM_VERSION=11.28.5 \
-t dsh-web:0.2.1-rc.2 . -t dsh-web:0.2.1-rc.2 .
docker build -t ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway \
examples/dsh-agents-as-a-service/gateway/
``` ```
See [`Dockerfile`](Dockerfile) for build args (`DSH_REPO`, `DSH_VERSION`, `PNPM_VERSION`) and See [`Dockerfile`](Dockerfile) for build args (`DSH_REPO`, `DSH_VERSION`, `PNPM_VERSION`) and
the runtime image contract (non-root `node` user, `DSH_HOME=/home/node/.dsh`, `EXPOSE 3080`). the runtime image contract (non-root `node` user, `DSH_HOME=/home/node/.dsh`, `EXPOSE 3080`).
`.env.sample` documents the runtime variables for both images.

75
build.sh Normal file → Executable file
View File

@@ -1,9 +1,68 @@
docker build \ #!/usr/bin/env bash
--build-arg DSH_VERSION=dsh-v0.2.1-rc.2 \ # Build the DSH Web and/or gateway images locally.
-t dsh-web:0.2.1-rc.2 . #
# Usage:
# ./build.sh # build the DSH Web image, then run it interactively
# ./build.sh web # build the DSH Web image only
# ./build.sh gateway # build the gateway image (examples/.../gateway/)
#
# Overridable environment:
# DSH_VERSION DSH git ref to build (default dsh-v0.2.1-rc.2)
# PNPM_VERSION pnpm version (default 11.28.5)
# IMAGE DSH Web image tag (default dsh-web:<version>)
# DSH_PORT host port for docker run (default 3080)
# GATEWAY_IMAGE gateway image tag (default ghcr.io/...:gateway)
# RUN 0 to build without running (default 1)
set -euo pipefail
cd "$(dirname "$0")"
docker run --rm -it \ DSH_VERSION="${DSH_VERSION:-dsh-v0.2.1-rc.2}"
-p 3080:3080 \ PNPM_VERSION="${PNPM_VERSION:-11.28.5}"
-v dsh-home:/home/node/.dsh \ IMAGE="${IMAGE:-dsh-web:${DSH_VERSION#dsh-v}}"
-e DEEPSEEK_API_KEY \ GATEWAY_IMAGE="${GATEWAY_IMAGE:-ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway}"
dsh-web:0.2.1-rc.2 DSH_PORT="${DSH_PORT:-3080}"
RUN="${RUN:-1}"
GATEWAY_DIR=examples/dsh-agents-as-a-service/gateway
build_web() {
docker build \
--build-arg "DSH_VERSION=${DSH_VERSION}" \
--build-arg "PNPM_VERSION=${PNPM_VERSION}" \
-t "${IMAGE}" \
"$@" \
.
}
build_gateway() {
docker build -t "${GATEWAY_IMAGE}" "$@" "${GATEWAY_DIR}/"
}
usage() {
echo "usage: $0 [run|web|gateway] [extra docker build args...]" >&2
exit 2
}
case "${1:-run}" in
run)
shift
build_web "$@"
if [ "${RUN}" = "1" ]; then
docker run --rm -it \
-p "${DSH_PORT}:3080" \
-v dsh-home:/home/node/.dsh \
-e DEEPSEEK_API_KEY \
"${IMAGE}"
fi
;;
web)
shift
build_web "$@"
;;
gateway)
shift
build_gateway "$@"
;;
*)
usage
;;
esac

30
deploy/k8s/sandbox-template.yaml
View File

@@ -3,6 +3,13 @@
# This is the "just one DSH Web" quick-start template. For the full # This is the "just one DSH Web" quick-start template. For the full
# multi-tenant platform (warm pool, per-user claims, gateway, idle-suspend), # multi-tenant platform (warm pool, per-user claims, gateway, idle-suspend),
# see examples/dsh-agents-as-a-service/. # see examples/dsh-agents-as-a-service/.
#
# Prerequisites (create these first, or apply 00-prereqs.yaml from the
# examples directory):
# - Namespace dsh-demo.
# - Secret dsh-provider-keys in dsh-demo with key DEEPSEEK_API_KEY
# (injected via envFrom; the pod starts without it, only model calls fail).
# The agent-sandbox controller (v1.0.7, API v1beta1) must be installed.
apiVersion: extensions.agents.x-k8s.io/v1beta1 apiVersion: extensions.agents.x-k8s.io/v1beta1
kind: SandboxTemplate kind: SandboxTemplate
metadata: metadata:
@@ -27,21 +34,40 @@ spec:
envFrom: envFrom:
- secretRef: - secretRef:
name: dsh-provider-keys # Secret with key DEEPSEEK_API_KEY name: dsh-provider-keys # Secret with key DEEPSEEK_API_KEY
env:
# Empty => DSH derives links from the request Host. Keep it empty
# when fronting the sandbox with a proxy/gateway.
- name: DSH_PUBLIC_URL
value: ""
# DSH Host/Origin-fences /api. Empty trusts only localhost, which
# is right for `kubectl port-forward`. Set this to the public host
# (e.g. dsh.example.com) if you expose DSH behind an Ingress, so
# requests forwarded with the browser's Host are accepted.
- name: DSH_TRUSTED_HOSTS
value: ""
volumeMounts: volumeMounts:
- name: dsh-home - name: dsh-home
mountPath: /home/node/.dsh # DSH_HOME mountPath: /home/node/.dsh # DSH_HOME
# / is behind DSH's launch-token cookie (401 without it); use an
# unauthenticated path for probes.
readinessProbe: readinessProbe:
httpGet: httpGet:
path: / path: /favicon.svg
port: 3080 port: 3080
initialDelaySeconds: 5 initialDelaySeconds: 5
periodSeconds: 5 periodSeconds: 5
livenessProbe: livenessProbe:
httpGet: httpGet:
path: / path: /favicon.svg
port: 3080 port: 3080
initialDelaySeconds: 15 initialDelaySeconds: 15
periodSeconds: 15 periodSeconds: 15
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
seccompProfile:
type: RuntimeDefault
volumeClaimTemplates: volumeClaimTemplates:
- metadata: - metadata:
name: dsh-home name: dsh-home

299
docs/architecture.md Normal file
View File

@@ -0,0 +1,299 @@
# Architecture
This document describes the browser-usable **agents-as-a-service** platform in
[`examples/dsh-agents-as-a-service/`](../examples/dsh-agents-as-a-service/): how the gateway,
Kubernetes, and DSH Web fit together, and why the design looks the way it does.
Scope: one Kubernetes namespace (`dsh-demo`), one shared DSH Web image, one gateway deployment,
and one DSH Web pod + PVC per user. The cluster is assumed to run
[agent-sandbox](https://github.com/kubernetes-sigs/agent-sandbox) v1.0.7 with the extensions
(`SandboxTemplate`, `SandboxWarmPool`, `SandboxClaim`).
## Container view
```mermaid
flowchart LR
browser["Browser"]
subgraph cluster["Kubernetes cluster - namespace dsh-demo"]
gw["Gateway Deployment<br/>Starlette ASGI on uvicorn :8080<br/>landing page + auth + proxy + sweeper"]
api["Kubernetes API Server"]
ctrl["agent-sandbox controller"]
pool["SandboxWarmPool dsh-pool<br/>2 pre-warmed spares"]
claim["SandboxClaim dsh-USER<br/>one per user"]
sb["Sandbox dsh-pool-xxxx<br/>spec.operatingMode"]
svc["Headless Service<br/>status.serviceFQDN"]
pod["DSH Web pod<br/>node process :3080"]
pvc[("PVC dsh-home-SANDBOX<br/>/home/node/.dsh")]
end
browser -->|"HTTP + WebSocket<br/>/u/USER/**"| gw
gw -->|"claims, sandboxes, pods, pod logs"| api
api --> ctrl
ctrl -->|"adopt / suspend / resume"| claim
claim --> sb
pool -.->|"replenishes"| sb
sb --> svc
svc --> pod
pod --> pvc
```
Responsibilities:
| Component | Responsibility |
| --- | --- |
| **Browser** | Holds only the gateway session cookie (`dsh_gw_session`). Talks exclusively to the gateway. |
| **Gateway** | Signup/login, session auth, `SandboxClaim` lifecycle, wake-on-connect, idle sweeper, HTTP/SSE/WebSocket reverse proxy, DSH launch-token handshake. |
| **Kubernetes API** | Source of truth for claims, sandboxes, and pods; also the log source for the DSH launch token. |
| **agent-sandbox controller** | Adopts warm spares on claim, flips `operatingMode` (Running/Suspended), garbage-collects sandbox + PVC on claim delete. |
| **DSH Web pod** | One `dsh web` process per user, state on the per-user PVC, listening on `3080`. |
| **Warm pool** | Keeps `N` Ready sandboxes so signup is a label flip instead of a cold start. |
## Flows
### Signup (browser)
```mermaid
sequenceDiagram
autonumber
actor B as Browser
participant G as Gateway
participant K as Kubernetes API
participant C as agent-sandbox controller
B->>G: POST /users (form: user=bob, Origin)
G->>G: validate user name and body size
G->>K: create SandboxClaim dsh-bob (warmPoolRef)
K->>C: reconcile claim
C-->>K: adopt spare, Ready, status.sandbox.name = dsh-pool-xxxx
G->>K: read claim (status) + sandbox + pod
G-->>B: 303 Location /u/bob/ + Set-Cookie dsh_gw_session
Note over G: JSON clients instead get 201 (Ready),<br/>202 (accepted, still waking), or 409 (conflict)
```
Form signup waits up to `WAKE_TIMEOUT` for the claim to become `Ready`, then redirects; the
first proxied request also waits for `Ready` and performs the DSH handshake (below). A `409`
means the user already has a claim, in which case `POST /login` (bearer token) or the existing
cookie is the right path.
### First request / proxy (with handshake)
```mermaid
sequenceDiagram
autonumber
actor B as Browser
participant G as Gateway
participant K as Kubernetes API
participant D as DSH Web pod
B->>G: GET /u/bob/ (Cookie dsh_gw_session)
G->>G: authenticate cookie, user = bob
G->>K: resolve claim -> sandbox -> pod
alt sandbox Suspended
G->>K: patch operatingMode=Running (wake)
loop until Ready or WAKE_TIMEOUT
G->>K: read sandbox Ready condition
end
end
G->>K: read pod logs (tail)
K-->>G: "dsh web: ...?token=..." line
G->>D: GET /?token=... (browser Host, X-Forwarded-*)
D-->>G: 303 + Set-Cookie dsh-auth-*
G->>G: cache cookie for (user, pod UID, Host)
G->>G: strip Cookie/Authorization/hop-by-hop headers
G->>D: replay request with dsh-auth cookie
D-->>G: 200 HTML / SSE stream
G-->>B: 200 (upstream Set-Cookie dropped, Location rewritten)
```
Handshake cache behavior: one entry per `(user, pod UID, Host)`; refreshed once when DSH answers
`401`; dropped when the sandbox is suspended or deleted. The launch token and the DSH auth
cookie never reach the browser.
### Wake-on-connect and idle suspend
```mermaid
sequenceDiagram
autonumber
participant S as Sweeper - lifespan task
participant G as Gateway
participant K as Kubernetes API
actor B as Browser
S->>G: no proxied request for IDLE_TIMEOUT
G->>K: patch sandbox operatingMode=Suspended
K-->>G: pod deleted, PVC + Service retained
Note over S,G: in-flight requests/WebSockets are tracked,<br/>so a busy turn is never suspended
B->>G: GET /u/bob/ after a while
G->>K: patch operatingMode=Running
G->>K: wait for Ready (WAKE_TIMEOUT)
G-->>B: request held, then served
```
Suspend deletes only the pod. The PVC keeps conversations, sessions, and config, so resume
reattaches exactly the same state.
### WebSocket
```mermaid
sequenceDiagram
autonumber
actor B as Browser
participant G as Gateway
participant D as DSH Web pod
B->>G: GET /u/bob/... Upgrade: websocket (Cookie dsh_gw_session)
G->>G: authenticate + mark user in-flight
alt cookie missing/invalid
G-->>B: 101 upgrade, then close 4401
else authenticated
G->>D: websockets.connect(path, Host, Cookie dsh-auth)
alt upstream handshake fails
G-->>B: 101 upgrade, then close 1011
else bridged
G-->>B: 101 Switching Protocols
loop frames
B->>D: frames proxied through G
D->>B: frames proxied through G
end
G->>G: clear in-flight on close
end
end
```
## Decisions and trade-offs
### ADR-1: Starlette ASGI instead of a sync Flask proxy
- **Decision:** implement the gateway as a Starlette app served by `uvicorn
gateway:create_app --factory`.
- **Why:** the browser needs WebSocket upgrades, streaming SSE responses, and concurrent
long-poll wake requests. A requests-based Flask proxy cannot upgrade connections and blocks
workers while a sandbox wakes.
- **Cost:** async code paths and explicit bridging for the sync `kubernetes` client (offloaded
to threads). Accepted: one small service, one framework.
### ADR-2: Path-based routing (`/u/{user}/`) instead of per-user subdomains
- **Decision:** serve each user under `https://gateway/u/{user}/`.
- **Why:** one hostname, one TLS certificate, no wildcard DNS or per-user Ingress objects. DSH
uses relative asset/API URLs and injects `<base href="./">`, so a path prefix works; the
gateway rewrites absolute-path `Location` headers and redirects `/u/{user}` to
`/u/{user}/`.
- **Cost:** any upstream code that emits root-absolute URLs would need rewriting; cookie
domains are shared (mitigated by stripping DSH `Set-Cookie`). If DSH ever changes its base
handling, per-user subdomains are the fallback.
### ADR-3: Server-side launch-token handshake instead of exposing the token
- **Decision:** the gateway reads DSH's launch token from pod logs, performs the token
handshake itself, caches DSH's auth cookie, and authenticates the browser with its own
`HttpOnly` cookie.
- **Why:** DSH has no token API and returns `401` on `/` without the launch-token cookie.
Handing the token to the browser would put it in URLs, history, and logs, and would let users
bypass gateway auth.
- **Cost:** log reading permission and a brittle parser; see [Upstream feature
request](#upstream-feature-request-remove-log-scraping). This is the least durable part of the
design and the first thing to replace.
### ADR-4: In-memory state, single replica
- **Decision:** idle clocks, handshake cache, and in-flight tracking live in gateway memory;
the Deployment runs one replica.
- **Why:** the simplest thing that works, with no extra datastore for a demo/reference
platform.
- **Cost:** a restart forgets idle timers (sandboxes stay Running until their next request) and
the handshake cache; two replicas would double-suspend and race. Scaling out requires moving
this state to a shared store (Redis/Postgres) or leader election. Documented instead of
hidden.
### ADR-5: Claims as the user record
- **Decision:** a user is a `SandboxClaim` named `dsh-<user>`; the bearer-token SHA-256 lives in
a claim annotation.
- **Why:** no database, no separate account service; deleting the claim is account deletion and
cascades everything.
- **Cost:** no password reset, no admin API, no audit log of its own; Kubernetes API load grows
with users. Fine for a reference pattern, not for a product.
### ADR-6: Gateway cookie session instead of bearer tokens in the browser
- **Decision:** browsers get `dsh_gw_session=<user>:<token>`, `HttpOnly`, `SameSite=Lax`
(`Secure` when enabled); API clients may still use `Authorization: Bearer`.
- **Why:** tokens never appear in URLs or JS-readable storage, and `SameSite=Lax` plus a strict
`Origin`/`Referer` check on cookie-authenticated mutations blocks basic CSRF.
- **Cost:** cookie-based auth needs TLS and correct `COOKIE_SECURE`; there is no signed-cookie
layer, so the gateway trusts the token hash comparison for validation.
## Security and RBAC
The gateway runs with a namespace-scoped ServiceAccount (`50-gateway.yaml`):
| API group | Resource | Verbs | Why |
| --- | --- | --- | --- |
| `extensions.agents.x-k8s.io` | `sandboxclaims` | `get`, `list`, `create`, `delete` | signup, state, account deletion |
| `agents.x-k8s.io` | `sandboxes` | `get`, `list`, `patch` | resolve DNS/pod, wake/suspend |
| `""` (core) | `pods` | `get`, `list` | resolve the pod behind a sandbox |
| `""` (core) | `pods/log` | `get` | launch-token handshake |
Notes and limits:
- **Pod logs are sensitive.** They contain the DSH launch token and possibly model output. The
permission is scoped to one namespace, but anyone who can read gateway memory or logs can
impersonate users. Prefer the upstream token API when available.
- **No cluster-wide access, no secret reads.** The gateway does not read `dsh-provider-keys`.
- **Browser-facing hardening:** `HttpOnly`/`SameSite=Lax`/optional `Secure` cookies,
timing-safe SHA-256 token comparison, strict same-origin check on cookie-auth mutations,
request-body limit (`MAX_BODY_MB`), hop-by-hop header stripping, upstream `Set-Cookie`
dropped, DSH `Location` headers rewritten under `/u/<user>/`.
- **Host allow-list:** DSH rejects unknown `Host` headers. The gateway forwards the browser's
`Host`, so every public hostname must be in `DSH_TRUSTED_HOSTS`; `localhost`/`127.0.0.1` are
always trusted.
- **TLS is external.** The gateway serves plain HTTP; terminate TLS at the Ingress and set
`COOKIE_SECURE=true`. Without TLS, session cookies are replayable on the network.
- **Tenant isolation** is pod + PVC separation; all tenants share one namespace, one DSH binary,
and one `DEEPSEEK_API_KEY`. Add NetworkPolicies and per-tenant credentials before calling it
multi-tenant-grade.
- **No gateway rate limiting.** Signup can create claims until the cluster or warm pool is
exhausted; put limits at the ingress for real deployments.
## Failure modes
| Failure | Symptom | Behavior / mitigation |
| --- | --- | --- |
| DSH log line changed or rotated away | Proxy returns `503` (missing launch token) | Fix the parser or restart the pod so the token line is re-emitted |
| Probe on `/` | Pod `CrashLoopBackOff`, `401` in probe logs | Use `/favicon.svg` for readiness/liveness (shipped templates do) |
| Warm pool exhausted | JSON signup returns `202` (state `Waking`); form signup is slow | Increase `replicas` in the warm pool |
| Gateway restart | Idle sandboxes stop suspending; handshake cache lost | Rebuilt lazily on next request; consider a shared store for production |
| DSH auth cookie expired mid-session | Proxy `401` once, then recovers | Gateway refreshes the handshake once |
| Upstream pod name changed after resume | Cached cookie keyed by old pod UID is useless | Cache is invalidated on suspend; next request re-handshakes |
| Two gateway replicas | Double suspend/wake races | Keep `replicas: 1` until state is externalized |
## Upstream feature request: remove log scraping
DSH should expose the launch token as a first-class, machine-readable interface so platforms do
not need `pods/log` access. Proposed shapes (any one suffices):
1. **Token file:** `dsh web --token-file /run/dsh/token` writes the current launch token (mode
`0600`) and rewrites it on rotation.
2. **Environment input:** accept `DSH_LAUNCH_TOKEN` (platform-generated) so the token never
has to be discovered.
3. **Loopback/trusted-host endpoint:** `GET /api/auth/launch-token` that only answers on the
pod's loopback or a trusted host and returns the token plus expiry.
4. **Token in a well-known file** under `DSH_HOME` with the same semantics as (1).
Why this matters: log scraping is brittle (rotation, restart, format changes), requires a broad
`pods/log` permission that also exposes application output, and can leak tokens into log
pipelines. Until an upstream API exists, the gateway's parser, `DSH_LOG_TAIL_LINES`, and the
single-retry refresh are the supported workaround.
## Evolution path
1. Replace log scraping with the upstream token API (above), deleting the `pods/log` permission.
2. Move idle clocks, in-flight tracking, and handshake cache to a shared store; then scale the
gateway horizontally (sticky routing not required if state is shared).
3. Add per-tenant API keys or an egress proxy so a single key is not shared by all sandboxes.
4. Add NetworkPolicies (gateway → sandboxes only), TLS everywhere, rate limits, and audit
logging at the ingress.
5. If DSH grows absolute URLs or per-user cookies that cannot be rewritten, switch to
per-user subdomains with wildcard DNS/TLS.

23
examples/dsh-agents-as-a-service/10-sandboxtemplate.yaml
View File

@@ -50,20 +50,39 @@ spec:
# gateway base URL if you want hard-coded absolute links instead. # gateway base URL if you want hard-coded absolute links instead.
- name: DSH_PUBLIC_URL - name: DSH_PUBLIC_URL
value: "" value: ""
# DSH Host/Origin-fences /api. The gateway forwards the browser's
# Host, so DSH must trust the public gateway host. Empty is right
# for port-forward/localhost; for the optional Ingress set this to
# the same host as 60-ingress.yaml (e.g. dsh.example.com).
- name: DSH_TRUSTED_HOSTS
value: ""
# Probes target /favicon.svg: DSH gates / behind its launch-token
# cookie and answers 401 to an unauthenticated GET, which would
# otherwise fail readiness and then get the pod killed by liveness.
# /favicon.svg is served without auth and is the cheapest liveness
# signal that proves the HTTP server is actually serving requests.
readinessProbe: readinessProbe:
httpGet: httpGet:
path: / path: /favicon.svg
port: 3080 port: 3080
initialDelaySeconds: 5 initialDelaySeconds: 5
periodSeconds: 5 periodSeconds: 5
failureThreshold: 30 failureThreshold: 30
livenessProbe: livenessProbe:
httpGet: httpGet:
path: / path: /favicon.svg
port: 3080 port: 3080
initialDelaySeconds: 15 initialDelaySeconds: 15
periodSeconds: 15 periodSeconds: 15
failureThreshold: 3 failureThreshold: 3
# The container only needs to serve HTTP; drop every capability and
# disallow privilege escalation. DSH writes only to DSH_HOME (PVC).
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
seccompProfile:
type: RuntimeDefault
resources: resources:
requests: {cpu: 250m, memory: 512Mi} requests: {cpu: 250m, memory: 512Mi}
limits: {cpu: "1", memory: 1Gi} limits: {cpu: "1", memory: 1Gi}

80
examples/dsh-agents-as-a-service/50-gateway.yaml
View File

@@ -1,15 +1,28 @@
# The gateway: control+data plane that turns the resources into a usable # The gateway: control+data plane that turns the resources into a usable
# service. Build the image first (see README step 8): # browser-facing service. It is a Starlette ASGI app (see gateway/) listening
# on :8080 and exposing:
# GET /healthz unauthenticated health probe
# GET / landing page (signup / token login)
# POST /users, POST /login, ... session cookie auth
# /u/<user>/... HTTP + WebSocket proxy to the user's DSH
#
# Image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway
# For local kind development you can build and load a local tag instead, then
# change `image` below to it:
# docker build -t dsh-aaas-gateway:demo gateway/ # docker build -t dsh-aaas-gateway:demo gateway/
# kind load docker-image dsh-aaas-gateway:demo # (kind clusters) # kind load docker-image dsh-aaas-gateway:demo
apiVersion: v1 apiVersion: v1
kind: ServiceAccount kind: ServiceAccount
metadata: metadata:
name: dsh-gateway name: dsh-gateway
namespace: dsh-demo namespace: dsh-demo
--- ---
# Least privilege: claims are created/deleted per user; sandboxes are only # Least privilege:
# read and patched (operatingMode flips for suspend/resume). # - claims are created/deleted per user (signup/delete);
# - sandboxes are read, and patched only to flip operatingMode (suspend/resume);
# - pods are read, and pods/log is read to capture DSH's launch token from
# the startup log line (`dsh web: ...?token=...`).
# configmaps are not needed: all state is in memory / claim annotations.
apiVersion: rbac.authorization.k8s.io/v1 apiVersion: rbac.authorization.k8s.io/v1
kind: Role kind: Role
metadata: metadata:
@@ -22,6 +35,12 @@ rules:
- apiGroups: ["agents.x-k8s.io"] - apiGroups: ["agents.x-k8s.io"]
resources: ["sandboxes"] resources: ["sandboxes"]
verbs: ["get", "list", "patch"] verbs: ["get", "list", "patch"]
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get"]
--- ---
apiVersion: rbac.authorization.k8s.io/v1 apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding kind: RoleBinding
@@ -43,7 +62,11 @@ metadata:
name: dsh-gateway name: dsh-gateway
namespace: dsh-demo namespace: dsh-demo
spec: spec:
replicas: 1 # in-memory idle clock => single replica by design # Single replica BY DESIGN: the idle clock, in-flight counters and the
# per-(user, pod, Host) DSH cookie cache are in-process memory. Running two
# replicas would fragment that state (double suspend/resume races, repeated
# log-token handshakes). Scale only after moving state to a shared store.
replicas: 1
selector: selector:
matchLabels: matchLabels:
app: dsh-gateway app: dsh-gateway
@@ -61,7 +84,7 @@ spec:
type: RuntimeDefault type: RuntimeDefault
containers: containers:
- name: gateway - name: gateway
image: dsh-aaas-gateway:demo image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway
imagePullPolicy: IfNotPresent imagePullPolicy: IfNotPresent
securityContext: securityContext:
allowPrivilegeEscalation: false allowPrivilegeEscalation: false
@@ -69,17 +92,60 @@ spec:
capabilities: capabilities:
drop: ["ALL"] drop: ["ALL"]
ports: ports:
- containerPort: 8080 - name: http
containerPort: 8080
env: env:
- name: NAMESPACE - name: NAMESPACE
value: dsh-demo value: dsh-demo
- name: POOL - name: POOL
value: dsh-pool value: dsh-pool
# Seconds a sandbox may sit idle before the sweeper suspends it
# (pod deleted, PVC kept), and the max time a proxy request waits
# for a suspended sandbox to become Ready again.
- name: IDLE_TIMEOUT - name: IDLE_TIMEOUT
value: "60" value: "60"
- name: WAKE_TIMEOUT
value: "120"
# Container name inside the per-user sandbox pods whose logs carry
# the DSH launch token.
- name: DSH_CONTAINER
value: dsh-web
# Cookie dsh_gw_session is HttpOnly + SameSite=Lax. Set to "true"
# when this Service is reached over HTTPS (e.g. the optional
# 60-ingress.yaml with TLS); "false" keeps the plain-HTTP demo
# usable via port-forward.
- name: COOKIE_SECURE
value: "false"
# Reject proxied request bodies larger than this (413).
- name: MAX_BODY_MB
value: "32"
# /healthz is unauthenticated and returns 200 without touching the
# cluster, so it is safe for both readiness and liveness.
readinessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 5
periodSeconds: 5
failureThreshold: 3
livenessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 10
periodSeconds: 10
failureThreshold: 3
resources: resources:
requests: {cpu: 50m, memory: 128Mi} requests: {cpu: 50m, memory: 128Mi}
limits: {cpu: 500m, memory: 256Mi} limits: {cpu: 500m, memory: 256Mi}
volumeMounts:
# readOnlyRootFilesystem leaves no writable HOME/tmp; the image sets
# HOME=/tmp, so back it with scratch space. No state lives here.
- name: tmp
mountPath: /tmp
volumes:
- name: tmp
emptyDir: {}
--- ---
apiVersion: v1 apiVersion: v1
kind: Service kind: Service

46
examples/dsh-agents-as-a-service/60-ingress.yaml Normal file
View File

@@ -0,0 +1,46 @@
# OPTIONAL browser entry point: expose the gateway through Traefik.
#
# Requires 50-gateway.yaml (Service dsh-gateway:8080) and a Traefik install
# with an IngressClass named "traefik" (k3s ships one). Replace the placeholder
# host below with a name that resolves to your ingress address, and either
# supply a real TLS Secret or delete the `tls:` block for plain HTTP.
#
# Two things must match this host:
# 1. SandboxTemplate (10-sandboxtemplate.yaml): set DSH_TRUSTED_HOSTS to
# this host so DSH accepts the forwarded Host/Origin on /api.
# 2. Gateway Deployment (50-gateway.yaml): set COOKIE_SECURE to "true"
# once this Ingress serves HTTPS, so the session cookie is Secure.
#
# WebSocket support is REQUIRED: the browser talks to /u/<user>/... and DSH
# uses WebSockets for live session traffic. Traefik upgrades WS/SSE
# transparently for ordinary HTTP routers, but do NOT attach buffering
# middlewares (e.g. compress, buffering) to this router — they break streaming
# and upgrades. `path: /` Prefix covers the landing page and every per-user
# proxy path, so no per-sandbox Ingress is needed.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: dsh-gateway
namespace: dsh-demo
# Typical Traefik + cert-manager wiring (uncomment and adapt):
# annotations:
# traefik.ingress.kubernetes.io/router.entrypoints: websecure
# cert-manager.io/cluster-issuer: letsencrypt-prod
annotations: {}
spec:
ingressClassName: traefik
tls:
- hosts:
- dsh.example.com # placeholder
secretName: dsh-example-tls # placeholder; create/PKI-provision this Secret
rules:
- host: dsh.example.com # placeholder
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: dsh-gateway
port:
number: 8080

203
examples/dsh-agents-as-a-service/README.md
View File

@@ -11,10 +11,11 @@ sessions, config) surviving on a per-user PVC at `/home/node/.dsh`.
It ships in two layers: It ships in two layers:
1. **The resource layer** (steps 1–7): pure manifests + kubectl, so every mechanism is visible. 1. **The resource layer** (steps 1–7): pure manifests + kubectl, so every mechanism is visible.
2. **The service layer** (step 8): a small included **gateway** (`gateway/`) that makes it 2. **The service layer** (step 8): a small included **gateway** (`gateway/`) that makes the
directly usable — signup API, per-user bearer tokens, a `/u/{user}/**` proxy that platform browser-usable — a signup page/API, `HttpOnly` cookie sessions, a `/u/{user}/**`
transparently **wakes suspended instances on connect**, and an idle sweeper that suspends proxy for HTTP **and WebSocket** that transparently **wakes suspended instances on
them again. connect**, and an idle sweeper that suspends them again. The gateway performs the DSH
launch-token handshake server-side, so the browser never sees the token.
What it demonstrates, feature by feature: What it demonstrates, feature by feature:
@@ -37,11 +38,14 @@ What it demonstrates, feature by feature:
instead of splitting `/v1/*` (API) from the dashboard. instead of splitting `/v1/*` (API) from the dashboard.
- **One shared secret**: only `DEEPSEEK_API_KEY` is injected (via `envFrom`); there is no - **One shared secret**: only `DEEPSEEK_API_KEY` is injected (via `envFrom`); there is no
dashboard basic-auth, session-cookie secret, or API-server key. dashboard basic-auth, session-cookie secret, or API-server key.
- **Host forwarding**: the gateway forwards the original `Host` + `X-Forwarded-*` (and keeps - **Launch-token handshake**: DSH gates `/` behind a token it prints only in its logs. The
`DSH_PUBLIC_URL` empty) so DSH generates reachable links from the request. gateway reads it, performs the handshake, and keeps the resulting auth cookie server-side;
- **WebSocket caveat**: the requests-based streaming proxy handles SSE but not WebSocket the browser authenticates to the gateway with its own cookie. See
upgrades. For full DSH web-GUI interactivity, front the gateway with a path-based Ingress [How the DSH handshake works](#how-the-dsh-handshake-works).
per user (or a WS-capable proxy). - **Host allow-list**: DSH rejects unknown `Host` headers. The gateway forwards the browser's
`Host`, so the public hostname must be added to `DSH_TRUSTED_HOSTS` in the template.
- **Probes target `/favicon.svg`**: `/` returns `401` without the launch token, so a probe on
`/` crash-loops the pod. `/favicon.svg` is unauthenticated.
## Prerequisites ## Prerequisites
@@ -55,7 +59,7 @@ What it demonstrates, feature by feature:
``` ```
Or run everything below in one shot with `./run-test-kind.sh` (expects a kind cluster + the Or run everything below in one shot with `./run-test-kind.sh` (expects a kind cluster + the
manifest above already applied). manifest above already applied; it creates and deletes the namespace itself).
## 1. Install the platform resources ## 1. Install the platform resources
@@ -111,14 +115,20 @@ Three things worth noticing:
## 3. Talk to the instance ## 3. Talk to the instance
DSH Web needs its launch token on first contact — a bare `/` returns `401`. The token is in the
pod logs:
```bash ```bash
kubectl -n dsh-demo port-forward "$SB" 18081:3080 kubectl -n dsh-demo logs "$SB" | grep -m1 '?token='
# open http://localhost:18081 — DSH Web for alice # dsh web: http://<pod-ip>:3080/?token=...
kubectl -n dsh-demo port-forward "pod/$SB" 18081:3080
# open http://localhost:18081/?token=... (from the log line)
``` ```
> **Local port note:** DSH Web listens on `3080` **in-cluster**. If `3080` is already in use > **Local port note:** DSH Web listens on `3080` **in-cluster**. If `3080` is already in use
> on your workstation (e.g. by a local DSH), forward to a different local port (here `18081`), > on your workstation (e.g. by a local DSH), forward to a different local port (here `18081`),
> never `3080`. > never `3080`. `localhost` is always in DSH's trusted-host list, so the forwarded request is
> accepted.
## 4. Write state on the PVC ## 4. Write state on the PVC
@@ -153,38 +163,169 @@ kubectl -n dsh-demo get sandboxclaim dsh-mallory \
kubectl delete -f 40-claim-rejected.yaml kubectl delete -f 40-claim-rejected.yaml
``` ```
## 8. The gateway: signup, proxy, idle-suspend, wake-on-connect ## 8. The gateway: signup, browser proxy, idle-suspend, wake-on-connect
Build the gateway image and deploy it: ### Deploy it
```bash ```bash
docker build -t dsh-aaas-gateway:demo gateway/ # Published image (on kind, load it into the cluster; on a real cluster it is
kind load docker-image dsh-aaas-gateway:demo # (kind clusters) # pulled from GHCR):
docker pull ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway
kind load docker-image ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway --name <kind-cluster>
# ... or build it locally and tag it exactly as 50-gateway.yaml expects:
docker build -t ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway gateway/
kind load docker-image ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway --name <kind-cluster>
kubectl apply -f 50-gateway.yaml kubectl apply -f 50-gateway.yaml
kubectl -n dsh-demo rollout status deploy/dsh-gateway --timeout=120s kubectl -n dsh-demo rollout status deploy/dsh-gateway --timeout=120s
kubectl -n dsh-demo port-forward svc/dsh-gateway 18080:8080 kubectl -n dsh-demo port-forward svc/dsh-gateway 18080:8080
``` ```
Then: > The gateway binds `8080`; the local port-forward uses `18080` and never `3080`.
>
> **Applying over an existing install?** Run `kubectl diff -f 10-sandboxtemplate.yaml`
> first. Cluster-specific fields such as `nodeAffinity` and the warm-pool replica count are
> intentionally not in the repo manifests; preserve them with a targeted patch instead of a
> blind apply. Moving the probes to `/favicon.svg` is the crash-loop fix.
### Browser flow
Open <http://localhost:18080>. The landing page has a one-field signup form. Enter `bob`, and
the gateway:
1. creates the `SandboxClaim` over the warm pool,
2. waits for it to become `Ready`,
3. sets an `HttpOnly`, `SameSite=Lax` cookie `dsh_gw_session=<user>:<token>` on the browser,
4. redirects to `/u/bob/`, where the DSH Web UI loads through the gateway.
From then on every request under `/u/bob/` (assets, API calls, SSE, WebSocket) is proxied to
bob's sandbox. The DSH launch token and DSH's own auth cookie stay in the gateway; the browser
only holds the gateway cookie.
The same flow with `curl`, no JavaScript (the session cookie is the only credential):
```bash ```bash
# signup → claim over the warm pool + one-time bearer token JAR=$(mktemp)
SIGNUP=$(curl -s -X POST http://localhost:18080/users -H 'Content-Type: application/json' -d '{"user":"bob"}') # signup (form POST, exactly what the landing page sends)
TOKEN=$(echo "$SIGNUP" | sed -n 's/.*"token":"\([^"]*\)".*/\1/p') curl -s -i -c "$JAR" -H 'Origin: http://localhost:18080' \
-d 'user=bob' http://localhost:18080/users
# proxy to bob's DSH Web (the gateway wakes it if suspended) # proxy: bob's DSH Web through the gateway
curl -s -H "Authorization: Bearer $TOKEN" http://localhost:18080/u/bob/ curl -s -b "$JAR" http://localhost:18080/u/bob/ | head -5
# without the cookie the gateway fails closed
# tokenless is rejected
curl -s -o /dev/null -w '%{http_code}\n' http://localhost:18080/u/bob/ # 401 curl -s -o /dev/null -w '%{http_code}\n' http://localhost:18080/u/bob/ # 401
# derived state: Ready / Waking / Suspended / ...
# after IDLE_TIMEOUT (60s) the sweeper suspends bob curl -s -b "$JAR" http://localhost:18080/users/bob
curl -s -H "Authorization: Bearer $TOKEN" http://localhost:18080/users/bob # state: Suspended
# delete bob (cascades sandbox + PVC) # delete bob (cascades sandbox + PVC)
curl -s -X DELETE -H "Authorization: Bearer $TOKEN" http://localhost:18080/users/bob curl -s -b "$JAR" -H 'Origin: http://localhost:18080' -X DELETE http://localhost:18080/users/bob
``` ```
API clients that prefer bearer tokens can use `POST /users` with a JSON body
(`{"user":"bob"}`) and the returned one-time token, `POST /login` to exchange a token for a
cookie, and `Authorization: Bearer <token>` on subsequent calls.
### Gateway API
| Route | Purpose |
| --- | --- |
| `GET /healthz` | Liveness/readiness; no cluster access |
| `GET /` | Landing page: signup, token login, logout |
| `POST /users` | Signup. JSON or form; `201` (JSON), `202` (waking), `303` (browser redirect), `409` if user exists; sets the session cookie |
| `POST /login` | Exchange a bearer token for the session cookie |
| `POST /logout` | Clear the session cookie |
| `GET /users/{user}` | Derived state (`Ready`/`Waking`/`Suspended`) |
| `DELETE /users/{user}` | Delete claim (cascades sandbox + PVC) |
| `GET /u/{user}` | `308` redirect to `/u/{user}/` |
| `GET /u/{user}/**` | HTTP/SSE proxy to the user's DSH Web |
| `WS /u/{user}/**` | WebSocket proxy to the user's DSH Web |
### How the DSH handshake works
DSH does not expose an API for its launch token; it prints a URL containing `?token=…` in the
pod logs. For each `(user, pod UID, Host)` the gateway:
1. resolves the claim → sandbox → pod (annotation `agents.x-k8s.io/pod-name`, or the sandbox
name as pod name),
2. reads the pod's logs (last `DSH_LOG_TAIL_LINES` lines) and parses the most recent
`dsh web: …?token=…` line,
3. requests `http://<serviceFQDN>:3080/?token=…` with the browser's `Host`,
4. captures DSH's `dsh-auth-*` `Set-Cookie` response and caches it,
5. injects that cookie (and strips the browser's `Cookie`/`Authorization` and any `?token=`)
on every proxied HTTP and WebSocket request,
6. refreshes the handshake once if DSH answers `401`, and drops the cached cookie when the
sandbox suspends or is deleted.
This is why the gateway's RBAC includes `pods` `get`/`list` and `pods/log` `get`. It is also
the most fragile part of the design; see [Limitations](#limitations).
### DSH_TRUSTED_HOSTS
DSH rejects requests whose `Host` is not allow-listed (`localhost` and `127.0.0.1` are always
trusted). Because the gateway forwards the browser's `Host`, set the public hostname in the
template:
```yaml
env:
- name: DSH_TRUSTED_HOSTS
value: "dsh.example.com" # comma or space separated; empty for port-forward only
```
The same applies to a local `docker run`: the printed token URL uses `DSH_PUBLIC_URL` if set,
otherwise `http://localhost:${DSH_PORT}`.
### WebSocket
The gateway is a Starlette ASGI app and proxies WebSocket upgrades with a bidirectional pump, so
the DSH web GUI is fully interactive. Requirements:
- the ingress/load balancer in front of the gateway must pass `Upgrade`/`Connection` headers;
`60-ingress.yaml` is a starting point (Traefik handles upgrades by default),
- idle suspension is disabled for users with in-flight proxied requests or open WebSockets, so
a long-running chat stream is not cut off mid-turn,
- missing/invalid gateway auth is rejected right after the upgrade with close code `4401`
(WebSocket auth has no HTTP `401` path); cross-origin or Origin-less upgrades with the cookie
are rejected the same way (CSWSH protection), so clients must send a matching `Origin`;
upstream bridging failures close with `1011`.
### Path routing
Each user's DSH Web is served under `/u/<user>/`. DSH injects `<base href="./">` and uses
relative asset/API URLs, so the UI works under a path prefix without a per-user hostname or
wildcard DNS. The gateway also rewrites absolute-path `Location` headers into `/u/<user>/…` and
redirects `/u/<user>` to the trailing-slash form with `308` so relative URLs resolve correctly.
## Troubleshooting
| Symptom | Cause / fix |
| --- | --- |
| Pod `CrashLoopBackOff`, probes failing with `401` | Probe path is `/`; change readiness/liveness to `/favicon.svg` |
| Browser shows `401` from DSH through the gateway | Gateway handshake failed — check `kubectl -n dsh-demo logs deploy/dsh-gateway` and that RBAC grants `pods/log` |
| `502` / `upstream unavailable` | Sandbox pod is not Ready (wake timed out) or the Service DNS is stale; check `kubectl get sandbox`, `WAKE_TIMEOUT` |
| `503` with `Retry-After` | The launch token was not found in the pod logs yet; the gateway retries after a short delay. Check the pod logs and `DSH_LOG_TAIL_LINES` |
| `409` on signup | User already has a claim; use `POST /login` with the existing token or delete the user first |
| Cookie set but browser keeps getting `401` | `COOKIE_SECURE=true` while serving over plain HTTP (secure cookies are not sent); use HTTPS or set `false` for local demos |
| WebSocket closes immediately (`4401`) | Missing/expired gateway cookie, or the upgrade's `Origin` does not match `Host` (CSWSH check); DSH rejecting `Host` instead surfaces as close `1011` — add the host to `DSH_TRUSTED_HOSTS` |
| Signup is slow / `202` then `Waking` | Warm pool exhausted (`kubectl get sandboxes -l agents.x-k8s.io/warm-pool-sandbox`); scale the pool |
| `403` from the Kubernetes API in gateway logs | The gateway ServiceAccount is missing claim/sandbox/pod-log verbs (see `50-gateway.yaml`) |
## Limitations
Deliberately **not** production code:
- **Single replica.** Idle clocks, in-flight tracking, and the handshake cache are in memory.
Running two gateway replicas would double-suspend or lose wake tracking; scale up only after
moving that state to a shared store.
- **Log scraping.** The launch token is recovered from pod logs. Log rotation, a restarted pod,
or a changed DSH log format can break the handshake until the next `401` refresh.
- **Shared provider key.** `DEEPSEEK_API_KEY` is shared by all tenants via the template. Real
multi-tenancy needs per-tenant credentials or an egress proxy.
- **No TLS in the gateway.** Terminate TLS at an Ingress/reverse proxy and set
`COOKIE_SECURE=true`.
- **In-memory user directory.** Users are claims + annotations; there is no database, password
reset, or admin API. Tokens are stored only as SHA-256 hashes in claim annotations.
- **Bearer tokens in URLs.** Avoid putting the signup token in a URL/log; the browser flow uses
the `HttpOnly` cookie instead.
## Cost model ## Cost model
The same economics as the upstream pattern: suspend-when-idle plus warm pools turn an The same economics as the upstream pattern: suspend-when-idle plus warm pools turn an

13
examples/dsh-agents-as-a-service/gateway/Dockerfile
View File

@@ -1,7 +1,18 @@
FROM python:3.12-slim FROM python:3.12-slim
# Read-only-rootfs friendly: no .pyc writes, no home directory writes.
ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1 \
HOME=/tmp
WORKDIR /app WORKDIR /app
COPY requirements.txt . COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt RUN pip install --no-cache-dir -r requirements.txt
COPY gateway.py . COPY gateway.py .
EXPOSE 8080
# Run as a fixed non-root UID; HOME=/tmp keeps caches out of the image.
USER 10000 USER 10000
CMD ["python", "gateway.py"]
CMD ["uvicorn", "gateway:create_app", "--factory", \
"--host", "0.0.0.0", "--port", "8080", "--no-access-log"]

184
examples/dsh-agents-as-a-service/gateway/README.md Normal file
View File

@@ -0,0 +1,184 @@
# DSH agents-as-a-service gateway
A single-file, browser-usable control + data plane for
[DeepSeek Harness Web (DSH)](https://github.com/deepseek-ai/deepseek-harness)
sandboxes running behind the `agent-sandbox` CRDs.
It does three things:
1. **Users** — creates a `SandboxClaim` from a warm pool, mints a bearer token
(only its SHA-256 is stored as a claim annotation), reports derived state,
and deletes the claim (cascading the sandbox and PVC).
2. **Browser sessions** — exchanges a user token for an HttpOnly session
cookie so the DSH UI works in a browser at `/u/<user>/`. Automation can
still use `Authorization: Bearer <token>`.
3. **Reverse proxy** — proxies HTTP (including SSE) and WebSocket traffic to
the sandbox's DSH Web server on `:3080`, including DSH's own cookie
handshake.
DSH is pinned to `dsh-v0.2.1-rc.2` in the example manifests. It binds `:3080`,
serves `/`, gates `/api` behind a `dsh-auth-*` cookie and requires a WebSocket
at `/api/remote.mux`.
## Flow
```
browser / curl gateway cluster
| | |
| POST /users {"user":"alice"} | |
|------------------------------>| create SandboxClaim ---------->|
| | wait Ready |
| 201 {token} + session cookie | |
|<------------------------------| |
| GET /u/alice/ (cookie) | |
|------------------------------>| claim -> sandbox -> pod |
| | read pod logs (pods/log) |
| | find "dsh web: ...?token=..." |
| | GET http://<fqdn>:3080/?token= |
| | capture dsh-auth-* Set-Cookie |
| | proxy with Host/Origin kept |
| HTML / SSE / WebSocket |<----------------------------->|
|<------------------------------| |
```
The launch token is printed by DSH to its pod stdout:
```
dsh web: http://<podIP>:3080/?token=<base64url>
```
The gateway reads the tail of the pod logs, parses the **last** such line and
performs the `?token=` handshake itself with the browser's `Host` header. The
resulting `dsh-auth-*` cookie is cached per `(user, pod UID, Host)` and injected
into proxied requests. The browser never sees the launch token or the bearer
token. A failed handshake returns `503` with `Retry-After: 10`.
## API
| Method | Path | Auth | Purpose |
| --- | --- | --- | --- |
| `GET` | `/healthz` | none | Liveness/readiness. Never touches the cluster. |
| `GET` | `/` | none | Landing page with signup + token login forms. |
| `POST` | `/users` | none | Signup. JSON body `{"user": "<dns-1123 label>"}` or form field `user`. JSON returns `201` when Ready, `202` when accepted but still Waking; form returns `303` to `/u/<user>/`. Both set the session cookie. `409` if the user exists, `400` if invalid, `413` if the body is too large. |
| `POST` | `/login` | none | Form (`user`, `token`) returns `303` to `/u/<user>/`; JSON returns `200`. Sets the session cookie. `401` on a bad token, `404` for an unknown user. |
| `POST` | `/logout` | none | Clears the session cookie and `303`s to `/`. |
| `GET` | `/users/{user}` | bearer or cookie | Derived state: `Provisioning`, `Waking`, `Ready`, `Suspending`, `Suspended`. |
| `DELETE` | `/users/{user}` | bearer or cookie | Delete the claim (sandbox + PVC cascade). Cookie-auth mutations require a same-origin `Origin`/`Referer`. |
| `GET` | `/u/{user}` | — | `308` redirect to `/u/{user}/`. |
| `*` | `/u/{user}/{path}` | bearer or cookie | HTTP proxy to DSH. Waits for a suspended sandbox to wake. Wakes on connect. |
| `WS` | `/u/{user}/{path}` | bearer or cookie | WebSocket proxy (e.g. `/api/remote.mux`). |
### Auth
* `Authorization: Bearer <token>` is checked with a timing-safe SHA-256
comparison against the `aaas.example.com/token-sha256` claim annotation.
* The browser session cookie is
`dsh_gw_session=<user>:<token>` (HttpOnly, SameSite=Lax, Path=/, optional
Secure). A session for user A never authorizes user B.
* Cookie-authenticated **mutations** (`POST`, `PUT`, `PATCH`, `DELETE`) and
cookie-authenticated **WebSocket upgrades** require the request `Origin`
(or `Referer` fallback) to match the `Host` header. Missing or mismatched
cross-origin metadata is rejected (`403` HTTP, close code `4401` for WS).
* The gateway's bearer token, the browser `Cookie` header and any `?token=`
query parameter are never forwarded upstream. Only the cached `dsh-auth-*`
cookie is injected.
* Upstream `Set-Cookie` headers are dropped. `Location` headers that are
absolute paths are rewritten under `/u/<user>/`, and any `token` query
parameter is stripped.
### Proxy failure semantics
| Situation | Result |
| --- | --- |
| Unknown user / claim | `404` |
| Missing or invalid credentials | `401` |
| Cross-origin cookie mutation | `403` |
| Request body over `MAX_BODY_MB` | `413` |
| Sandbox not provisioned, wake timeout, handshake failure | `503` + `Retry-After: 10` |
| Upstream connection/protocol failure | `502` |
| WS auth failure (unknown user, bad token, cross-origin) | accept, then close `4401` |
| WS upstream/handshake failure | accept, then close `1011` |
A close code can only be delivered after the WebSocket upgrade is accepted;
Starlette would turn an early close into an HTTP 403, which is why WS auth
failures are `101`, then close `4401`.
## Configuration
All values are read from the environment by `Settings.from_env()`; every one
can also be injected into `create_app` for tests.
| Variable | Default | Meaning |
| --- | --- | --- |
| `NAMESPACE` | `dsh-demo` | Namespace holding the claims, sandboxes and pods. |
| `POOL` | `dsh-pool` | `warmPoolRef.name` for new claims. |
| `IDLE_TIMEOUT` | `60` | Seconds of inactivity before the sweeper suspends a sandbox. |
| `WAKE_TIMEOUT` | `120` | Seconds to wait for a sandbox to become Ready (signup and wake-on-connect). |
| `DSH_PORT` | `3080` | Upstream DSH port. |
| `DSH_CONTAINER` | `dsh-web` | Container whose logs carry the launch token. |
| `DSH_LOG_TAIL_LINES` | `2000` | How many log lines to scan for the launch token. |
| `COOKIE_NAME` | `dsh_gw_session` | Browser session cookie name. |
| `COOKIE_SECURE` | `false` | Add `Secure` to the session cookie when served over TLS. |
| `COOKIE_MAX_AGE` | `604800` | Session cookie lifetime in seconds. |
| `MAX_BODY_MB` | `32` | Maximum buffered request body (and `413` threshold). |
| `HTTP_TIMEOUT` | `300` | Upstream connect/read timeout in seconds. |
| `SWEEP_INTERVAL` | `15` | Seconds between idle sweeps. |
### Required RBAC
The gateway's ServiceAccount needs, in its namespace:
* `sandboxclaims` (`extensions.agents.x-k8s.io`): get, list, create, delete
* `sandboxes` (`agents.x-k8s.io`): get, patch
* `pods`: get
* `pods/log`: get
## Running locally
The module has no import-time cluster access and the Kubernetes adapter is
lazy, so `/healthz` works without a kubeconfig:
```bash
python -m pip install -r requirements-dev.txt
python -m pytest -q
python -m ruff check .
# Uvicorn factory (loads in-cluster credentials only on the first claim call)
uvicorn gateway:create_app --factory --host 0.0.0.0 --port 8080
```
Container image:
```bash
docker build -t dsh-gateway examples/dsh-agents-as-a-service/gateway
```
The image runs as UID `10000`, sets `PYTHONDONTWRITEBYTECODE=1` and
`HOME=/tmp` (read-only-rootfs compatible), listens on `8080`, and disables
uvicorn's access log so bearer tokens in URLs are not logged.
## Known limitations
* **Log scraping is fragile.** The launch token is discovered from pod logs.
Log rotation, a very chatty container, or DSH changing its startup message
breaks the handshake. Prefer a first-class token/credential API when DSH
offers one.
* **Single replica, in-memory state.** The handshake cookie cache, idle
activity clocks and in-flight counters live in one process. Running more
than one replica would need shared state (or sticky routing), otherwise each
replica handshakes independently and the sweeper may race another replica.
* **In-memory token cache has no TTL.** It is keyed by pod UID, so pod
restarts invalidate it naturally; otherwise entries survive until the pod or
claim is deleted.
* **No TLS termination.** Run behind an ingress/load balancer; set
`COOKIE_SECURE=true` there.
* **Origin-based CSRF protection only.** Cookie-auth mutations require a
same-origin `Origin`/`Referer`; there is no per-form CSRF token, rate
limiting or login throttling.
* **HTTP request bodies are buffered** up to `MAX_BODY_MB`; uploads are not
streamed. Responses (including SSE) are streamed.
* **Upstream cookies are not forwarded.** DSH cannot rotate its own
`dsh-auth-*` cookie through the gateway; on an upstream `401` the gateway
re-runs the handshake once and replays the request.
* **Only DSH on `:3080` is proxied.** There is no dashboard/API-port split and
no platform API key injection.

1498
examples/dsh-agents-as-a-service/gateway/gateway.py
View File

File diff suppressed because it is too large Load Diff

14
examples/dsh-agents-as-a-service/gateway/pyproject.toml Normal file
View File

@@ -0,0 +1,14 @@
[tool.pytest.ini_options]
asyncio_mode = "auto"
testpaths = ["."]
[tool.ruff]
target-version = "py312"
line-length = 100
[tool.ruff.lint]
select = ["E", "F", "I", "B", "UP", "PIE", "RUF"]
ignore = ["RUF001", "RUF002", "RUF003"]
[tool.ruff.lint.per-file-ignores]
"test_gateway.py" = ["E501"]

4
examples/dsh-agents-as-a-service/gateway/requirements-dev.txt Normal file
View File

@@ -0,0 +1,4 @@
-r requirements.txt
pytest==9.1.1
pytest-asyncio==1.4.0
ruff==0.17.1

9
examples/dsh-agents-as-a-service/gateway/requirements.txt
View File

@@ -1,3 +1,6 @@
flask==3.1.* starlette==1.9.0
requests==2.32.* uvicorn[standard]==0.54.0
kubernetes==31.0.* httpx==0.28.1
websockets==17.3
kubernetes==37.0.1
python-multipart==0.0.32

1378
examples/dsh-agents-as-a-service/gateway/test_gateway.py
View File

File diff suppressed because it is too large Load Diff

157
examples/dsh-agents-as-a-service/run-test-kind.sh
View File

@@ -12,7 +12,11 @@ cd "$(dirname "$0")"
NS=dsh-demo NS=dsh-demo
pass() { echo "PASS: $1"; } pass() { echo "PASS: $1"; }
fail() { echo "FAIL: $1" >&2; exit 1; } fail() { echo "FAIL: $1" >&2; exit 1; }
cleanup() { kubectl delete namespace "$NS" --ignore-not-found --wait=false >/dev/null 2>&1 || true; } cleanup() {
kill "${PF:-}" 2>/dev/null || true
rm -f "${JAR:-}"
kubectl delete namespace "$NS" --ignore-not-found --wait=false >/dev/null 2>&1 || true
}
trap cleanup EXIT trap cleanup EXIT
echo "=== 1. install prereqs (throwaway key) + template + warm pool" echo "=== 1. install prereqs (throwaway key) + template + warm pool"
@@ -77,34 +81,146 @@ BOUND=$(kubectl -n "$NS" get sandboxclaim dsh-mallory -o jsonpath='{.status.sand
kubectl delete -f 40-claim-rejected.yaml >/dev/null kubectl delete -f 40-claim-rejected.yaml >/dev/null
pass "claim rejected with EnvVarsInjectionRejected and never bound" pass "claim rejected with EnvVarsInjectionRejected and never bound"
echo "=== 7. gateway: signup, proxy, idle-suspend, wake-on-connect" echo "=== 7. gateway: browser signup, proxy, idle-suspend, wake-on-connect"
if command -v docker >/dev/null && command -v kind >/dev/null; then if command -v docker >/dev/null && command -v kind >/dev/null; then
KIND_CLUSTER="${KIND_CLUSTER:-$(kubectl config current-context | sed 's/^kind-//')}" KIND_CLUSTER="${KIND_CLUSTER:-$(kubectl config current-context | sed 's/^kind-//')}"
docker build -q -t dsh-aaas-gateway:demo gateway/ >/dev/null # Tag exactly what 50-gateway.yaml references so the smoke test exercises
kind load docker-image dsh-aaas-gateway:demo --name "$KIND_CLUSTER" >/dev/null # the shipped manifest unchanged (imagePullPolicy: IfNotPresent uses the
# image loaded into kind).
GW_IMAGE=ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway
docker build -q -t "$GW_IMAGE" gateway/ >/dev/null
kind load docker-image "$GW_IMAGE" --name "$KIND_CLUSTER" >/dev/null
kubectl apply -f 50-gateway.yaml kubectl apply -f 50-gateway.yaml
kubectl -n "$NS" rollout status deploy/dsh-gateway --timeout=120s kubectl -n "$NS" rollout status deploy/dsh-gateway --timeout=120s
# Local port-forward uses 18080 (NOT 3080, which stays in-cluster). # Local port-forward uses 18080 (NOT 3080, which stays in-cluster).
kubectl -n "$NS" port-forward svc/dsh-gateway 18080:8080 >/dev/null 2>&1 & kubectl -n "$NS" port-forward svc/dsh-gateway 18080:8080 >/dev/null 2>&1 &
PF=$! PF=$!
sleep 3
GW=localhost:18080 GW=localhost:18080
JAR=$(mktemp)
SIGNUP=$(curl -s -X POST "http://$GW/users" -H 'Content-Type: application/json' -d '{"user":"bob"}') for _ in $(seq 1 20); do
TOKEN=$(echo "$SIGNUP" | sed -n 's/.*"token":"\([^"]*\)".*/\1/p') HEALTH=$(curl -s -o /dev/null -w '%{http_code}' "http://$GW/healthz" || true)
[ -n "$TOKEN" ] || fail "signup returned no token: $SIGNUP" [ "$HEALTH" = "200" ] && break
echo "$SIGNUP" | grep -q '"state":"Ready"' || fail "signup not Ready: $SIGNUP" sleep 1
pass "gateway signup: bob Ready with one-time token" done
[ "${HEALTH:-}" = "200" ] || fail "gateway /healthz was not reachable on $GW"
pass "gateway health: /healthz -> 200"
curl -sf -H "Authorization: Bearer $TOKEN" "http://$GW/u/bob/" >/dev/null \ # Signup exactly as the landing page does it: a form POST, answered with a
|| fail "proxied web call failed" # session cookie. The browser never handles the DSH launch token.
curl -s -o /dev/null -w '%{http_code}' "http://$GW/u/bob/" | grep -q 401 \ SIGNUP_CODE=$(curl -s -o /tmp/dsh-gw-signup.out -w '%{http_code}' \
|| fail "tokenless request was not rejected" -c "$JAR" -H "Origin: http://$GW" \
pass "gateway proxy: authed request 200, tokenless 401" -X POST "http://$GW/users" -d 'user=bob')
case "$SIGNUP_CODE" in
201|202|303) ;;
*) fail "signup returned HTTP $SIGNUP_CODE: $(cat /tmp/dsh-gw-signup.out)" ;;
esac
grep -q 'dsh_gw_session' "$JAR" || fail "signup did not set the dsh_gw_session cookie"
STATE=""
for _ in $(seq 1 30); do
STATE=$(curl -s -b "$JAR" "http://$GW/users/bob" \
| sed -n 's/.*"state":"\([^"]*\)".*/\1/p')
[ "$STATE" = "Ready" ] && break
sleep 2
done
[ "$STATE" = "Ready" ] || fail "bob's claim was not Ready (state=$STATE)"
pass "gateway signup: bob Ready with an HttpOnly session cookie"
CT=$(curl -s -o /tmp/dsh-u-bob.html -w '%{content_type}' -b "$JAR" "http://$GW/u/bob/")
case "$CT" in text/html*) ;; *) fail "/u/bob/ returned content-type '$CT', expected text/html";; esac
grep -qi '<html' /tmp/dsh-u-bob.html || fail "/u/bob/ did not return the DSH Web HTML document"
pass "gateway proxy: /u/bob/ returns DSH Web HTML through path routing"
CODE=$(curl -s -o /dev/null -w '%{http_code}' "http://$GW/u/bob/")
[ "$CODE" = "401" ] || fail "tokenless /u/bob/ returned $CODE, expected 401"
pass "gateway proxy: tokenless request rejected with 401"
# Optional WebSocket check: the gateway must reject upgrades without its
# session cookie and bridge authenticated upgrades to the sandbox.
if python3 -c 'import websockets' >/dev/null 2>&1; then
python3 - "$JAR" "$GW" <<'PY' || fail "WebSocket checks failed"
import sys
from websockets.exceptions import ConnectionClosed, InvalidStatus
from websockets.sync.client import connect
jar_path, gw = sys.argv[1], sys.argv[2]
cookie = None
with open(jar_path, encoding="utf-8") as jar:
for raw in jar:
line = raw.removeprefix("#HttpOnly_")
if not line.strip() or line.startswith("#"):
continue
fields = line.rstrip("\n").split("\t")
if len(fields) >= 7 and fields[5] == "dsh_gw_session":
cookie = f"{fields[5]}={fields[6]}"
break
if not cookie:
sys.exit("FAIL: no dsh_gw_session cookie in the jar")
uri = f"ws://{gw}/u/bob/api/remote.mux"
def connect_ws(headers):
try:
return connect(uri, additional_headers=headers, open_timeout=30)
except TypeError: # websockets < 14
return connect(uri, extra_headers=headers, open_timeout=30)
def status_of(exc):
response = getattr(exc, "response", None)
return getattr(response, "status_code", None) or getattr(exc, "status_code", None)
# 1. Without the gateway cookie the gateway accepts the upgrade and then
# closes with application code 4401 (WebSocket auth has no HTTP 401 path).
try:
with connect(uri, open_timeout=10) as ws:
try:
ws.recv(timeout=10)
except ConnectionClosed as exc:
code = exc.rcvd.code if exc.rcvd else None
if code != 4401:
sys.exit(f"FAIL: anonymous WebSocket closed with {code}, expected 4401")
print("anonymous WebSocket rejected with close code 4401")
else:
sys.exit("FAIL: anonymous WebSocket stayed open")
except InvalidStatus as exc:
sys.exit(f"FAIL: anonymous WebSocket handshake returned HTTP {status_of(exc)}")
except Exception as exc: # noqa: BLE001 - report any handshake failure
sys.exit(f"FAIL: anonymous WebSocket check raised {exc!r}")
# 2. With the browser session cookie the upgrade is accepted and bridged.
# DSH may not serve a WebSocket on '/', so an upstream close (1011) is
# reported but tolerated; a 4401 would mean the gateway lost the cookie.
try:
with connect_ws({"Cookie": cookie, "Origin": f"http://{gw}"}) as ws:
try:
ws.recv(timeout=10)
print("authenticated WebSocket accepted and streaming")
except TimeoutError:
print("authenticated WebSocket accepted (no immediate close)")
except ConnectionClosed as exc:
code = exc.rcvd.code if exc.rcvd else None
if code == 4401:
sys.exit(
"FAIL: authenticated WebSocket was treated as unauthenticated (4401)"
)
print(f"authenticated WebSocket accepted; upstream closed with {code}")
except InvalidStatus as exc:
sys.exit(f"FAIL: authenticated WebSocket handshake returned HTTP {status_of(exc)}")
except Exception as exc: # noqa: BLE001 - report any handshake failure
sys.exit(f"FAIL: authenticated WebSocket check raised {exc!r}")
PY
pass "gateway WebSocket: anonymous rejected (4401), authenticated upgrade bridged"
else
echo "WARNING: python websockets not installed; skipping the WebSocket check" >&2
fi
echo "waiting for idle sweeper (~75s)..." echo "waiting for idle sweeper (~75s)..."
for _ in $(seq 1 30); do for _ in $(seq 1 30); do
STATE=$(curl -s -H "Authorization: Bearer $TOKEN" "http://$GW/users/bob" \ STATE=$(curl -s -b "$JAR" "http://$GW/users/bob" \
| sed -n 's/.*"state":"\([^"]*\)".*/\1/p') | sed -n 's/.*"state":"\([^"]*\)".*/\1/p')
[ "$STATE" = "Suspended" ] && break [ "$STATE" = "Suspended" ] && break
sleep 5 sleep 5
@@ -113,16 +229,19 @@ if command -v docker >/dev/null && command -v kind >/dev/null; then
pass "idle sweeper suspended bob" pass "idle sweeper suspended bob"
T0=$(date +%s) T0=$(date +%s)
curl -sf -m 150 -H "Authorization: Bearer $TOKEN" "http://$GW/u/bob/" >/dev/null \ curl -sf -m 150 -b "$JAR" "http://$GW/u/bob/" >/dev/null \
|| fail "wake-on-connect request failed" || fail "wake-on-connect request failed"
pass "wake-on-connect: request held and served after $(( $(date +%s) - T0 ))s" pass "wake-on-connect: request held and served after $(( $(date +%s) - T0 ))s"
curl -sf -X DELETE -H "Authorization: Bearer $TOKEN" "http://$GW/users/bob" >/dev/null \ curl -sf -X DELETE -b "$JAR" -H "Origin: http://$GW" \
"http://$GW/users/bob" >/dev/null \
|| fail "gateway user deletion failed" || fail "gateway user deletion failed"
kill "$PF" 2>/dev/null || true kill "$PF" 2>/dev/null || true
rm -f "$JAR"
pass "gateway delete: bob removed"
elif [ "${SKIP_GATEWAY:-}" = "1" ]; then elif [ "${SKIP_GATEWAY:-}" = "1" ]; then
echo "WARNING: gateway checks SKIPPED (SKIP_GATEWAY=1) — signup, auth," >&2 echo "WARNING: gateway checks SKIPPED (SKIP_GATEWAY=1) — signup, auth," >&2
echo "WARNING: proxy, idle-suspend and wake-on-connect were NOT validated" >&2 echo "WARNING: proxy, WebSocket, idle-suspend and wake-on-connect were NOT validated" >&2
GATEWAY_SKIPPED=1 GATEWAY_SKIPPED=1
else else
fail "docker + kind are required for the gateway checks (set SKIP_GATEWAY=1 to run only the resource-layer checks)" fail "docker + kind are required for the gateway checks (set SKIP_GATEWAY=1 to run only the resource-layer checks)"