mirror of
https://github.com/sredevopsorg/deepseek-kubernetes-sandbox.git
synced 2026-11-03 01:13:24 +00:00
Make DSH agents-as-a-service browser-usable; fix probe crash-loop (#3)
* Rewrite DSH gateway for browser sessions, HTTP and WebSocket - Replace the Flask proxy with a Starlette ASGI app (uvicorn factory), keeping the control plane (claims, wake/suspend, idle sweeper). - Add HttpOnly gateway sessions and a server-side DSH launch-token handshake: read the token from pod logs, exchange it for a dsh-auth cookie, cache per (user, pod UID, Host), refresh once on 401. - Proxy HTTP/SSE and WebSocket (/api/remote.mux) under /u/<user>/, with header stripping/injection, Location rewriting and wake-on-connect. - Add a landing/signup/login page, health endpoint, pinned dependencies, a read-only-rootfs gateway image, 87 cluster-free tests and docs. * Fix DSH probe crash-loop and harden gateway manifests - Probe /favicon.svg instead of /: DSH gates / behind its launch-token cookie and returns 401, which made warm-pool pods crash-loop. - Pass DSH_TRUSTED_HOSTS through the entrypoint as --trusted-host args for public/ingress hosts; keep localhost trusted. - Deploy the published :gateway image, add pods/pods-log RBAC, /healthz probes, read-only rootfs with a /tmp emptyDir and non-root security. - Add optional Traefik Ingress and document the prerequisites. * Update CI, smoke tests, build tooling and docs - Run gateway pytest/ruff and build/smoke-test the image in CI. - Parameterize the DSH build version and pnpm version inputs. - Exercise cookie signup, HTML proxy, WebSocket auth, suspend/wake and delete in run-test-kind.sh. - Add Makefile/build.sh targets and document the architecture, handshake, probes, trusted hosts, troubleshooting and limitations. --------- Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
This commit is contained in:
SREDevOps.org Agent
committed by
GitHub
GitHub
parent
c9be929212
commit
6301c21bea
@@ -50,20 +50,39 @@ spec:
|
||||
# gateway base URL if you want hard-coded absolute links instead.
|
||||
- name: DSH_PUBLIC_URL
|
||||
value: ""
|
||||
# DSH Host/Origin-fences /api. The gateway forwards the browser's
|
||||
# Host, so DSH must trust the public gateway host. Empty is right
|
||||
# for port-forward/localhost; for the optional Ingress set this to
|
||||
# the same host as 60-ingress.yaml (e.g. dsh.example.com).
|
||||
- name: DSH_TRUSTED_HOSTS
|
||||
value: ""
|
||||
# Probes target /favicon.svg: DSH gates / behind its launch-token
|
||||
# cookie and answers 401 to an unauthenticated GET, which would
|
||||
# otherwise fail readiness and then get the pod killed by liveness.
|
||||
# /favicon.svg is served without auth and is the cheapest liveness
|
||||
# signal that proves the HTTP server is actually serving requests.
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /
|
||||
path: /favicon.svg
|
||||
port: 3080
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
failureThreshold: 30
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /
|
||||
path: /favicon.svg
|
||||
port: 3080
|
||||
initialDelaySeconds: 15
|
||||
periodSeconds: 15
|
||||
failureThreshold: 3
|
||||
# The container only needs to serve HTTP; drop every capability and
|
||||
# disallow privilege escalation. DSH writes only to DSH_HOME (PVC).
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
capabilities:
|
||||
drop: ["ALL"]
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
resources:
|
||||
requests: {cpu: 250m, memory: 512Mi}
|
||||
limits: {cpu: "1", memory: 1Gi}
|
||||
|
||||
@@ -1,15 +1,28 @@
|
||||
# The gateway: control+data plane that turns the resources into a usable
|
||||
# service. Build the image first (see README step 8):
|
||||
# browser-facing service. It is a Starlette ASGI app (see gateway/) listening
|
||||
# on :8080 and exposing:
|
||||
# GET /healthz unauthenticated health probe
|
||||
# GET / landing page (signup / token login)
|
||||
# POST /users, POST /login, ... session cookie auth
|
||||
# /u/<user>/... HTTP + WebSocket proxy to the user's DSH
|
||||
#
|
||||
# Image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway
|
||||
# For local kind development you can build and load a local tag instead, then
|
||||
# change `image` below to it:
|
||||
# docker build -t dsh-aaas-gateway:demo gateway/
|
||||
# kind load docker-image dsh-aaas-gateway:demo # (kind clusters)
|
||||
# kind load docker-image dsh-aaas-gateway:demo
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: dsh-gateway
|
||||
namespace: dsh-demo
|
||||
---
|
||||
# Least privilege: claims are created/deleted per user; sandboxes are only
|
||||
# read and patched (operatingMode flips for suspend/resume).
|
||||
# Least privilege:
|
||||
# - claims are created/deleted per user (signup/delete);
|
||||
# - sandboxes are read, and patched only to flip operatingMode (suspend/resume);
|
||||
# - pods are read, and pods/log is read to capture DSH's launch token from
|
||||
# the startup log line (`dsh web: ...?token=...`).
|
||||
# configmaps are not needed: all state is in memory / claim annotations.
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: Role
|
||||
metadata:
|
||||
@@ -22,6 +35,12 @@ rules:
|
||||
- apiGroups: ["agents.x-k8s.io"]
|
||||
resources: ["sandboxes"]
|
||||
verbs: ["get", "list", "patch"]
|
||||
- apiGroups: [""]
|
||||
resources: ["pods"]
|
||||
verbs: ["get", "list"]
|
||||
- apiGroups: [""]
|
||||
resources: ["pods/log"]
|
||||
verbs: ["get"]
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: RoleBinding
|
||||
@@ -43,7 +62,11 @@ metadata:
|
||||
name: dsh-gateway
|
||||
namespace: dsh-demo
|
||||
spec:
|
||||
replicas: 1 # in-memory idle clock => single replica by design
|
||||
# Single replica BY DESIGN: the idle clock, in-flight counters and the
|
||||
# per-(user, pod, Host) DSH cookie cache are in-process memory. Running two
|
||||
# replicas would fragment that state (double suspend/resume races, repeated
|
||||
# log-token handshakes). Scale only after moving state to a shared store.
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: dsh-gateway
|
||||
@@ -61,7 +84,7 @@ spec:
|
||||
type: RuntimeDefault
|
||||
containers:
|
||||
- name: gateway
|
||||
image: dsh-aaas-gateway:demo
|
||||
image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway
|
||||
imagePullPolicy: IfNotPresent
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
@@ -69,17 +92,60 @@ spec:
|
||||
capabilities:
|
||||
drop: ["ALL"]
|
||||
ports:
|
||||
- containerPort: 8080
|
||||
- name: http
|
||||
containerPort: 8080
|
||||
env:
|
||||
- name: NAMESPACE
|
||||
value: dsh-demo
|
||||
- name: POOL
|
||||
value: dsh-pool
|
||||
# Seconds a sandbox may sit idle before the sweeper suspends it
|
||||
# (pod deleted, PVC kept), and the max time a proxy request waits
|
||||
# for a suspended sandbox to become Ready again.
|
||||
- name: IDLE_TIMEOUT
|
||||
value: "60"
|
||||
- name: WAKE_TIMEOUT
|
||||
value: "120"
|
||||
# Container name inside the per-user sandbox pods whose logs carry
|
||||
# the DSH launch token.
|
||||
- name: DSH_CONTAINER
|
||||
value: dsh-web
|
||||
# Cookie dsh_gw_session is HttpOnly + SameSite=Lax. Set to "true"
|
||||
# when this Service is reached over HTTPS (e.g. the optional
|
||||
# 60-ingress.yaml with TLS); "false" keeps the plain-HTTP demo
|
||||
# usable via port-forward.
|
||||
- name: COOKIE_SECURE
|
||||
value: "false"
|
||||
# Reject proxied request bodies larger than this (413).
|
||||
- name: MAX_BODY_MB
|
||||
value: "32"
|
||||
# /healthz is unauthenticated and returns 200 without touching the
|
||||
# cluster, so it is safe for both readiness and liveness.
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /healthz
|
||||
port: 8080
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
failureThreshold: 3
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /healthz
|
||||
port: 8080
|
||||
initialDelaySeconds: 10
|
||||
periodSeconds: 10
|
||||
failureThreshold: 3
|
||||
resources:
|
||||
requests: {cpu: 50m, memory: 128Mi}
|
||||
limits: {cpu: 500m, memory: 256Mi}
|
||||
volumeMounts:
|
||||
# readOnlyRootFilesystem leaves no writable HOME/tmp; the image sets
|
||||
# HOME=/tmp, so back it with scratch space. No state lives here.
|
||||
- name: tmp
|
||||
mountPath: /tmp
|
||||
volumes:
|
||||
- name: tmp
|
||||
emptyDir: {}
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
|
||||
46
examples/dsh-agents-as-a-service/60-ingress.yaml
Normal file
46
examples/dsh-agents-as-a-service/60-ingress.yaml
Normal file
@@ -0,0 +1,46 @@
|
||||
# OPTIONAL browser entry point: expose the gateway through Traefik.
|
||||
#
|
||||
# Requires 50-gateway.yaml (Service dsh-gateway:8080) and a Traefik install
|
||||
# with an IngressClass named "traefik" (k3s ships one). Replace the placeholder
|
||||
# host below with a name that resolves to your ingress address, and either
|
||||
# supply a real TLS Secret or delete the `tls:` block for plain HTTP.
|
||||
#
|
||||
# Two things must match this host:
|
||||
# 1. SandboxTemplate (10-sandboxtemplate.yaml): set DSH_TRUSTED_HOSTS to
|
||||
# this host so DSH accepts the forwarded Host/Origin on /api.
|
||||
# 2. Gateway Deployment (50-gateway.yaml): set COOKIE_SECURE to "true"
|
||||
# once this Ingress serves HTTPS, so the session cookie is Secure.
|
||||
#
|
||||
# WebSocket support is REQUIRED: the browser talks to /u/<user>/... and DSH
|
||||
# uses WebSockets for live session traffic. Traefik upgrades WS/SSE
|
||||
# transparently for ordinary HTTP routers, but do NOT attach buffering
|
||||
# middlewares (e.g. compress, buffering) to this router — they break streaming
|
||||
# and upgrades. `path: /` Prefix covers the landing page and every per-user
|
||||
# proxy path, so no per-sandbox Ingress is needed.
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: dsh-gateway
|
||||
namespace: dsh-demo
|
||||
# Typical Traefik + cert-manager wiring (uncomment and adapt):
|
||||
# annotations:
|
||||
# traefik.ingress.kubernetes.io/router.entrypoints: websecure
|
||||
# cert-manager.io/cluster-issuer: letsencrypt-prod
|
||||
annotations: {}
|
||||
spec:
|
||||
ingressClassName: traefik
|
||||
tls:
|
||||
- hosts:
|
||||
- dsh.example.com # placeholder
|
||||
secretName: dsh-example-tls # placeholder; create/PKI-provision this Secret
|
||||
rules:
|
||||
- host: dsh.example.com # placeholder
|
||||
http:
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: dsh-gateway
|
||||
port:
|
||||
number: 8080
|
||||
@@ -11,10 +11,11 @@ sessions, config) surviving on a per-user PVC at `/home/node/.dsh`.
|
||||
It ships in two layers:
|
||||
|
||||
1. **The resource layer** (steps 1–7): pure manifests + kubectl, so every mechanism is visible.
|
||||
2. **The service layer** (step 8): a small included **gateway** (`gateway/`) that makes it
|
||||
directly usable — signup API, per-user bearer tokens, a `/u/{user}/**` proxy that
|
||||
transparently **wakes suspended instances on connect**, and an idle sweeper that suspends
|
||||
them again.
|
||||
2. **The service layer** (step 8): a small included **gateway** (`gateway/`) that makes the
|
||||
platform browser-usable — a signup page/API, `HttpOnly` cookie sessions, a `/u/{user}/**`
|
||||
proxy for HTTP **and WebSocket** that transparently **wakes suspended instances on
|
||||
connect**, and an idle sweeper that suspends them again. The gateway performs the DSH
|
||||
launch-token handshake server-side, so the browser never sees the token.
|
||||
|
||||
What it demonstrates, feature by feature:
|
||||
|
||||
@@ -37,11 +38,14 @@ What it demonstrates, feature by feature:
|
||||
instead of splitting `/v1/*` (API) from the dashboard.
|
||||
- **One shared secret**: only `DEEPSEEK_API_KEY` is injected (via `envFrom`); there is no
|
||||
dashboard basic-auth, session-cookie secret, or API-server key.
|
||||
- **Host forwarding**: the gateway forwards the original `Host` + `X-Forwarded-*` (and keeps
|
||||
`DSH_PUBLIC_URL` empty) so DSH generates reachable links from the request.
|
||||
- **WebSocket caveat**: the requests-based streaming proxy handles SSE but not WebSocket
|
||||
upgrades. For full DSH web-GUI interactivity, front the gateway with a path-based Ingress
|
||||
per user (or a WS-capable proxy).
|
||||
- **Launch-token handshake**: DSH gates `/` behind a token it prints only in its logs. The
|
||||
gateway reads it, performs the handshake, and keeps the resulting auth cookie server-side;
|
||||
the browser authenticates to the gateway with its own cookie. See
|
||||
[How the DSH handshake works](#how-the-dsh-handshake-works).
|
||||
- **Host allow-list**: DSH rejects unknown `Host` headers. The gateway forwards the browser's
|
||||
`Host`, so the public hostname must be added to `DSH_TRUSTED_HOSTS` in the template.
|
||||
- **Probes target `/favicon.svg`**: `/` returns `401` without the launch token, so a probe on
|
||||
`/` crash-loops the pod. `/favicon.svg` is unauthenticated.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
@@ -55,7 +59,7 @@ What it demonstrates, feature by feature:
|
||||
```
|
||||
|
||||
Or run everything below in one shot with `./run-test-kind.sh` (expects a kind cluster + the
|
||||
manifest above already applied).
|
||||
manifest above already applied; it creates and deletes the namespace itself).
|
||||
|
||||
## 1. Install the platform resources
|
||||
|
||||
@@ -111,14 +115,20 @@ Three things worth noticing:
|
||||
|
||||
## 3. Talk to the instance
|
||||
|
||||
DSH Web needs its launch token on first contact — a bare `/` returns `401`. The token is in the
|
||||
pod logs:
|
||||
|
||||
```bash
|
||||
kubectl -n dsh-demo port-forward "$SB" 18081:3080
|
||||
# open http://localhost:18081 — DSH Web for alice
|
||||
kubectl -n dsh-demo logs "$SB" | grep -m1 '?token='
|
||||
# dsh web: http://<pod-ip>:3080/?token=...
|
||||
kubectl -n dsh-demo port-forward "pod/$SB" 18081:3080
|
||||
# open http://localhost:18081/?token=... (from the log line)
|
||||
```
|
||||
|
||||
> **Local port note:** DSH Web listens on `3080` **in-cluster**. If `3080` is already in use
|
||||
> on your workstation (e.g. by a local DSH), forward to a different local port (here `18081`),
|
||||
> never `3080`.
|
||||
> never `3080`. `localhost` is always in DSH's trusted-host list, so the forwarded request is
|
||||
> accepted.
|
||||
|
||||
## 4. Write state on the PVC
|
||||
|
||||
@@ -153,38 +163,169 @@ kubectl -n dsh-demo get sandboxclaim dsh-mallory \
|
||||
kubectl delete -f 40-claim-rejected.yaml
|
||||
```
|
||||
|
||||
## 8. The gateway: signup, proxy, idle-suspend, wake-on-connect
|
||||
## 8. The gateway: signup, browser proxy, idle-suspend, wake-on-connect
|
||||
|
||||
Build the gateway image and deploy it:
|
||||
### Deploy it
|
||||
|
||||
```bash
|
||||
docker build -t dsh-aaas-gateway:demo gateway/
|
||||
kind load docker-image dsh-aaas-gateway:demo # (kind clusters)
|
||||
# Published image (on kind, load it into the cluster; on a real cluster it is
|
||||
# pulled from GHCR):
|
||||
docker pull ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway
|
||||
kind load docker-image ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway --name <kind-cluster>
|
||||
|
||||
# ... or build it locally and tag it exactly as 50-gateway.yaml expects:
|
||||
docker build -t ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway gateway/
|
||||
kind load docker-image ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway --name <kind-cluster>
|
||||
|
||||
kubectl apply -f 50-gateway.yaml
|
||||
kubectl -n dsh-demo rollout status deploy/dsh-gateway --timeout=120s
|
||||
kubectl -n dsh-demo port-forward svc/dsh-gateway 18080:8080
|
||||
```
|
||||
|
||||
Then:
|
||||
> The gateway binds `8080`; the local port-forward uses `18080` and never `3080`.
|
||||
>
|
||||
> **Applying over an existing install?** Run `kubectl diff -f 10-sandboxtemplate.yaml`
|
||||
> first. Cluster-specific fields such as `nodeAffinity` and the warm-pool replica count are
|
||||
> intentionally not in the repo manifests; preserve them with a targeted patch instead of a
|
||||
> blind apply. Moving the probes to `/favicon.svg` is the crash-loop fix.
|
||||
|
||||
### Browser flow
|
||||
|
||||
Open <http://localhost:18080>. The landing page has a one-field signup form. Enter `bob`, and
|
||||
the gateway:
|
||||
|
||||
1. creates the `SandboxClaim` over the warm pool,
|
||||
2. waits for it to become `Ready`,
|
||||
3. sets an `HttpOnly`, `SameSite=Lax` cookie `dsh_gw_session=<user>:<token>` on the browser,
|
||||
4. redirects to `/u/bob/`, where the DSH Web UI loads through the gateway.
|
||||
|
||||
From then on every request under `/u/bob/` (assets, API calls, SSE, WebSocket) is proxied to
|
||||
bob's sandbox. The DSH launch token and DSH's own auth cookie stay in the gateway; the browser
|
||||
only holds the gateway cookie.
|
||||
|
||||
The same flow with `curl`, no JavaScript (the session cookie is the only credential):
|
||||
|
||||
```bash
|
||||
# signup → claim over the warm pool + one-time bearer token
|
||||
SIGNUP=$(curl -s -X POST http://localhost:18080/users -H 'Content-Type: application/json' -d '{"user":"bob"}')
|
||||
TOKEN=$(echo "$SIGNUP" | sed -n 's/.*"token":"\([^"]*\)".*/\1/p')
|
||||
|
||||
# proxy to bob's DSH Web (the gateway wakes it if suspended)
|
||||
curl -s -H "Authorization: Bearer $TOKEN" http://localhost:18080/u/bob/
|
||||
|
||||
# tokenless is rejected
|
||||
JAR=$(mktemp)
|
||||
# signup (form POST, exactly what the landing page sends)
|
||||
curl -s -i -c "$JAR" -H 'Origin: http://localhost:18080' \
|
||||
-d 'user=bob' http://localhost:18080/users
|
||||
# proxy: bob's DSH Web through the gateway
|
||||
curl -s -b "$JAR" http://localhost:18080/u/bob/ | head -5
|
||||
# without the cookie the gateway fails closed
|
||||
curl -s -o /dev/null -w '%{http_code}\n' http://localhost:18080/u/bob/ # 401
|
||||
|
||||
# after IDLE_TIMEOUT (60s) the sweeper suspends bob
|
||||
curl -s -H "Authorization: Bearer $TOKEN" http://localhost:18080/users/bob # state: Suspended
|
||||
|
||||
# derived state: Ready / Waking / Suspended / ...
|
||||
curl -s -b "$JAR" http://localhost:18080/users/bob
|
||||
# delete bob (cascades sandbox + PVC)
|
||||
curl -s -X DELETE -H "Authorization: Bearer $TOKEN" http://localhost:18080/users/bob
|
||||
curl -s -b "$JAR" -H 'Origin: http://localhost:18080' -X DELETE http://localhost:18080/users/bob
|
||||
```
|
||||
|
||||
API clients that prefer bearer tokens can use `POST /users` with a JSON body
|
||||
(`{"user":"bob"}`) and the returned one-time token, `POST /login` to exchange a token for a
|
||||
cookie, and `Authorization: Bearer <token>` on subsequent calls.
|
||||
|
||||
### Gateway API
|
||||
|
||||
| Route | Purpose |
|
||||
| --- | --- |
|
||||
| `GET /healthz` | Liveness/readiness; no cluster access |
|
||||
| `GET /` | Landing page: signup, token login, logout |
|
||||
| `POST /users` | Signup. JSON or form; `201` (JSON), `202` (waking), `303` (browser redirect), `409` if user exists; sets the session cookie |
|
||||
| `POST /login` | Exchange a bearer token for the session cookie |
|
||||
| `POST /logout` | Clear the session cookie |
|
||||
| `GET /users/{user}` | Derived state (`Ready`/`Waking`/`Suspended`) |
|
||||
| `DELETE /users/{user}` | Delete claim (cascades sandbox + PVC) |
|
||||
| `GET /u/{user}` | `308` redirect to `/u/{user}/` |
|
||||
| `GET /u/{user}/**` | HTTP/SSE proxy to the user's DSH Web |
|
||||
| `WS /u/{user}/**` | WebSocket proxy to the user's DSH Web |
|
||||
|
||||
### How the DSH handshake works
|
||||
|
||||
DSH does not expose an API for its launch token; it prints a URL containing `?token=…` in the
|
||||
pod logs. For each `(user, pod UID, Host)` the gateway:
|
||||
|
||||
1. resolves the claim → sandbox → pod (annotation `agents.x-k8s.io/pod-name`, or the sandbox
|
||||
name as pod name),
|
||||
2. reads the pod's logs (last `DSH_LOG_TAIL_LINES` lines) and parses the most recent
|
||||
`dsh web: …?token=…` line,
|
||||
3. requests `http://<serviceFQDN>:3080/?token=…` with the browser's `Host`,
|
||||
4. captures DSH's `dsh-auth-*` `Set-Cookie` response and caches it,
|
||||
5. injects that cookie (and strips the browser's `Cookie`/`Authorization` and any `?token=`)
|
||||
on every proxied HTTP and WebSocket request,
|
||||
6. refreshes the handshake once if DSH answers `401`, and drops the cached cookie when the
|
||||
sandbox suspends or is deleted.
|
||||
|
||||
This is why the gateway's RBAC includes `pods` `get`/`list` and `pods/log` `get`. It is also
|
||||
the most fragile part of the design; see [Limitations](#limitations).
|
||||
|
||||
### DSH_TRUSTED_HOSTS
|
||||
|
||||
DSH rejects requests whose `Host` is not allow-listed (`localhost` and `127.0.0.1` are always
|
||||
trusted). Because the gateway forwards the browser's `Host`, set the public hostname in the
|
||||
template:
|
||||
|
||||
```yaml
|
||||
env:
|
||||
- name: DSH_TRUSTED_HOSTS
|
||||
value: "dsh.example.com" # comma or space separated; empty for port-forward only
|
||||
```
|
||||
|
||||
The same applies to a local `docker run`: the printed token URL uses `DSH_PUBLIC_URL` if set,
|
||||
otherwise `http://localhost:${DSH_PORT}`.
|
||||
|
||||
### WebSocket
|
||||
|
||||
The gateway is a Starlette ASGI app and proxies WebSocket upgrades with a bidirectional pump, so
|
||||
the DSH web GUI is fully interactive. Requirements:
|
||||
|
||||
- the ingress/load balancer in front of the gateway must pass `Upgrade`/`Connection` headers;
|
||||
`60-ingress.yaml` is a starting point (Traefik handles upgrades by default),
|
||||
- idle suspension is disabled for users with in-flight proxied requests or open WebSockets, so
|
||||
a long-running chat stream is not cut off mid-turn,
|
||||
- missing/invalid gateway auth is rejected right after the upgrade with close code `4401`
|
||||
(WebSocket auth has no HTTP `401` path); cross-origin or Origin-less upgrades with the cookie
|
||||
are rejected the same way (CSWSH protection), so clients must send a matching `Origin`;
|
||||
upstream bridging failures close with `1011`.
|
||||
|
||||
### Path routing
|
||||
|
||||
Each user's DSH Web is served under `/u/<user>/`. DSH injects `<base href="./">` and uses
|
||||
relative asset/API URLs, so the UI works under a path prefix without a per-user hostname or
|
||||
wildcard DNS. The gateway also rewrites absolute-path `Location` headers into `/u/<user>/…` and
|
||||
redirects `/u/<user>` to the trailing-slash form with `308` so relative URLs resolve correctly.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Cause / fix |
|
||||
| --- | --- |
|
||||
| Pod `CrashLoopBackOff`, probes failing with `401` | Probe path is `/`; change readiness/liveness to `/favicon.svg` |
|
||||
| Browser shows `401` from DSH through the gateway | Gateway handshake failed — check `kubectl -n dsh-demo logs deploy/dsh-gateway` and that RBAC grants `pods/log` |
|
||||
| `502` / `upstream unavailable` | Sandbox pod is not Ready (wake timed out) or the Service DNS is stale; check `kubectl get sandbox`, `WAKE_TIMEOUT` |
|
||||
| `503` with `Retry-After` | The launch token was not found in the pod logs yet; the gateway retries after a short delay. Check the pod logs and `DSH_LOG_TAIL_LINES` |
|
||||
| `409` on signup | User already has a claim; use `POST /login` with the existing token or delete the user first |
|
||||
| Cookie set but browser keeps getting `401` | `COOKIE_SECURE=true` while serving over plain HTTP (secure cookies are not sent); use HTTPS or set `false` for local demos |
|
||||
| WebSocket closes immediately (`4401`) | Missing/expired gateway cookie, or the upgrade's `Origin` does not match `Host` (CSWSH check); DSH rejecting `Host` instead surfaces as close `1011` — add the host to `DSH_TRUSTED_HOSTS` |
|
||||
| Signup is slow / `202` then `Waking` | Warm pool exhausted (`kubectl get sandboxes -l agents.x-k8s.io/warm-pool-sandbox`); scale the pool |
|
||||
| `403` from the Kubernetes API in gateway logs | The gateway ServiceAccount is missing claim/sandbox/pod-log verbs (see `50-gateway.yaml`) |
|
||||
|
||||
## Limitations
|
||||
|
||||
Deliberately **not** production code:
|
||||
|
||||
- **Single replica.** Idle clocks, in-flight tracking, and the handshake cache are in memory.
|
||||
Running two gateway replicas would double-suspend or lose wake tracking; scale up only after
|
||||
moving that state to a shared store.
|
||||
- **Log scraping.** The launch token is recovered from pod logs. Log rotation, a restarted pod,
|
||||
or a changed DSH log format can break the handshake until the next `401` refresh.
|
||||
- **Shared provider key.** `DEEPSEEK_API_KEY` is shared by all tenants via the template. Real
|
||||
multi-tenancy needs per-tenant credentials or an egress proxy.
|
||||
- **No TLS in the gateway.** Terminate TLS at an Ingress/reverse proxy and set
|
||||
`COOKIE_SECURE=true`.
|
||||
- **In-memory user directory.** Users are claims + annotations; there is no database, password
|
||||
reset, or admin API. Tokens are stored only as SHA-256 hashes in claim annotations.
|
||||
- **Bearer tokens in URLs.** Avoid putting the signup token in a URL/log; the browser flow uses
|
||||
the `HttpOnly` cookie instead.
|
||||
|
||||
## Cost model
|
||||
|
||||
The same economics as the upstream pattern: suspend-when-idle plus warm pools turn an
|
||||
|
||||
@@ -1,7 +1,18 @@
|
||||
FROM python:3.12-slim
|
||||
|
||||
# Read-only-rootfs friendly: no .pyc writes, no home directory writes.
|
||||
ENV PYTHONDONTWRITEBYTECODE=1 \
|
||||
PYTHONUNBUFFERED=1 \
|
||||
HOME=/tmp
|
||||
|
||||
WORKDIR /app
|
||||
COPY requirements.txt .
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
COPY gateway.py .
|
||||
|
||||
EXPOSE 8080
|
||||
# Run as a fixed non-root UID; HOME=/tmp keeps caches out of the image.
|
||||
USER 10000
|
||||
CMD ["python", "gateway.py"]
|
||||
|
||||
CMD ["uvicorn", "gateway:create_app", "--factory", \
|
||||
"--host", "0.0.0.0", "--port", "8080", "--no-access-log"]
|
||||
|
||||
184
examples/dsh-agents-as-a-service/gateway/README.md
Normal file
184
examples/dsh-agents-as-a-service/gateway/README.md
Normal file
@@ -0,0 +1,184 @@
|
||||
# DSH agents-as-a-service gateway
|
||||
|
||||
A single-file, browser-usable control + data plane for
|
||||
[DeepSeek Harness Web (DSH)](https://github.com/deepseek-ai/deepseek-harness)
|
||||
sandboxes running behind the `agent-sandbox` CRDs.
|
||||
|
||||
It does three things:
|
||||
|
||||
1. **Users** — creates a `SandboxClaim` from a warm pool, mints a bearer token
|
||||
(only its SHA-256 is stored as a claim annotation), reports derived state,
|
||||
and deletes the claim (cascading the sandbox and PVC).
|
||||
2. **Browser sessions** — exchanges a user token for an HttpOnly session
|
||||
cookie so the DSH UI works in a browser at `/u/<user>/`. Automation can
|
||||
still use `Authorization: Bearer <token>`.
|
||||
3. **Reverse proxy** — proxies HTTP (including SSE) and WebSocket traffic to
|
||||
the sandbox's DSH Web server on `:3080`, including DSH's own cookie
|
||||
handshake.
|
||||
|
||||
DSH is pinned to `dsh-v0.2.1-rc.2` in the example manifests. It binds `:3080`,
|
||||
serves `/`, gates `/api` behind a `dsh-auth-*` cookie and requires a WebSocket
|
||||
at `/api/remote.mux`.
|
||||
|
||||
## Flow
|
||||
|
||||
```
|
||||
browser / curl gateway cluster
|
||||
| | |
|
||||
| POST /users {"user":"alice"} | |
|
||||
|------------------------------>| create SandboxClaim ---------->|
|
||||
| | wait Ready |
|
||||
| 201 {token} + session cookie | |
|
||||
|<------------------------------| |
|
||||
| GET /u/alice/ (cookie) | |
|
||||
|------------------------------>| claim -> sandbox -> pod |
|
||||
| | read pod logs (pods/log) |
|
||||
| | find "dsh web: ...?token=..." |
|
||||
| | GET http://<fqdn>:3080/?token= |
|
||||
| | capture dsh-auth-* Set-Cookie |
|
||||
| | proxy with Host/Origin kept |
|
||||
| HTML / SSE / WebSocket |<----------------------------->|
|
||||
|<------------------------------| |
|
||||
```
|
||||
|
||||
The launch token is printed by DSH to its pod stdout:
|
||||
|
||||
```
|
||||
dsh web: http://<podIP>:3080/?token=<base64url>
|
||||
```
|
||||
|
||||
The gateway reads the tail of the pod logs, parses the **last** such line and
|
||||
performs the `?token=` handshake itself with the browser's `Host` header. The
|
||||
resulting `dsh-auth-*` cookie is cached per `(user, pod UID, Host)` and injected
|
||||
into proxied requests. The browser never sees the launch token or the bearer
|
||||
token. A failed handshake returns `503` with `Retry-After: 10`.
|
||||
|
||||
## API
|
||||
|
||||
| Method | Path | Auth | Purpose |
|
||||
| --- | --- | --- | --- |
|
||||
| `GET` | `/healthz` | none | Liveness/readiness. Never touches the cluster. |
|
||||
| `GET` | `/` | none | Landing page with signup + token login forms. |
|
||||
| `POST` | `/users` | none | Signup. JSON body `{"user": "<dns-1123 label>"}` or form field `user`. JSON returns `201` when Ready, `202` when accepted but still Waking; form returns `303` to `/u/<user>/`. Both set the session cookie. `409` if the user exists, `400` if invalid, `413` if the body is too large. |
|
||||
| `POST` | `/login` | none | Form (`user`, `token`) returns `303` to `/u/<user>/`; JSON returns `200`. Sets the session cookie. `401` on a bad token, `404` for an unknown user. |
|
||||
| `POST` | `/logout` | none | Clears the session cookie and `303`s to `/`. |
|
||||
| `GET` | `/users/{user}` | bearer or cookie | Derived state: `Provisioning`, `Waking`, `Ready`, `Suspending`, `Suspended`. |
|
||||
| `DELETE` | `/users/{user}` | bearer or cookie | Delete the claim (sandbox + PVC cascade). Cookie-auth mutations require a same-origin `Origin`/`Referer`. |
|
||||
| `GET` | `/u/{user}` | — | `308` redirect to `/u/{user}/`. |
|
||||
| `*` | `/u/{user}/{path}` | bearer or cookie | HTTP proxy to DSH. Waits for a suspended sandbox to wake. Wakes on connect. |
|
||||
| `WS` | `/u/{user}/{path}` | bearer or cookie | WebSocket proxy (e.g. `/api/remote.mux`). |
|
||||
|
||||
### Auth
|
||||
|
||||
* `Authorization: Bearer <token>` is checked with a timing-safe SHA-256
|
||||
comparison against the `aaas.example.com/token-sha256` claim annotation.
|
||||
* The browser session cookie is
|
||||
`dsh_gw_session=<user>:<token>` (HttpOnly, SameSite=Lax, Path=/, optional
|
||||
Secure). A session for user A never authorizes user B.
|
||||
* Cookie-authenticated **mutations** (`POST`, `PUT`, `PATCH`, `DELETE`) and
|
||||
cookie-authenticated **WebSocket upgrades** require the request `Origin`
|
||||
(or `Referer` fallback) to match the `Host` header. Missing or mismatched
|
||||
cross-origin metadata is rejected (`403` HTTP, close code `4401` for WS).
|
||||
* The gateway's bearer token, the browser `Cookie` header and any `?token=`
|
||||
query parameter are never forwarded upstream. Only the cached `dsh-auth-*`
|
||||
cookie is injected.
|
||||
* Upstream `Set-Cookie` headers are dropped. `Location` headers that are
|
||||
absolute paths are rewritten under `/u/<user>/`, and any `token` query
|
||||
parameter is stripped.
|
||||
|
||||
### Proxy failure semantics
|
||||
|
||||
| Situation | Result |
|
||||
| --- | --- |
|
||||
| Unknown user / claim | `404` |
|
||||
| Missing or invalid credentials | `401` |
|
||||
| Cross-origin cookie mutation | `403` |
|
||||
| Request body over `MAX_BODY_MB` | `413` |
|
||||
| Sandbox not provisioned, wake timeout, handshake failure | `503` + `Retry-After: 10` |
|
||||
| Upstream connection/protocol failure | `502` |
|
||||
| WS auth failure (unknown user, bad token, cross-origin) | accept, then close `4401` |
|
||||
| WS upstream/handshake failure | accept, then close `1011` |
|
||||
|
||||
A close code can only be delivered after the WebSocket upgrade is accepted;
|
||||
Starlette would turn an early close into an HTTP 403, which is why WS auth
|
||||
failures are `101`, then close `4401`.
|
||||
|
||||
## Configuration
|
||||
|
||||
All values are read from the environment by `Settings.from_env()`; every one
|
||||
can also be injected into `create_app` for tests.
|
||||
|
||||
| Variable | Default | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `NAMESPACE` | `dsh-demo` | Namespace holding the claims, sandboxes and pods. |
|
||||
| `POOL` | `dsh-pool` | `warmPoolRef.name` for new claims. |
|
||||
| `IDLE_TIMEOUT` | `60` | Seconds of inactivity before the sweeper suspends a sandbox. |
|
||||
| `WAKE_TIMEOUT` | `120` | Seconds to wait for a sandbox to become Ready (signup and wake-on-connect). |
|
||||
| `DSH_PORT` | `3080` | Upstream DSH port. |
|
||||
| `DSH_CONTAINER` | `dsh-web` | Container whose logs carry the launch token. |
|
||||
| `DSH_LOG_TAIL_LINES` | `2000` | How many log lines to scan for the launch token. |
|
||||
| `COOKIE_NAME` | `dsh_gw_session` | Browser session cookie name. |
|
||||
| `COOKIE_SECURE` | `false` | Add `Secure` to the session cookie when served over TLS. |
|
||||
| `COOKIE_MAX_AGE` | `604800` | Session cookie lifetime in seconds. |
|
||||
| `MAX_BODY_MB` | `32` | Maximum buffered request body (and `413` threshold). |
|
||||
| `HTTP_TIMEOUT` | `300` | Upstream connect/read timeout in seconds. |
|
||||
| `SWEEP_INTERVAL` | `15` | Seconds between idle sweeps. |
|
||||
|
||||
### Required RBAC
|
||||
|
||||
The gateway's ServiceAccount needs, in its namespace:
|
||||
|
||||
* `sandboxclaims` (`extensions.agents.x-k8s.io`): get, list, create, delete
|
||||
* `sandboxes` (`agents.x-k8s.io`): get, patch
|
||||
* `pods`: get
|
||||
* `pods/log`: get
|
||||
|
||||
## Running locally
|
||||
|
||||
The module has no import-time cluster access and the Kubernetes adapter is
|
||||
lazy, so `/healthz` works without a kubeconfig:
|
||||
|
||||
```bash
|
||||
python -m pip install -r requirements-dev.txt
|
||||
python -m pytest -q
|
||||
python -m ruff check .
|
||||
|
||||
# Uvicorn factory (loads in-cluster credentials only on the first claim call)
|
||||
uvicorn gateway:create_app --factory --host 0.0.0.0 --port 8080
|
||||
```
|
||||
|
||||
Container image:
|
||||
|
||||
```bash
|
||||
docker build -t dsh-gateway examples/dsh-agents-as-a-service/gateway
|
||||
```
|
||||
|
||||
The image runs as UID `10000`, sets `PYTHONDONTWRITEBYTECODE=1` and
|
||||
`HOME=/tmp` (read-only-rootfs compatible), listens on `8080`, and disables
|
||||
uvicorn's access log so bearer tokens in URLs are not logged.
|
||||
|
||||
## Known limitations
|
||||
|
||||
* **Log scraping is fragile.** The launch token is discovered from pod logs.
|
||||
Log rotation, a very chatty container, or DSH changing its startup message
|
||||
breaks the handshake. Prefer a first-class token/credential API when DSH
|
||||
offers one.
|
||||
* **Single replica, in-memory state.** The handshake cookie cache, idle
|
||||
activity clocks and in-flight counters live in one process. Running more
|
||||
than one replica would need shared state (or sticky routing), otherwise each
|
||||
replica handshakes independently and the sweeper may race another replica.
|
||||
* **In-memory token cache has no TTL.** It is keyed by pod UID, so pod
|
||||
restarts invalidate it naturally; otherwise entries survive until the pod or
|
||||
claim is deleted.
|
||||
* **No TLS termination.** Run behind an ingress/load balancer; set
|
||||
`COOKIE_SECURE=true` there.
|
||||
* **Origin-based CSRF protection only.** Cookie-auth mutations require a
|
||||
same-origin `Origin`/`Referer`; there is no per-form CSRF token, rate
|
||||
limiting or login throttling.
|
||||
* **HTTP request bodies are buffered** up to `MAX_BODY_MB`; uploads are not
|
||||
streamed. Responses (including SSE) are streamed.
|
||||
* **Upstream cookies are not forwarded.** DSH cannot rotate its own
|
||||
`dsh-auth-*` cookie through the gateway; on an upstream `401` the gateway
|
||||
re-runs the handshake once and replays the request.
|
||||
* **Only DSH on `:3080` is proxied.** There is no dashboard/API-port split and
|
||||
no platform API key injection.
|
||||
File diff suppressed because it is too large
Load Diff
14
examples/dsh-agents-as-a-service/gateway/pyproject.toml
Normal file
14
examples/dsh-agents-as-a-service/gateway/pyproject.toml
Normal file
@@ -0,0 +1,14 @@
|
||||
[tool.pytest.ini_options]
|
||||
asyncio_mode = "auto"
|
||||
testpaths = ["."]
|
||||
|
||||
[tool.ruff]
|
||||
target-version = "py312"
|
||||
line-length = 100
|
||||
|
||||
[tool.ruff.lint]
|
||||
select = ["E", "F", "I", "B", "UP", "PIE", "RUF"]
|
||||
ignore = ["RUF001", "RUF002", "RUF003"]
|
||||
|
||||
[tool.ruff.lint.per-file-ignores]
|
||||
"test_gateway.py" = ["E501"]
|
||||
@@ -0,0 +1,4 @@
|
||||
-r requirements.txt
|
||||
pytest==9.1.1
|
||||
pytest-asyncio==1.4.0
|
||||
ruff==0.17.1
|
||||
@@ -1,3 +1,6 @@
|
||||
flask==3.1.*
|
||||
requests==2.32.*
|
||||
kubernetes==31.0.*
|
||||
starlette==1.9.0
|
||||
uvicorn[standard]==0.54.0
|
||||
httpx==0.28.1
|
||||
websockets==17.3
|
||||
kubernetes==37.0.1
|
||||
python-multipart==0.0.32
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -12,7 +12,11 @@ cd "$(dirname "$0")"
|
||||
NS=dsh-demo
|
||||
pass() { echo "PASS: $1"; }
|
||||
fail() { echo "FAIL: $1" >&2; exit 1; }
|
||||
cleanup() { kubectl delete namespace "$NS" --ignore-not-found --wait=false >/dev/null 2>&1 || true; }
|
||||
cleanup() {
|
||||
kill "${PF:-}" 2>/dev/null || true
|
||||
rm -f "${JAR:-}"
|
||||
kubectl delete namespace "$NS" --ignore-not-found --wait=false >/dev/null 2>&1 || true
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
echo "=== 1. install prereqs (throwaway key) + template + warm pool"
|
||||
@@ -77,34 +81,146 @@ BOUND=$(kubectl -n "$NS" get sandboxclaim dsh-mallory -o jsonpath='{.status.sand
|
||||
kubectl delete -f 40-claim-rejected.yaml >/dev/null
|
||||
pass "claim rejected with EnvVarsInjectionRejected and never bound"
|
||||
|
||||
echo "=== 7. gateway: signup, proxy, idle-suspend, wake-on-connect"
|
||||
echo "=== 7. gateway: browser signup, proxy, idle-suspend, wake-on-connect"
|
||||
if command -v docker >/dev/null && command -v kind >/dev/null; then
|
||||
KIND_CLUSTER="${KIND_CLUSTER:-$(kubectl config current-context | sed 's/^kind-//')}"
|
||||
docker build -q -t dsh-aaas-gateway:demo gateway/ >/dev/null
|
||||
kind load docker-image dsh-aaas-gateway:demo --name "$KIND_CLUSTER" >/dev/null
|
||||
# Tag exactly what 50-gateway.yaml references so the smoke test exercises
|
||||
# the shipped manifest unchanged (imagePullPolicy: IfNotPresent uses the
|
||||
# image loaded into kind).
|
||||
GW_IMAGE=ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway
|
||||
docker build -q -t "$GW_IMAGE" gateway/ >/dev/null
|
||||
kind load docker-image "$GW_IMAGE" --name "$KIND_CLUSTER" >/dev/null
|
||||
kubectl apply -f 50-gateway.yaml
|
||||
kubectl -n "$NS" rollout status deploy/dsh-gateway --timeout=120s
|
||||
# Local port-forward uses 18080 (NOT 3080, which stays in-cluster).
|
||||
kubectl -n "$NS" port-forward svc/dsh-gateway 18080:8080 >/dev/null 2>&1 &
|
||||
PF=$!
|
||||
sleep 3
|
||||
GW=localhost:18080
|
||||
JAR=$(mktemp)
|
||||
|
||||
SIGNUP=$(curl -s -X POST "http://$GW/users" -H 'Content-Type: application/json' -d '{"user":"bob"}')
|
||||
TOKEN=$(echo "$SIGNUP" | sed -n 's/.*"token":"\([^"]*\)".*/\1/p')
|
||||
[ -n "$TOKEN" ] || fail "signup returned no token: $SIGNUP"
|
||||
echo "$SIGNUP" | grep -q '"state":"Ready"' || fail "signup not Ready: $SIGNUP"
|
||||
pass "gateway signup: bob Ready with one-time token"
|
||||
for _ in $(seq 1 20); do
|
||||
HEALTH=$(curl -s -o /dev/null -w '%{http_code}' "http://$GW/healthz" || true)
|
||||
[ "$HEALTH" = "200" ] && break
|
||||
sleep 1
|
||||
done
|
||||
[ "${HEALTH:-}" = "200" ] || fail "gateway /healthz was not reachable on $GW"
|
||||
pass "gateway health: /healthz -> 200"
|
||||
|
||||
curl -sf -H "Authorization: Bearer $TOKEN" "http://$GW/u/bob/" >/dev/null \
|
||||
|| fail "proxied web call failed"
|
||||
curl -s -o /dev/null -w '%{http_code}' "http://$GW/u/bob/" | grep -q 401 \
|
||||
|| fail "tokenless request was not rejected"
|
||||
pass "gateway proxy: authed request 200, tokenless 401"
|
||||
# Signup exactly as the landing page does it: a form POST, answered with a
|
||||
# session cookie. The browser never handles the DSH launch token.
|
||||
SIGNUP_CODE=$(curl -s -o /tmp/dsh-gw-signup.out -w '%{http_code}' \
|
||||
-c "$JAR" -H "Origin: http://$GW" \
|
||||
-X POST "http://$GW/users" -d 'user=bob')
|
||||
case "$SIGNUP_CODE" in
|
||||
201|202|303) ;;
|
||||
*) fail "signup returned HTTP $SIGNUP_CODE: $(cat /tmp/dsh-gw-signup.out)" ;;
|
||||
esac
|
||||
grep -q 'dsh_gw_session' "$JAR" || fail "signup did not set the dsh_gw_session cookie"
|
||||
STATE=""
|
||||
for _ in $(seq 1 30); do
|
||||
STATE=$(curl -s -b "$JAR" "http://$GW/users/bob" \
|
||||
| sed -n 's/.*"state":"\([^"]*\)".*/\1/p')
|
||||
[ "$STATE" = "Ready" ] && break
|
||||
sleep 2
|
||||
done
|
||||
[ "$STATE" = "Ready" ] || fail "bob's claim was not Ready (state=$STATE)"
|
||||
pass "gateway signup: bob Ready with an HttpOnly session cookie"
|
||||
|
||||
CT=$(curl -s -o /tmp/dsh-u-bob.html -w '%{content_type}' -b "$JAR" "http://$GW/u/bob/")
|
||||
case "$CT" in text/html*) ;; *) fail "/u/bob/ returned content-type '$CT', expected text/html";; esac
|
||||
grep -qi '<html' /tmp/dsh-u-bob.html || fail "/u/bob/ did not return the DSH Web HTML document"
|
||||
pass "gateway proxy: /u/bob/ returns DSH Web HTML through path routing"
|
||||
|
||||
CODE=$(curl -s -o /dev/null -w '%{http_code}' "http://$GW/u/bob/")
|
||||
[ "$CODE" = "401" ] || fail "tokenless /u/bob/ returned $CODE, expected 401"
|
||||
pass "gateway proxy: tokenless request rejected with 401"
|
||||
|
||||
# Optional WebSocket check: the gateway must reject upgrades without its
|
||||
# session cookie and bridge authenticated upgrades to the sandbox.
|
||||
if python3 -c 'import websockets' >/dev/null 2>&1; then
|
||||
python3 - "$JAR" "$GW" <<'PY' || fail "WebSocket checks failed"
|
||||
import sys
|
||||
|
||||
from websockets.exceptions import ConnectionClosed, InvalidStatus
|
||||
from websockets.sync.client import connect
|
||||
|
||||
jar_path, gw = sys.argv[1], sys.argv[2]
|
||||
cookie = None
|
||||
with open(jar_path, encoding="utf-8") as jar:
|
||||
for raw in jar:
|
||||
line = raw.removeprefix("#HttpOnly_")
|
||||
if not line.strip() or line.startswith("#"):
|
||||
continue
|
||||
fields = line.rstrip("\n").split("\t")
|
||||
if len(fields) >= 7 and fields[5] == "dsh_gw_session":
|
||||
cookie = f"{fields[5]}={fields[6]}"
|
||||
break
|
||||
if not cookie:
|
||||
sys.exit("FAIL: no dsh_gw_session cookie in the jar")
|
||||
|
||||
uri = f"ws://{gw}/u/bob/api/remote.mux"
|
||||
|
||||
|
||||
def connect_ws(headers):
|
||||
try:
|
||||
return connect(uri, additional_headers=headers, open_timeout=30)
|
||||
except TypeError: # websockets < 14
|
||||
return connect(uri, extra_headers=headers, open_timeout=30)
|
||||
|
||||
|
||||
def status_of(exc):
|
||||
response = getattr(exc, "response", None)
|
||||
return getattr(response, "status_code", None) or getattr(exc, "status_code", None)
|
||||
|
||||
|
||||
# 1. Without the gateway cookie the gateway accepts the upgrade and then
|
||||
# closes with application code 4401 (WebSocket auth has no HTTP 401 path).
|
||||
try:
|
||||
with connect(uri, open_timeout=10) as ws:
|
||||
try:
|
||||
ws.recv(timeout=10)
|
||||
except ConnectionClosed as exc:
|
||||
code = exc.rcvd.code if exc.rcvd else None
|
||||
if code != 4401:
|
||||
sys.exit(f"FAIL: anonymous WebSocket closed with {code}, expected 4401")
|
||||
print("anonymous WebSocket rejected with close code 4401")
|
||||
else:
|
||||
sys.exit("FAIL: anonymous WebSocket stayed open")
|
||||
except InvalidStatus as exc:
|
||||
sys.exit(f"FAIL: anonymous WebSocket handshake returned HTTP {status_of(exc)}")
|
||||
except Exception as exc: # noqa: BLE001 - report any handshake failure
|
||||
sys.exit(f"FAIL: anonymous WebSocket check raised {exc!r}")
|
||||
|
||||
# 2. With the browser session cookie the upgrade is accepted and bridged.
|
||||
# DSH may not serve a WebSocket on '/', so an upstream close (1011) is
|
||||
# reported but tolerated; a 4401 would mean the gateway lost the cookie.
|
||||
try:
|
||||
with connect_ws({"Cookie": cookie, "Origin": f"http://{gw}"}) as ws:
|
||||
try:
|
||||
ws.recv(timeout=10)
|
||||
print("authenticated WebSocket accepted and streaming")
|
||||
except TimeoutError:
|
||||
print("authenticated WebSocket accepted (no immediate close)")
|
||||
except ConnectionClosed as exc:
|
||||
code = exc.rcvd.code if exc.rcvd else None
|
||||
if code == 4401:
|
||||
sys.exit(
|
||||
"FAIL: authenticated WebSocket was treated as unauthenticated (4401)"
|
||||
)
|
||||
print(f"authenticated WebSocket accepted; upstream closed with {code}")
|
||||
except InvalidStatus as exc:
|
||||
sys.exit(f"FAIL: authenticated WebSocket handshake returned HTTP {status_of(exc)}")
|
||||
except Exception as exc: # noqa: BLE001 - report any handshake failure
|
||||
sys.exit(f"FAIL: authenticated WebSocket check raised {exc!r}")
|
||||
PY
|
||||
pass "gateway WebSocket: anonymous rejected (4401), authenticated upgrade bridged"
|
||||
else
|
||||
echo "WARNING: python websockets not installed; skipping the WebSocket check" >&2
|
||||
fi
|
||||
|
||||
echo "waiting for idle sweeper (~75s)..."
|
||||
for _ in $(seq 1 30); do
|
||||
STATE=$(curl -s -H "Authorization: Bearer $TOKEN" "http://$GW/users/bob" \
|
||||
STATE=$(curl -s -b "$JAR" "http://$GW/users/bob" \
|
||||
| sed -n 's/.*"state":"\([^"]*\)".*/\1/p')
|
||||
[ "$STATE" = "Suspended" ] && break
|
||||
sleep 5
|
||||
@@ -113,16 +229,19 @@ if command -v docker >/dev/null && command -v kind >/dev/null; then
|
||||
pass "idle sweeper suspended bob"
|
||||
|
||||
T0=$(date +%s)
|
||||
curl -sf -m 150 -H "Authorization: Bearer $TOKEN" "http://$GW/u/bob/" >/dev/null \
|
||||
curl -sf -m 150 -b "$JAR" "http://$GW/u/bob/" >/dev/null \
|
||||
|| fail "wake-on-connect request failed"
|
||||
pass "wake-on-connect: request held and served after $(( $(date +%s) - T0 ))s"
|
||||
|
||||
curl -sf -X DELETE -H "Authorization: Bearer $TOKEN" "http://$GW/users/bob" >/dev/null \
|
||||
curl -sf -X DELETE -b "$JAR" -H "Origin: http://$GW" \
|
||||
"http://$GW/users/bob" >/dev/null \
|
||||
|| fail "gateway user deletion failed"
|
||||
kill "$PF" 2>/dev/null || true
|
||||
rm -f "$JAR"
|
||||
pass "gateway delete: bob removed"
|
||||
elif [ "${SKIP_GATEWAY:-}" = "1" ]; then
|
||||
echo "WARNING: gateway checks SKIPPED (SKIP_GATEWAY=1) — signup, auth," >&2
|
||||
echo "WARNING: proxy, idle-suspend and wake-on-connect were NOT validated" >&2
|
||||
echo "WARNING: proxy, WebSocket, idle-suspend and wake-on-connect were NOT validated" >&2
|
||||
GATEWAY_SKIPPED=1
|
||||
else
|
||||
fail "docker + kind are required for the gateway checks (set SKIP_GATEWAY=1 to run only the resource-layer checks)"
|
||||
|
||||
Reference in New Issue
Block a user