nicolas-georger/deepseek-kubernetes-sandbox
1
0
Fork 0
mirror of https://github.com/sredevopsorg/deepseek-kubernetes-sandbox.git synced 2026-11-03 01:13:24 +00:00
Code Projects Releases Activity

fix(gateway)!: replace token cookies with revocable sessions and harden abuse controls

Browse Source 
- Replace user:token cookies with opaque server-side sessions, revocable on logout, delete, recreate and expiry; bind every credential to the path user and add cross-user regression tests.
- Rate-limit /users and /login, add optional signup invite codes and form same-origin checks, and unify login failures to 401.
- Require explicit ALLOW_INSECURE_COOKIES for plain HTTP, use __Host-dsh_gw_session when secure, and fix secure cookie deletion.
- Strip client-supplied forwarding/SSO headers, add an optional upstream allowlist, per-user in-flight and WebSocket message caps, credential-free handshake errors, and sweep-time state pruning.
- Digest-pin base images, hash-pin gateway dependencies, verify DSH_COMMIT in the build, and run the gateway image as non-root with a pinned Python base.
- Update manifests, production checklist, architecture/security docs and add an SSO preflight script.

BREAKING CHANGE: existing dsh_gw_session=<user>:<token> cookies are invalid; users must sign in again. Unknown users on POST /login now return 401 instead of 404, and the gateway refuses COOKIE_SECURE=false unless ALLOW_INSECURE_COOKIES=1.
This commit is contained in:
SREDevOpsAgents
2026-10-31 00:54:05 -03:00
parent 24f23e09ae
commit ae8aba8241
33 changed files with 5450 additions and 319 deletions
Download Patch File Download Diff File Expand all files Collapse all files

115
examples/dsh-agents-as-a-service/gateway/README.md
View File

@@ -10,11 +10,13 @@ It does four things:
(only its SHA-256 is stored as a claim annotation), reports derived state,
and deletes the claim (cascading the sandbox and PVC).
2. **Browser sessions** — exchanges a user token for an HttpOnly session
cookie so the DSH UI works in a browser at `/u/<user>/`. Automation can
still use `Authorization: Bearer <token>`.
cookie whose value is only an opaque, server-side session id (user and
expiry stay in the gateway's memory) so the DSH UI works in a browser at
`/u/<user>/`. Automation can still use
`Authorization: Bearer <token>`, which never comes from the cookie.
3. **Reverse proxy** — proxies HTTP (including SSE) and WebSocket traffic to
the sandbox's DSH Web server on `:3080`, including DSH's own cookie
handshake.
handshake. Client-supplied forwarding/identity headers are stripped.
4. **Optional SSO** — when `AUTH_PROXY_HEADER` is set, trusts the identity
header an authenticating reverse proxy (for example oauth2-proxy) forwards,
auto-provisions that user, and exposes `POST /recreate` for a clean restart.
@@ -63,10 +65,10 @@ token. A failed handshake returns `503` with `Retry-After: 10`.
| Method | Path | Auth | Purpose |
| --- | --- | --- | --- |
| `GET` | `/healthz` | none | Liveness/readiness. Never touches the cluster. |
| `GET` | `/` | none | Landing page with signup + token login forms. With SSO it auto-provisions the proxy-identified user and redirects to their workspace. |
| `POST` | `/users` | none | Signup. JSON body `{"user": "<dns-1123 label>"}` or form field `user`. JSON returns `201` when Ready, `202` when accepted but still Waking; form returns `303` to `/u/<user>/`. Both set the session cookie. `409` if the user exists, `400` if invalid, `413` if the body is too large. |
| `POST` | `/login` | none | Form (`user`, `token`) returns `303` to `/u/<user>/`; JSON returns `200`. Sets the session cookie. `401` on a bad token, `404` for an unknown user. |
| `POST` | `/logout` | none | Clears the session cookie and `303`s to `/`. |
| `GET` | `/` | none | Landing page with signup + token login forms (plus an invite-code input when `SIGNUP_INVITE_CODE` is set). With SSO it auto-provisions the proxy-identified user and redirects to their workspace. |
| `POST` | `/users` | none | Signup. JSON body `{"user": "<dns-1123 label>"}` or form field `user`; when `SIGNUP_INVITE_CODE` is set, an `invite` form field or `X-Signup-Invite` header must match it (`403` otherwise, checked before the user lookup). JSON returns `201` when Ready, `202` when accepted but still Waking; form returns `303` to `/u/<user>/`. Both set the session cookie. `409` if the user exists, `400` if invalid, `413` if the body is too large, `429` when the auth rate limit is hit. |
| `POST` | `/login` | none | Form (`user`, `token`) returns `303` to `/u/<user>/`; JSON returns `200`. Sets the session cookie. `401` on a bad token **and** for an unknown user (no username enumeration). `429` when the auth rate limit is hit. Both auth routes reject a form post whose `Origin`/`Referer` contradicts `Host` (`403`) when the browser sent one. |
| `POST` | `/logout` | none | Revokes the server-side session, clears the cookie and `303`s to `/`. |
| `POST` | `/recreate` | trusted proxy header | **SSO only.** Deletes the authenticated user's claim (cascading sandbox + PVC, so workspace data is wiped), waits out finalizers, provisions a fresh claim and `303`s to `/u/<user>/`. `401` without a valid identity header; cookie-authenticated calls must be same-origin. |
| `GET` | `/users/{user}` | bearer or cookie | Derived state: `Provisioning`, `Waking`, `Ready`, `Suspending`, `Suspended`. |
| `DELETE` | `/users/{user}` | bearer or cookie | Delete the claim (sandbox + PVC cascade). Cookie-auth mutations require a same-origin `Origin`/`Referer`. |
@@ -78,19 +80,49 @@ token. A failed handshake returns `503` with `Retry-After: 10`.
* `Authorization: Bearer <token>` is checked with a timing-safe SHA-256
comparison against the `aaas.example.com/token-sha256` claim annotation.
* The browser session cookie is
`dsh_gw_session=<user>:<token>` (HttpOnly, SameSite=Lax, Path=/, optional
Secure). A session for user A never authorizes user B.
* The browser session cookie is **opaque**: its value is a random
server-side session id; the user and expiry live in gateway memory
(`GatewayState.sessions`). A cookie can never be edited into another user's
credentials, a session id is not accepted as a bearer token, and a bearer
token never authorizes a different user. The cookie name defaults to
`__Host-dsh_gw_session` when `COOKIE_SECURE=true` (HTTPS) and to
`dsh_gw_session` for plain HTTP; attributes are HttpOnly, SameSite=Lax,
Path=/ and (with `COOKIE_SECURE`) Secure.
* Sessions are revoked on `POST /logout`, on `DELETE /users/{user}`, on
`POST /recreate`, and when they expire (looked up on every request; the
sweeper also drops expired entries). `from_env()` refuses to start without
`COOKIE_SECURE=true` or the explicit `ALLOW_INSECURE_COOKIES=1` opt-out for
local HTTP.
* Cookie-authenticated **mutations** (`POST`, `PUT`, `PATCH`, `DELETE`) and
cookie-authenticated **WebSocket upgrades** require the request `Origin`
(or `Referer` fallback) to match the `Host` header. Missing or mismatched
cross-origin metadata is rejected (`403` HTTP, close code `4401` for WS).
Form signup/login additionally reject a cross-origin `Origin`/`Referer`
when the browser sent one, before the body is read.
* `POST /users` and `POST /login` are rate limited per client:
`AUTH_RATE_LIMIT` attempts per `AUTH_RATE_WINDOW` seconds (default 30/60,
`0` disables). The client key is the rightmost `X-Forwarded-For` entry when
present (the address the last trusted proxy saw, not a spoofable leftmost
value), else the socket peer. Over-limit requests get `429` +
`Retry-After` before the body is read.
* Proxy requests strip client-supplied forwarding/identity headers in both
HTTP and WebSocket paths: `Forwarded`, every `X-Forwarded-*`
(including `X-Forwarded-User`), every `X-Auth-Request-*`, and `X-Real-IP`.
The gateway sets its own `X-Forwarded-For/Host/Proto/Port` values from the
connection. `UPSTREAM_HEADER_ALLOWLIST` optionally narrows the forwarded
client headers further (comma-separated, case-insensitive); unset keeps the
denylist behavior, and the gateway's own values are always added.
* The gateway's bearer token, the browser `Cookie` header and any `?token=`
query parameter are never forwarded upstream. Only the cached `dsh-auth-*`
cookie is injected.
* Upstream `Set-Cookie` headers are dropped. `Location` headers that are
absolute paths are rewritten under `/u/<user>/`, and any `token` query
parameter is stripped.
* Concurrency is capped per user: `MAX_IN_FLIGHT_PER_USER` proxied requests
(default 32) before the gateway answers `503` + `Retry-After` (HTTP) or
closes the WebSocket with `1011`. Messages received from DSH are limited to
`WS_MAX_MESSAGE_BYTES` (default 16 MiB); browser-to-DSH frames are bounded by
uvicorn's own WebSocket size limit.
### SSO (opt-in)
@@ -106,7 +138,8 @@ kubectl -n dsh-demo set env deploy/dsh-gateway AUTH_PROXY_HEADER=X-Forwarded-Use
When set, the gateway treats a request carrying that header as authenticated as
that user, provided the value is a valid DNS-1123 label; the header is never
trusted when the variable is unset. The landing page then auto-provisions the
user (creates their claim idempotently, mints a rotated session token) and
user (creates their claim idempotently, rotates the claim token when a new
session is needed and mints an opaque session cookie) and
redirects to `/u/<user>/`, and `POST /recreate` becomes available for a clean
restart. The base manifests leave it unset on purpose; the
[SSO exposure variant](../manifests/exposure/gateway-api-sso/) documents how to
@@ -133,14 +166,19 @@ oauth2-proxy's egress to the gateway, DNS and public HTTPS.
| --- | --- |
| Unknown user / claim | `404` |
| Missing or invalid credentials | `401` |
| Cross-origin cookie mutation | `403` |
| Unknown user on `/login` | `401` (same as a bad token; no username enumeration) |
| Cross-origin cookie mutation / cross-origin form signup or login | `403` |
| Auth route over `AUTH_RATE_LIMIT` | `429` + `Retry-After`, before the body is read |
| Request body over `MAX_BODY_MB` | `413` |
| Client aborts mid-body, before the response starts | `499` (`CLIENT_CLOSED_REQUEST`). Nothing is forwarded to DSH and nothing can be delivered; it is not an error, so no traceback is logged. Covers the body-reading control routes (`POST /users`, `POST /login`) as well as the proxy. |
| Client aborts mid-stream (SSE, long response) | The response already started, so its upstream status stands (typically `200`); the stream simply ends. No traceback, and the in-flight slot is released as the relay unwinds. |
| Sandbox not provisioned, wake timeout, handshake failure | `503` + `Retry-After: 10` |
| Sandbox not provisioned, wake timeout, handshake failure | `503` + `Retry-After: 10`. The handshake error body is a fixed message: it never contains the launch token, the handshake URL or pod/credential details. |
| User over `MAX_IN_FLIGHT_PER_USER` | `503` + `Retry-After` (HTTP); the existing in-flight requests are untouched |
| Upstream connection/protocol failure | `502` |
| WS auth failure (unknown user, bad token, cross-origin) | accept, then close `4401` |
| WS per-user concurrency cap | accept, then close `1011` |
| WS upstream/handshake failure | accept, then close `1011` |
| WS message from DSH over `WS_MAX_MESSAGE_BYTES` | the upstream/relay closes the connection (websockets' message-too-big path) |
A close code can only be delivered after the WebSocket upgrade is accepted;
Starlette would turn an early close into an HTTP 403, which is why WS auth
@@ -166,17 +204,27 @@ can also be injected into `create_app` for tests.
| `DSH_PORT` | `3080` | Upstream DSH port. |
| `DSH_CONTAINER` | `dsh-web` | Container whose logs carry the launch token. |
| `DSH_LOG_TAIL_LINES` | `2000` | How many log lines to scan for the launch token. |
| `COOKIE_NAME` | `dsh_gw_session` | Browser session cookie name. |
| `COOKIE_SECURE` | `false` | Add `Secure` to the session cookie when served over TLS. |
| `COOKIE_MAX_AGE` | `604800` | Session cookie lifetime in seconds. |
| `COOKIE_NAME` | `__Host-dsh_gw_session` when secure, else `dsh_gw_session` | Browser session cookie name. Only an opaque session id is stored in it. |
| `COOKIE_SECURE` | `false` | Add `Secure` to the session cookie. Required unless `ALLOW_INSECURE_COOKIES=1` is set for local HTTP. |
| `ALLOW_INSECURE_COOKIES` | `0` | Explicit opt-out for plain-HTTP demos. Without it, `Settings.from_env()` refuses to start unless `COOKIE_SECURE=true`. |
| `COOKIE_MAX_AGE` | `604800` | Session lifetime in seconds (server-side expiry and cookie `Max-Age`). |
| `MAX_BODY_MB` | `32` | Maximum buffered request body (and `413` threshold). |
| `HTTP_TIMEOUT` | `300` | Upstream connect/read timeout in seconds. Proxied streams (SSE, long responses) keep the connect/write budget but have **no read timeout**, so an idle event stream is not cut every five minutes. |
| `SWEEP_INTERVAL` | `15` | Seconds between idle sweeps. |
| `SWEEP_INTERVAL` | `15` | Seconds between idle sweeps. Sweeps also prune expired sessions and rate buckets and orphaned per-user state. |
| `SIGNUP_INVITE_CODE` | unset | When set, `POST /users` requires the matching `invite` form field or `X-Signup-Invite` header (timing-safe compare); the landing page then shows an invite input. Unset leaves signup open. |
| `AUTH_RATE_LIMIT` | `30` | Attempts allowed per client per `AUTH_RATE_WINDOW` on `POST /users` and `POST /login`; `0` disables the limit. |
| `AUTH_RATE_WINDOW` | `60` | Rate-limit window in seconds. |
| `UPSTREAM_HEADER_ALLOWLIST` | unset | Comma-separated, case-insensitive allowlist of client headers to forward upstream. Unset keeps the denylist behavior; forwarding/identity headers are stripped in both modes. |
| `MAX_IN_FLIGHT_PER_USER` | `32` | Concurrent proxied requests per user before `503`/WS `1011`. `0` disables the cap. |
| `WS_MAX_MESSAGE_BYTES` | `16777216` | Largest WebSocket message (16 MiB) accepted from DSH. Browser-to-DSH frames are bounded by uvicorn's own limit. |
| `AUTH_PROXY_HEADER` | unset | Opt-in SSO: name of the trusted identity header set by an authenticating reverse proxy (e.g. `X-Forwarded-User`). Unset disables SSO and the header is never trusted. Enable only when the Service is reachable exclusively through that proxy (see [SSO](#sso-opt-in)). |
The shipped `manifests/50-gateway.yaml` sets `IDLE_TIMEOUT=360` and
`WAKE_TIMEOUT=240` for the demo; the e2e script overrides them to 60/120 so the
smoke test does not wait minutes for the sweeper.
smoke test does not wait minutes for the sweeper. It also sets
`COOKIE_SECURE=false` together with the explicit `ALLOW_INSECURE_COOKIES=1`
acknowledgement because the demo is reached over plain HTTP (a `Secure` cookie
would never be sent). HTTPS deployments must set `COOKIE_SECURE=true`.
### Required RBAC
@@ -197,8 +245,10 @@ python -m pip install -r requirements-dev.txt
python -m pytest -q
python -m ruff check .
# Uvicorn factory (loads in-cluster credentials only on the first claim call)
uvicorn gateway:create_app --factory --host 0.0.0.0 --port 8080
# Uvicorn factory (loads in-cluster credentials only on the first claim call).
# from_env() insists on an explicit acknowledgement for plain HTTP; behind TLS
# set COOKIE_SECURE=true instead.
ALLOW_INSECURE_COOKIES=1 uvicorn gateway:create_app --factory --host 0.0.0.0 --port 8080
```
Container image, from the repository root:
@@ -226,18 +276,27 @@ uvicorn's access log so bearer tokens in URLs are not logged.
Log rotation, a very chatty container, or DSH changing its startup message
breaks the handshake. Prefer a first-class token/credential API when DSH
offers one.
* **Single replica, in-memory state.** The handshake cookie cache, idle
activity clocks and in-flight counters live in one process. Running more
than one replica would need shared state (or sticky routing), otherwise each
replica handshakes independently and the sweeper may race another replica.
* **Single replica, in-memory state.** The handshake cookie cache, the opaque
session table, idle activity clocks, in-flight counters and rate-limit
buckets live in one process. Running more than one replica would need shared
state (or sticky routing), otherwise each replica handshakes independently,
the sweeper may race another replica, and a session/rate budget is only seen
by the replica that created it.
* **Sessions die with the process.** An opaque session id is valid only in the
process that minted it and expires after `COOKIE_MAX_AGE`; a restart logs
every browser out (tokens are unaffected).
* **In-memory token cache has no TTL.** It is keyed by pod UID, so pod
restarts invalidate it naturally; otherwise entries survive until the pod or
claim is deleted.
* **No TLS termination.** Run behind an ingress/load balancer; set
`COOKIE_SECURE=true` there.
* **No TLS termination.** Run behind an ingress/load balancer and set
`COOKIE_SECURE=true`. `Settings.from_env()` refuses to start without it
unless `ALLOW_INSECURE_COOKIES=1` explicitly acknowledges plain-HTTP
sessions.
* **Origin-based CSRF protection only.** Cookie-auth mutations require a
same-origin `Origin`/`Referer`; there is no per-form CSRF token, rate
limiting or login throttling.
same-origin `Origin`/`Referer`, and form signup/login reject a contradicting
origin when the browser sends one; there is no per-form CSRF token. Auth
routes have per-client rate limiting and login throttling, but the budget is
in-process (per replica) and keyed by the rightmost `X-Forwarded-For` value.
* **HTTP request bodies are buffered** up to `MAX_BODY_MB`; uploads are not
streamed. Responses (including SSE) are streamed.
* **Upstream cookies are not forwarded.** DSH cannot rotate its own