1
0
mirror of https://github.com/sredevopsorg/deepseek-kubernetes-sandbox.git synced 2026-11-03 09:23:25 +00:00

4 Commits

Author SHA1 Message Date
Nicolás Georger
f4c0b52ee9 Merge branch 'chore/changelog-dsh-v0.2.1-rc.5-r6' into fix/security-gateway-sessions
Signed-off-by: Nicolás Georger <info@sredevops.org>
2026-10-31 07:38:56 -03:00
github-actions[bot]
96837a43b6 chore(changelog): refresh for dsh-v0.2.1-rc.5-r6 2026-10-31 09:31:08 +00:00
SREDevOpsAgents
2eb5c82ace Merge remote-tracking branch 'origin/main' into fix/security-gateway-sessions
# Conflicts:
#	docs/versioning.md
#	versions.env
2026-10-31 00:55:39 -03:00
SREDevOpsAgents
ae8aba8241 fix(gateway)!: replace token cookies with revocable sessions and harden abuse controls
- Replace user:token cookies with opaque server-side sessions, revocable on logout, delete, recreate and expiry; bind every credential to the path user and add cross-user regression tests.
- Rate-limit /users and /login, add optional signup invite codes and form same-origin checks, and unify login failures to 401.
- Require explicit ALLOW_INSECURE_COOKIES for plain HTTP, use __Host-dsh_gw_session when secure, and fix secure cookie deletion.
- Strip client-supplied forwarding/SSO headers, add an optional upstream allowlist, per-user in-flight and WebSocket message caps, credential-free handshake errors, and sweep-time state pruning.
- Digest-pin base images, hash-pin gateway dependencies, verify DSH_COMMIT in the build, and run the gateway image as non-root with a pinned Python base.
- Update manifests, production checklist, architecture/security docs and add an SSO preflight script.

BREAKING CHANGE: existing dsh_gw_session=<user>:<token> cookies are invalid; users must sign in again. Unknown users on POST /login now return 401 instead of 404, and the gateway refuses COOKIE_SECURE=false unless ALLOW_INSECURE_COOKIES=1.
2026-10-31 00:54:05 -03:00
15 changed files with 39 additions and 214 deletions

5
.entire/.gitignore vendored
View File

@@ -1,5 +0,0 @@
tmp/
settings.local.json
metadata/
logs/
redactors/local/

View File

@@ -1,16 +0,0 @@
{
"version": 1,
"agent": "dsh",
"bridge": "@sredevopsorg/dsh-entire-bridge",
"hook_names": [
"session-start",
"turn-start",
"turn-end",
"compaction",
"session-end",
"subagent-start",
"subagent-end"
],
"installed_at": "2026-11-02T06:02:27.816Z",
"local_dev": false
}

View File

@@ -1,9 +0,0 @@
{
"enabled": true,
"telemetry": true,
"checkpoints": {
"primary": {
"type": "git-refs"
}
}
}

View File

@@ -35,8 +35,7 @@ disclosure once a fix is available.
exposure variant's policy is required, not optional: an exposure without it exposure variant's policy is required, not optional: an exposure without it
is demo-only. Run `examples/dsh-agents-as-a-service/scripts/preflight-sso.sh` is demo-only. Run `examples/dsh-agents-as-a-service/scripts/preflight-sso.sh`
after applying the variant; it checks the Secrets, gateway env, TLS after applying the variant; it checks the Secrets, gateway env, TLS
reference, leftover placeholders, accidental second paths, the gateway reference, leftover placeholders and accidental second paths.
Role's `patch` verb on `sandboxclaims` and the variant NetworkPolicies.
- **Insecure-cookie guard.** TLS terminates outside the gateway. Set - **Insecure-cookie guard.** TLS terminates outside the gateway. Set
COOKIE_SECURE=true for HTTPS and remove ALLOW_INSECURE_COOKIES; the gateway COOKIE_SECURE=true for HTTPS and remove ALLOW_INSECURE_COOKIES; the gateway
refuses to start with COOKIE_SECURE=false unless ALLOW_INSECURE_COOKIES=1 is refuses to start with COOKIE_SECURE=false unless ALLOW_INSECURE_COOKIES=1 is

View File

@@ -518,10 +518,8 @@ front of **oauth2-proxy**, which authenticates the browser against GitHub
reports **before** pointing real traffic at the Gateway. It verifies the reports **before** pointing real traffic at the Gateway. It verifies the
Secrets (including a valid oauth2-proxy cookie secret length), the gateway's Secrets (including a valid oauth2-proxy cookie secret length), the gateway's
`AUTH_PROXY_HEADER`/`COOKIE_SECURE`, that the Gateway's TLS certificate `AUTH_PROXY_HEADER`/`COOKIE_SECURE`, that the Gateway's TLS certificate
reference resolves, that the gateway ServiceAccount may `patch` reference resolves, and that no `your-github-org`/`dsh-demo.example.com`
`sandboxclaims` (SSO token rotation), that the variant's NetworkPolicies are placeholder was left applied. Full verification steps are in the
applied, and that no `your-github-org`/`dsh-demo.example.com` placeholder was
left applied. Full verification steps are in the
[exposure README](manifests/exposure/README.md). [exposure README](manifests/exposure/README.md).
> **Exposure without the variant NetworkPolicy is demo-only.** The SSO > **Exposure without the variant NetworkPolicy is demo-only.** The SSO
@@ -601,7 +599,7 @@ applied by the base step.
| Every browser is logged out after a gateway rollout/restart | Expected: sessions are opaque ids in gateway memory and a restart drops them. Sign in again; a production deployment needs shared session state or per-request re-authentication. The DSH sandboxes and their data are unaffected | | Every browser is logged out after a gateway rollout/restart | Expected: sessions are opaque ids in gateway memory and a restart drops them. Sign in again; a production deployment needs shared session state or per-request re-authentication. The DSH sandboxes and their data are unaffected |
| WebSocket closes with `1011` for a busy client | `MAX_IN_FLIGHT_PER_USER` (per-user connection cap) or an upstream/handshake failure closed the socket; check the gateway logs to tell them apart | | WebSocket closes with `1011` for a busy client | `MAX_IN_FLIGHT_PER_USER` (per-user connection cap) or an upstream/handshake failure closed the socket; check the gateway logs to tell them apart |
| WebSocket closes after a very large message from DSH | A message received from DSH exceeded `WS_MAX_MESSAGE_BYTES` (16 MiB), so the relay closed it instead of buffering; raise the limit only if DSH legitimately sends larger frames. Browser-to-DSH frames are bounded by uvicorn's own limit | | WebSocket closes after a very large message from DSH | A message received from DSH exceeded `WS_MAX_MESSAGE_BYTES` (16 MiB), so the relay closed it instead of buffering; raise the limit only if DSH legitimately sends larger frames. Browser-to-DSH frames are bounded by uvicorn's own limit |
| `scripts/preflight-sso.sh` exits non-zero | It names the failing item: a missing Secret/key, a bad oauth2-proxy cookie-secret length, missing `AUTH_PROXY_HEADER`/`COOKIE_SECURE`, a dangling Gateway TLS reference, a leftover `your-github-org`/`dsh-demo.example.com` placeholder, a second path to the gateway (an Ingress), a gateway Role without `patch` on `sandboxclaims`, or a missing SSO NetworkPolicy. Fix all items before exposing the SSO variant | | `scripts/preflight-sso.sh` exits non-zero | It names the failing item: a missing Secret/key, a bad oauth2-proxy cookie-secret length, missing `AUTH_PROXY_HEADER`/`COOKIE_SECURE`, a dangling Gateway TLS reference, a leftover `your-github-org`/`dsh-demo.example.com` placeholder, or a second path to the gateway (an Ingress). Fix all items before exposing the SSO variant |
| `409` on signup | User already has a claim; use `POST /login` with the existing token or delete the user first | | `409` on signup | User already has a claim; use `POST /login` with the existing token or delete the user first |
| Cookie set but browser keeps getting `401` | `COOKIE_SECURE=true` while serving over plain HTTP (secure cookies are not sent); use HTTPS or set `false` **and** `ALLOW_INSECURE_COOKIES=1` for local demos | | Cookie set but browser keeps getting `401` | `COOKIE_SECURE=true` while serving over plain HTTP (secure cookies are not sent); use HTTPS or set `false` **and** `ALLOW_INSECURE_COOKIES=1` for local demos |
| WebSocket closes immediately (`4401`) | Missing/expired gateway cookie, or the upgrade's `Origin` does not match `Host` (CSWSH check); DSH rejecting `Host` instead surfaces as close `1011` — add the host to `DSH_TRUSTED_HOSTS` | | WebSocket closes immediately (`4401`) | Missing/expired gateway cookie, or the upgrade's `Origin` does not match `Host` (CSWSH check); DSH rejecting `Host` instead surfaces as close `1011` — add the host to `DSH_TRUSTED_HOSTS` |

View File

@@ -339,7 +339,7 @@ The gateway runs with a namespace-scoped ServiceAccount (`manifests/50-gateway.y
| API group | Resource | Verbs | Why | | API group | Resource | Verbs | Why |
| --- | --- | --- | --- | | --- | --- | --- | --- |
| `extensions.agents.x-k8s.io` | `sandboxclaims` | `get`, `list`, `create`, `patch`, `delete` | signup, state, account deletion, token rotation | | `extensions.agents.x-k8s.io` | `sandboxclaims` | `get`, `list`, `create`, `delete` | signup, state, account deletion |
| `agents.x-k8s.io` | `sandboxes` | `get`, `list`, `patch` | resolve DNS/pod, wake/suspend | | `agents.x-k8s.io` | `sandboxes` | `get`, `list`, `patch` | resolve DNS/pod, wake/suspend |
| `""` (core) | `pods` | `get`, `list` | resolve the pod behind a sandbox | | `""` (core) | `pods` | `get`, `list` | resolve the pod behind a sandbox |
| `""` (core) | `pods/log` | `get` | launch-token handshake | | `""` (core) | `pods/log` | `get` | launch-token handshake |
@@ -395,8 +395,7 @@ Notes and limits:
- **SSO is trust-on-hop.** `AUTH_PROXY_HEADER` makes the gateway trust the proxy-asserted - **SSO is trust-on-hop.** `AUTH_PROXY_HEADER` makes the gateway trust the proxy-asserted
identity; the SSO variant must remain the only path to the Service, and its NetworkPolicy is identity; the SSO variant must remain the only path to the Service, and its NetworkPolicy is
required, not optional. Run [`scripts/preflight-sso.sh`](../scripts/preflight-sso.sh) after required, not optional. Run [`scripts/preflight-sso.sh`](../scripts/preflight-sso.sh) after
applying it: it fails when the Role does not grant `patch` on `sandboxclaims` (SSO token applying it.
rotation) or when the variant's NetworkPolicies are missing.
## Failure modes ## Failure modes
@@ -405,7 +404,6 @@ Notes and limits:
| DSH log line changed or rotated away | Proxy returns `503` (missing launch token) | Fix the parser or restart the pod so the token line is re-emitted | | DSH log line changed or rotated away | Proxy returns `503` (missing launch token) | Fix the parser or restart the pod so the token line is re-emitted |
| Probe on `/` | Pod `CrashLoopBackOff`, `401` in probe logs | Use `/favicon.svg` for readiness/liveness (shipped templates do) | | Probe on `/` | Pod `CrashLoopBackOff`, `401` in probe logs | Use `/favicon.svg` for readiness/liveness (shipped templates do) |
| Warm pool exhausted | JSON signup returns `202` (state `Waking`); form signup is slow | Increase `replicas` in the warm pool | | Warm pool exhausted | JSON signup returns `202` (state `Waking`); form signup is slow | Increase `replicas` in the warm pool |
| Kubernetes API error (API down or throttled, missing RBAC verb) | HTTP `503` + `Retry-After`; WebSocket close `1011` | One credential-free log line with the API `status`/`reason`, plus a fixed body; the Kubernetes `Status` payload is never proxied. Check the ServiceAccount Role and the API server — RBAC is evaluated per request, so a Role fix needs no restart |
| Gateway restart | Idle sandboxes stop suspending; handshake cache and browser sessions lost | Idle clocks and handshake cache rebuild lazily; sessions do not, so browsers must sign in again after a rollout. Use a shared session store for production | | Gateway restart | Idle sandboxes stop suspending; handshake cache and browser sessions lost | Idle clocks and handshake cache rebuild lazily; sessions do not, so browsers must sign in again after a rollout. Use a shared session store for production |
| Gateway configured `COOKIE_SECURE=false` without `ALLOW_INSECURE_COOKIES=1` | Gateway pod exits at startup / `CrashLoopBackOff` | Intentional guard against an insecure cookie on an HTTPS deployment; set `ALLOW_INSECURE_COOKIES=1` only for the local plain-HTTP demo, or set `COOKIE_SECURE=true` | | Gateway configured `COOKIE_SECURE=false` without `ALLOW_INSECURE_COOKIES=1` | Gateway pod exits at startup / `CrashLoopBackOff` | Intentional guard against an insecure cookie on an HTTPS deployment; set `ALLOW_INSECURE_COOKIES=1` only for the local plain-HTTP demo, or set `COOKIE_SECURE=true` |
| Auth rate limit hit | `429` + `Retry-After` on signup/login | Per-client in-memory bucket (`AUTH_RATE_LIMIT`/`AUTH_RATE_WINDOW`); wait for the window or tune it. Behind a proxy, keying uses the rightmost `X-Forwarded-For` entry | | Auth rate limit hit | `429` + `Retry-After` on signup/login | Per-client in-memory bucket (`AUTH_RATE_LIMIT`/`AUTH_RATE_WINDOW`); wait for the window or tune it. Behind a proxy, keying uses the rightmost `X-Forwarded-For` entry |
@@ -413,7 +411,7 @@ Notes and limits:
| Per-user concurrency cap hit | HTTP `503` + `Retry-After`, or WebSocket close `1011` | `MAX_IN_FLIGHT_PER_USER` protects the single gateway from one busy tenant; reduce concurrency or raise the cap deliberately. These are per replica | | Per-user concurrency cap hit | HTTP `503` + `Retry-After`, or WebSocket close `1011` | `MAX_IN_FLIGHT_PER_USER` protects the single gateway from one busy tenant; reduce concurrency or raise the cap deliberately. These are per replica |
| Oversized WebSocket message from DSH | Connection closes | `WS_MAX_MESSAGE_BYTES` (16 MiB) is enforced on upstream messages; raise it only if DSH legitimately needs larger frames | | Oversized WebSocket message from DSH | Connection closes | `WS_MAX_MESSAGE_BYTES` (16 MiB) is enforced on upstream messages; raise it only if DSH legitimately needs larger frames |
| Session expiry / revocation | `401` (HTTP) or close `4401` (WebSocket) | Expected after logout, user delete, `/recreate`, `COOKIE_MAX_AGE` expiry or a gateway restart; sign in again | | Session expiry / revocation | `401` (HTTP) or close `4401` (WebSocket) | Expected after logout, user delete, `/recreate`, `COOKIE_MAX_AGE` expiry or a gateway restart; sign in again |
| SSO variant applied without its NetworkPolicy (or with an Ingress) | Gateway accepts spoofed `X-Forwarded-User` | Demo-only configuration; keep `75-networkpolicies.yaml`, remove the second path, and run `scripts/preflight-sso.sh`, which fails when either policy is missing | | SSO variant applied without its NetworkPolicy (or with an Ingress) | Gateway accepts spoofed `X-Forwarded-User` | Demo-only configuration; keep `75-networkpolicies.yaml`, remove the second path, and run `scripts/preflight-sso.sh` |
| Browser aborts a request (tab reload, back navigation) during a slow wake or handshake | No response reaches the client; nothing appears in the access log | If the abort lands before the response starts, the body read raises `ClientDisconnect` and the app answers `499` (`CLIENT_CLOSED_REQUEST`), which cannot reach the departed peer. Nothing reached DSH, no ERROR traceback is logged, and the in-flight slot is released. A request already streaming ends with its upstream status. | | Browser aborts a request (tab reload, back navigation) during a slow wake or handshake | No response reaches the client; nothing appears in the access log | If the abort lands before the response starts, the body read raises `ClientDisconnect` and the app answers `499` (`CLIENT_CLOSED_REQUEST`), which cannot reach the departed peer. Nothing reached DSH, no ERROR traceback is logged, and the in-flight slot is released. A request already streaming ends with its upstream status. |
| DSH auth cookie expired mid-session | Proxy `401` once, then recovers | Gateway refreshes the handshake once | | DSH auth cookie expired mid-session | Proxy `401` once, then recovers | Gateway refreshes the handshake once |
| Upstream pod name changed after resume | Cached cookie keyed by old pod UID is useless | Cache is invalidated on suspend; next request re-handshakes | | Upstream pod name changed after resume | Cached cookie keyed by old pod UID is useless | Cache is invalidated on suspend; next request re-handshakes |

View File

@@ -338,10 +338,8 @@ The SSO policy is what turns the trust boundary in the
The `dsh-gateway` Service must still stay ClusterIP; do not apply both exposure The `dsh-gateway` Service must still stay ClusterIP; do not apply both exposure
variants. After applying the SSO variant, run variants. After applying the SSO variant, run
[`scripts/preflight-sso.sh`](../scripts/preflight-sso.sh): it checks the [`scripts/preflight-sso.sh`](../scripts/preflight-sso.sh): it checks the
gateway's SSO env, the Secrets, the TLS reference, leftover placeholders, gateway's SSO env, the Secrets, the TLS reference, leftover placeholders and
accidental second paths (such as an Ingress), the gateway Role's `patch` verb on accidental second paths (such as an Ingress).
`sandboxclaims` (SSO token rotation) and that both SSO NetworkPolicies are
applied.
Verify the negative case from a sandbox pod (it is not the proxy): Verify the negative case from a sandbox pod (it is not the proxy):

View File

@@ -175,11 +175,9 @@ oauth2-proxy's egress to the gateway, DNS and public HTTPS.
| Sandbox not provisioned, wake timeout, handshake failure | `503` + `Retry-After: 10`. The handshake error body is a fixed message: it never contains the launch token, the handshake URL or pod/credential details. | | Sandbox not provisioned, wake timeout, handshake failure | `503` + `Retry-After: 10`. The handshake error body is a fixed message: it never contains the launch token, the handshake URL or pod/credential details. |
| User over `MAX_IN_FLIGHT_PER_USER` | `503` + `Retry-After` (HTTP); the existing in-flight requests are untouched | | User over `MAX_IN_FLIGHT_PER_USER` | `503` + `Retry-After` (HTTP); the existing in-flight requests are untouched |
| Upstream connection/protocol failure | `502` | | Upstream connection/protocol failure | `502` |
| Kubernetes API error (API unreachable/throttled, RBAC `403`, `5xx`) | `503` + `Retry-After: 10`. The body is a fixed message; the Kubernetes `Status` payload (which can name cluster objects) is never proxied, and the API `status`/`reason` is logged once |
| WS auth failure (unknown user, bad token, cross-origin) | accept, then close `4401` | | WS auth failure (unknown user, bad token, cross-origin) | accept, then close `4401` |
| WS per-user concurrency cap | accept, then close `1011` | | WS per-user concurrency cap | accept, then close `1011` |
| WS upstream/handshake failure | accept, then close `1011` | | WS upstream/handshake failure | accept, then close `1011` |
| WS Kubernetes API error | accept, then close `1011` |
| WS message from DSH over `WS_MAX_MESSAGE_BYTES` | the upstream/relay closes the connection (websockets' message-too-big path) | | WS message from DSH over `WS_MAX_MESSAGE_BYTES` | the upstream/relay closes the connection (websockets' message-too-big path) |
A close code can only be delivered after the WebSocket upgrade is accepted; A close code can only be delivered after the WebSocket upgrade is accepted;
@@ -232,8 +230,7 @@ would never be sent). HTTPS deployments must set `COOKIE_SECURE=true`.
The gateway's ServiceAccount needs, in its namespace: The gateway's ServiceAccount needs, in its namespace:
* `sandboxclaims` (`extensions.agents.x-k8s.io`): get, list, create, patch, delete * `sandboxclaims` (`extensions.agents.x-k8s.io`): get, list, create, delete
(patch rotates the token-hash annotation when a session is minted)
* `sandboxes` (`agents.x-k8s.io`): get, list, patch * `sandboxes` (`agents.x-k8s.io`): get, list, patch
* `pods`: get, list * `pods`: get, list
* `pods/log`: get * `pods/log`: get

View File

@@ -1510,13 +1510,7 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
user = websocket.path_params["user"] user = websocket.path_params["user"]
if not valid_user(user): if not valid_user(user):
return await self._ws_reject(websocket, 4401) return await self._ws_reject(websocket, 4401)
try: claim = await self.k8s.get_claim(user)
claim = await self.k8s.get_claim(user)
except ApiException:
# Starlette has no exception handler for WebSocket scopes, so an
# unhandled API error would escape as a traceback and an unclean
# close. 1011 keeps the HTTP 503 meaning on the WS channel.
return await self._ws_reject(websocket, 1011)
if claim is None: if claim is None:
return await self._ws_reject(websocket, 4401) return await self._ws_reject(websocket, 4401)
token, cookie_user = self.credentials(websocket.headers, websocket.cookies) token, cookie_user = self.credentials(websocket.headers, websocket.cookies)
@@ -1752,28 +1746,6 @@ async def client_gone(request: Request, exc: ClientDisconnect) -> Response:
return Response(status_code=CLIENT_CLOSED_REQUEST) return Response(status_code=CLIENT_CLOSED_REQUEST)
async def k8s_api_error(request: Request, exc: ApiException) -> Response:
"""Answer a Kubernetes API failure with 503 instead of a bare 500.
The gateway is the only path between the browser and the cluster, so an
API outage, a throttle or a missing RBAC verb is a server-side
availability problem, not a client error. A 503 keeps ingress and
browser retry semantics honest, and the body is a fixed message: the
Kubernetes ``Status`` payload can name cluster objects and is never
proxied. The API ``status``/``reason`` is logged once, credential-free.
"""
print(
"gateway: kubernetes API error: "
f"status={getattr(exc, 'status', None)} reason={getattr(exc, 'reason', None)}",
flush=True,
)
return JSONResponse(
{"error": "cluster API unavailable, retry shortly"},
status_code=503,
headers={"Retry-After": "10"},
)
def create_app( def create_app(
settings: Settings | None = None, settings: Settings | None = None,
k8s=None, k8s=None,
@@ -1806,7 +1778,7 @@ def create_app(
] ]
app = Starlette( app = Starlette(
routes=routes, routes=routes,
exception_handlers={ClientDisconnect: client_gone, ApiException: k8s_api_error}, exception_handlers={ClientDisconnect: client_gone},
lifespan=_lifespan(gateway, http_client if owns_http_client else None), lifespan=_lifespan(gateway, http_client if owns_http_client else None),
) )
app.state.gateway = gateway app.state.gateway = gateway

View File

@@ -1176,14 +1176,12 @@ async def test_delete_missing_claim_is_404(env):
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_unexpected_api_error_returns_503(env): async def test_unexpected_api_error_propagates(env):
add_user(env.k8s) add_user(env.k8s)
env.k8s.get_claim_error = ApiException(status=500) env.k8s.get_claim_error = ApiException(status=500)
async with asgi_client(env) as client: async with asgi_client(env) as client:
response = await client.get("/users/alice") with pytest.raises(ApiException):
assert response.status_code == 503 await client.get("/users/alice")
assert response.headers["retry-after"] == "10"
assert response.json() == {"error": "cluster API unavailable, retry shortly"}
@pytest.mark.asyncio @pytest.mark.asyncio
@@ -1833,16 +1831,6 @@ def test_ws_unauthenticated_closes_4401(env):
assert exc.value.code == 4401 assert exc.value.code == 4401
def test_ws_kubernetes_api_error_closes_1011(env):
add_user(env.k8s)
env.k8s.get_claim_error = ApiException(status=503)
with TestClient(env.app) as client:
with client.websocket_connect("/u/alice/api/remote.mux") as ws:
with pytest.raises(WebSocketDisconnect) as exc:
ws.receive_text()
assert exc.value.code == 1011
def test_ws_cookie_cross_origin_closes_4401(env): def test_ws_cookie_cross_origin_closes_4401(env):
add_user(env.k8s) add_user(env.k8s)
with TestClient(env.app) as client: with TestClient(env.app) as client:
@@ -2161,19 +2149,6 @@ def test_remote_user_returns_valid_and_rejects_invalid(env):
assert g.remote_user({}) is None assert g.remote_user({}) is None
@pytest.mark.asyncio
async def test_sso_landing_returns_503_when_claim_patch_is_forbidden(env):
"""A missing RBAC verb on ``patch sandboxclaims`` used to escape as a 500."""
enable_sso(env)
env.k8s.claims["alice"] = {"metadata": {"name": claim_name("alice")}}
env.k8s.patch_error = ApiException(status=403)
async with asgi_client(env) as client:
response = await client.get("/", headers={"X-Forwarded-User": "alice"})
assert response.status_code == 503
assert response.headers["retry-after"] == "10"
assert response.json() == {"error": "cluster API unavailable, retry shortly"}
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_sso_landing_new_user_shows_provisioning(env): async def test_sso_landing_new_user_shows_provisioning(env):
enable_sso(env) enable_sso(env)

View File

@@ -26,8 +26,7 @@ metadata:
namespace: dsh-demo namespace: dsh-demo
--- ---
# Least privilege: # Least privilege:
# - claims are created/deleted per user (signup/delete) and patched only to # - claims are created/deleted per user (signup/delete);
# rotate the token-hash annotation when a session is minted;
# - sandboxes are read, and patched only to flip operatingMode (suspend/resume); # - sandboxes are read, and patched only to flip operatingMode (suspend/resume);
# - pods are read, and pods/log is read to capture DSH's launch token from # - pods are read, and pods/log is read to capture DSH's launch token from
# the startup log line (`dsh web: ...?token=...`). # the startup log line (`dsh web: ...?token=...`).
@@ -40,7 +39,7 @@ metadata:
rules: rules:
- apiGroups: ["extensions.agents.x-k8s.io"] - apiGroups: ["extensions.agents.x-k8s.io"]
resources: ["sandboxclaims"] resources: ["sandboxclaims"]
verbs: ["get", "list", "create", "patch", "delete"] verbs: ["get", "list", "create", "delete"]
- apiGroups: ["agents.x-k8s.io"] - apiGroups: ["agents.x-k8s.io"]
resources: ["sandboxes"] resources: ["sandboxes"]
verbs: ["get", "list", "patch"] verbs: ["get", "list", "patch"]

View File

@@ -139,15 +139,13 @@ kubectl apply -f manifests/exposure/gateway-api-sso/
# 5. Preflight: fails loudly on missing Secrets, a bad oauth2-proxy cookie # 5. Preflight: fails loudly on missing Secrets, a bad oauth2-proxy cookie
# secret, missing gateway SSO env, a dangling TLS reference, leftover # secret, missing gateway SSO env, a dangling TLS reference, leftover
# placeholders, a second path to the gateway, a Role without `patch` on # placeholders, or a second path to the gateway.
# sandboxclaims, or missing SSO NetworkPolicies.
./scripts/preflight-sso.sh ./scripts/preflight-sso.sh
``` ```
The preflight is read-only and exits non-zero on any failure, naming the exact The preflight is read-only and exits non-zero on any failure, naming the exact
fix. Run it again after rotating a Secret or changing the public host; it also fix. Run it again after rotating a Secret or changing the public host; it also
verifies that the oauth2-proxy cookie secret is a valid AES key length (16, 24 verifies that the oauth2-proxy cookie secret decodes to 16, 24 or 32 bytes.
or 32 raw bytes, or URL-safe unpadded base64 that decodes to one of those).
### SSO trust boundary ### SSO trust boundary

View File

@@ -1,6 +1,5 @@
# REQUIRED with the SSO exposure variant: enforce the trust boundary at the # REQUIRED with the SSO exposure variant: enforce the trust boundary at the
# network layer. Applying the SSO variant without this file is demo-only. # network layer. Applying the SSO variant without this file is demo-only.
# `scripts/preflight-sso.sh` fails while either policy below is missing.
# #
# The chain is Traefik (Gateway API) -> oauth2-proxy -> dsh-gateway. These # The chain is Traefik (Gateway API) -> oauth2-proxy -> dsh-gateway. These
# policies make it the ONLY chain: # policies make it the ONLY chain:

View File

@@ -20,14 +20,7 @@
# applied oauth2-proxy args, Gateway listeners or HTTPRoute hostnames; # applied oauth2-proxy args, Gateway listeners or HTTPRoute hostnames;
# 6. (warning) no Ingress in the namespace creates a second path to # 6. (warning) no Ingress in the namespace creates a second path to
# dsh-gateway. With SSO enabled the Service must be reachable only through # dsh-gateway. With SSO enabled the Service must be reachable only through
# oauth2-proxy; # oauth2-proxy.
# 7. the gateway ServiceAccount may `patch` sandboxclaims. The SSO landing
# path rotates the token-hash annotation whenever the browser has no valid
# gateway session (first visit, expired/foreign cookie, gateway restart),
# so without that verb every such sign-in answers 503;
# 8. the SSO variant's required NetworkPolicies are applied. Without them any
# Pod in the cluster can reach dsh-gateway:8080 and spoof the identity
# header, so the exposure is demo-only.
# #
# Exit status: 0 when every check passes (warnings allowed), 1 on any failure. # Exit status: 0 when every check passes (warnings allowed), 1 on any failure.
# Every failure prints the fix. This script only reads cluster state. # Every failure prints the fix. This script only reads cluster state.
@@ -42,9 +35,6 @@
# OAUTH_SECRET oauth2-proxy Secret name (default: oauth2-proxy) # OAUTH_SECRET oauth2-proxy Secret name (default: oauth2-proxy)
# TLS_SECRET TLS Secret name the Gateway should reference (default: # TLS_SECRET TLS Secret name the Gateway should reference (default:
# dsh-demo-tls) # dsh-demo-tls)
# GATEWAY_ROLE Role/RoleBinding name that grants the gateway access
# (default: dsh-gateway; only used when impersonation is
# unavailable and the permission must be read statically)
set -euo pipefail set -euo pipefail
cd "$(dirname "$0")/.." cd "$(dirname "$0")/.."
@@ -53,7 +43,6 @@ GATEWAY_DEPLOY="${GATEWAY_DEPLOY:-dsh-gateway}"
OAUTH_DEPLOY="${OAUTH_DEPLOY:-oauth2-proxy}" OAUTH_DEPLOY="${OAUTH_DEPLOY:-oauth2-proxy}"
OAUTH_SECRET="${OAUTH_SECRET:-oauth2-proxy}" OAUTH_SECRET="${OAUTH_SECRET:-oauth2-proxy}"
TLS_SECRET="${TLS_SECRET:-dsh-demo-tls}" TLS_SECRET="${TLS_SECRET:-dsh-demo-tls}"
GATEWAY_ROLE="${GATEWAY_ROLE:-dsh-gateway}"
FAILED=0 FAILED=0
pass() { printf 'PASS %s\n' "$*"; } pass() { printf 'PASS %s\n' "$*"; }
@@ -94,38 +83,21 @@ check_secret_key "$OAUTH_SECRET" client-secret \
check_secret_key "$OAUTH_SECRET" cookie-secret \ check_secret_key "$OAUTH_SECRET" cookie-secret \
"Create it from manifests/exposure/gateway-api-sso/73-oauth2-proxy-secret.yaml.example." "Create it from manifests/exposure/gateway-api-sso/73-oauth2-proxy-secret.yaml.example."
# oauth2-proxy accepts the cookie secret either as raw bytes of length 16, 24 # The cookie secret must be URL-safe unpadded base64 of 16, 24 or 32 bytes;
# or 32, or as URL-safe unpadded base64 that decodes to one of those lengths # oauth2-proxy refuses anything else (the classic error is "44 bytes").
# (pkg/encryption.SecretBytes). The documented generator,
# `python3 -c 'import secrets; print(secrets.token_urlsafe(32))'`, produces the
# second form: 43 base64 characters that decode to 32 bytes.
if COOKIE_B64=$(kubectl -n "$NS" get secret "$OAUTH_SECRET" \ if COOKIE_B64=$(kubectl -n "$NS" get secret "$OAUTH_SECRET" \
-o jsonpath='{.data.cookie-secret}' 2>/dev/null); then -o jsonpath='{.data.cookie-secret}' 2>/dev/null); then
if [ -n "$COOKIE_B64" ]; then if [ -n "$COOKIE_B64" ]; then
COOKIE_RAW=$(printf '%s' "$COOKIE_B64" | base64 -d 2>/dev/null || true) if ! COOKIE_LEN=$(printf '%s' "$COOKIE_B64" | base64 -d 2>/dev/null | wc -c); then
COOKIE_LEN=${#COOKIE_RAW} fail "Secret $OAUTH_SECRET key 'cookie-secret' is not valid base64. Generate it with: python3 -c 'import secrets; print(secrets.token_urlsafe(32))'"
COOKIE_FORM=raw else
case "$COOKIE_LEN" in case "$COOKIE_LEN" in
16 | 24 | 32) ;; 16 | 24 | 32) pass "oauth2-proxy cookie-secret decodes to $COOKIE_LEN bytes" ;;
*) *)
# Mirror SecretBytes: only a strictly URL-safe base64 string is decoded; fail "oauth2-proxy cookie-secret decodes to $COOKIE_LEN bytes; it must be 16, 24 or 32. Generate a URL-safe unpadded value with: python3 -c 'import secrets; print(secrets.token_urlsafe(32))'"
# anything else is used as raw bytes (and is then the wrong length). ;;
COOKIE_FORM=base64 esac
case "$COOKIE_RAW" in fi
"" | *[!A-Za-z0-9_=-]*) COOKIE_LEN=0 ;;
*)
COOKIE_LEN=$(printf '%s' "$COOKIE_RAW" | tr -- '-_' '+/' | tr -d '=\n' \
| base64 -d 2>/dev/null | wc -c || true)
;;
esac
;;
esac
case "$COOKIE_LEN" in
16 | 24 | 32) pass "oauth2-proxy cookie-secret is $COOKIE_FORM with a $COOKIE_LEN-byte key" ;;
*)
fail "oauth2-proxy cookie-secret is not a valid length ($COOKIE_LEN bytes). It must be 16, 24 or 32 raw bytes, or URL-safe unpadded base64 that decodes to one of those lengths. Generate it with: python3 -c 'import secrets; print(secrets.token_urlsafe(32))'"
;;
esac
fi fi
fi fi
@@ -167,51 +139,6 @@ if env_is_set ALLOW_INSECURE_COOKIES; then
warn "ALLOW_INSECURE_COOKIES is still set on deploy/$GATEWAY_DEPLOY. It is ignored while COOKIE_SECURE=true but is a footgun: remove it with 'kubectl -n $NS set env deploy/$GATEWAY_DEPLOY ALLOW_INSECURE_COOKIES-'." warn "ALLOW_INSECURE_COOKIES is still set on deploy/$GATEWAY_DEPLOY. It is ignored while COOKIE_SECURE=true but is a footgun: remove it with 'kubectl -n $NS set env deploy/$GATEWAY_DEPLOY ALLOW_INSECURE_COOKIES-'."
fi fi
echo
echo "== Gateway RBAC (SSO token rotation)"
# The SSO landing path patches the token-hash annotation on the user's claim
# whenever the browser has no valid gateway session: first visit, an expired or
# foreign cookie, or a gateway restart (sessions live in the gateway's memory).
# Without `patch` every such GET / answers 503 and the user cannot sign in.
GATEWAY_SA=$(kubectl -n "$NS" get deployment "$GATEWAY_DEPLOY" \
-o jsonpath='{.spec.template.spec.serviceAccountName}' 2>/dev/null || true)
GATEWAY_SA=${GATEWAY_SA:-default}
CAN_I=$(kubectl auth can-i patch sandboxclaims.extensions.agents.x-k8s.io \
-n "$NS" --as="system:serviceaccount:$NS:$GATEWAY_SA" 2>&1 || true)
# read_gateway_rules: one "apiGroups|resources|verbs" line per rule of the
# Role (or ClusterRole) the gateway RoleBinding points at. Used only when
# `kubectl auth can-i --as` is unavailable (a read-only kubeconfig cannot
# impersonate); it is a static check, not an authoritative one.
read_gateway_rules() {
local rules_jsonpath kind name
rules_jsonpath='{range .rules[*]}{range .apiGroups[*]}{@}{","}{end}{"|"}{range .resources[*]}{@}{","}{end}{"|"}{range .verbs[*]}{@}{","}{end}{"\n"}{end}'
kind=$(kubectl -n "$NS" get rolebinding "$GATEWAY_ROLE" \
-o jsonpath='{.roleRef.kind}' 2>/dev/null || true)
name=$(kubectl -n "$NS" get rolebinding "$GATEWAY_ROLE" \
-o jsonpath='{.roleRef.name}' 2>/dev/null || true)
case "$kind" in
Role) kubectl -n "$NS" get role "$name" -o jsonpath="$rules_jsonpath" 2>/dev/null || true ;;
ClusterRole) kubectl get clusterrole "$name" -o jsonpath="$rules_jsonpath" 2>/dev/null || true ;;
esac
}
if [ "$CAN_I" = "yes" ]; then
pass "ServiceAccount $GATEWAY_SA can patch sandboxclaims (token rotation)"
elif [ "$CAN_I" = "no" ]; then
fail "ServiceAccount $GATEWAY_SA cannot patch sandboxclaims: SSO sign-in answers 503 whenever the browser has no valid gateway session (after a gateway restart, cookie expiry or in a new browser). Fix: grant 'patch' on sandboxclaims to Role $GATEWAY_ROLE (the shipped Role in manifests/50-gateway.yaml has it; re-applying just the Role/RoleBinding is enough). RBAC is evaluated per request; no gateway restart is needed."
else
RULES=$(read_gateway_rules)
if printf '%s\n' "$RULES" | awk -F'|' \
'$1 ~ /(^|,)extensions\.agents\.x-k8s\.io(,|$)/ && $2 ~ /(^|,)sandboxclaims(,|$)/ && $3 ~ /(^|,)patch(,|$)/ { found = 1 } END { exit found ? 0 : 1 }'; then
pass "the gateway Role grants patch on sandboxclaims (static check; impersonation unavailable)"
elif [ -n "$RULES" ]; then
fail "Role $GATEWAY_ROLE does not grant 'patch' on sandboxclaims: SSO sign-in answers 503 whenever the browser has no valid gateway session. Fix: grant that verb to the Role (the shipped Role in manifests/50-gateway.yaml has it; re-applying just the Role/RoleBinding is enough). RBAC is evaluated per request; no gateway restart is needed."
else
warn "could not verify that ServiceAccount $GATEWAY_SA may patch sandboxclaims ($CAN_I). Check manually that Role $GATEWAY_ROLE grants 'patch' on extensions.agents.x-k8s.io/sandboxclaims."
fi
fi
echo echo
echo "== SSO objects and placeholders" echo "== SSO objects and placeholders"
if kubectl -n "$NS" get deployment "$OAUTH_DEPLOY" >/dev/null 2>&1; then if kubectl -n "$NS" get deployment "$OAUTH_DEPLOY" >/dev/null 2>&1; then
@@ -299,18 +226,6 @@ if kubectl -n "$NS" get ingress >/dev/null 2>&1; then
fi fi
fi fi
# The SSO variant's policies are part of the variant: without them the
# gateway Service is reachable by any Pod, which can spoof the identity header.
echo
echo "== SSO NetworkPolicies (required with this variant)"
for POLICY in dsh-gateway-ingress oauth2-proxy-ingress; do
if kubectl -n "$NS" get networkpolicy "$POLICY" >/dev/null 2>&1; then
pass "NetworkPolicy $POLICY exists"
else
fail "NetworkPolicy $POLICY is missing: any Pod in the cluster can reach dsh-gateway:8080 and spoof X-Forwarded-User, so the SSO identity is not trustworthy. Fix: kubectl apply -f manifests/exposure/gateway-api-sso/75-networkpolicies.yaml"
fi
done
echo echo
if [ "$FAILED" -ne 0 ]; then if [ "$FAILED" -ne 0 ]; then
echo "preflight-sso: FAILED - fix the items above before exposing the gateway." >&2 echo "preflight-sso: FAILED - fix the items above before exposing the gateway." >&2

View File

@@ -29,6 +29,13 @@ NODE_RUNTIME_IMAGE=docker.io/node:24-trixie-slim@sha256:54f84b4cf8432b21aaffd5d3
# Base image for the example gateway (examples/dsh-agents-as-a-service/gateway). # Base image for the example gateway (examples/dsh-agents-as-a-service/gateway).
GATEWAY_PYTHON_IMAGE=docker.io/python:3.12-slim@sha256:aa943ce411423c7582288f18707ff66eddcbaeb510502d747a72aeb34f30d159 GATEWAY_PYTHON_IMAGE=docker.io/python:3.12-slim@sha256:aa943ce411423c7582288f18707ff66eddcbaeb510502d747a72aeb34f30d159
# Base images for the main DSH Web image (root Dockerfile), pinned by digest.
NODE_BUILD_IMAGE=docker.io/node:24-trixie@sha256:8c761eca2bf7eec965f946cf47b8090143cb41f45f2f8699af0cc0298d2d2d19
NODE_RUNTIME_IMAGE=docker.io/node:24-trixie-slim@sha256:54f84b4cf8432b21aaffd5d3ce8d41e12cea0d10d33c09d2f3c9879d49d6b446
# Base image for the example gateway (examples/dsh-agents-as-a-service/gateway).
GATEWAY_PYTHON_IMAGE=docker.io/python:3.12-slim@sha256:aa943ce411423c7582288f18707ff66eddcbaeb510502d747a72aeb34f30d159
# Control-plane versions used by the examples and their manifests. # Control-plane versions used by the examples and their manifests.
# The matching SHA-256s pin the release manifests that CI downloads and # The matching SHA-256s pin the release manifests that CI downloads and
# applies; update both the version and the digest when bumping either pin. # applies; update both the version and the digest when bumping either pin.