mirror of
https://github.com/sredevopsorg/deepseek-kubernetes-sandbox.git
synced 2026-11-03 09:23:25 +00:00
Compare commits
4 Commits
main
...
fix/securi
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
f4c0b52ee9 | ||
|
|
96837a43b6 | ||
|
|
2eb5c82ace | ||
|
|
ae8aba8241 |
5
.entire/.gitignore
vendored
5
.entire/.gitignore
vendored
@@ -1,5 +0,0 @@
|
|||||||
tmp/
|
|
||||||
settings.local.json
|
|
||||||
metadata/
|
|
||||||
logs/
|
|
||||||
redactors/local/
|
|
||||||
@@ -1,16 +0,0 @@
|
|||||||
{
|
|
||||||
"version": 1,
|
|
||||||
"agent": "dsh",
|
|
||||||
"bridge": "@sredevopsorg/dsh-entire-bridge",
|
|
||||||
"hook_names": [
|
|
||||||
"session-start",
|
|
||||||
"turn-start",
|
|
||||||
"turn-end",
|
|
||||||
"compaction",
|
|
||||||
"session-end",
|
|
||||||
"subagent-start",
|
|
||||||
"subagent-end"
|
|
||||||
],
|
|
||||||
"installed_at": "2026-11-02T06:02:27.816Z",
|
|
||||||
"local_dev": false
|
|
||||||
}
|
|
||||||
@@ -1,9 +0,0 @@
|
|||||||
{
|
|
||||||
"enabled": true,
|
|
||||||
"telemetry": true,
|
|
||||||
"checkpoints": {
|
|
||||||
"primary": {
|
|
||||||
"type": "git-refs"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -35,8 +35,7 @@ disclosure once a fix is available.
|
|||||||
exposure variant's policy is required, not optional: an exposure without it
|
exposure variant's policy is required, not optional: an exposure without it
|
||||||
is demo-only. Run `examples/dsh-agents-as-a-service/scripts/preflight-sso.sh`
|
is demo-only. Run `examples/dsh-agents-as-a-service/scripts/preflight-sso.sh`
|
||||||
after applying the variant; it checks the Secrets, gateway env, TLS
|
after applying the variant; it checks the Secrets, gateway env, TLS
|
||||||
reference, leftover placeholders, accidental second paths, the gateway
|
reference, leftover placeholders and accidental second paths.
|
||||||
Role's `patch` verb on `sandboxclaims` and the variant NetworkPolicies.
|
|
||||||
- **Insecure-cookie guard.** TLS terminates outside the gateway. Set
|
- **Insecure-cookie guard.** TLS terminates outside the gateway. Set
|
||||||
COOKIE_SECURE=true for HTTPS and remove ALLOW_INSECURE_COOKIES; the gateway
|
COOKIE_SECURE=true for HTTPS and remove ALLOW_INSECURE_COOKIES; the gateway
|
||||||
refuses to start with COOKIE_SECURE=false unless ALLOW_INSECURE_COOKIES=1 is
|
refuses to start with COOKIE_SECURE=false unless ALLOW_INSECURE_COOKIES=1 is
|
||||||
|
|||||||
@@ -518,10 +518,8 @@ front of **oauth2-proxy**, which authenticates the browser against GitHub
|
|||||||
reports **before** pointing real traffic at the Gateway. It verifies the
|
reports **before** pointing real traffic at the Gateway. It verifies the
|
||||||
Secrets (including a valid oauth2-proxy cookie secret length), the gateway's
|
Secrets (including a valid oauth2-proxy cookie secret length), the gateway's
|
||||||
`AUTH_PROXY_HEADER`/`COOKIE_SECURE`, that the Gateway's TLS certificate
|
`AUTH_PROXY_HEADER`/`COOKIE_SECURE`, that the Gateway's TLS certificate
|
||||||
reference resolves, that the gateway ServiceAccount may `patch`
|
reference resolves, and that no `your-github-org`/`dsh-demo.example.com`
|
||||||
`sandboxclaims` (SSO token rotation), that the variant's NetworkPolicies are
|
placeholder was left applied. Full verification steps are in the
|
||||||
applied, and that no `your-github-org`/`dsh-demo.example.com` placeholder was
|
|
||||||
left applied. Full verification steps are in the
|
|
||||||
[exposure README](manifests/exposure/README.md).
|
[exposure README](manifests/exposure/README.md).
|
||||||
|
|
||||||
> **Exposure without the variant NetworkPolicy is demo-only.** The SSO
|
> **Exposure without the variant NetworkPolicy is demo-only.** The SSO
|
||||||
@@ -601,7 +599,7 @@ applied by the base step.
|
|||||||
| Every browser is logged out after a gateway rollout/restart | Expected: sessions are opaque ids in gateway memory and a restart drops them. Sign in again; a production deployment needs shared session state or per-request re-authentication. The DSH sandboxes and their data are unaffected |
|
| Every browser is logged out after a gateway rollout/restart | Expected: sessions are opaque ids in gateway memory and a restart drops them. Sign in again; a production deployment needs shared session state or per-request re-authentication. The DSH sandboxes and their data are unaffected |
|
||||||
| WebSocket closes with `1011` for a busy client | `MAX_IN_FLIGHT_PER_USER` (per-user connection cap) or an upstream/handshake failure closed the socket; check the gateway logs to tell them apart |
|
| WebSocket closes with `1011` for a busy client | `MAX_IN_FLIGHT_PER_USER` (per-user connection cap) or an upstream/handshake failure closed the socket; check the gateway logs to tell them apart |
|
||||||
| WebSocket closes after a very large message from DSH | A message received from DSH exceeded `WS_MAX_MESSAGE_BYTES` (16 MiB), so the relay closed it instead of buffering; raise the limit only if DSH legitimately sends larger frames. Browser-to-DSH frames are bounded by uvicorn's own limit |
|
| WebSocket closes after a very large message from DSH | A message received from DSH exceeded `WS_MAX_MESSAGE_BYTES` (16 MiB), so the relay closed it instead of buffering; raise the limit only if DSH legitimately sends larger frames. Browser-to-DSH frames are bounded by uvicorn's own limit |
|
||||||
| `scripts/preflight-sso.sh` exits non-zero | It names the failing item: a missing Secret/key, a bad oauth2-proxy cookie-secret length, missing `AUTH_PROXY_HEADER`/`COOKIE_SECURE`, a dangling Gateway TLS reference, a leftover `your-github-org`/`dsh-demo.example.com` placeholder, a second path to the gateway (an Ingress), a gateway Role without `patch` on `sandboxclaims`, or a missing SSO NetworkPolicy. Fix all items before exposing the SSO variant |
|
| `scripts/preflight-sso.sh` exits non-zero | It names the failing item: a missing Secret/key, a bad oauth2-proxy cookie-secret length, missing `AUTH_PROXY_HEADER`/`COOKIE_SECURE`, a dangling Gateway TLS reference, a leftover `your-github-org`/`dsh-demo.example.com` placeholder, or a second path to the gateway (an Ingress). Fix all items before exposing the SSO variant |
|
||||||
| `409` on signup | User already has a claim; use `POST /login` with the existing token or delete the user first |
|
| `409` on signup | User already has a claim; use `POST /login` with the existing token or delete the user first |
|
||||||
| Cookie set but browser keeps getting `401` | `COOKIE_SECURE=true` while serving over plain HTTP (secure cookies are not sent); use HTTPS or set `false` **and** `ALLOW_INSECURE_COOKIES=1` for local demos |
|
| Cookie set but browser keeps getting `401` | `COOKIE_SECURE=true` while serving over plain HTTP (secure cookies are not sent); use HTTPS or set `false` **and** `ALLOW_INSECURE_COOKIES=1` for local demos |
|
||||||
| WebSocket closes immediately (`4401`) | Missing/expired gateway cookie, or the upgrade's `Origin` does not match `Host` (CSWSH check); DSH rejecting `Host` instead surfaces as close `1011` — add the host to `DSH_TRUSTED_HOSTS` |
|
| WebSocket closes immediately (`4401`) | Missing/expired gateway cookie, or the upgrade's `Origin` does not match `Host` (CSWSH check); DSH rejecting `Host` instead surfaces as close `1011` — add the host to `DSH_TRUSTED_HOSTS` |
|
||||||
|
|||||||
@@ -339,7 +339,7 @@ The gateway runs with a namespace-scoped ServiceAccount (`manifests/50-gateway.y
|
|||||||
|
|
||||||
| API group | Resource | Verbs | Why |
|
| API group | Resource | Verbs | Why |
|
||||||
| --- | --- | --- | --- |
|
| --- | --- | --- | --- |
|
||||||
| `extensions.agents.x-k8s.io` | `sandboxclaims` | `get`, `list`, `create`, `patch`, `delete` | signup, state, account deletion, token rotation |
|
| `extensions.agents.x-k8s.io` | `sandboxclaims` | `get`, `list`, `create`, `delete` | signup, state, account deletion |
|
||||||
| `agents.x-k8s.io` | `sandboxes` | `get`, `list`, `patch` | resolve DNS/pod, wake/suspend |
|
| `agents.x-k8s.io` | `sandboxes` | `get`, `list`, `patch` | resolve DNS/pod, wake/suspend |
|
||||||
| `""` (core) | `pods` | `get`, `list` | resolve the pod behind a sandbox |
|
| `""` (core) | `pods` | `get`, `list` | resolve the pod behind a sandbox |
|
||||||
| `""` (core) | `pods/log` | `get` | launch-token handshake |
|
| `""` (core) | `pods/log` | `get` | launch-token handshake |
|
||||||
@@ -395,8 +395,7 @@ Notes and limits:
|
|||||||
- **SSO is trust-on-hop.** `AUTH_PROXY_HEADER` makes the gateway trust the proxy-asserted
|
- **SSO is trust-on-hop.** `AUTH_PROXY_HEADER` makes the gateway trust the proxy-asserted
|
||||||
identity; the SSO variant must remain the only path to the Service, and its NetworkPolicy is
|
identity; the SSO variant must remain the only path to the Service, and its NetworkPolicy is
|
||||||
required, not optional. Run [`scripts/preflight-sso.sh`](../scripts/preflight-sso.sh) after
|
required, not optional. Run [`scripts/preflight-sso.sh`](../scripts/preflight-sso.sh) after
|
||||||
applying it: it fails when the Role does not grant `patch` on `sandboxclaims` (SSO token
|
applying it.
|
||||||
rotation) or when the variant's NetworkPolicies are missing.
|
|
||||||
|
|
||||||
## Failure modes
|
## Failure modes
|
||||||
|
|
||||||
@@ -405,7 +404,6 @@ Notes and limits:
|
|||||||
| DSH log line changed or rotated away | Proxy returns `503` (missing launch token) | Fix the parser or restart the pod so the token line is re-emitted |
|
| DSH log line changed or rotated away | Proxy returns `503` (missing launch token) | Fix the parser or restart the pod so the token line is re-emitted |
|
||||||
| Probe on `/` | Pod `CrashLoopBackOff`, `401` in probe logs | Use `/favicon.svg` for readiness/liveness (shipped templates do) |
|
| Probe on `/` | Pod `CrashLoopBackOff`, `401` in probe logs | Use `/favicon.svg` for readiness/liveness (shipped templates do) |
|
||||||
| Warm pool exhausted | JSON signup returns `202` (state `Waking`); form signup is slow | Increase `replicas` in the warm pool |
|
| Warm pool exhausted | JSON signup returns `202` (state `Waking`); form signup is slow | Increase `replicas` in the warm pool |
|
||||||
| Kubernetes API error (API down or throttled, missing RBAC verb) | HTTP `503` + `Retry-After`; WebSocket close `1011` | One credential-free log line with the API `status`/`reason`, plus a fixed body; the Kubernetes `Status` payload is never proxied. Check the ServiceAccount Role and the API server — RBAC is evaluated per request, so a Role fix needs no restart |
|
|
||||||
| Gateway restart | Idle sandboxes stop suspending; handshake cache and browser sessions lost | Idle clocks and handshake cache rebuild lazily; sessions do not, so browsers must sign in again after a rollout. Use a shared session store for production |
|
| Gateway restart | Idle sandboxes stop suspending; handshake cache and browser sessions lost | Idle clocks and handshake cache rebuild lazily; sessions do not, so browsers must sign in again after a rollout. Use a shared session store for production |
|
||||||
| Gateway configured `COOKIE_SECURE=false` without `ALLOW_INSECURE_COOKIES=1` | Gateway pod exits at startup / `CrashLoopBackOff` | Intentional guard against an insecure cookie on an HTTPS deployment; set `ALLOW_INSECURE_COOKIES=1` only for the local plain-HTTP demo, or set `COOKIE_SECURE=true` |
|
| Gateway configured `COOKIE_SECURE=false` without `ALLOW_INSECURE_COOKIES=1` | Gateway pod exits at startup / `CrashLoopBackOff` | Intentional guard against an insecure cookie on an HTTPS deployment; set `ALLOW_INSECURE_COOKIES=1` only for the local plain-HTTP demo, or set `COOKIE_SECURE=true` |
|
||||||
| Auth rate limit hit | `429` + `Retry-After` on signup/login | Per-client in-memory bucket (`AUTH_RATE_LIMIT`/`AUTH_RATE_WINDOW`); wait for the window or tune it. Behind a proxy, keying uses the rightmost `X-Forwarded-For` entry |
|
| Auth rate limit hit | `429` + `Retry-After` on signup/login | Per-client in-memory bucket (`AUTH_RATE_LIMIT`/`AUTH_RATE_WINDOW`); wait for the window or tune it. Behind a proxy, keying uses the rightmost `X-Forwarded-For` entry |
|
||||||
@@ -413,7 +411,7 @@ Notes and limits:
|
|||||||
| Per-user concurrency cap hit | HTTP `503` + `Retry-After`, or WebSocket close `1011` | `MAX_IN_FLIGHT_PER_USER` protects the single gateway from one busy tenant; reduce concurrency or raise the cap deliberately. These are per replica |
|
| Per-user concurrency cap hit | HTTP `503` + `Retry-After`, or WebSocket close `1011` | `MAX_IN_FLIGHT_PER_USER` protects the single gateway from one busy tenant; reduce concurrency or raise the cap deliberately. These are per replica |
|
||||||
| Oversized WebSocket message from DSH | Connection closes | `WS_MAX_MESSAGE_BYTES` (16 MiB) is enforced on upstream messages; raise it only if DSH legitimately needs larger frames |
|
| Oversized WebSocket message from DSH | Connection closes | `WS_MAX_MESSAGE_BYTES` (16 MiB) is enforced on upstream messages; raise it only if DSH legitimately needs larger frames |
|
||||||
| Session expiry / revocation | `401` (HTTP) or close `4401` (WebSocket) | Expected after logout, user delete, `/recreate`, `COOKIE_MAX_AGE` expiry or a gateway restart; sign in again |
|
| Session expiry / revocation | `401` (HTTP) or close `4401` (WebSocket) | Expected after logout, user delete, `/recreate`, `COOKIE_MAX_AGE` expiry or a gateway restart; sign in again |
|
||||||
| SSO variant applied without its NetworkPolicy (or with an Ingress) | Gateway accepts spoofed `X-Forwarded-User` | Demo-only configuration; keep `75-networkpolicies.yaml`, remove the second path, and run `scripts/preflight-sso.sh`, which fails when either policy is missing |
|
| SSO variant applied without its NetworkPolicy (or with an Ingress) | Gateway accepts spoofed `X-Forwarded-User` | Demo-only configuration; keep `75-networkpolicies.yaml`, remove the second path, and run `scripts/preflight-sso.sh` |
|
||||||
| Browser aborts a request (tab reload, back navigation) during a slow wake or handshake | No response reaches the client; nothing appears in the access log | If the abort lands before the response starts, the body read raises `ClientDisconnect` and the app answers `499` (`CLIENT_CLOSED_REQUEST`), which cannot reach the departed peer. Nothing reached DSH, no ERROR traceback is logged, and the in-flight slot is released. A request already streaming ends with its upstream status. |
|
| Browser aborts a request (tab reload, back navigation) during a slow wake or handshake | No response reaches the client; nothing appears in the access log | If the abort lands before the response starts, the body read raises `ClientDisconnect` and the app answers `499` (`CLIENT_CLOSED_REQUEST`), which cannot reach the departed peer. Nothing reached DSH, no ERROR traceback is logged, and the in-flight slot is released. A request already streaming ends with its upstream status. |
|
||||||
| DSH auth cookie expired mid-session | Proxy `401` once, then recovers | Gateway refreshes the handshake once |
|
| DSH auth cookie expired mid-session | Proxy `401` once, then recovers | Gateway refreshes the handshake once |
|
||||||
| Upstream pod name changed after resume | Cached cookie keyed by old pod UID is useless | Cache is invalidated on suspend; next request re-handshakes |
|
| Upstream pod name changed after resume | Cached cookie keyed by old pod UID is useless | Cache is invalidated on suspend; next request re-handshakes |
|
||||||
|
|||||||
@@ -338,10 +338,8 @@ The SSO policy is what turns the trust boundary in the
|
|||||||
The `dsh-gateway` Service must still stay ClusterIP; do not apply both exposure
|
The `dsh-gateway` Service must still stay ClusterIP; do not apply both exposure
|
||||||
variants. After applying the SSO variant, run
|
variants. After applying the SSO variant, run
|
||||||
[`scripts/preflight-sso.sh`](../scripts/preflight-sso.sh): it checks the
|
[`scripts/preflight-sso.sh`](../scripts/preflight-sso.sh): it checks the
|
||||||
gateway's SSO env, the Secrets, the TLS reference, leftover placeholders,
|
gateway's SSO env, the Secrets, the TLS reference, leftover placeholders and
|
||||||
accidental second paths (such as an Ingress), the gateway Role's `patch` verb on
|
accidental second paths (such as an Ingress).
|
||||||
`sandboxclaims` (SSO token rotation) and that both SSO NetworkPolicies are
|
|
||||||
applied.
|
|
||||||
|
|
||||||
Verify the negative case from a sandbox pod (it is not the proxy):
|
Verify the negative case from a sandbox pod (it is not the proxy):
|
||||||
|
|
||||||
|
|||||||
@@ -175,11 +175,9 @@ oauth2-proxy's egress to the gateway, DNS and public HTTPS.
|
|||||||
| Sandbox not provisioned, wake timeout, handshake failure | `503` + `Retry-After: 10`. The handshake error body is a fixed message: it never contains the launch token, the handshake URL or pod/credential details. |
|
| Sandbox not provisioned, wake timeout, handshake failure | `503` + `Retry-After: 10`. The handshake error body is a fixed message: it never contains the launch token, the handshake URL or pod/credential details. |
|
||||||
| User over `MAX_IN_FLIGHT_PER_USER` | `503` + `Retry-After` (HTTP); the existing in-flight requests are untouched |
|
| User over `MAX_IN_FLIGHT_PER_USER` | `503` + `Retry-After` (HTTP); the existing in-flight requests are untouched |
|
||||||
| Upstream connection/protocol failure | `502` |
|
| Upstream connection/protocol failure | `502` |
|
||||||
| Kubernetes API error (API unreachable/throttled, RBAC `403`, `5xx`) | `503` + `Retry-After: 10`. The body is a fixed message; the Kubernetes `Status` payload (which can name cluster objects) is never proxied, and the API `status`/`reason` is logged once |
|
|
||||||
| WS auth failure (unknown user, bad token, cross-origin) | accept, then close `4401` |
|
| WS auth failure (unknown user, bad token, cross-origin) | accept, then close `4401` |
|
||||||
| WS per-user concurrency cap | accept, then close `1011` |
|
| WS per-user concurrency cap | accept, then close `1011` |
|
||||||
| WS upstream/handshake failure | accept, then close `1011` |
|
| WS upstream/handshake failure | accept, then close `1011` |
|
||||||
| WS Kubernetes API error | accept, then close `1011` |
|
|
||||||
| WS message from DSH over `WS_MAX_MESSAGE_BYTES` | the upstream/relay closes the connection (websockets' message-too-big path) |
|
| WS message from DSH over `WS_MAX_MESSAGE_BYTES` | the upstream/relay closes the connection (websockets' message-too-big path) |
|
||||||
|
|
||||||
A close code can only be delivered after the WebSocket upgrade is accepted;
|
A close code can only be delivered after the WebSocket upgrade is accepted;
|
||||||
@@ -232,8 +230,7 @@ would never be sent). HTTPS deployments must set `COOKIE_SECURE=true`.
|
|||||||
|
|
||||||
The gateway's ServiceAccount needs, in its namespace:
|
The gateway's ServiceAccount needs, in its namespace:
|
||||||
|
|
||||||
* `sandboxclaims` (`extensions.agents.x-k8s.io`): get, list, create, patch, delete
|
* `sandboxclaims` (`extensions.agents.x-k8s.io`): get, list, create, delete
|
||||||
(patch rotates the token-hash annotation when a session is minted)
|
|
||||||
* `sandboxes` (`agents.x-k8s.io`): get, list, patch
|
* `sandboxes` (`agents.x-k8s.io`): get, list, patch
|
||||||
* `pods`: get, list
|
* `pods`: get, list
|
||||||
* `pods/log`: get
|
* `pods/log`: get
|
||||||
|
|||||||
@@ -1510,13 +1510,7 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
|
|||||||
user = websocket.path_params["user"]
|
user = websocket.path_params["user"]
|
||||||
if not valid_user(user):
|
if not valid_user(user):
|
||||||
return await self._ws_reject(websocket, 4401)
|
return await self._ws_reject(websocket, 4401)
|
||||||
try:
|
|
||||||
claim = await self.k8s.get_claim(user)
|
claim = await self.k8s.get_claim(user)
|
||||||
except ApiException:
|
|
||||||
# Starlette has no exception handler for WebSocket scopes, so an
|
|
||||||
# unhandled API error would escape as a traceback and an unclean
|
|
||||||
# close. 1011 keeps the HTTP 503 meaning on the WS channel.
|
|
||||||
return await self._ws_reject(websocket, 1011)
|
|
||||||
if claim is None:
|
if claim is None:
|
||||||
return await self._ws_reject(websocket, 4401)
|
return await self._ws_reject(websocket, 4401)
|
||||||
token, cookie_user = self.credentials(websocket.headers, websocket.cookies)
|
token, cookie_user = self.credentials(websocket.headers, websocket.cookies)
|
||||||
@@ -1752,28 +1746,6 @@ async def client_gone(request: Request, exc: ClientDisconnect) -> Response:
|
|||||||
return Response(status_code=CLIENT_CLOSED_REQUEST)
|
return Response(status_code=CLIENT_CLOSED_REQUEST)
|
||||||
|
|
||||||
|
|
||||||
async def k8s_api_error(request: Request, exc: ApiException) -> Response:
|
|
||||||
"""Answer a Kubernetes API failure with 503 instead of a bare 500.
|
|
||||||
|
|
||||||
The gateway is the only path between the browser and the cluster, so an
|
|
||||||
API outage, a throttle or a missing RBAC verb is a server-side
|
|
||||||
availability problem, not a client error. A 503 keeps ingress and
|
|
||||||
browser retry semantics honest, and the body is a fixed message: the
|
|
||||||
Kubernetes ``Status`` payload can name cluster objects and is never
|
|
||||||
proxied. The API ``status``/``reason`` is logged once, credential-free.
|
|
||||||
"""
|
|
||||||
print(
|
|
||||||
"gateway: kubernetes API error: "
|
|
||||||
f"status={getattr(exc, 'status', None)} reason={getattr(exc, 'reason', None)}",
|
|
||||||
flush=True,
|
|
||||||
)
|
|
||||||
return JSONResponse(
|
|
||||||
{"error": "cluster API unavailable, retry shortly"},
|
|
||||||
status_code=503,
|
|
||||||
headers={"Retry-After": "10"},
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def create_app(
|
def create_app(
|
||||||
settings: Settings | None = None,
|
settings: Settings | None = None,
|
||||||
k8s=None,
|
k8s=None,
|
||||||
@@ -1806,7 +1778,7 @@ def create_app(
|
|||||||
]
|
]
|
||||||
app = Starlette(
|
app = Starlette(
|
||||||
routes=routes,
|
routes=routes,
|
||||||
exception_handlers={ClientDisconnect: client_gone, ApiException: k8s_api_error},
|
exception_handlers={ClientDisconnect: client_gone},
|
||||||
lifespan=_lifespan(gateway, http_client if owns_http_client else None),
|
lifespan=_lifespan(gateway, http_client if owns_http_client else None),
|
||||||
)
|
)
|
||||||
app.state.gateway = gateway
|
app.state.gateway = gateway
|
||||||
|
|||||||
@@ -1176,14 +1176,12 @@ async def test_delete_missing_claim_is_404(env):
|
|||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
async def test_unexpected_api_error_returns_503(env):
|
async def test_unexpected_api_error_propagates(env):
|
||||||
add_user(env.k8s)
|
add_user(env.k8s)
|
||||||
env.k8s.get_claim_error = ApiException(status=500)
|
env.k8s.get_claim_error = ApiException(status=500)
|
||||||
async with asgi_client(env) as client:
|
async with asgi_client(env) as client:
|
||||||
response = await client.get("/users/alice")
|
with pytest.raises(ApiException):
|
||||||
assert response.status_code == 503
|
await client.get("/users/alice")
|
||||||
assert response.headers["retry-after"] == "10"
|
|
||||||
assert response.json() == {"error": "cluster API unavailable, retry shortly"}
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
@@ -1833,16 +1831,6 @@ def test_ws_unauthenticated_closes_4401(env):
|
|||||||
assert exc.value.code == 4401
|
assert exc.value.code == 4401
|
||||||
|
|
||||||
|
|
||||||
def test_ws_kubernetes_api_error_closes_1011(env):
|
|
||||||
add_user(env.k8s)
|
|
||||||
env.k8s.get_claim_error = ApiException(status=503)
|
|
||||||
with TestClient(env.app) as client:
|
|
||||||
with client.websocket_connect("/u/alice/api/remote.mux") as ws:
|
|
||||||
with pytest.raises(WebSocketDisconnect) as exc:
|
|
||||||
ws.receive_text()
|
|
||||||
assert exc.value.code == 1011
|
|
||||||
|
|
||||||
|
|
||||||
def test_ws_cookie_cross_origin_closes_4401(env):
|
def test_ws_cookie_cross_origin_closes_4401(env):
|
||||||
add_user(env.k8s)
|
add_user(env.k8s)
|
||||||
with TestClient(env.app) as client:
|
with TestClient(env.app) as client:
|
||||||
@@ -2161,19 +2149,6 @@ def test_remote_user_returns_valid_and_rejects_invalid(env):
|
|||||||
assert g.remote_user({}) is None
|
assert g.remote_user({}) is None
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
async def test_sso_landing_returns_503_when_claim_patch_is_forbidden(env):
|
|
||||||
"""A missing RBAC verb on ``patch sandboxclaims`` used to escape as a 500."""
|
|
||||||
enable_sso(env)
|
|
||||||
env.k8s.claims["alice"] = {"metadata": {"name": claim_name("alice")}}
|
|
||||||
env.k8s.patch_error = ApiException(status=403)
|
|
||||||
async with asgi_client(env) as client:
|
|
||||||
response = await client.get("/", headers={"X-Forwarded-User": "alice"})
|
|
||||||
assert response.status_code == 503
|
|
||||||
assert response.headers["retry-after"] == "10"
|
|
||||||
assert response.json() == {"error": "cluster API unavailable, retry shortly"}
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
async def test_sso_landing_new_user_shows_provisioning(env):
|
async def test_sso_landing_new_user_shows_provisioning(env):
|
||||||
enable_sso(env)
|
enable_sso(env)
|
||||||
|
|||||||
@@ -26,8 +26,7 @@ metadata:
|
|||||||
namespace: dsh-demo
|
namespace: dsh-demo
|
||||||
---
|
---
|
||||||
# Least privilege:
|
# Least privilege:
|
||||||
# - claims are created/deleted per user (signup/delete) and patched only to
|
# - claims are created/deleted per user (signup/delete);
|
||||||
# rotate the token-hash annotation when a session is minted;
|
|
||||||
# - sandboxes are read, and patched only to flip operatingMode (suspend/resume);
|
# - sandboxes are read, and patched only to flip operatingMode (suspend/resume);
|
||||||
# - pods are read, and pods/log is read to capture DSH's launch token from
|
# - pods are read, and pods/log is read to capture DSH's launch token from
|
||||||
# the startup log line (`dsh web: ...?token=...`).
|
# the startup log line (`dsh web: ...?token=...`).
|
||||||
@@ -40,7 +39,7 @@ metadata:
|
|||||||
rules:
|
rules:
|
||||||
- apiGroups: ["extensions.agents.x-k8s.io"]
|
- apiGroups: ["extensions.agents.x-k8s.io"]
|
||||||
resources: ["sandboxclaims"]
|
resources: ["sandboxclaims"]
|
||||||
verbs: ["get", "list", "create", "patch", "delete"]
|
verbs: ["get", "list", "create", "delete"]
|
||||||
- apiGroups: ["agents.x-k8s.io"]
|
- apiGroups: ["agents.x-k8s.io"]
|
||||||
resources: ["sandboxes"]
|
resources: ["sandboxes"]
|
||||||
verbs: ["get", "list", "patch"]
|
verbs: ["get", "list", "patch"]
|
||||||
|
|||||||
@@ -139,15 +139,13 @@ kubectl apply -f manifests/exposure/gateway-api-sso/
|
|||||||
|
|
||||||
# 5. Preflight: fails loudly on missing Secrets, a bad oauth2-proxy cookie
|
# 5. Preflight: fails loudly on missing Secrets, a bad oauth2-proxy cookie
|
||||||
# secret, missing gateway SSO env, a dangling TLS reference, leftover
|
# secret, missing gateway SSO env, a dangling TLS reference, leftover
|
||||||
# placeholders, a second path to the gateway, a Role without `patch` on
|
# placeholders, or a second path to the gateway.
|
||||||
# sandboxclaims, or missing SSO NetworkPolicies.
|
|
||||||
./scripts/preflight-sso.sh
|
./scripts/preflight-sso.sh
|
||||||
```
|
```
|
||||||
|
|
||||||
The preflight is read-only and exits non-zero on any failure, naming the exact
|
The preflight is read-only and exits non-zero on any failure, naming the exact
|
||||||
fix. Run it again after rotating a Secret or changing the public host; it also
|
fix. Run it again after rotating a Secret or changing the public host; it also
|
||||||
verifies that the oauth2-proxy cookie secret is a valid AES key length (16, 24
|
verifies that the oauth2-proxy cookie secret decodes to 16, 24 or 32 bytes.
|
||||||
or 32 raw bytes, or URL-safe unpadded base64 that decodes to one of those).
|
|
||||||
|
|
||||||
### SSO trust boundary
|
### SSO trust boundary
|
||||||
|
|
||||||
|
|||||||
@@ -1,6 +1,5 @@
|
|||||||
# REQUIRED with the SSO exposure variant: enforce the trust boundary at the
|
# REQUIRED with the SSO exposure variant: enforce the trust boundary at the
|
||||||
# network layer. Applying the SSO variant without this file is demo-only.
|
# network layer. Applying the SSO variant without this file is demo-only.
|
||||||
# `scripts/preflight-sso.sh` fails while either policy below is missing.
|
|
||||||
#
|
#
|
||||||
# The chain is Traefik (Gateway API) -> oauth2-proxy -> dsh-gateway. These
|
# The chain is Traefik (Gateway API) -> oauth2-proxy -> dsh-gateway. These
|
||||||
# policies make it the ONLY chain:
|
# policies make it the ONLY chain:
|
||||||
|
|||||||
@@ -20,14 +20,7 @@
|
|||||||
# applied oauth2-proxy args, Gateway listeners or HTTPRoute hostnames;
|
# applied oauth2-proxy args, Gateway listeners or HTTPRoute hostnames;
|
||||||
# 6. (warning) no Ingress in the namespace creates a second path to
|
# 6. (warning) no Ingress in the namespace creates a second path to
|
||||||
# dsh-gateway. With SSO enabled the Service must be reachable only through
|
# dsh-gateway. With SSO enabled the Service must be reachable only through
|
||||||
# oauth2-proxy;
|
# oauth2-proxy.
|
||||||
# 7. the gateway ServiceAccount may `patch` sandboxclaims. The SSO landing
|
|
||||||
# path rotates the token-hash annotation whenever the browser has no valid
|
|
||||||
# gateway session (first visit, expired/foreign cookie, gateway restart),
|
|
||||||
# so without that verb every such sign-in answers 503;
|
|
||||||
# 8. the SSO variant's required NetworkPolicies are applied. Without them any
|
|
||||||
# Pod in the cluster can reach dsh-gateway:8080 and spoof the identity
|
|
||||||
# header, so the exposure is demo-only.
|
|
||||||
#
|
#
|
||||||
# Exit status: 0 when every check passes (warnings allowed), 1 on any failure.
|
# Exit status: 0 when every check passes (warnings allowed), 1 on any failure.
|
||||||
# Every failure prints the fix. This script only reads cluster state.
|
# Every failure prints the fix. This script only reads cluster state.
|
||||||
@@ -42,9 +35,6 @@
|
|||||||
# OAUTH_SECRET oauth2-proxy Secret name (default: oauth2-proxy)
|
# OAUTH_SECRET oauth2-proxy Secret name (default: oauth2-proxy)
|
||||||
# TLS_SECRET TLS Secret name the Gateway should reference (default:
|
# TLS_SECRET TLS Secret name the Gateway should reference (default:
|
||||||
# dsh-demo-tls)
|
# dsh-demo-tls)
|
||||||
# GATEWAY_ROLE Role/RoleBinding name that grants the gateway access
|
|
||||||
# (default: dsh-gateway; only used when impersonation is
|
|
||||||
# unavailable and the permission must be read statically)
|
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
cd "$(dirname "$0")/.."
|
cd "$(dirname "$0")/.."
|
||||||
|
|
||||||
@@ -53,7 +43,6 @@ GATEWAY_DEPLOY="${GATEWAY_DEPLOY:-dsh-gateway}"
|
|||||||
OAUTH_DEPLOY="${OAUTH_DEPLOY:-oauth2-proxy}"
|
OAUTH_DEPLOY="${OAUTH_DEPLOY:-oauth2-proxy}"
|
||||||
OAUTH_SECRET="${OAUTH_SECRET:-oauth2-proxy}"
|
OAUTH_SECRET="${OAUTH_SECRET:-oauth2-proxy}"
|
||||||
TLS_SECRET="${TLS_SECRET:-dsh-demo-tls}"
|
TLS_SECRET="${TLS_SECRET:-dsh-demo-tls}"
|
||||||
GATEWAY_ROLE="${GATEWAY_ROLE:-dsh-gateway}"
|
|
||||||
|
|
||||||
FAILED=0
|
FAILED=0
|
||||||
pass() { printf 'PASS %s\n' "$*"; }
|
pass() { printf 'PASS %s\n' "$*"; }
|
||||||
@@ -94,39 +83,22 @@ check_secret_key "$OAUTH_SECRET" client-secret \
|
|||||||
check_secret_key "$OAUTH_SECRET" cookie-secret \
|
check_secret_key "$OAUTH_SECRET" cookie-secret \
|
||||||
"Create it from manifests/exposure/gateway-api-sso/73-oauth2-proxy-secret.yaml.example."
|
"Create it from manifests/exposure/gateway-api-sso/73-oauth2-proxy-secret.yaml.example."
|
||||||
|
|
||||||
# oauth2-proxy accepts the cookie secret either as raw bytes of length 16, 24
|
# The cookie secret must be URL-safe unpadded base64 of 16, 24 or 32 bytes;
|
||||||
# or 32, or as URL-safe unpadded base64 that decodes to one of those lengths
|
# oauth2-proxy refuses anything else (the classic error is "44 bytes").
|
||||||
# (pkg/encryption.SecretBytes). The documented generator,
|
|
||||||
# `python3 -c 'import secrets; print(secrets.token_urlsafe(32))'`, produces the
|
|
||||||
# second form: 43 base64 characters that decode to 32 bytes.
|
|
||||||
if COOKIE_B64=$(kubectl -n "$NS" get secret "$OAUTH_SECRET" \
|
if COOKIE_B64=$(kubectl -n "$NS" get secret "$OAUTH_SECRET" \
|
||||||
-o jsonpath='{.data.cookie-secret}' 2>/dev/null); then
|
-o jsonpath='{.data.cookie-secret}' 2>/dev/null); then
|
||||||
if [ -n "$COOKIE_B64" ]; then
|
if [ -n "$COOKIE_B64" ]; then
|
||||||
COOKIE_RAW=$(printf '%s' "$COOKIE_B64" | base64 -d 2>/dev/null || true)
|
if ! COOKIE_LEN=$(printf '%s' "$COOKIE_B64" | base64 -d 2>/dev/null | wc -c); then
|
||||||
COOKIE_LEN=${#COOKIE_RAW}
|
fail "Secret $OAUTH_SECRET key 'cookie-secret' is not valid base64. Generate it with: python3 -c 'import secrets; print(secrets.token_urlsafe(32))'"
|
||||||
COOKIE_FORM=raw
|
else
|
||||||
case "$COOKIE_LEN" in
|
case "$COOKIE_LEN" in
|
||||||
16 | 24 | 32) ;;
|
16 | 24 | 32) pass "oauth2-proxy cookie-secret decodes to $COOKIE_LEN bytes" ;;
|
||||||
*)
|
*)
|
||||||
# Mirror SecretBytes: only a strictly URL-safe base64 string is decoded;
|
fail "oauth2-proxy cookie-secret decodes to $COOKIE_LEN bytes; it must be 16, 24 or 32. Generate a URL-safe unpadded value with: python3 -c 'import secrets; print(secrets.token_urlsafe(32))'"
|
||||||
# anything else is used as raw bytes (and is then the wrong length).
|
|
||||||
COOKIE_FORM=base64
|
|
||||||
case "$COOKIE_RAW" in
|
|
||||||
"" | *[!A-Za-z0-9_=-]*) COOKIE_LEN=0 ;;
|
|
||||||
*)
|
|
||||||
COOKIE_LEN=$(printf '%s' "$COOKIE_RAW" | tr -- '-_' '+/' | tr -d '=\n' \
|
|
||||||
| base64 -d 2>/dev/null | wc -c || true)
|
|
||||||
;;
|
|
||||||
esac
|
|
||||||
;;
|
|
||||||
esac
|
|
||||||
case "$COOKIE_LEN" in
|
|
||||||
16 | 24 | 32) pass "oauth2-proxy cookie-secret is $COOKIE_FORM with a $COOKIE_LEN-byte key" ;;
|
|
||||||
*)
|
|
||||||
fail "oauth2-proxy cookie-secret is not a valid length ($COOKIE_LEN bytes). It must be 16, 24 or 32 raw bytes, or URL-safe unpadded base64 that decodes to one of those lengths. Generate it with: python3 -c 'import secrets; print(secrets.token_urlsafe(32))'"
|
|
||||||
;;
|
;;
|
||||||
esac
|
esac
|
||||||
fi
|
fi
|
||||||
|
fi
|
||||||
fi
|
fi
|
||||||
|
|
||||||
check_secret_key "$TLS_SECRET" tls.crt \
|
check_secret_key "$TLS_SECRET" tls.crt \
|
||||||
@@ -167,51 +139,6 @@ if env_is_set ALLOW_INSECURE_COOKIES; then
|
|||||||
warn "ALLOW_INSECURE_COOKIES is still set on deploy/$GATEWAY_DEPLOY. It is ignored while COOKIE_SECURE=true but is a footgun: remove it with 'kubectl -n $NS set env deploy/$GATEWAY_DEPLOY ALLOW_INSECURE_COOKIES-'."
|
warn "ALLOW_INSECURE_COOKIES is still set on deploy/$GATEWAY_DEPLOY. It is ignored while COOKIE_SECURE=true but is a footgun: remove it with 'kubectl -n $NS set env deploy/$GATEWAY_DEPLOY ALLOW_INSECURE_COOKIES-'."
|
||||||
fi
|
fi
|
||||||
|
|
||||||
echo
|
|
||||||
echo "== Gateway RBAC (SSO token rotation)"
|
|
||||||
# The SSO landing path patches the token-hash annotation on the user's claim
|
|
||||||
# whenever the browser has no valid gateway session: first visit, an expired or
|
|
||||||
# foreign cookie, or a gateway restart (sessions live in the gateway's memory).
|
|
||||||
# Without `patch` every such GET / answers 503 and the user cannot sign in.
|
|
||||||
GATEWAY_SA=$(kubectl -n "$NS" get deployment "$GATEWAY_DEPLOY" \
|
|
||||||
-o jsonpath='{.spec.template.spec.serviceAccountName}' 2>/dev/null || true)
|
|
||||||
GATEWAY_SA=${GATEWAY_SA:-default}
|
|
||||||
CAN_I=$(kubectl auth can-i patch sandboxclaims.extensions.agents.x-k8s.io \
|
|
||||||
-n "$NS" --as="system:serviceaccount:$NS:$GATEWAY_SA" 2>&1 || true)
|
|
||||||
|
|
||||||
# read_gateway_rules: one "apiGroups|resources|verbs" line per rule of the
|
|
||||||
# Role (or ClusterRole) the gateway RoleBinding points at. Used only when
|
|
||||||
# `kubectl auth can-i --as` is unavailable (a read-only kubeconfig cannot
|
|
||||||
# impersonate); it is a static check, not an authoritative one.
|
|
||||||
read_gateway_rules() {
|
|
||||||
local rules_jsonpath kind name
|
|
||||||
rules_jsonpath='{range .rules[*]}{range .apiGroups[*]}{@}{","}{end}{"|"}{range .resources[*]}{@}{","}{end}{"|"}{range .verbs[*]}{@}{","}{end}{"\n"}{end}'
|
|
||||||
kind=$(kubectl -n "$NS" get rolebinding "$GATEWAY_ROLE" \
|
|
||||||
-o jsonpath='{.roleRef.kind}' 2>/dev/null || true)
|
|
||||||
name=$(kubectl -n "$NS" get rolebinding "$GATEWAY_ROLE" \
|
|
||||||
-o jsonpath='{.roleRef.name}' 2>/dev/null || true)
|
|
||||||
case "$kind" in
|
|
||||||
Role) kubectl -n "$NS" get role "$name" -o jsonpath="$rules_jsonpath" 2>/dev/null || true ;;
|
|
||||||
ClusterRole) kubectl get clusterrole "$name" -o jsonpath="$rules_jsonpath" 2>/dev/null || true ;;
|
|
||||||
esac
|
|
||||||
}
|
|
||||||
|
|
||||||
if [ "$CAN_I" = "yes" ]; then
|
|
||||||
pass "ServiceAccount $GATEWAY_SA can patch sandboxclaims (token rotation)"
|
|
||||||
elif [ "$CAN_I" = "no" ]; then
|
|
||||||
fail "ServiceAccount $GATEWAY_SA cannot patch sandboxclaims: SSO sign-in answers 503 whenever the browser has no valid gateway session (after a gateway restart, cookie expiry or in a new browser). Fix: grant 'patch' on sandboxclaims to Role $GATEWAY_ROLE (the shipped Role in manifests/50-gateway.yaml has it; re-applying just the Role/RoleBinding is enough). RBAC is evaluated per request; no gateway restart is needed."
|
|
||||||
else
|
|
||||||
RULES=$(read_gateway_rules)
|
|
||||||
if printf '%s\n' "$RULES" | awk -F'|' \
|
|
||||||
'$1 ~ /(^|,)extensions\.agents\.x-k8s\.io(,|$)/ && $2 ~ /(^|,)sandboxclaims(,|$)/ && $3 ~ /(^|,)patch(,|$)/ { found = 1 } END { exit found ? 0 : 1 }'; then
|
|
||||||
pass "the gateway Role grants patch on sandboxclaims (static check; impersonation unavailable)"
|
|
||||||
elif [ -n "$RULES" ]; then
|
|
||||||
fail "Role $GATEWAY_ROLE does not grant 'patch' on sandboxclaims: SSO sign-in answers 503 whenever the browser has no valid gateway session. Fix: grant that verb to the Role (the shipped Role in manifests/50-gateway.yaml has it; re-applying just the Role/RoleBinding is enough). RBAC is evaluated per request; no gateway restart is needed."
|
|
||||||
else
|
|
||||||
warn "could not verify that ServiceAccount $GATEWAY_SA may patch sandboxclaims ($CAN_I). Check manually that Role $GATEWAY_ROLE grants 'patch' on extensions.agents.x-k8s.io/sandboxclaims."
|
|
||||||
fi
|
|
||||||
fi
|
|
||||||
|
|
||||||
echo
|
echo
|
||||||
echo "== SSO objects and placeholders"
|
echo "== SSO objects and placeholders"
|
||||||
if kubectl -n "$NS" get deployment "$OAUTH_DEPLOY" >/dev/null 2>&1; then
|
if kubectl -n "$NS" get deployment "$OAUTH_DEPLOY" >/dev/null 2>&1; then
|
||||||
@@ -299,18 +226,6 @@ if kubectl -n "$NS" get ingress >/dev/null 2>&1; then
|
|||||||
fi
|
fi
|
||||||
fi
|
fi
|
||||||
|
|
||||||
# The SSO variant's policies are part of the variant: without them the
|
|
||||||
# gateway Service is reachable by any Pod, which can spoof the identity header.
|
|
||||||
echo
|
|
||||||
echo "== SSO NetworkPolicies (required with this variant)"
|
|
||||||
for POLICY in dsh-gateway-ingress oauth2-proxy-ingress; do
|
|
||||||
if kubectl -n "$NS" get networkpolicy "$POLICY" >/dev/null 2>&1; then
|
|
||||||
pass "NetworkPolicy $POLICY exists"
|
|
||||||
else
|
|
||||||
fail "NetworkPolicy $POLICY is missing: any Pod in the cluster can reach dsh-gateway:8080 and spoof X-Forwarded-User, so the SSO identity is not trustworthy. Fix: kubectl apply -f manifests/exposure/gateway-api-sso/75-networkpolicies.yaml"
|
|
||||||
fi
|
|
||||||
done
|
|
||||||
|
|
||||||
echo
|
echo
|
||||||
if [ "$FAILED" -ne 0 ]; then
|
if [ "$FAILED" -ne 0 ]; then
|
||||||
echo "preflight-sso: FAILED - fix the items above before exposing the gateway." >&2
|
echo "preflight-sso: FAILED - fix the items above before exposing the gateway." >&2
|
||||||
|
|||||||
@@ -29,6 +29,13 @@ NODE_RUNTIME_IMAGE=docker.io/node:24-trixie-slim@sha256:54f84b4cf8432b21aaffd5d3
|
|||||||
# Base image for the example gateway (examples/dsh-agents-as-a-service/gateway).
|
# Base image for the example gateway (examples/dsh-agents-as-a-service/gateway).
|
||||||
GATEWAY_PYTHON_IMAGE=docker.io/python:3.12-slim@sha256:aa943ce411423c7582288f18707ff66eddcbaeb510502d747a72aeb34f30d159
|
GATEWAY_PYTHON_IMAGE=docker.io/python:3.12-slim@sha256:aa943ce411423c7582288f18707ff66eddcbaeb510502d747a72aeb34f30d159
|
||||||
|
|
||||||
|
# Base images for the main DSH Web image (root Dockerfile), pinned by digest.
|
||||||
|
NODE_BUILD_IMAGE=docker.io/node:24-trixie@sha256:8c761eca2bf7eec965f946cf47b8090143cb41f45f2f8699af0cc0298d2d2d19
|
||||||
|
NODE_RUNTIME_IMAGE=docker.io/node:24-trixie-slim@sha256:54f84b4cf8432b21aaffd5d3ce8d41e12cea0d10d33c09d2f3c9879d49d6b446
|
||||||
|
|
||||||
|
# Base image for the example gateway (examples/dsh-agents-as-a-service/gateway).
|
||||||
|
GATEWAY_PYTHON_IMAGE=docker.io/python:3.12-slim@sha256:aa943ce411423c7582288f18707ff66eddcbaeb510502d747a72aeb34f30d159
|
||||||
|
|
||||||
# Control-plane versions used by the examples and their manifests.
|
# Control-plane versions used by the examples and their manifests.
|
||||||
# The matching SHA-256s pin the release manifests that CI downloads and
|
# The matching SHA-256s pin the release manifests that CI downloads and
|
||||||
# applies; update both the version and the digest when bumping either pin.
|
# applies; update both the version and the digest when bumping either pin.
|
||||||
|
|||||||
Reference in New Issue
Block a user