mirror of
https://github.com/sredevopsorg/deepseek-kubernetes-sandbox.git
synced 2026-11-03 01:13:24 +00:00
Compare commits
3 Commits
fix/securi
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
2178e58d57 | ||
|
|
fbe886f3b9 | ||
|
|
c5b8afde48 |
5
.entire/.gitignore
vendored
Normal file
5
.entire/.gitignore
vendored
Normal file
@@ -0,0 +1,5 @@
|
|||||||
|
tmp/
|
||||||
|
settings.local.json
|
||||||
|
metadata/
|
||||||
|
logs/
|
||||||
|
redactors/local/
|
||||||
16
.entire/dsh-hooks.json
Normal file
16
.entire/dsh-hooks.json
Normal file
@@ -0,0 +1,16 @@
|
|||||||
|
{
|
||||||
|
"version": 1,
|
||||||
|
"agent": "dsh",
|
||||||
|
"bridge": "@sredevopsorg/dsh-entire-bridge",
|
||||||
|
"hook_names": [
|
||||||
|
"session-start",
|
||||||
|
"turn-start",
|
||||||
|
"turn-end",
|
||||||
|
"compaction",
|
||||||
|
"session-end",
|
||||||
|
"subagent-start",
|
||||||
|
"subagent-end"
|
||||||
|
],
|
||||||
|
"installed_at": "2026-11-02T06:02:27.816Z",
|
||||||
|
"local_dev": false
|
||||||
|
}
|
||||||
9
.entire/settings.json
Normal file
9
.entire/settings.json
Normal file
@@ -0,0 +1,9 @@
|
|||||||
|
{
|
||||||
|
"enabled": true,
|
||||||
|
"telemetry": true,
|
||||||
|
"checkpoints": {
|
||||||
|
"primary": {
|
||||||
|
"type": "git-refs"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -339,7 +339,7 @@ The gateway runs with a namespace-scoped ServiceAccount (`manifests/50-gateway.y
|
|||||||
|
|
||||||
| API group | Resource | Verbs | Why |
|
| API group | Resource | Verbs | Why |
|
||||||
| --- | --- | --- | --- |
|
| --- | --- | --- | --- |
|
||||||
| `extensions.agents.x-k8s.io` | `sandboxclaims` | `get`, `list`, `create`, `delete` | signup, state, account deletion |
|
| `extensions.agents.x-k8s.io` | `sandboxclaims` | `get`, `list`, `create`, `patch`, `delete` | signup, state, account deletion, token rotation |
|
||||||
| `agents.x-k8s.io` | `sandboxes` | `get`, `list`, `patch` | resolve DNS/pod, wake/suspend |
|
| `agents.x-k8s.io` | `sandboxes` | `get`, `list`, `patch` | resolve DNS/pod, wake/suspend |
|
||||||
| `""` (core) | `pods` | `get`, `list` | resolve the pod behind a sandbox |
|
| `""` (core) | `pods` | `get`, `list` | resolve the pod behind a sandbox |
|
||||||
| `""` (core) | `pods/log` | `get` | launch-token handshake |
|
| `""` (core) | `pods/log` | `get` | launch-token handshake |
|
||||||
@@ -404,6 +404,7 @@ Notes and limits:
|
|||||||
| DSH log line changed or rotated away | Proxy returns `503` (missing launch token) | Fix the parser or restart the pod so the token line is re-emitted |
|
| DSH log line changed or rotated away | Proxy returns `503` (missing launch token) | Fix the parser or restart the pod so the token line is re-emitted |
|
||||||
| Probe on `/` | Pod `CrashLoopBackOff`, `401` in probe logs | Use `/favicon.svg` for readiness/liveness (shipped templates do) |
|
| Probe on `/` | Pod `CrashLoopBackOff`, `401` in probe logs | Use `/favicon.svg` for readiness/liveness (shipped templates do) |
|
||||||
| Warm pool exhausted | JSON signup returns `202` (state `Waking`); form signup is slow | Increase `replicas` in the warm pool |
|
| Warm pool exhausted | JSON signup returns `202` (state `Waking`); form signup is slow | Increase `replicas` in the warm pool |
|
||||||
|
| Kubernetes API error (API down or throttled, missing RBAC verb) | HTTP `503` + `Retry-After`; WebSocket close `1011` | One credential-free log line with the API `status`/`reason`, plus a fixed body; the Kubernetes `Status` payload is never proxied. Check the ServiceAccount Role and the API server — RBAC is evaluated per request, so a Role fix needs no restart |
|
||||||
| Gateway restart | Idle sandboxes stop suspending; handshake cache and browser sessions lost | Idle clocks and handshake cache rebuild lazily; sessions do not, so browsers must sign in again after a rollout. Use a shared session store for production |
|
| Gateway restart | Idle sandboxes stop suspending; handshake cache and browser sessions lost | Idle clocks and handshake cache rebuild lazily; sessions do not, so browsers must sign in again after a rollout. Use a shared session store for production |
|
||||||
| Gateway configured `COOKIE_SECURE=false` without `ALLOW_INSECURE_COOKIES=1` | Gateway pod exits at startup / `CrashLoopBackOff` | Intentional guard against an insecure cookie on an HTTPS deployment; set `ALLOW_INSECURE_COOKIES=1` only for the local plain-HTTP demo, or set `COOKIE_SECURE=true` |
|
| Gateway configured `COOKIE_SECURE=false` without `ALLOW_INSECURE_COOKIES=1` | Gateway pod exits at startup / `CrashLoopBackOff` | Intentional guard against an insecure cookie on an HTTPS deployment; set `ALLOW_INSECURE_COOKIES=1` only for the local plain-HTTP demo, or set `COOKIE_SECURE=true` |
|
||||||
| Auth rate limit hit | `429` + `Retry-After` on signup/login | Per-client in-memory bucket (`AUTH_RATE_LIMIT`/`AUTH_RATE_WINDOW`); wait for the window or tune it. Behind a proxy, keying uses the rightmost `X-Forwarded-For` entry |
|
| Auth rate limit hit | `429` + `Retry-After` on signup/login | Per-client in-memory bucket (`AUTH_RATE_LIMIT`/`AUTH_RATE_WINDOW`); wait for the window or tune it. Behind a proxy, keying uses the rightmost `X-Forwarded-For` entry |
|
||||||
|
|||||||
@@ -175,9 +175,11 @@ oauth2-proxy's egress to the gateway, DNS and public HTTPS.
|
|||||||
| Sandbox not provisioned, wake timeout, handshake failure | `503` + `Retry-After: 10`. The handshake error body is a fixed message: it never contains the launch token, the handshake URL or pod/credential details. |
|
| Sandbox not provisioned, wake timeout, handshake failure | `503` + `Retry-After: 10`. The handshake error body is a fixed message: it never contains the launch token, the handshake URL or pod/credential details. |
|
||||||
| User over `MAX_IN_FLIGHT_PER_USER` | `503` + `Retry-After` (HTTP); the existing in-flight requests are untouched |
|
| User over `MAX_IN_FLIGHT_PER_USER` | `503` + `Retry-After` (HTTP); the existing in-flight requests are untouched |
|
||||||
| Upstream connection/protocol failure | `502` |
|
| Upstream connection/protocol failure | `502` |
|
||||||
|
| Kubernetes API error (API unreachable/throttled, RBAC `403`, `5xx`) | `503` + `Retry-After: 10`. The body is a fixed message; the Kubernetes `Status` payload (which can name cluster objects) is never proxied, and the API `status`/`reason` is logged once |
|
||||||
| WS auth failure (unknown user, bad token, cross-origin) | accept, then close `4401` |
|
| WS auth failure (unknown user, bad token, cross-origin) | accept, then close `4401` |
|
||||||
| WS per-user concurrency cap | accept, then close `1011` |
|
| WS per-user concurrency cap | accept, then close `1011` |
|
||||||
| WS upstream/handshake failure | accept, then close `1011` |
|
| WS upstream/handshake failure | accept, then close `1011` |
|
||||||
|
| WS Kubernetes API error | accept, then close `1011` |
|
||||||
| WS message from DSH over `WS_MAX_MESSAGE_BYTES` | the upstream/relay closes the connection (websockets' message-too-big path) |
|
| WS message from DSH over `WS_MAX_MESSAGE_BYTES` | the upstream/relay closes the connection (websockets' message-too-big path) |
|
||||||
|
|
||||||
A close code can only be delivered after the WebSocket upgrade is accepted;
|
A close code can only be delivered after the WebSocket upgrade is accepted;
|
||||||
@@ -230,7 +232,8 @@ would never be sent). HTTPS deployments must set `COOKIE_SECURE=true`.
|
|||||||
|
|
||||||
The gateway's ServiceAccount needs, in its namespace:
|
The gateway's ServiceAccount needs, in its namespace:
|
||||||
|
|
||||||
* `sandboxclaims` (`extensions.agents.x-k8s.io`): get, list, create, delete
|
* `sandboxclaims` (`extensions.agents.x-k8s.io`): get, list, create, patch, delete
|
||||||
|
(patch rotates the token-hash annotation when a session is minted)
|
||||||
* `sandboxes` (`agents.x-k8s.io`): get, list, patch
|
* `sandboxes` (`agents.x-k8s.io`): get, list, patch
|
||||||
* `pods`: get, list
|
* `pods`: get, list
|
||||||
* `pods/log`: get
|
* `pods/log`: get
|
||||||
|
|||||||
@@ -1510,7 +1510,13 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
|
|||||||
user = websocket.path_params["user"]
|
user = websocket.path_params["user"]
|
||||||
if not valid_user(user):
|
if not valid_user(user):
|
||||||
return await self._ws_reject(websocket, 4401)
|
return await self._ws_reject(websocket, 4401)
|
||||||
|
try:
|
||||||
claim = await self.k8s.get_claim(user)
|
claim = await self.k8s.get_claim(user)
|
||||||
|
except ApiException:
|
||||||
|
# Starlette has no exception handler for WebSocket scopes, so an
|
||||||
|
# unhandled API error would escape as a traceback and an unclean
|
||||||
|
# close. 1011 keeps the HTTP 503 meaning on the WS channel.
|
||||||
|
return await self._ws_reject(websocket, 1011)
|
||||||
if claim is None:
|
if claim is None:
|
||||||
return await self._ws_reject(websocket, 4401)
|
return await self._ws_reject(websocket, 4401)
|
||||||
token, cookie_user = self.credentials(websocket.headers, websocket.cookies)
|
token, cookie_user = self.credentials(websocket.headers, websocket.cookies)
|
||||||
@@ -1746,6 +1752,28 @@ async def client_gone(request: Request, exc: ClientDisconnect) -> Response:
|
|||||||
return Response(status_code=CLIENT_CLOSED_REQUEST)
|
return Response(status_code=CLIENT_CLOSED_REQUEST)
|
||||||
|
|
||||||
|
|
||||||
|
async def k8s_api_error(request: Request, exc: ApiException) -> Response:
|
||||||
|
"""Answer a Kubernetes API failure with 503 instead of a bare 500.
|
||||||
|
|
||||||
|
The gateway is the only path between the browser and the cluster, so an
|
||||||
|
API outage, a throttle or a missing RBAC verb is a server-side
|
||||||
|
availability problem, not a client error. A 503 keeps ingress and
|
||||||
|
browser retry semantics honest, and the body is a fixed message: the
|
||||||
|
Kubernetes ``Status`` payload can name cluster objects and is never
|
||||||
|
proxied. The API ``status``/``reason`` is logged once, credential-free.
|
||||||
|
"""
|
||||||
|
print(
|
||||||
|
"gateway: kubernetes API error: "
|
||||||
|
f"status={getattr(exc, 'status', None)} reason={getattr(exc, 'reason', None)}",
|
||||||
|
flush=True,
|
||||||
|
)
|
||||||
|
return JSONResponse(
|
||||||
|
{"error": "cluster API unavailable, retry shortly"},
|
||||||
|
status_code=503,
|
||||||
|
headers={"Retry-After": "10"},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def create_app(
|
def create_app(
|
||||||
settings: Settings | None = None,
|
settings: Settings | None = None,
|
||||||
k8s=None,
|
k8s=None,
|
||||||
@@ -1778,7 +1806,7 @@ def create_app(
|
|||||||
]
|
]
|
||||||
app = Starlette(
|
app = Starlette(
|
||||||
routes=routes,
|
routes=routes,
|
||||||
exception_handlers={ClientDisconnect: client_gone},
|
exception_handlers={ClientDisconnect: client_gone, ApiException: k8s_api_error},
|
||||||
lifespan=_lifespan(gateway, http_client if owns_http_client else None),
|
lifespan=_lifespan(gateway, http_client if owns_http_client else None),
|
||||||
)
|
)
|
||||||
app.state.gateway = gateway
|
app.state.gateway = gateway
|
||||||
|
|||||||
@@ -1176,12 +1176,14 @@ async def test_delete_missing_claim_is_404(env):
|
|||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
async def test_unexpected_api_error_propagates(env):
|
async def test_unexpected_api_error_returns_503(env):
|
||||||
add_user(env.k8s)
|
add_user(env.k8s)
|
||||||
env.k8s.get_claim_error = ApiException(status=500)
|
env.k8s.get_claim_error = ApiException(status=500)
|
||||||
async with asgi_client(env) as client:
|
async with asgi_client(env) as client:
|
||||||
with pytest.raises(ApiException):
|
response = await client.get("/users/alice")
|
||||||
await client.get("/users/alice")
|
assert response.status_code == 503
|
||||||
|
assert response.headers["retry-after"] == "10"
|
||||||
|
assert response.json() == {"error": "cluster API unavailable, retry shortly"}
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
@@ -1831,6 +1833,16 @@ def test_ws_unauthenticated_closes_4401(env):
|
|||||||
assert exc.value.code == 4401
|
assert exc.value.code == 4401
|
||||||
|
|
||||||
|
|
||||||
|
def test_ws_kubernetes_api_error_closes_1011(env):
|
||||||
|
add_user(env.k8s)
|
||||||
|
env.k8s.get_claim_error = ApiException(status=503)
|
||||||
|
with TestClient(env.app) as client:
|
||||||
|
with client.websocket_connect("/u/alice/api/remote.mux") as ws:
|
||||||
|
with pytest.raises(WebSocketDisconnect) as exc:
|
||||||
|
ws.receive_text()
|
||||||
|
assert exc.value.code == 1011
|
||||||
|
|
||||||
|
|
||||||
def test_ws_cookie_cross_origin_closes_4401(env):
|
def test_ws_cookie_cross_origin_closes_4401(env):
|
||||||
add_user(env.k8s)
|
add_user(env.k8s)
|
||||||
with TestClient(env.app) as client:
|
with TestClient(env.app) as client:
|
||||||
@@ -2149,6 +2161,19 @@ def test_remote_user_returns_valid_and_rejects_invalid(env):
|
|||||||
assert g.remote_user({}) is None
|
assert g.remote_user({}) is None
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_sso_landing_returns_503_when_claim_patch_is_forbidden(env):
|
||||||
|
"""A missing RBAC verb on ``patch sandboxclaims`` used to escape as a 500."""
|
||||||
|
enable_sso(env)
|
||||||
|
env.k8s.claims["alice"] = {"metadata": {"name": claim_name("alice")}}
|
||||||
|
env.k8s.patch_error = ApiException(status=403)
|
||||||
|
async with asgi_client(env) as client:
|
||||||
|
response = await client.get("/", headers={"X-Forwarded-User": "alice"})
|
||||||
|
assert response.status_code == 503
|
||||||
|
assert response.headers["retry-after"] == "10"
|
||||||
|
assert response.json() == {"error": "cluster API unavailable, retry shortly"}
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
async def test_sso_landing_new_user_shows_provisioning(env):
|
async def test_sso_landing_new_user_shows_provisioning(env):
|
||||||
enable_sso(env)
|
enable_sso(env)
|
||||||
|
|||||||
@@ -26,7 +26,8 @@ metadata:
|
|||||||
namespace: dsh-demo
|
namespace: dsh-demo
|
||||||
---
|
---
|
||||||
# Least privilege:
|
# Least privilege:
|
||||||
# - claims are created/deleted per user (signup/delete);
|
# - claims are created/deleted per user (signup/delete) and patched only to
|
||||||
|
# rotate the token-hash annotation when a session is minted;
|
||||||
# - sandboxes are read, and patched only to flip operatingMode (suspend/resume);
|
# - sandboxes are read, and patched only to flip operatingMode (suspend/resume);
|
||||||
# - pods are read, and pods/log is read to capture DSH's launch token from
|
# - pods are read, and pods/log is read to capture DSH's launch token from
|
||||||
# the startup log line (`dsh web: ...?token=...`).
|
# the startup log line (`dsh web: ...?token=...`).
|
||||||
@@ -39,7 +40,7 @@ metadata:
|
|||||||
rules:
|
rules:
|
||||||
- apiGroups: ["extensions.agents.x-k8s.io"]
|
- apiGroups: ["extensions.agents.x-k8s.io"]
|
||||||
resources: ["sandboxclaims"]
|
resources: ["sandboxclaims"]
|
||||||
verbs: ["get", "list", "create", "delete"]
|
verbs: ["get", "list", "create", "patch", "delete"]
|
||||||
- apiGroups: ["agents.x-k8s.io"]
|
- apiGroups: ["agents.x-k8s.io"]
|
||||||
resources: ["sandboxes"]
|
resources: ["sandboxes"]
|
||||||
verbs: ["get", "list", "patch"]
|
verbs: ["get", "list", "patch"]
|
||||||
|
|||||||
@@ -29,13 +29,6 @@ NODE_RUNTIME_IMAGE=docker.io/node:24-trixie-slim@sha256:54f84b4cf8432b21aaffd5d3
|
|||||||
# Base image for the example gateway (examples/dsh-agents-as-a-service/gateway).
|
# Base image for the example gateway (examples/dsh-agents-as-a-service/gateway).
|
||||||
GATEWAY_PYTHON_IMAGE=docker.io/python:3.12-slim@sha256:aa943ce411423c7582288f18707ff66eddcbaeb510502d747a72aeb34f30d159
|
GATEWAY_PYTHON_IMAGE=docker.io/python:3.12-slim@sha256:aa943ce411423c7582288f18707ff66eddcbaeb510502d747a72aeb34f30d159
|
||||||
|
|
||||||
# Base images for the main DSH Web image (root Dockerfile), pinned by digest.
|
|
||||||
NODE_BUILD_IMAGE=docker.io/node:24-trixie@sha256:8c761eca2bf7eec965f946cf47b8090143cb41f45f2f8699af0cc0298d2d2d19
|
|
||||||
NODE_RUNTIME_IMAGE=docker.io/node:24-trixie-slim@sha256:54f84b4cf8432b21aaffd5d3ce8d41e12cea0d10d33c09d2f3c9879d49d6b446
|
|
||||||
|
|
||||||
# Base image for the example gateway (examples/dsh-agents-as-a-service/gateway).
|
|
||||||
GATEWAY_PYTHON_IMAGE=docker.io/python:3.12-slim@sha256:aa943ce411423c7582288f18707ff66eddcbaeb510502d747a72aeb34f30d159
|
|
||||||
|
|
||||||
# Control-plane versions used by the examples and their manifests.
|
# Control-plane versions used by the examples and their manifests.
|
||||||
# The matching SHA-256s pin the release manifests that CI downloads and
|
# The matching SHA-256s pin the release manifests that CI downloads and
|
||||||
# applies; update both the version and the digest when bumping either pin.
|
# applies; update both the version and the digest when bumping either pin.
|
||||||
|
|||||||
Reference in New Issue
Block a user