nicolas-georger/deepseek-kubernetes-sandbox
1
0
Fork 0
mirror of https://github.com/sredevopsorg/deepseek-kubernetes-sandbox.git synced 2026-11-03 01:13:24 +00:00
Code Projects Releases Activity

fix(gateway)!: replace token cookies with revocable sessions and harden abuse controls (#40)

Browse Source 
- Replace user:token cookies with opaque server-side sessions, revocable on logout, delete, recreate and expiry; bind every credential to the path user and add cross-user regression tests.
- Rate-limit /users and /login, add optional signup invite codes and form same-origin checks, and unify login failures to 401.
- Require explicit ALLOW_INSECURE_COOKIES for plain HTTP, use __Host-dsh_gw_session when secure, and fix secure cookie deletion.
- Strip client-supplied forwarding/SSO headers, add an optional upstream allowlist, per-user in-flight and WebSocket message caps, credential-free handshake errors, and sweep-time state pruning.
- Digest-pin base images, hash-pin gateway dependencies, verify DSH_COMMIT in the build, and run the gateway image as non-root with a pinned Python base.
- Update manifests, production checklist, architecture/security docs and add an SSO preflight script.

BREAKING CHANGE: existing dsh_gw_session=<user>:<token> cookies are invalid; users must sign in again. Unknown users on POST /login now return 401 instead of 404, and the gateway refuses COOKIE_SECURE=false unless ALLOW_INSECURE_COOKIES=1.

Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
This commit is contained in:
SREDevOps.org Agent
2026-10-31 00:57:35 -03:00
committed by GitHub
parent 2f959e6afb
commit 1a929e3e24
33 changed files with 5450 additions and 319 deletions
Download Patch File Download Diff File Expand all files Collapse all files

134
examples/dsh-agents-as-a-service/docs/architecture.md
View File

@@ -72,8 +72,10 @@ Each variant also ships a NetworkPolicy: `ingress/61-networkpolicy.yaml` limits
to Traefik pods, and `gateway-api-sso/75-networkpolicies.yaml` limits the gateway to
oauth2-proxy, oauth2-proxy ingress to the Gateway implementation, and oauth2-proxy egress to
DNS/public HTTPS/the gateway. The SSO policy turns the trust boundary into pod-traffic
enforcement; the caveats (Traefik label, `hostNetwork`, kubelet probes) are in
[optional hardening](optional-hardening.md).
enforcement. These policies are **required for production**: exposing the gateway by applying
only the Gateway/HTTPRoute/Ingress objects is demo-only. The sandbox-side policy and the
gateway's own egress policy are separate (the API-server endpoint is cluster-specific); see
[optional hardening](optional-hardening.md#gateway-egress-required-for-production).
## Flows
@@ -87,20 +89,22 @@ sequenceDiagram
participant K as Kubernetes API
participant C as agent-sandbox controller
B->>G: POST /users (form: user=bob, Origin)
G->>G: validate user name and body size
B->>G: POST /users (form: user=bob, invite, Origin)
G->>G: rate limit, invite check, validate user name and body size
G->>K: create SandboxClaim dsh-bob (warmPoolRef)
K->>C: reconcile claim
C-->>K: adopt spare, Ready, status.sandbox.name = dsh-pool-xxxx
G->>K: read claim (status) + sandbox + pod
G-->>B: 303 Location /u/bob/ + Set-Cookie dsh_gw_session
G->>G: mint opaque sid, store sid -> (user, expiry)
G-->>B: 303 Location /u/bob/ + Set-Cookie dsh_gw_session=<sid>
Note over G: JSON clients instead get 201 (Ready),<br/>202 (accepted, still waking), or 409 (conflict)
```
Form signup waits up to `WAKE_TIMEOUT` for the claim to become `Ready`, then redirects; the
first proxied request also waits for `Ready` and performs the DSH handshake (below). A `409`
means the user already has a claim, in which case `POST /login` (bearer token) or the existing
cookie is the right path.
cookie is the right path. The cookie value is an opaque session id: it is meaningless outside
this gateway process and is never accepted as a bearer token.
### First request / proxy (with handshake)
@@ -112,8 +116,8 @@ sequenceDiagram
participant K as Kubernetes API
participant D as DSH Web pod
B->>G: GET /u/bob/ (Cookie dsh_gw_session)
G->>G: authenticate cookie, user = bob
B->>G: GET /u/bob/ (Cookie dsh_gw_session=<sid>)
G->>G: look up sid -> session (valid, not expired), user = bob
G->>K: resolve claim -> sandbox -> pod
alt sandbox Suspended
G->>K: patch operatingMode=Running (wake)
@@ -140,6 +144,32 @@ Inside `proxy_http` the order is auth, in-flight slot, sandbox resolution/wake,
body read, upstream send. An upload aborted mid-body may therefore already have paid for a wake
and a handshake, but it never reaches DSH.
### Browser sessions
The browser never holds a DSH credential. On signup/login the gateway mints a random session id
(`secrets.token_urlsafe(32)`), stores `sid -> Session(user, expires_at)` in memory, and sets it
as the `HttpOnly`, `SameSite=Lax` cookie `dsh_gw_session` (`__Host-dsh_gw_session` when
`COOKIE_SECURE=true`). Every browser request is resolved through that table; the cookie value is
not a bearer token and cannot be replayed against another user's path.
Sessions end on explicit logout, user delete, `/recreate` (all sessions for that user) and
expiry (`COOKIE_MAX_AGE`, default 7 days). Because the table is process memory, a gateway
restart logs every browser out; scaling beyond one replica therefore requires a shared session
store (or re-authentication on every request). The sweeper prunes expired entries and sessions
for users whose claim no longer exists.
### Auth abuse controls and proxy limits
Unauthenticated endpoints are throttled before the body is read: `AUTH_RATE_LIMIT` attempts per
client per `AUTH_RATE_WINDOW` (defaults 30/60s; keyed on the rightmost `X-Forwarded-For` entry
when present, else the peer address), answered with `429` + `Retry-After`. When
`SIGNUP_INVITE_CODE` is set, signup additionally requires a constant-time-matched code (form
field or `X-Signup-Invite`) and answers `403` otherwise. Proxied traffic is capped per user
(`MAX_IN_FLIGHT_PER_USER`, default 32 concurrent HTTP requests + WebSockets: `503` for HTTP,
close `1011` for WebSocket) and per WebSocket message received from DSH
(`WS_MAX_MESSAGE_BYTES`, 16 MiB). These are in-process guards for a single replica, not an ingress
rate limiter or quota system.
### Wake-on-connect and idle suspend
```mermaid
@@ -220,6 +250,14 @@ sequenceDiagram
end
```
An upgrade with a missing/expired session, or one that would exceed
`MAX_IN_FLIGHT_PER_USER`, is accepted and then closed: `4401` for authentication, `1011` for
the per-user socket cap. Messages received from DSH above `WS_MAX_MESSAGE_BYTES` (16 MiB) close
the connection instead of being buffered; browser-to-DSH frames are bounded by uvicorn's own
limit. Client-supplied `Forwarded`, `X-Forwarded-User`,
`X-Auth-Request-*` and `x-forwarded-*`/`x-auth-request-*` headers are stripped on the HTTP and
WebSocket paths; `UPSTREAM_HEADER_ALLOWLIST` replaces that denylist with an explicit allowlist.
## Decisions and trade-offs
### ADR-1: Starlette ASGI instead of a sync Flask proxy
@@ -279,14 +317,21 @@ sequenceDiagram
(`kubectl annotate sandboxclaim ...`) to keep being swept. Fine for a reference pattern, not
for a product.
### ADR-6: Gateway cookie session instead of bearer tokens in the browser
### ADR-6: Opaque server-side session cookie instead of bearer tokens in the browser
- **Decision:** browsers get `dsh_gw_session=<user>:<token>`, `HttpOnly`, `SameSite=Lax`
(`Secure` when enabled); API clients may still use `Authorization: Bearer`.
- **Why:** tokens never appear in URLs or JS-readable storage, and `SameSite=Lax` plus a strict
`Origin`/`Referer` check on cookie-authenticated mutations blocks basic CSRF.
- **Cost:** cookie-based auth needs TLS and correct `COOKIE_SECURE`; there is no signed-cookie
layer, so the gateway trusts the token hash comparison for validation.
- **Decision:** browsers get `dsh_gw_session=<sid>`, `HttpOnly`, `SameSite=Lax` (`Secure` when
enabled); the sid maps to `Session(user, expires_at)` in gateway memory. API clients may still
use `Authorization: Bearer`. The cookie name is `__Host-dsh_gw_session` when the Secure flag
is on and no explicit `COOKIE_NAME` is set.
- **Why:** tokens never appear in URLs or JS-readable storage, the cookie value is useless if
leaked outside the gateway session table, sessions can be revoked server-side (logout,
user delete, recreate, expiry), and a cookie cannot be reused as a bearer token for another
user's path. `SameSite=Lax` plus a strict `Origin`/`Referer` check on cookie-authenticated
mutations blocks basic CSRF.
- **Cost:** stateful auth in a single process: a restart logs everyone out and horizontal
scaling needs a shared session store. Cookie-based auth also needs TLS and the correct
`COOKIE_SECURE` setting; the gateway refuses to start with `COOKIE_SECURE=false` unless
`ALLOW_INSECURE_COOKIES=1` is set for a local HTTP demo.
## Security and RBAC
@@ -305,33 +350,52 @@ Notes and limits:
permission is scoped to one namespace, but anyone who can read gateway memory or logs can
impersonate users. Prefer the upstream token API when available.
- **No cluster-wide access, no secret reads.** The gateway does not read `dsh-provider-keys`.
- **Browser-facing hardening:** `HttpOnly`/`SameSite=Lax`/optional `Secure` cookies,
timing-safe SHA-256 token comparison, strict same-origin check on cookie-auth mutations,
request-body limit (`MAX_BODY_MB`), hop-by-hop header stripping, upstream `Set-Cookie`
dropped, DSH `Location` headers rewritten under `/u/<user>/`.
(It does read the optional signup invite code if it is mounted as an env var.)
- **Browser-facing hardening:** opaque server-side sessions instead of token-bearing cookies,
`HttpOnly`/`SameSite=Lax`/optional `Secure` (+ `__Host-` prefix when secure),
timing-safe SHA-256 bearer-token comparison, timing-safe invite-code comparison, strict
same-origin check on cookie-auth mutations, request-body limit (`MAX_BODY_MB`), hop-by-hop
header stripping, client identity/hop headers (`Forwarded`, `X-Forwarded-User`,
`X-Auth-Request-*`, `x-forwarded-*`/`x-auth-request-*`) stripped on HTTP and WebSocket,
upstream `Set-Cookie` dropped, DSH `Location` headers rewritten under `/u/<user>/`.
- **Session lifecycle:** login/signup mint an opaque sid; logout and user delete revoke it;
`/recreate` revokes every session for that user; expiry (`COOKIE_MAX_AGE`, default 7 days) is
enforced on lookup. The table is in memory, so a restart logs browsers out and a second
replica needs shared state.
- **Abuse controls (per replica):** `AUTH_RATE_LIMIT`/`AUTH_RATE_WINDOW` throttle
unauthenticated signup/login per client with `429` + `Retry-After`; `SIGNUP_INVITE_CODE`
gates signup; `MAX_IN_FLIGHT_PER_USER` caps concurrent proxied requests + WebSockets per user;
`WS_MAX_MESSAGE_BYTES` caps messages received from DSH; browser-to-DSH frames are bounded by
uvicorn. These are in-process guards, not an ingress rate limiter or audit log.
- **Host allow-list:** DSH rejects unknown `Host` headers. The gateway forwards the browser's
`Host`, so every public hostname must be in `DSH_TRUSTED_HOSTS`; `localhost`/`127.0.0.1` are
always trusted.
- **TLS is external.** The gateway serves plain HTTP; terminate TLS at the Ingress and set
`COOKIE_SECURE=true`. Without TLS, session cookies are replayable on the network.
`COOKIE_SECURE=true`. The base demo sets `COOKIE_SECURE=false` with
`ALLOW_INSECURE_COOKIES=1` for the port-forward walkthrough; without TLS, session cookies are
replayable on the network and that combination must never be exposed.
- **Tenant isolation** is pod + PVC separation; all tenants share one namespace, one DSH binary,
and the shared provider keys (`dsh-provider-keys` plus the optional
`dsh-provider-keys-extra`). `manifests/optional/network-policy/` adds an opt-in sandbox
NetworkPolicy, and gVisor/`nodeSelector` are opt-in PodSpec fields. Add per-tenant
credentials before calling it multi-tenant-grade.
`dsh-provider-keys-extra`). The base example is demo-only: production requires gVisor for the
sandboxes, the sandbox NetworkPolicy, the gateway egress policy and the exposure NetworkPolicy,
plus per-tenant credentials or an egress proxy. See the
[production checklist](optional-hardening.md#6-production-checklist).
- **DSH's inner file sandbox is disabled by deployment override.** The template sets
`DSH_PERMISSION_MODE=danger-full-access` because the in-container backends are unusable here:
`bubblewrap` ships in the image but is unsetuid and cannot create an unprivileged user namespace
under the pod's seccomp/capability surface, and no Landlock runner is available; the confined
executor would otherwise fail closed on every command. The pod remains the boundary, so an
agent can read its own `DSH_HOME`. Hosts with a working backend can keep a confined mode.
agent can read its own `DSH_HOME`. Hosts with a working backend can keep a confined mode;
multi-tenant deployments must put the pod in gVisor and a NetworkPolicy.
- **Optional hardening is opt-in.** gVisor, node selection, the sandbox/exposure NetworkPolicies,
extra provider keys and failed-Pod recovery are documented in
[optional hardening](optional-hardening.md). The base manifests stay portable (no gVisor
RuntimeClass, no active node labels, no NetworkPolicy, no extra Secret required, and the
controller's default `failedPodPolicy: Retain`).
- **No gateway rate limiting.** Signup can create claims until the cluster or warm pool is
exhausted; put limits at the ingress for real deployments.
- **SSO is trust-on-hop.** `AUTH_PROXY_HEADER` makes the gateway trust the proxy-asserted
identity; the SSO variant must remain the only path to the Service, and its NetworkPolicy is
required, not optional. Run [`scripts/preflight-sso.sh`](../scripts/preflight-sso.sh) after
applying it.
## Failure modes
@@ -340,7 +404,14 @@ Notes and limits:
| DSH log line changed or rotated away | Proxy returns `503` (missing launch token) | Fix the parser or restart the pod so the token line is re-emitted |
| Probe on `/` | Pod `CrashLoopBackOff`, `401` in probe logs | Use `/favicon.svg` for readiness/liveness (shipped templates do) |
| Warm pool exhausted | JSON signup returns `202` (state `Waking`); form signup is slow | Increase `replicas` in the warm pool |
| Gateway restart | Idle sandboxes stop suspending; handshake cache lost | Rebuilt lazily on next request; consider a shared store for production |
| Gateway restart | Idle sandboxes stop suspending; handshake cache and browser sessions lost | Idle clocks and handshake cache rebuild lazily; sessions do not, so browsers must sign in again after a rollout. Use a shared session store for production |
| Gateway configured `COOKIE_SECURE=false` without `ALLOW_INSECURE_COOKIES=1` | Gateway pod exits at startup / `CrashLoopBackOff` | Intentional guard against an insecure cookie on an HTTPS deployment; set `ALLOW_INSECURE_COOKIES=1` only for the local plain-HTTP demo, or set `COOKIE_SECURE=true` |
| Auth rate limit hit | `429` + `Retry-After` on signup/login | Per-client in-memory bucket (`AUTH_RATE_LIMIT`/`AUTH_RATE_WINDOW`); wait for the window or tune it. Behind a proxy, keying uses the rightmost `X-Forwarded-For` entry |
| Signup invite missing/wrong | `403` before the claim is created | `SIGNUP_INVITE_CODE` is set; send the `invite` form field or `X-Signup-Invite` header |
| Per-user concurrency cap hit | HTTP `503` + `Retry-After`, or WebSocket close `1011` | `MAX_IN_FLIGHT_PER_USER` protects the single gateway from one busy tenant; reduce concurrency or raise the cap deliberately. These are per replica |
| Oversized WebSocket message from DSH | Connection closes | `WS_MAX_MESSAGE_BYTES` (16 MiB) is enforced on upstream messages; raise it only if DSH legitimately needs larger frames |
| Session expiry / revocation | `401` (HTTP) or close `4401` (WebSocket) | Expected after logout, user delete, `/recreate`, `COOKIE_MAX_AGE` expiry or a gateway restart; sign in again |
| SSO variant applied without its NetworkPolicy (or with an Ingress) | Gateway accepts spoofed `X-Forwarded-User` | Demo-only configuration; keep `75-networkpolicies.yaml`, remove the second path, and run `scripts/preflight-sso.sh` |
| Browser aborts a request (tab reload, back navigation) during a slow wake or handshake | No response reaches the client; nothing appears in the access log | If the abort lands before the response starts, the body read raises `ClientDisconnect` and the app answers `499` (`CLIENT_CLOSED_REQUEST`), which cannot reach the departed peer. Nothing reached DSH, no ERROR traceback is logged, and the in-flight slot is released. A request already streaming ends with its upstream status. |
| DSH auth cookie expired mid-session | Proxy `401` once, then recovers | Gateway refreshes the handshake once |
| Upstream pod name changed after resume | Cached cookie keyed by old pod UID is useless | Cache is invalidated on suspend; next request re-handshakes |
@@ -372,10 +443,11 @@ single-retry refresh are the supported workaround.
## Evolution path
1. Replace log scraping with the upstream token API (above), deleting the `pods/log` permission.
2. Move idle clocks, in-flight tracking, and handshake cache to a shared store; then scale the
gateway horizontally (sticky routing not required if state is shared).
2. Move idle clocks, in-flight tracking, handshake cache **and sessions** to a shared store; then
scale the gateway horizontally (sticky routing not required if state is shared).
3. Add per-tenant API keys or an egress proxy so a single key is not shared by all sandboxes.
4. Add NetworkPolicies (gateway → sandboxes only), TLS everywhere, rate limits, and audit
logging at the ingress.
4. Add NetworkPolicies (gateway → sandboxes only, plus the documented gateway egress policy),
gVisor for sandbox pods, TLS everywhere, ingress-level rate limits/quota and audit logging.
The gateway's in-process limits are not a substitute.
5. If DSH grows absolute URLs or per-user cookies that cannot be rewritten, switch to
per-user subdomains with wildcard DNS/TLS.

124
examples/dsh-agents-as-a-service/docs/optional-hardening.md
View File

@@ -2,14 +2,26 @@
The base example is deliberately portable: it runs on kind and GKE without a
gVisor runtime, a tainted node pool or a NetworkPolicy-capable CNI, and it
boots with only the required `dsh-provider-keys` Secret. This document covers
four **opt-in** controls plus failed-Pod recovery:
boots with only the required `dsh-provider-keys` Secret. That portability is
exactly why the base manifests are **demo-only**. This document covers the
controls you add for production and the steps to enable or roll them back.
**Production is not the base walkthrough.** At minimum, a production deployment
needs: gVisor for the sandbox pods (or another kernel-isolation boundary), the
sandbox NetworkPolicy, the gateway egress policy, the exposure variant's
NetworkPolicy, TLS with `COOKIE_SECURE=true` (and no
`ALLOW_INSECURE_COOKIES`), SSO (or an equivalent authenticating proxy) as the
only path to the gateway, and per-tenant provider credentials/egress. The
checklist at the end summarizes this; without the network and kernel layers the
example is not tenant-isolated.
This document covers four **opt-in** controls plus failed-Pod recovery:
| Setting | Where it lives | What it changes |
| --- | --- | --- |
| gVisor | commented `runtimeClassName` in the SandboxTemplates and Deployments | puts DSH and/or the gateway in a userspace kernel |
| gVisor | commented `runtimeClassName` in the SandboxTemplates and Deployments | puts DSH and/or the gateway in a userspace kernel; required for untrusted agents in production |
| nodeSelector | commented `nodeSelector` in the same files | pins pods to a chosen node pool |
| NetworkPolicy | `manifests/optional/network-policy/`, `manifests/exposure/*/` | limits sandbox, gateway and proxy connectivity |
| NetworkPolicy | `manifests/optional/network-policy/`, `manifests/exposure/*/`, plus the gateway egress recipe | limits sandbox, gateway and proxy connectivity; the sandbox and gateway-egress policies are required for production |
| Provider keys | optional `dsh-provider-keys-extra` Secret | injects extra provider API keys into every sandbox |
| Failed-Pod recovery | commented `failedPodPolicy` in the SandboxTemplates | replaces a Failed sandbox Pod instead of parking the sandbox |
@@ -234,10 +246,17 @@ kubectl -n dsh-demo patch sandboxtemplate dsh-web --type merge \
The base example does **not** manage NetworkPolicy: the SandboxTemplate is
`networkPolicyManagement: Unmanaged`, and the gateway has no policy. That keeps
the walkthrough portable and independent of the sandbox-router; the opt-in
policies below are standard Kubernetes `NetworkPolicy` objects.
the walkthrough portable and independent of the sandbox-router. **For
production this is not optional**: without the sandbox policy every tenant can
reach every other tenant and cluster-internal endpoints, and without a gateway
egress policy the gateway can talk to anything the cluster network allows.
Apply both. The policies below are standard Kubernetes `NetworkPolicy` objects.
### Optional sandbox policy
> **Demo vs production.** An installation that skips the sandbox NetworkPolicy
> and the gateway egress policy is a demo. Exposure (Ingress/Gateway API) adds
> ingress protection only; it does not isolate tenants from each other.
### Sandbox policy (required for production)
[`manifests/optional/network-policy/10-sandbox-networkpolicy.yaml`](../manifests/optional/network-policy/10-sandbox-networkpolicy.yaml)
selects every Sandbox pod through the agent-sandbox tracking label
@@ -301,10 +320,10 @@ Caveats:
- `kubectl port-forward` still works for debugging (the connection is local to
the pod), except under gVisor.
### Exposure-variant policies
### Exposure-variant policies (required, part of the variant)
Each exposure variant ships its own policy. They are part of the variant, not a
separate opt-in step:
Each exposure variant ships its own policy. They are part of the variant and
**required for production**; an exposure without its policy is demo-only:
- [`manifests/exposure/ingress/61-networkpolicy.yaml`](../manifests/exposure/ingress/61-networkpolicy.yaml)
allows only Traefik pods (`app.kubernetes.io/name: traefik`) to reach the
@@ -317,7 +336,10 @@ separate opt-in step:
The SSO policy is what turns the trust boundary in the
[exposure README](../manifests/exposure/README.md) into network enforcement.
The `dsh-gateway` Service must still stay ClusterIP; do not apply both exposure
variants.
variants. After applying the SSO variant, run
[`scripts/preflight-sso.sh`](../scripts/preflight-sso.sh): it checks the
gateway's SSO env, the Secrets, the TLS reference, leftover placeholders and
accidental second paths (such as an Ingress).
Verify the negative case from a sandbox pod (it is not the proxy):
@@ -332,11 +354,12 @@ different pod label, replace the `podSelector` with the correct selector or an
`ipBlock`. As with the sandbox policy, kubelet probes can be affected on some
CNIs.
### Gateway egress (documented, not shipped)
### Gateway egress (required for production)
The gateway needs DNS, the Kubernetes API, and TCP `3080` to sandbox pods. The
API server endpoint is cluster-specific (Service CIDR or node IP/port), so an
egress policy is **not** shipped. If your cluster has a stable API-server CIDR
The gateway needs DNS, the Kubernetes API, and TCP `3080` to sandbox pods. An
egress policy is **required for production** but is not shipped, because the API
server endpoint is cluster-specific (Service CIDR or node IP/port) and a wrong
value silently breaks signup/proxy. If your cluster has a stable API-server CIDR
and port, add an egress policy like this (replace the placeholders; verify the
port, commonly `443` or `6443`):
@@ -597,7 +620,63 @@ done
Warm-pool spares pick the default up as they are claimed and replenished, or
delete them as in step 2 above.
## 6. Verification summary
## 6. Production checklist
The base walkthrough is a demo. A production install must satisfy every item
below; the sections above give the commands and caveats.
**Isolation and network**
- [ ] Sandbox pods run under gVisor (or an equivalent kernel-isolation
boundary) because DSH runs with `DSH_PERMISSION_MODE=danger-full-access`
in this image; the pod is the boundary. (Section 1.)
- [ ] The sandbox NetworkPolicy is applied and verified: no tenant→tenant,
no tenant→cluster-Service/API, DNS + public HTTPS only. (Section 3.)
- [ ] The gateway egress policy is applied with the cluster's real API-server
CIDR/port, scoped to DNS, the API server and sandbox pods on `3080`.
(Section 3, "Gateway egress".)
- [ ] The exposure variant's NetworkPolicy is applied. An exposure without it is
demo-only. For SSO, the gateway is reachable **only** through
oauth2-proxy; there is no second Ingress/Gateway/port-forward path.
- [ ] Nodes/CNI enforce NetworkPolicy (verify the negative cases, not just the
positive path).
- [ ] The gateway and (optionally) oauth2-proxy run with `runAsNonRoot`,
dropped capabilities, `RuntimeDefault` seccomp and a read-only root
filesystem, as shipped.
**Authentication, TLS and abuse controls**
- [ ] TLS terminates in front of the gateway, `COOKIE_SECURE=true`, and
`ALLOW_INSECURE_COOKIES` is **not** set. The gateway refuses to start with
an insecure cookie unless explicitly acknowledged.
- [ ] SSO (or an equivalent authenticating proxy) is enabled with
`AUTH_PROXY_HEADER=X-Forwarded-User`, and
[`scripts/preflight-sso.sh`](../scripts/preflight-sso.sh) exits `0`.
- [ ] No `your-github-org` / `dsh-demo.example.com` placeholder remains.
- [ ] `SIGNUP_INVITE_CODE` is set from a Secret if the gateway's own signup form
is reachable, and `AUTH_RATE_LIMIT` is set deliberately (`30`/60s shipped).
Add ingress-level rate limiting/quota too; the gateway's buckets are
in-process and reset on restart.
- [ ] Session lifetime (`COOKIE_MAX_AGE`) and the logout/delete revocation
behavior are acceptable; account for gateway restarts logging users out.
- [ ] `MAX_IN_FLIGHT_PER_USER` and `WS_MAX_MESSAGE_BYTES` fit the workload.
**Credentials, platform and operations**
- [ ] Provider keys are per-tenant (or routed through an egress proxy) instead
of the shared `dsh-provider-keys` Secrets; any shared key is treated as
visible to every tenant.
- [ ] RBAC stays namespace-scoped; `pods/log` is treated as sensitive because
it contains launch tokens and model output. Plan the upstream token API
migration.
- [ ] Images are digest-pinned and dependency installs are hash-locked.
- [ ] Gateway replicas stay at `1` (in-memory sessions/idle state) or the state
is externalized.
- [ ] `failedPodPolicy: Recreate` is set if node eviction/spot reclaim is
expected.
- [ ] Audit logging, alerting and backups of the PVCs are in place.
## 7. Verification summary
Required locally for any change to this example:
@@ -605,6 +684,7 @@ Required locally for any change to this example:
make check # versions, example layout, manifest YAML shape
yamllint .
make test # only when the gateway code changed
shellcheck examples/dsh-agents-as-a-service/scripts/*.sh
```
CI additionally parses every manifest and runs `kubectl apply
@@ -618,17 +698,25 @@ Manual checks, on a cluster with the relevant CNI/runtime:
gateway proxy works, port-forward limitation acknowledged.
- nodeSelector: pods land on the intended nodes; no `Pending` events.
- Sandbox policy: the allow/deny matrix above.
- Gateway egress policy: signup and proxy still work; the gateway cannot reach
an arbitrary cluster Service or the metadata server.
- Exposure policies: browser path works; a non-proxy pod cannot reach the
gateway; probes stay healthy.
gateway; probes stay healthy. `scripts/preflight-sso.sh` exits `0`.
- Cookie/TLS guard: with `COOKIE_SECURE=false` and no
`ALLOW_INSECURE_COOKIES` the gateway refuses to start; with TLS it starts and
the cookie is `Secure` (`__Host-` prefixed).
- Auth abuse controls: `429` + `Retry-After` after the configured limit, `403`
on a wrong invite code, and revocations on logout/delete/recreate.
- Provider keys: set/unset output (never the values) and the refresh procedure.
- Failed-Pod recovery: `spec.failedPodPolicy` on the Sandbox, and the drill
above replaces a `Failed` Pod while keeping the claim and PVC.
## 7. Rollback summary
## 8. Rollback summary
| Layer | Rollback |
| --- | --- |
| Sandbox policy | `kubectl delete -f manifests/optional/network-policy/` |
| Gateway egress policy | `kubectl delete networkpolicy dsh-gateway-egress -n dsh-demo` (do not leave it half-configured) |
| Exposure policy | delete the exposure variant directory |
| gVisor | patch `runtimeClassName: null`, recreate pods |
| nodeSelector | patch `nodeSelector: null`, recreate pods |