fix(gateway)!: replace token cookies with revocable sessions and harden abuse controls (#40)

- Replace user:token cookies with opaque server-side sessions, revocable on logout, delete, recreate and expiry; bind every credential to the path user and add cross-user regression tests. - Rate-limit /users and /login, add optional signup invite codes and form same-origin checks, and unify login failures to 401. - Require explicit ALLOW_INSECURE_COOKIES for plain HTTP, use __Host-dsh_gw_session when secure, and fix secure cookie deletion. - Strip client-supplied forwarding/SSO headers, add an optional upstream allowlist, per-user in-flight and WebSocket message caps, credential-free handshake errors, and sweep-time state pruning. - Digest-pin base images, hash-pin gateway dependencies, verify DSH_COMMIT in the build, and run the gateway image as non-root with a pinned Python base. - Update manifests, production checklist, architecture/security docs and add an SSO preflight script. BREAKING CHANGE: existing dsh_gw_session=<user>:<token> cookies are invalid; users must sign in again. Unknown users on POST /login now return 401 instead of 404, and the gateway refuses COOKIE_SECURE=false unless ALLOW_INSECURE_COOKIES=1. Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
2026-11-03 01:13:24 +00:00 · 2026-10-31 00:57:35 -03:00
parent 2f959e6afb
commit 1a929e3e24
33 changed files with 5450 additions and 319 deletions
--- a/examples/dsh-agents-as-a-service/manifests/10-sandboxtemplate.yaml
+++ b/examples/dsh-agents-as-a-service/manifests/10-sandboxtemplate.yaml
@@ -131,6 +131,24 @@ spec:
            # a private PVC), so trade DSH's inner file policy for a working
            # agent. Set this to workspace-write (or read-only) on hosts where a
            # backend is available; see the README troubleshooting row.
+            # SECURITY: the pod is the isolation boundary in this example.
+            # danger-full-access is set only because DSH's own command sandbox
+            # has no usable backend in this pod (see above), which means an
+            # agent command can read its own DSH_HOME (including
+            # .credentials.yaml) and write anywhere in the container. What
+            # still contains it is the pod itself: one pod per user, non-root,
+            # all capabilities dropped, RuntimeDefault seccomp, its own PVC and
+            # Service. That is NOT a hardened multi-tenant boundary on a shared
+            # kernel. For production/multi-tenant use, additionally run the
+            # sandbox under gVisor (`runtimeClassName: gvisor` above) so a
+            # container escape is trapped in a userspace kernel, and apply the
+            # sandbox NetworkPolicy (manifests/optional/network-policy/) so
+            # tenants cannot reach each other or cluster-internal endpoints.
+            # The base manifests without those two controls are a demo. Treat
+            # danger-full-access as a fallback, not a target: on hosts where
+            # DSH's command-sandbox backend works, prefer workspace-write (or
+            # read-only), and recreate the pod because the value is stamped at
+            # creation.
            - name: DSH_PERMISSION_MODE
              value: danger-full-access
          # Probes target /favicon.svg: DSH gates / behind its launch-token
--- a/examples/dsh-agents-as-a-service/manifests/50-gateway.yaml
+++ b/examples/dsh-agents-as-a-service/manifests/50-gateway.yaml
@@ -134,14 +134,30 @@ spec:
            # the DSH launch token.
            - name: DSH_CONTAINER
              value: dsh-web
-            # Cookie dsh_gw_session is HttpOnly + SameSite=Lax. "false" keeps
-            # the plain-HTTP demo usable through port-forward: a Secure cookie
-            # is never sent over HTTP, so login would silently fail.
-            # HTTPS deployments (the ingress/ variant with TLS, or
-            # gateway-api-sso/, which terminates TLS at the Gateway) MUST set
-            # this to "true".
+            # Browser session cookie: HttpOnly + SameSite=Lax, holds only an
+            # opaque, server-side session id (sid) - never the user's DSH
+            # token. Sessions can be revoked server-side: POST /logout, user
+            # delete, POST /recreate and expiry (COOKIE_MAX_AGE, default
+            # 7 days) all invalidate them. The session table is in process
+            # memory, so restarting this Deployment logs every browser out.
+            # With COOKIE_SECURE=true the cookie is named
+            # __Host-dsh_gw_session; COOKIE_NAME overrides the name.
+            #
+            # "false" keeps the plain-HTTP demo usable through port-forward: a
+            # Secure cookie is never sent over HTTP, so login would silently
+            # fail. The gateway REFUSES TO START with COOKIE_SECURE=false
+            # unless ALLOW_INSECURE_COOKIES=1 acknowledges that the session
+            # travels in clear text, so the base demo sets both. HTTPS
+            # deployments (the ingress/ variant with TLS, or gateway-api-sso/,
+            # which terminates TLS at the Gateway) MUST set this to "true".
            - name: COOKIE_SECURE
              value: "false"
+            # Explicit acknowledgement that this base demo serves plain HTTP.
+            # Required when COOKIE_SECURE=false (the gateway exits at startup
+            # otherwise); ignored once COOKIE_SECURE=true, so HTTPS installs
+            # should drop it. Never set this on an internet-facing deployment.
+            - name: ALLOW_INSECURE_COOKIES
+              value: "1"
            # SSO IS OPT-IN AND DISABLED HERE. With AUTH_PROXY_HEADER unset the
            # gateway trusts only its own bearer/session auth and ignores every
            # identity header. Enable it ONLY behind
@@ -152,9 +168,49 @@ spec:
            # reach :8080 can impersonate any user by sending that header.
            # - name: AUTH_PROXY_HEADER
            #   value: X-Forwarded-User
+            # Brute-force / abuse throttle for the unauthenticated auth
+            # endpoints (/users and /login): at most N attempts per client per
+            # AUTH_RATE_WINDOW seconds (default 60). A client is the rightmost
+            # X-Forwarded-For entry (the one added by the trusted proxy) when
+            # present, otherwise the TCP peer. Over the limit the gateway
+            # answers 429 + Retry-After BEFORE reading the body or touching the
+            # cluster; 0 disables the limiter. This is a per-process bucket, so
+            # it is not a substitute for an ingress rate limit.
+            - name: AUTH_RATE_LIMIT
+              value: "30"
+            # Window in seconds for AUTH_RATE_LIMIT (default 60).
+            # - name: AUTH_RATE_WINDOW
+            #   value: "60"
+            # Optional signup gate. When set, POST /users requires the invite
+            # code - form field "invite" or header X-Signup-Invite - compared in
+            # constant time. A missing/wrong code is rejected with 403 before
+            # the user is looked up, so it also stops unauthenticated claim
+            # creation. The landing page shows the invite input only when this
+            # is set. Keep the value in a Secret, create it before rollout and
+            # restart the gateway to pick up a rotation:
+            #   kubectl -n dsh-demo create secret generic dsh-signup-invite \
+            #     --from-literal=code="$(openssl rand -hex 16)"
+            # - name: SIGNUP_INVITE_CODE
+            #   valueFrom:
+            #     secretKeyRef:
+            #       name: dsh-signup-invite
+            #       key: code
            # Reject proxied request bodies larger than this (413).
            - name: MAX_BODY_MB
              value: "32"
+            # Per-user concurrency guard (default 32): at most this many
+            # proxied HTTP requests and WebSocket connections per user.
+            # Excess HTTP requests get 503 + Retry-After and excess WebSocket
+            # upgrades close with 1011, so one tenant cannot exhaust the
+            # single-replica gateway for everyone else.
+            # - name: MAX_IN_FLIGHT_PER_USER
+            #   value: "32"
+            # Largest WebSocket message received from DSH, in bytes (default
+            # 16 MiB); oversized frames close the connection. Browser-to-DSH
+            # frames are bounded by uvicorn's own limit. HTTP/SSE responses are
+            # streamed without this cap.
+            # - name: WS_MAX_MESSAGE_BYTES
+            #   value: "16777216"
          # /healthz is unauthenticated and returns 200 without touching the
          # cluster, so it is safe for both readiness and liveness.
          readinessProbe:
--- a/examples/dsh-agents-as-a-service/manifests/exposure/README.md
+++ b/examples/dsh-agents-as-a-service/manifests/exposure/README.md
@@ -16,6 +16,16 @@ Applying both gives the gateway two entry points, and the SSO variant is only
 safe when the gateway is reachable **exclusively** through oauth2-proxy. Do not
 mix them.

+> **Production requirement.** Each variant's NetworkPolicy is part of the
+> variant, and applying the variant without it is demo-only. The sandbox
+> NetworkPolicy and the gateway egress policy (the API server is cluster-specific)
+> are separate and equally required; see the
+> [production checklist](../../docs/optional-hardening.md#6-production-checklist)
+> and the
+> [gateway egress recipe](../../docs/optional-hardening.md#gateway-egress-required-for-production).
+> For SSO, run [`scripts/preflight-sso.sh`](../../scripts/preflight-sso.sh)
+> after applying the variant and before sending real traffic.
+
 Each variant also ships a NetworkPolicy, applied by the same directory apply:

 - `ingress/61-networkpolicy.yaml` allows only Traefik pods to reach the gateway
@@ -27,7 +37,8 @@ Each variant also ships a NetworkPolicy, applied by the same directory apply:
 The policies assume Traefik's `app.kubernetes.io/name: traefik` pod label.
 Adjust the selector for another implementation or a `hostNetwork` install; see
 [optional hardening](../../docs/optional-hardening.md) for the probe caveat and
-verification commands.
+verification commands. Keep the policies: an exposure without its NetworkPolicy
+is demo-only.

 ## Applying a variant

@@ -69,10 +80,13 @@ Secrets imperatively (commands below) before applying the directory.
 2. Set `DSH_TRUSTED_HOSTS` in the SandboxTemplate to the same public host:
   the gateway forwards the browser's `Host`, and DSH rejects unknown hosts on
   `/api` (see `../10-sandboxtemplate.yaml`).
-3. Set `COOKIE_SECURE=true` on the gateway once the Ingress serves HTTPS:
-   `kubectl -n dsh-demo set env deploy/dsh-gateway COOKIE_SECURE=true`. The
-   base manifest keeps `false` so the plain-HTTP port-forward demo works; a
-   `Secure` cookie is never sent over HTTP, which silently breaks login.
+3. Set `COOKIE_SECURE=true` and remove the plain-HTTP acknowledgement once the
+   Ingress serves HTTPS:
+   `kubectl -n dsh-demo set env deploy/dsh-gateway COOKIE_SECURE=true ALLOW_INSECURE_COOKIES-`.
+   The base manifest keeps `false` + `ALLOW_INSECURE_COOKIES=1` so the
+   plain-HTTP port-forward demo works; a `Secure` cookie is never sent over HTTP,
+   which silently breaks login, while the acknowledgement must not survive into
+   production.
 4. Do not attach buffering or compression middlewares to this router: they
   break WebSocket upgrades and streaming.
 5. `61-networkpolicy.yaml` is part of this variant: it allows only Traefik pods
@@ -116,14 +130,23 @@ kubectl -n dsh-demo create secret tls dsh-demo-tls --cert=tls.crt --key=tls.key

 # 3. Enable SSO on the gateway and trust the public host in the template.
 kubectl -n dsh-demo set env deploy/dsh-gateway \
-  AUTH_PROXY_HEADER=X-Forwarded-User COOKIE_SECURE=true
+  AUTH_PROXY_HEADER=X-Forwarded-User COOKIE_SECURE=true ALLOW_INSECURE_COOKIES-
 # then set DSH_TRUSTED_HOSTS=dsh-demo.example.com in the SandboxTemplate
 # (see ../10-sandboxtemplate.yaml) and re-apply it.

 # 4. Apply the variant.
 kubectl apply -f manifests/exposure/gateway-api-sso/
+
+# 5. Preflight: fails loudly on missing Secrets, a bad oauth2-proxy cookie
+#    secret, missing gateway SSO env, a dangling TLS reference, leftover
+#    placeholders, or a second path to the gateway.
+./scripts/preflight-sso.sh
 ```

+The preflight is read-only and exits non-zero on any failure, naming the exact
+fix. Run it again after rotating a Secret or changing the public host; it also
+verifies that the oauth2-proxy cookie secret decodes to 16, 24 or 32 bytes.
+
 ### SSO trust boundary

 `AUTH_PROXY_HEADER` is opt-in and deliberately unset in the base gateway
@@ -141,8 +164,14 @@ takes all of these:
   impersonate any user.
 3. Keep `75-networkpolicies.yaml` with the variant. It enforces the boundary in
   the network layer (gateway ← oauth2-proxy ← Traefik, and limited oauth2-proxy
-   egress). A custom exposure without an equivalent policy has no network
+   egress). This is required, not optional: an exposure without the policy is
+   demo-only, and a custom exposure without an equivalent policy has no network
   enforcement.
+4. Apply the gateway egress policy too: the gateway talks to the Kubernetes API
+   and to sandbox pods, and `75-networkpolicies.yaml` intentionally leaves that
+   direction unrestricted because the API-server endpoint is cluster-specific.
+   See
+   [gateway egress](../../docs/optional-hardening.md#gateway-egress-required-for-production).

 Also terminate TLS in front of the gateway (the Gateway listener does) and
 leave `COOKIE_SECURE=true` so the session cookie never travels in clear text.
--- a/examples/dsh-agents-as-a-service/manifests/exposure/gateway-api-sso/72-oauth2-proxy.yaml
+++ b/examples/dsh-agents-as-a-service/manifests/exposure/gateway-api-sso/72-oauth2-proxy.yaml
@@ -25,8 +25,9 @@
 # TLS terminates at the Gateway listener (70-gateway.yaml); keep
 # --cookie-secure=true and COOKIE_SECURE=true on the gateway.
 #
-# The image is pinned to quay.io/oauth2-proxy/oauth2-proxy:v7.15.5
-# (versions.env: OAUTH2_PROXY_IMAGE). Credentials come from the
+# The image is digest-pinned: quay.io/oauth2-proxy/oauth2-proxy:v7.15.5@sha256:...
+# (versions.env: OAUTH2_PROXY_IMAGE). The digest is the immutable multi-arch
+# index, so the tag can never be moved under us. Credentials come from the
 # "oauth2-proxy" Secret (see 73-oauth2-proxy-secret.yaml.example); this
 # manifest never carries secrets.
 apiVersion: apps/v1
@@ -61,7 +62,7 @@ spec:
          type: RuntimeDefault
      containers:
        - name: oauth2-proxy
-          image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.5
+          image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.5@sha256:8498b0d0ef0a7b29686414000a08aee467f02d0299c9ed1e006a8f33fc017916
          imagePullPolicy: IfNotPresent
          securityContext:
            allowPrivilegeEscalation: false
--- a/examples/dsh-agents-as-a-service/manifests/exposure/gateway-api-sso/75-networkpolicies.yaml
+++ b/examples/dsh-agents-as-a-service/manifests/exposure/gateway-api-sso/75-networkpolicies.yaml
@@ -1,5 +1,5 @@
-# OPTIONAL with the SSO exposure variant: enforce the trust boundary at the
-# network layer.
+# REQUIRED with the SSO exposure variant: enforce the trust boundary at the
+# network layer. Applying the SSO variant without this file is demo-only.
 #
 # The chain is Traefik (Gateway API) -> oauth2-proxy -> dsh-gateway. These
 # policies make it the ONLY chain:
@@ -8,8 +8,9 @@
 #   - oauth2-proxy egress is limited to DNS, public HTTPS (GitHub) and the
 #     gateway.
 # The gateway's own egress (Kubernetes API, sandbox pods) is intentionally not
-# restricted here: the API-server endpoint is cluster-specific. See
-# ../../../docs/optional-hardening.md for that recipe.
+# restricted here: the API-server endpoint is cluster-specific. Add it for
+# production. See
+# ../../../docs/optional-hardening.md#gateway-egress-required-for-production.
 #
 # This manifest assumes Traefik, matching gatewayClassName: traefik in
 # 70-gateway.yaml. Adjust the app.kubernetes.io/name selector for another
--- a/examples/dsh-agents-as-a-service/manifests/exposure/ingress/60-ingress.yaml
+++ b/examples/dsh-agents-as-a-service/manifests/exposure/ingress/60-ingress.yaml
@@ -14,8 +14,13 @@
 #      DSH_TRUSTED_HOSTS to this host so DSH accepts the forwarded Host/Origin
 #      on /api.
 #   2. Gateway Deployment (manifests/50-gateway.yaml): set COOKIE_SECURE to
-#      "true" once this Ingress serves HTTPS, so the session cookie is Secure.
-#      The base manifest sets "false" for the plain-HTTP port-forward demo.
+#      "true" and remove ALLOW_INSECURE_COOKIES once this Ingress serves HTTPS,
+#      so the session cookie is Secure:
+#        kubectl -n dsh-demo set env deploy/dsh-gateway \
+#          COOKIE_SECURE=true ALLOW_INSECURE_COOKIES-
+#      The base manifest sets "false" + ALLOW_INSECURE_COOKIES=1 for the
+#      plain-HTTP port-forward demo; without the acknowledgement the gateway
+#      refuses to start.
 #
 # WebSocket support is REQUIRED: the browser talks to /u/<user>/... and DSH
 # uses WebSockets for live session traffic. Traefik upgrades WS/SSE
@@ -24,9 +29,11 @@
 # and upgrades. `path: /` Prefix covers the landing page and every per-user
 # proxy path, so no per-sandbox Ingress is needed.
 #
-# 61-networkpolicy.yaml is part of this variant: it restricts the gateway's
-# ingress to Traefik pods. Adjust its selector if your ingress controller uses
-# a different label or hostNetwork.
+# 61-networkpolicy.yaml is part of this variant and required for production:
+# it restricts the gateway's ingress to Traefik pods. Adjust its selector if
+# your ingress controller uses a different label or hostNetwork. The sandbox
+# and gateway-egress NetworkPolicies are separate requirements; see
+# ../../../docs/optional-hardening.md#6-production-checklist.
 apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:
--- a/examples/dsh-agents-as-a-service/manifests/exposure/ingress/61-networkpolicy.yaml
+++ b/examples/dsh-agents-as-a-service/manifests/exposure/ingress/61-networkpolicy.yaml
@@ -1,11 +1,16 @@
-# OPTIONAL with the Ingress exposure variant: restrict the gateway's ingress
-# to the ingress controller.
+# REQUIRED with the Ingress exposure variant for production: restrict the
+# gateway's ingress to the ingress controller. Applying the variant without
+# this policy is demo-only.
 #
 # This manifest assumes Traefik, matching this variant's
 # `ingressClassName: traefik`. Traefik's Helm chart labels its pods
 # app.kubernetes.io/name=traefik; adjust the selector for another controller or
 # for a hostNetwork install (where the source is the node IP, not a pod IP).
 #
+# The sandbox NetworkPolicy and the gateway egress policy are separate and also
+# required for production; see
+# ../../../docs/optional-hardening.md#6-production-checklist.
+#
 # NetworkPolicy enforcement of kubelet probe traffic is CNI-dependent. If the
 # gateway starts failing its /healthz probes, allow the node CIDR on port 8080
 # or remove this policy. See ../../../docs/optional-hardening.md.
--- a/examples/dsh-agents-as-a-service/manifests/optional/README.md
+++ b/examples/dsh-agents-as-a-service/manifests/optional/README.md
@@ -23,6 +23,12 @@ It requires a CNI that enforces `NetworkPolicy`. The selector uses the
 agent-sandbox controller's pod tracking label, so the policy applies to
 existing warm-pool and claimed sandboxes without recreation.

+**This is opt-in for the demo but required for production.** Without it every
+tenant can reach every other tenant and cluster-internal endpoints. Production
+also needs the gateway egress policy (cluster-specific API-server endpoint) and
+the exposure variant's policy; see the
+[production checklist](../../docs/optional-hardening.md#6-production-checklist).
+
 ## gVisor, nodeSelector and provider keys

 These are configured in the main manifests or with an extra Secret:
--- a/examples/dsh-agents-as-a-service/manifests/optional/network-policy/10-sandbox-networkpolicy.yaml
+++ b/examples/dsh-agents-as-a-service/manifests/optional/network-policy/10-sandbox-networkpolicy.yaml
@@ -1,9 +1,15 @@
-# OPTIONAL, opt-in tenant network policy for the DSH sandboxes.
+# OPTIONAL for the base walkthrough, REQUIRED for production. Opt-in tenant
+# network policy for the DSH sandboxes: without it every tenant can reach every
+# other tenant and cluster-internal endpoints.
 #
 # It is NOT part of the top-level walkthrough. Apply and delete it explicitly:
 #   kubectl apply  -f manifests/optional/network-policy/
 #   kubectl delete -f manifests/optional/network-policy/
 #
+# Production also needs the gateway egress policy (cluster-specific API-server
+# endpoint) and the exposure variant's policy, plus gVisor
+# (docs/optional-hardening.md#6-production-checklist).
+#
 # The selector uses the agent-sandbox controller's pod tracking label, which is
 # present on every Sandbox pod (including warm-pool spares), so the policy
 # applies to existing sandboxes without recreating them. The SandboxTemplate