mirror of
https://github.com/sredevopsorg/deepseek-kubernetes-sandbox.git
synced 2026-11-03 09:23:25 +00:00
fix(gateway)!: replace token cookies with revocable sessions and harden abuse controls
- Replace user:token cookies with opaque server-side sessions, revocable on logout, delete, recreate and expiry; bind every credential to the path user and add cross-user regression tests. - Rate-limit /users and /login, add optional signup invite codes and form same-origin checks, and unify login failures to 401. - Require explicit ALLOW_INSECURE_COOKIES for plain HTTP, use __Host-dsh_gw_session when secure, and fix secure cookie deletion. - Strip client-supplied forwarding/SSO headers, add an optional upstream allowlist, per-user in-flight and WebSocket message caps, credential-free handshake errors, and sweep-time state pruning. - Digest-pin base images, hash-pin gateway dependencies, verify DSH_COMMIT in the build, and run the gateway image as non-root with a pinned Python base. - Update manifests, production checklist, architecture/security docs and add an SSO preflight script. BREAKING CHANGE: existing dsh_gw_session=<user>:<token> cookies are invalid; users must sign in again. Unknown users on POST /login now return 401 instead of 404, and the gateway refuses COOKIE_SECURE=false unless ALLOW_INSECURE_COOKIES=1.
This commit is contained in:
SREDevOpsAgents
@@ -131,6 +131,24 @@ spec:
|
||||
# a private PVC), so trade DSH's inner file policy for a working
|
||||
# agent. Set this to workspace-write (or read-only) on hosts where a
|
||||
# backend is available; see the README troubleshooting row.
|
||||
# SECURITY: the pod is the isolation boundary in this example.
|
||||
# danger-full-access is set only because DSH's own command sandbox
|
||||
# has no usable backend in this pod (see above), which means an
|
||||
# agent command can read its own DSH_HOME (including
|
||||
# .credentials.yaml) and write anywhere in the container. What
|
||||
# still contains it is the pod itself: one pod per user, non-root,
|
||||
# all capabilities dropped, RuntimeDefault seccomp, its own PVC and
|
||||
# Service. That is NOT a hardened multi-tenant boundary on a shared
|
||||
# kernel. For production/multi-tenant use, additionally run the
|
||||
# sandbox under gVisor (`runtimeClassName: gvisor` above) so a
|
||||
# container escape is trapped in a userspace kernel, and apply the
|
||||
# sandbox NetworkPolicy (manifests/optional/network-policy/) so
|
||||
# tenants cannot reach each other or cluster-internal endpoints.
|
||||
# The base manifests without those two controls are a demo. Treat
|
||||
# danger-full-access as a fallback, not a target: on hosts where
|
||||
# DSH's command-sandbox backend works, prefer workspace-write (or
|
||||
# read-only), and recreate the pod because the value is stamped at
|
||||
# creation.
|
||||
- name: DSH_PERMISSION_MODE
|
||||
value: danger-full-access
|
||||
# Probes target /favicon.svg: DSH gates / behind its launch-token
|
||||
|
||||
@@ -134,14 +134,30 @@ spec:
|
||||
# the DSH launch token.
|
||||
- name: DSH_CONTAINER
|
||||
value: dsh-web
|
||||
# Cookie dsh_gw_session is HttpOnly + SameSite=Lax. "false" keeps
|
||||
# the plain-HTTP demo usable through port-forward: a Secure cookie
|
||||
# is never sent over HTTP, so login would silently fail.
|
||||
# HTTPS deployments (the ingress/ variant with TLS, or
|
||||
# gateway-api-sso/, which terminates TLS at the Gateway) MUST set
|
||||
# this to "true".
|
||||
# Browser session cookie: HttpOnly + SameSite=Lax, holds only an
|
||||
# opaque, server-side session id (sid) - never the user's DSH
|
||||
# token. Sessions can be revoked server-side: POST /logout, user
|
||||
# delete, POST /recreate and expiry (COOKIE_MAX_AGE, default
|
||||
# 7 days) all invalidate them. The session table is in process
|
||||
# memory, so restarting this Deployment logs every browser out.
|
||||
# With COOKIE_SECURE=true the cookie is named
|
||||
# __Host-dsh_gw_session; COOKIE_NAME overrides the name.
|
||||
#
|
||||
# "false" keeps the plain-HTTP demo usable through port-forward: a
|
||||
# Secure cookie is never sent over HTTP, so login would silently
|
||||
# fail. The gateway REFUSES TO START with COOKIE_SECURE=false
|
||||
# unless ALLOW_INSECURE_COOKIES=1 acknowledges that the session
|
||||
# travels in clear text, so the base demo sets both. HTTPS
|
||||
# deployments (the ingress/ variant with TLS, or gateway-api-sso/,
|
||||
# which terminates TLS at the Gateway) MUST set this to "true".
|
||||
- name: COOKIE_SECURE
|
||||
value: "false"
|
||||
# Explicit acknowledgement that this base demo serves plain HTTP.
|
||||
# Required when COOKIE_SECURE=false (the gateway exits at startup
|
||||
# otherwise); ignored once COOKIE_SECURE=true, so HTTPS installs
|
||||
# should drop it. Never set this on an internet-facing deployment.
|
||||
- name: ALLOW_INSECURE_COOKIES
|
||||
value: "1"
|
||||
# SSO IS OPT-IN AND DISABLED HERE. With AUTH_PROXY_HEADER unset the
|
||||
# gateway trusts only its own bearer/session auth and ignores every
|
||||
# identity header. Enable it ONLY behind
|
||||
@@ -152,9 +168,49 @@ spec:
|
||||
# reach :8080 can impersonate any user by sending that header.
|
||||
# - name: AUTH_PROXY_HEADER
|
||||
# value: X-Forwarded-User
|
||||
# Brute-force / abuse throttle for the unauthenticated auth
|
||||
# endpoints (/users and /login): at most N attempts per client per
|
||||
# AUTH_RATE_WINDOW seconds (default 60). A client is the rightmost
|
||||
# X-Forwarded-For entry (the one added by the trusted proxy) when
|
||||
# present, otherwise the TCP peer. Over the limit the gateway
|
||||
# answers 429 + Retry-After BEFORE reading the body or touching the
|
||||
# cluster; 0 disables the limiter. This is a per-process bucket, so
|
||||
# it is not a substitute for an ingress rate limit.
|
||||
- name: AUTH_RATE_LIMIT
|
||||
value: "30"
|
||||
# Window in seconds for AUTH_RATE_LIMIT (default 60).
|
||||
# - name: AUTH_RATE_WINDOW
|
||||
# value: "60"
|
||||
# Optional signup gate. When set, POST /users requires the invite
|
||||
# code - form field "invite" or header X-Signup-Invite - compared in
|
||||
# constant time. A missing/wrong code is rejected with 403 before
|
||||
# the user is looked up, so it also stops unauthenticated claim
|
||||
# creation. The landing page shows the invite input only when this
|
||||
# is set. Keep the value in a Secret, create it before rollout and
|
||||
# restart the gateway to pick up a rotation:
|
||||
# kubectl -n dsh-demo create secret generic dsh-signup-invite \
|
||||
# --from-literal=code="$(openssl rand -hex 16)"
|
||||
# - name: SIGNUP_INVITE_CODE
|
||||
# valueFrom:
|
||||
# secretKeyRef:
|
||||
# name: dsh-signup-invite
|
||||
# key: code
|
||||
# Reject proxied request bodies larger than this (413).
|
||||
- name: MAX_BODY_MB
|
||||
value: "32"
|
||||
# Per-user concurrency guard (default 32): at most this many
|
||||
# proxied HTTP requests and WebSocket connections per user.
|
||||
# Excess HTTP requests get 503 + Retry-After and excess WebSocket
|
||||
# upgrades close with 1011, so one tenant cannot exhaust the
|
||||
# single-replica gateway for everyone else.
|
||||
# - name: MAX_IN_FLIGHT_PER_USER
|
||||
# value: "32"
|
||||
# Largest WebSocket message received from DSH, in bytes (default
|
||||
# 16 MiB); oversized frames close the connection. Browser-to-DSH
|
||||
# frames are bounded by uvicorn's own limit. HTTP/SSE responses are
|
||||
# streamed without this cap.
|
||||
# - name: WS_MAX_MESSAGE_BYTES
|
||||
# value: "16777216"
|
||||
# /healthz is unauthenticated and returns 200 without touching the
|
||||
# cluster, so it is safe for both readiness and liveness.
|
||||
readinessProbe:
|
||||
|
||||
@@ -16,6 +16,16 @@ Applying both gives the gateway two entry points, and the SSO variant is only
|
||||
safe when the gateway is reachable **exclusively** through oauth2-proxy. Do not
|
||||
mix them.
|
||||
|
||||
> **Production requirement.** Each variant's NetworkPolicy is part of the
|
||||
> variant, and applying the variant without it is demo-only. The sandbox
|
||||
> NetworkPolicy and the gateway egress policy (the API server is cluster-specific)
|
||||
> are separate and equally required; see the
|
||||
> [production checklist](../../docs/optional-hardening.md#6-production-checklist)
|
||||
> and the
|
||||
> [gateway egress recipe](../../docs/optional-hardening.md#gateway-egress-required-for-production).
|
||||
> For SSO, run [`scripts/preflight-sso.sh`](../../scripts/preflight-sso.sh)
|
||||
> after applying the variant and before sending real traffic.
|
||||
|
||||
Each variant also ships a NetworkPolicy, applied by the same directory apply:
|
||||
|
||||
- `ingress/61-networkpolicy.yaml` allows only Traefik pods to reach the gateway
|
||||
@@ -27,7 +37,8 @@ Each variant also ships a NetworkPolicy, applied by the same directory apply:
|
||||
The policies assume Traefik's `app.kubernetes.io/name: traefik` pod label.
|
||||
Adjust the selector for another implementation or a `hostNetwork` install; see
|
||||
[optional hardening](../../docs/optional-hardening.md) for the probe caveat and
|
||||
verification commands.
|
||||
verification commands. Keep the policies: an exposure without its NetworkPolicy
|
||||
is demo-only.
|
||||
|
||||
## Applying a variant
|
||||
|
||||
@@ -69,10 +80,13 @@ Secrets imperatively (commands below) before applying the directory.
|
||||
2. Set `DSH_TRUSTED_HOSTS` in the SandboxTemplate to the same public host:
|
||||
the gateway forwards the browser's `Host`, and DSH rejects unknown hosts on
|
||||
`/api` (see `../10-sandboxtemplate.yaml`).
|
||||
3. Set `COOKIE_SECURE=true` on the gateway once the Ingress serves HTTPS:
|
||||
`kubectl -n dsh-demo set env deploy/dsh-gateway COOKIE_SECURE=true`. The
|
||||
base manifest keeps `false` so the plain-HTTP port-forward demo works; a
|
||||
`Secure` cookie is never sent over HTTP, which silently breaks login.
|
||||
3. Set `COOKIE_SECURE=true` and remove the plain-HTTP acknowledgement once the
|
||||
Ingress serves HTTPS:
|
||||
`kubectl -n dsh-demo set env deploy/dsh-gateway COOKIE_SECURE=true ALLOW_INSECURE_COOKIES-`.
|
||||
The base manifest keeps `false` + `ALLOW_INSECURE_COOKIES=1` so the
|
||||
plain-HTTP port-forward demo works; a `Secure` cookie is never sent over HTTP,
|
||||
which silently breaks login, while the acknowledgement must not survive into
|
||||
production.
|
||||
4. Do not attach buffering or compression middlewares to this router: they
|
||||
break WebSocket upgrades and streaming.
|
||||
5. `61-networkpolicy.yaml` is part of this variant: it allows only Traefik pods
|
||||
@@ -116,14 +130,23 @@ kubectl -n dsh-demo create secret tls dsh-demo-tls --cert=tls.crt --key=tls.key
|
||||
|
||||
# 3. Enable SSO on the gateway and trust the public host in the template.
|
||||
kubectl -n dsh-demo set env deploy/dsh-gateway \
|
||||
AUTH_PROXY_HEADER=X-Forwarded-User COOKIE_SECURE=true
|
||||
AUTH_PROXY_HEADER=X-Forwarded-User COOKIE_SECURE=true ALLOW_INSECURE_COOKIES-
|
||||
# then set DSH_TRUSTED_HOSTS=dsh-demo.example.com in the SandboxTemplate
|
||||
# (see ../10-sandboxtemplate.yaml) and re-apply it.
|
||||
|
||||
# 4. Apply the variant.
|
||||
kubectl apply -f manifests/exposure/gateway-api-sso/
|
||||
|
||||
# 5. Preflight: fails loudly on missing Secrets, a bad oauth2-proxy cookie
|
||||
# secret, missing gateway SSO env, a dangling TLS reference, leftover
|
||||
# placeholders, or a second path to the gateway.
|
||||
./scripts/preflight-sso.sh
|
||||
```
|
||||
|
||||
The preflight is read-only and exits non-zero on any failure, naming the exact
|
||||
fix. Run it again after rotating a Secret or changing the public host; it also
|
||||
verifies that the oauth2-proxy cookie secret decodes to 16, 24 or 32 bytes.
|
||||
|
||||
### SSO trust boundary
|
||||
|
||||
`AUTH_PROXY_HEADER` is opt-in and deliberately unset in the base gateway
|
||||
@@ -141,8 +164,14 @@ takes all of these:
|
||||
impersonate any user.
|
||||
3. Keep `75-networkpolicies.yaml` with the variant. It enforces the boundary in
|
||||
the network layer (gateway ← oauth2-proxy ← Traefik, and limited oauth2-proxy
|
||||
egress). A custom exposure without an equivalent policy has no network
|
||||
egress). This is required, not optional: an exposure without the policy is
|
||||
demo-only, and a custom exposure without an equivalent policy has no network
|
||||
enforcement.
|
||||
4. Apply the gateway egress policy too: the gateway talks to the Kubernetes API
|
||||
and to sandbox pods, and `75-networkpolicies.yaml` intentionally leaves that
|
||||
direction unrestricted because the API-server endpoint is cluster-specific.
|
||||
See
|
||||
[gateway egress](../../docs/optional-hardening.md#gateway-egress-required-for-production).
|
||||
|
||||
Also terminate TLS in front of the gateway (the Gateway listener does) and
|
||||
leave `COOKIE_SECURE=true` so the session cookie never travels in clear text.
|
||||
|
||||
@@ -25,8 +25,9 @@
|
||||
# TLS terminates at the Gateway listener (70-gateway.yaml); keep
|
||||
# --cookie-secure=true and COOKIE_SECURE=true on the gateway.
|
||||
#
|
||||
# The image is pinned to quay.io/oauth2-proxy/oauth2-proxy:v7.15.5
|
||||
# (versions.env: OAUTH2_PROXY_IMAGE). Credentials come from the
|
||||
# The image is digest-pinned: quay.io/oauth2-proxy/oauth2-proxy:v7.15.5@sha256:...
|
||||
# (versions.env: OAUTH2_PROXY_IMAGE). The digest is the immutable multi-arch
|
||||
# index, so the tag can never be moved under us. Credentials come from the
|
||||
# "oauth2-proxy" Secret (see 73-oauth2-proxy-secret.yaml.example); this
|
||||
# manifest never carries secrets.
|
||||
apiVersion: apps/v1
|
||||
@@ -61,7 +62,7 @@ spec:
|
||||
type: RuntimeDefault
|
||||
containers:
|
||||
- name: oauth2-proxy
|
||||
image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.5
|
||||
image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.5@sha256:8498b0d0ef0a7b29686414000a08aee467f02d0299c9ed1e006a8f33fc017916
|
||||
imagePullPolicy: IfNotPresent
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
# OPTIONAL with the SSO exposure variant: enforce the trust boundary at the
|
||||
# network layer.
|
||||
# REQUIRED with the SSO exposure variant: enforce the trust boundary at the
|
||||
# network layer. Applying the SSO variant without this file is demo-only.
|
||||
#
|
||||
# The chain is Traefik (Gateway API) -> oauth2-proxy -> dsh-gateway. These
|
||||
# policies make it the ONLY chain:
|
||||
@@ -8,8 +8,9 @@
|
||||
# - oauth2-proxy egress is limited to DNS, public HTTPS (GitHub) and the
|
||||
# gateway.
|
||||
# The gateway's own egress (Kubernetes API, sandbox pods) is intentionally not
|
||||
# restricted here: the API-server endpoint is cluster-specific. See
|
||||
# ../../../docs/optional-hardening.md for that recipe.
|
||||
# restricted here: the API-server endpoint is cluster-specific. Add it for
|
||||
# production. See
|
||||
# ../../../docs/optional-hardening.md#gateway-egress-required-for-production.
|
||||
#
|
||||
# This manifest assumes Traefik, matching gatewayClassName: traefik in
|
||||
# 70-gateway.yaml. Adjust the app.kubernetes.io/name selector for another
|
||||
|
||||
@@ -14,8 +14,13 @@
|
||||
# DSH_TRUSTED_HOSTS to this host so DSH accepts the forwarded Host/Origin
|
||||
# on /api.
|
||||
# 2. Gateway Deployment (manifests/50-gateway.yaml): set COOKIE_SECURE to
|
||||
# "true" once this Ingress serves HTTPS, so the session cookie is Secure.
|
||||
# The base manifest sets "false" for the plain-HTTP port-forward demo.
|
||||
# "true" and remove ALLOW_INSECURE_COOKIES once this Ingress serves HTTPS,
|
||||
# so the session cookie is Secure:
|
||||
# kubectl -n dsh-demo set env deploy/dsh-gateway \
|
||||
# COOKIE_SECURE=true ALLOW_INSECURE_COOKIES-
|
||||
# The base manifest sets "false" + ALLOW_INSECURE_COOKIES=1 for the
|
||||
# plain-HTTP port-forward demo; without the acknowledgement the gateway
|
||||
# refuses to start.
|
||||
#
|
||||
# WebSocket support is REQUIRED: the browser talks to /u/<user>/... and DSH
|
||||
# uses WebSockets for live session traffic. Traefik upgrades WS/SSE
|
||||
@@ -24,9 +29,11 @@
|
||||
# and upgrades. `path: /` Prefix covers the landing page and every per-user
|
||||
# proxy path, so no per-sandbox Ingress is needed.
|
||||
#
|
||||
# 61-networkpolicy.yaml is part of this variant: it restricts the gateway's
|
||||
# ingress to Traefik pods. Adjust its selector if your ingress controller uses
|
||||
# a different label or hostNetwork.
|
||||
# 61-networkpolicy.yaml is part of this variant and required for production:
|
||||
# it restricts the gateway's ingress to Traefik pods. Adjust its selector if
|
||||
# your ingress controller uses a different label or hostNetwork. The sandbox
|
||||
# and gateway-egress NetworkPolicies are separate requirements; see
|
||||
# ../../../docs/optional-hardening.md#6-production-checklist.
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
|
||||
@@ -1,11 +1,16 @@
|
||||
# OPTIONAL with the Ingress exposure variant: restrict the gateway's ingress
|
||||
# to the ingress controller.
|
||||
# REQUIRED with the Ingress exposure variant for production: restrict the
|
||||
# gateway's ingress to the ingress controller. Applying the variant without
|
||||
# this policy is demo-only.
|
||||
#
|
||||
# This manifest assumes Traefik, matching this variant's
|
||||
# `ingressClassName: traefik`. Traefik's Helm chart labels its pods
|
||||
# app.kubernetes.io/name=traefik; adjust the selector for another controller or
|
||||
# for a hostNetwork install (where the source is the node IP, not a pod IP).
|
||||
#
|
||||
# The sandbox NetworkPolicy and the gateway egress policy are separate and also
|
||||
# required for production; see
|
||||
# ../../../docs/optional-hardening.md#6-production-checklist.
|
||||
#
|
||||
# NetworkPolicy enforcement of kubelet probe traffic is CNI-dependent. If the
|
||||
# gateway starts failing its /healthz probes, allow the node CIDR on port 8080
|
||||
# or remove this policy. See ../../../docs/optional-hardening.md.
|
||||
|
||||
@@ -23,6 +23,12 @@ It requires a CNI that enforces `NetworkPolicy`. The selector uses the
|
||||
agent-sandbox controller's pod tracking label, so the policy applies to
|
||||
existing warm-pool and claimed sandboxes without recreation.
|
||||
|
||||
**This is opt-in for the demo but required for production.** Without it every
|
||||
tenant can reach every other tenant and cluster-internal endpoints. Production
|
||||
also needs the gateway egress policy (cluster-specific API-server endpoint) and
|
||||
the exposure variant's policy; see the
|
||||
[production checklist](../../docs/optional-hardening.md#6-production-checklist).
|
||||
|
||||
## gVisor, nodeSelector and provider keys
|
||||
|
||||
These are configured in the main manifests or with an extra Secret:
|
||||
|
||||
@@ -1,9 +1,15 @@
|
||||
# OPTIONAL, opt-in tenant network policy for the DSH sandboxes.
|
||||
# OPTIONAL for the base walkthrough, REQUIRED for production. Opt-in tenant
|
||||
# network policy for the DSH sandboxes: without it every tenant can reach every
|
||||
# other tenant and cluster-internal endpoints.
|
||||
#
|
||||
# It is NOT part of the top-level walkthrough. Apply and delete it explicitly:
|
||||
# kubectl apply -f manifests/optional/network-policy/
|
||||
# kubectl delete -f manifests/optional/network-policy/
|
||||
#
|
||||
# Production also needs the gateway egress policy (cluster-specific API-server
|
||||
# endpoint) and the exposure variant's policy, plus gVisor
|
||||
# (docs/optional-hardening.md#6-production-checklist).
|
||||
#
|
||||
# The selector uses the agent-sandbox controller's pod tracking label, which is
|
||||
# present on every Sandbox pod (including warm-pool spares), so the policy
|
||||
# applies to existing sandboxes without recreating them. The SandboxTemplate
|
||||
|
||||
Reference in New Issue
Block a user