deepseek-kubernetes-sandbox/examples/dsh-agents-as-a-service/manifests/exposure/README.md

Exposing the gateway

The base manifests deploy the gateway as a ClusterIP Service (dsh-gateway:8080) and expose nothing outside the cluster. WebSocket upgrades on /u/<user>/** are part of the contract, so the entry point must forward Upgrade/Connection headers (Traefik does by default).

There are two mutually exclusive exposure variants. Pick exactly one:

Variant	Entry point	Authentication	TLS
ingress/60-ingress.yaml	Traefik Ingress	the gateway's own signup/session cookie	optional, recommended
gateway-api-sso/	Gateway API Gateway + HTTPRoute	GitHub OAuth via oauth2-proxy, then the gateway cookie	required (the Gateway listener terminates it)

Applying both gives the gateway two entry points, and the SSO variant is only safe when the gateway is reachable exclusively through oauth2-proxy. Do not mix them.

Production requirement. Each variant's NetworkPolicy is part of the variant, and applying the variant without it is demo-only. The sandbox NetworkPolicy and the gateway egress policy (the API server is cluster-specific) are separate and equally required; see the production checklist and the gateway egress recipe. For SSO, run scripts/preflight-sso.sh after applying the variant and before sending real traffic.

Each variant also ships a NetworkPolicy, applied by the same directory apply:

• ingress/61-networkpolicy.yaml allows only Traefik pods to reach the gateway on 8080.
• gateway-api-sso/75-networkpolicies.yaml allows only oauth2-proxy to reach the gateway, only the Gateway implementation (Traefik) to reach oauth2-proxy, and limits oauth2-proxy egress to DNS, public HTTPS (GitHub) and the gateway.

The policies assume Traefik's app.kubernetes.io/name: traefik pod label. Adjust the selector for another implementation or a hostNetwork install; see optional hardening for the probe caveat and verification commands. Keep the policies: an exposure without its NetworkPolicy is demo-only.

Applying a variant

Run these commands from the example root (examples/dsh-agents-as-a-service/), the same working directory the example README uses.

kubectl apply -f manifests/ is not recursive: it applies only the top-level numbered files and does not descend into exposure/. Apply the base resources first, then the chosen variant explicitly:

# Base resources: namespace, template, warm pool, claims, gateway.
kubectl apply -f manifests/

# Plain Ingress exposure:
kubectl apply -f manifests/exposure/ingress/

# ... or SSO exposure (create the two Secrets and replace the example.com
# placeholders first — see below):
kubectl apply -f manifests/exposure/gateway-api-sso/

Never use kubectl apply -R (recursive): it would apply both variants at once and hand out a second, unauthenticated path to the same gateway. Apply the variant directory on purpose, or the individual files by name.

kubectl only reads .yaml, .yml and .json files from a directory, so the *.yaml.example secret templates are skipped automatically; create the real Secrets imperatively (commands below) before applying the directory.

ingress/: plain Ingress, no SSO

60-ingress.yaml routes a host through the cluster's Traefik (ingressClassName: traefik; k3s ships one) to the gateway Service.

1. Replace the dsh.example.com host and the dsh-example-tls Secret name with real values, or delete the tls: block to serve plain HTTP.
2. Set DSH_TRUSTED_HOSTS in the SandboxTemplate to the same public host: the gateway forwards the browser's Host, and DSH rejects unknown hosts on /api (see ../10-sandboxtemplate.yaml).
3. Set COOKIE_SECURE=true and remove the plain-HTTP acknowledgement once the Ingress serves HTTPS: kubectl -n dsh-demo set env deploy/dsh-gateway COOKIE_SECURE=true ALLOW_INSECURE_COOKIES-. The base manifest keeps false + ALLOW_INSECURE_COOKIES=1 so the plain-HTTP port-forward demo works; a Secure cookie is never sent over HTTP, which silently breaks login, while the acknowledgement must not survive into production.
4. Do not attach buffering or compression middlewares to this router: they break WebSocket upgrades and streaming.
5. 61-networkpolicy.yaml is part of this variant: it allows only Traefik pods to reach the gateway on 8080. Adjust its selector for another ingress controller or hostNetwork; if kubelet probes fail, see the hardening doc.

Verify: curl -sI https://<host>/healthz returns 200.

gateway-api-sso/: TLS + GitHub OAuth

70-gateway.yaml defines the TLS Gateway, 71-httproute.yaml sends every request to oauth2-proxy, and 72-oauth2-proxy.yaml authenticates the browser against GitHub and proxies to the gateway with X-Forwarded-User set. 75-networkpolicies.yaml makes this the only network path to the gateway.

Prerequisites:

• Gateway API CRDs and a Gateway implementation installed; gatewayClassName: traefik matches the Traefik Gateway provider. The CRD version is pinned in versions.env (GATEWAY_API_VERSION).
• The OAuth App credentials and a cookie secret, from 73-oauth2-proxy-secret.yaml.example.
• A TLS certificate/key for the public host, from 74-tls-secret.yaml.example.
• oauth2-proxy is pinned in the manifest to quay.io/oauth2-proxy/oauth2-proxy:v7.15.5 (versions.env: OAUTH2_PROXY_IMAGE).

Steps:

# 1. Real Secrets (the committed files are *.yaml.example templates).
kubectl -n dsh-demo create secret generic oauth2-proxy \
  --from-literal=client-id='<GitHub OAuth App client ID>' \
  --from-literal=client-secret='<GitHub OAuth App client secret>' \
  --from-literal=cookie-secret="$(python3 -c 'import secrets; print(secrets.token_urlsafe(32))')"
kubectl -n dsh-demo create secret tls dsh-demo-tls --cert=tls.crt --key=tls.key

# 2. Replace dsh-demo.example.com (70/71/72) and your-github-org (72) with
#    the real public host and allowed GitHub organization.

# 3. Enable SSO on the gateway and trust the public host in the template.
kubectl -n dsh-demo set env deploy/dsh-gateway \
  AUTH_PROXY_HEADER=X-Forwarded-User COOKIE_SECURE=true ALLOW_INSECURE_COOKIES-
# then set DSH_TRUSTED_HOSTS=dsh-demo.example.com in the SandboxTemplate
# (see ../10-sandboxtemplate.yaml) and re-apply it.

# 4. Apply the variant.
kubectl apply -f manifests/exposure/gateway-api-sso/

# 5. Preflight: fails loudly on missing Secrets, a bad oauth2-proxy cookie
#    secret, missing gateway SSO env, a dangling TLS reference, leftover
#    placeholders, a second path to the gateway, a Role without `patch` on
#    sandboxclaims, or missing SSO NetworkPolicies.
./scripts/preflight-sso.sh

The preflight is read-only and exits non-zero on any failure, naming the exact fix. Run it again after rotating a Secret or changing the public host; it also verifies that the oauth2-proxy cookie secret is a valid AES key length (16, 24 or 32 raw bytes, or URL-safe unpadded base64 that decodes to one of those).

SSO trust boundary

AUTH_PROXY_HEADER is opt-in and deliberately unset in the base gateway manifest: the gateway ignores identity headers until it is set. Enabling SSO takes all of these:

1. AUTH_PROXY_HEADER=X-Forwarded-User on deploy/dsh-gateway (step 3 above). Without it, oauth2-proxy's login has no effect on the gateway and users are still asked for the gateway's own token.
2. The gateway Service must be reachable only through oauth2-proxy: keep it ClusterIP, route all browser traffic through the HTTPRoute → oauth2-proxy → dsh-gateway chain, and never apply ingress/ (or any other path to the Service) while SSO is enabled. A client that can reach dsh-gateway:8080 directly can set X-Forwarded-User itself and impersonate any user.
3. Keep 75-networkpolicies.yaml with the variant. It enforces the boundary in the network layer (gateway ← oauth2-proxy ← Traefik, and limited oauth2-proxy egress). This is required, not optional: an exposure without the policy is demo-only, and a custom exposure without an equivalent policy has no network enforcement.
4. Apply the gateway egress policy too: the gateway talks to the Kubernetes API and to sandbox pods, and 75-networkpolicies.yaml intentionally leaves that direction unrestricted because the API-server endpoint is cluster-specific. See gateway egress.

Also terminate TLS in front of the gateway (the Gateway listener does) and leave COOKIE_SECURE=true so the session cookie never travels in clear text.

Switching variants

Delete the variant you had before applying the other one, e.g. kubectl delete -f manifests/exposure/ingress/, and revert AUTH_PROXY_HEADER/COOKIE_SECURE on the gateway. Verify with kubectl -n dsh-demo get ingress,gateway,httproute that exactly one path to the gateway remains.