deepseek-kubernetes-sandbox/examples/dsh-agents-as-a-service/docs/optional-hardening.md

Optional hardening: gVisor, nodeSelector, NetworkPolicy, provider keys and failed-Pod recovery

The base example is deliberately portable: it runs on kind and GKE without a gVisor runtime, a tainted node pool or a NetworkPolicy-capable CNI, and it boots with only the required dsh-provider-keys Secret. That portability is exactly why the base manifests are demo-only. This document covers the controls you add for production and the steps to enable or roll them back.

Production is not the base walkthrough. At minimum, a production deployment needs: gVisor for the sandbox pods (or another kernel-isolation boundary), the sandbox NetworkPolicy, the gateway egress policy, the exposure variant's NetworkPolicy, TLS with COOKIE_SECURE=true (and no ALLOW_INSECURE_COOKIES), SSO (or an equivalent authenticating proxy) as the only path to the gateway, and per-tenant provider credentials/egress. The checklist at the end summarizes this; without the network and kernel layers the example is not tenant-isolated.

This document covers four opt-in controls plus failed-Pod recovery:

Setting	Where it lives	What it changes
gVisor	commented runtimeClassName in the SandboxTemplates and Deployments	puts DSH and/or the gateway in a userspace kernel; required for untrusted agents in production
nodeSelector	commented nodeSelector in the same files	pins pods to a chosen node pool
NetworkPolicy	manifests/optional/network-policy/, manifests/exposure/*/, plus the gateway egress recipe	limits sandbox, gateway and proxy connectivity; the sandbox and gateway-egress policies are required for production
Provider keys	optional dsh-provider-keys-extra Secret	injects extra provider API keys into every sandbox
Failed-Pod recovery	commented failedPodPolicy in the SandboxTemplates	replaces a Failed sandbox Pod instead of parking the sandbox

Each setting is independent. Enable only what the cluster supports and test the walkthrough after each change.

1. gVisor

What it changes

The DSH sandbox and the gateway (and optionally oauth2-proxy) run under the gvisor RuntimeClass. agent-sandbox passes runtimeClassName straight to the PodSpec; no CRD change is involved.

Prerequisites

• A RuntimeClass named gvisor:

  kubectl get runtimeclass gvisor -o yaml
  

• Nodes capable of running it (runsc installed and registered with the container runtime). How you get there is cluster-specific:

  • GKE Sandbox: create a node pool with --sandbox=type=gvisor; GKE labels the nodes sandbox.gke.io/runtime: gvisor, taints them sandbox.gke.io/runtime=gvisor:NoSchedule, and its gvisor RuntimeClass applies the matching affinity and toleration automatically.
  • Self-managed: install gVisor, create the gvisor RuntimeClass with handler: runsc, and label/taint a dedicated node pool yourself.

• A scheduling plan: a gVisor RuntimeClass without capable nodes makes pods fail to start. Use the RuntimeClass scheduling fields or the optional nodeSelector/toleration described in section 2.

Enable it

Fresh install: uncomment runtimeClassName: gvisor before applying:

• manifests/10-sandboxtemplate.yaml
• manifests/50-gateway.yaml
• manifests/exposure/gateway-api-sso/72-oauth2-proxy.yaml (only with the SSO variant)
• minimal template

Existing install: a template edit does not update existing Sandboxes, and warm-pool spares use OnReplenish, so patch explicitly. Plan a maintenance window: claimed users are disconnected when their pod is recreated.

# 1. The blueprint for future sandboxes.
kubectl -n dsh-demo patch sandboxtemplate dsh-web --type merge \
  -p '{"spec":{"podTemplate":{"spec":{"runtimeClassName":"gvisor"}}}}'

# 2. Warm-pool spares: delete them; the pool replenishes from the template.
kubectl -n dsh-demo delete sandboxes -l agents.x-k8s.io/warm-pool-sandbox

# 3. Claimed sandboxes: snapshot their pod template, then suspend/resume.
for sb in $(kubectl -n dsh-demo get sandboxes \
  -l '!agents.x-k8s.io/warm-pool-sandbox' -o jsonpath='{.items[*].metadata.name}'); do
  kubectl -n dsh-demo patch sandbox "$sb" --type merge \
    -p '{"spec":{"podTemplate":{"spec":{"runtimeClassName":"gvisor"}}}}'
  kubectl -n dsh-demo patch sandbox "$sb" --type merge \
    -p '{"spec":{"operatingMode":"Suspended"}}'
  kubectl -n dsh-demo patch sandbox "$sb" --type merge \
    -p '{"spec":{"operatingMode":"Running"}}'
done

# 4. The gateway (and oauth2-proxy when the SSO variant is installed).
kubectl -n dsh-demo patch deploy dsh-gateway --type merge \
  -p '{"spec":{"template":{"spec":{"runtimeClassName":"gvisor"}}}}'
kubectl -n dsh-demo rollout status deploy/dsh-gateway --timeout=180s
# kubectl -n dsh-demo patch deploy oauth2-proxy --type merge \
#   -p '{"spec":{"template":{"spec":{"runtimeClassName":"gvisor"}}}}'

If the pool's nodes are tainted and the RuntimeClass does not already add a toleration, add the matching tolerations block (section 2) in the same patch or in the manifests.

Verify

SB=$(kubectl -n dsh-demo get sandboxclaim dsh-alice -o jsonpath='{.status.sandbox.name}')
POD=$(kubectl -n dsh-demo get sandbox "$SB" \
  -o jsonpath='{.metadata.annotations.agents\.x-k8s\.io/pod-name}'); POD=${POD:-$SB}

kubectl -n dsh-demo get pod "$POD" -o jsonpath='{.spec.runtimeClassName}{"\n"}'
# gvisor
kubectl -n dsh-demo exec "$POD" -- uname -r
kubectl -n dsh-demo get node "$(kubectl -n dsh-demo get pod "$POD" -o jsonpath='{.spec.nodeName}')" \
  -o jsonpath='{.status.nodeInfo.kernelVersion}{"\n"}'

The pod's runtimeClassName is authoritative. The kernel release reported by uname -r inside a gVisor pod is the sandbox kernel, not the host kernel; on GKE it looks like 4.19.0-gvisor.

Caveats

• kubectl port-forward does not work with gVisor. The multi-tenant gateway path (HTTP/WebSocket to the pod) works; the minimal example's port-forward quickstart does not. Use the gateway example or a reachable Service (NodePort/LoadBalancer/Sandbox Router) instead.
• gVisor has performance overhead and does not implement every syscall. Test DSH and the agent tools you rely on (git, gh, uv, language toolchains).
• Node scheduling is the hard part. A gvisor RuntimeClass on a cluster with no capable nodes leaves pods unschedulable or failing at container start.
• Existing Sandboxes are snapshots. The loop above is required for every claimed sandbox; deleting the SandboxClaim would also delete its PVC and workspace data, so do not use claim re-creation as a shortcut.
• A resumed sandbox starts cold; the startup probe gives it up to 120s.

Roll back

kubectl -n dsh-demo patch sandboxtemplate dsh-web --type merge \
  -p '{"spec":{"podTemplate":{"spec":{"runtimeClassName":null}}}}'
for sb in $(kubectl -n dsh-demo get sandboxes -o jsonpath='{.items[*].metadata.name}'); do
  kubectl -n dsh-demo patch sandbox "$sb" --type merge \
    -p '{"spec":{"podTemplate":{"spec":{"runtimeClassName":null}}}}'
done
kubectl -n dsh-demo delete sandboxes -l agents.x-k8s.io/warm-pool-sandbox
kubectl -n dsh-demo patch deploy dsh-gateway --type merge \
  -p '{"spec":{"template":{"spec":{"runtimeClassName":null}}}}'
# kubectl -n dsh-demo patch deploy oauth2-proxy --type merge \
#   -p '{"spec":{"template":{"spec":{"runtimeClassName":null}}}}'

Recreate the pods (delete spares, suspend/resume claimed sandboxes, let the Deployment roll out) so the cleared field takes effect.

2. nodeSelector

What it changes

Pods get a nodeSelector so they land only on nodes carrying the chosen label. The same field is available in both SandboxTemplates and both Deployments; the shipped value is commented out because active cluster-specific scheduling does not belong in the repository.

Choose a label

kubectl get nodes --show-labels

• GKE Sandbox: the nodes carry sandbox.gke.io/runtime: gvisor and are tainted sandbox.gke.io/runtime=gvisor:NoSchedule. With runtimeClassName: gvisor, GKE already schedules the pod onto those nodes; add a nodeSelector only when you need a specific sandbox pool or ComputeClass (cloud.google.com/compute-class: <name>).
• Self-managed: label the pool you want, for example dsh.example.com/node-pool: gvisor, and use that label.
• General pinning: any stable node-pool label works. Keep the real key and value in a local patch; the manifests use dsh.example.com/node-pool as an obvious placeholder.

Enable it

Uncomment and replace the placeholder before a fresh apply:

      nodeSelector:
        dsh.example.com/node-pool: gvisor

For an existing install, patch the template, each claimed Sandbox and the Deployments, then recreate pods exactly as in section 1:

kubectl -n dsh-demo patch sandboxtemplate dsh-web --type merge \
  -p '{"spec":{"podTemplate":{"spec":{"nodeSelector":{"dsh.example.com/node-pool":"gvisor"}}}}}'
# ... same warm-pool delete / claimed-sandbox suspend-resume / Deployment
# patch and rollout as in section 1, with nodeSelector instead of
# runtimeClassName.

If the selected nodes are tainted, add a matching toleration. This is the companion block; do not uncomment it unless the pool is actually tainted:

      tolerations:
        - key: dsh.example.com/node-pool
          operator: Equal
          value: gvisor
          effect: NoSchedule

GKE Sandbox's taint is sandbox.gke.io/runtime=gvisor:NoSchedule; the GKE gvisor RuntimeClass adds that toleration automatically, so a manual copy is only needed when you schedule without the RuntimeClass or with a different pool.

Verify and troubleshoot

kubectl -n dsh-demo get pod "$POD" -o wide
kubectl -n dsh-demo get pod "$POD" -o jsonpath='{.spec.nodeSelector}{"\n"}'
kubectl -n dsh-demo describe pod "$POD" | sed -n '/Events:/,$p'

• Pending with 0/N nodes are available: the label is wrong, the pool is full, or the pod needs a toleration for a taint.
• A pod-level nodeSelector is merged with the RuntimeClass scheduling.nodeSelector. If both are set and conflict, no node matches.
• Labels changed after scheduling do not move running pods. Patch/recreate.
• A nodeSelector that excludes every gVisor node is a valid way to keep the gateway on a normal pool while the sandboxes run under gVisor.

Roll back

Remove the field from the manifests, or patch it to null, then recreate the pods:

kubectl -n dsh-demo patch sandboxtemplate dsh-web --type merge \
  -p '{"spec":{"podTemplate":{"spec":{"nodeSelector":null}}}}'
# ... same per-Sandbox and Deployment loop as section 1.

3. NetworkPolicy

The base example does not manage NetworkPolicy: the SandboxTemplate is networkPolicyManagement: Unmanaged, and the gateway has no policy. That keeps the walkthrough portable and independent of the sandbox-router. For production this is not optional: without the sandbox policy every tenant can reach every other tenant and cluster-internal endpoints, and without a gateway egress policy the gateway can talk to anything the cluster network allows. Apply both. The policies below are standard Kubernetes NetworkPolicy objects.

Demo vs production. An installation that skips the sandbox NetworkPolicy and the gateway egress policy is a demo. Exposure (Ingress/Gateway API) adds ingress protection only; it does not isolate tenants from each other.

Sandbox policy (required for production)

manifests/optional/network-policy/10-sandbox-networkpolicy.yaml selects every Sandbox pod through the agent-sandbox tracking label agents.x-k8s.io/sandbox-name-hash. It applies to existing warm-pool and claimed sandboxes without recreating them.

kubectl apply -f manifests/optional/network-policy/
kubectl delete -f manifests/optional/network-policy/   # rollback

What it allows:

Direction	Allowed	Blocked
Ingress	gateway pods (app: dsh-gateway) on TCP 3080	everything else, including other sandboxes and cluster Services
Egress	UDP/TCP 53 to kube-dns and NodeLocal DNSCache; TCP 443 to public IPs	RFC1918, CGNAT, link-local/metadata, cluster Services/API, plain HTTP, other ports

Verify from inside the pods (NetworkPolicy does not affect kubectl exec itself, only the traffic the commands generate):

SB=$(kubectl -n dsh-demo get sandboxclaim dsh-alice -o jsonpath='{.status.sandbox.name}')
POD=$(kubectl -n dsh-demo get sandbox "$SB" \
  -o jsonpath='{.metadata.annotations.agents\.x-k8s\.io/pod-name}'); POD=${POD:-$SB}

# Allowed: gateway -> sandbox.
kubectl -n dsh-demo exec deploy/dsh-gateway -- python -c \
  "import urllib.request; print(urllib.request.urlopen('http://$POD:3080/favicon.svg', timeout=5).status)"

# Allowed: DNS and public HTTPS from the sandbox.
kubectl -n dsh-demo exec "$POD" -- getent hosts api.deepseek.com
kubectl -n dsh-demo exec "$POD" -- curl -fsS -o /dev/null -w '%{http_code}\n' https://example.com

# Denied: another sandbox cannot reach this one on 3080 (expect a timeout).
OTHER_SB=$(kubectl -n dsh-demo get sandboxclaim dsh-bob -o jsonpath='{.status.sandbox.name}')
OTHER_POD=$(kubectl -n dsh-demo get sandbox "$OTHER_SB" \
  -o jsonpath='{.metadata.annotations.agents\.x-k8s\.io/pod-name}'); OTHER_POD=${OTHER_POD:-$OTHER_SB}
kubectl -n dsh-demo exec "$OTHER_POD" -- curl --max-time 3 -sS "http://$POD:3080/favicon.svg"

Caveats:

• The CNI must enforce NetworkPolicy. If it does not, the file applies but has no effect.
• DNS: the default assumes CoreDNS/kube-dns pods labelled k8s-app: kube-dns and covers NodeLocal DNSCache at 169.254.20.10. A different DNS layout needs a selector/ipBlock edit.
• Kubelet probes from the node are CNI-dependent. If a pod starts failing its probes, allow the node CIDR on the probe port or remove the ingress policy.
• The policy blocks cluster Services and the Kubernetes API from the sandbox. DSH does not need them, but an agent tool that does needs an explicit rule.
• It allows any public host on TCP 443, not an FQDN allowlist. It is network isolation, not data-loss prevention.
• Custom provider base URLs, an HTTP proxy, port 80, SSH git (port 22) or a non-443 provider endpoint need explicit egress additions.
• Do not combine this standalone policy with networkPolicyManagement: Managed unless you migrate the rules into spec.networkPolicy. NetworkPolicies are additive; the controller's Managed secure default would also be present.
• kubectl port-forward still works for debugging (the connection is local to the pod), except under gVisor.

Exposure-variant policies (required, part of the variant)

Each exposure variant ships its own policy. They are part of the variant and required for production; an exposure without its policy is demo-only:

• manifests/exposure/ingress/61-networkpolicy.yaml allows only Traefik pods (app.kubernetes.io/name: traefik) to reach the gateway on 8080.
• manifests/exposure/gateway-api-sso/75-networkpolicies.yaml allows only oauth2-proxy to reach the gateway, only the Gateway implementation (Traefik) to reach oauth2-proxy, and limits oauth2-proxy egress to DNS, public HTTPS (GitHub) and the gateway.

The SSO policy is what turns the trust boundary in the exposure README into network enforcement. The dsh-gateway Service must still stay ClusterIP; do not apply both exposure variants. After applying the SSO variant, run scripts/preflight-sso.sh: it checks the gateway's SSO env, the Secrets, the TLS reference, leftover placeholders and accidental second paths (such as an Ingress).

Verify the negative case from a sandbox pod (it is not the proxy):

kubectl -n dsh-demo exec "$POD" -- curl --max-time 3 -sS http://dsh-gateway:8080/healthz
# expected: timeout/connection failure

Then verify the real browser path through the Ingress or Gateway, and confirm the gateway/proxy pods are Ready. If Traefik uses hostNetwork or a different pod label, replace the podSelector with the correct selector or an ipBlock. As with the sandbox policy, kubelet probes can be affected on some CNIs.

Gateway egress (required for production)

The gateway needs DNS, the Kubernetes API, and TCP 3080 to sandbox pods. An egress policy is required for production but is not shipped, because the API server endpoint is cluster-specific (Service CIDR or node IP/port) and a wrong value silently breaks signup/proxy. If your cluster has a stable API-server CIDR and port, add an egress policy like this (replace the placeholders; verify the port, commonly 443 or 6443):

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: dsh-gateway-egress
  namespace: dsh-demo
spec:
  podSelector:
    matchLabels:
      app: dsh-gateway
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53
    - to:
        - ipBlock:
            cidr: 10.96.0.1/32   # placeholder: API-server Service/cluster endpoint
      ports:
        - protocol: TCP
          port: 443
    - to:
        - podSelector:
            matchExpressions:
              - key: agents.x-k8s.io/sandbox-name-hash
                operator: Exists
      ports:
        - protocol: TCP
          port: 3080

Cilium users can express the API server as toEntities: kube-apiserver (or toServices) instead of a CIDR. Do not apply the sample blindly: a wrong CIDR or port silently breaks signup/proxy.

4. Optional provider keys

How injection works

The SandboxTemplate injects two Secrets via envFrom:

• dsh-provider-keys — required by the walkthrough; holds DEEPSEEK_API_KEY.
• dsh-provider-keys-extra — optional; referenced with optional: true, so a missing Secret does not block sandbox startup. Every key in it becomes an environment variable in every sandbox.

Create the extra Secret before the warm pool (pod env is stamped at creation):

kubectl -n dsh-demo create secret generic dsh-provider-keys-extra \
  --from-literal=OPENROUTER_API_KEY="sk-or-..." \
  --from-literal=GEMINI_API_KEY="..." \
  --from-literal=NVIDIA_API_KEY="nvapi-..." \
  --dry-run=client -o yaml | kubectl apply -f -

Use the exact env-var names (uppercase, underscores) — invalid Secret keys are skipped by the kubelet with an event. Do not duplicate a key across the two Secrets; when the same key exists twice, the later envFrom entry wins.

Supported names (curated)

These are the names the pinned DSH release resolves through its packaged provider catalog; the catalog has more entries, these are the common ones:

Provider	Environment variable
DeepSeek	DEEPSEEK_API_KEY
OpenRouter	OPENROUTER_API_KEY
Google Gemini	GEMINI_API_KEY
NVIDIA	NVIDIA_API_KEY
OpenAI	OPENAI_API_KEY
Anthropic	ANTHROPIC_API_KEY
Groq	GROQ_API_KEY
xAI	XAI_API_KEY
Mistral	MISTRAL_API_KEY
Together	TOGETHER_API_KEY
Cerebras	CEREBRAS_API_KEY
Vercel AI Gateway	AI_GATEWAY_API_KEY
Hugging Face	HF_TOKEN

Aliases to avoid: GOOGLE_API_KEY, HUGGINGFACE_API_KEY and VERCEL_AI_GATEWAY_API_KEY. The packaged catalog looks for GEMINI_API_KEY, HF_TOKEN and AI_GATEWAY_API_KEY instead.

Picking up a key change

A Secret update does not change running pods. The base key and extra keys are both snapshotted at container start:

# Warm-pool spares: delete them so the pool recreates them from the template.
kubectl -n dsh-demo delete sandboxes -l agents.x-k8s.io/warm-pool-sandbox

# Running claimed sandboxes: delete the pod (the Sandbox controller recreates
# it) or suspend/resume the sandbox.
POD=...   # resolve it as in section 3
kubectl -n dsh-demo delete pod "$POD"

New claims get the new values automatically. Confirm names without printing secret values:

kubectl -n dsh-demo exec "$POD" -- sh -c '
  for k in DEEPSEEK_API_KEY OPENROUTER_API_KEY GEMINI_API_KEY NVIDIA_API_KEY; do
    if [ -n "$(printenv "$k" || true)" ]; then echo "$k=set"; else echo "$k=unset"; fi
  done'

Precedence and shared-key limits

• DSH's stored credential file ($DSH_HOME/.credentials.yaml, mode 0600) wins over the environment. If a tenant saved a key through the DSH UI, changing the Secret will not override it.
• Both Secrets are shared by every tenant, and any agent can read its own environment. This is a warm-pool constraint, not tenant isolation. A tenant can override its own key through the DSH credential store on its PVC, but the platform does not manage that; real multi-tenancy needs per-tenant credentials or an egress proxy.
• The optional Secret is created imperatively; the repository never contains a real key. Keep it out of Git and out of shell history where possible.

5. Failed-Pod recovery

What it changes

When a sandbox Pod reaches phase Failed, the controller's default policy, Retain, leaves the Pod in place and surfaces Finished=True on the Sandbox. The Sandbox, its claim and its PVC survive, but nothing runs: the gateway keeps proxying to a dead Pod and every request fails until someone deletes the Pod or the claim.

agent-sandbox v1.0.8+ (the current pin in versions.env is newer) adds spec.failedPodPolicy to Sandbox and to SandboxTemplate:

Value	Behavior
Retain (default)	The Failed Pod stays and the Sandbox reports Finished=True; the evidence stays available for debugging.
Recreate	The controller deletes the Failed Pod and starts a replacement; the Sandbox identity and its Sandbox-owned PVCs are retained.

Repeated failures back off like a Deployment - 5 s, doubling to 5 min between creations - so a crash loop cannot hammer the API server. The backoff resets once the replacement Pod has been continuously Running for 10 minutes, and it lives in memory only, so a controller restart clears it.

The base templates ship it commented out, for two reasons:

• A deterministic failure (bad image, bad Secret, a wrong probe) would recreate forever while hiding the cause.
• The templates use restartPolicy: OnFailure, so the kubelet restarts a crashing container in place and the Pod does not reach phase Failed anyway. This policy matters for terminal failures - most commonly node eviction (Evicted), when a spot/preemptible node is reclaimed or the node hits memory pressure.

No gateway change is needed: the handshake cache is keyed by Pod UID, so a replacement Pod re-handshakes on the next request, and the existing 401 retry covers containers restarted in place.

Enable it

# 1. The blueprint for future sandboxes.
kubectl -n dsh-demo patch sandboxtemplate dsh-web --type merge \
  -p '{"spec":{"failedPodPolicy":"Recreate"}}'

# 2. Warm-pool spares were stamped at creation: delete them and let the pool
#    replenish from the updated template.
kubectl -n dsh-demo delete sandboxes -l agents.x-k8s.io/warm-pool-sandbox

# 3. Claimed sandboxes keep their own spec: patch them in place.
for sb in $(kubectl -n dsh-demo get sandboxes \
  -l '!agents.x-k8s.io/warm-pool-sandbox' -o jsonpath='{.items[*].metadata.name}'); do
  kubectl -n dsh-demo patch sandbox "$sb" --type merge \
    -p '{"spec":{"failedPodPolicy":"Recreate"}}'
done

For a fresh install, uncomment failedPodPolicy: Recreate in the template instead.

Verify

SB=$(kubectl -n dsh-demo get sandboxclaim dsh-alice -o jsonpath='{.status.sandbox.name}')
kubectl -n dsh-demo get sandbox "$SB" -o jsonpath='{.spec.failedPodPolicy}{"\n"}'
# Recreate

A failure drill that never touches a user sandbox - a Sandbox with restartPolicy: Never and a container that exits non-zero (swap the image if Docker Hub is unreachable):

kubectl -n dsh-demo apply -f - <<'EOF'
apiVersion: agents.x-k8s.io/v1beta1
kind: Sandbox
metadata:
  name: failed-pod-drill
spec:
  failedPodPolicy: Recreate
  podTemplate:
    spec:
      restartPolicy: Never
      containers:
        - name: drill
          image: busybox:1.36
          command: ["sh", "-c", "exit 1"]
EOF

kubectl -n dsh-demo get pods -w
# The drill Pod reaches phase Failed; a replacement appears within ~5 s, then
# 10 s, 20 s, ... as the container keeps exiting non-zero.
kubectl -n dsh-demo delete sandbox failed-pod-drill

Repeat the drill with failedPodPolicy: Retain (or omit the field) to see the default: the Pod stays Failed and the Sandbox condition Finished becomes True.

Caveats

• Recreate deletes the Failed Pod, so its logs go with it. Capture kubectl get events or switch back to Retain when debugging.
• The replacement is a cold start: DSH boots again and issues a new launch token. The workspace comes back from the PVC; in-flight sessions do not.
• The field needs agent-sandbox v1.0.8+. A v1.0.7 cluster rejects it as an unknown field.
• It applies to sandbox Pods only. The gateway and oauth2-proxy are plain Deployments, and their ReplicaSets already replace Failed Pods.

Roll back

kubectl -n dsh-demo patch sandboxtemplate dsh-web --type merge \
  -p '{"spec":{"failedPodPolicy":"Retain"}}'
for sb in $(kubectl -n dsh-demo get sandboxes -o jsonpath='{.items[*].metadata.name}'); do
  kubectl -n dsh-demo patch sandbox "$sb" --type merge \
    -p '{"spec":{"failedPodPolicy":"Retain"}}'
done

Warm-pool spares pick the default up as they are claimed and replenished, or delete them as in step 2 above.

6. Production checklist

The base walkthrough is a demo. A production install must satisfy every item below; the sections above give the commands and caveats.

Isolation and network

• Sandbox pods run under gVisor (or an equivalent kernel-isolation boundary) because DSH runs with DSH_PERMISSION_MODE=danger-full-access in this image; the pod is the boundary. (Section 1.)
• The sandbox NetworkPolicy is applied and verified: no tenant→tenant, no tenant→cluster-Service/API, DNS + public HTTPS only. (Section 3.)
• The gateway egress policy is applied with the cluster's real API-server CIDR/port, scoped to DNS, the API server and sandbox pods on 3080. (Section 3, "Gateway egress".)
• The exposure variant's NetworkPolicy is applied. An exposure without it is demo-only. For SSO, the gateway is reachable only through oauth2-proxy; there is no second Ingress/Gateway/port-forward path.
• Nodes/CNI enforce NetworkPolicy (verify the negative cases, not just the positive path).
• The gateway and (optionally) oauth2-proxy run with runAsNonRoot, dropped capabilities, RuntimeDefault seccomp and a read-only root filesystem, as shipped.

Authentication, TLS and abuse controls

• TLS terminates in front of the gateway, COOKIE_SECURE=true, and ALLOW_INSECURE_COOKIES is not set. The gateway refuses to start with an insecure cookie unless explicitly acknowledged.
• SSO (or an equivalent authenticating proxy) is enabled with AUTH_PROXY_HEADER=X-Forwarded-User, and scripts/preflight-sso.sh exits 0.
• No your-github-org / dsh-demo.example.com placeholder remains.
• SIGNUP_INVITE_CODE is set from a Secret if the gateway's own signup form is reachable, and AUTH_RATE_LIMIT is set deliberately (30/60s shipped). Add ingress-level rate limiting/quota too; the gateway's buckets are in-process and reset on restart.
• Session lifetime (COOKIE_MAX_AGE) and the logout/delete revocation behavior are acceptable; account for gateway restarts logging users out.
• MAX_IN_FLIGHT_PER_USER and WS_MAX_MESSAGE_BYTES fit the workload.

Credentials, platform and operations

• Provider keys are per-tenant (or routed through an egress proxy) instead of the shared dsh-provider-keys Secrets; any shared key is treated as visible to every tenant.
• RBAC stays namespace-scoped; pods/log is treated as sensitive because it contains launch tokens and model output. Plan the upstream token API migration.
• Images are digest-pinned and dependency installs are hash-locked.
• Gateway replicas stay at 1 (in-memory sessions/idle state) or the state is externalized.
• failedPodPolicy: Recreate is set if node eviction/spot reclaim is expected.
• Audit logging, alerting and backups of the PVCs are in place.

7. Verification summary

Required locally for any change to this example:

make check          # versions, example layout, manifest YAML shape
yamllint .
make test           # only when the gateway code changed
shellcheck examples/dsh-agents-as-a-service/scripts/*.sh

CI additionally parses every manifest and runs kubectl apply --dry-run=server against the real agent-sandbox CRDs. The weekly kind e2e runs the base walkthrough; it also proves the optional extra Secret may be absent.

Manual checks, on a cluster with the relevant CNI/runtime:

• gVisor: pod runtimeClassName, uname -r differs from the node kernel, gateway proxy works, port-forward limitation acknowledged.
• nodeSelector: pods land on the intended nodes; no Pending events.
• Sandbox policy: the allow/deny matrix above.
• Gateway egress policy: signup and proxy still work; the gateway cannot reach an arbitrary cluster Service or the metadata server.
• Exposure policies: browser path works; a non-proxy pod cannot reach the gateway; probes stay healthy. scripts/preflight-sso.sh exits 0.
• Cookie/TLS guard: with COOKIE_SECURE=false and no ALLOW_INSECURE_COOKIES the gateway refuses to start; with TLS it starts and the cookie is Secure (__Host- prefixed).
• Auth abuse controls: 429 + Retry-After after the configured limit, 403 on a wrong invite code, and revocations on logout/delete/recreate.
• Provider keys: set/unset output (never the values) and the refresh procedure.
• Failed-Pod recovery: spec.failedPodPolicy on the Sandbox, and the drill above replaces a Failed Pod while keeping the claim and PVC.

8. Rollback summary

Layer	Rollback
Sandbox policy	kubectl delete -f manifests/optional/network-policy/
Gateway egress policy	kubectl delete networkpolicy dsh-gateway-egress -n dsh-demo (do not leave it half-configured)
Exposure policy	delete the exposure variant directory
gVisor	patch runtimeClassName: null, recreate pods
nodeSelector	patch nodeSelector: null, recreate pods
Provider keys	delete the extra Secret or a key, recreate pods
Failed-Pod recovery	patch failedPodPolicy: Retain, delete warm-pool spares

No PVC or workspace data is touched by any rollback. Deleting a SandboxClaim is different: it cascades to the Sandbox and its PVC, so do not use claim deletion to apply a pod-spec change.