feat(example): add optional failed-Pod recovery (#26)

agent-sandbox v1.0.8 adds spec.failedPodPolicy to Sandbox and
SandboxTemplate. The default, Retain, leaves a Failed Pod in place and
surfaces Finished=True; Recreate deletes it and starts a replacement with
the same Sandbox identity and PVCs.

Ship it as an opt-in toggle next to the other commented fields, with a
new section 5 in the optional-hardening doc: what it changes, how to
enable it for new, warm-pool and already-claimed Sandboxes (the policy is
stamped at creation), a failure drill, caveats and rollback. The base
walkthrough stays on Retain because a deterministic failure would
otherwise recreate forever, and because the templates use
restartPolicy OnFailure, so the policy only matters for terminal cases
such as node eviction.

Cross-references updated in the example README, the optional-manifests
index, the architecture failure-mode table and the platform skill's
troubleshooting table.

Co-authored-by: SREDevOpsAgents <agents@sredevops.org>

.dsh/skills/dsh-agent-sandbox-platform/references/troubleshooting.md

@@ -20,6 +20,7 @@ events, then compare desired vs observed state. Keep output bounded. Never
 | Restart with mode-660 error | logs mention `.credentials.yaml` | kubelet fsGroup widened the file; suspend, `chmod 600` from a throwaway pod, patch `fsGroupChangePolicy: OnRootMismatch`, resume. |
 | PVC `Pending` | `kubectl describe pvc`, StorageClass | No default provisioner, access mode, zone binding. |
 | Launch token not found | `kubectl logs <pod> --tail=2000 | grep 'dsh web:'` | Pod not ready yet, or logs rotated. Do not guess tokens; wait for readiness. |
+| Sandbox reports `Finished=True` | pod phase, `kubectl get events` | The backing Pod reached `Failed` (eviction, spot reclaim) and `failedPodPolicy` defaults to `Retain`. Delete the Pod to force a replacement, or set `Recreate` (v1.0.8+) on the Sandbox/template; the claim and PVCs survive either way. |
 | Hand-applied sandbox disappeared | claim annotations, gateway logs | Idle sweeper only manages claims with `aaas.example.com/managed-by=dsh-gateway`; if it was managed, recreate/resume. |
 | Gateway 401 | gateway logs, cookie flags | Missing/expired cookie, DSH handshake failed, or `COOKIE_SECURE=true` over HTTP. |
 | Gateway 502/503 | sandbox Ready, gateway logs | Wake timeout, stale DNS, token not yet in logs; check `WAKE_TIMEOUT` and RBAC `pods/log`. |

examples/dsh-agents-as-a-service/README.md

@@ -541,8 +541,8 @@ gateway remains.
 
 ## Optional hardening
 
-The base walkthrough is deliberately portable. Four opt-in layers are documented
+The base walkthrough is deliberately portable. Four opt-in isolation layers are
-in [optional hardening](docs/optional-hardening.md):
+documented in [optional hardening](docs/optional-hardening.md):
 
 - **gVisor** — uncomment `runtimeClassName: gvisor` before applying, or patch an
   existing install. Requires a `gvisor` RuntimeClass and capable nodes.
@@ -554,6 +554,9 @@ in [optional hardening](docs/optional-hardening.md):
   `kubectl apply -f manifests/optional/network-policy/` allows only
   gateway→sandbox on `3080` and sandbox→DNS/public `443`.
 - **Provider keys** — create `dsh-provider-keys-extra` as shown in step 1.
+- **Failed-Pod recovery** — uncomment `failedPodPolicy: Recreate` so the
+  controller replaces a `Failed` sandbox Pod (node eviction) instead of parking
+  the sandbox at `Finished=True`. Requires agent-sandbox v1.0.8+.
 
 Apply each independently and re-run the walkthrough. `kubectl apply -f manifests/`
 is non-recursive, so the optional manifest is not applied by the base step.
@@ -586,6 +589,7 @@ is non-recursive, so the optional manifest is not applied by the base step.
 | Sandbox cannot reach a model API after applying the optional NetworkPolicy | The policy allows DNS and public TCP `443` only; a proxy, custom base URL, port `80` or non-443 endpoint needs an explicit egress rule |
 | Gateway cannot reach a sandbox after applying an exposure NetworkPolicy | Check that the CNI enforces NetworkPolicy and that the Traefik/oauth2-proxy pod labels match; kubelet probes may also need the node CIDR |
 | New provider key not visible in a sandbox | Pod env is stamped at creation: delete the warm-pool spares and recreate claimed pods. A key saved in DSH's credential store also wins over env |
+| Sandbox reports `Finished=True` and every gateway request fails | The sandbox Pod reached phase `Failed` (node eviction, spot reclaim) and the default `Retain` keeps it. Delete the Pod so the controller recreates it, or set `failedPodPolicy: Recreate` (agent-sandbox v1.0.8+; see optional hardening) |
 
 ## Limitations
 
@@ -649,4 +653,5 @@ delete the Secrets with the namespace and revoke the OAuth App in GitHub.
   security and RBAC, failure modes, evolution path.
 - [Exposure README](manifests/exposure/README.md) — the two variants in detail.
 - [Optional hardening](docs/optional-hardening.md) — gVisor, nodeSelector,
-  NetworkPolicy and extra provider keys, with verification and rollback.
+  NetworkPolicy, extra provider keys and failed-Pod recovery, with verification
+  and rollback.

examples/dsh-agents-as-a-service/docs/architecture.md

@@ -301,10 +301,11 @@ Notes and limits:
   `dsh-provider-keys-extra`). `manifests/optional/network-policy/` adds an opt-in sandbox
   NetworkPolicy, and gVisor/`nodeSelector` are opt-in PodSpec fields. Add per-tenant
   credentials before calling it multi-tenant-grade.
-- **Optional hardening is opt-in.** gVisor, node selection, the sandbox/exposure NetworkPolicies
+- **Optional hardening is opt-in.** gVisor, node selection, the sandbox/exposure NetworkPolicies,
-  and extra provider keys are documented in [optional hardening](optional-hardening.md). The
+  extra provider keys and failed-Pod recovery are documented in
-  base manifests stay portable (no gVisor RuntimeClass, no active node labels, no
+  [optional hardening](optional-hardening.md). The base manifests stay portable (no gVisor
-  NetworkPolicy and no extra Secret required).
+  RuntimeClass, no active node labels, no NetworkPolicy, no extra Secret required, and the
+  controller's default `failedPodPolicy: Retain`).
 - **No gateway rate limiting.** Signup can create claims until the cluster or warm pool is
   exhausted; put limits at the ingress for real deployments.
 
@@ -323,6 +324,7 @@ Notes and limits:
 | gVisor + `kubectl port-forward` | Forward refuses or returns nothing | Port-forward is incompatible with gVisor; use the gateway or a reachable Service |
 | Provider key added but missing in the pod | Requests still fail with `MISSING_CREDENTIAL` | Pod env is stamped at creation; recreate spares/pods. A stored credential wins over env |
 | `nodeSelector` matches no node | Pod `Pending`, `0/N nodes are available` | Fix the label, pool capacity or taint/toleration; RuntimeClass scheduling may also constrain it |
+| Sandbox Pod reaches phase `Failed` (node eviction, spot reclaim) | Sandbox reports `Finished=True`; the gateway keeps failing against a dead Pod | Set `failedPodPolicy: Recreate` (agent-sandbox v1.0.8+) so the controller replaces the Pod with the same identity and PVCs; otherwise delete the Pod by hand. See the hardening doc |
 
 ## Upstream feature request: remove log scraping
 

examples/dsh-agents-as-a-service/docs/optional-hardening.md

@@ -1,18 +1,19 @@
-# Optional hardening: gVisor, nodeSelector, NetworkPolicy and provider keys
+# Optional hardening: gVisor, nodeSelector, NetworkPolicy, provider keys and failed-Pod recovery
 
 The base example is deliberately portable: it runs on kind and GKE without a
 gVisor runtime, a tainted node pool or a NetworkPolicy-capable CNI, and it
 boots with only the required `dsh-provider-keys` Secret. This document covers
-four **opt-in** layers:
+four **opt-in** isolation layers, plus one availability setting:
 
-| Layer | Where it lives | What it changes |
+| Setting | Where it lives | What it changes |
 | --- | --- | --- |
 | gVisor | commented `runtimeClassName` in the SandboxTemplates and Deployments | puts DSH and/or the gateway in a userspace kernel |
 | nodeSelector | commented `nodeSelector` in the same files | pins pods to a chosen node pool |
 | NetworkPolicy | `manifests/optional/network-policy/`, `manifests/exposure/*/` | limits sandbox, gateway and proxy connectivity |
 | Provider keys | optional `dsh-provider-keys-extra` Secret | injects extra provider API keys into every sandbox |
+| Failed-Pod recovery | commented `failedPodPolicy` in the SandboxTemplates | replaces a Failed sandbox Pod instead of parking the sandbox |
 
-Each layer is independent. Enable only what the cluster supports and test the
+Each setting is independent. Enable only what the cluster supports and test the
 walkthrough after each change.
 
 ## 1. gVisor
@@ -473,7 +474,130 @@ kubectl -n dsh-demo exec "$POD" -- sh -c '
 - The optional Secret is created imperatively; the repository never contains a
   real key. Keep it out of Git and out of shell history where possible.
 
-## 5. Verification summary
+## 5. Failed-Pod recovery
+
+### What it changes
+
+When a sandbox Pod reaches phase `Failed`, the controller's default policy,
+`Retain`, leaves the Pod in place and surfaces `Finished=True` on the Sandbox.
+The Sandbox, its claim and its PVC survive, but nothing runs: the gateway keeps
+proxying to a dead Pod and every request fails until someone deletes the Pod or
+the claim.
+
+agent-sandbox v1.0.8 (the pin in [versions.env](../../../versions.env)) adds
+`spec.failedPodPolicy` to `Sandbox` and to `SandboxTemplate`:
+
+| Value | Behavior |
+| --- | --- |
+| `Retain` (default) | The Failed Pod stays and the Sandbox reports `Finished=True`; the evidence stays available for debugging. |
+| `Recreate` | The controller deletes the Failed Pod and starts a replacement; the Sandbox identity and its Sandbox-owned PVCs are retained. |
+
+Repeated failures back off like a Deployment - 5 s, doubling to 5 min between
+creations - so a crash loop cannot hammer the API server. The backoff resets
+once the replacement Pod has been continuously `Running` for 10 minutes, and it
+lives in memory only, so a controller restart clears it.
+
+The base templates ship it commented out, for two reasons:
+
+- A deterministic failure (bad image, bad Secret, a wrong probe) would recreate
+  forever while hiding the cause.
+- The templates use `restartPolicy: OnFailure`, so the kubelet restarts a
+  crashing container in place and the Pod does not reach phase `Failed` anyway.
+  This policy matters for terminal failures - most commonly node eviction
+  (`Evicted`), when a spot/preemptible node is reclaimed or the node hits memory
+  pressure.
+
+No gateway change is needed: the handshake cache is keyed by Pod UID, so a
+replacement Pod re-handshakes on the next request, and the existing `401` retry
+covers containers restarted in place.
+
+### Enable it
+
+```bash
+# 1. The blueprint for future sandboxes.
+kubectl -n dsh-demo patch sandboxtemplate dsh-web --type merge \
+  -p '{"spec":{"failedPodPolicy":"Recreate"}}'
+
+# 2. Warm-pool spares were stamped at creation: delete them and let the pool
+#    replenish from the updated template.
+kubectl -n dsh-demo delete sandboxes -l agents.x-k8s.io/warm-pool-sandbox
+
+# 3. Claimed sandboxes keep their own spec: patch them in place.
+for sb in $(kubectl -n dsh-demo get sandboxes \
+  -l '!agents.x-k8s.io/warm-pool-sandbox' -o jsonpath='{.items[*].metadata.name}'); do
+  kubectl -n dsh-demo patch sandbox "$sb" --type merge \
+    -p '{"spec":{"failedPodPolicy":"Recreate"}}'
+done
+```
+
+For a fresh install, uncomment `failedPodPolicy: Recreate` in the template
+instead.
+
+### Verify
+
+```bash
+SB=$(kubectl -n dsh-demo get sandboxclaim dsh-alice -o jsonpath='{.status.sandbox.name}')
+kubectl -n dsh-demo get sandbox "$SB" -o jsonpath='{.spec.failedPodPolicy}{"\n"}'
+# Recreate
+```
+
+A failure drill that never touches a user sandbox - a `Sandbox` with
+`restartPolicy: Never` and a container that exits non-zero (swap the image if
+Docker Hub is unreachable):
+
+```bash
+kubectl -n dsh-demo apply -f - <<'EOF'
+apiVersion: agents.x-k8s.io/v1beta1
+kind: Sandbox
+metadata:
+  name: failed-pod-drill
+spec:
+  failedPodPolicy: Recreate
+  podTemplate:
+    spec:
+      restartPolicy: Never
+      containers:
+        - name: drill
+          image: busybox:1.36
+          command: ["sh", "-c", "exit 1"]
+EOF
+
+kubectl -n dsh-demo get pods -w
+# The drill Pod reaches phase Failed; a replacement appears within ~5 s, then
+# 10 s, 20 s, ... as the container keeps exiting non-zero.
+kubectl -n dsh-demo delete sandbox failed-pod-drill
+```
+
+Repeat the drill with `failedPodPolicy: Retain` (or omit the field) to see the
+default: the Pod stays `Failed` and the Sandbox condition `Finished` becomes
+`True`.
+
+### Caveats
+
+- `Recreate` deletes the Failed Pod, so its logs go with it. Capture
+  `kubectl get events` or switch back to `Retain` when debugging.
+- The replacement is a cold start: DSH boots again and issues a new launch
+  token. The workspace comes back from the PVC; in-flight sessions do not.
+- The field needs agent-sandbox v1.0.8+. A v1.0.7 cluster rejects it as an
+  unknown field.
+- It applies to sandbox Pods only. The gateway and oauth2-proxy are plain
+  Deployments, and their ReplicaSets already replace Failed Pods.
+
+### Roll back
+
+```bash
+kubectl -n dsh-demo patch sandboxtemplate dsh-web --type merge \
+  -p '{"spec":{"failedPodPolicy":"Retain"}}'
+for sb in $(kubectl -n dsh-demo get sandboxes -o jsonpath='{.items[*].metadata.name}'); do
+  kubectl -n dsh-demo patch sandbox "$sb" --type merge \
+    -p '{"spec":{"failedPodPolicy":"Retain"}}'
+done
+```
+
+Warm-pool spares pick the default up as they are claimed and replenished, or
+delete them as in step 2 above.
+
+## 6. Verification summary
 
 Required locally for any change to this example:
 
@@ -497,8 +621,10 @@ Manual checks, on a cluster with the relevant CNI/runtime:
 - Exposure policies: browser path works; a non-proxy pod cannot reach the
   gateway; probes stay healthy.
 - Provider keys: set/unset output (never the values) and the refresh procedure.
+- Failed-Pod recovery: `spec.failedPodPolicy` on the Sandbox, and the drill
+  above replaces a `Failed` Pod while keeping the claim and PVC.
 
-## 6. Rollback summary
+## 7. Rollback summary
 
 | Layer | Rollback |
 | --- | --- |
@@ -507,6 +633,7 @@ Manual checks, on a cluster with the relevant CNI/runtime:
 | gVisor | patch `runtimeClassName: null`, recreate pods |
 | nodeSelector | patch `nodeSelector: null`, recreate pods |
 | Provider keys | delete the extra Secret or a key, recreate pods |
+| Failed-Pod recovery | patch `failedPodPolicy: Retain`, delete warm-pool spares |
 
 No PVC or workspace data is touched by any rollback. Deleting a `SandboxClaim`
 is different: it cascades to the Sandbox **and** its PVC, so do not use claim

examples/dsh-agents-as-a-service/manifests/10-sandboxtemplate.yaml

@@ -29,6 +29,15 @@ spec:
   # be blocked. See docs/optional-hardening.md.
   networkPolicyManagement: Unmanaged
 
+  # OPTIONAL: replace a Failed sandbox Pod instead of parking the sandbox.
+  # The default, Retain, leaves the Failed Pod in place and surfaces
+  # Finished=True on the Sandbox until a human intervenes. Recreate deletes the
+  # Failed Pod and starts a replacement with the same Sandbox identity and
+  # Sandbox-owned PVCs, backing off (5s doubling to 5m) on repeated failures.
+  # Requires agent-sandbox v1.0.8+; existing Sandboxes keep their own stamped
+  # policy, so patch them separately. See docs/optional-hardening.md.
+  # failedPodPolicy: Recreate
+
   podTemplate:
     spec:
       # DSH Web is multi-arch (amd64 + arm64); no arch nodeSelector is needed.

examples/dsh-agents-as-a-service/manifests/optional/README.md

@@ -30,6 +30,9 @@ These are configured in the main manifests or with an extra Secret:
 - **gVisor:** uncomment `runtimeClassName: gvisor` in the SandboxTemplate and
   the Deployments before applying, or patch an existing install.
 - **nodeSelector:** uncomment the field and replace the placeholder node label.
+- **Failed-Pod recovery:** uncomment `failedPodPolicy: Recreate` in the
+  SandboxTemplate (agent-sandbox v1.0.8+) so a Failed Pod is replaced instead
+  of parked.
 - **Provider keys:** create the optional `dsh-provider-keys-extra` Secret; the
   base `dsh-provider-keys` Secret remains the one required key.
 

examples/minimal-single-tenant/manifests/sandbox-template.yaml

@@ -26,6 +26,10 @@ spec:
   # quickstart reaches the sandbox through `kubectl port-forward`, which is
   # local to the pod. The multi-tenant example documents opt-in policies in
   # examples/dsh-agents-as-a-service/docs/optional-hardening.md.
+  #
+  # OPTIONAL: recreate a Failed sandbox Pod instead of retaining it
+  # (agent-sandbox v1.0.8+). Documented with the multi-tenant example.
+  # failedPodPolicy: Recreate
   podTemplate:
     spec:
       # OPTIONAL: run under gVisor. gVisor breaks `kubectl port-forward`, so