diff --git a/.dsh/skills/dsh-agent-sandbox-platform/references/troubleshooting.md b/.dsh/skills/dsh-agent-sandbox-platform/references/troubleshooting.md index 894918f..7750740 100644 --- a/.dsh/skills/dsh-agent-sandbox-platform/references/troubleshooting.md +++ b/.dsh/skills/dsh-agent-sandbox-platform/references/troubleshooting.md @@ -20,6 +20,7 @@ events, then compare desired vs observed state. Keep output bounded. Never | Restart with mode-660 error | logs mention `.credentials.yaml` | kubelet fsGroup widened the file; suspend, `chmod 600` from a throwaway pod, patch `fsGroupChangePolicy: OnRootMismatch`, resume. | | PVC `Pending` | `kubectl describe pvc`, StorageClass | No default provisioner, access mode, zone binding. | | Launch token not found | `kubectl logs --tail=2000 | grep 'dsh web:'` | Pod not ready yet, or logs rotated. Do not guess tokens; wait for readiness. | +| Sandbox reports `Finished=True` | pod phase, `kubectl get events` | The backing Pod reached `Failed` (eviction, spot reclaim) and `failedPodPolicy` defaults to `Retain`. Delete the Pod to force a replacement, or set `Recreate` (v1.0.8+) on the Sandbox/template; the claim and PVCs survive either way. | | Hand-applied sandbox disappeared | claim annotations, gateway logs | Idle sweeper only manages claims with `aaas.example.com/managed-by=dsh-gateway`; if it was managed, recreate/resume. | | Gateway 401 | gateway logs, cookie flags | Missing/expired cookie, DSH handshake failed, or `COOKIE_SECURE=true` over HTTP. | | Gateway 502/503 | sandbox Ready, gateway logs | Wake timeout, stale DNS, token not yet in logs; check `WAKE_TIMEOUT` and RBAC `pods/log`. | diff --git a/examples/dsh-agents-as-a-service/README.md b/examples/dsh-agents-as-a-service/README.md index 417e927..c645fbc 100644 --- a/examples/dsh-agents-as-a-service/README.md +++ b/examples/dsh-agents-as-a-service/README.md @@ -541,8 +541,8 @@ gateway remains. ## Optional hardening -The base walkthrough is deliberately portable. Four opt-in layers are documented -in [optional hardening](docs/optional-hardening.md): +The base walkthrough is deliberately portable. Four opt-in isolation layers are +documented in [optional hardening](docs/optional-hardening.md): - **gVisor** — uncomment `runtimeClassName: gvisor` before applying, or patch an existing install. Requires a `gvisor` RuntimeClass and capable nodes. @@ -554,6 +554,9 @@ in [optional hardening](docs/optional-hardening.md): `kubectl apply -f manifests/optional/network-policy/` allows only gateway→sandbox on `3080` and sandbox→DNS/public `443`. - **Provider keys** — create `dsh-provider-keys-extra` as shown in step 1. +- **Failed-Pod recovery** — uncomment `failedPodPolicy: Recreate` so the + controller replaces a `Failed` sandbox Pod (node eviction) instead of parking + the sandbox at `Finished=True`. Requires agent-sandbox v1.0.8+. Apply each independently and re-run the walkthrough. `kubectl apply -f manifests/` is non-recursive, so the optional manifest is not applied by the base step. @@ -586,6 +589,7 @@ is non-recursive, so the optional manifest is not applied by the base step. | Sandbox cannot reach a model API after applying the optional NetworkPolicy | The policy allows DNS and public TCP `443` only; a proxy, custom base URL, port `80` or non-443 endpoint needs an explicit egress rule | | Gateway cannot reach a sandbox after applying an exposure NetworkPolicy | Check that the CNI enforces NetworkPolicy and that the Traefik/oauth2-proxy pod labels match; kubelet probes may also need the node CIDR | | New provider key not visible in a sandbox | Pod env is stamped at creation: delete the warm-pool spares and recreate claimed pods. A key saved in DSH's credential store also wins over env | +| Sandbox reports `Finished=True` and every gateway request fails | The sandbox Pod reached phase `Failed` (node eviction, spot reclaim) and the default `Retain` keeps it. Delete the Pod so the controller recreates it, or set `failedPodPolicy: Recreate` (agent-sandbox v1.0.8+; see optional hardening) | ## Limitations @@ -649,4 +653,5 @@ delete the Secrets with the namespace and revoke the OAuth App in GitHub. security and RBAC, failure modes, evolution path. - [Exposure README](manifests/exposure/README.md) — the two variants in detail. - [Optional hardening](docs/optional-hardening.md) — gVisor, nodeSelector, - NetworkPolicy and extra provider keys, with verification and rollback. + NetworkPolicy, extra provider keys and failed-Pod recovery, with verification + and rollback. diff --git a/examples/dsh-agents-as-a-service/docs/architecture.md b/examples/dsh-agents-as-a-service/docs/architecture.md index 48df152..5bd5de4 100644 --- a/examples/dsh-agents-as-a-service/docs/architecture.md +++ b/examples/dsh-agents-as-a-service/docs/architecture.md @@ -301,10 +301,11 @@ Notes and limits: `dsh-provider-keys-extra`). `manifests/optional/network-policy/` adds an opt-in sandbox NetworkPolicy, and gVisor/`nodeSelector` are opt-in PodSpec fields. Add per-tenant credentials before calling it multi-tenant-grade. -- **Optional hardening is opt-in.** gVisor, node selection, the sandbox/exposure NetworkPolicies - and extra provider keys are documented in [optional hardening](optional-hardening.md). The - base manifests stay portable (no gVisor RuntimeClass, no active node labels, no - NetworkPolicy and no extra Secret required). +- **Optional hardening is opt-in.** gVisor, node selection, the sandbox/exposure NetworkPolicies, + extra provider keys and failed-Pod recovery are documented in + [optional hardening](optional-hardening.md). The base manifests stay portable (no gVisor + RuntimeClass, no active node labels, no NetworkPolicy, no extra Secret required, and the + controller's default `failedPodPolicy: Retain`). - **No gateway rate limiting.** Signup can create claims until the cluster or warm pool is exhausted; put limits at the ingress for real deployments. @@ -323,6 +324,7 @@ Notes and limits: | gVisor + `kubectl port-forward` | Forward refuses or returns nothing | Port-forward is incompatible with gVisor; use the gateway or a reachable Service | | Provider key added but missing in the pod | Requests still fail with `MISSING_CREDENTIAL` | Pod env is stamped at creation; recreate spares/pods. A stored credential wins over env | | `nodeSelector` matches no node | Pod `Pending`, `0/N nodes are available` | Fix the label, pool capacity or taint/toleration; RuntimeClass scheduling may also constrain it | +| Sandbox Pod reaches phase `Failed` (node eviction, spot reclaim) | Sandbox reports `Finished=True`; the gateway keeps failing against a dead Pod | Set `failedPodPolicy: Recreate` (agent-sandbox v1.0.8+) so the controller replaces the Pod with the same identity and PVCs; otherwise delete the Pod by hand. See the hardening doc | ## Upstream feature request: remove log scraping diff --git a/examples/dsh-agents-as-a-service/docs/optional-hardening.md b/examples/dsh-agents-as-a-service/docs/optional-hardening.md index d88501c..8a25c46 100644 --- a/examples/dsh-agents-as-a-service/docs/optional-hardening.md +++ b/examples/dsh-agents-as-a-service/docs/optional-hardening.md @@ -1,18 +1,19 @@ -# Optional hardening: gVisor, nodeSelector, NetworkPolicy and provider keys +# Optional hardening: gVisor, nodeSelector, NetworkPolicy, provider keys and failed-Pod recovery The base example is deliberately portable: it runs on kind and GKE without a gVisor runtime, a tainted node pool or a NetworkPolicy-capable CNI, and it boots with only the required `dsh-provider-keys` Secret. This document covers -four **opt-in** layers: +four **opt-in** isolation layers, plus one availability setting: -| Layer | Where it lives | What it changes | +| Setting | Where it lives | What it changes | | --- | --- | --- | | gVisor | commented `runtimeClassName` in the SandboxTemplates and Deployments | puts DSH and/or the gateway in a userspace kernel | | nodeSelector | commented `nodeSelector` in the same files | pins pods to a chosen node pool | | NetworkPolicy | `manifests/optional/network-policy/`, `manifests/exposure/*/` | limits sandbox, gateway and proxy connectivity | | Provider keys | optional `dsh-provider-keys-extra` Secret | injects extra provider API keys into every sandbox | +| Failed-Pod recovery | commented `failedPodPolicy` in the SandboxTemplates | replaces a Failed sandbox Pod instead of parking the sandbox | -Each layer is independent. Enable only what the cluster supports and test the +Each setting is independent. Enable only what the cluster supports and test the walkthrough after each change. ## 1. gVisor @@ -473,7 +474,130 @@ kubectl -n dsh-demo exec "$POD" -- sh -c ' - The optional Secret is created imperatively; the repository never contains a real key. Keep it out of Git and out of shell history where possible. -## 5. Verification summary +## 5. Failed-Pod recovery + +### What it changes + +When a sandbox Pod reaches phase `Failed`, the controller's default policy, +`Retain`, leaves the Pod in place and surfaces `Finished=True` on the Sandbox. +The Sandbox, its claim and its PVC survive, but nothing runs: the gateway keeps +proxying to a dead Pod and every request fails until someone deletes the Pod or +the claim. + +agent-sandbox v1.0.8 (the pin in [versions.env](../../../versions.env)) adds +`spec.failedPodPolicy` to `Sandbox` and to `SandboxTemplate`: + +| Value | Behavior | +| --- | --- | +| `Retain` (default) | The Failed Pod stays and the Sandbox reports `Finished=True`; the evidence stays available for debugging. | +| `Recreate` | The controller deletes the Failed Pod and starts a replacement; the Sandbox identity and its Sandbox-owned PVCs are retained. | + +Repeated failures back off like a Deployment - 5 s, doubling to 5 min between +creations - so a crash loop cannot hammer the API server. The backoff resets +once the replacement Pod has been continuously `Running` for 10 minutes, and it +lives in memory only, so a controller restart clears it. + +The base templates ship it commented out, for two reasons: + +- A deterministic failure (bad image, bad Secret, a wrong probe) would recreate + forever while hiding the cause. +- The templates use `restartPolicy: OnFailure`, so the kubelet restarts a + crashing container in place and the Pod does not reach phase `Failed` anyway. + This policy matters for terminal failures - most commonly node eviction + (`Evicted`), when a spot/preemptible node is reclaimed or the node hits memory + pressure. + +No gateway change is needed: the handshake cache is keyed by Pod UID, so a +replacement Pod re-handshakes on the next request, and the existing `401` retry +covers containers restarted in place. + +### Enable it + +```bash +# 1. The blueprint for future sandboxes. +kubectl -n dsh-demo patch sandboxtemplate dsh-web --type merge \ + -p '{"spec":{"failedPodPolicy":"Recreate"}}' + +# 2. Warm-pool spares were stamped at creation: delete them and let the pool +# replenish from the updated template. +kubectl -n dsh-demo delete sandboxes -l agents.x-k8s.io/warm-pool-sandbox + +# 3. Claimed sandboxes keep their own spec: patch them in place. +for sb in $(kubectl -n dsh-demo get sandboxes \ + -l '!agents.x-k8s.io/warm-pool-sandbox' -o jsonpath='{.items[*].metadata.name}'); do + kubectl -n dsh-demo patch sandbox "$sb" --type merge \ + -p '{"spec":{"failedPodPolicy":"Recreate"}}' +done +``` + +For a fresh install, uncomment `failedPodPolicy: Recreate` in the template +instead. + +### Verify + +```bash +SB=$(kubectl -n dsh-demo get sandboxclaim dsh-alice -o jsonpath='{.status.sandbox.name}') +kubectl -n dsh-demo get sandbox "$SB" -o jsonpath='{.spec.failedPodPolicy}{"\n"}' +# Recreate +``` + +A failure drill that never touches a user sandbox - a `Sandbox` with +`restartPolicy: Never` and a container that exits non-zero (swap the image if +Docker Hub is unreachable): + +```bash +kubectl -n dsh-demo apply -f - <<'EOF' +apiVersion: agents.x-k8s.io/v1beta1 +kind: Sandbox +metadata: + name: failed-pod-drill +spec: + failedPodPolicy: Recreate + podTemplate: + spec: + restartPolicy: Never + containers: + - name: drill + image: busybox:1.36 + command: ["sh", "-c", "exit 1"] +EOF + +kubectl -n dsh-demo get pods -w +# The drill Pod reaches phase Failed; a replacement appears within ~5 s, then +# 10 s, 20 s, ... as the container keeps exiting non-zero. +kubectl -n dsh-demo delete sandbox failed-pod-drill +``` + +Repeat the drill with `failedPodPolicy: Retain` (or omit the field) to see the +default: the Pod stays `Failed` and the Sandbox condition `Finished` becomes +`True`. + +### Caveats + +- `Recreate` deletes the Failed Pod, so its logs go with it. Capture + `kubectl get events` or switch back to `Retain` when debugging. +- The replacement is a cold start: DSH boots again and issues a new launch + token. The workspace comes back from the PVC; in-flight sessions do not. +- The field needs agent-sandbox v1.0.8+. A v1.0.7 cluster rejects it as an + unknown field. +- It applies to sandbox Pods only. The gateway and oauth2-proxy are plain + Deployments, and their ReplicaSets already replace Failed Pods. + +### Roll back + +```bash +kubectl -n dsh-demo patch sandboxtemplate dsh-web --type merge \ + -p '{"spec":{"failedPodPolicy":"Retain"}}' +for sb in $(kubectl -n dsh-demo get sandboxes -o jsonpath='{.items[*].metadata.name}'); do + kubectl -n dsh-demo patch sandbox "$sb" --type merge \ + -p '{"spec":{"failedPodPolicy":"Retain"}}' +done +``` + +Warm-pool spares pick the default up as they are claimed and replenished, or +delete them as in step 2 above. + +## 6. Verification summary Required locally for any change to this example: @@ -497,8 +621,10 @@ Manual checks, on a cluster with the relevant CNI/runtime: - Exposure policies: browser path works; a non-proxy pod cannot reach the gateway; probes stay healthy. - Provider keys: set/unset output (never the values) and the refresh procedure. +- Failed-Pod recovery: `spec.failedPodPolicy` on the Sandbox, and the drill + above replaces a `Failed` Pod while keeping the claim and PVC. -## 6. Rollback summary +## 7. Rollback summary | Layer | Rollback | | --- | --- | @@ -507,6 +633,7 @@ Manual checks, on a cluster with the relevant CNI/runtime: | gVisor | patch `runtimeClassName: null`, recreate pods | | nodeSelector | patch `nodeSelector: null`, recreate pods | | Provider keys | delete the extra Secret or a key, recreate pods | +| Failed-Pod recovery | patch `failedPodPolicy: Retain`, delete warm-pool spares | No PVC or workspace data is touched by any rollback. Deleting a `SandboxClaim` is different: it cascades to the Sandbox **and** its PVC, so do not use claim diff --git a/examples/dsh-agents-as-a-service/manifests/10-sandboxtemplate.yaml b/examples/dsh-agents-as-a-service/manifests/10-sandboxtemplate.yaml index 5210cd2..9583765 100644 --- a/examples/dsh-agents-as-a-service/manifests/10-sandboxtemplate.yaml +++ b/examples/dsh-agents-as-a-service/manifests/10-sandboxtemplate.yaml @@ -29,6 +29,15 @@ spec: # be blocked. See docs/optional-hardening.md. networkPolicyManagement: Unmanaged + # OPTIONAL: replace a Failed sandbox Pod instead of parking the sandbox. + # The default, Retain, leaves the Failed Pod in place and surfaces + # Finished=True on the Sandbox until a human intervenes. Recreate deletes the + # Failed Pod and starts a replacement with the same Sandbox identity and + # Sandbox-owned PVCs, backing off (5s doubling to 5m) on repeated failures. + # Requires agent-sandbox v1.0.8+; existing Sandboxes keep their own stamped + # policy, so patch them separately. See docs/optional-hardening.md. + # failedPodPolicy: Recreate + podTemplate: spec: # DSH Web is multi-arch (amd64 + arm64); no arch nodeSelector is needed. diff --git a/examples/dsh-agents-as-a-service/manifests/optional/README.md b/examples/dsh-agents-as-a-service/manifests/optional/README.md index 1156cfb..41fb21f 100644 --- a/examples/dsh-agents-as-a-service/manifests/optional/README.md +++ b/examples/dsh-agents-as-a-service/manifests/optional/README.md @@ -30,6 +30,9 @@ These are configured in the main manifests or with an extra Secret: - **gVisor:** uncomment `runtimeClassName: gvisor` in the SandboxTemplate and the Deployments before applying, or patch an existing install. - **nodeSelector:** uncomment the field and replace the placeholder node label. +- **Failed-Pod recovery:** uncomment `failedPodPolicy: Recreate` in the + SandboxTemplate (agent-sandbox v1.0.8+) so a Failed Pod is replaced instead + of parked. - **Provider keys:** create the optional `dsh-provider-keys-extra` Secret; the base `dsh-provider-keys` Secret remains the one required key. diff --git a/examples/minimal-single-tenant/manifests/sandbox-template.yaml b/examples/minimal-single-tenant/manifests/sandbox-template.yaml index 98512c1..53ee509 100644 --- a/examples/minimal-single-tenant/manifests/sandbox-template.yaml +++ b/examples/minimal-single-tenant/manifests/sandbox-template.yaml @@ -26,6 +26,10 @@ spec: # quickstart reaches the sandbox through `kubectl port-forward`, which is # local to the pod. The multi-tenant example documents opt-in policies in # examples/dsh-agents-as-a-service/docs/optional-hardening.md. + # + # OPTIONAL: recreate a Failed sandbox Pod instead of retaining it + # (agent-sandbox v1.0.8+). Documented with the multi-tenant example. + # failedPodPolicy: Recreate podTemplate: spec: # OPTIONAL: run under gVisor. gVisor breaks `kubectl port-forward`, so