mirror of
https://github.com/sredevopsorg/deepseek-kubernetes-sandbox.git
synced 2026-11-03 01:13:24 +00:00
feat(example): add optional failed-Pod recovery (#26)
agent-sandbox v1.0.8 adds spec.failedPodPolicy to Sandbox and SandboxTemplate. The default, Retain, leaves a Failed Pod in place and surfaces Finished=True; Recreate deletes it and starts a replacement with the same Sandbox identity and PVCs. Ship it as an opt-in toggle next to the other commented fields, with a new section 5 in the optional-hardening doc: what it changes, how to enable it for new, warm-pool and already-claimed Sandboxes (the policy is stamped at creation), a failure drill, caveats and rollback. The base walkthrough stays on Retain because a deterministic failure would otherwise recreate forever, and because the templates use restartPolicy OnFailure, so the policy only matters for terminal cases such as node eviction. Cross-references updated in the example README, the optional-manifests index, the architecture failure-mode table and the platform skill's troubleshooting table. Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
This commit is contained in:
SREDevOps.org Agent
committed by
GitHub
GitHub
parent
96d3030a49
commit
14c3fb840e
@@ -20,6 +20,7 @@ events, then compare desired vs observed state. Keep output bounded. Never
|
||||
| Restart with mode-660 error | logs mention `.credentials.yaml` | kubelet fsGroup widened the file; suspend, `chmod 600` from a throwaway pod, patch `fsGroupChangePolicy: OnRootMismatch`, resume. |
|
||||
| PVC `Pending` | `kubectl describe pvc`, StorageClass | No default provisioner, access mode, zone binding. |
|
||||
| Launch token not found | `kubectl logs <pod> --tail=2000 | grep 'dsh web:'` | Pod not ready yet, or logs rotated. Do not guess tokens; wait for readiness. |
|
||||
| Sandbox reports `Finished=True` | pod phase, `kubectl get events` | The backing Pod reached `Failed` (eviction, spot reclaim) and `failedPodPolicy` defaults to `Retain`. Delete the Pod to force a replacement, or set `Recreate` (v1.0.8+) on the Sandbox/template; the claim and PVCs survive either way. |
|
||||
| Hand-applied sandbox disappeared | claim annotations, gateway logs | Idle sweeper only manages claims with `aaas.example.com/managed-by=dsh-gateway`; if it was managed, recreate/resume. |
|
||||
| Gateway 401 | gateway logs, cookie flags | Missing/expired cookie, DSH handshake failed, or `COOKIE_SECURE=true` over HTTP. |
|
||||
| Gateway 502/503 | sandbox Ready, gateway logs | Wake timeout, stale DNS, token not yet in logs; check `WAKE_TIMEOUT` and RBAC `pods/log`. |
|
||||
|
||||
@@ -541,8 +541,8 @@ gateway remains.
|
||||
|
||||
## Optional hardening
|
||||
|
||||
The base walkthrough is deliberately portable. Four opt-in layers are documented
|
||||
in [optional hardening](docs/optional-hardening.md):
|
||||
The base walkthrough is deliberately portable. Four opt-in isolation layers are
|
||||
documented in [optional hardening](docs/optional-hardening.md):
|
||||
|
||||
- **gVisor** — uncomment `runtimeClassName: gvisor` before applying, or patch an
|
||||
existing install. Requires a `gvisor` RuntimeClass and capable nodes.
|
||||
@@ -554,6 +554,9 @@ in [optional hardening](docs/optional-hardening.md):
|
||||
`kubectl apply -f manifests/optional/network-policy/` allows only
|
||||
gateway→sandbox on `3080` and sandbox→DNS/public `443`.
|
||||
- **Provider keys** — create `dsh-provider-keys-extra` as shown in step 1.
|
||||
- **Failed-Pod recovery** — uncomment `failedPodPolicy: Recreate` so the
|
||||
controller replaces a `Failed` sandbox Pod (node eviction) instead of parking
|
||||
the sandbox at `Finished=True`. Requires agent-sandbox v1.0.8+.
|
||||
|
||||
Apply each independently and re-run the walkthrough. `kubectl apply -f manifests/`
|
||||
is non-recursive, so the optional manifest is not applied by the base step.
|
||||
@@ -586,6 +589,7 @@ is non-recursive, so the optional manifest is not applied by the base step.
|
||||
| Sandbox cannot reach a model API after applying the optional NetworkPolicy | The policy allows DNS and public TCP `443` only; a proxy, custom base URL, port `80` or non-443 endpoint needs an explicit egress rule |
|
||||
| Gateway cannot reach a sandbox after applying an exposure NetworkPolicy | Check that the CNI enforces NetworkPolicy and that the Traefik/oauth2-proxy pod labels match; kubelet probes may also need the node CIDR |
|
||||
| New provider key not visible in a sandbox | Pod env is stamped at creation: delete the warm-pool spares and recreate claimed pods. A key saved in DSH's credential store also wins over env |
|
||||
| Sandbox reports `Finished=True` and every gateway request fails | The sandbox Pod reached phase `Failed` (node eviction, spot reclaim) and the default `Retain` keeps it. Delete the Pod so the controller recreates it, or set `failedPodPolicy: Recreate` (agent-sandbox v1.0.8+; see optional hardening) |
|
||||
|
||||
## Limitations
|
||||
|
||||
@@ -649,4 +653,5 @@ delete the Secrets with the namespace and revoke the OAuth App in GitHub.
|
||||
security and RBAC, failure modes, evolution path.
|
||||
- [Exposure README](manifests/exposure/README.md) — the two variants in detail.
|
||||
- [Optional hardening](docs/optional-hardening.md) — gVisor, nodeSelector,
|
||||
NetworkPolicy and extra provider keys, with verification and rollback.
|
||||
NetworkPolicy, extra provider keys and failed-Pod recovery, with verification
|
||||
and rollback.
|
||||
|
||||
@@ -301,10 +301,11 @@ Notes and limits:
|
||||
`dsh-provider-keys-extra`). `manifests/optional/network-policy/` adds an opt-in sandbox
|
||||
NetworkPolicy, and gVisor/`nodeSelector` are opt-in PodSpec fields. Add per-tenant
|
||||
credentials before calling it multi-tenant-grade.
|
||||
- **Optional hardening is opt-in.** gVisor, node selection, the sandbox/exposure NetworkPolicies
|
||||
and extra provider keys are documented in [optional hardening](optional-hardening.md). The
|
||||
base manifests stay portable (no gVisor RuntimeClass, no active node labels, no
|
||||
NetworkPolicy and no extra Secret required).
|
||||
- **Optional hardening is opt-in.** gVisor, node selection, the sandbox/exposure NetworkPolicies,
|
||||
extra provider keys and failed-Pod recovery are documented in
|
||||
[optional hardening](optional-hardening.md). The base manifests stay portable (no gVisor
|
||||
RuntimeClass, no active node labels, no NetworkPolicy, no extra Secret required, and the
|
||||
controller's default `failedPodPolicy: Retain`).
|
||||
- **No gateway rate limiting.** Signup can create claims until the cluster or warm pool is
|
||||
exhausted; put limits at the ingress for real deployments.
|
||||
|
||||
@@ -323,6 +324,7 @@ Notes and limits:
|
||||
| gVisor + `kubectl port-forward` | Forward refuses or returns nothing | Port-forward is incompatible with gVisor; use the gateway or a reachable Service |
|
||||
| Provider key added but missing in the pod | Requests still fail with `MISSING_CREDENTIAL` | Pod env is stamped at creation; recreate spares/pods. A stored credential wins over env |
|
||||
| `nodeSelector` matches no node | Pod `Pending`, `0/N nodes are available` | Fix the label, pool capacity or taint/toleration; RuntimeClass scheduling may also constrain it |
|
||||
| Sandbox Pod reaches phase `Failed` (node eviction, spot reclaim) | Sandbox reports `Finished=True`; the gateway keeps failing against a dead Pod | Set `failedPodPolicy: Recreate` (agent-sandbox v1.0.8+) so the controller replaces the Pod with the same identity and PVCs; otherwise delete the Pod by hand. See the hardening doc |
|
||||
|
||||
## Upstream feature request: remove log scraping
|
||||
|
||||
|
||||
@@ -1,18 +1,19 @@
|
||||
# Optional hardening: gVisor, nodeSelector, NetworkPolicy and provider keys
|
||||
# Optional hardening: gVisor, nodeSelector, NetworkPolicy, provider keys and failed-Pod recovery
|
||||
|
||||
The base example is deliberately portable: it runs on kind and GKE without a
|
||||
gVisor runtime, a tainted node pool or a NetworkPolicy-capable CNI, and it
|
||||
boots with only the required `dsh-provider-keys` Secret. This document covers
|
||||
four **opt-in** layers:
|
||||
four **opt-in** isolation layers, plus one availability setting:
|
||||
|
||||
| Layer | Where it lives | What it changes |
|
||||
| Setting | Where it lives | What it changes |
|
||||
| --- | --- | --- |
|
||||
| gVisor | commented `runtimeClassName` in the SandboxTemplates and Deployments | puts DSH and/or the gateway in a userspace kernel |
|
||||
| nodeSelector | commented `nodeSelector` in the same files | pins pods to a chosen node pool |
|
||||
| NetworkPolicy | `manifests/optional/network-policy/`, `manifests/exposure/*/` | limits sandbox, gateway and proxy connectivity |
|
||||
| Provider keys | optional `dsh-provider-keys-extra` Secret | injects extra provider API keys into every sandbox |
|
||||
| Failed-Pod recovery | commented `failedPodPolicy` in the SandboxTemplates | replaces a Failed sandbox Pod instead of parking the sandbox |
|
||||
|
||||
Each layer is independent. Enable only what the cluster supports and test the
|
||||
Each setting is independent. Enable only what the cluster supports and test the
|
||||
walkthrough after each change.
|
||||
|
||||
## 1. gVisor
|
||||
@@ -473,7 +474,130 @@ kubectl -n dsh-demo exec "$POD" -- sh -c '
|
||||
- The optional Secret is created imperatively; the repository never contains a
|
||||
real key. Keep it out of Git and out of shell history where possible.
|
||||
|
||||
## 5. Verification summary
|
||||
## 5. Failed-Pod recovery
|
||||
|
||||
### What it changes
|
||||
|
||||
When a sandbox Pod reaches phase `Failed`, the controller's default policy,
|
||||
`Retain`, leaves the Pod in place and surfaces `Finished=True` on the Sandbox.
|
||||
The Sandbox, its claim and its PVC survive, but nothing runs: the gateway keeps
|
||||
proxying to a dead Pod and every request fails until someone deletes the Pod or
|
||||
the claim.
|
||||
|
||||
agent-sandbox v1.0.8 (the pin in [versions.env](../../../versions.env)) adds
|
||||
`spec.failedPodPolicy` to `Sandbox` and to `SandboxTemplate`:
|
||||
|
||||
| Value | Behavior |
|
||||
| --- | --- |
|
||||
| `Retain` (default) | The Failed Pod stays and the Sandbox reports `Finished=True`; the evidence stays available for debugging. |
|
||||
| `Recreate` | The controller deletes the Failed Pod and starts a replacement; the Sandbox identity and its Sandbox-owned PVCs are retained. |
|
||||
|
||||
Repeated failures back off like a Deployment - 5 s, doubling to 5 min between
|
||||
creations - so a crash loop cannot hammer the API server. The backoff resets
|
||||
once the replacement Pod has been continuously `Running` for 10 minutes, and it
|
||||
lives in memory only, so a controller restart clears it.
|
||||
|
||||
The base templates ship it commented out, for two reasons:
|
||||
|
||||
- A deterministic failure (bad image, bad Secret, a wrong probe) would recreate
|
||||
forever while hiding the cause.
|
||||
- The templates use `restartPolicy: OnFailure`, so the kubelet restarts a
|
||||
crashing container in place and the Pod does not reach phase `Failed` anyway.
|
||||
This policy matters for terminal failures - most commonly node eviction
|
||||
(`Evicted`), when a spot/preemptible node is reclaimed or the node hits memory
|
||||
pressure.
|
||||
|
||||
No gateway change is needed: the handshake cache is keyed by Pod UID, so a
|
||||
replacement Pod re-handshakes on the next request, and the existing `401` retry
|
||||
covers containers restarted in place.
|
||||
|
||||
### Enable it
|
||||
|
||||
```bash
|
||||
# 1. The blueprint for future sandboxes.
|
||||
kubectl -n dsh-demo patch sandboxtemplate dsh-web --type merge \
|
||||
-p '{"spec":{"failedPodPolicy":"Recreate"}}'
|
||||
|
||||
# 2. Warm-pool spares were stamped at creation: delete them and let the pool
|
||||
# replenish from the updated template.
|
||||
kubectl -n dsh-demo delete sandboxes -l agents.x-k8s.io/warm-pool-sandbox
|
||||
|
||||
# 3. Claimed sandboxes keep their own spec: patch them in place.
|
||||
for sb in $(kubectl -n dsh-demo get sandboxes \
|
||||
-l '!agents.x-k8s.io/warm-pool-sandbox' -o jsonpath='{.items[*].metadata.name}'); do
|
||||
kubectl -n dsh-demo patch sandbox "$sb" --type merge \
|
||||
-p '{"spec":{"failedPodPolicy":"Recreate"}}'
|
||||
done
|
||||
```
|
||||
|
||||
For a fresh install, uncomment `failedPodPolicy: Recreate` in the template
|
||||
instead.
|
||||
|
||||
### Verify
|
||||
|
||||
```bash
|
||||
SB=$(kubectl -n dsh-demo get sandboxclaim dsh-alice -o jsonpath='{.status.sandbox.name}')
|
||||
kubectl -n dsh-demo get sandbox "$SB" -o jsonpath='{.spec.failedPodPolicy}{"\n"}'
|
||||
# Recreate
|
||||
```
|
||||
|
||||
A failure drill that never touches a user sandbox - a `Sandbox` with
|
||||
`restartPolicy: Never` and a container that exits non-zero (swap the image if
|
||||
Docker Hub is unreachable):
|
||||
|
||||
```bash
|
||||
kubectl -n dsh-demo apply -f - <<'EOF'
|
||||
apiVersion: agents.x-k8s.io/v1beta1
|
||||
kind: Sandbox
|
||||
metadata:
|
||||
name: failed-pod-drill
|
||||
spec:
|
||||
failedPodPolicy: Recreate
|
||||
podTemplate:
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
containers:
|
||||
- name: drill
|
||||
image: busybox:1.36
|
||||
command: ["sh", "-c", "exit 1"]
|
||||
EOF
|
||||
|
||||
kubectl -n dsh-demo get pods -w
|
||||
# The drill Pod reaches phase Failed; a replacement appears within ~5 s, then
|
||||
# 10 s, 20 s, ... as the container keeps exiting non-zero.
|
||||
kubectl -n dsh-demo delete sandbox failed-pod-drill
|
||||
```
|
||||
|
||||
Repeat the drill with `failedPodPolicy: Retain` (or omit the field) to see the
|
||||
default: the Pod stays `Failed` and the Sandbox condition `Finished` becomes
|
||||
`True`.
|
||||
|
||||
### Caveats
|
||||
|
||||
- `Recreate` deletes the Failed Pod, so its logs go with it. Capture
|
||||
`kubectl get events` or switch back to `Retain` when debugging.
|
||||
- The replacement is a cold start: DSH boots again and issues a new launch
|
||||
token. The workspace comes back from the PVC; in-flight sessions do not.
|
||||
- The field needs agent-sandbox v1.0.8+. A v1.0.7 cluster rejects it as an
|
||||
unknown field.
|
||||
- It applies to sandbox Pods only. The gateway and oauth2-proxy are plain
|
||||
Deployments, and their ReplicaSets already replace Failed Pods.
|
||||
|
||||
### Roll back
|
||||
|
||||
```bash
|
||||
kubectl -n dsh-demo patch sandboxtemplate dsh-web --type merge \
|
||||
-p '{"spec":{"failedPodPolicy":"Retain"}}'
|
||||
for sb in $(kubectl -n dsh-demo get sandboxes -o jsonpath='{.items[*].metadata.name}'); do
|
||||
kubectl -n dsh-demo patch sandbox "$sb" --type merge \
|
||||
-p '{"spec":{"failedPodPolicy":"Retain"}}'
|
||||
done
|
||||
```
|
||||
|
||||
Warm-pool spares pick the default up as they are claimed and replenished, or
|
||||
delete them as in step 2 above.
|
||||
|
||||
## 6. Verification summary
|
||||
|
||||
Required locally for any change to this example:
|
||||
|
||||
@@ -497,8 +621,10 @@ Manual checks, on a cluster with the relevant CNI/runtime:
|
||||
- Exposure policies: browser path works; a non-proxy pod cannot reach the
|
||||
gateway; probes stay healthy.
|
||||
- Provider keys: set/unset output (never the values) and the refresh procedure.
|
||||
- Failed-Pod recovery: `spec.failedPodPolicy` on the Sandbox, and the drill
|
||||
above replaces a `Failed` Pod while keeping the claim and PVC.
|
||||
|
||||
## 6. Rollback summary
|
||||
## 7. Rollback summary
|
||||
|
||||
| Layer | Rollback |
|
||||
| --- | --- |
|
||||
@@ -507,6 +633,7 @@ Manual checks, on a cluster with the relevant CNI/runtime:
|
||||
| gVisor | patch `runtimeClassName: null`, recreate pods |
|
||||
| nodeSelector | patch `nodeSelector: null`, recreate pods |
|
||||
| Provider keys | delete the extra Secret or a key, recreate pods |
|
||||
| Failed-Pod recovery | patch `failedPodPolicy: Retain`, delete warm-pool spares |
|
||||
|
||||
No PVC or workspace data is touched by any rollback. Deleting a `SandboxClaim`
|
||||
is different: it cascades to the Sandbox **and** its PVC, so do not use claim
|
||||
|
||||
@@ -29,6 +29,15 @@ spec:
|
||||
# be blocked. See docs/optional-hardening.md.
|
||||
networkPolicyManagement: Unmanaged
|
||||
|
||||
# OPTIONAL: replace a Failed sandbox Pod instead of parking the sandbox.
|
||||
# The default, Retain, leaves the Failed Pod in place and surfaces
|
||||
# Finished=True on the Sandbox until a human intervenes. Recreate deletes the
|
||||
# Failed Pod and starts a replacement with the same Sandbox identity and
|
||||
# Sandbox-owned PVCs, backing off (5s doubling to 5m) on repeated failures.
|
||||
# Requires agent-sandbox v1.0.8+; existing Sandboxes keep their own stamped
|
||||
# policy, so patch them separately. See docs/optional-hardening.md.
|
||||
# failedPodPolicy: Recreate
|
||||
|
||||
podTemplate:
|
||||
spec:
|
||||
# DSH Web is multi-arch (amd64 + arm64); no arch nodeSelector is needed.
|
||||
|
||||
@@ -30,6 +30,9 @@ These are configured in the main manifests or with an extra Secret:
|
||||
- **gVisor:** uncomment `runtimeClassName: gvisor` in the SandboxTemplate and
|
||||
the Deployments before applying, or patch an existing install.
|
||||
- **nodeSelector:** uncomment the field and replace the placeholder node label.
|
||||
- **Failed-Pod recovery:** uncomment `failedPodPolicy: Recreate` in the
|
||||
SandboxTemplate (agent-sandbox v1.0.8+) so a Failed Pod is replaced instead
|
||||
of parked.
|
||||
- **Provider keys:** create the optional `dsh-provider-keys-extra` Secret; the
|
||||
base `dsh-provider-keys` Secret remains the one required key.
|
||||
|
||||
|
||||
@@ -26,6 +26,10 @@ spec:
|
||||
# quickstart reaches the sandbox through `kubectl port-forward`, which is
|
||||
# local to the pod. The multi-tenant example documents opt-in policies in
|
||||
# examples/dsh-agents-as-a-service/docs/optional-hardening.md.
|
||||
#
|
||||
# OPTIONAL: recreate a Failed sandbox Pod instead of retaining it
|
||||
# (agent-sandbox v1.0.8+). Documented with the multi-tenant example.
|
||||
# failedPodPolicy: Recreate
|
||||
podTemplate:
|
||||
spec:
|
||||
# OPTIONAL: run under gVisor. gVisor breaks `kubectl port-forward`, so
|
||||
|
||||
Reference in New Issue
Block a user