diff --git a/deploy/k8s/sandbox-template.yaml b/deploy/k8s/sandbox-template.yaml index 6b63433..3954cef 100644 --- a/deploy/k8s/sandbox-template.yaml +++ b/deploy/k8s/sandbox-template.yaml @@ -24,6 +24,15 @@ spec: runAsUser: 1000 # Matches the UID of the 'node' user runAsGroup: 1000 fsGroup: 1000 + # DSH's credentials plugin refuses to start when .credentials.yaml has + # any group/other permission bit (it creates the file 0600). The + # default kubelet fsGroup policy ("Always") recursively ORs 0660 into + # every file each time the pod starts, so a recreated/resumed sandbox + # comes back with 0660 and crash-loops. OnRootMismatch skips that walk + # once the volume root already has the fsGroup ownership/mode from the + # first mount. Keep in sync with + # examples/dsh-agents-as-a-service/10-sandboxtemplate.yaml. + fsGroupChangePolicy: OnRootMismatch containers: - name: dsh-web image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.2 diff --git a/docs/architecture.md b/docs/architecture.md index b949c62..654af9b 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -140,6 +140,15 @@ sweeper skips everything else, so it can never suspend a sandbox an operator is that marker the sweeper would suspend a hand-applied claim 60s after first sighting and break `kubectl logs`/`port-forward` against its pod. +**Resume and file ownership.** The pod template sets `fsGroupChangePolicy: OnRootMismatch` on +purpose. DSH's credentials plugin refuses to boot when `.credentials.yaml` has any group/other +permission bit (it creates the file `0600`). The kubelet's default fsGroup policy (`Always`) +recursively ORs `0660` into every file on each pod start, so the first resume of a sandbox would +crash-loop with `mode 660`. `OnRootMismatch` only rewrites ownership/permissions when the volume +root does not already match the fsGroup, which it does after the first mount — so suspend/resume +keeps working. Sandboxes snapshot their pod template, so an updated `SandboxTemplate` does not +retroactively change existing ones. + ### WebSocket ```mermaid diff --git a/examples/dsh-agents-as-a-service/10-sandboxtemplate.yaml b/examples/dsh-agents-as-a-service/10-sandboxtemplate.yaml index dadc541..cebaa46 100644 --- a/examples/dsh-agents-as-a-service/10-sandboxtemplate.yaml +++ b/examples/dsh-agents-as-a-service/10-sandboxtemplate.yaml @@ -34,6 +34,15 @@ spec: runAsGroup: 1000 fsGroup: 1000 runAsNonRoot: true + # DSH's credentials plugin refuses to start when .credentials.yaml has + # any group/other permission bit (it creates the file 0600). The + # default kubelet fsGroup policy ("Always") recursively ORs 0660 into + # every file each time the pod starts, so a resumed sandbox comes back + # with 0660 and crash-loops before the gateway can proxy to it. + # OnRootMismatch skips that walk once the volume root already carries + # the fsGroup ownership/mode applied on the first mount. + # Keep in sync with deploy/k8s/sandbox-template.yaml. + fsGroupChangePolicy: OnRootMismatch containers: - name: dsh-web image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.2 diff --git a/examples/dsh-agents-as-a-service/README.md b/examples/dsh-agents-as-a-service/README.md index d823d61..c24bf3e 100644 --- a/examples/dsh-agents-as-a-service/README.md +++ b/examples/dsh-agents-as-a-service/README.md @@ -159,6 +159,12 @@ kubectl -n dsh-demo wait sandbox "$SB" --for=condition=Ready --timeout=180s kubectl -n dsh-demo exec "$SB" -- cat /home/node/.dsh/marker.txt # "remember me" ``` +> **Why resume works:** the pod template sets `fsGroupChangePolicy: OnRootMismatch`. DSH requires +> its `.credentials.yaml` to be `0600`, while the kubelet's default fsGroup policy widens every +> file to `0660` on each pod start — a resumed sandbox would crash-loop. See +> [Troubleshooting](#troubleshooting) for the one-time fix on sandboxes created before this +> setting existed. + ## 7. Injection policy rejects env-carrying claims ```bash @@ -194,6 +200,11 @@ kubectl -n dsh-demo port-forward svc/dsh-gateway 18080:8080 > first. Cluster-specific fields such as `nodeAffinity` and the warm-pool replica count are > intentionally not in the repo manifests; preserve them with a targeted patch instead of a > blind apply. Moving the probes to `/favicon.svg` is the crash-loop fix. +> +> The template also sets `fsGroupChangePolicy: OnRootMismatch` (see the note in step 6). +> Existing `Sandbox` objects snapshot their pod template, so a template apply does not update +> them: patch each sandbox's `spec.podTemplate.spec.securityContext.fsGroupChangePolicy` and +> `chmod 600` its `.credentials.yaml`, or recreate it. ### Browser flow @@ -314,6 +325,7 @@ redirects `/u/` to the trailing-slash form with `308` so relative URLs res | Signup is slow / `202` then `Waking` | Warm pool exhausted (`kubectl get sandboxes -l agents.x-k8s.io/warm-pool-sandbox`); scale the pool | | `403` from the Kubernetes API in gateway logs | The gateway ServiceAccount is missing claim/sandbox/pod-log verbs (see `50-gateway.yaml`) | | Step 3: `pod "$SB" not found` after deploying the gateway | The claim was suspended by the idle sweeper. Claims created by the gateway carry `aaas.example.com/managed-by=dsh-gateway`; hand-applied claims (steps 1–7) do not and are never swept. Upgrade the gateway to a build that honors the marker, or resume the sandbox: `kubectl -n dsh-demo patch sandbox "$SB" --type merge -p '{"spec":{"operatingMode":"Running"}}'` | +| Resume crash-loops: `credentials-local: … is readable beyond its owner (mode 660)` | The pod template lacks `fsGroupChangePolicy: OnRootMismatch`, so the kubelet's default fsGroup policy rewrote `.credentials.yaml` to `0660`; DSH requires `0600` and refuses to start. New installs get the policy from the template. For an existing sandbox: suspend it, run a throwaway pod on `dsh-home-$SB` to `chmod 600 /home/node/.dsh/.credentials.yaml`, patch the sandbox's `podTemplate` with the policy, then resume. | ## Limitations