Fix DSH crash-loop after suspend/resume (fsGroup chmod) (#5)

Resuming a sandbox recreates its pod on the same PVC. The kubelet's default
fsGroup policy ("Always") recursively ORs 0660 into every file on each pod
start, so DSH's .credentials.yaml — created 0600 — comes back as 0660. The
credentials plugin refuses to boot in that case:

  credentials-local: .../.credentials.yaml is readable beyond its owner
  (mode 660); run "chmod 600 ..."

That makes the first resume after a suspend crash-loop, which breaks the
example's step 6 and any gateway wake-on-connect.

Set fsGroupChangePolicy: OnRootMismatch on both sandbox templates. The
kubelet then skips the recursive walk when the volume root already carries
the fsGroup ownership/mode applied on the first mount — exactly the
suspend/resume case, while a fresh PVC is still chowned correctly.

Document the upgrade path: Sandbox objects snapshot their pod template, so
existing sandboxes and warm-pool members need the policy patched (plus a
one-time chmod 600) or must be recreated.

Co-authored-by: SREDevOpsAgents <agents@sredevops.org>

deploy/k8s/sandbox-template.yaml

@@ -24,6 +24,15 @@ spec:
         runAsUser: 1000 # Matches the UID of the 'node' user
         runAsGroup: 1000
         fsGroup: 1000
+        # DSH's credentials plugin refuses to start when .credentials.yaml has
+        # any group/other permission bit (it creates the file 0600). The
+        # default kubelet fsGroup policy ("Always") recursively ORs 0660 into
+        # every file each time the pod starts, so a recreated/resumed sandbox
+        # comes back with 0660 and crash-loops. OnRootMismatch skips that walk
+        # once the volume root already has the fsGroup ownership/mode from the
+        # first mount. Keep in sync with
+        # examples/dsh-agents-as-a-service/10-sandboxtemplate.yaml.
+        fsGroupChangePolicy: OnRootMismatch
       containers:
         - name: dsh-web
           image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.2

docs/architecture.md

@@ -140,6 +140,15 @@ sweeper skips everything else, so it can never suspend a sandbox an operator is
 that marker the sweeper would suspend a hand-applied claim 60s after first sighting and break
 `kubectl logs`/`port-forward` against its pod.
 
+**Resume and file ownership.** The pod template sets `fsGroupChangePolicy: OnRootMismatch` on
+purpose. DSH's credentials plugin refuses to boot when `.credentials.yaml` has any group/other
+permission bit (it creates the file `0600`). The kubelet's default fsGroup policy (`Always`)
+recursively ORs `0660` into every file on each pod start, so the first resume of a sandbox would
+crash-loop with `mode 660`. `OnRootMismatch` only rewrites ownership/permissions when the volume
+root does not already match the fsGroup, which it does after the first mount — so suspend/resume
+keeps working. Sandboxes snapshot their pod template, so an updated `SandboxTemplate` does not
+retroactively change existing ones.
+
 ### WebSocket
 
 ```mermaid

examples/dsh-agents-as-a-service/10-sandboxtemplate.yaml

@@ -34,6 +34,15 @@ spec:
         runAsGroup: 1000
         fsGroup: 1000
         runAsNonRoot: true
+        # DSH's credentials plugin refuses to start when .credentials.yaml has
+        # any group/other permission bit (it creates the file 0600). The
+        # default kubelet fsGroup policy ("Always") recursively ORs 0660 into
+        # every file each time the pod starts, so a resumed sandbox comes back
+        # with 0660 and crash-loops before the gateway can proxy to it.
+        # OnRootMismatch skips that walk once the volume root already carries
+        # the fsGroup ownership/mode applied on the first mount.
+        # Keep in sync with deploy/k8s/sandbox-template.yaml.
+        fsGroupChangePolicy: OnRootMismatch
       containers:
         - name: dsh-web
           image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.2

examples/dsh-agents-as-a-service/README.md

@@ -159,6 +159,12 @@ kubectl -n dsh-demo wait sandbox "$SB" --for=condition=Ready --timeout=180s
 kubectl -n dsh-demo exec "$SB" -- cat /home/node/.dsh/marker.txt   # "remember me"
 ```
 
+> **Why resume works:** the pod template sets `fsGroupChangePolicy: OnRootMismatch`. DSH requires
+> its `.credentials.yaml` to be `0600`, while the kubelet's default fsGroup policy widens every
+> file to `0660` on each pod start — a resumed sandbox would crash-loop. See
+> [Troubleshooting](#troubleshooting) for the one-time fix on sandboxes created before this
+> setting existed.
+
 ## 7. Injection policy rejects env-carrying claims
 
 ```bash
@@ -194,6 +200,11 @@ kubectl -n dsh-demo port-forward svc/dsh-gateway 18080:8080
 > first. Cluster-specific fields such as `nodeAffinity` and the warm-pool replica count are
 > intentionally not in the repo manifests; preserve them with a targeted patch instead of a
 > blind apply. Moving the probes to `/favicon.svg` is the crash-loop fix.
+>
+> The template also sets `fsGroupChangePolicy: OnRootMismatch` (see the note in step 6).
+> Existing `Sandbox` objects snapshot their pod template, so a template apply does not update
+> them: patch each sandbox's `spec.podTemplate.spec.securityContext.fsGroupChangePolicy` and
+> `chmod 600` its `.credentials.yaml`, or recreate it.
 
 ### Browser flow
 
@@ -314,6 +325,7 @@ redirects `/u/<user>` to the trailing-slash form with `308` so relative URLs res
 | Signup is slow / `202` then `Waking` | Warm pool exhausted (`kubectl get sandboxes -l agents.x-k8s.io/warm-pool-sandbox`); scale the pool |
 | `403` from the Kubernetes API in gateway logs | The gateway ServiceAccount is missing claim/sandbox/pod-log verbs (see `50-gateway.yaml`) |
 | Step 3: `pod "$SB" not found` after deploying the gateway | The claim was suspended by the idle sweeper. Claims created by the gateway carry `aaas.example.com/managed-by=dsh-gateway`; hand-applied claims (steps 1–7) do not and are never swept. Upgrade the gateway to a build that honors the marker, or resume the sandbox: `kubectl -n dsh-demo patch sandbox "$SB" --type merge -p '{"spec":{"operatingMode":"Running"}}'` |
+| Resume crash-loops: `credentials-local: … is readable beyond its owner (mode 660)` | The pod template lacks `fsGroupChangePolicy: OnRootMismatch`, so the kubelet's default fsGroup policy rewrote `.credentials.yaml` to `0660`; DSH requires `0600` and refuses to start. New installs get the policy from the template. For an existing sandbox: suspend it, run a throwaway pod on `dsh-home-$SB` to `chmod 600 /home/node/.dsh/.credentials.yaml`, patch the sandbox's `podTemplate` with the policy, then resume. |
 
 ## Limitations