feat(example): persist the agent workspace on the PVC via subPath (#34)

/workspace - the DSH image's WORKDIR, where agents keep files - was plain
container storage, so every suspend/resume or pod replacement silently threw
agent work away while DSH state survived on the PVC.

Mount the same per-sandbox `dsh-home` PVC a second time at /workspace with
`subPath: workspace`. One PVC, no extra provisioning, and the subPath keeps
DSH state (sessions, config and .credentials.yaml at the volume root) out of
the agent-visible tree; mounting the volume root at /workspace would have
exposed the credential file.

The kubelet creates the missing subPath directory itself and chmods it to the
volume's fsGroup permissions, and the volume root's setgid bit makes the new
directory inherit gid 1000, so UID 1000 can write with no init container and
no root. The rare interrupted-kubelet case that leaves 0755 is documented with
a recovery procedure (kubernetes#142464) rather than papered over.

- e2e-kind: write a /workspace marker alongside the DSH_HOME marker, assert
  the mount is the dsh-home subPath, and verify both survive suspend/resume.
- Docs: README (overview, steps 4-6, troubleshooting row), architecture
  (PVC diagram, persistence, component table), main-image working-directory
  row, platform runbook PVC row.

Verified: make check (14 manifests), make lint, plus YAML assertions on the
two mounts and the unchanged volumeClaimTemplates.

Co-authored-by: SREDevOpsAgents <agents@sredevops.org>

examples/dsh-agents-as-a-service/README.md

@@ -6,8 +6,8 @@ platform pattern into pure agent-sandbox resources + kubectl, running **DeepSeek
 Harness Web (DSH)** instead of Hermes. Every user of such a platform gets a
 personal, long-lived DSH Web instance that is **provisioned in ~2 seconds from a
 warm pool, suspended when idle to save cost, and woken transparently when the
-user returns** — with all state (conversations, sessions, config) surviving on a
-per-user PVC at `/home/node/.dsh`.
+user returns** — with all state (conversations, sessions, config, and the agent
+workspace) surviving on the per-user PVC at `/home/node/.dsh` + `/workspace`.
 
 It ships in three layers:
 
@@ -46,7 +46,7 @@ What it demonstrates, feature by feature:
 
 | agent-sandbox feature | Role in the platform |
 | --- | --- |
-| `SandboxTemplate` with `volumeClaimTemplates` | Identical DSH pods; per-user state on a PVC at `/home/node/.dsh` |
+| `SandboxTemplate` with `volumeClaimTemplates` | Identical DSH pods; per-user state at `/home/node/.dsh` and the agent workspace at `/workspace`, on one PVC (the workspace is a `subPath` mount, so DSH state stays private) |
 | `envVarsInjectionPolicy` / `volumeClaimTemplatesPolicy: Disallowed` | Claims that would bypass the warm pool are **rejected loudly**, not cold-started silently |
 | `service: true` | Stable per-sandbox DNS (`status.serviceFQDN`) for the gateway to route on |
 | `SandboxWarmPool` | Signup latency is pre-paid: image pull, PVC provision, DSH boot all happen before any user exists |
@@ -202,8 +202,14 @@ kubectl -n dsh-demo port-forward "pod/$SB" 18081:3080
 
 ```bash
 kubectl -n dsh-demo exec "$SB" -- sh -c 'echo "remember me" > /home/node/.dsh/marker.txt'
+kubectl -n dsh-demo exec "$SB" -- sh -c 'echo "remember me too" > /workspace/marker.txt'
 ```
 
+`/workspace` (the DSH image's `WORKDIR`, where agents keep files) is the same
+`dsh-home` PVC mounted again with `subPath: workspace`. The two trees stay
+separate, so `.credentials.yaml` and the session store are never part of the
+agent-visible workspace.
+
 ## 5. Suspend: delete the pod, keep the PVC + Service
 
 ```bash
@@ -219,13 +225,15 @@ kubectl -n dsh-demo get svc "$SB"            # still here
 kubectl -n dsh-demo patch sandbox "$SB" --type merge -p '{"spec":{"operatingMode":"Running"}}'
 kubectl -n dsh-demo wait sandbox "$SB" --for=condition=Ready --timeout=180s
 kubectl -n dsh-demo exec "$SB" -- cat /home/node/.dsh/marker.txt   # "remember me"
+kubectl -n dsh-demo exec "$SB" -- cat /workspace/marker.txt        # "remember me too"
 ```
 
 > **Why resume works:** the pod template sets `fsGroupChangePolicy: OnRootMismatch`.
 > DSH requires its `.credentials.yaml` to be `0600`, while the kubelet's default
 > fsGroup policy widens every file to `0660` on each pod start — a resumed sandbox
 > would crash-loop. See [Troubleshooting](#troubleshooting) for the one-time fix
-> on sandboxes created before this setting existed.
+> on sandboxes created before this setting existed. The `/workspace` mount does not
+> change that: `subPath` is a separate directory, and only the volume root is walked.
 
 ## 7. Injection policy rejects env-carrying claims
 
@@ -579,6 +587,7 @@ is non-recursive, so the optional manifest is not applied by the base step.
 | Gateway pod `ErrImagePull` / `ImagePullBackOff` (especially on kind) | The manifest uses `imagePullPolicy: IfNotPresent` with the release identity. Build with `make gateway`, retag to exactly the manifest's image, `kind load docker-image <that-ref>`, or set the Deployment image to your local tag / pinned digest. On a real cluster, make sure the GHCR package is readable (public or a pull secret) |
 | Step 3: `pod "$SB" not found` after deploying the gateway | The claim was suspended by the idle sweeper. Claims created by the gateway carry `aaas.example.com/managed-by=dsh-gateway`; hand-applied claims (steps 1–7) do not and are never swept. Resume the sandbox: `kubectl -n dsh-demo patch sandbox "$SB" --type merge -p '{"spec":{"operatingMode":"Running"}}'` |
 | Resume crash-loops: `credentials-local: … is readable beyond its owner (mode 660)` | The pod template lacks `fsGroupChangePolicy: OnRootMismatch`, so the kubelet's default fsGroup policy rewrote `.credentials.yaml` to `0660`; DSH requires `0600` and refuses to start. New installs get the policy from the template. For an existing sandbox: suspend it, run a throwaway pod on `dsh-home-$SB` to `chmod 600 /home/node/.dsh/.credentials.yaml`, patch the sandbox's `podTemplate` with the policy, then resume |
+| Agents get `Permission denied` writing to `/workspace` on a fresh or resumed pod | The `/workspace` mount is the `dsh-home` PVC's `workspace/` subPath, and the kubelet is known to leave a `subPath` directory at `0755` if it is interrupted between creating and chmodding it ([kubernetes#142464](https://github.com/kubernetes/kubernetes/issues/142464)); later mounts never correct it. Suspend the sandbox, run a throwaway pod on `dsh-home-$SB` mounted at `/data`, `chmod 2775 /data/workspace` (do **not** chown/chmod recursively — that rewrites `.credentials.yaml`), then resume. If you recreate the sandbox instead, its current `/workspace` contents go with the PVC |
 | SSO: GitHub login succeeds but the gateway asks for a token / returns `401` to `/recreate` | `AUTH_PROXY_HEADER` is not set on `deploy/dsh-gateway` (SSO is opt-in). Set `AUTH_PROXY_HEADER=X-Forwarded-User` and confirm the request actually reaches the gateway through oauth2-proxy |
 | SSO: gateway treats a caller as another user / bad `X-Forwarded-User` accepted | Trust boundary violation: something other than oauth2-proxy can reach `dsh-gateway:8080`. Remove the second path (Ingress/Gateway/port-forward), keep the Service ClusterIP, and only re-enable SSO once the proxy is the sole entry point. Never put a trusted identity header in front of a service that is directly reachable |
 | SSO: oauth2-proxy `cookie_secret must be 16, 24, or 32 bytes ... but is 44 bytes` | The cookie secret was generated as standard base64. Use URL-safe unpadded base64 (see `73-oauth2-proxy-secret.yaml.example`) |

examples/dsh-agents-as-a-service/docs/architecture.md

@@ -24,7 +24,7 @@ flowchart LR
         sb["Sandbox dsh-pool-xxxx<br/>spec.operatingMode"]
         svc["Headless Service<br/>status.serviceFQDN"]
         pod["DSH Web pod<br/>node process :3080"]
-        pvc[("PVC dsh-home-SANDBOX<br/>/home/node/.dsh")]
+        pvc[("PVC dsh-home-SANDBOX<br/>/home/node/.dsh + /workspace (subPath)")]
     end
 
     browser -->|"HTTP + WebSocket<br/>/u/USER/**"| gw
@@ -46,7 +46,7 @@ Responsibilities:
 | **Gateway** | Signup/login, session auth, `SandboxClaim` lifecycle, wake-on-connect, idle sweeper, HTTP/SSE/WebSocket reverse proxy, DSH launch-token handshake. |
 | **Kubernetes API** | Source of truth for claims, sandboxes, and pods; also the log source for the DSH launch token. |
 | **agent-sandbox controller** | Adopts warm spares on claim, flips `operatingMode` (Running/Suspended), garbage-collects sandbox + PVC on claim delete. |
-| **DSH Web pod** | One `dsh web` process per user, state on the per-user PVC, listening on `3080`. |
+| **DSH Web pod** | One `dsh web` process per user, DSH state and the agent workspace on the per-user PVC (`/home/node/.dsh` + `/workspace`), listening on `3080`. |
 | **Warm pool** | Keeps `N` Ready sandboxes so signup is a label flip instead of a cold start. |
 
 ## Exposure variants
@@ -159,8 +159,12 @@ sequenceDiagram
     G-->>B: request held, then served
 ```
 
-Suspend deletes only the pod. The PVC keeps conversations, sessions, and config, so resume
-reattaches exactly the same state.
+Suspend deletes only the pod. The PVC keeps conversations, sessions, config **and the agent
+workspace**, so resume reattaches exactly the same state. Both trees live on one PVC: DSH state at
+`/home/node/.dsh` (the volume root) and the workspace at `/workspace` (a second mount of the same
+volume with `subPath: workspace`, which keeps `.credentials.yaml` out of the agent-visible tree).
+The idle sweeper never touches the volume; deleting the claim still garbage-collects it, so
+`POST /recreate` wipes state and workspace together.
 
 **Ownership boundary.** The gateway shares its namespace with claims created outside it — the
 example's resource-layer walkthrough applies `dsh-alice` by hand, and every claim is named

examples/dsh-agents-as-a-service/manifests/10-sandboxtemplate.yaml

@@ -135,7 +135,8 @@ spec:
             periodSeconds: 15
             failureThreshold: 3
           # The container only needs to serve HTTP; drop every capability and
-          # disallow privilege escalation. DSH writes only to DSH_HOME (PVC).
+          # disallow privilege escalation. DSH writes only to its PVC-backed
+          # directories (DSH_HOME and the agent workspace).
           securityContext:
             allowPrivilegeEscalation: false
             capabilities:
@@ -148,6 +149,16 @@ spec:
           volumeMounts:
             - name: dsh-home
               mountPath: /home/node/.dsh # DSH_HOME: sessions, config, state
+            # The agent workspace (the image's WORKDIR) is PVC-backed too, but
+            # under its own subPath so DSH state - including .credentials.yaml -
+            # never shows up as workspace data. Kubelet creates workspace/ on
+            # first mount with the volume's fsGroup permissions (0775, setgid
+            # inherited from the volume root), so UID 1000 can write; no init
+            # container is needed. If the kubelet is killed mid-creation the
+            # directory stays 0755: see Troubleshooting in the README.
+            - name: dsh-home
+              mountPath: /workspace
+              subPath: workspace
   volumeClaimTemplates:
     - metadata:
         name: dsh-home

examples/dsh-agents-as-a-service/scripts/e2e-kind.sh

@@ -73,7 +73,13 @@ pass "claim Ready in ${ELAPSED}s, adopted $SB"
 echo "=== 3. write state on the PVC"
 POD="$SB"
 kubectl -n "$NS" exec "$POD" -- sh -c 'echo "remember me" > /home/node/.dsh/marker.txt'
-pass "marker written in $POD"
+pass "state marker written in $POD"
+# The agent workspace must be the dsh-home PVC's workspace/ subPath, writable
+# as UID 1000 without an init container.
+SUBPATH=$(kubectl -n "$NS" get pod "$POD" -o jsonpath='{range .spec.containers[0].volumeMounts[?(@.mountPath=="/workspace")]}{.subPath}{end}')
+[ "$SUBPATH" = "workspace" ] || fail "/workspace is not mounted from the dsh-home PVC subPath"
+kubectl -n "$NS" exec "$POD" -- sh -c 'echo "remember me too" > /workspace/marker.txt'
+pass "workspace marker written in $POD (subPath=$SUBPATH)"
 
 echo "=== 4. suspend (pod deleted, PVC + Service retained)"
 kubectl -n "$NS" patch sandbox "$SB" --type merge -p '{"spec":{"operatingMode":"Suspended"}}'
@@ -87,8 +93,11 @@ kubectl -n "$NS" patch sandbox "$SB" --type merge -p '{"spec":{"operatingMode":"
 kubectl -n "$NS" wait sandbox "$SB" --for=condition=Ready --timeout=180s
 POD="$SB"
 MARKER=$(kubectl -n "$NS" exec "$POD" -- cat /home/node/.dsh/marker.txt)
-[ "$MARKER" = "remember me" ] || fail "marker did not survive suspend/resume"
-pass "resumed: state survived on the reattached PVC"
+[ "$MARKER" = "remember me" ] || fail "state marker did not survive suspend/resume"
+pass "resumed: DSH state survived on the reattached PVC"
+WS_MARKER=$(kubectl -n "$NS" exec "$POD" -- cat /workspace/marker.txt)
+[ "$WS_MARKER" = "remember me too" ] || fail "workspace marker did not survive suspend/resume"
+pass "resumed: agent workspace survived on the PVC subPath"
 
 echo "=== 6. injection policy rejects env-carrying claims"
 kubectl apply -f manifests/40-claim-rejected.yaml