From 367294b2308060084a884d824256ae582034e7dc Mon Sep 17 00:00:00 2001 From: "SREDevOps.org Agent" Date: Fri, 30 Oct 2026 08:29:39 -0300 Subject: [PATCH] feat(example): persist the agent workspace on the PVC via subPath (#34) /workspace - the DSH image's WORKDIR, where agents keep files - was plain container storage, so every suspend/resume or pod replacement silently threw agent work away while DSH state survived on the PVC. Mount the same per-sandbox `dsh-home` PVC a second time at /workspace with `subPath: workspace`. One PVC, no extra provisioning, and the subPath keeps DSH state (sessions, config and .credentials.yaml at the volume root) out of the agent-visible tree; mounting the volume root at /workspace would have exposed the credential file. The kubelet creates the missing subPath directory itself and chmods it to the volume's fsGroup permissions, and the volume root's setgid bit makes the new directory inherit gid 1000, so UID 1000 can write with no init container and no root. The rare interrupted-kubelet case that leaves 0755 is documented with a recovery procedure (kubernetes#142464) rather than papered over. - e2e-kind: write a /workspace marker alongside the DSH_HOME marker, assert the mount is the dsh-home subPath, and verify both survive suspend/resume. - Docs: README (overview, steps 4-6, troubleshooting row), architecture (PVC diagram, persistence, component table), main-image working-directory row, platform runbook PVC row. Verified: make check (14 manifests), make lint, plus YAML assertions on the two mounts and the unchanged volumeClaimTemplates. Co-authored-by: SREDevOpsAgents --- .../references/platform-runbook.md | 4 ++-- docs/main-image.md | 2 +- examples/dsh-agents-as-a-service/README.md | 17 +++++++++++++---- .../docs/architecture.md | 12 ++++++++---- .../manifests/10-sandboxtemplate.yaml | 13 ++++++++++++- .../dsh-agents-as-a-service/scripts/e2e-kind.sh | 15 ++++++++++++--- 6 files changed, 48 insertions(+), 15 deletions(-) diff --git a/.dsh/skills/dsh-agent-sandbox-platform/references/platform-runbook.md b/.dsh/skills/dsh-agent-sandbox-platform/references/platform-runbook.md index a29664c..2f9496f 100644 --- a/.dsh/skills/dsh-agent-sandbox-platform/references/platform-runbook.md +++ b/.dsh/skills/dsh-agent-sandbox-platform/references/platform-runbook.md @@ -28,8 +28,8 @@ a missing CRD is a prerequisite failure, not an application bug. | `SandboxWarmPool` | Pre-warmed spares from one template | `spec.replicas`, `sandboxTemplateRef` | | `SandboxClaim` | One tenant/user request; adopts a spare | `spec.warmPoolRef`, `additionalPodMetadata`, `status.sandbox.name` | | `Sandbox` | Adopted or directly created pod + Service + PVC | `spec.operatingMode`, `status.conditions`, `status.serviceFQDN`, `status.podIPs` | -| `Pod` | The DSH Web process | `:3080`, PVC at `/home/node/.dsh` | -| `PersistentVolumeClaim` | Per-sandbox state | `dsh-home-` | +| `Pod` | The DSH Web process | `:3080`, PVC at `/home/node/.dsh` and `/workspace` (`subPath: workspace`) | +| `PersistentVolumeClaim` | Per-sandbox DSH state + agent workspace | `dsh-home-` | | `Service` | Stable per-sandbox DNS | ``, `status.serviceFQDN` | Delete a claim to cascade-delete its sandbox and PVC (account deletion). Never diff --git a/docs/main-image.md b/docs/main-image.md index 6789bcb..31d71ef 100644 --- a/docs/main-image.md +++ b/docs/main-image.md @@ -108,7 +108,7 @@ procedure. | --- | --- | --- | | Exposed port | `3080/tcp` | `EXPOSE 3080`; one port serves the UI, assets, API, SSE and WebSocket traffic. | | State volume | `/home/node/.dsh` | Declared with `VOLUME`; use a named volume (Docker) or PVC (Kubernetes). | -| Working directory | `/workspace` | The agent workspace; mount a volume here if agents should keep files. | +| Working directory | `/workspace` | The agent workspace. Mount a volume here if agents should keep files; the agents-as-a-service example mounts the same PVC's `workspace` subPath here. | | Stop signal | `SIGTERM` | `tini` forwards it to `dsh web`; allow a graceful shutdown period. | | User | `node`, UID/GID `1000` | The process is non-root. A PVC mounted at `DSH_HOME` must be writable by UID 1000. | diff --git a/examples/dsh-agents-as-a-service/README.md b/examples/dsh-agents-as-a-service/README.md index 2ac2d15..8e70444 100644 --- a/examples/dsh-agents-as-a-service/README.md +++ b/examples/dsh-agents-as-a-service/README.md @@ -6,8 +6,8 @@ platform pattern into pure agent-sandbox resources + kubectl, running **DeepSeek Harness Web (DSH)** instead of Hermes. Every user of such a platform gets a personal, long-lived DSH Web instance that is **provisioned in ~2 seconds from a warm pool, suspended when idle to save cost, and woken transparently when the -user returns** — with all state (conversations, sessions, config) surviving on a -per-user PVC at `/home/node/.dsh`. +user returns** — with all state (conversations, sessions, config, and the agent +workspace) surviving on the per-user PVC at `/home/node/.dsh` + `/workspace`. It ships in three layers: @@ -46,7 +46,7 @@ What it demonstrates, feature by feature: | agent-sandbox feature | Role in the platform | | --- | --- | -| `SandboxTemplate` with `volumeClaimTemplates` | Identical DSH pods; per-user state on a PVC at `/home/node/.dsh` | +| `SandboxTemplate` with `volumeClaimTemplates` | Identical DSH pods; per-user state at `/home/node/.dsh` and the agent workspace at `/workspace`, on one PVC (the workspace is a `subPath` mount, so DSH state stays private) | | `envVarsInjectionPolicy` / `volumeClaimTemplatesPolicy: Disallowed` | Claims that would bypass the warm pool are **rejected loudly**, not cold-started silently | | `service: true` | Stable per-sandbox DNS (`status.serviceFQDN`) for the gateway to route on | | `SandboxWarmPool` | Signup latency is pre-paid: image pull, PVC provision, DSH boot all happen before any user exists | @@ -202,8 +202,14 @@ kubectl -n dsh-demo port-forward "pod/$SB" 18081:3080 ```bash kubectl -n dsh-demo exec "$SB" -- sh -c 'echo "remember me" > /home/node/.dsh/marker.txt' +kubectl -n dsh-demo exec "$SB" -- sh -c 'echo "remember me too" > /workspace/marker.txt' ``` +`/workspace` (the DSH image's `WORKDIR`, where agents keep files) is the same +`dsh-home` PVC mounted again with `subPath: workspace`. The two trees stay +separate, so `.credentials.yaml` and the session store are never part of the +agent-visible workspace. + ## 5. Suspend: delete the pod, keep the PVC + Service ```bash @@ -219,13 +225,15 @@ kubectl -n dsh-demo get svc "$SB" # still here kubectl -n dsh-demo patch sandbox "$SB" --type merge -p '{"spec":{"operatingMode":"Running"}}' kubectl -n dsh-demo wait sandbox "$SB" --for=condition=Ready --timeout=180s kubectl -n dsh-demo exec "$SB" -- cat /home/node/.dsh/marker.txt # "remember me" +kubectl -n dsh-demo exec "$SB" -- cat /workspace/marker.txt # "remember me too" ``` > **Why resume works:** the pod template sets `fsGroupChangePolicy: OnRootMismatch`. > DSH requires its `.credentials.yaml` to be `0600`, while the kubelet's default > fsGroup policy widens every file to `0660` on each pod start — a resumed sandbox > would crash-loop. See [Troubleshooting](#troubleshooting) for the one-time fix -> on sandboxes created before this setting existed. +> on sandboxes created before this setting existed. The `/workspace` mount does not +> change that: `subPath` is a separate directory, and only the volume root is walked. ## 7. Injection policy rejects env-carrying claims @@ -579,6 +587,7 @@ is non-recursive, so the optional manifest is not applied by the base step. | Gateway pod `ErrImagePull` / `ImagePullBackOff` (especially on kind) | The manifest uses `imagePullPolicy: IfNotPresent` with the release identity. Build with `make gateway`, retag to exactly the manifest's image, `kind load docker-image `, or set the Deployment image to your local tag / pinned digest. On a real cluster, make sure the GHCR package is readable (public or a pull secret) | | Step 3: `pod "$SB" not found` after deploying the gateway | The claim was suspended by the idle sweeper. Claims created by the gateway carry `aaas.example.com/managed-by=dsh-gateway`; hand-applied claims (steps 1–7) do not and are never swept. Resume the sandbox: `kubectl -n dsh-demo patch sandbox "$SB" --type merge -p '{"spec":{"operatingMode":"Running"}}'` | | Resume crash-loops: `credentials-local: … is readable beyond its owner (mode 660)` | The pod template lacks `fsGroupChangePolicy: OnRootMismatch`, so the kubelet's default fsGroup policy rewrote `.credentials.yaml` to `0660`; DSH requires `0600` and refuses to start. New installs get the policy from the template. For an existing sandbox: suspend it, run a throwaway pod on `dsh-home-$SB` to `chmod 600 /home/node/.dsh/.credentials.yaml`, patch the sandbox's `podTemplate` with the policy, then resume | +| Agents get `Permission denied` writing to `/workspace` on a fresh or resumed pod | The `/workspace` mount is the `dsh-home` PVC's `workspace/` subPath, and the kubelet is known to leave a `subPath` directory at `0755` if it is interrupted between creating and chmodding it ([kubernetes#142464](https://github.com/kubernetes/kubernetes/issues/142464)); later mounts never correct it. Suspend the sandbox, run a throwaway pod on `dsh-home-$SB` mounted at `/data`, `chmod 2775 /data/workspace` (do **not** chown/chmod recursively — that rewrites `.credentials.yaml`), then resume. If you recreate the sandbox instead, its current `/workspace` contents go with the PVC | | SSO: GitHub login succeeds but the gateway asks for a token / returns `401` to `/recreate` | `AUTH_PROXY_HEADER` is not set on `deploy/dsh-gateway` (SSO is opt-in). Set `AUTH_PROXY_HEADER=X-Forwarded-User` and confirm the request actually reaches the gateway through oauth2-proxy | | SSO: gateway treats a caller as another user / bad `X-Forwarded-User` accepted | Trust boundary violation: something other than oauth2-proxy can reach `dsh-gateway:8080`. Remove the second path (Ingress/Gateway/port-forward), keep the Service ClusterIP, and only re-enable SSO once the proxy is the sole entry point. Never put a trusted identity header in front of a service that is directly reachable | | SSO: oauth2-proxy `cookie_secret must be 16, 24, or 32 bytes ... but is 44 bytes` | The cookie secret was generated as standard base64. Use URL-safe unpadded base64 (see `73-oauth2-proxy-secret.yaml.example`) | diff --git a/examples/dsh-agents-as-a-service/docs/architecture.md b/examples/dsh-agents-as-a-service/docs/architecture.md index 6a07578..c33c802 100644 --- a/examples/dsh-agents-as-a-service/docs/architecture.md +++ b/examples/dsh-agents-as-a-service/docs/architecture.md @@ -24,7 +24,7 @@ flowchart LR sb["Sandbox dsh-pool-xxxx
spec.operatingMode"] svc["Headless Service
status.serviceFQDN"] pod["DSH Web pod
node process :3080"] - pvc[("PVC dsh-home-SANDBOX
/home/node/.dsh")] + pvc[("PVC dsh-home-SANDBOX
/home/node/.dsh + /workspace (subPath)")] end browser -->|"HTTP + WebSocket
/u/USER/**"| gw @@ -46,7 +46,7 @@ Responsibilities: | **Gateway** | Signup/login, session auth, `SandboxClaim` lifecycle, wake-on-connect, idle sweeper, HTTP/SSE/WebSocket reverse proxy, DSH launch-token handshake. | | **Kubernetes API** | Source of truth for claims, sandboxes, and pods; also the log source for the DSH launch token. | | **agent-sandbox controller** | Adopts warm spares on claim, flips `operatingMode` (Running/Suspended), garbage-collects sandbox + PVC on claim delete. | -| **DSH Web pod** | One `dsh web` process per user, state on the per-user PVC, listening on `3080`. | +| **DSH Web pod** | One `dsh web` process per user, DSH state and the agent workspace on the per-user PVC (`/home/node/.dsh` + `/workspace`), listening on `3080`. | | **Warm pool** | Keeps `N` Ready sandboxes so signup is a label flip instead of a cold start. | ## Exposure variants @@ -159,8 +159,12 @@ sequenceDiagram G-->>B: request held, then served ``` -Suspend deletes only the pod. The PVC keeps conversations, sessions, and config, so resume -reattaches exactly the same state. +Suspend deletes only the pod. The PVC keeps conversations, sessions, config **and the agent +workspace**, so resume reattaches exactly the same state. Both trees live on one PVC: DSH state at +`/home/node/.dsh` (the volume root) and the workspace at `/workspace` (a second mount of the same +volume with `subPath: workspace`, which keeps `.credentials.yaml` out of the agent-visible tree). +The idle sweeper never touches the volume; deleting the claim still garbage-collects it, so +`POST /recreate` wipes state and workspace together. **Ownership boundary.** The gateway shares its namespace with claims created outside it — the example's resource-layer walkthrough applies `dsh-alice` by hand, and every claim is named diff --git a/examples/dsh-agents-as-a-service/manifests/10-sandboxtemplate.yaml b/examples/dsh-agents-as-a-service/manifests/10-sandboxtemplate.yaml index 9583fe7..0366f9f 100644 --- a/examples/dsh-agents-as-a-service/manifests/10-sandboxtemplate.yaml +++ b/examples/dsh-agents-as-a-service/manifests/10-sandboxtemplate.yaml @@ -135,7 +135,8 @@ spec: periodSeconds: 15 failureThreshold: 3 # The container only needs to serve HTTP; drop every capability and - # disallow privilege escalation. DSH writes only to DSH_HOME (PVC). + # disallow privilege escalation. DSH writes only to its PVC-backed + # directories (DSH_HOME and the agent workspace). securityContext: allowPrivilegeEscalation: false capabilities: @@ -148,6 +149,16 @@ spec: volumeMounts: - name: dsh-home mountPath: /home/node/.dsh # DSH_HOME: sessions, config, state + # The agent workspace (the image's WORKDIR) is PVC-backed too, but + # under its own subPath so DSH state - including .credentials.yaml - + # never shows up as workspace data. Kubelet creates workspace/ on + # first mount with the volume's fsGroup permissions (0775, setgid + # inherited from the volume root), so UID 1000 can write; no init + # container is needed. If the kubelet is killed mid-creation the + # directory stays 0755: see Troubleshooting in the README. + - name: dsh-home + mountPath: /workspace + subPath: workspace volumeClaimTemplates: - metadata: name: dsh-home diff --git a/examples/dsh-agents-as-a-service/scripts/e2e-kind.sh b/examples/dsh-agents-as-a-service/scripts/e2e-kind.sh index 850417c..1e4cacf 100755 --- a/examples/dsh-agents-as-a-service/scripts/e2e-kind.sh +++ b/examples/dsh-agents-as-a-service/scripts/e2e-kind.sh @@ -73,7 +73,13 @@ pass "claim Ready in ${ELAPSED}s, adopted $SB" echo "=== 3. write state on the PVC" POD="$SB" kubectl -n "$NS" exec "$POD" -- sh -c 'echo "remember me" > /home/node/.dsh/marker.txt' -pass "marker written in $POD" +pass "state marker written in $POD" +# The agent workspace must be the dsh-home PVC's workspace/ subPath, writable +# as UID 1000 without an init container. +SUBPATH=$(kubectl -n "$NS" get pod "$POD" -o jsonpath='{range .spec.containers[0].volumeMounts[?(@.mountPath=="/workspace")]}{.subPath}{end}') +[ "$SUBPATH" = "workspace" ] || fail "/workspace is not mounted from the dsh-home PVC subPath" +kubectl -n "$NS" exec "$POD" -- sh -c 'echo "remember me too" > /workspace/marker.txt' +pass "workspace marker written in $POD (subPath=$SUBPATH)" echo "=== 4. suspend (pod deleted, PVC + Service retained)" kubectl -n "$NS" patch sandbox "$SB" --type merge -p '{"spec":{"operatingMode":"Suspended"}}' @@ -87,8 +93,11 @@ kubectl -n "$NS" patch sandbox "$SB" --type merge -p '{"spec":{"operatingMode":" kubectl -n "$NS" wait sandbox "$SB" --for=condition=Ready --timeout=180s POD="$SB" MARKER=$(kubectl -n "$NS" exec "$POD" -- cat /home/node/.dsh/marker.txt) -[ "$MARKER" = "remember me" ] || fail "marker did not survive suspend/resume" -pass "resumed: state survived on the reattached PVC" +[ "$MARKER" = "remember me" ] || fail "state marker did not survive suspend/resume" +pass "resumed: DSH state survived on the reattached PVC" +WS_MARKER=$(kubectl -n "$NS" exec "$POD" -- cat /workspace/marker.txt) +[ "$WS_MARKER" = "remember me too" ] || fail "workspace marker did not survive suspend/resume" +pass "resumed: agent workspace survived on the PVC subPath" echo "=== 6. injection policy rejects env-carrying claims" kubectl apply -f manifests/40-claim-rejected.yaml