diff --git a/.dsh/skills/dsh-agent-sandbox-platform/references/platform-runbook.md b/.dsh/skills/dsh-agent-sandbox-platform/references/platform-runbook.md index a29664c..2f9496f 100644 --- a/.dsh/skills/dsh-agent-sandbox-platform/references/platform-runbook.md +++ b/.dsh/skills/dsh-agent-sandbox-platform/references/platform-runbook.md @@ -28,8 +28,8 @@ a missing CRD is a prerequisite failure, not an application bug. | `SandboxWarmPool` | Pre-warmed spares from one template | `spec.replicas`, `sandboxTemplateRef` | | `SandboxClaim` | One tenant/user request; adopts a spare | `spec.warmPoolRef`, `additionalPodMetadata`, `status.sandbox.name` | | `Sandbox` | Adopted or directly created pod + Service + PVC | `spec.operatingMode`, `status.conditions`, `status.serviceFQDN`, `status.podIPs` | -| `Pod` | The DSH Web process | `:3080`, PVC at `/home/node/.dsh` | -| `PersistentVolumeClaim` | Per-sandbox state | `dsh-home-` | +| `Pod` | The DSH Web process | `:3080`, PVC at `/home/node/.dsh` and `/workspace` (`subPath: workspace`) | +| `PersistentVolumeClaim` | Per-sandbox DSH state + agent workspace | `dsh-home-` | | `Service` | Stable per-sandbox DNS | ``, `status.serviceFQDN` | Delete a claim to cascade-delete its sandbox and PVC (account deletion). Never diff --git a/docs/main-image.md b/docs/main-image.md index 6789bcb..31d71ef 100644 --- a/docs/main-image.md +++ b/docs/main-image.md @@ -108,7 +108,7 @@ procedure. | --- | --- | --- | | Exposed port | `3080/tcp` | `EXPOSE 3080`; one port serves the UI, assets, API, SSE and WebSocket traffic. | | State volume | `/home/node/.dsh` | Declared with `VOLUME`; use a named volume (Docker) or PVC (Kubernetes). | -| Working directory | `/workspace` | The agent workspace; mount a volume here if agents should keep files. | +| Working directory | `/workspace` | The agent workspace. Mount a volume here if agents should keep files; the agents-as-a-service example mounts the same PVC's `workspace` subPath here. | | Stop signal | `SIGTERM` | `tini` forwards it to `dsh web`; allow a graceful shutdown period. | | User | `node`, UID/GID `1000` | The process is non-root. A PVC mounted at `DSH_HOME` must be writable by UID 1000. | diff --git a/examples/dsh-agents-as-a-service/README.md b/examples/dsh-agents-as-a-service/README.md index 2ac2d15..8e70444 100644 --- a/examples/dsh-agents-as-a-service/README.md +++ b/examples/dsh-agents-as-a-service/README.md @@ -6,8 +6,8 @@ platform pattern into pure agent-sandbox resources + kubectl, running **DeepSeek Harness Web (DSH)** instead of Hermes. Every user of such a platform gets a personal, long-lived DSH Web instance that is **provisioned in ~2 seconds from a warm pool, suspended when idle to save cost, and woken transparently when the -user returns** — with all state (conversations, sessions, config) surviving on a -per-user PVC at `/home/node/.dsh`. +user returns** — with all state (conversations, sessions, config, and the agent +workspace) surviving on the per-user PVC at `/home/node/.dsh` + `/workspace`. It ships in three layers: @@ -46,7 +46,7 @@ What it demonstrates, feature by feature: | agent-sandbox feature | Role in the platform | | --- | --- | -| `SandboxTemplate` with `volumeClaimTemplates` | Identical DSH pods; per-user state on a PVC at `/home/node/.dsh` | +| `SandboxTemplate` with `volumeClaimTemplates` | Identical DSH pods; per-user state at `/home/node/.dsh` and the agent workspace at `/workspace`, on one PVC (the workspace is a `subPath` mount, so DSH state stays private) | | `envVarsInjectionPolicy` / `volumeClaimTemplatesPolicy: Disallowed` | Claims that would bypass the warm pool are **rejected loudly**, not cold-started silently | | `service: true` | Stable per-sandbox DNS (`status.serviceFQDN`) for the gateway to route on | | `SandboxWarmPool` | Signup latency is pre-paid: image pull, PVC provision, DSH boot all happen before any user exists | @@ -202,8 +202,14 @@ kubectl -n dsh-demo port-forward "pod/$SB" 18081:3080 ```bash kubectl -n dsh-demo exec "$SB" -- sh -c 'echo "remember me" > /home/node/.dsh/marker.txt' +kubectl -n dsh-demo exec "$SB" -- sh -c 'echo "remember me too" > /workspace/marker.txt' ``` +`/workspace` (the DSH image's `WORKDIR`, where agents keep files) is the same +`dsh-home` PVC mounted again with `subPath: workspace`. The two trees stay +separate, so `.credentials.yaml` and the session store are never part of the +agent-visible workspace. + ## 5. Suspend: delete the pod, keep the PVC + Service ```bash @@ -219,13 +225,15 @@ kubectl -n dsh-demo get svc "$SB" # still here kubectl -n dsh-demo patch sandbox "$SB" --type merge -p '{"spec":{"operatingMode":"Running"}}' kubectl -n dsh-demo wait sandbox "$SB" --for=condition=Ready --timeout=180s kubectl -n dsh-demo exec "$SB" -- cat /home/node/.dsh/marker.txt # "remember me" +kubectl -n dsh-demo exec "$SB" -- cat /workspace/marker.txt # "remember me too" ``` > **Why resume works:** the pod template sets `fsGroupChangePolicy: OnRootMismatch`. > DSH requires its `.credentials.yaml` to be `0600`, while the kubelet's default > fsGroup policy widens every file to `0660` on each pod start — a resumed sandbox > would crash-loop. See [Troubleshooting](#troubleshooting) for the one-time fix -> on sandboxes created before this setting existed. +> on sandboxes created before this setting existed. The `/workspace` mount does not +> change that: `subPath` is a separate directory, and only the volume root is walked. ## 7. Injection policy rejects env-carrying claims @@ -579,6 +587,7 @@ is non-recursive, so the optional manifest is not applied by the base step. | Gateway pod `ErrImagePull` / `ImagePullBackOff` (especially on kind) | The manifest uses `imagePullPolicy: IfNotPresent` with the release identity. Build with `make gateway`, retag to exactly the manifest's image, `kind load docker-image `, or set the Deployment image to your local tag / pinned digest. On a real cluster, make sure the GHCR package is readable (public or a pull secret) | | Step 3: `pod "$SB" not found` after deploying the gateway | The claim was suspended by the idle sweeper. Claims created by the gateway carry `aaas.example.com/managed-by=dsh-gateway`; hand-applied claims (steps 1–7) do not and are never swept. Resume the sandbox: `kubectl -n dsh-demo patch sandbox "$SB" --type merge -p '{"spec":{"operatingMode":"Running"}}'` | | Resume crash-loops: `credentials-local: … is readable beyond its owner (mode 660)` | The pod template lacks `fsGroupChangePolicy: OnRootMismatch`, so the kubelet's default fsGroup policy rewrote `.credentials.yaml` to `0660`; DSH requires `0600` and refuses to start. New installs get the policy from the template. For an existing sandbox: suspend it, run a throwaway pod on `dsh-home-$SB` to `chmod 600 /home/node/.dsh/.credentials.yaml`, patch the sandbox's `podTemplate` with the policy, then resume | +| Agents get `Permission denied` writing to `/workspace` on a fresh or resumed pod | The `/workspace` mount is the `dsh-home` PVC's `workspace/` subPath, and the kubelet is known to leave a `subPath` directory at `0755` if it is interrupted between creating and chmodding it ([kubernetes#142464](https://github.com/kubernetes/kubernetes/issues/142464)); later mounts never correct it. Suspend the sandbox, run a throwaway pod on `dsh-home-$SB` mounted at `/data`, `chmod 2775 /data/workspace` (do **not** chown/chmod recursively — that rewrites `.credentials.yaml`), then resume. If you recreate the sandbox instead, its current `/workspace` contents go with the PVC | | SSO: GitHub login succeeds but the gateway asks for a token / returns `401` to `/recreate` | `AUTH_PROXY_HEADER` is not set on `deploy/dsh-gateway` (SSO is opt-in). Set `AUTH_PROXY_HEADER=X-Forwarded-User` and confirm the request actually reaches the gateway through oauth2-proxy | | SSO: gateway treats a caller as another user / bad `X-Forwarded-User` accepted | Trust boundary violation: something other than oauth2-proxy can reach `dsh-gateway:8080`. Remove the second path (Ingress/Gateway/port-forward), keep the Service ClusterIP, and only re-enable SSO once the proxy is the sole entry point. Never put a trusted identity header in front of a service that is directly reachable | | SSO: oauth2-proxy `cookie_secret must be 16, 24, or 32 bytes ... but is 44 bytes` | The cookie secret was generated as standard base64. Use URL-safe unpadded base64 (see `73-oauth2-proxy-secret.yaml.example`) | diff --git a/examples/dsh-agents-as-a-service/docs/architecture.md b/examples/dsh-agents-as-a-service/docs/architecture.md index 6a07578..c33c802 100644 --- a/examples/dsh-agents-as-a-service/docs/architecture.md +++ b/examples/dsh-agents-as-a-service/docs/architecture.md @@ -24,7 +24,7 @@ flowchart LR sb["Sandbox dsh-pool-xxxx
spec.operatingMode"] svc["Headless Service
status.serviceFQDN"] pod["DSH Web pod
node process :3080"] - pvc[("PVC dsh-home-SANDBOX
/home/node/.dsh")] + pvc[("PVC dsh-home-SANDBOX
/home/node/.dsh + /workspace (subPath)")] end browser -->|"HTTP + WebSocket
/u/USER/**"| gw @@ -46,7 +46,7 @@ Responsibilities: | **Gateway** | Signup/login, session auth, `SandboxClaim` lifecycle, wake-on-connect, idle sweeper, HTTP/SSE/WebSocket reverse proxy, DSH launch-token handshake. | | **Kubernetes API** | Source of truth for claims, sandboxes, and pods; also the log source for the DSH launch token. | | **agent-sandbox controller** | Adopts warm spares on claim, flips `operatingMode` (Running/Suspended), garbage-collects sandbox + PVC on claim delete. | -| **DSH Web pod** | One `dsh web` process per user, state on the per-user PVC, listening on `3080`. | +| **DSH Web pod** | One `dsh web` process per user, DSH state and the agent workspace on the per-user PVC (`/home/node/.dsh` + `/workspace`), listening on `3080`. | | **Warm pool** | Keeps `N` Ready sandboxes so signup is a label flip instead of a cold start. | ## Exposure variants @@ -159,8 +159,12 @@ sequenceDiagram G-->>B: request held, then served ``` -Suspend deletes only the pod. The PVC keeps conversations, sessions, and config, so resume -reattaches exactly the same state. +Suspend deletes only the pod. The PVC keeps conversations, sessions, config **and the agent +workspace**, so resume reattaches exactly the same state. Both trees live on one PVC: DSH state at +`/home/node/.dsh` (the volume root) and the workspace at `/workspace` (a second mount of the same +volume with `subPath: workspace`, which keeps `.credentials.yaml` out of the agent-visible tree). +The idle sweeper never touches the volume; deleting the claim still garbage-collects it, so +`POST /recreate` wipes state and workspace together. **Ownership boundary.** The gateway shares its namespace with claims created outside it — the example's resource-layer walkthrough applies `dsh-alice` by hand, and every claim is named diff --git a/examples/dsh-agents-as-a-service/manifests/10-sandboxtemplate.yaml b/examples/dsh-agents-as-a-service/manifests/10-sandboxtemplate.yaml index 9583fe7..0366f9f 100644 --- a/examples/dsh-agents-as-a-service/manifests/10-sandboxtemplate.yaml +++ b/examples/dsh-agents-as-a-service/manifests/10-sandboxtemplate.yaml @@ -135,7 +135,8 @@ spec: periodSeconds: 15 failureThreshold: 3 # The container only needs to serve HTTP; drop every capability and - # disallow privilege escalation. DSH writes only to DSH_HOME (PVC). + # disallow privilege escalation. DSH writes only to its PVC-backed + # directories (DSH_HOME and the agent workspace). securityContext: allowPrivilegeEscalation: false capabilities: @@ -148,6 +149,16 @@ spec: volumeMounts: - name: dsh-home mountPath: /home/node/.dsh # DSH_HOME: sessions, config, state + # The agent workspace (the image's WORKDIR) is PVC-backed too, but + # under its own subPath so DSH state - including .credentials.yaml - + # never shows up as workspace data. Kubelet creates workspace/ on + # first mount with the volume's fsGroup permissions (0775, setgid + # inherited from the volume root), so UID 1000 can write; no init + # container is needed. If the kubelet is killed mid-creation the + # directory stays 0755: see Troubleshooting in the README. + - name: dsh-home + mountPath: /workspace + subPath: workspace volumeClaimTemplates: - metadata: name: dsh-home diff --git a/examples/dsh-agents-as-a-service/scripts/e2e-kind.sh b/examples/dsh-agents-as-a-service/scripts/e2e-kind.sh index 850417c..1e4cacf 100755 --- a/examples/dsh-agents-as-a-service/scripts/e2e-kind.sh +++ b/examples/dsh-agents-as-a-service/scripts/e2e-kind.sh @@ -73,7 +73,13 @@ pass "claim Ready in ${ELAPSED}s, adopted $SB" echo "=== 3. write state on the PVC" POD="$SB" kubectl -n "$NS" exec "$POD" -- sh -c 'echo "remember me" > /home/node/.dsh/marker.txt' -pass "marker written in $POD" +pass "state marker written in $POD" +# The agent workspace must be the dsh-home PVC's workspace/ subPath, writable +# as UID 1000 without an init container. +SUBPATH=$(kubectl -n "$NS" get pod "$POD" -o jsonpath='{range .spec.containers[0].volumeMounts[?(@.mountPath=="/workspace")]}{.subPath}{end}') +[ "$SUBPATH" = "workspace" ] || fail "/workspace is not mounted from the dsh-home PVC subPath" +kubectl -n "$NS" exec "$POD" -- sh -c 'echo "remember me too" > /workspace/marker.txt' +pass "workspace marker written in $POD (subPath=$SUBPATH)" echo "=== 4. suspend (pod deleted, PVC + Service retained)" kubectl -n "$NS" patch sandbox "$SB" --type merge -p '{"spec":{"operatingMode":"Suspended"}}' @@ -87,8 +93,11 @@ kubectl -n "$NS" patch sandbox "$SB" --type merge -p '{"spec":{"operatingMode":" kubectl -n "$NS" wait sandbox "$SB" --for=condition=Ready --timeout=180s POD="$SB" MARKER=$(kubectl -n "$NS" exec "$POD" -- cat /home/node/.dsh/marker.txt) -[ "$MARKER" = "remember me" ] || fail "marker did not survive suspend/resume" -pass "resumed: state survived on the reattached PVC" +[ "$MARKER" = "remember me" ] || fail "state marker did not survive suspend/resume" +pass "resumed: DSH state survived on the reattached PVC" +WS_MARKER=$(kubectl -n "$NS" exec "$POD" -- cat /workspace/marker.txt) +[ "$WS_MARKER" = "remember me too" ] || fail "workspace marker did not survive suspend/resume" +pass "resumed: agent workspace survived on the PVC subPath" echo "=== 6. injection policy rejects env-carrying claims" kubectl apply -f manifests/40-claim-rejected.yaml