feat(skills): add agent-sandbox platform skill (#20)

Add the project-local dsh-agent-sandbox-platform skill covering platform operations, the optional agent-sandbox MCP server, feature-extension recipes, and a read-only preflight. Document project skills in README, docs and AGENTS.md. Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
2026-11-03 01:13:24 +00:00 · 2026-10-26 08:10:59 -03:00
parent 8681412be4
commit bf24306421
10 changed files with 1046 additions and 0 deletions
--- a/.dsh/skills/dsh-agent-sandbox-platform/references/extending-the-repo.md
+++ b/.dsh/skills/dsh-agent-sandbox-platform/references/extending-the-repo.md
@@ -0,0 +1,131 @@
+# Extending the repository
+
+Read `AGENTS.md`, `docs/examples.md`, and the example README before editing.
+This repository is image-first: the Dockerfile is the primary artifact, every
+implementation is a self-contained example under `examples/<name>/`, and
+`versions.env` is the only source of pins. Documentation is part of the change.
+
+## Invariants
+
+- Never hard-code a version, image tag, or registry path; edit `versions.env`.
+  For the DSH release identity use `make set-version`, never ad-hoc `sed`.
+- Never commit a secret. Secret-shaped manifests are `*.yaml.example` with
+  placeholders; keys are documented only in `.env.sample`.
+- Manifests are numbered `NN-*.yaml` and apply in order with plain `kubectl`.
+  Exposure variants live under `manifests/exposure/<variant>/`, are documented
+  by one README, and are mutually exclusive.
+- Keep the gateway HTTP/WebSocket contract stable unless the change is explicitly
+  about it.
+- Do not edit `CHANGELOG.md` (generated by git-cliff) or rewrite git history.
+- Keep diffs scoped; do not mix a version bump with an unrelated feature.
+
+## Change matrix
+
+| Change | Edit | Minimum verification |
+| --- | --- | --- |
+| Documentation only | `*.md`, `docs/` | `make check`; relative links resolve |
+| Example manifest | `examples/<name>/manifests/NN-*.yaml`, README step | `make check`; server-side dry-run in CI |
+| Gateway behavior/route | `gateway/gateway.py`, `gateway/test_gateway.py`, gateway README, architecture | `make test`, `make lint`; update tests and API docs |
+| Exposure variant | `manifests/exposure/<variant>/`, exposure README, example README | `make check`; apply exactly one variant |
+| New example | full `examples/<name>/` layout, `examples/README.md` | `make check`; CI dry-run; optional e2e |
+| Version bump | `versions.env`, affected docs/manifests | `make check`; CI installs the pin; e2e on a cluster |
+| MCP integration | deployment/RBAC, DSH row, docs | read-only `/readyz`; no secret in repo |
+| Workflow | `.github/workflows/` | actionlint; SHA-pinned actions; no hard-coded versions |
+
+## Recipes
+
+### Add or change a resource step
+
+1. Keep numbering stable; add `NN-*.yaml` at the end unless order matters.
+2. Update the matching numbered step in the example README.
+3. Keep cluster-specific state (node affinity, tuned replica counts) out of the
+   shipped manifest; document it as a local patch.
+4. Run `make check`. CI additionally parses every manifest and runs a
+   server-side dry-run against the agent-sandbox CRDs.
+
+### Change the gateway
+
+The gateway is a single-file Starlette app. A change usually touches:
+
+- `gateway/gateway.py` (route, auth, proxy, sweeper);
+- `gateway/test_gateway.py` (unit tests; no cluster required);
+- `gateway/README.md` (API table, configuration, auth semantics);
+- `examples/dsh-agents-as-a-service/docs/architecture.md` when flows change;
+- `examples/dsh-agents-as-a-service/README.md` when the walkthrough changes.
+
+Run `make test` and `make lint`. Preserve session-cookie auth, same-origin rules
+for cookie mutations and WebSocket upgrades, and the "waking"/`502`/`503`
+semantics unless the change is explicitly about them. Do not expose a trusted
+identity header unless the Service is reachable only through the proxy.
+
+### Add an exposure variant
+
+Create `manifests/exposure/<variant>/NN-*.yaml`, update
+`manifests/exposure/README.md` and the example README, and state the trust
+boundary. Variants are alternatives: never apply two paths to the same gateway.
+Set `DSH_TRUSTED_HOSTS` in the template for the public host and
+`COOKIE_SECURE=true` for HTTPS.
+
+### Add an example
+
+Follow `docs/examples.md`: `README.md`, `manifests/NN-*.yaml`, optional
+`manifests/exposure/` with a README, optional components with Dockerfile +
+README + tests, and executable, idempotent, self-cleaning scripts. Add the
+literal `examples/<name>/` entry to `examples/README.md`; `make check` enforces
+the layout. Publish component images under
+`<registry>/<owner>/<repo>/<example>/<component>`.
+
+### Bump a version
+
+- **DSH release identity:** `make set-version DSH_VERSION=<ref>
+  [RELEASE_REVISION=rN]`, then `make check`, then build and smoke-test the image.
+  Tags are immutable; fix a bad release with `-r(N+1)`.
+- **agent-sandbox:** edit `AGENT_SANDBOX_VERSION` in `versions.env`, update the
+  prerequisites in the example README, and note behavior changes (connection
+  types, MCP output limits, CRD fields). Install the new release YAML on a kind
+  cluster, then run `make check` and `make e2e`. Do not bump it inside an
+  unrelated PR.
+
+### Integrate the agent-sandbox MCP server
+
+Build from the cluster's pinned tag, give it a dedicated namespace and
+least-privilege ServiceAccount, keep it ClusterIP behind an authenticating
+proxy, and document how to reach `/readyz`. Add a DSH profile row only with user
+approval; a row affects every session. Never commit its credentials or a
+`cluster-admin` binding.
+
+### Documentation
+
+Update the doc whose behavior changed in the same commit: root README for the
+front door, `docs/` for repository-wide contracts, the example README for that
+walkthrough, the gateway README for API/config/auth. Skill docs live in
+`.dsh/skills/`; keep `SKILL.md` under ~8 KB and put depth in `references/`.
+`scripts/check-versions.sh` scans every `*.md` for stale `dsh-v…` tokens, so
+refer to "the release identity in `versions.env`" rather than copying a tag.
+
+## Verification and review
+
+Run at least `make check`; add `make test` and `make lint` for gateway/script
+changes. For manifests, use `kubectl apply --dry-run=server` against a cluster
+with the CRDs. For behavior changes, run the example or `make e2e`.
+
+Before reporting, review:
+
+- [ ] Diff is scoped; no generated files or unrelated changes.
+- [ ] Docs and examples match the new behavior.
+- [ ] Pins come only from `versions.env`; no stale version tokens.
+- [ ] No secrets or usable credentials; secret-shaped files are `.example`.
+- [ ] Manifests parse; exposure variants remain mutually exclusive.
+- [ ] Gateway tests and public contract unchanged or intentionally updated.
+- [ ] Skills: frontmatter valid, body within budget, scripts shellcheck-clean.
+- [ ] `CHANGELOG.md` untouched; branch/commit follow `docs/gitflow.md`.
+
+## Safe follow-ups enabled by this skill
+
+- A read-only preflight job or scheduled report using
+  `scripts/platform-preflight.sh`.
+- An MCP readiness check in the example e2e (health endpoints only).
+- A separate runtime warm pool for untrusted execution, leaving the DSH template
+  unchanged.
+- A combined DSH + sandboxd image only with an explicit maintenance/exec design
+  and a documented trust boundary.
--- a/.dsh/skills/dsh-agent-sandbox-platform/references/mcp-deployment.md
+++ b/.dsh/skills/dsh-agent-sandbox-platform/references/mcp-deployment.md
@@ -0,0 +1,94 @@
+# Deploying the agent-sandbox MCP server
+
+Read [the MCP reference](mcp-server.md) first. This file covers the connection
+matrix, a safe deployment shape, and the DSH profile row. The server is
+unauthenticated; its service account is the security boundary.
+
+## Connection matrix
+
+| `K8S_SANDBOX_CONNECTION__TYPE` | Required env | Use when |
+| --- | --- | --- |
+| `gateway` | `GATEWAY_NAME`, `GATEWAY_NAMESPACE=default`, `GATEWAY_READY_TIMEOUT=180`, `SERVER_PORT=8888` | Gateway API + implementation exist |
+| `direct` | `API_URL` (router URL), `SERVER_PORT=8888` | sandbox-router is deployed |
+| `in-cluster` | `USE_POD_IP=false` (DNS) or `true` (`status.podIPs`), `SERVER_PORT=8888` | Bypass the router from inside the cluster |
+| `sandboxd-in-cluster` (v1.0.8+) | `MODE=in-cluster-service` or `in-cluster-pod-ip`, `REST_PORT=8080`, `GRPC_PORT=9090` | Template runs sandboxd |
+
+Also set `K8S_SANDBOX_PROBE_NAMESPACE` to a namespace the server's service
+account can list claims in, and optionally
+`K8S_SANDBOX_SESSION_ID_LABEL_KEY` (default
+`mcp.k8s-agent-sandbox/session-id`).
+
+For `sandboxd-in-cluster` service mode the template needs `spec.service: true`.
+The default managed NetworkPolicy admits only sandbox-router, so add an ingress
+rule for the MCP server on 8080/9090. Supplying `spec.networkPolicy` replaces the
+secure defaults: also add the egress rules the sandbox needs, because an omitted
+egress list denies all outbound traffic. RBAC for that mode is
+`sandboxclaims` create/get/list/watch/delete and `sandboxes` get/list/watch; no
+`pods/portforward`.
+
+## Build and deploy (sketch)
+
+No official image reference is documented in the upstream README; pin what you
+build.
+
+```bash
+# AGENT_SANDBOX_VERSION comes from versions.env; never hard-code it.
+tmp="$(mktemp -d)"; trap 'rm -rf "$tmp"' EXIT
+git clone --depth 1 --branch "$AGENT_SANDBOX_VERSION" \
+  https://github.com/kubernetes-sigs/agent-sandbox.git "$tmp"
+docker build -t "$MCP_IMAGE" "$tmp/clients/integrations/mcp-server"
+```
+
+Kubernetes shape:
+
+- Dedicated namespace and ServiceAccount; a namespace-scoped Role granting only
+  `sandboxclaims` create/get/list/watch/delete and `sandboxes` get/list/watch.
+  No Secrets, no `pods/exec`, no cluster-admin.
+- Deployment: one replica, non-root, `readOnlyRootFilesystem`, dropped
+  capabilities, `K8S_SANDBOX_CONNECTION__TYPE=in-cluster`,
+  `K8S_SANDBOX_PROBE_NAMESPACE`, resource limits.
+- Probes: `GET /healthz` (liveness; never calls the API) and `GET /readyz`
+  (readiness; lists claims; use `periodSeconds: 15`, not aggressive polling).
+- ClusterIP Service on `8000`. Add a NetworkPolicy that allows only the intended
+  client namespace/pod. Never expose it through Ingress/LoadBalancer without an
+  authenticating proxy. The service account is the real boundary.
+
+## Connect it to DSH
+
+Add a profile MCP row (through the Plugins page/`plugin_manager`; see
+`cordis-plugin-development`). A row affects every session and needs approval.
+
+```yaml
+- insert:
+    - id: mcp-agent-sandbox
+      name: '@deepseek-ai/dsh-mcp-client'
+      config:
+        serverName: agent-sandbox
+        transport: streamable-http
+        url: http://agent-sandbox-mcp.example.svc.cluster.local:8000/mcp
+        toolCallTimeoutMs: 240000
+        failOnStartupError: false
+```
+
+- `serverName` must match `[A-Za-z0-9_-]{1,32}`; tools appear as
+  `mcp__<serverName>__<tool>`. Check the actual names in the tool list; do not
+  assume hyphens/underscores.
+- Raise `toolCallTimeoutMs` above `sandbox_ready_timeout` with headroom; the DSH
+  default is 60 s and would cancel a 180 s creation.
+- If the server is behind an authenticating proxy, pass its token via the row's
+  `headers` and an environment expression; never commit it.
+- To reach a locally port-forwarded server, use a loopback URL; the MCP session
+  still needs streamable HTTP.
+- Remove the row when it was enabled only for one task.
+
+## Deployment safety checklist
+
+- [ ] Build tag equals the cluster's `AGENT_SANDBOX_VERSION`.
+- [ ] Dedicated namespace/SA; RBAC limited to claims and sandboxes.
+- [ ] Service is ClusterIP; no public route without authentication.
+- [ ] NetworkPolicy admits only the DSH client and needed sandbox traffic.
+- [ ] `/readyz` succeeds with `K8S_SANDBOX_PROBE_NAMESPACE` set to a listable
+      namespace.
+- [ ] No Secret data in manifests, logs, or the repository.
+- [ ] Cleanup path documented: delete session-owned claims or wait for the
+      shutdown timeout.
--- a/.dsh/skills/dsh-agent-sandbox-platform/references/mcp-server.md
+++ b/.dsh/skills/dsh-agent-sandbox-platform/references/mcp-server.md
@@ -0,0 +1,103 @@
+# agent-sandbox MCP server reference
+
+Upstream: [clients/integrations/mcp-server](https://github.com/kubernetes-sigs/agent-sandbox/tree/main/clients/integrations/mcp-server).
+It is a Python/FastAPI + FastMCP service (streamable HTTP on `:8000`, MCP
+endpoint `/mcp`) that provisions sandboxes from a warm pool and runs commands or
+file operations through a sandbox runtime. It ships with **no authentication**.
+
+Build the server from the same tag as the cluster: read `AGENT_SANDBOX_VERSION`
+in `versions.env`, and check the installed controller image. Do not build from
+`main` against an older cluster; tool schemas and connection types differ.
+
+## Tool surface
+
+| Tool | Arguments (defaults) | Returns / notes |
+| --- | --- | --- |
+| `create_sandbox` | `warmpool`, `namespace`, `sandbox_ready_timeout=180`, `labels`, `shutdown_after_seconds=300` (`null` disables), `pod_labels`, `pod_annotations` | `sandbox_claim_name`; adds the MCP session label |
+| `delete_sandbox` | `sandbox_claim_name`, `namespace` | Deletes the session-owned claim; PVC cascades |
+| `execute_command` | `sandbox_claim_name`, `namespace`, `command`, `timeout=60` (max 600), `max_output_chars` only on v1.0.8+ (128-300,000, default 30,000) | `exit_code`, `stdout`, `stderr`; v1.0.8 adds `*_total_chars` and `*_truncated` |
+| `upload_file` | `sandbox_claim_name`, `namespace`, `path`, `content`, `binary=false`, `timeout=60` | Bytes written; base64 when `binary=true` |
+| `download_file` | `sandbox_claim_name`, `namespace`, `path`, `binary=false`, `timeout=60` | Content + bytes read |
+| `list_files` | `sandbox_claim_name`, `namespace`, `path`, `timeout=60`, `max_entries=1000` (max 10,000) | Entries (`name`, `size`, `type`, `mod_time`), `total_entries`, `truncated` |
+| `file_exists` | `sandbox_claim_name`, `namespace`, `path`, `timeout=60` | `exists`; a missing path is not an error |
+| `get_sandbox_status` | `sandbox_claim_name`, `namespace` | `SandboxReady` / `SandboxNotReady` / `SandboxNotFound`, `ready`, `message` |
+
+Resource: `sandboxes://{namespace}` lists sandboxes for the current MCP session.
+
+## Session scoping is the most important behavior
+
+Every tool, including `get_sandbox_status`, calls `get_sandbox()` ->
+`ensure_session_owns()`, which lists sandboxes with the label selector
+`<K8S_SANDBOX_SESSION_ID_LABEL_KEY>=<MCP session id>`. The resource uses the same
+selector. Consequences:
+
+- The server can only see/exec/delete claims **it created in the same MCP
+  session**. It cannot inspect or manage DSH platform claims created by the
+  gateway or `kubectl`, or claims from another session.
+- On a DSH MCP reconnect, the MCP session changes; earlier claims become
+  invisible to the tools. They keep running until `shutdown_after_seconds`
+  expires, or until you remove them by label with a confirmed `kubectl delete`.
+- A "Sandbox claim '<name>' is not found in namespace '<ns>'" error usually
+  means wrong namespace **or** the claim belongs to a different session.
+- Use `kubectl` for global inventory; use the MCP only for sandboxes it created
+  in this task.
+
+Session identity requires a session-providing transport. With stdio,
+`ctx.session_id` is `None` and creation fails. Connect DSH over
+`streamable-http`.
+
+## Runtime prerequisite
+
+The server talks to a sandbox runtime, not to arbitrary pods:
+
+- `gateway`, `direct`, `in-cluster`: runtime HTTP on `SERVER_PORT=8888`
+  (sandbox-router API or direct pod runtime).
+- `sandboxd-in-cluster` (v1.0.8+): sandboxd REST `8080`, gRPC `9090`.
+
+The shipped DSH template runs only DSH Web on `3080`, so no connection type can
+exec into it. A sandboxd **sidecar** executes commands in the sidecar's own
+container, not in the DSH container; shared volumes share files, not binaries or
+processes. To exec inside DSH, build a combined image (sandboxd injected into the
+DSH image) or use `kubectl exec` with explicit RBAC and confirmation. For
+untrusted code, deploy a separate runtime warm pool rather than modifying the
+DSH template.
+
+## Version differences
+
+| Area | Before v1.0.8 | v1.0.8+ |
+| --- | --- | --- |
+| Connection types | `gateway`, `direct`, `in-cluster` | Adds `sandboxd-in-cluster` |
+| `execute_command` output | Unbounded | Bounded by `max_output_chars`; four metadata fields |
+| Failed pod recovery | `Retain` only | `spec.failedPodPolicy: Recreate` |
+| Router install | `kubectl apply -f sandbox-router/deploy/` | `kubectl apply -k sandbox-router/deploy/`; projected token |
+
+Never pass `max_output_chars` to a pre-v1.0.8 server. On older servers, redirect
+verbose output to a file and `download_file` it, or pipe through `tail`/`head`.
+
+## Safety and cleanup
+
+- No auth: anyone reaching `:8000` gets every tool and the service account's
+  access. Keep it ClusterIP, authenticated, and namespace-scoped.
+- Treat commands as untrusted code: they can read files and environment
+  variables inside the sandbox, including provider keys in a combined image.
+- `delete_sandbox` cascades the claim, sandbox, and PVC: confirm the target and
+  namespace. Never delete a namespace or PVC as cleanup.
+- Clean up in the same session that created the sandbox. If the session is gone,
+  list claims by the session label and delete them explicitly (requires write
+  RBAC); otherwise rely on the shutdown timeout.
+
+## Troubleshooting
+
+| Symptom | Cause / fix |
+| --- | --- |
+| Tools missing | Row not enabled, wrong URL, connection failed; check logs and `/readyz` |
+| 404 on connect | URL is missing `/mcp` or points at the wrong port |
+| `ctx.session_id is None` | Connected over stdio; use streamable HTTP |
+| `Sandbox claim ... is not found` | Wrong namespace, or claim created by another MCP session |
+| Creation times out at ~60 s | MCP session alive but DSH cancelled the call; raise `toolCallTimeoutMs` |
+| `execute_command` connection error | No runtime on 8888/8080/9090, or wrong connection type; the DSH template is unsupported |
+| Output flood | Pre-v1.0.8 server; bound the command output yourself |
+| Empty sandbox list | The resource is session-scoped; use `kubectl` for global inventory |
+
+Deployment, connection matrix, RBAC, and the DSH row:
+[MCP deployment](mcp-deployment.md).
--- a/.dsh/skills/dsh-agent-sandbox-platform/references/platform-runbook.md
+++ b/.dsh/skills/dsh-agent-sandbox-platform/references/platform-runbook.md
@@ -0,0 +1,176 @@
+# Platform runbook
+
+Scope: the resource layer and gateway in
+`examples/dsh-agents-as-a-service/`, one namespace, one DSH Web sandbox per
+user. Read the example README and `AGENTS.md` first. All commands are read-only
+unless a section is explicitly a change; confirm before changing anything.
+
+## 1. Context, version, CRDs
+
+```bash
+kubectl config current-context
+kubectl auth whoami
+kubectl config view --minify -o jsonpath='{..namespace}{"\n"}'
+kubectl get crd | grep -E 'agents\.x-k8s\.io'
+kubectl -n agent-sandbox-system get deploy agent-sandbox-controller \
+  -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
+```
+
+Compare the controller image tag with `AGENT_SANDBOX_VERSION` in
+`versions.env`. The example needs the extension CRDs plus core `sandboxes`;
+a missing CRD is a prerequisite failure, not an application bug.
+
+## 2. Resource model
+
+| Object | Role | Key fields |
+| --- | --- | --- |
+| `SandboxTemplate` | Blueprint; pinned image, probes, security context, PVC template | `envVarsInjectionPolicy`, `volumeClaimTemplatesPolicy`, `service`, `podTemplate` |
+| `SandboxWarmPool` | Pre-warmed spares from one template | `spec.replicas`, `sandboxTemplateRef` |
+| `SandboxClaim` | One tenant/user request; adopts a spare | `spec.warmPoolRef`, `additionalPodMetadata`, `status.sandbox.name` |
+| `Sandbox` | Adopted or directly created pod + Service + PVC | `spec.operatingMode`, `status.conditions`, `status.serviceFQDN`, `status.podIPs` |
+| `Pod` | The DSH Web process | `:3080`, PVC at `/home/node/.dsh` |
+| `PersistentVolumeClaim` | Per-sandbox state | `dsh-home-<sandbox>` |
+| `Service` | Stable per-sandbox DNS | `<sandbox>`, `status.serviceFQDN` |
+
+Delete a claim to cascade-delete its sandbox and PVC (account deletion). Never
+delete a PVC or namespace without an explicit, named confirmation.
+
+## 3. Read-only inspection
+
+```bash
+kubectl -n "$NS" get sandboxtemplates,sandboxwarmpools,sandboxclaims,sandboxes -o wide
+kubectl -n "$NS" get pods,pvc,svc -o wide
+kubectl -n "$NS" get events --sort-by=.lastTimestamp | tail -30
+kubectl -n "$NS" get sandboxclaim "$CLAIM" -o yaml
+kubectl -n "$NS" get sandbox "$SB" -o yaml
+```
+
+Bounded logs: `kubectl -n "$NS" logs "$POD" --tail=200` (add `--previous`
+only after a restart).
+
+## 4. Claim -> sandbox -> pod
+
+```bash
+SB=$(kubectl -n "$NS" get sandboxclaim "$CLAIM" -o jsonpath='{.status.sandbox.name}')
+POD=$(kubectl -n "$NS" get sandbox "$SB" -o jsonpath='{.metadata.annotations.agents\.x-k8s\.io/pod-name}')
+POD=${POD:-$SB}
+kubectl -n "$NS" get sandbox "$SB" -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}{"\n"}{end}'
+kubectl -n "$NS" get sandbox "$SB" -o jsonpath='{.spec.operatingMode}{" "}{.status.serviceFQDN}{" "}{.status.podIPs}{"\n"}'
+```
+
+Rules:
+
+- Resolve user -> sandbox only through `claim.status.sandbox.name`; the sandbox
+  keeps its pool-generated name, not the claim name.
+- A claim tracks binding; for runtime state read `Sandbox.spec.operatingMode`
+  and the `Suspended`/`Ready` conditions. A suspended sandbox is not serving
+  even when the claim still exists.
+- A missing `claim.status.sandbox.name` means the claim did not bind: inspect
+  the Ready condition `reason` and events. The pod is normally named after the
+  sandbox; use the annotation when present.
+
+## 5. Suspend, resume, wake
+
+Suspend keeps the PVC and Service, deletes the pod:
+
+```bash
+kubectl -n "$NS" patch sandbox "$SB" --type merge \
+  -p '{"spec":{"operatingMode":"Suspended"}}'
+kubectl -n "$NS" wait --for=delete "pod/$POD" --timeout=120s
+kubectl -n "$NS" get pvc "dsh-home-$SB"
+kubectl -n "$NS" get svc "$SB"
+```
+
+Resume and verify state:
+
+```bash
+kubectl -n "$NS" patch sandbox "$SB" --type merge \
+  -p '{"spec":{"operatingMode":"Running"}}'
+kubectl -n "$NS" wait sandbox "$SB" --for=condition=Ready --timeout=180s
+kubectl -n "$NS" exec "$SB" -- cat /home/node/.dsh/marker.txt
+```
+
+The gateway wakes a suspended sandbox on the first request. The idle sweeper
+only suspends claims it created (`aaas.example.com/managed-by=dsh-gateway`);
+hand-applied claims are never swept.
+
+## 6. Warm pool
+
+```bash
+kubectl -n "$NS" get sandboxwarmpool "$POOL" -o yaml
+kubectl -n "$NS" get sandboxes -l agents.x-k8s.io/warm-pool-sandbox -o wide
+kubectl -n "$NS" get sandboxwarmpool "$POOL" -o jsonpath='{.spec.replicas}{"\n"}'
+```
+
+Adopted sandboxes lose the warm-pool label; count current spares against
+`spec.replicas`. Zero spares or `spec.replicas: 0` means cold, slow signup.
+Scale only when asked:
+
+```bash
+kubectl -n "$NS" patch sandboxwarmpool "$POOL" --type merge -p '{"spec":{"replicas":2}}'
+```
+
+## 7. DSH runtime specifics
+
+- DSH Web listens on `3080` in-cluster. Port-forward to a free local port such
+  as `18081`, never `3080` if a local DSH already uses it.
+- The launch token is printed once as `dsh web: http://<ip>:3080/?token=...`.
+  Treat it as a credential: do not paste it into reports; hand the user the URL.
+- `/` returns `401` without the token. Probes must target `/favicon.svg`.
+- `DSH_TRUSTED_HOSTS` must include any public host that a proxy forwards; DSH
+  fences on `Host`/`Origin`.
+- State lives at `/home/node/.dsh`. DSH requires `.credentials.yaml` to be
+  `0600`; the template uses `fsGroupChangePolicy: OnRootMismatch`. If a resumed
+  sandbox crash-loops with a mode-660 error, suspend it, fix the file with a
+  throwaway pod, patch the policy, then resume.
+
+## 8. Gateway layer
+
+```bash
+kubectl -n "$NS" get deploy,svc,pods -l app=dsh-gateway -o wide
+kubectl -n "$NS" get endpoints dsh-gateway
+kubectl -n "$NS" logs deploy/dsh-gateway --tail=100
+```
+
+The gateway is a ClusterIP Service on `8080`; it proxies under `/u/{user}/` and
+derives state (`Provisioning`, `Waking`, `Ready`, `Suspending`, `Suspended`).
+It performs the DSH launch-token handshake server-side. With SSO enabled
+(`AUTH_PROXY_HEADER`), the Service must be reachable only through the
+authenticating proxy; never add a second path or port-forward it on a shared
+host. A resource-only install has no gateway: that is a warning, not a failure.
+
+## 9. Security boundaries
+
+- One `DEEPSEEK_API_KEY` is shared by all sandboxes; the example is not
+  tenant-isolated.
+- Secret data, kubeconfigs, and pod logs are sensitive; RBAC may allow Secret
+  access, policy does not. Keep gateway RBAC namespace-scoped; it needs pod logs
+  to find the launch token.
+- TLS terminates outside the gateway; set `COOKIE_SECURE=true` for HTTPS.
+
+## 10. Failure matrix
+
+| Symptom | First evidence | Likely cause / fix |
+| --- | --- | --- |
+| Claim not Ready, no sandbox | Ready condition reason, events | Empty pool, image pull, quota, unschedulable |
+| `EnvVarsInjectionRejected` | claim Ready reason | Claim carries env/volume templates; template policy rejects it |
+| Pod Pending | `kubectl describe pod` | Requests/taints/affinity, unbound PVC |
+| CrashLoopBackOff | logs, `--previous` | Startup failure, bad config, probe 401, fsGroup mode 660 |
+| ImagePullBackOff | describe events | Wrong tag, private registry, missing pull secret |
+| DSH 401 in browser | gateway logs | Launch-token handshake failed; check gateway RBAC `pods/log` |
+| 502/503 through gateway | sandbox Ready, gateway logs | Wake timeout, DNS stale, token not in logs yet |
+| Warm pool empty | sandboxes label count | Scale the pool or wait for replenishment |
+| Suspended but expected running | `operatingMode` | Resume it; the gateway only wakes on a request |
+| WS closes `1011` | gateway logs | DSH rejected the upstream Host/authority; check trusted hosts |
+
+## 11. Cleanup and reporting
+
+Delete only claims/sandboxes you created, by name:
+
+```bash
+kubectl -n "$NS" delete sandboxclaim "$CLAIM" --dry-run=server
+kubectl -n "$NS" delete sandboxclaim "$CLAIM"
+```
+
+Report context/namespace, evidence, conclusion, change, rollback, and what was
+deleted. Leave the namespace, pools, templates, and unrelated claims alone.
--- a/.dsh/skills/dsh-agent-sandbox-platform/references/troubleshooting.md
+++ b/.dsh/skills/dsh-agent-sandbox-platform/references/troubleshooting.md
@@ -0,0 +1,71 @@
+# Troubleshooting
+
+Start with evidence, not fixes: resolve context/namespace, read conditions and
+events, then compare desired vs observed state. Keep output bounded. Never
+"fix" an access error by broadening credentials.
+
+## Platform
+
+| Symptom | First evidence | Likely cause / fix |
+| --- | --- | --- |
+| `Forbidden` on every call | `kubectl auth whoami`, `kubectl auth can-i --list -n <ns>` | Wrong context or RBAC. Do not retry with broader credentials; ask for the right kubeconfig. |
+| CRDs not found | `kubectl get crd | grep agents.x-k8s.io` | agent-sandbox extensions not installed. Install the pinned release YAML; that is a prerequisite, not a code change. |
+| Claim created, no sandbox | `claim.status.conditions[?(@.type=="Ready")]`, events | Empty pool, image pull, quota, unschedulable; or policy rejection. |
+| Claim rejected | Ready reason `EnvVarsInjectionRejected` | Claim carries `env`/`volumeClaimTemplates` while the template disallows injection. Remove them; do not bypass the pool. |
+| User is suspended but the claim exists | `Sandbox.spec.operatingMode`, `Suspended` condition | Resume with `operatingMode: Running`; a gateway request would wake it. |
+| Warm pool exhausted | `kubectl get sandboxes -l agents.x-k8s.io/warm-pool-sandbox` | Scale `spec.replicas` (with confirmation) or accept slow cold provisioning. |
+| Pod `Pending` | `kubectl describe pod` | Requests/limits, taints, affinity, unbound PVC, storage class. |
+| `ImagePullBackOff` | describe events | Wrong image/tag, private GHCR package, missing pull secret; on kind, load the image under the exact manifest ref. |
+| `CrashLoopBackOff`, probe 401 | logs, `describe` | Probe path is `/`; use `/favicon.svg`. |
+| Restart with mode-660 error | logs mention `.credentials.yaml` | kubelet fsGroup widened the file; suspend, `chmod 600` from a throwaway pod, patch `fsGroupChangePolicy: OnRootMismatch`, resume. |
+| PVC `Pending` | `kubectl describe pvc`, StorageClass | No default provisioner, access mode, zone binding. |
+| Launch token not found | `kubectl logs <pod> --tail=2000 | grep 'dsh web:'` | Pod not ready yet, or logs rotated. Do not guess tokens; wait for readiness. |
+| Hand-applied sandbox disappeared | claim annotations, gateway logs | Idle sweeper only manages claims with `aaas.example.com/managed-by=dsh-gateway`; if it was managed, recreate/resume. |
+| Gateway 401 | gateway logs, cookie flags | Missing/expired cookie, DSH handshake failed, or `COOKIE_SECURE=true` over HTTP. |
+| Gateway 502/503 | sandbox Ready, gateway logs | Wake timeout, stale DNS, token not yet in logs; check `WAKE_TIMEOUT` and RBAC `pods/log`. |
+| WebSocket closes 1011 | gateway logs, `Host`/Origin | DSH rejected the upstream authority; check `DSH_TRUSTED_HOSTS` and the gateway's pod-IP dial path. |
+| WebSocket closes 4401 | cookie, Origin | Missing gateway auth or cross-origin upgrade; send a matching `Origin`. |
+| SSO header ignored | gateway env | `AUTH_PROXY_HEADER` unset; setting it requires the proxy to be the only path to the ClusterIP Service. |
+| Someone is treated as another user | service paths | Trust-boundary violation: the gateway is directly reachable. Remove the extra path and disable SSO until fixed. |
+
+## Agent-sandbox MCP
+
+| Symptom | First evidence | Likely cause / fix |
+| --- | --- | --- |
+| No `mcp__*__create_sandbox` tools | tool list, DSH logs | Server not deployed or row not enabled. Use `kubectl`; connect only with approval. |
+| 404 on connect | curl `/mcp`, `/healthz` | Wrong path/port; MCP endpoint is `/mcp`, health is at the root. |
+| `/readyz` 503 | response body, server logs | API unreachable or namespace `K8S_SANDBOX_PROBE_NAMESPACE` not listable; fix RBAC/config. |
+| `ctx.session_id is None` | server error | Connected over stdio; use streamable HTTP. |
+| `Sandbox claim ... is not found` | server error, `kubectl get claim` | Wrong namespace, or the claim belongs to another MCP session. List with `kubectl`; use the MCP only for its own claims. |
+| Creation cancelled at ~60 s | MCP call log | DSH `toolCallTimeoutMs` is 60 s while creation waits up to 180 s; raise the timeout. |
+| `execute_command` connection error | server logs, template ports | No runtime on 8888/8080/9090. The DSH template only listens on 3080; use a runtime template or a combined image. |
+| Output truncated / flooded | tool result | Pre-v1.0.8 is unbounded; v1.0.8 bounds output. Redirect verbose output to a file and read/download it; never pass `max_output_chars` to an old server. |
+| Orphaned claims after reconnect | `kubectl get claims -l mcp.k8s-agent-sandbox/session-id` | MCP sessions are transient; wait for `shutdown_after_seconds` or delete by label with explicit confirmation. |
+| `Forbidden` from the MCP server | server logs | Its ServiceAccount lacks claim/sandbox verbs, or RBAC is in another namespace; fix the Role, not the kubeconfig. |
+| Sandbox expired | `get_sandbox_status` -> `SandboxNotFound` | `shutdown_after_seconds` elapsed; create a new sandbox and keep the task inside the budget. |
+
+## Repository checks and CI
+
+| Symptom | Cause / fix |
+| --- | --- |
+| `check-versions: stale version reference in docs: dsh-v...` | A copied release tag in a Markdown file. Refer to the identity in `versions.env`; run `make set-version` only for a real bump. |
+| `check-versions: hard-coded version outside versions.env` | A tag in `.github/workflows` or the Makefile; move it to `versions.env`. |
+| `check-examples` layout error | Missing `README.md`/`manifests/`, stray YAML at the example root, missing exposure README, non-executable script, or missing `examples/README.md` entry. |
+| Manifest dry-run fails in CI | Invalid YAML or CRD schema; parse locally and run `kubectl apply --dry-run=server` against the pinned CRDs. |
+| `shellcheck` failure | Quote expansions, use arrays for kubectl args, avoid unused variables; run shellcheck locally. |
+| Lychee link failure | A relative/absolute link does not resolve; the link checker runs offline on the listed docs. |
+| PR-title check failure | Use Conventional Commits: `feat(scope): ...`, `fix(scope): ...`, `docs(scope): ...`. |
+| Changelog conflict | Never edit `CHANGELOG.md`; it is generated from commits. |
+
+## Do not
+
+- Do not use `--force --grace-period=0` to clear a stuck resource before
+  understanding why it is stuck.
+- Do not delete namespaces, PVCs, PVs, CRDs, or cluster-scoped objects as a
+  cleanup shortcut.
+- Do not read Secret data, kubeconfigs, or node logs to "debug" a permission
+  problem, and do not paste pod logs containing the DSH launch token into a
+  report.
+- Do not expose the unauthenticated MCP server or enable `AUTH_PROXY_HEADER`
+  while another path to the gateway exists.
+- Do not retry a `Forbidden` with a stronger kubeconfig; fix the RBAC scope.