mirror of
https://github.com/sredevopsorg/deepseek-kubernetes-sandbox.git
synced 2026-11-03 01:13:24 +00:00
feat(skills): add agent-sandbox platform skill (#20)
Add the project-local dsh-agent-sandbox-platform skill covering platform operations, the optional agent-sandbox MCP server, feature-extension recipes, and a read-only preflight. Document project skills in README, docs and AGENTS.md. Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
This commit is contained in:
151
.dsh/skills/dsh-agent-sandbox-platform/SKILL.md
Normal file
151
.dsh/skills/dsh-agent-sandbox-platform/SKILL.md
Normal file
@@ -0,0 +1,151 @@
|
||||
---
|
||||
name: dsh-agent-sandbox-platform
|
||||
description: >-
|
||||
Operate and extend this repository's DeepSeek Harness-on-agent-sandbox platform:
|
||||
warm pools, claims, sandbox suspend/resume, the gateway, and the optional
|
||||
agent-sandbox MCP server, with version-aware safety and a read-only preflight.
|
||||
whenToUse: >-
|
||||
Use when working in deepseek-kubernetes-sandbox or with Kubernetes agent-sandbox:
|
||||
inspect or provision sandboxes, debug warm pools or claims, use
|
||||
mcp__*__create_sandbox/execute_command, connect the agent-sandbox MCP server, or
|
||||
add a manifest, gateway route, exposure variant, example, or version bump here.
|
||||
metadata:
|
||||
author: sredevopsorg
|
||||
version: "1.0.0"
|
||||
---
|
||||
|
||||
# DSH agent-sandbox platform
|
||||
|
||||
This repository runs DeepSeek Harness Web (DSH) as one sandbox per user on
|
||||
[agent-sandbox](https://github.com/kubernetes-sigs/agent-sandbox), with an
|
||||
optional official MCP server for sandbox execution. For generic cluster work
|
||||
load `kubernetes-kubectl-operations`; for profile MCP rows load
|
||||
`cordis-plugin-development`.
|
||||
|
||||
## 0. Facts before action
|
||||
|
||||
1. Read `AGENTS.md`, `versions.env`, the example README/architecture. The
|
||||
control-plane pin is `AGENT_SANDBOX_VERSION`; release identity is
|
||||
`DSH_VERSION` + `RELEASE_REVISION`.
|
||||
2. Establish context, user and namespace; never assume `dsh-demo`.
|
||||
3. Run `scripts/platform-preflight.sh --namespace <ns>` (read-only).
|
||||
4. If `mcp__*__create_sandbox`-style tools are absent, use `kubectl`; do not
|
||||
install or enable a server on your own.
|
||||
|
||||
## 1. Non-negotiables
|
||||
|
||||
- **Read-only by default.** Change a cluster only when the user asked for that
|
||||
exact change; show the command and rollback first.
|
||||
- **Scope every command** with `--context` and `-n`. `--all`/`-A` mutations and
|
||||
deleting namespaces, PVCs, PVs, CRDs, or cluster-scoped objects need an
|
||||
explicit, named confirmation.
|
||||
- **Secrets and logs stay private.** Never print Secret data, kubeconfigs, or
|
||||
node logs. Pod logs can contain the DSH launch token.
|
||||
- **The MCP server has no authentication.** It is a full-access management
|
||||
endpoint; never expose it. A DSH profile row affects every session.
|
||||
- **Clean up what you create.** Delete MCP-created sandboxes in the same
|
||||
session; never leave orphaned claims running.
|
||||
- **Bound output.** Use `--tail`, `--since`, selectors, JSONPath, and
|
||||
head/tail excerpts instead of whole manifests or huge logs.
|
||||
|
||||
## 2. Choose the path
|
||||
|
||||
| Need | Path |
|
||||
| --- | --- |
|
||||
| Inspect or change claims, pools, sandboxes, PVCs, gateway | `kubectl` + [platform runbook](references/platform-runbook.md) |
|
||||
| Run code or move files in a sandbox | agent-sandbox MCP, only for a runtime-capable template ([MCP reference](references/mcp-server.md)) |
|
||||
| Add or change a repository feature | [repo extension guide](references/extending-the-repo.md) |
|
||||
|
||||
The MCP server cannot manage this example's DSH claims: its tools are scoped to
|
||||
claims created by the same MCP session. Use `kubectl` for the platform.
|
||||
|
||||
## 3. Operate the platform
|
||||
|
||||
Resolve user -> claim -> sandbox -> pod:
|
||||
|
||||
```bash
|
||||
SB=$(kubectl -n "$NS" get sandboxclaim "$CLAIM" -o jsonpath='{.status.sandbox.name}')
|
||||
kubectl -n "$NS" get sandbox "$SB" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}{" "}{.status.conditions[?(@.type=="Ready")].reason}{"\n"}{.status.serviceFQDN}{"\n"}'
|
||||
POD=$(kubectl -n "$NS" get sandbox "$SB" -o jsonpath='{.metadata.annotations.agents\.x-k8s\.io/pod-name}'); POD=${POD:-$SB}
|
||||
```
|
||||
|
||||
- Read runtime state from the Sandbox, not the claim: `spec.operatingMode`,
|
||||
`Suspended`, `Ready`. The claim tracks binding, not liveness.
|
||||
- Suspend = patch `operatingMode: Suspended` (pod deleted, PVC + Service kept);
|
||||
resume = `Running` + wait for `Ready`. The gateway wakes on connect.
|
||||
- Warm-pool spares carry `agents.x-k8s.io/warm-pool-sandbox`; adopted sandboxes
|
||||
lose it. A pool with no spares makes signup cold and slow.
|
||||
- The DSH launch token appears once in pod logs; hand the user the `?token=` URL,
|
||||
never the token in reports. Probe `/favicon.svg`.
|
||||
- The idle sweeper only suspends claims the gateway created.
|
||||
|
||||
Full commands, security boundaries and failure matrix:
|
||||
[platform runbook](references/platform-runbook.md).
|
||||
|
||||
## 4. Use the agent-sandbox MCP server within its limits
|
||||
|
||||
Verified from the upstream server:
|
||||
|
||||
- Tools: `create_sandbox`, `delete_sandbox`, `execute_command`, `upload_file`,
|
||||
`download_file`, `list_files`, `file_exists`, `get_sandbox_status`; resource
|
||||
`sandboxes://{namespace}`.
|
||||
- **Session-scoped:** every tool and the resource filter claims by
|
||||
`K8S_SANDBOX_SESSION_ID_LABEL_KEY=<MCP session id>`. It cannot read, exec in,
|
||||
or delete DSH platform claims or claims from a prior/reconnected session.
|
||||
- **Streamable HTTP only:** stdio has no MCP session id and fails. A DSH row must
|
||||
use `transport: streamable-http`; health endpoints are `/healthz` and `/readyz`
|
||||
at the root, the MCP endpoint is `/mcp`.
|
||||
- **Runtime required:** connection types target a sandbox runtime on 8888, or
|
||||
sandboxd on 8080/9090. The DSH template listens only on 3080, so
|
||||
`execute_command` cannot work against it. A sandboxd sidecar executes in its
|
||||
own container, not DSH's.
|
||||
- **Timeouts:** `create_sandbox` waits up to 180 s, but the DSH MCP client's
|
||||
default call timeout is 60 s; raise `toolCallTimeoutMs`.
|
||||
`shutdown_after_seconds` defaults to 300 s (`null` disables it).
|
||||
- **Output:** servers before v1.0.8 return unbounded `execute_command` output.
|
||||
v1.0.8 adds `max_output_chars` (128-300,000, default 30,000) and truncation
|
||||
metadata. Never pass the new argument to an older server; redirect verbose
|
||||
output and read a file instead.
|
||||
|
||||
Deployment, connection matrix and DSH row:
|
||||
[MCP deployment](references/mcp-deployment.md). Limits and cleanup:
|
||||
[MCP reference](references/mcp-server.md).
|
||||
|
||||
## 5. Extend the project
|
||||
|
||||
Read `docs/examples.md` and the testing table in `AGENTS.md` before editing.
|
||||
|
||||
- **Manifest step:** numbered `manifests/NN-*.yaml`, update the README step, run
|
||||
`make check`; CI dry-runs it.
|
||||
- **Gateway:** edit `gateway/gateway.py` and `gateway/test_gateway.py`, update
|
||||
the gateway README API table and architecture doc, run `make test` and
|
||||
`make lint`. Keep the HTTP/WebSocket contract stable.
|
||||
- **Exposure variant:** one variant under `manifests/exposure/<variant>/`,
|
||||
update the exposure README and trusted-host/cookie settings. Variants are
|
||||
alternatives, never a stack.
|
||||
- **New example:** follow `docs/examples.md`, add the index entry, keep scripts
|
||||
executable/idempotent/self-cleaning, run `make check`.
|
||||
- **Version bump:** edit `versions.env` only, update prerequisites, run
|
||||
`make check` and the e2e. Never mix a bump with an unrelated feature.
|
||||
- **MCP integration:** pin the build to the cluster's tag, least-privilege SA,
|
||||
ClusterIP + auth proxy, raised DSH timeout, no secrets in the repo.
|
||||
- **Docs:** update the affected doc in the same change.
|
||||
|
||||
Details and review checklist:
|
||||
[repo extension guide](references/extending-the-repo.md).
|
||||
|
||||
## 6. Report and clean up
|
||||
|
||||
State context/namespace, bounded evidence, conclusion, exact change, rollback,
|
||||
and resources deleted. Disable any MCP row enabled only for the task. Do not
|
||||
leave sandboxes, port-forwards, or temporary files behind.
|
||||
|
||||
## References
|
||||
|
||||
- [Platform runbook](references/platform-runbook.md)
|
||||
- [MCP deployment](references/mcp-deployment.md)
|
||||
- [MCP server reference](references/mcp-server.md)
|
||||
- [Repo extension guide](references/extending-the-repo.md)
|
||||
- [Troubleshooting](references/troubleshooting.md)
|
||||
- Upstream MCP server README:
|
||||
https://github.com/kubernetes-sigs/agent-sandbox/blob/main/clients/integrations/mcp-server/README.md
|
||||
@@ -0,0 +1,131 @@
|
||||
# Extending the repository
|
||||
|
||||
Read `AGENTS.md`, `docs/examples.md`, and the example README before editing.
|
||||
This repository is image-first: the Dockerfile is the primary artifact, every
|
||||
implementation is a self-contained example under `examples/<name>/`, and
|
||||
`versions.env` is the only source of pins. Documentation is part of the change.
|
||||
|
||||
## Invariants
|
||||
|
||||
- Never hard-code a version, image tag, or registry path; edit `versions.env`.
|
||||
For the DSH release identity use `make set-version`, never ad-hoc `sed`.
|
||||
- Never commit a secret. Secret-shaped manifests are `*.yaml.example` with
|
||||
placeholders; keys are documented only in `.env.sample`.
|
||||
- Manifests are numbered `NN-*.yaml` and apply in order with plain `kubectl`.
|
||||
Exposure variants live under `manifests/exposure/<variant>/`, are documented
|
||||
by one README, and are mutually exclusive.
|
||||
- Keep the gateway HTTP/WebSocket contract stable unless the change is explicitly
|
||||
about it.
|
||||
- Do not edit `CHANGELOG.md` (generated by git-cliff) or rewrite git history.
|
||||
- Keep diffs scoped; do not mix a version bump with an unrelated feature.
|
||||
|
||||
## Change matrix
|
||||
|
||||
| Change | Edit | Minimum verification |
|
||||
| --- | --- | --- |
|
||||
| Documentation only | `*.md`, `docs/` | `make check`; relative links resolve |
|
||||
| Example manifest | `examples/<name>/manifests/NN-*.yaml`, README step | `make check`; server-side dry-run in CI |
|
||||
| Gateway behavior/route | `gateway/gateway.py`, `gateway/test_gateway.py`, gateway README, architecture | `make test`, `make lint`; update tests and API docs |
|
||||
| Exposure variant | `manifests/exposure/<variant>/`, exposure README, example README | `make check`; apply exactly one variant |
|
||||
| New example | full `examples/<name>/` layout, `examples/README.md` | `make check`; CI dry-run; optional e2e |
|
||||
| Version bump | `versions.env`, affected docs/manifests | `make check`; CI installs the pin; e2e on a cluster |
|
||||
| MCP integration | deployment/RBAC, DSH row, docs | read-only `/readyz`; no secret in repo |
|
||||
| Workflow | `.github/workflows/` | actionlint; SHA-pinned actions; no hard-coded versions |
|
||||
|
||||
## Recipes
|
||||
|
||||
### Add or change a resource step
|
||||
|
||||
1. Keep numbering stable; add `NN-*.yaml` at the end unless order matters.
|
||||
2. Update the matching numbered step in the example README.
|
||||
3. Keep cluster-specific state (node affinity, tuned replica counts) out of the
|
||||
shipped manifest; document it as a local patch.
|
||||
4. Run `make check`. CI additionally parses every manifest and runs a
|
||||
server-side dry-run against the agent-sandbox CRDs.
|
||||
|
||||
### Change the gateway
|
||||
|
||||
The gateway is a single-file Starlette app. A change usually touches:
|
||||
|
||||
- `gateway/gateway.py` (route, auth, proxy, sweeper);
|
||||
- `gateway/test_gateway.py` (unit tests; no cluster required);
|
||||
- `gateway/README.md` (API table, configuration, auth semantics);
|
||||
- `examples/dsh-agents-as-a-service/docs/architecture.md` when flows change;
|
||||
- `examples/dsh-agents-as-a-service/README.md` when the walkthrough changes.
|
||||
|
||||
Run `make test` and `make lint`. Preserve session-cookie auth, same-origin rules
|
||||
for cookie mutations and WebSocket upgrades, and the "waking"/`502`/`503`
|
||||
semantics unless the change is explicitly about them. Do not expose a trusted
|
||||
identity header unless the Service is reachable only through the proxy.
|
||||
|
||||
### Add an exposure variant
|
||||
|
||||
Create `manifests/exposure/<variant>/NN-*.yaml`, update
|
||||
`manifests/exposure/README.md` and the example README, and state the trust
|
||||
boundary. Variants are alternatives: never apply two paths to the same gateway.
|
||||
Set `DSH_TRUSTED_HOSTS` in the template for the public host and
|
||||
`COOKIE_SECURE=true` for HTTPS.
|
||||
|
||||
### Add an example
|
||||
|
||||
Follow `docs/examples.md`: `README.md`, `manifests/NN-*.yaml`, optional
|
||||
`manifests/exposure/` with a README, optional components with Dockerfile +
|
||||
README + tests, and executable, idempotent, self-cleaning scripts. Add the
|
||||
literal `examples/<name>/` entry to `examples/README.md`; `make check` enforces
|
||||
the layout. Publish component images under
|
||||
`<registry>/<owner>/<repo>/<example>/<component>`.
|
||||
|
||||
### Bump a version
|
||||
|
||||
- **DSH release identity:** `make set-version DSH_VERSION=<ref>
|
||||
[RELEASE_REVISION=rN]`, then `make check`, then build and smoke-test the image.
|
||||
Tags are immutable; fix a bad release with `-r(N+1)`.
|
||||
- **agent-sandbox:** edit `AGENT_SANDBOX_VERSION` in `versions.env`, update the
|
||||
prerequisites in the example README, and note behavior changes (connection
|
||||
types, MCP output limits, CRD fields). Install the new release YAML on a kind
|
||||
cluster, then run `make check` and `make e2e`. Do not bump it inside an
|
||||
unrelated PR.
|
||||
|
||||
### Integrate the agent-sandbox MCP server
|
||||
|
||||
Build from the cluster's pinned tag, give it a dedicated namespace and
|
||||
least-privilege ServiceAccount, keep it ClusterIP behind an authenticating
|
||||
proxy, and document how to reach `/readyz`. Add a DSH profile row only with user
|
||||
approval; a row affects every session. Never commit its credentials or a
|
||||
`cluster-admin` binding.
|
||||
|
||||
### Documentation
|
||||
|
||||
Update the doc whose behavior changed in the same commit: root README for the
|
||||
front door, `docs/` for repository-wide contracts, the example README for that
|
||||
walkthrough, the gateway README for API/config/auth. Skill docs live in
|
||||
`.dsh/skills/`; keep `SKILL.md` under ~8 KB and put depth in `references/`.
|
||||
`scripts/check-versions.sh` scans every `*.md` for stale `dsh-v…` tokens, so
|
||||
refer to "the release identity in `versions.env`" rather than copying a tag.
|
||||
|
||||
## Verification and review
|
||||
|
||||
Run at least `make check`; add `make test` and `make lint` for gateway/script
|
||||
changes. For manifests, use `kubectl apply --dry-run=server` against a cluster
|
||||
with the CRDs. For behavior changes, run the example or `make e2e`.
|
||||
|
||||
Before reporting, review:
|
||||
|
||||
- [ ] Diff is scoped; no generated files or unrelated changes.
|
||||
- [ ] Docs and examples match the new behavior.
|
||||
- [ ] Pins come only from `versions.env`; no stale version tokens.
|
||||
- [ ] No secrets or usable credentials; secret-shaped files are `.example`.
|
||||
- [ ] Manifests parse; exposure variants remain mutually exclusive.
|
||||
- [ ] Gateway tests and public contract unchanged or intentionally updated.
|
||||
- [ ] Skills: frontmatter valid, body within budget, scripts shellcheck-clean.
|
||||
- [ ] `CHANGELOG.md` untouched; branch/commit follow `docs/gitflow.md`.
|
||||
|
||||
## Safe follow-ups enabled by this skill
|
||||
|
||||
- A read-only preflight job or scheduled report using
|
||||
`scripts/platform-preflight.sh`.
|
||||
- An MCP readiness check in the example e2e (health endpoints only).
|
||||
- A separate runtime warm pool for untrusted execution, leaving the DSH template
|
||||
unchanged.
|
||||
- A combined DSH + sandboxd image only with an explicit maintenance/exec design
|
||||
and a documented trust boundary.
|
||||
@@ -0,0 +1,94 @@
|
||||
# Deploying the agent-sandbox MCP server
|
||||
|
||||
Read [the MCP reference](mcp-server.md) first. This file covers the connection
|
||||
matrix, a safe deployment shape, and the DSH profile row. The server is
|
||||
unauthenticated; its service account is the security boundary.
|
||||
|
||||
## Connection matrix
|
||||
|
||||
| `K8S_SANDBOX_CONNECTION__TYPE` | Required env | Use when |
|
||||
| --- | --- | --- |
|
||||
| `gateway` | `GATEWAY_NAME`, `GATEWAY_NAMESPACE=default`, `GATEWAY_READY_TIMEOUT=180`, `SERVER_PORT=8888` | Gateway API + implementation exist |
|
||||
| `direct` | `API_URL` (router URL), `SERVER_PORT=8888` | sandbox-router is deployed |
|
||||
| `in-cluster` | `USE_POD_IP=false` (DNS) or `true` (`status.podIPs`), `SERVER_PORT=8888` | Bypass the router from inside the cluster |
|
||||
| `sandboxd-in-cluster` (v1.0.8+) | `MODE=in-cluster-service` or `in-cluster-pod-ip`, `REST_PORT=8080`, `GRPC_PORT=9090` | Template runs sandboxd |
|
||||
|
||||
Also set `K8S_SANDBOX_PROBE_NAMESPACE` to a namespace the server's service
|
||||
account can list claims in, and optionally
|
||||
`K8S_SANDBOX_SESSION_ID_LABEL_KEY` (default
|
||||
`mcp.k8s-agent-sandbox/session-id`).
|
||||
|
||||
For `sandboxd-in-cluster` service mode the template needs `spec.service: true`.
|
||||
The default managed NetworkPolicy admits only sandbox-router, so add an ingress
|
||||
rule for the MCP server on 8080/9090. Supplying `spec.networkPolicy` replaces the
|
||||
secure defaults: also add the egress rules the sandbox needs, because an omitted
|
||||
egress list denies all outbound traffic. RBAC for that mode is
|
||||
`sandboxclaims` create/get/list/watch/delete and `sandboxes` get/list/watch; no
|
||||
`pods/portforward`.
|
||||
|
||||
## Build and deploy (sketch)
|
||||
|
||||
No official image reference is documented in the upstream README; pin what you
|
||||
build.
|
||||
|
||||
```bash
|
||||
# AGENT_SANDBOX_VERSION comes from versions.env; never hard-code it.
|
||||
tmp="$(mktemp -d)"; trap 'rm -rf "$tmp"' EXIT
|
||||
git clone --depth 1 --branch "$AGENT_SANDBOX_VERSION" \
|
||||
https://github.com/kubernetes-sigs/agent-sandbox.git "$tmp"
|
||||
docker build -t "$MCP_IMAGE" "$tmp/clients/integrations/mcp-server"
|
||||
```
|
||||
|
||||
Kubernetes shape:
|
||||
|
||||
- Dedicated namespace and ServiceAccount; a namespace-scoped Role granting only
|
||||
`sandboxclaims` create/get/list/watch/delete and `sandboxes` get/list/watch.
|
||||
No Secrets, no `pods/exec`, no cluster-admin.
|
||||
- Deployment: one replica, non-root, `readOnlyRootFilesystem`, dropped
|
||||
capabilities, `K8S_SANDBOX_CONNECTION__TYPE=in-cluster`,
|
||||
`K8S_SANDBOX_PROBE_NAMESPACE`, resource limits.
|
||||
- Probes: `GET /healthz` (liveness; never calls the API) and `GET /readyz`
|
||||
(readiness; lists claims; use `periodSeconds: 15`, not aggressive polling).
|
||||
- ClusterIP Service on `8000`. Add a NetworkPolicy that allows only the intended
|
||||
client namespace/pod. Never expose it through Ingress/LoadBalancer without an
|
||||
authenticating proxy. The service account is the real boundary.
|
||||
|
||||
## Connect it to DSH
|
||||
|
||||
Add a profile MCP row (through the Plugins page/`plugin_manager`; see
|
||||
`cordis-plugin-development`). A row affects every session and needs approval.
|
||||
|
||||
```yaml
|
||||
- insert:
|
||||
- id: mcp-agent-sandbox
|
||||
name: '@deepseek-ai/dsh-mcp-client'
|
||||
config:
|
||||
serverName: agent-sandbox
|
||||
transport: streamable-http
|
||||
url: http://agent-sandbox-mcp.example.svc.cluster.local:8000/mcp
|
||||
toolCallTimeoutMs: 240000
|
||||
failOnStartupError: false
|
||||
```
|
||||
|
||||
- `serverName` must match `[A-Za-z0-9_-]{1,32}`; tools appear as
|
||||
`mcp__<serverName>__<tool>`. Check the actual names in the tool list; do not
|
||||
assume hyphens/underscores.
|
||||
- Raise `toolCallTimeoutMs` above `sandbox_ready_timeout` with headroom; the DSH
|
||||
default is 60 s and would cancel a 180 s creation.
|
||||
- If the server is behind an authenticating proxy, pass its token via the row's
|
||||
`headers` and an environment expression; never commit it.
|
||||
- To reach a locally port-forwarded server, use a loopback URL; the MCP session
|
||||
still needs streamable HTTP.
|
||||
- Remove the row when it was enabled only for one task.
|
||||
|
||||
## Deployment safety checklist
|
||||
|
||||
- [ ] Build tag equals the cluster's `AGENT_SANDBOX_VERSION`.
|
||||
- [ ] Dedicated namespace/SA; RBAC limited to claims and sandboxes.
|
||||
- [ ] Service is ClusterIP; no public route without authentication.
|
||||
- [ ] NetworkPolicy admits only the DSH client and needed sandbox traffic.
|
||||
- [ ] `/readyz` succeeds with `K8S_SANDBOX_PROBE_NAMESPACE` set to a listable
|
||||
namespace.
|
||||
- [ ] No Secret data in manifests, logs, or the repository.
|
||||
- [ ] Cleanup path documented: delete session-owned claims or wait for the
|
||||
shutdown timeout.
|
||||
103
.dsh/skills/dsh-agent-sandbox-platform/references/mcp-server.md
Normal file
103
.dsh/skills/dsh-agent-sandbox-platform/references/mcp-server.md
Normal file
@@ -0,0 +1,103 @@
|
||||
# agent-sandbox MCP server reference
|
||||
|
||||
Upstream: [clients/integrations/mcp-server](https://github.com/kubernetes-sigs/agent-sandbox/tree/main/clients/integrations/mcp-server).
|
||||
It is a Python/FastAPI + FastMCP service (streamable HTTP on `:8000`, MCP
|
||||
endpoint `/mcp`) that provisions sandboxes from a warm pool and runs commands or
|
||||
file operations through a sandbox runtime. It ships with **no authentication**.
|
||||
|
||||
Build the server from the same tag as the cluster: read `AGENT_SANDBOX_VERSION`
|
||||
in `versions.env`, and check the installed controller image. Do not build from
|
||||
`main` against an older cluster; tool schemas and connection types differ.
|
||||
|
||||
## Tool surface
|
||||
|
||||
| Tool | Arguments (defaults) | Returns / notes |
|
||||
| --- | --- | --- |
|
||||
| `create_sandbox` | `warmpool`, `namespace`, `sandbox_ready_timeout=180`, `labels`, `shutdown_after_seconds=300` (`null` disables), `pod_labels`, `pod_annotations` | `sandbox_claim_name`; adds the MCP session label |
|
||||
| `delete_sandbox` | `sandbox_claim_name`, `namespace` | Deletes the session-owned claim; PVC cascades |
|
||||
| `execute_command` | `sandbox_claim_name`, `namespace`, `command`, `timeout=60` (max 600), `max_output_chars` only on v1.0.8+ (128-300,000, default 30,000) | `exit_code`, `stdout`, `stderr`; v1.0.8 adds `*_total_chars` and `*_truncated` |
|
||||
| `upload_file` | `sandbox_claim_name`, `namespace`, `path`, `content`, `binary=false`, `timeout=60` | Bytes written; base64 when `binary=true` |
|
||||
| `download_file` | `sandbox_claim_name`, `namespace`, `path`, `binary=false`, `timeout=60` | Content + bytes read |
|
||||
| `list_files` | `sandbox_claim_name`, `namespace`, `path`, `timeout=60`, `max_entries=1000` (max 10,000) | Entries (`name`, `size`, `type`, `mod_time`), `total_entries`, `truncated` |
|
||||
| `file_exists` | `sandbox_claim_name`, `namespace`, `path`, `timeout=60` | `exists`; a missing path is not an error |
|
||||
| `get_sandbox_status` | `sandbox_claim_name`, `namespace` | `SandboxReady` / `SandboxNotReady` / `SandboxNotFound`, `ready`, `message` |
|
||||
|
||||
Resource: `sandboxes://{namespace}` lists sandboxes for the current MCP session.
|
||||
|
||||
## Session scoping is the most important behavior
|
||||
|
||||
Every tool, including `get_sandbox_status`, calls `get_sandbox()` ->
|
||||
`ensure_session_owns()`, which lists sandboxes with the label selector
|
||||
`<K8S_SANDBOX_SESSION_ID_LABEL_KEY>=<MCP session id>`. The resource uses the same
|
||||
selector. Consequences:
|
||||
|
||||
- The server can only see/exec/delete claims **it created in the same MCP
|
||||
session**. It cannot inspect or manage DSH platform claims created by the
|
||||
gateway or `kubectl`, or claims from another session.
|
||||
- On a DSH MCP reconnect, the MCP session changes; earlier claims become
|
||||
invisible to the tools. They keep running until `shutdown_after_seconds`
|
||||
expires, or until you remove them by label with a confirmed `kubectl delete`.
|
||||
- A "Sandbox claim '<name>' is not found in namespace '<ns>'" error usually
|
||||
means wrong namespace **or** the claim belongs to a different session.
|
||||
- Use `kubectl` for global inventory; use the MCP only for sandboxes it created
|
||||
in this task.
|
||||
|
||||
Session identity requires a session-providing transport. With stdio,
|
||||
`ctx.session_id` is `None` and creation fails. Connect DSH over
|
||||
`streamable-http`.
|
||||
|
||||
## Runtime prerequisite
|
||||
|
||||
The server talks to a sandbox runtime, not to arbitrary pods:
|
||||
|
||||
- `gateway`, `direct`, `in-cluster`: runtime HTTP on `SERVER_PORT=8888`
|
||||
(sandbox-router API or direct pod runtime).
|
||||
- `sandboxd-in-cluster` (v1.0.8+): sandboxd REST `8080`, gRPC `9090`.
|
||||
|
||||
The shipped DSH template runs only DSH Web on `3080`, so no connection type can
|
||||
exec into it. A sandboxd **sidecar** executes commands in the sidecar's own
|
||||
container, not in the DSH container; shared volumes share files, not binaries or
|
||||
processes. To exec inside DSH, build a combined image (sandboxd injected into the
|
||||
DSH image) or use `kubectl exec` with explicit RBAC and confirmation. For
|
||||
untrusted code, deploy a separate runtime warm pool rather than modifying the
|
||||
DSH template.
|
||||
|
||||
## Version differences
|
||||
|
||||
| Area | Before v1.0.8 | v1.0.8+ |
|
||||
| --- | --- | --- |
|
||||
| Connection types | `gateway`, `direct`, `in-cluster` | Adds `sandboxd-in-cluster` |
|
||||
| `execute_command` output | Unbounded | Bounded by `max_output_chars`; four metadata fields |
|
||||
| Failed pod recovery | `Retain` only | `spec.failedPodPolicy: Recreate` |
|
||||
| Router install | `kubectl apply -f sandbox-router/deploy/` | `kubectl apply -k sandbox-router/deploy/`; projected token |
|
||||
|
||||
Never pass `max_output_chars` to a pre-v1.0.8 server. On older servers, redirect
|
||||
verbose output to a file and `download_file` it, or pipe through `tail`/`head`.
|
||||
|
||||
## Safety and cleanup
|
||||
|
||||
- No auth: anyone reaching `:8000` gets every tool and the service account's
|
||||
access. Keep it ClusterIP, authenticated, and namespace-scoped.
|
||||
- Treat commands as untrusted code: they can read files and environment
|
||||
variables inside the sandbox, including provider keys in a combined image.
|
||||
- `delete_sandbox` cascades the claim, sandbox, and PVC: confirm the target and
|
||||
namespace. Never delete a namespace or PVC as cleanup.
|
||||
- Clean up in the same session that created the sandbox. If the session is gone,
|
||||
list claims by the session label and delete them explicitly (requires write
|
||||
RBAC); otherwise rely on the shutdown timeout.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Cause / fix |
|
||||
| --- | --- |
|
||||
| Tools missing | Row not enabled, wrong URL, connection failed; check logs and `/readyz` |
|
||||
| 404 on connect | URL is missing `/mcp` or points at the wrong port |
|
||||
| `ctx.session_id is None` | Connected over stdio; use streamable HTTP |
|
||||
| `Sandbox claim ... is not found` | Wrong namespace, or claim created by another MCP session |
|
||||
| Creation times out at ~60 s | MCP session alive but DSH cancelled the call; raise `toolCallTimeoutMs` |
|
||||
| `execute_command` connection error | No runtime on 8888/8080/9090, or wrong connection type; the DSH template is unsupported |
|
||||
| Output flood | Pre-v1.0.8 server; bound the command output yourself |
|
||||
| Empty sandbox list | The resource is session-scoped; use `kubectl` for global inventory |
|
||||
|
||||
Deployment, connection matrix, RBAC, and the DSH row:
|
||||
[MCP deployment](mcp-deployment.md).
|
||||
@@ -0,0 +1,176 @@
|
||||
# Platform runbook
|
||||
|
||||
Scope: the resource layer and gateway in
|
||||
`examples/dsh-agents-as-a-service/`, one namespace, one DSH Web sandbox per
|
||||
user. Read the example README and `AGENTS.md` first. All commands are read-only
|
||||
unless a section is explicitly a change; confirm before changing anything.
|
||||
|
||||
## 1. Context, version, CRDs
|
||||
|
||||
```bash
|
||||
kubectl config current-context
|
||||
kubectl auth whoami
|
||||
kubectl config view --minify -o jsonpath='{..namespace}{"\n"}'
|
||||
kubectl get crd | grep -E 'agents\.x-k8s\.io'
|
||||
kubectl -n agent-sandbox-system get deploy agent-sandbox-controller \
|
||||
-o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
|
||||
```
|
||||
|
||||
Compare the controller image tag with `AGENT_SANDBOX_VERSION` in
|
||||
`versions.env`. The example needs the extension CRDs plus core `sandboxes`;
|
||||
a missing CRD is a prerequisite failure, not an application bug.
|
||||
|
||||
## 2. Resource model
|
||||
|
||||
| Object | Role | Key fields |
|
||||
| --- | --- | --- |
|
||||
| `SandboxTemplate` | Blueprint; pinned image, probes, security context, PVC template | `envVarsInjectionPolicy`, `volumeClaimTemplatesPolicy`, `service`, `podTemplate` |
|
||||
| `SandboxWarmPool` | Pre-warmed spares from one template | `spec.replicas`, `sandboxTemplateRef` |
|
||||
| `SandboxClaim` | One tenant/user request; adopts a spare | `spec.warmPoolRef`, `additionalPodMetadata`, `status.sandbox.name` |
|
||||
| `Sandbox` | Adopted or directly created pod + Service + PVC | `spec.operatingMode`, `status.conditions`, `status.serviceFQDN`, `status.podIPs` |
|
||||
| `Pod` | The DSH Web process | `:3080`, PVC at `/home/node/.dsh` |
|
||||
| `PersistentVolumeClaim` | Per-sandbox state | `dsh-home-<sandbox>` |
|
||||
| `Service` | Stable per-sandbox DNS | `<sandbox>`, `status.serviceFQDN` |
|
||||
|
||||
Delete a claim to cascade-delete its sandbox and PVC (account deletion). Never
|
||||
delete a PVC or namespace without an explicit, named confirmation.
|
||||
|
||||
## 3. Read-only inspection
|
||||
|
||||
```bash
|
||||
kubectl -n "$NS" get sandboxtemplates,sandboxwarmpools,sandboxclaims,sandboxes -o wide
|
||||
kubectl -n "$NS" get pods,pvc,svc -o wide
|
||||
kubectl -n "$NS" get events --sort-by=.lastTimestamp | tail -30
|
||||
kubectl -n "$NS" get sandboxclaim "$CLAIM" -o yaml
|
||||
kubectl -n "$NS" get sandbox "$SB" -o yaml
|
||||
```
|
||||
|
||||
Bounded logs: `kubectl -n "$NS" logs "$POD" --tail=200` (add `--previous`
|
||||
only after a restart).
|
||||
|
||||
## 4. Claim -> sandbox -> pod
|
||||
|
||||
```bash
|
||||
SB=$(kubectl -n "$NS" get sandboxclaim "$CLAIM" -o jsonpath='{.status.sandbox.name}')
|
||||
POD=$(kubectl -n "$NS" get sandbox "$SB" -o jsonpath='{.metadata.annotations.agents\.x-k8s\.io/pod-name}')
|
||||
POD=${POD:-$SB}
|
||||
kubectl -n "$NS" get sandbox "$SB" -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}{"\n"}{end}'
|
||||
kubectl -n "$NS" get sandbox "$SB" -o jsonpath='{.spec.operatingMode}{" "}{.status.serviceFQDN}{" "}{.status.podIPs}{"\n"}'
|
||||
```
|
||||
|
||||
Rules:
|
||||
|
||||
- Resolve user -> sandbox only through `claim.status.sandbox.name`; the sandbox
|
||||
keeps its pool-generated name, not the claim name.
|
||||
- A claim tracks binding; for runtime state read `Sandbox.spec.operatingMode`
|
||||
and the `Suspended`/`Ready` conditions. A suspended sandbox is not serving
|
||||
even when the claim still exists.
|
||||
- A missing `claim.status.sandbox.name` means the claim did not bind: inspect
|
||||
the Ready condition `reason` and events. The pod is normally named after the
|
||||
sandbox; use the annotation when present.
|
||||
|
||||
## 5. Suspend, resume, wake
|
||||
|
||||
Suspend keeps the PVC and Service, deletes the pod:
|
||||
|
||||
```bash
|
||||
kubectl -n "$NS" patch sandbox "$SB" --type merge \
|
||||
-p '{"spec":{"operatingMode":"Suspended"}}'
|
||||
kubectl -n "$NS" wait --for=delete "pod/$POD" --timeout=120s
|
||||
kubectl -n "$NS" get pvc "dsh-home-$SB"
|
||||
kubectl -n "$NS" get svc "$SB"
|
||||
```
|
||||
|
||||
Resume and verify state:
|
||||
|
||||
```bash
|
||||
kubectl -n "$NS" patch sandbox "$SB" --type merge \
|
||||
-p '{"spec":{"operatingMode":"Running"}}'
|
||||
kubectl -n "$NS" wait sandbox "$SB" --for=condition=Ready --timeout=180s
|
||||
kubectl -n "$NS" exec "$SB" -- cat /home/node/.dsh/marker.txt
|
||||
```
|
||||
|
||||
The gateway wakes a suspended sandbox on the first request. The idle sweeper
|
||||
only suspends claims it created (`aaas.example.com/managed-by=dsh-gateway`);
|
||||
hand-applied claims are never swept.
|
||||
|
||||
## 6. Warm pool
|
||||
|
||||
```bash
|
||||
kubectl -n "$NS" get sandboxwarmpool "$POOL" -o yaml
|
||||
kubectl -n "$NS" get sandboxes -l agents.x-k8s.io/warm-pool-sandbox -o wide
|
||||
kubectl -n "$NS" get sandboxwarmpool "$POOL" -o jsonpath='{.spec.replicas}{"\n"}'
|
||||
```
|
||||
|
||||
Adopted sandboxes lose the warm-pool label; count current spares against
|
||||
`spec.replicas`. Zero spares or `spec.replicas: 0` means cold, slow signup.
|
||||
Scale only when asked:
|
||||
|
||||
```bash
|
||||
kubectl -n "$NS" patch sandboxwarmpool "$POOL" --type merge -p '{"spec":{"replicas":2}}'
|
||||
```
|
||||
|
||||
## 7. DSH runtime specifics
|
||||
|
||||
- DSH Web listens on `3080` in-cluster. Port-forward to a free local port such
|
||||
as `18081`, never `3080` if a local DSH already uses it.
|
||||
- The launch token is printed once as `dsh web: http://<ip>:3080/?token=...`.
|
||||
Treat it as a credential: do not paste it into reports; hand the user the URL.
|
||||
- `/` returns `401` without the token. Probes must target `/favicon.svg`.
|
||||
- `DSH_TRUSTED_HOSTS` must include any public host that a proxy forwards; DSH
|
||||
fences on `Host`/`Origin`.
|
||||
- State lives at `/home/node/.dsh`. DSH requires `.credentials.yaml` to be
|
||||
`0600`; the template uses `fsGroupChangePolicy: OnRootMismatch`. If a resumed
|
||||
sandbox crash-loops with a mode-660 error, suspend it, fix the file with a
|
||||
throwaway pod, patch the policy, then resume.
|
||||
|
||||
## 8. Gateway layer
|
||||
|
||||
```bash
|
||||
kubectl -n "$NS" get deploy,svc,pods -l app=dsh-gateway -o wide
|
||||
kubectl -n "$NS" get endpoints dsh-gateway
|
||||
kubectl -n "$NS" logs deploy/dsh-gateway --tail=100
|
||||
```
|
||||
|
||||
The gateway is a ClusterIP Service on `8080`; it proxies under `/u/{user}/` and
|
||||
derives state (`Provisioning`, `Waking`, `Ready`, `Suspending`, `Suspended`).
|
||||
It performs the DSH launch-token handshake server-side. With SSO enabled
|
||||
(`AUTH_PROXY_HEADER`), the Service must be reachable only through the
|
||||
authenticating proxy; never add a second path or port-forward it on a shared
|
||||
host. A resource-only install has no gateway: that is a warning, not a failure.
|
||||
|
||||
## 9. Security boundaries
|
||||
|
||||
- One `DEEPSEEK_API_KEY` is shared by all sandboxes; the example is not
|
||||
tenant-isolated.
|
||||
- Secret data, kubeconfigs, and pod logs are sensitive; RBAC may allow Secret
|
||||
access, policy does not. Keep gateway RBAC namespace-scoped; it needs pod logs
|
||||
to find the launch token.
|
||||
- TLS terminates outside the gateway; set `COOKIE_SECURE=true` for HTTPS.
|
||||
|
||||
## 10. Failure matrix
|
||||
|
||||
| Symptom | First evidence | Likely cause / fix |
|
||||
| --- | --- | --- |
|
||||
| Claim not Ready, no sandbox | Ready condition reason, events | Empty pool, image pull, quota, unschedulable |
|
||||
| `EnvVarsInjectionRejected` | claim Ready reason | Claim carries env/volume templates; template policy rejects it |
|
||||
| Pod Pending | `kubectl describe pod` | Requests/taints/affinity, unbound PVC |
|
||||
| CrashLoopBackOff | logs, `--previous` | Startup failure, bad config, probe 401, fsGroup mode 660 |
|
||||
| ImagePullBackOff | describe events | Wrong tag, private registry, missing pull secret |
|
||||
| DSH 401 in browser | gateway logs | Launch-token handshake failed; check gateway RBAC `pods/log` |
|
||||
| 502/503 through gateway | sandbox Ready, gateway logs | Wake timeout, DNS stale, token not in logs yet |
|
||||
| Warm pool empty | sandboxes label count | Scale the pool or wait for replenishment |
|
||||
| Suspended but expected running | `operatingMode` | Resume it; the gateway only wakes on a request |
|
||||
| WS closes `1011` | gateway logs | DSH rejected the upstream Host/authority; check trusted hosts |
|
||||
|
||||
## 11. Cleanup and reporting
|
||||
|
||||
Delete only claims/sandboxes you created, by name:
|
||||
|
||||
```bash
|
||||
kubectl -n "$NS" delete sandboxclaim "$CLAIM" --dry-run=server
|
||||
kubectl -n "$NS" delete sandboxclaim "$CLAIM"
|
||||
```
|
||||
|
||||
Report context/namespace, evidence, conclusion, change, rollback, and what was
|
||||
deleted. Leave the namespace, pools, templates, and unrelated claims alone.
|
||||
@@ -0,0 +1,71 @@
|
||||
# Troubleshooting
|
||||
|
||||
Start with evidence, not fixes: resolve context/namespace, read conditions and
|
||||
events, then compare desired vs observed state. Keep output bounded. Never
|
||||
"fix" an access error by broadening credentials.
|
||||
|
||||
## Platform
|
||||
|
||||
| Symptom | First evidence | Likely cause / fix |
|
||||
| --- | --- | --- |
|
||||
| `Forbidden` on every call | `kubectl auth whoami`, `kubectl auth can-i --list -n <ns>` | Wrong context or RBAC. Do not retry with broader credentials; ask for the right kubeconfig. |
|
||||
| CRDs not found | `kubectl get crd | grep agents.x-k8s.io` | agent-sandbox extensions not installed. Install the pinned release YAML; that is a prerequisite, not a code change. |
|
||||
| Claim created, no sandbox | `claim.status.conditions[?(@.type=="Ready")]`, events | Empty pool, image pull, quota, unschedulable; or policy rejection. |
|
||||
| Claim rejected | Ready reason `EnvVarsInjectionRejected` | Claim carries `env`/`volumeClaimTemplates` while the template disallows injection. Remove them; do not bypass the pool. |
|
||||
| User is suspended but the claim exists | `Sandbox.spec.operatingMode`, `Suspended` condition | Resume with `operatingMode: Running`; a gateway request would wake it. |
|
||||
| Warm pool exhausted | `kubectl get sandboxes -l agents.x-k8s.io/warm-pool-sandbox` | Scale `spec.replicas` (with confirmation) or accept slow cold provisioning. |
|
||||
| Pod `Pending` | `kubectl describe pod` | Requests/limits, taints, affinity, unbound PVC, storage class. |
|
||||
| `ImagePullBackOff` | describe events | Wrong image/tag, private GHCR package, missing pull secret; on kind, load the image under the exact manifest ref. |
|
||||
| `CrashLoopBackOff`, probe 401 | logs, `describe` | Probe path is `/`; use `/favicon.svg`. |
|
||||
| Restart with mode-660 error | logs mention `.credentials.yaml` | kubelet fsGroup widened the file; suspend, `chmod 600` from a throwaway pod, patch `fsGroupChangePolicy: OnRootMismatch`, resume. |
|
||||
| PVC `Pending` | `kubectl describe pvc`, StorageClass | No default provisioner, access mode, zone binding. |
|
||||
| Launch token not found | `kubectl logs <pod> --tail=2000 | grep 'dsh web:'` | Pod not ready yet, or logs rotated. Do not guess tokens; wait for readiness. |
|
||||
| Hand-applied sandbox disappeared | claim annotations, gateway logs | Idle sweeper only manages claims with `aaas.example.com/managed-by=dsh-gateway`; if it was managed, recreate/resume. |
|
||||
| Gateway 401 | gateway logs, cookie flags | Missing/expired cookie, DSH handshake failed, or `COOKIE_SECURE=true` over HTTP. |
|
||||
| Gateway 502/503 | sandbox Ready, gateway logs | Wake timeout, stale DNS, token not yet in logs; check `WAKE_TIMEOUT` and RBAC `pods/log`. |
|
||||
| WebSocket closes 1011 | gateway logs, `Host`/Origin | DSH rejected the upstream authority; check `DSH_TRUSTED_HOSTS` and the gateway's pod-IP dial path. |
|
||||
| WebSocket closes 4401 | cookie, Origin | Missing gateway auth or cross-origin upgrade; send a matching `Origin`. |
|
||||
| SSO header ignored | gateway env | `AUTH_PROXY_HEADER` unset; setting it requires the proxy to be the only path to the ClusterIP Service. |
|
||||
| Someone is treated as another user | service paths | Trust-boundary violation: the gateway is directly reachable. Remove the extra path and disable SSO until fixed. |
|
||||
|
||||
## Agent-sandbox MCP
|
||||
|
||||
| Symptom | First evidence | Likely cause / fix |
|
||||
| --- | --- | --- |
|
||||
| No `mcp__*__create_sandbox` tools | tool list, DSH logs | Server not deployed or row not enabled. Use `kubectl`; connect only with approval. |
|
||||
| 404 on connect | curl `/mcp`, `/healthz` | Wrong path/port; MCP endpoint is `/mcp`, health is at the root. |
|
||||
| `/readyz` 503 | response body, server logs | API unreachable or namespace `K8S_SANDBOX_PROBE_NAMESPACE` not listable; fix RBAC/config. |
|
||||
| `ctx.session_id is None` | server error | Connected over stdio; use streamable HTTP. |
|
||||
| `Sandbox claim ... is not found` | server error, `kubectl get claim` | Wrong namespace, or the claim belongs to another MCP session. List with `kubectl`; use the MCP only for its own claims. |
|
||||
| Creation cancelled at ~60 s | MCP call log | DSH `toolCallTimeoutMs` is 60 s while creation waits up to 180 s; raise the timeout. |
|
||||
| `execute_command` connection error | server logs, template ports | No runtime on 8888/8080/9090. The DSH template only listens on 3080; use a runtime template or a combined image. |
|
||||
| Output truncated / flooded | tool result | Pre-v1.0.8 is unbounded; v1.0.8 bounds output. Redirect verbose output to a file and read/download it; never pass `max_output_chars` to an old server. |
|
||||
| Orphaned claims after reconnect | `kubectl get claims -l mcp.k8s-agent-sandbox/session-id` | MCP sessions are transient; wait for `shutdown_after_seconds` or delete by label with explicit confirmation. |
|
||||
| `Forbidden` from the MCP server | server logs | Its ServiceAccount lacks claim/sandbox verbs, or RBAC is in another namespace; fix the Role, not the kubeconfig. |
|
||||
| Sandbox expired | `get_sandbox_status` -> `SandboxNotFound` | `shutdown_after_seconds` elapsed; create a new sandbox and keep the task inside the budget. |
|
||||
|
||||
## Repository checks and CI
|
||||
|
||||
| Symptom | Cause / fix |
|
||||
| --- | --- |
|
||||
| `check-versions: stale version reference in docs: dsh-v...` | A copied release tag in a Markdown file. Refer to the identity in `versions.env`; run `make set-version` only for a real bump. |
|
||||
| `check-versions: hard-coded version outside versions.env` | A tag in `.github/workflows` or the Makefile; move it to `versions.env`. |
|
||||
| `check-examples` layout error | Missing `README.md`/`manifests/`, stray YAML at the example root, missing exposure README, non-executable script, or missing `examples/README.md` entry. |
|
||||
| Manifest dry-run fails in CI | Invalid YAML or CRD schema; parse locally and run `kubectl apply --dry-run=server` against the pinned CRDs. |
|
||||
| `shellcheck` failure | Quote expansions, use arrays for kubectl args, avoid unused variables; run shellcheck locally. |
|
||||
| Lychee link failure | A relative/absolute link does not resolve; the link checker runs offline on the listed docs. |
|
||||
| PR-title check failure | Use Conventional Commits: `feat(scope): ...`, `fix(scope): ...`, `docs(scope): ...`. |
|
||||
| Changelog conflict | Never edit `CHANGELOG.md`; it is generated from commits. |
|
||||
|
||||
## Do not
|
||||
|
||||
- Do not use `--force --grace-period=0` to clear a stuck resource before
|
||||
understanding why it is stuck.
|
||||
- Do not delete namespaces, PVCs, PVs, CRDs, or cluster-scoped objects as a
|
||||
cleanup shortcut.
|
||||
- Do not read Secret data, kubeconfigs, or node logs to "debug" a permission
|
||||
problem, and do not paste pod logs containing the DSH launch token into a
|
||||
report.
|
||||
- Do not expose the unauthenticated MCP server or enable `AUTH_PROXY_HEADER`
|
||||
while another path to the gateway exists.
|
||||
- Do not retry a `Forbidden` with a stronger kubeconfig; fix the RBAC scope.
|
||||
298
.dsh/skills/dsh-agent-sandbox-platform/scripts/platform-preflight.sh
Executable file
298
.dsh/skills/dsh-agent-sandbox-platform/scripts/platform-preflight.sh
Executable file
@@ -0,0 +1,298 @@
|
||||
#!/usr/bin/env bash
|
||||
# Read-only preflight for the DSH agents-as-a-service platform.
|
||||
#
|
||||
# Checks the kubectl target, agent-sandbox CRDs/version, warm pools, claims,
|
||||
# sandboxes, gateway and (optionally) an agent-sandbox MCP server. It never
|
||||
# mutates the cluster, never reads Secret data, and never port-forwards.
|
||||
#
|
||||
# Usage:
|
||||
# platform-preflight.sh [--context NAME] [--namespace NS] [--pool NAME]
|
||||
# [--template NAME] [--mcp-url URL] [--help]
|
||||
#
|
||||
# Exit codes: 0 = no failures, 1 = at least one failure, 2 = usage error.
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
||||
REPO_ROOT="$(git -C "$SCRIPT_DIR" rev-parse --show-toplevel 2>/dev/null || true)"
|
||||
|
||||
CTX=""
|
||||
NS="${NAMESPACE:-dsh-demo}"
|
||||
POOL=""
|
||||
TEMPLATE=""
|
||||
MCP_URL=""
|
||||
|
||||
PASS=0
|
||||
WARN=0
|
||||
FAIL=0
|
||||
|
||||
usage() {
|
||||
cat <<'EOF'
|
||||
Read-only preflight for the DSH agents-as-a-service platform.
|
||||
|
||||
Options:
|
||||
--context NAME kubectl context (default: current context)
|
||||
--namespace NS namespace to inspect (default: $NAMESPACE or dsh-demo)
|
||||
--pool NAME check only this warm pool
|
||||
--template NAME check only this SandboxTemplate
|
||||
--mcp-url URL also check an MCP server's /healthz and /readyz
|
||||
--help show this help
|
||||
|
||||
The script makes no changes and never reads Secret data.
|
||||
EOF
|
||||
}
|
||||
|
||||
while [ "$#" -gt 0 ]; do
|
||||
case "$1" in
|
||||
--context) CTX="${2:-}"; shift 2 ;;
|
||||
--namespace) NS="${2:-}"; shift 2 ;;
|
||||
--pool) POOL="${2:-}"; shift 2 ;;
|
||||
--template) TEMPLATE="${2:-}"; shift 2 ;;
|
||||
--mcp-url) MCP_URL="${2:-}"; shift 2 ;;
|
||||
--help|-h) usage; exit 0 ;;
|
||||
*) echo "unknown argument: $1" >&2; usage >&2; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
if [ -z "$NS" ]; then
|
||||
echo "namespace must not be empty" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
section() { printf '\n== %s ==\n' "$1"; }
|
||||
pass() { PASS=$((PASS + 1)); printf 'PASS %s\n' "$1"; }
|
||||
warn() { WARN=$((WARN + 1)); printf 'WARN %s\n' "$1"; }
|
||||
fail() { FAIL=$((FAIL + 1)); printf 'FAIL %s\n' "$1"; }
|
||||
info() { printf 'INFO %s\n' "$1"; }
|
||||
|
||||
command -v kubectl >/dev/null 2>&1 || {
|
||||
echo "FAIL kubectl is not installed" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
if [ -n "$CTX" ]; then
|
||||
if ! kubectl config get-contexts -o name 2>/dev/null | grep -Fxq "$CTX"; then
|
||||
echo "FAIL context '$CTX' is not in the kubeconfig" >&2
|
||||
exit 1
|
||||
fi
|
||||
KUBECTL=(kubectl --context "$CTX" --request-timeout=15s)
|
||||
else
|
||||
KUBECTL=(kubectl --request-timeout=15s)
|
||||
fi
|
||||
|
||||
k() { "${KUBECTL[@]}" "$@"; }
|
||||
|
||||
section "Target"
|
||||
if current="$(k config current-context 2>/dev/null)"; then
|
||||
info "context: $current"
|
||||
pass "kubeconfig is readable"
|
||||
else
|
||||
fail "cannot read the current kubectl context"
|
||||
fi
|
||||
if user="$(k auth whoami -o jsonpath='{.status.userInfo.username}' 2>/dev/null)" && [ -n "$user" ]; then
|
||||
info "identity: $user"
|
||||
else
|
||||
fallback_user="$(k config view --minify -o jsonpath='{.users[0].name}' 2>/dev/null || true)"
|
||||
info "identity: ${fallback_user:-unknown}"
|
||||
fi
|
||||
if k get namespace "$NS" -o name >/dev/null 2>&1; then
|
||||
pass "namespace $NS exists"
|
||||
else
|
||||
fail "namespace $NS is missing or not readable"
|
||||
fi
|
||||
|
||||
if [ -n "$REPO_ROOT" ] && [ -f "$REPO_ROOT/versions.env" ]; then
|
||||
# shellcheck source=/dev/null
|
||||
. "$REPO_ROOT/versions.env"
|
||||
info "expected agent-sandbox pin: ${AGENT_SANDBOX_VERSION:-unknown}"
|
||||
else
|
||||
warn "versions.env not found; skipping the pin comparison"
|
||||
fi
|
||||
|
||||
section "Capabilities (informational; never widen them)"
|
||||
for check in "create sandboxclaims" "delete sandboxclaims" "create pods/exec"; do
|
||||
verb="${check%% *}"
|
||||
resource="${check#* }"
|
||||
if [ "$(k auth can-i "$verb" "$resource" -n "$NS" 2>/dev/null || echo unknown)" = "yes" ]; then
|
||||
info "can $verb $resource in $NS: yes"
|
||||
else
|
||||
info "can $verb $resource in $NS: no"
|
||||
fi
|
||||
done
|
||||
if [ "$(k auth can-i get secrets -n "$NS" 2>/dev/null || echo unknown)" = "yes" ]; then
|
||||
warn "RBAC allows reading Secrets; policy still forbids printing Secret data"
|
||||
else
|
||||
info "can get secrets in $NS: no"
|
||||
fi
|
||||
|
||||
section "Agent Sandbox CRDs"
|
||||
for crd in \
|
||||
sandboxtemplates.extensions.agents.x-k8s.io \
|
||||
sandboxwarmpools.extensions.agents.x-k8s.io \
|
||||
sandboxclaims.extensions.agents.x-k8s.io \
|
||||
sandboxes.agents.x-k8s.io; do
|
||||
if k get crd "$crd" -o name >/dev/null 2>&1; then
|
||||
pass "CRD $crd"
|
||||
else
|
||||
fail "CRD $crd is missing or not readable"
|
||||
fi
|
||||
done
|
||||
|
||||
section "Controller"
|
||||
controller_ns="agent-sandbox-system"
|
||||
controller_image="$(k -n "$controller_ns" get deployment agent-sandbox-controller \
|
||||
-o jsonpath='{.spec.template.spec.containers[0].image}' 2>/dev/null || true)"
|
||||
if [ -n "$controller_image" ]; then
|
||||
info "controller image: $controller_image"
|
||||
controller_tag="${controller_image##*:}"
|
||||
if [ -n "${AGENT_SANDBOX_VERSION:-}" ] && [ "$controller_tag" != "$AGENT_SANDBOX_VERSION" ]; then
|
||||
warn "controller tag '$controller_tag' differs from AGENT_SANDBOX_VERSION '$AGENT_SANDBOX_VERSION'"
|
||||
else
|
||||
pass "controller version matches versions.env"
|
||||
fi
|
||||
else
|
||||
warn "controller deployment not found in $controller_ns; cannot verify the installed version"
|
||||
fi
|
||||
|
||||
section "Templates"
|
||||
templates="$(k -n "$NS" get sandboxtemplates \
|
||||
-o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' 2>/dev/null || true)"
|
||||
if [ -z "$templates" ]; then
|
||||
warn "no SandboxTemplates in $NS"
|
||||
fi
|
||||
if [ -n "$TEMPLATE" ] && ! printf '%s\n' "$templates" | grep -Fxq "$TEMPLATE"; then
|
||||
fail "SandboxTemplate $TEMPLATE not found in $NS"
|
||||
fi
|
||||
for tpl in $templates; do
|
||||
[ -n "$TEMPLATE" ] && [ "$tpl" != "$TEMPLATE" ] && continue
|
||||
desc="$(k -n "$NS" get sandboxtemplate "$tpl" \
|
||||
-o jsonpath='{range .spec.podTemplate.spec.containers[*]}{.name}{":"}{range .ports[*]}{.containerPort}{","}{end}{";"}{end}' \
|
||||
2>/dev/null || true)"
|
||||
info "template $tpl containers: ${desc:-<none>}"
|
||||
case "$desc" in
|
||||
*sandboxd*|*:8080,*|*:9090,*)
|
||||
pass "template $tpl exposes a sandboxd runtime (MCP sandboxd-in-cluster)" ;;
|
||||
*:8888,*)
|
||||
pass "template $tpl exposes a runtime HTTP port (MCP gateway/direct/in-cluster)" ;;
|
||||
*:3080,*)
|
||||
warn "template $tpl is DSH Web only (:3080); MCP execute_command cannot run against it" ;;
|
||||
*)
|
||||
warn "template $tpl exposes no known sandbox runtime port" ;;
|
||||
esac
|
||||
done
|
||||
|
||||
section "Warm pools"
|
||||
pools="$(k -n "$NS" get sandboxwarmpools \
|
||||
-o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' 2>/dev/null || true)"
|
||||
if [ -z "$pools" ]; then
|
||||
warn "no SandboxWarmPools in $NS; claims will provision cold"
|
||||
fi
|
||||
if [ -n "$POOL" ] && ! printf '%s\n' "$pools" | grep -Fxq "$POOL"; then
|
||||
fail "SandboxWarmPool $POOL not found in $NS"
|
||||
fi
|
||||
for pool in $pools; do
|
||||
[ -n "$POOL" ] && [ "$pool" != "$POOL" ] && continue
|
||||
replicas="$(k -n "$NS" get sandboxwarmpool "$pool" -o jsonpath='{.spec.replicas}' 2>/dev/null || true)"
|
||||
selector="$(k -n "$NS" get sandboxwarmpool "$pool" -o jsonpath='{.status.selector}' 2>/dev/null || true)"
|
||||
if [ -n "$selector" ]; then
|
||||
ready="$(k -n "$NS" get sandboxes -l "$selector" \
|
||||
-o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' \
|
||||
2>/dev/null | grep -c '^True$' || true)"
|
||||
else
|
||||
ready="$(k -n "$NS" get sandboxes -l agents.x-k8s.io/warm-pool-sandbox \
|
||||
-o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' \
|
||||
2>/dev/null | grep -c '^True$' || true)"
|
||||
fi
|
||||
replicas="${replicas:-0}"
|
||||
info "pool $pool: $ready/$replicas warm spares Ready"
|
||||
if [ "$replicas" -eq 0 ]; then
|
||||
warn "pool $pool has spec.replicas=0; signup will cold-provision"
|
||||
elif [ "$ready" -lt "$replicas" ]; then
|
||||
warn "pool $pool is below its desired spare count"
|
||||
else
|
||||
pass "pool $pool is at capacity"
|
||||
fi
|
||||
done
|
||||
|
||||
section "Claims"
|
||||
claims="$(k -n "$NS" get sandboxclaims \
|
||||
-o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.conditions[?(@.type=="Ready")].status}{"\t"}{.status.conditions[?(@.type=="Ready")].reason}{"\t"}{.status.sandbox.name}{"\n"}{end}' \
|
||||
2>/dev/null || true)"
|
||||
if [ -z "$claims" ]; then
|
||||
warn "no SandboxClaims in $NS"
|
||||
else
|
||||
while IFS=$'\t' read -r name ready reason sandbox; do
|
||||
[ -n "$name" ] || continue
|
||||
if [ "$ready" != "True" ]; then
|
||||
warn "claim $name is not Ready (status='${ready:-unknown}', reason='${reason:-unknown}')"
|
||||
elif [ -z "$sandbox" ]; then
|
||||
warn "claim $name is Ready but has no bound sandbox"
|
||||
else
|
||||
pass "claim $name -> sandbox $sandbox"
|
||||
fi
|
||||
done <<< "$claims"
|
||||
fi
|
||||
|
||||
section "Sandboxes"
|
||||
sandboxes="$(k -n "$NS" get sandboxes \
|
||||
-o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.operatingMode}{"\t"}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' \
|
||||
2>/dev/null || true)"
|
||||
if [ -z "$sandboxes" ]; then
|
||||
warn "no Sandboxes in $NS"
|
||||
else
|
||||
while IFS=$'\t' read -r name mode ready; do
|
||||
[ -n "$name" ] || continue
|
||||
info "sandbox $name mode=${mode:-Running} ready=${ready:-unknown}"
|
||||
done <<< "$sandboxes"
|
||||
fi
|
||||
|
||||
section "Storage and Services"
|
||||
pvc_count="$(k -n "$NS" get pvc -o name 2>/dev/null | wc -l | tr -d ' ')"
|
||||
svc_count="$(k -n "$NS" get service -o name 2>/dev/null | wc -l | tr -d ' ')"
|
||||
info "persistent volume claims: ${pvc_count:-0}"
|
||||
info "services: ${svc_count:-0}"
|
||||
[ "${pvc_count:-0}" -gt 0 ] || warn "no PVCs in $NS"
|
||||
[ "${svc_count:-0}" -gt 0 ] || warn "no Services in $NS"
|
||||
|
||||
section "Gateway"
|
||||
if k -n "$NS" get deployment dsh-gateway -o name >/dev/null 2>&1; then
|
||||
pass "gateway deployment dsh-gateway exists"
|
||||
if k -n "$NS" get service dsh-gateway -o name >/dev/null 2>&1; then
|
||||
endpoints="$(k -n "$NS" get endpoints dsh-gateway \
|
||||
-o jsonpath='{range .subsets[*].addresses[*]}{.ip}{" "}{end}' 2>/dev/null || true)"
|
||||
if [ -n "$endpoints" ]; then
|
||||
pass "gateway endpoints: $endpoints"
|
||||
else
|
||||
warn "gateway Service has no ready endpoints"
|
||||
fi
|
||||
else
|
||||
warn "gateway Service dsh-gateway is missing"
|
||||
fi
|
||||
else
|
||||
warn "no dsh-gateway deployment; this is a resource-layer-only install"
|
||||
fi
|
||||
|
||||
section "MCP server"
|
||||
if [ -n "$MCP_URL" ]; then
|
||||
if ! command -v curl >/dev/null 2>&1; then
|
||||
fail "curl is required for --mcp-url"
|
||||
else
|
||||
base="${MCP_URL%/}"
|
||||
base="${base%/mcp}"
|
||||
for endpoint in healthz readyz; do
|
||||
body="$(curl -fsS --max-time 5 "$base/$endpoint" 2>/dev/null | head -c 300 || true)"
|
||||
if [ -n "$body" ]; then
|
||||
pass "MCP /$endpoint: $body"
|
||||
else
|
||||
fail "MCP /$endpoint is unreachable at $base/$endpoint"
|
||||
fi
|
||||
done
|
||||
fi
|
||||
else
|
||||
info "no --mcp-url supplied; skipping server health checks"
|
||||
fi
|
||||
|
||||
printf '\n== Summary ==\nPASS=%d WARN=%d FAIL=%d\n' "$PASS" "$WARN" "$FAIL"
|
||||
if [ "$FAIL" -gt 0 ]; then
|
||||
exit 1
|
||||
fi
|
||||
exit 0
|
||||
@@ -48,6 +48,7 @@ before changing anything; it encodes the conventions the CI enforces.
|
||||
| examples/dsh-agents-as-a-service/ | The multi-tenant platform example (manifests, gateway, docs, scripts) |
|
||||
| examples/minimal-single-tenant/ | The just-one-DSH-Web quickstart |
|
||||
| scripts/ | check-versions.sh, check-examples.sh, check-manifests.py, set-version.sh |
|
||||
| .dsh/skills/ | Project-local DSH skills (auto-discovered); dsh-agent-sandbox-platform covers platform operation, MCP limits and extension recipes |
|
||||
| .github/workflows/ | ci.yaml, reusable _build-image.yaml, build-*, release.yaml, e2e-kind.yaml, pr-title.yaml, codeql.yml |
|
||||
|
||||
## Commands
|
||||
@@ -75,6 +76,7 @@ needs kind plus the agent-sandbox CRDs (see the example README).
|
||||
| Dockerfile / versions.env | make check; make image (amd64) and a container smoke test |
|
||||
| Workflow | actionlint in CI; keep actions SHA-pinned; no hard-coded versions |
|
||||
| New example | make check; all files described in docs/examples.md |
|
||||
| Project skill (.dsh/skills/) | make check; frontmatter and size check; shellcheck skill scripts; load it with the skill tool |
|
||||
|
||||
## Versioning and releases
|
||||
|
||||
@@ -136,5 +138,6 @@ needs kind plus the agent-sandbox CRDs (see the example README).
|
||||
- [docs/examples.md](docs/examples.md) - example authoring rules
|
||||
- [docs/gitflow.md](docs/gitflow.md) - branches, PRs, required checks
|
||||
- [docs/versioning.md](docs/versioning.md) - versioning, changelog, releases
|
||||
- [.dsh/skills/dsh-agent-sandbox-platform/SKILL.md](.dsh/skills/dsh-agent-sandbox-platform/SKILL.md) - platform operations, MCP limits, extension recipes
|
||||
- [CONTRIBUTING.md](CONTRIBUTING.md) - development setup
|
||||
- [SECURITY.md](SECURITY.md) - vulnerability reporting and threat notes
|
||||
|
||||
10
README.md
10
README.md
@@ -132,6 +132,15 @@ The gateway is a component of the multi-tenant example and has its own
|
||||
[README](examples/dsh-agents-as-a-service/gateway/README.md) (API, configuration,
|
||||
auth/SSO, limitations).
|
||||
|
||||
## Agent skills
|
||||
|
||||
DSH discovers project skills from `.dsh/skills/<name>/SKILL.md`. The repository
|
||||
ships [`dsh-agent-sandbox-platform`](.dsh/skills/dsh-agent-sandbox-platform/SKILL.md),
|
||||
which covers operating and extending the platform, the optional
|
||||
[agent-sandbox MCP server](https://github.com/kubernetes-sigs/agent-sandbox/blob/main/clients/integrations/mcp-server/README.md),
|
||||
and a read-only cluster preflight. Runbooks, the MCP reference, the extension
|
||||
guide and troubleshooting notes live beside `SKILL.md`.
|
||||
|
||||
## Repository layout
|
||||
|
||||
| Path | Purpose |
|
||||
@@ -144,6 +153,7 @@ auth/SSO, limitations).
|
||||
| [`examples/dsh-agents-as-a-service/`](examples/dsh-agents-as-a-service/) | Multi-tenant platform example: `manifests/`, `gateway/`, `docs/`, `scripts/`. |
|
||||
| [`examples/minimal-single-tenant/`](examples/minimal-single-tenant/) | Single-sandbox quickstart. |
|
||||
| [`scripts/`](scripts/) | Repository checks: versions, example layout, manifest YAML, version bump. |
|
||||
| [`.dsh/skills/`](.dsh/skills/) | Project-local DSH skills, auto-discovered from the git root. |
|
||||
| [`.github/`](.github/) | CI, release automation, dependency updates, PR/issue templates. |
|
||||
| [`.env.sample`](.env.sample) | Documented local configuration (copy to `.env`; never commit it). |
|
||||
|
||||
|
||||
@@ -17,6 +17,15 @@ gets a DSH Web container running before anything else.
|
||||
| [gitflow.md](gitflow.md) | Branches, protected `main`, squash merges, Conventional PR titles, required CI checks, release branches, hotfixes and the one-time GitHub settings checklist. |
|
||||
| [versioning.md](versioning.md) | Release identity (`DSH ref` + optional `-rN`), why the current identity is `dsh-v0.2.1-rc.3-r3`, channels, compatibility matrix, bumping pins, the release checklist, rollback, git-cliff and digest pinning. |
|
||||
|
||||
## Project skills
|
||||
|
||||
DSH discovers project skills from `<git root>/.dsh/skills/<name>/SKILL.md`
|
||||
(the nearest `.git` ancestor). This repository ships one:
|
||||
|
||||
| Skill | Read it for |
|
||||
| --- | --- |
|
||||
| [dsh-agent-sandbox-platform](../.dsh/skills/dsh-agent-sandbox-platform/SKILL.md) | Operating and extending the agents-as-a-service platform, agent-sandbox MCP limits and deployment, and a read-only preflight. |
|
||||
|
||||
## Example documentation
|
||||
|
||||
| Document | Read it for |
|
||||
|
||||
Reference in New Issue
Block a user