1
0
mirror of https://github.com/sredevopsorg/deepseek-kubernetes-sandbox.git synced 2026-11-03 01:13:24 +00:00

9 Commits

Author SHA1 Message Date
SREDevOps.org Agent
bed8e5675a chore(release): dsh-v0.2.1-rc.5 (#29)
Bump DSH_VERSION to dsh-v0.2.1-rc.5 and clear RELEASE_REVISION, making the release identity dsh-v0.2.1-rc.5. Update every derived manifest, image tag, and doc, and remove prose that described the previous identity's legacy-tag history.

Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
2026-10-30 04:54:37 -03:00
SREDevOps.org Agent
14c3fb840e feat(example): add optional failed-Pod recovery (#26)
agent-sandbox v1.0.8 adds spec.failedPodPolicy to Sandbox and
SandboxTemplate. The default, Retain, leaves a Failed Pod in place and
surfaces Finished=True; Recreate deletes it and starts a replacement with
the same Sandbox identity and PVCs.

Ship it as an opt-in toggle next to the other commented fields, with a
new section 5 in the optional-hardening doc: what it changes, how to
enable it for new, warm-pool and already-claimed Sandboxes (the policy is
stamped at creation), a failure drill, caveats and rollback. The base
walkthrough stays on Retain because a deterministic failure would
otherwise recreate forever, and because the templates use
restartPolicy OnFailure, so the policy only matters for terminal cases
such as node eviction.

Cross-references updated in the example README, the optional-manifests
index, the architecture failure-mode table and the platform skill's
troubleshooting table.

Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
2026-10-26 14:29:01 -03:00
SREDevOps.org Agent
96d3030a49 chore(deps): bump agent-sandbox to v1.0.8 (#25)
Update AGENT_SANDBOX_VERSION to v1.0.8 and the docs that quote the
control-plane pin (example prerequisites, architecture, versioning table,
minimal-single-tenant template comment, platform skill).

The v1.0.8 release adds only one CRD field, spec.failedPodPolicy
(Sandbox and SandboxTemplate, enum Retain|Recreate, default Retain), so
the existing manifests keep their behavior: Retain is the default and
neither the API version (v1beta1) nor any field we set changed. All five
custom resources still validate against the v1.0.8 CRD schemas.

Behavior changes in the controller itself (bounded MCP output, router
kustomization, creation-latency metric timing) do not affect the
platform because it uses neither the sandbox-router nor the MCP server.

Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
2026-10-26 14:17:28 -03:00
SREDevOps.org Agent
2b43d5dfd5 fix(ci): make release image digests unique and allow changelog PRs (#24)
The reusable image build ran once for the main image and once for the gateway in the same release run, but both used digests-<platform> artifact names, so the main merge downloaded a gateway digest and failed. Include a short hash of the full image name in the artifact name and download pattern.

The publish-release job also lacked pull-requests: write, so the generated CHANGELOG PR failed with 'Resource not accessible by integration'. Grant the job contents and pull-requests write.

Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
2026-10-26 10:34:21 -03:00
SREDevOps.org Agent
97abf75dfe chore(changelog): refresh for dsh-v0.2.1-rc.3-r4 (#23)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-10-26 10:27:02 -03:00
SREDevOps.org Agent
e2f1fe567a chore(release): dsh-v0.2.1-rc.3-r4 (#22)
Bump RELEASE_REVISION to r4 for the optional-hardening feature release and refresh every consuming manifest/doc. CHANGELOG.md is generated by the release workflow and is intentionally left untouched.

Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
2026-10-26 10:17:43 -03:00
SREDevOps.org Agent
3e9ba38fc1 feat(example): add optional gVisor, nodeSelector, NetworkPolicy and provider keys (#21)
Add opt-in hardening to the agents-as-a-service example:
- commented runtimeClassName/nodeSelector fields for sandboxes and Deployments
- optional sandbox NetworkPolicy plus exposure-variant policies
- optional dsh-provider-keys-extra Secret for extra provider keys
- docs/optional-hardening.md with verification, caveats and rollback

The base walkthrough stays portable; no gateway code, routes or auth changed.

Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
2026-10-26 10:03:30 -03:00
Nicolás Georger
bf24306421 feat(skills): add agent-sandbox platform skill (#20)
Add the project-local dsh-agent-sandbox-platform skill covering platform operations, the optional agent-sandbox MCP server, feature-extension recipes, and a read-only preflight. Document project skills in README, docs and AGENTS.md.

Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
2026-10-26 08:10:59 -03:00
Nicolás Georger
8681412be4 chore(changelog): refresh for dsh-v0.2.1-rc.3-r3 (#19)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-10-25 19:42:02 -03:00
41 changed files with 2479 additions and 122 deletions

View File

@@ -0,0 +1,156 @@
---
name: dsh-agent-sandbox-platform
description: >-
Operate and extend this repository's DeepSeek Harness-on-agent-sandbox platform:
warm pools, claims, sandbox suspend/resume, the gateway, and the optional
agent-sandbox MCP server, with version-aware safety and a read-only preflight.
whenToUse: >-
Use when working in deepseek-kubernetes-sandbox or with Kubernetes agent-sandbox:
inspect or provision sandboxes, debug warm pools or claims, use
mcp__*__create_sandbox/execute_command, connect the agent-sandbox MCP server, or
add a manifest, gateway route, exposure variant, example, or version bump here.
metadata:
author: sredevopsorg
version: "1.0.0"
---
# DSH agent-sandbox platform
This repository runs DeepSeek Harness Web (DSH) as one sandbox per user on
[agent-sandbox](https://github.com/kubernetes-sigs/agent-sandbox), with an
optional official MCP server for sandbox execution. For generic cluster work
load `kubernetes-kubectl-operations`; for profile MCP rows load
`cordis-plugin-development`.
## 0. Facts before action
1. Read `AGENTS.md`, `versions.env`, the example README/architecture. The
control-plane pin is `AGENT_SANDBOX_VERSION`; release identity is
`DSH_VERSION` + `RELEASE_REVISION`.
2. Establish context, user and namespace; never assume `dsh-demo`.
3. Run `scripts/platform-preflight.sh --namespace <ns>` (read-only).
4. If `mcp__*__create_sandbox`-style tools are absent, use `kubectl`; do not
install or enable a server on your own.
## 1. Non-negotiables
- **Read-only by default.** Change a cluster only when the user asked for that
exact change; show the command and rollback first.
- **Scope every command** with `--context` and `-n`. `--all`/`-A` mutations and
deleting namespaces, PVCs, PVs, CRDs, or cluster-scoped objects need an
explicit, named confirmation.
- **Secrets and logs stay private.** Never print Secret data, kubeconfigs, or
node logs. Pod logs can contain the DSH launch token.
- **The MCP server has no authentication.** It is a full-access management
endpoint; never expose it. A DSH profile row affects every session.
- **Clean up what you create.** Delete MCP-created sandboxes in the same
session; never leave orphaned claims running.
- **Bound output.** Use `--tail`, `--since`, selectors, JSONPath, and
head/tail excerpts instead of whole manifests or huge logs.
## 2. Choose the path
| Need | Path |
| --- | --- |
| Inspect or change claims, pools, sandboxes, PVCs, gateway | `kubectl` + [platform runbook](references/platform-runbook.md) |
| Run code or move files in a sandbox | agent-sandbox MCP, only for a runtime-capable template ([MCP reference](references/mcp-server.md)) |
| Add or change a repository feature | [repo extension guide](references/extending-the-repo.md) |
The MCP server cannot manage this example's DSH claims: its tools are scoped to
claims created by the same MCP session. Use `kubectl` for the platform.
## 3. Operate the platform
Resolve user -> claim -> sandbox -> pod:
```bash
SB=$(kubectl -n "$NS" get sandboxclaim "$CLAIM" -o jsonpath='{.status.sandbox.name}')
kubectl -n "$NS" get sandbox "$SB" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}{" "}{.status.conditions[?(@.type=="Ready")].reason}{"\n"}{.status.serviceFQDN}{"\n"}'
POD=$(kubectl -n "$NS" get sandbox "$SB" -o jsonpath='{.metadata.annotations.agents\.x-k8s\.io/pod-name}'); POD=${POD:-$SB}
```
- Read runtime state from the Sandbox, not the claim: `spec.operatingMode`,
`Suspended`, `Ready`. The claim tracks binding, not liveness.
- Suspend = patch `operatingMode: Suspended` (pod deleted, PVC + Service kept);
resume = `Running` + wait for `Ready`. The gateway wakes on connect.
- Warm-pool spares carry `agents.x-k8s.io/warm-pool-sandbox`; adopted sandboxes
lose it. A pool with no spares makes signup cold and slow.
- The DSH launch token appears once in pod logs; hand the user the `?token=` URL,
never the token in reports. Probe `/favicon.svg`.
- The idle sweeper only suspends claims the gateway created.
Full commands, security boundaries and failure matrix:
[platform runbook](references/platform-runbook.md).
## 4. Use the agent-sandbox MCP server within its limits
Verified from the upstream server:
- Tools: `create_sandbox`, `delete_sandbox`, `execute_command`, `upload_file`,
`download_file`, `list_files`, `file_exists`, `get_sandbox_status`; resource
`sandboxes://{namespace}`.
- **Session-scoped:** every tool and the resource filter claims by
`K8S_SANDBOX_SESSION_ID_LABEL_KEY=<MCP session id>`. It cannot read, exec in,
or delete DSH platform claims or claims from a prior/reconnected session.
- **Streamable HTTP only:** stdio has no MCP session id and fails. A DSH row must
use `transport: streamable-http`; health endpoints are `/healthz` and `/readyz`
at the root, the MCP endpoint is `/mcp`.
- **Runtime required:** connection types target a sandbox runtime on 8888, or
sandboxd on 8080/9090. The DSH template listens only on 3080, so
`execute_command` cannot work against it. A sandboxd sidecar executes in its
own container, not DSH's.
- **Timeouts:** `create_sandbox` waits up to 180 s, but the DSH MCP client's
default call timeout is 60 s; raise `toolCallTimeoutMs`.
`shutdown_after_seconds` defaults to 300 s (`null` disables it).
- **Output:** servers before v1.0.8 return unbounded `execute_command` output.
v1.0.8 adds `max_output_chars` (128-300,000, default 30,000) and truncation
metadata. The repository pins v1.0.8 (`AGENT_SANDBOX_VERSION`), so the
documented deployment is bounded; never pass the new argument to an older
server, and on one redirect verbose output and read a file instead.
Deployment, connection matrix and DSH row:
[MCP deployment](references/mcp-deployment.md). Limits and cleanup:
[MCP reference](references/mcp-server.md).
## 5. Extend the project
Read `docs/examples.md` and the testing table in `AGENTS.md` before editing.
- **Manifest step:** numbered `manifests/NN-*.yaml`, update the README step, run
`make check`; CI dry-runs it.
- **Gateway:** edit `gateway/gateway.py` and `gateway/test_gateway.py`, update
the gateway README API table and architecture doc, run `make test` and
`make lint`. Keep the HTTP/WebSocket contract stable.
- **Exposure variant:** one variant under `manifests/exposure/<variant>/`,
update the exposure README and trusted-host/cookie settings. Variants are
alternatives, never a stack.
- **New example:** follow `docs/examples.md`, add the index entry, keep scripts
executable/idempotent/self-cleaning, run `make check`.
- **Version bump:** edit `versions.env` only, update prerequisites, run
`make check` and the e2e. Never mix a bump with an unrelated feature.
- **MCP integration:** pin the build to the cluster's tag, least-privilege SA,
ClusterIP + auth proxy, raised DSH timeout, no secrets in the repo.
- **Optional hardening:** gVisor, `nodeSelector`, NetworkPolicy and extra
provider keys stay opt-in; the sandbox policy lives under
`manifests/optional/`. All four are documented in
`examples/dsh-agents-as-a-service/docs/optional-hardening.md`.
- **Docs:** update the affected doc in the same change.
Details and review checklist:
[repo extension guide](references/extending-the-repo.md).
## 6. Report and clean up
State context/namespace, bounded evidence, conclusion, exact change, rollback,
and resources deleted. Disable any MCP row enabled only for the task. Do not
leave sandboxes, port-forwards, or temporary files behind.
## References
- [Platform runbook](references/platform-runbook.md)
- [MCP deployment](references/mcp-deployment.md)
- [MCP server reference](references/mcp-server.md)
- [Repo extension guide](references/extending-the-repo.md)
- [Troubleshooting](references/troubleshooting.md)
- Upstream MCP server README:
https://github.com/kubernetes-sigs/agent-sandbox/blob/main/clients/integrations/mcp-server/README.md

View File

@@ -0,0 +1,132 @@
# Extending the repository
Read `AGENTS.md`, `docs/examples.md`, and the example README before editing.
This repository is image-first: the Dockerfile is the primary artifact, every
implementation is a self-contained example under `examples/<name>/`, and
`versions.env` is the only source of pins. Documentation is part of the change.
## Invariants
- Never hard-code a version, image tag, or registry path; edit `versions.env`.
For the DSH release identity use `make set-version`, never ad-hoc `sed`.
- Never commit a secret. Secret-shaped manifests are `*.yaml.example` with
placeholders; keys are documented only in `.env.sample`.
- Manifests are numbered `NN-*.yaml` and apply in order with plain `kubectl`.
Exposure variants live under `manifests/exposure/<variant>/`, are documented
by one README, and are mutually exclusive.
- Keep the gateway HTTP/WebSocket contract stable unless the change is explicitly
about it.
- Do not edit `CHANGELOG.md` (generated by git-cliff) or rewrite git history.
- Keep diffs scoped; do not mix a version bump with an unrelated feature.
## Change matrix
| Change | Edit | Minimum verification |
| --- | --- | --- |
| Documentation only | `*.md`, `docs/` | `make check`; relative links resolve |
| Example manifest | `examples/<name>/manifests/NN-*.yaml`, README step | `make check`; server-side dry-run in CI |
| Optional hardening | `manifests/optional/`, exposure NPs, example `docs/optional-hardening.md` | `make check`; manual verification on a capable cluster |
| Gateway behavior/route | `gateway/gateway.py`, `gateway/test_gateway.py`, gateway README, architecture | `make test`, `make lint`; update tests and API docs |
| Exposure variant | `manifests/exposure/<variant>/`, exposure README, example README | `make check`; apply exactly one variant |
| New example | full `examples/<name>/` layout, `examples/README.md` | `make check`; CI dry-run; optional e2e |
| Version bump | `versions.env`, affected docs/manifests | `make check`; CI installs the pin; e2e on a cluster |
| MCP integration | deployment/RBAC, DSH row, docs | read-only `/readyz`; no secret in repo |
| Workflow | `.github/workflows/` | actionlint; SHA-pinned actions; no hard-coded versions |
## Recipes
### Add or change a resource step
1. Keep numbering stable; add `NN-*.yaml` at the end unless order matters.
2. Update the matching numbered step in the example README.
3. Keep cluster-specific state (node affinity, tuned replica counts) out of the
shipped manifest; document it as a local patch.
4. Run `make check`. CI additionally parses every manifest and runs a
server-side dry-run against the agent-sandbox CRDs.
### Change the gateway
The gateway is a single-file Starlette app. A change usually touches:
- `gateway/gateway.py` (route, auth, proxy, sweeper);
- `gateway/test_gateway.py` (unit tests; no cluster required);
- `gateway/README.md` (API table, configuration, auth semantics);
- `examples/dsh-agents-as-a-service/docs/architecture.md` when flows change;
- `examples/dsh-agents-as-a-service/README.md` when the walkthrough changes.
Run `make test` and `make lint`. Preserve session-cookie auth, same-origin rules
for cookie mutations and WebSocket upgrades, and the "waking"/`502`/`503`
semantics unless the change is explicitly about them. Do not expose a trusted
identity header unless the Service is reachable only through the proxy.
### Add an exposure variant
Create `manifests/exposure/<variant>/NN-*.yaml`, update
`manifests/exposure/README.md` and the example README, and state the trust
boundary. Variants are alternatives: never apply two paths to the same gateway.
Set `DSH_TRUSTED_HOSTS` in the template for the public host and
`COOKIE_SECURE=true` for HTTPS.
### Add an example
Follow `docs/examples.md`: `README.md`, `manifests/NN-*.yaml`, optional
`manifests/exposure/` with a README, optional components with Dockerfile +
README + tests, and executable, idempotent, self-cleaning scripts. Add the
literal `examples/<name>/` entry to `examples/README.md`; `make check` enforces
the layout. Publish component images under
`<registry>/<owner>/<repo>/<example>/<component>`.
### Bump a version
- **DSH release identity:** `make set-version DSH_VERSION=<ref>
[RELEASE_REVISION=rN]`, then `make check`, then build and smoke-test the image.
Tags are immutable; fix a bad release with `-r(N+1)`.
- **agent-sandbox:** edit `AGENT_SANDBOX_VERSION` in `versions.env`, update the
prerequisites in the example README, and note behavior changes (connection
types, MCP output limits, CRD fields). Install the new release YAML on a kind
cluster, then run `make check` and `make e2e`. Do not bump it inside an
unrelated PR.
### Integrate the agent-sandbox MCP server
Build from the cluster's pinned tag, give it a dedicated namespace and
least-privilege ServiceAccount, keep it ClusterIP behind an authenticating
proxy, and document how to reach `/readyz`. Add a DSH profile row only with user
approval; a row affects every session. Never commit its credentials or a
`cluster-admin` binding.
### Documentation
Update the doc whose behavior changed in the same commit: root README for the
front door, `docs/` for repository-wide contracts, the example README for that
walkthrough, the gateway README for API/config/auth. Skill docs live in
`.dsh/skills/`; keep `SKILL.md` under ~8 KB and put depth in `references/`.
`scripts/check-versions.sh` scans every `*.md` for stale `dsh-v…` tokens, so
refer to "the release identity in `versions.env`" rather than copying a tag.
## Verification and review
Run at least `make check`; add `make test` and `make lint` for gateway/script
changes. For manifests, use `kubectl apply --dry-run=server` against a cluster
with the CRDs. For behavior changes, run the example or `make e2e`.
Before reporting, review:
- [ ] Diff is scoped; no generated files or unrelated changes.
- [ ] Docs and examples match the new behavior.
- [ ] Pins come only from `versions.env`; no stale version tokens.
- [ ] No secrets or usable credentials; secret-shaped files are `.example`.
- [ ] Manifests parse; exposure variants remain mutually exclusive.
- [ ] Gateway tests and public contract unchanged or intentionally updated.
- [ ] Skills: frontmatter valid, body within budget, scripts shellcheck-clean.
- [ ] `CHANGELOG.md` untouched; branch/commit follow `docs/gitflow.md`.
## Safe follow-ups enabled by this skill
- A read-only preflight job or scheduled report using
`scripts/platform-preflight.sh`.
- An MCP readiness check in the example e2e (health endpoints only).
- A separate runtime warm pool for untrusted execution, leaving the DSH template
unchanged.
- A combined DSH + sandboxd image only with an explicit maintenance/exec design
and a documented trust boundary.

View File

@@ -0,0 +1,94 @@
# Deploying the agent-sandbox MCP server
Read [the MCP reference](mcp-server.md) first. This file covers the connection
matrix, a safe deployment shape, and the DSH profile row. The server is
unauthenticated; its service account is the security boundary.
## Connection matrix
| `K8S_SANDBOX_CONNECTION__TYPE` | Required env | Use when |
| --- | --- | --- |
| `gateway` | `GATEWAY_NAME`, `GATEWAY_NAMESPACE=default`, `GATEWAY_READY_TIMEOUT=180`, `SERVER_PORT=8888` | Gateway API + implementation exist |
| `direct` | `API_URL` (router URL), `SERVER_PORT=8888` | sandbox-router is deployed |
| `in-cluster` | `USE_POD_IP=false` (DNS) or `true` (`status.podIPs`), `SERVER_PORT=8888` | Bypass the router from inside the cluster |
| `sandboxd-in-cluster` (v1.0.8+) | `MODE=in-cluster-service` or `in-cluster-pod-ip`, `REST_PORT=8080`, `GRPC_PORT=9090` | Template runs sandboxd |
Also set `K8S_SANDBOX_PROBE_NAMESPACE` to a namespace the server's service
account can list claims in, and optionally
`K8S_SANDBOX_SESSION_ID_LABEL_KEY` (default
`mcp.k8s-agent-sandbox/session-id`).
For `sandboxd-in-cluster` service mode the template needs `spec.service: true`.
The default managed NetworkPolicy admits only sandbox-router, so add an ingress
rule for the MCP server on 8080/9090. Supplying `spec.networkPolicy` replaces the
secure defaults: also add the egress rules the sandbox needs, because an omitted
egress list denies all outbound traffic. RBAC for that mode is
`sandboxclaims` create/get/list/watch/delete and `sandboxes` get/list/watch; no
`pods/portforward`.
## Build and deploy (sketch)
No official image reference is documented in the upstream README; pin what you
build.
```bash
# AGENT_SANDBOX_VERSION comes from versions.env; never hard-code it.
tmp="$(mktemp -d)"; trap 'rm -rf "$tmp"' EXIT
git clone --depth 1 --branch "$AGENT_SANDBOX_VERSION" \
https://github.com/kubernetes-sigs/agent-sandbox.git "$tmp"
docker build -t "$MCP_IMAGE" "$tmp/clients/integrations/mcp-server"
```
Kubernetes shape:
- Dedicated namespace and ServiceAccount; a namespace-scoped Role granting only
`sandboxclaims` create/get/list/watch/delete and `sandboxes` get/list/watch.
No Secrets, no `pods/exec`, no cluster-admin.
- Deployment: one replica, non-root, `readOnlyRootFilesystem`, dropped
capabilities, `K8S_SANDBOX_CONNECTION__TYPE=in-cluster`,
`K8S_SANDBOX_PROBE_NAMESPACE`, resource limits.
- Probes: `GET /healthz` (liveness; never calls the API) and `GET /readyz`
(readiness; lists claims; use `periodSeconds: 15`, not aggressive polling).
- ClusterIP Service on `8000`. Add a NetworkPolicy that allows only the intended
client namespace/pod. Never expose it through Ingress/LoadBalancer without an
authenticating proxy. The service account is the real boundary.
## Connect it to DSH
Add a profile MCP row (through the Plugins page/`plugin_manager`; see
`cordis-plugin-development`). A row affects every session and needs approval.
```yaml
- insert:
- id: mcp-agent-sandbox
name: '@deepseek-ai/dsh-mcp-client'
config:
serverName: agent-sandbox
transport: streamable-http
url: http://agent-sandbox-mcp.example.svc.cluster.local:8000/mcp
toolCallTimeoutMs: 240000
failOnStartupError: false
```
- `serverName` must match `[A-Za-z0-9_-]{1,32}`; tools appear as
`mcp__<serverName>__<tool>`. Check the actual names in the tool list; do not
assume hyphens/underscores.
- Raise `toolCallTimeoutMs` above `sandbox_ready_timeout` with headroom; the DSH
default is 60 s and would cancel a 180 s creation.
- If the server is behind an authenticating proxy, pass its token via the row's
`headers` and an environment expression; never commit it.
- To reach a locally port-forwarded server, use a loopback URL; the MCP session
still needs streamable HTTP.
- Remove the row when it was enabled only for one task.
## Deployment safety checklist
- [ ] Build tag equals the cluster's `AGENT_SANDBOX_VERSION`.
- [ ] Dedicated namespace/SA; RBAC limited to claims and sandboxes.
- [ ] Service is ClusterIP; no public route without authentication.
- [ ] NetworkPolicy admits only the DSH client and needed sandbox traffic.
- [ ] `/readyz` succeeds with `K8S_SANDBOX_PROBE_NAMESPACE` set to a listable
namespace.
- [ ] No Secret data in manifests, logs, or the repository.
- [ ] Cleanup path documented: delete session-owned claims or wait for the
shutdown timeout.

View File

@@ -0,0 +1,107 @@
# agent-sandbox MCP server reference
Upstream: [clients/integrations/mcp-server](https://github.com/kubernetes-sigs/agent-sandbox/tree/main/clients/integrations/mcp-server).
It is a Python/FastAPI + FastMCP service (streamable HTTP on `:8000`, MCP
endpoint `/mcp`) that provisions sandboxes from a warm pool and runs commands or
file operations through a sandbox runtime. It ships with **no authentication**.
Build the server from the same tag as the cluster: read `AGENT_SANDBOX_VERSION`
in `versions.env`, and check the installed controller image. Do not build from
`main` against an older cluster; tool schemas and connection types differ.
## Tool surface
| Tool | Arguments (defaults) | Returns / notes |
| --- | --- | --- |
| `create_sandbox` | `warmpool`, `namespace`, `sandbox_ready_timeout=180`, `labels`, `shutdown_after_seconds=300` (`null` disables), `pod_labels`, `pod_annotations` | `sandbox_claim_name`; adds the MCP session label |
| `delete_sandbox` | `sandbox_claim_name`, `namespace` | Deletes the session-owned claim; PVC cascades |
| `execute_command` | `sandbox_claim_name`, `namespace`, `command`, `timeout=60` (max 600), `max_output_chars` only on v1.0.8+ (128-300,000, default 30,000) | `exit_code`, `stdout`, `stderr`; v1.0.8 adds `*_total_chars` and `*_truncated` |
| `upload_file` | `sandbox_claim_name`, `namespace`, `path`, `content`, `binary=false`, `timeout=60` | Bytes written; base64 when `binary=true` |
| `download_file` | `sandbox_claim_name`, `namespace`, `path`, `binary=false`, `timeout=60` | Content + bytes read |
| `list_files` | `sandbox_claim_name`, `namespace`, `path`, `timeout=60`, `max_entries=1000` (max 10,000) | Entries (`name`, `size`, `type`, `mod_time`), `total_entries`, `truncated` |
| `file_exists` | `sandbox_claim_name`, `namespace`, `path`, `timeout=60` | `exists`; a missing path is not an error |
| `get_sandbox_status` | `sandbox_claim_name`, `namespace` | `SandboxReady` / `SandboxNotReady` / `SandboxNotFound`, `ready`, `message` |
Resource: `sandboxes://{namespace}` lists sandboxes for the current MCP session.
## Session scoping is the most important behavior
Every tool, including `get_sandbox_status`, calls `get_sandbox()` ->
`ensure_session_owns()`, which lists sandboxes with the label selector
`<K8S_SANDBOX_SESSION_ID_LABEL_KEY>=<MCP session id>`. The resource uses the same
selector. Consequences:
- The server can only see/exec/delete claims **it created in the same MCP
session**. It cannot inspect or manage DSH platform claims created by the
gateway or `kubectl`, or claims from another session.
- On a DSH MCP reconnect, the MCP session changes; earlier claims become
invisible to the tools. They keep running until `shutdown_after_seconds`
expires, or until you remove them by label with a confirmed `kubectl delete`.
- A "Sandbox claim '<name>' is not found in namespace '<ns>'" error usually
means wrong namespace **or** the claim belongs to a different session.
- Use `kubectl` for global inventory; use the MCP only for sandboxes it created
in this task.
Session identity requires a session-providing transport. With stdio,
`ctx.session_id` is `None` and creation fails. Connect DSH over
`streamable-http`.
## Runtime prerequisite
The server talks to a sandbox runtime, not to arbitrary pods:
- `gateway`, `direct`, `in-cluster`: runtime HTTP on `SERVER_PORT=8888`
(sandbox-router API or direct pod runtime).
- `sandboxd-in-cluster` (v1.0.8+): sandboxd REST `8080`, gRPC `9090`.
The shipped DSH template runs only DSH Web on `3080`, so no connection type can
exec into it. A sandboxd **sidecar** executes commands in the sidecar's own
container, not in the DSH container; shared volumes share files, not binaries or
processes. To exec inside DSH, build a combined image (sandboxd injected into the
DSH image) or use `kubectl exec` with explicit RBAC and confirmation. For
untrusted code, deploy a separate runtime warm pool rather than modifying the
DSH template.
## Version differences
| Area | Before v1.0.8 | v1.0.8+ |
| --- | --- | --- |
| Connection types | `gateway`, `direct`, `in-cluster` | Adds `sandboxd-in-cluster` |
| `execute_command` output | Unbounded | Bounded by `max_output_chars`; four metadata fields |
| Failed pod recovery | `Retain` only | `spec.failedPodPolicy: Recreate` |
| Router install | `kubectl apply -f sandbox-router/deploy/` | `kubectl apply -k sandbox-router/deploy/`; projected token |
Never pass `max_output_chars` to a pre-v1.0.8 server. On older servers, redirect
verbose output to a file and `download_file` it, or pipe through `tail`/`head`.
The repository pins `AGENT_SANDBOX_VERSION=v1.0.8`, so the right-hand column is
what the documented platform and a same-tag MCP build provide; keep the
before-column for clusters still on v1.0.7.
## Safety and cleanup
- No auth: anyone reaching `:8000` gets every tool and the service account's
access. Keep it ClusterIP, authenticated, and namespace-scoped.
- Treat commands as untrusted code: they can read files and environment
variables inside the sandbox, including provider keys in a combined image.
- `delete_sandbox` cascades the claim, sandbox, and PVC: confirm the target and
namespace. Never delete a namespace or PVC as cleanup.
- Clean up in the same session that created the sandbox. If the session is gone,
list claims by the session label and delete them explicitly (requires write
RBAC); otherwise rely on the shutdown timeout.
## Troubleshooting
| Symptom | Cause / fix |
| --- | --- |
| Tools missing | Row not enabled, wrong URL, connection failed; check logs and `/readyz` |
| 404 on connect | URL is missing `/mcp` or points at the wrong port |
| `ctx.session_id is None` | Connected over stdio; use streamable HTTP |
| `Sandbox claim ... is not found` | Wrong namespace, or claim created by another MCP session |
| Creation times out at ~60 s | MCP session alive but DSH cancelled the call; raise `toolCallTimeoutMs` |
| `execute_command` connection error | No runtime on 8888/8080/9090, or wrong connection type; the DSH template is unsupported |
| Output flood | Pre-v1.0.8 server; bound the command output yourself |
| Empty sandbox list | The resource is session-scoped; use `kubectl` for global inventory |
Deployment, connection matrix, RBAC, and the DSH row:
[MCP deployment](mcp-deployment.md).

View File

@@ -0,0 +1,181 @@
# Platform runbook
Scope: the resource layer and gateway in
`examples/dsh-agents-as-a-service/`, one namespace, one DSH Web sandbox per
user. Read the example README and `AGENTS.md` first. All commands are read-only
unless a section is explicitly a change; confirm before changing anything.
## 1. Context, version, CRDs
```bash
kubectl config current-context
kubectl auth whoami
kubectl config view --minify -o jsonpath='{..namespace}{"\n"}'
kubectl get crd | grep -E 'agents\.x-k8s\.io'
kubectl -n agent-sandbox-system get deploy agent-sandbox-controller \
-o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
```
Compare the controller image tag with `AGENT_SANDBOX_VERSION` in
`versions.env`. The example needs the extension CRDs plus core `sandboxes`;
a missing CRD is a prerequisite failure, not an application bug.
## 2. Resource model
| Object | Role | Key fields |
| --- | --- | --- |
| `SandboxTemplate` | Blueprint; pinned image, probes, security context, PVC template | `envVarsInjectionPolicy`, `volumeClaimTemplatesPolicy`, `service`, `podTemplate` |
| `SandboxWarmPool` | Pre-warmed spares from one template | `spec.replicas`, `sandboxTemplateRef` |
| `SandboxClaim` | One tenant/user request; adopts a spare | `spec.warmPoolRef`, `additionalPodMetadata`, `status.sandbox.name` |
| `Sandbox` | Adopted or directly created pod + Service + PVC | `spec.operatingMode`, `status.conditions`, `status.serviceFQDN`, `status.podIPs` |
| `Pod` | The DSH Web process | `:3080`, PVC at `/home/node/.dsh` |
| `PersistentVolumeClaim` | Per-sandbox state | `dsh-home-<sandbox>` |
| `Service` | Stable per-sandbox DNS | `<sandbox>`, `status.serviceFQDN` |
Delete a claim to cascade-delete its sandbox and PVC (account deletion). Never
delete a PVC or namespace without an explicit, named confirmation.
## 3. Read-only inspection
```bash
kubectl -n "$NS" get sandboxtemplates,sandboxwarmpools,sandboxclaims,sandboxes -o wide
kubectl -n "$NS" get pods,pvc,svc -o wide
kubectl -n "$NS" get events --sort-by=.lastTimestamp | tail -30
kubectl -n "$NS" get sandboxclaim "$CLAIM" -o yaml
kubectl -n "$NS" get sandbox "$SB" -o yaml
```
Bounded logs: `kubectl -n "$NS" logs "$POD" --tail=200` (add `--previous`
only after a restart).
## 4. Claim -> sandbox -> pod
```bash
SB=$(kubectl -n "$NS" get sandboxclaim "$CLAIM" -o jsonpath='{.status.sandbox.name}')
POD=$(kubectl -n "$NS" get sandbox "$SB" -o jsonpath='{.metadata.annotations.agents\.x-k8s\.io/pod-name}')
POD=${POD:-$SB}
kubectl -n "$NS" get sandbox "$SB" -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}{"\n"}{end}'
kubectl -n "$NS" get sandbox "$SB" -o jsonpath='{.spec.operatingMode}{" "}{.status.serviceFQDN}{" "}{.status.podIPs}{"\n"}'
```
Rules:
- Resolve user -> sandbox only through `claim.status.sandbox.name`; the sandbox
keeps its pool-generated name, not the claim name.
- A claim tracks binding; for runtime state read `Sandbox.spec.operatingMode`
and the `Suspended`/`Ready` conditions. A suspended sandbox is not serving
even when the claim still exists.
- A missing `claim.status.sandbox.name` means the claim did not bind: inspect
the Ready condition `reason` and events. The pod is normally named after the
sandbox; use the annotation when present.
## 5. Suspend, resume, wake
Suspend keeps the PVC and Service, deletes the pod:
```bash
kubectl -n "$NS" patch sandbox "$SB" --type merge \
-p '{"spec":{"operatingMode":"Suspended"}}'
kubectl -n "$NS" wait --for=delete "pod/$POD" --timeout=120s
kubectl -n "$NS" get pvc "dsh-home-$SB"
kubectl -n "$NS" get svc "$SB"
```
Resume and verify state:
```bash
kubectl -n "$NS" patch sandbox "$SB" --type merge \
-p '{"spec":{"operatingMode":"Running"}}'
kubectl -n "$NS" wait sandbox "$SB" --for=condition=Ready --timeout=180s
kubectl -n "$NS" exec "$SB" -- cat /home/node/.dsh/marker.txt
```
The gateway wakes a suspended sandbox on the first request. The idle sweeper
only suspends claims it created (`aaas.example.com/managed-by=dsh-gateway`);
hand-applied claims are never swept.
## 6. Warm pool
```bash
kubectl -n "$NS" get sandboxwarmpool "$POOL" -o yaml
kubectl -n "$NS" get sandboxes -l agents.x-k8s.io/warm-pool-sandbox -o wide
kubectl -n "$NS" get sandboxwarmpool "$POOL" -o jsonpath='{.spec.replicas}{"\n"}'
```
Adopted sandboxes lose the warm-pool label; count current spares against
`spec.replicas`. Zero spares or `spec.replicas: 0` means cold, slow signup.
Scale only when asked:
```bash
kubectl -n "$NS" patch sandboxwarmpool "$POOL" --type merge -p '{"spec":{"replicas":2}}'
```
## 7. DSH runtime specifics
- DSH Web listens on `3080` in-cluster. Port-forward to a free local port such
as `18081`, never `3080` if a local DSH already uses it.
- The launch token is printed once as `dsh web: http://<ip>:3080/?token=...`.
Treat it as a credential: do not paste it into reports; hand the user the URL.
- `/` returns `401` without the token. Probes must target `/favicon.svg`.
- `DSH_TRUSTED_HOSTS` must include any public host that a proxy forwards; DSH
fences on `Host`/`Origin`.
- State lives at `/home/node/.dsh`. DSH requires `.credentials.yaml` to be
`0600`; the template uses `fsGroupChangePolicy: OnRootMismatch`. If a resumed
sandbox crash-loops with a mode-660 error, suspend it, fix the file with a
throwaway pod, patch the policy, then resume.
## 8. Gateway layer
```bash
kubectl -n "$NS" get deploy,svc,pods -l app=dsh-gateway -o wide
kubectl -n "$NS" get endpoints dsh-gateway
kubectl -n "$NS" logs deploy/dsh-gateway --tail=100
```
The gateway is a ClusterIP Service on `8080`; it proxies under `/u/{user}/` and
derives state (`Provisioning`, `Waking`, `Ready`, `Suspending`, `Suspended`).
It performs the DSH launch-token handshake server-side. With SSO enabled
(`AUTH_PROXY_HEADER`), the Service must be reachable only through the
authenticating proxy; never add a second path or port-forward it on a shared
host. A resource-only install has no gateway: that is a warning, not a failure.
## 9. Security boundaries
- Provider keys (`dsh-provider-keys` plus the optional
`dsh-provider-keys-extra`) are shared by all sandboxes; the example is not
tenant-isolated.
- gVisor, `nodeSelector`, the sandbox/exposure NetworkPolicies and extra
provider keys are opt-in, not active by default. A NetworkPolicy only helps
with a CNI that enforces it; see
[optional hardening](../../../../examples/dsh-agents-as-a-service/docs/optional-hardening.md).
- Secret data, kubeconfigs, and pod logs are sensitive; RBAC may allow Secret
access, policy does not. Keep gateway RBAC namespace-scoped; it needs pod logs
to find the launch token.
- TLS terminates outside the gateway; set `COOKIE_SECURE=true` for HTTPS.
## 10. Failure matrix
| Symptom | First evidence | Likely cause / fix |
| --- | --- | --- |
| Claim not Ready, no sandbox | Ready condition reason, events | Empty pool, image pull, quota, unschedulable |
| `EnvVarsInjectionRejected` | claim Ready reason | Claim carries env/volume templates; template policy rejects it |
| Pod Pending | `kubectl describe pod` | Requests/taints/affinity, unbound PVC |
| CrashLoopBackOff | logs, `--previous` | Startup failure, bad config, probe 401, fsGroup mode 660 |
| ImagePullBackOff | describe events | Wrong tag, private registry, missing pull secret |
| DSH 401 in browser | gateway logs | Launch-token handshake failed; check gateway RBAC `pods/log` |
| 502/503 through gateway | sandbox Ready, gateway logs | Wake timeout, DNS stale, token not in logs yet |
| Warm pool empty | sandboxes label count | Scale the pool or wait for replenishment |
| Suspended but expected running | `operatingMode` | Resume it; the gateway only wakes on a request |
| WS closes `1011` | gateway logs | DSH rejected the upstream Host/authority; check trusted hosts |
## 11. Cleanup and reporting
Delete only claims/sandboxes you created, by name:
```bash
kubectl -n "$NS" delete sandboxclaim "$CLAIM" --dry-run=server
kubectl -n "$NS" delete sandboxclaim "$CLAIM"
```
Report context/namespace, evidence, conclusion, change, rollback, and what was
deleted. Leave the namespace, pools, templates, and unrelated claims alone.

View File

@@ -0,0 +1,72 @@
# Troubleshooting
Start with evidence, not fixes: resolve context/namespace, read conditions and
events, then compare desired vs observed state. Keep output bounded. Never
"fix" an access error by broadening credentials.
## Platform
| Symptom | First evidence | Likely cause / fix |
| --- | --- | --- |
| `Forbidden` on every call | `kubectl auth whoami`, `kubectl auth can-i --list -n <ns>` | Wrong context or RBAC. Do not retry with broader credentials; ask for the right kubeconfig. |
| CRDs not found | `kubectl get crd | grep agents.x-k8s.io` | agent-sandbox extensions not installed. Install the pinned release YAML; that is a prerequisite, not a code change. |
| Claim created, no sandbox | `claim.status.conditions[?(@.type=="Ready")]`, events | Empty pool, image pull, quota, unschedulable; or policy rejection. |
| Claim rejected | Ready reason `EnvVarsInjectionRejected` | Claim carries `env`/`volumeClaimTemplates` while the template disallows injection. Remove them; do not bypass the pool. |
| User is suspended but the claim exists | `Sandbox.spec.operatingMode`, `Suspended` condition | Resume with `operatingMode: Running`; a gateway request would wake it. |
| Warm pool exhausted | `kubectl get sandboxes -l agents.x-k8s.io/warm-pool-sandbox` | Scale `spec.replicas` (with confirmation) or accept slow cold provisioning. |
| Pod `Pending` | `kubectl describe pod` | Requests/limits, taints, affinity, unbound PVC, storage class. |
| `ImagePullBackOff` | describe events | Wrong image/tag, private GHCR package, missing pull secret; on kind, load the image under the exact manifest ref. |
| `CrashLoopBackOff`, probe 401 | logs, `describe` | Probe path is `/`; use `/favicon.svg`. |
| Restart with mode-660 error | logs mention `.credentials.yaml` | kubelet fsGroup widened the file; suspend, `chmod 600` from a throwaway pod, patch `fsGroupChangePolicy: OnRootMismatch`, resume. |
| PVC `Pending` | `kubectl describe pvc`, StorageClass | No default provisioner, access mode, zone binding. |
| Launch token not found | `kubectl logs <pod> --tail=2000 | grep 'dsh web:'` | Pod not ready yet, or logs rotated. Do not guess tokens; wait for readiness. |
| Sandbox reports `Finished=True` | pod phase, `kubectl get events` | The backing Pod reached `Failed` (eviction, spot reclaim) and `failedPodPolicy` defaults to `Retain`. Delete the Pod to force a replacement, or set `Recreate` (v1.0.8+) on the Sandbox/template; the claim and PVCs survive either way. |
| Hand-applied sandbox disappeared | claim annotations, gateway logs | Idle sweeper only manages claims with `aaas.example.com/managed-by=dsh-gateway`; if it was managed, recreate/resume. |
| Gateway 401 | gateway logs, cookie flags | Missing/expired cookie, DSH handshake failed, or `COOKIE_SECURE=true` over HTTP. |
| Gateway 502/503 | sandbox Ready, gateway logs | Wake timeout, stale DNS, token not yet in logs; check `WAKE_TIMEOUT` and RBAC `pods/log`. |
| WebSocket closes 1011 | gateway logs, `Host`/Origin | DSH rejected the upstream authority; check `DSH_TRUSTED_HOSTS` and the gateway's pod-IP dial path. |
| WebSocket closes 4401 | cookie, Origin | Missing gateway auth or cross-origin upgrade; send a matching `Origin`. |
| SSO header ignored | gateway env | `AUTH_PROXY_HEADER` unset; setting it requires the proxy to be the only path to the ClusterIP Service. |
| Someone is treated as another user | service paths | Trust-boundary violation: the gateway is directly reachable. Remove the extra path and disable SSO until fixed. |
## Agent-sandbox MCP
| Symptom | First evidence | Likely cause / fix |
| --- | --- | --- |
| No `mcp__*__create_sandbox` tools | tool list, DSH logs | Server not deployed or row not enabled. Use `kubectl`; connect only with approval. |
| 404 on connect | curl `/mcp`, `/healthz` | Wrong path/port; MCP endpoint is `/mcp`, health is at the root. |
| `/readyz` 503 | response body, server logs | API unreachable or namespace `K8S_SANDBOX_PROBE_NAMESPACE` not listable; fix RBAC/config. |
| `ctx.session_id is None` | server error | Connected over stdio; use streamable HTTP. |
| `Sandbox claim ... is not found` | server error, `kubectl get claim` | Wrong namespace, or the claim belongs to another MCP session. List with `kubectl`; use the MCP only for its own claims. |
| Creation cancelled at ~60 s | MCP call log | DSH `toolCallTimeoutMs` is 60 s while creation waits up to 180 s; raise the timeout. |
| `execute_command` connection error | server logs, template ports | No runtime on 8888/8080/9090. The DSH template only listens on 3080; use a runtime template or a combined image. |
| Output truncated / flooded | tool result | Pre-v1.0.8 is unbounded; v1.0.8 bounds output. Redirect verbose output to a file and read/download it; never pass `max_output_chars` to an old server. |
| Orphaned claims after reconnect | `kubectl get claims -l mcp.k8s-agent-sandbox/session-id` | MCP sessions are transient; wait for `shutdown_after_seconds` or delete by label with explicit confirmation. |
| `Forbidden` from the MCP server | server logs | Its ServiceAccount lacks claim/sandbox verbs, or RBAC is in another namespace; fix the Role, not the kubeconfig. |
| Sandbox expired | `get_sandbox_status` -> `SandboxNotFound` | `shutdown_after_seconds` elapsed; create a new sandbox and keep the task inside the budget. |
## Repository checks and CI
| Symptom | Cause / fix |
| --- | --- |
| `check-versions: stale version reference in docs: dsh-v...` | A copied release tag in a Markdown file. Refer to the identity in `versions.env`; run `make set-version` only for a real bump. |
| `check-versions: hard-coded version outside versions.env` | A tag in `.github/workflows` or the Makefile; move it to `versions.env`. |
| `check-examples` layout error | Missing `README.md`/`manifests/`, stray YAML at the example root, missing exposure README, non-executable script, or missing `examples/README.md` entry. |
| Manifest dry-run fails in CI | Invalid YAML or CRD schema; parse locally and run `kubectl apply --dry-run=server` against the pinned CRDs. |
| `shellcheck` failure | Quote expansions, use arrays for kubectl args, avoid unused variables; run shellcheck locally. |
| Lychee link failure | A relative/absolute link does not resolve; the link checker runs offline on the listed docs. |
| PR-title check failure | Use Conventional Commits: `feat(scope): ...`, `fix(scope): ...`, `docs(scope): ...`. |
| Changelog conflict | Never edit `CHANGELOG.md`; it is generated from commits. |
## Do not
- Do not use `--force --grace-period=0` to clear a stuck resource before
understanding why it is stuck.
- Do not delete namespaces, PVCs, PVs, CRDs, or cluster-scoped objects as a
cleanup shortcut.
- Do not read Secret data, kubeconfigs, or node logs to "debug" a permission
problem, and do not paste pod logs containing the DSH launch token into a
report.
- Do not expose the unauthenticated MCP server or enable `AUTH_PROXY_HEADER`
while another path to the gateway exists.
- Do not retry a `Forbidden` with a stronger kubeconfig; fix the RBAC scope.

View File

@@ -0,0 +1,298 @@
#!/usr/bin/env bash
# Read-only preflight for the DSH agents-as-a-service platform.
#
# Checks the kubectl target, agent-sandbox CRDs/version, warm pools, claims,
# sandboxes, gateway and (optionally) an agent-sandbox MCP server. It never
# mutates the cluster, never reads Secret data, and never port-forwards.
#
# Usage:
# platform-preflight.sh [--context NAME] [--namespace NS] [--pool NAME]
# [--template NAME] [--mcp-url URL] [--help]
#
# Exit codes: 0 = no failures, 1 = at least one failure, 2 = usage error.
set -euo pipefail
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(git -C "$SCRIPT_DIR" rev-parse --show-toplevel 2>/dev/null || true)"
CTX=""
NS="${NAMESPACE:-dsh-demo}"
POOL=""
TEMPLATE=""
MCP_URL=""
PASS=0
WARN=0
FAIL=0
usage() {
cat <<'EOF'
Read-only preflight for the DSH agents-as-a-service platform.
Options:
--context NAME kubectl context (default: current context)
--namespace NS namespace to inspect (default: $NAMESPACE or dsh-demo)
--pool NAME check only this warm pool
--template NAME check only this SandboxTemplate
--mcp-url URL also check an MCP server's /healthz and /readyz
--help show this help
The script makes no changes and never reads Secret data.
EOF
}
while [ "$#" -gt 0 ]; do
case "$1" in
--context) CTX="${2:-}"; shift 2 ;;
--namespace) NS="${2:-}"; shift 2 ;;
--pool) POOL="${2:-}"; shift 2 ;;
--template) TEMPLATE="${2:-}"; shift 2 ;;
--mcp-url) MCP_URL="${2:-}"; shift 2 ;;
--help|-h) usage; exit 0 ;;
*) echo "unknown argument: $1" >&2; usage >&2; exit 2 ;;
esac
done
if [ -z "$NS" ]; then
echo "namespace must not be empty" >&2
exit 2
fi
section() { printf '\n== %s ==\n' "$1"; }
pass() { PASS=$((PASS + 1)); printf 'PASS %s\n' "$1"; }
warn() { WARN=$((WARN + 1)); printf 'WARN %s\n' "$1"; }
fail() { FAIL=$((FAIL + 1)); printf 'FAIL %s\n' "$1"; }
info() { printf 'INFO %s\n' "$1"; }
command -v kubectl >/dev/null 2>&1 || {
echo "FAIL kubectl is not installed" >&2
exit 1
}
if [ -n "$CTX" ]; then
if ! kubectl config get-contexts -o name 2>/dev/null | grep -Fxq "$CTX"; then
echo "FAIL context '$CTX' is not in the kubeconfig" >&2
exit 1
fi
KUBECTL=(kubectl --context "$CTX" --request-timeout=15s)
else
KUBECTL=(kubectl --request-timeout=15s)
fi
k() { "${KUBECTL[@]}" "$@"; }
section "Target"
if current="$(k config current-context 2>/dev/null)"; then
info "context: $current"
pass "kubeconfig is readable"
else
fail "cannot read the current kubectl context"
fi
if user="$(k auth whoami -o jsonpath='{.status.userInfo.username}' 2>/dev/null)" && [ -n "$user" ]; then
info "identity: $user"
else
fallback_user="$(k config view --minify -o jsonpath='{.users[0].name}' 2>/dev/null || true)"
info "identity: ${fallback_user:-unknown}"
fi
if k get namespace "$NS" -o name >/dev/null 2>&1; then
pass "namespace $NS exists"
else
fail "namespace $NS is missing or not readable"
fi
if [ -n "$REPO_ROOT" ] && [ -f "$REPO_ROOT/versions.env" ]; then
# shellcheck source=/dev/null
. "$REPO_ROOT/versions.env"
info "expected agent-sandbox pin: ${AGENT_SANDBOX_VERSION:-unknown}"
else
warn "versions.env not found; skipping the pin comparison"
fi
section "Capabilities (informational; never widen them)"
for check in "create sandboxclaims" "delete sandboxclaims" "create pods/exec"; do
verb="${check%% *}"
resource="${check#* }"
if [ "$(k auth can-i "$verb" "$resource" -n "$NS" 2>/dev/null || echo unknown)" = "yes" ]; then
info "can $verb $resource in $NS: yes"
else
info "can $verb $resource in $NS: no"
fi
done
if [ "$(k auth can-i get secrets -n "$NS" 2>/dev/null || echo unknown)" = "yes" ]; then
warn "RBAC allows reading Secrets; policy still forbids printing Secret data"
else
info "can get secrets in $NS: no"
fi
section "Agent Sandbox CRDs"
for crd in \
sandboxtemplates.extensions.agents.x-k8s.io \
sandboxwarmpools.extensions.agents.x-k8s.io \
sandboxclaims.extensions.agents.x-k8s.io \
sandboxes.agents.x-k8s.io; do
if k get crd "$crd" -o name >/dev/null 2>&1; then
pass "CRD $crd"
else
fail "CRD $crd is missing or not readable"
fi
done
section "Controller"
controller_ns="agent-sandbox-system"
controller_image="$(k -n "$controller_ns" get deployment agent-sandbox-controller \
-o jsonpath='{.spec.template.spec.containers[0].image}' 2>/dev/null || true)"
if [ -n "$controller_image" ]; then
info "controller image: $controller_image"
controller_tag="${controller_image##*:}"
if [ -n "${AGENT_SANDBOX_VERSION:-}" ] && [ "$controller_tag" != "$AGENT_SANDBOX_VERSION" ]; then
warn "controller tag '$controller_tag' differs from AGENT_SANDBOX_VERSION '$AGENT_SANDBOX_VERSION'"
else
pass "controller version matches versions.env"
fi
else
warn "controller deployment not found in $controller_ns; cannot verify the installed version"
fi
section "Templates"
templates="$(k -n "$NS" get sandboxtemplates \
-o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' 2>/dev/null || true)"
if [ -z "$templates" ]; then
warn "no SandboxTemplates in $NS"
fi
if [ -n "$TEMPLATE" ] && ! printf '%s\n' "$templates" | grep -Fxq "$TEMPLATE"; then
fail "SandboxTemplate $TEMPLATE not found in $NS"
fi
for tpl in $templates; do
[ -n "$TEMPLATE" ] && [ "$tpl" != "$TEMPLATE" ] && continue
desc="$(k -n "$NS" get sandboxtemplate "$tpl" \
-o jsonpath='{range .spec.podTemplate.spec.containers[*]}{.name}{":"}{range .ports[*]}{.containerPort}{","}{end}{";"}{end}' \
2>/dev/null || true)"
info "template $tpl containers: ${desc:-<none>}"
case "$desc" in
*sandboxd*|*:8080,*|*:9090,*)
pass "template $tpl exposes a sandboxd runtime (MCP sandboxd-in-cluster)" ;;
*:8888,*)
pass "template $tpl exposes a runtime HTTP port (MCP gateway/direct/in-cluster)" ;;
*:3080,*)
warn "template $tpl is DSH Web only (:3080); MCP execute_command cannot run against it" ;;
*)
warn "template $tpl exposes no known sandbox runtime port" ;;
esac
done
section "Warm pools"
pools="$(k -n "$NS" get sandboxwarmpools \
-o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' 2>/dev/null || true)"
if [ -z "$pools" ]; then
warn "no SandboxWarmPools in $NS; claims will provision cold"
fi
if [ -n "$POOL" ] && ! printf '%s\n' "$pools" | grep -Fxq "$POOL"; then
fail "SandboxWarmPool $POOL not found in $NS"
fi
for pool in $pools; do
[ -n "$POOL" ] && [ "$pool" != "$POOL" ] && continue
replicas="$(k -n "$NS" get sandboxwarmpool "$pool" -o jsonpath='{.spec.replicas}' 2>/dev/null || true)"
selector="$(k -n "$NS" get sandboxwarmpool "$pool" -o jsonpath='{.status.selector}' 2>/dev/null || true)"
if [ -n "$selector" ]; then
ready="$(k -n "$NS" get sandboxes -l "$selector" \
-o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' \
2>/dev/null | grep -c '^True$' || true)"
else
ready="$(k -n "$NS" get sandboxes -l agents.x-k8s.io/warm-pool-sandbox \
-o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' \
2>/dev/null | grep -c '^True$' || true)"
fi
replicas="${replicas:-0}"
info "pool $pool: $ready/$replicas warm spares Ready"
if [ "$replicas" -eq 0 ]; then
warn "pool $pool has spec.replicas=0; signup will cold-provision"
elif [ "$ready" -lt "$replicas" ]; then
warn "pool $pool is below its desired spare count"
else
pass "pool $pool is at capacity"
fi
done
section "Claims"
claims="$(k -n "$NS" get sandboxclaims \
-o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.conditions[?(@.type=="Ready")].status}{"\t"}{.status.conditions[?(@.type=="Ready")].reason}{"\t"}{.status.sandbox.name}{"\n"}{end}' \
2>/dev/null || true)"
if [ -z "$claims" ]; then
warn "no SandboxClaims in $NS"
else
while IFS=$'\t' read -r name ready reason sandbox; do
[ -n "$name" ] || continue
if [ "$ready" != "True" ]; then
warn "claim $name is not Ready (status='${ready:-unknown}', reason='${reason:-unknown}')"
elif [ -z "$sandbox" ]; then
warn "claim $name is Ready but has no bound sandbox"
else
pass "claim $name -> sandbox $sandbox"
fi
done <<< "$claims"
fi
section "Sandboxes"
sandboxes="$(k -n "$NS" get sandboxes \
-o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.operatingMode}{"\t"}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' \
2>/dev/null || true)"
if [ -z "$sandboxes" ]; then
warn "no Sandboxes in $NS"
else
while IFS=$'\t' read -r name mode ready; do
[ -n "$name" ] || continue
info "sandbox $name mode=${mode:-Running} ready=${ready:-unknown}"
done <<< "$sandboxes"
fi
section "Storage and Services"
pvc_count="$(k -n "$NS" get pvc -o name 2>/dev/null | wc -l | tr -d ' ')"
svc_count="$(k -n "$NS" get service -o name 2>/dev/null | wc -l | tr -d ' ')"
info "persistent volume claims: ${pvc_count:-0}"
info "services: ${svc_count:-0}"
[ "${pvc_count:-0}" -gt 0 ] || warn "no PVCs in $NS"
[ "${svc_count:-0}" -gt 0 ] || warn "no Services in $NS"
section "Gateway"
if k -n "$NS" get deployment dsh-gateway -o name >/dev/null 2>&1; then
pass "gateway deployment dsh-gateway exists"
if k -n "$NS" get service dsh-gateway -o name >/dev/null 2>&1; then
endpoints="$(k -n "$NS" get endpoints dsh-gateway \
-o jsonpath='{range .subsets[*].addresses[*]}{.ip}{" "}{end}' 2>/dev/null || true)"
if [ -n "$endpoints" ]; then
pass "gateway endpoints: $endpoints"
else
warn "gateway Service has no ready endpoints"
fi
else
warn "gateway Service dsh-gateway is missing"
fi
else
warn "no dsh-gateway deployment; this is a resource-layer-only install"
fi
section "MCP server"
if [ -n "$MCP_URL" ]; then
if ! command -v curl >/dev/null 2>&1; then
fail "curl is required for --mcp-url"
else
base="${MCP_URL%/}"
base="${base%/mcp}"
for endpoint in healthz readyz; do
body="$(curl -fsS --max-time 5 "$base/$endpoint" 2>/dev/null | head -c 300 || true)"
if [ -n "$body" ]; then
pass "MCP /$endpoint: $body"
else
fail "MCP /$endpoint is unreachable at $base/$endpoint"
fi
done
fi
else
info "no --mcp-url supplied; skipping server health checks"
fi
printf '\n== Summary ==\nPASS=%d WARN=%d FAIL=%d\n' "$PASS" "$WARN" "$FAIL"
if [ "$FAIL" -gt 0 ]; then
exit 1
fi
exit 0

View File

@@ -5,8 +5,23 @@
# --- Main DSH Web image (root Dockerfile) --- # --- Main DSH Web image (root Dockerfile) ---
DEEPSEEK_API_KEY= DEEPSEEK_API_KEY=
DSH_VERSION=dsh-v0.2.1-rc.3 # Optional additional provider keys consumed by DSH. For Docker, pass them with
RELEASE_REVISION=r3 # -e; the Kubernetes examples inject them through the optional
# dsh-provider-keys-extra Secret.
OPENROUTER_API_KEY=
GEMINI_API_KEY=
NVIDIA_API_KEY=
OPENAI_API_KEY=
ANTHROPIC_API_KEY=
GROQ_API_KEY=
XAI_API_KEY=
MISTRAL_API_KEY=
TOGETHER_API_KEY=
CEREBRAS_API_KEY=
AI_GATEWAY_API_KEY=
HF_TOKEN=
DSH_VERSION=dsh-v0.2.1-rc.5
RELEASE_REVISION=
PNPM_VERSION=11.28.5 PNPM_VERSION=11.28.5
# Container port, and the host port published by "make run". # Container port, and the host port published by "make run".
DSH_PORT=3080 DSH_PORT=3080

View File

@@ -28,7 +28,7 @@ body:
id: image-tag id: image-tag
attributes: attributes:
label: Image or release label: Image or release
description: The pinned tag or digest, for example dsh-v0.2.1-rc.3-r3. description: The pinned tag or digest, for example dsh-v0.2.1-rc.5.
- type: input - type: input
id: cluster id: cluster
attributes: attributes:

View File

@@ -60,8 +60,17 @@ jobs:
run: | run: |
set -euo pipefail set -euo pipefail
pair=$(printf '%s' "${{ matrix.platform }}" | tr '/' '-') pair=$(printf '%s' "${{ matrix.platform }}" | tr '/' '-')
echo "PLATFORM_PAIR=$pair" >> "$GITHUB_ENV" # This reusable workflow runs once for the main image and once for the
echo "CACHE_SCOPE=$(printf '%s' "$IMAGE" | tr '/' '-')" >> "$GITHUB_ENV" # gateway in the same release run. Artifact names must include a
# unique token for the image: the repository path is a prefix of the
# gateway path, so a path-derived pattern would match both. Use a
# short hash of the full image name.
image_hash=$(printf '%s' "$IMAGE" | sha256sum | cut -c1-12)
{
echo "PLATFORM_PAIR=$pair"
echo "CACHE_SCOPE=$(printf '%s' "$IMAGE" | tr '/' '-')"
echo "ARTIFACT_PREFIX=$image_hash"
} >> "$GITHUB_ENV"
- name: Set up Docker Buildx - name: Set up Docker Buildx
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1 uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
@@ -101,7 +110,7 @@ jobs:
- name: Upload digest - name: Upload digest
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with: with:
name: digests-${{ env.PLATFORM_PAIR }} name: digests-${{ env.ARTIFACT_PREFIX }}-${{ env.PLATFORM_PAIR }}
path: /tmp/digests/* path: /tmp/digests/*
if-no-files-found: error if-no-files-found: error
retention-days: 1 retention-days: 1
@@ -112,11 +121,18 @@ jobs:
runs-on: ubuntu-latest runs-on: ubuntu-latest
timeout-minutes: 30 timeout-minutes: 30
steps: steps:
- name: Prepare environment
env:
IMAGE: ${{ inputs.image }}
run: |
set -euo pipefail
echo "ARTIFACT_PREFIX=$(printf '%s' "$IMAGE" | sha256sum | cut -c1-12)" >> "$GITHUB_ENV"
- name: Download digests - name: Download digests
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2 uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with: with:
path: /tmp/digests path: /tmp/digests
pattern: digests-* pattern: digests-${{ env.ARTIFACT_PREFIX }}-*
merge-multiple: true merge-multiple: true
- name: Set up Docker Buildx - name: Set up Docker Buildx

View File

@@ -205,6 +205,9 @@ jobs:
needs: [meta, build-main, build-gateway] needs: [meta, build-main, build-gateway]
runs-on: ubuntu-latest runs-on: ubuntu-latest
timeout-minutes: 20 timeout-minutes: 20
permissions:
contents: write
pull-requests: write
steps: steps:
- name: Checkout - name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

View File

@@ -48,6 +48,7 @@ before changing anything; it encodes the conventions the CI enforces.
| examples/dsh-agents-as-a-service/ | The multi-tenant platform example (manifests, gateway, docs, scripts) | | examples/dsh-agents-as-a-service/ | The multi-tenant platform example (manifests, gateway, docs, scripts) |
| examples/minimal-single-tenant/ | The just-one-DSH-Web quickstart | | examples/minimal-single-tenant/ | The just-one-DSH-Web quickstart |
| scripts/ | check-versions.sh, check-examples.sh, check-manifests.py, set-version.sh | | scripts/ | check-versions.sh, check-examples.sh, check-manifests.py, set-version.sh |
| .dsh/skills/ | Project-local DSH skills (auto-discovered); dsh-agent-sandbox-platform covers platform operation, MCP limits and extension recipes |
| .github/workflows/ | ci.yaml, reusable _build-image.yaml, build-*, release.yaml, e2e-kind.yaml, pr-title.yaml, codeql.yml | | .github/workflows/ | ci.yaml, reusable _build-image.yaml, build-*, release.yaml, e2e-kind.yaml, pr-title.yaml, codeql.yml |
## Commands ## Commands
@@ -71,18 +72,21 @@ needs kind plus the agent-sandbox CRDs (see the example README).
| --- | --- | | --- | --- |
| Documentation only | make check; verify relative links | | Documentation only | make check; verify relative links |
| Example manifest | make check; kubectl apply --dry-run=server in CI | | Example manifest | make check; kubectl apply --dry-run=server in CI |
| Optional hardening (gVisor, nodeSelector, NetworkPolicy, provider keys) | make check; server dry-run in CI; manual verification on a capable cluster |
| Gateway code | make test; make lint; add or update tests | | Gateway code | make test; make lint; add or update tests |
| Dockerfile / versions.env | make check; make image (amd64) and a container smoke test | | Dockerfile / versions.env | make check; make image (amd64) and a container smoke test |
| Workflow | actionlint in CI; keep actions SHA-pinned; no hard-coded versions | | Workflow | actionlint in CI; keep actions SHA-pinned; no hard-coded versions |
| New example | make check; all files described in docs/examples.md | | New example | make check; all files described in docs/examples.md |
| Project skill (.dsh/skills/) | make check; frontmatter and size check; shellcheck skill scripts; load it with the skill tool |
## Versioning and releases ## Versioning and releases
- Release identity = DSH_VERSION, optionally with a repository revision suffix - Release identity = DSH_VERSION, optionally with a repository revision suffix
for re-releases at the same DSH ref (first release: dsh-v0.2.1-rc.3-r3). for re-releases at the same DSH ref (current identity:
- The current identity is dsh-v0.2.1-rc.3-r3 (stored in versions.env). The dsh-v0.2.1-rc.5).
dsh-v0.2.1-rc.3 image tag already exists and tags are immutable, which is why - The current identity is dsh-v0.2.1-rc.5 (stored in versions.env). It is the
the first release under this scheme carries -r2. first release at this DSH ref, so it carries no revision suffix; a
repository-only change at the same ref publishes -r2.
- Channels: release tags are immutable; latest follows the newest release; - Channels: release tags are immutable; latest follows the newest release;
main is the development channel for both image packages. main is the development channel for both image packages.
- A broken release is fixed by publishing -r(N+1), never by re-tagging. - A broken release is fixed by publishing -r(N+1), never by re-tagging.
@@ -118,8 +122,11 @@ needs kind plus the agent-sandbox CRDs (see the example README).
can spoof X-Forwarded-User. can spoof X-Forwarded-User.
- The example gateway reads the DSH launch token from pod logs. Treat pod logs - The example gateway reads the DSH launch token from pod logs. Treat pod logs
as sensitive and keep its RBAC namespace-scoped. as sensitive and keep its RBAC namespace-scoped.
- The example shares one DEEPSEEK_API_KEY across sandboxes; do not describe it - The example shares provider keys (dsh-provider-keys plus the optional
as tenant-isolated. dsh-provider-keys-extra) across sandboxes; do not describe it as
tenant-isolated.
- gVisor, nodeSelector and the NetworkPolicies are opt-in and are not active in
the base manifests; do not describe the base example as hardened.
- TLS terminates outside the gateway; set COOKIE_SECURE=true for HTTPS and keep - TLS terminates outside the gateway; set COOKIE_SECURE=true for HTTPS and keep
the base manifest safe for local HTTP. the base manifest safe for local HTTP.
@@ -136,5 +143,6 @@ needs kind plus the agent-sandbox CRDs (see the example README).
- [docs/examples.md](docs/examples.md) - example authoring rules - [docs/examples.md](docs/examples.md) - example authoring rules
- [docs/gitflow.md](docs/gitflow.md) - branches, PRs, required checks - [docs/gitflow.md](docs/gitflow.md) - branches, PRs, required checks
- [docs/versioning.md](docs/versioning.md) - versioning, changelog, releases - [docs/versioning.md](docs/versioning.md) - versioning, changelog, releases
- [.dsh/skills/dsh-agent-sandbox-platform/SKILL.md](.dsh/skills/dsh-agent-sandbox-platform/SKILL.md) - platform operations, MCP limits, extension recipes
- [CONTRIBUTING.md](CONTRIBUTING.md) - development setup - [CONTRIBUTING.md](CONTRIBUTING.md) - development setup
- [SECURITY.md](SECURITY.md) - vulnerability reporting and threat notes - [SECURITY.md](SECURITY.md) - vulnerability reporting and threat notes

View File

@@ -3,24 +3,45 @@
All notable changes to this repository are documented here. All notable changes to this repository are documented here.
Generated by git-cliff from Conventional Commits; do not edit this file by hand. Generated by git-cliff from Conventional Commits; do not edit this file by hand.
## [Unreleased] ## [dsh-v0.2.1-rc.3-r4] - 2026-10-26
### Added
- **skills:** Add agent-sandbox platform skill (#20) (bf24306)
- **example:** Add optional gVisor, nodeSelector, NetworkPolicy and provider keys (#21) (3e9ba38)
### Maintenance
- **changelog:** Refresh for dsh-v0.2.1-rc.3-r3 (#19) (8681412)
- **release:** Dsh-v0.2.1-rc.3-r4 (#22) (e2f1fe5)
## [dsh-v0.2.1-rc.3-r3] - 2026-10-25
### Added ### Added
- **gateway-api:** Expose dsh-gateway externally via Traefik Gateway API (#12) (7d3faf0) - **gateway-api:** Expose dsh-gateway externally via Traefik Gateway API (#12) (7d3faf0)
- **gateway-api:** GitHub SSO via oauth2-proxy in front of dsh-gateway (#13) (3f46279) - **gateway-api:** GitHub SSO via oauth2-proxy in front of dsh-gateway (#13) (3f46279)
- **gateway:** SSO landing auto-connects and offers recreate sandbox (#15) (ed1398d) - **gateway:** SSO landing auto-connects and offers recreate sandbox (#15) (ed1398d)
- **repo:** Restructure around the dsh-harness image and one example per implementation (e6e109d) - **repo:** Repo reorganization (#16) (3b70f17)
### Fixed ### Fixed
- **Dockerfile:** Update runtime dependencies and enable corepack for pnpm (6d97f61) - **Dockerfile:** Update runtime dependencies and enable corepack for pnpm (6d97f61)
- **gateway-api:** Correct oauth2-proxy cookie-secret generation (#14) (73a65d9) - **gateway-api:** Correct oauth2-proxy cookie-secret generation (#14) (73a65d9)
### Maintenance
- Bump the actions group across 1 directory with 3 updates (#17) (6a8fe54)
### Other ### Other
- Dockerfile extra tools (2b47afa) - Dockerfile extra tools (2b47afa)
- Modify gateway.yaml for image policy and timeout values (53e55fc) - Modify gateway.yaml for image policy and timeout values (53e55fc)
- Add commit message prefix for Dependabot updates (6876871)
- Versioning update (#18) (ecba8b9)
- Refactor environment variable loading in build workflow (d627526)
- Refactor environment variable loading in build-gateway.yaml (027f74f)
- Refactor release workflow for environment variable sourcing (958cbe2)
## [v0.0.1] - 2026-10-21 ## [v0.0.1] - 2026-10-21

View File

@@ -20,7 +20,7 @@ ARG NODE_RUNTIME_IMAGE=docker.io/node:24-trixie-slim
FROM $NODE_BUILD_IMAGE AS build FROM $NODE_BUILD_IMAGE AS build
ARG DSH_REPO=https://github.com/deepseek-ai/deepseek-harness.git ARG DSH_REPO=https://github.com/deepseek-ai/deepseek-harness.git
ARG DSH_VERSION=dsh-v0.2.1-rc.3 ARG DSH_VERSION=dsh-v0.2.1-rc.5
ARG PNPM_VERSION=11.28.5 ARG PNPM_VERSION=11.28.5
ENV CI=true \ ENV CI=true \
@@ -57,8 +57,8 @@ RUN cd /opt/dsh \
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
FROM $NODE_RUNTIME_IMAGE AS runtime FROM $NODE_RUNTIME_IMAGE AS runtime
ARG DSH_VERSION=dsh-v0.2.1-rc.3 ARG DSH_VERSION=dsh-v0.2.1-rc.5
ARG RELEASE_REVISION=r3 ARG RELEASE_REVISION=
ARG PNPM_VERSION=11.28.5 ARG PNPM_VERSION=11.28.5
ARG VCS_REF=unknown ARG VCS_REF=unknown

View File

@@ -20,7 +20,7 @@ docker run -d --name dsh-web \
-p 3080:3080 \ -p 3080:3080 \
-v dsh-home:/home/node/.dsh \ -v dsh-home:/home/node/.dsh \
-e DEEPSEEK_API_KEY \ -e DEEPSEEK_API_KEY \
ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.3-r3 ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.5
``` ```
DSH Web is **not usable at a bare `/`**. Without its launch-token cookie it DSH Web is **not usable at a bare `/`**. Without its launch-token cookie it
@@ -45,7 +45,7 @@ docker run -d --name dsh-web -p 18081:3080 \
-e DSH_PUBLIC_URL=http://localhost:18081 \ -e DSH_PUBLIC_URL=http://localhost:18081 \
-v dsh-home:/home/node/.dsh \ -v dsh-home:/home/node/.dsh \
-e DEEPSEEK_API_KEY \ -e DEEPSEEK_API_KEY \
ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.3-r3 ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.5
# then open http://localhost:18081/?token=... # then open http://localhost:18081/?token=...
``` ```
@@ -72,6 +72,13 @@ docker run -d --name dsh-web -p 18081:3080 \
| `DSH_TRUSTED_HOSTS` | empty | Extra trusted `Host`/`Origin` entries, comma/space separated. | | `DSH_TRUSTED_HOSTS` | empty | Extra trusted `Host`/`Origin` entries, comma/space separated. |
| `DSH_HOME` | `/home/node/.dsh` | DSH state directory (mount a volume here). | | `DSH_HOME` | `/home/node/.dsh` | DSH state directory (mount a volume here). |
DSH also resolves the standard per-provider environment names —
`OPENROUTER_API_KEY`, `GEMINI_API_KEY`, `NVIDIA_API_KEY`, `OPENAI_API_KEY`,
`ANTHROPIC_API_KEY`, `GROQ_API_KEY`, `XAI_API_KEY`, `HF_TOKEN`, and others —
through its packaged provider catalog. Inject them the same way; a key saved in
DSH's credential store wins over the environment. See
[provider credentials](docs/main-image.md#provider-credentials).
Full contract, build args, probes, volumes and signals: Full contract, build args, probes, volumes and signals:
[docs/main-image.md](docs/main-image.md). [docs/main-image.md](docs/main-image.md).
@@ -87,21 +94,23 @@ ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway
| Tag | Contents | Meaning | | Tag | Contents | Meaning |
| --- | --- | --- | | --- | --- | --- |
| `dsh-v0.2.1-rc.3-r3` | DSH Web, port `3080` | **Current release identity.** Immutable: once published it never moves. | | `dsh-v0.2.1-rc.5` | DSH Web, port `3080` | **Current release identity.** Immutable: once published it never moves. |
| `latest` | DSH Web | Most recent release; same digest as the current identity. | | `latest` | DSH Web | Most recent release; same digest as the current identity. |
| `main` | DSH Web | Development channel rebuilt from every push to `main`. Not a release; not for production. | | `main` | DSH Web | Development channel rebuilt from every push to `main`. Not a release; not for production. |
| `dsh-v0.2.1-rc.3` | DSH Web | **Legacy tag** published before the release-identity scheme. It is not updated. New deployments use `dsh-v0.2.1-rc.3-r3`. |
Tags published before the release-identity scheme remain on GHCR but are not
updated; new deployments use the current identity.
The example gateway is published under its nested path with the same release The example gateway is published under its nested path with the same release
identity, e.g. identity, e.g.
`ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway:dsh-v0.2.1-rc.3-r3`. `ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway:dsh-v0.2.1-rc.5`.
Release identities, channels and rollback rules are explained in Release identities, channels and rollback rules are explained in
[docs/versioning.md](docs/versioning.md). Production deployments should pin a [docs/versioning.md](docs/versioning.md). Production deployments should pin a
digest: digest:
```bash ```bash
docker buildx imagetools inspect ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.3-r3 docker buildx imagetools inspect ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.5
# image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox@sha256:<digest> # image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox@sha256:<digest>
``` ```
@@ -125,12 +134,23 @@ Self-contained implementations that consume the image. Index and contract:
| Example | What it demonstrates | | Example | What it demonstrates |
| --- | --- | | --- | --- |
| [`examples/dsh-agents-as-a-service/`](examples/dsh-agents-as-a-service/) | Full multi-tenant platform: SandboxTemplate with per-user PVCs, warm pool, one claim per user, suspend-when-idle, wake-on-return, browser gateway with signup + `HttpOnly` sessions + HTTP/WebSocket proxy, Ingress or Gateway API + GitHub SSO exposure. Start with the [walkthrough](examples/dsh-agents-as-a-service/README.md) and the [architecture](examples/dsh-agents-as-a-service/docs/architecture.md). | | [`examples/dsh-agents-as-a-service/`](examples/dsh-agents-as-a-service/) | Full multi-tenant platform: SandboxTemplate with per-user PVCs, warm pool, one claim per user, suspend-when-idle, wake-on-return, browser gateway with signup + `HttpOnly` sessions + HTTP/WebSocket proxy, Ingress or Gateway API + GitHub SSO exposure, and opt-in gVisor/NetworkPolicy/provider-key hardening. Start with the [walkthrough](examples/dsh-agents-as-a-service/README.md) and the [architecture](examples/dsh-agents-as-a-service/docs/architecture.md). |
| [`examples/minimal-single-tenant/`](examples/minimal-single-tenant/) | The smallest useful setup: one `SandboxTemplate` and one `SandboxClaim`, reached through `kubectl port-forward` + the launch-token URL. Manifest: [`manifests/sandbox-template.yaml`](examples/minimal-single-tenant/manifests/sandbox-template.yaml). | | [`examples/minimal-single-tenant/`](examples/minimal-single-tenant/) | The smallest useful setup: one `SandboxTemplate` and one `SandboxClaim`, reached through `kubectl port-forward` + the launch-token URL. Manifest: [`manifests/sandbox-template.yaml`](examples/minimal-single-tenant/manifests/sandbox-template.yaml). |
The gateway is a component of the multi-tenant example and has its own The gateway is a component of the multi-tenant example and has its own
[README](examples/dsh-agents-as-a-service/gateway/README.md) (API, configuration, [README](examples/dsh-agents-as-a-service/gateway/README.md) (API, configuration,
auth/SSO, limitations). auth/SSO, limitations). The example also documents opt-in gVisor,
`nodeSelector`, NetworkPolicy and extra provider-key hardening in
[optional hardening](examples/dsh-agents-as-a-service/docs/optional-hardening.md).
## Agent skills
DSH discovers project skills from `.dsh/skills/<name>/SKILL.md`. The repository
ships [`dsh-agent-sandbox-platform`](.dsh/skills/dsh-agent-sandbox-platform/SKILL.md),
which covers operating and extending the platform, the optional
[agent-sandbox MCP server](https://github.com/kubernetes-sigs/agent-sandbox/blob/main/clients/integrations/mcp-server/README.md),
and a read-only cluster preflight. Runbooks, the MCP reference, the extension
guide and troubleshooting notes live beside `SKILL.md`.
## Repository layout ## Repository layout
@@ -144,6 +164,7 @@ auth/SSO, limitations).
| [`examples/dsh-agents-as-a-service/`](examples/dsh-agents-as-a-service/) | Multi-tenant platform example: `manifests/`, `gateway/`, `docs/`, `scripts/`. | | [`examples/dsh-agents-as-a-service/`](examples/dsh-agents-as-a-service/) | Multi-tenant platform example: `manifests/`, `gateway/`, `docs/`, `scripts/`. |
| [`examples/minimal-single-tenant/`](examples/minimal-single-tenant/) | Single-sandbox quickstart. | | [`examples/minimal-single-tenant/`](examples/minimal-single-tenant/) | Single-sandbox quickstart. |
| [`scripts/`](scripts/) | Repository checks: versions, example layout, manifest YAML, version bump. | | [`scripts/`](scripts/) | Repository checks: versions, example layout, manifest YAML, version bump. |
| [`.dsh/skills/`](.dsh/skills/) | Project-local DSH skills, auto-discovered from the git root. |
| [`.github/`](.github/) | CI, release automation, dependency updates, PR/issue templates. | | [`.github/`](.github/) | CI, release automation, dependency updates, PR/issue templates. |
| [`.env.sample`](.env.sample) | Documented local configuration (copy to `.env`; never commit it). | | [`.env.sample`](.env.sample) | Documented local configuration (copy to `.env`; never commit it). |
@@ -170,10 +191,9 @@ required checks and releases: [docs/gitflow.md](docs/gitflow.md).
The release identity is the upstream DSH ref plus an optional repository The release identity is the upstream DSH ref plus an optional repository
revision: `DSH_VERSION` and `RELEASE_REVISION` in [versions.env](versions.env). revision: `DSH_VERSION` and `RELEASE_REVISION` in [versions.env](versions.env).
The current identity is **`dsh-v0.2.1-rc.3-r3`** — the first release under the The current identity is **`dsh-v0.2.1-rc.5`** — the first release at this
mirrored-DSH scheme at the same upstream DSH ref as the legacy upstream DSH ref, so it carries no repository revision. Tags are immutable; a
`dsh-v0.2.1-rc.3` tag. Tags are immutable; a broken release is fixed by broken release is fixed by publishing `-r(N+1)`, never by re-tagging.
publishing `-r(N+1)`, never by re-tagging.
`CHANGELOG.md` is generated by [git-cliff](https://git-cliff.org) from `CHANGELOG.md` is generated by [git-cliff](https://git-cliff.org) from
Conventional Commits and is never edited by hand. See Conventional Commits and is never edited by hand. See

View File

@@ -25,10 +25,17 @@ disclosure once a fix is available.
- **SSO header trust boundary.** AUTH_PROXY_HEADER makes the gateway trust an - **SSO header trust boundary.** AUTH_PROXY_HEADER makes the gateway trust an
identity header (for example X-Forwarded-User). Enable it only when the identity header (for example X-Forwarded-User). Enable it only when the
gateway Service is reachable exclusively through the authenticating proxy; gateway Service is reachable exclusively through the authenticating proxy;
otherwise any client can spoof the header. otherwise any client can spoof the header. The SSO exposure variant ships a
- **Shared provider key.** The example injects one DEEPSEEK_API_KEY into every NetworkPolicy that restricts gateway access to oauth2-proxy pods; a custom
tenant sandbox (a warm-pool constraint). Treat tenants as mutually trusting exposure needs an equivalent.
until per-tenant credentials or an egress proxy exist. - **Shared provider keys.** The example injects DEEPSEEK_API_KEY and any keys in
the optional dsh-provider-keys-extra Secret into every tenant sandbox (a
warm-pool constraint). Treat tenants as mutually trusting until per-tenant
credentials or an egress proxy exist.
- **Optional isolation is opt-in.** gVisor, nodeSelector, the sandbox/exposure
NetworkPolicies and extra provider keys are opt-in. Without a
NetworkPolicy-capable CNI or a gVisor runtime, the base example is not
tenant-isolated; see examples/dsh-agents-as-a-service/docs/optional-hardening.md.
- **TLS is external.** Terminate TLS at an Ingress or Gateway and set - **TLS is external.** Terminate TLS at an Ingress or Gateway and set
COOKIE_SECURE=true. Plain HTTP outside localhost exposes session cookies. COOKIE_SECURE=true. Plain HTTP outside localhost exposes session cookies.
- **No secrets in the repository.** The repository only ships empty Secret - **No secrets in the repository.** The repository only ships empty Secret

View File

@@ -15,7 +15,16 @@ gets a DSH Web container running before anything else.
| [main-image.md](main-image.md) | The primary artifact: build args, runtime environment, entrypoint, probes, volumes/ports/signals, non-root contract, image names, tags and digests. | | [main-image.md](main-image.md) | The primary artifact: build args, runtime environment, entrypoint, probes, volumes/ports/signals, non-root contract, image names, tags and digests. |
| [examples.md](examples.md) | The example contract: required layout, manifest rules, exposure variants, components and image naming, scripts, docs, and how to add a new example. | | [examples.md](examples.md) | The example contract: required layout, manifest rules, exposure variants, components and image naming, scripts, docs, and how to add a new example. |
| [gitflow.md](gitflow.md) | Branches, protected `main`, squash merges, Conventional PR titles, required CI checks, release branches, hotfixes and the one-time GitHub settings checklist. | | [gitflow.md](gitflow.md) | Branches, protected `main`, squash merges, Conventional PR titles, required CI checks, release branches, hotfixes and the one-time GitHub settings checklist. |
| [versioning.md](versioning.md) | Release identity (`DSH ref` + optional `-rN`), why the current identity is `dsh-v0.2.1-rc.3-r3`, channels, compatibility matrix, bumping pins, the release checklist, rollback, git-cliff and digest pinning. | | [versioning.md](versioning.md) | Release identity (`DSH ref` + optional `-rN`), the current identity `dsh-v0.2.1-rc.5`, why identities are immutable, channels, compatibility matrix, bumping pins, the release checklist, rollback, git-cliff and digest pinning. |
## Project skills
DSH discovers project skills from `<git root>/.dsh/skills/<name>/SKILL.md`
(the nearest `.git` ancestor). This repository ships one:
| Skill | Read it for |
| --- | --- |
| [dsh-agent-sandbox-platform](../.dsh/skills/dsh-agent-sandbox-platform/SKILL.md) | Operating and extending the agents-as-a-service platform, agent-sandbox MCP limits and deployment, and a read-only preflight. |
## Example documentation ## Example documentation

View File

@@ -18,6 +18,9 @@ examples/<name>/
| # ordered steps, troubleshooting, cleanup | # ordered steps, troubleshooting, cleanup
+-- manifests/ # required +-- manifests/ # required
| +-- NN-*.yaml # numbered top-level manifests, plain kubectl | +-- NN-*.yaml # numbered top-level manifests, plain kubectl
| +-- optional/ # optional opt-in manifests, applied by path
| +-- README.md # required when optional/ exists
| +-- <feature>/NN-*.yaml # e.g. network-policy/
| +-- exposure/ # optional exposure variants | +-- exposure/ # optional exposure variants
| +-- README.md # required when exposure/ exists | +-- README.md # required when exposure/ exists
| +-- <variant>/NN-*.yaml # e.g. ingress/, gateway-api-sso/ | +-- <variant>/NN-*.yaml # e.g. ingress/, gateway-api-sso/
@@ -41,6 +44,8 @@ examples/<name>/
- a `*.yaml` file sits at the example root instead of under `manifests/`; - a `*.yaml` file sits at the example root instead of under `manifests/`;
- `manifests/exposure/` exists without a `manifests/exposure/README.md` describing - `manifests/exposure/` exists without a `manifests/exposure/README.md` describing
the variants; the variants;
- `manifests/optional/` exists without a `manifests/optional/README.md`
describing the opt-in manifests;
- a subdirectory with a `Dockerfile` has no `README.md`; - a subdirectory with a `Dockerfile` has no `README.md`;
- a file in `scripts/` is not executable. - a file in `scripts/` is not executable.
@@ -68,7 +73,12 @@ make check
command; real values are generated per install. command; real values are generated per install.
- **Cluster-specific state stays out.** Node affinity, replica counts tuned for a - **Cluster-specific state stays out.** Node affinity, replica counts tuned for a
cluster, and similar fields belong in a local patch, not in the shipped cluster, and similar fields belong in a local patch, not in the shipped
manifest. manifest. Optional scheduler/runtime hints may ship **commented out** with
placeholder labels; they must not be active.
- **Optional manifests are opt-in.** Put them under
`manifests/optional/<feature>/`, document them in
`manifests/optional/README.md`, and apply them by path. The top-level
`kubectl apply -f manifests/` is non-recursive and does not apply them.
- **Placeholders are explicit.** Hostnames, GitHub orgs and TLS material use - **Placeholders are explicit.** Hostnames, GitHub orgs and TLS material use
values like `dsh.example.com` / `your-github-org` with a TODO comment. values like `dsh.example.com` / `your-github-org` with a TODO comment.

View File

@@ -31,11 +31,11 @@ To reproduce the equivalent by hand:
```bash ```bash
docker build \ docker build \
--build-arg DSH_VERSION=dsh-v0.2.1-rc.3 \ --build-arg DSH_VERSION=dsh-v0.2.1-rc.5 \
--build-arg RELEASE_REVISION=r2 \ --build-arg RELEASE_REVISION= \
--build-arg PNPM_VERSION=11.28.5 \ --build-arg PNPM_VERSION=11.28.5 \
--build-arg VCS_REF="$(git rev-parse --short HEAD)" \ --build-arg VCS_REF="$(git rev-parse --short HEAD)" \
-t ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.3-r3 . -t ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.5 .
``` ```
### Build args ### Build args
@@ -80,6 +80,28 @@ The entrypoint is `tini -- /usr/local/bin/dsh-entrypoint`:
The image also sets `NODE_ENV=production`, `HOME=/home/node`, and a `PATH` that The image also sets `NODE_ENV=production`, `HOME=/home/node`, and a `PATH` that
includes `/app/node_modules/.bin`. includes `/app/node_modules/.bin`.
### Provider credentials
DSH resolves provider API keys from the environment when no stored credential
exists. The packaged provider catalog maps each provider to its standard
environment name. Commonly used names:
`DEEPSEEK_API_KEY`, `OPENROUTER_API_KEY`, `GEMINI_API_KEY`, `NVIDIA_API_KEY`,
`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GROQ_API_KEY`, `XAI_API_KEY`,
`MISTRAL_API_KEY`, `TOGETHER_API_KEY`, `CEREBRAS_API_KEY`,
`AI_GATEWAY_API_KEY`, `HF_TOKEN`.
Use `GEMINI_API_KEY` for Google Gemini (not `GOOGLE_API_KEY`), `HF_TOKEN` for
Hugging Face, and `AI_GATEWAY_API_KEY` for the Vercel AI Gateway. Inject keys
with `-e`/`--env-file` in Docker or a Secret + `envFrom` in Kubernetes; never
bake them into the image.
A key saved through DSH's credential store (`$DSH_HOME/.credentials.yaml`) wins
over the environment. The
[agents-as-a-service example](../examples/dsh-agents-as-a-service/docs/optional-hardening.md)
documents the optional `dsh-provider-keys-extra` Secret and the pod-refresh
procedure.
### Ports, volumes and signals ### Ports, volumes and signals
| Item | Value | Notes | | Item | Value | Notes |
@@ -142,17 +164,19 @@ GitHub Container Registry:
| Tag | Meaning | | Tag | Meaning |
| --- | --- | | --- | --- |
| `dsh-v0.2.1-rc.3-r3` | Current release identity. **Immutable**: once published it never moves. | | `dsh-v0.2.1-rc.5` | Current release identity. **Immutable**: once published it never moves. |
| `latest` | Most recent release; points at the same digest as the current identity. | | `latest` | Most recent release; points at the same digest as the current identity. |
| `main` | Development channel rebuilt from every push to `main`; not a release, not for production. | | `main` | Development channel rebuilt from every push to `main`; not a release, not for production. |
| `dsh-v0.2.1-rc.3` | Legacy tag published before the release-identity scheme existed. Do not use it for new deployments; see the migration table in the [root README](../README.md). |
Tags published before the release-identity scheme remain published but are not
updated; new deployments use the current identity.
Release identities, channels and rollback rules are defined in Release identities, channels and rollback rules are defined in
[versioning](versioning.md). Tags are immutable, so production deployments should [versioning](versioning.md). Tags are immutable, so production deployments should
pin a digest: pin a digest:
```bash ```bash
docker buildx imagetools inspect ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.3-r3 docker buildx imagetools inspect ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.5
# ... pin the digest in your deployment manifests: # ... pin the digest in your deployment manifests:
# image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox@sha256:<digest> # image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox@sha256:<digest>
``` ```
@@ -160,7 +184,7 @@ docker buildx imagetools inspect ghcr.io/sredevopsorg/deepseek-kubernetes-sandbo
## Verifying an image ## Verifying an image
```bash ```bash
docker inspect ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.3-r3 \ docker inspect ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.5 \
--format '{{json .Config.Labels}}' --format '{{json .Config.Labels}}'
# org.opencontainers.image.title / .version / .source # org.opencontainers.image.title / .version / .source
``` ```

View File

@@ -15,46 +15,47 @@ release identity = DSH_VERSION + optional "-" + RELEASE_REVISION
| Piece | Meaning | | Piece | Meaning |
| --- | --- | | --- | --- |
| `DSH_VERSION` | The upstream DeepSeek Harness git ref being packaged, e.g. `dsh-v0.2.1-rc.3`. It mirrors upstream's own tag, which is what "mirrored-DSH scheme" means. | | `DSH_VERSION` | The upstream DeepSeek Harness git ref being packaged, e.g. `dsh-v0.2.1-rc.5`. It mirrors upstream's own tag, which is what "mirrored-DSH scheme" means. |
| `RELEASE_REVISION` | An optional repository revision at that DSH ref: empty for the first release, `rN` (`N >= 2`) for a repository-only re-release. | | `RELEASE_REVISION` | An optional repository revision at that DSH ref: empty for the first release, `rN` (`N >= 2`) for a repository-only re-release. |
Examples: Examples:
- `DSH_VERSION=dsh-v0.2.1-rc.3`, `RELEASE_REVISION=` → identity `dsh-v0.2.1-rc.3` - `DSH_VERSION=dsh-v0.2.1-rc.5`, `RELEASE_REVISION=` → identity `dsh-v0.2.1-rc.5`
(first release at that upstream ref). (first release at that upstream ref).
- `DSH_VERSION=dsh-v0.2.1-rc.3`, `RELEASE_REVISION=r2` → identity - `DSH_VERSION=<ref>`, `RELEASE_REVISION=rN` → identity `<ref>-rN` (for example
`dsh-v0.2.1-rc.3-r3` (documentation, CI or gateway changes on top of the same documentation, CI or gateway changes on top of an already-released ref; no DSH
upstream DSH; no DSH upgrade). upgrade). Use `r2` for the first re-release: revision 1 is the plain tag.
A DSH upgrade produces a new base identity (for example the next upstream ref A DSH upgrade produces a new base identity (for example the next upstream ref
with an empty revision); a repository-only change at the same DSH ref increments with an empty revision); a repository-only change at the same DSH ref increments
`rN`. `rN`.
## Why the current identity is `dsh-v0.2.1-rc.3-r3` ## Why release identities are immutable
The image tag `dsh-v0.2.1-rc.3` was already published to GHCR for an earlier **Container tags are immutable** in this project: re-pushing a tag would silently
repository state. **Container tags are immutable** in this project: re-pushing a change what an existing deployment runs. A DSH upgrade therefore produces a new
tag would silently change what an existing deployment runs. The mirrored-DSH base identity with an empty revision; a repository-only change at an
scheme therefore starts its first identity at `-r2`: the plain tag is already-released ref publishes `-rN`, where `r2` is the first re-release (the
conceptually revision 1 and is taken, so the first publishable repository plain tag is conceptually revision 1 and is taken).
revision at the same upstream ref is `r2`.
The identity is stored in [versions.env](../versions.env) and consumed everywhere: The current identity is `dsh-v0.2.1-rc.5`, the first release at this upstream
the Dockerfile label, the example manifests, the Makefile and the release ref, so it carries no revision suffix. The identity is stored in
workflow. The legacy `dsh-v0.2.1-rc.3` tag remains published but is not updated; [versions.env](../versions.env) and consumed everywhere: the Dockerfile label,
new deployments use `dsh-v0.2.1-rc.3-r3`. the example manifests, the Makefile and the release workflow. Tags published
before the release-identity scheme remain published but are not updated; new
deployments use the current identity.
The gateway image moved from the repository-level `:gateway` tag to a nested The gateway image moved from the repository-level `:gateway` tag to a nested
package (`<repository>/dsh-agents-as-a-service/gateway`). For exactly one package (`<repository>/dsh-agents-as-a-service/gateway`). The release workflow
release the release workflow also republishes the legacy also republishes the legacy `<repository>:gateway` tag as a deprecated
`<repository>:gateway` tag as a deprecated alias; it is removed in the next compatibility alias; it may be removed in a later release, so deployments
release, so deployments should pin the nested package. should pin the nested package.
## Channels ## Channels
| Channel | Image tag | Meaning | | Channel | Image tag | Meaning |
| --- | --- | --- | | --- | --- | --- |
| Release | `dsh-v0.2.1-rc.3-r3` | The current immutable release identity. What production should pin (preferably by digest). | | Release | `dsh-v0.2.1-rc.5` | The current immutable release identity. What production should pin (preferably by digest). |
| `latest` | `latest` | Alias for the most recent release; moves only when a new release is published. | | `latest` | `latest` | Alias for the most recent release; moves only when a new release is published. |
| `main` | `main` | Development channel, rebuilt from every push to `main`. Never treat it as a release; it exists so `main` can be tested. | | `main` | `main` | Development channel, rebuilt from every push to `main`. Never treat it as a release; it exists so `main` can be tested. |
@@ -68,10 +69,10 @@ Values from [versions.env](../versions.env) at the current identity:
| Component | Pin | Source | | Component | Pin | Source |
| --- | --- | --- | | --- | --- | --- |
| Upstream DSH source | `DSH_REPO` | [versions.env](../versions.env) | | Upstream DSH source | `DSH_REPO` | [versions.env](../versions.env) |
| DSH ref | `dsh-v0.2.1-rc.3` | `DSH_VERSION` | | DSH ref | `dsh-v0.2.1-rc.5` | `DSH_VERSION` |
| Repository revision | `r2` | `RELEASE_REVISION` | | Repository revision | `(empty)` | `RELEASE_REVISION` |
| pnpm | `11.28.5` | `PNPM_VERSION` | | pnpm | `11.28.5` | `PNPM_VERSION` |
| agent-sandbox (CRDs) | `v1.0.7` | `AGENT_SANDBOX_VERSION`, used by the examples and CI dry-run | | agent-sandbox (CRDs) | `v1.0.8` | `AGENT_SANDBOX_VERSION`, used by the examples and CI dry-run |
| Kubernetes Gateway API | `v1.2.1` | `GATEWAY_API_VERSION`, used by the SSO exposure variant | | Kubernetes Gateway API | `v1.2.1` | `GATEWAY_API_VERSION`, used by the SSO exposure variant |
| oauth2-proxy | `quay.io/oauth2-proxy/oauth2-proxy:v7.15.5` | `OAUTH2_PROXY_IMAGE` | | oauth2-proxy | `quay.io/oauth2-proxy/oauth2-proxy:v7.15.5` | `OAUTH2_PROXY_IMAGE` |
| Main image | `ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox` | `GHCR_IMAGE` | | Main image | `ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox` | `GHCR_IMAGE` |
@@ -153,7 +154,7 @@ Release identities are mutable only in theory; deployments should still pin the
resolved digest for reproducibility and supply-chain safety: resolved digest for reproducibility and supply-chain safety:
```bash ```bash
docker buildx imagetools inspect ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.3-r3 docker buildx imagetools inspect ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.5
# then, in the deployment: # then, in the deployment:
# image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox@sha256:<digest> # image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox@sha256:<digest>
``` ```

View File

@@ -10,6 +10,10 @@ reference material, not libraries.
| [`examples/dsh-agents-as-a-service/`](dsh-agents-as-a-service/) | The full multi-tenant platform pattern: one `SandboxTemplate` with per-user PVCs, a warm pool, one ~10-line `SandboxClaim` per user, suspend-when-idle / wake-on-return, and a browser gateway with signup, `HttpOnly` sessions, an HTTP + WebSocket proxy and two exposure variants (Ingress, or Gateway API + GitHub SSO). | agent-sandbox with extensions; kind for the e2e script. | | [`examples/dsh-agents-as-a-service/`](dsh-agents-as-a-service/) | The full multi-tenant platform pattern: one `SandboxTemplate` with per-user PVCs, a warm pool, one ~10-line `SandboxClaim` per user, suspend-when-idle / wake-on-return, and a browser gateway with signup, `HttpOnly` sessions, an HTTP + WebSocket proxy and two exposure variants (Ingress, or Gateway API + GitHub SSO). | agent-sandbox with extensions; kind for the e2e script. |
| [`examples/minimal-single-tenant/`](minimal-single-tenant/) | The smallest useful setup: a single `SandboxTemplate` and one `SandboxClaim`, no gateway — reach DSH Web with a launch-token URL through `kubectl port-forward`. | agent-sandbox with extensions. | | [`examples/minimal-single-tenant/`](minimal-single-tenant/) | The smallest useful setup: a single `SandboxTemplate` and one `SandboxClaim`, no gateway — reach DSH Web with a launch-token URL through `kubectl port-forward`. | agent-sandbox with extensions. |
The multi-tenant example also documents opt-in gVisor, `nodeSelector`,
NetworkPolicy and extra provider-key hardening in
[optional hardening](dsh-agents-as-a-service/docs/optional-hardening.md).
## The example contract (short version) ## The example contract (short version)
Full rules: [docs/examples.md](../docs/examples.md). `scripts/check-examples.sh` Full rules: [docs/examples.md](../docs/examples.md). `scripts/check-examples.sh`
@@ -19,6 +23,8 @@ enforces the layout, and `make check` runs it.
examples/<name>/ examples/<name>/
+-- README.md # walkthrough: prerequisites, steps, troubleshooting, cleanup +-- README.md # walkthrough: prerequisites, steps, troubleshooting, cleanup
+-- manifests/ # NN-*.yaml, applied with plain kubectl +-- manifests/ # NN-*.yaml, applied with plain kubectl
| +-- optional/<feature>/ # optional opt-in manifests...
| +-- optional/README.md # ...required whenever optional/ exists
| +-- exposure/<variant>/ # optional per-variant manifests... | +-- exposure/<variant>/ # optional per-variant manifests...
| +-- exposure/README.md # ...required whenever exposure/ exists | +-- exposure/README.md # ...required whenever exposure/ exists
+-- scripts/ # optional executable, idempotent, self-cleaning *.sh +-- scripts/ # optional executable, idempotent, self-cleaning *.sh
@@ -44,4 +50,6 @@ kubectl apply -f examples/dsh-agents-as-a-service/manifests/
examples/dsh-agents-as-a-service/scripts/e2e-kind.sh examples/dsh-agents-as-a-service/scripts/e2e-kind.sh
``` ```
Each example README has the exact prerequisites, exposure options and teardown. `kubectl apply -f manifests/` is non-recursive: `optional/` and `exposure/`
subdirectories are applied explicitly. Each example README has the exact
prerequisites, optional hardening, exposure options and teardown.

View File

@@ -32,10 +32,14 @@ manifests/
+-- 30-claim-alice.yaml # one user +-- 30-claim-alice.yaml # one user
+-- 40-claim-rejected.yaml # negative demo: env injection rejected +-- 40-claim-rejected.yaml # negative demo: env injection rejected
+-- 50-gateway.yaml # Deployment/RBAC/Service for the gateway +-- 50-gateway.yaml # Deployment/RBAC/Service for the gateway
+-- optional/
| +-- README.md # opt-in manifests, applied by path
| +-- network-policy/10-sandbox-networkpolicy.yaml
+-- exposure/ +-- exposure/
+-- README.md # the two exposure variants, when to use which +-- README.md # the two exposure variants, when to use which
+-- ingress/60-ingress.yaml +-- ingress/60-ingress.yaml
+-- gateway-api-sso/70-gateway.yaml ... 74-tls-secret.yaml.example +-- ingress/61-networkpolicy.yaml
+-- gateway-api-sso/70-gateway.yaml ... 75-networkpolicies.yaml
``` ```
What it demonstrates, feature by feature: What it demonstrates, feature by feature:
@@ -57,8 +61,10 @@ What it demonstrates, feature by feature:
`kubernetes.io/arch: amd64` nodeSelector. `kubernetes.io/arch: amd64` nodeSelector.
- **Single port**: DSH Web serves everything on `3080`; the gateway proxies to one - **Single port**: DSH Web serves everything on `3080`; the gateway proxies to one
upstream instead of splitting `/v1/*` (API) from the dashboard. upstream instead of splitting `/v1/*` (API) from the dashboard.
- **One shared secret**: only `DEEPSEEK_API_KEY` is injected (via `envFrom`); there - **Shared provider keys**: `DEEPSEEK_API_KEY` plus any keys in the optional
is no dashboard basic-auth, session-cookie secret, or API-server key. `dsh-provider-keys-extra` Secret are injected via `envFrom` into every
sandbox. There is no per-tenant key, dashboard basic-auth, session-cookie
secret, or API-server key.
- **Launch-token handshake**: DSH gates `/` behind a token it prints only in its - **Launch-token handshake**: DSH gates `/` behind a token it prints only in its
logs. The gateway reads it, performs the handshake, and keeps the resulting logs. The gateway reads it, performs the handshake, and keeps the resulting
auth cookie server-side; the browser authenticates to the gateway with its own auth cookie server-side; the browser authenticates to the gateway with its own
@@ -77,7 +83,7 @@ What it demonstrates, feature by feature:
- agent-sandbox with extensions installed: - agent-sandbox with extensions installed:
```bash ```bash
VERSION=v1.0.7 # AGENT_SANDBOX_VERSION in versions.env VERSION=v1.0.8 # AGENT_SANDBOX_VERSION in versions.env
kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/download/$VERSION/sandbox-with-extensions.yaml kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/download/$VERSION/sandbox-with-extensions.yaml
kubectl -n agent-sandbox-system rollout status deployment --timeout=180s kubectl -n agent-sandbox-system rollout status deployment --timeout=180s
``` ```
@@ -93,14 +99,39 @@ What it demonstrates, feature by feature:
## 1. Install the platform resources ## 1. Install the platform resources
First the namespace, then the shared provider key. The key reaches **every** First the namespace, then the shared provider keys. The keys reach **every**
tenant sandbox (a warm-pool constraint: pod env is stamped before any user tenant sandbox (a warm-pool constraint: pod env is stamped before any user
exists), so it is **never committed** — generate a fresh value per install: exists), so they are **never committed** — generate fresh values per install.
`00-prereqs.yaml` already creates the empty `dsh-provider-keys` Secret, so use
an upsert:
```bash ```bash
kubectl apply -f manifests/00-prereqs.yaml kubectl apply -f manifests/00-prereqs.yaml
kubectl -n dsh-demo create secret generic dsh-provider-keys \ kubectl -n dsh-demo create secret generic dsh-provider-keys \
--from-literal=DEEPSEEK_API_KEY="sk-..." --from-literal=DEEPSEEK_API_KEY="sk-..." \
--dry-run=client -o yaml | kubectl apply -f -
```
Optional: additional providers (OpenRouter, Gemini, NVIDIA, OpenAI, Anthropic,
...) go in a second Secret. It is referenced with `optional: true`, so the
sandboxes start without it, and every key becomes an env var:
```bash
kubectl -n dsh-demo create secret generic dsh-provider-keys-extra \
--from-literal=OPENROUTER_API_KEY="sk-or-..." \
--from-literal=GEMINI_API_KEY="..." \
--from-literal=NVIDIA_API_KEY="nvapi-..." \
--dry-run=client -o yaml | kubectl apply -f -
```
Create both Secrets **before** the template/warm pool. A later Secret change
does not update running pods: delete the warm-pool spares and recreate claimed
pods to pick up new keys. Keys are shared by all tenants; a key saved in DSH's
credential store wins over env. See
[optional hardening](docs/optional-hardening.md) for the full env-name list and
the refresh procedure.
```bash
kubectl apply -f manifests/10-sandboxtemplate.yaml -f manifests/20-sandboxwarmpool.yaml kubectl apply -f manifests/10-sandboxtemplate.yaml -f manifests/20-sandboxwarmpool.yaml
``` ```
@@ -213,7 +244,7 @@ kubectl delete -f manifests/40-claim-rejected.yaml
```bash ```bash
# Published image (on kind, load it into the cluster; on a real cluster the # Published image (on kind, load it into the cluster; on a real cluster the
# kubelet pulls it from GHCR): # kubelet pulls it from GHCR):
GW=ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway:dsh-v0.2.1-rc.3-r3 GW=ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway:dsh-v0.2.1-rc.5
docker pull "$GW" docker pull "$GW"
kind load docker-image "$GW" --name <kind-cluster> kind load docker-image "$GW" --name <kind-cluster>
@@ -253,6 +284,12 @@ kubectl -n dsh-demo port-forward svc/dsh-gateway 18080:8080
> apply does not update them: patch each sandbox's > apply does not update them: patch each sandbox's
> `spec.podTemplate.spec.securityContext.fsGroupChangePolicy` and > `spec.podTemplate.spec.securityContext.fsGroupChangePolicy` and
> `chmod 600` its `.credentials.yaml`, or recreate it. > `chmod 600` its `.credentials.yaml`, or recreate it.
>
> The optional gVisor/`nodeSelector` fields are pod-spec changes too: existing
> Sandboxes snapshot them, and warm-pool spares are not replaced by a template
> edit. See [optional hardening](docs/optional-hardening.md) for the patch and
> recreation commands. A provider-key Secret change likewise needs pod
> recreation to take effect.
### Browser flow ### Browser flow
@@ -388,6 +425,13 @@ not a stack. Details, commands and the switch procedure are in the
> `kubectl apply -R`: it would apply both variants and create two paths to the > `kubectl apply -R`: it would apply both variants and create two paths to the
> same gateway, which is unsafe once SSO is enabled. > same gateway, which is unsafe once SSO is enabled.
Both variants also ship NetworkPolicies: `ingress/61-networkpolicy.yaml` limits
gateway ingress to Traefik pods, and `gateway-api-sso/75-networkpolicies.yaml`
limits the gateway to oauth2-proxy and oauth2-proxy egress to DNS, public HTTPS
and the gateway. They are part of the variant; see the
[exposure README](manifests/exposure/README.md) and
[optional hardening](docs/optional-hardening.md).
### Option A — plain Ingress (gateway's own session auth) ### Option A — plain Ingress (gateway's own session auth)
```bash ```bash
@@ -404,6 +448,10 @@ kubectl apply -f manifests/exposure/ingress/
(The base manifest keeps `false` so the plain-HTTP port-forward demo works.) (The base manifest keeps `false` so the plain-HTTP port-forward demo works.)
3. Verify: `curl -sI https://<host>/healthz` returns `200`; sign up in a browser 3. Verify: `curl -sI https://<host>/healthz` returns `200`; sign up in a browser
at `https://<host>/`. at `https://<host>/`.
4. `manifests/exposure/ingress/61-networkpolicy.yaml` is part of this variant:
it allows only Traefik pods to reach the gateway on `8080`. Adjust its
selector if your ingress controller uses a different pod label or
`hostNetwork`; if kubelet probes fail, see the hardening doc.
Keep the router free of buffering/compression middlewares: they break WebSocket Keep the router free of buffering/compression middlewares: they break WebSocket
upgrades and streaming. upgrades and streaming.
@@ -472,13 +520,16 @@ can be cleared at `/oauth2/sign_out`.
> **SSO trust boundary — read this before enabling it.** After SSO is on, the > **SSO trust boundary — read this before enabling it.** After SSO is on, the
> `dsh-gateway` Service must be reachable **only** through oauth2-proxy (it is > `dsh-gateway` Service must be reachable **only** through oauth2-proxy (it is
> ClusterIP). Anyone who can reach `dsh-gateway:8080` directly can send > ClusterIP). The variant's `75-networkpolicies.yaml` restricts pod traffic
> `X-Forwarded-User: <victim>` and be treated as that user. Never apply the > accordingly: the gateway accepts traffic only from oauth2-proxy pods,
> Ingress variant (or any other route to the Service) while SSO is enabled, and > oauth2-proxy only from the Gateway implementation, and oauth2-proxy egress is
> do not expose the gateway Service with `kubectl port-forward` on a shared > limited to DNS, public HTTPS and the gateway. Anyone who can reach
> machine while `AUTH_PROXY_HEADER` is set. To turn SSO off, remove the variable > `dsh-gateway:8080` directly can still send `X-Forwarded-User: <victim>` and be
> (`kubectl -n dsh-demo set env deploy/dsh-gateway AUTH_PROXY_HEADER-`) and > treated as that user, so never add a second path (the Ingress variant, a
> delete the SSO variant; the gateway falls back to bearer-token/session auth. > shared-host port-forward, a permissive policy). To turn SSO off, remove the
> variable (`kubectl -n dsh-demo set env deploy/dsh-gateway AUTH_PROXY_HEADER-`)
> and delete the SSO variant; the gateway falls back to bearer-token/session
> auth.
### Switching variants ### Switching variants
@@ -488,6 +539,28 @@ Delete the variant you had before applying the other one
`kubectl -n dsh-demo get ingress,gateway,httproute` that exactly one path to the `kubectl -n dsh-demo get ingress,gateway,httproute` that exactly one path to the
gateway remains. gateway remains.
## Optional hardening
The base walkthrough is deliberately portable. Four opt-in isolation layers are
documented in [optional hardening](docs/optional-hardening.md):
- **gVisor** — uncomment `runtimeClassName: gvisor` before applying, or patch an
existing install. Requires a `gvisor` RuntimeClass and capable nodes.
`kubectl port-forward` does not work under gVisor; the gateway path does.
- **nodeSelector** — uncomment and replace the placeholder node label to pin
sandboxes and the gateway to a dedicated pool; add a toleration for tainted
pools.
- **Sandbox NetworkPolicy** —
`kubectl apply -f manifests/optional/network-policy/` allows only
gateway→sandbox on `3080` and sandbox→DNS/public `443`.
- **Provider keys** — create `dsh-provider-keys-extra` as shown in step 1.
- **Failed-Pod recovery** — uncomment `failedPodPolicy: Recreate` so the
controller replaces a `Failed` sandbox Pod (node eviction) instead of parking
the sandbox at `Finished=True`. Requires agent-sandbox v1.0.8+.
Apply each independently and re-run the walkthrough. `kubectl apply -f manifests/`
is non-recursive, so the optional manifest is not applied by the base step.
## Troubleshooting ## Troubleshooting
| Symptom | Cause / fix | | Symptom | Cause / fix |
@@ -511,6 +584,12 @@ gateway remains.
| SSO: oauth2-proxy `cookie_secret must be 16, 24, or 32 bytes ... but is 44 bytes` | The cookie secret was generated as standard base64. Use URL-safe unpadded base64 (see `73-oauth2-proxy-secret.yaml.example`) | | SSO: oauth2-proxy `cookie_secret must be 16, 24, or 32 bytes ... but is 44 bytes` | The cookie secret was generated as standard base64. Use URL-safe unpadded base64 (see `73-oauth2-proxy-secret.yaml.example`) |
| SSO: redirect loop (`ERR_TOO_MANY_REDIRECTS`) | oauth2-proxy `--redirect-url` / callback does not match the public FQDN, or TLS is not terminating at the Gateway listener; fix the OAuth App URLs and the manifest placeholders | | SSO: redirect loop (`ERR_TOO_MANY_REDIRECTS`) | oauth2-proxy `--redirect-url` / callback does not match the public FQDN, or TLS is not terminating at the Gateway listener; fix the OAuth App URLs and the manifest placeholders |
| SSO: DSH rejects the public host behind the proxy | Add the public hostname to `DSH_TRUSTED_HOSTS` in `manifests/10-sandboxtemplate.yaml`; the gateway forwards the browser's `Host` | | SSO: DSH rejects the public host behind the proxy | Add the public hostname to `DSH_TRUSTED_HOSTS` in `manifests/10-sandboxtemplate.yaml`; the gateway forwards the browser's `Host` |
| Pods `Pending` after setting `nodeSelector` | No node matches the label, the pool is full, or a tainted pool needs a toleration; inspect `kubectl describe pod` and the node labels |
| gVisor: `kubectl port-forward` fails or the connection is refused | Port-forward is incompatible with gVisor; reach DSH through the gateway (or a reachable Service/Sandbox Router) |
| Sandbox cannot reach a model API after applying the optional NetworkPolicy | The policy allows DNS and public TCP `443` only; a proxy, custom base URL, port `80` or non-443 endpoint needs an explicit egress rule |
| Gateway cannot reach a sandbox after applying an exposure NetworkPolicy | Check that the CNI enforces NetworkPolicy and that the Traefik/oauth2-proxy pod labels match; kubelet probes may also need the node CIDR |
| New provider key not visible in a sandbox | Pod env is stamped at creation: delete the warm-pool spares and recreate claimed pods. A key saved in DSH's credential store also wins over env |
| Sandbox reports `Finished=True` and every gateway request fails | The sandbox Pod reached phase `Failed` (node eviction, spot reclaim) and the default `Retain` keeps it. Delete the Pod so the controller recreates it, or set `failedPodPolicy: Recreate` (agent-sandbox v1.0.8+; see optional hardening) |
## Limitations ## Limitations
@@ -522,13 +601,22 @@ Deliberately **not** production code:
- **Log scraping.** The launch token is recovered from pod logs. Log rotation, a - **Log scraping.** The launch token is recovered from pod logs. Log rotation, a
restarted pod, or a changed DSH log format can break the handshake until the restarted pod, or a changed DSH log format can break the handshake until the
next `401` refresh. next `401` refresh.
- **Shared provider key.** `DEEPSEEK_API_KEY` is shared by all tenants via the - **Shared provider keys.** `DEEPSEEK_API_KEY` and any keys in the optional
template. Real multi-tenancy needs per-tenant credentials or an egress proxy. `dsh-provider-keys-extra` Secret are shared by all tenants via the template.
Tenants can override a key through DSH's stored credential file on their PVC,
but the platform does not manage that. Real multi-tenancy needs per-tenant
credentials or an egress proxy.
- **No TLS in the gateway.** Terminate TLS at an Ingress or Gateway and set - **No TLS in the gateway.** Terminate TLS at an Ingress or Gateway and set
`COOKIE_SECURE=true`. `COOKIE_SECURE=true`.
- **SSO trust is hop-based.** `AUTH_PROXY_HEADER` trusts whoever sets the header; - **SSO trust is hop-based.** `AUTH_PROXY_HEADER` trusts whoever sets the header.
the only guarantee is that the gateway Service is not directly reachable. The SSO variant ships `75-networkpolicies.yaml`, which restricts the gateway
There is no signature, mTLS, or NetworkPolicy enforcing that in this example. to oauth2-proxy and oauth2-proxy egress to the gateway/DNS/public HTTPS; a
custom exposure without that policy has no signature, mTLS or network
enforcement.
- **Isolation is opt-in.** gVisor, the sandbox NetworkPolicy and the exposure
policies are opt-in; without them, sandboxes share the host kernel and can
reach the cluster network. The base example is a demo, not a hardened
multi-tenant platform.
- **OAuth org is the only authorization.** Any GitHub user in the configured - **OAuth org is the only authorization.** Any GitHub user in the configured
organization gets an auto-provisioned sandbox; there is no per-user allow-list, organization gets an auto-provisioned sandbox; there is no per-user allow-list,
quota, or admin API. quota, or admin API.
@@ -564,3 +652,6 @@ delete the Secrets with the namespace and revoke the OAuth App in GitHub.
- [Architecture](docs/architecture.md) — container/sequence diagrams, ADRs, - [Architecture](docs/architecture.md) — container/sequence diagrams, ADRs,
security and RBAC, failure modes, evolution path. security and RBAC, failure modes, evolution path.
- [Exposure README](manifests/exposure/README.md) — the two variants in detail. - [Exposure README](manifests/exposure/README.md) — the two variants in detail.
- [Optional hardening](docs/optional-hardening.md) — gVisor, nodeSelector,
NetworkPolicy, extra provider keys and failed-Pod recovery, with verification
and rollback.

View File

@@ -6,7 +6,7 @@ Kubernetes, and DSH Web fit together, and why the design looks the way it does.
Scope: one Kubernetes namespace (`dsh-demo`), one shared DSH Web image, one gateway deployment, Scope: one Kubernetes namespace (`dsh-demo`), one shared DSH Web image, one gateway deployment,
and one DSH Web pod + PVC per user. The cluster is assumed to run and one DSH Web pod + PVC per user. The cluster is assumed to run
[agent-sandbox](https://github.com/kubernetes-sigs/agent-sandbox) v1.0.7 with the extensions [agent-sandbox](https://github.com/kubernetes-sigs/agent-sandbox) v1.0.8 with the extensions
(`SandboxTemplate`, `SandboxWarmPool`, `SandboxClaim`). (`SandboxTemplate`, `SandboxWarmPool`, `SandboxClaim`).
## Container view ## Container view
@@ -67,6 +67,13 @@ as that user. The gateway ignores the header unless `AUTH_PROXY_HEADER` is set.
[exposure README](../manifests/exposure/README.md) and the [exposure README](../manifests/exposure/README.md) and the
[gateway README](../gateway/README.md#sso-opt-in). [gateway README](../gateway/README.md#sso-opt-in).
Each variant also ships a NetworkPolicy: `ingress/61-networkpolicy.yaml` limits gateway ingress
to Traefik pods, and `gateway-api-sso/75-networkpolicies.yaml` limits the gateway to
oauth2-proxy, oauth2-proxy ingress to the Gateway implementation, and oauth2-proxy egress to
DNS/public HTTPS/the gateway. The SSO policy turns the trust boundary into pod-traffic
enforcement; the caveats (Traefik label, `hostNetwork`, kubelet probes) are in
[optional hardening](optional-hardening.md).
## Flows ## Flows
### Signup (browser) ### Signup (browser)
@@ -290,8 +297,15 @@ Notes and limits:
- **TLS is external.** The gateway serves plain HTTP; terminate TLS at the Ingress and set - **TLS is external.** The gateway serves plain HTTP; terminate TLS at the Ingress and set
`COOKIE_SECURE=true`. Without TLS, session cookies are replayable on the network. `COOKIE_SECURE=true`. Without TLS, session cookies are replayable on the network.
- **Tenant isolation** is pod + PVC separation; all tenants share one namespace, one DSH binary, - **Tenant isolation** is pod + PVC separation; all tenants share one namespace, one DSH binary,
and one `DEEPSEEK_API_KEY`. Add NetworkPolicies and per-tenant credentials before calling it and the shared provider keys (`dsh-provider-keys` plus the optional
multi-tenant-grade. `dsh-provider-keys-extra`). `manifests/optional/network-policy/` adds an opt-in sandbox
NetworkPolicy, and gVisor/`nodeSelector` are opt-in PodSpec fields. Add per-tenant
credentials before calling it multi-tenant-grade.
- **Optional hardening is opt-in.** gVisor, node selection, the sandbox/exposure NetworkPolicies,
extra provider keys and failed-Pod recovery are documented in
[optional hardening](optional-hardening.md). The base manifests stay portable (no gVisor
RuntimeClass, no active node labels, no NetworkPolicy, no extra Secret required, and the
controller's default `failedPodPolicy: Retain`).
- **No gateway rate limiting.** Signup can create claims until the cluster or warm pool is - **No gateway rate limiting.** Signup can create claims until the cluster or warm pool is
exhausted; put limits at the ingress for real deployments. exhausted; put limits at the ingress for real deployments.
@@ -306,6 +320,11 @@ Notes and limits:
| DSH auth cookie expired mid-session | Proxy `401` once, then recovers | Gateway refreshes the handshake once | | DSH auth cookie expired mid-session | Proxy `401` once, then recovers | Gateway refreshes the handshake once |
| Upstream pod name changed after resume | Cached cookie keyed by old pod UID is useless | Cache is invalidated on suspend; next request re-handshakes | | Upstream pod name changed after resume | Cached cookie keyed by old pod UID is useless | Cache is invalidated on suspend; next request re-handshakes |
| Two gateway replicas | Double suspend/wake races | Keep `replicas: 1` until state is externalized | | Two gateway replicas | Double suspend/wake races | Keep `replicas: 1` until state is externalized |
| Optional NetworkPolicy blocks DNS/probes | Sandbox 502/503, pod restarts, failed `getent` | Check the CNI, DNS selector/NodeLocal address and kubelet-probe traffic; see the hardening doc |
| gVisor + `kubectl port-forward` | Forward refuses or returns nothing | Port-forward is incompatible with gVisor; use the gateway or a reachable Service |
| Provider key added but missing in the pod | Requests still fail with `MISSING_CREDENTIAL` | Pod env is stamped at creation; recreate spares/pods. A stored credential wins over env |
| `nodeSelector` matches no node | Pod `Pending`, `0/N nodes are available` | Fix the label, pool capacity or taint/toleration; RuntimeClass scheduling may also constrain it |
| Sandbox Pod reaches phase `Failed` (node eviction, spot reclaim) | Sandbox reports `Finished=True`; the gateway keeps failing against a dead Pod | Set `failedPodPolicy: Recreate` (agent-sandbox v1.0.8+) so the controller replaces the Pod with the same identity and PVCs; otherwise delete the Pod by hand. See the hardening doc |
## Upstream feature request: remove log scraping ## Upstream feature request: remove log scraping

View File

@@ -0,0 +1,640 @@
# Optional hardening: gVisor, nodeSelector, NetworkPolicy, provider keys and failed-Pod recovery
The base example is deliberately portable: it runs on kind and GKE without a
gVisor runtime, a tainted node pool or a NetworkPolicy-capable CNI, and it
boots with only the required `dsh-provider-keys` Secret. This document covers
four **opt-in** isolation layers, plus one availability setting:
| Setting | Where it lives | What it changes |
| --- | --- | --- |
| gVisor | commented `runtimeClassName` in the SandboxTemplates and Deployments | puts DSH and/or the gateway in a userspace kernel |
| nodeSelector | commented `nodeSelector` in the same files | pins pods to a chosen node pool |
| NetworkPolicy | `manifests/optional/network-policy/`, `manifests/exposure/*/` | limits sandbox, gateway and proxy connectivity |
| Provider keys | optional `dsh-provider-keys-extra` Secret | injects extra provider API keys into every sandbox |
| Failed-Pod recovery | commented `failedPodPolicy` in the SandboxTemplates | replaces a Failed sandbox Pod instead of parking the sandbox |
Each setting is independent. Enable only what the cluster supports and test the
walkthrough after each change.
## 1. gVisor
### What it changes
The DSH sandbox and the gateway (and optionally oauth2-proxy) run under the
`gvisor` RuntimeClass. `agent-sandbox` passes `runtimeClassName` straight to the
PodSpec; no CRD change is involved.
### Prerequisites
- A `RuntimeClass` named `gvisor`:
```bash
kubectl get runtimeclass gvisor -o yaml
```
- Nodes capable of running it (`runsc` installed and registered with the
container runtime). How you get there is cluster-specific:
- GKE Sandbox: create a node pool with `--sandbox=type=gvisor`; GKE labels
the nodes `sandbox.gke.io/runtime: gvisor`, taints them
`sandbox.gke.io/runtime=gvisor:NoSchedule`, and its `gvisor` RuntimeClass
applies the matching affinity and toleration automatically.
- Self-managed: install gVisor, create the `gvisor` RuntimeClass with
`handler: runsc`, and label/taint a dedicated node pool yourself.
- A scheduling plan: a gVisor RuntimeClass without capable nodes makes pods
fail to start. Use the RuntimeClass `scheduling` fields or the optional
`nodeSelector`/toleration described in section 2.
### Enable it
**Fresh install:** uncomment `runtimeClassName: gvisor` before applying:
- [`manifests/10-sandboxtemplate.yaml`](../manifests/10-sandboxtemplate.yaml)
- [`manifests/50-gateway.yaml`](../manifests/50-gateway.yaml)
- [`manifests/exposure/gateway-api-sso/72-oauth2-proxy.yaml`](../manifests/exposure/gateway-api-sso/72-oauth2-proxy.yaml)
(only with the SSO variant)
- [minimal sandbox-template.yaml](../../minimal-single-tenant/manifests/sandbox-template.yaml)
**Existing install:** a template edit does not update existing Sandboxes, and
warm-pool spares use `OnReplenish`, so patch explicitly. Plan a maintenance
window: claimed users are disconnected when their pod is recreated.
```bash
# 1. The blueprint for future sandboxes.
kubectl -n dsh-demo patch sandboxtemplate dsh-web --type merge \
-p '{"spec":{"podTemplate":{"spec":{"runtimeClassName":"gvisor"}}}}'
# 2. Warm-pool spares: delete them; the pool replenishes from the template.
kubectl -n dsh-demo delete sandboxes -l agents.x-k8s.io/warm-pool-sandbox
# 3. Claimed sandboxes: snapshot their pod template, then suspend/resume.
for sb in $(kubectl -n dsh-demo get sandboxes \
-l '!agents.x-k8s.io/warm-pool-sandbox' -o jsonpath='{.items[*].metadata.name}'); do
kubectl -n dsh-demo patch sandbox "$sb" --type merge \
-p '{"spec":{"podTemplate":{"spec":{"runtimeClassName":"gvisor"}}}}'
kubectl -n dsh-demo patch sandbox "$sb" --type merge \
-p '{"spec":{"operatingMode":"Suspended"}}'
kubectl -n dsh-demo patch sandbox "$sb" --type merge \
-p '{"spec":{"operatingMode":"Running"}}'
done
# 4. The gateway (and oauth2-proxy when the SSO variant is installed).
kubectl -n dsh-demo patch deploy dsh-gateway --type merge \
-p '{"spec":{"template":{"spec":{"runtimeClassName":"gvisor"}}}}'
kubectl -n dsh-demo rollout status deploy/dsh-gateway --timeout=180s
# kubectl -n dsh-demo patch deploy oauth2-proxy --type merge \
# -p '{"spec":{"template":{"spec":{"runtimeClassName":"gvisor"}}}}'
```
If the pool's nodes are tainted and the RuntimeClass does not already add a
toleration, add the matching `tolerations` block (section 2) in the same patch
or in the manifests.
### Verify
```bash
SB=$(kubectl -n dsh-demo get sandboxclaim dsh-alice -o jsonpath='{.status.sandbox.name}')
POD=$(kubectl -n dsh-demo get sandbox "$SB" \
-o jsonpath='{.metadata.annotations.agents\.x-k8s\.io/pod-name}'); POD=${POD:-$SB}
kubectl -n dsh-demo get pod "$POD" -o jsonpath='{.spec.runtimeClassName}{"\n"}'
# gvisor
kubectl -n dsh-demo exec "$POD" -- uname -r
kubectl -n dsh-demo get node "$(kubectl -n dsh-demo get pod "$POD" -o jsonpath='{.spec.nodeName}')" \
-o jsonpath='{.status.nodeInfo.kernelVersion}{"\n"}'
```
The pod's `runtimeClassName` is authoritative. The kernel release reported by
`uname -r` inside a gVisor pod is the sandbox kernel, not the host kernel; on
GKE it looks like `4.19.0-gvisor`.
### Caveats
- **`kubectl port-forward` does not work with gVisor.** The multi-tenant
gateway path (HTTP/WebSocket to the pod) works; the minimal example's
port-forward quickstart does not. Use the gateway example or a reachable
Service (NodePort/LoadBalancer/Sandbox Router) instead.
- gVisor has performance overhead and does not implement every syscall. Test
DSH and the agent tools you rely on (`git`, `gh`, `uv`, language toolchains).
- Node scheduling is the hard part. A `gvisor` RuntimeClass on a cluster with
no capable nodes leaves pods unschedulable or failing at container start.
- Existing Sandboxes are snapshots. The loop above is required for every
claimed sandbox; deleting the `SandboxClaim` would also delete its PVC and
workspace data, so do not use claim re-creation as a shortcut.
- A resumed sandbox starts cold; the startup probe gives it up to 120s.
### Roll back
```bash
kubectl -n dsh-demo patch sandboxtemplate dsh-web --type merge \
-p '{"spec":{"podTemplate":{"spec":{"runtimeClassName":null}}}}'
for sb in $(kubectl -n dsh-demo get sandboxes -o jsonpath='{.items[*].metadata.name}'); do
kubectl -n dsh-demo patch sandbox "$sb" --type merge \
-p '{"spec":{"podTemplate":{"spec":{"runtimeClassName":null}}}}'
done
kubectl -n dsh-demo delete sandboxes -l agents.x-k8s.io/warm-pool-sandbox
kubectl -n dsh-demo patch deploy dsh-gateway --type merge \
-p '{"spec":{"template":{"spec":{"runtimeClassName":null}}}}'
# kubectl -n dsh-demo patch deploy oauth2-proxy --type merge \
# -p '{"spec":{"template":{"spec":{"runtimeClassName":null}}}}'
```
Recreate the pods (delete spares, suspend/resume claimed sandboxes, let the
Deployment roll out) so the cleared field takes effect.
## 2. nodeSelector
### What it changes
Pods get a `nodeSelector` so they land only on nodes carrying the chosen label.
The same field is available in both SandboxTemplates and both Deployments;
the shipped value is **commented out** because active cluster-specific
scheduling does not belong in the repository.
### Choose a label
```bash
kubectl get nodes --show-labels
```
- **GKE Sandbox:** the nodes carry `sandbox.gke.io/runtime: gvisor` and are
tainted `sandbox.gke.io/runtime=gvisor:NoSchedule`. With
`runtimeClassName: gvisor`, GKE already schedules the pod onto those nodes;
add a `nodeSelector` only when you need a specific sandbox pool or
ComputeClass (`cloud.google.com/compute-class: <name>`).
- **Self-managed:** label the pool you want, for example
`dsh.example.com/node-pool: gvisor`, and use that label.
- **General pinning:** any stable node-pool label works. Keep the real key and
value in a local patch; the manifests use `dsh.example.com/node-pool` as an
obvious placeholder.
### Enable it
Uncomment and replace the placeholder before a fresh apply:
```yaml
nodeSelector:
dsh.example.com/node-pool: gvisor
```
For an existing install, patch the template, each claimed Sandbox and the
Deployments, then recreate pods exactly as in section 1:
```bash
kubectl -n dsh-demo patch sandboxtemplate dsh-web --type merge \
-p '{"spec":{"podTemplate":{"spec":{"nodeSelector":{"dsh.example.com/node-pool":"gvisor"}}}}}'
# ... same warm-pool delete / claimed-sandbox suspend-resume / Deployment
# patch and rollout as in section 1, with nodeSelector instead of
# runtimeClassName.
```
If the selected nodes are tainted, add a matching toleration. This is the
companion block; do **not** uncomment it unless the pool is actually tainted:
```yaml
tolerations:
- key: dsh.example.com/node-pool
operator: Equal
value: gvisor
effect: NoSchedule
```
GKE Sandbox's taint is `sandbox.gke.io/runtime=gvisor:NoSchedule`; the GKE
`gvisor` RuntimeClass adds that toleration automatically, so a manual copy is
only needed when you schedule without the RuntimeClass or with a different
pool.
### Verify and troubleshoot
```bash
kubectl -n dsh-demo get pod "$POD" -o wide
kubectl -n dsh-demo get pod "$POD" -o jsonpath='{.spec.nodeSelector}{"\n"}'
kubectl -n dsh-demo describe pod "$POD" | sed -n '/Events:/,$p'
```
- `Pending` with `0/N nodes are available`: the label is wrong, the pool is
full, or the pod needs a toleration for a taint.
- A pod-level `nodeSelector` is merged with the RuntimeClass
`scheduling.nodeSelector`. If both are set and conflict, no node matches.
- Labels changed after scheduling do not move running pods. Patch/recreate.
- A `nodeSelector` that excludes every gVisor node is a valid way to keep the
gateway on a normal pool while the sandboxes run under gVisor.
### Roll back
Remove the field from the manifests, or patch it to `null`, then recreate the
pods:
```bash
kubectl -n dsh-demo patch sandboxtemplate dsh-web --type merge \
-p '{"spec":{"podTemplate":{"spec":{"nodeSelector":null}}}}'
# ... same per-Sandbox and Deployment loop as section 1.
```
## 3. NetworkPolicy
The base example does **not** manage NetworkPolicy: the SandboxTemplate is
`networkPolicyManagement: Unmanaged`, and the gateway has no policy. That keeps
the walkthrough portable and independent of the sandbox-router; the opt-in
policies below are standard Kubernetes `NetworkPolicy` objects.
### Optional sandbox policy
[`manifests/optional/network-policy/10-sandbox-networkpolicy.yaml`](../manifests/optional/network-policy/10-sandbox-networkpolicy.yaml)
selects every Sandbox pod through the agent-sandbox tracking label
`agents.x-k8s.io/sandbox-name-hash`. It applies to existing warm-pool and
claimed sandboxes without recreating them.
```bash
kubectl apply -f manifests/optional/network-policy/
kubectl delete -f manifests/optional/network-policy/ # rollback
```
What it allows:
| Direction | Allowed | Blocked |
| --- | --- | --- |
| Ingress | gateway pods (`app: dsh-gateway`) on TCP `3080` | everything else, including other sandboxes and cluster Services |
| Egress | UDP/TCP `53` to kube-dns and NodeLocal DNSCache; TCP `443` to public IPs | RFC1918, CGNAT, link-local/metadata, cluster Services/API, plain HTTP, other ports |
Verify from inside the pods (NetworkPolicy does not affect `kubectl exec`
itself, only the traffic the commands generate):
```bash
SB=$(kubectl -n dsh-demo get sandboxclaim dsh-alice -o jsonpath='{.status.sandbox.name}')
POD=$(kubectl -n dsh-demo get sandbox "$SB" \
-o jsonpath='{.metadata.annotations.agents\.x-k8s\.io/pod-name}'); POD=${POD:-$SB}
# Allowed: gateway -> sandbox.
kubectl -n dsh-demo exec deploy/dsh-gateway -- python -c \
"import urllib.request; print(urllib.request.urlopen('http://$POD:3080/favicon.svg', timeout=5).status)"
# Allowed: DNS and public HTTPS from the sandbox.
kubectl -n dsh-demo exec "$POD" -- getent hosts api.deepseek.com
kubectl -n dsh-demo exec "$POD" -- curl -fsS -o /dev/null -w '%{http_code}\n' https://example.com
# Denied: another sandbox cannot reach this one on 3080 (expect a timeout).
OTHER_SB=$(kubectl -n dsh-demo get sandboxclaim dsh-bob -o jsonpath='{.status.sandbox.name}')
OTHER_POD=$(kubectl -n dsh-demo get sandbox "$OTHER_SB" \
-o jsonpath='{.metadata.annotations.agents\.x-k8s\.io/pod-name}'); OTHER_POD=${OTHER_POD:-$OTHER_SB}
kubectl -n dsh-demo exec "$OTHER_POD" -- curl --max-time 3 -sS "http://$POD:3080/favicon.svg"
```
Caveats:
- **The CNI must enforce NetworkPolicy.** If it does not, the file applies but
has no effect.
- DNS: the default assumes CoreDNS/kube-dns pods labelled `k8s-app: kube-dns`
and covers NodeLocal DNSCache at `169.254.20.10`. A different DNS layout
needs a selector/`ipBlock` edit.
- **Kubelet probes** from the node are CNI-dependent. If a pod starts failing
its probes, allow the node CIDR on the probe port or remove the ingress
policy.
- The policy blocks cluster Services and the Kubernetes API from the sandbox.
DSH does not need them, but an agent tool that does needs an explicit rule.
- It allows any public host on TCP `443`, not an FQDN allowlist. It is network
isolation, not data-loss prevention.
- Custom provider base URLs, an HTTP proxy, port `80`, SSH `git` (port `22`) or
a non-443 provider endpoint need explicit egress additions.
- Do not combine this standalone policy with `networkPolicyManagement: Managed`
unless you migrate the rules into `spec.networkPolicy`. NetworkPolicies are
additive; the controller's Managed secure default would also be present.
- `kubectl port-forward` still works for debugging (the connection is local to
the pod), except under gVisor.
### Exposure-variant policies
Each exposure variant ships its own policy. They are part of the variant, not a
separate opt-in step:
- [`manifests/exposure/ingress/61-networkpolicy.yaml`](../manifests/exposure/ingress/61-networkpolicy.yaml)
allows only Traefik pods (`app.kubernetes.io/name: traefik`) to reach the
gateway on `8080`.
- [`manifests/exposure/gateway-api-sso/75-networkpolicies.yaml`](../manifests/exposure/gateway-api-sso/75-networkpolicies.yaml)
allows only oauth2-proxy to reach the gateway, only the Gateway
implementation (Traefik) to reach oauth2-proxy, and limits oauth2-proxy
egress to DNS, public HTTPS (GitHub) and the gateway.
The SSO policy is what turns the trust boundary in the
[exposure README](../manifests/exposure/README.md) into network enforcement.
The `dsh-gateway` Service must still stay ClusterIP; do not apply both exposure
variants.
Verify the negative case from a sandbox pod (it is not the proxy):
```bash
kubectl -n dsh-demo exec "$POD" -- curl --max-time 3 -sS http://dsh-gateway:8080/healthz
# expected: timeout/connection failure
```
Then verify the real browser path through the Ingress or Gateway, and confirm
the gateway/proxy pods are `Ready`. If Traefik uses `hostNetwork` or a
different pod label, replace the `podSelector` with the correct selector or an
`ipBlock`. As with the sandbox policy, kubelet probes can be affected on some
CNIs.
### Gateway egress (documented, not shipped)
The gateway needs DNS, the Kubernetes API, and TCP `3080` to sandbox pods. The
API server endpoint is cluster-specific (Service CIDR or node IP/port), so an
egress policy is **not** shipped. If your cluster has a stable API-server CIDR
and port, add an egress policy like this (replace the placeholders; verify the
port, commonly `443` or `6443`):
```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: dsh-gateway-egress
namespace: dsh-demo
spec:
podSelector:
matchLabels:
app: dsh-gateway
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
- to:
- ipBlock:
cidr: 10.96.0.1/32 # placeholder: API-server Service/cluster endpoint
ports:
- protocol: TCP
port: 443
- to:
- podSelector:
matchExpressions:
- key: agents.x-k8s.io/sandbox-name-hash
operator: Exists
ports:
- protocol: TCP
port: 3080
```
Cilium users can express the API server as `toEntities: kube-apiserver` (or
`toServices`) instead of a CIDR. Do not apply the sample blindly: a wrong CIDR
or port silently breaks signup/proxy.
## 4. Optional provider keys
### How injection works
The SandboxTemplate injects two Secrets via `envFrom`:
- `dsh-provider-keys` — required by the walkthrough; holds
`DEEPSEEK_API_KEY`.
- `dsh-provider-keys-extra` — optional; referenced with `optional: true`, so a
missing Secret does not block sandbox startup. Every key in it becomes an
environment variable in every sandbox.
Create the extra Secret before the warm pool (pod env is stamped at creation):
```bash
kubectl -n dsh-demo create secret generic dsh-provider-keys-extra \
--from-literal=OPENROUTER_API_KEY="sk-or-..." \
--from-literal=GEMINI_API_KEY="..." \
--from-literal=NVIDIA_API_KEY="nvapi-..." \
--dry-run=client -o yaml | kubectl apply -f -
```
Use the exact env-var names (uppercase, underscores) — invalid Secret keys are
skipped by the kubelet with an event. Do not duplicate a key across the two
Secrets; when the same key exists twice, the later `envFrom` entry wins.
### Supported names (curated)
These are the names the pinned DSH release resolves through its packaged provider
catalog; the catalog has more entries, these are the common ones:
| Provider | Environment variable |
| --- | --- |
| DeepSeek | `DEEPSEEK_API_KEY` |
| OpenRouter | `OPENROUTER_API_KEY` |
| Google Gemini | `GEMINI_API_KEY` |
| NVIDIA | `NVIDIA_API_KEY` |
| OpenAI | `OPENAI_API_KEY` |
| Anthropic | `ANTHROPIC_API_KEY` |
| Groq | `GROQ_API_KEY` |
| xAI | `XAI_API_KEY` |
| Mistral | `MISTRAL_API_KEY` |
| Together | `TOGETHER_API_KEY` |
| Cerebras | `CEREBRAS_API_KEY` |
| Vercel AI Gateway | `AI_GATEWAY_API_KEY` |
| Hugging Face | `HF_TOKEN` |
Aliases to avoid: `GOOGLE_API_KEY`, `HUGGINGFACE_API_KEY` and
`VERCEL_AI_GATEWAY_API_KEY`. The packaged catalog looks for `GEMINI_API_KEY`,
`HF_TOKEN` and `AI_GATEWAY_API_KEY` instead.
### Picking up a key change
A Secret update does not change running pods. The base key and extra keys are
both snapshotted at container start:
```bash
# Warm-pool spares: delete them so the pool recreates them from the template.
kubectl -n dsh-demo delete sandboxes -l agents.x-k8s.io/warm-pool-sandbox
# Running claimed sandboxes: delete the pod (the Sandbox controller recreates
# it) or suspend/resume the sandbox.
POD=... # resolve it as in section 3
kubectl -n dsh-demo delete pod "$POD"
```
New claims get the new values automatically. Confirm names without printing
secret values:
```bash
kubectl -n dsh-demo exec "$POD" -- sh -c '
for k in DEEPSEEK_API_KEY OPENROUTER_API_KEY GEMINI_API_KEY NVIDIA_API_KEY; do
if [ -n "$(printenv "$k" || true)" ]; then echo "$k=set"; else echo "$k=unset"; fi
done'
```
### Precedence and shared-key limits
- DSH's stored credential file (`$DSH_HOME/.credentials.yaml`, mode `0600`)
**wins over the environment**. If a tenant saved a key through the DSH UI,
changing the Secret will not override it.
- Both Secrets are shared by **every** tenant, and any agent can read its own
environment. This is a warm-pool constraint, not tenant isolation. A tenant
can override its own key through the DSH credential store on its PVC, but the
platform does not manage that; real multi-tenancy needs per-tenant
credentials or an egress proxy.
- The optional Secret is created imperatively; the repository never contains a
real key. Keep it out of Git and out of shell history where possible.
## 5. Failed-Pod recovery
### What it changes
When a sandbox Pod reaches phase `Failed`, the controller's default policy,
`Retain`, leaves the Pod in place and surfaces `Finished=True` on the Sandbox.
The Sandbox, its claim and its PVC survive, but nothing runs: the gateway keeps
proxying to a dead Pod and every request fails until someone deletes the Pod or
the claim.
agent-sandbox v1.0.8 (the pin in [versions.env](../../../versions.env)) adds
`spec.failedPodPolicy` to `Sandbox` and to `SandboxTemplate`:
| Value | Behavior |
| --- | --- |
| `Retain` (default) | The Failed Pod stays and the Sandbox reports `Finished=True`; the evidence stays available for debugging. |
| `Recreate` | The controller deletes the Failed Pod and starts a replacement; the Sandbox identity and its Sandbox-owned PVCs are retained. |
Repeated failures back off like a Deployment - 5 s, doubling to 5 min between
creations - so a crash loop cannot hammer the API server. The backoff resets
once the replacement Pod has been continuously `Running` for 10 minutes, and it
lives in memory only, so a controller restart clears it.
The base templates ship it commented out, for two reasons:
- A deterministic failure (bad image, bad Secret, a wrong probe) would recreate
forever while hiding the cause.
- The templates use `restartPolicy: OnFailure`, so the kubelet restarts a
crashing container in place and the Pod does not reach phase `Failed` anyway.
This policy matters for terminal failures - most commonly node eviction
(`Evicted`), when a spot/preemptible node is reclaimed or the node hits memory
pressure.
No gateway change is needed: the handshake cache is keyed by Pod UID, so a
replacement Pod re-handshakes on the next request, and the existing `401` retry
covers containers restarted in place.
### Enable it
```bash
# 1. The blueprint for future sandboxes.
kubectl -n dsh-demo patch sandboxtemplate dsh-web --type merge \
-p '{"spec":{"failedPodPolicy":"Recreate"}}'
# 2. Warm-pool spares were stamped at creation: delete them and let the pool
# replenish from the updated template.
kubectl -n dsh-demo delete sandboxes -l agents.x-k8s.io/warm-pool-sandbox
# 3. Claimed sandboxes keep their own spec: patch them in place.
for sb in $(kubectl -n dsh-demo get sandboxes \
-l '!agents.x-k8s.io/warm-pool-sandbox' -o jsonpath='{.items[*].metadata.name}'); do
kubectl -n dsh-demo patch sandbox "$sb" --type merge \
-p '{"spec":{"failedPodPolicy":"Recreate"}}'
done
```
For a fresh install, uncomment `failedPodPolicy: Recreate` in the template
instead.
### Verify
```bash
SB=$(kubectl -n dsh-demo get sandboxclaim dsh-alice -o jsonpath='{.status.sandbox.name}')
kubectl -n dsh-demo get sandbox "$SB" -o jsonpath='{.spec.failedPodPolicy}{"\n"}'
# Recreate
```
A failure drill that never touches a user sandbox - a `Sandbox` with
`restartPolicy: Never` and a container that exits non-zero (swap the image if
Docker Hub is unreachable):
```bash
kubectl -n dsh-demo apply -f - <<'EOF'
apiVersion: agents.x-k8s.io/v1beta1
kind: Sandbox
metadata:
name: failed-pod-drill
spec:
failedPodPolicy: Recreate
podTemplate:
spec:
restartPolicy: Never
containers:
- name: drill
image: busybox:1.36
command: ["sh", "-c", "exit 1"]
EOF
kubectl -n dsh-demo get pods -w
# The drill Pod reaches phase Failed; a replacement appears within ~5 s, then
# 10 s, 20 s, ... as the container keeps exiting non-zero.
kubectl -n dsh-demo delete sandbox failed-pod-drill
```
Repeat the drill with `failedPodPolicy: Retain` (or omit the field) to see the
default: the Pod stays `Failed` and the Sandbox condition `Finished` becomes
`True`.
### Caveats
- `Recreate` deletes the Failed Pod, so its logs go with it. Capture
`kubectl get events` or switch back to `Retain` when debugging.
- The replacement is a cold start: DSH boots again and issues a new launch
token. The workspace comes back from the PVC; in-flight sessions do not.
- The field needs agent-sandbox v1.0.8+. A v1.0.7 cluster rejects it as an
unknown field.
- It applies to sandbox Pods only. The gateway and oauth2-proxy are plain
Deployments, and their ReplicaSets already replace Failed Pods.
### Roll back
```bash
kubectl -n dsh-demo patch sandboxtemplate dsh-web --type merge \
-p '{"spec":{"failedPodPolicy":"Retain"}}'
for sb in $(kubectl -n dsh-demo get sandboxes -o jsonpath='{.items[*].metadata.name}'); do
kubectl -n dsh-demo patch sandbox "$sb" --type merge \
-p '{"spec":{"failedPodPolicy":"Retain"}}'
done
```
Warm-pool spares pick the default up as they are claimed and replenished, or
delete them as in step 2 above.
## 6. Verification summary
Required locally for any change to this example:
```bash
make check # versions, example layout, manifest YAML shape
yamllint .
make test # gateway unit tests (gateway code is untouched)
```
CI additionally parses every manifest and runs `kubectl apply
--dry-run=server` against the real agent-sandbox CRDs. The weekly kind e2e
runs the base walkthrough; it also proves the optional extra Secret may be
absent.
Manual checks, on a cluster with the relevant CNI/runtime:
- gVisor: pod `runtimeClassName`, `uname -r` differs from the node kernel,
gateway proxy works, port-forward limitation acknowledged.
- nodeSelector: pods land on the intended nodes; no `Pending` events.
- Sandbox policy: the allow/deny matrix above.
- Exposure policies: browser path works; a non-proxy pod cannot reach the
gateway; probes stay healthy.
- Provider keys: set/unset output (never the values) and the refresh procedure.
- Failed-Pod recovery: `spec.failedPodPolicy` on the Sandbox, and the drill
above replaces a `Failed` Pod while keeping the claim and PVC.
## 7. Rollback summary
| Layer | Rollback |
| --- | --- |
| Sandbox policy | `kubectl delete -f manifests/optional/network-policy/` |
| Exposure policy | delete the exposure variant directory |
| gVisor | patch `runtimeClassName: null`, recreate pods |
| nodeSelector | patch `nodeSelector: null`, recreate pods |
| Provider keys | delete the extra Secret or a key, recreate pods |
| Failed-Pod recovery | patch `failedPodPolicy: Retain`, delete warm-pool spares |
No PVC or workspace data is touched by any rollback. Deleting a `SandboxClaim`
is different: it cascades to the Sandbox **and** its PVC, so do not use claim
deletion to apply a pod-spec change.

View File

@@ -22,7 +22,7 @@ It does four things:
The DSH version pin lives in [`versions.env`](../../../versions.env) The DSH version pin lives in [`versions.env`](../../../versions.env)
(`DSH_VERSION` plus the optional repository `RELEASE_REVISION`); the example (`DSH_VERSION` plus the optional repository `RELEASE_REVISION`); the example
manifests and published images use the release identity built from it (currently manifests and published images use the release identity built from it (currently
`dsh-v0.2.1-rc.3-r3`). DSH binds `:3080`, serves `/`, gates `/api` behind a `dsh-v0.2.1-rc.5`). DSH binds `:3080`, serves `/`, gates `/api` behind a
`dsh-auth-*` cookie and requires a WebSocket at `/api/remote.mux`. `dsh-auth-*` cookie and requires a WebSocket at `/api/remote.mux`.
## Flow ## Flow
@@ -110,7 +110,9 @@ user (creates their claim idempotently, mints a rotated session token) and
redirects to `/u/<user>/`, and `POST /recreate` becomes available for a clean redirects to `/u/<user>/`, and `POST /recreate` becomes available for a clean
restart. The base manifests leave it unset on purpose; the restart. The base manifests leave it unset on purpose; the
[SSO exposure variant](../manifests/exposure/gateway-api-sso/) documents how to [SSO exposure variant](../manifests/exposure/gateway-api-sso/) documents how to
enable it on the Deployment (`kubectl set env`). enable it on the Deployment (`kubectl set env`). That variant also ships
`75-networkpolicies.yaml`, which restricts the gateway to oauth2-proxy and
oauth2-proxy's egress to the gateway, DNS and public HTTPS.
> **Trust boundary — this is the security-critical part.** A trusted identity > **Trust boundary — this is the security-critical part.** A trusted identity
> header is only safe when **nothing else can reach the gateway**. The gateway > header is only safe when **nothing else can reach the gateway**. The gateway
@@ -118,6 +120,8 @@ enable it on the Deployment (`kubectl set env`).
> authenticating proxy; if a client can reach `dsh-gateway:8080` directly (a > authenticating proxy; if a client can reach `dsh-gateway:8080` directly (a
> second Ingress/Gateway/Route, a port-forward on a shared host, a broad > second Ingress/Gateway/Route, a port-forward on a shared host, a broad
> NetworkPolicy), it can send `X-Forwarded-User` itself and impersonate any user. > NetworkPolicy), it can send `X-Forwarded-User` itself and impersonate any user.
> The variant's `75-networkpolicies.yaml` restricts pod traffic to the proxy; a
> custom exposure without an equivalent policy has no network enforcement.
> Treat `AUTH_PROXY_HEADER` as "the proxy already authenticated this request", > Treat `AUTH_PROXY_HEADER` as "the proxy already authenticated this request",
> not as an authentication mechanism of its own. Disable SSO by removing the > not as an authentication mechanism of its own. Disable SSO by removing the
> variable: the header is then ignored and bearer-token/session auth applies > variable: the header is then ignored and bearer-token/session auth applies
@@ -232,5 +236,6 @@ uvicorn's access log so bearer tokens in URLs are not logged.
* **SSO trusts a header.** With `AUTH_PROXY_HEADER` set, any client that can * **SSO trusts a header.** With `AUTH_PROXY_HEADER` set, any client that can
reach the gateway Service directly can spoof the identity header. The gateway reach the gateway Service directly can spoof the identity header. The gateway
cannot enforce the trust boundary itself: keep the Service ClusterIP and cannot enforce the trust boundary itself: keep the Service ClusterIP and
reachable only through the authenticating proxy. There is no mTLS or signed reachable only through the authenticating proxy. The SSO exposure variant
identity assertion. ships a NetworkPolicy that restricts pod traffic to the proxy; without it
there is no mTLS or signed identity assertion.

View File

@@ -1,14 +1,31 @@
# DSH "Agents as a Service" — prerequisites: namespace + shared provider key. # DSH "Agents as a Service" — prerequisites: namespace + shared provider keys.
# #
# The dsh-provider-keys Secret is deliberately NOT committed with a usable # The dsh-provider-keys Secret is deliberately NOT committed with a usable
# value. DEEPSEEK_API_KEY reaches every tenant sandbox (injected via envFrom # value. DEEPSEEK_API_KEY reaches every tenant sandbox (injected via envFrom
# on the SandboxTemplate, stamped before any user exists — a warm-pool # on the SandboxTemplate, stamped before any user exists — a warm-pool
# constraint), so a real key committed here would be shared by every install # constraint), so a real key committed here would be shared by every install
# of this example. Generate a fresh value per install (README step 1; # of this example. Generate a fresh value per install (README step 1;
# scripts/e2e-kind.sh uses a throwaway): # scripts/e2e-kind.sh uses a throwaway). Use an upsert: this manifest already
# creates the empty Secret, so a plain `kubectl create` would fail:
# #
# kubectl -n dsh-demo create secret generic dsh-provider-keys \ # kubectl -n dsh-demo create secret generic dsh-provider-keys \
# --from-literal=DEEPSEEK_API_KEY="sk-..." # --from-literal=DEEPSEEK_API_KEY="sk-..." \
# --dry-run=client -o yaml | kubectl apply -f -
#
# Optional: additional providers (OpenRouter, Gemini, NVIDIA, OpenAI, ...) go
# in a second, optional Secret. The SandboxTemplate references it with
# `optional: true`, so sandboxes start even when it does not exist, and every
# key in it becomes an environment variable. Create it BEFORE the warm pool,
# because pod env is stamped at pod creation:
#
# kubectl -n dsh-demo create secret generic dsh-provider-keys-extra \
# --from-literal=OPENROUTER_API_KEY="sk-or-..." \
# --from-literal=GEMINI_API_KEY="..." \
# --from-literal=NVIDIA_API_KEY="nvapi-..." \
# --dry-run=client -o yaml | kubectl apply -f -
#
# Supported env names, precedence, shared-key caveats and the pod-refresh
# procedure are in docs/optional-hardening.md.
# #
# All the infrastructure in this example works without a real key; only # All the infrastructure in this example works without a real key; only
# actual model calls need one. # actual model calls need one.

View File

@@ -1,6 +1,6 @@
# The reusable DSH Web blueprint. Warm pools stamp pod env BEFORE any user # The reusable DSH Web blueprint. Warm pools stamp pod env BEFORE any user
# exists, so per-user secrets cannot live in pod env; the shared provider key # exists, so per-user secrets cannot live in pod env; the shared provider keys
# is injected via envFrom, and claims that would bypass the warm pool are # are injected via envFrom, and claims that would bypass the warm pool are
# rejected outright (see 40-claim-rejected.yaml). # rejected outright (see 40-claim-rejected.yaml).
# #
# A stripped-down single-tenant variant of this template (no warm pool, no # A stripped-down single-tenant variant of this template (no warm pool, no
@@ -21,16 +21,40 @@ spec:
# The gateway routes each user to their DSH Web instance by this name. # The gateway routes each user to their DSH Web instance by this name.
service: true service: true
# The example gateway does not manage NetworkPolicy, so it is left # The example leaves NetworkPolicy to an optional, opt-in manifest
# unmanaged here. A production platform would use # (manifests/optional/network-policy/): the base walkthrough should not
# networkPolicyManagement: Managed and restrict ingress to its gateway on # require a policy-capable CNI, and the controller's Managed secure default
# :3080 — with the shared provider key, that NetworkPolicy is the per-user # only allows the sandbox-router, which this example does not deploy. If you
# isolation boundary. # switch to Managed, define spec.networkPolicy explicitly or the gateway will
# be blocked. See docs/optional-hardening.md.
networkPolicyManagement: Unmanaged networkPolicyManagement: Unmanaged
# OPTIONAL: replace a Failed sandbox Pod instead of parking the sandbox.
# The default, Retain, leaves the Failed Pod in place and surfaces
# Finished=True on the Sandbox until a human intervenes. Recreate deletes the
# Failed Pod and starts a replacement with the same Sandbox identity and
# Sandbox-owned PVCs, backing off (5s doubling to 5m) on repeated failures.
# Requires agent-sandbox v1.0.8+; existing Sandboxes keep their own stamped
# policy, so patch them separately. See docs/optional-hardening.md.
# failedPodPolicy: Recreate
podTemplate: podTemplate:
spec: spec:
# DSH Web is multi-arch (amd64 + arm64); no arch nodeSelector needed. # DSH Web is multi-arch (amd64 + arm64); no arch nodeSelector is needed.
#
# OPTIONAL: run the sandbox under gVisor. Requires a RuntimeClass named
# "gvisor" and nodes that can run it; gVisor breaks `kubectl port-forward`
# into the sandbox (use the gateway). Existing Sandboxes snapshot their
# pod template, so enabling it later requires patching/recreating them.
# See docs/optional-hardening.md.
# runtimeClassName: gvisor
#
# OPTIONAL: pin sandboxes to a dedicated node pool (for example a gVisor
# pool). Replace this placeholder with a label your nodes actually carry;
# add a matching toleration if the pool is tainted. Active
# cluster-specific scheduling belongs in a local patch, not here.
# nodeSelector:
# dsh.example.com/node-pool: dsh
restartPolicy: "OnFailure" restartPolicy: "OnFailure"
securityContext: securityContext:
runAsUser: 1000 # the 'node' user in the DSH image runAsUser: 1000 # the 'node' user in the DSH image
@@ -49,7 +73,7 @@ spec:
fsGroupChangePolicy: OnRootMismatch fsGroupChangePolicy: OnRootMismatch
containers: containers:
- name: dsh-web - name: dsh-web
image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.3-r3 image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.5
imagePullPolicy: IfNotPresent imagePullPolicy: IfNotPresent
ports: ports:
- name: web - name: web
@@ -57,6 +81,13 @@ spec:
envFrom: envFrom:
- secretRef: - secretRef:
name: dsh-provider-keys name: dsh-provider-keys
# Optional extra provider keys (OpenRouter, Gemini, NVIDIA, ...).
# The Secret may be absent; every key in it becomes an env var.
# Create it before the warm pool, because pod env is stamped at
# pod creation. See docs/optional-hardening.md.
- secretRef:
name: dsh-provider-keys-extra
optional: true
env: env:
# Empty => DSH derives links from the request Host, which the # Empty => DSH derives links from the request Host, which the
# gateway forwards (plus X-Forwarded-*). Point this at the # gateway forwards (plus X-Forwarded-*). Point this at the

View File

@@ -84,6 +84,15 @@ spec:
app: dsh-gateway app: dsh-gateway
spec: spec:
serviceAccountName: dsh-gateway serviceAccountName: dsh-gateway
# OPTIONAL: run the gateway under gVisor. Requires a RuntimeClass named
# "gvisor" and nodes that can run it. See docs/optional-hardening.md.
# runtimeClassName: gvisor
#
# OPTIONAL: pin the gateway to a dedicated node pool. Replace the
# placeholder with a label your nodes carry; add a toleration for a
# tainted pool. Keep active cluster-specific scheduling in a local patch.
# nodeSelector:
# dsh.example.com/node-pool: dsh
securityContext: securityContext:
runAsNonRoot: true runAsNonRoot: true
runAsUser: 10000 # matches USER in gateway/Dockerfile runAsUser: 10000 # matches USER in gateway/Dockerfile
@@ -92,7 +101,7 @@ spec:
type: RuntimeDefault type: RuntimeDefault
containers: containers:
- name: gateway - name: gateway
image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway:dsh-v0.2.1-rc.3-r3 image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway:dsh-v0.2.1-rc.5
imagePullPolicy: IfNotPresent imagePullPolicy: IfNotPresent
securityContext: securityContext:
allowPrivilegeEscalation: false allowPrivilegeEscalation: false

View File

@@ -16,6 +16,19 @@ Applying both gives the gateway two entry points, and the SSO variant is only
safe when the gateway is reachable **exclusively** through oauth2-proxy. Do not safe when the gateway is reachable **exclusively** through oauth2-proxy. Do not
mix them. mix them.
Each variant also ships a NetworkPolicy, applied by the same directory apply:
- `ingress/61-networkpolicy.yaml` allows only Traefik pods to reach the gateway
on `8080`.
- `gateway-api-sso/75-networkpolicies.yaml` allows only oauth2-proxy to reach
the gateway, only the Gateway implementation (Traefik) to reach oauth2-proxy,
and limits oauth2-proxy egress to DNS, public HTTPS (GitHub) and the gateway.
The policies assume Traefik's `app.kubernetes.io/name: traefik` pod label.
Adjust the selector for another implementation or a `hostNetwork` install; see
[optional hardening](../../docs/optional-hardening.md) for the probe caveat and
verification commands.
## Applying a variant ## Applying a variant
Run these commands from the example root Run these commands from the example root
@@ -62,6 +75,9 @@ Secrets imperatively (commands below) before applying the directory.
`Secure` cookie is never sent over HTTP, which silently breaks login. `Secure` cookie is never sent over HTTP, which silently breaks login.
4. Do not attach buffering or compression middlewares to this router: they 4. Do not attach buffering or compression middlewares to this router: they
break WebSocket upgrades and streaming. break WebSocket upgrades and streaming.
5. `61-networkpolicy.yaml` is part of this variant: it allows only Traefik pods
to reach the gateway on `8080`. Adjust its selector for another ingress
controller or `hostNetwork`; if kubelet probes fail, see the hardening doc.
Verify: `curl -sI https://<host>/healthz` returns `200`. Verify: `curl -sI https://<host>/healthz` returns `200`.
@@ -70,6 +86,7 @@ Verify: `curl -sI https://<host>/healthz` returns `200`.
`70-gateway.yaml` defines the TLS `Gateway`, `71-httproute.yaml` sends every `70-gateway.yaml` defines the TLS `Gateway`, `71-httproute.yaml` sends every
request to oauth2-proxy, and `72-oauth2-proxy.yaml` authenticates the browser request to oauth2-proxy, and `72-oauth2-proxy.yaml` authenticates the browser
against GitHub and proxies to the gateway with `X-Forwarded-User` set. against GitHub and proxies to the gateway with `X-Forwarded-User` set.
`75-networkpolicies.yaml` makes this the only network path to the gateway.
Prerequisites: Prerequisites:
@@ -111,7 +128,7 @@ kubectl apply -f manifests/exposure/gateway-api-sso/
`AUTH_PROXY_HEADER` is opt-in and deliberately unset in the base gateway `AUTH_PROXY_HEADER` is opt-in and deliberately unset in the base gateway
manifest: the gateway ignores identity headers until it is set. Enabling SSO manifest: the gateway ignores identity headers until it is set. Enabling SSO
takes **both** sides: takes all of these:
1. `AUTH_PROXY_HEADER=X-Forwarded-User` on `deploy/dsh-gateway` (step 3 1. `AUTH_PROXY_HEADER=X-Forwarded-User` on `deploy/dsh-gateway` (step 3
above). Without it, oauth2-proxy's login has no effect on the gateway and above). Without it, oauth2-proxy's login has no effect on the gateway and
@@ -122,6 +139,10 @@ takes **both** sides:
other path to the Service) while SSO is enabled. A client that can reach other path to the Service) while SSO is enabled. A client that can reach
`dsh-gateway:8080` directly can set `X-Forwarded-User` itself and `dsh-gateway:8080` directly can set `X-Forwarded-User` itself and
impersonate any user. impersonate any user.
3. Keep `75-networkpolicies.yaml` with the variant. It enforces the boundary in
the network layer (gateway ← oauth2-proxy ← Traefik, and limited oauth2-proxy
egress). A custom exposure without an equivalent policy has no network
enforcement.
Also terminate TLS in front of the gateway (the Gateway listener does) and Also terminate TLS in front of the gateway (the Gateway listener does) and
leave `COOKIE_SECURE=true` so the session cookie never travels in clear text. leave `COOKIE_SECURE=true` so the session cookie never travels in clear text.

View File

@@ -16,6 +16,12 @@
# apply ../ingress/ (or anything else that reaches the gateway). A # apply ../ingress/ (or anything else that reaches the gateway). A
# client that can reach dsh-gateway:8080 can spoof X-Forwarded-User and # client that can reach dsh-gateway:8080 can spoof X-Forwarded-User and
# impersonate any user. # impersonate any user.
#
# 75-networkpolicies.yaml enforces that boundary at the network layer: the
# gateway accepts traffic only from oauth2-proxy pods, oauth2-proxy only from
# the Gateway implementation, and oauth2-proxy egress is limited to DNS,
# public HTTPS and the gateway. Apply it with this variant.
#
# TLS terminates at the Gateway listener (70-gateway.yaml); keep # TLS terminates at the Gateway listener (70-gateway.yaml); keep
# --cookie-secure=true and COOKIE_SECURE=true on the gateway. # --cookie-secure=true and COOKIE_SECURE=true on the gateway.
# #
@@ -42,6 +48,11 @@ spec:
app: oauth2-proxy app: oauth2-proxy
app.kubernetes.io/part-of: dsh-demo app.kubernetes.io/part-of: dsh-demo
spec: spec:
# OPTIONAL: run the proxy under gVisor and/or pin it to a node pool.
# See ../../../docs/optional-hardening.md.
# runtimeClassName: gvisor
# nodeSelector:
# dsh.example.com/node-pool: dsh
securityContext: securityContext:
runAsNonRoot: true runAsNonRoot: true
runAsUser: 2000 runAsUser: 2000

View File

@@ -0,0 +1,120 @@
# OPTIONAL with the SSO exposure variant: enforce the trust boundary at the
# network layer.
#
# The chain is Traefik (Gateway API) -> oauth2-proxy -> dsh-gateway. These
# policies make it the ONLY chain:
# - the gateway accepts traffic only from oauth2-proxy pods;
# - oauth2-proxy accepts traffic only from the Gateway implementation;
# - oauth2-proxy egress is limited to DNS, public HTTPS (GitHub) and the
# gateway.
# The gateway's own egress (Kubernetes API, sandbox pods) is intentionally not
# restricted here: the API-server endpoint is cluster-specific. See
# ../../../docs/optional-hardening.md for that recipe.
#
# This manifest assumes Traefik, matching gatewayClassName: traefik in
# 70-gateway.yaml. Adjust the app.kubernetes.io/name selector for another
# implementation. NetworkPolicy enforcement of kubelet probes is CNI-dependent;
# if the gateway or proxy fails its probes, allow the node CIDR on the probe
# port or remove the ingress policy.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: dsh-gateway-ingress
namespace: dsh-demo
spec:
podSelector:
matchLabels:
app: dsh-gateway
policyTypes:
- Ingress
ingress:
- from:
- podSelector:
matchLabels:
app: oauth2-proxy
ports:
- protocol: TCP
port: 8080
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: oauth2-proxy-ingress
namespace: dsh-demo
spec:
podSelector:
matchLabels:
app: oauth2-proxy
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector: {}
podSelector:
matchLabels:
app.kubernetes.io/name: traefik
ports:
- protocol: TCP
port: 4180
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: oauth2-proxy-egress
namespace: dsh-demo
spec:
podSelector:
matchLabels:
app: oauth2-proxy
policyTypes:
- Egress
egress:
# Cluster DNS.
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
# NodeLocal DNSCache.
- to:
- ipBlock:
cidr: 169.254.20.10/32
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
# The gateway backend.
- to:
- podSelector:
matchLabels:
app: dsh-gateway
ports:
- protocol: TCP
port: 8080
# GitHub OAuth endpoints are public HTTPS. Private ranges and the metadata
# server stay blocked.
- to:
- ipBlock:
cidr: 0.0.0.0/0
except:
- 10.0.0.0/8
- 172.16.0.0/12
- 192.168.0.0/16
- 169.254.0.0/16
- 100.64.0.0/10
- ipBlock:
cidr: "::/0"
except:
- "fc00::/7"
- "fe80::/10"
ports:
- protocol: TCP
port: 443

View File

@@ -23,6 +23,10 @@
# middlewares (e.g. compress, buffering) to this router — they break streaming # middlewares (e.g. compress, buffering) to this router — they break streaming
# and upgrades. `path: /` Prefix covers the landing page and every per-user # and upgrades. `path: /` Prefix covers the landing page and every per-user
# proxy path, so no per-sandbox Ingress is needed. # proxy path, so no per-sandbox Ingress is needed.
#
# 61-networkpolicy.yaml is part of this variant: it restricts the gateway's
# ingress to Traefik pods. Adjust its selector if your ingress controller uses
# a different label or hostNetwork.
apiVersion: networking.k8s.io/v1 apiVersion: networking.k8s.io/v1
kind: Ingress kind: Ingress
metadata: metadata:

View File

@@ -0,0 +1,31 @@
# OPTIONAL with the Ingress exposure variant: restrict the gateway's ingress
# to the ingress controller.
#
# This manifest assumes Traefik, matching this variant's
# `ingressClassName: traefik`. Traefik's Helm chart labels its pods
# app.kubernetes.io/name=traefik; adjust the selector for another controller or
# for a hostNetwork install (where the source is the node IP, not a pod IP).
#
# NetworkPolicy enforcement of kubelet probe traffic is CNI-dependent. If the
# gateway starts failing its /healthz probes, allow the node CIDR on port 8080
# or remove this policy. See ../../../docs/optional-hardening.md.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: dsh-gateway-ingress
namespace: dsh-demo
spec:
podSelector:
matchLabels:
app: dsh-gateway
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector: {}
podSelector:
matchLabels:
app.kubernetes.io/name: traefik
ports:
- protocol: TCP
port: 8080

View File

@@ -0,0 +1,40 @@
# Optional manifests
Opt-in manifests that are **not** part of the top-level walkthrough.
`kubectl apply -f manifests/` is non-recursive, so it does not apply this
directory; apply a subdirectory explicitly.
## network-policy/
A Kubernetes `NetworkPolicy` for the DSH sandboxes:
- ingress only from the gateway pods on TCP `3080` (HTTP + WebSocket);
- egress only to DNS and public HTTPS (provider APIs, package mirrors,
git-over-HTTPS).
Apply and delete it explicitly:
```bash
kubectl apply -f manifests/optional/network-policy/
kubectl delete -f manifests/optional/network-policy/
```
It requires a CNI that enforces `NetworkPolicy`. The selector uses the
agent-sandbox controller's pod tracking label, so the policy applies to
existing warm-pool and claimed sandboxes without recreation.
## gVisor, nodeSelector and provider keys
These are configured in the main manifests or with an extra Secret:
- **gVisor:** uncomment `runtimeClassName: gvisor` in the SandboxTemplate and
the Deployments before applying, or patch an existing install.
- **nodeSelector:** uncomment the field and replace the placeholder node label.
- **Failed-Pod recovery:** uncomment `failedPodPolicy: Recreate` in the
SandboxTemplate (agent-sandbox v1.0.8+) so a Failed Pod is replaced instead
of parked.
- **Provider keys:** create the optional `dsh-provider-keys-extra` Secret; the
base `dsh-provider-keys` Secret remains the one required key.
Full prerequisites, exact commands, verification and caveats:
[optional hardening](../../docs/optional-hardening.md).

View File

@@ -0,0 +1,80 @@
# OPTIONAL, opt-in tenant network policy for the DSH sandboxes.
#
# It is NOT part of the top-level walkthrough. Apply and delete it explicitly:
# kubectl apply -f manifests/optional/network-policy/
# kubectl delete -f manifests/optional/network-policy/
#
# The selector uses the agent-sandbox controller's pod tracking label, which is
# present on every Sandbox pod (including warm-pool spares), so the policy
# applies to existing sandboxes without recreating them. The SandboxTemplate
# stays networkPolicyManagement: Unmanaged. Do not also switch it to Managed
# unless you migrate these rules into spec.networkPolicy; NetworkPolicies are
# additive and the two configurations would be confusing.
#
# Requires a CNI that enforces NetworkPolicy. See docs/optional-hardening.md
# for the connectivity matrix, DNS variants, kubelet-probe caveat and rollback.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: dsh-sandbox
namespace: dsh-demo
spec:
podSelector:
matchExpressions:
- key: agents.x-k8s.io/sandbox-name-hash
operator: Exists
policyTypes:
- Ingress
- Egress
ingress:
# The gateway is the only intended client of a sandbox (HTTP + WebSocket).
- from:
- podSelector:
matchLabels:
app: dsh-gateway
ports:
- protocol: TCP
port: 3080
egress:
# Cluster DNS. CoreDNS/kube-dns pods carry k8s-app=kube-dns.
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
# NodeLocal DNSCache, for clusters that do not send DNS to the kube-dns
# Service. 169.254.20.10 is its standard link-local address.
- to:
- ipBlock:
cidr: 169.254.20.10/32
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
# Public HTTPS only: provider APIs, package mirrors, git-over-HTTPS.
# Private ranges, CGNAT and the metadata server stay blocked.
- to:
- ipBlock:
cidr: 0.0.0.0/0
except:
- 10.0.0.0/8
- 172.16.0.0/12
- 192.168.0.0/16
- 169.254.0.0/16
- 100.64.0.0/10
- ipBlock:
cidr: "::/0"
except:
- "fc00::/7"
- "fe80::/10"
ports:
- protocol: TCP
port: 443

View File

@@ -49,6 +49,8 @@ kubectl apply -f manifests/00-prereqs.yaml
kubectl -n "$NS" delete secret dsh-provider-keys --ignore-not-found >/dev/null kubectl -n "$NS" delete secret dsh-provider-keys --ignore-not-found >/dev/null
kubectl -n "$NS" create secret generic dsh-provider-keys \ kubectl -n "$NS" create secret generic dsh-provider-keys \
--from-literal=DEEPSEEK_API_KEY="sk-test-$(openssl rand -hex 16)" --from-literal=DEEPSEEK_API_KEY="sk-test-$(openssl rand -hex 16)"
# dsh-provider-keys-extra is intentionally absent: the template's optional
# envFrom reference must not block the warm pool.
kubectl apply -f manifests/10-sandboxtemplate.yaml -f manifests/20-sandboxwarmpool.yaml kubectl apply -f manifests/10-sandboxtemplate.yaml -f manifests/20-sandboxwarmpool.yaml
for _ in $(seq 1 120); do for _ in $(seq 1 120); do
READY=$(kubectl -n "$NS" get sandboxes -l agents.x-k8s.io/warm-pool-sandbox \ READY=$(kubectl -n "$NS" get sandboxes -l agents.x-k8s.io/warm-pool-sandbox \

View File

@@ -25,19 +25,28 @@ This example deliberately has no gateway: you reach DSH through a
``` ```
- A DeepSeek API key for real model calls. The pod starts without one; only - A DeepSeek API key for real model calls. The pod starts without one; only
model requests fail. model requests fail. Optionally create `dsh-provider-keys-extra` for
OpenRouter, Gemini, NVIDIA or other provider keys (the template references it
with `optional: true`).
## Quickstart ## Quickstart
Run these from this example's directory (`examples/minimal-single-tenant/`). Run these from this example's directory (`examples/minimal-single-tenant/`).
```bash ```bash
# 1. Namespace + shared provider key. The key is injected into the sandbox via # 1. Namespace + shared provider keys. They are injected into the sandbox via
# envFrom and is never committed. # envFrom and are never committed.
kubectl create namespace dsh-demo kubectl create namespace dsh-demo
kubectl -n dsh-demo create secret generic dsh-provider-keys \ kubectl -n dsh-demo create secret generic dsh-provider-keys \
--from-literal=DEEPSEEK_API_KEY="sk-..." --from-literal=DEEPSEEK_API_KEY="sk-..."
# Optional: extra providers, injected through a second Secret. Create it before
# the warm pool so the pods are stamped with the keys.
kubectl -n dsh-demo create secret generic dsh-provider-keys-extra \
--from-literal=OPENROUTER_API_KEY="sk-or-..." \
--from-literal=GEMINI_API_KEY="..." \
--from-literal=NVIDIA_API_KEY="nvapi-..."
# 2. The template itself (image pinned in the manifest). # 2. The template itself (image pinned in the manifest).
kubectl apply -f manifests/sandbox-template.yaml kubectl apply -f manifests/sandbox-template.yaml
@@ -86,6 +95,22 @@ kubectl -n dsh-demo port-forward "pod/$SB" 18081:3080
> use on your workstation, forward to another local port (here `18081`), never > use on your workstation, forward to another local port (here `18081`), never
> `3080`. `localhost` is always in DSH's trusted-host list, so the forwarded > `3080`. `localhost` is always in DSH's trusted-host list, so the forwarded
> request is accepted. > request is accepted.
>
> **gVisor caveat:** if you uncomment `runtimeClassName: gvisor` in the
> template, `kubectl port-forward` no longer works. Use the multi-tenant gateway
> example or a reachable Service instead; see
> [optional hardening](../dsh-agents-as-a-service/docs/optional-hardening.md).
## Network policy and optional hardening
The template omits `networkPolicyManagement`, so the agent-sandbox controller
uses its Managed secure default: ingress only from the sandbox-router, public
egress, and no internal cluster DNS. `kubectl port-forward` is local to the pod
and still works (except under gVisor).
gVisor, an optional `nodeSelector`, extra provider keys and a stricter sandbox
NetworkPolicy are documented in
[optional hardening](../dsh-agents-as-a-service/docs/optional-hardening.md).
## Probes and trusted hosts ## Probes and trusted hosts

View File

@@ -10,7 +10,10 @@
# - Namespace dsh-demo. # - Namespace dsh-demo.
# - Secret dsh-provider-keys in dsh-demo with key DEEPSEEK_API_KEY # - Secret dsh-provider-keys in dsh-demo with key DEEPSEEK_API_KEY
# (injected via envFrom; the pod starts without it, only model calls fail). # (injected via envFrom; the pod starts without it, only model calls fail).
# The agent-sandbox controller (v1.0.7, API v1beta1) must be installed. # - Optional: Secret dsh-provider-keys-extra with additional provider keys
# (OPENROUTER_API_KEY, GEMINI_API_KEY, NVIDIA_API_KEY, ...). It is
# referenced with `optional: true`, so it may be absent.
# The agent-sandbox controller (v1.0.8, API v1beta1) must be installed.
apiVersion: extensions.agents.x-k8s.io/v1beta1 apiVersion: extensions.agents.x-k8s.io/v1beta1
kind: SandboxTemplate kind: SandboxTemplate
metadata: metadata:
@@ -18,8 +21,26 @@ metadata:
namespace: dsh-demo namespace: dsh-demo
spec: spec:
service: true # headless Service per Sandbox -> stable DNS (status.serviceFQDN) service: true # headless Service per Sandbox -> stable DNS (status.serviceFQDN)
# networkPolicyManagement is omitted, so the controller uses its Managed
# secure default (ingress only from the sandbox-router, public egress). This
# quickstart reaches the sandbox through `kubectl port-forward`, which is
# local to the pod. The multi-tenant example documents opt-in policies in
# examples/dsh-agents-as-a-service/docs/optional-hardening.md.
#
# OPTIONAL: recreate a Failed sandbox Pod instead of retaining it
# (agent-sandbox v1.0.8+). Documented with the multi-tenant example.
# failedPodPolicy: Recreate
podTemplate: podTemplate:
spec: spec:
# OPTIONAL: run under gVisor. gVisor breaks `kubectl port-forward`, so
# this quickstart's only access path will not work; use the multi-tenant
# gateway example or a Service/NodePort instead.
# runtimeClassName: gvisor
#
# OPTIONAL: pin to a dedicated node pool. Replace the placeholder with a
# node label; add a matching toleration if the pool is tainted.
# nodeSelector:
# dsh.example.com/node-pool: dsh
securityContext: securityContext:
runAsNonRoot: true runAsNonRoot: true
runAsUser: 1000 # Matches the UID of the 'node' user runAsUser: 1000 # Matches the UID of the 'node' user
@@ -35,7 +56,7 @@ spec:
fsGroupChangePolicy: OnRootMismatch fsGroupChangePolicy: OnRootMismatch
containers: containers:
- name: dsh-web - name: dsh-web
image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.3-r3 image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.5
imagePullPolicy: IfNotPresent imagePullPolicy: IfNotPresent
ports: ports:
- name: web - name: web
@@ -43,6 +64,10 @@ spec:
envFrom: envFrom:
- secretRef: - secretRef:
name: dsh-provider-keys # Secret with key DEEPSEEK_API_KEY name: dsh-provider-keys # Secret with key DEEPSEEK_API_KEY
# Optional extra provider keys; an absent Secret is safe.
- secretRef:
name: dsh-provider-keys-extra
optional: true
env: env:
# Empty => DSH derives links from the request Host. Keep it empty # Empty => DSH derives links from the request Host. Keep it empty
# when fronting the sandbox with a proxy/gateway. # when fronting the sandbox with a proxy/gateway.

View File

@@ -29,6 +29,10 @@ for dir in examples/*/; do
err "$name: manifests/exposure/ requires a README.md describing the variants" err "$name: manifests/exposure/ requires a README.md describing the variants"
fi fi
if [ -d "${dir}manifests/optional" ] && [ ! -f "${dir}manifests/optional/README.md" ]; then
err "$name: manifests/optional/ requires a README.md describing the opt-in manifests"
fi
for component in "$dir"*/; do for component in "$dir"*/; do
[ -d "$component" ] || continue [ -d "$component" ] || continue
case "$(basename "$component")" in case "$(basename "$component")" in

View File

@@ -3,7 +3,7 @@
# "make check". # "make check".
# #
# ./scripts/set-version.sh dsh-v0.2.2-rc.1 # new DSH ref, no revision # ./scripts/set-version.sh dsh-v0.2.2-rc.1 # new DSH ref, no revision
# ./scripts/set-version.sh dsh-v0.2.1-rc.3 r3 # same ref, new repo revision # ./scripts/set-version.sh dsh-v0.2.1-rc.5 r3 # same ref, new repo revision
set -euo pipefail set -euo pipefail
cd "$(dirname "$0")/.." cd "$(dirname "$0")/.."

View File

@@ -5,18 +5,18 @@
# Upstream DeepSeek Harness source, and the ref this repository packages. # Upstream DeepSeek Harness source, and the ref this repository packages.
DSH_REPO=https://github.com/deepseek-ai/deepseek-harness.git DSH_REPO=https://github.com/deepseek-ai/deepseek-harness.git
DSH_VERSION=dsh-v0.2.1-rc.3 DSH_VERSION=dsh-v0.2.1-rc.5
PNPM_VERSION=11.28.5 PNPM_VERSION=11.28.5
# Release identity = DSH_VERSION plus an optional repository revision. # Release identity = DSH_VERSION plus an optional repository revision.
# An empty RELEASE_REVISION is the first release at this DSH ref; rN (N >= 2) # An empty RELEASE_REVISION is the first release at this DSH ref; rN (N >= 2)
# marks a repository-only re-release at the same DSH ref. The dsh-v0.2.1-rc.3 # marks a repository-only re-release at the same DSH ref. This is the first
# image tag is already published on GHCR and tags are immutable, so the first # release at dsh-v0.2.1-rc.5, so the identity is the plain tag; a later
# release under the current scheme is dsh-v0.2.1-rc.3-r3. # repository-only change at the same ref becomes -r2.
RELEASE_REVISION=r3 RELEASE_REVISION=
# Control-plane versions used by the examples and their manifests. # Control-plane versions used by the examples and their manifests.
AGENT_SANDBOX_VERSION=v1.0.7 AGENT_SANDBOX_VERSION=v1.0.8
GATEWAY_API_VERSION=v1.2.1 GATEWAY_API_VERSION=v1.2.1
OAUTH2_PROXY_IMAGE=quay.io/oauth2-proxy/oauth2-proxy:v7.15.5 OAUTH2_PROXY_IMAGE=quay.io/oauth2-proxy/oauth2-proxy:v7.15.5