1
0
mirror of https://github.com/sredevopsorg/deepseek-kubernetes-sandbox.git synced 2026-11-03 01:13:24 +00:00

23 Commits

Author SHA1 Message Date
SREDevOps.org Agent
bed8e5675a chore(release): dsh-v0.2.1-rc.5 (#29)
Bump DSH_VERSION to dsh-v0.2.1-rc.5 and clear RELEASE_REVISION, making the release identity dsh-v0.2.1-rc.5. Update every derived manifest, image tag, and doc, and remove prose that described the previous identity's legacy-tag history.

Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
2026-10-30 04:54:37 -03:00
SREDevOps.org Agent
14c3fb840e feat(example): add optional failed-Pod recovery (#26)
agent-sandbox v1.0.8 adds spec.failedPodPolicy to Sandbox and
SandboxTemplate. The default, Retain, leaves a Failed Pod in place and
surfaces Finished=True; Recreate deletes it and starts a replacement with
the same Sandbox identity and PVCs.

Ship it as an opt-in toggle next to the other commented fields, with a
new section 5 in the optional-hardening doc: what it changes, how to
enable it for new, warm-pool and already-claimed Sandboxes (the policy is
stamped at creation), a failure drill, caveats and rollback. The base
walkthrough stays on Retain because a deterministic failure would
otherwise recreate forever, and because the templates use
restartPolicy OnFailure, so the policy only matters for terminal cases
such as node eviction.

Cross-references updated in the example README, the optional-manifests
index, the architecture failure-mode table and the platform skill's
troubleshooting table.

Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
2026-10-26 14:29:01 -03:00
SREDevOps.org Agent
96d3030a49 chore(deps): bump agent-sandbox to v1.0.8 (#25)
Update AGENT_SANDBOX_VERSION to v1.0.8 and the docs that quote the
control-plane pin (example prerequisites, architecture, versioning table,
minimal-single-tenant template comment, platform skill).

The v1.0.8 release adds only one CRD field, spec.failedPodPolicy
(Sandbox and SandboxTemplate, enum Retain|Recreate, default Retain), so
the existing manifests keep their behavior: Retain is the default and
neither the API version (v1beta1) nor any field we set changed. All five
custom resources still validate against the v1.0.8 CRD schemas.

Behavior changes in the controller itself (bounded MCP output, router
kustomization, creation-latency metric timing) do not affect the
platform because it uses neither the sandbox-router nor the MCP server.

Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
2026-10-26 14:17:28 -03:00
SREDevOps.org Agent
2b43d5dfd5 fix(ci): make release image digests unique and allow changelog PRs (#24)
The reusable image build ran once for the main image and once for the gateway in the same release run, but both used digests-<platform> artifact names, so the main merge downloaded a gateway digest and failed. Include a short hash of the full image name in the artifact name and download pattern.

The publish-release job also lacked pull-requests: write, so the generated CHANGELOG PR failed with 'Resource not accessible by integration'. Grant the job contents and pull-requests write.

Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
2026-10-26 10:34:21 -03:00
SREDevOps.org Agent
97abf75dfe chore(changelog): refresh for dsh-v0.2.1-rc.3-r4 (#23)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-10-26 10:27:02 -03:00
SREDevOps.org Agent
e2f1fe567a chore(release): dsh-v0.2.1-rc.3-r4 (#22)
Bump RELEASE_REVISION to r4 for the optional-hardening feature release and refresh every consuming manifest/doc. CHANGELOG.md is generated by the release workflow and is intentionally left untouched.

Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
2026-10-26 10:17:43 -03:00
SREDevOps.org Agent
3e9ba38fc1 feat(example): add optional gVisor, nodeSelector, NetworkPolicy and provider keys (#21)
Add opt-in hardening to the agents-as-a-service example:
- commented runtimeClassName/nodeSelector fields for sandboxes and Deployments
- optional sandbox NetworkPolicy plus exposure-variant policies
- optional dsh-provider-keys-extra Secret for extra provider keys
- docs/optional-hardening.md with verification, caveats and rollback

The base walkthrough stays portable; no gateway code, routes or auth changed.

Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
2026-10-26 10:03:30 -03:00
Nicolás Georger
bf24306421 feat(skills): add agent-sandbox platform skill (#20)
Add the project-local dsh-agent-sandbox-platform skill covering platform operations, the optional agent-sandbox MCP server, feature-extension recipes, and a read-only preflight. Document project skills in README, docs and AGENTS.md.

Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
2026-10-26 08:10:59 -03:00
Nicolás Georger
8681412be4 chore(changelog): refresh for dsh-v0.2.1-rc.3-r3 (#19)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-10-25 19:42:02 -03:00
Nicolás Georger
958cbe25a5 Refactor release workflow for environment variable sourcing
Updated the release workflow to source environment variables from versions.env and modified the tag comparison logic.

Signed-off-by: Nicolás Georger <info@sredevops.org>
2026-10-25 19:19:26 -03:00
Nicolás Georger
027f74f8e9 Refactor environment variable loading in build-gateway.yaml
Updated the environment variable loading method in the GitHub Actions workflow.

Signed-off-by: Nicolás Georger <info@sredevops.org>
2026-10-25 19:03:16 -03:00
Nicolás Georger
d627526e3e Refactor environment variable loading in build workflow
Update the script to read environment variables from versions.env using 'source' instead of a loop.

Signed-off-by: Nicolás Georger <info@sredevops.org>
2026-10-25 18:56:53 -03:00
Nicolás Georger
ecba8b96a6 Versioning update (#18)
Signed-off-by: Nicolás Georger <info@sredevops.org>
2026-10-25 18:46:09 -03:00
dependabot[bot]
6a8fe54abc chore: bump the actions group across 1 directory with 3 updates (#17)
Bumps the actions group with 3 updates in the / directory: [actions/setup-python](https://github.com/actions/setup-python), [actions/dependency-review-action](https://github.com/actions/dependency-review-action) and [amannn/action-semantic-pull-request](https://github.com/amannn/action-semantic-pull-request).


Updates `actions/setup-python` from 7.0.0 to 7.1.0
- [Release notes](https://github.com/actions/setup-python/releases)
- [Commits](5fda3b95a4...6d4fd22b20)

Updates `actions/dependency-review-action` from 4.9.0 to 5.0.0
- [Release notes](https://github.com/actions/dependency-review-action/releases)
- [Commits](2031cfc080...a1d282b36b)

Updates `amannn/action-semantic-pull-request` from 5 to 6
- [Release notes](https://github.com/amannn/action-semantic-pull-request/releases)
- [Changelog](https://github.com/amannn/action-semantic-pull-request/blob/main/CHANGELOG.md)
- [Commits](e32d7e603d...48f256284b)

---
updated-dependencies:
- dependency-name: actions/dependency-review-action
  dependency-version: 5.0.0
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: actions
- dependency-name: actions/setup-python
  dependency-version: 7.1.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: actions
- dependency-name: amannn/action-semantic-pull-request
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: actions
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-25 14:42:29 -03:00
Nicolás Georger
6876871110 Add commit message prefix for Dependabot updates
Signed-off-by: Nicolás Georger <info@sredevops.org>
2026-10-25 14:37:11 -03:00
Nicolás Georger
3b70f179ac feat(repo)!: repo reorganization (#16)
* feat(repo)!: restructure around the dsh-harness image and one example per implementation

- the root Dockerfile is the primary artifact; versions.env is the single source of truth
- merge gateway-api/dsh-demo into examples/dsh-agents-as-a-service (manifests/, exposure variants)
- move the single-tenant template into examples/minimal-single-tenant
- publish the gateway as a nested package under the repository image
- mirror the DSH ref for release identities (first release dsh-v0.2.1-rc.2-r2)
- CI: lint/python/manifests/gateway-image, reusable multi-arch build, tag-driven release, weekly e2e
- docs: README, docs/{main-image,examples,gitflow,versioning}, example READMEs, AGENTS.md, governance
- SSO is opt-in again; gateway pull policy and cookie defaults match the documented demo

BREAKING CHANGE: example manifest paths, the gateway image name, and the removal
of deploy/ and gateway-api/ affect existing deployments; see the migration table
in README.md.

* docs(changelog): generate initial changelog from history

* fix(scripts): make set-version update Dockerfile and sample revision pins

* build(hadolint): document intentional lint exceptions for the main image

---------

Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
2026-10-25 08:37:43 -03:00
Nicolás Georger
53e55fc851 Modify gateway.yaml for image policy and timeout values
Updated imagePullPolicy to always pull the latest image and increased timeout values for IDLE_TIMEOUT and WAKE_TIMEOUT.

Signed-off-by: Nicolás Georger <info@sredevops.org>
2026-10-25 04:41:00 -03:00
SREDevOps.org Agent
ed1398dfd1 feat(gateway): SSO landing auto-connects and offers recreate sandbox (#15)
After oauth2-proxy authentication, the gateway now renders a minimal SSO
landing (no signup/token forms) that auto-connects to the sandbox and offers
a single 'Recreate sandbox' action for a clean start.

- _sso_session: stop blocking on wait_ready (provisioning is non-blocking;
  the proxy path still awaits readiness), so the landing reports
  'Provisioning' instead of stalling for WAKE_TIMEOUT
- landing: SSO branch renders a minimal page (signed-in user + sandbox state
  + auto-connect meta-refresh), replacing the generic signup/token forms
- new POST /recreate: deletes the claim (cascade pod+PVC), waits out
  finalizers, re-provisions, and reconnects; CSRF-protected, SSO-only
- tests: 109 passing (SSO landing provision/connect/recreate cases)

Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
2026-10-25 04:34:13 -03:00
SREDevOps.org Agent
73a65d98d2 fix(gateway-api): correct oauth2-proxy cookie-secret generation (#14)
oauth2-proxy base64-decodes cookie_secret and requires it to decode to 16/24/32
bytes. Plain 'openssl rand -base64 32' yields 44 chars of standard base64
(padding + possibly '+/'), which it cannot decode and treats as 44 raw bytes,
failing with 'cookie_secret must be 16, 24, or 32 bytes ... but is 44 bytes'.

Use URL-safe, unpadded base64 instead (secrets.token_urlsafe(32) or the
openssl|tr variant).

Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
2026-10-25 04:12:56 -03:00
SREDevOps.org Agent
3f46279b39 feat(gateway-api): GitHub SSO via oauth2-proxy in front of dsh-gateway (#13)
Add oauth2-proxy (GitHub provider, org-restricted) as a reverse proxy in
front of the dsh-gateway, wired through the Traefik Gateway API. The gateway
inherits the authenticated GitHub username via X-Forwarded-User (SSO).

- gateway.py: opt-in AUTH_PROXY_HEADER trust, remote_user(), SSO bootstrap
  (auto-provision claim + mint/rotate token + session cookie), patch_token()
- 50-gateway.yaml: AUTH_PROXY_HEADER=X-Forwarded-User, COOKIE_SECURE=true
- gateway-api/dsh-demo: oauth2-proxy Deployment/Service + secret example,
  HTTPRoute backend -> oauth2-proxy:4180
- tests: 10 new SSO cases (106 passing)

Placeholders (dsh-demo.example.com, your-github-org) are substituted at
apply time.

Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
2026-10-25 03:43:32 -03:00
Nicolás Georger
2b47afa22b Dockerfile extra tools
Signed-off-by: Nicolás Georger <info@sredevops.org>
2026-10-25 01:39:03 -03:00
Nicolás Georger
6d97f6193b fix(Dockerfile): update runtime dependencies and enable corepack for pnpm 2026-10-21 23:16:12 -03:00
SREDevOps.org Agent
7d3faf07dd feat(gateway-api): expose dsh-gateway externally via Traefik Gateway API (#12)
Add Gateway and HTTPRoute manifests to route external traffic to the
dsh-gateway service (dsh-demo:8080).

- Gateway dsh-gateway: HTTPS listener with TLS termination against a
  dsh-demo-tls cert secret.
- HTTPRoute dsh-gateway: hostname -> dsh-gateway:8080.
- Example TLS secret manifest + README with apply/verify/rollback.

Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
2026-10-21 18:58:38 -03:00
81 changed files with 6713 additions and 1538 deletions

View File

@@ -1,149 +1,6 @@
# Logs
logs
*.log
npm-debug.log*
yarn-debug.log*
yarn-error.log*
lerna-debug.log*
# Diagnostic reports (https://nodejs.org/api/report.html)
report.[0-9]*.[0-9]*.[0-9]*.[0-9]*.json
# Runtime data
pids
*.pid
*.seed
*.pid.lock
# Directory for instrumented libs generated by jscoverage/JSCover
lib-cov
# Coverage directory used by tools like istanbul
coverage
*.lcov
# nyc test coverage
.nyc_output
# Grunt intermediate storage (https://gruntjs.com/creating-plugins#storing-task-files)
.grunt
# Bower dependency directory (https://bower.io/)
bower_components
# node-waf configuration
.lock-wscript
# Compiled binary addons (https://nodejs.org/api/addons.html)
build/Release
# Dependency directories
node_modules/
jspm_packages/
# Snowpack dependency directory (https://snowpack.dev/)
web_modules/
# TypeScript cache
*.tsbuildinfo
# Optional npm cache directory
.npm
# Optional eslint cache
.eslintcache
# Optional stylelint cache
.stylelintcache
# Optional REPL history
.node_repl_history
# Output of 'npm pack'
*.tgz
# Yarn Integrity file
.yarn-integrity
# dotenv environment variable files
.env
.env.*
!.env.example
# parcel-bundler cache (https://parceljs.org/)
.cache
.parcel-cache
# Next.js build output
.next
out
# Nuxt.js build / generate output
.nuxt
dist
.output
# Gatsby files
.cache/
# Comment in the public line in if your project uses Gatsby and not Next.js
# https://nextjs.org/blog/next-9-1#public-directory-support
# public
# vuepress build output
.vuepress/dist
# vuepress v2.x temp directory
.temp
# Sveltekit cache directory
.svelte-kit/
# vitepress build output
**/.vitepress/dist
# vitepress cache directory
**/.vitepress/cache
# Docusaurus cache and generated files
.docusaurus
# Serverless directories
.serverless/
# FuseBox cache
.fusebox/
# DynamoDB Local files
.dynamodb/
# Firebase cache directory
.firebase/
# TernJS port file
.tern-port
# Stores Visual Studio Code versions used for testing Visual Studio Code extensions
.vscode-test
# pnpm
.pnpm-store
# yarn v3
.pnp.*
.yarn/*
!.yarn/patches
!.yarn/plugins
!.yarn/releases
!.yarn/sdks
!.yarn/versions
# Vite files
vite.config.js.timestamp-*
vite.config.ts.timestamp-*
.vite/
# Astro generated types
.astro/
# Turbo Repo
.turbo
# The main image build clones DeepSeek Harness from git and copies nothing
# from this repository, so the build context is intentionally reduced to the
# Dockerfile alone. This keeps builds fast and prevents the rest of the repo
# (examples, docs, git history) from being sent to the builder.
*
!Dockerfile

View File

@@ -0,0 +1,156 @@
---
name: dsh-agent-sandbox-platform
description: >-
Operate and extend this repository's DeepSeek Harness-on-agent-sandbox platform:
warm pools, claims, sandbox suspend/resume, the gateway, and the optional
agent-sandbox MCP server, with version-aware safety and a read-only preflight.
whenToUse: >-
Use when working in deepseek-kubernetes-sandbox or with Kubernetes agent-sandbox:
inspect or provision sandboxes, debug warm pools or claims, use
mcp__*__create_sandbox/execute_command, connect the agent-sandbox MCP server, or
add a manifest, gateway route, exposure variant, example, or version bump here.
metadata:
author: sredevopsorg
version: "1.0.0"
---
# DSH agent-sandbox platform
This repository runs DeepSeek Harness Web (DSH) as one sandbox per user on
[agent-sandbox](https://github.com/kubernetes-sigs/agent-sandbox), with an
optional official MCP server for sandbox execution. For generic cluster work
load `kubernetes-kubectl-operations`; for profile MCP rows load
`cordis-plugin-development`.
## 0. Facts before action
1. Read `AGENTS.md`, `versions.env`, the example README/architecture. The
control-plane pin is `AGENT_SANDBOX_VERSION`; release identity is
`DSH_VERSION` + `RELEASE_REVISION`.
2. Establish context, user and namespace; never assume `dsh-demo`.
3. Run `scripts/platform-preflight.sh --namespace <ns>` (read-only).
4. If `mcp__*__create_sandbox`-style tools are absent, use `kubectl`; do not
install or enable a server on your own.
## 1. Non-negotiables
- **Read-only by default.** Change a cluster only when the user asked for that
exact change; show the command and rollback first.
- **Scope every command** with `--context` and `-n`. `--all`/`-A` mutations and
deleting namespaces, PVCs, PVs, CRDs, or cluster-scoped objects need an
explicit, named confirmation.
- **Secrets and logs stay private.** Never print Secret data, kubeconfigs, or
node logs. Pod logs can contain the DSH launch token.
- **The MCP server has no authentication.** It is a full-access management
endpoint; never expose it. A DSH profile row affects every session.
- **Clean up what you create.** Delete MCP-created sandboxes in the same
session; never leave orphaned claims running.
- **Bound output.** Use `--tail`, `--since`, selectors, JSONPath, and
head/tail excerpts instead of whole manifests or huge logs.
## 2. Choose the path
| Need | Path |
| --- | --- |
| Inspect or change claims, pools, sandboxes, PVCs, gateway | `kubectl` + [platform runbook](references/platform-runbook.md) |
| Run code or move files in a sandbox | agent-sandbox MCP, only for a runtime-capable template ([MCP reference](references/mcp-server.md)) |
| Add or change a repository feature | [repo extension guide](references/extending-the-repo.md) |
The MCP server cannot manage this example's DSH claims: its tools are scoped to
claims created by the same MCP session. Use `kubectl` for the platform.
## 3. Operate the platform
Resolve user -> claim -> sandbox -> pod:
```bash
SB=$(kubectl -n "$NS" get sandboxclaim "$CLAIM" -o jsonpath='{.status.sandbox.name}')
kubectl -n "$NS" get sandbox "$SB" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}{" "}{.status.conditions[?(@.type=="Ready")].reason}{"\n"}{.status.serviceFQDN}{"\n"}'
POD=$(kubectl -n "$NS" get sandbox "$SB" -o jsonpath='{.metadata.annotations.agents\.x-k8s\.io/pod-name}'); POD=${POD:-$SB}
```
- Read runtime state from the Sandbox, not the claim: `spec.operatingMode`,
`Suspended`, `Ready`. The claim tracks binding, not liveness.
- Suspend = patch `operatingMode: Suspended` (pod deleted, PVC + Service kept);
resume = `Running` + wait for `Ready`. The gateway wakes on connect.
- Warm-pool spares carry `agents.x-k8s.io/warm-pool-sandbox`; adopted sandboxes
lose it. A pool with no spares makes signup cold and slow.
- The DSH launch token appears once in pod logs; hand the user the `?token=` URL,
never the token in reports. Probe `/favicon.svg`.
- The idle sweeper only suspends claims the gateway created.
Full commands, security boundaries and failure matrix:
[platform runbook](references/platform-runbook.md).
## 4. Use the agent-sandbox MCP server within its limits
Verified from the upstream server:
- Tools: `create_sandbox`, `delete_sandbox`, `execute_command`, `upload_file`,
`download_file`, `list_files`, `file_exists`, `get_sandbox_status`; resource
`sandboxes://{namespace}`.
- **Session-scoped:** every tool and the resource filter claims by
`K8S_SANDBOX_SESSION_ID_LABEL_KEY=<MCP session id>`. It cannot read, exec in,
or delete DSH platform claims or claims from a prior/reconnected session.
- **Streamable HTTP only:** stdio has no MCP session id and fails. A DSH row must
use `transport: streamable-http`; health endpoints are `/healthz` and `/readyz`
at the root, the MCP endpoint is `/mcp`.
- **Runtime required:** connection types target a sandbox runtime on 8888, or
sandboxd on 8080/9090. The DSH template listens only on 3080, so
`execute_command` cannot work against it. A sandboxd sidecar executes in its
own container, not DSH's.
- **Timeouts:** `create_sandbox` waits up to 180 s, but the DSH MCP client's
default call timeout is 60 s; raise `toolCallTimeoutMs`.
`shutdown_after_seconds` defaults to 300 s (`null` disables it).
- **Output:** servers before v1.0.8 return unbounded `execute_command` output.
v1.0.8 adds `max_output_chars` (128-300,000, default 30,000) and truncation
metadata. The repository pins v1.0.8 (`AGENT_SANDBOX_VERSION`), so the
documented deployment is bounded; never pass the new argument to an older
server, and on one redirect verbose output and read a file instead.
Deployment, connection matrix and DSH row:
[MCP deployment](references/mcp-deployment.md). Limits and cleanup:
[MCP reference](references/mcp-server.md).
## 5. Extend the project
Read `docs/examples.md` and the testing table in `AGENTS.md` before editing.
- **Manifest step:** numbered `manifests/NN-*.yaml`, update the README step, run
`make check`; CI dry-runs it.
- **Gateway:** edit `gateway/gateway.py` and `gateway/test_gateway.py`, update
the gateway README API table and architecture doc, run `make test` and
`make lint`. Keep the HTTP/WebSocket contract stable.
- **Exposure variant:** one variant under `manifests/exposure/<variant>/`,
update the exposure README and trusted-host/cookie settings. Variants are
alternatives, never a stack.
- **New example:** follow `docs/examples.md`, add the index entry, keep scripts
executable/idempotent/self-cleaning, run `make check`.
- **Version bump:** edit `versions.env` only, update prerequisites, run
`make check` and the e2e. Never mix a bump with an unrelated feature.
- **MCP integration:** pin the build to the cluster's tag, least-privilege SA,
ClusterIP + auth proxy, raised DSH timeout, no secrets in the repo.
- **Optional hardening:** gVisor, `nodeSelector`, NetworkPolicy and extra
provider keys stay opt-in; the sandbox policy lives under
`manifests/optional/`. All four are documented in
`examples/dsh-agents-as-a-service/docs/optional-hardening.md`.
- **Docs:** update the affected doc in the same change.
Details and review checklist:
[repo extension guide](references/extending-the-repo.md).
## 6. Report and clean up
State context/namespace, bounded evidence, conclusion, exact change, rollback,
and resources deleted. Disable any MCP row enabled only for the task. Do not
leave sandboxes, port-forwards, or temporary files behind.
## References
- [Platform runbook](references/platform-runbook.md)
- [MCP deployment](references/mcp-deployment.md)
- [MCP server reference](references/mcp-server.md)
- [Repo extension guide](references/extending-the-repo.md)
- [Troubleshooting](references/troubleshooting.md)
- Upstream MCP server README:
https://github.com/kubernetes-sigs/agent-sandbox/blob/main/clients/integrations/mcp-server/README.md

View File

@@ -0,0 +1,132 @@
# Extending the repository
Read `AGENTS.md`, `docs/examples.md`, and the example README before editing.
This repository is image-first: the Dockerfile is the primary artifact, every
implementation is a self-contained example under `examples/<name>/`, and
`versions.env` is the only source of pins. Documentation is part of the change.
## Invariants
- Never hard-code a version, image tag, or registry path; edit `versions.env`.
For the DSH release identity use `make set-version`, never ad-hoc `sed`.
- Never commit a secret. Secret-shaped manifests are `*.yaml.example` with
placeholders; keys are documented only in `.env.sample`.
- Manifests are numbered `NN-*.yaml` and apply in order with plain `kubectl`.
Exposure variants live under `manifests/exposure/<variant>/`, are documented
by one README, and are mutually exclusive.
- Keep the gateway HTTP/WebSocket contract stable unless the change is explicitly
about it.
- Do not edit `CHANGELOG.md` (generated by git-cliff) or rewrite git history.
- Keep diffs scoped; do not mix a version bump with an unrelated feature.
## Change matrix
| Change | Edit | Minimum verification |
| --- | --- | --- |
| Documentation only | `*.md`, `docs/` | `make check`; relative links resolve |
| Example manifest | `examples/<name>/manifests/NN-*.yaml`, README step | `make check`; server-side dry-run in CI |
| Optional hardening | `manifests/optional/`, exposure NPs, example `docs/optional-hardening.md` | `make check`; manual verification on a capable cluster |
| Gateway behavior/route | `gateway/gateway.py`, `gateway/test_gateway.py`, gateway README, architecture | `make test`, `make lint`; update tests and API docs |
| Exposure variant | `manifests/exposure/<variant>/`, exposure README, example README | `make check`; apply exactly one variant |
| New example | full `examples/<name>/` layout, `examples/README.md` | `make check`; CI dry-run; optional e2e |
| Version bump | `versions.env`, affected docs/manifests | `make check`; CI installs the pin; e2e on a cluster |
| MCP integration | deployment/RBAC, DSH row, docs | read-only `/readyz`; no secret in repo |
| Workflow | `.github/workflows/` | actionlint; SHA-pinned actions; no hard-coded versions |
## Recipes
### Add or change a resource step
1. Keep numbering stable; add `NN-*.yaml` at the end unless order matters.
2. Update the matching numbered step in the example README.
3. Keep cluster-specific state (node affinity, tuned replica counts) out of the
shipped manifest; document it as a local patch.
4. Run `make check`. CI additionally parses every manifest and runs a
server-side dry-run against the agent-sandbox CRDs.
### Change the gateway
The gateway is a single-file Starlette app. A change usually touches:
- `gateway/gateway.py` (route, auth, proxy, sweeper);
- `gateway/test_gateway.py` (unit tests; no cluster required);
- `gateway/README.md` (API table, configuration, auth semantics);
- `examples/dsh-agents-as-a-service/docs/architecture.md` when flows change;
- `examples/dsh-agents-as-a-service/README.md` when the walkthrough changes.
Run `make test` and `make lint`. Preserve session-cookie auth, same-origin rules
for cookie mutations and WebSocket upgrades, and the "waking"/`502`/`503`
semantics unless the change is explicitly about them. Do not expose a trusted
identity header unless the Service is reachable only through the proxy.
### Add an exposure variant
Create `manifests/exposure/<variant>/NN-*.yaml`, update
`manifests/exposure/README.md` and the example README, and state the trust
boundary. Variants are alternatives: never apply two paths to the same gateway.
Set `DSH_TRUSTED_HOSTS` in the template for the public host and
`COOKIE_SECURE=true` for HTTPS.
### Add an example
Follow `docs/examples.md`: `README.md`, `manifests/NN-*.yaml`, optional
`manifests/exposure/` with a README, optional components with Dockerfile +
README + tests, and executable, idempotent, self-cleaning scripts. Add the
literal `examples/<name>/` entry to `examples/README.md`; `make check` enforces
the layout. Publish component images under
`<registry>/<owner>/<repo>/<example>/<component>`.
### Bump a version
- **DSH release identity:** `make set-version DSH_VERSION=<ref>
[RELEASE_REVISION=rN]`, then `make check`, then build and smoke-test the image.
Tags are immutable; fix a bad release with `-r(N+1)`.
- **agent-sandbox:** edit `AGENT_SANDBOX_VERSION` in `versions.env`, update the
prerequisites in the example README, and note behavior changes (connection
types, MCP output limits, CRD fields). Install the new release YAML on a kind
cluster, then run `make check` and `make e2e`. Do not bump it inside an
unrelated PR.
### Integrate the agent-sandbox MCP server
Build from the cluster's pinned tag, give it a dedicated namespace and
least-privilege ServiceAccount, keep it ClusterIP behind an authenticating
proxy, and document how to reach `/readyz`. Add a DSH profile row only with user
approval; a row affects every session. Never commit its credentials or a
`cluster-admin` binding.
### Documentation
Update the doc whose behavior changed in the same commit: root README for the
front door, `docs/` for repository-wide contracts, the example README for that
walkthrough, the gateway README for API/config/auth. Skill docs live in
`.dsh/skills/`; keep `SKILL.md` under ~8 KB and put depth in `references/`.
`scripts/check-versions.sh` scans every `*.md` for stale `dsh-v…` tokens, so
refer to "the release identity in `versions.env`" rather than copying a tag.
## Verification and review
Run at least `make check`; add `make test` and `make lint` for gateway/script
changes. For manifests, use `kubectl apply --dry-run=server` against a cluster
with the CRDs. For behavior changes, run the example or `make e2e`.
Before reporting, review:
- [ ] Diff is scoped; no generated files or unrelated changes.
- [ ] Docs and examples match the new behavior.
- [ ] Pins come only from `versions.env`; no stale version tokens.
- [ ] No secrets or usable credentials; secret-shaped files are `.example`.
- [ ] Manifests parse; exposure variants remain mutually exclusive.
- [ ] Gateway tests and public contract unchanged or intentionally updated.
- [ ] Skills: frontmatter valid, body within budget, scripts shellcheck-clean.
- [ ] `CHANGELOG.md` untouched; branch/commit follow `docs/gitflow.md`.
## Safe follow-ups enabled by this skill
- A read-only preflight job or scheduled report using
`scripts/platform-preflight.sh`.
- An MCP readiness check in the example e2e (health endpoints only).
- A separate runtime warm pool for untrusted execution, leaving the DSH template
unchanged.
- A combined DSH + sandboxd image only with an explicit maintenance/exec design
and a documented trust boundary.

View File

@@ -0,0 +1,94 @@
# Deploying the agent-sandbox MCP server
Read [the MCP reference](mcp-server.md) first. This file covers the connection
matrix, a safe deployment shape, and the DSH profile row. The server is
unauthenticated; its service account is the security boundary.
## Connection matrix
| `K8S_SANDBOX_CONNECTION__TYPE` | Required env | Use when |
| --- | --- | --- |
| `gateway` | `GATEWAY_NAME`, `GATEWAY_NAMESPACE=default`, `GATEWAY_READY_TIMEOUT=180`, `SERVER_PORT=8888` | Gateway API + implementation exist |
| `direct` | `API_URL` (router URL), `SERVER_PORT=8888` | sandbox-router is deployed |
| `in-cluster` | `USE_POD_IP=false` (DNS) or `true` (`status.podIPs`), `SERVER_PORT=8888` | Bypass the router from inside the cluster |
| `sandboxd-in-cluster` (v1.0.8+) | `MODE=in-cluster-service` or `in-cluster-pod-ip`, `REST_PORT=8080`, `GRPC_PORT=9090` | Template runs sandboxd |
Also set `K8S_SANDBOX_PROBE_NAMESPACE` to a namespace the server's service
account can list claims in, and optionally
`K8S_SANDBOX_SESSION_ID_LABEL_KEY` (default
`mcp.k8s-agent-sandbox/session-id`).
For `sandboxd-in-cluster` service mode the template needs `spec.service: true`.
The default managed NetworkPolicy admits only sandbox-router, so add an ingress
rule for the MCP server on 8080/9090. Supplying `spec.networkPolicy` replaces the
secure defaults: also add the egress rules the sandbox needs, because an omitted
egress list denies all outbound traffic. RBAC for that mode is
`sandboxclaims` create/get/list/watch/delete and `sandboxes` get/list/watch; no
`pods/portforward`.
## Build and deploy (sketch)
No official image reference is documented in the upstream README; pin what you
build.
```bash
# AGENT_SANDBOX_VERSION comes from versions.env; never hard-code it.
tmp="$(mktemp -d)"; trap 'rm -rf "$tmp"' EXIT
git clone --depth 1 --branch "$AGENT_SANDBOX_VERSION" \
https://github.com/kubernetes-sigs/agent-sandbox.git "$tmp"
docker build -t "$MCP_IMAGE" "$tmp/clients/integrations/mcp-server"
```
Kubernetes shape:
- Dedicated namespace and ServiceAccount; a namespace-scoped Role granting only
`sandboxclaims` create/get/list/watch/delete and `sandboxes` get/list/watch.
No Secrets, no `pods/exec`, no cluster-admin.
- Deployment: one replica, non-root, `readOnlyRootFilesystem`, dropped
capabilities, `K8S_SANDBOX_CONNECTION__TYPE=in-cluster`,
`K8S_SANDBOX_PROBE_NAMESPACE`, resource limits.
- Probes: `GET /healthz` (liveness; never calls the API) and `GET /readyz`
(readiness; lists claims; use `periodSeconds: 15`, not aggressive polling).
- ClusterIP Service on `8000`. Add a NetworkPolicy that allows only the intended
client namespace/pod. Never expose it through Ingress/LoadBalancer without an
authenticating proxy. The service account is the real boundary.
## Connect it to DSH
Add a profile MCP row (through the Plugins page/`plugin_manager`; see
`cordis-plugin-development`). A row affects every session and needs approval.
```yaml
- insert:
- id: mcp-agent-sandbox
name: '@deepseek-ai/dsh-mcp-client'
config:
serverName: agent-sandbox
transport: streamable-http
url: http://agent-sandbox-mcp.example.svc.cluster.local:8000/mcp
toolCallTimeoutMs: 240000
failOnStartupError: false
```
- `serverName` must match `[A-Za-z0-9_-]{1,32}`; tools appear as
`mcp__<serverName>__<tool>`. Check the actual names in the tool list; do not
assume hyphens/underscores.
- Raise `toolCallTimeoutMs` above `sandbox_ready_timeout` with headroom; the DSH
default is 60 s and would cancel a 180 s creation.
- If the server is behind an authenticating proxy, pass its token via the row's
`headers` and an environment expression; never commit it.
- To reach a locally port-forwarded server, use a loopback URL; the MCP session
still needs streamable HTTP.
- Remove the row when it was enabled only for one task.
## Deployment safety checklist
- [ ] Build tag equals the cluster's `AGENT_SANDBOX_VERSION`.
- [ ] Dedicated namespace/SA; RBAC limited to claims and sandboxes.
- [ ] Service is ClusterIP; no public route without authentication.
- [ ] NetworkPolicy admits only the DSH client and needed sandbox traffic.
- [ ] `/readyz` succeeds with `K8S_SANDBOX_PROBE_NAMESPACE` set to a listable
namespace.
- [ ] No Secret data in manifests, logs, or the repository.
- [ ] Cleanup path documented: delete session-owned claims or wait for the
shutdown timeout.

View File

@@ -0,0 +1,107 @@
# agent-sandbox MCP server reference
Upstream: [clients/integrations/mcp-server](https://github.com/kubernetes-sigs/agent-sandbox/tree/main/clients/integrations/mcp-server).
It is a Python/FastAPI + FastMCP service (streamable HTTP on `:8000`, MCP
endpoint `/mcp`) that provisions sandboxes from a warm pool and runs commands or
file operations through a sandbox runtime. It ships with **no authentication**.
Build the server from the same tag as the cluster: read `AGENT_SANDBOX_VERSION`
in `versions.env`, and check the installed controller image. Do not build from
`main` against an older cluster; tool schemas and connection types differ.
## Tool surface
| Tool | Arguments (defaults) | Returns / notes |
| --- | --- | --- |
| `create_sandbox` | `warmpool`, `namespace`, `sandbox_ready_timeout=180`, `labels`, `shutdown_after_seconds=300` (`null` disables), `pod_labels`, `pod_annotations` | `sandbox_claim_name`; adds the MCP session label |
| `delete_sandbox` | `sandbox_claim_name`, `namespace` | Deletes the session-owned claim; PVC cascades |
| `execute_command` | `sandbox_claim_name`, `namespace`, `command`, `timeout=60` (max 600), `max_output_chars` only on v1.0.8+ (128-300,000, default 30,000) | `exit_code`, `stdout`, `stderr`; v1.0.8 adds `*_total_chars` and `*_truncated` |
| `upload_file` | `sandbox_claim_name`, `namespace`, `path`, `content`, `binary=false`, `timeout=60` | Bytes written; base64 when `binary=true` |
| `download_file` | `sandbox_claim_name`, `namespace`, `path`, `binary=false`, `timeout=60` | Content + bytes read |
| `list_files` | `sandbox_claim_name`, `namespace`, `path`, `timeout=60`, `max_entries=1000` (max 10,000) | Entries (`name`, `size`, `type`, `mod_time`), `total_entries`, `truncated` |
| `file_exists` | `sandbox_claim_name`, `namespace`, `path`, `timeout=60` | `exists`; a missing path is not an error |
| `get_sandbox_status` | `sandbox_claim_name`, `namespace` | `SandboxReady` / `SandboxNotReady` / `SandboxNotFound`, `ready`, `message` |
Resource: `sandboxes://{namespace}` lists sandboxes for the current MCP session.
## Session scoping is the most important behavior
Every tool, including `get_sandbox_status`, calls `get_sandbox()` ->
`ensure_session_owns()`, which lists sandboxes with the label selector
`<K8S_SANDBOX_SESSION_ID_LABEL_KEY>=<MCP session id>`. The resource uses the same
selector. Consequences:
- The server can only see/exec/delete claims **it created in the same MCP
session**. It cannot inspect or manage DSH platform claims created by the
gateway or `kubectl`, or claims from another session.
- On a DSH MCP reconnect, the MCP session changes; earlier claims become
invisible to the tools. They keep running until `shutdown_after_seconds`
expires, or until you remove them by label with a confirmed `kubectl delete`.
- A "Sandbox claim '<name>' is not found in namespace '<ns>'" error usually
means wrong namespace **or** the claim belongs to a different session.
- Use `kubectl` for global inventory; use the MCP only for sandboxes it created
in this task.
Session identity requires a session-providing transport. With stdio,
`ctx.session_id` is `None` and creation fails. Connect DSH over
`streamable-http`.
## Runtime prerequisite
The server talks to a sandbox runtime, not to arbitrary pods:
- `gateway`, `direct`, `in-cluster`: runtime HTTP on `SERVER_PORT=8888`
(sandbox-router API or direct pod runtime).
- `sandboxd-in-cluster` (v1.0.8+): sandboxd REST `8080`, gRPC `9090`.
The shipped DSH template runs only DSH Web on `3080`, so no connection type can
exec into it. A sandboxd **sidecar** executes commands in the sidecar's own
container, not in the DSH container; shared volumes share files, not binaries or
processes. To exec inside DSH, build a combined image (sandboxd injected into the
DSH image) or use `kubectl exec` with explicit RBAC and confirmation. For
untrusted code, deploy a separate runtime warm pool rather than modifying the
DSH template.
## Version differences
| Area | Before v1.0.8 | v1.0.8+ |
| --- | --- | --- |
| Connection types | `gateway`, `direct`, `in-cluster` | Adds `sandboxd-in-cluster` |
| `execute_command` output | Unbounded | Bounded by `max_output_chars`; four metadata fields |
| Failed pod recovery | `Retain` only | `spec.failedPodPolicy: Recreate` |
| Router install | `kubectl apply -f sandbox-router/deploy/` | `kubectl apply -k sandbox-router/deploy/`; projected token |
Never pass `max_output_chars` to a pre-v1.0.8 server. On older servers, redirect
verbose output to a file and `download_file` it, or pipe through `tail`/`head`.
The repository pins `AGENT_SANDBOX_VERSION=v1.0.8`, so the right-hand column is
what the documented platform and a same-tag MCP build provide; keep the
before-column for clusters still on v1.0.7.
## Safety and cleanup
- No auth: anyone reaching `:8000` gets every tool and the service account's
access. Keep it ClusterIP, authenticated, and namespace-scoped.
- Treat commands as untrusted code: they can read files and environment
variables inside the sandbox, including provider keys in a combined image.
- `delete_sandbox` cascades the claim, sandbox, and PVC: confirm the target and
namespace. Never delete a namespace or PVC as cleanup.
- Clean up in the same session that created the sandbox. If the session is gone,
list claims by the session label and delete them explicitly (requires write
RBAC); otherwise rely on the shutdown timeout.
## Troubleshooting
| Symptom | Cause / fix |
| --- | --- |
| Tools missing | Row not enabled, wrong URL, connection failed; check logs and `/readyz` |
| 404 on connect | URL is missing `/mcp` or points at the wrong port |
| `ctx.session_id is None` | Connected over stdio; use streamable HTTP |
| `Sandbox claim ... is not found` | Wrong namespace, or claim created by another MCP session |
| Creation times out at ~60 s | MCP session alive but DSH cancelled the call; raise `toolCallTimeoutMs` |
| `execute_command` connection error | No runtime on 8888/8080/9090, or wrong connection type; the DSH template is unsupported |
| Output flood | Pre-v1.0.8 server; bound the command output yourself |
| Empty sandbox list | The resource is session-scoped; use `kubectl` for global inventory |
Deployment, connection matrix, RBAC, and the DSH row:
[MCP deployment](mcp-deployment.md).

View File

@@ -0,0 +1,181 @@
# Platform runbook
Scope: the resource layer and gateway in
`examples/dsh-agents-as-a-service/`, one namespace, one DSH Web sandbox per
user. Read the example README and `AGENTS.md` first. All commands are read-only
unless a section is explicitly a change; confirm before changing anything.
## 1. Context, version, CRDs
```bash
kubectl config current-context
kubectl auth whoami
kubectl config view --minify -o jsonpath='{..namespace}{"\n"}'
kubectl get crd | grep -E 'agents\.x-k8s\.io'
kubectl -n agent-sandbox-system get deploy agent-sandbox-controller \
-o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
```
Compare the controller image tag with `AGENT_SANDBOX_VERSION` in
`versions.env`. The example needs the extension CRDs plus core `sandboxes`;
a missing CRD is a prerequisite failure, not an application bug.
## 2. Resource model
| Object | Role | Key fields |
| --- | --- | --- |
| `SandboxTemplate` | Blueprint; pinned image, probes, security context, PVC template | `envVarsInjectionPolicy`, `volumeClaimTemplatesPolicy`, `service`, `podTemplate` |
| `SandboxWarmPool` | Pre-warmed spares from one template | `spec.replicas`, `sandboxTemplateRef` |
| `SandboxClaim` | One tenant/user request; adopts a spare | `spec.warmPoolRef`, `additionalPodMetadata`, `status.sandbox.name` |
| `Sandbox` | Adopted or directly created pod + Service + PVC | `spec.operatingMode`, `status.conditions`, `status.serviceFQDN`, `status.podIPs` |
| `Pod` | The DSH Web process | `:3080`, PVC at `/home/node/.dsh` |
| `PersistentVolumeClaim` | Per-sandbox state | `dsh-home-<sandbox>` |
| `Service` | Stable per-sandbox DNS | `<sandbox>`, `status.serviceFQDN` |
Delete a claim to cascade-delete its sandbox and PVC (account deletion). Never
delete a PVC or namespace without an explicit, named confirmation.
## 3. Read-only inspection
```bash
kubectl -n "$NS" get sandboxtemplates,sandboxwarmpools,sandboxclaims,sandboxes -o wide
kubectl -n "$NS" get pods,pvc,svc -o wide
kubectl -n "$NS" get events --sort-by=.lastTimestamp | tail -30
kubectl -n "$NS" get sandboxclaim "$CLAIM" -o yaml
kubectl -n "$NS" get sandbox "$SB" -o yaml
```
Bounded logs: `kubectl -n "$NS" logs "$POD" --tail=200` (add `--previous`
only after a restart).
## 4. Claim -> sandbox -> pod
```bash
SB=$(kubectl -n "$NS" get sandboxclaim "$CLAIM" -o jsonpath='{.status.sandbox.name}')
POD=$(kubectl -n "$NS" get sandbox "$SB" -o jsonpath='{.metadata.annotations.agents\.x-k8s\.io/pod-name}')
POD=${POD:-$SB}
kubectl -n "$NS" get sandbox "$SB" -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}{"\n"}{end}'
kubectl -n "$NS" get sandbox "$SB" -o jsonpath='{.spec.operatingMode}{" "}{.status.serviceFQDN}{" "}{.status.podIPs}{"\n"}'
```
Rules:
- Resolve user -> sandbox only through `claim.status.sandbox.name`; the sandbox
keeps its pool-generated name, not the claim name.
- A claim tracks binding; for runtime state read `Sandbox.spec.operatingMode`
and the `Suspended`/`Ready` conditions. A suspended sandbox is not serving
even when the claim still exists.
- A missing `claim.status.sandbox.name` means the claim did not bind: inspect
the Ready condition `reason` and events. The pod is normally named after the
sandbox; use the annotation when present.
## 5. Suspend, resume, wake
Suspend keeps the PVC and Service, deletes the pod:
```bash
kubectl -n "$NS" patch sandbox "$SB" --type merge \
-p '{"spec":{"operatingMode":"Suspended"}}'
kubectl -n "$NS" wait --for=delete "pod/$POD" --timeout=120s
kubectl -n "$NS" get pvc "dsh-home-$SB"
kubectl -n "$NS" get svc "$SB"
```
Resume and verify state:
```bash
kubectl -n "$NS" patch sandbox "$SB" --type merge \
-p '{"spec":{"operatingMode":"Running"}}'
kubectl -n "$NS" wait sandbox "$SB" --for=condition=Ready --timeout=180s
kubectl -n "$NS" exec "$SB" -- cat /home/node/.dsh/marker.txt
```
The gateway wakes a suspended sandbox on the first request. The idle sweeper
only suspends claims it created (`aaas.example.com/managed-by=dsh-gateway`);
hand-applied claims are never swept.
## 6. Warm pool
```bash
kubectl -n "$NS" get sandboxwarmpool "$POOL" -o yaml
kubectl -n "$NS" get sandboxes -l agents.x-k8s.io/warm-pool-sandbox -o wide
kubectl -n "$NS" get sandboxwarmpool "$POOL" -o jsonpath='{.spec.replicas}{"\n"}'
```
Adopted sandboxes lose the warm-pool label; count current spares against
`spec.replicas`. Zero spares or `spec.replicas: 0` means cold, slow signup.
Scale only when asked:
```bash
kubectl -n "$NS" patch sandboxwarmpool "$POOL" --type merge -p '{"spec":{"replicas":2}}'
```
## 7. DSH runtime specifics
- DSH Web listens on `3080` in-cluster. Port-forward to a free local port such
as `18081`, never `3080` if a local DSH already uses it.
- The launch token is printed once as `dsh web: http://<ip>:3080/?token=...`.
Treat it as a credential: do not paste it into reports; hand the user the URL.
- `/` returns `401` without the token. Probes must target `/favicon.svg`.
- `DSH_TRUSTED_HOSTS` must include any public host that a proxy forwards; DSH
fences on `Host`/`Origin`.
- State lives at `/home/node/.dsh`. DSH requires `.credentials.yaml` to be
`0600`; the template uses `fsGroupChangePolicy: OnRootMismatch`. If a resumed
sandbox crash-loops with a mode-660 error, suspend it, fix the file with a
throwaway pod, patch the policy, then resume.
## 8. Gateway layer
```bash
kubectl -n "$NS" get deploy,svc,pods -l app=dsh-gateway -o wide
kubectl -n "$NS" get endpoints dsh-gateway
kubectl -n "$NS" logs deploy/dsh-gateway --tail=100
```
The gateway is a ClusterIP Service on `8080`; it proxies under `/u/{user}/` and
derives state (`Provisioning`, `Waking`, `Ready`, `Suspending`, `Suspended`).
It performs the DSH launch-token handshake server-side. With SSO enabled
(`AUTH_PROXY_HEADER`), the Service must be reachable only through the
authenticating proxy; never add a second path or port-forward it on a shared
host. A resource-only install has no gateway: that is a warning, not a failure.
## 9. Security boundaries
- Provider keys (`dsh-provider-keys` plus the optional
`dsh-provider-keys-extra`) are shared by all sandboxes; the example is not
tenant-isolated.
- gVisor, `nodeSelector`, the sandbox/exposure NetworkPolicies and extra
provider keys are opt-in, not active by default. A NetworkPolicy only helps
with a CNI that enforces it; see
[optional hardening](../../../../examples/dsh-agents-as-a-service/docs/optional-hardening.md).
- Secret data, kubeconfigs, and pod logs are sensitive; RBAC may allow Secret
access, policy does not. Keep gateway RBAC namespace-scoped; it needs pod logs
to find the launch token.
- TLS terminates outside the gateway; set `COOKIE_SECURE=true` for HTTPS.
## 10. Failure matrix
| Symptom | First evidence | Likely cause / fix |
| --- | --- | --- |
| Claim not Ready, no sandbox | Ready condition reason, events | Empty pool, image pull, quota, unschedulable |
| `EnvVarsInjectionRejected` | claim Ready reason | Claim carries env/volume templates; template policy rejects it |
| Pod Pending | `kubectl describe pod` | Requests/taints/affinity, unbound PVC |
| CrashLoopBackOff | logs, `--previous` | Startup failure, bad config, probe 401, fsGroup mode 660 |
| ImagePullBackOff | describe events | Wrong tag, private registry, missing pull secret |
| DSH 401 in browser | gateway logs | Launch-token handshake failed; check gateway RBAC `pods/log` |
| 502/503 through gateway | sandbox Ready, gateway logs | Wake timeout, DNS stale, token not in logs yet |
| Warm pool empty | sandboxes label count | Scale the pool or wait for replenishment |
| Suspended but expected running | `operatingMode` | Resume it; the gateway only wakes on a request |
| WS closes `1011` | gateway logs | DSH rejected the upstream Host/authority; check trusted hosts |
## 11. Cleanup and reporting
Delete only claims/sandboxes you created, by name:
```bash
kubectl -n "$NS" delete sandboxclaim "$CLAIM" --dry-run=server
kubectl -n "$NS" delete sandboxclaim "$CLAIM"
```
Report context/namespace, evidence, conclusion, change, rollback, and what was
deleted. Leave the namespace, pools, templates, and unrelated claims alone.

View File

@@ -0,0 +1,72 @@
# Troubleshooting
Start with evidence, not fixes: resolve context/namespace, read conditions and
events, then compare desired vs observed state. Keep output bounded. Never
"fix" an access error by broadening credentials.
## Platform
| Symptom | First evidence | Likely cause / fix |
| --- | --- | --- |
| `Forbidden` on every call | `kubectl auth whoami`, `kubectl auth can-i --list -n <ns>` | Wrong context or RBAC. Do not retry with broader credentials; ask for the right kubeconfig. |
| CRDs not found | `kubectl get crd | grep agents.x-k8s.io` | agent-sandbox extensions not installed. Install the pinned release YAML; that is a prerequisite, not a code change. |
| Claim created, no sandbox | `claim.status.conditions[?(@.type=="Ready")]`, events | Empty pool, image pull, quota, unschedulable; or policy rejection. |
| Claim rejected | Ready reason `EnvVarsInjectionRejected` | Claim carries `env`/`volumeClaimTemplates` while the template disallows injection. Remove them; do not bypass the pool. |
| User is suspended but the claim exists | `Sandbox.spec.operatingMode`, `Suspended` condition | Resume with `operatingMode: Running`; a gateway request would wake it. |
| Warm pool exhausted | `kubectl get sandboxes -l agents.x-k8s.io/warm-pool-sandbox` | Scale `spec.replicas` (with confirmation) or accept slow cold provisioning. |
| Pod `Pending` | `kubectl describe pod` | Requests/limits, taints, affinity, unbound PVC, storage class. |
| `ImagePullBackOff` | describe events | Wrong image/tag, private GHCR package, missing pull secret; on kind, load the image under the exact manifest ref. |
| `CrashLoopBackOff`, probe 401 | logs, `describe` | Probe path is `/`; use `/favicon.svg`. |
| Restart with mode-660 error | logs mention `.credentials.yaml` | kubelet fsGroup widened the file; suspend, `chmod 600` from a throwaway pod, patch `fsGroupChangePolicy: OnRootMismatch`, resume. |
| PVC `Pending` | `kubectl describe pvc`, StorageClass | No default provisioner, access mode, zone binding. |
| Launch token not found | `kubectl logs <pod> --tail=2000 | grep 'dsh web:'` | Pod not ready yet, or logs rotated. Do not guess tokens; wait for readiness. |
| Sandbox reports `Finished=True` | pod phase, `kubectl get events` | The backing Pod reached `Failed` (eviction, spot reclaim) and `failedPodPolicy` defaults to `Retain`. Delete the Pod to force a replacement, or set `Recreate` (v1.0.8+) on the Sandbox/template; the claim and PVCs survive either way. |
| Hand-applied sandbox disappeared | claim annotations, gateway logs | Idle sweeper only manages claims with `aaas.example.com/managed-by=dsh-gateway`; if it was managed, recreate/resume. |
| Gateway 401 | gateway logs, cookie flags | Missing/expired cookie, DSH handshake failed, or `COOKIE_SECURE=true` over HTTP. |
| Gateway 502/503 | sandbox Ready, gateway logs | Wake timeout, stale DNS, token not yet in logs; check `WAKE_TIMEOUT` and RBAC `pods/log`. |
| WebSocket closes 1011 | gateway logs, `Host`/Origin | DSH rejected the upstream authority; check `DSH_TRUSTED_HOSTS` and the gateway's pod-IP dial path. |
| WebSocket closes 4401 | cookie, Origin | Missing gateway auth or cross-origin upgrade; send a matching `Origin`. |
| SSO header ignored | gateway env | `AUTH_PROXY_HEADER` unset; setting it requires the proxy to be the only path to the ClusterIP Service. |
| Someone is treated as another user | service paths | Trust-boundary violation: the gateway is directly reachable. Remove the extra path and disable SSO until fixed. |
## Agent-sandbox MCP
| Symptom | First evidence | Likely cause / fix |
| --- | --- | --- |
| No `mcp__*__create_sandbox` tools | tool list, DSH logs | Server not deployed or row not enabled. Use `kubectl`; connect only with approval. |
| 404 on connect | curl `/mcp`, `/healthz` | Wrong path/port; MCP endpoint is `/mcp`, health is at the root. |
| `/readyz` 503 | response body, server logs | API unreachable or namespace `K8S_SANDBOX_PROBE_NAMESPACE` not listable; fix RBAC/config. |
| `ctx.session_id is None` | server error | Connected over stdio; use streamable HTTP. |
| `Sandbox claim ... is not found` | server error, `kubectl get claim` | Wrong namespace, or the claim belongs to another MCP session. List with `kubectl`; use the MCP only for its own claims. |
| Creation cancelled at ~60 s | MCP call log | DSH `toolCallTimeoutMs` is 60 s while creation waits up to 180 s; raise the timeout. |
| `execute_command` connection error | server logs, template ports | No runtime on 8888/8080/9090. The DSH template only listens on 3080; use a runtime template or a combined image. |
| Output truncated / flooded | tool result | Pre-v1.0.8 is unbounded; v1.0.8 bounds output. Redirect verbose output to a file and read/download it; never pass `max_output_chars` to an old server. |
| Orphaned claims after reconnect | `kubectl get claims -l mcp.k8s-agent-sandbox/session-id` | MCP sessions are transient; wait for `shutdown_after_seconds` or delete by label with explicit confirmation. |
| `Forbidden` from the MCP server | server logs | Its ServiceAccount lacks claim/sandbox verbs, or RBAC is in another namespace; fix the Role, not the kubeconfig. |
| Sandbox expired | `get_sandbox_status` -> `SandboxNotFound` | `shutdown_after_seconds` elapsed; create a new sandbox and keep the task inside the budget. |
## Repository checks and CI
| Symptom | Cause / fix |
| --- | --- |
| `check-versions: stale version reference in docs: dsh-v...` | A copied release tag in a Markdown file. Refer to the identity in `versions.env`; run `make set-version` only for a real bump. |
| `check-versions: hard-coded version outside versions.env` | A tag in `.github/workflows` or the Makefile; move it to `versions.env`. |
| `check-examples` layout error | Missing `README.md`/`manifests/`, stray YAML at the example root, missing exposure README, non-executable script, or missing `examples/README.md` entry. |
| Manifest dry-run fails in CI | Invalid YAML or CRD schema; parse locally and run `kubectl apply --dry-run=server` against the pinned CRDs. |
| `shellcheck` failure | Quote expansions, use arrays for kubectl args, avoid unused variables; run shellcheck locally. |
| Lychee link failure | A relative/absolute link does not resolve; the link checker runs offline on the listed docs. |
| PR-title check failure | Use Conventional Commits: `feat(scope): ...`, `fix(scope): ...`, `docs(scope): ...`. |
| Changelog conflict | Never edit `CHANGELOG.md`; it is generated from commits. |
## Do not
- Do not use `--force --grace-period=0` to clear a stuck resource before
understanding why it is stuck.
- Do not delete namespaces, PVCs, PVs, CRDs, or cluster-scoped objects as a
cleanup shortcut.
- Do not read Secret data, kubeconfigs, or node logs to "debug" a permission
problem, and do not paste pod logs containing the DSH launch token into a
report.
- Do not expose the unauthenticated MCP server or enable `AUTH_PROXY_HEADER`
while another path to the gateway exists.
- Do not retry a `Forbidden` with a stronger kubeconfig; fix the RBAC scope.

View File

@@ -0,0 +1,298 @@
#!/usr/bin/env bash
# Read-only preflight for the DSH agents-as-a-service platform.
#
# Checks the kubectl target, agent-sandbox CRDs/version, warm pools, claims,
# sandboxes, gateway and (optionally) an agent-sandbox MCP server. It never
# mutates the cluster, never reads Secret data, and never port-forwards.
#
# Usage:
# platform-preflight.sh [--context NAME] [--namespace NS] [--pool NAME]
# [--template NAME] [--mcp-url URL] [--help]
#
# Exit codes: 0 = no failures, 1 = at least one failure, 2 = usage error.
set -euo pipefail
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(git -C "$SCRIPT_DIR" rev-parse --show-toplevel 2>/dev/null || true)"
CTX=""
NS="${NAMESPACE:-dsh-demo}"
POOL=""
TEMPLATE=""
MCP_URL=""
PASS=0
WARN=0
FAIL=0
usage() {
cat <<'EOF'
Read-only preflight for the DSH agents-as-a-service platform.
Options:
--context NAME kubectl context (default: current context)
--namespace NS namespace to inspect (default: $NAMESPACE or dsh-demo)
--pool NAME check only this warm pool
--template NAME check only this SandboxTemplate
--mcp-url URL also check an MCP server's /healthz and /readyz
--help show this help
The script makes no changes and never reads Secret data.
EOF
}
while [ "$#" -gt 0 ]; do
case "$1" in
--context) CTX="${2:-}"; shift 2 ;;
--namespace) NS="${2:-}"; shift 2 ;;
--pool) POOL="${2:-}"; shift 2 ;;
--template) TEMPLATE="${2:-}"; shift 2 ;;
--mcp-url) MCP_URL="${2:-}"; shift 2 ;;
--help|-h) usage; exit 0 ;;
*) echo "unknown argument: $1" >&2; usage >&2; exit 2 ;;
esac
done
if [ -z "$NS" ]; then
echo "namespace must not be empty" >&2
exit 2
fi
section() { printf '\n== %s ==\n' "$1"; }
pass() { PASS=$((PASS + 1)); printf 'PASS %s\n' "$1"; }
warn() { WARN=$((WARN + 1)); printf 'WARN %s\n' "$1"; }
fail() { FAIL=$((FAIL + 1)); printf 'FAIL %s\n' "$1"; }
info() { printf 'INFO %s\n' "$1"; }
command -v kubectl >/dev/null 2>&1 || {
echo "FAIL kubectl is not installed" >&2
exit 1
}
if [ -n "$CTX" ]; then
if ! kubectl config get-contexts -o name 2>/dev/null | grep -Fxq "$CTX"; then
echo "FAIL context '$CTX' is not in the kubeconfig" >&2
exit 1
fi
KUBECTL=(kubectl --context "$CTX" --request-timeout=15s)
else
KUBECTL=(kubectl --request-timeout=15s)
fi
k() { "${KUBECTL[@]}" "$@"; }
section "Target"
if current="$(k config current-context 2>/dev/null)"; then
info "context: $current"
pass "kubeconfig is readable"
else
fail "cannot read the current kubectl context"
fi
if user="$(k auth whoami -o jsonpath='{.status.userInfo.username}' 2>/dev/null)" && [ -n "$user" ]; then
info "identity: $user"
else
fallback_user="$(k config view --minify -o jsonpath='{.users[0].name}' 2>/dev/null || true)"
info "identity: ${fallback_user:-unknown}"
fi
if k get namespace "$NS" -o name >/dev/null 2>&1; then
pass "namespace $NS exists"
else
fail "namespace $NS is missing or not readable"
fi
if [ -n "$REPO_ROOT" ] && [ -f "$REPO_ROOT/versions.env" ]; then
# shellcheck source=/dev/null
. "$REPO_ROOT/versions.env"
info "expected agent-sandbox pin: ${AGENT_SANDBOX_VERSION:-unknown}"
else
warn "versions.env not found; skipping the pin comparison"
fi
section "Capabilities (informational; never widen them)"
for check in "create sandboxclaims" "delete sandboxclaims" "create pods/exec"; do
verb="${check%% *}"
resource="${check#* }"
if [ "$(k auth can-i "$verb" "$resource" -n "$NS" 2>/dev/null || echo unknown)" = "yes" ]; then
info "can $verb $resource in $NS: yes"
else
info "can $verb $resource in $NS: no"
fi
done
if [ "$(k auth can-i get secrets -n "$NS" 2>/dev/null || echo unknown)" = "yes" ]; then
warn "RBAC allows reading Secrets; policy still forbids printing Secret data"
else
info "can get secrets in $NS: no"
fi
section "Agent Sandbox CRDs"
for crd in \
sandboxtemplates.extensions.agents.x-k8s.io \
sandboxwarmpools.extensions.agents.x-k8s.io \
sandboxclaims.extensions.agents.x-k8s.io \
sandboxes.agents.x-k8s.io; do
if k get crd "$crd" -o name >/dev/null 2>&1; then
pass "CRD $crd"
else
fail "CRD $crd is missing or not readable"
fi
done
section "Controller"
controller_ns="agent-sandbox-system"
controller_image="$(k -n "$controller_ns" get deployment agent-sandbox-controller \
-o jsonpath='{.spec.template.spec.containers[0].image}' 2>/dev/null || true)"
if [ -n "$controller_image" ]; then
info "controller image: $controller_image"
controller_tag="${controller_image##*:}"
if [ -n "${AGENT_SANDBOX_VERSION:-}" ] && [ "$controller_tag" != "$AGENT_SANDBOX_VERSION" ]; then
warn "controller tag '$controller_tag' differs from AGENT_SANDBOX_VERSION '$AGENT_SANDBOX_VERSION'"
else
pass "controller version matches versions.env"
fi
else
warn "controller deployment not found in $controller_ns; cannot verify the installed version"
fi
section "Templates"
templates="$(k -n "$NS" get sandboxtemplates \
-o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' 2>/dev/null || true)"
if [ -z "$templates" ]; then
warn "no SandboxTemplates in $NS"
fi
if [ -n "$TEMPLATE" ] && ! printf '%s\n' "$templates" | grep -Fxq "$TEMPLATE"; then
fail "SandboxTemplate $TEMPLATE not found in $NS"
fi
for tpl in $templates; do
[ -n "$TEMPLATE" ] && [ "$tpl" != "$TEMPLATE" ] && continue
desc="$(k -n "$NS" get sandboxtemplate "$tpl" \
-o jsonpath='{range .spec.podTemplate.spec.containers[*]}{.name}{":"}{range .ports[*]}{.containerPort}{","}{end}{";"}{end}' \
2>/dev/null || true)"
info "template $tpl containers: ${desc:-<none>}"
case "$desc" in
*sandboxd*|*:8080,*|*:9090,*)
pass "template $tpl exposes a sandboxd runtime (MCP sandboxd-in-cluster)" ;;
*:8888,*)
pass "template $tpl exposes a runtime HTTP port (MCP gateway/direct/in-cluster)" ;;
*:3080,*)
warn "template $tpl is DSH Web only (:3080); MCP execute_command cannot run against it" ;;
*)
warn "template $tpl exposes no known sandbox runtime port" ;;
esac
done
section "Warm pools"
pools="$(k -n "$NS" get sandboxwarmpools \
-o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' 2>/dev/null || true)"
if [ -z "$pools" ]; then
warn "no SandboxWarmPools in $NS; claims will provision cold"
fi
if [ -n "$POOL" ] && ! printf '%s\n' "$pools" | grep -Fxq "$POOL"; then
fail "SandboxWarmPool $POOL not found in $NS"
fi
for pool in $pools; do
[ -n "$POOL" ] && [ "$pool" != "$POOL" ] && continue
replicas="$(k -n "$NS" get sandboxwarmpool "$pool" -o jsonpath='{.spec.replicas}' 2>/dev/null || true)"
selector="$(k -n "$NS" get sandboxwarmpool "$pool" -o jsonpath='{.status.selector}' 2>/dev/null || true)"
if [ -n "$selector" ]; then
ready="$(k -n "$NS" get sandboxes -l "$selector" \
-o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' \
2>/dev/null | grep -c '^True$' || true)"
else
ready="$(k -n "$NS" get sandboxes -l agents.x-k8s.io/warm-pool-sandbox \
-o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' \
2>/dev/null | grep -c '^True$' || true)"
fi
replicas="${replicas:-0}"
info "pool $pool: $ready/$replicas warm spares Ready"
if [ "$replicas" -eq 0 ]; then
warn "pool $pool has spec.replicas=0; signup will cold-provision"
elif [ "$ready" -lt "$replicas" ]; then
warn "pool $pool is below its desired spare count"
else
pass "pool $pool is at capacity"
fi
done
section "Claims"
claims="$(k -n "$NS" get sandboxclaims \
-o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.conditions[?(@.type=="Ready")].status}{"\t"}{.status.conditions[?(@.type=="Ready")].reason}{"\t"}{.status.sandbox.name}{"\n"}{end}' \
2>/dev/null || true)"
if [ -z "$claims" ]; then
warn "no SandboxClaims in $NS"
else
while IFS=$'\t' read -r name ready reason sandbox; do
[ -n "$name" ] || continue
if [ "$ready" != "True" ]; then
warn "claim $name is not Ready (status='${ready:-unknown}', reason='${reason:-unknown}')"
elif [ -z "$sandbox" ]; then
warn "claim $name is Ready but has no bound sandbox"
else
pass "claim $name -> sandbox $sandbox"
fi
done <<< "$claims"
fi
section "Sandboxes"
sandboxes="$(k -n "$NS" get sandboxes \
-o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.operatingMode}{"\t"}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' \
2>/dev/null || true)"
if [ -z "$sandboxes" ]; then
warn "no Sandboxes in $NS"
else
while IFS=$'\t' read -r name mode ready; do
[ -n "$name" ] || continue
info "sandbox $name mode=${mode:-Running} ready=${ready:-unknown}"
done <<< "$sandboxes"
fi
section "Storage and Services"
pvc_count="$(k -n "$NS" get pvc -o name 2>/dev/null | wc -l | tr -d ' ')"
svc_count="$(k -n "$NS" get service -o name 2>/dev/null | wc -l | tr -d ' ')"
info "persistent volume claims: ${pvc_count:-0}"
info "services: ${svc_count:-0}"
[ "${pvc_count:-0}" -gt 0 ] || warn "no PVCs in $NS"
[ "${svc_count:-0}" -gt 0 ] || warn "no Services in $NS"
section "Gateway"
if k -n "$NS" get deployment dsh-gateway -o name >/dev/null 2>&1; then
pass "gateway deployment dsh-gateway exists"
if k -n "$NS" get service dsh-gateway -o name >/dev/null 2>&1; then
endpoints="$(k -n "$NS" get endpoints dsh-gateway \
-o jsonpath='{range .subsets[*].addresses[*]}{.ip}{" "}{end}' 2>/dev/null || true)"
if [ -n "$endpoints" ]; then
pass "gateway endpoints: $endpoints"
else
warn "gateway Service has no ready endpoints"
fi
else
warn "gateway Service dsh-gateway is missing"
fi
else
warn "no dsh-gateway deployment; this is a resource-layer-only install"
fi
section "MCP server"
if [ -n "$MCP_URL" ]; then
if ! command -v curl >/dev/null 2>&1; then
fail "curl is required for --mcp-url"
else
base="${MCP_URL%/}"
base="${base%/mcp}"
for endpoint in healthz readyz; do
body="$(curl -fsS --max-time 5 "$base/$endpoint" 2>/dev/null | head -c 300 || true)"
if [ -n "$body" ]; then
pass "MCP /$endpoint: $body"
else
fail "MCP /$endpoint is unreachable at $base/$endpoint"
fi
done
fi
else
info "no --mcp-url supplied; skipping server health checks"
fi
printf '\n== Summary ==\nPASS=%d WARN=%d FAIL=%d\n' "$PASS" "$WARN" "$FAIL"
if [ "$FAIL" -gt 0 ]; then
exit 1
fi
exit 0

15
.editorconfig Normal file
View File

@@ -0,0 +1,15 @@
root = true
[*]
charset = utf-8
end_of_line = lf
insert_final_newline = true
trim_trailing_whitespace = true
indent_style = space
indent_size = 2
[*.py]
indent_size = 4
[Makefile]
indent_style = tab

View File

@@ -1,29 +1,50 @@
# Copy to .env and fill in. Never commit a real key.
#
# The version values below are documentation only: versions.env is the single
# source of truth, and scripts/check-versions.sh fails if they drift.
# --- DSH Web image (Dockerfile / build.sh) ---
# --- Main DSH Web image (root Dockerfile) ---
DEEPSEEK_API_KEY=
DSH_VERSION=dsh-v0.2.1-rc.2
# Optional additional provider keys consumed by DSH. For Docker, pass them with
# -e; the Kubernetes examples inject them through the optional
# dsh-provider-keys-extra Secret.
OPENROUTER_API_KEY=
GEMINI_API_KEY=
NVIDIA_API_KEY=
OPENAI_API_KEY=
ANTHROPIC_API_KEY=
GROQ_API_KEY=
XAI_API_KEY=
MISTRAL_API_KEY=
TOGETHER_API_KEY=
CEREBRAS_API_KEY=
AI_GATEWAY_API_KEY=
HF_TOKEN=
DSH_VERSION=dsh-v0.2.1-rc.5
RELEASE_REVISION=
PNPM_VERSION=11.28.5
# DSH Web port inside the container, and the local port published by
# build.sh / docker run. The gateway also uses it as the upstream port.
# Container port, and the host port published by "make run".
DSH_PORT=3080
# Comma/space separated extra Host allow-list entries for DSH (for an
# Ingress/public hostname). localhost and 127.0.0.1 are always trusted.
# Comma/space separated extra Host allow-list entries for DSH (for a proxy or
# public hostname). localhost and 127.0.0.1 are always trusted.
DSH_TRUSTED_HOSTS=
# --- Gateway (examples/dsh-agents-as-a-service) ---
GATEWAY_IMAGE=ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway
# --- agents-as-a-service example ---
NAMESPACE=dsh-demo
POOL=dsh-pool
# Seconds a sandbox may stay idle before the sweeper suspends it.
IDLE_TIMEOUT=60
# Seconds a waking request may wait for the sandbox to become Ready.
WAKE_TIMEOUT=120
# Seconds a sandbox may stay idle before the sweeper suspends it, and the max
# time a proxied request waits for a suspended sandbox to become Ready.
IDLE_TIMEOUT=360
WAKE_TIMEOUT=240
DSH_CONTAINER=dsh-web
DSH_LOG_TAIL_LINES=2000
COOKIE_NAME=dsh_gw_session
# Set to true only behind HTTPS; secure cookies are not sent over plain HTTP.
# Set to true only behind HTTPS; secure cookies are not sent over plain HTTP
# (localhost counts as a secure context in browsers).
COOKIE_SECURE=false
COOKIE_MAX_AGE=604800
MAX_BODY_MB=32
HTTP_TIMEOUT=300
# SSO is opt-in. Set this only when the gateway Service is reachable
# exclusively through oauth2-proxy (see the example exposure docs).
# AUTH_PROXY_HEADER=X-Forwarded-User

42
.github/ISSUE_TEMPLATE/bug_report.yml vendored Normal file
View File

@@ -0,0 +1,42 @@
name: Bug report
description: Report a reproducible problem with the image, an example or the tooling.
labels: ["bug"]
body:
- type: markdown
attributes:
value: |
Do not include API keys, tokens or other secrets. Report security
issues privately via SECURITY.md instead.
- type: textarea
id: what-happened
attributes:
label: What happened
description: The observed behavior and the expected behavior.
validations:
required: true
- type: textarea
id: reproduce
attributes:
label: Reproduction steps
description: Commands, manifests and versions. Redact secrets.
placeholder: |
make image
docker run ...
validations:
required: true
- type: input
id: image-tag
attributes:
label: Image or release
description: The pinned tag or digest, for example dsh-v0.2.1-rc.5.
- type: input
id: cluster
attributes:
label: Environment
description: Kubernetes distribution/version, kind version, agent-sandbox version.
- type: textarea
id: logs
attributes:
label: Relevant logs
description: Paste the smallest useful excerpt. Remove tokens and API keys.
render: shell

8
.github/ISSUE_TEMPLATE/config.yml vendored Normal file
View File

@@ -0,0 +1,8 @@
blank_issues_enabled: false
contact_links:
- name: Security vulnerability
url: https://github.com/sredevopsorg/deepseek-kubernetes-sandbox/security/advisories/new
about: Report security issues privately; do not open a public issue.
- name: Documentation
url: https://github.com/sredevopsorg/deepseek-kubernetes-sandbox/tree/main/docs
about: Read the repository docs before opening an issue.

View File

@@ -0,0 +1,35 @@
name: Feature request
description: Propose a new example, image capability or workflow improvement.
labels: ["enhancement"]
body:
- type: textarea
id: problem
attributes:
label: Problem or use case
description: What are you trying to do, and what blocks you today?
validations:
required: true
- type: textarea
id: proposal
attributes:
label: Proposed solution
description: Describe the shape you have in mind. For a new example, say which implementation it demonstrates.
validations:
required: true
- type: dropdown
id: area
attributes:
label: Area
options:
- Main DSH Web image
- Existing example
- New example
- CI / release automation
- Documentation
- Other
validations:
required: true
- type: textarea
id: alternatives
attributes:
label: Alternatives considered

36
.github/dependabot.yml vendored Normal file
View File

@@ -0,0 +1,36 @@
version: 2
updates:
- package-ecosystem: github-actions
directory: /
schedule:
interval: weekly
groups:
actions:
patterns: ["*"]
commit-message:
prefix: "chore: "
- package-ecosystem: pip
directory: /examples/dsh-agents-as-a-service/gateway
schedule:
interval: weekly
groups:
gateway-python:
patterns: ["*"]
commit-message:
prefix: "chore: "
- package-ecosystem: docker
directory: /
schedule:
interval: weekly
commit-message:
prefix: "chore: "
- package-ecosystem: docker
directory: /examples/dsh-agents-as-a-service/gateway
schedule:
interval: weekly
commit-message:
prefix: "chore: "

31
.github/pull_request_template.md vendored Normal file
View File

@@ -0,0 +1,31 @@
## Summary
<!-- What changes, and why? Link the issue: Closes #123 -->
## Type of change
- [ ] feat: new capability or example
- [ ] fix: bug fix
- [ ] docs: documentation only
- [ ] refactor: no behavior change
- [ ] build / ci: tooling, images, workflows
- [ ] chore: maintenance
## Breaking change
- [ ] Yes, and the migration is documented in the description
- [ ] No
## Checklist
- [ ] make check passes (versions, example layout, manifest YAML)
- [ ] make test passes (gateway unit tests)
- [ ] make lint passes (ruff, shellcheck; yamllint/actionlint where installed)
- [ ] Docs updated for the change (root README, docs/, example README)
- [ ] No secrets or usable credentials committed
- [ ] CHANGELOG.md was not edited by hand (generated by git-cliff)
- [ ] Version pins changed only through versions.env / make set-version
## Testing
<!-- Commands run, clusters used, results. Mention if kind e2e was skipped. -->

176
.github/workflows/_build-image.yaml vendored Normal file
View File

@@ -0,0 +1,176 @@
# Reusable multi-arch image build: one workflow, called by build-main,
# build-gateway and release. Builds natively per platform (no QEMU), pushes by
# digest, then merges the manifest list with SBOM and provenance attestations.
name: Reusable multi-arch image build
on:
workflow_call:
inputs:
image:
description: Fully qualified image name without a tag
required: true
type: string
version:
description: Version recorded in the OCI labels
required: true
type: string
context:
description: Docker build context
required: true
type: string
dockerfile:
description: Dockerfile path, relative to the context
required: false
default: Dockerfile
type: string
build-args:
description: Newline-separated build args
required: false
default: ""
type: string
tags:
description: Newline-separated fully qualified tags to publish
required: true
type: string
permissions:
contents: read
packages: write
id-token: write
attestations: write
jobs:
build:
name: Build ${{ matrix.platform }}
runs-on: ${{ matrix.platform == 'linux/amd64' && 'ubuntu-latest' || 'ubuntu-24.04-arm' }}
timeout-minutes: 120
strategy:
fail-fast: false
matrix:
platform:
- linux/amd64
- linux/arm64
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Prepare environment
env:
IMAGE: ${{ inputs.image }}
run: |
set -euo pipefail
pair=$(printf '%s' "${{ matrix.platform }}" | tr '/' '-')
# This reusable workflow runs once for the main image and once for the
# gateway in the same release run. Artifact names must include a
# unique token for the image: the repository path is a prefix of the
# gateway path, so a path-derived pattern would match both. Use a
# short hash of the full image name.
image_hash=$(printf '%s' "$IMAGE" | sha256sum | cut -c1-12)
{
echo "PLATFORM_PAIR=$pair"
echo "CACHE_SCOPE=$(printf '%s' "$IMAGE" | tr '/' '-')"
echo "ARTIFACT_PREFIX=$image_hash"
} >> "$GITHUB_ENV"
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
- name: Login to GitHub Container Registry
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Build and push by digest
id: build
uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7.4.0
with:
context: ${{ inputs.context }}
file: ${{ inputs.context }}/${{ inputs.dockerfile }}
platforms: ${{ matrix.platform }}
build-args: ${{ inputs.build-args }}
labels: |
org.opencontainers.image.source=https://github.com/${{ github.repository }}
org.opencontainers.image.version=${{ inputs.version }}
org.opencontainers.image.revision=${{ github.sha }}
outputs: type=image,name=${{ inputs.image }},push-by-digest=true,name-canonical=true,push=true,oci-mediatypes=true
sbom: true
provenance: mode=max
cache-from: type=gha,scope=${{ env.CACHE_SCOPE }}-${{ matrix.platform }}
cache-to: type=gha,mode=max,scope=${{ env.CACHE_SCOPE }}-${{ matrix.platform }}
- name: Export digest
run: |
set -euo pipefail
mkdir -p /tmp/digests
digest="${{ steps.build.outputs.digest }}"
touch "/tmp/digests/${digest#sha256:}"
- name: Upload digest
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: digests-${{ env.ARTIFACT_PREFIX }}-${{ env.PLATFORM_PAIR }}
path: /tmp/digests/*
if-no-files-found: error
retention-days: 1
merge:
name: Merge manifest list
needs: build
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- name: Prepare environment
env:
IMAGE: ${{ inputs.image }}
run: |
set -euo pipefail
echo "ARTIFACT_PREFIX=$(printf '%s' "$IMAGE" | sha256sum | cut -c1-12)" >> "$GITHUB_ENV"
- name: Download digests
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with:
path: /tmp/digests
pattern: digests-${{ env.ARTIFACT_PREFIX }}-*
merge-multiple: true
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
- name: Login to GitHub Container Registry
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Create and push manifest list
working-directory: /tmp/digests
env:
IMAGE: ${{ inputs.image }}
TAGS: ${{ inputs.tags }}
run: |
set -euo pipefail
args=()
while IFS= read -r tag; do
if [ -n "$tag" ]; then
args+=("-t" "$tag")
fi
done <<< "$TAGS"
refs=()
for digest in *; do
refs+=("$IMAGE@sha256:$digest")
done
docker buildx imagetools create "${args[@]}" "${refs[@]}"
- name: Inspect image
env:
IMAGE: ${{ inputs.image }}
TAGS: ${{ inputs.tags }}
run: |
set -euo pipefail
first=$(printf '%s\n' "$TAGS" | head -n1)
docker buildx imagetools inspect "$first"
echo "### Published ${{ inputs.image }}" >> "$GITHUB_STEP_SUMMARY"
printf '%s\n' "$TAGS" | sed 's/^/- /' >> "$GITHUB_STEP_SUMMARY"

70
.github/workflows/build-gateway.yaml vendored Normal file
View File

@@ -0,0 +1,70 @@
# Development channel for the example gateway image: a merge that touches the
# gateway publishes ghcr.io/<owner>/<repo>/dsh-agents-as-a-service/gateway:main.
name: Build gateway image
on:
workflow_dispatch:
push:
branches: [main]
paths:
- examples/dsh-agents-as-a-service/gateway/**
- versions.env
- .github/workflows/build-gateway.yaml
- .github/workflows/_build-image.yaml
permissions:
contents: read
packages: write
id-token: write
attestations: write
concurrency:
group: build-gateway-${{ github.ref }}
cancel-in-progress: true
jobs:
meta:
name: Read versions
runs-on: ubuntu-latest
timeout-minutes: 5
outputs:
identity: ${{ steps.meta.outputs.identity }}
gateway_image: ${{ steps.meta.outputs.gateway_image }}
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Read versions.env
id: meta
shell: bash
run: |
set -euo pipefail
set -a
# shellcheck disable=SC1091
source versions.env
set +a
identity="$DSH_VERSION"
[ -z "$RELEASE_REVISION" ] || identity="$DSH_VERSION-$RELEASE_REVISION"
{
echo "identity=$identity"
echo "gateway_image=$GATEWAY_IMAGE"
} >> "$GITHUB_OUTPUT"
build:
name: Build and push gateway image
needs: meta
uses: ./.github/workflows/_build-image.yaml
permissions:
contents: read
packages: write
id-token: write
attestations: write
with:
image: ${{ needs.meta.outputs.gateway_image }}
version: ${{ needs.meta.outputs.identity }}
context: examples/dsh-agents-as-a-service/gateway
dockerfile: Dockerfile
tags: |
${{ needs.meta.outputs.gateway_image }}:main

84
.github/workflows/build-main.yaml vendored Normal file
View File

@@ -0,0 +1,84 @@
# Development channel for the main DSH Web image: every merge to main that
# touches the image definition publishes ghcr.io/<owner>/<repo>:main.
# Releases are published by release.yaml with immutable version tags.
name: Build main image
on:
workflow_dispatch:
push:
branches: [main]
paths:
- Dockerfile
- versions.env
- .github/workflows/build-main.yaml
- .github/workflows/_build-image.yaml
permissions:
contents: read
packages: write
id-token: write
attestations: write
concurrency:
group: build-main-${{ github.ref }}
cancel-in-progress: true
jobs:
meta:
name: Read versions
runs-on: ubuntu-latest
timeout-minutes: 5
outputs:
identity: ${{ steps.meta.outputs.identity }}
dsh_version: ${{ steps.meta.outputs.dsh_version }}
pnpm_version: ${{ steps.meta.outputs.pnpm_version }}
release_revision: ${{ steps.meta.outputs.release_revision }}
ghcr_image: ${{ steps.meta.outputs.ghcr_image }}
gateway_image: ${{ steps.meta.outputs.gateway_image }}
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Read versions.env
id: meta
shell: bash
run: |
set -euo pipefail
set -a
# shellcheck disable=SC1091
source versions.env
set +a
identity="$DSH_VERSION"
[ -z "$RELEASE_REVISION" ] || identity="$DSH_VERSION-$RELEASE_REVISION"
{
echo "identity=$identity"
echo "dsh_version=$DSH_VERSION"
echo "pnpm_version=$PNPM_VERSION"
echo "release_revision=$RELEASE_REVISION"
echo "ghcr_image=$GHCR_IMAGE"
echo "gateway_image=$GATEWAY_IMAGE"
} >> "$GITHUB_OUTPUT"
build:
name: Build and push main image
needs: meta
uses: ./.github/workflows/_build-image.yaml
permissions:
contents: read
packages: write
id-token: write
attestations: write
with:
image: ${{ needs.meta.outputs.ghcr_image }}
version: ${{ needs.meta.outputs.identity }}
context: .
dockerfile: Dockerfile
build-args: |
DSH_VERSION=${{ needs.meta.outputs.dsh_version }}
RELEASE_REVISION=${{ needs.meta.outputs.release_revision }}
PNPM_VERSION=${{ needs.meta.outputs.pnpm_version }}
VCS_REF=${{ github.sha }}
tags: |
${{ needs.meta.outputs.ghcr_image }}:main

257
.github/workflows/ci.yaml vendored Normal file
View File

@@ -0,0 +1,257 @@
# Continuous integration: lint, unit tests, manifest dry-run, gateway image
# smoke test and dependency review. Job names are the required branch-protection
# checks documented in docs/gitflow.md.
name: CI
on:
pull_request:
push:
branches: [main]
workflow_dispatch:
permissions:
contents: read
concurrency:
group: ci-${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
lint:
name: lint
runs-on: ubuntu-latest
timeout-minutes: 20
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Set up Python
uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
with:
python-version: "3.12"
- name: Install Python linters
run: |
set -euo pipefail
python -m pip install --upgrade pip
pip install "yamllint==1.38.0" "pyyaml==6.0.3"
- name: Install actionlint
env:
ACTIONLINT_VERSION: v1.7.12
run: |
set -euo pipefail
version="${ACTIONLINT_VERSION#v}"
curl -fsSL -o /tmp/actionlint.tar.gz "https://github.com/rhysd/actionlint/releases/download/$ACTIONLINT_VERSION/actionlint_${version}_linux_amd64.tar.gz"
tar -xzf /tmp/actionlint.tar.gz -C /tmp actionlint
install -m 0755 /tmp/actionlint /usr/local/bin/actionlint
actionlint -version
- name: Install hadolint
env:
HADOLINT_VERSION: v2.12.0
run: |
set -euo pipefail
curl -fsSL -o /tmp/hadolint "https://github.com/hadolint/hadolint/releases/download/$HADOLINT_VERSION/hadolint-Linux-x86_64"
install -m 0755 /tmp/hadolint /usr/local/bin/hadolint
hadolint --version
- name: Repository checks
run: |
set -euo pipefail
./scripts/check-versions.sh
./scripts/check-examples.sh
python scripts/check-manifests.py
- name: Shellcheck
run: shellcheck -x build.sh scripts/*.sh examples/*/scripts/*.sh
- name: Actionlint
run: actionlint -color
- name: Yamllint
run: yamllint .
- name: Hadolint
run: |
set -euo pipefail
hadolint Dockerfile
hadolint examples/dsh-agents-as-a-service/gateway/Dockerfile
- name: Documentation links
uses: lycheeverse/lychee-action@e7477775783ea5526144ba13e8db5eec57747ce8 # v2
with:
args: >-
--offline --no-progress --verbose
./README.md ./CONTRIBUTING.md ./SECURITY.md ./AGENTS.md ./docs ./examples
python:
name: python
runs-on: ubuntu-latest
timeout-minutes: 20
defaults:
run:
working-directory: examples/dsh-agents-as-a-service/gateway
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Set up Python
uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
with:
python-version: "3.12"
cache: pip
cache-dependency-path: |
examples/dsh-agents-as-a-service/gateway/requirements.txt
examples/dsh-agents-as-a-service/gateway/requirements-dev.txt
- name: Install dependencies
run: |
set -euo pipefail
python -m pip install --upgrade pip
pip install -r requirements.txt -r requirements-dev.txt
- name: Lint
run: |
set -euo pipefail
python -m ruff check .
python -m ruff format --check .
- name: Unit tests with coverage
run: >-
python -m pytest -q
--cov=. --cov-report=xml:coverage.xml --cov-report=term
--junitxml=junit.xml
- name: Coverage summary
run: |
set -euo pipefail
{
echo "### Gateway test coverage"
python -m coverage report
} >> "$GITHUB_STEP_SUMMARY"
- name: Upload test artefacts
if: always()
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: gateway-test-results
path: |
examples/dsh-agents-as-a-service/gateway/junit.xml
examples/dsh-agents-as-a-service/gateway/coverage.xml
if-no-files-found: warn
retention-days: 7
manifests:
name: manifests
runs-on: ubuntu-latest
timeout-minutes: 25
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Create kind cluster
uses: helm/kind-action@1544676c07bb3570fcb70166beaa07c4528fd4aa # v1.15.1
with:
version: v0.33.0
cluster_name: dsh-ci
- name: Read versions.env
run: |
set -euo pipefail
while IFS= read -r line; do
case "$line" in ''|'#'*) continue ;; esac
echo "$line" >> "$GITHUB_ENV"
done < versions.env
- name: Install agent-sandbox and Gateway API CRDs
run: |
set -euo pipefail
kubectl apply -f "https://github.com/kubernetes-sigs/agent-sandbox/releases/download/$AGENT_SANDBOX_VERSION/sandbox-with-extensions.yaml"
kubectl wait --for=condition=Established crd/sandboxtemplates.extensions.agents.x-k8s.io --timeout=180s
kubectl wait --for=condition=Established crd/sandboxwarmpools.extensions.agents.x-k8s.io --timeout=180s
kubectl wait --for=condition=Established crd/sandboxclaims.extensions.agents.x-k8s.io --timeout=180s
kubectl wait --for=condition=Established crd/sandboxes.agents.x-k8s.io --timeout=180s
kubectl apply -f "https://github.com/kubernetes-sigs/gateway-api/releases/download/$GATEWAY_API_VERSION/standard-install.yaml"
kubectl wait --for=condition=Established crd/gateways.gateway.networking.k8s.io --timeout=180s
kubectl wait --for=condition=Established crd/httproutes.gateway.networking.k8s.io --timeout=180s
- name: Set up Python
uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
with:
python-version: "3.12"
- name: Install check dependencies
run: pip install "pyyaml==6.0.3"
- name: Validate manifest shape
run: python scripts/check-manifests.py
- name: Validate manifests (server dry-run)
run: |
set -euo pipefail
kubectl create namespace dsh-demo
shopt -s globstar nullglob
count=0
for file in examples/*/manifests/**/*.yaml; do
case "$file" in *.yaml.example) continue ;; esac
echo "== $file =="
kubectl apply --dry-run=server -f "$file"
count=$((count + 1))
done
if [ "$count" -eq 0 ]; then
echo "no manifests were validated" >&2
exit 1
fi
echo "validated $count manifests"
gateway-image:
name: gateway-image
runs-on: ubuntu-latest
timeout-minutes: 20
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Build gateway image
run: docker build -t dsh-aaas-gateway:ci examples/dsh-agents-as-a-service/gateway/
- name: Smoke-test /healthz without a cluster
run: |
set -euo pipefail
cat > /tmp/kubeconfig <<'EOF'
apiVersion: v1
kind: Config
clusters:
- name: dummy
cluster:
server: https://127.0.0.1:6443
insecure-skip-tls-verify: true
contexts:
- name: dummy
context:
cluster: dummy
user: dummy
users:
- name: dummy
user: {}
current-context: dummy
EOF
docker run -d --name gateway-smoke \
-p 18080:8080 \
-e NAMESPACE=dsh-demo \
-e POOL=dsh-pool \
-e KUBECONFIG=/tmp/kubeconfig \
-v /tmp/kubeconfig:/tmp/kubeconfig:ro \
dsh-aaas-gateway:ci
trap 'docker logs gateway-smoke || true; docker rm -f gateway-smoke || true' EXIT
code=""
for _ in $(seq 1 30); do
code=$(curl -s -o /dev/null -w '%{http_code}' http://localhost:18080/healthz || true)
[ "$code" = "200" ] && break
sleep 1
done
if [ "$code" != "200" ]; then
echo "gateway /healthz returned '$code' instead of 200" >&2
exit 1
fi
echo "gateway image healthy: GET /healthz -> 200"
dependency-review:
name: dependency-review
if: github.event_name == 'pull_request'
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
contents: read
pull-requests: write
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Dependency review
uses: actions/dependency-review-action@a1d282b36b6f3519aa1f3fc636f609c47dddb294 # v4

View File

@@ -57,7 +57,7 @@ jobs:
# your codebase is analyzed, see https://docs.github.com/en/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/codeql-code-scanning-for-compiled-languages
steps:
- name: Checkout repository
uses: actions/checkout@v7
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
# Add any setup steps before running the `github/codeql-action/init` action.
# This includes steps like installing compilers or runtimes (`actions/setup-node`
@@ -67,7 +67,7 @@ jobs:
# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL
uses: github/codeql-action/init@v4
uses: github/codeql-action/init@9b3254085494b55a0f7c4c2b3d2d52cc712ce370 # v4
with:
languages: ${{ matrix.language }}
build-mode: ${{ matrix.build-mode }}
@@ -96,6 +96,6 @@ jobs:
exit 1
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v4
uses: github/codeql-action/analyze@9b3254085494b55a0f7c4c2b3d2d52cc712ce370 # v4
with:
category: "/language:${{matrix.language}}"

78
.github/workflows/e2e-kind.yaml vendored Normal file
View File

@@ -0,0 +1,78 @@
# Weekly end-to-end run of the agents-as-a-service example against a real kind
# cluster with agent-sandbox installed. Scheduled (not PR-blocking) so upstream
# controller churn does not block merges; also available on demand.
name: E2E (kind)
on:
schedule:
- cron: "0 6 * * 1"
workflow_dispatch:
permissions:
contents: read
concurrency:
group: e2e-kind
cancel-in-progress: false
jobs:
e2e:
name: Example walkthrough on kind
runs-on: ubuntu-latest
timeout-minutes: 45
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Create kind cluster
uses: helm/kind-action@1544676c07bb3570fcb70166beaa07c4528fd4aa # v1.15.1
with:
version: v0.33.0
cluster_name: dsh-e2e
- name: Read versions.env
run: |
set -euo pipefail
while IFS= read -r line; do
case "$line" in ''|'#'*) continue ;; esac
echo "$line" >> "$GITHUB_ENV"
done < versions.env
- name: Install agent-sandbox
run: |
set -euo pipefail
kubectl apply -f "https://github.com/kubernetes-sigs/agent-sandbox/releases/download/$AGENT_SANDBOX_VERSION/sandbox-with-extensions.yaml"
kubectl -n agent-sandbox-system rollout status deployment --timeout=180s
kubectl wait --for=condition=Established crd/sandboxtemplates.extensions.agents.x-k8s.io --timeout=180s
kubectl wait --for=condition=Established crd/sandboxwarmpools.extensions.agents.x-k8s.io --timeout=180s
kubectl wait --for=condition=Established crd/sandboxclaims.extensions.agents.x-k8s.io --timeout=180s
kubectl wait --for=condition=Established crd/sandboxes.agents.x-k8s.io --timeout=180s
- name: Set up Python (WebSocket checks)
uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
with:
python-version: "3.12"
- name: Install WebSocket client
run: pip install "websockets==17.3"
- name: Run the example walkthrough
env:
KIND_CLUSTER: dsh-e2e
run: examples/dsh-agents-as-a-service/scripts/e2e-kind.sh
- name: Collect cluster state
if: always()
run: |
set -euo pipefail
{
echo "== resources =="
kubectl get all,pvc -n dsh-demo -o wide 2>&1 || true
echo "== events =="
kubectl get events -n dsh-demo --sort-by=.lastTimestamp 2>&1 || true
echo "== gateway logs =="
kubectl -n dsh-demo logs deploy/dsh-gateway --tail=200 2>&1 || true
} > /tmp/e2e-cluster.txt
- name: Upload cluster state
if: always()
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: e2e-cluster-state
path: /tmp/e2e-cluster.txt
if-no-files-found: warn
retention-days: 7

View File

@@ -1,277 +0,0 @@
# This workflow builds the multi-arch gateway image (Starlette ASGI, served by
# uvicorn on :8080) and pushes it to GHCR as
# ghcr.io/<owner>/<repo>:gateway. The DSH Web image is built by multi-build.yaml.
# It uses Docker Buildx with native AMD64 and ARM64 runners, without QEMU.
name: Gateway Build multi arch Docker Image with separate Github Runners
on:
workflow_dispatch:
push:
branches:
- main
paths:
- 'examples/dsh-agents-as-a-service/gateway/**'
- '.github/workflows/multi-build-gateway.yaml'
env:
# ghcr.io/<owner>/<repo>; this workflow always publishes the `:gateway` tag.
GHCR_IMAGE: ghcr.io/${{ github.repository }}
permissions:
# Global permissions for the workflow, which can be overridden at the job level
contents: read
concurrency:
# This concurrency group ensures that only one job in the group runs at a time.
# If a new job is triggered, the previous one will be canceled.
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
# The build job builds the Docker image for each platform specified in the matrix.
build:
strategy:
fail-fast: false
matrix:
platform:
- linux/amd64
- linux/arm64
# The matrix includes two platforms: linux/amd64 and linux/arm64.
# The build job will run for each platform in the matrix.
permissions:
# Permissions for the build job, which can be overridden at the step level
# The permissions are set to allow the job to write to the GitHub Container Registry (GHCR) and read from the repository.
attestations: write
actions: read
checks: write
contents: write
deployments: none
id-token: write
issues: read
discussions: read
packages: write
pages: none
pull-requests: read
repository-projects: read
security-events: read
statuses: read
runs-on: ${{ matrix.platform == 'linux/amd64' && 'ubuntu-latest' || matrix.platform == 'linux/arm64' && 'ubuntu-24.04-arm' }}
# The job runs on different runners based on the platform.
# For linux/amd64, it runs on the latest Ubuntu runner.
# For linux/arm64, it runs on an Ubuntu 24.04 ARM runner.
# The runner is selected based on the platform specified in the matrix.
name: Build Docker image for ${{ matrix.platform }}
steps:
-
name: Prepare environment for current platform
# This step sets up the environment for the current platform being built.
# It replaces the '/' character in the platform name with '-' and sets it as an environment variable.
# This is useful for naming artifacts and other resources that cannot contain '/'.
# The environment variable PLATFORMS_PAIR will be used later in the workflow.
id: prepare
run: |
platform=${{ matrix.platform }}
echo "PLATFORM_PAIR=${platform//\//-}" >> $GITHUB_ENV
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
# This step checks out the code from the repository.
# It uses the actions/checkout action to clone the repository into the runner's workspace.
- name: Docker meta default
# This step generates metadata for the Docker image.
# It uses the docker/metadata-action to create metadata based on the repository information.
# The metadata includes information such as the image name, tags, and labels.
# The metadata will be used later in the workflow to build and push the Docker image.
id: meta
uses: docker/metadata-action@dc802804100637a589fabce1cb79ff13a1411302 # v6.2.0
with:
images: ${{ env.GHCR_IMAGE }}
tags: |
type=raw,value=gateway
- name: Set up Docker Context for Buildx
# This step sets up a Docker context for Buildx.
# It creates a new context named "builders" that will be used for building the Docker image.
# The context allows Buildx to use the Docker daemon for building images.
id: buildx-context
run: |
docker context create builders
- name: Set up Docker Buildx
# This step sets up Docker Buildx, which is a Docker CLI plugin for extended build capabilities with BuildKit.
# It uses the docker/setup-buildx-action to configure Buildx with the specified context and platforms.
# The platforms are specified in the matrix and will be used for building the Docker image.
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
with:
endpoint: builders
platforms: ${{ matrix.platform }}
- name: Login to GitHub Container Registry
# This step logs in to the GitHub Container Registry (GHCR) using the docker/login-action.
# It uses the GitHub actor's username and the GITHUB_TOKEN secret for authentication.
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Build and push by digest
# This step builds and pushes the Docker image using Buildx.
# It uses the docker/build-push-action to build the image with the specified context and platforms.
# The image is built with the labels and annotations generated in the previous steps.
# The outputs are configured to push the image by digest, which allows for better caching and versioning.
# The cache-from and cache-to options are used to enable caching for the build process.
# The cache is stored in GitHub Actions cache and is scoped to the repository, branch, and platform.
id: build
uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7.4.0
env:
DOCKER_BUILDKIT: 1
with:
context: ./examples/dsh-agents-as-a-service/gateway/
platforms: ${{ matrix.platform }}
labels: ${{ steps.meta.outputs.labels }}
annotations: ${{ steps.meta.outputs.annotations }}
outputs: type=image,name=${{ env.GHCR_IMAGE }},push-by-digest=true,name-canonical=true,push=true,oci-mediatypes=true
cache-from: type=gha,scope=${{ github.repository }}-${{ github.ref_name }}-${{ matrix.platform }}
cache-to: type=gha,scope=${{ github.repository }}-${{ github.ref_name }}-${{ matrix.platform }}
- name: Export digest
# This step exports the digest of the built image to a file.
# It creates a directory in /tmp/digests and saves the digest of the image to a file.
# The digest is obtained from the output of the build step.
# The digest is used to uniquely identify the built image and can be used for further processing or verification.
run: |
mkdir -p /tmp/digests
digest="${{ steps.build.outputs.digest }}"
touch "/tmp/digests/${digest#sha256:}"
- name: Upload digest
# This step uploads the digest file to the GitHub Actions artifact storage.
# It uses the actions/upload-artifact action to upload the file created in the previous step.
# The artifact is named digests-${{ env.PLATFORM_PAIR }}, where PLATFORM_PAIR is the platform name with '/' replaced by '-'.
# The artifact is retained for 1 day, and if no files are found, it will throw an error.
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: digests-${{ env.PLATFORM_PAIR }}
path: /tmp/digests/*
if-no-files-found: error
retention-days: 1
merge:
# This job merges the Docker manifests for the different platforms built in the previous job.
name: Merge Docker manifests
runs-on: ubuntu-latest
permissions:
attestations: write
actions: read
checks: read
contents: read
deployments: none
id-token: write
issues: read
discussions: read
packages: write
pages: none
pull-requests: read
repository-projects: read
security-events: read
statuses: read
needs:
- build
# This job depends on the build job to complete before it starts.
# It ensures that the Docker images for all platforms are built before merging the manifests.
steps:
- name: Download digests
# This step downloads the digest files uploaded in the build job.
# It uses the actions/download-artifact action to download the artifacts with the pattern digests-*.
# The downloaded files are merged into the /tmp/digests directory.
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with:
path: /tmp/digests
pattern: digests-*
merge-multiple: true
- name: Docker meta
# This step generates metadata for the Docker image.
# It uses the docker/metadata-action to create metadata based on the repository information.
# The metadata includes information such as the image name, tags, and labels.
id: meta
uses: docker/metadata-action@dc802804100637a589fabce1cb79ff13a1411302 # v6.2.0
with:
images: ${{ env.GHCR_IMAGE }}
annotations: |
type=org.opencontainers.image.description,value=${{ github.event.repository.description || 'No description provided' }}
tags: |
type=raw,value=gateway
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
# This step sets up Docker Buildx, which is a Docker CLI plugin for extended build capabilities with BuildKit.
with:
driver-opts: |
network=host
- name: Login to GitHub Container Registry
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
# This step logs in to the GitHub Container Registry (GHCR) using the docker/login-action.
# It uses the GitHub actor's username and the GITHUB_TOKEN secret for authentication.
# The login is necessary to push the merged manifest list to GHCR.
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Get execution timestamp with RFC3339 format
# This step gets the current execution timestamp in RFC3339 format.
# It uses the date command to get the current UTC time and formats it as a string.
# The timestamp is used for annotating the Docker manifest list.
id: timestamp
run: |
echo "timestamp=$(date -u +"%Y-%m-%dT%H:%M:%SZ")" >> $GITHUB_OUTPUT
- name: Create manifest list and pushs
# This step creates a manifest list for the Docker images built for different platforms.
# It uses the docker buildx imagetools create command to create the manifest list.
# The manifest list is annotated with metadata such as description, creation timestamp, and source URL.
# The annotations are obtained from the metadata generated in the previous steps.
# The manifest list is pushed to the GitHub Container Registry (GHCR) with the specified tags.
working-directory: /tmp/digests
id: manifest-annotate
continue-on-error: true
run: |
docker buildx imagetools create \
$(jq -cr '.tags | map("-t " + .) | join(" ")' <<< "$DOCKER_METADATA_OUTPUT_JSON") \
--annotation='index:org.opencontainers.image.description=${{ github.event.repository.description }}' \
--annotation='index:org.opencontainers.image.created=${{ steps.timestamp.outputs.timestamp }}' \
--annotation='index:org.opencontainers.image.url=${{ github.event.repository.url }}' \
--annotation='index:org.opencontainers.image.source=${{ github.event.repository.url }}' \
$(printf '${{ env.GHCR_IMAGE }}@sha256:%s ' *)
- name: Create manifest list and push without annotations
# This step creates a manifest list for the Docker images built for different platforms.
# It uses the docker buildx imagetools create command to create the manifest list.
# The manifest list is created without annotations if the previous step fails.
# The manifest list is pushed to the GitHub Container Registry (GHCR) with the specified tags.
if: steps.manifest-annotate.outcome == 'failure'
working-directory: /tmp/digests
run: |
docker buildx imagetools create $(jq -cr '.tags | map("-t " + .) | join(" ")' <<< "$DOCKER_METADATA_OUTPUT_JSON") \
$(printf '${{ env.GHCR_IMAGE }}@sha256:%s ' *)
- name: Inspect image
# This step inspects the created manifest list to verify its contents.
# It uses the docker buildx imagetools inspect command to display information about the manifest list.
# The inspection output will show the platforms and tags associated with the manifest list.
id: inspect
run: |
docker buildx imagetools inspect '${{ env.GHCR_IMAGE }}:${{ steps.meta.outputs.version }}'

View File

@@ -1,295 +0,0 @@
# This workflow builds a multi-arch Docker image using GitHub Actions and separated Github Runners with native support for ARM64 and AMD64 architectures, without using QEMU emulation.
# It uses Docker Buildx to build and push the image to GitHub Container Registry (GHCR).
name: Build multi arch Docker Image with separate Github Runners
on:
workflow_dispatch:
inputs:
dsh_version:
description: "DSH git ref to build (tag or branch)"
required: true
default: dsh-v0.2.1-rc.2
type: string
pnpm_version:
description: "pnpm version used to build DSH"
required: true
default: "11.28.5"
type: string
push:
branches:
- main
paths:
- 'Dockerfile'
- '.github/workflows/multi-build.yaml'
env:
# The name of the Docker image to be built and pushed to GHCR
# The image name is derived from the GitHub repository name and the GitHub Container Registry (GHCR) URL.
# The image name will be in the format: ghcr.io/<owner>/<repo>
GHCR_IMAGE: ghcr.io/${{ github.repository }}
# workflow_dispatch inputs win; pushes to main fall back to the pinned
# defaults. Both values flow into the Docker build args and the image tags.
DSH_VERSION: ${{ inputs.dsh_version || 'dsh-v0.2.1-rc.2' }}
PNPM_VERSION: ${{ inputs.pnpm_version || '11.28.5' }}
permissions:
# Global permissions for the workflow, which can be overridden at the job level
contents: read
concurrency:
# This concurrency group ensures that only one job in the group runs at a time.
# If a new job is triggered, the previous one will be canceled.
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
# The build job builds the Docker image for each platform specified in the matrix.
build:
strategy:
fail-fast: false
matrix:
platform:
- linux/amd64
- linux/arm64
# The matrix includes two platforms: linux/amd64 and linux/arm64.
# The build job will run for each platform in the matrix.
permissions:
# Permissions for the build job, which can be overridden at the step level
# The permissions are set to allow the job to write to the GitHub Container Registry (GHCR) and read from the repository.
attestations: write
actions: read
checks: write
contents: write
deployments: none
id-token: write
issues: read
discussions: read
packages: write
pages: none
pull-requests: read
repository-projects: read
security-events: read
statuses: read
runs-on: ${{ matrix.platform == 'linux/amd64' && 'ubuntu-latest' || matrix.platform == 'linux/arm64' && 'ubuntu-24.04-arm' }}
# The job runs on different runners based on the platform.
# For linux/amd64, it runs on the latest Ubuntu runner.
# For linux/arm64, it runs on an Ubuntu 24.04 ARM runner.
# The runner is selected based on the platform specified in the matrix.
name: Build Docker image for ${{ matrix.platform }}
steps:
-
name: Prepare environment for current platform
# This step sets up the environment for the current platform being built.
# It replaces the '/' character in the platform name with '-' and sets it as an environment variable.
# This is useful for naming artifacts and other resources that cannot contain '/'.
# The environment variable PLATFORMS_PAIR will be used later in the workflow.
id: prepare
run: |
platform=${{ matrix.platform }}
echo "PLATFORM_PAIR=${platform//\//-}" >> $GITHUB_ENV
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
# This step checks out the code from the repository.
# It uses the actions/checkout action to clone the repository into the runner's workspace.
- name: Docker meta default
# This step generates metadata for the Docker image.
# It uses the docker/metadata-action to create metadata based on the repository information.
# The metadata includes information such as the image name, tags, and labels.
# The metadata will be used later in the workflow to build and push the Docker image.
id: meta
uses: docker/metadata-action@dc802804100637a589fabce1cb79ff13a1411302 # v6.2.0
with:
images: ${{ env.GHCR_IMAGE }}
- name: Set up Docker Context for Buildx
# This step sets up a Docker context for Buildx.
# It creates a new context named "builders" that will be used for building the Docker image.
# The context allows Buildx to use the Docker daemon for building images.
id: buildx-context
run: |
docker context create builders
- name: Set up Docker Buildx
# This step sets up Docker Buildx, which is a Docker CLI plugin for extended build capabilities with BuildKit.
# It uses the docker/setup-buildx-action to configure Buildx with the specified context and platforms.
# The platforms are specified in the matrix and will be used for building the Docker image.
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
with:
endpoint: builders
platforms: ${{ matrix.platform }}
- name: Login to GitHub Container Registry
# This step logs in to the GitHub Container Registry (GHCR) using the docker/login-action.
# It uses the GitHub actor's username and the GITHUB_TOKEN secret for authentication.
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Build and push by digest
# This step builds and pushes the Docker image using Buildx.
# It uses the docker/build-push-action to build the image with the specified context and platforms.
# The image is built with the labels and annotations generated in the previous steps.
# The outputs are configured to push the image by digest, which allows for better caching and versioning.
# The cache-from and cache-to options are used to enable caching for the build process.
# The cache is stored in GitHub Actions cache and is scoped to the repository, branch, and platform.
id: build
uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7.4.0
env:
DOCKER_BUILDKIT: 1
DSH_PORT: 3080
with:
context: .
platforms: ${{ matrix.platform }}
labels: ${{ steps.meta.outputs.labels }}
build-args: |
DSH_VERSION=${{ env.DSH_VERSION }}
PNPM_VERSION=${{ env.PNPM_VERSION }}
annotations: ${{ steps.meta.outputs.annotations }}
outputs: type=image,name=${{ env.GHCR_IMAGE }},push-by-digest=true,name-canonical=true,push=true,oci-mediatypes=true
cache-from: type=gha,scope=${{ github.repository }}-${{ github.ref_name }}-${{ matrix.platform }}
cache-to: type=gha,scope=${{ github.repository }}-${{ github.ref_name }}-${{ matrix.platform }}
- name: Export digest
# This step exports the digest of the built image to a file.
# It creates a directory in /tmp/digests and saves the digest of the image to a file.
# The digest is obtained from the output of the build step.
# The digest is used to uniquely identify the built image and can be used for further processing or verification.
run: |
mkdir -p /tmp/digests
digest="${{ steps.build.outputs.digest }}"
touch "/tmp/digests/${digest#sha256:}"
- name: Upload digest
# This step uploads the digest file to the GitHub Actions artifact storage.
# It uses the actions/upload-artifact action to upload the file created in the previous step.
# The artifact is named digests-${{ env.PLATFORM_PAIR }}, where PLATFORM_PAIR is the platform name with '/' replaced by '-'.
# The artifact is retained for 1 day, and if no files are found, it will throw an error.
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: digests-${{ env.PLATFORM_PAIR }}
path: /tmp/digests/*
if-no-files-found: error
retention-days: 1
merge:
# This job merges the Docker manifests for the different platforms built in the previous job.
name: Merge Docker manifests
runs-on: ubuntu-latest
permissions:
attestations: write
actions: read
checks: read
contents: read
deployments: none
id-token: write
issues: read
discussions: read
packages: write
pages: none
pull-requests: read
repository-projects: read
security-events: read
statuses: read
needs:
- build
# This job depends on the build job to complete before it starts.
# It ensures that the Docker images for all platforms are built before merging the manifests.
steps:
- name: Download digests
# This step downloads the digest files uploaded in the build job.
# It uses the actions/download-artifact action to download the artifacts with the pattern digests-*.
# The downloaded files are merged into the /tmp/digests directory.
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with:
path: /tmp/digests
pattern: digests-*
merge-multiple: true
- name: Docker meta
# This step generates metadata for the Docker image.
# It uses the docker/metadata-action to create metadata based on the repository information.
# The metadata includes information such as the image name, tags, and labels.
id: meta
uses: docker/metadata-action@dc802804100637a589fabce1cb79ff13a1411302 # v6.2.0
with:
images: ${{ env.GHCR_IMAGE }}
annotations: |
type=org.opencontainers.image.description,value=${{ github.event.repository.description || 'No description provided' }}
tags: |
type=raw,value=main,enable=${{ github.ref_name == 'main' }}
type=raw,value=latest,enable=${{ github.ref_name == 'main' }}
type=raw,value=${{ env.DSH_VERSION }},enable=${{ github.ref_name == 'main' || github.event_name == 'workflow_dispatch' }}
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
# This step sets up Docker Buildx, which is a Docker CLI plugin for extended build capabilities with BuildKit.
with:
driver-opts: |
network=host
- name: Login to GitHub Container Registry
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
# This step logs in to the GitHub Container Registry (GHCR) using the docker/login-action.
# It uses the GitHub actor's username and the GITHUB_TOKEN secret for authentication.
# The login is necessary to push the merged manifest list to GHCR.
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Get execution timestamp with RFC3339 format
# This step gets the current execution timestamp in RFC3339 format.
# It uses the date command to get the current UTC time and formats it as a string.
# The timestamp is used for annotating the Docker manifest list.
id: timestamp
run: |
echo "timestamp=$(date -u +"%Y-%m-%dT%H:%M:%SZ")" >> $GITHUB_OUTPUT
- name: Create manifest list and pushs
# This step creates a manifest list for the Docker images built for different platforms.
# It uses the docker buildx imagetools create command to create the manifest list.
# The manifest list is annotated with metadata such as description, creation timestamp, and source URL.
# The annotations are obtained from the metadata generated in the previous steps.
# The manifest list is pushed to the GitHub Container Registry (GHCR) with the specified tags.
working-directory: /tmp/digests
id: manifest-annotate
continue-on-error: true
run: |
docker buildx imagetools create \
$(jq -cr '.tags | map("-t " + .) | join(" ")' <<< "$DOCKER_METADATA_OUTPUT_JSON") \
--annotation='index:org.opencontainers.image.description=${{ github.event.repository.description }}' \
--annotation='index:org.opencontainers.image.created=${{ steps.timestamp.outputs.timestamp }}' \
--annotation='index:org.opencontainers.image.url=${{ github.event.repository.url }}' \
--annotation='index:org.opencontainers.image.source=${{ github.event.repository.url }}' \
$(printf '${{ env.GHCR_IMAGE }}@sha256:%s ' *)
- name: Create manifest list and push without annotations
# This step creates a manifest list for the Docker images built for different platforms.
# It uses the docker buildx imagetools create command to create the manifest list.
# The manifest list is created without annotations if the previous step fails.
# The manifest list is pushed to the GitHub Container Registry (GHCR) with the specified tags.
if: steps.manifest-annotate.outcome == 'failure'
working-directory: /tmp/digests
run: |
docker buildx imagetools create $(jq -cr '.tags | map("-t " + .) | join(" ")' <<< "$DOCKER_METADATA_OUTPUT_JSON") \
$(printf '${{ env.GHCR_IMAGE }}@sha256:%s ' *)
- name: Inspect image
# This step inspects the created manifest list to verify its contents.
# It uses the docker buildx imagetools inspect command to display information about the manifest list.
# The inspection output will show the platforms and tags associated with the manifest list.
id: inspect
run: |
docker buildx imagetools inspect '${{ env.GHCR_IMAGE }}:${{ steps.meta.outputs.version }}'

38
.github/workflows/pr-title.yaml vendored Normal file
View File

@@ -0,0 +1,38 @@
# Enforce Conventional Commit pull request titles so git-cliff can group
# release notes. The PR is squash-merged, so the title becomes the commit.
name: PR title
on:
pull_request:
types: [opened, edited, synchronize, reopened]
permissions:
contents: read
pull-requests: read
jobs:
lint:
name: pr-title
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- name: Validate title
uses: amannn/action-semantic-pull-request@48f256284bd46cdaab1048c3721360e808335d50 # v6.1.1
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
types: |
feat
fix
docs
refactor
perf
test
build
ci
chore
revert
requireScope: false
subjectPattern: ^(?![A-Z]).+$
subjectPatternError: |
The PR title subject must start in lowercase and must not end with a period.

271
.github/workflows/release.yaml vendored Normal file
View File

@@ -0,0 +1,271 @@
# Release pipeline. Pushing a tag that matches the identity in versions.env
# publishes both images (version + latest), creates the GitHub Release from
# git-cliff, and opens a PR that refreshes CHANGELOG.md on main.
#
# The tag must be dsh-vX.Y.Z or dsh-vX.Y.Z-rN and must match versions.env;
# tags are immutable, so a broken release is fixed with a new -rN revision.
name: Release
on:
push:
tags:
- "dsh-v*"
workflow_dispatch:
inputs:
tag:
description: Existing tag to release (must match versions.env)
required: true
type: string
publish_legacy_gateway_alias:
description: Also publish the legacy :gateway alias (one release only)
required: false
default: true
type: boolean
permissions:
contents: write
packages: write
id-token: write
attestations: write
concurrency:
group: release-${{ github.ref }}
cancel-in-progress: false
jobs:
meta:
name: Validate release identity
runs-on: ubuntu-latest
timeout-minutes: 15
outputs:
identity: ${{ steps.meta.outputs.identity }}
dsh_version: ${{ steps.meta.outputs.dsh_version }}
pnpm_version: ${{ steps.meta.outputs.pnpm_version }}
release_revision: ${{ steps.meta.outputs.release_revision }}
ghcr_image: ${{ steps.meta.outputs.ghcr_image }}
gateway_image: ${{ steps.meta.outputs.gateway_image }}
tag: ${{ steps.meta.outputs.tag }}
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
fetch-depth: 0
persist-credentials: false
- name: Validate tag against versions.env
id: meta
shell: bash
run: |
set -euo pipefail
set -a
# shellcheck disable=SC1091
source versions.env
set +a
tag="${{ inputs.tag || github.ref_name }}"
identity="$DSH_VERSION"
[ -z "$RELEASE_REVISION" ] || identity="$DSH_VERSION-$RELEASE_REVISION"
if [ "$tag" != "$identity" ]; then
echo "tag '$tag' does not match versions.env identity '$identity'" >&2
exit 1
fi
if git rev-parse -q --verify "refs/tags/$tag" >/dev/null; then
git fetch origin main '+refs/heads/release/*:refs/remotes/origin/release/*'
commit=$(git rev-list -n1 "refs/tags/$tag")
allowed=0
for ref in origin/main origin/release/*; do
if git rev-parse -q --verify "$ref" >/dev/null && git merge-base --is-ancestor "$commit" "$ref"; then
allowed=1
break
fi
done
if [ "$allowed" -ne 1 ]; then
echo "tag commit $commit is not on main or a release branch" >&2
exit 1
fi
else
echo "tag '$tag' does not exist in this repository" >&2
exit 1
fi
if ! git ls-remote --tags --exit-code "$DSH_REPO" "$DSH_VERSION" >/dev/null; then
echo "upstream ref '$DSH_VERSION' not found in $DSH_REPO" >&2
exit 1
fi
{
echo "identity=$identity"
echo "dsh_version=$DSH_VERSION"
echo "pnpm_version=$PNPM_VERSION"
echo "release_revision=$RELEASE_REVISION"
echo "ghcr_image=$GHCR_IMAGE"
echo "gateway_image=$GATEWAY_IMAGE"
echo "tag=$tag"
} >> "$GITHUB_OUTPUT"
- name: Verify repository pins
run: ./scripts/check-versions.sh
tests:
name: Unit tests and lint
needs: meta
runs-on: ubuntu-latest
timeout-minutes: 15
defaults:
run:
working-directory: examples/dsh-agents-as-a-service/gateway
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Set up Python
uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
with:
python-version: "3.12"
cache: pip
cache-dependency-path: |
examples/dsh-agents-as-a-service/gateway/requirements.txt
examples/dsh-agents-as-a-service/gateway/requirements-dev.txt
- name: Install dependencies
run: |
set -euo pipefail
python -m pip install --upgrade pip
pip install -r requirements.txt -r requirements-dev.txt
- name: Lint
run: |
set -euo pipefail
python -m ruff check .
python -m ruff format --check .
- name: Unit tests
run: python -m pytest -q
build-main:
name: Publish main image
needs: [meta, tests]
uses: ./.github/workflows/_build-image.yaml
permissions:
contents: read
packages: write
id-token: write
attestations: write
with:
image: ${{ needs.meta.outputs.ghcr_image }}
version: ${{ needs.meta.outputs.identity }}
context: .
dockerfile: Dockerfile
build-args: |
DSH_VERSION=${{ needs.meta.outputs.dsh_version }}
RELEASE_REVISION=${{ needs.meta.outputs.release_revision }}
PNPM_VERSION=${{ needs.meta.outputs.pnpm_version }}
VCS_REF=${{ github.sha }}
tags: |
${{ needs.meta.outputs.ghcr_image }}:${{ needs.meta.outputs.identity }}
${{ needs.meta.outputs.ghcr_image }}:latest
build-gateway:
name: Publish gateway image
needs: [meta, tests]
uses: ./.github/workflows/_build-image.yaml
permissions:
contents: read
packages: write
id-token: write
attestations: write
with:
image: ${{ needs.meta.outputs.gateway_image }}
version: ${{ needs.meta.outputs.identity }}
context: examples/dsh-agents-as-a-service/gateway
dockerfile: Dockerfile
tags: |
${{ needs.meta.outputs.gateway_image }}:${{ needs.meta.outputs.identity }}
${{ needs.meta.outputs.gateway_image }}:latest
legacy-gateway-alias:
name: Publish legacy :gateway alias (temporary)
needs: [meta, build-gateway]
if: github.event_name != 'workflow_dispatch' || inputs.publish_legacy_gateway_alias
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- name: Login to GitHub Container Registry
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Retag gateway as the legacy package tag
env:
SOURCE: ${{ needs.meta.outputs.gateway_image }}:${{ needs.meta.outputs.identity }}
LEGACY: ${{ needs.meta.outputs.ghcr_image }}:gateway
run: |
set -euo pipefail
docker buildx imagetools create -t "$LEGACY" "$SOURCE"
echo "Deprecated alias $LEGACY now points at $SOURCE" >> "$GITHUB_STEP_SUMMARY"
publish-release:
name: Changelog and GitHub Release
needs: [meta, build-main, build-gateway]
runs-on: ubuntu-latest
timeout-minutes: 20
permissions:
contents: write
pull-requests: write
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
fetch-depth: 0
- name: Install git-cliff
env:
GIT_CLIFF_VERSION: v2.14.2
run: |
set -euo pipefail
version="${GIT_CLIFF_VERSION#v}"
curl -fsSL -o /tmp/git-cliff.tar.gz "https://github.com/orhun/git-cliff/releases/download/$GIT_CLIFF_VERSION/git-cliff-$version-x86_64-unknown-linux-gnu.tar.gz"
tar -xzf /tmp/git-cliff.tar.gz -C /tmp
install -m 0755 "/tmp/git-cliff-$version/git-cliff" /usr/local/bin/git-cliff
git-cliff --version
- name: Generate CHANGELOG and release notes
env:
IDENTITY: ${{ needs.meta.outputs.identity }}
run: |
set -euo pipefail
git-cliff --tag "$IDENTITY" --output CHANGELOG.md
git-cliff --tag "$IDENTITY" --latest --strip header --output /tmp/release-notes.md
echo "### Release $IDENTITY" >> "$GITHUB_STEP_SUMMARY"
head -n 40 /tmp/release-notes.md >> "$GITHUB_STEP_SUMMARY"
- name: Create GitHub Release
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
IDENTITY: ${{ needs.meta.outputs.identity }}
run: |
set -euo pipefail
if gh release view "$IDENTITY" >/dev/null 2>&1; then
echo "release $IDENTITY already exists; skipping" >&2
else
gh release create "$IDENTITY" --title "$IDENTITY" --notes-file /tmp/release-notes.md --verify-tag
fi
- name: Open CHANGELOG pull request
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
IDENTITY: ${{ needs.meta.outputs.identity }}
run: |
set -euo pipefail
git config user.name "github-actions[bot]"
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
if git diff --quiet -- CHANGELOG.md; then
echo "CHANGELOG.md already up to date"
exit 0
fi
branch="chore/changelog-$IDENTITY"
git switch -c "$branch"
git add CHANGELOG.md
git commit -m "chore(changelog): refresh for $IDENTITY"
git push origin "$branch"
gh pr create \
--base main \
--head "$branch" \
--title "chore(changelog): refresh for $IDENTITY" \
--body "Generated by git-cliff for $IDENTITY. Do not edit CHANGELOG.md by hand."

View File

@@ -1,137 +0,0 @@
name: Validate
# Validates the Kubernetes manifests against the real agent-sandbox CRD
# schema (server-side dry-run on a throwaway kind cluster), runs the gateway
# unit tests + lint, and builds/smoke-tests the gateway image without a
# cluster. Runs on every PR and on pushes to main.
on:
pull_request:
push:
branches: [main]
permissions:
contents: read
jobs:
validate-manifests:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Create kind cluster
uses: helm/kind-action@1544676c07bb3570fcb70166beaa07c4528fd4aa # v1.15.1
with:
version: v0.33.0
cluster_name: dsh-validate
- name: Install agent-sandbox (extensions)
run: |
VERSION=v1.0.7
kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/download/${VERSION}/sandbox-with-extensions.yaml
kubectl wait --for=condition=Established crd/sandboxtemplates.extensions.agents.x-k8s.io --timeout=180s
kubectl wait --for=condition=Established crd/sandboxwarmpools.extensions.agents.x-k8s.io --timeout=180s
kubectl wait --for=condition=Established crd/sandboxclaims.extensions.agents.x-k8s.io --timeout=180s
kubectl wait --for=condition=Established crd/sandboxes.agents.x-k8s.io --timeout=180s
- name: Validate manifests (server dry-run)
run: |
set -euo pipefail
kubectl create namespace dsh-demo
for f in examples/dsh-agents-as-a-service/*.yaml; do
echo "== $f =="
kubectl apply --dry-run=server -f "$f"
done
kubectl apply --dry-run=server -f deploy/k8s/sandbox-template.yaml
echo "all manifests validated"
gateway-tests:
# The gateway lives in examples/dsh-agents-as-a-service/gateway/. Its
# runtime pins and dev pins are kept separate so the image only ships
# requirements.txt while CI installs both files.
name: Gateway unit tests and lint
runs-on: ubuntu-latest
defaults:
run:
working-directory: examples/dsh-agents-as-a-service/gateway
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
cache: pip
cache-dependency-path: |
examples/dsh-agents-as-a-service/gateway/requirements.txt
examples/dsh-agents-as-a-service/gateway/requirements-dev.txt
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt -r requirements-dev.txt
- name: Lint (ruff)
run: python -m ruff check .
- name: Run gateway unit tests
run: python -m pytest -v
gateway-image:
# No cluster and no real credentials: the gateway must start and answer
# /healthz with a dummy kubeconfig, which catches broken imports, missing
# runtime deps, and an unresponsive ASGI entrypoint.
name: Build and smoke-test gateway image
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Build gateway image
run: |
docker build \
-t dsh-aaas-gateway:ci \
examples/dsh-agents-as-a-service/gateway/
- name: Smoke-test /healthz without a cluster
run: |
set -euo pipefail
cat > /tmp/kubeconfig <<'EOF'
apiVersion: v1
kind: Config
clusters:
- name: dummy
cluster:
server: https://127.0.0.1:6443
insecure-skip-tls-verify: true
contexts:
- name: dummy
context:
cluster: dummy
user: dummy
users:
- name: dummy
user: {}
current-context: dummy
EOF
docker run -d --name gateway-smoke \
-p 18080:8080 \
-e NAMESPACE=dsh-demo \
-e POOL=dsh-pool \
-e KUBECONFIG=/tmp/kubeconfig \
-v /tmp/kubeconfig:/tmp/kubeconfig:ro \
dsh-aaas-gateway:ci
trap 'docker logs gateway-smoke || true; docker rm -f gateway-smoke || true' EXIT
code=""
for _ in $(seq 1 30); do
code=$(curl -s -o /dev/null -w '%{http_code}' http://localhost:18080/healthz || true)
[ "$code" = "200" ] && break
sleep 1
done
if [ "$code" != "200" ]; then
echo "gateway /healthz returned '$code' instead of 200" >&2
exit 1
fi
echo "gateway image healthy: GET /healthz -> 200"

5
.gitignore vendored
View File

@@ -159,3 +159,8 @@ __pycache__/
htmlcov/
.venv/
venv/
# Local repository tooling
.uvcache/
coverage.xml

11
.hadolint.yaml Normal file
View File

@@ -0,0 +1,11 @@
# Container lint policy. Base images are supplied as build args defaulting to
# pinned tags, and the image is rebuilt on every DSH release, so unpinned
# packages inside a pinned base image are intentional.
failure-threshold: warning
ignored:
- DL3008 # apt-get packages are intentionally not version-pinned
- DL3018 # apk packages are intentionally not version-pinned
- DL3006 # FROM uses build args (NODE_BUILD_IMAGE/NODE_RUNTIME_IMAGE) from versions.env
- DL3003 # the DSH packaging step deliberately cds inside a single layer
- DL3059 # consecutive RUN layers are kept for cache reuse and readability
- DL4006 # the upstream uv installer pipes curl to sh by design

13
.lychee.toml Normal file
View File

@@ -0,0 +1,13 @@
# Link checker for README/docs. Placeholders used by the examples are excluded
# on purpose; real URLs must resolve.
verbose = "info"
no_progress = true
max_retries = 2
exclude = [
"localhost",
"127\\.0\\.0\\.1",
"example\\.com",
"your-github-org",
"mailto:",
"ghcr\\.io/.*@sha256",
]

19
.yamllint.yml Normal file
View File

@@ -0,0 +1,19 @@
extends: default
rules:
document-start: disable
line-length:
max: 200
level: error
truthy:
check-keys: false
comments-indentation: disable
comments:
min-spaces-from-content: 1
ignore: |
.venv/
.uvcache/
node_modules/
# Upstream CodeQL boilerplate: long documentation comments, linted by actionlint.
.github/workflows/codeql.yml

148
AGENTS.md Normal file
View File

@@ -0,0 +1,148 @@
# AGENTS.md
Guidance for AI coding agents (and humans) working in this repository. Read it
before changing anything; it encodes the conventions the CI enforces.
## What this repository is
- **The root is the primary artifact.** The Dockerfile packages DeepSeek
Harness Web (dsh web) as a slim, non-root, multi-arch image. Everything else
exists to support it or to demonstrate an implementation on top of it.
- **Every implementation is a self-contained example** under examples/<name>/.
There is exactly one example for the agents-as-a-service platform; do not
split it into parallel directories.
- **versions.env is the single source of truth** for the DSH ref, pnpm, the
control-plane versions and the image names.
- **Docs describe the shipped behavior**, not intentions. If you change a
manifest, port, env var or workflow, update the docs in the same change.
- **CHANGELOG.md is generated** by git-cliff from Conventional Commits. Never
edit it by hand.
## Golden rules
1. Keep the Dockerfile working for both linux/amd64 and linux/arm64; it must
stay the first-class artifact.
2. Never hard-code a version, image tag or registry path. Edit versions.env and
run make set-version when bumping.
3. Never commit a real secret or usable credential. Secret-shaped YAML lives in
*.yaml.example files; keys are documented in .env.sample only.
4. Every manifest under examples/*/manifests/ must parse and pass a server-side
dry-run in CI. Keep manifests numbered (NN-*.yaml), and keep exposure
variants under manifests/exposure/<variant>/ documented by the single
manifests/exposure/README.md (variants are mutually exclusive).
5. Do not edit CHANGELOG.md, and do not move or delete published tags.
Releasing is documented in docs/versioning.md.
6. Keep the gateway HTTP/WebSocket contract stable unless the change is
explicitly about it; clients and examples depend on it.
## Repository map
| Path | What lives there |
| --- | --- |
| Dockerfile | Main DSH Web image (build + runtime stages, entrypoint) |
| versions.env | Every pinned version and image name |
| Makefile | Canonical local entrypoint |
| build.sh | Deprecated shim that forwards to make |
| docs/ | main-image.md, examples.md, gitflow.md, versioning.md |
| examples/ | One implementation per directory (contract in docs/examples.md) |
| examples/dsh-agents-as-a-service/ | The multi-tenant platform example (manifests, gateway, docs, scripts) |
| examples/minimal-single-tenant/ | The just-one-DSH-Web quickstart |
| scripts/ | check-versions.sh, check-examples.sh, check-manifests.py, set-version.sh |
| .dsh/skills/ | Project-local DSH skills (auto-discovered); dsh-agent-sandbox-platform covers platform operation, MCP limits and extension recipes |
| .github/workflows/ | ci.yaml, reusable _build-image.yaml, build-*, release.yaml, e2e-kind.yaml, pr-title.yaml, codeql.yml |
## Commands
make check # versions, example layout and manifest YAML (run before every commit)
make test # gateway unit tests (109+ tests)
make lint # ruff + shellcheck (+ yamllint/actionlint when installed)
make image # build the main DSH Web image (slow: clones and builds DSH)
make gateway # build the example gateway image (fast)
make e2e # full kind walkthrough; needs a cluster with agent-sandbox
make changelog # regenerate CHANGELOG.md with git-cliff (release only)
make release-dry-run
make set-version DSH_VERSION=dsh-vX.Y.Z RELEASE_REVISION=rN
No cluster is needed for check, test, lint, gateway or image. Only make e2e
needs kind plus the agent-sandbox CRDs (see the example README).
## Testing expectations per change
| Change | Minimum verification |
| --- | --- |
| Documentation only | make check; verify relative links |
| Example manifest | make check; kubectl apply --dry-run=server in CI |
| Optional hardening (gVisor, nodeSelector, NetworkPolicy, provider keys) | make check; server dry-run in CI; manual verification on a capable cluster |
| Gateway code | make test; make lint; add or update tests |
| Dockerfile / versions.env | make check; make image (amd64) and a container smoke test |
| Workflow | actionlint in CI; keep actions SHA-pinned; no hard-coded versions |
| New example | make check; all files described in docs/examples.md |
| Project skill (.dsh/skills/) | make check; frontmatter and size check; shellcheck skill scripts; load it with the skill tool |
## Versioning and releases
- Release identity = DSH_VERSION, optionally with a repository revision suffix
for re-releases at the same DSH ref (current identity:
dsh-v0.2.1-rc.5).
- The current identity is dsh-v0.2.1-rc.5 (stored in versions.env). It is the
first release at this DSH ref, so it carries no revision suffix; a
repository-only change at the same ref publishes -r2.
- Channels: release tags are immutable; latest follows the newest release;
main is the development channel for both image packages.
- A broken release is fixed by publishing -r(N+1), never by re-tagging.
- The full procedure, compatibility table and rollback rules are in
docs/versioning.md.
## Branch and PR conventions
- Trunk-based with release branches, documented in docs/gitflow.md.
- Branch names: feat/, fix/, docs/, chore/, refactor/, test/.
- Conventional Commit subjects and PR titles (feat(scope): ..., fix(scope): ...).
- PRs are squash-merged; main stays linear and protected.
- Required checks: lint, python, manifests, gateway-image, pr-title, CodeQL,
dependency-review.
- Keep diffs scoped; update docs with the code; never mix a version bump with
unrelated changes.
## Adding an example
1. Read docs/examples.md and examples/README.md.
2. Create examples/<name>/ with README.md, manifests/NN-*.yaml, optional
manifests/exposure/<variant>/ (plus README), optional components with their
own Dockerfile/README/tests, and scripts/ (executable, idempotent,
self-cleaning).
3. Publish component images under <registry>/<owner>/<repo>/<example>/<component>.
4. Add the example to examples/README.md; scripts/check-examples.sh enforces
the layout.
## Security notes
- AUTH_PROXY_HEADER is opt-in. Only enable it when the gateway Service is
reachable exclusively through the authenticating proxy; otherwise any client
can spoof X-Forwarded-User.
- The example gateway reads the DSH launch token from pod logs. Treat pod logs
as sensitive and keep its RBAC namespace-scoped.
- The example shares provider keys (dsh-provider-keys plus the optional
dsh-provider-keys-extra) across sandboxes; do not describe it as
tenant-isolated.
- gVisor, nodeSelector and the NetworkPolicies are opt-in and are not active in
the base manifests; do not describe the base example as hardened.
- TLS terminates outside the gateway; set COOKIE_SECURE=true for HTTPS and keep
the base manifest safe for local HTTP.
## Ask before
- Changing release workflows, tag formats or published image names.
- Editing the gateway's public routes or auth semantics.
- Adding a new top-level tool, service or dependency to the main image.
- Rewriting generated files (CHANGELOG.md) or git history.
## Where to look first
- [docs/main-image.md](docs/main-image.md) - container contract and tags
- [docs/examples.md](docs/examples.md) - example authoring rules
- [docs/gitflow.md](docs/gitflow.md) - branches, PRs, required checks
- [docs/versioning.md](docs/versioning.md) - versioning, changelog, releases
- [.dsh/skills/dsh-agent-sandbox-platform/SKILL.md](.dsh/skills/dsh-agent-sandbox-platform/SKILL.md) - platform operations, MCP limits, extension recipes
- [CONTRIBUTING.md](CONTRIBUTING.md) - development setup
- [SECURITY.md](SECURITY.md) - vulnerability reporting and threat notes

67
CHANGELOG.md Normal file
View File

@@ -0,0 +1,67 @@
# Changelog
All notable changes to this repository are documented here.
Generated by git-cliff from Conventional Commits; do not edit this file by hand.
## [dsh-v0.2.1-rc.3-r4] - 2026-10-26
### Added
- **skills:** Add agent-sandbox platform skill (#20) (bf24306)
- **example:** Add optional gVisor, nodeSelector, NetworkPolicy and provider keys (#21) (3e9ba38)
### Maintenance
- **changelog:** Refresh for dsh-v0.2.1-rc.3-r3 (#19) (8681412)
- **release:** Dsh-v0.2.1-rc.3-r4 (#22) (e2f1fe5)
## [dsh-v0.2.1-rc.3-r3] - 2026-10-25
### Added
- **gateway-api:** Expose dsh-gateway externally via Traefik Gateway API (#12) (7d3faf0)
- **gateway-api:** GitHub SSO via oauth2-proxy in front of dsh-gateway (#13) (3f46279)
- **gateway:** SSO landing auto-connects and offers recreate sandbox (#15) (ed1398d)
- **repo:** Repo reorganization (#16) (3b70f17)
### Fixed
- **Dockerfile:** Update runtime dependencies and enable corepack for pnpm (6d97f61)
- **gateway-api:** Correct oauth2-proxy cookie-secret generation (#14) (73a65d9)
### Maintenance
- Bump the actions group across 1 directory with 3 updates (#17) (6a8fe54)
### Other
- Dockerfile extra tools (2b47afa)
- Modify gateway.yaml for image policy and timeout values (53e55fc)
- Add commit message prefix for Dependabot updates (6876871)
- Versioning update (#18) (ecba8b9)
- Refactor environment variable loading in build workflow (d627526)
- Refactor environment variable loading in build-gateway.yaml (027f74f)
- Refactor release workflow for environment variable sourcing (958cbe2)
## [v0.0.1] - 2026-10-21
### Other
- Initial commit (f3d4312)
- Update Dockerfile, README, and Kubernetes manifests for DSH Web deployment (#1) (0d66cba)
- Add DeepSeek Harness Web agents-as-a-service platform pattern (#2) (98cb547)
- Update action versions and cluster version (d9cb265)
- Add Python to CodeQL workflow languages (b0f992f)
- Add multi-arch Docker image build workflow with GitHub Actions (de11d34)
- Add step to create manifest list for Docker images (c9be929)
- Make DSH agents-as-a-service browser-usable; fix probe crash-loop (#3) (6301c21)
- Fix idle sweeper suspending claims the gateway does not own (#4) (29eefd6)
- Fix DSH crash-loop after suspend/resume (fsGroup chmod) (#5) (a8ef211)
- Add a startup probe so slow DSH cold starts are not killed (#6) (c193ed2)
- Normalise the pod model returned by CoreV1Api (#7) (9dea4a7)
- Forward the upstream query byte-for-byte (#8) (6876406)
- Do not read-time-out proxied upstream streams (#9) (163abae)
- Dial the sandbox pod IP for WebSocket upgrades (#10) (abe5b4d)
- Handshake under the pod-IP authority for WebSocket upgrades (#11) (0a61e57)

46
CODE_OF_CONDUCT.md Normal file
View File

@@ -0,0 +1,46 @@
# Code of Conduct
This project adopts the Contributor Covenant, version 2.1:
https://www.contributor-covenant.org/version/2/1/code_of_conduct/
## Our standards
We are committed to a welcoming, harassment-free experience for everyone,
regardless of age, body size, visible or invisible disability, ethnicity, sex
characteristics, gender identity and expression, level of experience,
education, socio-economic status, nationality, personal appearance, race,
religion, or sexual identity and orientation.
Examples of behavior that contributes to a positive environment:
- Demonstrating empathy and kindness toward other people
- Being respectful of differing opinions, viewpoints, and experiences
- Giving and gracefully accepting constructive feedback
- Accepting responsibility and apologizing to those affected by our mistakes
- Focusing on what is best for the community and the project
Examples of unacceptable behavior:
- Sexualized language or imagery, and sexual attention or advances
- Trolling, insulting or derogatory comments, and personal or political attacks
- Public or private harassment
- Publishing others' private information without explicit permission
- Other conduct which could reasonably be considered inappropriate in a
professional setting
## Enforcement
Maintainers are responsible for clarifying and enforcing these standards and
may remove, edit, or reject comments, commits, code, issues, and other
contributions that do not align with this Code of Conduct.
Report abusive, harassing, or otherwise unacceptable behavior to the
maintainers through a private GitHub Security Advisory or by contacting the
repository owner on GitHub. All complaints will be reviewed and investigated
promptly and fairly. Maintainers must respect the privacy and security of the
reporter.
## Attribution
This Code of Conduct is adapted from the Contributor Covenant, version 2.1,
available at https://www.contributor-covenant.org/version/2/1/code_of_conduct/

88
CONTRIBUTING.md Normal file
View File

@@ -0,0 +1,88 @@
# Contributing
Thanks for improving deepseek-kubernetes-sandbox. This repository packages
DeepSeek Harness Web as a container and ships self-contained implementation
examples around it. The root is the container; every implementation lives
under examples/.
## Repository layout
| Path | Purpose |
| --- | --- |
| Dockerfile | The primary artifact: the DSH Web image |
| versions.env | Single source of truth for every pinned version |
| Makefile | Canonical local entrypoint (build, test, lint, check, e2e) |
| docs/ | Repository-wide docs: main image, example contract, gitflow, versioning |
| examples/ | One self-contained implementation per directory |
| scripts/ | Repository checks (versions, example layout, manifests, version bump) |
| .github/ | CI, releases, dependency updates, templates |
## Requirements
- Docker (for image builds)
- Python 3.12+ (gateway tests and repository checks)
- make and git
- kubectl and kind only for the end-to-end example
## Local workflow
git switch -c feat/<short-slug>
make check # versions, example layout, manifest YAML
make test # gateway unit tests
make lint # ruff, shellcheck (+ yamllint/actionlint when installed)
make image # build the main DSH Web image
make gateway # build the example gateway image
make e2e # full kind walkthrough (optional, slow)
Install the gateway dev dependencies once:
python -m pip install -r examples/dsh-agents-as-a-service/gateway/requirements-dev.txt
## Pull requests
- Use a Conventional Commit title, for example feat(gateway): add wake metrics
or docs(examples): clarify SSO trust boundary. PR titles are linted.
- Keep the diff scoped. Squash merge into main; history stays linear.
- Required checks: lint, python, manifests, gateway-image, pr-title, CodeQL,
and dependency-review. Branch protection is described in docs/gitflow.md.
- Update the docs that the change affects; docs are part of the change, not a
follow-up.
- Never commit a real secret. Secret-shaped YAML belongs in a *.yaml.example
file, and versions live in versions.env only.
## Changing versions
make set-version DSH_VERSION=dsh-vX.Y.Z RELEASE_REVISION=r2
make check
CHANGELOG.md is generated by git-cliff from Conventional Commits. Do not edit
it by hand; the release workflow regenerates it.
## Adding an example
Read docs/examples.md, then add examples/<name>/ with:
- README.md (what it demonstrates, prerequisites, steps, teardown)
- manifests/NN-*.yaml (numbered, plain kubectl)
- optional manifests/exposure/<variant>/ with its own README.md
- optional components (a Dockerfile plus README and tests per component)
- scripts/ with executable, idempotent, self-cleaning scripts
- docs/ for deeper material
List it in examples/README.md. scripts/check-examples.sh enforces this layout.
## Documentation
- Root README.md is the front door: main image first, then examples.
- docs/ holds repository-wide material; example-specific material belongs in
the example directory.
## Branches and releases
See docs/gitflow.md for the branch model and docs/versioning.md for the
release identity, channels and rollback rules.
## Security
See SECURITY.md. Report vulnerabilities privately through GitHub Security
Advisories; do not open a public issue.

View File

@@ -1,26 +1,37 @@
# syntax=docker/dockerfile:1.7
#
# The primary artifact of this repository: DeepSeek Harness Web (dsh web),
# packaged as a slim, non-root, multi-arch runtime image.
#
# Build args (all pinned in versions.env, verified by scripts/check-versions.sh):
# DSH_REPO upstream git remote to package
# DSH_VERSION upstream git ref to build (tag or branch)
# RELEASE_REVISION optional repository revision label (r2, r3, ...)
# PNPM_VERSION pnpm used to build DSH
# VCS_REF this repository's commit, recorded in OCI labels
# NODE_BUILD_IMAGE node image used to build DSH
# NODE_RUNTIME_IMAGE node image used at runtime
ARG NODE_BUILD_IMAGE=docker.io/node:24-trixie
ARG NODE_RUNTIME_IMAGE=docker.io/node:24-trixie-slim
# ---------------------------------------------------------------------------
# Build stage
# Build stage: clone DSH at the pinned ref, build, and package the CLI.
# ---------------------------------------------------------------------------
FROM ${NODE_BUILD_IMAGE} AS build
FROM $NODE_BUILD_IMAGE AS build
ARG DSH_REPO=https://github.com/deepseek-ai/deepseek-harness.git
ARG DSH_VERSION=dsh-v0.2.1-rc.2
ARG DSH_VERSION=dsh-v0.2.1-rc.5
ARG PNPM_VERSION=11.28.5
ENV CI=true \
NODE_OPTIONS=--max-old-space-size=4096
RUN corepack enable \
&& corepack prepare "pnpm@${PNPM_VERSION}" --activate
&& corepack prepare "pnpm@$PNPM_VERSION" --activate
WORKDIR /src
RUN git clone --depth 1 --branch "${DSH_VERSION}" "${DSH_REPO}" .
RUN git clone --depth 1 --branch "$DSH_VERSION" "$DSH_REPO" .
RUN pnpm install --frozen-lockfile
@@ -44,29 +55,47 @@ RUN cd /opt/dsh \
# ---------------------------------------------------------------------------
# Runtime stage
# ---------------------------------------------------------------------------
FROM ${NODE_RUNTIME_IMAGE} AS runtime
FROM $NODE_RUNTIME_IMAGE AS runtime
ARG DSH_VERSION
ARG DSH_VERSION=dsh-v0.2.1-rc.5
ARG RELEASE_REVISION=
ARG PNPM_VERSION=11.28.5
ARG VCS_REF=unknown
LABEL org.opencontainers.image.title="DeepSeek Harness Web packaged by SREDevOps.org" \
org.opencontainers.image.version="${DSH_VERSION}" \
org.opencontainers.image.source="https://github.com/sredevopsorg/deepseek-kubernetes-sandbox"
org.opencontainers.image.description="DeepSeek Harness Web (dsh web) as a slim, non-root, Kubernetes-ready image" \
org.opencontainers.image.source="https://github.com/sredevopsorg/deepseek-kubernetes-sandbox" \
org.opencontainers.image.version="$DSH_VERSION" \
org.opencontainers.image.revision="$VCS_REF" \
org.opencontainers.image.licenses="MIT" \
io.sredevopsorg.dsh.version="$DSH_VERSION" \
io.sredevopsorg.release.revision="$RELEASE_REVISION"
ENV NODE_ENV=production \
HOME=/home/node \
DSH_HOME=/home/node/.dsh \
DSH_PORT=3080 \
PATH="/app/node_modules/.bin:${PATH}"
PATH="/app/node_modules/.bin:$PATH"
# git/ripgrep are useful for DSH agent tools; tini handles signals.
# git/ripgrep/jq/yq/curl/gh are deliberate agent tooling parity: a sandboxed DSH
# instance expects the usual developer CLI. tini reaps signals for a clean stop.
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
ca-certificates \
git \
ripgrep \
jq \
yq \
curl \
wget \
tini \
gh \
&& rm -rf /var/lib/apt/lists/* \
&& install -d -o node -g node -m 0755 /workspace "${DSH_HOME}"
&& curl -LsSf https://astral.sh/uv/install.sh | sh \
&& install -d -o node -g node -m 0755 /workspace "$DSH_HOME" \
&& corepack enable \
&& corepack prepare "pnpm@$PNPM_VERSION" --activate \
&& pnpm --version
COPY --from=build --chown=node:node /opt/dsh /app
@@ -88,7 +117,7 @@ if [ -n "$DSH_PUBLIC_URL" ]; then
set -- --public-url "$DSH_PUBLIC_URL" "$@"
fi
# Additional trusted hosts (e.g. the public gateway host behind an Ingress).
# Additional trusted hosts (e.g. the public gateway host behind a proxy).
# DSH Host/Origin-fences /api: without this, requests forwarded with the
# browser's Host header are rejected. Accepts comma- or space-separated names;
# localhost/127.0.0.1 are always trusted below.
@@ -117,4 +146,4 @@ USER node
EXPOSE 3080
VOLUME ["/home/node/.dsh"]
STOPSIGNAL SIGTERM
ENTRYPOINT ["/usr/bin/tini", "--", "/usr/local/bin/dsh-entrypoint"]
ENTRYPOINT ["/usr/bin/tini", "--", "/usr/local/bin/dsh-entrypoint"]

21
LICENSE Normal file
View File

@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2026 Nicolás Georger
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

View File

@@ -1,37 +1,82 @@
# Local build and test helpers. Every variable can be overridden, e.g.
# make image DSH_VERSION=dsh-v0.2.1-rc.2 IMAGE=dsh-web:dev
# make test
DSH_VERSION ?= dsh-v0.2.1-rc.2
PNPM_VERSION ?= 11.28.5
IMAGE ?= dsh-web:0.2.1-rc.2
GATEWAY_IMAGE ?= ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway
GATEWAY_DIR := examples/dsh-agents-as-a-service/gateway
PYTHON ?= python3
DSH_PORT ?= 3080
# Canonical local entrypoint. Includes versions.env so there is exactly one
# place to bump a version (run "make check" to verify pins across the repo).
#
# make image build the main DSH Web image
# make gateway build the example gateway image
# make test gateway unit tests
# make lint ruff + shellcheck (+ yamllint/actionlint when installed)
# make check versions, example layout and manifest YAML checks
# make e2e full example walkthrough on a kind cluster
# make changelog regenerate CHANGELOG.md with git-cliff
include versions.env
.PHONY: help image gateway run test lint
RELEASE_IDENTITY := $(DSH_VERSION)$(if $(RELEASE_REVISION),-$(RELEASE_REVISION))
IMAGE ?= $(GHCR_IMAGE):$(RELEASE_IDENTITY)
GATEWAY_LOCAL_IMAGE ?= dsh-aaas-gateway:dev
GATEWAY_DIR := examples/dsh-agents-as-a-service/gateway
EXAMPLE_DIR := examples/dsh-agents-as-a-service
PYTHON ?= $(if $(wildcard $(CURDIR)/.venv/bin/python),$(CURDIR)/.venv/bin/python,python3)
DSH_PORT ?= 3080
VCS_REF ?= $(shell git rev-parse --short HEAD 2>/dev/null || echo unknown)
.PHONY: help image run gateway test cov lint check e2e changelog release-dry-run set-version clean
help: ## Show available targets
@grep -E '^[a-z-]+:.*?## ' $(MAKEFILE_LIST) | sed 's/:.*## /\t/'
@grep -E '^[a-z-]+:.*?## ' $(MAKEFILE_LIST) | sed 's/:.*## / /'
image: ## Build the DSH Web image
image: ## Build the main DSH Web image (no push)
docker build \
--build-arg DSH_VERSION=$(DSH_VERSION) \
--build-arg RELEASE_REVISION=$(RELEASE_REVISION) \
--build-arg PNPM_VERSION=$(PNPM_VERSION) \
--build-arg VCS_REF=$(VCS_REF) \
-t $(IMAGE) .
gateway: ## Build the gateway image
docker build -t $(GATEWAY_IMAGE) $(GATEWAY_DIR)/
run: image ## Build and run the DSH Web image
run: image ## Build and run the main image (needs DEEPSEEK_API_KEY)
docker run --rm -it \
-p $(DSH_PORT):3080 \
-v dsh-home:/home/node/.dsh \
-e DEEPSEEK_API_KEY \
$(IMAGE)
test: ## Run the gateway unit tests (needs requirements + requirements-dev installed)
gateway: ## Build the example gateway image locally
docker build -t $(GATEWAY_LOCAL_IMAGE) $(GATEWAY_DIR)/
test: ## Gateway unit tests
cd $(GATEWAY_DIR) && $(PYTHON) -m pytest -q
lint: ## Lint the gateway with ruff
cd $(GATEWAY_DIR) && ruff check .
cov: ## Gateway unit tests with coverage (needs pytest-cov)
cd $(GATEWAY_DIR) && $(PYTHON) -m pytest -q --cov=. --cov-report=term-missing --cov-report=xml:coverage.xml
lint: ## Lint Python, shell, YAML and Dockerfiles (skips missing tools locally)
cd $(GATEWAY_DIR) && $(PYTHON) -m ruff check .
cd $(GATEWAY_DIR) && $(PYTHON) -m ruff format --check .
shellcheck build.sh scripts/*.sh $(EXAMPLE_DIR)/scripts/*.sh
@command -v yamllint >/dev/null 2>&1 && yamllint . || echo "yamllint not installed; CI enforces it"
@command -v actionlint >/dev/null 2>&1 && actionlint || echo "actionlint not installed; CI enforces it"
check: ## Verify version pins, example layout and manifest YAML
./scripts/check-versions.sh
./scripts/check-examples.sh
$(PYTHON) scripts/check-manifests.py
e2e: ## Run the full example walkthrough on the current kind cluster
$(EXAMPLE_DIR)/scripts/e2e-kind.sh
changelog: ## Regenerate CHANGELOG.md from Conventional Commits
@command -v git-cliff >/dev/null 2>&1 || { echo "install git-cliff: https://git-cliff.org" >&2; exit 1; }
git-cliff --output CHANGELOG.md
release-dry-run: check ## Validate the release identity and preview release notes
@echo "release identity: $(RELEASE_IDENTITY)"
@echo "main image: $(GHCR_IMAGE):$(RELEASE_IDENTITY)"
@echo "gateway image: $(GATEWAY_IMAGE):$(RELEASE_IDENTITY)"
@if git rev-parse -q --verify "refs/tags/$(RELEASE_IDENTITY)" >/dev/null; then echo "ERROR: tag already exists" >&2; exit 1; fi
@command -v git-cliff >/dev/null 2>&1 && git-cliff --unreleased --tag $(RELEASE_IDENTITY) || echo "git-cliff not installed; skipping notes preview"
set-version: ## Bump pins: make set-version DSH_VERSION=dsh-vX [RELEASE_REVISION=rN]
@test -n "$(DSH_VERSION)" || { echo "DSH_VERSION is required" >&2; exit 1; }
./scripts/set-version.sh "$(DSH_VERSION)" "$(RELEASE_REVISION)"
clean: ## Remove local build/test caches
rm -rf .pytest_cache .ruff_cache $(GATEWAY_DIR)/.pytest_cache $(GATEWAY_DIR)/.ruff_cache $(GATEWAY_DIR)/__pycache__

344
README.md
View File

@@ -1,180 +1,234 @@
# deepseek-kubernetes-sandbox
Packages [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) Web (`dsh web`) into
a multi-arch, Kubernetes-ready container image, and ships the
[agents-as-a-service](examples/dsh-agents-as-a-service/) platform pattern — running a personal
DSH Web instance per user on [kubernetes-sigs/agent-sandbox](https://github.com/kubernetes-sigs/agent-sandbox)
behind a small browser-usable gateway.
**DeepSeek Harness Web, packaged as a container.** This repository's primary
artifact is a slim, non-root, multi-arch image of
[DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) Web
(`dsh web`), ready for Docker and Kubernetes. Everything else exists to support
that image or to demonstrate an implementation on top of it.
## What's in this repo
The flagship implementation is
[`examples/dsh-agents-as-a-service/`](examples/dsh-agents-as-a-service/): the
"agents-as-a-service" platform pattern with one DSH Web instance per user,
provisioned from a warm pool on
[kubernetes-sigs/agent-sandbox](https://github.com/kubernetes-sigs/agent-sandbox),
suspended when idle, and fronted by a small browser-usable gateway.
| Path | Purpose |
| --- | --- |
| [`Dockerfile`](Dockerfile) | Multi-stage build that clones DSH, `pnpm pack`s the CLI, and produces a slim runtime image (`:3080`) |
| [`build.sh`](build.sh), [`Makefile`](Makefile) | Local image builds: DSH Web and gateway, plus test/lint targets |
| [`examples/dsh-agents-as-a-service/`](examples/dsh-agents-as-a-service/) | The full multi-tenant platform pattern (warm pool, per-user claims, gateway, smoke test) |
| [`examples/dsh-agents-as-a-service/gateway/`](examples/dsh-agents-as-a-service/gateway/) | Starlette ASGI gateway: signup, cookie sessions, `/u/{user}/**` HTTP + WebSocket proxy, idle sweeper |
| [`deploy/k8s/sandbox-template.yaml`](deploy/k8s/sandbox-template.yaml) | Minimal single-tenant `SandboxTemplate` |
| [`docs/architecture.md`](docs/architecture.md) | Container and sequence diagrams, design decisions, security notes |
| [`.github/workflows/`](.github/workflows/) | CI: manifests dry-run, gateway tests/lint, multi-arch image builds |
## Published images
Everything is published to GitHub Container Registry as
`ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox`:
| Tag | Contents | Port |
| --- | --- | --- |
| `dsh-v0.2.1-rc.2` (also `main`, `latest`) | DSH Web image | `3080` |
| `gateway` | Starlette gateway image | `8080` |
The DSH Web tag matches the upstream DSH git tag it was built from.
## Quick start (DSH Web image)
## Quick start
```bash
docker run -d --name dsh-web \
-p 3080:3080 \
-v dsh-home:/home/node/.dsh \
-e DEEPSEEK_API_KEY \
ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.2
ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.5
```
**DSH Web is not usable at a bare `/`.** Without its launch-token cookie it answers `401`, and
the token is only printed in the startup logs:
DSH Web is **not usable at a bare `/`**. Without its launch-token cookie it
answers `401`, and the token is printed only in the startup logs:
```bash
docker logs dsh-web 2>&1 | grep -m1 '?token='
# dsh web: http://localhost:3080/?token=...
```
Open the printed `?token=...` URL once — DSH sets its auth cookie and the UI loads normally
afterwards. Hitting `/` directly (or after the cookie expires) returns `401` by design; that is
why Kubernetes probes use `/favicon.svg` (see below) and why the gateway performs this
handshake for the browser.
Open the printed `?token=...` URL once: DSH sets its auth cookie and the UI loads
normally afterwards. Hitting `/` directly (or after the cookie expires) returns
`401` by design.
> **Different host port?** If `3080` is already taken on your workstation, map another port and
> set the public URL so the printed token URL matches: `-p 18081:3080 -e
> DSH_PUBLIC_URL=http://localhost:18081`, then open `http://localhost:18081/?token=...`.
### Host port and public URL
## Quick start (Kubernetes)
If `3080` is taken locally, map another port and set the public URL so the quoted
token URL matches:
```bash
kubectl create namespace dsh-demo --dry-run=client -o yaml | kubectl apply -f -
kubectl -n dsh-demo create secret generic dsh-provider-keys \
--from-literal=DEEPSEEK_API_KEY="sk-..."
kubectl apply -f deploy/k8s/sandbox-template.yaml
# A template needs a warm pool and a claim to become a running Sandbox.
kubectl apply -f - <<'EOF'
apiVersion: extensions.agents.x-k8s.io/v1beta1
kind: SandboxWarmPool
metadata:
name: dsh-pool
namespace: dsh-demo
spec:
replicas: 1
sandboxTemplateRef:
name: dsh-web-template
---
apiVersion: extensions.agents.x-k8s.io/v1beta1
kind: SandboxClaim
metadata:
name: dsh
namespace: dsh-demo
spec:
warmPoolRef:
name: dsh-pool
EOF
kubectl -n dsh-demo wait sandboxclaim/dsh --for=condition=Ready --timeout=180s
docker run -d --name dsh-web -p 18081:3080 \
-e DSH_PUBLIC_URL=http://localhost:18081 \
-v dsh-home:/home/node/.dsh \
-e DEEPSEEK_API_KEY \
ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.5
# then open http://localhost:18081/?token=...
```
Then reach the instance through its launch-token URL:
### Hosts, probes and the 401
- **Probes must not hit `/`.** It returns `401` without the launch token, so a
readiness/liveness probe on `/` crash-loops the pod. Probe `/favicon.svg`
instead: it is unauthenticated. The shipped manifests use it.
- **Host allow-listing.** DSH rejects unknown `Host` headers. `localhost` and
`127.0.0.1` are always trusted; add any other public hostname to
`DSH_TRUSTED_HOSTS` (comma or space separated). A gateway or Ingress that
forwards the browser's `Host` needs its hostname listed there.
- **State lives in `/home/node/.dsh`.** Mount a volume (Docker) or PVC
(Kubernetes) to keep conversations, sessions and config across restarts.
### Runtime environment
| Variable | Default | Purpose |
| --- | --- | --- |
| `DEEPSEEK_API_KEY` | empty | Provider credential consumed by DSH. Inject it; never bake it into the image or commit it. |
| `DSH_PORT` | `3080` | Port DSH Web listens on inside the container. |
| `DSH_HOST` | auto-detected pod/container IP | Bind address. Leave empty in Kubernetes. |
| `DSH_PUBLIC_URL` | `http://localhost:${DSH_PORT}` | Public base URL used in generated links and the launch-token URL in the logs. |
| `DSH_TRUSTED_HOSTS` | empty | Extra trusted `Host`/`Origin` entries, comma/space separated. |
| `DSH_HOME` | `/home/node/.dsh` | DSH state directory (mount a volume here). |
DSH also resolves the standard per-provider environment names —
`OPENROUTER_API_KEY`, `GEMINI_API_KEY`, `NVIDIA_API_KEY`, `OPENAI_API_KEY`,
`ANTHROPIC_API_KEY`, `GROQ_API_KEY`, `XAI_API_KEY`, `HF_TOKEN`, and others —
through its packaged provider catalog. Inject them the same way; a key saved in
DSH's credential store wins over the environment. See
[provider credentials](docs/main-image.md#provider-credentials).
Full contract, build args, probes, volumes and signals:
[docs/main-image.md](docs/main-image.md).
## Published images
Published to GitHub Container Registry. The main image is the repository image;
example components are nested packages below it:
```
ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox # DSH Web (main image)
ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway # example gateway
```
| Tag | Contents | Meaning |
| --- | --- | --- |
| `dsh-v0.2.1-rc.5` | DSH Web, port `3080` | **Current release identity.** Immutable: once published it never moves. |
| `latest` | DSH Web | Most recent release; same digest as the current identity. |
| `main` | DSH Web | Development channel rebuilt from every push to `main`. Not a release; not for production. |
Tags published before the release-identity scheme remain on GHCR but are not
updated; new deployments use the current identity.
The example gateway is published under its nested path with the same release
identity, e.g.
`ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway:dsh-v0.2.1-rc.5`.
Release identities, channels and rollback rules are explained in
[docs/versioning.md](docs/versioning.md). Production deployments should pin a
digest:
```bash
SB=$(kubectl -n dsh-demo get sandboxclaim dsh -o jsonpath='{.status.sandbox.name}')
kubectl -n dsh-demo logs "$SB" | grep -m1 '?token='
kubectl -n dsh-demo port-forward "pod/$SB" 18081:3080
# open http://localhost:18081/?token=... from the log line
docker buildx imagetools inspect ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.5
# image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox@sha256:<digest>
```
Two things bite first-time setups:
- **Probes must not hit `/`.** DSH returns `401` there without the launch token, so a
readiness/liveness probe on `/` crash-loops the pod. The shipped templates use
`/favicon.svg`, which is unauthenticated; keep that path in custom templates.
- **Host allow-listing.** DSH rejects unknown `Host` headers. `localhost` and `127.0.0.1` are
always trusted; for any other hostname add it to the template's `DSH_TRUSTED_HOSTS` (comma or
space separated). The gateway forwards the browser's `Host`, so the public gateway hostname
must be listed there.
## The multi-user platform pattern
`examples/dsh-agents-as-a-service/` is a port of the upstream
[hermes-agents-as-a-service](https://github.com/kubernetes-sigs/agent-sandbox/tree/main/examples/hermes-agents-as-a-service)
example to DSH Web. It demonstrates the platform shape — one `SandboxTemplate`, one warm pool,
one tiny `SandboxClaim` per user, and `operatingMode` as the cost dial — plus a gateway that
turns it into a usable service.
**Browser flow:** open the gateway landing page, sign up, and get an `HttpOnly` session cookie.
The browser then browses `http://<gateway>/u/<user>/`, which proxies HTTP **and WebSocket** to
that user's DSH Web. The gateway reads the DSH launch token from the pod logs, performs the
token handshake server-side, and injects the resulting DSH cookie on every proxied request — the
browser never sees the token. Suspended instances are woken transparently on the next request;
idle ones are suspended again after `IDLE_TIMEOUT`.
See the [example README](examples/dsh-agents-as-a-service/README.md) for the full walkthrough
and [docs/architecture.md](docs/architecture.md) for the diagrams and trade-offs.
### Mechanism mapping: Hermes → DSH
| Hermes example mechanism | Role | Adaptation for DSH |
| :--- | :--- | :--- |
| `SandboxTemplate` | Reusable Pod blueprint | Image `dsh-web`, port `3080`, probes on `/favicon.svg` |
| `volumeClaimTemplates` | Auto-create a PVC per user | Mount `/home/node/.dsh` (DSH state) |
| `SandboxWarmPool` | Pre-warmed instances | Keep as-is; size for expected concurrency |
| `SandboxClaim` + `warmPoolRef` | "Registration" adopts a pooled instance | Keep exactly as-is; one claim per user |
| `service: true` | Stable DNS per Sandbox | Enabled, so the gateway routes to each user's DSH Web |
| `envVarsInjectionPolicy: Disallowed` | Reject claims that would cold-start | `DEEPSEEK_API_KEY` injected via `envFrom` on the template (shared platform key) |
| `operatingMode` | Cost switch: Running / Suspended | Reused directly — suspend idle instances to save cost |
### Key differences from Hermes
- **Multi-arch** — the DSH image is built for amd64 and arm64 (no amd64 nodeSelector).
- **Single port** — DSH Web serves everything on `3080`; no API/dashboard port split.
- **One secret** — only `DEEPSEEK_API_KEY`; no dashboard/session/api-server credentials.
- **Host forwarding** — the gateway forwards `Host` + `X-Forwarded-*` (and keeps
`DSH_PUBLIC_URL` empty) so DSH generates reachable links; add the public hostname to
`DSH_TRUSTED_HOSTS`.
- **Browser sessions + WebSocket** — the gateway is Starlette ASGI, so it proxies WebSocket
upgrades as well as HTTP/SSE, and authenticates browsers with an `HttpOnly` cookie instead of
exposing per-user bearer tokens in URLs.
## Build it yourself
## Build from source
```bash
./build.sh # build the DSH Web image, then run it interactively
./build.sh web # build the DSH Web image only
./build.sh gateway # build the gateway image
make image # same as ./build.sh web
make gateway # gateway image
make test # gateway unit tests (pytest)
make lint # gateway lint (ruff)
make image # build the main DSH Web image (clones and builds upstream DSH)
make run # build it, then run it interactively (needs DEEPSEEK_API_KEY)
```
Or directly:
The pins (`DSH_VERSION`, `RELEASE_REVISION`, `PNPM_VERSION`) come from
[versions.env](versions.env) — never hard-code them
(`make check` verifies). [`build.sh`](build.sh) is a deprecated shim that
forwards to `make`; use the Makefile targets.
```bash
docker build \
--build-arg DSH_VERSION=dsh-v0.2.1-rc.2 \
--build-arg PNPM_VERSION=11.28.5 \
-t dsh-web:0.2.1-rc.2 .
## Examples
docker build -t ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway \
examples/dsh-agents-as-a-service/gateway/
```
Self-contained implementations that consume the image. Index and contract:
[`examples/README.md`](examples/README.md) and
[docs/examples.md](docs/examples.md).
See [`Dockerfile`](Dockerfile) for build args (`DSH_REPO`, `DSH_VERSION`, `PNPM_VERSION`) and
the runtime image contract (non-root `node` user, `DSH_HOME=/home/node/.dsh`, `EXPOSE 3080`).
`.env.sample` documents the runtime variables for both images.
| Example | What it demonstrates |
| --- | --- |
| [`examples/dsh-agents-as-a-service/`](examples/dsh-agents-as-a-service/) | Full multi-tenant platform: SandboxTemplate with per-user PVCs, warm pool, one claim per user, suspend-when-idle, wake-on-return, browser gateway with signup + `HttpOnly` sessions + HTTP/WebSocket proxy, Ingress or Gateway API + GitHub SSO exposure, and opt-in gVisor/NetworkPolicy/provider-key hardening. Start with the [walkthrough](examples/dsh-agents-as-a-service/README.md) and the [architecture](examples/dsh-agents-as-a-service/docs/architecture.md). |
| [`examples/minimal-single-tenant/`](examples/minimal-single-tenant/) | The smallest useful setup: one `SandboxTemplate` and one `SandboxClaim`, reached through `kubectl port-forward` + the launch-token URL. Manifest: [`manifests/sandbox-template.yaml`](examples/minimal-single-tenant/manifests/sandbox-template.yaml). |
The gateway is a component of the multi-tenant example and has its own
[README](examples/dsh-agents-as-a-service/gateway/README.md) (API, configuration,
auth/SSO, limitations). The example also documents opt-in gVisor,
`nodeSelector`, NetworkPolicy and extra provider-key hardening in
[optional hardening](examples/dsh-agents-as-a-service/docs/optional-hardening.md).
## Agent skills
DSH discovers project skills from `.dsh/skills/<name>/SKILL.md`. The repository
ships [`dsh-agent-sandbox-platform`](.dsh/skills/dsh-agent-sandbox-platform/SKILL.md),
which covers operating and extending the platform, the optional
[agent-sandbox MCP server](https://github.com/kubernetes-sigs/agent-sandbox/blob/main/clients/integrations/mcp-server/README.md),
and a read-only cluster preflight. Runbooks, the MCP reference, the extension
guide and troubleshooting notes live beside `SKILL.md`.
## Repository layout
| Path | Purpose |
| --- | --- |
| [`Dockerfile`](Dockerfile) | Primary artifact: the multi-stage DSH Web image (port `3080`). |
| [`versions.env`](versions.env) | Single source of truth for the DSH ref, repository revision, pnpm/control-plane versions and image names. |
| [`Makefile`](Makefile), [`build.sh`](build.sh) | Canonical local entrypoint; `build.sh` is a deprecated shim. |
| [`docs/`](docs/README.md) | Repository docs: image contract, example contract, gitflow, versioning. |
| [`examples/`](examples/README.md) | One self-contained implementation per directory. |
| [`examples/dsh-agents-as-a-service/`](examples/dsh-agents-as-a-service/) | Multi-tenant platform example: `manifests/`, `gateway/`, `docs/`, `scripts/`. |
| [`examples/minimal-single-tenant/`](examples/minimal-single-tenant/) | Single-sandbox quickstart. |
| [`scripts/`](scripts/) | Repository checks: versions, example layout, manifest YAML, version bump. |
| [`.dsh/skills/`](.dsh/skills/) | Project-local DSH skills, auto-discovered from the git root. |
| [`.github/`](.github/) | CI, release automation, dependency updates, PR/issue templates. |
| [`.env.sample`](.env.sample) | Documented local configuration (copy to `.env`; never commit it). |
## Development
The Makefile is the canonical entrypoint. No cluster is needed for `check`,
`test`, `lint`, `gateway` or `image`:
| Target | What it does |
| --- | --- |
| `make check` | Version pins, example layout and manifest YAML checks. Run before every commit. |
| `make test` | Gateway unit tests (`pytest`). |
| `make lint` | `ruff` + `shellcheck` (+ `yamllint`/`actionlint` when installed). |
| `make image` | Build the main DSH Web image. |
| `make gateway` | Build the example gateway image locally (`dsh-aaas-gateway:dev`). |
| `make e2e` | Full kind walkthrough of the multi-tenant example (needs a cluster with agent-sandbox). |
| `make changelog` | Regenerate `CHANGELOG.md` with git-cliff (release only). |
| `make set-version` | Bump pins in `versions.env`; e.g. `make set-version DSH_VERSION=<ref> RELEASE_REVISION=rN`. |
Local setup and the PR workflow: [CONTRIBUTING.md](CONTRIBUTING.md); branches,
required checks and releases: [docs/gitflow.md](docs/gitflow.md).
## Versioning and releases
The release identity is the upstream DSH ref plus an optional repository
revision: `DSH_VERSION` and `RELEASE_REVISION` in [versions.env](versions.env).
The current identity is **`dsh-v0.2.1-rc.5`** — the first release at this
upstream DSH ref, so it carries no repository revision. Tags are immutable; a
broken release is fixed by publishing `-r(N+1)`, never by re-tagging.
`CHANGELOG.md` is generated by [git-cliff](https://git-cliff.org) from
Conventional Commits and is never edited by hand. See
[`CHANGELOG.md`](CHANGELOG.md), [docs/versioning.md](docs/versioning.md) and
[cliff.toml](cliff.toml).
## Migrating from the previous layout
If you are coming from an older checkout, these paths moved. Nothing else changed
semantically; the content was split into the example it belongs to.
| Old path | New path |
| --- | --- |
| `gateway-api/dsh-demo/` (Gateway API + oauth2-proxy / GitHub SSO) | `examples/dsh-agents-as-a-service/manifests/exposure/gateway-api-sso/` (now `70-gateway.yaml` ... `74-tls-secret.yaml.example`) |
| `examples/dsh-agents-as-a-service/60-ingress.yaml` | `examples/dsh-agents-as-a-service/manifests/exposure/ingress/60-ingress.yaml` |
| `examples/dsh-agents-as-a-service/NN-*.yaml` (`00-` ... `50-`) | `examples/dsh-agents-as-a-service/manifests/NN-*.yaml` |
| `examples/dsh-agents-as-a-service/run-test-kind.sh` | `examples/dsh-agents-as-a-service/scripts/e2e-kind.sh` |
| `deploy/k8s/sandbox-template.yaml` | `examples/minimal-single-tenant/manifests/sandbox-template.yaml` |
| `docs/architecture.md` | `examples/dsh-agents-as-a-service/docs/architecture.md` |
The SSO walkthrough that lived in `gateway-api/dsh-demo/README.md` is merged
into the [multi-tenant example README](examples/dsh-agents-as-a-service/README.md)
(step 9) and the [exposure README](examples/dsh-agents-as-a-service/manifests/exposure/README.md).
## Contributing
- [CONTRIBUTING.md](CONTRIBUTING.md) — local setup, PR workflow, what a change needs.
- [AGENTS.md](AGENTS.md) — repository conventions for AI coding agents (and humans) and the CI commands.
- [SECURITY.md](SECURITY.md) — private vulnerability reporting and project threat notes.
- [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md) — community standards.
- [docs/gitflow.md](docs/gitflow.md) — branches, protected `main`, required checks, release process.
Do not commit secrets; secret-shaped YAML lives in `*.yaml.example` files, and
version pins change only through `versions.env` / `make set-version`.
## License
[MIT](LICENSE) © Nicolás Georger.

48
SECURITY.md Normal file
View File

@@ -0,0 +1,48 @@
# Security policy
## Supported versions
Only the latest published release is supported. Image tags are immutable;
fixes ship as a new release (see docs/versioning.md).
## Reporting a vulnerability
Report vulnerabilities privately through GitHub Security Advisories:
https://github.com/sredevopsorg/deepseek-kubernetes-sandbox/security/advisories/new
Do not open a public issue, and do not include live credentials in a report.
You can expect an acknowledgement within a few days and a coordinated
disclosure once a fix is available.
## Threat notes specific to this project
- **Launch token in pod logs.** The example gateway reads DSH's launch token
from pod logs (pods/log). Anyone with access to those logs can impersonate a
tenant. Keep the gateway's RBAC namespace-scoped and treat pod logs as
sensitive. Replacing log scraping with an upstream token API is tracked in
the example architecture document.
- **SSO header trust boundary.** AUTH_PROXY_HEADER makes the gateway trust an
identity header (for example X-Forwarded-User). Enable it only when the
gateway Service is reachable exclusively through the authenticating proxy;
otherwise any client can spoof the header. The SSO exposure variant ships a
NetworkPolicy that restricts gateway access to oauth2-proxy pods; a custom
exposure needs an equivalent.
- **Shared provider keys.** The example injects DEEPSEEK_API_KEY and any keys in
the optional dsh-provider-keys-extra Secret into every tenant sandbox (a
warm-pool constraint). Treat tenants as mutually trusting until per-tenant
credentials or an egress proxy exist.
- **Optional isolation is opt-in.** gVisor, nodeSelector, the sandbox/exposure
NetworkPolicies and extra provider keys are opt-in. Without a
NetworkPolicy-capable CNI or a gVisor runtime, the base example is not
tenant-isolated; see examples/dsh-agents-as-a-service/docs/optional-hardening.md.
- **TLS is external.** Terminate TLS at an Ingress or Gateway and set
COOKIE_SECURE=true. Plain HTTP outside localhost exposes session cookies.
- **No secrets in the repository.** The repository only ships empty Secret
manifests and *.yaml.example files.
## Dependency and supply chain
GitHub Actions are pinned by commit SHA. The CodeQL workflow scans Python and
Actions. The release workflow publishes images with SBOM and provenance
attestations.

View File

@@ -1,68 +1,19 @@
#!/usr/bin/env bash
# Build the DSH Web and/or gateway images locally.
# Deprecated compatibility shim: "make" is the canonical entrypoint.
#
# Usage:
# ./build.sh # build the DSH Web image, then run it interactively
# ./build.sh web # build the DSH Web image only
# ./build.sh gateway # build the gateway image (examples/.../gateway/)
# ./build.sh -> make run (build the main image, then run it)
# ./build.sh web -> make image
# ./build.sh gateway -> make gateway
#
# Overridable environment:
# DSH_VERSION DSH git ref to build (default dsh-v0.2.1-rc.2)
# PNPM_VERSION pnpm version (default 11.28.5)
# IMAGE DSH Web image tag (default dsh-web:<version>)
# DSH_PORT host port for docker run (default 3080)
# GATEWAY_IMAGE gateway image tag (default ghcr.io/...:gateway)
# RUN 0 to build without running (default 1)
# Use the Makefile targets directly for everything else (test, lint, check, e2e).
set -euo pipefail
cd "$(dirname "$0")"
DSH_VERSION="${DSH_VERSION:-dsh-v0.2.1-rc.2}"
PNPM_VERSION="${PNPM_VERSION:-11.28.5}"
IMAGE="${IMAGE:-dsh-web:${DSH_VERSION#dsh-v}}"
GATEWAY_IMAGE="${GATEWAY_IMAGE:-ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway}"
DSH_PORT="${DSH_PORT:-3080}"
RUN="${RUN:-1}"
GATEWAY_DIR=examples/dsh-agents-as-a-service/gateway
build_web() {
docker build \
--build-arg "DSH_VERSION=${DSH_VERSION}" \
--build-arg "PNPM_VERSION=${PNPM_VERSION}" \
-t "${IMAGE}" \
"$@" \
.
}
build_gateway() {
docker build -t "${GATEWAY_IMAGE}" "$@" "${GATEWAY_DIR}/"
}
usage() {
echo "usage: $0 [run|web|gateway] [extra docker build args...]" >&2
exit 2
}
echo "build.sh is deprecated; use 'make image', 'make run' or 'make gateway'." >&2
case "${1:-run}" in
run)
shift
build_web "$@"
if [ "${RUN}" = "1" ]; then
docker run --rm -it \
-p "${DSH_PORT}:3080" \
-v dsh-home:/home/node/.dsh \
-e DEEPSEEK_API_KEY \
"${IMAGE}"
fi
;;
web)
shift
build_web "$@"
;;
gateway)
shift
build_gateway "$@"
;;
*)
usage
;;
run) exec make run ;;
web) exec make image ;;
gateway) exec make gateway ;;
*) echo "usage: $0 [run|web|gateway]" >&2; exit 2 ;;
esac

49
cliff.toml Normal file
View File

@@ -0,0 +1,49 @@
# git-cliff configuration: release notes and CHANGELOG.md are generated from
# Conventional Commits. See docs/versioning.md for the release procedure.
[changelog]
header = """
# Changelog
All notable changes to this repository are documented here.
Generated by git-cliff from Conventional Commits; do not edit this file by hand.
"""
body = """
{% if version %}\
## [{{ version }}] - {{ timestamp | date(format="%Y-%m-%d") }}
{% else %}\
## [Unreleased]
{% endif %}\
{% for group, commits in commits | group_by(attribute="group") %}
### {{ group | upper_first }}
{% for commit in commits %}
- {% if commit.scope %}**{{ commit.scope }}:** {% endif %}{{ commit.message | split(pat="\n") | first | upper_first }} ({{ commit.id | truncate(length=7, end="") }})
{%- endfor %}
{% endfor %}\n
"""
footer = ""
trim = true
[git]
conventional_commits = true
filter_unconventional = false
split_commits = false
commit_parsers = [
{ message = "^feat", group = "Added" },
{ message = "^fix", group = "Fixed" },
{ message = "^perf", group = "Performance" },
{ message = "^refactor", group = "Changed" },
{ message = "^docs", group = "Documentation" },
{ message = "^build", group = "Build" },
{ message = "^ci", group = "CI" },
{ message = "^test", group = "Tests" },
{ message = "^chore", group = "Maintenance" },
{ message = "^revert", group = "Reverted" },
{ message = ".*", group = "Other" },
]
protect_breaking_commits = false
filter_commits = false
# Release tags are dsh-vX.Y.Z[-rN]; the single legacy pre-scheme tag is v0.0.1.
tag_pattern = "^(v|dsh-v).*$"
sort_commits = "oldest"

47
docs/README.md Normal file
View File

@@ -0,0 +1,47 @@
# Repository documentation
Repository-wide documentation lives here. Example-specific material lives with
its example under [`examples/<name>/docs/`](../examples/) — see the
[agents-as-a-service architecture](../examples/dsh-agents-as-a-service/docs/architecture.md)
for the deep-dive that used to live in this directory.
If you are new, read the [root README](../README.md) first: it is image-first and
gets a DSH Web container running before anything else.
## Index
| Document | Read it for |
| --- | --- |
| [main-image.md](main-image.md) | The primary artifact: build args, runtime environment, entrypoint, probes, volumes/ports/signals, non-root contract, image names, tags and digests. |
| [examples.md](examples.md) | The example contract: required layout, manifest rules, exposure variants, components and image naming, scripts, docs, and how to add a new example. |
| [gitflow.md](gitflow.md) | Branches, protected `main`, squash merges, Conventional PR titles, required CI checks, release branches, hotfixes and the one-time GitHub settings checklist. |
| [versioning.md](versioning.md) | Release identity (`DSH ref` + optional `-rN`), the current identity `dsh-v0.2.1-rc.5`, why identities are immutable, channels, compatibility matrix, bumping pins, the release checklist, rollback, git-cliff and digest pinning. |
## Project skills
DSH discovers project skills from `<git root>/.dsh/skills/<name>/SKILL.md`
(the nearest `.git` ancestor). This repository ships one:
| Skill | Read it for |
| --- | --- |
| [dsh-agent-sandbox-platform](../.dsh/skills/dsh-agent-sandbox-platform/SKILL.md) | Operating and extending the agents-as-a-service platform, agent-sandbox MCP limits and deployment, and a read-only preflight. |
## Example documentation
| Document | Read it for |
| --- | --- |
| [examples/README.md](../examples/README.md) | The index of implementations in this repository. |
| [dsh-agents-as-a-service README](../examples/dsh-agents-as-a-service/README.md) | The multi-tenant platform walkthrough: warm pool, claims, suspend/resume, gateway, exposure and troubleshooting. |
| [dsh-agents-as-a-service architecture](../examples/dsh-agents-as-a-service/docs/architecture.md) | Container/sequence diagrams, ADRs, security and RBAC, failure modes and the evolution path. |
| [gateway README](../examples/dsh-agents-as-a-service/gateway/README.md) | The gateway's HTTP/WebSocket API, configuration, auth (including SSO), and limitations. |
## Project policies
| Document | Read it for |
| --- | --- |
| [AGENTS.md](../AGENTS.md) | Repository conventions for AI agents and humans, and the commands the CI runs. |
| [CONTRIBUTING.md](../CONTRIBUTING.md) | Local setup, the change workflow and what a pull request needs. |
| [SECURITY.md](../SECURITY.md) | Vulnerability reporting and project-specific threat notes (pod logs, SSO header trust, shared provider key, TLS). |
| [CODE_OF_CONDUCT.md](../CODE_OF_CONDUCT.md) | Community standards. |
| [CHANGELOG.md](../CHANGELOG.md) | Generated release history (do not edit by hand). |
| [LICENSE](../LICENSE) | MIT. |

159
docs/examples.md Normal file
View File

@@ -0,0 +1,159 @@
# Example contract
Every implementation in this repository is a **self-contained example** under
[`examples/<name>/`](../examples/). Examples demonstrate how to run the main
DSH Web image (and, when needed, auxiliary components such as the
agents-as-a-service gateway) on Kubernetes. They are reference material, not
libraries: each one is applied with plain `kubectl`, documented end to end, and
deletable in one command.
[`examples/README.md`](../examples/README.md) is the index. If an example is not
listed there, it does not exist as far as `make check` is concerned.
## Required layout
```
examples/<name>/
+-- README.md # required: what it demonstrates, prerequisites,
| # ordered steps, troubleshooting, cleanup
+-- manifests/ # required
| +-- NN-*.yaml # numbered top-level manifests, plain kubectl
| +-- optional/ # optional opt-in manifests, applied by path
| +-- README.md # required when optional/ exists
| +-- <feature>/NN-*.yaml # e.g. network-policy/
| +-- exposure/ # optional exposure variants
| +-- README.md # required when exposure/ exists
| +-- <variant>/NN-*.yaml # e.g. ingress/, gateway-api-sso/
+-- scripts/ # optional
| +-- *.sh # executable, idempotent, self-cleaning
+-- docs/ # optional deeper material (architecture, ADRs)
+-- <component>/ # optional published component
+-- Dockerfile
+-- README.md # required whenever a Dockerfile exists
+-- tests...
```
### Rules the CI enforces
[scripts/check-examples.sh](../scripts/check-examples.sh) fails the build when:
- `examples/README.md` is missing or does not mention `examples/<name>/` for every
example directory;
- an example has no `README.md` or no `manifests/` directory;
- `manifests/` contains no top-level `*.yaml` files;
- a `*.yaml` file sits at the example root instead of under `manifests/`;
- `manifests/exposure/` exists without a `manifests/exposure/README.md` describing
the variants;
- `manifests/optional/` exists without a `manifests/optional/README.md`
describing the opt-in manifests;
- a subdirectory with a `Dockerfile` has no `README.md`;
- a file in `scripts/` is not executable.
[scripts/check-manifests.py](../scripts/check-manifests.py) additionally parses
every `manifests/**/*.yaml` and requires `apiVersion`, `kind` and
`metadata.name`. CI also runs a server-side dry-run against the real
agent-sandbox CRDs on a throwaway kind cluster.
Run both locally with:
```bash
make check
```
## Manifests
- **Numbered and ordered.** `NN-*.yaml` files are applied in filename order and
read as a walkthrough: `00-prereqs.yaml`, `10-sandboxtemplate.yaml`,
`20-sandboxwarmpool.yaml`, ... Keep the numbering stable; it is referenced from
the example README.
- **Plain kubectl.** No Helm, Kustomize, or custom installer. `kubectl apply -f`
against a namespace the example creates is the whole interface.
- **Safe by default.** Never ship a usable Secret. Secret-shaped manifests are
`*.yaml.example` files with obvious placeholders and an inline creation
command; real values are generated per install.
- **Cluster-specific state stays out.** Node affinity, replica counts tuned for a
cluster, and similar fields belong in a local patch, not in the shipped
manifest. Optional scheduler/runtime hints may ship **commented out** with
placeholder labels; they must not be active.
- **Optional manifests are opt-in.** Put them under
`manifests/optional/<feature>/`, document them in
`manifests/optional/README.md`, and apply them by path. The top-level
`kubectl apply -f manifests/` is non-recursive and does not apply them.
- **Placeholders are explicit.** Hostnames, GitHub orgs and TLS material use
values like `dsh.example.com` / `your-github-org` with a TODO comment.
## Exposure variants
An example that can be reached from outside the cluster puts each option under
`manifests/exposure/<variant>/` and describes the trade-offs in
`manifests/exposure/README.md`. Variants are alternatives, not a stack: apply one.
`dsh-agents-as-a-service` currently ships:
| Variant | Path | What it gives you |
| --- | --- | --- |
| Ingress | `manifests/exposure/ingress/` | Traefik-style Ingress + TLS; the gateway does its own session auth. |
| Gateway API + SSO | `manifests/exposure/gateway-api-sso/` | Gateway API `Gateway`/`HTTPRoute` in front of oauth2-proxy (GitHub OAuth); the gateway inherits `X-Forwarded-User` via `AUTH_PROXY_HEADER`. |
## Components
A component is a program the example needs beyond the main image: today only the
agents-as-a-service gateway. A component lives in its own subdirectory with:
- `Dockerfile` — its image build;
- `README.md` — purpose, API/config reference, local development, limitations;
- tests — unit tests that run without a cluster, plus whatever else applies.
Component images are published as **nested packages below the repository image**:
```
<registry>/<owner>/<repo>/<example>/<component>
```
so the gateway is
`ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway`.
Never create an unrelated top-level package; `check-versions.sh` verifies the
nesting.
## Scripts
`scripts/*.sh` must be:
- **executable** (the CI checks the mode bit);
- **idempotent** — safe to re-run, cleaning up before creating;
- **self-cleaning** — a trap removes the namespace, port-forwards and temp files,
even on failure;
- **non-interactive** — usable in CI, with environment-variable knobs instead of
prompts.
`examples/dsh-agents-as-a-service/scripts/e2e-kind.sh` is the reference
implementation: it runs the README walkthrough and deletes everything it created.
## Documentation
- The example `README.md` is the walkthrough. Keep it runnable: every command
block should work when pasted in order.
- Deeper material (architecture, sequence diagrams, ADRs) belongs in the
example's `docs/`, not in the repository-level [docs/](../docs/).
- Repository-wide docs describe the container and the repo rules; example docs
describe one implementation.
## Adding a new example
1. Create `examples/<name>/` with the layout above.
2. Write `README.md` first: what it demonstrates, prerequisites, numbered steps,
troubleshooting, limitations, cleanup.
3. Add `manifests/NN-*.yaml`; keep secrets as `*.yaml.example`.
4. If a component needs an image, publish it under
`<registry>/<owner>/<repo>/<example>/<component>`.
5. Add an entry to [`examples/README.md`](../examples/README.md) using the literal
`examples/<name>/` path.
6. Run `make check` and, if you added a component, `make test` and `make lint`.
7. If the example is expensive to run (kind, images, cluster CRDs), ship a
`scripts/e2e-kind.sh` and wire it into CI.
## Current examples
| Example | Demonstrates |
| --- | --- |
| [`examples/dsh-agents-as-a-service/`](../examples/dsh-agents-as-a-service/) | The full multi-tenant platform pattern: SandboxTemplate, warm pool, per-user claims, suspend/resume, and the browser gateway. |
| [`examples/minimal-single-tenant/`](../examples/minimal-single-tenant/) | A single DSH Web sandbox: one SandboxTemplate and a claim. |

128
docs/gitflow.md Normal file
View File

@@ -0,0 +1,128 @@
# Gitflow: branches, pull requests and CI
This repository uses **trunk-based development with release branches**. `main` is
always releasable; feature work happens on short-lived branches; releases are cut
from `main` onto a release branch and tagged with an immutable release identity.
See [versioning](versioning.md) for what the tags mean.
## Branches
| Branch | Branched from | Merged into | Purpose |
| --- | --- | --- | --- |
| `main` | — | — | The integration trunk. Protected, linear history, always releasable. |
| `feat/<slug>` | `main` | `main` (squash) | New capability or example. |
| `fix/<slug>` | `main` | `main` (squash) | Bug fix. |
| `docs/<slug>` | `main` | `main` (squash) | Documentation only. |
| `refactor/<slug>`, `chore/<slug>`, `test/<slug>` | `main` | `main` (squash) | Non-behavioral rework, maintenance, tests. |
| `release/<identity>` | `main` at the release commit | — | Release stabilization and tagging. Only release-blocking fixes land here. |
| `hotfix/<slug>` | The release branch (or the tagged commit) | `main` and the release branch | Urgent production fix. |
Keep branches short-lived. Rebase on `main` rather than merging `main` into the
branch; the PR is squash-merged, so individual branch commits are never part of
`main` history.
## Pull requests
1. **One scoped change per PR.** Do not mix a version bump with unrelated work,
and update the docs the change touches in the same PR.
2. **Conventional PR title.** The title becomes the squash commit message, so it
is linted by the `pr-title` check: `feat(scope): ...`, `fix(scope): ...`,
`docs(scope): ...`, `build: ...`, `ci: ...`, `refactor: ...`, `test: ...`,
`chore: ...`. Breaking changes use `feat(scope)!: ...` or a
`BREAKING CHANGE:` footer.
3. **CI must be green.** All required checks below must pass before merge.
4. **Squash merge only.** `main` stays linear and protected; merged PRs are one
commit each.
5. **No direct pushes to `main`** or to a release branch. Both are protected.
6. **Review for behavior changes.** Infrastructure, workflow, release and
gateway-contract changes need a maintainer review; trivial docs changes still
need the checks.
## Required checks
| Check | Workflow / job | What it verifies |
| --- | --- | --- |
| `lint` | `ci.yaml` | `scripts/check-versions.sh` (version pins and markdown tokens), ruff on the gateway, shellcheck on shell scripts, yamllint/actionlint when available. |
| `python` | `ci.yaml` | Gateway unit tests (`pytest`). |
| `manifests` | `ci.yaml` | `scripts/check-examples.sh`, `scripts/check-manifests.py` and a server-side dry-run of every example manifest against the agent-sandbox CRDs. |
| `gateway-image` | `ci.yaml` | Builds the gateway image and smoke-tests `/healthz` without a cluster. |
| `pr-title` | `pr-title.yaml` | Conventional Commit PR title. |
| `CodeQL` | `codeql.yml` | Static analysis for Python and GitHub Actions. |
| `dependency-review` | `ci.yaml` | Dependency changes introduced by the PR (job `dependency-review`; runs on pull requests only). |
The image build workflows (`_build-image.yaml`, `build-main.yaml`,
`build-gateway.yaml`) and `e2e-kind.yaml` are **not** required on every PR: the
image builds run where they are needed, and the kind end-to-end suite is reserved
for release validation and manual dispatch because it is slow.
## Releases
Releases are cut from `main`:
1. Confirm `main` is green and `make check`, `make test`, `make lint` pass.
2. Bump [versions.env](../versions.env) if the DSH ref changed, using
`make set-version`; otherwise keep the same `DSH_VERSION` and bump
`RELEASE_REVISION`.
3. Create `release/<identity>` from the release commit.
4. Run `make release-dry-run` and review the generated notes.
5. Create and push the annotated `dsh-*` release tag. The `release.yaml`
workflow validates that the tag matches [versions.env](../versions.env), builds
and pushes the multi-arch images (version + latest), creates the GitHub
Release from the git-cliff notes, and opens a `chore(changelog)` PR that
refreshes `CHANGELOG.md` on `main`.
6. Merge the changelog PR and any release-branch-only fixes back to `main`.
Full procedure: [versioning](versioning.md).
## Hotfixes
A released tag is **never** moved or re-tagged — tags and image identities are
immutable. To fix a broken release:
1. Branch `hotfix/<slug>` from the release tag (or its release branch).
2. Make the minimal fix, add a test, open a PR; the same required checks apply.
3. Publish a new identity with a higher `RELEASE_REVISION` (`-r(N+1)`). Do not
reuse or delete the broken tag.
4. Merge the fix back into `main` (or cherry-pick it) so the trunk is fixed too.
Rollback is also a re-release, never a retag: see
[versioning](versioning.md#rollback).
## Tag protection
All release tags match `dsh-*` and are protected:
- only maintainers (the release workflow) may create them;
- they cannot be deleted or force-updated;
- they are annotated and point at the release commit on a release branch.
Image tags derive from the same identity and are equally immutable; the mutable
`latest` and `main` image channels always point at a release or a trunk build
respectively.
## One-time GitHub settings checklist
Configure these on the repository before relying on the workflow:
- [ ] **Branches → main**: require a pull request; require the checks `lint`,
`python`, `manifests`, `gateway-image`, `pr-title`, `CodeQL`,
`dependency-review`; require branches to be up to date; require conversation
resolution; restrict direct pushes; allow only squash merges and enable
"Automatically delete head branches".
- [ ] **Branches → release/***: same protections, plus allow the release workflow
to push tags.
- [ ] **Tags → dsh-***: restrict creation/update/deletion to maintainers; a tag
push triggers the release workflow.
- [ ] **Actions → General**: allow GitHub Actions; set workflow permissions to
read-only by default; allow `GITHUB_TOKEN` to create pull requests only
where required.
- [ ] **Actions → Runners**: make the arm64 runners (`ubuntu-24.04-arm`) available
to the public/private repository as appropriate.
- [ ] **Packages**: grant the release workflow `packages: write` (workflow
permissions do this) and protect the
`ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox` packages from deletion.
- [ ] **Security**: enable CodeQL default setup **or** keep the advanced
`codeql.yml` workflow (not both), enable Dependabot alerts and
dependency review.
- [ ] **Secrets/Variables**: none are needed for image publishing; the workflows
use `GITHUB_TOKEN`.

193
docs/main-image.md Normal file
View File

@@ -0,0 +1,193 @@
# The main DSH Web image
This repository's primary artifact is a container image that packages
[DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) Web
(`dsh web`) for Kubernetes and plain Docker. The [Dockerfile](../Dockerfile) at the
repository root builds it; every version pin it uses lives in
[versions.env](../versions.env).
The image is image-first: it is the product, and the examples under
[`examples/`](../examples/) are self-contained implementations that consume it.
## What the build does
The Dockerfile is a two-stage build:
1. **Build stage** (`node:24-trixie`) clones DSH at the pinned git ref, installs
dependencies with pnpm, runs the production build, packages the `@deepseek-ai/dsh`
CLI with `pnpm pack`, and installs its production dependencies with npm.
2. **Runtime stage** (`node:24-trixie-slim`) copies `/opt/dsh` to `/app`, adds the
CLI tools DSH agents commonly use (`git`, `ripgrep`, `jq`, `yq`, `curl`, `wget`,
`gh`, `tini`, `uv`), and starts `dsh web` through a small entrypoint.
## Build it
```bash
make image # canonical: builds the DSH Web image, no push
```
`make image` reads [versions.env](../versions.env) and passes the pins as build args.
To reproduce the equivalent by hand:
```bash
docker build \
--build-arg DSH_VERSION=dsh-v0.2.1-rc.5 \
--build-arg RELEASE_REVISION= \
--build-arg PNPM_VERSION=11.28.5 \
--build-arg VCS_REF="$(git rev-parse --short HEAD)" \
-t ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.5 .
```
### Build args
| Arg | Default / source | Purpose |
| --- | --- | --- |
| `DSH_REPO` | `https://github.com/deepseek-ai/deepseek-harness.git` | Upstream DSH repository to clone. |
| `DSH_VERSION` | `DSH_VERSION` in [versions.env](../versions.env) | Upstream DSH git tag/ref to package; recorded as `org.opencontainers.image.version`. |
| `RELEASE_REVISION` | `RELEASE_REVISION` in [versions.env](../versions.env) | Repository revision suffix (`rN`); the release identity is `DSH_VERSION-RELEASE_REVISION`, recorded in the `io.sredevopsorg.release.revision` label. See [versioning](versioning.md). |
| `PNPM_VERSION` | `PNPM_VERSION` in [versions.env](../versions.env) | pnpm version activated with corepack in both stages. |
| `VCS_REF` | `git rev-parse --short HEAD` (via `make image`) | Source revision recorded for traceability. |
| `NODE_BUILD_IMAGE` | `docker.io/node:24-trixie` | Build-stage base image. |
| `NODE_RUNTIME_IMAGE` | `docker.io/node:24-trixie-slim` | Runtime-stage base image. |
Never hard-code a version outside [versions.env](../versions.env):
`./scripts/check-versions.sh` (and `make check`) fails when a pin drifts.
## Runtime contract
### Entrypoint
The entrypoint is `tini -- /usr/local/bin/dsh-entrypoint`:
1. `tini` runs as PID 1 and reaps children/forwards signals.
2. The shell entrypoint applies defaults, derives a bind address when `DSH_HOST` is
empty (the pod/container's first non-loopback IPv4), builds the
`dsh web` command line, and `exec`s it so signals reach the Node process.
3. `localhost` and `127.0.0.1` are always added to DSH's trusted-host list; each
`DSH_TRUSTED_HOSTS` entry is added as an extra `--trusted-host`.
### Environment variables
| Variable | Default | Purpose |
| --- | --- | --- |
| `DSH_PORT` | `3080` | Port DSH Web listens on inside the container. |
| `DSH_HOST` | auto-detected non-loopback IPv4 | Bind address. Leave empty in Kubernetes so the pod IP is used automatically. |
| `DSH_PUBLIC_URL` | `http://localhost:${DSH_PORT}` | Public base URL used to build links and the launch-token URL printed in the logs. Set it when the container is published on a different host port or behind a proxy. |
| `DSH_TRUSTED_HOSTS` | empty | Extra `Host`/`Origin` allow-list entries, comma- or space-separated. `localhost`/`127.0.0.1` are always trusted. Required when a proxy forwards the browser's `Host`. |
| `DEEPSEEK_API_KEY` | empty | Provider credential read by DSH. Inject it with `-e`/`envFrom`; never bake it into the image. Only model calls fail when it is missing. |
| `DSH_HOME` | `/home/node/.dsh` | DSH state directory (conversations, sessions, config, credentials). Mount a volume here to persist it. |
The image also sets `NODE_ENV=production`, `HOME=/home/node`, and a `PATH` that
includes `/app/node_modules/.bin`.
### Provider credentials
DSH resolves provider API keys from the environment when no stored credential
exists. The packaged provider catalog maps each provider to its standard
environment name. Commonly used names:
`DEEPSEEK_API_KEY`, `OPENROUTER_API_KEY`, `GEMINI_API_KEY`, `NVIDIA_API_KEY`,
`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GROQ_API_KEY`, `XAI_API_KEY`,
`MISTRAL_API_KEY`, `TOGETHER_API_KEY`, `CEREBRAS_API_KEY`,
`AI_GATEWAY_API_KEY`, `HF_TOKEN`.
Use `GEMINI_API_KEY` for Google Gemini (not `GOOGLE_API_KEY`), `HF_TOKEN` for
Hugging Face, and `AI_GATEWAY_API_KEY` for the Vercel AI Gateway. Inject keys
with `-e`/`--env-file` in Docker or a Secret + `envFrom` in Kubernetes; never
bake them into the image.
A key saved through DSH's credential store (`$DSH_HOME/.credentials.yaml`) wins
over the environment. The
[agents-as-a-service example](../examples/dsh-agents-as-a-service/docs/optional-hardening.md)
documents the optional `dsh-provider-keys-extra` Secret and the pod-refresh
procedure.
### Ports, volumes and signals
| Item | Value | Notes |
| --- | --- | --- |
| Exposed port | `3080/tcp` | `EXPOSE 3080`; one port serves the UI, assets, API, SSE and WebSocket traffic. |
| State volume | `/home/node/.dsh` | Declared with `VOLUME`; use a named volume (Docker) or PVC (Kubernetes). |
| Working directory | `/workspace` | The agent workspace; mount a volume here if agents should keep files. |
| Stop signal | `SIGTERM` | `tini` forwards it to `dsh web`; allow a graceful shutdown period. |
| User | `node`, UID/GID `1000` | The process is non-root. A PVC mounted at `DSH_HOME` must be writable by UID 1000. |
### Health probes
`/` is **not** a health endpoint: DSH returns `401` until a browser completes the
launch-token handshake. Probe `/favicon.svg` instead — it is unauthenticated and
served by the web layer, so it proves the HTTP server is up without triggering auth.
```yaml
startupProbe:
httpGet: {path: /favicon.svg, port: 3080}
periodSeconds: 5
failureThreshold: 24
readinessProbe:
httpGet: {path: /favicon.svg, port: 3080}
periodSeconds: 5
livenessProbe:
httpGet: {path: /favicon.svg, port: 3080}
periodSeconds: 15
```
The shipped templates use exactly this shape; see
[`examples/minimal-single-tenant/manifests/sandbox-template.yaml`](../examples/minimal-single-tenant/manifests/sandbox-template.yaml).
A `startupProbe` is important because plugin activation (or the first start of a
large sandbox) can exceed a default liveness budget.
### Launch token
DSH prints its one-time launch URL to the logs:
```
dsh web: http://<host>:${DSH_PORT}/?token=...
```
Open that URL once to obtain DSH's auth cookie. After that the UI works at the bare
URL; without the cookie `/` answers `401` by design. The
[agents-as-a-service gateway](../examples/dsh-agents-as-a-service/gateway/) performs
this handshake server-side so browsers never see the token.
## Published image names and tags
Registry layout: the main image is the repository itself, and every example
component is a nested package below it:
```
ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox # main DSH Web image
ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway # example component
```
The main image is published multi-arch (`linux/amd64`, `linux/arm64`) to
GitHub Container Registry:
| Tag | Meaning |
| --- | --- |
| `dsh-v0.2.1-rc.5` | Current release identity. **Immutable**: once published it never moves. |
| `latest` | Most recent release; points at the same digest as the current identity. |
| `main` | Development channel rebuilt from every push to `main`; not a release, not for production. |
Tags published before the release-identity scheme remain published but are not
updated; new deployments use the current identity.
Release identities, channels and rollback rules are defined in
[versioning](versioning.md). Tags are immutable, so production deployments should
pin a digest:
```bash
docker buildx imagetools inspect ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.5
# ... pin the digest in your deployment manifests:
# image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox@sha256:<digest>
```
## Verifying an image
```bash
docker inspect ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.5 \
--format '{{json .Config.Labels}}'
# org.opencontainers.image.title / .version / .source
```
Release builds also publish SBOM and provenance attestations from the release
workflow (see [SECURITY.md](../SECURITY.md) and [gitflow](gitflow.md)).

164
docs/versioning.md Normal file
View File

@@ -0,0 +1,164 @@
# Versioning, releases and changelog
[versions.env](../versions.env) is the **single source of truth** for the DSH ref,
the repository revision, the pnpm version, the control-plane component versions
and the image names. Nothing else may hard-code a version:
[scripts/check-versions.sh](../scripts/check-versions.sh) fails when a pin drifts,
and it also verifies that prose mentions only the current release identity or its
base DSH tag.
## Release identity
```
release identity = DSH_VERSION + optional "-" + RELEASE_REVISION
```
| Piece | Meaning |
| --- | --- |
| `DSH_VERSION` | The upstream DeepSeek Harness git ref being packaged, e.g. `dsh-v0.2.1-rc.5`. It mirrors upstream's own tag, which is what "mirrored-DSH scheme" means. |
| `RELEASE_REVISION` | An optional repository revision at that DSH ref: empty for the first release, `rN` (`N >= 2`) for a repository-only re-release. |
Examples:
- `DSH_VERSION=dsh-v0.2.1-rc.5`, `RELEASE_REVISION=` → identity `dsh-v0.2.1-rc.5`
(first release at that upstream ref).
- `DSH_VERSION=<ref>`, `RELEASE_REVISION=rN` → identity `<ref>-rN` (for example
documentation, CI or gateway changes on top of an already-released ref; no DSH
upgrade). Use `r2` for the first re-release: revision 1 is the plain tag.
A DSH upgrade produces a new base identity (for example the next upstream ref
with an empty revision); a repository-only change at the same DSH ref increments
`rN`.
## Why release identities are immutable
**Container tags are immutable** in this project: re-pushing a tag would silently
change what an existing deployment runs. A DSH upgrade therefore produces a new
base identity with an empty revision; a repository-only change at an
already-released ref publishes `-rN`, where `r2` is the first re-release (the
plain tag is conceptually revision 1 and is taken).
The current identity is `dsh-v0.2.1-rc.5`, the first release at this upstream
ref, so it carries no revision suffix. The identity is stored in
[versions.env](../versions.env) and consumed everywhere: the Dockerfile label,
the example manifests, the Makefile and the release workflow. Tags published
before the release-identity scheme remain published but are not updated; new
deployments use the current identity.
The gateway image moved from the repository-level `:gateway` tag to a nested
package (`<repository>/dsh-agents-as-a-service/gateway`). The release workflow
also republishes the legacy `<repository>:gateway` tag as a deprecated
compatibility alias; it may be removed in a later release, so deployments
should pin the nested package.
## Channels
| Channel | Image tag | Meaning |
| --- | --- | --- |
| Release | `dsh-v0.2.1-rc.5` | The current immutable release identity. What production should pin (preferably by digest). |
| `latest` | `latest` | Alias for the most recent release; moves only when a new release is published. |
| `main` | `main` | Development channel, rebuilt from every push to `main`. Never treat it as a release; it exists so `main` can be tested. |
The same channels exist for every published package, including the nested
example component image.
## Compatibility matrix
Values from [versions.env](../versions.env) at the current identity:
| Component | Pin | Source |
| --- | --- | --- |
| Upstream DSH source | `DSH_REPO` | [versions.env](../versions.env) |
| DSH ref | `dsh-v0.2.1-rc.5` | `DSH_VERSION` |
| Repository revision | `(empty)` | `RELEASE_REVISION` |
| pnpm | `11.28.5` | `PNPM_VERSION` |
| agent-sandbox (CRDs) | `v1.0.8` | `AGENT_SANDBOX_VERSION`, used by the examples and CI dry-run |
| Kubernetes Gateway API | `v1.2.1` | `GATEWAY_API_VERSION`, used by the SSO exposure variant |
| oauth2-proxy | `quay.io/oauth2-proxy/oauth2-proxy:v7.15.5` | `OAUTH2_PROXY_IMAGE` |
| Main image | `ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox` | `GHCR_IMAGE` |
| Gateway image | `ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway` | `GATEWAY_IMAGE`, nested below `GHCR_IMAGE` |
Example component images are always nested below the main package:
`<GHCR_IMAGE>/<example>/<component>`. `check-versions.sh` rejects an unrelated
top-level package.
## Bumping versions
```bash
make set-version DSH_VERSION=dsh-vX.Y.Z RELEASE_REVISION=rN
make check
```
`set-version` updates [versions.env](../versions.env) through
[scripts/set-version.sh](../scripts/set-version.sh) and leaves every consuming
file to derive from it. Never edit the Dockerfile, manifests, Makefile or
workflows with a literal version.
- New upstream DSH ref: change `DSH_VERSION` and clear `RELEASE_REVISION`.
- Repository-only change at the same DSH ref: keep `DSH_VERSION`, set
`RELEASE_REVISION=r(N+1)`.
## Release checklist
1. `main` is green and `make check`, `make test`, `make lint` pass locally.
2. `make image` and `make gateway` build; smoke-test the images.
3. Bump the pins with `make set-version` (commit that change through a PR).
4. `make release-dry-run` — verifies the identity, prints the image names, checks
the tag does not already exist, and previews the generated notes.
5. Create and push the annotated `dsh-*` release tag (or dispatch
`release.yaml` with an existing tag). The workflow validates that the tag
matches [versions.env](../versions.env).
6. The workflow builds and pushes multi-arch images for the main image and the
example components (the identity tag plus `latest`), creates the GitHub
Release from the git-cliff notes, records the published digests, and opens a
`chore(changelog)` PR that refreshes `CHANGELOG.md` on `main`.
7. Merge the changelog PR. Verify:
`docker buildx imagetools inspect <image>:<identity>` shows both platforms,
and the GitHub Release references the tag and the digest.
8. Announce the identity and the digest; deployments pin the digest.
## Rollback
Tags and release identities are **immutable**, so a bad release is never re-tagged
or deleted. Fix forward:
1. Fix the problem on a `hotfix/<slug>` branch (see [gitflow](gitflow.md)).
2. Publish a new identity with the next revision, `-r(N+1)`, at the same
`DSH_VERSION`.
3. Point deployments at the new digest; keep the broken tag for audit history.
Emergency rollback to the previous release is also a re-deploy of a **previous
digest**, not a move of `latest`.
## Changelog
`CHANGELOG.md` is generated by [git-cliff](https://git-cliff.org) from
Conventional Commits, configured in [cliff.toml](../cliff.toml):
```bash
make changelog # regenerate CHANGELOG.md locally
```
- Do not edit `CHANGELOG.md` by hand; the release workflow regenerates it and opens a PR to refresh it on `main`.
- Commit messages and PR titles must be Conventional Commits so they are grouped
correctly (Added / Fixed / Changed / Documentation / Build / CI / Tests /
Maintenance).
- The changelog groups commits since the previous `dsh-*` tag; release notes use
the same rendering.
- The changelog is excluded from the version-token check because it records
historical identities on purpose.
## Digest pinning
Release identities are mutable only in theory; deployments should still pin the
resolved digest for reproducibility and supply-chain safety:
```bash
docker buildx imagetools inspect ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.5
# then, in the deployment:
# image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox@sha256:<digest>
```
The example manifests use the identity tag so they stay readable and upgrade with
`versions.env`; production overlays should pin the digest. Release images carry
SBOM and provenance attestations (see [SECURITY.md](../SECURITY.md)).

55
examples/README.md Normal file
View File

@@ -0,0 +1,55 @@
# Examples
Every implementation in this repository is a self-contained example: one
directory, plain `kubectl`, an end-to-end README, and a cleanup command.
Examples consume the [main DSH Web image](../docs/main-image.md); they are
reference material, not libraries.
| Example | What it demonstrates | Needs |
| --- | --- | --- |
| [`examples/dsh-agents-as-a-service/`](dsh-agents-as-a-service/) | The full multi-tenant platform pattern: one `SandboxTemplate` with per-user PVCs, a warm pool, one ~10-line `SandboxClaim` per user, suspend-when-idle / wake-on-return, and a browser gateway with signup, `HttpOnly` sessions, an HTTP + WebSocket proxy and two exposure variants (Ingress, or Gateway API + GitHub SSO). | agent-sandbox with extensions; kind for the e2e script. |
| [`examples/minimal-single-tenant/`](minimal-single-tenant/) | The smallest useful setup: a single `SandboxTemplate` and one `SandboxClaim`, no gateway — reach DSH Web with a launch-token URL through `kubectl port-forward`. | agent-sandbox with extensions. |
The multi-tenant example also documents opt-in gVisor, `nodeSelector`,
NetworkPolicy and extra provider-key hardening in
[optional hardening](dsh-agents-as-a-service/docs/optional-hardening.md).
## The example contract (short version)
Full rules: [docs/examples.md](../docs/examples.md). `scripts/check-examples.sh`
enforces the layout, and `make check` runs it.
```
examples/<name>/
+-- README.md # walkthrough: prerequisites, steps, troubleshooting, cleanup
+-- manifests/ # NN-*.yaml, applied with plain kubectl
| +-- optional/<feature>/ # optional opt-in manifests...
| +-- optional/README.md # ...required whenever optional/ exists
| +-- exposure/<variant>/ # optional per-variant manifests...
| +-- exposure/README.md # ...required whenever exposure/ exists
+-- scripts/ # optional executable, idempotent, self-cleaning *.sh
+-- docs/ # optional architecture/ADR material
+-- <component>/ # optional component with Dockerfile + README + tests
```
- Secrets never ship populated: use `*.yaml.example` files with placeholders.
- Manifests are numbered (`00-`, `10-`, ...) and read as an ordered walkthrough.
- Component images are published nested below the main image:
`<registry>/<owner>/<repo>/<example>/<component>`.
- Every example must be listed above with its literal `examples/<name>/` path.
- Scripts under `scripts/` must be executable, idempotent and clean up after
themselves.
## Running an example
```bash
# e.g. the multi-tenant platform example
kubectl apply -f examples/dsh-agents-as-a-service/manifests/
# or run the whole walkthrough (kind + agent-sandbox required) and let it clean up
examples/dsh-agents-as-a-service/scripts/e2e-kind.sh
```
`kubectl apply -f manifests/` is non-recursive: `optional/` and `exposure/`
subdirectories are applied explicitly. Each example README has the exact
prerequisites, optional hardening, exposure options and teardown.

View File

@@ -1,27 +0,0 @@
# DSH "Agents as a Service" — prerequisites: namespace + shared provider key.
#
# The dsh-provider-keys Secret is deliberately NOT committed with a usable
# value. DEEPSEEK_API_KEY reaches every tenant sandbox (injected via envFrom
# on the SandboxTemplate, stamped before any user exists — a warm-pool
# constraint), so a real key committed here would be shared by every install
# of this example. Generate a fresh value per install (README step 1;
# run-test-kind.sh uses a throwaway):
#
# kubectl -n dsh-demo create secret generic dsh-provider-keys \
# --from-literal=DEEPSEEK_API_KEY="sk-..."
#
# All the infrastructure in this example works without a real key; only
# actual model calls need one.
apiVersion: v1
kind: Namespace
metadata:
name: dsh-demo
---
apiVersion: v1
kind: Secret
metadata:
name: dsh-provider-keys
namespace: dsh-demo
type: Opaque
# stringData:
# DEEPSEEK_API_KEY: "sk-your-key-here"

View File

@@ -2,20 +2,45 @@
This example distills the upstream
[hermes-agents-as-a-service](https://github.com/kubernetes-sigs/agent-sandbox/tree/main/examples/hermes-agents-as-a-service)
platform pattern into pure agent-sandbox resources + kubectl, running **DeepSeek Harness Web
(DSH)** instead of Hermes. Every user of such a platform gets a personal, long-lived DSH Web
instance that is **provisioned in ~2 seconds from a warm pool, suspended when idle to save
cost, and woken transparently when the user returns** — with all state (conversations,
sessions, config) surviving on a per-user PVC at `/home/node/.dsh`.
platform pattern into pure agent-sandbox resources + kubectl, running **DeepSeek
Harness Web (DSH)** instead of Hermes. Every user of such a platform gets a
personal, long-lived DSH Web instance that is **provisioned in ~2 seconds from a
warm pool, suspended when idle to save cost, and woken transparently when the
user returns** — with all state (conversations, sessions, config) surviving on a
per-user PVC at `/home/node/.dsh`.
It ships in two layers:
It ships in three layers:
1. **The resource layer** (steps 1–7): pure manifests + kubectl, so every mechanism is visible.
2. **The service layer** (step 8): a small included **gateway** (`gateway/`) that makes the
platform browser-usable — a signup page/API, `HttpOnly` cookie sessions, a `/u/{user}/**`
proxy for HTTP **and WebSocket** that transparently **wakes suspended instances on
connect**, and an idle sweeper that suspends them again. The gateway performs the DSH
launch-token handshake server-side, so the browser never sees the token.
1. **The resource layer** (steps 1–7): pure manifests + kubectl, so every
mechanism is visible.
2. **The service layer** (step 8): a small included **gateway** (`gateway/`) that
makes the platform browser-usable — a signup page/API, `HttpOnly` cookie
sessions, a `/u/{user}/**` proxy for HTTP **and WebSocket** that transparently
**wakes suspended instances on connect**, and an idle sweeper that suspends
them again. The gateway performs the DSH launch-token handshake server-side,
so the browser never sees the token.
3. **The exposure layer** (step 9): choose exactly one way in — a plain Traefik
Ingress, or Gateway API + oauth2-proxy (GitHub SSO).
All manifests live under [`manifests/`](manifests/):
```
manifests/
+-- 00-prereqs.yaml # namespace + empty provider-key Secret
+-- 10-sandboxtemplate.yaml # the DSH Web blueprint
+-- 20-sandboxwarmpool.yaml # 2 pre-warmed spares
+-- 30-claim-alice.yaml # one user
+-- 40-claim-rejected.yaml # negative demo: env injection rejected
+-- 50-gateway.yaml # Deployment/RBAC/Service for the gateway
+-- optional/
| +-- README.md # opt-in manifests, applied by path
| +-- network-policy/10-sandbox-networkpolicy.yaml
+-- exposure/
+-- README.md # the two exposure variants, when to use which
+-- ingress/60-ingress.yaml
+-- ingress/61-networkpolicy.yaml
+-- gateway-api-sso/70-gateway.yaml ... 75-networkpolicies.yaml
```
What it demonstrates, feature by feature:
@@ -34,20 +59,23 @@ What it demonstrates, feature by feature:
- **Multi-arch**: the DSH image is built for amd64 and arm64, so there is no
`kubernetes.io/arch: amd64` nodeSelector.
- **Single port**: DSH Web serves everything on `3080`; the gateway proxies to one upstream
instead of splitting `/v1/*` (API) from the dashboard.
- **One shared secret**: only `DEEPSEEK_API_KEY` is injected (via `envFrom`); there is no
dashboard basic-auth, session-cookie secret, or API-server key.
- **Launch-token handshake**: DSH gates `/` behind a token it prints only in its logs. The
gateway reads it, performs the handshake, and keeps the resulting auth cookie server-side;
the browser authenticates to the gateway with its own cookie. See
[How the DSH handshake works](#how-the-dsh-handshake-works).
- **Host allow-list**: DSH rejects unknown `Host` headers. The gateway forwards the browser's
`Host`, so the public hostname must be added to `DSH_TRUSTED_HOSTS` in the template.
- **Probes target `/favicon.svg`**: `/` returns `401` without the launch token, so a probe on
`/` crash-loops the pod. `/favicon.svg` is unauthenticated, and a `startupProbe` gates
liveness/readiness so a slow cold start (plugin activation, or a resumed sandbox) is not
restarted mid-boot.
- **Single port**: DSH Web serves everything on `3080`; the gateway proxies to one
upstream instead of splitting `/v1/*` (API) from the dashboard.
- **Shared provider keys**: `DEEPSEEK_API_KEY` plus any keys in the optional
`dsh-provider-keys-extra` Secret are injected via `envFrom` into every
sandbox. There is no per-tenant key, dashboard basic-auth, session-cookie
secret, or API-server key.
- **Launch-token handshake**: DSH gates `/` behind a token it prints only in its
logs. The gateway reads it, performs the handshake, and keeps the resulting
auth cookie server-side; the browser authenticates to the gateway with its own
cookie. See [How the DSH handshake works](#how-the-dsh-handshake-works).
- **Host allow-list**: DSH rejects unknown `Host` headers. The gateway forwards
the browser's `Host`, so the public hostname must be added to
`DSH_TRUSTED_HOSTS` in the template.
- **Probes target `/favicon.svg`**: `/` returns `401` without the launch token, so
a probe on `/` crash-loops the pod. `/favicon.svg` is unauthenticated, and a
`startupProbe` gates liveness/readiness so a slow cold start (plugin
activation, or a resumed sandbox) is not restarted mid-boot.
## Prerequisites
@@ -55,43 +83,74 @@ What it demonstrates, feature by feature:
- agent-sandbox with extensions installed:
```bash
VERSION=v1.0.7
kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/download/${VERSION}/sandbox-with-extensions.yaml
VERSION=v1.0.8 # AGENT_SANDBOX_VERSION in versions.env
kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/download/$VERSION/sandbox-with-extensions.yaml
kubectl -n agent-sandbox-system rollout status deployment --timeout=180s
```
Or run everything below in one shot with `./run-test-kind.sh` (expects a kind cluster + the
manifest above already applied; it creates and deletes the namespace itself).
For step 9's SSO variant you additionally need the Gateway API CRDs and a
Gateway implementation (`gatewayClassName: traefik` in the manifests); see the
[exposure README](manifests/exposure/README.md).
- Optional: run the whole resource + gateway walkthrough in one shot with
[`scripts/e2e-kind.sh`](scripts/e2e-kind.sh). It expects a kind cluster with
agent-sandbox installed, then creates and deletes the `dsh-demo` namespace
itself. It stops after the gateway layer (step 8); exposure is not applied.
## 1. Install the platform resources
First the namespace, then the shared provider key. The key reaches **every** tenant sandbox
(a warm-pool constraint: pod env is stamped before any user exists), so it is **never
committed** — generate a fresh value per install:
First the namespace, then the shared provider keys. The keys reach **every**
tenant sandbox (a warm-pool constraint: pod env is stamped before any user
exists), so they are **never committed** — generate fresh values per install.
`00-prereqs.yaml` already creates the empty `dsh-provider-keys` Secret, so use
an upsert:
```bash
kubectl apply -f 00-prereqs.yaml
kubectl apply -f manifests/00-prereqs.yaml
kubectl -n dsh-demo create secret generic dsh-provider-keys \
--from-literal=DEEPSEEK_API_KEY="sk-..."
kubectl apply -f 10-sandboxtemplate.yaml -f 20-sandboxwarmpool.yaml
--from-literal=DEEPSEEK_API_KEY="sk-..." \
--dry-run=client -o yaml | kubectl apply -f -
```
Watch the two warm spares boot (first time pulls the image — subsequent operations never pay
that again; that is the point of the pool):
Optional: additional providers (OpenRouter, Gemini, NVIDIA, OpenAI, Anthropic,
...) go in a second Secret. It is referenced with `optional: true`, so the
sandboxes start without it, and every key becomes an env var:
```bash
kubectl -n dsh-demo create secret generic dsh-provider-keys-extra \
--from-literal=OPENROUTER_API_KEY="sk-or-..." \
--from-literal=GEMINI_API_KEY="..." \
--from-literal=NVIDIA_API_KEY="nvapi-..." \
--dry-run=client -o yaml | kubectl apply -f -
```
Create both Secrets **before** the template/warm pool. A later Secret change
does not update running pods: delete the warm-pool spares and recreate claimed
pods to pick up new keys. Keys are shared by all tenants; a key saved in DSH's
credential store wins over env. See
[optional hardening](docs/optional-hardening.md) for the full env-name list and
the refresh procedure.
```bash
kubectl apply -f manifests/10-sandboxtemplate.yaml -f manifests/20-sandboxwarmpool.yaml
```
Watch the two warm spares boot (first time pulls the image — subsequent operations
never pay that again; that is the point of the pool):
```bash
kubectl -n dsh-demo get sandboxes -l agents.x-k8s.io/warm-pool-sandbox -w
```
Wait until both show `READY: True`. Each spare already has its PVC bound and DSH Web serving
on port `3080`.
Wait until both show `READY: True`. Each spare already has its PVC bound and DSH
Web serving on port `3080`.
## 2. "Signup": claim an instance (~2s)
A user signing up is one small claim — `warmPoolRef` plus a pod label:
```bash
kubectl apply -f 30-claim-alice.yaml
kubectl apply -f manifests/30-claim-alice.yaml
kubectl -n dsh-demo wait sandboxclaim dsh-alice --for=condition=Ready --timeout=60s
```
@@ -105,20 +164,21 @@ kubectl -n dsh-demo get sandboxes --show-labels
Three things worth noticing:
- **The sandbox keeps its pool-generated name** (`dsh-pool-…`), not the claim's name —
adoption re-labels an existing warm sandbox rather than creating one. A platform must always
resolve user → sandbox through `claim.status.sandbox.name`.
- The `agents.x-k8s.io/warm-pool-sandbox` label is **gone** from the adopted sandbox (it left
the pool's inventory), and the pool has already started replenishing back to 2 spares.
- **The sandbox keeps its pool-generated name** (`dsh-pool-…`), not the claim's
name — adoption re-labels an existing warm sandbox rather than creating one. A
platform must always resolve user → sandbox through `claim.status.sandbox.name`.
- The `agents.x-k8s.io/warm-pool-sandbox` label is **gone** from the adopted
sandbox (it left the pool's inventory), and the pool has already started
replenishing back to 2 spares.
- The pod carries the claim's `additionalPodMetadata` label
(`sandbox.users.io/dsh-user=alice`), and
`kubectl -n dsh-demo get sandbox $SB -o jsonpath='{.status.serviceFQDN}'` shows the stable
DNS name the gateway proxies to.
`kubectl -n dsh-demo get sandbox $SB -o jsonpath='{.status.serviceFQDN}'` shows
the stable DNS name the gateway proxies to.
## 3. Talk to the instance
DSH Web needs its launch token on first contact — a bare `/` returns `401`. The token is in the
pod logs:
DSH Web needs its launch token on first contact — a bare `/` returns `401`. The
token is in the pod logs:
```bash
kubectl -n dsh-demo logs "$SB" | grep -m1 '?token='
@@ -127,16 +187,16 @@ kubectl -n dsh-demo port-forward "pod/$SB" 18081:3080
# open http://localhost:18081/?token=... (from the log line)
```
> **Local port note:** DSH Web listens on `3080` **in-cluster**. If `3080` is already in use
> on your workstation (e.g. by a local DSH), forward to a different local port (here `18081`),
> never `3080`. `localhost` is always in DSH's trusted-host list, so the forwarded request is
> accepted.
> **Local port note:** DSH Web listens on `3080` **in-cluster**. If `3080` is
> already in use on your workstation (e.g. by a local DSH), forward to a
> different local port (here `18081`), never `3080`. `localhost` is always in
> DSH's trusted-host list, so the forwarded request is accepted.
> **Why the pod stays up:** this claim was applied by hand, so it carries no gateway ownership
> annotation (`aaas.example.com/managed-by`). The gateway's idle sweeper only suspends claims it
> created through `POST /users`; the resource-layer claims from these manifests are never
> touched. You can deploy the gateway (step 8) and still keep a sandbox you are port-forwarding
> to.
> **Why the pod stays up:** this claim was applied by hand, so it carries no
> gateway ownership annotation (`aaas.example.com/managed-by`). The gateway's idle
> sweeper only suspends claims it created through `POST /users`; the resource-layer
> claims from these manifests are never touched. You can deploy the gateway
> (step 8) and still keep a sandbox you are port-forwarding to.
## 4. Write state on the PVC
@@ -161,20 +221,20 @@ kubectl -n dsh-demo wait sandbox "$SB" --for=condition=Ready --timeout=180s
kubectl -n dsh-demo exec "$SB" -- cat /home/node/.dsh/marker.txt # "remember me"
```
> **Why resume works:** the pod template sets `fsGroupChangePolicy: OnRootMismatch`. DSH requires
> its `.credentials.yaml` to be `0600`, while the kubelet's default fsGroup policy widens every
> file to `0660` on each pod start — a resumed sandbox would crash-loop. See
> [Troubleshooting](#troubleshooting) for the one-time fix on sandboxes created before this
> setting existed.
> **Why resume works:** the pod template sets `fsGroupChangePolicy: OnRootMismatch`.
> DSH requires its `.credentials.yaml` to be `0600`, while the kubelet's default
> fsGroup policy widens every file to `0660` on each pod start — a resumed sandbox
> would crash-loop. See [Troubleshooting](#troubleshooting) for the one-time fix
> on sandboxes created before this setting existed.
## 7. Injection policy rejects env-carrying claims
```bash
kubectl apply -f 40-claim-rejected.yaml
kubectl apply -f manifests/40-claim-rejected.yaml
kubectl -n dsh-demo get sandboxclaim dsh-mallory \
-o jsonpath='{.status.conditions[?(@.type=="Ready")].reason}'
# EnvVarsInjectionRejected — the claim never bound a sandbox
kubectl delete -f 40-claim-rejected.yaml
kubectl delete -f manifests/40-claim-rejected.yaml
```
## 8. The gateway: signup, browser proxy, idle-suspend, wake-on-connect
@@ -182,59 +242,72 @@ kubectl delete -f 40-claim-rejected.yaml
### Deploy it
```bash
# Published image (on kind, load it into the cluster; on a real cluster it is
# pulled from GHCR):
docker pull ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway
kind load docker-image ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway --name <kind-cluster>
# Published image (on kind, load it into the cluster; on a real cluster the
# kubelet pulls it from GHCR):
GW=ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway:dsh-v0.2.1-rc.5
docker pull "$GW"
kind load docker-image "$GW" --name <kind-cluster>
# ... or build it locally and tag it exactly as 50-gateway.yaml expects:
docker build -t ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway gateway/
kind load docker-image ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway --name <kind-cluster>
# ... or build it locally and tag it exactly as manifests/50-gateway.yaml
# expects (make gateway builds dsh-aaas-gateway:dev):
make gateway
docker tag dsh-aaas-gateway:dev "$GW"
kind load docker-image "$GW" --name <kind-cluster>
kubectl apply -f 50-gateway.yaml
kubectl apply -f manifests/50-gateway.yaml
kubectl -n dsh-demo rollout status deploy/dsh-gateway --timeout=120s
kubectl -n dsh-demo port-forward svc/dsh-gateway 18080:8080
```
> The gateway binds `8080`; the local port-forward uses `18080` and never `3080`.
>
> **Re-deploying after a new image?** The workflow publishes the mutable `:gateway` tag and the
> Deployment uses `imagePullPolicy: IfNotPresent`, so a node that already cached the tag keeps
> the old image across `kubectl apply`/`rollout restart`. Pin the digest for a reproducible
> rollout (the workflow's `Inspect image` step and `docker buildx imagetools inspect` both show
> it):
> **Image pull policy.** `manifests/50-gateway.yaml` uses
> `imagePullPolicy: IfNotPresent` with the release identity, so a locally built
> image must be loaded into the cluster under exactly that name (kind:
> `kind load`). A node that already cached a different tag is the usual cause of
> "my new build is not running"; pin the digest for a reproducible rollout:
>
> ```bash
> kubectl -n dsh-demo set image deploy/dsh-gateway \
> gateway=ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox@sha256:<digest>
> gateway=ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway@sha256:<digest>
> kubectl -n dsh-demo rollout status deploy/dsh-gateway --timeout=120s
> ```
>
> **Applying over an existing install?** Run `kubectl diff -f 10-sandboxtemplate.yaml`
> first. Cluster-specific fields such as `nodeAffinity` and the warm-pool replica count are
> intentionally not in the repo manifests; preserve them with a targeted patch instead of a
> **Applying over an existing install?** Run
> `kubectl diff -f manifests/10-sandboxtemplate.yaml` first. Cluster-specific
> fields such as `nodeAffinity` and the warm-pool replica count are intentionally
> not in the repo manifests; preserve them with a targeted patch instead of a
> blind apply. Moving the probes to `/favicon.svg` is the crash-loop fix.
>
> The template also sets `fsGroupChangePolicy: OnRootMismatch` (see the note in step 6).
> Existing `Sandbox` objects snapshot their pod template, so a template apply does not update
> them: patch each sandbox's `spec.podTemplate.spec.securityContext.fsGroupChangePolicy` and
> The template also sets `fsGroupChangePolicy: OnRootMismatch` (see the note in
> step 6). Existing `Sandbox` objects snapshot their pod template, so a template
> apply does not update them: patch each sandbox's
> `spec.podTemplate.spec.securityContext.fsGroupChangePolicy` and
> `chmod 600` its `.credentials.yaml`, or recreate it.
>
> The optional gVisor/`nodeSelector` fields are pod-spec changes too: existing
> Sandboxes snapshot them, and warm-pool spares are not replaced by a template
> edit. See [optional hardening](docs/optional-hardening.md) for the patch and
> recreation commands. A provider-key Secret change likewise needs pod
> recreation to take effect.
### Browser flow
Open <http://localhost:18080>. The landing page has a one-field signup form. Enter `bob`, and
the gateway:
Open <http://localhost:18080>. The landing page has a one-field signup form.
Enter `bob`, and the gateway:
1. creates the `SandboxClaim` over the warm pool,
2. waits for it to become `Ready`,
3. sets an `HttpOnly`, `SameSite=Lax` cookie `dsh_gw_session=<user>:<token>` on the browser,
3. sets an `HttpOnly`, `SameSite=Lax` cookie `dsh_gw_session=<user>:<token>` on
the browser,
4. redirects to `/u/bob/`, where the DSH Web UI loads through the gateway.
From then on every request under `/u/bob/` (assets, API calls, SSE, WebSocket) is proxied to
bob's sandbox. The DSH launch token and DSH's own auth cookie stay in the gateway; the browser
only holds the gateway cookie.
From then on every request under `/u/bob/` (assets, API calls, SSE, WebSocket) is
proxied to bob's sandbox. The DSH launch token and DSH's own auth cookie stay in
the gateway; the browser only holds the gateway cookie.
The same flow with `curl`, no JavaScript (the session cookie is the only credential):
The same flow with `curl`, no JavaScript (the session cookie is the only
credential):
```bash
JAR=$(mktemp)
@@ -252,48 +325,53 @@ curl -s -b "$JAR" -H 'Origin: http://localhost:18080' -X DELETE http://localhost
```
API clients that prefer bearer tokens can use `POST /users` with a JSON body
(`{"user":"bob"}`) and the returned one-time token, `POST /login` to exchange a token for a
cookie, and `Authorization: Bearer <token>` on subsequent calls.
(`{"user":"bob"}`) and the returned one-time token, `POST /login` to exchange a
token for a cookie, and `Authorization: Bearer <token>` on subsequent calls.
### Gateway API
| Route | Purpose |
| --- | --- |
| `GET /healthz` | Liveness/readiness; no cluster access |
| `GET /` | Landing page: signup, token login, logout |
| `GET /` | Landing page: signup, token login, logout. With SSO it auto-provisions and connects instead. |
| `POST /users` | Signup. JSON or form; `201` (JSON), `202` (waking), `303` (browser redirect), `409` if user exists; sets the session cookie |
| `POST /login` | Exchange a bearer token for the session cookie |
| `POST /logout` | Clear the session cookie |
| `POST /recreate` | SSO only: delete the authenticated user's claim (wiping workspace data) and provision a fresh one; `303` to `/u/{user}/`. Cookie-authenticated calls must be same-origin. |
| `GET /users/{user}` | Derived state (`Ready`/`Waking`/`Suspended`) |
| `DELETE /users/{user}` | Delete claim (cascades sandbox + PVC) |
| `GET /u/{user}` | `308` redirect to `/u/{user}/` |
| `GET /u/{user}/**` | HTTP/SSE proxy to the user's DSH Web |
| `WS /u/{user}/**` | WebSocket proxy to the user's DSH Web |
The full API reference, configuration table, auth semantics and failure matrix
live in the [gateway README](gateway/README.md).
### How the DSH handshake works
DSH does not expose an API for its launch token; it prints a URL containing `?token=…` in the
pod logs. For each `(user, pod UID, Host)` the gateway:
DSH does not expose an API for its launch token; it prints a URL containing
`?token=…` in the pod logs. For each `(user, pod UID, Host)` the gateway:
1. resolves the claim → sandbox → pod (annotation `agents.x-k8s.io/pod-name`, or the sandbox
name as pod name),
2. reads the pod's logs (last `DSH_LOG_TAIL_LINES` lines) and parses the most recent
`dsh web: …?token=…` line,
1. resolves the claim → sandbox → pod (annotation `agents.x-k8s.io/pod-name`, or
the sandbox name as pod name),
2. reads the pod's logs (last `DSH_LOG_TAIL_LINES` lines) and parses the most
recent `dsh web: …?token=…` line,
3. requests `http://<serviceFQDN>:3080/?token=…` with the browser's `Host`,
4. captures DSH's `dsh-auth-*` `Set-Cookie` response and caches it,
5. injects that cookie (and strips the browser's `Cookie`/`Authorization` and any `?token=`)
on every proxied HTTP and WebSocket request,
6. refreshes the handshake once if DSH answers `401`, and drops the cached cookie when the
sandbox suspends or is deleted.
5. injects that cookie (and strips the browser's `Cookie`/`Authorization` and any
`?token=`) on every proxied HTTP and WebSocket request,
6. refreshes the handshake once if DSH answers `401`, and drops the cached cookie
when the sandbox suspends or is deleted.
This is why the gateway's RBAC includes `pods` `get`/`list` and `pods/log` `get`. It is also
the most fragile part of the design; see [Limitations](#limitations).
This is why the gateway's RBAC includes `pods` `get`/`list` and `pods/log`
`get`. It is also the most fragile part of the design; see
[Limitations](#limitations).
### DSH_TRUSTED_HOSTS
DSH rejects requests whose `Host` is not allow-listed (`localhost` and `127.0.0.1` are always
trusted). Because the gateway forwards the browser's `Host`, set the public hostname in the
template:
DSH rejects requests whose `Host` is not allow-listed (`localhost` and
`127.0.0.1` are always trusted). Because the gateway forwards the browser's
`Host`, set the public hostname in the template:
```yaml
env:
@@ -301,36 +379,187 @@ env:
value: "dsh.example.com" # comma or space separated; empty for port-forward only
```
The same applies to a local `docker run`: the printed token URL uses `DSH_PUBLIC_URL` if set,
otherwise `http://localhost:${DSH_PORT}`.
The same applies to a local `docker run`: the printed token URL uses
`DSH_PUBLIC_URL` if set, otherwise `http://localhost:<DSH_PORT>`.
### WebSocket
The gateway is a Starlette ASGI app and proxies WebSocket upgrades with a bidirectional pump, so
the DSH web GUI is fully interactive. Requirements:
The gateway is a Starlette ASGI app and proxies WebSocket upgrades with a
bidirectional pump, so the DSH web GUI is fully interactive. Requirements:
- the ingress/load balancer in front of the gateway must pass `Upgrade`/`Connection` headers;
`60-ingress.yaml` is a starting point (Traefik handles upgrades by default),
- idle suspension is disabled for users with in-flight proxied requests or open WebSockets, so
a long-running chat stream is not cut off mid-turn,
- the upstream upgrade dials the sandbox **pod IP**, forwards no `Host`/`Origin`, and performs
the launch-token handshake under that same `pod-IP:3080` authority. DSH fences on `Host`
(only its bind address, loopback, or an allow-listed host passes) and binds its auth cookie to
the request authority (`payload.authority === authority`), while `websockets` derives `Host`
from the URI and cannot be overridden. A browser-host cookie or forwarded origin makes the
upgrade fail (`401`/`403`, surfaced as close `1011`), which the client retries forever as
`[connection] connection lost, retry #N`,
- missing/invalid gateway auth is rejected right after the upgrade with close code `4401`
(WebSocket auth has no HTTP `401` path); cross-origin or Origin-less upgrades with the cookie
are rejected the same way (CSWSH protection), so clients must send a matching `Origin`;
upstream bridging failures close with `1011`.
- the ingress/load balancer in front of the gateway must pass `Upgrade`/`Connection`
headers; [`manifests/exposure/ingress/60-ingress.yaml`](manifests/exposure/ingress/60-ingress.yaml)
is a starting point (Traefik handles upgrades by default),
- idle suspension is disabled for users with in-flight proxied requests or open
WebSockets, so a long-running chat stream is not cut off mid-turn,
- the upstream upgrade dials the sandbox **pod IP**, forwards no `Host`/`Origin`,
and performs the launch-token handshake under that same `pod-IP:3080` authority.
DSH fences on `Host` (only its bind address, loopback, or an allow-listed host
passes) and binds its auth cookie to the request authority
(`payload.authority === authority`), while `websockets` derives `Host` from the
URI and cannot be overridden. A browser-host cookie or forwarded origin makes
the upgrade fail (`401`/`403`, surfaced as close `1011`), which the client
retries forever as `[connection] connection lost, retry #N`,
- missing/invalid gateway auth is rejected right after the upgrade with close code
`4401` (WebSocket auth has no HTTP `401` path); cross-origin or Origin-less
upgrades with the cookie are rejected the same way (CSWSH protection), so
clients must send a matching `Origin`; upstream bridging failures close with
`1011`.
### Path routing
Each user's DSH Web is served under `/u/<user>/`. DSH injects `<base href="./">` and uses
relative asset/API URLs, so the UI works under a path prefix without a per-user hostname or
wildcard DNS. The gateway also rewrites absolute-path `Location` headers into `/u/<user>/…` and
redirects `/u/<user>` to the trailing-slash form with `308` so relative URLs resolve correctly.
Each user's DSH Web is served under `/u/<user>/`. DSH injects `<base href="./">`
and uses relative asset/API URLs, so the UI works under a path prefix without a
per-user hostname or wildcard DNS. The gateway also rewrites absolute-path
`Location` headers into `/u/<user>/…` and redirects `/u/<user>` to the
trailing-slash form with `308` so relative URLs resolve correctly.
## 9. Expose it: choose exactly one variant
The base manifests expose nothing outside the cluster (`dsh-gateway` is a
ClusterIP Service on port `8080`). Pick **one** variant — they are alternatives,
not a stack. Details, commands and the switch procedure are in the
[exposure README](manifests/exposure/README.md).
> Apply the base resources first, then the variant **by path**. Do not use
> `kubectl apply -R`: it would apply both variants and create two paths to the
> same gateway, which is unsafe once SSO is enabled.
Both variants also ship NetworkPolicies: `ingress/61-networkpolicy.yaml` limits
gateway ingress to Traefik pods, and `gateway-api-sso/75-networkpolicies.yaml`
limits the gateway to oauth2-proxy and oauth2-proxy egress to DNS, public HTTPS
and the gateway. They are part of the variant; see the
[exposure README](manifests/exposure/README.md) and
[optional hardening](docs/optional-hardening.md).
### Option A — plain Ingress (gateway's own session auth)
```bash
# Replace the dsh.example.com host and TLS Secret (or drop the tls: block for
# plain HTTP), then:
kubectl apply -f manifests/exposure/ingress/
```
1. Set `DSH_TRUSTED_HOSTS` in `manifests/10-sandboxtemplate.yaml` to the same
public host (the gateway forwards the browser's `Host`) and re-apply the
template.
2. Set `COOKIE_SECURE=true` on the gateway once the Ingress serves HTTPS:
`kubectl -n dsh-demo set env deploy/dsh-gateway COOKIE_SECURE=true`.
(The base manifest keeps `false` so the plain-HTTP port-forward demo works.)
3. Verify: `curl -sI https://<host>/healthz` returns `200`; sign up in a browser
at `https://<host>/`.
4. `manifests/exposure/ingress/61-networkpolicy.yaml` is part of this variant:
it allows only Traefik pods to reach the gateway on `8080`. Adjust its
selector if your ingress controller uses a different pod label or
`hostNetwork`; if kubelet probes fail, see the hardening doc.
Keep the router free of buffering/compression middlewares: they break WebSocket
upgrades and streaming.
### Option B — Gateway API + GitHub SSO (oauth2-proxy)
`manifests/exposure/gateway-api-sso/` puts a TLS `Gateway` + `HTTPRoute` in
front of **oauth2-proxy**, which authenticates the browser against GitHub
(restricted to a configured organization) and forwards to the gateway with
`X-Forwarded-User` set. The gateway trusts that header only when
`AUTH_PROXY_HEADER` is set (SSO is **opt-in**).
Prerequisites:
- Gateway API CRDs and a Gateway implementation; the manifests assume
`gatewayClassName: traefik` (the CRD version is pinned as
`GATEWAY_API_VERSION` in `versions.env`).
- A GitHub OAuth App whose homepage/callback match the public FQDN
(`https://<host>/` and `https://<host>/oauth2/callback`).
- The oauth2-proxy credentials Secret (template:
`73-oauth2-proxy-secret.yaml.example`) and a TLS Secret for the Gateway
listener (template: `74-tls-secret.yaml.example`).
Steps:
```bash
# 1. Real Secrets; the committed *.yaml.example files are templates only.
kubectl -n dsh-demo create secret generic oauth2-proxy \
--from-literal=client-id='<GitHub OAuth App client ID>' \
--from-literal=client-secret='<GitHub OAuth App client secret>' \
--from-literal=cookie-secret="$(python3 -c 'import secrets; print(secrets.token_urlsafe(32))')"
kubectl -n dsh-demo create secret tls dsh-demo-tls --cert=tls.crt --key=tls.key
# 2. Replace dsh-demo.example.com (70/71/72) and your-github-org (72).
# 3. Enable SSO on the gateway and trust the public host in the template.
kubectl -n dsh-demo set env deploy/dsh-gateway \
AUTH_PROXY_HEADER=X-Forwarded-User COOKIE_SECURE=true
# ... and set DSH_TRUSTED_HOSTS=dsh-demo.example.com in
# manifests/10-sandboxtemplate.yaml, then re-apply the template.
# 4. Apply the variant.
kubectl apply -f manifests/exposure/gateway-api-sso/
```
Verification:
```bash
kubectl get gateway dsh-gateway -n dsh-demo -o yaml # Programmed=True
kubectl get httproute -n dsh-demo -o wide # backend oauth2-proxy:4180, Accepted
kubectl get deploy oauth2-proxy -n dsh-demo # 1/1 Ready
kubectl get endpoints oauth2-proxy -n dsh-demo # backend present
kubectl -n dsh-demo logs deploy/oauth2-proxy --tail=50 # no config errors
```
Browser check: open `https://<host>/` — you are redirected to GitHub, and after
signing in as a member of the allowed organization the gateway auto-provisions
your sandbox (the first page can show `Provisioning`) and connects you. A
non-member is denied before reaching the gateway. GitHub logins are lowercase
DNS-1123 labels, so they map 1:1 onto the gateway's `user` validation. The SSO
landing page also offers **Recreate sandbox** (`POST /recreate`), which deletes
the claim — wiping the workspace PVC — and provisions a fresh one.
Sign-out: the gateway `/logout` clears `dsh_gw_session`; oauth2-proxy's session
can be cleared at `/oauth2/sign_out`.
> **SSO trust boundary — read this before enabling it.** After SSO is on, the
> `dsh-gateway` Service must be reachable **only** through oauth2-proxy (it is
> ClusterIP). The variant's `75-networkpolicies.yaml` restricts pod traffic
> accordingly: the gateway accepts traffic only from oauth2-proxy pods,
> oauth2-proxy only from the Gateway implementation, and oauth2-proxy egress is
> limited to DNS, public HTTPS and the gateway. Anyone who can reach
> `dsh-gateway:8080` directly can still send `X-Forwarded-User: <victim>` and be
> treated as that user, so never add a second path (the Ingress variant, a
> shared-host port-forward, a permissive policy). To turn SSO off, remove the
> variable (`kubectl -n dsh-demo set env deploy/dsh-gateway AUTH_PROXY_HEADER-`)
> and delete the SSO variant; the gateway falls back to bearer-token/session
> auth.
### Switching variants
Delete the variant you had before applying the other one
(`kubectl delete -f manifests/exposure/ingress/`), revert
`AUTH_PROXY_HEADER`/`COOKIE_SECURE`, and verify with
`kubectl -n dsh-demo get ingress,gateway,httproute` that exactly one path to the
gateway remains.
## Optional hardening
The base walkthrough is deliberately portable. Four opt-in isolation layers are
documented in [optional hardening](docs/optional-hardening.md):
- **gVisor** — uncomment `runtimeClassName: gvisor` before applying, or patch an
existing install. Requires a `gvisor` RuntimeClass and capable nodes.
`kubectl port-forward` does not work under gVisor; the gateway path does.
- **nodeSelector** — uncomment and replace the placeholder node label to pin
sandboxes and the gateway to a dedicated pool; add a toleration for tainted
pools.
- **Sandbox NetworkPolicy** —
`kubectl apply -f manifests/optional/network-policy/` allows only
gateway→sandbox on `3080` and sandbox→DNS/public `443`.
- **Provider keys** — create `dsh-provider-keys-extra` as shown in step 1.
- **Failed-Pod recovery** — uncomment `failedPodPolicy: Recreate` so the
controller replaces a `Failed` sandbox Pod (node eviction) instead of parking
the sandbox at `Finished=True`. Requires agent-sandbox v1.0.8+.
Apply each independently and re-run the walkthrough. `kubectl apply -f manifests/`
is non-recursive, so the optional manifest is not applied by the base step.
## Troubleshooting
@@ -346,36 +575,83 @@ redirects `/u/<user>` to the trailing-slash form with `308` so relative URLs res
| Console spams `[connection] connection lost, retry #N` | The upstream WS upgrade is being rejected, surfaced as close `1011`. The gateway dials the sandbox **pod IP**, forwards no `Host`/`Origin`, and handshakes under the same `pod-IP:3080` authority DSH binds its cookie to; an older gateway image, or a sandbox with an empty `status.podIPs`, hits this loop |
| `manifest.webmanifest` returns `401` in the console | Browsers fetch the Web App Manifest without credentials, so the gateway's session cookie is absent and the request fails closed. The UI is unaffected; only the PWA install prompt is lost |
| Signup is slow / `202` then `Waking` | Warm pool exhausted (`kubectl get sandboxes -l agents.x-k8s.io/warm-pool-sandbox`); scale the pool |
| `403` from the Kubernetes API in gateway logs | The gateway ServiceAccount is missing claim/sandbox/pod-log verbs (see `50-gateway.yaml`) |
| Step 3: `pod "$SB" not found` after deploying the gateway | The claim was suspended by the idle sweeper. Claims created by the gateway carry `aaas.example.com/managed-by=dsh-gateway`; hand-applied claims (steps 1–7) do not and are never swept. Upgrade the gateway to a build that honors the marker, or resume the sandbox: `kubectl -n dsh-demo patch sandbox "$SB" --type merge -p '{"spec":{"operatingMode":"Running"}}'` |
| Resume crash-loops: `credentials-local: … is readable beyond its owner (mode 660)` | The pod template lacks `fsGroupChangePolicy: OnRootMismatch`, so the kubelet's default fsGroup policy rewrote `.credentials.yaml` to `0660`; DSH requires `0600` and refuses to start. New installs get the policy from the template. For an existing sandbox: suspend it, run a throwaway pod on `dsh-home-$SB` to `chmod 600 /home/node/.dsh/.credentials.yaml`, patch the sandbox's `podTemplate` with the policy, then resume. |
| `403` from the Kubernetes API in gateway logs | The gateway ServiceAccount is missing claim/sandbox/pod-log verbs (see `manifests/50-gateway.yaml`) |
| Gateway pod `ErrImagePull` / `ImagePullBackOff` (especially on kind) | The manifest uses `imagePullPolicy: IfNotPresent` with the release identity. Build with `make gateway`, retag to exactly the manifest's image, `kind load docker-image <that-ref>`, or set the Deployment image to your local tag / pinned digest. On a real cluster, make sure the GHCR package is readable (public or a pull secret) |
| Step 3: `pod "$SB" not found` after deploying the gateway | The claim was suspended by the idle sweeper. Claims created by the gateway carry `aaas.example.com/managed-by=dsh-gateway`; hand-applied claims (steps 1–7) do not and are never swept. Resume the sandbox: `kubectl -n dsh-demo patch sandbox "$SB" --type merge -p '{"spec":{"operatingMode":"Running"}}'` |
| Resume crash-loops: `credentials-local: … is readable beyond its owner (mode 660)` | The pod template lacks `fsGroupChangePolicy: OnRootMismatch`, so the kubelet's default fsGroup policy rewrote `.credentials.yaml` to `0660`; DSH requires `0600` and refuses to start. New installs get the policy from the template. For an existing sandbox: suspend it, run a throwaway pod on `dsh-home-$SB` to `chmod 600 /home/node/.dsh/.credentials.yaml`, patch the sandbox's `podTemplate` with the policy, then resume |
| SSO: GitHub login succeeds but the gateway asks for a token / returns `401` to `/recreate` | `AUTH_PROXY_HEADER` is not set on `deploy/dsh-gateway` (SSO is opt-in). Set `AUTH_PROXY_HEADER=X-Forwarded-User` and confirm the request actually reaches the gateway through oauth2-proxy |
| SSO: gateway treats a caller as another user / bad `X-Forwarded-User` accepted | Trust boundary violation: something other than oauth2-proxy can reach `dsh-gateway:8080`. Remove the second path (Ingress/Gateway/port-forward), keep the Service ClusterIP, and only re-enable SSO once the proxy is the sole entry point. Never put a trusted identity header in front of a service that is directly reachable |
| SSO: oauth2-proxy `cookie_secret must be 16, 24, or 32 bytes ... but is 44 bytes` | The cookie secret was generated as standard base64. Use URL-safe unpadded base64 (see `73-oauth2-proxy-secret.yaml.example`) |
| SSO: redirect loop (`ERR_TOO_MANY_REDIRECTS`) | oauth2-proxy `--redirect-url` / callback does not match the public FQDN, or TLS is not terminating at the Gateway listener; fix the OAuth App URLs and the manifest placeholders |
| SSO: DSH rejects the public host behind the proxy | Add the public hostname to `DSH_TRUSTED_HOSTS` in `manifests/10-sandboxtemplate.yaml`; the gateway forwards the browser's `Host` |
| Pods `Pending` after setting `nodeSelector` | No node matches the label, the pool is full, or a tainted pool needs a toleration; inspect `kubectl describe pod` and the node labels |
| gVisor: `kubectl port-forward` fails or the connection is refused | Port-forward is incompatible with gVisor; reach DSH through the gateway (or a reachable Service/Sandbox Router) |
| Sandbox cannot reach a model API after applying the optional NetworkPolicy | The policy allows DNS and public TCP `443` only; a proxy, custom base URL, port `80` or non-443 endpoint needs an explicit egress rule |
| Gateway cannot reach a sandbox after applying an exposure NetworkPolicy | Check that the CNI enforces NetworkPolicy and that the Traefik/oauth2-proxy pod labels match; kubelet probes may also need the node CIDR |
| New provider key not visible in a sandbox | Pod env is stamped at creation: delete the warm-pool spares and recreate claimed pods. A key saved in DSH's credential store also wins over env |
| Sandbox reports `Finished=True` and every gateway request fails | The sandbox Pod reached phase `Failed` (node eviction, spot reclaim) and the default `Retain` keeps it. Delete the Pod so the controller recreates it, or set `failedPodPolicy: Recreate` (agent-sandbox v1.0.8+; see optional hardening) |
## Limitations
Deliberately **not** production code:
- **Single replica.** Idle clocks, in-flight tracking, and the handshake cache are in memory.
Running two gateway replicas would double-suspend or lose wake tracking; scale up only after
moving that state to a shared store.
- **Log scraping.** The launch token is recovered from pod logs. Log rotation, a restarted pod,
or a changed DSH log format can break the handshake until the next `401` refresh.
- **Shared provider key.** `DEEPSEEK_API_KEY` is shared by all tenants via the template. Real
multi-tenancy needs per-tenant credentials or an egress proxy.
- **No TLS in the gateway.** Terminate TLS at an Ingress/reverse proxy and set
- **Single replica.** Idle clocks, in-flight tracking, and the handshake cache
are in memory. Running two gateway replicas would double-suspend or lose wake
tracking; scale up only after moving that state to a shared store.
- **Log scraping.** The launch token is recovered from pod logs. Log rotation, a
restarted pod, or a changed DSH log format can break the handshake until the
next `401` refresh.
- **Shared provider keys.** `DEEPSEEK_API_KEY` and any keys in the optional
`dsh-provider-keys-extra` Secret are shared by all tenants via the template.
Tenants can override a key through DSH's stored credential file on their PVC,
but the platform does not manage that. Real multi-tenancy needs per-tenant
credentials or an egress proxy.
- **No TLS in the gateway.** Terminate TLS at an Ingress or Gateway and set
`COOKIE_SECURE=true`.
- **In-memory user directory.** Users are claims + annotations; there is no database, password
reset, or admin API. Tokens are stored only as SHA-256 hashes in claim annotations.
- **Bearer tokens in URLs.** Avoid putting the signup token in a URL/log; the browser flow uses
the `HttpOnly` cookie instead.
- **SSO trust is hop-based.** `AUTH_PROXY_HEADER` trusts whoever sets the header.
The SSO variant ships `75-networkpolicies.yaml`, which restricts the gateway
to oauth2-proxy and oauth2-proxy egress to the gateway/DNS/public HTTPS; a
custom exposure without that policy has no signature, mTLS or network
enforcement.
- **Isolation is opt-in.** gVisor, the sandbox NetworkPolicy and the exposure
policies are opt-in; without them, sandboxes share the host kernel and can
reach the cluster network. The base example is a demo, not a hardened
multi-tenant platform.
- **OAuth org is the only authorization.** Any GitHub user in the configured
organization gets an auto-provisioned sandbox; there is no per-user allow-list,
quota, or admin API.
- **In-memory user directory.** Users are claims + annotations; there is no
database, password reset, or admin API. Tokens are stored only as SHA-256
hashes in claim annotations.
- **Bearer tokens in URLs.** Avoid putting the signup token in a URL/log; the
browser flow uses the `HttpOnly` cookie instead.
## Cost model
The same economics as the upstream pattern: suspend-when-idle plus warm pools turn an
always-on instance into a sub-second cold-start pause per return-after-absence, at a fraction
of the cost.
The same economics as the upstream pattern: suspend-when-idle plus warm pools turn
an always-on instance into a sub-second cold-start pause per return-after-absence,
at a fraction of the cost.
## Cleanup
```bash
# Delete an exposure variant first if you applied one (optional; namespace
# deletion below removes everything):
kubectl delete -f manifests/exposure/ingress/ # or gateway-api-sso/
# Then the whole example:
kubectl delete namespace dsh-demo
```
If you created the oauth2-proxy/TLS Secrets or the GitHub OAuth App for step 9,
delete the Secrets with the namespace and revoke the OAuth App in GitHub.
## More
- [Gateway README](gateway/README.md) — API, configuration, auth/SSO, failure
semantics, limitations.
- [Architecture](docs/architecture.md) — container/sequence diagrams, ADRs,
security and RBAC, failure modes, evolution path.
- [Exposure README](manifests/exposure/README.md) — the two variants in detail.
- [Optional hardening](docs/optional-hardening.md) — gVisor, nodeSelector,
NetworkPolicy, extra provider keys and failed-Pod recovery, with verification
and rollback.

View File

@@ -1,12 +1,12 @@
# Architecture
This document describes the browser-usable **agents-as-a-service** platform in
[`examples/dsh-agents-as-a-service/`](../examples/dsh-agents-as-a-service/): how the gateway,
This document describes the browser-usable **agents-as-a-service** platform in this example
([README](../README.md), [manifests](../manifests/), [gateway](../gateway/)): how the gateway,
Kubernetes, and DSH Web fit together, and why the design looks the way it does.
Scope: one Kubernetes namespace (`dsh-demo`), one shared DSH Web image, one gateway deployment,
and one DSH Web pod + PVC per user. The cluster is assumed to run
[agent-sandbox](https://github.com/kubernetes-sigs/agent-sandbox) v1.0.7 with the extensions
[agent-sandbox](https://github.com/kubernetes-sigs/agent-sandbox) v1.0.8 with the extensions
(`SandboxTemplate`, `SandboxWarmPool`, `SandboxClaim`).
## Container view
@@ -49,6 +49,31 @@ Responsibilities:
| **DSH Web pod** | One `dsh web` process per user, state on the per-user PVC, listening on `3080`. |
| **Warm pool** | Keeps `N` Ready sandboxes so signup is a label flip instead of a cold start. |
## Exposure variants
The base manifests deploy the gateway as a ClusterIP Service (`dsh-gateway:8080`) and expose
nothing outside the cluster. Two **mutually exclusive** variants live under
[`manifests/exposure/`](../manifests/exposure/):
| Variant | Entry point | Authentication | Notes |
| --- | --- | --- | --- |
| [Ingress](../manifests/exposure/ingress/60-ingress.yaml) | Traefik `Ingress` + optional TLS | the gateway's own signup/session cookie | simplest; no external IdP; set `COOKIE_SECURE=true` with HTTPS |
| [Gateway API + SSO](../manifests/exposure/gateway-api-sso/70-gateway.yaml) | Gateway API `Gateway`/`HTTPRoute` terminating TLS | GitHub OAuth via [oauth2-proxy](../manifests/exposure/gateway-api-sso/72-oauth2-proxy.yaml), identity forwarded as `X-Forwarded-User` | requires `AUTH_PROXY_HEADER` on the gateway; the trust boundary applies |
They are alternatives, not a stack: applying both creates a second path to the same gateway
and defeats SSO. When SSO is enabled, the Service must stay ClusterIP and be reachable only
through oauth2-proxy — otherwise any client can send `X-Forwarded-User` itself and be treated
as that user. The gateway ignores the header unless `AUTH_PROXY_HEADER` is set. See the
[exposure README](../manifests/exposure/README.md) and the
[gateway README](../gateway/README.md#sso-opt-in).
Each variant also ships a NetworkPolicy: `ingress/61-networkpolicy.yaml` limits gateway ingress
to Traefik pods, and `gateway-api-sso/75-networkpolicies.yaml` limits the gateway to
oauth2-proxy, oauth2-proxy ingress to the Gateway implementation, and oauth2-proxy egress to
DNS/public HTTPS/the gateway. The SSO policy turns the trust boundary into pod-traffic
enforcement; the caveats (Traefik label, `hostNetwork`, kubelet probes) are in
[optional hardening](optional-hardening.md).
## Flows
### Signup (browser)
@@ -247,7 +272,7 @@ sequenceDiagram
## Security and RBAC
The gateway runs with a namespace-scoped ServiceAccount (`50-gateway.yaml`):
The gateway runs with a namespace-scoped ServiceAccount (`manifests/50-gateway.yaml`):
| API group | Resource | Verbs | Why |
| --- | --- | --- | --- |
@@ -272,8 +297,15 @@ Notes and limits:
- **TLS is external.** The gateway serves plain HTTP; terminate TLS at the Ingress and set
`COOKIE_SECURE=true`. Without TLS, session cookies are replayable on the network.
- **Tenant isolation** is pod + PVC separation; all tenants share one namespace, one DSH binary,
and one `DEEPSEEK_API_KEY`. Add NetworkPolicies and per-tenant credentials before calling it
multi-tenant-grade.
and the shared provider keys (`dsh-provider-keys` plus the optional
`dsh-provider-keys-extra`). `manifests/optional/network-policy/` adds an opt-in sandbox
NetworkPolicy, and gVisor/`nodeSelector` are opt-in PodSpec fields. Add per-tenant
credentials before calling it multi-tenant-grade.
- **Optional hardening is opt-in.** gVisor, node selection, the sandbox/exposure NetworkPolicies,
extra provider keys and failed-Pod recovery are documented in
[optional hardening](optional-hardening.md). The base manifests stay portable (no gVisor
RuntimeClass, no active node labels, no NetworkPolicy, no extra Secret required, and the
controller's default `failedPodPolicy: Retain`).
- **No gateway rate limiting.** Signup can create claims until the cluster or warm pool is
exhausted; put limits at the ingress for real deployments.
@@ -288,6 +320,11 @@ Notes and limits:
| DSH auth cookie expired mid-session | Proxy `401` once, then recovers | Gateway refreshes the handshake once |
| Upstream pod name changed after resume | Cached cookie keyed by old pod UID is useless | Cache is invalidated on suspend; next request re-handshakes |
| Two gateway replicas | Double suspend/wake races | Keep `replicas: 1` until state is externalized |
| Optional NetworkPolicy blocks DNS/probes | Sandbox 502/503, pod restarts, failed `getent` | Check the CNI, DNS selector/NodeLocal address and kubelet-probe traffic; see the hardening doc |
| gVisor + `kubectl port-forward` | Forward refuses or returns nothing | Port-forward is incompatible with gVisor; use the gateway or a reachable Service |
| Provider key added but missing in the pod | Requests still fail with `MISSING_CREDENTIAL` | Pod env is stamped at creation; recreate spares/pods. A stored credential wins over env |
| `nodeSelector` matches no node | Pod `Pending`, `0/N nodes are available` | Fix the label, pool capacity or taint/toleration; RuntimeClass scheduling may also constrain it |
| Sandbox Pod reaches phase `Failed` (node eviction, spot reclaim) | Sandbox reports `Finished=True`; the gateway keeps failing against a dead Pod | Set `failedPodPolicy: Recreate` (agent-sandbox v1.0.8+) so the controller replaces the Pod with the same identity and PVCs; otherwise delete the Pod by hand. See the hardening doc |
## Upstream feature request: remove log scraping

View File

@@ -0,0 +1,640 @@
# Optional hardening: gVisor, nodeSelector, NetworkPolicy, provider keys and failed-Pod recovery
The base example is deliberately portable: it runs on kind and GKE without a
gVisor runtime, a tainted node pool or a NetworkPolicy-capable CNI, and it
boots with only the required `dsh-provider-keys` Secret. This document covers
four **opt-in** isolation layers, plus one availability setting:
| Setting | Where it lives | What it changes |
| --- | --- | --- |
| gVisor | commented `runtimeClassName` in the SandboxTemplates and Deployments | puts DSH and/or the gateway in a userspace kernel |
| nodeSelector | commented `nodeSelector` in the same files | pins pods to a chosen node pool |
| NetworkPolicy | `manifests/optional/network-policy/`, `manifests/exposure/*/` | limits sandbox, gateway and proxy connectivity |
| Provider keys | optional `dsh-provider-keys-extra` Secret | injects extra provider API keys into every sandbox |
| Failed-Pod recovery | commented `failedPodPolicy` in the SandboxTemplates | replaces a Failed sandbox Pod instead of parking the sandbox |
Each setting is independent. Enable only what the cluster supports and test the
walkthrough after each change.
## 1. gVisor
### What it changes
The DSH sandbox and the gateway (and optionally oauth2-proxy) run under the
`gvisor` RuntimeClass. `agent-sandbox` passes `runtimeClassName` straight to the
PodSpec; no CRD change is involved.
### Prerequisites
- A `RuntimeClass` named `gvisor`:
```bash
kubectl get runtimeclass gvisor -o yaml
```
- Nodes capable of running it (`runsc` installed and registered with the
container runtime). How you get there is cluster-specific:
- GKE Sandbox: create a node pool with `--sandbox=type=gvisor`; GKE labels
the nodes `sandbox.gke.io/runtime: gvisor`, taints them
`sandbox.gke.io/runtime=gvisor:NoSchedule`, and its `gvisor` RuntimeClass
applies the matching affinity and toleration automatically.
- Self-managed: install gVisor, create the `gvisor` RuntimeClass with
`handler: runsc`, and label/taint a dedicated node pool yourself.
- A scheduling plan: a gVisor RuntimeClass without capable nodes makes pods
fail to start. Use the RuntimeClass `scheduling` fields or the optional
`nodeSelector`/toleration described in section 2.
### Enable it
**Fresh install:** uncomment `runtimeClassName: gvisor` before applying:
- [`manifests/10-sandboxtemplate.yaml`](../manifests/10-sandboxtemplate.yaml)
- [`manifests/50-gateway.yaml`](../manifests/50-gateway.yaml)
- [`manifests/exposure/gateway-api-sso/72-oauth2-proxy.yaml`](../manifests/exposure/gateway-api-sso/72-oauth2-proxy.yaml)
(only with the SSO variant)
- [minimal sandbox-template.yaml](../../minimal-single-tenant/manifests/sandbox-template.yaml)
**Existing install:** a template edit does not update existing Sandboxes, and
warm-pool spares use `OnReplenish`, so patch explicitly. Plan a maintenance
window: claimed users are disconnected when their pod is recreated.
```bash
# 1. The blueprint for future sandboxes.
kubectl -n dsh-demo patch sandboxtemplate dsh-web --type merge \
-p '{"spec":{"podTemplate":{"spec":{"runtimeClassName":"gvisor"}}}}'
# 2. Warm-pool spares: delete them; the pool replenishes from the template.
kubectl -n dsh-demo delete sandboxes -l agents.x-k8s.io/warm-pool-sandbox
# 3. Claimed sandboxes: snapshot their pod template, then suspend/resume.
for sb in $(kubectl -n dsh-demo get sandboxes \
-l '!agents.x-k8s.io/warm-pool-sandbox' -o jsonpath='{.items[*].metadata.name}'); do
kubectl -n dsh-demo patch sandbox "$sb" --type merge \
-p '{"spec":{"podTemplate":{"spec":{"runtimeClassName":"gvisor"}}}}'
kubectl -n dsh-demo patch sandbox "$sb" --type merge \
-p '{"spec":{"operatingMode":"Suspended"}}'
kubectl -n dsh-demo patch sandbox "$sb" --type merge \
-p '{"spec":{"operatingMode":"Running"}}'
done
# 4. The gateway (and oauth2-proxy when the SSO variant is installed).
kubectl -n dsh-demo patch deploy dsh-gateway --type merge \
-p '{"spec":{"template":{"spec":{"runtimeClassName":"gvisor"}}}}'
kubectl -n dsh-demo rollout status deploy/dsh-gateway --timeout=180s
# kubectl -n dsh-demo patch deploy oauth2-proxy --type merge \
# -p '{"spec":{"template":{"spec":{"runtimeClassName":"gvisor"}}}}'
```
If the pool's nodes are tainted and the RuntimeClass does not already add a
toleration, add the matching `tolerations` block (section 2) in the same patch
or in the manifests.
### Verify
```bash
SB=$(kubectl -n dsh-demo get sandboxclaim dsh-alice -o jsonpath='{.status.sandbox.name}')
POD=$(kubectl -n dsh-demo get sandbox "$SB" \
-o jsonpath='{.metadata.annotations.agents\.x-k8s\.io/pod-name}'); POD=${POD:-$SB}
kubectl -n dsh-demo get pod "$POD" -o jsonpath='{.spec.runtimeClassName}{"\n"}'
# gvisor
kubectl -n dsh-demo exec "$POD" -- uname -r
kubectl -n dsh-demo get node "$(kubectl -n dsh-demo get pod "$POD" -o jsonpath='{.spec.nodeName}')" \
-o jsonpath='{.status.nodeInfo.kernelVersion}{"\n"}'
```
The pod's `runtimeClassName` is authoritative. The kernel release reported by
`uname -r` inside a gVisor pod is the sandbox kernel, not the host kernel; on
GKE it looks like `4.19.0-gvisor`.
### Caveats
- **`kubectl port-forward` does not work with gVisor.** The multi-tenant
gateway path (HTTP/WebSocket to the pod) works; the minimal example's
port-forward quickstart does not. Use the gateway example or a reachable
Service (NodePort/LoadBalancer/Sandbox Router) instead.
- gVisor has performance overhead and does not implement every syscall. Test
DSH and the agent tools you rely on (`git`, `gh`, `uv`, language toolchains).
- Node scheduling is the hard part. A `gvisor` RuntimeClass on a cluster with
no capable nodes leaves pods unschedulable or failing at container start.
- Existing Sandboxes are snapshots. The loop above is required for every
claimed sandbox; deleting the `SandboxClaim` would also delete its PVC and
workspace data, so do not use claim re-creation as a shortcut.
- A resumed sandbox starts cold; the startup probe gives it up to 120s.
### Roll back
```bash
kubectl -n dsh-demo patch sandboxtemplate dsh-web --type merge \
-p '{"spec":{"podTemplate":{"spec":{"runtimeClassName":null}}}}'
for sb in $(kubectl -n dsh-demo get sandboxes -o jsonpath='{.items[*].metadata.name}'); do
kubectl -n dsh-demo patch sandbox "$sb" --type merge \
-p '{"spec":{"podTemplate":{"spec":{"runtimeClassName":null}}}}'
done
kubectl -n dsh-demo delete sandboxes -l agents.x-k8s.io/warm-pool-sandbox
kubectl -n dsh-demo patch deploy dsh-gateway --type merge \
-p '{"spec":{"template":{"spec":{"runtimeClassName":null}}}}'
# kubectl -n dsh-demo patch deploy oauth2-proxy --type merge \
# -p '{"spec":{"template":{"spec":{"runtimeClassName":null}}}}'
```
Recreate the pods (delete spares, suspend/resume claimed sandboxes, let the
Deployment roll out) so the cleared field takes effect.
## 2. nodeSelector
### What it changes
Pods get a `nodeSelector` so they land only on nodes carrying the chosen label.
The same field is available in both SandboxTemplates and both Deployments;
the shipped value is **commented out** because active cluster-specific
scheduling does not belong in the repository.
### Choose a label
```bash
kubectl get nodes --show-labels
```
- **GKE Sandbox:** the nodes carry `sandbox.gke.io/runtime: gvisor` and are
tainted `sandbox.gke.io/runtime=gvisor:NoSchedule`. With
`runtimeClassName: gvisor`, GKE already schedules the pod onto those nodes;
add a `nodeSelector` only when you need a specific sandbox pool or
ComputeClass (`cloud.google.com/compute-class: <name>`).
- **Self-managed:** label the pool you want, for example
`dsh.example.com/node-pool: gvisor`, and use that label.
- **General pinning:** any stable node-pool label works. Keep the real key and
value in a local patch; the manifests use `dsh.example.com/node-pool` as an
obvious placeholder.
### Enable it
Uncomment and replace the placeholder before a fresh apply:
```yaml
nodeSelector:
dsh.example.com/node-pool: gvisor
```
For an existing install, patch the template, each claimed Sandbox and the
Deployments, then recreate pods exactly as in section 1:
```bash
kubectl -n dsh-demo patch sandboxtemplate dsh-web --type merge \
-p '{"spec":{"podTemplate":{"spec":{"nodeSelector":{"dsh.example.com/node-pool":"gvisor"}}}}}'
# ... same warm-pool delete / claimed-sandbox suspend-resume / Deployment
# patch and rollout as in section 1, with nodeSelector instead of
# runtimeClassName.
```
If the selected nodes are tainted, add a matching toleration. This is the
companion block; do **not** uncomment it unless the pool is actually tainted:
```yaml
tolerations:
- key: dsh.example.com/node-pool
operator: Equal
value: gvisor
effect: NoSchedule
```
GKE Sandbox's taint is `sandbox.gke.io/runtime=gvisor:NoSchedule`; the GKE
`gvisor` RuntimeClass adds that toleration automatically, so a manual copy is
only needed when you schedule without the RuntimeClass or with a different
pool.
### Verify and troubleshoot
```bash
kubectl -n dsh-demo get pod "$POD" -o wide
kubectl -n dsh-demo get pod "$POD" -o jsonpath='{.spec.nodeSelector}{"\n"}'
kubectl -n dsh-demo describe pod "$POD" | sed -n '/Events:/,$p'
```
- `Pending` with `0/N nodes are available`: the label is wrong, the pool is
full, or the pod needs a toleration for a taint.
- A pod-level `nodeSelector` is merged with the RuntimeClass
`scheduling.nodeSelector`. If both are set and conflict, no node matches.
- Labels changed after scheduling do not move running pods. Patch/recreate.
- A `nodeSelector` that excludes every gVisor node is a valid way to keep the
gateway on a normal pool while the sandboxes run under gVisor.
### Roll back
Remove the field from the manifests, or patch it to `null`, then recreate the
pods:
```bash
kubectl -n dsh-demo patch sandboxtemplate dsh-web --type merge \
-p '{"spec":{"podTemplate":{"spec":{"nodeSelector":null}}}}'
# ... same per-Sandbox and Deployment loop as section 1.
```
## 3. NetworkPolicy
The base example does **not** manage NetworkPolicy: the SandboxTemplate is
`networkPolicyManagement: Unmanaged`, and the gateway has no policy. That keeps
the walkthrough portable and independent of the sandbox-router; the opt-in
policies below are standard Kubernetes `NetworkPolicy` objects.
### Optional sandbox policy
[`manifests/optional/network-policy/10-sandbox-networkpolicy.yaml`](../manifests/optional/network-policy/10-sandbox-networkpolicy.yaml)
selects every Sandbox pod through the agent-sandbox tracking label
`agents.x-k8s.io/sandbox-name-hash`. It applies to existing warm-pool and
claimed sandboxes without recreating them.
```bash
kubectl apply -f manifests/optional/network-policy/
kubectl delete -f manifests/optional/network-policy/ # rollback
```
What it allows:
| Direction | Allowed | Blocked |
| --- | --- | --- |
| Ingress | gateway pods (`app: dsh-gateway`) on TCP `3080` | everything else, including other sandboxes and cluster Services |
| Egress | UDP/TCP `53` to kube-dns and NodeLocal DNSCache; TCP `443` to public IPs | RFC1918, CGNAT, link-local/metadata, cluster Services/API, plain HTTP, other ports |
Verify from inside the pods (NetworkPolicy does not affect `kubectl exec`
itself, only the traffic the commands generate):
```bash
SB=$(kubectl -n dsh-demo get sandboxclaim dsh-alice -o jsonpath='{.status.sandbox.name}')
POD=$(kubectl -n dsh-demo get sandbox "$SB" \
-o jsonpath='{.metadata.annotations.agents\.x-k8s\.io/pod-name}'); POD=${POD:-$SB}
# Allowed: gateway -> sandbox.
kubectl -n dsh-demo exec deploy/dsh-gateway -- python -c \
"import urllib.request; print(urllib.request.urlopen('http://$POD:3080/favicon.svg', timeout=5).status)"
# Allowed: DNS and public HTTPS from the sandbox.
kubectl -n dsh-demo exec "$POD" -- getent hosts api.deepseek.com
kubectl -n dsh-demo exec "$POD" -- curl -fsS -o /dev/null -w '%{http_code}\n' https://example.com
# Denied: another sandbox cannot reach this one on 3080 (expect a timeout).
OTHER_SB=$(kubectl -n dsh-demo get sandboxclaim dsh-bob -o jsonpath='{.status.sandbox.name}')
OTHER_POD=$(kubectl -n dsh-demo get sandbox "$OTHER_SB" \
-o jsonpath='{.metadata.annotations.agents\.x-k8s\.io/pod-name}'); OTHER_POD=${OTHER_POD:-$OTHER_SB}
kubectl -n dsh-demo exec "$OTHER_POD" -- curl --max-time 3 -sS "http://$POD:3080/favicon.svg"
```
Caveats:
- **The CNI must enforce NetworkPolicy.** If it does not, the file applies but
has no effect.
- DNS: the default assumes CoreDNS/kube-dns pods labelled `k8s-app: kube-dns`
and covers NodeLocal DNSCache at `169.254.20.10`. A different DNS layout
needs a selector/`ipBlock` edit.
- **Kubelet probes** from the node are CNI-dependent. If a pod starts failing
its probes, allow the node CIDR on the probe port or remove the ingress
policy.
- The policy blocks cluster Services and the Kubernetes API from the sandbox.
DSH does not need them, but an agent tool that does needs an explicit rule.
- It allows any public host on TCP `443`, not an FQDN allowlist. It is network
isolation, not data-loss prevention.
- Custom provider base URLs, an HTTP proxy, port `80`, SSH `git` (port `22`) or
a non-443 provider endpoint need explicit egress additions.
- Do not combine this standalone policy with `networkPolicyManagement: Managed`
unless you migrate the rules into `spec.networkPolicy`. NetworkPolicies are
additive; the controller's Managed secure default would also be present.
- `kubectl port-forward` still works for debugging (the connection is local to
the pod), except under gVisor.
### Exposure-variant policies
Each exposure variant ships its own policy. They are part of the variant, not a
separate opt-in step:
- [`manifests/exposure/ingress/61-networkpolicy.yaml`](../manifests/exposure/ingress/61-networkpolicy.yaml)
allows only Traefik pods (`app.kubernetes.io/name: traefik`) to reach the
gateway on `8080`.
- [`manifests/exposure/gateway-api-sso/75-networkpolicies.yaml`](../manifests/exposure/gateway-api-sso/75-networkpolicies.yaml)
allows only oauth2-proxy to reach the gateway, only the Gateway
implementation (Traefik) to reach oauth2-proxy, and limits oauth2-proxy
egress to DNS, public HTTPS (GitHub) and the gateway.
The SSO policy is what turns the trust boundary in the
[exposure README](../manifests/exposure/README.md) into network enforcement.
The `dsh-gateway` Service must still stay ClusterIP; do not apply both exposure
variants.
Verify the negative case from a sandbox pod (it is not the proxy):
```bash
kubectl -n dsh-demo exec "$POD" -- curl --max-time 3 -sS http://dsh-gateway:8080/healthz
# expected: timeout/connection failure
```
Then verify the real browser path through the Ingress or Gateway, and confirm
the gateway/proxy pods are `Ready`. If Traefik uses `hostNetwork` or a
different pod label, replace the `podSelector` with the correct selector or an
`ipBlock`. As with the sandbox policy, kubelet probes can be affected on some
CNIs.
### Gateway egress (documented, not shipped)
The gateway needs DNS, the Kubernetes API, and TCP `3080` to sandbox pods. The
API server endpoint is cluster-specific (Service CIDR or node IP/port), so an
egress policy is **not** shipped. If your cluster has a stable API-server CIDR
and port, add an egress policy like this (replace the placeholders; verify the
port, commonly `443` or `6443`):
```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: dsh-gateway-egress
namespace: dsh-demo
spec:
podSelector:
matchLabels:
app: dsh-gateway
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
- to:
- ipBlock:
cidr: 10.96.0.1/32 # placeholder: API-server Service/cluster endpoint
ports:
- protocol: TCP
port: 443
- to:
- podSelector:
matchExpressions:
- key: agents.x-k8s.io/sandbox-name-hash
operator: Exists
ports:
- protocol: TCP
port: 3080
```
Cilium users can express the API server as `toEntities: kube-apiserver` (or
`toServices`) instead of a CIDR. Do not apply the sample blindly: a wrong CIDR
or port silently breaks signup/proxy.
## 4. Optional provider keys
### How injection works
The SandboxTemplate injects two Secrets via `envFrom`:
- `dsh-provider-keys` — required by the walkthrough; holds
`DEEPSEEK_API_KEY`.
- `dsh-provider-keys-extra` — optional; referenced with `optional: true`, so a
missing Secret does not block sandbox startup. Every key in it becomes an
environment variable in every sandbox.
Create the extra Secret before the warm pool (pod env is stamped at creation):
```bash
kubectl -n dsh-demo create secret generic dsh-provider-keys-extra \
--from-literal=OPENROUTER_API_KEY="sk-or-..." \
--from-literal=GEMINI_API_KEY="..." \
--from-literal=NVIDIA_API_KEY="nvapi-..." \
--dry-run=client -o yaml | kubectl apply -f -
```
Use the exact env-var names (uppercase, underscores) — invalid Secret keys are
skipped by the kubelet with an event. Do not duplicate a key across the two
Secrets; when the same key exists twice, the later `envFrom` entry wins.
### Supported names (curated)
These are the names the pinned DSH release resolves through its packaged provider
catalog; the catalog has more entries, these are the common ones:
| Provider | Environment variable |
| --- | --- |
| DeepSeek | `DEEPSEEK_API_KEY` |
| OpenRouter | `OPENROUTER_API_KEY` |
| Google Gemini | `GEMINI_API_KEY` |
| NVIDIA | `NVIDIA_API_KEY` |
| OpenAI | `OPENAI_API_KEY` |
| Anthropic | `ANTHROPIC_API_KEY` |
| Groq | `GROQ_API_KEY` |
| xAI | `XAI_API_KEY` |
| Mistral | `MISTRAL_API_KEY` |
| Together | `TOGETHER_API_KEY` |
| Cerebras | `CEREBRAS_API_KEY` |
| Vercel AI Gateway | `AI_GATEWAY_API_KEY` |
| Hugging Face | `HF_TOKEN` |
Aliases to avoid: `GOOGLE_API_KEY`, `HUGGINGFACE_API_KEY` and
`VERCEL_AI_GATEWAY_API_KEY`. The packaged catalog looks for `GEMINI_API_KEY`,
`HF_TOKEN` and `AI_GATEWAY_API_KEY` instead.
### Picking up a key change
A Secret update does not change running pods. The base key and extra keys are
both snapshotted at container start:
```bash
# Warm-pool spares: delete them so the pool recreates them from the template.
kubectl -n dsh-demo delete sandboxes -l agents.x-k8s.io/warm-pool-sandbox
# Running claimed sandboxes: delete the pod (the Sandbox controller recreates
# it) or suspend/resume the sandbox.
POD=... # resolve it as in section 3
kubectl -n dsh-demo delete pod "$POD"
```
New claims get the new values automatically. Confirm names without printing
secret values:
```bash
kubectl -n dsh-demo exec "$POD" -- sh -c '
for k in DEEPSEEK_API_KEY OPENROUTER_API_KEY GEMINI_API_KEY NVIDIA_API_KEY; do
if [ -n "$(printenv "$k" || true)" ]; then echo "$k=set"; else echo "$k=unset"; fi
done'
```
### Precedence and shared-key limits
- DSH's stored credential file (`$DSH_HOME/.credentials.yaml`, mode `0600`)
**wins over the environment**. If a tenant saved a key through the DSH UI,
changing the Secret will not override it.
- Both Secrets are shared by **every** tenant, and any agent can read its own
environment. This is a warm-pool constraint, not tenant isolation. A tenant
can override its own key through the DSH credential store on its PVC, but the
platform does not manage that; real multi-tenancy needs per-tenant
credentials or an egress proxy.
- The optional Secret is created imperatively; the repository never contains a
real key. Keep it out of Git and out of shell history where possible.
## 5. Failed-Pod recovery
### What it changes
When a sandbox Pod reaches phase `Failed`, the controller's default policy,
`Retain`, leaves the Pod in place and surfaces `Finished=True` on the Sandbox.
The Sandbox, its claim and its PVC survive, but nothing runs: the gateway keeps
proxying to a dead Pod and every request fails until someone deletes the Pod or
the claim.
agent-sandbox v1.0.8 (the pin in [versions.env](../../../versions.env)) adds
`spec.failedPodPolicy` to `Sandbox` and to `SandboxTemplate`:
| Value | Behavior |
| --- | --- |
| `Retain` (default) | The Failed Pod stays and the Sandbox reports `Finished=True`; the evidence stays available for debugging. |
| `Recreate` | The controller deletes the Failed Pod and starts a replacement; the Sandbox identity and its Sandbox-owned PVCs are retained. |
Repeated failures back off like a Deployment - 5 s, doubling to 5 min between
creations - so a crash loop cannot hammer the API server. The backoff resets
once the replacement Pod has been continuously `Running` for 10 minutes, and it
lives in memory only, so a controller restart clears it.
The base templates ship it commented out, for two reasons:
- A deterministic failure (bad image, bad Secret, a wrong probe) would recreate
forever while hiding the cause.
- The templates use `restartPolicy: OnFailure`, so the kubelet restarts a
crashing container in place and the Pod does not reach phase `Failed` anyway.
This policy matters for terminal failures - most commonly node eviction
(`Evicted`), when a spot/preemptible node is reclaimed or the node hits memory
pressure.
No gateway change is needed: the handshake cache is keyed by Pod UID, so a
replacement Pod re-handshakes on the next request, and the existing `401` retry
covers containers restarted in place.
### Enable it
```bash
# 1. The blueprint for future sandboxes.
kubectl -n dsh-demo patch sandboxtemplate dsh-web --type merge \
-p '{"spec":{"failedPodPolicy":"Recreate"}}'
# 2. Warm-pool spares were stamped at creation: delete them and let the pool
# replenish from the updated template.
kubectl -n dsh-demo delete sandboxes -l agents.x-k8s.io/warm-pool-sandbox
# 3. Claimed sandboxes keep their own spec: patch them in place.
for sb in $(kubectl -n dsh-demo get sandboxes \
-l '!agents.x-k8s.io/warm-pool-sandbox' -o jsonpath='{.items[*].metadata.name}'); do
kubectl -n dsh-demo patch sandbox "$sb" --type merge \
-p '{"spec":{"failedPodPolicy":"Recreate"}}'
done
```
For a fresh install, uncomment `failedPodPolicy: Recreate` in the template
instead.
### Verify
```bash
SB=$(kubectl -n dsh-demo get sandboxclaim dsh-alice -o jsonpath='{.status.sandbox.name}')
kubectl -n dsh-demo get sandbox "$SB" -o jsonpath='{.spec.failedPodPolicy}{"\n"}'
# Recreate
```
A failure drill that never touches a user sandbox - a `Sandbox` with
`restartPolicy: Never` and a container that exits non-zero (swap the image if
Docker Hub is unreachable):
```bash
kubectl -n dsh-demo apply -f - <<'EOF'
apiVersion: agents.x-k8s.io/v1beta1
kind: Sandbox
metadata:
name: failed-pod-drill
spec:
failedPodPolicy: Recreate
podTemplate:
spec:
restartPolicy: Never
containers:
- name: drill
image: busybox:1.36
command: ["sh", "-c", "exit 1"]
EOF
kubectl -n dsh-demo get pods -w
# The drill Pod reaches phase Failed; a replacement appears within ~5 s, then
# 10 s, 20 s, ... as the container keeps exiting non-zero.
kubectl -n dsh-demo delete sandbox failed-pod-drill
```
Repeat the drill with `failedPodPolicy: Retain` (or omit the field) to see the
default: the Pod stays `Failed` and the Sandbox condition `Finished` becomes
`True`.
### Caveats
- `Recreate` deletes the Failed Pod, so its logs go with it. Capture
`kubectl get events` or switch back to `Retain` when debugging.
- The replacement is a cold start: DSH boots again and issues a new launch
token. The workspace comes back from the PVC; in-flight sessions do not.
- The field needs agent-sandbox v1.0.8+. A v1.0.7 cluster rejects it as an
unknown field.
- It applies to sandbox Pods only. The gateway and oauth2-proxy are plain
Deployments, and their ReplicaSets already replace Failed Pods.
### Roll back
```bash
kubectl -n dsh-demo patch sandboxtemplate dsh-web --type merge \
-p '{"spec":{"failedPodPolicy":"Retain"}}'
for sb in $(kubectl -n dsh-demo get sandboxes -o jsonpath='{.items[*].metadata.name}'); do
kubectl -n dsh-demo patch sandbox "$sb" --type merge \
-p '{"spec":{"failedPodPolicy":"Retain"}}'
done
```
Warm-pool spares pick the default up as they are claimed and replenished, or
delete them as in step 2 above.
## 6. Verification summary
Required locally for any change to this example:
```bash
make check # versions, example layout, manifest YAML shape
yamllint .
make test # gateway unit tests (gateway code is untouched)
```
CI additionally parses every manifest and runs `kubectl apply
--dry-run=server` against the real agent-sandbox CRDs. The weekly kind e2e
runs the base walkthrough; it also proves the optional extra Secret may be
absent.
Manual checks, on a cluster with the relevant CNI/runtime:
- gVisor: pod `runtimeClassName`, `uname -r` differs from the node kernel,
gateway proxy works, port-forward limitation acknowledged.
- nodeSelector: pods land on the intended nodes; no `Pending` events.
- Sandbox policy: the allow/deny matrix above.
- Exposure policies: browser path works; a non-proxy pod cannot reach the
gateway; probes stay healthy.
- Provider keys: set/unset output (never the values) and the refresh procedure.
- Failed-Pod recovery: `spec.failedPodPolicy` on the Sandbox, and the drill
above replaces a `Failed` Pod while keeping the claim and PVC.
## 7. Rollback summary
| Layer | Rollback |
| --- | --- |
| Sandbox policy | `kubectl delete -f manifests/optional/network-policy/` |
| Exposure policy | delete the exposure variant directory |
| gVisor | patch `runtimeClassName: null`, recreate pods |
| nodeSelector | patch `nodeSelector: null`, recreate pods |
| Provider keys | delete the extra Secret or a key, recreate pods |
| Failed-Pod recovery | patch `failedPodPolicy: Retain`, delete warm-pool spares |
No PVC or workspace data is touched by any rollback. Deleting a `SandboxClaim`
is different: it cascades to the Sandbox **and** its PVC, so do not use claim
deletion to apply a pod-spec change.

View File

@@ -0,0 +1,11 @@
# Runtime image only needs the app and its pinned runtime requirements.
__pycache__/
*.py[cod]
.pytest_cache/
.ruff_cache/
.venv/
test_gateway.py
requirements-dev.txt
pyproject.toml
README.md
coverage.xml

View File

@@ -4,7 +4,7 @@ A single-file, browser-usable control + data plane for
[DeepSeek Harness Web (DSH)](https://github.com/deepseek-ai/deepseek-harness)
sandboxes running behind the `agent-sandbox` CRDs.
It does three things:
It does four things:
1. **Users** — creates a `SandboxClaim` from a warm pool, mints a bearer token
(only its SHA-256 is stored as a claim annotation), reports derived state,
@@ -15,10 +15,15 @@ It does three things:
3. **Reverse proxy** — proxies HTTP (including SSE) and WebSocket traffic to
the sandbox's DSH Web server on `:3080`, including DSH's own cookie
handshake.
4. **Optional SSO** — when `AUTH_PROXY_HEADER` is set, trusts the identity
header an authenticating reverse proxy (for example oauth2-proxy) forwards,
auto-provisions that user, and exposes `POST /recreate` for a clean restart.
DSH is pinned to `dsh-v0.2.1-rc.2` in the example manifests. It binds `:3080`,
serves `/`, gates `/api` behind a `dsh-auth-*` cookie and requires a WebSocket
at `/api/remote.mux`.
The DSH version pin lives in [`versions.env`](../../../versions.env)
(`DSH_VERSION` plus the optional repository `RELEASE_REVISION`); the example
manifests and published images use the release identity built from it (currently
`dsh-v0.2.1-rc.5`). DSH binds `:3080`, serves `/`, gates `/api` behind a
`dsh-auth-*` cookie and requires a WebSocket at `/api/remote.mux`.
## Flow
@@ -58,10 +63,11 @@ token. A failed handshake returns `503` with `Retry-After: 10`.
| Method | Path | Auth | Purpose |
| --- | --- | --- | --- |
| `GET` | `/healthz` | none | Liveness/readiness. Never touches the cluster. |
| `GET` | `/` | none | Landing page with signup + token login forms. |
| `GET` | `/` | none | Landing page with signup + token login forms. With SSO it auto-provisions the proxy-identified user and redirects to their workspace. |
| `POST` | `/users` | none | Signup. JSON body `{"user": "<dns-1123 label>"}` or form field `user`. JSON returns `201` when Ready, `202` when accepted but still Waking; form returns `303` to `/u/<user>/`. Both set the session cookie. `409` if the user exists, `400` if invalid, `413` if the body is too large. |
| `POST` | `/login` | none | Form (`user`, `token`) returns `303` to `/u/<user>/`; JSON returns `200`. Sets the session cookie. `401` on a bad token, `404` for an unknown user. |
| `POST` | `/logout` | none | Clears the session cookie and `303`s to `/`. |
| `POST` | `/recreate` | trusted proxy header | **SSO only.** Deletes the authenticated user's claim (cascading sandbox + PVC, so workspace data is wiped), waits out finalizers, provisions a fresh claim and `303`s to `/u/<user>/`. `401` without a valid identity header; cookie-authenticated calls must be same-origin. |
| `GET` | `/users/{user}` | bearer or cookie | Derived state: `Provisioning`, `Waking`, `Ready`, `Suspending`, `Suspended`. |
| `DELETE` | `/users/{user}` | bearer or cookie | Delete the claim (sandbox + PVC cascade). Cookie-auth mutations require a same-origin `Origin`/`Referer`. |
| `GET` | `/u/{user}` | — | `308` redirect to `/u/{user}/`. |
@@ -86,6 +92,41 @@ token. A failed handshake returns `503` with `Retry-After: 10`.
absolute paths are rewritten under `/u/<user>/`, and any `token` query
parameter is stripped.
### SSO (opt-in)
By default the gateway is the only authenticator: users sign up with a token or
the session cookie, and the gateway ignores all identity headers. Set
`AUTH_PROXY_HEADER` to enable SSO behind an authenticating reverse proxy such as
oauth2-proxy:
```bash
kubectl -n dsh-demo set env deploy/dsh-gateway AUTH_PROXY_HEADER=X-Forwarded-User
```
When set, the gateway treats a request carrying that header as authenticated as
that user, provided the value is a valid DNS-1123 label; the header is never
trusted when the variable is unset. The landing page then auto-provisions the
user (creates their claim idempotently, mints a rotated session token) and
redirects to `/u/<user>/`, and `POST /recreate` becomes available for a clean
restart. The base manifests leave it unset on purpose; the
[SSO exposure variant](../manifests/exposure/gateway-api-sso/) documents how to
enable it on the Deployment (`kubectl set env`). That variant also ships
`75-networkpolicies.yaml`, which restricts the gateway to oauth2-proxy and
oauth2-proxy's egress to the gateway, DNS and public HTTPS.
> **Trust boundary — this is the security-critical part.** A trusted identity
> header is only safe when **nothing else can reach the gateway**. The gateway
> Service must stay ClusterIP and be reachable exclusively through the
> authenticating proxy; if a client can reach `dsh-gateway:8080` directly (a
> second Ingress/Gateway/Route, a port-forward on a shared host, a broad
> NetworkPolicy), it can send `X-Forwarded-User` itself and impersonate any user.
> The variant's `75-networkpolicies.yaml` restricts pod traffic to the proxy; a
> custom exposure without an equivalent policy has no network enforcement.
> Treat `AUTH_PROXY_HEADER` as "the proxy already authenticated this request",
> not as an authentication mechanism of its own. Disable SSO by removing the
> variable: the header is then ignored and bearer-token/session auth applies
> again.
### Proxy failure semantics
| Situation | Result |
@@ -123,6 +164,7 @@ can also be injected into `create_app` for tests.
| `MAX_BODY_MB` | `32` | Maximum buffered request body (and `413` threshold). |
| `HTTP_TIMEOUT` | `300` | Upstream connect/read timeout in seconds. Proxied streams (SSE, long responses) keep the connect/write budget but have **no read timeout**, so an idle event stream is not cut every five minutes. |
| `SWEEP_INTERVAL` | `15` | Seconds between idle sweeps. |
| `AUTH_PROXY_HEADER` | unset | Opt-in SSO: name of the trusted identity header set by an authenticating reverse proxy (e.g. `X-Forwarded-User`). Unset disables SSO and the header is never trusted. Enable only when the Service is reachable exclusively through that proxy (see [SSO](#sso-opt-in)). |
### Required RBAC
@@ -147,12 +189,21 @@ python -m ruff check .
uvicorn gateway:create_app --factory --host 0.0.0.0 --port 8080
```
Container image:
Container image, from the repository root:
```bash
docker build -t dsh-gateway examples/dsh-agents-as-a-service/gateway
make gateway # builds dsh-aaas-gateway:dev from this directory
```
The equivalent direct build is
`docker build -t dsh-aaas-gateway:dev examples/dsh-agents-as-a-service/gateway`.
For kind, retag it to the image reference in
`examples/dsh-agents-as-a-service/manifests/50-gateway.yaml` and `kind load` it.
Published images are nested below the main image as
`<GHCR_IMAGE>/dsh-agents-as-a-service/gateway:<identity>`, where the identity
comes from [`versions.env`](../../../versions.env) — see
[`docs/versioning.md`](../../../docs/versioning.md).
The image runs as UID `10000`, sets `PYTHONDONTWRITEBYTECODE=1` and
`HOME=/tmp` (read-only-rootfs compatible), listens on `8080`, and disables
uvicorn's access log so bearer tokens in URLs are not logged.
@@ -182,3 +233,9 @@ uvicorn's access log so bearer tokens in URLs are not logged.
re-runs the handshake once and replays the request.
* **Only DSH on `:3080` is proxied.** There is no dashboard/API-port split and
no platform API key injection.
* **SSO trusts a header.** With `AUTH_PROXY_HEADER` set, any client that can
reach the gateway Service directly can spoof the identity header. The gateway
cannot enforce the trust boundary itself: keep the Service ClusterIP and
reachable only through the authenticating proxy. The SSO exposure variant
ships a NetworkPolicy that restricts pod traffic to the proxy; without it
there is no mTLS or signed identity assertion.

View File

@@ -195,9 +195,7 @@ def derive_state(sandbox: dict | None) -> str:
def token_matches(claim: dict, token: str | None) -> bool:
"""Timing-safe check of a user token against the stored SHA-256."""
want = (claim.get("metadata", {}).get("annotations") or {}).get(
TOKEN_ANNOTATION, ""
)
want = (claim.get("metadata", {}).get("annotations") or {}).get(TOKEN_ANNOTATION, "")
if not token or not want:
return False
got = hashlib.sha256(token.encode()).hexdigest()
@@ -331,6 +329,11 @@ class Settings:
# Connection retries while a Ready pod is still binding its port.
upstream_attempts: int = 3
upstream_retry_delay: float = 1.0
# Optional identity header (e.g. "X-Forwarded-User") set by an authenticating
# reverse proxy such as oauth2-proxy. When configured, requests carrying a
# valid user in this header are treated as authenticated (SSO); unset (None)
# disables SSO entirely and the header is never trusted.
auth_proxy_header: str | None = None
@classmethod
def from_env(cls) -> Settings:
@@ -348,6 +351,7 @@ class Settings:
max_body_bytes=_env_int("MAX_BODY_MB", 32) * 1024 * 1024,
http_timeout=_env_float("HTTP_TIMEOUT", 300.0),
sweep_interval=_env_float("SWEEP_INTERVAL", 15.0),
auth_proxy_header=os.environ.get("AUTH_PROXY_HEADER") or None,
)
@@ -496,13 +500,23 @@ class Kubernetes:
{"spec": {"operatingMode": mode}},
)
async def patch_token(self, user: str, token_hash: str) -> None:
await self._call(
"crd",
"patch_namespaced_custom_object",
GROUP,
VERSION,
self.namespace,
"sandboxclaims",
claim_name(user),
{"metadata": {"annotations": {TOKEN_ANNOTATION: token_hash}}},
)
# -- pods and logs -----------------------------------------------------
async def get_pod(self, name: str) -> dict | None:
try:
pod = await self._call(
"core", "read_namespaced_pod", name, self.namespace
)
pod = await self._call("core", "read_namespaced_pod", name, self.namespace)
except ApiException as exc:
if exc.status == 404:
return None
@@ -512,9 +526,7 @@ class Kubernetes:
# so callers can keep using ``pod.get(...)``.
return k8s_client.ApiClient().sanitize_for_serialization(pod)
async def read_pod_logs(
self, name: str, container: str, tail_lines: int
) -> str:
async def read_pod_logs(self, name: str, container: str, tail_lines: int) -> str:
result = await self._call(
"core",
"read_namespaced_pod_log",
@@ -571,17 +583,66 @@ class Gateway:
return token, user
return None, None
def remote_user(self, headers) -> str | None:
"""The user asserted by a trusted reverse proxy, when SSO is enabled.
Honored only when ``auth_proxy_header`` is configured, and only when the
value is a valid dns-1123 label — a spoofed or malformed value is never
trusted.
"""
name = self.settings.auth_proxy_header
if not name:
return None
raw = (headers.get(name) or "").strip()
return raw if valid_user(raw) else None
def _authorized(
self,
claim: dict,
token: str | None,
cookie_user: str | None,
user: str,
remote: str | None = None,
) -> bool:
if remote == user:
return True # identity asserted by the authenticating proxy
if cookie_user is not None and cookie_user != user:
return False # a session for another user never authorizes this one
return token_matches(claim, token)
async def _sso_session(self, remote: str, headers, cookies, response: Response) -> dict | None:
"""Provision ``remote`` idempotently and mint/refresh a session token.
The authenticating proxy has already verified ``remote``, so it is
trusted. A brand-new user is provisioned (claim created, token minted);
an existing user whose session cookie is missing or no longer matches
gets a rotated token so a fresh cookie can be issued. A still-valid
session is reused untouched. Provisioning is non-blocking: the sandbox
is awaited by the proxy path (``_resolve_sandbox``), so the landing page
can report "Provisioning" instead of stalling for ``WAKE_TIMEOUT``.
"""
claim = await self.k8s.get_claim(remote)
token, cookie_user = self.credentials(headers, cookies)
if claim is not None and cookie_user == remote and token_matches(claim, token):
return claim
token = secrets.token_urlsafe(32)
token_hash = hashlib.sha256(token.encode()).hexdigest()
is_new = claim is None
if is_new:
try:
await self.k8s.create_claim(remote, token_hash, self.settings.pool)
except UserExists:
# Lost a concurrent signup race: rotate the existing token.
is_new = False
await self.k8s.patch_token(remote, token_hash)
else:
await self._touch(remote)
else:
await self.k8s.patch_token(remote, token_hash)
await self._touch(remote)
self.set_session(response, remote, token)
return await self.k8s.get_claim(remote) if is_new else claim
@staticmethod
def _mutation_from_cookie(headers, method: str, cookie_user: str | None) -> bool:
return (
@@ -613,9 +674,7 @@ class Gateway:
while True:
async with self.state.lock:
if not self.state.maintenance.get(user):
self.state.in_flight[user] = (
self.state.in_flight.get(user, 0) + 1
)
self.state.in_flight[user] = self.state.in_flight.get(user, 0) + 1
self.state.last_activity[user] = time.time()
return True
if time.monotonic() > deadline:
@@ -624,9 +683,7 @@ class Gateway:
async def _release(self, user: str) -> None:
async with self.state.lock:
self.state.in_flight[user] = max(
self.state.in_flight.get(user, 1) - 1, 0
)
self.state.in_flight[user] = max(self.state.in_flight.get(user, 1) - 1, 0)
self.state.last_activity[user] = time.time()
# -- sandbox / handshake helpers --------------------------------------
@@ -655,9 +712,7 @@ class Gateway:
# Wake-on-connect: safe in ANY state, including mid-suspension.
self._clear_handshake(user)
try:
await self.k8s.patch_operating_mode(
sandbox["metadata"]["name"], "Running"
)
await self.k8s.patch_operating_mode(sandbox["metadata"]["name"], "Running")
except Exception as exc:
raise SandboxUnavailable("could not wake the sandbox") from exc
sandbox = await self.wait_ready(user, self.settings.wake_timeout)
@@ -671,10 +726,7 @@ class Gateway:
def _pod_name(self, sandbox: dict) -> str:
annotations = sandbox.get("metadata", {}).get("annotations") or {}
return (
annotations.get(POD_NAME_ANNOTATION)
or sandbox["metadata"]["name"]
)
return annotations.get(POD_NAME_ANNOTATION) or sandbox["metadata"]["name"]
@staticmethod
def _pod_ip(sandbox: dict) -> str | None:
@@ -718,16 +770,11 @@ class Gateway:
self.settings.dsh_log_tail_lines,
)
except ApiException as exc:
raise HandshakeError(
f"could not read logs for {pod_name}: {exc}"
) from exc
raise HandshakeError(f"could not read logs for {pod_name}: {exc}") from exc
launch_token = parse_dsh_launch_token(logs)
if not launch_token:
raise HandshakeError("DSH launch token not found in pod logs")
url = (
f"http://{fqdn}:{self.settings.dsh_port}/"
f"?token={quote(launch_token, safe='')}"
)
url = f"http://{fqdn}:{self.settings.dsh_port}/?token={quote(launch_token, safe='')}"
headers = {
"Host": host,
"X-Forwarded-Host": host,
@@ -743,9 +790,7 @@ class Gateway:
except httpx.HTTPError as exc:
raise HandshakeError(f"DSH handshake failed: {exc}") from exc
if response.status_code >= 400:
raise HandshakeError(
f"DSH handshake returned {response.status_code}"
)
raise HandshakeError(f"DSH handshake returned {response.status_code}")
cookie = extract_dsh_cookies(response.headers.get_list("set-cookie"))
if not cookie:
raise HandshakeError("DSH handshake set no dsh-auth cookie")
@@ -767,7 +812,63 @@ class Gateway:
async def healthz(self, request: Request) -> JSONResponse:
return JSONResponse({"status": "ok"})
async def landing(self, request: Request) -> HTMLResponse:
@staticmethod
def _sso_page(user: str, state: str) -> str:
"""Minimal SSO landing: connect to the sandbox, or offer to recreate it.
No signup/token forms — the authenticating proxy has already identified
the user. A "Provisioning" sandbox refreshes itself until a pod exists;
any other state auto-connects to ``/u/<user>/`` and offers the single
"Recreate sandbox" action.
"""
safe_user = html.escape(user)
path = quote(user)
if state == "Provisioning":
status = "Provisioning"
body = (
'<meta http-equiv="refresh" content="3">'
"<p>Provisioning your sandbox… this can take a minute.</p>"
)
else:
status = html.escape(state)
confirm = (
"onsubmit=\"return confirm('Delete this sandbox and start fresh? "
"Your workspace data will be wiped.')\""
)
body = (
f'<meta http-equiv="refresh" content="1;url=/u/{path}/">'
"<p>Connecting to your sandbox…</p>"
f'<form method="post" action="/recreate" {confirm}>'
'<button type="submit">Recreate sandbox</button></form>'
)
return (
'<!doctype html><html lang="en"><head><meta charset="utf-8">'
'<meta name="viewport" content="width=device-width, initial-scale=1">'
"<title>DSH sandbox</title><style>"
"body{font-family:system-ui,sans-serif;max-width:42rem;margin:3rem auto;"
"padding:0 1rem;line-height:1.5}"
"form{margin:1rem 0}button{padding:.4rem 1rem}"
"</style></head><body>"
"<h1>DeepSeek Harness sandbox</h1>"
f"<p>Signed in as <strong>{safe_user}</strong>.</p>"
f"<p>Sandbox status: <strong>{status}</strong>.</p>"
f"{body}"
"</body></html>"
)
async def landing(self, request: Request) -> Response:
remote = self.remote_user(request.headers)
if remote:
# SSO: auto-provision + connect. A temporary response captures the
# session cookie minted by _sso_session; the real page is rendered
# once the sandbox state is known.
response = HTMLResponse()
claim = await self._sso_session(remote, request.headers, request.cookies, response)
sandbox = await self._sandbox_of(claim) if claim else None
page = HTMLResponse(self._sso_page(remote, derive_state(sandbox)))
if "set-cookie" in response.headers:
page.headers["set-cookie"] = response.headers["set-cookie"]
return page
token, user = self.credentials(request.headers, request.cookies)
session = ""
if user and token:
@@ -826,14 +927,10 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
@staticmethod
def _wants_form(request: Request) -> bool:
ctype = request.headers.get("content-type", "").lower()
return ctype.startswith(
("application/x-www-form-urlencoded", "multipart/form-data")
)
return ctype.startswith(("application/x-www-form-urlencoded", "multipart/form-data"))
async def _read_payload(self, request: Request) -> dict:
if request.headers.get("content-type", "").lower().startswith(
"application/json"
):
if request.headers.get("content-type", "").lower().startswith("application/json"):
data = await self._read_body(request)
try:
payload = json.loads(data)
@@ -870,9 +967,7 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
return self._error(413, "request body too large")
user = payload.get("user", "")
if not valid_user(user):
return self._error(
400, "body must be {'user': '<dns-1123 label>'}"
)
return self._error(400, "body must be {'user': '<dns-1123 label>'}")
if await self.k8s.get_claim(user) is not None:
return self._error(409, "user exists")
token = secrets.token_urlsafe(32)
@@ -932,7 +1027,7 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
if claim is None:
return self._error(404, "not found")
token, cookie_user = self.credentials(request.headers, request.cookies)
if not self._authorized(claim, token, cookie_user, user):
if not self._authorized(claim, token, cookie_user, user, self.remote_user(request.headers)):
return self._error(401, "unauthorized")
sandbox = await self._sandbox_of(claim)
return JSONResponse(
@@ -951,7 +1046,7 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
if claim is None:
return self._error(404, "not found")
token, cookie_user = self.credentials(request.headers, request.cookies)
if not self._authorized(claim, token, cookie_user, user):
if not self._authorized(claim, token, cookie_user, user, self.remote_user(request.headers)):
return self._error(401, "unauthorized")
if self._mutation_from_cookie(request.headers, "DELETE", cookie_user):
return self._error(403, "cross-origin request rejected")
@@ -960,9 +1055,37 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
async with self.state.lock:
self.state.last_activity.pop(user, None)
self.state.in_flight.pop(user, None)
return JSONResponse(
{"user": user, "note": "claim deleted; sandbox and PVC cascade"}
)
return JSONResponse({"user": user, "note": "claim deleted; sandbox and PVC cascade"})
async def recreate(self, request: Request) -> Response:
"""Delete and re-provision the SSO user's sandbox for a clean start.
SSO-only: the identity comes from the trusted proxy header. Deleting the
claim cascades to the sandbox pod and its PVC, wiping user state; we poll
until the claim is fully gone (finalizers) before recreating to avoid a
name-collision 409, then reconnect.
"""
remote = self.remote_user(request.headers)
if not remote:
return self._error(401, "unauthorized")
_token, cookie_user = self.credentials(request.headers, request.cookies)
if cookie_user is not None and cookie_user != remote:
return self._error(401, "unauthorized")
if self._mutation_from_cookie(request.headers, "POST", cookie_user):
return self._error(403, "cross-origin request rejected")
await self.k8s.delete_claim(remote)
self._clear_handshake(remote)
async with self.state.lock:
self.state.last_activity.pop(remote, None)
self.state.in_flight.pop(remote, None)
# Wait out any finalizer so the claim name is reusable.
for _ in range(30):
if await self.k8s.get_claim(remote) is None:
break
await asyncio.sleep(1)
response = RedirectResponse(f"/u/{remote}/", status_code=303)
await self._sso_session(remote, request.headers, request.cookies, response)
return response
# -- HTTP proxy --------------------------------------------------------
@@ -970,41 +1093,38 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
user = request.path_params["user"]
if not valid_user(user):
return self._error(404, "unknown user")
remote = self.remote_user(request.headers)
claim = await self.k8s.get_claim(user)
if claim is None:
if remote == user:
# SSO first hit straight at a deep link: let landing bootstrap.
return RedirectResponse("/", status_code=303)
return self._error(404, "unknown user")
token, cookie_user = self.credentials(request.headers, request.cookies)
if not self._authorized(claim, token, cookie_user, user):
if remote == user and not (cookie_user == user and token_matches(claim, token)):
# SSO user without a valid session cookie: bootstrap, then continue.
return RedirectResponse("/", status_code=303)
if not self._authorized(claim, token, cookie_user, user, remote):
return self._error(401, "unauthorized")
if self._mutation_from_cookie(request.headers, request.method, cookie_user):
return self._error(403, "cross-origin request rejected")
if not await self._acquire(user):
return self._error(
503, "gateway busy, retry shortly", {"Retry-After": "5"}
)
return self._error(503, "gateway busy, retry shortly", {"Retry-After": "5"})
streaming = False
try:
sandbox, fqdn = await self._resolve_sandbox(user, claim)
host = request.headers.get("host", "")
cookie, _, was_cached = await self.dsh_cookie(
user, sandbox, fqdn, host
)
cookie, _, was_cached = await self.dsh_cookie(user, sandbox, fqdn, host)
body = await self._read_body(request)
upstream = await self._send_upstream(
request, user, fqdn, cookie, body
)
upstream = await self._send_upstream(request, user, fqdn, cookie, body)
if upstream.status_code == 401 and was_cached:
# The cached DSH cookie expired (pod restart, logout, ...):
# handshake once more and replay the (already buffered) request.
await upstream.aclose()
cookie, _, _ = await self.dsh_cookie(
user, sandbox, fqdn, host, force=True
)
upstream = await self._send_upstream(
request, user, fqdn, cookie, body
)
cookie, _, _ = await self.dsh_cookie(user, sandbox, fqdn, host, force=True)
upstream = await self._send_upstream(request, user, fqdn, cookie, body)
response = StreamingResponse(
self._relay(user, upstream),
@@ -1044,8 +1164,7 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
headers = {
key: value
for key, value in request.headers.items()
if key.lower() not in OWNED_REQUEST_HEADERS
and key.lower() != "content-length"
if key.lower() not in OWNED_REQUEST_HEADERS and key.lower() != "content-length"
}
host = request.headers.get("host", "")
headers["host"] = host
@@ -1057,9 +1176,7 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
else:
headers["x-forwarded-for"] = ""
port = host.rsplit(":", 1)[1] if ":" in host else ""
headers["x-forwarded-port"] = port or (
"443" if request.url.scheme == "https" else "80"
)
headers["x-forwarded-port"] = port or ("443" if request.url.scheme == "https" else "80")
last_error: Exception | None = None
for attempt in range(self.settings.upstream_attempts):
@@ -1143,7 +1260,9 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
if claim is None:
return await self._ws_reject(websocket, 4401)
token, cookie_user = self.credentials(websocket.headers, websocket.cookies)
if not self._authorized(claim, token, cookie_user, user):
if not self._authorized(
claim, token, cookie_user, user, self.remote_user(websocket.headers)
):
return await self._ws_reject(websocket, 4401)
if cookie_user is not None and not same_origin(websocket.headers):
return await self._ws_reject(websocket, 4401)
@@ -1167,9 +1286,7 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
if ":" in upstream_host and not upstream_host.startswith("["):
upstream_host = f"[{upstream_host}]" # IPv6 literal
authority = f"{upstream_host}:{self.settings.dsh_port}"
cookie, _, _ = await self.dsh_cookie(
user, sandbox, fqdn, authority
)
cookie, _, _ = await self.dsh_cookie(user, sandbox, fqdn, authority)
url = f"ws://{authority}/{path}"
query = filtered_query(websocket.url.query)
if query:
@@ -1188,9 +1305,7 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
headers["x-forwarded-for"] = websocket.client.host
subprotocols = [
part.strip()
for part in websocket.headers.get(
"sec-websocket-protocol", ""
).split(",")
for part in websocket.headers.get("sec-websocket-protocol", "").split(",")
if part.strip()
]
upstream_cm = self.ws_connect(
@@ -1206,16 +1321,12 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
return await self._ws_reject(websocket, 1011)
try:
await websocket.accept(
subprotocol=getattr(upstream, "subprotocol", None)
)
await websocket.accept(subprotocol=getattr(upstream, "subprotocol", None))
except RuntimeError:
pass
await self._pump(websocket, upstream)
with contextlib.suppress(Exception):
await websocket.close(
code=self._upstream_close_code(upstream)
)
await websocket.close(code=self._upstream_close_code(upstream))
finally:
if upstream_cm is not None:
with contextlib.suppress(Exception):
@@ -1255,9 +1366,7 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
asyncio.create_task(client_to_upstream()),
asyncio.create_task(upstream_to_client()),
]
done, pending = await asyncio.wait(
tasks, return_when=asyncio.FIRST_COMPLETED
)
done, pending = await asyncio.wait(tasks, return_when=asyncio.FIRST_COMPLETED)
for task in pending:
task.cancel()
await asyncio.gather(*done, *pending, return_exceptions=True)
@@ -1284,17 +1393,13 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
continue
if self.state.in_flight.get(user, 0) > 0:
continue # requests in flight => the user is active
idle = time.time() - self.state.last_activity.setdefault(
user, time.time()
)
idle = time.time() - self.state.last_activity.setdefault(user, time.time())
if idle < self.settings.idle_timeout:
continue
self.state.maintenance[user] = True
try:
print(f"suspending {user} (idle {int(idle)}s)", flush=True)
await self.k8s.patch_operating_mode(
sandbox["metadata"]["name"], "Suspended"
)
await self.k8s.patch_operating_mode(sandbox["metadata"]["name"], "Suspended")
self._clear_handshake(user)
finally:
async with self.state.lock:
@@ -1345,18 +1450,15 @@ def create_app(
k8s = k8s if k8s is not None else Kubernetes(settings.namespace)
owns_http_client = http_client is None
if http_client is None:
http_client = httpx.AsyncClient(
follow_redirects=False, timeout=settings.http_timeout
)
gateway = Gateway(
settings, k8s, http_client, ws_connect or websockets.connect
)
http_client = httpx.AsyncClient(follow_redirects=False, timeout=settings.http_timeout)
gateway = Gateway(settings, k8s, http_client, ws_connect or websockets.connect)
routes = [
Route("/healthz", gateway.healthz),
Route("/", gateway.landing),
Route("/users", gateway.signup, methods=["POST"]),
Route("/login", gateway.login, methods=["POST"]),
Route("/logout", gateway.logout, methods=["POST"]),
Route("/recreate", gateway.recreate, methods=["POST"]),
Route("/users/{user}", gateway.get_user),
Route("/users/{user}", gateway.delete_user, methods=["DELETE"]),
Route("/u/{user}", gateway.redirect_to_workspace),

View File

@@ -1,4 +1,5 @@
-r requirements.txt
pytest==9.1.1
pytest-asyncio==1.4.0
pytest-cov==7.1.0
ruff==0.17.1

View File

@@ -45,9 +45,7 @@ FQDN = "dsh-alice-sbx.test-ns.svc.cluster.local"
def dsh_response(status=200, content=b"", headers=None):
"""A MockTransport response that is a real stream (``aiter_raw`` works)."""
return httpx.Response(
status, headers=headers, stream=httpx.ByteStream(content)
)
return httpx.Response(status, headers=headers, stream=httpx.ByteStream(content))
# --- fakes -----------------------------------------------------------------
@@ -62,6 +60,7 @@ class FakeK8s:
self.created = []
self.deleted = []
self.patches = []
self.token_patches = []
self.conflict_on_create = False
self.get_claim_error = None
self.patch_error = None
@@ -92,6 +91,15 @@ class FakeK8s:
raise self.patch_error
self.patches.append((name, mode))
async def patch_token(self, user, token_hash):
if self.patch_error is not None:
raise self.patch_error
self.token_patches.append((user, token_hash))
claim = self.claims.get(user)
if claim is not None:
annotations = claim.setdefault("metadata", {}).setdefault("annotations", {})
annotations["aaas.example.com/token-sha256"] = token_hash
async def get_pod(self, name):
return self.pods.get(name)
@@ -106,9 +114,7 @@ class FakeUpstream:
self.requests = []
self.handshake_requests = []
self.handshake_status = 303
self.handshake_cookies = [
"dsh-auth-a=one; Path=/; HttpOnly; SameSite=Strict"
]
self.handshake_cookies = ["dsh-auth-a=one; Path=/; HttpOnly; SameSite=Strict"]
self.proxy_handler = None
self.proxy_requests = []
@@ -128,9 +134,7 @@ class FakeUpstream:
self.proxy_requests.append(request)
if self.proxy_handler is not None:
return self.proxy_handler(request)
return dsh_response(
200, b"hello from DSH", {"content-type": "text/plain"}
)
return dsh_response(200, b"hello from DSH", {"content-type": "text/plain"})
class FakeWS:
@@ -195,9 +199,7 @@ def add_user(k8s, user="alice", token=TOKEN, ready=True, pod_name=None, suffix="
"metadata": {
"name": claim_name(user),
"annotations": {
"aaas.example.com/token-sha256": hashlib.sha256(
token.encode()
).hexdigest()
"aaas.example.com/token-sha256": hashlib.sha256(token.encode()).hexdigest()
},
},
"status": {"sandbox": {"name": sandbox_name}},
@@ -212,10 +214,7 @@ def add_user(k8s, user="alice", token=TOKEN, ready=True, pod_name=None, suffix="
},
}
k8s.pods[pod_name] = {"metadata": {"name": pod_name, "uid": POD_UID}}
k8s.logs[pod_name] = (
"booting\n"
f"dsh web: http://10.0.0.7:3080/?token={TOKEN}-launch\n"
)
k8s.logs[pod_name] = f"booting\ndsh web: http://10.0.0.7:3080/?token={TOKEN}-launch\n"
return sandbox_name, pod_name
@@ -231,9 +230,7 @@ def env():
)
k8s = FakeK8s()
upstream = FakeUpstream()
http_client = httpx.AsyncClient(
transport=httpx.MockTransport(upstream), timeout=5
)
http_client = httpx.AsyncClient(transport=httpx.MockTransport(upstream), timeout=5)
ws_connect = FakeWSConnect()
app = create_app(
settings=settings,
@@ -263,9 +260,7 @@ def asgi_client(env, cookies=None):
def stale_handshake(env):
env.gateway.state.handshake[("alice", POD_UID, "gateway.test")] = (
"dsh-auth-a=stale"
)
env.gateway.state.handshake[("alice", POD_UID, "gateway.test")] = "dsh-auth-a=stale"
# --- pure helpers ----------------------------------------------------------
@@ -407,9 +402,7 @@ def test_token_matches_is_timing_safe_hash_check():
claim = {
"metadata": {
"annotations": {
"aaas.example.com/token-sha256": hashlib.sha256(
TOKEN.encode()
).hexdigest()
"aaas.example.com/token-sha256": hashlib.sha256(TOKEN.encode()).hexdigest()
}
}
}
@@ -431,6 +424,7 @@ def test_settings_from_env(monkeypatch):
monkeypatch.setenv("COOKIE_MAX_AGE", "100")
monkeypatch.setenv("MAX_BODY_MB", "2")
monkeypatch.setenv("HTTP_TIMEOUT", "12.5")
monkeypatch.setenv("AUTH_PROXY_HEADER", "X-Forwarded-User")
settings = Settings.from_env()
assert settings.namespace == "ns-x"
assert settings.pool == "pool-x"
@@ -443,6 +437,12 @@ def test_settings_from_env(monkeypatch):
assert settings.cookie_max_age == 100
assert settings.max_body_bytes == 2 * 1024 * 1024
assert settings.http_timeout == 12.5
assert settings.auth_proxy_header == "X-Forwarded-User"
def test_settings_auth_proxy_header_defaults_to_none(monkeypatch):
monkeypatch.delenv("AUTH_PROXY_HEADER", raising=False)
assert Settings.from_env().auth_proxy_header is None
# --- Kubernetes adapter ----------------------------------------------------
@@ -467,14 +467,9 @@ async def test_kubernetes_create_claim_body(monkeypatch):
body = args[4]
assert body["metadata"]["name"] == "dsh-alice"
assert body["metadata"]["annotations"]["aaas.example.com/token-sha256"] == "hash"
assert (
body["metadata"]["annotations"]["aaas.example.com/managed-by"]
== "dsh-gateway"
)
assert body["metadata"]["annotations"]["aaas.example.com/managed-by"] == "dsh-gateway"
assert body["spec"]["warmPoolRef"]["name"] == "test-pool"
assert body["spec"]["additionalPodMetadata"]["labels"][
"sandbox.users.io/dsh-user"
] == "alice"
assert body["spec"]["additionalPodMetadata"]["labels"]["sandbox.users.io/dsh-user"] == "alice"
@pytest.mark.asyncio
@@ -606,9 +601,7 @@ async def test_landing_page_has_signup_and_login_forms(env):
@pytest.mark.asyncio
async def test_landing_page_shows_session_when_cookie_present(env):
async with asgi_client(
env, cookies={"dsh_gw_session": f"alice:{TOKEN}"}
) as client:
async with asgi_client(env, cookies={"dsh_gw_session": f"alice:{TOKEN}"}) as client:
response = await client.get("/")
assert "Signed in as" in response.text
assert "/u/alice/" in response.text
@@ -626,9 +619,7 @@ async def test_workspace_redirect_adds_trailing_slash(env):
@pytest.mark.asyncio
@pytest.mark.parametrize(
"bad_user", ["Alice", "-alice", "alice-", "", "a_b", 123, None, "a" * 60]
)
@pytest.mark.parametrize("bad_user", ["Alice", "-alice", "alice-", "", "a_b", 123, None, "a" * 60])
async def test_signup_rejects_invalid_dns1123_label(env, bad_user):
async with asgi_client(env) as client:
response = await client.post("/users", json={"user": bad_user})
@@ -762,9 +753,7 @@ async def test_login_json_and_failures(env):
@pytest.mark.asyncio
async def test_logout_clears_cookie(env):
async with asgi_client(
env, cookies={"dsh_gw_session": f"alice:{TOKEN}"}
) as client:
async with asgi_client(env, cookies={"dsh_gw_session": f"alice:{TOKEN}"}) as client:
response = await client.post("/logout")
assert response.status_code == 303
assert response.headers["location"] == "/"
@@ -779,12 +768,8 @@ async def test_logout_clears_cookie(env):
async def test_get_user_requires_valid_bearer_or_cookie(env):
add_user(env.k8s)
async with asgi_client(env) as client:
forbidden = await client.get(
"/users/alice", headers={"Authorization": "Bearer wrong"}
)
bearer = await client.get(
"/users/alice", headers={"Authorization": f"Bearer {TOKEN}"}
)
forbidden = await client.get("/users/alice", headers={"Authorization": "Bearer wrong"})
bearer = await client.get("/users/alice", headers={"Authorization": f"Bearer {TOKEN}"})
cookie = await client.get(
"/users/alice",
headers={"Cookie": f"dsh_gw_session=alice:{TOKEN}"},
@@ -820,9 +805,7 @@ async def test_invalid_user_in_path_is_404(env, path):
async def test_delete_user_requires_auth(env):
add_user(env.k8s)
async with asgi_client(env) as client:
response = await client.delete(
"/users/alice", headers={"Authorization": "Bearer wrong"}
)
response = await client.delete("/users/alice", headers={"Authorization": "Bearer wrong"})
assert response.status_code == 401
assert env.k8s.deleted == []
@@ -834,12 +817,8 @@ async def test_delete_with_cookie_requires_same_origin(env):
cookies = {"dsh_gw_session": f"alice:{TOKEN}"}
async with asgi_client(env, cookies=cookies) as client:
missing = await client.delete("/users/alice")
cross = await client.delete(
"/users/alice", headers={"Origin": "http://evil.test"}
)
good = await client.delete(
"/users/alice", headers={"Origin": "http://gateway.test"}
)
cross = await client.delete("/users/alice", headers={"Origin": "http://evil.test"})
good = await client.delete("/users/alice", headers={"Origin": "http://gateway.test"})
assert missing.status_code == 403
assert cross.status_code == 403
assert good.status_code == 200
@@ -851,9 +830,7 @@ async def test_delete_with_cookie_requires_same_origin(env):
async def test_delete_with_bearer_does_not_need_origin(env):
add_user(env.k8s)
async with asgi_client(env) as client:
response = await client.delete(
"/users/alice", headers={"Authorization": f"Bearer {TOKEN}"}
)
response = await client.delete("/users/alice", headers={"Authorization": f"Bearer {TOKEN}"})
assert response.status_code == 200
@@ -862,9 +839,7 @@ async def test_delete_missing_claim_is_404(env):
add_user(env.k8s)
env.k8s.claims.clear()
async with asgi_client(env) as client:
response = await client.delete(
"/users/alice", headers={"Authorization": f"Bearer {TOKEN}"}
)
response = await client.delete("/users/alice", headers={"Authorization": f"Bearer {TOKEN}"})
assert response.status_code == 404
@@ -939,9 +914,9 @@ async def test_missing_cookie_in_handshake_is_503(env):
@pytest.mark.asyncio
async def test_pod_annotation_overrides_sandbox_name(env):
sandbox_name, pod_name = add_user(env.k8s)
env.k8s.sandboxes[sandbox_name]["metadata"]["annotations"][
"agents.x-k8s.io/pod-name"
] = "custom-pod"
env.k8s.sandboxes[sandbox_name]["metadata"]["annotations"]["agents.x-k8s.io/pod-name"] = (
"custom-pod"
)
env.k8s.pods["custom-pod"] = {"metadata": {"uid": "other-uid"}}
env.k8s.logs["custom-pod"] = env.k8s.logs[pod_name]
async with asgi_client(env, cookies={"dsh_gw_session": f"alice:{TOKEN}"}) as client:
@@ -1063,9 +1038,7 @@ async def test_proxy_upstream_connection_error_is_502(env):
@pytest.mark.asyncio
async def test_proxy_body_limit_returns_413(env):
add_user(env.k8s)
env.gateway.settings = Settings(
**{**env.settings.__dict__, "max_body_bytes": 4}
)
env.gateway.settings = Settings(**{**env.settings.__dict__, "max_body_bytes": 4})
async with asgi_client(env) as client:
response = await client.post(
"/u/alice/api/echo",
@@ -1175,9 +1148,7 @@ async def test_proxied_requests_use_the_stream_timeout(env, monkeypatch):
"server": ("gateway.test", 80),
"path_params": {"user": "alice", "path": "events"},
}
response = await env.gateway._send_upstream(
Request(scope), "alice", FQDN, "dsh-auth-a=1", b""
)
response = await env.gateway._send_upstream(Request(scope), "alice", FQDN, "dsh-auth-a=1", b"")
await response.aclose()
assert captured["timeout"].read is None
@@ -1340,9 +1311,7 @@ async def test_sweeper_skips_claim_it_does_not_own(env):
assert "alice" not in env.gateway.state.last_activity
# ...and does not suspend it even when it looks long-idle.
env.gateway.state.last_activity["alice"] = (
time.time() - env.settings.idle_timeout - 10
)
env.gateway.state.last_activity["alice"] = time.time() - env.settings.idle_timeout - 10
await env.gateway.sweep_once()
assert env.k8s.patches == []
@@ -1351,9 +1320,7 @@ async def test_sweeper_skips_claim_it_does_not_own(env):
async def test_sweeper_suspends_idle_ready_sandbox(env):
env.k8s.claims["alice"] = ready_claim()
env.k8s.sandboxes["dsh-alice-sbx"] = ready_sandbox()
env.gateway.state.last_activity["alice"] = (
time.time() - env.settings.idle_timeout - 10
)
env.gateway.state.last_activity["alice"] = time.time() - env.settings.idle_timeout - 10
env.gateway.state.handshake[("alice", POD_UID, "gateway.test")] = "dsh-auth-a=x"
await env.gateway.sweep_once()
@@ -1413,3 +1380,146 @@ async def test_sweeper_survives_transient_api_error(env):
# The per-user maintenance flag must be released even on failure.
assert env.gateway.state.maintenance["alice"] is False
# --- SSO (reverse-proxy identity) ------------------------------------------
def enable_sso(env, header="X-Forwarded-User"):
"""Flip an env into SSO mode: the gateway trusts the named identity header."""
env.settings = Settings(**{**env.settings.__dict__, "auth_proxy_header": header})
env.gateway.settings = env.settings
return env
def test_remote_user_none_when_sso_disabled(env):
assert env.gateway.remote_user({"X-Forwarded-User": "alice"}) is None
def test_remote_user_returns_valid_and_rejects_invalid(env):
enable_sso(env)
g = env.gateway
assert g.remote_user({"X-Forwarded-User": "alice"}) == "alice"
assert g.remote_user({"X-Forwarded-User": " bob "}) == "bob"
assert g.remote_user({"X-Forwarded-User": "UPPER"}) is None
assert g.remote_user({"X-Forwarded-User": "bad name"}) is None
assert g.remote_user({"X-Forwarded-User": ""}) is None
assert g.remote_user({}) is None
@pytest.mark.asyncio
async def test_sso_landing_new_user_shows_provisioning(env):
enable_sso(env)
async with asgi_client(env) as client:
response = await client.get("/", headers={"X-Forwarded-User": "alice"})
assert response.status_code == 200
assert "text/html" in response.headers["content-type"]
assert "Signed in as <strong>alice</strong>" in response.text
assert "Provisioning" in response.text
assert "Recreate sandbox" not in response.text
assert re_cookie_user(response.headers["set-cookie"]) == "alice"
assert [c[0] for c in env.k8s.created] == ["alice"]
assert env.k8s.token_patches == []
@pytest.mark.asyncio
async def test_sso_landing_ready_connects_and_offers_recreate_only(env):
enable_sso(env)
add_user(env.k8s)
async with asgi_client(env, cookies={"dsh_gw_session": f"alice:{TOKEN}"}) as client:
response = await client.get("/", headers={"X-Forwarded-User": "alice"})
assert response.status_code == 200
assert "Signed in as <strong>alice</strong>" in response.text
assert "Recreate sandbox" in response.text
# No signup/token forms for SSO users.
assert 'action="/users"' not in response.text
assert 'action="/login"' not in response.text
# Auto-connects to the workspace.
assert "url=/u/alice/" in response.text
assert env.k8s.created == []
assert env.k8s.token_patches == []
@pytest.mark.asyncio
async def test_sso_landing_rotates_token_when_session_missing(env):
enable_sso(env)
add_user(env.k8s)
async with asgi_client(env) as client:
response = await client.get("/", headers={"X-Forwarded-User": "alice"})
assert response.status_code == 200
assert "Recreate sandbox" in response.text
assert re_cookie_user(response.headers["set-cookie"]) == "alice"
assert [p[0] for p in env.k8s.token_patches] == ["alice"]
assert env.k8s.created == []
@pytest.mark.asyncio
async def test_recreate_deletes_and_reprovisions(env):
enable_sso(env)
add_user(env.k8s)
async with asgi_client(env) as client:
response = await client.post("/recreate", headers={"X-Forwarded-User": "alice"})
assert response.status_code == 303
assert response.headers["location"] == "/u/alice/"
assert env.k8s.deleted == ["alice"]
assert [c[0] for c in env.k8s.created] == ["alice"]
assert re_cookie_user(response.headers["set-cookie"]) == "alice"
@pytest.mark.asyncio
async def test_recreate_requires_sso(env):
add_user(env.k8s)
async with asgi_client(env) as client:
response = await client.post("/recreate", headers={"X-Forwarded-User": "alice"})
assert response.status_code == 401
assert env.k8s.deleted == []
@pytest.mark.asyncio
async def test_recreate_rejects_cross_origin(env):
enable_sso(env)
add_user(env.k8s)
async with asgi_client(env, cookies={"dsh_gw_session": f"alice:{TOKEN}"}) as client:
response = await client.post(
"/recreate",
headers={"X-Forwarded-User": "alice", "Origin": "http://evil.test"},
)
assert response.status_code == 403
assert env.k8s.deleted == []
@pytest.mark.asyncio
async def test_sso_get_user_authorizes_via_header(env):
enable_sso(env)
add_user(env.k8s)
async with asgi_client(env) as client:
response = await client.get("/users/alice", headers={"X-Forwarded-User": "alice"})
assert response.status_code == 200
assert response.json()["user"] == "alice"
@pytest.mark.asyncio
async def test_sso_proxy_redirects_to_bootstrap_without_session(env):
enable_sso(env)
add_user(env.k8s)
async with asgi_client(env) as client:
response = await client.get("/u/alice/", headers={"X-Forwarded-User": "alice"})
assert response.status_code == 303
assert response.headers["location"] == "/"
@pytest.mark.asyncio
async def test_sso_proxy_redirects_to_bootstrap_when_claim_missing(env):
enable_sso(env)
async with asgi_client(env) as client:
response = await client.get("/u/alice/", headers={"X-Forwarded-User": "alice"})
assert response.status_code == 303
assert response.headers["location"] == "/"
@pytest.mark.asyncio
async def test_spoofed_header_ignored_when_sso_disabled(env):
add_user(env.k8s)
async with asgi_client(env) as client:
response = await client.get("/u/alice/", headers={"X-Forwarded-User": "alice"})
assert response.status_code == 401

View File

@@ -0,0 +1,44 @@
# DSH "Agents as a Service" — prerequisites: namespace + shared provider keys.
#
# The dsh-provider-keys Secret is deliberately NOT committed with a usable
# value. DEEPSEEK_API_KEY reaches every tenant sandbox (injected via envFrom
# on the SandboxTemplate, stamped before any user exists — a warm-pool
# constraint), so a real key committed here would be shared by every install
# of this example. Generate a fresh value per install (README step 1;
# scripts/e2e-kind.sh uses a throwaway). Use an upsert: this manifest already
# creates the empty Secret, so a plain `kubectl create` would fail:
#
# kubectl -n dsh-demo create secret generic dsh-provider-keys \
# --from-literal=DEEPSEEK_API_KEY="sk-..." \
# --dry-run=client -o yaml | kubectl apply -f -
#
# Optional: additional providers (OpenRouter, Gemini, NVIDIA, OpenAI, ...) go
# in a second, optional Secret. The SandboxTemplate references it with
# `optional: true`, so sandboxes start even when it does not exist, and every
# key in it becomes an environment variable. Create it BEFORE the warm pool,
# because pod env is stamped at pod creation:
#
# kubectl -n dsh-demo create secret generic dsh-provider-keys-extra \
# --from-literal=OPENROUTER_API_KEY="sk-or-..." \
# --from-literal=GEMINI_API_KEY="..." \
# --from-literal=NVIDIA_API_KEY="nvapi-..." \
# --dry-run=client -o yaml | kubectl apply -f -
#
# Supported env names, precedence, shared-key caveats and the pod-refresh
# procedure are in docs/optional-hardening.md.
#
# All the infrastructure in this example works without a real key; only
# actual model calls need one.
apiVersion: v1
kind: Namespace
metadata:
name: dsh-demo
---
apiVersion: v1
kind: Secret
metadata:
name: dsh-provider-keys
namespace: dsh-demo
type: Opaque
# stringData:
# DEEPSEEK_API_KEY: "sk-your-key-here"

View File

@@ -1,7 +1,10 @@
# The reusable DSH Web blueprint. Warm pools stamp pod env BEFORE any user
# exists, so per-user secrets cannot live in pod env; the shared provider key
# is injected via envFrom, and claims that would bypass the warm pool are
# exists, so per-user secrets cannot live in pod env; the shared provider keys
# are injected via envFrom, and claims that would bypass the warm pool are
# rejected outright (see 40-claim-rejected.yaml).
#
# A stripped-down single-tenant variant of this template (no warm pool, no
# policy enforcement) lives in examples/minimal-single-tenant/.
apiVersion: extensions.agents.x-k8s.io/v1beta1
kind: SandboxTemplate
metadata:
@@ -18,16 +21,40 @@ spec:
# The gateway routes each user to their DSH Web instance by this name.
service: true
# No gateway pod scopes ingress in this standalone demo, so the
# NetworkPolicy is left unmanaged. A production platform would use
# networkPolicyManagement: Managed and restrict ingress to its gateway on
# :3080 — with the shared provider key, that NetworkPolicy is the per-user
# isolation boundary.
# The example leaves NetworkPolicy to an optional, opt-in manifest
# (manifests/optional/network-policy/): the base walkthrough should not
# require a policy-capable CNI, and the controller's Managed secure default
# only allows the sandbox-router, which this example does not deploy. If you
# switch to Managed, define spec.networkPolicy explicitly or the gateway will
# be blocked. See docs/optional-hardening.md.
networkPolicyManagement: Unmanaged
# OPTIONAL: replace a Failed sandbox Pod instead of parking the sandbox.
# The default, Retain, leaves the Failed Pod in place and surfaces
# Finished=True on the Sandbox until a human intervenes. Recreate deletes the
# Failed Pod and starts a replacement with the same Sandbox identity and
# Sandbox-owned PVCs, backing off (5s doubling to 5m) on repeated failures.
# Requires agent-sandbox v1.0.8+; existing Sandboxes keep their own stamped
# policy, so patch them separately. See docs/optional-hardening.md.
# failedPodPolicy: Recreate
podTemplate:
spec:
# DSH Web is multi-arch (amd64 + arm64); no arch nodeSelector needed.
# DSH Web is multi-arch (amd64 + arm64); no arch nodeSelector is needed.
#
# OPTIONAL: run the sandbox under gVisor. Requires a RuntimeClass named
# "gvisor" and nodes that can run it; gVisor breaks `kubectl port-forward`
# into the sandbox (use the gateway). Existing Sandboxes snapshot their
# pod template, so enabling it later requires patching/recreating them.
# See docs/optional-hardening.md.
# runtimeClassName: gvisor
#
# OPTIONAL: pin sandboxes to a dedicated node pool (for example a gVisor
# pool). Replace this placeholder with a label your nodes actually carry;
# add a matching toleration if the pool is tainted. Active
# cluster-specific scheduling belongs in a local patch, not here.
# nodeSelector:
# dsh.example.com/node-pool: dsh
restartPolicy: "OnFailure"
securityContext:
runAsUser: 1000 # the 'node' user in the DSH image
@@ -41,11 +68,12 @@ spec:
# with 0660 and crash-loops before the gateway can proxy to it.
# OnRootMismatch skips that walk once the volume root already carries
# the fsGroup ownership/mode applied on the first mount.
# Keep in sync with deploy/k8s/sandbox-template.yaml.
# Keep in sync with the minimal variant:
# examples/minimal-single-tenant/manifests/sandbox-template.yaml.
fsGroupChangePolicy: OnRootMismatch
containers:
- name: dsh-web
image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.2
image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.5
imagePullPolicy: IfNotPresent
ports:
- name: web
@@ -53,6 +81,13 @@ spec:
envFrom:
- secretRef:
name: dsh-provider-keys
# Optional extra provider keys (OpenRouter, Gemini, NVIDIA, ...).
# The Secret may be absent; every key in it becomes an env var.
# Create it before the warm pool, because pod env is stamped at
# pod creation. See docs/optional-hardening.md.
- secretRef:
name: dsh-provider-keys-extra
optional: true
env:
# Empty => DSH derives links from the request Host, which the
# gateway forwards (plus X-Forwarded-*). Point this at the
@@ -61,8 +96,10 @@ spec:
value: ""
# DSH Host/Origin-fences /api. The gateway forwards the browser's
# Host, so DSH must trust the public gateway host. Empty is right
# for port-forward/localhost; for the optional Ingress set this to
# the same host as 60-ingress.yaml (e.g. dsh.example.com).
# for port-forward/localhost; for either exposure variant set this
# to the public host (e.g. dsh.example.com for
# manifests/exposure/ingress/60-ingress.yaml, or
# dsh-demo.example.com for the gateway-api-sso/ HTTPRoute).
- name: DSH_TRUSTED_HOSTS
value: ""
# Probes target /favicon.svg: DSH gates / behind its launch-token

View File

@@ -6,11 +6,19 @@
# POST /users, POST /login, ... session cookie auth
# /u/<user>/... HTTP + WebSocket proxy to the user's DSH
#
# Image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway
# For local kind development you can build and load a local tag instead, then
# change `image` below to it:
# docker build -t dsh-aaas-gateway:demo gateway/
# kind load docker-image dsh-aaas-gateway:demo
# Image: the release image published for this example, pinned by tag.
# imagePullPolicy: IfNotPresent lets a local kind build loaded under the same
# tag be used without a registry round-trip (scripts/e2e-kind.sh builds
# dsh-aaas-gateway:e2e and overrides this image).
#
# SSO IS OPT-IN: this manifest runs the gateway with its own bearer/session
# auth only, so AUTH_PROXY_HEADER is intentionally NOT set (see the env
# block). To put GitHub OAuth in front of it, use
# manifests/exposure/gateway-api-sso/.
#
# This Service is ClusterIP and is the only intended entry point: expose it
# through exactly ONE of the documented variants in manifests/exposure/
# (ingress/ or gateway-api-sso/), never both and never directly.
apiVersion: v1
kind: ServiceAccount
metadata:
@@ -76,6 +84,15 @@ spec:
app: dsh-gateway
spec:
serviceAccountName: dsh-gateway
# OPTIONAL: run the gateway under gVisor. Requires a RuntimeClass named
# "gvisor" and nodes that can run it. See docs/optional-hardening.md.
# runtimeClassName: gvisor
#
# OPTIONAL: pin the gateway to a dedicated node pool. Replace the
# placeholder with a label your nodes carry; add a toleration for a
# tainted pool. Keep active cluster-specific scheduling in a local patch.
# nodeSelector:
# dsh.example.com/node-pool: dsh
securityContext:
runAsNonRoot: true
runAsUser: 10000 # matches USER in gateway/Dockerfile
@@ -84,7 +101,7 @@ spec:
type: RuntimeDefault
containers:
- name: gateway
image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway
image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway:dsh-v0.2.1-rc.5
imagePullPolicy: IfNotPresent
securityContext:
allowPrivilegeEscalation: false
@@ -105,20 +122,36 @@ spec:
# gateway created are swept: a claim applied by hand from the
# resource-layer manifests carries no aaas.example.com/managed-by
# annotation and is never suspended.
#
# These are the demo values (360/240). scripts/e2e-kind.sh
# overrides both to 60/120 with `kubectl set env` so the smoke
# test does not wait six minutes for the sweeper.
- name: IDLE_TIMEOUT
value: "60"
value: "360"
- name: WAKE_TIMEOUT
value: "120"
value: "240"
# Container name inside the per-user sandbox pods whose logs carry
# the DSH launch token.
- name: DSH_CONTAINER
value: dsh-web
# Cookie dsh_gw_session is HttpOnly + SameSite=Lax. Set to "true"
# when this Service is reached over HTTPS (e.g. the optional
# 60-ingress.yaml with TLS); "false" keeps the plain-HTTP demo
# usable via port-forward.
# Cookie dsh_gw_session is HttpOnly + SameSite=Lax. "false" keeps
# the plain-HTTP demo usable through port-forward: a Secure cookie
# is never sent over HTTP, so login would silently fail.
# HTTPS deployments (the ingress/ variant with TLS, or
# gateway-api-sso/, which terminates TLS at the Gateway) MUST set
# this to "true".
- name: COOKIE_SECURE
value: "false"
# SSO IS OPT-IN AND DISABLED HERE. With AUTH_PROXY_HEADER unset the
# gateway trusts only its own bearer/session auth and ignores every
# identity header. Enable it ONLY behind
# manifests/exposure/gateway-api-sso/, where oauth2-proxy is the
# sole route to this Service: set it to X-Forwarded-User, keep the
# Service ClusterIP and reachable only through the proxy, and
# terminate TLS in front of the gateway. Otherwise anyone who can
# reach :8080 can impersonate any user by sending that header.
# - name: AUTH_PROXY_HEADER
# value: X-Forwarded-User
# Reject proxied request bodies larger than this (413).
- name: MAX_BODY_MB
value: "32"

View File

@@ -0,0 +1,156 @@
# Exposing the gateway
The base manifests deploy the gateway as a ClusterIP Service
(`dsh-gateway:8080`) and expose nothing outside the cluster. WebSocket
upgrades on `/u/<user>/**` are part of the contract, so the entry point must
forward `Upgrade`/`Connection` headers (Traefik does by default).
There are two **mutually exclusive** exposure variants. Pick exactly one:
| Variant | Entry point | Authentication | TLS |
| --- | --- | --- | --- |
| [`ingress/60-ingress.yaml`](ingress/60-ingress.yaml) | Traefik `Ingress` | the gateway's own signup/session cookie | optional, recommended |
| [`gateway-api-sso/`](gateway-api-sso/72-oauth2-proxy.yaml) | Gateway API `Gateway` + `HTTPRoute` | GitHub OAuth via oauth2-proxy, then the gateway cookie | required (the Gateway listener terminates it) |
Applying both gives the gateway two entry points, and the SSO variant is only
safe when the gateway is reachable **exclusively** through oauth2-proxy. Do not
mix them.
Each variant also ships a NetworkPolicy, applied by the same directory apply:
- `ingress/61-networkpolicy.yaml` allows only Traefik pods to reach the gateway
on `8080`.
- `gateway-api-sso/75-networkpolicies.yaml` allows only oauth2-proxy to reach
the gateway, only the Gateway implementation (Traefik) to reach oauth2-proxy,
and limits oauth2-proxy egress to DNS, public HTTPS (GitHub) and the gateway.
The policies assume Traefik's `app.kubernetes.io/name: traefik` pod label.
Adjust the selector for another implementation or a `hostNetwork` install; see
[optional hardening](../../docs/optional-hardening.md) for the probe caveat and
verification commands.
## Applying a variant
Run these commands from the example root
(`examples/dsh-agents-as-a-service/`), the same working directory the example
README uses.
`kubectl apply -f manifests/` is **not recursive**: it applies only the
top-level numbered files and does not descend into `exposure/`. Apply the base
resources first, then the chosen variant explicitly:
```bash
# Base resources: namespace, template, warm pool, claims, gateway.
kubectl apply -f manifests/
# Plain Ingress exposure:
kubectl apply -f manifests/exposure/ingress/
# ... or SSO exposure (create the two Secrets and replace the example.com
# placeholders first — see below):
kubectl apply -f manifests/exposure/gateway-api-sso/
```
**Never use `kubectl apply -R`** (recursive): it would apply both variants at
once and hand out a second, unauthenticated path to the same gateway. Apply the
variant directory on purpose, or the individual files by name.
`kubectl` only reads `.yaml`, `.yml` and `.json` files from a directory, so
the `*.yaml.example` secret templates are skipped automatically; create the real
Secrets imperatively (commands below) before applying the directory.
## `ingress/`: plain Ingress, no SSO
`60-ingress.yaml` routes a host through the cluster's Traefik
(`ingressClassName: traefik`; k3s ships one) to the gateway Service.
1. Replace the `dsh.example.com` host and the `dsh-example-tls` Secret name
with real values, or delete the `tls:` block to serve plain HTTP.
2. Set `DSH_TRUSTED_HOSTS` in the SandboxTemplate to the same public host:
the gateway forwards the browser's `Host`, and DSH rejects unknown hosts on
`/api` (see `../10-sandboxtemplate.yaml`).
3. Set `COOKIE_SECURE=true` on the gateway once the Ingress serves HTTPS:
`kubectl -n dsh-demo set env deploy/dsh-gateway COOKIE_SECURE=true`. The
base manifest keeps `false` so the plain-HTTP port-forward demo works; a
`Secure` cookie is never sent over HTTP, which silently breaks login.
4. Do not attach buffering or compression middlewares to this router: they
break WebSocket upgrades and streaming.
5. `61-networkpolicy.yaml` is part of this variant: it allows only Traefik pods
to reach the gateway on `8080`. Adjust its selector for another ingress
controller or `hostNetwork`; if kubelet probes fail, see the hardening doc.
Verify: `curl -sI https://<host>/healthz` returns `200`.
## `gateway-api-sso/`: TLS + GitHub OAuth
`70-gateway.yaml` defines the TLS `Gateway`, `71-httproute.yaml` sends every
request to oauth2-proxy, and `72-oauth2-proxy.yaml` authenticates the browser
against GitHub and proxies to the gateway with `X-Forwarded-User` set.
`75-networkpolicies.yaml` makes this the only network path to the gateway.
Prerequisites:
- Gateway API CRDs and a Gateway implementation installed;
`gatewayClassName: traefik` matches the Traefik Gateway provider. The CRD
version is pinned in `versions.env` (`GATEWAY_API_VERSION`).
- The OAuth App credentials and a cookie secret, from
`73-oauth2-proxy-secret.yaml.example`.
- A TLS certificate/key for the public host, from
`74-tls-secret.yaml.example`.
- oauth2-proxy is pinned in the manifest to
`quay.io/oauth2-proxy/oauth2-proxy:v7.15.5` (`versions.env`:
`OAUTH2_PROXY_IMAGE`).
Steps:
```bash
# 1. Real Secrets (the committed files are *.yaml.example templates).
kubectl -n dsh-demo create secret generic oauth2-proxy \
--from-literal=client-id='<GitHub OAuth App client ID>' \
--from-literal=client-secret='<GitHub OAuth App client secret>' \
--from-literal=cookie-secret="$(python3 -c 'import secrets; print(secrets.token_urlsafe(32))')"
kubectl -n dsh-demo create secret tls dsh-demo-tls --cert=tls.crt --key=tls.key
# 2. Replace dsh-demo.example.com (70/71/72) and your-github-org (72) with
# the real public host and allowed GitHub organization.
# 3. Enable SSO on the gateway and trust the public host in the template.
kubectl -n dsh-demo set env deploy/dsh-gateway \
AUTH_PROXY_HEADER=X-Forwarded-User COOKIE_SECURE=true
# then set DSH_TRUSTED_HOSTS=dsh-demo.example.com in the SandboxTemplate
# (see ../10-sandboxtemplate.yaml) and re-apply it.
# 4. Apply the variant.
kubectl apply -f manifests/exposure/gateway-api-sso/
```
### SSO trust boundary
`AUTH_PROXY_HEADER` is opt-in and deliberately unset in the base gateway
manifest: the gateway ignores identity headers until it is set. Enabling SSO
takes all of these:
1. `AUTH_PROXY_HEADER=X-Forwarded-User` on `deploy/dsh-gateway` (step 3
above). Without it, oauth2-proxy's login has no effect on the gateway and
users are still asked for the gateway's own token.
2. The gateway Service must be reachable **only through oauth2-proxy**: keep it
ClusterIP, route all browser traffic through the `HTTPRoute` →
oauth2-proxy → `dsh-gateway` chain, and never apply `ingress/` (or any
other path to the Service) while SSO is enabled. A client that can reach
`dsh-gateway:8080` directly can set `X-Forwarded-User` itself and
impersonate any user.
3. Keep `75-networkpolicies.yaml` with the variant. It enforces the boundary in
the network layer (gateway ← oauth2-proxy ← Traefik, and limited oauth2-proxy
egress). A custom exposure without an equivalent policy has no network
enforcement.
Also terminate TLS in front of the gateway (the Gateway listener does) and
leave `COOKIE_SECURE=true` so the session cookie never travels in clear text.
## Switching variants
Delete the variant you had before applying the other one, e.g.
`kubectl delete -f manifests/exposure/ingress/`, and revert
`AUTH_PROXY_HEADER`/`COOKIE_SECURE` on the gateway. Verify with
`kubectl -n dsh-demo get ingress,gateway,httproute` that exactly one path to
the gateway remains.

View File

@@ -0,0 +1,25 @@
# Browser entry point (TLS) for the gateway-api-sso variant: terminates TLS
# for dsh-demo.example.com and routes through 71-httproute.yaml to
# oauth2-proxy, which authenticates the browser before the gateway.
# Mutually exclusive with ../../ingress/ (see ../../README.md). The
# "dsh-demo-tls" Secret comes from 74-tls-secret.yaml.example.
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: dsh-gateway
namespace: dsh-demo
labels:
app.kubernetes.io/part-of: dsh-demo
spec:
gatewayClassName: traefik
listeners:
- name: https
port: 8443
protocol: HTTPS
tls:
mode: Terminate
certificateRefs:
- name: dsh-demo-tls
allowedRoutes:
namespaces:
from: Same

View File

@@ -0,0 +1,23 @@
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: dsh-gateway
namespace: dsh-demo
labels:
app.kubernetes.io/part-of: dsh-demo
spec:
parentRefs:
- name: dsh-gateway
namespace: dsh-demo
sectionName: https
hostnames:
# TODO: replace with the actual public FQDN before applying.
- dsh-demo.example.com
rules:
- backendRefs:
# Route through oauth2-proxy (GitHub OAuth) first; it authenticates the
# browser and forwards to dsh-gateway:8080 with X-Forwarded-User set.
# The gateway only trusts that header when AUTH_PROXY_HEADER is set on
# its Deployment — see 72-oauth2-proxy.yaml and ../README.md.
- name: oauth2-proxy
port: 4180

View File

@@ -0,0 +1,145 @@
# oauth2-proxy — GitHub OAuth login in front of the dsh-gateway.
#
# Runs as a reverse proxy: it authenticates the browser against GitHub
# (restricted to a configured organization), then forwards the request to
# the dsh-gateway Service with the authenticated identity in X-Forwarded-User
# (plus X-Auth-Request-* via --set-xauthrequest). The gateway trusts that
# header only when AUTH_PROXY_HEADER is set on its Deployment.
#
# ENABLING SSO TAKES BOTH SIDES (see ../../../exposure/README.md):
# 1. Set AUTH_PROXY_HEADER=X-Forwarded-User on deploy/dsh-gateway, otherwise
# the gateway ignores the header and the login is cosmetic:
# kubectl -n dsh-demo set env deploy/dsh-gateway \
# AUTH_PROXY_HEADER=X-Forwarded-User COOKIE_SECURE=true
# 2. Make dsh-gateway reachable ONLY through this proxy: keep the Service
# ClusterIP, route browser traffic through 71-httproute.yaml, and do not
# apply ../ingress/ (or anything else that reaches the gateway). A
# client that can reach dsh-gateway:8080 can spoof X-Forwarded-User and
# impersonate any user.
#
# 75-networkpolicies.yaml enforces that boundary at the network layer: the
# gateway accepts traffic only from oauth2-proxy pods, oauth2-proxy only from
# the Gateway implementation, and oauth2-proxy egress is limited to DNS,
# public HTTPS and the gateway. Apply it with this variant.
#
# TLS terminates at the Gateway listener (70-gateway.yaml); keep
# --cookie-secure=true and COOKIE_SECURE=true on the gateway.
#
# The image is pinned to quay.io/oauth2-proxy/oauth2-proxy:v7.15.5
# (versions.env: OAUTH2_PROXY_IMAGE). Credentials come from the
# "oauth2-proxy" Secret (see 73-oauth2-proxy-secret.yaml.example); this
# manifest never carries secrets.
apiVersion: apps/v1
kind: Deployment
metadata:
name: oauth2-proxy
namespace: dsh-demo
labels:
app: oauth2-proxy
app.kubernetes.io/part-of: dsh-demo
spec:
replicas: 1
selector:
matchLabels:
app: oauth2-proxy
template:
metadata:
labels:
app: oauth2-proxy
app.kubernetes.io/part-of: dsh-demo
spec:
# OPTIONAL: run the proxy under gVisor and/or pin it to a node pool.
# See ../../../docs/optional-hardening.md.
# runtimeClassName: gvisor
# nodeSelector:
# dsh.example.com/node-pool: dsh
securityContext:
runAsNonRoot: true
runAsUser: 2000
runAsGroup: 2000
seccompProfile:
type: RuntimeDefault
containers:
- name: oauth2-proxy
image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.5
imagePullPolicy: IfNotPresent
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: ["ALL"]
args:
- --http-address=0.0.0.0:4180
- --provider=github
# TODO: replace with the GitHub organization allowed to sign in.
- --github-org=your-github-org
- --email-domain=*
# Reverse-proxy target: the gateway backend Service.
- --upstream=http://dsh-gateway:8080
# Public callback (TLS terminates at the Gateway listener).
# TODO: replace with the actual public FQDN.
- --redirect-url=https://dsh-demo.example.com/oauth2/callback
- --cookie-secure=true
- --cookie-name=_oauth2_proxy
# Also expose X-Auth-Request-User/Email/Groups upstream.
- --set-xauthrequest
# Single provider: skip the provider-selection page.
- --skip-provider-button
env:
- name: OAUTH2_PROXY_CLIENT_ID
valueFrom:
secretKeyRef:
name: oauth2-proxy
key: client-id
- name: OAUTH2_PROXY_CLIENT_SECRET
valueFrom:
secretKeyRef:
name: oauth2-proxy
key: client-secret
- name: OAUTH2_PROXY_COOKIE_SECRET
valueFrom:
secretKeyRef:
name: oauth2-proxy
key: cookie-secret
ports:
- name: http
containerPort: 4180
livenessProbe:
httpGet:
path: /ping
port: http
initialDelaySeconds: 5
periodSeconds: 10
timeoutSeconds: 1
failureThreshold: 3
readinessProbe:
httpGet:
path: /ready
port: http
initialDelaySeconds: 5
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
resources:
requests:
cpu: 10m
memory: 32Mi
limits:
cpu: 200m
memory: 128Mi
---
apiVersion: v1
kind: Service
metadata:
name: oauth2-proxy
namespace: dsh-demo
labels:
app: oauth2-proxy
app.kubernetes.io/part-of: dsh-demo
spec:
selector:
app: oauth2-proxy
ports:
- name: http
port: 4180
targetPort: http

View File

@@ -0,0 +1,39 @@
# EXAMPLE ONLY — do not apply as-is.
#
# oauth2-proxy reads its GitHub OAuth credentials and cookie-signing secret from
# this Secret. Populate real values and apply:
#
# kubectl -n dsh-demo create secret generic oauth2-proxy \
# --from-literal=client-id='<GitHub OAuth App client ID>' \
# --from-literal=client-secret='<GitHub OAuth App client secret>' \
# --from-literal=cookie-secret="$(python3 -c 'import secrets; print(secrets.token_urlsafe(32))')"
#
# The cookie secret must decode (base64) to exactly 16, 24, or 32 bytes.
# oauth2-proxy base64-decodes it, so it must be URL-safe, unpadded base64.
# Generate it with either of:
#
# python3 -c 'import secrets; print(secrets.token_urlsafe(32))'
# openssl rand -base64 32 | tr -- '+/' '-_' | tr -d '='
#
# Do NOT use plain `openssl rand -base64 32`: that yields 44 characters of
# standard base64 (with "=" and possibly "+"/"/"), which oauth2-proxy cannot
# decode and therefore treats as 44 raw bytes, failing with "cookie_secret must
# be 16, 24, or 32 bytes ... but is 44 bytes".
#
# GitHub OAuth App (https://github.com/settings/developers):
# Homepage URL https://dsh-demo.example.com
# Authorization callback https://dsh-demo.example.com/oauth2/callback
# (replace dsh-demo.example.com with the actual public FQDN)
#
# NOTE: pass the base64 value as-is via --from-literal (kubectl stores the
# literal; it does not base64-encode it a second time).
apiVersion: v1
kind: Secret
metadata:
name: oauth2-proxy
namespace: dsh-demo
type: Opaque
stringData:
client-id: "<github-oauth-app-client-id>"
client-secret: "<github-oauth-app-client-secret>"
cookie-secret: "<urlsafe-unpadded-base64-of-32-random-bytes>"

View File

@@ -0,0 +1,17 @@
# EXAMPLE ONLY — do not apply as-is.
#
# Copy to tls-secret.yaml and populate with real base64-encoded PEM material,
# then apply. The secret must live in the dsh-demo namespace and be named
# "dsh-demo-tls" (referenced by 70-gateway.yaml).
#
# kubectl -n dsh-demo create secret tls dsh-demo-tls \
# --cert=tls.crt --key=tls.key --dry-run=client -o yaml > tls-secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: dsh-demo-tls
namespace: dsh-demo
type: kubernetes.io/tls
data:
tls.crt: <base64-encoded-certificate>
tls.key: <base64-encoded-private-key>

View File

@@ -0,0 +1,120 @@
# OPTIONAL with the SSO exposure variant: enforce the trust boundary at the
# network layer.
#
# The chain is Traefik (Gateway API) -> oauth2-proxy -> dsh-gateway. These
# policies make it the ONLY chain:
# - the gateway accepts traffic only from oauth2-proxy pods;
# - oauth2-proxy accepts traffic only from the Gateway implementation;
# - oauth2-proxy egress is limited to DNS, public HTTPS (GitHub) and the
# gateway.
# The gateway's own egress (Kubernetes API, sandbox pods) is intentionally not
# restricted here: the API-server endpoint is cluster-specific. See
# ../../../docs/optional-hardening.md for that recipe.
#
# This manifest assumes Traefik, matching gatewayClassName: traefik in
# 70-gateway.yaml. Adjust the app.kubernetes.io/name selector for another
# implementation. NetworkPolicy enforcement of kubelet probes is CNI-dependent;
# if the gateway or proxy fails its probes, allow the node CIDR on the probe
# port or remove the ingress policy.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: dsh-gateway-ingress
namespace: dsh-demo
spec:
podSelector:
matchLabels:
app: dsh-gateway
policyTypes:
- Ingress
ingress:
- from:
- podSelector:
matchLabels:
app: oauth2-proxy
ports:
- protocol: TCP
port: 8080
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: oauth2-proxy-ingress
namespace: dsh-demo
spec:
podSelector:
matchLabels:
app: oauth2-proxy
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector: {}
podSelector:
matchLabels:
app.kubernetes.io/name: traefik
ports:
- protocol: TCP
port: 4180
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: oauth2-proxy-egress
namespace: dsh-demo
spec:
podSelector:
matchLabels:
app: oauth2-proxy
policyTypes:
- Egress
egress:
# Cluster DNS.
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
# NodeLocal DNSCache.
- to:
- ipBlock:
cidr: 169.254.20.10/32
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
# The gateway backend.
- to:
- podSelector:
matchLabels:
app: dsh-gateway
ports:
- protocol: TCP
port: 8080
# GitHub OAuth endpoints are public HTTPS. Private ranges and the metadata
# server stay blocked.
- to:
- ipBlock:
cidr: 0.0.0.0/0
except:
- 10.0.0.0/8
- 172.16.0.0/12
- 192.168.0.0/16
- 169.254.0.0/16
- 100.64.0.0/10
- ipBlock:
cidr: "::/0"
except:
- "fc00::/7"
- "fe80::/10"
ports:
- protocol: TCP
port: 443

View File

@@ -1,15 +1,21 @@
# OPTIONAL browser entry point: expose the gateway through Traefik.
#
# Requires 50-gateway.yaml (Service dsh-gateway:8080) and a Traefik install
# with an IngressClass named "traefik" (k3s ships one). Replace the placeholder
# host below with a name that resolves to your ingress address, and either
# supply a real TLS Secret or delete the `tls:` block for plain HTTP.
# This is the "no SSO" exposure variant and is MUTUALLY EXCLUSIVE with
# ../gateway-api-sso/ — apply exactly one of them (see ../README.md).
#
# Requires the base manifests (Service dsh-gateway:8080 from
# manifests/50-gateway.yaml) and a Traefik install with an IngressClass named
# "traefik" (k3s ships one). Replace the placeholder host below with a name
# that resolves to your ingress address, and either supply a real TLS Secret or
# delete the `tls:` block for plain HTTP.
#
# Two things must match this host:
# 1. SandboxTemplate (10-sandboxtemplate.yaml): set DSH_TRUSTED_HOSTS to
# this host so DSH accepts the forwarded Host/Origin on /api.
# 2. Gateway Deployment (50-gateway.yaml): set COOKIE_SECURE to "true"
# once this Ingress serves HTTPS, so the session cookie is Secure.
# 1. SandboxTemplate (manifests/10-sandboxtemplate.yaml): set
# DSH_TRUSTED_HOSTS to this host so DSH accepts the forwarded Host/Origin
# on /api.
# 2. Gateway Deployment (manifests/50-gateway.yaml): set COOKIE_SECURE to
# "true" once this Ingress serves HTTPS, so the session cookie is Secure.
# The base manifest sets "false" for the plain-HTTP port-forward demo.
#
# WebSocket support is REQUIRED: the browser talks to /u/<user>/... and DSH
# uses WebSockets for live session traffic. Traefik upgrades WS/SSE
@@ -17,6 +23,10 @@
# middlewares (e.g. compress, buffering) to this router — they break streaming
# and upgrades. `path: /` Prefix covers the landing page and every per-user
# proxy path, so no per-sandbox Ingress is needed.
#
# 61-networkpolicy.yaml is part of this variant: it restricts the gateway's
# ingress to Traefik pods. Adjust its selector if your ingress controller uses
# a different label or hostNetwork.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:

View File

@@ -0,0 +1,31 @@
# OPTIONAL with the Ingress exposure variant: restrict the gateway's ingress
# to the ingress controller.
#
# This manifest assumes Traefik, matching this variant's
# `ingressClassName: traefik`. Traefik's Helm chart labels its pods
# app.kubernetes.io/name=traefik; adjust the selector for another controller or
# for a hostNetwork install (where the source is the node IP, not a pod IP).
#
# NetworkPolicy enforcement of kubelet probe traffic is CNI-dependent. If the
# gateway starts failing its /healthz probes, allow the node CIDR on port 8080
# or remove this policy. See ../../../docs/optional-hardening.md.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: dsh-gateway-ingress
namespace: dsh-demo
spec:
podSelector:
matchLabels:
app: dsh-gateway
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector: {}
podSelector:
matchLabels:
app.kubernetes.io/name: traefik
ports:
- protocol: TCP
port: 8080

View File

@@ -0,0 +1,40 @@
# Optional manifests
Opt-in manifests that are **not** part of the top-level walkthrough.
`kubectl apply -f manifests/` is non-recursive, so it does not apply this
directory; apply a subdirectory explicitly.
## network-policy/
A Kubernetes `NetworkPolicy` for the DSH sandboxes:
- ingress only from the gateway pods on TCP `3080` (HTTP + WebSocket);
- egress only to DNS and public HTTPS (provider APIs, package mirrors,
git-over-HTTPS).
Apply and delete it explicitly:
```bash
kubectl apply -f manifests/optional/network-policy/
kubectl delete -f manifests/optional/network-policy/
```
It requires a CNI that enforces `NetworkPolicy`. The selector uses the
agent-sandbox controller's pod tracking label, so the policy applies to
existing warm-pool and claimed sandboxes without recreation.
## gVisor, nodeSelector and provider keys
These are configured in the main manifests or with an extra Secret:
- **gVisor:** uncomment `runtimeClassName: gvisor` in the SandboxTemplate and
the Deployments before applying, or patch an existing install.
- **nodeSelector:** uncomment the field and replace the placeholder node label.
- **Failed-Pod recovery:** uncomment `failedPodPolicy: Recreate` in the
SandboxTemplate (agent-sandbox v1.0.8+) so a Failed Pod is replaced instead
of parked.
- **Provider keys:** create the optional `dsh-provider-keys-extra` Secret; the
base `dsh-provider-keys` Secret remains the one required key.
Full prerequisites, exact commands, verification and caveats:
[optional hardening](../../docs/optional-hardening.md).

View File

@@ -0,0 +1,80 @@
# OPTIONAL, opt-in tenant network policy for the DSH sandboxes.
#
# It is NOT part of the top-level walkthrough. Apply and delete it explicitly:
# kubectl apply -f manifests/optional/network-policy/
# kubectl delete -f manifests/optional/network-policy/
#
# The selector uses the agent-sandbox controller's pod tracking label, which is
# present on every Sandbox pod (including warm-pool spares), so the policy
# applies to existing sandboxes without recreating them. The SandboxTemplate
# stays networkPolicyManagement: Unmanaged. Do not also switch it to Managed
# unless you migrate these rules into spec.networkPolicy; NetworkPolicies are
# additive and the two configurations would be confusing.
#
# Requires a CNI that enforces NetworkPolicy. See docs/optional-hardening.md
# for the connectivity matrix, DNS variants, kubelet-probe caveat and rollback.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: dsh-sandbox
namespace: dsh-demo
spec:
podSelector:
matchExpressions:
- key: agents.x-k8s.io/sandbox-name-hash
operator: Exists
policyTypes:
- Ingress
- Egress
ingress:
# The gateway is the only intended client of a sandbox (HTTP + WebSocket).
- from:
- podSelector:
matchLabels:
app: dsh-gateway
ports:
- protocol: TCP
port: 3080
egress:
# Cluster DNS. CoreDNS/kube-dns pods carry k8s-app=kube-dns.
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
# NodeLocal DNSCache, for clusters that do not send DNS to the kube-dns
# Service. 169.254.20.10 is its standard link-local address.
- to:
- ipBlock:
cidr: 169.254.20.10/32
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
# Public HTTPS only: provider APIs, package mirrors, git-over-HTTPS.
# Private ranges, CGNAT and the metadata server stay blocked.
- to:
- ipBlock:
cidr: 0.0.0.0/0
except:
- 10.0.0.0/8
- 172.16.0.0/12
- 192.168.0.0/16
- 169.254.0.0/16
- 100.64.0.0/10
- ipBlock:
cidr: "::/0"
except:
- "fc00::/7"
- "fe80::/10"
ports:
- protocol: TCP
port: 443

View File

@@ -1,13 +1,37 @@
#!/usr/bin/env bash
# Smoke test for the dsh-agents-as-a-service example: runs the full README
# walkthrough non-interactively against the current kubectl context.
# Expects agent-sandbox (with extensions) already installed.
# End-to-end smoke test for the dsh-agents-as-a-service example: runs the
# README walkthrough non-interactively against the current kubectl context.
#
# Prerequisites:
# - A kind cluster whose name matches the current kubectl context, or set
# KIND_CLUSTER=<name>.
# - agent-sandbox with extensions installed (API v1beta1). The release
# manifest is pinned by AGENT_SANDBOX_VERSION in versions.env:
# . ../../../versions.env
# kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/download/${AGENT_SANDBOX_VERSION}/sandbox-with-extensions.yaml
# - docker, to build the gateway image (skipped with SKIP_GATEWAY=1).
#
# Environment:
# SKIP_GATEWAY=1 run only the resource-layer checks (no gateway build/load,
# no signup/proxy/WebSocket/idle/wake checks)
# KIND_CLUSTER kind cluster name for "kind load docker-image"
# (default: derived from the current kubectl context)
# GW_LOCAL_IMAGE local gateway image tag
# (default: dsh-aaas-gateway:e2e)
#
# The gateway is built from the working tree (gateway/) and loaded into kind,
# so this tests the example as checked out, not the published image.
#
# The script is destructive only inside the dsh-demo namespace: it creates the
# namespace (and a throwaway provider key) and DELETES it on exit via the
# cleanup trap.
#
# NOTE: the only local port bound is the gateway port-forward on 18080.
# The DSH Web container port (3080) is in-cluster only and is never bound
# on the local host.
set -euo pipefail
cd "$(dirname "$0")"
# Example root: paths below are manifests/... and gateway/... from here.
cd "$(dirname "$0")/.."
NS=dsh-demo
pass() { echo "PASS: $1"; }
@@ -20,12 +44,14 @@ cleanup() {
trap cleanup EXIT
echo "=== 1. install prereqs (throwaway key) + template + warm pool"
kubectl apply -f 00-prereqs.yaml
kubectl apply -f manifests/00-prereqs.yaml
# The provider key is never committed; generate a throwaway value.
kubectl -n "$NS" delete secret dsh-provider-keys --ignore-not-found >/dev/null
kubectl -n "$NS" create secret generic dsh-provider-keys \
--from-literal=DEEPSEEK_API_KEY="sk-test-$(openssl rand -hex 16)"
kubectl apply -f 10-sandboxtemplate.yaml -f 20-sandboxwarmpool.yaml
# dsh-provider-keys-extra is intentionally absent: the template's optional
# envFrom reference must not block the warm pool.
kubectl apply -f manifests/10-sandboxtemplate.yaml -f manifests/20-sandboxwarmpool.yaml
for _ in $(seq 1 120); do
READY=$(kubectl -n "$NS" get sandboxes -l agents.x-k8s.io/warm-pool-sandbox \
-o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' | grep -c True || true)
@@ -37,7 +63,7 @@ pass "warm pool: 2 spares Ready"
echo "=== 2. signup via claim (warm adoption)"
T0=$(date +%s)
kubectl apply -f 30-claim-alice.yaml
kubectl apply -f manifests/30-claim-alice.yaml
kubectl -n "$NS" wait sandboxclaim dsh-alice --for=condition=Ready --timeout=60s
ELAPSED=$(( $(date +%s) - T0 ))
SB=$(kubectl -n "$NS" get sandboxclaim dsh-alice -o jsonpath='{.status.sandbox.name}')
@@ -65,7 +91,7 @@ MARKER=$(kubectl -n "$NS" exec "$POD" -- cat /home/node/.dsh/marker.txt)
pass "resumed: state survived on the reattached PVC"
echo "=== 6. injection policy rejects env-carrying claims"
kubectl apply -f 40-claim-rejected.yaml
kubectl apply -f manifests/40-claim-rejected.yaml
# Wait for the POSITIVE rejection signal, not merely "not bound yet".
REASON=""
for _ in $(seq 1 12); do
@@ -78,19 +104,25 @@ done
|| fail "claim was not rejected (Ready reason='$REASON')"
BOUND=$(kubectl -n "$NS" get sandboxclaim dsh-mallory -o jsonpath='{.status.sandbox.name}' 2>/dev/null || true)
[ -z "$BOUND" ] || fail "env-carrying claim unexpectedly bound sandbox '$BOUND'"
kubectl delete -f 40-claim-rejected.yaml >/dev/null
kubectl delete -f manifests/40-claim-rejected.yaml >/dev/null
pass "claim rejected with EnvVarsInjectionRejected and never bound"
echo "=== 7. gateway: browser signup, proxy, idle-suspend, wake-on-connect"
if command -v docker >/dev/null && command -v kind >/dev/null; then
KIND_CLUSTER="${KIND_CLUSTER:-$(kubectl config current-context | sed 's/^kind-//')}"
# Tag exactly what 50-gateway.yaml references so the smoke test exercises
# the shipped manifest unchanged (imagePullPolicy: IfNotPresent uses the
# image loaded into kind).
GW_IMAGE=ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway
docker build -q -t "$GW_IMAGE" gateway/ >/dev/null
kind load docker-image "$GW_IMAGE" --name "$KIND_CLUSTER" >/dev/null
kubectl apply -f 50-gateway.yaml
GW_LOCAL_IMAGE="${GW_LOCAL_IMAGE:-dsh-aaas-gateway:e2e}"
# Test the working tree, not the published image: build the gateway from
# gateway/, load it into kind, then point the Deployment at that tag. The
# shipped manifest pins the release image; imagePullPolicy: IfNotPresent
# makes the node use the loaded one.
docker build -q -t "$GW_LOCAL_IMAGE" gateway/ >/dev/null
kind load docker-image "$GW_LOCAL_IMAGE" --name "$KIND_CLUSTER" >/dev/null
kubectl apply -f manifests/50-gateway.yaml
kubectl -n "$NS" set image deploy/dsh-gateway gateway="$GW_LOCAL_IMAGE"
# Override the demo timeouts from manifests/50-gateway.yaml (360/240) with the
# gateway's working-tree defaults, so idle-suspend and wake-on-connect are
# exercised in seconds instead of minutes.
kubectl -n "$NS" set env deploy/dsh-gateway IDLE_TIMEOUT=60 WAKE_TIMEOUT=120
kubectl -n "$NS" rollout status deploy/dsh-gateway --timeout=120s
# Local port-forward uses 18080 (NOT 3080, which stays in-cluster).
kubectl -n "$NS" port-forward svc/dsh-gateway 18080:8080 >/dev/null 2>&1 &

View File

@@ -0,0 +1,152 @@
# Minimal single-tenant DSH Web
The smallest Kubernetes deployment of DeepSeek Harness (DSH) Web: one
`SandboxTemplate`, a one-replica warm pool and one claim. It is the "just one
DSH Web instance" quickstart.
For the full platform — warm pool sizing, per-user claims, a browser-facing
gateway with cookie sessions, idle-suspend and wake-on-connect — see
[`examples/dsh-agents-as-a-service/`](../dsh-agents-as-a-service/README.md).
This example deliberately has no gateway: you reach DSH through a
`kubectl port-forward` and its launch-token URL.
## Prerequisites
- A Kubernetes cluster with a default StorageClass (kind and GKE both work).
- The agent-sandbox controller with extensions (API `v1beta1`) installed.
The version is pinned as `AGENT_SANDBOX_VERSION` in
[`versions.env`](../../versions.env):
```bash
# from the repository root:
. versions.env
kubectl apply -f "https://github.com/kubernetes-sigs/agent-sandbox/releases/download/${AGENT_SANDBOX_VERSION}/sandbox-with-extensions.yaml"
kubectl -n agent-sandbox-system rollout status deployment --timeout=180s
```
- A DeepSeek API key for real model calls. The pod starts without one; only
model requests fail. Optionally create `dsh-provider-keys-extra` for
OpenRouter, Gemini, NVIDIA or other provider keys (the template references it
with `optional: true`).
## Quickstart
Run these from this example's directory (`examples/minimal-single-tenant/`).
```bash
# 1. Namespace + shared provider keys. They are injected into the sandbox via
# envFrom and are never committed.
kubectl create namespace dsh-demo
kubectl -n dsh-demo create secret generic dsh-provider-keys \
--from-literal=DEEPSEEK_API_KEY="sk-..."
# Optional: extra providers, injected through a second Secret. Create it before
# the warm pool so the pods are stamped with the keys.
kubectl -n dsh-demo create secret generic dsh-provider-keys-extra \
--from-literal=OPENROUTER_API_KEY="sk-or-..." \
--from-literal=GEMINI_API_KEY="..." \
--from-literal=NVIDIA_API_KEY="nvapi-..."
# 2. The template itself (image pinned in the manifest).
kubectl apply -f manifests/sandbox-template.yaml
# 3. A template only describes pods; a warm pool turns it into running
# Sandboxes and a claim adopts one. One replica is enough here.
kubectl apply -f - <<'EOF'
apiVersion: extensions.agents.x-k8s.io/v1beta1
kind: SandboxWarmPool
metadata:
name: dsh-pool
namespace: dsh-demo
spec:
replicas: 1
sandboxTemplateRef:
name: dsh-web-template
---
apiVersion: extensions.agents.x-k8s.io/v1beta1
kind: SandboxClaim
metadata:
name: dsh
namespace: dsh-demo
spec:
warmPoolRef:
name: dsh-pool
EOF
kubectl -n dsh-demo wait sandboxclaim/dsh --for=condition=Ready --timeout=180s
```
## Talk to the instance
DSH gates `/` behind a launch token it prints only in its startup logs. Open
the printed URL once; DSH then sets its own auth cookie and the UI loads
normally.
```bash
SB=$(kubectl -n dsh-demo get sandboxclaim dsh -o jsonpath='{.status.sandbox.name}')
kubectl -n dsh-demo logs "$SB" | grep -m1 '?token='
# dsh web: http://<pod-ip>:3080/?token=...
kubectl -n dsh-demo port-forward "pod/$SB" 18081:3080
# open http://localhost:18081/?token=... from the log line
```
> **Port note:** DSH Web listens on `3080` in-cluster. If `3080` is already in
> use on your workstation, forward to another local port (here `18081`), never
> `3080`. `localhost` is always in DSH's trusted-host list, so the forwarded
> request is accepted.
>
> **gVisor caveat:** if you uncomment `runtimeClassName: gvisor` in the
> template, `kubectl port-forward` no longer works. Use the multi-tenant gateway
> example or a reachable Service instead; see
> [optional hardening](../dsh-agents-as-a-service/docs/optional-hardening.md).
## Network policy and optional hardening
The template omits `networkPolicyManagement`, so the agent-sandbox controller
uses its Managed secure default: ingress only from the sandbox-router, public
egress, and no internal cluster DNS. `kubectl port-forward` is local to the pod
and still works (except under gVisor).
gVisor, an optional `nodeSelector`, extra provider keys and a stricter sandbox
NetworkPolicy are documented in
[optional hardening](../dsh-agents-as-a-service/docs/optional-hardening.md).
## Probes and trusted hosts
Two settings in `manifests/sandbox-template.yaml` are easy to get wrong:
- **Probe path.** `/` returns `401` without the launch-token cookie, so a probe
there crash-loops the pod. The template probes `/favicon.svg` (served
unauthenticated) and gates liveness/readiness behind a `startupProbe`, which
gives a cold start (plugin activation, or the first resume) up to 120s before
a restart.
- **Host allow-list.** DSH rejects unknown `Host` headers. `localhost` and
`127.0.0.1` are always trusted; if you later front the sandbox with an
Ingress or proxy, set `DSH_TRUSTED_HOSTS` to that public host (comma or space
separated). The full example explains both exposure variants in
[`manifests/exposure/README.md`](../dsh-agents-as-a-service/manifests/exposure/README.md).
The template also sets `fsGroupChangePolicy: OnRootMismatch`. DSH's
credentials plugin requires `.credentials.yaml` to be `0600`, while the
kubelet's default fsGroup policy rewrites it to `0660` on every pod start —
without this policy a recreated or resumed sandbox crash-loops. The rationale
is spelled out in
[`10-sandboxtemplate.yaml`](../dsh-agents-as-a-service/manifests/10-sandboxtemplate.yaml).
## Suspend, resume and clean up
```bash
# Suspend: the pod is deleted, the PVC (state) and Service are kept.
kubectl -n dsh-demo patch sandbox "$SB" --type merge -p '{"spec":{"operatingMode":"Suspended"}}'
# Resume: a new pod reattaches the same PVC; conversations survive.
kubectl -n dsh-demo patch sandbox "$SB" --type merge -p '{"spec":{"operatingMode":"Running"}}'
kubectl -n dsh-demo wait sandbox "$SB" --for=condition=Ready --timeout=180s
# Delete everything this quickstart created.
kubectl -n dsh-demo delete sandboxclaim dsh
kubectl delete namespace dsh-demo
```
Deleting the claim cascades to the sandbox and its PVC.

View File

@@ -4,12 +4,16 @@
# multi-tenant platform (warm pool, per-user claims, gateway, idle-suspend),
# see examples/dsh-agents-as-a-service/.
#
# Prerequisites (create these first, or apply 00-prereqs.yaml from the
# examples directory):
# Claims reference this template directly (spec.sandboxTemplateRef): there is
# no warm pool and no per-user platform here. See the README for the full
# quickstart; in short the prerequisites are:
# - Namespace dsh-demo.
# - Secret dsh-provider-keys in dsh-demo with key DEEPSEEK_API_KEY
# (injected via envFrom; the pod starts without it, only model calls fail).
# The agent-sandbox controller (v1.0.7, API v1beta1) must be installed.
# - Optional: Secret dsh-provider-keys-extra with additional provider keys
# (OPENROUTER_API_KEY, GEMINI_API_KEY, NVIDIA_API_KEY, ...). It is
# referenced with `optional: true`, so it may be absent.
# The agent-sandbox controller (v1.0.8, API v1beta1) must be installed.
apiVersion: extensions.agents.x-k8s.io/v1beta1
kind: SandboxTemplate
metadata:
@@ -17,8 +21,26 @@ metadata:
namespace: dsh-demo
spec:
service: true # headless Service per Sandbox -> stable DNS (status.serviceFQDN)
# networkPolicyManagement is omitted, so the controller uses its Managed
# secure default (ingress only from the sandbox-router, public egress). This
# quickstart reaches the sandbox through `kubectl port-forward`, which is
# local to the pod. The multi-tenant example documents opt-in policies in
# examples/dsh-agents-as-a-service/docs/optional-hardening.md.
#
# OPTIONAL: recreate a Failed sandbox Pod instead of retaining it
# (agent-sandbox v1.0.8+). Documented with the multi-tenant example.
# failedPodPolicy: Recreate
podTemplate:
spec:
# OPTIONAL: run under gVisor. gVisor breaks `kubectl port-forward`, so
# this quickstart's only access path will not work; use the multi-tenant
# gateway example or a Service/NodePort instead.
# runtimeClassName: gvisor
#
# OPTIONAL: pin to a dedicated node pool. Replace the placeholder with a
# node label; add a matching toleration if the pool is tainted.
# nodeSelector:
# dsh.example.com/node-pool: dsh
securityContext:
runAsNonRoot: true
runAsUser: 1000 # Matches the UID of the 'node' user
@@ -26,16 +48,15 @@ spec:
fsGroup: 1000
# DSH's credentials plugin refuses to start when .credentials.yaml has
# any group/other permission bit (it creates the file 0600). The
# default kubelet fsGroup policy ("Always") recursively ORs 0660 into
# every file each time the pod starts, so a recreated/resumed sandbox
# comes back with 0660 and crash-loops. OnRootMismatch skips that walk
# once the volume root already has the fsGroup ownership/mode from the
# first mount. Keep in sync with
# examples/dsh-agents-as-a-service/10-sandboxtemplate.yaml.
# default kubelet fsGroup policy ("Always") rewrites it to 0660 on every
# pod start, so a recreated or resumed sandbox crash-loops; OnRootMismatch
# avoids that walk once the volume root already carries the ownership.
# Rationale in full at
# examples/dsh-agents-as-a-service/manifests/10-sandboxtemplate.yaml.
fsGroupChangePolicy: OnRootMismatch
containers:
- name: dsh-web
image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.2
image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.5
imagePullPolicy: IfNotPresent
ports:
- name: web
@@ -43,6 +64,10 @@ spec:
envFrom:
- secretRef:
name: dsh-provider-keys # Secret with key DEEPSEEK_API_KEY
# Optional extra provider keys; an absent Secret is safe.
- secretRef:
name: dsh-provider-keys-extra
optional: true
env:
# Empty => DSH derives links from the request Host. Keep it empty
# when fronting the sandbox with a proxy/gateway.
@@ -50,8 +75,10 @@ spec:
value: ""
# DSH Host/Origin-fences /api. Empty trusts only localhost, which
# is right for `kubectl port-forward`. Set this to the public host
# (e.g. dsh.example.com) if you expose DSH behind an Ingress, so
# requests forwarded with the browser's Host are accepted.
# (e.g. dsh.example.com) if you front DSH with a proxy or Ingress,
# so requests forwarded with the browser's Host are accepted. The
# full example documents both exposure variants in
# examples/dsh-agents-as-a-service/manifests/exposure/README.md.
- name: DSH_TRUSTED_HOSTS
value: ""
volumeMounts:

60
scripts/check-examples.sh Executable file
View File

@@ -0,0 +1,60 @@
#!/usr/bin/env bash
# Enforce the example contract documented in docs/examples.md.
# Run by "make check" locally and by the CI lint job.
set -euo pipefail
cd "$(dirname "$0")/.."
index=examples/README.md
fail=0
err() { echo "check-examples: $*" >&2; fail=1; }
[ -f "$index" ] || err "missing $index"
for dir in examples/*/; do
name="$(basename "$dir")"
[ "$name" = "_template" ] && continue
[ -f "${dir}README.md" ] || err "$name: missing README.md"
[ -d "${dir}manifests" ] || err "$name: missing manifests/"
if [ -d "${dir}manifests" ]; then
count=$(find "${dir}manifests" -maxdepth 1 -type f -name '*.yaml' | wc -l | tr -d ' ')
[ "$count" -gt 0 ] || err "$name: manifests/ has no top-level *.yaml files"
fi
stray=$(find "$dir" -maxdepth 1 -type f -name '*.yaml' | wc -l | tr -d ' ')
[ "$stray" -eq 0 ] || err "$name: manifests belong under manifests/, not at the example root"
if [ -d "${dir}manifests/exposure" ] && [ ! -f "${dir}manifests/exposure/README.md" ]; then
err "$name: manifests/exposure/ requires a README.md describing the variants"
fi
if [ -d "${dir}manifests/optional" ] && [ ! -f "${dir}manifests/optional/README.md" ]; then
err "$name: manifests/optional/ requires a README.md describing the opt-in manifests"
fi
for component in "$dir"*/; do
[ -d "$component" ] || continue
case "$(basename "$component")" in
manifests|scripts|docs) continue ;;
esac
if [ -f "${component}Dockerfile" ] && [ ! -f "${component}README.md" ]; then
err "$name/$(basename "$component"): component with a Dockerfile needs a README.md"
fi
done
if ! grep -Fq "examples/$name/" "$index"; then
err "$name: not listed in examples/README.md"
fi
for script in "$dir"scripts/*.sh; do
[ -e "$script" ] || continue
[ -x "$script" ] || err "$script: must be executable"
done
done
if [ "$fail" -ne 0 ]; then
echo "check-examples: FAILED" >&2
exit 1
fi
echo "check-examples: ok"

View File

@@ -0,0 +1,47 @@
#!/usr/bin/env python3
"""Parse every example manifest and assert the minimal Kubernetes shape."""
from __future__ import annotations
import sys
from pathlib import Path
try:
import yaml
except ModuleNotFoundError: # pragma: no cover - environment guard
sys.exit("check-manifests: PyYAML is required (python -m pip install pyyaml)")
ROOT = Path(__file__).resolve().parent.parent
FILES = sorted(ROOT.glob("examples/*/manifests/**/*.yaml"))
if not FILES:
sys.exit("check-manifests: no manifests found under examples/*/manifests/")
errors: list[str] = []
documents = 0
for path in FILES:
relative = path.relative_to(ROOT)
try:
loaded = list(yaml.safe_load_all(path.read_text(encoding="utf-8")))
except yaml.YAMLError as exc:
errors.append(f"{relative}: invalid YAML: {exc}")
continue
for position, document in enumerate(loaded, start=1):
if document is None:
continue
documents += 1
if not isinstance(document, dict):
errors.append(f"{relative} document {position}: not a mapping")
continue
missing = [key for key in ("apiVersion", "kind", "metadata") if key not in document]
if missing:
errors.append(f"{relative} document {position}: missing {', '.join(missing)}")
continue
metadata = document["metadata"]
if not isinstance(metadata, dict) or not metadata.get("name"):
errors.append(f"{relative} document {position}: metadata.name is required")
if errors:
print("\n".join(errors), file=sys.stderr)
sys.exit(1)
print(f"check-manifests: ok ({len(FILES)} files, {documents} documents)")

70
scripts/check-versions.sh Executable file
View File

@@ -0,0 +1,70 @@
#!/usr/bin/env bash
# Verify that every version pin in the repository agrees with versions.env.
# Run by "make check" locally and by the CI lint job.
set -euo pipefail
cd "$(dirname "$0")/.."
# shellcheck source=/dev/null
. ./versions.env
IDENTITY="$DSH_VERSION"
if [ -n "${RELEASE_REVISION:-}" ]; then
IDENTITY="$DSH_VERSION-$RELEASE_REVISION"
fi
fail=0
err() { echo "check-versions: $*" >&2; fail=1; }
expect() { # <file> <literal>
if ! grep -Fq -- "$2" "$1"; then
err "$1: expected to contain: $2"
fi
}
# Main Dockerfile build args
expect Dockerfile "ARG DSH_VERSION=$DSH_VERSION"
expect Dockerfile "ARG PNPM_VERSION=$PNPM_VERSION"
expect Dockerfile "ARG RELEASE_REVISION=$RELEASE_REVISION"
# Local configuration mirrored for humans
expect .env.sample "DSH_VERSION=$DSH_VERSION"
expect .env.sample "PNPM_VERSION=$PNPM_VERSION"
expect .env.sample "RELEASE_REVISION=$RELEASE_REVISION"
# Image references in the example manifests
for f in \
examples/dsh-agents-as-a-service/manifests/10-sandboxtemplate.yaml \
examples/minimal-single-tenant/manifests/sandbox-template.yaml; do
expect "$f" "image: $GHCR_IMAGE:$IDENTITY"
done
expect examples/dsh-agents-as-a-service/manifests/50-gateway.yaml "image: $GATEWAY_IMAGE:$IDENTITY"
# The gateway image must live below the repository image (one package per
# example component, no unrelated top-level packages).
case "$GATEWAY_IMAGE" in
"$GHCR_IMAGE"/*) ;;
*) err "GATEWAY_IMAGE ($GATEWAY_IMAGE) must be nested below GHCR_IMAGE ($GHCR_IMAGE)" ;;
esac
# Build tooling must read versions.env instead of hard-coding a version.
# (scripts/ is excluded: set-version.sh documents usage examples in comments.)
while IFS= read -r hit; do
[ -n "$hit" ] || continue
err "hard-coded version outside versions.env: $hit"
done < <(grep -rnE 'dsh-v[0-9]' .github/workflows Makefile 2>/dev/null || true)
# Prose may mention versions, but only the current identity or its base tag.
while IFS= read -r token; do
[ -n "$token" ] || continue
case "$token" in
"$IDENTITY"|"$DSH_VERSION") ;;
*) err "stale version reference in docs: $token" ;;
esac
done < <(grep -rhoE 'dsh-v[0-9][A-Za-z0-9._-]*' \
--include='*.md' --exclude-dir=.git --exclude-dir=.venv --exclude-dir=.uvcache \
--exclude=CHANGELOG.md . | sort -u)
if [ "$fail" -ne 0 ]; then
echo "check-versions: FAILED" >&2
exit 1
fi
echo "check-versions: ok ($IDENTITY)"

64
scripts/set-version.sh Executable file
View File

@@ -0,0 +1,64 @@
#!/usr/bin/env bash
# Rewrite every version pin in one pass, then leave the tree ready for
# "make check".
#
# ./scripts/set-version.sh dsh-v0.2.2-rc.1 # new DSH ref, no revision
# ./scripts/set-version.sh dsh-v0.2.1-rc.5 r3 # same ref, new repo revision
set -euo pipefail
cd "$(dirname "$0")/.."
new_version="${1:-}"
new_revision="${2:-}"
case "$new_version" in
dsh-v[0-9]*) ;;
*) echo "usage: $0 <dsh-vX.Y.Z> [rN]" >&2; exit 2 ;;
esac
if [ -n "$new_revision" ]; then
case "$new_revision" in
r[2-9]|r[1-9][0-9]*) ;;
*) echo "revision must look like r2, r3, ..." >&2; exit 2 ;;
esac
fi
# shellcheck source=/dev/null
old_version="$(. ./versions.env; printf '%s' "$DSH_VERSION")"
# shellcheck source=/dev/null
old_revision="$(. ./versions.env; printf '%s' "${RELEASE_REVISION:-}")"
old_identity="$old_version"
[ -z "$old_revision" ] || old_identity="$old_version-$old_revision"
new_identity="$new_version"
[ -z "$new_revision" ] || new_identity="$new_version-$new_revision"
if [ "$old_identity" = "$new_identity" ]; then
echo "set-version: already at $new_identity"
exit 0
fi
# Replace the identity first (it contains the base version), then the base
# version itself. CHANGELOG.md is generated from tags, so it is left alone.
while IFS= read -r file; do
[ -f "$file" ] || continue
case "$file" in
.env.sample|Dockerfile|Makefile|*.md|*.yaml|*.yml|*.sh|*.py|*.toml|*.env) ;;
*) continue ;;
esac
sed -i "s/$old_identity/$new_identity/g; s/$old_version/$new_version/g" "$file"
done < <(git ls-files)
# versions.env is authoritative: write both keys explicitly so an empty
# revision is stored as an empty value.
sed -i "s/^DSH_VERSION=.*/DSH_VERSION=$new_version/" versions.env
sed -i "s/^RELEASE_REVISION=.*/RELEASE_REVISION=$new_revision/" versions.env
# Values stored separately from the full identity: Dockerfile ARG defaults and
# the human-facing sample cannot be reached by the identity replacement above.
sed -i "s/^ARG DSH_VERSION=.*/ARG DSH_VERSION=$new_version/" Dockerfile
sed -i "s/^ARG RELEASE_REVISION=.*/ARG RELEASE_REVISION=$new_revision/" Dockerfile
sed -i "s/^DSH_VERSION=.*/DSH_VERSION=$new_version/" .env.sample
sed -i "s/^RELEASE_REVISION=.*/RELEASE_REVISION=$new_revision/" .env.sample
echo "set-version: $old_identity -> $new_identity"
echo "set-version: run 'make check' and review the diff"

26
versions.env Normal file
View File

@@ -0,0 +1,26 @@
# Single source of truth for every pinned version and image name in this
# repository. Consumed by the Makefile, build.sh, scripts/*.sh and the GitHub
# Actions workflows. Never hard-code these values elsewhere: run "make check"
# (scripts/check-versions.sh) to verify every pin.
# Upstream DeepSeek Harness source, and the ref this repository packages.
DSH_REPO=https://github.com/deepseek-ai/deepseek-harness.git
DSH_VERSION=dsh-v0.2.1-rc.5
PNPM_VERSION=11.28.5
# Release identity = DSH_VERSION plus an optional repository revision.
# An empty RELEASE_REVISION is the first release at this DSH ref; rN (N >= 2)
# marks a repository-only re-release at the same DSH ref. This is the first
# release at dsh-v0.2.1-rc.5, so the identity is the plain tag; a later
# repository-only change at the same ref becomes -r2.
RELEASE_REVISION=
# Control-plane versions used by the examples and their manifests.
AGENT_SANDBOX_VERSION=v1.0.8
GATEWAY_API_VERSION=v1.2.1
OAUTH2_PROXY_IMAGE=quay.io/oauth2-proxy/oauth2-proxy:v7.15.5
# Registry layout: the main image is the repository itself, and every example
# component is a nested package below it (<registry>/<owner>/<repo>/<example>/<component>).
GHCR_IMAGE=ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox
GATEWAY_IMAGE=ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway