1
0
mirror of https://github.com/sredevopsorg/deepseek-kubernetes-sandbox.git synced 2026-11-03 01:13:24 +00:00

39 Commits
v0.0.1 ... main

Author SHA1 Message Date
SREDevOps.org Agent
2178e58d57 chore: enable entire dsh (#46)
* Enable Entire

* entire dsh

---------

Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
2026-11-02 20:56:51 -03:00
SREDevOps.org Agent
fbe886f3b9 fix(gateway): grant sandboxclaims patch and map API errors to 503 (#45)
* fix(gateway): grant patch on sandboxclaims for token rotation

The SSO landing path calls K8sClient.patch_token(), which patches the
`dsh-<user>` SandboxClaim's token-hash annotation whenever the browser has
no valid gateway session: first visit, an expired or foreign cookie, or a
gateway restart that drops the in-memory session store. The gateway Role
only granted get/list/create/delete on sandboxclaims, so the patch returned
403 and, because ApiException is not handled, every such GET / answered a
bare 500 "Internal Server Error".

Grant the missing `patch` verb on
extensions.agents.x-k8s.io/sandboxclaims, and update the least-privilege
comment plus the two docs that enumerate the Role verbs.

Verified with `make check` (versions, example layout, manifests) and
`yamllint`; the 403 and the resulting 500 were reproduced from a running
install and cleared by applying the equivalent Role patch (RBAC is evaluated
per request, so no gateway restart is needed).

* fix(gateway): return 503 instead of a bare 500 on Kubernetes API errors

Every gateway route talks to the Kubernetes API, and ApiException used to be
unhandled: Starlette's ServerErrorMiddleware turned it into a bare
500 "Internal Server Error" plus an ERROR traceback. That masked a missing
RBAC verb as a client-visible crash on GET / (the SSO landing path), and it
would equally hide an API outage or throttling.

Register an ApiException handler that logs one credential-free line
(status/reason) and answers 503 + Retry-After with a fixed body, so the
Kubernetes Status payload (which can name cluster objects) is never proxied.
Guard the first get_claim() in the WebSocket route the same way: WebSocket
scopes bypass Starlette's HTTP exception handlers, so it now closes with 1011
instead of escaping as a traceback.

Tests assert the 503 + Retry-After response, cover the SSO landing path that
produced the reported 500 (forbidden claim patch), and assert the WebSocket
close code. The failure-semantics tables document the new behaviour.

---------

Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
2026-11-01 21:38:49 -03:00
Nicolás Georger
c5b8afde48 chore(changelog): refresh for dsh-v0.2.1-rc.5-r6 (#44)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-10-31 07:48:51 -03:00
Nicolás Georger
8e9bd478b4 chore: improve package installation and add Python depend… (#42)
* chore(dockerfile): improve package installation and add Python dependencies

* chore(dockerfile): update package installation commands for consistency

* chore: update release version to dsh-v0.2.1-rc.5-r6 across documentation and configuration files
2026-10-31 06:25:39 -03:00
SREDevOps.org Agent
1a929e3e24 fix(gateway)!: replace token cookies with revocable sessions and harden abuse controls (#40)
- Replace user:token cookies with opaque server-side sessions, revocable on logout, delete, recreate and expiry; bind every credential to the path user and add cross-user regression tests.
- Rate-limit /users and /login, add optional signup invite codes and form same-origin checks, and unify login failures to 401.
- Require explicit ALLOW_INSECURE_COOKIES for plain HTTP, use __Host-dsh_gw_session when secure, and fix secure cookie deletion.
- Strip client-supplied forwarding/SSO headers, add an optional upstream allowlist, per-user in-flight and WebSocket message caps, credential-free handshake errors, and sweep-time state pruning.
- Digest-pin base images, hash-pin gateway dependencies, verify DSH_COMMIT in the build, and run the gateway image as non-root with a pinned Python base.
- Update manifests, production checklist, architecture/security docs and add an SSO preflight script.

BREAKING CHANGE: existing dsh_gw_session=<user>:<token> cookies are invalid; users must sign in again. Unknown users on POST /login now return 401 instead of 404, and the gateway refuses COOKIE_SECURE=false unless ALLOW_INSECURE_COOKIES=1.

Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
2026-10-31 00:57:35 -03:00
SREDevOps.org Agent
2f959e6afb fix(ci): harden workflow token scope and download integrity (#39)
Security audit of .github/workflows found a script-injection vector in the
release workflow, over-broad token grants on non-publishing jobs, and
downloaded tooling/manifests executed without integrity checks.

- release: pass the tag/dispatch input through env instead of interpolating
  it into the validation script; default workflow permissions to contents:read
- build-main/build-gateway: grant write scopes only to the build job
- _build-image/codeql: stop persisting checkout credentials
- ci/release: verify actionlint, hadolint and git-cliff against pinned SHA-256s
- ci-manifests/e2e: verify upstream CRD manifests against versions.env pins
- e2e: redact token material before uploading cluster diagnostics
- docs: add workflow-security.md with controls, audit record and residual
  repository-setting recommendations

Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
2026-10-30 15:16:51 -03:00
SREDevOps.org Agent
4c073eef71 chore(deps): bump agent-sandbox to v1.0.9 (#38)
Update AGENT_SANDBOX_VERSION from v1.0.8 to v1.0.9, the pin the examples
and the CI manifest dry-run consume. Refresh the compatibility matrix in
docs/versioning.md and clarify the failedPodPolicy sentence in the
hardening doc, which described v1.0.8 as the pin.

Reviewed the v1.0.9 release for breaking changes: the new
volumeClaimTemplates admission rules accept the named "dsh-home" templates
in both examples, and the router NetworkPolicy / Go label-constant changes
do not apply to this repository.

Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
2026-10-30 13:11:39 -03:00
SREDevOps.org Agent
24f23e09ae docs: align readmes, docs and agent guidance with current behavior (#37)
Remove obsolete layout/migration prose and legacy gateway-alias references, reconcile the required-check policy, fix version-label/RBAC/workspace/channel drift, and de-duplicate the example and gateway docs.

Rename the minimal manifest to 10-sandboxtemplate.yaml and mount the dsh-home PVC's workspace subPath at /workspace so the documented persistence is real. Enforce numbered top-level manifests in check-examples.sh.

Verified with make check, yamllint, shellcheck, a Markdown link/anchor scan, and git diff --check.

Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
2026-10-30 12:31:50 -03:00
SREDevOps.org Agent
82ee0d0cdb feat(example): start DSH's folder picker on the persisted workspace (#36)
The directory picker has no start-path setting: it lists $HOME whenever it
opens. With the image default HOME=/home/node the folder explorer started on
a throwaway directory, so point HOME at the PVC-backed /workspace in both
sandbox templates. `~` and $HOME-relative tool caches/config follow; DSH_HOME
stays /home/node/.dsh, so sessions, config and credentials are unaffected.

Document the trade-off in the image contract and both examples, add
troubleshooting for sandboxes whose pod env was stamped before the change,
and assert HOME/~ in the e2e walkthrough.

Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
2026-10-30 11:19:03 -03:00
SREDevOps.org Agent
a09d4ac340 fix: ship bubblewrap for DSH's command sandbox, keep agents usable without it (#35)
* fix(example): keep agents usable when DSH's inner sandbox has no backend

DSH confines bash commands itself (bubblewrap, else Landlock) and fails closed
with SANDBOX_UNAVAILABLE when no runner is usable. In this hardened pod -
non-root, capabilities dropped, seccomp RuntimeDefault - bubblewrap cannot
start (it needs CAP_SYS_ADMIN) and the pod exposes no Landlock runner, so the
shipped workspace-write default made every command fail:

  sandbox mode "workspace-write" is requested but no sandbox backend is usable

The DSH base bundle at the pinned ref documents DSH_PERMISSION_MODE as the
explicit deployment override, so set it to danger-full-access in both sandbox
templates. The per-user pod is the isolation boundary in this example (its own
pod, PVC and Service; non-root; all capabilities dropped), so a working agent
is worth more than an inner file policy that cannot be enforced. Note that
danger-full-access also turns approval prompts off.

Trade-off documented rather than hidden: an agent command can now read its own
DSH_HOME (including .credentials.yaml) and write anywhere in the container.
Hosts where a runner works should keep workspace-write or read-only.

- main-image docs: DSH_PERMISSION_MODE row plus a note on the runners and the
  fail-closed behavior
- example README: limits bullet and a troubleshooting row with the in-pod
  backend check and how to restore a confined mode
- architecture: tenant-isolation bullet extended
- skill runbook/troubleshooting: operation guidance; never grant CAP_SYS_ADMIN
  or seccompProfile: Unconfined to "fix" this without an explicit decision

Verified: make check (14 manifests), make lint, YAML assertions on both
templates, and the override expression at the pinned dsh-v0.2.1-rc.5 base
bundle (mode: !!js process.env.DSH_PERMISSION_MODE ?? 'workspace-write').

* fix(image): install bubblewrap for DSH's command sandbox

DSH's Linux sandbox runner chain is bubblewrap, then Landlock, and it fails
closed with SANDBOX_UNAVAILABLE when neither works. The image shipped neither,
so a sandboxed agent had no way to execute confined commands.

Add bubblewrap to the runtime stage. Debian ships it unsetuid (0.12.0), so it
still needs unprivileged user namespaces plus the namespace/mount syscalls: in a
hardened pod (all capabilities dropped, RuntimeDefault seccomp) it usually
cannot create one, and DSH then falls back to Landlock or, failing that, to the
DSH_PERMISSION_MODE=danger-full-access override the templates already set.
Where the host allows an unprivileged user namespace, the first-choice runner
now exists, so the confined modes can be restored.

Docs corrected to match: bubblewrap is no longer "absent"; the blocker is the
unsetuid binary's namespace requirement, not a missing CAP_SYS_ADMIN grant.

Verified: make check, make lint; the package exists in the pinned base
(node:24-trixie-slim, bubblewrap 0.12.0-1~deb13u1); a throwaway container with
the pod's flags (--user 1000 --cap-drop ALL --security-opt no-new-privileges)
still cannot create a namespace, which is why the override stays. No local image
build was run.

* ci(image): build and smoke-test the main image on PRs that touch it

The Dockerfile and the pins in versions.env could only be validated by a merge
to main or by running `make image` by hand; neither gives a PR-time signal.

Add ci-image.yaml: on pull requests that change Dockerfile, .dockerignore or
versions.env (and on manual dispatch) it builds linux/amd64 natively with the
pins sourced from versions.env - no registry login, no push - loads the image
locally and smoke-tests it:

- the packaged CLI runs and reports the pinned version
- the container is the non-root node user (1000) and owns /workspace + DSH_HOME
- the bubblewrap command-sandbox runner is present
- the OCI version label matches versions.env
- a container serves /favicon.svg (the same unauthenticated probe the shipped
  manifests use), with logs filtered so the launch token is never printed

Path-filtered like the other on-demand checks, so it is not a required status
check; the published multi-arch build stays in build-main.yaml and release.yaml.

Docs updated in the same change: AGENTS.md repository map and testing table,
docs/gitflow.md checks section, docs/main-image.md build section.

Verified: actionlint 1.7.12 and yamllint 1.38.0 (CI versions) clean over the
whole repository, plus make check and make lint.

---------

Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
2026-10-30 08:36:21 -03:00
SREDevOps.org Agent
367294b230 feat(example): persist the agent workspace on the PVC via subPath (#34)
/workspace - the DSH image's WORKDIR, where agents keep files - was plain
container storage, so every suspend/resume or pod replacement silently threw
agent work away while DSH state survived on the PVC.

Mount the same per-sandbox `dsh-home` PVC a second time at /workspace with
`subPath: workspace`. One PVC, no extra provisioning, and the subPath keeps
DSH state (sessions, config and .credentials.yaml at the volume root) out of
the agent-visible tree; mounting the volume root at /workspace would have
exposed the credential file.

The kubelet creates the missing subPath directory itself and chmods it to the
volume's fsGroup permissions, and the volume root's setgid bit makes the new
directory inherit gid 1000, so UID 1000 can write with no init container and
no root. The rare interrupted-kubelet case that leaves 0755 is documented with
a recovery procedure (kubernetes#142464) rather than papered over.

- e2e-kind: write a /workspace marker alongside the DSH_HOME marker, assert
  the mount is the dsh-home subPath, and verify both survive suspend/resume.
- Docs: README (overview, steps 4-6, troubleshooting row), architecture
  (PVC diagram, persistence, component table), main-image working-directory
  row, platform runbook PVC row.

Verified: make check (14 manifests), make lint, plus YAML assertions on the
two mounts and the unchanged volumeClaimTemplates.

Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
2026-10-30 08:29:39 -03:00
SREDevOps.org Agent
cbcc56f97e fix(gateway): answer client-aborted requests with 499, not a traceback (#32)
* fix(gateway): answer client-aborted requests with 499, not a traceback

A browser reload, back navigation or proxy timeout can abort a request while
its body is still arriving. Starlette then raises ClientDisconnect from
request.stream(); nothing caught it, so ServerErrorMiddleware logged a full
ERROR traceback per aborted request and buried real faults. The request never
reaches DSH (the body is buffered before the upstream send) and the peer can
no longer be answered, so this is not an error.

- Register a ClientDisconnect exception handler that answers 499
  (CLIENT_CLOSED_REQUEST). One handler covers the proxy and the body-reading
  control routes (signup, login). A request already streaming keeps its
  upstream status because the response has started.
- Make _release await-free, and reach it even when the upstream close is
  cancelled. CancelledError is a BaseException, so suppress(Exception) around
  upstream.aclose() skipped the release and stranded the in-flight slot,
  leaving that user's sandbox looking busy forever (the idle sweeper then
  never suspends it).
- Add raw-ASGI regression tests: mid-body abort on the proxy and on signup,
  mid-stream abort, a cancelled release, and a cancelled upstream close.
- Document the 499 semantics in the gateway README failure table and the
  architecture failure modes.

Verified: make check, make test (114 passed), make lint. Against the pinned
uvicorn 0.54 the abort reproduces the original traceback before the fix and
logs no traceback after it.

* chore(deps): bump gateway starlette to 1.10.0 and ruff to 0.17.2

Absorbs Dependabot PR #27 against current main: that branch predates the
dsh-v0.2.1-rc.5 changelog/release commits, so merging it as-is would have
reverted them.

starlette 1.10.0 reworks URL-encoded form parsing (fields are built in the
parser callbacks, and the last field is kept when the stream ends without an
empty chunk) and clears tracebacks of handled exceptions. The gateway reads
JSON and form bodies, so the existing suite exercises signup/login forms and
the 499 client-abort path against the new release. ruff 0.17.2 only adds
preview and fix-safety changes; the configured rule set reports nothing new.

Verified: make check, make test (114 passed), make lint, make gateway (image
builds on python:3.12-slim and imports starlette 1.10.0), plus a direct ASGI
abort probe under 1.10.0 (499, in-flight slot released, nothing forwarded).

---------

Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
2026-10-30 07:14:49 -03:00
dependabot[bot]
8c34f43789 chore: bump docker/login-action (#28)
Bumps the actions group with 1 update in the / directory: [docker/login-action](https://github.com/docker/login-action).


Updates `docker/login-action` from 4.6.0 to 4.7.0
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](dbcb813823...81c6f93d4b)

---
updated-dependencies:
- dependency-name: docker/login-action
  dependency-version: 4.7.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: actions
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-30 06:15:25 -03:00
Nicolás Georger
97c563296a Add continue-on-error to CHANGELOG PR step
Signed-off-by: Nicolás Georger <info@sredevops.org>
2026-10-30 06:14:27 -03:00
SREDevOps.org Agent
dcb5b436f6 ci(workflows): streamline CI and remove redundant builds (#31)
Streamline the CI and release workflows:

- ci.yaml is now PR-only (no push-to-main re-run) and keeps only lint and
  dependency-review; the gateway-image local build/smoke job is removed.
- Split the path-sensitive jobs into ci-python.yaml and ci-manifests.yaml,
  each using native on.pull_request.paths filtering so unrelated PRs skip them.
- Drop the duplicate check-manifests.py run from the lint job.
- release.yaml: skip the tests job steps for tags already on main (where PR CI
  already ran), keeping them only for release-branch tags.

Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
2026-10-30 05:48:18 -03:00
SREDevOps.org Agent
b57916d854 chore(changelog): refresh for dsh-v0.2.1-rc.5 (#30)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-10-30 05:07:57 -03:00
SREDevOps.org Agent
bed8e5675a chore(release): dsh-v0.2.1-rc.5 (#29)
Bump DSH_VERSION to dsh-v0.2.1-rc.5 and clear RELEASE_REVISION, making the release identity dsh-v0.2.1-rc.5. Update every derived manifest, image tag, and doc, and remove prose that described the previous identity's legacy-tag history.

Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
2026-10-30 04:54:37 -03:00
SREDevOps.org Agent
14c3fb840e feat(example): add optional failed-Pod recovery (#26)
agent-sandbox v1.0.8 adds spec.failedPodPolicy to Sandbox and
SandboxTemplate. The default, Retain, leaves a Failed Pod in place and
surfaces Finished=True; Recreate deletes it and starts a replacement with
the same Sandbox identity and PVCs.

Ship it as an opt-in toggle next to the other commented fields, with a
new section 5 in the optional-hardening doc: what it changes, how to
enable it for new, warm-pool and already-claimed Sandboxes (the policy is
stamped at creation), a failure drill, caveats and rollback. The base
walkthrough stays on Retain because a deterministic failure would
otherwise recreate forever, and because the templates use
restartPolicy OnFailure, so the policy only matters for terminal cases
such as node eviction.

Cross-references updated in the example README, the optional-manifests
index, the architecture failure-mode table and the platform skill's
troubleshooting table.

Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
2026-10-26 14:29:01 -03:00
SREDevOps.org Agent
96d3030a49 chore(deps): bump agent-sandbox to v1.0.8 (#25)
Update AGENT_SANDBOX_VERSION to v1.0.8 and the docs that quote the
control-plane pin (example prerequisites, architecture, versioning table,
minimal-single-tenant template comment, platform skill).

The v1.0.8 release adds only one CRD field, spec.failedPodPolicy
(Sandbox and SandboxTemplate, enum Retain|Recreate, default Retain), so
the existing manifests keep their behavior: Retain is the default and
neither the API version (v1beta1) nor any field we set changed. All five
custom resources still validate against the v1.0.8 CRD schemas.

Behavior changes in the controller itself (bounded MCP output, router
kustomization, creation-latency metric timing) do not affect the
platform because it uses neither the sandbox-router nor the MCP server.

Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
2026-10-26 14:17:28 -03:00
SREDevOps.org Agent
2b43d5dfd5 fix(ci): make release image digests unique and allow changelog PRs (#24)
The reusable image build ran once for the main image and once for the gateway in the same release run, but both used digests-<platform> artifact names, so the main merge downloaded a gateway digest and failed. Include a short hash of the full image name in the artifact name and download pattern.

The publish-release job also lacked pull-requests: write, so the generated CHANGELOG PR failed with 'Resource not accessible by integration'. Grant the job contents and pull-requests write.

Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
2026-10-26 10:34:21 -03:00
SREDevOps.org Agent
97abf75dfe chore(changelog): refresh for dsh-v0.2.1-rc.3-r4 (#23)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-10-26 10:27:02 -03:00
SREDevOps.org Agent
e2f1fe567a chore(release): dsh-v0.2.1-rc.3-r4 (#22)
Bump RELEASE_REVISION to r4 for the optional-hardening feature release and refresh every consuming manifest/doc. CHANGELOG.md is generated by the release workflow and is intentionally left untouched.

Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
2026-10-26 10:17:43 -03:00
SREDevOps.org Agent
3e9ba38fc1 feat(example): add optional gVisor, nodeSelector, NetworkPolicy and provider keys (#21)
Add opt-in hardening to the agents-as-a-service example:
- commented runtimeClassName/nodeSelector fields for sandboxes and Deployments
- optional sandbox NetworkPolicy plus exposure-variant policies
- optional dsh-provider-keys-extra Secret for extra provider keys
- docs/optional-hardening.md with verification, caveats and rollback

The base walkthrough stays portable; no gateway code, routes or auth changed.

Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
2026-10-26 10:03:30 -03:00
Nicolás Georger
bf24306421 feat(skills): add agent-sandbox platform skill (#20)
Add the project-local dsh-agent-sandbox-platform skill covering platform operations, the optional agent-sandbox MCP server, feature-extension recipes, and a read-only preflight. Document project skills in README, docs and AGENTS.md.

Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
2026-10-26 08:10:59 -03:00
Nicolás Georger
8681412be4 chore(changelog): refresh for dsh-v0.2.1-rc.3-r3 (#19)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-10-25 19:42:02 -03:00
Nicolás Georger
958cbe25a5 Refactor release workflow for environment variable sourcing
Updated the release workflow to source environment variables from versions.env and modified the tag comparison logic.

Signed-off-by: Nicolás Georger <info@sredevops.org>
2026-10-25 19:19:26 -03:00
Nicolás Georger
027f74f8e9 Refactor environment variable loading in build-gateway.yaml
Updated the environment variable loading method in the GitHub Actions workflow.

Signed-off-by: Nicolás Georger <info@sredevops.org>
2026-10-25 19:03:16 -03:00
Nicolás Georger
d627526e3e Refactor environment variable loading in build workflow
Update the script to read environment variables from versions.env using 'source' instead of a loop.

Signed-off-by: Nicolás Georger <info@sredevops.org>
2026-10-25 18:56:53 -03:00
Nicolás Georger
ecba8b96a6 Versioning update (#18)
Signed-off-by: Nicolás Georger <info@sredevops.org>
2026-10-25 18:46:09 -03:00
dependabot[bot]
6a8fe54abc chore: bump the actions group across 1 directory with 3 updates (#17)
Bumps the actions group with 3 updates in the / directory: [actions/setup-python](https://github.com/actions/setup-python), [actions/dependency-review-action](https://github.com/actions/dependency-review-action) and [amannn/action-semantic-pull-request](https://github.com/amannn/action-semantic-pull-request).


Updates `actions/setup-python` from 7.0.0 to 7.1.0
- [Release notes](https://github.com/actions/setup-python/releases)
- [Commits](5fda3b95a4...6d4fd22b20)

Updates `actions/dependency-review-action` from 4.9.0 to 5.0.0
- [Release notes](https://github.com/actions/dependency-review-action/releases)
- [Commits](2031cfc080...a1d282b36b)

Updates `amannn/action-semantic-pull-request` from 5 to 6
- [Release notes](https://github.com/amannn/action-semantic-pull-request/releases)
- [Changelog](https://github.com/amannn/action-semantic-pull-request/blob/main/CHANGELOG.md)
- [Commits](e32d7e603d...48f256284b)

---
updated-dependencies:
- dependency-name: actions/dependency-review-action
  dependency-version: 5.0.0
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: actions
- dependency-name: actions/setup-python
  dependency-version: 7.1.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: actions
- dependency-name: amannn/action-semantic-pull-request
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: actions
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-25 14:42:29 -03:00
Nicolás Georger
6876871110 Add commit message prefix for Dependabot updates
Signed-off-by: Nicolás Georger <info@sredevops.org>
2026-10-25 14:37:11 -03:00
Nicolás Georger
3b70f179ac feat(repo)!: repo reorganization (#16)
* feat(repo)!: restructure around the dsh-harness image and one example per implementation

- the root Dockerfile is the primary artifact; versions.env is the single source of truth
- merge gateway-api/dsh-demo into examples/dsh-agents-as-a-service (manifests/, exposure variants)
- move the single-tenant template into examples/minimal-single-tenant
- publish the gateway as a nested package under the repository image
- mirror the DSH ref for release identities (first release dsh-v0.2.1-rc.2-r2)
- CI: lint/python/manifests/gateway-image, reusable multi-arch build, tag-driven release, weekly e2e
- docs: README, docs/{main-image,examples,gitflow,versioning}, example READMEs, AGENTS.md, governance
- SSO is opt-in again; gateway pull policy and cookie defaults match the documented demo

BREAKING CHANGE: example manifest paths, the gateway image name, and the removal
of deploy/ and gateway-api/ affect existing deployments; see the migration table
in README.md.

* docs(changelog): generate initial changelog from history

* fix(scripts): make set-version update Dockerfile and sample revision pins

* build(hadolint): document intentional lint exceptions for the main image

---------

Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
2026-10-25 08:37:43 -03:00
Nicolás Georger
53e55fc851 Modify gateway.yaml for image policy and timeout values
Updated imagePullPolicy to always pull the latest image and increased timeout values for IDLE_TIMEOUT and WAKE_TIMEOUT.

Signed-off-by: Nicolás Georger <info@sredevops.org>
2026-10-25 04:41:00 -03:00
SREDevOps.org Agent
ed1398dfd1 feat(gateway): SSO landing auto-connects and offers recreate sandbox (#15)
After oauth2-proxy authentication, the gateway now renders a minimal SSO
landing (no signup/token forms) that auto-connects to the sandbox and offers
a single 'Recreate sandbox' action for a clean start.

- _sso_session: stop blocking on wait_ready (provisioning is non-blocking;
  the proxy path still awaits readiness), so the landing reports
  'Provisioning' instead of stalling for WAKE_TIMEOUT
- landing: SSO branch renders a minimal page (signed-in user + sandbox state
  + auto-connect meta-refresh), replacing the generic signup/token forms
- new POST /recreate: deletes the claim (cascade pod+PVC), waits out
  finalizers, re-provisions, and reconnects; CSRF-protected, SSO-only
- tests: 109 passing (SSO landing provision/connect/recreate cases)

Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
2026-10-25 04:34:13 -03:00
SREDevOps.org Agent
73a65d98d2 fix(gateway-api): correct oauth2-proxy cookie-secret generation (#14)
oauth2-proxy base64-decodes cookie_secret and requires it to decode to 16/24/32
bytes. Plain 'openssl rand -base64 32' yields 44 chars of standard base64
(padding + possibly '+/'), which it cannot decode and treats as 44 raw bytes,
failing with 'cookie_secret must be 16, 24, or 32 bytes ... but is 44 bytes'.

Use URL-safe, unpadded base64 instead (secrets.token_urlsafe(32) or the
openssl|tr variant).

Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
2026-10-25 04:12:56 -03:00
SREDevOps.org Agent
3f46279b39 feat(gateway-api): GitHub SSO via oauth2-proxy in front of dsh-gateway (#13)
Add oauth2-proxy (GitHub provider, org-restricted) as a reverse proxy in
front of the dsh-gateway, wired through the Traefik Gateway API. The gateway
inherits the authenticated GitHub username via X-Forwarded-User (SSO).

- gateway.py: opt-in AUTH_PROXY_HEADER trust, remote_user(), SSO bootstrap
  (auto-provision claim + mint/rotate token + session cookie), patch_token()
- 50-gateway.yaml: AUTH_PROXY_HEADER=X-Forwarded-User, COOKIE_SECURE=true
- gateway-api/dsh-demo: oauth2-proxy Deployment/Service + secret example,
  HTTPRoute backend -> oauth2-proxy:4180
- tests: 10 new SSO cases (106 passing)

Placeholders (dsh-demo.example.com, your-github-org) are substituted at
apply time.

Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
2026-10-25 03:43:32 -03:00
Nicolás Georger
2b47afa22b Dockerfile extra tools
Signed-off-by: Nicolás Georger <info@sredevops.org>
2026-10-25 01:39:03 -03:00
Nicolás Georger
6d97f6193b fix(Dockerfile): update runtime dependencies and enable corepack for pnpm 2026-10-21 23:16:12 -03:00
SREDevOps.org Agent
7d3faf07dd feat(gateway-api): expose dsh-gateway externally via Traefik Gateway API (#12)
Add Gateway and HTTPRoute manifests to route external traffic to the
dsh-gateway service (dsh-demo:8080).

- Gateway dsh-gateway: HTTPS listener with TLS termination against a
  dsh-demo-tls cert secret.
- HTTPRoute dsh-gateway: hostname -> dsh-gateway:8080.
- Example TLS secret manifest + README with apply/verify/rollback.

Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
2026-10-21 18:58:38 -03:00
96 changed files with 13420 additions and 2405 deletions

View File

@@ -1,149 +1,6 @@
# Logs
logs
*.log
npm-debug.log*
yarn-debug.log*
yarn-error.log*
lerna-debug.log*
# Diagnostic reports (https://nodejs.org/api/report.html)
report.[0-9]*.[0-9]*.[0-9]*.[0-9]*.json
# Runtime data
pids
*.pid
*.seed
*.pid.lock
# Directory for instrumented libs generated by jscoverage/JSCover
lib-cov
# Coverage directory used by tools like istanbul
coverage
*.lcov
# nyc test coverage
.nyc_output
# Grunt intermediate storage (https://gruntjs.com/creating-plugins#storing-task-files)
.grunt
# Bower dependency directory (https://bower.io/)
bower_components
# node-waf configuration
.lock-wscript
# Compiled binary addons (https://nodejs.org/api/addons.html)
build/Release
# Dependency directories
node_modules/
jspm_packages/
# Snowpack dependency directory (https://snowpack.dev/)
web_modules/
# TypeScript cache
*.tsbuildinfo
# Optional npm cache directory
.npm
# Optional eslint cache
.eslintcache
# Optional stylelint cache
.stylelintcache
# Optional REPL history
.node_repl_history
# Output of 'npm pack'
*.tgz
# Yarn Integrity file
.yarn-integrity
# dotenv environment variable files
.env
.env.*
!.env.example
# parcel-bundler cache (https://parceljs.org/)
.cache
.parcel-cache
# Next.js build output
.next
out
# Nuxt.js build / generate output
.nuxt
dist
.output
# Gatsby files
.cache/
# Comment in the public line in if your project uses Gatsby and not Next.js
# https://nextjs.org/blog/next-9-1#public-directory-support
# public
# vuepress build output
.vuepress/dist
# vuepress v2.x temp directory
.temp
# Sveltekit cache directory
.svelte-kit/
# vitepress build output
**/.vitepress/dist
# vitepress cache directory
**/.vitepress/cache
# Docusaurus cache and generated files
.docusaurus
# Serverless directories
.serverless/
# FuseBox cache
.fusebox/
# DynamoDB Local files
.dynamodb/
# Firebase cache directory
.firebase/
# TernJS port file
.tern-port
# Stores Visual Studio Code versions used for testing Visual Studio Code extensions
.vscode-test
# pnpm
.pnpm-store
# yarn v3
.pnp.*
.yarn/*
!.yarn/patches
!.yarn/plugins
!.yarn/releases
!.yarn/sdks
!.yarn/versions
# Vite files
vite.config.js.timestamp-*
vite.config.ts.timestamp-*
.vite/
# Astro generated types
.astro/
# Turbo Repo
.turbo
# The main image build clones DeepSeek Harness from git and copies nothing
# from this repository, so the build context is intentionally reduced to the
# Dockerfile alone. This keeps builds fast and prevents the rest of the repo
# (examples, docs, git history) from being sent to the builder.
*
!Dockerfile

View File

@@ -0,0 +1,157 @@
---
name: dsh-agent-sandbox-platform
description: >-
Operate and extend this repository's DeepSeek Harness-on-agent-sandbox platform:
warm pools, claims, sandbox suspend/resume, the gateway, and the optional
agent-sandbox MCP server, with version-aware safety and a read-only preflight.
whenToUse: >-
Use when working in deepseek-kubernetes-sandbox or with Kubernetes agent-sandbox:
inspect or provision sandboxes, debug warm pools or claims, use
mcp__*__create_sandbox/execute_command, connect the agent-sandbox MCP server, or
add a manifest, gateway route, exposure variant, example, or version bump here.
metadata:
author: sredevopsorg
version: "1.0.0"
---
# DSH agent-sandbox platform
This repository runs DeepSeek Harness Web (DSH) as one sandbox per user on
[agent-sandbox](https://github.com/kubernetes-sigs/agent-sandbox), with an
optional official MCP server for sandbox execution. For generic cluster work
load `kubernetes-kubectl-operations`; for profile MCP rows load
`cordis-plugin-development`.
## 0. Facts before action
1. Read `AGENTS.md`, `versions.env`, the example README/architecture. The
control-plane pin is `AGENT_SANDBOX_VERSION`; release identity is
`DSH_VERSION` + `RELEASE_REVISION`.
2. Establish context, user and namespace; never assume `dsh-demo`.
3. Run `scripts/platform-preflight.sh --namespace <ns>` (read-only).
4. If `mcp__*__create_sandbox`-style tools are absent, use `kubectl`; do not
install or enable a server on your own.
## 1. Non-negotiables
- **Read-only by default.** Change a cluster only when the user asked for that
exact change; show the command and rollback first.
- **Scope every command** with `--context` and `-n`. `--all`/`-A` mutations and
deleting namespaces, PVCs, PVs, CRDs, or cluster-scoped objects need an
explicit, named confirmation.
- **Secrets and logs stay private.** Never print Secret data, kubeconfigs, or
node logs. Pod logs can contain the DSH launch token.
- **The MCP server has no authentication.** It is a full-access management
endpoint; never expose it. A DSH profile row affects every session.
- **Clean up what you create.** Delete MCP-created sandboxes in the same
session; never leave orphaned claims running.
- **Bound output.** Use `--tail`, `--since`, selectors, JSONPath, and
head/tail excerpts instead of whole manifests or huge logs.
## 2. Choose the path
| Need | Path |
| --- | --- |
| Inspect or change claims, pools, sandboxes, PVCs, gateway | `kubectl` + [platform runbook](references/platform-runbook.md) |
| Run code or move files in a sandbox | agent-sandbox MCP, only for a runtime-capable template ([MCP reference](references/mcp-server.md)) |
| Add or change a repository feature | [repo extension guide](references/extending-the-repo.md) |
The MCP server cannot manage this example's DSH claims: its tools are scoped to
claims created by the same MCP session. Use `kubectl` for the platform.
## 3. Operate the platform
Resolve user -> claim -> sandbox -> pod:
```bash
SB=$(kubectl -n "$NS" get sandboxclaim "$CLAIM" -o jsonpath='{.status.sandbox.name}')
kubectl -n "$NS" get sandbox "$SB" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}{" "}{.status.conditions[?(@.type=="Ready")].reason}{"\n"}{.status.serviceFQDN}{"\n"}'
POD=$(kubectl -n "$NS" get sandbox "$SB" -o jsonpath='{.metadata.annotations.agents\.x-k8s\.io/pod-name}'); POD=${POD:-$SB}
```
- Read runtime state from the Sandbox, not the claim: `spec.operatingMode`,
`Suspended`, `Ready`. The claim tracks binding, not liveness.
- Suspend = patch `operatingMode: Suspended` (pod deleted, PVC + Service kept);
resume = `Running` + wait for `Ready`. The gateway wakes on connect.
- Warm-pool spares carry `agents.x-k8s.io/warm-pool-sandbox`; adopted sandboxes
lose it. A pool with no spares makes signup cold and slow.
- The DSH launch token appears once in pod logs; hand the user the `?token=` URL,
never the token in reports. Probe `/favicon.svg`.
- The idle sweeper only suspends claims the gateway created.
Full commands, security boundaries and failure matrix:
[platform runbook](references/platform-runbook.md).
## 4. Use the agent-sandbox MCP server within its limits
Verified from the upstream server:
- Tools: `create_sandbox`, `delete_sandbox`, `execute_command`, `upload_file`,
`download_file`, `list_files`, `file_exists`, `get_sandbox_status`; resource
`sandboxes://{namespace}`.
- **Session-scoped:** every tool and the resource filter claims by
`K8S_SANDBOX_SESSION_ID_LABEL_KEY=<MCP session id>`. It cannot read, exec in,
or delete DSH platform claims or claims from a prior/reconnected session.
- **Streamable HTTP only:** stdio has no MCP session id and fails. A DSH row must
use `transport: streamable-http`; health endpoints are `/healthz` and `/readyz`
at the root, the MCP endpoint is `/mcp`.
- **Runtime required:** connection types target a sandbox runtime on 8888, or
sandboxd on 8080/9090. The DSH template listens only on 3080, so
`execute_command` cannot work against it. A sandboxd sidecar executes in its
own container, not DSH's.
- **Timeouts:** `create_sandbox` waits up to 180 s, but the DSH MCP client's
default call timeout is 60 s; raise `toolCallTimeoutMs`.
`shutdown_after_seconds` defaults to 300 s (`null` disables it).
- **Output:** servers before v1.0.8 return unbounded `execute_command` output.
v1.0.8 adds `max_output_chars` (128-300,000, default 30,000) and truncation
metadata. The repository pins v1.0.8 (`AGENT_SANDBOX_VERSION`), so the
documented deployment is bounded; never pass the new argument to an older
server, and on one redirect verbose output and read a file instead.
Deployment, connection matrix and DSH row:
[MCP deployment](references/mcp-deployment.md). Limits and cleanup:
[MCP reference](references/mcp-server.md).
## 5. Extend the project
Read `docs/examples.md` and the testing table in `AGENTS.md` before editing.
- **Manifest step:** numbered `manifests/NN-*.yaml`, update the README step, run
`make check`; CI dry-runs it.
- **Gateway:** edit `gateway/gateway.py` and `gateway/test_gateway.py`, update
the gateway README API table and architecture doc, run `make test` and
`make lint`. Keep the HTTP/WebSocket contract stable.
- **Exposure variant:** one variant under `manifests/exposure/<variant>/`,
update the exposure README and trusted-host/cookie settings. Variants are
alternatives, never a stack.
- **New example:** follow `docs/examples.md`, add the index entry, keep scripts
executable/idempotent/self-cleaning, run `make check`.
- **Version bump:** use `make set-version`; never edit pins by hand. Update
prerequisites, run `make check` and the e2e on a cluster. Never mix a bump
with an unrelated feature.
- **MCP integration:** pin the build to the cluster's tag, least-privilege SA,
ClusterIP + auth proxy, raised DSH timeout, no secrets in the repo.
- **Optional hardening:** gVisor, `nodeSelector`, NetworkPolicy, extra
provider keys and failed-Pod recovery stay opt-in; the sandbox policy lives
under `manifests/optional/`. All of them are documented in
`examples/dsh-agents-as-a-service/docs/optional-hardening.md`.
- **Docs:** update the affected doc in the same change.
Details and review checklist:
[repo extension guide](references/extending-the-repo.md).
## 6. Report and clean up
State context/namespace, bounded evidence, conclusion, exact change, rollback,
and resources deleted. Disable any MCP row enabled only for the task. Do not
leave sandboxes, port-forwards, or temporary files behind.
## References
- [Platform runbook](references/platform-runbook.md)
- [MCP deployment](references/mcp-deployment.md)
- [MCP server reference](references/mcp-server.md)
- [Repo extension guide](references/extending-the-repo.md)
- [Troubleshooting](references/troubleshooting.md)
- Upstream MCP server README:
https://github.com/kubernetes-sigs/agent-sandbox/blob/main/clients/integrations/mcp-server/README.md

View File

@@ -0,0 +1,132 @@
# Extending the repository
Read `AGENTS.md`, `docs/examples.md`, and the example README before editing.
This repository is image-first: the Dockerfile is the primary artifact, every
implementation is a self-contained example under `examples/<name>/`, and
`versions.env` is the only source of pins. Documentation is part of the change.
## Invariants
- Never hard-code a version, image tag, or registry path; edit `versions.env`.
For the DSH release identity use `make set-version`, never ad-hoc `sed`.
- Never commit a secret. Secret-shaped manifests are `*.yaml.example` with
placeholders; keys are documented only in `.env.sample`.
- Manifests are numbered `NN-*.yaml` and apply in order with plain `kubectl`.
Exposure variants live under `manifests/exposure/<variant>/`, are documented
by one README, and are mutually exclusive.
- Keep the gateway HTTP/WebSocket contract stable unless the change is explicitly
about it.
- Do not edit `CHANGELOG.md` (generated by git-cliff) or rewrite git history.
- Keep diffs scoped; do not mix a version bump with an unrelated feature.
## Change matrix
| Change | Edit | Minimum verification |
| --- | --- | --- |
| Documentation only | `*.md`, `docs/` | `make check`; relative links resolve |
| Example manifest | `examples/<name>/manifests/NN-*.yaml`, README step | `make check`; server-side dry-run in CI |
| Optional hardening | `manifests/optional/`, exposure NPs, example `docs/optional-hardening.md` | `make check`; manual verification on a capable cluster |
| Gateway behavior/route | `gateway/gateway.py`, `gateway/test_gateway.py`, gateway README, architecture | `make test`, `make lint`; update tests and API docs |
| Exposure variant | `manifests/exposure/<variant>/`, exposure README, example README | `make check`; apply exactly one variant |
| New example | full `examples/<name>/` layout, `examples/README.md` | `make check`; CI dry-run; optional e2e |
| Version bump | `versions.env`, affected docs/manifests | `make check`; CI installs the pin; e2e on a cluster |
| MCP integration | deployment/RBAC, DSH row, docs | read-only `/readyz`; no secret in repo |
| Workflow | `.github/workflows/` | actionlint; SHA-pinned actions; no hard-coded versions |
## Recipes
### Add or change a resource step
1. Keep numbering stable; add `NN-*.yaml` at the end unless order matters.
2. Update the matching numbered step in the example README.
3. Keep cluster-specific state (node affinity, tuned replica counts) out of the
shipped manifest; document it as a local patch.
4. Run `make check`. CI additionally parses every manifest and runs a
server-side dry-run against the agent-sandbox CRDs.
### Change the gateway
The gateway is a single-file Starlette app. A change usually touches:
- `gateway/gateway.py` (route, auth, proxy, sweeper);
- `gateway/test_gateway.py` (unit tests; no cluster required);
- `gateway/README.md` (API table, configuration, auth semantics);
- `examples/dsh-agents-as-a-service/docs/architecture.md` when flows change;
- `examples/dsh-agents-as-a-service/README.md` when the walkthrough changes.
Run `make test` and `make lint`. Preserve session-cookie auth, same-origin rules
for cookie mutations and WebSocket upgrades, and the "waking"/`502`/`503`
semantics unless the change is explicitly about them. Do not expose a trusted
identity header unless the Service is reachable only through the proxy.
### Add an exposure variant
Create `manifests/exposure/<variant>/NN-*.yaml`, update
`manifests/exposure/README.md` and the example README, and state the trust
boundary. Variants are alternatives: never apply two paths to the same gateway.
Set `DSH_TRUSTED_HOSTS` in the template for the public host and
`COOKIE_SECURE=true` for HTTPS.
### Add an example
Follow `docs/examples.md`: `README.md`, `manifests/NN-*.yaml`, optional
`manifests/exposure/` with a README, optional components with Dockerfile +
README + tests, and executable, idempotent, self-cleaning scripts. Add the
literal `examples/<name>/` entry to `examples/README.md`; `make check` enforces
the layout. Publish component images under
`<registry>/<owner>/<repo>/<example>/<component>`.
### Bump a version
- **DSH release identity:** `make set-version DSH_VERSION=<ref>
[RELEASE_REVISION=rN]`, then `make check`, then build and smoke-test the image.
Tags are immutable; fix a bad release with `-r(N+1)`.
- **agent-sandbox:** edit `AGENT_SANDBOX_VERSION` in `versions.env`, update the
prerequisites in the example README, and note behavior changes (connection
types, MCP output limits, CRD fields). Install the new release YAML on a kind
cluster, then run `make check` and `make e2e`. Do not bump it inside an
unrelated PR.
### Integrate the agent-sandbox MCP server
Build from the cluster's pinned tag, give it a dedicated namespace and
least-privilege ServiceAccount, keep it ClusterIP behind an authenticating
proxy, and document how to reach `/readyz`. Add a DSH profile row only with user
approval; a row affects every session. Never commit its credentials or a
`cluster-admin` binding.
### Documentation
Update the doc whose behavior changed in the same commit: root README for the
front door, `docs/` for repository-wide contracts, the example README for that
walkthrough, the gateway README for API/config/auth. Skill docs live in
`.dsh/skills/`; keep `SKILL.md` under ~8 KB and put depth in `references/`.
`scripts/check-versions.sh` scans every `*.md` for stale `dsh-v…` tokens, so
refer to "the release identity in `versions.env`" rather than copying a tag.
## Verification and review
Run at least `make check`; add `make test` and `make lint` for gateway/script
changes. For manifests, use `kubectl apply --dry-run=server` against a cluster
with the CRDs. For behavior changes, run the example or `make e2e`.
Before reporting, review:
- [ ] Diff is scoped; no generated files or unrelated changes.
- [ ] Docs and examples match the new behavior.
- [ ] Pins come only from `versions.env`; no stale version tokens.
- [ ] No secrets or usable credentials; secret-shaped files are `.example`.
- [ ] Manifests parse; exposure variants remain mutually exclusive.
- [ ] Gateway tests and public contract unchanged or intentionally updated.
- [ ] Skills: frontmatter valid, body within budget, scripts shellcheck-clean.
- [ ] `CHANGELOG.md` untouched; branch/commit follow `docs/gitflow.md`.
## Safe follow-ups enabled by this skill
- A read-only preflight job or scheduled report using
`scripts/platform-preflight.sh`.
- An MCP readiness check in the example e2e (health endpoints only).
- A separate runtime warm pool for untrusted execution, leaving the DSH template
unchanged.
- A combined DSH + sandboxd image only with an explicit maintenance/exec design
and a documented trust boundary.

View File

@@ -0,0 +1,94 @@
# Deploying the agent-sandbox MCP server
Read [the MCP reference](mcp-server.md) first. This file covers the connection
matrix, a safe deployment shape, and the DSH profile row. The server is
unauthenticated; its service account is the security boundary.
## Connection matrix
| `K8S_SANDBOX_CONNECTION__TYPE` | Required env | Use when |
| --- | --- | --- |
| `gateway` | `GATEWAY_NAME`, `GATEWAY_NAMESPACE=default`, `GATEWAY_READY_TIMEOUT=180`, `SERVER_PORT=8888` | Gateway API + implementation exist |
| `direct` | `API_URL` (router URL), `SERVER_PORT=8888` | sandbox-router is deployed |
| `in-cluster` | `USE_POD_IP=false` (DNS) or `true` (`status.podIPs`), `SERVER_PORT=8888` | Bypass the router from inside the cluster |
| `sandboxd-in-cluster` (v1.0.8+) | `MODE=in-cluster-service` or `in-cluster-pod-ip`, `REST_PORT=8080`, `GRPC_PORT=9090` | Template runs sandboxd |
Also set `K8S_SANDBOX_PROBE_NAMESPACE` to a namespace the server's service
account can list claims in, and optionally
`K8S_SANDBOX_SESSION_ID_LABEL_KEY` (default
`mcp.k8s-agent-sandbox/session-id`).
For `sandboxd-in-cluster` service mode the template needs `spec.service: true`.
The default managed NetworkPolicy admits only sandbox-router, so add an ingress
rule for the MCP server on 8080/9090. Supplying `spec.networkPolicy` replaces the
secure defaults: also add the egress rules the sandbox needs, because an omitted
egress list denies all outbound traffic. RBAC for that mode is
`sandboxclaims` create/get/list/watch/delete and `sandboxes` get/list/watch; no
`pods/portforward`.
## Build and deploy (sketch)
No official image reference is documented in the upstream README; pin what you
build.
```bash
# AGENT_SANDBOX_VERSION comes from versions.env; never hard-code it.
tmp="$(mktemp -d)"; trap 'rm -rf "$tmp"' EXIT
git clone --depth 1 --branch "$AGENT_SANDBOX_VERSION" \
https://github.com/kubernetes-sigs/agent-sandbox.git "$tmp"
docker build -t "$MCP_IMAGE" "$tmp/clients/integrations/mcp-server"
```
Kubernetes shape:
- Dedicated namespace and ServiceAccount; a namespace-scoped Role granting only
`sandboxclaims` create/get/list/watch/delete and `sandboxes` get/list/watch.
No Secrets, no `pods/exec`, no cluster-admin.
- Deployment: one replica, non-root, `readOnlyRootFilesystem`, dropped
capabilities, `K8S_SANDBOX_CONNECTION__TYPE=in-cluster`,
`K8S_SANDBOX_PROBE_NAMESPACE`, resource limits.
- Probes: `GET /healthz` (liveness; never calls the API) and `GET /readyz`
(readiness; lists claims; use `periodSeconds: 15`, not aggressive polling).
- ClusterIP Service on `8000`. Add a NetworkPolicy that allows only the intended
client namespace/pod. Never expose it through Ingress/LoadBalancer without an
authenticating proxy. The service account is the real boundary.
## Connect it to DSH
Add a profile MCP row (through the Plugins page/`plugin_manager`; see
`cordis-plugin-development`). A row affects every session and needs approval.
```yaml
- insert:
- id: mcp-agent-sandbox
name: '@deepseek-ai/dsh-mcp-client'
config:
serverName: agent-sandbox
transport: streamable-http
url: http://agent-sandbox-mcp.example.svc.cluster.local:8000/mcp
toolCallTimeoutMs: 240000
failOnStartupError: false
```
- `serverName` must match `[A-Za-z0-9_-]{1,32}`; tools appear as
`mcp__<serverName>__<tool>`. Check the actual names in the tool list; do not
assume hyphens/underscores.
- Raise `toolCallTimeoutMs` above `sandbox_ready_timeout` with headroom; the DSH
default is 60 s and would cancel a 180 s creation.
- If the server is behind an authenticating proxy, pass its token via the row's
`headers` and an environment expression; never commit it.
- To reach a locally port-forwarded server, use a loopback URL; the MCP session
still needs streamable HTTP.
- Remove the row when it was enabled only for one task.
## Deployment safety checklist
- [ ] Build tag equals the cluster's `AGENT_SANDBOX_VERSION`.
- [ ] Dedicated namespace/SA; RBAC limited to claims and sandboxes.
- [ ] Service is ClusterIP; no public route without authentication.
- [ ] NetworkPolicy admits only the DSH client and needed sandbox traffic.
- [ ] `/readyz` succeeds with `K8S_SANDBOX_PROBE_NAMESPACE` set to a listable
namespace.
- [ ] No Secret data in manifests, logs, or the repository.
- [ ] Cleanup path documented: delete session-owned claims or wait for the
shutdown timeout.

View File

@@ -0,0 +1,107 @@
# agent-sandbox MCP server reference
Upstream: [clients/integrations/mcp-server](https://github.com/kubernetes-sigs/agent-sandbox/tree/main/clients/integrations/mcp-server).
It is a Python/FastAPI + FastMCP service (streamable HTTP on `:8000`, MCP
endpoint `/mcp`) that provisions sandboxes from a warm pool and runs commands or
file operations through a sandbox runtime. It ships with **no authentication**.
Build the server from the same tag as the cluster: read `AGENT_SANDBOX_VERSION`
in `versions.env`, and check the installed controller image. Do not build from
`main` against an older cluster; tool schemas and connection types differ.
## Tool surface
| Tool | Arguments (defaults) | Returns / notes |
| --- | --- | --- |
| `create_sandbox` | `warmpool`, `namespace`, `sandbox_ready_timeout=180`, `labels`, `shutdown_after_seconds=300` (`null` disables), `pod_labels`, `pod_annotations` | `sandbox_claim_name`; adds the MCP session label |
| `delete_sandbox` | `sandbox_claim_name`, `namespace` | Deletes the session-owned claim; PVC cascades |
| `execute_command` | `sandbox_claim_name`, `namespace`, `command`, `timeout=60` (max 600), `max_output_chars` only on v1.0.8+ (128-300,000, default 30,000) | `exit_code`, `stdout`, `stderr`; v1.0.8 adds `*_total_chars` and `*_truncated` |
| `upload_file` | `sandbox_claim_name`, `namespace`, `path`, `content`, `binary=false`, `timeout=60` | Bytes written; base64 when `binary=true` |
| `download_file` | `sandbox_claim_name`, `namespace`, `path`, `binary=false`, `timeout=60` | Content + bytes read |
| `list_files` | `sandbox_claim_name`, `namespace`, `path`, `timeout=60`, `max_entries=1000` (max 10,000) | Entries (`name`, `size`, `type`, `mod_time`), `total_entries`, `truncated` |
| `file_exists` | `sandbox_claim_name`, `namespace`, `path`, `timeout=60` | `exists`; a missing path is not an error |
| `get_sandbox_status` | `sandbox_claim_name`, `namespace` | `SandboxReady` / `SandboxNotReady` / `SandboxNotFound`, `ready`, `message` |
Resource: `sandboxes://{namespace}` lists sandboxes for the current MCP session.
## Session scoping is the most important behavior
Every tool, including `get_sandbox_status`, calls `get_sandbox()` ->
`ensure_session_owns()`, which lists sandboxes with the label selector
`<K8S_SANDBOX_SESSION_ID_LABEL_KEY>=<MCP session id>`. The resource uses the same
selector. Consequences:
- The server can only see/exec/delete claims **it created in the same MCP
session**. It cannot inspect or manage DSH platform claims created by the
gateway or `kubectl`, or claims from another session.
- On a DSH MCP reconnect, the MCP session changes; earlier claims become
invisible to the tools. They keep running until `shutdown_after_seconds`
expires, or until you remove them by label with a confirmed `kubectl delete`.
- A "Sandbox claim '<name>' is not found in namespace '<ns>'" error usually
means wrong namespace **or** the claim belongs to a different session.
- Use `kubectl` for global inventory; use the MCP only for sandboxes it created
in this task.
Session identity requires a session-providing transport. With stdio,
`ctx.session_id` is `None` and creation fails. Connect DSH over
`streamable-http`.
## Runtime prerequisite
The server talks to a sandbox runtime, not to arbitrary pods:
- `gateway`, `direct`, `in-cluster`: runtime HTTP on `SERVER_PORT=8888`
(sandbox-router API or direct pod runtime).
- `sandboxd-in-cluster` (v1.0.8+): sandboxd REST `8080`, gRPC `9090`.
The shipped DSH template runs only DSH Web on `3080`, so no connection type can
exec into it. A sandboxd **sidecar** executes commands in the sidecar's own
container, not in the DSH container; shared volumes share files, not binaries or
processes. To exec inside DSH, build a combined image (sandboxd injected into the
DSH image) or use `kubectl exec` with explicit RBAC and confirmation. For
untrusted code, deploy a separate runtime warm pool rather than modifying the
DSH template.
## Version differences
| Area | Before v1.0.8 | v1.0.8+ |
| --- | --- | --- |
| Connection types | `gateway`, `direct`, `in-cluster` | Adds `sandboxd-in-cluster` |
| `execute_command` output | Unbounded | Bounded by `max_output_chars`; four metadata fields |
| Failed pod recovery | `Retain` only | `spec.failedPodPolicy: Recreate` |
| Router install | `kubectl apply -f sandbox-router/deploy/` | `kubectl apply -k sandbox-router/deploy/`; projected token |
Never pass `max_output_chars` to a pre-v1.0.8 server. On older servers, redirect
verbose output to a file and `download_file` it, or pipe through `tail`/`head`.
The repository pins `AGENT_SANDBOX_VERSION=v1.0.8`, so the right-hand column is
what the documented platform and a same-tag MCP build provide; keep the
before-column for clusters still on v1.0.7.
## Safety and cleanup
- No auth: anyone reaching `:8000` gets every tool and the service account's
access. Keep it ClusterIP, authenticated, and namespace-scoped.
- Treat commands as untrusted code: they can read files and environment
variables inside the sandbox, including provider keys in a combined image.
- `delete_sandbox` cascades the claim, sandbox, and PVC: confirm the target and
namespace. Never delete a namespace or PVC as cleanup.
- Clean up in the same session that created the sandbox. If the session is gone,
list claims by the session label and delete them explicitly (requires write
RBAC); otherwise rely on the shutdown timeout.
## Troubleshooting
| Symptom | Cause / fix |
| --- | --- |
| Tools missing | Row not enabled, wrong URL, connection failed; check logs and `/readyz` |
| 404 on connect | URL is missing `/mcp` or points at the wrong port |
| `ctx.session_id is None` | Connected over stdio; use streamable HTTP |
| `Sandbox claim ... is not found` | Wrong namespace, or claim created by another MCP session |
| Creation times out at ~60 s | MCP session alive but DSH cancelled the call; raise `toolCallTimeoutMs` |
| `execute_command` connection error | No runtime on 8888/8080/9090, or wrong connection type; the DSH template is unsupported |
| Output flood | Pre-v1.0.8 server; bound the command output yourself |
| Empty sandbox list | The resource is session-scoped; use `kubectl` for global inventory |
Deployment, connection matrix, RBAC, and the DSH row:
[MCP deployment](mcp-deployment.md).

View File

@@ -0,0 +1,200 @@
# Platform runbook
Scope: the resource layer and gateway in
`examples/dsh-agents-as-a-service/`, one namespace, one DSH Web sandbox per
user. Read the example README and `AGENTS.md` first. All commands are read-only
unless a section is explicitly a change; confirm before changing anything.
## 1. Context, version, CRDs
```bash
kubectl config current-context
kubectl auth whoami
kubectl config view --minify -o jsonpath='{..namespace}{"\n"}'
kubectl get crd | grep -E 'agents\.x-k8s\.io'
kubectl -n agent-sandbox-system get deploy agent-sandbox-controller \
-o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
```
Compare the controller image tag with `AGENT_SANDBOX_VERSION` in
`versions.env`. The example needs the extension CRDs plus core `sandboxes`;
a missing CRD is a prerequisite failure, not an application bug.
## 2. Resource model
| Object | Role | Key fields |
| --- | --- | --- |
| `SandboxTemplate` | Blueprint; pinned image, probes, security context, PVC template | `envVarsInjectionPolicy`, `volumeClaimTemplatesPolicy`, `service`, `podTemplate` |
| `SandboxWarmPool` | Pre-warmed spares from one template | `spec.replicas`, `sandboxTemplateRef` |
| `SandboxClaim` | One tenant/user request; adopts a spare | `spec.warmPoolRef`, `additionalPodMetadata`, `status.sandbox.name` |
| `Sandbox` | Adopted or directly created pod + Service + PVC | `spec.operatingMode`, `status.conditions`, `status.serviceFQDN`, `status.podIPs` |
| `Pod` | The DSH Web process | `:3080`, PVC at `/home/node/.dsh` and `/workspace` (`subPath: workspace`) |
| `PersistentVolumeClaim` | Per-sandbox DSH state + agent workspace | `dsh-home-<sandbox>` |
| `Service` | Stable per-sandbox DNS | `<sandbox>`, `status.serviceFQDN` |
Delete a claim to cascade-delete its sandbox and PVC (account deletion). Never
delete a PVC or namespace without an explicit, named confirmation.
## 3. Read-only inspection
```bash
kubectl -n "$NS" get sandboxtemplates,sandboxwarmpools,sandboxclaims,sandboxes -o wide
kubectl -n "$NS" get pods,pvc,svc -o wide
kubectl -n "$NS" get events --sort-by=.lastTimestamp | tail -30
kubectl -n "$NS" get sandboxclaim "$CLAIM" -o yaml
kubectl -n "$NS" get sandbox "$SB" -o yaml
```
Bounded logs: `kubectl -n "$NS" logs "$POD" --tail=200` (add `--previous`
only after a restart).
## 4. Claim -> sandbox -> pod
```bash
SB=$(kubectl -n "$NS" get sandboxclaim "$CLAIM" -o jsonpath='{.status.sandbox.name}')
POD=$(kubectl -n "$NS" get sandbox "$SB" -o jsonpath='{.metadata.annotations.agents\.x-k8s\.io/pod-name}')
POD=${POD:-$SB}
kubectl -n "$NS" get sandbox "$SB" -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}{"\n"}{end}'
kubectl -n "$NS" get sandbox "$SB" -o jsonpath='{.spec.operatingMode}{" "}{.status.serviceFQDN}{" "}{.status.podIPs}{"\n"}'
```
Rules:
- Resolve user -> sandbox only through `claim.status.sandbox.name`; the sandbox
keeps its pool-generated name, not the claim name.
- A claim tracks binding; for runtime state read `Sandbox.spec.operatingMode`
and the `Suspended`/`Ready` conditions. A suspended sandbox is not serving
even when the claim still exists.
- A missing `claim.status.sandbox.name` means the claim did not bind: inspect
the Ready condition `reason` and events. The pod is normally named after the
sandbox; use the annotation when present.
## 5. Suspend, resume, wake
Suspend keeps the PVC and Service, deletes the pod:
```bash
kubectl -n "$NS" patch sandbox "$SB" --type merge \
-p '{"spec":{"operatingMode":"Suspended"}}'
kubectl -n "$NS" wait --for=delete "pod/$POD" --timeout=120s
kubectl -n "$NS" get pvc "dsh-home-$SB"
kubectl -n "$NS" get svc "$SB"
```
Resume and verify state:
```bash
kubectl -n "$NS" patch sandbox "$SB" --type merge \
-p '{"spec":{"operatingMode":"Running"}}'
kubectl -n "$NS" wait sandbox "$SB" --for=condition=Ready --timeout=180s
kubectl -n "$NS" exec "$SB" -- cat /home/node/.dsh/marker.txt
```
The gateway wakes a suspended sandbox on the first request. The idle sweeper
only suspends claims it created (`aaas.example.com/managed-by=dsh-gateway`);
hand-applied claims are never swept.
## 6. Warm pool
```bash
kubectl -n "$NS" get sandboxwarmpool "$POOL" -o yaml
kubectl -n "$NS" get sandboxes -l agents.x-k8s.io/warm-pool-sandbox -o wide
kubectl -n "$NS" get sandboxwarmpool "$POOL" -o jsonpath='{.spec.replicas}{"\n"}'
```
Adopted sandboxes lose the warm-pool label; count current spares against
`spec.replicas`. Zero spares or `spec.replicas: 0` means cold, slow signup.
Scale only when asked:
```bash
kubectl -n "$NS" patch sandboxwarmpool "$POOL" --type merge -p '{"spec":{"replicas":2}}'
```
## 7. DSH runtime specifics
- DSH Web listens on `3080` in-cluster. Port-forward to a free local port such
as `18081`, never `3080` if a local DSH already uses it.
- The launch token is printed once as `dsh web: http://<ip>:3080/?token=...`.
Treat it as a credential: do not paste it into reports; hand the user the URL.
- `/` returns `401` without the token. Probes must target `/favicon.svg`.
- `DSH_TRUSTED_HOSTS` must include any public host that a proxy forwards; DSH
fences on `Host`/`Origin`.
- State lives at `/home/node/.dsh`. DSH requires `.credentials.yaml` to be
`0600`; the template uses `fsGroupChangePolicy: OnRootMismatch`. If a resumed
sandbox crash-loops with a mode-660 error, suspend it, fix the file with a
throwaway pod, patch the policy, then resume.
- Agent files live at `/workspace`, the same PVC mounted with
`subPath: workspace`; it survives suspend/resume and dies with the PVC.
- The templates set `HOME=/workspace` (the image default is `/home/node`). DSH's
directory picker lists `$HOME` when it opens and has no start-path setting, so
this is what makes the folder explorer start on the persisted workspace; `~`
and `$HOME`-relative tool caches/config also resolve there. `DSH_HOME` remains
`/home/node/.dsh`, so state and credentials stay out of the workspace. Pod env
is stamped at sandbox creation: a sandbox created before the override keeps
the old `HOME` until it is recreated (or its `podTemplate` is patched and the
sandbox resumed). Verify with
`kubectl -n "$NS" exec "$POD" -- printenv HOME`.
- DSH confines bash with its own file sandbox (`bubblewrap`, else Landlock) and
fails closed with `SANDBOX_UNAVAILABLE` when neither works. The image ships
`bubblewrap`, but it is unsetuid and needs an unprivileged user namespace that
this hardened pod (all capabilities dropped, RuntimeDefault seccomp) does not
grant; that plus no Landlock runner is why the template sets
`DSH_PERMISSION_MODE=danger-full-access` and treats the pod as the boundary.
Never "fix" it by granting `CAP_SYS_ADMIN` or `seccompProfile: Unconfined`
without an explicit decision.
## 8. Gateway layer
```bash
kubectl -n "$NS" get deploy,svc,pods -l app=dsh-gateway -o wide
kubectl -n "$NS" get endpoints dsh-gateway
kubectl -n "$NS" logs deploy/dsh-gateway --tail=100
```
The gateway is a ClusterIP Service on `8080`; it proxies under `/u/{user}/` and
derives state (`Provisioning`, `Waking`, `Ready`, `Suspending`, `Suspended`).
It performs the DSH launch-token handshake server-side. With SSO enabled
(`AUTH_PROXY_HEADER`), the Service must be reachable only through the
authenticating proxy; never add a second path or port-forward it on a shared
host. A resource-only install has no gateway: that is a warning, not a failure.
## 9. Security boundaries
- Provider keys (`dsh-provider-keys` plus the optional
`dsh-provider-keys-extra`) are shared by all sandboxes; the example is not
tenant-isolated.
- gVisor, `nodeSelector`, the sandbox/exposure NetworkPolicies and extra
provider keys are opt-in, not active by default. A NetworkPolicy only helps
with a CNI that enforces it; see
[optional hardening](../../../../examples/dsh-agents-as-a-service/docs/optional-hardening.md).
- Secret data, kubeconfigs, and pod logs are sensitive; RBAC may allow Secret
access, policy does not. Keep gateway RBAC namespace-scoped; it needs pod logs
to find the launch token.
- TLS terminates outside the gateway; set `COOKIE_SECURE=true` for HTTPS.
## 10. Failure matrix
| Symptom | First evidence | Likely cause / fix |
| --- | --- | --- |
| Claim not Ready, no sandbox | Ready condition reason, events | Empty pool, image pull, quota, unschedulable |
| `EnvVarsInjectionRejected` | claim Ready reason | Claim carries env/volume templates; template policy rejects it |
| Pod Pending | `kubectl describe pod` | Requests/taints/affinity, unbound PVC |
| CrashLoopBackOff | logs, `--previous` | Startup failure, bad config, probe 401, fsGroup mode 660 |
| ImagePullBackOff | describe events | Wrong tag, private registry, missing pull secret |
| DSH 401 in browser | gateway logs | Launch-token handshake failed; check gateway RBAC `pods/log` |
| 502/503 through gateway | sandbox Ready, gateway logs | Wake timeout, DNS stale, token not in logs yet |
| Warm pool empty | sandboxes label count | Scale the pool or wait for replenishment |
| Suspended but expected running | `operatingMode` | Resume it; the gateway only wakes on a request |
| WS closes `1011` | gateway logs | DSH rejected the upstream Host/authority; check trusted hosts |
## 11. Cleanup and reporting
Delete only claims/sandboxes you created, by name:
```bash
kubectl -n "$NS" delete sandboxclaim "$CLAIM" --dry-run=server
kubectl -n "$NS" delete sandboxclaim "$CLAIM"
```
Report context/namespace, evidence, conclusion, change, rollback, and what was
deleted. Leave the namespace, pools, templates, and unrelated claims alone.

View File

@@ -0,0 +1,74 @@
# Troubleshooting
Start with evidence, not fixes: resolve context/namespace, read conditions and
events, then compare desired vs observed state. Keep output bounded. Never
"fix" an access error by broadening credentials.
## Platform
| Symptom | First evidence | Likely cause / fix |
| --- | --- | --- |
| `Forbidden` on every call | `kubectl auth whoami`, `kubectl auth can-i --list -n <ns>` | Wrong context or RBAC. Do not retry with broader credentials; ask for the right kubeconfig. |
| CRDs not found | `kubectl get crd | grep agents.x-k8s.io` | agent-sandbox extensions not installed. Install the pinned release YAML; that is a prerequisite, not a code change. |
| Claim created, no sandbox | `claim.status.conditions[?(@.type=="Ready")]`, events | Empty pool, image pull, quota, unschedulable; or policy rejection. |
| Claim rejected | Ready reason `EnvVarsInjectionRejected` | Claim carries `env`/`volumeClaimTemplates` while the template disallows injection. Remove them; do not bypass the pool. |
| User is suspended but the claim exists | `Sandbox.spec.operatingMode`, `Suspended` condition | Resume with `operatingMode: Running`; a gateway request would wake it. |
| Warm pool exhausted | `kubectl get sandboxes -l agents.x-k8s.io/warm-pool-sandbox` | Scale `spec.replicas` (with confirmation) or accept slow cold provisioning. |
| Pod `Pending` | `kubectl describe pod` | Requests/limits, taints, affinity, unbound PVC, storage class. |
| `ImagePullBackOff` | describe events | Wrong image/tag, private GHCR package, missing pull secret; on kind, load the image under the exact manifest ref. |
| `CrashLoopBackOff`, probe 401 | logs, `describe` | Probe path is `/`; use `/favicon.svg`. |
| Restart with mode-660 error | logs mention `.credentials.yaml` | kubelet fsGroup widened the file; suspend, `chmod 600` from a throwaway pod, patch `fsGroupChangePolicy: OnRootMismatch`, resume. |
| Every bash command fails with `SANDBOX_UNAVAILABLE` | DSH tool result names the requested mode and backends | DSH's own command sandbox has no usable runner: `bubblewrap` ships in the image but is unsetuid and needs an unprivileged user namespace this hardened pod does not grant, and no Landlock runner is available. The example template sets `DSH_PERMISSION_MODE=danger-full-access`; do not grant capabilities to restore it unless the operator explicitly accepts that downgrade. |
| Folder picker opens on `/home/node`, not the workspace | `kubectl exec <pod> -- printenv HOME` | The sandbox predates the template's `HOME=/workspace` and pod env is stamped at creation. Recreate the sandbox (PVC survives) or patch its `podTemplate` with `HOME`, then resume. Workspace dotfiles (`.cache`, `.config`, `.npm`, `.agents`) are the expected trade-off of the override, not a fault. |
| PVC `Pending` | `kubectl describe pvc`, StorageClass | No default provisioner, access mode, zone binding. |
| Launch token not found | `kubectl logs <pod> --tail=2000 | grep 'dsh web:'` | Pod not ready yet, or logs rotated. Do not guess tokens; wait for readiness. |
| Sandbox reports `Finished=True` | pod phase, `kubectl get events` | The backing Pod reached `Failed` (eviction, spot reclaim) and `failedPodPolicy` defaults to `Retain`. Delete the Pod to force a replacement, or set `Recreate` (v1.0.8+) on the Sandbox/template; the claim and PVCs survive either way. |
| Hand-applied sandbox disappeared | claim annotations, gateway logs | Idle sweeper only manages claims with `aaas.example.com/managed-by=dsh-gateway`; if it was managed, recreate/resume. |
| Gateway 401 | gateway logs, cookie flags | Missing/expired cookie, DSH handshake failed, or `COOKIE_SECURE=true` over HTTP. |
| Gateway 502/503 | sandbox Ready, gateway logs | Wake timeout, stale DNS, token not yet in logs; check `WAKE_TIMEOUT` and RBAC `pods/log`. |
| WebSocket closes 1011 | gateway logs, `Host`/Origin | DSH rejected the upstream authority; check `DSH_TRUSTED_HOSTS` and the gateway's pod-IP dial path. |
| WebSocket closes 4401 | cookie, Origin | Missing gateway auth or cross-origin upgrade; send a matching `Origin`. |
| SSO header ignored | gateway env | `AUTH_PROXY_HEADER` unset; setting it requires the proxy to be the only path to the ClusterIP Service. |
| Someone is treated as another user | service paths | Trust-boundary violation: the gateway is directly reachable. Remove the extra path and disable SSO until fixed. |
## Agent-sandbox MCP
| Symptom | First evidence | Likely cause / fix |
| --- | --- | --- |
| No `mcp__*__create_sandbox` tools | tool list, DSH logs | Server not deployed or row not enabled. Use `kubectl`; connect only with approval. |
| 404 on connect | curl `/mcp`, `/healthz` | Wrong path/port; MCP endpoint is `/mcp`, health is at the root. |
| `/readyz` 503 | response body, server logs | API unreachable or namespace `K8S_SANDBOX_PROBE_NAMESPACE` not listable; fix RBAC/config. |
| `ctx.session_id is None` | server error | Connected over stdio; use streamable HTTP. |
| `Sandbox claim ... is not found` | server error, `kubectl get claim` | Wrong namespace, or the claim belongs to another MCP session. List with `kubectl`; use the MCP only for its own claims. |
| Creation cancelled at ~60 s | MCP call log | DSH `toolCallTimeoutMs` is 60 s while creation waits up to 180 s; raise the timeout. |
| `execute_command` connection error | server logs, template ports | No runtime on 8888/8080/9090. The DSH template only listens on 3080; use a runtime template or a combined image. |
| Output truncated / flooded | tool result | Pre-v1.0.8 is unbounded; v1.0.8 bounds output. Redirect verbose output to a file and read/download it; never pass `max_output_chars` to an old server. |
| Orphaned claims after reconnect | `kubectl get claims -l mcp.k8s-agent-sandbox/session-id` | MCP sessions are transient; wait for `shutdown_after_seconds` or delete by label with explicit confirmation. |
| `Forbidden` from the MCP server | server logs | Its ServiceAccount lacks claim/sandbox verbs, or RBAC is in another namespace; fix the Role, not the kubeconfig. |
| Sandbox expired | `get_sandbox_status` -> `SandboxNotFound` | `shutdown_after_seconds` elapsed; create a new sandbox and keep the task inside the budget. |
## Repository checks and CI
| Symptom | Cause / fix |
| --- | --- |
| `check-versions: stale version reference in docs: dsh-v...` | A copied release tag in a Markdown file. Refer to the identity in `versions.env`; run `make set-version` only for a real bump. |
| `check-versions: hard-coded version outside versions.env` | A tag in `.github/workflows` or the Makefile; move it to `versions.env`. |
| `check-examples` layout error | Missing `README.md`/`manifests/`, stray YAML at the example root, missing exposure README, non-executable script, or missing `examples/README.md` entry. |
| Manifest dry-run fails in CI | Invalid YAML or CRD schema; parse locally and run `kubectl apply --dry-run=server` against the pinned CRDs. |
| `shellcheck` failure | Quote expansions, use arrays for kubectl args, avoid unused variables; run shellcheck locally. |
| Lychee link failure | A relative/absolute link does not resolve; the link checker runs offline on the listed docs. |
| PR-title check failure | Use Conventional Commits: `feat(scope): ...`, `fix(scope): ...`, `docs(scope): ...`. |
| Changelog conflict | Never edit `CHANGELOG.md`; it is generated from commits. |
## Do not
- Do not use `--force --grace-period=0` to clear a stuck resource before
understanding why it is stuck.
- Do not delete namespaces, PVCs, PVs, CRDs, or cluster-scoped objects as a
cleanup shortcut.
- Do not read Secret data, kubeconfigs, or node logs to "debug" a permission
problem, and do not paste pod logs containing the DSH launch token into a
report.
- Do not expose the unauthenticated MCP server or enable `AUTH_PROXY_HEADER`
while another path to the gateway exists.
- Do not retry a `Forbidden` with a stronger kubeconfig; fix the RBAC scope.

View File

@@ -0,0 +1,298 @@
#!/usr/bin/env bash
# Read-only preflight for the DSH agents-as-a-service platform.
#
# Checks the kubectl target, agent-sandbox CRDs/version, warm pools, claims,
# sandboxes, gateway and (optionally) an agent-sandbox MCP server. It never
# mutates the cluster, never reads Secret data, and never port-forwards.
#
# Usage:
# platform-preflight.sh [--context NAME] [--namespace NS] [--pool NAME]
# [--template NAME] [--mcp-url URL] [--help]
#
# Exit codes: 0 = no failures, 1 = at least one failure, 2 = usage error.
set -euo pipefail
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(git -C "$SCRIPT_DIR" rev-parse --show-toplevel 2>/dev/null || true)"
CTX=""
NS="${NAMESPACE:-dsh-demo}"
POOL=""
TEMPLATE=""
MCP_URL=""
PASS=0
WARN=0
FAIL=0
usage() {
cat <<'EOF'
Read-only preflight for the DSH agents-as-a-service platform.
Options:
--context NAME kubectl context (default: current context)
--namespace NS namespace to inspect (default: $NAMESPACE or dsh-demo)
--pool NAME check only this warm pool
--template NAME check only this SandboxTemplate
--mcp-url URL also check an MCP server's /healthz and /readyz
--help show this help
The script makes no changes and never reads Secret data.
EOF
}
while [ "$#" -gt 0 ]; do
case "$1" in
--context) CTX="${2:-}"; shift 2 ;;
--namespace) NS="${2:-}"; shift 2 ;;
--pool) POOL="${2:-}"; shift 2 ;;
--template) TEMPLATE="${2:-}"; shift 2 ;;
--mcp-url) MCP_URL="${2:-}"; shift 2 ;;
--help|-h) usage; exit 0 ;;
*) echo "unknown argument: $1" >&2; usage >&2; exit 2 ;;
esac
done
if [ -z "$NS" ]; then
echo "namespace must not be empty" >&2
exit 2
fi
section() { printf '\n== %s ==\n' "$1"; }
pass() { PASS=$((PASS + 1)); printf 'PASS %s\n' "$1"; }
warn() { WARN=$((WARN + 1)); printf 'WARN %s\n' "$1"; }
fail() { FAIL=$((FAIL + 1)); printf 'FAIL %s\n' "$1"; }
info() { printf 'INFO %s\n' "$1"; }
command -v kubectl >/dev/null 2>&1 || {
echo "FAIL kubectl is not installed" >&2
exit 1
}
if [ -n "$CTX" ]; then
if ! kubectl config get-contexts -o name 2>/dev/null | grep -Fxq "$CTX"; then
echo "FAIL context '$CTX' is not in the kubeconfig" >&2
exit 1
fi
KUBECTL=(kubectl --context "$CTX" --request-timeout=15s)
else
KUBECTL=(kubectl --request-timeout=15s)
fi
k() { "${KUBECTL[@]}" "$@"; }
section "Target"
if current="$(k config current-context 2>/dev/null)"; then
info "context: $current"
pass "kubeconfig is readable"
else
fail "cannot read the current kubectl context"
fi
if user="$(k auth whoami -o jsonpath='{.status.userInfo.username}' 2>/dev/null)" && [ -n "$user" ]; then
info "identity: $user"
else
fallback_user="$(k config view --minify -o jsonpath='{.users[0].name}' 2>/dev/null || true)"
info "identity: ${fallback_user:-unknown}"
fi
if k get namespace "$NS" -o name >/dev/null 2>&1; then
pass "namespace $NS exists"
else
fail "namespace $NS is missing or not readable"
fi
if [ -n "$REPO_ROOT" ] && [ -f "$REPO_ROOT/versions.env" ]; then
# shellcheck source=/dev/null
. "$REPO_ROOT/versions.env"
info "expected agent-sandbox pin: ${AGENT_SANDBOX_VERSION:-unknown}"
else
warn "versions.env not found; skipping the pin comparison"
fi
section "Capabilities (informational; never widen them)"
for check in "create sandboxclaims" "delete sandboxclaims" "create pods/exec"; do
verb="${check%% *}"
resource="${check#* }"
if [ "$(k auth can-i "$verb" "$resource" -n "$NS" 2>/dev/null || echo unknown)" = "yes" ]; then
info "can $verb $resource in $NS: yes"
else
info "can $verb $resource in $NS: no"
fi
done
if [ "$(k auth can-i get secrets -n "$NS" 2>/dev/null || echo unknown)" = "yes" ]; then
warn "RBAC allows reading Secrets; policy still forbids printing Secret data"
else
info "can get secrets in $NS: no"
fi
section "Agent Sandbox CRDs"
for crd in \
sandboxtemplates.extensions.agents.x-k8s.io \
sandboxwarmpools.extensions.agents.x-k8s.io \
sandboxclaims.extensions.agents.x-k8s.io \
sandboxes.agents.x-k8s.io; do
if k get crd "$crd" -o name >/dev/null 2>&1; then
pass "CRD $crd"
else
fail "CRD $crd is missing or not readable"
fi
done
section "Controller"
controller_ns="agent-sandbox-system"
controller_image="$(k -n "$controller_ns" get deployment agent-sandbox-controller \
-o jsonpath='{.spec.template.spec.containers[0].image}' 2>/dev/null || true)"
if [ -n "$controller_image" ]; then
info "controller image: $controller_image"
controller_tag="${controller_image##*:}"
if [ -n "${AGENT_SANDBOX_VERSION:-}" ] && [ "$controller_tag" != "$AGENT_SANDBOX_VERSION" ]; then
warn "controller tag '$controller_tag' differs from AGENT_SANDBOX_VERSION '$AGENT_SANDBOX_VERSION'"
else
pass "controller version matches versions.env"
fi
else
warn "controller deployment not found in $controller_ns; cannot verify the installed version"
fi
section "Templates"
templates="$(k -n "$NS" get sandboxtemplates \
-o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' 2>/dev/null || true)"
if [ -z "$templates" ]; then
warn "no SandboxTemplates in $NS"
fi
if [ -n "$TEMPLATE" ] && ! printf '%s\n' "$templates" | grep -Fxq "$TEMPLATE"; then
fail "SandboxTemplate $TEMPLATE not found in $NS"
fi
for tpl in $templates; do
[ -n "$TEMPLATE" ] && [ "$tpl" != "$TEMPLATE" ] && continue
desc="$(k -n "$NS" get sandboxtemplate "$tpl" \
-o jsonpath='{range .spec.podTemplate.spec.containers[*]}{.name}{":"}{range .ports[*]}{.containerPort}{","}{end}{";"}{end}' \
2>/dev/null || true)"
info "template $tpl containers: ${desc:-<none>}"
case "$desc" in
*sandboxd*|*:8080,*|*:9090,*)
pass "template $tpl exposes a sandboxd runtime (MCP sandboxd-in-cluster)" ;;
*:8888,*)
pass "template $tpl exposes a runtime HTTP port (MCP gateway/direct/in-cluster)" ;;
*:3080,*)
warn "template $tpl is DSH Web only (:3080); MCP execute_command cannot run against it" ;;
*)
warn "template $tpl exposes no known sandbox runtime port" ;;
esac
done
section "Warm pools"
pools="$(k -n "$NS" get sandboxwarmpools \
-o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' 2>/dev/null || true)"
if [ -z "$pools" ]; then
warn "no SandboxWarmPools in $NS; claims will provision cold"
fi
if [ -n "$POOL" ] && ! printf '%s\n' "$pools" | grep -Fxq "$POOL"; then
fail "SandboxWarmPool $POOL not found in $NS"
fi
for pool in $pools; do
[ -n "$POOL" ] && [ "$pool" != "$POOL" ] && continue
replicas="$(k -n "$NS" get sandboxwarmpool "$pool" -o jsonpath='{.spec.replicas}' 2>/dev/null || true)"
selector="$(k -n "$NS" get sandboxwarmpool "$pool" -o jsonpath='{.status.selector}' 2>/dev/null || true)"
if [ -n "$selector" ]; then
ready="$(k -n "$NS" get sandboxes -l "$selector" \
-o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' \
2>/dev/null | grep -c '^True$' || true)"
else
ready="$(k -n "$NS" get sandboxes -l agents.x-k8s.io/warm-pool-sandbox \
-o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' \
2>/dev/null | grep -c '^True$' || true)"
fi
replicas="${replicas:-0}"
info "pool $pool: $ready/$replicas warm spares Ready"
if [ "$replicas" -eq 0 ]; then
warn "pool $pool has spec.replicas=0; signup will cold-provision"
elif [ "$ready" -lt "$replicas" ]; then
warn "pool $pool is below its desired spare count"
else
pass "pool $pool is at capacity"
fi
done
section "Claims"
claims="$(k -n "$NS" get sandboxclaims \
-o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.conditions[?(@.type=="Ready")].status}{"\t"}{.status.conditions[?(@.type=="Ready")].reason}{"\t"}{.status.sandbox.name}{"\n"}{end}' \
2>/dev/null || true)"
if [ -z "$claims" ]; then
warn "no SandboxClaims in $NS"
else
while IFS=$'\t' read -r name ready reason sandbox; do
[ -n "$name" ] || continue
if [ "$ready" != "True" ]; then
warn "claim $name is not Ready (status='${ready:-unknown}', reason='${reason:-unknown}')"
elif [ -z "$sandbox" ]; then
warn "claim $name is Ready but has no bound sandbox"
else
pass "claim $name -> sandbox $sandbox"
fi
done <<< "$claims"
fi
section "Sandboxes"
sandboxes="$(k -n "$NS" get sandboxes \
-o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.operatingMode}{"\t"}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' \
2>/dev/null || true)"
if [ -z "$sandboxes" ]; then
warn "no Sandboxes in $NS"
else
while IFS=$'\t' read -r name mode ready; do
[ -n "$name" ] || continue
info "sandbox $name mode=${mode:-Running} ready=${ready:-unknown}"
done <<< "$sandboxes"
fi
section "Storage and Services"
pvc_count="$(k -n "$NS" get pvc -o name 2>/dev/null | wc -l | tr -d ' ')"
svc_count="$(k -n "$NS" get service -o name 2>/dev/null | wc -l | tr -d ' ')"
info "persistent volume claims: ${pvc_count:-0}"
info "services: ${svc_count:-0}"
[ "${pvc_count:-0}" -gt 0 ] || warn "no PVCs in $NS"
[ "${svc_count:-0}" -gt 0 ] || warn "no Services in $NS"
section "Gateway"
if k -n "$NS" get deployment dsh-gateway -o name >/dev/null 2>&1; then
pass "gateway deployment dsh-gateway exists"
if k -n "$NS" get service dsh-gateway -o name >/dev/null 2>&1; then
endpoints="$(k -n "$NS" get endpoints dsh-gateway \
-o jsonpath='{range .subsets[*].addresses[*]}{.ip}{" "}{end}' 2>/dev/null || true)"
if [ -n "$endpoints" ]; then
pass "gateway endpoints: $endpoints"
else
warn "gateway Service has no ready endpoints"
fi
else
warn "gateway Service dsh-gateway is missing"
fi
else
warn "no dsh-gateway deployment; this is a resource-layer-only install"
fi
section "MCP server"
if [ -n "$MCP_URL" ]; then
if ! command -v curl >/dev/null 2>&1; then
fail "curl is required for --mcp-url"
else
base="${MCP_URL%/}"
base="${base%/mcp}"
for endpoint in healthz readyz; do
body="$(curl -fsS --max-time 5 "$base/$endpoint" 2>/dev/null | head -c 300 || true)"
if [ -n "$body" ]; then
pass "MCP /$endpoint: $body"
else
fail "MCP /$endpoint is unreachable at $base/$endpoint"
fi
done
fi
else
info "no --mcp-url supplied; skipping server health checks"
fi
printf '\n== Summary ==\nPASS=%d WARN=%d FAIL=%d\n' "$PASS" "$WARN" "$FAIL"
if [ "$FAIL" -gt 0 ]; then
exit 1
fi
exit 0

15
.editorconfig Normal file
View File

@@ -0,0 +1,15 @@
root = true
[*]
charset = utf-8
end_of_line = lf
insert_final_newline = true
trim_trailing_whitespace = true
indent_style = space
indent_size = 2
[*.py]
indent_size = 4
[Makefile]
indent_style = tab

5
.entire/.gitignore vendored Normal file
View File

@@ -0,0 +1,5 @@
tmp/
settings.local.json
metadata/
logs/
redactors/local/

16
.entire/dsh-hooks.json Normal file
View File

@@ -0,0 +1,16 @@
{
"version": 1,
"agent": "dsh",
"bridge": "@sredevopsorg/dsh-entire-bridge",
"hook_names": [
"session-start",
"turn-start",
"turn-end",
"compaction",
"session-end",
"subagent-start",
"subagent-end"
],
"installed_at": "2026-11-02T06:02:27.816Z",
"local_dev": false
}

9
.entire/settings.json Normal file
View File

@@ -0,0 +1,9 @@
{
"enabled": true,
"telemetry": true,
"checkpoints": {
"primary": {
"type": "git-refs"
}
}
}

View File

@@ -1,29 +1,40 @@
# Copy to .env and fill in. Never commit a real key.
# Reference environment names for local runs. Nothing sources this file
# automatically: export the values or pass them with `docker run -e/--env-file`.
# Copy it to .env if you prefer --env-file. Never commit a real key.
#
# The version values below are documentation only: versions.env is the single
# source of truth, and scripts/check-versions.sh fails if they drift.
# --- DSH Web image (Dockerfile / build.sh) ---
# --- Main DSH Web image (root Dockerfile) ---
DEEPSEEK_API_KEY=
DSH_VERSION=dsh-v0.2.1-rc.2
# Optional additional provider keys consumed by DSH. For Docker, pass them with
# -e; the Kubernetes examples inject them through the optional
# dsh-provider-keys-extra Secret.
OPENROUTER_API_KEY=
GEMINI_API_KEY=
NVIDIA_API_KEY=
OPENAI_API_KEY=
ANTHROPIC_API_KEY=
GROQ_API_KEY=
XAI_API_KEY=
MISTRAL_API_KEY=
TOGETHER_API_KEY=
CEREBRAS_API_KEY=
AI_GATEWAY_API_KEY=
HF_TOKEN=
DSH_VERSION=dsh-v0.2.1-rc.5
# Full upstream commit SHA that DSH_VERSION resolves to; the main image build
# fails if the clone's HEAD differs (guards against a moved tag).
DSH_COMMIT=b830ec950faba0a46dda564be3e2d854afed94f1
RELEASE_REVISION=r6
PNPM_VERSION=11.28.5
# DSH Web port inside the container, and the local port published by
# build.sh / docker run. The gateway also uses it as the upstream port.
# Container port, and the host port published by "make run".
DSH_PORT=3080
# Comma/space separated extra Host allow-list entries for DSH (for an
# Ingress/public hostname). localhost and 127.0.0.1 are always trusted.
# Comma/space separated extra Host allow-list entries for DSH (for a proxy or
# public hostname). localhost and 127.0.0.1 are always trusted.
DSH_TRUSTED_HOSTS=
# --- Gateway (examples/dsh-agents-as-a-service) ---
GATEWAY_IMAGE=ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway
NAMESPACE=dsh-demo
POOL=dsh-pool
# Seconds a sandbox may stay idle before the sweeper suspends it.
IDLE_TIMEOUT=60
# Seconds a waking request may wait for the sandbox to become Ready.
WAKE_TIMEOUT=120
DSH_CONTAINER=dsh-web
DSH_LOG_TAIL_LINES=2000
COOKIE_NAME=dsh_gw_session
# Set to true only behind HTTPS; secure cookies are not sent over plain HTTP.
COOKIE_SECURE=false
COOKIE_MAX_AGE=604800
MAX_BODY_MB=32
HTTP_TIMEOUT=300
# The example gateway's runtime settings (NAMESPACE, POOL, IDLE_TIMEOUT,
# WAKE_TIMEOUT, COOKIE_*, AUTH_PROXY_HEADER, ...) are documented in
# examples/dsh-agents-as-a-service/gateway/README.md and set in
# examples/dsh-agents-as-a-service/manifests/50-gateway.yaml.

42
.github/ISSUE_TEMPLATE/bug_report.yml vendored Normal file
View File

@@ -0,0 +1,42 @@
name: Bug report
description: Report a reproducible problem with the image, an example or the tooling.
labels: ["bug"]
body:
- type: markdown
attributes:
value: |
Do not include API keys, tokens or other secrets. Report security
issues privately via SECURITY.md instead.
- type: textarea
id: what-happened
attributes:
label: What happened
description: The observed behavior and the expected behavior.
validations:
required: true
- type: textarea
id: reproduce
attributes:
label: Reproduction steps
description: Commands, manifests and versions. Redact secrets.
placeholder: |
make image
docker run ...
validations:
required: true
- type: input
id: image-tag
attributes:
label: Image or release
description: The tag or digest you deployed (from the manifest or release metadata).
- type: input
id: cluster
attributes:
label: Environment
description: Kubernetes distribution/version, kind version, agent-sandbox version.
- type: textarea
id: logs
attributes:
label: Relevant logs
description: Paste the smallest useful excerpt. Remove tokens and API keys.
render: shell

8
.github/ISSUE_TEMPLATE/config.yml vendored Normal file
View File

@@ -0,0 +1,8 @@
blank_issues_enabled: false
contact_links:
- name: Security vulnerability
url: https://github.com/sredevopsorg/deepseek-kubernetes-sandbox/security/advisories/new
about: Report security issues privately; do not open a public issue.
- name: Documentation
url: https://github.com/sredevopsorg/deepseek-kubernetes-sandbox/tree/main/docs
about: Read the repository docs before opening an issue.

View File

@@ -0,0 +1,35 @@
name: Feature request
description: Propose a new example, image capability or workflow improvement.
labels: ["enhancement"]
body:
- type: textarea
id: problem
attributes:
label: Problem or use case
description: What are you trying to do, and what blocks you today?
validations:
required: true
- type: textarea
id: proposal
attributes:
label: Proposed solution
description: Describe the shape you have in mind. For a new example, say which implementation it demonstrates.
validations:
required: true
- type: dropdown
id: area
attributes:
label: Area
options:
- Main DSH Web image
- Existing example
- New example
- CI / release automation
- Documentation
- Other
validations:
required: true
- type: textarea
id: alternatives
attributes:
label: Alternatives considered

36
.github/dependabot.yml vendored Normal file
View File

@@ -0,0 +1,36 @@
version: 2
updates:
- package-ecosystem: github-actions
directory: /
schedule:
interval: weekly
groups:
actions:
patterns: ["*"]
commit-message:
prefix: "chore: "
- package-ecosystem: pip
directory: /examples/dsh-agents-as-a-service/gateway
schedule:
interval: weekly
groups:
gateway-python:
patterns: ["*"]
commit-message:
prefix: "chore: "
- package-ecosystem: docker
directory: /
schedule:
interval: weekly
commit-message:
prefix: "chore: "
- package-ecosystem: docker
directory: /examples/dsh-agents-as-a-service/gateway
schedule:
interval: weekly
commit-message:
prefix: "chore: "

30
.github/pull_request_template.md vendored Normal file
View File

@@ -0,0 +1,30 @@
## Summary
<!-- What changes, and why? Link the issue: Closes #123 -->
## Type of change
- [ ] feat: new capability or example
- [ ] fix: bug fix
- [ ] docs: documentation only
- [ ] refactor: no behavior change
- [ ] build / ci: tooling, images, workflows
- [ ] chore: maintenance
## Breaking change
- [ ] Yes, and the migration is documented in the description
- [ ] No
## Checklist
- [ ] make check passes (versions, example layout, manifest YAML)
- [ ] make test / make lint pass when the change touches code (gateway unit tests, ruff, shellcheck; yamllint/actionlint where installed)
- [ ] Docs updated for the change (root README, docs/, example README)
- [ ] No secrets or usable credentials committed
- [ ] CHANGELOG.md was not edited by hand (generated by git-cliff)
- [ ] Version pins changed only through versions.env / make set-version
## Testing
<!-- Commands run, clusters used, results. Mention if kind e2e was skipped. -->

178
.github/workflows/_build-image.yaml vendored Normal file
View File

@@ -0,0 +1,178 @@
# Reusable multi-arch image build: one workflow, called by build-main,
# build-gateway and release. Builds natively per platform (no QEMU), pushes by
# digest, then merges the manifest list with SBOM and provenance attestations.
name: Reusable multi-arch image build
on:
workflow_call:
inputs:
image:
description: Fully qualified image name without a tag
required: true
type: string
version:
description: Version recorded in the OCI labels
required: true
type: string
context:
description: Docker build context
required: true
type: string
dockerfile:
description: Dockerfile path, relative to the context
required: false
default: Dockerfile
type: string
build-args:
description: Newline-separated build args
required: false
default: ""
type: string
tags:
description: Newline-separated fully qualified tags to publish
required: true
type: string
permissions:
contents: read
packages: write
id-token: write
attestations: write
jobs:
build:
name: Build ${{ matrix.platform }}
runs-on: ${{ matrix.platform == 'linux/amd64' && 'ubuntu-latest' || 'ubuntu-24.04-arm' }}
timeout-minutes: 120
strategy:
fail-fast: false
matrix:
platform:
- linux/amd64
- linux/arm64
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Prepare environment
env:
IMAGE: ${{ inputs.image }}
run: |
set -euo pipefail
pair=$(printf '%s' "${{ matrix.platform }}" | tr '/' '-')
# This reusable workflow runs once for the main image and once for the
# gateway in the same release run. Artifact names must include a
# unique token for the image: the repository path is a prefix of the
# gateway path, so a path-derived pattern would match both. Use a
# short hash of the full image name.
image_hash=$(printf '%s' "$IMAGE" | sha256sum | cut -c1-12)
{
echo "PLATFORM_PAIR=$pair"
echo "CACHE_SCOPE=$(printf '%s' "$IMAGE" | tr '/' '-')"
echo "ARTIFACT_PREFIX=$image_hash"
} >> "$GITHUB_ENV"
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
- name: Login to GitHub Container Registry
uses: docker/login-action@81c6f93d4bc991e510b342a00b419b3bb4efc5c6 # v4.7.0
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Build and push by digest
id: build
uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7.4.0
with:
context: ${{ inputs.context }}
file: ${{ inputs.context }}/${{ inputs.dockerfile }}
platforms: ${{ matrix.platform }}
build-args: ${{ inputs.build-args }}
labels: |
org.opencontainers.image.source=https://github.com/${{ github.repository }}
org.opencontainers.image.version=${{ inputs.version }}
org.opencontainers.image.revision=${{ github.sha }}
outputs: type=image,name=${{ inputs.image }},push-by-digest=true,name-canonical=true,push=true,oci-mediatypes=true
sbom: true
provenance: mode=max
cache-from: type=gha,scope=${{ env.CACHE_SCOPE }}-${{ matrix.platform }}
cache-to: type=gha,mode=max,scope=${{ env.CACHE_SCOPE }}-${{ matrix.platform }}
- name: Export digest
run: |
set -euo pipefail
mkdir -p /tmp/digests
digest="${{ steps.build.outputs.digest }}"
touch "/tmp/digests/${digest#sha256:}"
- name: Upload digest
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: digests-${{ env.ARTIFACT_PREFIX }}-${{ env.PLATFORM_PAIR }}
path: /tmp/digests/*
if-no-files-found: error
retention-days: 1
merge:
name: Merge manifest list
needs: build
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- name: Prepare environment
env:
IMAGE: ${{ inputs.image }}
run: |
set -euo pipefail
echo "ARTIFACT_PREFIX=$(printf '%s' "$IMAGE" | sha256sum | cut -c1-12)" >> "$GITHUB_ENV"
- name: Download digests
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with:
path: /tmp/digests
pattern: digests-${{ env.ARTIFACT_PREFIX }}-*
merge-multiple: true
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
- name: Login to GitHub Container Registry
uses: docker/login-action@81c6f93d4bc991e510b342a00b419b3bb4efc5c6 # v4.7.0
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Create and push manifest list
working-directory: /tmp/digests
env:
IMAGE: ${{ inputs.image }}
TAGS: ${{ inputs.tags }}
run: |
set -euo pipefail
args=()
while IFS= read -r tag; do
if [ -n "$tag" ]; then
args+=("-t" "$tag")
fi
done <<< "$TAGS"
refs=()
for digest in *; do
refs+=("$IMAGE@sha256:$digest")
done
docker buildx imagetools create "${args[@]}" "${refs[@]}"
- name: Inspect image
env:
IMAGE: ${{ inputs.image }}
TAGS: ${{ inputs.tags }}
run: |
set -euo pipefail
first=$(printf '%s\n' "$TAGS" | head -n1)
docker buildx imagetools inspect "$first"
echo "### Published ${{ inputs.image }}" >> "$GITHUB_STEP_SUMMARY"
printf '%s\n' "$TAGS" | sed 's/^/- /' >> "$GITHUB_STEP_SUMMARY"

68
.github/workflows/build-gateway.yaml vendored Normal file
View File

@@ -0,0 +1,68 @@
# Development channel for the example gateway image: a merge that touches the
# gateway publishes ghcr.io/<owner>/<repo>/dsh-agents-as-a-service/gateway:main.
name: Build gateway image
on:
workflow_dispatch:
push:
branches: [main]
paths:
- examples/dsh-agents-as-a-service/gateway/**
- versions.env
- .github/workflows/build-gateway.yaml
- .github/workflows/_build-image.yaml
# Write scopes are granted to the build job only; every other job stays read-only.
permissions:
contents: read
concurrency:
group: build-gateway-${{ github.ref }}
cancel-in-progress: true
jobs:
meta:
name: Read versions
runs-on: ubuntu-latest
timeout-minutes: 5
outputs:
identity: ${{ steps.meta.outputs.identity }}
gateway_image: ${{ steps.meta.outputs.gateway_image }}
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Read versions.env
id: meta
shell: bash
run: |
set -euo pipefail
set -a
# shellcheck disable=SC1091
source versions.env
set +a
identity="$DSH_VERSION"
[ -z "$RELEASE_REVISION" ] || identity="$DSH_VERSION-$RELEASE_REVISION"
{
echo "identity=$identity"
echo "gateway_image=$GATEWAY_IMAGE"
} >> "$GITHUB_OUTPUT"
build:
name: Build and push gateway image
needs: meta
uses: ./.github/workflows/_build-image.yaml
permissions:
contents: read
packages: write
id-token: write
attestations: write
with:
image: ${{ needs.meta.outputs.gateway_image }}
version: ${{ needs.meta.outputs.identity }}
context: examples/dsh-agents-as-a-service/gateway
dockerfile: Dockerfile
tags: |
${{ needs.meta.outputs.gateway_image }}:main

85
.github/workflows/build-main.yaml vendored Normal file
View File

@@ -0,0 +1,85 @@
# Development channel for the main DSH Web image: every merge to main that
# touches the image definition publishes ghcr.io/<owner>/<repo>:main.
# Releases are published by release.yaml with immutable version tags.
name: Build main image
on:
workflow_dispatch:
push:
branches: [main]
paths:
- Dockerfile
- versions.env
- .github/workflows/build-main.yaml
- .github/workflows/_build-image.yaml
# Write scopes are granted to the build job only; every other job stays read-only.
permissions:
contents: read
concurrency:
group: build-main-${{ github.ref }}
cancel-in-progress: true
jobs:
meta:
name: Read versions
runs-on: ubuntu-latest
timeout-minutes: 5
outputs:
identity: ${{ steps.meta.outputs.identity }}
dsh_version: ${{ steps.meta.outputs.dsh_version }}
dsh_commit: ${{ steps.meta.outputs.dsh_commit }}
pnpm_version: ${{ steps.meta.outputs.pnpm_version }}
release_revision: ${{ steps.meta.outputs.release_revision }}
ghcr_image: ${{ steps.meta.outputs.ghcr_image }}
gateway_image: ${{ steps.meta.outputs.gateway_image }}
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Read versions.env
id: meta
shell: bash
run: |
set -euo pipefail
set -a
# shellcheck disable=SC1091
source versions.env
set +a
identity="$DSH_VERSION"
[ -z "$RELEASE_REVISION" ] || identity="$DSH_VERSION-$RELEASE_REVISION"
{
echo "identity=$identity"
echo "dsh_version=$DSH_VERSION"
echo "dsh_commit=$DSH_COMMIT"
echo "pnpm_version=$PNPM_VERSION"
echo "release_revision=$RELEASE_REVISION"
echo "ghcr_image=$GHCR_IMAGE"
echo "gateway_image=$GATEWAY_IMAGE"
} >> "$GITHUB_OUTPUT"
build:
name: Build and push main image
needs: meta
uses: ./.github/workflows/_build-image.yaml
permissions:
contents: read
packages: write
id-token: write
attestations: write
with:
image: ${{ needs.meta.outputs.ghcr_image }}
version: ${{ needs.meta.outputs.identity }}
context: .
dockerfile: Dockerfile
build-args: |
DSH_VERSION=${{ needs.meta.outputs.dsh_version }}
DSH_COMMIT=${{ needs.meta.outputs.dsh_commit }}
RELEASE_REVISION=${{ needs.meta.outputs.release_revision }}
PNPM_VERSION=${{ needs.meta.outputs.pnpm_version }}
VCS_REF=${{ github.sha }}
tags: |
${{ needs.meta.outputs.ghcr_image }}:main

128
.github/workflows/ci-image.yaml vendored Normal file
View File

@@ -0,0 +1,128 @@
# Build-only check for the main DSH Web image: linux/amd64, no registry login,
# no push. It proves the Dockerfile still builds from the pinned versions.env and
# that the resulting image can actually run DSH Web. The published multi-arch
# build stays in build-main.yaml (merge to main) and release.yaml (tags).
#
# Like ci-python.yaml and ci-manifests.yaml this is path-filtered, so it is an
# on-demand check rather than a required status check.
name: CI image
on:
pull_request:
paths:
- Dockerfile
- .dockerignore
- versions.env
- .github/workflows/ci-image.yaml
workflow_dispatch:
permissions:
contents: read
concurrency:
group: ci-image-${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
build-amd64:
name: build-amd64
runs-on: ubuntu-latest
timeout-minutes: 120
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Read versions.env
id: meta
shell: bash
run: |
set -euo pipefail
set -a
# shellcheck disable=SC1091
source versions.env
set +a
identity="$DSH_VERSION"
[ -z "${RELEASE_REVISION:-}" ] || identity="$DSH_VERSION-$RELEASE_REVISION"
{
echo "identity=$identity"
echo "dsh_version=$DSH_VERSION"
echo "dsh_commit=$DSH_COMMIT"
echo "pnpm_version=$PNPM_VERSION"
echo "release_revision=${RELEASE_REVISION:-}"
} >> "$GITHUB_OUTPUT"
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
- name: Build linux/amd64 (load locally, never push)
uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7.4.0
with:
context: .
file: Dockerfile
platforms: linux/amd64
push: false
load: true
tags: dsh-web:ci
build-args: |
DSH_VERSION=${{ steps.meta.outputs.dsh_version }}
DSH_COMMIT=${{ steps.meta.outputs.dsh_commit }}
RELEASE_REVISION=${{ steps.meta.outputs.release_revision }}
PNPM_VERSION=${{ steps.meta.outputs.pnpm_version }}
VCS_REF=${{ github.sha }}
labels: |
org.opencontainers.image.source=https://github.com/${{ github.repository }}
org.opencontainers.image.version=${{ steps.meta.outputs.identity }}
org.opencontainers.image.revision=${{ github.sha }}
cache-from: type=gha,scope=ci-image-linux-amd64
cache-to: type=gha,mode=max,scope=ci-image-linux-amd64
- name: Smoke test the image
env:
IDENTITY: ${{ steps.meta.outputs.identity }}
DSH_VERSION: ${{ steps.meta.outputs.dsh_version }}
run: |
set -euo pipefail
image=dsh-web:ci
# The packaged CLI runs and reports the version pinned in versions.env.
version="$(docker run --rm --entrypoint node "$image" /app/lib/bin.js --version)"
printf 'DSH CLI: %s\n' "$version"
grep -q "${DSH_VERSION#dsh-v}" <<<"$version"
# The container runs as the non-root node user and its writable
# directories exist and are owned by that user.
[ "$(docker run --rm --entrypoint id "$image" -u)" = "1000" ]
docker run --rm --entrypoint sh "$image" -c \
'test -d /workspace && test -d "$DSH_HOME" && [ "$(stat -c %u /workspace)" = 1000 ]'
# The command-sandbox runner this image is expected to ship.
docker run --rm --entrypoint bwrap "$image" --version
# Labels come from the build args rather than a hard-coded value.
label="$(docker inspect -f '{{ index .Config.Labels "org.opencontainers.image.version" }}' "$image")"
printf 'image version label: %s\n' "$label"
[ "$label" = "$IDENTITY" ]
- name: Serve DSH Web
run: |
set -euo pipefail
docker run -d --name dsh-ci -p 18080:3080 \
-e DSH_TRUSTED_HOSTS=127.0.0.1:18080 dsh-web:ci >/dev/null
trap 'docker rm -f dsh-ci >/dev/null 2>&1 || true' EXIT
# /favicon.svg is unauthenticated and served by the web layer, which is
# why the shipped probes use it too.
for second in $(seq 1 60); do
if curl -fsS -o /dev/null http://127.0.0.1:18080/favicon.svg; then
printf 'DSH Web served /favicon.svg after ~%ss\n' "$((second * 2))"
exit 0
fi
sleep 2
done
echo "::error::DSH Web did not serve /favicon.svg within 120s"
# Pod logs can hold the DSH launch token: never dump them verbatim.
docker logs dsh-ci 2>&1 | grep -v 'token=' | tail -50 || true
exit 1

88
.github/workflows/ci-manifests.yaml vendored Normal file
View File

@@ -0,0 +1,88 @@
# Manifest shape validation and server-side dry-run. Split from ci.yaml so this
# job can use native on.pull_request.paths filtering and skip PRs that do not
# touch manifests, scripts or versions.env. The job name is `manifests`; because
# it is path-filtered it is an on-demand check, not a required branch-protection
# check (see docs/gitflow.md).
name: CI (manifests)
on:
pull_request:
paths:
- examples/**/manifests/**
- scripts/check-manifests.py
- versions.env
- .github/workflows/ci-manifests.yaml
workflow_dispatch:
permissions:
contents: read
concurrency:
group: ci-manifests-${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
manifests:
name: manifests
runs-on: ubuntu-latest
timeout-minutes: 25
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Create kind cluster
uses: helm/kind-action@1544676c07bb3570fcb70166beaa07c4528fd4aa # v1.15.1
with:
version: v0.33.0
cluster_name: dsh-ci
- name: Read versions.env
run: |
set -euo pipefail
while IFS= read -r line; do
case "$line" in ''|'#'*) continue ;; esac
echo "$line" >> "$GITHUB_ENV"
done < versions.env
- name: Install agent-sandbox and Gateway API CRDs
run: |
set -euo pipefail
# Download, verify the pinned digest from versions.env, then apply.
curl -fsSL --proto '=https' -o /tmp/agent-sandbox.yaml \
"https://github.com/kubernetes-sigs/agent-sandbox/releases/download/$AGENT_SANDBOX_VERSION/sandbox-with-extensions.yaml"
printf '%s %s\n' "$AGENT_SANDBOX_MANIFEST_SHA256" /tmp/agent-sandbox.yaml | sha256sum -c -
kubectl apply -f /tmp/agent-sandbox.yaml
kubectl wait --for=condition=Established crd/sandboxtemplates.extensions.agents.x-k8s.io --timeout=180s
kubectl wait --for=condition=Established crd/sandboxwarmpools.extensions.agents.x-k8s.io --timeout=180s
kubectl wait --for=condition=Established crd/sandboxclaims.extensions.agents.x-k8s.io --timeout=180s
kubectl wait --for=condition=Established crd/sandboxes.agents.x-k8s.io --timeout=180s
curl -fsSL --proto '=https' -o /tmp/gateway-api.yaml \
"https://github.com/kubernetes-sigs/gateway-api/releases/download/$GATEWAY_API_VERSION/standard-install.yaml"
printf '%s %s\n' "$GATEWAY_API_MANIFEST_SHA256" /tmp/gateway-api.yaml | sha256sum -c -
kubectl apply -f /tmp/gateway-api.yaml
kubectl wait --for=condition=Established crd/gateways.gateway.networking.k8s.io --timeout=180s
kubectl wait --for=condition=Established crd/httproutes.gateway.networking.k8s.io --timeout=180s
- name: Set up Python
uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
with:
python-version: "3.12"
- name: Install check dependencies
run: pip install "pyyaml==6.0.3"
- name: Validate manifest shape
run: python scripts/check-manifests.py
- name: Validate manifests (server dry-run)
run: |
set -euo pipefail
kubectl create namespace dsh-demo
shopt -s globstar nullglob
count=0
for file in examples/*/manifests/**/*.yaml; do
case "$file" in *.yaml.example) continue ;; esac
echo "== $file =="
kubectl apply --dry-run=server -f "$file"
count=$((count + 1))
done
if [ "$count" -eq 0 ]; then
echo "no manifests were validated" >&2
exit 1
fi
echo "validated $count manifests"

75
.github/workflows/ci-python.yaml vendored Normal file
View File

@@ -0,0 +1,75 @@
# Gateway unit tests and lint. Split from ci.yaml so this job can use native
# on.pull_request.paths filtering and skip PRs that do not touch the gateway.
# The job name is `python`; because it is path-filtered it is an on-demand check,
# not a required branch-protection check (see docs/gitflow.md).
name: CI (python)
on:
pull_request:
paths:
- examples/dsh-agents-as-a-service/gateway/**
- .github/workflows/ci-python.yaml
workflow_dispatch:
permissions:
contents: read
concurrency:
group: ci-python-${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
python:
name: python
runs-on: ubuntu-latest
timeout-minutes: 20
defaults:
run:
working-directory: examples/dsh-agents-as-a-service/gateway
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Set up Python
uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
with:
python-version: "3.12"
cache: pip
cache-dependency-path: |
examples/dsh-agents-as-a-service/gateway/requirements.txt
examples/dsh-agents-as-a-service/gateway/requirements-dev.txt
- name: Install dependencies
run: |
set -euo pipefail
python -m pip install --upgrade pip
# Both files are fully hash-pinned (uv pip compile --generate-hashes);
# pip refuses to install anything not listed with a matching hash.
pip install --require-hashes -r requirements.txt -r requirements-dev.txt
- name: Lint
run: |
set -euo pipefail
python -m ruff check .
python -m ruff format --check .
- name: Unit tests with coverage
run: >-
python -m pytest -q
--cov=. --cov-report=xml:coverage.xml --cov-report=term
--junitxml=junit.xml
- name: Coverage summary
run: |
set -euo pipefail
{
echo "### Gateway test coverage"
python -m coverage report
} >> "$GITHUB_STEP_SUMMARY"
- name: Upload test artefacts
if: always()
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: gateway-test-results
path: |
examples/dsh-agents-as-a-service/gateway/junit.xml
examples/dsh-agents-as-a-service/gateway/coverage.xml
if-no-files-found: warn
retention-days: 7

102
.github/workflows/ci.yaml vendored Normal file
View File

@@ -0,0 +1,102 @@
# Continuous integration: lint and dependency review. Job names are the required
# branch-protection checks documented in docs/gitflow.md. The gateway unit tests
# and the manifest dry-run live in ci-python.yaml / ci-manifests.yaml so each can
# use native on.pull_request.paths filtering and skip unrelated changes.
name: CI
on:
pull_request:
workflow_dispatch:
permissions:
contents: read
concurrency:
group: ci-${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
lint:
name: lint
runs-on: ubuntu-latest
timeout-minutes: 20
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Set up Python
uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
with:
python-version: "3.12"
- name: Install Python linters
run: |
set -euo pipefail
python -m pip install --upgrade pip
pip install "yamllint==1.38.0" "pyyaml==6.0.3"
- name: Install actionlint
env:
ACTIONLINT_VERSION: v1.7.12
# SHA-256 of actionlint_1.7.12_linux_amd64.tar.gz, published in the
# release's actionlint_1.7.12_checksums.txt. Update it with the version.
ACTIONLINT_SHA256: 8aca8db96f1b94770f1b0d72b6dddcb1ebb8123cb3712530b08cc387b349a3d8
run: |
set -euo pipefail
version="${ACTIONLINT_VERSION#v}"
asset="actionlint_${version}_linux_amd64.tar.gz"
curl -fsSL --proto '=https' -o "/tmp/$asset" \
"https://github.com/rhysd/actionlint/releases/download/$ACTIONLINT_VERSION/$asset"
printf '%s %s\n' "$ACTIONLINT_SHA256" "/tmp/$asset" | sha256sum -c -
tar -xzf "/tmp/$asset" -C /tmp actionlint
install -m 0755 /tmp/actionlint /usr/local/bin/actionlint
actionlint -version
- name: Install hadolint
env:
HADOLINT_VERSION: v2.12.0
# SHA-256 of hadolint-Linux-x86_64, published next to the download.
HADOLINT_SHA256: 56de6d5e5ec427e17b74fa48d51271c7fc0d61244bf5c90e828aab8362d55010
run: |
set -euo pipefail
curl -fsSL --proto '=https' -o /tmp/hadolint \
"https://github.com/hadolint/hadolint/releases/download/$HADOLINT_VERSION/hadolint-Linux-x86_64"
printf '%s %s\n' "$HADOLINT_SHA256" /tmp/hadolint | sha256sum -c -
install -m 0755 /tmp/hadolint /usr/local/bin/hadolint
hadolint --version
- name: Repository checks
run: |
set -euo pipefail
./scripts/check-versions.sh
./scripts/check-examples.sh
- name: Shellcheck
run: shellcheck -x build.sh scripts/*.sh examples/*/scripts/*.sh
- name: Actionlint
run: actionlint -color
- name: Yamllint
run: yamllint .
- name: Hadolint
run: |
set -euo pipefail
hadolint Dockerfile
hadolint examples/dsh-agents-as-a-service/gateway/Dockerfile
- name: Documentation links
uses: lycheeverse/lychee-action@e7477775783ea5526144ba13e8db5eec57747ce8 # v2
with:
args: >-
--offline --no-progress --verbose
./README.md ./CONTRIBUTING.md ./SECURITY.md ./AGENTS.md ./docs ./examples
dependency-review:
name: dependency-review
if: github.event_name == 'pull_request'
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
contents: read
pull-requests: write
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Dependency review
uses: actions/dependency-review-action@a1d282b36b6f3519aa1f3fc636f609c47dddb294 # v4

View File

@@ -57,7 +57,9 @@ jobs:
# your codebase is analyzed, see https://docs.github.com/en/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/codeql-code-scanning-for-compiled-languages
steps:
- name: Checkout repository
uses: actions/checkout@v7
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
# Add any setup steps before running the `github/codeql-action/init` action.
# This includes steps like installing compilers or runtimes (`actions/setup-node`
@@ -67,7 +69,7 @@ jobs:
# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL
uses: github/codeql-action/init@v4
uses: github/codeql-action/init@9b3254085494b55a0f7c4c2b3d2d52cc712ce370 # v4
with:
languages: ${{ matrix.language }}
build-mode: ${{ matrix.build-mode }}
@@ -96,6 +98,6 @@ jobs:
exit 1
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v4
uses: github/codeql-action/analyze@9b3254085494b55a0f7c4c2b3d2d52cc712ce370 # v4
with:
category: "/language:${{matrix.language}}"

83
.github/workflows/e2e-kind.yaml vendored Normal file
View File

@@ -0,0 +1,83 @@
# Weekly end-to-end run of the agents-as-a-service example against a real kind
# cluster with agent-sandbox installed. Scheduled (not PR-blocking) so upstream
# controller churn does not block merges; also available on demand.
name: E2E (kind)
on:
schedule:
- cron: "0 6 * * 1"
workflow_dispatch:
permissions:
contents: read
concurrency:
group: e2e-kind
cancel-in-progress: false
jobs:
e2e:
name: Example walkthrough on kind
runs-on: ubuntu-latest
timeout-minutes: 45
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Create kind cluster
uses: helm/kind-action@1544676c07bb3570fcb70166beaa07c4528fd4aa # v1.15.1
with:
version: v0.33.0
cluster_name: dsh-e2e
- name: Read versions.env
run: |
set -euo pipefail
while IFS= read -r line; do
case "$line" in ''|'#'*) continue ;; esac
echo "$line" >> "$GITHUB_ENV"
done < versions.env
- name: Install agent-sandbox
run: |
set -euo pipefail
curl -fsSL --proto '=https' -o /tmp/agent-sandbox.yaml \
"https://github.com/kubernetes-sigs/agent-sandbox/releases/download/$AGENT_SANDBOX_VERSION/sandbox-with-extensions.yaml"
printf '%s %s\n' "$AGENT_SANDBOX_MANIFEST_SHA256" /tmp/agent-sandbox.yaml | sha256sum -c -
kubectl apply -f /tmp/agent-sandbox.yaml
kubectl -n agent-sandbox-system rollout status deployment --timeout=180s
kubectl wait --for=condition=Established crd/sandboxtemplates.extensions.agents.x-k8s.io --timeout=180s
kubectl wait --for=condition=Established crd/sandboxwarmpools.extensions.agents.x-k8s.io --timeout=180s
kubectl wait --for=condition=Established crd/sandboxclaims.extensions.agents.x-k8s.io --timeout=180s
kubectl wait --for=condition=Established crd/sandboxes.agents.x-k8s.io --timeout=180s
- name: Set up Python (WebSocket checks)
uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
with:
python-version: "3.12"
- name: Install WebSocket client
run: pip install "websockets==17.3"
- name: Run the example walkthrough
env:
KIND_CLUSTER: dsh-e2e
run: examples/dsh-agents-as-a-service/scripts/e2e-kind.sh
- name: Collect cluster state
if: always()
run: |
set -euo pipefail
{
echo "== resources =="
kubectl get all,pvc -n dsh-demo -o wide 2>&1 || true
echo "== events =="
kubectl get events -n dsh-demo --sort-by=.lastTimestamp 2>&1 || true
echo "== gateway logs =="
# Redact any token material before this file becomes an artifact.
kubectl -n dsh-demo logs deploy/dsh-gateway --tail=200 2>&1 \
| sed -E 's/(token=)[^&"[:space:]]+/\1[REDACTED]/g' || true
} > /tmp/e2e-cluster.txt
- name: Upload cluster state
if: always()
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: e2e-cluster-state
path: /tmp/e2e-cluster.txt
if-no-files-found: warn
retention-days: 7

View File

@@ -1,277 +0,0 @@
# This workflow builds the multi-arch gateway image (Starlette ASGI, served by
# uvicorn on :8080) and pushes it to GHCR as
# ghcr.io/<owner>/<repo>:gateway. The DSH Web image is built by multi-build.yaml.
# It uses Docker Buildx with native AMD64 and ARM64 runners, without QEMU.
name: Gateway Build multi arch Docker Image with separate Github Runners
on:
workflow_dispatch:
push:
branches:
- main
paths:
- 'examples/dsh-agents-as-a-service/gateway/**'
- '.github/workflows/multi-build-gateway.yaml'
env:
# ghcr.io/<owner>/<repo>; this workflow always publishes the `:gateway` tag.
GHCR_IMAGE: ghcr.io/${{ github.repository }}
permissions:
# Global permissions for the workflow, which can be overridden at the job level
contents: read
concurrency:
# This concurrency group ensures that only one job in the group runs at a time.
# If a new job is triggered, the previous one will be canceled.
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
# The build job builds the Docker image for each platform specified in the matrix.
build:
strategy:
fail-fast: false
matrix:
platform:
- linux/amd64
- linux/arm64
# The matrix includes two platforms: linux/amd64 and linux/arm64.
# The build job will run for each platform in the matrix.
permissions:
# Permissions for the build job, which can be overridden at the step level
# The permissions are set to allow the job to write to the GitHub Container Registry (GHCR) and read from the repository.
attestations: write
actions: read
checks: write
contents: write
deployments: none
id-token: write
issues: read
discussions: read
packages: write
pages: none
pull-requests: read
repository-projects: read
security-events: read
statuses: read
runs-on: ${{ matrix.platform == 'linux/amd64' && 'ubuntu-latest' || matrix.platform == 'linux/arm64' && 'ubuntu-24.04-arm' }}
# The job runs on different runners based on the platform.
# For linux/amd64, it runs on the latest Ubuntu runner.
# For linux/arm64, it runs on an Ubuntu 24.04 ARM runner.
# The runner is selected based on the platform specified in the matrix.
name: Build Docker image for ${{ matrix.platform }}
steps:
-
name: Prepare environment for current platform
# This step sets up the environment for the current platform being built.
# It replaces the '/' character in the platform name with '-' and sets it as an environment variable.
# This is useful for naming artifacts and other resources that cannot contain '/'.
# The environment variable PLATFORMS_PAIR will be used later in the workflow.
id: prepare
run: |
platform=${{ matrix.platform }}
echo "PLATFORM_PAIR=${platform//\//-}" >> $GITHUB_ENV
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
# This step checks out the code from the repository.
# It uses the actions/checkout action to clone the repository into the runner's workspace.
- name: Docker meta default
# This step generates metadata for the Docker image.
# It uses the docker/metadata-action to create metadata based on the repository information.
# The metadata includes information such as the image name, tags, and labels.
# The metadata will be used later in the workflow to build and push the Docker image.
id: meta
uses: docker/metadata-action@dc802804100637a589fabce1cb79ff13a1411302 # v6.2.0
with:
images: ${{ env.GHCR_IMAGE }}
tags: |
type=raw,value=gateway
- name: Set up Docker Context for Buildx
# This step sets up a Docker context for Buildx.
# It creates a new context named "builders" that will be used for building the Docker image.
# The context allows Buildx to use the Docker daemon for building images.
id: buildx-context
run: |
docker context create builders
- name: Set up Docker Buildx
# This step sets up Docker Buildx, which is a Docker CLI plugin for extended build capabilities with BuildKit.
# It uses the docker/setup-buildx-action to configure Buildx with the specified context and platforms.
# The platforms are specified in the matrix and will be used for building the Docker image.
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
with:
endpoint: builders
platforms: ${{ matrix.platform }}
- name: Login to GitHub Container Registry
# This step logs in to the GitHub Container Registry (GHCR) using the docker/login-action.
# It uses the GitHub actor's username and the GITHUB_TOKEN secret for authentication.
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Build and push by digest
# This step builds and pushes the Docker image using Buildx.
# It uses the docker/build-push-action to build the image with the specified context and platforms.
# The image is built with the labels and annotations generated in the previous steps.
# The outputs are configured to push the image by digest, which allows for better caching and versioning.
# The cache-from and cache-to options are used to enable caching for the build process.
# The cache is stored in GitHub Actions cache and is scoped to the repository, branch, and platform.
id: build
uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7.4.0
env:
DOCKER_BUILDKIT: 1
with:
context: ./examples/dsh-agents-as-a-service/gateway/
platforms: ${{ matrix.platform }}
labels: ${{ steps.meta.outputs.labels }}
annotations: ${{ steps.meta.outputs.annotations }}
outputs: type=image,name=${{ env.GHCR_IMAGE }},push-by-digest=true,name-canonical=true,push=true,oci-mediatypes=true
cache-from: type=gha,scope=${{ github.repository }}-${{ github.ref_name }}-${{ matrix.platform }}
cache-to: type=gha,scope=${{ github.repository }}-${{ github.ref_name }}-${{ matrix.platform }}
- name: Export digest
# This step exports the digest of the built image to a file.
# It creates a directory in /tmp/digests and saves the digest of the image to a file.
# The digest is obtained from the output of the build step.
# The digest is used to uniquely identify the built image and can be used for further processing or verification.
run: |
mkdir -p /tmp/digests
digest="${{ steps.build.outputs.digest }}"
touch "/tmp/digests/${digest#sha256:}"
- name: Upload digest
# This step uploads the digest file to the GitHub Actions artifact storage.
# It uses the actions/upload-artifact action to upload the file created in the previous step.
# The artifact is named digests-${{ env.PLATFORM_PAIR }}, where PLATFORM_PAIR is the platform name with '/' replaced by '-'.
# The artifact is retained for 1 day, and if no files are found, it will throw an error.
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: digests-${{ env.PLATFORM_PAIR }}
path: /tmp/digests/*
if-no-files-found: error
retention-days: 1
merge:
# This job merges the Docker manifests for the different platforms built in the previous job.
name: Merge Docker manifests
runs-on: ubuntu-latest
permissions:
attestations: write
actions: read
checks: read
contents: read
deployments: none
id-token: write
issues: read
discussions: read
packages: write
pages: none
pull-requests: read
repository-projects: read
security-events: read
statuses: read
needs:
- build
# This job depends on the build job to complete before it starts.
# It ensures that the Docker images for all platforms are built before merging the manifests.
steps:
- name: Download digests
# This step downloads the digest files uploaded in the build job.
# It uses the actions/download-artifact action to download the artifacts with the pattern digests-*.
# The downloaded files are merged into the /tmp/digests directory.
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with:
path: /tmp/digests
pattern: digests-*
merge-multiple: true
- name: Docker meta
# This step generates metadata for the Docker image.
# It uses the docker/metadata-action to create metadata based on the repository information.
# The metadata includes information such as the image name, tags, and labels.
id: meta
uses: docker/metadata-action@dc802804100637a589fabce1cb79ff13a1411302 # v6.2.0
with:
images: ${{ env.GHCR_IMAGE }}
annotations: |
type=org.opencontainers.image.description,value=${{ github.event.repository.description || 'No description provided' }}
tags: |
type=raw,value=gateway
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
# This step sets up Docker Buildx, which is a Docker CLI plugin for extended build capabilities with BuildKit.
with:
driver-opts: |
network=host
- name: Login to GitHub Container Registry
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
# This step logs in to the GitHub Container Registry (GHCR) using the docker/login-action.
# It uses the GitHub actor's username and the GITHUB_TOKEN secret for authentication.
# The login is necessary to push the merged manifest list to GHCR.
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Get execution timestamp with RFC3339 format
# This step gets the current execution timestamp in RFC3339 format.
# It uses the date command to get the current UTC time and formats it as a string.
# The timestamp is used for annotating the Docker manifest list.
id: timestamp
run: |
echo "timestamp=$(date -u +"%Y-%m-%dT%H:%M:%SZ")" >> $GITHUB_OUTPUT
- name: Create manifest list and pushs
# This step creates a manifest list for the Docker images built for different platforms.
# It uses the docker buildx imagetools create command to create the manifest list.
# The manifest list is annotated with metadata such as description, creation timestamp, and source URL.
# The annotations are obtained from the metadata generated in the previous steps.
# The manifest list is pushed to the GitHub Container Registry (GHCR) with the specified tags.
working-directory: /tmp/digests
id: manifest-annotate
continue-on-error: true
run: |
docker buildx imagetools create \
$(jq -cr '.tags | map("-t " + .) | join(" ")' <<< "$DOCKER_METADATA_OUTPUT_JSON") \
--annotation='index:org.opencontainers.image.description=${{ github.event.repository.description }}' \
--annotation='index:org.opencontainers.image.created=${{ steps.timestamp.outputs.timestamp }}' \
--annotation='index:org.opencontainers.image.url=${{ github.event.repository.url }}' \
--annotation='index:org.opencontainers.image.source=${{ github.event.repository.url }}' \
$(printf '${{ env.GHCR_IMAGE }}@sha256:%s ' *)
- name: Create manifest list and push without annotations
# This step creates a manifest list for the Docker images built for different platforms.
# It uses the docker buildx imagetools create command to create the manifest list.
# The manifest list is created without annotations if the previous step fails.
# The manifest list is pushed to the GitHub Container Registry (GHCR) with the specified tags.
if: steps.manifest-annotate.outcome == 'failure'
working-directory: /tmp/digests
run: |
docker buildx imagetools create $(jq -cr '.tags | map("-t " + .) | join(" ")' <<< "$DOCKER_METADATA_OUTPUT_JSON") \
$(printf '${{ env.GHCR_IMAGE }}@sha256:%s ' *)
- name: Inspect image
# This step inspects the created manifest list to verify its contents.
# It uses the docker buildx imagetools inspect command to display information about the manifest list.
# The inspection output will show the platforms and tags associated with the manifest list.
id: inspect
run: |
docker buildx imagetools inspect '${{ env.GHCR_IMAGE }}:${{ steps.meta.outputs.version }}'

View File

@@ -1,295 +0,0 @@
# This workflow builds a multi-arch Docker image using GitHub Actions and separated Github Runners with native support for ARM64 and AMD64 architectures, without using QEMU emulation.
# It uses Docker Buildx to build and push the image to GitHub Container Registry (GHCR).
name: Build multi arch Docker Image with separate Github Runners
on:
workflow_dispatch:
inputs:
dsh_version:
description: "DSH git ref to build (tag or branch)"
required: true
default: dsh-v0.2.1-rc.2
type: string
pnpm_version:
description: "pnpm version used to build DSH"
required: true
default: "11.28.5"
type: string
push:
branches:
- main
paths:
- 'Dockerfile'
- '.github/workflows/multi-build.yaml'
env:
# The name of the Docker image to be built and pushed to GHCR
# The image name is derived from the GitHub repository name and the GitHub Container Registry (GHCR) URL.
# The image name will be in the format: ghcr.io/<owner>/<repo>
GHCR_IMAGE: ghcr.io/${{ github.repository }}
# workflow_dispatch inputs win; pushes to main fall back to the pinned
# defaults. Both values flow into the Docker build args and the image tags.
DSH_VERSION: ${{ inputs.dsh_version || 'dsh-v0.2.1-rc.2' }}
PNPM_VERSION: ${{ inputs.pnpm_version || '11.28.5' }}
permissions:
# Global permissions for the workflow, which can be overridden at the job level
contents: read
concurrency:
# This concurrency group ensures that only one job in the group runs at a time.
# If a new job is triggered, the previous one will be canceled.
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
# The build job builds the Docker image for each platform specified in the matrix.
build:
strategy:
fail-fast: false
matrix:
platform:
- linux/amd64
- linux/arm64
# The matrix includes two platforms: linux/amd64 and linux/arm64.
# The build job will run for each platform in the matrix.
permissions:
# Permissions for the build job, which can be overridden at the step level
# The permissions are set to allow the job to write to the GitHub Container Registry (GHCR) and read from the repository.
attestations: write
actions: read
checks: write
contents: write
deployments: none
id-token: write
issues: read
discussions: read
packages: write
pages: none
pull-requests: read
repository-projects: read
security-events: read
statuses: read
runs-on: ${{ matrix.platform == 'linux/amd64' && 'ubuntu-latest' || matrix.platform == 'linux/arm64' && 'ubuntu-24.04-arm' }}
# The job runs on different runners based on the platform.
# For linux/amd64, it runs on the latest Ubuntu runner.
# For linux/arm64, it runs on an Ubuntu 24.04 ARM runner.
# The runner is selected based on the platform specified in the matrix.
name: Build Docker image for ${{ matrix.platform }}
steps:
-
name: Prepare environment for current platform
# This step sets up the environment for the current platform being built.
# It replaces the '/' character in the platform name with '-' and sets it as an environment variable.
# This is useful for naming artifacts and other resources that cannot contain '/'.
# The environment variable PLATFORMS_PAIR will be used later in the workflow.
id: prepare
run: |
platform=${{ matrix.platform }}
echo "PLATFORM_PAIR=${platform//\//-}" >> $GITHUB_ENV
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
# This step checks out the code from the repository.
# It uses the actions/checkout action to clone the repository into the runner's workspace.
- name: Docker meta default
# This step generates metadata for the Docker image.
# It uses the docker/metadata-action to create metadata based on the repository information.
# The metadata includes information such as the image name, tags, and labels.
# The metadata will be used later in the workflow to build and push the Docker image.
id: meta
uses: docker/metadata-action@dc802804100637a589fabce1cb79ff13a1411302 # v6.2.0
with:
images: ${{ env.GHCR_IMAGE }}
- name: Set up Docker Context for Buildx
# This step sets up a Docker context for Buildx.
# It creates a new context named "builders" that will be used for building the Docker image.
# The context allows Buildx to use the Docker daemon for building images.
id: buildx-context
run: |
docker context create builders
- name: Set up Docker Buildx
# This step sets up Docker Buildx, which is a Docker CLI plugin for extended build capabilities with BuildKit.
# It uses the docker/setup-buildx-action to configure Buildx with the specified context and platforms.
# The platforms are specified in the matrix and will be used for building the Docker image.
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
with:
endpoint: builders
platforms: ${{ matrix.platform }}
- name: Login to GitHub Container Registry
# This step logs in to the GitHub Container Registry (GHCR) using the docker/login-action.
# It uses the GitHub actor's username and the GITHUB_TOKEN secret for authentication.
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Build and push by digest
# This step builds and pushes the Docker image using Buildx.
# It uses the docker/build-push-action to build the image with the specified context and platforms.
# The image is built with the labels and annotations generated in the previous steps.
# The outputs are configured to push the image by digest, which allows for better caching and versioning.
# The cache-from and cache-to options are used to enable caching for the build process.
# The cache is stored in GitHub Actions cache and is scoped to the repository, branch, and platform.
id: build
uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7.4.0
env:
DOCKER_BUILDKIT: 1
DSH_PORT: 3080
with:
context: .
platforms: ${{ matrix.platform }}
labels: ${{ steps.meta.outputs.labels }}
build-args: |
DSH_VERSION=${{ env.DSH_VERSION }}
PNPM_VERSION=${{ env.PNPM_VERSION }}
annotations: ${{ steps.meta.outputs.annotations }}
outputs: type=image,name=${{ env.GHCR_IMAGE }},push-by-digest=true,name-canonical=true,push=true,oci-mediatypes=true
cache-from: type=gha,scope=${{ github.repository }}-${{ github.ref_name }}-${{ matrix.platform }}
cache-to: type=gha,scope=${{ github.repository }}-${{ github.ref_name }}-${{ matrix.platform }}
- name: Export digest
# This step exports the digest of the built image to a file.
# It creates a directory in /tmp/digests and saves the digest of the image to a file.
# The digest is obtained from the output of the build step.
# The digest is used to uniquely identify the built image and can be used for further processing or verification.
run: |
mkdir -p /tmp/digests
digest="${{ steps.build.outputs.digest }}"
touch "/tmp/digests/${digest#sha256:}"
- name: Upload digest
# This step uploads the digest file to the GitHub Actions artifact storage.
# It uses the actions/upload-artifact action to upload the file created in the previous step.
# The artifact is named digests-${{ env.PLATFORM_PAIR }}, where PLATFORM_PAIR is the platform name with '/' replaced by '-'.
# The artifact is retained for 1 day, and if no files are found, it will throw an error.
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: digests-${{ env.PLATFORM_PAIR }}
path: /tmp/digests/*
if-no-files-found: error
retention-days: 1
merge:
# This job merges the Docker manifests for the different platforms built in the previous job.
name: Merge Docker manifests
runs-on: ubuntu-latest
permissions:
attestations: write
actions: read
checks: read
contents: read
deployments: none
id-token: write
issues: read
discussions: read
packages: write
pages: none
pull-requests: read
repository-projects: read
security-events: read
statuses: read
needs:
- build
# This job depends on the build job to complete before it starts.
# It ensures that the Docker images for all platforms are built before merging the manifests.
steps:
- name: Download digests
# This step downloads the digest files uploaded in the build job.
# It uses the actions/download-artifact action to download the artifacts with the pattern digests-*.
# The downloaded files are merged into the /tmp/digests directory.
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with:
path: /tmp/digests
pattern: digests-*
merge-multiple: true
- name: Docker meta
# This step generates metadata for the Docker image.
# It uses the docker/metadata-action to create metadata based on the repository information.
# The metadata includes information such as the image name, tags, and labels.
id: meta
uses: docker/metadata-action@dc802804100637a589fabce1cb79ff13a1411302 # v6.2.0
with:
images: ${{ env.GHCR_IMAGE }}
annotations: |
type=org.opencontainers.image.description,value=${{ github.event.repository.description || 'No description provided' }}
tags: |
type=raw,value=main,enable=${{ github.ref_name == 'main' }}
type=raw,value=latest,enable=${{ github.ref_name == 'main' }}
type=raw,value=${{ env.DSH_VERSION }},enable=${{ github.ref_name == 'main' || github.event_name == 'workflow_dispatch' }}
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
# This step sets up Docker Buildx, which is a Docker CLI plugin for extended build capabilities with BuildKit.
with:
driver-opts: |
network=host
- name: Login to GitHub Container Registry
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
# This step logs in to the GitHub Container Registry (GHCR) using the docker/login-action.
# It uses the GitHub actor's username and the GITHUB_TOKEN secret for authentication.
# The login is necessary to push the merged manifest list to GHCR.
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Get execution timestamp with RFC3339 format
# This step gets the current execution timestamp in RFC3339 format.
# It uses the date command to get the current UTC time and formats it as a string.
# The timestamp is used for annotating the Docker manifest list.
id: timestamp
run: |
echo "timestamp=$(date -u +"%Y-%m-%dT%H:%M:%SZ")" >> $GITHUB_OUTPUT
- name: Create manifest list and pushs
# This step creates a manifest list for the Docker images built for different platforms.
# It uses the docker buildx imagetools create command to create the manifest list.
# The manifest list is annotated with metadata such as description, creation timestamp, and source URL.
# The annotations are obtained from the metadata generated in the previous steps.
# The manifest list is pushed to the GitHub Container Registry (GHCR) with the specified tags.
working-directory: /tmp/digests
id: manifest-annotate
continue-on-error: true
run: |
docker buildx imagetools create \
$(jq -cr '.tags | map("-t " + .) | join(" ")' <<< "$DOCKER_METADATA_OUTPUT_JSON") \
--annotation='index:org.opencontainers.image.description=${{ github.event.repository.description }}' \
--annotation='index:org.opencontainers.image.created=${{ steps.timestamp.outputs.timestamp }}' \
--annotation='index:org.opencontainers.image.url=${{ github.event.repository.url }}' \
--annotation='index:org.opencontainers.image.source=${{ github.event.repository.url }}' \
$(printf '${{ env.GHCR_IMAGE }}@sha256:%s ' *)
- name: Create manifest list and push without annotations
# This step creates a manifest list for the Docker images built for different platforms.
# It uses the docker buildx imagetools create command to create the manifest list.
# The manifest list is created without annotations if the previous step fails.
# The manifest list is pushed to the GitHub Container Registry (GHCR) with the specified tags.
if: steps.manifest-annotate.outcome == 'failure'
working-directory: /tmp/digests
run: |
docker buildx imagetools create $(jq -cr '.tags | map("-t " + .) | join(" ")' <<< "$DOCKER_METADATA_OUTPUT_JSON") \
$(printf '${{ env.GHCR_IMAGE }}@sha256:%s ' *)
- name: Inspect image
# This step inspects the created manifest list to verify its contents.
# It uses the docker buildx imagetools inspect command to display information about the manifest list.
# The inspection output will show the platforms and tags associated with the manifest list.
id: inspect
run: |
docker buildx imagetools inspect '${{ env.GHCR_IMAGE }}:${{ steps.meta.outputs.version }}'

38
.github/workflows/pr-title.yaml vendored Normal file
View File

@@ -0,0 +1,38 @@
# Enforce Conventional Commit pull request titles so git-cliff can group
# release notes. The PR is squash-merged, so the title becomes the commit.
name: PR title
on:
pull_request:
types: [opened, edited, synchronize, reopened]
permissions:
contents: read
pull-requests: read
jobs:
lint:
name: pr-title
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- name: Validate title
uses: amannn/action-semantic-pull-request@48f256284bd46cdaab1048c3721360e808335d50 # v6.1.1
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
types: |
feat
fix
docs
refactor
perf
test
build
ci
chore
revert
requireScope: false
subjectPattern: ^(?![A-Z]).+$
subjectPatternError: |
The PR title subject must start in lowercase and must not end with a period.

286
.github/workflows/release.yaml vendored Normal file
View File

@@ -0,0 +1,286 @@
# Release pipeline. Pushing a tag that matches the identity in versions.env
# publishes both images (version + latest), creates the GitHub Release from
# git-cliff, and opens a PR that refreshes CHANGELOG.md on main.
#
# The tag must be dsh-vX.Y.Z or dsh-vX.Y.Z-rN and must match versions.env;
# tags are immutable, so a broken release is fixed with a new -rN revision.
name: Release
on:
push:
tags:
- "dsh-v*"
workflow_dispatch:
inputs:
tag:
description: Existing tag to release (must match versions.env)
required: true
type: string
# Read-only by default; the build and publish jobs add the write scopes they
# need as job-level grants.
permissions:
contents: read
concurrency:
group: release-${{ github.ref }}
cancel-in-progress: false
jobs:
meta:
name: Validate release identity
runs-on: ubuntu-latest
timeout-minutes: 15
outputs:
identity: ${{ steps.meta.outputs.identity }}
dsh_version: ${{ steps.meta.outputs.dsh_version }}
dsh_commit: ${{ steps.meta.outputs.dsh_commit }}
pnpm_version: ${{ steps.meta.outputs.pnpm_version }}
release_revision: ${{ steps.meta.outputs.release_revision }}
ghcr_image: ${{ steps.meta.outputs.ghcr_image }}
gateway_image: ${{ steps.meta.outputs.gateway_image }}
tag: ${{ steps.meta.outputs.tag }}
run_tests: ${{ steps.meta.outputs.run_tests }}
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
fetch-depth: 0
persist-credentials: false
- name: Validate tag against versions.env
id: meta
shell: bash
env:
# Passed through the environment, never interpolated into the script:
# a tag name or dispatch input can contain shell metacharacters.
REQUESTED_TAG: ${{ inputs.tag || github.ref_name }}
run: |
set -euo pipefail
set -a
# shellcheck disable=SC1091
source versions.env
set +a
tag="$REQUESTED_TAG"
identity="$DSH_VERSION"
[ -z "$RELEASE_REVISION" ] || identity="$DSH_VERSION-$RELEASE_REVISION"
if [ "$tag" != "$identity" ]; then
echo "tag '$tag' does not match versions.env identity '$identity'" >&2
exit 1
fi
if git rev-parse -q --verify "refs/tags/$tag" >/dev/null; then
git fetch origin main '+refs/heads/release/*:refs/remotes/origin/release/*'
commit=$(git rev-list -n1 "refs/tags/$tag")
allowed=0
for ref in origin/main origin/release/*; do
if git rev-parse -q --verify "$ref" >/dev/null && git merge-base --is-ancestor "$commit" "$ref"; then
allowed=1
break
fi
done
if [ "$allowed" -ne 1 ]; then
echo "tag commit $commit is not on main or a release branch" >&2
exit 1
fi
# Tests already ran in PR CI when the tag is on main; only re-run
# them for release-branch tags (where ci.yaml does not trigger).
run_tests=true
if git rev-parse -q --verify origin/main >/dev/null && git merge-base --is-ancestor "$commit" origin/main; then
run_tests=false
fi
else
echo "tag '$tag' does not exist in this repository" >&2
exit 1
fi
if ! upstream="$(git ls-remote --tags --exit-code "$DSH_REPO" \
"refs/tags/$DSH_VERSION^{}" "refs/tags/$DSH_VERSION")"; then
echo "upstream ref '$DSH_VERSION' not found in $DSH_REPO" >&2
exit 1
fi
# Never release a ref that no longer resolves to the pinned commit: an
# annotated tag has a peeled ^{} line, a lightweight tag only the ref.
resolved="$(printf '%s\n' "$upstream" \
| awk -v t="refs/tags/$DSH_VERSION^{}" '$2 == t {print $1}' | head -n1)"
if [ -z "$resolved" ]; then
resolved="$(printf '%s\n' "$upstream" \
| awk -v t="refs/tags/$DSH_VERSION" '$2 == t {print $1}' | head -n1)"
fi
if [ "$resolved" != "$DSH_COMMIT" ]; then
echo "upstream $DSH_REPO@$DSH_VERSION resolved to '$resolved'," \
"versions.env pins DSH_COMMIT=$DSH_COMMIT" >&2
exit 1
fi
{
echo "identity=$identity"
echo "dsh_version=$DSH_VERSION"
echo "dsh_commit=$DSH_COMMIT"
echo "pnpm_version=$PNPM_VERSION"
echo "release_revision=$RELEASE_REVISION"
echo "ghcr_image=$GHCR_IMAGE"
echo "gateway_image=$GATEWAY_IMAGE"
echo "tag=$tag"
echo "run_tests=$run_tests"
} >> "$GITHUB_OUTPUT"
- name: Verify repository pins
run: ./scripts/check-versions.sh
tests:
name: Unit tests and lint
needs: meta
# The test steps are guarded individually rather than the whole job so a
# main-branch tag (run_tests=false) still leaves this job "successful" and
# does not cascade-skip build-main/build-gateway.
runs-on: ubuntu-latest
timeout-minutes: 15
defaults:
run:
working-directory: examples/dsh-agents-as-a-service/gateway
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Set up Python
if: needs.meta.outputs.run_tests == 'true'
uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
with:
python-version: "3.12"
cache: pip
cache-dependency-path: |
examples/dsh-agents-as-a-service/gateway/requirements.txt
examples/dsh-agents-as-a-service/gateway/requirements-dev.txt
- name: Install dependencies
if: needs.meta.outputs.run_tests == 'true'
run: |
set -euo pipefail
python -m pip install --upgrade pip
pip install --require-hashes -r requirements.txt -r requirements-dev.txt
- name: Lint
if: needs.meta.outputs.run_tests == 'true'
run: |
set -euo pipefail
python -m ruff check .
python -m ruff format --check .
- name: Unit tests
if: needs.meta.outputs.run_tests == 'true'
run: python -m pytest -q
build-main:
name: Publish main image
needs: [meta, tests]
uses: ./.github/workflows/_build-image.yaml
permissions:
contents: read
packages: write
id-token: write
attestations: write
with:
image: ${{ needs.meta.outputs.ghcr_image }}
version: ${{ needs.meta.outputs.identity }}
context: .
dockerfile: Dockerfile
build-args: |
DSH_VERSION=${{ needs.meta.outputs.dsh_version }}
DSH_COMMIT=${{ needs.meta.outputs.dsh_commit }}
RELEASE_REVISION=${{ needs.meta.outputs.release_revision }}
PNPM_VERSION=${{ needs.meta.outputs.pnpm_version }}
VCS_REF=${{ github.sha }}
tags: |
${{ needs.meta.outputs.ghcr_image }}:${{ needs.meta.outputs.identity }}
${{ needs.meta.outputs.ghcr_image }}:latest
build-gateway:
name: Publish gateway image
needs: [meta, tests]
uses: ./.github/workflows/_build-image.yaml
permissions:
contents: read
packages: write
id-token: write
attestations: write
with:
image: ${{ needs.meta.outputs.gateway_image }}
version: ${{ needs.meta.outputs.identity }}
context: examples/dsh-agents-as-a-service/gateway
dockerfile: Dockerfile
tags: |
${{ needs.meta.outputs.gateway_image }}:${{ needs.meta.outputs.identity }}
${{ needs.meta.outputs.gateway_image }}:latest
publish-release:
name: Changelog and GitHub Release
needs: [meta, build-main, build-gateway]
runs-on: ubuntu-latest
timeout-minutes: 20
permissions:
contents: write
pull-requests: write
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
fetch-depth: 0
- name: Install git-cliff
env:
GIT_CLIFF_VERSION: v2.14.2
# SHA-256 of git-cliff-2.14.2-x86_64-unknown-linux-gnu.tar.gz; upstream
# publishes the matching SHA-512 next to the asset. Update with the version.
GIT_CLIFF_SHA256: 24f397c733add5390fdceee3a2088588ab0d5f944ce00d34cb7029b888cf2db4
run: |
set -euo pipefail
version="${GIT_CLIFF_VERSION#v}"
asset="git-cliff-${version}-x86_64-unknown-linux-gnu.tar.gz"
curl -fsSL --proto '=https' -o "/tmp/$asset" \
"https://github.com/orhun/git-cliff/releases/download/$GIT_CLIFF_VERSION/$asset"
printf '%s %s\n' "$GIT_CLIFF_SHA256" "/tmp/$asset" | sha256sum -c -
tar -xzf "/tmp/$asset" -C /tmp
install -m 0755 "/tmp/git-cliff-$version/git-cliff" /usr/local/bin/git-cliff
git-cliff --version
- name: Generate CHANGELOG and release notes
env:
IDENTITY: ${{ needs.meta.outputs.identity }}
run: |
set -euo pipefail
git-cliff --tag "$IDENTITY" --output CHANGELOG.md
git-cliff --tag "$IDENTITY" --latest --strip header --output /tmp/release-notes.md
echo "### Release $IDENTITY" >> "$GITHUB_STEP_SUMMARY"
head -n 40 /tmp/release-notes.md >> "$GITHUB_STEP_SUMMARY"
- name: Create GitHub Release
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
IDENTITY: ${{ needs.meta.outputs.identity }}
run: |
set -euo pipefail
if gh release view "$IDENTITY" >/dev/null 2>&1; then
echo "release $IDENTITY already exists; skipping" >&2
else
gh release create "$IDENTITY" --title "$IDENTITY" --notes-file /tmp/release-notes.md --verify-tag
fi
- name: Open CHANGELOG pull request
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
IDENTITY: ${{ needs.meta.outputs.identity }}
continue-on-error: true
run: |
set -euo pipefail
git config user.name "github-actions[bot]"
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
if git diff --quiet -- CHANGELOG.md; then
echo "CHANGELOG.md already up to date"
exit 0
fi
branch="chore/changelog-$IDENTITY"
git switch -c "$branch"
git add CHANGELOG.md
git commit -m "chore(changelog): refresh for $IDENTITY"
git push origin "$branch"
gh pr create \
--base main \
--head "$branch" \
--title "chore(changelog): refresh for $IDENTITY" \
--body "Generated by git-cliff for $IDENTITY. Do not edit CHANGELOG.md by hand."

View File

@@ -1,137 +0,0 @@
name: Validate
# Validates the Kubernetes manifests against the real agent-sandbox CRD
# schema (server-side dry-run on a throwaway kind cluster), runs the gateway
# unit tests + lint, and builds/smoke-tests the gateway image without a
# cluster. Runs on every PR and on pushes to main.
on:
pull_request:
push:
branches: [main]
permissions:
contents: read
jobs:
validate-manifests:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Create kind cluster
uses: helm/kind-action@1544676c07bb3570fcb70166beaa07c4528fd4aa # v1.15.1
with:
version: v0.33.0
cluster_name: dsh-validate
- name: Install agent-sandbox (extensions)
run: |
VERSION=v1.0.7
kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/download/${VERSION}/sandbox-with-extensions.yaml
kubectl wait --for=condition=Established crd/sandboxtemplates.extensions.agents.x-k8s.io --timeout=180s
kubectl wait --for=condition=Established crd/sandboxwarmpools.extensions.agents.x-k8s.io --timeout=180s
kubectl wait --for=condition=Established crd/sandboxclaims.extensions.agents.x-k8s.io --timeout=180s
kubectl wait --for=condition=Established crd/sandboxes.agents.x-k8s.io --timeout=180s
- name: Validate manifests (server dry-run)
run: |
set -euo pipefail
kubectl create namespace dsh-demo
for f in examples/dsh-agents-as-a-service/*.yaml; do
echo "== $f =="
kubectl apply --dry-run=server -f "$f"
done
kubectl apply --dry-run=server -f deploy/k8s/sandbox-template.yaml
echo "all manifests validated"
gateway-tests:
# The gateway lives in examples/dsh-agents-as-a-service/gateway/. Its
# runtime pins and dev pins are kept separate so the image only ships
# requirements.txt while CI installs both files.
name: Gateway unit tests and lint
runs-on: ubuntu-latest
defaults:
run:
working-directory: examples/dsh-agents-as-a-service/gateway
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
cache: pip
cache-dependency-path: |
examples/dsh-agents-as-a-service/gateway/requirements.txt
examples/dsh-agents-as-a-service/gateway/requirements-dev.txt
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt -r requirements-dev.txt
- name: Lint (ruff)
run: python -m ruff check .
- name: Run gateway unit tests
run: python -m pytest -v
gateway-image:
# No cluster and no real credentials: the gateway must start and answer
# /healthz with a dummy kubeconfig, which catches broken imports, missing
# runtime deps, and an unresponsive ASGI entrypoint.
name: Build and smoke-test gateway image
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Build gateway image
run: |
docker build \
-t dsh-aaas-gateway:ci \
examples/dsh-agents-as-a-service/gateway/
- name: Smoke-test /healthz without a cluster
run: |
set -euo pipefail
cat > /tmp/kubeconfig <<'EOF'
apiVersion: v1
kind: Config
clusters:
- name: dummy
cluster:
server: https://127.0.0.1:6443
insecure-skip-tls-verify: true
contexts:
- name: dummy
context:
cluster: dummy
user: dummy
users:
- name: dummy
user: {}
current-context: dummy
EOF
docker run -d --name gateway-smoke \
-p 18080:8080 \
-e NAMESPACE=dsh-demo \
-e POOL=dsh-pool \
-e KUBECONFIG=/tmp/kubeconfig \
-v /tmp/kubeconfig:/tmp/kubeconfig:ro \
dsh-aaas-gateway:ci
trap 'docker logs gateway-smoke || true; docker rm -f gateway-smoke || true' EXIT
code=""
for _ in $(seq 1 30); do
code=$(curl -s -o /dev/null -w '%{http_code}' http://localhost:18080/healthz || true)
[ "$code" = "200" ] && break
sleep 1
done
if [ "$code" != "200" ]; then
echo "gateway /healthz returned '$code' instead of 200" >&2
exit 1
fi
echo "gateway image healthy: GET /healthz -> 200"

5
.gitignore vendored
View File

@@ -159,3 +159,8 @@ __pycache__/
htmlcov/
.venv/
venv/
# Local repository tooling
.uvcache/
coverage.xml

11
.hadolint.yaml Normal file
View File

@@ -0,0 +1,11 @@
# Container lint policy. Base images are supplied as build args defaulting to
# pinned tags, and the image is rebuilt on every DSH release, so unpinned
# packages inside a pinned base image are intentional.
failure-threshold: warning
ignored:
- DL3008 # apt-get packages are intentionally not version-pinned
- DL3018 # apk packages are intentionally not version-pinned
- DL3006 # FROM uses build args (NODE_BUILD_IMAGE/NODE_RUNTIME_IMAGE) from versions.env
- DL3003 # the DSH packaging step deliberately cds inside a single layer
- DL3059 # consecutive RUN layers are kept for cache reuse and readability
- DL4006 # the upstream uv installer pipes curl to sh by design

13
.lychee.toml Normal file
View File

@@ -0,0 +1,13 @@
# Link checker for README/docs. Placeholders used by the examples are excluded
# on purpose; real URLs must resolve.
verbose = "info"
no_progress = true
max_retries = 2
exclude = [
"localhost",
"127\\.0\\.0\\.1",
"example\\.com",
"your-github-org",
"mailto:",
"ghcr\\.io/.*@sha256",
]

19
.yamllint.yml Normal file
View File

@@ -0,0 +1,19 @@
extends: default
rules:
document-start: disable
line-length:
max: 200
level: error
truthy:
check-keys: false
comments-indentation: disable
comments:
min-spaces-from-content: 1
ignore: |
.venv/
.uvcache/
node_modules/
# Upstream CodeQL boilerplate: long documentation comments, linted by actionlint.
.github/workflows/codeql.yml

163
AGENTS.md Normal file
View File

@@ -0,0 +1,163 @@
# AGENTS.md
Guidance for AI coding agents (and humans) working in this repository. Read it
before changing anything; it encodes the conventions the CI enforces.
## What this repository is
- **The root is the primary artifact.** The Dockerfile packages DeepSeek
Harness Web (dsh web) as a slim, non-root, multi-arch image. Everything else
exists to support it or to demonstrate an implementation on top of it.
- **Every implementation is a self-contained example** under examples/<name>/.
The multi-tenant platform lives in exactly one directory; do not split it into
parallel examples.
- **versions.env is the single source of truth** for the DSH ref, pnpm, the
control-plane versions and the image names.
- **Docs describe the shipped behavior**, not intentions. If you change a
manifest, port, env var or workflow, update the docs in the same change.
- **CHANGELOG.md is generated** by git-cliff from Conventional Commits. Never
edit it by hand.
## Golden rules
1. Keep the Dockerfile working for both linux/amd64 and linux/arm64; it must
stay the first-class artifact.
2. Never hard-code a version, image tag or registry path. Edit versions.env and
run make set-version when bumping.
3. Never commit a real secret or usable credential. Secret-shaped YAML lives in
*.yaml.example files; keys are documented in .env.sample only.
4. Every manifest under examples/*/manifests/ must parse and pass a server-side
dry-run in CI. Keep manifests numbered (NN-*.yaml), and keep exposure
variants under manifests/exposure/<variant>/ documented by the single
manifests/exposure/README.md (variants are mutually exclusive).
5. Do not edit CHANGELOG.md, and do not move or delete published tags.
Releasing is documented in docs/versioning.md.
6. Keep the gateway HTTP/WebSocket contract stable unless the change is
explicitly about it; clients and examples depend on it.
## Repository map
| Path | What lives there |
| --- | --- |
| Dockerfile | Main DSH Web image (build + runtime stages, entrypoint) |
| versions.env | Every pinned version and image name |
| Makefile | Canonical local entrypoint |
| build.sh | Deprecated shim that forwards to make |
| docs/ | main-image.md, examples.md, gitflow.md, versioning.md (index: README.md) |
| examples/ | One implementation per directory (contract in docs/examples.md) |
| examples/dsh-agents-as-a-service/ | The multi-tenant platform example (manifests, gateway, docs, scripts) |
| examples/minimal-single-tenant/ | The just-one-DSH-Web quickstart |
| scripts/ | check-versions.sh, check-examples.sh, check-manifests.py, set-version.sh |
| .dsh/skills/ | Project-local DSH skills (auto-discovered); dsh-agent-sandbox-platform covers platform operation, MCP limits and extension recipes |
| .github/workflows/ | ci.yaml (lint, dependency-review), ci-python.yaml, ci-manifests.yaml, ci-image.yaml (amd64 build + smoke test on Dockerfile/versions.env changes), reusable _build-image.yaml, build-*, release.yaml, e2e-kind.yaml, pr-title.yaml, codeql.yml |
## Commands
make check # versions, example layout and manifest YAML (run before every commit)
make test # gateway unit tests (pytest)
make lint # ruff + shellcheck (+ yamllint/actionlint when installed)
make image # build the main DSH Web image (slow: clones and builds DSH)
make gateway # build the example gateway image (fast)
make e2e # full kind walkthrough; needs a cluster with agent-sandbox
make changelog # regenerate CHANGELOG.md with git-cliff (release only)
make release-dry-run
make set-version DSH_VERSION=dsh-vX.Y.Z RELEASE_REVISION=rN
No cluster is needed for check, test, lint, gateway or image. Only make e2e
needs kind plus the agent-sandbox CRDs (see the example README).
## Testing expectations per change
| Change | Minimum verification |
| --- | --- |
| Documentation only | make check; verify relative links |
| Example manifest | make check; kubectl apply --dry-run=server in CI |
| Optional hardening (gVisor, nodeSelector, NetworkPolicy, provider keys, failed-Pod recovery) | make check; server dry-run in CI; manual verification on a capable cluster |
| Gateway code | make test; make lint; add or update tests |
| Dockerfile / versions.env | make check; the CI `build-amd64` job (`ci-image.yaml`) builds and smoke-tests the image on the PR; run `make image` locally only when you need the image |
| Workflow | actionlint in CI; keep actions SHA-pinned; no hard-coded versions |
| New example | make check; all files described in docs/examples.md |
| Project skill (.dsh/skills/) | make check; validate frontmatter; keep SKILL.md under ~8 KB; shellcheck skill scripts; load it with the skill tool |
## Versioning and releases
- Release identity = DSH_VERSION plus an optional -rN repository revision.
The current identity is dsh-v0.2.1-rc.5-r6 (first release at this DSH ref, so no
suffix); a repository-only change at the same ref publishes -r2.
- Channels: release tags are immutable; latest follows the newest release;
main is the development channel for both image packages.
- A broken release is fixed by publishing -r(N+1), never by re-tagging.
- The full procedure, compatibility table and rollback rules are in
docs/versioning.md.
## Branch and PR conventions
- Trunk-based with release branches, documented in docs/gitflow.md.
- Branch names: feat/, fix/, docs/, chore/, refactor/, test/.
- Conventional Commit subjects and PR titles (feat(scope): ..., fix(scope): ...).
- PRs are squash-merged; main stays linear and protected.
- Required on every PR: lint, pr-title, CodeQL (both Analyze jobs) and
dependency-review. python, manifests and build-amd64 are path-filtered
on-demand checks; run them when the change touches their scope.
- Keep diffs scoped; update docs with the code; never mix a version bump with
unrelated changes.
## Adding an example
1. Read docs/examples.md and examples/README.md.
2. Create examples/<name>/ with README.md, manifests/NN-*.yaml, optional
manifests/optional/<feature>/ or manifests/exposure/<variant>/ (each with a
README), optional components with their own Dockerfile/README/tests, and
scripts/ (executable, idempotent, self-cleaning).
3. Publish component images under <registry>/<owner>/<repo>/<example>/<component>.
4. Add the example to examples/README.md; scripts/check-examples.sh enforces
the layout.
## Security notes
- AUTH_PROXY_HEADER is opt-in. Only enable it when the gateway Service is
reachable exclusively through the authenticating proxy; otherwise any client
can spoof X-Forwarded-User. The SSO variant's NetworkPolicy is required for
production, and `examples/dsh-agents-as-a-service/scripts/preflight-sso.sh`
verifies the setup (Secrets, gateway env, TLS reference, placeholders, second
paths).
- The example gateway reads the DSH launch token from pod logs. Treat pod logs
as sensitive and keep its RBAC namespace-scoped.
- Browser sessions are opaque server-side ids held in gateway memory: they are
revocable (logout/delete/recreate/expiry) but a gateway restart logs everyone
out and horizontal scaling needs a shared store. Session cookies are never
accepted as bearer tokens.
- AUTH_RATE_LIMIT, SIGNUP_INVITE_CODE, MAX_IN_FLIGHT_PER_USER and
WS_MAX_MESSAGE_BYTES are in-process abuse guards, not ingress rate limits,
quotas or an audit log. Add those for an internet-facing deployment.
- The example shares provider keys (dsh-provider-keys plus the optional
dsh-provider-keys-extra) across sandboxes; do not describe it as
tenant-isolated.
- gVisor, nodeSelector, the shipped sandbox/exposure NetworkPolicies and the
documented gateway egress policy, extra provider keys and failed-Pod recovery
are opt-in and are not active in the base manifests; do not describe the base
example as hardened.
The sandbox and gateway-egress policies are required production steps; an
exposure without its NetworkPolicy is demo-only. See the example's
production checklist.
- TLS terminates outside the gateway. Set COOKIE_SECURE=true for HTTPS and drop
ALLOW_INSECURE_COOKIES; the gateway refuses COOKIE_SECURE=false unless
ALLOW_INSECURE_COOKIES=1, which is only for the plain-HTTP local demo. Never
expose that combination.
## Ask before
- Changing release workflows, tag formats or published image names.
- Editing the gateway's public routes or auth semantics.
- Adding a new top-level tool, service or dependency to the main image.
- Rewriting generated files (CHANGELOG.md) or git history.
## Where to look first
- [docs/main-image.md](docs/main-image.md) - container contract and tags
- [docs/examples.md](docs/examples.md) - example authoring rules
- [docs/gitflow.md](docs/gitflow.md) - branches, PRs, required checks
- [docs/versioning.md](docs/versioning.md) - versioning, changelog, releases
- [.dsh/skills/dsh-agent-sandbox-platform/SKILL.md](.dsh/skills/dsh-agent-sandbox-platform/SKILL.md) - platform operations, MCP limits, extension recipes
- [CONTRIBUTING.md](CONTRIBUTING.md) - development setup
- [SECURITY.md](SECURITY.md) - vulnerability reporting and threat notes

116
CHANGELOG.md Normal file
View File

@@ -0,0 +1,116 @@
# Changelog
All notable changes to this repository are documented here.
Generated by git-cliff from Conventional Commits; do not edit this file by hand.
## [dsh-v0.2.1-rc.5-r6] - 2026-10-31
### Added
- **example:** Persist the agent workspace on the PVC via subPath (#34) (367294b)
- **example:** Start DSH's folder picker on the persisted workspace (#36) (82ee0d0)
### CI
- **workflows:** Streamline CI and remove redundant builds (#31) (dcb5b43)
### Documentation
- Align readmes, docs and agent guidance with current behavior (#37) (24f23e0)
### Fixed
- **gateway:** Answer client-aborted requests with 499, not a traceback (#32) (cbcc56f)
- Ship bubblewrap for DSH's command sandbox, keep agents usable without it (#35) (a09d4ac)
- **ci:** Harden workflow token scope and download integrity (#39) (2f959e6)
- **gateway:** Replace token cookies with revocable sessions and harden abuse controls (#40) (1a929e3)
### Maintenance
- **changelog:** Refresh for dsh-v0.2.1-rc.5 (#30) (b57916d)
- Bump docker/login-action (#28) (8c34f43)
- **deps:** Bump agent-sandbox to v1.0.9 (#38) (4c073ee)
- Improve package installation and add Python depend… (#42) (8e9bd47)
### Other
- Add continue-on-error to CHANGELOG PR step (97c5632)
## [dsh-v0.2.1-rc.5] - 2026-10-30
### Added
- **example:** Add optional failed-Pod recovery (#26) (14c3fb8)
### Fixed
- **ci:** Make release image digests unique and allow changelog PRs (#24) (2b43d5d)
### Maintenance
- **changelog:** Refresh for dsh-v0.2.1-rc.3-r4 (#23) (97abf75)
- **deps:** Bump agent-sandbox to v1.0.8 (#25) (96d3030)
- **release:** Dsh-v0.2.1-rc.5 (#29) (bed8e56)
## [dsh-v0.2.1-rc.3-r4] - 2026-10-26
### Added
- **skills:** Add agent-sandbox platform skill (#20) (bf24306)
- **example:** Add optional gVisor, nodeSelector, NetworkPolicy and provider keys (#21) (3e9ba38)
### Maintenance
- **changelog:** Refresh for dsh-v0.2.1-rc.3-r3 (#19) (8681412)
- **release:** Dsh-v0.2.1-rc.3-r4 (#22) (e2f1fe5)
## [dsh-v0.2.1-rc.3-r3] - 2026-10-25
### Added
- **gateway-api:** Expose dsh-gateway externally via Traefik Gateway API (#12) (7d3faf0)
- **gateway-api:** GitHub SSO via oauth2-proxy in front of dsh-gateway (#13) (3f46279)
- **gateway:** SSO landing auto-connects and offers recreate sandbox (#15) (ed1398d)
- **repo:** Repo reorganization (#16) (3b70f17)
### Fixed
- **Dockerfile:** Update runtime dependencies and enable corepack for pnpm (6d97f61)
- **gateway-api:** Correct oauth2-proxy cookie-secret generation (#14) (73a65d9)
### Maintenance
- Bump the actions group across 1 directory with 3 updates (#17) (6a8fe54)
### Other
- Dockerfile extra tools (2b47afa)
- Modify gateway.yaml for image policy and timeout values (53e55fc)
- Add commit message prefix for Dependabot updates (6876871)
- Versioning update (#18) (ecba8b9)
- Refactor environment variable loading in build workflow (d627526)
- Refactor environment variable loading in build-gateway.yaml (027f74f)
- Refactor release workflow for environment variable sourcing (958cbe2)
## [v0.0.1] - 2026-10-21
### Other
- Initial commit (f3d4312)
- Update Dockerfile, README, and Kubernetes manifests for DSH Web deployment (#1) (0d66cba)
- Add DeepSeek Harness Web agents-as-a-service platform pattern (#2) (98cb547)
- Update action versions and cluster version (d9cb265)
- Add Python to CodeQL workflow languages (b0f992f)
- Add multi-arch Docker image build workflow with GitHub Actions (de11d34)
- Add step to create manifest list for Docker images (c9be929)
- Make DSH agents-as-a-service browser-usable; fix probe crash-loop (#3) (6301c21)
- Fix idle sweeper suspending claims the gateway does not own (#4) (29eefd6)
- Fix DSH crash-loop after suspend/resume (fsGroup chmod) (#5) (a8ef211)
- Add a startup probe so slow DSH cold starts are not killed (#6) (c193ed2)
- Normalise the pod model returned by CoreV1Api (#7) (9dea4a7)
- Forward the upstream query byte-for-byte (#8) (6876406)
- Do not read-time-out proxied upstream streams (#9) (163abae)
- Dial the sandbox pod IP for WebSocket upgrades (#10) (abe5b4d)
- Handshake under the pod-IP authority for WebSocket upgrades (#11) (0a61e57)

46
CODE_OF_CONDUCT.md Normal file
View File

@@ -0,0 +1,46 @@
# Code of Conduct
This project adopts the Contributor Covenant, version 2.1:
https://www.contributor-covenant.org/version/2/1/code_of_conduct/
## Our standards
We are committed to a welcoming, harassment-free experience for everyone,
regardless of age, body size, visible or invisible disability, ethnicity, sex
characteristics, gender identity and expression, level of experience,
education, socio-economic status, nationality, personal appearance, race,
religion, or sexual identity and orientation.
Examples of behavior that contributes to a positive environment:
- Demonstrating empathy and kindness toward other people
- Being respectful of differing opinions, viewpoints, and experiences
- Giving and gracefully accepting constructive feedback
- Accepting responsibility and apologizing to those affected by our mistakes
- Focusing on what is best for the community and the project
Examples of unacceptable behavior:
- Sexualized language or imagery, and sexual attention or advances
- Trolling, insulting or derogatory comments, and personal or political attacks
- Public or private harassment
- Publishing others' private information without explicit permission
- Other conduct which could reasonably be considered inappropriate in a
professional setting
## Enforcement
Maintainers are responsible for clarifying and enforcing these standards and
may remove, edit, or reject comments, commits, code, issues, and other
contributions that do not align with this Code of Conduct.
Report abusive, harassing, or otherwise unacceptable behavior to the
maintainers through a private GitHub Security Advisory or by contacting the
repository owner on GitHub. All complaints will be reviewed and investigated
promptly and fairly. Maintainers must respect the privacy and security of the
reporter.
## Attribution
This Code of Conduct is adapted from the Contributor Covenant, version 2.1,
available at https://www.contributor-covenant.org/version/2/1/code_of_conduct/

86
CONTRIBUTING.md Normal file
View File

@@ -0,0 +1,86 @@
# Contributing
Thanks for improving deepseek-kubernetes-sandbox. This repository packages
DeepSeek Harness Web as a container and ships self-contained implementation
examples around it. The root is the container; every implementation lives
under examples/.
## Repository layout
| Path | Purpose |
| --- | --- |
| Dockerfile | The primary artifact: the DSH Web image |
| versions.env | Single source of truth for every pinned version |
| Makefile | Canonical local entrypoint (build, test, lint, check, e2e) |
| docs/ | Repository-wide docs: main image, example contract, gitflow, versioning |
| examples/ | One self-contained implementation per directory |
| scripts/ | Repository checks (versions, example layout, manifests, version bump) |
| .github/ | CI, releases, dependency updates, templates |
## Requirements
- Docker (for image builds)
- Python 3.12+ (gateway tests and repository checks)
- make and git
- kubectl and kind only for the end-to-end example
## Local workflow
git switch -c feat/<short-slug>
make check # versions, example layout, manifest YAML
make test # gateway unit tests
make lint # ruff, shellcheck (+ yamllint/actionlint when installed)
make image # build the main DSH Web image
make gateway # build the example gateway image
make e2e # full kind walkthrough (optional, slow)
Install the gateway dev dependencies once:
python -m pip install -r examples/dsh-agents-as-a-service/gateway/requirements-dev.txt
## Pull requests
- Use a Conventional Commit title, for example feat(gateway): add wake metrics
or docs(examples): clarify SSO trust boundary. PR titles are linted.
- Keep the diff scoped. Squash merge into main; history stays linear.
- Required on every PR: lint, pr-title, CodeQL (both Analyze jobs) and
dependency-review. python, manifests and build-amd64 are path-filtered
on-demand checks; run them when the change touches their scope. Branch
protection is described in docs/gitflow.md.
- Update the docs that the change affects; docs are part of the change, not a
follow-up.
- Never commit a real secret. Secret-shaped YAML belongs in a *.yaml.example
file, and versions live in versions.env only.
## Changing versions
make set-version DSH_VERSION=dsh-vX.Y.Z RELEASE_REVISION=r2
make check
CHANGELOG.md is generated by git-cliff from Conventional Commits. Do not edit
it by hand; the release workflow regenerates it.
## Adding an example
Read docs/examples.md, then add examples/<name>/ with a README.md,
numbered manifests/NN-*.yaml, optional manifests/optional/<feature>/ or
manifests/exposure/<variant>/ (each with a README), optional components (a
Dockerfile plus README and tests), and executable, idempotent, self-cleaning
scripts. Add the literal examples/<name>/ entry to examples/README.md;
scripts/check-examples.sh enforces the layout.
## Documentation
- Root README.md is the front door: main image first, then examples.
- docs/ holds repository-wide material; example-specific material belongs in
the example directory.
## Branches and releases
See docs/gitflow.md for the branch model and docs/versioning.md for the
release identity, channels and rollback rules.
## Security
See SECURITY.md. Report vulnerabilities privately through GitHub Security
Advisories; do not open a public issue.

View File

@@ -1,26 +1,47 @@
# syntax=docker/dockerfile:1.7
ARG NODE_BUILD_IMAGE=docker.io/node:24-trixie
ARG NODE_RUNTIME_IMAGE=docker.io/node:24-trixie-slim
#
# The primary artifact of this repository: DeepSeek Harness Web (dsh web),
# packaged as a slim, non-root, multi-arch runtime image.
#
# Build args (all pinned in versions.env, verified by scripts/check-versions.sh):
# DSH_REPO upstream git remote to package
# DSH_VERSION upstream git ref to build (tag or branch)
# DSH_COMMIT full commit SHA DSH_VERSION must resolve to
# RELEASE_REVISION optional repository revision label (r2, r3, ...)
# PNPM_VERSION pnpm used to build DSH
# VCS_REF this repository's commit, recorded in OCI labels
# NODE_BUILD_IMAGE node image used to build DSH (pinned by digest)
# NODE_RUNTIME_IMAGE node image used at runtime (pinned by digest)
ARG NODE_BUILD_IMAGE=docker.io/node:24-trixie@sha256:8c761eca2bf7eec965f946cf47b8090143cb41f45f2f8699af0cc0298d2d2d19
ARG NODE_RUNTIME_IMAGE=docker.io/node:24-trixie-slim@sha256:54f84b4cf8432b21aaffd5d3ce8d41e12cea0d10d33c09d2f3c9879d49d6b446
# ---------------------------------------------------------------------------
# Build stage
# Build stage: clone DSH at the pinned ref, build, and package the CLI.
# ---------------------------------------------------------------------------
FROM ${NODE_BUILD_IMAGE} AS build
FROM $NODE_BUILD_IMAGE AS build
ARG DSH_REPO=https://github.com/deepseek-ai/deepseek-harness.git
ARG DSH_VERSION=dsh-v0.2.1-rc.2
ARG DSH_VERSION=dsh-v0.2.1-rc.5
ARG DSH_COMMIT=b830ec950faba0a46dda564be3e2d854afed94f1
ARG PNPM_VERSION=11.28.5
ENV CI=true \
NODE_OPTIONS=--max-old-space-size=4096
RUN corepack enable \
&& corepack prepare "pnpm@${PNPM_VERSION}" --activate
&& corepack prepare "pnpm@$PNPM_VERSION" --activate
WORKDIR /src
RUN git clone --depth 1 --branch "${DSH_VERSION}" "${DSH_REPO}" .
# Clone the pinned ref and fail closed unless it resolves to DSH_COMMIT. A moved
# tag would otherwise silently change what the image packages.
RUN git clone --depth 1 --branch "$DSH_VERSION" "$DSH_REPO" . \
&& actual="$(git -C /src rev-parse HEAD)" \
&& if [ "$actual" != "$DSH_COMMIT" ]; then \
echo "error: $DSH_REPO $DSH_VERSION resolved to $actual, expected DSH_COMMIT $DSH_COMMIT" >&2; \
echo "error: update DSH_VERSION/DSH_COMMIT in versions.env with scripts/set-version.sh" >&2; \
exit 1; \
fi
RUN pnpm install --frozen-lockfile
@@ -44,29 +65,56 @@ RUN cd /opt/dsh \
# ---------------------------------------------------------------------------
# Runtime stage
# ---------------------------------------------------------------------------
FROM ${NODE_RUNTIME_IMAGE} AS runtime
FROM $NODE_RUNTIME_IMAGE AS runtime
ARG DSH_VERSION
ARG DSH_VERSION=dsh-v0.2.1-rc.5
ARG RELEASE_REVISION=r6
ARG PNPM_VERSION=11.28.5
ARG VCS_REF=unknown
LABEL org.opencontainers.image.title="DeepSeek Harness Web packaged by SREDevOps.org" \
org.opencontainers.image.version="${DSH_VERSION}" \
org.opencontainers.image.source="https://github.com/sredevopsorg/deepseek-kubernetes-sandbox"
org.opencontainers.image.description="DeepSeek Harness Web (dsh web) as a slim, non-root, Kubernetes-ready image" \
org.opencontainers.image.source="https://github.com/sredevopsorg/deepseek-kubernetes-sandbox" \
org.opencontainers.image.version="$DSH_VERSION" \
org.opencontainers.image.revision="$VCS_REF" \
org.opencontainers.image.licenses="MIT" \
io.sredevopsorg.dsh.version="$DSH_VERSION" \
io.sredevopsorg.release.revision="$RELEASE_REVISION"
ENV NODE_ENV=production \
HOME=/home/node \
DSH_HOME=/home/node/.dsh \
DSH_PORT=3080 \
PATH="/app/node_modules/.bin:${PATH}"
PATH="/app/node_modules/.bin:$PATH"
# git/ripgrep are useful for DSH agent tools; tini handles signals.
# git/ripgrep/jq/yq/curl/gh are deliberate agent tooling parity: a sandboxed DSH
# instance expects the usual developer CLI. tini reaps signals for a clean stop.
# bubblewrap is DSH's first-choice command-sandbox runner on Linux. It ships
# unsetuid, so it needs unprivileged user namespaces plus the namespace/mount
# syscalls; a hardened pod (all capabilities dropped, RuntimeDefault seccomp)
# usually cannot create one, which is why DSH then falls back to Landlock and,
# when that is unavailable too, to DSH_PERMISSION_MODE=danger-full-access.
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
ca-certificates \
git \
ripgrep \
jq \
yq \
curl \
wget \
tini \
gh \
bubblewrap \
python3 \
python3-pip \
python3-venv \
&& rm -rf /var/lib/apt/lists/* \
&& install -d -o node -g node -m 0755 /workspace "${DSH_HOME}"
&& apt-get autoclean -y \
&& install -d -o node -g node -m 0755 /workspace "$DSH_HOME" \
&& corepack enable \
&& corepack prepare "pnpm@$PNPM_VERSION" --activate \
&& pnpm --version
COPY --from=build --chown=node:node /opt/dsh /app
@@ -88,7 +136,7 @@ if [ -n "$DSH_PUBLIC_URL" ]; then
set -- --public-url "$DSH_PUBLIC_URL" "$@"
fi
# Additional trusted hosts (e.g. the public gateway host behind an Ingress).
# Additional trusted hosts (e.g. the public gateway host behind a proxy).
# DSH Host/Origin-fences /api: without this, requests forwarded with the
# browser's Host header are rejected. Accepts comma- or space-separated names;
# localhost/127.0.0.1 are always trusted below.
@@ -115,6 +163,6 @@ WORKDIR /workspace
USER node
EXPOSE 3080
VOLUME ["/home/node/.dsh"]
VOLUME ["/home/node/.dsh", "/workspace"]
STOPSIGNAL SIGTERM
ENTRYPOINT ["/usr/bin/tini", "--", "/usr/local/bin/dsh-entrypoint"]
ENTRYPOINT ["/usr/bin/tini", "--", "/usr/local/bin/dsh-entrypoint"]

21
LICENSE Normal file
View File

@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2026 Nicolás Georger
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

View File

@@ -1,37 +1,83 @@
# Local build and test helpers. Every variable can be overridden, e.g.
# make image DSH_VERSION=dsh-v0.2.1-rc.2 IMAGE=dsh-web:dev
# make test
DSH_VERSION ?= dsh-v0.2.1-rc.2
PNPM_VERSION ?= 11.28.5
IMAGE ?= dsh-web:0.2.1-rc.2
GATEWAY_IMAGE ?= ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway
GATEWAY_DIR := examples/dsh-agents-as-a-service/gateway
PYTHON ?= python3
DSH_PORT ?= 3080
# Canonical local entrypoint. Includes versions.env so there is exactly one
# place to bump a version (run "make check" to verify pins across the repo).
#
# make image build the main DSH Web image
# make gateway build the example gateway image
# make test gateway unit tests
# make lint ruff + shellcheck (+ yamllint/actionlint when installed)
# make check versions, example layout and manifest YAML checks
# make e2e full example walkthrough on a kind cluster
# make changelog regenerate CHANGELOG.md with git-cliff
include versions.env
.PHONY: help image gateway run test lint
RELEASE_IDENTITY := $(DSH_VERSION)$(if $(RELEASE_REVISION),-$(RELEASE_REVISION))
IMAGE ?= $(GHCR_IMAGE):$(RELEASE_IDENTITY)
GATEWAY_LOCAL_IMAGE ?= dsh-aaas-gateway:dev
GATEWAY_DIR := examples/dsh-agents-as-a-service/gateway
EXAMPLE_DIR := examples/dsh-agents-as-a-service
PYTHON ?= $(if $(wildcard $(CURDIR)/.venv/bin/python),$(CURDIR)/.venv/bin/python,python3)
DSH_PORT ?= 3080
VCS_REF ?= $(shell git rev-parse --short HEAD 2>/dev/null || echo unknown)
.PHONY: help image run gateway test cov lint check e2e changelog release-dry-run set-version clean
help: ## Show available targets
@grep -E '^[a-z-]+:.*?## ' $(MAKEFILE_LIST) | sed 's/:.*## /\t/'
@grep -E '^[a-z-]+:.*?## ' $(MAKEFILE_LIST) | sed 's/:.*## / /'
image: ## Build the DSH Web image
image: ## Build the main DSH Web image (no push)
docker build \
--build-arg DSH_VERSION=$(DSH_VERSION) \
--build-arg DSH_COMMIT=$(DSH_COMMIT) \
--build-arg RELEASE_REVISION=$(RELEASE_REVISION) \
--build-arg PNPM_VERSION=$(PNPM_VERSION) \
--build-arg VCS_REF=$(VCS_REF) \
-t $(IMAGE) .
gateway: ## Build the gateway image
docker build -t $(GATEWAY_IMAGE) $(GATEWAY_DIR)/
run: image ## Build and run the DSH Web image
run: image ## Build and run the main image (needs DEEPSEEK_API_KEY)
docker run --rm -it \
-p $(DSH_PORT):3080 \
-v dsh-home:/home/node/.dsh \
-e DEEPSEEK_API_KEY \
$(IMAGE)
test: ## Run the gateway unit tests (needs requirements + requirements-dev installed)
gateway: ## Build the example gateway image locally
docker build -t $(GATEWAY_LOCAL_IMAGE) $(GATEWAY_DIR)/
test: ## Gateway unit tests
cd $(GATEWAY_DIR) && $(PYTHON) -m pytest -q
lint: ## Lint the gateway with ruff
cd $(GATEWAY_DIR) && ruff check .
cov: ## Gateway unit tests with coverage (needs pytest-cov)
cd $(GATEWAY_DIR) && $(PYTHON) -m pytest -q --cov=. --cov-report=term-missing --cov-report=xml:coverage.xml
lint: ## Lint Python, shell, YAML and Dockerfiles (skips missing tools locally)
cd $(GATEWAY_DIR) && $(PYTHON) -m ruff check .
cd $(GATEWAY_DIR) && $(PYTHON) -m ruff format --check .
shellcheck build.sh scripts/*.sh $(EXAMPLE_DIR)/scripts/*.sh
@command -v yamllint >/dev/null 2>&1 && yamllint . || echo "yamllint not installed; CI enforces it"
@command -v actionlint >/dev/null 2>&1 && actionlint || echo "actionlint not installed; CI enforces it"
check: ## Verify version pins, example layout and manifest YAML
./scripts/check-versions.sh
./scripts/check-examples.sh
$(PYTHON) scripts/check-manifests.py
e2e: ## Run the full example walkthrough on the current kind cluster
$(EXAMPLE_DIR)/scripts/e2e-kind.sh
changelog: ## Regenerate CHANGELOG.md from Conventional Commits
@command -v git-cliff >/dev/null 2>&1 || { echo "install git-cliff: https://git-cliff.org" >&2; exit 1; }
git-cliff --output CHANGELOG.md
release-dry-run: check ## Validate the release identity and preview release notes
@echo "release identity: $(RELEASE_IDENTITY)"
@echo "main image: $(GHCR_IMAGE):$(RELEASE_IDENTITY)"
@echo "gateway image: $(GATEWAY_IMAGE):$(RELEASE_IDENTITY)"
@if git rev-parse -q --verify "refs/tags/$(RELEASE_IDENTITY)" >/dev/null; then echo "ERROR: tag already exists" >&2; exit 1; fi
@command -v git-cliff >/dev/null 2>&1 && git-cliff --unreleased --tag $(RELEASE_IDENTITY) || echo "git-cliff not installed; skipping notes preview"
set-version: ## Bump pins: make set-version DSH_VERSION=dsh-vX [RELEASE_REVISION=rN]
@test -n "$(DSH_VERSION)" || { echo "DSH_VERSION is required" >&2; exit 1; }
./scripts/set-version.sh "$(DSH_VERSION)" "$(RELEASE_REVISION)"
clean: ## Remove local build/test caches
rm -rf .pytest_cache .ruff_cache $(GATEWAY_DIR)/.pytest_cache $(GATEWAY_DIR)/.ruff_cache $(GATEWAY_DIR)/__pycache__

323
README.md
View File

@@ -1,180 +1,213 @@
# deepseek-kubernetes-sandbox
Packages [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) Web (`dsh web`) into
a multi-arch, Kubernetes-ready container image, and ships the
[agents-as-a-service](examples/dsh-agents-as-a-service/) platform pattern — running a personal
DSH Web instance per user on [kubernetes-sigs/agent-sandbox](https://github.com/kubernetes-sigs/agent-sandbox)
behind a small browser-usable gateway.
**DeepSeek Harness Web, packaged as a container.** This repository's primary
artifact is a slim, non-root, multi-arch image of
[DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) Web
(`dsh web`), ready for Docker and Kubernetes. Everything else exists to support
that image or to demonstrate an implementation on top of it.
## What's in this repo
The flagship implementation is
[`examples/dsh-agents-as-a-service/`](examples/dsh-agents-as-a-service/): the
"agents-as-a-service" platform pattern with one DSH Web instance per user,
provisioned from a warm pool on
[kubernetes-sigs/agent-sandbox](https://github.com/kubernetes-sigs/agent-sandbox),
suspended when idle, and fronted by a small browser-usable gateway.
| Path | Purpose |
| --- | --- |
| [`Dockerfile`](Dockerfile) | Multi-stage build that clones DSH, `pnpm pack`s the CLI, and produces a slim runtime image (`:3080`) |
| [`build.sh`](build.sh), [`Makefile`](Makefile) | Local image builds: DSH Web and gateway, plus test/lint targets |
| [`examples/dsh-agents-as-a-service/`](examples/dsh-agents-as-a-service/) | The full multi-tenant platform pattern (warm pool, per-user claims, gateway, smoke test) |
| [`examples/dsh-agents-as-a-service/gateway/`](examples/dsh-agents-as-a-service/gateway/) | Starlette ASGI gateway: signup, cookie sessions, `/u/{user}/**` HTTP + WebSocket proxy, idle sweeper |
| [`deploy/k8s/sandbox-template.yaml`](deploy/k8s/sandbox-template.yaml) | Minimal single-tenant `SandboxTemplate` |
| [`docs/architecture.md`](docs/architecture.md) | Container and sequence diagrams, design decisions, security notes |
| [`.github/workflows/`](.github/workflows/) | CI: manifests dry-run, gateway tests/lint, multi-arch image builds |
## Published images
Everything is published to GitHub Container Registry as
`ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox`:
| Tag | Contents | Port |
| --- | --- | --- |
| `dsh-v0.2.1-rc.2` (also `main`, `latest`) | DSH Web image | `3080` |
| `gateway` | Starlette gateway image | `8080` |
The DSH Web tag matches the upstream DSH git tag it was built from.
## Quick start (DSH Web image)
## Quick start
```bash
docker run -d --name dsh-web \
-p 3080:3080 \
-v dsh-home:/home/node/.dsh \
-e DEEPSEEK_API_KEY \
ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.2
ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.5-r6
```
**DSH Web is not usable at a bare `/`.** Without its launch-token cookie it answers `401`, and
the token is only printed in the startup logs:
DSH Web is **not usable at a bare `/`**. Without its launch-token cookie it
answers `401`, and the token is printed only in the startup logs:
```bash
docker logs dsh-web 2>&1 | grep -m1 '?token='
# dsh web: http://localhost:3080/?token=...
```
Open the printed `?token=...` URL once — DSH sets its auth cookie and the UI loads normally
afterwards. Hitting `/` directly (or after the cookie expires) returns `401` by design; that is
why Kubernetes probes use `/favicon.svg` (see below) and why the gateway performs this
handshake for the browser.
Open the printed `?token=...` URL once: DSH sets its auth cookie and the UI loads
normally afterwards. Hitting `/` directly (or after the cookie expires) returns
`401` by design.
> **Different host port?** If `3080` is already taken on your workstation, map another port and
> set the public URL so the printed token URL matches: `-p 18081:3080 -e
> DSH_PUBLIC_URL=http://localhost:18081`, then open `http://localhost:18081/?token=...`.
### Host port and public URL
## Quick start (Kubernetes)
If `3080` is taken locally, map another port and set the public URL so the quoted
token URL matches:
```bash
kubectl create namespace dsh-demo --dry-run=client -o yaml | kubectl apply -f -
kubectl -n dsh-demo create secret generic dsh-provider-keys \
--from-literal=DEEPSEEK_API_KEY="sk-..."
kubectl apply -f deploy/k8s/sandbox-template.yaml
# A template needs a warm pool and a claim to become a running Sandbox.
kubectl apply -f - <<'EOF'
apiVersion: extensions.agents.x-k8s.io/v1beta1
kind: SandboxWarmPool
metadata:
name: dsh-pool
namespace: dsh-demo
spec:
replicas: 1
sandboxTemplateRef:
name: dsh-web-template
---
apiVersion: extensions.agents.x-k8s.io/v1beta1
kind: SandboxClaim
metadata:
name: dsh
namespace: dsh-demo
spec:
warmPoolRef:
name: dsh-pool
EOF
kubectl -n dsh-demo wait sandboxclaim/dsh --for=condition=Ready --timeout=180s
docker run -d --name dsh-web -p 18081:3080 \
-e DSH_PUBLIC_URL=http://localhost:18081 \
-v dsh-home:/home/node/.dsh \
-e DEEPSEEK_API_KEY \
ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.5-r6
# then open http://localhost:18081/?token=...
```
Then reach the instance through its launch-token URL:
### Hosts, probes and the 401
- **Probes must not hit `/`.** It returns `401` without the launch token, so a
readiness/liveness probe on `/` crash-loops the pod. Probe `/favicon.svg`
instead: it is unauthenticated. The shipped manifests use it.
- **Host allow-listing.** DSH rejects unknown `Host` headers. `localhost` and
`127.0.0.1` are always trusted; add any other public hostname to
`DSH_TRUSTED_HOSTS` (comma or space separated). A gateway or Ingress that
forwards the browser's `Host` needs its hostname listed there.
- **State lives in `/home/node/.dsh`.** Mount a volume (Docker) or PVC
(Kubernetes) to keep conversations, sessions and config across restarts.
### Runtime environment
| Variable | Default | Purpose |
| --- | --- | --- |
| `DEEPSEEK_API_KEY` | empty | Provider credential consumed by DSH. Inject it; never bake it into the image or commit it. |
| `DSH_PORT` | `3080` | Port DSH Web listens on inside the container. |
| `DSH_HOST` | auto-detected pod/container IP | Bind address. Leave empty in Kubernetes. |
| `DSH_PUBLIC_URL` | `http://localhost:${DSH_PORT}` | Public base URL used in generated links and the launch-token URL in the logs. |
| `DSH_TRUSTED_HOSTS` | empty | Extra trusted `Host`/`Origin` entries, comma/space separated. |
| `DSH_HOME` | `/home/node/.dsh` | DSH state directory (mount a volume here). |
DSH also resolves the standard per-provider environment names —
`OPENROUTER_API_KEY`, `GEMINI_API_KEY`, `NVIDIA_API_KEY`, `OPENAI_API_KEY`,
`ANTHROPIC_API_KEY`, `GROQ_API_KEY`, `XAI_API_KEY`, `HF_TOKEN`, and others —
through its packaged provider catalog. Inject them the same way; a key saved in
DSH's credential store wins over the environment. See
[provider credentials](docs/main-image.md#provider-credentials).
Full contract, build args, probes, volumes and signals:
[docs/main-image.md](docs/main-image.md).
## Published images
Published to GitHub Container Registry. The main image is the repository image;
example components are nested packages below it:
```
ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox # DSH Web (main image)
ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway # example gateway
```
| Tag | Contents | Meaning |
| --- | --- | --- |
| `dsh-v0.2.1-rc.5-r6` | DSH Web, port `3080` | **Current release identity.** Immutable: once published it never moves. |
| `latest` | DSH Web | Most recent release; same digest as the current identity. |
| `main` | DSH Web | Development channel rebuilt when image-definition files change on `main`. Not a release; not for production. |
The example gateway is published under its nested path with the same release
identity, e.g.
`ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway:dsh-v0.2.1-rc.5-r6`.
Release identities, channels and rollback rules are explained in
[docs/versioning.md](docs/versioning.md). Production deployments should pin a
digest:
```bash
SB=$(kubectl -n dsh-demo get sandboxclaim dsh -o jsonpath='{.status.sandbox.name}')
kubectl -n dsh-demo logs "$SB" | grep -m1 '?token='
kubectl -n dsh-demo port-forward "pod/$SB" 18081:3080
# open http://localhost:18081/?token=... from the log line
docker buildx imagetools inspect ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.5-r6
# image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox@sha256:<digest>
```
Two things bite first-time setups:
- **Probes must not hit `/`.** DSH returns `401` there without the launch token, so a
readiness/liveness probe on `/` crash-loops the pod. The shipped templates use
`/favicon.svg`, which is unauthenticated; keep that path in custom templates.
- **Host allow-listing.** DSH rejects unknown `Host` headers. `localhost` and `127.0.0.1` are
always trusted; for any other hostname add it to the template's `DSH_TRUSTED_HOSTS` (comma or
space separated). The gateway forwards the browser's `Host`, so the public gateway hostname
must be listed there.
## The multi-user platform pattern
`examples/dsh-agents-as-a-service/` is a port of the upstream
[hermes-agents-as-a-service](https://github.com/kubernetes-sigs/agent-sandbox/tree/main/examples/hermes-agents-as-a-service)
example to DSH Web. It demonstrates the platform shape — one `SandboxTemplate`, one warm pool,
one tiny `SandboxClaim` per user, and `operatingMode` as the cost dial — plus a gateway that
turns it into a usable service.
**Browser flow:** open the gateway landing page, sign up, and get an `HttpOnly` session cookie.
The browser then browses `http://<gateway>/u/<user>/`, which proxies HTTP **and WebSocket** to
that user's DSH Web. The gateway reads the DSH launch token from the pod logs, performs the
token handshake server-side, and injects the resulting DSH cookie on every proxied request — the
browser never sees the token. Suspended instances are woken transparently on the next request;
idle ones are suspended again after `IDLE_TIMEOUT`.
See the [example README](examples/dsh-agents-as-a-service/README.md) for the full walkthrough
and [docs/architecture.md](docs/architecture.md) for the diagrams and trade-offs.
### Mechanism mapping: Hermes → DSH
| Hermes example mechanism | Role | Adaptation for DSH |
| :--- | :--- | :--- |
| `SandboxTemplate` | Reusable Pod blueprint | Image `dsh-web`, port `3080`, probes on `/favicon.svg` |
| `volumeClaimTemplates` | Auto-create a PVC per user | Mount `/home/node/.dsh` (DSH state) |
| `SandboxWarmPool` | Pre-warmed instances | Keep as-is; size for expected concurrency |
| `SandboxClaim` + `warmPoolRef` | "Registration" adopts a pooled instance | Keep exactly as-is; one claim per user |
| `service: true` | Stable DNS per Sandbox | Enabled, so the gateway routes to each user's DSH Web |
| `envVarsInjectionPolicy: Disallowed` | Reject claims that would cold-start | `DEEPSEEK_API_KEY` injected via `envFrom` on the template (shared platform key) |
| `operatingMode` | Cost switch: Running / Suspended | Reused directly — suspend idle instances to save cost |
### Key differences from Hermes
- **Multi-arch** — the DSH image is built for amd64 and arm64 (no amd64 nodeSelector).
- **Single port** — DSH Web serves everything on `3080`; no API/dashboard port split.
- **One secret** — only `DEEPSEEK_API_KEY`; no dashboard/session/api-server credentials.
- **Host forwarding** — the gateway forwards `Host` + `X-Forwarded-*` (and keeps
`DSH_PUBLIC_URL` empty) so DSH generates reachable links; add the public hostname to
`DSH_TRUSTED_HOSTS`.
- **Browser sessions + WebSocket** — the gateway is Starlette ASGI, so it proxies WebSocket
upgrades as well as HTTP/SSE, and authenticates browsers with an `HttpOnly` cookie instead of
exposing per-user bearer tokens in URLs.
## Build it yourself
## Build from source
```bash
./build.sh # build the DSH Web image, then run it interactively
./build.sh web # build the DSH Web image only
./build.sh gateway # build the gateway image
make image # same as ./build.sh web
make gateway # gateway image
make test # gateway unit tests (pytest)
make lint # gateway lint (ruff)
make image # build the main DSH Web image (clones and builds upstream DSH)
make run # build it, then run it interactively (needs DEEPSEEK_API_KEY)
```
Or directly:
The pins (`DSH_VERSION`, `RELEASE_REVISION`, `PNPM_VERSION`) come from
[versions.env](versions.env) — never hard-code them
(`make check` verifies). [`build.sh`](build.sh) is a deprecated shim that
forwards to `make`; use the Makefile targets.
```bash
docker build \
--build-arg DSH_VERSION=dsh-v0.2.1-rc.2 \
--build-arg PNPM_VERSION=11.28.5 \
-t dsh-web:0.2.1-rc.2 .
## Examples
docker build -t ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway \
examples/dsh-agents-as-a-service/gateway/
```
Self-contained implementations that consume the image. Index and contract:
[`examples/README.md`](examples/README.md) and
[docs/examples.md](docs/examples.md).
See [`Dockerfile`](Dockerfile) for build args (`DSH_REPO`, `DSH_VERSION`, `PNPM_VERSION`) and
the runtime image contract (non-root `node` user, `DSH_HOME=/home/node/.dsh`, `EXPOSE 3080`).
`.env.sample` documents the runtime variables for both images.
| Example | What it demonstrates |
| --- | --- |
| [`examples/dsh-agents-as-a-service/`](examples/dsh-agents-as-a-service/) | Full multi-tenant platform: SandboxTemplate with per-user PVCs, warm pool, one claim per user, suspend-when-idle, wake-on-return, browser gateway with signup + `HttpOnly` sessions + HTTP/WebSocket proxy, Ingress or Gateway API + GitHub SSO exposure, and opt-in gVisor/NetworkPolicy/provider-key hardening and failed-Pod recovery. Start with the [walkthrough](examples/dsh-agents-as-a-service/README.md) and the [architecture](examples/dsh-agents-as-a-service/docs/architecture.md). |
| [`examples/minimal-single-tenant/`](examples/minimal-single-tenant/) | The smallest useful setup: one `SandboxTemplate` and one `SandboxClaim`, reached through `kubectl port-forward` + the launch-token URL. Manifest: [`manifests/10-sandboxtemplate.yaml`](examples/minimal-single-tenant/manifests/10-sandboxtemplate.yaml). |
The gateway is a component of the multi-tenant example and has its own
[README](examples/dsh-agents-as-a-service/gateway/README.md) (API, configuration,
auth/SSO, limitations). The example also documents opt-in gVisor,
`nodeSelector`, NetworkPolicy and extra provider-key hardening in
[optional hardening](examples/dsh-agents-as-a-service/docs/optional-hardening.md).
## Agent skills
DSH discovers project skills from `.dsh/skills/<name>/SKILL.md`. The repository
ships [`dsh-agent-sandbox-platform`](.dsh/skills/dsh-agent-sandbox-platform/SKILL.md),
which covers operating and extending the platform, the optional
[agent-sandbox MCP server](https://github.com/kubernetes-sigs/agent-sandbox/blob/main/clients/integrations/mcp-server/README.md),
and a read-only cluster preflight. Runbooks, the MCP reference, the extension
guide and troubleshooting notes live beside `SKILL.md`.
## Repository layout
| Path | Purpose |
| --- | --- |
| [`Dockerfile`](Dockerfile) | Primary artifact: the multi-stage DSH Web image (port `3080`). |
| [`versions.env`](versions.env) | Single source of truth for the DSH ref, repository revision, pnpm/control-plane versions and image names. |
| [`Makefile`](Makefile), [`build.sh`](build.sh) | Canonical local entrypoint; `build.sh` is a deprecated shim. |
| [`docs/`](docs/README.md) | Repository docs: image contract, example contract, gitflow, versioning. |
| [`examples/`](examples/README.md) | One self-contained implementation per directory. |
| [`examples/dsh-agents-as-a-service/`](examples/dsh-agents-as-a-service/) | Multi-tenant platform example: `manifests/`, `gateway/`, `docs/`, `scripts/`. |
| [`examples/minimal-single-tenant/`](examples/minimal-single-tenant/) | Single-sandbox quickstart. |
| [`scripts/`](scripts/) | Repository checks: versions, example layout, manifest YAML, version bump. |
| [`.dsh/skills/`](.dsh/skills/) | Project-local DSH skills, auto-discovered from the git root. |
| [`.github/`](.github/) | CI, release automation, dependency updates, PR/issue templates. |
| [`.env.sample`](.env.sample) | Reference environment names for local runs; nothing reads it automatically. Never commit a real key. |
## Development
The Makefile is the canonical entrypoint. No cluster is needed for `check`,
`test`, `lint`, `gateway` or `image`:
| Target | What it does |
| --- | --- |
| `make check` | Version pins, example layout and manifest YAML checks. Run before every commit. |
| `make test` | Gateway unit tests (`pytest`). |
| `make lint` | `ruff` + `shellcheck` (+ `yamllint`/`actionlint` when installed). |
| `make image` | Build the main DSH Web image. |
| `make gateway` | Build the example gateway image locally (`dsh-aaas-gateway:dev`). |
| `make e2e` | Full kind walkthrough of the multi-tenant example (needs a cluster with agent-sandbox). |
| `make changelog` | Regenerate `CHANGELOG.md` with git-cliff (release only). |
| `make set-version` | Bump pins in `versions.env`; e.g. `make set-version DSH_VERSION=<ref> RELEASE_REVISION=rN`. |
Local setup and the PR workflow: [CONTRIBUTING.md](CONTRIBUTING.md); branches,
required checks and releases: [docs/gitflow.md](docs/gitflow.md).
## Versioning and releases
The release identity is the upstream DSH ref plus an optional repository
revision: `DSH_VERSION` and `RELEASE_REVISION` in [versions.env](versions.env).
The current identity is **`dsh-v0.2.1-rc.5-r6`** — the first release at this
upstream DSH ref, so it carries no repository revision. Tags are immutable; a
broken release is fixed by publishing `-r(N+1)`, never by re-tagging.
`CHANGELOG.md` is generated by [git-cliff](https://git-cliff.org) from
Conventional Commits and is never edited by hand. See
[`CHANGELOG.md`](CHANGELOG.md), [docs/versioning.md](docs/versioning.md) and
[cliff.toml](cliff.toml).
## Contributing
- [CONTRIBUTING.md](CONTRIBUTING.md) — local setup, PR workflow, what a change needs.
- [AGENTS.md](AGENTS.md) — repository conventions for AI coding agents (and humans) and the CI commands.
- [SECURITY.md](SECURITY.md) — private vulnerability reporting and project threat notes.
- [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md) — community standards.
- [docs/gitflow.md](docs/gitflow.md) — branches, protected `main`, required checks, release process.
Do not commit secrets; secret-shaped YAML lives in `*.yaml.example` files, and
version pins change only through `versions.env` / `make set-version`.
## License
[MIT](LICENSE) © Nicolás Georger.

66
SECURITY.md Normal file
View File

@@ -0,0 +1,66 @@
# Security policy
## Supported versions
Only the latest published release is supported. Image tags are immutable;
fixes ship as a new release (see docs/versioning.md).
## Reporting a vulnerability
Report vulnerabilities privately through GitHub Security Advisories:
https://github.com/sredevopsorg/deepseek-kubernetes-sandbox/security/advisories/new
Do not open a public issue, and do not include live credentials in a report.
You can expect an acknowledgement within a few days and a coordinated
disclosure once a fix is available.
## Threat notes specific to this project
- **Launch token in pod logs.** The example gateway reads DSH's launch token
from pod logs (pods/log). Anyone with access to those logs can impersonate a
tenant. Keep the gateway's RBAC namespace-scoped and treat pod logs as
sensitive. Replacing log scraping with an upstream token API is tracked in
the [example architecture document](examples/dsh-agents-as-a-service/docs/architecture.md#upstream-feature-request-remove-log-scraping).
- **Browser sessions are server-side.** The gateway session cookie holds an
opaque session id, not a user name or token; it is not accepted as a bearer
token. Sessions are revoked on logout, user delete, `/recreate` and expiry,
but they live in gateway memory, so a gateway restart logs every browser out
and scaling beyond one replica requires a shared store. Do not treat the
cookie as durable or cross-replica state.
- **SSO header trust boundary.** AUTH_PROXY_HEADER makes the gateway trust an
identity header (for example X-Forwarded-User). Enable it only when the
gateway Service is reachable exclusively through the authenticating proxy
and its NetworkPolicy; otherwise any client can spoof the header. The SSO
exposure variant's policy is required, not optional: an exposure without it
is demo-only. Run `examples/dsh-agents-as-a-service/scripts/preflight-sso.sh`
after applying the variant; it checks the Secrets, gateway env, TLS
reference, leftover placeholders and accidental second paths.
- **Insecure-cookie guard.** TLS terminates outside the gateway. Set
COOKIE_SECURE=true for HTTPS and remove ALLOW_INSECURE_COOKIES; the gateway
refuses to start with COOKIE_SECURE=false unless ALLOW_INSECURE_COOKIES=1 is
explicitly set, which exists only for the plain-HTTP local demo. Never expose
that combination outside localhost.
- **Abuse controls are in-process.** AUTH_RATE_LIMIT/AUTH_RATE_WINDOW,
SIGNUP_INVITE_CODE, MAX_IN_FLIGHT_PER_USER and WS_MAX_MESSAGE_BYTES bound
casual abuse and resource exhaustion on the single gateway replica. They
reset on restart and are not an ingress rate limiter, WAF, quota system or
audit log; add those for an internet-facing deployment.
- **Shared provider keys.** The example injects DEEPSEEK_API_KEY and any keys in
the optional dsh-provider-keys-extra Secret into every tenant sandbox (a
warm-pool constraint). Treat tenants as mutually trusting until per-tenant
credentials or an egress proxy exist.
- **The base example is demo-only, not hardened.** gVisor, nodeSelector, the
sandbox/exposure NetworkPolicies, the gateway egress policy and extra provider
keys are opt-in. Without gVisor and the network policies there is no
tenant-to-tenant isolation; the sandboxes share the host kernel and can reach
cluster-internal endpoints. Production requirements are enumerated in the
[production checklist](examples/dsh-agents-as-a-service/docs/optional-hardening.md#6-production-checklist).
- **No secrets in the repository.** The repository only ships empty Secret
manifests and *.yaml.example files.
## Dependency and supply chain
GitHub Actions are pinned by commit SHA. The CodeQL workflow scans Python and
Actions. The release workflow publishes images with SBOM and provenance
attestations.

View File

@@ -1,68 +1,19 @@
#!/usr/bin/env bash
# Build the DSH Web and/or gateway images locally.
# Deprecated compatibility shim: "make" is the canonical entrypoint.
#
# Usage:
# ./build.sh # build the DSH Web image, then run it interactively
# ./build.sh web # build the DSH Web image only
# ./build.sh gateway # build the gateway image (examples/.../gateway/)
# ./build.sh -> make run (build the main image, then run it)
# ./build.sh web -> make image
# ./build.sh gateway -> make gateway
#
# Overridable environment:
# DSH_VERSION DSH git ref to build (default dsh-v0.2.1-rc.2)
# PNPM_VERSION pnpm version (default 11.28.5)
# IMAGE DSH Web image tag (default dsh-web:<version>)
# DSH_PORT host port for docker run (default 3080)
# GATEWAY_IMAGE gateway image tag (default ghcr.io/...:gateway)
# RUN 0 to build without running (default 1)
# Use the Makefile targets directly for everything else (test, lint, check, e2e).
set -euo pipefail
cd "$(dirname "$0")"
DSH_VERSION="${DSH_VERSION:-dsh-v0.2.1-rc.2}"
PNPM_VERSION="${PNPM_VERSION:-11.28.5}"
IMAGE="${IMAGE:-dsh-web:${DSH_VERSION#dsh-v}}"
GATEWAY_IMAGE="${GATEWAY_IMAGE:-ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway}"
DSH_PORT="${DSH_PORT:-3080}"
RUN="${RUN:-1}"
GATEWAY_DIR=examples/dsh-agents-as-a-service/gateway
build_web() {
docker build \
--build-arg "DSH_VERSION=${DSH_VERSION}" \
--build-arg "PNPM_VERSION=${PNPM_VERSION}" \
-t "${IMAGE}" \
"$@" \
.
}
build_gateway() {
docker build -t "${GATEWAY_IMAGE}" "$@" "${GATEWAY_DIR}/"
}
usage() {
echo "usage: $0 [run|web|gateway] [extra docker build args...]" >&2
exit 2
}
echo "build.sh is deprecated; use 'make image', 'make run' or 'make gateway'." >&2
case "${1:-run}" in
run)
shift
build_web "$@"
if [ "${RUN}" = "1" ]; then
docker run --rm -it \
-p "${DSH_PORT}:3080" \
-v dsh-home:/home/node/.dsh \
-e DEEPSEEK_API_KEY \
"${IMAGE}"
fi
;;
web)
shift
build_web "$@"
;;
gateway)
shift
build_gateway "$@"
;;
*)
usage
;;
run) exec make run ;;
web) exec make image ;;
gateway) exec make gateway ;;
*) echo "usage: $0 [run|web|gateway]" >&2; exit 2 ;;
esac

49
cliff.toml Normal file
View File

@@ -0,0 +1,49 @@
# git-cliff configuration: release notes and CHANGELOG.md are generated from
# Conventional Commits. See docs/versioning.md for the release procedure.
[changelog]
header = """
# Changelog
All notable changes to this repository are documented here.
Generated by git-cliff from Conventional Commits; do not edit this file by hand.
"""
body = """
{% if version %}\
## [{{ version }}] - {{ timestamp | date(format="%Y-%m-%d") }}
{% else %}\
## [Unreleased]
{% endif %}\
{% for group, commits in commits | group_by(attribute="group") %}
### {{ group | upper_first }}
{% for commit in commits %}
- {% if commit.scope %}**{{ commit.scope }}:** {% endif %}{{ commit.message | split(pat="\n") | first | upper_first }} ({{ commit.id | truncate(length=7, end="") }})
{%- endfor %}
{% endfor %}\n
"""
footer = ""
trim = true
[git]
conventional_commits = true
filter_unconventional = false
split_commits = false
commit_parsers = [
{ message = "^feat", group = "Added" },
{ message = "^fix", group = "Fixed" },
{ message = "^perf", group = "Performance" },
{ message = "^refactor", group = "Changed" },
{ message = "^docs", group = "Documentation" },
{ message = "^build", group = "Build" },
{ message = "^ci", group = "CI" },
{ message = "^test", group = "Tests" },
{ message = "^chore", group = "Maintenance" },
{ message = "^revert", group = "Reverted" },
{ message = ".*", group = "Other" },
]
protect_breaking_commits = false
filter_commits = false
# Release tags are dsh-vX.Y.Z[-rN]; the single pre-scheme tag is v0.0.1.
tag_pattern = "^(v|dsh-v).*$"
sort_commits = "oldest"

View File

@@ -1,96 +0,0 @@
# Minimal single-tenant SandboxTemplate for DeepSeek Harness Web.
#
# This is the "just one DSH Web" quick-start template. For the full
# multi-tenant platform (warm pool, per-user claims, gateway, idle-suspend),
# see examples/dsh-agents-as-a-service/.
#
# Prerequisites (create these first, or apply 00-prereqs.yaml from the
# examples directory):
# - Namespace dsh-demo.
# - Secret dsh-provider-keys in dsh-demo with key DEEPSEEK_API_KEY
# (injected via envFrom; the pod starts without it, only model calls fail).
# The agent-sandbox controller (v1.0.7, API v1beta1) must be installed.
apiVersion: extensions.agents.x-k8s.io/v1beta1
kind: SandboxTemplate
metadata:
name: dsh-web-template
namespace: dsh-demo
spec:
service: true # headless Service per Sandbox -> stable DNS (status.serviceFQDN)
podTemplate:
spec:
securityContext:
runAsNonRoot: true
runAsUser: 1000 # Matches the UID of the 'node' user
runAsGroup: 1000
fsGroup: 1000
# DSH's credentials plugin refuses to start when .credentials.yaml has
# any group/other permission bit (it creates the file 0600). The
# default kubelet fsGroup policy ("Always") recursively ORs 0660 into
# every file each time the pod starts, so a recreated/resumed sandbox
# comes back with 0660 and crash-loops. OnRootMismatch skips that walk
# once the volume root already has the fsGroup ownership/mode from the
# first mount. Keep in sync with
# examples/dsh-agents-as-a-service/10-sandboxtemplate.yaml.
fsGroupChangePolicy: OnRootMismatch
containers:
- name: dsh-web
image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.2
imagePullPolicy: IfNotPresent
ports:
- name: web
containerPort: 3080
envFrom:
- secretRef:
name: dsh-provider-keys # Secret with key DEEPSEEK_API_KEY
env:
# Empty => DSH derives links from the request Host. Keep it empty
# when fronting the sandbox with a proxy/gateway.
- name: DSH_PUBLIC_URL
value: ""
# DSH Host/Origin-fences /api. Empty trusts only localhost, which
# is right for `kubectl port-forward`. Set this to the public host
# (e.g. dsh.example.com) if you expose DSH behind an Ingress, so
# requests forwarded with the browser's Host are accepted.
- name: DSH_TRUSTED_HOSTS
value: ""
volumeMounts:
- name: dsh-home
mountPath: /home/node/.dsh # DSH_HOME
# / is behind DSH's launch-token cookie (401 without it); use an
# unauthenticated path for probes. The startup probe gives a cold
# start (plugin activation on a small node, or the first resume of a
# suspended sandbox) up to 120s before liveness can restart it.
startupProbe:
httpGet:
path: /favicon.svg
port: 3080
periodSeconds: 5
failureThreshold: 24
readinessProbe:
httpGet:
path: /favicon.svg
port: 3080
initialDelaySeconds: 5
periodSeconds: 5
livenessProbe:
httpGet:
path: /favicon.svg
port: 3080
initialDelaySeconds: 15
periodSeconds: 15
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
seccompProfile:
type: RuntimeDefault
volumeClaimTemplates:
- metadata:
name: dsh-home
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 5Gi
# storageClassName omitted => cluster default

49
docs/README.md Normal file
View File

@@ -0,0 +1,49 @@
# Repository documentation
Repository-wide documentation lives here. Example-specific material lives with
its example under [`examples/<name>/docs/`](../examples/) — see the
[agents-as-a-service architecture](../examples/dsh-agents-as-a-service/docs/architecture.md).
If you are new, read the [root README](../README.md) first: it is image-first and
gets a DSH Web container running before anything else.
## Index
| Document | Read it for |
| --- | --- |
| [main-image.md](main-image.md) | The primary artifact: build args, runtime environment, entrypoint, probes, volumes/ports/signals, non-root contract, image names, tags and digests. |
| [examples.md](examples.md) | The example contract: required layout, manifest rules, exposure variants, components and image naming, scripts, docs, and how to add a new example. |
| [gitflow.md](gitflow.md) | Branches, protected `main`, squash merges, Conventional PR titles, required CI checks, release branches, hotfixes and the one-time GitHub settings checklist. |
| [versioning.md](versioning.md) | Release identity (`DSH ref` + optional `-rN`), the current identity `dsh-v0.2.1-rc.5-r6`, why identities are immutable, channels, compatibility matrix, bumping pins, the release checklist, rollback, git-cliff and digest pinning. |
| [workflow-security.md](workflow-security.md) | GitHub Actions security posture: least-privilege tokens, pinned actions, untrusted-input handling, download integrity checks, artifact hygiene, the latest audit record and residual repository-setting recommendations. |
## Project skills
DSH discovers project skills from `<git root>/.dsh/skills/<name>/SKILL.md`
(the nearest `.git` ancestor). This repository ships one:
| Skill | Read it for |
| --- | --- |
| [dsh-agent-sandbox-platform](../.dsh/skills/dsh-agent-sandbox-platform/SKILL.md) | Operating and extending the agents-as-a-service platform, agent-sandbox MCP limits and deployment, and a read-only preflight. |
## Example documentation
| Document | Read it for |
| --- | --- |
| [examples/README.md](../examples/README.md) | The index of implementations in this repository. |
| [dsh-agents-as-a-service README](../examples/dsh-agents-as-a-service/README.md) | The multi-tenant platform walkthrough: warm pool, claims, suspend/resume, gateway, exposure and troubleshooting. |
| [dsh-agents-as-a-service architecture](../examples/dsh-agents-as-a-service/docs/architecture.md) | Container/sequence diagrams, ADRs, security and RBAC, failure modes and the evolution path. |
| [gateway README](../examples/dsh-agents-as-a-service/gateway/README.md) | The gateway's HTTP/WebSocket API, configuration, auth (including SSO), and limitations. |
| [optional hardening](../examples/dsh-agents-as-a-service/docs/optional-hardening.md) | Opt-in gVisor, nodeSelector, NetworkPolicy, provider keys and failed-Pod recovery. |
| [exposure variants](../examples/dsh-agents-as-a-service/manifests/exposure/README.md) | The mutually exclusive Ingress and Gateway API + SSO exposure paths. |
## Project policies
| Document | Read it for |
| --- | --- |
| [AGENTS.md](../AGENTS.md) | Repository conventions for AI agents and humans, and the commands the CI runs. |
| [CONTRIBUTING.md](../CONTRIBUTING.md) | Local setup, the change workflow and what a pull request needs. |
| [SECURITY.md](../SECURITY.md) | Vulnerability reporting and project-specific threat notes (pod logs, SSO header trust, shared provider key, TLS). |
| [CODE_OF_CONDUCT.md](../CODE_OF_CONDUCT.md) | Community standards. |
| [CHANGELOG.md](../CHANGELOG.md) | Generated release history (do not edit by hand). |
| [LICENSE](../LICENSE) | MIT. |

View File

@@ -1,319 +0,0 @@
# Architecture
This document describes the browser-usable **agents-as-a-service** platform in
[`examples/dsh-agents-as-a-service/`](../examples/dsh-agents-as-a-service/): how the gateway,
Kubernetes, and DSH Web fit together, and why the design looks the way it does.
Scope: one Kubernetes namespace (`dsh-demo`), one shared DSH Web image, one gateway deployment,
and one DSH Web pod + PVC per user. The cluster is assumed to run
[agent-sandbox](https://github.com/kubernetes-sigs/agent-sandbox) v1.0.7 with the extensions
(`SandboxTemplate`, `SandboxWarmPool`, `SandboxClaim`).
## Container view
```mermaid
flowchart LR
browser["Browser"]
subgraph cluster["Kubernetes cluster - namespace dsh-demo"]
gw["Gateway Deployment<br/>Starlette ASGI on uvicorn :8080<br/>landing page + auth + proxy + sweeper"]
api["Kubernetes API Server"]
ctrl["agent-sandbox controller"]
pool["SandboxWarmPool dsh-pool<br/>2 pre-warmed spares"]
claim["SandboxClaim dsh-USER<br/>one per user"]
sb["Sandbox dsh-pool-xxxx<br/>spec.operatingMode"]
svc["Headless Service<br/>status.serviceFQDN"]
pod["DSH Web pod<br/>node process :3080"]
pvc[("PVC dsh-home-SANDBOX<br/>/home/node/.dsh")]
end
browser -->|"HTTP + WebSocket<br/>/u/USER/**"| gw
gw -->|"claims, sandboxes, pods, pod logs"| api
api --> ctrl
ctrl -->|"adopt / suspend / resume"| claim
claim --> sb
pool -.->|"replenishes"| sb
sb --> svc
svc --> pod
pod --> pvc
```
Responsibilities:
| Component | Responsibility |
| --- | --- |
| **Browser** | Holds only the gateway session cookie (`dsh_gw_session`). Talks exclusively to the gateway. |
| **Gateway** | Signup/login, session auth, `SandboxClaim` lifecycle, wake-on-connect, idle sweeper, HTTP/SSE/WebSocket reverse proxy, DSH launch-token handshake. |
| **Kubernetes API** | Source of truth for claims, sandboxes, and pods; also the log source for the DSH launch token. |
| **agent-sandbox controller** | Adopts warm spares on claim, flips `operatingMode` (Running/Suspended), garbage-collects sandbox + PVC on claim delete. |
| **DSH Web pod** | One `dsh web` process per user, state on the per-user PVC, listening on `3080`. |
| **Warm pool** | Keeps `N` Ready sandboxes so signup is a label flip instead of a cold start. |
## Flows
### Signup (browser)
```mermaid
sequenceDiagram
autonumber
actor B as Browser
participant G as Gateway
participant K as Kubernetes API
participant C as agent-sandbox controller
B->>G: POST /users (form: user=bob, Origin)
G->>G: validate user name and body size
G->>K: create SandboxClaim dsh-bob (warmPoolRef)
K->>C: reconcile claim
C-->>K: adopt spare, Ready, status.sandbox.name = dsh-pool-xxxx
G->>K: read claim (status) + sandbox + pod
G-->>B: 303 Location /u/bob/ + Set-Cookie dsh_gw_session
Note over G: JSON clients instead get 201 (Ready),<br/>202 (accepted, still waking), or 409 (conflict)
```
Form signup waits up to `WAKE_TIMEOUT` for the claim to become `Ready`, then redirects; the
first proxied request also waits for `Ready` and performs the DSH handshake (below). A `409`
means the user already has a claim, in which case `POST /login` (bearer token) or the existing
cookie is the right path.
### First request / proxy (with handshake)
```mermaid
sequenceDiagram
autonumber
actor B as Browser
participant G as Gateway
participant K as Kubernetes API
participant D as DSH Web pod
B->>G: GET /u/bob/ (Cookie dsh_gw_session)
G->>G: authenticate cookie, user = bob
G->>K: resolve claim -> sandbox -> pod
alt sandbox Suspended
G->>K: patch operatingMode=Running (wake)
loop until Ready or WAKE_TIMEOUT
G->>K: read sandbox Ready condition
end
end
G->>K: read pod logs (tail)
K-->>G: "dsh web: ...?token=..." line
G->>D: GET /?token=... (browser Host, X-Forwarded-*)
D-->>G: 303 + Set-Cookie dsh-auth-*
G->>G: cache cookie for (user, pod UID, Host)
G->>G: strip Cookie/Authorization/hop-by-hop headers
G->>D: replay request with dsh-auth cookie
D-->>G: 200 HTML / SSE stream
G-->>B: 200 (upstream Set-Cookie dropped, Location rewritten)
```
Handshake cache behavior: one entry per `(user, pod UID, Host)`; refreshed once when DSH answers
`401`; dropped when the sandbox is suspended or deleted. The launch token and the DSH auth
cookie never reach the browser.
### Wake-on-connect and idle suspend
```mermaid
sequenceDiagram
autonumber
participant S as Sweeper - lifespan task
participant G as Gateway
participant K as Kubernetes API
actor B as Browser
S->>G: no proxied request for IDLE_TIMEOUT
G->>K: patch sandbox operatingMode=Suspended
K-->>G: pod deleted, PVC + Service retained
Note over S,G: in-flight requests/WebSockets are tracked,<br/>so a busy turn is never suspended
B->>G: GET /u/bob/ after a while
G->>K: patch operatingMode=Running
G->>K: wait for Ready (WAKE_TIMEOUT)
G-->>B: request held, then served
```
Suspend deletes only the pod. The PVC keeps conversations, sessions, and config, so resume
reattaches exactly the same state.
**Ownership boundary.** The gateway shares its namespace with claims created outside it — the
example's resource-layer walkthrough applies `dsh-alice` by hand, and every claim is named
`dsh-<user>`. Claims the gateway creates carry `aaas.example.com/managed-by=dsh-gateway`; the
sweeper skips everything else, so it can never suspend a sandbox an operator is using. Without
that marker the sweeper would suspend a hand-applied claim 60s after first sighting and break
`kubectl logs`/`port-forward` against its pod.
**Resume and file ownership.** The pod template sets `fsGroupChangePolicy: OnRootMismatch` on
purpose. DSH's credentials plugin refuses to boot when `.credentials.yaml` has any group/other
permission bit (it creates the file `0600`). The kubelet's default fsGroup policy (`Always`)
recursively ORs `0660` into every file on each pod start, so the first resume of a sandbox would
crash-loop with `mode 660`. `OnRootMismatch` only rewrites ownership/permissions when the volume
root does not already match the fsGroup, which it does after the first mount — so suspend/resume
keeps working. Sandboxes snapshot their pod template, so an updated `SandboxTemplate` does not
retroactively change existing ones.
### WebSocket
```mermaid
sequenceDiagram
autonumber
actor B as Browser
participant G as Gateway
participant D as DSH Web pod
B->>G: GET /u/bob/... Upgrade: websocket (Cookie dsh_gw_session)
G->>G: authenticate + mark user in-flight
alt cookie missing/invalid
G-->>B: 101 upgrade, then close 4401
else authenticated
G->>D: websockets.connect(path, Host, Cookie dsh-auth)
alt upstream handshake fails
G-->>B: 101 upgrade, then close 1011
else bridged
G-->>B: 101 Switching Protocols
loop frames
B->>D: frames proxied through G
D->>B: frames proxied through G
end
G->>G: clear in-flight on close
end
end
```
## Decisions and trade-offs
### ADR-1: Starlette ASGI instead of a sync Flask proxy
- **Decision:** implement the gateway as a Starlette app served by `uvicorn
gateway:create_app --factory`.
- **Why:** the browser needs WebSocket upgrades, streaming SSE responses, and concurrent
long-poll wake requests. A requests-based Flask proxy cannot upgrade connections and blocks
workers while a sandbox wakes.
- **Cost:** async code paths and explicit bridging for the sync `kubernetes` client (offloaded
to threads). Accepted: one small service, one framework.
### ADR-2: Path-based routing (`/u/{user}/`) instead of per-user subdomains
- **Decision:** serve each user under `https://gateway/u/{user}/`.
- **Why:** one hostname, one TLS certificate, no wildcard DNS or per-user Ingress objects. DSH
uses relative asset/API URLs and injects `<base href="./">`, so a path prefix works; the
gateway rewrites absolute-path `Location` headers and redirects `/u/{user}` to
`/u/{user}/`.
- **Cost:** any upstream code that emits root-absolute URLs would need rewriting; cookie
domains are shared (mitigated by stripping DSH `Set-Cookie`). If DSH ever changes its base
handling, per-user subdomains are the fallback.
### ADR-3: Server-side launch-token handshake instead of exposing the token
- **Decision:** the gateway reads DSH's launch token from pod logs, performs the token
handshake itself, caches DSH's auth cookie, and authenticates the browser with its own
`HttpOnly` cookie.
- **Why:** DSH has no token API and returns `401` on `/` without the launch-token cookie.
Handing the token to the browser would put it in URLs, history, and logs, and would let users
bypass gateway auth.
- **Cost:** log reading permission and a brittle parser; see [Upstream feature
request](#upstream-feature-request-remove-log-scraping). This is the least durable part of the
design and the first thing to replace.
### ADR-4: In-memory state, single replica
- **Decision:** idle clocks, handshake cache, and in-flight tracking live in gateway memory;
the Deployment runs one replica.
- **Why:** the simplest thing that works, with no extra datastore for a demo/reference
platform.
- **Cost:** a restart forgets idle timers (sandboxes stay Running until their next request) and
the handshake cache; two replicas would double-suspend and race. Scaling out requires moving
this state to a shared store (Redis/Postgres) or leader election. Documented instead of
hidden.
### ADR-5: Claims as the user record
- **Decision:** a user is a `SandboxClaim` named `dsh-<user>`; the bearer-token SHA-256 lives in
a claim annotation, and the gateway marks its own claims with
`aaas.example.com/managed-by=dsh-gateway`.
- **Why:** no database, no separate account service; deleting the claim is account deletion and
cascades everything. The ownership annotation keeps a namespaced claim list safe to share with
objects created by hand or by other controllers.
- **Cost:** no password reset, no admin API, no audit log of its own; Kubernetes API load grows
with users. Claims created before the ownership marker existed must be annotated
(`kubectl annotate sandboxclaim ...`) to keep being swept. Fine for a reference pattern, not
for a product.
### ADR-6: Gateway cookie session instead of bearer tokens in the browser
- **Decision:** browsers get `dsh_gw_session=<user>:<token>`, `HttpOnly`, `SameSite=Lax`
(`Secure` when enabled); API clients may still use `Authorization: Bearer`.
- **Why:** tokens never appear in URLs or JS-readable storage, and `SameSite=Lax` plus a strict
`Origin`/`Referer` check on cookie-authenticated mutations blocks basic CSRF.
- **Cost:** cookie-based auth needs TLS and correct `COOKIE_SECURE`; there is no signed-cookie
layer, so the gateway trusts the token hash comparison for validation.
## Security and RBAC
The gateway runs with a namespace-scoped ServiceAccount (`50-gateway.yaml`):
| API group | Resource | Verbs | Why |
| --- | --- | --- | --- |
| `extensions.agents.x-k8s.io` | `sandboxclaims` | `get`, `list`, `create`, `delete` | signup, state, account deletion |
| `agents.x-k8s.io` | `sandboxes` | `get`, `list`, `patch` | resolve DNS/pod, wake/suspend |
| `""` (core) | `pods` | `get`, `list` | resolve the pod behind a sandbox |
| `""` (core) | `pods/log` | `get` | launch-token handshake |
Notes and limits:
- **Pod logs are sensitive.** They contain the DSH launch token and possibly model output. The
permission is scoped to one namespace, but anyone who can read gateway memory or logs can
impersonate users. Prefer the upstream token API when available.
- **No cluster-wide access, no secret reads.** The gateway does not read `dsh-provider-keys`.
- **Browser-facing hardening:** `HttpOnly`/`SameSite=Lax`/optional `Secure` cookies,
timing-safe SHA-256 token comparison, strict same-origin check on cookie-auth mutations,
request-body limit (`MAX_BODY_MB`), hop-by-hop header stripping, upstream `Set-Cookie`
dropped, DSH `Location` headers rewritten under `/u/<user>/`.
- **Host allow-list:** DSH rejects unknown `Host` headers. The gateway forwards the browser's
`Host`, so every public hostname must be in `DSH_TRUSTED_HOSTS`; `localhost`/`127.0.0.1` are
always trusted.
- **TLS is external.** The gateway serves plain HTTP; terminate TLS at the Ingress and set
`COOKIE_SECURE=true`. Without TLS, session cookies are replayable on the network.
- **Tenant isolation** is pod + PVC separation; all tenants share one namespace, one DSH binary,
and one `DEEPSEEK_API_KEY`. Add NetworkPolicies and per-tenant credentials before calling it
multi-tenant-grade.
- **No gateway rate limiting.** Signup can create claims until the cluster or warm pool is
exhausted; put limits at the ingress for real deployments.
## Failure modes
| Failure | Symptom | Behavior / mitigation |
| --- | --- | --- |
| DSH log line changed or rotated away | Proxy returns `503` (missing launch token) | Fix the parser or restart the pod so the token line is re-emitted |
| Probe on `/` | Pod `CrashLoopBackOff`, `401` in probe logs | Use `/favicon.svg` for readiness/liveness (shipped templates do) |
| Warm pool exhausted | JSON signup returns `202` (state `Waking`); form signup is slow | Increase `replicas` in the warm pool |
| Gateway restart | Idle sandboxes stop suspending; handshake cache lost | Rebuilt lazily on next request; consider a shared store for production |
| DSH auth cookie expired mid-session | Proxy `401` once, then recovers | Gateway refreshes the handshake once |
| Upstream pod name changed after resume | Cached cookie keyed by old pod UID is useless | Cache is invalidated on suspend; next request re-handshakes |
| Two gateway replicas | Double suspend/wake races | Keep `replicas: 1` until state is externalized |
## Upstream feature request: remove log scraping
DSH should expose the launch token as a first-class, machine-readable interface so platforms do
not need `pods/log` access. Proposed shapes (any one suffices):
1. **Token file:** `dsh web --token-file /run/dsh/token` writes the current launch token (mode
`0600`) and rewrites it on rotation.
2. **Environment input:** accept `DSH_LAUNCH_TOKEN` (platform-generated) so the token never
has to be discovered.
3. **Loopback/trusted-host endpoint:** `GET /api/auth/launch-token` that only answers on the
pod's loopback or a trusted host and returns the token plus expiry.
4. **Token in a well-known file** under `DSH_HOME` with the same semantics as (1).
Why this matters: log scraping is brittle (rotation, restart, format changes), requires a broad
`pods/log` permission that also exposes application output, and can leak tokens into log
pipelines. Until an upstream API exists, the gateway's parser, `DSH_LOG_TAIL_LINES`, and the
single-retry refresh are the supported workaround.
## Evolution path
1. Replace log scraping with the upstream token API (above), deleting the `pods/log` permission.
2. Move idle clocks, in-flight tracking, and handshake cache to a shared store; then scale the
gateway horizontally (sticky routing not required if state is shared).
3. Add per-tenant API keys or an egress proxy so a single key is not shared by all sandboxes.
4. Add NetworkPolicies (gateway → sandboxes only), TLS everywhere, rate limits, and audit
logging at the ingress.
5. If DSH grows absolute URLs or per-user cookies that cannot be rewritten, switch to
per-user subdomains with wildcard DNS/TLS.

160
docs/examples.md Normal file
View File

@@ -0,0 +1,160 @@
# Example contract
Every implementation in this repository is a **self-contained example** under
[`examples/<name>/`](../examples/). Examples demonstrate how to run the main
DSH Web image (and, when needed, auxiliary components such as the
agents-as-a-service gateway) on Kubernetes. They are reference material, not
libraries: each one is applied with plain `kubectl`, documented end to end, and
deletable in one command.
[`examples/README.md`](../examples/README.md) is the index. If an example is not
listed there, it does not exist as far as `make check` is concerned.
## Required layout
```
examples/<name>/
+-- README.md # required: what it demonstrates, prerequisites,
| # ordered steps, troubleshooting, cleanup
+-- manifests/ # required
| +-- NN-*.yaml # numbered top-level manifests, plain kubectl
| +-- optional/ # optional opt-in manifests, applied by path
| +-- README.md # required when optional/ exists
| +-- <feature>/NN-*.yaml # e.g. network-policy/
| +-- exposure/ # optional exposure variants
| +-- README.md # required when exposure/ exists
| +-- <variant>/NN-*.yaml # e.g. ingress/, gateway-api-sso/
+-- scripts/ # optional
| +-- *.sh # executable, idempotent, self-cleaning
+-- docs/ # optional deeper material (architecture, ADRs)
+-- <component>/ # optional published component
+-- Dockerfile
+-- README.md # required whenever a Dockerfile exists
+-- tests...
```
### Rules the CI enforces
[scripts/check-examples.sh](../scripts/check-examples.sh) fails the build when:
- `examples/README.md` is missing or does not mention `examples/<name>/` for every
example directory;
- an example has no `README.md` or no `manifests/` directory;
- `manifests/` contains no top-level `*.yaml` files;
- a top-level `manifests/*.yaml` is not named `NN-*.yaml`;
- a `*.yaml` file sits at the example root instead of under `manifests/`;
- `manifests/exposure/` exists without a `manifests/exposure/README.md` describing
the variants;
- `manifests/optional/` exists without a `manifests/optional/README.md`
describing the opt-in manifests;
- a subdirectory with a `Dockerfile` has no `README.md`;
- a file in `scripts/` is not executable.
[scripts/check-manifests.py](../scripts/check-manifests.py) additionally parses
every `manifests/**/*.yaml` and requires `apiVersion`, `kind` and
`metadata.name`. CI also runs a server-side dry-run against the real
agent-sandbox CRDs on a throwaway kind cluster.
Run both locally with:
```bash
make check
```
## Manifests
- **Numbered and ordered.** `NN-*.yaml` files are applied in filename order and
read as a walkthrough: `00-prereqs.yaml`, `10-sandboxtemplate.yaml`,
`20-sandboxwarmpool.yaml`, ... Keep the numbering stable; it is referenced from
the example README.
- **Plain kubectl.** No Helm, Kustomize, or custom installer. `kubectl apply -f`
against a namespace the example creates is the whole interface.
- **Safe by default.** Never ship a usable Secret. Secret-shaped manifests are
`*.yaml.example` files with obvious placeholders and an inline creation
command; real values are generated per install.
- **Cluster-specific state stays out.** Node affinity, replica counts tuned for a
cluster, and similar fields belong in a local patch, not in the shipped
manifest. Optional scheduler/runtime hints may ship **commented out** with
placeholder labels; they must not be active.
- **Optional manifests are opt-in.** Put them under
`manifests/optional/<feature>/`, document them in
`manifests/optional/README.md`, and apply them by path. The top-level
`kubectl apply -f manifests/` is non-recursive and does not apply them.
- **Placeholders are explicit.** Hostnames, GitHub orgs and TLS material use
values like `dsh.example.com` / `your-github-org` with a TODO comment.
## Exposure variants
An example that can be reached from outside the cluster puts each option under
`manifests/exposure/<variant>/` and describes the trade-offs in
`manifests/exposure/README.md`. Variants are alternatives, not a stack: apply one.
`dsh-agents-as-a-service` currently ships:
| Variant | Path | What it gives you |
| --- | --- | --- |
| Ingress | `manifests/exposure/ingress/` | Traefik-style Ingress + TLS; the gateway does its own session auth. |
| Gateway API + SSO | `manifests/exposure/gateway-api-sso/` | Gateway API `Gateway`/`HTTPRoute` in front of oauth2-proxy (GitHub OAuth); the gateway inherits `X-Forwarded-User` via `AUTH_PROXY_HEADER`. |
## Components
A component is a program the example needs beyond the main image: today only the
agents-as-a-service gateway. A component lives in its own subdirectory with:
- `Dockerfile` — its image build;
- `README.md` — purpose, API/config reference, local development, limitations;
- tests — unit tests that run without a cluster, plus whatever else applies.
Component images are published as **nested packages below the repository image**:
```
<registry>/<owner>/<repo>/<example>/<component>
```
so the gateway is
`ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway`.
Never create an unrelated top-level package; `check-versions.sh` verifies the
nesting.
## Scripts
`scripts/*.sh` must be:
- **executable** (the CI checks the mode bit);
- **idempotent** — safe to re-run, cleaning up before creating;
- **self-cleaning** — a trap removes the namespace, port-forwards and temp files,
even on failure;
- **non-interactive** — usable in CI, with environment-variable knobs instead of
prompts.
`examples/dsh-agents-as-a-service/scripts/e2e-kind.sh` is the reference
implementation: it runs the README walkthrough and deletes everything it created.
## Documentation
- The example `README.md` is the walkthrough. Keep it runnable: every command
block should work when pasted in order.
- Deeper material (architecture, sequence diagrams, ADRs) belongs in the
example's `docs/`, not in the repository-level [docs/](../docs/).
- Repository-wide docs describe the container and the repo rules; example docs
describe one implementation.
## Adding a new example
1. Create `examples/<name>/` with the layout above.
2. Write `README.md` first: what it demonstrates, prerequisites, numbered steps,
troubleshooting, limitations, cleanup.
3. Add `manifests/NN-*.yaml`; keep secrets as `*.yaml.example`.
4. If a component needs an image, publish it under
`<registry>/<owner>/<repo>/<example>/<component>`.
5. Add an entry to [`examples/README.md`](../examples/README.md) using the literal
`examples/<name>/` path.
6. Run `make check` and, if you added a component, `make test` and `make lint`.
7. If the example is expensive to run (kind, images, cluster CRDs), ship a
`scripts/e2e-kind.sh` and wire it into CI.
## Current examples
| Example | Demonstrates |
| --- | --- |
| [`examples/dsh-agents-as-a-service/`](../examples/dsh-agents-as-a-service/) | The full multi-tenant platform pattern: SandboxTemplate, warm pool, per-user claims, suspend/resume, and the browser gateway. |
| [`examples/minimal-single-tenant/`](../examples/minimal-single-tenant/) | A single DSH Web sandbox: one SandboxTemplate and a claim. |

148
docs/gitflow.md Normal file
View File

@@ -0,0 +1,148 @@
# Gitflow: branches, pull requests and CI
This repository uses **trunk-based development with release branches**. `main` is
always releasable; feature work happens on short-lived branches; releases are cut
from `main` onto a release branch and tagged with an immutable release identity.
See [versioning](versioning.md) for what the tags mean.
## Branches
| Branch | Branched from | Merged into | Purpose |
| --- | --- | --- | --- |
| `main` | — | — | The integration trunk. Protected, linear history, always releasable. |
| `feat/<slug>` | `main` | `main` (squash) | New capability or example. |
| `fix/<slug>` | `main` | `main` (squash) | Bug fix. |
| `docs/<slug>` | `main` | `main` (squash) | Documentation only. |
| `refactor/<slug>`, `chore/<slug>`, `test/<slug>` | `main` | `main` (squash) | Non-behavioral rework, maintenance, tests. |
| `release/<identity>` | `main` at the release commit | — | Release stabilization and tagging. Only release-blocking fixes land here. |
| `hotfix/<slug>` | The release branch (or the tagged commit) | `main` and the release branch | Urgent production fix. |
Keep branches short-lived. Rebase on `main` rather than merging `main` into the
branch; the PR is squash-merged, so individual branch commits are never part of
`main` history.
## Pull requests
1. **One scoped change per PR.** Do not mix a version bump with unrelated work,
and update the docs the change touches in the same PR.
2. **Conventional PR title.** The title becomes the squash commit message, so it
is linted by the `pr-title` check: `feat(scope): ...`, `fix(scope): ...`,
`docs(scope): ...`, `build: ...`, `ci: ...`, `refactor: ...`, `test: ...`,
`chore: ...`. Breaking changes use `feat(scope)!: ...` or a
`BREAKING CHANGE:` footer.
3. **CI must be green.** All applicable checks below must pass before merge.
4. **Squash merge only.** `main` stays linear and protected; merged PRs are one
commit each.
5. **No direct pushes to `main`** or to a release branch. Both are protected.
6. **Review for behavior changes.** Infrastructure, workflow, release and
gateway-contract changes need a maintainer review; trivial docs changes still
need the checks.
## Required checks
| Check | Workflow / job | Required on every PR? | What it verifies |
| --- | --- | --- | --- |
| `lint` | `ci.yaml` | yes | `scripts/check-versions.sh` (version pins and markdown tokens), `scripts/check-examples.sh` (example layout and numbered manifests), shellcheck, actionlint, yamllint, hadolint and documentation links. |
| `python` | `ci-python.yaml` | no (path-filtered) | Gateway ruff check/format and unit tests (`pytest` with coverage). Runs only when the gateway or this workflow changes. |
| `manifests` | `ci-manifests.yaml` | no (path-filtered) | `scripts/check-manifests.py` (manifest shape) and a server-side dry-run of every example manifest against the agent-sandbox CRDs. Runs when manifest files, `scripts/check-manifests.py` or `versions.env` change. |
| `pr-title` | `pr-title.yaml` | yes | Conventional Commit PR title. |
| `Analyze (python)`, `Analyze (actions)` | `codeql.yml` | yes (both jobs) | CodeQL static analysis for Python and GitHub Actions. |
| `dependency-review` | `ci.yaml` | yes | Dependency changes introduced by the PR (job `dependency-review`; runs on pull requests only). |
`python`, `manifests` and `build-amd64` use native `on.pull_request.paths`
filtering, so PRs that do not touch the gateway, the manifests or the image
definition skip them. The gateway image is built and published by
`build-gateway.yaml` on merge; `/healthz` is covered by the gateway unit tests and
the Dockerfile is linted by `lint` (hadolint).
`ci-image.yaml` (`build-amd64`) is the PR-side check for the main image: when
`Dockerfile`, `.dockerignore` or `versions.env` change it builds `linux/amd64`
with the pins from `versions.env` — no registry login, no push — and smoke-tests
the result (`--version`, non-root UID, `/workspace` + `DSH_HOME`, the
`bubblewrap` runner, the version label, and `/favicon.svg` served from a running
container). The published multi-arch build still happens only in
`build-main.yaml` on merge.
Because a skipped path-filtered check is not a passing check, require only the
always-on set: `lint`, `pr-title`, both CodeQL `Analyze` jobs and
`dependency-review`. Keep `python`, `manifests` and `build-amd64` as on-demand
checks that skip when unrelated to the change.
The image build workflows (`_build-image.yaml`, `build-main.yaml`,
`build-gateway.yaml`) and `e2e-kind.yaml` are **not** required on every PR: the
image builds run where they are needed, and the kind end-to-end suite is reserved
for release validation and manual dispatch because it is slow.
## Releases
Releases are cut from `main`:
1. Confirm `main` is green and `make check`, `make test`, `make lint` pass.
2. Bump [versions.env](../versions.env) if the DSH ref changed, using
`make set-version`; otherwise keep the same `DSH_VERSION` and bump
`RELEASE_REVISION`.
3. Create `release/<identity>` from the release commit.
4. Run `make release-dry-run` and review the generated notes.
5. Create and push the annotated `dsh-*` release tag. The `release.yaml`
workflow validates that the tag matches [versions.env](../versions.env), builds
and pushes the multi-arch images (version + latest), creates the GitHub
Release from the git-cliff notes, and opens a `chore(changelog)` PR that
refreshes `CHANGELOG.md` on `main`.
6. Merge the changelog PR and any release-branch-only fixes back to `main`.
Full procedure: [versioning](versioning.md).
## Hotfixes
A released tag is **never** moved or re-tagged — tags and image identities are
immutable. To fix a broken release:
1. Branch `hotfix/<slug>` from the release tag (or its release branch).
2. Make the minimal fix, add a test, open a PR; the same required checks apply.
3. Publish a new identity with a higher `RELEASE_REVISION` (`-r(N+1)`). Do not
reuse or delete the broken tag.
4. Merge the fix back into `main` (or cherry-pick it) so the trunk is fixed too.
Rollback is also a re-release, never a retag: see
[versioning](versioning.md#rollback).
## Tag protection
Release tags follow `dsh-*` and are protected (the pre-scheme `v0.0.1` tag is
historical and unmaintained):
- only maintainers (the release workflow) may create them;
- they cannot be deleted or force-updated;
- they are annotated and point at the release commit on a release branch.
Image tags derive from the same identity and are equally immutable; the mutable
`latest` and `main` image channels always point at a release or a trunk build
respectively.
## One-time GitHub settings checklist
Configure these on the repository before relying on the workflow:
- [ ] **Branches → main**: require a pull request; require the always-on checks
`lint`, `pr-title`, both CodeQL `Analyze` jobs and `dependency-review`
(`python`, `manifests` and `build-amd64` are path-filtered and must not be
required); require branches to be up to date; require conversation
resolution; restrict direct pushes; allow only squash merges and enable
"Automatically delete head branches".
- [ ] **Branches → release/***: same protections, plus allow the release workflow
to push tags.
- [ ] **Tags → dsh-***: restrict creation/update/deletion to maintainers; a tag
push triggers the release workflow.
- [ ] **Actions → General**: allow GitHub Actions; set workflow permissions to
read-only by default; allow `GITHUB_TOKEN` to create pull requests only
where required.
- [ ] **Actions → Runners**: make the arm64 runners (`ubuntu-24.04-arm`) available
to the public/private repository as appropriate.
- [ ] **Packages**: grant the release workflow `packages: write` (workflow
permissions do this) and protect the
`ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox` packages from deletion.
- [ ] **Security**: enable CodeQL default setup **or** keep the advanced
`codeql.yml` workflow (not both), enable Dependabot alerts and
dependency review.
- [ ] **Secrets/Variables**: none are needed for image publishing; the workflows
use `GITHUB_TOKEN`.

236
docs/main-image.md Normal file
View File

@@ -0,0 +1,236 @@
# The main DSH Web image
This repository's primary artifact is a container image that packages
[DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) Web
(`dsh web`) for Kubernetes and plain Docker. The [Dockerfile](../Dockerfile) at the
repository root builds it; every version pin it uses lives in
[versions.env](../versions.env).
The image is image-first: it is the product, and the examples under
[`examples/`](../examples/) are self-contained implementations that consume it.
## What the build does
The Dockerfile is a two-stage build. Both stages use the `node` images pinned by
digest in [versions.env](../versions.env): a mutable tag alone is not trusted, and
`./scripts/check-versions.sh` rejects an image reference without a `@sha256:` digest.
1. **Build stage** (`NODE_BUILD_IMAGE`) clones DSH at the pinned git ref, verifies
that the clone's `HEAD` is exactly `DSH_COMMIT` (the build fails otherwise, so a
moved upstream tag cannot silently change what the image packages), installs
dependencies with pnpm, runs the production build, packages the `@deepseek-ai/dsh`
CLI with `pnpm pack`, and installs its production dependencies with npm.
2. **Runtime stage** (`NODE_RUNTIME_IMAGE`) copies `/opt/dsh` to `/app`, adds the
CLI tools DSH agents commonly use (`git`, `ripgrep`, `jq`, `yq`, `curl`, `wget`,
`gh`, `tini`, `uv`), installs `bubblewrap` as DSH's preferred command-sandbox
runner, and starts `dsh web` through a small entrypoint.
## Build it
```bash
make image # canonical: builds the DSH Web image, no push
```
`make image` reads [versions.env](../versions.env) and passes the pins as build args.
Pull requests that touch `Dockerfile`, `.dockerignore` or `versions.env` get the
same build for `linux/amd64` from the [`ci-image.yaml`](../.github/workflows/ci-image.yaml)
workflow — no registry login, no push — followed by a container smoke test.
To reproduce the equivalent by hand:
```bash
docker build \
--build-arg DSH_VERSION=dsh-v0.2.1-rc.5-r6 \
--build-arg DSH_COMMIT=b830ec950faba0a46dda564be3e2d854afed94f1 \
--build-arg RELEASE_REVISION= \
--build-arg PNPM_VERSION=11.28.5 \
--build-arg VCS_REF="$(git rev-parse --short HEAD)" \
-t ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.5-r6 .
```
### Build args
| Arg | Default / source | Purpose |
| --- | --- | --- |
| `DSH_REPO` | `https://github.com/deepseek-ai/deepseek-harness.git` | Upstream DSH repository to clone. |
| `DSH_VERSION` | `DSH_VERSION` in [versions.env](../versions.env) | Upstream DSH git tag/ref to package. The Dockerfile records it as `io.sredevopsorg.dsh.version` and (for local builds) `org.opencontainers.image.version`; published builds set the version label to the full release identity. |
| `DSH_COMMIT` | `DSH_COMMIT` in [versions.env](../versions.env) | Full commit SHA `DSH_VERSION` must resolve to upstream. The build stage fails unless `git rev-parse HEAD` in the clone equals this value, which pins the source independently of a movable tag. |
| `RELEASE_REVISION` | `RELEASE_REVISION` in [versions.env](../versions.env) | Repository revision suffix (`rN`), recorded in `io.sredevopsorg.release.revision`. The release identity is `DSH_VERSION` plus `-RELEASE_REVISION` when set. See [versioning](versioning.md). |
| `PNPM_VERSION` | `PNPM_VERSION` in [versions.env](../versions.env) | pnpm version activated with corepack in both stages. |
| `VCS_REF` | `git rev-parse --short HEAD` (via `make image`) | Source revision recorded for traceability. |
| `NODE_BUILD_IMAGE` | `docker.io/node:24-trixie@sha256:8c761eca2bf7eec965f946cf47b8090143cb41f45f2f8699af0cc0298d2d2d19` | Build-stage base image, pinned to the multi-arch index digest. |
| `NODE_RUNTIME_IMAGE` | `docker.io/node:24-trixie-slim@sha256:54f84b4cf8432b21aaffd5d3ce8d41e12cea0d10d33c09d2f3c9879d49d6b446` | Runtime-stage base image, pinned to the multi-arch index digest. |
Base images are pinned by digest because tags are mutable: `node:24-trixie` can be
re-pointed at a new build at any time. Bumping the tag and its digest is a
deliberate, reviewable change in [versions.env](../versions.env); environments that
mirror the image should also mirror the `node` digest they build from.
Never hard-code a version outside [versions.env](../versions.env):
`./scripts/check-versions.sh` (and `make check`) fails when a pin drifts.
## Runtime contract
### Entrypoint
The entrypoint is `tini -- /usr/local/bin/dsh-entrypoint`:
1. `tini` runs as PID 1 and reaps children/forwards signals.
2. The shell entrypoint applies defaults, derives a bind address when `DSH_HOST` is
empty (the pod/container's first non-loopback IPv4), builds the
`dsh web` command line, and `exec`s it so signals reach the Node process.
3. `localhost` and `127.0.0.1` are always added to DSH's trusted-host list; each
`DSH_TRUSTED_HOSTS` entry is added as an extra `--trusted-host`.
### Environment variables
| Variable | Default | Purpose |
| --- | --- | --- |
| `DSH_PORT` | `3080` | Port DSH Web listens on inside the container. |
| `DSH_HOST` | auto-detected non-loopback IPv4 | Bind address. Leave empty in Kubernetes so the pod IP is used automatically. |
| `DSH_PUBLIC_URL` | `http://localhost:${DSH_PORT}` | Public base URL used to build links and the launch-token URL printed in the logs. Set it when the container is published on a different host port or behind a proxy. |
| `DSH_TRUSTED_HOSTS` | empty | Extra `Host`/`Origin` allow-list entries, comma- or space-separated. `localhost`/`127.0.0.1` are always trusted. Required when a proxy forwards the browser's `Host`. |
| `DEEPSEEK_API_KEY` | empty | Provider credential read by DSH. Inject it with `-e`/`envFrom`; never bake it into the image. Only model calls fail when it is missing. |
| `DSH_HOME` | `/home/node/.dsh` | DSH state directory (conversations, sessions, config, credentials). Mount a volume here to persist it. |
| `HOME` | `/home/node` | OS home: where `~` resolves and where tools keep caches and config. DSH's directory picker has no start-path setting, so it lists `$HOME` when opened. The shipped sandbox templates override it to `/workspace` so the picker opens on the persisted workspace; the trade-offs are below. |
| `DSH_PERMISSION_MODE` | `workspace-write` | Deployment default for DSH's file policy: `read-only`, `workspace-write`, or `danger-full-access` (which also turns approval prompts off). The confined modes need `bubblewrap` or Landlock inside the container; where neither is usable every bash command fails closed with `SANDBOX_UNAVAILABLE`. See the note below. |
The image also sets `NODE_ENV=production`, `HOME=/home/node`, and a `PATH` that
includes `/app/node_modules/.bin`.
`HOME` decides where DSH's folder picker starts. DSH has no start-path setting:
the picker's browse backend lists `$HOME` when the dialog opens and labels that
listing *Home*. With the image default (`/home/node`) that listing is outside a
mounted volume; the example sandbox templates
([minimal](../examples/minimal-single-tenant/manifests/10-sandboxtemplate.yaml),
[agents as a service](../examples/dsh-agents-as-a-service/manifests/10-sandboxtemplate.yaml))
mount the same PVC's `workspace` subPath at `/workspace` and set `HOME=/workspace`
so the picker opens on the persisted agent workspace. `~`
and every tool that keeps caches or config under `$HOME` (`npm`/`pnpm`, `uv`,
`git`, `gh`, DSH's global `~/.agents` instructions and skills) follow, so those
dotfiles then live on the workspace volume too. `DSH_HOME` is separate and stays
`/home/node/.dsh`, so sessions, config and credentials stay out of the
workspace. An existing sandbox stamps this environment at pod creation, so the
override only applies to sandboxes created after it was added.
DSH confines bash commands itself (a second layer inside whatever container
sandbox you already run). Its Linux runners are `bubblewrap` and then Landlock,
and it fails closed rather than running unconfined. The image ships
`bubblewrap`, but Debian's binary is unsetuid: it needs **unprivileged user
namespaces** plus the namespace/mount syscalls. A hardened pod — non-root,
`capabilities.drop: ["ALL"]`, `seccompProfile: RuntimeDefault` — usually cannot
create one, and it may expose no Landlock runner either, in which case every
command fails with `SANDBOX_UNAVAILABLE`. Set
`DSH_PERMISSION_MODE=danger-full-access` only when the surrounding pod or VM is
the isolation boundary you rely on; keep the confined modes when the host
provides a working runner (check `grep -o landlock /sys/kernel/security/lsm` and
`bwrap --ro-bind / / true` inside the container first).
### Provider credentials
DSH resolves provider API keys from the environment when no stored credential
exists. The packaged provider catalog maps each provider to its standard
environment name. Commonly used names:
`DEEPSEEK_API_KEY`, `OPENROUTER_API_KEY`, `GEMINI_API_KEY`, `NVIDIA_API_KEY`,
`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GROQ_API_KEY`, `XAI_API_KEY`,
`MISTRAL_API_KEY`, `TOGETHER_API_KEY`, `CEREBRAS_API_KEY`,
`AI_GATEWAY_API_KEY`, `HF_TOKEN`.
Use `GEMINI_API_KEY` for Google Gemini (not `GOOGLE_API_KEY`), `HF_TOKEN` for
Hugging Face, and `AI_GATEWAY_API_KEY` for the Vercel AI Gateway. Inject keys
with `-e`/`--env-file` in Docker or a Secret + `envFrom` in Kubernetes; never
bake them into the image.
A key saved through DSH's credential store (`$DSH_HOME/.credentials.yaml`) wins
over the environment. The
[agents-as-a-service example](../examples/dsh-agents-as-a-service/docs/optional-hardening.md)
documents the optional `dsh-provider-keys-extra` Secret and the pod-refresh
procedure.
### Ports, volumes and signals
| Item | Value | Notes |
| --- | --- | --- |
| Exposed port | `3080/tcp` | `EXPOSE 3080`; one port serves the UI, assets, API, SSE and WebSocket traffic. |
| State volume | `/home/node/.dsh` | Declared with `VOLUME`; use a named volume (Docker) or PVC (Kubernetes). |
| Working directory | `/workspace` | The agent workspace. Mount a volume here if agents should keep files; the shipped example templates mount the same PVC's `workspace` subPath here and point `HOME` at it, so DSH's folder picker opens on it. |
| Stop signal | `SIGTERM` | `tini` forwards it to `dsh web`; allow a graceful shutdown period. |
| User | `node`, UID/GID `1000` | The process is non-root. A PVC mounted at `DSH_HOME` must be writable by UID 1000. |
### Health probes
`/` is **not** a health endpoint: DSH returns `401` until a browser completes the
launch-token handshake. Probe `/favicon.svg` instead — it is unauthenticated and
served by the web layer, so it proves the HTTP server is up without triggering auth.
```yaml
startupProbe:
httpGet: {path: /favicon.svg, port: 3080}
periodSeconds: 5
failureThreshold: 24
readinessProbe:
httpGet: {path: /favicon.svg, port: 3080}
periodSeconds: 5
livenessProbe:
httpGet: {path: /favicon.svg, port: 3080}
periodSeconds: 15
```
The shipped templates use the same path and startup budget; see
[`examples/minimal-single-tenant/manifests/10-sandboxtemplate.yaml`](../examples/minimal-single-tenant/manifests/10-sandboxtemplate.yaml).
A `startupProbe` is important because plugin activation (or the first start of a
large sandbox) can exceed a default liveness budget.
### Launch token
DSH prints its one-time launch URL to the logs:
```
dsh web: http://<host>:${DSH_PORT}/?token=...
```
Open that URL once to obtain DSH's auth cookie. After that the UI works at the bare
URL; without the cookie `/` answers `401` by design. The
[agents-as-a-service gateway](../examples/dsh-agents-as-a-service/gateway/) performs
this handshake server-side so browsers never see the token.
## Published image names and tags
Registry layout: the main image is the repository itself, and every example
component is a nested package below it:
```
ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox # main DSH Web image
ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway # example component
```
The main image is published multi-arch (`linux/amd64`, `linux/arm64`) to
GitHub Container Registry:
| Tag | Meaning |
| --- | --- |
| `dsh-v0.2.1-rc.5-r6` | Current release identity. **Immutable**: once published it never moves. |
| `latest` | Most recent release; points at the same digest as the current identity. |
| `main` | Development channel rebuilt when image-definition files change on `main`; not a release, not for production. |
Release identities, channels and rollback rules are defined in
[versioning](versioning.md). Tags are immutable, so production deployments should
pin a digest:
```bash
docker buildx imagetools inspect ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.5-r6
# ... pin the digest in your deployment manifests:
# image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox@sha256:<digest>
```
## Verifying an image
```bash
docker inspect ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.5-r6 \
--format '{{json .Config.Labels}}'
# org.opencontainers.image.title / .version / .source
# io.sredevopsorg.dsh.version / io.sredevopsorg.release.revision
```
Release builds also publish SBOM and provenance attestations from the release
workflow (see [SECURITY.md](../SECURITY.md) and [gitflow](gitflow.md)).

186
docs/versioning.md Normal file
View File

@@ -0,0 +1,186 @@
# Versioning, releases and changelog
[versions.env](../versions.env) is the **single source of truth** for the DSH ref,
the repository revision, the pnpm version, the control-plane component versions
and the image names. Nothing else may hard-code a version:
[scripts/check-versions.sh](../scripts/check-versions.sh) fails when a pin drifts,
and it also verifies that prose mentions only the current release identity or its
base DSH tag.
## Release identity
```
release identity = DSH_VERSION + optional "-" + RELEASE_REVISION
```
| Piece | Meaning |
| --- | --- |
| `DSH_VERSION` | The upstream DeepSeek Harness git ref being packaged, e.g. `dsh-v0.2.1-rc.5-r6`. It mirrors upstream's own tag. |
| `RELEASE_REVISION` | An optional repository revision at that DSH ref: empty for the first release, `rN` (`N >= 2`) for a repository-only re-release. |
Examples:
- `DSH_VERSION=dsh-v0.2.1-rc.5-r6`, `RELEASE_REVISION=` → identity `dsh-v0.2.1-rc.5-r6`
(first release at that upstream ref).
- `DSH_VERSION=<ref>`, `RELEASE_REVISION=rN` → identity `<ref>-rN` (for example
documentation, CI or gateway changes on top of an already-released ref; no DSH
upgrade). Use `r2` for the first re-release: revision 1 is the plain tag.
A DSH upgrade produces a new base identity (for example the next upstream ref
with an empty revision); a repository-only change at the same DSH ref increments
`rN`.
## Why release identities are immutable
**Container tags are immutable** in this project: re-pushing a tag would silently
change what an existing deployment runs. A fix therefore publishes a new identity;
see the rules above.
The current identity is `dsh-v0.2.1-rc.5-r6`, the first release at this upstream
ref, so it carries no revision suffix. The identity is stored in
[versions.env](../versions.env) and consumed everywhere: the Dockerfile labels
(version and revision), the example manifests, the Makefile and the release
workflow.
## Channels
| Channel | Image tag | Meaning |
| --- | --- | --- |
| Release | `dsh-v0.2.1-rc.5-r6` | The current immutable release identity. What production should pin (preferably by digest). |
| `latest` | `latest` | Alias for the most recent release; moves only when a new release is published. |
| `main` | `main` | Development channel, rebuilt when image or gateway sources change on `main`. Never treat it as a release; it exists so `main` can be tested. |
The same channels exist for every published package, including the nested
example component image.
## Compatibility matrix
Values from [versions.env](../versions.env) at the current identity:
| Component | Pin | Source |
| --- | --- | --- |
| Upstream DSH source | `DSH_REPO` | [versions.env](../versions.env) |
| DSH ref | `dsh-v0.2.1-rc.5-r6` | `DSH_VERSION` |
| DSH commit | `b830ec950faba0a46dda564be3e2d854afed94f1` | `DSH_COMMIT`; the main image build fails unless the clone's `HEAD` matches |
| Repository revision | `(empty)` | `RELEASE_REVISION` |
| pnpm | `11.28.5` | `PNPM_VERSION` |
| Node build image | `docker.io/node:24-trixie@sha256:8c761eca2bf7eec965f946cf47b8090143cb41f45f2f8699af0cc0298d2d2d19` | `NODE_BUILD_IMAGE`, digest-pinned |
| Node runtime image | `docker.io/node:24-trixie-slim@sha256:54f84b4cf8432b21aaffd5d3ce8d41e12cea0d10d33c09d2f3c9879d49d6b446` | `NODE_RUNTIME_IMAGE`, digest-pinned |
| Gateway Python image | `docker.io/python:3.12-slim@sha256:aa943ce411423c7582288f18707ff66eddcbaeb510502d747a72aeb34f30d159` | `GATEWAY_PYTHON_IMAGE`, digest-pinned |
| agent-sandbox (CRDs) | `v1.0.9` | `AGENT_SANDBOX_VERSION`, used by the examples and CI dry-run |
| Kubernetes Gateway API | `v1.2.1` | `GATEWAY_API_VERSION`, used by the SSO exposure variant |
| oauth2-proxy | `quay.io/oauth2-proxy/oauth2-proxy:v7.15.5@sha256:8498b0d0ef0a7b29686414000a08aee467f02d0299c9ed1e006a8f33fc017916` | `OAUTH2_PROXY_IMAGE`, digest-pinned and used by the SSO exposure variant |
| Main image | `ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox` | `GHCR_IMAGE` |
| Gateway image | `ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway` | `GATEWAY_IMAGE`, nested below `GHCR_IMAGE` |
Every container base image is referenced as `name:tag@sha256:<index digest>`, not
by tag alone: `check-versions.sh` rejects a pin without a digest, so a re-pushed
tag cannot change what we build from. The example gateway's Python dependencies
are fully hash-pinned in
[requirements.txt](../examples/dsh-agents-as-a-service/gateway/requirements.txt)
and
[requirements-dev.txt](../examples/dsh-agents-as-a-service/gateway/requirements-dev.txt)
and installed with `pip install --require-hashes`.
Example component images are always nested below the main package:
`<GHCR_IMAGE>/<example>/<component>`. `check-versions.sh` rejects an unrelated
top-level package.
## Bumping versions
```bash
make set-version DSH_VERSION=dsh-vX.Y.Z RELEASE_REVISION=rN
make check
```
`set-version` updates [versions.env](../versions.env) through
[scripts/set-version.sh](../scripts/set-version.sh) and leaves every consuming
file to derive from it. Never edit the Dockerfile, manifests, Makefile or
workflows with a literal version.
- New upstream DSH ref: change `DSH_VERSION` and clear `RELEASE_REVISION`.
`set-version` resolves the new ref's commit with `git ls-remote` (peeled `^{}`
commit first, tag ref as fallback) and rewrites `DSH_COMMIT` in `versions.env`,
the Dockerfile ARG default and `.env.sample`; the build then verifies it.
- Repository-only change at the same DSH ref: keep `DSH_VERSION` and `DSH_COMMIT`,
set `RELEASE_REVISION=r(N+1)`.
- Base images (Node and Python) are bumped by hand: change the tag *and* the
`@sha256:` digest in `versions.env` together, then run `make check` and a build.
There is deliberately no automated digest refresh, because a silent base-image
change is exactly what digest pinning exists to prevent.
## Release checklist
1. `main` is green and `make check`, `make test`, `make lint` pass locally.
2. `make image` and `make gateway` build; smoke-test the images.
3. Bump the pins with `make set-version` (commit that change through a PR).
4. `make release-dry-run` — verifies the identity, prints the image names, checks
the tag does not already exist, and previews the generated notes.
5. Create and push the annotated `dsh-*` release tag (or dispatch
`release.yaml` with an existing tag). The workflow validates that the tag
matches [versions.env](../versions.env) and that the upstream ref still
resolves to `DSH_COMMIT` before building anything.
6. The workflow builds and pushes multi-arch images for the main image and the
example components (the identity tag plus `latest`), creates the GitHub
Release from the git-cliff notes, and opens a `chore(changelog)` PR that
refreshes `CHANGELOG.md` on `main`. The publish log shows each image manifest
and digest.
7. Merge the changelog PR. Verify:
`docker buildx imagetools inspect <image>:<identity>` shows both platforms;
the GitHub Release references the tag, and the workflow log shows the digest.
8. Announce the identity and the digest; deployments pin the digest.
## Rollback
Tags and release identities are **immutable**, so a bad release is never re-tagged
or deleted. Fix forward:
1. Fix the problem on a `hotfix/<slug>` branch (see [gitflow](gitflow.md)).
2. Publish a new identity with the next revision, `-r(N+1)`, at the same
`DSH_VERSION`.
3. Point deployments at the new digest; keep the broken tag for audit history.
Emergency rollback to the previous release is also a re-deploy of a **previous
digest**, not a move of `latest`.
## Changelog
`CHANGELOG.md` is generated by [git-cliff](https://git-cliff.org) from
Conventional Commits, configured in [cliff.toml](../cliff.toml):
```bash
make changelog # regenerate CHANGELOG.md locally
```
- Do not edit `CHANGELOG.md` by hand; the release workflow regenerates it and opens a PR to refresh it on `main`.
- Commit messages and PR titles must be Conventional Commits so they are grouped
correctly (Added / Fixed / Changed / Documentation / Build / CI / Tests /
Maintenance).
- The changelog groups commits since the previous `dsh-*` tag; release notes use
the same rendering.
- The changelog is excluded from the version-token check because it records
historical identities on purpose.
## Digest pinning
Supply-chain inputs are pinned to immutable content, not to mutable tags:
- **Build inputs**: every container base image is referenced by multi-arch index
digest in [versions.env](../versions.env), and the example gateway's Python
dependencies are hash-pinned (`--require-hashes`).
- **Upstream source**: the Dockerfile clones `DSH_VERSION` and fails unless the
clone's `HEAD` equals `DSH_COMMIT`, so a moved upstream tag cannot change what
a rebuild packages.
Release identities are immutable in theory; deployments should still pin the
_resolved_ digest for reproducibility and supply-chain safety:
```bash
docker buildx imagetools inspect ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.5-r6
# then, in the deployment:
# image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox@sha256:<digest>
```
The example manifests use the identity tag so they stay readable and upgrade with
`versions.env`; production overlays should pin the digest. Release images carry
SBOM and provenance attestations (see [SECURITY.md](../SECURITY.md)).

69
docs/workflow-security.md Normal file
View File

@@ -0,0 +1,69 @@
# Workflow security (GitHub Actions)
The CI/CD surface lives entirely in [`.github/workflows/`](../.github/workflows/)
and runs on GitHub-hosted runners. This page describes the controls those
workflows enforce and records the most recent audit. They complement the threat
notes in [SECURITY.md](../SECURITY.md) and the release procedure in
[versioning.md](versioning.md).
## Controls
| Control | Where | What it does |
| --- | --- | --- |
| Least-privilege tokens | Every workflow | A top-level `permissions:` block defaults to read-only; `packages: write`, `contents: write`, `id-token: write` and `attestations: write` are granted only to the jobs that publish images, attestations or releases. |
| Pinned actions | Every `uses:` | Third-party actions and reusable workflows are pinned to a full 40-character commit SHA (the version is kept in the trailing comment). Dependabot updates the pins weekly. The only unpinned reference is the in-repository `./.github/workflows/_build-image.yaml`. |
| No privileged untrusted triggers | Trigger blocks | No workflow uses `pull_request_target` or `workflow_run`. PR-triggered workflows run untrusted code with a read-only token and no secrets. |
| Untrusted input is data | `release.yaml` | The release tag (from a tag push or `workflow_dispatch` input) is passed through `env:` and read as a shell variable; it is never interpolated into a `run:` script, so metacharacters in a tag name cannot execute. |
| Checkout credential hygiene | Checkout steps | `persist-credentials: false` is set wherever the job does not need to push. The changelog job keeps credentials because it pushes a branch and opens a PR with `GITHUB_TOKEN`. |
| Download integrity | `ci.yaml`, `release.yaml` | actionlint, hadolint and git-cliff are downloaded with `curl --proto '=https'` and verified against a pinned SHA-256 with `sha256sum -c` before execution. |
| Cluster manifest integrity | `ci-manifests.yaml`, `e2e-kind.yaml` | The agent-sandbox and Gateway API release manifests are downloaded and checked against the `*_MANIFEST_SHA256` pins in [versions.env](../versions.env) before `kubectl apply`. |
| Log and artifact hygiene | `ci-image.yaml`, `e2e-kind.yaml` | Pod logs are truncated and token material is redacted before logs are printed or diagnostic files are uploaded as artifacts. |
| Supply-chain monitoring | Repository settings | Dependabot (GitHub Actions, pip, Docker), CodeQL advanced setup and the dependency-review action run on every applicable change. CODEOWNERS covers the whole repository, including `.github/workflows/`. |
## Maintaining the pins
- The tool versions and their SHA-256 digests live together in the install step
(`ACTIONLINT_SHA256`, `HADOLINT_SHA256`, `GIT_CLIFF_SHA256`); update both
values when bumping a version.
- Control-plane versions are in `versions.env`; update
`AGENT_SANDBOX_MANIFEST_SHA256` or `GATEWAY_API_MANIFEST_SHA256` together with
the corresponding version, otherwise CI fails the digest check by design.
- GitHub Actions pins are updated by Dependabot; never replace a SHA with a tag.
## Audit record (2026-10-30)
Scope: all 11 workflows under `.github/workflows/`, plus `.github/CODEOWNERS`
and `.github/dependabot.yml`. Findings are grouped by the categories in
GitHub's [Secure use reference](https://docs.github.com/en/actions/reference/security/secure-use).
| # | Category | File | Severity | Finding | Status |
| --- | --- | --- | --- | --- | --- |
| 1 | Script injection | `release.yaml` | High | Tag name / dispatch input interpolated into a privileged `run:` block | Fixed: passed through `env` |
| 2 | Token permissions | `release.yaml` | Medium | `meta` and `tests` inherited workflow-level write scopes | Fixed: workflow default is `contents: read` |
| 3 | Token permissions | `build-main.yaml`, `build-gateway.yaml` | Low | `meta` inherited `packages`/`id-token`/`attestations` write | Fixed: workflow default is `contents: read` |
| 4 | Checkout | `_build-image.yaml` | Low | Checkout persisted the token in `.git/config` | Fixed: `persist-credentials: false` |
| 5 | Checkout | `codeql.yml` | Low | Checkout persisted the token in `.git/config` | Fixed: `persist-credentials: false` |
| 6 | Supply chain | `ci.yaml` | Medium | actionlint and hadolint were executed without a checksum | Fixed: pinned SHA-256 verified before install |
| 7 | Supply chain | `release.yaml` | Medium | git-cliff was executed without a checksum in the release job | Fixed: pinned SHA-256 verified before install |
| 8 | Supply chain | `ci-manifests.yaml`, `e2e-kind.yaml` | Medium | Upstream CRD/controller manifests applied without integrity checks | Fixed: pinned SHA-256 in `versions.env` |
| 9 | Secret hygiene | `e2e-kind.yaml` | Low | Raw gateway logs uploaded in a 7-day artifact | Fixed: `token=` values redacted |
| 10 | Repository controls | Repository settings | Low | No environment approval gate on publishing; single CODEOWNER | Recommendation: see below |
Checked and clean: no `pull_request_target`/`workflow_run`, no self-hosted
runners, all third-party actions pinned, PR workflows read-only, no secrets in
workflow files, no secret interpolation into scripts.
## Residual recommendations (repository settings)
These cannot be enforced from workflow files alone:
- Add a protected `release` environment with required reviewers and attach it to
the `build-main`, `build-gateway` and `publish-release` jobs. This requires the
environment to exist in repository settings first, so it is intentionally not
wired into the workflow yet.
- Keep "Allow GitHub Actions to create and approve pull requests" limited to
creating PRs (the changelog job needs create, not approve).
- Require review from Code Owners for `.github/workflows/` and add a second
owner so workflow changes are never single-person.
- Optionally run OpenSSF Scorecard on a schedule for continuous checks on these
controls.

44
examples/README.md Normal file
View File

@@ -0,0 +1,44 @@
# Examples
Every implementation in this repository is a self-contained example: one
directory, plain `kubectl`, an end-to-end README, and a cleanup command.
Examples consume the [main DSH Web image](../docs/main-image.md); they are
reference material, not libraries.
| Example | What it demonstrates | Needs |
| --- | --- | --- |
| [`examples/dsh-agents-as-a-service/`](dsh-agents-as-a-service/) | The full multi-tenant platform pattern: one `SandboxTemplate` with per-user PVCs, a warm pool, one ~10-line `SandboxClaim` per user, suspend-when-idle / wake-on-return, and a browser gateway with signup, `HttpOnly` sessions, an HTTP + WebSocket proxy and two exposure variants (Ingress, or Gateway API + GitHub SSO). | agent-sandbox with extensions; kind for the e2e script. |
| [`examples/minimal-single-tenant/`](minimal-single-tenant/) | The smallest useful setup: a single `SandboxTemplate` and one `SandboxClaim`, no gateway — reach DSH Web with a launch-token URL through `kubectl port-forward`. | agent-sandbox with extensions. |
The multi-tenant example also documents opt-in gVisor, `nodeSelector`,
NetworkPolicy, extra provider-key hardening and failed-Pod recovery in
[optional hardening](dsh-agents-as-a-service/docs/optional-hardening.md).
## The example contract (short version)
Full rules: [docs/examples.md](../docs/examples.md). `scripts/check-examples.sh`
enforces the layout, and `make check` runs it.
- `README.md` is the walkthrough; `manifests/NN-*.yaml` is applied with plain
kubectl and read in order.
- Optional `manifests/optional/<feature>/` and `manifests/exposure/<variant>/`
each ship a README; exposure variants are mutually exclusive.
- Optional `scripts/` must be executable, idempotent and self-cleaning; `docs/`
holds deeper material.
- Components with a Dockerfile need a README and tests; images are published
nested below the main image (`<registry>/<owner>/<repo>/<example>/<component>`).
- Secrets are `*.yaml.example` with placeholders; every example is listed above
with its literal `examples/<name>/` path.
## Running an example
Each example README has the exact prerequisites, commands, optional hardening,
exposure options and teardown. The multi-tenant example also has a full kind
walkthrough that cleans up after itself:
```bash
examples/dsh-agents-as-a-service/scripts/e2e-kind.sh
```
`kubectl apply -f manifests/` is non-recursive: `optional/` and `exposure/`
subdirectories are applied explicitly.

View File

@@ -1,27 +0,0 @@
# DSH "Agents as a Service" — prerequisites: namespace + shared provider key.
#
# The dsh-provider-keys Secret is deliberately NOT committed with a usable
# value. DEEPSEEK_API_KEY reaches every tenant sandbox (injected via envFrom
# on the SandboxTemplate, stamped before any user exists — a warm-pool
# constraint), so a real key committed here would be shared by every install
# of this example. Generate a fresh value per install (README step 1;
# run-test-kind.sh uses a throwaway):
#
# kubectl -n dsh-demo create secret generic dsh-provider-keys \
# --from-literal=DEEPSEEK_API_KEY="sk-..."
#
# All the infrastructure in this example works without a real key; only
# actual model calls need one.
apiVersion: v1
kind: Namespace
metadata:
name: dsh-demo
---
apiVersion: v1
kind: Secret
metadata:
name: dsh-provider-keys
namespace: dsh-demo
type: Opaque
# stringData:
# DEEPSEEK_API_KEY: "sk-your-key-here"

View File

@@ -1,122 +0,0 @@
# The reusable DSH Web blueprint. Warm pools stamp pod env BEFORE any user
# exists, so per-user secrets cannot live in pod env; the shared provider key
# is injected via envFrom, and claims that would bypass the warm pool are
# rejected outright (see 40-claim-rejected.yaml).
apiVersion: extensions.agents.x-k8s.io/v1beta1
kind: SandboxTemplate
metadata:
name: dsh-web
namespace: dsh-demo
spec:
# Claims must not inject env or volumeClaimTemplates: with Disallowed, such
# claims are REJECTED instead of silently taking the cold (non-warm-pool)
# provisioning path. Demonstrated in step 6.
envVarsInjectionPolicy: Disallowed
volumeClaimTemplatesPolicy: Disallowed
# Per-sandbox headless Service -> stable DNS (sandbox.status.serviceFQDN).
# The gateway routes each user to their DSH Web instance by this name.
service: true
# No gateway pod scopes ingress in this standalone demo, so the
# NetworkPolicy is left unmanaged. A production platform would use
# networkPolicyManagement: Managed and restrict ingress to its gateway on
# :3080 — with the shared provider key, that NetworkPolicy is the per-user
# isolation boundary.
networkPolicyManagement: Unmanaged
podTemplate:
spec:
# DSH Web is multi-arch (amd64 + arm64); no arch nodeSelector needed.
restartPolicy: "OnFailure"
securityContext:
runAsUser: 1000 # the 'node' user in the DSH image
runAsGroup: 1000
fsGroup: 1000
runAsNonRoot: true
# DSH's credentials plugin refuses to start when .credentials.yaml has
# any group/other permission bit (it creates the file 0600). The
# default kubelet fsGroup policy ("Always") recursively ORs 0660 into
# every file each time the pod starts, so a resumed sandbox comes back
# with 0660 and crash-loops before the gateway can proxy to it.
# OnRootMismatch skips that walk once the volume root already carries
# the fsGroup ownership/mode applied on the first mount.
# Keep in sync with deploy/k8s/sandbox-template.yaml.
fsGroupChangePolicy: OnRootMismatch
containers:
- name: dsh-web
image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.2
imagePullPolicy: IfNotPresent
ports:
- name: web
containerPort: 3080
envFrom:
- secretRef:
name: dsh-provider-keys
env:
# Empty => DSH derives links from the request Host, which the
# gateway forwards (plus X-Forwarded-*). Point this at the
# gateway base URL if you want hard-coded absolute links instead.
- name: DSH_PUBLIC_URL
value: ""
# DSH Host/Origin-fences /api. The gateway forwards the browser's
# Host, so DSH must trust the public gateway host. Empty is right
# for port-forward/localhost; for the optional Ingress set this to
# the same host as 60-ingress.yaml (e.g. dsh.example.com).
- name: DSH_TRUSTED_HOSTS
value: ""
# Probes target /favicon.svg: DSH gates / behind its launch-token
# cookie and answers 401 to an unauthenticated GET, which would
# otherwise fail readiness and then get the pod killed by liveness.
# /favicon.svg is served without auth and is the cheapest liveness
# signal that proves the HTTP server is actually serving requests.
#
# Gate liveness/readiness behind a startup probe: a cold start on a
# small node takes tens of seconds to load plugins and start serving
# (a resumed sandbox starts cold too). Without it the kubelet can kill
# DSH mid-startup, and a SIGTERM'd DSH exits 0, which the sandbox
# controller records as finished. 24 x 5s = 120s, matching the
# gateway's WAKE_TIMEOUT.
startupProbe:
httpGet:
path: /favicon.svg
port: 3080
periodSeconds: 5
failureThreshold: 24
readinessProbe:
httpGet:
path: /favicon.svg
port: 3080
initialDelaySeconds: 5
periodSeconds: 5
failureThreshold: 30
livenessProbe:
httpGet:
path: /favicon.svg
port: 3080
initialDelaySeconds: 15
periodSeconds: 15
failureThreshold: 3
# The container only needs to serve HTTP; drop every capability and
# disallow privilege escalation. DSH writes only to DSH_HOME (PVC).
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
seccompProfile:
type: RuntimeDefault
resources:
requests: {cpu: 250m, memory: 512Mi}
limits: {cpu: "1", memory: 1Gi}
volumeMounts:
- name: dsh-home
mountPath: /home/node/.dsh # DSH_HOME: sessions, config, state
volumeClaimTemplates:
- metadata:
name: dsh-home
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 5Gi
# storageClassName omitted => cluster default (kind & GKE both work)

View File

@@ -1,163 +0,0 @@
# The gateway: control+data plane that turns the resources into a usable
# browser-facing service. It is a Starlette ASGI app (see gateway/) listening
# on :8080 and exposing:
# GET /healthz unauthenticated health probe
# GET / landing page (signup / token login)
# POST /users, POST /login, ... session cookie auth
# /u/<user>/... HTTP + WebSocket proxy to the user's DSH
#
# Image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway
# For local kind development you can build and load a local tag instead, then
# change `image` below to it:
# docker build -t dsh-aaas-gateway:demo gateway/
# kind load docker-image dsh-aaas-gateway:demo
apiVersion: v1
kind: ServiceAccount
metadata:
name: dsh-gateway
namespace: dsh-demo
---
# Least privilege:
# - claims are created/deleted per user (signup/delete);
# - sandboxes are read, and patched only to flip operatingMode (suspend/resume);
# - pods are read, and pods/log is read to capture DSH's launch token from
# the startup log line (`dsh web: ...?token=...`).
# configmaps are not needed: all state is in memory / claim annotations.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: dsh-gateway
namespace: dsh-demo
rules:
- apiGroups: ["extensions.agents.x-k8s.io"]
resources: ["sandboxclaims"]
verbs: ["get", "list", "create", "delete"]
- apiGroups: ["agents.x-k8s.io"]
resources: ["sandboxes"]
verbs: ["get", "list", "patch"]
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: dsh-gateway
namespace: dsh-demo
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: dsh-gateway
subjects:
- kind: ServiceAccount
name: dsh-gateway
namespace: dsh-demo
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: dsh-gateway
namespace: dsh-demo
spec:
# Single replica BY DESIGN: the idle clock, in-flight counters and the
# per-(user, pod, Host) DSH cookie cache are in-process memory. Running two
# replicas would fragment that state (double suspend/resume races, repeated
# log-token handshakes). Scale only after moving state to a shared store.
replicas: 1
selector:
matchLabels:
app: dsh-gateway
template:
metadata:
labels:
app: dsh-gateway
spec:
serviceAccountName: dsh-gateway
securityContext:
runAsNonRoot: true
runAsUser: 10000 # matches USER in gateway/Dockerfile
runAsGroup: 10000
seccompProfile:
type: RuntimeDefault
containers:
- name: gateway
image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway
imagePullPolicy: IfNotPresent
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: ["ALL"]
ports:
- name: http
containerPort: 8080
env:
- name: NAMESPACE
value: dsh-demo
- name: POOL
value: dsh-pool
# Seconds a sandbox may sit idle before the sweeper suspends it
# (pod deleted, PVC kept), and the max time a proxy request waits
# for a suspended sandbox to become Ready again. Only claims this
# gateway created are swept: a claim applied by hand from the
# resource-layer manifests carries no aaas.example.com/managed-by
# annotation and is never suspended.
- name: IDLE_TIMEOUT
value: "60"
- name: WAKE_TIMEOUT
value: "120"
# Container name inside the per-user sandbox pods whose logs carry
# the DSH launch token.
- name: DSH_CONTAINER
value: dsh-web
# Cookie dsh_gw_session is HttpOnly + SameSite=Lax. Set to "true"
# when this Service is reached over HTTPS (e.g. the optional
# 60-ingress.yaml with TLS); "false" keeps the plain-HTTP demo
# usable via port-forward.
- name: COOKIE_SECURE
value: "false"
# Reject proxied request bodies larger than this (413).
- name: MAX_BODY_MB
value: "32"
# /healthz is unauthenticated and returns 200 without touching the
# cluster, so it is safe for both readiness and liveness.
readinessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 5
periodSeconds: 5
failureThreshold: 3
livenessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 10
periodSeconds: 10
failureThreshold: 3
resources:
requests: {cpu: 50m, memory: 128Mi}
limits: {cpu: 500m, memory: 256Mi}
volumeMounts:
# readOnlyRootFilesystem leaves no writable HOME/tmp; the image sets
# HOME=/tmp, so back it with scratch space. No state lives here.
- name: tmp
mountPath: /tmp
volumes:
- name: tmp
emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
name: dsh-gateway
namespace: dsh-demo
spec:
selector:
app: dsh-gateway
ports:
- port: 8080
targetPort: 8080

View File

@@ -1,46 +0,0 @@
# OPTIONAL browser entry point: expose the gateway through Traefik.
#
# Requires 50-gateway.yaml (Service dsh-gateway:8080) and a Traefik install
# with an IngressClass named "traefik" (k3s ships one). Replace the placeholder
# host below with a name that resolves to your ingress address, and either
# supply a real TLS Secret or delete the `tls:` block for plain HTTP.
#
# Two things must match this host:
# 1. SandboxTemplate (10-sandboxtemplate.yaml): set DSH_TRUSTED_HOSTS to
# this host so DSH accepts the forwarded Host/Origin on /api.
# 2. Gateway Deployment (50-gateway.yaml): set COOKIE_SECURE to "true"
# once this Ingress serves HTTPS, so the session cookie is Secure.
#
# WebSocket support is REQUIRED: the browser talks to /u/<user>/... and DSH
# uses WebSockets for live session traffic. Traefik upgrades WS/SSE
# transparently for ordinary HTTP routers, but do NOT attach buffering
# middlewares (e.g. compress, buffering) to this router — they break streaming
# and upgrades. `path: /` Prefix covers the landing page and every per-user
# proxy path, so no per-sandbox Ingress is needed.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: dsh-gateway
namespace: dsh-demo
# Typical Traefik + cert-manager wiring (uncomment and adapt):
# annotations:
# traefik.ingress.kubernetes.io/router.entrypoints: websecure
# cert-manager.io/cluster-issuer: letsencrypt-prod
annotations: {}
spec:
ingressClassName: traefik
tls:
- hosts:
- dsh.example.com # placeholder
secretName: dsh-example-tls # placeholder; create/PKI-provision this Secret
rules:
- host: dsh.example.com # placeholder
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: dsh-gateway
port:
number: 8080

View File

@@ -2,26 +2,54 @@
This example distills the upstream
[hermes-agents-as-a-service](https://github.com/kubernetes-sigs/agent-sandbox/tree/main/examples/hermes-agents-as-a-service)
platform pattern into pure agent-sandbox resources + kubectl, running **DeepSeek Harness Web
(DSH)** instead of Hermes. Every user of such a platform gets a personal, long-lived DSH Web
instance that is **provisioned in ~2 seconds from a warm pool, suspended when idle to save
cost, and woken transparently when the user returns** — with all state (conversations,
sessions, config) surviving on a per-user PVC at `/home/node/.dsh`.
platform pattern into pure agent-sandbox resources + kubectl, running **DeepSeek
Harness Web (DSH)** instead of Hermes. Every user of such a platform gets a
personal, long-lived DSH Web instance that is **provisioned in ~2 seconds from a
warm pool, suspended when idle to save cost, and woken transparently when the
user returns** — with all state (conversations, sessions, config, and the agent
workspace) surviving on the per-user PVC at `/home/node/.dsh` + `/workspace`.
Suspend-when-idle plus the warm pool keep cost proportional to use: a return
after absence pays a short cold-start pause, not an always-on instance.
It ships in two layers:
It ships in three layers:
1. **The resource layer** (steps 1–7): pure manifests + kubectl, so every mechanism is visible.
2. **The service layer** (step 8): a small included **gateway** (`gateway/`) that makes the
platform browser-usable — a signup page/API, `HttpOnly` cookie sessions, a `/u/{user}/**`
proxy for HTTP **and WebSocket** that transparently **wakes suspended instances on
connect**, and an idle sweeper that suspends them again. The gateway performs the DSH
launch-token handshake server-side, so the browser never sees the token.
1. **The resource layer** (steps 1–7): pure manifests + kubectl, so every
mechanism is visible.
2. **The service layer** (step 8): a small included **gateway** (`gateway/`) that
makes the platform browser-usable — a signup page/API, revocable server-side
**opaque-cookie sessions**, auth rate limiting and an optional signup invite
code, a `/u/{user}/**` proxy for HTTP **and WebSocket** that transparently
**wakes suspended instances on connect**, and an idle sweeper that suspends
them again. The gateway performs the DSH launch-token handshake server-side,
so the browser never sees the token.
3. **The exposure layer** (step 9): choose exactly one way in — a plain Traefik
Ingress, or Gateway API + oauth2-proxy (GitHub SSO).
All manifests live under [`manifests/`](manifests/):
```
manifests/
+-- 00-prereqs.yaml # namespace + empty provider-key Secret
+-- 10-sandboxtemplate.yaml # the DSH Web blueprint
+-- 20-sandboxwarmpool.yaml # 2 pre-warmed spares
+-- 30-claim-alice.yaml # one user
+-- 40-claim-rejected.yaml # negative demo: env injection rejected
+-- 50-gateway.yaml # Deployment/RBAC/Service for the gateway
+-- optional/
| +-- README.md # opt-in manifests, applied by path
| +-- network-policy/10-sandbox-networkpolicy.yaml
+-- exposure/
+-- README.md # the two exposure variants, when to use which
+-- ingress/60-ingress.yaml
+-- ingress/61-networkpolicy.yaml
+-- gateway-api-sso/70-gateway.yaml ... 75-networkpolicies.yaml
```
What it demonstrates, feature by feature:
| agent-sandbox feature | Role in the platform |
| --- | --- |
| `SandboxTemplate` with `volumeClaimTemplates` | Identical DSH pods; per-user state on a PVC at `/home/node/.dsh` |
| `SandboxTemplate` with `volumeClaimTemplates` | Identical DSH pods; per-user state at `/home/node/.dsh` and the agent workspace at `/workspace`, on one PVC (the workspace is a `subPath` mount, so DSH state stays private) |
| `envVarsInjectionPolicy` / `volumeClaimTemplatesPolicy: Disallowed` | Claims that would bypass the warm pool are **rejected loudly**, not cold-started silently |
| `service: true` | Stable per-sandbox DNS (`status.serviceFQDN`) for the gateway to route on |
| `SandboxWarmPool` | Signup latency is pre-paid: image pull, PVC provision, DSH boot all happen before any user exists |
@@ -34,64 +62,101 @@ What it demonstrates, feature by feature:
- **Multi-arch**: the DSH image is built for amd64 and arm64, so there is no
`kubernetes.io/arch: amd64` nodeSelector.
- **Single port**: DSH Web serves everything on `3080`; the gateway proxies to one upstream
instead of splitting `/v1/*` (API) from the dashboard.
- **One shared secret**: only `DEEPSEEK_API_KEY` is injected (via `envFrom`); there is no
dashboard basic-auth, session-cookie secret, or API-server key.
- **Launch-token handshake**: DSH gates `/` behind a token it prints only in its logs. The
gateway reads it, performs the handshake, and keeps the resulting auth cookie server-side;
the browser authenticates to the gateway with its own cookie. See
- **Single port**: DSH Web serves everything on `3080`; the gateway proxies to one
upstream instead of splitting `/v1/*` (API) from the dashboard.
- **Shared provider keys**: `DEEPSEEK_API_KEY` plus any keys in the optional
`dsh-provider-keys-extra` Secret are injected via `envFrom` into every
sandbox. There is no per-tenant key, dashboard basic-auth, signed-session
secret (sessions are opaque server-side ids), or API-server key.
- **Launch-token handshake**: DSH gates `/` behind a token it prints only in its
logs. The gateway reads it, performs the handshake, and keeps the resulting
auth cookie server-side; the browser authenticates to the gateway with its own
opaque session cookie, never the token. See
[How the DSH handshake works](#how-the-dsh-handshake-works).
- **Host allow-list**: DSH rejects unknown `Host` headers. The gateway forwards the browser's
`Host`, so the public hostname must be added to `DSH_TRUSTED_HOSTS` in the template.
- **Probes target `/favicon.svg`**: `/` returns `401` without the launch token, so a probe on
`/` crash-loops the pod. `/favicon.svg` is unauthenticated, and a `startupProbe` gates
liveness/readiness so a slow cold start (plugin activation, or a resumed sandbox) is not
restarted mid-boot.
- **Host allow-list**: DSH rejects unknown `Host` headers. The gateway forwards
the browser's `Host`, so the public hostname must be added to
`DSH_TRUSTED_HOSTS` in the template.
- **Probes target `/favicon.svg`**: `/` returns `401` without the launch token, so
a probe on `/` crash-loops the pod. `/favicon.svg` is unauthenticated, and a
`startupProbe` gates liveness/readiness so a slow cold start (plugin
activation, or a resumed sandbox) is not restarted mid-boot.
## Prerequisites
Run the walkthrough from this example's directory (`examples/dsh-agents-as-a-service/`).
- A Kubernetes cluster with a default StorageClass (kind and GKE both work as-is).
- agent-sandbox with extensions installed:
```bash
VERSION=v1.0.7
kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/download/${VERSION}/sandbox-with-extensions.yaml
. ../../versions.env
kubectl apply -f "https://github.com/kubernetes-sigs/agent-sandbox/releases/download/${AGENT_SANDBOX_VERSION}/sandbox-with-extensions.yaml"
kubectl -n agent-sandbox-system rollout status deployment --timeout=180s
```
Or run everything below in one shot with `./run-test-kind.sh` (expects a kind cluster + the
manifest above already applied; it creates and deletes the namespace itself).
For step 9's SSO variant you additionally need the Gateway API CRDs and a
Gateway implementation (`gatewayClassName: traefik` in the manifests); see the
[exposure README](manifests/exposure/README.md).
- Optional: run the whole resource + gateway walkthrough in one shot with
[`scripts/e2e-kind.sh`](scripts/e2e-kind.sh). It expects a kind cluster with
agent-sandbox installed, then creates and deletes the `dsh-demo` namespace
itself. It stops after the gateway layer (step 8); exposure is not applied.
## 1. Install the platform resources
First the namespace, then the shared provider key. The key reaches **every** tenant sandbox
(a warm-pool constraint: pod env is stamped before any user exists), so it is **never
committed** — generate a fresh value per install:
First the namespace, then the shared provider keys. The keys reach **every**
tenant sandbox (a warm-pool constraint: pod env is stamped before any user
exists), so they are **never committed** — generate fresh values per install.
`00-prereqs.yaml` already creates the empty `dsh-provider-keys` Secret, so use
an upsert:
```bash
kubectl apply -f 00-prereqs.yaml
kubectl apply -f manifests/00-prereqs.yaml
kubectl -n dsh-demo create secret generic dsh-provider-keys \
--from-literal=DEEPSEEK_API_KEY="sk-..."
kubectl apply -f 10-sandboxtemplate.yaml -f 20-sandboxwarmpool.yaml
--from-literal=DEEPSEEK_API_KEY="sk-..." \
--dry-run=client -o yaml | kubectl apply -f -
```
Watch the two warm spares boot (first time pulls the image — subsequent operations never pay
that again; that is the point of the pool):
Optional: additional providers (OpenRouter, Gemini, NVIDIA, OpenAI, Anthropic,
...) go in a second Secret. It is referenced with `optional: true`, so the
sandboxes start without it, and every key becomes an env var:
```bash
kubectl -n dsh-demo create secret generic dsh-provider-keys-extra \
--from-literal=OPENROUTER_API_KEY="sk-or-..." \
--from-literal=GEMINI_API_KEY="..." \
--from-literal=NVIDIA_API_KEY="nvapi-..." \
--dry-run=client -o yaml | kubectl apply -f -
```
Create both Secrets **before** the template/warm pool. A later Secret change
does not update running pods: delete the warm-pool spares and recreate claimed
pods to pick up new keys. Keys are shared by all tenants; a key saved in DSH's
credential store wins over env. See
[optional hardening](docs/optional-hardening.md) for the full env-name list and
the refresh procedure.
```bash
kubectl apply -f manifests/10-sandboxtemplate.yaml -f manifests/20-sandboxwarmpool.yaml
```
Watch the two warm spares boot (first time pulls the image — subsequent operations
never pay that again; that is the point of the pool):
```bash
kubectl -n dsh-demo get sandboxes -l agents.x-k8s.io/warm-pool-sandbox -w
```
Wait until both show `READY: True`. Each spare already has its PVC bound and DSH Web serving
on port `3080`.
Wait until both show `READY: True`. Each spare already has its PVC bound and DSH
Web serving on port `3080`.
## 2. "Signup": claim an instance (~2s)
A user signing up is one small claim — `warmPoolRef` plus a pod label:
```bash
kubectl apply -f 30-claim-alice.yaml
kubectl apply -f manifests/30-claim-alice.yaml
kubectl -n dsh-demo wait sandboxclaim dsh-alice --for=condition=Ready --timeout=60s
```
@@ -105,20 +170,21 @@ kubectl -n dsh-demo get sandboxes --show-labels
Three things worth noticing:
- **The sandbox keeps its pool-generated name** (`dsh-pool-…`), not the claim's name —
adoption re-labels an existing warm sandbox rather than creating one. A platform must always
resolve user → sandbox through `claim.status.sandbox.name`.
- The `agents.x-k8s.io/warm-pool-sandbox` label is **gone** from the adopted sandbox (it left
the pool's inventory), and the pool has already started replenishing back to 2 spares.
- **The sandbox keeps its pool-generated name** (`dsh-pool-…`), not the claim's
name — adoption re-labels an existing warm sandbox rather than creating one. A
platform must always resolve user → sandbox through `claim.status.sandbox.name`.
- The `agents.x-k8s.io/warm-pool-sandbox` label is **gone** from the adopted
sandbox (it left the pool's inventory), and the pool has already started
replenishing back to 2 spares.
- The pod carries the claim's `additionalPodMetadata` label
(`sandbox.users.io/dsh-user=alice`), and
`kubectl -n dsh-demo get sandbox $SB -o jsonpath='{.status.serviceFQDN}'` shows the stable
DNS name the gateway proxies to.
`kubectl -n dsh-demo get sandbox $SB -o jsonpath='{.status.serviceFQDN}'` shows
the stable DNS name the gateway proxies to.
## 3. Talk to the instance
DSH Web needs its launch token on first contact — a bare `/` returns `401`. The token is in the
pod logs:
DSH Web needs its launch token on first contact — a bare `/` returns `401`. The
token is in the pod logs:
```bash
kubectl -n dsh-demo logs "$SB" | grep -m1 '?token='
@@ -127,23 +193,44 @@ kubectl -n dsh-demo port-forward "pod/$SB" 18081:3080
# open http://localhost:18081/?token=... (from the log line)
```
> **Local port note:** DSH Web listens on `3080` **in-cluster**. If `3080` is already in use
> on your workstation (e.g. by a local DSH), forward to a different local port (here `18081`),
> never `3080`. `localhost` is always in DSH's trusted-host list, so the forwarded request is
> accepted.
> **Local port note:** DSH Web listens on `3080` **in-cluster**. If `3080` is
> already in use on your workstation (e.g. by a local DSH), forward to a
> different local port (here `18081`), never `3080`. `localhost` is always in
> DSH's trusted-host list, so the forwarded request is accepted.
> **Why the pod stays up:** this claim was applied by hand, so it carries no gateway ownership
> annotation (`aaas.example.com/managed-by`). The gateway's idle sweeper only suspends claims it
> created through `POST /users`; the resource-layer claims from these manifests are never
> touched. You can deploy the gateway (step 8) and still keep a sandbox you are port-forwarding
> to.
> **Why the pod stays up:** this claim was applied by hand, so it carries no
> gateway ownership annotation (`aaas.example.com/managed-by`). The gateway's idle
> sweeper only suspends claims it created through `POST /users`; the resource-layer
> claims from these manifests are never touched. You can deploy the gateway
> (step 8) and still keep a sandbox you are port-forwarding to.
## 4. Write state on the PVC
```bash
kubectl -n dsh-demo exec "$SB" -- printenv HOME # /workspace
kubectl -n dsh-demo exec "$SB" -- sh -c 'echo "remember me" > /home/node/.dsh/marker.txt'
kubectl -n dsh-demo exec "$SB" -- sh -c 'echo "remember me too" > /workspace/marker.txt'
```
`/workspace` (the DSH image's `WORKDIR`, where agents keep files) is the same
`dsh-home` PVC mounted again with `subPath: workspace`. The two trees stay
separate, so `.credentials.yaml` and the session store are never part of the
agent-visible workspace.
The template also sets `HOME=/workspace`. DSH's directory picker (the folder
explorer used to choose a workspace) has no start-path setting: when it opens,
its browse backend lists `$HOME`, so with the image default (`/home/node`, which
is not on a volume) it would start outside the persisted tree. Pointing `HOME`
at the workspace makes the picker open on it (labelled *Home* in its breadcrumbs)
and makes `~` in agent commands mean `/workspace`. The trade-off is that tools
keeping caches or config under `$HOME` (`npm`/`pnpm`, `uv`, `git`, `gh`, and
DSH's global `~/.agents` instructions and skills) write dotfiles onto the PVC
inside the agent workspace, where the agent and the file explorer can see them.
`DSH_HOME` is explicit (`/home/node/.dsh`), so sessions, config and credentials
are unaffected. Pod env is stamped when a sandbox is created: a sandbox from
before this setting keeps `HOME=/home/node` until it is recreated (see
[Troubleshooting](#troubleshooting)).
## 5. Suspend: delete the pod, keep the PVC + Service
```bash
@@ -159,22 +246,21 @@ kubectl -n dsh-demo get svc "$SB" # still here
kubectl -n dsh-demo patch sandbox "$SB" --type merge -p '{"spec":{"operatingMode":"Running"}}'
kubectl -n dsh-demo wait sandbox "$SB" --for=condition=Ready --timeout=180s
kubectl -n dsh-demo exec "$SB" -- cat /home/node/.dsh/marker.txt # "remember me"
kubectl -n dsh-demo exec "$SB" -- cat /workspace/marker.txt # "remember me too"
```
> **Why resume works:** the pod template sets `fsGroupChangePolicy: OnRootMismatch`. DSH requires
> its `.credentials.yaml` to be `0600`, while the kubelet's default fsGroup policy widens every
> file to `0660` on each pod start — a resumed sandbox would crash-loop. See
> [Troubleshooting](#troubleshooting) for the one-time fix on sandboxes created before this
> setting existed.
> **Why resume works:** the pod template sets `fsGroupChangePolicy: OnRootMismatch`;
> without it the kubelet widens `.credentials.yaml` to `0660` and DSH crash-loops.
> See [Troubleshooting](#troubleshooting) for the one-time fix on older sandboxes.
## 7. Injection policy rejects env-carrying claims
```bash
kubectl apply -f 40-claim-rejected.yaml
kubectl apply -f manifests/40-claim-rejected.yaml
kubectl -n dsh-demo get sandboxclaim dsh-mallory \
-o jsonpath='{.status.conditions[?(@.type=="Ready")].reason}'
# EnvVarsInjectionRejected — the claim never bound a sandbox
kubectl delete -f 40-claim-rejected.yaml
kubectl delete -f manifests/40-claim-rejected.yaml
```
## 8. The gateway: signup, browser proxy, idle-suspend, wake-on-connect
@@ -182,63 +268,100 @@ kubectl delete -f 40-claim-rejected.yaml
### Deploy it
```bash
# Published image (on kind, load it into the cluster; on a real cluster it is
# pulled from GHCR):
docker pull ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway
kind load docker-image ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway --name <kind-cluster>
# Published image (on kind, load it into the cluster; on a real cluster the
# kubelet pulls it from GHCR):
GW=ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway:dsh-v0.2.1-rc.5-r6
docker pull "$GW"
kind load docker-image "$GW" --name <kind-cluster>
# ... or build it locally and tag it exactly as 50-gateway.yaml expects:
docker build -t ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway gateway/
kind load docker-image ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway --name <kind-cluster>
# ... or build it locally and tag it exactly as manifests/50-gateway.yaml
# expects (make gateway builds dsh-aaas-gateway:dev):
make -C ../.. gateway
docker tag dsh-aaas-gateway:dev "$GW"
kind load docker-image "$GW" --name <kind-cluster>
kubectl apply -f 50-gateway.yaml
kubectl apply -f manifests/50-gateway.yaml
kubectl -n dsh-demo rollout status deploy/dsh-gateway --timeout=120s
kubectl -n dsh-demo port-forward svc/dsh-gateway 18080:8080
```
> The gateway binds `8080`; the local port-forward uses `18080` and never `3080`.
>
> **Re-deploying after a new image?** The workflow publishes the mutable `:gateway` tag and the
> Deployment uses `imagePullPolicy: IfNotPresent`, so a node that already cached the tag keeps
> the old image across `kubectl apply`/`rollout restart`. Pin the digest for a reproducible
> rollout (the workflow's `Inspect image` step and `docker buildx imagetools inspect` both show
> it):
> **Cookie/TLS guard.** The base manifest serves plain HTTP through port-forward,
> so it sets `COOKIE_SECURE=false` **and** `ALLOW_INSECURE_COOKIES=1`. The gateway
> refuses to start when `COOKIE_SECURE=false` unless
> `ALLOW_INSECURE_COOKIES=1` explicitly acknowledges that the session cookie
> travels in clear text; that prevents an HTTPS deployment from silently serving
> an insecure cookie. Before exposing the gateway over TLS, set
> `COOKIE_SECURE=true` and remove the acknowledgement:
> `kubectl -n dsh-demo set env deploy/dsh-gateway COOKIE_SECURE=true ALLOW_INSECURE_COOKIES-`.
>
> **Image pull policy.** `manifests/50-gateway.yaml` uses
> `imagePullPolicy: IfNotPresent` with the release identity, so a locally built
> image must be loaded into the cluster under exactly that name (kind:
> `kind load`). A node that already cached a different tag is the usual cause of
> "my new build is not running"; pin the digest for a reproducible rollout:
>
> ```bash
> kubectl -n dsh-demo set image deploy/dsh-gateway \
> gateway=ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox@sha256:<digest>
> gateway=ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway@sha256:<digest>
> kubectl -n dsh-demo rollout status deploy/dsh-gateway --timeout=120s
> ```
>
> **Applying over an existing install?** Run `kubectl diff -f 10-sandboxtemplate.yaml`
> first. Cluster-specific fields such as `nodeAffinity` and the warm-pool replica count are
> intentionally not in the repo manifests; preserve them with a targeted patch instead of a
> **Applying over an existing install?** Run
> `kubectl diff -f manifests/10-sandboxtemplate.yaml` first. Cluster-specific
> fields such as `nodeAffinity` and the warm-pool replica count are intentionally
> not in the repo manifests; preserve them with a targeted patch instead of a
> blind apply. Moving the probes to `/favicon.svg` is the crash-loop fix.
>
> The template also sets `fsGroupChangePolicy: OnRootMismatch` (see the note in step 6).
> Existing `Sandbox` objects snapshot their pod template, so a template apply does not update
> them: patch each sandbox's `spec.podTemplate.spec.securityContext.fsGroupChangePolicy` and
> The template also sets `fsGroupChangePolicy: OnRootMismatch` (see the note in
> step 6). Existing `Sandbox` objects snapshot their pod template, so a template
> apply does not update them: patch each sandbox's
> `spec.podTemplate.spec.securityContext.fsGroupChangePolicy` and
> `chmod 600` its `.credentials.yaml`, or recreate it.
>
> The optional gVisor/`nodeSelector` fields are pod-spec changes too: existing
> Sandboxes snapshot them, and warm-pool spares are not replaced by a template
> edit. See [optional hardening](docs/optional-hardening.md) for the patch and
> recreation commands. A provider-key Secret change likewise needs pod
> recreation to take effect.
### Browser flow
Open <http://localhost:18080>. The landing page has a one-field signup form. Enter `bob`, and
Open <http://localhost:18080>. The landing page has a one-field signup form (plus
an invite-code field when `SIGNUP_INVITE_CODE` is configured). Enter `bob`, and
the gateway:
1. creates the `SandboxClaim` over the warm pool,
2. waits for it to become `Ready`,
3. sets an `HttpOnly`, `SameSite=Lax` cookie `dsh_gw_session=<user>:<token>` on the browser,
4. redirects to `/u/bob/`, where the DSH Web UI loads through the gateway.
1. enforces the auth rate limit and, if configured, the signup invite code,
2. creates the `SandboxClaim` over the warm pool,
3. waits for it to become `Ready`,
4. mints a random, server-side session (the cookie value is an opaque `sid`, not
a user name and not a token), stores `sid -> (user, expiry)` in memory, and
sets the `HttpOnly`, `SameSite=Lax` cookie `dsh_gw_session=<sid>` on the
browser (`__Host-dsh_gw_session` when `COOKIE_SECURE=true`),
5. redirects to `/u/bob/`, where the DSH Web UI loads through the gateway.
From then on every request under `/u/bob/` (assets, API calls, SSE, WebSocket) is proxied to
bob's sandbox. The DSH launch token and DSH's own auth cookie stay in the gateway; the browser
only holds the gateway cookie.
From then on every request under `/u/bob/` (assets, API calls, SSE, WebSocket) is
proxied to bob's sandbox. The DSH launch token and DSH's own auth cookie stay in
the gateway; the browser only holds the opaque gateway session id. The cookie is
useless as a bearer token.
The same flow with `curl`, no JavaScript (the session cookie is the only credential):
**Sessions are revocable and mostly short-lived in practice.** `POST /logout`,
deleting the user, `POST /recreate` and cookie expiry (default 7 days,
`COOKIE_MAX_AGE`) invalidate the server-side session. The session table lives in
gateway memory, so **restarting the gateway logs every browser out** — expected
for this single-replica demo; a production deployment needs a shared session
store and sticky routing or per-request re-authentication. With SSO enabled, the
gateway mints a session for the identity oauth2-proxy asserts instead of asking
for a token.
The same flow with `curl`, no JavaScript (the session cookie is the only
credential):
```bash
JAR=$(mktemp)
# signup (form POST, exactly what the landing page sends)
# signup (form POST, exactly what the landing page sends); add -d 'invite=...'
# when SIGNUP_INVITE_CODE is configured, or send X-Signup-Invite
curl -s -i -c "$JAR" -H 'Origin: http://localhost:18080' \
-d 'user=bob' http://localhost:18080/users
# proxy: bob's DSH Web through the gateway
@@ -247,53 +370,60 @@ curl -s -b "$JAR" http://localhost:18080/u/bob/ | head -5
curl -s -o /dev/null -w '%{http_code}\n' http://localhost:18080/u/bob/ # 401
# derived state: Ready / Waking / Suspended / ...
curl -s -b "$JAR" http://localhost:18080/users/bob
# delete bob (cascades sandbox + PVC)
# sign out (revokes the server-side session)
curl -s -b "$JAR" -H 'Origin: http://localhost:18080' -X POST http://localhost:18080/logout
# delete bob (cascades sandbox + PVC and revokes every session for bob)
curl -s -b "$JAR" -H 'Origin: http://localhost:18080' -X DELETE http://localhost:18080/users/bob
```
API clients that prefer bearer tokens can use `POST /users` with a JSON body
(`{"user":"bob"}`) and the returned one-time token, `POST /login` to exchange a token for a
cookie, and `Authorization: Bearer <token>` on subsequent calls.
(`{"user":"bob"}`) and the returned one-time token, `POST /login` to exchange a
token for a cookie, and `Authorization: Bearer <token>` on subsequent calls. A
cookie/session value is never accepted as a bearer token, and a token for one
user cannot read another user's path.
### Gateway API
### Auth abuse controls and proxy limits
| Route | Purpose |
| --- | --- |
| `GET /healthz` | Liveness/readiness; no cluster access |
| `GET /` | Landing page: signup, token login, logout |
| `POST /users` | Signup. JSON or form; `201` (JSON), `202` (waking), `303` (browser redirect), `409` if user exists; sets the session cookie |
| `POST /login` | Exchange a bearer token for the session cookie |
| `POST /logout` | Clear the session cookie |
| `GET /users/{user}` | Derived state (`Ready`/`Waking`/`Suspended`) |
| `DELETE /users/{user}` | Delete claim (cascades sandbox + PVC) |
| `GET /u/{user}` | `308` redirect to `/u/{user}/` |
| `GET /u/{user}/**` | HTTP/SSE proxy to the user's DSH Web |
| `WS /u/{user}/**` | WebSocket proxy to the user's DSH Web |
The gateway is a single process, so these limits are per replica. They stop
casual abuse and one tenant monopolizing the service; an internet-facing
deployment still needs ingress-level rate limiting and an audit trail.
| Setting | Default | Behavior |
| --- | --- | --- |
| `AUTH_RATE_LIMIT` / `AUTH_RATE_WINDOW` | `30` per `60` s | Throttles `/users` and `/login` per client (rightmost `X-Forwarded-For` when behind a proxy, else the TCP peer) and answers `429` + `Retry-After` before reading the body or touching the cluster. `0` disables it. The base manifest sets it explicitly. |
| `SIGNUP_INVITE_CODE` | unset | When set, `POST /users` requires the code as form field `invite` or header `X-Signup-Invite` (constant-time compare, `403` before user lookup). Put it in a Secret and restart the gateway to rotate. |
| `MAX_IN_FLIGHT_PER_USER` | `32` | Caps concurrent proxied requests + WebSockets per user; excess HTTP gets `503` + `Retry-After`, excess WebSocket upgrades close with `1011`. |
| `WS_MAX_MESSAGE_BYTES` | `16 MiB` | Largest WebSocket message received from DSH; oversized frames close the connection. Browser-to-DSH frames are bounded by uvicorn's own limit. HTTP/SSE streaming is not capped by this. |
| `MAX_BODY_MB` | `32` | Largest proxied HTTP request body (`413`). |
The gateway also strips client-supplied `Forwarded`, `X-Forwarded-User` and
`X-Auth-Request-*` headers (and `x-forwarded-*` / `x-auth-request-*` prefixes)
before proxying, so a tenant cannot inject identity or spoof hop headers into
DSH. `UPSTREAM_HEADER_ALLOWLIST` can replace that denylist with an explicit
comma-separated allowlist. See the [gateway README](gateway/README.md) for the
full configuration.
### Gateway routes
The browser flow above uses the gateway's main routes. The full route table,
status codes, configuration and failure matrix live in the
[gateway README](gateway/README.md).
### How the DSH handshake works
DSH does not expose an API for its launch token; it prints a URL containing `?token=…` in the
pod logs. For each `(user, pod UID, Host)` the gateway:
1. resolves the claim → sandbox → pod (annotation `agents.x-k8s.io/pod-name`, or the sandbox
name as pod name),
2. reads the pod's logs (last `DSH_LOG_TAIL_LINES` lines) and parses the most recent
`dsh web: …?token=…` line,
3. requests `http://<serviceFQDN>:3080/?token=…` with the browser's `Host`,
4. captures DSH's `dsh-auth-*` `Set-Cookie` response and caches it,
5. injects that cookie (and strips the browser's `Cookie`/`Authorization` and any `?token=`)
on every proxied HTTP and WebSocket request,
6. refreshes the handshake once if DSH answers `401`, and drops the cached cookie when the
sandbox suspends or is deleted.
This is why the gateway's RBAC includes `pods` `get`/`list` and `pods/log` `get`. It is also
the most fragile part of the design; see [Limitations](#limitations).
DSH prints its launch token only in pod logs. The gateway resolves
claim → sandbox → pod, reads the pod-log tail, performs the `?token=` handshake
under the browser's `Host`, caches `dsh-auth-*` per `(user, pod UID, Host)`, and
refreshes once on a `401`. That is why its RBAC includes `pods` `get`/`list` and
`pods/log` `get`; it is also the most fragile part of the design (see
[Limitations](#limitations) and the
[architecture ADR](docs/architecture.md#adr-3-server-side-launch-token-handshake-instead-of-exposing-the-token)).
### DSH_TRUSTED_HOSTS
DSH rejects requests whose `Host` is not allow-listed (`localhost` and `127.0.0.1` are always
trusted). Because the gateway forwards the browser's `Host`, set the public hostname in the
template:
DSH rejects requests whose `Host` is not allow-listed (`localhost` and
`127.0.0.1` are always trusted). Because the gateway forwards the browser's
`Host`, set the public hostname in the template:
```yaml
env:
@@ -301,36 +431,159 @@ env:
value: "dsh.example.com" # comma or space separated; empty for port-forward only
```
The same applies to a local `docker run`: the printed token URL uses `DSH_PUBLIC_URL` if set,
otherwise `http://localhost:${DSH_PORT}`.
The same applies to a local `docker run`: the printed token URL uses
`DSH_PUBLIC_URL` if set, otherwise `http://localhost:<DSH_PORT>`.
### WebSocket
The gateway is a Starlette ASGI app and proxies WebSocket upgrades with a bidirectional pump, so
the DSH web GUI is fully interactive. Requirements:
The gateway proxies WebSocket upgrades with a bidirectional pump, so the DSH web
GUI is fully interactive. The entry point must pass `Upgrade`/`Connection`
headers; [`manifests/exposure/ingress/60-ingress.yaml`](manifests/exposure/ingress/60-ingress.yaml)
is a starting point (Traefik handles upgrades by default). Idle suspension is
disabled while a proxied request or socket is in flight, so a long-running turn is
not cut off.
- the ingress/load balancer in front of the gateway must pass `Upgrade`/`Connection` headers;
`60-ingress.yaml` is a starting point (Traefik handles upgrades by default),
- idle suspension is disabled for users with in-flight proxied requests or open WebSockets, so
a long-running chat stream is not cut off mid-turn,
- the upstream upgrade dials the sandbox **pod IP**, forwards no `Host`/`Origin`, and performs
the launch-token handshake under that same `pod-IP:3080` authority. DSH fences on `Host`
(only its bind address, loopback, or an allow-listed host passes) and binds its auth cookie to
the request authority (`payload.authority === authority`), while `websockets` derives `Host`
from the URI and cannot be overridden. A browser-host cookie or forwarded origin makes the
upgrade fail (`401`/`403`, surfaced as close `1011`), which the client retries forever as
`[connection] connection lost, retry #N`,
- missing/invalid gateway auth is rejected right after the upgrade with close code `4401`
(WebSocket auth has no HTTP `401` path); cross-origin or Origin-less upgrades with the cookie
are rejected the same way (CSWSH protection), so clients must send a matching `Origin`;
upstream bridging failures close with `1011`.
The upstream upgrade dials the sandbox **pod IP**, forwards no `Host`/`Origin`,
and performs the launch-token handshake under the same `pod-IP:3080` authority.
Missing or cross-origin auth closes with `4401`; upstream bridging failures close
with `1011`, as does exceeding `MAX_IN_FLIGHT_PER_USER` (per-user socket cap).
Messages received from DSH larger than `WS_MAX_MESSAGE_BYTES` (16 MiB by
default) close the connection instead of being buffered; browser-to-DSH frames
are bounded by uvicorn's own limit. See the
[gateway README](gateway/README.md#proxy-failure-semantics) for the full matrix.
### Path routing
Each user's DSH Web is served under `/u/<user>/`. DSH injects `<base href="./">` and uses
relative asset/API URLs, so the UI works under a path prefix without a per-user hostname or
wildcard DNS. The gateway also rewrites absolute-path `Location` headers into `/u/<user>/…` and
redirects `/u/<user>` to the trailing-slash form with `308` so relative URLs resolve correctly.
Each user's DSH Web is served under `/u/<user>/`. DSH injects `<base href="./">`
and uses relative asset/API URLs, so the UI works under a path prefix without a
per-user hostname or wildcard DNS. The gateway also rewrites absolute-path
`Location` headers into `/u/<user>/…` and redirects `/u/<user>` to the
trailing-slash form with `308` so relative URLs resolve correctly.
## 9. Expose it: choose exactly one variant
The base manifests expose nothing outside the cluster (`dsh-gateway` is a
ClusterIP Service on port `8080`). Pick **one** variant — they are alternatives,
not a stack. Details, commands and the switch procedure are in the
[exposure README](manifests/exposure/README.md).
> Apply the base resources first, then the variant **by path**. Do not use
> `kubectl apply -R`: it would apply both variants and create two paths to the
> same gateway, which is unsafe once SSO is enabled.
Both variants also ship NetworkPolicies: `ingress/61-networkpolicy.yaml` limits
gateway ingress to Traefik pods, and `gateway-api-sso/75-networkpolicies.yaml`
limits the gateway to oauth2-proxy and oauth2-proxy egress to DNS, public HTTPS
and the gateway. They are part of the variant **and required for production**:
an exposure applied without its policy is demo-only. The sandbox side needs a
separate policy, and the gateway needs an egress policy; both are covered in
[optional hardening](docs/optional-hardening.md) (see the production checklist)
and summarized in the [exposure README](manifests/exposure/README.md).
### Option A — plain Ingress (gateway's own session auth)
1. Set the Ingress host/TLS (or drop the `tls:` block), and set
`DSH_TRUSTED_HOSTS` in `manifests/10-sandboxtemplate.yaml` to the same public
host; re-apply the template.
2. Set `COOKIE_SECURE=true` and drop the plain-HTTP acknowledgement once the
Ingress serves HTTPS:
`kubectl -n dsh-demo set env deploy/dsh-gateway COOKIE_SECURE=true ALLOW_INSECURE_COOKIES-`.
3. `kubectl apply -f manifests/exposure/ingress/`; verify
`curl -sI https://<host>/healthz` returns `200`.
Keep the router free of buffering/compression middlewares: they break WebSocket
upgrades and streaming. The variant's `61-networkpolicy.yaml` allows only Traefik
pods to reach the gateway; see the [exposure README](manifests/exposure/README.md)
for selector and probe caveats.
### Option B — Gateway API + GitHub SSO (oauth2-proxy)
`manifests/exposure/gateway-api-sso/` puts a TLS `Gateway` + `HTTPRoute` in
front of **oauth2-proxy**, which authenticates the browser against GitHub
(restricted to a configured organization) and forwards to the gateway with
`X-Forwarded-User` set. The gateway trusts that header only when
`AUTH_PROXY_HEADER` is set (SSO is **opt-in**).
1. Install Gateway API CRDs and a Gateway implementation; the manifests assume
`gatewayClassName: traefik` (the CRD version is `GATEWAY_API_VERSION` in
`versions.env`).
2. Create a GitHub OAuth App whose homepage/callback match the public FQDN, and
create the oauth2-proxy and TLS Secrets from
`73-oauth2-proxy-secret.yaml.example` and `74-tls-secret.yaml.example`.
3. Replace `dsh-demo.example.com` and `your-github-org` in the variant
manifests, set `DSH_TRUSTED_HOSTS` in the template, and enable SSO:
`kubectl -n dsh-demo set env deploy/dsh-gateway AUTH_PROXY_HEADER=X-Forwarded-User COOKIE_SECURE=true ALLOW_INSECURE_COOKIES-`.
4. `kubectl apply -f manifests/exposure/gateway-api-sso/`; then run
[`scripts/preflight-sso.sh`](scripts/preflight-sso.sh) and fix everything it
reports **before** pointing real traffic at the Gateway. It verifies the
Secrets (including a valid oauth2-proxy cookie secret length), the gateway's
`AUTH_PROXY_HEADER`/`COOKIE_SECURE`, that the Gateway's TLS certificate
reference resolves, and that no `your-github-org`/`dsh-demo.example.com`
placeholder was left applied. Full verification steps are in the
[exposure README](manifests/exposure/README.md).
> **Exposure without the variant NetworkPolicy is demo-only.** The SSO
> variant's `75-networkpolicies.yaml` is not optional in practice: it is the
> only thing that makes oauth2-proxy the sole path to the gateway. Applying
> only the Gateway/HTTPRoute/oauth2-proxy objects leaves the trust boundary
> unenforced, so anyone who can reach `dsh-gateway:8080` can send
> `X-Forwarded-User` and impersonate the user they claim to be.
>
> **SSO trust boundary — read this before enabling it.** After SSO is on, the
> `dsh-gateway` Service must be reachable **only** through oauth2-proxy (it is
> ClusterIP). The variant's `75-networkpolicies.yaml` restricts pod traffic
> accordingly: the gateway accepts traffic only from oauth2-proxy pods,
> oauth2-proxy only from the Gateway implementation, and its egress only to DNS,
> public HTTPS and the gateway. The gateway's own egress (Kubernetes API,
> sandbox pods) is not restricted by that file because the API-server endpoint is
> cluster-specific; add the
> [gateway egress policy](docs/optional-hardening.md#gateway-egress-required-for-production)
> for production. Anyone who can reach `dsh-gateway:8080` directly can send
> `X-Forwarded-User: <victim>`, so never add a second path. To turn SSO off,
> remove the variable
> (`kubectl -n dsh-demo set env deploy/dsh-gateway AUTH_PROXY_HEADER-`) and
> delete the SSO variant.
### Switching variants
Delete the variant you had before applying the other one, revert
`AUTH_PROXY_HEADER`/`COOKIE_SECURE`, and verify with
`kubectl -n dsh-demo get ingress,gateway,httproute` that exactly one path to the
gateway remains. Full commands and verification are in the
[exposure README](manifests/exposure/README.md).
## Optional hardening
The base walkthrough is deliberately portable, which means it is **not
hardened**: without the policies and gVisor it is demo-only, not a safe
multi-tenant platform. [Optional hardening](docs/optional-hardening.md) has the
settings and a production checklist. In short:
- **gVisor** — uncomment `runtimeClassName: gvisor` before applying, or patch an
existing install. Requires a `gvisor` RuntimeClass and capable nodes.
`kubectl port-forward` does not work under gVisor; the gateway path does.
Required for untrusted agents in production.
- **nodeSelector** — uncomment and replace the placeholder node label to pin
sandboxes and the gateway to a dedicated pool; add a toleration for tainted
pools.
- **Sandbox NetworkPolicy** —
`kubectl apply -f manifests/optional/network-policy/` allows only
gateway→sandbox on `3080` and sandbox→DNS/public `443`. Required for
production; exposure without it is demo-only.
- **Gateway egress policy** — the gateway needs DNS, the Kubernetes API and
`3080` to sandbox pods. It is not shipped because the API-server endpoint is
cluster-specific; use the
[recipe](docs/optional-hardening.md#gateway-egress-required-for-production).
Required for production.
- **Provider keys** — create `dsh-provider-keys-extra` as shown in step 1.
- **Failed-Pod recovery** — uncomment `failedPodPolicy: Recreate` so the
controller replaces a `Failed` sandbox Pod (node eviction) instead of parking
the sandbox at `Finished=True`. Requires agent-sandbox v1.0.8+.
The **exposure variants' NetworkPolicies** are not optional in production
either; see step 9. Apply each setting independently and re-run the walkthrough.
`kubectl apply -f manifests/` is non-recursive, so the optional manifests are not
applied by the base step.
## Troubleshooting
@@ -339,43 +592,124 @@ redirects `/u/<user>` to the trailing-slash form with `308` so relative URLs res
| Pod `CrashLoopBackOff`, probes failing with `401` | Probe path is `/`; change readiness/liveness to `/favicon.svg` |
| Browser shows `401` from DSH through the gateway | Gateway handshake failed — check `kubectl -n dsh-demo logs deploy/dsh-gateway` and that RBAC grants `pods/log` |
| `502` / `upstream unavailable` | Sandbox pod is not Ready (wake timed out) or the Service DNS is stale; check `kubectl get sandbox`, `WAKE_TIMEOUT` |
| `503` with `Retry-After` | The launch token was not found in the pod logs yet; the gateway retries after a short delay. Check the pod logs and `DSH_LOG_TAIL_LINES` |
| `503` with `Retry-After` | Either the launch token was not found in the pod logs yet (the gateway retries after a short delay; check the pod logs and `DSH_LOG_TAIL_LINES`) or the user hit `MAX_IN_FLIGHT_PER_USER` (default 32 concurrent requests/WebSockets); reduce client concurrency or raise the limit deliberately |
| `429` with `Retry-After` on signup/login | The per-client auth rate limit (`AUTH_RATE_LIMIT`, default explicitly `30` per `AUTH_RATE_WINDOW=60` s) rejected the attempt before reading the body. Wait for the window, or raise the limit; behind a proxy make sure the real client IP is the rightmost `X-Forwarded-For` entry, not a shared NAT address |
| `403` on signup: invite required/invalid | `SIGNUP_INVITE_CODE` is configured. Send the code as form field `invite` or header `X-Signup-Invite`; the landing page shows the field when enabled. A wrong code is rejected before the user lookup |
| Gateway pod `CrashLoopBackOff` with `refusing to start with an insecure session cookie` | Intentional guard: plain HTTP is only allowed when explicitly acknowledged. For the local demo set `ALLOW_INSECURE_COOKIES=1` (the base manifest does); for HTTPS set `COOKIE_SECURE=true` and remove `ALLOW_INSECURE_COOKIES` |
| Every browser is logged out after a gateway rollout/restart | Expected: sessions are opaque ids in gateway memory and a restart drops them. Sign in again; a production deployment needs shared session state or per-request re-authentication. The DSH sandboxes and their data are unaffected |
| WebSocket closes with `1011` for a busy client | `MAX_IN_FLIGHT_PER_USER` (per-user connection cap) or an upstream/handshake failure closed the socket; check the gateway logs to tell them apart |
| WebSocket closes after a very large message from DSH | A message received from DSH exceeded `WS_MAX_MESSAGE_BYTES` (16 MiB), so the relay closed it instead of buffering; raise the limit only if DSH legitimately sends larger frames. Browser-to-DSH frames are bounded by uvicorn's own limit |
| `scripts/preflight-sso.sh` exits non-zero | It names the failing item: a missing Secret/key, a bad oauth2-proxy cookie-secret length, missing `AUTH_PROXY_HEADER`/`COOKIE_SECURE`, a dangling Gateway TLS reference, a leftover `your-github-org`/`dsh-demo.example.com` placeholder, or a second path to the gateway (an Ingress). Fix all items before exposing the SSO variant |
| `409` on signup | User already has a claim; use `POST /login` with the existing token or delete the user first |
| Cookie set but browser keeps getting `401` | `COOKIE_SECURE=true` while serving over plain HTTP (secure cookies are not sent); use HTTPS or set `false` for local demos |
| Cookie set but browser keeps getting `401` | `COOKIE_SECURE=true` while serving over plain HTTP (secure cookies are not sent); use HTTPS or set `false` **and** `ALLOW_INSECURE_COOKIES=1` for local demos |
| WebSocket closes immediately (`4401`) | Missing/expired gateway cookie, or the upgrade's `Origin` does not match `Host` (CSWSH check); DSH rejecting `Host` instead surfaces as close `1011` — add the host to `DSH_TRUSTED_HOSTS` |
| Console spams `[connection] connection lost, retry #N` | The upstream WS upgrade is being rejected, surfaced as close `1011`. The gateway dials the sandbox **pod IP**, forwards no `Host`/`Origin`, and handshakes under the same `pod-IP:3080` authority DSH binds its cookie to; an older gateway image, or a sandbox with an empty `status.podIPs`, hits this loop |
| `manifest.webmanifest` returns `401` in the console | Browsers fetch the Web App Manifest without credentials, so the gateway's session cookie is absent and the request fails closed. The UI is unaffected; only the PWA install prompt is lost |
| Signup is slow / `202` then `Waking` | Warm pool exhausted (`kubectl get sandboxes -l agents.x-k8s.io/warm-pool-sandbox`); scale the pool |
| `403` from the Kubernetes API in gateway logs | The gateway ServiceAccount is missing claim/sandbox/pod-log verbs (see `50-gateway.yaml`) |
| Step 3: `pod "$SB" not found` after deploying the gateway | The claim was suspended by the idle sweeper. Claims created by the gateway carry `aaas.example.com/managed-by=dsh-gateway`; hand-applied claims (steps 1–7) do not and are never swept. Upgrade the gateway to a build that honors the marker, or resume the sandbox: `kubectl -n dsh-demo patch sandbox "$SB" --type merge -p '{"spec":{"operatingMode":"Running"}}'` |
| Resume crash-loops: `credentials-local: … is readable beyond its owner (mode 660)` | The pod template lacks `fsGroupChangePolicy: OnRootMismatch`, so the kubelet's default fsGroup policy rewrote `.credentials.yaml` to `0660`; DSH requires `0600` and refuses to start. New installs get the policy from the template. For an existing sandbox: suspend it, run a throwaway pod on `dsh-home-$SB` to `chmod 600 /home/node/.dsh/.credentials.yaml`, patch the sandbox's `podTemplate` with the policy, then resume. |
| `403` from the Kubernetes API in gateway logs | The gateway ServiceAccount is missing claim/sandbox/pod-log verbs (see `manifests/50-gateway.yaml`) |
| Gateway pod `ErrImagePull` / `ImagePullBackOff` (especially on kind) | The manifest uses `imagePullPolicy: IfNotPresent` with the release identity. Build with `make -C ../.. gateway`, retag to exactly the manifest's image, `kind load docker-image <that-ref>`, or set the Deployment image to your local tag / pinned digest. On a real cluster, make sure the GHCR package is readable (public or a pull secret) |
| Step 3: `pod "$SB" not found` after deploying the gateway | The claim was suspended by the idle sweeper. Claims created by the gateway carry `aaas.example.com/managed-by=dsh-gateway`; hand-applied claims (steps 1–7) do not and are never swept. Resume the sandbox: `kubectl -n dsh-demo patch sandbox "$SB" --type merge -p '{"spec":{"operatingMode":"Running"}}'` |
| Resume crash-loops: `credentials-local: … is readable beyond its owner (mode 660)` | The pod template lacks `fsGroupChangePolicy: OnRootMismatch`, so the kubelet's default fsGroup policy rewrote `.credentials.yaml` to `0660`; DSH requires `0600` and refuses to start. New installs get the policy from the template. For an existing sandbox: suspend it, run a throwaway pod on `dsh-home-$SB` to `chmod 600 /home/node/.dsh/.credentials.yaml`, patch the sandbox's `podTemplate` with the policy, then resume |
| Agents get `Permission denied` writing to `/workspace` on a fresh or resumed pod | The `/workspace` mount is the `dsh-home` PVC's `workspace/` subPath, and the kubelet is known to leave a `subPath` directory at `0755` if it is interrupted between creating and chmodding it ([kubernetes#142464](https://github.com/kubernetes/kubernetes/issues/142464)); later mounts never correct it. Suspend the sandbox, run a throwaway pod on `dsh-home-$SB` mounted at `/data`, `chmod 2775 /data/workspace` (do **not** chown/chmod recursively — that rewrites `.credentials.yaml`), then resume. If you recreate the sandbox instead, its current `/workspace` contents go with the PVC |
| Every bash command fails: `SANDBOX_UNAVAILABLE`, `sandbox mode "workspace-write" is requested but no sandbox backend is usable` | DSH's own command sandbox found neither backend: `bubblewrap` ships in the image but is unsetuid, so it needs an unprivileged user namespace this hardened pod does not grant, and no Landlock runner exists on this kernel/runtime — the confined executor fails closed. The template sets `DSH_PERMISSION_MODE=danger-full-access` to keep agents usable inside the pod boundary. To restore confinement, confirm a backend inside the pod (`bwrap --ro-bind / / true`; `grep -o landlock /sys/kernel/security/lsm`), set `DSH_PERMISSION_MODE=workspace-write` (or `read-only`), and recreate the pod — the value is stamped at creation. A session that explicitly chose a mode keeps that choice |
| SSO: GitHub login succeeds but the gateway asks for a token / returns `401` to `/recreate` | `AUTH_PROXY_HEADER` is not set on `deploy/dsh-gateway` (SSO is opt-in). Set `AUTH_PROXY_HEADER=X-Forwarded-User` and confirm the request actually reaches the gateway through oauth2-proxy |
| SSO: gateway treats a caller as another user / bad `X-Forwarded-User` accepted | Trust boundary violation: something other than oauth2-proxy can reach `dsh-gateway:8080`. Remove the second path (Ingress/Gateway/port-forward), keep the Service ClusterIP, and only re-enable SSO once the proxy is the sole entry point. Never put a trusted identity header in front of a service that is directly reachable |
| SSO: oauth2-proxy `cookie_secret must be 16, 24, or 32 bytes ... but is 44 bytes` | The cookie secret was generated as standard base64. Use URL-safe unpadded base64 (see `73-oauth2-proxy-secret.yaml.example`) |
| SSO: redirect loop (`ERR_TOO_MANY_REDIRECTS`) | oauth2-proxy `--redirect-url` / callback does not match the public FQDN, or TLS is not terminating at the Gateway listener; fix the OAuth App URLs and the manifest placeholders |
| SSO: DSH rejects the public host behind the proxy | Add the public hostname to `DSH_TRUSTED_HOSTS` in `manifests/10-sandboxtemplate.yaml`; the gateway forwards the browser's `Host` |
| Pods `Pending` after setting `nodeSelector` | No node matches the label, the pool is full, or a tainted pool needs a toleration; inspect `kubectl describe pod` and the node labels |
| gVisor: `kubectl port-forward` fails or the connection is refused | Port-forward is incompatible with gVisor; reach DSH through the gateway (or a reachable Service/Sandbox Router) |
| Sandbox cannot reach a model API after applying the optional NetworkPolicy | The policy allows DNS and public TCP `443` only; a proxy, custom base URL, port `80` or non-443 endpoint needs an explicit egress rule |
| Gateway cannot reach a sandbox after applying an exposure NetworkPolicy | Check that the CNI enforces NetworkPolicy and that the Traefik/oauth2-proxy pod labels match; kubelet probes may also need the node CIDR |
| New provider key not visible in a sandbox | Pod env is stamped at creation: delete the warm-pool spares and recreate claimed pods. A key saved in DSH's credential store also wins over env |
| Folder picker opens on `/home/node` instead of the workspace | The sandbox was created before the template set `HOME=/workspace`, and pod env is stamped at creation. Verify with `kubectl -n dsh-demo exec "$SB" -- printenv HOME`; patch the sandbox's `podTemplate` with the env (or recreate the sandbox — the PVC keeps DSH state and workspace) and resume |
| File explorer and `ls /workspace` show `.cache`, `.config`, `.local`, `.npm`, `.agents` | Expected with `HOME=/workspace`: `~`-relative tool caches and DSH's global agent instructions/skills live on the PVC inside the workspace. Remove the `HOME` env from the template (and recreate the sandbox) if you would rather keep the picker on the ephemeral image home; point caches elsewhere per tool if only they are unwanted |
| Sandbox reports `Finished=True` and every gateway request fails | The sandbox Pod reached phase `Failed` (node eviction, spot reclaim) and the default `Retain` keeps it. Delete the Pod so the controller recreates it, or set `failedPodPolicy: Recreate` (agent-sandbox v1.0.8+; see optional hardening) |
## Limitations
Deliberately **not** production code:
- **Single replica.** Idle clocks, in-flight tracking, and the handshake cache are in memory.
Running two gateway replicas would double-suspend or lose wake tracking; scale up only after
moving that state to a shared store.
- **Log scraping.** The launch token is recovered from pod logs. Log rotation, a restarted pod,
or a changed DSH log format can break the handshake until the next `401` refresh.
- **Shared provider key.** `DEEPSEEK_API_KEY` is shared by all tenants via the template. Real
multi-tenancy needs per-tenant credentials or an egress proxy.
- **No TLS in the gateway.** Terminate TLS at an Ingress/reverse proxy and set
`COOKIE_SECURE=true`.
- **In-memory user directory.** Users are claims + annotations; there is no database, password
reset, or admin API. Tokens are stored only as SHA-256 hashes in claim annotations.
- **Bearer tokens in URLs.** Avoid putting the signup token in a URL/log; the browser flow uses
the `HttpOnly` cookie instead.
## Cost model
The same economics as the upstream pattern: suspend-when-idle plus warm pools turn an
always-on instance into a sub-second cold-start pause per return-after-absence, at a fraction
of the cost.
- **Single replica.** Idle clocks, in-flight tracking, the handshake cache and
the browser session table are in memory. Running two gateway replicas would
double-suspend or lose wake tracking, and sessions would land on the wrong
replica; scale up only after moving that state to a shared store.
- **Sessions die with the process.** Browser sessions are opaque ids held in
gateway memory (revocable on logout/delete/recreate, but not durable). A
gateway restart logs every browser out. There is no SSO refresh-token or
password store; identity is re-established by oauth2-proxy on the next
request.
- **Abuse controls are per-process and coarse.** `AUTH_RATE_LIMIT`, the signup
invite code, `MAX_IN_FLIGHT_PER_USER` and `WS_MAX_MESSAGE_BYTES` limit casual
abuse, but they are not a substitute for ingress rate limiting, WAF/quota, or
an audit log, and the in-memory rate buckets reset on restart.
- **Log scraping.** The launch token is recovered from pod logs. Log rotation, a
restarted pod, or a changed DSH log format can break the handshake until the
next `401` refresh.
- **Shared provider keys.** `DEEPSEEK_API_KEY` and any keys in the optional
`dsh-provider-keys-extra` Secret are shared by all tenants via the template.
Tenants can override a key through DSH's stored credential file on their PVC,
but the platform does not manage that. Real multi-tenancy needs per-tenant
credentials or an egress proxy.
- **No TLS in the gateway.** Terminate TLS at an Ingress or Gateway and set
`COOKIE_SECURE=true`, then remove `ALLOW_INSECURE_COOKIES`. The base manifest
runs plain HTTP for the port-forward demo and is demo-only.
- **SSO trust is hop-based.** `AUTH_PROXY_HEADER` trusts whoever sets the header.
The SSO variant ships `75-networkpolicies.yaml`, which restricts the gateway
to oauth2-proxy and oauth2-proxy egress to the gateway/DNS/public HTTPS; a
custom exposure without that policy has no signature, mTLS or network
enforcement. Exposure without the policy is demo-only.
- **DSH's inner file sandbox is off here.** The template sets
`DSH_PERMISSION_MODE=danger-full-access` because neither backend of DSH's own
command sandbox is usable in this pod: `bubblewrap` is installed in the image
but is unsetuid, so it needs an unprivileged user namespace that the pod's
seccomp/capability surface does not grant (`capabilities.drop: ["ALL"]`), and
the kernel/runtime exposes no Landlock runner. The executor therefore fails
closed with `SANDBOX_UNAVAILABLE` and every bash command fails. The per-user
pod remains the boundary — non-root,
capabilities dropped, `seccompProfile: RuntimeDefault`, its own PVC and Service
— but an agent command can now read its own `DSH_HOME` (including
`.credentials.yaml`) and write anywhere in the container. Run sandboxes under
gVisor in production. On hosts where a backend works, set `workspace-write`
(or `read-only`) and verify before relying on the inner policy.
- **Isolation is opt-in.** gVisor, the sandbox NetworkPolicy, the gateway egress
policy and the exposure policies are opt-in; without them, sandboxes share the
host kernel and can reach the cluster network. The base example is a demo, not
a hardened multi-tenant platform. See the
[production checklist](docs/optional-hardening.md#6-production-checklist).
- **OAuth org is the only authorization.** Any GitHub user in the configured
organization gets an auto-provisioned sandbox; there is no per-user allow-list,
quota, or admin API. `SIGNUP_INVITE_CODE` gates the gateway's own signup form,
not SSO auto-provisioning.
- **In-memory user directory.** Users are claims + annotations; there is no
database, password reset, or admin API. Tokens are stored only as SHA-256
hashes in claim annotations.
- **Bearer tokens in URLs.** Avoid putting the signup token in a URL/log; the
browser flow uses the `HttpOnly` cookie instead. The cookie no longer contains
the token, and a cookie value is never accepted as a bearer token.
## Cleanup
```bash
# Delete an exposure variant first if you applied one (optional; namespace
# deletion below removes everything):
kubectl delete -f manifests/exposure/ingress/ # or gateway-api-sso/
# Then the whole example:
kubectl delete namespace dsh-demo
```
If you created the oauth2-proxy/TLS Secrets or the GitHub OAuth App for step 9,
delete the Secrets with the namespace and revoke the OAuth App in GitHub.
## More
- [Gateway README](gateway/README.md) — API, configuration, auth/SSO, failure
semantics, limitations.
- [Architecture](docs/architecture.md) — container/sequence diagrams, ADRs,
security and RBAC, failure modes, evolution path.
- [Exposure README](manifests/exposure/README.md) — the two variants in detail.
- [SSO preflight](scripts/preflight-sso.sh) — read-only checks for the SSO
variant (Secrets, gateway env, TLS reference, placeholders, second paths).
- [Optional hardening](docs/optional-hardening.md) — gVisor, nodeSelector,
NetworkPolicy, extra provider keys and failed-Pod recovery, with verification,
the production checklist and rollback.

View File

@@ -0,0 +1,454 @@
# Architecture
This document describes the browser-usable **agents-as-a-service** platform in this example
([README](../README.md), [manifests](../manifests/), [gateway](../gateway/)): how the gateway,
Kubernetes, and DSH Web fit together, and why the design looks the way it does.
Scope: one Kubernetes namespace (`dsh-demo`), one shared DSH Web image, one gateway deployment,
and one DSH Web pod + PVC per user. The cluster is assumed to run
[agent-sandbox](https://github.com/kubernetes-sigs/agent-sandbox) with the pinned
`AGENT_SANDBOX_VERSION` from [`versions.env`](../../../versions.env) and the extensions
(`SandboxTemplate`, `SandboxWarmPool`, `SandboxClaim`).
## Container view
```mermaid
flowchart LR
browser["Browser"]
subgraph cluster["Kubernetes cluster - namespace dsh-demo"]
gw["Gateway Deployment<br/>Starlette ASGI on uvicorn :8080<br/>landing page + auth + proxy + sweeper"]
api["Kubernetes API Server"]
ctrl["agent-sandbox controller"]
pool["SandboxWarmPool dsh-pool<br/>2 pre-warmed spares"]
claim["SandboxClaim dsh-USER<br/>one per user"]
sb["Sandbox dsh-pool-xxxx<br/>spec.operatingMode"]
svc["Headless Service<br/>status.serviceFQDN"]
pod["DSH Web pod<br/>node process :3080"]
pvc[("PVC dsh-home-SANDBOX<br/>/home/node/.dsh + /workspace (subPath)")]
end
browser -->|"HTTP + WebSocket<br/>/u/USER/**"| gw
gw -->|"claims, sandboxes, pods, pod logs"| api
api --> ctrl
ctrl -->|"adopt / suspend / resume"| claim
claim --> sb
pool -.->|"replenishes"| sb
sb --> svc
svc --> pod
pod --> pvc
```
Responsibilities:
| Component | Responsibility |
| --- | --- |
| **Browser** | Holds only the gateway session cookie (`dsh_gw_session`). Talks exclusively to the gateway. |
| **Gateway** | Signup/login, session auth, `SandboxClaim` lifecycle, wake-on-connect, idle sweeper, HTTP/SSE/WebSocket reverse proxy, DSH launch-token handshake. |
| **Kubernetes API** | Source of truth for claims, sandboxes, and pods; also the log source for the DSH launch token. |
| **agent-sandbox controller** | Adopts warm spares on claim, flips `operatingMode` (Running/Suspended), garbage-collects sandbox + PVC on claim delete. |
| **DSH Web pod** | One `dsh web` process per user, DSH state and the agent workspace on the per-user PVC (`/home/node/.dsh` + `/workspace`), listening on `3080`. |
| **Warm pool** | Keeps `N` Ready sandboxes so signup is a label flip instead of a cold start. |
## Exposure variants
The base manifests deploy the gateway as a ClusterIP Service (`dsh-gateway:8080`) and expose
nothing outside the cluster. Two **mutually exclusive** variants live under
[`manifests/exposure/`](../manifests/exposure/):
| Variant | Entry point | Authentication | Notes |
| --- | --- | --- | --- |
| [Ingress](../manifests/exposure/ingress/60-ingress.yaml) | Traefik `Ingress` + optional TLS | the gateway's own signup/session cookie | simplest; no external IdP; set `COOKIE_SECURE=true` with HTTPS |
| [Gateway API + SSO](../manifests/exposure/gateway-api-sso/70-gateway.yaml) | Gateway API `Gateway`/`HTTPRoute` terminating TLS | GitHub OAuth via [oauth2-proxy](../manifests/exposure/gateway-api-sso/72-oauth2-proxy.yaml), identity forwarded as `X-Forwarded-User` | requires `AUTH_PROXY_HEADER` on the gateway; the trust boundary applies |
They are alternatives, not a stack: applying both creates a second path to the same gateway
and defeats SSO. When SSO is enabled, the Service must stay ClusterIP and be reachable only
through oauth2-proxy — otherwise any client can send `X-Forwarded-User` itself and be treated
as that user. The gateway ignores the header unless `AUTH_PROXY_HEADER` is set. See the
[exposure README](../manifests/exposure/README.md) and the
[gateway README](../gateway/README.md#sso-opt-in).
Each variant also ships a NetworkPolicy: `ingress/61-networkpolicy.yaml` limits gateway ingress
to Traefik pods, and `gateway-api-sso/75-networkpolicies.yaml` limits the gateway to
oauth2-proxy, oauth2-proxy ingress to the Gateway implementation, and oauth2-proxy egress to
DNS/public HTTPS/the gateway. The SSO policy turns the trust boundary into pod-traffic
enforcement. These policies are **required for production**: exposing the gateway by applying
only the Gateway/HTTPRoute/Ingress objects is demo-only. The sandbox-side policy and the
gateway's own egress policy are separate (the API-server endpoint is cluster-specific); see
[optional hardening](optional-hardening.md#gateway-egress-required-for-production).
## Flows
### Signup (browser)
```mermaid
sequenceDiagram
autonumber
actor B as Browser
participant G as Gateway
participant K as Kubernetes API
participant C as agent-sandbox controller
B->>G: POST /users (form: user=bob, invite, Origin)
G->>G: rate limit, invite check, validate user name and body size
G->>K: create SandboxClaim dsh-bob (warmPoolRef)
K->>C: reconcile claim
C-->>K: adopt spare, Ready, status.sandbox.name = dsh-pool-xxxx
G->>K: read claim (status) + sandbox + pod
G->>G: mint opaque sid, store sid -> (user, expiry)
G-->>B: 303 Location /u/bob/ + Set-Cookie dsh_gw_session=<sid>
Note over G: JSON clients instead get 201 (Ready),<br/>202 (accepted, still waking), or 409 (conflict)
```
Form signup waits up to `WAKE_TIMEOUT` for the claim to become `Ready`, then redirects; the
first proxied request also waits for `Ready` and performs the DSH handshake (below). A `409`
means the user already has a claim, in which case `POST /login` (bearer token) or the existing
cookie is the right path. The cookie value is an opaque session id: it is meaningless outside
this gateway process and is never accepted as a bearer token.
### First request / proxy (with handshake)
```mermaid
sequenceDiagram
autonumber
actor B as Browser
participant G as Gateway
participant K as Kubernetes API
participant D as DSH Web pod
B->>G: GET /u/bob/ (Cookie dsh_gw_session=<sid>)
G->>G: look up sid -> session (valid, not expired), user = bob
G->>K: resolve claim -> sandbox -> pod
alt sandbox Suspended
G->>K: patch operatingMode=Running (wake)
loop until Ready or WAKE_TIMEOUT
G->>K: read sandbox Ready condition
end
end
G->>K: read pod logs (tail)
K-->>G: "dsh web: ...?token=..." line
G->>D: GET /?token=... (browser Host, X-Forwarded-*)
D-->>G: 303 + Set-Cookie dsh-auth-*
G->>G: cache cookie for (user, pod UID, Host)
G->>G: strip Cookie/Authorization/hop-by-hop headers
G->>D: replay request with dsh-auth cookie
D-->>G: 200 HTML / SSE stream
G-->>B: 200 (upstream Set-Cookie dropped, Location rewritten)
```
Handshake cache behavior: one entry per `(user, pod UID, Host)`; refreshed once when DSH answers
`401`; dropped when the sandbox is suspended or deleted. The launch token and the DSH auth
cookie never reach the browser.
Inside `proxy_http` the order is auth, in-flight slot, sandbox resolution/wake, DSH handshake,
body read, upstream send. An upload aborted mid-body may therefore already have paid for a wake
and a handshake, but it never reaches DSH.
### Browser sessions
The browser never holds a DSH credential. On signup/login the gateway mints a random session id
(`secrets.token_urlsafe(32)`), stores `sid -> Session(user, expires_at)` in memory, and sets it
as the `HttpOnly`, `SameSite=Lax` cookie `dsh_gw_session` (`__Host-dsh_gw_session` when
`COOKIE_SECURE=true`). Every browser request is resolved through that table; the cookie value is
not a bearer token and cannot be replayed against another user's path.
Sessions end on explicit logout, user delete, `/recreate` (all sessions for that user) and
expiry (`COOKIE_MAX_AGE`, default 7 days). Because the table is process memory, a gateway
restart logs every browser out; scaling beyond one replica therefore requires a shared session
store (or re-authentication on every request). The sweeper prunes expired entries and sessions
for users whose claim no longer exists.
### Auth abuse controls and proxy limits
Unauthenticated endpoints are throttled before the body is read: `AUTH_RATE_LIMIT` attempts per
client per `AUTH_RATE_WINDOW` (defaults 30/60s; keyed on the rightmost `X-Forwarded-For` entry
when present, else the peer address), answered with `429` + `Retry-After`. When
`SIGNUP_INVITE_CODE` is set, signup additionally requires a constant-time-matched code (form
field or `X-Signup-Invite`) and answers `403` otherwise. Proxied traffic is capped per user
(`MAX_IN_FLIGHT_PER_USER`, default 32 concurrent HTTP requests + WebSockets: `503` for HTTP,
close `1011` for WebSocket) and per WebSocket message received from DSH
(`WS_MAX_MESSAGE_BYTES`, 16 MiB). These are in-process guards for a single replica, not an ingress
rate limiter or quota system.
### Wake-on-connect and idle suspend
```mermaid
sequenceDiagram
autonumber
participant S as Sweeper - lifespan task
participant G as Gateway
participant K as Kubernetes API
actor B as Browser
S->>G: no proxied request for IDLE_TIMEOUT
G->>K: patch sandbox operatingMode=Suspended
K-->>G: pod deleted, PVC + Service retained
Note over S,G: in-flight requests/WebSockets are tracked,<br/>so a busy turn is never suspended
B->>G: GET /u/bob/ after a while
G->>K: patch operatingMode=Running
G->>K: wait for Ready (WAKE_TIMEOUT)
G-->>B: request held, then served
```
Suspend deletes only the pod. The PVC keeps conversations, sessions, config **and the agent
workspace**, so resume reattaches exactly the same state. Both trees live on one PVC: DSH state at
`/home/node/.dsh` (the volume root) and the workspace at `/workspace` (a second mount of the same
volume with `subPath: workspace`, which keeps `.credentials.yaml` out of the agent-visible tree).
The idle sweeper never touches the volume; deleting the claim still garbage-collects it, so
`POST /recreate` wipes state and workspace together.
**Workspace as home.** The template sets `HOME=/workspace`. DSH has no start-path setting for its
directory picker: the browse backend lists `$HOME` when the dialog opens, so with the image default
(`/home/node`, off the PVC) the picker would start on a throwaway directory. The override makes the
picker open on the persisted workspace and makes `~` resolve there; the cost is that tool caches
and config kept under `$HOME` (`npm`/`pnpm`, `uv`, `git`, `gh`, and DSH's global `~/.agents`
instructions and skills) become dotfiles in the workspace. `DSH_HOME` stays `/home/node/.dsh`, so
sessions, config and credentials remain outside the agent-visible tree. Pod env is stamped when a
sandbox is created, so an existing sandbox keeps the old `HOME` until it is recreated.
**Ownership boundary.** The gateway shares its namespace with claims created outside it — the
example's resource-layer walkthrough applies `dsh-alice` by hand, and every claim is named
`dsh-<user>`. Claims the gateway creates carry `aaas.example.com/managed-by=dsh-gateway`; the
sweeper skips everything else, so it can never suspend a sandbox an operator is using. Without
that marker the sweeper would suspend a hand-applied claim 60s after first sighting and break
`kubectl logs`/`port-forward` against its pod.
**Resume and file ownership.** The pod template sets `fsGroupChangePolicy: OnRootMismatch` on
purpose. DSH's credentials plugin refuses to boot when `.credentials.yaml` has any group/other
permission bit (it creates the file `0600`). The kubelet's default fsGroup policy (`Always`)
recursively ORs `0660` into every file on each pod start, so the first resume of a sandbox would
crash-loop with `mode 660`. `OnRootMismatch` only rewrites ownership/permissions when the volume
root does not already match the fsGroup, which it does after the first mount — so suspend/resume
keeps working. Sandboxes snapshot their pod template, so an updated `SandboxTemplate` does not
retroactively change existing ones.
### WebSocket
```mermaid
sequenceDiagram
autonumber
actor B as Browser
participant G as Gateway
participant D as DSH Web pod
B->>G: GET /u/bob/... Upgrade: websocket (Cookie dsh_gw_session)
G->>G: authenticate + mark user in-flight
alt cookie missing/invalid
G-->>B: 101 upgrade, then close 4401
else authenticated
G->>D: websockets.connect(path, Host, Cookie dsh-auth)
alt upstream handshake fails
G-->>B: 101 upgrade, then close 1011
else bridged
G-->>B: 101 Switching Protocols
loop frames
B->>D: frames proxied through G
D->>B: frames proxied through G
end
G->>G: clear in-flight on close
end
end
```
An upgrade with a missing/expired session, or one that would exceed
`MAX_IN_FLIGHT_PER_USER`, is accepted and then closed: `4401` for authentication, `1011` for
the per-user socket cap. Messages received from DSH above `WS_MAX_MESSAGE_BYTES` (16 MiB) close
the connection instead of being buffered; browser-to-DSH frames are bounded by uvicorn's own
limit. Client-supplied `Forwarded`, `X-Forwarded-User`,
`X-Auth-Request-*` and `x-forwarded-*`/`x-auth-request-*` headers are stripped on the HTTP and
WebSocket paths; `UPSTREAM_HEADER_ALLOWLIST` replaces that denylist with an explicit allowlist.
## Decisions and trade-offs
### ADR-1: Starlette ASGI instead of a sync Flask proxy
- **Decision:** implement the gateway as a Starlette app served by `uvicorn
gateway:create_app --factory`.
- **Why:** the browser needs WebSocket upgrades, streaming SSE responses, and concurrent
long-poll wake requests. A requests-based Flask proxy cannot upgrade connections and blocks
workers while a sandbox wakes.
- **Cost:** async code paths and explicit bridging for the sync `kubernetes` client (offloaded
to threads). Accepted: one small service, one framework.
### ADR-2: Path-based routing (`/u/{user}/`) instead of per-user subdomains
- **Decision:** serve each user under `https://gateway/u/{user}/`.
- **Why:** one hostname, one TLS certificate, no wildcard DNS or per-user Ingress objects. DSH
uses relative asset/API URLs and injects `<base href="./">`, so a path prefix works; the
gateway rewrites absolute-path `Location` headers and redirects `/u/{user}` to
`/u/{user}/`.
- **Cost:** any upstream code that emits root-absolute URLs would need rewriting; cookie
domains are shared (mitigated by stripping DSH `Set-Cookie`). If DSH ever changes its base
handling, per-user subdomains are the fallback.
### ADR-3: Server-side launch-token handshake instead of exposing the token
- **Decision:** the gateway reads DSH's launch token from pod logs, performs the token
handshake itself, caches DSH's auth cookie, and authenticates the browser with its own
`HttpOnly` cookie.
- **Why:** DSH has no token API and returns `401` on `/` without the launch-token cookie.
Handing the token to the browser would put it in URLs, history, and logs, and would let users
bypass gateway auth.
- **Cost:** log reading permission and a brittle parser; see [Upstream feature
request](#upstream-feature-request-remove-log-scraping). This is the least durable part of the
design and the first thing to replace.
### ADR-4: In-memory state, single replica
- **Decision:** idle clocks, handshake cache, and in-flight tracking live in gateway memory;
the Deployment runs one replica.
- **Why:** the simplest thing that works, with no extra datastore for a demo/reference
platform.
- **Cost:** a restart forgets idle timers (sandboxes stay Running until their next request) and
the handshake cache; two replicas would double-suspend and race. Scaling out requires moving
this state to a shared store (Redis/Postgres) or leader election. Documented instead of
hidden.
### ADR-5: Claims as the user record
- **Decision:** a user is a `SandboxClaim` named `dsh-<user>`; the bearer-token SHA-256 lives in
a claim annotation, and the gateway marks its own claims with
`aaas.example.com/managed-by=dsh-gateway`.
- **Why:** no database, no separate account service; deleting the claim is account deletion and
cascades everything. The ownership annotation keeps a namespaced claim list safe to share with
objects created by hand or by other controllers.
- **Cost:** no password reset, no admin API, no audit log of its own; Kubernetes API load grows
with users. Claims created before the ownership marker existed must be annotated
(`kubectl annotate sandboxclaim ...`) to keep being swept. Fine for a reference pattern, not
for a product.
### ADR-6: Opaque server-side session cookie instead of bearer tokens in the browser
- **Decision:** browsers get `dsh_gw_session=<sid>`, `HttpOnly`, `SameSite=Lax` (`Secure` when
enabled); the sid maps to `Session(user, expires_at)` in gateway memory. API clients may still
use `Authorization: Bearer`. The cookie name is `__Host-dsh_gw_session` when the Secure flag
is on and no explicit `COOKIE_NAME` is set.
- **Why:** tokens never appear in URLs or JS-readable storage, the cookie value is useless if
leaked outside the gateway session table, sessions can be revoked server-side (logout,
user delete, recreate, expiry), and a cookie cannot be reused as a bearer token for another
user's path. `SameSite=Lax` plus a strict `Origin`/`Referer` check on cookie-authenticated
mutations blocks basic CSRF.
- **Cost:** stateful auth in a single process: a restart logs everyone out and horizontal
scaling needs a shared session store. Cookie-based auth also needs TLS and the correct
`COOKIE_SECURE` setting; the gateway refuses to start with `COOKIE_SECURE=false` unless
`ALLOW_INSECURE_COOKIES=1` is set for a local HTTP demo.
## Security and RBAC
The gateway runs with a namespace-scoped ServiceAccount (`manifests/50-gateway.yaml`):
| API group | Resource | Verbs | Why |
| --- | --- | --- | --- |
| `extensions.agents.x-k8s.io` | `sandboxclaims` | `get`, `list`, `create`, `patch`, `delete` | signup, state, account deletion, token rotation |
| `agents.x-k8s.io` | `sandboxes` | `get`, `list`, `patch` | resolve DNS/pod, wake/suspend |
| `""` (core) | `pods` | `get`, `list` | resolve the pod behind a sandbox |
| `""` (core) | `pods/log` | `get` | launch-token handshake |
Notes and limits:
- **Pod logs are sensitive.** They contain the DSH launch token and possibly model output. The
permission is scoped to one namespace, but anyone who can read gateway memory or logs can
impersonate users. Prefer the upstream token API when available.
- **No cluster-wide access, no secret reads.** The gateway does not read `dsh-provider-keys`.
(It does read the optional signup invite code if it is mounted as an env var.)
- **Browser-facing hardening:** opaque server-side sessions instead of token-bearing cookies,
`HttpOnly`/`SameSite=Lax`/optional `Secure` (+ `__Host-` prefix when secure),
timing-safe SHA-256 bearer-token comparison, timing-safe invite-code comparison, strict
same-origin check on cookie-auth mutations, request-body limit (`MAX_BODY_MB`), hop-by-hop
header stripping, client identity/hop headers (`Forwarded`, `X-Forwarded-User`,
`X-Auth-Request-*`, `x-forwarded-*`/`x-auth-request-*`) stripped on HTTP and WebSocket,
upstream `Set-Cookie` dropped, DSH `Location` headers rewritten under `/u/<user>/`.
- **Session lifecycle:** login/signup mint an opaque sid; logout and user delete revoke it;
`/recreate` revokes every session for that user; expiry (`COOKIE_MAX_AGE`, default 7 days) is
enforced on lookup. The table is in memory, so a restart logs browsers out and a second
replica needs shared state.
- **Abuse controls (per replica):** `AUTH_RATE_LIMIT`/`AUTH_RATE_WINDOW` throttle
unauthenticated signup/login per client with `429` + `Retry-After`; `SIGNUP_INVITE_CODE`
gates signup; `MAX_IN_FLIGHT_PER_USER` caps concurrent proxied requests + WebSockets per user;
`WS_MAX_MESSAGE_BYTES` caps messages received from DSH; browser-to-DSH frames are bounded by
uvicorn. These are in-process guards, not an ingress rate limiter or audit log.
- **Host allow-list:** DSH rejects unknown `Host` headers. The gateway forwards the browser's
`Host`, so every public hostname must be in `DSH_TRUSTED_HOSTS`; `localhost`/`127.0.0.1` are
always trusted.
- **TLS is external.** The gateway serves plain HTTP; terminate TLS at the Ingress and set
`COOKIE_SECURE=true`. The base demo sets `COOKIE_SECURE=false` with
`ALLOW_INSECURE_COOKIES=1` for the port-forward walkthrough; without TLS, session cookies are
replayable on the network and that combination must never be exposed.
- **Tenant isolation** is pod + PVC separation; all tenants share one namespace, one DSH binary,
and the shared provider keys (`dsh-provider-keys` plus the optional
`dsh-provider-keys-extra`). The base example is demo-only: production requires gVisor for the
sandboxes, the sandbox NetworkPolicy, the gateway egress policy and the exposure NetworkPolicy,
plus per-tenant credentials or an egress proxy. See the
[production checklist](optional-hardening.md#6-production-checklist).
- **DSH's inner file sandbox is disabled by deployment override.** The template sets
`DSH_PERMISSION_MODE=danger-full-access` because the in-container backends are unusable here:
`bubblewrap` ships in the image but is unsetuid and cannot create an unprivileged user namespace
under the pod's seccomp/capability surface, and no Landlock runner is available; the confined
executor would otherwise fail closed on every command. The pod remains the boundary, so an
agent can read its own `DSH_HOME`. Hosts with a working backend can keep a confined mode;
multi-tenant deployments must put the pod in gVisor and a NetworkPolicy.
- **Optional hardening is opt-in.** gVisor, node selection, the sandbox/exposure NetworkPolicies,
extra provider keys and failed-Pod recovery are documented in
[optional hardening](optional-hardening.md). The base manifests stay portable (no gVisor
RuntimeClass, no active node labels, no NetworkPolicy, no extra Secret required, and the
controller's default `failedPodPolicy: Retain`).
- **SSO is trust-on-hop.** `AUTH_PROXY_HEADER` makes the gateway trust the proxy-asserted
identity; the SSO variant must remain the only path to the Service, and its NetworkPolicy is
required, not optional. Run [`scripts/preflight-sso.sh`](../scripts/preflight-sso.sh) after
applying it.
## Failure modes
| Failure | Symptom | Behavior / mitigation |
| --- | --- | --- |
| DSH log line changed or rotated away | Proxy returns `503` (missing launch token) | Fix the parser or restart the pod so the token line is re-emitted |
| Probe on `/` | Pod `CrashLoopBackOff`, `401` in probe logs | Use `/favicon.svg` for readiness/liveness (shipped templates do) |
| Warm pool exhausted | JSON signup returns `202` (state `Waking`); form signup is slow | Increase `replicas` in the warm pool |
| Kubernetes API error (API down or throttled, missing RBAC verb) | HTTP `503` + `Retry-After`; WebSocket close `1011` | One credential-free log line with the API `status`/`reason`, plus a fixed body; the Kubernetes `Status` payload is never proxied. Check the ServiceAccount Role and the API server — RBAC is evaluated per request, so a Role fix needs no restart |
| Gateway restart | Idle sandboxes stop suspending; handshake cache and browser sessions lost | Idle clocks and handshake cache rebuild lazily; sessions do not, so browsers must sign in again after a rollout. Use a shared session store for production |
| Gateway configured `COOKIE_SECURE=false` without `ALLOW_INSECURE_COOKIES=1` | Gateway pod exits at startup / `CrashLoopBackOff` | Intentional guard against an insecure cookie on an HTTPS deployment; set `ALLOW_INSECURE_COOKIES=1` only for the local plain-HTTP demo, or set `COOKIE_SECURE=true` |
| Auth rate limit hit | `429` + `Retry-After` on signup/login | Per-client in-memory bucket (`AUTH_RATE_LIMIT`/`AUTH_RATE_WINDOW`); wait for the window or tune it. Behind a proxy, keying uses the rightmost `X-Forwarded-For` entry |
| Signup invite missing/wrong | `403` before the claim is created | `SIGNUP_INVITE_CODE` is set; send the `invite` form field or `X-Signup-Invite` header |
| Per-user concurrency cap hit | HTTP `503` + `Retry-After`, or WebSocket close `1011` | `MAX_IN_FLIGHT_PER_USER` protects the single gateway from one busy tenant; reduce concurrency or raise the cap deliberately. These are per replica |
| Oversized WebSocket message from DSH | Connection closes | `WS_MAX_MESSAGE_BYTES` (16 MiB) is enforced on upstream messages; raise it only if DSH legitimately needs larger frames |
| Session expiry / revocation | `401` (HTTP) or close `4401` (WebSocket) | Expected after logout, user delete, `/recreate`, `COOKIE_MAX_AGE` expiry or a gateway restart; sign in again |
| SSO variant applied without its NetworkPolicy (or with an Ingress) | Gateway accepts spoofed `X-Forwarded-User` | Demo-only configuration; keep `75-networkpolicies.yaml`, remove the second path, and run `scripts/preflight-sso.sh` |
| Browser aborts a request (tab reload, back navigation) during a slow wake or handshake | No response reaches the client; nothing appears in the access log | If the abort lands before the response starts, the body read raises `ClientDisconnect` and the app answers `499` (`CLIENT_CLOSED_REQUEST`), which cannot reach the departed peer. Nothing reached DSH, no ERROR traceback is logged, and the in-flight slot is released. A request already streaming ends with its upstream status. |
| DSH auth cookie expired mid-session | Proxy `401` once, then recovers | Gateway refreshes the handshake once |
| Upstream pod name changed after resume | Cached cookie keyed by old pod UID is useless | Cache is invalidated on suspend; next request re-handshakes |
| Two gateway replicas | Double suspend/wake races | Keep `replicas: 1` until state is externalized |
| Optional NetworkPolicy blocks DNS/probes | Sandbox 502/503, pod restarts, failed `getent` | Check the CNI, DNS selector/NodeLocal address and kubelet-probe traffic; see the hardening doc |
| gVisor + `kubectl port-forward` | Forward refuses or returns nothing | Port-forward is incompatible with gVisor; use the gateway or a reachable Service |
| Provider key added but missing in the pod | Requests still fail with `MISSING_CREDENTIAL` | Pod env is stamped at creation; recreate spares/pods. A stored credential wins over env |
| `nodeSelector` matches no node | Pod `Pending`, `0/N nodes are available` | Fix the label, pool capacity or taint/toleration; RuntimeClass scheduling may also constrain it |
| Sandbox Pod reaches phase `Failed` (node eviction, spot reclaim) | Sandbox reports `Finished=True`; the gateway keeps failing against a dead Pod | Set `failedPodPolicy: Recreate` (agent-sandbox v1.0.8+) so the controller replaces the Pod with the same identity and PVCs; otherwise delete the Pod by hand. See the hardening doc |
## Upstream feature request: remove log scraping
DSH should expose the launch token as a first-class, machine-readable interface so platforms do
not need `pods/log` access. Proposed shapes (any one suffices):
1. **Token file:** `dsh web --token-file /run/dsh/token` writes the current launch token (mode
`0600`) and rewrites it on rotation.
2. **Environment input:** accept `DSH_LAUNCH_TOKEN` (platform-generated) so the token never
has to be discovered.
3. **Loopback/trusted-host endpoint:** `GET /api/auth/launch-token` that only answers on the
pod's loopback or a trusted host and returns the token plus expiry.
4. **Token in a well-known file** under `DSH_HOME` with the same semantics as (1).
Why this matters: log scraping is brittle (rotation, restart, format changes), requires a broad
`pods/log` permission that also exposes application output, and can leak tokens into log
pipelines. Until an upstream API exists, the gateway's parser, `DSH_LOG_TAIL_LINES`, and the
single-retry refresh are the supported workaround.
## Evolution path
1. Replace log scraping with the upstream token API (above), deleting the `pods/log` permission.
2. Move idle clocks, in-flight tracking, handshake cache **and sessions** to a shared store; then
scale the gateway horizontally (sticky routing not required if state is shared).
3. Add per-tenant API keys or an egress proxy so a single key is not shared by all sandboxes.
4. Add NetworkPolicies (gateway → sandboxes only, plus the documented gateway egress policy),
gVisor for sandbox pods, TLS everywhere, ingress-level rate limits/quota and audit logging.
The gateway's in-process limits are not a substitute.
5. If DSH grows absolute URLs or per-user cookies that cannot be rewritten, switch to
per-user subdomains with wildcard DNS/TLS.

View File

@@ -0,0 +1,728 @@
# Optional hardening: gVisor, nodeSelector, NetworkPolicy, provider keys and failed-Pod recovery
The base example is deliberately portable: it runs on kind and GKE without a
gVisor runtime, a tainted node pool or a NetworkPolicy-capable CNI, and it
boots with only the required `dsh-provider-keys` Secret. That portability is
exactly why the base manifests are **demo-only**. This document covers the
controls you add for production and the steps to enable or roll them back.
**Production is not the base walkthrough.** At minimum, a production deployment
needs: gVisor for the sandbox pods (or another kernel-isolation boundary), the
sandbox NetworkPolicy, the gateway egress policy, the exposure variant's
NetworkPolicy, TLS with `COOKIE_SECURE=true` (and no
`ALLOW_INSECURE_COOKIES`), SSO (or an equivalent authenticating proxy) as the
only path to the gateway, and per-tenant provider credentials/egress. The
checklist at the end summarizes this; without the network and kernel layers the
example is not tenant-isolated.
This document covers four **opt-in** controls plus failed-Pod recovery:
| Setting | Where it lives | What it changes |
| --- | --- | --- |
| gVisor | commented `runtimeClassName` in the SandboxTemplates and Deployments | puts DSH and/or the gateway in a userspace kernel; required for untrusted agents in production |
| nodeSelector | commented `nodeSelector` in the same files | pins pods to a chosen node pool |
| NetworkPolicy | `manifests/optional/network-policy/`, `manifests/exposure/*/`, plus the gateway egress recipe | limits sandbox, gateway and proxy connectivity; the sandbox and gateway-egress policies are required for production |
| Provider keys | optional `dsh-provider-keys-extra` Secret | injects extra provider API keys into every sandbox |
| Failed-Pod recovery | commented `failedPodPolicy` in the SandboxTemplates | replaces a Failed sandbox Pod instead of parking the sandbox |
Each setting is independent. Enable only what the cluster supports and test the
walkthrough after each change.
## 1. gVisor
### What it changes
The DSH sandbox and the gateway (and optionally oauth2-proxy) run under the
`gvisor` RuntimeClass. `agent-sandbox` passes `runtimeClassName` straight to the
PodSpec; no CRD change is involved.
### Prerequisites
- A `RuntimeClass` named `gvisor`:
```bash
kubectl get runtimeclass gvisor -o yaml
```
- Nodes capable of running it (`runsc` installed and registered with the
container runtime). How you get there is cluster-specific:
- GKE Sandbox: create a node pool with `--sandbox=type=gvisor`; GKE labels
the nodes `sandbox.gke.io/runtime: gvisor`, taints them
`sandbox.gke.io/runtime=gvisor:NoSchedule`, and its `gvisor` RuntimeClass
applies the matching affinity and toleration automatically.
- Self-managed: install gVisor, create the `gvisor` RuntimeClass with
`handler: runsc`, and label/taint a dedicated node pool yourself.
- A scheduling plan: a gVisor RuntimeClass without capable nodes makes pods
fail to start. Use the RuntimeClass `scheduling` fields or the optional
`nodeSelector`/toleration described in section 2.
### Enable it
**Fresh install:** uncomment `runtimeClassName: gvisor` before applying:
- [`manifests/10-sandboxtemplate.yaml`](../manifests/10-sandboxtemplate.yaml)
- [`manifests/50-gateway.yaml`](../manifests/50-gateway.yaml)
- [`manifests/exposure/gateway-api-sso/72-oauth2-proxy.yaml`](../manifests/exposure/gateway-api-sso/72-oauth2-proxy.yaml)
(only with the SSO variant)
- [minimal template](../../minimal-single-tenant/manifests/10-sandboxtemplate.yaml)
**Existing install:** a template edit does not update existing Sandboxes, and
warm-pool spares use `OnReplenish`, so patch explicitly. Plan a maintenance
window: claimed users are disconnected when their pod is recreated.
```bash
# 1. The blueprint for future sandboxes.
kubectl -n dsh-demo patch sandboxtemplate dsh-web --type merge \
-p '{"spec":{"podTemplate":{"spec":{"runtimeClassName":"gvisor"}}}}'
# 2. Warm-pool spares: delete them; the pool replenishes from the template.
kubectl -n dsh-demo delete sandboxes -l agents.x-k8s.io/warm-pool-sandbox
# 3. Claimed sandboxes: snapshot their pod template, then suspend/resume.
for sb in $(kubectl -n dsh-demo get sandboxes \
-l '!agents.x-k8s.io/warm-pool-sandbox' -o jsonpath='{.items[*].metadata.name}'); do
kubectl -n dsh-demo patch sandbox "$sb" --type merge \
-p '{"spec":{"podTemplate":{"spec":{"runtimeClassName":"gvisor"}}}}'
kubectl -n dsh-demo patch sandbox "$sb" --type merge \
-p '{"spec":{"operatingMode":"Suspended"}}'
kubectl -n dsh-demo patch sandbox "$sb" --type merge \
-p '{"spec":{"operatingMode":"Running"}}'
done
# 4. The gateway (and oauth2-proxy when the SSO variant is installed).
kubectl -n dsh-demo patch deploy dsh-gateway --type merge \
-p '{"spec":{"template":{"spec":{"runtimeClassName":"gvisor"}}}}'
kubectl -n dsh-demo rollout status deploy/dsh-gateway --timeout=180s
# kubectl -n dsh-demo patch deploy oauth2-proxy --type merge \
# -p '{"spec":{"template":{"spec":{"runtimeClassName":"gvisor"}}}}'
```
If the pool's nodes are tainted and the RuntimeClass does not already add a
toleration, add the matching `tolerations` block (section 2) in the same patch
or in the manifests.
### Verify
```bash
SB=$(kubectl -n dsh-demo get sandboxclaim dsh-alice -o jsonpath='{.status.sandbox.name}')
POD=$(kubectl -n dsh-demo get sandbox "$SB" \
-o jsonpath='{.metadata.annotations.agents\.x-k8s\.io/pod-name}'); POD=${POD:-$SB}
kubectl -n dsh-demo get pod "$POD" -o jsonpath='{.spec.runtimeClassName}{"\n"}'
# gvisor
kubectl -n dsh-demo exec "$POD" -- uname -r
kubectl -n dsh-demo get node "$(kubectl -n dsh-demo get pod "$POD" -o jsonpath='{.spec.nodeName}')" \
-o jsonpath='{.status.nodeInfo.kernelVersion}{"\n"}'
```
The pod's `runtimeClassName` is authoritative. The kernel release reported by
`uname -r` inside a gVisor pod is the sandbox kernel, not the host kernel; on
GKE it looks like `4.19.0-gvisor`.
### Caveats
- **`kubectl port-forward` does not work with gVisor.** The multi-tenant
gateway path (HTTP/WebSocket to the pod) works; the minimal example's
port-forward quickstart does not. Use the gateway example or a reachable
Service (NodePort/LoadBalancer/Sandbox Router) instead.
- gVisor has performance overhead and does not implement every syscall. Test
DSH and the agent tools you rely on (`git`, `gh`, `uv`, language toolchains).
- Node scheduling is the hard part. A `gvisor` RuntimeClass on a cluster with
no capable nodes leaves pods unschedulable or failing at container start.
- Existing Sandboxes are snapshots. The loop above is required for every
claimed sandbox; deleting the `SandboxClaim` would also delete its PVC and
workspace data, so do not use claim re-creation as a shortcut.
- A resumed sandbox starts cold; the startup probe gives it up to 120s.
### Roll back
```bash
kubectl -n dsh-demo patch sandboxtemplate dsh-web --type merge \
-p '{"spec":{"podTemplate":{"spec":{"runtimeClassName":null}}}}'
for sb in $(kubectl -n dsh-demo get sandboxes -o jsonpath='{.items[*].metadata.name}'); do
kubectl -n dsh-demo patch sandbox "$sb" --type merge \
-p '{"spec":{"podTemplate":{"spec":{"runtimeClassName":null}}}}'
done
kubectl -n dsh-demo delete sandboxes -l agents.x-k8s.io/warm-pool-sandbox
kubectl -n dsh-demo patch deploy dsh-gateway --type merge \
-p '{"spec":{"template":{"spec":{"runtimeClassName":null}}}}'
# kubectl -n dsh-demo patch deploy oauth2-proxy --type merge \
# -p '{"spec":{"template":{"spec":{"runtimeClassName":null}}}}'
```
Recreate the pods (delete spares, suspend/resume claimed sandboxes, let the
Deployment roll out) so the cleared field takes effect.
## 2. nodeSelector
### What it changes
Pods get a `nodeSelector` so they land only on nodes carrying the chosen label.
The same field is available in both SandboxTemplates and both Deployments;
the shipped value is **commented out** because active cluster-specific
scheduling does not belong in the repository.
### Choose a label
```bash
kubectl get nodes --show-labels
```
- **GKE Sandbox:** the nodes carry `sandbox.gke.io/runtime: gvisor` and are
tainted `sandbox.gke.io/runtime=gvisor:NoSchedule`. With
`runtimeClassName: gvisor`, GKE already schedules the pod onto those nodes;
add a `nodeSelector` only when you need a specific sandbox pool or
ComputeClass (`cloud.google.com/compute-class: <name>`).
- **Self-managed:** label the pool you want, for example
`dsh.example.com/node-pool: gvisor`, and use that label.
- **General pinning:** any stable node-pool label works. Keep the real key and
value in a local patch; the manifests use `dsh.example.com/node-pool` as an
obvious placeholder.
### Enable it
Uncomment and replace the placeholder before a fresh apply:
```yaml
nodeSelector:
dsh.example.com/node-pool: gvisor
```
For an existing install, patch the template, each claimed Sandbox and the
Deployments, then recreate pods exactly as in section 1:
```bash
kubectl -n dsh-demo patch sandboxtemplate dsh-web --type merge \
-p '{"spec":{"podTemplate":{"spec":{"nodeSelector":{"dsh.example.com/node-pool":"gvisor"}}}}}'
# ... same warm-pool delete / claimed-sandbox suspend-resume / Deployment
# patch and rollout as in section 1, with nodeSelector instead of
# runtimeClassName.
```
If the selected nodes are tainted, add a matching toleration. This is the
companion block; do **not** uncomment it unless the pool is actually tainted:
```yaml
tolerations:
- key: dsh.example.com/node-pool
operator: Equal
value: gvisor
effect: NoSchedule
```
GKE Sandbox's taint is `sandbox.gke.io/runtime=gvisor:NoSchedule`; the GKE
`gvisor` RuntimeClass adds that toleration automatically, so a manual copy is
only needed when you schedule without the RuntimeClass or with a different
pool.
### Verify and troubleshoot
```bash
kubectl -n dsh-demo get pod "$POD" -o wide
kubectl -n dsh-demo get pod "$POD" -o jsonpath='{.spec.nodeSelector}{"\n"}'
kubectl -n dsh-demo describe pod "$POD" | sed -n '/Events:/,$p'
```
- `Pending` with `0/N nodes are available`: the label is wrong, the pool is
full, or the pod needs a toleration for a taint.
- A pod-level `nodeSelector` is merged with the RuntimeClass
`scheduling.nodeSelector`. If both are set and conflict, no node matches.
- Labels changed after scheduling do not move running pods. Patch/recreate.
- A `nodeSelector` that excludes every gVisor node is a valid way to keep the
gateway on a normal pool while the sandboxes run under gVisor.
### Roll back
Remove the field from the manifests, or patch it to `null`, then recreate the
pods:
```bash
kubectl -n dsh-demo patch sandboxtemplate dsh-web --type merge \
-p '{"spec":{"podTemplate":{"spec":{"nodeSelector":null}}}}'
# ... same per-Sandbox and Deployment loop as section 1.
```
## 3. NetworkPolicy
The base example does **not** manage NetworkPolicy: the SandboxTemplate is
`networkPolicyManagement: Unmanaged`, and the gateway has no policy. That keeps
the walkthrough portable and independent of the sandbox-router. **For
production this is not optional**: without the sandbox policy every tenant can
reach every other tenant and cluster-internal endpoints, and without a gateway
egress policy the gateway can talk to anything the cluster network allows.
Apply both. The policies below are standard Kubernetes `NetworkPolicy` objects.
> **Demo vs production.** An installation that skips the sandbox NetworkPolicy
> and the gateway egress policy is a demo. Exposure (Ingress/Gateway API) adds
> ingress protection only; it does not isolate tenants from each other.
### Sandbox policy (required for production)
[`manifests/optional/network-policy/10-sandbox-networkpolicy.yaml`](../manifests/optional/network-policy/10-sandbox-networkpolicy.yaml)
selects every Sandbox pod through the agent-sandbox tracking label
`agents.x-k8s.io/sandbox-name-hash`. It applies to existing warm-pool and
claimed sandboxes without recreating them.
```bash
kubectl apply -f manifests/optional/network-policy/
kubectl delete -f manifests/optional/network-policy/ # rollback
```
What it allows:
| Direction | Allowed | Blocked |
| --- | --- | --- |
| Ingress | gateway pods (`app: dsh-gateway`) on TCP `3080` | everything else, including other sandboxes and cluster Services |
| Egress | UDP/TCP `53` to kube-dns and NodeLocal DNSCache; TCP `443` to public IPs | RFC1918, CGNAT, link-local/metadata, cluster Services/API, plain HTTP, other ports |
Verify from inside the pods (NetworkPolicy does not affect `kubectl exec`
itself, only the traffic the commands generate):
```bash
SB=$(kubectl -n dsh-demo get sandboxclaim dsh-alice -o jsonpath='{.status.sandbox.name}')
POD=$(kubectl -n dsh-demo get sandbox "$SB" \
-o jsonpath='{.metadata.annotations.agents\.x-k8s\.io/pod-name}'); POD=${POD:-$SB}
# Allowed: gateway -> sandbox.
kubectl -n dsh-demo exec deploy/dsh-gateway -- python -c \
"import urllib.request; print(urllib.request.urlopen('http://$POD:3080/favicon.svg', timeout=5).status)"
# Allowed: DNS and public HTTPS from the sandbox.
kubectl -n dsh-demo exec "$POD" -- getent hosts api.deepseek.com
kubectl -n dsh-demo exec "$POD" -- curl -fsS -o /dev/null -w '%{http_code}\n' https://example.com
# Denied: another sandbox cannot reach this one on 3080 (expect a timeout).
OTHER_SB=$(kubectl -n dsh-demo get sandboxclaim dsh-bob -o jsonpath='{.status.sandbox.name}')
OTHER_POD=$(kubectl -n dsh-demo get sandbox "$OTHER_SB" \
-o jsonpath='{.metadata.annotations.agents\.x-k8s\.io/pod-name}'); OTHER_POD=${OTHER_POD:-$OTHER_SB}
kubectl -n dsh-demo exec "$OTHER_POD" -- curl --max-time 3 -sS "http://$POD:3080/favicon.svg"
```
Caveats:
- **The CNI must enforce NetworkPolicy.** If it does not, the file applies but
has no effect.
- DNS: the default assumes CoreDNS/kube-dns pods labelled `k8s-app: kube-dns`
and covers NodeLocal DNSCache at `169.254.20.10`. A different DNS layout
needs a selector/`ipBlock` edit.
- **Kubelet probes** from the node are CNI-dependent. If a pod starts failing
its probes, allow the node CIDR on the probe port or remove the ingress
policy.
- The policy blocks cluster Services and the Kubernetes API from the sandbox.
DSH does not need them, but an agent tool that does needs an explicit rule.
- It allows any public host on TCP `443`, not an FQDN allowlist. It is network
isolation, not data-loss prevention.
- Custom provider base URLs, an HTTP proxy, port `80`, SSH `git` (port `22`) or
a non-443 provider endpoint need explicit egress additions.
- Do not combine this standalone policy with `networkPolicyManagement: Managed`
unless you migrate the rules into `spec.networkPolicy`. NetworkPolicies are
additive; the controller's Managed secure default would also be present.
- `kubectl port-forward` still works for debugging (the connection is local to
the pod), except under gVisor.
### Exposure-variant policies (required, part of the variant)
Each exposure variant ships its own policy. They are part of the variant and
**required for production**; an exposure without its policy is demo-only:
- [`manifests/exposure/ingress/61-networkpolicy.yaml`](../manifests/exposure/ingress/61-networkpolicy.yaml)
allows only Traefik pods (`app.kubernetes.io/name: traefik`) to reach the
gateway on `8080`.
- [`manifests/exposure/gateway-api-sso/75-networkpolicies.yaml`](../manifests/exposure/gateway-api-sso/75-networkpolicies.yaml)
allows only oauth2-proxy to reach the gateway, only the Gateway
implementation (Traefik) to reach oauth2-proxy, and limits oauth2-proxy
egress to DNS, public HTTPS (GitHub) and the gateway.
The SSO policy is what turns the trust boundary in the
[exposure README](../manifests/exposure/README.md) into network enforcement.
The `dsh-gateway` Service must still stay ClusterIP; do not apply both exposure
variants. After applying the SSO variant, run
[`scripts/preflight-sso.sh`](../scripts/preflight-sso.sh): it checks the
gateway's SSO env, the Secrets, the TLS reference, leftover placeholders and
accidental second paths (such as an Ingress).
Verify the negative case from a sandbox pod (it is not the proxy):
```bash
kubectl -n dsh-demo exec "$POD" -- curl --max-time 3 -sS http://dsh-gateway:8080/healthz
# expected: timeout/connection failure
```
Then verify the real browser path through the Ingress or Gateway, and confirm
the gateway/proxy pods are `Ready`. If Traefik uses `hostNetwork` or a
different pod label, replace the `podSelector` with the correct selector or an
`ipBlock`. As with the sandbox policy, kubelet probes can be affected on some
CNIs.
### Gateway egress (required for production)
The gateway needs DNS, the Kubernetes API, and TCP `3080` to sandbox pods. An
egress policy is **required for production** but is not shipped, because the API
server endpoint is cluster-specific (Service CIDR or node IP/port) and a wrong
value silently breaks signup/proxy. If your cluster has a stable API-server CIDR
and port, add an egress policy like this (replace the placeholders; verify the
port, commonly `443` or `6443`):
```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: dsh-gateway-egress
namespace: dsh-demo
spec:
podSelector:
matchLabels:
app: dsh-gateway
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
- to:
- ipBlock:
cidr: 10.96.0.1/32 # placeholder: API-server Service/cluster endpoint
ports:
- protocol: TCP
port: 443
- to:
- podSelector:
matchExpressions:
- key: agents.x-k8s.io/sandbox-name-hash
operator: Exists
ports:
- protocol: TCP
port: 3080
```
Cilium users can express the API server as `toEntities: kube-apiserver` (or
`toServices`) instead of a CIDR. Do not apply the sample blindly: a wrong CIDR
or port silently breaks signup/proxy.
## 4. Optional provider keys
### How injection works
The SandboxTemplate injects two Secrets via `envFrom`:
- `dsh-provider-keys` — required by the walkthrough; holds
`DEEPSEEK_API_KEY`.
- `dsh-provider-keys-extra` — optional; referenced with `optional: true`, so a
missing Secret does not block sandbox startup. Every key in it becomes an
environment variable in every sandbox.
Create the extra Secret before the warm pool (pod env is stamped at creation):
```bash
kubectl -n dsh-demo create secret generic dsh-provider-keys-extra \
--from-literal=OPENROUTER_API_KEY="sk-or-..." \
--from-literal=GEMINI_API_KEY="..." \
--from-literal=NVIDIA_API_KEY="nvapi-..." \
--dry-run=client -o yaml | kubectl apply -f -
```
Use the exact env-var names (uppercase, underscores) — invalid Secret keys are
skipped by the kubelet with an event. Do not duplicate a key across the two
Secrets; when the same key exists twice, the later `envFrom` entry wins.
### Supported names (curated)
These are the names the pinned DSH release resolves through its packaged provider
catalog; the catalog has more entries, these are the common ones:
| Provider | Environment variable |
| --- | --- |
| DeepSeek | `DEEPSEEK_API_KEY` |
| OpenRouter | `OPENROUTER_API_KEY` |
| Google Gemini | `GEMINI_API_KEY` |
| NVIDIA | `NVIDIA_API_KEY` |
| OpenAI | `OPENAI_API_KEY` |
| Anthropic | `ANTHROPIC_API_KEY` |
| Groq | `GROQ_API_KEY` |
| xAI | `XAI_API_KEY` |
| Mistral | `MISTRAL_API_KEY` |
| Together | `TOGETHER_API_KEY` |
| Cerebras | `CEREBRAS_API_KEY` |
| Vercel AI Gateway | `AI_GATEWAY_API_KEY` |
| Hugging Face | `HF_TOKEN` |
Aliases to avoid: `GOOGLE_API_KEY`, `HUGGINGFACE_API_KEY` and
`VERCEL_AI_GATEWAY_API_KEY`. The packaged catalog looks for `GEMINI_API_KEY`,
`HF_TOKEN` and `AI_GATEWAY_API_KEY` instead.
### Picking up a key change
A Secret update does not change running pods. The base key and extra keys are
both snapshotted at container start:
```bash
# Warm-pool spares: delete them so the pool recreates them from the template.
kubectl -n dsh-demo delete sandboxes -l agents.x-k8s.io/warm-pool-sandbox
# Running claimed sandboxes: delete the pod (the Sandbox controller recreates
# it) or suspend/resume the sandbox.
POD=... # resolve it as in section 3
kubectl -n dsh-demo delete pod "$POD"
```
New claims get the new values automatically. Confirm names without printing
secret values:
```bash
kubectl -n dsh-demo exec "$POD" -- sh -c '
for k in DEEPSEEK_API_KEY OPENROUTER_API_KEY GEMINI_API_KEY NVIDIA_API_KEY; do
if [ -n "$(printenv "$k" || true)" ]; then echo "$k=set"; else echo "$k=unset"; fi
done'
```
### Precedence and shared-key limits
- DSH's stored credential file (`$DSH_HOME/.credentials.yaml`, mode `0600`)
**wins over the environment**. If a tenant saved a key through the DSH UI,
changing the Secret will not override it.
- Both Secrets are shared by **every** tenant, and any agent can read its own
environment. This is a warm-pool constraint, not tenant isolation. A tenant
can override its own key through the DSH credential store on its PVC, but the
platform does not manage that; real multi-tenancy needs per-tenant
credentials or an egress proxy.
- The optional Secret is created imperatively; the repository never contains a
real key. Keep it out of Git and out of shell history where possible.
## 5. Failed-Pod recovery
### What it changes
When a sandbox Pod reaches phase `Failed`, the controller's default policy,
`Retain`, leaves the Pod in place and surfaces `Finished=True` on the Sandbox.
The Sandbox, its claim and its PVC survive, but nothing runs: the gateway keeps
proxying to a dead Pod and every request fails until someone deletes the Pod or
the claim.
agent-sandbox v1.0.8+ (the current pin in [versions.env](../../../versions.env) is newer) adds
`spec.failedPodPolicy` to `Sandbox` and to `SandboxTemplate`:
| Value | Behavior |
| --- | --- |
| `Retain` (default) | The Failed Pod stays and the Sandbox reports `Finished=True`; the evidence stays available for debugging. |
| `Recreate` | The controller deletes the Failed Pod and starts a replacement; the Sandbox identity and its Sandbox-owned PVCs are retained. |
Repeated failures back off like a Deployment - 5 s, doubling to 5 min between
creations - so a crash loop cannot hammer the API server. The backoff resets
once the replacement Pod has been continuously `Running` for 10 minutes, and it
lives in memory only, so a controller restart clears it.
The base templates ship it commented out, for two reasons:
- A deterministic failure (bad image, bad Secret, a wrong probe) would recreate
forever while hiding the cause.
- The templates use `restartPolicy: OnFailure`, so the kubelet restarts a
crashing container in place and the Pod does not reach phase `Failed` anyway.
This policy matters for terminal failures - most commonly node eviction
(`Evicted`), when a spot/preemptible node is reclaimed or the node hits memory
pressure.
No gateway change is needed: the handshake cache is keyed by Pod UID, so a
replacement Pod re-handshakes on the next request, and the existing `401` retry
covers containers restarted in place.
### Enable it
```bash
# 1. The blueprint for future sandboxes.
kubectl -n dsh-demo patch sandboxtemplate dsh-web --type merge \
-p '{"spec":{"failedPodPolicy":"Recreate"}}'
# 2. Warm-pool spares were stamped at creation: delete them and let the pool
# replenish from the updated template.
kubectl -n dsh-demo delete sandboxes -l agents.x-k8s.io/warm-pool-sandbox
# 3. Claimed sandboxes keep their own spec: patch them in place.
for sb in $(kubectl -n dsh-demo get sandboxes \
-l '!agents.x-k8s.io/warm-pool-sandbox' -o jsonpath='{.items[*].metadata.name}'); do
kubectl -n dsh-demo patch sandbox "$sb" --type merge \
-p '{"spec":{"failedPodPolicy":"Recreate"}}'
done
```
For a fresh install, uncomment `failedPodPolicy: Recreate` in the template
instead.
### Verify
```bash
SB=$(kubectl -n dsh-demo get sandboxclaim dsh-alice -o jsonpath='{.status.sandbox.name}')
kubectl -n dsh-demo get sandbox "$SB" -o jsonpath='{.spec.failedPodPolicy}{"\n"}'
# Recreate
```
A failure drill that never touches a user sandbox - a `Sandbox` with
`restartPolicy: Never` and a container that exits non-zero (swap the image if
Docker Hub is unreachable):
```bash
kubectl -n dsh-demo apply -f - <<'EOF'
apiVersion: agents.x-k8s.io/v1beta1
kind: Sandbox
metadata:
name: failed-pod-drill
spec:
failedPodPolicy: Recreate
podTemplate:
spec:
restartPolicy: Never
containers:
- name: drill
image: busybox:1.36
command: ["sh", "-c", "exit 1"]
EOF
kubectl -n dsh-demo get pods -w
# The drill Pod reaches phase Failed; a replacement appears within ~5 s, then
# 10 s, 20 s, ... as the container keeps exiting non-zero.
kubectl -n dsh-demo delete sandbox failed-pod-drill
```
Repeat the drill with `failedPodPolicy: Retain` (or omit the field) to see the
default: the Pod stays `Failed` and the Sandbox condition `Finished` becomes
`True`.
### Caveats
- `Recreate` deletes the Failed Pod, so its logs go with it. Capture
`kubectl get events` or switch back to `Retain` when debugging.
- The replacement is a cold start: DSH boots again and issues a new launch
token. The workspace comes back from the PVC; in-flight sessions do not.
- The field needs agent-sandbox v1.0.8+. A v1.0.7 cluster rejects it as an
unknown field.
- It applies to sandbox Pods only. The gateway and oauth2-proxy are plain
Deployments, and their ReplicaSets already replace Failed Pods.
### Roll back
```bash
kubectl -n dsh-demo patch sandboxtemplate dsh-web --type merge \
-p '{"spec":{"failedPodPolicy":"Retain"}}'
for sb in $(kubectl -n dsh-demo get sandboxes -o jsonpath='{.items[*].metadata.name}'); do
kubectl -n dsh-demo patch sandbox "$sb" --type merge \
-p '{"spec":{"failedPodPolicy":"Retain"}}'
done
```
Warm-pool spares pick the default up as they are claimed and replenished, or
delete them as in step 2 above.
## 6. Production checklist
The base walkthrough is a demo. A production install must satisfy every item
below; the sections above give the commands and caveats.
**Isolation and network**
- [ ] Sandbox pods run under gVisor (or an equivalent kernel-isolation
boundary) because DSH runs with `DSH_PERMISSION_MODE=danger-full-access`
in this image; the pod is the boundary. (Section 1.)
- [ ] The sandbox NetworkPolicy is applied and verified: no tenant→tenant,
no tenant→cluster-Service/API, DNS + public HTTPS only. (Section 3.)
- [ ] The gateway egress policy is applied with the cluster's real API-server
CIDR/port, scoped to DNS, the API server and sandbox pods on `3080`.
(Section 3, "Gateway egress".)
- [ ] The exposure variant's NetworkPolicy is applied. An exposure without it is
demo-only. For SSO, the gateway is reachable **only** through
oauth2-proxy; there is no second Ingress/Gateway/port-forward path.
- [ ] Nodes/CNI enforce NetworkPolicy (verify the negative cases, not just the
positive path).
- [ ] The gateway and (optionally) oauth2-proxy run with `runAsNonRoot`,
dropped capabilities, `RuntimeDefault` seccomp and a read-only root
filesystem, as shipped.
**Authentication, TLS and abuse controls**
- [ ] TLS terminates in front of the gateway, `COOKIE_SECURE=true`, and
`ALLOW_INSECURE_COOKIES` is **not** set. The gateway refuses to start with
an insecure cookie unless explicitly acknowledged.
- [ ] SSO (or an equivalent authenticating proxy) is enabled with
`AUTH_PROXY_HEADER=X-Forwarded-User`, and
[`scripts/preflight-sso.sh`](../scripts/preflight-sso.sh) exits `0`.
- [ ] No `your-github-org` / `dsh-demo.example.com` placeholder remains.
- [ ] `SIGNUP_INVITE_CODE` is set from a Secret if the gateway's own signup form
is reachable, and `AUTH_RATE_LIMIT` is set deliberately (`30`/60s shipped).
Add ingress-level rate limiting/quota too; the gateway's buckets are
in-process and reset on restart.
- [ ] Session lifetime (`COOKIE_MAX_AGE`) and the logout/delete revocation
behavior are acceptable; account for gateway restarts logging users out.
- [ ] `MAX_IN_FLIGHT_PER_USER` and `WS_MAX_MESSAGE_BYTES` fit the workload.
**Credentials, platform and operations**
- [ ] Provider keys are per-tenant (or routed through an egress proxy) instead
of the shared `dsh-provider-keys` Secrets; any shared key is treated as
visible to every tenant.
- [ ] RBAC stays namespace-scoped; `pods/log` is treated as sensitive because
it contains launch tokens and model output. Plan the upstream token API
migration.
- [ ] Images are digest-pinned and dependency installs are hash-locked.
- [ ] Gateway replicas stay at `1` (in-memory sessions/idle state) or the state
is externalized.
- [ ] `failedPodPolicy: Recreate` is set if node eviction/spot reclaim is
expected.
- [ ] Audit logging, alerting and backups of the PVCs are in place.
## 7. Verification summary
Required locally for any change to this example:
```bash
make check # versions, example layout, manifest YAML shape
yamllint .
make test # only when the gateway code changed
shellcheck examples/dsh-agents-as-a-service/scripts/*.sh
```
CI additionally parses every manifest and runs `kubectl apply
--dry-run=server` against the real agent-sandbox CRDs. The weekly kind e2e
runs the base walkthrough; it also proves the optional extra Secret may be
absent.
Manual checks, on a cluster with the relevant CNI/runtime:
- gVisor: pod `runtimeClassName`, `uname -r` differs from the node kernel,
gateway proxy works, port-forward limitation acknowledged.
- nodeSelector: pods land on the intended nodes; no `Pending` events.
- Sandbox policy: the allow/deny matrix above.
- Gateway egress policy: signup and proxy still work; the gateway cannot reach
an arbitrary cluster Service or the metadata server.
- Exposure policies: browser path works; a non-proxy pod cannot reach the
gateway; probes stay healthy. `scripts/preflight-sso.sh` exits `0`.
- Cookie/TLS guard: with `COOKIE_SECURE=false` and no
`ALLOW_INSECURE_COOKIES` the gateway refuses to start; with TLS it starts and
the cookie is `Secure` (`__Host-` prefixed).
- Auth abuse controls: `429` + `Retry-After` after the configured limit, `403`
on a wrong invite code, and revocations on logout/delete/recreate.
- Provider keys: set/unset output (never the values) and the refresh procedure.
- Failed-Pod recovery: `spec.failedPodPolicy` on the Sandbox, and the drill
above replaces a `Failed` Pod while keeping the claim and PVC.
## 8. Rollback summary
| Layer | Rollback |
| --- | --- |
| Sandbox policy | `kubectl delete -f manifests/optional/network-policy/` |
| Gateway egress policy | `kubectl delete networkpolicy dsh-gateway-egress -n dsh-demo` (do not leave it half-configured) |
| Exposure policy | delete the exposure variant directory |
| gVisor | patch `runtimeClassName: null`, recreate pods |
| nodeSelector | patch `nodeSelector: null`, recreate pods |
| Provider keys | delete the extra Secret or a key, recreate pods |
| Failed-Pod recovery | patch `failedPodPolicy: Retain`, delete warm-pool spares |
No PVC or workspace data is touched by any rollback. Deleting a `SandboxClaim`
is different: it cascades to the Sandbox **and** its PVC, so do not use claim
deletion to apply a pod-spec change.

View File

@@ -0,0 +1,11 @@
# Runtime image only needs the app and its pinned runtime requirements.
__pycache__/
*.py[cod]
.pytest_cache/
.ruff_cache/
.venv/
test_gateway.py
requirements-dev.txt
pyproject.toml
README.md
coverage.xml

View File

@@ -1,16 +1,73 @@
FROM python:3.12-slim
# syntax=docker/dockerfile:1.7
#
# Example gateway image: the tenant-facing proxy in front of DSH sandboxes.
# The base image is digest-pinned in versions.env (GATEWAY_PYTHON_IMAGE) and
# scripts/check-versions.sh verifies the ARG default below.
#
# Two stages on that base:
# build installs the fully hash-pinned runtime dependencies into /venv as
# a dedicated non-root UID, so the runtime stage copies a venv that
# contains no root-owned files.
# runtime copies /venv and the app, runs as UID 10000 with HOME=/tmp and no
# writable location of its own: usable with readOnlyRootFilesystem.
ARG GATEWAY_PYTHON_IMAGE=docker.io/python:3.12-slim@sha256:aa943ce411423c7582288f18707ff66eddcbaeb510502d747a72aeb34f30d159
# Read-only-rootfs friendly: no .pyc writes, no home directory writes.
# ---------------------------------------------------------------------------
# Build stage: hash-pinned virtualenv owned by the runtime identity.
# ---------------------------------------------------------------------------
FROM $GATEWAY_PYTHON_IMAGE AS build
ARG APP_UID=10000
ARG APP_GID=10000
# No .pyc writes and no user cache dirs: the runtime image stays read-only.
ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1 \
PIP_NO_CACHE_DIR=1 \
HOME=/tmp
WORKDIR /app
RUN groupadd --gid "$APP_GID" app \
&& useradd --uid "$APP_UID" --gid "$APP_GID" --no-create-home \
--home-dir /tmp --shell /usr/sbin/nologin app \
&& python -m venv /venv \
&& chown -R "$APP_UID:$APP_GID" /venv
WORKDIR /build
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY gateway.py .
# Install as the unprivileged user so every file in /venv is owned by UID 10000.
# --require-hashes makes pip reject any requirement without a matching hash:
# the dependency set in requirements.txt is fully pinned and verified. Use the
# venv's own pip: the system pip would user-install into $HOME instead.
USER $APP_UID:$APP_GID
RUN /venv/bin/pip install --no-cache-dir --require-hashes -r requirements.txt
# ---------------------------------------------------------------------------
# Runtime stage
# ---------------------------------------------------------------------------
FROM $GATEWAY_PYTHON_IMAGE AS runtime
ARG APP_UID=10000
ARG APP_GID=10000
ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1 \
HOME=/tmp \
PATH="/venv/bin:$PATH"
RUN groupadd --gid "$APP_GID" app \
&& useradd --uid "$APP_UID" --gid "$APP_GID" --no-create-home \
--home-dir /tmp --shell /usr/sbin/nologin app
WORKDIR /app
# Dependencies come from the build stage: same interpreter, no build tools left
# in the runtime image and nothing owned by root inside /venv.
COPY --from=build --chown=$APP_UID:$APP_GID /venv /venv
COPY --chown=$APP_UID:$APP_GID gateway.py .
EXPOSE 8080
# Run as a fixed non-root UID; HOME=/tmp keeps caches out of the image.
USER 10000

View File

@@ -4,21 +4,28 @@ A single-file, browser-usable control + data plane for
[DeepSeek Harness Web (DSH)](https://github.com/deepseek-ai/deepseek-harness)
sandboxes running behind the `agent-sandbox` CRDs.
It does three things:
It does four things:
1. **Users** — creates a `SandboxClaim` from a warm pool, mints a bearer token
(only its SHA-256 is stored as a claim annotation), reports derived state,
and deletes the claim (cascading the sandbox and PVC).
2. **Browser sessions** — exchanges a user token for an HttpOnly session
cookie so the DSH UI works in a browser at `/u/<user>/`. Automation can
still use `Authorization: Bearer <token>`.
cookie whose value is only an opaque, server-side session id (user and
expiry stay in the gateway's memory) so the DSH UI works in a browser at
`/u/<user>/`. Automation can still use
`Authorization: Bearer <token>`, which never comes from the cookie.
3. **Reverse proxy** — proxies HTTP (including SSE) and WebSocket traffic to
the sandbox's DSH Web server on `:3080`, including DSH's own cookie
handshake.
handshake. Client-supplied forwarding/identity headers are stripped.
4. **Optional SSO** — when `AUTH_PROXY_HEADER` is set, trusts the identity
header an authenticating reverse proxy (for example oauth2-proxy) forwards,
auto-provisions that user, and exposes `POST /recreate` for a clean restart.
DSH is pinned to `dsh-v0.2.1-rc.2` in the example manifests. It binds `:3080`,
serves `/`, gates `/api` behind a `dsh-auth-*` cookie and requires a WebSocket
at `/api/remote.mux`.
The DSH version pin lives in [`versions.env`](../../../versions.env)
(`DSH_VERSION` plus the optional repository `RELEASE_REVISION`); the example
manifests and published images use the release identity built from it (currently
`dsh-v0.2.1-rc.5-r6`). DSH binds `:3080`, gates the UI and `/api` behind a
`dsh-auth-*` cookie and requires a WebSocket at `/api/remote.mux`.
## Flow
@@ -58,10 +65,11 @@ token. A failed handshake returns `503` with `Retry-After: 10`.
| Method | Path | Auth | Purpose |
| --- | --- | --- | --- |
| `GET` | `/healthz` | none | Liveness/readiness. Never touches the cluster. |
| `GET` | `/` | none | Landing page with signup + token login forms. |
| `POST` | `/users` | none | Signup. JSON body `{"user": "<dns-1123 label>"}` or form field `user`. JSON returns `201` when Ready, `202` when accepted but still Waking; form returns `303` to `/u/<user>/`. Both set the session cookie. `409` if the user exists, `400` if invalid, `413` if the body is too large. |
| `POST` | `/login` | none | Form (`user`, `token`) returns `303` to `/u/<user>/`; JSON returns `200`. Sets the session cookie. `401` on a bad token, `404` for an unknown user. |
| `POST` | `/logout` | none | Clears the session cookie and `303`s to `/`. |
| `GET` | `/` | none | Landing page with signup + token login forms (plus an invite-code input when `SIGNUP_INVITE_CODE` is set). With SSO it auto-provisions the proxy-identified user and redirects to their workspace. |
| `POST` | `/users` | none | Signup. JSON body `{"user": "<dns-1123 label>"}` or form field `user`; when `SIGNUP_INVITE_CODE` is set, an `invite` form field or `X-Signup-Invite` header must match it (`403` otherwise, checked before the user lookup). JSON returns `201` when Ready, `202` when accepted but still Waking; form returns `303` to `/u/<user>/`. Both set the session cookie. `409` if the user exists, `400` if invalid, `413` if the body is too large, `429` when the auth rate limit is hit. |
| `POST` | `/login` | none | Form (`user`, `token`) returns `303` to `/u/<user>/`; JSON returns `200`. Sets the session cookie. `401` on a bad token **and** for an unknown user (no username enumeration). `429` when the auth rate limit is hit. Both auth routes reject a form post whose `Origin`/`Referer` contradicts `Host` (`403`) when the browser sent one. |
| `POST` | `/logout` | none | Revokes the server-side session, clears the cookie and `303`s to `/`. |
| `POST` | `/recreate` | trusted proxy header | **SSO only.** Deletes the authenticated user's claim (cascading sandbox + PVC, so workspace data is wiped), waits out finalizers, provisions a fresh claim and `303`s to `/u/<user>/`. `401` without a valid identity header; cookie-authenticated calls must be same-origin. |
| `GET` | `/users/{user}` | bearer or cookie | Derived state: `Provisioning`, `Waking`, `Ready`, `Suspending`, `Suspended`. |
| `DELETE` | `/users/{user}` | bearer or cookie | Delete the claim (sandbox + PVC cascade). Cookie-auth mutations require a same-origin `Origin`/`Referer`. |
| `GET` | `/u/{user}` | — | `308` redirect to `/u/{user}/`. |
@@ -72,19 +80,85 @@ token. A failed handshake returns `503` with `Retry-After: 10`.
* `Authorization: Bearer <token>` is checked with a timing-safe SHA-256
comparison against the `aaas.example.com/token-sha256` claim annotation.
* The browser session cookie is
`dsh_gw_session=<user>:<token>` (HttpOnly, SameSite=Lax, Path=/, optional
Secure). A session for user A never authorizes user B.
* The browser session cookie is **opaque**: its value is a random
server-side session id; the user and expiry live in gateway memory
(`GatewayState.sessions`). A cookie can never be edited into another user's
credentials, a session id is not accepted as a bearer token, and a bearer
token never authorizes a different user. The cookie name defaults to
`__Host-dsh_gw_session` when `COOKIE_SECURE=true` (HTTPS) and to
`dsh_gw_session` for plain HTTP; attributes are HttpOnly, SameSite=Lax,
Path=/ and (with `COOKIE_SECURE`) Secure.
* Sessions are revoked on `POST /logout`, on `DELETE /users/{user}`, on
`POST /recreate`, and when they expire (looked up on every request; the
sweeper also drops expired entries). `from_env()` refuses to start without
`COOKIE_SECURE=true` or the explicit `ALLOW_INSECURE_COOKIES=1` opt-out for
local HTTP.
* Cookie-authenticated **mutations** (`POST`, `PUT`, `PATCH`, `DELETE`) and
cookie-authenticated **WebSocket upgrades** require the request `Origin`
(or `Referer` fallback) to match the `Host` header. Missing or mismatched
cross-origin metadata is rejected (`403` HTTP, close code `4401` for WS).
Form signup/login additionally reject a cross-origin `Origin`/`Referer`
when the browser sent one, before the body is read.
* `POST /users` and `POST /login` are rate limited per client:
`AUTH_RATE_LIMIT` attempts per `AUTH_RATE_WINDOW` seconds (default 30/60,
`0` disables). The client key is the rightmost `X-Forwarded-For` entry when
present (the address the last trusted proxy saw, not a spoofable leftmost
value), else the socket peer. Over-limit requests get `429` +
`Retry-After` before the body is read.
* Proxy requests strip client-supplied forwarding/identity headers in both
HTTP and WebSocket paths: `Forwarded`, every `X-Forwarded-*`
(including `X-Forwarded-User`), every `X-Auth-Request-*`, and `X-Real-IP`.
The gateway sets its own `X-Forwarded-For/Host/Proto/Port` values from the
connection. `UPSTREAM_HEADER_ALLOWLIST` optionally narrows the forwarded
client headers further (comma-separated, case-insensitive); unset keeps the
denylist behavior, and the gateway's own values are always added.
* The gateway's bearer token, the browser `Cookie` header and any `?token=`
query parameter are never forwarded upstream. Only the cached `dsh-auth-*`
cookie is injected.
* Upstream `Set-Cookie` headers are dropped. `Location` headers that are
absolute paths are rewritten under `/u/<user>/`, and any `token` query
parameter is stripped.
* Concurrency is capped per user: `MAX_IN_FLIGHT_PER_USER` proxied requests
(default 32) before the gateway answers `503` + `Retry-After` (HTTP) or
closes the WebSocket with `1011`. Messages received from DSH are limited to
`WS_MAX_MESSAGE_BYTES` (default 16 MiB); browser-to-DSH frames are bounded by
uvicorn's own WebSocket size limit.
### SSO (opt-in)
By default the gateway is the only authenticator: users sign up with a token or
the session cookie, and the gateway ignores all identity headers. Set
`AUTH_PROXY_HEADER` to enable SSO behind an authenticating reverse proxy such as
oauth2-proxy:
```bash
kubectl -n dsh-demo set env deploy/dsh-gateway AUTH_PROXY_HEADER=X-Forwarded-User
```
When set, the gateway treats a request carrying that header as authenticated as
that user, provided the value is a valid DNS-1123 label; the header is never
trusted when the variable is unset. The landing page then auto-provisions the
user (creates their claim idempotently, rotates the claim token when a new
session is needed and mints an opaque session cookie) and
redirects to `/u/<user>/`, and `POST /recreate` becomes available for a clean
restart. The base manifests leave it unset on purpose; the
[SSO exposure variant](../manifests/exposure/gateway-api-sso/) documents how to
enable it on the Deployment (`kubectl set env`). That variant also ships
`75-networkpolicies.yaml`, which restricts the gateway to oauth2-proxy and
oauth2-proxy's egress to the gateway, DNS and public HTTPS.
> **Trust boundary — this is the security-critical part.** A trusted identity
> header is only safe when **nothing else can reach the gateway**. The gateway
> Service must stay ClusterIP and be reachable exclusively through the
> authenticating proxy; if a client can reach `dsh-gateway:8080` directly (a
> second Ingress/Gateway/Route, a port-forward on a shared host, a broad
> NetworkPolicy), it can send `X-Forwarded-User` itself and impersonate any user.
> The variant's `75-networkpolicies.yaml` restricts pod traffic to the proxy; a
> custom exposure without an equivalent policy has no network enforcement.
> Treat `AUTH_PROXY_HEADER` as "the proxy already authenticated this request",
> not as an authentication mechanism of its own. Disable SSO by removing the
> variable: the header is then ignored and bearer-token/session auth applies
> again.
### Proxy failure semantics
@@ -92,17 +166,32 @@ token. A failed handshake returns `503` with `Retry-After: 10`.
| --- | --- |
| Unknown user / claim | `404` |
| Missing or invalid credentials | `401` |
| Cross-origin cookie mutation | `403` |
| Unknown user on `/login` | `401` (same as a bad token; no username enumeration) |
| Cross-origin cookie mutation / cross-origin form signup or login | `403` |
| Auth route over `AUTH_RATE_LIMIT` | `429` + `Retry-After`, before the body is read |
| Request body over `MAX_BODY_MB` | `413` |
| Sandbox not provisioned, wake timeout, handshake failure | `503` + `Retry-After: 10` |
| Client aborts mid-body, before the response starts | `499` (`CLIENT_CLOSED_REQUEST`). Nothing is forwarded to DSH and nothing can be delivered; it is not an error, so no traceback is logged. Covers the body-reading control routes (`POST /users`, `POST /login`) as well as the proxy. |
| Client aborts mid-stream (SSE, long response) | The response already started, so its upstream status stands (typically `200`); the stream simply ends. No traceback, and the in-flight slot is released as the relay unwinds. |
| Sandbox not provisioned, wake timeout, handshake failure | `503` + `Retry-After: 10`. The handshake error body is a fixed message: it never contains the launch token, the handshake URL or pod/credential details. |
| User over `MAX_IN_FLIGHT_PER_USER` | `503` + `Retry-After` (HTTP); the existing in-flight requests are untouched |
| Upstream connection/protocol failure | `502` |
| Kubernetes API error (API unreachable/throttled, RBAC `403`, `5xx`) | `503` + `Retry-After: 10`. The body is a fixed message; the Kubernetes `Status` payload (which can name cluster objects) is never proxied, and the API `status`/`reason` is logged once |
| WS auth failure (unknown user, bad token, cross-origin) | accept, then close `4401` |
| WS per-user concurrency cap | accept, then close `1011` |
| WS upstream/handshake failure | accept, then close `1011` |
| WS Kubernetes API error | accept, then close `1011` |
| WS message from DSH over `WS_MAX_MESSAGE_BYTES` | the upstream/relay closes the connection (websockets' message-too-big path) |
A close code can only be delivered after the WebSocket upgrade is accepted;
Starlette would turn an early close into an HTTP 403, which is why WS auth
failures are `101`, then close `4401`.
An aborted request is deliberately silent: uvicorn runs with `--no-access-log`,
and the `499` cannot reach a peer that has already left (uvicorn's `send()` is
a no-op once the socket is closed). Sandbox resolution, the wake and the DSH
handshake run before the body is read, so an aborted upload can cost a wake or
a handshake but never reaches DSH.
## Configuration
All values are read from the environment by `Settings.from_env()`; every one
@@ -117,20 +206,36 @@ can also be injected into `create_app` for tests.
| `DSH_PORT` | `3080` | Upstream DSH port. |
| `DSH_CONTAINER` | `dsh-web` | Container whose logs carry the launch token. |
| `DSH_LOG_TAIL_LINES` | `2000` | How many log lines to scan for the launch token. |
| `COOKIE_NAME` | `dsh_gw_session` | Browser session cookie name. |
| `COOKIE_SECURE` | `false` | Add `Secure` to the session cookie when served over TLS. |
| `COOKIE_MAX_AGE` | `604800` | Session cookie lifetime in seconds. |
| `COOKIE_NAME` | `__Host-dsh_gw_session` when secure, else `dsh_gw_session` | Browser session cookie name. Only an opaque session id is stored in it. |
| `COOKIE_SECURE` | `false` | Add `Secure` to the session cookie. Required unless `ALLOW_INSECURE_COOKIES=1` is set for local HTTP. |
| `ALLOW_INSECURE_COOKIES` | `0` | Explicit opt-out for plain-HTTP demos. Without it, `Settings.from_env()` refuses to start unless `COOKIE_SECURE=true`. |
| `COOKIE_MAX_AGE` | `604800` | Session lifetime in seconds (server-side expiry and cookie `Max-Age`). |
| `MAX_BODY_MB` | `32` | Maximum buffered request body (and `413` threshold). |
| `HTTP_TIMEOUT` | `300` | Upstream connect/read timeout in seconds. Proxied streams (SSE, long responses) keep the connect/write budget but have **no read timeout**, so an idle event stream is not cut every five minutes. |
| `SWEEP_INTERVAL` | `15` | Seconds between idle sweeps. |
| `SWEEP_INTERVAL` | `15` | Seconds between idle sweeps. Sweeps also prune expired sessions and rate buckets and orphaned per-user state. |
| `SIGNUP_INVITE_CODE` | unset | When set, `POST /users` requires the matching `invite` form field or `X-Signup-Invite` header (timing-safe compare); the landing page then shows an invite input. Unset leaves signup open. |
| `AUTH_RATE_LIMIT` | `30` | Attempts allowed per client per `AUTH_RATE_WINDOW` on `POST /users` and `POST /login`; `0` disables the limit. |
| `AUTH_RATE_WINDOW` | `60` | Rate-limit window in seconds. |
| `UPSTREAM_HEADER_ALLOWLIST` | unset | Comma-separated, case-insensitive allowlist of client headers to forward upstream. Unset keeps the denylist behavior; forwarding/identity headers are stripped in both modes. |
| `MAX_IN_FLIGHT_PER_USER` | `32` | Concurrent proxied requests per user before `503`/WS `1011`. `0` disables the cap. |
| `WS_MAX_MESSAGE_BYTES` | `16777216` | Largest WebSocket message (16 MiB) accepted from DSH. Browser-to-DSH frames are bounded by uvicorn's own limit. |
| `AUTH_PROXY_HEADER` | unset | Opt-in SSO: name of the trusted identity header set by an authenticating reverse proxy (e.g. `X-Forwarded-User`). Unset disables SSO and the header is never trusted. Enable only when the Service is reachable exclusively through that proxy (see [SSO](#sso-opt-in)). |
The shipped `manifests/50-gateway.yaml` sets `IDLE_TIMEOUT=360` and
`WAKE_TIMEOUT=240` for the demo; the e2e script overrides them to 60/120 so the
smoke test does not wait minutes for the sweeper. It also sets
`COOKIE_SECURE=false` together with the explicit `ALLOW_INSECURE_COOKIES=1`
acknowledgement because the demo is reached over plain HTTP (a `Secure` cookie
would never be sent). HTTPS deployments must set `COOKIE_SECURE=true`.
### Required RBAC
The gateway's ServiceAccount needs, in its namespace:
* `sandboxclaims` (`extensions.agents.x-k8s.io`): get, list, create, delete
* `sandboxes` (`agents.x-k8s.io`): get, patch
* `pods`: get
* `sandboxclaims` (`extensions.agents.x-k8s.io`): get, list, create, patch, delete
(patch rotates the token-hash annotation when a session is minted)
* `sandboxes` (`agents.x-k8s.io`): get, list, patch
* `pods`: get, list
* `pods/log`: get
## Running locally
@@ -143,16 +248,27 @@ python -m pip install -r requirements-dev.txt
python -m pytest -q
python -m ruff check .
# Uvicorn factory (loads in-cluster credentials only on the first claim call)
uvicorn gateway:create_app --factory --host 0.0.0.0 --port 8080
# Uvicorn factory (loads in-cluster credentials only on the first claim call).
# from_env() insists on an explicit acknowledgement for plain HTTP; behind TLS
# set COOKIE_SECURE=true instead.
ALLOW_INSECURE_COOKIES=1 uvicorn gateway:create_app --factory --host 0.0.0.0 --port 8080
```
Container image:
Container image, from the repository root:
```bash
docker build -t dsh-gateway examples/dsh-agents-as-a-service/gateway
make gateway # builds dsh-aaas-gateway:dev from this directory
```
The equivalent direct build is
`docker build -t dsh-aaas-gateway:dev examples/dsh-agents-as-a-service/gateway`.
For kind, retag it to the image reference in
`examples/dsh-agents-as-a-service/manifests/50-gateway.yaml` and `kind load` it.
Published images are nested below the main image as
`<GHCR_IMAGE>/dsh-agents-as-a-service/gateway:<identity>`, where the identity
comes from [`versions.env`](../../../versions.env) — see
[`docs/versioning.md`](../../../docs/versioning.md).
The image runs as UID `10000`, sets `PYTHONDONTWRITEBYTECODE=1` and
`HOME=/tmp` (read-only-rootfs compatible), listens on `8080`, and disables
uvicorn's access log so bearer tokens in URLs are not logged.
@@ -163,18 +279,27 @@ uvicorn's access log so bearer tokens in URLs are not logged.
Log rotation, a very chatty container, or DSH changing its startup message
breaks the handshake. Prefer a first-class token/credential API when DSH
offers one.
* **Single replica, in-memory state.** The handshake cookie cache, idle
activity clocks and in-flight counters live in one process. Running more
than one replica would need shared state (or sticky routing), otherwise each
replica handshakes independently and the sweeper may race another replica.
* **Single replica, in-memory state.** The handshake cookie cache, the opaque
session table, idle activity clocks, in-flight counters and rate-limit
buckets live in one process. Running more than one replica would need shared
state (or sticky routing), otherwise each replica handshakes independently,
the sweeper may race another replica, and a session/rate budget is only seen
by the replica that created it.
* **Sessions die with the process.** An opaque session id is valid only in the
process that minted it and expires after `COOKIE_MAX_AGE`; a restart logs
every browser out (tokens are unaffected).
* **In-memory token cache has no TTL.** It is keyed by pod UID, so pod
restarts invalidate it naturally; otherwise entries survive until the pod or
claim is deleted.
* **No TLS termination.** Run behind an ingress/load balancer; set
`COOKIE_SECURE=true` there.
* **No TLS termination.** Run behind an ingress/load balancer and set
`COOKIE_SECURE=true`. `Settings.from_env()` refuses to start without it
unless `ALLOW_INSECURE_COOKIES=1` explicitly acknowledges plain-HTTP
sessions.
* **Origin-based CSRF protection only.** Cookie-auth mutations require a
same-origin `Origin`/`Referer`; there is no per-form CSRF token, rate
limiting or login throttling.
same-origin `Origin`/`Referer`, and form signup/login reject a contradicting
origin when the browser sends one; there is no per-form CSRF token. Auth
routes have per-client rate limiting and login throttling, but the budget is
in-process (per replica) and keyed by the rightmost `X-Forwarded-For` value.
* **HTTP request bodies are buffered** up to `MAX_BODY_MB`; uploads are not
streamed. Responses (including SSE) are streamed.
* **Upstream cookies are not forwarded.** DSH cannot rotate its own
@@ -182,3 +307,9 @@ uvicorn's access log so bearer tokens in URLs are not logged.
re-runs the handshake once and replays the request.
* **Only DSH on `:3080` is proxied.** There is no dashboard/API-port split and
no platform API key injection.
* **SSO trusts a header.** With `AUTH_PROXY_HEADER` set, any client that can
reach the gateway Service directly can spoof the identity header. The gateway
cannot enforce the trust boundary itself: keep the Service ClusterIP and
reachable only through the authenticating proxy. The SSO exposure variant
ships a NetworkPolicy that restricts pod traffic to the proxy; without it
there is no mTLS or signed identity assertion.

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,44 @@
# DSH "Agents as a Service" — prerequisites: namespace + shared provider keys.
#
# The dsh-provider-keys Secret is deliberately NOT committed with a usable
# value. DEEPSEEK_API_KEY reaches every tenant sandbox (injected via envFrom
# on the SandboxTemplate, stamped before any user exists — a warm-pool
# constraint), so a real key committed here would be shared by every install
# of this example. Generate a fresh value per install (README step 1;
# scripts/e2e-kind.sh uses a throwaway). Use an upsert: this manifest already
# creates the empty Secret, so a plain `kubectl create` would fail:
#
# kubectl -n dsh-demo create secret generic dsh-provider-keys \
# --from-literal=DEEPSEEK_API_KEY="sk-..." \
# --dry-run=client -o yaml | kubectl apply -f -
#
# Optional: additional providers (OpenRouter, Gemini, NVIDIA, OpenAI, ...) go
# in a second, optional Secret. The SandboxTemplate references it with
# `optional: true`, so sandboxes start even when it does not exist, and every
# key in it becomes an environment variable. Create it BEFORE the warm pool,
# because pod env is stamped at pod creation:
#
# kubectl -n dsh-demo create secret generic dsh-provider-keys-extra \
# --from-literal=OPENROUTER_API_KEY="sk-or-..." \
# --from-literal=GEMINI_API_KEY="..." \
# --from-literal=NVIDIA_API_KEY="nvapi-..." \
# --dry-run=client -o yaml | kubectl apply -f -
#
# Supported env names, precedence, shared-key caveats and the pod-refresh
# procedure are in docs/optional-hardening.md.
#
# All the infrastructure in this example works without a real key; only
# actual model calls need one.
apiVersion: v1
kind: Namespace
metadata:
name: dsh-demo
---
apiVersion: v1
kind: Secret
metadata:
name: dsh-provider-keys
namespace: dsh-demo
type: Opaque
# stringData:
# DEEPSEEK_API_KEY: "sk-your-key-here"

View File

@@ -0,0 +1,219 @@
# The reusable DSH Web blueprint. Warm pools stamp pod env BEFORE any user
# exists, so per-user secrets cannot live in pod env; the shared provider keys
# are injected via envFrom, and claims that would bypass the warm pool are
# rejected outright (see 40-claim-rejected.yaml).
#
# A stripped-down single-tenant variant of this template (no warm pool, no
# policy enforcement) lives in examples/minimal-single-tenant/.
apiVersion: extensions.agents.x-k8s.io/v1beta1
kind: SandboxTemplate
metadata:
name: dsh-web
namespace: dsh-demo
spec:
# Claims must not inject env or volumeClaimTemplates: with Disallowed, such
# claims are REJECTED instead of silently taking the cold (non-warm-pool)
# provisioning path. Demonstrated in step 6.
envVarsInjectionPolicy: Disallowed
volumeClaimTemplatesPolicy: Disallowed
# Per-sandbox headless Service -> stable DNS (sandbox.status.serviceFQDN).
# The gateway routes each user to their DSH Web instance by this name.
service: true
# The example leaves NetworkPolicy to an optional, opt-in manifest
# (manifests/optional/network-policy/): the base walkthrough should not
# require a policy-capable CNI, and the controller's Managed secure default
# only allows the sandbox-router, which this example does not deploy. If you
# switch to Managed, define spec.networkPolicy explicitly or the gateway will
# be blocked. See docs/optional-hardening.md.
networkPolicyManagement: Unmanaged
# OPTIONAL: replace a Failed sandbox Pod instead of parking the sandbox.
# The default, Retain, leaves the Failed Pod in place and surfaces
# Finished=True on the Sandbox until a human intervenes. Recreate deletes the
# Failed Pod and starts a replacement with the same Sandbox identity and
# Sandbox-owned PVCs, backing off (5s doubling to 5m) on repeated failures.
# Requires agent-sandbox v1.0.8+; existing Sandboxes keep their own stamped
# policy, so patch them separately. See docs/optional-hardening.md.
# failedPodPolicy: Recreate
podTemplate:
spec:
# DSH Web is multi-arch (amd64 + arm64); no arch nodeSelector is needed.
#
# OPTIONAL: run the sandbox under gVisor. Requires a RuntimeClass named
# "gvisor" and nodes that can run it; gVisor breaks `kubectl port-forward`
# into the sandbox (use the gateway). Existing Sandboxes snapshot their
# pod template, so enabling it later requires patching/recreating them.
# See docs/optional-hardening.md.
# runtimeClassName: gvisor
#
# OPTIONAL: pin sandboxes to a dedicated node pool (for example a gVisor
# pool). Replace this placeholder with a label your nodes actually carry;
# add a matching toleration if the pool is tainted. Active
# cluster-specific scheduling belongs in a local patch, not here.
# nodeSelector:
# dsh.example.com/node-pool: dsh
restartPolicy: "OnFailure"
securityContext:
runAsUser: 1000 # the 'node' user in the DSH image
runAsGroup: 1000
fsGroup: 1000
runAsNonRoot: true
# DSH's credentials plugin refuses to start when .credentials.yaml has
# any group/other permission bit (it creates the file 0600). The
# default kubelet fsGroup policy ("Always") recursively ORs 0660 into
# every file each time the pod starts, so a resumed sandbox comes back
# with 0660 and crash-loops before the gateway can proxy to it.
# OnRootMismatch skips that walk once the volume root already carries
# the fsGroup ownership/mode applied on the first mount.
# Keep in sync with the minimal variant:
# examples/minimal-single-tenant/manifests/10-sandboxtemplate.yaml.
fsGroupChangePolicy: OnRootMismatch
containers:
- name: dsh-web
image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.5-r6
imagePullPolicy: IfNotPresent
ports:
- name: web
containerPort: 3080
envFrom:
- secretRef:
name: dsh-provider-keys
# Optional extra provider keys (OpenRouter, Gemini, NVIDIA, ...).
# The Secret may be absent; every key in it becomes an env var.
# Create it before the warm pool, because pod env is stamped at
# pod creation. See docs/optional-hardening.md.
- secretRef:
name: dsh-provider-keys-extra
optional: true
env:
# Point the OS home at the persisted agent workspace. DSH's
# directory picker (the folder explorer used to choose a workspace)
# has no start-path setting: when it opens it lists $HOME, and
# os.homedir() follows this variable. The image default is
# /home/node, which is not on a volume, so the picker would open on
# a throwaway directory; /workspace is the PVC-backed WORKDIR. `~`
# in agent commands resolves here too. Trade-off: tools that keep
# caches/config under $HOME (npm/pnpm, uv, git, gh, and DSH's global
# ~/.agents instructions and skills) now write dotfiles onto the PVC
# inside the agent workspace, so they are visible to the agent and
# in the file explorer. DSH_HOME is explicit (/home/node/.dsh), so
# sessions, config and credentials are unaffected. Existing
# sandboxes stamp pod env at creation and keep the old HOME until
# recreated. Keep in sync with the minimal variant:
# examples/minimal-single-tenant/manifests/10-sandboxtemplate.yaml.
- name: HOME
value: /workspace
# Empty => DSH derives links from the request Host, which the
# gateway forwards (plus X-Forwarded-*). Point this at the
# gateway base URL if you want hard-coded absolute links instead.
- name: DSH_PUBLIC_URL
value: ""
# DSH Host/Origin-fences /api. The gateway forwards the browser's
# Host, so DSH must trust the public gateway host. Empty is right
# for port-forward/localhost; for either exposure variant set this
# to the public host (e.g. dsh.example.com for
# manifests/exposure/ingress/60-ingress.yaml, or
# dsh-demo.example.com for the gateway-api-sso/ HTTPRoute).
- name: DSH_TRUSTED_HOSTS
value: ""
# DSH confines every bash command with its own file sandbox
# (bubblewrap, else Landlock). Neither backend is usable inside this
# pod: bubblewrap ships in the image but is unsetuid, so it needs an
# unprivileged user namespace that the pod's RuntimeDefault seccomp
# and dropped capabilities do not grant, and the runtime/kernel
# surface here exposes no Landlock runner. The executor therefore
# fails closed with SANDBOX_UNAVAILABLE and every command fails. The
# pod is the isolation boundary in this example (one pod
# per user, non-root, capabilities dropped, seccomp RuntimeDefault,
# a private PVC), so trade DSH's inner file policy for a working
# agent. Set this to workspace-write (or read-only) on hosts where a
# backend is available; see the README troubleshooting row.
# SECURITY: the pod is the isolation boundary in this example.
# danger-full-access is set only because DSH's own command sandbox
# has no usable backend in this pod (see above), which means an
# agent command can read its own DSH_HOME (including
# .credentials.yaml) and write anywhere in the container. What
# still contains it is the pod itself: one pod per user, non-root,
# all capabilities dropped, RuntimeDefault seccomp, its own PVC and
# Service. That is NOT a hardened multi-tenant boundary on a shared
# kernel. For production/multi-tenant use, additionally run the
# sandbox under gVisor (`runtimeClassName: gvisor` above) so a
# container escape is trapped in a userspace kernel, and apply the
# sandbox NetworkPolicy (manifests/optional/network-policy/) so
# tenants cannot reach each other or cluster-internal endpoints.
# The base manifests without those two controls are a demo. Treat
# danger-full-access as a fallback, not a target: on hosts where
# DSH's command-sandbox backend works, prefer workspace-write (or
# read-only), and recreate the pod because the value is stamped at
# creation.
- name: DSH_PERMISSION_MODE
value: danger-full-access
# Probes target /favicon.svg: DSH gates / behind its launch-token
# cookie and answers 401 to an unauthenticated GET, which would
# otherwise fail readiness and then get the pod killed by liveness.
# /favicon.svg is served without auth and is the cheapest liveness
# signal that proves the HTTP server is actually serving requests.
#
# Gate liveness/readiness behind a startup probe: a cold start on a
# small node takes tens of seconds to load plugins and start serving
# (a resumed sandbox starts cold too). Without it the kubelet can kill
# DSH mid-startup, and a SIGTERM'd DSH exits 0, which the sandbox
# controller records as finished. 24 x 5s = 120s, matching the
# gateway's WAKE_TIMEOUT.
startupProbe:
httpGet:
path: /favicon.svg
port: 3080
periodSeconds: 5
failureThreshold: 24
readinessProbe:
httpGet:
path: /favicon.svg
port: 3080
initialDelaySeconds: 5
periodSeconds: 5
failureThreshold: 30
livenessProbe:
httpGet:
path: /favicon.svg
port: 3080
initialDelaySeconds: 15
periodSeconds: 15
failureThreshold: 3
# The container only needs to serve HTTP; drop every capability and
# disallow privilege escalation. DSH writes only to its PVC-backed
# directories (DSH_HOME and the agent workspace).
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
seccompProfile:
type: RuntimeDefault
resources:
requests: {cpu: 250m, memory: 512Mi}
limits: {cpu: "1", memory: 1Gi}
volumeMounts:
- name: dsh-home
mountPath: /home/node/.dsh # DSH_HOME: sessions, config, state
# The agent workspace (the image's WORKDIR) is PVC-backed too, but
# under its own subPath so DSH state - including .credentials.yaml -
# never shows up as workspace data. Kubelet creates workspace/ on
# first mount with the volume's fsGroup permissions (0775, setgid
# inherited from the volume root), so UID 1000 can write; no init
# container is needed. If the kubelet is killed mid-creation the
# directory stays 0755: see Troubleshooting in the README.
- name: dsh-home
mountPath: /workspace
subPath: workspace
volumeClaimTemplates:
- metadata:
name: dsh-home
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 5Gi
# storageClassName omitted => cluster default (kind & GKE both work)

View File

@@ -0,0 +1,253 @@
# The gateway: control+data plane that turns the resources into a usable
# browser-facing service. It is a Starlette ASGI app (see gateway/) listening
# on :8080 and exposing:
# GET /healthz unauthenticated health probe
# GET / landing page (signup / token login)
# POST /users, POST /login, ... session cookie auth
# /u/<user>/... HTTP + WebSocket proxy to the user's DSH
#
# Image: the release image published for this example, pinned by tag.
# imagePullPolicy: IfNotPresent lets a local kind build loaded under the same
# tag be used without a registry round-trip (scripts/e2e-kind.sh builds
# dsh-aaas-gateway:e2e and overrides this image).
#
# SSO IS OPT-IN: this manifest runs the gateway with its own bearer/session
# auth only, so AUTH_PROXY_HEADER is intentionally NOT set (see the env
# block). To put GitHub OAuth in front of it, use
# manifests/exposure/gateway-api-sso/.
#
# This Service is ClusterIP and is the only intended entry point: expose it
# through exactly ONE of the documented variants in manifests/exposure/
# (ingress/ or gateway-api-sso/), never both and never directly.
apiVersion: v1
kind: ServiceAccount
metadata:
name: dsh-gateway
namespace: dsh-demo
---
# Least privilege:
# - claims are created/deleted per user (signup/delete) and patched only to
# rotate the token-hash annotation when a session is minted;
# - sandboxes are read, and patched only to flip operatingMode (suspend/resume);
# - pods are read, and pods/log is read to capture DSH's launch token from
# the startup log line (`dsh web: ...?token=...`).
# configmaps are not needed: all state is in memory / claim annotations.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: dsh-gateway
namespace: dsh-demo
rules:
- apiGroups: ["extensions.agents.x-k8s.io"]
resources: ["sandboxclaims"]
verbs: ["get", "list", "create", "patch", "delete"]
- apiGroups: ["agents.x-k8s.io"]
resources: ["sandboxes"]
verbs: ["get", "list", "patch"]
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: dsh-gateway
namespace: dsh-demo
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: dsh-gateway
subjects:
- kind: ServiceAccount
name: dsh-gateway
namespace: dsh-demo
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: dsh-gateway
namespace: dsh-demo
spec:
# Single replica BY DESIGN: the idle clock, in-flight counters and the
# per-(user, pod, Host) DSH cookie cache are in-process memory. Running two
# replicas would fragment that state (double suspend/resume races, repeated
# log-token handshakes). Scale only after moving state to a shared store.
replicas: 1
selector:
matchLabels:
app: dsh-gateway
template:
metadata:
labels:
app: dsh-gateway
spec:
serviceAccountName: dsh-gateway
# OPTIONAL: run the gateway under gVisor. Requires a RuntimeClass named
# "gvisor" and nodes that can run it. See docs/optional-hardening.md.
# runtimeClassName: gvisor
#
# OPTIONAL: pin the gateway to a dedicated node pool. Replace the
# placeholder with a label your nodes carry; add a toleration for a
# tainted pool. Keep active cluster-specific scheduling in a local patch.
# nodeSelector:
# dsh.example.com/node-pool: dsh
securityContext:
runAsNonRoot: true
runAsUser: 10000 # matches USER in gateway/Dockerfile
runAsGroup: 10000
seccompProfile:
type: RuntimeDefault
containers:
- name: gateway
image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway:dsh-v0.2.1-rc.5-r6
imagePullPolicy: IfNotPresent
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: ["ALL"]
ports:
- name: http
containerPort: 8080
env:
- name: NAMESPACE
value: dsh-demo
- name: POOL
value: dsh-pool
# Seconds a sandbox may sit idle before the sweeper suspends it
# (pod deleted, PVC kept), and the max time a proxy request waits
# for a suspended sandbox to become Ready again. Only claims this
# gateway created are swept: a claim applied by hand from the
# resource-layer manifests carries no aaas.example.com/managed-by
# annotation and is never suspended.
#
# These are the demo values (360/240). scripts/e2e-kind.sh
# overrides both to 60/120 with `kubectl set env` so the smoke
# test does not wait six minutes for the sweeper.
- name: IDLE_TIMEOUT
value: "360"
- name: WAKE_TIMEOUT
value: "240"
# Container name inside the per-user sandbox pods whose logs carry
# the DSH launch token.
- name: DSH_CONTAINER
value: dsh-web
# Browser session cookie: HttpOnly + SameSite=Lax, holds only an
# opaque, server-side session id (sid) - never the user's DSH
# token. Sessions can be revoked server-side: POST /logout, user
# delete, POST /recreate and expiry (COOKIE_MAX_AGE, default
# 7 days) all invalidate them. The session table is in process
# memory, so restarting this Deployment logs every browser out.
# With COOKIE_SECURE=true the cookie is named
# __Host-dsh_gw_session; COOKIE_NAME overrides the name.
#
# "false" keeps the plain-HTTP demo usable through port-forward: a
# Secure cookie is never sent over HTTP, so login would silently
# fail. The gateway REFUSES TO START with COOKIE_SECURE=false
# unless ALLOW_INSECURE_COOKIES=1 acknowledges that the session
# travels in clear text, so the base demo sets both. HTTPS
# deployments (the ingress/ variant with TLS, or gateway-api-sso/,
# which terminates TLS at the Gateway) MUST set this to "true".
- name: COOKIE_SECURE
value: "false"
# Explicit acknowledgement that this base demo serves plain HTTP.
# Required when COOKIE_SECURE=false (the gateway exits at startup
# otherwise); ignored once COOKIE_SECURE=true, so HTTPS installs
# should drop it. Never set this on an internet-facing deployment.
- name: ALLOW_INSECURE_COOKIES
value: "1"
# SSO IS OPT-IN AND DISABLED HERE. With AUTH_PROXY_HEADER unset the
# gateway trusts only its own bearer/session auth and ignores every
# identity header. Enable it ONLY behind
# manifests/exposure/gateway-api-sso/, where oauth2-proxy is the
# sole route to this Service: set it to X-Forwarded-User, keep the
# Service ClusterIP and reachable only through the proxy, and
# terminate TLS in front of the gateway. Otherwise anyone who can
# reach :8080 can impersonate any user by sending that header.
# - name: AUTH_PROXY_HEADER
# value: X-Forwarded-User
# Brute-force / abuse throttle for the unauthenticated auth
# endpoints (/users and /login): at most N attempts per client per
# AUTH_RATE_WINDOW seconds (default 60). A client is the rightmost
# X-Forwarded-For entry (the one added by the trusted proxy) when
# present, otherwise the TCP peer. Over the limit the gateway
# answers 429 + Retry-After BEFORE reading the body or touching the
# cluster; 0 disables the limiter. This is a per-process bucket, so
# it is not a substitute for an ingress rate limit.
- name: AUTH_RATE_LIMIT
value: "30"
# Window in seconds for AUTH_RATE_LIMIT (default 60).
# - name: AUTH_RATE_WINDOW
# value: "60"
# Optional signup gate. When set, POST /users requires the invite
# code - form field "invite" or header X-Signup-Invite - compared in
# constant time. A missing/wrong code is rejected with 403 before
# the user is looked up, so it also stops unauthenticated claim
# creation. The landing page shows the invite input only when this
# is set. Keep the value in a Secret, create it before rollout and
# restart the gateway to pick up a rotation:
# kubectl -n dsh-demo create secret generic dsh-signup-invite \
# --from-literal=code="$(openssl rand -hex 16)"
# - name: SIGNUP_INVITE_CODE
# valueFrom:
# secretKeyRef:
# name: dsh-signup-invite
# key: code
# Reject proxied request bodies larger than this (413).
- name: MAX_BODY_MB
value: "32"
# Per-user concurrency guard (default 32): at most this many
# proxied HTTP requests and WebSocket connections per user.
# Excess HTTP requests get 503 + Retry-After and excess WebSocket
# upgrades close with 1011, so one tenant cannot exhaust the
# single-replica gateway for everyone else.
# - name: MAX_IN_FLIGHT_PER_USER
# value: "32"
# Largest WebSocket message received from DSH, in bytes (default
# 16 MiB); oversized frames close the connection. Browser-to-DSH
# frames are bounded by uvicorn's own limit. HTTP/SSE responses are
# streamed without this cap.
# - name: WS_MAX_MESSAGE_BYTES
# value: "16777216"
# /healthz is unauthenticated and returns 200 without touching the
# cluster, so it is safe for both readiness and liveness.
readinessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 5
periodSeconds: 5
failureThreshold: 3
livenessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 10
periodSeconds: 10
failureThreshold: 3
resources:
requests: {cpu: 50m, memory: 128Mi}
limits: {cpu: 500m, memory: 256Mi}
volumeMounts:
# readOnlyRootFilesystem leaves no writable HOME/tmp; the image sets
# HOME=/tmp, so back it with scratch space. No state lives here.
- name: tmp
mountPath: /tmp
volumes:
- name: tmp
emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
name: dsh-gateway
namespace: dsh-demo
spec:
selector:
app: dsh-gateway
ports:
- port: 8080
targetPort: 8080

View File

@@ -0,0 +1,185 @@
# Exposing the gateway
The base manifests deploy the gateway as a ClusterIP Service
(`dsh-gateway:8080`) and expose nothing outside the cluster. WebSocket
upgrades on `/u/<user>/**` are part of the contract, so the entry point must
forward `Upgrade`/`Connection` headers (Traefik does by default).
There are two **mutually exclusive** exposure variants. Pick exactly one:
| Variant | Entry point | Authentication | TLS |
| --- | --- | --- | --- |
| [`ingress/60-ingress.yaml`](ingress/60-ingress.yaml) | Traefik `Ingress` | the gateway's own signup/session cookie | optional, recommended |
| [`gateway-api-sso/`](gateway-api-sso/72-oauth2-proxy.yaml) | Gateway API `Gateway` + `HTTPRoute` | GitHub OAuth via oauth2-proxy, then the gateway cookie | required (the Gateway listener terminates it) |
Applying both gives the gateway two entry points, and the SSO variant is only
safe when the gateway is reachable **exclusively** through oauth2-proxy. Do not
mix them.
> **Production requirement.** Each variant's NetworkPolicy is part of the
> variant, and applying the variant without it is demo-only. The sandbox
> NetworkPolicy and the gateway egress policy (the API server is cluster-specific)
> are separate and equally required; see the
> [production checklist](../../docs/optional-hardening.md#6-production-checklist)
> and the
> [gateway egress recipe](../../docs/optional-hardening.md#gateway-egress-required-for-production).
> For SSO, run [`scripts/preflight-sso.sh`](../../scripts/preflight-sso.sh)
> after applying the variant and before sending real traffic.
Each variant also ships a NetworkPolicy, applied by the same directory apply:
- `ingress/61-networkpolicy.yaml` allows only Traefik pods to reach the gateway
on `8080`.
- `gateway-api-sso/75-networkpolicies.yaml` allows only oauth2-proxy to reach
the gateway, only the Gateway implementation (Traefik) to reach oauth2-proxy,
and limits oauth2-proxy egress to DNS, public HTTPS (GitHub) and the gateway.
The policies assume Traefik's `app.kubernetes.io/name: traefik` pod label.
Adjust the selector for another implementation or a `hostNetwork` install; see
[optional hardening](../../docs/optional-hardening.md) for the probe caveat and
verification commands. Keep the policies: an exposure without its NetworkPolicy
is demo-only.
## Applying a variant
Run these commands from the example root
(`examples/dsh-agents-as-a-service/`), the same working directory the example
README uses.
`kubectl apply -f manifests/` is **not recursive**: it applies only the
top-level numbered files and does not descend into `exposure/`. Apply the base
resources first, then the chosen variant explicitly:
```bash
# Base resources: namespace, template, warm pool, claims, gateway.
kubectl apply -f manifests/
# Plain Ingress exposure:
kubectl apply -f manifests/exposure/ingress/
# ... or SSO exposure (create the two Secrets and replace the example.com
# placeholders first — see below):
kubectl apply -f manifests/exposure/gateway-api-sso/
```
**Never use `kubectl apply -R`** (recursive): it would apply both variants at
once and hand out a second, unauthenticated path to the same gateway. Apply the
variant directory on purpose, or the individual files by name.
`kubectl` only reads `.yaml`, `.yml` and `.json` files from a directory, so
the `*.yaml.example` secret templates are skipped automatically; create the real
Secrets imperatively (commands below) before applying the directory.
## `ingress/`: plain Ingress, no SSO
`60-ingress.yaml` routes a host through the cluster's Traefik
(`ingressClassName: traefik`; k3s ships one) to the gateway Service.
1. Replace the `dsh.example.com` host and the `dsh-example-tls` Secret name
with real values, or delete the `tls:` block to serve plain HTTP.
2. Set `DSH_TRUSTED_HOSTS` in the SandboxTemplate to the same public host:
the gateway forwards the browser's `Host`, and DSH rejects unknown hosts on
`/api` (see `../10-sandboxtemplate.yaml`).
3. Set `COOKIE_SECURE=true` and remove the plain-HTTP acknowledgement once the
Ingress serves HTTPS:
`kubectl -n dsh-demo set env deploy/dsh-gateway COOKIE_SECURE=true ALLOW_INSECURE_COOKIES-`.
The base manifest keeps `false` + `ALLOW_INSECURE_COOKIES=1` so the
plain-HTTP port-forward demo works; a `Secure` cookie is never sent over HTTP,
which silently breaks login, while the acknowledgement must not survive into
production.
4. Do not attach buffering or compression middlewares to this router: they
break WebSocket upgrades and streaming.
5. `61-networkpolicy.yaml` is part of this variant: it allows only Traefik pods
to reach the gateway on `8080`. Adjust its selector for another ingress
controller or `hostNetwork`; if kubelet probes fail, see the hardening doc.
Verify: `curl -sI https://<host>/healthz` returns `200`.
## `gateway-api-sso/`: TLS + GitHub OAuth
`70-gateway.yaml` defines the TLS `Gateway`, `71-httproute.yaml` sends every
request to oauth2-proxy, and `72-oauth2-proxy.yaml` authenticates the browser
against GitHub and proxies to the gateway with `X-Forwarded-User` set.
`75-networkpolicies.yaml` makes this the only network path to the gateway.
Prerequisites:
- Gateway API CRDs and a Gateway implementation installed;
`gatewayClassName: traefik` matches the Traefik Gateway provider. The CRD
version is pinned in `versions.env` (`GATEWAY_API_VERSION`).
- The OAuth App credentials and a cookie secret, from
`73-oauth2-proxy-secret.yaml.example`.
- A TLS certificate/key for the public host, from
`74-tls-secret.yaml.example`.
- oauth2-proxy is pinned in the manifest to
`quay.io/oauth2-proxy/oauth2-proxy:v7.15.5` (`versions.env`:
`OAUTH2_PROXY_IMAGE`).
Steps:
```bash
# 1. Real Secrets (the committed files are *.yaml.example templates).
kubectl -n dsh-demo create secret generic oauth2-proxy \
--from-literal=client-id='<GitHub OAuth App client ID>' \
--from-literal=client-secret='<GitHub OAuth App client secret>' \
--from-literal=cookie-secret="$(python3 -c 'import secrets; print(secrets.token_urlsafe(32))')"
kubectl -n dsh-demo create secret tls dsh-demo-tls --cert=tls.crt --key=tls.key
# 2. Replace dsh-demo.example.com (70/71/72) and your-github-org (72) with
# the real public host and allowed GitHub organization.
# 3. Enable SSO on the gateway and trust the public host in the template.
kubectl -n dsh-demo set env deploy/dsh-gateway \
AUTH_PROXY_HEADER=X-Forwarded-User COOKIE_SECURE=true ALLOW_INSECURE_COOKIES-
# then set DSH_TRUSTED_HOSTS=dsh-demo.example.com in the SandboxTemplate
# (see ../10-sandboxtemplate.yaml) and re-apply it.
# 4. Apply the variant.
kubectl apply -f manifests/exposure/gateway-api-sso/
# 5. Preflight: fails loudly on missing Secrets, a bad oauth2-proxy cookie
# secret, missing gateway SSO env, a dangling TLS reference, leftover
# placeholders, or a second path to the gateway.
./scripts/preflight-sso.sh
```
The preflight is read-only and exits non-zero on any failure, naming the exact
fix. Run it again after rotating a Secret or changing the public host; it also
verifies that the oauth2-proxy cookie secret decodes to 16, 24 or 32 bytes.
### SSO trust boundary
`AUTH_PROXY_HEADER` is opt-in and deliberately unset in the base gateway
manifest: the gateway ignores identity headers until it is set. Enabling SSO
takes all of these:
1. `AUTH_PROXY_HEADER=X-Forwarded-User` on `deploy/dsh-gateway` (step 3
above). Without it, oauth2-proxy's login has no effect on the gateway and
users are still asked for the gateway's own token.
2. The gateway Service must be reachable **only through oauth2-proxy**: keep it
ClusterIP, route all browser traffic through the `HTTPRoute` →
oauth2-proxy → `dsh-gateway` chain, and never apply `ingress/` (or any
other path to the Service) while SSO is enabled. A client that can reach
`dsh-gateway:8080` directly can set `X-Forwarded-User` itself and
impersonate any user.
3. Keep `75-networkpolicies.yaml` with the variant. It enforces the boundary in
the network layer (gateway ← oauth2-proxy ← Traefik, and limited oauth2-proxy
egress). This is required, not optional: an exposure without the policy is
demo-only, and a custom exposure without an equivalent policy has no network
enforcement.
4. Apply the gateway egress policy too: the gateway talks to the Kubernetes API
and to sandbox pods, and `75-networkpolicies.yaml` intentionally leaves that
direction unrestricted because the API-server endpoint is cluster-specific.
See
[gateway egress](../../docs/optional-hardening.md#gateway-egress-required-for-production).
Also terminate TLS in front of the gateway (the Gateway listener does) and
leave `COOKIE_SECURE=true` so the session cookie never travels in clear text.
## Switching variants
Delete the variant you had before applying the other one, e.g.
`kubectl delete -f manifests/exposure/ingress/`, and revert
`AUTH_PROXY_HEADER`/`COOKIE_SECURE` on the gateway. Verify with
`kubectl -n dsh-demo get ingress,gateway,httproute` that exactly one path to
the gateway remains.

View File

@@ -0,0 +1,25 @@
# Browser entry point (TLS) for the gateway-api-sso variant: terminates TLS
# for dsh-demo.example.com and routes through 71-httproute.yaml to
# oauth2-proxy, which authenticates the browser before the gateway.
# Mutually exclusive with ../../ingress/ (see ../../README.md). The
# "dsh-demo-tls" Secret comes from 74-tls-secret.yaml.example.
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: dsh-gateway
namespace: dsh-demo
labels:
app.kubernetes.io/part-of: dsh-demo
spec:
gatewayClassName: traefik
listeners:
- name: https
port: 8443
protocol: HTTPS
tls:
mode: Terminate
certificateRefs:
- name: dsh-demo-tls
allowedRoutes:
namespaces:
from: Same

View File

@@ -0,0 +1,23 @@
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: dsh-gateway
namespace: dsh-demo
labels:
app.kubernetes.io/part-of: dsh-demo
spec:
parentRefs:
- name: dsh-gateway
namespace: dsh-demo
sectionName: https
hostnames:
# TODO: replace with the actual public FQDN before applying.
- dsh-demo.example.com
rules:
- backendRefs:
# Route through oauth2-proxy (GitHub OAuth) first; it authenticates the
# browser and forwards to dsh-gateway:8080 with X-Forwarded-User set.
# The gateway only trusts that header when AUTH_PROXY_HEADER is set on
# its Deployment — see 72-oauth2-proxy.yaml and ../README.md.
- name: oauth2-proxy
port: 4180

View File

@@ -0,0 +1,146 @@
# oauth2-proxy — GitHub OAuth login in front of the dsh-gateway.
#
# Runs as a reverse proxy: it authenticates the browser against GitHub
# (restricted to a configured organization), then forwards the request to
# the dsh-gateway Service with the authenticated identity in X-Forwarded-User
# (plus X-Auth-Request-* via --set-xauthrequest). The gateway trusts that
# header only when AUTH_PROXY_HEADER is set on its Deployment.
#
# ENABLING SSO TAKES BOTH SIDES (see ../../../exposure/README.md):
# 1. Set AUTH_PROXY_HEADER=X-Forwarded-User on deploy/dsh-gateway, otherwise
# the gateway ignores the header and the login is cosmetic:
# kubectl -n dsh-demo set env deploy/dsh-gateway \
# AUTH_PROXY_HEADER=X-Forwarded-User COOKIE_SECURE=true
# 2. Make dsh-gateway reachable ONLY through this proxy: keep the Service
# ClusterIP, route browser traffic through 71-httproute.yaml, and do not
# apply ../ingress/ (or anything else that reaches the gateway). A
# client that can reach dsh-gateway:8080 can spoof X-Forwarded-User and
# impersonate any user.
#
# 75-networkpolicies.yaml enforces that boundary at the network layer: the
# gateway accepts traffic only from oauth2-proxy pods, oauth2-proxy only from
# the Gateway implementation, and oauth2-proxy egress is limited to DNS,
# public HTTPS and the gateway. Apply it with this variant.
#
# TLS terminates at the Gateway listener (70-gateway.yaml); keep
# --cookie-secure=true and COOKIE_SECURE=true on the gateway.
#
# The image is digest-pinned: quay.io/oauth2-proxy/oauth2-proxy:v7.15.5@sha256:...
# (versions.env: OAUTH2_PROXY_IMAGE). The digest is the immutable multi-arch
# index, so the tag can never be moved under us. Credentials come from the
# "oauth2-proxy" Secret (see 73-oauth2-proxy-secret.yaml.example); this
# manifest never carries secrets.
apiVersion: apps/v1
kind: Deployment
metadata:
name: oauth2-proxy
namespace: dsh-demo
labels:
app: oauth2-proxy
app.kubernetes.io/part-of: dsh-demo
spec:
replicas: 1
selector:
matchLabels:
app: oauth2-proxy
template:
metadata:
labels:
app: oauth2-proxy
app.kubernetes.io/part-of: dsh-demo
spec:
# OPTIONAL: run the proxy under gVisor and/or pin it to a node pool.
# See ../../../docs/optional-hardening.md.
# runtimeClassName: gvisor
# nodeSelector:
# dsh.example.com/node-pool: dsh
securityContext:
runAsNonRoot: true
runAsUser: 2000
runAsGroup: 2000
seccompProfile:
type: RuntimeDefault
containers:
- name: oauth2-proxy
image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.5@sha256:8498b0d0ef0a7b29686414000a08aee467f02d0299c9ed1e006a8f33fc017916
imagePullPolicy: IfNotPresent
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: ["ALL"]
args:
- --http-address=0.0.0.0:4180
- --provider=github
# TODO: replace with the GitHub organization allowed to sign in.
- --github-org=your-github-org
- --email-domain=*
# Reverse-proxy target: the gateway backend Service.
- --upstream=http://dsh-gateway:8080
# Public callback (TLS terminates at the Gateway listener).
# TODO: replace with the actual public FQDN.
- --redirect-url=https://dsh-demo.example.com/oauth2/callback
- --cookie-secure=true
- --cookie-name=_oauth2_proxy
# Also expose X-Auth-Request-User/Email/Groups upstream.
- --set-xauthrequest
# Single provider: skip the provider-selection page.
- --skip-provider-button
env:
- name: OAUTH2_PROXY_CLIENT_ID
valueFrom:
secretKeyRef:
name: oauth2-proxy
key: client-id
- name: OAUTH2_PROXY_CLIENT_SECRET
valueFrom:
secretKeyRef:
name: oauth2-proxy
key: client-secret
- name: OAUTH2_PROXY_COOKIE_SECRET
valueFrom:
secretKeyRef:
name: oauth2-proxy
key: cookie-secret
ports:
- name: http
containerPort: 4180
livenessProbe:
httpGet:
path: /ping
port: http
initialDelaySeconds: 5
periodSeconds: 10
timeoutSeconds: 1
failureThreshold: 3
readinessProbe:
httpGet:
path: /ready
port: http
initialDelaySeconds: 5
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
resources:
requests:
cpu: 10m
memory: 32Mi
limits:
cpu: 200m
memory: 128Mi
---
apiVersion: v1
kind: Service
metadata:
name: oauth2-proxy
namespace: dsh-demo
labels:
app: oauth2-proxy
app.kubernetes.io/part-of: dsh-demo
spec:
selector:
app: oauth2-proxy
ports:
- name: http
port: 4180
targetPort: http

View File

@@ -0,0 +1,39 @@
# EXAMPLE ONLY — do not apply as-is.
#
# oauth2-proxy reads its GitHub OAuth credentials and cookie-signing secret from
# this Secret. Populate real values and apply:
#
# kubectl -n dsh-demo create secret generic oauth2-proxy \
# --from-literal=client-id='<GitHub OAuth App client ID>' \
# --from-literal=client-secret='<GitHub OAuth App client secret>' \
# --from-literal=cookie-secret="$(python3 -c 'import secrets; print(secrets.token_urlsafe(32))')"
#
# The cookie secret must decode (base64) to exactly 16, 24, or 32 bytes.
# oauth2-proxy base64-decodes it, so it must be URL-safe, unpadded base64.
# Generate it with either of:
#
# python3 -c 'import secrets; print(secrets.token_urlsafe(32))'
# openssl rand -base64 32 | tr -- '+/' '-_' | tr -d '='
#
# Do NOT use plain `openssl rand -base64 32`: that yields 44 characters of
# standard base64 (with "=" and possibly "+"/"/"), which oauth2-proxy cannot
# decode and therefore treats as 44 raw bytes, failing with "cookie_secret must
# be 16, 24, or 32 bytes ... but is 44 bytes".
#
# GitHub OAuth App (https://github.com/settings/developers):
# Homepage URL https://dsh-demo.example.com
# Authorization callback https://dsh-demo.example.com/oauth2/callback
# (replace dsh-demo.example.com with the actual public FQDN)
#
# NOTE: pass the base64 value as-is via --from-literal (kubectl stores the
# literal; it does not base64-encode it a second time).
apiVersion: v1
kind: Secret
metadata:
name: oauth2-proxy
namespace: dsh-demo
type: Opaque
stringData:
client-id: "<github-oauth-app-client-id>"
client-secret: "<github-oauth-app-client-secret>"
cookie-secret: "<urlsafe-unpadded-base64-of-32-random-bytes>"

View File

@@ -0,0 +1,17 @@
# EXAMPLE ONLY — do not apply as-is.
#
# Copy to tls-secret.yaml and populate with real base64-encoded PEM material,
# then apply. The secret must live in the dsh-demo namespace and be named
# "dsh-demo-tls" (referenced by 70-gateway.yaml).
#
# kubectl -n dsh-demo create secret tls dsh-demo-tls \
# --cert=tls.crt --key=tls.key --dry-run=client -o yaml > tls-secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: dsh-demo-tls
namespace: dsh-demo
type: kubernetes.io/tls
data:
tls.crt: <base64-encoded-certificate>
tls.key: <base64-encoded-private-key>

View File

@@ -0,0 +1,121 @@
# REQUIRED with the SSO exposure variant: enforce the trust boundary at the
# network layer. Applying the SSO variant without this file is demo-only.
#
# The chain is Traefik (Gateway API) -> oauth2-proxy -> dsh-gateway. These
# policies make it the ONLY chain:
# - the gateway accepts traffic only from oauth2-proxy pods;
# - oauth2-proxy accepts traffic only from the Gateway implementation;
# - oauth2-proxy egress is limited to DNS, public HTTPS (GitHub) and the
# gateway.
# The gateway's own egress (Kubernetes API, sandbox pods) is intentionally not
# restricted here: the API-server endpoint is cluster-specific. Add it for
# production. See
# ../../../docs/optional-hardening.md#gateway-egress-required-for-production.
#
# This manifest assumes Traefik, matching gatewayClassName: traefik in
# 70-gateway.yaml. Adjust the app.kubernetes.io/name selector for another
# implementation. NetworkPolicy enforcement of kubelet probes is CNI-dependent;
# if the gateway or proxy fails its probes, allow the node CIDR on the probe
# port or remove the ingress policy.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: dsh-gateway-ingress
namespace: dsh-demo
spec:
podSelector:
matchLabels:
app: dsh-gateway
policyTypes:
- Ingress
ingress:
- from:
- podSelector:
matchLabels:
app: oauth2-proxy
ports:
- protocol: TCP
port: 8080
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: oauth2-proxy-ingress
namespace: dsh-demo
spec:
podSelector:
matchLabels:
app: oauth2-proxy
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector: {}
podSelector:
matchLabels:
app.kubernetes.io/name: traefik
ports:
- protocol: TCP
port: 4180
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: oauth2-proxy-egress
namespace: dsh-demo
spec:
podSelector:
matchLabels:
app: oauth2-proxy
policyTypes:
- Egress
egress:
# Cluster DNS.
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
# NodeLocal DNSCache.
- to:
- ipBlock:
cidr: 169.254.20.10/32
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
# The gateway backend.
- to:
- podSelector:
matchLabels:
app: dsh-gateway
ports:
- protocol: TCP
port: 8080
# GitHub OAuth endpoints are public HTTPS. Private ranges and the metadata
# server stay blocked.
- to:
- ipBlock:
cidr: 0.0.0.0/0
except:
- 10.0.0.0/8
- 172.16.0.0/12
- 192.168.0.0/16
- 169.254.0.0/16
- 100.64.0.0/10
- ipBlock:
cidr: "::/0"
except:
- "fc00::/7"
- "fe80::/10"
ports:
- protocol: TCP
port: 443

View File

@@ -0,0 +1,63 @@
# OPTIONAL browser entry point: expose the gateway through Traefik.
#
# This is the "no SSO" exposure variant and is MUTUALLY EXCLUSIVE with
# ../gateway-api-sso/ — apply exactly one of them (see ../README.md).
#
# Requires the base manifests (Service dsh-gateway:8080 from
# manifests/50-gateway.yaml) and a Traefik install with an IngressClass named
# "traefik" (k3s ships one). Replace the placeholder host below with a name
# that resolves to your ingress address, and either supply a real TLS Secret or
# delete the `tls:` block for plain HTTP.
#
# Two things must match this host:
# 1. SandboxTemplate (manifests/10-sandboxtemplate.yaml): set
# DSH_TRUSTED_HOSTS to this host so DSH accepts the forwarded Host/Origin
# on /api.
# 2. Gateway Deployment (manifests/50-gateway.yaml): set COOKIE_SECURE to
# "true" and remove ALLOW_INSECURE_COOKIES once this Ingress serves HTTPS,
# so the session cookie is Secure:
# kubectl -n dsh-demo set env deploy/dsh-gateway \
# COOKIE_SECURE=true ALLOW_INSECURE_COOKIES-
# The base manifest sets "false" + ALLOW_INSECURE_COOKIES=1 for the
# plain-HTTP port-forward demo; without the acknowledgement the gateway
# refuses to start.
#
# WebSocket support is REQUIRED: the browser talks to /u/<user>/... and DSH
# uses WebSockets for live session traffic. Traefik upgrades WS/SSE
# transparently for ordinary HTTP routers, but do NOT attach buffering
# middlewares (e.g. compress, buffering) to this router — they break streaming
# and upgrades. `path: /` Prefix covers the landing page and every per-user
# proxy path, so no per-sandbox Ingress is needed.
#
# 61-networkpolicy.yaml is part of this variant and required for production:
# it restricts the gateway's ingress to Traefik pods. Adjust its selector if
# your ingress controller uses a different label or hostNetwork. The sandbox
# and gateway-egress NetworkPolicies are separate requirements; see
# ../../../docs/optional-hardening.md#6-production-checklist.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: dsh-gateway
namespace: dsh-demo
# Typical Traefik + cert-manager wiring (uncomment and adapt):
# annotations:
# traefik.ingress.kubernetes.io/router.entrypoints: websecure
# cert-manager.io/cluster-issuer: letsencrypt-prod
annotations: {}
spec:
ingressClassName: traefik
tls:
- hosts:
- dsh.example.com # placeholder
secretName: dsh-example-tls # placeholder; create/PKI-provision this Secret
rules:
- host: dsh.example.com # placeholder
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: dsh-gateway
port:
number: 8080

View File

@@ -0,0 +1,36 @@
# REQUIRED with the Ingress exposure variant for production: restrict the
# gateway's ingress to the ingress controller. Applying the variant without
# this policy is demo-only.
#
# This manifest assumes Traefik, matching this variant's
# `ingressClassName: traefik`. Traefik's Helm chart labels its pods
# app.kubernetes.io/name=traefik; adjust the selector for another controller or
# for a hostNetwork install (where the source is the node IP, not a pod IP).
#
# The sandbox NetworkPolicy and the gateway egress policy are separate and also
# required for production; see
# ../../../docs/optional-hardening.md#6-production-checklist.
#
# NetworkPolicy enforcement of kubelet probe traffic is CNI-dependent. If the
# gateway starts failing its /healthz probes, allow the node CIDR on port 8080
# or remove this policy. See ../../../docs/optional-hardening.md.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: dsh-gateway-ingress
namespace: dsh-demo
spec:
podSelector:
matchLabels:
app: dsh-gateway
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector: {}
podSelector:
matchLabels:
app.kubernetes.io/name: traefik
ports:
- protocol: TCP
port: 8080

View File

@@ -0,0 +1,46 @@
# Optional manifests
Opt-in manifests that are **not** part of the top-level walkthrough.
`kubectl apply -f manifests/` is non-recursive, so it does not apply this
directory; apply a subdirectory explicitly.
## network-policy/
A Kubernetes `NetworkPolicy` for the DSH sandboxes:
- ingress only from the gateway pods on TCP `3080` (HTTP + WebSocket);
- egress only to DNS and public HTTPS (provider APIs, package mirrors,
git-over-HTTPS).
Apply and delete it explicitly:
```bash
kubectl apply -f manifests/optional/network-policy/
kubectl delete -f manifests/optional/network-policy/
```
It requires a CNI that enforces `NetworkPolicy`. The selector uses the
agent-sandbox controller's pod tracking label, so the policy applies to
existing warm-pool and claimed sandboxes without recreation.
**This is opt-in for the demo but required for production.** Without it every
tenant can reach every other tenant and cluster-internal endpoints. Production
also needs the gateway egress policy (cluster-specific API-server endpoint) and
the exposure variant's policy; see the
[production checklist](../../docs/optional-hardening.md#6-production-checklist).
## gVisor, nodeSelector and provider keys
These are configured in the main manifests or with an extra Secret:
- **gVisor:** uncomment `runtimeClassName: gvisor` in the SandboxTemplate and
the Deployments before applying, or patch an existing install.
- **nodeSelector:** uncomment the field and replace the placeholder node label.
- **Failed-Pod recovery:** uncomment `failedPodPolicy: Recreate` in the
SandboxTemplate (agent-sandbox v1.0.8+) so a Failed Pod is replaced instead
of parked.
- **Provider keys:** create the optional `dsh-provider-keys-extra` Secret; the
base `dsh-provider-keys` Secret remains the one required key.
Full prerequisites, exact commands, verification and caveats:
[optional hardening](../../docs/optional-hardening.md).

View File

@@ -0,0 +1,86 @@
# OPTIONAL for the base walkthrough, REQUIRED for production. Opt-in tenant
# network policy for the DSH sandboxes: without it every tenant can reach every
# other tenant and cluster-internal endpoints.
#
# It is NOT part of the top-level walkthrough. Apply and delete it explicitly:
# kubectl apply -f manifests/optional/network-policy/
# kubectl delete -f manifests/optional/network-policy/
#
# Production also needs the gateway egress policy (cluster-specific API-server
# endpoint) and the exposure variant's policy, plus gVisor
# (docs/optional-hardening.md#6-production-checklist).
#
# The selector uses the agent-sandbox controller's pod tracking label, which is
# present on every Sandbox pod (including warm-pool spares), so the policy
# applies to existing sandboxes without recreating them. The SandboxTemplate
# stays networkPolicyManagement: Unmanaged. Do not also switch it to Managed
# unless you migrate these rules into spec.networkPolicy; NetworkPolicies are
# additive and the two configurations would be confusing.
#
# Requires a CNI that enforces NetworkPolicy. See docs/optional-hardening.md
# for the connectivity matrix, DNS variants, kubelet-probe caveat and rollback.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: dsh-sandbox
namespace: dsh-demo
spec:
podSelector:
matchExpressions:
- key: agents.x-k8s.io/sandbox-name-hash
operator: Exists
policyTypes:
- Ingress
- Egress
ingress:
# The gateway is the only intended client of a sandbox (HTTP + WebSocket).
- from:
- podSelector:
matchLabels:
app: dsh-gateway
ports:
- protocol: TCP
port: 3080
egress:
# Cluster DNS. CoreDNS/kube-dns pods carry k8s-app=kube-dns.
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
# NodeLocal DNSCache, for clusters that do not send DNS to the kube-dns
# Service. 169.254.20.10 is its standard link-local address.
- to:
- ipBlock:
cidr: 169.254.20.10/32
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
# Public HTTPS only: provider APIs, package mirrors, git-over-HTTPS.
# Private ranges, CGNAT and the metadata server stay blocked.
- to:
- ipBlock:
cidr: 0.0.0.0/0
except:
- 10.0.0.0/8
- 172.16.0.0/12
- 192.168.0.0/16
- 169.254.0.0/16
- 100.64.0.0/10
- ipBlock:
cidr: "::/0"
except:
- "fc00::/7"
- "fe80::/10"
ports:
- protocol: TCP
port: 443

View File

@@ -1,13 +1,37 @@
#!/usr/bin/env bash
# Smoke test for the dsh-agents-as-a-service example: runs the full README
# walkthrough non-interactively against the current kubectl context.
# Expects agent-sandbox (with extensions) already installed.
# End-to-end smoke test for the dsh-agents-as-a-service example: runs the
# README walkthrough non-interactively against the current kubectl context.
#
# Prerequisites:
# - A kind cluster whose name matches the current kubectl context, or set
# KIND_CLUSTER=<name>.
# - agent-sandbox with extensions installed (API v1beta1). The release
# manifest is pinned by AGENT_SANDBOX_VERSION in versions.env:
# . ../../../versions.env
# kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/download/${AGENT_SANDBOX_VERSION}/sandbox-with-extensions.yaml
# - docker, to build the gateway image (skipped with SKIP_GATEWAY=1).
#
# Environment:
# SKIP_GATEWAY=1 run only the resource-layer checks (no gateway build/load,
# no signup/proxy/WebSocket/idle/wake checks)
# KIND_CLUSTER kind cluster name for "kind load docker-image"
# (default: derived from the current kubectl context)
# GW_LOCAL_IMAGE local gateway image tag
# (default: dsh-aaas-gateway:e2e)
#
# The gateway is built from the working tree (gateway/) and loaded into kind,
# so this tests the example as checked out, not the published image.
#
# The script is destructive only inside the dsh-demo namespace: it creates the
# namespace (and a throwaway provider key) and DELETES it on exit via the
# cleanup trap.
#
# NOTE: the only local port bound is the gateway port-forward on 18080.
# The DSH Web container port (3080) is in-cluster only and is never bound
# on the local host.
set -euo pipefail
cd "$(dirname "$0")"
# Example root: paths below are manifests/... and gateway/... from here.
cd "$(dirname "$0")/.."
NS=dsh-demo
pass() { echo "PASS: $1"; }
@@ -20,12 +44,14 @@ cleanup() {
trap cleanup EXIT
echo "=== 1. install prereqs (throwaway key) + template + warm pool"
kubectl apply -f 00-prereqs.yaml
kubectl apply -f manifests/00-prereqs.yaml
# The provider key is never committed; generate a throwaway value.
kubectl -n "$NS" delete secret dsh-provider-keys --ignore-not-found >/dev/null
kubectl -n "$NS" create secret generic dsh-provider-keys \
--from-literal=DEEPSEEK_API_KEY="sk-test-$(openssl rand -hex 16)"
kubectl apply -f 10-sandboxtemplate.yaml -f 20-sandboxwarmpool.yaml
# dsh-provider-keys-extra is intentionally absent: the template's optional
# envFrom reference must not block the warm pool.
kubectl apply -f manifests/10-sandboxtemplate.yaml -f manifests/20-sandboxwarmpool.yaml
for _ in $(seq 1 120); do
READY=$(kubectl -n "$NS" get sandboxes -l agents.x-k8s.io/warm-pool-sandbox \
-o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' | grep -c True || true)
@@ -37,7 +63,7 @@ pass "warm pool: 2 spares Ready"
echo "=== 2. signup via claim (warm adoption)"
T0=$(date +%s)
kubectl apply -f 30-claim-alice.yaml
kubectl apply -f manifests/30-claim-alice.yaml
kubectl -n "$NS" wait sandboxclaim dsh-alice --for=condition=Ready --timeout=60s
ELAPSED=$(( $(date +%s) - T0 ))
SB=$(kubectl -n "$NS" get sandboxclaim dsh-alice -o jsonpath='{.status.sandbox.name}')
@@ -46,8 +72,23 @@ pass "claim Ready in ${ELAPSED}s, adopted $SB"
echo "=== 3. write state on the PVC"
POD="$SB"
# HOME must point at the PVC-backed workspace: DSH's directory picker lists
# $HOME when it opens and has no start-path setting, so this is what makes the
# folder explorer start on the persisted tree instead of the ephemeral image
# home. It also makes ~ resolve to /workspace inside agent commands.
HOME_VALUE=$(kubectl -n "$NS" exec "$POD" -- printenv HOME)
[ "$HOME_VALUE" = "/workspace" ] || fail "HOME is '$HOME_VALUE', expected /workspace"
TILDE=$(kubectl -n "$NS" exec "$POD" -- sh -c 'cd ~ && pwd')
[ "$TILDE" = "/workspace" ] || fail "~ resolves to '$TILDE', expected /workspace"
pass "HOME=$HOME_VALUE and ~ resolve to the workspace"
kubectl -n "$NS" exec "$POD" -- sh -c 'echo "remember me" > /home/node/.dsh/marker.txt'
pass "marker written in $POD"
pass "state marker written in $POD"
# The agent workspace must be the dsh-home PVC's workspace/ subPath, writable
# as UID 1000 without an init container.
SUBPATH=$(kubectl -n "$NS" get pod "$POD" -o jsonpath='{range .spec.containers[0].volumeMounts[?(@.mountPath=="/workspace")]}{.subPath}{end}')
[ "$SUBPATH" = "workspace" ] || fail "/workspace is not mounted from the dsh-home PVC subPath"
kubectl -n "$NS" exec "$POD" -- sh -c 'echo "remember me too" > /workspace/marker.txt'
pass "workspace marker written in $POD (subPath=$SUBPATH)"
echo "=== 4. suspend (pod deleted, PVC + Service retained)"
kubectl -n "$NS" patch sandbox "$SB" --type merge -p '{"spec":{"operatingMode":"Suspended"}}'
@@ -61,11 +102,14 @@ kubectl -n "$NS" patch sandbox "$SB" --type merge -p '{"spec":{"operatingMode":"
kubectl -n "$NS" wait sandbox "$SB" --for=condition=Ready --timeout=180s
POD="$SB"
MARKER=$(kubectl -n "$NS" exec "$POD" -- cat /home/node/.dsh/marker.txt)
[ "$MARKER" = "remember me" ] || fail "marker did not survive suspend/resume"
pass "resumed: state survived on the reattached PVC"
[ "$MARKER" = "remember me" ] || fail "state marker did not survive suspend/resume"
pass "resumed: DSH state survived on the reattached PVC"
WS_MARKER=$(kubectl -n "$NS" exec "$POD" -- cat /workspace/marker.txt)
[ "$WS_MARKER" = "remember me too" ] || fail "workspace marker did not survive suspend/resume"
pass "resumed: agent workspace survived on the PVC subPath"
echo "=== 6. injection policy rejects env-carrying claims"
kubectl apply -f 40-claim-rejected.yaml
kubectl apply -f manifests/40-claim-rejected.yaml
# Wait for the POSITIVE rejection signal, not merely "not bound yet".
REASON=""
for _ in $(seq 1 12); do
@@ -78,19 +122,25 @@ done
|| fail "claim was not rejected (Ready reason='$REASON')"
BOUND=$(kubectl -n "$NS" get sandboxclaim dsh-mallory -o jsonpath='{.status.sandbox.name}' 2>/dev/null || true)
[ -z "$BOUND" ] || fail "env-carrying claim unexpectedly bound sandbox '$BOUND'"
kubectl delete -f 40-claim-rejected.yaml >/dev/null
kubectl delete -f manifests/40-claim-rejected.yaml >/dev/null
pass "claim rejected with EnvVarsInjectionRejected and never bound"
echo "=== 7. gateway: browser signup, proxy, idle-suspend, wake-on-connect"
if command -v docker >/dev/null && command -v kind >/dev/null; then
KIND_CLUSTER="${KIND_CLUSTER:-$(kubectl config current-context | sed 's/^kind-//')}"
# Tag exactly what 50-gateway.yaml references so the smoke test exercises
# the shipped manifest unchanged (imagePullPolicy: IfNotPresent uses the
# image loaded into kind).
GW_IMAGE=ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway
docker build -q -t "$GW_IMAGE" gateway/ >/dev/null
kind load docker-image "$GW_IMAGE" --name "$KIND_CLUSTER" >/dev/null
kubectl apply -f 50-gateway.yaml
GW_LOCAL_IMAGE="${GW_LOCAL_IMAGE:-dsh-aaas-gateway:e2e}"
# Test the working tree, not the published image: build the gateway from
# gateway/, load it into kind, then point the Deployment at that tag. The
# shipped manifest pins the release image; imagePullPolicy: IfNotPresent
# makes the node use the loaded one.
docker build -q -t "$GW_LOCAL_IMAGE" gateway/ >/dev/null
kind load docker-image "$GW_LOCAL_IMAGE" --name "$KIND_CLUSTER" >/dev/null
kubectl apply -f manifests/50-gateway.yaml
kubectl -n "$NS" set image deploy/dsh-gateway gateway="$GW_LOCAL_IMAGE"
# Override the demo timeouts from manifests/50-gateway.yaml (360/240) with the
# gateway's working-tree defaults, so idle-suspend and wake-on-connect are
# exercised in seconds instead of minutes.
kubectl -n "$NS" set env deploy/dsh-gateway IDLE_TIMEOUT=60 WAKE_TIMEOUT=120
kubectl -n "$NS" rollout status deploy/dsh-gateway --timeout=120s
# Local port-forward uses 18080 (NOT 3080, which stays in-cluster).
kubectl -n "$NS" port-forward svc/dsh-gateway 18080:8080 >/dev/null 2>&1 &

View File

@@ -0,0 +1,234 @@
#!/usr/bin/env bash
# Preflight for the gateway-api-sso exposure variant.
#
# Run this AFTER creating the oauth2-proxy/TLS Secrets, setting the gateway
# env and applying manifests/exposure/gateway-api-sso/, and BEFORE sending real
# traffic. It checks the configuration that is easy to get wrong and whose
# failure is either silent (login appears to work but the gateway still asks
# for its own token) or dangerous (a spoofable identity header, a placeholder
# host, a TLS Secret that does not exist).
#
# Checks:
# 1. kubectl reaches the cluster and the SSO objects exist;
# 2. the oauth2-proxy Secret has client-id, client-secret and a cookie-secret
# that decodes to 16, 24 or 32 bytes; the TLS Secret exists;
# 3. deploy/dsh-gateway has AUTH_PROXY_HEADER=X-Forwarded-User and
# COOKIE_SECURE=true (both are opt-in and both are required for SSO over
# TLS);
# 4. the Gateway's TLS certificate reference resolves to an existing Secret;
# 5. no your-github-org / dsh-demo.example.com placeholder is left in the
# applied oauth2-proxy args, Gateway listeners or HTTPRoute hostnames;
# 6. (warning) no Ingress in the namespace creates a second path to
# dsh-gateway. With SSO enabled the Service must be reachable only through
# oauth2-proxy.
#
# Exit status: 0 when every check passes (warnings allowed), 1 on any failure.
# Every failure prints the fix. This script only reads cluster state.
#
# Usage:
# examples/dsh-agents-as-a-service/scripts/preflight-sso.sh
#
# Environment overrides:
# NAMESPACE namespace (default: dsh-demo)
# GATEWAY_DEPLOY gateway Deployment name (default: dsh-gateway)
# OAUTH_DEPLOY oauth2-proxy Deployment name (default: oauth2-proxy)
# OAUTH_SECRET oauth2-proxy Secret name (default: oauth2-proxy)
# TLS_SECRET TLS Secret name the Gateway should reference (default:
# dsh-demo-tls)
set -euo pipefail
cd "$(dirname "$0")/.."
NS="${NAMESPACE:-dsh-demo}"
GATEWAY_DEPLOY="${GATEWAY_DEPLOY:-dsh-gateway}"
OAUTH_DEPLOY="${OAUTH_DEPLOY:-oauth2-proxy}"
OAUTH_SECRET="${OAUTH_SECRET:-oauth2-proxy}"
TLS_SECRET="${TLS_SECRET:-dsh-demo-tls}"
FAILED=0
pass() { printf 'PASS %s\n' "$*"; }
warn() { printf 'WARN %s\n' "$*" >&2; }
fail() { printf 'FAIL %s\n' "$*" >&2; FAILED=1; }
if ! command -v kubectl >/dev/null 2>&1; then
echo "FAIL kubectl is not installed; install it and select the target context" >&2
exit 1
fi
if ! kubectl -n "$NS" get deployment "$GATEWAY_DEPLOY" >/dev/null 2>&1; then
echo "FAIL deploy/$GATEWAY_DEPLOY not found in namespace $NS." >&2
echo " Apply the base manifests first: kubectl apply -f manifests/" >&2
echo " Then apply the variant: kubectl apply -f manifests/exposure/gateway-api-sso/" >&2
exit 1
fi
echo "== Secrets"
# check_secret_key <secret> <key> <hint>
check_secret_key() {
local secret=$1 key=$2 hint=$3 value escaped
# Kubernetes JSONPath treats "." as a path separator; escape it in keys such
# as tls.crt.
escaped=${key//./\\.}
value=$(kubectl -n "$NS" get secret "$secret" -o jsonpath="{.data.${escaped}}" 2>/dev/null || true)
if [ -n "$value" ]; then
pass "Secret $secret has non-empty key '$key'"
else
fail "Secret $secret is missing a non-empty '$key' key. $hint"
fi
}
check_secret_key "$OAUTH_SECRET" client-id \
"Create it from manifests/exposure/gateway-api-sso/73-oauth2-proxy-secret.yaml.example."
check_secret_key "$OAUTH_SECRET" client-secret \
"Create it from manifests/exposure/gateway-api-sso/73-oauth2-proxy-secret.yaml.example."
check_secret_key "$OAUTH_SECRET" cookie-secret \
"Create it from manifests/exposure/gateway-api-sso/73-oauth2-proxy-secret.yaml.example."
# The cookie secret must be URL-safe unpadded base64 of 16, 24 or 32 bytes;
# oauth2-proxy refuses anything else (the classic error is "44 bytes").
if COOKIE_B64=$(kubectl -n "$NS" get secret "$OAUTH_SECRET" \
-o jsonpath='{.data.cookie-secret}' 2>/dev/null); then
if [ -n "$COOKIE_B64" ]; then
if ! COOKIE_LEN=$(printf '%s' "$COOKIE_B64" | base64 -d 2>/dev/null | wc -c); then
fail "Secret $OAUTH_SECRET key 'cookie-secret' is not valid base64. Generate it with: python3 -c 'import secrets; print(secrets.token_urlsafe(32))'"
else
case "$COOKIE_LEN" in
16 | 24 | 32) pass "oauth2-proxy cookie-secret decodes to $COOKIE_LEN bytes" ;;
*)
fail "oauth2-proxy cookie-secret decodes to $COOKIE_LEN bytes; it must be 16, 24 or 32. Generate a URL-safe unpadded value with: python3 -c 'import secrets; print(secrets.token_urlsafe(32))'"
;;
esac
fi
fi
fi
check_secret_key "$TLS_SECRET" tls.crt \
"Create the TLS Secret with: kubectl -n $NS create secret tls $TLS_SECRET --cert=tls.crt --key=tls.key"
check_secret_key "$TLS_SECRET" tls.key \
"Create the TLS Secret with: kubectl -n $NS create secret tls $TLS_SECRET --cert=tls.crt --key=tls.key"
echo
echo "== Gateway SSO configuration (deploy/$GATEWAY_DEPLOY)"
GATEWAY_ENV=$(kubectl -n "$NS" get deployment "$GATEWAY_DEPLOY" \
-o jsonpath='{range .spec.template.spec.containers[*].env[*]}{.name}={.value}{"\n"}{end}')
# env_value <NAME>: print the literal env value, or nothing when unset.
env_value() {
printf '%s\n' "$GATEWAY_ENV" | awk -F= -v key="$1" '$1 == key { sub(/^[^=]*=/, ""); print }'
}
env_is_set() {
printf '%s\n' "$GATEWAY_ENV" | awk -F= -v key="$1" '$1 == key { found = 1 } END { exit found ? 0 : 1 }'
}
if ! env_is_set AUTH_PROXY_HEADER; then
fail "AUTH_PROXY_HEADER is not set on deploy/$GATEWAY_DEPLOY, so the gateway ignores oauth2-proxy's identity header and users are still asked for the gateway token. Fix: kubectl -n $NS set env deploy/$GATEWAY_DEPLOY AUTH_PROXY_HEADER=X-Forwarded-User"
elif [ "$(env_value AUTH_PROXY_HEADER)" != "X-Forwarded-User" ]; then
fail "AUTH_PROXY_HEADER is '$(env_value AUTH_PROXY_HEADER)', expected 'X-Forwarded-User'. Fix: kubectl -n $NS set env deploy/$GATEWAY_DEPLOY AUTH_PROXY_HEADER=X-Forwarded-User"
else
pass "AUTH_PROXY_HEADER=X-Forwarded-User"
fi
if ! env_is_set COOKIE_SECURE; then
fail "COOKIE_SECURE is not set on deploy/$GATEWAY_DEPLOY; the session cookie must be Secure behind TLS. Fix: kubectl -n $NS set env deploy/$GATEWAY_DEPLOY COOKIE_SECURE=true"
elif [ "$(env_value COOKIE_SECURE | tr '[:upper:]' '[:lower:]')" != "true" ]; then
fail "COOKIE_SECURE is '$(env_value COOKIE_SECURE)', expected 'true' behind TLS. Fix: kubectl -n $NS set env deploy/$GATEWAY_DEPLOY COOKIE_SECURE=true"
else
pass "COOKIE_SECURE=true"
fi
if env_is_set ALLOW_INSECURE_COOKIES; then
warn "ALLOW_INSECURE_COOKIES is still set on deploy/$GATEWAY_DEPLOY. It is ignored while COOKIE_SECURE=true but is a footgun: remove it with 'kubectl -n $NS set env deploy/$GATEWAY_DEPLOY ALLOW_INSECURE_COOKIES-'."
fi
echo
echo "== SSO objects and placeholders"
if kubectl -n "$NS" get deployment "$OAUTH_DEPLOY" >/dev/null 2>&1; then
pass "deploy/$OAUTH_DEPLOY exists"
OAUTH_ARGS=$(kubectl -n "$NS" get deployment "$OAUTH_DEPLOY" \
-o jsonpath='{range .spec.template.spec.containers[*].args[*]}{@}{"\n"}{end}')
else
OAUTH_ARGS=""
fail "deploy/$OAUTH_DEPLOY not found. Apply the variant: kubectl apply -f manifests/exposure/gateway-api-sso/"
fi
# GitHub org restriction is the only authorization in the shipped variant.
# --email-domain=* is acceptable only because --github-org narrows membership;
# a missing or empty org value would accept any GitHub account.
if [ -n "$OAUTH_ARGS" ]; then
if printf '%s\n' "$OAUTH_ARGS" | grep -q '^--provider=github$'; then
pass "oauth2-proxy uses --provider=github"
else
fail "oauth2-proxy is not configured with --provider=github. Fix: add --provider=github to deploy/$OAUTH_DEPLOY."
fi
if printf '%s\n' "$OAUTH_ARGS" | grep -Eq '^--github-org=.+$'; then
pass "oauth2-proxy restricts membership with --github-org"
else
fail "oauth2-proxy has no non-empty --github-org. Fix: set --github-org=<your-org> in deploy/$OAUTH_DEPLOY; otherwise --email-domain=* accepts any GitHub account."
fi
fi
if kubectl -n "$NS" get httproute dsh-gateway >/dev/null 2>&1; then
pass "HTTPRoute dsh-gateway exists"
ROUTE_STATE=$(kubectl -n "$NS" get httproute dsh-gateway \
-o jsonpath='{.spec.hostnames[*]}' 2>/dev/null || true)
else
ROUTE_STATE=""
fail "HTTPRoute dsh-gateway not found. Apply the variant: kubectl apply -f manifests/exposure/gateway-api-sso/"
fi
if kubectl -n "$NS" get gateway dsh-gateway >/dev/null 2>&1; then
pass "Gateway dsh-gateway exists"
GATEWAY_STATE=$(kubectl -n "$NS" get gateway dsh-gateway \
-o jsonpath='{range .spec.listeners[*]}{.hostname}{"\n"}{end}' 2>/dev/null || true)
CERT_REFS=$(kubectl -n "$NS" get gateway dsh-gateway \
-o jsonpath='{range .spec.listeners[*].tls.certificateRefs[*]}{.namespace}{"/"}{.name}{"\n"}{end}' 2>/dev/null || true)
else
GATEWAY_STATE=""
CERT_REFS=""
fail "Gateway dsh-gateway not found. Apply the variant: kubectl apply -f manifests/exposure/gateway-api-sso/"
fi
# Placeholders are the default values in the committed manifests. They reach
# the cluster if the variant is applied before editing them.
PLACEHOLDER_RE='your-github-org|dsh-demo\.example\.com'
PLACEHOLDER_STATE=$(printf '%s\n%s\n%s\n' "$OAUTH_ARGS" "$ROUTE_STATE" "$GATEWAY_STATE")
if printf '%s\n' "$PLACEHOLDER_STATE" | grep -En "$PLACEHOLDER_RE" >/dev/null 2>&1; then
fail "the applied SSO objects still contain a placeholder (your-github-org or dsh-demo.example.com):"
printf '%s\n' "$PLACEHOLDER_STATE" | grep -En "$PLACEHOLDER_RE" | sed 's/^/ /' >&2
echo " Replace --github-org, --redirect-url, the Gateway listener host and the HTTPRoute hostname with the real public FQDN and organization, then re-apply and restart the proxy." >&2
else
pass "no your-github-org / dsh-demo.example.com placeholder in the applied SSO objects"
fi
# The Gateway must reference an existing TLS Secret; the default name is
# dsh-demo-tls, but 70-gateway.yaml can be edited to another name.
if [ -n "$CERT_REFS" ]; then
while IFS= read -r ref; do
[ -n "$ref" ] || continue
ref_ns=${ref%%/*}
ref_name=${ref#*/}
ref_ns=${ref_ns:-$NS}
if kubectl -n "$ref_ns" get secret "$ref_name" >/dev/null 2>&1; then
pass "Gateway TLS certificate reference ${ref_ns}/${ref_name} exists"
else
fail "Gateway references TLS Secret ${ref_ns}/${ref_name} but it does not exist. Create it (see manifests/exposure/gateway-api-sso/74-tls-secret.yaml.example)."
fi
done <<<"$CERT_REFS"
else
warn "the Gateway has no TLS certificateRefs; SSO over plain HTTP would expose the session cookie. Terminate TLS at the Gateway listener."
fi
# A second path to the gateway defeats SSO: anyone who can reach
# dsh-gateway:8080 can spoof X-Forwarded-User. This is a best-effort check.
if kubectl -n "$NS" get ingress >/dev/null 2>&1; then
INGRESS_STATE=$(kubectl -n "$NS" get ingress -o json 2>/dev/null || true)
if printf '%s' "$INGRESS_STATE" | grep -q '"name":"dsh-gateway"'; then
fail "an Ingress in namespace $NS also routes to dsh-gateway. With SSO enabled the gateway must be reachable only through oauth2-proxy: delete the Ingress variant (kubectl delete -f manifests/exposure/ingress/)."
fi
fi
echo
if [ "$FAILED" -ne 0 ]; then
echo "preflight-sso: FAILED - fix the items above before exposing the gateway." >&2
exit 1
fi
echo "preflight-sso: ok - the SSO variant looks correctly configured."

View File

@@ -0,0 +1,174 @@
# Minimal single-tenant DSH Web
The smallest Kubernetes deployment of DeepSeek Harness (DSH) Web: one
`SandboxTemplate`, a one-replica warm pool and one claim. It is the "just one
DSH Web instance" quickstart.
For the full platform — warm pool sizing, per-user claims, a browser-facing
gateway with cookie sessions, idle-suspend and wake-on-connect — see
[`examples/dsh-agents-as-a-service/`](../dsh-agents-as-a-service/README.md).
This example deliberately has no gateway: you reach DSH through a
`kubectl port-forward` and its launch-token URL.
## Prerequisites
- A Kubernetes cluster with a default StorageClass (kind and GKE both work).
- The agent-sandbox controller with extensions (API `v1beta1`) installed.
The version is pinned as `AGENT_SANDBOX_VERSION` in
[`versions.env`](../../versions.env):
```bash
# from the repository root:
. versions.env
kubectl apply -f "https://github.com/kubernetes-sigs/agent-sandbox/releases/download/${AGENT_SANDBOX_VERSION}/sandbox-with-extensions.yaml"
kubectl -n agent-sandbox-system rollout status deployment --timeout=180s
```
- A DeepSeek API key for real model calls. The pod starts without one; only
model requests fail. Optionally create `dsh-provider-keys-extra` for
OpenRouter, Gemini, NVIDIA or other provider keys (the template references it
with `optional: true`).
## Quickstart
Run these from this example's directory (`examples/minimal-single-tenant/`).
```bash
# 1. Namespace + shared provider keys. They are injected into the sandbox via
# envFrom and are never committed.
kubectl create namespace dsh-demo
kubectl -n dsh-demo create secret generic dsh-provider-keys \
--from-literal=DEEPSEEK_API_KEY="sk-..."
# Optional: extra providers, injected through a second Secret. Create it before
# the warm pool so the pods are stamped with the keys.
kubectl -n dsh-demo create secret generic dsh-provider-keys-extra \
--from-literal=OPENROUTER_API_KEY="sk-or-..." \
--from-literal=GEMINI_API_KEY="..." \
--from-literal=NVIDIA_API_KEY="nvapi-..."
# 2. The template itself (image pinned in the manifest).
kubectl apply -f manifests/10-sandboxtemplate.yaml
# 3. A template only describes pods; a warm pool turns it into running
# Sandboxes and a claim adopts one. One replica is enough here.
kubectl apply -f - <<'EOF'
apiVersion: extensions.agents.x-k8s.io/v1beta1
kind: SandboxWarmPool
metadata:
name: dsh-pool
namespace: dsh-demo
spec:
replicas: 1
sandboxTemplateRef:
name: dsh-web-template
---
apiVersion: extensions.agents.x-k8s.io/v1beta1
kind: SandboxClaim
metadata:
name: dsh
namespace: dsh-demo
spec:
warmPoolRef:
name: dsh-pool
EOF
kubectl -n dsh-demo wait sandboxclaim/dsh --for=condition=Ready --timeout=180s
```
## Talk to the instance
DSH gates `/` behind a launch token it prints only in its startup logs. Open
the printed URL once; DSH then sets its own auth cookie and the UI loads
normally.
```bash
SB=$(kubectl -n dsh-demo get sandboxclaim dsh -o jsonpath='{.status.sandbox.name}')
kubectl -n dsh-demo logs "$SB" | grep -m1 '?token='
# dsh web: http://<pod-ip>:3080/?token=...
kubectl -n dsh-demo port-forward "pod/$SB" 18081:3080
# open http://localhost:18081/?token=... from the log line
```
> **Port note:** DSH Web listens on `3080` in-cluster. If `3080` is already in
> use on your workstation, forward to another local port (here `18081`), never
> `3080`. `localhost` is always in DSH's trusted-host list, so the forwarded
> request is accepted.
>
> **gVisor caveat:** if you uncomment `runtimeClassName: gvisor` in the
> template, `kubectl port-forward` no longer works. Use the multi-tenant gateway
> example or a reachable Service instead; see
> [optional hardening](../dsh-agents-as-a-service/docs/optional-hardening.md).
## Folder picker and the workspace
The template mounts the `dsh-home` PVC's `workspace/` subPath at
`/workspace` and sets `HOME=/workspace`. DSH's directory picker (the folder
explorer used to choose a workspace) has no start-path setting: it lists `$HOME`
when it opens, so with the image default (`/home/node`) it would start on a
directory that is not persisted. Pointing `HOME` at the PVC-backed workspace makes
the picker open there (labelled *Home* in its breadcrumbs) and makes `~` in agent
commands resolve to `/workspace`. The trade-off is that `$HOME`-relative tool
caches and config (`.cache`, `.config`, `.local`, `.npm`, `.agents`) are written
into the workspace on the PVC. DSH state is unaffected: `DSH_HOME` stays
`/home/node/.dsh`. The rationale and the list of affected paths are in
[`10-sandboxtemplate.yaml`](../dsh-agents-as-a-service/manifests/10-sandboxtemplate.yaml).
Pod specs are stamped at creation, so an existing sandbox keeps its old mounts and
`HOME` until it is recreated; the PVC retains DSH state and workspace data.
Verify inside a running sandbox:
```bash
kubectl -n dsh-demo exec "$SB" -- printenv HOME # /workspace
```
## Network policy and optional hardening
The template omits `networkPolicyManagement`, so the agent-sandbox controller
uses its Managed secure default: ingress only from the sandbox-router, public
egress, and no internal cluster DNS. `kubectl port-forward` is local to the pod
and still works (except under gVisor).
gVisor, an optional `nodeSelector`, extra provider keys and a stricter sandbox
NetworkPolicy are documented in
[optional hardening](../dsh-agents-as-a-service/docs/optional-hardening.md).
## Probes and trusted hosts
Two settings in `manifests/10-sandboxtemplate.yaml` are easy to get wrong:
- **Probe path.** `/` returns `401` without the launch-token cookie, so a probe
there crash-loops the pod. The template probes `/favicon.svg` (served
unauthenticated) and gates liveness/readiness behind a `startupProbe`, which
gives a cold start (plugin activation, or the first resume) up to 120s before
a restart.
- **Host allow-list.** DSH rejects unknown `Host` headers. `localhost` and
`127.0.0.1` are always trusted; if you later front the sandbox with an
Ingress or proxy, set `DSH_TRUSTED_HOSTS` to that public host (comma or space
separated). The full example explains both exposure variants in
[`manifests/exposure/README.md`](../dsh-agents-as-a-service/manifests/exposure/README.md).
The template also sets `fsGroupChangePolicy: OnRootMismatch`. DSH's
credentials plugin requires `.credentials.yaml` to be `0600`, while the
kubelet's default fsGroup policy rewrites it to `0660` on every pod start —
without this policy a recreated or resumed sandbox crash-loops. The rationale
is spelled out in
[`10-sandboxtemplate.yaml`](../dsh-agents-as-a-service/manifests/10-sandboxtemplate.yaml).
## Suspend, resume and clean up
```bash
# Suspend: the pod is deleted, the PVC (state) and Service are kept.
kubectl -n dsh-demo patch sandbox "$SB" --type merge -p '{"spec":{"operatingMode":"Suspended"}}'
# Resume: a new pod reattaches the same PVC; conversations survive.
kubectl -n dsh-demo patch sandbox "$SB" --type merge -p '{"spec":{"operatingMode":"Running"}}'
kubectl -n dsh-demo wait sandbox "$SB" --for=condition=Ready --timeout=180s
# Delete everything this quickstart created.
kubectl -n dsh-demo delete sandboxclaim dsh
kubectl delete namespace dsh-demo
```
Deleting the claim cascades to the sandbox and its PVC.

View File

@@ -0,0 +1,154 @@
# Minimal single-tenant SandboxTemplate for DeepSeek Harness Web.
#
# This is the "just one DSH Web" quick-start template. For the full
# multi-tenant platform (warm pool, per-user claims, gateway, idle-suspend),
# see examples/dsh-agents-as-a-service/.
#
# Claims reference this template directly (spec.sandboxTemplateRef): there is
# no warm pool and no per-user platform here. See the README for the full
# quickstart; in short the prerequisites are:
# - Namespace dsh-demo.
# - Secret dsh-provider-keys in dsh-demo with key DEEPSEEK_API_KEY
# (injected via envFrom; the pod starts without it, only model calls fail).
# - Optional: Secret dsh-provider-keys-extra with additional provider keys
# (OPENROUTER_API_KEY, GEMINI_API_KEY, NVIDIA_API_KEY, ...). It is
# referenced with `optional: true`, so it may be absent.
# The agent-sandbox controller (v1.0.8, API v1beta1) must be installed.
apiVersion: extensions.agents.x-k8s.io/v1beta1
kind: SandboxTemplate
metadata:
name: dsh-web-template
namespace: dsh-demo
spec:
service: true # headless Service per Sandbox -> stable DNS (status.serviceFQDN)
# networkPolicyManagement is omitted, so the controller uses its Managed
# secure default (ingress only from the sandbox-router, public egress). This
# quickstart reaches the sandbox through `kubectl port-forward`, which is
# local to the pod. The multi-tenant example documents opt-in policies in
# examples/dsh-agents-as-a-service/docs/optional-hardening.md.
#
# OPTIONAL: recreate a Failed sandbox Pod instead of retaining it
# (agent-sandbox v1.0.8+). Documented with the multi-tenant example.
# failedPodPolicy: Recreate
podTemplate:
spec:
# OPTIONAL: run under gVisor. gVisor breaks `kubectl port-forward`, so
# this quickstart's only access path will not work; use the multi-tenant
# gateway example or a Service/NodePort instead.
# runtimeClassName: gvisor
#
# OPTIONAL: pin to a dedicated node pool. Replace the placeholder with a
# node label; add a matching toleration if the pool is tainted.
# nodeSelector:
# dsh.example.com/node-pool: dsh
securityContext:
runAsNonRoot: true
runAsUser: 1000 # Matches the UID of the 'node' user
runAsGroup: 1000
fsGroup: 1000
# DSH's credentials plugin refuses to start when .credentials.yaml has
# any group/other permission bit (it creates the file 0600). The
# default kubelet fsGroup policy ("Always") rewrites it to 0660 on every
# pod start, so a recreated or resumed sandbox crash-loops; OnRootMismatch
# avoids that walk once the volume root already carries the ownership.
# Rationale in full at
# examples/dsh-agents-as-a-service/manifests/10-sandboxtemplate.yaml.
fsGroupChangePolicy: OnRootMismatch
containers:
- name: dsh-web
image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.5-r6
imagePullPolicy: IfNotPresent
ports:
- name: web
containerPort: 3080
envFrom:
- secretRef:
name: dsh-provider-keys # Secret with key DEEPSEEK_API_KEY
# Optional extra provider keys; an absent Secret is safe.
- secretRef:
name: dsh-provider-keys-extra
optional: true
env:
# Point the OS home at the persisted agent workspace. DSH's
# directory picker (the folder explorer used to choose a workspace)
# lists $HOME when it opens and has no start-path setting, so with
# the image default (/home/node, not on a volume) it would open on a
# throwaway directory. /workspace is the PVC-backed WORKDIR, so the
# picker opens on the tree that survives suspend/resume, and `~` in
# agent commands resolves there. Trade-off: tools that keep caches
# or config under $HOME (npm/pnpm, uv, git, gh, DSH's global
# ~/.agents instructions and skills) write dotfiles onto the PVC
# inside the agent workspace. DSH_HOME is explicit
# (/home/node/.dsh), so sessions, config and credentials are
# unaffected. Rationale in full at
# examples/dsh-agents-as-a-service/manifests/10-sandboxtemplate.yaml.
- name: HOME
value: /workspace
# Empty => DSH derives links from the request Host. Keep it empty
# when fronting the sandbox with a proxy/gateway.
- name: DSH_PUBLIC_URL
value: ""
# DSH Host/Origin-fences /api. Empty trusts only localhost, which
# is right for `kubectl port-forward`. Set this to the public host
# (e.g. dsh.example.com) if you front DSH with a proxy or Ingress,
# so requests forwarded with the browser's Host are accepted. The
# full example documents both exposure variants in
# examples/dsh-agents-as-a-service/manifests/exposure/README.md.
- name: DSH_TRUSTED_HOSTS
value: ""
# DSH's own bash file sandbox (bubblewrap, else Landlock) cannot be
# enforced in this hardened pod - bubblewrap is installed but is
# unsetuid and needs an unprivileged user namespace - and fails
# closed with SANDBOX_UNAVAILABLE. The pod is the boundary here;
# keep the deployment override in sync with the multi-tenant example
# and see its README troubleshooting row to restore a confined mode.
- name: DSH_PERMISSION_MODE
value: danger-full-access
volumeMounts:
- name: dsh-home
mountPath: /home/node/.dsh # DSH_HOME
# The agent workspace (the image's WORKDIR) is the same PVC under
# its own subPath, so DSH state - including .credentials.yaml -
# never shows up as workspace data. Kubelet creates workspace/ on
# first mount with the volume's fsGroup permissions; no init
# container is needed. See the multi-tenant template for details.
- name: dsh-home
mountPath: /workspace
subPath: workspace
# / is behind DSH's launch-token cookie (401 without it); use an
# unauthenticated path for probes. The startup probe gives a cold
# start (plugin activation on a small node, or the first resume of a
# suspended sandbox) up to 120s before liveness can restart it.
startupProbe:
httpGet:
path: /favicon.svg
port: 3080
periodSeconds: 5
failureThreshold: 24
readinessProbe:
httpGet:
path: /favicon.svg
port: 3080
initialDelaySeconds: 5
periodSeconds: 5
livenessProbe:
httpGet:
path: /favicon.svg
port: 3080
initialDelaySeconds: 15
periodSeconds: 15
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
seccompProfile:
type: RuntimeDefault
volumeClaimTemplates:
- metadata:
name: dsh-home
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 5Gi
# storageClassName omitted => cluster default

67
scripts/check-examples.sh Executable file
View File

@@ -0,0 +1,67 @@
#!/usr/bin/env bash
# Enforce the example contract documented in docs/examples.md.
# Run by "make check" locally and by the CI lint job.
set -euo pipefail
cd "$(dirname "$0")/.."
index=examples/README.md
fail=0
err() { echo "check-examples: $*" >&2; fail=1; }
[ -f "$index" ] || err "missing $index"
for dir in examples/*/; do
name="$(basename "$dir")"
[ "$name" = "_template" ] && continue
[ -f "${dir}README.md" ] || err "$name: missing README.md"
[ -d "${dir}manifests" ] || err "$name: missing manifests/"
if [ -d "${dir}manifests" ]; then
count=$(find "${dir}manifests" -maxdepth 1 -type f -name '*.yaml' | wc -l | tr -d ' ')
[ "$count" -gt 0 ] || err "$name: manifests/ has no top-level *.yaml files"
for manifest in "${dir}"manifests/*.yaml; do
[ -e "$manifest" ] || continue
case "$(basename "$manifest")" in
[0-9][0-9]-*.yaml) ;;
*) err "$name: top-level manifest $(basename "$manifest") must be named NN-*.yaml" ;;
esac
done
fi
stray=$(find "$dir" -maxdepth 1 -type f -name '*.yaml' | wc -l | tr -d ' ')
[ "$stray" -eq 0 ] || err "$name: manifests belong under manifests/, not at the example root"
if [ -d "${dir}manifests/exposure" ] && [ ! -f "${dir}manifests/exposure/README.md" ]; then
err "$name: manifests/exposure/ requires a README.md describing the variants"
fi
if [ -d "${dir}manifests/optional" ] && [ ! -f "${dir}manifests/optional/README.md" ]; then
err "$name: manifests/optional/ requires a README.md describing the opt-in manifests"
fi
for component in "$dir"*/; do
[ -d "$component" ] || continue
case "$(basename "$component")" in
manifests|scripts|docs) continue ;;
esac
if [ -f "${component}Dockerfile" ] && [ ! -f "${component}README.md" ]; then
err "$name/$(basename "$component"): component with a Dockerfile needs a README.md"
fi
done
if ! grep -Fq "examples/$name/" "$index"; then
err "$name: not listed in examples/README.md"
fi
for script in "$dir"scripts/*.sh; do
[ -e "$script" ] || continue
[ -x "$script" ] || err "$script: must be executable"
done
done
if [ "$fail" -ne 0 ]; then
echo "check-examples: FAILED" >&2
exit 1
fi
echo "check-examples: ok"

View File

@@ -0,0 +1,47 @@
#!/usr/bin/env python3
"""Parse every example manifest and assert the minimal Kubernetes shape."""
from __future__ import annotations
import sys
from pathlib import Path
try:
import yaml
except ModuleNotFoundError: # pragma: no cover - environment guard
sys.exit("check-manifests: PyYAML is required (python -m pip install pyyaml)")
ROOT = Path(__file__).resolve().parent.parent
FILES = sorted(ROOT.glob("examples/*/manifests/**/*.yaml"))
if not FILES:
sys.exit("check-manifests: no manifests found under examples/*/manifests/")
errors: list[str] = []
documents = 0
for path in FILES:
relative = path.relative_to(ROOT)
try:
loaded = list(yaml.safe_load_all(path.read_text(encoding="utf-8")))
except yaml.YAMLError as exc:
errors.append(f"{relative}: invalid YAML: {exc}")
continue
for position, document in enumerate(loaded, start=1):
if document is None:
continue
documents += 1
if not isinstance(document, dict):
errors.append(f"{relative} document {position}: not a mapping")
continue
missing = [key for key in ("apiVersion", "kind", "metadata") if key not in document]
if missing:
errors.append(f"{relative} document {position}: missing {', '.join(missing)}")
continue
metadata = document["metadata"]
if not isinstance(metadata, dict) or not metadata.get("name"):
errors.append(f"{relative} document {position}: metadata.name is required")
if errors:
print("\n".join(errors), file=sys.stderr)
sys.exit(1)
print(f"check-manifests: ok ({len(FILES)} files, {documents} documents)")

108
scripts/check-versions.sh Executable file
View File

@@ -0,0 +1,108 @@
#!/usr/bin/env bash
# Verify that every version pin in the repository agrees with versions.env.
# Run by "make check" locally and by the CI lint job.
set -euo pipefail
cd "$(dirname "$0")/.."
# shellcheck source=/dev/null
. ./versions.env
IDENTITY="$DSH_VERSION"
if [ -n "${RELEASE_REVISION:-}" ]; then
IDENTITY="$DSH_VERSION-$RELEASE_REVISION"
fi
fail=0
err() { echo "check-versions: $*" >&2; fail=1; }
expect() { # <file> <literal>
if ! grep -Fq -- "$2" "$1"; then
err "$1: expected to contain: $2"
fi
}
# Main Dockerfile build args
expect Dockerfile "ARG DSH_VERSION=$DSH_VERSION"
expect Dockerfile "ARG DSH_COMMIT=$DSH_COMMIT"
expect Dockerfile "ARG PNPM_VERSION=$PNPM_VERSION"
expect Dockerfile "ARG RELEASE_REVISION=$RELEASE_REVISION"
expect Dockerfile "ARG NODE_BUILD_IMAGE=$NODE_BUILD_IMAGE"
expect Dockerfile "ARG NODE_RUNTIME_IMAGE=$NODE_RUNTIME_IMAGE"
# The build stage must fail closed when the tag no longer resolves to DSH_COMMIT.
expect Dockerfile "git -C /src rev-parse HEAD"
# Example gateway build base image
expect examples/dsh-agents-as-a-service/gateway/Dockerfile \
"ARG GATEWAY_PYTHON_IMAGE=$GATEWAY_PYTHON_IMAGE"
# Local configuration mirrored for humans
expect .env.sample "DSH_VERSION=$DSH_VERSION"
expect .env.sample "DSH_COMMIT=$DSH_COMMIT"
expect .env.sample "PNPM_VERSION=$PNPM_VERSION"
expect .env.sample "RELEASE_REVISION=$RELEASE_REVISION"
# Human-facing docs embed the resolved commit in the build example and table.
expect docs/main-image.md "$DSH_COMMIT"
expect docs/versioning.md "$DSH_COMMIT"
# Every container image is pinned to an immutable multi-arch index digest, not
# a mutable tag. This is a supply-chain invariant: don't "fix" a failing check
# by dropping the digest.
for image_var in NODE_BUILD_IMAGE NODE_RUNTIME_IMAGE GATEWAY_PYTHON_IMAGE OAUTH2_PROXY_IMAGE; do
image_value="${!image_var:-}"
if ! printf '%s' "$image_value" | grep -qE '@sha256:[0-9a-f]{64}$'; then
err "versions.env: $image_var must be digest-pinned (name:tag@sha256:<64 hex>), got: ${image_value:-<unset>}"
fi
done
if ! printf '%s' "${DSH_COMMIT:-}" | grep -qE '^[0-9a-f]{40}$'; then
err "versions.env: DSH_COMMIT must be a full 40-character commit SHA, got: ${DSH_COMMIT:-<unset>}"
fi
# Build entrypoints must pass DSH_COMMIT rather than trusting the tag alone.
expect Makefile "--build-arg DSH_COMMIT=\$(DSH_COMMIT)"
for wf in build-main.yaml release.yaml ci-image.yaml; do
expect ".github/workflows/$wf" "DSH_COMMIT="
done
# Image references in the example manifests
for f in \
examples/dsh-agents-as-a-service/manifests/10-sandboxtemplate.yaml \
examples/minimal-single-tenant/manifests/10-sandboxtemplate.yaml; do
expect "$f" "image: $GHCR_IMAGE:$IDENTITY"
done
expect examples/dsh-agents-as-a-service/manifests/50-gateway.yaml "image: $GATEWAY_IMAGE:$IDENTITY"
# The oauth2-proxy exposure variant pins the digest, not the mutable tag.
expect examples/dsh-agents-as-a-service/manifests/exposure/gateway-api-sso/72-oauth2-proxy.yaml \
"image: $OAUTH2_PROXY_IMAGE"
# The gateway image must live below the repository image (one package per
# example component, no unrelated top-level packages).
case "$GATEWAY_IMAGE" in
"$GHCR_IMAGE"/*) ;;
*) err "GATEWAY_IMAGE ($GATEWAY_IMAGE) must be nested below GHCR_IMAGE ($GHCR_IMAGE)" ;;
esac
# Build tooling must read versions.env instead of hard-coding a version.
# (scripts/ is excluded: set-version.sh documents usage examples in comments.)
while IFS= read -r hit; do
[ -n "$hit" ] || continue
err "hard-coded version outside versions.env: $hit"
done < <(grep -rnE 'dsh-v[0-9]' .github/workflows Makefile 2>/dev/null || true)
# Prose may mention versions, but only the current identity or its base tag.
while IFS= read -r token; do
[ -n "$token" ] || continue
case "$token" in
"$IDENTITY"|"$DSH_VERSION") ;;
*) err "stale version reference in docs: $token" ;;
esac
done < <(grep -rhoE 'dsh-v[0-9][A-Za-z0-9._-]*' \
--include='*.md' --exclude-dir=.git --exclude-dir=.venv --exclude-dir=.uvcache \
--exclude=CHANGELOG.md . | sort -u)
if [ "$fail" -ne 0 ]; then
echo "check-versions: FAILED" >&2
exit 1
fi
echo "check-versions: ok ($IDENTITY)"

107
scripts/set-version.sh Executable file
View File

@@ -0,0 +1,107 @@
#!/usr/bin/env bash
# Rewrite every version pin in one pass, then leave the tree ready for
# "make check".
#
# ./scripts/set-version.sh dsh-v0.2.2-rc.1 # new DSH ref, no revision
# ./scripts/set-version.sh dsh-v0.2.1-rc.5-r6 r3 # same ref, new repo revision
#
# For a new DSH ref the script resolves the tag's commit with "git ls-remote"
# (peeled ^{} commit first, tag ref as fallback) and rewrites DSH_COMMIT in
# versions.env, the Dockerfile ARG default and .env.sample. A revision-only bump
# at the same ref keeps the existing commit and needs no network access.
set -euo pipefail
cd "$(dirname "$0")/.."
new_version="${1:-}"
new_revision="${2:-}"
case "$new_version" in
dsh-v[0-9]*) ;;
*) echo "usage: $0 <dsh-vX.Y.Z> [rN]" >&2; exit 2 ;;
esac
if [ -n "$new_revision" ]; then
case "$new_revision" in
r[2-9]|r[1-9][0-9]*) ;;
*) echo "revision must look like r2, r3, ..." >&2; exit 2 ;;
esac
fi
# shellcheck source=/dev/null
. ./versions.env
old_version="$DSH_VERSION"
old_revision="${RELEASE_REVISION:-}"
old_commit="${DSH_COMMIT:-}"
old_identity="$old_version"
[ -z "$old_revision" ] || old_identity="$old_version-$old_revision"
new_identity="$new_version"
[ -z "$new_revision" ] || new_identity="$new_version-$new_revision"
if [ "$old_identity" = "$new_identity" ]; then
echo "set-version: already at $new_identity"
exit 0
fi
# Resolve the commit the new upstream tag points at. Annotated tags expose a
# peeled ^{} line (the commit) and a tag-object line; prefer the peeled commit
# and fall back to the tag ref for lightweight tags. A repository-only revision
# bump keeps the already pinned commit, so it needs no network.
new_commit="$old_commit"
if [ "$new_version" != "$old_version" ] || [ -z "$new_commit" ]; then
if ! refs="$(git ls-remote --tags --exit-code "$DSH_REPO" \
"refs/tags/$new_version^{}" "refs/tags/$new_version")"; then
echo "set-version: upstream tag '$new_version' not found in $DSH_REPO" >&2
exit 1
fi
new_commit="$(printf '%s\n' "$refs" \
| awk -v t="refs/tags/$new_version^{}" '$2 == t {print $1}' | head -n1)"
if [ -z "$new_commit" ]; then
new_commit="$(printf '%s\n' "$refs" \
| awk -v t="refs/tags/$new_version" '$2 == t {print $1}' | head -n1)"
fi
if ! printf '%s' "$new_commit" | grep -qE '^[0-9a-f]{40}$'; then
echo "set-version: could not resolve a commit for '$new_version' in $DSH_REPO" >&2
exit 1
fi
fi
# Replace the identity first (it contains the base version), then the base
# version itself. CHANGELOG.md is generated from tags, so it is left alone.
while IFS= read -r file; do
[ -f "$file" ] || continue
case "$file" in
.env.sample|Dockerfile|Makefile|*.md|*.yaml|*.yml|*.sh|*.py|*.toml|*.env) ;;
*) continue ;;
esac
sed -i "s/$old_identity/$new_identity/g; s/$old_version/$new_version/g" "$file"
done < <(git ls-files)
# versions.env is authoritative: write these keys explicitly so an empty
# revision is stored as an empty value.
sed -i "s/^DSH_VERSION=.*/DSH_VERSION=$new_version/" versions.env
sed -i "s/^RELEASE_REVISION=.*/RELEASE_REVISION=$new_revision/" versions.env
sed -i "s/^DSH_COMMIT=.*/DSH_COMMIT=$new_commit/" versions.env
# Values stored separately from the full identity: Dockerfile ARG defaults and
# the human-facing sample cannot be reached by the identity replacement above.
sed -i "s/^ARG DSH_VERSION=.*/ARG DSH_VERSION=$new_version/" Dockerfile
sed -i "s/^ARG RELEASE_REVISION=.*/ARG RELEASE_REVISION=$new_revision/" Dockerfile
sed -i "s/^ARG DSH_COMMIT=.*/ARG DSH_COMMIT=$new_commit/" Dockerfile
sed -i "s/^DSH_VERSION=.*/DSH_VERSION=$new_version/" .env.sample
sed -i "s/^RELEASE_REVISION=.*/RELEASE_REVISION=$new_revision/" .env.sample
sed -i "s/^DSH_COMMIT=.*/DSH_COMMIT=$new_commit/" .env.sample
# The human-facing build examples embed the full SHA, which the version
# replacement above cannot see. Keep them in step; check-versions.sh verifies.
if [ -n "$old_commit" ] && [ "$old_commit" != "$new_commit" ]; then
for doc in docs/main-image.md docs/versioning.md; do
[ -f "$doc" ] || continue
sed -i "s/$old_commit/$new_commit/g" "$doc"
done
fi
echo "set-version: $old_identity -> $new_identity"
echo "set-version: DSH_COMMIT=$new_commit"
echo "set-version: run 'make check' and review the diff"

44
versions.env Normal file
View File

@@ -0,0 +1,44 @@
# Single source of truth for every pinned version and image name in this
# repository. Consumed by the Makefile, build.sh, scripts/*.sh and the GitHub
# Actions workflows. Never hard-code these values elsewhere: run "make check"
# (scripts/check-versions.sh) to verify every pin.
#
# Every container image below is pinned to an immutable multi-arch index digest
# (name:tag@sha256:...), not just a mutable tag. Bump the tag and the digest
# together; scripts/check-versions.sh requires the digest form.
# Upstream DeepSeek Harness source, and the ref this repository packages.
# DSH_COMMIT is the full commit SHA that DSH_VERSION resolves to upstream; the
# Dockerfile verifies the clone's HEAD against it and fails the build on drift.
DSH_REPO=https://github.com/deepseek-ai/deepseek-harness.git
DSH_VERSION=dsh-v0.2.1-rc.5
DSH_COMMIT=b830ec950faba0a46dda564be3e2d854afed94f1
PNPM_VERSION=11.28.5
# Release identity = DSH_VERSION plus an optional repository revision.
# An empty RELEASE_REVISION is the first release at this DSH ref; rN (N >= 2)
# marks a repository-only re-release at the same DSH ref. This is the first
# release at dsh-v0.2.1-rc.5-r6, so the identity is the plain tag; a later
# repository-only change at the same ref becomes -r2.
RELEASE_REVISION=r6
# Base images for the main DSH Web image (root Dockerfile), pinned by digest.
NODE_BUILD_IMAGE=docker.io/node:24-trixie@sha256:8c761eca2bf7eec965f946cf47b8090143cb41f45f2f8699af0cc0298d2d2d19
NODE_RUNTIME_IMAGE=docker.io/node:24-trixie-slim@sha256:54f84b4cf8432b21aaffd5d3ce8d41e12cea0d10d33c09d2f3c9879d49d6b446
# Base image for the example gateway (examples/dsh-agents-as-a-service/gateway).
GATEWAY_PYTHON_IMAGE=docker.io/python:3.12-slim@sha256:aa943ce411423c7582288f18707ff66eddcbaeb510502d747a72aeb34f30d159
# Control-plane versions used by the examples and their manifests.
# The matching SHA-256s pin the release manifests that CI downloads and
# applies; update both the version and the digest when bumping either pin.
AGENT_SANDBOX_VERSION=v1.0.9
AGENT_SANDBOX_MANIFEST_SHA256=4bf9ee5558e9a663f91cfa5963e96f2591bfaa1dcf79f03bfab65bc5d65cc699
GATEWAY_API_VERSION=v1.2.1
GATEWAY_API_MANIFEST_SHA256=97598bf6ab3b33b9b5c5432bdd24de091e4e9c3aa0575ebb0710a2a19cd64d64
OAUTH2_PROXY_IMAGE=quay.io/oauth2-proxy/oauth2-proxy:v7.15.5@sha256:8498b0d0ef0a7b29686414000a08aee467f02d0299c9ed1e006a8f33fc017916
# Registry layout: the main image is the repository itself, and every example
# component is a nested package below it (<registry>/<owner>/<repo>/<example>/<component>).
GHCR_IMAGE=ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox
GATEWAY_IMAGE=ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway