1
0
mirror of https://github.com/sredevopsorg/deepseek-kubernetes-sandbox.git synced 2026-11-03 01:13:24 +00:00

docs: align readmes, docs and agent guidance with current behavior (#37)

Remove obsolete layout/migration prose and legacy gateway-alias references, reconcile the required-check policy, fix version-label/RBAC/workspace/channel drift, and de-duplicate the example and gateway docs.

Rename the minimal manifest to 10-sandboxtemplate.yaml and mount the dsh-home PVC's workspace subPath at /workspace so the documented persistence is real. Enforce numbered top-level manifests in check-examples.sh.

Verified with make check, yamllint, shellcheck, a Markdown link/anchor scan, and git diff --check.

Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
This commit is contained in:
SREDevOps.org Agent
2026-10-30 12:31:50 -03:00
committed by GitHub
parent 82ee0d0cdb
commit 24f23e09ae
27 changed files with 228 additions and 372 deletions

View File

@@ -126,13 +126,14 @@ Read `docs/examples.md` and the testing table in `AGENTS.md` before editing.
alternatives, never a stack. alternatives, never a stack.
- **New example:** follow `docs/examples.md`, add the index entry, keep scripts - **New example:** follow `docs/examples.md`, add the index entry, keep scripts
executable/idempotent/self-cleaning, run `make check`. executable/idempotent/self-cleaning, run `make check`.
- **Version bump:** edit `versions.env` only, update prerequisites, run - **Version bump:** use `make set-version`; never edit pins by hand. Update
`make check` and the e2e. Never mix a bump with an unrelated feature. prerequisites, run `make check` and the e2e on a cluster. Never mix a bump
with an unrelated feature.
- **MCP integration:** pin the build to the cluster's tag, least-privilege SA, - **MCP integration:** pin the build to the cluster's tag, least-privilege SA,
ClusterIP + auth proxy, raised DSH timeout, no secrets in the repo. ClusterIP + auth proxy, raised DSH timeout, no secrets in the repo.
- **Optional hardening:** gVisor, `nodeSelector`, NetworkPolicy and extra - **Optional hardening:** gVisor, `nodeSelector`, NetworkPolicy, extra
provider keys stay opt-in; the sandbox policy lives under provider keys and failed-Pod recovery stay opt-in; the sandbox policy lives
`manifests/optional/`. All four are documented in under `manifests/optional/`. All of them are documented in
`examples/dsh-agents-as-a-service/docs/optional-hardening.md`. `examples/dsh-agents-as-a-service/docs/optional-hardening.md`.
- **Docs:** update the affected doc in the same change. - **Docs:** update the affected doc in the same change.

View File

@@ -1,4 +1,6 @@
# Copy to .env and fill in. Never commit a real key. # Reference environment names for local runs. Nothing sources this file
# automatically: export the values or pass them with `docker run -e/--env-file`.
# Copy it to .env if you prefer --env-file. Never commit a real key.
# #
# The version values below are documentation only: versions.env is the single # The version values below are documentation only: versions.env is the single
# source of truth, and scripts/check-versions.sh fails if they drift. # source of truth, and scripts/check-versions.sh fails if they drift.
@@ -29,22 +31,7 @@ DSH_PORT=3080
# public hostname). localhost and 127.0.0.1 are always trusted. # public hostname). localhost and 127.0.0.1 are always trusted.
DSH_TRUSTED_HOSTS= DSH_TRUSTED_HOSTS=
# --- agents-as-a-service example --- # The example gateway's runtime settings (NAMESPACE, POOL, IDLE_TIMEOUT,
NAMESPACE=dsh-demo # WAKE_TIMEOUT, COOKIE_*, AUTH_PROXY_HEADER, ...) are documented in
POOL=dsh-pool # examples/dsh-agents-as-a-service/gateway/README.md and set in
# Seconds a sandbox may stay idle before the sweeper suspends it, and the max # examples/dsh-agents-as-a-service/manifests/50-gateway.yaml.
# time a proxied request waits for a suspended sandbox to become Ready.
IDLE_TIMEOUT=360
WAKE_TIMEOUT=240
DSH_CONTAINER=dsh-web
DSH_LOG_TAIL_LINES=2000
COOKIE_NAME=dsh_gw_session
# Set to true only behind HTTPS; secure cookies are not sent over plain HTTP
# (localhost counts as a secure context in browsers).
COOKIE_SECURE=false
COOKIE_MAX_AGE=604800
MAX_BODY_MB=32
HTTP_TIMEOUT=300
# SSO is opt-in. Set this only when the gateway Service is reachable
# exclusively through oauth2-proxy (see the example exposure docs).
# AUTH_PROXY_HEADER=X-Forwarded-User

View File

@@ -28,7 +28,7 @@ body:
id: image-tag id: image-tag
attributes: attributes:
label: Image or release label: Image or release
description: The pinned tag or digest, for example dsh-v0.2.1-rc.5. description: The tag or digest you deployed (from the manifest or release metadata).
- type: input - type: input
id: cluster id: cluster
attributes: attributes:

View File

@@ -19,8 +19,7 @@
## Checklist ## Checklist
- [ ] make check passes (versions, example layout, manifest YAML) - [ ] make check passes (versions, example layout, manifest YAML)
- [ ] make test passes (gateway unit tests) - [ ] make test / make lint pass when the change touches code (gateway unit tests, ruff, shellcheck; yamllint/actionlint where installed)
- [ ] make lint passes (ruff, shellcheck; yamllint/actionlint where installed)
- [ ] Docs updated for the change (root README, docs/, example README) - [ ] Docs updated for the change (root README, docs/, example README)
- [ ] No secrets or usable credentials committed - [ ] No secrets or usable credentials committed
- [ ] CHANGELOG.md was not edited by hand (generated by git-cliff) - [ ] CHANGELOG.md was not edited by hand (generated by git-cliff)

View File

@@ -1,7 +1,8 @@
# Manifest shape validation and server-side dry-run. Split from ci.yaml so this # Manifest shape validation and server-side dry-run. Split from ci.yaml so this
# job can use native on.pull_request.paths filtering and skip PRs that do not # job can use native on.pull_request.paths filtering and skip PRs that do not
# touch manifests, scripts or versions.env. The job name is the required # touch manifests, scripts or versions.env. The job name is `manifests`; because
# branch-protection check `manifests`. # it is path-filtered it is an on-demand check, not a required branch-protection
# check (see docs/gitflow.md).
name: CI (manifests) name: CI (manifests)
on: on:

View File

@@ -1,6 +1,7 @@
# Gateway unit tests and lint. Split from ci.yaml so this job can use native # Gateway unit tests and lint. Split from ci.yaml so this job can use native
# on.pull_request.paths filtering and skip PRs that do not touch the gateway. # on.pull_request.paths filtering and skip PRs that do not touch the gateway.
# The job name is the required branch-protection check `python`. # The job name is `python`; because it is path-filtered it is an on-demand check,
# not a required branch-protection check (see docs/gitflow.md).
name: CI (python) name: CI (python)
on: on:

View File

@@ -16,11 +16,6 @@ on:
description: Existing tag to release (must match versions.env) description: Existing tag to release (must match versions.env)
required: true required: true
type: string type: string
publish_legacy_gateway_alias:
description: Also publish the legacy :gateway alias (one release only)
required: false
default: true
type: boolean
permissions: permissions:
contents: write contents: write
@@ -193,28 +188,6 @@ jobs:
${{ needs.meta.outputs.gateway_image }}:${{ needs.meta.outputs.identity }} ${{ needs.meta.outputs.gateway_image }}:${{ needs.meta.outputs.identity }}
${{ needs.meta.outputs.gateway_image }}:latest ${{ needs.meta.outputs.gateway_image }}:latest
legacy-gateway-alias:
name: Publish legacy :gateway alias (temporary)
needs: [meta, build-gateway]
if: github.event_name != 'workflow_dispatch' || inputs.publish_legacy_gateway_alias
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- name: Login to GitHub Container Registry
uses: docker/login-action@81c6f93d4bc991e510b342a00b419b3bb4efc5c6 # v4.7.0
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Retag gateway as the legacy package tag
env:
SOURCE: ${{ needs.meta.outputs.gateway_image }}:${{ needs.meta.outputs.identity }}
LEGACY: ${{ needs.meta.outputs.ghcr_image }}:gateway
run: |
set -euo pipefail
docker buildx imagetools create -t "$LEGACY" "$SOURCE"
echo "Deprecated alias $LEGACY now points at $SOURCE" >> "$GITHUB_STEP_SUMMARY"
publish-release: publish-release:
name: Changelog and GitHub Release name: Changelog and GitHub Release
needs: [meta, build-main, build-gateway] needs: [meta, build-main, build-gateway]

View File

@@ -9,8 +9,8 @@ before changing anything; it encodes the conventions the CI enforces.
Harness Web (dsh web) as a slim, non-root, multi-arch image. Everything else Harness Web (dsh web) as a slim, non-root, multi-arch image. Everything else
exists to support it or to demonstrate an implementation on top of it. exists to support it or to demonstrate an implementation on top of it.
- **Every implementation is a self-contained example** under examples/<name>/. - **Every implementation is a self-contained example** under examples/<name>/.
There is exactly one example for the agents-as-a-service platform; do not The multi-tenant platform lives in exactly one directory; do not split it into
split it into parallel directories. parallel examples.
- **versions.env is the single source of truth** for the DSH ref, pnpm, the - **versions.env is the single source of truth** for the DSH ref, pnpm, the
control-plane versions and the image names. control-plane versions and the image names.
- **Docs describe the shipped behavior**, not intentions. If you change a - **Docs describe the shipped behavior**, not intentions. If you change a
@@ -43,7 +43,7 @@ before changing anything; it encodes the conventions the CI enforces.
| versions.env | Every pinned version and image name | | versions.env | Every pinned version and image name |
| Makefile | Canonical local entrypoint | | Makefile | Canonical local entrypoint |
| build.sh | Deprecated shim that forwards to make | | build.sh | Deprecated shim that forwards to make |
| docs/ | main-image.md, examples.md, gitflow.md, versioning.md | | docs/ | main-image.md, examples.md, gitflow.md, versioning.md (index: README.md) |
| examples/ | One implementation per directory (contract in docs/examples.md) | | examples/ | One implementation per directory (contract in docs/examples.md) |
| examples/dsh-agents-as-a-service/ | The multi-tenant platform example (manifests, gateway, docs, scripts) | | examples/dsh-agents-as-a-service/ | The multi-tenant platform example (manifests, gateway, docs, scripts) |
| examples/minimal-single-tenant/ | The just-one-DSH-Web quickstart | | examples/minimal-single-tenant/ | The just-one-DSH-Web quickstart |
@@ -54,7 +54,7 @@ before changing anything; it encodes the conventions the CI enforces.
## Commands ## Commands
make check # versions, example layout and manifest YAML (run before every commit) make check # versions, example layout and manifest YAML (run before every commit)
make test # gateway unit tests (109+ tests) make test # gateway unit tests (pytest)
make lint # ruff + shellcheck (+ yamllint/actionlint when installed) make lint # ruff + shellcheck (+ yamllint/actionlint when installed)
make image # build the main DSH Web image (slow: clones and builds DSH) make image # build the main DSH Web image (slow: clones and builds DSH)
make gateway # build the example gateway image (fast) make gateway # build the example gateway image (fast)
@@ -72,21 +72,18 @@ needs kind plus the agent-sandbox CRDs (see the example README).
| --- | --- | | --- | --- |
| Documentation only | make check; verify relative links | | Documentation only | make check; verify relative links |
| Example manifest | make check; kubectl apply --dry-run=server in CI | | Example manifest | make check; kubectl apply --dry-run=server in CI |
| Optional hardening (gVisor, nodeSelector, NetworkPolicy, provider keys) | make check; server dry-run in CI; manual verification on a capable cluster | | Optional hardening (gVisor, nodeSelector, NetworkPolicy, provider keys, failed-Pod recovery) | make check; server dry-run in CI; manual verification on a capable cluster |
| Gateway code | make test; make lint; add or update tests | | Gateway code | make test; make lint; add or update tests |
| Dockerfile / versions.env | make check; the CI `build-amd64` job (`ci-image.yaml`) builds and smoke-tests the image on the PR; run `make image` locally only when you need the image | | Dockerfile / versions.env | make check; the CI `build-amd64` job (`ci-image.yaml`) builds and smoke-tests the image on the PR; run `make image` locally only when you need the image |
| Workflow | actionlint in CI; keep actions SHA-pinned; no hard-coded versions | | Workflow | actionlint in CI; keep actions SHA-pinned; no hard-coded versions |
| New example | make check; all files described in docs/examples.md | | New example | make check; all files described in docs/examples.md |
| Project skill (.dsh/skills/) | make check; frontmatter and size check; shellcheck skill scripts; load it with the skill tool | | Project skill (.dsh/skills/) | make check; validate frontmatter; keep SKILL.md under ~8 KB; shellcheck skill scripts; load it with the skill tool |
## Versioning and releases ## Versioning and releases
- Release identity = DSH_VERSION, optionally with a repository revision suffix - Release identity = DSH_VERSION plus an optional -rN repository revision.
for re-releases at the same DSH ref (current identity: The current identity is dsh-v0.2.1-rc.5 (first release at this DSH ref, so no
dsh-v0.2.1-rc.5). suffix); a repository-only change at the same ref publishes -r2.
- The current identity is dsh-v0.2.1-rc.5 (stored in versions.env). It is the
first release at this DSH ref, so it carries no revision suffix; a
repository-only change at the same ref publishes -r2.
- Channels: release tags are immutable; latest follows the newest release; - Channels: release tags are immutable; latest follows the newest release;
main is the development channel for both image packages. main is the development channel for both image packages.
- A broken release is fixed by publishing -r(N+1), never by re-tagging. - A broken release is fixed by publishing -r(N+1), never by re-tagging.
@@ -99,7 +96,9 @@ needs kind plus the agent-sandbox CRDs (see the example README).
- Branch names: feat/, fix/, docs/, chore/, refactor/, test/. - Branch names: feat/, fix/, docs/, chore/, refactor/, test/.
- Conventional Commit subjects and PR titles (feat(scope): ..., fix(scope): ...). - Conventional Commit subjects and PR titles (feat(scope): ..., fix(scope): ...).
- PRs are squash-merged; main stays linear and protected. - PRs are squash-merged; main stays linear and protected.
- Required checks: lint, python, manifests, pr-title, CodeQL, dependency-review. - Required on every PR: lint, pr-title, CodeQL (both Analyze jobs) and
dependency-review. python, manifests and build-amd64 are path-filtered
on-demand checks; run them when the change touches their scope.
- Keep diffs scoped; update docs with the code; never mix a version bump with - Keep diffs scoped; update docs with the code; never mix a version bump with
unrelated changes. unrelated changes.
@@ -107,9 +106,9 @@ needs kind plus the agent-sandbox CRDs (see the example README).
1. Read docs/examples.md and examples/README.md. 1. Read docs/examples.md and examples/README.md.
2. Create examples/<name>/ with README.md, manifests/NN-*.yaml, optional 2. Create examples/<name>/ with README.md, manifests/NN-*.yaml, optional
manifests/exposure/<variant>/ (plus README), optional components with their manifests/optional/<feature>/ or manifests/exposure/<variant>/ (each with a
own Dockerfile/README/tests, and scripts/ (executable, idempotent, README), optional components with their own Dockerfile/README/tests, and
self-cleaning). scripts/ (executable, idempotent, self-cleaning).
3. Publish component images under <registry>/<owner>/<repo>/<example>/<component>. 3. Publish component images under <registry>/<owner>/<repo>/<example>/<component>.
4. Add the example to examples/README.md; scripts/check-examples.sh enforces 4. Add the example to examples/README.md; scripts/check-examples.sh enforces
the layout. the layout.
@@ -124,8 +123,9 @@ needs kind plus the agent-sandbox CRDs (see the example README).
- The example shares provider keys (dsh-provider-keys plus the optional - The example shares provider keys (dsh-provider-keys plus the optional
dsh-provider-keys-extra) across sandboxes; do not describe it as dsh-provider-keys-extra) across sandboxes; do not describe it as
tenant-isolated. tenant-isolated.
- gVisor, nodeSelector and the NetworkPolicies are opt-in and are not active in - gVisor, nodeSelector, the NetworkPolicies, extra provider keys and failed-Pod
the base manifests; do not describe the base example as hardened. recovery are opt-in and are not active in the base manifests; do not describe
the base example as hardened.
- TLS terminates outside the gateway; set COOKIE_SECURE=true for HTTPS and keep - TLS terminates outside the gateway; set COOKIE_SECURE=true for HTTPS and keep
the base manifest safe for local HTTP. the base manifest safe for local HTTP.

View File

@@ -43,8 +43,10 @@ Install the gateway dev dependencies once:
- Use a Conventional Commit title, for example feat(gateway): add wake metrics - Use a Conventional Commit title, for example feat(gateway): add wake metrics
or docs(examples): clarify SSO trust boundary. PR titles are linted. or docs(examples): clarify SSO trust boundary. PR titles are linted.
- Keep the diff scoped. Squash merge into main; history stays linear. - Keep the diff scoped. Squash merge into main; history stays linear.
- Required checks: lint, python, manifests, pr-title, CodeQL, and - Required on every PR: lint, pr-title, CodeQL (both Analyze jobs) and
dependency-review. Branch protection is described in docs/gitflow.md. dependency-review. python, manifests and build-amd64 are path-filtered
on-demand checks; run them when the change touches their scope. Branch
protection is described in docs/gitflow.md.
- Update the docs that the change affects; docs are part of the change, not a - Update the docs that the change affects; docs are part of the change, not a
follow-up. follow-up.
- Never commit a real secret. Secret-shaped YAML belongs in a *.yaml.example - Never commit a real secret. Secret-shaped YAML belongs in a *.yaml.example
@@ -60,16 +62,12 @@ it by hand; the release workflow regenerates it.
## Adding an example ## Adding an example
Read docs/examples.md, then add examples/<name>/ with: Read docs/examples.md, then add examples/<name>/ with a README.md,
numbered manifests/NN-*.yaml, optional manifests/optional/<feature>/ or
- README.md (what it demonstrates, prerequisites, steps, teardown) manifests/exposure/<variant>/ (each with a README), optional components (a
- manifests/NN-*.yaml (numbered, plain kubectl) Dockerfile plus README and tests), and executable, idempotent, self-cleaning
- optional manifests/exposure/<variant>/ with its own README.md scripts. Add the literal examples/<name>/ entry to examples/README.md;
- optional components (a Dockerfile plus README and tests per component) scripts/check-examples.sh enforces the layout.
- scripts/ with executable, idempotent, self-cleaning scripts
- docs/ for deeper material
List it in examples/README.md. scripts/check-examples.sh enforces this layout.
## Documentation ## Documentation

View File

@@ -96,10 +96,7 @@ ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway
| --- | --- | --- | | --- | --- | --- |
| `dsh-v0.2.1-rc.5` | DSH Web, port `3080` | **Current release identity.** Immutable: once published it never moves. | | `dsh-v0.2.1-rc.5` | DSH Web, port `3080` | **Current release identity.** Immutable: once published it never moves. |
| `latest` | DSH Web | Most recent release; same digest as the current identity. | | `latest` | DSH Web | Most recent release; same digest as the current identity. |
| `main` | DSH Web | Development channel rebuilt from every push to `main`. Not a release; not for production. | | `main` | DSH Web | Development channel rebuilt when image-definition files change on `main`. Not a release; not for production. |
Tags published before the release-identity scheme remain on GHCR but are not
updated; new deployments use the current identity.
The example gateway is published under its nested path with the same release The example gateway is published under its nested path with the same release
identity, e.g. identity, e.g.
@@ -134,8 +131,8 @@ Self-contained implementations that consume the image. Index and contract:
| Example | What it demonstrates | | Example | What it demonstrates |
| --- | --- | | --- | --- |
| [`examples/dsh-agents-as-a-service/`](examples/dsh-agents-as-a-service/) | Full multi-tenant platform: SandboxTemplate with per-user PVCs, warm pool, one claim per user, suspend-when-idle, wake-on-return, browser gateway with signup + `HttpOnly` sessions + HTTP/WebSocket proxy, Ingress or Gateway API + GitHub SSO exposure, and opt-in gVisor/NetworkPolicy/provider-key hardening. Start with the [walkthrough](examples/dsh-agents-as-a-service/README.md) and the [architecture](examples/dsh-agents-as-a-service/docs/architecture.md). | | [`examples/dsh-agents-as-a-service/`](examples/dsh-agents-as-a-service/) | Full multi-tenant platform: SandboxTemplate with per-user PVCs, warm pool, one claim per user, suspend-when-idle, wake-on-return, browser gateway with signup + `HttpOnly` sessions + HTTP/WebSocket proxy, Ingress or Gateway API + GitHub SSO exposure, and opt-in gVisor/NetworkPolicy/provider-key hardening and failed-Pod recovery. Start with the [walkthrough](examples/dsh-agents-as-a-service/README.md) and the [architecture](examples/dsh-agents-as-a-service/docs/architecture.md). |
| [`examples/minimal-single-tenant/`](examples/minimal-single-tenant/) | The smallest useful setup: one `SandboxTemplate` and one `SandboxClaim`, reached through `kubectl port-forward` + the launch-token URL. Manifest: [`manifests/sandbox-template.yaml`](examples/minimal-single-tenant/manifests/sandbox-template.yaml). | | [`examples/minimal-single-tenant/`](examples/minimal-single-tenant/) | The smallest useful setup: one `SandboxTemplate` and one `SandboxClaim`, reached through `kubectl port-forward` + the launch-token URL. Manifest: [`manifests/10-sandboxtemplate.yaml`](examples/minimal-single-tenant/manifests/10-sandboxtemplate.yaml). |
The gateway is a component of the multi-tenant example and has its own The gateway is a component of the multi-tenant example and has its own
[README](examples/dsh-agents-as-a-service/gateway/README.md) (API, configuration, [README](examples/dsh-agents-as-a-service/gateway/README.md) (API, configuration,
@@ -166,7 +163,7 @@ guide and troubleshooting notes live beside `SKILL.md`.
| [`scripts/`](scripts/) | Repository checks: versions, example layout, manifest YAML, version bump. | | [`scripts/`](scripts/) | Repository checks: versions, example layout, manifest YAML, version bump. |
| [`.dsh/skills/`](.dsh/skills/) | Project-local DSH skills, auto-discovered from the git root. | | [`.dsh/skills/`](.dsh/skills/) | Project-local DSH skills, auto-discovered from the git root. |
| [`.github/`](.github/) | CI, release automation, dependency updates, PR/issue templates. | | [`.github/`](.github/) | CI, release automation, dependency updates, PR/issue templates. |
| [`.env.sample`](.env.sample) | Documented local configuration (copy to `.env`; never commit it). | | [`.env.sample`](.env.sample) | Reference environment names for local runs; nothing reads it automatically. Never commit a real key. |
## Development ## Development
@@ -200,24 +197,6 @@ Conventional Commits and is never edited by hand. See
[`CHANGELOG.md`](CHANGELOG.md), [docs/versioning.md](docs/versioning.md) and [`CHANGELOG.md`](CHANGELOG.md), [docs/versioning.md](docs/versioning.md) and
[cliff.toml](cliff.toml). [cliff.toml](cliff.toml).
## Migrating from the previous layout
If you are coming from an older checkout, these paths moved. Nothing else changed
semantically; the content was split into the example it belongs to.
| Old path | New path |
| --- | --- |
| `gateway-api/dsh-demo/` (Gateway API + oauth2-proxy / GitHub SSO) | `examples/dsh-agents-as-a-service/manifests/exposure/gateway-api-sso/` (now `70-gateway.yaml` ... `74-tls-secret.yaml.example`) |
| `examples/dsh-agents-as-a-service/60-ingress.yaml` | `examples/dsh-agents-as-a-service/manifests/exposure/ingress/60-ingress.yaml` |
| `examples/dsh-agents-as-a-service/NN-*.yaml` (`00-` ... `50-`) | `examples/dsh-agents-as-a-service/manifests/NN-*.yaml` |
| `examples/dsh-agents-as-a-service/run-test-kind.sh` | `examples/dsh-agents-as-a-service/scripts/e2e-kind.sh` |
| `deploy/k8s/sandbox-template.yaml` | `examples/minimal-single-tenant/manifests/sandbox-template.yaml` |
| `docs/architecture.md` | `examples/dsh-agents-as-a-service/docs/architecture.md` |
The SSO walkthrough that lived in `gateway-api/dsh-demo/README.md` is merged
into the [multi-tenant example README](examples/dsh-agents-as-a-service/README.md)
(step 9) and the [exposure README](examples/dsh-agents-as-a-service/manifests/exposure/README.md).
## Contributing ## Contributing
- [CONTRIBUTING.md](CONTRIBUTING.md) — local setup, PR workflow, what a change needs. - [CONTRIBUTING.md](CONTRIBUTING.md) — local setup, PR workflow, what a change needs.

View File

@@ -21,7 +21,7 @@ disclosure once a fix is available.
from pod logs (pods/log). Anyone with access to those logs can impersonate a from pod logs (pods/log). Anyone with access to those logs can impersonate a
tenant. Keep the gateway's RBAC namespace-scoped and treat pod logs as tenant. Keep the gateway's RBAC namespace-scoped and treat pod logs as
sensitive. Replacing log scraping with an upstream token API is tracked in sensitive. Replacing log scraping with an upstream token API is tracked in
the example architecture document. the [example architecture document](examples/dsh-agents-as-a-service/docs/architecture.md#upstream-feature-request-remove-log-scraping).
- **SSO header trust boundary.** AUTH_PROXY_HEADER makes the gateway trust an - **SSO header trust boundary.** AUTH_PROXY_HEADER makes the gateway trust an
identity header (for example X-Forwarded-User). Enable it only when the identity header (for example X-Forwarded-User). Enable it only when the
gateway Service is reachable exclusively through the authenticating proxy; gateway Service is reachable exclusively through the authenticating proxy;

View File

@@ -44,6 +44,6 @@ commit_parsers = [
] ]
protect_breaking_commits = false protect_breaking_commits = false
filter_commits = false filter_commits = false
# Release tags are dsh-vX.Y.Z[-rN]; the single legacy pre-scheme tag is v0.0.1. # Release tags are dsh-vX.Y.Z[-rN]; the single pre-scheme tag is v0.0.1.
tag_pattern = "^(v|dsh-v).*$" tag_pattern = "^(v|dsh-v).*$"
sort_commits = "oldest" sort_commits = "oldest"

View File

@@ -2,8 +2,7 @@
Repository-wide documentation lives here. Example-specific material lives with Repository-wide documentation lives here. Example-specific material lives with
its example under [`examples/<name>/docs/`](../examples/) — see the its example under [`examples/<name>/docs/`](../examples/) — see the
[agents-as-a-service architecture](../examples/dsh-agents-as-a-service/docs/architecture.md) [agents-as-a-service architecture](../examples/dsh-agents-as-a-service/docs/architecture.md).
for the deep-dive that used to live in this directory.
If you are new, read the [root README](../README.md) first: it is image-first and If you are new, read the [root README](../README.md) first: it is image-first and
gets a DSH Web container running before anything else. gets a DSH Web container running before anything else.
@@ -34,6 +33,8 @@ DSH discovers project skills from `<git root>/.dsh/skills/<name>/SKILL.md`
| [dsh-agents-as-a-service README](../examples/dsh-agents-as-a-service/README.md) | The multi-tenant platform walkthrough: warm pool, claims, suspend/resume, gateway, exposure and troubleshooting. | | [dsh-agents-as-a-service README](../examples/dsh-agents-as-a-service/README.md) | The multi-tenant platform walkthrough: warm pool, claims, suspend/resume, gateway, exposure and troubleshooting. |
| [dsh-agents-as-a-service architecture](../examples/dsh-agents-as-a-service/docs/architecture.md) | Container/sequence diagrams, ADRs, security and RBAC, failure modes and the evolution path. | | [dsh-agents-as-a-service architecture](../examples/dsh-agents-as-a-service/docs/architecture.md) | Container/sequence diagrams, ADRs, security and RBAC, failure modes and the evolution path. |
| [gateway README](../examples/dsh-agents-as-a-service/gateway/README.md) | The gateway's HTTP/WebSocket API, configuration, auth (including SSO), and limitations. | | [gateway README](../examples/dsh-agents-as-a-service/gateway/README.md) | The gateway's HTTP/WebSocket API, configuration, auth (including SSO), and limitations. |
| [optional hardening](../examples/dsh-agents-as-a-service/docs/optional-hardening.md) | Opt-in gVisor, nodeSelector, NetworkPolicy, provider keys and failed-Pod recovery. |
| [exposure variants](../examples/dsh-agents-as-a-service/manifests/exposure/README.md) | The mutually exclusive Ingress and Gateway API + SSO exposure paths. |
## Project policies ## Project policies

View File

@@ -41,6 +41,7 @@ examples/<name>/
example directory; example directory;
- an example has no `README.md` or no `manifests/` directory; - an example has no `README.md` or no `manifests/` directory;
- `manifests/` contains no top-level `*.yaml` files; - `manifests/` contains no top-level `*.yaml` files;
- a top-level `manifests/*.yaml` is not named `NN-*.yaml`;
- a `*.yaml` file sits at the example root instead of under `manifests/`; - a `*.yaml` file sits at the example root instead of under `manifests/`;
- `manifests/exposure/` exists without a `manifests/exposure/README.md` describing - `manifests/exposure/` exists without a `manifests/exposure/README.md` describing
the variants; the variants;

View File

@@ -30,7 +30,7 @@ branch; the PR is squash-merged, so individual branch commits are never part of
`docs(scope): ...`, `build: ...`, `ci: ...`, `refactor: ...`, `test: ...`, `docs(scope): ...`, `build: ...`, `ci: ...`, `refactor: ...`, `test: ...`,
`chore: ...`. Breaking changes use `feat(scope)!: ...` or a `chore: ...`. Breaking changes use `feat(scope)!: ...` or a
`BREAKING CHANGE:` footer. `BREAKING CHANGE:` footer.
3. **CI must be green.** All required checks below must pass before merge. 3. **CI must be green.** All applicable checks below must pass before merge.
4. **Squash merge only.** `main` stays linear and protected; merged PRs are one 4. **Squash merge only.** `main` stays linear and protected; merged PRs are one
commit each. commit each.
5. **No direct pushes to `main`** or to a release branch. Both are protected. 5. **No direct pushes to `main`** or to a release branch. Both are protected.
@@ -40,21 +40,20 @@ branch; the PR is squash-merged, so individual branch commits are never part of
## Required checks ## Required checks
| Check | Workflow / job | What it verifies | | Check | Workflow / job | Required on every PR? | What it verifies |
| --- | --- | --- | | --- | --- | --- | --- |
| `lint` | `ci.yaml` | `scripts/check-versions.sh` (version pins and markdown tokens), `scripts/check-examples.sh` (example layout), shellcheck, actionlint, yamllint, hadolint and documentation links. | | `lint` | `ci.yaml` | yes | `scripts/check-versions.sh` (version pins and markdown tokens), `scripts/check-examples.sh` (example layout and numbered manifests), shellcheck, actionlint, yamllint, hadolint and documentation links. |
| `python` | `ci-python.yaml` | Gateway ruff check/format and unit tests (`pytest` with coverage). Runs only when the gateway or this workflow changes. | | `python` | `ci-python.yaml` | no (path-filtered) | Gateway ruff check/format and unit tests (`pytest` with coverage). Runs only when the gateway or this workflow changes. |
| `manifests` | `ci-manifests.yaml` | `scripts/check-manifests.py` (manifest shape) and a server-side dry-run of every example manifest against the agent-sandbox CRDs. Runs only when manifests, check scripts or `versions.env` change. | | `manifests` | `ci-manifests.yaml` | no (path-filtered) | `scripts/check-manifests.py` (manifest shape) and a server-side dry-run of every example manifest against the agent-sandbox CRDs. Runs when manifest files, `scripts/check-manifests.py` or `versions.env` change. |
| `pr-title` | `pr-title.yaml` | Conventional Commit PR title. | | `pr-title` | `pr-title.yaml` | yes | Conventional Commit PR title. |
| `CodeQL` | `codeql.yml` | Static analysis for Python and GitHub Actions. | | `Analyze (python)`, `Analyze (actions)` | `codeql.yml` | yes (both jobs) | CodeQL static analysis for Python and GitHub Actions. |
| `dependency-review` | `ci.yaml` | Dependency changes introduced by the PR (job `dependency-review`; runs on pull requests only). | | `dependency-review` | `ci.yaml` | yes | Dependency changes introduced by the PR (job `dependency-review`; runs on pull requests only). |
`python`, `manifests` and `build-amd64` use native `on.pull_request.paths` `python`, `manifests` and `build-amd64` use native `on.pull_request.paths`
filtering, so PRs that do not touch the gateway, the manifests or the image filtering, so PRs that do not touch the gateway, the manifests or the image
definition skip them. The gateway image build and `/healthz` are no longer definition skip them. The gateway image is built and published by
smoke-tested in a dedicated job: `/healthz` is covered by the gateway unit tests, `build-gateway.yaml` on merge; `/healthz` is covered by the gateway unit tests and
the Dockerfile is linted by `lint` (hadolint), and the multi-arch image build is the Dockerfile is linted by `lint` (hadolint).
published by `build-gateway.yaml` on merge.
`ci-image.yaml` (`build-amd64`) is the PR-side check for the main image: when `ci-image.yaml` (`build-amd64`) is the PR-side check for the main image: when
`Dockerfile`, `.dockerignore` or `versions.env` change it builds `linux/amd64` `Dockerfile`, `.dockerignore` or `versions.env` change it builds `linux/amd64`
@@ -64,9 +63,9 @@ the result (`--version`, non-root UID, `/workspace` + `DSH_HOME`, the
container). The published multi-arch build still happens only in container). The published multi-arch build still happens only in
`build-main.yaml` on merge. `build-main.yaml` on merge.
Because a skipped path-filtered check is not a passing check, only the always-on Because a skipped path-filtered check is not a passing check, require only the
checks (`lint`, `pr-title`, `CodeQL`, `dependency-review`) should be enforced as always-on set: `lint`, `pr-title`, both CodeQL `Analyze` jobs and
required status checks; `python`, `manifests` and `build-amd64` are on-demand `dependency-review`. Keep `python`, `manifests` and `build-amd64` as on-demand
checks that skip when unrelated to the change. checks that skip when unrelated to the change.
The image build workflows (`_build-image.yaml`, `build-main.yaml`, The image build workflows (`_build-image.yaml`, `build-main.yaml`,
@@ -109,7 +108,8 @@ Rollback is also a re-release, never a retag: see
## Tag protection ## Tag protection
All release tags match `dsh-*` and are protected: Release tags follow `dsh-*` and are protected (the pre-scheme `v0.0.1` tag is
historical and unmaintained):
- only maintainers (the release workflow) may create them; - only maintainers (the release workflow) may create them;
- they cannot be deleted or force-updated; - they cannot be deleted or force-updated;
@@ -123,11 +123,12 @@ respectively.
Configure these on the repository before relying on the workflow: Configure these on the repository before relying on the workflow:
- [ ] **Branches → main**: require a pull request; require the checks `lint`, - [ ] **Branches → main**: require a pull request; require the always-on checks
`python`, `manifests`, `pr-title`, `CodeQL`, `dependency-review`; require `lint`, `pr-title`, both CodeQL `Analyze` jobs and `dependency-review`
branches to be up to date; require conversation resolution; restrict direct (`python`, `manifests` and `build-amd64` are path-filtered and must not be
pushes; allow only squash merges and enable "Automatically delete head required); require branches to be up to date; require conversation
branches". resolution; restrict direct pushes; allow only squash merges and enable
"Automatically delete head branches".
- [ ] **Branches → release/***: same protections, plus allow the release workflow - [ ] **Branches → release/***: same protections, plus allow the release workflow
to push tags. to push tags.
- [ ] **Tags → dsh-***: restrict creation/update/deletion to maintainers; a tag - [ ] **Tags → dsh-***: restrict creation/update/deletion to maintainers; a tag

View File

@@ -47,8 +47,8 @@ docker build \
| Arg | Default / source | Purpose | | Arg | Default / source | Purpose |
| --- | --- | --- | | --- | --- | --- |
| `DSH_REPO` | `https://github.com/deepseek-ai/deepseek-harness.git` | Upstream DSH repository to clone. | | `DSH_REPO` | `https://github.com/deepseek-ai/deepseek-harness.git` | Upstream DSH repository to clone. |
| `DSH_VERSION` | `DSH_VERSION` in [versions.env](../versions.env) | Upstream DSH git tag/ref to package; recorded as `org.opencontainers.image.version`. | | `DSH_VERSION` | `DSH_VERSION` in [versions.env](../versions.env) | Upstream DSH git tag/ref to package. The Dockerfile records it as `io.sredevopsorg.dsh.version` and (for local builds) `org.opencontainers.image.version`; published builds set the version label to the full release identity. |
| `RELEASE_REVISION` | `RELEASE_REVISION` in [versions.env](../versions.env) | Repository revision suffix (`rN`); the release identity is `DSH_VERSION-RELEASE_REVISION`, recorded in the `io.sredevopsorg.release.revision` label. See [versioning](versioning.md). | | `RELEASE_REVISION` | `RELEASE_REVISION` in [versions.env](../versions.env) | Repository revision suffix (`rN`), recorded in `io.sredevopsorg.release.revision`. The release identity is `DSH_VERSION` plus `-RELEASE_REVISION` when set. See [versioning](versioning.md). |
| `PNPM_VERSION` | `PNPM_VERSION` in [versions.env](../versions.env) | pnpm version activated with corepack in both stages. | | `PNPM_VERSION` | `PNPM_VERSION` in [versions.env](../versions.env) | pnpm version activated with corepack in both stages. |
| `VCS_REF` | `git rev-parse --short HEAD` (via `make image`) | Source revision recorded for traceability. | | `VCS_REF` | `git rev-parse --short HEAD` (via `make image`) | Source revision recorded for traceability. |
| `NODE_BUILD_IMAGE` | `docker.io/node:24-trixie` | Build-stage base image. | | `NODE_BUILD_IMAGE` | `docker.io/node:24-trixie` | Build-stage base image. |
@@ -90,9 +90,10 @@ includes `/app/node_modules/.bin`.
the picker's browse backend lists `$HOME` when the dialog opens and labels that the picker's browse backend lists `$HOME` when the dialog opens and labels that
listing *Home*. With the image default (`/home/node`) that listing is outside a listing *Home*. With the image default (`/home/node`) that listing is outside a
mounted volume; the example sandbox templates mounted volume; the example sandbox templates
([minimal](../examples/minimal-single-tenant/manifests/sandbox-template.yaml), ([minimal](../examples/minimal-single-tenant/manifests/10-sandboxtemplate.yaml),
[agents as a service](../examples/dsh-agents-as-a-service/manifests/10-sandboxtemplate.yaml)) [agents as a service](../examples/dsh-agents-as-a-service/manifests/10-sandboxtemplate.yaml))
set `HOME=/workspace` so the picker opens on the persisted agent workspace. `~` mount the same PVC's `workspace` subPath at `/workspace` and set `HOME=/workspace`
so the picker opens on the persisted agent workspace. `~`
and every tool that keeps caches or config under `$HOME` (`npm`/`pnpm`, `uv`, and every tool that keeps caches or config under `$HOME` (`npm`/`pnpm`, `uv`,
`git`, `gh`, DSH's global `~/.agents` instructions and skills) follow, so those `git`, `gh`, DSH's global `~/.agents` instructions and skills) follow, so those
dotfiles then live on the workspace volume too. `DSH_HOME` is separate and stays dotfiles then live on the workspace volume too. `DSH_HOME` is separate and stays
@@ -141,7 +142,7 @@ procedure.
| --- | --- | --- | | --- | --- | --- |
| Exposed port | `3080/tcp` | `EXPOSE 3080`; one port serves the UI, assets, API, SSE and WebSocket traffic. | | Exposed port | `3080/tcp` | `EXPOSE 3080`; one port serves the UI, assets, API, SSE and WebSocket traffic. |
| State volume | `/home/node/.dsh` | Declared with `VOLUME`; use a named volume (Docker) or PVC (Kubernetes). | | State volume | `/home/node/.dsh` | Declared with `VOLUME`; use a named volume (Docker) or PVC (Kubernetes). |
| Working directory | `/workspace` | The agent workspace. Mount a volume here if agents should keep files; the agents-as-a-service example mounts the same PVC's `workspace` subPath here and points `HOME` at it, so DSH's folder picker opens on it. | | Working directory | `/workspace` | The agent workspace. Mount a volume here if agents should keep files; the shipped example templates mount the same PVC's `workspace` subPath here and point `HOME` at it, so DSH's folder picker opens on it. |
| Stop signal | `SIGTERM` | `tini` forwards it to `dsh web`; allow a graceful shutdown period. | | Stop signal | `SIGTERM` | `tini` forwards it to `dsh web`; allow a graceful shutdown period. |
| User | `node`, UID/GID `1000` | The process is non-root. A PVC mounted at `DSH_HOME` must be writable by UID 1000. | | User | `node`, UID/GID `1000` | The process is non-root. A PVC mounted at `DSH_HOME` must be writable by UID 1000. |
@@ -164,8 +165,8 @@ livenessProbe:
periodSeconds: 15 periodSeconds: 15
``` ```
The shipped templates use exactly this shape; see The shipped templates use the same path and startup budget; see
[`examples/minimal-single-tenant/manifests/sandbox-template.yaml`](../examples/minimal-single-tenant/manifests/sandbox-template.yaml). [`examples/minimal-single-tenant/manifests/10-sandboxtemplate.yaml`](../examples/minimal-single-tenant/manifests/10-sandboxtemplate.yaml).
A `startupProbe` is important because plugin activation (or the first start of a A `startupProbe` is important because plugin activation (or the first start of a
large sandbox) can exceed a default liveness budget. large sandbox) can exceed a default liveness budget.
@@ -199,10 +200,7 @@ GitHub Container Registry:
| --- | --- | | --- | --- |
| `dsh-v0.2.1-rc.5` | Current release identity. **Immutable**: once published it never moves. | | `dsh-v0.2.1-rc.5` | Current release identity. **Immutable**: once published it never moves. |
| `latest` | Most recent release; points at the same digest as the current identity. | | `latest` | Most recent release; points at the same digest as the current identity. |
| `main` | Development channel rebuilt from every push to `main`; not a release, not for production. | | `main` | Development channel rebuilt when image-definition files change on `main`; not a release, not for production. |
Tags published before the release-identity scheme remain published but are not
updated; new deployments use the current identity.
Release identities, channels and rollback rules are defined in Release identities, channels and rollback rules are defined in
[versioning](versioning.md). Tags are immutable, so production deployments should [versioning](versioning.md). Tags are immutable, so production deployments should
@@ -220,6 +218,7 @@ docker buildx imagetools inspect ghcr.io/sredevopsorg/deepseek-kubernetes-sandbo
docker inspect ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.5 \ docker inspect ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.5 \
--format '{{json .Config.Labels}}' --format '{{json .Config.Labels}}'
# org.opencontainers.image.title / .version / .source # org.opencontainers.image.title / .version / .source
# io.sredevopsorg.dsh.version / io.sredevopsorg.release.revision
``` ```
Release builds also publish SBOM and provenance attestations from the release Release builds also publish SBOM and provenance attestations from the release

View File

@@ -15,7 +15,7 @@ release identity = DSH_VERSION + optional "-" + RELEASE_REVISION
| Piece | Meaning | | Piece | Meaning |
| --- | --- | | --- | --- |
| `DSH_VERSION` | The upstream DeepSeek Harness git ref being packaged, e.g. `dsh-v0.2.1-rc.5`. It mirrors upstream's own tag, which is what "mirrored-DSH scheme" means. | | `DSH_VERSION` | The upstream DeepSeek Harness git ref being packaged, e.g. `dsh-v0.2.1-rc.5`. It mirrors upstream's own tag. |
| `RELEASE_REVISION` | An optional repository revision at that DSH ref: empty for the first release, `rN` (`N >= 2`) for a repository-only re-release. | | `RELEASE_REVISION` | An optional repository revision at that DSH ref: empty for the first release, `rN` (`N >= 2`) for a repository-only re-release. |
Examples: Examples:
@@ -33,23 +33,14 @@ with an empty revision); a repository-only change at the same DSH ref increments
## Why release identities are immutable ## Why release identities are immutable
**Container tags are immutable** in this project: re-pushing a tag would silently **Container tags are immutable** in this project: re-pushing a tag would silently
change what an existing deployment runs. A DSH upgrade therefore produces a new change what an existing deployment runs. A fix therefore publishes a new identity;
base identity with an empty revision; a repository-only change at an see the rules above.
already-released ref publishes `-rN`, where `r2` is the first re-release (the
plain tag is conceptually revision 1 and is taken).
The current identity is `dsh-v0.2.1-rc.5`, the first release at this upstream The current identity is `dsh-v0.2.1-rc.5`, the first release at this upstream
ref, so it carries no revision suffix. The identity is stored in ref, so it carries no revision suffix. The identity is stored in
[versions.env](../versions.env) and consumed everywhere: the Dockerfile label, [versions.env](../versions.env) and consumed everywhere: the Dockerfile labels
the example manifests, the Makefile and the release workflow. Tags published (version and revision), the example manifests, the Makefile and the release
before the release-identity scheme remain published but are not updated; new workflow.
deployments use the current identity.
The gateway image moved from the repository-level `:gateway` tag to a nested
package (`<repository>/dsh-agents-as-a-service/gateway`). The release workflow
also republishes the legacy `<repository>:gateway` tag as a deprecated
compatibility alias; it may be removed in a later release, so deployments
should pin the nested package.
## Channels ## Channels
@@ -57,7 +48,7 @@ should pin the nested package.
| --- | --- | --- | | --- | --- | --- |
| Release | `dsh-v0.2.1-rc.5` | The current immutable release identity. What production should pin (preferably by digest). | | Release | `dsh-v0.2.1-rc.5` | The current immutable release identity. What production should pin (preferably by digest). |
| `latest` | `latest` | Alias for the most recent release; moves only when a new release is published. | | `latest` | `latest` | Alias for the most recent release; moves only when a new release is published. |
| `main` | `main` | Development channel, rebuilt from every push to `main`. Never treat it as a release; it exists so `main` can be tested. | | `main` | `main` | Development channel, rebuilt when image or gateway sources change on `main`. Never treat it as a release; it exists so `main` can be tested. |
The same channels exist for every published package, including the nested The same channels exist for every published package, including the nested
example component image. example component image.
@@ -110,11 +101,12 @@ workflows with a literal version.
matches [versions.env](../versions.env). matches [versions.env](../versions.env).
6. The workflow builds and pushes multi-arch images for the main image and the 6. The workflow builds and pushes multi-arch images for the main image and the
example components (the identity tag plus `latest`), creates the GitHub example components (the identity tag plus `latest`), creates the GitHub
Release from the git-cliff notes, records the published digests, and opens a Release from the git-cliff notes, and opens a `chore(changelog)` PR that
`chore(changelog)` PR that refreshes `CHANGELOG.md` on `main`. refreshes `CHANGELOG.md` on `main`. The publish log shows each image manifest
and digest.
7. Merge the changelog PR. Verify: 7. Merge the changelog PR. Verify:
`docker buildx imagetools inspect <image>:<identity>` shows both platforms, `docker buildx imagetools inspect <image>:<identity>` shows both platforms;
and the GitHub Release references the tag and the digest. the GitHub Release references the tag, and the workflow log shows the digest.
8. Announce the identity and the digest; deployments pin the digest. 8. Announce the identity and the digest; deployments pin the digest.
## Rollback ## Rollback

View File

@@ -11,7 +11,7 @@ reference material, not libraries.
| [`examples/minimal-single-tenant/`](minimal-single-tenant/) | The smallest useful setup: a single `SandboxTemplate` and one `SandboxClaim`, no gateway — reach DSH Web with a launch-token URL through `kubectl port-forward`. | agent-sandbox with extensions. | | [`examples/minimal-single-tenant/`](minimal-single-tenant/) | The smallest useful setup: a single `SandboxTemplate` and one `SandboxClaim`, no gateway — reach DSH Web with a launch-token URL through `kubectl port-forward`. | agent-sandbox with extensions. |
The multi-tenant example also documents opt-in gVisor, `nodeSelector`, The multi-tenant example also documents opt-in gVisor, `nodeSelector`,
NetworkPolicy and extra provider-key hardening in NetworkPolicy, extra provider-key hardening and failed-Pod recovery in
[optional hardening](dsh-agents-as-a-service/docs/optional-hardening.md). [optional hardening](dsh-agents-as-a-service/docs/optional-hardening.md).
## The example contract (short version) ## The example contract (short version)
@@ -19,37 +19,26 @@ NetworkPolicy and extra provider-key hardening in
Full rules: [docs/examples.md](../docs/examples.md). `scripts/check-examples.sh` Full rules: [docs/examples.md](../docs/examples.md). `scripts/check-examples.sh`
enforces the layout, and `make check` runs it. enforces the layout, and `make check` runs it.
``` - `README.md` is the walkthrough; `manifests/NN-*.yaml` is applied with plain
examples/<name>/ kubectl and read in order.
+-- README.md # walkthrough: prerequisites, steps, troubleshooting, cleanup - Optional `manifests/optional/<feature>/` and `manifests/exposure/<variant>/`
+-- manifests/ # NN-*.yaml, applied with plain kubectl each ship a README; exposure variants are mutually exclusive.
| +-- optional/<feature>/ # optional opt-in manifests... - Optional `scripts/` must be executable, idempotent and self-cleaning; `docs/`
| +-- optional/README.md # ...required whenever optional/ exists holds deeper material.
| +-- exposure/<variant>/ # optional per-variant manifests... - Components with a Dockerfile need a README and tests; images are published
| +-- exposure/README.md # ...required whenever exposure/ exists nested below the main image (`<registry>/<owner>/<repo>/<example>/<component>`).
+-- scripts/ # optional executable, idempotent, self-cleaning *.sh - Secrets are `*.yaml.example` with placeholders; every example is listed above
+-- docs/ # optional architecture/ADR material with its literal `examples/<name>/` path.
+-- <component>/ # optional component with Dockerfile + README + tests
```
- Secrets never ship populated: use `*.yaml.example` files with placeholders.
- Manifests are numbered (`00-`, `10-`, ...) and read as an ordered walkthrough.
- Component images are published nested below the main image:
`<registry>/<owner>/<repo>/<example>/<component>`.
- Every example must be listed above with its literal `examples/<name>/` path.
- Scripts under `scripts/` must be executable, idempotent and clean up after
themselves.
## Running an example ## Running an example
```bash Each example README has the exact prerequisites, commands, optional hardening,
# e.g. the multi-tenant platform example exposure options and teardown. The multi-tenant example also has a full kind
kubectl apply -f examples/dsh-agents-as-a-service/manifests/ walkthrough that cleans up after itself:
# or run the whole walkthrough (kind + agent-sandbox required) and let it clean up ```bash
examples/dsh-agents-as-a-service/scripts/e2e-kind.sh examples/dsh-agents-as-a-service/scripts/e2e-kind.sh
``` ```
`kubectl apply -f manifests/` is non-recursive: `optional/` and `exposure/` `kubectl apply -f manifests/` is non-recursive: `optional/` and `exposure/`
subdirectories are applied explicitly. Each example README has the exact subdirectories are applied explicitly.
prerequisites, optional hardening, exposure options and teardown.

View File

@@ -8,6 +8,8 @@ personal, long-lived DSH Web instance that is **provisioned in ~2 seconds from a
warm pool, suspended when idle to save cost, and woken transparently when the warm pool, suspended when idle to save cost, and woken transparently when the
user returns** — with all state (conversations, sessions, config, and the agent user returns** — with all state (conversations, sessions, config, and the agent
workspace) surviving on the per-user PVC at `/home/node/.dsh` + `/workspace`. workspace) surviving on the per-user PVC at `/home/node/.dsh` + `/workspace`.
Suspend-when-idle plus the warm pool keep cost proportional to use: a return
after absence pays a short cold-start pause, not an always-on instance.
It ships in three layers: It ships in three layers:
@@ -79,12 +81,14 @@ What it demonstrates, feature by feature:
## Prerequisites ## Prerequisites
Run the walkthrough from this example's directory (`examples/dsh-agents-as-a-service/`).
- A Kubernetes cluster with a default StorageClass (kind and GKE both work as-is). - A Kubernetes cluster with a default StorageClass (kind and GKE both work as-is).
- agent-sandbox with extensions installed: - agent-sandbox with extensions installed:
```bash ```bash
VERSION=v1.0.8 # AGENT_SANDBOX_VERSION in versions.env . ../../versions.env
kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/download/$VERSION/sandbox-with-extensions.yaml kubectl apply -f "https://github.com/kubernetes-sigs/agent-sandbox/releases/download/${AGENT_SANDBOX_VERSION}/sandbox-with-extensions.yaml"
kubectl -n agent-sandbox-system rollout status deployment --timeout=180s kubectl -n agent-sandbox-system rollout status deployment --timeout=180s
``` ```
@@ -243,12 +247,9 @@ kubectl -n dsh-demo exec "$SB" -- cat /home/node/.dsh/marker.txt # "remember m
kubectl -n dsh-demo exec "$SB" -- cat /workspace/marker.txt # "remember me too" kubectl -n dsh-demo exec "$SB" -- cat /workspace/marker.txt # "remember me too"
``` ```
> **Why resume works:** the pod template sets `fsGroupChangePolicy: OnRootMismatch`. > **Why resume works:** the pod template sets `fsGroupChangePolicy: OnRootMismatch`;
> DSH requires its `.credentials.yaml` to be `0600`, while the kubelet's default > without it the kubelet widens `.credentials.yaml` to `0660` and DSH crash-loops.
> fsGroup policy widens every file to `0660` on each pod start — a resumed sandbox > See [Troubleshooting](#troubleshooting) for the one-time fix on older sandboxes.
> would crash-loop. See [Troubleshooting](#troubleshooting) for the one-time fix
> on sandboxes created before this setting existed. The `/workspace` mount does not
> change that: `subPath` is a separate directory, and only the volume root is walked.
## 7. Injection policy rejects env-carrying claims ## 7. Injection policy rejects env-carrying claims
@@ -273,7 +274,7 @@ kind load docker-image "$GW" --name <kind-cluster>
# ... or build it locally and tag it exactly as manifests/50-gateway.yaml # ... or build it locally and tag it exactly as manifests/50-gateway.yaml
# expects (make gateway builds dsh-aaas-gateway:dev): # expects (make gateway builds dsh-aaas-gateway:dev):
make gateway make -C ../.. gateway
docker tag dsh-aaas-gateway:dev "$GW" docker tag dsh-aaas-gateway:dev "$GW"
kind load docker-image "$GW" --name <kind-cluster> kind load docker-image "$GW" --name <kind-cluster>
@@ -351,44 +352,21 @@ API clients that prefer bearer tokens can use `POST /users` with a JSON body
(`{"user":"bob"}`) and the returned one-time token, `POST /login` to exchange a (`{"user":"bob"}`) and the returned one-time token, `POST /login` to exchange a
token for a cookie, and `Authorization: Bearer <token>` on subsequent calls. token for a cookie, and `Authorization: Bearer <token>` on subsequent calls.
### Gateway API ### Gateway routes
| Route | Purpose | The browser flow above uses the gateway's main routes. The full route table,
| --- | --- | status codes, configuration and failure matrix live in the
| `GET /healthz` | Liveness/readiness; no cluster access | [gateway README](gateway/README.md).
| `GET /` | Landing page: signup, token login, logout. With SSO it auto-provisions and connects instead. |
| `POST /users` | Signup. JSON or form; `201` (JSON), `202` (waking), `303` (browser redirect), `409` if user exists; sets the session cookie |
| `POST /login` | Exchange a bearer token for the session cookie |
| `POST /logout` | Clear the session cookie |
| `POST /recreate` | SSO only: delete the authenticated user's claim (wiping workspace data) and provision a fresh one; `303` to `/u/{user}/`. Cookie-authenticated calls must be same-origin. |
| `GET /users/{user}` | Derived state (`Ready`/`Waking`/`Suspended`) |
| `DELETE /users/{user}` | Delete claim (cascades sandbox + PVC) |
| `GET /u/{user}` | `308` redirect to `/u/{user}/` |
| `GET /u/{user}/**` | HTTP/SSE proxy to the user's DSH Web |
| `WS /u/{user}/**` | WebSocket proxy to the user's DSH Web |
The full API reference, configuration table, auth semantics and failure matrix
live in the [gateway README](gateway/README.md).
### How the DSH handshake works ### How the DSH handshake works
DSH does not expose an API for its launch token; it prints a URL containing DSH prints its launch token only in pod logs. The gateway resolves
`?token=…` in the pod logs. For each `(user, pod UID, Host)` the gateway: claim → sandbox → pod, reads the pod-log tail, performs the `?token=` handshake
under the browser's `Host`, caches `dsh-auth-*` per `(user, pod UID, Host)`, and
1. resolves the claim → sandbox → pod (annotation `agents.x-k8s.io/pod-name`, or refreshes once on a `401`. That is why its RBAC includes `pods` `get`/`list` and
the sandbox name as pod name), `pods/log` `get`; it is also the most fragile part of the design (see
2. reads the pod's logs (last `DSH_LOG_TAIL_LINES` lines) and parses the most [Limitations](#limitations) and the
recent `dsh web: …?token=…` line, [architecture ADR](docs/architecture.md#adr-3-server-side-launch-token-handshake-instead-of-exposing-the-token)).
3. requests `http://<serviceFQDN>:3080/?token=…` with the browser's `Host`,
4. captures DSH's `dsh-auth-*` `Set-Cookie` response and caches it,
5. injects that cookie (and strips the browser's `Cookie`/`Authorization` and any
`?token=`) on every proxied HTTP and WebSocket request,
6. refreshes the handshake once if DSH answers `401`, and drops the cached cookie
when the sandbox suspends or is deleted.
This is why the gateway's RBAC includes `pods` `get`/`list` and `pods/log`
`get`. It is also the most fragile part of the design; see
[Limitations](#limitations).
### DSH_TRUSTED_HOSTS ### DSH_TRUSTED_HOSTS
@@ -407,27 +385,18 @@ The same applies to a local `docker run`: the printed token URL uses
### WebSocket ### WebSocket
The gateway is a Starlette ASGI app and proxies WebSocket upgrades with a The gateway proxies WebSocket upgrades with a bidirectional pump, so the DSH web
bidirectional pump, so the DSH web GUI is fully interactive. Requirements: GUI is fully interactive. The entry point must pass `Upgrade`/`Connection`
- the ingress/load balancer in front of the gateway must pass `Upgrade`/`Connection`
headers; [`manifests/exposure/ingress/60-ingress.yaml`](manifests/exposure/ingress/60-ingress.yaml) headers; [`manifests/exposure/ingress/60-ingress.yaml`](manifests/exposure/ingress/60-ingress.yaml)
is a starting point (Traefik handles upgrades by default), is a starting point (Traefik handles upgrades by default). Idle suspension is
- idle suspension is disabled for users with in-flight proxied requests or open disabled while a proxied request or socket is in flight, so a long-running turn is
WebSockets, so a long-running chat stream is not cut off mid-turn, not cut off.
- the upstream upgrade dials the sandbox **pod IP**, forwards no `Host`/`Origin`,
and performs the launch-token handshake under that same `pod-IP:3080` authority. The upstream upgrade dials the sandbox **pod IP**, forwards no `Host`/`Origin`,
DSH fences on `Host` (only its bind address, loopback, or an allow-listed host and performs the launch-token handshake under the same `pod-IP:3080` authority.
passes) and binds its auth cookie to the request authority Missing or cross-origin auth closes with `4401`; upstream bridging failures close
(`payload.authority === authority`), while `websockets` derives `Host` from the with `1011`. See the [gateway README](gateway/README.md#proxy-failure-semantics)
URI and cannot be overridden. A browser-host cookie or forwarded origin makes for the full matrix.
the upgrade fail (`401`/`403`, surfaced as close `1011`), which the client
retries forever as `[connection] connection lost, retry #N`,
- missing/invalid gateway auth is rejected right after the upgrade with close code
`4401` (WebSocket auth has no HTTP `401` path); cross-origin or Origin-less
upgrades with the cookie are rejected the same way (CSWSH protection), so
clients must send a matching `Origin`; upstream bridging failures close with
`1011`.
### Path routing ### Path routing
@@ -457,27 +426,18 @@ and the gateway. They are part of the variant; see the
### Option A — plain Ingress (gateway's own session auth) ### Option A — plain Ingress (gateway's own session auth)
```bash 1. Set the Ingress host/TLS (or drop the `tls:` block), and set
# Replace the dsh.example.com host and TLS Secret (or drop the tls: block for `DSH_TRUSTED_HOSTS` in `manifests/10-sandboxtemplate.yaml` to the same public
# plain HTTP), then: host; re-apply the template.
kubectl apply -f manifests/exposure/ingress/ 2. Set `COOKIE_SECURE=true` once the Ingress serves HTTPS:
```
1. Set `DSH_TRUSTED_HOSTS` in `manifests/10-sandboxtemplate.yaml` to the same
public host (the gateway forwards the browser's `Host`) and re-apply the
template.
2. Set `COOKIE_SECURE=true` on the gateway once the Ingress serves HTTPS:
`kubectl -n dsh-demo set env deploy/dsh-gateway COOKIE_SECURE=true`. `kubectl -n dsh-demo set env deploy/dsh-gateway COOKIE_SECURE=true`.
(The base manifest keeps `false` so the plain-HTTP port-forward demo works.) 3. `kubectl apply -f manifests/exposure/ingress/`; verify
3. Verify: `curl -sI https://<host>/healthz` returns `200`; sign up in a browser `curl -sI https://<host>/healthz` returns `200`.
at `https://<host>/`.
4. `manifests/exposure/ingress/61-networkpolicy.yaml` is part of this variant:
it allows only Traefik pods to reach the gateway on `8080`. Adjust its
selector if your ingress controller uses a different pod label or
`hostNetwork`; if kubelet probes fail, see the hardening doc.
Keep the router free of buffering/compression middlewares: they break WebSocket Keep the router free of buffering/compression middlewares: they break WebSocket
upgrades and streaming. upgrades and streaming. The variant's `61-networkpolicy.yaml` allows only Traefik
pods to reach the gateway; see the [exposure README](manifests/exposure/README.md)
for selector and probe caveats.
### Option B — Gateway API + GitHub SSO (oauth2-proxy) ### Option B — Gateway API + GitHub SSO (oauth2-proxy)
@@ -487,84 +447,41 @@ front of **oauth2-proxy**, which authenticates the browser against GitHub
`X-Forwarded-User` set. The gateway trusts that header only when `X-Forwarded-User` set. The gateway trusts that header only when
`AUTH_PROXY_HEADER` is set (SSO is **opt-in**). `AUTH_PROXY_HEADER` is set (SSO is **opt-in**).
Prerequisites: 1. Install Gateway API CRDs and a Gateway implementation; the manifests assume
`gatewayClassName: traefik` (the CRD version is `GATEWAY_API_VERSION` in
- Gateway API CRDs and a Gateway implementation; the manifests assume `versions.env`).
`gatewayClassName: traefik` (the CRD version is pinned as 2. Create a GitHub OAuth App whose homepage/callback match the public FQDN, and
`GATEWAY_API_VERSION` in `versions.env`). create the oauth2-proxy and TLS Secrets from
- A GitHub OAuth App whose homepage/callback match the public FQDN `73-oauth2-proxy-secret.yaml.example` and `74-tls-secret.yaml.example`.
(`https://<host>/` and `https://<host>/oauth2/callback`). 3. Replace `dsh-demo.example.com` and `your-github-org`, set
- The oauth2-proxy credentials Secret (template: `DSH_TRUSTED_HOSTS` in the template, and enable SSO:
`73-oauth2-proxy-secret.yaml.example`) and a TLS Secret for the Gateway `kubectl -n dsh-demo set env deploy/dsh-gateway AUTH_PROXY_HEADER=X-Forwarded-User COOKIE_SECURE=true`.
listener (template: `74-tls-secret.yaml.example`). 4. `kubectl apply -f manifests/exposure/gateway-api-sso/`; verify the Gateway,
HTTPRoute, oauth2-proxy Deployment and endpoints as shown in the
Steps: [exposure README](manifests/exposure/README.md).
```bash
# 1. Real Secrets; the committed *.yaml.example files are templates only.
kubectl -n dsh-demo create secret generic oauth2-proxy \
--from-literal=client-id='<GitHub OAuth App client ID>' \
--from-literal=client-secret='<GitHub OAuth App client secret>' \
--from-literal=cookie-secret="$(python3 -c 'import secrets; print(secrets.token_urlsafe(32))')"
kubectl -n dsh-demo create secret tls dsh-demo-tls --cert=tls.crt --key=tls.key
# 2. Replace dsh-demo.example.com (70/71/72) and your-github-org (72).
# 3. Enable SSO on the gateway and trust the public host in the template.
kubectl -n dsh-demo set env deploy/dsh-gateway \
AUTH_PROXY_HEADER=X-Forwarded-User COOKIE_SECURE=true
# ... and set DSH_TRUSTED_HOSTS=dsh-demo.example.com in
# manifests/10-sandboxtemplate.yaml, then re-apply the template.
# 4. Apply the variant.
kubectl apply -f manifests/exposure/gateway-api-sso/
```
Verification:
```bash
kubectl get gateway dsh-gateway -n dsh-demo -o yaml # Programmed=True
kubectl get httproute -n dsh-demo -o wide # backend oauth2-proxy:4180, Accepted
kubectl get deploy oauth2-proxy -n dsh-demo # 1/1 Ready
kubectl get endpoints oauth2-proxy -n dsh-demo # backend present
kubectl -n dsh-demo logs deploy/oauth2-proxy --tail=50 # no config errors
```
Browser check: open `https://<host>/` — you are redirected to GitHub, and after
signing in as a member of the allowed organization the gateway auto-provisions
your sandbox (the first page can show `Provisioning`) and connects you. A
non-member is denied before reaching the gateway. GitHub logins are lowercase
DNS-1123 labels, so they map 1:1 onto the gateway's `user` validation. The SSO
landing page also offers **Recreate sandbox** (`POST /recreate`), which deletes
the claim — wiping the workspace PVC — and provisions a fresh one.
Sign-out: the gateway `/logout` clears `dsh_gw_session`; oauth2-proxy's session
can be cleared at `/oauth2/sign_out`.
> **SSO trust boundary — read this before enabling it.** After SSO is on, the > **SSO trust boundary — read this before enabling it.** After SSO is on, the
> `dsh-gateway` Service must be reachable **only** through oauth2-proxy (it is > `dsh-gateway` Service must be reachable **only** through oauth2-proxy (it is
> ClusterIP). The variant's `75-networkpolicies.yaml` restricts pod traffic > ClusterIP). The variant's `75-networkpolicies.yaml` restricts pod traffic
> accordingly: the gateway accepts traffic only from oauth2-proxy pods, > accordingly: the gateway accepts traffic only from oauth2-proxy pods,
> oauth2-proxy only from the Gateway implementation, and oauth2-proxy egress is > oauth2-proxy only from the Gateway implementation, and its egress only to DNS,
> limited to DNS, public HTTPS and the gateway. Anyone who can reach > public HTTPS and the gateway. Anyone who can reach `dsh-gateway:8080` directly
> `dsh-gateway:8080` directly can still send `X-Forwarded-User: <victim>` and be > can send `X-Forwarded-User: <victim>`, so never add a second path. To turn SSO
> treated as that user, so never add a second path (the Ingress variant, a > off, remove the variable
> shared-host port-forward, a permissive policy). To turn SSO off, remove the > (`kubectl -n dsh-demo set env deploy/dsh-gateway AUTH_PROXY_HEADER-`) and
> variable (`kubectl -n dsh-demo set env deploy/dsh-gateway AUTH_PROXY_HEADER-`) > delete the SSO variant.
> and delete the SSO variant; the gateway falls back to bearer-token/session
> auth.
### Switching variants ### Switching variants
Delete the variant you had before applying the other one Delete the variant you had before applying the other one, revert
(`kubectl delete -f manifests/exposure/ingress/`), revert
`AUTH_PROXY_HEADER`/`COOKIE_SECURE`, and verify with `AUTH_PROXY_HEADER`/`COOKIE_SECURE`, and verify with
`kubectl -n dsh-demo get ingress,gateway,httproute` that exactly one path to the `kubectl -n dsh-demo get ingress,gateway,httproute` that exactly one path to the
gateway remains. gateway remains. Full commands and verification are in the
[exposure README](manifests/exposure/README.md).
## Optional hardening ## Optional hardening
The base walkthrough is deliberately portable. Four opt-in isolation layers are The base walkthrough is deliberately portable. Five opt-in settings are
documented in [optional hardening](docs/optional-hardening.md): documented in [optional hardening](docs/optional-hardening.md):
- **gVisor** — uncomment `runtimeClassName: gvisor` before applying, or patch an - **gVisor** — uncomment `runtimeClassName: gvisor` before applying, or patch an
@@ -582,7 +499,7 @@ documented in [optional hardening](docs/optional-hardening.md):
the sandbox at `Finished=True`. Requires agent-sandbox v1.0.8+. the sandbox at `Finished=True`. Requires agent-sandbox v1.0.8+.
Apply each independently and re-run the walkthrough. `kubectl apply -f manifests/` Apply each independently and re-run the walkthrough. `kubectl apply -f manifests/`
is non-recursive, so the optional manifest is not applied by the base step. is non-recursive, so the optional manifests are not applied by the base step.
## Troubleshooting ## Troubleshooting
@@ -599,7 +516,7 @@ is non-recursive, so the optional manifest is not applied by the base step.
| `manifest.webmanifest` returns `401` in the console | Browsers fetch the Web App Manifest without credentials, so the gateway's session cookie is absent and the request fails closed. The UI is unaffected; only the PWA install prompt is lost | | `manifest.webmanifest` returns `401` in the console | Browsers fetch the Web App Manifest without credentials, so the gateway's session cookie is absent and the request fails closed. The UI is unaffected; only the PWA install prompt is lost |
| Signup is slow / `202` then `Waking` | Warm pool exhausted (`kubectl get sandboxes -l agents.x-k8s.io/warm-pool-sandbox`); scale the pool | | Signup is slow / `202` then `Waking` | Warm pool exhausted (`kubectl get sandboxes -l agents.x-k8s.io/warm-pool-sandbox`); scale the pool |
| `403` from the Kubernetes API in gateway logs | The gateway ServiceAccount is missing claim/sandbox/pod-log verbs (see `manifests/50-gateway.yaml`) | | `403` from the Kubernetes API in gateway logs | The gateway ServiceAccount is missing claim/sandbox/pod-log verbs (see `manifests/50-gateway.yaml`) |
| Gateway pod `ErrImagePull` / `ImagePullBackOff` (especially on kind) | The manifest uses `imagePullPolicy: IfNotPresent` with the release identity. Build with `make gateway`, retag to exactly the manifest's image, `kind load docker-image <that-ref>`, or set the Deployment image to your local tag / pinned digest. On a real cluster, make sure the GHCR package is readable (public or a pull secret) | | Gateway pod `ErrImagePull` / `ImagePullBackOff` (especially on kind) | The manifest uses `imagePullPolicy: IfNotPresent` with the release identity. Build with `make -C ../.. gateway`, retag to exactly the manifest's image, `kind load docker-image <that-ref>`, or set the Deployment image to your local tag / pinned digest. On a real cluster, make sure the GHCR package is readable (public or a pull secret) |
| Step 3: `pod "$SB" not found` after deploying the gateway | The claim was suspended by the idle sweeper. Claims created by the gateway carry `aaas.example.com/managed-by=dsh-gateway`; hand-applied claims (steps 1–7) do not and are never swept. Resume the sandbox: `kubectl -n dsh-demo patch sandbox "$SB" --type merge -p '{"spec":{"operatingMode":"Running"}}'` | | Step 3: `pod "$SB" not found` after deploying the gateway | The claim was suspended by the idle sweeper. Claims created by the gateway carry `aaas.example.com/managed-by=dsh-gateway`; hand-applied claims (steps 1–7) do not and are never swept. Resume the sandbox: `kubectl -n dsh-demo patch sandbox "$SB" --type merge -p '{"spec":{"operatingMode":"Running"}}'` |
| Resume crash-loops: `credentials-local: … is readable beyond its owner (mode 660)` | The pod template lacks `fsGroupChangePolicy: OnRootMismatch`, so the kubelet's default fsGroup policy rewrote `.credentials.yaml` to `0660`; DSH requires `0600` and refuses to start. New installs get the policy from the template. For an existing sandbox: suspend it, run a throwaway pod on `dsh-home-$SB` to `chmod 600 /home/node/.dsh/.credentials.yaml`, patch the sandbox's `podTemplate` with the policy, then resume | | Resume crash-loops: `credentials-local: … is readable beyond its owner (mode 660)` | The pod template lacks `fsGroupChangePolicy: OnRootMismatch`, so the kubelet's default fsGroup policy rewrote `.credentials.yaml` to `0660`; DSH requires `0600` and refuses to start. New installs get the policy from the template. For an existing sandbox: suspend it, run a throwaway pod on `dsh-home-$SB` to `chmod 600 /home/node/.dsh/.credentials.yaml`, patch the sandbox's `podTemplate` with the policy, then resume |
| Agents get `Permission denied` writing to `/workspace` on a fresh or resumed pod | The `/workspace` mount is the `dsh-home` PVC's `workspace/` subPath, and the kubelet is known to leave a `subPath` directory at `0755` if it is interrupted between creating and chmodding it ([kubernetes#142464](https://github.com/kubernetes/kubernetes/issues/142464)); later mounts never correct it. Suspend the sandbox, run a throwaway pod on `dsh-home-$SB` mounted at `/data`, `chmod 2775 /data/workspace` (do **not** chown/chmod recursively — that rewrites `.credentials.yaml`), then resume. If you recreate the sandbox instead, its current `/workspace` contents go with the PVC | | Agents get `Permission denied` writing to `/workspace` on a fresh or resumed pod | The `/workspace` mount is the `dsh-home` PVC's `workspace/` subPath, and the kubelet is known to leave a `subPath` directory at `0755` if it is interrupted between creating and chmodding it ([kubernetes#142464](https://github.com/kubernetes/kubernetes/issues/142464)); later mounts never correct it. Suspend the sandbox, run a throwaway pod on `dsh-home-$SB` mounted at `/data`, `chmod 2775 /data/workspace` (do **not** chown/chmod recursively — that rewrites `.credentials.yaml`), then resume. If you recreate the sandbox instead, its current `/workspace` contents go with the PVC |
@@ -666,12 +583,6 @@ Deliberately **not** production code:
- **Bearer tokens in URLs.** Avoid putting the signup token in a URL/log; the - **Bearer tokens in URLs.** Avoid putting the signup token in a URL/log; the
browser flow uses the `HttpOnly` cookie instead. browser flow uses the `HttpOnly` cookie instead.
## Cost model
The same economics as the upstream pattern: suspend-when-idle plus warm pools turn
an always-on instance into a sub-second cold-start pause per return-after-absence,
at a fraction of the cost.
## Cleanup ## Cleanup
```bash ```bash

View File

@@ -6,7 +6,8 @@ Kubernetes, and DSH Web fit together, and why the design looks the way it does.
Scope: one Kubernetes namespace (`dsh-demo`), one shared DSH Web image, one gateway deployment, Scope: one Kubernetes namespace (`dsh-demo`), one shared DSH Web image, one gateway deployment,
and one DSH Web pod + PVC per user. The cluster is assumed to run and one DSH Web pod + PVC per user. The cluster is assumed to run
[agent-sandbox](https://github.com/kubernetes-sigs/agent-sandbox) v1.0.8 with the extensions [agent-sandbox](https://github.com/kubernetes-sigs/agent-sandbox) with the pinned
`AGENT_SANDBOX_VERSION` from [`versions.env`](../../../versions.env) and the extensions
(`SandboxTemplate`, `SandboxWarmPool`, `SandboxClaim`). (`SandboxTemplate`, `SandboxWarmPool`, `SandboxClaim`).
## Container view ## Container view
@@ -340,7 +341,7 @@ Notes and limits:
| Probe on `/` | Pod `CrashLoopBackOff`, `401` in probe logs | Use `/favicon.svg` for readiness/liveness (shipped templates do) | | Probe on `/` | Pod `CrashLoopBackOff`, `401` in probe logs | Use `/favicon.svg` for readiness/liveness (shipped templates do) |
| Warm pool exhausted | JSON signup returns `202` (state `Waking`); form signup is slow | Increase `replicas` in the warm pool | | Warm pool exhausted | JSON signup returns `202` (state `Waking`); form signup is slow | Increase `replicas` in the warm pool |
| Gateway restart | Idle sandboxes stop suspending; handshake cache lost | Rebuilt lazily on next request; consider a shared store for production | | Gateway restart | Idle sandboxes stop suspending; handshake cache lost | Rebuilt lazily on next request; consider a shared store for production |
| Browser aborts a request (tab reload, back navigation) during a slow wake or handshake | No response reaches the client; nothing appears in the access log | If the abort lands before the response starts, the body read raises `ClientDisconnect` and the app answers `499` (`CLIENT_CLOSED_REQUEST`), which cannot reach the departed peer. Nothing reached DSH, no ERROR traceback is logged, and the in-flight slot is released. A request already streaming ends with its upstream status. Before the fix every mid-body abort logged a `ClientDisconnect` traceback | | Browser aborts a request (tab reload, back navigation) during a slow wake or handshake | No response reaches the client; nothing appears in the access log | If the abort lands before the response starts, the body read raises `ClientDisconnect` and the app answers `499` (`CLIENT_CLOSED_REQUEST`), which cannot reach the departed peer. Nothing reached DSH, no ERROR traceback is logged, and the in-flight slot is released. A request already streaming ends with its upstream status. |
| DSH auth cookie expired mid-session | Proxy `401` once, then recovers | Gateway refreshes the handshake once | | DSH auth cookie expired mid-session | Proxy `401` once, then recovers | Gateway refreshes the handshake once |
| Upstream pod name changed after resume | Cached cookie keyed by old pod UID is useless | Cache is invalidated on suspend; next request re-handshakes | | Upstream pod name changed after resume | Cached cookie keyed by old pod UID is useless | Cache is invalidated on suspend; next request re-handshakes |
| Two gateway replicas | Double suspend/wake races | Keep `replicas: 1` until state is externalized | | Two gateway replicas | Double suspend/wake races | Keep `replicas: 1` until state is externalized |

View File

@@ -3,7 +3,7 @@
The base example is deliberately portable: it runs on kind and GKE without a The base example is deliberately portable: it runs on kind and GKE without a
gVisor runtime, a tainted node pool or a NetworkPolicy-capable CNI, and it gVisor runtime, a tainted node pool or a NetworkPolicy-capable CNI, and it
boots with only the required `dsh-provider-keys` Secret. This document covers boots with only the required `dsh-provider-keys` Secret. This document covers
four **opt-in** isolation layers, plus one availability setting: four **opt-in** controls plus failed-Pod recovery:
| Setting | Where it lives | What it changes | | Setting | Where it lives | What it changes |
| --- | --- | --- | | --- | --- | --- |
@@ -52,7 +52,7 @@ PodSpec; no CRD change is involved.
- [`manifests/50-gateway.yaml`](../manifests/50-gateway.yaml) - [`manifests/50-gateway.yaml`](../manifests/50-gateway.yaml)
- [`manifests/exposure/gateway-api-sso/72-oauth2-proxy.yaml`](../manifests/exposure/gateway-api-sso/72-oauth2-proxy.yaml) - [`manifests/exposure/gateway-api-sso/72-oauth2-proxy.yaml`](../manifests/exposure/gateway-api-sso/72-oauth2-proxy.yaml)
(only with the SSO variant) (only with the SSO variant)
- [minimal sandbox-template.yaml](../../minimal-single-tenant/manifests/sandbox-template.yaml) - [minimal template](../../minimal-single-tenant/manifests/10-sandboxtemplate.yaml)
**Existing install:** a template edit does not update existing Sandboxes, and **Existing install:** a template edit does not update existing Sandboxes, and
warm-pool spares use `OnReplenish`, so patch explicitly. Plan a maintenance warm-pool spares use `OnReplenish`, so patch explicitly. Plan a maintenance
@@ -604,7 +604,7 @@ Required locally for any change to this example:
```bash ```bash
make check # versions, example layout, manifest YAML shape make check # versions, example layout, manifest YAML shape
yamllint . yamllint .
make test # gateway unit tests (gateway code is untouched) make test # only when the gateway code changed
``` ```
CI additionally parses every manifest and runs `kubectl apply CI additionally parses every manifest and runs `kubectl apply

View File

@@ -22,7 +22,7 @@ It does four things:
The DSH version pin lives in [`versions.env`](../../../versions.env) The DSH version pin lives in [`versions.env`](../../../versions.env)
(`DSH_VERSION` plus the optional repository `RELEASE_REVISION`); the example (`DSH_VERSION` plus the optional repository `RELEASE_REVISION`); the example
manifests and published images use the release identity built from it (currently manifests and published images use the release identity built from it (currently
`dsh-v0.2.1-rc.5`). DSH binds `:3080`, serves `/`, gates `/api` behind a `dsh-v0.2.1-rc.5`). DSH binds `:3080`, gates the UI and `/api` behind a
`dsh-auth-*` cookie and requires a WebSocket at `/api/remote.mux`. `dsh-auth-*` cookie and requires a WebSocket at `/api/remote.mux`.
## Flow ## Flow
@@ -174,13 +174,17 @@ can also be injected into `create_app` for tests.
| `SWEEP_INTERVAL` | `15` | Seconds between idle sweeps. | | `SWEEP_INTERVAL` | `15` | Seconds between idle sweeps. |
| `AUTH_PROXY_HEADER` | unset | Opt-in SSO: name of the trusted identity header set by an authenticating reverse proxy (e.g. `X-Forwarded-User`). Unset disables SSO and the header is never trusted. Enable only when the Service is reachable exclusively through that proxy (see [SSO](#sso-opt-in)). | | `AUTH_PROXY_HEADER` | unset | Opt-in SSO: name of the trusted identity header set by an authenticating reverse proxy (e.g. `X-Forwarded-User`). Unset disables SSO and the header is never trusted. Enable only when the Service is reachable exclusively through that proxy (see [SSO](#sso-opt-in)). |
The shipped `manifests/50-gateway.yaml` sets `IDLE_TIMEOUT=360` and
`WAKE_TIMEOUT=240` for the demo; the e2e script overrides them to 60/120 so the
smoke test does not wait minutes for the sweeper.
### Required RBAC ### Required RBAC
The gateway's ServiceAccount needs, in its namespace: The gateway's ServiceAccount needs, in its namespace:
* `sandboxclaims` (`extensions.agents.x-k8s.io`): get, list, create, delete * `sandboxclaims` (`extensions.agents.x-k8s.io`): get, list, create, delete
* `sandboxes` (`agents.x-k8s.io`): get, patch * `sandboxes` (`agents.x-k8s.io`): get, list, patch
* `pods`: get * `pods`: get, list
* `pods/log`: get * `pods/log`: get
## Running locally ## Running locally

View File

@@ -69,7 +69,7 @@ spec:
# OnRootMismatch skips that walk once the volume root already carries # OnRootMismatch skips that walk once the volume root already carries
# the fsGroup ownership/mode applied on the first mount. # the fsGroup ownership/mode applied on the first mount.
# Keep in sync with the minimal variant: # Keep in sync with the minimal variant:
# examples/minimal-single-tenant/manifests/sandbox-template.yaml. # examples/minimal-single-tenant/manifests/10-sandboxtemplate.yaml.
fsGroupChangePolicy: OnRootMismatch fsGroupChangePolicy: OnRootMismatch
containers: containers:
- name: dsh-web - name: dsh-web
@@ -103,7 +103,7 @@ spec:
# sessions, config and credentials are unaffected. Existing # sessions, config and credentials are unaffected. Existing
# sandboxes stamp pod env at creation and keep the old HOME until # sandboxes stamp pod env at creation and keep the old HOME until
# recreated. Keep in sync with the minimal variant: # recreated. Keep in sync with the minimal variant:
# examples/minimal-single-tenant/manifests/sandbox-template.yaml. # examples/minimal-single-tenant/manifests/10-sandboxtemplate.yaml.
- name: HOME - name: HOME
value: /workspace value: /workspace
# Empty => DSH derives links from the request Host, which the # Empty => DSH derives links from the request Host, which the

View File

@@ -48,7 +48,7 @@ kubectl -n dsh-demo create secret generic dsh-provider-keys-extra \
--from-literal=NVIDIA_API_KEY="nvapi-..." --from-literal=NVIDIA_API_KEY="nvapi-..."
# 2. The template itself (image pinned in the manifest). # 2. The template itself (image pinned in the manifest).
kubectl apply -f manifests/sandbox-template.yaml kubectl apply -f manifests/10-sandboxtemplate.yaml
# 3. A template only describes pods; a warm pool turns it into running # 3. A template only describes pods; a warm pool turns it into running
# Sandboxes and a claim adopts one. One replica is enough here. # Sandboxes and a claim adopts one. One replica is enough here.
@@ -103,16 +103,19 @@ kubectl -n dsh-demo port-forward "pod/$SB" 18081:3080
## Folder picker and the workspace ## Folder picker and the workspace
The template sets `HOME=/workspace`. DSH's directory picker (the folder explorer The template mounts the `dsh-home` PVC's `workspace/` subPath at
used to choose a workspace) has no start-path setting: it lists `$HOME` when it `/workspace` and sets `HOME=/workspace`. DSH's directory picker (the folder
opens, so with the image default (`/home/node`) it would start on a directory that explorer used to choose a workspace) has no start-path setting: it lists `$HOME`
is not persisted. Pointing `HOME` at the PVC-backed workspace makes the picker when it opens, so with the image default (`/home/node`) it would start on a
open there (labelled *Home* in its breadcrumbs) and makes `~` in agent commands directory that is not persisted. Pointing `HOME` at the PVC-backed workspace makes
resolve to `/workspace`. The trade-off is that `$HOME`-relative tool caches and the picker open there (labelled *Home* in its breadcrumbs) and makes `~` in agent
config (`.cache`, `.config`, `.local`, `.npm`, `.agents`) are written into the commands resolve to `/workspace`. The trade-off is that `$HOME`-relative tool
workspace on the PVC. DSH state is unaffected: `DSH_HOME` stays caches and config (`.cache`, `.config`, `.local`, `.npm`, `.agents`) are written
into the workspace on the PVC. DSH state is unaffected: `DSH_HOME` stays
`/home/node/.dsh`. The rationale and the list of affected paths are in `/home/node/.dsh`. The rationale and the list of affected paths are in
[`10-sandboxtemplate.yaml`](../dsh-agents-as-a-service/manifests/10-sandboxtemplate.yaml). [`10-sandboxtemplate.yaml`](../dsh-agents-as-a-service/manifests/10-sandboxtemplate.yaml).
Pod specs are stamped at creation, so an existing sandbox keeps its old mounts and
`HOME` until it is recreated; the PVC retains DSH state and workspace data.
Verify inside a running sandbox: Verify inside a running sandbox:
@@ -133,7 +136,7 @@ NetworkPolicy are documented in
## Probes and trusted hosts ## Probes and trusted hosts
Two settings in `manifests/sandbox-template.yaml` are easy to get wrong: Two settings in `manifests/10-sandboxtemplate.yaml` are easy to get wrong:
- **Probe path.** `/` returns `401` without the launch-token cookie, so a probe - **Probe path.** `/` returns `401` without the launch-token cookie, so a probe
there crash-loops the pod. The template probes `/favicon.svg` (served there crash-loops the pod. The template probes `/favicon.svg` (served

View File

@@ -107,6 +107,14 @@ spec:
volumeMounts: volumeMounts:
- name: dsh-home - name: dsh-home
mountPath: /home/node/.dsh # DSH_HOME mountPath: /home/node/.dsh # DSH_HOME
# The agent workspace (the image's WORKDIR) is the same PVC under
# its own subPath, so DSH state - including .credentials.yaml -
# never shows up as workspace data. Kubelet creates workspace/ on
# first mount with the volume's fsGroup permissions; no init
# container is needed. See the multi-tenant template for details.
- name: dsh-home
mountPath: /workspace
subPath: workspace
# / is behind DSH's launch-token cookie (401 without it); use an # / is behind DSH's launch-token cookie (401 without it); use an
# unauthenticated path for probes. The startup probe gives a cold # unauthenticated path for probes. The startup probe gives a cold
# start (plugin activation on a small node, or the first resume of a # start (plugin activation on a small node, or the first resume of a

View File

@@ -20,6 +20,13 @@ for dir in examples/*/; do
if [ -d "${dir}manifests" ]; then if [ -d "${dir}manifests" ]; then
count=$(find "${dir}manifests" -maxdepth 1 -type f -name '*.yaml' | wc -l | tr -d ' ') count=$(find "${dir}manifests" -maxdepth 1 -type f -name '*.yaml' | wc -l | tr -d ' ')
[ "$count" -gt 0 ] || err "$name: manifests/ has no top-level *.yaml files" [ "$count" -gt 0 ] || err "$name: manifests/ has no top-level *.yaml files"
for manifest in "${dir}"manifests/*.yaml; do
[ -e "$manifest" ] || continue
case "$(basename "$manifest")" in
[0-9][0-9]-*.yaml) ;;
*) err "$name: top-level manifest $(basename "$manifest") must be named NN-*.yaml" ;;
esac
done
fi fi
stray=$(find "$dir" -maxdepth 1 -type f -name '*.yaml' | wc -l | tr -d ' ') stray=$(find "$dir" -maxdepth 1 -type f -name '*.yaml' | wc -l | tr -d ' ')

View File

@@ -33,7 +33,7 @@ expect .env.sample "RELEASE_REVISION=$RELEASE_REVISION"
# Image references in the example manifests # Image references in the example manifests
for f in \ for f in \
examples/dsh-agents-as-a-service/manifests/10-sandboxtemplate.yaml \ examples/dsh-agents-as-a-service/manifests/10-sandboxtemplate.yaml \
examples/minimal-single-tenant/manifests/sandbox-template.yaml; do examples/minimal-single-tenant/manifests/10-sandboxtemplate.yaml; do
expect "$f" "image: $GHCR_IMAGE:$IDENTITY" expect "$f" "image: $GHCR_IMAGE:$IDENTITY"
done done
expect examples/dsh-agents-as-a-service/manifests/50-gateway.yaml "image: $GATEWAY_IMAGE:$IDENTITY" expect examples/dsh-agents-as-a-service/manifests/50-gateway.yaml "image: $GATEWAY_IMAGE:$IDENTITY"