mirror of
https://github.com/sredevopsorg/deepseek-kubernetes-sandbox.git
synced 2026-11-03 01:13:24 +00:00
docs: align readmes, docs and agent guidance with current behavior (#37)
Remove obsolete layout/migration prose and legacy gateway-alias references, reconcile the required-check policy, fix version-label/RBAC/workspace/channel drift, and de-duplicate the example and gateway docs. Rename the minimal manifest to 10-sandboxtemplate.yaml and mount the dsh-home PVC's workspace subPath at /workspace so the documented persistence is real. Enforce numbered top-level manifests in check-examples.sh. Verified with make check, yamllint, shellcheck, a Markdown link/anchor scan, and git diff --check. Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
This commit is contained in:
SREDevOps.org Agent
committed by
GitHub
GitHub
parent
82ee0d0cdb
commit
24f23e09ae
@@ -126,13 +126,14 @@ Read `docs/examples.md` and the testing table in `AGENTS.md` before editing.
|
||||
alternatives, never a stack.
|
||||
- **New example:** follow `docs/examples.md`, add the index entry, keep scripts
|
||||
executable/idempotent/self-cleaning, run `make check`.
|
||||
- **Version bump:** edit `versions.env` only, update prerequisites, run
|
||||
`make check` and the e2e. Never mix a bump with an unrelated feature.
|
||||
- **Version bump:** use `make set-version`; never edit pins by hand. Update
|
||||
prerequisites, run `make check` and the e2e on a cluster. Never mix a bump
|
||||
with an unrelated feature.
|
||||
- **MCP integration:** pin the build to the cluster's tag, least-privilege SA,
|
||||
ClusterIP + auth proxy, raised DSH timeout, no secrets in the repo.
|
||||
- **Optional hardening:** gVisor, `nodeSelector`, NetworkPolicy and extra
|
||||
provider keys stay opt-in; the sandbox policy lives under
|
||||
`manifests/optional/`. All four are documented in
|
||||
- **Optional hardening:** gVisor, `nodeSelector`, NetworkPolicy, extra
|
||||
provider keys and failed-Pod recovery stay opt-in; the sandbox policy lives
|
||||
under `manifests/optional/`. All of them are documented in
|
||||
`examples/dsh-agents-as-a-service/docs/optional-hardening.md`.
|
||||
- **Docs:** update the affected doc in the same change.
|
||||
|
||||
|
||||
27
.env.sample
27
.env.sample
@@ -1,4 +1,6 @@
|
||||
# Copy to .env and fill in. Never commit a real key.
|
||||
# Reference environment names for local runs. Nothing sources this file
|
||||
# automatically: export the values or pass them with `docker run -e/--env-file`.
|
||||
# Copy it to .env if you prefer --env-file. Never commit a real key.
|
||||
#
|
||||
# The version values below are documentation only: versions.env is the single
|
||||
# source of truth, and scripts/check-versions.sh fails if they drift.
|
||||
@@ -29,22 +31,7 @@ DSH_PORT=3080
|
||||
# public hostname). localhost and 127.0.0.1 are always trusted.
|
||||
DSH_TRUSTED_HOSTS=
|
||||
|
||||
# --- agents-as-a-service example ---
|
||||
NAMESPACE=dsh-demo
|
||||
POOL=dsh-pool
|
||||
# Seconds a sandbox may stay idle before the sweeper suspends it, and the max
|
||||
# time a proxied request waits for a suspended sandbox to become Ready.
|
||||
IDLE_TIMEOUT=360
|
||||
WAKE_TIMEOUT=240
|
||||
DSH_CONTAINER=dsh-web
|
||||
DSH_LOG_TAIL_LINES=2000
|
||||
COOKIE_NAME=dsh_gw_session
|
||||
# Set to true only behind HTTPS; secure cookies are not sent over plain HTTP
|
||||
# (localhost counts as a secure context in browsers).
|
||||
COOKIE_SECURE=false
|
||||
COOKIE_MAX_AGE=604800
|
||||
MAX_BODY_MB=32
|
||||
HTTP_TIMEOUT=300
|
||||
# SSO is opt-in. Set this only when the gateway Service is reachable
|
||||
# exclusively through oauth2-proxy (see the example exposure docs).
|
||||
# AUTH_PROXY_HEADER=X-Forwarded-User
|
||||
# The example gateway's runtime settings (NAMESPACE, POOL, IDLE_TIMEOUT,
|
||||
# WAKE_TIMEOUT, COOKIE_*, AUTH_PROXY_HEADER, ...) are documented in
|
||||
# examples/dsh-agents-as-a-service/gateway/README.md and set in
|
||||
# examples/dsh-agents-as-a-service/manifests/50-gateway.yaml.
|
||||
|
||||
2
.github/ISSUE_TEMPLATE/bug_report.yml
vendored
2
.github/ISSUE_TEMPLATE/bug_report.yml
vendored
@@ -28,7 +28,7 @@ body:
|
||||
id: image-tag
|
||||
attributes:
|
||||
label: Image or release
|
||||
description: The pinned tag or digest, for example dsh-v0.2.1-rc.5.
|
||||
description: The tag or digest you deployed (from the manifest or release metadata).
|
||||
- type: input
|
||||
id: cluster
|
||||
attributes:
|
||||
|
||||
3
.github/pull_request_template.md
vendored
3
.github/pull_request_template.md
vendored
@@ -19,8 +19,7 @@
|
||||
## Checklist
|
||||
|
||||
- [ ] make check passes (versions, example layout, manifest YAML)
|
||||
- [ ] make test passes (gateway unit tests)
|
||||
- [ ] make lint passes (ruff, shellcheck; yamllint/actionlint where installed)
|
||||
- [ ] make test / make lint pass when the change touches code (gateway unit tests, ruff, shellcheck; yamllint/actionlint where installed)
|
||||
- [ ] Docs updated for the change (root README, docs/, example README)
|
||||
- [ ] No secrets or usable credentials committed
|
||||
- [ ] CHANGELOG.md was not edited by hand (generated by git-cliff)
|
||||
|
||||
5
.github/workflows/ci-manifests.yaml
vendored
5
.github/workflows/ci-manifests.yaml
vendored
@@ -1,7 +1,8 @@
|
||||
# Manifest shape validation and server-side dry-run. Split from ci.yaml so this
|
||||
# job can use native on.pull_request.paths filtering and skip PRs that do not
|
||||
# touch manifests, scripts or versions.env. The job name is the required
|
||||
# branch-protection check `manifests`.
|
||||
# touch manifests, scripts or versions.env. The job name is `manifests`; because
|
||||
# it is path-filtered it is an on-demand check, not a required branch-protection
|
||||
# check (see docs/gitflow.md).
|
||||
name: CI (manifests)
|
||||
|
||||
on:
|
||||
|
||||
3
.github/workflows/ci-python.yaml
vendored
3
.github/workflows/ci-python.yaml
vendored
@@ -1,6 +1,7 @@
|
||||
# Gateway unit tests and lint. Split from ci.yaml so this job can use native
|
||||
# on.pull_request.paths filtering and skip PRs that do not touch the gateway.
|
||||
# The job name is the required branch-protection check `python`.
|
||||
# The job name is `python`; because it is path-filtered it is an on-demand check,
|
||||
# not a required branch-protection check (see docs/gitflow.md).
|
||||
name: CI (python)
|
||||
|
||||
on:
|
||||
|
||||
27
.github/workflows/release.yaml
vendored
27
.github/workflows/release.yaml
vendored
@@ -16,11 +16,6 @@ on:
|
||||
description: Existing tag to release (must match versions.env)
|
||||
required: true
|
||||
type: string
|
||||
publish_legacy_gateway_alias:
|
||||
description: Also publish the legacy :gateway alias (one release only)
|
||||
required: false
|
||||
default: true
|
||||
type: boolean
|
||||
|
||||
permissions:
|
||||
contents: write
|
||||
@@ -193,28 +188,6 @@ jobs:
|
||||
${{ needs.meta.outputs.gateway_image }}:${{ needs.meta.outputs.identity }}
|
||||
${{ needs.meta.outputs.gateway_image }}:latest
|
||||
|
||||
legacy-gateway-alias:
|
||||
name: Publish legacy :gateway alias (temporary)
|
||||
needs: [meta, build-gateway]
|
||||
if: github.event_name != 'workflow_dispatch' || inputs.publish_legacy_gateway_alias
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 15
|
||||
steps:
|
||||
- name: Login to GitHub Container Registry
|
||||
uses: docker/login-action@81c6f93d4bc991e510b342a00b419b3bb4efc5c6 # v4.7.0
|
||||
with:
|
||||
registry: ghcr.io
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.GITHUB_TOKEN }}
|
||||
- name: Retag gateway as the legacy package tag
|
||||
env:
|
||||
SOURCE: ${{ needs.meta.outputs.gateway_image }}:${{ needs.meta.outputs.identity }}
|
||||
LEGACY: ${{ needs.meta.outputs.ghcr_image }}:gateway
|
||||
run: |
|
||||
set -euo pipefail
|
||||
docker buildx imagetools create -t "$LEGACY" "$SOURCE"
|
||||
echo "Deprecated alias $LEGACY now points at $SOURCE" >> "$GITHUB_STEP_SUMMARY"
|
||||
|
||||
publish-release:
|
||||
name: Changelog and GitHub Release
|
||||
needs: [meta, build-main, build-gateway]
|
||||
|
||||
36
AGENTS.md
36
AGENTS.md
@@ -9,8 +9,8 @@ before changing anything; it encodes the conventions the CI enforces.
|
||||
Harness Web (dsh web) as a slim, non-root, multi-arch image. Everything else
|
||||
exists to support it or to demonstrate an implementation on top of it.
|
||||
- **Every implementation is a self-contained example** under examples/<name>/.
|
||||
There is exactly one example for the agents-as-a-service platform; do not
|
||||
split it into parallel directories.
|
||||
The multi-tenant platform lives in exactly one directory; do not split it into
|
||||
parallel examples.
|
||||
- **versions.env is the single source of truth** for the DSH ref, pnpm, the
|
||||
control-plane versions and the image names.
|
||||
- **Docs describe the shipped behavior**, not intentions. If you change a
|
||||
@@ -43,7 +43,7 @@ before changing anything; it encodes the conventions the CI enforces.
|
||||
| versions.env | Every pinned version and image name |
|
||||
| Makefile | Canonical local entrypoint |
|
||||
| build.sh | Deprecated shim that forwards to make |
|
||||
| docs/ | main-image.md, examples.md, gitflow.md, versioning.md |
|
||||
| docs/ | main-image.md, examples.md, gitflow.md, versioning.md (index: README.md) |
|
||||
| examples/ | One implementation per directory (contract in docs/examples.md) |
|
||||
| examples/dsh-agents-as-a-service/ | The multi-tenant platform example (manifests, gateway, docs, scripts) |
|
||||
| examples/minimal-single-tenant/ | The just-one-DSH-Web quickstart |
|
||||
@@ -54,7 +54,7 @@ before changing anything; it encodes the conventions the CI enforces.
|
||||
## Commands
|
||||
|
||||
make check # versions, example layout and manifest YAML (run before every commit)
|
||||
make test # gateway unit tests (109+ tests)
|
||||
make test # gateway unit tests (pytest)
|
||||
make lint # ruff + shellcheck (+ yamllint/actionlint when installed)
|
||||
make image # build the main DSH Web image (slow: clones and builds DSH)
|
||||
make gateway # build the example gateway image (fast)
|
||||
@@ -72,21 +72,18 @@ needs kind plus the agent-sandbox CRDs (see the example README).
|
||||
| --- | --- |
|
||||
| Documentation only | make check; verify relative links |
|
||||
| Example manifest | make check; kubectl apply --dry-run=server in CI |
|
||||
| Optional hardening (gVisor, nodeSelector, NetworkPolicy, provider keys) | make check; server dry-run in CI; manual verification on a capable cluster |
|
||||
| Optional hardening (gVisor, nodeSelector, NetworkPolicy, provider keys, failed-Pod recovery) | make check; server dry-run in CI; manual verification on a capable cluster |
|
||||
| Gateway code | make test; make lint; add or update tests |
|
||||
| Dockerfile / versions.env | make check; the CI `build-amd64` job (`ci-image.yaml`) builds and smoke-tests the image on the PR; run `make image` locally only when you need the image |
|
||||
| Workflow | actionlint in CI; keep actions SHA-pinned; no hard-coded versions |
|
||||
| New example | make check; all files described in docs/examples.md |
|
||||
| Project skill (.dsh/skills/) | make check; frontmatter and size check; shellcheck skill scripts; load it with the skill tool |
|
||||
| Project skill (.dsh/skills/) | make check; validate frontmatter; keep SKILL.md under ~8 KB; shellcheck skill scripts; load it with the skill tool |
|
||||
|
||||
## Versioning and releases
|
||||
|
||||
- Release identity = DSH_VERSION, optionally with a repository revision suffix
|
||||
for re-releases at the same DSH ref (current identity:
|
||||
dsh-v0.2.1-rc.5).
|
||||
- The current identity is dsh-v0.2.1-rc.5 (stored in versions.env). It is the
|
||||
first release at this DSH ref, so it carries no revision suffix; a
|
||||
repository-only change at the same ref publishes -r2.
|
||||
- Release identity = DSH_VERSION plus an optional -rN repository revision.
|
||||
The current identity is dsh-v0.2.1-rc.5 (first release at this DSH ref, so no
|
||||
suffix); a repository-only change at the same ref publishes -r2.
|
||||
- Channels: release tags are immutable; latest follows the newest release;
|
||||
main is the development channel for both image packages.
|
||||
- A broken release is fixed by publishing -r(N+1), never by re-tagging.
|
||||
@@ -99,7 +96,9 @@ needs kind plus the agent-sandbox CRDs (see the example README).
|
||||
- Branch names: feat/, fix/, docs/, chore/, refactor/, test/.
|
||||
- Conventional Commit subjects and PR titles (feat(scope): ..., fix(scope): ...).
|
||||
- PRs are squash-merged; main stays linear and protected.
|
||||
- Required checks: lint, python, manifests, pr-title, CodeQL, dependency-review.
|
||||
- Required on every PR: lint, pr-title, CodeQL (both Analyze jobs) and
|
||||
dependency-review. python, manifests and build-amd64 are path-filtered
|
||||
on-demand checks; run them when the change touches their scope.
|
||||
- Keep diffs scoped; update docs with the code; never mix a version bump with
|
||||
unrelated changes.
|
||||
|
||||
@@ -107,9 +106,9 @@ needs kind plus the agent-sandbox CRDs (see the example README).
|
||||
|
||||
1. Read docs/examples.md and examples/README.md.
|
||||
2. Create examples/<name>/ with README.md, manifests/NN-*.yaml, optional
|
||||
manifests/exposure/<variant>/ (plus README), optional components with their
|
||||
own Dockerfile/README/tests, and scripts/ (executable, idempotent,
|
||||
self-cleaning).
|
||||
manifests/optional/<feature>/ or manifests/exposure/<variant>/ (each with a
|
||||
README), optional components with their own Dockerfile/README/tests, and
|
||||
scripts/ (executable, idempotent, self-cleaning).
|
||||
3. Publish component images under <registry>/<owner>/<repo>/<example>/<component>.
|
||||
4. Add the example to examples/README.md; scripts/check-examples.sh enforces
|
||||
the layout.
|
||||
@@ -124,8 +123,9 @@ needs kind plus the agent-sandbox CRDs (see the example README).
|
||||
- The example shares provider keys (dsh-provider-keys plus the optional
|
||||
dsh-provider-keys-extra) across sandboxes; do not describe it as
|
||||
tenant-isolated.
|
||||
- gVisor, nodeSelector and the NetworkPolicies are opt-in and are not active in
|
||||
the base manifests; do not describe the base example as hardened.
|
||||
- gVisor, nodeSelector, the NetworkPolicies, extra provider keys and failed-Pod
|
||||
recovery are opt-in and are not active in the base manifests; do not describe
|
||||
the base example as hardened.
|
||||
- TLS terminates outside the gateway; set COOKIE_SECURE=true for HTTPS and keep
|
||||
the base manifest safe for local HTTP.
|
||||
|
||||
|
||||
@@ -43,8 +43,10 @@ Install the gateway dev dependencies once:
|
||||
- Use a Conventional Commit title, for example feat(gateway): add wake metrics
|
||||
or docs(examples): clarify SSO trust boundary. PR titles are linted.
|
||||
- Keep the diff scoped. Squash merge into main; history stays linear.
|
||||
- Required checks: lint, python, manifests, pr-title, CodeQL, and
|
||||
dependency-review. Branch protection is described in docs/gitflow.md.
|
||||
- Required on every PR: lint, pr-title, CodeQL (both Analyze jobs) and
|
||||
dependency-review. python, manifests and build-amd64 are path-filtered
|
||||
on-demand checks; run them when the change touches their scope. Branch
|
||||
protection is described in docs/gitflow.md.
|
||||
- Update the docs that the change affects; docs are part of the change, not a
|
||||
follow-up.
|
||||
- Never commit a real secret. Secret-shaped YAML belongs in a *.yaml.example
|
||||
@@ -60,16 +62,12 @@ it by hand; the release workflow regenerates it.
|
||||
|
||||
## Adding an example
|
||||
|
||||
Read docs/examples.md, then add examples/<name>/ with:
|
||||
|
||||
- README.md (what it demonstrates, prerequisites, steps, teardown)
|
||||
- manifests/NN-*.yaml (numbered, plain kubectl)
|
||||
- optional manifests/exposure/<variant>/ with its own README.md
|
||||
- optional components (a Dockerfile plus README and tests per component)
|
||||
- scripts/ with executable, idempotent, self-cleaning scripts
|
||||
- docs/ for deeper material
|
||||
|
||||
List it in examples/README.md. scripts/check-examples.sh enforces this layout.
|
||||
Read docs/examples.md, then add examples/<name>/ with a README.md,
|
||||
numbered manifests/NN-*.yaml, optional manifests/optional/<feature>/ or
|
||||
manifests/exposure/<variant>/ (each with a README), optional components (a
|
||||
Dockerfile plus README and tests), and executable, idempotent, self-cleaning
|
||||
scripts. Add the literal examples/<name>/ entry to examples/README.md;
|
||||
scripts/check-examples.sh enforces the layout.
|
||||
|
||||
## Documentation
|
||||
|
||||
|
||||
29
README.md
29
README.md
@@ -96,10 +96,7 @@ ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway
|
||||
| --- | --- | --- |
|
||||
| `dsh-v0.2.1-rc.5` | DSH Web, port `3080` | **Current release identity.** Immutable: once published it never moves. |
|
||||
| `latest` | DSH Web | Most recent release; same digest as the current identity. |
|
||||
| `main` | DSH Web | Development channel rebuilt from every push to `main`. Not a release; not for production. |
|
||||
|
||||
Tags published before the release-identity scheme remain on GHCR but are not
|
||||
updated; new deployments use the current identity.
|
||||
| `main` | DSH Web | Development channel rebuilt when image-definition files change on `main`. Not a release; not for production. |
|
||||
|
||||
The example gateway is published under its nested path with the same release
|
||||
identity, e.g.
|
||||
@@ -134,8 +131,8 @@ Self-contained implementations that consume the image. Index and contract:
|
||||
|
||||
| Example | What it demonstrates |
|
||||
| --- | --- |
|
||||
| [`examples/dsh-agents-as-a-service/`](examples/dsh-agents-as-a-service/) | Full multi-tenant platform: SandboxTemplate with per-user PVCs, warm pool, one claim per user, suspend-when-idle, wake-on-return, browser gateway with signup + `HttpOnly` sessions + HTTP/WebSocket proxy, Ingress or Gateway API + GitHub SSO exposure, and opt-in gVisor/NetworkPolicy/provider-key hardening. Start with the [walkthrough](examples/dsh-agents-as-a-service/README.md) and the [architecture](examples/dsh-agents-as-a-service/docs/architecture.md). |
|
||||
| [`examples/minimal-single-tenant/`](examples/minimal-single-tenant/) | The smallest useful setup: one `SandboxTemplate` and one `SandboxClaim`, reached through `kubectl port-forward` + the launch-token URL. Manifest: [`manifests/sandbox-template.yaml`](examples/minimal-single-tenant/manifests/sandbox-template.yaml). |
|
||||
| [`examples/dsh-agents-as-a-service/`](examples/dsh-agents-as-a-service/) | Full multi-tenant platform: SandboxTemplate with per-user PVCs, warm pool, one claim per user, suspend-when-idle, wake-on-return, browser gateway with signup + `HttpOnly` sessions + HTTP/WebSocket proxy, Ingress or Gateway API + GitHub SSO exposure, and opt-in gVisor/NetworkPolicy/provider-key hardening and failed-Pod recovery. Start with the [walkthrough](examples/dsh-agents-as-a-service/README.md) and the [architecture](examples/dsh-agents-as-a-service/docs/architecture.md). |
|
||||
| [`examples/minimal-single-tenant/`](examples/minimal-single-tenant/) | The smallest useful setup: one `SandboxTemplate` and one `SandboxClaim`, reached through `kubectl port-forward` + the launch-token URL. Manifest: [`manifests/10-sandboxtemplate.yaml`](examples/minimal-single-tenant/manifests/10-sandboxtemplate.yaml). |
|
||||
|
||||
The gateway is a component of the multi-tenant example and has its own
|
||||
[README](examples/dsh-agents-as-a-service/gateway/README.md) (API, configuration,
|
||||
@@ -166,7 +163,7 @@ guide and troubleshooting notes live beside `SKILL.md`.
|
||||
| [`scripts/`](scripts/) | Repository checks: versions, example layout, manifest YAML, version bump. |
|
||||
| [`.dsh/skills/`](.dsh/skills/) | Project-local DSH skills, auto-discovered from the git root. |
|
||||
| [`.github/`](.github/) | CI, release automation, dependency updates, PR/issue templates. |
|
||||
| [`.env.sample`](.env.sample) | Documented local configuration (copy to `.env`; never commit it). |
|
||||
| [`.env.sample`](.env.sample) | Reference environment names for local runs; nothing reads it automatically. Never commit a real key. |
|
||||
|
||||
## Development
|
||||
|
||||
@@ -200,24 +197,6 @@ Conventional Commits and is never edited by hand. See
|
||||
[`CHANGELOG.md`](CHANGELOG.md), [docs/versioning.md](docs/versioning.md) and
|
||||
[cliff.toml](cliff.toml).
|
||||
|
||||
## Migrating from the previous layout
|
||||
|
||||
If you are coming from an older checkout, these paths moved. Nothing else changed
|
||||
semantically; the content was split into the example it belongs to.
|
||||
|
||||
| Old path | New path |
|
||||
| --- | --- |
|
||||
| `gateway-api/dsh-demo/` (Gateway API + oauth2-proxy / GitHub SSO) | `examples/dsh-agents-as-a-service/manifests/exposure/gateway-api-sso/` (now `70-gateway.yaml` ... `74-tls-secret.yaml.example`) |
|
||||
| `examples/dsh-agents-as-a-service/60-ingress.yaml` | `examples/dsh-agents-as-a-service/manifests/exposure/ingress/60-ingress.yaml` |
|
||||
| `examples/dsh-agents-as-a-service/NN-*.yaml` (`00-` ... `50-`) | `examples/dsh-agents-as-a-service/manifests/NN-*.yaml` |
|
||||
| `examples/dsh-agents-as-a-service/run-test-kind.sh` | `examples/dsh-agents-as-a-service/scripts/e2e-kind.sh` |
|
||||
| `deploy/k8s/sandbox-template.yaml` | `examples/minimal-single-tenant/manifests/sandbox-template.yaml` |
|
||||
| `docs/architecture.md` | `examples/dsh-agents-as-a-service/docs/architecture.md` |
|
||||
|
||||
The SSO walkthrough that lived in `gateway-api/dsh-demo/README.md` is merged
|
||||
into the [multi-tenant example README](examples/dsh-agents-as-a-service/README.md)
|
||||
(step 9) and the [exposure README](examples/dsh-agents-as-a-service/manifests/exposure/README.md).
|
||||
|
||||
## Contributing
|
||||
|
||||
- [CONTRIBUTING.md](CONTRIBUTING.md) — local setup, PR workflow, what a change needs.
|
||||
|
||||
@@ -21,7 +21,7 @@ disclosure once a fix is available.
|
||||
from pod logs (pods/log). Anyone with access to those logs can impersonate a
|
||||
tenant. Keep the gateway's RBAC namespace-scoped and treat pod logs as
|
||||
sensitive. Replacing log scraping with an upstream token API is tracked in
|
||||
the example architecture document.
|
||||
the [example architecture document](examples/dsh-agents-as-a-service/docs/architecture.md#upstream-feature-request-remove-log-scraping).
|
||||
- **SSO header trust boundary.** AUTH_PROXY_HEADER makes the gateway trust an
|
||||
identity header (for example X-Forwarded-User). Enable it only when the
|
||||
gateway Service is reachable exclusively through the authenticating proxy;
|
||||
|
||||
@@ -44,6 +44,6 @@ commit_parsers = [
|
||||
]
|
||||
protect_breaking_commits = false
|
||||
filter_commits = false
|
||||
# Release tags are dsh-vX.Y.Z[-rN]; the single legacy pre-scheme tag is v0.0.1.
|
||||
# Release tags are dsh-vX.Y.Z[-rN]; the single pre-scheme tag is v0.0.1.
|
||||
tag_pattern = "^(v|dsh-v).*$"
|
||||
sort_commits = "oldest"
|
||||
|
||||
@@ -2,8 +2,7 @@
|
||||
|
||||
Repository-wide documentation lives here. Example-specific material lives with
|
||||
its example under [`examples/<name>/docs/`](../examples/) — see the
|
||||
[agents-as-a-service architecture](../examples/dsh-agents-as-a-service/docs/architecture.md)
|
||||
for the deep-dive that used to live in this directory.
|
||||
[agents-as-a-service architecture](../examples/dsh-agents-as-a-service/docs/architecture.md).
|
||||
|
||||
If you are new, read the [root README](../README.md) first: it is image-first and
|
||||
gets a DSH Web container running before anything else.
|
||||
@@ -34,6 +33,8 @@ DSH discovers project skills from `<git root>/.dsh/skills/<name>/SKILL.md`
|
||||
| [dsh-agents-as-a-service README](../examples/dsh-agents-as-a-service/README.md) | The multi-tenant platform walkthrough: warm pool, claims, suspend/resume, gateway, exposure and troubleshooting. |
|
||||
| [dsh-agents-as-a-service architecture](../examples/dsh-agents-as-a-service/docs/architecture.md) | Container/sequence diagrams, ADRs, security and RBAC, failure modes and the evolution path. |
|
||||
| [gateway README](../examples/dsh-agents-as-a-service/gateway/README.md) | The gateway's HTTP/WebSocket API, configuration, auth (including SSO), and limitations. |
|
||||
| [optional hardening](../examples/dsh-agents-as-a-service/docs/optional-hardening.md) | Opt-in gVisor, nodeSelector, NetworkPolicy, provider keys and failed-Pod recovery. |
|
||||
| [exposure variants](../examples/dsh-agents-as-a-service/manifests/exposure/README.md) | The mutually exclusive Ingress and Gateway API + SSO exposure paths. |
|
||||
|
||||
## Project policies
|
||||
|
||||
|
||||
@@ -41,6 +41,7 @@ examples/<name>/
|
||||
example directory;
|
||||
- an example has no `README.md` or no `manifests/` directory;
|
||||
- `manifests/` contains no top-level `*.yaml` files;
|
||||
- a top-level `manifests/*.yaml` is not named `NN-*.yaml`;
|
||||
- a `*.yaml` file sits at the example root instead of under `manifests/`;
|
||||
- `manifests/exposure/` exists without a `manifests/exposure/README.md` describing
|
||||
the variants;
|
||||
|
||||
@@ -30,7 +30,7 @@ branch; the PR is squash-merged, so individual branch commits are never part of
|
||||
`docs(scope): ...`, `build: ...`, `ci: ...`, `refactor: ...`, `test: ...`,
|
||||
`chore: ...`. Breaking changes use `feat(scope)!: ...` or a
|
||||
`BREAKING CHANGE:` footer.
|
||||
3. **CI must be green.** All required checks below must pass before merge.
|
||||
3. **CI must be green.** All applicable checks below must pass before merge.
|
||||
4. **Squash merge only.** `main` stays linear and protected; merged PRs are one
|
||||
commit each.
|
||||
5. **No direct pushes to `main`** or to a release branch. Both are protected.
|
||||
@@ -40,21 +40,20 @@ branch; the PR is squash-merged, so individual branch commits are never part of
|
||||
|
||||
## Required checks
|
||||
|
||||
| Check | Workflow / job | What it verifies |
|
||||
| --- | --- | --- |
|
||||
| `lint` | `ci.yaml` | `scripts/check-versions.sh` (version pins and markdown tokens), `scripts/check-examples.sh` (example layout), shellcheck, actionlint, yamllint, hadolint and documentation links. |
|
||||
| `python` | `ci-python.yaml` | Gateway ruff check/format and unit tests (`pytest` with coverage). Runs only when the gateway or this workflow changes. |
|
||||
| `manifests` | `ci-manifests.yaml` | `scripts/check-manifests.py` (manifest shape) and a server-side dry-run of every example manifest against the agent-sandbox CRDs. Runs only when manifests, check scripts or `versions.env` change. |
|
||||
| `pr-title` | `pr-title.yaml` | Conventional Commit PR title. |
|
||||
| `CodeQL` | `codeql.yml` | Static analysis for Python and GitHub Actions. |
|
||||
| `dependency-review` | `ci.yaml` | Dependency changes introduced by the PR (job `dependency-review`; runs on pull requests only). |
|
||||
| Check | Workflow / job | Required on every PR? | What it verifies |
|
||||
| --- | --- | --- | --- |
|
||||
| `lint` | `ci.yaml` | yes | `scripts/check-versions.sh` (version pins and markdown tokens), `scripts/check-examples.sh` (example layout and numbered manifests), shellcheck, actionlint, yamllint, hadolint and documentation links. |
|
||||
| `python` | `ci-python.yaml` | no (path-filtered) | Gateway ruff check/format and unit tests (`pytest` with coverage). Runs only when the gateway or this workflow changes. |
|
||||
| `manifests` | `ci-manifests.yaml` | no (path-filtered) | `scripts/check-manifests.py` (manifest shape) and a server-side dry-run of every example manifest against the agent-sandbox CRDs. Runs when manifest files, `scripts/check-manifests.py` or `versions.env` change. |
|
||||
| `pr-title` | `pr-title.yaml` | yes | Conventional Commit PR title. |
|
||||
| `Analyze (python)`, `Analyze (actions)` | `codeql.yml` | yes (both jobs) | CodeQL static analysis for Python and GitHub Actions. |
|
||||
| `dependency-review` | `ci.yaml` | yes | Dependency changes introduced by the PR (job `dependency-review`; runs on pull requests only). |
|
||||
|
||||
`python`, `manifests` and `build-amd64` use native `on.pull_request.paths`
|
||||
filtering, so PRs that do not touch the gateway, the manifests or the image
|
||||
definition skip them. The gateway image build and `/healthz` are no longer
|
||||
smoke-tested in a dedicated job: `/healthz` is covered by the gateway unit tests,
|
||||
the Dockerfile is linted by `lint` (hadolint), and the multi-arch image build is
|
||||
published by `build-gateway.yaml` on merge.
|
||||
definition skip them. The gateway image is built and published by
|
||||
`build-gateway.yaml` on merge; `/healthz` is covered by the gateway unit tests and
|
||||
the Dockerfile is linted by `lint` (hadolint).
|
||||
|
||||
`ci-image.yaml` (`build-amd64`) is the PR-side check for the main image: when
|
||||
`Dockerfile`, `.dockerignore` or `versions.env` change it builds `linux/amd64`
|
||||
@@ -64,9 +63,9 @@ the result (`--version`, non-root UID, `/workspace` + `DSH_HOME`, the
|
||||
container). The published multi-arch build still happens only in
|
||||
`build-main.yaml` on merge.
|
||||
|
||||
Because a skipped path-filtered check is not a passing check, only the always-on
|
||||
checks (`lint`, `pr-title`, `CodeQL`, `dependency-review`) should be enforced as
|
||||
required status checks; `python`, `manifests` and `build-amd64` are on-demand
|
||||
Because a skipped path-filtered check is not a passing check, require only the
|
||||
always-on set: `lint`, `pr-title`, both CodeQL `Analyze` jobs and
|
||||
`dependency-review`. Keep `python`, `manifests` and `build-amd64` as on-demand
|
||||
checks that skip when unrelated to the change.
|
||||
|
||||
The image build workflows (`_build-image.yaml`, `build-main.yaml`,
|
||||
@@ -109,7 +108,8 @@ Rollback is also a re-release, never a retag: see
|
||||
|
||||
## Tag protection
|
||||
|
||||
All release tags match `dsh-*` and are protected:
|
||||
Release tags follow `dsh-*` and are protected (the pre-scheme `v0.0.1` tag is
|
||||
historical and unmaintained):
|
||||
|
||||
- only maintainers (the release workflow) may create them;
|
||||
- they cannot be deleted or force-updated;
|
||||
@@ -123,11 +123,12 @@ respectively.
|
||||
|
||||
Configure these on the repository before relying on the workflow:
|
||||
|
||||
- [ ] **Branches → main**: require a pull request; require the checks `lint`,
|
||||
`python`, `manifests`, `pr-title`, `CodeQL`, `dependency-review`; require
|
||||
branches to be up to date; require conversation resolution; restrict direct
|
||||
pushes; allow only squash merges and enable "Automatically delete head
|
||||
branches".
|
||||
- [ ] **Branches → main**: require a pull request; require the always-on checks
|
||||
`lint`, `pr-title`, both CodeQL `Analyze` jobs and `dependency-review`
|
||||
(`python`, `manifests` and `build-amd64` are path-filtered and must not be
|
||||
required); require branches to be up to date; require conversation
|
||||
resolution; restrict direct pushes; allow only squash merges and enable
|
||||
"Automatically delete head branches".
|
||||
- [ ] **Branches → release/***: same protections, plus allow the release workflow
|
||||
to push tags.
|
||||
- [ ] **Tags → dsh-***: restrict creation/update/deletion to maintainers; a tag
|
||||
|
||||
@@ -47,8 +47,8 @@ docker build \
|
||||
| Arg | Default / source | Purpose |
|
||||
| --- | --- | --- |
|
||||
| `DSH_REPO` | `https://github.com/deepseek-ai/deepseek-harness.git` | Upstream DSH repository to clone. |
|
||||
| `DSH_VERSION` | `DSH_VERSION` in [versions.env](../versions.env) | Upstream DSH git tag/ref to package; recorded as `org.opencontainers.image.version`. |
|
||||
| `RELEASE_REVISION` | `RELEASE_REVISION` in [versions.env](../versions.env) | Repository revision suffix (`rN`); the release identity is `DSH_VERSION-RELEASE_REVISION`, recorded in the `io.sredevopsorg.release.revision` label. See [versioning](versioning.md). |
|
||||
| `DSH_VERSION` | `DSH_VERSION` in [versions.env](../versions.env) | Upstream DSH git tag/ref to package. The Dockerfile records it as `io.sredevopsorg.dsh.version` and (for local builds) `org.opencontainers.image.version`; published builds set the version label to the full release identity. |
|
||||
| `RELEASE_REVISION` | `RELEASE_REVISION` in [versions.env](../versions.env) | Repository revision suffix (`rN`), recorded in `io.sredevopsorg.release.revision`. The release identity is `DSH_VERSION` plus `-RELEASE_REVISION` when set. See [versioning](versioning.md). |
|
||||
| `PNPM_VERSION` | `PNPM_VERSION` in [versions.env](../versions.env) | pnpm version activated with corepack in both stages. |
|
||||
| `VCS_REF` | `git rev-parse --short HEAD` (via `make image`) | Source revision recorded for traceability. |
|
||||
| `NODE_BUILD_IMAGE` | `docker.io/node:24-trixie` | Build-stage base image. |
|
||||
@@ -90,9 +90,10 @@ includes `/app/node_modules/.bin`.
|
||||
the picker's browse backend lists `$HOME` when the dialog opens and labels that
|
||||
listing *Home*. With the image default (`/home/node`) that listing is outside a
|
||||
mounted volume; the example sandbox templates
|
||||
([minimal](../examples/minimal-single-tenant/manifests/sandbox-template.yaml),
|
||||
([minimal](../examples/minimal-single-tenant/manifests/10-sandboxtemplate.yaml),
|
||||
[agents as a service](../examples/dsh-agents-as-a-service/manifests/10-sandboxtemplate.yaml))
|
||||
set `HOME=/workspace` so the picker opens on the persisted agent workspace. `~`
|
||||
mount the same PVC's `workspace` subPath at `/workspace` and set `HOME=/workspace`
|
||||
so the picker opens on the persisted agent workspace. `~`
|
||||
and every tool that keeps caches or config under `$HOME` (`npm`/`pnpm`, `uv`,
|
||||
`git`, `gh`, DSH's global `~/.agents` instructions and skills) follow, so those
|
||||
dotfiles then live on the workspace volume too. `DSH_HOME` is separate and stays
|
||||
@@ -141,7 +142,7 @@ procedure.
|
||||
| --- | --- | --- |
|
||||
| Exposed port | `3080/tcp` | `EXPOSE 3080`; one port serves the UI, assets, API, SSE and WebSocket traffic. |
|
||||
| State volume | `/home/node/.dsh` | Declared with `VOLUME`; use a named volume (Docker) or PVC (Kubernetes). |
|
||||
| Working directory | `/workspace` | The agent workspace. Mount a volume here if agents should keep files; the agents-as-a-service example mounts the same PVC's `workspace` subPath here and points `HOME` at it, so DSH's folder picker opens on it. |
|
||||
| Working directory | `/workspace` | The agent workspace. Mount a volume here if agents should keep files; the shipped example templates mount the same PVC's `workspace` subPath here and point `HOME` at it, so DSH's folder picker opens on it. |
|
||||
| Stop signal | `SIGTERM` | `tini` forwards it to `dsh web`; allow a graceful shutdown period. |
|
||||
| User | `node`, UID/GID `1000` | The process is non-root. A PVC mounted at `DSH_HOME` must be writable by UID 1000. |
|
||||
|
||||
@@ -164,8 +165,8 @@ livenessProbe:
|
||||
periodSeconds: 15
|
||||
```
|
||||
|
||||
The shipped templates use exactly this shape; see
|
||||
[`examples/minimal-single-tenant/manifests/sandbox-template.yaml`](../examples/minimal-single-tenant/manifests/sandbox-template.yaml).
|
||||
The shipped templates use the same path and startup budget; see
|
||||
[`examples/minimal-single-tenant/manifests/10-sandboxtemplate.yaml`](../examples/minimal-single-tenant/manifests/10-sandboxtemplate.yaml).
|
||||
A `startupProbe` is important because plugin activation (or the first start of a
|
||||
large sandbox) can exceed a default liveness budget.
|
||||
|
||||
@@ -199,10 +200,7 @@ GitHub Container Registry:
|
||||
| --- | --- |
|
||||
| `dsh-v0.2.1-rc.5` | Current release identity. **Immutable**: once published it never moves. |
|
||||
| `latest` | Most recent release; points at the same digest as the current identity. |
|
||||
| `main` | Development channel rebuilt from every push to `main`; not a release, not for production. |
|
||||
|
||||
Tags published before the release-identity scheme remain published but are not
|
||||
updated; new deployments use the current identity.
|
||||
| `main` | Development channel rebuilt when image-definition files change on `main`; not a release, not for production. |
|
||||
|
||||
Release identities, channels and rollback rules are defined in
|
||||
[versioning](versioning.md). Tags are immutable, so production deployments should
|
||||
@@ -220,6 +218,7 @@ docker buildx imagetools inspect ghcr.io/sredevopsorg/deepseek-kubernetes-sandbo
|
||||
docker inspect ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.5 \
|
||||
--format '{{json .Config.Labels}}'
|
||||
# org.opencontainers.image.title / .version / .source
|
||||
# io.sredevopsorg.dsh.version / io.sredevopsorg.release.revision
|
||||
```
|
||||
|
||||
Release builds also publish SBOM and provenance attestations from the release
|
||||
|
||||
@@ -15,7 +15,7 @@ release identity = DSH_VERSION + optional "-" + RELEASE_REVISION
|
||||
|
||||
| Piece | Meaning |
|
||||
| --- | --- |
|
||||
| `DSH_VERSION` | The upstream DeepSeek Harness git ref being packaged, e.g. `dsh-v0.2.1-rc.5`. It mirrors upstream's own tag, which is what "mirrored-DSH scheme" means. |
|
||||
| `DSH_VERSION` | The upstream DeepSeek Harness git ref being packaged, e.g. `dsh-v0.2.1-rc.5`. It mirrors upstream's own tag. |
|
||||
| `RELEASE_REVISION` | An optional repository revision at that DSH ref: empty for the first release, `rN` (`N >= 2`) for a repository-only re-release. |
|
||||
|
||||
Examples:
|
||||
@@ -33,23 +33,14 @@ with an empty revision); a repository-only change at the same DSH ref increments
|
||||
## Why release identities are immutable
|
||||
|
||||
**Container tags are immutable** in this project: re-pushing a tag would silently
|
||||
change what an existing deployment runs. A DSH upgrade therefore produces a new
|
||||
base identity with an empty revision; a repository-only change at an
|
||||
already-released ref publishes `-rN`, where `r2` is the first re-release (the
|
||||
plain tag is conceptually revision 1 and is taken).
|
||||
change what an existing deployment runs. A fix therefore publishes a new identity;
|
||||
see the rules above.
|
||||
|
||||
The current identity is `dsh-v0.2.1-rc.5`, the first release at this upstream
|
||||
ref, so it carries no revision suffix. The identity is stored in
|
||||
[versions.env](../versions.env) and consumed everywhere: the Dockerfile label,
|
||||
the example manifests, the Makefile and the release workflow. Tags published
|
||||
before the release-identity scheme remain published but are not updated; new
|
||||
deployments use the current identity.
|
||||
|
||||
The gateway image moved from the repository-level `:gateway` tag to a nested
|
||||
package (`<repository>/dsh-agents-as-a-service/gateway`). The release workflow
|
||||
also republishes the legacy `<repository>:gateway` tag as a deprecated
|
||||
compatibility alias; it may be removed in a later release, so deployments
|
||||
should pin the nested package.
|
||||
[versions.env](../versions.env) and consumed everywhere: the Dockerfile labels
|
||||
(version and revision), the example manifests, the Makefile and the release
|
||||
workflow.
|
||||
|
||||
## Channels
|
||||
|
||||
@@ -57,7 +48,7 @@ should pin the nested package.
|
||||
| --- | --- | --- |
|
||||
| Release | `dsh-v0.2.1-rc.5` | The current immutable release identity. What production should pin (preferably by digest). |
|
||||
| `latest` | `latest` | Alias for the most recent release; moves only when a new release is published. |
|
||||
| `main` | `main` | Development channel, rebuilt from every push to `main`. Never treat it as a release; it exists so `main` can be tested. |
|
||||
| `main` | `main` | Development channel, rebuilt when image or gateway sources change on `main`. Never treat it as a release; it exists so `main` can be tested. |
|
||||
|
||||
The same channels exist for every published package, including the nested
|
||||
example component image.
|
||||
@@ -110,11 +101,12 @@ workflows with a literal version.
|
||||
matches [versions.env](../versions.env).
|
||||
6. The workflow builds and pushes multi-arch images for the main image and the
|
||||
example components (the identity tag plus `latest`), creates the GitHub
|
||||
Release from the git-cliff notes, records the published digests, and opens a
|
||||
`chore(changelog)` PR that refreshes `CHANGELOG.md` on `main`.
|
||||
Release from the git-cliff notes, and opens a `chore(changelog)` PR that
|
||||
refreshes `CHANGELOG.md` on `main`. The publish log shows each image manifest
|
||||
and digest.
|
||||
7. Merge the changelog PR. Verify:
|
||||
`docker buildx imagetools inspect <image>:<identity>` shows both platforms,
|
||||
and the GitHub Release references the tag and the digest.
|
||||
`docker buildx imagetools inspect <image>:<identity>` shows both platforms;
|
||||
the GitHub Release references the tag, and the workflow log shows the digest.
|
||||
8. Announce the identity and the digest; deployments pin the digest.
|
||||
|
||||
## Rollback
|
||||
|
||||
@@ -11,7 +11,7 @@ reference material, not libraries.
|
||||
| [`examples/minimal-single-tenant/`](minimal-single-tenant/) | The smallest useful setup: a single `SandboxTemplate` and one `SandboxClaim`, no gateway — reach DSH Web with a launch-token URL through `kubectl port-forward`. | agent-sandbox with extensions. |
|
||||
|
||||
The multi-tenant example also documents opt-in gVisor, `nodeSelector`,
|
||||
NetworkPolicy and extra provider-key hardening in
|
||||
NetworkPolicy, extra provider-key hardening and failed-Pod recovery in
|
||||
[optional hardening](dsh-agents-as-a-service/docs/optional-hardening.md).
|
||||
|
||||
## The example contract (short version)
|
||||
@@ -19,37 +19,26 @@ NetworkPolicy and extra provider-key hardening in
|
||||
Full rules: [docs/examples.md](../docs/examples.md). `scripts/check-examples.sh`
|
||||
enforces the layout, and `make check` runs it.
|
||||
|
||||
```
|
||||
examples/<name>/
|
||||
+-- README.md # walkthrough: prerequisites, steps, troubleshooting, cleanup
|
||||
+-- manifests/ # NN-*.yaml, applied with plain kubectl
|
||||
| +-- optional/<feature>/ # optional opt-in manifests...
|
||||
| +-- optional/README.md # ...required whenever optional/ exists
|
||||
| +-- exposure/<variant>/ # optional per-variant manifests...
|
||||
| +-- exposure/README.md # ...required whenever exposure/ exists
|
||||
+-- scripts/ # optional executable, idempotent, self-cleaning *.sh
|
||||
+-- docs/ # optional architecture/ADR material
|
||||
+-- <component>/ # optional component with Dockerfile + README + tests
|
||||
```
|
||||
|
||||
- Secrets never ship populated: use `*.yaml.example` files with placeholders.
|
||||
- Manifests are numbered (`00-`, `10-`, ...) and read as an ordered walkthrough.
|
||||
- Component images are published nested below the main image:
|
||||
`<registry>/<owner>/<repo>/<example>/<component>`.
|
||||
- Every example must be listed above with its literal `examples/<name>/` path.
|
||||
- Scripts under `scripts/` must be executable, idempotent and clean up after
|
||||
themselves.
|
||||
- `README.md` is the walkthrough; `manifests/NN-*.yaml` is applied with plain
|
||||
kubectl and read in order.
|
||||
- Optional `manifests/optional/<feature>/` and `manifests/exposure/<variant>/`
|
||||
each ship a README; exposure variants are mutually exclusive.
|
||||
- Optional `scripts/` must be executable, idempotent and self-cleaning; `docs/`
|
||||
holds deeper material.
|
||||
- Components with a Dockerfile need a README and tests; images are published
|
||||
nested below the main image (`<registry>/<owner>/<repo>/<example>/<component>`).
|
||||
- Secrets are `*.yaml.example` with placeholders; every example is listed above
|
||||
with its literal `examples/<name>/` path.
|
||||
|
||||
## Running an example
|
||||
|
||||
```bash
|
||||
# e.g. the multi-tenant platform example
|
||||
kubectl apply -f examples/dsh-agents-as-a-service/manifests/
|
||||
Each example README has the exact prerequisites, commands, optional hardening,
|
||||
exposure options and teardown. The multi-tenant example also has a full kind
|
||||
walkthrough that cleans up after itself:
|
||||
|
||||
# or run the whole walkthrough (kind + agent-sandbox required) and let it clean up
|
||||
```bash
|
||||
examples/dsh-agents-as-a-service/scripts/e2e-kind.sh
|
||||
```
|
||||
|
||||
`kubectl apply -f manifests/` is non-recursive: `optional/` and `exposure/`
|
||||
subdirectories are applied explicitly. Each example README has the exact
|
||||
prerequisites, optional hardening, exposure options and teardown.
|
||||
subdirectories are applied explicitly.
|
||||
|
||||
@@ -8,6 +8,8 @@ personal, long-lived DSH Web instance that is **provisioned in ~2 seconds from a
|
||||
warm pool, suspended when idle to save cost, and woken transparently when the
|
||||
user returns** — with all state (conversations, sessions, config, and the agent
|
||||
workspace) surviving on the per-user PVC at `/home/node/.dsh` + `/workspace`.
|
||||
Suspend-when-idle plus the warm pool keep cost proportional to use: a return
|
||||
after absence pays a short cold-start pause, not an always-on instance.
|
||||
|
||||
It ships in three layers:
|
||||
|
||||
@@ -79,12 +81,14 @@ What it demonstrates, feature by feature:
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Run the walkthrough from this example's directory (`examples/dsh-agents-as-a-service/`).
|
||||
|
||||
- A Kubernetes cluster with a default StorageClass (kind and GKE both work as-is).
|
||||
- agent-sandbox with extensions installed:
|
||||
|
||||
```bash
|
||||
VERSION=v1.0.8 # AGENT_SANDBOX_VERSION in versions.env
|
||||
kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/download/$VERSION/sandbox-with-extensions.yaml
|
||||
. ../../versions.env
|
||||
kubectl apply -f "https://github.com/kubernetes-sigs/agent-sandbox/releases/download/${AGENT_SANDBOX_VERSION}/sandbox-with-extensions.yaml"
|
||||
kubectl -n agent-sandbox-system rollout status deployment --timeout=180s
|
||||
```
|
||||
|
||||
@@ -243,12 +247,9 @@ kubectl -n dsh-demo exec "$SB" -- cat /home/node/.dsh/marker.txt # "remember m
|
||||
kubectl -n dsh-demo exec "$SB" -- cat /workspace/marker.txt # "remember me too"
|
||||
```
|
||||
|
||||
> **Why resume works:** the pod template sets `fsGroupChangePolicy: OnRootMismatch`.
|
||||
> DSH requires its `.credentials.yaml` to be `0600`, while the kubelet's default
|
||||
> fsGroup policy widens every file to `0660` on each pod start — a resumed sandbox
|
||||
> would crash-loop. See [Troubleshooting](#troubleshooting) for the one-time fix
|
||||
> on sandboxes created before this setting existed. The `/workspace` mount does not
|
||||
> change that: `subPath` is a separate directory, and only the volume root is walked.
|
||||
> **Why resume works:** the pod template sets `fsGroupChangePolicy: OnRootMismatch`;
|
||||
> without it the kubelet widens `.credentials.yaml` to `0660` and DSH crash-loops.
|
||||
> See [Troubleshooting](#troubleshooting) for the one-time fix on older sandboxes.
|
||||
|
||||
## 7. Injection policy rejects env-carrying claims
|
||||
|
||||
@@ -273,7 +274,7 @@ kind load docker-image "$GW" --name <kind-cluster>
|
||||
|
||||
# ... or build it locally and tag it exactly as manifests/50-gateway.yaml
|
||||
# expects (make gateway builds dsh-aaas-gateway:dev):
|
||||
make gateway
|
||||
make -C ../.. gateway
|
||||
docker tag dsh-aaas-gateway:dev "$GW"
|
||||
kind load docker-image "$GW" --name <kind-cluster>
|
||||
|
||||
@@ -351,44 +352,21 @@ API clients that prefer bearer tokens can use `POST /users` with a JSON body
|
||||
(`{"user":"bob"}`) and the returned one-time token, `POST /login` to exchange a
|
||||
token for a cookie, and `Authorization: Bearer <token>` on subsequent calls.
|
||||
|
||||
### Gateway API
|
||||
### Gateway routes
|
||||
|
||||
| Route | Purpose |
|
||||
| --- | --- |
|
||||
| `GET /healthz` | Liveness/readiness; no cluster access |
|
||||
| `GET /` | Landing page: signup, token login, logout. With SSO it auto-provisions and connects instead. |
|
||||
| `POST /users` | Signup. JSON or form; `201` (JSON), `202` (waking), `303` (browser redirect), `409` if user exists; sets the session cookie |
|
||||
| `POST /login` | Exchange a bearer token for the session cookie |
|
||||
| `POST /logout` | Clear the session cookie |
|
||||
| `POST /recreate` | SSO only: delete the authenticated user's claim (wiping workspace data) and provision a fresh one; `303` to `/u/{user}/`. Cookie-authenticated calls must be same-origin. |
|
||||
| `GET /users/{user}` | Derived state (`Ready`/`Waking`/`Suspended`) |
|
||||
| `DELETE /users/{user}` | Delete claim (cascades sandbox + PVC) |
|
||||
| `GET /u/{user}` | `308` redirect to `/u/{user}/` |
|
||||
| `GET /u/{user}/**` | HTTP/SSE proxy to the user's DSH Web |
|
||||
| `WS /u/{user}/**` | WebSocket proxy to the user's DSH Web |
|
||||
|
||||
The full API reference, configuration table, auth semantics and failure matrix
|
||||
live in the [gateway README](gateway/README.md).
|
||||
The browser flow above uses the gateway's main routes. The full route table,
|
||||
status codes, configuration and failure matrix live in the
|
||||
[gateway README](gateway/README.md).
|
||||
|
||||
### How the DSH handshake works
|
||||
|
||||
DSH does not expose an API for its launch token; it prints a URL containing
|
||||
`?token=…` in the pod logs. For each `(user, pod UID, Host)` the gateway:
|
||||
|
||||
1. resolves the claim → sandbox → pod (annotation `agents.x-k8s.io/pod-name`, or
|
||||
the sandbox name as pod name),
|
||||
2. reads the pod's logs (last `DSH_LOG_TAIL_LINES` lines) and parses the most
|
||||
recent `dsh web: …?token=…` line,
|
||||
3. requests `http://<serviceFQDN>:3080/?token=…` with the browser's `Host`,
|
||||
4. captures DSH's `dsh-auth-*` `Set-Cookie` response and caches it,
|
||||
5. injects that cookie (and strips the browser's `Cookie`/`Authorization` and any
|
||||
`?token=`) on every proxied HTTP and WebSocket request,
|
||||
6. refreshes the handshake once if DSH answers `401`, and drops the cached cookie
|
||||
when the sandbox suspends or is deleted.
|
||||
|
||||
This is why the gateway's RBAC includes `pods` `get`/`list` and `pods/log`
|
||||
`get`. It is also the most fragile part of the design; see
|
||||
[Limitations](#limitations).
|
||||
DSH prints its launch token only in pod logs. The gateway resolves
|
||||
claim → sandbox → pod, reads the pod-log tail, performs the `?token=` handshake
|
||||
under the browser's `Host`, caches `dsh-auth-*` per `(user, pod UID, Host)`, and
|
||||
refreshes once on a `401`. That is why its RBAC includes `pods` `get`/`list` and
|
||||
`pods/log` `get`; it is also the most fragile part of the design (see
|
||||
[Limitations](#limitations) and the
|
||||
[architecture ADR](docs/architecture.md#adr-3-server-side-launch-token-handshake-instead-of-exposing-the-token)).
|
||||
|
||||
### DSH_TRUSTED_HOSTS
|
||||
|
||||
@@ -407,27 +385,18 @@ The same applies to a local `docker run`: the printed token URL uses
|
||||
|
||||
### WebSocket
|
||||
|
||||
The gateway is a Starlette ASGI app and proxies WebSocket upgrades with a
|
||||
bidirectional pump, so the DSH web GUI is fully interactive. Requirements:
|
||||
|
||||
- the ingress/load balancer in front of the gateway must pass `Upgrade`/`Connection`
|
||||
The gateway proxies WebSocket upgrades with a bidirectional pump, so the DSH web
|
||||
GUI is fully interactive. The entry point must pass `Upgrade`/`Connection`
|
||||
headers; [`manifests/exposure/ingress/60-ingress.yaml`](manifests/exposure/ingress/60-ingress.yaml)
|
||||
is a starting point (Traefik handles upgrades by default),
|
||||
- idle suspension is disabled for users with in-flight proxied requests or open
|
||||
WebSockets, so a long-running chat stream is not cut off mid-turn,
|
||||
- the upstream upgrade dials the sandbox **pod IP**, forwards no `Host`/`Origin`,
|
||||
and performs the launch-token handshake under that same `pod-IP:3080` authority.
|
||||
DSH fences on `Host` (only its bind address, loopback, or an allow-listed host
|
||||
passes) and binds its auth cookie to the request authority
|
||||
(`payload.authority === authority`), while `websockets` derives `Host` from the
|
||||
URI and cannot be overridden. A browser-host cookie or forwarded origin makes
|
||||
the upgrade fail (`401`/`403`, surfaced as close `1011`), which the client
|
||||
retries forever as `[connection] connection lost, retry #N`,
|
||||
- missing/invalid gateway auth is rejected right after the upgrade with close code
|
||||
`4401` (WebSocket auth has no HTTP `401` path); cross-origin or Origin-less
|
||||
upgrades with the cookie are rejected the same way (CSWSH protection), so
|
||||
clients must send a matching `Origin`; upstream bridging failures close with
|
||||
`1011`.
|
||||
is a starting point (Traefik handles upgrades by default). Idle suspension is
|
||||
disabled while a proxied request or socket is in flight, so a long-running turn is
|
||||
not cut off.
|
||||
|
||||
The upstream upgrade dials the sandbox **pod IP**, forwards no `Host`/`Origin`,
|
||||
and performs the launch-token handshake under the same `pod-IP:3080` authority.
|
||||
Missing or cross-origin auth closes with `4401`; upstream bridging failures close
|
||||
with `1011`. See the [gateway README](gateway/README.md#proxy-failure-semantics)
|
||||
for the full matrix.
|
||||
|
||||
### Path routing
|
||||
|
||||
@@ -457,27 +426,18 @@ and the gateway. They are part of the variant; see the
|
||||
|
||||
### Option A — plain Ingress (gateway's own session auth)
|
||||
|
||||
```bash
|
||||
# Replace the dsh.example.com host and TLS Secret (or drop the tls: block for
|
||||
# plain HTTP), then:
|
||||
kubectl apply -f manifests/exposure/ingress/
|
||||
```
|
||||
|
||||
1. Set `DSH_TRUSTED_HOSTS` in `manifests/10-sandboxtemplate.yaml` to the same
|
||||
public host (the gateway forwards the browser's `Host`) and re-apply the
|
||||
template.
|
||||
2. Set `COOKIE_SECURE=true` on the gateway once the Ingress serves HTTPS:
|
||||
1. Set the Ingress host/TLS (or drop the `tls:` block), and set
|
||||
`DSH_TRUSTED_HOSTS` in `manifests/10-sandboxtemplate.yaml` to the same public
|
||||
host; re-apply the template.
|
||||
2. Set `COOKIE_SECURE=true` once the Ingress serves HTTPS:
|
||||
`kubectl -n dsh-demo set env deploy/dsh-gateway COOKIE_SECURE=true`.
|
||||
(The base manifest keeps `false` so the plain-HTTP port-forward demo works.)
|
||||
3. Verify: `curl -sI https://<host>/healthz` returns `200`; sign up in a browser
|
||||
at `https://<host>/`.
|
||||
4. `manifests/exposure/ingress/61-networkpolicy.yaml` is part of this variant:
|
||||
it allows only Traefik pods to reach the gateway on `8080`. Adjust its
|
||||
selector if your ingress controller uses a different pod label or
|
||||
`hostNetwork`; if kubelet probes fail, see the hardening doc.
|
||||
3. `kubectl apply -f manifests/exposure/ingress/`; verify
|
||||
`curl -sI https://<host>/healthz` returns `200`.
|
||||
|
||||
Keep the router free of buffering/compression middlewares: they break WebSocket
|
||||
upgrades and streaming.
|
||||
upgrades and streaming. The variant's `61-networkpolicy.yaml` allows only Traefik
|
||||
pods to reach the gateway; see the [exposure README](manifests/exposure/README.md)
|
||||
for selector and probe caveats.
|
||||
|
||||
### Option B — Gateway API + GitHub SSO (oauth2-proxy)
|
||||
|
||||
@@ -487,84 +447,41 @@ front of **oauth2-proxy**, which authenticates the browser against GitHub
|
||||
`X-Forwarded-User` set. The gateway trusts that header only when
|
||||
`AUTH_PROXY_HEADER` is set (SSO is **opt-in**).
|
||||
|
||||
Prerequisites:
|
||||
|
||||
- Gateway API CRDs and a Gateway implementation; the manifests assume
|
||||
`gatewayClassName: traefik` (the CRD version is pinned as
|
||||
`GATEWAY_API_VERSION` in `versions.env`).
|
||||
- A GitHub OAuth App whose homepage/callback match the public FQDN
|
||||
(`https://<host>/` and `https://<host>/oauth2/callback`).
|
||||
- The oauth2-proxy credentials Secret (template:
|
||||
`73-oauth2-proxy-secret.yaml.example`) and a TLS Secret for the Gateway
|
||||
listener (template: `74-tls-secret.yaml.example`).
|
||||
|
||||
Steps:
|
||||
|
||||
```bash
|
||||
# 1. Real Secrets; the committed *.yaml.example files are templates only.
|
||||
kubectl -n dsh-demo create secret generic oauth2-proxy \
|
||||
--from-literal=client-id='<GitHub OAuth App client ID>' \
|
||||
--from-literal=client-secret='<GitHub OAuth App client secret>' \
|
||||
--from-literal=cookie-secret="$(python3 -c 'import secrets; print(secrets.token_urlsafe(32))')"
|
||||
kubectl -n dsh-demo create secret tls dsh-demo-tls --cert=tls.crt --key=tls.key
|
||||
|
||||
# 2. Replace dsh-demo.example.com (70/71/72) and your-github-org (72).
|
||||
|
||||
# 3. Enable SSO on the gateway and trust the public host in the template.
|
||||
kubectl -n dsh-demo set env deploy/dsh-gateway \
|
||||
AUTH_PROXY_HEADER=X-Forwarded-User COOKIE_SECURE=true
|
||||
# ... and set DSH_TRUSTED_HOSTS=dsh-demo.example.com in
|
||||
# manifests/10-sandboxtemplate.yaml, then re-apply the template.
|
||||
|
||||
# 4. Apply the variant.
|
||||
kubectl apply -f manifests/exposure/gateway-api-sso/
|
||||
```
|
||||
|
||||
Verification:
|
||||
|
||||
```bash
|
||||
kubectl get gateway dsh-gateway -n dsh-demo -o yaml # Programmed=True
|
||||
kubectl get httproute -n dsh-demo -o wide # backend oauth2-proxy:4180, Accepted
|
||||
kubectl get deploy oauth2-proxy -n dsh-demo # 1/1 Ready
|
||||
kubectl get endpoints oauth2-proxy -n dsh-demo # backend present
|
||||
kubectl -n dsh-demo logs deploy/oauth2-proxy --tail=50 # no config errors
|
||||
```
|
||||
|
||||
Browser check: open `https://<host>/` — you are redirected to GitHub, and after
|
||||
signing in as a member of the allowed organization the gateway auto-provisions
|
||||
your sandbox (the first page can show `Provisioning`) and connects you. A
|
||||
non-member is denied before reaching the gateway. GitHub logins are lowercase
|
||||
DNS-1123 labels, so they map 1:1 onto the gateway's `user` validation. The SSO
|
||||
landing page also offers **Recreate sandbox** (`POST /recreate`), which deletes
|
||||
the claim — wiping the workspace PVC — and provisions a fresh one.
|
||||
|
||||
Sign-out: the gateway `/logout` clears `dsh_gw_session`; oauth2-proxy's session
|
||||
can be cleared at `/oauth2/sign_out`.
|
||||
1. Install Gateway API CRDs and a Gateway implementation; the manifests assume
|
||||
`gatewayClassName: traefik` (the CRD version is `GATEWAY_API_VERSION` in
|
||||
`versions.env`).
|
||||
2. Create a GitHub OAuth App whose homepage/callback match the public FQDN, and
|
||||
create the oauth2-proxy and TLS Secrets from
|
||||
`73-oauth2-proxy-secret.yaml.example` and `74-tls-secret.yaml.example`.
|
||||
3. Replace `dsh-demo.example.com` and `your-github-org`, set
|
||||
`DSH_TRUSTED_HOSTS` in the template, and enable SSO:
|
||||
`kubectl -n dsh-demo set env deploy/dsh-gateway AUTH_PROXY_HEADER=X-Forwarded-User COOKIE_SECURE=true`.
|
||||
4. `kubectl apply -f manifests/exposure/gateway-api-sso/`; verify the Gateway,
|
||||
HTTPRoute, oauth2-proxy Deployment and endpoints as shown in the
|
||||
[exposure README](manifests/exposure/README.md).
|
||||
|
||||
> **SSO trust boundary — read this before enabling it.** After SSO is on, the
|
||||
> `dsh-gateway` Service must be reachable **only** through oauth2-proxy (it is
|
||||
> ClusterIP). The variant's `75-networkpolicies.yaml` restricts pod traffic
|
||||
> accordingly: the gateway accepts traffic only from oauth2-proxy pods,
|
||||
> oauth2-proxy only from the Gateway implementation, and oauth2-proxy egress is
|
||||
> limited to DNS, public HTTPS and the gateway. Anyone who can reach
|
||||
> `dsh-gateway:8080` directly can still send `X-Forwarded-User: <victim>` and be
|
||||
> treated as that user, so never add a second path (the Ingress variant, a
|
||||
> shared-host port-forward, a permissive policy). To turn SSO off, remove the
|
||||
> variable (`kubectl -n dsh-demo set env deploy/dsh-gateway AUTH_PROXY_HEADER-`)
|
||||
> and delete the SSO variant; the gateway falls back to bearer-token/session
|
||||
> auth.
|
||||
> oauth2-proxy only from the Gateway implementation, and its egress only to DNS,
|
||||
> public HTTPS and the gateway. Anyone who can reach `dsh-gateway:8080` directly
|
||||
> can send `X-Forwarded-User: <victim>`, so never add a second path. To turn SSO
|
||||
> off, remove the variable
|
||||
> (`kubectl -n dsh-demo set env deploy/dsh-gateway AUTH_PROXY_HEADER-`) and
|
||||
> delete the SSO variant.
|
||||
|
||||
### Switching variants
|
||||
|
||||
Delete the variant you had before applying the other one
|
||||
(`kubectl delete -f manifests/exposure/ingress/`), revert
|
||||
Delete the variant you had before applying the other one, revert
|
||||
`AUTH_PROXY_HEADER`/`COOKIE_SECURE`, and verify with
|
||||
`kubectl -n dsh-demo get ingress,gateway,httproute` that exactly one path to the
|
||||
gateway remains.
|
||||
gateway remains. Full commands and verification are in the
|
||||
[exposure README](manifests/exposure/README.md).
|
||||
|
||||
## Optional hardening
|
||||
|
||||
The base walkthrough is deliberately portable. Four opt-in isolation layers are
|
||||
The base walkthrough is deliberately portable. Five opt-in settings are
|
||||
documented in [optional hardening](docs/optional-hardening.md):
|
||||
|
||||
- **gVisor** — uncomment `runtimeClassName: gvisor` before applying, or patch an
|
||||
@@ -582,7 +499,7 @@ documented in [optional hardening](docs/optional-hardening.md):
|
||||
the sandbox at `Finished=True`. Requires agent-sandbox v1.0.8+.
|
||||
|
||||
Apply each independently and re-run the walkthrough. `kubectl apply -f manifests/`
|
||||
is non-recursive, so the optional manifest is not applied by the base step.
|
||||
is non-recursive, so the optional manifests are not applied by the base step.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
@@ -599,7 +516,7 @@ is non-recursive, so the optional manifest is not applied by the base step.
|
||||
| `manifest.webmanifest` returns `401` in the console | Browsers fetch the Web App Manifest without credentials, so the gateway's session cookie is absent and the request fails closed. The UI is unaffected; only the PWA install prompt is lost |
|
||||
| Signup is slow / `202` then `Waking` | Warm pool exhausted (`kubectl get sandboxes -l agents.x-k8s.io/warm-pool-sandbox`); scale the pool |
|
||||
| `403` from the Kubernetes API in gateway logs | The gateway ServiceAccount is missing claim/sandbox/pod-log verbs (see `manifests/50-gateway.yaml`) |
|
||||
| Gateway pod `ErrImagePull` / `ImagePullBackOff` (especially on kind) | The manifest uses `imagePullPolicy: IfNotPresent` with the release identity. Build with `make gateway`, retag to exactly the manifest's image, `kind load docker-image <that-ref>`, or set the Deployment image to your local tag / pinned digest. On a real cluster, make sure the GHCR package is readable (public or a pull secret) |
|
||||
| Gateway pod `ErrImagePull` / `ImagePullBackOff` (especially on kind) | The manifest uses `imagePullPolicy: IfNotPresent` with the release identity. Build with `make -C ../.. gateway`, retag to exactly the manifest's image, `kind load docker-image <that-ref>`, or set the Deployment image to your local tag / pinned digest. On a real cluster, make sure the GHCR package is readable (public or a pull secret) |
|
||||
| Step 3: `pod "$SB" not found` after deploying the gateway | The claim was suspended by the idle sweeper. Claims created by the gateway carry `aaas.example.com/managed-by=dsh-gateway`; hand-applied claims (steps 1–7) do not and are never swept. Resume the sandbox: `kubectl -n dsh-demo patch sandbox "$SB" --type merge -p '{"spec":{"operatingMode":"Running"}}'` |
|
||||
| Resume crash-loops: `credentials-local: … is readable beyond its owner (mode 660)` | The pod template lacks `fsGroupChangePolicy: OnRootMismatch`, so the kubelet's default fsGroup policy rewrote `.credentials.yaml` to `0660`; DSH requires `0600` and refuses to start. New installs get the policy from the template. For an existing sandbox: suspend it, run a throwaway pod on `dsh-home-$SB` to `chmod 600 /home/node/.dsh/.credentials.yaml`, patch the sandbox's `podTemplate` with the policy, then resume |
|
||||
| Agents get `Permission denied` writing to `/workspace` on a fresh or resumed pod | The `/workspace` mount is the `dsh-home` PVC's `workspace/` subPath, and the kubelet is known to leave a `subPath` directory at `0755` if it is interrupted between creating and chmodding it ([kubernetes#142464](https://github.com/kubernetes/kubernetes/issues/142464)); later mounts never correct it. Suspend the sandbox, run a throwaway pod on `dsh-home-$SB` mounted at `/data`, `chmod 2775 /data/workspace` (do **not** chown/chmod recursively — that rewrites `.credentials.yaml`), then resume. If you recreate the sandbox instead, its current `/workspace` contents go with the PVC |
|
||||
@@ -666,12 +583,6 @@ Deliberately **not** production code:
|
||||
- **Bearer tokens in URLs.** Avoid putting the signup token in a URL/log; the
|
||||
browser flow uses the `HttpOnly` cookie instead.
|
||||
|
||||
## Cost model
|
||||
|
||||
The same economics as the upstream pattern: suspend-when-idle plus warm pools turn
|
||||
an always-on instance into a sub-second cold-start pause per return-after-absence,
|
||||
at a fraction of the cost.
|
||||
|
||||
## Cleanup
|
||||
|
||||
```bash
|
||||
|
||||
@@ -6,7 +6,8 @@ Kubernetes, and DSH Web fit together, and why the design looks the way it does.
|
||||
|
||||
Scope: one Kubernetes namespace (`dsh-demo`), one shared DSH Web image, one gateway deployment,
|
||||
and one DSH Web pod + PVC per user. The cluster is assumed to run
|
||||
[agent-sandbox](https://github.com/kubernetes-sigs/agent-sandbox) v1.0.8 with the extensions
|
||||
[agent-sandbox](https://github.com/kubernetes-sigs/agent-sandbox) with the pinned
|
||||
`AGENT_SANDBOX_VERSION` from [`versions.env`](../../../versions.env) and the extensions
|
||||
(`SandboxTemplate`, `SandboxWarmPool`, `SandboxClaim`).
|
||||
|
||||
## Container view
|
||||
@@ -340,7 +341,7 @@ Notes and limits:
|
||||
| Probe on `/` | Pod `CrashLoopBackOff`, `401` in probe logs | Use `/favicon.svg` for readiness/liveness (shipped templates do) |
|
||||
| Warm pool exhausted | JSON signup returns `202` (state `Waking`); form signup is slow | Increase `replicas` in the warm pool |
|
||||
| Gateway restart | Idle sandboxes stop suspending; handshake cache lost | Rebuilt lazily on next request; consider a shared store for production |
|
||||
| Browser aborts a request (tab reload, back navigation) during a slow wake or handshake | No response reaches the client; nothing appears in the access log | If the abort lands before the response starts, the body read raises `ClientDisconnect` and the app answers `499` (`CLIENT_CLOSED_REQUEST`), which cannot reach the departed peer. Nothing reached DSH, no ERROR traceback is logged, and the in-flight slot is released. A request already streaming ends with its upstream status. Before the fix every mid-body abort logged a `ClientDisconnect` traceback |
|
||||
| Browser aborts a request (tab reload, back navigation) during a slow wake or handshake | No response reaches the client; nothing appears in the access log | If the abort lands before the response starts, the body read raises `ClientDisconnect` and the app answers `499` (`CLIENT_CLOSED_REQUEST`), which cannot reach the departed peer. Nothing reached DSH, no ERROR traceback is logged, and the in-flight slot is released. A request already streaming ends with its upstream status. |
|
||||
| DSH auth cookie expired mid-session | Proxy `401` once, then recovers | Gateway refreshes the handshake once |
|
||||
| Upstream pod name changed after resume | Cached cookie keyed by old pod UID is useless | Cache is invalidated on suspend; next request re-handshakes |
|
||||
| Two gateway replicas | Double suspend/wake races | Keep `replicas: 1` until state is externalized |
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
The base example is deliberately portable: it runs on kind and GKE without a
|
||||
gVisor runtime, a tainted node pool or a NetworkPolicy-capable CNI, and it
|
||||
boots with only the required `dsh-provider-keys` Secret. This document covers
|
||||
four **opt-in** isolation layers, plus one availability setting:
|
||||
four **opt-in** controls plus failed-Pod recovery:
|
||||
|
||||
| Setting | Where it lives | What it changes |
|
||||
| --- | --- | --- |
|
||||
@@ -52,7 +52,7 @@ PodSpec; no CRD change is involved.
|
||||
- [`manifests/50-gateway.yaml`](../manifests/50-gateway.yaml)
|
||||
- [`manifests/exposure/gateway-api-sso/72-oauth2-proxy.yaml`](../manifests/exposure/gateway-api-sso/72-oauth2-proxy.yaml)
|
||||
(only with the SSO variant)
|
||||
- [minimal sandbox-template.yaml](../../minimal-single-tenant/manifests/sandbox-template.yaml)
|
||||
- [minimal template](../../minimal-single-tenant/manifests/10-sandboxtemplate.yaml)
|
||||
|
||||
**Existing install:** a template edit does not update existing Sandboxes, and
|
||||
warm-pool spares use `OnReplenish`, so patch explicitly. Plan a maintenance
|
||||
@@ -604,7 +604,7 @@ Required locally for any change to this example:
|
||||
```bash
|
||||
make check # versions, example layout, manifest YAML shape
|
||||
yamllint .
|
||||
make test # gateway unit tests (gateway code is untouched)
|
||||
make test # only when the gateway code changed
|
||||
```
|
||||
|
||||
CI additionally parses every manifest and runs `kubectl apply
|
||||
|
||||
@@ -22,7 +22,7 @@ It does four things:
|
||||
The DSH version pin lives in [`versions.env`](../../../versions.env)
|
||||
(`DSH_VERSION` plus the optional repository `RELEASE_REVISION`); the example
|
||||
manifests and published images use the release identity built from it (currently
|
||||
`dsh-v0.2.1-rc.5`). DSH binds `:3080`, serves `/`, gates `/api` behind a
|
||||
`dsh-v0.2.1-rc.5`). DSH binds `:3080`, gates the UI and `/api` behind a
|
||||
`dsh-auth-*` cookie and requires a WebSocket at `/api/remote.mux`.
|
||||
|
||||
## Flow
|
||||
@@ -174,13 +174,17 @@ can also be injected into `create_app` for tests.
|
||||
| `SWEEP_INTERVAL` | `15` | Seconds between idle sweeps. |
|
||||
| `AUTH_PROXY_HEADER` | unset | Opt-in SSO: name of the trusted identity header set by an authenticating reverse proxy (e.g. `X-Forwarded-User`). Unset disables SSO and the header is never trusted. Enable only when the Service is reachable exclusively through that proxy (see [SSO](#sso-opt-in)). |
|
||||
|
||||
The shipped `manifests/50-gateway.yaml` sets `IDLE_TIMEOUT=360` and
|
||||
`WAKE_TIMEOUT=240` for the demo; the e2e script overrides them to 60/120 so the
|
||||
smoke test does not wait minutes for the sweeper.
|
||||
|
||||
### Required RBAC
|
||||
|
||||
The gateway's ServiceAccount needs, in its namespace:
|
||||
|
||||
* `sandboxclaims` (`extensions.agents.x-k8s.io`): get, list, create, delete
|
||||
* `sandboxes` (`agents.x-k8s.io`): get, patch
|
||||
* `pods`: get
|
||||
* `sandboxes` (`agents.x-k8s.io`): get, list, patch
|
||||
* `pods`: get, list
|
||||
* `pods/log`: get
|
||||
|
||||
## Running locally
|
||||
|
||||
@@ -69,7 +69,7 @@ spec:
|
||||
# OnRootMismatch skips that walk once the volume root already carries
|
||||
# the fsGroup ownership/mode applied on the first mount.
|
||||
# Keep in sync with the minimal variant:
|
||||
# examples/minimal-single-tenant/manifests/sandbox-template.yaml.
|
||||
# examples/minimal-single-tenant/manifests/10-sandboxtemplate.yaml.
|
||||
fsGroupChangePolicy: OnRootMismatch
|
||||
containers:
|
||||
- name: dsh-web
|
||||
@@ -103,7 +103,7 @@ spec:
|
||||
# sessions, config and credentials are unaffected. Existing
|
||||
# sandboxes stamp pod env at creation and keep the old HOME until
|
||||
# recreated. Keep in sync with the minimal variant:
|
||||
# examples/minimal-single-tenant/manifests/sandbox-template.yaml.
|
||||
# examples/minimal-single-tenant/manifests/10-sandboxtemplate.yaml.
|
||||
- name: HOME
|
||||
value: /workspace
|
||||
# Empty => DSH derives links from the request Host, which the
|
||||
|
||||
@@ -48,7 +48,7 @@ kubectl -n dsh-demo create secret generic dsh-provider-keys-extra \
|
||||
--from-literal=NVIDIA_API_KEY="nvapi-..."
|
||||
|
||||
# 2. The template itself (image pinned in the manifest).
|
||||
kubectl apply -f manifests/sandbox-template.yaml
|
||||
kubectl apply -f manifests/10-sandboxtemplate.yaml
|
||||
|
||||
# 3. A template only describes pods; a warm pool turns it into running
|
||||
# Sandboxes and a claim adopts one. One replica is enough here.
|
||||
@@ -103,16 +103,19 @@ kubectl -n dsh-demo port-forward "pod/$SB" 18081:3080
|
||||
|
||||
## Folder picker and the workspace
|
||||
|
||||
The template sets `HOME=/workspace`. DSH's directory picker (the folder explorer
|
||||
used to choose a workspace) has no start-path setting: it lists `$HOME` when it
|
||||
opens, so with the image default (`/home/node`) it would start on a directory that
|
||||
is not persisted. Pointing `HOME` at the PVC-backed workspace makes the picker
|
||||
open there (labelled *Home* in its breadcrumbs) and makes `~` in agent commands
|
||||
resolve to `/workspace`. The trade-off is that `$HOME`-relative tool caches and
|
||||
config (`.cache`, `.config`, `.local`, `.npm`, `.agents`) are written into the
|
||||
workspace on the PVC. DSH state is unaffected: `DSH_HOME` stays
|
||||
The template mounts the `dsh-home` PVC's `workspace/` subPath at
|
||||
`/workspace` and sets `HOME=/workspace`. DSH's directory picker (the folder
|
||||
explorer used to choose a workspace) has no start-path setting: it lists `$HOME`
|
||||
when it opens, so with the image default (`/home/node`) it would start on a
|
||||
directory that is not persisted. Pointing `HOME` at the PVC-backed workspace makes
|
||||
the picker open there (labelled *Home* in its breadcrumbs) and makes `~` in agent
|
||||
commands resolve to `/workspace`. The trade-off is that `$HOME`-relative tool
|
||||
caches and config (`.cache`, `.config`, `.local`, `.npm`, `.agents`) are written
|
||||
into the workspace on the PVC. DSH state is unaffected: `DSH_HOME` stays
|
||||
`/home/node/.dsh`. The rationale and the list of affected paths are in
|
||||
[`10-sandboxtemplate.yaml`](../dsh-agents-as-a-service/manifests/10-sandboxtemplate.yaml).
|
||||
Pod specs are stamped at creation, so an existing sandbox keeps its old mounts and
|
||||
`HOME` until it is recreated; the PVC retains DSH state and workspace data.
|
||||
|
||||
Verify inside a running sandbox:
|
||||
|
||||
@@ -133,7 +136,7 @@ NetworkPolicy are documented in
|
||||
|
||||
## Probes and trusted hosts
|
||||
|
||||
Two settings in `manifests/sandbox-template.yaml` are easy to get wrong:
|
||||
Two settings in `manifests/10-sandboxtemplate.yaml` are easy to get wrong:
|
||||
|
||||
- **Probe path.** `/` returns `401` without the launch-token cookie, so a probe
|
||||
there crash-loops the pod. The template probes `/favicon.svg` (served
|
||||
|
||||
@@ -107,6 +107,14 @@ spec:
|
||||
volumeMounts:
|
||||
- name: dsh-home
|
||||
mountPath: /home/node/.dsh # DSH_HOME
|
||||
# The agent workspace (the image's WORKDIR) is the same PVC under
|
||||
# its own subPath, so DSH state - including .credentials.yaml -
|
||||
# never shows up as workspace data. Kubelet creates workspace/ on
|
||||
# first mount with the volume's fsGroup permissions; no init
|
||||
# container is needed. See the multi-tenant template for details.
|
||||
- name: dsh-home
|
||||
mountPath: /workspace
|
||||
subPath: workspace
|
||||
# / is behind DSH's launch-token cookie (401 without it); use an
|
||||
# unauthenticated path for probes. The startup probe gives a cold
|
||||
# start (plugin activation on a small node, or the first resume of a
|
||||
@@ -20,6 +20,13 @@ for dir in examples/*/; do
|
||||
if [ -d "${dir}manifests" ]; then
|
||||
count=$(find "${dir}manifests" -maxdepth 1 -type f -name '*.yaml' | wc -l | tr -d ' ')
|
||||
[ "$count" -gt 0 ] || err "$name: manifests/ has no top-level *.yaml files"
|
||||
for manifest in "${dir}"manifests/*.yaml; do
|
||||
[ -e "$manifest" ] || continue
|
||||
case "$(basename "$manifest")" in
|
||||
[0-9][0-9]-*.yaml) ;;
|
||||
*) err "$name: top-level manifest $(basename "$manifest") must be named NN-*.yaml" ;;
|
||||
esac
|
||||
done
|
||||
fi
|
||||
|
||||
stray=$(find "$dir" -maxdepth 1 -type f -name '*.yaml' | wc -l | tr -d ' ')
|
||||
|
||||
@@ -33,7 +33,7 @@ expect .env.sample "RELEASE_REVISION=$RELEASE_REVISION"
|
||||
# Image references in the example manifests
|
||||
for f in \
|
||||
examples/dsh-agents-as-a-service/manifests/10-sandboxtemplate.yaml \
|
||||
examples/minimal-single-tenant/manifests/sandbox-template.yaml; do
|
||||
examples/minimal-single-tenant/manifests/10-sandboxtemplate.yaml; do
|
||||
expect "$f" "image: $GHCR_IMAGE:$IDENTITY"
|
||||
done
|
||||
expect examples/dsh-agents-as-a-service/manifests/50-gateway.yaml "image: $GATEWAY_IMAGE:$IDENTITY"
|
||||
|
||||
Reference in New Issue
Block a user