nicolas-georger/deepseek-kubernetes-sandbox
1
0
Fork 0
mirror of https://github.com/sredevopsorg/deepseek-kubernetes-sandbox.git synced 2026-11-03 01:13:24 +00:00
Code Projects Releases Activity

feat(repo)!: restructure around the dsh-harness image and one example per implementation

Browse Source 
- the root Dockerfile is the primary artifact; versions.env is the single source of truth
- merge gateway-api/dsh-demo into examples/dsh-agents-as-a-service (manifests/, exposure variants)
- move the single-tenant template into examples/minimal-single-tenant
- publish the gateway as a nested package under the repository image
- mirror the DSH ref for release identities (first release dsh-v0.2.1-rc.2-r2)
- CI: lint/python/manifests/gateway-image, reusable multi-arch build, tag-driven release, weekly e2e
- docs: README, docs/{main-image,examples,gitflow,versioning}, example READMEs, AGENTS.md, governance
- SSO is opt-in again; gateway pull policy and cookie defaults match the documented demo

BREAKING CHANGE: example manifest paths, the gateway image name, and the removal
of deploy/ and gateway-api/ affect existing deployments; see the migration table
in README.md.
This commit is contained in:
SREDevOpsAgents
2026-10-25 06:37:28 -03:00
parent 53e55fc851
commit e6e109d9a9
68 changed files with 3713 additions and 1643 deletions
Download Patch File Download Diff File Expand all files Collapse all files

38
docs/README.md Normal file
View File

@@ -0,0 +1,38 @@
# Repository documentation
Repository-wide documentation lives here. Example-specific material lives with
its example under [`examples/<name>/docs/`](../examples/) — see the
[agents-as-a-service architecture](../examples/dsh-agents-as-a-service/docs/architecture.md)
for the deep-dive that used to live in this directory.
If you are new, read the [root README](../README.md) first: it is image-first and
gets a DSH Web container running before anything else.
## Index
| Document | Read it for |
| --- | --- |
| [main-image.md](main-image.md) | The primary artifact: build args, runtime environment, entrypoint, probes, volumes/ports/signals, non-root contract, image names, tags and digests. |
| [examples.md](examples.md) | The example contract: required layout, manifest rules, exposure variants, components and image naming, scripts, docs, and how to add a new example. |
| [gitflow.md](gitflow.md) | Branches, protected `main`, squash merges, Conventional PR titles, required CI checks, release branches, hotfixes and the one-time GitHub settings checklist. |
| [versioning.md](versioning.md) | Release identity (`DSH ref` + optional `-rN`), why the current identity is `dsh-v0.2.1-rc.2-r2`, channels, compatibility matrix, bumping pins, the release checklist, rollback, git-cliff and digest pinning. |
## Example documentation
| Document | Read it for |
| --- | --- |
| [examples/README.md](../examples/README.md) | The index of implementations in this repository. |
| [dsh-agents-as-a-service README](../examples/dsh-agents-as-a-service/README.md) | The multi-tenant platform walkthrough: warm pool, claims, suspend/resume, gateway, exposure and troubleshooting. |
| [dsh-agents-as-a-service architecture](../examples/dsh-agents-as-a-service/docs/architecture.md) | Container/sequence diagrams, ADRs, security and RBAC, failure modes and the evolution path. |
| [gateway README](../examples/dsh-agents-as-a-service/gateway/README.md) | The gateway's HTTP/WebSocket API, configuration, auth (including SSO), and limitations. |
## Project policies
| Document | Read it for |
| --- | --- |
| [AGENTS.md](../AGENTS.md) | Repository conventions for AI agents and humans, and the commands the CI runs. |
| [CONTRIBUTING.md](../CONTRIBUTING.md) | Local setup, the change workflow and what a pull request needs. |
| [SECURITY.md](../SECURITY.md) | Vulnerability reporting and project-specific threat notes (pod logs, SSO header trust, shared provider key, TLS). |
| [CODE_OF_CONDUCT.md](../CODE_OF_CONDUCT.md) | Community standards. |
| [CHANGELOG.md](../CHANGELOG.md) | Generated release history (do not edit by hand). |
| [LICENSE](../LICENSE) | MIT. |

319
docs/architecture.md
View File

@@ -1,319 +0,0 @@
# Architecture
This document describes the browser-usable **agents-as-a-service** platform in
[`examples/dsh-agents-as-a-service/`](../examples/dsh-agents-as-a-service/): how the gateway,
Kubernetes, and DSH Web fit together, and why the design looks the way it does.
Scope: one Kubernetes namespace (`dsh-demo`), one shared DSH Web image, one gateway deployment,
and one DSH Web pod + PVC per user. The cluster is assumed to run
[agent-sandbox](https://github.com/kubernetes-sigs/agent-sandbox) v1.0.7 with the extensions
(`SandboxTemplate`, `SandboxWarmPool`, `SandboxClaim`).
## Container view
```mermaid
flowchart LR
browser["Browser"]
subgraph cluster["Kubernetes cluster - namespace dsh-demo"]
gw["Gateway Deployment<br/>Starlette ASGI on uvicorn :8080<br/>landing page + auth + proxy + sweeper"]
api["Kubernetes API Server"]
ctrl["agent-sandbox controller"]
pool["SandboxWarmPool dsh-pool<br/>2 pre-warmed spares"]
claim["SandboxClaim dsh-USER<br/>one per user"]
sb["Sandbox dsh-pool-xxxx<br/>spec.operatingMode"]
svc["Headless Service<br/>status.serviceFQDN"]
pod["DSH Web pod<br/>node process :3080"]
pvc[("PVC dsh-home-SANDBOX<br/>/home/node/.dsh")]
end
browser -->|"HTTP + WebSocket<br/>/u/USER/**"| gw
gw -->|"claims, sandboxes, pods, pod logs"| api
api --> ctrl
ctrl -->|"adopt / suspend / resume"| claim
claim --> sb
pool -.->|"replenishes"| sb
sb --> svc
svc --> pod
pod --> pvc
```
Responsibilities:
| Component | Responsibility |
| --- | --- |
| **Browser** | Holds only the gateway session cookie (`dsh_gw_session`). Talks exclusively to the gateway. |
| **Gateway** | Signup/login, session auth, `SandboxClaim` lifecycle, wake-on-connect, idle sweeper, HTTP/SSE/WebSocket reverse proxy, DSH launch-token handshake. |
| **Kubernetes API** | Source of truth for claims, sandboxes, and pods; also the log source for the DSH launch token. |
| **agent-sandbox controller** | Adopts warm spares on claim, flips `operatingMode` (Running/Suspended), garbage-collects sandbox + PVC on claim delete. |
| **DSH Web pod** | One `dsh web` process per user, state on the per-user PVC, listening on `3080`. |
| **Warm pool** | Keeps `N` Ready sandboxes so signup is a label flip instead of a cold start. |
## Flows
### Signup (browser)
```mermaid
sequenceDiagram
autonumber
actor B as Browser
participant G as Gateway
participant K as Kubernetes API
participant C as agent-sandbox controller
B->>G: POST /users (form: user=bob, Origin)
G->>G: validate user name and body size
G->>K: create SandboxClaim dsh-bob (warmPoolRef)
K->>C: reconcile claim
C-->>K: adopt spare, Ready, status.sandbox.name = dsh-pool-xxxx
G->>K: read claim (status) + sandbox + pod
G-->>B: 303 Location /u/bob/ + Set-Cookie dsh_gw_session
Note over G: JSON clients instead get 201 (Ready),<br/>202 (accepted, still waking), or 409 (conflict)
```
Form signup waits up to `WAKE_TIMEOUT` for the claim to become `Ready`, then redirects; the
first proxied request also waits for `Ready` and performs the DSH handshake (below). A `409`
means the user already has a claim, in which case `POST /login` (bearer token) or the existing
cookie is the right path.
### First request / proxy (with handshake)
```mermaid
sequenceDiagram
autonumber
actor B as Browser
participant G as Gateway
participant K as Kubernetes API
participant D as DSH Web pod
B->>G: GET /u/bob/ (Cookie dsh_gw_session)
G->>G: authenticate cookie, user = bob
G->>K: resolve claim -> sandbox -> pod
alt sandbox Suspended
G->>K: patch operatingMode=Running (wake)
loop until Ready or WAKE_TIMEOUT
G->>K: read sandbox Ready condition
end
end
G->>K: read pod logs (tail)
K-->>G: "dsh web: ...?token=..." line
G->>D: GET /?token=... (browser Host, X-Forwarded-*)
D-->>G: 303 + Set-Cookie dsh-auth-*
G->>G: cache cookie for (user, pod UID, Host)
G->>G: strip Cookie/Authorization/hop-by-hop headers
G->>D: replay request with dsh-auth cookie
D-->>G: 200 HTML / SSE stream
G-->>B: 200 (upstream Set-Cookie dropped, Location rewritten)
```
Handshake cache behavior: one entry per `(user, pod UID, Host)`; refreshed once when DSH answers
`401`; dropped when the sandbox is suspended or deleted. The launch token and the DSH auth
cookie never reach the browser.
### Wake-on-connect and idle suspend
```mermaid
sequenceDiagram
autonumber
participant S as Sweeper - lifespan task
participant G as Gateway
participant K as Kubernetes API
actor B as Browser
S->>G: no proxied request for IDLE_TIMEOUT
G->>K: patch sandbox operatingMode=Suspended
K-->>G: pod deleted, PVC + Service retained
Note over S,G: in-flight requests/WebSockets are tracked,<br/>so a busy turn is never suspended
B->>G: GET /u/bob/ after a while
G->>K: patch operatingMode=Running
G->>K: wait for Ready (WAKE_TIMEOUT)
G-->>B: request held, then served
```
Suspend deletes only the pod. The PVC keeps conversations, sessions, and config, so resume
reattaches exactly the same state.
**Ownership boundary.** The gateway shares its namespace with claims created outside it — the
example's resource-layer walkthrough applies `dsh-alice` by hand, and every claim is named
`dsh-<user>`. Claims the gateway creates carry `aaas.example.com/managed-by=dsh-gateway`; the
sweeper skips everything else, so it can never suspend a sandbox an operator is using. Without
that marker the sweeper would suspend a hand-applied claim 60s after first sighting and break
`kubectl logs`/`port-forward` against its pod.
**Resume and file ownership.** The pod template sets `fsGroupChangePolicy: OnRootMismatch` on
purpose. DSH's credentials plugin refuses to boot when `.credentials.yaml` has any group/other
permission bit (it creates the file `0600`). The kubelet's default fsGroup policy (`Always`)
recursively ORs `0660` into every file on each pod start, so the first resume of a sandbox would
crash-loop with `mode 660`. `OnRootMismatch` only rewrites ownership/permissions when the volume
root does not already match the fsGroup, which it does after the first mount — so suspend/resume
keeps working. Sandboxes snapshot their pod template, so an updated `SandboxTemplate` does not
retroactively change existing ones.
### WebSocket
```mermaid
sequenceDiagram
autonumber
actor B as Browser
participant G as Gateway
participant D as DSH Web pod
B->>G: GET /u/bob/... Upgrade: websocket (Cookie dsh_gw_session)
G->>G: authenticate + mark user in-flight
alt cookie missing/invalid
G-->>B: 101 upgrade, then close 4401
else authenticated
G->>D: websockets.connect(path, Host, Cookie dsh-auth)
alt upstream handshake fails
G-->>B: 101 upgrade, then close 1011
else bridged
G-->>B: 101 Switching Protocols
loop frames
B->>D: frames proxied through G
D->>B: frames proxied through G
end
G->>G: clear in-flight on close
end
end
```
## Decisions and trade-offs
### ADR-1: Starlette ASGI instead of a sync Flask proxy
- **Decision:** implement the gateway as a Starlette app served by `uvicorn
gateway:create_app --factory`.
- **Why:** the browser needs WebSocket upgrades, streaming SSE responses, and concurrent
long-poll wake requests. A requests-based Flask proxy cannot upgrade connections and blocks
workers while a sandbox wakes.
- **Cost:** async code paths and explicit bridging for the sync `kubernetes` client (offloaded
to threads). Accepted: one small service, one framework.
### ADR-2: Path-based routing (`/u/{user}/`) instead of per-user subdomains
- **Decision:** serve each user under `https://gateway/u/{user}/`.
- **Why:** one hostname, one TLS certificate, no wildcard DNS or per-user Ingress objects. DSH
uses relative asset/API URLs and injects `<base href="./">`, so a path prefix works; the
gateway rewrites absolute-path `Location` headers and redirects `/u/{user}` to
`/u/{user}/`.
- **Cost:** any upstream code that emits root-absolute URLs would need rewriting; cookie
domains are shared (mitigated by stripping DSH `Set-Cookie`). If DSH ever changes its base
handling, per-user subdomains are the fallback.
### ADR-3: Server-side launch-token handshake instead of exposing the token
- **Decision:** the gateway reads DSH's launch token from pod logs, performs the token
handshake itself, caches DSH's auth cookie, and authenticates the browser with its own
`HttpOnly` cookie.
- **Why:** DSH has no token API and returns `401` on `/` without the launch-token cookie.
Handing the token to the browser would put it in URLs, history, and logs, and would let users
bypass gateway auth.
- **Cost:** log reading permission and a brittle parser; see [Upstream feature
request](#upstream-feature-request-remove-log-scraping). This is the least durable part of the
design and the first thing to replace.
### ADR-4: In-memory state, single replica
- **Decision:** idle clocks, handshake cache, and in-flight tracking live in gateway memory;
the Deployment runs one replica.
- **Why:** the simplest thing that works, with no extra datastore for a demo/reference
platform.
- **Cost:** a restart forgets idle timers (sandboxes stay Running until their next request) and
the handshake cache; two replicas would double-suspend and race. Scaling out requires moving
this state to a shared store (Redis/Postgres) or leader election. Documented instead of
hidden.
### ADR-5: Claims as the user record
- **Decision:** a user is a `SandboxClaim` named `dsh-<user>`; the bearer-token SHA-256 lives in
a claim annotation, and the gateway marks its own claims with
`aaas.example.com/managed-by=dsh-gateway`.
- **Why:** no database, no separate account service; deleting the claim is account deletion and
cascades everything. The ownership annotation keeps a namespaced claim list safe to share with
objects created by hand or by other controllers.
- **Cost:** no password reset, no admin API, no audit log of its own; Kubernetes API load grows
with users. Claims created before the ownership marker existed must be annotated
(`kubectl annotate sandboxclaim ...`) to keep being swept. Fine for a reference pattern, not
for a product.
### ADR-6: Gateway cookie session instead of bearer tokens in the browser
- **Decision:** browsers get `dsh_gw_session=<user>:<token>`, `HttpOnly`, `SameSite=Lax`
(`Secure` when enabled); API clients may still use `Authorization: Bearer`.
- **Why:** tokens never appear in URLs or JS-readable storage, and `SameSite=Lax` plus a strict
`Origin`/`Referer` check on cookie-authenticated mutations blocks basic CSRF.
- **Cost:** cookie-based auth needs TLS and correct `COOKIE_SECURE`; there is no signed-cookie
layer, so the gateway trusts the token hash comparison for validation.
## Security and RBAC
The gateway runs with a namespace-scoped ServiceAccount (`50-gateway.yaml`):
| API group | Resource | Verbs | Why |
| --- | --- | --- | --- |
| `extensions.agents.x-k8s.io` | `sandboxclaims` | `get`, `list`, `create`, `delete` | signup, state, account deletion |
| `agents.x-k8s.io` | `sandboxes` | `get`, `list`, `patch` | resolve DNS/pod, wake/suspend |
| `""` (core) | `pods` | `get`, `list` | resolve the pod behind a sandbox |
| `""` (core) | `pods/log` | `get` | launch-token handshake |
Notes and limits:
- **Pod logs are sensitive.** They contain the DSH launch token and possibly model output. The
permission is scoped to one namespace, but anyone who can read gateway memory or logs can
impersonate users. Prefer the upstream token API when available.
- **No cluster-wide access, no secret reads.** The gateway does not read `dsh-provider-keys`.
- **Browser-facing hardening:** `HttpOnly`/`SameSite=Lax`/optional `Secure` cookies,
timing-safe SHA-256 token comparison, strict same-origin check on cookie-auth mutations,
request-body limit (`MAX_BODY_MB`), hop-by-hop header stripping, upstream `Set-Cookie`
dropped, DSH `Location` headers rewritten under `/u/<user>/`.
- **Host allow-list:** DSH rejects unknown `Host` headers. The gateway forwards the browser's
`Host`, so every public hostname must be in `DSH_TRUSTED_HOSTS`; `localhost`/`127.0.0.1` are
always trusted.
- **TLS is external.** The gateway serves plain HTTP; terminate TLS at the Ingress and set
`COOKIE_SECURE=true`. Without TLS, session cookies are replayable on the network.
- **Tenant isolation** is pod + PVC separation; all tenants share one namespace, one DSH binary,
and one `DEEPSEEK_API_KEY`. Add NetworkPolicies and per-tenant credentials before calling it
multi-tenant-grade.
- **No gateway rate limiting.** Signup can create claims until the cluster or warm pool is
exhausted; put limits at the ingress for real deployments.
## Failure modes
| Failure | Symptom | Behavior / mitigation |
| --- | --- | --- |
| DSH log line changed or rotated away | Proxy returns `503` (missing launch token) | Fix the parser or restart the pod so the token line is re-emitted |
| Probe on `/` | Pod `CrashLoopBackOff`, `401` in probe logs | Use `/favicon.svg` for readiness/liveness (shipped templates do) |
| Warm pool exhausted | JSON signup returns `202` (state `Waking`); form signup is slow | Increase `replicas` in the warm pool |
| Gateway restart | Idle sandboxes stop suspending; handshake cache lost | Rebuilt lazily on next request; consider a shared store for production |
| DSH auth cookie expired mid-session | Proxy `401` once, then recovers | Gateway refreshes the handshake once |
| Upstream pod name changed after resume | Cached cookie keyed by old pod UID is useless | Cache is invalidated on suspend; next request re-handshakes |
| Two gateway replicas | Double suspend/wake races | Keep `replicas: 1` until state is externalized |
## Upstream feature request: remove log scraping
DSH should expose the launch token as a first-class, machine-readable interface so platforms do
not need `pods/log` access. Proposed shapes (any one suffices):
1. **Token file:** `dsh web --token-file /run/dsh/token` writes the current launch token (mode
`0600`) and rewrites it on rotation.
2. **Environment input:** accept `DSH_LAUNCH_TOKEN` (platform-generated) so the token never
has to be discovered.
3. **Loopback/trusted-host endpoint:** `GET /api/auth/launch-token` that only answers on the
pod's loopback or a trusted host and returns the token plus expiry.
4. **Token in a well-known file** under `DSH_HOME` with the same semantics as (1).
Why this matters: log scraping is brittle (rotation, restart, format changes), requires a broad
`pods/log` permission that also exposes application output, and can leak tokens into log
pipelines. Until an upstream API exists, the gateway's parser, `DSH_LOG_TAIL_LINES`, and the
single-retry refresh are the supported workaround.
## Evolution path
1. Replace log scraping with the upstream token API (above), deleting the `pods/log` permission.
2. Move idle clocks, in-flight tracking, and handshake cache to a shared store; then scale the
gateway horizontally (sticky routing not required if state is shared).
3. Add per-tenant API keys or an egress proxy so a single key is not shared by all sandboxes.
4. Add NetworkPolicies (gateway → sandboxes only), TLS everywhere, rate limits, and audit
logging at the ingress.
5. If DSH grows absolute URLs or per-user cookies that cannot be rewritten, switch to
per-user subdomains with wildcard DNS/TLS.

149
docs/examples.md Normal file
View File

@@ -0,0 +1,149 @@
# Example contract
Every implementation in this repository is a **self-contained example** under
[`examples/<name>/`](../examples/). Examples demonstrate how to run the main
DSH Web image (and, when needed, auxiliary components such as the
agents-as-a-service gateway) on Kubernetes. They are reference material, not
libraries: each one is applied with plain `kubectl`, documented end to end, and
deletable in one command.
[`examples/README.md`](../examples/README.md) is the index. If an example is not
listed there, it does not exist as far as `make check` is concerned.
## Required layout
```
examples/<name>/
+-- README.md # required: what it demonstrates, prerequisites,
| # ordered steps, troubleshooting, cleanup
+-- manifests/ # required
| +-- NN-*.yaml # numbered top-level manifests, plain kubectl
| +-- exposure/ # optional exposure variants
| +-- README.md # required when exposure/ exists
| +-- <variant>/NN-*.yaml # e.g. ingress/, gateway-api-sso/
+-- scripts/ # optional
| +-- *.sh # executable, idempotent, self-cleaning
+-- docs/ # optional deeper material (architecture, ADRs)
+-- <component>/ # optional published component
+-- Dockerfile
+-- README.md # required whenever a Dockerfile exists
+-- tests...
```
### Rules the CI enforces
[scripts/check-examples.sh](../scripts/check-examples.sh) fails the build when:
- `examples/README.md` is missing or does not mention `examples/<name>/` for every
example directory;
- an example has no `README.md` or no `manifests/` directory;
- `manifests/` contains no top-level `*.yaml` files;
- a `*.yaml` file sits at the example root instead of under `manifests/`;
- `manifests/exposure/` exists without a `manifests/exposure/README.md` describing
the variants;
- a subdirectory with a `Dockerfile` has no `README.md`;
- a file in `scripts/` is not executable.
[scripts/check-manifests.py](../scripts/check-manifests.py) additionally parses
every `manifests/**/*.yaml` and requires `apiVersion`, `kind` and
`metadata.name`. CI also runs a server-side dry-run against the real
agent-sandbox CRDs on a throwaway kind cluster.
Run both locally with:
```bash
make check
```
## Manifests
- **Numbered and ordered.** `NN-*.yaml` files are applied in filename order and
read as a walkthrough: `00-prereqs.yaml`, `10-sandboxtemplate.yaml`,
`20-sandboxwarmpool.yaml`, ... Keep the numbering stable; it is referenced from
the example README.
- **Plain kubectl.** No Helm, Kustomize, or custom installer. `kubectl apply -f`
against a namespace the example creates is the whole interface.
- **Safe by default.** Never ship a usable Secret. Secret-shaped manifests are
`*.yaml.example` files with obvious placeholders and an inline creation
command; real values are generated per install.
- **Cluster-specific state stays out.** Node affinity, replica counts tuned for a
cluster, and similar fields belong in a local patch, not in the shipped
manifest.
- **Placeholders are explicit.** Hostnames, GitHub orgs and TLS material use
values like `dsh.example.com` / `your-github-org` with a TODO comment.
## Exposure variants
An example that can be reached from outside the cluster puts each option under
`manifests/exposure/<variant>/` and describes the trade-offs in
`manifests/exposure/README.md`. Variants are alternatives, not a stack: apply one.
`dsh-agents-as-a-service` currently ships:
| Variant | Path | What it gives you |
| --- | --- | --- |
| Ingress | `manifests/exposure/ingress/` | Traefik-style Ingress + TLS; the gateway does its own session auth. |
| Gateway API + SSO | `manifests/exposure/gateway-api-sso/` | Gateway API `Gateway`/`HTTPRoute` in front of oauth2-proxy (GitHub OAuth); the gateway inherits `X-Forwarded-User` via `AUTH_PROXY_HEADER`. |
## Components
A component is a program the example needs beyond the main image: today only the
agents-as-a-service gateway. A component lives in its own subdirectory with:
- `Dockerfile` — its image build;
- `README.md` — purpose, API/config reference, local development, limitations;
- tests — unit tests that run without a cluster, plus whatever else applies.
Component images are published as **nested packages below the repository image**:
```
<registry>/<owner>/<repo>/<example>/<component>
```
so the gateway is
`ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway`.
Never create an unrelated top-level package; `check-versions.sh` verifies the
nesting.
## Scripts
`scripts/*.sh` must be:
- **executable** (the CI checks the mode bit);
- **idempotent** — safe to re-run, cleaning up before creating;
- **self-cleaning** — a trap removes the namespace, port-forwards and temp files,
even on failure;
- **non-interactive** — usable in CI, with environment-variable knobs instead of
prompts.
`examples/dsh-agents-as-a-service/scripts/e2e-kind.sh` is the reference
implementation: it runs the README walkthrough and deletes everything it created.
## Documentation
- The example `README.md` is the walkthrough. Keep it runnable: every command
block should work when pasted in order.
- Deeper material (architecture, sequence diagrams, ADRs) belongs in the
example's `docs/`, not in the repository-level [docs/](../docs/).
- Repository-wide docs describe the container and the repo rules; example docs
describe one implementation.
## Adding a new example
1. Create `examples/<name>/` with the layout above.
2. Write `README.md` first: what it demonstrates, prerequisites, numbered steps,
troubleshooting, limitations, cleanup.
3. Add `manifests/NN-*.yaml`; keep secrets as `*.yaml.example`.
4. If a component needs an image, publish it under
`<registry>/<owner>/<repo>/<example>/<component>`.
5. Add an entry to [`examples/README.md`](../examples/README.md) using the literal
`examples/<name>/` path.
6. Run `make check` and, if you added a component, `make test` and `make lint`.
7. If the example is expensive to run (kind, images, cluster CRDs), ship a
`scripts/e2e-kind.sh` and wire it into CI.
## Current examples
| Example | Demonstrates |
| --- | --- |
| [`examples/dsh-agents-as-a-service/`](../examples/dsh-agents-as-a-service/) | The full multi-tenant platform pattern: SandboxTemplate, warm pool, per-user claims, suspend/resume, and the browser gateway. |
| [`examples/minimal-single-tenant/`](../examples/minimal-single-tenant/) | A single DSH Web sandbox: one SandboxTemplate and a claim. |

128
docs/gitflow.md Normal file
View File

@@ -0,0 +1,128 @@
# Gitflow: branches, pull requests and CI
This repository uses **trunk-based development with release branches**. `main` is
always releasable; feature work happens on short-lived branches; releases are cut
from `main` onto a release branch and tagged with an immutable release identity.
See [versioning](versioning.md) for what the tags mean.
## Branches
| Branch | Branched from | Merged into | Purpose |
| --- | --- | --- | --- |
| `main` | — | — | The integration trunk. Protected, linear history, always releasable. |
| `feat/<slug>` | `main` | `main` (squash) | New capability or example. |
| `fix/<slug>` | `main` | `main` (squash) | Bug fix. |
| `docs/<slug>` | `main` | `main` (squash) | Documentation only. |
| `refactor/<slug>`, `chore/<slug>`, `test/<slug>` | `main` | `main` (squash) | Non-behavioral rework, maintenance, tests. |
| `release/<identity>` | `main` at the release commit | — | Release stabilization and tagging. Only release-blocking fixes land here. |
| `hotfix/<slug>` | The release branch (or the tagged commit) | `main` and the release branch | Urgent production fix. |
Keep branches short-lived. Rebase on `main` rather than merging `main` into the
branch; the PR is squash-merged, so individual branch commits are never part of
`main` history.
## Pull requests
1. **One scoped change per PR.** Do not mix a version bump with unrelated work,
and update the docs the change touches in the same PR.
2. **Conventional PR title.** The title becomes the squash commit message, so it
is linted by the `pr-title` check: `feat(scope): ...`, `fix(scope): ...`,
`docs(scope): ...`, `build: ...`, `ci: ...`, `refactor: ...`, `test: ...`,
`chore: ...`. Breaking changes use `feat(scope)!: ...` or a
`BREAKING CHANGE:` footer.
3. **CI must be green.** All required checks below must pass before merge.
4. **Squash merge only.** `main` stays linear and protected; merged PRs are one
commit each.
5. **No direct pushes to `main`** or to a release branch. Both are protected.
6. **Review for behavior changes.** Infrastructure, workflow, release and
gateway-contract changes need a maintainer review; trivial docs changes still
need the checks.
## Required checks
| Check | Workflow / job | What it verifies |
| --- | --- | --- |
| `lint` | `ci.yaml` | `scripts/check-versions.sh` (version pins and markdown tokens), ruff on the gateway, shellcheck on shell scripts, yamllint/actionlint when available. |
| `python` | `ci.yaml` | Gateway unit tests (`pytest`). |
| `manifests` | `ci.yaml` | `scripts/check-examples.sh`, `scripts/check-manifests.py` and a server-side dry-run of every example manifest against the agent-sandbox CRDs. |
| `gateway-image` | `ci.yaml` | Builds the gateway image and smoke-tests `/healthz` without a cluster. |
| `pr-title` | `pr-title.yaml` | Conventional Commit PR title. |
| `CodeQL` | `codeql.yml` | Static analysis for Python and GitHub Actions. |
| `dependency-review` | `ci.yaml` | Dependency changes introduced by the PR (job `dependency-review`; runs on pull requests only). |
The image build workflows (`_build-image.yaml`, `build-main.yaml`,
`build-gateway.yaml`) and `e2e-kind.yaml` are **not** required on every PR: the
image builds run where they are needed, and the kind end-to-end suite is reserved
for release validation and manual dispatch because it is slow.
## Releases
Releases are cut from `main`:
1. Confirm `main` is green and `make check`, `make test`, `make lint` pass.
2. Bump [versions.env](../versions.env) if the DSH ref changed, using
`make set-version`; otherwise keep the same `DSH_VERSION` and bump
`RELEASE_REVISION`.
3. Create `release/<identity>` from the release commit.
4. Run `make release-dry-run` and review the generated notes.
5. Create and push the annotated `dsh-*` release tag. The `release.yaml`
workflow validates that the tag matches [versions.env](../versions.env), builds
and pushes the multi-arch images (version + latest), creates the GitHub
Release from the git-cliff notes, and opens a `chore(changelog)` PR that
refreshes `CHANGELOG.md` on `main`.
6. Merge the changelog PR and any release-branch-only fixes back to `main`.
Full procedure: [versioning](versioning.md).
## Hotfixes
A released tag is **never** moved or re-tagged — tags and image identities are
immutable. To fix a broken release:
1. Branch `hotfix/<slug>` from the release tag (or its release branch).
2. Make the minimal fix, add a test, open a PR; the same required checks apply.
3. Publish a new identity with a higher `RELEASE_REVISION` (`-r(N+1)`). Do not
reuse or delete the broken tag.
4. Merge the fix back into `main` (or cherry-pick it) so the trunk is fixed too.
Rollback is also a re-release, never a retag: see
[versioning](versioning.md#rollback).
## Tag protection
All release tags match `dsh-*` and are protected:
- only maintainers (the release workflow) may create them;
- they cannot be deleted or force-updated;
- they are annotated and point at the release commit on a release branch.
Image tags derive from the same identity and are equally immutable; the mutable
`latest` and `main` image channels always point at a release or a trunk build
respectively.
## One-time GitHub settings checklist
Configure these on the repository before relying on the workflow:
- [ ] **Branches → main**: require a pull request; require the checks `lint`,
`python`, `manifests`, `gateway-image`, `pr-title`, `CodeQL`,
`dependency-review`; require branches to be up to date; require conversation
resolution; restrict direct pushes; allow only squash merges and enable
"Automatically delete head branches".
- [ ] **Branches → release/***: same protections, plus allow the release workflow
to push tags.
- [ ] **Tags → dsh-***: restrict creation/update/deletion to maintainers; a tag
push triggers the release workflow.
- [ ] **Actions → General**: allow GitHub Actions; set workflow permissions to
read-only by default; allow `GITHUB_TOKEN` to create pull requests only
where required.
- [ ] **Actions → Runners**: make the arm64 runners (`ubuntu-24.04-arm`) available
to the public/private repository as appropriate.
- [ ] **Packages**: grant the release workflow `packages: write` (workflow
permissions do this) and protect the
`ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox` packages from deletion.
- [ ] **Security**: enable CodeQL default setup **or** keep the advanced
`codeql.yml` workflow (not both), enable Dependabot alerts and
dependency review.
- [ ] **Secrets/Variables**: none are needed for image publishing; the workflows
use `GITHUB_TOKEN`.

169
docs/main-image.md Normal file
View File

@@ -0,0 +1,169 @@
# The main DSH Web image
This repository's primary artifact is a container image that packages
[DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) Web
(`dsh web`) for Kubernetes and plain Docker. The [Dockerfile](../Dockerfile) at the
repository root builds it; every version pin it uses lives in
[versions.env](../versions.env).
The image is image-first: it is the product, and the examples under
[`examples/`](../examples/) are self-contained implementations that consume it.
## What the build does
The Dockerfile is a two-stage build:
1. **Build stage** (`node:24-trixie`) clones DSH at the pinned git ref, installs
dependencies with pnpm, runs the production build, packages the `@deepseek-ai/dsh`
CLI with `pnpm pack`, and installs its production dependencies with npm.
2. **Runtime stage** (`node:24-trixie-slim`) copies `/opt/dsh` to `/app`, adds the
CLI tools DSH agents commonly use (`git`, `ripgrep`, `jq`, `yq`, `curl`, `wget`,
`gh`, `tini`, `uv`), and starts `dsh web` through a small entrypoint.
## Build it
```bash
make image # canonical: builds the DSH Web image, no push
```
`make image` reads [versions.env](../versions.env) and passes the pins as build args.
To reproduce the equivalent by hand:
```bash
docker build \
--build-arg DSH_VERSION=dsh-v0.2.1-rc.2 \
--build-arg RELEASE_REVISION=r2 \
--build-arg PNPM_VERSION=11.28.5 \
--build-arg VCS_REF="$(git rev-parse --short HEAD)" \
-t ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.2-r2 .
```
### Build args
| Arg | Default / source | Purpose |
| --- | --- | --- |
| `DSH_REPO` | `https://github.com/deepseek-ai/deepseek-harness.git` | Upstream DSH repository to clone. |
| `DSH_VERSION` | `DSH_VERSION` in [versions.env](../versions.env) | Upstream DSH git tag/ref to package; recorded as `org.opencontainers.image.version`. |
| `RELEASE_REVISION` | `RELEASE_REVISION` in [versions.env](../versions.env) | Repository revision suffix (`rN`); the release identity is `DSH_VERSION-RELEASE_REVISION`, recorded in the `io.sredevopsorg.release.revision` label. See [versioning](versioning.md). |
| `PNPM_VERSION` | `PNPM_VERSION` in [versions.env](../versions.env) | pnpm version activated with corepack in both stages. |
| `VCS_REF` | `git rev-parse --short HEAD` (via `make image`) | Source revision recorded for traceability. |
| `NODE_BUILD_IMAGE` | `docker.io/node:24-trixie` | Build-stage base image. |
| `NODE_RUNTIME_IMAGE` | `docker.io/node:24-trixie-slim` | Runtime-stage base image. |
Never hard-code a version outside [versions.env](../versions.env):
`./scripts/check-versions.sh` (and `make check`) fails when a pin drifts.
## Runtime contract
### Entrypoint
The entrypoint is `tini -- /usr/local/bin/dsh-entrypoint`:
1. `tini` runs as PID 1 and reaps children/forwards signals.
2. The shell entrypoint applies defaults, derives a bind address when `DSH_HOST` is
empty (the pod/container's first non-loopback IPv4), builds the
`dsh web` command line, and `exec`s it so signals reach the Node process.
3. `localhost` and `127.0.0.1` are always added to DSH's trusted-host list; each
`DSH_TRUSTED_HOSTS` entry is added as an extra `--trusted-host`.
### Environment variables
| Variable | Default | Purpose |
| --- | --- | --- |
| `DSH_PORT` | `3080` | Port DSH Web listens on inside the container. |
| `DSH_HOST` | auto-detected non-loopback IPv4 | Bind address. Leave empty in Kubernetes so the pod IP is used automatically. |
| `DSH_PUBLIC_URL` | `http://localhost:${DSH_PORT}` | Public base URL used to build links and the launch-token URL printed in the logs. Set it when the container is published on a different host port or behind a proxy. |
| `DSH_TRUSTED_HOSTS` | empty | Extra `Host`/`Origin` allow-list entries, comma- or space-separated. `localhost`/`127.0.0.1` are always trusted. Required when a proxy forwards the browser's `Host`. |
| `DEEPSEEK_API_KEY` | empty | Provider credential read by DSH. Inject it with `-e`/`envFrom`; never bake it into the image. Only model calls fail when it is missing. |
| `DSH_HOME` | `/home/node/.dsh` | DSH state directory (conversations, sessions, config, credentials). Mount a volume here to persist it. |
The image also sets `NODE_ENV=production`, `HOME=/home/node`, and a `PATH` that
includes `/app/node_modules/.bin`.
### Ports, volumes and signals
| Item | Value | Notes |
| --- | --- | --- |
| Exposed port | `3080/tcp` | `EXPOSE 3080`; one port serves the UI, assets, API, SSE and WebSocket traffic. |
| State volume | `/home/node/.dsh` | Declared with `VOLUME`; use a named volume (Docker) or PVC (Kubernetes). |
| Working directory | `/workspace` | The agent workspace; mount a volume here if agents should keep files. |
| Stop signal | `SIGTERM` | `tini` forwards it to `dsh web`; allow a graceful shutdown period. |
| User | `node`, UID/GID `1000` | The process is non-root. A PVC mounted at `DSH_HOME` must be writable by UID 1000. |
### Health probes
`/` is **not** a health endpoint: DSH returns `401` until a browser completes the
launch-token handshake. Probe `/favicon.svg` instead — it is unauthenticated and
served by the web layer, so it proves the HTTP server is up without triggering auth.
```yaml
startupProbe:
httpGet: {path: /favicon.svg, port: 3080}
periodSeconds: 5
failureThreshold: 24
readinessProbe:
httpGet: {path: /favicon.svg, port: 3080}
periodSeconds: 5
livenessProbe:
httpGet: {path: /favicon.svg, port: 3080}
periodSeconds: 15
```
The shipped templates use exactly this shape; see
[`examples/minimal-single-tenant/manifests/sandbox-template.yaml`](../examples/minimal-single-tenant/manifests/sandbox-template.yaml).
A `startupProbe` is important because plugin activation (or the first start of a
large sandbox) can exceed a default liveness budget.
### Launch token
DSH prints its one-time launch URL to the logs:
```
dsh web: http://<host>:${DSH_PORT}/?token=...
```
Open that URL once to obtain DSH's auth cookie. After that the UI works at the bare
URL; without the cookie `/` answers `401` by design. The
[agents-as-a-service gateway](../examples/dsh-agents-as-a-service/gateway/) performs
this handshake server-side so browsers never see the token.
## Published image names and tags
Registry layout: the main image is the repository itself, and every example
component is a nested package below it:
```
ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox # main DSH Web image
ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway # example component
```
The main image is published multi-arch (`linux/amd64`, `linux/arm64`) to
GitHub Container Registry:
| Tag | Meaning |
| --- | --- |
| `dsh-v0.2.1-rc.2-r2` | Current release identity. **Immutable**: once published it never moves. |
| `latest` | Most recent release; points at the same digest as the current identity. |
| `main` | Development channel rebuilt from every push to `main`; not a release, not for production. |
| `dsh-v0.2.1-rc.2` | Legacy tag published before the release-identity scheme existed. Do not use it for new deployments; see the migration table in the [root README](../README.md). |
Release identities, channels and rollback rules are defined in
[versioning](versioning.md). Tags are immutable, so production deployments should
pin a digest:
```bash
docker buildx imagetools inspect ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.2-r2
# ... pin the digest in your deployment manifests:
# image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox@sha256:<digest>
```
## Verifying an image
```bash
docker inspect ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.2-r2 \
--format '{{json .Config.Labels}}'
# org.opencontainers.image.title / .version / .source
```
Release builds also publish SBOM and provenance attestations from the release
workflow (see [SECURITY.md](../SECURITY.md) and [gitflow](gitflow.md)).

163
docs/versioning.md Normal file
View File

@@ -0,0 +1,163 @@
# Versioning, releases and changelog
[versions.env](../versions.env) is the **single source of truth** for the DSH ref,
the repository revision, the pnpm version, the control-plane component versions
and the image names. Nothing else may hard-code a version:
[scripts/check-versions.sh](../scripts/check-versions.sh) fails when a pin drifts,
and it also verifies that prose mentions only the current release identity or its
base DSH tag.
## Release identity
```
release identity = DSH_VERSION + optional "-" + RELEASE_REVISION
```
| Piece | Meaning |
| --- | --- |
| `DSH_VERSION` | The upstream DeepSeek Harness git ref being packaged, e.g. `dsh-v0.2.1-rc.2`. It mirrors upstream's own tag, which is what "mirrored-DSH scheme" means. |
| `RELEASE_REVISION` | An optional repository revision at that DSH ref: empty for the first release, `rN` (`N >= 2`) for a repository-only re-release. |
Examples:
- `DSH_VERSION=dsh-v0.2.1-rc.2`, `RELEASE_REVISION=` → identity `dsh-v0.2.1-rc.2`
(first release at that upstream ref).
- `DSH_VERSION=dsh-v0.2.1-rc.2`, `RELEASE_REVISION=r2` → identity
`dsh-v0.2.1-rc.2-r2` (documentation, CI or gateway changes on top of the same
upstream DSH; no DSH upgrade).
A DSH upgrade produces a new base identity (for example the next upstream ref
with an empty revision); a repository-only change at the same DSH ref increments
`rN`.
## Why the current identity is `dsh-v0.2.1-rc.2-r2`
The image tag `dsh-v0.2.1-rc.2` was already published to GHCR for an earlier
repository state. **Container tags are immutable** in this project: re-pushing a
tag would silently change what an existing deployment runs. The mirrored-DSH
scheme therefore starts its first identity at `-r2`: the plain tag is
conceptually revision 1 and is taken, so the first publishable repository
revision at the same upstream ref is `r2`.
The identity is stored in [versions.env](../versions.env) and consumed everywhere:
the Dockerfile label, the example manifests, the Makefile and the release
workflow. The legacy `dsh-v0.2.1-rc.2` tag remains published but is not updated;
new deployments use `dsh-v0.2.1-rc.2-r2`.
The gateway image moved from the repository-level `:gateway` tag to a nested
package (`<repository>/dsh-agents-as-a-service/gateway`). For exactly one
release the release workflow also republishes the legacy
`<repository>:gateway` tag as a deprecated alias; it is removed in the next
release, so deployments should pin the nested package.
## Channels
| Channel | Image tag | Meaning |
| --- | --- | --- |
| Release | `dsh-v0.2.1-rc.2-r2` | The current immutable release identity. What production should pin (preferably by digest). |
| `latest` | `latest` | Alias for the most recent release; moves only when a new release is published. |
| `main` | `main` | Development channel, rebuilt from every push to `main`. Never treat it as a release; it exists so `main` can be tested. |
The same channels exist for every published package, including the nested
example component image.
## Compatibility matrix
Values from [versions.env](../versions.env) at the current identity:
| Component | Pin | Source |
| --- | --- | --- |
| Upstream DSH source | `DSH_REPO` | [versions.env](../versions.env) |
| DSH ref | `dsh-v0.2.1-rc.2` | `DSH_VERSION` |
| Repository revision | `r2` | `RELEASE_REVISION` |
| pnpm | `11.28.5` | `PNPM_VERSION` |
| agent-sandbox (CRDs) | `v1.0.7` | `AGENT_SANDBOX_VERSION`, used by the examples and CI dry-run |
| Kubernetes Gateway API | `v1.2.1` | `GATEWAY_API_VERSION`, used by the SSO exposure variant |
| oauth2-proxy | `quay.io/oauth2-proxy/oauth2-proxy:v7.15.5` | `OAUTH2_PROXY_IMAGE` |
| Main image | `ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox` | `GHCR_IMAGE` |
| Gateway image | `ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway` | `GATEWAY_IMAGE`, nested below `GHCR_IMAGE` |
Example component images are always nested below the main package:
`<GHCR_IMAGE>/<example>/<component>`. `check-versions.sh` rejects an unrelated
top-level package.
## Bumping versions
```bash
make set-version DSH_VERSION=dsh-vX.Y.Z RELEASE_REVISION=rN
make check
```
`set-version` updates [versions.env](../versions.env) through
[scripts/set-version.sh](../scripts/set-version.sh) and leaves every consuming
file to derive from it. Never edit the Dockerfile, manifests, Makefile or
workflows with a literal version.
- New upstream DSH ref: change `DSH_VERSION` and clear `RELEASE_REVISION`.
- Repository-only change at the same DSH ref: keep `DSH_VERSION`, set
`RELEASE_REVISION=r(N+1)`.
## Release checklist
1. `main` is green and `make check`, `make test`, `make lint` pass locally.
2. `make image` and `make gateway` build; smoke-test the images.
3. Bump the pins with `make set-version` (commit that change through a PR).
4. `make release-dry-run` — verifies the identity, prints the image names, checks
the tag does not already exist, and previews the generated notes.
5. Create and push the annotated `dsh-*` release tag (or dispatch
`release.yaml` with an existing tag). The workflow validates that the tag
matches [versions.env](../versions.env).
6. The workflow builds and pushes multi-arch images for the main image and the
example components (the identity tag plus `latest`), creates the GitHub
Release from the git-cliff notes, records the published digests, and opens a
`chore(changelog)` PR that refreshes `CHANGELOG.md` on `main`.
7. Merge the changelog PR. Verify:
`docker buildx imagetools inspect <image>:<identity>` shows both platforms,
and the GitHub Release references the tag and the digest.
8. Announce the identity and the digest; deployments pin the digest.
## Rollback
Tags and release identities are **immutable**, so a bad release is never re-tagged
or deleted. Fix forward:
1. Fix the problem on a `hotfix/<slug>` branch (see [gitflow](gitflow.md)).
2. Publish a new identity with the next revision, `-r(N+1)`, at the same
`DSH_VERSION`.
3. Point deployments at the new digest; keep the broken tag for audit history.
Emergency rollback to the previous release is also a re-deploy of a **previous
digest**, not a move of `latest`.
## Changelog
`CHANGELOG.md` is generated by [git-cliff](https://git-cliff.org) from
Conventional Commits, configured in [cliff.toml](../cliff.toml):
```bash
make changelog # regenerate CHANGELOG.md locally
```
- Do not edit `CHANGELOG.md` by hand; the release workflow regenerates it and opens a PR to refresh it on `main`.
- Commit messages and PR titles must be Conventional Commits so they are grouped
correctly (Added / Fixed / Changed / Documentation / Build / CI / Tests /
Maintenance).
- The changelog groups commits since the previous `dsh-*` tag; release notes use
the same rendering.
- The changelog is excluded from the version-token check because it records
historical identities on purpose.
## Digest pinning
Release identities are mutable only in theory; deployments should still pin the
resolved digest for reproducibility and supply-chain safety:
```bash
docker buildx imagetools inspect ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.2-r2
# then, in the deployment:
# image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox@sha256:<digest>
```
The example manifests use the identity tag so they stay readable and upgrade with
`versions.env`; production overlays should pin the digest. Release images carry
SBOM and provenance attestations (see [SECURITY.md](../SECURITY.md)).