mirror of
https://github.com/sredevopsorg/deepseek-kubernetes-sandbox.git
synced 2026-11-03 01:13:24 +00:00
Compare commits
24 Commits
dsh-v0.2.1
...
chore/enti
| Author | SHA1 | Date | |
|---|---|---|---|
|
a585e6be13 | ||
|
|
c5b8afde48 | ||
|
|
8e9bd478b4 | ||
|
|
1a929e3e24 | ||
|
|
2f959e6afb | ||
|
|
4c073eef71 | ||
|
|
24f23e09ae | ||
|
|
82ee0d0cdb | ||
|
|
a09d4ac340 | ||
|
|
367294b230 | ||
|
|
cbcc56f97e | ||
|
|
8c34f43789 | ||
|
|
97c563296a | ||
|
|
dcb5b436f6 | ||
|
|
b57916d854 | ||
|
|
bed8e5675a | ||
|
|
14c3fb840e | ||
|
|
96d3030a49 | ||
|
|
2b43d5dfd5 | ||
|
|
97abf75dfe | ||
|
|
e2f1fe567a | ||
|
|
3e9ba38fc1 | ||
|
|
bf24306421 | ||
|
|
8681412be4 |
157
.dsh/skills/dsh-agent-sandbox-platform/SKILL.md
Normal file
157
.dsh/skills/dsh-agent-sandbox-platform/SKILL.md
Normal file
@@ -0,0 +1,157 @@
|
||||
---
|
||||
name: dsh-agent-sandbox-platform
|
||||
description: >-
|
||||
Operate and extend this repository's DeepSeek Harness-on-agent-sandbox platform:
|
||||
warm pools, claims, sandbox suspend/resume, the gateway, and the optional
|
||||
agent-sandbox MCP server, with version-aware safety and a read-only preflight.
|
||||
whenToUse: >-
|
||||
Use when working in deepseek-kubernetes-sandbox or with Kubernetes agent-sandbox:
|
||||
inspect or provision sandboxes, debug warm pools or claims, use
|
||||
mcp__*__create_sandbox/execute_command, connect the agent-sandbox MCP server, or
|
||||
add a manifest, gateway route, exposure variant, example, or version bump here.
|
||||
metadata:
|
||||
author: sredevopsorg
|
||||
version: "1.0.0"
|
||||
---
|
||||
|
||||
# DSH agent-sandbox platform
|
||||
|
||||
This repository runs DeepSeek Harness Web (DSH) as one sandbox per user on
|
||||
[agent-sandbox](https://github.com/kubernetes-sigs/agent-sandbox), with an
|
||||
optional official MCP server for sandbox execution. For generic cluster work
|
||||
load `kubernetes-kubectl-operations`; for profile MCP rows load
|
||||
`cordis-plugin-development`.
|
||||
|
||||
## 0. Facts before action
|
||||
|
||||
1. Read `AGENTS.md`, `versions.env`, the example README/architecture. The
|
||||
control-plane pin is `AGENT_SANDBOX_VERSION`; release identity is
|
||||
`DSH_VERSION` + `RELEASE_REVISION`.
|
||||
2. Establish context, user and namespace; never assume `dsh-demo`.
|
||||
3. Run `scripts/platform-preflight.sh --namespace <ns>` (read-only).
|
||||
4. If `mcp__*__create_sandbox`-style tools are absent, use `kubectl`; do not
|
||||
install or enable a server on your own.
|
||||
|
||||
## 1. Non-negotiables
|
||||
|
||||
- **Read-only by default.** Change a cluster only when the user asked for that
|
||||
exact change; show the command and rollback first.
|
||||
- **Scope every command** with `--context` and `-n`. `--all`/`-A` mutations and
|
||||
deleting namespaces, PVCs, PVs, CRDs, or cluster-scoped objects need an
|
||||
explicit, named confirmation.
|
||||
- **Secrets and logs stay private.** Never print Secret data, kubeconfigs, or
|
||||
node logs. Pod logs can contain the DSH launch token.
|
||||
- **The MCP server has no authentication.** It is a full-access management
|
||||
endpoint; never expose it. A DSH profile row affects every session.
|
||||
- **Clean up what you create.** Delete MCP-created sandboxes in the same
|
||||
session; never leave orphaned claims running.
|
||||
- **Bound output.** Use `--tail`, `--since`, selectors, JSONPath, and
|
||||
head/tail excerpts instead of whole manifests or huge logs.
|
||||
|
||||
## 2. Choose the path
|
||||
|
||||
| Need | Path |
|
||||
| --- | --- |
|
||||
| Inspect or change claims, pools, sandboxes, PVCs, gateway | `kubectl` + [platform runbook](references/platform-runbook.md) |
|
||||
| Run code or move files in a sandbox | agent-sandbox MCP, only for a runtime-capable template ([MCP reference](references/mcp-server.md)) |
|
||||
| Add or change a repository feature | [repo extension guide](references/extending-the-repo.md) |
|
||||
|
||||
The MCP server cannot manage this example's DSH claims: its tools are scoped to
|
||||
claims created by the same MCP session. Use `kubectl` for the platform.
|
||||
|
||||
## 3. Operate the platform
|
||||
|
||||
Resolve user -> claim -> sandbox -> pod:
|
||||
|
||||
```bash
|
||||
SB=$(kubectl -n "$NS" get sandboxclaim "$CLAIM" -o jsonpath='{.status.sandbox.name}')
|
||||
kubectl -n "$NS" get sandbox "$SB" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}{" "}{.status.conditions[?(@.type=="Ready")].reason}{"\n"}{.status.serviceFQDN}{"\n"}'
|
||||
POD=$(kubectl -n "$NS" get sandbox "$SB" -o jsonpath='{.metadata.annotations.agents\.x-k8s\.io/pod-name}'); POD=${POD:-$SB}
|
||||
```
|
||||
|
||||
- Read runtime state from the Sandbox, not the claim: `spec.operatingMode`,
|
||||
`Suspended`, `Ready`. The claim tracks binding, not liveness.
|
||||
- Suspend = patch `operatingMode: Suspended` (pod deleted, PVC + Service kept);
|
||||
resume = `Running` + wait for `Ready`. The gateway wakes on connect.
|
||||
- Warm-pool spares carry `agents.x-k8s.io/warm-pool-sandbox`; adopted sandboxes
|
||||
lose it. A pool with no spares makes signup cold and slow.
|
||||
- The DSH launch token appears once in pod logs; hand the user the `?token=` URL,
|
||||
never the token in reports. Probe `/favicon.svg`.
|
||||
- The idle sweeper only suspends claims the gateway created.
|
||||
|
||||
Full commands, security boundaries and failure matrix:
|
||||
[platform runbook](references/platform-runbook.md).
|
||||
|
||||
## 4. Use the agent-sandbox MCP server within its limits
|
||||
|
||||
Verified from the upstream server:
|
||||
|
||||
- Tools: `create_sandbox`, `delete_sandbox`, `execute_command`, `upload_file`,
|
||||
`download_file`, `list_files`, `file_exists`, `get_sandbox_status`; resource
|
||||
`sandboxes://{namespace}`.
|
||||
- **Session-scoped:** every tool and the resource filter claims by
|
||||
`K8S_SANDBOX_SESSION_ID_LABEL_KEY=<MCP session id>`. It cannot read, exec in,
|
||||
or delete DSH platform claims or claims from a prior/reconnected session.
|
||||
- **Streamable HTTP only:** stdio has no MCP session id and fails. A DSH row must
|
||||
use `transport: streamable-http`; health endpoints are `/healthz` and `/readyz`
|
||||
at the root, the MCP endpoint is `/mcp`.
|
||||
- **Runtime required:** connection types target a sandbox runtime on 8888, or
|
||||
sandboxd on 8080/9090. The DSH template listens only on 3080, so
|
||||
`execute_command` cannot work against it. A sandboxd sidecar executes in its
|
||||
own container, not DSH's.
|
||||
- **Timeouts:** `create_sandbox` waits up to 180 s, but the DSH MCP client's
|
||||
default call timeout is 60 s; raise `toolCallTimeoutMs`.
|
||||
`shutdown_after_seconds` defaults to 300 s (`null` disables it).
|
||||
- **Output:** servers before v1.0.8 return unbounded `execute_command` output.
|
||||
v1.0.8 adds `max_output_chars` (128-300,000, default 30,000) and truncation
|
||||
metadata. The repository pins v1.0.8 (`AGENT_SANDBOX_VERSION`), so the
|
||||
documented deployment is bounded; never pass the new argument to an older
|
||||
server, and on one redirect verbose output and read a file instead.
|
||||
|
||||
Deployment, connection matrix and DSH row:
|
||||
[MCP deployment](references/mcp-deployment.md). Limits and cleanup:
|
||||
[MCP reference](references/mcp-server.md).
|
||||
|
||||
## 5. Extend the project
|
||||
|
||||
Read `docs/examples.md` and the testing table in `AGENTS.md` before editing.
|
||||
|
||||
- **Manifest step:** numbered `manifests/NN-*.yaml`, update the README step, run
|
||||
`make check`; CI dry-runs it.
|
||||
- **Gateway:** edit `gateway/gateway.py` and `gateway/test_gateway.py`, update
|
||||
the gateway README API table and architecture doc, run `make test` and
|
||||
`make lint`. Keep the HTTP/WebSocket contract stable.
|
||||
- **Exposure variant:** one variant under `manifests/exposure/<variant>/`,
|
||||
update the exposure README and trusted-host/cookie settings. Variants are
|
||||
alternatives, never a stack.
|
||||
- **New example:** follow `docs/examples.md`, add the index entry, keep scripts
|
||||
executable/idempotent/self-cleaning, run `make check`.
|
||||
- **Version bump:** use `make set-version`; never edit pins by hand. Update
|
||||
prerequisites, run `make check` and the e2e on a cluster. Never mix a bump
|
||||
with an unrelated feature.
|
||||
- **MCP integration:** pin the build to the cluster's tag, least-privilege SA,
|
||||
ClusterIP + auth proxy, raised DSH timeout, no secrets in the repo.
|
||||
- **Optional hardening:** gVisor, `nodeSelector`, NetworkPolicy, extra
|
||||
provider keys and failed-Pod recovery stay opt-in; the sandbox policy lives
|
||||
under `manifests/optional/`. All of them are documented in
|
||||
`examples/dsh-agents-as-a-service/docs/optional-hardening.md`.
|
||||
- **Docs:** update the affected doc in the same change.
|
||||
|
||||
Details and review checklist:
|
||||
[repo extension guide](references/extending-the-repo.md).
|
||||
|
||||
## 6. Report and clean up
|
||||
|
||||
State context/namespace, bounded evidence, conclusion, exact change, rollback,
|
||||
and resources deleted. Disable any MCP row enabled only for the task. Do not
|
||||
leave sandboxes, port-forwards, or temporary files behind.
|
||||
|
||||
## References
|
||||
|
||||
- [Platform runbook](references/platform-runbook.md)
|
||||
- [MCP deployment](references/mcp-deployment.md)
|
||||
- [MCP server reference](references/mcp-server.md)
|
||||
- [Repo extension guide](references/extending-the-repo.md)
|
||||
- [Troubleshooting](references/troubleshooting.md)
|
||||
- Upstream MCP server README:
|
||||
https://github.com/kubernetes-sigs/agent-sandbox/blob/main/clients/integrations/mcp-server/README.md
|
||||
@@ -0,0 +1,132 @@
|
||||
# Extending the repository
|
||||
|
||||
Read `AGENTS.md`, `docs/examples.md`, and the example README before editing.
|
||||
This repository is image-first: the Dockerfile is the primary artifact, every
|
||||
implementation is a self-contained example under `examples/<name>/`, and
|
||||
`versions.env` is the only source of pins. Documentation is part of the change.
|
||||
|
||||
## Invariants
|
||||
|
||||
- Never hard-code a version, image tag, or registry path; edit `versions.env`.
|
||||
For the DSH release identity use `make set-version`, never ad-hoc `sed`.
|
||||
- Never commit a secret. Secret-shaped manifests are `*.yaml.example` with
|
||||
placeholders; keys are documented only in `.env.sample`.
|
||||
- Manifests are numbered `NN-*.yaml` and apply in order with plain `kubectl`.
|
||||
Exposure variants live under `manifests/exposure/<variant>/`, are documented
|
||||
by one README, and are mutually exclusive.
|
||||
- Keep the gateway HTTP/WebSocket contract stable unless the change is explicitly
|
||||
about it.
|
||||
- Do not edit `CHANGELOG.md` (generated by git-cliff) or rewrite git history.
|
||||
- Keep diffs scoped; do not mix a version bump with an unrelated feature.
|
||||
|
||||
## Change matrix
|
||||
|
||||
| Change | Edit | Minimum verification |
|
||||
| --- | --- | --- |
|
||||
| Documentation only | `*.md`, `docs/` | `make check`; relative links resolve |
|
||||
| Example manifest | `examples/<name>/manifests/NN-*.yaml`, README step | `make check`; server-side dry-run in CI |
|
||||
| Optional hardening | `manifests/optional/`, exposure NPs, example `docs/optional-hardening.md` | `make check`; manual verification on a capable cluster |
|
||||
| Gateway behavior/route | `gateway/gateway.py`, `gateway/test_gateway.py`, gateway README, architecture | `make test`, `make lint`; update tests and API docs |
|
||||
| Exposure variant | `manifests/exposure/<variant>/`, exposure README, example README | `make check`; apply exactly one variant |
|
||||
| New example | full `examples/<name>/` layout, `examples/README.md` | `make check`; CI dry-run; optional e2e |
|
||||
| Version bump | `versions.env`, affected docs/manifests | `make check`; CI installs the pin; e2e on a cluster |
|
||||
| MCP integration | deployment/RBAC, DSH row, docs | read-only `/readyz`; no secret in repo |
|
||||
| Workflow | `.github/workflows/` | actionlint; SHA-pinned actions; no hard-coded versions |
|
||||
|
||||
## Recipes
|
||||
|
||||
### Add or change a resource step
|
||||
|
||||
1. Keep numbering stable; add `NN-*.yaml` at the end unless order matters.
|
||||
2. Update the matching numbered step in the example README.
|
||||
3. Keep cluster-specific state (node affinity, tuned replica counts) out of the
|
||||
shipped manifest; document it as a local patch.
|
||||
4. Run `make check`. CI additionally parses every manifest and runs a
|
||||
server-side dry-run against the agent-sandbox CRDs.
|
||||
|
||||
### Change the gateway
|
||||
|
||||
The gateway is a single-file Starlette app. A change usually touches:
|
||||
|
||||
- `gateway/gateway.py` (route, auth, proxy, sweeper);
|
||||
- `gateway/test_gateway.py` (unit tests; no cluster required);
|
||||
- `gateway/README.md` (API table, configuration, auth semantics);
|
||||
- `examples/dsh-agents-as-a-service/docs/architecture.md` when flows change;
|
||||
- `examples/dsh-agents-as-a-service/README.md` when the walkthrough changes.
|
||||
|
||||
Run `make test` and `make lint`. Preserve session-cookie auth, same-origin rules
|
||||
for cookie mutations and WebSocket upgrades, and the "waking"/`502`/`503`
|
||||
semantics unless the change is explicitly about them. Do not expose a trusted
|
||||
identity header unless the Service is reachable only through the proxy.
|
||||
|
||||
### Add an exposure variant
|
||||
|
||||
Create `manifests/exposure/<variant>/NN-*.yaml`, update
|
||||
`manifests/exposure/README.md` and the example README, and state the trust
|
||||
boundary. Variants are alternatives: never apply two paths to the same gateway.
|
||||
Set `DSH_TRUSTED_HOSTS` in the template for the public host and
|
||||
`COOKIE_SECURE=true` for HTTPS.
|
||||
|
||||
### Add an example
|
||||
|
||||
Follow `docs/examples.md`: `README.md`, `manifests/NN-*.yaml`, optional
|
||||
`manifests/exposure/` with a README, optional components with Dockerfile +
|
||||
README + tests, and executable, idempotent, self-cleaning scripts. Add the
|
||||
literal `examples/<name>/` entry to `examples/README.md`; `make check` enforces
|
||||
the layout. Publish component images under
|
||||
`<registry>/<owner>/<repo>/<example>/<component>`.
|
||||
|
||||
### Bump a version
|
||||
|
||||
- **DSH release identity:** `make set-version DSH_VERSION=<ref>
|
||||
[RELEASE_REVISION=rN]`, then `make check`, then build and smoke-test the image.
|
||||
Tags are immutable; fix a bad release with `-r(N+1)`.
|
||||
- **agent-sandbox:** edit `AGENT_SANDBOX_VERSION` in `versions.env`, update the
|
||||
prerequisites in the example README, and note behavior changes (connection
|
||||
types, MCP output limits, CRD fields). Install the new release YAML on a kind
|
||||
cluster, then run `make check` and `make e2e`. Do not bump it inside an
|
||||
unrelated PR.
|
||||
|
||||
### Integrate the agent-sandbox MCP server
|
||||
|
||||
Build from the cluster's pinned tag, give it a dedicated namespace and
|
||||
least-privilege ServiceAccount, keep it ClusterIP behind an authenticating
|
||||
proxy, and document how to reach `/readyz`. Add a DSH profile row only with user
|
||||
approval; a row affects every session. Never commit its credentials or a
|
||||
`cluster-admin` binding.
|
||||
|
||||
### Documentation
|
||||
|
||||
Update the doc whose behavior changed in the same commit: root README for the
|
||||
front door, `docs/` for repository-wide contracts, the example README for that
|
||||
walkthrough, the gateway README for API/config/auth. Skill docs live in
|
||||
`.dsh/skills/`; keep `SKILL.md` under ~8 KB and put depth in `references/`.
|
||||
`scripts/check-versions.sh` scans every `*.md` for stale `dsh-v…` tokens, so
|
||||
refer to "the release identity in `versions.env`" rather than copying a tag.
|
||||
|
||||
## Verification and review
|
||||
|
||||
Run at least `make check`; add `make test` and `make lint` for gateway/script
|
||||
changes. For manifests, use `kubectl apply --dry-run=server` against a cluster
|
||||
with the CRDs. For behavior changes, run the example or `make e2e`.
|
||||
|
||||
Before reporting, review:
|
||||
|
||||
- [ ] Diff is scoped; no generated files or unrelated changes.
|
||||
- [ ] Docs and examples match the new behavior.
|
||||
- [ ] Pins come only from `versions.env`; no stale version tokens.
|
||||
- [ ] No secrets or usable credentials; secret-shaped files are `.example`.
|
||||
- [ ] Manifests parse; exposure variants remain mutually exclusive.
|
||||
- [ ] Gateway tests and public contract unchanged or intentionally updated.
|
||||
- [ ] Skills: frontmatter valid, body within budget, scripts shellcheck-clean.
|
||||
- [ ] `CHANGELOG.md` untouched; branch/commit follow `docs/gitflow.md`.
|
||||
|
||||
## Safe follow-ups enabled by this skill
|
||||
|
||||
- A read-only preflight job or scheduled report using
|
||||
`scripts/platform-preflight.sh`.
|
||||
- An MCP readiness check in the example e2e (health endpoints only).
|
||||
- A separate runtime warm pool for untrusted execution, leaving the DSH template
|
||||
unchanged.
|
||||
- A combined DSH + sandboxd image only with an explicit maintenance/exec design
|
||||
and a documented trust boundary.
|
||||
@@ -0,0 +1,94 @@
|
||||
# Deploying the agent-sandbox MCP server
|
||||
|
||||
Read [the MCP reference](mcp-server.md) first. This file covers the connection
|
||||
matrix, a safe deployment shape, and the DSH profile row. The server is
|
||||
unauthenticated; its service account is the security boundary.
|
||||
|
||||
## Connection matrix
|
||||
|
||||
| `K8S_SANDBOX_CONNECTION__TYPE` | Required env | Use when |
|
||||
| --- | --- | --- |
|
||||
| `gateway` | `GATEWAY_NAME`, `GATEWAY_NAMESPACE=default`, `GATEWAY_READY_TIMEOUT=180`, `SERVER_PORT=8888` | Gateway API + implementation exist |
|
||||
| `direct` | `API_URL` (router URL), `SERVER_PORT=8888` | sandbox-router is deployed |
|
||||
| `in-cluster` | `USE_POD_IP=false` (DNS) or `true` (`status.podIPs`), `SERVER_PORT=8888` | Bypass the router from inside the cluster |
|
||||
| `sandboxd-in-cluster` (v1.0.8+) | `MODE=in-cluster-service` or `in-cluster-pod-ip`, `REST_PORT=8080`, `GRPC_PORT=9090` | Template runs sandboxd |
|
||||
|
||||
Also set `K8S_SANDBOX_PROBE_NAMESPACE` to a namespace the server's service
|
||||
account can list claims in, and optionally
|
||||
`K8S_SANDBOX_SESSION_ID_LABEL_KEY` (default
|
||||
`mcp.k8s-agent-sandbox/session-id`).
|
||||
|
||||
For `sandboxd-in-cluster` service mode the template needs `spec.service: true`.
|
||||
The default managed NetworkPolicy admits only sandbox-router, so add an ingress
|
||||
rule for the MCP server on 8080/9090. Supplying `spec.networkPolicy` replaces the
|
||||
secure defaults: also add the egress rules the sandbox needs, because an omitted
|
||||
egress list denies all outbound traffic. RBAC for that mode is
|
||||
`sandboxclaims` create/get/list/watch/delete and `sandboxes` get/list/watch; no
|
||||
`pods/portforward`.
|
||||
|
||||
## Build and deploy (sketch)
|
||||
|
||||
No official image reference is documented in the upstream README; pin what you
|
||||
build.
|
||||
|
||||
```bash
|
||||
# AGENT_SANDBOX_VERSION comes from versions.env; never hard-code it.
|
||||
tmp="$(mktemp -d)"; trap 'rm -rf "$tmp"' EXIT
|
||||
git clone --depth 1 --branch "$AGENT_SANDBOX_VERSION" \
|
||||
https://github.com/kubernetes-sigs/agent-sandbox.git "$tmp"
|
||||
docker build -t "$MCP_IMAGE" "$tmp/clients/integrations/mcp-server"
|
||||
```
|
||||
|
||||
Kubernetes shape:
|
||||
|
||||
- Dedicated namespace and ServiceAccount; a namespace-scoped Role granting only
|
||||
`sandboxclaims` create/get/list/watch/delete and `sandboxes` get/list/watch.
|
||||
No Secrets, no `pods/exec`, no cluster-admin.
|
||||
- Deployment: one replica, non-root, `readOnlyRootFilesystem`, dropped
|
||||
capabilities, `K8S_SANDBOX_CONNECTION__TYPE=in-cluster`,
|
||||
`K8S_SANDBOX_PROBE_NAMESPACE`, resource limits.
|
||||
- Probes: `GET /healthz` (liveness; never calls the API) and `GET /readyz`
|
||||
(readiness; lists claims; use `periodSeconds: 15`, not aggressive polling).
|
||||
- ClusterIP Service on `8000`. Add a NetworkPolicy that allows only the intended
|
||||
client namespace/pod. Never expose it through Ingress/LoadBalancer without an
|
||||
authenticating proxy. The service account is the real boundary.
|
||||
|
||||
## Connect it to DSH
|
||||
|
||||
Add a profile MCP row (through the Plugins page/`plugin_manager`; see
|
||||
`cordis-plugin-development`). A row affects every session and needs approval.
|
||||
|
||||
```yaml
|
||||
- insert:
|
||||
- id: mcp-agent-sandbox
|
||||
name: '@deepseek-ai/dsh-mcp-client'
|
||||
config:
|
||||
serverName: agent-sandbox
|
||||
transport: streamable-http
|
||||
url: http://agent-sandbox-mcp.example.svc.cluster.local:8000/mcp
|
||||
toolCallTimeoutMs: 240000
|
||||
failOnStartupError: false
|
||||
```
|
||||
|
||||
- `serverName` must match `[A-Za-z0-9_-]{1,32}`; tools appear as
|
||||
`mcp__<serverName>__<tool>`. Check the actual names in the tool list; do not
|
||||
assume hyphens/underscores.
|
||||
- Raise `toolCallTimeoutMs` above `sandbox_ready_timeout` with headroom; the DSH
|
||||
default is 60 s and would cancel a 180 s creation.
|
||||
- If the server is behind an authenticating proxy, pass its token via the row's
|
||||
`headers` and an environment expression; never commit it.
|
||||
- To reach a locally port-forwarded server, use a loopback URL; the MCP session
|
||||
still needs streamable HTTP.
|
||||
- Remove the row when it was enabled only for one task.
|
||||
|
||||
## Deployment safety checklist
|
||||
|
||||
- [ ] Build tag equals the cluster's `AGENT_SANDBOX_VERSION`.
|
||||
- [ ] Dedicated namespace/SA; RBAC limited to claims and sandboxes.
|
||||
- [ ] Service is ClusterIP; no public route without authentication.
|
||||
- [ ] NetworkPolicy admits only the DSH client and needed sandbox traffic.
|
||||
- [ ] `/readyz` succeeds with `K8S_SANDBOX_PROBE_NAMESPACE` set to a listable
|
||||
namespace.
|
||||
- [ ] No Secret data in manifests, logs, or the repository.
|
||||
- [ ] Cleanup path documented: delete session-owned claims or wait for the
|
||||
shutdown timeout.
|
||||
107
.dsh/skills/dsh-agent-sandbox-platform/references/mcp-server.md
Normal file
107
.dsh/skills/dsh-agent-sandbox-platform/references/mcp-server.md
Normal file
@@ -0,0 +1,107 @@
|
||||
# agent-sandbox MCP server reference
|
||||
|
||||
Upstream: [clients/integrations/mcp-server](https://github.com/kubernetes-sigs/agent-sandbox/tree/main/clients/integrations/mcp-server).
|
||||
It is a Python/FastAPI + FastMCP service (streamable HTTP on `:8000`, MCP
|
||||
endpoint `/mcp`) that provisions sandboxes from a warm pool and runs commands or
|
||||
file operations through a sandbox runtime. It ships with **no authentication**.
|
||||
|
||||
Build the server from the same tag as the cluster: read `AGENT_SANDBOX_VERSION`
|
||||
in `versions.env`, and check the installed controller image. Do not build from
|
||||
`main` against an older cluster; tool schemas and connection types differ.
|
||||
|
||||
## Tool surface
|
||||
|
||||
| Tool | Arguments (defaults) | Returns / notes |
|
||||
| --- | --- | --- |
|
||||
| `create_sandbox` | `warmpool`, `namespace`, `sandbox_ready_timeout=180`, `labels`, `shutdown_after_seconds=300` (`null` disables), `pod_labels`, `pod_annotations` | `sandbox_claim_name`; adds the MCP session label |
|
||||
| `delete_sandbox` | `sandbox_claim_name`, `namespace` | Deletes the session-owned claim; PVC cascades |
|
||||
| `execute_command` | `sandbox_claim_name`, `namespace`, `command`, `timeout=60` (max 600), `max_output_chars` only on v1.0.8+ (128-300,000, default 30,000) | `exit_code`, `stdout`, `stderr`; v1.0.8 adds `*_total_chars` and `*_truncated` |
|
||||
| `upload_file` | `sandbox_claim_name`, `namespace`, `path`, `content`, `binary=false`, `timeout=60` | Bytes written; base64 when `binary=true` |
|
||||
| `download_file` | `sandbox_claim_name`, `namespace`, `path`, `binary=false`, `timeout=60` | Content + bytes read |
|
||||
| `list_files` | `sandbox_claim_name`, `namespace`, `path`, `timeout=60`, `max_entries=1000` (max 10,000) | Entries (`name`, `size`, `type`, `mod_time`), `total_entries`, `truncated` |
|
||||
| `file_exists` | `sandbox_claim_name`, `namespace`, `path`, `timeout=60` | `exists`; a missing path is not an error |
|
||||
| `get_sandbox_status` | `sandbox_claim_name`, `namespace` | `SandboxReady` / `SandboxNotReady` / `SandboxNotFound`, `ready`, `message` |
|
||||
|
||||
Resource: `sandboxes://{namespace}` lists sandboxes for the current MCP session.
|
||||
|
||||
## Session scoping is the most important behavior
|
||||
|
||||
Every tool, including `get_sandbox_status`, calls `get_sandbox()` ->
|
||||
`ensure_session_owns()`, which lists sandboxes with the label selector
|
||||
`<K8S_SANDBOX_SESSION_ID_LABEL_KEY>=<MCP session id>`. The resource uses the same
|
||||
selector. Consequences:
|
||||
|
||||
- The server can only see/exec/delete claims **it created in the same MCP
|
||||
session**. It cannot inspect or manage DSH platform claims created by the
|
||||
gateway or `kubectl`, or claims from another session.
|
||||
- On a DSH MCP reconnect, the MCP session changes; earlier claims become
|
||||
invisible to the tools. They keep running until `shutdown_after_seconds`
|
||||
expires, or until you remove them by label with a confirmed `kubectl delete`.
|
||||
- A "Sandbox claim '<name>' is not found in namespace '<ns>'" error usually
|
||||
means wrong namespace **or** the claim belongs to a different session.
|
||||
- Use `kubectl` for global inventory; use the MCP only for sandboxes it created
|
||||
in this task.
|
||||
|
||||
Session identity requires a session-providing transport. With stdio,
|
||||
`ctx.session_id` is `None` and creation fails. Connect DSH over
|
||||
`streamable-http`.
|
||||
|
||||
## Runtime prerequisite
|
||||
|
||||
The server talks to a sandbox runtime, not to arbitrary pods:
|
||||
|
||||
- `gateway`, `direct`, `in-cluster`: runtime HTTP on `SERVER_PORT=8888`
|
||||
(sandbox-router API or direct pod runtime).
|
||||
- `sandboxd-in-cluster` (v1.0.8+): sandboxd REST `8080`, gRPC `9090`.
|
||||
|
||||
The shipped DSH template runs only DSH Web on `3080`, so no connection type can
|
||||
exec into it. A sandboxd **sidecar** executes commands in the sidecar's own
|
||||
container, not in the DSH container; shared volumes share files, not binaries or
|
||||
processes. To exec inside DSH, build a combined image (sandboxd injected into the
|
||||
DSH image) or use `kubectl exec` with explicit RBAC and confirmation. For
|
||||
untrusted code, deploy a separate runtime warm pool rather than modifying the
|
||||
DSH template.
|
||||
|
||||
## Version differences
|
||||
|
||||
| Area | Before v1.0.8 | v1.0.8+ |
|
||||
| --- | --- | --- |
|
||||
| Connection types | `gateway`, `direct`, `in-cluster` | Adds `sandboxd-in-cluster` |
|
||||
| `execute_command` output | Unbounded | Bounded by `max_output_chars`; four metadata fields |
|
||||
| Failed pod recovery | `Retain` only | `spec.failedPodPolicy: Recreate` |
|
||||
| Router install | `kubectl apply -f sandbox-router/deploy/` | `kubectl apply -k sandbox-router/deploy/`; projected token |
|
||||
|
||||
Never pass `max_output_chars` to a pre-v1.0.8 server. On older servers, redirect
|
||||
verbose output to a file and `download_file` it, or pipe through `tail`/`head`.
|
||||
|
||||
The repository pins `AGENT_SANDBOX_VERSION=v1.0.8`, so the right-hand column is
|
||||
what the documented platform and a same-tag MCP build provide; keep the
|
||||
before-column for clusters still on v1.0.7.
|
||||
|
||||
## Safety and cleanup
|
||||
|
||||
- No auth: anyone reaching `:8000` gets every tool and the service account's
|
||||
access. Keep it ClusterIP, authenticated, and namespace-scoped.
|
||||
- Treat commands as untrusted code: they can read files and environment
|
||||
variables inside the sandbox, including provider keys in a combined image.
|
||||
- `delete_sandbox` cascades the claim, sandbox, and PVC: confirm the target and
|
||||
namespace. Never delete a namespace or PVC as cleanup.
|
||||
- Clean up in the same session that created the sandbox. If the session is gone,
|
||||
list claims by the session label and delete them explicitly (requires write
|
||||
RBAC); otherwise rely on the shutdown timeout.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Cause / fix |
|
||||
| --- | --- |
|
||||
| Tools missing | Row not enabled, wrong URL, connection failed; check logs and `/readyz` |
|
||||
| 404 on connect | URL is missing `/mcp` or points at the wrong port |
|
||||
| `ctx.session_id is None` | Connected over stdio; use streamable HTTP |
|
||||
| `Sandbox claim ... is not found` | Wrong namespace, or claim created by another MCP session |
|
||||
| Creation times out at ~60 s | MCP session alive but DSH cancelled the call; raise `toolCallTimeoutMs` |
|
||||
| `execute_command` connection error | No runtime on 8888/8080/9090, or wrong connection type; the DSH template is unsupported |
|
||||
| Output flood | Pre-v1.0.8 server; bound the command output yourself |
|
||||
| Empty sandbox list | The resource is session-scoped; use `kubectl` for global inventory |
|
||||
|
||||
Deployment, connection matrix, RBAC, and the DSH row:
|
||||
[MCP deployment](mcp-deployment.md).
|
||||
@@ -0,0 +1,200 @@
|
||||
# Platform runbook
|
||||
|
||||
Scope: the resource layer and gateway in
|
||||
`examples/dsh-agents-as-a-service/`, one namespace, one DSH Web sandbox per
|
||||
user. Read the example README and `AGENTS.md` first. All commands are read-only
|
||||
unless a section is explicitly a change; confirm before changing anything.
|
||||
|
||||
## 1. Context, version, CRDs
|
||||
|
||||
```bash
|
||||
kubectl config current-context
|
||||
kubectl auth whoami
|
||||
kubectl config view --minify -o jsonpath='{..namespace}{"\n"}'
|
||||
kubectl get crd | grep -E 'agents\.x-k8s\.io'
|
||||
kubectl -n agent-sandbox-system get deploy agent-sandbox-controller \
|
||||
-o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
|
||||
```
|
||||
|
||||
Compare the controller image tag with `AGENT_SANDBOX_VERSION` in
|
||||
`versions.env`. The example needs the extension CRDs plus core `sandboxes`;
|
||||
a missing CRD is a prerequisite failure, not an application bug.
|
||||
|
||||
## 2. Resource model
|
||||
|
||||
| Object | Role | Key fields |
|
||||
| --- | --- | --- |
|
||||
| `SandboxTemplate` | Blueprint; pinned image, probes, security context, PVC template | `envVarsInjectionPolicy`, `volumeClaimTemplatesPolicy`, `service`, `podTemplate` |
|
||||
| `SandboxWarmPool` | Pre-warmed spares from one template | `spec.replicas`, `sandboxTemplateRef` |
|
||||
| `SandboxClaim` | One tenant/user request; adopts a spare | `spec.warmPoolRef`, `additionalPodMetadata`, `status.sandbox.name` |
|
||||
| `Sandbox` | Adopted or directly created pod + Service + PVC | `spec.operatingMode`, `status.conditions`, `status.serviceFQDN`, `status.podIPs` |
|
||||
| `Pod` | The DSH Web process | `:3080`, PVC at `/home/node/.dsh` and `/workspace` (`subPath: workspace`) |
|
||||
| `PersistentVolumeClaim` | Per-sandbox DSH state + agent workspace | `dsh-home-<sandbox>` |
|
||||
| `Service` | Stable per-sandbox DNS | `<sandbox>`, `status.serviceFQDN` |
|
||||
|
||||
Delete a claim to cascade-delete its sandbox and PVC (account deletion). Never
|
||||
delete a PVC or namespace without an explicit, named confirmation.
|
||||
|
||||
## 3. Read-only inspection
|
||||
|
||||
```bash
|
||||
kubectl -n "$NS" get sandboxtemplates,sandboxwarmpools,sandboxclaims,sandboxes -o wide
|
||||
kubectl -n "$NS" get pods,pvc,svc -o wide
|
||||
kubectl -n "$NS" get events --sort-by=.lastTimestamp | tail -30
|
||||
kubectl -n "$NS" get sandboxclaim "$CLAIM" -o yaml
|
||||
kubectl -n "$NS" get sandbox "$SB" -o yaml
|
||||
```
|
||||
|
||||
Bounded logs: `kubectl -n "$NS" logs "$POD" --tail=200` (add `--previous`
|
||||
only after a restart).
|
||||
|
||||
## 4. Claim -> sandbox -> pod
|
||||
|
||||
```bash
|
||||
SB=$(kubectl -n "$NS" get sandboxclaim "$CLAIM" -o jsonpath='{.status.sandbox.name}')
|
||||
POD=$(kubectl -n "$NS" get sandbox "$SB" -o jsonpath='{.metadata.annotations.agents\.x-k8s\.io/pod-name}')
|
||||
POD=${POD:-$SB}
|
||||
kubectl -n "$NS" get sandbox "$SB" -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}{"\n"}{end}'
|
||||
kubectl -n "$NS" get sandbox "$SB" -o jsonpath='{.spec.operatingMode}{" "}{.status.serviceFQDN}{" "}{.status.podIPs}{"\n"}'
|
||||
```
|
||||
|
||||
Rules:
|
||||
|
||||
- Resolve user -> sandbox only through `claim.status.sandbox.name`; the sandbox
|
||||
keeps its pool-generated name, not the claim name.
|
||||
- A claim tracks binding; for runtime state read `Sandbox.spec.operatingMode`
|
||||
and the `Suspended`/`Ready` conditions. A suspended sandbox is not serving
|
||||
even when the claim still exists.
|
||||
- A missing `claim.status.sandbox.name` means the claim did not bind: inspect
|
||||
the Ready condition `reason` and events. The pod is normally named after the
|
||||
sandbox; use the annotation when present.
|
||||
|
||||
## 5. Suspend, resume, wake
|
||||
|
||||
Suspend keeps the PVC and Service, deletes the pod:
|
||||
|
||||
```bash
|
||||
kubectl -n "$NS" patch sandbox "$SB" --type merge \
|
||||
-p '{"spec":{"operatingMode":"Suspended"}}'
|
||||
kubectl -n "$NS" wait --for=delete "pod/$POD" --timeout=120s
|
||||
kubectl -n "$NS" get pvc "dsh-home-$SB"
|
||||
kubectl -n "$NS" get svc "$SB"
|
||||
```
|
||||
|
||||
Resume and verify state:
|
||||
|
||||
```bash
|
||||
kubectl -n "$NS" patch sandbox "$SB" --type merge \
|
||||
-p '{"spec":{"operatingMode":"Running"}}'
|
||||
kubectl -n "$NS" wait sandbox "$SB" --for=condition=Ready --timeout=180s
|
||||
kubectl -n "$NS" exec "$SB" -- cat /home/node/.dsh/marker.txt
|
||||
```
|
||||
|
||||
The gateway wakes a suspended sandbox on the first request. The idle sweeper
|
||||
only suspends claims it created (`aaas.example.com/managed-by=dsh-gateway`);
|
||||
hand-applied claims are never swept.
|
||||
|
||||
## 6. Warm pool
|
||||
|
||||
```bash
|
||||
kubectl -n "$NS" get sandboxwarmpool "$POOL" -o yaml
|
||||
kubectl -n "$NS" get sandboxes -l agents.x-k8s.io/warm-pool-sandbox -o wide
|
||||
kubectl -n "$NS" get sandboxwarmpool "$POOL" -o jsonpath='{.spec.replicas}{"\n"}'
|
||||
```
|
||||
|
||||
Adopted sandboxes lose the warm-pool label; count current spares against
|
||||
`spec.replicas`. Zero spares or `spec.replicas: 0` means cold, slow signup.
|
||||
Scale only when asked:
|
||||
|
||||
```bash
|
||||
kubectl -n "$NS" patch sandboxwarmpool "$POOL" --type merge -p '{"spec":{"replicas":2}}'
|
||||
```
|
||||
|
||||
## 7. DSH runtime specifics
|
||||
|
||||
- DSH Web listens on `3080` in-cluster. Port-forward to a free local port such
|
||||
as `18081`, never `3080` if a local DSH already uses it.
|
||||
- The launch token is printed once as `dsh web: http://<ip>:3080/?token=...`.
|
||||
Treat it as a credential: do not paste it into reports; hand the user the URL.
|
||||
- `/` returns `401` without the token. Probes must target `/favicon.svg`.
|
||||
- `DSH_TRUSTED_HOSTS` must include any public host that a proxy forwards; DSH
|
||||
fences on `Host`/`Origin`.
|
||||
- State lives at `/home/node/.dsh`. DSH requires `.credentials.yaml` to be
|
||||
`0600`; the template uses `fsGroupChangePolicy: OnRootMismatch`. If a resumed
|
||||
sandbox crash-loops with a mode-660 error, suspend it, fix the file with a
|
||||
throwaway pod, patch the policy, then resume.
|
||||
- Agent files live at `/workspace`, the same PVC mounted with
|
||||
`subPath: workspace`; it survives suspend/resume and dies with the PVC.
|
||||
- The templates set `HOME=/workspace` (the image default is `/home/node`). DSH's
|
||||
directory picker lists `$HOME` when it opens and has no start-path setting, so
|
||||
this is what makes the folder explorer start on the persisted workspace; `~`
|
||||
and `$HOME`-relative tool caches/config also resolve there. `DSH_HOME` remains
|
||||
`/home/node/.dsh`, so state and credentials stay out of the workspace. Pod env
|
||||
is stamped at sandbox creation: a sandbox created before the override keeps
|
||||
the old `HOME` until it is recreated (or its `podTemplate` is patched and the
|
||||
sandbox resumed). Verify with
|
||||
`kubectl -n "$NS" exec "$POD" -- printenv HOME`.
|
||||
- DSH confines bash with its own file sandbox (`bubblewrap`, else Landlock) and
|
||||
fails closed with `SANDBOX_UNAVAILABLE` when neither works. The image ships
|
||||
`bubblewrap`, but it is unsetuid and needs an unprivileged user namespace that
|
||||
this hardened pod (all capabilities dropped, RuntimeDefault seccomp) does not
|
||||
grant; that plus no Landlock runner is why the template sets
|
||||
`DSH_PERMISSION_MODE=danger-full-access` and treats the pod as the boundary.
|
||||
Never "fix" it by granting `CAP_SYS_ADMIN` or `seccompProfile: Unconfined`
|
||||
without an explicit decision.
|
||||
|
||||
## 8. Gateway layer
|
||||
|
||||
```bash
|
||||
kubectl -n "$NS" get deploy,svc,pods -l app=dsh-gateway -o wide
|
||||
kubectl -n "$NS" get endpoints dsh-gateway
|
||||
kubectl -n "$NS" logs deploy/dsh-gateway --tail=100
|
||||
```
|
||||
|
||||
The gateway is a ClusterIP Service on `8080`; it proxies under `/u/{user}/` and
|
||||
derives state (`Provisioning`, `Waking`, `Ready`, `Suspending`, `Suspended`).
|
||||
It performs the DSH launch-token handshake server-side. With SSO enabled
|
||||
(`AUTH_PROXY_HEADER`), the Service must be reachable only through the
|
||||
authenticating proxy; never add a second path or port-forward it on a shared
|
||||
host. A resource-only install has no gateway: that is a warning, not a failure.
|
||||
|
||||
## 9. Security boundaries
|
||||
|
||||
- Provider keys (`dsh-provider-keys` plus the optional
|
||||
`dsh-provider-keys-extra`) are shared by all sandboxes; the example is not
|
||||
tenant-isolated.
|
||||
- gVisor, `nodeSelector`, the sandbox/exposure NetworkPolicies and extra
|
||||
provider keys are opt-in, not active by default. A NetworkPolicy only helps
|
||||
with a CNI that enforces it; see
|
||||
[optional hardening](../../../../examples/dsh-agents-as-a-service/docs/optional-hardening.md).
|
||||
- Secret data, kubeconfigs, and pod logs are sensitive; RBAC may allow Secret
|
||||
access, policy does not. Keep gateway RBAC namespace-scoped; it needs pod logs
|
||||
to find the launch token.
|
||||
- TLS terminates outside the gateway; set `COOKIE_SECURE=true` for HTTPS.
|
||||
|
||||
## 10. Failure matrix
|
||||
|
||||
| Symptom | First evidence | Likely cause / fix |
|
||||
| --- | --- | --- |
|
||||
| Claim not Ready, no sandbox | Ready condition reason, events | Empty pool, image pull, quota, unschedulable |
|
||||
| `EnvVarsInjectionRejected` | claim Ready reason | Claim carries env/volume templates; template policy rejects it |
|
||||
| Pod Pending | `kubectl describe pod` | Requests/taints/affinity, unbound PVC |
|
||||
| CrashLoopBackOff | logs, `--previous` | Startup failure, bad config, probe 401, fsGroup mode 660 |
|
||||
| ImagePullBackOff | describe events | Wrong tag, private registry, missing pull secret |
|
||||
| DSH 401 in browser | gateway logs | Launch-token handshake failed; check gateway RBAC `pods/log` |
|
||||
| 502/503 through gateway | sandbox Ready, gateway logs | Wake timeout, DNS stale, token not in logs yet |
|
||||
| Warm pool empty | sandboxes label count | Scale the pool or wait for replenishment |
|
||||
| Suspended but expected running | `operatingMode` | Resume it; the gateway only wakes on a request |
|
||||
| WS closes `1011` | gateway logs | DSH rejected the upstream Host/authority; check trusted hosts |
|
||||
|
||||
## 11. Cleanup and reporting
|
||||
|
||||
Delete only claims/sandboxes you created, by name:
|
||||
|
||||
```bash
|
||||
kubectl -n "$NS" delete sandboxclaim "$CLAIM" --dry-run=server
|
||||
kubectl -n "$NS" delete sandboxclaim "$CLAIM"
|
||||
```
|
||||
|
||||
Report context/namespace, evidence, conclusion, change, rollback, and what was
|
||||
deleted. Leave the namespace, pools, templates, and unrelated claims alone.
|
||||
@@ -0,0 +1,74 @@
|
||||
# Troubleshooting
|
||||
|
||||
Start with evidence, not fixes: resolve context/namespace, read conditions and
|
||||
events, then compare desired vs observed state. Keep output bounded. Never
|
||||
"fix" an access error by broadening credentials.
|
||||
|
||||
## Platform
|
||||
|
||||
| Symptom | First evidence | Likely cause / fix |
|
||||
| --- | --- | --- |
|
||||
| `Forbidden` on every call | `kubectl auth whoami`, `kubectl auth can-i --list -n <ns>` | Wrong context or RBAC. Do not retry with broader credentials; ask for the right kubeconfig. |
|
||||
| CRDs not found | `kubectl get crd | grep agents.x-k8s.io` | agent-sandbox extensions not installed. Install the pinned release YAML; that is a prerequisite, not a code change. |
|
||||
| Claim created, no sandbox | `claim.status.conditions[?(@.type=="Ready")]`, events | Empty pool, image pull, quota, unschedulable; or policy rejection. |
|
||||
| Claim rejected | Ready reason `EnvVarsInjectionRejected` | Claim carries `env`/`volumeClaimTemplates` while the template disallows injection. Remove them; do not bypass the pool. |
|
||||
| User is suspended but the claim exists | `Sandbox.spec.operatingMode`, `Suspended` condition | Resume with `operatingMode: Running`; a gateway request would wake it. |
|
||||
| Warm pool exhausted | `kubectl get sandboxes -l agents.x-k8s.io/warm-pool-sandbox` | Scale `spec.replicas` (with confirmation) or accept slow cold provisioning. |
|
||||
| Pod `Pending` | `kubectl describe pod` | Requests/limits, taints, affinity, unbound PVC, storage class. |
|
||||
| `ImagePullBackOff` | describe events | Wrong image/tag, private GHCR package, missing pull secret; on kind, load the image under the exact manifest ref. |
|
||||
| `CrashLoopBackOff`, probe 401 | logs, `describe` | Probe path is `/`; use `/favicon.svg`. |
|
||||
| Restart with mode-660 error | logs mention `.credentials.yaml` | kubelet fsGroup widened the file; suspend, `chmod 600` from a throwaway pod, patch `fsGroupChangePolicy: OnRootMismatch`, resume. |
|
||||
| Every bash command fails with `SANDBOX_UNAVAILABLE` | DSH tool result names the requested mode and backends | DSH's own command sandbox has no usable runner: `bubblewrap` ships in the image but is unsetuid and needs an unprivileged user namespace this hardened pod does not grant, and no Landlock runner is available. The example template sets `DSH_PERMISSION_MODE=danger-full-access`; do not grant capabilities to restore it unless the operator explicitly accepts that downgrade. |
|
||||
| Folder picker opens on `/home/node`, not the workspace | `kubectl exec <pod> -- printenv HOME` | The sandbox predates the template's `HOME=/workspace` and pod env is stamped at creation. Recreate the sandbox (PVC survives) or patch its `podTemplate` with `HOME`, then resume. Workspace dotfiles (`.cache`, `.config`, `.npm`, `.agents`) are the expected trade-off of the override, not a fault. |
|
||||
| PVC `Pending` | `kubectl describe pvc`, StorageClass | No default provisioner, access mode, zone binding. |
|
||||
| Launch token not found | `kubectl logs <pod> --tail=2000 | grep 'dsh web:'` | Pod not ready yet, or logs rotated. Do not guess tokens; wait for readiness. |
|
||||
| Sandbox reports `Finished=True` | pod phase, `kubectl get events` | The backing Pod reached `Failed` (eviction, spot reclaim) and `failedPodPolicy` defaults to `Retain`. Delete the Pod to force a replacement, or set `Recreate` (v1.0.8+) on the Sandbox/template; the claim and PVCs survive either way. |
|
||||
| Hand-applied sandbox disappeared | claim annotations, gateway logs | Idle sweeper only manages claims with `aaas.example.com/managed-by=dsh-gateway`; if it was managed, recreate/resume. |
|
||||
| Gateway 401 | gateway logs, cookie flags | Missing/expired cookie, DSH handshake failed, or `COOKIE_SECURE=true` over HTTP. |
|
||||
| Gateway 502/503 | sandbox Ready, gateway logs | Wake timeout, stale DNS, token not yet in logs; check `WAKE_TIMEOUT` and RBAC `pods/log`. |
|
||||
| WebSocket closes 1011 | gateway logs, `Host`/Origin | DSH rejected the upstream authority; check `DSH_TRUSTED_HOSTS` and the gateway's pod-IP dial path. |
|
||||
| WebSocket closes 4401 | cookie, Origin | Missing gateway auth or cross-origin upgrade; send a matching `Origin`. |
|
||||
| SSO header ignored | gateway env | `AUTH_PROXY_HEADER` unset; setting it requires the proxy to be the only path to the ClusterIP Service. |
|
||||
| Someone is treated as another user | service paths | Trust-boundary violation: the gateway is directly reachable. Remove the extra path and disable SSO until fixed. |
|
||||
|
||||
## Agent-sandbox MCP
|
||||
|
||||
| Symptom | First evidence | Likely cause / fix |
|
||||
| --- | --- | --- |
|
||||
| No `mcp__*__create_sandbox` tools | tool list, DSH logs | Server not deployed or row not enabled. Use `kubectl`; connect only with approval. |
|
||||
| 404 on connect | curl `/mcp`, `/healthz` | Wrong path/port; MCP endpoint is `/mcp`, health is at the root. |
|
||||
| `/readyz` 503 | response body, server logs | API unreachable or namespace `K8S_SANDBOX_PROBE_NAMESPACE` not listable; fix RBAC/config. |
|
||||
| `ctx.session_id is None` | server error | Connected over stdio; use streamable HTTP. |
|
||||
| `Sandbox claim ... is not found` | server error, `kubectl get claim` | Wrong namespace, or the claim belongs to another MCP session. List with `kubectl`; use the MCP only for its own claims. |
|
||||
| Creation cancelled at ~60 s | MCP call log | DSH `toolCallTimeoutMs` is 60 s while creation waits up to 180 s; raise the timeout. |
|
||||
| `execute_command` connection error | server logs, template ports | No runtime on 8888/8080/9090. The DSH template only listens on 3080; use a runtime template or a combined image. |
|
||||
| Output truncated / flooded | tool result | Pre-v1.0.8 is unbounded; v1.0.8 bounds output. Redirect verbose output to a file and read/download it; never pass `max_output_chars` to an old server. |
|
||||
| Orphaned claims after reconnect | `kubectl get claims -l mcp.k8s-agent-sandbox/session-id` | MCP sessions are transient; wait for `shutdown_after_seconds` or delete by label with explicit confirmation. |
|
||||
| `Forbidden` from the MCP server | server logs | Its ServiceAccount lacks claim/sandbox verbs, or RBAC is in another namespace; fix the Role, not the kubeconfig. |
|
||||
| Sandbox expired | `get_sandbox_status` -> `SandboxNotFound` | `shutdown_after_seconds` elapsed; create a new sandbox and keep the task inside the budget. |
|
||||
|
||||
## Repository checks and CI
|
||||
|
||||
| Symptom | Cause / fix |
|
||||
| --- | --- |
|
||||
| `check-versions: stale version reference in docs: dsh-v...` | A copied release tag in a Markdown file. Refer to the identity in `versions.env`; run `make set-version` only for a real bump. |
|
||||
| `check-versions: hard-coded version outside versions.env` | A tag in `.github/workflows` or the Makefile; move it to `versions.env`. |
|
||||
| `check-examples` layout error | Missing `README.md`/`manifests/`, stray YAML at the example root, missing exposure README, non-executable script, or missing `examples/README.md` entry. |
|
||||
| Manifest dry-run fails in CI | Invalid YAML or CRD schema; parse locally and run `kubectl apply --dry-run=server` against the pinned CRDs. |
|
||||
| `shellcheck` failure | Quote expansions, use arrays for kubectl args, avoid unused variables; run shellcheck locally. |
|
||||
| Lychee link failure | A relative/absolute link does not resolve; the link checker runs offline on the listed docs. |
|
||||
| PR-title check failure | Use Conventional Commits: `feat(scope): ...`, `fix(scope): ...`, `docs(scope): ...`. |
|
||||
| Changelog conflict | Never edit `CHANGELOG.md`; it is generated from commits. |
|
||||
|
||||
## Do not
|
||||
|
||||
- Do not use `--force --grace-period=0` to clear a stuck resource before
|
||||
understanding why it is stuck.
|
||||
- Do not delete namespaces, PVCs, PVs, CRDs, or cluster-scoped objects as a
|
||||
cleanup shortcut.
|
||||
- Do not read Secret data, kubeconfigs, or node logs to "debug" a permission
|
||||
problem, and do not paste pod logs containing the DSH launch token into a
|
||||
report.
|
||||
- Do not expose the unauthenticated MCP server or enable `AUTH_PROXY_HEADER`
|
||||
while another path to the gateway exists.
|
||||
- Do not retry a `Forbidden` with a stronger kubeconfig; fix the RBAC scope.
|
||||
298
.dsh/skills/dsh-agent-sandbox-platform/scripts/platform-preflight.sh
Executable file
298
.dsh/skills/dsh-agent-sandbox-platform/scripts/platform-preflight.sh
Executable file
@@ -0,0 +1,298 @@
|
||||
#!/usr/bin/env bash
|
||||
# Read-only preflight for the DSH agents-as-a-service platform.
|
||||
#
|
||||
# Checks the kubectl target, agent-sandbox CRDs/version, warm pools, claims,
|
||||
# sandboxes, gateway and (optionally) an agent-sandbox MCP server. It never
|
||||
# mutates the cluster, never reads Secret data, and never port-forwards.
|
||||
#
|
||||
# Usage:
|
||||
# platform-preflight.sh [--context NAME] [--namespace NS] [--pool NAME]
|
||||
# [--template NAME] [--mcp-url URL] [--help]
|
||||
#
|
||||
# Exit codes: 0 = no failures, 1 = at least one failure, 2 = usage error.
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
||||
REPO_ROOT="$(git -C "$SCRIPT_DIR" rev-parse --show-toplevel 2>/dev/null || true)"
|
||||
|
||||
CTX=""
|
||||
NS="${NAMESPACE:-dsh-demo}"
|
||||
POOL=""
|
||||
TEMPLATE=""
|
||||
MCP_URL=""
|
||||
|
||||
PASS=0
|
||||
WARN=0
|
||||
FAIL=0
|
||||
|
||||
usage() {
|
||||
cat <<'EOF'
|
||||
Read-only preflight for the DSH agents-as-a-service platform.
|
||||
|
||||
Options:
|
||||
--context NAME kubectl context (default: current context)
|
||||
--namespace NS namespace to inspect (default: $NAMESPACE or dsh-demo)
|
||||
--pool NAME check only this warm pool
|
||||
--template NAME check only this SandboxTemplate
|
||||
--mcp-url URL also check an MCP server's /healthz and /readyz
|
||||
--help show this help
|
||||
|
||||
The script makes no changes and never reads Secret data.
|
||||
EOF
|
||||
}
|
||||
|
||||
while [ "$#" -gt 0 ]; do
|
||||
case "$1" in
|
||||
--context) CTX="${2:-}"; shift 2 ;;
|
||||
--namespace) NS="${2:-}"; shift 2 ;;
|
||||
--pool) POOL="${2:-}"; shift 2 ;;
|
||||
--template) TEMPLATE="${2:-}"; shift 2 ;;
|
||||
--mcp-url) MCP_URL="${2:-}"; shift 2 ;;
|
||||
--help|-h) usage; exit 0 ;;
|
||||
*) echo "unknown argument: $1" >&2; usage >&2; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
if [ -z "$NS" ]; then
|
||||
echo "namespace must not be empty" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
section() { printf '\n== %s ==\n' "$1"; }
|
||||
pass() { PASS=$((PASS + 1)); printf 'PASS %s\n' "$1"; }
|
||||
warn() { WARN=$((WARN + 1)); printf 'WARN %s\n' "$1"; }
|
||||
fail() { FAIL=$((FAIL + 1)); printf 'FAIL %s\n' "$1"; }
|
||||
info() { printf 'INFO %s\n' "$1"; }
|
||||
|
||||
command -v kubectl >/dev/null 2>&1 || {
|
||||
echo "FAIL kubectl is not installed" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
if [ -n "$CTX" ]; then
|
||||
if ! kubectl config get-contexts -o name 2>/dev/null | grep -Fxq "$CTX"; then
|
||||
echo "FAIL context '$CTX' is not in the kubeconfig" >&2
|
||||
exit 1
|
||||
fi
|
||||
KUBECTL=(kubectl --context "$CTX" --request-timeout=15s)
|
||||
else
|
||||
KUBECTL=(kubectl --request-timeout=15s)
|
||||
fi
|
||||
|
||||
k() { "${KUBECTL[@]}" "$@"; }
|
||||
|
||||
section "Target"
|
||||
if current="$(k config current-context 2>/dev/null)"; then
|
||||
info "context: $current"
|
||||
pass "kubeconfig is readable"
|
||||
else
|
||||
fail "cannot read the current kubectl context"
|
||||
fi
|
||||
if user="$(k auth whoami -o jsonpath='{.status.userInfo.username}' 2>/dev/null)" && [ -n "$user" ]; then
|
||||
info "identity: $user"
|
||||
else
|
||||
fallback_user="$(k config view --minify -o jsonpath='{.users[0].name}' 2>/dev/null || true)"
|
||||
info "identity: ${fallback_user:-unknown}"
|
||||
fi
|
||||
if k get namespace "$NS" -o name >/dev/null 2>&1; then
|
||||
pass "namespace $NS exists"
|
||||
else
|
||||
fail "namespace $NS is missing or not readable"
|
||||
fi
|
||||
|
||||
if [ -n "$REPO_ROOT" ] && [ -f "$REPO_ROOT/versions.env" ]; then
|
||||
# shellcheck source=/dev/null
|
||||
. "$REPO_ROOT/versions.env"
|
||||
info "expected agent-sandbox pin: ${AGENT_SANDBOX_VERSION:-unknown}"
|
||||
else
|
||||
warn "versions.env not found; skipping the pin comparison"
|
||||
fi
|
||||
|
||||
section "Capabilities (informational; never widen them)"
|
||||
for check in "create sandboxclaims" "delete sandboxclaims" "create pods/exec"; do
|
||||
verb="${check%% *}"
|
||||
resource="${check#* }"
|
||||
if [ "$(k auth can-i "$verb" "$resource" -n "$NS" 2>/dev/null || echo unknown)" = "yes" ]; then
|
||||
info "can $verb $resource in $NS: yes"
|
||||
else
|
||||
info "can $verb $resource in $NS: no"
|
||||
fi
|
||||
done
|
||||
if [ "$(k auth can-i get secrets -n "$NS" 2>/dev/null || echo unknown)" = "yes" ]; then
|
||||
warn "RBAC allows reading Secrets; policy still forbids printing Secret data"
|
||||
else
|
||||
info "can get secrets in $NS: no"
|
||||
fi
|
||||
|
||||
section "Agent Sandbox CRDs"
|
||||
for crd in \
|
||||
sandboxtemplates.extensions.agents.x-k8s.io \
|
||||
sandboxwarmpools.extensions.agents.x-k8s.io \
|
||||
sandboxclaims.extensions.agents.x-k8s.io \
|
||||
sandboxes.agents.x-k8s.io; do
|
||||
if k get crd "$crd" -o name >/dev/null 2>&1; then
|
||||
pass "CRD $crd"
|
||||
else
|
||||
fail "CRD $crd is missing or not readable"
|
||||
fi
|
||||
done
|
||||
|
||||
section "Controller"
|
||||
controller_ns="agent-sandbox-system"
|
||||
controller_image="$(k -n "$controller_ns" get deployment agent-sandbox-controller \
|
||||
-o jsonpath='{.spec.template.spec.containers[0].image}' 2>/dev/null || true)"
|
||||
if [ -n "$controller_image" ]; then
|
||||
info "controller image: $controller_image"
|
||||
controller_tag="${controller_image##*:}"
|
||||
if [ -n "${AGENT_SANDBOX_VERSION:-}" ] && [ "$controller_tag" != "$AGENT_SANDBOX_VERSION" ]; then
|
||||
warn "controller tag '$controller_tag' differs from AGENT_SANDBOX_VERSION '$AGENT_SANDBOX_VERSION'"
|
||||
else
|
||||
pass "controller version matches versions.env"
|
||||
fi
|
||||
else
|
||||
warn "controller deployment not found in $controller_ns; cannot verify the installed version"
|
||||
fi
|
||||
|
||||
section "Templates"
|
||||
templates="$(k -n "$NS" get sandboxtemplates \
|
||||
-o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' 2>/dev/null || true)"
|
||||
if [ -z "$templates" ]; then
|
||||
warn "no SandboxTemplates in $NS"
|
||||
fi
|
||||
if [ -n "$TEMPLATE" ] && ! printf '%s\n' "$templates" | grep -Fxq "$TEMPLATE"; then
|
||||
fail "SandboxTemplate $TEMPLATE not found in $NS"
|
||||
fi
|
||||
for tpl in $templates; do
|
||||
[ -n "$TEMPLATE" ] && [ "$tpl" != "$TEMPLATE" ] && continue
|
||||
desc="$(k -n "$NS" get sandboxtemplate "$tpl" \
|
||||
-o jsonpath='{range .spec.podTemplate.spec.containers[*]}{.name}{":"}{range .ports[*]}{.containerPort}{","}{end}{";"}{end}' \
|
||||
2>/dev/null || true)"
|
||||
info "template $tpl containers: ${desc:-<none>}"
|
||||
case "$desc" in
|
||||
*sandboxd*|*:8080,*|*:9090,*)
|
||||
pass "template $tpl exposes a sandboxd runtime (MCP sandboxd-in-cluster)" ;;
|
||||
*:8888,*)
|
||||
pass "template $tpl exposes a runtime HTTP port (MCP gateway/direct/in-cluster)" ;;
|
||||
*:3080,*)
|
||||
warn "template $tpl is DSH Web only (:3080); MCP execute_command cannot run against it" ;;
|
||||
*)
|
||||
warn "template $tpl exposes no known sandbox runtime port" ;;
|
||||
esac
|
||||
done
|
||||
|
||||
section "Warm pools"
|
||||
pools="$(k -n "$NS" get sandboxwarmpools \
|
||||
-o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' 2>/dev/null || true)"
|
||||
if [ -z "$pools" ]; then
|
||||
warn "no SandboxWarmPools in $NS; claims will provision cold"
|
||||
fi
|
||||
if [ -n "$POOL" ] && ! printf '%s\n' "$pools" | grep -Fxq "$POOL"; then
|
||||
fail "SandboxWarmPool $POOL not found in $NS"
|
||||
fi
|
||||
for pool in $pools; do
|
||||
[ -n "$POOL" ] && [ "$pool" != "$POOL" ] && continue
|
||||
replicas="$(k -n "$NS" get sandboxwarmpool "$pool" -o jsonpath='{.spec.replicas}' 2>/dev/null || true)"
|
||||
selector="$(k -n "$NS" get sandboxwarmpool "$pool" -o jsonpath='{.status.selector}' 2>/dev/null || true)"
|
||||
if [ -n "$selector" ]; then
|
||||
ready="$(k -n "$NS" get sandboxes -l "$selector" \
|
||||
-o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' \
|
||||
2>/dev/null | grep -c '^True$' || true)"
|
||||
else
|
||||
ready="$(k -n "$NS" get sandboxes -l agents.x-k8s.io/warm-pool-sandbox \
|
||||
-o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' \
|
||||
2>/dev/null | grep -c '^True$' || true)"
|
||||
fi
|
||||
replicas="${replicas:-0}"
|
||||
info "pool $pool: $ready/$replicas warm spares Ready"
|
||||
if [ "$replicas" -eq 0 ]; then
|
||||
warn "pool $pool has spec.replicas=0; signup will cold-provision"
|
||||
elif [ "$ready" -lt "$replicas" ]; then
|
||||
warn "pool $pool is below its desired spare count"
|
||||
else
|
||||
pass "pool $pool is at capacity"
|
||||
fi
|
||||
done
|
||||
|
||||
section "Claims"
|
||||
claims="$(k -n "$NS" get sandboxclaims \
|
||||
-o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.conditions[?(@.type=="Ready")].status}{"\t"}{.status.conditions[?(@.type=="Ready")].reason}{"\t"}{.status.sandbox.name}{"\n"}{end}' \
|
||||
2>/dev/null || true)"
|
||||
if [ -z "$claims" ]; then
|
||||
warn "no SandboxClaims in $NS"
|
||||
else
|
||||
while IFS=$'\t' read -r name ready reason sandbox; do
|
||||
[ -n "$name" ] || continue
|
||||
if [ "$ready" != "True" ]; then
|
||||
warn "claim $name is not Ready (status='${ready:-unknown}', reason='${reason:-unknown}')"
|
||||
elif [ -z "$sandbox" ]; then
|
||||
warn "claim $name is Ready but has no bound sandbox"
|
||||
else
|
||||
pass "claim $name -> sandbox $sandbox"
|
||||
fi
|
||||
done <<< "$claims"
|
||||
fi
|
||||
|
||||
section "Sandboxes"
|
||||
sandboxes="$(k -n "$NS" get sandboxes \
|
||||
-o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.operatingMode}{"\t"}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' \
|
||||
2>/dev/null || true)"
|
||||
if [ -z "$sandboxes" ]; then
|
||||
warn "no Sandboxes in $NS"
|
||||
else
|
||||
while IFS=$'\t' read -r name mode ready; do
|
||||
[ -n "$name" ] || continue
|
||||
info "sandbox $name mode=${mode:-Running} ready=${ready:-unknown}"
|
||||
done <<< "$sandboxes"
|
||||
fi
|
||||
|
||||
section "Storage and Services"
|
||||
pvc_count="$(k -n "$NS" get pvc -o name 2>/dev/null | wc -l | tr -d ' ')"
|
||||
svc_count="$(k -n "$NS" get service -o name 2>/dev/null | wc -l | tr -d ' ')"
|
||||
info "persistent volume claims: ${pvc_count:-0}"
|
||||
info "services: ${svc_count:-0}"
|
||||
[ "${pvc_count:-0}" -gt 0 ] || warn "no PVCs in $NS"
|
||||
[ "${svc_count:-0}" -gt 0 ] || warn "no Services in $NS"
|
||||
|
||||
section "Gateway"
|
||||
if k -n "$NS" get deployment dsh-gateway -o name >/dev/null 2>&1; then
|
||||
pass "gateway deployment dsh-gateway exists"
|
||||
if k -n "$NS" get service dsh-gateway -o name >/dev/null 2>&1; then
|
||||
endpoints="$(k -n "$NS" get endpoints dsh-gateway \
|
||||
-o jsonpath='{range .subsets[*].addresses[*]}{.ip}{" "}{end}' 2>/dev/null || true)"
|
||||
if [ -n "$endpoints" ]; then
|
||||
pass "gateway endpoints: $endpoints"
|
||||
else
|
||||
warn "gateway Service has no ready endpoints"
|
||||
fi
|
||||
else
|
||||
warn "gateway Service dsh-gateway is missing"
|
||||
fi
|
||||
else
|
||||
warn "no dsh-gateway deployment; this is a resource-layer-only install"
|
||||
fi
|
||||
|
||||
section "MCP server"
|
||||
if [ -n "$MCP_URL" ]; then
|
||||
if ! command -v curl >/dev/null 2>&1; then
|
||||
fail "curl is required for --mcp-url"
|
||||
else
|
||||
base="${MCP_URL%/}"
|
||||
base="${base%/mcp}"
|
||||
for endpoint in healthz readyz; do
|
||||
body="$(curl -fsS --max-time 5 "$base/$endpoint" 2>/dev/null | head -c 300 || true)"
|
||||
if [ -n "$body" ]; then
|
||||
pass "MCP /$endpoint: $body"
|
||||
else
|
||||
fail "MCP /$endpoint is unreachable at $base/$endpoint"
|
||||
fi
|
||||
done
|
||||
fi
|
||||
else
|
||||
info "no --mcp-url supplied; skipping server health checks"
|
||||
fi
|
||||
|
||||
printf '\n== Summary ==\nPASS=%d WARN=%d FAIL=%d\n' "$PASS" "$WARN" "$FAIL"
|
||||
if [ "$FAIL" -gt 0 ]; then
|
||||
exit 1
|
||||
fi
|
||||
exit 0
|
||||
5
.entire/.gitignore
vendored
Normal file
5
.entire/.gitignore
vendored
Normal file
@@ -0,0 +1,5 @@
|
||||
tmp/
|
||||
settings.local.json
|
||||
metadata/
|
||||
logs/
|
||||
redactors/local/
|
||||
9
.entire/settings.json
Normal file
9
.entire/settings.json
Normal file
@@ -0,0 +1,9 @@
|
||||
{
|
||||
"enabled": true,
|
||||
"telemetry": true,
|
||||
"checkpoints": {
|
||||
"primary": {
|
||||
"type": "git-refs"
|
||||
}
|
||||
}
|
||||
}
|
||||
49
.env.sample
49
.env.sample
@@ -1,12 +1,32 @@
|
||||
# Copy to .env and fill in. Never commit a real key.
|
||||
# Reference environment names for local runs. Nothing sources this file
|
||||
# automatically: export the values or pass them with `docker run -e/--env-file`.
|
||||
# Copy it to .env if you prefer --env-file. Never commit a real key.
|
||||
#
|
||||
# The version values below are documentation only: versions.env is the single
|
||||
# source of truth, and scripts/check-versions.sh fails if they drift.
|
||||
|
||||
# --- Main DSH Web image (root Dockerfile) ---
|
||||
DEEPSEEK_API_KEY=
|
||||
DSH_VERSION=dsh-v0.2.1-rc.3
|
||||
RELEASE_REVISION=r3
|
||||
# Optional additional provider keys consumed by DSH. For Docker, pass them with
|
||||
# -e; the Kubernetes examples inject them through the optional
|
||||
# dsh-provider-keys-extra Secret.
|
||||
OPENROUTER_API_KEY=
|
||||
GEMINI_API_KEY=
|
||||
NVIDIA_API_KEY=
|
||||
OPENAI_API_KEY=
|
||||
ANTHROPIC_API_KEY=
|
||||
GROQ_API_KEY=
|
||||
XAI_API_KEY=
|
||||
MISTRAL_API_KEY=
|
||||
TOGETHER_API_KEY=
|
||||
CEREBRAS_API_KEY=
|
||||
AI_GATEWAY_API_KEY=
|
||||
HF_TOKEN=
|
||||
DSH_VERSION=dsh-v0.2.1-rc.5
|
||||
# Full upstream commit SHA that DSH_VERSION resolves to; the main image build
|
||||
# fails if the clone's HEAD differs (guards against a moved tag).
|
||||
DSH_COMMIT=b830ec950faba0a46dda564be3e2d854afed94f1
|
||||
RELEASE_REVISION=r6
|
||||
PNPM_VERSION=11.28.5
|
||||
# Container port, and the host port published by "make run".
|
||||
DSH_PORT=3080
|
||||
@@ -14,22 +34,7 @@ DSH_PORT=3080
|
||||
# public hostname). localhost and 127.0.0.1 are always trusted.
|
||||
DSH_TRUSTED_HOSTS=
|
||||
|
||||
# --- agents-as-a-service example ---
|
||||
NAMESPACE=dsh-demo
|
||||
POOL=dsh-pool
|
||||
# Seconds a sandbox may stay idle before the sweeper suspends it, and the max
|
||||
# time a proxied request waits for a suspended sandbox to become Ready.
|
||||
IDLE_TIMEOUT=360
|
||||
WAKE_TIMEOUT=240
|
||||
DSH_CONTAINER=dsh-web
|
||||
DSH_LOG_TAIL_LINES=2000
|
||||
COOKIE_NAME=dsh_gw_session
|
||||
# Set to true only behind HTTPS; secure cookies are not sent over plain HTTP
|
||||
# (localhost counts as a secure context in browsers).
|
||||
COOKIE_SECURE=false
|
||||
COOKIE_MAX_AGE=604800
|
||||
MAX_BODY_MB=32
|
||||
HTTP_TIMEOUT=300
|
||||
# SSO is opt-in. Set this only when the gateway Service is reachable
|
||||
# exclusively through oauth2-proxy (see the example exposure docs).
|
||||
# AUTH_PROXY_HEADER=X-Forwarded-User
|
||||
# The example gateway's runtime settings (NAMESPACE, POOL, IDLE_TIMEOUT,
|
||||
# WAKE_TIMEOUT, COOKIE_*, AUTH_PROXY_HEADER, ...) are documented in
|
||||
# examples/dsh-agents-as-a-service/gateway/README.md and set in
|
||||
# examples/dsh-agents-as-a-service/manifests/50-gateway.yaml.
|
||||
|
||||
2
.github/ISSUE_TEMPLATE/bug_report.yml
vendored
2
.github/ISSUE_TEMPLATE/bug_report.yml
vendored
@@ -28,7 +28,7 @@ body:
|
||||
id: image-tag
|
||||
attributes:
|
||||
label: Image or release
|
||||
description: The pinned tag or digest, for example dsh-v0.2.1-rc.3-r3.
|
||||
description: The tag or digest you deployed (from the manifest or release metadata).
|
||||
- type: input
|
||||
id: cluster
|
||||
attributes:
|
||||
|
||||
3
.github/pull_request_template.md
vendored
3
.github/pull_request_template.md
vendored
@@ -19,8 +19,7 @@
|
||||
## Checklist
|
||||
|
||||
- [ ] make check passes (versions, example layout, manifest YAML)
|
||||
- [ ] make test passes (gateway unit tests)
|
||||
- [ ] make lint passes (ruff, shellcheck; yamllint/actionlint where installed)
|
||||
- [ ] make test / make lint pass when the change touches code (gateway unit tests, ruff, shellcheck; yamllint/actionlint where installed)
|
||||
- [ ] Docs updated for the change (root README, docs/, example README)
|
||||
- [ ] No secrets or usable credentials committed
|
||||
- [ ] CHANGELOG.md was not edited by hand (generated by git-cliff)
|
||||
|
||||
30
.github/workflows/_build-image.yaml
vendored
30
.github/workflows/_build-image.yaml
vendored
@@ -53,6 +53,8 @@ jobs:
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Prepare environment
|
||||
env:
|
||||
@@ -60,14 +62,23 @@ jobs:
|
||||
run: |
|
||||
set -euo pipefail
|
||||
pair=$(printf '%s' "${{ matrix.platform }}" | tr '/' '-')
|
||||
echo "PLATFORM_PAIR=$pair" >> "$GITHUB_ENV"
|
||||
echo "CACHE_SCOPE=$(printf '%s' "$IMAGE" | tr '/' '-')" >> "$GITHUB_ENV"
|
||||
# This reusable workflow runs once for the main image and once for the
|
||||
# gateway in the same release run. Artifact names must include a
|
||||
# unique token for the image: the repository path is a prefix of the
|
||||
# gateway path, so a path-derived pattern would match both. Use a
|
||||
# short hash of the full image name.
|
||||
image_hash=$(printf '%s' "$IMAGE" | sha256sum | cut -c1-12)
|
||||
{
|
||||
echo "PLATFORM_PAIR=$pair"
|
||||
echo "CACHE_SCOPE=$(printf '%s' "$IMAGE" | tr '/' '-')"
|
||||
echo "ARTIFACT_PREFIX=$image_hash"
|
||||
} >> "$GITHUB_ENV"
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
|
||||
|
||||
- name: Login to GitHub Container Registry
|
||||
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
|
||||
uses: docker/login-action@81c6f93d4bc991e510b342a00b419b3bb4efc5c6 # v4.7.0
|
||||
with:
|
||||
registry: ghcr.io
|
||||
username: ${{ github.actor }}
|
||||
@@ -101,7 +112,7 @@ jobs:
|
||||
- name: Upload digest
|
||||
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
|
||||
with:
|
||||
name: digests-${{ env.PLATFORM_PAIR }}
|
||||
name: digests-${{ env.ARTIFACT_PREFIX }}-${{ env.PLATFORM_PAIR }}
|
||||
path: /tmp/digests/*
|
||||
if-no-files-found: error
|
||||
retention-days: 1
|
||||
@@ -112,18 +123,25 @@ jobs:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
steps:
|
||||
- name: Prepare environment
|
||||
env:
|
||||
IMAGE: ${{ inputs.image }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
echo "ARTIFACT_PREFIX=$(printf '%s' "$IMAGE" | sha256sum | cut -c1-12)" >> "$GITHUB_ENV"
|
||||
|
||||
- name: Download digests
|
||||
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
|
||||
with:
|
||||
path: /tmp/digests
|
||||
pattern: digests-*
|
||||
pattern: digests-${{ env.ARTIFACT_PREFIX }}-*
|
||||
merge-multiple: true
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
|
||||
|
||||
- name: Login to GitHub Container Registry
|
||||
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
|
||||
uses: docker/login-action@81c6f93d4bc991e510b342a00b419b3bb4efc5c6 # v4.7.0
|
||||
with:
|
||||
registry: ghcr.io
|
||||
username: ${{ github.actor }}
|
||||
|
||||
4
.github/workflows/build-gateway.yaml
vendored
4
.github/workflows/build-gateway.yaml
vendored
@@ -12,11 +12,9 @@ on:
|
||||
- .github/workflows/build-gateway.yaml
|
||||
- .github/workflows/_build-image.yaml
|
||||
|
||||
# Write scopes are granted to the build job only; every other job stays read-only.
|
||||
permissions:
|
||||
contents: read
|
||||
packages: write
|
||||
id-token: write
|
||||
attestations: write
|
||||
|
||||
concurrency:
|
||||
group: build-gateway-${{ github.ref }}
|
||||
|
||||
7
.github/workflows/build-main.yaml
vendored
7
.github/workflows/build-main.yaml
vendored
@@ -13,11 +13,9 @@ on:
|
||||
- .github/workflows/build-main.yaml
|
||||
- .github/workflows/_build-image.yaml
|
||||
|
||||
# Write scopes are granted to the build job only; every other job stays read-only.
|
||||
permissions:
|
||||
contents: read
|
||||
packages: write
|
||||
id-token: write
|
||||
attestations: write
|
||||
|
||||
concurrency:
|
||||
group: build-main-${{ github.ref }}
|
||||
@@ -31,6 +29,7 @@ jobs:
|
||||
outputs:
|
||||
identity: ${{ steps.meta.outputs.identity }}
|
||||
dsh_version: ${{ steps.meta.outputs.dsh_version }}
|
||||
dsh_commit: ${{ steps.meta.outputs.dsh_commit }}
|
||||
pnpm_version: ${{ steps.meta.outputs.pnpm_version }}
|
||||
release_revision: ${{ steps.meta.outputs.release_revision }}
|
||||
ghcr_image: ${{ steps.meta.outputs.ghcr_image }}
|
||||
@@ -55,6 +54,7 @@ jobs:
|
||||
{
|
||||
echo "identity=$identity"
|
||||
echo "dsh_version=$DSH_VERSION"
|
||||
echo "dsh_commit=$DSH_COMMIT"
|
||||
echo "pnpm_version=$PNPM_VERSION"
|
||||
echo "release_revision=$RELEASE_REVISION"
|
||||
echo "ghcr_image=$GHCR_IMAGE"
|
||||
@@ -77,6 +77,7 @@ jobs:
|
||||
dockerfile: Dockerfile
|
||||
build-args: |
|
||||
DSH_VERSION=${{ needs.meta.outputs.dsh_version }}
|
||||
DSH_COMMIT=${{ needs.meta.outputs.dsh_commit }}
|
||||
RELEASE_REVISION=${{ needs.meta.outputs.release_revision }}
|
||||
PNPM_VERSION=${{ needs.meta.outputs.pnpm_version }}
|
||||
VCS_REF=${{ github.sha }}
|
||||
|
||||
128
.github/workflows/ci-image.yaml
vendored
Normal file
128
.github/workflows/ci-image.yaml
vendored
Normal file
@@ -0,0 +1,128 @@
|
||||
# Build-only check for the main DSH Web image: linux/amd64, no registry login,
|
||||
# no push. It proves the Dockerfile still builds from the pinned versions.env and
|
||||
# that the resulting image can actually run DSH Web. The published multi-arch
|
||||
# build stays in build-main.yaml (merge to main) and release.yaml (tags).
|
||||
#
|
||||
# Like ci-python.yaml and ci-manifests.yaml this is path-filtered, so it is an
|
||||
# on-demand check rather than a required status check.
|
||||
name: CI image
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
paths:
|
||||
- Dockerfile
|
||||
- .dockerignore
|
||||
- versions.env
|
||||
- .github/workflows/ci-image.yaml
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
concurrency:
|
||||
group: ci-image-${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
build-amd64:
|
||||
name: build-amd64
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 120
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Read versions.env
|
||||
id: meta
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
set -a
|
||||
# shellcheck disable=SC1091
|
||||
source versions.env
|
||||
set +a
|
||||
identity="$DSH_VERSION"
|
||||
[ -z "${RELEASE_REVISION:-}" ] || identity="$DSH_VERSION-$RELEASE_REVISION"
|
||||
{
|
||||
echo "identity=$identity"
|
||||
echo "dsh_version=$DSH_VERSION"
|
||||
echo "dsh_commit=$DSH_COMMIT"
|
||||
echo "pnpm_version=$PNPM_VERSION"
|
||||
echo "release_revision=${RELEASE_REVISION:-}"
|
||||
} >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
|
||||
|
||||
- name: Build linux/amd64 (load locally, never push)
|
||||
uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7.4.0
|
||||
with:
|
||||
context: .
|
||||
file: Dockerfile
|
||||
platforms: linux/amd64
|
||||
push: false
|
||||
load: true
|
||||
tags: dsh-web:ci
|
||||
build-args: |
|
||||
DSH_VERSION=${{ steps.meta.outputs.dsh_version }}
|
||||
DSH_COMMIT=${{ steps.meta.outputs.dsh_commit }}
|
||||
RELEASE_REVISION=${{ steps.meta.outputs.release_revision }}
|
||||
PNPM_VERSION=${{ steps.meta.outputs.pnpm_version }}
|
||||
VCS_REF=${{ github.sha }}
|
||||
labels: |
|
||||
org.opencontainers.image.source=https://github.com/${{ github.repository }}
|
||||
org.opencontainers.image.version=${{ steps.meta.outputs.identity }}
|
||||
org.opencontainers.image.revision=${{ github.sha }}
|
||||
cache-from: type=gha,scope=ci-image-linux-amd64
|
||||
cache-to: type=gha,mode=max,scope=ci-image-linux-amd64
|
||||
|
||||
- name: Smoke test the image
|
||||
env:
|
||||
IDENTITY: ${{ steps.meta.outputs.identity }}
|
||||
DSH_VERSION: ${{ steps.meta.outputs.dsh_version }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
image=dsh-web:ci
|
||||
|
||||
# The packaged CLI runs and reports the version pinned in versions.env.
|
||||
version="$(docker run --rm --entrypoint node "$image" /app/lib/bin.js --version)"
|
||||
printf 'DSH CLI: %s\n' "$version"
|
||||
grep -q "${DSH_VERSION#dsh-v}" <<<"$version"
|
||||
|
||||
# The container runs as the non-root node user and its writable
|
||||
# directories exist and are owned by that user.
|
||||
[ "$(docker run --rm --entrypoint id "$image" -u)" = "1000" ]
|
||||
docker run --rm --entrypoint sh "$image" -c \
|
||||
'test -d /workspace && test -d "$DSH_HOME" && [ "$(stat -c %u /workspace)" = 1000 ]'
|
||||
|
||||
# The command-sandbox runner this image is expected to ship.
|
||||
docker run --rm --entrypoint bwrap "$image" --version
|
||||
|
||||
# Labels come from the build args rather than a hard-coded value.
|
||||
label="$(docker inspect -f '{{ index .Config.Labels "org.opencontainers.image.version" }}' "$image")"
|
||||
printf 'image version label: %s\n' "$label"
|
||||
[ "$label" = "$IDENTITY" ]
|
||||
|
||||
- name: Serve DSH Web
|
||||
run: |
|
||||
set -euo pipefail
|
||||
docker run -d --name dsh-ci -p 18080:3080 \
|
||||
-e DSH_TRUSTED_HOSTS=127.0.0.1:18080 dsh-web:ci >/dev/null
|
||||
trap 'docker rm -f dsh-ci >/dev/null 2>&1 || true' EXIT
|
||||
|
||||
# /favicon.svg is unauthenticated and served by the web layer, which is
|
||||
# why the shipped probes use it too.
|
||||
for second in $(seq 1 60); do
|
||||
if curl -fsS -o /dev/null http://127.0.0.1:18080/favicon.svg; then
|
||||
printf 'DSH Web served /favicon.svg after ~%ss\n' "$((second * 2))"
|
||||
exit 0
|
||||
fi
|
||||
sleep 2
|
||||
done
|
||||
|
||||
echo "::error::DSH Web did not serve /favicon.svg within 120s"
|
||||
# Pod logs can hold the DSH launch token: never dump them verbatim.
|
||||
docker logs dsh-ci 2>&1 | grep -v 'token=' | tail -50 || true
|
||||
exit 1
|
||||
88
.github/workflows/ci-manifests.yaml
vendored
Normal file
88
.github/workflows/ci-manifests.yaml
vendored
Normal file
@@ -0,0 +1,88 @@
|
||||
# Manifest shape validation and server-side dry-run. Split from ci.yaml so this
|
||||
# job can use native on.pull_request.paths filtering and skip PRs that do not
|
||||
# touch manifests, scripts or versions.env. The job name is `manifests`; because
|
||||
# it is path-filtered it is an on-demand check, not a required branch-protection
|
||||
# check (see docs/gitflow.md).
|
||||
name: CI (manifests)
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
paths:
|
||||
- examples/**/manifests/**
|
||||
- scripts/check-manifests.py
|
||||
- versions.env
|
||||
- .github/workflows/ci-manifests.yaml
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
concurrency:
|
||||
group: ci-manifests-${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
manifests:
|
||||
name: manifests
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 25
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
||||
with:
|
||||
persist-credentials: false
|
||||
- name: Create kind cluster
|
||||
uses: helm/kind-action@1544676c07bb3570fcb70166beaa07c4528fd4aa # v1.15.1
|
||||
with:
|
||||
version: v0.33.0
|
||||
cluster_name: dsh-ci
|
||||
- name: Read versions.env
|
||||
run: |
|
||||
set -euo pipefail
|
||||
while IFS= read -r line; do
|
||||
case "$line" in ''|'#'*) continue ;; esac
|
||||
echo "$line" >> "$GITHUB_ENV"
|
||||
done < versions.env
|
||||
- name: Install agent-sandbox and Gateway API CRDs
|
||||
run: |
|
||||
set -euo pipefail
|
||||
# Download, verify the pinned digest from versions.env, then apply.
|
||||
curl -fsSL --proto '=https' -o /tmp/agent-sandbox.yaml \
|
||||
"https://github.com/kubernetes-sigs/agent-sandbox/releases/download/$AGENT_SANDBOX_VERSION/sandbox-with-extensions.yaml"
|
||||
printf '%s %s\n' "$AGENT_SANDBOX_MANIFEST_SHA256" /tmp/agent-sandbox.yaml | sha256sum -c -
|
||||
kubectl apply -f /tmp/agent-sandbox.yaml
|
||||
kubectl wait --for=condition=Established crd/sandboxtemplates.extensions.agents.x-k8s.io --timeout=180s
|
||||
kubectl wait --for=condition=Established crd/sandboxwarmpools.extensions.agents.x-k8s.io --timeout=180s
|
||||
kubectl wait --for=condition=Established crd/sandboxclaims.extensions.agents.x-k8s.io --timeout=180s
|
||||
kubectl wait --for=condition=Established crd/sandboxes.agents.x-k8s.io --timeout=180s
|
||||
curl -fsSL --proto '=https' -o /tmp/gateway-api.yaml \
|
||||
"https://github.com/kubernetes-sigs/gateway-api/releases/download/$GATEWAY_API_VERSION/standard-install.yaml"
|
||||
printf '%s %s\n' "$GATEWAY_API_MANIFEST_SHA256" /tmp/gateway-api.yaml | sha256sum -c -
|
||||
kubectl apply -f /tmp/gateway-api.yaml
|
||||
kubectl wait --for=condition=Established crd/gateways.gateway.networking.k8s.io --timeout=180s
|
||||
kubectl wait --for=condition=Established crd/httproutes.gateway.networking.k8s.io --timeout=180s
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
|
||||
with:
|
||||
python-version: "3.12"
|
||||
- name: Install check dependencies
|
||||
run: pip install "pyyaml==6.0.3"
|
||||
- name: Validate manifest shape
|
||||
run: python scripts/check-manifests.py
|
||||
- name: Validate manifests (server dry-run)
|
||||
run: |
|
||||
set -euo pipefail
|
||||
kubectl create namespace dsh-demo
|
||||
shopt -s globstar nullglob
|
||||
count=0
|
||||
for file in examples/*/manifests/**/*.yaml; do
|
||||
case "$file" in *.yaml.example) continue ;; esac
|
||||
echo "== $file =="
|
||||
kubectl apply --dry-run=server -f "$file"
|
||||
count=$((count + 1))
|
||||
done
|
||||
if [ "$count" -eq 0 ]; then
|
||||
echo "no manifests were validated" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "validated $count manifests"
|
||||
75
.github/workflows/ci-python.yaml
vendored
Normal file
75
.github/workflows/ci-python.yaml
vendored
Normal file
@@ -0,0 +1,75 @@
|
||||
# Gateway unit tests and lint. Split from ci.yaml so this job can use native
|
||||
# on.pull_request.paths filtering and skip PRs that do not touch the gateway.
|
||||
# The job name is `python`; because it is path-filtered it is an on-demand check,
|
||||
# not a required branch-protection check (see docs/gitflow.md).
|
||||
name: CI (python)
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
paths:
|
||||
- examples/dsh-agents-as-a-service/gateway/**
|
||||
- .github/workflows/ci-python.yaml
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
concurrency:
|
||||
group: ci-python-${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
python:
|
||||
name: python
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 20
|
||||
defaults:
|
||||
run:
|
||||
working-directory: examples/dsh-agents-as-a-service/gateway
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
||||
with:
|
||||
persist-credentials: false
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
|
||||
with:
|
||||
python-version: "3.12"
|
||||
cache: pip
|
||||
cache-dependency-path: |
|
||||
examples/dsh-agents-as-a-service/gateway/requirements.txt
|
||||
examples/dsh-agents-as-a-service/gateway/requirements-dev.txt
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
set -euo pipefail
|
||||
python -m pip install --upgrade pip
|
||||
# Both files are fully hash-pinned (uv pip compile --generate-hashes);
|
||||
# pip refuses to install anything not listed with a matching hash.
|
||||
pip install --require-hashes -r requirements.txt -r requirements-dev.txt
|
||||
- name: Lint
|
||||
run: |
|
||||
set -euo pipefail
|
||||
python -m ruff check .
|
||||
python -m ruff format --check .
|
||||
- name: Unit tests with coverage
|
||||
run: >-
|
||||
python -m pytest -q
|
||||
--cov=. --cov-report=xml:coverage.xml --cov-report=term
|
||||
--junitxml=junit.xml
|
||||
- name: Coverage summary
|
||||
run: |
|
||||
set -euo pipefail
|
||||
{
|
||||
echo "### Gateway test coverage"
|
||||
python -m coverage report
|
||||
} >> "$GITHUB_STEP_SUMMARY"
|
||||
- name: Upload test artefacts
|
||||
if: always()
|
||||
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
|
||||
with:
|
||||
name: gateway-test-results
|
||||
path: |
|
||||
examples/dsh-agents-as-a-service/gateway/junit.xml
|
||||
examples/dsh-agents-as-a-service/gateway/coverage.xml
|
||||
if-no-files-found: warn
|
||||
retention-days: 7
|
||||
189
.github/workflows/ci.yaml
vendored
189
.github/workflows/ci.yaml
vendored
@@ -1,12 +1,11 @@
|
||||
# Continuous integration: lint, unit tests, manifest dry-run, gateway image
|
||||
# smoke test and dependency review. Job names are the required branch-protection
|
||||
# checks documented in docs/gitflow.md.
|
||||
# Continuous integration: lint and dependency review. Job names are the required
|
||||
# branch-protection checks documented in docs/gitflow.md. The gateway unit tests
|
||||
# and the manifest dry-run live in ci-python.yaml / ci-manifests.yaml so each can
|
||||
# use native on.pull_request.paths filtering and skip unrelated changes.
|
||||
name: CI
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
push:
|
||||
branches: [main]
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
@@ -38,19 +37,29 @@ jobs:
|
||||
- name: Install actionlint
|
||||
env:
|
||||
ACTIONLINT_VERSION: v1.7.12
|
||||
# SHA-256 of actionlint_1.7.12_linux_amd64.tar.gz, published in the
|
||||
# release's actionlint_1.7.12_checksums.txt. Update it with the version.
|
||||
ACTIONLINT_SHA256: 8aca8db96f1b94770f1b0d72b6dddcb1ebb8123cb3712530b08cc387b349a3d8
|
||||
run: |
|
||||
set -euo pipefail
|
||||
version="${ACTIONLINT_VERSION#v}"
|
||||
curl -fsSL -o /tmp/actionlint.tar.gz "https://github.com/rhysd/actionlint/releases/download/$ACTIONLINT_VERSION/actionlint_${version}_linux_amd64.tar.gz"
|
||||
tar -xzf /tmp/actionlint.tar.gz -C /tmp actionlint
|
||||
asset="actionlint_${version}_linux_amd64.tar.gz"
|
||||
curl -fsSL --proto '=https' -o "/tmp/$asset" \
|
||||
"https://github.com/rhysd/actionlint/releases/download/$ACTIONLINT_VERSION/$asset"
|
||||
printf '%s %s\n' "$ACTIONLINT_SHA256" "/tmp/$asset" | sha256sum -c -
|
||||
tar -xzf "/tmp/$asset" -C /tmp actionlint
|
||||
install -m 0755 /tmp/actionlint /usr/local/bin/actionlint
|
||||
actionlint -version
|
||||
- name: Install hadolint
|
||||
env:
|
||||
HADOLINT_VERSION: v2.12.0
|
||||
# SHA-256 of hadolint-Linux-x86_64, published next to the download.
|
||||
HADOLINT_SHA256: 56de6d5e5ec427e17b74fa48d51271c7fc0d61244bf5c90e828aab8362d55010
|
||||
run: |
|
||||
set -euo pipefail
|
||||
curl -fsSL -o /tmp/hadolint "https://github.com/hadolint/hadolint/releases/download/$HADOLINT_VERSION/hadolint-Linux-x86_64"
|
||||
curl -fsSL --proto '=https' -o /tmp/hadolint \
|
||||
"https://github.com/hadolint/hadolint/releases/download/$HADOLINT_VERSION/hadolint-Linux-x86_64"
|
||||
printf '%s %s\n' "$HADOLINT_SHA256" /tmp/hadolint | sha256sum -c -
|
||||
install -m 0755 /tmp/hadolint /usr/local/bin/hadolint
|
||||
hadolint --version
|
||||
- name: Repository checks
|
||||
@@ -58,7 +67,6 @@ jobs:
|
||||
set -euo pipefail
|
||||
./scripts/check-versions.sh
|
||||
./scripts/check-examples.sh
|
||||
python scripts/check-manifests.py
|
||||
- name: Shellcheck
|
||||
run: shellcheck -x build.sh scripts/*.sh examples/*/scripts/*.sh
|
||||
- name: Actionlint
|
||||
@@ -77,169 +85,6 @@ jobs:
|
||||
--offline --no-progress --verbose
|
||||
./README.md ./CONTRIBUTING.md ./SECURITY.md ./AGENTS.md ./docs ./examples
|
||||
|
||||
python:
|
||||
name: python
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 20
|
||||
defaults:
|
||||
run:
|
||||
working-directory: examples/dsh-agents-as-a-service/gateway
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
||||
with:
|
||||
persist-credentials: false
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
|
||||
with:
|
||||
python-version: "3.12"
|
||||
cache: pip
|
||||
cache-dependency-path: |
|
||||
examples/dsh-agents-as-a-service/gateway/requirements.txt
|
||||
examples/dsh-agents-as-a-service/gateway/requirements-dev.txt
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
set -euo pipefail
|
||||
python -m pip install --upgrade pip
|
||||
pip install -r requirements.txt -r requirements-dev.txt
|
||||
- name: Lint
|
||||
run: |
|
||||
set -euo pipefail
|
||||
python -m ruff check .
|
||||
python -m ruff format --check .
|
||||
- name: Unit tests with coverage
|
||||
run: >-
|
||||
python -m pytest -q
|
||||
--cov=. --cov-report=xml:coverage.xml --cov-report=term
|
||||
--junitxml=junit.xml
|
||||
- name: Coverage summary
|
||||
run: |
|
||||
set -euo pipefail
|
||||
{
|
||||
echo "### Gateway test coverage"
|
||||
python -m coverage report
|
||||
} >> "$GITHUB_STEP_SUMMARY"
|
||||
- name: Upload test artefacts
|
||||
if: always()
|
||||
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
|
||||
with:
|
||||
name: gateway-test-results
|
||||
path: |
|
||||
examples/dsh-agents-as-a-service/gateway/junit.xml
|
||||
examples/dsh-agents-as-a-service/gateway/coverage.xml
|
||||
if-no-files-found: warn
|
||||
retention-days: 7
|
||||
|
||||
manifests:
|
||||
name: manifests
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 25
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
||||
with:
|
||||
persist-credentials: false
|
||||
- name: Create kind cluster
|
||||
uses: helm/kind-action@1544676c07bb3570fcb70166beaa07c4528fd4aa # v1.15.1
|
||||
with:
|
||||
version: v0.33.0
|
||||
cluster_name: dsh-ci
|
||||
- name: Read versions.env
|
||||
run: |
|
||||
set -euo pipefail
|
||||
while IFS= read -r line; do
|
||||
case "$line" in ''|'#'*) continue ;; esac
|
||||
echo "$line" >> "$GITHUB_ENV"
|
||||
done < versions.env
|
||||
- name: Install agent-sandbox and Gateway API CRDs
|
||||
run: |
|
||||
set -euo pipefail
|
||||
kubectl apply -f "https://github.com/kubernetes-sigs/agent-sandbox/releases/download/$AGENT_SANDBOX_VERSION/sandbox-with-extensions.yaml"
|
||||
kubectl wait --for=condition=Established crd/sandboxtemplates.extensions.agents.x-k8s.io --timeout=180s
|
||||
kubectl wait --for=condition=Established crd/sandboxwarmpools.extensions.agents.x-k8s.io --timeout=180s
|
||||
kubectl wait --for=condition=Established crd/sandboxclaims.extensions.agents.x-k8s.io --timeout=180s
|
||||
kubectl wait --for=condition=Established crd/sandboxes.agents.x-k8s.io --timeout=180s
|
||||
kubectl apply -f "https://github.com/kubernetes-sigs/gateway-api/releases/download/$GATEWAY_API_VERSION/standard-install.yaml"
|
||||
kubectl wait --for=condition=Established crd/gateways.gateway.networking.k8s.io --timeout=180s
|
||||
kubectl wait --for=condition=Established crd/httproutes.gateway.networking.k8s.io --timeout=180s
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
|
||||
with:
|
||||
python-version: "3.12"
|
||||
- name: Install check dependencies
|
||||
run: pip install "pyyaml==6.0.3"
|
||||
- name: Validate manifest shape
|
||||
run: python scripts/check-manifests.py
|
||||
- name: Validate manifests (server dry-run)
|
||||
run: |
|
||||
set -euo pipefail
|
||||
kubectl create namespace dsh-demo
|
||||
shopt -s globstar nullglob
|
||||
count=0
|
||||
for file in examples/*/manifests/**/*.yaml; do
|
||||
case "$file" in *.yaml.example) continue ;; esac
|
||||
echo "== $file =="
|
||||
kubectl apply --dry-run=server -f "$file"
|
||||
count=$((count + 1))
|
||||
done
|
||||
if [ "$count" -eq 0 ]; then
|
||||
echo "no manifests were validated" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "validated $count manifests"
|
||||
|
||||
gateway-image:
|
||||
name: gateway-image
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 20
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
||||
with:
|
||||
persist-credentials: false
|
||||
- name: Build gateway image
|
||||
run: docker build -t dsh-aaas-gateway:ci examples/dsh-agents-as-a-service/gateway/
|
||||
- name: Smoke-test /healthz without a cluster
|
||||
run: |
|
||||
set -euo pipefail
|
||||
cat > /tmp/kubeconfig <<'EOF'
|
||||
apiVersion: v1
|
||||
kind: Config
|
||||
clusters:
|
||||
- name: dummy
|
||||
cluster:
|
||||
server: https://127.0.0.1:6443
|
||||
insecure-skip-tls-verify: true
|
||||
contexts:
|
||||
- name: dummy
|
||||
context:
|
||||
cluster: dummy
|
||||
user: dummy
|
||||
users:
|
||||
- name: dummy
|
||||
user: {}
|
||||
current-context: dummy
|
||||
EOF
|
||||
docker run -d --name gateway-smoke \
|
||||
-p 18080:8080 \
|
||||
-e NAMESPACE=dsh-demo \
|
||||
-e POOL=dsh-pool \
|
||||
-e KUBECONFIG=/tmp/kubeconfig \
|
||||
-v /tmp/kubeconfig:/tmp/kubeconfig:ro \
|
||||
dsh-aaas-gateway:ci
|
||||
trap 'docker logs gateway-smoke || true; docker rm -f gateway-smoke || true' EXIT
|
||||
code=""
|
||||
for _ in $(seq 1 30); do
|
||||
code=$(curl -s -o /dev/null -w '%{http_code}' http://localhost:18080/healthz || true)
|
||||
[ "$code" = "200" ] && break
|
||||
sleep 1
|
||||
done
|
||||
if [ "$code" != "200" ]; then
|
||||
echo "gateway /healthz returned '$code' instead of 200" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "gateway image healthy: GET /healthz -> 200"
|
||||
|
||||
dependency-review:
|
||||
name: dependency-review
|
||||
if: github.event_name == 'pull_request'
|
||||
|
||||
2
.github/workflows/codeql.yml
vendored
2
.github/workflows/codeql.yml
vendored
@@ -58,6 +58,8 @@ jobs:
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
# Add any setup steps before running the `github/codeql-action/init` action.
|
||||
# This includes steps like installing compilers or runtimes (`actions/setup-node`
|
||||
|
||||
9
.github/workflows/e2e-kind.yaml
vendored
9
.github/workflows/e2e-kind.yaml
vendored
@@ -40,7 +40,10 @@ jobs:
|
||||
- name: Install agent-sandbox
|
||||
run: |
|
||||
set -euo pipefail
|
||||
kubectl apply -f "https://github.com/kubernetes-sigs/agent-sandbox/releases/download/$AGENT_SANDBOX_VERSION/sandbox-with-extensions.yaml"
|
||||
curl -fsSL --proto '=https' -o /tmp/agent-sandbox.yaml \
|
||||
"https://github.com/kubernetes-sigs/agent-sandbox/releases/download/$AGENT_SANDBOX_VERSION/sandbox-with-extensions.yaml"
|
||||
printf '%s %s\n' "$AGENT_SANDBOX_MANIFEST_SHA256" /tmp/agent-sandbox.yaml | sha256sum -c -
|
||||
kubectl apply -f /tmp/agent-sandbox.yaml
|
||||
kubectl -n agent-sandbox-system rollout status deployment --timeout=180s
|
||||
kubectl wait --for=condition=Established crd/sandboxtemplates.extensions.agents.x-k8s.io --timeout=180s
|
||||
kubectl wait --for=condition=Established crd/sandboxwarmpools.extensions.agents.x-k8s.io --timeout=180s
|
||||
@@ -66,7 +69,9 @@ jobs:
|
||||
echo "== events =="
|
||||
kubectl get events -n dsh-demo --sort-by=.lastTimestamp 2>&1 || true
|
||||
echo "== gateway logs =="
|
||||
kubectl -n dsh-demo logs deploy/dsh-gateway --tail=200 2>&1 || true
|
||||
# Redact any token material before this file becomes an artifact.
|
||||
kubectl -n dsh-demo logs deploy/dsh-gateway --tail=200 2>&1 \
|
||||
| sed -E 's/(token=)[^&"[:space:]]+/\1[REDACTED]/g' || true
|
||||
} > /tmp/e2e-cluster.txt
|
||||
- name: Upload cluster state
|
||||
if: always()
|
||||
|
||||
90
.github/workflows/release.yaml
vendored
90
.github/workflows/release.yaml
vendored
@@ -16,17 +16,11 @@ on:
|
||||
description: Existing tag to release (must match versions.env)
|
||||
required: true
|
||||
type: string
|
||||
publish_legacy_gateway_alias:
|
||||
description: Also publish the legacy :gateway alias (one release only)
|
||||
required: false
|
||||
default: true
|
||||
type: boolean
|
||||
|
||||
# Read-only by default; the build and publish jobs add the write scopes they
|
||||
# need as job-level grants.
|
||||
permissions:
|
||||
contents: write
|
||||
packages: write
|
||||
id-token: write
|
||||
attestations: write
|
||||
contents: read
|
||||
|
||||
concurrency:
|
||||
group: release-${{ github.ref }}
|
||||
@@ -40,11 +34,13 @@ jobs:
|
||||
outputs:
|
||||
identity: ${{ steps.meta.outputs.identity }}
|
||||
dsh_version: ${{ steps.meta.outputs.dsh_version }}
|
||||
dsh_commit: ${{ steps.meta.outputs.dsh_commit }}
|
||||
pnpm_version: ${{ steps.meta.outputs.pnpm_version }}
|
||||
release_revision: ${{ steps.meta.outputs.release_revision }}
|
||||
ghcr_image: ${{ steps.meta.outputs.ghcr_image }}
|
||||
gateway_image: ${{ steps.meta.outputs.gateway_image }}
|
||||
tag: ${{ steps.meta.outputs.tag }}
|
||||
run_tests: ${{ steps.meta.outputs.run_tests }}
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
||||
@@ -55,13 +51,17 @@ jobs:
|
||||
- name: Validate tag against versions.env
|
||||
id: meta
|
||||
shell: bash
|
||||
env:
|
||||
# Passed through the environment, never interpolated into the script:
|
||||
# a tag name or dispatch input can contain shell metacharacters.
|
||||
REQUESTED_TAG: ${{ inputs.tag || github.ref_name }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
set -a
|
||||
# shellcheck disable=SC1091
|
||||
source versions.env
|
||||
set +a
|
||||
tag="${{ inputs.tag || github.ref_name }}"
|
||||
tag="$REQUESTED_TAG"
|
||||
identity="$DSH_VERSION"
|
||||
[ -z "$RELEASE_REVISION" ] || identity="$DSH_VERSION-$RELEASE_REVISION"
|
||||
if [ "$tag" != "$identity" ]; then
|
||||
@@ -82,22 +82,44 @@ jobs:
|
||||
echo "tag commit $commit is not on main or a release branch" >&2
|
||||
exit 1
|
||||
fi
|
||||
# Tests already ran in PR CI when the tag is on main; only re-run
|
||||
# them for release-branch tags (where ci.yaml does not trigger).
|
||||
run_tests=true
|
||||
if git rev-parse -q --verify origin/main >/dev/null && git merge-base --is-ancestor "$commit" origin/main; then
|
||||
run_tests=false
|
||||
fi
|
||||
else
|
||||
echo "tag '$tag' does not exist in this repository" >&2
|
||||
exit 1
|
||||
fi
|
||||
if ! git ls-remote --tags --exit-code "$DSH_REPO" "$DSH_VERSION" >/dev/null; then
|
||||
if ! upstream="$(git ls-remote --tags --exit-code "$DSH_REPO" \
|
||||
"refs/tags/$DSH_VERSION^{}" "refs/tags/$DSH_VERSION")"; then
|
||||
echo "upstream ref '$DSH_VERSION' not found in $DSH_REPO" >&2
|
||||
exit 1
|
||||
fi
|
||||
# Never release a ref that no longer resolves to the pinned commit: an
|
||||
# annotated tag has a peeled ^{} line, a lightweight tag only the ref.
|
||||
resolved="$(printf '%s\n' "$upstream" \
|
||||
| awk -v t="refs/tags/$DSH_VERSION^{}" '$2 == t {print $1}' | head -n1)"
|
||||
if [ -z "$resolved" ]; then
|
||||
resolved="$(printf '%s\n' "$upstream" \
|
||||
| awk -v t="refs/tags/$DSH_VERSION" '$2 == t {print $1}' | head -n1)"
|
||||
fi
|
||||
if [ "$resolved" != "$DSH_COMMIT" ]; then
|
||||
echo "upstream $DSH_REPO@$DSH_VERSION resolved to '$resolved'," \
|
||||
"versions.env pins DSH_COMMIT=$DSH_COMMIT" >&2
|
||||
exit 1
|
||||
fi
|
||||
{
|
||||
echo "identity=$identity"
|
||||
echo "dsh_version=$DSH_VERSION"
|
||||
echo "dsh_commit=$DSH_COMMIT"
|
||||
echo "pnpm_version=$PNPM_VERSION"
|
||||
echo "release_revision=$RELEASE_REVISION"
|
||||
echo "ghcr_image=$GHCR_IMAGE"
|
||||
echo "gateway_image=$GATEWAY_IMAGE"
|
||||
echo "tag=$tag"
|
||||
echo "run_tests=$run_tests"
|
||||
} >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Verify repository pins
|
||||
@@ -106,6 +128,9 @@ jobs:
|
||||
tests:
|
||||
name: Unit tests and lint
|
||||
needs: meta
|
||||
# The test steps are guarded individually rather than the whole job so a
|
||||
# main-branch tag (run_tests=false) still leaves this job "successful" and
|
||||
# does not cascade-skip build-main/build-gateway.
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 15
|
||||
defaults:
|
||||
@@ -117,6 +142,7 @@ jobs:
|
||||
with:
|
||||
persist-credentials: false
|
||||
- name: Set up Python
|
||||
if: needs.meta.outputs.run_tests == 'true'
|
||||
uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
|
||||
with:
|
||||
python-version: "3.12"
|
||||
@@ -125,16 +151,19 @@ jobs:
|
||||
examples/dsh-agents-as-a-service/gateway/requirements.txt
|
||||
examples/dsh-agents-as-a-service/gateway/requirements-dev.txt
|
||||
- name: Install dependencies
|
||||
if: needs.meta.outputs.run_tests == 'true'
|
||||
run: |
|
||||
set -euo pipefail
|
||||
python -m pip install --upgrade pip
|
||||
pip install -r requirements.txt -r requirements-dev.txt
|
||||
pip install --require-hashes -r requirements.txt -r requirements-dev.txt
|
||||
- name: Lint
|
||||
if: needs.meta.outputs.run_tests == 'true'
|
||||
run: |
|
||||
set -euo pipefail
|
||||
python -m ruff check .
|
||||
python -m ruff format --check .
|
||||
- name: Unit tests
|
||||
if: needs.meta.outputs.run_tests == 'true'
|
||||
run: python -m pytest -q
|
||||
|
||||
build-main:
|
||||
@@ -153,6 +182,7 @@ jobs:
|
||||
dockerfile: Dockerfile
|
||||
build-args: |
|
||||
DSH_VERSION=${{ needs.meta.outputs.dsh_version }}
|
||||
DSH_COMMIT=${{ needs.meta.outputs.dsh_commit }}
|
||||
RELEASE_REVISION=${{ needs.meta.outputs.release_revision }}
|
||||
PNPM_VERSION=${{ needs.meta.outputs.pnpm_version }}
|
||||
VCS_REF=${{ github.sha }}
|
||||
@@ -178,33 +208,14 @@ jobs:
|
||||
${{ needs.meta.outputs.gateway_image }}:${{ needs.meta.outputs.identity }}
|
||||
${{ needs.meta.outputs.gateway_image }}:latest
|
||||
|
||||
legacy-gateway-alias:
|
||||
name: Publish legacy :gateway alias (temporary)
|
||||
needs: [meta, build-gateway]
|
||||
if: github.event_name != 'workflow_dispatch' || inputs.publish_legacy_gateway_alias
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 15
|
||||
steps:
|
||||
- name: Login to GitHub Container Registry
|
||||
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
|
||||
with:
|
||||
registry: ghcr.io
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.GITHUB_TOKEN }}
|
||||
- name: Retag gateway as the legacy package tag
|
||||
env:
|
||||
SOURCE: ${{ needs.meta.outputs.gateway_image }}:${{ needs.meta.outputs.identity }}
|
||||
LEGACY: ${{ needs.meta.outputs.ghcr_image }}:gateway
|
||||
run: |
|
||||
set -euo pipefail
|
||||
docker buildx imagetools create -t "$LEGACY" "$SOURCE"
|
||||
echo "Deprecated alias $LEGACY now points at $SOURCE" >> "$GITHUB_STEP_SUMMARY"
|
||||
|
||||
publish-release:
|
||||
name: Changelog and GitHub Release
|
||||
needs: [meta, build-main, build-gateway]
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 20
|
||||
permissions:
|
||||
contents: write
|
||||
pull-requests: write
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
||||
@@ -214,11 +225,17 @@ jobs:
|
||||
- name: Install git-cliff
|
||||
env:
|
||||
GIT_CLIFF_VERSION: v2.14.2
|
||||
# SHA-256 of git-cliff-2.14.2-x86_64-unknown-linux-gnu.tar.gz; upstream
|
||||
# publishes the matching SHA-512 next to the asset. Update with the version.
|
||||
GIT_CLIFF_SHA256: 24f397c733add5390fdceee3a2088588ab0d5f944ce00d34cb7029b888cf2db4
|
||||
run: |
|
||||
set -euo pipefail
|
||||
version="${GIT_CLIFF_VERSION#v}"
|
||||
curl -fsSL -o /tmp/git-cliff.tar.gz "https://github.com/orhun/git-cliff/releases/download/$GIT_CLIFF_VERSION/git-cliff-$version-x86_64-unknown-linux-gnu.tar.gz"
|
||||
tar -xzf /tmp/git-cliff.tar.gz -C /tmp
|
||||
asset="git-cliff-${version}-x86_64-unknown-linux-gnu.tar.gz"
|
||||
curl -fsSL --proto '=https' -o "/tmp/$asset" \
|
||||
"https://github.com/orhun/git-cliff/releases/download/$GIT_CLIFF_VERSION/$asset"
|
||||
printf '%s %s\n' "$GIT_CLIFF_SHA256" "/tmp/$asset" | sha256sum -c -
|
||||
tar -xzf "/tmp/$asset" -C /tmp
|
||||
install -m 0755 "/tmp/git-cliff-$version/git-cliff" /usr/local/bin/git-cliff
|
||||
git-cliff --version
|
||||
|
||||
@@ -248,6 +265,7 @@ jobs:
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
IDENTITY: ${{ needs.meta.outputs.identity }}
|
||||
continue-on-error: true
|
||||
run: |
|
||||
set -euo pipefail
|
||||
git config user.name "github-actions[bot]"
|
||||
|
||||
585
.opencode/plugins/entire.ts
Normal file
585
.opencode/plugins/entire.ts
Normal file
@@ -0,0 +1,585 @@
|
||||
// Entire CLI plugin for OpenCode
|
||||
// Auto-generated by `entire enable --agent opencode`
|
||||
// Do not edit manually — changes will be overwritten on next install.
|
||||
//
|
||||
// Uses node:child_process so hooks work under both Bun (OpenCode CLI/TUI)
|
||||
// and Node (OpenCode Desktop's Electron sidecar). See #2014.
|
||||
import { spawn, spawnSync } from "node:child_process"
|
||||
import type { Plugin } from "@opencode-ai/plugin"
|
||||
|
||||
export const EntirePlugin: Plugin = async ({ directory }) => {
|
||||
// Track seen user messages to fire turn-start only once per message
|
||||
const seenUserMessages = new Set<string>()
|
||||
// Track current session ID for message events (which don't include sessionID)
|
||||
let currentSessionID: string | null = null
|
||||
// Track the model used by the most recent assistant message
|
||||
let currentModel: string | null = null
|
||||
// In-memory store for message metadata (role, tokens, etc.)
|
||||
const messageStore = new Map<string, any>()
|
||||
// One-time model-context injection captured from the turn-start hook's stdout,
|
||||
// applied on the next LLM call via experimental.chat.system.transform.
|
||||
let pendingInjection: string | null = null
|
||||
// Child (subagent) sessions. OpenCode's task tool runs each subagent as a
|
||||
// real session, so without this set a child would register as the user's
|
||||
// session: its write would take the checkpoint and the parent would log
|
||||
// "no files modified". Learned from session.* events' `parentID`, and from the child
|
||||
// ID surfaced in the parent's task part / tool.execute.after — the latter
|
||||
// two also cover a child session resumed via `task_id` from an earlier
|
||||
// process, whose own session.created predates this plugin instance. These
|
||||
// sets live for the process and are cleared only on server.instance.disposed.
|
||||
const childSessions = new Set<string>()
|
||||
// child session ID -> the top-level (user's) session it descends from. With
|
||||
// subagent_depth > 1 a child can itself call the task tool; that
|
||||
// grandchild's task is recorded on the top-level session, the only one
|
||||
// Entire tracks, keyed by its own callID like any other task.
|
||||
const rootOf = new Map<string, string>()
|
||||
// child session ID -> the session that launched it, so aborting a session
|
||||
// can end the background calls it launched.
|
||||
const parentOf = new Map<string, string>()
|
||||
// The per-call maps and sets below are keyed by callKey: the calling
|
||||
// session plus the hook key. A callID is unique only within one session
|
||||
// (some providers number calls per conversation), so a child's own task
|
||||
// call can share one with its parent's running call.
|
||||
// Call keys already announced via subagent-start (the running part update
|
||||
// repeats), so a late running update cannot announce a task again after its
|
||||
// stop. Cleared for a key when a new call starts under it (a reused callID).
|
||||
const announcedTasks = new Set<string>()
|
||||
// Call key -> Date.now() at tool.execute.before, sent as started_at. A
|
||||
// child resumed via `task_id` holds every earlier call's messages too; this
|
||||
// is what lets the CLI keep only the ones this call produced. The hook key
|
||||
// is the part's callID, except for a command subtask (`subtask: true`),
|
||||
// whose before/after hooks carry the task part's own id instead.
|
||||
const taskStartedAt = new Map<string, number>()
|
||||
// Announced tasks not yet stopped: call key -> its subagent-stop payload.
|
||||
// Every stop path takes the entry out, so a task stops at most once.
|
||||
const liveTasks = new Map<string, Record<string, unknown>>()
|
||||
// Call key of a task part id -> call key of its callID, to map a command
|
||||
// subtask's hook key to the callID its subagent-start used.
|
||||
const partCallID = new Map<string, string>()
|
||||
// Call keys whose stop fired before their running part was announced. The
|
||||
// late announcement is then skipped rather than opening a task that would
|
||||
// never stop.
|
||||
const stoppedBeforeStart = new Set<string>()
|
||||
// Call keys of tasks stopped from their error part, so a
|
||||
// tool.execute.after that still arrives does not stop them again.
|
||||
const stoppedOnError = new Set<string>()
|
||||
// Background children (OPENCODE_EXPERIMENTAL_BACKGROUND_SUBAGENTS, or a
|
||||
// foreground task promoted to the background): child ID -> subagent-stop
|
||||
// payloads held back until the child's own session goes idle. A queue: a
|
||||
// resume via `task_id` that joins a running child is queued behind the
|
||||
// current run and runs as its own busy -> idle cycle, so each idle ends the
|
||||
// oldest held call.
|
||||
const backgroundTasks = new Map<string, Record<string, unknown>[]>()
|
||||
// Sessions whose last session.status was not idle. An idle only ends a
|
||||
// held call when it follows a busy period: an errored run reports idle
|
||||
// twice.
|
||||
const busySessions = new Set<string>()
|
||||
// Children whose held stops all end at their next idle: an abort reached
|
||||
// them mid-run. Cancelling a job does not cancel a joined run already
|
||||
// executing, so the child's own idle, not the abort, is when it ends.
|
||||
const drainAtIdle = new Set<string>()
|
||||
|
||||
/**
|
||||
* Build the shell command for a hook invocation.
|
||||
* Uses sh -c so the command is guarded by a `command -v` probe: when the
|
||||
* entire binary is not on PATH the hook exits 0 rather than failing the
|
||||
* surrounding OpenCode operation.
|
||||
*/
|
||||
function hookCmd(hookName: string): string[] {
|
||||
return ["sh", "-c", `if ! command -v entire >/dev/null 2>&1; then exit 0; fi; exec entire hooks opencode ${hookName}`]
|
||||
}
|
||||
|
||||
/**
|
||||
* Pipe JSON payload to an entire hooks command (async).
|
||||
* Errors are logged but never thrown — plugin failures must not crash OpenCode.
|
||||
*/
|
||||
async function callHook(hookName: string, payload: Record<string, unknown>) {
|
||||
try {
|
||||
const json = JSON.stringify(payload)
|
||||
const [cmd, ...args] = hookCmd(hookName)
|
||||
await new Promise<void>((resolve) => {
|
||||
const proc = spawn(cmd, args, {
|
||||
cwd: directory,
|
||||
stdio: ["pipe", "ignore", "ignore"],
|
||||
})
|
||||
proc.on("error", () => resolve())
|
||||
proc.on("close", () => resolve())
|
||||
proc.stdin?.end(json + "\n")
|
||||
})
|
||||
} catch {
|
||||
// Silently ignore — plugin failures must not crash OpenCode
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Synchronous variant for hooks that must complete before subsequent agent work
|
||||
* or process exit. `turn-start` must finish initializing session state before a
|
||||
* fast mid-turn commit can hit git hooks, and `turn-end` / `session-end` must
|
||||
* finish before `opencode run` tears down its event loop.
|
||||
*/
|
||||
function callHookSync(hookName: string, payload: Record<string, unknown>) {
|
||||
try {
|
||||
const json = JSON.stringify(payload)
|
||||
const [cmd, ...args] = hookCmd(hookName)
|
||||
spawnSync(cmd, args, {
|
||||
cwd: directory,
|
||||
input: json + "\n",
|
||||
stdio: ["pipe", "ignore", "ignore"],
|
||||
})
|
||||
} catch {
|
||||
// Silently ignore — plugin failures must not crash OpenCode
|
||||
}
|
||||
}
|
||||
|
||||
// parseInjectedContext scans a hook's stdout for Entire's injection envelope
|
||||
// ({"inject_context":"..."}) and returns the text to inject, or null.
|
||||
function parseInjectedContext(stdout: string): string | null {
|
||||
if (!stdout) return null
|
||||
for (const line of stdout.split("\n")) {
|
||||
const trimmed = line.trim()
|
||||
if (!trimmed.startsWith("{")) continue
|
||||
try {
|
||||
const parsed = JSON.parse(trimmed) as { inject_context?: unknown }
|
||||
if (typeof parsed.inject_context === "string" && parsed.inject_context.length > 0) {
|
||||
return parsed.inject_context
|
||||
}
|
||||
} catch {
|
||||
// not our envelope — ignore
|
||||
}
|
||||
}
|
||||
return null
|
||||
}
|
||||
|
||||
// fireTurnStart fires turn-start synchronously (state must be ready before
|
||||
// mid-turn commits) and stashes any model-context injection emitted on stdout
|
||||
// for experimental.chat.system.transform to apply. Entire emits the injection
|
||||
// at most once per session, so pendingInjection is set on the first turn only.
|
||||
function fireTurnStart(payload: Record<string, unknown>) {
|
||||
try {
|
||||
const json = JSON.stringify(payload)
|
||||
const [cmd, ...args] = hookCmd("turn-start")
|
||||
const proc = spawnSync(cmd, args, {
|
||||
cwd: directory,
|
||||
input: json + "\n",
|
||||
encoding: "utf8",
|
||||
stdio: ["pipe", "pipe", "ignore"],
|
||||
})
|
||||
const out = typeof proc.stdout === "string" ? proc.stdout : ""
|
||||
const injected = parseInjectedContext(out)
|
||||
if (injected) pendingInjection = injected
|
||||
} catch {
|
||||
// Silently ignore — plugin failures must not crash OpenCode
|
||||
}
|
||||
}
|
||||
|
||||
// trackChild records a child and the top-level session it descends from.
|
||||
function trackChild(childID: string, parentID: string) {
|
||||
childSessions.add(childID)
|
||||
if (!rootOf.has(childID)) rootOf.set(childID, rootOf.get(parentID) ?? parentID)
|
||||
if (!parentOf.has(childID)) parentOf.set(childID, parentID)
|
||||
}
|
||||
|
||||
// callKey scopes a hook key (a callID or a task part id) to the session
|
||||
// that made the call.
|
||||
function callKey(sessionID: string, hookKey: string): string {
|
||||
return sessionID + "\0" + hookKey
|
||||
}
|
||||
|
||||
// topLevelSession maps a session to the top-level session it belongs to.
|
||||
function topLevelSession(sessionID: string): string {
|
||||
return rootOf.get(sessionID) ?? sessionID
|
||||
}
|
||||
|
||||
// announceTask fires subagent-start the first time a task part is running
|
||||
// with its child bound (state.metadata.sessionId). tool.execute.before fires
|
||||
// earlier but has no child ID yet, and session.created for the child can
|
||||
// interleave with a sibling's, so neither is a safe join. The running update
|
||||
// repeats, hence once per call key. Called for task parts in child sessions
|
||||
// too, so a nested subagent is announced on the top-level session.
|
||||
function announceTask(part: any) {
|
||||
if (!(part?.type === "tool" && part.tool === "task" && part.callID &&
|
||||
part.state?.status === "running" && part.state?.metadata?.sessionId)) return
|
||||
const sessionID = part.sessionID ?? currentSessionID
|
||||
if (!sessionID) return
|
||||
const key = callKey(sessionID, part.callID)
|
||||
const partKey = callKey(sessionID, part.id)
|
||||
if (announcedTasks.has(key)) return
|
||||
announcedTasks.add(key)
|
||||
trackChild(part.state.metadata.sessionId, sessionID)
|
||||
// Without the before hook (the plugin loaded mid-call), the part's own
|
||||
// start, OpenCode's clock before the child's prompt, still bounds it.
|
||||
const startedAt = taskStartedAt.get(key) ?? taskStartedAt.get(partKey) ?? part.state?.time?.start ?? 0
|
||||
if (stoppedBeforeStart.delete(key) || stoppedBeforeStart.delete(partKey)) {
|
||||
forgetTaskKeys(part, sessionID)
|
||||
return
|
||||
}
|
||||
const payload = {
|
||||
session_id: topLevelSession(sessionID),
|
||||
tool_use_id: part.callID,
|
||||
subagent_id: part.state.metadata.sessionId,
|
||||
subagent_type: part.state?.input?.subagent_type ?? "",
|
||||
task_description: part.state?.input?.description ?? "",
|
||||
started_at: startedAt,
|
||||
}
|
||||
liveTasks.set(key, payload)
|
||||
if (part.id) partCallID.set(partKey, key)
|
||||
callHookSync("subagent-start", payload)
|
||||
}
|
||||
|
||||
// announcedKey maps a call key to the call key the task was announced under.
|
||||
function announcedKey(key: string): string {
|
||||
return partCallID.get(key) ?? key
|
||||
}
|
||||
|
||||
// takeLiveTask removes and returns the announced, not yet stopped task a
|
||||
// call key belongs to.
|
||||
function takeLiveTask(key: string): Record<string, unknown> | undefined {
|
||||
const announced = announcedKey(key)
|
||||
partCallID.delete(key)
|
||||
const payload = liveTasks.get(announced)
|
||||
liveTasks.delete(announced)
|
||||
return payload
|
||||
}
|
||||
|
||||
function forgetTaskKeys(part: any, sessionID: string) {
|
||||
taskStartedAt.delete(callKey(sessionID, part.callID))
|
||||
if (part.id) taskStartedAt.delete(callKey(sessionID, part.id))
|
||||
}
|
||||
|
||||
// stopFailedTask fires subagent-stop for an announced task whose part
|
||||
// ended in error without tool.execute.after: OpenCode skips that hook when
|
||||
// the call is aborted (Esc, `opencode run` teardown) or its execute throws.
|
||||
// A task that failed before its child was bound was never announced, so
|
||||
// nothing fires for it.
|
||||
function stopFailedTask(part: any) {
|
||||
if (!(part?.type === "tool" && part.tool === "task" && part.callID &&
|
||||
part.state?.status === "error")) return
|
||||
const sessionID = part.sessionID ?? currentSessionID ?? ""
|
||||
const key = callKey(sessionID, part.callID)
|
||||
const partKey = callKey(sessionID, part.id)
|
||||
const payload = takeLiveTask(key)
|
||||
forgetTaskKeys(part, sessionID)
|
||||
stoppedBeforeStart.delete(key)
|
||||
if (part.id) {
|
||||
partCallID.delete(partKey)
|
||||
stoppedBeforeStart.delete(partKey)
|
||||
}
|
||||
if (!payload) return
|
||||
stoppedOnError.add(key)
|
||||
if (part.id) stoppedOnError.add(partKey)
|
||||
callHookSync("subagent-stop", payload)
|
||||
}
|
||||
|
||||
// finishBackgroundTask fires the oldest held subagent-stop for a background
|
||||
// child whose run went idle: its work finished, failed, or was aborted
|
||||
// (`opencode run` exiting with the child still running). Each case ends the
|
||||
// call; a later call queued on the same child ends at its own run's idle.
|
||||
function finishBackgroundTask(childID: string) {
|
||||
const payloads = backgroundTasks.get(childID)
|
||||
const payload = payloads?.shift()
|
||||
if (!payload) return
|
||||
if (payloads.length === 0) backgroundTasks.delete(childID)
|
||||
callHookSync("subagent-stop", payload)
|
||||
}
|
||||
|
||||
// drainBackgroundTasks fires every held subagent-stop for a child whose
|
||||
// background job is over. A job that fails or is cancelled drops the runs
|
||||
// still queued on it, so their calls get no idle of their own; left held,
|
||||
// the next resume of the child would end them at its idle instead.
|
||||
function drainBackgroundTasks(childID: string) {
|
||||
const payloads = backgroundTasks.get(childID)
|
||||
if (!payloads) return
|
||||
backgroundTasks.delete(childID)
|
||||
for (const payload of payloads) callHookSync("subagent-stop", payload)
|
||||
}
|
||||
|
||||
// A background job's end reaches the session that launched it as a
|
||||
// synthetic text part: <task id="<child>" state="completed|error">.
|
||||
const backgroundResult = /^<task id="([^"]+)" state="(completed|error)">/
|
||||
|
||||
// backgroundSessionAborted ends the held calls of an aborted session and of
|
||||
// the children it launched (Esc cancels its background jobs and reports no
|
||||
// result): at once for an idle child, at its next idle for a busy one.
|
||||
function backgroundSessionAborted(sessionID: string) {
|
||||
const targets = [sessionID]
|
||||
for (const [childID, parentID] of parentOf) if (parentID === sessionID) targets.push(childID)
|
||||
for (const target of targets) {
|
||||
if (!backgroundTasks.has(target)) continue
|
||||
if (busySessions.has(target)) drainAtIdle.add(target)
|
||||
else drainBackgroundTasks(target)
|
||||
}
|
||||
}
|
||||
|
||||
function resetSessionTracking(sessionID: string) {
|
||||
if (currentSessionID === sessionID) {
|
||||
return false
|
||||
}
|
||||
seenUserMessages.clear()
|
||||
messageStore.clear()
|
||||
currentModel = null
|
||||
currentSessionID = sessionID
|
||||
// Drop any turn-start injection captured for the prior session so it
|
||||
// can't leak into the new session's system prompt.
|
||||
pendingInjection = null
|
||||
return true
|
||||
}
|
||||
|
||||
return {
|
||||
// Apply the one-time Entire context injection captured at turn-start by
|
||||
// appending it to the system prompt for this LLM call.
|
||||
"experimental.chat.system.transform": async (input: { sessionID?: string }, output: { system: string[] }) => {
|
||||
if (input?.sessionID && childSessions.has(input.sessionID)) return
|
||||
if (pendingInjection && Array.isArray(output.system)) {
|
||||
output.system.push(pendingInjection)
|
||||
pendingInjection = null
|
||||
}
|
||||
},
|
||||
// Subagent call start. No child ID exists yet, so this only records the
|
||||
// time; subagent-start fires from the running task part below.
|
||||
"tool.execute.before": async (input) => {
|
||||
try {
|
||||
if (input.tool !== "task") return
|
||||
const key = callKey(input.sessionID, input.callID)
|
||||
// A new call under this key: some providers number calls per
|
||||
// conversation, so an earlier call's markers must not dedupe or
|
||||
// swallow this one's start and stop.
|
||||
announcedTasks.delete(key)
|
||||
stoppedBeforeStart.delete(key)
|
||||
stoppedOnError.delete(key)
|
||||
taskStartedAt.set(key, Date.now())
|
||||
} catch {
|
||||
// Silently ignore — plugin failures must not crash OpenCode
|
||||
}
|
||||
},
|
||||
// Subagent completion. tool.execute.after for the task tool fires once, with
|
||||
// the child session ID in output.metadata: at true completion for a
|
||||
// foreground task, immediately for a background one (metadata.background),
|
||||
// whose stop is held until the child goes idle. Synchronous: `opencode run`
|
||||
// exits on the parent's idle right after this, and an async hook would be
|
||||
// killed before it finished.
|
||||
"tool.execute.after": async (input, output) => {
|
||||
try {
|
||||
if (input.tool !== "task") return
|
||||
const key = callKey(input.sessionID, input.callID)
|
||||
const announced = takeLiveTask(key)
|
||||
if (stoppedOnError.delete(key) && !announced) {
|
||||
// Already stopped: its part ended in error first.
|
||||
taskStartedAt.delete(key)
|
||||
return
|
||||
}
|
||||
// A failed command subtask fires this hook with no output; its
|
||||
// announced child still ends here.
|
||||
const childID = output?.metadata?.sessionId ?? announced?.subagent_id
|
||||
if (!childID) {
|
||||
taskStartedAt.delete(key)
|
||||
return
|
||||
}
|
||||
trackChild(childID, input.sessionID)
|
||||
const startedAt = taskStartedAt.get(key) ?? 0
|
||||
taskStartedAt.delete(key)
|
||||
if (!announced) stoppedBeforeStart.add(key)
|
||||
// A child's own task call (subagent_depth > 1) is recorded on the
|
||||
// top-level session: the child has no Entire session of its own.
|
||||
const payload = announced ?? {
|
||||
session_id: topLevelSession(input.sessionID),
|
||||
tool_use_id: input.callID,
|
||||
subagent_id: childID,
|
||||
subagent_type: input.args?.subagent_type ?? "",
|
||||
task_description: input.args?.description ?? "",
|
||||
started_at: startedAt,
|
||||
}
|
||||
if (output?.metadata?.background === true) {
|
||||
backgroundTasks.set(childID, [...(backgroundTasks.get(childID) ?? []), payload])
|
||||
return
|
||||
}
|
||||
callHookSync("subagent-stop", payload)
|
||||
} catch {
|
||||
// Silently ignore — plugin failures must not crash OpenCode
|
||||
}
|
||||
},
|
||||
event: async ({ event }) => {
|
||||
try {
|
||||
const props = (event as any).properties
|
||||
const info = props?.info
|
||||
if (event.type.startsWith("session.") && info?.parentID && info?.id) trackChild(info.id, info.parentID)
|
||||
if (event.type === "message.part.updated") {
|
||||
announceTask(props?.part)
|
||||
stopFailedTask(props?.part)
|
||||
const part = props?.part
|
||||
// Only a part in a tracked session: synthetic text also carries
|
||||
// attachment contents, which could spell the same envelope.
|
||||
const sid = part?.sessionID
|
||||
const tracked = sid === currentSessionID || childSessions.has(sid) || [...parentOf.values()].includes(sid)
|
||||
if (tracked && part?.type === "text" && part.synthetic === true && typeof part.text === "string") {
|
||||
const result = backgroundResult.exec(part.text)
|
||||
if (result) drainBackgroundTasks(result[1])
|
||||
}
|
||||
}
|
||||
// An aborted message, reported on session.error, or on the message
|
||||
// itself when the run was interrupted outside the processor.
|
||||
if (event.type === "session.error" && props?.error?.name === "MessageAbortedError" && props?.sessionID) {
|
||||
backgroundSessionAborted(props.sessionID)
|
||||
}
|
||||
if (event.type === "message.updated" && info?.error?.name === "MessageAbortedError" && info?.sessionID) {
|
||||
backgroundSessionAborted(info.sessionID)
|
||||
}
|
||||
if (event.type === "session.status" && props?.sessionID) {
|
||||
if (props?.status?.type !== "idle") busySessions.add(props.sessionID)
|
||||
else if (busySessions.delete(props.sessionID)) {
|
||||
if (drainAtIdle.delete(props.sessionID)) drainBackgroundTasks(props.sessionID)
|
||||
else finishBackgroundTask(props.sessionID)
|
||||
}
|
||||
}
|
||||
const eventSessionID: string | undefined =
|
||||
props?.sessionID ?? info?.sessionID ?? info?.id ?? props?.part?.sessionID
|
||||
if (eventSessionID && childSessions.has(eventSessionID)) return
|
||||
switch (event.type) {
|
||||
case "session.created": {
|
||||
const session = info
|
||||
if (!session?.id) break
|
||||
// Reset per-session tracking state when switching sessions.
|
||||
if (resetSessionTracking(session.id)) {
|
||||
await callHook("session-start", {
|
||||
session_id: session.id,
|
||||
})
|
||||
}
|
||||
break
|
||||
}
|
||||
|
||||
case "message.updated": {
|
||||
const msg = info
|
||||
if (!msg) break
|
||||
|
||||
if (msg.sessionID && resetSessionTracking(msg.sessionID)) {
|
||||
callHookSync("session-start", {
|
||||
session_id: msg.sessionID,
|
||||
})
|
||||
}
|
||||
|
||||
// Store message metadata (role, time, tokens, etc.)
|
||||
messageStore.set(msg.id, msg)
|
||||
// Track model from assistant messages
|
||||
if (msg.role === "assistant" && msg.modelID) {
|
||||
currentModel = msg.modelID
|
||||
}
|
||||
|
||||
// Fallback: some opencode run flows commit before any message.part.updated
|
||||
// event is delivered for the user's prompt. Start the turn from the
|
||||
// user message itself so git hooks see an ACTIVE session in time.
|
||||
if (msg.role === "user" && !seenUserMessages.has(msg.id)) {
|
||||
seenUserMessages.add(msg.id)
|
||||
const sessionID = msg.sessionID ?? currentSessionID
|
||||
if (sessionID) {
|
||||
fireTurnStart({
|
||||
session_id: sessionID,
|
||||
prompt: "",
|
||||
model: currentModel ?? "",
|
||||
})
|
||||
}
|
||||
}
|
||||
break
|
||||
}
|
||||
|
||||
case "message.part.updated": {
|
||||
const part = props?.part
|
||||
if (!part?.messageID) break
|
||||
|
||||
// Fire turn-start on the first text part of a new user message
|
||||
const msg = messageStore.get(part.messageID)
|
||||
if (msg?.role === "user" && part.type === "text" && !seenUserMessages.has(msg.id)) {
|
||||
seenUserMessages.add(msg.id)
|
||||
const sessionID = msg.sessionID ?? currentSessionID
|
||||
if (sessionID) {
|
||||
// A synthetic part is OpenCode writing, not the user: a
|
||||
// background task's result arrives this way. The parent does
|
||||
// take a turn on it, so the turn still starts, with no prompt.
|
||||
fireTurnStart({
|
||||
session_id: sessionID,
|
||||
prompt: part.synthetic === true ? "" : (part.text ?? ""),
|
||||
model: currentModel ?? "",
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
break
|
||||
}
|
||||
|
||||
case "session.status": {
|
||||
// session.status fires in both TUI and non-interactive (run) mode.
|
||||
// session.idle is deprecated and not reliably emitted in run mode.
|
||||
if (props?.status?.type !== "idle") break
|
||||
const sessionID = props?.sessionID ?? currentSessionID
|
||||
if (!sessionID) break
|
||||
// Use sync variant: `opencode run` exits on the same idle event,
|
||||
// so an async hook would be killed before completing.
|
||||
callHookSync("turn-end", {
|
||||
session_id: sessionID,
|
||||
model: currentModel ?? "",
|
||||
})
|
||||
break
|
||||
}
|
||||
|
||||
case "session.compacted": {
|
||||
const sessionID = props?.sessionID
|
||||
if (!sessionID) break
|
||||
await callHook("compaction", {
|
||||
session_id: sessionID,
|
||||
})
|
||||
break
|
||||
}
|
||||
|
||||
case "session.deleted": {
|
||||
const session = info
|
||||
if (!session?.id) break
|
||||
seenUserMessages.clear()
|
||||
messageStore.clear()
|
||||
currentSessionID = null
|
||||
pendingInjection = null
|
||||
// childSessions/announcedTasks are NOT cleared here: a child's own
|
||||
// deletion never reaches this case (the guard above returns first),
|
||||
// so this only ever fires for an unrelated top-level session, and
|
||||
// clearing here would drop a live child back to top-level mid-task.
|
||||
// Use sync variant: session-end may fire during shutdown.
|
||||
callHookSync("session-end", {
|
||||
session_id: session.id,
|
||||
})
|
||||
break
|
||||
}
|
||||
|
||||
case "server.instance.disposed": {
|
||||
// Fires when OpenCode shuts down (TUI close or `opencode run` exit).
|
||||
// session.deleted only fires on explicit user deletion, not on quit,
|
||||
// so this is the only reliable way to end sessions on exit.
|
||||
if (!currentSessionID) break
|
||||
const sessionID = currentSessionID
|
||||
seenUserMessages.clear()
|
||||
messageStore.clear()
|
||||
currentSessionID = null
|
||||
pendingInjection = null
|
||||
childSessions.clear()
|
||||
rootOf.clear()
|
||||
parentOf.clear()
|
||||
announcedTasks.clear()
|
||||
taskStartedAt.clear()
|
||||
liveTasks.clear()
|
||||
partCallID.clear()
|
||||
stoppedBeforeStart.clear()
|
||||
stoppedOnError.clear()
|
||||
// A background child still running here is ended by the Go side's
|
||||
// SessionEnd sweep, which completes every live task record.
|
||||
backgroundTasks.clear()
|
||||
busySessions.clear()
|
||||
drainAtIdle.clear()
|
||||
// Use sync variant: this is the last event before process exit.
|
||||
callHookSync("session-end", {
|
||||
session_id: sessionID,
|
||||
})
|
||||
break
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// Silently ignore — plugin failures must not crash OpenCode
|
||||
}
|
||||
},
|
||||
}
|
||||
}
|
||||
65
AGENTS.md
65
AGENTS.md
@@ -9,8 +9,8 @@ before changing anything; it encodes the conventions the CI enforces.
|
||||
Harness Web (dsh web) as a slim, non-root, multi-arch image. Everything else
|
||||
exists to support it or to demonstrate an implementation on top of it.
|
||||
- **Every implementation is a self-contained example** under examples/<name>/.
|
||||
There is exactly one example for the agents-as-a-service platform; do not
|
||||
split it into parallel directories.
|
||||
The multi-tenant platform lives in exactly one directory; do not split it into
|
||||
parallel examples.
|
||||
- **versions.env is the single source of truth** for the DSH ref, pnpm, the
|
||||
control-plane versions and the image names.
|
||||
- **Docs describe the shipped behavior**, not intentions. If you change a
|
||||
@@ -43,17 +43,18 @@ before changing anything; it encodes the conventions the CI enforces.
|
||||
| versions.env | Every pinned version and image name |
|
||||
| Makefile | Canonical local entrypoint |
|
||||
| build.sh | Deprecated shim that forwards to make |
|
||||
| docs/ | main-image.md, examples.md, gitflow.md, versioning.md |
|
||||
| docs/ | main-image.md, examples.md, gitflow.md, versioning.md (index: README.md) |
|
||||
| examples/ | One implementation per directory (contract in docs/examples.md) |
|
||||
| examples/dsh-agents-as-a-service/ | The multi-tenant platform example (manifests, gateway, docs, scripts) |
|
||||
| examples/minimal-single-tenant/ | The just-one-DSH-Web quickstart |
|
||||
| scripts/ | check-versions.sh, check-examples.sh, check-manifests.py, set-version.sh |
|
||||
| .github/workflows/ | ci.yaml, reusable _build-image.yaml, build-*, release.yaml, e2e-kind.yaml, pr-title.yaml, codeql.yml |
|
||||
| .dsh/skills/ | Project-local DSH skills (auto-discovered); dsh-agent-sandbox-platform covers platform operation, MCP limits and extension recipes |
|
||||
| .github/workflows/ | ci.yaml (lint, dependency-review), ci-python.yaml, ci-manifests.yaml, ci-image.yaml (amd64 build + smoke test on Dockerfile/versions.env changes), reusable _build-image.yaml, build-*, release.yaml, e2e-kind.yaml, pr-title.yaml, codeql.yml |
|
||||
|
||||
## Commands
|
||||
|
||||
make check # versions, example layout and manifest YAML (run before every commit)
|
||||
make test # gateway unit tests (109+ tests)
|
||||
make test # gateway unit tests (pytest)
|
||||
make lint # ruff + shellcheck (+ yamllint/actionlint when installed)
|
||||
make image # build the main DSH Web image (slow: clones and builds DSH)
|
||||
make gateway # build the example gateway image (fast)
|
||||
@@ -71,18 +72,18 @@ needs kind plus the agent-sandbox CRDs (see the example README).
|
||||
| --- | --- |
|
||||
| Documentation only | make check; verify relative links |
|
||||
| Example manifest | make check; kubectl apply --dry-run=server in CI |
|
||||
| Optional hardening (gVisor, nodeSelector, NetworkPolicy, provider keys, failed-Pod recovery) | make check; server dry-run in CI; manual verification on a capable cluster |
|
||||
| Gateway code | make test; make lint; add or update tests |
|
||||
| Dockerfile / versions.env | make check; make image (amd64) and a container smoke test |
|
||||
| Dockerfile / versions.env | make check; the CI `build-amd64` job (`ci-image.yaml`) builds and smoke-tests the image on the PR; run `make image` locally only when you need the image |
|
||||
| Workflow | actionlint in CI; keep actions SHA-pinned; no hard-coded versions |
|
||||
| New example | make check; all files described in docs/examples.md |
|
||||
| Project skill (.dsh/skills/) | make check; validate frontmatter; keep SKILL.md under ~8 KB; shellcheck skill scripts; load it with the skill tool |
|
||||
|
||||
## Versioning and releases
|
||||
|
||||
- Release identity = DSH_VERSION, optionally with a repository revision suffix
|
||||
for re-releases at the same DSH ref (first release: dsh-v0.2.1-rc.3-r3).
|
||||
- The current identity is dsh-v0.2.1-rc.3-r3 (stored in versions.env). The
|
||||
dsh-v0.2.1-rc.3 image tag already exists and tags are immutable, which is why
|
||||
the first release under this scheme carries -r2.
|
||||
- Release identity = DSH_VERSION plus an optional -rN repository revision.
|
||||
The current identity is dsh-v0.2.1-rc.5-r6 (first release at this DSH ref, so no
|
||||
suffix); a repository-only change at the same ref publishes -r2.
|
||||
- Channels: release tags are immutable; latest follows the newest release;
|
||||
main is the development channel for both image packages.
|
||||
- A broken release is fixed by publishing -r(N+1), never by re-tagging.
|
||||
@@ -95,8 +96,9 @@ needs kind plus the agent-sandbox CRDs (see the example README).
|
||||
- Branch names: feat/, fix/, docs/, chore/, refactor/, test/.
|
||||
- Conventional Commit subjects and PR titles (feat(scope): ..., fix(scope): ...).
|
||||
- PRs are squash-merged; main stays linear and protected.
|
||||
- Required checks: lint, python, manifests, gateway-image, pr-title, CodeQL,
|
||||
dependency-review.
|
||||
- Required on every PR: lint, pr-title, CodeQL (both Analyze jobs) and
|
||||
dependency-review. python, manifests and build-amd64 are path-filtered
|
||||
on-demand checks; run them when the change touches their scope.
|
||||
- Keep diffs scoped; update docs with the code; never mix a version bump with
|
||||
unrelated changes.
|
||||
|
||||
@@ -104,9 +106,9 @@ needs kind plus the agent-sandbox CRDs (see the example README).
|
||||
|
||||
1. Read docs/examples.md and examples/README.md.
|
||||
2. Create examples/<name>/ with README.md, manifests/NN-*.yaml, optional
|
||||
manifests/exposure/<variant>/ (plus README), optional components with their
|
||||
own Dockerfile/README/tests, and scripts/ (executable, idempotent,
|
||||
self-cleaning).
|
||||
manifests/optional/<feature>/ or manifests/exposure/<variant>/ (each with a
|
||||
README), optional components with their own Dockerfile/README/tests, and
|
||||
scripts/ (executable, idempotent, self-cleaning).
|
||||
3. Publish component images under <registry>/<owner>/<repo>/<example>/<component>.
|
||||
4. Add the example to examples/README.md; scripts/check-examples.sh enforces
|
||||
the layout.
|
||||
@@ -115,13 +117,33 @@ needs kind plus the agent-sandbox CRDs (see the example README).
|
||||
|
||||
- AUTH_PROXY_HEADER is opt-in. Only enable it when the gateway Service is
|
||||
reachable exclusively through the authenticating proxy; otherwise any client
|
||||
can spoof X-Forwarded-User.
|
||||
can spoof X-Forwarded-User. The SSO variant's NetworkPolicy is required for
|
||||
production, and `examples/dsh-agents-as-a-service/scripts/preflight-sso.sh`
|
||||
verifies the setup (Secrets, gateway env, TLS reference, placeholders, second
|
||||
paths).
|
||||
- The example gateway reads the DSH launch token from pod logs. Treat pod logs
|
||||
as sensitive and keep its RBAC namespace-scoped.
|
||||
- The example shares one DEEPSEEK_API_KEY across sandboxes; do not describe it
|
||||
as tenant-isolated.
|
||||
- TLS terminates outside the gateway; set COOKIE_SECURE=true for HTTPS and keep
|
||||
the base manifest safe for local HTTP.
|
||||
- Browser sessions are opaque server-side ids held in gateway memory: they are
|
||||
revocable (logout/delete/recreate/expiry) but a gateway restart logs everyone
|
||||
out and horizontal scaling needs a shared store. Session cookies are never
|
||||
accepted as bearer tokens.
|
||||
- AUTH_RATE_LIMIT, SIGNUP_INVITE_CODE, MAX_IN_FLIGHT_PER_USER and
|
||||
WS_MAX_MESSAGE_BYTES are in-process abuse guards, not ingress rate limits,
|
||||
quotas or an audit log. Add those for an internet-facing deployment.
|
||||
- The example shares provider keys (dsh-provider-keys plus the optional
|
||||
dsh-provider-keys-extra) across sandboxes; do not describe it as
|
||||
tenant-isolated.
|
||||
- gVisor, nodeSelector, the shipped sandbox/exposure NetworkPolicies and the
|
||||
documented gateway egress policy, extra provider keys and failed-Pod recovery
|
||||
are opt-in and are not active in the base manifests; do not describe the base
|
||||
example as hardened.
|
||||
The sandbox and gateway-egress policies are required production steps; an
|
||||
exposure without its NetworkPolicy is demo-only. See the example's
|
||||
production checklist.
|
||||
- TLS terminates outside the gateway. Set COOKIE_SECURE=true for HTTPS and drop
|
||||
ALLOW_INSECURE_COOKIES; the gateway refuses COOKIE_SECURE=false unless
|
||||
ALLOW_INSECURE_COOKIES=1, which is only for the plain-HTTP local demo. Never
|
||||
expose that combination.
|
||||
|
||||
## Ask before
|
||||
|
||||
@@ -136,5 +158,6 @@ needs kind plus the agent-sandbox CRDs (see the example README).
|
||||
- [docs/examples.md](docs/examples.md) - example authoring rules
|
||||
- [docs/gitflow.md](docs/gitflow.md) - branches, PRs, required checks
|
||||
- [docs/versioning.md](docs/versioning.md) - versioning, changelog, releases
|
||||
- [.dsh/skills/dsh-agent-sandbox-platform/SKILL.md](.dsh/skills/dsh-agent-sandbox-platform/SKILL.md) - platform operations, MCP limits, extension recipes
|
||||
- [CONTRIBUTING.md](CONTRIBUTING.md) - development setup
|
||||
- [SECURITY.md](SECURITY.md) - vulnerability reporting and threat notes
|
||||
|
||||
74
CHANGELOG.md
74
CHANGELOG.md
@@ -3,24 +3,94 @@
|
||||
All notable changes to this repository are documented here.
|
||||
Generated by git-cliff from Conventional Commits; do not edit this file by hand.
|
||||
|
||||
## [Unreleased]
|
||||
## [dsh-v0.2.1-rc.5-r6] - 2026-10-31
|
||||
|
||||
### Added
|
||||
|
||||
- **example:** Persist the agent workspace on the PVC via subPath (#34) (367294b)
|
||||
- **example:** Start DSH's folder picker on the persisted workspace (#36) (82ee0d0)
|
||||
|
||||
### CI
|
||||
|
||||
- **workflows:** Streamline CI and remove redundant builds (#31) (dcb5b43)
|
||||
|
||||
### Documentation
|
||||
|
||||
- Align readmes, docs and agent guidance with current behavior (#37) (24f23e0)
|
||||
|
||||
### Fixed
|
||||
|
||||
- **gateway:** Answer client-aborted requests with 499, not a traceback (#32) (cbcc56f)
|
||||
- Ship bubblewrap for DSH's command sandbox, keep agents usable without it (#35) (a09d4ac)
|
||||
- **ci:** Harden workflow token scope and download integrity (#39) (2f959e6)
|
||||
- **gateway:** Replace token cookies with revocable sessions and harden abuse controls (#40) (1a929e3)
|
||||
|
||||
### Maintenance
|
||||
|
||||
- **changelog:** Refresh for dsh-v0.2.1-rc.5 (#30) (b57916d)
|
||||
- Bump docker/login-action (#28) (8c34f43)
|
||||
- **deps:** Bump agent-sandbox to v1.0.9 (#38) (4c073ee)
|
||||
- Improve package installation and add Python depend… (#42) (8e9bd47)
|
||||
|
||||
### Other
|
||||
|
||||
- Add continue-on-error to CHANGELOG PR step (97c5632)
|
||||
|
||||
## [dsh-v0.2.1-rc.5] - 2026-10-30
|
||||
|
||||
### Added
|
||||
|
||||
- **example:** Add optional failed-Pod recovery (#26) (14c3fb8)
|
||||
|
||||
### Fixed
|
||||
|
||||
- **ci:** Make release image digests unique and allow changelog PRs (#24) (2b43d5d)
|
||||
|
||||
### Maintenance
|
||||
|
||||
- **changelog:** Refresh for dsh-v0.2.1-rc.3-r4 (#23) (97abf75)
|
||||
- **deps:** Bump agent-sandbox to v1.0.8 (#25) (96d3030)
|
||||
- **release:** Dsh-v0.2.1-rc.5 (#29) (bed8e56)
|
||||
|
||||
## [dsh-v0.2.1-rc.3-r4] - 2026-10-26
|
||||
|
||||
### Added
|
||||
|
||||
- **skills:** Add agent-sandbox platform skill (#20) (bf24306)
|
||||
- **example:** Add optional gVisor, nodeSelector, NetworkPolicy and provider keys (#21) (3e9ba38)
|
||||
|
||||
### Maintenance
|
||||
|
||||
- **changelog:** Refresh for dsh-v0.2.1-rc.3-r3 (#19) (8681412)
|
||||
- **release:** Dsh-v0.2.1-rc.3-r4 (#22) (e2f1fe5)
|
||||
|
||||
## [dsh-v0.2.1-rc.3-r3] - 2026-10-25
|
||||
|
||||
### Added
|
||||
|
||||
- **gateway-api:** Expose dsh-gateway externally via Traefik Gateway API (#12) (7d3faf0)
|
||||
- **gateway-api:** GitHub SSO via oauth2-proxy in front of dsh-gateway (#13) (3f46279)
|
||||
- **gateway:** SSO landing auto-connects and offers recreate sandbox (#15) (ed1398d)
|
||||
- **repo:** Restructure around the dsh-harness image and one example per implementation (e6e109d)
|
||||
- **repo:** Repo reorganization (#16) (3b70f17)
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Dockerfile:** Update runtime dependencies and enable corepack for pnpm (6d97f61)
|
||||
- **gateway-api:** Correct oauth2-proxy cookie-secret generation (#14) (73a65d9)
|
||||
|
||||
### Maintenance
|
||||
|
||||
- Bump the actions group across 1 directory with 3 updates (#17) (6a8fe54)
|
||||
|
||||
### Other
|
||||
|
||||
- Dockerfile extra tools (2b47afa)
|
||||
- Modify gateway.yaml for image policy and timeout values (53e55fc)
|
||||
- Add commit message prefix for Dependabot updates (6876871)
|
||||
- Versioning update (#18) (ecba8b9)
|
||||
- Refactor environment variable loading in build workflow (d627526)
|
||||
- Refactor environment variable loading in build-gateway.yaml (027f74f)
|
||||
- Refactor release workflow for environment variable sourcing (958cbe2)
|
||||
|
||||
## [v0.0.1] - 2026-10-21
|
||||
|
||||
|
||||
@@ -43,8 +43,10 @@ Install the gateway dev dependencies once:
|
||||
- Use a Conventional Commit title, for example feat(gateway): add wake metrics
|
||||
or docs(examples): clarify SSO trust boundary. PR titles are linted.
|
||||
- Keep the diff scoped. Squash merge into main; history stays linear.
|
||||
- Required checks: lint, python, manifests, gateway-image, pr-title, CodeQL,
|
||||
and dependency-review. Branch protection is described in docs/gitflow.md.
|
||||
- Required on every PR: lint, pr-title, CodeQL (both Analyze jobs) and
|
||||
dependency-review. python, manifests and build-amd64 are path-filtered
|
||||
on-demand checks; run them when the change touches their scope. Branch
|
||||
protection is described in docs/gitflow.md.
|
||||
- Update the docs that the change affects; docs are part of the change, not a
|
||||
follow-up.
|
||||
- Never commit a real secret. Secret-shaped YAML belongs in a *.yaml.example
|
||||
@@ -60,16 +62,12 @@ it by hand; the release workflow regenerates it.
|
||||
|
||||
## Adding an example
|
||||
|
||||
Read docs/examples.md, then add examples/<name>/ with:
|
||||
|
||||
- README.md (what it demonstrates, prerequisites, steps, teardown)
|
||||
- manifests/NN-*.yaml (numbered, plain kubectl)
|
||||
- optional manifests/exposure/<variant>/ with its own README.md
|
||||
- optional components (a Dockerfile plus README and tests per component)
|
||||
- scripts/ with executable, idempotent, self-cleaning scripts
|
||||
- docs/ for deeper material
|
||||
|
||||
List it in examples/README.md. scripts/check-examples.sh enforces this layout.
|
||||
Read docs/examples.md, then add examples/<name>/ with a README.md,
|
||||
numbered manifests/NN-*.yaml, optional manifests/optional/<feature>/ or
|
||||
manifests/exposure/<variant>/ (each with a README), optional components (a
|
||||
Dockerfile plus README and tests), and executable, idempotent, self-cleaning
|
||||
scripts. Add the literal examples/<name>/ entry to examples/README.md;
|
||||
scripts/check-examples.sh enforces the layout.
|
||||
|
||||
## Documentation
|
||||
|
||||
|
||||
39
Dockerfile
39
Dockerfile
@@ -6,13 +6,14 @@
|
||||
# Build args (all pinned in versions.env, verified by scripts/check-versions.sh):
|
||||
# DSH_REPO upstream git remote to package
|
||||
# DSH_VERSION upstream git ref to build (tag or branch)
|
||||
# DSH_COMMIT full commit SHA DSH_VERSION must resolve to
|
||||
# RELEASE_REVISION optional repository revision label (r2, r3, ...)
|
||||
# PNPM_VERSION pnpm used to build DSH
|
||||
# VCS_REF this repository's commit, recorded in OCI labels
|
||||
# NODE_BUILD_IMAGE node image used to build DSH
|
||||
# NODE_RUNTIME_IMAGE node image used at runtime
|
||||
ARG NODE_BUILD_IMAGE=docker.io/node:24-trixie
|
||||
ARG NODE_RUNTIME_IMAGE=docker.io/node:24-trixie-slim
|
||||
# NODE_BUILD_IMAGE node image used to build DSH (pinned by digest)
|
||||
# NODE_RUNTIME_IMAGE node image used at runtime (pinned by digest)
|
||||
ARG NODE_BUILD_IMAGE=docker.io/node:24-trixie@sha256:8c761eca2bf7eec965f946cf47b8090143cb41f45f2f8699af0cc0298d2d2d19
|
||||
ARG NODE_RUNTIME_IMAGE=docker.io/node:24-trixie-slim@sha256:54f84b4cf8432b21aaffd5d3ce8d41e12cea0d10d33c09d2f3c9879d49d6b446
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Build stage: clone DSH at the pinned ref, build, and package the CLI.
|
||||
@@ -20,7 +21,8 @@ ARG NODE_RUNTIME_IMAGE=docker.io/node:24-trixie-slim
|
||||
FROM $NODE_BUILD_IMAGE AS build
|
||||
|
||||
ARG DSH_REPO=https://github.com/deepseek-ai/deepseek-harness.git
|
||||
ARG DSH_VERSION=dsh-v0.2.1-rc.3
|
||||
ARG DSH_VERSION=dsh-v0.2.1-rc.5
|
||||
ARG DSH_COMMIT=b830ec950faba0a46dda564be3e2d854afed94f1
|
||||
ARG PNPM_VERSION=11.28.5
|
||||
|
||||
ENV CI=true \
|
||||
@@ -31,7 +33,15 @@ RUN corepack enable \
|
||||
|
||||
WORKDIR /src
|
||||
|
||||
RUN git clone --depth 1 --branch "$DSH_VERSION" "$DSH_REPO" .
|
||||
# Clone the pinned ref and fail closed unless it resolves to DSH_COMMIT. A moved
|
||||
# tag would otherwise silently change what the image packages.
|
||||
RUN git clone --depth 1 --branch "$DSH_VERSION" "$DSH_REPO" . \
|
||||
&& actual="$(git -C /src rev-parse HEAD)" \
|
||||
&& if [ "$actual" != "$DSH_COMMIT" ]; then \
|
||||
echo "error: $DSH_REPO $DSH_VERSION resolved to $actual, expected DSH_COMMIT $DSH_COMMIT" >&2; \
|
||||
echo "error: update DSH_VERSION/DSH_COMMIT in versions.env with scripts/set-version.sh" >&2; \
|
||||
exit 1; \
|
||||
fi
|
||||
|
||||
RUN pnpm install --frozen-lockfile
|
||||
|
||||
@@ -57,8 +67,8 @@ RUN cd /opt/dsh \
|
||||
# ---------------------------------------------------------------------------
|
||||
FROM $NODE_RUNTIME_IMAGE AS runtime
|
||||
|
||||
ARG DSH_VERSION=dsh-v0.2.1-rc.3
|
||||
ARG RELEASE_REVISION=r3
|
||||
ARG DSH_VERSION=dsh-v0.2.1-rc.5
|
||||
ARG RELEASE_REVISION=r6
|
||||
ARG PNPM_VERSION=11.28.5
|
||||
ARG VCS_REF=unknown
|
||||
|
||||
@@ -79,6 +89,11 @@ ENV NODE_ENV=production \
|
||||
|
||||
# git/ripgrep/jq/yq/curl/gh are deliberate agent tooling parity: a sandboxed DSH
|
||||
# instance expects the usual developer CLI. tini reaps signals for a clean stop.
|
||||
# bubblewrap is DSH's first-choice command-sandbox runner on Linux. It ships
|
||||
# unsetuid, so it needs unprivileged user namespaces plus the namespace/mount
|
||||
# syscalls; a hardened pod (all capabilities dropped, RuntimeDefault seccomp)
|
||||
# usually cannot create one, which is why DSH then falls back to Landlock and,
|
||||
# when that is unavailable too, to DSH_PERMISSION_MODE=danger-full-access.
|
||||
RUN apt-get update \
|
||||
&& apt-get install -y --no-install-recommends \
|
||||
ca-certificates \
|
||||
@@ -90,8 +105,12 @@ RUN apt-get update \
|
||||
wget \
|
||||
tini \
|
||||
gh \
|
||||
bubblewrap \
|
||||
python3 \
|
||||
python3-pip \
|
||||
python3-venv \
|
||||
&& rm -rf /var/lib/apt/lists/* \
|
||||
&& curl -LsSf https://astral.sh/uv/install.sh | sh \
|
||||
&& apt-get autoclean -y \
|
||||
&& install -d -o node -g node -m 0755 /workspace "$DSH_HOME" \
|
||||
&& corepack enable \
|
||||
&& corepack prepare "pnpm@$PNPM_VERSION" --activate \
|
||||
@@ -144,6 +163,6 @@ WORKDIR /workspace
|
||||
USER node
|
||||
|
||||
EXPOSE 3080
|
||||
VOLUME ["/home/node/.dsh"]
|
||||
VOLUME ["/home/node/.dsh", "/workspace"]
|
||||
STOPSIGNAL SIGTERM
|
||||
ENTRYPOINT ["/usr/bin/tini", "--", "/usr/local/bin/dsh-entrypoint"]
|
||||
|
||||
1
Makefile
1
Makefile
@@ -27,6 +27,7 @@ help: ## Show available targets
|
||||
image: ## Build the main DSH Web image (no push)
|
||||
docker build \
|
||||
--build-arg DSH_VERSION=$(DSH_VERSION) \
|
||||
--build-arg DSH_COMMIT=$(DSH_COMMIT) \
|
||||
--build-arg RELEASE_REVISION=$(RELEASE_REVISION) \
|
||||
--build-arg PNPM_VERSION=$(PNPM_VERSION) \
|
||||
--build-arg VCS_REF=$(VCS_REF) \
|
||||
|
||||
65
README.md
65
README.md
@@ -20,7 +20,7 @@ docker run -d --name dsh-web \
|
||||
-p 3080:3080 \
|
||||
-v dsh-home:/home/node/.dsh \
|
||||
-e DEEPSEEK_API_KEY \
|
||||
ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.3-r3
|
||||
ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.5-r6
|
||||
```
|
||||
|
||||
DSH Web is **not usable at a bare `/`**. Without its launch-token cookie it
|
||||
@@ -45,7 +45,7 @@ docker run -d --name dsh-web -p 18081:3080 \
|
||||
-e DSH_PUBLIC_URL=http://localhost:18081 \
|
||||
-v dsh-home:/home/node/.dsh \
|
||||
-e DEEPSEEK_API_KEY \
|
||||
ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.3-r3
|
||||
ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.5-r6
|
||||
# then open http://localhost:18081/?token=...
|
||||
```
|
||||
|
||||
@@ -72,6 +72,13 @@ docker run -d --name dsh-web -p 18081:3080 \
|
||||
| `DSH_TRUSTED_HOSTS` | empty | Extra trusted `Host`/`Origin` entries, comma/space separated. |
|
||||
| `DSH_HOME` | `/home/node/.dsh` | DSH state directory (mount a volume here). |
|
||||
|
||||
DSH also resolves the standard per-provider environment names —
|
||||
`OPENROUTER_API_KEY`, `GEMINI_API_KEY`, `NVIDIA_API_KEY`, `OPENAI_API_KEY`,
|
||||
`ANTHROPIC_API_KEY`, `GROQ_API_KEY`, `XAI_API_KEY`, `HF_TOKEN`, and others —
|
||||
through its packaged provider catalog. Inject them the same way; a key saved in
|
||||
DSH's credential store wins over the environment. See
|
||||
[provider credentials](docs/main-image.md#provider-credentials).
|
||||
|
||||
Full contract, build args, probes, volumes and signals:
|
||||
[docs/main-image.md](docs/main-image.md).
|
||||
|
||||
@@ -87,21 +94,20 @@ ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway
|
||||
|
||||
| Tag | Contents | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `dsh-v0.2.1-rc.3-r3` | DSH Web, port `3080` | **Current release identity.** Immutable: once published it never moves. |
|
||||
| `dsh-v0.2.1-rc.5-r6` | DSH Web, port `3080` | **Current release identity.** Immutable: once published it never moves. |
|
||||
| `latest` | DSH Web | Most recent release; same digest as the current identity. |
|
||||
| `main` | DSH Web | Development channel rebuilt from every push to `main`. Not a release; not for production. |
|
||||
| `dsh-v0.2.1-rc.3` | DSH Web | **Legacy tag** published before the release-identity scheme. It is not updated. New deployments use `dsh-v0.2.1-rc.3-r3`. |
|
||||
| `main` | DSH Web | Development channel rebuilt when image-definition files change on `main`. Not a release; not for production. |
|
||||
|
||||
The example gateway is published under its nested path with the same release
|
||||
identity, e.g.
|
||||
`ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway:dsh-v0.2.1-rc.3-r3`.
|
||||
`ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway:dsh-v0.2.1-rc.5-r6`.
|
||||
|
||||
Release identities, channels and rollback rules are explained in
|
||||
[docs/versioning.md](docs/versioning.md). Production deployments should pin a
|
||||
digest:
|
||||
|
||||
```bash
|
||||
docker buildx imagetools inspect ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.3-r3
|
||||
docker buildx imagetools inspect ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.5-r6
|
||||
# image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox@sha256:<digest>
|
||||
```
|
||||
|
||||
@@ -125,12 +131,23 @@ Self-contained implementations that consume the image. Index and contract:
|
||||
|
||||
| Example | What it demonstrates |
|
||||
| --- | --- |
|
||||
| [`examples/dsh-agents-as-a-service/`](examples/dsh-agents-as-a-service/) | Full multi-tenant platform: SandboxTemplate with per-user PVCs, warm pool, one claim per user, suspend-when-idle, wake-on-return, browser gateway with signup + `HttpOnly` sessions + HTTP/WebSocket proxy, Ingress or Gateway API + GitHub SSO exposure. Start with the [walkthrough](examples/dsh-agents-as-a-service/README.md) and the [architecture](examples/dsh-agents-as-a-service/docs/architecture.md). |
|
||||
| [`examples/minimal-single-tenant/`](examples/minimal-single-tenant/) | The smallest useful setup: one `SandboxTemplate` and one `SandboxClaim`, reached through `kubectl port-forward` + the launch-token URL. Manifest: [`manifests/sandbox-template.yaml`](examples/minimal-single-tenant/manifests/sandbox-template.yaml). |
|
||||
| [`examples/dsh-agents-as-a-service/`](examples/dsh-agents-as-a-service/) | Full multi-tenant platform: SandboxTemplate with per-user PVCs, warm pool, one claim per user, suspend-when-idle, wake-on-return, browser gateway with signup + `HttpOnly` sessions + HTTP/WebSocket proxy, Ingress or Gateway API + GitHub SSO exposure, and opt-in gVisor/NetworkPolicy/provider-key hardening and failed-Pod recovery. Start with the [walkthrough](examples/dsh-agents-as-a-service/README.md) and the [architecture](examples/dsh-agents-as-a-service/docs/architecture.md). |
|
||||
| [`examples/minimal-single-tenant/`](examples/minimal-single-tenant/) | The smallest useful setup: one `SandboxTemplate` and one `SandboxClaim`, reached through `kubectl port-forward` + the launch-token URL. Manifest: [`manifests/10-sandboxtemplate.yaml`](examples/minimal-single-tenant/manifests/10-sandboxtemplate.yaml). |
|
||||
|
||||
The gateway is a component of the multi-tenant example and has its own
|
||||
[README](examples/dsh-agents-as-a-service/gateway/README.md) (API, configuration,
|
||||
auth/SSO, limitations).
|
||||
auth/SSO, limitations). The example also documents opt-in gVisor,
|
||||
`nodeSelector`, NetworkPolicy and extra provider-key hardening in
|
||||
[optional hardening](examples/dsh-agents-as-a-service/docs/optional-hardening.md).
|
||||
|
||||
## Agent skills
|
||||
|
||||
DSH discovers project skills from `.dsh/skills/<name>/SKILL.md`. The repository
|
||||
ships [`dsh-agent-sandbox-platform`](.dsh/skills/dsh-agent-sandbox-platform/SKILL.md),
|
||||
which covers operating and extending the platform, the optional
|
||||
[agent-sandbox MCP server](https://github.com/kubernetes-sigs/agent-sandbox/blob/main/clients/integrations/mcp-server/README.md),
|
||||
and a read-only cluster preflight. Runbooks, the MCP reference, the extension
|
||||
guide and troubleshooting notes live beside `SKILL.md`.
|
||||
|
||||
## Repository layout
|
||||
|
||||
@@ -144,8 +161,9 @@ auth/SSO, limitations).
|
||||
| [`examples/dsh-agents-as-a-service/`](examples/dsh-agents-as-a-service/) | Multi-tenant platform example: `manifests/`, `gateway/`, `docs/`, `scripts/`. |
|
||||
| [`examples/minimal-single-tenant/`](examples/minimal-single-tenant/) | Single-sandbox quickstart. |
|
||||
| [`scripts/`](scripts/) | Repository checks: versions, example layout, manifest YAML, version bump. |
|
||||
| [`.dsh/skills/`](.dsh/skills/) | Project-local DSH skills, auto-discovered from the git root. |
|
||||
| [`.github/`](.github/) | CI, release automation, dependency updates, PR/issue templates. |
|
||||
| [`.env.sample`](.env.sample) | Documented local configuration (copy to `.env`; never commit it). |
|
||||
| [`.env.sample`](.env.sample) | Reference environment names for local runs; nothing reads it automatically. Never commit a real key. |
|
||||
|
||||
## Development
|
||||
|
||||
@@ -170,34 +188,15 @@ required checks and releases: [docs/gitflow.md](docs/gitflow.md).
|
||||
|
||||
The release identity is the upstream DSH ref plus an optional repository
|
||||
revision: `DSH_VERSION` and `RELEASE_REVISION` in [versions.env](versions.env).
|
||||
The current identity is **`dsh-v0.2.1-rc.3-r3`** — the first release under the
|
||||
mirrored-DSH scheme at the same upstream DSH ref as the legacy
|
||||
`dsh-v0.2.1-rc.3` tag. Tags are immutable; a broken release is fixed by
|
||||
publishing `-r(N+1)`, never by re-tagging.
|
||||
The current identity is **`dsh-v0.2.1-rc.5-r6`** — the first release at this
|
||||
upstream DSH ref, so it carries no repository revision. Tags are immutable; a
|
||||
broken release is fixed by publishing `-r(N+1)`, never by re-tagging.
|
||||
|
||||
`CHANGELOG.md` is generated by [git-cliff](https://git-cliff.org) from
|
||||
Conventional Commits and is never edited by hand. See
|
||||
[`CHANGELOG.md`](CHANGELOG.md), [docs/versioning.md](docs/versioning.md) and
|
||||
[cliff.toml](cliff.toml).
|
||||
|
||||
## Migrating from the previous layout
|
||||
|
||||
If you are coming from an older checkout, these paths moved. Nothing else changed
|
||||
semantically; the content was split into the example it belongs to.
|
||||
|
||||
| Old path | New path |
|
||||
| --- | --- |
|
||||
| `gateway-api/dsh-demo/` (Gateway API + oauth2-proxy / GitHub SSO) | `examples/dsh-agents-as-a-service/manifests/exposure/gateway-api-sso/` (now `70-gateway.yaml` ... `74-tls-secret.yaml.example`) |
|
||||
| `examples/dsh-agents-as-a-service/60-ingress.yaml` | `examples/dsh-agents-as-a-service/manifests/exposure/ingress/60-ingress.yaml` |
|
||||
| `examples/dsh-agents-as-a-service/NN-*.yaml` (`00-` ... `50-`) | `examples/dsh-agents-as-a-service/manifests/NN-*.yaml` |
|
||||
| `examples/dsh-agents-as-a-service/run-test-kind.sh` | `examples/dsh-agents-as-a-service/scripts/e2e-kind.sh` |
|
||||
| `deploy/k8s/sandbox-template.yaml` | `examples/minimal-single-tenant/manifests/sandbox-template.yaml` |
|
||||
| `docs/architecture.md` | `examples/dsh-agents-as-a-service/docs/architecture.md` |
|
||||
|
||||
The SSO walkthrough that lived in `gateway-api/dsh-demo/README.md` is merged
|
||||
into the [multi-tenant example README](examples/dsh-agents-as-a-service/README.md)
|
||||
(step 9) and the [exposure README](examples/dsh-agents-as-a-service/manifests/exposure/README.md).
|
||||
|
||||
## Contributing
|
||||
|
||||
- [CONTRIBUTING.md](CONTRIBUTING.md) — local setup, PR workflow, what a change needs.
|
||||
|
||||
41
SECURITY.md
41
SECURITY.md
@@ -21,16 +21,41 @@ disclosure once a fix is available.
|
||||
from pod logs (pods/log). Anyone with access to those logs can impersonate a
|
||||
tenant. Keep the gateway's RBAC namespace-scoped and treat pod logs as
|
||||
sensitive. Replacing log scraping with an upstream token API is tracked in
|
||||
the example architecture document.
|
||||
the [example architecture document](examples/dsh-agents-as-a-service/docs/architecture.md#upstream-feature-request-remove-log-scraping).
|
||||
- **Browser sessions are server-side.** The gateway session cookie holds an
|
||||
opaque session id, not a user name or token; it is not accepted as a bearer
|
||||
token. Sessions are revoked on logout, user delete, `/recreate` and expiry,
|
||||
but they live in gateway memory, so a gateway restart logs every browser out
|
||||
and scaling beyond one replica requires a shared store. Do not treat the
|
||||
cookie as durable or cross-replica state.
|
||||
- **SSO header trust boundary.** AUTH_PROXY_HEADER makes the gateway trust an
|
||||
identity header (for example X-Forwarded-User). Enable it only when the
|
||||
gateway Service is reachable exclusively through the authenticating proxy;
|
||||
otherwise any client can spoof the header.
|
||||
- **Shared provider key.** The example injects one DEEPSEEK_API_KEY into every
|
||||
tenant sandbox (a warm-pool constraint). Treat tenants as mutually trusting
|
||||
until per-tenant credentials or an egress proxy exist.
|
||||
- **TLS is external.** Terminate TLS at an Ingress or Gateway and set
|
||||
COOKIE_SECURE=true. Plain HTTP outside localhost exposes session cookies.
|
||||
gateway Service is reachable exclusively through the authenticating proxy
|
||||
and its NetworkPolicy; otherwise any client can spoof the header. The SSO
|
||||
exposure variant's policy is required, not optional: an exposure without it
|
||||
is demo-only. Run `examples/dsh-agents-as-a-service/scripts/preflight-sso.sh`
|
||||
after applying the variant; it checks the Secrets, gateway env, TLS
|
||||
reference, leftover placeholders and accidental second paths.
|
||||
- **Insecure-cookie guard.** TLS terminates outside the gateway. Set
|
||||
COOKIE_SECURE=true for HTTPS and remove ALLOW_INSECURE_COOKIES; the gateway
|
||||
refuses to start with COOKIE_SECURE=false unless ALLOW_INSECURE_COOKIES=1 is
|
||||
explicitly set, which exists only for the plain-HTTP local demo. Never expose
|
||||
that combination outside localhost.
|
||||
- **Abuse controls are in-process.** AUTH_RATE_LIMIT/AUTH_RATE_WINDOW,
|
||||
SIGNUP_INVITE_CODE, MAX_IN_FLIGHT_PER_USER and WS_MAX_MESSAGE_BYTES bound
|
||||
casual abuse and resource exhaustion on the single gateway replica. They
|
||||
reset on restart and are not an ingress rate limiter, WAF, quota system or
|
||||
audit log; add those for an internet-facing deployment.
|
||||
- **Shared provider keys.** The example injects DEEPSEEK_API_KEY and any keys in
|
||||
the optional dsh-provider-keys-extra Secret into every tenant sandbox (a
|
||||
warm-pool constraint). Treat tenants as mutually trusting until per-tenant
|
||||
credentials or an egress proxy exist.
|
||||
- **The base example is demo-only, not hardened.** gVisor, nodeSelector, the
|
||||
sandbox/exposure NetworkPolicies, the gateway egress policy and extra provider
|
||||
keys are opt-in. Without gVisor and the network policies there is no
|
||||
tenant-to-tenant isolation; the sandboxes share the host kernel and can reach
|
||||
cluster-internal endpoints. Production requirements are enumerated in the
|
||||
[production checklist](examples/dsh-agents-as-a-service/docs/optional-hardening.md#6-production-checklist).
|
||||
- **No secrets in the repository.** The repository only ships empty Secret
|
||||
manifests and *.yaml.example files.
|
||||
|
||||
|
||||
@@ -44,6 +44,6 @@ commit_parsers = [
|
||||
]
|
||||
protect_breaking_commits = false
|
||||
filter_commits = false
|
||||
# Release tags are dsh-vX.Y.Z[-rN]; the single legacy pre-scheme tag is v0.0.1.
|
||||
# Release tags are dsh-vX.Y.Z[-rN]; the single pre-scheme tag is v0.0.1.
|
||||
tag_pattern = "^(v|dsh-v).*$"
|
||||
sort_commits = "oldest"
|
||||
|
||||
@@ -2,8 +2,7 @@
|
||||
|
||||
Repository-wide documentation lives here. Example-specific material lives with
|
||||
its example under [`examples/<name>/docs/`](../examples/) — see the
|
||||
[agents-as-a-service architecture](../examples/dsh-agents-as-a-service/docs/architecture.md)
|
||||
for the deep-dive that used to live in this directory.
|
||||
[agents-as-a-service architecture](../examples/dsh-agents-as-a-service/docs/architecture.md).
|
||||
|
||||
If you are new, read the [root README](../README.md) first: it is image-first and
|
||||
gets a DSH Web container running before anything else.
|
||||
@@ -15,7 +14,17 @@ gets a DSH Web container running before anything else.
|
||||
| [main-image.md](main-image.md) | The primary artifact: build args, runtime environment, entrypoint, probes, volumes/ports/signals, non-root contract, image names, tags and digests. |
|
||||
| [examples.md](examples.md) | The example contract: required layout, manifest rules, exposure variants, components and image naming, scripts, docs, and how to add a new example. |
|
||||
| [gitflow.md](gitflow.md) | Branches, protected `main`, squash merges, Conventional PR titles, required CI checks, release branches, hotfixes and the one-time GitHub settings checklist. |
|
||||
| [versioning.md](versioning.md) | Release identity (`DSH ref` + optional `-rN`), why the current identity is `dsh-v0.2.1-rc.3-r3`, channels, compatibility matrix, bumping pins, the release checklist, rollback, git-cliff and digest pinning. |
|
||||
| [versioning.md](versioning.md) | Release identity (`DSH ref` + optional `-rN`), the current identity `dsh-v0.2.1-rc.5-r6`, why identities are immutable, channels, compatibility matrix, bumping pins, the release checklist, rollback, git-cliff and digest pinning. |
|
||||
| [workflow-security.md](workflow-security.md) | GitHub Actions security posture: least-privilege tokens, pinned actions, untrusted-input handling, download integrity checks, artifact hygiene, the latest audit record and residual repository-setting recommendations. |
|
||||
|
||||
## Project skills
|
||||
|
||||
DSH discovers project skills from `<git root>/.dsh/skills/<name>/SKILL.md`
|
||||
(the nearest `.git` ancestor). This repository ships one:
|
||||
|
||||
| Skill | Read it for |
|
||||
| --- | --- |
|
||||
| [dsh-agent-sandbox-platform](../.dsh/skills/dsh-agent-sandbox-platform/SKILL.md) | Operating and extending the agents-as-a-service platform, agent-sandbox MCP limits and deployment, and a read-only preflight. |
|
||||
|
||||
## Example documentation
|
||||
|
||||
@@ -25,6 +34,8 @@ gets a DSH Web container running before anything else.
|
||||
| [dsh-agents-as-a-service README](../examples/dsh-agents-as-a-service/README.md) | The multi-tenant platform walkthrough: warm pool, claims, suspend/resume, gateway, exposure and troubleshooting. |
|
||||
| [dsh-agents-as-a-service architecture](../examples/dsh-agents-as-a-service/docs/architecture.md) | Container/sequence diagrams, ADRs, security and RBAC, failure modes and the evolution path. |
|
||||
| [gateway README](../examples/dsh-agents-as-a-service/gateway/README.md) | The gateway's HTTP/WebSocket API, configuration, auth (including SSO), and limitations. |
|
||||
| [optional hardening](../examples/dsh-agents-as-a-service/docs/optional-hardening.md) | Opt-in gVisor, nodeSelector, NetworkPolicy, provider keys and failed-Pod recovery. |
|
||||
| [exposure variants](../examples/dsh-agents-as-a-service/manifests/exposure/README.md) | The mutually exclusive Ingress and Gateway API + SSO exposure paths. |
|
||||
|
||||
## Project policies
|
||||
|
||||
|
||||
@@ -18,6 +18,9 @@ examples/<name>/
|
||||
| # ordered steps, troubleshooting, cleanup
|
||||
+-- manifests/ # required
|
||||
| +-- NN-*.yaml # numbered top-level manifests, plain kubectl
|
||||
| +-- optional/ # optional opt-in manifests, applied by path
|
||||
| +-- README.md # required when optional/ exists
|
||||
| +-- <feature>/NN-*.yaml # e.g. network-policy/
|
||||
| +-- exposure/ # optional exposure variants
|
||||
| +-- README.md # required when exposure/ exists
|
||||
| +-- <variant>/NN-*.yaml # e.g. ingress/, gateway-api-sso/
|
||||
@@ -38,9 +41,12 @@ examples/<name>/
|
||||
example directory;
|
||||
- an example has no `README.md` or no `manifests/` directory;
|
||||
- `manifests/` contains no top-level `*.yaml` files;
|
||||
- a top-level `manifests/*.yaml` is not named `NN-*.yaml`;
|
||||
- a `*.yaml` file sits at the example root instead of under `manifests/`;
|
||||
- `manifests/exposure/` exists without a `manifests/exposure/README.md` describing
|
||||
the variants;
|
||||
- `manifests/optional/` exists without a `manifests/optional/README.md`
|
||||
describing the opt-in manifests;
|
||||
- a subdirectory with a `Dockerfile` has no `README.md`;
|
||||
- a file in `scripts/` is not executable.
|
||||
|
||||
@@ -68,7 +74,12 @@ make check
|
||||
command; real values are generated per install.
|
||||
- **Cluster-specific state stays out.** Node affinity, replica counts tuned for a
|
||||
cluster, and similar fields belong in a local patch, not in the shipped
|
||||
manifest.
|
||||
manifest. Optional scheduler/runtime hints may ship **commented out** with
|
||||
placeholder labels; they must not be active.
|
||||
- **Optional manifests are opt-in.** Put them under
|
||||
`manifests/optional/<feature>/`, document them in
|
||||
`manifests/optional/README.md`, and apply them by path. The top-level
|
||||
`kubectl apply -f manifests/` is non-recursive and does not apply them.
|
||||
- **Placeholders are explicit.** Hostnames, GitHub orgs and TLS material use
|
||||
values like `dsh.example.com` / `your-github-org` with a TODO comment.
|
||||
|
||||
|
||||
@@ -30,7 +30,7 @@ branch; the PR is squash-merged, so individual branch commits are never part of
|
||||
`docs(scope): ...`, `build: ...`, `ci: ...`, `refactor: ...`, `test: ...`,
|
||||
`chore: ...`. Breaking changes use `feat(scope)!: ...` or a
|
||||
`BREAKING CHANGE:` footer.
|
||||
3. **CI must be green.** All required checks below must pass before merge.
|
||||
3. **CI must be green.** All applicable checks below must pass before merge.
|
||||
4. **Squash merge only.** `main` stays linear and protected; merged PRs are one
|
||||
commit each.
|
||||
5. **No direct pushes to `main`** or to a release branch. Both are protected.
|
||||
@@ -40,15 +40,33 @@ branch; the PR is squash-merged, so individual branch commits are never part of
|
||||
|
||||
## Required checks
|
||||
|
||||
| Check | Workflow / job | What it verifies |
|
||||
| --- | --- | --- |
|
||||
| `lint` | `ci.yaml` | `scripts/check-versions.sh` (version pins and markdown tokens), ruff on the gateway, shellcheck on shell scripts, yamllint/actionlint when available. |
|
||||
| `python` | `ci.yaml` | Gateway unit tests (`pytest`). |
|
||||
| `manifests` | `ci.yaml` | `scripts/check-examples.sh`, `scripts/check-manifests.py` and a server-side dry-run of every example manifest against the agent-sandbox CRDs. |
|
||||
| `gateway-image` | `ci.yaml` | Builds the gateway image and smoke-tests `/healthz` without a cluster. |
|
||||
| `pr-title` | `pr-title.yaml` | Conventional Commit PR title. |
|
||||
| `CodeQL` | `codeql.yml` | Static analysis for Python and GitHub Actions. |
|
||||
| `dependency-review` | `ci.yaml` | Dependency changes introduced by the PR (job `dependency-review`; runs on pull requests only). |
|
||||
| Check | Workflow / job | Required on every PR? | What it verifies |
|
||||
| --- | --- | --- | --- |
|
||||
| `lint` | `ci.yaml` | yes | `scripts/check-versions.sh` (version pins and markdown tokens), `scripts/check-examples.sh` (example layout and numbered manifests), shellcheck, actionlint, yamllint, hadolint and documentation links. |
|
||||
| `python` | `ci-python.yaml` | no (path-filtered) | Gateway ruff check/format and unit tests (`pytest` with coverage). Runs only when the gateway or this workflow changes. |
|
||||
| `manifests` | `ci-manifests.yaml` | no (path-filtered) | `scripts/check-manifests.py` (manifest shape) and a server-side dry-run of every example manifest against the agent-sandbox CRDs. Runs when manifest files, `scripts/check-manifests.py` or `versions.env` change. |
|
||||
| `pr-title` | `pr-title.yaml` | yes | Conventional Commit PR title. |
|
||||
| `Analyze (python)`, `Analyze (actions)` | `codeql.yml` | yes (both jobs) | CodeQL static analysis for Python and GitHub Actions. |
|
||||
| `dependency-review` | `ci.yaml` | yes | Dependency changes introduced by the PR (job `dependency-review`; runs on pull requests only). |
|
||||
|
||||
`python`, `manifests` and `build-amd64` use native `on.pull_request.paths`
|
||||
filtering, so PRs that do not touch the gateway, the manifests or the image
|
||||
definition skip them. The gateway image is built and published by
|
||||
`build-gateway.yaml` on merge; `/healthz` is covered by the gateway unit tests and
|
||||
the Dockerfile is linted by `lint` (hadolint).
|
||||
|
||||
`ci-image.yaml` (`build-amd64`) is the PR-side check for the main image: when
|
||||
`Dockerfile`, `.dockerignore` or `versions.env` change it builds `linux/amd64`
|
||||
with the pins from `versions.env` — no registry login, no push — and smoke-tests
|
||||
the result (`--version`, non-root UID, `/workspace` + `DSH_HOME`, the
|
||||
`bubblewrap` runner, the version label, and `/favicon.svg` served from a running
|
||||
container). The published multi-arch build still happens only in
|
||||
`build-main.yaml` on merge.
|
||||
|
||||
Because a skipped path-filtered check is not a passing check, require only the
|
||||
always-on set: `lint`, `pr-title`, both CodeQL `Analyze` jobs and
|
||||
`dependency-review`. Keep `python`, `manifests` and `build-amd64` as on-demand
|
||||
checks that skip when unrelated to the change.
|
||||
|
||||
The image build workflows (`_build-image.yaml`, `build-main.yaml`,
|
||||
`build-gateway.yaml`) and `e2e-kind.yaml` are **not** required on every PR: the
|
||||
@@ -90,7 +108,8 @@ Rollback is also a re-release, never a retag: see
|
||||
|
||||
## Tag protection
|
||||
|
||||
All release tags match `dsh-*` and are protected:
|
||||
Release tags follow `dsh-*` and are protected (the pre-scheme `v0.0.1` tag is
|
||||
historical and unmaintained):
|
||||
|
||||
- only maintainers (the release workflow) may create them;
|
||||
- they cannot be deleted or force-updated;
|
||||
@@ -104,9 +123,10 @@ respectively.
|
||||
|
||||
Configure these on the repository before relying on the workflow:
|
||||
|
||||
- [ ] **Branches → main**: require a pull request; require the checks `lint`,
|
||||
`python`, `manifests`, `gateway-image`, `pr-title`, `CodeQL`,
|
||||
`dependency-review`; require branches to be up to date; require conversation
|
||||
- [ ] **Branches → main**: require a pull request; require the always-on checks
|
||||
`lint`, `pr-title`, both CodeQL `Analyze` jobs and `dependency-review`
|
||||
(`python`, `manifests` and `build-amd64` are path-filtered and must not be
|
||||
required); require branches to be up to date; require conversation
|
||||
resolution; restrict direct pushes; allow only squash merges and enable
|
||||
"Automatically delete head branches".
|
||||
- [ ] **Branches → release/***: same protections, plus allow the release workflow
|
||||
|
||||
@@ -11,14 +11,19 @@ The image is image-first: it is the product, and the examples under
|
||||
|
||||
## What the build does
|
||||
|
||||
The Dockerfile is a two-stage build:
|
||||
The Dockerfile is a two-stage build. Both stages use the `node` images pinned by
|
||||
digest in [versions.env](../versions.env): a mutable tag alone is not trusted, and
|
||||
`./scripts/check-versions.sh` rejects an image reference without a `@sha256:` digest.
|
||||
|
||||
1. **Build stage** (`node:24-trixie`) clones DSH at the pinned git ref, installs
|
||||
1. **Build stage** (`NODE_BUILD_IMAGE`) clones DSH at the pinned git ref, verifies
|
||||
that the clone's `HEAD` is exactly `DSH_COMMIT` (the build fails otherwise, so a
|
||||
moved upstream tag cannot silently change what the image packages), installs
|
||||
dependencies with pnpm, runs the production build, packages the `@deepseek-ai/dsh`
|
||||
CLI with `pnpm pack`, and installs its production dependencies with npm.
|
||||
2. **Runtime stage** (`node:24-trixie-slim`) copies `/opt/dsh` to `/app`, adds the
|
||||
2. **Runtime stage** (`NODE_RUNTIME_IMAGE`) copies `/opt/dsh` to `/app`, adds the
|
||||
CLI tools DSH agents commonly use (`git`, `ripgrep`, `jq`, `yq`, `curl`, `wget`,
|
||||
`gh`, `tini`, `uv`), and starts `dsh web` through a small entrypoint.
|
||||
`gh`, `tini`, `uv`), installs `bubblewrap` as DSH's preferred command-sandbox
|
||||
runner, and starts `dsh web` through a small entrypoint.
|
||||
|
||||
## Build it
|
||||
|
||||
@@ -27,15 +32,19 @@ make image # canonical: builds the DSH Web image, no push
|
||||
```
|
||||
|
||||
`make image` reads [versions.env](../versions.env) and passes the pins as build args.
|
||||
Pull requests that touch `Dockerfile`, `.dockerignore` or `versions.env` get the
|
||||
same build for `linux/amd64` from the [`ci-image.yaml`](../.github/workflows/ci-image.yaml)
|
||||
workflow — no registry login, no push — followed by a container smoke test.
|
||||
To reproduce the equivalent by hand:
|
||||
|
||||
```bash
|
||||
docker build \
|
||||
--build-arg DSH_VERSION=dsh-v0.2.1-rc.3 \
|
||||
--build-arg RELEASE_REVISION=r2 \
|
||||
--build-arg DSH_VERSION=dsh-v0.2.1-rc.5-r6 \
|
||||
--build-arg DSH_COMMIT=b830ec950faba0a46dda564be3e2d854afed94f1 \
|
||||
--build-arg RELEASE_REVISION= \
|
||||
--build-arg PNPM_VERSION=11.28.5 \
|
||||
--build-arg VCS_REF="$(git rev-parse --short HEAD)" \
|
||||
-t ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.3-r3 .
|
||||
-t ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.5-r6 .
|
||||
```
|
||||
|
||||
### Build args
|
||||
@@ -43,12 +52,18 @@ docker build \
|
||||
| Arg | Default / source | Purpose |
|
||||
| --- | --- | --- |
|
||||
| `DSH_REPO` | `https://github.com/deepseek-ai/deepseek-harness.git` | Upstream DSH repository to clone. |
|
||||
| `DSH_VERSION` | `DSH_VERSION` in [versions.env](../versions.env) | Upstream DSH git tag/ref to package; recorded as `org.opencontainers.image.version`. |
|
||||
| `RELEASE_REVISION` | `RELEASE_REVISION` in [versions.env](../versions.env) | Repository revision suffix (`rN`); the release identity is `DSH_VERSION-RELEASE_REVISION`, recorded in the `io.sredevopsorg.release.revision` label. See [versioning](versioning.md). |
|
||||
| `DSH_VERSION` | `DSH_VERSION` in [versions.env](../versions.env) | Upstream DSH git tag/ref to package. The Dockerfile records it as `io.sredevopsorg.dsh.version` and (for local builds) `org.opencontainers.image.version`; published builds set the version label to the full release identity. |
|
||||
| `DSH_COMMIT` | `DSH_COMMIT` in [versions.env](../versions.env) | Full commit SHA `DSH_VERSION` must resolve to upstream. The build stage fails unless `git rev-parse HEAD` in the clone equals this value, which pins the source independently of a movable tag. |
|
||||
| `RELEASE_REVISION` | `RELEASE_REVISION` in [versions.env](../versions.env) | Repository revision suffix (`rN`), recorded in `io.sredevopsorg.release.revision`. The release identity is `DSH_VERSION` plus `-RELEASE_REVISION` when set. See [versioning](versioning.md). |
|
||||
| `PNPM_VERSION` | `PNPM_VERSION` in [versions.env](../versions.env) | pnpm version activated with corepack in both stages. |
|
||||
| `VCS_REF` | `git rev-parse --short HEAD` (via `make image`) | Source revision recorded for traceability. |
|
||||
| `NODE_BUILD_IMAGE` | `docker.io/node:24-trixie` | Build-stage base image. |
|
||||
| `NODE_RUNTIME_IMAGE` | `docker.io/node:24-trixie-slim` | Runtime-stage base image. |
|
||||
| `NODE_BUILD_IMAGE` | `docker.io/node:24-trixie@sha256:8c761eca2bf7eec965f946cf47b8090143cb41f45f2f8699af0cc0298d2d2d19` | Build-stage base image, pinned to the multi-arch index digest. |
|
||||
| `NODE_RUNTIME_IMAGE` | `docker.io/node:24-trixie-slim@sha256:54f84b4cf8432b21aaffd5d3ce8d41e12cea0d10d33c09d2f3c9879d49d6b446` | Runtime-stage base image, pinned to the multi-arch index digest. |
|
||||
|
||||
Base images are pinned by digest because tags are mutable: `node:24-trixie` can be
|
||||
re-pointed at a new build at any time. Bumping the tag and its digest is a
|
||||
deliberate, reviewable change in [versions.env](../versions.env); environments that
|
||||
mirror the image should also mirror the `node` digest they build from.
|
||||
|
||||
Never hard-code a version outside [versions.env](../versions.env):
|
||||
`./scripts/check-versions.sh` (and `make check`) fails when a pin drifts.
|
||||
@@ -76,17 +91,69 @@ The entrypoint is `tini -- /usr/local/bin/dsh-entrypoint`:
|
||||
| `DSH_TRUSTED_HOSTS` | empty | Extra `Host`/`Origin` allow-list entries, comma- or space-separated. `localhost`/`127.0.0.1` are always trusted. Required when a proxy forwards the browser's `Host`. |
|
||||
| `DEEPSEEK_API_KEY` | empty | Provider credential read by DSH. Inject it with `-e`/`envFrom`; never bake it into the image. Only model calls fail when it is missing. |
|
||||
| `DSH_HOME` | `/home/node/.dsh` | DSH state directory (conversations, sessions, config, credentials). Mount a volume here to persist it. |
|
||||
| `HOME` | `/home/node` | OS home: where `~` resolves and where tools keep caches and config. DSH's directory picker has no start-path setting, so it lists `$HOME` when opened. The shipped sandbox templates override it to `/workspace` so the picker opens on the persisted workspace; the trade-offs are below. |
|
||||
| `DSH_PERMISSION_MODE` | `workspace-write` | Deployment default for DSH's file policy: `read-only`, `workspace-write`, or `danger-full-access` (which also turns approval prompts off). The confined modes need `bubblewrap` or Landlock inside the container; where neither is usable every bash command fails closed with `SANDBOX_UNAVAILABLE`. See the note below. |
|
||||
|
||||
The image also sets `NODE_ENV=production`, `HOME=/home/node`, and a `PATH` that
|
||||
includes `/app/node_modules/.bin`.
|
||||
|
||||
`HOME` decides where DSH's folder picker starts. DSH has no start-path setting:
|
||||
the picker's browse backend lists `$HOME` when the dialog opens and labels that
|
||||
listing *Home*. With the image default (`/home/node`) that listing is outside a
|
||||
mounted volume; the example sandbox templates
|
||||
([minimal](../examples/minimal-single-tenant/manifests/10-sandboxtemplate.yaml),
|
||||
[agents as a service](../examples/dsh-agents-as-a-service/manifests/10-sandboxtemplate.yaml))
|
||||
mount the same PVC's `workspace` subPath at `/workspace` and set `HOME=/workspace`
|
||||
so the picker opens on the persisted agent workspace. `~`
|
||||
and every tool that keeps caches or config under `$HOME` (`npm`/`pnpm`, `uv`,
|
||||
`git`, `gh`, DSH's global `~/.agents` instructions and skills) follow, so those
|
||||
dotfiles then live on the workspace volume too. `DSH_HOME` is separate and stays
|
||||
`/home/node/.dsh`, so sessions, config and credentials stay out of the
|
||||
workspace. An existing sandbox stamps this environment at pod creation, so the
|
||||
override only applies to sandboxes created after it was added.
|
||||
|
||||
DSH confines bash commands itself (a second layer inside whatever container
|
||||
sandbox you already run). Its Linux runners are `bubblewrap` and then Landlock,
|
||||
and it fails closed rather than running unconfined. The image ships
|
||||
`bubblewrap`, but Debian's binary is unsetuid: it needs **unprivileged user
|
||||
namespaces** plus the namespace/mount syscalls. A hardened pod — non-root,
|
||||
`capabilities.drop: ["ALL"]`, `seccompProfile: RuntimeDefault` — usually cannot
|
||||
create one, and it may expose no Landlock runner either, in which case every
|
||||
command fails with `SANDBOX_UNAVAILABLE`. Set
|
||||
`DSH_PERMISSION_MODE=danger-full-access` only when the surrounding pod or VM is
|
||||
the isolation boundary you rely on; keep the confined modes when the host
|
||||
provides a working runner (check `grep -o landlock /sys/kernel/security/lsm` and
|
||||
`bwrap --ro-bind / / true` inside the container first).
|
||||
|
||||
### Provider credentials
|
||||
|
||||
DSH resolves provider API keys from the environment when no stored credential
|
||||
exists. The packaged provider catalog maps each provider to its standard
|
||||
environment name. Commonly used names:
|
||||
|
||||
`DEEPSEEK_API_KEY`, `OPENROUTER_API_KEY`, `GEMINI_API_KEY`, `NVIDIA_API_KEY`,
|
||||
`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GROQ_API_KEY`, `XAI_API_KEY`,
|
||||
`MISTRAL_API_KEY`, `TOGETHER_API_KEY`, `CEREBRAS_API_KEY`,
|
||||
`AI_GATEWAY_API_KEY`, `HF_TOKEN`.
|
||||
|
||||
Use `GEMINI_API_KEY` for Google Gemini (not `GOOGLE_API_KEY`), `HF_TOKEN` for
|
||||
Hugging Face, and `AI_GATEWAY_API_KEY` for the Vercel AI Gateway. Inject keys
|
||||
with `-e`/`--env-file` in Docker or a Secret + `envFrom` in Kubernetes; never
|
||||
bake them into the image.
|
||||
|
||||
A key saved through DSH's credential store (`$DSH_HOME/.credentials.yaml`) wins
|
||||
over the environment. The
|
||||
[agents-as-a-service example](../examples/dsh-agents-as-a-service/docs/optional-hardening.md)
|
||||
documents the optional `dsh-provider-keys-extra` Secret and the pod-refresh
|
||||
procedure.
|
||||
|
||||
### Ports, volumes and signals
|
||||
|
||||
| Item | Value | Notes |
|
||||
| --- | --- | --- |
|
||||
| Exposed port | `3080/tcp` | `EXPOSE 3080`; one port serves the UI, assets, API, SSE and WebSocket traffic. |
|
||||
| State volume | `/home/node/.dsh` | Declared with `VOLUME`; use a named volume (Docker) or PVC (Kubernetes). |
|
||||
| Working directory | `/workspace` | The agent workspace; mount a volume here if agents should keep files. |
|
||||
| Working directory | `/workspace` | The agent workspace. Mount a volume here if agents should keep files; the shipped example templates mount the same PVC's `workspace` subPath here and point `HOME` at it, so DSH's folder picker opens on it. |
|
||||
| Stop signal | `SIGTERM` | `tini` forwards it to `dsh web`; allow a graceful shutdown period. |
|
||||
| User | `node`, UID/GID `1000` | The process is non-root. A PVC mounted at `DSH_HOME` must be writable by UID 1000. |
|
||||
|
||||
@@ -109,8 +176,8 @@ livenessProbe:
|
||||
periodSeconds: 15
|
||||
```
|
||||
|
||||
The shipped templates use exactly this shape; see
|
||||
[`examples/minimal-single-tenant/manifests/sandbox-template.yaml`](../examples/minimal-single-tenant/manifests/sandbox-template.yaml).
|
||||
The shipped templates use the same path and startup budget; see
|
||||
[`examples/minimal-single-tenant/manifests/10-sandboxtemplate.yaml`](../examples/minimal-single-tenant/manifests/10-sandboxtemplate.yaml).
|
||||
A `startupProbe` is important because plugin activation (or the first start of a
|
||||
large sandbox) can exceed a default liveness budget.
|
||||
|
||||
@@ -142,17 +209,16 @@ GitHub Container Registry:
|
||||
|
||||
| Tag | Meaning |
|
||||
| --- | --- |
|
||||
| `dsh-v0.2.1-rc.3-r3` | Current release identity. **Immutable**: once published it never moves. |
|
||||
| `dsh-v0.2.1-rc.5-r6` | Current release identity. **Immutable**: once published it never moves. |
|
||||
| `latest` | Most recent release; points at the same digest as the current identity. |
|
||||
| `main` | Development channel rebuilt from every push to `main`; not a release, not for production. |
|
||||
| `dsh-v0.2.1-rc.3` | Legacy tag published before the release-identity scheme existed. Do not use it for new deployments; see the migration table in the [root README](../README.md). |
|
||||
| `main` | Development channel rebuilt when image-definition files change on `main`; not a release, not for production. |
|
||||
|
||||
Release identities, channels and rollback rules are defined in
|
||||
[versioning](versioning.md). Tags are immutable, so production deployments should
|
||||
pin a digest:
|
||||
|
||||
```bash
|
||||
docker buildx imagetools inspect ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.3-r3
|
||||
docker buildx imagetools inspect ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.5-r6
|
||||
# ... pin the digest in your deployment manifests:
|
||||
# image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox@sha256:<digest>
|
||||
```
|
||||
@@ -160,9 +226,10 @@ docker buildx imagetools inspect ghcr.io/sredevopsorg/deepseek-kubernetes-sandbo
|
||||
## Verifying an image
|
||||
|
||||
```bash
|
||||
docker inspect ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.3-r3 \
|
||||
docker inspect ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.5-r6 \
|
||||
--format '{{json .Config.Labels}}'
|
||||
# org.opencontainers.image.title / .version / .source
|
||||
# io.sredevopsorg.dsh.version / io.sredevopsorg.release.revision
|
||||
```
|
||||
|
||||
Release builds also publish SBOM and provenance attestations from the release
|
||||
|
||||
@@ -15,48 +15,40 @@ release identity = DSH_VERSION + optional "-" + RELEASE_REVISION
|
||||
|
||||
| Piece | Meaning |
|
||||
| --- | --- |
|
||||
| `DSH_VERSION` | The upstream DeepSeek Harness git ref being packaged, e.g. `dsh-v0.2.1-rc.3`. It mirrors upstream's own tag, which is what "mirrored-DSH scheme" means. |
|
||||
| `DSH_VERSION` | The upstream DeepSeek Harness git ref being packaged, e.g. `dsh-v0.2.1-rc.5-r6`. It mirrors upstream's own tag. |
|
||||
| `RELEASE_REVISION` | An optional repository revision at that DSH ref: empty for the first release, `rN` (`N >= 2`) for a repository-only re-release. |
|
||||
|
||||
Examples:
|
||||
|
||||
- `DSH_VERSION=dsh-v0.2.1-rc.3`, `RELEASE_REVISION=` → identity `dsh-v0.2.1-rc.3`
|
||||
- `DSH_VERSION=dsh-v0.2.1-rc.5-r6`, `RELEASE_REVISION=` → identity `dsh-v0.2.1-rc.5-r6`
|
||||
(first release at that upstream ref).
|
||||
- `DSH_VERSION=dsh-v0.2.1-rc.3`, `RELEASE_REVISION=r2` → identity
|
||||
`dsh-v0.2.1-rc.3-r3` (documentation, CI or gateway changes on top of the same
|
||||
upstream DSH; no DSH upgrade).
|
||||
- `DSH_VERSION=<ref>`, `RELEASE_REVISION=rN` → identity `<ref>-rN` (for example
|
||||
documentation, CI or gateway changes on top of an already-released ref; no DSH
|
||||
upgrade). Use `r2` for the first re-release: revision 1 is the plain tag.
|
||||
|
||||
A DSH upgrade produces a new base identity (for example the next upstream ref
|
||||
with an empty revision); a repository-only change at the same DSH ref increments
|
||||
`rN`.
|
||||
|
||||
## Why the current identity is `dsh-v0.2.1-rc.3-r3`
|
||||
## Why release identities are immutable
|
||||
|
||||
The image tag `dsh-v0.2.1-rc.3` was already published to GHCR for an earlier
|
||||
repository state. **Container tags are immutable** in this project: re-pushing a
|
||||
tag would silently change what an existing deployment runs. The mirrored-DSH
|
||||
scheme therefore starts its first identity at `-r2`: the plain tag is
|
||||
conceptually revision 1 and is taken, so the first publishable repository
|
||||
revision at the same upstream ref is `r2`.
|
||||
**Container tags are immutable** in this project: re-pushing a tag would silently
|
||||
change what an existing deployment runs. A fix therefore publishes a new identity;
|
||||
see the rules above.
|
||||
|
||||
The identity is stored in [versions.env](../versions.env) and consumed everywhere:
|
||||
the Dockerfile label, the example manifests, the Makefile and the release
|
||||
workflow. The legacy `dsh-v0.2.1-rc.3` tag remains published but is not updated;
|
||||
new deployments use `dsh-v0.2.1-rc.3-r3`.
|
||||
|
||||
The gateway image moved from the repository-level `:gateway` tag to a nested
|
||||
package (`<repository>/dsh-agents-as-a-service/gateway`). For exactly one
|
||||
release the release workflow also republishes the legacy
|
||||
`<repository>:gateway` tag as a deprecated alias; it is removed in the next
|
||||
release, so deployments should pin the nested package.
|
||||
The current identity is `dsh-v0.2.1-rc.5-r6`, the first release at this upstream
|
||||
ref, so it carries no revision suffix. The identity is stored in
|
||||
[versions.env](../versions.env) and consumed everywhere: the Dockerfile labels
|
||||
(version and revision), the example manifests, the Makefile and the release
|
||||
workflow.
|
||||
|
||||
## Channels
|
||||
|
||||
| Channel | Image tag | Meaning |
|
||||
| --- | --- | --- |
|
||||
| Release | `dsh-v0.2.1-rc.3-r3` | The current immutable release identity. What production should pin (preferably by digest). |
|
||||
| Release | `dsh-v0.2.1-rc.5-r6` | The current immutable release identity. What production should pin (preferably by digest). |
|
||||
| `latest` | `latest` | Alias for the most recent release; moves only when a new release is published. |
|
||||
| `main` | `main` | Development channel, rebuilt from every push to `main`. Never treat it as a release; it exists so `main` can be tested. |
|
||||
| `main` | `main` | Development channel, rebuilt when image or gateway sources change on `main`. Never treat it as a release; it exists so `main` can be tested. |
|
||||
|
||||
The same channels exist for every published package, including the nested
|
||||
example component image.
|
||||
@@ -68,15 +60,28 @@ Values from [versions.env](../versions.env) at the current identity:
|
||||
| Component | Pin | Source |
|
||||
| --- | --- | --- |
|
||||
| Upstream DSH source | `DSH_REPO` | [versions.env](../versions.env) |
|
||||
| DSH ref | `dsh-v0.2.1-rc.3` | `DSH_VERSION` |
|
||||
| Repository revision | `r2` | `RELEASE_REVISION` |
|
||||
| DSH ref | `dsh-v0.2.1-rc.5-r6` | `DSH_VERSION` |
|
||||
| DSH commit | `b830ec950faba0a46dda564be3e2d854afed94f1` | `DSH_COMMIT`; the main image build fails unless the clone's `HEAD` matches |
|
||||
| Repository revision | `(empty)` | `RELEASE_REVISION` |
|
||||
| pnpm | `11.28.5` | `PNPM_VERSION` |
|
||||
| agent-sandbox (CRDs) | `v1.0.7` | `AGENT_SANDBOX_VERSION`, used by the examples and CI dry-run |
|
||||
| Node build image | `docker.io/node:24-trixie@sha256:8c761eca2bf7eec965f946cf47b8090143cb41f45f2f8699af0cc0298d2d2d19` | `NODE_BUILD_IMAGE`, digest-pinned |
|
||||
| Node runtime image | `docker.io/node:24-trixie-slim@sha256:54f84b4cf8432b21aaffd5d3ce8d41e12cea0d10d33c09d2f3c9879d49d6b446` | `NODE_RUNTIME_IMAGE`, digest-pinned |
|
||||
| Gateway Python image | `docker.io/python:3.12-slim@sha256:aa943ce411423c7582288f18707ff66eddcbaeb510502d747a72aeb34f30d159` | `GATEWAY_PYTHON_IMAGE`, digest-pinned |
|
||||
| agent-sandbox (CRDs) | `v1.0.9` | `AGENT_SANDBOX_VERSION`, used by the examples and CI dry-run |
|
||||
| Kubernetes Gateway API | `v1.2.1` | `GATEWAY_API_VERSION`, used by the SSO exposure variant |
|
||||
| oauth2-proxy | `quay.io/oauth2-proxy/oauth2-proxy:v7.15.5` | `OAUTH2_PROXY_IMAGE` |
|
||||
| oauth2-proxy | `quay.io/oauth2-proxy/oauth2-proxy:v7.15.5@sha256:8498b0d0ef0a7b29686414000a08aee467f02d0299c9ed1e006a8f33fc017916` | `OAUTH2_PROXY_IMAGE`, digest-pinned and used by the SSO exposure variant |
|
||||
| Main image | `ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox` | `GHCR_IMAGE` |
|
||||
| Gateway image | `ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway` | `GATEWAY_IMAGE`, nested below `GHCR_IMAGE` |
|
||||
|
||||
Every container base image is referenced as `name:tag@sha256:<index digest>`, not
|
||||
by tag alone: `check-versions.sh` rejects a pin without a digest, so a re-pushed
|
||||
tag cannot change what we build from. The example gateway's Python dependencies
|
||||
are fully hash-pinned in
|
||||
[requirements.txt](../examples/dsh-agents-as-a-service/gateway/requirements.txt)
|
||||
and
|
||||
[requirements-dev.txt](../examples/dsh-agents-as-a-service/gateway/requirements-dev.txt)
|
||||
and installed with `pip install --require-hashes`.
|
||||
|
||||
Example component images are always nested below the main package:
|
||||
`<GHCR_IMAGE>/<example>/<component>`. `check-versions.sh` rejects an unrelated
|
||||
top-level package.
|
||||
@@ -94,8 +99,15 @@ file to derive from it. Never edit the Dockerfile, manifests, Makefile or
|
||||
workflows with a literal version.
|
||||
|
||||
- New upstream DSH ref: change `DSH_VERSION` and clear `RELEASE_REVISION`.
|
||||
- Repository-only change at the same DSH ref: keep `DSH_VERSION`, set
|
||||
`RELEASE_REVISION=r(N+1)`.
|
||||
`set-version` resolves the new ref's commit with `git ls-remote` (peeled `^{}`
|
||||
commit first, tag ref as fallback) and rewrites `DSH_COMMIT` in `versions.env`,
|
||||
the Dockerfile ARG default and `.env.sample`; the build then verifies it.
|
||||
- Repository-only change at the same DSH ref: keep `DSH_VERSION` and `DSH_COMMIT`,
|
||||
set `RELEASE_REVISION=r(N+1)`.
|
||||
- Base images (Node and Python) are bumped by hand: change the tag *and* the
|
||||
`@sha256:` digest in `versions.env` together, then run `make check` and a build.
|
||||
There is deliberately no automated digest refresh, because a silent base-image
|
||||
change is exactly what digest pinning exists to prevent.
|
||||
|
||||
## Release checklist
|
||||
|
||||
@@ -106,14 +118,16 @@ workflows with a literal version.
|
||||
the tag does not already exist, and previews the generated notes.
|
||||
5. Create and push the annotated `dsh-*` release tag (or dispatch
|
||||
`release.yaml` with an existing tag). The workflow validates that the tag
|
||||
matches [versions.env](../versions.env).
|
||||
matches [versions.env](../versions.env) and that the upstream ref still
|
||||
resolves to `DSH_COMMIT` before building anything.
|
||||
6. The workflow builds and pushes multi-arch images for the main image and the
|
||||
example components (the identity tag plus `latest`), creates the GitHub
|
||||
Release from the git-cliff notes, records the published digests, and opens a
|
||||
`chore(changelog)` PR that refreshes `CHANGELOG.md` on `main`.
|
||||
Release from the git-cliff notes, and opens a `chore(changelog)` PR that
|
||||
refreshes `CHANGELOG.md` on `main`. The publish log shows each image manifest
|
||||
and digest.
|
||||
7. Merge the changelog PR. Verify:
|
||||
`docker buildx imagetools inspect <image>:<identity>` shows both platforms,
|
||||
and the GitHub Release references the tag and the digest.
|
||||
`docker buildx imagetools inspect <image>:<identity>` shows both platforms;
|
||||
the GitHub Release references the tag, and the workflow log shows the digest.
|
||||
8. Announce the identity and the digest; deployments pin the digest.
|
||||
|
||||
## Rollback
|
||||
@@ -149,11 +163,20 @@ make changelog # regenerate CHANGELOG.md locally
|
||||
|
||||
## Digest pinning
|
||||
|
||||
Release identities are mutable only in theory; deployments should still pin the
|
||||
resolved digest for reproducibility and supply-chain safety:
|
||||
Supply-chain inputs are pinned to immutable content, not to mutable tags:
|
||||
|
||||
- **Build inputs**: every container base image is referenced by multi-arch index
|
||||
digest in [versions.env](../versions.env), and the example gateway's Python
|
||||
dependencies are hash-pinned (`--require-hashes`).
|
||||
- **Upstream source**: the Dockerfile clones `DSH_VERSION` and fails unless the
|
||||
clone's `HEAD` equals `DSH_COMMIT`, so a moved upstream tag cannot change what
|
||||
a rebuild packages.
|
||||
|
||||
Release identities are immutable in theory; deployments should still pin the
|
||||
_resolved_ digest for reproducibility and supply-chain safety:
|
||||
|
||||
```bash
|
||||
docker buildx imagetools inspect ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.3-r3
|
||||
docker buildx imagetools inspect ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.5-r6
|
||||
# then, in the deployment:
|
||||
# image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox@sha256:<digest>
|
||||
```
|
||||
|
||||
69
docs/workflow-security.md
Normal file
69
docs/workflow-security.md
Normal file
@@ -0,0 +1,69 @@
|
||||
# Workflow security (GitHub Actions)
|
||||
|
||||
The CI/CD surface lives entirely in [`.github/workflows/`](../.github/workflows/)
|
||||
and runs on GitHub-hosted runners. This page describes the controls those
|
||||
workflows enforce and records the most recent audit. They complement the threat
|
||||
notes in [SECURITY.md](../SECURITY.md) and the release procedure in
|
||||
[versioning.md](versioning.md).
|
||||
|
||||
## Controls
|
||||
|
||||
| Control | Where | What it does |
|
||||
| --- | --- | --- |
|
||||
| Least-privilege tokens | Every workflow | A top-level `permissions:` block defaults to read-only; `packages: write`, `contents: write`, `id-token: write` and `attestations: write` are granted only to the jobs that publish images, attestations or releases. |
|
||||
| Pinned actions | Every `uses:` | Third-party actions and reusable workflows are pinned to a full 40-character commit SHA (the version is kept in the trailing comment). Dependabot updates the pins weekly. The only unpinned reference is the in-repository `./.github/workflows/_build-image.yaml`. |
|
||||
| No privileged untrusted triggers | Trigger blocks | No workflow uses `pull_request_target` or `workflow_run`. PR-triggered workflows run untrusted code with a read-only token and no secrets. |
|
||||
| Untrusted input is data | `release.yaml` | The release tag (from a tag push or `workflow_dispatch` input) is passed through `env:` and read as a shell variable; it is never interpolated into a `run:` script, so metacharacters in a tag name cannot execute. |
|
||||
| Checkout credential hygiene | Checkout steps | `persist-credentials: false` is set wherever the job does not need to push. The changelog job keeps credentials because it pushes a branch and opens a PR with `GITHUB_TOKEN`. |
|
||||
| Download integrity | `ci.yaml`, `release.yaml` | actionlint, hadolint and git-cliff are downloaded with `curl --proto '=https'` and verified against a pinned SHA-256 with `sha256sum -c` before execution. |
|
||||
| Cluster manifest integrity | `ci-manifests.yaml`, `e2e-kind.yaml` | The agent-sandbox and Gateway API release manifests are downloaded and checked against the `*_MANIFEST_SHA256` pins in [versions.env](../versions.env) before `kubectl apply`. |
|
||||
| Log and artifact hygiene | `ci-image.yaml`, `e2e-kind.yaml` | Pod logs are truncated and token material is redacted before logs are printed or diagnostic files are uploaded as artifacts. |
|
||||
| Supply-chain monitoring | Repository settings | Dependabot (GitHub Actions, pip, Docker), CodeQL advanced setup and the dependency-review action run on every applicable change. CODEOWNERS covers the whole repository, including `.github/workflows/`. |
|
||||
|
||||
## Maintaining the pins
|
||||
|
||||
- The tool versions and their SHA-256 digests live together in the install step
|
||||
(`ACTIONLINT_SHA256`, `HADOLINT_SHA256`, `GIT_CLIFF_SHA256`); update both
|
||||
values when bumping a version.
|
||||
- Control-plane versions are in `versions.env`; update
|
||||
`AGENT_SANDBOX_MANIFEST_SHA256` or `GATEWAY_API_MANIFEST_SHA256` together with
|
||||
the corresponding version, otherwise CI fails the digest check by design.
|
||||
- GitHub Actions pins are updated by Dependabot; never replace a SHA with a tag.
|
||||
|
||||
## Audit record (2026-10-30)
|
||||
|
||||
Scope: all 11 workflows under `.github/workflows/`, plus `.github/CODEOWNERS`
|
||||
and `.github/dependabot.yml`. Findings are grouped by the categories in
|
||||
GitHub's [Secure use reference](https://docs.github.com/en/actions/reference/security/secure-use).
|
||||
|
||||
| # | Category | File | Severity | Finding | Status |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| 1 | Script injection | `release.yaml` | High | Tag name / dispatch input interpolated into a privileged `run:` block | Fixed: passed through `env` |
|
||||
| 2 | Token permissions | `release.yaml` | Medium | `meta` and `tests` inherited workflow-level write scopes | Fixed: workflow default is `contents: read` |
|
||||
| 3 | Token permissions | `build-main.yaml`, `build-gateway.yaml` | Low | `meta` inherited `packages`/`id-token`/`attestations` write | Fixed: workflow default is `contents: read` |
|
||||
| 4 | Checkout | `_build-image.yaml` | Low | Checkout persisted the token in `.git/config` | Fixed: `persist-credentials: false` |
|
||||
| 5 | Checkout | `codeql.yml` | Low | Checkout persisted the token in `.git/config` | Fixed: `persist-credentials: false` |
|
||||
| 6 | Supply chain | `ci.yaml` | Medium | actionlint and hadolint were executed without a checksum | Fixed: pinned SHA-256 verified before install |
|
||||
| 7 | Supply chain | `release.yaml` | Medium | git-cliff was executed without a checksum in the release job | Fixed: pinned SHA-256 verified before install |
|
||||
| 8 | Supply chain | `ci-manifests.yaml`, `e2e-kind.yaml` | Medium | Upstream CRD/controller manifests applied without integrity checks | Fixed: pinned SHA-256 in `versions.env` |
|
||||
| 9 | Secret hygiene | `e2e-kind.yaml` | Low | Raw gateway logs uploaded in a 7-day artifact | Fixed: `token=` values redacted |
|
||||
| 10 | Repository controls | Repository settings | Low | No environment approval gate on publishing; single CODEOWNER | Recommendation: see below |
|
||||
|
||||
Checked and clean: no `pull_request_target`/`workflow_run`, no self-hosted
|
||||
runners, all third-party actions pinned, PR workflows read-only, no secrets in
|
||||
workflow files, no secret interpolation into scripts.
|
||||
|
||||
## Residual recommendations (repository settings)
|
||||
|
||||
These cannot be enforced from workflow files alone:
|
||||
|
||||
- Add a protected `release` environment with required reviewers and attach it to
|
||||
the `build-main`, `build-gateway` and `publish-release` jobs. This requires the
|
||||
environment to exist in repository settings first, so it is intentionally not
|
||||
wired into the workflow yet.
|
||||
- Keep "Allow GitHub Actions to create and approve pull requests" limited to
|
||||
creating PRs (the changelog job needs create, not approve).
|
||||
- Require review from Code Owners for `.github/workflows/` and add a second
|
||||
owner so workflow changes are never single-person.
|
||||
- Optionally run OpenSSF Scorecard on a schedule for continuous checks on these
|
||||
controls.
|
||||
@@ -10,38 +10,35 @@ reference material, not libraries.
|
||||
| [`examples/dsh-agents-as-a-service/`](dsh-agents-as-a-service/) | The full multi-tenant platform pattern: one `SandboxTemplate` with per-user PVCs, a warm pool, one ~10-line `SandboxClaim` per user, suspend-when-idle / wake-on-return, and a browser gateway with signup, `HttpOnly` sessions, an HTTP + WebSocket proxy and two exposure variants (Ingress, or Gateway API + GitHub SSO). | agent-sandbox with extensions; kind for the e2e script. |
|
||||
| [`examples/minimal-single-tenant/`](minimal-single-tenant/) | The smallest useful setup: a single `SandboxTemplate` and one `SandboxClaim`, no gateway — reach DSH Web with a launch-token URL through `kubectl port-forward`. | agent-sandbox with extensions. |
|
||||
|
||||
The multi-tenant example also documents opt-in gVisor, `nodeSelector`,
|
||||
NetworkPolicy, extra provider-key hardening and failed-Pod recovery in
|
||||
[optional hardening](dsh-agents-as-a-service/docs/optional-hardening.md).
|
||||
|
||||
## The example contract (short version)
|
||||
|
||||
Full rules: [docs/examples.md](../docs/examples.md). `scripts/check-examples.sh`
|
||||
enforces the layout, and `make check` runs it.
|
||||
|
||||
```
|
||||
examples/<name>/
|
||||
+-- README.md # walkthrough: prerequisites, steps, troubleshooting, cleanup
|
||||
+-- manifests/ # NN-*.yaml, applied with plain kubectl
|
||||
| +-- exposure/<variant>/ # optional per-variant manifests...
|
||||
| +-- exposure/README.md # ...required whenever exposure/ exists
|
||||
+-- scripts/ # optional executable, idempotent, self-cleaning *.sh
|
||||
+-- docs/ # optional architecture/ADR material
|
||||
+-- <component>/ # optional component with Dockerfile + README + tests
|
||||
```
|
||||
|
||||
- Secrets never ship populated: use `*.yaml.example` files with placeholders.
|
||||
- Manifests are numbered (`00-`, `10-`, ...) and read as an ordered walkthrough.
|
||||
- Component images are published nested below the main image:
|
||||
`<registry>/<owner>/<repo>/<example>/<component>`.
|
||||
- Every example must be listed above with its literal `examples/<name>/` path.
|
||||
- Scripts under `scripts/` must be executable, idempotent and clean up after
|
||||
themselves.
|
||||
- `README.md` is the walkthrough; `manifests/NN-*.yaml` is applied with plain
|
||||
kubectl and read in order.
|
||||
- Optional `manifests/optional/<feature>/` and `manifests/exposure/<variant>/`
|
||||
each ship a README; exposure variants are mutually exclusive.
|
||||
- Optional `scripts/` must be executable, idempotent and self-cleaning; `docs/`
|
||||
holds deeper material.
|
||||
- Components with a Dockerfile need a README and tests; images are published
|
||||
nested below the main image (`<registry>/<owner>/<repo>/<example>/<component>`).
|
||||
- Secrets are `*.yaml.example` with placeholders; every example is listed above
|
||||
with its literal `examples/<name>/` path.
|
||||
|
||||
## Running an example
|
||||
|
||||
```bash
|
||||
# e.g. the multi-tenant platform example
|
||||
kubectl apply -f examples/dsh-agents-as-a-service/manifests/
|
||||
Each example README has the exact prerequisites, commands, optional hardening,
|
||||
exposure options and teardown. The multi-tenant example also has a full kind
|
||||
walkthrough that cleans up after itself:
|
||||
|
||||
# or run the whole walkthrough (kind + agent-sandbox required) and let it clean up
|
||||
```bash
|
||||
examples/dsh-agents-as-a-service/scripts/e2e-kind.sh
|
||||
```
|
||||
|
||||
Each example README has the exact prerequisites, exposure options and teardown.
|
||||
`kubectl apply -f manifests/` is non-recursive: `optional/` and `exposure/`
|
||||
subdirectories are applied explicitly.
|
||||
|
||||
@@ -6,16 +6,19 @@ platform pattern into pure agent-sandbox resources + kubectl, running **DeepSeek
|
||||
Harness Web (DSH)** instead of Hermes. Every user of such a platform gets a
|
||||
personal, long-lived DSH Web instance that is **provisioned in ~2 seconds from a
|
||||
warm pool, suspended when idle to save cost, and woken transparently when the
|
||||
user returns** — with all state (conversations, sessions, config) surviving on a
|
||||
per-user PVC at `/home/node/.dsh`.
|
||||
user returns** — with all state (conversations, sessions, config, and the agent
|
||||
workspace) surviving on the per-user PVC at `/home/node/.dsh` + `/workspace`.
|
||||
Suspend-when-idle plus the warm pool keep cost proportional to use: a return
|
||||
after absence pays a short cold-start pause, not an always-on instance.
|
||||
|
||||
It ships in three layers:
|
||||
|
||||
1. **The resource layer** (steps 1–7): pure manifests + kubectl, so every
|
||||
mechanism is visible.
|
||||
2. **The service layer** (step 8): a small included **gateway** (`gateway/`) that
|
||||
makes the platform browser-usable — a signup page/API, `HttpOnly` cookie
|
||||
sessions, a `/u/{user}/**` proxy for HTTP **and WebSocket** that transparently
|
||||
makes the platform browser-usable — a signup page/API, revocable server-side
|
||||
**opaque-cookie sessions**, auth rate limiting and an optional signup invite
|
||||
code, a `/u/{user}/**` proxy for HTTP **and WebSocket** that transparently
|
||||
**wakes suspended instances on connect**, and an idle sweeper that suspends
|
||||
them again. The gateway performs the DSH launch-token handshake server-side,
|
||||
so the browser never sees the token.
|
||||
@@ -32,17 +35,21 @@ manifests/
|
||||
+-- 30-claim-alice.yaml # one user
|
||||
+-- 40-claim-rejected.yaml # negative demo: env injection rejected
|
||||
+-- 50-gateway.yaml # Deployment/RBAC/Service for the gateway
|
||||
+-- optional/
|
||||
| +-- README.md # opt-in manifests, applied by path
|
||||
| +-- network-policy/10-sandbox-networkpolicy.yaml
|
||||
+-- exposure/
|
||||
+-- README.md # the two exposure variants, when to use which
|
||||
+-- ingress/60-ingress.yaml
|
||||
+-- gateway-api-sso/70-gateway.yaml ... 74-tls-secret.yaml.example
|
||||
+-- ingress/61-networkpolicy.yaml
|
||||
+-- gateway-api-sso/70-gateway.yaml ... 75-networkpolicies.yaml
|
||||
```
|
||||
|
||||
What it demonstrates, feature by feature:
|
||||
|
||||
| agent-sandbox feature | Role in the platform |
|
||||
| --- | --- |
|
||||
| `SandboxTemplate` with `volumeClaimTemplates` | Identical DSH pods; per-user state on a PVC at `/home/node/.dsh` |
|
||||
| `SandboxTemplate` with `volumeClaimTemplates` | Identical DSH pods; per-user state at `/home/node/.dsh` and the agent workspace at `/workspace`, on one PVC (the workspace is a `subPath` mount, so DSH state stays private) |
|
||||
| `envVarsInjectionPolicy` / `volumeClaimTemplatesPolicy: Disallowed` | Claims that would bypass the warm pool are **rejected loudly**, not cold-started silently |
|
||||
| `service: true` | Stable per-sandbox DNS (`status.serviceFQDN`) for the gateway to route on |
|
||||
| `SandboxWarmPool` | Signup latency is pre-paid: image pull, PVC provision, DSH boot all happen before any user exists |
|
||||
@@ -57,12 +64,15 @@ What it demonstrates, feature by feature:
|
||||
`kubernetes.io/arch: amd64` nodeSelector.
|
||||
- **Single port**: DSH Web serves everything on `3080`; the gateway proxies to one
|
||||
upstream instead of splitting `/v1/*` (API) from the dashboard.
|
||||
- **One shared secret**: only `DEEPSEEK_API_KEY` is injected (via `envFrom`); there
|
||||
is no dashboard basic-auth, session-cookie secret, or API-server key.
|
||||
- **Shared provider keys**: `DEEPSEEK_API_KEY` plus any keys in the optional
|
||||
`dsh-provider-keys-extra` Secret are injected via `envFrom` into every
|
||||
sandbox. There is no per-tenant key, dashboard basic-auth, signed-session
|
||||
secret (sessions are opaque server-side ids), or API-server key.
|
||||
- **Launch-token handshake**: DSH gates `/` behind a token it prints only in its
|
||||
logs. The gateway reads it, performs the handshake, and keeps the resulting
|
||||
auth cookie server-side; the browser authenticates to the gateway with its own
|
||||
cookie. See [How the DSH handshake works](#how-the-dsh-handshake-works).
|
||||
opaque session cookie, never the token. See
|
||||
[How the DSH handshake works](#how-the-dsh-handshake-works).
|
||||
- **Host allow-list**: DSH rejects unknown `Host` headers. The gateway forwards
|
||||
the browser's `Host`, so the public hostname must be added to
|
||||
`DSH_TRUSTED_HOSTS` in the template.
|
||||
@@ -73,12 +83,14 @@ What it demonstrates, feature by feature:
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Run the walkthrough from this example's directory (`examples/dsh-agents-as-a-service/`).
|
||||
|
||||
- A Kubernetes cluster with a default StorageClass (kind and GKE both work as-is).
|
||||
- agent-sandbox with extensions installed:
|
||||
|
||||
```bash
|
||||
VERSION=v1.0.7 # AGENT_SANDBOX_VERSION in versions.env
|
||||
kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/download/$VERSION/sandbox-with-extensions.yaml
|
||||
. ../../versions.env
|
||||
kubectl apply -f "https://github.com/kubernetes-sigs/agent-sandbox/releases/download/${AGENT_SANDBOX_VERSION}/sandbox-with-extensions.yaml"
|
||||
kubectl -n agent-sandbox-system rollout status deployment --timeout=180s
|
||||
```
|
||||
|
||||
@@ -93,14 +105,39 @@ What it demonstrates, feature by feature:
|
||||
|
||||
## 1. Install the platform resources
|
||||
|
||||
First the namespace, then the shared provider key. The key reaches **every**
|
||||
First the namespace, then the shared provider keys. The keys reach **every**
|
||||
tenant sandbox (a warm-pool constraint: pod env is stamped before any user
|
||||
exists), so it is **never committed** — generate a fresh value per install:
|
||||
exists), so they are **never committed** — generate fresh values per install.
|
||||
`00-prereqs.yaml` already creates the empty `dsh-provider-keys` Secret, so use
|
||||
an upsert:
|
||||
|
||||
```bash
|
||||
kubectl apply -f manifests/00-prereqs.yaml
|
||||
kubectl -n dsh-demo create secret generic dsh-provider-keys \
|
||||
--from-literal=DEEPSEEK_API_KEY="sk-..."
|
||||
--from-literal=DEEPSEEK_API_KEY="sk-..." \
|
||||
--dry-run=client -o yaml | kubectl apply -f -
|
||||
```
|
||||
|
||||
Optional: additional providers (OpenRouter, Gemini, NVIDIA, OpenAI, Anthropic,
|
||||
...) go in a second Secret. It is referenced with `optional: true`, so the
|
||||
sandboxes start without it, and every key becomes an env var:
|
||||
|
||||
```bash
|
||||
kubectl -n dsh-demo create secret generic dsh-provider-keys-extra \
|
||||
--from-literal=OPENROUTER_API_KEY="sk-or-..." \
|
||||
--from-literal=GEMINI_API_KEY="..." \
|
||||
--from-literal=NVIDIA_API_KEY="nvapi-..." \
|
||||
--dry-run=client -o yaml | kubectl apply -f -
|
||||
```
|
||||
|
||||
Create both Secrets **before** the template/warm pool. A later Secret change
|
||||
does not update running pods: delete the warm-pool spares and recreate claimed
|
||||
pods to pick up new keys. Keys are shared by all tenants; a key saved in DSH's
|
||||
credential store wins over env. See
|
||||
[optional hardening](docs/optional-hardening.md) for the full env-name list and
|
||||
the refresh procedure.
|
||||
|
||||
```bash
|
||||
kubectl apply -f manifests/10-sandboxtemplate.yaml -f manifests/20-sandboxwarmpool.yaml
|
||||
```
|
||||
|
||||
@@ -170,9 +207,30 @@ kubectl -n dsh-demo port-forward "pod/$SB" 18081:3080
|
||||
## 4. Write state on the PVC
|
||||
|
||||
```bash
|
||||
kubectl -n dsh-demo exec "$SB" -- printenv HOME # /workspace
|
||||
kubectl -n dsh-demo exec "$SB" -- sh -c 'echo "remember me" > /home/node/.dsh/marker.txt'
|
||||
kubectl -n dsh-demo exec "$SB" -- sh -c 'echo "remember me too" > /workspace/marker.txt'
|
||||
```
|
||||
|
||||
`/workspace` (the DSH image's `WORKDIR`, where agents keep files) is the same
|
||||
`dsh-home` PVC mounted again with `subPath: workspace`. The two trees stay
|
||||
separate, so `.credentials.yaml` and the session store are never part of the
|
||||
agent-visible workspace.
|
||||
|
||||
The template also sets `HOME=/workspace`. DSH's directory picker (the folder
|
||||
explorer used to choose a workspace) has no start-path setting: when it opens,
|
||||
its browse backend lists `$HOME`, so with the image default (`/home/node`, which
|
||||
is not on a volume) it would start outside the persisted tree. Pointing `HOME`
|
||||
at the workspace makes the picker open on it (labelled *Home* in its breadcrumbs)
|
||||
and makes `~` in agent commands mean `/workspace`. The trade-off is that tools
|
||||
keeping caches or config under `$HOME` (`npm`/`pnpm`, `uv`, `git`, `gh`, and
|
||||
DSH's global `~/.agents` instructions and skills) write dotfiles onto the PVC
|
||||
inside the agent workspace, where the agent and the file explorer can see them.
|
||||
`DSH_HOME` is explicit (`/home/node/.dsh`), so sessions, config and credentials
|
||||
are unaffected. Pod env is stamped when a sandbox is created: a sandbox from
|
||||
before this setting keeps `HOME=/home/node` until it is recreated (see
|
||||
[Troubleshooting](#troubleshooting)).
|
||||
|
||||
## 5. Suspend: delete the pod, keep the PVC + Service
|
||||
|
||||
```bash
|
||||
@@ -188,13 +246,12 @@ kubectl -n dsh-demo get svc "$SB" # still here
|
||||
kubectl -n dsh-demo patch sandbox "$SB" --type merge -p '{"spec":{"operatingMode":"Running"}}'
|
||||
kubectl -n dsh-demo wait sandbox "$SB" --for=condition=Ready --timeout=180s
|
||||
kubectl -n dsh-demo exec "$SB" -- cat /home/node/.dsh/marker.txt # "remember me"
|
||||
kubectl -n dsh-demo exec "$SB" -- cat /workspace/marker.txt # "remember me too"
|
||||
```
|
||||
|
||||
> **Why resume works:** the pod template sets `fsGroupChangePolicy: OnRootMismatch`.
|
||||
> DSH requires its `.credentials.yaml` to be `0600`, while the kubelet's default
|
||||
> fsGroup policy widens every file to `0660` on each pod start — a resumed sandbox
|
||||
> would crash-loop. See [Troubleshooting](#troubleshooting) for the one-time fix
|
||||
> on sandboxes created before this setting existed.
|
||||
> **Why resume works:** the pod template sets `fsGroupChangePolicy: OnRootMismatch`;
|
||||
> without it the kubelet widens `.credentials.yaml` to `0660` and DSH crash-loops.
|
||||
> See [Troubleshooting](#troubleshooting) for the one-time fix on older sandboxes.
|
||||
|
||||
## 7. Injection policy rejects env-carrying claims
|
||||
|
||||
@@ -213,13 +270,13 @@ kubectl delete -f manifests/40-claim-rejected.yaml
|
||||
```bash
|
||||
# Published image (on kind, load it into the cluster; on a real cluster the
|
||||
# kubelet pulls it from GHCR):
|
||||
GW=ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway:dsh-v0.2.1-rc.3-r3
|
||||
GW=ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway:dsh-v0.2.1-rc.5-r6
|
||||
docker pull "$GW"
|
||||
kind load docker-image "$GW" --name <kind-cluster>
|
||||
|
||||
# ... or build it locally and tag it exactly as manifests/50-gateway.yaml
|
||||
# expects (make gateway builds dsh-aaas-gateway:dev):
|
||||
make gateway
|
||||
make -C ../.. gateway
|
||||
docker tag dsh-aaas-gateway:dev "$GW"
|
||||
kind load docker-image "$GW" --name <kind-cluster>
|
||||
|
||||
@@ -230,6 +287,15 @@ kubectl -n dsh-demo port-forward svc/dsh-gateway 18080:8080
|
||||
|
||||
> The gateway binds `8080`; the local port-forward uses `18080` and never `3080`.
|
||||
>
|
||||
> **Cookie/TLS guard.** The base manifest serves plain HTTP through port-forward,
|
||||
> so it sets `COOKIE_SECURE=false` **and** `ALLOW_INSECURE_COOKIES=1`. The gateway
|
||||
> refuses to start when `COOKIE_SECURE=false` unless
|
||||
> `ALLOW_INSECURE_COOKIES=1` explicitly acknowledges that the session cookie
|
||||
> travels in clear text; that prevents an HTTPS deployment from silently serving
|
||||
> an insecure cookie. Before exposing the gateway over TLS, set
|
||||
> `COOKIE_SECURE=true` and remove the acknowledgement:
|
||||
> `kubectl -n dsh-demo set env deploy/dsh-gateway COOKIE_SECURE=true ALLOW_INSECURE_COOKIES-`.
|
||||
>
|
||||
> **Image pull policy.** `manifests/50-gateway.yaml` uses
|
||||
> `imagePullPolicy: IfNotPresent` with the release identity, so a locally built
|
||||
> image must be loaded into the cluster under exactly that name (kind:
|
||||
@@ -253,28 +319,49 @@ kubectl -n dsh-demo port-forward svc/dsh-gateway 18080:8080
|
||||
> apply does not update them: patch each sandbox's
|
||||
> `spec.podTemplate.spec.securityContext.fsGroupChangePolicy` and
|
||||
> `chmod 600` its `.credentials.yaml`, or recreate it.
|
||||
>
|
||||
> The optional gVisor/`nodeSelector` fields are pod-spec changes too: existing
|
||||
> Sandboxes snapshot them, and warm-pool spares are not replaced by a template
|
||||
> edit. See [optional hardening](docs/optional-hardening.md) for the patch and
|
||||
> recreation commands. A provider-key Secret change likewise needs pod
|
||||
> recreation to take effect.
|
||||
|
||||
### Browser flow
|
||||
|
||||
Open <http://localhost:18080>. The landing page has a one-field signup form.
|
||||
Enter `bob`, and the gateway:
|
||||
Open <http://localhost:18080>. The landing page has a one-field signup form (plus
|
||||
an invite-code field when `SIGNUP_INVITE_CODE` is configured). Enter `bob`, and
|
||||
the gateway:
|
||||
|
||||
1. creates the `SandboxClaim` over the warm pool,
|
||||
2. waits for it to become `Ready`,
|
||||
3. sets an `HttpOnly`, `SameSite=Lax` cookie `dsh_gw_session=<user>:<token>` on
|
||||
the browser,
|
||||
4. redirects to `/u/bob/`, where the DSH Web UI loads through the gateway.
|
||||
1. enforces the auth rate limit and, if configured, the signup invite code,
|
||||
2. creates the `SandboxClaim` over the warm pool,
|
||||
3. waits for it to become `Ready`,
|
||||
4. mints a random, server-side session (the cookie value is an opaque `sid`, not
|
||||
a user name and not a token), stores `sid -> (user, expiry)` in memory, and
|
||||
sets the `HttpOnly`, `SameSite=Lax` cookie `dsh_gw_session=<sid>` on the
|
||||
browser (`__Host-dsh_gw_session` when `COOKIE_SECURE=true`),
|
||||
5. redirects to `/u/bob/`, where the DSH Web UI loads through the gateway.
|
||||
|
||||
From then on every request under `/u/bob/` (assets, API calls, SSE, WebSocket) is
|
||||
proxied to bob's sandbox. The DSH launch token and DSH's own auth cookie stay in
|
||||
the gateway; the browser only holds the gateway cookie.
|
||||
the gateway; the browser only holds the opaque gateway session id. The cookie is
|
||||
useless as a bearer token.
|
||||
|
||||
**Sessions are revocable and mostly short-lived in practice.** `POST /logout`,
|
||||
deleting the user, `POST /recreate` and cookie expiry (default 7 days,
|
||||
`COOKIE_MAX_AGE`) invalidate the server-side session. The session table lives in
|
||||
gateway memory, so **restarting the gateway logs every browser out** — expected
|
||||
for this single-replica demo; a production deployment needs a shared session
|
||||
store and sticky routing or per-request re-authentication. With SSO enabled, the
|
||||
gateway mints a session for the identity oauth2-proxy asserts instead of asking
|
||||
for a token.
|
||||
|
||||
The same flow with `curl`, no JavaScript (the session cookie is the only
|
||||
credential):
|
||||
|
||||
```bash
|
||||
JAR=$(mktemp)
|
||||
# signup (form POST, exactly what the landing page sends)
|
||||
# signup (form POST, exactly what the landing page sends); add -d 'invite=...'
|
||||
# when SIGNUP_INVITE_CODE is configured, or send X-Signup-Invite
|
||||
curl -s -i -c "$JAR" -H 'Origin: http://localhost:18080' \
|
||||
-d 'user=bob' http://localhost:18080/users
|
||||
# proxy: bob's DSH Web through the gateway
|
||||
@@ -283,52 +370,54 @@ curl -s -b "$JAR" http://localhost:18080/u/bob/ | head -5
|
||||
curl -s -o /dev/null -w '%{http_code}\n' http://localhost:18080/u/bob/ # 401
|
||||
# derived state: Ready / Waking / Suspended / ...
|
||||
curl -s -b "$JAR" http://localhost:18080/users/bob
|
||||
# delete bob (cascades sandbox + PVC)
|
||||
# sign out (revokes the server-side session)
|
||||
curl -s -b "$JAR" -H 'Origin: http://localhost:18080' -X POST http://localhost:18080/logout
|
||||
# delete bob (cascades sandbox + PVC and revokes every session for bob)
|
||||
curl -s -b "$JAR" -H 'Origin: http://localhost:18080' -X DELETE http://localhost:18080/users/bob
|
||||
```
|
||||
|
||||
API clients that prefer bearer tokens can use `POST /users` with a JSON body
|
||||
(`{"user":"bob"}`) and the returned one-time token, `POST /login` to exchange a
|
||||
token for a cookie, and `Authorization: Bearer <token>` on subsequent calls.
|
||||
token for a cookie, and `Authorization: Bearer <token>` on subsequent calls. A
|
||||
cookie/session value is never accepted as a bearer token, and a token for one
|
||||
user cannot read another user's path.
|
||||
|
||||
### Gateway API
|
||||
### Auth abuse controls and proxy limits
|
||||
|
||||
| Route | Purpose |
|
||||
| --- | --- |
|
||||
| `GET /healthz` | Liveness/readiness; no cluster access |
|
||||
| `GET /` | Landing page: signup, token login, logout. With SSO it auto-provisions and connects instead. |
|
||||
| `POST /users` | Signup. JSON or form; `201` (JSON), `202` (waking), `303` (browser redirect), `409` if user exists; sets the session cookie |
|
||||
| `POST /login` | Exchange a bearer token for the session cookie |
|
||||
| `POST /logout` | Clear the session cookie |
|
||||
| `POST /recreate` | SSO only: delete the authenticated user's claim (wiping workspace data) and provision a fresh one; `303` to `/u/{user}/`. Cookie-authenticated calls must be same-origin. |
|
||||
| `GET /users/{user}` | Derived state (`Ready`/`Waking`/`Suspended`) |
|
||||
| `DELETE /users/{user}` | Delete claim (cascades sandbox + PVC) |
|
||||
| `GET /u/{user}` | `308` redirect to `/u/{user}/` |
|
||||
| `GET /u/{user}/**` | HTTP/SSE proxy to the user's DSH Web |
|
||||
| `WS /u/{user}/**` | WebSocket proxy to the user's DSH Web |
|
||||
The gateway is a single process, so these limits are per replica. They stop
|
||||
casual abuse and one tenant monopolizing the service; an internet-facing
|
||||
deployment still needs ingress-level rate limiting and an audit trail.
|
||||
|
||||
The full API reference, configuration table, auth semantics and failure matrix
|
||||
live in the [gateway README](gateway/README.md).
|
||||
| Setting | Default | Behavior |
|
||||
| --- | --- | --- |
|
||||
| `AUTH_RATE_LIMIT` / `AUTH_RATE_WINDOW` | `30` per `60` s | Throttles `/users` and `/login` per client (rightmost `X-Forwarded-For` when behind a proxy, else the TCP peer) and answers `429` + `Retry-After` before reading the body or touching the cluster. `0` disables it. The base manifest sets it explicitly. |
|
||||
| `SIGNUP_INVITE_CODE` | unset | When set, `POST /users` requires the code as form field `invite` or header `X-Signup-Invite` (constant-time compare, `403` before user lookup). Put it in a Secret and restart the gateway to rotate. |
|
||||
| `MAX_IN_FLIGHT_PER_USER` | `32` | Caps concurrent proxied requests + WebSockets per user; excess HTTP gets `503` + `Retry-After`, excess WebSocket upgrades close with `1011`. |
|
||||
| `WS_MAX_MESSAGE_BYTES` | `16 MiB` | Largest WebSocket message received from DSH; oversized frames close the connection. Browser-to-DSH frames are bounded by uvicorn's own limit. HTTP/SSE streaming is not capped by this. |
|
||||
| `MAX_BODY_MB` | `32` | Largest proxied HTTP request body (`413`). |
|
||||
|
||||
The gateway also strips client-supplied `Forwarded`, `X-Forwarded-User` and
|
||||
`X-Auth-Request-*` headers (and `x-forwarded-*` / `x-auth-request-*` prefixes)
|
||||
before proxying, so a tenant cannot inject identity or spoof hop headers into
|
||||
DSH. `UPSTREAM_HEADER_ALLOWLIST` can replace that denylist with an explicit
|
||||
comma-separated allowlist. See the [gateway README](gateway/README.md) for the
|
||||
full configuration.
|
||||
|
||||
### Gateway routes
|
||||
|
||||
The browser flow above uses the gateway's main routes. The full route table,
|
||||
status codes, configuration and failure matrix live in the
|
||||
[gateway README](gateway/README.md).
|
||||
|
||||
### How the DSH handshake works
|
||||
|
||||
DSH does not expose an API for its launch token; it prints a URL containing
|
||||
`?token=…` in the pod logs. For each `(user, pod UID, Host)` the gateway:
|
||||
|
||||
1. resolves the claim → sandbox → pod (annotation `agents.x-k8s.io/pod-name`, or
|
||||
the sandbox name as pod name),
|
||||
2. reads the pod's logs (last `DSH_LOG_TAIL_LINES` lines) and parses the most
|
||||
recent `dsh web: …?token=…` line,
|
||||
3. requests `http://<serviceFQDN>:3080/?token=…` with the browser's `Host`,
|
||||
4. captures DSH's `dsh-auth-*` `Set-Cookie` response and caches it,
|
||||
5. injects that cookie (and strips the browser's `Cookie`/`Authorization` and any
|
||||
`?token=`) on every proxied HTTP and WebSocket request,
|
||||
6. refreshes the handshake once if DSH answers `401`, and drops the cached cookie
|
||||
when the sandbox suspends or is deleted.
|
||||
|
||||
This is why the gateway's RBAC includes `pods` `get`/`list` and `pods/log`
|
||||
`get`. It is also the most fragile part of the design; see
|
||||
[Limitations](#limitations).
|
||||
DSH prints its launch token only in pod logs. The gateway resolves
|
||||
claim → sandbox → pod, reads the pod-log tail, performs the `?token=` handshake
|
||||
under the browser's `Host`, caches `dsh-auth-*` per `(user, pod UID, Host)`, and
|
||||
refreshes once on a `401`. That is why its RBAC includes `pods` `get`/`list` and
|
||||
`pods/log` `get`; it is also the most fragile part of the design (see
|
||||
[Limitations](#limitations) and the
|
||||
[architecture ADR](docs/architecture.md#adr-3-server-side-launch-token-handshake-instead-of-exposing-the-token)).
|
||||
|
||||
### DSH_TRUSTED_HOSTS
|
||||
|
||||
@@ -347,27 +436,21 @@ The same applies to a local `docker run`: the printed token URL uses
|
||||
|
||||
### WebSocket
|
||||
|
||||
The gateway is a Starlette ASGI app and proxies WebSocket upgrades with a
|
||||
bidirectional pump, so the DSH web GUI is fully interactive. Requirements:
|
||||
The gateway proxies WebSocket upgrades with a bidirectional pump, so the DSH web
|
||||
GUI is fully interactive. The entry point must pass `Upgrade`/`Connection`
|
||||
headers; [`manifests/exposure/ingress/60-ingress.yaml`](manifests/exposure/ingress/60-ingress.yaml)
|
||||
is a starting point (Traefik handles upgrades by default). Idle suspension is
|
||||
disabled while a proxied request or socket is in flight, so a long-running turn is
|
||||
not cut off.
|
||||
|
||||
- the ingress/load balancer in front of the gateway must pass `Upgrade`/`Connection`
|
||||
headers; [`manifests/exposure/ingress/60-ingress.yaml`](manifests/exposure/ingress/60-ingress.yaml)
|
||||
is a starting point (Traefik handles upgrades by default),
|
||||
- idle suspension is disabled for users with in-flight proxied requests or open
|
||||
WebSockets, so a long-running chat stream is not cut off mid-turn,
|
||||
- the upstream upgrade dials the sandbox **pod IP**, forwards no `Host`/`Origin`,
|
||||
and performs the launch-token handshake under that same `pod-IP:3080` authority.
|
||||
DSH fences on `Host` (only its bind address, loopback, or an allow-listed host
|
||||
passes) and binds its auth cookie to the request authority
|
||||
(`payload.authority === authority`), while `websockets` derives `Host` from the
|
||||
URI and cannot be overridden. A browser-host cookie or forwarded origin makes
|
||||
the upgrade fail (`401`/`403`, surfaced as close `1011`), which the client
|
||||
retries forever as `[connection] connection lost, retry #N`,
|
||||
- missing/invalid gateway auth is rejected right after the upgrade with close code
|
||||
`4401` (WebSocket auth has no HTTP `401` path); cross-origin or Origin-less
|
||||
upgrades with the cookie are rejected the same way (CSWSH protection), so
|
||||
clients must send a matching `Origin`; upstream bridging failures close with
|
||||
`1011`.
|
||||
The upstream upgrade dials the sandbox **pod IP**, forwards no `Host`/`Origin`,
|
||||
and performs the launch-token handshake under the same `pod-IP:3080` authority.
|
||||
Missing or cross-origin auth closes with `4401`; upstream bridging failures close
|
||||
with `1011`, as does exceeding `MAX_IN_FLIGHT_PER_USER` (per-user socket cap).
|
||||
Messages received from DSH larger than `WS_MAX_MESSAGE_BYTES` (16 MiB by
|
||||
default) close the connection instead of being buffered; browser-to-DSH frames
|
||||
are bounded by uvicorn's own limit. See the
|
||||
[gateway README](gateway/README.md#proxy-failure-semantics) for the full matrix.
|
||||
|
||||
### Path routing
|
||||
|
||||
@@ -388,25 +471,30 @@ not a stack. Details, commands and the switch procedure are in the
|
||||
> `kubectl apply -R`: it would apply both variants and create two paths to the
|
||||
> same gateway, which is unsafe once SSO is enabled.
|
||||
|
||||
Both variants also ship NetworkPolicies: `ingress/61-networkpolicy.yaml` limits
|
||||
gateway ingress to Traefik pods, and `gateway-api-sso/75-networkpolicies.yaml`
|
||||
limits the gateway to oauth2-proxy and oauth2-proxy egress to DNS, public HTTPS
|
||||
and the gateway. They are part of the variant **and required for production**:
|
||||
an exposure applied without its policy is demo-only. The sandbox side needs a
|
||||
separate policy, and the gateway needs an egress policy; both are covered in
|
||||
[optional hardening](docs/optional-hardening.md) (see the production checklist)
|
||||
and summarized in the [exposure README](manifests/exposure/README.md).
|
||||
|
||||
### Option A — plain Ingress (gateway's own session auth)
|
||||
|
||||
```bash
|
||||
# Replace the dsh.example.com host and TLS Secret (or drop the tls: block for
|
||||
# plain HTTP), then:
|
||||
kubectl apply -f manifests/exposure/ingress/
|
||||
```
|
||||
|
||||
1. Set `DSH_TRUSTED_HOSTS` in `manifests/10-sandboxtemplate.yaml` to the same
|
||||
public host (the gateway forwards the browser's `Host`) and re-apply the
|
||||
template.
|
||||
2. Set `COOKIE_SECURE=true` on the gateway once the Ingress serves HTTPS:
|
||||
`kubectl -n dsh-demo set env deploy/dsh-gateway COOKIE_SECURE=true`.
|
||||
(The base manifest keeps `false` so the plain-HTTP port-forward demo works.)
|
||||
3. Verify: `curl -sI https://<host>/healthz` returns `200`; sign up in a browser
|
||||
at `https://<host>/`.
|
||||
1. Set the Ingress host/TLS (or drop the `tls:` block), and set
|
||||
`DSH_TRUSTED_HOSTS` in `manifests/10-sandboxtemplate.yaml` to the same public
|
||||
host; re-apply the template.
|
||||
2. Set `COOKIE_SECURE=true` and drop the plain-HTTP acknowledgement once the
|
||||
Ingress serves HTTPS:
|
||||
`kubectl -n dsh-demo set env deploy/dsh-gateway COOKIE_SECURE=true ALLOW_INSECURE_COOKIES-`.
|
||||
3. `kubectl apply -f manifests/exposure/ingress/`; verify
|
||||
`curl -sI https://<host>/healthz` returns `200`.
|
||||
|
||||
Keep the router free of buffering/compression middlewares: they break WebSocket
|
||||
upgrades and streaming.
|
||||
upgrades and streaming. The variant's `61-networkpolicy.yaml` allows only Traefik
|
||||
pods to reach the gateway; see the [exposure README](manifests/exposure/README.md)
|
||||
for selector and probe caveats.
|
||||
|
||||
### Option B — Gateway API + GitHub SSO (oauth2-proxy)
|
||||
|
||||
@@ -416,77 +504,86 @@ front of **oauth2-proxy**, which authenticates the browser against GitHub
|
||||
`X-Forwarded-User` set. The gateway trusts that header only when
|
||||
`AUTH_PROXY_HEADER` is set (SSO is **opt-in**).
|
||||
|
||||
Prerequisites:
|
||||
|
||||
- Gateway API CRDs and a Gateway implementation; the manifests assume
|
||||
`gatewayClassName: traefik` (the CRD version is pinned as
|
||||
`GATEWAY_API_VERSION` in `versions.env`).
|
||||
- A GitHub OAuth App whose homepage/callback match the public FQDN
|
||||
(`https://<host>/` and `https://<host>/oauth2/callback`).
|
||||
- The oauth2-proxy credentials Secret (template:
|
||||
`73-oauth2-proxy-secret.yaml.example`) and a TLS Secret for the Gateway
|
||||
listener (template: `74-tls-secret.yaml.example`).
|
||||
|
||||
Steps:
|
||||
|
||||
```bash
|
||||
# 1. Real Secrets; the committed *.yaml.example files are templates only.
|
||||
kubectl -n dsh-demo create secret generic oauth2-proxy \
|
||||
--from-literal=client-id='<GitHub OAuth App client ID>' \
|
||||
--from-literal=client-secret='<GitHub OAuth App client secret>' \
|
||||
--from-literal=cookie-secret="$(python3 -c 'import secrets; print(secrets.token_urlsafe(32))')"
|
||||
kubectl -n dsh-demo create secret tls dsh-demo-tls --cert=tls.crt --key=tls.key
|
||||
|
||||
# 2. Replace dsh-demo.example.com (70/71/72) and your-github-org (72).
|
||||
|
||||
# 3. Enable SSO on the gateway and trust the public host in the template.
|
||||
kubectl -n dsh-demo set env deploy/dsh-gateway \
|
||||
AUTH_PROXY_HEADER=X-Forwarded-User COOKIE_SECURE=true
|
||||
# ... and set DSH_TRUSTED_HOSTS=dsh-demo.example.com in
|
||||
# manifests/10-sandboxtemplate.yaml, then re-apply the template.
|
||||
|
||||
# 4. Apply the variant.
|
||||
kubectl apply -f manifests/exposure/gateway-api-sso/
|
||||
```
|
||||
|
||||
Verification:
|
||||
|
||||
```bash
|
||||
kubectl get gateway dsh-gateway -n dsh-demo -o yaml # Programmed=True
|
||||
kubectl get httproute -n dsh-demo -o wide # backend oauth2-proxy:4180, Accepted
|
||||
kubectl get deploy oauth2-proxy -n dsh-demo # 1/1 Ready
|
||||
kubectl get endpoints oauth2-proxy -n dsh-demo # backend present
|
||||
kubectl -n dsh-demo logs deploy/oauth2-proxy --tail=50 # no config errors
|
||||
```
|
||||
|
||||
Browser check: open `https://<host>/` — you are redirected to GitHub, and after
|
||||
signing in as a member of the allowed organization the gateway auto-provisions
|
||||
your sandbox (the first page can show `Provisioning`) and connects you. A
|
||||
non-member is denied before reaching the gateway. GitHub logins are lowercase
|
||||
DNS-1123 labels, so they map 1:1 onto the gateway's `user` validation. The SSO
|
||||
landing page also offers **Recreate sandbox** (`POST /recreate`), which deletes
|
||||
the claim — wiping the workspace PVC — and provisions a fresh one.
|
||||
|
||||
Sign-out: the gateway `/logout` clears `dsh_gw_session`; oauth2-proxy's session
|
||||
can be cleared at `/oauth2/sign_out`.
|
||||
1. Install Gateway API CRDs and a Gateway implementation; the manifests assume
|
||||
`gatewayClassName: traefik` (the CRD version is `GATEWAY_API_VERSION` in
|
||||
`versions.env`).
|
||||
2. Create a GitHub OAuth App whose homepage/callback match the public FQDN, and
|
||||
create the oauth2-proxy and TLS Secrets from
|
||||
`73-oauth2-proxy-secret.yaml.example` and `74-tls-secret.yaml.example`.
|
||||
3. Replace `dsh-demo.example.com` and `your-github-org` in the variant
|
||||
manifests, set `DSH_TRUSTED_HOSTS` in the template, and enable SSO:
|
||||
`kubectl -n dsh-demo set env deploy/dsh-gateway AUTH_PROXY_HEADER=X-Forwarded-User COOKIE_SECURE=true ALLOW_INSECURE_COOKIES-`.
|
||||
4. `kubectl apply -f manifests/exposure/gateway-api-sso/`; then run
|
||||
[`scripts/preflight-sso.sh`](scripts/preflight-sso.sh) and fix everything it
|
||||
reports **before** pointing real traffic at the Gateway. It verifies the
|
||||
Secrets (including a valid oauth2-proxy cookie secret length), the gateway's
|
||||
`AUTH_PROXY_HEADER`/`COOKIE_SECURE`, that the Gateway's TLS certificate
|
||||
reference resolves, and that no `your-github-org`/`dsh-demo.example.com`
|
||||
placeholder was left applied. Full verification steps are in the
|
||||
[exposure README](manifests/exposure/README.md).
|
||||
|
||||
> **Exposure without the variant NetworkPolicy is demo-only.** The SSO
|
||||
> variant's `75-networkpolicies.yaml` is not optional in practice: it is the
|
||||
> only thing that makes oauth2-proxy the sole path to the gateway. Applying
|
||||
> only the Gateway/HTTPRoute/oauth2-proxy objects leaves the trust boundary
|
||||
> unenforced, so anyone who can reach `dsh-gateway:8080` can send
|
||||
> `X-Forwarded-User` and impersonate the user they claim to be.
|
||||
>
|
||||
> **SSO trust boundary — read this before enabling it.** After SSO is on, the
|
||||
> `dsh-gateway` Service must be reachable **only** through oauth2-proxy (it is
|
||||
> ClusterIP). Anyone who can reach `dsh-gateway:8080` directly can send
|
||||
> `X-Forwarded-User: <victim>` and be treated as that user. Never apply the
|
||||
> Ingress variant (or any other route to the Service) while SSO is enabled, and
|
||||
> do not expose the gateway Service with `kubectl port-forward` on a shared
|
||||
> machine while `AUTH_PROXY_HEADER` is set. To turn SSO off, remove the variable
|
||||
> ClusterIP). The variant's `75-networkpolicies.yaml` restricts pod traffic
|
||||
> accordingly: the gateway accepts traffic only from oauth2-proxy pods,
|
||||
> oauth2-proxy only from the Gateway implementation, and its egress only to DNS,
|
||||
> public HTTPS and the gateway. The gateway's own egress (Kubernetes API,
|
||||
> sandbox pods) is not restricted by that file because the API-server endpoint is
|
||||
> cluster-specific; add the
|
||||
> [gateway egress policy](docs/optional-hardening.md#gateway-egress-required-for-production)
|
||||
> for production. Anyone who can reach `dsh-gateway:8080` directly can send
|
||||
> `X-Forwarded-User: <victim>`, so never add a second path. To turn SSO off,
|
||||
> remove the variable
|
||||
> (`kubectl -n dsh-demo set env deploy/dsh-gateway AUTH_PROXY_HEADER-`) and
|
||||
> delete the SSO variant; the gateway falls back to bearer-token/session auth.
|
||||
> delete the SSO variant.
|
||||
|
||||
### Switching variants
|
||||
|
||||
Delete the variant you had before applying the other one
|
||||
(`kubectl delete -f manifests/exposure/ingress/`), revert
|
||||
Delete the variant you had before applying the other one, revert
|
||||
`AUTH_PROXY_HEADER`/`COOKIE_SECURE`, and verify with
|
||||
`kubectl -n dsh-demo get ingress,gateway,httproute` that exactly one path to the
|
||||
gateway remains.
|
||||
gateway remains. Full commands and verification are in the
|
||||
[exposure README](manifests/exposure/README.md).
|
||||
|
||||
## Optional hardening
|
||||
|
||||
The base walkthrough is deliberately portable, which means it is **not
|
||||
hardened**: without the policies and gVisor it is demo-only, not a safe
|
||||
multi-tenant platform. [Optional hardening](docs/optional-hardening.md) has the
|
||||
settings and a production checklist. In short:
|
||||
|
||||
- **gVisor** — uncomment `runtimeClassName: gvisor` before applying, or patch an
|
||||
existing install. Requires a `gvisor` RuntimeClass and capable nodes.
|
||||
`kubectl port-forward` does not work under gVisor; the gateway path does.
|
||||
Required for untrusted agents in production.
|
||||
- **nodeSelector** — uncomment and replace the placeholder node label to pin
|
||||
sandboxes and the gateway to a dedicated pool; add a toleration for tainted
|
||||
pools.
|
||||
- **Sandbox NetworkPolicy** —
|
||||
`kubectl apply -f manifests/optional/network-policy/` allows only
|
||||
gateway→sandbox on `3080` and sandbox→DNS/public `443`. Required for
|
||||
production; exposure without it is demo-only.
|
||||
- **Gateway egress policy** — the gateway needs DNS, the Kubernetes API and
|
||||
`3080` to sandbox pods. It is not shipped because the API-server endpoint is
|
||||
cluster-specific; use the
|
||||
[recipe](docs/optional-hardening.md#gateway-egress-required-for-production).
|
||||
Required for production.
|
||||
- **Provider keys** — create `dsh-provider-keys-extra` as shown in step 1.
|
||||
- **Failed-Pod recovery** — uncomment `failedPodPolicy: Recreate` so the
|
||||
controller replaces a `Failed` sandbox Pod (node eviction) instead of parking
|
||||
the sandbox at `Finished=True`. Requires agent-sandbox v1.0.8+.
|
||||
|
||||
The **exposure variants' NetworkPolicies** are not optional in production
|
||||
either; see step 9. Apply each setting independently and re-run the walkthrough.
|
||||
`kubectl apply -f manifests/` is non-recursive, so the optional manifests are not
|
||||
applied by the base step.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
@@ -495,54 +592,101 @@ gateway remains.
|
||||
| Pod `CrashLoopBackOff`, probes failing with `401` | Probe path is `/`; change readiness/liveness to `/favicon.svg` |
|
||||
| Browser shows `401` from DSH through the gateway | Gateway handshake failed — check `kubectl -n dsh-demo logs deploy/dsh-gateway` and that RBAC grants `pods/log` |
|
||||
| `502` / `upstream unavailable` | Sandbox pod is not Ready (wake timed out) or the Service DNS is stale; check `kubectl get sandbox`, `WAKE_TIMEOUT` |
|
||||
| `503` with `Retry-After` | The launch token was not found in the pod logs yet; the gateway retries after a short delay. Check the pod logs and `DSH_LOG_TAIL_LINES` |
|
||||
| `503` with `Retry-After` | Either the launch token was not found in the pod logs yet (the gateway retries after a short delay; check the pod logs and `DSH_LOG_TAIL_LINES`) or the user hit `MAX_IN_FLIGHT_PER_USER` (default 32 concurrent requests/WebSockets); reduce client concurrency or raise the limit deliberately |
|
||||
| `429` with `Retry-After` on signup/login | The per-client auth rate limit (`AUTH_RATE_LIMIT`, default explicitly `30` per `AUTH_RATE_WINDOW=60` s) rejected the attempt before reading the body. Wait for the window, or raise the limit; behind a proxy make sure the real client IP is the rightmost `X-Forwarded-For` entry, not a shared NAT address |
|
||||
| `403` on signup: invite required/invalid | `SIGNUP_INVITE_CODE` is configured. Send the code as form field `invite` or header `X-Signup-Invite`; the landing page shows the field when enabled. A wrong code is rejected before the user lookup |
|
||||
| Gateway pod `CrashLoopBackOff` with `refusing to start with an insecure session cookie` | Intentional guard: plain HTTP is only allowed when explicitly acknowledged. For the local demo set `ALLOW_INSECURE_COOKIES=1` (the base manifest does); for HTTPS set `COOKIE_SECURE=true` and remove `ALLOW_INSECURE_COOKIES` |
|
||||
| Every browser is logged out after a gateway rollout/restart | Expected: sessions are opaque ids in gateway memory and a restart drops them. Sign in again; a production deployment needs shared session state or per-request re-authentication. The DSH sandboxes and their data are unaffected |
|
||||
| WebSocket closes with `1011` for a busy client | `MAX_IN_FLIGHT_PER_USER` (per-user connection cap) or an upstream/handshake failure closed the socket; check the gateway logs to tell them apart |
|
||||
| WebSocket closes after a very large message from DSH | A message received from DSH exceeded `WS_MAX_MESSAGE_BYTES` (16 MiB), so the relay closed it instead of buffering; raise the limit only if DSH legitimately sends larger frames. Browser-to-DSH frames are bounded by uvicorn's own limit |
|
||||
| `scripts/preflight-sso.sh` exits non-zero | It names the failing item: a missing Secret/key, a bad oauth2-proxy cookie-secret length, missing `AUTH_PROXY_HEADER`/`COOKIE_SECURE`, a dangling Gateway TLS reference, a leftover `your-github-org`/`dsh-demo.example.com` placeholder, or a second path to the gateway (an Ingress). Fix all items before exposing the SSO variant |
|
||||
| `409` on signup | User already has a claim; use `POST /login` with the existing token or delete the user first |
|
||||
| Cookie set but browser keeps getting `401` | `COOKIE_SECURE=true` while serving over plain HTTP (secure cookies are not sent); use HTTPS or set `false` for local demos |
|
||||
| Cookie set but browser keeps getting `401` | `COOKIE_SECURE=true` while serving over plain HTTP (secure cookies are not sent); use HTTPS or set `false` **and** `ALLOW_INSECURE_COOKIES=1` for local demos |
|
||||
| WebSocket closes immediately (`4401`) | Missing/expired gateway cookie, or the upgrade's `Origin` does not match `Host` (CSWSH check); DSH rejecting `Host` instead surfaces as close `1011` — add the host to `DSH_TRUSTED_HOSTS` |
|
||||
| Console spams `[connection] connection lost, retry #N` | The upstream WS upgrade is being rejected, surfaced as close `1011`. The gateway dials the sandbox **pod IP**, forwards no `Host`/`Origin`, and handshakes under the same `pod-IP:3080` authority DSH binds its cookie to; an older gateway image, or a sandbox with an empty `status.podIPs`, hits this loop |
|
||||
| `manifest.webmanifest` returns `401` in the console | Browsers fetch the Web App Manifest without credentials, so the gateway's session cookie is absent and the request fails closed. The UI is unaffected; only the PWA install prompt is lost |
|
||||
| Signup is slow / `202` then `Waking` | Warm pool exhausted (`kubectl get sandboxes -l agents.x-k8s.io/warm-pool-sandbox`); scale the pool |
|
||||
| `403` from the Kubernetes API in gateway logs | The gateway ServiceAccount is missing claim/sandbox/pod-log verbs (see `manifests/50-gateway.yaml`) |
|
||||
| Gateway pod `ErrImagePull` / `ImagePullBackOff` (especially on kind) | The manifest uses `imagePullPolicy: IfNotPresent` with the release identity. Build with `make gateway`, retag to exactly the manifest's image, `kind load docker-image <that-ref>`, or set the Deployment image to your local tag / pinned digest. On a real cluster, make sure the GHCR package is readable (public or a pull secret) |
|
||||
| Gateway pod `ErrImagePull` / `ImagePullBackOff` (especially on kind) | The manifest uses `imagePullPolicy: IfNotPresent` with the release identity. Build with `make -C ../.. gateway`, retag to exactly the manifest's image, `kind load docker-image <that-ref>`, or set the Deployment image to your local tag / pinned digest. On a real cluster, make sure the GHCR package is readable (public or a pull secret) |
|
||||
| Step 3: `pod "$SB" not found` after deploying the gateway | The claim was suspended by the idle sweeper. Claims created by the gateway carry `aaas.example.com/managed-by=dsh-gateway`; hand-applied claims (steps 1–7) do not and are never swept. Resume the sandbox: `kubectl -n dsh-demo patch sandbox "$SB" --type merge -p '{"spec":{"operatingMode":"Running"}}'` |
|
||||
| Resume crash-loops: `credentials-local: … is readable beyond its owner (mode 660)` | The pod template lacks `fsGroupChangePolicy: OnRootMismatch`, so the kubelet's default fsGroup policy rewrote `.credentials.yaml` to `0660`; DSH requires `0600` and refuses to start. New installs get the policy from the template. For an existing sandbox: suspend it, run a throwaway pod on `dsh-home-$SB` to `chmod 600 /home/node/.dsh/.credentials.yaml`, patch the sandbox's `podTemplate` with the policy, then resume |
|
||||
| Agents get `Permission denied` writing to `/workspace` on a fresh or resumed pod | The `/workspace` mount is the `dsh-home` PVC's `workspace/` subPath, and the kubelet is known to leave a `subPath` directory at `0755` if it is interrupted between creating and chmodding it ([kubernetes#142464](https://github.com/kubernetes/kubernetes/issues/142464)); later mounts never correct it. Suspend the sandbox, run a throwaway pod on `dsh-home-$SB` mounted at `/data`, `chmod 2775 /data/workspace` (do **not** chown/chmod recursively — that rewrites `.credentials.yaml`), then resume. If you recreate the sandbox instead, its current `/workspace` contents go with the PVC |
|
||||
| Every bash command fails: `SANDBOX_UNAVAILABLE`, `sandbox mode "workspace-write" is requested but no sandbox backend is usable` | DSH's own command sandbox found neither backend: `bubblewrap` ships in the image but is unsetuid, so it needs an unprivileged user namespace this hardened pod does not grant, and no Landlock runner exists on this kernel/runtime — the confined executor fails closed. The template sets `DSH_PERMISSION_MODE=danger-full-access` to keep agents usable inside the pod boundary. To restore confinement, confirm a backend inside the pod (`bwrap --ro-bind / / true`; `grep -o landlock /sys/kernel/security/lsm`), set `DSH_PERMISSION_MODE=workspace-write` (or `read-only`), and recreate the pod — the value is stamped at creation. A session that explicitly chose a mode keeps that choice |
|
||||
| SSO: GitHub login succeeds but the gateway asks for a token / returns `401` to `/recreate` | `AUTH_PROXY_HEADER` is not set on `deploy/dsh-gateway` (SSO is opt-in). Set `AUTH_PROXY_HEADER=X-Forwarded-User` and confirm the request actually reaches the gateway through oauth2-proxy |
|
||||
| SSO: gateway treats a caller as another user / bad `X-Forwarded-User` accepted | Trust boundary violation: something other than oauth2-proxy can reach `dsh-gateway:8080`. Remove the second path (Ingress/Gateway/port-forward), keep the Service ClusterIP, and only re-enable SSO once the proxy is the sole entry point. Never put a trusted identity header in front of a service that is directly reachable |
|
||||
| SSO: oauth2-proxy `cookie_secret must be 16, 24, or 32 bytes ... but is 44 bytes` | The cookie secret was generated as standard base64. Use URL-safe unpadded base64 (see `73-oauth2-proxy-secret.yaml.example`) |
|
||||
| SSO: redirect loop (`ERR_TOO_MANY_REDIRECTS`) | oauth2-proxy `--redirect-url` / callback does not match the public FQDN, or TLS is not terminating at the Gateway listener; fix the OAuth App URLs and the manifest placeholders |
|
||||
| SSO: DSH rejects the public host behind the proxy | Add the public hostname to `DSH_TRUSTED_HOSTS` in `manifests/10-sandboxtemplate.yaml`; the gateway forwards the browser's `Host` |
|
||||
| Pods `Pending` after setting `nodeSelector` | No node matches the label, the pool is full, or a tainted pool needs a toleration; inspect `kubectl describe pod` and the node labels |
|
||||
| gVisor: `kubectl port-forward` fails or the connection is refused | Port-forward is incompatible with gVisor; reach DSH through the gateway (or a reachable Service/Sandbox Router) |
|
||||
| Sandbox cannot reach a model API after applying the optional NetworkPolicy | The policy allows DNS and public TCP `443` only; a proxy, custom base URL, port `80` or non-443 endpoint needs an explicit egress rule |
|
||||
| Gateway cannot reach a sandbox after applying an exposure NetworkPolicy | Check that the CNI enforces NetworkPolicy and that the Traefik/oauth2-proxy pod labels match; kubelet probes may also need the node CIDR |
|
||||
| New provider key not visible in a sandbox | Pod env is stamped at creation: delete the warm-pool spares and recreate claimed pods. A key saved in DSH's credential store also wins over env |
|
||||
| Folder picker opens on `/home/node` instead of the workspace | The sandbox was created before the template set `HOME=/workspace`, and pod env is stamped at creation. Verify with `kubectl -n dsh-demo exec "$SB" -- printenv HOME`; patch the sandbox's `podTemplate` with the env (or recreate the sandbox — the PVC keeps DSH state and workspace) and resume |
|
||||
| File explorer and `ls /workspace` show `.cache`, `.config`, `.local`, `.npm`, `.agents` | Expected with `HOME=/workspace`: `~`-relative tool caches and DSH's global agent instructions/skills live on the PVC inside the workspace. Remove the `HOME` env from the template (and recreate the sandbox) if you would rather keep the picker on the ephemeral image home; point caches elsewhere per tool if only they are unwanted |
|
||||
| Sandbox reports `Finished=True` and every gateway request fails | The sandbox Pod reached phase `Failed` (node eviction, spot reclaim) and the default `Retain` keeps it. Delete the Pod so the controller recreates it, or set `failedPodPolicy: Recreate` (agent-sandbox v1.0.8+; see optional hardening) |
|
||||
|
||||
## Limitations
|
||||
|
||||
Deliberately **not** production code:
|
||||
|
||||
- **Single replica.** Idle clocks, in-flight tracking, and the handshake cache
|
||||
are in memory. Running two gateway replicas would double-suspend or lose wake
|
||||
tracking; scale up only after moving that state to a shared store.
|
||||
- **Single replica.** Idle clocks, in-flight tracking, the handshake cache and
|
||||
the browser session table are in memory. Running two gateway replicas would
|
||||
double-suspend or lose wake tracking, and sessions would land on the wrong
|
||||
replica; scale up only after moving that state to a shared store.
|
||||
- **Sessions die with the process.** Browser sessions are opaque ids held in
|
||||
gateway memory (revocable on logout/delete/recreate, but not durable). A
|
||||
gateway restart logs every browser out. There is no SSO refresh-token or
|
||||
password store; identity is re-established by oauth2-proxy on the next
|
||||
request.
|
||||
- **Abuse controls are per-process and coarse.** `AUTH_RATE_LIMIT`, the signup
|
||||
invite code, `MAX_IN_FLIGHT_PER_USER` and `WS_MAX_MESSAGE_BYTES` limit casual
|
||||
abuse, but they are not a substitute for ingress rate limiting, WAF/quota, or
|
||||
an audit log, and the in-memory rate buckets reset on restart.
|
||||
- **Log scraping.** The launch token is recovered from pod logs. Log rotation, a
|
||||
restarted pod, or a changed DSH log format can break the handshake until the
|
||||
next `401` refresh.
|
||||
- **Shared provider key.** `DEEPSEEK_API_KEY` is shared by all tenants via the
|
||||
template. Real multi-tenancy needs per-tenant credentials or an egress proxy.
|
||||
- **Shared provider keys.** `DEEPSEEK_API_KEY` and any keys in the optional
|
||||
`dsh-provider-keys-extra` Secret are shared by all tenants via the template.
|
||||
Tenants can override a key through DSH's stored credential file on their PVC,
|
||||
but the platform does not manage that. Real multi-tenancy needs per-tenant
|
||||
credentials or an egress proxy.
|
||||
- **No TLS in the gateway.** Terminate TLS at an Ingress or Gateway and set
|
||||
`COOKIE_SECURE=true`.
|
||||
- **SSO trust is hop-based.** `AUTH_PROXY_HEADER` trusts whoever sets the header;
|
||||
the only guarantee is that the gateway Service is not directly reachable.
|
||||
There is no signature, mTLS, or NetworkPolicy enforcing that in this example.
|
||||
`COOKIE_SECURE=true`, then remove `ALLOW_INSECURE_COOKIES`. The base manifest
|
||||
runs plain HTTP for the port-forward demo and is demo-only.
|
||||
- **SSO trust is hop-based.** `AUTH_PROXY_HEADER` trusts whoever sets the header.
|
||||
The SSO variant ships `75-networkpolicies.yaml`, which restricts the gateway
|
||||
to oauth2-proxy and oauth2-proxy egress to the gateway/DNS/public HTTPS; a
|
||||
custom exposure without that policy has no signature, mTLS or network
|
||||
enforcement. Exposure without the policy is demo-only.
|
||||
- **DSH's inner file sandbox is off here.** The template sets
|
||||
`DSH_PERMISSION_MODE=danger-full-access` because neither backend of DSH's own
|
||||
command sandbox is usable in this pod: `bubblewrap` is installed in the image
|
||||
but is unsetuid, so it needs an unprivileged user namespace that the pod's
|
||||
seccomp/capability surface does not grant (`capabilities.drop: ["ALL"]`), and
|
||||
the kernel/runtime exposes no Landlock runner. The executor therefore fails
|
||||
closed with `SANDBOX_UNAVAILABLE` and every bash command fails. The per-user
|
||||
pod remains the boundary — non-root,
|
||||
capabilities dropped, `seccompProfile: RuntimeDefault`, its own PVC and Service
|
||||
— but an agent command can now read its own `DSH_HOME` (including
|
||||
`.credentials.yaml`) and write anywhere in the container. Run sandboxes under
|
||||
gVisor in production. On hosts where a backend works, set `workspace-write`
|
||||
(or `read-only`) and verify before relying on the inner policy.
|
||||
- **Isolation is opt-in.** gVisor, the sandbox NetworkPolicy, the gateway egress
|
||||
policy and the exposure policies are opt-in; without them, sandboxes share the
|
||||
host kernel and can reach the cluster network. The base example is a demo, not
|
||||
a hardened multi-tenant platform. See the
|
||||
[production checklist](docs/optional-hardening.md#6-production-checklist).
|
||||
- **OAuth org is the only authorization.** Any GitHub user in the configured
|
||||
organization gets an auto-provisioned sandbox; there is no per-user allow-list,
|
||||
quota, or admin API.
|
||||
quota, or admin API. `SIGNUP_INVITE_CODE` gates the gateway's own signup form,
|
||||
not SSO auto-provisioning.
|
||||
- **In-memory user directory.** Users are claims + annotations; there is no
|
||||
database, password reset, or admin API. Tokens are stored only as SHA-256
|
||||
hashes in claim annotations.
|
||||
- **Bearer tokens in URLs.** Avoid putting the signup token in a URL/log; the
|
||||
browser flow uses the `HttpOnly` cookie instead.
|
||||
|
||||
## Cost model
|
||||
|
||||
The same economics as the upstream pattern: suspend-when-idle plus warm pools turn
|
||||
an always-on instance into a sub-second cold-start pause per return-after-absence,
|
||||
at a fraction of the cost.
|
||||
browser flow uses the `HttpOnly` cookie instead. The cookie no longer contains
|
||||
the token, and a cookie value is never accepted as a bearer token.
|
||||
|
||||
## Cleanup
|
||||
|
||||
@@ -564,3 +708,8 @@ delete the Secrets with the namespace and revoke the OAuth App in GitHub.
|
||||
- [Architecture](docs/architecture.md) — container/sequence diagrams, ADRs,
|
||||
security and RBAC, failure modes, evolution path.
|
||||
- [Exposure README](manifests/exposure/README.md) — the two variants in detail.
|
||||
- [SSO preflight](scripts/preflight-sso.sh) — read-only checks for the SSO
|
||||
variant (Secrets, gateway env, TLS reference, placeholders, second paths).
|
||||
- [Optional hardening](docs/optional-hardening.md) — gVisor, nodeSelector,
|
||||
NetworkPolicy, extra provider keys and failed-Pod recovery, with verification,
|
||||
the production checklist and rollback.
|
||||
|
||||
@@ -6,7 +6,8 @@ Kubernetes, and DSH Web fit together, and why the design looks the way it does.
|
||||
|
||||
Scope: one Kubernetes namespace (`dsh-demo`), one shared DSH Web image, one gateway deployment,
|
||||
and one DSH Web pod + PVC per user. The cluster is assumed to run
|
||||
[agent-sandbox](https://github.com/kubernetes-sigs/agent-sandbox) v1.0.7 with the extensions
|
||||
[agent-sandbox](https://github.com/kubernetes-sigs/agent-sandbox) with the pinned
|
||||
`AGENT_SANDBOX_VERSION` from [`versions.env`](../../../versions.env) and the extensions
|
||||
(`SandboxTemplate`, `SandboxWarmPool`, `SandboxClaim`).
|
||||
|
||||
## Container view
|
||||
@@ -24,7 +25,7 @@ flowchart LR
|
||||
sb["Sandbox dsh-pool-xxxx<br/>spec.operatingMode"]
|
||||
svc["Headless Service<br/>status.serviceFQDN"]
|
||||
pod["DSH Web pod<br/>node process :3080"]
|
||||
pvc[("PVC dsh-home-SANDBOX<br/>/home/node/.dsh")]
|
||||
pvc[("PVC dsh-home-SANDBOX<br/>/home/node/.dsh + /workspace (subPath)")]
|
||||
end
|
||||
|
||||
browser -->|"HTTP + WebSocket<br/>/u/USER/**"| gw
|
||||
@@ -46,7 +47,7 @@ Responsibilities:
|
||||
| **Gateway** | Signup/login, session auth, `SandboxClaim` lifecycle, wake-on-connect, idle sweeper, HTTP/SSE/WebSocket reverse proxy, DSH launch-token handshake. |
|
||||
| **Kubernetes API** | Source of truth for claims, sandboxes, and pods; also the log source for the DSH launch token. |
|
||||
| **agent-sandbox controller** | Adopts warm spares on claim, flips `operatingMode` (Running/Suspended), garbage-collects sandbox + PVC on claim delete. |
|
||||
| **DSH Web pod** | One `dsh web` process per user, state on the per-user PVC, listening on `3080`. |
|
||||
| **DSH Web pod** | One `dsh web` process per user, DSH state and the agent workspace on the per-user PVC (`/home/node/.dsh` + `/workspace`), listening on `3080`. |
|
||||
| **Warm pool** | Keeps `N` Ready sandboxes so signup is a label flip instead of a cold start. |
|
||||
|
||||
## Exposure variants
|
||||
@@ -67,6 +68,15 @@ as that user. The gateway ignores the header unless `AUTH_PROXY_HEADER` is set.
|
||||
[exposure README](../manifests/exposure/README.md) and the
|
||||
[gateway README](../gateway/README.md#sso-opt-in).
|
||||
|
||||
Each variant also ships a NetworkPolicy: `ingress/61-networkpolicy.yaml` limits gateway ingress
|
||||
to Traefik pods, and `gateway-api-sso/75-networkpolicies.yaml` limits the gateway to
|
||||
oauth2-proxy, oauth2-proxy ingress to the Gateway implementation, and oauth2-proxy egress to
|
||||
DNS/public HTTPS/the gateway. The SSO policy turns the trust boundary into pod-traffic
|
||||
enforcement. These policies are **required for production**: exposing the gateway by applying
|
||||
only the Gateway/HTTPRoute/Ingress objects is demo-only. The sandbox-side policy and the
|
||||
gateway's own egress policy are separate (the API-server endpoint is cluster-specific); see
|
||||
[optional hardening](optional-hardening.md#gateway-egress-required-for-production).
|
||||
|
||||
## Flows
|
||||
|
||||
### Signup (browser)
|
||||
@@ -79,20 +89,22 @@ sequenceDiagram
|
||||
participant K as Kubernetes API
|
||||
participant C as agent-sandbox controller
|
||||
|
||||
B->>G: POST /users (form: user=bob, Origin)
|
||||
G->>G: validate user name and body size
|
||||
B->>G: POST /users (form: user=bob, invite, Origin)
|
||||
G->>G: rate limit, invite check, validate user name and body size
|
||||
G->>K: create SandboxClaim dsh-bob (warmPoolRef)
|
||||
K->>C: reconcile claim
|
||||
C-->>K: adopt spare, Ready, status.sandbox.name = dsh-pool-xxxx
|
||||
G->>K: read claim (status) + sandbox + pod
|
||||
G-->>B: 303 Location /u/bob/ + Set-Cookie dsh_gw_session
|
||||
G->>G: mint opaque sid, store sid -> (user, expiry)
|
||||
G-->>B: 303 Location /u/bob/ + Set-Cookie dsh_gw_session=<sid>
|
||||
Note over G: JSON clients instead get 201 (Ready),<br/>202 (accepted, still waking), or 409 (conflict)
|
||||
```
|
||||
|
||||
Form signup waits up to `WAKE_TIMEOUT` for the claim to become `Ready`, then redirects; the
|
||||
first proxied request also waits for `Ready` and performs the DSH handshake (below). A `409`
|
||||
means the user already has a claim, in which case `POST /login` (bearer token) or the existing
|
||||
cookie is the right path.
|
||||
cookie is the right path. The cookie value is an opaque session id: it is meaningless outside
|
||||
this gateway process and is never accepted as a bearer token.
|
||||
|
||||
### First request / proxy (with handshake)
|
||||
|
||||
@@ -104,8 +116,8 @@ sequenceDiagram
|
||||
participant K as Kubernetes API
|
||||
participant D as DSH Web pod
|
||||
|
||||
B->>G: GET /u/bob/ (Cookie dsh_gw_session)
|
||||
G->>G: authenticate cookie, user = bob
|
||||
B->>G: GET /u/bob/ (Cookie dsh_gw_session=<sid>)
|
||||
G->>G: look up sid -> session (valid, not expired), user = bob
|
||||
G->>K: resolve claim -> sandbox -> pod
|
||||
alt sandbox Suspended
|
||||
G->>K: patch operatingMode=Running (wake)
|
||||
@@ -128,6 +140,36 @@ Handshake cache behavior: one entry per `(user, pod UID, Host)`; refreshed once
|
||||
`401`; dropped when the sandbox is suspended or deleted. The launch token and the DSH auth
|
||||
cookie never reach the browser.
|
||||
|
||||
Inside `proxy_http` the order is auth, in-flight slot, sandbox resolution/wake, DSH handshake,
|
||||
body read, upstream send. An upload aborted mid-body may therefore already have paid for a wake
|
||||
and a handshake, but it never reaches DSH.
|
||||
|
||||
### Browser sessions
|
||||
|
||||
The browser never holds a DSH credential. On signup/login the gateway mints a random session id
|
||||
(`secrets.token_urlsafe(32)`), stores `sid -> Session(user, expires_at)` in memory, and sets it
|
||||
as the `HttpOnly`, `SameSite=Lax` cookie `dsh_gw_session` (`__Host-dsh_gw_session` when
|
||||
`COOKIE_SECURE=true`). Every browser request is resolved through that table; the cookie value is
|
||||
not a bearer token and cannot be replayed against another user's path.
|
||||
|
||||
Sessions end on explicit logout, user delete, `/recreate` (all sessions for that user) and
|
||||
expiry (`COOKIE_MAX_AGE`, default 7 days). Because the table is process memory, a gateway
|
||||
restart logs every browser out; scaling beyond one replica therefore requires a shared session
|
||||
store (or re-authentication on every request). The sweeper prunes expired entries and sessions
|
||||
for users whose claim no longer exists.
|
||||
|
||||
### Auth abuse controls and proxy limits
|
||||
|
||||
Unauthenticated endpoints are throttled before the body is read: `AUTH_RATE_LIMIT` attempts per
|
||||
client per `AUTH_RATE_WINDOW` (defaults 30/60s; keyed on the rightmost `X-Forwarded-For` entry
|
||||
when present, else the peer address), answered with `429` + `Retry-After`. When
|
||||
`SIGNUP_INVITE_CODE` is set, signup additionally requires a constant-time-matched code (form
|
||||
field or `X-Signup-Invite`) and answers `403` otherwise. Proxied traffic is capped per user
|
||||
(`MAX_IN_FLIGHT_PER_USER`, default 32 concurrent HTTP requests + WebSockets: `503` for HTTP,
|
||||
close `1011` for WebSocket) and per WebSocket message received from DSH
|
||||
(`WS_MAX_MESSAGE_BYTES`, 16 MiB). These are in-process guards for a single replica, not an ingress
|
||||
rate limiter or quota system.
|
||||
|
||||
### Wake-on-connect and idle suspend
|
||||
|
||||
```mermaid
|
||||
@@ -148,8 +190,21 @@ sequenceDiagram
|
||||
G-->>B: request held, then served
|
||||
```
|
||||
|
||||
Suspend deletes only the pod. The PVC keeps conversations, sessions, and config, so resume
|
||||
reattaches exactly the same state.
|
||||
Suspend deletes only the pod. The PVC keeps conversations, sessions, config **and the agent
|
||||
workspace**, so resume reattaches exactly the same state. Both trees live on one PVC: DSH state at
|
||||
`/home/node/.dsh` (the volume root) and the workspace at `/workspace` (a second mount of the same
|
||||
volume with `subPath: workspace`, which keeps `.credentials.yaml` out of the agent-visible tree).
|
||||
The idle sweeper never touches the volume; deleting the claim still garbage-collects it, so
|
||||
`POST /recreate` wipes state and workspace together.
|
||||
|
||||
**Workspace as home.** The template sets `HOME=/workspace`. DSH has no start-path setting for its
|
||||
directory picker: the browse backend lists `$HOME` when the dialog opens, so with the image default
|
||||
(`/home/node`, off the PVC) the picker would start on a throwaway directory. The override makes the
|
||||
picker open on the persisted workspace and makes `~` resolve there; the cost is that tool caches
|
||||
and config kept under `$HOME` (`npm`/`pnpm`, `uv`, `git`, `gh`, and DSH's global `~/.agents`
|
||||
instructions and skills) become dotfiles in the workspace. `DSH_HOME` stays `/home/node/.dsh`, so
|
||||
sessions, config and credentials remain outside the agent-visible tree. Pod env is stamped when a
|
||||
sandbox is created, so an existing sandbox keeps the old `HOME` until it is recreated.
|
||||
|
||||
**Ownership boundary.** The gateway shares its namespace with claims created outside it — the
|
||||
example's resource-layer walkthrough applies `dsh-alice` by hand, and every claim is named
|
||||
@@ -195,6 +250,14 @@ sequenceDiagram
|
||||
end
|
||||
```
|
||||
|
||||
An upgrade with a missing/expired session, or one that would exceed
|
||||
`MAX_IN_FLIGHT_PER_USER`, is accepted and then closed: `4401` for authentication, `1011` for
|
||||
the per-user socket cap. Messages received from DSH above `WS_MAX_MESSAGE_BYTES` (16 MiB) close
|
||||
the connection instead of being buffered; browser-to-DSH frames are bounded by uvicorn's own
|
||||
limit. Client-supplied `Forwarded`, `X-Forwarded-User`,
|
||||
`X-Auth-Request-*` and `x-forwarded-*`/`x-auth-request-*` headers are stripped on the HTTP and
|
||||
WebSocket paths; `UPSTREAM_HEADER_ALLOWLIST` replaces that denylist with an explicit allowlist.
|
||||
|
||||
## Decisions and trade-offs
|
||||
|
||||
### ADR-1: Starlette ASGI instead of a sync Flask proxy
|
||||
@@ -254,14 +317,21 @@ sequenceDiagram
|
||||
(`kubectl annotate sandboxclaim ...`) to keep being swept. Fine for a reference pattern, not
|
||||
for a product.
|
||||
|
||||
### ADR-6: Gateway cookie session instead of bearer tokens in the browser
|
||||
### ADR-6: Opaque server-side session cookie instead of bearer tokens in the browser
|
||||
|
||||
- **Decision:** browsers get `dsh_gw_session=<user>:<token>`, `HttpOnly`, `SameSite=Lax`
|
||||
(`Secure` when enabled); API clients may still use `Authorization: Bearer`.
|
||||
- **Why:** tokens never appear in URLs or JS-readable storage, and `SameSite=Lax` plus a strict
|
||||
`Origin`/`Referer` check on cookie-authenticated mutations blocks basic CSRF.
|
||||
- **Cost:** cookie-based auth needs TLS and correct `COOKIE_SECURE`; there is no signed-cookie
|
||||
layer, so the gateway trusts the token hash comparison for validation.
|
||||
- **Decision:** browsers get `dsh_gw_session=<sid>`, `HttpOnly`, `SameSite=Lax` (`Secure` when
|
||||
enabled); the sid maps to `Session(user, expires_at)` in gateway memory. API clients may still
|
||||
use `Authorization: Bearer`. The cookie name is `__Host-dsh_gw_session` when the Secure flag
|
||||
is on and no explicit `COOKIE_NAME` is set.
|
||||
- **Why:** tokens never appear in URLs or JS-readable storage, the cookie value is useless if
|
||||
leaked outside the gateway session table, sessions can be revoked server-side (logout,
|
||||
user delete, recreate, expiry), and a cookie cannot be reused as a bearer token for another
|
||||
user's path. `SameSite=Lax` plus a strict `Origin`/`Referer` check on cookie-authenticated
|
||||
mutations blocks basic CSRF.
|
||||
- **Cost:** stateful auth in a single process: a restart logs everyone out and horizontal
|
||||
scaling needs a shared session store. Cookie-based auth also needs TLS and the correct
|
||||
`COOKIE_SECURE` setting; the gateway refuses to start with `COOKIE_SECURE=false` unless
|
||||
`ALLOW_INSECURE_COOKIES=1` is set for a local HTTP demo.
|
||||
|
||||
## Security and RBAC
|
||||
|
||||
@@ -280,20 +350,52 @@ Notes and limits:
|
||||
permission is scoped to one namespace, but anyone who can read gateway memory or logs can
|
||||
impersonate users. Prefer the upstream token API when available.
|
||||
- **No cluster-wide access, no secret reads.** The gateway does not read `dsh-provider-keys`.
|
||||
- **Browser-facing hardening:** `HttpOnly`/`SameSite=Lax`/optional `Secure` cookies,
|
||||
timing-safe SHA-256 token comparison, strict same-origin check on cookie-auth mutations,
|
||||
request-body limit (`MAX_BODY_MB`), hop-by-hop header stripping, upstream `Set-Cookie`
|
||||
dropped, DSH `Location` headers rewritten under `/u/<user>/`.
|
||||
(It does read the optional signup invite code if it is mounted as an env var.)
|
||||
- **Browser-facing hardening:** opaque server-side sessions instead of token-bearing cookies,
|
||||
`HttpOnly`/`SameSite=Lax`/optional `Secure` (+ `__Host-` prefix when secure),
|
||||
timing-safe SHA-256 bearer-token comparison, timing-safe invite-code comparison, strict
|
||||
same-origin check on cookie-auth mutations, request-body limit (`MAX_BODY_MB`), hop-by-hop
|
||||
header stripping, client identity/hop headers (`Forwarded`, `X-Forwarded-User`,
|
||||
`X-Auth-Request-*`, `x-forwarded-*`/`x-auth-request-*`) stripped on HTTP and WebSocket,
|
||||
upstream `Set-Cookie` dropped, DSH `Location` headers rewritten under `/u/<user>/`.
|
||||
- **Session lifecycle:** login/signup mint an opaque sid; logout and user delete revoke it;
|
||||
`/recreate` revokes every session for that user; expiry (`COOKIE_MAX_AGE`, default 7 days) is
|
||||
enforced on lookup. The table is in memory, so a restart logs browsers out and a second
|
||||
replica needs shared state.
|
||||
- **Abuse controls (per replica):** `AUTH_RATE_LIMIT`/`AUTH_RATE_WINDOW` throttle
|
||||
unauthenticated signup/login per client with `429` + `Retry-After`; `SIGNUP_INVITE_CODE`
|
||||
gates signup; `MAX_IN_FLIGHT_PER_USER` caps concurrent proxied requests + WebSockets per user;
|
||||
`WS_MAX_MESSAGE_BYTES` caps messages received from DSH; browser-to-DSH frames are bounded by
|
||||
uvicorn. These are in-process guards, not an ingress rate limiter or audit log.
|
||||
- **Host allow-list:** DSH rejects unknown `Host` headers. The gateway forwards the browser's
|
||||
`Host`, so every public hostname must be in `DSH_TRUSTED_HOSTS`; `localhost`/`127.0.0.1` are
|
||||
always trusted.
|
||||
- **TLS is external.** The gateway serves plain HTTP; terminate TLS at the Ingress and set
|
||||
`COOKIE_SECURE=true`. Without TLS, session cookies are replayable on the network.
|
||||
`COOKIE_SECURE=true`. The base demo sets `COOKIE_SECURE=false` with
|
||||
`ALLOW_INSECURE_COOKIES=1` for the port-forward walkthrough; without TLS, session cookies are
|
||||
replayable on the network and that combination must never be exposed.
|
||||
- **Tenant isolation** is pod + PVC separation; all tenants share one namespace, one DSH binary,
|
||||
and one `DEEPSEEK_API_KEY`. Add NetworkPolicies and per-tenant credentials before calling it
|
||||
multi-tenant-grade.
|
||||
- **No gateway rate limiting.** Signup can create claims until the cluster or warm pool is
|
||||
exhausted; put limits at the ingress for real deployments.
|
||||
and the shared provider keys (`dsh-provider-keys` plus the optional
|
||||
`dsh-provider-keys-extra`). The base example is demo-only: production requires gVisor for the
|
||||
sandboxes, the sandbox NetworkPolicy, the gateway egress policy and the exposure NetworkPolicy,
|
||||
plus per-tenant credentials or an egress proxy. See the
|
||||
[production checklist](optional-hardening.md#6-production-checklist).
|
||||
- **DSH's inner file sandbox is disabled by deployment override.** The template sets
|
||||
`DSH_PERMISSION_MODE=danger-full-access` because the in-container backends are unusable here:
|
||||
`bubblewrap` ships in the image but is unsetuid and cannot create an unprivileged user namespace
|
||||
under the pod's seccomp/capability surface, and no Landlock runner is available; the confined
|
||||
executor would otherwise fail closed on every command. The pod remains the boundary, so an
|
||||
agent can read its own `DSH_HOME`. Hosts with a working backend can keep a confined mode;
|
||||
multi-tenant deployments must put the pod in gVisor and a NetworkPolicy.
|
||||
- **Optional hardening is opt-in.** gVisor, node selection, the sandbox/exposure NetworkPolicies,
|
||||
extra provider keys and failed-Pod recovery are documented in
|
||||
[optional hardening](optional-hardening.md). The base manifests stay portable (no gVisor
|
||||
RuntimeClass, no active node labels, no NetworkPolicy, no extra Secret required, and the
|
||||
controller's default `failedPodPolicy: Retain`).
|
||||
- **SSO is trust-on-hop.** `AUTH_PROXY_HEADER` makes the gateway trust the proxy-asserted
|
||||
identity; the SSO variant must remain the only path to the Service, and its NetworkPolicy is
|
||||
required, not optional. Run [`scripts/preflight-sso.sh`](../scripts/preflight-sso.sh) after
|
||||
applying it.
|
||||
|
||||
## Failure modes
|
||||
|
||||
@@ -302,10 +404,23 @@ Notes and limits:
|
||||
| DSH log line changed or rotated away | Proxy returns `503` (missing launch token) | Fix the parser or restart the pod so the token line is re-emitted |
|
||||
| Probe on `/` | Pod `CrashLoopBackOff`, `401` in probe logs | Use `/favicon.svg` for readiness/liveness (shipped templates do) |
|
||||
| Warm pool exhausted | JSON signup returns `202` (state `Waking`); form signup is slow | Increase `replicas` in the warm pool |
|
||||
| Gateway restart | Idle sandboxes stop suspending; handshake cache lost | Rebuilt lazily on next request; consider a shared store for production |
|
||||
| Gateway restart | Idle sandboxes stop suspending; handshake cache and browser sessions lost | Idle clocks and handshake cache rebuild lazily; sessions do not, so browsers must sign in again after a rollout. Use a shared session store for production |
|
||||
| Gateway configured `COOKIE_SECURE=false` without `ALLOW_INSECURE_COOKIES=1` | Gateway pod exits at startup / `CrashLoopBackOff` | Intentional guard against an insecure cookie on an HTTPS deployment; set `ALLOW_INSECURE_COOKIES=1` only for the local plain-HTTP demo, or set `COOKIE_SECURE=true` |
|
||||
| Auth rate limit hit | `429` + `Retry-After` on signup/login | Per-client in-memory bucket (`AUTH_RATE_LIMIT`/`AUTH_RATE_WINDOW`); wait for the window or tune it. Behind a proxy, keying uses the rightmost `X-Forwarded-For` entry |
|
||||
| Signup invite missing/wrong | `403` before the claim is created | `SIGNUP_INVITE_CODE` is set; send the `invite` form field or `X-Signup-Invite` header |
|
||||
| Per-user concurrency cap hit | HTTP `503` + `Retry-After`, or WebSocket close `1011` | `MAX_IN_FLIGHT_PER_USER` protects the single gateway from one busy tenant; reduce concurrency or raise the cap deliberately. These are per replica |
|
||||
| Oversized WebSocket message from DSH | Connection closes | `WS_MAX_MESSAGE_BYTES` (16 MiB) is enforced on upstream messages; raise it only if DSH legitimately needs larger frames |
|
||||
| Session expiry / revocation | `401` (HTTP) or close `4401` (WebSocket) | Expected after logout, user delete, `/recreate`, `COOKIE_MAX_AGE` expiry or a gateway restart; sign in again |
|
||||
| SSO variant applied without its NetworkPolicy (or with an Ingress) | Gateway accepts spoofed `X-Forwarded-User` | Demo-only configuration; keep `75-networkpolicies.yaml`, remove the second path, and run `scripts/preflight-sso.sh` |
|
||||
| Browser aborts a request (tab reload, back navigation) during a slow wake or handshake | No response reaches the client; nothing appears in the access log | If the abort lands before the response starts, the body read raises `ClientDisconnect` and the app answers `499` (`CLIENT_CLOSED_REQUEST`), which cannot reach the departed peer. Nothing reached DSH, no ERROR traceback is logged, and the in-flight slot is released. A request already streaming ends with its upstream status. |
|
||||
| DSH auth cookie expired mid-session | Proxy `401` once, then recovers | Gateway refreshes the handshake once |
|
||||
| Upstream pod name changed after resume | Cached cookie keyed by old pod UID is useless | Cache is invalidated on suspend; next request re-handshakes |
|
||||
| Two gateway replicas | Double suspend/wake races | Keep `replicas: 1` until state is externalized |
|
||||
| Optional NetworkPolicy blocks DNS/probes | Sandbox 502/503, pod restarts, failed `getent` | Check the CNI, DNS selector/NodeLocal address and kubelet-probe traffic; see the hardening doc |
|
||||
| gVisor + `kubectl port-forward` | Forward refuses or returns nothing | Port-forward is incompatible with gVisor; use the gateway or a reachable Service |
|
||||
| Provider key added but missing in the pod | Requests still fail with `MISSING_CREDENTIAL` | Pod env is stamped at creation; recreate spares/pods. A stored credential wins over env |
|
||||
| `nodeSelector` matches no node | Pod `Pending`, `0/N nodes are available` | Fix the label, pool capacity or taint/toleration; RuntimeClass scheduling may also constrain it |
|
||||
| Sandbox Pod reaches phase `Failed` (node eviction, spot reclaim) | Sandbox reports `Finished=True`; the gateway keeps failing against a dead Pod | Set `failedPodPolicy: Recreate` (agent-sandbox v1.0.8+) so the controller replaces the Pod with the same identity and PVCs; otherwise delete the Pod by hand. See the hardening doc |
|
||||
|
||||
## Upstream feature request: remove log scraping
|
||||
|
||||
@@ -328,10 +443,11 @@ single-retry refresh are the supported workaround.
|
||||
## Evolution path
|
||||
|
||||
1. Replace log scraping with the upstream token API (above), deleting the `pods/log` permission.
|
||||
2. Move idle clocks, in-flight tracking, and handshake cache to a shared store; then scale the
|
||||
gateway horizontally (sticky routing not required if state is shared).
|
||||
2. Move idle clocks, in-flight tracking, handshake cache **and sessions** to a shared store; then
|
||||
scale the gateway horizontally (sticky routing not required if state is shared).
|
||||
3. Add per-tenant API keys or an egress proxy so a single key is not shared by all sandboxes.
|
||||
4. Add NetworkPolicies (gateway → sandboxes only), TLS everywhere, rate limits, and audit
|
||||
logging at the ingress.
|
||||
4. Add NetworkPolicies (gateway → sandboxes only, plus the documented gateway egress policy),
|
||||
gVisor for sandbox pods, TLS everywhere, ingress-level rate limits/quota and audit logging.
|
||||
The gateway's in-process limits are not a substitute.
|
||||
5. If DSH grows absolute URLs or per-user cookies that cannot be rewritten, switch to
|
||||
per-user subdomains with wildcard DNS/TLS.
|
||||
|
||||
728
examples/dsh-agents-as-a-service/docs/optional-hardening.md
Normal file
728
examples/dsh-agents-as-a-service/docs/optional-hardening.md
Normal file
@@ -0,0 +1,728 @@
|
||||
# Optional hardening: gVisor, nodeSelector, NetworkPolicy, provider keys and failed-Pod recovery
|
||||
|
||||
The base example is deliberately portable: it runs on kind and GKE without a
|
||||
gVisor runtime, a tainted node pool or a NetworkPolicy-capable CNI, and it
|
||||
boots with only the required `dsh-provider-keys` Secret. That portability is
|
||||
exactly why the base manifests are **demo-only**. This document covers the
|
||||
controls you add for production and the steps to enable or roll them back.
|
||||
|
||||
**Production is not the base walkthrough.** At minimum, a production deployment
|
||||
needs: gVisor for the sandbox pods (or another kernel-isolation boundary), the
|
||||
sandbox NetworkPolicy, the gateway egress policy, the exposure variant's
|
||||
NetworkPolicy, TLS with `COOKIE_SECURE=true` (and no
|
||||
`ALLOW_INSECURE_COOKIES`), SSO (or an equivalent authenticating proxy) as the
|
||||
only path to the gateway, and per-tenant provider credentials/egress. The
|
||||
checklist at the end summarizes this; without the network and kernel layers the
|
||||
example is not tenant-isolated.
|
||||
|
||||
This document covers four **opt-in** controls plus failed-Pod recovery:
|
||||
|
||||
| Setting | Where it lives | What it changes |
|
||||
| --- | --- | --- |
|
||||
| gVisor | commented `runtimeClassName` in the SandboxTemplates and Deployments | puts DSH and/or the gateway in a userspace kernel; required for untrusted agents in production |
|
||||
| nodeSelector | commented `nodeSelector` in the same files | pins pods to a chosen node pool |
|
||||
| NetworkPolicy | `manifests/optional/network-policy/`, `manifests/exposure/*/`, plus the gateway egress recipe | limits sandbox, gateway and proxy connectivity; the sandbox and gateway-egress policies are required for production |
|
||||
| Provider keys | optional `dsh-provider-keys-extra` Secret | injects extra provider API keys into every sandbox |
|
||||
| Failed-Pod recovery | commented `failedPodPolicy` in the SandboxTemplates | replaces a Failed sandbox Pod instead of parking the sandbox |
|
||||
|
||||
Each setting is independent. Enable only what the cluster supports and test the
|
||||
walkthrough after each change.
|
||||
|
||||
## 1. gVisor
|
||||
|
||||
### What it changes
|
||||
|
||||
The DSH sandbox and the gateway (and optionally oauth2-proxy) run under the
|
||||
`gvisor` RuntimeClass. `agent-sandbox` passes `runtimeClassName` straight to the
|
||||
PodSpec; no CRD change is involved.
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- A `RuntimeClass` named `gvisor`:
|
||||
|
||||
```bash
|
||||
kubectl get runtimeclass gvisor -o yaml
|
||||
```
|
||||
|
||||
- Nodes capable of running it (`runsc` installed and registered with the
|
||||
container runtime). How you get there is cluster-specific:
|
||||
- GKE Sandbox: create a node pool with `--sandbox=type=gvisor`; GKE labels
|
||||
the nodes `sandbox.gke.io/runtime: gvisor`, taints them
|
||||
`sandbox.gke.io/runtime=gvisor:NoSchedule`, and its `gvisor` RuntimeClass
|
||||
applies the matching affinity and toleration automatically.
|
||||
- Self-managed: install gVisor, create the `gvisor` RuntimeClass with
|
||||
`handler: runsc`, and label/taint a dedicated node pool yourself.
|
||||
- A scheduling plan: a gVisor RuntimeClass without capable nodes makes pods
|
||||
fail to start. Use the RuntimeClass `scheduling` fields or the optional
|
||||
`nodeSelector`/toleration described in section 2.
|
||||
|
||||
### Enable it
|
||||
|
||||
**Fresh install:** uncomment `runtimeClassName: gvisor` before applying:
|
||||
|
||||
- [`manifests/10-sandboxtemplate.yaml`](../manifests/10-sandboxtemplate.yaml)
|
||||
- [`manifests/50-gateway.yaml`](../manifests/50-gateway.yaml)
|
||||
- [`manifests/exposure/gateway-api-sso/72-oauth2-proxy.yaml`](../manifests/exposure/gateway-api-sso/72-oauth2-proxy.yaml)
|
||||
(only with the SSO variant)
|
||||
- [minimal template](../../minimal-single-tenant/manifests/10-sandboxtemplate.yaml)
|
||||
|
||||
**Existing install:** a template edit does not update existing Sandboxes, and
|
||||
warm-pool spares use `OnReplenish`, so patch explicitly. Plan a maintenance
|
||||
window: claimed users are disconnected when their pod is recreated.
|
||||
|
||||
```bash
|
||||
# 1. The blueprint for future sandboxes.
|
||||
kubectl -n dsh-demo patch sandboxtemplate dsh-web --type merge \
|
||||
-p '{"spec":{"podTemplate":{"spec":{"runtimeClassName":"gvisor"}}}}'
|
||||
|
||||
# 2. Warm-pool spares: delete them; the pool replenishes from the template.
|
||||
kubectl -n dsh-demo delete sandboxes -l agents.x-k8s.io/warm-pool-sandbox
|
||||
|
||||
# 3. Claimed sandboxes: snapshot their pod template, then suspend/resume.
|
||||
for sb in $(kubectl -n dsh-demo get sandboxes \
|
||||
-l '!agents.x-k8s.io/warm-pool-sandbox' -o jsonpath='{.items[*].metadata.name}'); do
|
||||
kubectl -n dsh-demo patch sandbox "$sb" --type merge \
|
||||
-p '{"spec":{"podTemplate":{"spec":{"runtimeClassName":"gvisor"}}}}'
|
||||
kubectl -n dsh-demo patch sandbox "$sb" --type merge \
|
||||
-p '{"spec":{"operatingMode":"Suspended"}}'
|
||||
kubectl -n dsh-demo patch sandbox "$sb" --type merge \
|
||||
-p '{"spec":{"operatingMode":"Running"}}'
|
||||
done
|
||||
|
||||
# 4. The gateway (and oauth2-proxy when the SSO variant is installed).
|
||||
kubectl -n dsh-demo patch deploy dsh-gateway --type merge \
|
||||
-p '{"spec":{"template":{"spec":{"runtimeClassName":"gvisor"}}}}'
|
||||
kubectl -n dsh-demo rollout status deploy/dsh-gateway --timeout=180s
|
||||
# kubectl -n dsh-demo patch deploy oauth2-proxy --type merge \
|
||||
# -p '{"spec":{"template":{"spec":{"runtimeClassName":"gvisor"}}}}'
|
||||
```
|
||||
|
||||
If the pool's nodes are tainted and the RuntimeClass does not already add a
|
||||
toleration, add the matching `tolerations` block (section 2) in the same patch
|
||||
or in the manifests.
|
||||
|
||||
### Verify
|
||||
|
||||
```bash
|
||||
SB=$(kubectl -n dsh-demo get sandboxclaim dsh-alice -o jsonpath='{.status.sandbox.name}')
|
||||
POD=$(kubectl -n dsh-demo get sandbox "$SB" \
|
||||
-o jsonpath='{.metadata.annotations.agents\.x-k8s\.io/pod-name}'); POD=${POD:-$SB}
|
||||
|
||||
kubectl -n dsh-demo get pod "$POD" -o jsonpath='{.spec.runtimeClassName}{"\n"}'
|
||||
# gvisor
|
||||
kubectl -n dsh-demo exec "$POD" -- uname -r
|
||||
kubectl -n dsh-demo get node "$(kubectl -n dsh-demo get pod "$POD" -o jsonpath='{.spec.nodeName}')" \
|
||||
-o jsonpath='{.status.nodeInfo.kernelVersion}{"\n"}'
|
||||
```
|
||||
|
||||
The pod's `runtimeClassName` is authoritative. The kernel release reported by
|
||||
`uname -r` inside a gVisor pod is the sandbox kernel, not the host kernel; on
|
||||
GKE it looks like `4.19.0-gvisor`.
|
||||
|
||||
### Caveats
|
||||
|
||||
- **`kubectl port-forward` does not work with gVisor.** The multi-tenant
|
||||
gateway path (HTTP/WebSocket to the pod) works; the minimal example's
|
||||
port-forward quickstart does not. Use the gateway example or a reachable
|
||||
Service (NodePort/LoadBalancer/Sandbox Router) instead.
|
||||
- gVisor has performance overhead and does not implement every syscall. Test
|
||||
DSH and the agent tools you rely on (`git`, `gh`, `uv`, language toolchains).
|
||||
- Node scheduling is the hard part. A `gvisor` RuntimeClass on a cluster with
|
||||
no capable nodes leaves pods unschedulable or failing at container start.
|
||||
- Existing Sandboxes are snapshots. The loop above is required for every
|
||||
claimed sandbox; deleting the `SandboxClaim` would also delete its PVC and
|
||||
workspace data, so do not use claim re-creation as a shortcut.
|
||||
- A resumed sandbox starts cold; the startup probe gives it up to 120s.
|
||||
|
||||
### Roll back
|
||||
|
||||
```bash
|
||||
kubectl -n dsh-demo patch sandboxtemplate dsh-web --type merge \
|
||||
-p '{"spec":{"podTemplate":{"spec":{"runtimeClassName":null}}}}'
|
||||
for sb in $(kubectl -n dsh-demo get sandboxes -o jsonpath='{.items[*].metadata.name}'); do
|
||||
kubectl -n dsh-demo patch sandbox "$sb" --type merge \
|
||||
-p '{"spec":{"podTemplate":{"spec":{"runtimeClassName":null}}}}'
|
||||
done
|
||||
kubectl -n dsh-demo delete sandboxes -l agents.x-k8s.io/warm-pool-sandbox
|
||||
kubectl -n dsh-demo patch deploy dsh-gateway --type merge \
|
||||
-p '{"spec":{"template":{"spec":{"runtimeClassName":null}}}}'
|
||||
# kubectl -n dsh-demo patch deploy oauth2-proxy --type merge \
|
||||
# -p '{"spec":{"template":{"spec":{"runtimeClassName":null}}}}'
|
||||
```
|
||||
|
||||
Recreate the pods (delete spares, suspend/resume claimed sandboxes, let the
|
||||
Deployment roll out) so the cleared field takes effect.
|
||||
|
||||
## 2. nodeSelector
|
||||
|
||||
### What it changes
|
||||
|
||||
Pods get a `nodeSelector` so they land only on nodes carrying the chosen label.
|
||||
The same field is available in both SandboxTemplates and both Deployments;
|
||||
the shipped value is **commented out** because active cluster-specific
|
||||
scheduling does not belong in the repository.
|
||||
|
||||
### Choose a label
|
||||
|
||||
```bash
|
||||
kubectl get nodes --show-labels
|
||||
```
|
||||
|
||||
- **GKE Sandbox:** the nodes carry `sandbox.gke.io/runtime: gvisor` and are
|
||||
tainted `sandbox.gke.io/runtime=gvisor:NoSchedule`. With
|
||||
`runtimeClassName: gvisor`, GKE already schedules the pod onto those nodes;
|
||||
add a `nodeSelector` only when you need a specific sandbox pool or
|
||||
ComputeClass (`cloud.google.com/compute-class: <name>`).
|
||||
- **Self-managed:** label the pool you want, for example
|
||||
`dsh.example.com/node-pool: gvisor`, and use that label.
|
||||
- **General pinning:** any stable node-pool label works. Keep the real key and
|
||||
value in a local patch; the manifests use `dsh.example.com/node-pool` as an
|
||||
obvious placeholder.
|
||||
|
||||
### Enable it
|
||||
|
||||
Uncomment and replace the placeholder before a fresh apply:
|
||||
|
||||
```yaml
|
||||
nodeSelector:
|
||||
dsh.example.com/node-pool: gvisor
|
||||
```
|
||||
|
||||
For an existing install, patch the template, each claimed Sandbox and the
|
||||
Deployments, then recreate pods exactly as in section 1:
|
||||
|
||||
```bash
|
||||
kubectl -n dsh-demo patch sandboxtemplate dsh-web --type merge \
|
||||
-p '{"spec":{"podTemplate":{"spec":{"nodeSelector":{"dsh.example.com/node-pool":"gvisor"}}}}}'
|
||||
# ... same warm-pool delete / claimed-sandbox suspend-resume / Deployment
|
||||
# patch and rollout as in section 1, with nodeSelector instead of
|
||||
# runtimeClassName.
|
||||
```
|
||||
|
||||
If the selected nodes are tainted, add a matching toleration. This is the
|
||||
companion block; do **not** uncomment it unless the pool is actually tainted:
|
||||
|
||||
```yaml
|
||||
tolerations:
|
||||
- key: dsh.example.com/node-pool
|
||||
operator: Equal
|
||||
value: gvisor
|
||||
effect: NoSchedule
|
||||
```
|
||||
|
||||
GKE Sandbox's taint is `sandbox.gke.io/runtime=gvisor:NoSchedule`; the GKE
|
||||
`gvisor` RuntimeClass adds that toleration automatically, so a manual copy is
|
||||
only needed when you schedule without the RuntimeClass or with a different
|
||||
pool.
|
||||
|
||||
### Verify and troubleshoot
|
||||
|
||||
```bash
|
||||
kubectl -n dsh-demo get pod "$POD" -o wide
|
||||
kubectl -n dsh-demo get pod "$POD" -o jsonpath='{.spec.nodeSelector}{"\n"}'
|
||||
kubectl -n dsh-demo describe pod "$POD" | sed -n '/Events:/,$p'
|
||||
```
|
||||
|
||||
- `Pending` with `0/N nodes are available`: the label is wrong, the pool is
|
||||
full, or the pod needs a toleration for a taint.
|
||||
- A pod-level `nodeSelector` is merged with the RuntimeClass
|
||||
`scheduling.nodeSelector`. If both are set and conflict, no node matches.
|
||||
- Labels changed after scheduling do not move running pods. Patch/recreate.
|
||||
- A `nodeSelector` that excludes every gVisor node is a valid way to keep the
|
||||
gateway on a normal pool while the sandboxes run under gVisor.
|
||||
|
||||
### Roll back
|
||||
|
||||
Remove the field from the manifests, or patch it to `null`, then recreate the
|
||||
pods:
|
||||
|
||||
```bash
|
||||
kubectl -n dsh-demo patch sandboxtemplate dsh-web --type merge \
|
||||
-p '{"spec":{"podTemplate":{"spec":{"nodeSelector":null}}}}'
|
||||
# ... same per-Sandbox and Deployment loop as section 1.
|
||||
```
|
||||
|
||||
## 3. NetworkPolicy
|
||||
|
||||
The base example does **not** manage NetworkPolicy: the SandboxTemplate is
|
||||
`networkPolicyManagement: Unmanaged`, and the gateway has no policy. That keeps
|
||||
the walkthrough portable and independent of the sandbox-router. **For
|
||||
production this is not optional**: without the sandbox policy every tenant can
|
||||
reach every other tenant and cluster-internal endpoints, and without a gateway
|
||||
egress policy the gateway can talk to anything the cluster network allows.
|
||||
Apply both. The policies below are standard Kubernetes `NetworkPolicy` objects.
|
||||
|
||||
> **Demo vs production.** An installation that skips the sandbox NetworkPolicy
|
||||
> and the gateway egress policy is a demo. Exposure (Ingress/Gateway API) adds
|
||||
> ingress protection only; it does not isolate tenants from each other.
|
||||
|
||||
### Sandbox policy (required for production)
|
||||
|
||||
[`manifests/optional/network-policy/10-sandbox-networkpolicy.yaml`](../manifests/optional/network-policy/10-sandbox-networkpolicy.yaml)
|
||||
selects every Sandbox pod through the agent-sandbox tracking label
|
||||
`agents.x-k8s.io/sandbox-name-hash`. It applies to existing warm-pool and
|
||||
claimed sandboxes without recreating them.
|
||||
|
||||
```bash
|
||||
kubectl apply -f manifests/optional/network-policy/
|
||||
kubectl delete -f manifests/optional/network-policy/ # rollback
|
||||
```
|
||||
|
||||
What it allows:
|
||||
|
||||
| Direction | Allowed | Blocked |
|
||||
| --- | --- | --- |
|
||||
| Ingress | gateway pods (`app: dsh-gateway`) on TCP `3080` | everything else, including other sandboxes and cluster Services |
|
||||
| Egress | UDP/TCP `53` to kube-dns and NodeLocal DNSCache; TCP `443` to public IPs | RFC1918, CGNAT, link-local/metadata, cluster Services/API, plain HTTP, other ports |
|
||||
|
||||
Verify from inside the pods (NetworkPolicy does not affect `kubectl exec`
|
||||
itself, only the traffic the commands generate):
|
||||
|
||||
```bash
|
||||
SB=$(kubectl -n dsh-demo get sandboxclaim dsh-alice -o jsonpath='{.status.sandbox.name}')
|
||||
POD=$(kubectl -n dsh-demo get sandbox "$SB" \
|
||||
-o jsonpath='{.metadata.annotations.agents\.x-k8s\.io/pod-name}'); POD=${POD:-$SB}
|
||||
|
||||
# Allowed: gateway -> sandbox.
|
||||
kubectl -n dsh-demo exec deploy/dsh-gateway -- python -c \
|
||||
"import urllib.request; print(urllib.request.urlopen('http://$POD:3080/favicon.svg', timeout=5).status)"
|
||||
|
||||
# Allowed: DNS and public HTTPS from the sandbox.
|
||||
kubectl -n dsh-demo exec "$POD" -- getent hosts api.deepseek.com
|
||||
kubectl -n dsh-demo exec "$POD" -- curl -fsS -o /dev/null -w '%{http_code}\n' https://example.com
|
||||
|
||||
# Denied: another sandbox cannot reach this one on 3080 (expect a timeout).
|
||||
OTHER_SB=$(kubectl -n dsh-demo get sandboxclaim dsh-bob -o jsonpath='{.status.sandbox.name}')
|
||||
OTHER_POD=$(kubectl -n dsh-demo get sandbox "$OTHER_SB" \
|
||||
-o jsonpath='{.metadata.annotations.agents\.x-k8s\.io/pod-name}'); OTHER_POD=${OTHER_POD:-$OTHER_SB}
|
||||
kubectl -n dsh-demo exec "$OTHER_POD" -- curl --max-time 3 -sS "http://$POD:3080/favicon.svg"
|
||||
```
|
||||
|
||||
Caveats:
|
||||
|
||||
- **The CNI must enforce NetworkPolicy.** If it does not, the file applies but
|
||||
has no effect.
|
||||
- DNS: the default assumes CoreDNS/kube-dns pods labelled `k8s-app: kube-dns`
|
||||
and covers NodeLocal DNSCache at `169.254.20.10`. A different DNS layout
|
||||
needs a selector/`ipBlock` edit.
|
||||
- **Kubelet probes** from the node are CNI-dependent. If a pod starts failing
|
||||
its probes, allow the node CIDR on the probe port or remove the ingress
|
||||
policy.
|
||||
- The policy blocks cluster Services and the Kubernetes API from the sandbox.
|
||||
DSH does not need them, but an agent tool that does needs an explicit rule.
|
||||
- It allows any public host on TCP `443`, not an FQDN allowlist. It is network
|
||||
isolation, not data-loss prevention.
|
||||
- Custom provider base URLs, an HTTP proxy, port `80`, SSH `git` (port `22`) or
|
||||
a non-443 provider endpoint need explicit egress additions.
|
||||
- Do not combine this standalone policy with `networkPolicyManagement: Managed`
|
||||
unless you migrate the rules into `spec.networkPolicy`. NetworkPolicies are
|
||||
additive; the controller's Managed secure default would also be present.
|
||||
- `kubectl port-forward` still works for debugging (the connection is local to
|
||||
the pod), except under gVisor.
|
||||
|
||||
### Exposure-variant policies (required, part of the variant)
|
||||
|
||||
Each exposure variant ships its own policy. They are part of the variant and
|
||||
**required for production**; an exposure without its policy is demo-only:
|
||||
|
||||
- [`manifests/exposure/ingress/61-networkpolicy.yaml`](../manifests/exposure/ingress/61-networkpolicy.yaml)
|
||||
allows only Traefik pods (`app.kubernetes.io/name: traefik`) to reach the
|
||||
gateway on `8080`.
|
||||
- [`manifests/exposure/gateway-api-sso/75-networkpolicies.yaml`](../manifests/exposure/gateway-api-sso/75-networkpolicies.yaml)
|
||||
allows only oauth2-proxy to reach the gateway, only the Gateway
|
||||
implementation (Traefik) to reach oauth2-proxy, and limits oauth2-proxy
|
||||
egress to DNS, public HTTPS (GitHub) and the gateway.
|
||||
|
||||
The SSO policy is what turns the trust boundary in the
|
||||
[exposure README](../manifests/exposure/README.md) into network enforcement.
|
||||
The `dsh-gateway` Service must still stay ClusterIP; do not apply both exposure
|
||||
variants. After applying the SSO variant, run
|
||||
[`scripts/preflight-sso.sh`](../scripts/preflight-sso.sh): it checks the
|
||||
gateway's SSO env, the Secrets, the TLS reference, leftover placeholders and
|
||||
accidental second paths (such as an Ingress).
|
||||
|
||||
Verify the negative case from a sandbox pod (it is not the proxy):
|
||||
|
||||
```bash
|
||||
kubectl -n dsh-demo exec "$POD" -- curl --max-time 3 -sS http://dsh-gateway:8080/healthz
|
||||
# expected: timeout/connection failure
|
||||
```
|
||||
|
||||
Then verify the real browser path through the Ingress or Gateway, and confirm
|
||||
the gateway/proxy pods are `Ready`. If Traefik uses `hostNetwork` or a
|
||||
different pod label, replace the `podSelector` with the correct selector or an
|
||||
`ipBlock`. As with the sandbox policy, kubelet probes can be affected on some
|
||||
CNIs.
|
||||
|
||||
### Gateway egress (required for production)
|
||||
|
||||
The gateway needs DNS, the Kubernetes API, and TCP `3080` to sandbox pods. An
|
||||
egress policy is **required for production** but is not shipped, because the API
|
||||
server endpoint is cluster-specific (Service CIDR or node IP/port) and a wrong
|
||||
value silently breaks signup/proxy. If your cluster has a stable API-server CIDR
|
||||
and port, add an egress policy like this (replace the placeholders; verify the
|
||||
port, commonly `443` or `6443`):
|
||||
|
||||
```yaml
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: dsh-gateway-egress
|
||||
namespace: dsh-demo
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app: dsh-gateway
|
||||
policyTypes:
|
||||
- Egress
|
||||
egress:
|
||||
- to:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: kube-system
|
||||
podSelector:
|
||||
matchLabels:
|
||||
k8s-app: kube-dns
|
||||
ports:
|
||||
- protocol: UDP
|
||||
port: 53
|
||||
- protocol: TCP
|
||||
port: 53
|
||||
- to:
|
||||
- ipBlock:
|
||||
cidr: 10.96.0.1/32 # placeholder: API-server Service/cluster endpoint
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 443
|
||||
- to:
|
||||
- podSelector:
|
||||
matchExpressions:
|
||||
- key: agents.x-k8s.io/sandbox-name-hash
|
||||
operator: Exists
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 3080
|
||||
```
|
||||
|
||||
Cilium users can express the API server as `toEntities: kube-apiserver` (or
|
||||
`toServices`) instead of a CIDR. Do not apply the sample blindly: a wrong CIDR
|
||||
or port silently breaks signup/proxy.
|
||||
|
||||
## 4. Optional provider keys
|
||||
|
||||
### How injection works
|
||||
|
||||
The SandboxTemplate injects two Secrets via `envFrom`:
|
||||
|
||||
- `dsh-provider-keys` — required by the walkthrough; holds
|
||||
`DEEPSEEK_API_KEY`.
|
||||
- `dsh-provider-keys-extra` — optional; referenced with `optional: true`, so a
|
||||
missing Secret does not block sandbox startup. Every key in it becomes an
|
||||
environment variable in every sandbox.
|
||||
|
||||
Create the extra Secret before the warm pool (pod env is stamped at creation):
|
||||
|
||||
```bash
|
||||
kubectl -n dsh-demo create secret generic dsh-provider-keys-extra \
|
||||
--from-literal=OPENROUTER_API_KEY="sk-or-..." \
|
||||
--from-literal=GEMINI_API_KEY="..." \
|
||||
--from-literal=NVIDIA_API_KEY="nvapi-..." \
|
||||
--dry-run=client -o yaml | kubectl apply -f -
|
||||
```
|
||||
|
||||
Use the exact env-var names (uppercase, underscores) — invalid Secret keys are
|
||||
skipped by the kubelet with an event. Do not duplicate a key across the two
|
||||
Secrets; when the same key exists twice, the later `envFrom` entry wins.
|
||||
|
||||
### Supported names (curated)
|
||||
|
||||
These are the names the pinned DSH release resolves through its packaged provider
|
||||
catalog; the catalog has more entries, these are the common ones:
|
||||
|
||||
| Provider | Environment variable |
|
||||
| --- | --- |
|
||||
| DeepSeek | `DEEPSEEK_API_KEY` |
|
||||
| OpenRouter | `OPENROUTER_API_KEY` |
|
||||
| Google Gemini | `GEMINI_API_KEY` |
|
||||
| NVIDIA | `NVIDIA_API_KEY` |
|
||||
| OpenAI | `OPENAI_API_KEY` |
|
||||
| Anthropic | `ANTHROPIC_API_KEY` |
|
||||
| Groq | `GROQ_API_KEY` |
|
||||
| xAI | `XAI_API_KEY` |
|
||||
| Mistral | `MISTRAL_API_KEY` |
|
||||
| Together | `TOGETHER_API_KEY` |
|
||||
| Cerebras | `CEREBRAS_API_KEY` |
|
||||
| Vercel AI Gateway | `AI_GATEWAY_API_KEY` |
|
||||
| Hugging Face | `HF_TOKEN` |
|
||||
|
||||
Aliases to avoid: `GOOGLE_API_KEY`, `HUGGINGFACE_API_KEY` and
|
||||
`VERCEL_AI_GATEWAY_API_KEY`. The packaged catalog looks for `GEMINI_API_KEY`,
|
||||
`HF_TOKEN` and `AI_GATEWAY_API_KEY` instead.
|
||||
|
||||
### Picking up a key change
|
||||
|
||||
A Secret update does not change running pods. The base key and extra keys are
|
||||
both snapshotted at container start:
|
||||
|
||||
```bash
|
||||
# Warm-pool spares: delete them so the pool recreates them from the template.
|
||||
kubectl -n dsh-demo delete sandboxes -l agents.x-k8s.io/warm-pool-sandbox
|
||||
|
||||
# Running claimed sandboxes: delete the pod (the Sandbox controller recreates
|
||||
# it) or suspend/resume the sandbox.
|
||||
POD=... # resolve it as in section 3
|
||||
kubectl -n dsh-demo delete pod "$POD"
|
||||
```
|
||||
|
||||
New claims get the new values automatically. Confirm names without printing
|
||||
secret values:
|
||||
|
||||
```bash
|
||||
kubectl -n dsh-demo exec "$POD" -- sh -c '
|
||||
for k in DEEPSEEK_API_KEY OPENROUTER_API_KEY GEMINI_API_KEY NVIDIA_API_KEY; do
|
||||
if [ -n "$(printenv "$k" || true)" ]; then echo "$k=set"; else echo "$k=unset"; fi
|
||||
done'
|
||||
```
|
||||
|
||||
### Precedence and shared-key limits
|
||||
|
||||
- DSH's stored credential file (`$DSH_HOME/.credentials.yaml`, mode `0600`)
|
||||
**wins over the environment**. If a tenant saved a key through the DSH UI,
|
||||
changing the Secret will not override it.
|
||||
- Both Secrets are shared by **every** tenant, and any agent can read its own
|
||||
environment. This is a warm-pool constraint, not tenant isolation. A tenant
|
||||
can override its own key through the DSH credential store on its PVC, but the
|
||||
platform does not manage that; real multi-tenancy needs per-tenant
|
||||
credentials or an egress proxy.
|
||||
- The optional Secret is created imperatively; the repository never contains a
|
||||
real key. Keep it out of Git and out of shell history where possible.
|
||||
|
||||
## 5. Failed-Pod recovery
|
||||
|
||||
### What it changes
|
||||
|
||||
When a sandbox Pod reaches phase `Failed`, the controller's default policy,
|
||||
`Retain`, leaves the Pod in place and surfaces `Finished=True` on the Sandbox.
|
||||
The Sandbox, its claim and its PVC survive, but nothing runs: the gateway keeps
|
||||
proxying to a dead Pod and every request fails until someone deletes the Pod or
|
||||
the claim.
|
||||
|
||||
agent-sandbox v1.0.8+ (the current pin in [versions.env](../../../versions.env) is newer) adds
|
||||
`spec.failedPodPolicy` to `Sandbox` and to `SandboxTemplate`:
|
||||
|
||||
| Value | Behavior |
|
||||
| --- | --- |
|
||||
| `Retain` (default) | The Failed Pod stays and the Sandbox reports `Finished=True`; the evidence stays available for debugging. |
|
||||
| `Recreate` | The controller deletes the Failed Pod and starts a replacement; the Sandbox identity and its Sandbox-owned PVCs are retained. |
|
||||
|
||||
Repeated failures back off like a Deployment - 5 s, doubling to 5 min between
|
||||
creations - so a crash loop cannot hammer the API server. The backoff resets
|
||||
once the replacement Pod has been continuously `Running` for 10 minutes, and it
|
||||
lives in memory only, so a controller restart clears it.
|
||||
|
||||
The base templates ship it commented out, for two reasons:
|
||||
|
||||
- A deterministic failure (bad image, bad Secret, a wrong probe) would recreate
|
||||
forever while hiding the cause.
|
||||
- The templates use `restartPolicy: OnFailure`, so the kubelet restarts a
|
||||
crashing container in place and the Pod does not reach phase `Failed` anyway.
|
||||
This policy matters for terminal failures - most commonly node eviction
|
||||
(`Evicted`), when a spot/preemptible node is reclaimed or the node hits memory
|
||||
pressure.
|
||||
|
||||
No gateway change is needed: the handshake cache is keyed by Pod UID, so a
|
||||
replacement Pod re-handshakes on the next request, and the existing `401` retry
|
||||
covers containers restarted in place.
|
||||
|
||||
### Enable it
|
||||
|
||||
```bash
|
||||
# 1. The blueprint for future sandboxes.
|
||||
kubectl -n dsh-demo patch sandboxtemplate dsh-web --type merge \
|
||||
-p '{"spec":{"failedPodPolicy":"Recreate"}}'
|
||||
|
||||
# 2. Warm-pool spares were stamped at creation: delete them and let the pool
|
||||
# replenish from the updated template.
|
||||
kubectl -n dsh-demo delete sandboxes -l agents.x-k8s.io/warm-pool-sandbox
|
||||
|
||||
# 3. Claimed sandboxes keep their own spec: patch them in place.
|
||||
for sb in $(kubectl -n dsh-demo get sandboxes \
|
||||
-l '!agents.x-k8s.io/warm-pool-sandbox' -o jsonpath='{.items[*].metadata.name}'); do
|
||||
kubectl -n dsh-demo patch sandbox "$sb" --type merge \
|
||||
-p '{"spec":{"failedPodPolicy":"Recreate"}}'
|
||||
done
|
||||
```
|
||||
|
||||
For a fresh install, uncomment `failedPodPolicy: Recreate` in the template
|
||||
instead.
|
||||
|
||||
### Verify
|
||||
|
||||
```bash
|
||||
SB=$(kubectl -n dsh-demo get sandboxclaim dsh-alice -o jsonpath='{.status.sandbox.name}')
|
||||
kubectl -n dsh-demo get sandbox "$SB" -o jsonpath='{.spec.failedPodPolicy}{"\n"}'
|
||||
# Recreate
|
||||
```
|
||||
|
||||
A failure drill that never touches a user sandbox - a `Sandbox` with
|
||||
`restartPolicy: Never` and a container that exits non-zero (swap the image if
|
||||
Docker Hub is unreachable):
|
||||
|
||||
```bash
|
||||
kubectl -n dsh-demo apply -f - <<'EOF'
|
||||
apiVersion: agents.x-k8s.io/v1beta1
|
||||
kind: Sandbox
|
||||
metadata:
|
||||
name: failed-pod-drill
|
||||
spec:
|
||||
failedPodPolicy: Recreate
|
||||
podTemplate:
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
containers:
|
||||
- name: drill
|
||||
image: busybox:1.36
|
||||
command: ["sh", "-c", "exit 1"]
|
||||
EOF
|
||||
|
||||
kubectl -n dsh-demo get pods -w
|
||||
# The drill Pod reaches phase Failed; a replacement appears within ~5 s, then
|
||||
# 10 s, 20 s, ... as the container keeps exiting non-zero.
|
||||
kubectl -n dsh-demo delete sandbox failed-pod-drill
|
||||
```
|
||||
|
||||
Repeat the drill with `failedPodPolicy: Retain` (or omit the field) to see the
|
||||
default: the Pod stays `Failed` and the Sandbox condition `Finished` becomes
|
||||
`True`.
|
||||
|
||||
### Caveats
|
||||
|
||||
- `Recreate` deletes the Failed Pod, so its logs go with it. Capture
|
||||
`kubectl get events` or switch back to `Retain` when debugging.
|
||||
- The replacement is a cold start: DSH boots again and issues a new launch
|
||||
token. The workspace comes back from the PVC; in-flight sessions do not.
|
||||
- The field needs agent-sandbox v1.0.8+. A v1.0.7 cluster rejects it as an
|
||||
unknown field.
|
||||
- It applies to sandbox Pods only. The gateway and oauth2-proxy are plain
|
||||
Deployments, and their ReplicaSets already replace Failed Pods.
|
||||
|
||||
### Roll back
|
||||
|
||||
```bash
|
||||
kubectl -n dsh-demo patch sandboxtemplate dsh-web --type merge \
|
||||
-p '{"spec":{"failedPodPolicy":"Retain"}}'
|
||||
for sb in $(kubectl -n dsh-demo get sandboxes -o jsonpath='{.items[*].metadata.name}'); do
|
||||
kubectl -n dsh-demo patch sandbox "$sb" --type merge \
|
||||
-p '{"spec":{"failedPodPolicy":"Retain"}}'
|
||||
done
|
||||
```
|
||||
|
||||
Warm-pool spares pick the default up as they are claimed and replenished, or
|
||||
delete them as in step 2 above.
|
||||
|
||||
## 6. Production checklist
|
||||
|
||||
The base walkthrough is a demo. A production install must satisfy every item
|
||||
below; the sections above give the commands and caveats.
|
||||
|
||||
**Isolation and network**
|
||||
|
||||
- [ ] Sandbox pods run under gVisor (or an equivalent kernel-isolation
|
||||
boundary) because DSH runs with `DSH_PERMISSION_MODE=danger-full-access`
|
||||
in this image; the pod is the boundary. (Section 1.)
|
||||
- [ ] The sandbox NetworkPolicy is applied and verified: no tenant→tenant,
|
||||
no tenant→cluster-Service/API, DNS + public HTTPS only. (Section 3.)
|
||||
- [ ] The gateway egress policy is applied with the cluster's real API-server
|
||||
CIDR/port, scoped to DNS, the API server and sandbox pods on `3080`.
|
||||
(Section 3, "Gateway egress".)
|
||||
- [ ] The exposure variant's NetworkPolicy is applied. An exposure without it is
|
||||
demo-only. For SSO, the gateway is reachable **only** through
|
||||
oauth2-proxy; there is no second Ingress/Gateway/port-forward path.
|
||||
- [ ] Nodes/CNI enforce NetworkPolicy (verify the negative cases, not just the
|
||||
positive path).
|
||||
- [ ] The gateway and (optionally) oauth2-proxy run with `runAsNonRoot`,
|
||||
dropped capabilities, `RuntimeDefault` seccomp and a read-only root
|
||||
filesystem, as shipped.
|
||||
|
||||
**Authentication, TLS and abuse controls**
|
||||
|
||||
- [ ] TLS terminates in front of the gateway, `COOKIE_SECURE=true`, and
|
||||
`ALLOW_INSECURE_COOKIES` is **not** set. The gateway refuses to start with
|
||||
an insecure cookie unless explicitly acknowledged.
|
||||
- [ ] SSO (or an equivalent authenticating proxy) is enabled with
|
||||
`AUTH_PROXY_HEADER=X-Forwarded-User`, and
|
||||
[`scripts/preflight-sso.sh`](../scripts/preflight-sso.sh) exits `0`.
|
||||
- [ ] No `your-github-org` / `dsh-demo.example.com` placeholder remains.
|
||||
- [ ] `SIGNUP_INVITE_CODE` is set from a Secret if the gateway's own signup form
|
||||
is reachable, and `AUTH_RATE_LIMIT` is set deliberately (`30`/60s shipped).
|
||||
Add ingress-level rate limiting/quota too; the gateway's buckets are
|
||||
in-process and reset on restart.
|
||||
- [ ] Session lifetime (`COOKIE_MAX_AGE`) and the logout/delete revocation
|
||||
behavior are acceptable; account for gateway restarts logging users out.
|
||||
- [ ] `MAX_IN_FLIGHT_PER_USER` and `WS_MAX_MESSAGE_BYTES` fit the workload.
|
||||
|
||||
**Credentials, platform and operations**
|
||||
|
||||
- [ ] Provider keys are per-tenant (or routed through an egress proxy) instead
|
||||
of the shared `dsh-provider-keys` Secrets; any shared key is treated as
|
||||
visible to every tenant.
|
||||
- [ ] RBAC stays namespace-scoped; `pods/log` is treated as sensitive because
|
||||
it contains launch tokens and model output. Plan the upstream token API
|
||||
migration.
|
||||
- [ ] Images are digest-pinned and dependency installs are hash-locked.
|
||||
- [ ] Gateway replicas stay at `1` (in-memory sessions/idle state) or the state
|
||||
is externalized.
|
||||
- [ ] `failedPodPolicy: Recreate` is set if node eviction/spot reclaim is
|
||||
expected.
|
||||
- [ ] Audit logging, alerting and backups of the PVCs are in place.
|
||||
|
||||
## 7. Verification summary
|
||||
|
||||
Required locally for any change to this example:
|
||||
|
||||
```bash
|
||||
make check # versions, example layout, manifest YAML shape
|
||||
yamllint .
|
||||
make test # only when the gateway code changed
|
||||
shellcheck examples/dsh-agents-as-a-service/scripts/*.sh
|
||||
```
|
||||
|
||||
CI additionally parses every manifest and runs `kubectl apply
|
||||
--dry-run=server` against the real agent-sandbox CRDs. The weekly kind e2e
|
||||
runs the base walkthrough; it also proves the optional extra Secret may be
|
||||
absent.
|
||||
|
||||
Manual checks, on a cluster with the relevant CNI/runtime:
|
||||
|
||||
- gVisor: pod `runtimeClassName`, `uname -r` differs from the node kernel,
|
||||
gateway proxy works, port-forward limitation acknowledged.
|
||||
- nodeSelector: pods land on the intended nodes; no `Pending` events.
|
||||
- Sandbox policy: the allow/deny matrix above.
|
||||
- Gateway egress policy: signup and proxy still work; the gateway cannot reach
|
||||
an arbitrary cluster Service or the metadata server.
|
||||
- Exposure policies: browser path works; a non-proxy pod cannot reach the
|
||||
gateway; probes stay healthy. `scripts/preflight-sso.sh` exits `0`.
|
||||
- Cookie/TLS guard: with `COOKIE_SECURE=false` and no
|
||||
`ALLOW_INSECURE_COOKIES` the gateway refuses to start; with TLS it starts and
|
||||
the cookie is `Secure` (`__Host-` prefixed).
|
||||
- Auth abuse controls: `429` + `Retry-After` after the configured limit, `403`
|
||||
on a wrong invite code, and revocations on logout/delete/recreate.
|
||||
- Provider keys: set/unset output (never the values) and the refresh procedure.
|
||||
- Failed-Pod recovery: `spec.failedPodPolicy` on the Sandbox, and the drill
|
||||
above replaces a `Failed` Pod while keeping the claim and PVC.
|
||||
|
||||
## 8. Rollback summary
|
||||
|
||||
| Layer | Rollback |
|
||||
| --- | --- |
|
||||
| Sandbox policy | `kubectl delete -f manifests/optional/network-policy/` |
|
||||
| Gateway egress policy | `kubectl delete networkpolicy dsh-gateway-egress -n dsh-demo` (do not leave it half-configured) |
|
||||
| Exposure policy | delete the exposure variant directory |
|
||||
| gVisor | patch `runtimeClassName: null`, recreate pods |
|
||||
| nodeSelector | patch `nodeSelector: null`, recreate pods |
|
||||
| Provider keys | delete the extra Secret or a key, recreate pods |
|
||||
| Failed-Pod recovery | patch `failedPodPolicy: Retain`, delete warm-pool spares |
|
||||
|
||||
No PVC or workspace data is touched by any rollback. Deleting a `SandboxClaim`
|
||||
is different: it cascades to the Sandbox **and** its PVC, so do not use claim
|
||||
deletion to apply a pod-spec change.
|
||||
@@ -1,16 +1,73 @@
|
||||
FROM python:3.12-slim
|
||||
# syntax=docker/dockerfile:1.7
|
||||
#
|
||||
# Example gateway image: the tenant-facing proxy in front of DSH sandboxes.
|
||||
# The base image is digest-pinned in versions.env (GATEWAY_PYTHON_IMAGE) and
|
||||
# scripts/check-versions.sh verifies the ARG default below.
|
||||
#
|
||||
# Two stages on that base:
|
||||
# build installs the fully hash-pinned runtime dependencies into /venv as
|
||||
# a dedicated non-root UID, so the runtime stage copies a venv that
|
||||
# contains no root-owned files.
|
||||
# runtime copies /venv and the app, runs as UID 10000 with HOME=/tmp and no
|
||||
# writable location of its own: usable with readOnlyRootFilesystem.
|
||||
ARG GATEWAY_PYTHON_IMAGE=docker.io/python:3.12-slim@sha256:aa943ce411423c7582288f18707ff66eddcbaeb510502d747a72aeb34f30d159
|
||||
|
||||
# Read-only-rootfs friendly: no .pyc writes, no home directory writes.
|
||||
# ---------------------------------------------------------------------------
|
||||
# Build stage: hash-pinned virtualenv owned by the runtime identity.
|
||||
# ---------------------------------------------------------------------------
|
||||
FROM $GATEWAY_PYTHON_IMAGE AS build
|
||||
|
||||
ARG APP_UID=10000
|
||||
ARG APP_GID=10000
|
||||
|
||||
# No .pyc writes and no user cache dirs: the runtime image stays read-only.
|
||||
ENV PYTHONDONTWRITEBYTECODE=1 \
|
||||
PYTHONUNBUFFERED=1 \
|
||||
PIP_NO_CACHE_DIR=1 \
|
||||
HOME=/tmp
|
||||
|
||||
WORKDIR /app
|
||||
RUN groupadd --gid "$APP_GID" app \
|
||||
&& useradd --uid "$APP_UID" --gid "$APP_GID" --no-create-home \
|
||||
--home-dir /tmp --shell /usr/sbin/nologin app \
|
||||
&& python -m venv /venv \
|
||||
&& chown -R "$APP_UID:$APP_GID" /venv
|
||||
|
||||
WORKDIR /build
|
||||
COPY requirements.txt .
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
COPY gateway.py .
|
||||
|
||||
# Install as the unprivileged user so every file in /venv is owned by UID 10000.
|
||||
# --require-hashes makes pip reject any requirement without a matching hash:
|
||||
# the dependency set in requirements.txt is fully pinned and verified. Use the
|
||||
# venv's own pip: the system pip would user-install into $HOME instead.
|
||||
USER $APP_UID:$APP_GID
|
||||
RUN /venv/bin/pip install --no-cache-dir --require-hashes -r requirements.txt
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Runtime stage
|
||||
# ---------------------------------------------------------------------------
|
||||
FROM $GATEWAY_PYTHON_IMAGE AS runtime
|
||||
|
||||
ARG APP_UID=10000
|
||||
ARG APP_GID=10000
|
||||
|
||||
ENV PYTHONDONTWRITEBYTECODE=1 \
|
||||
PYTHONUNBUFFERED=1 \
|
||||
HOME=/tmp \
|
||||
PATH="/venv/bin:$PATH"
|
||||
|
||||
RUN groupadd --gid "$APP_GID" app \
|
||||
&& useradd --uid "$APP_UID" --gid "$APP_GID" --no-create-home \
|
||||
--home-dir /tmp --shell /usr/sbin/nologin app
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Dependencies come from the build stage: same interpreter, no build tools left
|
||||
# in the runtime image and nothing owned by root inside /venv.
|
||||
COPY --from=build --chown=$APP_UID:$APP_GID /venv /venv
|
||||
COPY --chown=$APP_UID:$APP_GID gateway.py .
|
||||
|
||||
EXPOSE 8080
|
||||
|
||||
# Run as a fixed non-root UID; HOME=/tmp keeps caches out of the image.
|
||||
USER 10000
|
||||
|
||||
|
||||
@@ -10,11 +10,13 @@ It does four things:
|
||||
(only its SHA-256 is stored as a claim annotation), reports derived state,
|
||||
and deletes the claim (cascading the sandbox and PVC).
|
||||
2. **Browser sessions** — exchanges a user token for an HttpOnly session
|
||||
cookie so the DSH UI works in a browser at `/u/<user>/`. Automation can
|
||||
still use `Authorization: Bearer <token>`.
|
||||
cookie whose value is only an opaque, server-side session id (user and
|
||||
expiry stay in the gateway's memory) so the DSH UI works in a browser at
|
||||
`/u/<user>/`. Automation can still use
|
||||
`Authorization: Bearer <token>`, which never comes from the cookie.
|
||||
3. **Reverse proxy** — proxies HTTP (including SSE) and WebSocket traffic to
|
||||
the sandbox's DSH Web server on `:3080`, including DSH's own cookie
|
||||
handshake.
|
||||
handshake. Client-supplied forwarding/identity headers are stripped.
|
||||
4. **Optional SSO** — when `AUTH_PROXY_HEADER` is set, trusts the identity
|
||||
header an authenticating reverse proxy (for example oauth2-proxy) forwards,
|
||||
auto-provisions that user, and exposes `POST /recreate` for a clean restart.
|
||||
@@ -22,7 +24,7 @@ It does four things:
|
||||
The DSH version pin lives in [`versions.env`](../../../versions.env)
|
||||
(`DSH_VERSION` plus the optional repository `RELEASE_REVISION`); the example
|
||||
manifests and published images use the release identity built from it (currently
|
||||
`dsh-v0.2.1-rc.3-r3`). DSH binds `:3080`, serves `/`, gates `/api` behind a
|
||||
`dsh-v0.2.1-rc.5-r6`). DSH binds `:3080`, gates the UI and `/api` behind a
|
||||
`dsh-auth-*` cookie and requires a WebSocket at `/api/remote.mux`.
|
||||
|
||||
## Flow
|
||||
@@ -63,10 +65,10 @@ token. A failed handshake returns `503` with `Retry-After: 10`.
|
||||
| Method | Path | Auth | Purpose |
|
||||
| --- | --- | --- | --- |
|
||||
| `GET` | `/healthz` | none | Liveness/readiness. Never touches the cluster. |
|
||||
| `GET` | `/` | none | Landing page with signup + token login forms. With SSO it auto-provisions the proxy-identified user and redirects to their workspace. |
|
||||
| `POST` | `/users` | none | Signup. JSON body `{"user": "<dns-1123 label>"}` or form field `user`. JSON returns `201` when Ready, `202` when accepted but still Waking; form returns `303` to `/u/<user>/`. Both set the session cookie. `409` if the user exists, `400` if invalid, `413` if the body is too large. |
|
||||
| `POST` | `/login` | none | Form (`user`, `token`) returns `303` to `/u/<user>/`; JSON returns `200`. Sets the session cookie. `401` on a bad token, `404` for an unknown user. |
|
||||
| `POST` | `/logout` | none | Clears the session cookie and `303`s to `/`. |
|
||||
| `GET` | `/` | none | Landing page with signup + token login forms (plus an invite-code input when `SIGNUP_INVITE_CODE` is set). With SSO it auto-provisions the proxy-identified user and redirects to their workspace. |
|
||||
| `POST` | `/users` | none | Signup. JSON body `{"user": "<dns-1123 label>"}` or form field `user`; when `SIGNUP_INVITE_CODE` is set, an `invite` form field or `X-Signup-Invite` header must match it (`403` otherwise, checked before the user lookup). JSON returns `201` when Ready, `202` when accepted but still Waking; form returns `303` to `/u/<user>/`. Both set the session cookie. `409` if the user exists, `400` if invalid, `413` if the body is too large, `429` when the auth rate limit is hit. |
|
||||
| `POST` | `/login` | none | Form (`user`, `token`) returns `303` to `/u/<user>/`; JSON returns `200`. Sets the session cookie. `401` on a bad token **and** for an unknown user (no username enumeration). `429` when the auth rate limit is hit. Both auth routes reject a form post whose `Origin`/`Referer` contradicts `Host` (`403`) when the browser sent one. |
|
||||
| `POST` | `/logout` | none | Revokes the server-side session, clears the cookie and `303`s to `/`. |
|
||||
| `POST` | `/recreate` | trusted proxy header | **SSO only.** Deletes the authenticated user's claim (cascading sandbox + PVC, so workspace data is wiped), waits out finalizers, provisions a fresh claim and `303`s to `/u/<user>/`. `401` without a valid identity header; cookie-authenticated calls must be same-origin. |
|
||||
| `GET` | `/users/{user}` | bearer or cookie | Derived state: `Provisioning`, `Waking`, `Ready`, `Suspending`, `Suspended`. |
|
||||
| `DELETE` | `/users/{user}` | bearer or cookie | Delete the claim (sandbox + PVC cascade). Cookie-auth mutations require a same-origin `Origin`/`Referer`. |
|
||||
@@ -78,19 +80,49 @@ token. A failed handshake returns `503` with `Retry-After: 10`.
|
||||
|
||||
* `Authorization: Bearer <token>` is checked with a timing-safe SHA-256
|
||||
comparison against the `aaas.example.com/token-sha256` claim annotation.
|
||||
* The browser session cookie is
|
||||
`dsh_gw_session=<user>:<token>` (HttpOnly, SameSite=Lax, Path=/, optional
|
||||
Secure). A session for user A never authorizes user B.
|
||||
* The browser session cookie is **opaque**: its value is a random
|
||||
server-side session id; the user and expiry live in gateway memory
|
||||
(`GatewayState.sessions`). A cookie can never be edited into another user's
|
||||
credentials, a session id is not accepted as a bearer token, and a bearer
|
||||
token never authorizes a different user. The cookie name defaults to
|
||||
`__Host-dsh_gw_session` when `COOKIE_SECURE=true` (HTTPS) and to
|
||||
`dsh_gw_session` for plain HTTP; attributes are HttpOnly, SameSite=Lax,
|
||||
Path=/ and (with `COOKIE_SECURE`) Secure.
|
||||
* Sessions are revoked on `POST /logout`, on `DELETE /users/{user}`, on
|
||||
`POST /recreate`, and when they expire (looked up on every request; the
|
||||
sweeper also drops expired entries). `from_env()` refuses to start without
|
||||
`COOKIE_SECURE=true` or the explicit `ALLOW_INSECURE_COOKIES=1` opt-out for
|
||||
local HTTP.
|
||||
* Cookie-authenticated **mutations** (`POST`, `PUT`, `PATCH`, `DELETE`) and
|
||||
cookie-authenticated **WebSocket upgrades** require the request `Origin`
|
||||
(or `Referer` fallback) to match the `Host` header. Missing or mismatched
|
||||
cross-origin metadata is rejected (`403` HTTP, close code `4401` for WS).
|
||||
Form signup/login additionally reject a cross-origin `Origin`/`Referer`
|
||||
when the browser sent one, before the body is read.
|
||||
* `POST /users` and `POST /login` are rate limited per client:
|
||||
`AUTH_RATE_LIMIT` attempts per `AUTH_RATE_WINDOW` seconds (default 30/60,
|
||||
`0` disables). The client key is the rightmost `X-Forwarded-For` entry when
|
||||
present (the address the last trusted proxy saw, not a spoofable leftmost
|
||||
value), else the socket peer. Over-limit requests get `429` +
|
||||
`Retry-After` before the body is read.
|
||||
* Proxy requests strip client-supplied forwarding/identity headers in both
|
||||
HTTP and WebSocket paths: `Forwarded`, every `X-Forwarded-*`
|
||||
(including `X-Forwarded-User`), every `X-Auth-Request-*`, and `X-Real-IP`.
|
||||
The gateway sets its own `X-Forwarded-For/Host/Proto/Port` values from the
|
||||
connection. `UPSTREAM_HEADER_ALLOWLIST` optionally narrows the forwarded
|
||||
client headers further (comma-separated, case-insensitive); unset keeps the
|
||||
denylist behavior, and the gateway's own values are always added.
|
||||
* The gateway's bearer token, the browser `Cookie` header and any `?token=`
|
||||
query parameter are never forwarded upstream. Only the cached `dsh-auth-*`
|
||||
cookie is injected.
|
||||
* Upstream `Set-Cookie` headers are dropped. `Location` headers that are
|
||||
absolute paths are rewritten under `/u/<user>/`, and any `token` query
|
||||
parameter is stripped.
|
||||
* Concurrency is capped per user: `MAX_IN_FLIGHT_PER_USER` proxied requests
|
||||
(default 32) before the gateway answers `503` + `Retry-After` (HTTP) or
|
||||
closes the WebSocket with `1011`. Messages received from DSH are limited to
|
||||
`WS_MAX_MESSAGE_BYTES` (default 16 MiB); browser-to-DSH frames are bounded by
|
||||
uvicorn's own WebSocket size limit.
|
||||
|
||||
### SSO (opt-in)
|
||||
|
||||
@@ -106,11 +138,14 @@ kubectl -n dsh-demo set env deploy/dsh-gateway AUTH_PROXY_HEADER=X-Forwarded-Use
|
||||
When set, the gateway treats a request carrying that header as authenticated as
|
||||
that user, provided the value is a valid DNS-1123 label; the header is never
|
||||
trusted when the variable is unset. The landing page then auto-provisions the
|
||||
user (creates their claim idempotently, mints a rotated session token) and
|
||||
user (creates their claim idempotently, rotates the claim token when a new
|
||||
session is needed and mints an opaque session cookie) and
|
||||
redirects to `/u/<user>/`, and `POST /recreate` becomes available for a clean
|
||||
restart. The base manifests leave it unset on purpose; the
|
||||
[SSO exposure variant](../manifests/exposure/gateway-api-sso/) documents how to
|
||||
enable it on the Deployment (`kubectl set env`).
|
||||
enable it on the Deployment (`kubectl set env`). That variant also ships
|
||||
`75-networkpolicies.yaml`, which restricts the gateway to oauth2-proxy and
|
||||
oauth2-proxy's egress to the gateway, DNS and public HTTPS.
|
||||
|
||||
> **Trust boundary — this is the security-critical part.** A trusted identity
|
||||
> header is only safe when **nothing else can reach the gateway**. The gateway
|
||||
@@ -118,6 +153,8 @@ enable it on the Deployment (`kubectl set env`).
|
||||
> authenticating proxy; if a client can reach `dsh-gateway:8080` directly (a
|
||||
> second Ingress/Gateway/Route, a port-forward on a shared host, a broad
|
||||
> NetworkPolicy), it can send `X-Forwarded-User` itself and impersonate any user.
|
||||
> The variant's `75-networkpolicies.yaml` restricts pod traffic to the proxy; a
|
||||
> custom exposure without an equivalent policy has no network enforcement.
|
||||
> Treat `AUTH_PROXY_HEADER` as "the proxy already authenticated this request",
|
||||
> not as an authentication mechanism of its own. Disable SSO by removing the
|
||||
> variable: the header is then ignored and bearer-token/session auth applies
|
||||
@@ -129,17 +166,30 @@ enable it on the Deployment (`kubectl set env`).
|
||||
| --- | --- |
|
||||
| Unknown user / claim | `404` |
|
||||
| Missing or invalid credentials | `401` |
|
||||
| Cross-origin cookie mutation | `403` |
|
||||
| Unknown user on `/login` | `401` (same as a bad token; no username enumeration) |
|
||||
| Cross-origin cookie mutation / cross-origin form signup or login | `403` |
|
||||
| Auth route over `AUTH_RATE_LIMIT` | `429` + `Retry-After`, before the body is read |
|
||||
| Request body over `MAX_BODY_MB` | `413` |
|
||||
| Sandbox not provisioned, wake timeout, handshake failure | `503` + `Retry-After: 10` |
|
||||
| Client aborts mid-body, before the response starts | `499` (`CLIENT_CLOSED_REQUEST`). Nothing is forwarded to DSH and nothing can be delivered; it is not an error, so no traceback is logged. Covers the body-reading control routes (`POST /users`, `POST /login`) as well as the proxy. |
|
||||
| Client aborts mid-stream (SSE, long response) | The response already started, so its upstream status stands (typically `200`); the stream simply ends. No traceback, and the in-flight slot is released as the relay unwinds. |
|
||||
| Sandbox not provisioned, wake timeout, handshake failure | `503` + `Retry-After: 10`. The handshake error body is a fixed message: it never contains the launch token, the handshake URL or pod/credential details. |
|
||||
| User over `MAX_IN_FLIGHT_PER_USER` | `503` + `Retry-After` (HTTP); the existing in-flight requests are untouched |
|
||||
| Upstream connection/protocol failure | `502` |
|
||||
| WS auth failure (unknown user, bad token, cross-origin) | accept, then close `4401` |
|
||||
| WS per-user concurrency cap | accept, then close `1011` |
|
||||
| WS upstream/handshake failure | accept, then close `1011` |
|
||||
| WS message from DSH over `WS_MAX_MESSAGE_BYTES` | the upstream/relay closes the connection (websockets' message-too-big path) |
|
||||
|
||||
A close code can only be delivered after the WebSocket upgrade is accepted;
|
||||
Starlette would turn an early close into an HTTP 403, which is why WS auth
|
||||
failures are `101`, then close `4401`.
|
||||
|
||||
An aborted request is deliberately silent: uvicorn runs with `--no-access-log`,
|
||||
and the `499` cannot reach a peer that has already left (uvicorn's `send()` is
|
||||
a no-op once the socket is closed). Sandbox resolution, the wake and the DSH
|
||||
handshake run before the body is read, so an aborted upload can cost a wake or
|
||||
a handshake but never reaches DSH.
|
||||
|
||||
## Configuration
|
||||
|
||||
All values are read from the environment by `Settings.from_env()`; every one
|
||||
@@ -154,21 +204,35 @@ can also be injected into `create_app` for tests.
|
||||
| `DSH_PORT` | `3080` | Upstream DSH port. |
|
||||
| `DSH_CONTAINER` | `dsh-web` | Container whose logs carry the launch token. |
|
||||
| `DSH_LOG_TAIL_LINES` | `2000` | How many log lines to scan for the launch token. |
|
||||
| `COOKIE_NAME` | `dsh_gw_session` | Browser session cookie name. |
|
||||
| `COOKIE_SECURE` | `false` | Add `Secure` to the session cookie when served over TLS. |
|
||||
| `COOKIE_MAX_AGE` | `604800` | Session cookie lifetime in seconds. |
|
||||
| `COOKIE_NAME` | `__Host-dsh_gw_session` when secure, else `dsh_gw_session` | Browser session cookie name. Only an opaque session id is stored in it. |
|
||||
| `COOKIE_SECURE` | `false` | Add `Secure` to the session cookie. Required unless `ALLOW_INSECURE_COOKIES=1` is set for local HTTP. |
|
||||
| `ALLOW_INSECURE_COOKIES` | `0` | Explicit opt-out for plain-HTTP demos. Without it, `Settings.from_env()` refuses to start unless `COOKIE_SECURE=true`. |
|
||||
| `COOKIE_MAX_AGE` | `604800` | Session lifetime in seconds (server-side expiry and cookie `Max-Age`). |
|
||||
| `MAX_BODY_MB` | `32` | Maximum buffered request body (and `413` threshold). |
|
||||
| `HTTP_TIMEOUT` | `300` | Upstream connect/read timeout in seconds. Proxied streams (SSE, long responses) keep the connect/write budget but have **no read timeout**, so an idle event stream is not cut every five minutes. |
|
||||
| `SWEEP_INTERVAL` | `15` | Seconds between idle sweeps. |
|
||||
| `SWEEP_INTERVAL` | `15` | Seconds between idle sweeps. Sweeps also prune expired sessions and rate buckets and orphaned per-user state. |
|
||||
| `SIGNUP_INVITE_CODE` | unset | When set, `POST /users` requires the matching `invite` form field or `X-Signup-Invite` header (timing-safe compare); the landing page then shows an invite input. Unset leaves signup open. |
|
||||
| `AUTH_RATE_LIMIT` | `30` | Attempts allowed per client per `AUTH_RATE_WINDOW` on `POST /users` and `POST /login`; `0` disables the limit. |
|
||||
| `AUTH_RATE_WINDOW` | `60` | Rate-limit window in seconds. |
|
||||
| `UPSTREAM_HEADER_ALLOWLIST` | unset | Comma-separated, case-insensitive allowlist of client headers to forward upstream. Unset keeps the denylist behavior; forwarding/identity headers are stripped in both modes. |
|
||||
| `MAX_IN_FLIGHT_PER_USER` | `32` | Concurrent proxied requests per user before `503`/WS `1011`. `0` disables the cap. |
|
||||
| `WS_MAX_MESSAGE_BYTES` | `16777216` | Largest WebSocket message (16 MiB) accepted from DSH. Browser-to-DSH frames are bounded by uvicorn's own limit. |
|
||||
| `AUTH_PROXY_HEADER` | unset | Opt-in SSO: name of the trusted identity header set by an authenticating reverse proxy (e.g. `X-Forwarded-User`). Unset disables SSO and the header is never trusted. Enable only when the Service is reachable exclusively through that proxy (see [SSO](#sso-opt-in)). |
|
||||
|
||||
The shipped `manifests/50-gateway.yaml` sets `IDLE_TIMEOUT=360` and
|
||||
`WAKE_TIMEOUT=240` for the demo; the e2e script overrides them to 60/120 so the
|
||||
smoke test does not wait minutes for the sweeper. It also sets
|
||||
`COOKIE_SECURE=false` together with the explicit `ALLOW_INSECURE_COOKIES=1`
|
||||
acknowledgement because the demo is reached over plain HTTP (a `Secure` cookie
|
||||
would never be sent). HTTPS deployments must set `COOKIE_SECURE=true`.
|
||||
|
||||
### Required RBAC
|
||||
|
||||
The gateway's ServiceAccount needs, in its namespace:
|
||||
|
||||
* `sandboxclaims` (`extensions.agents.x-k8s.io`): get, list, create, delete
|
||||
* `sandboxes` (`agents.x-k8s.io`): get, patch
|
||||
* `pods`: get
|
||||
* `sandboxes` (`agents.x-k8s.io`): get, list, patch
|
||||
* `pods`: get, list
|
||||
* `pods/log`: get
|
||||
|
||||
## Running locally
|
||||
@@ -181,8 +245,10 @@ python -m pip install -r requirements-dev.txt
|
||||
python -m pytest -q
|
||||
python -m ruff check .
|
||||
|
||||
# Uvicorn factory (loads in-cluster credentials only on the first claim call)
|
||||
uvicorn gateway:create_app --factory --host 0.0.0.0 --port 8080
|
||||
# Uvicorn factory (loads in-cluster credentials only on the first claim call).
|
||||
# from_env() insists on an explicit acknowledgement for plain HTTP; behind TLS
|
||||
# set COOKIE_SECURE=true instead.
|
||||
ALLOW_INSECURE_COOKIES=1 uvicorn gateway:create_app --factory --host 0.0.0.0 --port 8080
|
||||
```
|
||||
|
||||
Container image, from the repository root:
|
||||
@@ -210,18 +276,27 @@ uvicorn's access log so bearer tokens in URLs are not logged.
|
||||
Log rotation, a very chatty container, or DSH changing its startup message
|
||||
breaks the handshake. Prefer a first-class token/credential API when DSH
|
||||
offers one.
|
||||
* **Single replica, in-memory state.** The handshake cookie cache, idle
|
||||
activity clocks and in-flight counters live in one process. Running more
|
||||
than one replica would need shared state (or sticky routing), otherwise each
|
||||
replica handshakes independently and the sweeper may race another replica.
|
||||
* **Single replica, in-memory state.** The handshake cookie cache, the opaque
|
||||
session table, idle activity clocks, in-flight counters and rate-limit
|
||||
buckets live in one process. Running more than one replica would need shared
|
||||
state (or sticky routing), otherwise each replica handshakes independently,
|
||||
the sweeper may race another replica, and a session/rate budget is only seen
|
||||
by the replica that created it.
|
||||
* **Sessions die with the process.** An opaque session id is valid only in the
|
||||
process that minted it and expires after `COOKIE_MAX_AGE`; a restart logs
|
||||
every browser out (tokens are unaffected).
|
||||
* **In-memory token cache has no TTL.** It is keyed by pod UID, so pod
|
||||
restarts invalidate it naturally; otherwise entries survive until the pod or
|
||||
claim is deleted.
|
||||
* **No TLS termination.** Run behind an ingress/load balancer; set
|
||||
`COOKIE_SECURE=true` there.
|
||||
* **No TLS termination.** Run behind an ingress/load balancer and set
|
||||
`COOKIE_SECURE=true`. `Settings.from_env()` refuses to start without it
|
||||
unless `ALLOW_INSECURE_COOKIES=1` explicitly acknowledges plain-HTTP
|
||||
sessions.
|
||||
* **Origin-based CSRF protection only.** Cookie-auth mutations require a
|
||||
same-origin `Origin`/`Referer`; there is no per-form CSRF token, rate
|
||||
limiting or login throttling.
|
||||
same-origin `Origin`/`Referer`, and form signup/login reject a contradicting
|
||||
origin when the browser sends one; there is no per-form CSRF token. Auth
|
||||
routes have per-client rate limiting and login throttling, but the budget is
|
||||
in-process (per replica) and keyed by the rightmost `X-Forwarded-For` value.
|
||||
* **HTTP request bodies are buffered** up to `MAX_BODY_MB`; uploads are not
|
||||
streamed. Responses (including SSE) are streamed.
|
||||
* **Upstream cookies are not forwarded.** DSH cannot rotate its own
|
||||
@@ -232,5 +307,6 @@ uvicorn's access log so bearer tokens in URLs are not logged.
|
||||
* **SSO trusts a header.** With `AUTH_PROXY_HEADER` set, any client that can
|
||||
reach the gateway Service directly can spoof the identity header. The gateway
|
||||
cannot enforce the trust boundary itself: keep the Service ClusterIP and
|
||||
reachable only through the authenticating proxy. There is no mTLS or signed
|
||||
identity assertion.
|
||||
reachable only through the authenticating proxy. The SSO exposure variant
|
||||
ships a NetworkPolicy that restricts pod traffic to the proxy; without it
|
||||
there is no mTLS or signed identity assertion.
|
||||
|
||||
@@ -7,9 +7,12 @@ single teaching-friendly module with three jobs:
|
||||
(only its SHA-256 is stored, as a claim annotation), report derived state,
|
||||
and delete the claim (which cascades the sandbox and its PVC).
|
||||
* **Browser sessions** -- exchange a user token for an HttpOnly session cookie
|
||||
so the DSH UI can be opened in a browser at ``/u/<user>/``. Browser requests
|
||||
authenticate with that cookie; automation can keep using
|
||||
``Authorization: Bearer <token>``.
|
||||
whose value is an opaque, server-side session id (the user and expiry live in
|
||||
memory) so the DSH UI can be opened in a browser at ``/u/<user>/``. Browser
|
||||
requests authenticate with that cookie; automation can keep using
|
||||
``Authorization: Bearer <token>``. Sessions are revoked on logout, user
|
||||
deletion, recreate and expiry, and a cookie is never accepted as a bearer
|
||||
token (nor the other way around).
|
||||
* **Reverse proxy** -- HTTP (including SSE) and WebSocket proxying to the
|
||||
sandbox's DSH Web server on ``:3080``. DSH itself is cookie-gated: the
|
||||
gateway scrapes the launch token from the pod logs, performs the ``?token=``
|
||||
@@ -38,6 +41,7 @@ import contextlib
|
||||
import hashlib
|
||||
import html
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import re
|
||||
import secrets
|
||||
@@ -62,7 +66,7 @@ from kubernetes import config as k8s_config
|
||||
from kubernetes.client.exceptions import ApiException
|
||||
from starlette.applications import Starlette
|
||||
from starlette.datastructures import Headers
|
||||
from starlette.requests import Request
|
||||
from starlette.requests import ClientDisconnect, Request
|
||||
from starlette.responses import (
|
||||
HTMLResponse,
|
||||
JSONResponse,
|
||||
@@ -121,6 +125,19 @@ OWNED_REQUEST_HEADERS = HOP_BY_HOP | {
|
||||
"x-real-ip",
|
||||
}
|
||||
|
||||
# Default session cookie names. ``__Host-`` is only legal over HTTPS and pins
|
||||
# the cookie to Path=/ with no Domain attribute; the insecure default keeps the
|
||||
# example usable over plain HTTP behind a port-forward.
|
||||
DEFAULT_COOKIE_NAME = "dsh_gw_session"
|
||||
SECURE_COOKIE_NAME = "__Host-dsh_gw_session"
|
||||
|
||||
# Header prefixes a client must never be able to smuggle upstream. The gateway
|
||||
# derives its own X-Forwarded-* values from the connection; anything the browser
|
||||
# sends with these names is attacker-controlled and is dropped in both the HTTP
|
||||
# and WebSocket proxy paths.
|
||||
FORWARDING_HEADER_PREFIXES = ("x-forwarded-", "x-auth-request-")
|
||||
FORWARDING_HEADERS = frozenset({"forwarded"})
|
||||
|
||||
_DSH_WEB_LINE = re.compile(r"dsh web:\s+(?P<url>https?://\S+)")
|
||||
|
||||
|
||||
@@ -144,6 +161,11 @@ class UpstreamError(Exception):
|
||||
"""The upstream DSH connection failed."""
|
||||
|
||||
|
||||
# nginx's "client closed request": the peer went away before it could be
|
||||
# answered, so no status is ever delivered.
|
||||
CLIENT_CLOSED_REQUEST = 499
|
||||
|
||||
|
||||
# --- Pure helpers (unit-testable without an app) --------------------------
|
||||
|
||||
|
||||
@@ -244,6 +266,33 @@ def same_origin(headers) -> bool:
|
||||
return False
|
||||
|
||||
|
||||
def strip_forwarding_header(name: str) -> bool:
|
||||
"""True for client-supplied identity/forwarding headers we never forward.
|
||||
|
||||
``X-Forwarded-User`` and ``X-Auth-Request-*`` carry identities an
|
||||
authenticating proxy or the gateway itself owns; ``Forwarded`` /
|
||||
``X-Forwarded-*`` describe the hop chain. Forwarding any of them lets a
|
||||
browser spoof its origin or a trusted SSO user upstream.
|
||||
"""
|
||||
lowered = name.lower()
|
||||
return lowered in FORWARDING_HEADERS or lowered.startswith(FORWARDING_HEADER_PREFIXES)
|
||||
|
||||
|
||||
def client_identity(headers, client) -> str:
|
||||
"""The rate-limit bucket for a request.
|
||||
|
||||
The rightmost ``X-Forwarded-For`` entry is the address the last trusted
|
||||
proxy saw (each proxy appends), so it cannot be spoofed by prepending
|
||||
arbitrary values. Without the header fall back to the socket peer.
|
||||
"""
|
||||
forwarded = (headers.get("x-forwarded-for") or "").strip()
|
||||
if forwarded:
|
||||
parts = [part.strip() for part in forwarded.split(",") if part.strip()]
|
||||
if parts:
|
||||
return parts[-1]
|
||||
return getattr(client, "host", None) or "unknown"
|
||||
|
||||
|
||||
def rewrite_location(location: str, user: str) -> str:
|
||||
"""Prefix absolute-path redirects so the browser stays under ``/u/<user>/``."""
|
||||
if not location.startswith("/") or location.startswith("//"):
|
||||
@@ -301,6 +350,14 @@ def _env_float(name: str, default: float) -> float:
|
||||
return default
|
||||
|
||||
|
||||
def _env_allowlist(name: str) -> frozenset[str] | None:
|
||||
"""Parse a comma-separated header allowlist; unset means "denylist mode"."""
|
||||
raw = os.environ.get(name)
|
||||
if raw is None:
|
||||
return None
|
||||
return frozenset(part.strip().lower() for part in raw.split(",") if part.strip())
|
||||
|
||||
|
||||
# --- Settings --------------------------------------------------------------
|
||||
|
||||
|
||||
@@ -320,7 +377,7 @@ class Settings:
|
||||
dsh_port: int = 3080
|
||||
dsh_container: str = "dsh-web"
|
||||
dsh_log_tail_lines: int = 2000
|
||||
cookie_name: str = "dsh_gw_session"
|
||||
cookie_name: str = DEFAULT_COOKIE_NAME
|
||||
cookie_secure: bool = False
|
||||
cookie_max_age: int = 7 * 24 * 60 * 60
|
||||
max_body_bytes: int = 32 * 1024 * 1024
|
||||
@@ -334,9 +391,36 @@ class Settings:
|
||||
# valid user in this header are treated as authenticated (SSO); unset (None)
|
||||
# disables SSO entirely and the header is never trusted.
|
||||
auth_proxy_header: str | None = None
|
||||
# Abuse controls. AUTH_RATE_LIMIT applies to the unauthenticated auth routes
|
||||
# (POST /users and POST /login) per client and window; 0 disables it.
|
||||
auth_rate_limit: int = 30
|
||||
auth_rate_window: int = 60
|
||||
# When set, signup requires this invite code (form field ``invite`` or the
|
||||
# ``X-Signup-Invite`` header). None keeps signup open, as in the demo.
|
||||
signup_invite_code: str | None = None
|
||||
# Optional allowlist of client request headers to forward upstream. None
|
||||
# keeps the default denylist behavior; a set (possibly empty) forwards only
|
||||
# the named headers (plus the gateway's own X-Forwarded-* values).
|
||||
upstream_header_allowlist: frozenset[str] | None = None
|
||||
# Concurrent proxied requests per user before 503/1011 is returned.
|
||||
max_in_flight_per_user: int = 32
|
||||
# Largest WebSocket message accepted from DSH, in bytes. Browser-to-DSH
|
||||
# frames are bounded by the ASGI server's own WebSocket limit.
|
||||
ws_max_message_bytes: int = 16 * 1024 * 1024
|
||||
|
||||
@classmethod
|
||||
def from_env(cls) -> Settings:
|
||||
cookie_secure = _env_bool("COOKIE_SECURE", False)
|
||||
allow_insecure = _env_bool("ALLOW_INSECURE_COOKIES", False)
|
||||
if not cookie_secure and not allow_insecure:
|
||||
raise RuntimeError(
|
||||
"refusing to start with an insecure session cookie: set "
|
||||
"COOKIE_SECURE=true behind TLS, or ALLOW_INSECURE_COOKIES=1 "
|
||||
"for a local plain-HTTP demo"
|
||||
)
|
||||
cookie_name = os.environ.get("COOKIE_NAME") or (
|
||||
SECURE_COOKIE_NAME if cookie_secure else DEFAULT_COOKIE_NAME
|
||||
)
|
||||
return cls(
|
||||
namespace=os.environ.get("NAMESPACE", "dsh-demo"),
|
||||
pool=os.environ.get("POOL", "dsh-pool"),
|
||||
@@ -345,13 +429,19 @@ class Settings:
|
||||
dsh_port=_env_int("DSH_PORT", 3080),
|
||||
dsh_container=os.environ.get("DSH_CONTAINER", "dsh-web"),
|
||||
dsh_log_tail_lines=_env_int("DSH_LOG_TAIL_LINES", 2000),
|
||||
cookie_name=os.environ.get("COOKIE_NAME", "dsh_gw_session"),
|
||||
cookie_secure=_env_bool("COOKIE_SECURE", False),
|
||||
cookie_name=cookie_name,
|
||||
cookie_secure=cookie_secure,
|
||||
cookie_max_age=_env_int("COOKIE_MAX_AGE", 7 * 24 * 60 * 60),
|
||||
max_body_bytes=_env_int("MAX_BODY_MB", 32) * 1024 * 1024,
|
||||
http_timeout=_env_float("HTTP_TIMEOUT", 300.0),
|
||||
sweep_interval=_env_float("SWEEP_INTERVAL", 15.0),
|
||||
auth_proxy_header=os.environ.get("AUTH_PROXY_HEADER") or None,
|
||||
auth_rate_limit=_env_int("AUTH_RATE_LIMIT", 30),
|
||||
auth_rate_window=_env_int("AUTH_RATE_WINDOW", 60),
|
||||
signup_invite_code=os.environ.get("SIGNUP_INVITE_CODE") or None,
|
||||
upstream_header_allowlist=_env_allowlist("UPSTREAM_HEADER_ALLOWLIST"),
|
||||
max_in_flight_per_user=_env_int("MAX_IN_FLIGHT_PER_USER", 32),
|
||||
ws_max_message_bytes=_env_int("WS_MAX_MESSAGE_BYTES", 16 * 1024 * 1024),
|
||||
)
|
||||
|
||||
|
||||
@@ -541,6 +631,19 @@ class Kubernetes:
|
||||
# --- Gateway ---------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Session:
|
||||
"""A browser session stored server-side, keyed by an opaque random id.
|
||||
|
||||
The cookie only carries the id; the user and the expiry stay in memory, so
|
||||
a cookie can never be edited into another user's credentials and a stolen
|
||||
token is not recoverable from the browser.
|
||||
"""
|
||||
|
||||
user: str
|
||||
expires_at: float
|
||||
|
||||
|
||||
@dataclass
|
||||
class GatewayState:
|
||||
"""In-memory, per-process state (single replica by design)."""
|
||||
@@ -550,6 +653,10 @@ class GatewayState:
|
||||
maintenance: dict[str, bool] = field(default_factory=dict)
|
||||
# (user, pod UID, browser Host) -> "dsh-auth-a=...; dsh-auth-b=..."
|
||||
handshake: dict[tuple[str, str, str], str] = field(default_factory=dict)
|
||||
# Opaque session id -> server-side session (user + expiry).
|
||||
sessions: dict[str, Session] = field(default_factory=dict)
|
||||
# Auth rate limiting: client -> (window start, attempts in window).
|
||||
auth_rate: dict[str, tuple[float, int]] = field(default_factory=dict)
|
||||
lock: asyncio.Lock = field(default_factory=asyncio.Lock)
|
||||
|
||||
|
||||
@@ -571,17 +678,40 @@ class Gateway:
|
||||
|
||||
# -- credentials -------------------------------------------------------
|
||||
|
||||
def session_for(self, sid: str | None) -> Session | None:
|
||||
"""Look up a live session, dropping it when it has expired."""
|
||||
if not sid:
|
||||
return None
|
||||
session = self.state.sessions.get(sid)
|
||||
if session is None:
|
||||
return None
|
||||
if session.expires_at <= time.time():
|
||||
self.state.sessions.pop(sid, None)
|
||||
return None
|
||||
return session
|
||||
|
||||
def revoke_session(self, sid: str | None) -> None:
|
||||
if sid:
|
||||
self.state.sessions.pop(sid, None)
|
||||
|
||||
def revoke_sessions(self, user: str) -> None:
|
||||
for sid in [key for key, session in self.state.sessions.items() if session.user == user]:
|
||||
self.state.sessions.pop(sid, None)
|
||||
|
||||
def credentials(self, headers, cookies) -> tuple[str | None, str | None]:
|
||||
"""Return ``(token, cookie_user)``; bearer wins over the session cookie."""
|
||||
"""Return ``(token, session_user)``; bearer wins over the session cookie.
|
||||
|
||||
A bearer token is never resolved from a cookie and a session id is never
|
||||
accepted as a bearer token. An unknown or expired session is treated as
|
||||
anonymous (and the expired entry is dropped).
|
||||
"""
|
||||
auth = headers.get("authorization", "")
|
||||
if auth.lower().startswith("bearer "):
|
||||
return auth[7:].strip(), None
|
||||
raw = cookies.get(self.settings.cookie_name)
|
||||
if raw:
|
||||
user, sep, token = raw.partition(":")
|
||||
if sep and user and token:
|
||||
return token, user
|
||||
return None, None
|
||||
session = self.session_for(cookies.get(self.settings.cookie_name))
|
||||
if session is None:
|
||||
return None, None
|
||||
return None, session.user
|
||||
|
||||
def remote_user(self, headers) -> str | None:
|
||||
"""The user asserted by a trusted reverse proxy, when SSO is enabled.
|
||||
@@ -600,31 +730,37 @@ class Gateway:
|
||||
self,
|
||||
claim: dict,
|
||||
token: str | None,
|
||||
cookie_user: str | None,
|
||||
session_user: str | None,
|
||||
user: str,
|
||||
remote: str | None = None,
|
||||
) -> bool:
|
||||
if remote == user:
|
||||
return True # identity asserted by the authenticating proxy
|
||||
if cookie_user is not None and cookie_user != user:
|
||||
return False # a session for another user never authorizes this one
|
||||
if session_user is not None:
|
||||
return session_user == user # server-side session bound to this user
|
||||
return token_matches(claim, token)
|
||||
|
||||
async def _sso_session(self, remote: str, headers, cookies, response: Response) -> dict | None:
|
||||
"""Provision ``remote`` idempotently and mint/refresh a session token.
|
||||
"""Provision ``remote`` idempotently and mint/refresh a session.
|
||||
|
||||
The authenticating proxy has already verified ``remote``, so it is
|
||||
trusted. A brand-new user is provisioned (claim created, token minted);
|
||||
an existing user whose session cookie is missing or no longer matches
|
||||
gets a rotated token so a fresh cookie can be issued. A still-valid
|
||||
session is reused untouched. Provisioning is non-blocking: the sandbox
|
||||
is awaited by the proxy path (``_resolve_sandbox``), so the landing page
|
||||
can report "Provisioning" instead of stalling for ``WAKE_TIMEOUT``.
|
||||
an existing user whose session cookie is missing, foreign or no longer
|
||||
matches gets a rotated token so a fresh cookie can be issued. A still-valid
|
||||
session for the same user is reused untouched. Provisioning is
|
||||
non-blocking: the sandbox is awaited by the proxy path
|
||||
(``_resolve_sandbox``), so the landing page can report "Provisioning"
|
||||
instead of stalling for ``WAKE_TIMEOUT``.
|
||||
"""
|
||||
claim = await self.k8s.get_claim(remote)
|
||||
token, cookie_user = self.credentials(headers, cookies)
|
||||
if claim is not None and cookie_user == remote and token_matches(claim, token):
|
||||
return claim
|
||||
sid = cookies.get(self.settings.cookie_name)
|
||||
session = self.session_for(sid)
|
||||
if session is not None:
|
||||
if session.user != remote:
|
||||
# A session for another identity never authorizes this user.
|
||||
self.revoke_session(sid)
|
||||
elif claim is not None:
|
||||
return claim
|
||||
token = secrets.token_urlsafe(32)
|
||||
token_hash = hashlib.sha256(token.encode()).hexdigest()
|
||||
is_new = claim is None
|
||||
@@ -640,7 +776,7 @@ class Gateway:
|
||||
else:
|
||||
await self.k8s.patch_token(remote, token_hash)
|
||||
await self._touch(remote)
|
||||
self.set_session(response, remote, token)
|
||||
self.set_session(response, remote)
|
||||
return await self.k8s.get_claim(remote) if is_new else claim
|
||||
|
||||
@staticmethod
|
||||
@@ -651,16 +787,73 @@ class Gateway:
|
||||
and not same_origin(headers)
|
||||
)
|
||||
|
||||
def set_session(self, response: Response, user: str, token: str) -> None:
|
||||
def set_session(self, response: Response, user: str) -> str:
|
||||
"""Mint an opaque server-side session and set the browser cookie."""
|
||||
sid = secrets.token_urlsafe(32)
|
||||
self.state.sessions[sid] = Session(
|
||||
user=user,
|
||||
expires_at=time.time() + self.settings.cookie_max_age,
|
||||
)
|
||||
response.set_cookie(
|
||||
self.settings.cookie_name,
|
||||
f"{user}:{token}",
|
||||
sid,
|
||||
max_age=self.settings.cookie_max_age,
|
||||
path="/",
|
||||
httponly=True,
|
||||
samesite="lax",
|
||||
secure=self.settings.cookie_secure,
|
||||
)
|
||||
return sid
|
||||
|
||||
# -- abuse controls ----------------------------------------------------
|
||||
|
||||
def _rate_limit(self, request: Request) -> Response | None:
|
||||
"""Return a 429 response when this client is over the auth budget."""
|
||||
limit = self.settings.auth_rate_limit
|
||||
if limit <= 0:
|
||||
return None
|
||||
window = max(1, self.settings.auth_rate_window)
|
||||
key = client_identity(request.headers, request.client)
|
||||
now = time.time()
|
||||
bucket = self.state.auth_rate.get(key)
|
||||
if bucket is None or now - bucket[0] >= window:
|
||||
self.state.auth_rate[key] = (now, 1)
|
||||
return None
|
||||
start, count = bucket
|
||||
if count >= limit:
|
||||
retry_after = max(1, math.ceil(start + window - now))
|
||||
return self._error(
|
||||
429,
|
||||
"too many authentication attempts, retry shortly",
|
||||
{"Retry-After": str(retry_after)},
|
||||
)
|
||||
self.state.auth_rate[key] = (start, count + 1)
|
||||
return None
|
||||
|
||||
def _invite_allowed(self, request: Request, payload: dict) -> bool:
|
||||
"""Check the signup invite code, when one is configured."""
|
||||
expected = self.settings.signup_invite_code
|
||||
if expected is None:
|
||||
return True
|
||||
provided = payload.get("invite")
|
||||
if not isinstance(provided, str) or not provided:
|
||||
provided = request.headers.get("x-signup-invite", "")
|
||||
if not isinstance(provided, str) or not provided:
|
||||
return False
|
||||
return secrets.compare_digest(provided.encode("utf-8"), expected.encode("utf-8"))
|
||||
|
||||
def _form_cross_origin(self, request: Request) -> bool:
|
||||
"""Reject a form signup/login whose Origin/Referer contradicts Host.
|
||||
|
||||
Only checked when the browser actually sent one: non-browser clients
|
||||
(automation, tests) commonly omit both headers and must keep working.
|
||||
"""
|
||||
if not self._wants_form(request):
|
||||
return False
|
||||
headers = request.headers
|
||||
if not (headers.get("origin") or headers.get("referer")):
|
||||
return False
|
||||
return not same_origin(headers)
|
||||
|
||||
# -- activity / maintenance -------------------------------------------
|
||||
|
||||
@@ -669,12 +862,21 @@ class Gateway:
|
||||
self.state.last_activity[user] = time.time()
|
||||
|
||||
async def _acquire(self, user: str) -> bool:
|
||||
"""Reserve an in-flight slot, waiting out a concurrent suspend."""
|
||||
"""Reserve an in-flight slot, waiting out a concurrent suspend.
|
||||
|
||||
Returns False when the user already has ``MAX_IN_FLIGHT_PER_USER``
|
||||
requests in flight (an abuse guard, not a transient error) or when a
|
||||
suspend does not finish within ``WAKE_TIMEOUT``.
|
||||
"""
|
||||
deadline = time.monotonic() + self.settings.wake_timeout
|
||||
cap = self.settings.max_in_flight_per_user
|
||||
while True:
|
||||
async with self.state.lock:
|
||||
if not self.state.maintenance.get(user):
|
||||
self.state.in_flight[user] = self.state.in_flight.get(user, 0) + 1
|
||||
current = self.state.in_flight.get(user, 0)
|
||||
if cap > 0 and current >= cap:
|
||||
return False
|
||||
self.state.in_flight[user] = current + 1
|
||||
self.state.last_activity[user] = time.time()
|
||||
return True
|
||||
if time.monotonic() > deadline:
|
||||
@@ -682,9 +884,14 @@ class Gateway:
|
||||
await asyncio.sleep(0.1)
|
||||
|
||||
async def _release(self, user: str) -> None:
|
||||
async with self.state.lock:
|
||||
self.state.in_flight[user] = max(self.state.in_flight.get(user, 1) - 1, 0)
|
||||
self.state.last_activity[user] = time.time()
|
||||
# Deliberately await-free: this runs from ``finally`` blocks, and the
|
||||
# slot must be freed even when a client disconnect cancels the request
|
||||
# mid-stream. Awaiting the lock there can be interrupted before the
|
||||
# decrement, stranding the slot (the user's sandbox would then never
|
||||
# look idle again). A plain dict update is atomic because the event
|
||||
# loop cannot switch tasks between two synchronous statements.
|
||||
self.state.in_flight[user] = max(self.state.in_flight.get(user, 1) - 1, 0)
|
||||
self.state.last_activity[user] = time.time()
|
||||
|
||||
# -- sandbox / handshake helpers --------------------------------------
|
||||
|
||||
@@ -749,14 +956,19 @@ class Gateway:
|
||||
host: str,
|
||||
force: bool = False,
|
||||
) -> tuple[str, tuple[str, str, str], bool]:
|
||||
"""Return ``(cookie, cache_key, was_cached)``, handshaking on a miss."""
|
||||
"""Return ``(cookie, cache_key, was_cached)``, handshaking on a miss.
|
||||
|
||||
Every failure raises ``HandshakeError`` with a fixed, credential-free
|
||||
message: the exception reaches the browser in the 503 body, so it must
|
||||
never carry the launch token, the handshake URL or pod details.
|
||||
"""
|
||||
pod_name = self._pod_name(sandbox)
|
||||
try:
|
||||
pod = await self.k8s.get_pod(pod_name)
|
||||
except ApiException as exc:
|
||||
raise HandshakeError(f"could not read pod {pod_name}: {exc}") from exc
|
||||
raise HandshakeError("could not read the DSH handshake pod") from exc
|
||||
if pod is None:
|
||||
raise HandshakeError(f"pod {pod_name} not found")
|
||||
raise HandshakeError("the DSH handshake pod is not available")
|
||||
uid = (pod.get("metadata") or {}).get("uid") or pod_name
|
||||
key = (user, uid, host)
|
||||
if not force:
|
||||
@@ -770,7 +982,7 @@ class Gateway:
|
||||
self.settings.dsh_log_tail_lines,
|
||||
)
|
||||
except ApiException as exc:
|
||||
raise HandshakeError(f"could not read logs for {pod_name}: {exc}") from exc
|
||||
raise HandshakeError("could not read the pod logs for the handshake") from exc
|
||||
launch_token = parse_dsh_launch_token(logs)
|
||||
if not launch_token:
|
||||
raise HandshakeError("DSH launch token not found in pod logs")
|
||||
@@ -788,12 +1000,12 @@ class Gateway:
|
||||
timeout=self.settings.http_timeout,
|
||||
)
|
||||
except httpx.HTTPError as exc:
|
||||
raise HandshakeError(f"DSH handshake failed: {exc}") from exc
|
||||
raise HandshakeError("the DSH handshake request failed") from exc
|
||||
if response.status_code >= 400:
|
||||
raise HandshakeError(f"DSH handshake returned {response.status_code}")
|
||||
raise HandshakeError("the DSH handshake returned an error status")
|
||||
cookie = extract_dsh_cookies(response.headers.get_list("set-cookie"))
|
||||
if not cookie:
|
||||
raise HandshakeError("DSH handshake set no dsh-auth cookie")
|
||||
raise HandshakeError("the DSH handshake set no auth cookie")
|
||||
self.state.handshake[key] = cookie
|
||||
return cookie, key, False
|
||||
|
||||
@@ -869,9 +1081,9 @@ class Gateway:
|
||||
if "set-cookie" in response.headers:
|
||||
page.headers["set-cookie"] = response.headers["set-cookie"]
|
||||
return page
|
||||
token, user = self.credentials(request.headers, request.cookies)
|
||||
_token, user = self.credentials(request.headers, request.cookies)
|
||||
session = ""
|
||||
if user and token:
|
||||
if user:
|
||||
safe_user = html.escape(user)
|
||||
session = (
|
||||
f'<p class="session">Signed in as <strong>{safe_user}</strong>. '
|
||||
@@ -879,6 +1091,11 @@ class Gateway:
|
||||
'<form method="post" action="/logout">'
|
||||
'<button type="submit">Sign out</button></form>'
|
||||
)
|
||||
invite_input = (
|
||||
'<input name="invite" type="password" placeholder="invite code" required>'
|
||||
if self.settings.signup_invite_code is not None
|
||||
else ""
|
||||
)
|
||||
body = f"""<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
@@ -901,6 +1118,7 @@ class Gateway:
|
||||
<h2>Create a sandbox</h2>
|
||||
<form method="post" action="/users">
|
||||
<input name="user" placeholder="username (dns-1123 label)" required>
|
||||
{invite_input}
|
||||
<button type="submit">Create</button>
|
||||
</form>
|
||||
</section>
|
||||
@@ -960,11 +1178,18 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
|
||||
# -- users -------------------------------------------------------------
|
||||
|
||||
async def signup(self, request: Request) -> Response:
|
||||
limited = self._rate_limit(request)
|
||||
if limited is not None:
|
||||
return limited
|
||||
if self._form_cross_origin(request):
|
||||
return self._error(403, "cross-origin request rejected")
|
||||
wants_form = self._wants_form(request)
|
||||
try:
|
||||
payload = await self._read_payload(request)
|
||||
except BodyTooLarge:
|
||||
return self._error(413, "request body too large")
|
||||
if not self._invite_allowed(request, payload):
|
||||
return self._error(403, "invalid or missing invite code")
|
||||
user = payload.get("user", "")
|
||||
if not valid_user(user):
|
||||
return self._error(400, "body must be {'user': '<dns-1123 label>'}")
|
||||
@@ -982,16 +1207,21 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
|
||||
sandbox_name = (sandbox or {}).get("metadata", {}).get("name")
|
||||
if wants_form:
|
||||
response = RedirectResponse(f"/u/{user}/", status_code=303)
|
||||
self.set_session(response, user, token)
|
||||
self.set_session(response, user)
|
||||
return response
|
||||
response = JSONResponse(
|
||||
{"user": user, "token": token, "state": state, "sandbox": sandbox_name},
|
||||
status_code=201 if state == "Ready" else 202,
|
||||
)
|
||||
self.set_session(response, user, token)
|
||||
self.set_session(response, user)
|
||||
return response
|
||||
|
||||
async def login(self, request: Request) -> Response:
|
||||
limited = self._rate_limit(request)
|
||||
if limited is not None:
|
||||
return limited
|
||||
if self._form_cross_origin(request):
|
||||
return self._error(403, "cross-origin request rejected")
|
||||
wants_form = self._wants_form(request)
|
||||
try:
|
||||
payload = await self._read_payload(request)
|
||||
@@ -1002,21 +1232,29 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
|
||||
if not isinstance(user, str) or not isinstance(token, str):
|
||||
return self._error(400, "user and token are required")
|
||||
claim = await self.k8s.get_claim(user)
|
||||
if claim is None:
|
||||
return self._error(404, "not found")
|
||||
if not token_matches(claim, token):
|
||||
# An unknown user is indistinguishable from a bad token to the client:
|
||||
# never reveal which usernames exist.
|
||||
if claim is None or not token_matches(claim, token):
|
||||
return self._error(401, "unauthorized")
|
||||
await self._touch(user)
|
||||
if wants_form:
|
||||
response = RedirectResponse(f"/u/{user}/", status_code=303)
|
||||
else:
|
||||
response = JSONResponse({"user": user, "state": "ok"})
|
||||
self.set_session(response, user, token)
|
||||
self.set_session(response, user)
|
||||
return response
|
||||
|
||||
async def logout(self, request: Request) -> Response:
|
||||
self.revoke_session(request.cookies.get(self.settings.cookie_name))
|
||||
response = RedirectResponse("/", status_code=303)
|
||||
response.delete_cookie(self.settings.cookie_name, path="/")
|
||||
# Match the attributes the cookie was set with: a __Host- cookie
|
||||
# deletion that lacks Secure is rejected by browsers, leaving a stale
|
||||
# (though already revoked) cookie behind.
|
||||
response.delete_cookie(
|
||||
self.settings.cookie_name,
|
||||
path="/",
|
||||
secure=self.settings.cookie_secure,
|
||||
)
|
||||
return response
|
||||
|
||||
async def get_user(self, request: Request) -> Response:
|
||||
@@ -1051,6 +1289,7 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
|
||||
if self._mutation_from_cookie(request.headers, "DELETE", cookie_user):
|
||||
return self._error(403, "cross-origin request rejected")
|
||||
await self.k8s.delete_claim(user)
|
||||
self.revoke_sessions(user)
|
||||
self._clear_handshake(user)
|
||||
async with self.state.lock:
|
||||
self.state.last_activity.pop(user, None)
|
||||
@@ -1074,6 +1313,7 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
|
||||
if self._mutation_from_cookie(request.headers, "POST", cookie_user):
|
||||
return self._error(403, "cross-origin request rejected")
|
||||
await self.k8s.delete_claim(remote)
|
||||
self.revoke_sessions(remote)
|
||||
self._clear_handshake(remote)
|
||||
async with self.state.lock:
|
||||
self.state.last_activity.pop(remote, None)
|
||||
@@ -1101,7 +1341,7 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
|
||||
return RedirectResponse("/", status_code=303)
|
||||
return self._error(404, "unknown user")
|
||||
token, cookie_user = self.credentials(request.headers, request.cookies)
|
||||
if remote == user and not (cookie_user == user and token_matches(claim, token)):
|
||||
if remote == user and cookie_user != user:
|
||||
# SSO user without a valid session cookie: bootstrap, then continue.
|
||||
return RedirectResponse("/", status_code=303)
|
||||
if not self._authorized(claim, token, cookie_user, user, remote):
|
||||
@@ -1161,11 +1401,17 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
|
||||
if query:
|
||||
url = f"{url}?{query}"
|
||||
|
||||
headers = {
|
||||
key: value
|
||||
for key, value in request.headers.items()
|
||||
if key.lower() not in OWNED_REQUEST_HEADERS and key.lower() != "content-length"
|
||||
}
|
||||
headers = {}
|
||||
allowlist = self.settings.upstream_header_allowlist
|
||||
for key, value in request.headers.items():
|
||||
lowered = key.lower()
|
||||
if lowered in OWNED_REQUEST_HEADERS or lowered == "content-length":
|
||||
continue
|
||||
if strip_forwarding_header(lowered):
|
||||
continue # client-supplied identity/hop-chain headers are spoofable
|
||||
if allowlist is not None and lowered not in allowlist:
|
||||
continue
|
||||
headers[key] = value
|
||||
host = request.headers.get("host", "")
|
||||
headers["host"] = host
|
||||
headers["cookie"] = cookie # only the cached DSH auth cookie
|
||||
@@ -1238,9 +1484,17 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
|
||||
print(f"stream for {user} ended early: {exc!r}", flush=True)
|
||||
return
|
||||
finally:
|
||||
with contextlib.suppress(Exception):
|
||||
await upstream.aclose()
|
||||
await self._release(user)
|
||||
# The slot must be freed even when this ``finally`` runs inside a
|
||||
# cancelled task: on a client disconnect Starlette cancels the
|
||||
# streaming task through an anyio cancel scope, and CancelledError
|
||||
# is a BaseException, so ``suppress(Exception)`` would let a close
|
||||
# that awaits abort the whole block before ``_release`` ran. A
|
||||
# stranded slot keeps the user's sandbox looking busy forever.
|
||||
try:
|
||||
with contextlib.suppress(Exception):
|
||||
await upstream.aclose()
|
||||
finally:
|
||||
await self._release(user)
|
||||
|
||||
# -- WebSocket proxy ---------------------------------------------------
|
||||
|
||||
@@ -1291,13 +1545,19 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
|
||||
query = filtered_query(websocket.url.query)
|
||||
if query:
|
||||
url = f"{url}?{query}"
|
||||
headers = {
|
||||
key: value
|
||||
for key, value in websocket.headers.items()
|
||||
if key.lower() not in OWNED_REQUEST_HEADERS
|
||||
and key.lower() != "origin"
|
||||
and not key.lower().startswith("sec-websocket-")
|
||||
}
|
||||
headers = {}
|
||||
allowlist = self.settings.upstream_header_allowlist
|
||||
for key, value in websocket.headers.items():
|
||||
lowered = key.lower()
|
||||
if lowered in OWNED_REQUEST_HEADERS or lowered == "origin":
|
||||
continue
|
||||
if lowered.startswith("sec-websocket-"):
|
||||
continue
|
||||
if strip_forwarding_header(lowered):
|
||||
continue # client-supplied identity/hop-chain headers
|
||||
if allowlist is not None and lowered not in allowlist:
|
||||
continue
|
||||
headers[key] = value
|
||||
headers["cookie"] = cookie
|
||||
headers["x-forwarded-host"] = host
|
||||
headers["x-forwarded-proto"] = websocket.url.scheme
|
||||
@@ -1313,7 +1573,7 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
|
||||
additional_headers=headers,
|
||||
subprotocols=subprotocols or None,
|
||||
open_timeout=self.settings.wake_timeout,
|
||||
max_size=None,
|
||||
max_size=self.settings.ws_max_message_bytes,
|
||||
proxy=None,
|
||||
)
|
||||
upstream = await upstream_cm.__aenter__()
|
||||
@@ -1373,8 +1633,41 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
|
||||
|
||||
# -- idle sweeper ------------------------------------------------------
|
||||
|
||||
def _prune_state(self, claims: list[dict]) -> None:
|
||||
"""Drop state that can no longer be reached.
|
||||
|
||||
Sessions and activity clocks for a user whose claim is gone, expired
|
||||
sessions and expired rate-limit buckets would otherwise accumulate for
|
||||
the lifetime of the process. In-flight counters are only pruned at
|
||||
zero, and a user with work in flight is always preserved.
|
||||
"""
|
||||
now = time.time()
|
||||
present = set()
|
||||
for claim in claims:
|
||||
name = (claim.get("metadata") or {}).get("name") or ""
|
||||
if name.startswith("dsh-"):
|
||||
present.add(name[len("dsh-") :])
|
||||
for user in [user for user in self.state.last_activity if user not in present]:
|
||||
self.state.last_activity.pop(user, None)
|
||||
for user, count in list(self.state.in_flight.items()):
|
||||
if user not in present and count == 0:
|
||||
self.state.in_flight.pop(user, None)
|
||||
for user, flag in list(self.state.maintenance.items()):
|
||||
if user not in present and not flag:
|
||||
self.state.maintenance.pop(user, None)
|
||||
for key in [key for key in self.state.handshake if key[0] not in present]:
|
||||
self.state.handshake.pop(key, None)
|
||||
for sid, session in list(self.state.sessions.items()):
|
||||
if session.user not in present or session.expires_at <= now:
|
||||
self.state.sessions.pop(sid, None)
|
||||
window = max(1, self.settings.auth_rate_window)
|
||||
for key, (start, _count) in list(self.state.auth_rate.items()):
|
||||
if now - start >= window:
|
||||
self.state.auth_rate.pop(key, None)
|
||||
|
||||
async def sweep_once(self) -> None:
|
||||
claims = await self.k8s.list_claims()
|
||||
self._prune_state(claims)
|
||||
for claim in claims:
|
||||
# Only suspend claims this gateway created. Everything else in
|
||||
# the namespace (resource-layer claims from the README walkthrough,
|
||||
@@ -1439,6 +1732,20 @@ def _lifespan(gateway: Gateway, close_http_client: httpx.AsyncClient | None = No
|
||||
return lifespan
|
||||
|
||||
|
||||
async def client_gone(request: Request, exc: ClientDisconnect) -> Response:
|
||||
"""Answer a client that vanished mid-request instead of logging a traceback.
|
||||
|
||||
Reading a request body raises ``ClientDisconnect`` when a browser reload, a
|
||||
back navigation or an ingress timeout aborts the upload. The request never
|
||||
reached DSH, and nothing can be delivered to the peer that left, so the
|
||||
honest outcome is 499 (nginx's "client closed request"). Letting it escape
|
||||
would make Starlette's ``ServerErrorMiddleware`` log one ERROR traceback per
|
||||
aborted request, which buries real faults. It also covers the control
|
||||
routes that read a body (signup and login), not just the proxy.
|
||||
"""
|
||||
return Response(status_code=CLIENT_CLOSED_REQUEST)
|
||||
|
||||
|
||||
def create_app(
|
||||
settings: Settings | None = None,
|
||||
k8s=None,
|
||||
@@ -1471,6 +1778,7 @@ def create_app(
|
||||
]
|
||||
app = Starlette(
|
||||
routes=routes,
|
||||
exception_handlers={ClientDisconnect: client_gone},
|
||||
lifespan=_lifespan(gateway, http_client if owns_http_client else None),
|
||||
)
|
||||
app.state.gateway = gateway
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -1,14 +1,31 @@
|
||||
# DSH "Agents as a Service" — prerequisites: namespace + shared provider key.
|
||||
# DSH "Agents as a Service" — prerequisites: namespace + shared provider keys.
|
||||
#
|
||||
# The dsh-provider-keys Secret is deliberately NOT committed with a usable
|
||||
# value. DEEPSEEK_API_KEY reaches every tenant sandbox (injected via envFrom
|
||||
# on the SandboxTemplate, stamped before any user exists — a warm-pool
|
||||
# constraint), so a real key committed here would be shared by every install
|
||||
# of this example. Generate a fresh value per install (README step 1;
|
||||
# scripts/e2e-kind.sh uses a throwaway):
|
||||
# scripts/e2e-kind.sh uses a throwaway). Use an upsert: this manifest already
|
||||
# creates the empty Secret, so a plain `kubectl create` would fail:
|
||||
#
|
||||
# kubectl -n dsh-demo create secret generic dsh-provider-keys \
|
||||
# --from-literal=DEEPSEEK_API_KEY="sk-..."
|
||||
# --from-literal=DEEPSEEK_API_KEY="sk-..." \
|
||||
# --dry-run=client -o yaml | kubectl apply -f -
|
||||
#
|
||||
# Optional: additional providers (OpenRouter, Gemini, NVIDIA, OpenAI, ...) go
|
||||
# in a second, optional Secret. The SandboxTemplate references it with
|
||||
# `optional: true`, so sandboxes start even when it does not exist, and every
|
||||
# key in it becomes an environment variable. Create it BEFORE the warm pool,
|
||||
# because pod env is stamped at pod creation:
|
||||
#
|
||||
# kubectl -n dsh-demo create secret generic dsh-provider-keys-extra \
|
||||
# --from-literal=OPENROUTER_API_KEY="sk-or-..." \
|
||||
# --from-literal=GEMINI_API_KEY="..." \
|
||||
# --from-literal=NVIDIA_API_KEY="nvapi-..." \
|
||||
# --dry-run=client -o yaml | kubectl apply -f -
|
||||
#
|
||||
# Supported env names, precedence, shared-key caveats and the pod-refresh
|
||||
# procedure are in docs/optional-hardening.md.
|
||||
#
|
||||
# All the infrastructure in this example works without a real key; only
|
||||
# actual model calls need one.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# The reusable DSH Web blueprint. Warm pools stamp pod env BEFORE any user
|
||||
# exists, so per-user secrets cannot live in pod env; the shared provider key
|
||||
# is injected via envFrom, and claims that would bypass the warm pool are
|
||||
# exists, so per-user secrets cannot live in pod env; the shared provider keys
|
||||
# are injected via envFrom, and claims that would bypass the warm pool are
|
||||
# rejected outright (see 40-claim-rejected.yaml).
|
||||
#
|
||||
# A stripped-down single-tenant variant of this template (no warm pool, no
|
||||
@@ -21,16 +21,40 @@ spec:
|
||||
# The gateway routes each user to their DSH Web instance by this name.
|
||||
service: true
|
||||
|
||||
# The example gateway does not manage NetworkPolicy, so it is left
|
||||
# unmanaged here. A production platform would use
|
||||
# networkPolicyManagement: Managed and restrict ingress to its gateway on
|
||||
# :3080 — with the shared provider key, that NetworkPolicy is the per-user
|
||||
# isolation boundary.
|
||||
# The example leaves NetworkPolicy to an optional, opt-in manifest
|
||||
# (manifests/optional/network-policy/): the base walkthrough should not
|
||||
# require a policy-capable CNI, and the controller's Managed secure default
|
||||
# only allows the sandbox-router, which this example does not deploy. If you
|
||||
# switch to Managed, define spec.networkPolicy explicitly or the gateway will
|
||||
# be blocked. See docs/optional-hardening.md.
|
||||
networkPolicyManagement: Unmanaged
|
||||
|
||||
# OPTIONAL: replace a Failed sandbox Pod instead of parking the sandbox.
|
||||
# The default, Retain, leaves the Failed Pod in place and surfaces
|
||||
# Finished=True on the Sandbox until a human intervenes. Recreate deletes the
|
||||
# Failed Pod and starts a replacement with the same Sandbox identity and
|
||||
# Sandbox-owned PVCs, backing off (5s doubling to 5m) on repeated failures.
|
||||
# Requires agent-sandbox v1.0.8+; existing Sandboxes keep their own stamped
|
||||
# policy, so patch them separately. See docs/optional-hardening.md.
|
||||
# failedPodPolicy: Recreate
|
||||
|
||||
podTemplate:
|
||||
spec:
|
||||
# DSH Web is multi-arch (amd64 + arm64); no arch nodeSelector needed.
|
||||
# DSH Web is multi-arch (amd64 + arm64); no arch nodeSelector is needed.
|
||||
#
|
||||
# OPTIONAL: run the sandbox under gVisor. Requires a RuntimeClass named
|
||||
# "gvisor" and nodes that can run it; gVisor breaks `kubectl port-forward`
|
||||
# into the sandbox (use the gateway). Existing Sandboxes snapshot their
|
||||
# pod template, so enabling it later requires patching/recreating them.
|
||||
# See docs/optional-hardening.md.
|
||||
# runtimeClassName: gvisor
|
||||
#
|
||||
# OPTIONAL: pin sandboxes to a dedicated node pool (for example a gVisor
|
||||
# pool). Replace this placeholder with a label your nodes actually carry;
|
||||
# add a matching toleration if the pool is tainted. Active
|
||||
# cluster-specific scheduling belongs in a local patch, not here.
|
||||
# nodeSelector:
|
||||
# dsh.example.com/node-pool: dsh
|
||||
restartPolicy: "OnFailure"
|
||||
securityContext:
|
||||
runAsUser: 1000 # the 'node' user in the DSH image
|
||||
@@ -45,11 +69,11 @@ spec:
|
||||
# OnRootMismatch skips that walk once the volume root already carries
|
||||
# the fsGroup ownership/mode applied on the first mount.
|
||||
# Keep in sync with the minimal variant:
|
||||
# examples/minimal-single-tenant/manifests/sandbox-template.yaml.
|
||||
# examples/minimal-single-tenant/manifests/10-sandboxtemplate.yaml.
|
||||
fsGroupChangePolicy: OnRootMismatch
|
||||
containers:
|
||||
- name: dsh-web
|
||||
image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.3-r3
|
||||
image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.5-r6
|
||||
imagePullPolicy: IfNotPresent
|
||||
ports:
|
||||
- name: web
|
||||
@@ -57,7 +81,31 @@ spec:
|
||||
envFrom:
|
||||
- secretRef:
|
||||
name: dsh-provider-keys
|
||||
# Optional extra provider keys (OpenRouter, Gemini, NVIDIA, ...).
|
||||
# The Secret may be absent; every key in it becomes an env var.
|
||||
# Create it before the warm pool, because pod env is stamped at
|
||||
# pod creation. See docs/optional-hardening.md.
|
||||
- secretRef:
|
||||
name: dsh-provider-keys-extra
|
||||
optional: true
|
||||
env:
|
||||
# Point the OS home at the persisted agent workspace. DSH's
|
||||
# directory picker (the folder explorer used to choose a workspace)
|
||||
# has no start-path setting: when it opens it lists $HOME, and
|
||||
# os.homedir() follows this variable. The image default is
|
||||
# /home/node, which is not on a volume, so the picker would open on
|
||||
# a throwaway directory; /workspace is the PVC-backed WORKDIR. `~`
|
||||
# in agent commands resolves here too. Trade-off: tools that keep
|
||||
# caches/config under $HOME (npm/pnpm, uv, git, gh, and DSH's global
|
||||
# ~/.agents instructions and skills) now write dotfiles onto the PVC
|
||||
# inside the agent workspace, so they are visible to the agent and
|
||||
# in the file explorer. DSH_HOME is explicit (/home/node/.dsh), so
|
||||
# sessions, config and credentials are unaffected. Existing
|
||||
# sandboxes stamp pod env at creation and keep the old HOME until
|
||||
# recreated. Keep in sync with the minimal variant:
|
||||
# examples/minimal-single-tenant/manifests/10-sandboxtemplate.yaml.
|
||||
- name: HOME
|
||||
value: /workspace
|
||||
# Empty => DSH derives links from the request Host, which the
|
||||
# gateway forwards (plus X-Forwarded-*). Point this at the
|
||||
# gateway base URL if you want hard-coded absolute links instead.
|
||||
@@ -71,6 +119,38 @@ spec:
|
||||
# dsh-demo.example.com for the gateway-api-sso/ HTTPRoute).
|
||||
- name: DSH_TRUSTED_HOSTS
|
||||
value: ""
|
||||
# DSH confines every bash command with its own file sandbox
|
||||
# (bubblewrap, else Landlock). Neither backend is usable inside this
|
||||
# pod: bubblewrap ships in the image but is unsetuid, so it needs an
|
||||
# unprivileged user namespace that the pod's RuntimeDefault seccomp
|
||||
# and dropped capabilities do not grant, and the runtime/kernel
|
||||
# surface here exposes no Landlock runner. The executor therefore
|
||||
# fails closed with SANDBOX_UNAVAILABLE and every command fails. The
|
||||
# pod is the isolation boundary in this example (one pod
|
||||
# per user, non-root, capabilities dropped, seccomp RuntimeDefault,
|
||||
# a private PVC), so trade DSH's inner file policy for a working
|
||||
# agent. Set this to workspace-write (or read-only) on hosts where a
|
||||
# backend is available; see the README troubleshooting row.
|
||||
# SECURITY: the pod is the isolation boundary in this example.
|
||||
# danger-full-access is set only because DSH's own command sandbox
|
||||
# has no usable backend in this pod (see above), which means an
|
||||
# agent command can read its own DSH_HOME (including
|
||||
# .credentials.yaml) and write anywhere in the container. What
|
||||
# still contains it is the pod itself: one pod per user, non-root,
|
||||
# all capabilities dropped, RuntimeDefault seccomp, its own PVC and
|
||||
# Service. That is NOT a hardened multi-tenant boundary on a shared
|
||||
# kernel. For production/multi-tenant use, additionally run the
|
||||
# sandbox under gVisor (`runtimeClassName: gvisor` above) so a
|
||||
# container escape is trapped in a userspace kernel, and apply the
|
||||
# sandbox NetworkPolicy (manifests/optional/network-policy/) so
|
||||
# tenants cannot reach each other or cluster-internal endpoints.
|
||||
# The base manifests without those two controls are a demo. Treat
|
||||
# danger-full-access as a fallback, not a target: on hosts where
|
||||
# DSH's command-sandbox backend works, prefer workspace-write (or
|
||||
# read-only), and recreate the pod because the value is stamped at
|
||||
# creation.
|
||||
- name: DSH_PERMISSION_MODE
|
||||
value: danger-full-access
|
||||
# Probes target /favicon.svg: DSH gates / behind its launch-token
|
||||
# cookie and answers 401 to an unauthenticated GET, which would
|
||||
# otherwise fail readiness and then get the pod killed by liveness.
|
||||
@@ -104,7 +184,8 @@ spec:
|
||||
periodSeconds: 15
|
||||
failureThreshold: 3
|
||||
# The container only needs to serve HTTP; drop every capability and
|
||||
# disallow privilege escalation. DSH writes only to DSH_HOME (PVC).
|
||||
# disallow privilege escalation. DSH writes only to its PVC-backed
|
||||
# directories (DSH_HOME and the agent workspace).
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
capabilities:
|
||||
@@ -117,6 +198,16 @@ spec:
|
||||
volumeMounts:
|
||||
- name: dsh-home
|
||||
mountPath: /home/node/.dsh # DSH_HOME: sessions, config, state
|
||||
# The agent workspace (the image's WORKDIR) is PVC-backed too, but
|
||||
# under its own subPath so DSH state - including .credentials.yaml -
|
||||
# never shows up as workspace data. Kubelet creates workspace/ on
|
||||
# first mount with the volume's fsGroup permissions (0775, setgid
|
||||
# inherited from the volume root), so UID 1000 can write; no init
|
||||
# container is needed. If the kubelet is killed mid-creation the
|
||||
# directory stays 0755: see Troubleshooting in the README.
|
||||
- name: dsh-home
|
||||
mountPath: /workspace
|
||||
subPath: workspace
|
||||
volumeClaimTemplates:
|
||||
- metadata:
|
||||
name: dsh-home
|
||||
|
||||
@@ -84,6 +84,15 @@ spec:
|
||||
app: dsh-gateway
|
||||
spec:
|
||||
serviceAccountName: dsh-gateway
|
||||
# OPTIONAL: run the gateway under gVisor. Requires a RuntimeClass named
|
||||
# "gvisor" and nodes that can run it. See docs/optional-hardening.md.
|
||||
# runtimeClassName: gvisor
|
||||
#
|
||||
# OPTIONAL: pin the gateway to a dedicated node pool. Replace the
|
||||
# placeholder with a label your nodes carry; add a toleration for a
|
||||
# tainted pool. Keep active cluster-specific scheduling in a local patch.
|
||||
# nodeSelector:
|
||||
# dsh.example.com/node-pool: dsh
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 10000 # matches USER in gateway/Dockerfile
|
||||
@@ -92,7 +101,7 @@ spec:
|
||||
type: RuntimeDefault
|
||||
containers:
|
||||
- name: gateway
|
||||
image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway:dsh-v0.2.1-rc.3-r3
|
||||
image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway:dsh-v0.2.1-rc.5-r6
|
||||
imagePullPolicy: IfNotPresent
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
@@ -125,14 +134,30 @@ spec:
|
||||
# the DSH launch token.
|
||||
- name: DSH_CONTAINER
|
||||
value: dsh-web
|
||||
# Cookie dsh_gw_session is HttpOnly + SameSite=Lax. "false" keeps
|
||||
# the plain-HTTP demo usable through port-forward: a Secure cookie
|
||||
# is never sent over HTTP, so login would silently fail.
|
||||
# HTTPS deployments (the ingress/ variant with TLS, or
|
||||
# gateway-api-sso/, which terminates TLS at the Gateway) MUST set
|
||||
# this to "true".
|
||||
# Browser session cookie: HttpOnly + SameSite=Lax, holds only an
|
||||
# opaque, server-side session id (sid) - never the user's DSH
|
||||
# token. Sessions can be revoked server-side: POST /logout, user
|
||||
# delete, POST /recreate and expiry (COOKIE_MAX_AGE, default
|
||||
# 7 days) all invalidate them. The session table is in process
|
||||
# memory, so restarting this Deployment logs every browser out.
|
||||
# With COOKIE_SECURE=true the cookie is named
|
||||
# __Host-dsh_gw_session; COOKIE_NAME overrides the name.
|
||||
#
|
||||
# "false" keeps the plain-HTTP demo usable through port-forward: a
|
||||
# Secure cookie is never sent over HTTP, so login would silently
|
||||
# fail. The gateway REFUSES TO START with COOKIE_SECURE=false
|
||||
# unless ALLOW_INSECURE_COOKIES=1 acknowledges that the session
|
||||
# travels in clear text, so the base demo sets both. HTTPS
|
||||
# deployments (the ingress/ variant with TLS, or gateway-api-sso/,
|
||||
# which terminates TLS at the Gateway) MUST set this to "true".
|
||||
- name: COOKIE_SECURE
|
||||
value: "false"
|
||||
# Explicit acknowledgement that this base demo serves plain HTTP.
|
||||
# Required when COOKIE_SECURE=false (the gateway exits at startup
|
||||
# otherwise); ignored once COOKIE_SECURE=true, so HTTPS installs
|
||||
# should drop it. Never set this on an internet-facing deployment.
|
||||
- name: ALLOW_INSECURE_COOKIES
|
||||
value: "1"
|
||||
# SSO IS OPT-IN AND DISABLED HERE. With AUTH_PROXY_HEADER unset the
|
||||
# gateway trusts only its own bearer/session auth and ignores every
|
||||
# identity header. Enable it ONLY behind
|
||||
@@ -143,9 +168,49 @@ spec:
|
||||
# reach :8080 can impersonate any user by sending that header.
|
||||
# - name: AUTH_PROXY_HEADER
|
||||
# value: X-Forwarded-User
|
||||
# Brute-force / abuse throttle for the unauthenticated auth
|
||||
# endpoints (/users and /login): at most N attempts per client per
|
||||
# AUTH_RATE_WINDOW seconds (default 60). A client is the rightmost
|
||||
# X-Forwarded-For entry (the one added by the trusted proxy) when
|
||||
# present, otherwise the TCP peer. Over the limit the gateway
|
||||
# answers 429 + Retry-After BEFORE reading the body or touching the
|
||||
# cluster; 0 disables the limiter. This is a per-process bucket, so
|
||||
# it is not a substitute for an ingress rate limit.
|
||||
- name: AUTH_RATE_LIMIT
|
||||
value: "30"
|
||||
# Window in seconds for AUTH_RATE_LIMIT (default 60).
|
||||
# - name: AUTH_RATE_WINDOW
|
||||
# value: "60"
|
||||
# Optional signup gate. When set, POST /users requires the invite
|
||||
# code - form field "invite" or header X-Signup-Invite - compared in
|
||||
# constant time. A missing/wrong code is rejected with 403 before
|
||||
# the user is looked up, so it also stops unauthenticated claim
|
||||
# creation. The landing page shows the invite input only when this
|
||||
# is set. Keep the value in a Secret, create it before rollout and
|
||||
# restart the gateway to pick up a rotation:
|
||||
# kubectl -n dsh-demo create secret generic dsh-signup-invite \
|
||||
# --from-literal=code="$(openssl rand -hex 16)"
|
||||
# - name: SIGNUP_INVITE_CODE
|
||||
# valueFrom:
|
||||
# secretKeyRef:
|
||||
# name: dsh-signup-invite
|
||||
# key: code
|
||||
# Reject proxied request bodies larger than this (413).
|
||||
- name: MAX_BODY_MB
|
||||
value: "32"
|
||||
# Per-user concurrency guard (default 32): at most this many
|
||||
# proxied HTTP requests and WebSocket connections per user.
|
||||
# Excess HTTP requests get 503 + Retry-After and excess WebSocket
|
||||
# upgrades close with 1011, so one tenant cannot exhaust the
|
||||
# single-replica gateway for everyone else.
|
||||
# - name: MAX_IN_FLIGHT_PER_USER
|
||||
# value: "32"
|
||||
# Largest WebSocket message received from DSH, in bytes (default
|
||||
# 16 MiB); oversized frames close the connection. Browser-to-DSH
|
||||
# frames are bounded by uvicorn's own limit. HTTP/SSE responses are
|
||||
# streamed without this cap.
|
||||
# - name: WS_MAX_MESSAGE_BYTES
|
||||
# value: "16777216"
|
||||
# /healthz is unauthenticated and returns 200 without touching the
|
||||
# cluster, so it is safe for both readiness and liveness.
|
||||
readinessProbe:
|
||||
|
||||
@@ -16,6 +16,30 @@ Applying both gives the gateway two entry points, and the SSO variant is only
|
||||
safe when the gateway is reachable **exclusively** through oauth2-proxy. Do not
|
||||
mix them.
|
||||
|
||||
> **Production requirement.** Each variant's NetworkPolicy is part of the
|
||||
> variant, and applying the variant without it is demo-only. The sandbox
|
||||
> NetworkPolicy and the gateway egress policy (the API server is cluster-specific)
|
||||
> are separate and equally required; see the
|
||||
> [production checklist](../../docs/optional-hardening.md#6-production-checklist)
|
||||
> and the
|
||||
> [gateway egress recipe](../../docs/optional-hardening.md#gateway-egress-required-for-production).
|
||||
> For SSO, run [`scripts/preflight-sso.sh`](../../scripts/preflight-sso.sh)
|
||||
> after applying the variant and before sending real traffic.
|
||||
|
||||
Each variant also ships a NetworkPolicy, applied by the same directory apply:
|
||||
|
||||
- `ingress/61-networkpolicy.yaml` allows only Traefik pods to reach the gateway
|
||||
on `8080`.
|
||||
- `gateway-api-sso/75-networkpolicies.yaml` allows only oauth2-proxy to reach
|
||||
the gateway, only the Gateway implementation (Traefik) to reach oauth2-proxy,
|
||||
and limits oauth2-proxy egress to DNS, public HTTPS (GitHub) and the gateway.
|
||||
|
||||
The policies assume Traefik's `app.kubernetes.io/name: traefik` pod label.
|
||||
Adjust the selector for another implementation or a `hostNetwork` install; see
|
||||
[optional hardening](../../docs/optional-hardening.md) for the probe caveat and
|
||||
verification commands. Keep the policies: an exposure without its NetworkPolicy
|
||||
is demo-only.
|
||||
|
||||
## Applying a variant
|
||||
|
||||
Run these commands from the example root
|
||||
@@ -56,12 +80,18 @@ Secrets imperatively (commands below) before applying the directory.
|
||||
2. Set `DSH_TRUSTED_HOSTS` in the SandboxTemplate to the same public host:
|
||||
the gateway forwards the browser's `Host`, and DSH rejects unknown hosts on
|
||||
`/api` (see `../10-sandboxtemplate.yaml`).
|
||||
3. Set `COOKIE_SECURE=true` on the gateway once the Ingress serves HTTPS:
|
||||
`kubectl -n dsh-demo set env deploy/dsh-gateway COOKIE_SECURE=true`. The
|
||||
base manifest keeps `false` so the plain-HTTP port-forward demo works; a
|
||||
`Secure` cookie is never sent over HTTP, which silently breaks login.
|
||||
3. Set `COOKIE_SECURE=true` and remove the plain-HTTP acknowledgement once the
|
||||
Ingress serves HTTPS:
|
||||
`kubectl -n dsh-demo set env deploy/dsh-gateway COOKIE_SECURE=true ALLOW_INSECURE_COOKIES-`.
|
||||
The base manifest keeps `false` + `ALLOW_INSECURE_COOKIES=1` so the
|
||||
plain-HTTP port-forward demo works; a `Secure` cookie is never sent over HTTP,
|
||||
which silently breaks login, while the acknowledgement must not survive into
|
||||
production.
|
||||
4. Do not attach buffering or compression middlewares to this router: they
|
||||
break WebSocket upgrades and streaming.
|
||||
5. `61-networkpolicy.yaml` is part of this variant: it allows only Traefik pods
|
||||
to reach the gateway on `8080`. Adjust its selector for another ingress
|
||||
controller or `hostNetwork`; if kubelet probes fail, see the hardening doc.
|
||||
|
||||
Verify: `curl -sI https://<host>/healthz` returns `200`.
|
||||
|
||||
@@ -70,6 +100,7 @@ Verify: `curl -sI https://<host>/healthz` returns `200`.
|
||||
`70-gateway.yaml` defines the TLS `Gateway`, `71-httproute.yaml` sends every
|
||||
request to oauth2-proxy, and `72-oauth2-proxy.yaml` authenticates the browser
|
||||
against GitHub and proxies to the gateway with `X-Forwarded-User` set.
|
||||
`75-networkpolicies.yaml` makes this the only network path to the gateway.
|
||||
|
||||
Prerequisites:
|
||||
|
||||
@@ -99,19 +130,28 @@ kubectl -n dsh-demo create secret tls dsh-demo-tls --cert=tls.crt --key=tls.key
|
||||
|
||||
# 3. Enable SSO on the gateway and trust the public host in the template.
|
||||
kubectl -n dsh-demo set env deploy/dsh-gateway \
|
||||
AUTH_PROXY_HEADER=X-Forwarded-User COOKIE_SECURE=true
|
||||
AUTH_PROXY_HEADER=X-Forwarded-User COOKIE_SECURE=true ALLOW_INSECURE_COOKIES-
|
||||
# then set DSH_TRUSTED_HOSTS=dsh-demo.example.com in the SandboxTemplate
|
||||
# (see ../10-sandboxtemplate.yaml) and re-apply it.
|
||||
|
||||
# 4. Apply the variant.
|
||||
kubectl apply -f manifests/exposure/gateway-api-sso/
|
||||
|
||||
# 5. Preflight: fails loudly on missing Secrets, a bad oauth2-proxy cookie
|
||||
# secret, missing gateway SSO env, a dangling TLS reference, leftover
|
||||
# placeholders, or a second path to the gateway.
|
||||
./scripts/preflight-sso.sh
|
||||
```
|
||||
|
||||
The preflight is read-only and exits non-zero on any failure, naming the exact
|
||||
fix. Run it again after rotating a Secret or changing the public host; it also
|
||||
verifies that the oauth2-proxy cookie secret decodes to 16, 24 or 32 bytes.
|
||||
|
||||
### SSO trust boundary
|
||||
|
||||
`AUTH_PROXY_HEADER` is opt-in and deliberately unset in the base gateway
|
||||
manifest: the gateway ignores identity headers until it is set. Enabling SSO
|
||||
takes **both** sides:
|
||||
takes all of these:
|
||||
|
||||
1. `AUTH_PROXY_HEADER=X-Forwarded-User` on `deploy/dsh-gateway` (step 3
|
||||
above). Without it, oauth2-proxy's login has no effect on the gateway and
|
||||
@@ -122,6 +162,16 @@ takes **both** sides:
|
||||
other path to the Service) while SSO is enabled. A client that can reach
|
||||
`dsh-gateway:8080` directly can set `X-Forwarded-User` itself and
|
||||
impersonate any user.
|
||||
3. Keep `75-networkpolicies.yaml` with the variant. It enforces the boundary in
|
||||
the network layer (gateway ← oauth2-proxy ← Traefik, and limited oauth2-proxy
|
||||
egress). This is required, not optional: an exposure without the policy is
|
||||
demo-only, and a custom exposure without an equivalent policy has no network
|
||||
enforcement.
|
||||
4. Apply the gateway egress policy too: the gateway talks to the Kubernetes API
|
||||
and to sandbox pods, and `75-networkpolicies.yaml` intentionally leaves that
|
||||
direction unrestricted because the API-server endpoint is cluster-specific.
|
||||
See
|
||||
[gateway egress](../../docs/optional-hardening.md#gateway-egress-required-for-production).
|
||||
|
||||
Also terminate TLS in front of the gateway (the Gateway listener does) and
|
||||
leave `COOKIE_SECURE=true` so the session cookie never travels in clear text.
|
||||
|
||||
@@ -16,11 +16,18 @@
|
||||
# apply ../ingress/ (or anything else that reaches the gateway). A
|
||||
# client that can reach dsh-gateway:8080 can spoof X-Forwarded-User and
|
||||
# impersonate any user.
|
||||
#
|
||||
# 75-networkpolicies.yaml enforces that boundary at the network layer: the
|
||||
# gateway accepts traffic only from oauth2-proxy pods, oauth2-proxy only from
|
||||
# the Gateway implementation, and oauth2-proxy egress is limited to DNS,
|
||||
# public HTTPS and the gateway. Apply it with this variant.
|
||||
#
|
||||
# TLS terminates at the Gateway listener (70-gateway.yaml); keep
|
||||
# --cookie-secure=true and COOKIE_SECURE=true on the gateway.
|
||||
#
|
||||
# The image is pinned to quay.io/oauth2-proxy/oauth2-proxy:v7.15.5
|
||||
# (versions.env: OAUTH2_PROXY_IMAGE). Credentials come from the
|
||||
# The image is digest-pinned: quay.io/oauth2-proxy/oauth2-proxy:v7.15.5@sha256:...
|
||||
# (versions.env: OAUTH2_PROXY_IMAGE). The digest is the immutable multi-arch
|
||||
# index, so the tag can never be moved under us. Credentials come from the
|
||||
# "oauth2-proxy" Secret (see 73-oauth2-proxy-secret.yaml.example); this
|
||||
# manifest never carries secrets.
|
||||
apiVersion: apps/v1
|
||||
@@ -42,6 +49,11 @@ spec:
|
||||
app: oauth2-proxy
|
||||
app.kubernetes.io/part-of: dsh-demo
|
||||
spec:
|
||||
# OPTIONAL: run the proxy under gVisor and/or pin it to a node pool.
|
||||
# See ../../../docs/optional-hardening.md.
|
||||
# runtimeClassName: gvisor
|
||||
# nodeSelector:
|
||||
# dsh.example.com/node-pool: dsh
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 2000
|
||||
@@ -50,7 +62,7 @@ spec:
|
||||
type: RuntimeDefault
|
||||
containers:
|
||||
- name: oauth2-proxy
|
||||
image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.5
|
||||
image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.5@sha256:8498b0d0ef0a7b29686414000a08aee467f02d0299c9ed1e006a8f33fc017916
|
||||
imagePullPolicy: IfNotPresent
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
|
||||
@@ -0,0 +1,121 @@
|
||||
# REQUIRED with the SSO exposure variant: enforce the trust boundary at the
|
||||
# network layer. Applying the SSO variant without this file is demo-only.
|
||||
#
|
||||
# The chain is Traefik (Gateway API) -> oauth2-proxy -> dsh-gateway. These
|
||||
# policies make it the ONLY chain:
|
||||
# - the gateway accepts traffic only from oauth2-proxy pods;
|
||||
# - oauth2-proxy accepts traffic only from the Gateway implementation;
|
||||
# - oauth2-proxy egress is limited to DNS, public HTTPS (GitHub) and the
|
||||
# gateway.
|
||||
# The gateway's own egress (Kubernetes API, sandbox pods) is intentionally not
|
||||
# restricted here: the API-server endpoint is cluster-specific. Add it for
|
||||
# production. See
|
||||
# ../../../docs/optional-hardening.md#gateway-egress-required-for-production.
|
||||
#
|
||||
# This manifest assumes Traefik, matching gatewayClassName: traefik in
|
||||
# 70-gateway.yaml. Adjust the app.kubernetes.io/name selector for another
|
||||
# implementation. NetworkPolicy enforcement of kubelet probes is CNI-dependent;
|
||||
# if the gateway or proxy fails its probes, allow the node CIDR on the probe
|
||||
# port or remove the ingress policy.
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: dsh-gateway-ingress
|
||||
namespace: dsh-demo
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app: dsh-gateway
|
||||
policyTypes:
|
||||
- Ingress
|
||||
ingress:
|
||||
- from:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app: oauth2-proxy
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 8080
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: oauth2-proxy-ingress
|
||||
namespace: dsh-demo
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app: oauth2-proxy
|
||||
policyTypes:
|
||||
- Ingress
|
||||
ingress:
|
||||
- from:
|
||||
- namespaceSelector: {}
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: traefik
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 4180
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: oauth2-proxy-egress
|
||||
namespace: dsh-demo
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app: oauth2-proxy
|
||||
policyTypes:
|
||||
- Egress
|
||||
egress:
|
||||
# Cluster DNS.
|
||||
- to:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: kube-system
|
||||
podSelector:
|
||||
matchLabels:
|
||||
k8s-app: kube-dns
|
||||
ports:
|
||||
- protocol: UDP
|
||||
port: 53
|
||||
- protocol: TCP
|
||||
port: 53
|
||||
# NodeLocal DNSCache.
|
||||
- to:
|
||||
- ipBlock:
|
||||
cidr: 169.254.20.10/32
|
||||
ports:
|
||||
- protocol: UDP
|
||||
port: 53
|
||||
- protocol: TCP
|
||||
port: 53
|
||||
# The gateway backend.
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app: dsh-gateway
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 8080
|
||||
# GitHub OAuth endpoints are public HTTPS. Private ranges and the metadata
|
||||
# server stay blocked.
|
||||
- to:
|
||||
- ipBlock:
|
||||
cidr: 0.0.0.0/0
|
||||
except:
|
||||
- 10.0.0.0/8
|
||||
- 172.16.0.0/12
|
||||
- 192.168.0.0/16
|
||||
- 169.254.0.0/16
|
||||
- 100.64.0.0/10
|
||||
- ipBlock:
|
||||
cidr: "::/0"
|
||||
except:
|
||||
- "fc00::/7"
|
||||
- "fe80::/10"
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 443
|
||||
@@ -14,8 +14,13 @@
|
||||
# DSH_TRUSTED_HOSTS to this host so DSH accepts the forwarded Host/Origin
|
||||
# on /api.
|
||||
# 2. Gateway Deployment (manifests/50-gateway.yaml): set COOKIE_SECURE to
|
||||
# "true" once this Ingress serves HTTPS, so the session cookie is Secure.
|
||||
# The base manifest sets "false" for the plain-HTTP port-forward demo.
|
||||
# "true" and remove ALLOW_INSECURE_COOKIES once this Ingress serves HTTPS,
|
||||
# so the session cookie is Secure:
|
||||
# kubectl -n dsh-demo set env deploy/dsh-gateway \
|
||||
# COOKIE_SECURE=true ALLOW_INSECURE_COOKIES-
|
||||
# The base manifest sets "false" + ALLOW_INSECURE_COOKIES=1 for the
|
||||
# plain-HTTP port-forward demo; without the acknowledgement the gateway
|
||||
# refuses to start.
|
||||
#
|
||||
# WebSocket support is REQUIRED: the browser talks to /u/<user>/... and DSH
|
||||
# uses WebSockets for live session traffic. Traefik upgrades WS/SSE
|
||||
@@ -23,6 +28,12 @@
|
||||
# middlewares (e.g. compress, buffering) to this router — they break streaming
|
||||
# and upgrades. `path: /` Prefix covers the landing page and every per-user
|
||||
# proxy path, so no per-sandbox Ingress is needed.
|
||||
#
|
||||
# 61-networkpolicy.yaml is part of this variant and required for production:
|
||||
# it restricts the gateway's ingress to Traefik pods. Adjust its selector if
|
||||
# your ingress controller uses a different label or hostNetwork. The sandbox
|
||||
# and gateway-egress NetworkPolicies are separate requirements; see
|
||||
# ../../../docs/optional-hardening.md#6-production-checklist.
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
|
||||
@@ -0,0 +1,36 @@
|
||||
# REQUIRED with the Ingress exposure variant for production: restrict the
|
||||
# gateway's ingress to the ingress controller. Applying the variant without
|
||||
# this policy is demo-only.
|
||||
#
|
||||
# This manifest assumes Traefik, matching this variant's
|
||||
# `ingressClassName: traefik`. Traefik's Helm chart labels its pods
|
||||
# app.kubernetes.io/name=traefik; adjust the selector for another controller or
|
||||
# for a hostNetwork install (where the source is the node IP, not a pod IP).
|
||||
#
|
||||
# The sandbox NetworkPolicy and the gateway egress policy are separate and also
|
||||
# required for production; see
|
||||
# ../../../docs/optional-hardening.md#6-production-checklist.
|
||||
#
|
||||
# NetworkPolicy enforcement of kubelet probe traffic is CNI-dependent. If the
|
||||
# gateway starts failing its /healthz probes, allow the node CIDR on port 8080
|
||||
# or remove this policy. See ../../../docs/optional-hardening.md.
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: dsh-gateway-ingress
|
||||
namespace: dsh-demo
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app: dsh-gateway
|
||||
policyTypes:
|
||||
- Ingress
|
||||
ingress:
|
||||
- from:
|
||||
- namespaceSelector: {}
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: traefik
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 8080
|
||||
@@ -0,0 +1,46 @@
|
||||
# Optional manifests
|
||||
|
||||
Opt-in manifests that are **not** part of the top-level walkthrough.
|
||||
`kubectl apply -f manifests/` is non-recursive, so it does not apply this
|
||||
directory; apply a subdirectory explicitly.
|
||||
|
||||
## network-policy/
|
||||
|
||||
A Kubernetes `NetworkPolicy` for the DSH sandboxes:
|
||||
|
||||
- ingress only from the gateway pods on TCP `3080` (HTTP + WebSocket);
|
||||
- egress only to DNS and public HTTPS (provider APIs, package mirrors,
|
||||
git-over-HTTPS).
|
||||
|
||||
Apply and delete it explicitly:
|
||||
|
||||
```bash
|
||||
kubectl apply -f manifests/optional/network-policy/
|
||||
kubectl delete -f manifests/optional/network-policy/
|
||||
```
|
||||
|
||||
It requires a CNI that enforces `NetworkPolicy`. The selector uses the
|
||||
agent-sandbox controller's pod tracking label, so the policy applies to
|
||||
existing warm-pool and claimed sandboxes without recreation.
|
||||
|
||||
**This is opt-in for the demo but required for production.** Without it every
|
||||
tenant can reach every other tenant and cluster-internal endpoints. Production
|
||||
also needs the gateway egress policy (cluster-specific API-server endpoint) and
|
||||
the exposure variant's policy; see the
|
||||
[production checklist](../../docs/optional-hardening.md#6-production-checklist).
|
||||
|
||||
## gVisor, nodeSelector and provider keys
|
||||
|
||||
These are configured in the main manifests or with an extra Secret:
|
||||
|
||||
- **gVisor:** uncomment `runtimeClassName: gvisor` in the SandboxTemplate and
|
||||
the Deployments before applying, or patch an existing install.
|
||||
- **nodeSelector:** uncomment the field and replace the placeholder node label.
|
||||
- **Failed-Pod recovery:** uncomment `failedPodPolicy: Recreate` in the
|
||||
SandboxTemplate (agent-sandbox v1.0.8+) so a Failed Pod is replaced instead
|
||||
of parked.
|
||||
- **Provider keys:** create the optional `dsh-provider-keys-extra` Secret; the
|
||||
base `dsh-provider-keys` Secret remains the one required key.
|
||||
|
||||
Full prerequisites, exact commands, verification and caveats:
|
||||
[optional hardening](../../docs/optional-hardening.md).
|
||||
@@ -0,0 +1,86 @@
|
||||
# OPTIONAL for the base walkthrough, REQUIRED for production. Opt-in tenant
|
||||
# network policy for the DSH sandboxes: without it every tenant can reach every
|
||||
# other tenant and cluster-internal endpoints.
|
||||
#
|
||||
# It is NOT part of the top-level walkthrough. Apply and delete it explicitly:
|
||||
# kubectl apply -f manifests/optional/network-policy/
|
||||
# kubectl delete -f manifests/optional/network-policy/
|
||||
#
|
||||
# Production also needs the gateway egress policy (cluster-specific API-server
|
||||
# endpoint) and the exposure variant's policy, plus gVisor
|
||||
# (docs/optional-hardening.md#6-production-checklist).
|
||||
#
|
||||
# The selector uses the agent-sandbox controller's pod tracking label, which is
|
||||
# present on every Sandbox pod (including warm-pool spares), so the policy
|
||||
# applies to existing sandboxes without recreating them. The SandboxTemplate
|
||||
# stays networkPolicyManagement: Unmanaged. Do not also switch it to Managed
|
||||
# unless you migrate these rules into spec.networkPolicy; NetworkPolicies are
|
||||
# additive and the two configurations would be confusing.
|
||||
#
|
||||
# Requires a CNI that enforces NetworkPolicy. See docs/optional-hardening.md
|
||||
# for the connectivity matrix, DNS variants, kubelet-probe caveat and rollback.
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: dsh-sandbox
|
||||
namespace: dsh-demo
|
||||
spec:
|
||||
podSelector:
|
||||
matchExpressions:
|
||||
- key: agents.x-k8s.io/sandbox-name-hash
|
||||
operator: Exists
|
||||
policyTypes:
|
||||
- Ingress
|
||||
- Egress
|
||||
ingress:
|
||||
# The gateway is the only intended client of a sandbox (HTTP + WebSocket).
|
||||
- from:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app: dsh-gateway
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 3080
|
||||
egress:
|
||||
# Cluster DNS. CoreDNS/kube-dns pods carry k8s-app=kube-dns.
|
||||
- to:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: kube-system
|
||||
podSelector:
|
||||
matchLabels:
|
||||
k8s-app: kube-dns
|
||||
ports:
|
||||
- protocol: UDP
|
||||
port: 53
|
||||
- protocol: TCP
|
||||
port: 53
|
||||
# NodeLocal DNSCache, for clusters that do not send DNS to the kube-dns
|
||||
# Service. 169.254.20.10 is its standard link-local address.
|
||||
- to:
|
||||
- ipBlock:
|
||||
cidr: 169.254.20.10/32
|
||||
ports:
|
||||
- protocol: UDP
|
||||
port: 53
|
||||
- protocol: TCP
|
||||
port: 53
|
||||
# Public HTTPS only: provider APIs, package mirrors, git-over-HTTPS.
|
||||
# Private ranges, CGNAT and the metadata server stay blocked.
|
||||
- to:
|
||||
- ipBlock:
|
||||
cidr: 0.0.0.0/0
|
||||
except:
|
||||
- 10.0.0.0/8
|
||||
- 172.16.0.0/12
|
||||
- 192.168.0.0/16
|
||||
- 169.254.0.0/16
|
||||
- 100.64.0.0/10
|
||||
- ipBlock:
|
||||
cidr: "::/0"
|
||||
except:
|
||||
- "fc00::/7"
|
||||
- "fe80::/10"
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 443
|
||||
@@ -49,6 +49,8 @@ kubectl apply -f manifests/00-prereqs.yaml
|
||||
kubectl -n "$NS" delete secret dsh-provider-keys --ignore-not-found >/dev/null
|
||||
kubectl -n "$NS" create secret generic dsh-provider-keys \
|
||||
--from-literal=DEEPSEEK_API_KEY="sk-test-$(openssl rand -hex 16)"
|
||||
# dsh-provider-keys-extra is intentionally absent: the template's optional
|
||||
# envFrom reference must not block the warm pool.
|
||||
kubectl apply -f manifests/10-sandboxtemplate.yaml -f manifests/20-sandboxwarmpool.yaml
|
||||
for _ in $(seq 1 120); do
|
||||
READY=$(kubectl -n "$NS" get sandboxes -l agents.x-k8s.io/warm-pool-sandbox \
|
||||
@@ -70,8 +72,23 @@ pass "claim Ready in ${ELAPSED}s, adopted $SB"
|
||||
|
||||
echo "=== 3. write state on the PVC"
|
||||
POD="$SB"
|
||||
# HOME must point at the PVC-backed workspace: DSH's directory picker lists
|
||||
# $HOME when it opens and has no start-path setting, so this is what makes the
|
||||
# folder explorer start on the persisted tree instead of the ephemeral image
|
||||
# home. It also makes ~ resolve to /workspace inside agent commands.
|
||||
HOME_VALUE=$(kubectl -n "$NS" exec "$POD" -- printenv HOME)
|
||||
[ "$HOME_VALUE" = "/workspace" ] || fail "HOME is '$HOME_VALUE', expected /workspace"
|
||||
TILDE=$(kubectl -n "$NS" exec "$POD" -- sh -c 'cd ~ && pwd')
|
||||
[ "$TILDE" = "/workspace" ] || fail "~ resolves to '$TILDE', expected /workspace"
|
||||
pass "HOME=$HOME_VALUE and ~ resolve to the workspace"
|
||||
kubectl -n "$NS" exec "$POD" -- sh -c 'echo "remember me" > /home/node/.dsh/marker.txt'
|
||||
pass "marker written in $POD"
|
||||
pass "state marker written in $POD"
|
||||
# The agent workspace must be the dsh-home PVC's workspace/ subPath, writable
|
||||
# as UID 1000 without an init container.
|
||||
SUBPATH=$(kubectl -n "$NS" get pod "$POD" -o jsonpath='{range .spec.containers[0].volumeMounts[?(@.mountPath=="/workspace")]}{.subPath}{end}')
|
||||
[ "$SUBPATH" = "workspace" ] || fail "/workspace is not mounted from the dsh-home PVC subPath"
|
||||
kubectl -n "$NS" exec "$POD" -- sh -c 'echo "remember me too" > /workspace/marker.txt'
|
||||
pass "workspace marker written in $POD (subPath=$SUBPATH)"
|
||||
|
||||
echo "=== 4. suspend (pod deleted, PVC + Service retained)"
|
||||
kubectl -n "$NS" patch sandbox "$SB" --type merge -p '{"spec":{"operatingMode":"Suspended"}}'
|
||||
@@ -85,8 +102,11 @@ kubectl -n "$NS" patch sandbox "$SB" --type merge -p '{"spec":{"operatingMode":"
|
||||
kubectl -n "$NS" wait sandbox "$SB" --for=condition=Ready --timeout=180s
|
||||
POD="$SB"
|
||||
MARKER=$(kubectl -n "$NS" exec "$POD" -- cat /home/node/.dsh/marker.txt)
|
||||
[ "$MARKER" = "remember me" ] || fail "marker did not survive suspend/resume"
|
||||
pass "resumed: state survived on the reattached PVC"
|
||||
[ "$MARKER" = "remember me" ] || fail "state marker did not survive suspend/resume"
|
||||
pass "resumed: DSH state survived on the reattached PVC"
|
||||
WS_MARKER=$(kubectl -n "$NS" exec "$POD" -- cat /workspace/marker.txt)
|
||||
[ "$WS_MARKER" = "remember me too" ] || fail "workspace marker did not survive suspend/resume"
|
||||
pass "resumed: agent workspace survived on the PVC subPath"
|
||||
|
||||
echo "=== 6. injection policy rejects env-carrying claims"
|
||||
kubectl apply -f manifests/40-claim-rejected.yaml
|
||||
|
||||
234
examples/dsh-agents-as-a-service/scripts/preflight-sso.sh
Executable file
234
examples/dsh-agents-as-a-service/scripts/preflight-sso.sh
Executable file
@@ -0,0 +1,234 @@
|
||||
#!/usr/bin/env bash
|
||||
# Preflight for the gateway-api-sso exposure variant.
|
||||
#
|
||||
# Run this AFTER creating the oauth2-proxy/TLS Secrets, setting the gateway
|
||||
# env and applying manifests/exposure/gateway-api-sso/, and BEFORE sending real
|
||||
# traffic. It checks the configuration that is easy to get wrong and whose
|
||||
# failure is either silent (login appears to work but the gateway still asks
|
||||
# for its own token) or dangerous (a spoofable identity header, a placeholder
|
||||
# host, a TLS Secret that does not exist).
|
||||
#
|
||||
# Checks:
|
||||
# 1. kubectl reaches the cluster and the SSO objects exist;
|
||||
# 2. the oauth2-proxy Secret has client-id, client-secret and a cookie-secret
|
||||
# that decodes to 16, 24 or 32 bytes; the TLS Secret exists;
|
||||
# 3. deploy/dsh-gateway has AUTH_PROXY_HEADER=X-Forwarded-User and
|
||||
# COOKIE_SECURE=true (both are opt-in and both are required for SSO over
|
||||
# TLS);
|
||||
# 4. the Gateway's TLS certificate reference resolves to an existing Secret;
|
||||
# 5. no your-github-org / dsh-demo.example.com placeholder is left in the
|
||||
# applied oauth2-proxy args, Gateway listeners or HTTPRoute hostnames;
|
||||
# 6. (warning) no Ingress in the namespace creates a second path to
|
||||
# dsh-gateway. With SSO enabled the Service must be reachable only through
|
||||
# oauth2-proxy.
|
||||
#
|
||||
# Exit status: 0 when every check passes (warnings allowed), 1 on any failure.
|
||||
# Every failure prints the fix. This script only reads cluster state.
|
||||
#
|
||||
# Usage:
|
||||
# examples/dsh-agents-as-a-service/scripts/preflight-sso.sh
|
||||
#
|
||||
# Environment overrides:
|
||||
# NAMESPACE namespace (default: dsh-demo)
|
||||
# GATEWAY_DEPLOY gateway Deployment name (default: dsh-gateway)
|
||||
# OAUTH_DEPLOY oauth2-proxy Deployment name (default: oauth2-proxy)
|
||||
# OAUTH_SECRET oauth2-proxy Secret name (default: oauth2-proxy)
|
||||
# TLS_SECRET TLS Secret name the Gateway should reference (default:
|
||||
# dsh-demo-tls)
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
NS="${NAMESPACE:-dsh-demo}"
|
||||
GATEWAY_DEPLOY="${GATEWAY_DEPLOY:-dsh-gateway}"
|
||||
OAUTH_DEPLOY="${OAUTH_DEPLOY:-oauth2-proxy}"
|
||||
OAUTH_SECRET="${OAUTH_SECRET:-oauth2-proxy}"
|
||||
TLS_SECRET="${TLS_SECRET:-dsh-demo-tls}"
|
||||
|
||||
FAILED=0
|
||||
pass() { printf 'PASS %s\n' "$*"; }
|
||||
warn() { printf 'WARN %s\n' "$*" >&2; }
|
||||
fail() { printf 'FAIL %s\n' "$*" >&2; FAILED=1; }
|
||||
|
||||
if ! command -v kubectl >/dev/null 2>&1; then
|
||||
echo "FAIL kubectl is not installed; install it and select the target context" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if ! kubectl -n "$NS" get deployment "$GATEWAY_DEPLOY" >/dev/null 2>&1; then
|
||||
echo "FAIL deploy/$GATEWAY_DEPLOY not found in namespace $NS." >&2
|
||||
echo " Apply the base manifests first: kubectl apply -f manifests/" >&2
|
||||
echo " Then apply the variant: kubectl apply -f manifests/exposure/gateway-api-sso/" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "== Secrets"
|
||||
# check_secret_key <secret> <key> <hint>
|
||||
check_secret_key() {
|
||||
local secret=$1 key=$2 hint=$3 value escaped
|
||||
# Kubernetes JSONPath treats "." as a path separator; escape it in keys such
|
||||
# as tls.crt.
|
||||
escaped=${key//./\\.}
|
||||
value=$(kubectl -n "$NS" get secret "$secret" -o jsonpath="{.data.${escaped}}" 2>/dev/null || true)
|
||||
if [ -n "$value" ]; then
|
||||
pass "Secret $secret has non-empty key '$key'"
|
||||
else
|
||||
fail "Secret $secret is missing a non-empty '$key' key. $hint"
|
||||
fi
|
||||
}
|
||||
|
||||
check_secret_key "$OAUTH_SECRET" client-id \
|
||||
"Create it from manifests/exposure/gateway-api-sso/73-oauth2-proxy-secret.yaml.example."
|
||||
check_secret_key "$OAUTH_SECRET" client-secret \
|
||||
"Create it from manifests/exposure/gateway-api-sso/73-oauth2-proxy-secret.yaml.example."
|
||||
check_secret_key "$OAUTH_SECRET" cookie-secret \
|
||||
"Create it from manifests/exposure/gateway-api-sso/73-oauth2-proxy-secret.yaml.example."
|
||||
|
||||
# The cookie secret must be URL-safe unpadded base64 of 16, 24 or 32 bytes;
|
||||
# oauth2-proxy refuses anything else (the classic error is "44 bytes").
|
||||
if COOKIE_B64=$(kubectl -n "$NS" get secret "$OAUTH_SECRET" \
|
||||
-o jsonpath='{.data.cookie-secret}' 2>/dev/null); then
|
||||
if [ -n "$COOKIE_B64" ]; then
|
||||
if ! COOKIE_LEN=$(printf '%s' "$COOKIE_B64" | base64 -d 2>/dev/null | wc -c); then
|
||||
fail "Secret $OAUTH_SECRET key 'cookie-secret' is not valid base64. Generate it with: python3 -c 'import secrets; print(secrets.token_urlsafe(32))'"
|
||||
else
|
||||
case "$COOKIE_LEN" in
|
||||
16 | 24 | 32) pass "oauth2-proxy cookie-secret decodes to $COOKIE_LEN bytes" ;;
|
||||
*)
|
||||
fail "oauth2-proxy cookie-secret decodes to $COOKIE_LEN bytes; it must be 16, 24 or 32. Generate a URL-safe unpadded value with: python3 -c 'import secrets; print(secrets.token_urlsafe(32))'"
|
||||
;;
|
||||
esac
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
check_secret_key "$TLS_SECRET" tls.crt \
|
||||
"Create the TLS Secret with: kubectl -n $NS create secret tls $TLS_SECRET --cert=tls.crt --key=tls.key"
|
||||
check_secret_key "$TLS_SECRET" tls.key \
|
||||
"Create the TLS Secret with: kubectl -n $NS create secret tls $TLS_SECRET --cert=tls.crt --key=tls.key"
|
||||
|
||||
echo
|
||||
echo "== Gateway SSO configuration (deploy/$GATEWAY_DEPLOY)"
|
||||
GATEWAY_ENV=$(kubectl -n "$NS" get deployment "$GATEWAY_DEPLOY" \
|
||||
-o jsonpath='{range .spec.template.spec.containers[*].env[*]}{.name}={.value}{"\n"}{end}')
|
||||
|
||||
# env_value <NAME>: print the literal env value, or nothing when unset.
|
||||
env_value() {
|
||||
printf '%s\n' "$GATEWAY_ENV" | awk -F= -v key="$1" '$1 == key { sub(/^[^=]*=/, ""); print }'
|
||||
}
|
||||
env_is_set() {
|
||||
printf '%s\n' "$GATEWAY_ENV" | awk -F= -v key="$1" '$1 == key { found = 1 } END { exit found ? 0 : 1 }'
|
||||
}
|
||||
|
||||
if ! env_is_set AUTH_PROXY_HEADER; then
|
||||
fail "AUTH_PROXY_HEADER is not set on deploy/$GATEWAY_DEPLOY, so the gateway ignores oauth2-proxy's identity header and users are still asked for the gateway token. Fix: kubectl -n $NS set env deploy/$GATEWAY_DEPLOY AUTH_PROXY_HEADER=X-Forwarded-User"
|
||||
elif [ "$(env_value AUTH_PROXY_HEADER)" != "X-Forwarded-User" ]; then
|
||||
fail "AUTH_PROXY_HEADER is '$(env_value AUTH_PROXY_HEADER)', expected 'X-Forwarded-User'. Fix: kubectl -n $NS set env deploy/$GATEWAY_DEPLOY AUTH_PROXY_HEADER=X-Forwarded-User"
|
||||
else
|
||||
pass "AUTH_PROXY_HEADER=X-Forwarded-User"
|
||||
fi
|
||||
|
||||
if ! env_is_set COOKIE_SECURE; then
|
||||
fail "COOKIE_SECURE is not set on deploy/$GATEWAY_DEPLOY; the session cookie must be Secure behind TLS. Fix: kubectl -n $NS set env deploy/$GATEWAY_DEPLOY COOKIE_SECURE=true"
|
||||
elif [ "$(env_value COOKIE_SECURE | tr '[:upper:]' '[:lower:]')" != "true" ]; then
|
||||
fail "COOKIE_SECURE is '$(env_value COOKIE_SECURE)', expected 'true' behind TLS. Fix: kubectl -n $NS set env deploy/$GATEWAY_DEPLOY COOKIE_SECURE=true"
|
||||
else
|
||||
pass "COOKIE_SECURE=true"
|
||||
fi
|
||||
|
||||
if env_is_set ALLOW_INSECURE_COOKIES; then
|
||||
warn "ALLOW_INSECURE_COOKIES is still set on deploy/$GATEWAY_DEPLOY. It is ignored while COOKIE_SECURE=true but is a footgun: remove it with 'kubectl -n $NS set env deploy/$GATEWAY_DEPLOY ALLOW_INSECURE_COOKIES-'."
|
||||
fi
|
||||
|
||||
echo
|
||||
echo "== SSO objects and placeholders"
|
||||
if kubectl -n "$NS" get deployment "$OAUTH_DEPLOY" >/dev/null 2>&1; then
|
||||
pass "deploy/$OAUTH_DEPLOY exists"
|
||||
OAUTH_ARGS=$(kubectl -n "$NS" get deployment "$OAUTH_DEPLOY" \
|
||||
-o jsonpath='{range .spec.template.spec.containers[*].args[*]}{@}{"\n"}{end}')
|
||||
else
|
||||
OAUTH_ARGS=""
|
||||
fail "deploy/$OAUTH_DEPLOY not found. Apply the variant: kubectl apply -f manifests/exposure/gateway-api-sso/"
|
||||
fi
|
||||
|
||||
# GitHub org restriction is the only authorization in the shipped variant.
|
||||
# --email-domain=* is acceptable only because --github-org narrows membership;
|
||||
# a missing or empty org value would accept any GitHub account.
|
||||
if [ -n "$OAUTH_ARGS" ]; then
|
||||
if printf '%s\n' "$OAUTH_ARGS" | grep -q '^--provider=github$'; then
|
||||
pass "oauth2-proxy uses --provider=github"
|
||||
else
|
||||
fail "oauth2-proxy is not configured with --provider=github. Fix: add --provider=github to deploy/$OAUTH_DEPLOY."
|
||||
fi
|
||||
if printf '%s\n' "$OAUTH_ARGS" | grep -Eq '^--github-org=.+$'; then
|
||||
pass "oauth2-proxy restricts membership with --github-org"
|
||||
else
|
||||
fail "oauth2-proxy has no non-empty --github-org. Fix: set --github-org=<your-org> in deploy/$OAUTH_DEPLOY; otherwise --email-domain=* accepts any GitHub account."
|
||||
fi
|
||||
fi
|
||||
|
||||
if kubectl -n "$NS" get httproute dsh-gateway >/dev/null 2>&1; then
|
||||
pass "HTTPRoute dsh-gateway exists"
|
||||
ROUTE_STATE=$(kubectl -n "$NS" get httproute dsh-gateway \
|
||||
-o jsonpath='{.spec.hostnames[*]}' 2>/dev/null || true)
|
||||
else
|
||||
ROUTE_STATE=""
|
||||
fail "HTTPRoute dsh-gateway not found. Apply the variant: kubectl apply -f manifests/exposure/gateway-api-sso/"
|
||||
fi
|
||||
|
||||
if kubectl -n "$NS" get gateway dsh-gateway >/dev/null 2>&1; then
|
||||
pass "Gateway dsh-gateway exists"
|
||||
GATEWAY_STATE=$(kubectl -n "$NS" get gateway dsh-gateway \
|
||||
-o jsonpath='{range .spec.listeners[*]}{.hostname}{"\n"}{end}' 2>/dev/null || true)
|
||||
CERT_REFS=$(kubectl -n "$NS" get gateway dsh-gateway \
|
||||
-o jsonpath='{range .spec.listeners[*].tls.certificateRefs[*]}{.namespace}{"/"}{.name}{"\n"}{end}' 2>/dev/null || true)
|
||||
else
|
||||
GATEWAY_STATE=""
|
||||
CERT_REFS=""
|
||||
fail "Gateway dsh-gateway not found. Apply the variant: kubectl apply -f manifests/exposure/gateway-api-sso/"
|
||||
fi
|
||||
|
||||
# Placeholders are the default values in the committed manifests. They reach
|
||||
# the cluster if the variant is applied before editing them.
|
||||
PLACEHOLDER_RE='your-github-org|dsh-demo\.example\.com'
|
||||
PLACEHOLDER_STATE=$(printf '%s\n%s\n%s\n' "$OAUTH_ARGS" "$ROUTE_STATE" "$GATEWAY_STATE")
|
||||
if printf '%s\n' "$PLACEHOLDER_STATE" | grep -En "$PLACEHOLDER_RE" >/dev/null 2>&1; then
|
||||
fail "the applied SSO objects still contain a placeholder (your-github-org or dsh-demo.example.com):"
|
||||
printf '%s\n' "$PLACEHOLDER_STATE" | grep -En "$PLACEHOLDER_RE" | sed 's/^/ /' >&2
|
||||
echo " Replace --github-org, --redirect-url, the Gateway listener host and the HTTPRoute hostname with the real public FQDN and organization, then re-apply and restart the proxy." >&2
|
||||
else
|
||||
pass "no your-github-org / dsh-demo.example.com placeholder in the applied SSO objects"
|
||||
fi
|
||||
|
||||
# The Gateway must reference an existing TLS Secret; the default name is
|
||||
# dsh-demo-tls, but 70-gateway.yaml can be edited to another name.
|
||||
if [ -n "$CERT_REFS" ]; then
|
||||
while IFS= read -r ref; do
|
||||
[ -n "$ref" ] || continue
|
||||
ref_ns=${ref%%/*}
|
||||
ref_name=${ref#*/}
|
||||
ref_ns=${ref_ns:-$NS}
|
||||
if kubectl -n "$ref_ns" get secret "$ref_name" >/dev/null 2>&1; then
|
||||
pass "Gateway TLS certificate reference ${ref_ns}/${ref_name} exists"
|
||||
else
|
||||
fail "Gateway references TLS Secret ${ref_ns}/${ref_name} but it does not exist. Create it (see manifests/exposure/gateway-api-sso/74-tls-secret.yaml.example)."
|
||||
fi
|
||||
done <<<"$CERT_REFS"
|
||||
else
|
||||
warn "the Gateway has no TLS certificateRefs; SSO over plain HTTP would expose the session cookie. Terminate TLS at the Gateway listener."
|
||||
fi
|
||||
|
||||
# A second path to the gateway defeats SSO: anyone who can reach
|
||||
# dsh-gateway:8080 can spoof X-Forwarded-User. This is a best-effort check.
|
||||
if kubectl -n "$NS" get ingress >/dev/null 2>&1; then
|
||||
INGRESS_STATE=$(kubectl -n "$NS" get ingress -o json 2>/dev/null || true)
|
||||
if printf '%s' "$INGRESS_STATE" | grep -q '"name":"dsh-gateway"'; then
|
||||
fail "an Ingress in namespace $NS also routes to dsh-gateway. With SSO enabled the gateway must be reachable only through oauth2-proxy: delete the Ingress variant (kubectl delete -f manifests/exposure/ingress/)."
|
||||
fi
|
||||
fi
|
||||
|
||||
echo
|
||||
if [ "$FAILED" -ne 0 ]; then
|
||||
echo "preflight-sso: FAILED - fix the items above before exposing the gateway." >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "preflight-sso: ok - the SSO variant looks correctly configured."
|
||||
@@ -25,21 +25,30 @@ This example deliberately has no gateway: you reach DSH through a
|
||||
```
|
||||
|
||||
- A DeepSeek API key for real model calls. The pod starts without one; only
|
||||
model requests fail.
|
||||
model requests fail. Optionally create `dsh-provider-keys-extra` for
|
||||
OpenRouter, Gemini, NVIDIA or other provider keys (the template references it
|
||||
with `optional: true`).
|
||||
|
||||
## Quickstart
|
||||
|
||||
Run these from this example's directory (`examples/minimal-single-tenant/`).
|
||||
|
||||
```bash
|
||||
# 1. Namespace + shared provider key. The key is injected into the sandbox via
|
||||
# envFrom and is never committed.
|
||||
# 1. Namespace + shared provider keys. They are injected into the sandbox via
|
||||
# envFrom and are never committed.
|
||||
kubectl create namespace dsh-demo
|
||||
kubectl -n dsh-demo create secret generic dsh-provider-keys \
|
||||
--from-literal=DEEPSEEK_API_KEY="sk-..."
|
||||
|
||||
# Optional: extra providers, injected through a second Secret. Create it before
|
||||
# the warm pool so the pods are stamped with the keys.
|
||||
kubectl -n dsh-demo create secret generic dsh-provider-keys-extra \
|
||||
--from-literal=OPENROUTER_API_KEY="sk-or-..." \
|
||||
--from-literal=GEMINI_API_KEY="..." \
|
||||
--from-literal=NVIDIA_API_KEY="nvapi-..."
|
||||
|
||||
# 2. The template itself (image pinned in the manifest).
|
||||
kubectl apply -f manifests/sandbox-template.yaml
|
||||
kubectl apply -f manifests/10-sandboxtemplate.yaml
|
||||
|
||||
# 3. A template only describes pods; a warm pool turns it into running
|
||||
# Sandboxes and a claim adopts one. One replica is enough here.
|
||||
@@ -86,10 +95,48 @@ kubectl -n dsh-demo port-forward "pod/$SB" 18081:3080
|
||||
> use on your workstation, forward to another local port (here `18081`), never
|
||||
> `3080`. `localhost` is always in DSH's trusted-host list, so the forwarded
|
||||
> request is accepted.
|
||||
>
|
||||
> **gVisor caveat:** if you uncomment `runtimeClassName: gvisor` in the
|
||||
> template, `kubectl port-forward` no longer works. Use the multi-tenant gateway
|
||||
> example or a reachable Service instead; see
|
||||
> [optional hardening](../dsh-agents-as-a-service/docs/optional-hardening.md).
|
||||
|
||||
## Folder picker and the workspace
|
||||
|
||||
The template mounts the `dsh-home` PVC's `workspace/` subPath at
|
||||
`/workspace` and sets `HOME=/workspace`. DSH's directory picker (the folder
|
||||
explorer used to choose a workspace) has no start-path setting: it lists `$HOME`
|
||||
when it opens, so with the image default (`/home/node`) it would start on a
|
||||
directory that is not persisted. Pointing `HOME` at the PVC-backed workspace makes
|
||||
the picker open there (labelled *Home* in its breadcrumbs) and makes `~` in agent
|
||||
commands resolve to `/workspace`. The trade-off is that `$HOME`-relative tool
|
||||
caches and config (`.cache`, `.config`, `.local`, `.npm`, `.agents`) are written
|
||||
into the workspace on the PVC. DSH state is unaffected: `DSH_HOME` stays
|
||||
`/home/node/.dsh`. The rationale and the list of affected paths are in
|
||||
[`10-sandboxtemplate.yaml`](../dsh-agents-as-a-service/manifests/10-sandboxtemplate.yaml).
|
||||
Pod specs are stamped at creation, so an existing sandbox keeps its old mounts and
|
||||
`HOME` until it is recreated; the PVC retains DSH state and workspace data.
|
||||
|
||||
Verify inside a running sandbox:
|
||||
|
||||
```bash
|
||||
kubectl -n dsh-demo exec "$SB" -- printenv HOME # /workspace
|
||||
```
|
||||
|
||||
## Network policy and optional hardening
|
||||
|
||||
The template omits `networkPolicyManagement`, so the agent-sandbox controller
|
||||
uses its Managed secure default: ingress only from the sandbox-router, public
|
||||
egress, and no internal cluster DNS. `kubectl port-forward` is local to the pod
|
||||
and still works (except under gVisor).
|
||||
|
||||
gVisor, an optional `nodeSelector`, extra provider keys and a stricter sandbox
|
||||
NetworkPolicy are documented in
|
||||
[optional hardening](../dsh-agents-as-a-service/docs/optional-hardening.md).
|
||||
|
||||
## Probes and trusted hosts
|
||||
|
||||
Two settings in `manifests/sandbox-template.yaml` are easy to get wrong:
|
||||
Two settings in `manifests/10-sandboxtemplate.yaml` are easy to get wrong:
|
||||
|
||||
- **Probe path.** `/` returns `401` without the launch-token cookie, so a probe
|
||||
there crash-loops the pod. The template probes `/favicon.svg` (served
|
||||
|
||||
@@ -10,7 +10,10 @@
|
||||
# - Namespace dsh-demo.
|
||||
# - Secret dsh-provider-keys in dsh-demo with key DEEPSEEK_API_KEY
|
||||
# (injected via envFrom; the pod starts without it, only model calls fail).
|
||||
# The agent-sandbox controller (v1.0.7, API v1beta1) must be installed.
|
||||
# - Optional: Secret dsh-provider-keys-extra with additional provider keys
|
||||
# (OPENROUTER_API_KEY, GEMINI_API_KEY, NVIDIA_API_KEY, ...). It is
|
||||
# referenced with `optional: true`, so it may be absent.
|
||||
# The agent-sandbox controller (v1.0.8, API v1beta1) must be installed.
|
||||
apiVersion: extensions.agents.x-k8s.io/v1beta1
|
||||
kind: SandboxTemplate
|
||||
metadata:
|
||||
@@ -18,8 +21,26 @@ metadata:
|
||||
namespace: dsh-demo
|
||||
spec:
|
||||
service: true # headless Service per Sandbox -> stable DNS (status.serviceFQDN)
|
||||
# networkPolicyManagement is omitted, so the controller uses its Managed
|
||||
# secure default (ingress only from the sandbox-router, public egress). This
|
||||
# quickstart reaches the sandbox through `kubectl port-forward`, which is
|
||||
# local to the pod. The multi-tenant example documents opt-in policies in
|
||||
# examples/dsh-agents-as-a-service/docs/optional-hardening.md.
|
||||
#
|
||||
# OPTIONAL: recreate a Failed sandbox Pod instead of retaining it
|
||||
# (agent-sandbox v1.0.8+). Documented with the multi-tenant example.
|
||||
# failedPodPolicy: Recreate
|
||||
podTemplate:
|
||||
spec:
|
||||
# OPTIONAL: run under gVisor. gVisor breaks `kubectl port-forward`, so
|
||||
# this quickstart's only access path will not work; use the multi-tenant
|
||||
# gateway example or a Service/NodePort instead.
|
||||
# runtimeClassName: gvisor
|
||||
#
|
||||
# OPTIONAL: pin to a dedicated node pool. Replace the placeholder with a
|
||||
# node label; add a matching toleration if the pool is tainted.
|
||||
# nodeSelector:
|
||||
# dsh.example.com/node-pool: dsh
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 1000 # Matches the UID of the 'node' user
|
||||
@@ -35,7 +56,7 @@ spec:
|
||||
fsGroupChangePolicy: OnRootMismatch
|
||||
containers:
|
||||
- name: dsh-web
|
||||
image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.3-r3
|
||||
image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.5-r6
|
||||
imagePullPolicy: IfNotPresent
|
||||
ports:
|
||||
- name: web
|
||||
@@ -43,7 +64,26 @@ spec:
|
||||
envFrom:
|
||||
- secretRef:
|
||||
name: dsh-provider-keys # Secret with key DEEPSEEK_API_KEY
|
||||
# Optional extra provider keys; an absent Secret is safe.
|
||||
- secretRef:
|
||||
name: dsh-provider-keys-extra
|
||||
optional: true
|
||||
env:
|
||||
# Point the OS home at the persisted agent workspace. DSH's
|
||||
# directory picker (the folder explorer used to choose a workspace)
|
||||
# lists $HOME when it opens and has no start-path setting, so with
|
||||
# the image default (/home/node, not on a volume) it would open on a
|
||||
# throwaway directory. /workspace is the PVC-backed WORKDIR, so the
|
||||
# picker opens on the tree that survives suspend/resume, and `~` in
|
||||
# agent commands resolves there. Trade-off: tools that keep caches
|
||||
# or config under $HOME (npm/pnpm, uv, git, gh, DSH's global
|
||||
# ~/.agents instructions and skills) write dotfiles onto the PVC
|
||||
# inside the agent workspace. DSH_HOME is explicit
|
||||
# (/home/node/.dsh), so sessions, config and credentials are
|
||||
# unaffected. Rationale in full at
|
||||
# examples/dsh-agents-as-a-service/manifests/10-sandboxtemplate.yaml.
|
||||
- name: HOME
|
||||
value: /workspace
|
||||
# Empty => DSH derives links from the request Host. Keep it empty
|
||||
# when fronting the sandbox with a proxy/gateway.
|
||||
- name: DSH_PUBLIC_URL
|
||||
@@ -56,9 +96,25 @@ spec:
|
||||
# examples/dsh-agents-as-a-service/manifests/exposure/README.md.
|
||||
- name: DSH_TRUSTED_HOSTS
|
||||
value: ""
|
||||
# DSH's own bash file sandbox (bubblewrap, else Landlock) cannot be
|
||||
# enforced in this hardened pod - bubblewrap is installed but is
|
||||
# unsetuid and needs an unprivileged user namespace - and fails
|
||||
# closed with SANDBOX_UNAVAILABLE. The pod is the boundary here;
|
||||
# keep the deployment override in sync with the multi-tenant example
|
||||
# and see its README troubleshooting row to restore a confined mode.
|
||||
- name: DSH_PERMISSION_MODE
|
||||
value: danger-full-access
|
||||
volumeMounts:
|
||||
- name: dsh-home
|
||||
mountPath: /home/node/.dsh # DSH_HOME
|
||||
# The agent workspace (the image's WORKDIR) is the same PVC under
|
||||
# its own subPath, so DSH state - including .credentials.yaml -
|
||||
# never shows up as workspace data. Kubelet creates workspace/ on
|
||||
# first mount with the volume's fsGroup permissions; no init
|
||||
# container is needed. See the multi-tenant template for details.
|
||||
- name: dsh-home
|
||||
mountPath: /workspace
|
||||
subPath: workspace
|
||||
# / is behind DSH's launch-token cookie (401 without it); use an
|
||||
# unauthenticated path for probes. The startup probe gives a cold
|
||||
# start (plugin activation on a small node, or the first resume of a
|
||||
@@ -20,6 +20,13 @@ for dir in examples/*/; do
|
||||
if [ -d "${dir}manifests" ]; then
|
||||
count=$(find "${dir}manifests" -maxdepth 1 -type f -name '*.yaml' | wc -l | tr -d ' ')
|
||||
[ "$count" -gt 0 ] || err "$name: manifests/ has no top-level *.yaml files"
|
||||
for manifest in "${dir}"manifests/*.yaml; do
|
||||
[ -e "$manifest" ] || continue
|
||||
case "$(basename "$manifest")" in
|
||||
[0-9][0-9]-*.yaml) ;;
|
||||
*) err "$name: top-level manifest $(basename "$manifest") must be named NN-*.yaml" ;;
|
||||
esac
|
||||
done
|
||||
fi
|
||||
|
||||
stray=$(find "$dir" -maxdepth 1 -type f -name '*.yaml' | wc -l | tr -d ' ')
|
||||
@@ -29,6 +36,10 @@ for dir in examples/*/; do
|
||||
err "$name: manifests/exposure/ requires a README.md describing the variants"
|
||||
fi
|
||||
|
||||
if [ -d "${dir}manifests/optional" ] && [ ! -f "${dir}manifests/optional/README.md" ]; then
|
||||
err "$name: manifests/optional/ requires a README.md describing the opt-in manifests"
|
||||
fi
|
||||
|
||||
for component in "$dir"*/; do
|
||||
[ -d "$component" ] || continue
|
||||
case "$(basename "$component")" in
|
||||
|
||||
@@ -22,22 +22,60 @@ expect() { # <file> <literal>
|
||||
|
||||
# Main Dockerfile build args
|
||||
expect Dockerfile "ARG DSH_VERSION=$DSH_VERSION"
|
||||
expect Dockerfile "ARG DSH_COMMIT=$DSH_COMMIT"
|
||||
expect Dockerfile "ARG PNPM_VERSION=$PNPM_VERSION"
|
||||
expect Dockerfile "ARG RELEASE_REVISION=$RELEASE_REVISION"
|
||||
expect Dockerfile "ARG NODE_BUILD_IMAGE=$NODE_BUILD_IMAGE"
|
||||
expect Dockerfile "ARG NODE_RUNTIME_IMAGE=$NODE_RUNTIME_IMAGE"
|
||||
# The build stage must fail closed when the tag no longer resolves to DSH_COMMIT.
|
||||
expect Dockerfile "git -C /src rev-parse HEAD"
|
||||
|
||||
# Example gateway build base image
|
||||
expect examples/dsh-agents-as-a-service/gateway/Dockerfile \
|
||||
"ARG GATEWAY_PYTHON_IMAGE=$GATEWAY_PYTHON_IMAGE"
|
||||
|
||||
# Local configuration mirrored for humans
|
||||
expect .env.sample "DSH_VERSION=$DSH_VERSION"
|
||||
expect .env.sample "DSH_COMMIT=$DSH_COMMIT"
|
||||
expect .env.sample "PNPM_VERSION=$PNPM_VERSION"
|
||||
expect .env.sample "RELEASE_REVISION=$RELEASE_REVISION"
|
||||
|
||||
# Human-facing docs embed the resolved commit in the build example and table.
|
||||
expect docs/main-image.md "$DSH_COMMIT"
|
||||
expect docs/versioning.md "$DSH_COMMIT"
|
||||
|
||||
# Every container image is pinned to an immutable multi-arch index digest, not
|
||||
# a mutable tag. This is a supply-chain invariant: don't "fix" a failing check
|
||||
# by dropping the digest.
|
||||
for image_var in NODE_BUILD_IMAGE NODE_RUNTIME_IMAGE GATEWAY_PYTHON_IMAGE OAUTH2_PROXY_IMAGE; do
|
||||
image_value="${!image_var:-}"
|
||||
if ! printf '%s' "$image_value" | grep -qE '@sha256:[0-9a-f]{64}$'; then
|
||||
err "versions.env: $image_var must be digest-pinned (name:tag@sha256:<64 hex>), got: ${image_value:-<unset>}"
|
||||
fi
|
||||
done
|
||||
|
||||
if ! printf '%s' "${DSH_COMMIT:-}" | grep -qE '^[0-9a-f]{40}$'; then
|
||||
err "versions.env: DSH_COMMIT must be a full 40-character commit SHA, got: ${DSH_COMMIT:-<unset>}"
|
||||
fi
|
||||
|
||||
# Build entrypoints must pass DSH_COMMIT rather than trusting the tag alone.
|
||||
expect Makefile "--build-arg DSH_COMMIT=\$(DSH_COMMIT)"
|
||||
for wf in build-main.yaml release.yaml ci-image.yaml; do
|
||||
expect ".github/workflows/$wf" "DSH_COMMIT="
|
||||
done
|
||||
|
||||
# Image references in the example manifests
|
||||
for f in \
|
||||
examples/dsh-agents-as-a-service/manifests/10-sandboxtemplate.yaml \
|
||||
examples/minimal-single-tenant/manifests/sandbox-template.yaml; do
|
||||
examples/minimal-single-tenant/manifests/10-sandboxtemplate.yaml; do
|
||||
expect "$f" "image: $GHCR_IMAGE:$IDENTITY"
|
||||
done
|
||||
expect examples/dsh-agents-as-a-service/manifests/50-gateway.yaml "image: $GATEWAY_IMAGE:$IDENTITY"
|
||||
|
||||
# The oauth2-proxy exposure variant pins the digest, not the mutable tag.
|
||||
expect examples/dsh-agents-as-a-service/manifests/exposure/gateway-api-sso/72-oauth2-proxy.yaml \
|
||||
"image: $OAUTH2_PROXY_IMAGE"
|
||||
|
||||
# The gateway image must live below the repository image (one package per
|
||||
# example component, no unrelated top-level packages).
|
||||
case "$GATEWAY_IMAGE" in
|
||||
|
||||
@@ -3,7 +3,12 @@
|
||||
# "make check".
|
||||
#
|
||||
# ./scripts/set-version.sh dsh-v0.2.2-rc.1 # new DSH ref, no revision
|
||||
# ./scripts/set-version.sh dsh-v0.2.1-rc.3 r3 # same ref, new repo revision
|
||||
# ./scripts/set-version.sh dsh-v0.2.1-rc.5-r6 r3 # same ref, new repo revision
|
||||
#
|
||||
# For a new DSH ref the script resolves the tag's commit with "git ls-remote"
|
||||
# (peeled ^{} commit first, tag ref as fallback) and rewrites DSH_COMMIT in
|
||||
# versions.env, the Dockerfile ARG default and .env.sample. A revision-only bump
|
||||
# at the same ref keeps the existing commit and needs no network access.
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
@@ -23,9 +28,11 @@ if [ -n "$new_revision" ]; then
|
||||
fi
|
||||
|
||||
# shellcheck source=/dev/null
|
||||
old_version="$(. ./versions.env; printf '%s' "$DSH_VERSION")"
|
||||
# shellcheck source=/dev/null
|
||||
old_revision="$(. ./versions.env; printf '%s' "${RELEASE_REVISION:-}")"
|
||||
. ./versions.env
|
||||
|
||||
old_version="$DSH_VERSION"
|
||||
old_revision="${RELEASE_REVISION:-}"
|
||||
old_commit="${DSH_COMMIT:-}"
|
||||
|
||||
old_identity="$old_version"
|
||||
[ -z "$old_revision" ] || old_identity="$old_version-$old_revision"
|
||||
@@ -37,6 +44,29 @@ if [ "$old_identity" = "$new_identity" ]; then
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Resolve the commit the new upstream tag points at. Annotated tags expose a
|
||||
# peeled ^{} line (the commit) and a tag-object line; prefer the peeled commit
|
||||
# and fall back to the tag ref for lightweight tags. A repository-only revision
|
||||
# bump keeps the already pinned commit, so it needs no network.
|
||||
new_commit="$old_commit"
|
||||
if [ "$new_version" != "$old_version" ] || [ -z "$new_commit" ]; then
|
||||
if ! refs="$(git ls-remote --tags --exit-code "$DSH_REPO" \
|
||||
"refs/tags/$new_version^{}" "refs/tags/$new_version")"; then
|
||||
echo "set-version: upstream tag '$new_version' not found in $DSH_REPO" >&2
|
||||
exit 1
|
||||
fi
|
||||
new_commit="$(printf '%s\n' "$refs" \
|
||||
| awk -v t="refs/tags/$new_version^{}" '$2 == t {print $1}' | head -n1)"
|
||||
if [ -z "$new_commit" ]; then
|
||||
new_commit="$(printf '%s\n' "$refs" \
|
||||
| awk -v t="refs/tags/$new_version" '$2 == t {print $1}' | head -n1)"
|
||||
fi
|
||||
if ! printf '%s' "$new_commit" | grep -qE '^[0-9a-f]{40}$'; then
|
||||
echo "set-version: could not resolve a commit for '$new_version' in $DSH_REPO" >&2
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
# Replace the identity first (it contains the base version), then the base
|
||||
# version itself. CHANGELOG.md is generated from tags, so it is left alone.
|
||||
while IFS= read -r file; do
|
||||
@@ -48,17 +78,30 @@ while IFS= read -r file; do
|
||||
sed -i "s/$old_identity/$new_identity/g; s/$old_version/$new_version/g" "$file"
|
||||
done < <(git ls-files)
|
||||
|
||||
# versions.env is authoritative: write both keys explicitly so an empty
|
||||
# versions.env is authoritative: write these keys explicitly so an empty
|
||||
# revision is stored as an empty value.
|
||||
sed -i "s/^DSH_VERSION=.*/DSH_VERSION=$new_version/" versions.env
|
||||
sed -i "s/^RELEASE_REVISION=.*/RELEASE_REVISION=$new_revision/" versions.env
|
||||
sed -i "s/^DSH_COMMIT=.*/DSH_COMMIT=$new_commit/" versions.env
|
||||
|
||||
# Values stored separately from the full identity: Dockerfile ARG defaults and
|
||||
# the human-facing sample cannot be reached by the identity replacement above.
|
||||
sed -i "s/^ARG DSH_VERSION=.*/ARG DSH_VERSION=$new_version/" Dockerfile
|
||||
sed -i "s/^ARG RELEASE_REVISION=.*/ARG RELEASE_REVISION=$new_revision/" Dockerfile
|
||||
sed -i "s/^ARG DSH_COMMIT=.*/ARG DSH_COMMIT=$new_commit/" Dockerfile
|
||||
sed -i "s/^DSH_VERSION=.*/DSH_VERSION=$new_version/" .env.sample
|
||||
sed -i "s/^RELEASE_REVISION=.*/RELEASE_REVISION=$new_revision/" .env.sample
|
||||
sed -i "s/^DSH_COMMIT=.*/DSH_COMMIT=$new_commit/" .env.sample
|
||||
|
||||
# The human-facing build examples embed the full SHA, which the version
|
||||
# replacement above cannot see. Keep them in step; check-versions.sh verifies.
|
||||
if [ -n "$old_commit" ] && [ "$old_commit" != "$new_commit" ]; then
|
||||
for doc in docs/main-image.md docs/versioning.md; do
|
||||
[ -f "$doc" ] || continue
|
||||
sed -i "s/$old_commit/$new_commit/g" "$doc"
|
||||
done
|
||||
fi
|
||||
|
||||
echo "set-version: $old_identity -> $new_identity"
|
||||
echo "set-version: DSH_COMMIT=$new_commit"
|
||||
echo "set-version: run 'make check' and review the diff"
|
||||
|
||||
32
versions.env
32
versions.env
@@ -2,23 +2,41 @@
|
||||
# repository. Consumed by the Makefile, build.sh, scripts/*.sh and the GitHub
|
||||
# Actions workflows. Never hard-code these values elsewhere: run "make check"
|
||||
# (scripts/check-versions.sh) to verify every pin.
|
||||
#
|
||||
# Every container image below is pinned to an immutable multi-arch index digest
|
||||
# (name:tag@sha256:...), not just a mutable tag. Bump the tag and the digest
|
||||
# together; scripts/check-versions.sh requires the digest form.
|
||||
|
||||
# Upstream DeepSeek Harness source, and the ref this repository packages.
|
||||
# DSH_COMMIT is the full commit SHA that DSH_VERSION resolves to upstream; the
|
||||
# Dockerfile verifies the clone's HEAD against it and fails the build on drift.
|
||||
DSH_REPO=https://github.com/deepseek-ai/deepseek-harness.git
|
||||
DSH_VERSION=dsh-v0.2.1-rc.3
|
||||
DSH_VERSION=dsh-v0.2.1-rc.5
|
||||
DSH_COMMIT=b830ec950faba0a46dda564be3e2d854afed94f1
|
||||
PNPM_VERSION=11.28.5
|
||||
|
||||
# Release identity = DSH_VERSION plus an optional repository revision.
|
||||
# An empty RELEASE_REVISION is the first release at this DSH ref; rN (N >= 2)
|
||||
# marks a repository-only re-release at the same DSH ref. The dsh-v0.2.1-rc.3
|
||||
# image tag is already published on GHCR and tags are immutable, so the first
|
||||
# release under the current scheme is dsh-v0.2.1-rc.3-r3.
|
||||
RELEASE_REVISION=r3
|
||||
# marks a repository-only re-release at the same DSH ref. This is the first
|
||||
# release at dsh-v0.2.1-rc.5-r6, so the identity is the plain tag; a later
|
||||
# repository-only change at the same ref becomes -r2.
|
||||
RELEASE_REVISION=r6
|
||||
|
||||
# Base images for the main DSH Web image (root Dockerfile), pinned by digest.
|
||||
NODE_BUILD_IMAGE=docker.io/node:24-trixie@sha256:8c761eca2bf7eec965f946cf47b8090143cb41f45f2f8699af0cc0298d2d2d19
|
||||
NODE_RUNTIME_IMAGE=docker.io/node:24-trixie-slim@sha256:54f84b4cf8432b21aaffd5d3ce8d41e12cea0d10d33c09d2f3c9879d49d6b446
|
||||
|
||||
# Base image for the example gateway (examples/dsh-agents-as-a-service/gateway).
|
||||
GATEWAY_PYTHON_IMAGE=docker.io/python:3.12-slim@sha256:aa943ce411423c7582288f18707ff66eddcbaeb510502d747a72aeb34f30d159
|
||||
|
||||
# Control-plane versions used by the examples and their manifests.
|
||||
AGENT_SANDBOX_VERSION=v1.0.7
|
||||
# The matching SHA-256s pin the release manifests that CI downloads and
|
||||
# applies; update both the version and the digest when bumping either pin.
|
||||
AGENT_SANDBOX_VERSION=v1.0.9
|
||||
AGENT_SANDBOX_MANIFEST_SHA256=4bf9ee5558e9a663f91cfa5963e96f2591bfaa1dcf79f03bfab65bc5d65cc699
|
||||
GATEWAY_API_VERSION=v1.2.1
|
||||
OAUTH2_PROXY_IMAGE=quay.io/oauth2-proxy/oauth2-proxy:v7.15.5
|
||||
GATEWAY_API_MANIFEST_SHA256=97598bf6ab3b33b9b5c5432bdd24de091e4e9c3aa0575ebb0710a2a19cd64d64
|
||||
OAUTH2_PROXY_IMAGE=quay.io/oauth2-proxy/oauth2-proxy:v7.15.5@sha256:8498b0d0ef0a7b29686414000a08aee467f02d0299c9ed1e006a8f33fc017916
|
||||
|
||||
# Registry layout: the main image is the repository itself, and every example
|
||||
# component is a nested package below it (<registry>/<owner>/<repo>/<example>/<component>).
|
||||
|
||||
Reference in New Issue
Block a user