diff --git a/.dsh/skills/dsh-agent-sandbox-platform/SKILL.md b/.dsh/skills/dsh-agent-sandbox-platform/SKILL.md index a573f96..1ef09b4 100644 --- a/.dsh/skills/dsh-agent-sandbox-platform/SKILL.md +++ b/.dsh/skills/dsh-agent-sandbox-platform/SKILL.md @@ -129,6 +129,10 @@ Read `docs/examples.md` and the testing table in `AGENTS.md` before editing. `make check` and the e2e. Never mix a bump with an unrelated feature. - **MCP integration:** pin the build to the cluster's tag, least-privilege SA, ClusterIP + auth proxy, raised DSH timeout, no secrets in the repo. +- **Optional hardening:** gVisor, `nodeSelector`, NetworkPolicy and extra + provider keys stay opt-in; the sandbox policy lives under + `manifests/optional/`. All four are documented in + `examples/dsh-agents-as-a-service/docs/optional-hardening.md`. - **Docs:** update the affected doc in the same change. Details and review checklist: diff --git a/.dsh/skills/dsh-agent-sandbox-platform/references/extending-the-repo.md b/.dsh/skills/dsh-agent-sandbox-platform/references/extending-the-repo.md index 692903f..7de9bba 100644 --- a/.dsh/skills/dsh-agent-sandbox-platform/references/extending-the-repo.md +++ b/.dsh/skills/dsh-agent-sandbox-platform/references/extending-the-repo.md @@ -25,6 +25,7 @@ implementation is a self-contained example under `examples//`, and | --- | --- | --- | | Documentation only | `*.md`, `docs/` | `make check`; relative links resolve | | Example manifest | `examples//manifests/NN-*.yaml`, README step | `make check`; server-side dry-run in CI | +| Optional hardening | `manifests/optional/`, exposure NPs, example `docs/optional-hardening.md` | `make check`; manual verification on a capable cluster | | Gateway behavior/route | `gateway/gateway.py`, `gateway/test_gateway.py`, gateway README, architecture | `make test`, `make lint`; update tests and API docs | | Exposure variant | `manifests/exposure//`, exposure README, example README | `make check`; apply exactly one variant | | New example | full `examples//` layout, `examples/README.md` | `make check`; CI dry-run; optional e2e | diff --git a/.dsh/skills/dsh-agent-sandbox-platform/references/platform-runbook.md b/.dsh/skills/dsh-agent-sandbox-platform/references/platform-runbook.md index c0de958..a29664c 100644 --- a/.dsh/skills/dsh-agent-sandbox-platform/references/platform-runbook.md +++ b/.dsh/skills/dsh-agent-sandbox-platform/references/platform-runbook.md @@ -141,8 +141,13 @@ host. A resource-only install has no gateway: that is a warning, not a failure. ## 9. Security boundaries -- One `DEEPSEEK_API_KEY` is shared by all sandboxes; the example is not +- Provider keys (`dsh-provider-keys` plus the optional + `dsh-provider-keys-extra`) are shared by all sandboxes; the example is not tenant-isolated. +- gVisor, `nodeSelector`, the sandbox/exposure NetworkPolicies and extra + provider keys are opt-in, not active by default. A NetworkPolicy only helps + with a CNI that enforces it; see + [optional hardening](../../../../examples/dsh-agents-as-a-service/docs/optional-hardening.md). - Secret data, kubeconfigs, and pod logs are sensitive; RBAC may allow Secret access, policy does not. Keep gateway RBAC namespace-scoped; it needs pod logs to find the launch token. diff --git a/.env.sample b/.env.sample index 1afcf45..a095090 100644 --- a/.env.sample +++ b/.env.sample @@ -5,6 +5,21 @@ # --- Main DSH Web image (root Dockerfile) --- DEEPSEEK_API_KEY= +# Optional additional provider keys consumed by DSH. For Docker, pass them with +# -e; the Kubernetes examples inject them through the optional +# dsh-provider-keys-extra Secret. +OPENROUTER_API_KEY= +GEMINI_API_KEY= +NVIDIA_API_KEY= +OPENAI_API_KEY= +ANTHROPIC_API_KEY= +GROQ_API_KEY= +XAI_API_KEY= +MISTRAL_API_KEY= +TOGETHER_API_KEY= +CEREBRAS_API_KEY= +AI_GATEWAY_API_KEY= +HF_TOKEN= DSH_VERSION=dsh-v0.2.1-rc.3 RELEASE_REVISION=r3 PNPM_VERSION=11.28.5 diff --git a/AGENTS.md b/AGENTS.md index 3eea7f3..159a6e8 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -72,6 +72,7 @@ needs kind plus the agent-sandbox CRDs (see the example README). | --- | --- | | Documentation only | make check; verify relative links | | Example manifest | make check; kubectl apply --dry-run=server in CI | +| Optional hardening (gVisor, nodeSelector, NetworkPolicy, provider keys) | make check; server dry-run in CI; manual verification on a capable cluster | | Gateway code | make test; make lint; add or update tests | | Dockerfile / versions.env | make check; make image (amd64) and a container smoke test | | Workflow | actionlint in CI; keep actions SHA-pinned; no hard-coded versions | @@ -120,8 +121,11 @@ needs kind plus the agent-sandbox CRDs (see the example README). can spoof X-Forwarded-User. - The example gateway reads the DSH launch token from pod logs. Treat pod logs as sensitive and keep its RBAC namespace-scoped. -- The example shares one DEEPSEEK_API_KEY across sandboxes; do not describe it - as tenant-isolated. +- The example shares provider keys (dsh-provider-keys plus the optional + dsh-provider-keys-extra) across sandboxes; do not describe it as + tenant-isolated. +- gVisor, nodeSelector and the NetworkPolicies are opt-in and are not active in + the base manifests; do not describe the base example as hardened. - TLS terminates outside the gateway; set COOKIE_SECURE=true for HTTPS and keep the base manifest safe for local HTTP. diff --git a/README.md b/README.md index 75e3f34..5dcbbb7 100644 --- a/README.md +++ b/README.md @@ -72,6 +72,13 @@ docker run -d --name dsh-web -p 18081:3080 \ | `DSH_TRUSTED_HOSTS` | empty | Extra trusted `Host`/`Origin` entries, comma/space separated. | | `DSH_HOME` | `/home/node/.dsh` | DSH state directory (mount a volume here). | +DSH also resolves the standard per-provider environment names — +`OPENROUTER_API_KEY`, `GEMINI_API_KEY`, `NVIDIA_API_KEY`, `OPENAI_API_KEY`, +`ANTHROPIC_API_KEY`, `GROQ_API_KEY`, `XAI_API_KEY`, `HF_TOKEN`, and others — +through its packaged provider catalog. Inject them the same way; a key saved in +DSH's credential store wins over the environment. See +[provider credentials](docs/main-image.md#provider-credentials). + Full contract, build args, probes, volumes and signals: [docs/main-image.md](docs/main-image.md). @@ -125,12 +132,14 @@ Self-contained implementations that consume the image. Index and contract: | Example | What it demonstrates | | --- | --- | -| [`examples/dsh-agents-as-a-service/`](examples/dsh-agents-as-a-service/) | Full multi-tenant platform: SandboxTemplate with per-user PVCs, warm pool, one claim per user, suspend-when-idle, wake-on-return, browser gateway with signup + `HttpOnly` sessions + HTTP/WebSocket proxy, Ingress or Gateway API + GitHub SSO exposure. Start with the [walkthrough](examples/dsh-agents-as-a-service/README.md) and the [architecture](examples/dsh-agents-as-a-service/docs/architecture.md). | +| [`examples/dsh-agents-as-a-service/`](examples/dsh-agents-as-a-service/) | Full multi-tenant platform: SandboxTemplate with per-user PVCs, warm pool, one claim per user, suspend-when-idle, wake-on-return, browser gateway with signup + `HttpOnly` sessions + HTTP/WebSocket proxy, Ingress or Gateway API + GitHub SSO exposure, and opt-in gVisor/NetworkPolicy/provider-key hardening. Start with the [walkthrough](examples/dsh-agents-as-a-service/README.md) and the [architecture](examples/dsh-agents-as-a-service/docs/architecture.md). | | [`examples/minimal-single-tenant/`](examples/minimal-single-tenant/) | The smallest useful setup: one `SandboxTemplate` and one `SandboxClaim`, reached through `kubectl port-forward` + the launch-token URL. Manifest: [`manifests/sandbox-template.yaml`](examples/minimal-single-tenant/manifests/sandbox-template.yaml). | The gateway is a component of the multi-tenant example and has its own [README](examples/dsh-agents-as-a-service/gateway/README.md) (API, configuration, -auth/SSO, limitations). +auth/SSO, limitations). The example also documents opt-in gVisor, +`nodeSelector`, NetworkPolicy and extra provider-key hardening in +[optional hardening](examples/dsh-agents-as-a-service/docs/optional-hardening.md). ## Agent skills diff --git a/SECURITY.md b/SECURITY.md index f12987e..7942ffa 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -25,10 +25,17 @@ disclosure once a fix is available. - **SSO header trust boundary.** AUTH_PROXY_HEADER makes the gateway trust an identity header (for example X-Forwarded-User). Enable it only when the gateway Service is reachable exclusively through the authenticating proxy; - otherwise any client can spoof the header. -- **Shared provider key.** The example injects one DEEPSEEK_API_KEY into every - tenant sandbox (a warm-pool constraint). Treat tenants as mutually trusting - until per-tenant credentials or an egress proxy exist. + otherwise any client can spoof the header. The SSO exposure variant ships a + NetworkPolicy that restricts gateway access to oauth2-proxy pods; a custom + exposure needs an equivalent. +- **Shared provider keys.** The example injects DEEPSEEK_API_KEY and any keys in + the optional dsh-provider-keys-extra Secret into every tenant sandbox (a + warm-pool constraint). Treat tenants as mutually trusting until per-tenant + credentials or an egress proxy exist. +- **Optional isolation is opt-in.** gVisor, nodeSelector, the sandbox/exposure + NetworkPolicies and extra provider keys are opt-in. Without a + NetworkPolicy-capable CNI or a gVisor runtime, the base example is not + tenant-isolated; see examples/dsh-agents-as-a-service/docs/optional-hardening.md. - **TLS is external.** Terminate TLS at an Ingress or Gateway and set COOKIE_SECURE=true. Plain HTTP outside localhost exposes session cookies. - **No secrets in the repository.** The repository only ships empty Secret diff --git a/docs/examples.md b/docs/examples.md index 6e6d132..9f81eea 100644 --- a/docs/examples.md +++ b/docs/examples.md @@ -18,6 +18,9 @@ examples// | # ordered steps, troubleshooting, cleanup +-- manifests/ # required | +-- NN-*.yaml # numbered top-level manifests, plain kubectl +| +-- optional/ # optional opt-in manifests, applied by path +| +-- README.md # required when optional/ exists +| +-- /NN-*.yaml # e.g. network-policy/ | +-- exposure/ # optional exposure variants | +-- README.md # required when exposure/ exists | +-- /NN-*.yaml # e.g. ingress/, gateway-api-sso/ @@ -41,6 +44,8 @@ examples// - a `*.yaml` file sits at the example root instead of under `manifests/`; - `manifests/exposure/` exists without a `manifests/exposure/README.md` describing the variants; +- `manifests/optional/` exists without a `manifests/optional/README.md` + describing the opt-in manifests; - a subdirectory with a `Dockerfile` has no `README.md`; - a file in `scripts/` is not executable. @@ -68,7 +73,12 @@ make check command; real values are generated per install. - **Cluster-specific state stays out.** Node affinity, replica counts tuned for a cluster, and similar fields belong in a local patch, not in the shipped - manifest. + manifest. Optional scheduler/runtime hints may ship **commented out** with + placeholder labels; they must not be active. +- **Optional manifests are opt-in.** Put them under + `manifests/optional//`, document them in + `manifests/optional/README.md`, and apply them by path. The top-level + `kubectl apply -f manifests/` is non-recursive and does not apply them. - **Placeholders are explicit.** Hostnames, GitHub orgs and TLS material use values like `dsh.example.com` / `your-github-org` with a TODO comment. diff --git a/docs/main-image.md b/docs/main-image.md index 03d267a..c973996 100644 --- a/docs/main-image.md +++ b/docs/main-image.md @@ -80,6 +80,28 @@ The entrypoint is `tini -- /usr/local/bin/dsh-entrypoint`: The image also sets `NODE_ENV=production`, `HOME=/home/node`, and a `PATH` that includes `/app/node_modules/.bin`. +### Provider credentials + +DSH resolves provider API keys from the environment when no stored credential +exists. The packaged provider catalog maps each provider to its standard +environment name. Commonly used names: + +`DEEPSEEK_API_KEY`, `OPENROUTER_API_KEY`, `GEMINI_API_KEY`, `NVIDIA_API_KEY`, +`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GROQ_API_KEY`, `XAI_API_KEY`, +`MISTRAL_API_KEY`, `TOGETHER_API_KEY`, `CEREBRAS_API_KEY`, +`AI_GATEWAY_API_KEY`, `HF_TOKEN`. + +Use `GEMINI_API_KEY` for Google Gemini (not `GOOGLE_API_KEY`), `HF_TOKEN` for +Hugging Face, and `AI_GATEWAY_API_KEY` for the Vercel AI Gateway. Inject keys +with `-e`/`--env-file` in Docker or a Secret + `envFrom` in Kubernetes; never +bake them into the image. + +A key saved through DSH's credential store (`$DSH_HOME/.credentials.yaml`) wins +over the environment. The +[agents-as-a-service example](../examples/dsh-agents-as-a-service/docs/optional-hardening.md) +documents the optional `dsh-provider-keys-extra` Secret and the pod-refresh +procedure. + ### Ports, volumes and signals | Item | Value | Notes | diff --git a/examples/README.md b/examples/README.md index 0c233bb..066daca 100644 --- a/examples/README.md +++ b/examples/README.md @@ -10,6 +10,10 @@ reference material, not libraries. | [`examples/dsh-agents-as-a-service/`](dsh-agents-as-a-service/) | The full multi-tenant platform pattern: one `SandboxTemplate` with per-user PVCs, a warm pool, one ~10-line `SandboxClaim` per user, suspend-when-idle / wake-on-return, and a browser gateway with signup, `HttpOnly` sessions, an HTTP + WebSocket proxy and two exposure variants (Ingress, or Gateway API + GitHub SSO). | agent-sandbox with extensions; kind for the e2e script. | | [`examples/minimal-single-tenant/`](minimal-single-tenant/) | The smallest useful setup: a single `SandboxTemplate` and one `SandboxClaim`, no gateway — reach DSH Web with a launch-token URL through `kubectl port-forward`. | agent-sandbox with extensions. | +The multi-tenant example also documents opt-in gVisor, `nodeSelector`, +NetworkPolicy and extra provider-key hardening in +[optional hardening](dsh-agents-as-a-service/docs/optional-hardening.md). + ## The example contract (short version) Full rules: [docs/examples.md](../docs/examples.md). `scripts/check-examples.sh` @@ -19,6 +23,8 @@ enforces the layout, and `make check` runs it. examples// +-- README.md # walkthrough: prerequisites, steps, troubleshooting, cleanup +-- manifests/ # NN-*.yaml, applied with plain kubectl +| +-- optional// # optional opt-in manifests... +| +-- optional/README.md # ...required whenever optional/ exists | +-- exposure// # optional per-variant manifests... | +-- exposure/README.md # ...required whenever exposure/ exists +-- scripts/ # optional executable, idempotent, self-cleaning *.sh @@ -44,4 +50,6 @@ kubectl apply -f examples/dsh-agents-as-a-service/manifests/ examples/dsh-agents-as-a-service/scripts/e2e-kind.sh ``` -Each example README has the exact prerequisites, exposure options and teardown. +`kubectl apply -f manifests/` is non-recursive: `optional/` and `exposure/` +subdirectories are applied explicitly. Each example README has the exact +prerequisites, optional hardening, exposure options and teardown. diff --git a/examples/dsh-agents-as-a-service/README.md b/examples/dsh-agents-as-a-service/README.md index 3d9b1dd..0665196 100644 --- a/examples/dsh-agents-as-a-service/README.md +++ b/examples/dsh-agents-as-a-service/README.md @@ -32,10 +32,14 @@ manifests/ +-- 30-claim-alice.yaml # one user +-- 40-claim-rejected.yaml # negative demo: env injection rejected +-- 50-gateway.yaml # Deployment/RBAC/Service for the gateway ++-- optional/ +| +-- README.md # opt-in manifests, applied by path +| +-- network-policy/10-sandbox-networkpolicy.yaml +-- exposure/ +-- README.md # the two exposure variants, when to use which +-- ingress/60-ingress.yaml - +-- gateway-api-sso/70-gateway.yaml ... 74-tls-secret.yaml.example + +-- ingress/61-networkpolicy.yaml + +-- gateway-api-sso/70-gateway.yaml ... 75-networkpolicies.yaml ``` What it demonstrates, feature by feature: @@ -57,8 +61,10 @@ What it demonstrates, feature by feature: `kubernetes.io/arch: amd64` nodeSelector. - **Single port**: DSH Web serves everything on `3080`; the gateway proxies to one upstream instead of splitting `/v1/*` (API) from the dashboard. -- **One shared secret**: only `DEEPSEEK_API_KEY` is injected (via `envFrom`); there - is no dashboard basic-auth, session-cookie secret, or API-server key. +- **Shared provider keys**: `DEEPSEEK_API_KEY` plus any keys in the optional + `dsh-provider-keys-extra` Secret are injected via `envFrom` into every + sandbox. There is no per-tenant key, dashboard basic-auth, session-cookie + secret, or API-server key. - **Launch-token handshake**: DSH gates `/` behind a token it prints only in its logs. The gateway reads it, performs the handshake, and keeps the resulting auth cookie server-side; the browser authenticates to the gateway with its own @@ -93,14 +99,39 @@ What it demonstrates, feature by feature: ## 1. Install the platform resources -First the namespace, then the shared provider key. The key reaches **every** +First the namespace, then the shared provider keys. The keys reach **every** tenant sandbox (a warm-pool constraint: pod env is stamped before any user -exists), so it is **never committed** — generate a fresh value per install: +exists), so they are **never committed** — generate fresh values per install. +`00-prereqs.yaml` already creates the empty `dsh-provider-keys` Secret, so use +an upsert: ```bash kubectl apply -f manifests/00-prereqs.yaml kubectl -n dsh-demo create secret generic dsh-provider-keys \ - --from-literal=DEEPSEEK_API_KEY="sk-..." + --from-literal=DEEPSEEK_API_KEY="sk-..." \ + --dry-run=client -o yaml | kubectl apply -f - +``` + +Optional: additional providers (OpenRouter, Gemini, NVIDIA, OpenAI, Anthropic, +...) go in a second Secret. It is referenced with `optional: true`, so the +sandboxes start without it, and every key becomes an env var: + +```bash +kubectl -n dsh-demo create secret generic dsh-provider-keys-extra \ + --from-literal=OPENROUTER_API_KEY="sk-or-..." \ + --from-literal=GEMINI_API_KEY="..." \ + --from-literal=NVIDIA_API_KEY="nvapi-..." \ + --dry-run=client -o yaml | kubectl apply -f - +``` + +Create both Secrets **before** the template/warm pool. A later Secret change +does not update running pods: delete the warm-pool spares and recreate claimed +pods to pick up new keys. Keys are shared by all tenants; a key saved in DSH's +credential store wins over env. See +[optional hardening](docs/optional-hardening.md) for the full env-name list and +the refresh procedure. + +```bash kubectl apply -f manifests/10-sandboxtemplate.yaml -f manifests/20-sandboxwarmpool.yaml ``` @@ -253,6 +284,12 @@ kubectl -n dsh-demo port-forward svc/dsh-gateway 18080:8080 > apply does not update them: patch each sandbox's > `spec.podTemplate.spec.securityContext.fsGroupChangePolicy` and > `chmod 600` its `.credentials.yaml`, or recreate it. +> +> The optional gVisor/`nodeSelector` fields are pod-spec changes too: existing +> Sandboxes snapshot them, and warm-pool spares are not replaced by a template +> edit. See [optional hardening](docs/optional-hardening.md) for the patch and +> recreation commands. A provider-key Secret change likewise needs pod +> recreation to take effect. ### Browser flow @@ -388,6 +425,13 @@ not a stack. Details, commands and the switch procedure are in the > `kubectl apply -R`: it would apply both variants and create two paths to the > same gateway, which is unsafe once SSO is enabled. +Both variants also ship NetworkPolicies: `ingress/61-networkpolicy.yaml` limits +gateway ingress to Traefik pods, and `gateway-api-sso/75-networkpolicies.yaml` +limits the gateway to oauth2-proxy and oauth2-proxy egress to DNS, public HTTPS +and the gateway. They are part of the variant; see the +[exposure README](manifests/exposure/README.md) and +[optional hardening](docs/optional-hardening.md). + ### Option A — plain Ingress (gateway's own session auth) ```bash @@ -404,6 +448,10 @@ kubectl apply -f manifests/exposure/ingress/ (The base manifest keeps `false` so the plain-HTTP port-forward demo works.) 3. Verify: `curl -sI https:///healthz` returns `200`; sign up in a browser at `https:///`. +4. `manifests/exposure/ingress/61-networkpolicy.yaml` is part of this variant: + it allows only Traefik pods to reach the gateway on `8080`. Adjust its + selector if your ingress controller uses a different pod label or + `hostNetwork`; if kubelet probes fail, see the hardening doc. Keep the router free of buffering/compression middlewares: they break WebSocket upgrades and streaming. @@ -472,13 +520,16 @@ can be cleared at `/oauth2/sign_out`. > **SSO trust boundary — read this before enabling it.** After SSO is on, the > `dsh-gateway` Service must be reachable **only** through oauth2-proxy (it is -> ClusterIP). Anyone who can reach `dsh-gateway:8080` directly can send -> `X-Forwarded-User: ` and be treated as that user. Never apply the -> Ingress variant (or any other route to the Service) while SSO is enabled, and -> do not expose the gateway Service with `kubectl port-forward` on a shared -> machine while `AUTH_PROXY_HEADER` is set. To turn SSO off, remove the variable -> (`kubectl -n dsh-demo set env deploy/dsh-gateway AUTH_PROXY_HEADER-`) and -> delete the SSO variant; the gateway falls back to bearer-token/session auth. +> ClusterIP). The variant's `75-networkpolicies.yaml` restricts pod traffic +> accordingly: the gateway accepts traffic only from oauth2-proxy pods, +> oauth2-proxy only from the Gateway implementation, and oauth2-proxy egress is +> limited to DNS, public HTTPS and the gateway. Anyone who can reach +> `dsh-gateway:8080` directly can still send `X-Forwarded-User: ` and be +> treated as that user, so never add a second path (the Ingress variant, a +> shared-host port-forward, a permissive policy). To turn SSO off, remove the +> variable (`kubectl -n dsh-demo set env deploy/dsh-gateway AUTH_PROXY_HEADER-`) +> and delete the SSO variant; the gateway falls back to bearer-token/session +> auth. ### Switching variants @@ -488,6 +539,25 @@ Delete the variant you had before applying the other one `kubectl -n dsh-demo get ingress,gateway,httproute` that exactly one path to the gateway remains. +## Optional hardening + +The base walkthrough is deliberately portable. Four opt-in layers are documented +in [optional hardening](docs/optional-hardening.md): + +- **gVisor** — uncomment `runtimeClassName: gvisor` before applying, or patch an + existing install. Requires a `gvisor` RuntimeClass and capable nodes. + `kubectl port-forward` does not work under gVisor; the gateway path does. +- **nodeSelector** — uncomment and replace the placeholder node label to pin + sandboxes and the gateway to a dedicated pool; add a toleration for tainted + pools. +- **Sandbox NetworkPolicy** — + `kubectl apply -f manifests/optional/network-policy/` allows only + gateway→sandbox on `3080` and sandbox→DNS/public `443`. +- **Provider keys** — create `dsh-provider-keys-extra` as shown in step 1. + +Apply each independently and re-run the walkthrough. `kubectl apply -f manifests/` +is non-recursive, so the optional manifest is not applied by the base step. + ## Troubleshooting | Symptom | Cause / fix | @@ -511,6 +581,11 @@ gateway remains. | SSO: oauth2-proxy `cookie_secret must be 16, 24, or 32 bytes ... but is 44 bytes` | The cookie secret was generated as standard base64. Use URL-safe unpadded base64 (see `73-oauth2-proxy-secret.yaml.example`) | | SSO: redirect loop (`ERR_TOO_MANY_REDIRECTS`) | oauth2-proxy `--redirect-url` / callback does not match the public FQDN, or TLS is not terminating at the Gateway listener; fix the OAuth App URLs and the manifest placeholders | | SSO: DSH rejects the public host behind the proxy | Add the public hostname to `DSH_TRUSTED_HOSTS` in `manifests/10-sandboxtemplate.yaml`; the gateway forwards the browser's `Host` | +| Pods `Pending` after setting `nodeSelector` | No node matches the label, the pool is full, or a tainted pool needs a toleration; inspect `kubectl describe pod` and the node labels | +| gVisor: `kubectl port-forward` fails or the connection is refused | Port-forward is incompatible with gVisor; reach DSH through the gateway (or a reachable Service/Sandbox Router) | +| Sandbox cannot reach a model API after applying the optional NetworkPolicy | The policy allows DNS and public TCP `443` only; a proxy, custom base URL, port `80` or non-443 endpoint needs an explicit egress rule | +| Gateway cannot reach a sandbox after applying an exposure NetworkPolicy | Check that the CNI enforces NetworkPolicy and that the Traefik/oauth2-proxy pod labels match; kubelet probes may also need the node CIDR | +| New provider key not visible in a sandbox | Pod env is stamped at creation: delete the warm-pool spares and recreate claimed pods. A key saved in DSH's credential store also wins over env | ## Limitations @@ -522,13 +597,22 @@ Deliberately **not** production code: - **Log scraping.** The launch token is recovered from pod logs. Log rotation, a restarted pod, or a changed DSH log format can break the handshake until the next `401` refresh. -- **Shared provider key.** `DEEPSEEK_API_KEY` is shared by all tenants via the - template. Real multi-tenancy needs per-tenant credentials or an egress proxy. +- **Shared provider keys.** `DEEPSEEK_API_KEY` and any keys in the optional + `dsh-provider-keys-extra` Secret are shared by all tenants via the template. + Tenants can override a key through DSH's stored credential file on their PVC, + but the platform does not manage that. Real multi-tenancy needs per-tenant + credentials or an egress proxy. - **No TLS in the gateway.** Terminate TLS at an Ingress or Gateway and set `COOKIE_SECURE=true`. -- **SSO trust is hop-based.** `AUTH_PROXY_HEADER` trusts whoever sets the header; - the only guarantee is that the gateway Service is not directly reachable. - There is no signature, mTLS, or NetworkPolicy enforcing that in this example. +- **SSO trust is hop-based.** `AUTH_PROXY_HEADER` trusts whoever sets the header. + The SSO variant ships `75-networkpolicies.yaml`, which restricts the gateway + to oauth2-proxy and oauth2-proxy egress to the gateway/DNS/public HTTPS; a + custom exposure without that policy has no signature, mTLS or network + enforcement. +- **Isolation is opt-in.** gVisor, the sandbox NetworkPolicy and the exposure + policies are opt-in; without them, sandboxes share the host kernel and can + reach the cluster network. The base example is a demo, not a hardened + multi-tenant platform. - **OAuth org is the only authorization.** Any GitHub user in the configured organization gets an auto-provisioned sandbox; there is no per-user allow-list, quota, or admin API. @@ -564,3 +648,5 @@ delete the Secrets with the namespace and revoke the OAuth App in GitHub. - [Architecture](docs/architecture.md) — container/sequence diagrams, ADRs, security and RBAC, failure modes, evolution path. - [Exposure README](manifests/exposure/README.md) — the two variants in detail. +- [Optional hardening](docs/optional-hardening.md) — gVisor, nodeSelector, + NetworkPolicy and extra provider keys, with verification and rollback. diff --git a/examples/dsh-agents-as-a-service/docs/architecture.md b/examples/dsh-agents-as-a-service/docs/architecture.md index 1da55ce..755c38a 100644 --- a/examples/dsh-agents-as-a-service/docs/architecture.md +++ b/examples/dsh-agents-as-a-service/docs/architecture.md @@ -67,6 +67,13 @@ as that user. The gateway ignores the header unless `AUTH_PROXY_HEADER` is set. [exposure README](../manifests/exposure/README.md) and the [gateway README](../gateway/README.md#sso-opt-in). +Each variant also ships a NetworkPolicy: `ingress/61-networkpolicy.yaml` limits gateway ingress +to Traefik pods, and `gateway-api-sso/75-networkpolicies.yaml` limits the gateway to +oauth2-proxy, oauth2-proxy ingress to the Gateway implementation, and oauth2-proxy egress to +DNS/public HTTPS/the gateway. The SSO policy turns the trust boundary into pod-traffic +enforcement; the caveats (Traefik label, `hostNetwork`, kubelet probes) are in +[optional hardening](optional-hardening.md). + ## Flows ### Signup (browser) @@ -290,8 +297,14 @@ Notes and limits: - **TLS is external.** The gateway serves plain HTTP; terminate TLS at the Ingress and set `COOKIE_SECURE=true`. Without TLS, session cookies are replayable on the network. - **Tenant isolation** is pod + PVC separation; all tenants share one namespace, one DSH binary, - and one `DEEPSEEK_API_KEY`. Add NetworkPolicies and per-tenant credentials before calling it - multi-tenant-grade. + and the shared provider keys (`dsh-provider-keys` plus the optional + `dsh-provider-keys-extra`). `manifests/optional/network-policy/` adds an opt-in sandbox + NetworkPolicy, and gVisor/`nodeSelector` are opt-in PodSpec fields. Add per-tenant + credentials before calling it multi-tenant-grade. +- **Optional hardening is opt-in.** gVisor, node selection, the sandbox/exposure NetworkPolicies + and extra provider keys are documented in [optional hardening](optional-hardening.md). The + base manifests stay portable (no gVisor RuntimeClass, no active node labels, no + NetworkPolicy and no extra Secret required). - **No gateway rate limiting.** Signup can create claims until the cluster or warm pool is exhausted; put limits at the ingress for real deployments. @@ -306,6 +319,10 @@ Notes and limits: | DSH auth cookie expired mid-session | Proxy `401` once, then recovers | Gateway refreshes the handshake once | | Upstream pod name changed after resume | Cached cookie keyed by old pod UID is useless | Cache is invalidated on suspend; next request re-handshakes | | Two gateway replicas | Double suspend/wake races | Keep `replicas: 1` until state is externalized | +| Optional NetworkPolicy blocks DNS/probes | Sandbox 502/503, pod restarts, failed `getent` | Check the CNI, DNS selector/NodeLocal address and kubelet-probe traffic; see the hardening doc | +| gVisor + `kubectl port-forward` | Forward refuses or returns nothing | Port-forward is incompatible with gVisor; use the gateway or a reachable Service | +| Provider key added but missing in the pod | Requests still fail with `MISSING_CREDENTIAL` | Pod env is stamped at creation; recreate spares/pods. A stored credential wins over env | +| `nodeSelector` matches no node | Pod `Pending`, `0/N nodes are available` | Fix the label, pool capacity or taint/toleration; RuntimeClass scheduling may also constrain it | ## Upstream feature request: remove log scraping diff --git a/examples/dsh-agents-as-a-service/docs/optional-hardening.md b/examples/dsh-agents-as-a-service/docs/optional-hardening.md new file mode 100644 index 0000000..d88501c --- /dev/null +++ b/examples/dsh-agents-as-a-service/docs/optional-hardening.md @@ -0,0 +1,513 @@ +# Optional hardening: gVisor, nodeSelector, NetworkPolicy and provider keys + +The base example is deliberately portable: it runs on kind and GKE without a +gVisor runtime, a tainted node pool or a NetworkPolicy-capable CNI, and it +boots with only the required `dsh-provider-keys` Secret. This document covers +four **opt-in** layers: + +| Layer | Where it lives | What it changes | +| --- | --- | --- | +| gVisor | commented `runtimeClassName` in the SandboxTemplates and Deployments | puts DSH and/or the gateway in a userspace kernel | +| nodeSelector | commented `nodeSelector` in the same files | pins pods to a chosen node pool | +| NetworkPolicy | `manifests/optional/network-policy/`, `manifests/exposure/*/` | limits sandbox, gateway and proxy connectivity | +| Provider keys | optional `dsh-provider-keys-extra` Secret | injects extra provider API keys into every sandbox | + +Each layer is independent. Enable only what the cluster supports and test the +walkthrough after each change. + +## 1. gVisor + +### What it changes + +The DSH sandbox and the gateway (and optionally oauth2-proxy) run under the +`gvisor` RuntimeClass. `agent-sandbox` passes `runtimeClassName` straight to the +PodSpec; no CRD change is involved. + +### Prerequisites + +- A `RuntimeClass` named `gvisor`: + + ```bash + kubectl get runtimeclass gvisor -o yaml + ``` + +- Nodes capable of running it (`runsc` installed and registered with the + container runtime). How you get there is cluster-specific: + - GKE Sandbox: create a node pool with `--sandbox=type=gvisor`; GKE labels + the nodes `sandbox.gke.io/runtime: gvisor`, taints them + `sandbox.gke.io/runtime=gvisor:NoSchedule`, and its `gvisor` RuntimeClass + applies the matching affinity and toleration automatically. + - Self-managed: install gVisor, create the `gvisor` RuntimeClass with + `handler: runsc`, and label/taint a dedicated node pool yourself. +- A scheduling plan: a gVisor RuntimeClass without capable nodes makes pods + fail to start. Use the RuntimeClass `scheduling` fields or the optional + `nodeSelector`/toleration described in section 2. + +### Enable it + +**Fresh install:** uncomment `runtimeClassName: gvisor` before applying: + +- [`manifests/10-sandboxtemplate.yaml`](../manifests/10-sandboxtemplate.yaml) +- [`manifests/50-gateway.yaml`](../manifests/50-gateway.yaml) +- [`manifests/exposure/gateway-api-sso/72-oauth2-proxy.yaml`](../manifests/exposure/gateway-api-sso/72-oauth2-proxy.yaml) + (only with the SSO variant) +- [minimal sandbox-template.yaml](../../minimal-single-tenant/manifests/sandbox-template.yaml) + +**Existing install:** a template edit does not update existing Sandboxes, and +warm-pool spares use `OnReplenish`, so patch explicitly. Plan a maintenance +window: claimed users are disconnected when their pod is recreated. + +```bash +# 1. The blueprint for future sandboxes. +kubectl -n dsh-demo patch sandboxtemplate dsh-web --type merge \ + -p '{"spec":{"podTemplate":{"spec":{"runtimeClassName":"gvisor"}}}}' + +# 2. Warm-pool spares: delete them; the pool replenishes from the template. +kubectl -n dsh-demo delete sandboxes -l agents.x-k8s.io/warm-pool-sandbox + +# 3. Claimed sandboxes: snapshot their pod template, then suspend/resume. +for sb in $(kubectl -n dsh-demo get sandboxes \ + -l '!agents.x-k8s.io/warm-pool-sandbox' -o jsonpath='{.items[*].metadata.name}'); do + kubectl -n dsh-demo patch sandbox "$sb" --type merge \ + -p '{"spec":{"podTemplate":{"spec":{"runtimeClassName":"gvisor"}}}}' + kubectl -n dsh-demo patch sandbox "$sb" --type merge \ + -p '{"spec":{"operatingMode":"Suspended"}}' + kubectl -n dsh-demo patch sandbox "$sb" --type merge \ + -p '{"spec":{"operatingMode":"Running"}}' +done + +# 4. The gateway (and oauth2-proxy when the SSO variant is installed). +kubectl -n dsh-demo patch deploy dsh-gateway --type merge \ + -p '{"spec":{"template":{"spec":{"runtimeClassName":"gvisor"}}}}' +kubectl -n dsh-demo rollout status deploy/dsh-gateway --timeout=180s +# kubectl -n dsh-demo patch deploy oauth2-proxy --type merge \ +# -p '{"spec":{"template":{"spec":{"runtimeClassName":"gvisor"}}}}' +``` + +If the pool's nodes are tainted and the RuntimeClass does not already add a +toleration, add the matching `tolerations` block (section 2) in the same patch +or in the manifests. + +### Verify + +```bash +SB=$(kubectl -n dsh-demo get sandboxclaim dsh-alice -o jsonpath='{.status.sandbox.name}') +POD=$(kubectl -n dsh-demo get sandbox "$SB" \ + -o jsonpath='{.metadata.annotations.agents\.x-k8s\.io/pod-name}'); POD=${POD:-$SB} + +kubectl -n dsh-demo get pod "$POD" -o jsonpath='{.spec.runtimeClassName}{"\n"}' +# gvisor +kubectl -n dsh-demo exec "$POD" -- uname -r +kubectl -n dsh-demo get node "$(kubectl -n dsh-demo get pod "$POD" -o jsonpath='{.spec.nodeName}')" \ + -o jsonpath='{.status.nodeInfo.kernelVersion}{"\n"}' +``` + +The pod's `runtimeClassName` is authoritative. The kernel release reported by +`uname -r` inside a gVisor pod is the sandbox kernel, not the host kernel; on +GKE it looks like `4.19.0-gvisor`. + +### Caveats + +- **`kubectl port-forward` does not work with gVisor.** The multi-tenant + gateway path (HTTP/WebSocket to the pod) works; the minimal example's + port-forward quickstart does not. Use the gateway example or a reachable + Service (NodePort/LoadBalancer/Sandbox Router) instead. +- gVisor has performance overhead and does not implement every syscall. Test + DSH and the agent tools you rely on (`git`, `gh`, `uv`, language toolchains). +- Node scheduling is the hard part. A `gvisor` RuntimeClass on a cluster with + no capable nodes leaves pods unschedulable or failing at container start. +- Existing Sandboxes are snapshots. The loop above is required for every + claimed sandbox; deleting the `SandboxClaim` would also delete its PVC and + workspace data, so do not use claim re-creation as a shortcut. +- A resumed sandbox starts cold; the startup probe gives it up to 120s. + +### Roll back + +```bash +kubectl -n dsh-demo patch sandboxtemplate dsh-web --type merge \ + -p '{"spec":{"podTemplate":{"spec":{"runtimeClassName":null}}}}' +for sb in $(kubectl -n dsh-demo get sandboxes -o jsonpath='{.items[*].metadata.name}'); do + kubectl -n dsh-demo patch sandbox "$sb" --type merge \ + -p '{"spec":{"podTemplate":{"spec":{"runtimeClassName":null}}}}' +done +kubectl -n dsh-demo delete sandboxes -l agents.x-k8s.io/warm-pool-sandbox +kubectl -n dsh-demo patch deploy dsh-gateway --type merge \ + -p '{"spec":{"template":{"spec":{"runtimeClassName":null}}}}' +# kubectl -n dsh-demo patch deploy oauth2-proxy --type merge \ +# -p '{"spec":{"template":{"spec":{"runtimeClassName":null}}}}' +``` + +Recreate the pods (delete spares, suspend/resume claimed sandboxes, let the +Deployment roll out) so the cleared field takes effect. + +## 2. nodeSelector + +### What it changes + +Pods get a `nodeSelector` so they land only on nodes carrying the chosen label. +The same field is available in both SandboxTemplates and both Deployments; +the shipped value is **commented out** because active cluster-specific +scheduling does not belong in the repository. + +### Choose a label + +```bash +kubectl get nodes --show-labels +``` + +- **GKE Sandbox:** the nodes carry `sandbox.gke.io/runtime: gvisor` and are + tainted `sandbox.gke.io/runtime=gvisor:NoSchedule`. With + `runtimeClassName: gvisor`, GKE already schedules the pod onto those nodes; + add a `nodeSelector` only when you need a specific sandbox pool or + ComputeClass (`cloud.google.com/compute-class: `). +- **Self-managed:** label the pool you want, for example + `dsh.example.com/node-pool: gvisor`, and use that label. +- **General pinning:** any stable node-pool label works. Keep the real key and + value in a local patch; the manifests use `dsh.example.com/node-pool` as an + obvious placeholder. + +### Enable it + +Uncomment and replace the placeholder before a fresh apply: + +```yaml + nodeSelector: + dsh.example.com/node-pool: gvisor +``` + +For an existing install, patch the template, each claimed Sandbox and the +Deployments, then recreate pods exactly as in section 1: + +```bash +kubectl -n dsh-demo patch sandboxtemplate dsh-web --type merge \ + -p '{"spec":{"podTemplate":{"spec":{"nodeSelector":{"dsh.example.com/node-pool":"gvisor"}}}}}' +# ... same warm-pool delete / claimed-sandbox suspend-resume / Deployment +# patch and rollout as in section 1, with nodeSelector instead of +# runtimeClassName. +``` + +If the selected nodes are tainted, add a matching toleration. This is the +companion block; do **not** uncomment it unless the pool is actually tainted: + +```yaml + tolerations: + - key: dsh.example.com/node-pool + operator: Equal + value: gvisor + effect: NoSchedule +``` + +GKE Sandbox's taint is `sandbox.gke.io/runtime=gvisor:NoSchedule`; the GKE +`gvisor` RuntimeClass adds that toleration automatically, so a manual copy is +only needed when you schedule without the RuntimeClass or with a different +pool. + +### Verify and troubleshoot + +```bash +kubectl -n dsh-demo get pod "$POD" -o wide +kubectl -n dsh-demo get pod "$POD" -o jsonpath='{.spec.nodeSelector}{"\n"}' +kubectl -n dsh-demo describe pod "$POD" | sed -n '/Events:/,$p' +``` + +- `Pending` with `0/N nodes are available`: the label is wrong, the pool is + full, or the pod needs a toleration for a taint. +- A pod-level `nodeSelector` is merged with the RuntimeClass + `scheduling.nodeSelector`. If both are set and conflict, no node matches. +- Labels changed after scheduling do not move running pods. Patch/recreate. +- A `nodeSelector` that excludes every gVisor node is a valid way to keep the + gateway on a normal pool while the sandboxes run under gVisor. + +### Roll back + +Remove the field from the manifests, or patch it to `null`, then recreate the +pods: + +```bash +kubectl -n dsh-demo patch sandboxtemplate dsh-web --type merge \ + -p '{"spec":{"podTemplate":{"spec":{"nodeSelector":null}}}}' +# ... same per-Sandbox and Deployment loop as section 1. +``` + +## 3. NetworkPolicy + +The base example does **not** manage NetworkPolicy: the SandboxTemplate is +`networkPolicyManagement: Unmanaged`, and the gateway has no policy. That keeps +the walkthrough portable and independent of the sandbox-router; the opt-in +policies below are standard Kubernetes `NetworkPolicy` objects. + +### Optional sandbox policy + +[`manifests/optional/network-policy/10-sandbox-networkpolicy.yaml`](../manifests/optional/network-policy/10-sandbox-networkpolicy.yaml) +selects every Sandbox pod through the agent-sandbox tracking label +`agents.x-k8s.io/sandbox-name-hash`. It applies to existing warm-pool and +claimed sandboxes without recreating them. + +```bash +kubectl apply -f manifests/optional/network-policy/ +kubectl delete -f manifests/optional/network-policy/ # rollback +``` + +What it allows: + +| Direction | Allowed | Blocked | +| --- | --- | --- | +| Ingress | gateway pods (`app: dsh-gateway`) on TCP `3080` | everything else, including other sandboxes and cluster Services | +| Egress | UDP/TCP `53` to kube-dns and NodeLocal DNSCache; TCP `443` to public IPs | RFC1918, CGNAT, link-local/metadata, cluster Services/API, plain HTTP, other ports | + +Verify from inside the pods (NetworkPolicy does not affect `kubectl exec` +itself, only the traffic the commands generate): + +```bash +SB=$(kubectl -n dsh-demo get sandboxclaim dsh-alice -o jsonpath='{.status.sandbox.name}') +POD=$(kubectl -n dsh-demo get sandbox "$SB" \ + -o jsonpath='{.metadata.annotations.agents\.x-k8s\.io/pod-name}'); POD=${POD:-$SB} + +# Allowed: gateway -> sandbox. +kubectl -n dsh-demo exec deploy/dsh-gateway -- python -c \ + "import urllib.request; print(urllib.request.urlopen('http://$POD:3080/favicon.svg', timeout=5).status)" + +# Allowed: DNS and public HTTPS from the sandbox. +kubectl -n dsh-demo exec "$POD" -- getent hosts api.deepseek.com +kubectl -n dsh-demo exec "$POD" -- curl -fsS -o /dev/null -w '%{http_code}\n' https://example.com + +# Denied: another sandbox cannot reach this one on 3080 (expect a timeout). +OTHER_SB=$(kubectl -n dsh-demo get sandboxclaim dsh-bob -o jsonpath='{.status.sandbox.name}') +OTHER_POD=$(kubectl -n dsh-demo get sandbox "$OTHER_SB" \ + -o jsonpath='{.metadata.annotations.agents\.x-k8s\.io/pod-name}'); OTHER_POD=${OTHER_POD:-$OTHER_SB} +kubectl -n dsh-demo exec "$OTHER_POD" -- curl --max-time 3 -sS "http://$POD:3080/favicon.svg" +``` + +Caveats: + +- **The CNI must enforce NetworkPolicy.** If it does not, the file applies but + has no effect. +- DNS: the default assumes CoreDNS/kube-dns pods labelled `k8s-app: kube-dns` + and covers NodeLocal DNSCache at `169.254.20.10`. A different DNS layout + needs a selector/`ipBlock` edit. +- **Kubelet probes** from the node are CNI-dependent. If a pod starts failing + its probes, allow the node CIDR on the probe port or remove the ingress + policy. +- The policy blocks cluster Services and the Kubernetes API from the sandbox. + DSH does not need them, but an agent tool that does needs an explicit rule. +- It allows any public host on TCP `443`, not an FQDN allowlist. It is network + isolation, not data-loss prevention. +- Custom provider base URLs, an HTTP proxy, port `80`, SSH `git` (port `22`) or + a non-443 provider endpoint need explicit egress additions. +- Do not combine this standalone policy with `networkPolicyManagement: Managed` + unless you migrate the rules into `spec.networkPolicy`. NetworkPolicies are + additive; the controller's Managed secure default would also be present. +- `kubectl port-forward` still works for debugging (the connection is local to + the pod), except under gVisor. + +### Exposure-variant policies + +Each exposure variant ships its own policy. They are part of the variant, not a +separate opt-in step: + +- [`manifests/exposure/ingress/61-networkpolicy.yaml`](../manifests/exposure/ingress/61-networkpolicy.yaml) + allows only Traefik pods (`app.kubernetes.io/name: traefik`) to reach the + gateway on `8080`. +- [`manifests/exposure/gateway-api-sso/75-networkpolicies.yaml`](../manifests/exposure/gateway-api-sso/75-networkpolicies.yaml) + allows only oauth2-proxy to reach the gateway, only the Gateway + implementation (Traefik) to reach oauth2-proxy, and limits oauth2-proxy + egress to DNS, public HTTPS (GitHub) and the gateway. + +The SSO policy is what turns the trust boundary in the +[exposure README](../manifests/exposure/README.md) into network enforcement. +The `dsh-gateway` Service must still stay ClusterIP; do not apply both exposure +variants. + +Verify the negative case from a sandbox pod (it is not the proxy): + +```bash +kubectl -n dsh-demo exec "$POD" -- curl --max-time 3 -sS http://dsh-gateway:8080/healthz +# expected: timeout/connection failure +``` + +Then verify the real browser path through the Ingress or Gateway, and confirm +the gateway/proxy pods are `Ready`. If Traefik uses `hostNetwork` or a +different pod label, replace the `podSelector` with the correct selector or an +`ipBlock`. As with the sandbox policy, kubelet probes can be affected on some +CNIs. + +### Gateway egress (documented, not shipped) + +The gateway needs DNS, the Kubernetes API, and TCP `3080` to sandbox pods. The +API server endpoint is cluster-specific (Service CIDR or node IP/port), so an +egress policy is **not** shipped. If your cluster has a stable API-server CIDR +and port, add an egress policy like this (replace the placeholders; verify the +port, commonly `443` or `6443`): + +```yaml +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: dsh-gateway-egress + namespace: dsh-demo +spec: + podSelector: + matchLabels: + app: dsh-gateway + policyTypes: + - Egress + egress: + - to: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: kube-system + podSelector: + matchLabels: + k8s-app: kube-dns + ports: + - protocol: UDP + port: 53 + - protocol: TCP + port: 53 + - to: + - ipBlock: + cidr: 10.96.0.1/32 # placeholder: API-server Service/cluster endpoint + ports: + - protocol: TCP + port: 443 + - to: + - podSelector: + matchExpressions: + - key: agents.x-k8s.io/sandbox-name-hash + operator: Exists + ports: + - protocol: TCP + port: 3080 +``` + +Cilium users can express the API server as `toEntities: kube-apiserver` (or +`toServices`) instead of a CIDR. Do not apply the sample blindly: a wrong CIDR +or port silently breaks signup/proxy. + +## 4. Optional provider keys + +### How injection works + +The SandboxTemplate injects two Secrets via `envFrom`: + +- `dsh-provider-keys` — required by the walkthrough; holds + `DEEPSEEK_API_KEY`. +- `dsh-provider-keys-extra` — optional; referenced with `optional: true`, so a + missing Secret does not block sandbox startup. Every key in it becomes an + environment variable in every sandbox. + +Create the extra Secret before the warm pool (pod env is stamped at creation): + +```bash +kubectl -n dsh-demo create secret generic dsh-provider-keys-extra \ + --from-literal=OPENROUTER_API_KEY="sk-or-..." \ + --from-literal=GEMINI_API_KEY="..." \ + --from-literal=NVIDIA_API_KEY="nvapi-..." \ + --dry-run=client -o yaml | kubectl apply -f - +``` + +Use the exact env-var names (uppercase, underscores) — invalid Secret keys are +skipped by the kubelet with an event. Do not duplicate a key across the two +Secrets; when the same key exists twice, the later `envFrom` entry wins. + +### Supported names (curated) + +These are the names DSH v0.2.1-rc.3 resolves through its packaged provider +catalog; the catalog has more entries, these are the common ones: + +| Provider | Environment variable | +| --- | --- | +| DeepSeek | `DEEPSEEK_API_KEY` | +| OpenRouter | `OPENROUTER_API_KEY` | +| Google Gemini | `GEMINI_API_KEY` | +| NVIDIA | `NVIDIA_API_KEY` | +| OpenAI | `OPENAI_API_KEY` | +| Anthropic | `ANTHROPIC_API_KEY` | +| Groq | `GROQ_API_KEY` | +| xAI | `XAI_API_KEY` | +| Mistral | `MISTRAL_API_KEY` | +| Together | `TOGETHER_API_KEY` | +| Cerebras | `CEREBRAS_API_KEY` | +| Vercel AI Gateway | `AI_GATEWAY_API_KEY` | +| Hugging Face | `HF_TOKEN` | + +Aliases to avoid: `GOOGLE_API_KEY`, `HUGGINGFACE_API_KEY` and +`VERCEL_AI_GATEWAY_API_KEY`. The packaged catalog looks for `GEMINI_API_KEY`, +`HF_TOKEN` and `AI_GATEWAY_API_KEY` instead. + +### Picking up a key change + +A Secret update does not change running pods. The base key and extra keys are +both snapshotted at container start: + +```bash +# Warm-pool spares: delete them so the pool recreates them from the template. +kubectl -n dsh-demo delete sandboxes -l agents.x-k8s.io/warm-pool-sandbox + +# Running claimed sandboxes: delete the pod (the Sandbox controller recreates +# it) or suspend/resume the sandbox. +POD=... # resolve it as in section 3 +kubectl -n dsh-demo delete pod "$POD" +``` + +New claims get the new values automatically. Confirm names without printing +secret values: + +```bash +kubectl -n dsh-demo exec "$POD" -- sh -c ' + for k in DEEPSEEK_API_KEY OPENROUTER_API_KEY GEMINI_API_KEY NVIDIA_API_KEY; do + if [ -n "$(printenv "$k" || true)" ]; then echo "$k=set"; else echo "$k=unset"; fi + done' +``` + +### Precedence and shared-key limits + +- DSH's stored credential file (`$DSH_HOME/.credentials.yaml`, mode `0600`) + **wins over the environment**. If a tenant saved a key through the DSH UI, + changing the Secret will not override it. +- Both Secrets are shared by **every** tenant, and any agent can read its own + environment. This is a warm-pool constraint, not tenant isolation. A tenant + can override its own key through the DSH credential store on its PVC, but the + platform does not manage that; real multi-tenancy needs per-tenant + credentials or an egress proxy. +- The optional Secret is created imperatively; the repository never contains a + real key. Keep it out of Git and out of shell history where possible. + +## 5. Verification summary + +Required locally for any change to this example: + +```bash +make check # versions, example layout, manifest YAML shape +yamllint . +make test # gateway unit tests (gateway code is untouched) +``` + +CI additionally parses every manifest and runs `kubectl apply +--dry-run=server` against the real agent-sandbox CRDs. The weekly kind e2e +runs the base walkthrough; it also proves the optional extra Secret may be +absent. + +Manual checks, on a cluster with the relevant CNI/runtime: + +- gVisor: pod `runtimeClassName`, `uname -r` differs from the node kernel, + gateway proxy works, port-forward limitation acknowledged. +- nodeSelector: pods land on the intended nodes; no `Pending` events. +- Sandbox policy: the allow/deny matrix above. +- Exposure policies: browser path works; a non-proxy pod cannot reach the + gateway; probes stay healthy. +- Provider keys: set/unset output (never the values) and the refresh procedure. + +## 6. Rollback summary + +| Layer | Rollback | +| --- | --- | +| Sandbox policy | `kubectl delete -f manifests/optional/network-policy/` | +| Exposure policy | delete the exposure variant directory | +| gVisor | patch `runtimeClassName: null`, recreate pods | +| nodeSelector | patch `nodeSelector: null`, recreate pods | +| Provider keys | delete the extra Secret or a key, recreate pods | + +No PVC or workspace data is touched by any rollback. Deleting a `SandboxClaim` +is different: it cascades to the Sandbox **and** its PVC, so do not use claim +deletion to apply a pod-spec change. diff --git a/examples/dsh-agents-as-a-service/gateway/README.md b/examples/dsh-agents-as-a-service/gateway/README.md index da51243..2e233f8 100644 --- a/examples/dsh-agents-as-a-service/gateway/README.md +++ b/examples/dsh-agents-as-a-service/gateway/README.md @@ -110,7 +110,9 @@ user (creates their claim idempotently, mints a rotated session token) and redirects to `/u//`, and `POST /recreate` becomes available for a clean restart. The base manifests leave it unset on purpose; the [SSO exposure variant](../manifests/exposure/gateway-api-sso/) documents how to -enable it on the Deployment (`kubectl set env`). +enable it on the Deployment (`kubectl set env`). That variant also ships +`75-networkpolicies.yaml`, which restricts the gateway to oauth2-proxy and +oauth2-proxy's egress to the gateway, DNS and public HTTPS. > **Trust boundary — this is the security-critical part.** A trusted identity > header is only safe when **nothing else can reach the gateway**. The gateway @@ -118,6 +120,8 @@ enable it on the Deployment (`kubectl set env`). > authenticating proxy; if a client can reach `dsh-gateway:8080` directly (a > second Ingress/Gateway/Route, a port-forward on a shared host, a broad > NetworkPolicy), it can send `X-Forwarded-User` itself and impersonate any user. +> The variant's `75-networkpolicies.yaml` restricts pod traffic to the proxy; a +> custom exposure without an equivalent policy has no network enforcement. > Treat `AUTH_PROXY_HEADER` as "the proxy already authenticated this request", > not as an authentication mechanism of its own. Disable SSO by removing the > variable: the header is then ignored and bearer-token/session auth applies @@ -232,5 +236,6 @@ uvicorn's access log so bearer tokens in URLs are not logged. * **SSO trusts a header.** With `AUTH_PROXY_HEADER` set, any client that can reach the gateway Service directly can spoof the identity header. The gateway cannot enforce the trust boundary itself: keep the Service ClusterIP and - reachable only through the authenticating proxy. There is no mTLS or signed - identity assertion. + reachable only through the authenticating proxy. The SSO exposure variant + ships a NetworkPolicy that restricts pod traffic to the proxy; without it + there is no mTLS or signed identity assertion. diff --git a/examples/dsh-agents-as-a-service/manifests/00-prereqs.yaml b/examples/dsh-agents-as-a-service/manifests/00-prereqs.yaml index 141d6ca..5989095 100644 --- a/examples/dsh-agents-as-a-service/manifests/00-prereqs.yaml +++ b/examples/dsh-agents-as-a-service/manifests/00-prereqs.yaml @@ -1,14 +1,31 @@ -# DSH "Agents as a Service" — prerequisites: namespace + shared provider key. +# DSH "Agents as a Service" — prerequisites: namespace + shared provider keys. # # The dsh-provider-keys Secret is deliberately NOT committed with a usable # value. DEEPSEEK_API_KEY reaches every tenant sandbox (injected via envFrom # on the SandboxTemplate, stamped before any user exists — a warm-pool # constraint), so a real key committed here would be shared by every install # of this example. Generate a fresh value per install (README step 1; -# scripts/e2e-kind.sh uses a throwaway): +# scripts/e2e-kind.sh uses a throwaway). Use an upsert: this manifest already +# creates the empty Secret, so a plain `kubectl create` would fail: # # kubectl -n dsh-demo create secret generic dsh-provider-keys \ -# --from-literal=DEEPSEEK_API_KEY="sk-..." +# --from-literal=DEEPSEEK_API_KEY="sk-..." \ +# --dry-run=client -o yaml | kubectl apply -f - +# +# Optional: additional providers (OpenRouter, Gemini, NVIDIA, OpenAI, ...) go +# in a second, optional Secret. The SandboxTemplate references it with +# `optional: true`, so sandboxes start even when it does not exist, and every +# key in it becomes an environment variable. Create it BEFORE the warm pool, +# because pod env is stamped at pod creation: +# +# kubectl -n dsh-demo create secret generic dsh-provider-keys-extra \ +# --from-literal=OPENROUTER_API_KEY="sk-or-..." \ +# --from-literal=GEMINI_API_KEY="..." \ +# --from-literal=NVIDIA_API_KEY="nvapi-..." \ +# --dry-run=client -o yaml | kubectl apply -f - +# +# Supported env names, precedence, shared-key caveats and the pod-refresh +# procedure are in docs/optional-hardening.md. # # All the infrastructure in this example works without a real key; only # actual model calls need one. diff --git a/examples/dsh-agents-as-a-service/manifests/10-sandboxtemplate.yaml b/examples/dsh-agents-as-a-service/manifests/10-sandboxtemplate.yaml index 3151415..75a6294 100644 --- a/examples/dsh-agents-as-a-service/manifests/10-sandboxtemplate.yaml +++ b/examples/dsh-agents-as-a-service/manifests/10-sandboxtemplate.yaml @@ -1,6 +1,6 @@ # The reusable DSH Web blueprint. Warm pools stamp pod env BEFORE any user -# exists, so per-user secrets cannot live in pod env; the shared provider key -# is injected via envFrom, and claims that would bypass the warm pool are +# exists, so per-user secrets cannot live in pod env; the shared provider keys +# are injected via envFrom, and claims that would bypass the warm pool are # rejected outright (see 40-claim-rejected.yaml). # # A stripped-down single-tenant variant of this template (no warm pool, no @@ -21,16 +21,31 @@ spec: # The gateway routes each user to their DSH Web instance by this name. service: true - # The example gateway does not manage NetworkPolicy, so it is left - # unmanaged here. A production platform would use - # networkPolicyManagement: Managed and restrict ingress to its gateway on - # :3080 — with the shared provider key, that NetworkPolicy is the per-user - # isolation boundary. + # The example leaves NetworkPolicy to an optional, opt-in manifest + # (manifests/optional/network-policy/): the base walkthrough should not + # require a policy-capable CNI, and the controller's Managed secure default + # only allows the sandbox-router, which this example does not deploy. If you + # switch to Managed, define spec.networkPolicy explicitly or the gateway will + # be blocked. See docs/optional-hardening.md. networkPolicyManagement: Unmanaged podTemplate: spec: - # DSH Web is multi-arch (amd64 + arm64); no arch nodeSelector needed. + # DSH Web is multi-arch (amd64 + arm64); no arch nodeSelector is needed. + # + # OPTIONAL: run the sandbox under gVisor. Requires a RuntimeClass named + # "gvisor" and nodes that can run it; gVisor breaks `kubectl port-forward` + # into the sandbox (use the gateway). Existing Sandboxes snapshot their + # pod template, so enabling it later requires patching/recreating them. + # See docs/optional-hardening.md. + # runtimeClassName: gvisor + # + # OPTIONAL: pin sandboxes to a dedicated node pool (for example a gVisor + # pool). Replace this placeholder with a label your nodes actually carry; + # add a matching toleration if the pool is tainted. Active + # cluster-specific scheduling belongs in a local patch, not here. + # nodeSelector: + # dsh.example.com/node-pool: dsh restartPolicy: "OnFailure" securityContext: runAsUser: 1000 # the 'node' user in the DSH image @@ -57,6 +72,13 @@ spec: envFrom: - secretRef: name: dsh-provider-keys + # Optional extra provider keys (OpenRouter, Gemini, NVIDIA, ...). + # The Secret may be absent; every key in it becomes an env var. + # Create it before the warm pool, because pod env is stamped at + # pod creation. See docs/optional-hardening.md. + - secretRef: + name: dsh-provider-keys-extra + optional: true env: # Empty => DSH derives links from the request Host, which the # gateway forwards (plus X-Forwarded-*). Point this at the diff --git a/examples/dsh-agents-as-a-service/manifests/50-gateway.yaml b/examples/dsh-agents-as-a-service/manifests/50-gateway.yaml index b492eae..db21600 100644 --- a/examples/dsh-agents-as-a-service/manifests/50-gateway.yaml +++ b/examples/dsh-agents-as-a-service/manifests/50-gateway.yaml @@ -84,6 +84,15 @@ spec: app: dsh-gateway spec: serviceAccountName: dsh-gateway + # OPTIONAL: run the gateway under gVisor. Requires a RuntimeClass named + # "gvisor" and nodes that can run it. See docs/optional-hardening.md. + # runtimeClassName: gvisor + # + # OPTIONAL: pin the gateway to a dedicated node pool. Replace the + # placeholder with a label your nodes carry; add a toleration for a + # tainted pool. Keep active cluster-specific scheduling in a local patch. + # nodeSelector: + # dsh.example.com/node-pool: dsh securityContext: runAsNonRoot: true runAsUser: 10000 # matches USER in gateway/Dockerfile diff --git a/examples/dsh-agents-as-a-service/manifests/exposure/README.md b/examples/dsh-agents-as-a-service/manifests/exposure/README.md index 42b5bb3..a796575 100644 --- a/examples/dsh-agents-as-a-service/manifests/exposure/README.md +++ b/examples/dsh-agents-as-a-service/manifests/exposure/README.md @@ -16,6 +16,19 @@ Applying both gives the gateway two entry points, and the SSO variant is only safe when the gateway is reachable **exclusively** through oauth2-proxy. Do not mix them. +Each variant also ships a NetworkPolicy, applied by the same directory apply: + +- `ingress/61-networkpolicy.yaml` allows only Traefik pods to reach the gateway + on `8080`. +- `gateway-api-sso/75-networkpolicies.yaml` allows only oauth2-proxy to reach + the gateway, only the Gateway implementation (Traefik) to reach oauth2-proxy, + and limits oauth2-proxy egress to DNS, public HTTPS (GitHub) and the gateway. + +The policies assume Traefik's `app.kubernetes.io/name: traefik` pod label. +Adjust the selector for another implementation or a `hostNetwork` install; see +[optional hardening](../../docs/optional-hardening.md) for the probe caveat and +verification commands. + ## Applying a variant Run these commands from the example root @@ -62,6 +75,9 @@ Secrets imperatively (commands below) before applying the directory. `Secure` cookie is never sent over HTTP, which silently breaks login. 4. Do not attach buffering or compression middlewares to this router: they break WebSocket upgrades and streaming. +5. `61-networkpolicy.yaml` is part of this variant: it allows only Traefik pods + to reach the gateway on `8080`. Adjust its selector for another ingress + controller or `hostNetwork`; if kubelet probes fail, see the hardening doc. Verify: `curl -sI https:///healthz` returns `200`. @@ -70,6 +86,7 @@ Verify: `curl -sI https:///healthz` returns `200`. `70-gateway.yaml` defines the TLS `Gateway`, `71-httproute.yaml` sends every request to oauth2-proxy, and `72-oauth2-proxy.yaml` authenticates the browser against GitHub and proxies to the gateway with `X-Forwarded-User` set. +`75-networkpolicies.yaml` makes this the only network path to the gateway. Prerequisites: @@ -111,7 +128,7 @@ kubectl apply -f manifests/exposure/gateway-api-sso/ `AUTH_PROXY_HEADER` is opt-in and deliberately unset in the base gateway manifest: the gateway ignores identity headers until it is set. Enabling SSO -takes **both** sides: +takes all of these: 1. `AUTH_PROXY_HEADER=X-Forwarded-User` on `deploy/dsh-gateway` (step 3 above). Without it, oauth2-proxy's login has no effect on the gateway and @@ -122,6 +139,10 @@ takes **both** sides: other path to the Service) while SSO is enabled. A client that can reach `dsh-gateway:8080` directly can set `X-Forwarded-User` itself and impersonate any user. +3. Keep `75-networkpolicies.yaml` with the variant. It enforces the boundary in + the network layer (gateway ← oauth2-proxy ← Traefik, and limited oauth2-proxy + egress). A custom exposure without an equivalent policy has no network + enforcement. Also terminate TLS in front of the gateway (the Gateway listener does) and leave `COOKIE_SECURE=true` so the session cookie never travels in clear text. diff --git a/examples/dsh-agents-as-a-service/manifests/exposure/gateway-api-sso/72-oauth2-proxy.yaml b/examples/dsh-agents-as-a-service/manifests/exposure/gateway-api-sso/72-oauth2-proxy.yaml index 52dfa8b..881f852 100644 --- a/examples/dsh-agents-as-a-service/manifests/exposure/gateway-api-sso/72-oauth2-proxy.yaml +++ b/examples/dsh-agents-as-a-service/manifests/exposure/gateway-api-sso/72-oauth2-proxy.yaml @@ -16,6 +16,12 @@ # apply ../ingress/ (or anything else that reaches the gateway). A # client that can reach dsh-gateway:8080 can spoof X-Forwarded-User and # impersonate any user. +# +# 75-networkpolicies.yaml enforces that boundary at the network layer: the +# gateway accepts traffic only from oauth2-proxy pods, oauth2-proxy only from +# the Gateway implementation, and oauth2-proxy egress is limited to DNS, +# public HTTPS and the gateway. Apply it with this variant. +# # TLS terminates at the Gateway listener (70-gateway.yaml); keep # --cookie-secure=true and COOKIE_SECURE=true on the gateway. # @@ -42,6 +48,11 @@ spec: app: oauth2-proxy app.kubernetes.io/part-of: dsh-demo spec: + # OPTIONAL: run the proxy under gVisor and/or pin it to a node pool. + # See ../../../docs/optional-hardening.md. + # runtimeClassName: gvisor + # nodeSelector: + # dsh.example.com/node-pool: dsh securityContext: runAsNonRoot: true runAsUser: 2000 diff --git a/examples/dsh-agents-as-a-service/manifests/exposure/gateway-api-sso/75-networkpolicies.yaml b/examples/dsh-agents-as-a-service/manifests/exposure/gateway-api-sso/75-networkpolicies.yaml new file mode 100644 index 0000000..289ceee --- /dev/null +++ b/examples/dsh-agents-as-a-service/manifests/exposure/gateway-api-sso/75-networkpolicies.yaml @@ -0,0 +1,120 @@ +# OPTIONAL with the SSO exposure variant: enforce the trust boundary at the +# network layer. +# +# The chain is Traefik (Gateway API) -> oauth2-proxy -> dsh-gateway. These +# policies make it the ONLY chain: +# - the gateway accepts traffic only from oauth2-proxy pods; +# - oauth2-proxy accepts traffic only from the Gateway implementation; +# - oauth2-proxy egress is limited to DNS, public HTTPS (GitHub) and the +# gateway. +# The gateway's own egress (Kubernetes API, sandbox pods) is intentionally not +# restricted here: the API-server endpoint is cluster-specific. See +# ../../../docs/optional-hardening.md for that recipe. +# +# This manifest assumes Traefik, matching gatewayClassName: traefik in +# 70-gateway.yaml. Adjust the app.kubernetes.io/name selector for another +# implementation. NetworkPolicy enforcement of kubelet probes is CNI-dependent; +# if the gateway or proxy fails its probes, allow the node CIDR on the probe +# port or remove the ingress policy. +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: dsh-gateway-ingress + namespace: dsh-demo +spec: + podSelector: + matchLabels: + app: dsh-gateway + policyTypes: + - Ingress + ingress: + - from: + - podSelector: + matchLabels: + app: oauth2-proxy + ports: + - protocol: TCP + port: 8080 +--- +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: oauth2-proxy-ingress + namespace: dsh-demo +spec: + podSelector: + matchLabels: + app: oauth2-proxy + policyTypes: + - Ingress + ingress: + - from: + - namespaceSelector: {} + podSelector: + matchLabels: + app.kubernetes.io/name: traefik + ports: + - protocol: TCP + port: 4180 +--- +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: oauth2-proxy-egress + namespace: dsh-demo +spec: + podSelector: + matchLabels: + app: oauth2-proxy + policyTypes: + - Egress + egress: + # Cluster DNS. + - to: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: kube-system + podSelector: + matchLabels: + k8s-app: kube-dns + ports: + - protocol: UDP + port: 53 + - protocol: TCP + port: 53 + # NodeLocal DNSCache. + - to: + - ipBlock: + cidr: 169.254.20.10/32 + ports: + - protocol: UDP + port: 53 + - protocol: TCP + port: 53 + # The gateway backend. + - to: + - podSelector: + matchLabels: + app: dsh-gateway + ports: + - protocol: TCP + port: 8080 + # GitHub OAuth endpoints are public HTTPS. Private ranges and the metadata + # server stay blocked. + - to: + - ipBlock: + cidr: 0.0.0.0/0 + except: + - 10.0.0.0/8 + - 172.16.0.0/12 + - 192.168.0.0/16 + - 169.254.0.0/16 + - 100.64.0.0/10 + - ipBlock: + cidr: "::/0" + except: + - "fc00::/7" + - "fe80::/10" + ports: + - protocol: TCP + port: 443 diff --git a/examples/dsh-agents-as-a-service/manifests/exposure/ingress/60-ingress.yaml b/examples/dsh-agents-as-a-service/manifests/exposure/ingress/60-ingress.yaml index 974f354..55760be 100644 --- a/examples/dsh-agents-as-a-service/manifests/exposure/ingress/60-ingress.yaml +++ b/examples/dsh-agents-as-a-service/manifests/exposure/ingress/60-ingress.yaml @@ -23,6 +23,10 @@ # middlewares (e.g. compress, buffering) to this router — they break streaming # and upgrades. `path: /` Prefix covers the landing page and every per-user # proxy path, so no per-sandbox Ingress is needed. +# +# 61-networkpolicy.yaml is part of this variant: it restricts the gateway's +# ingress to Traefik pods. Adjust its selector if your ingress controller uses +# a different label or hostNetwork. apiVersion: networking.k8s.io/v1 kind: Ingress metadata: diff --git a/examples/dsh-agents-as-a-service/manifests/exposure/ingress/61-networkpolicy.yaml b/examples/dsh-agents-as-a-service/manifests/exposure/ingress/61-networkpolicy.yaml new file mode 100644 index 0000000..15a4be2 --- /dev/null +++ b/examples/dsh-agents-as-a-service/manifests/exposure/ingress/61-networkpolicy.yaml @@ -0,0 +1,31 @@ +# OPTIONAL with the Ingress exposure variant: restrict the gateway's ingress +# to the ingress controller. +# +# This manifest assumes Traefik, matching this variant's +# `ingressClassName: traefik`. Traefik's Helm chart labels its pods +# app.kubernetes.io/name=traefik; adjust the selector for another controller or +# for a hostNetwork install (where the source is the node IP, not a pod IP). +# +# NetworkPolicy enforcement of kubelet probe traffic is CNI-dependent. If the +# gateway starts failing its /healthz probes, allow the node CIDR on port 8080 +# or remove this policy. See ../../../docs/optional-hardening.md. +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: dsh-gateway-ingress + namespace: dsh-demo +spec: + podSelector: + matchLabels: + app: dsh-gateway + policyTypes: + - Ingress + ingress: + - from: + - namespaceSelector: {} + podSelector: + matchLabels: + app.kubernetes.io/name: traefik + ports: + - protocol: TCP + port: 8080 diff --git a/examples/dsh-agents-as-a-service/manifests/optional/README.md b/examples/dsh-agents-as-a-service/manifests/optional/README.md new file mode 100644 index 0000000..1156cfb --- /dev/null +++ b/examples/dsh-agents-as-a-service/manifests/optional/README.md @@ -0,0 +1,37 @@ +# Optional manifests + +Opt-in manifests that are **not** part of the top-level walkthrough. +`kubectl apply -f manifests/` is non-recursive, so it does not apply this +directory; apply a subdirectory explicitly. + +## network-policy/ + +A Kubernetes `NetworkPolicy` for the DSH sandboxes: + +- ingress only from the gateway pods on TCP `3080` (HTTP + WebSocket); +- egress only to DNS and public HTTPS (provider APIs, package mirrors, + git-over-HTTPS). + +Apply and delete it explicitly: + +```bash +kubectl apply -f manifests/optional/network-policy/ +kubectl delete -f manifests/optional/network-policy/ +``` + +It requires a CNI that enforces `NetworkPolicy`. The selector uses the +agent-sandbox controller's pod tracking label, so the policy applies to +existing warm-pool and claimed sandboxes without recreation. + +## gVisor, nodeSelector and provider keys + +These are configured in the main manifests or with an extra Secret: + +- **gVisor:** uncomment `runtimeClassName: gvisor` in the SandboxTemplate and + the Deployments before applying, or patch an existing install. +- **nodeSelector:** uncomment the field and replace the placeholder node label. +- **Provider keys:** create the optional `dsh-provider-keys-extra` Secret; the + base `dsh-provider-keys` Secret remains the one required key. + +Full prerequisites, exact commands, verification and caveats: +[optional hardening](../../docs/optional-hardening.md). diff --git a/examples/dsh-agents-as-a-service/manifests/optional/network-policy/10-sandbox-networkpolicy.yaml b/examples/dsh-agents-as-a-service/manifests/optional/network-policy/10-sandbox-networkpolicy.yaml new file mode 100644 index 0000000..ee087bd --- /dev/null +++ b/examples/dsh-agents-as-a-service/manifests/optional/network-policy/10-sandbox-networkpolicy.yaml @@ -0,0 +1,80 @@ +# OPTIONAL, opt-in tenant network policy for the DSH sandboxes. +# +# It is NOT part of the top-level walkthrough. Apply and delete it explicitly: +# kubectl apply -f manifests/optional/network-policy/ +# kubectl delete -f manifests/optional/network-policy/ +# +# The selector uses the agent-sandbox controller's pod tracking label, which is +# present on every Sandbox pod (including warm-pool spares), so the policy +# applies to existing sandboxes without recreating them. The SandboxTemplate +# stays networkPolicyManagement: Unmanaged. Do not also switch it to Managed +# unless you migrate these rules into spec.networkPolicy; NetworkPolicies are +# additive and the two configurations would be confusing. +# +# Requires a CNI that enforces NetworkPolicy. See docs/optional-hardening.md +# for the connectivity matrix, DNS variants, kubelet-probe caveat and rollback. +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: dsh-sandbox + namespace: dsh-demo +spec: + podSelector: + matchExpressions: + - key: agents.x-k8s.io/sandbox-name-hash + operator: Exists + policyTypes: + - Ingress + - Egress + ingress: + # The gateway is the only intended client of a sandbox (HTTP + WebSocket). + - from: + - podSelector: + matchLabels: + app: dsh-gateway + ports: + - protocol: TCP + port: 3080 + egress: + # Cluster DNS. CoreDNS/kube-dns pods carry k8s-app=kube-dns. + - to: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: kube-system + podSelector: + matchLabels: + k8s-app: kube-dns + ports: + - protocol: UDP + port: 53 + - protocol: TCP + port: 53 + # NodeLocal DNSCache, for clusters that do not send DNS to the kube-dns + # Service. 169.254.20.10 is its standard link-local address. + - to: + - ipBlock: + cidr: 169.254.20.10/32 + ports: + - protocol: UDP + port: 53 + - protocol: TCP + port: 53 + # Public HTTPS only: provider APIs, package mirrors, git-over-HTTPS. + # Private ranges, CGNAT and the metadata server stay blocked. + - to: + - ipBlock: + cidr: 0.0.0.0/0 + except: + - 10.0.0.0/8 + - 172.16.0.0/12 + - 192.168.0.0/16 + - 169.254.0.0/16 + - 100.64.0.0/10 + - ipBlock: + cidr: "::/0" + except: + - "fc00::/7" + - "fe80::/10" + ports: + - protocol: TCP + port: 443 diff --git a/examples/dsh-agents-as-a-service/scripts/e2e-kind.sh b/examples/dsh-agents-as-a-service/scripts/e2e-kind.sh index 2aaf1e4..850417c 100755 --- a/examples/dsh-agents-as-a-service/scripts/e2e-kind.sh +++ b/examples/dsh-agents-as-a-service/scripts/e2e-kind.sh @@ -49,6 +49,8 @@ kubectl apply -f manifests/00-prereqs.yaml kubectl -n "$NS" delete secret dsh-provider-keys --ignore-not-found >/dev/null kubectl -n "$NS" create secret generic dsh-provider-keys \ --from-literal=DEEPSEEK_API_KEY="sk-test-$(openssl rand -hex 16)" +# dsh-provider-keys-extra is intentionally absent: the template's optional +# envFrom reference must not block the warm pool. kubectl apply -f manifests/10-sandboxtemplate.yaml -f manifests/20-sandboxwarmpool.yaml for _ in $(seq 1 120); do READY=$(kubectl -n "$NS" get sandboxes -l agents.x-k8s.io/warm-pool-sandbox \ diff --git a/examples/minimal-single-tenant/README.md b/examples/minimal-single-tenant/README.md index dec87ac..87fffd9 100644 --- a/examples/minimal-single-tenant/README.md +++ b/examples/minimal-single-tenant/README.md @@ -25,19 +25,28 @@ This example deliberately has no gateway: you reach DSH through a ``` - A DeepSeek API key for real model calls. The pod starts without one; only - model requests fail. + model requests fail. Optionally create `dsh-provider-keys-extra` for + OpenRouter, Gemini, NVIDIA or other provider keys (the template references it + with `optional: true`). ## Quickstart Run these from this example's directory (`examples/minimal-single-tenant/`). ```bash -# 1. Namespace + shared provider key. The key is injected into the sandbox via -# envFrom and is never committed. +# 1. Namespace + shared provider keys. They are injected into the sandbox via +# envFrom and are never committed. kubectl create namespace dsh-demo kubectl -n dsh-demo create secret generic dsh-provider-keys \ --from-literal=DEEPSEEK_API_KEY="sk-..." +# Optional: extra providers, injected through a second Secret. Create it before +# the warm pool so the pods are stamped with the keys. +kubectl -n dsh-demo create secret generic dsh-provider-keys-extra \ + --from-literal=OPENROUTER_API_KEY="sk-or-..." \ + --from-literal=GEMINI_API_KEY="..." \ + --from-literal=NVIDIA_API_KEY="nvapi-..." + # 2. The template itself (image pinned in the manifest). kubectl apply -f manifests/sandbox-template.yaml @@ -86,6 +95,22 @@ kubectl -n dsh-demo port-forward "pod/$SB" 18081:3080 > use on your workstation, forward to another local port (here `18081`), never > `3080`. `localhost` is always in DSH's trusted-host list, so the forwarded > request is accepted. +> +> **gVisor caveat:** if you uncomment `runtimeClassName: gvisor` in the +> template, `kubectl port-forward` no longer works. Use the multi-tenant gateway +> example or a reachable Service instead; see +> [optional hardening](../dsh-agents-as-a-service/docs/optional-hardening.md). + +## Network policy and optional hardening + +The template omits `networkPolicyManagement`, so the agent-sandbox controller +uses its Managed secure default: ingress only from the sandbox-router, public +egress, and no internal cluster DNS. `kubectl port-forward` is local to the pod +and still works (except under gVisor). + +gVisor, an optional `nodeSelector`, extra provider keys and a stricter sandbox +NetworkPolicy are documented in +[optional hardening](../dsh-agents-as-a-service/docs/optional-hardening.md). ## Probes and trusted hosts diff --git a/examples/minimal-single-tenant/manifests/sandbox-template.yaml b/examples/minimal-single-tenant/manifests/sandbox-template.yaml index 1b17827..1b673ae 100644 --- a/examples/minimal-single-tenant/manifests/sandbox-template.yaml +++ b/examples/minimal-single-tenant/manifests/sandbox-template.yaml @@ -10,6 +10,9 @@ # - Namespace dsh-demo. # - Secret dsh-provider-keys in dsh-demo with key DEEPSEEK_API_KEY # (injected via envFrom; the pod starts without it, only model calls fail). +# - Optional: Secret dsh-provider-keys-extra with additional provider keys +# (OPENROUTER_API_KEY, GEMINI_API_KEY, NVIDIA_API_KEY, ...). It is +# referenced with `optional: true`, so it may be absent. # The agent-sandbox controller (v1.0.7, API v1beta1) must be installed. apiVersion: extensions.agents.x-k8s.io/v1beta1 kind: SandboxTemplate @@ -18,8 +21,22 @@ metadata: namespace: dsh-demo spec: service: true # headless Service per Sandbox -> stable DNS (status.serviceFQDN) + # networkPolicyManagement is omitted, so the controller uses its Managed + # secure default (ingress only from the sandbox-router, public egress). This + # quickstart reaches the sandbox through `kubectl port-forward`, which is + # local to the pod. The multi-tenant example documents opt-in policies in + # examples/dsh-agents-as-a-service/docs/optional-hardening.md. podTemplate: spec: + # OPTIONAL: run under gVisor. gVisor breaks `kubectl port-forward`, so + # this quickstart's only access path will not work; use the multi-tenant + # gateway example or a Service/NodePort instead. + # runtimeClassName: gvisor + # + # OPTIONAL: pin to a dedicated node pool. Replace the placeholder with a + # node label; add a matching toleration if the pool is tainted. + # nodeSelector: + # dsh.example.com/node-pool: dsh securityContext: runAsNonRoot: true runAsUser: 1000 # Matches the UID of the 'node' user @@ -43,6 +60,10 @@ spec: envFrom: - secretRef: name: dsh-provider-keys # Secret with key DEEPSEEK_API_KEY + # Optional extra provider keys; an absent Secret is safe. + - secretRef: + name: dsh-provider-keys-extra + optional: true env: # Empty => DSH derives links from the request Host. Keep it empty # when fronting the sandbox with a proxy/gateway. diff --git a/scripts/check-examples.sh b/scripts/check-examples.sh index 4cb2941..e0dcf40 100755 --- a/scripts/check-examples.sh +++ b/scripts/check-examples.sh @@ -29,6 +29,10 @@ for dir in examples/*/; do err "$name: manifests/exposure/ requires a README.md describing the variants" fi + if [ -d "${dir}manifests/optional" ] && [ ! -f "${dir}manifests/optional/README.md" ]; then + err "$name: manifests/optional/ requires a README.md describing the opt-in manifests" + fi + for component in "$dir"*/; do [ -d "$component" ] || continue case "$(basename "$component")" in