mirror of
https://github.com/sredevopsorg/deepseek-kubernetes-sandbox.git
synced 2026-11-03 01:13:24 +00:00
feat(example): add optional gVisor, nodeSelector, NetworkPolicy and provider keys (#21)
Add opt-in hardening to the agents-as-a-service example: - commented runtimeClassName/nodeSelector fields for sandboxes and Deployments - optional sandbox NetworkPolicy plus exposure-variant policies - optional dsh-provider-keys-extra Secret for extra provider keys - docs/optional-hardening.md with verification, caveats and rollback The base walkthrough stays portable; no gateway code, routes or auth changed. Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
This commit is contained in:
SREDevOps.org Agent
committed by
GitHub
GitHub
parent
bf24306421
commit
3e9ba38fc1
@@ -129,6 +129,10 @@ Read `docs/examples.md` and the testing table in `AGENTS.md` before editing.
|
||||
`make check` and the e2e. Never mix a bump with an unrelated feature.
|
||||
- **MCP integration:** pin the build to the cluster's tag, least-privilege SA,
|
||||
ClusterIP + auth proxy, raised DSH timeout, no secrets in the repo.
|
||||
- **Optional hardening:** gVisor, `nodeSelector`, NetworkPolicy and extra
|
||||
provider keys stay opt-in; the sandbox policy lives under
|
||||
`manifests/optional/`. All four are documented in
|
||||
`examples/dsh-agents-as-a-service/docs/optional-hardening.md`.
|
||||
- **Docs:** update the affected doc in the same change.
|
||||
|
||||
Details and review checklist:
|
||||
|
||||
@@ -25,6 +25,7 @@ implementation is a self-contained example under `examples/<name>/`, and
|
||||
| --- | --- | --- |
|
||||
| Documentation only | `*.md`, `docs/` | `make check`; relative links resolve |
|
||||
| Example manifest | `examples/<name>/manifests/NN-*.yaml`, README step | `make check`; server-side dry-run in CI |
|
||||
| Optional hardening | `manifests/optional/`, exposure NPs, example `docs/optional-hardening.md` | `make check`; manual verification on a capable cluster |
|
||||
| Gateway behavior/route | `gateway/gateway.py`, `gateway/test_gateway.py`, gateway README, architecture | `make test`, `make lint`; update tests and API docs |
|
||||
| Exposure variant | `manifests/exposure/<variant>/`, exposure README, example README | `make check`; apply exactly one variant |
|
||||
| New example | full `examples/<name>/` layout, `examples/README.md` | `make check`; CI dry-run; optional e2e |
|
||||
|
||||
@@ -141,8 +141,13 @@ host. A resource-only install has no gateway: that is a warning, not a failure.
|
||||
|
||||
## 9. Security boundaries
|
||||
|
||||
- One `DEEPSEEK_API_KEY` is shared by all sandboxes; the example is not
|
||||
- Provider keys (`dsh-provider-keys` plus the optional
|
||||
`dsh-provider-keys-extra`) are shared by all sandboxes; the example is not
|
||||
tenant-isolated.
|
||||
- gVisor, `nodeSelector`, the sandbox/exposure NetworkPolicies and extra
|
||||
provider keys are opt-in, not active by default. A NetworkPolicy only helps
|
||||
with a CNI that enforces it; see
|
||||
[optional hardening](../../../../examples/dsh-agents-as-a-service/docs/optional-hardening.md).
|
||||
- Secret data, kubeconfigs, and pod logs are sensitive; RBAC may allow Secret
|
||||
access, policy does not. Keep gateway RBAC namespace-scoped; it needs pod logs
|
||||
to find the launch token.
|
||||
|
||||
15
.env.sample
15
.env.sample
@@ -5,6 +5,21 @@
|
||||
|
||||
# --- Main DSH Web image (root Dockerfile) ---
|
||||
DEEPSEEK_API_KEY=
|
||||
# Optional additional provider keys consumed by DSH. For Docker, pass them with
|
||||
# -e; the Kubernetes examples inject them through the optional
|
||||
# dsh-provider-keys-extra Secret.
|
||||
OPENROUTER_API_KEY=
|
||||
GEMINI_API_KEY=
|
||||
NVIDIA_API_KEY=
|
||||
OPENAI_API_KEY=
|
||||
ANTHROPIC_API_KEY=
|
||||
GROQ_API_KEY=
|
||||
XAI_API_KEY=
|
||||
MISTRAL_API_KEY=
|
||||
TOGETHER_API_KEY=
|
||||
CEREBRAS_API_KEY=
|
||||
AI_GATEWAY_API_KEY=
|
||||
HF_TOKEN=
|
||||
DSH_VERSION=dsh-v0.2.1-rc.3
|
||||
RELEASE_REVISION=r3
|
||||
PNPM_VERSION=11.28.5
|
||||
|
||||
@@ -72,6 +72,7 @@ needs kind plus the agent-sandbox CRDs (see the example README).
|
||||
| --- | --- |
|
||||
| Documentation only | make check; verify relative links |
|
||||
| Example manifest | make check; kubectl apply --dry-run=server in CI |
|
||||
| Optional hardening (gVisor, nodeSelector, NetworkPolicy, provider keys) | make check; server dry-run in CI; manual verification on a capable cluster |
|
||||
| Gateway code | make test; make lint; add or update tests |
|
||||
| Dockerfile / versions.env | make check; make image (amd64) and a container smoke test |
|
||||
| Workflow | actionlint in CI; keep actions SHA-pinned; no hard-coded versions |
|
||||
@@ -120,8 +121,11 @@ needs kind plus the agent-sandbox CRDs (see the example README).
|
||||
can spoof X-Forwarded-User.
|
||||
- The example gateway reads the DSH launch token from pod logs. Treat pod logs
|
||||
as sensitive and keep its RBAC namespace-scoped.
|
||||
- The example shares one DEEPSEEK_API_KEY across sandboxes; do not describe it
|
||||
as tenant-isolated.
|
||||
- The example shares provider keys (dsh-provider-keys plus the optional
|
||||
dsh-provider-keys-extra) across sandboxes; do not describe it as
|
||||
tenant-isolated.
|
||||
- gVisor, nodeSelector and the NetworkPolicies are opt-in and are not active in
|
||||
the base manifests; do not describe the base example as hardened.
|
||||
- TLS terminates outside the gateway; set COOKIE_SECURE=true for HTTPS and keep
|
||||
the base manifest safe for local HTTP.
|
||||
|
||||
|
||||
13
README.md
13
README.md
@@ -72,6 +72,13 @@ docker run -d --name dsh-web -p 18081:3080 \
|
||||
| `DSH_TRUSTED_HOSTS` | empty | Extra trusted `Host`/`Origin` entries, comma/space separated. |
|
||||
| `DSH_HOME` | `/home/node/.dsh` | DSH state directory (mount a volume here). |
|
||||
|
||||
DSH also resolves the standard per-provider environment names —
|
||||
`OPENROUTER_API_KEY`, `GEMINI_API_KEY`, `NVIDIA_API_KEY`, `OPENAI_API_KEY`,
|
||||
`ANTHROPIC_API_KEY`, `GROQ_API_KEY`, `XAI_API_KEY`, `HF_TOKEN`, and others —
|
||||
through its packaged provider catalog. Inject them the same way; a key saved in
|
||||
DSH's credential store wins over the environment. See
|
||||
[provider credentials](docs/main-image.md#provider-credentials).
|
||||
|
||||
Full contract, build args, probes, volumes and signals:
|
||||
[docs/main-image.md](docs/main-image.md).
|
||||
|
||||
@@ -125,12 +132,14 @@ Self-contained implementations that consume the image. Index and contract:
|
||||
|
||||
| Example | What it demonstrates |
|
||||
| --- | --- |
|
||||
| [`examples/dsh-agents-as-a-service/`](examples/dsh-agents-as-a-service/) | Full multi-tenant platform: SandboxTemplate with per-user PVCs, warm pool, one claim per user, suspend-when-idle, wake-on-return, browser gateway with signup + `HttpOnly` sessions + HTTP/WebSocket proxy, Ingress or Gateway API + GitHub SSO exposure. Start with the [walkthrough](examples/dsh-agents-as-a-service/README.md) and the [architecture](examples/dsh-agents-as-a-service/docs/architecture.md). |
|
||||
| [`examples/dsh-agents-as-a-service/`](examples/dsh-agents-as-a-service/) | Full multi-tenant platform: SandboxTemplate with per-user PVCs, warm pool, one claim per user, suspend-when-idle, wake-on-return, browser gateway with signup + `HttpOnly` sessions + HTTP/WebSocket proxy, Ingress or Gateway API + GitHub SSO exposure, and opt-in gVisor/NetworkPolicy/provider-key hardening. Start with the [walkthrough](examples/dsh-agents-as-a-service/README.md) and the [architecture](examples/dsh-agents-as-a-service/docs/architecture.md). |
|
||||
| [`examples/minimal-single-tenant/`](examples/minimal-single-tenant/) | The smallest useful setup: one `SandboxTemplate` and one `SandboxClaim`, reached through `kubectl port-forward` + the launch-token URL. Manifest: [`manifests/sandbox-template.yaml`](examples/minimal-single-tenant/manifests/sandbox-template.yaml). |
|
||||
|
||||
The gateway is a component of the multi-tenant example and has its own
|
||||
[README](examples/dsh-agents-as-a-service/gateway/README.md) (API, configuration,
|
||||
auth/SSO, limitations).
|
||||
auth/SSO, limitations). The example also documents opt-in gVisor,
|
||||
`nodeSelector`, NetworkPolicy and extra provider-key hardening in
|
||||
[optional hardening](examples/dsh-agents-as-a-service/docs/optional-hardening.md).
|
||||
|
||||
## Agent skills
|
||||
|
||||
|
||||
15
SECURITY.md
15
SECURITY.md
@@ -25,10 +25,17 @@ disclosure once a fix is available.
|
||||
- **SSO header trust boundary.** AUTH_PROXY_HEADER makes the gateway trust an
|
||||
identity header (for example X-Forwarded-User). Enable it only when the
|
||||
gateway Service is reachable exclusively through the authenticating proxy;
|
||||
otherwise any client can spoof the header.
|
||||
- **Shared provider key.** The example injects one DEEPSEEK_API_KEY into every
|
||||
tenant sandbox (a warm-pool constraint). Treat tenants as mutually trusting
|
||||
until per-tenant credentials or an egress proxy exist.
|
||||
otherwise any client can spoof the header. The SSO exposure variant ships a
|
||||
NetworkPolicy that restricts gateway access to oauth2-proxy pods; a custom
|
||||
exposure needs an equivalent.
|
||||
- **Shared provider keys.** The example injects DEEPSEEK_API_KEY and any keys in
|
||||
the optional dsh-provider-keys-extra Secret into every tenant sandbox (a
|
||||
warm-pool constraint). Treat tenants as mutually trusting until per-tenant
|
||||
credentials or an egress proxy exist.
|
||||
- **Optional isolation is opt-in.** gVisor, nodeSelector, the sandbox/exposure
|
||||
NetworkPolicies and extra provider keys are opt-in. Without a
|
||||
NetworkPolicy-capable CNI or a gVisor runtime, the base example is not
|
||||
tenant-isolated; see examples/dsh-agents-as-a-service/docs/optional-hardening.md.
|
||||
- **TLS is external.** Terminate TLS at an Ingress or Gateway and set
|
||||
COOKIE_SECURE=true. Plain HTTP outside localhost exposes session cookies.
|
||||
- **No secrets in the repository.** The repository only ships empty Secret
|
||||
|
||||
@@ -18,6 +18,9 @@ examples/<name>/
|
||||
| # ordered steps, troubleshooting, cleanup
|
||||
+-- manifests/ # required
|
||||
| +-- NN-*.yaml # numbered top-level manifests, plain kubectl
|
||||
| +-- optional/ # optional opt-in manifests, applied by path
|
||||
| +-- README.md # required when optional/ exists
|
||||
| +-- <feature>/NN-*.yaml # e.g. network-policy/
|
||||
| +-- exposure/ # optional exposure variants
|
||||
| +-- README.md # required when exposure/ exists
|
||||
| +-- <variant>/NN-*.yaml # e.g. ingress/, gateway-api-sso/
|
||||
@@ -41,6 +44,8 @@ examples/<name>/
|
||||
- a `*.yaml` file sits at the example root instead of under `manifests/`;
|
||||
- `manifests/exposure/` exists without a `manifests/exposure/README.md` describing
|
||||
the variants;
|
||||
- `manifests/optional/` exists without a `manifests/optional/README.md`
|
||||
describing the opt-in manifests;
|
||||
- a subdirectory with a `Dockerfile` has no `README.md`;
|
||||
- a file in `scripts/` is not executable.
|
||||
|
||||
@@ -68,7 +73,12 @@ make check
|
||||
command; real values are generated per install.
|
||||
- **Cluster-specific state stays out.** Node affinity, replica counts tuned for a
|
||||
cluster, and similar fields belong in a local patch, not in the shipped
|
||||
manifest.
|
||||
manifest. Optional scheduler/runtime hints may ship **commented out** with
|
||||
placeholder labels; they must not be active.
|
||||
- **Optional manifests are opt-in.** Put them under
|
||||
`manifests/optional/<feature>/`, document them in
|
||||
`manifests/optional/README.md`, and apply them by path. The top-level
|
||||
`kubectl apply -f manifests/` is non-recursive and does not apply them.
|
||||
- **Placeholders are explicit.** Hostnames, GitHub orgs and TLS material use
|
||||
values like `dsh.example.com` / `your-github-org` with a TODO comment.
|
||||
|
||||
|
||||
@@ -80,6 +80,28 @@ The entrypoint is `tini -- /usr/local/bin/dsh-entrypoint`:
|
||||
The image also sets `NODE_ENV=production`, `HOME=/home/node`, and a `PATH` that
|
||||
includes `/app/node_modules/.bin`.
|
||||
|
||||
### Provider credentials
|
||||
|
||||
DSH resolves provider API keys from the environment when no stored credential
|
||||
exists. The packaged provider catalog maps each provider to its standard
|
||||
environment name. Commonly used names:
|
||||
|
||||
`DEEPSEEK_API_KEY`, `OPENROUTER_API_KEY`, `GEMINI_API_KEY`, `NVIDIA_API_KEY`,
|
||||
`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GROQ_API_KEY`, `XAI_API_KEY`,
|
||||
`MISTRAL_API_KEY`, `TOGETHER_API_KEY`, `CEREBRAS_API_KEY`,
|
||||
`AI_GATEWAY_API_KEY`, `HF_TOKEN`.
|
||||
|
||||
Use `GEMINI_API_KEY` for Google Gemini (not `GOOGLE_API_KEY`), `HF_TOKEN` for
|
||||
Hugging Face, and `AI_GATEWAY_API_KEY` for the Vercel AI Gateway. Inject keys
|
||||
with `-e`/`--env-file` in Docker or a Secret + `envFrom` in Kubernetes; never
|
||||
bake them into the image.
|
||||
|
||||
A key saved through DSH's credential store (`$DSH_HOME/.credentials.yaml`) wins
|
||||
over the environment. The
|
||||
[agents-as-a-service example](../examples/dsh-agents-as-a-service/docs/optional-hardening.md)
|
||||
documents the optional `dsh-provider-keys-extra` Secret and the pod-refresh
|
||||
procedure.
|
||||
|
||||
### Ports, volumes and signals
|
||||
|
||||
| Item | Value | Notes |
|
||||
|
||||
@@ -10,6 +10,10 @@ reference material, not libraries.
|
||||
| [`examples/dsh-agents-as-a-service/`](dsh-agents-as-a-service/) | The full multi-tenant platform pattern: one `SandboxTemplate` with per-user PVCs, a warm pool, one ~10-line `SandboxClaim` per user, suspend-when-idle / wake-on-return, and a browser gateway with signup, `HttpOnly` sessions, an HTTP + WebSocket proxy and two exposure variants (Ingress, or Gateway API + GitHub SSO). | agent-sandbox with extensions; kind for the e2e script. |
|
||||
| [`examples/minimal-single-tenant/`](minimal-single-tenant/) | The smallest useful setup: a single `SandboxTemplate` and one `SandboxClaim`, no gateway — reach DSH Web with a launch-token URL through `kubectl port-forward`. | agent-sandbox with extensions. |
|
||||
|
||||
The multi-tenant example also documents opt-in gVisor, `nodeSelector`,
|
||||
NetworkPolicy and extra provider-key hardening in
|
||||
[optional hardening](dsh-agents-as-a-service/docs/optional-hardening.md).
|
||||
|
||||
## The example contract (short version)
|
||||
|
||||
Full rules: [docs/examples.md](../docs/examples.md). `scripts/check-examples.sh`
|
||||
@@ -19,6 +23,8 @@ enforces the layout, and `make check` runs it.
|
||||
examples/<name>/
|
||||
+-- README.md # walkthrough: prerequisites, steps, troubleshooting, cleanup
|
||||
+-- manifests/ # NN-*.yaml, applied with plain kubectl
|
||||
| +-- optional/<feature>/ # optional opt-in manifests...
|
||||
| +-- optional/README.md # ...required whenever optional/ exists
|
||||
| +-- exposure/<variant>/ # optional per-variant manifests...
|
||||
| +-- exposure/README.md # ...required whenever exposure/ exists
|
||||
+-- scripts/ # optional executable, idempotent, self-cleaning *.sh
|
||||
@@ -44,4 +50,6 @@ kubectl apply -f examples/dsh-agents-as-a-service/manifests/
|
||||
examples/dsh-agents-as-a-service/scripts/e2e-kind.sh
|
||||
```
|
||||
|
||||
Each example README has the exact prerequisites, exposure options and teardown.
|
||||
`kubectl apply -f manifests/` is non-recursive: `optional/` and `exposure/`
|
||||
subdirectories are applied explicitly. Each example README has the exact
|
||||
prerequisites, optional hardening, exposure options and teardown.
|
||||
|
||||
@@ -32,10 +32,14 @@ manifests/
|
||||
+-- 30-claim-alice.yaml # one user
|
||||
+-- 40-claim-rejected.yaml # negative demo: env injection rejected
|
||||
+-- 50-gateway.yaml # Deployment/RBAC/Service for the gateway
|
||||
+-- optional/
|
||||
| +-- README.md # opt-in manifests, applied by path
|
||||
| +-- network-policy/10-sandbox-networkpolicy.yaml
|
||||
+-- exposure/
|
||||
+-- README.md # the two exposure variants, when to use which
|
||||
+-- ingress/60-ingress.yaml
|
||||
+-- gateway-api-sso/70-gateway.yaml ... 74-tls-secret.yaml.example
|
||||
+-- ingress/61-networkpolicy.yaml
|
||||
+-- gateway-api-sso/70-gateway.yaml ... 75-networkpolicies.yaml
|
||||
```
|
||||
|
||||
What it demonstrates, feature by feature:
|
||||
@@ -57,8 +61,10 @@ What it demonstrates, feature by feature:
|
||||
`kubernetes.io/arch: amd64` nodeSelector.
|
||||
- **Single port**: DSH Web serves everything on `3080`; the gateway proxies to one
|
||||
upstream instead of splitting `/v1/*` (API) from the dashboard.
|
||||
- **One shared secret**: only `DEEPSEEK_API_KEY` is injected (via `envFrom`); there
|
||||
is no dashboard basic-auth, session-cookie secret, or API-server key.
|
||||
- **Shared provider keys**: `DEEPSEEK_API_KEY` plus any keys in the optional
|
||||
`dsh-provider-keys-extra` Secret are injected via `envFrom` into every
|
||||
sandbox. There is no per-tenant key, dashboard basic-auth, session-cookie
|
||||
secret, or API-server key.
|
||||
- **Launch-token handshake**: DSH gates `/` behind a token it prints only in its
|
||||
logs. The gateway reads it, performs the handshake, and keeps the resulting
|
||||
auth cookie server-side; the browser authenticates to the gateway with its own
|
||||
@@ -93,14 +99,39 @@ What it demonstrates, feature by feature:
|
||||
|
||||
## 1. Install the platform resources
|
||||
|
||||
First the namespace, then the shared provider key. The key reaches **every**
|
||||
First the namespace, then the shared provider keys. The keys reach **every**
|
||||
tenant sandbox (a warm-pool constraint: pod env is stamped before any user
|
||||
exists), so it is **never committed** — generate a fresh value per install:
|
||||
exists), so they are **never committed** — generate fresh values per install.
|
||||
`00-prereqs.yaml` already creates the empty `dsh-provider-keys` Secret, so use
|
||||
an upsert:
|
||||
|
||||
```bash
|
||||
kubectl apply -f manifests/00-prereqs.yaml
|
||||
kubectl -n dsh-demo create secret generic dsh-provider-keys \
|
||||
--from-literal=DEEPSEEK_API_KEY="sk-..."
|
||||
--from-literal=DEEPSEEK_API_KEY="sk-..." \
|
||||
--dry-run=client -o yaml | kubectl apply -f -
|
||||
```
|
||||
|
||||
Optional: additional providers (OpenRouter, Gemini, NVIDIA, OpenAI, Anthropic,
|
||||
...) go in a second Secret. It is referenced with `optional: true`, so the
|
||||
sandboxes start without it, and every key becomes an env var:
|
||||
|
||||
```bash
|
||||
kubectl -n dsh-demo create secret generic dsh-provider-keys-extra \
|
||||
--from-literal=OPENROUTER_API_KEY="sk-or-..." \
|
||||
--from-literal=GEMINI_API_KEY="..." \
|
||||
--from-literal=NVIDIA_API_KEY="nvapi-..." \
|
||||
--dry-run=client -o yaml | kubectl apply -f -
|
||||
```
|
||||
|
||||
Create both Secrets **before** the template/warm pool. A later Secret change
|
||||
does not update running pods: delete the warm-pool spares and recreate claimed
|
||||
pods to pick up new keys. Keys are shared by all tenants; a key saved in DSH's
|
||||
credential store wins over env. See
|
||||
[optional hardening](docs/optional-hardening.md) for the full env-name list and
|
||||
the refresh procedure.
|
||||
|
||||
```bash
|
||||
kubectl apply -f manifests/10-sandboxtemplate.yaml -f manifests/20-sandboxwarmpool.yaml
|
||||
```
|
||||
|
||||
@@ -253,6 +284,12 @@ kubectl -n dsh-demo port-forward svc/dsh-gateway 18080:8080
|
||||
> apply does not update them: patch each sandbox's
|
||||
> `spec.podTemplate.spec.securityContext.fsGroupChangePolicy` and
|
||||
> `chmod 600` its `.credentials.yaml`, or recreate it.
|
||||
>
|
||||
> The optional gVisor/`nodeSelector` fields are pod-spec changes too: existing
|
||||
> Sandboxes snapshot them, and warm-pool spares are not replaced by a template
|
||||
> edit. See [optional hardening](docs/optional-hardening.md) for the patch and
|
||||
> recreation commands. A provider-key Secret change likewise needs pod
|
||||
> recreation to take effect.
|
||||
|
||||
### Browser flow
|
||||
|
||||
@@ -388,6 +425,13 @@ not a stack. Details, commands and the switch procedure are in the
|
||||
> `kubectl apply -R`: it would apply both variants and create two paths to the
|
||||
> same gateway, which is unsafe once SSO is enabled.
|
||||
|
||||
Both variants also ship NetworkPolicies: `ingress/61-networkpolicy.yaml` limits
|
||||
gateway ingress to Traefik pods, and `gateway-api-sso/75-networkpolicies.yaml`
|
||||
limits the gateway to oauth2-proxy and oauth2-proxy egress to DNS, public HTTPS
|
||||
and the gateway. They are part of the variant; see the
|
||||
[exposure README](manifests/exposure/README.md) and
|
||||
[optional hardening](docs/optional-hardening.md).
|
||||
|
||||
### Option A — plain Ingress (gateway's own session auth)
|
||||
|
||||
```bash
|
||||
@@ -404,6 +448,10 @@ kubectl apply -f manifests/exposure/ingress/
|
||||
(The base manifest keeps `false` so the plain-HTTP port-forward demo works.)
|
||||
3. Verify: `curl -sI https://<host>/healthz` returns `200`; sign up in a browser
|
||||
at `https://<host>/`.
|
||||
4. `manifests/exposure/ingress/61-networkpolicy.yaml` is part of this variant:
|
||||
it allows only Traefik pods to reach the gateway on `8080`. Adjust its
|
||||
selector if your ingress controller uses a different pod label or
|
||||
`hostNetwork`; if kubelet probes fail, see the hardening doc.
|
||||
|
||||
Keep the router free of buffering/compression middlewares: they break WebSocket
|
||||
upgrades and streaming.
|
||||
@@ -472,13 +520,16 @@ can be cleared at `/oauth2/sign_out`.
|
||||
|
||||
> **SSO trust boundary — read this before enabling it.** After SSO is on, the
|
||||
> `dsh-gateway` Service must be reachable **only** through oauth2-proxy (it is
|
||||
> ClusterIP). Anyone who can reach `dsh-gateway:8080` directly can send
|
||||
> `X-Forwarded-User: <victim>` and be treated as that user. Never apply the
|
||||
> Ingress variant (or any other route to the Service) while SSO is enabled, and
|
||||
> do not expose the gateway Service with `kubectl port-forward` on a shared
|
||||
> machine while `AUTH_PROXY_HEADER` is set. To turn SSO off, remove the variable
|
||||
> (`kubectl -n dsh-demo set env deploy/dsh-gateway AUTH_PROXY_HEADER-`) and
|
||||
> delete the SSO variant; the gateway falls back to bearer-token/session auth.
|
||||
> ClusterIP). The variant's `75-networkpolicies.yaml` restricts pod traffic
|
||||
> accordingly: the gateway accepts traffic only from oauth2-proxy pods,
|
||||
> oauth2-proxy only from the Gateway implementation, and oauth2-proxy egress is
|
||||
> limited to DNS, public HTTPS and the gateway. Anyone who can reach
|
||||
> `dsh-gateway:8080` directly can still send `X-Forwarded-User: <victim>` and be
|
||||
> treated as that user, so never add a second path (the Ingress variant, a
|
||||
> shared-host port-forward, a permissive policy). To turn SSO off, remove the
|
||||
> variable (`kubectl -n dsh-demo set env deploy/dsh-gateway AUTH_PROXY_HEADER-`)
|
||||
> and delete the SSO variant; the gateway falls back to bearer-token/session
|
||||
> auth.
|
||||
|
||||
### Switching variants
|
||||
|
||||
@@ -488,6 +539,25 @@ Delete the variant you had before applying the other one
|
||||
`kubectl -n dsh-demo get ingress,gateway,httproute` that exactly one path to the
|
||||
gateway remains.
|
||||
|
||||
## Optional hardening
|
||||
|
||||
The base walkthrough is deliberately portable. Four opt-in layers are documented
|
||||
in [optional hardening](docs/optional-hardening.md):
|
||||
|
||||
- **gVisor** — uncomment `runtimeClassName: gvisor` before applying, or patch an
|
||||
existing install. Requires a `gvisor` RuntimeClass and capable nodes.
|
||||
`kubectl port-forward` does not work under gVisor; the gateway path does.
|
||||
- **nodeSelector** — uncomment and replace the placeholder node label to pin
|
||||
sandboxes and the gateway to a dedicated pool; add a toleration for tainted
|
||||
pools.
|
||||
- **Sandbox NetworkPolicy** —
|
||||
`kubectl apply -f manifests/optional/network-policy/` allows only
|
||||
gateway→sandbox on `3080` and sandbox→DNS/public `443`.
|
||||
- **Provider keys** — create `dsh-provider-keys-extra` as shown in step 1.
|
||||
|
||||
Apply each independently and re-run the walkthrough. `kubectl apply -f manifests/`
|
||||
is non-recursive, so the optional manifest is not applied by the base step.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Cause / fix |
|
||||
@@ -511,6 +581,11 @@ gateway remains.
|
||||
| SSO: oauth2-proxy `cookie_secret must be 16, 24, or 32 bytes ... but is 44 bytes` | The cookie secret was generated as standard base64. Use URL-safe unpadded base64 (see `73-oauth2-proxy-secret.yaml.example`) |
|
||||
| SSO: redirect loop (`ERR_TOO_MANY_REDIRECTS`) | oauth2-proxy `--redirect-url` / callback does not match the public FQDN, or TLS is not terminating at the Gateway listener; fix the OAuth App URLs and the manifest placeholders |
|
||||
| SSO: DSH rejects the public host behind the proxy | Add the public hostname to `DSH_TRUSTED_HOSTS` in `manifests/10-sandboxtemplate.yaml`; the gateway forwards the browser's `Host` |
|
||||
| Pods `Pending` after setting `nodeSelector` | No node matches the label, the pool is full, or a tainted pool needs a toleration; inspect `kubectl describe pod` and the node labels |
|
||||
| gVisor: `kubectl port-forward` fails or the connection is refused | Port-forward is incompatible with gVisor; reach DSH through the gateway (or a reachable Service/Sandbox Router) |
|
||||
| Sandbox cannot reach a model API after applying the optional NetworkPolicy | The policy allows DNS and public TCP `443` only; a proxy, custom base URL, port `80` or non-443 endpoint needs an explicit egress rule |
|
||||
| Gateway cannot reach a sandbox after applying an exposure NetworkPolicy | Check that the CNI enforces NetworkPolicy and that the Traefik/oauth2-proxy pod labels match; kubelet probes may also need the node CIDR |
|
||||
| New provider key not visible in a sandbox | Pod env is stamped at creation: delete the warm-pool spares and recreate claimed pods. A key saved in DSH's credential store also wins over env |
|
||||
|
||||
## Limitations
|
||||
|
||||
@@ -522,13 +597,22 @@ Deliberately **not** production code:
|
||||
- **Log scraping.** The launch token is recovered from pod logs. Log rotation, a
|
||||
restarted pod, or a changed DSH log format can break the handshake until the
|
||||
next `401` refresh.
|
||||
- **Shared provider key.** `DEEPSEEK_API_KEY` is shared by all tenants via the
|
||||
template. Real multi-tenancy needs per-tenant credentials or an egress proxy.
|
||||
- **Shared provider keys.** `DEEPSEEK_API_KEY` and any keys in the optional
|
||||
`dsh-provider-keys-extra` Secret are shared by all tenants via the template.
|
||||
Tenants can override a key through DSH's stored credential file on their PVC,
|
||||
but the platform does not manage that. Real multi-tenancy needs per-tenant
|
||||
credentials or an egress proxy.
|
||||
- **No TLS in the gateway.** Terminate TLS at an Ingress or Gateway and set
|
||||
`COOKIE_SECURE=true`.
|
||||
- **SSO trust is hop-based.** `AUTH_PROXY_HEADER` trusts whoever sets the header;
|
||||
the only guarantee is that the gateway Service is not directly reachable.
|
||||
There is no signature, mTLS, or NetworkPolicy enforcing that in this example.
|
||||
- **SSO trust is hop-based.** `AUTH_PROXY_HEADER` trusts whoever sets the header.
|
||||
The SSO variant ships `75-networkpolicies.yaml`, which restricts the gateway
|
||||
to oauth2-proxy and oauth2-proxy egress to the gateway/DNS/public HTTPS; a
|
||||
custom exposure without that policy has no signature, mTLS or network
|
||||
enforcement.
|
||||
- **Isolation is opt-in.** gVisor, the sandbox NetworkPolicy and the exposure
|
||||
policies are opt-in; without them, sandboxes share the host kernel and can
|
||||
reach the cluster network. The base example is a demo, not a hardened
|
||||
multi-tenant platform.
|
||||
- **OAuth org is the only authorization.** Any GitHub user in the configured
|
||||
organization gets an auto-provisioned sandbox; there is no per-user allow-list,
|
||||
quota, or admin API.
|
||||
@@ -564,3 +648,5 @@ delete the Secrets with the namespace and revoke the OAuth App in GitHub.
|
||||
- [Architecture](docs/architecture.md) — container/sequence diagrams, ADRs,
|
||||
security and RBAC, failure modes, evolution path.
|
||||
- [Exposure README](manifests/exposure/README.md) — the two variants in detail.
|
||||
- [Optional hardening](docs/optional-hardening.md) — gVisor, nodeSelector,
|
||||
NetworkPolicy and extra provider keys, with verification and rollback.
|
||||
|
||||
@@ -67,6 +67,13 @@ as that user. The gateway ignores the header unless `AUTH_PROXY_HEADER` is set.
|
||||
[exposure README](../manifests/exposure/README.md) and the
|
||||
[gateway README](../gateway/README.md#sso-opt-in).
|
||||
|
||||
Each variant also ships a NetworkPolicy: `ingress/61-networkpolicy.yaml` limits gateway ingress
|
||||
to Traefik pods, and `gateway-api-sso/75-networkpolicies.yaml` limits the gateway to
|
||||
oauth2-proxy, oauth2-proxy ingress to the Gateway implementation, and oauth2-proxy egress to
|
||||
DNS/public HTTPS/the gateway. The SSO policy turns the trust boundary into pod-traffic
|
||||
enforcement; the caveats (Traefik label, `hostNetwork`, kubelet probes) are in
|
||||
[optional hardening](optional-hardening.md).
|
||||
|
||||
## Flows
|
||||
|
||||
### Signup (browser)
|
||||
@@ -290,8 +297,14 @@ Notes and limits:
|
||||
- **TLS is external.** The gateway serves plain HTTP; terminate TLS at the Ingress and set
|
||||
`COOKIE_SECURE=true`. Without TLS, session cookies are replayable on the network.
|
||||
- **Tenant isolation** is pod + PVC separation; all tenants share one namespace, one DSH binary,
|
||||
and one `DEEPSEEK_API_KEY`. Add NetworkPolicies and per-tenant credentials before calling it
|
||||
multi-tenant-grade.
|
||||
and the shared provider keys (`dsh-provider-keys` plus the optional
|
||||
`dsh-provider-keys-extra`). `manifests/optional/network-policy/` adds an opt-in sandbox
|
||||
NetworkPolicy, and gVisor/`nodeSelector` are opt-in PodSpec fields. Add per-tenant
|
||||
credentials before calling it multi-tenant-grade.
|
||||
- **Optional hardening is opt-in.** gVisor, node selection, the sandbox/exposure NetworkPolicies
|
||||
and extra provider keys are documented in [optional hardening](optional-hardening.md). The
|
||||
base manifests stay portable (no gVisor RuntimeClass, no active node labels, no
|
||||
NetworkPolicy and no extra Secret required).
|
||||
- **No gateway rate limiting.** Signup can create claims until the cluster or warm pool is
|
||||
exhausted; put limits at the ingress for real deployments.
|
||||
|
||||
@@ -306,6 +319,10 @@ Notes and limits:
|
||||
| DSH auth cookie expired mid-session | Proxy `401` once, then recovers | Gateway refreshes the handshake once |
|
||||
| Upstream pod name changed after resume | Cached cookie keyed by old pod UID is useless | Cache is invalidated on suspend; next request re-handshakes |
|
||||
| Two gateway replicas | Double suspend/wake races | Keep `replicas: 1` until state is externalized |
|
||||
| Optional NetworkPolicy blocks DNS/probes | Sandbox 502/503, pod restarts, failed `getent` | Check the CNI, DNS selector/NodeLocal address and kubelet-probe traffic; see the hardening doc |
|
||||
| gVisor + `kubectl port-forward` | Forward refuses or returns nothing | Port-forward is incompatible with gVisor; use the gateway or a reachable Service |
|
||||
| Provider key added but missing in the pod | Requests still fail with `MISSING_CREDENTIAL` | Pod env is stamped at creation; recreate spares/pods. A stored credential wins over env |
|
||||
| `nodeSelector` matches no node | Pod `Pending`, `0/N nodes are available` | Fix the label, pool capacity or taint/toleration; RuntimeClass scheduling may also constrain it |
|
||||
|
||||
## Upstream feature request: remove log scraping
|
||||
|
||||
|
||||
513
examples/dsh-agents-as-a-service/docs/optional-hardening.md
Normal file
513
examples/dsh-agents-as-a-service/docs/optional-hardening.md
Normal file
@@ -0,0 +1,513 @@
|
||||
# Optional hardening: gVisor, nodeSelector, NetworkPolicy and provider keys
|
||||
|
||||
The base example is deliberately portable: it runs on kind and GKE without a
|
||||
gVisor runtime, a tainted node pool or a NetworkPolicy-capable CNI, and it
|
||||
boots with only the required `dsh-provider-keys` Secret. This document covers
|
||||
four **opt-in** layers:
|
||||
|
||||
| Layer | Where it lives | What it changes |
|
||||
| --- | --- | --- |
|
||||
| gVisor | commented `runtimeClassName` in the SandboxTemplates and Deployments | puts DSH and/or the gateway in a userspace kernel |
|
||||
| nodeSelector | commented `nodeSelector` in the same files | pins pods to a chosen node pool |
|
||||
| NetworkPolicy | `manifests/optional/network-policy/`, `manifests/exposure/*/` | limits sandbox, gateway and proxy connectivity |
|
||||
| Provider keys | optional `dsh-provider-keys-extra` Secret | injects extra provider API keys into every sandbox |
|
||||
|
||||
Each layer is independent. Enable only what the cluster supports and test the
|
||||
walkthrough after each change.
|
||||
|
||||
## 1. gVisor
|
||||
|
||||
### What it changes
|
||||
|
||||
The DSH sandbox and the gateway (and optionally oauth2-proxy) run under the
|
||||
`gvisor` RuntimeClass. `agent-sandbox` passes `runtimeClassName` straight to the
|
||||
PodSpec; no CRD change is involved.
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- A `RuntimeClass` named `gvisor`:
|
||||
|
||||
```bash
|
||||
kubectl get runtimeclass gvisor -o yaml
|
||||
```
|
||||
|
||||
- Nodes capable of running it (`runsc` installed and registered with the
|
||||
container runtime). How you get there is cluster-specific:
|
||||
- GKE Sandbox: create a node pool with `--sandbox=type=gvisor`; GKE labels
|
||||
the nodes `sandbox.gke.io/runtime: gvisor`, taints them
|
||||
`sandbox.gke.io/runtime=gvisor:NoSchedule`, and its `gvisor` RuntimeClass
|
||||
applies the matching affinity and toleration automatically.
|
||||
- Self-managed: install gVisor, create the `gvisor` RuntimeClass with
|
||||
`handler: runsc`, and label/taint a dedicated node pool yourself.
|
||||
- A scheduling plan: a gVisor RuntimeClass without capable nodes makes pods
|
||||
fail to start. Use the RuntimeClass `scheduling` fields or the optional
|
||||
`nodeSelector`/toleration described in section 2.
|
||||
|
||||
### Enable it
|
||||
|
||||
**Fresh install:** uncomment `runtimeClassName: gvisor` before applying:
|
||||
|
||||
- [`manifests/10-sandboxtemplate.yaml`](../manifests/10-sandboxtemplate.yaml)
|
||||
- [`manifests/50-gateway.yaml`](../manifests/50-gateway.yaml)
|
||||
- [`manifests/exposure/gateway-api-sso/72-oauth2-proxy.yaml`](../manifests/exposure/gateway-api-sso/72-oauth2-proxy.yaml)
|
||||
(only with the SSO variant)
|
||||
- [minimal sandbox-template.yaml](../../minimal-single-tenant/manifests/sandbox-template.yaml)
|
||||
|
||||
**Existing install:** a template edit does not update existing Sandboxes, and
|
||||
warm-pool spares use `OnReplenish`, so patch explicitly. Plan a maintenance
|
||||
window: claimed users are disconnected when their pod is recreated.
|
||||
|
||||
```bash
|
||||
# 1. The blueprint for future sandboxes.
|
||||
kubectl -n dsh-demo patch sandboxtemplate dsh-web --type merge \
|
||||
-p '{"spec":{"podTemplate":{"spec":{"runtimeClassName":"gvisor"}}}}'
|
||||
|
||||
# 2. Warm-pool spares: delete them; the pool replenishes from the template.
|
||||
kubectl -n dsh-demo delete sandboxes -l agents.x-k8s.io/warm-pool-sandbox
|
||||
|
||||
# 3. Claimed sandboxes: snapshot their pod template, then suspend/resume.
|
||||
for sb in $(kubectl -n dsh-demo get sandboxes \
|
||||
-l '!agents.x-k8s.io/warm-pool-sandbox' -o jsonpath='{.items[*].metadata.name}'); do
|
||||
kubectl -n dsh-demo patch sandbox "$sb" --type merge \
|
||||
-p '{"spec":{"podTemplate":{"spec":{"runtimeClassName":"gvisor"}}}}'
|
||||
kubectl -n dsh-demo patch sandbox "$sb" --type merge \
|
||||
-p '{"spec":{"operatingMode":"Suspended"}}'
|
||||
kubectl -n dsh-demo patch sandbox "$sb" --type merge \
|
||||
-p '{"spec":{"operatingMode":"Running"}}'
|
||||
done
|
||||
|
||||
# 4. The gateway (and oauth2-proxy when the SSO variant is installed).
|
||||
kubectl -n dsh-demo patch deploy dsh-gateway --type merge \
|
||||
-p '{"spec":{"template":{"spec":{"runtimeClassName":"gvisor"}}}}'
|
||||
kubectl -n dsh-demo rollout status deploy/dsh-gateway --timeout=180s
|
||||
# kubectl -n dsh-demo patch deploy oauth2-proxy --type merge \
|
||||
# -p '{"spec":{"template":{"spec":{"runtimeClassName":"gvisor"}}}}'
|
||||
```
|
||||
|
||||
If the pool's nodes are tainted and the RuntimeClass does not already add a
|
||||
toleration, add the matching `tolerations` block (section 2) in the same patch
|
||||
or in the manifests.
|
||||
|
||||
### Verify
|
||||
|
||||
```bash
|
||||
SB=$(kubectl -n dsh-demo get sandboxclaim dsh-alice -o jsonpath='{.status.sandbox.name}')
|
||||
POD=$(kubectl -n dsh-demo get sandbox "$SB" \
|
||||
-o jsonpath='{.metadata.annotations.agents\.x-k8s\.io/pod-name}'); POD=${POD:-$SB}
|
||||
|
||||
kubectl -n dsh-demo get pod "$POD" -o jsonpath='{.spec.runtimeClassName}{"\n"}'
|
||||
# gvisor
|
||||
kubectl -n dsh-demo exec "$POD" -- uname -r
|
||||
kubectl -n dsh-demo get node "$(kubectl -n dsh-demo get pod "$POD" -o jsonpath='{.spec.nodeName}')" \
|
||||
-o jsonpath='{.status.nodeInfo.kernelVersion}{"\n"}'
|
||||
```
|
||||
|
||||
The pod's `runtimeClassName` is authoritative. The kernel release reported by
|
||||
`uname -r` inside a gVisor pod is the sandbox kernel, not the host kernel; on
|
||||
GKE it looks like `4.19.0-gvisor`.
|
||||
|
||||
### Caveats
|
||||
|
||||
- **`kubectl port-forward` does not work with gVisor.** The multi-tenant
|
||||
gateway path (HTTP/WebSocket to the pod) works; the minimal example's
|
||||
port-forward quickstart does not. Use the gateway example or a reachable
|
||||
Service (NodePort/LoadBalancer/Sandbox Router) instead.
|
||||
- gVisor has performance overhead and does not implement every syscall. Test
|
||||
DSH and the agent tools you rely on (`git`, `gh`, `uv`, language toolchains).
|
||||
- Node scheduling is the hard part. A `gvisor` RuntimeClass on a cluster with
|
||||
no capable nodes leaves pods unschedulable or failing at container start.
|
||||
- Existing Sandboxes are snapshots. The loop above is required for every
|
||||
claimed sandbox; deleting the `SandboxClaim` would also delete its PVC and
|
||||
workspace data, so do not use claim re-creation as a shortcut.
|
||||
- A resumed sandbox starts cold; the startup probe gives it up to 120s.
|
||||
|
||||
### Roll back
|
||||
|
||||
```bash
|
||||
kubectl -n dsh-demo patch sandboxtemplate dsh-web --type merge \
|
||||
-p '{"spec":{"podTemplate":{"spec":{"runtimeClassName":null}}}}'
|
||||
for sb in $(kubectl -n dsh-demo get sandboxes -o jsonpath='{.items[*].metadata.name}'); do
|
||||
kubectl -n dsh-demo patch sandbox "$sb" --type merge \
|
||||
-p '{"spec":{"podTemplate":{"spec":{"runtimeClassName":null}}}}'
|
||||
done
|
||||
kubectl -n dsh-demo delete sandboxes -l agents.x-k8s.io/warm-pool-sandbox
|
||||
kubectl -n dsh-demo patch deploy dsh-gateway --type merge \
|
||||
-p '{"spec":{"template":{"spec":{"runtimeClassName":null}}}}'
|
||||
# kubectl -n dsh-demo patch deploy oauth2-proxy --type merge \
|
||||
# -p '{"spec":{"template":{"spec":{"runtimeClassName":null}}}}'
|
||||
```
|
||||
|
||||
Recreate the pods (delete spares, suspend/resume claimed sandboxes, let the
|
||||
Deployment roll out) so the cleared field takes effect.
|
||||
|
||||
## 2. nodeSelector
|
||||
|
||||
### What it changes
|
||||
|
||||
Pods get a `nodeSelector` so they land only on nodes carrying the chosen label.
|
||||
The same field is available in both SandboxTemplates and both Deployments;
|
||||
the shipped value is **commented out** because active cluster-specific
|
||||
scheduling does not belong in the repository.
|
||||
|
||||
### Choose a label
|
||||
|
||||
```bash
|
||||
kubectl get nodes --show-labels
|
||||
```
|
||||
|
||||
- **GKE Sandbox:** the nodes carry `sandbox.gke.io/runtime: gvisor` and are
|
||||
tainted `sandbox.gke.io/runtime=gvisor:NoSchedule`. With
|
||||
`runtimeClassName: gvisor`, GKE already schedules the pod onto those nodes;
|
||||
add a `nodeSelector` only when you need a specific sandbox pool or
|
||||
ComputeClass (`cloud.google.com/compute-class: <name>`).
|
||||
- **Self-managed:** label the pool you want, for example
|
||||
`dsh.example.com/node-pool: gvisor`, and use that label.
|
||||
- **General pinning:** any stable node-pool label works. Keep the real key and
|
||||
value in a local patch; the manifests use `dsh.example.com/node-pool` as an
|
||||
obvious placeholder.
|
||||
|
||||
### Enable it
|
||||
|
||||
Uncomment and replace the placeholder before a fresh apply:
|
||||
|
||||
```yaml
|
||||
nodeSelector:
|
||||
dsh.example.com/node-pool: gvisor
|
||||
```
|
||||
|
||||
For an existing install, patch the template, each claimed Sandbox and the
|
||||
Deployments, then recreate pods exactly as in section 1:
|
||||
|
||||
```bash
|
||||
kubectl -n dsh-demo patch sandboxtemplate dsh-web --type merge \
|
||||
-p '{"spec":{"podTemplate":{"spec":{"nodeSelector":{"dsh.example.com/node-pool":"gvisor"}}}}}'
|
||||
# ... same warm-pool delete / claimed-sandbox suspend-resume / Deployment
|
||||
# patch and rollout as in section 1, with nodeSelector instead of
|
||||
# runtimeClassName.
|
||||
```
|
||||
|
||||
If the selected nodes are tainted, add a matching toleration. This is the
|
||||
companion block; do **not** uncomment it unless the pool is actually tainted:
|
||||
|
||||
```yaml
|
||||
tolerations:
|
||||
- key: dsh.example.com/node-pool
|
||||
operator: Equal
|
||||
value: gvisor
|
||||
effect: NoSchedule
|
||||
```
|
||||
|
||||
GKE Sandbox's taint is `sandbox.gke.io/runtime=gvisor:NoSchedule`; the GKE
|
||||
`gvisor` RuntimeClass adds that toleration automatically, so a manual copy is
|
||||
only needed when you schedule without the RuntimeClass or with a different
|
||||
pool.
|
||||
|
||||
### Verify and troubleshoot
|
||||
|
||||
```bash
|
||||
kubectl -n dsh-demo get pod "$POD" -o wide
|
||||
kubectl -n dsh-demo get pod "$POD" -o jsonpath='{.spec.nodeSelector}{"\n"}'
|
||||
kubectl -n dsh-demo describe pod "$POD" | sed -n '/Events:/,$p'
|
||||
```
|
||||
|
||||
- `Pending` with `0/N nodes are available`: the label is wrong, the pool is
|
||||
full, or the pod needs a toleration for a taint.
|
||||
- A pod-level `nodeSelector` is merged with the RuntimeClass
|
||||
`scheduling.nodeSelector`. If both are set and conflict, no node matches.
|
||||
- Labels changed after scheduling do not move running pods. Patch/recreate.
|
||||
- A `nodeSelector` that excludes every gVisor node is a valid way to keep the
|
||||
gateway on a normal pool while the sandboxes run under gVisor.
|
||||
|
||||
### Roll back
|
||||
|
||||
Remove the field from the manifests, or patch it to `null`, then recreate the
|
||||
pods:
|
||||
|
||||
```bash
|
||||
kubectl -n dsh-demo patch sandboxtemplate dsh-web --type merge \
|
||||
-p '{"spec":{"podTemplate":{"spec":{"nodeSelector":null}}}}'
|
||||
# ... same per-Sandbox and Deployment loop as section 1.
|
||||
```
|
||||
|
||||
## 3. NetworkPolicy
|
||||
|
||||
The base example does **not** manage NetworkPolicy: the SandboxTemplate is
|
||||
`networkPolicyManagement: Unmanaged`, and the gateway has no policy. That keeps
|
||||
the walkthrough portable and independent of the sandbox-router; the opt-in
|
||||
policies below are standard Kubernetes `NetworkPolicy` objects.
|
||||
|
||||
### Optional sandbox policy
|
||||
|
||||
[`manifests/optional/network-policy/10-sandbox-networkpolicy.yaml`](../manifests/optional/network-policy/10-sandbox-networkpolicy.yaml)
|
||||
selects every Sandbox pod through the agent-sandbox tracking label
|
||||
`agents.x-k8s.io/sandbox-name-hash`. It applies to existing warm-pool and
|
||||
claimed sandboxes without recreating them.
|
||||
|
||||
```bash
|
||||
kubectl apply -f manifests/optional/network-policy/
|
||||
kubectl delete -f manifests/optional/network-policy/ # rollback
|
||||
```
|
||||
|
||||
What it allows:
|
||||
|
||||
| Direction | Allowed | Blocked |
|
||||
| --- | --- | --- |
|
||||
| Ingress | gateway pods (`app: dsh-gateway`) on TCP `3080` | everything else, including other sandboxes and cluster Services |
|
||||
| Egress | UDP/TCP `53` to kube-dns and NodeLocal DNSCache; TCP `443` to public IPs | RFC1918, CGNAT, link-local/metadata, cluster Services/API, plain HTTP, other ports |
|
||||
|
||||
Verify from inside the pods (NetworkPolicy does not affect `kubectl exec`
|
||||
itself, only the traffic the commands generate):
|
||||
|
||||
```bash
|
||||
SB=$(kubectl -n dsh-demo get sandboxclaim dsh-alice -o jsonpath='{.status.sandbox.name}')
|
||||
POD=$(kubectl -n dsh-demo get sandbox "$SB" \
|
||||
-o jsonpath='{.metadata.annotations.agents\.x-k8s\.io/pod-name}'); POD=${POD:-$SB}
|
||||
|
||||
# Allowed: gateway -> sandbox.
|
||||
kubectl -n dsh-demo exec deploy/dsh-gateway -- python -c \
|
||||
"import urllib.request; print(urllib.request.urlopen('http://$POD:3080/favicon.svg', timeout=5).status)"
|
||||
|
||||
# Allowed: DNS and public HTTPS from the sandbox.
|
||||
kubectl -n dsh-demo exec "$POD" -- getent hosts api.deepseek.com
|
||||
kubectl -n dsh-demo exec "$POD" -- curl -fsS -o /dev/null -w '%{http_code}\n' https://example.com
|
||||
|
||||
# Denied: another sandbox cannot reach this one on 3080 (expect a timeout).
|
||||
OTHER_SB=$(kubectl -n dsh-demo get sandboxclaim dsh-bob -o jsonpath='{.status.sandbox.name}')
|
||||
OTHER_POD=$(kubectl -n dsh-demo get sandbox "$OTHER_SB" \
|
||||
-o jsonpath='{.metadata.annotations.agents\.x-k8s\.io/pod-name}'); OTHER_POD=${OTHER_POD:-$OTHER_SB}
|
||||
kubectl -n dsh-demo exec "$OTHER_POD" -- curl --max-time 3 -sS "http://$POD:3080/favicon.svg"
|
||||
```
|
||||
|
||||
Caveats:
|
||||
|
||||
- **The CNI must enforce NetworkPolicy.** If it does not, the file applies but
|
||||
has no effect.
|
||||
- DNS: the default assumes CoreDNS/kube-dns pods labelled `k8s-app: kube-dns`
|
||||
and covers NodeLocal DNSCache at `169.254.20.10`. A different DNS layout
|
||||
needs a selector/`ipBlock` edit.
|
||||
- **Kubelet probes** from the node are CNI-dependent. If a pod starts failing
|
||||
its probes, allow the node CIDR on the probe port or remove the ingress
|
||||
policy.
|
||||
- The policy blocks cluster Services and the Kubernetes API from the sandbox.
|
||||
DSH does not need them, but an agent tool that does needs an explicit rule.
|
||||
- It allows any public host on TCP `443`, not an FQDN allowlist. It is network
|
||||
isolation, not data-loss prevention.
|
||||
- Custom provider base URLs, an HTTP proxy, port `80`, SSH `git` (port `22`) or
|
||||
a non-443 provider endpoint need explicit egress additions.
|
||||
- Do not combine this standalone policy with `networkPolicyManagement: Managed`
|
||||
unless you migrate the rules into `spec.networkPolicy`. NetworkPolicies are
|
||||
additive; the controller's Managed secure default would also be present.
|
||||
- `kubectl port-forward` still works for debugging (the connection is local to
|
||||
the pod), except under gVisor.
|
||||
|
||||
### Exposure-variant policies
|
||||
|
||||
Each exposure variant ships its own policy. They are part of the variant, not a
|
||||
separate opt-in step:
|
||||
|
||||
- [`manifests/exposure/ingress/61-networkpolicy.yaml`](../manifests/exposure/ingress/61-networkpolicy.yaml)
|
||||
allows only Traefik pods (`app.kubernetes.io/name: traefik`) to reach the
|
||||
gateway on `8080`.
|
||||
- [`manifests/exposure/gateway-api-sso/75-networkpolicies.yaml`](../manifests/exposure/gateway-api-sso/75-networkpolicies.yaml)
|
||||
allows only oauth2-proxy to reach the gateway, only the Gateway
|
||||
implementation (Traefik) to reach oauth2-proxy, and limits oauth2-proxy
|
||||
egress to DNS, public HTTPS (GitHub) and the gateway.
|
||||
|
||||
The SSO policy is what turns the trust boundary in the
|
||||
[exposure README](../manifests/exposure/README.md) into network enforcement.
|
||||
The `dsh-gateway` Service must still stay ClusterIP; do not apply both exposure
|
||||
variants.
|
||||
|
||||
Verify the negative case from a sandbox pod (it is not the proxy):
|
||||
|
||||
```bash
|
||||
kubectl -n dsh-demo exec "$POD" -- curl --max-time 3 -sS http://dsh-gateway:8080/healthz
|
||||
# expected: timeout/connection failure
|
||||
```
|
||||
|
||||
Then verify the real browser path through the Ingress or Gateway, and confirm
|
||||
the gateway/proxy pods are `Ready`. If Traefik uses `hostNetwork` or a
|
||||
different pod label, replace the `podSelector` with the correct selector or an
|
||||
`ipBlock`. As with the sandbox policy, kubelet probes can be affected on some
|
||||
CNIs.
|
||||
|
||||
### Gateway egress (documented, not shipped)
|
||||
|
||||
The gateway needs DNS, the Kubernetes API, and TCP `3080` to sandbox pods. The
|
||||
API server endpoint is cluster-specific (Service CIDR or node IP/port), so an
|
||||
egress policy is **not** shipped. If your cluster has a stable API-server CIDR
|
||||
and port, add an egress policy like this (replace the placeholders; verify the
|
||||
port, commonly `443` or `6443`):
|
||||
|
||||
```yaml
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: dsh-gateway-egress
|
||||
namespace: dsh-demo
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app: dsh-gateway
|
||||
policyTypes:
|
||||
- Egress
|
||||
egress:
|
||||
- to:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: kube-system
|
||||
podSelector:
|
||||
matchLabels:
|
||||
k8s-app: kube-dns
|
||||
ports:
|
||||
- protocol: UDP
|
||||
port: 53
|
||||
- protocol: TCP
|
||||
port: 53
|
||||
- to:
|
||||
- ipBlock:
|
||||
cidr: 10.96.0.1/32 # placeholder: API-server Service/cluster endpoint
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 443
|
||||
- to:
|
||||
- podSelector:
|
||||
matchExpressions:
|
||||
- key: agents.x-k8s.io/sandbox-name-hash
|
||||
operator: Exists
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 3080
|
||||
```
|
||||
|
||||
Cilium users can express the API server as `toEntities: kube-apiserver` (or
|
||||
`toServices`) instead of a CIDR. Do not apply the sample blindly: a wrong CIDR
|
||||
or port silently breaks signup/proxy.
|
||||
|
||||
## 4. Optional provider keys
|
||||
|
||||
### How injection works
|
||||
|
||||
The SandboxTemplate injects two Secrets via `envFrom`:
|
||||
|
||||
- `dsh-provider-keys` — required by the walkthrough; holds
|
||||
`DEEPSEEK_API_KEY`.
|
||||
- `dsh-provider-keys-extra` — optional; referenced with `optional: true`, so a
|
||||
missing Secret does not block sandbox startup. Every key in it becomes an
|
||||
environment variable in every sandbox.
|
||||
|
||||
Create the extra Secret before the warm pool (pod env is stamped at creation):
|
||||
|
||||
```bash
|
||||
kubectl -n dsh-demo create secret generic dsh-provider-keys-extra \
|
||||
--from-literal=OPENROUTER_API_KEY="sk-or-..." \
|
||||
--from-literal=GEMINI_API_KEY="..." \
|
||||
--from-literal=NVIDIA_API_KEY="nvapi-..." \
|
||||
--dry-run=client -o yaml | kubectl apply -f -
|
||||
```
|
||||
|
||||
Use the exact env-var names (uppercase, underscores) — invalid Secret keys are
|
||||
skipped by the kubelet with an event. Do not duplicate a key across the two
|
||||
Secrets; when the same key exists twice, the later `envFrom` entry wins.
|
||||
|
||||
### Supported names (curated)
|
||||
|
||||
These are the names DSH v0.2.1-rc.3 resolves through its packaged provider
|
||||
catalog; the catalog has more entries, these are the common ones:
|
||||
|
||||
| Provider | Environment variable |
|
||||
| --- | --- |
|
||||
| DeepSeek | `DEEPSEEK_API_KEY` |
|
||||
| OpenRouter | `OPENROUTER_API_KEY` |
|
||||
| Google Gemini | `GEMINI_API_KEY` |
|
||||
| NVIDIA | `NVIDIA_API_KEY` |
|
||||
| OpenAI | `OPENAI_API_KEY` |
|
||||
| Anthropic | `ANTHROPIC_API_KEY` |
|
||||
| Groq | `GROQ_API_KEY` |
|
||||
| xAI | `XAI_API_KEY` |
|
||||
| Mistral | `MISTRAL_API_KEY` |
|
||||
| Together | `TOGETHER_API_KEY` |
|
||||
| Cerebras | `CEREBRAS_API_KEY` |
|
||||
| Vercel AI Gateway | `AI_GATEWAY_API_KEY` |
|
||||
| Hugging Face | `HF_TOKEN` |
|
||||
|
||||
Aliases to avoid: `GOOGLE_API_KEY`, `HUGGINGFACE_API_KEY` and
|
||||
`VERCEL_AI_GATEWAY_API_KEY`. The packaged catalog looks for `GEMINI_API_KEY`,
|
||||
`HF_TOKEN` and `AI_GATEWAY_API_KEY` instead.
|
||||
|
||||
### Picking up a key change
|
||||
|
||||
A Secret update does not change running pods. The base key and extra keys are
|
||||
both snapshotted at container start:
|
||||
|
||||
```bash
|
||||
# Warm-pool spares: delete them so the pool recreates them from the template.
|
||||
kubectl -n dsh-demo delete sandboxes -l agents.x-k8s.io/warm-pool-sandbox
|
||||
|
||||
# Running claimed sandboxes: delete the pod (the Sandbox controller recreates
|
||||
# it) or suspend/resume the sandbox.
|
||||
POD=... # resolve it as in section 3
|
||||
kubectl -n dsh-demo delete pod "$POD"
|
||||
```
|
||||
|
||||
New claims get the new values automatically. Confirm names without printing
|
||||
secret values:
|
||||
|
||||
```bash
|
||||
kubectl -n dsh-demo exec "$POD" -- sh -c '
|
||||
for k in DEEPSEEK_API_KEY OPENROUTER_API_KEY GEMINI_API_KEY NVIDIA_API_KEY; do
|
||||
if [ -n "$(printenv "$k" || true)" ]; then echo "$k=set"; else echo "$k=unset"; fi
|
||||
done'
|
||||
```
|
||||
|
||||
### Precedence and shared-key limits
|
||||
|
||||
- DSH's stored credential file (`$DSH_HOME/.credentials.yaml`, mode `0600`)
|
||||
**wins over the environment**. If a tenant saved a key through the DSH UI,
|
||||
changing the Secret will not override it.
|
||||
- Both Secrets are shared by **every** tenant, and any agent can read its own
|
||||
environment. This is a warm-pool constraint, not tenant isolation. A tenant
|
||||
can override its own key through the DSH credential store on its PVC, but the
|
||||
platform does not manage that; real multi-tenancy needs per-tenant
|
||||
credentials or an egress proxy.
|
||||
- The optional Secret is created imperatively; the repository never contains a
|
||||
real key. Keep it out of Git and out of shell history where possible.
|
||||
|
||||
## 5. Verification summary
|
||||
|
||||
Required locally for any change to this example:
|
||||
|
||||
```bash
|
||||
make check # versions, example layout, manifest YAML shape
|
||||
yamllint .
|
||||
make test # gateway unit tests (gateway code is untouched)
|
||||
```
|
||||
|
||||
CI additionally parses every manifest and runs `kubectl apply
|
||||
--dry-run=server` against the real agent-sandbox CRDs. The weekly kind e2e
|
||||
runs the base walkthrough; it also proves the optional extra Secret may be
|
||||
absent.
|
||||
|
||||
Manual checks, on a cluster with the relevant CNI/runtime:
|
||||
|
||||
- gVisor: pod `runtimeClassName`, `uname -r` differs from the node kernel,
|
||||
gateway proxy works, port-forward limitation acknowledged.
|
||||
- nodeSelector: pods land on the intended nodes; no `Pending` events.
|
||||
- Sandbox policy: the allow/deny matrix above.
|
||||
- Exposure policies: browser path works; a non-proxy pod cannot reach the
|
||||
gateway; probes stay healthy.
|
||||
- Provider keys: set/unset output (never the values) and the refresh procedure.
|
||||
|
||||
## 6. Rollback summary
|
||||
|
||||
| Layer | Rollback |
|
||||
| --- | --- |
|
||||
| Sandbox policy | `kubectl delete -f manifests/optional/network-policy/` |
|
||||
| Exposure policy | delete the exposure variant directory |
|
||||
| gVisor | patch `runtimeClassName: null`, recreate pods |
|
||||
| nodeSelector | patch `nodeSelector: null`, recreate pods |
|
||||
| Provider keys | delete the extra Secret or a key, recreate pods |
|
||||
|
||||
No PVC or workspace data is touched by any rollback. Deleting a `SandboxClaim`
|
||||
is different: it cascades to the Sandbox **and** its PVC, so do not use claim
|
||||
deletion to apply a pod-spec change.
|
||||
@@ -110,7 +110,9 @@ user (creates their claim idempotently, mints a rotated session token) and
|
||||
redirects to `/u/<user>/`, and `POST /recreate` becomes available for a clean
|
||||
restart. The base manifests leave it unset on purpose; the
|
||||
[SSO exposure variant](../manifests/exposure/gateway-api-sso/) documents how to
|
||||
enable it on the Deployment (`kubectl set env`).
|
||||
enable it on the Deployment (`kubectl set env`). That variant also ships
|
||||
`75-networkpolicies.yaml`, which restricts the gateway to oauth2-proxy and
|
||||
oauth2-proxy's egress to the gateway, DNS and public HTTPS.
|
||||
|
||||
> **Trust boundary — this is the security-critical part.** A trusted identity
|
||||
> header is only safe when **nothing else can reach the gateway**. The gateway
|
||||
@@ -118,6 +120,8 @@ enable it on the Deployment (`kubectl set env`).
|
||||
> authenticating proxy; if a client can reach `dsh-gateway:8080` directly (a
|
||||
> second Ingress/Gateway/Route, a port-forward on a shared host, a broad
|
||||
> NetworkPolicy), it can send `X-Forwarded-User` itself and impersonate any user.
|
||||
> The variant's `75-networkpolicies.yaml` restricts pod traffic to the proxy; a
|
||||
> custom exposure without an equivalent policy has no network enforcement.
|
||||
> Treat `AUTH_PROXY_HEADER` as "the proxy already authenticated this request",
|
||||
> not as an authentication mechanism of its own. Disable SSO by removing the
|
||||
> variable: the header is then ignored and bearer-token/session auth applies
|
||||
@@ -232,5 +236,6 @@ uvicorn's access log so bearer tokens in URLs are not logged.
|
||||
* **SSO trusts a header.** With `AUTH_PROXY_HEADER` set, any client that can
|
||||
reach the gateway Service directly can spoof the identity header. The gateway
|
||||
cannot enforce the trust boundary itself: keep the Service ClusterIP and
|
||||
reachable only through the authenticating proxy. There is no mTLS or signed
|
||||
identity assertion.
|
||||
reachable only through the authenticating proxy. The SSO exposure variant
|
||||
ships a NetworkPolicy that restricts pod traffic to the proxy; without it
|
||||
there is no mTLS or signed identity assertion.
|
||||
|
||||
@@ -1,14 +1,31 @@
|
||||
# DSH "Agents as a Service" — prerequisites: namespace + shared provider key.
|
||||
# DSH "Agents as a Service" — prerequisites: namespace + shared provider keys.
|
||||
#
|
||||
# The dsh-provider-keys Secret is deliberately NOT committed with a usable
|
||||
# value. DEEPSEEK_API_KEY reaches every tenant sandbox (injected via envFrom
|
||||
# on the SandboxTemplate, stamped before any user exists — a warm-pool
|
||||
# constraint), so a real key committed here would be shared by every install
|
||||
# of this example. Generate a fresh value per install (README step 1;
|
||||
# scripts/e2e-kind.sh uses a throwaway):
|
||||
# scripts/e2e-kind.sh uses a throwaway). Use an upsert: this manifest already
|
||||
# creates the empty Secret, so a plain `kubectl create` would fail:
|
||||
#
|
||||
# kubectl -n dsh-demo create secret generic dsh-provider-keys \
|
||||
# --from-literal=DEEPSEEK_API_KEY="sk-..."
|
||||
# --from-literal=DEEPSEEK_API_KEY="sk-..." \
|
||||
# --dry-run=client -o yaml | kubectl apply -f -
|
||||
#
|
||||
# Optional: additional providers (OpenRouter, Gemini, NVIDIA, OpenAI, ...) go
|
||||
# in a second, optional Secret. The SandboxTemplate references it with
|
||||
# `optional: true`, so sandboxes start even when it does not exist, and every
|
||||
# key in it becomes an environment variable. Create it BEFORE the warm pool,
|
||||
# because pod env is stamped at pod creation:
|
||||
#
|
||||
# kubectl -n dsh-demo create secret generic dsh-provider-keys-extra \
|
||||
# --from-literal=OPENROUTER_API_KEY="sk-or-..." \
|
||||
# --from-literal=GEMINI_API_KEY="..." \
|
||||
# --from-literal=NVIDIA_API_KEY="nvapi-..." \
|
||||
# --dry-run=client -o yaml | kubectl apply -f -
|
||||
#
|
||||
# Supported env names, precedence, shared-key caveats and the pod-refresh
|
||||
# procedure are in docs/optional-hardening.md.
|
||||
#
|
||||
# All the infrastructure in this example works without a real key; only
|
||||
# actual model calls need one.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# The reusable DSH Web blueprint. Warm pools stamp pod env BEFORE any user
|
||||
# exists, so per-user secrets cannot live in pod env; the shared provider key
|
||||
# is injected via envFrom, and claims that would bypass the warm pool are
|
||||
# exists, so per-user secrets cannot live in pod env; the shared provider keys
|
||||
# are injected via envFrom, and claims that would bypass the warm pool are
|
||||
# rejected outright (see 40-claim-rejected.yaml).
|
||||
#
|
||||
# A stripped-down single-tenant variant of this template (no warm pool, no
|
||||
@@ -21,16 +21,31 @@ spec:
|
||||
# The gateway routes each user to their DSH Web instance by this name.
|
||||
service: true
|
||||
|
||||
# The example gateway does not manage NetworkPolicy, so it is left
|
||||
# unmanaged here. A production platform would use
|
||||
# networkPolicyManagement: Managed and restrict ingress to its gateway on
|
||||
# :3080 — with the shared provider key, that NetworkPolicy is the per-user
|
||||
# isolation boundary.
|
||||
# The example leaves NetworkPolicy to an optional, opt-in manifest
|
||||
# (manifests/optional/network-policy/): the base walkthrough should not
|
||||
# require a policy-capable CNI, and the controller's Managed secure default
|
||||
# only allows the sandbox-router, which this example does not deploy. If you
|
||||
# switch to Managed, define spec.networkPolicy explicitly or the gateway will
|
||||
# be blocked. See docs/optional-hardening.md.
|
||||
networkPolicyManagement: Unmanaged
|
||||
|
||||
podTemplate:
|
||||
spec:
|
||||
# DSH Web is multi-arch (amd64 + arm64); no arch nodeSelector needed.
|
||||
# DSH Web is multi-arch (amd64 + arm64); no arch nodeSelector is needed.
|
||||
#
|
||||
# OPTIONAL: run the sandbox under gVisor. Requires a RuntimeClass named
|
||||
# "gvisor" and nodes that can run it; gVisor breaks `kubectl port-forward`
|
||||
# into the sandbox (use the gateway). Existing Sandboxes snapshot their
|
||||
# pod template, so enabling it later requires patching/recreating them.
|
||||
# See docs/optional-hardening.md.
|
||||
# runtimeClassName: gvisor
|
||||
#
|
||||
# OPTIONAL: pin sandboxes to a dedicated node pool (for example a gVisor
|
||||
# pool). Replace this placeholder with a label your nodes actually carry;
|
||||
# add a matching toleration if the pool is tainted. Active
|
||||
# cluster-specific scheduling belongs in a local patch, not here.
|
||||
# nodeSelector:
|
||||
# dsh.example.com/node-pool: dsh
|
||||
restartPolicy: "OnFailure"
|
||||
securityContext:
|
||||
runAsUser: 1000 # the 'node' user in the DSH image
|
||||
@@ -57,6 +72,13 @@ spec:
|
||||
envFrom:
|
||||
- secretRef:
|
||||
name: dsh-provider-keys
|
||||
# Optional extra provider keys (OpenRouter, Gemini, NVIDIA, ...).
|
||||
# The Secret may be absent; every key in it becomes an env var.
|
||||
# Create it before the warm pool, because pod env is stamped at
|
||||
# pod creation. See docs/optional-hardening.md.
|
||||
- secretRef:
|
||||
name: dsh-provider-keys-extra
|
||||
optional: true
|
||||
env:
|
||||
# Empty => DSH derives links from the request Host, which the
|
||||
# gateway forwards (plus X-Forwarded-*). Point this at the
|
||||
|
||||
@@ -84,6 +84,15 @@ spec:
|
||||
app: dsh-gateway
|
||||
spec:
|
||||
serviceAccountName: dsh-gateway
|
||||
# OPTIONAL: run the gateway under gVisor. Requires a RuntimeClass named
|
||||
# "gvisor" and nodes that can run it. See docs/optional-hardening.md.
|
||||
# runtimeClassName: gvisor
|
||||
#
|
||||
# OPTIONAL: pin the gateway to a dedicated node pool. Replace the
|
||||
# placeholder with a label your nodes carry; add a toleration for a
|
||||
# tainted pool. Keep active cluster-specific scheduling in a local patch.
|
||||
# nodeSelector:
|
||||
# dsh.example.com/node-pool: dsh
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 10000 # matches USER in gateway/Dockerfile
|
||||
|
||||
@@ -16,6 +16,19 @@ Applying both gives the gateway two entry points, and the SSO variant is only
|
||||
safe when the gateway is reachable **exclusively** through oauth2-proxy. Do not
|
||||
mix them.
|
||||
|
||||
Each variant also ships a NetworkPolicy, applied by the same directory apply:
|
||||
|
||||
- `ingress/61-networkpolicy.yaml` allows only Traefik pods to reach the gateway
|
||||
on `8080`.
|
||||
- `gateway-api-sso/75-networkpolicies.yaml` allows only oauth2-proxy to reach
|
||||
the gateway, only the Gateway implementation (Traefik) to reach oauth2-proxy,
|
||||
and limits oauth2-proxy egress to DNS, public HTTPS (GitHub) and the gateway.
|
||||
|
||||
The policies assume Traefik's `app.kubernetes.io/name: traefik` pod label.
|
||||
Adjust the selector for another implementation or a `hostNetwork` install; see
|
||||
[optional hardening](../../docs/optional-hardening.md) for the probe caveat and
|
||||
verification commands.
|
||||
|
||||
## Applying a variant
|
||||
|
||||
Run these commands from the example root
|
||||
@@ -62,6 +75,9 @@ Secrets imperatively (commands below) before applying the directory.
|
||||
`Secure` cookie is never sent over HTTP, which silently breaks login.
|
||||
4. Do not attach buffering or compression middlewares to this router: they
|
||||
break WebSocket upgrades and streaming.
|
||||
5. `61-networkpolicy.yaml` is part of this variant: it allows only Traefik pods
|
||||
to reach the gateway on `8080`. Adjust its selector for another ingress
|
||||
controller or `hostNetwork`; if kubelet probes fail, see the hardening doc.
|
||||
|
||||
Verify: `curl -sI https://<host>/healthz` returns `200`.
|
||||
|
||||
@@ -70,6 +86,7 @@ Verify: `curl -sI https://<host>/healthz` returns `200`.
|
||||
`70-gateway.yaml` defines the TLS `Gateway`, `71-httproute.yaml` sends every
|
||||
request to oauth2-proxy, and `72-oauth2-proxy.yaml` authenticates the browser
|
||||
against GitHub and proxies to the gateway with `X-Forwarded-User` set.
|
||||
`75-networkpolicies.yaml` makes this the only network path to the gateway.
|
||||
|
||||
Prerequisites:
|
||||
|
||||
@@ -111,7 +128,7 @@ kubectl apply -f manifests/exposure/gateway-api-sso/
|
||||
|
||||
`AUTH_PROXY_HEADER` is opt-in and deliberately unset in the base gateway
|
||||
manifest: the gateway ignores identity headers until it is set. Enabling SSO
|
||||
takes **both** sides:
|
||||
takes all of these:
|
||||
|
||||
1. `AUTH_PROXY_HEADER=X-Forwarded-User` on `deploy/dsh-gateway` (step 3
|
||||
above). Without it, oauth2-proxy's login has no effect on the gateway and
|
||||
@@ -122,6 +139,10 @@ takes **both** sides:
|
||||
other path to the Service) while SSO is enabled. A client that can reach
|
||||
`dsh-gateway:8080` directly can set `X-Forwarded-User` itself and
|
||||
impersonate any user.
|
||||
3. Keep `75-networkpolicies.yaml` with the variant. It enforces the boundary in
|
||||
the network layer (gateway ← oauth2-proxy ← Traefik, and limited oauth2-proxy
|
||||
egress). A custom exposure without an equivalent policy has no network
|
||||
enforcement.
|
||||
|
||||
Also terminate TLS in front of the gateway (the Gateway listener does) and
|
||||
leave `COOKIE_SECURE=true` so the session cookie never travels in clear text.
|
||||
|
||||
@@ -16,6 +16,12 @@
|
||||
# apply ../ingress/ (or anything else that reaches the gateway). A
|
||||
# client that can reach dsh-gateway:8080 can spoof X-Forwarded-User and
|
||||
# impersonate any user.
|
||||
#
|
||||
# 75-networkpolicies.yaml enforces that boundary at the network layer: the
|
||||
# gateway accepts traffic only from oauth2-proxy pods, oauth2-proxy only from
|
||||
# the Gateway implementation, and oauth2-proxy egress is limited to DNS,
|
||||
# public HTTPS and the gateway. Apply it with this variant.
|
||||
#
|
||||
# TLS terminates at the Gateway listener (70-gateway.yaml); keep
|
||||
# --cookie-secure=true and COOKIE_SECURE=true on the gateway.
|
||||
#
|
||||
@@ -42,6 +48,11 @@ spec:
|
||||
app: oauth2-proxy
|
||||
app.kubernetes.io/part-of: dsh-demo
|
||||
spec:
|
||||
# OPTIONAL: run the proxy under gVisor and/or pin it to a node pool.
|
||||
# See ../../../docs/optional-hardening.md.
|
||||
# runtimeClassName: gvisor
|
||||
# nodeSelector:
|
||||
# dsh.example.com/node-pool: dsh
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 2000
|
||||
|
||||
@@ -0,0 +1,120 @@
|
||||
# OPTIONAL with the SSO exposure variant: enforce the trust boundary at the
|
||||
# network layer.
|
||||
#
|
||||
# The chain is Traefik (Gateway API) -> oauth2-proxy -> dsh-gateway. These
|
||||
# policies make it the ONLY chain:
|
||||
# - the gateway accepts traffic only from oauth2-proxy pods;
|
||||
# - oauth2-proxy accepts traffic only from the Gateway implementation;
|
||||
# - oauth2-proxy egress is limited to DNS, public HTTPS (GitHub) and the
|
||||
# gateway.
|
||||
# The gateway's own egress (Kubernetes API, sandbox pods) is intentionally not
|
||||
# restricted here: the API-server endpoint is cluster-specific. See
|
||||
# ../../../docs/optional-hardening.md for that recipe.
|
||||
#
|
||||
# This manifest assumes Traefik, matching gatewayClassName: traefik in
|
||||
# 70-gateway.yaml. Adjust the app.kubernetes.io/name selector for another
|
||||
# implementation. NetworkPolicy enforcement of kubelet probes is CNI-dependent;
|
||||
# if the gateway or proxy fails its probes, allow the node CIDR on the probe
|
||||
# port or remove the ingress policy.
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: dsh-gateway-ingress
|
||||
namespace: dsh-demo
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app: dsh-gateway
|
||||
policyTypes:
|
||||
- Ingress
|
||||
ingress:
|
||||
- from:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app: oauth2-proxy
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 8080
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: oauth2-proxy-ingress
|
||||
namespace: dsh-demo
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app: oauth2-proxy
|
||||
policyTypes:
|
||||
- Ingress
|
||||
ingress:
|
||||
- from:
|
||||
- namespaceSelector: {}
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: traefik
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 4180
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: oauth2-proxy-egress
|
||||
namespace: dsh-demo
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app: oauth2-proxy
|
||||
policyTypes:
|
||||
- Egress
|
||||
egress:
|
||||
# Cluster DNS.
|
||||
- to:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: kube-system
|
||||
podSelector:
|
||||
matchLabels:
|
||||
k8s-app: kube-dns
|
||||
ports:
|
||||
- protocol: UDP
|
||||
port: 53
|
||||
- protocol: TCP
|
||||
port: 53
|
||||
# NodeLocal DNSCache.
|
||||
- to:
|
||||
- ipBlock:
|
||||
cidr: 169.254.20.10/32
|
||||
ports:
|
||||
- protocol: UDP
|
||||
port: 53
|
||||
- protocol: TCP
|
||||
port: 53
|
||||
# The gateway backend.
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app: dsh-gateway
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 8080
|
||||
# GitHub OAuth endpoints are public HTTPS. Private ranges and the metadata
|
||||
# server stay blocked.
|
||||
- to:
|
||||
- ipBlock:
|
||||
cidr: 0.0.0.0/0
|
||||
except:
|
||||
- 10.0.0.0/8
|
||||
- 172.16.0.0/12
|
||||
- 192.168.0.0/16
|
||||
- 169.254.0.0/16
|
||||
- 100.64.0.0/10
|
||||
- ipBlock:
|
||||
cidr: "::/0"
|
||||
except:
|
||||
- "fc00::/7"
|
||||
- "fe80::/10"
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 443
|
||||
@@ -23,6 +23,10 @@
|
||||
# middlewares (e.g. compress, buffering) to this router — they break streaming
|
||||
# and upgrades. `path: /` Prefix covers the landing page and every per-user
|
||||
# proxy path, so no per-sandbox Ingress is needed.
|
||||
#
|
||||
# 61-networkpolicy.yaml is part of this variant: it restricts the gateway's
|
||||
# ingress to Traefik pods. Adjust its selector if your ingress controller uses
|
||||
# a different label or hostNetwork.
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
|
||||
@@ -0,0 +1,31 @@
|
||||
# OPTIONAL with the Ingress exposure variant: restrict the gateway's ingress
|
||||
# to the ingress controller.
|
||||
#
|
||||
# This manifest assumes Traefik, matching this variant's
|
||||
# `ingressClassName: traefik`. Traefik's Helm chart labels its pods
|
||||
# app.kubernetes.io/name=traefik; adjust the selector for another controller or
|
||||
# for a hostNetwork install (where the source is the node IP, not a pod IP).
|
||||
#
|
||||
# NetworkPolicy enforcement of kubelet probe traffic is CNI-dependent. If the
|
||||
# gateway starts failing its /healthz probes, allow the node CIDR on port 8080
|
||||
# or remove this policy. See ../../../docs/optional-hardening.md.
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: dsh-gateway-ingress
|
||||
namespace: dsh-demo
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app: dsh-gateway
|
||||
policyTypes:
|
||||
- Ingress
|
||||
ingress:
|
||||
- from:
|
||||
- namespaceSelector: {}
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: traefik
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 8080
|
||||
@@ -0,0 +1,37 @@
|
||||
# Optional manifests
|
||||
|
||||
Opt-in manifests that are **not** part of the top-level walkthrough.
|
||||
`kubectl apply -f manifests/` is non-recursive, so it does not apply this
|
||||
directory; apply a subdirectory explicitly.
|
||||
|
||||
## network-policy/
|
||||
|
||||
A Kubernetes `NetworkPolicy` for the DSH sandboxes:
|
||||
|
||||
- ingress only from the gateway pods on TCP `3080` (HTTP + WebSocket);
|
||||
- egress only to DNS and public HTTPS (provider APIs, package mirrors,
|
||||
git-over-HTTPS).
|
||||
|
||||
Apply and delete it explicitly:
|
||||
|
||||
```bash
|
||||
kubectl apply -f manifests/optional/network-policy/
|
||||
kubectl delete -f manifests/optional/network-policy/
|
||||
```
|
||||
|
||||
It requires a CNI that enforces `NetworkPolicy`. The selector uses the
|
||||
agent-sandbox controller's pod tracking label, so the policy applies to
|
||||
existing warm-pool and claimed sandboxes without recreation.
|
||||
|
||||
## gVisor, nodeSelector and provider keys
|
||||
|
||||
These are configured in the main manifests or with an extra Secret:
|
||||
|
||||
- **gVisor:** uncomment `runtimeClassName: gvisor` in the SandboxTemplate and
|
||||
the Deployments before applying, or patch an existing install.
|
||||
- **nodeSelector:** uncomment the field and replace the placeholder node label.
|
||||
- **Provider keys:** create the optional `dsh-provider-keys-extra` Secret; the
|
||||
base `dsh-provider-keys` Secret remains the one required key.
|
||||
|
||||
Full prerequisites, exact commands, verification and caveats:
|
||||
[optional hardening](../../docs/optional-hardening.md).
|
||||
@@ -0,0 +1,80 @@
|
||||
# OPTIONAL, opt-in tenant network policy for the DSH sandboxes.
|
||||
#
|
||||
# It is NOT part of the top-level walkthrough. Apply and delete it explicitly:
|
||||
# kubectl apply -f manifests/optional/network-policy/
|
||||
# kubectl delete -f manifests/optional/network-policy/
|
||||
#
|
||||
# The selector uses the agent-sandbox controller's pod tracking label, which is
|
||||
# present on every Sandbox pod (including warm-pool spares), so the policy
|
||||
# applies to existing sandboxes without recreating them. The SandboxTemplate
|
||||
# stays networkPolicyManagement: Unmanaged. Do not also switch it to Managed
|
||||
# unless you migrate these rules into spec.networkPolicy; NetworkPolicies are
|
||||
# additive and the two configurations would be confusing.
|
||||
#
|
||||
# Requires a CNI that enforces NetworkPolicy. See docs/optional-hardening.md
|
||||
# for the connectivity matrix, DNS variants, kubelet-probe caveat and rollback.
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: dsh-sandbox
|
||||
namespace: dsh-demo
|
||||
spec:
|
||||
podSelector:
|
||||
matchExpressions:
|
||||
- key: agents.x-k8s.io/sandbox-name-hash
|
||||
operator: Exists
|
||||
policyTypes:
|
||||
- Ingress
|
||||
- Egress
|
||||
ingress:
|
||||
# The gateway is the only intended client of a sandbox (HTTP + WebSocket).
|
||||
- from:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app: dsh-gateway
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 3080
|
||||
egress:
|
||||
# Cluster DNS. CoreDNS/kube-dns pods carry k8s-app=kube-dns.
|
||||
- to:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: kube-system
|
||||
podSelector:
|
||||
matchLabels:
|
||||
k8s-app: kube-dns
|
||||
ports:
|
||||
- protocol: UDP
|
||||
port: 53
|
||||
- protocol: TCP
|
||||
port: 53
|
||||
# NodeLocal DNSCache, for clusters that do not send DNS to the kube-dns
|
||||
# Service. 169.254.20.10 is its standard link-local address.
|
||||
- to:
|
||||
- ipBlock:
|
||||
cidr: 169.254.20.10/32
|
||||
ports:
|
||||
- protocol: UDP
|
||||
port: 53
|
||||
- protocol: TCP
|
||||
port: 53
|
||||
# Public HTTPS only: provider APIs, package mirrors, git-over-HTTPS.
|
||||
# Private ranges, CGNAT and the metadata server stay blocked.
|
||||
- to:
|
||||
- ipBlock:
|
||||
cidr: 0.0.0.0/0
|
||||
except:
|
||||
- 10.0.0.0/8
|
||||
- 172.16.0.0/12
|
||||
- 192.168.0.0/16
|
||||
- 169.254.0.0/16
|
||||
- 100.64.0.0/10
|
||||
- ipBlock:
|
||||
cidr: "::/0"
|
||||
except:
|
||||
- "fc00::/7"
|
||||
- "fe80::/10"
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 443
|
||||
@@ -49,6 +49,8 @@ kubectl apply -f manifests/00-prereqs.yaml
|
||||
kubectl -n "$NS" delete secret dsh-provider-keys --ignore-not-found >/dev/null
|
||||
kubectl -n "$NS" create secret generic dsh-provider-keys \
|
||||
--from-literal=DEEPSEEK_API_KEY="sk-test-$(openssl rand -hex 16)"
|
||||
# dsh-provider-keys-extra is intentionally absent: the template's optional
|
||||
# envFrom reference must not block the warm pool.
|
||||
kubectl apply -f manifests/10-sandboxtemplate.yaml -f manifests/20-sandboxwarmpool.yaml
|
||||
for _ in $(seq 1 120); do
|
||||
READY=$(kubectl -n "$NS" get sandboxes -l agents.x-k8s.io/warm-pool-sandbox \
|
||||
|
||||
@@ -25,19 +25,28 @@ This example deliberately has no gateway: you reach DSH through a
|
||||
```
|
||||
|
||||
- A DeepSeek API key for real model calls. The pod starts without one; only
|
||||
model requests fail.
|
||||
model requests fail. Optionally create `dsh-provider-keys-extra` for
|
||||
OpenRouter, Gemini, NVIDIA or other provider keys (the template references it
|
||||
with `optional: true`).
|
||||
|
||||
## Quickstart
|
||||
|
||||
Run these from this example's directory (`examples/minimal-single-tenant/`).
|
||||
|
||||
```bash
|
||||
# 1. Namespace + shared provider key. The key is injected into the sandbox via
|
||||
# envFrom and is never committed.
|
||||
# 1. Namespace + shared provider keys. They are injected into the sandbox via
|
||||
# envFrom and are never committed.
|
||||
kubectl create namespace dsh-demo
|
||||
kubectl -n dsh-demo create secret generic dsh-provider-keys \
|
||||
--from-literal=DEEPSEEK_API_KEY="sk-..."
|
||||
|
||||
# Optional: extra providers, injected through a second Secret. Create it before
|
||||
# the warm pool so the pods are stamped with the keys.
|
||||
kubectl -n dsh-demo create secret generic dsh-provider-keys-extra \
|
||||
--from-literal=OPENROUTER_API_KEY="sk-or-..." \
|
||||
--from-literal=GEMINI_API_KEY="..." \
|
||||
--from-literal=NVIDIA_API_KEY="nvapi-..."
|
||||
|
||||
# 2. The template itself (image pinned in the manifest).
|
||||
kubectl apply -f manifests/sandbox-template.yaml
|
||||
|
||||
@@ -86,6 +95,22 @@ kubectl -n dsh-demo port-forward "pod/$SB" 18081:3080
|
||||
> use on your workstation, forward to another local port (here `18081`), never
|
||||
> `3080`. `localhost` is always in DSH's trusted-host list, so the forwarded
|
||||
> request is accepted.
|
||||
>
|
||||
> **gVisor caveat:** if you uncomment `runtimeClassName: gvisor` in the
|
||||
> template, `kubectl port-forward` no longer works. Use the multi-tenant gateway
|
||||
> example or a reachable Service instead; see
|
||||
> [optional hardening](../dsh-agents-as-a-service/docs/optional-hardening.md).
|
||||
|
||||
## Network policy and optional hardening
|
||||
|
||||
The template omits `networkPolicyManagement`, so the agent-sandbox controller
|
||||
uses its Managed secure default: ingress only from the sandbox-router, public
|
||||
egress, and no internal cluster DNS. `kubectl port-forward` is local to the pod
|
||||
and still works (except under gVisor).
|
||||
|
||||
gVisor, an optional `nodeSelector`, extra provider keys and a stricter sandbox
|
||||
NetworkPolicy are documented in
|
||||
[optional hardening](../dsh-agents-as-a-service/docs/optional-hardening.md).
|
||||
|
||||
## Probes and trusted hosts
|
||||
|
||||
|
||||
@@ -10,6 +10,9 @@
|
||||
# - Namespace dsh-demo.
|
||||
# - Secret dsh-provider-keys in dsh-demo with key DEEPSEEK_API_KEY
|
||||
# (injected via envFrom; the pod starts without it, only model calls fail).
|
||||
# - Optional: Secret dsh-provider-keys-extra with additional provider keys
|
||||
# (OPENROUTER_API_KEY, GEMINI_API_KEY, NVIDIA_API_KEY, ...). It is
|
||||
# referenced with `optional: true`, so it may be absent.
|
||||
# The agent-sandbox controller (v1.0.7, API v1beta1) must be installed.
|
||||
apiVersion: extensions.agents.x-k8s.io/v1beta1
|
||||
kind: SandboxTemplate
|
||||
@@ -18,8 +21,22 @@ metadata:
|
||||
namespace: dsh-demo
|
||||
spec:
|
||||
service: true # headless Service per Sandbox -> stable DNS (status.serviceFQDN)
|
||||
# networkPolicyManagement is omitted, so the controller uses its Managed
|
||||
# secure default (ingress only from the sandbox-router, public egress). This
|
||||
# quickstart reaches the sandbox through `kubectl port-forward`, which is
|
||||
# local to the pod. The multi-tenant example documents opt-in policies in
|
||||
# examples/dsh-agents-as-a-service/docs/optional-hardening.md.
|
||||
podTemplate:
|
||||
spec:
|
||||
# OPTIONAL: run under gVisor. gVisor breaks `kubectl port-forward`, so
|
||||
# this quickstart's only access path will not work; use the multi-tenant
|
||||
# gateway example or a Service/NodePort instead.
|
||||
# runtimeClassName: gvisor
|
||||
#
|
||||
# OPTIONAL: pin to a dedicated node pool. Replace the placeholder with a
|
||||
# node label; add a matching toleration if the pool is tainted.
|
||||
# nodeSelector:
|
||||
# dsh.example.com/node-pool: dsh
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 1000 # Matches the UID of the 'node' user
|
||||
@@ -43,6 +60,10 @@ spec:
|
||||
envFrom:
|
||||
- secretRef:
|
||||
name: dsh-provider-keys # Secret with key DEEPSEEK_API_KEY
|
||||
# Optional extra provider keys; an absent Secret is safe.
|
||||
- secretRef:
|
||||
name: dsh-provider-keys-extra
|
||||
optional: true
|
||||
env:
|
||||
# Empty => DSH derives links from the request Host. Keep it empty
|
||||
# when fronting the sandbox with a proxy/gateway.
|
||||
|
||||
@@ -29,6 +29,10 @@ for dir in examples/*/; do
|
||||
err "$name: manifests/exposure/ requires a README.md describing the variants"
|
||||
fi
|
||||
|
||||
if [ -d "${dir}manifests/optional" ] && [ ! -f "${dir}manifests/optional/README.md" ]; then
|
||||
err "$name: manifests/optional/ requires a README.md describing the opt-in manifests"
|
||||
fi
|
||||
|
||||
for component in "$dir"*/; do
|
||||
[ -d "$component" ] || continue
|
||||
case "$(basename "$component")" in
|
||||
|
||||
Reference in New Issue
Block a user