nicolas-georger/deepseek-kubernetes-sandbox
1
0
Fork 0
mirror of https://github.com/sredevopsorg/deepseek-kubernetes-sandbox.git synced 2026-11-03 01:13:24 +00:00
Code Projects Releases Activity

feat(example): add optional gVisor, nodeSelector, NetworkPolicy and provider keys (#21)

Browse Source 
Add opt-in hardening to the agents-as-a-service example:
- commented runtimeClassName/nodeSelector fields for sandboxes and Deployments
- optional sandbox NetworkPolicy plus exposure-variant policies
- optional dsh-provider-keys-extra Secret for extra provider keys
- docs/optional-hardening.md with verification, caveats and rollback

The base walkthrough stays portable; no gateway code, routes or auth changed.

Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
This commit is contained in:
SREDevOps.org Agent
2026-10-26 10:03:30 -03:00
committed by GitHub
parent bf24306421
commit 3e9ba38fc1
28 changed files with 1159 additions and 49 deletions
Download Patch File Download Diff File Expand all files Collapse all files

122
examples/dsh-agents-as-a-service/README.md
View File

@@ -32,10 +32,14 @@ manifests/
+-- 30-claim-alice.yaml # one user
+-- 40-claim-rejected.yaml # negative demo: env injection rejected
+-- 50-gateway.yaml # Deployment/RBAC/Service for the gateway
+-- optional/
| +-- README.md # opt-in manifests, applied by path
| +-- network-policy/10-sandbox-networkpolicy.yaml
+-- exposure/
+-- README.md # the two exposure variants, when to use which
+-- ingress/60-ingress.yaml
+-- gateway-api-sso/70-gateway.yaml ... 74-tls-secret.yaml.example
+-- ingress/61-networkpolicy.yaml
+-- gateway-api-sso/70-gateway.yaml ... 75-networkpolicies.yaml
```
What it demonstrates, feature by feature:
@@ -57,8 +61,10 @@ What it demonstrates, feature by feature:
`kubernetes.io/arch: amd64` nodeSelector.
- **Single port**: DSH Web serves everything on `3080`; the gateway proxies to one
upstream instead of splitting `/v1/*` (API) from the dashboard.
- **One shared secret**: only `DEEPSEEK_API_KEY` is injected (via `envFrom`); there
is no dashboard basic-auth, session-cookie secret, or API-server key.
- **Shared provider keys**: `DEEPSEEK_API_KEY` plus any keys in the optional
`dsh-provider-keys-extra` Secret are injected via `envFrom` into every
sandbox. There is no per-tenant key, dashboard basic-auth, session-cookie
secret, or API-server key.
- **Launch-token handshake**: DSH gates `/` behind a token it prints only in its
logs. The gateway reads it, performs the handshake, and keeps the resulting
auth cookie server-side; the browser authenticates to the gateway with its own
@@ -93,14 +99,39 @@ What it demonstrates, feature by feature:
## 1. Install the platform resources
First the namespace, then the shared provider key. The key reaches **every**
First the namespace, then the shared provider keys. The keys reach **every**
tenant sandbox (a warm-pool constraint: pod env is stamped before any user
exists), so it is **never committed** — generate a fresh value per install:
exists), so they are **never committed** — generate fresh values per install.
`00-prereqs.yaml` already creates the empty `dsh-provider-keys` Secret, so use
an upsert:
```bash
kubectl apply -f manifests/00-prereqs.yaml
kubectl -n dsh-demo create secret generic dsh-provider-keys \
--from-literal=DEEPSEEK_API_KEY="sk-..."
--from-literal=DEEPSEEK_API_KEY="sk-..." \
--dry-run=client -o yaml | kubectl apply -f -
```
Optional: additional providers (OpenRouter, Gemini, NVIDIA, OpenAI, Anthropic,
...) go in a second Secret. It is referenced with `optional: true`, so the
sandboxes start without it, and every key becomes an env var:
```bash
kubectl -n dsh-demo create secret generic dsh-provider-keys-extra \
--from-literal=OPENROUTER_API_KEY="sk-or-..." \
--from-literal=GEMINI_API_KEY="..." \
--from-literal=NVIDIA_API_KEY="nvapi-..." \
--dry-run=client -o yaml | kubectl apply -f -
```
Create both Secrets **before** the template/warm pool. A later Secret change
does not update running pods: delete the warm-pool spares and recreate claimed
pods to pick up new keys. Keys are shared by all tenants; a key saved in DSH's
credential store wins over env. See
[optional hardening](docs/optional-hardening.md) for the full env-name list and
the refresh procedure.
```bash
kubectl apply -f manifests/10-sandboxtemplate.yaml -f manifests/20-sandboxwarmpool.yaml
```
@@ -253,6 +284,12 @@ kubectl -n dsh-demo port-forward svc/dsh-gateway 18080:8080
> apply does not update them: patch each sandbox's
> `spec.podTemplate.spec.securityContext.fsGroupChangePolicy` and
> `chmod 600` its `.credentials.yaml`, or recreate it.
>
> The optional gVisor/`nodeSelector` fields are pod-spec changes too: existing
> Sandboxes snapshot them, and warm-pool spares are not replaced by a template
> edit. See [optional hardening](docs/optional-hardening.md) for the patch and
> recreation commands. A provider-key Secret change likewise needs pod
> recreation to take effect.
### Browser flow
@@ -388,6 +425,13 @@ not a stack. Details, commands and the switch procedure are in the
> `kubectl apply -R`: it would apply both variants and create two paths to the
> same gateway, which is unsafe once SSO is enabled.
Both variants also ship NetworkPolicies: `ingress/61-networkpolicy.yaml` limits
gateway ingress to Traefik pods, and `gateway-api-sso/75-networkpolicies.yaml`
limits the gateway to oauth2-proxy and oauth2-proxy egress to DNS, public HTTPS
and the gateway. They are part of the variant; see the
[exposure README](manifests/exposure/README.md) and
[optional hardening](docs/optional-hardening.md).
### Option A — plain Ingress (gateway's own session auth)
```bash
@@ -404,6 +448,10 @@ kubectl apply -f manifests/exposure/ingress/
(The base manifest keeps `false` so the plain-HTTP port-forward demo works.)
3. Verify: `curl -sI https://<host>/healthz` returns `200`; sign up in a browser
at `https://<host>/`.
4. `manifests/exposure/ingress/61-networkpolicy.yaml` is part of this variant:
it allows only Traefik pods to reach the gateway on `8080`. Adjust its
selector if your ingress controller uses a different pod label or
`hostNetwork`; if kubelet probes fail, see the hardening doc.
Keep the router free of buffering/compression middlewares: they break WebSocket
upgrades and streaming.
@@ -472,13 +520,16 @@ can be cleared at `/oauth2/sign_out`.
> **SSO trust boundary — read this before enabling it.** After SSO is on, the
> `dsh-gateway` Service must be reachable **only** through oauth2-proxy (it is
> ClusterIP). Anyone who can reach `dsh-gateway:8080` directly can send
> `X-Forwarded-User: <victim>` and be treated as that user. Never apply the
> Ingress variant (or any other route to the Service) while SSO is enabled, and
> do not expose the gateway Service with `kubectl port-forward` on a shared
> machine while `AUTH_PROXY_HEADER` is set. To turn SSO off, remove the variable
> (`kubectl -n dsh-demo set env deploy/dsh-gateway AUTH_PROXY_HEADER-`) and
> delete the SSO variant; the gateway falls back to bearer-token/session auth.
> ClusterIP). The variant's `75-networkpolicies.yaml` restricts pod traffic
> accordingly: the gateway accepts traffic only from oauth2-proxy pods,
> oauth2-proxy only from the Gateway implementation, and oauth2-proxy egress is
> limited to DNS, public HTTPS and the gateway. Anyone who can reach
> `dsh-gateway:8080` directly can still send `X-Forwarded-User: <victim>` and be
> treated as that user, so never add a second path (the Ingress variant, a
> shared-host port-forward, a permissive policy). To turn SSO off, remove the
> variable (`kubectl -n dsh-demo set env deploy/dsh-gateway AUTH_PROXY_HEADER-`)
> and delete the SSO variant; the gateway falls back to bearer-token/session
> auth.
### Switching variants
@@ -488,6 +539,25 @@ Delete the variant you had before applying the other one
`kubectl -n dsh-demo get ingress,gateway,httproute` that exactly one path to the
gateway remains.
## Optional hardening
The base walkthrough is deliberately portable. Four opt-in layers are documented
in [optional hardening](docs/optional-hardening.md):
- **gVisor** — uncomment `runtimeClassName: gvisor` before applying, or patch an
existing install. Requires a `gvisor` RuntimeClass and capable nodes.
`kubectl port-forward` does not work under gVisor; the gateway path does.
- **nodeSelector** — uncomment and replace the placeholder node label to pin
sandboxes and the gateway to a dedicated pool; add a toleration for tainted
pools.
- **Sandbox NetworkPolicy** —
`kubectl apply -f manifests/optional/network-policy/` allows only
gateway→sandbox on `3080` and sandbox→DNS/public `443`.
- **Provider keys** — create `dsh-provider-keys-extra` as shown in step 1.
Apply each independently and re-run the walkthrough. `kubectl apply -f manifests/`
is non-recursive, so the optional manifest is not applied by the base step.
## Troubleshooting
| Symptom | Cause / fix |
@@ -511,6 +581,11 @@ gateway remains.
| SSO: oauth2-proxy `cookie_secret must be 16, 24, or 32 bytes ... but is 44 bytes` | The cookie secret was generated as standard base64. Use URL-safe unpadded base64 (see `73-oauth2-proxy-secret.yaml.example`) |
| SSO: redirect loop (`ERR_TOO_MANY_REDIRECTS`) | oauth2-proxy `--redirect-url` / callback does not match the public FQDN, or TLS is not terminating at the Gateway listener; fix the OAuth App URLs and the manifest placeholders |
| SSO: DSH rejects the public host behind the proxy | Add the public hostname to `DSH_TRUSTED_HOSTS` in `manifests/10-sandboxtemplate.yaml`; the gateway forwards the browser's `Host` |
| Pods `Pending` after setting `nodeSelector` | No node matches the label, the pool is full, or a tainted pool needs a toleration; inspect `kubectl describe pod` and the node labels |
| gVisor: `kubectl port-forward` fails or the connection is refused | Port-forward is incompatible with gVisor; reach DSH through the gateway (or a reachable Service/Sandbox Router) |
| Sandbox cannot reach a model API after applying the optional NetworkPolicy | The policy allows DNS and public TCP `443` only; a proxy, custom base URL, port `80` or non-443 endpoint needs an explicit egress rule |
| Gateway cannot reach a sandbox after applying an exposure NetworkPolicy | Check that the CNI enforces NetworkPolicy and that the Traefik/oauth2-proxy pod labels match; kubelet probes may also need the node CIDR |
| New provider key not visible in a sandbox | Pod env is stamped at creation: delete the warm-pool spares and recreate claimed pods. A key saved in DSH's credential store also wins over env |
## Limitations
@@ -522,13 +597,22 @@ Deliberately **not** production code:
- **Log scraping.** The launch token is recovered from pod logs. Log rotation, a
restarted pod, or a changed DSH log format can break the handshake until the
next `401` refresh.
- **Shared provider key.** `DEEPSEEK_API_KEY` is shared by all tenants via the
template. Real multi-tenancy needs per-tenant credentials or an egress proxy.
- **Shared provider keys.** `DEEPSEEK_API_KEY` and any keys in the optional
`dsh-provider-keys-extra` Secret are shared by all tenants via the template.
Tenants can override a key through DSH's stored credential file on their PVC,
but the platform does not manage that. Real multi-tenancy needs per-tenant
credentials or an egress proxy.
- **No TLS in the gateway.** Terminate TLS at an Ingress or Gateway and set
`COOKIE_SECURE=true`.
- **SSO trust is hop-based.** `AUTH_PROXY_HEADER` trusts whoever sets the header;
the only guarantee is that the gateway Service is not directly reachable.
There is no signature, mTLS, or NetworkPolicy enforcing that in this example.
- **SSO trust is hop-based.** `AUTH_PROXY_HEADER` trusts whoever sets the header.
The SSO variant ships `75-networkpolicies.yaml`, which restricts the gateway
to oauth2-proxy and oauth2-proxy egress to the gateway/DNS/public HTTPS; a
custom exposure without that policy has no signature, mTLS or network
enforcement.
- **Isolation is opt-in.** gVisor, the sandbox NetworkPolicy and the exposure
policies are opt-in; without them, sandboxes share the host kernel and can
reach the cluster network. The base example is a demo, not a hardened
multi-tenant platform.
- **OAuth org is the only authorization.** Any GitHub user in the configured
organization gets an auto-provisioned sandbox; there is no per-user allow-list,
quota, or admin API.
@@ -564,3 +648,5 @@ delete the Secrets with the namespace and revoke the OAuth App in GitHub.
- [Architecture](docs/architecture.md) — container/sequence diagrams, ADRs,
security and RBAC, failure modes, evolution path.
- [Exposure README](manifests/exposure/README.md) — the two variants in detail.
- [Optional hardening](docs/optional-hardening.md) — gVisor, nodeSelector,
NetworkPolicy and extra provider keys, with verification and rollback.

21
examples/dsh-agents-as-a-service/docs/architecture.md
View File

@@ -67,6 +67,13 @@ as that user. The gateway ignores the header unless `AUTH_PROXY_HEADER` is set.
[exposure README](../manifests/exposure/README.md) and the
[gateway README](../gateway/README.md#sso-opt-in).
Each variant also ships a NetworkPolicy: `ingress/61-networkpolicy.yaml` limits gateway ingress
to Traefik pods, and `gateway-api-sso/75-networkpolicies.yaml` limits the gateway to
oauth2-proxy, oauth2-proxy ingress to the Gateway implementation, and oauth2-proxy egress to
DNS/public HTTPS/the gateway. The SSO policy turns the trust boundary into pod-traffic
enforcement; the caveats (Traefik label, `hostNetwork`, kubelet probes) are in
[optional hardening](optional-hardening.md).
## Flows
### Signup (browser)
@@ -290,8 +297,14 @@ Notes and limits:
- **TLS is external.** The gateway serves plain HTTP; terminate TLS at the Ingress and set
`COOKIE_SECURE=true`. Without TLS, session cookies are replayable on the network.
- **Tenant isolation** is pod + PVC separation; all tenants share one namespace, one DSH binary,
and one `DEEPSEEK_API_KEY`. Add NetworkPolicies and per-tenant credentials before calling it
multi-tenant-grade.
and the shared provider keys (`dsh-provider-keys` plus the optional
`dsh-provider-keys-extra`). `manifests/optional/network-policy/` adds an opt-in sandbox
NetworkPolicy, and gVisor/`nodeSelector` are opt-in PodSpec fields. Add per-tenant
credentials before calling it multi-tenant-grade.
- **Optional hardening is opt-in.** gVisor, node selection, the sandbox/exposure NetworkPolicies
and extra provider keys are documented in [optional hardening](optional-hardening.md). The
base manifests stay portable (no gVisor RuntimeClass, no active node labels, no
NetworkPolicy and no extra Secret required).
- **No gateway rate limiting.** Signup can create claims until the cluster or warm pool is
exhausted; put limits at the ingress for real deployments.
@@ -306,6 +319,10 @@ Notes and limits:
| DSH auth cookie expired mid-session | Proxy `401` once, then recovers | Gateway refreshes the handshake once |
| Upstream pod name changed after resume | Cached cookie keyed by old pod UID is useless | Cache is invalidated on suspend; next request re-handshakes |
| Two gateway replicas | Double suspend/wake races | Keep `replicas: 1` until state is externalized |
| Optional NetworkPolicy blocks DNS/probes | Sandbox 502/503, pod restarts, failed `getent` | Check the CNI, DNS selector/NodeLocal address and kubelet-probe traffic; see the hardening doc |
| gVisor + `kubectl port-forward` | Forward refuses or returns nothing | Port-forward is incompatible with gVisor; use the gateway or a reachable Service |
| Provider key added but missing in the pod | Requests still fail with `MISSING_CREDENTIAL` | Pod env is stamped at creation; recreate spares/pods. A stored credential wins over env |
| `nodeSelector` matches no node | Pod `Pending`, `0/N nodes are available` | Fix the label, pool capacity or taint/toleration; RuntimeClass scheduling may also constrain it |
## Upstream feature request: remove log scraping

513
examples/dsh-agents-as-a-service/docs/optional-hardening.md Normal file
View File

@@ -0,0 +1,513 @@
# Optional hardening: gVisor, nodeSelector, NetworkPolicy and provider keys
The base example is deliberately portable: it runs on kind and GKE without a
gVisor runtime, a tainted node pool or a NetworkPolicy-capable CNI, and it
boots with only the required `dsh-provider-keys` Secret. This document covers
four **opt-in** layers:
| Layer | Where it lives | What it changes |
| --- | --- | --- |
| gVisor | commented `runtimeClassName` in the SandboxTemplates and Deployments | puts DSH and/or the gateway in a userspace kernel |
| nodeSelector | commented `nodeSelector` in the same files | pins pods to a chosen node pool |
| NetworkPolicy | `manifests/optional/network-policy/`, `manifests/exposure/*/` | limits sandbox, gateway and proxy connectivity |
| Provider keys | optional `dsh-provider-keys-extra` Secret | injects extra provider API keys into every sandbox |
Each layer is independent. Enable only what the cluster supports and test the
walkthrough after each change.
## 1. gVisor
### What it changes
The DSH sandbox and the gateway (and optionally oauth2-proxy) run under the
`gvisor` RuntimeClass. `agent-sandbox` passes `runtimeClassName` straight to the
PodSpec; no CRD change is involved.
### Prerequisites
- A `RuntimeClass` named `gvisor`:
```bash
kubectl get runtimeclass gvisor -o yaml
```
- Nodes capable of running it (`runsc` installed and registered with the
container runtime). How you get there is cluster-specific:
- GKE Sandbox: create a node pool with `--sandbox=type=gvisor`; GKE labels
the nodes `sandbox.gke.io/runtime: gvisor`, taints them
`sandbox.gke.io/runtime=gvisor:NoSchedule`, and its `gvisor` RuntimeClass
applies the matching affinity and toleration automatically.
- Self-managed: install gVisor, create the `gvisor` RuntimeClass with
`handler: runsc`, and label/taint a dedicated node pool yourself.
- A scheduling plan: a gVisor RuntimeClass without capable nodes makes pods
fail to start. Use the RuntimeClass `scheduling` fields or the optional
`nodeSelector`/toleration described in section 2.
### Enable it
**Fresh install:** uncomment `runtimeClassName: gvisor` before applying:
- [`manifests/10-sandboxtemplate.yaml`](../manifests/10-sandboxtemplate.yaml)
- [`manifests/50-gateway.yaml`](../manifests/50-gateway.yaml)
- [`manifests/exposure/gateway-api-sso/72-oauth2-proxy.yaml`](../manifests/exposure/gateway-api-sso/72-oauth2-proxy.yaml)
(only with the SSO variant)
- [minimal sandbox-template.yaml](../../minimal-single-tenant/manifests/sandbox-template.yaml)
**Existing install:** a template edit does not update existing Sandboxes, and
warm-pool spares use `OnReplenish`, so patch explicitly. Plan a maintenance
window: claimed users are disconnected when their pod is recreated.
```bash
# 1. The blueprint for future sandboxes.
kubectl -n dsh-demo patch sandboxtemplate dsh-web --type merge \
-p '{"spec":{"podTemplate":{"spec":{"runtimeClassName":"gvisor"}}}}'
# 2. Warm-pool spares: delete them; the pool replenishes from the template.
kubectl -n dsh-demo delete sandboxes -l agents.x-k8s.io/warm-pool-sandbox
# 3. Claimed sandboxes: snapshot their pod template, then suspend/resume.
for sb in $(kubectl -n dsh-demo get sandboxes \
-l '!agents.x-k8s.io/warm-pool-sandbox' -o jsonpath='{.items[*].metadata.name}'); do
kubectl -n dsh-demo patch sandbox "$sb" --type merge \
-p '{"spec":{"podTemplate":{"spec":{"runtimeClassName":"gvisor"}}}}'
kubectl -n dsh-demo patch sandbox "$sb" --type merge \
-p '{"spec":{"operatingMode":"Suspended"}}'
kubectl -n dsh-demo patch sandbox "$sb" --type merge \
-p '{"spec":{"operatingMode":"Running"}}'
done
# 4. The gateway (and oauth2-proxy when the SSO variant is installed).
kubectl -n dsh-demo patch deploy dsh-gateway --type merge \
-p '{"spec":{"template":{"spec":{"runtimeClassName":"gvisor"}}}}'
kubectl -n dsh-demo rollout status deploy/dsh-gateway --timeout=180s
# kubectl -n dsh-demo patch deploy oauth2-proxy --type merge \
# -p '{"spec":{"template":{"spec":{"runtimeClassName":"gvisor"}}}}'
```
If the pool's nodes are tainted and the RuntimeClass does not already add a
toleration, add the matching `tolerations` block (section 2) in the same patch
or in the manifests.
### Verify
```bash
SB=$(kubectl -n dsh-demo get sandboxclaim dsh-alice -o jsonpath='{.status.sandbox.name}')
POD=$(kubectl -n dsh-demo get sandbox "$SB" \
-o jsonpath='{.metadata.annotations.agents\.x-k8s\.io/pod-name}'); POD=${POD:-$SB}
kubectl -n dsh-demo get pod "$POD" -o jsonpath='{.spec.runtimeClassName}{"\n"}'
# gvisor
kubectl -n dsh-demo exec "$POD" -- uname -r
kubectl -n dsh-demo get node "$(kubectl -n dsh-demo get pod "$POD" -o jsonpath='{.spec.nodeName}')" \
-o jsonpath='{.status.nodeInfo.kernelVersion}{"\n"}'
```
The pod's `runtimeClassName` is authoritative. The kernel release reported by
`uname -r` inside a gVisor pod is the sandbox kernel, not the host kernel; on
GKE it looks like `4.19.0-gvisor`.
### Caveats
- **`kubectl port-forward` does not work with gVisor.** The multi-tenant
gateway path (HTTP/WebSocket to the pod) works; the minimal example's
port-forward quickstart does not. Use the gateway example or a reachable
Service (NodePort/LoadBalancer/Sandbox Router) instead.
- gVisor has performance overhead and does not implement every syscall. Test
DSH and the agent tools you rely on (`git`, `gh`, `uv`, language toolchains).
- Node scheduling is the hard part. A `gvisor` RuntimeClass on a cluster with
no capable nodes leaves pods unschedulable or failing at container start.
- Existing Sandboxes are snapshots. The loop above is required for every
claimed sandbox; deleting the `SandboxClaim` would also delete its PVC and
workspace data, so do not use claim re-creation as a shortcut.
- A resumed sandbox starts cold; the startup probe gives it up to 120s.
### Roll back
```bash
kubectl -n dsh-demo patch sandboxtemplate dsh-web --type merge \
-p '{"spec":{"podTemplate":{"spec":{"runtimeClassName":null}}}}'
for sb in $(kubectl -n dsh-demo get sandboxes -o jsonpath='{.items[*].metadata.name}'); do
kubectl -n dsh-demo patch sandbox "$sb" --type merge \
-p '{"spec":{"podTemplate":{"spec":{"runtimeClassName":null}}}}'
done
kubectl -n dsh-demo delete sandboxes -l agents.x-k8s.io/warm-pool-sandbox
kubectl -n dsh-demo patch deploy dsh-gateway --type merge \
-p '{"spec":{"template":{"spec":{"runtimeClassName":null}}}}'
# kubectl -n dsh-demo patch deploy oauth2-proxy --type merge \
# -p '{"spec":{"template":{"spec":{"runtimeClassName":null}}}}'
```
Recreate the pods (delete spares, suspend/resume claimed sandboxes, let the
Deployment roll out) so the cleared field takes effect.
## 2. nodeSelector
### What it changes
Pods get a `nodeSelector` so they land only on nodes carrying the chosen label.
The same field is available in both SandboxTemplates and both Deployments;
the shipped value is **commented out** because active cluster-specific
scheduling does not belong in the repository.
### Choose a label
```bash
kubectl get nodes --show-labels
```
- **GKE Sandbox:** the nodes carry `sandbox.gke.io/runtime: gvisor` and are
tainted `sandbox.gke.io/runtime=gvisor:NoSchedule`. With
`runtimeClassName: gvisor`, GKE already schedules the pod onto those nodes;
add a `nodeSelector` only when you need a specific sandbox pool or
ComputeClass (`cloud.google.com/compute-class: <name>`).
- **Self-managed:** label the pool you want, for example
`dsh.example.com/node-pool: gvisor`, and use that label.
- **General pinning:** any stable node-pool label works. Keep the real key and
value in a local patch; the manifests use `dsh.example.com/node-pool` as an
obvious placeholder.
### Enable it
Uncomment and replace the placeholder before a fresh apply:
```yaml
nodeSelector:
dsh.example.com/node-pool: gvisor
```
For an existing install, patch the template, each claimed Sandbox and the
Deployments, then recreate pods exactly as in section 1:
```bash
kubectl -n dsh-demo patch sandboxtemplate dsh-web --type merge \
-p '{"spec":{"podTemplate":{"spec":{"nodeSelector":{"dsh.example.com/node-pool":"gvisor"}}}}}'
# ... same warm-pool delete / claimed-sandbox suspend-resume / Deployment
# patch and rollout as in section 1, with nodeSelector instead of
# runtimeClassName.
```
If the selected nodes are tainted, add a matching toleration. This is the
companion block; do **not** uncomment it unless the pool is actually tainted:
```yaml
tolerations:
- key: dsh.example.com/node-pool
operator: Equal
value: gvisor
effect: NoSchedule
```
GKE Sandbox's taint is `sandbox.gke.io/runtime=gvisor:NoSchedule`; the GKE
`gvisor` RuntimeClass adds that toleration automatically, so a manual copy is
only needed when you schedule without the RuntimeClass or with a different
pool.
### Verify and troubleshoot
```bash
kubectl -n dsh-demo get pod "$POD" -o wide
kubectl -n dsh-demo get pod "$POD" -o jsonpath='{.spec.nodeSelector}{"\n"}'
kubectl -n dsh-demo describe pod "$POD" | sed -n '/Events:/,$p'
```
- `Pending` with `0/N nodes are available`: the label is wrong, the pool is
full, or the pod needs a toleration for a taint.
- A pod-level `nodeSelector` is merged with the RuntimeClass
`scheduling.nodeSelector`. If both are set and conflict, no node matches.
- Labels changed after scheduling do not move running pods. Patch/recreate.
- A `nodeSelector` that excludes every gVisor node is a valid way to keep the
gateway on a normal pool while the sandboxes run under gVisor.
### Roll back
Remove the field from the manifests, or patch it to `null`, then recreate the
pods:
```bash
kubectl -n dsh-demo patch sandboxtemplate dsh-web --type merge \
-p '{"spec":{"podTemplate":{"spec":{"nodeSelector":null}}}}'
# ... same per-Sandbox and Deployment loop as section 1.
```
## 3. NetworkPolicy
The base example does **not** manage NetworkPolicy: the SandboxTemplate is
`networkPolicyManagement: Unmanaged`, and the gateway has no policy. That keeps
the walkthrough portable and independent of the sandbox-router; the opt-in
policies below are standard Kubernetes `NetworkPolicy` objects.
### Optional sandbox policy
[`manifests/optional/network-policy/10-sandbox-networkpolicy.yaml`](../manifests/optional/network-policy/10-sandbox-networkpolicy.yaml)
selects every Sandbox pod through the agent-sandbox tracking label
`agents.x-k8s.io/sandbox-name-hash`. It applies to existing warm-pool and
claimed sandboxes without recreating them.
```bash
kubectl apply -f manifests/optional/network-policy/
kubectl delete -f manifests/optional/network-policy/ # rollback
```
What it allows:
| Direction | Allowed | Blocked |
| --- | --- | --- |
| Ingress | gateway pods (`app: dsh-gateway`) on TCP `3080` | everything else, including other sandboxes and cluster Services |
| Egress | UDP/TCP `53` to kube-dns and NodeLocal DNSCache; TCP `443` to public IPs | RFC1918, CGNAT, link-local/metadata, cluster Services/API, plain HTTP, other ports |
Verify from inside the pods (NetworkPolicy does not affect `kubectl exec`
itself, only the traffic the commands generate):
```bash
SB=$(kubectl -n dsh-demo get sandboxclaim dsh-alice -o jsonpath='{.status.sandbox.name}')
POD=$(kubectl -n dsh-demo get sandbox "$SB" \
-o jsonpath='{.metadata.annotations.agents\.x-k8s\.io/pod-name}'); POD=${POD:-$SB}
# Allowed: gateway -> sandbox.
kubectl -n dsh-demo exec deploy/dsh-gateway -- python -c \
"import urllib.request; print(urllib.request.urlopen('http://$POD:3080/favicon.svg', timeout=5).status)"
# Allowed: DNS and public HTTPS from the sandbox.
kubectl -n dsh-demo exec "$POD" -- getent hosts api.deepseek.com
kubectl -n dsh-demo exec "$POD" -- curl -fsS -o /dev/null -w '%{http_code}\n' https://example.com
# Denied: another sandbox cannot reach this one on 3080 (expect a timeout).
OTHER_SB=$(kubectl -n dsh-demo get sandboxclaim dsh-bob -o jsonpath='{.status.sandbox.name}')
OTHER_POD=$(kubectl -n dsh-demo get sandbox "$OTHER_SB" \
-o jsonpath='{.metadata.annotations.agents\.x-k8s\.io/pod-name}'); OTHER_POD=${OTHER_POD:-$OTHER_SB}
kubectl -n dsh-demo exec "$OTHER_POD" -- curl --max-time 3 -sS "http://$POD:3080/favicon.svg"
```
Caveats:
- **The CNI must enforce NetworkPolicy.** If it does not, the file applies but
has no effect.
- DNS: the default assumes CoreDNS/kube-dns pods labelled `k8s-app: kube-dns`
and covers NodeLocal DNSCache at `169.254.20.10`. A different DNS layout
needs a selector/`ipBlock` edit.
- **Kubelet probes** from the node are CNI-dependent. If a pod starts failing
its probes, allow the node CIDR on the probe port or remove the ingress
policy.
- The policy blocks cluster Services and the Kubernetes API from the sandbox.
DSH does not need them, but an agent tool that does needs an explicit rule.
- It allows any public host on TCP `443`, not an FQDN allowlist. It is network
isolation, not data-loss prevention.
- Custom provider base URLs, an HTTP proxy, port `80`, SSH `git` (port `22`) or
a non-443 provider endpoint need explicit egress additions.
- Do not combine this standalone policy with `networkPolicyManagement: Managed`
unless you migrate the rules into `spec.networkPolicy`. NetworkPolicies are
additive; the controller's Managed secure default would also be present.
- `kubectl port-forward` still works for debugging (the connection is local to
the pod), except under gVisor.
### Exposure-variant policies
Each exposure variant ships its own policy. They are part of the variant, not a
separate opt-in step:
- [`manifests/exposure/ingress/61-networkpolicy.yaml`](../manifests/exposure/ingress/61-networkpolicy.yaml)
allows only Traefik pods (`app.kubernetes.io/name: traefik`) to reach the
gateway on `8080`.
- [`manifests/exposure/gateway-api-sso/75-networkpolicies.yaml`](../manifests/exposure/gateway-api-sso/75-networkpolicies.yaml)
allows only oauth2-proxy to reach the gateway, only the Gateway
implementation (Traefik) to reach oauth2-proxy, and limits oauth2-proxy
egress to DNS, public HTTPS (GitHub) and the gateway.
The SSO policy is what turns the trust boundary in the
[exposure README](../manifests/exposure/README.md) into network enforcement.
The `dsh-gateway` Service must still stay ClusterIP; do not apply both exposure
variants.
Verify the negative case from a sandbox pod (it is not the proxy):
```bash
kubectl -n dsh-demo exec "$POD" -- curl --max-time 3 -sS http://dsh-gateway:8080/healthz
# expected: timeout/connection failure
```
Then verify the real browser path through the Ingress or Gateway, and confirm
the gateway/proxy pods are `Ready`. If Traefik uses `hostNetwork` or a
different pod label, replace the `podSelector` with the correct selector or an
`ipBlock`. As with the sandbox policy, kubelet probes can be affected on some
CNIs.
### Gateway egress (documented, not shipped)
The gateway needs DNS, the Kubernetes API, and TCP `3080` to sandbox pods. The
API server endpoint is cluster-specific (Service CIDR or node IP/port), so an
egress policy is **not** shipped. If your cluster has a stable API-server CIDR
and port, add an egress policy like this (replace the placeholders; verify the
port, commonly `443` or `6443`):
```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: dsh-gateway-egress
namespace: dsh-demo
spec:
podSelector:
matchLabels:
app: dsh-gateway
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
- to:
- ipBlock:
cidr: 10.96.0.1/32 # placeholder: API-server Service/cluster endpoint
ports:
- protocol: TCP
port: 443
- to:
- podSelector:
matchExpressions:
- key: agents.x-k8s.io/sandbox-name-hash
operator: Exists
ports:
- protocol: TCP
port: 3080
```
Cilium users can express the API server as `toEntities: kube-apiserver` (or
`toServices`) instead of a CIDR. Do not apply the sample blindly: a wrong CIDR
or port silently breaks signup/proxy.
## 4. Optional provider keys
### How injection works
The SandboxTemplate injects two Secrets via `envFrom`:
- `dsh-provider-keys` — required by the walkthrough; holds
`DEEPSEEK_API_KEY`.
- `dsh-provider-keys-extra` — optional; referenced with `optional: true`, so a
missing Secret does not block sandbox startup. Every key in it becomes an
environment variable in every sandbox.
Create the extra Secret before the warm pool (pod env is stamped at creation):
```bash
kubectl -n dsh-demo create secret generic dsh-provider-keys-extra \
--from-literal=OPENROUTER_API_KEY="sk-or-..." \
--from-literal=GEMINI_API_KEY="..." \
--from-literal=NVIDIA_API_KEY="nvapi-..." \
--dry-run=client -o yaml | kubectl apply -f -
```
Use the exact env-var names (uppercase, underscores) — invalid Secret keys are
skipped by the kubelet with an event. Do not duplicate a key across the two
Secrets; when the same key exists twice, the later `envFrom` entry wins.
### Supported names (curated)
These are the names DSH v0.2.1-rc.3 resolves through its packaged provider
catalog; the catalog has more entries, these are the common ones:
| Provider | Environment variable |
| --- | --- |
| DeepSeek | `DEEPSEEK_API_KEY` |
| OpenRouter | `OPENROUTER_API_KEY` |
| Google Gemini | `GEMINI_API_KEY` |
| NVIDIA | `NVIDIA_API_KEY` |
| OpenAI | `OPENAI_API_KEY` |
| Anthropic | `ANTHROPIC_API_KEY` |
| Groq | `GROQ_API_KEY` |
| xAI | `XAI_API_KEY` |
| Mistral | `MISTRAL_API_KEY` |
| Together | `TOGETHER_API_KEY` |
| Cerebras | `CEREBRAS_API_KEY` |
| Vercel AI Gateway | `AI_GATEWAY_API_KEY` |
| Hugging Face | `HF_TOKEN` |
Aliases to avoid: `GOOGLE_API_KEY`, `HUGGINGFACE_API_KEY` and
`VERCEL_AI_GATEWAY_API_KEY`. The packaged catalog looks for `GEMINI_API_KEY`,
`HF_TOKEN` and `AI_GATEWAY_API_KEY` instead.
### Picking up a key change
A Secret update does not change running pods. The base key and extra keys are
both snapshotted at container start:
```bash
# Warm-pool spares: delete them so the pool recreates them from the template.
kubectl -n dsh-demo delete sandboxes -l agents.x-k8s.io/warm-pool-sandbox
# Running claimed sandboxes: delete the pod (the Sandbox controller recreates
# it) or suspend/resume the sandbox.
POD=... # resolve it as in section 3
kubectl -n dsh-demo delete pod "$POD"
```
New claims get the new values automatically. Confirm names without printing
secret values:
```bash
kubectl -n dsh-demo exec "$POD" -- sh -c '
for k in DEEPSEEK_API_KEY OPENROUTER_API_KEY GEMINI_API_KEY NVIDIA_API_KEY; do
if [ -n "$(printenv "$k" || true)" ]; then echo "$k=set"; else echo "$k=unset"; fi
done'
```
### Precedence and shared-key limits
- DSH's stored credential file (`$DSH_HOME/.credentials.yaml`, mode `0600`)
**wins over the environment**. If a tenant saved a key through the DSH UI,
changing the Secret will not override it.
- Both Secrets are shared by **every** tenant, and any agent can read its own
environment. This is a warm-pool constraint, not tenant isolation. A tenant
can override its own key through the DSH credential store on its PVC, but the
platform does not manage that; real multi-tenancy needs per-tenant
credentials or an egress proxy.
- The optional Secret is created imperatively; the repository never contains a
real key. Keep it out of Git and out of shell history where possible.
## 5. Verification summary
Required locally for any change to this example:
```bash
make check # versions, example layout, manifest YAML shape
yamllint .
make test # gateway unit tests (gateway code is untouched)
```
CI additionally parses every manifest and runs `kubectl apply
--dry-run=server` against the real agent-sandbox CRDs. The weekly kind e2e
runs the base walkthrough; it also proves the optional extra Secret may be
absent.
Manual checks, on a cluster with the relevant CNI/runtime:
- gVisor: pod `runtimeClassName`, `uname -r` differs from the node kernel,
gateway proxy works, port-forward limitation acknowledged.
- nodeSelector: pods land on the intended nodes; no `Pending` events.
- Sandbox policy: the allow/deny matrix above.
- Exposure policies: browser path works; a non-proxy pod cannot reach the
gateway; probes stay healthy.
- Provider keys: set/unset output (never the values) and the refresh procedure.
## 6. Rollback summary
| Layer | Rollback |
| --- | --- |
| Sandbox policy | `kubectl delete -f manifests/optional/network-policy/` |
| Exposure policy | delete the exposure variant directory |
| gVisor | patch `runtimeClassName: null`, recreate pods |
| nodeSelector | patch `nodeSelector: null`, recreate pods |
| Provider keys | delete the extra Secret or a key, recreate pods |
No PVC or workspace data is touched by any rollback. Deleting a `SandboxClaim`
is different: it cascades to the Sandbox **and** its PVC, so do not use claim
deletion to apply a pod-spec change.

11
examples/dsh-agents-as-a-service/gateway/README.md
View File

@@ -110,7 +110,9 @@ user (creates their claim idempotently, mints a rotated session token) and
redirects to `/u/<user>/`, and `POST /recreate` becomes available for a clean
restart. The base manifests leave it unset on purpose; the
[SSO exposure variant](../manifests/exposure/gateway-api-sso/) documents how to
enable it on the Deployment (`kubectl set env`).
enable it on the Deployment (`kubectl set env`). That variant also ships
`75-networkpolicies.yaml`, which restricts the gateway to oauth2-proxy and
oauth2-proxy's egress to the gateway, DNS and public HTTPS.
> **Trust boundary — this is the security-critical part.** A trusted identity
> header is only safe when **nothing else can reach the gateway**. The gateway
@@ -118,6 +120,8 @@ enable it on the Deployment (`kubectl set env`).
> authenticating proxy; if a client can reach `dsh-gateway:8080` directly (a
> second Ingress/Gateway/Route, a port-forward on a shared host, a broad
> NetworkPolicy), it can send `X-Forwarded-User` itself and impersonate any user.
> The variant's `75-networkpolicies.yaml` restricts pod traffic to the proxy; a
> custom exposure without an equivalent policy has no network enforcement.
> Treat `AUTH_PROXY_HEADER` as "the proxy already authenticated this request",
> not as an authentication mechanism of its own. Disable SSO by removing the
> variable: the header is then ignored and bearer-token/session auth applies
@@ -232,5 +236,6 @@ uvicorn's access log so bearer tokens in URLs are not logged.
* **SSO trusts a header.** With `AUTH_PROXY_HEADER` set, any client that can
reach the gateway Service directly can spoof the identity header. The gateway
cannot enforce the trust boundary itself: keep the Service ClusterIP and
reachable only through the authenticating proxy. There is no mTLS or signed
identity assertion.
reachable only through the authenticating proxy. The SSO exposure variant
ships a NetworkPolicy that restricts pod traffic to the proxy; without it
there is no mTLS or signed identity assertion.

23
examples/dsh-agents-as-a-service/manifests/00-prereqs.yaml
View File

@@ -1,14 +1,31 @@
# DSH "Agents as a Service" — prerequisites: namespace + shared provider key.
# DSH "Agents as a Service" — prerequisites: namespace + shared provider keys.
#
# The dsh-provider-keys Secret is deliberately NOT committed with a usable
# value. DEEPSEEK_API_KEY reaches every tenant sandbox (injected via envFrom
# on the SandboxTemplate, stamped before any user exists — a warm-pool
# constraint), so a real key committed here would be shared by every install
# of this example. Generate a fresh value per install (README step 1;
# scripts/e2e-kind.sh uses a throwaway):
# scripts/e2e-kind.sh uses a throwaway). Use an upsert: this manifest already
# creates the empty Secret, so a plain `kubectl create` would fail:
#
# kubectl -n dsh-demo create secret generic dsh-provider-keys \
# --from-literal=DEEPSEEK_API_KEY="sk-..."
# --from-literal=DEEPSEEK_API_KEY="sk-..." \
# --dry-run=client -o yaml | kubectl apply -f -
#
# Optional: additional providers (OpenRouter, Gemini, NVIDIA, OpenAI, ...) go
# in a second, optional Secret. The SandboxTemplate references it with
# `optional: true`, so sandboxes start even when it does not exist, and every
# key in it becomes an environment variable. Create it BEFORE the warm pool,
# because pod env is stamped at pod creation:
#
# kubectl -n dsh-demo create secret generic dsh-provider-keys-extra \
# --from-literal=OPENROUTER_API_KEY="sk-or-..." \
# --from-literal=GEMINI_API_KEY="..." \
# --from-literal=NVIDIA_API_KEY="nvapi-..." \
# --dry-run=client -o yaml | kubectl apply -f -
#
# Supported env names, precedence, shared-key caveats and the pod-refresh
# procedure are in docs/optional-hardening.md.
#
# All the infrastructure in this example works without a real key; only
# actual model calls need one.

38
examples/dsh-agents-as-a-service/manifests/10-sandboxtemplate.yaml
View File

@@ -1,6 +1,6 @@
# The reusable DSH Web blueprint. Warm pools stamp pod env BEFORE any user
# exists, so per-user secrets cannot live in pod env; the shared provider key
# is injected via envFrom, and claims that would bypass the warm pool are
# exists, so per-user secrets cannot live in pod env; the shared provider keys
# are injected via envFrom, and claims that would bypass the warm pool are
# rejected outright (see 40-claim-rejected.yaml).
#
# A stripped-down single-tenant variant of this template (no warm pool, no
@@ -21,16 +21,31 @@ spec:
# The gateway routes each user to their DSH Web instance by this name.
service: true
# The example gateway does not manage NetworkPolicy, so it is left
# unmanaged here. A production platform would use
# networkPolicyManagement: Managed and restrict ingress to its gateway on
# :3080 — with the shared provider key, that NetworkPolicy is the per-user
# isolation boundary.
# The example leaves NetworkPolicy to an optional, opt-in manifest
# (manifests/optional/network-policy/): the base walkthrough should not
# require a policy-capable CNI, and the controller's Managed secure default
# only allows the sandbox-router, which this example does not deploy. If you
# switch to Managed, define spec.networkPolicy explicitly or the gateway will
# be blocked. See docs/optional-hardening.md.
networkPolicyManagement: Unmanaged
podTemplate:
spec:
# DSH Web is multi-arch (amd64 + arm64); no arch nodeSelector needed.
# DSH Web is multi-arch (amd64 + arm64); no arch nodeSelector is needed.
#
# OPTIONAL: run the sandbox under gVisor. Requires a RuntimeClass named
# "gvisor" and nodes that can run it; gVisor breaks `kubectl port-forward`
# into the sandbox (use the gateway). Existing Sandboxes snapshot their
# pod template, so enabling it later requires patching/recreating them.
# See docs/optional-hardening.md.
# runtimeClassName: gvisor
#
# OPTIONAL: pin sandboxes to a dedicated node pool (for example a gVisor
# pool). Replace this placeholder with a label your nodes actually carry;
# add a matching toleration if the pool is tainted. Active
# cluster-specific scheduling belongs in a local patch, not here.
# nodeSelector:
# dsh.example.com/node-pool: dsh
restartPolicy: "OnFailure"
securityContext:
runAsUser: 1000 # the 'node' user in the DSH image
@@ -57,6 +72,13 @@ spec:
envFrom:
- secretRef:
name: dsh-provider-keys
# Optional extra provider keys (OpenRouter, Gemini, NVIDIA, ...).
# The Secret may be absent; every key in it becomes an env var.
# Create it before the warm pool, because pod env is stamped at
# pod creation. See docs/optional-hardening.md.
- secretRef:
name: dsh-provider-keys-extra
optional: true
env:
# Empty => DSH derives links from the request Host, which the
# gateway forwards (plus X-Forwarded-*). Point this at the

9
examples/dsh-agents-as-a-service/manifests/50-gateway.yaml
View File

@@ -84,6 +84,15 @@ spec:
app: dsh-gateway
spec:
serviceAccountName: dsh-gateway
# OPTIONAL: run the gateway under gVisor. Requires a RuntimeClass named
# "gvisor" and nodes that can run it. See docs/optional-hardening.md.
# runtimeClassName: gvisor
#
# OPTIONAL: pin the gateway to a dedicated node pool. Replace the
# placeholder with a label your nodes carry; add a toleration for a
# tainted pool. Keep active cluster-specific scheduling in a local patch.
# nodeSelector:
# dsh.example.com/node-pool: dsh
securityContext:
runAsNonRoot: true
runAsUser: 10000 # matches USER in gateway/Dockerfile

23
examples/dsh-agents-as-a-service/manifests/exposure/README.md
View File

@@ -16,6 +16,19 @@ Applying both gives the gateway two entry points, and the SSO variant is only
safe when the gateway is reachable **exclusively** through oauth2-proxy. Do not
mix them.
Each variant also ships a NetworkPolicy, applied by the same directory apply:
- `ingress/61-networkpolicy.yaml` allows only Traefik pods to reach the gateway
on `8080`.
- `gateway-api-sso/75-networkpolicies.yaml` allows only oauth2-proxy to reach
the gateway, only the Gateway implementation (Traefik) to reach oauth2-proxy,
and limits oauth2-proxy egress to DNS, public HTTPS (GitHub) and the gateway.
The policies assume Traefik's `app.kubernetes.io/name: traefik` pod label.
Adjust the selector for another implementation or a `hostNetwork` install; see
[optional hardening](../../docs/optional-hardening.md) for the probe caveat and
verification commands.
## Applying a variant
Run these commands from the example root
@@ -62,6 +75,9 @@ Secrets imperatively (commands below) before applying the directory.
`Secure` cookie is never sent over HTTP, which silently breaks login.
4. Do not attach buffering or compression middlewares to this router: they
break WebSocket upgrades and streaming.
5. `61-networkpolicy.yaml` is part of this variant: it allows only Traefik pods
to reach the gateway on `8080`. Adjust its selector for another ingress
controller or `hostNetwork`; if kubelet probes fail, see the hardening doc.
Verify: `curl -sI https://<host>/healthz` returns `200`.
@@ -70,6 +86,7 @@ Verify: `curl -sI https://<host>/healthz` returns `200`.
`70-gateway.yaml` defines the TLS `Gateway`, `71-httproute.yaml` sends every
request to oauth2-proxy, and `72-oauth2-proxy.yaml` authenticates the browser
against GitHub and proxies to the gateway with `X-Forwarded-User` set.
`75-networkpolicies.yaml` makes this the only network path to the gateway.
Prerequisites:
@@ -111,7 +128,7 @@ kubectl apply -f manifests/exposure/gateway-api-sso/
`AUTH_PROXY_HEADER` is opt-in and deliberately unset in the base gateway
manifest: the gateway ignores identity headers until it is set. Enabling SSO
takes **both** sides:
takes all of these:
1. `AUTH_PROXY_HEADER=X-Forwarded-User` on `deploy/dsh-gateway` (step 3
above). Without it, oauth2-proxy's login has no effect on the gateway and
@@ -122,6 +139,10 @@ takes **both** sides:
other path to the Service) while SSO is enabled. A client that can reach
`dsh-gateway:8080` directly can set `X-Forwarded-User` itself and
impersonate any user.
3. Keep `75-networkpolicies.yaml` with the variant. It enforces the boundary in
the network layer (gateway ← oauth2-proxy ← Traefik, and limited oauth2-proxy
egress). A custom exposure without an equivalent policy has no network
enforcement.
Also terminate TLS in front of the gateway (the Gateway listener does) and
leave `COOKIE_SECURE=true` so the session cookie never travels in clear text.

11
examples/dsh-agents-as-a-service/manifests/exposure/gateway-api-sso/72-oauth2-proxy.yaml
View File

@@ -16,6 +16,12 @@
# apply ../ingress/ (or anything else that reaches the gateway). A
# client that can reach dsh-gateway:8080 can spoof X-Forwarded-User and
# impersonate any user.
#
# 75-networkpolicies.yaml enforces that boundary at the network layer: the
# gateway accepts traffic only from oauth2-proxy pods, oauth2-proxy only from
# the Gateway implementation, and oauth2-proxy egress is limited to DNS,
# public HTTPS and the gateway. Apply it with this variant.
#
# TLS terminates at the Gateway listener (70-gateway.yaml); keep
# --cookie-secure=true and COOKIE_SECURE=true on the gateway.
#
@@ -42,6 +48,11 @@ spec:
app: oauth2-proxy
app.kubernetes.io/part-of: dsh-demo
spec:
# OPTIONAL: run the proxy under gVisor and/or pin it to a node pool.
# See ../../../docs/optional-hardening.md.
# runtimeClassName: gvisor
# nodeSelector:
# dsh.example.com/node-pool: dsh
securityContext:
runAsNonRoot: true
runAsUser: 2000

120
examples/dsh-agents-as-a-service/manifests/exposure/gateway-api-sso/75-networkpolicies.yaml Normal file
View File

@@ -0,0 +1,120 @@
# OPTIONAL with the SSO exposure variant: enforce the trust boundary at the
# network layer.
#
# The chain is Traefik (Gateway API) -> oauth2-proxy -> dsh-gateway. These
# policies make it the ONLY chain:
# - the gateway accepts traffic only from oauth2-proxy pods;
# - oauth2-proxy accepts traffic only from the Gateway implementation;
# - oauth2-proxy egress is limited to DNS, public HTTPS (GitHub) and the
# gateway.
# The gateway's own egress (Kubernetes API, sandbox pods) is intentionally not
# restricted here: the API-server endpoint is cluster-specific. See
# ../../../docs/optional-hardening.md for that recipe.
#
# This manifest assumes Traefik, matching gatewayClassName: traefik in
# 70-gateway.yaml. Adjust the app.kubernetes.io/name selector for another
# implementation. NetworkPolicy enforcement of kubelet probes is CNI-dependent;
# if the gateway or proxy fails its probes, allow the node CIDR on the probe
# port or remove the ingress policy.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: dsh-gateway-ingress
namespace: dsh-demo
spec:
podSelector:
matchLabels:
app: dsh-gateway
policyTypes:
- Ingress
ingress:
- from:
- podSelector:
matchLabels:
app: oauth2-proxy
ports:
- protocol: TCP
port: 8080
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: oauth2-proxy-ingress
namespace: dsh-demo
spec:
podSelector:
matchLabels:
app: oauth2-proxy
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector: {}
podSelector:
matchLabels:
app.kubernetes.io/name: traefik
ports:
- protocol: TCP
port: 4180
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: oauth2-proxy-egress
namespace: dsh-demo
spec:
podSelector:
matchLabels:
app: oauth2-proxy
policyTypes:
- Egress
egress:
# Cluster DNS.
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
# NodeLocal DNSCache.
- to:
- ipBlock:
cidr: 169.254.20.10/32
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
# The gateway backend.
- to:
- podSelector:
matchLabels:
app: dsh-gateway
ports:
- protocol: TCP
port: 8080
# GitHub OAuth endpoints are public HTTPS. Private ranges and the metadata
# server stay blocked.
- to:
- ipBlock:
cidr: 0.0.0.0/0
except:
- 10.0.0.0/8
- 172.16.0.0/12
- 192.168.0.0/16
- 169.254.0.0/16
- 100.64.0.0/10
- ipBlock:
cidr: "::/0"
except:
- "fc00::/7"
- "fe80::/10"
ports:
- protocol: TCP
port: 443

4
examples/dsh-agents-as-a-service/manifests/exposure/ingress/60-ingress.yaml
View File

@@ -23,6 +23,10 @@
# middlewares (e.g. compress, buffering) to this router — they break streaming
# and upgrades. `path: /` Prefix covers the landing page and every per-user
# proxy path, so no per-sandbox Ingress is needed.
#
# 61-networkpolicy.yaml is part of this variant: it restricts the gateway's
# ingress to Traefik pods. Adjust its selector if your ingress controller uses
# a different label or hostNetwork.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:

31
examples/dsh-agents-as-a-service/manifests/exposure/ingress/61-networkpolicy.yaml Normal file
View File

@@ -0,0 +1,31 @@
# OPTIONAL with the Ingress exposure variant: restrict the gateway's ingress
# to the ingress controller.
#
# This manifest assumes Traefik, matching this variant's
# `ingressClassName: traefik`. Traefik's Helm chart labels its pods
# app.kubernetes.io/name=traefik; adjust the selector for another controller or
# for a hostNetwork install (where the source is the node IP, not a pod IP).
#
# NetworkPolicy enforcement of kubelet probe traffic is CNI-dependent. If the
# gateway starts failing its /healthz probes, allow the node CIDR on port 8080
# or remove this policy. See ../../../docs/optional-hardening.md.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: dsh-gateway-ingress
namespace: dsh-demo
spec:
podSelector:
matchLabels:
app: dsh-gateway
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector: {}
podSelector:
matchLabels:
app.kubernetes.io/name: traefik
ports:
- protocol: TCP
port: 8080

37
examples/dsh-agents-as-a-service/manifests/optional/README.md Normal file
View File

@@ -0,0 +1,37 @@
# Optional manifests
Opt-in manifests that are **not** part of the top-level walkthrough.
`kubectl apply -f manifests/` is non-recursive, so it does not apply this
directory; apply a subdirectory explicitly.
## network-policy/
A Kubernetes `NetworkPolicy` for the DSH sandboxes:
- ingress only from the gateway pods on TCP `3080` (HTTP + WebSocket);
- egress only to DNS and public HTTPS (provider APIs, package mirrors,
git-over-HTTPS).
Apply and delete it explicitly:
```bash
kubectl apply -f manifests/optional/network-policy/
kubectl delete -f manifests/optional/network-policy/
```
It requires a CNI that enforces `NetworkPolicy`. The selector uses the
agent-sandbox controller's pod tracking label, so the policy applies to
existing warm-pool and claimed sandboxes without recreation.
## gVisor, nodeSelector and provider keys
These are configured in the main manifests or with an extra Secret:
- **gVisor:** uncomment `runtimeClassName: gvisor` in the SandboxTemplate and
the Deployments before applying, or patch an existing install.
- **nodeSelector:** uncomment the field and replace the placeholder node label.
- **Provider keys:** create the optional `dsh-provider-keys-extra` Secret; the
base `dsh-provider-keys` Secret remains the one required key.
Full prerequisites, exact commands, verification and caveats:
[optional hardening](../../docs/optional-hardening.md).

80
examples/dsh-agents-as-a-service/manifests/optional/network-policy/10-sandbox-networkpolicy.yaml Normal file
View File

@@ -0,0 +1,80 @@
# OPTIONAL, opt-in tenant network policy for the DSH sandboxes.
#
# It is NOT part of the top-level walkthrough. Apply and delete it explicitly:
# kubectl apply -f manifests/optional/network-policy/
# kubectl delete -f manifests/optional/network-policy/
#
# The selector uses the agent-sandbox controller's pod tracking label, which is
# present on every Sandbox pod (including warm-pool spares), so the policy
# applies to existing sandboxes without recreating them. The SandboxTemplate
# stays networkPolicyManagement: Unmanaged. Do not also switch it to Managed
# unless you migrate these rules into spec.networkPolicy; NetworkPolicies are
# additive and the two configurations would be confusing.
#
# Requires a CNI that enforces NetworkPolicy. See docs/optional-hardening.md
# for the connectivity matrix, DNS variants, kubelet-probe caveat and rollback.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: dsh-sandbox
namespace: dsh-demo
spec:
podSelector:
matchExpressions:
- key: agents.x-k8s.io/sandbox-name-hash
operator: Exists
policyTypes:
- Ingress
- Egress
ingress:
# The gateway is the only intended client of a sandbox (HTTP + WebSocket).
- from:
- podSelector:
matchLabels:
app: dsh-gateway
ports:
- protocol: TCP
port: 3080
egress:
# Cluster DNS. CoreDNS/kube-dns pods carry k8s-app=kube-dns.
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
# NodeLocal DNSCache, for clusters that do not send DNS to the kube-dns
# Service. 169.254.20.10 is its standard link-local address.
- to:
- ipBlock:
cidr: 169.254.20.10/32
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
# Public HTTPS only: provider APIs, package mirrors, git-over-HTTPS.
# Private ranges, CGNAT and the metadata server stay blocked.
- to:
- ipBlock:
cidr: 0.0.0.0/0
except:
- 10.0.0.0/8
- 172.16.0.0/12
- 192.168.0.0/16
- 169.254.0.0/16
- 100.64.0.0/10
- ipBlock:
cidr: "::/0"
except:
- "fc00::/7"
- "fe80::/10"
ports:
- protocol: TCP
port: 443

2
examples/dsh-agents-as-a-service/scripts/e2e-kind.sh
View File

@@ -49,6 +49,8 @@ kubectl apply -f manifests/00-prereqs.yaml
kubectl -n "$NS" delete secret dsh-provider-keys --ignore-not-found >/dev/null
kubectl -n "$NS" create secret generic dsh-provider-keys \
--from-literal=DEEPSEEK_API_KEY="sk-test-$(openssl rand -hex 16)"
# dsh-provider-keys-extra is intentionally absent: the template's optional
# envFrom reference must not block the warm pool.
kubectl apply -f manifests/10-sandboxtemplate.yaml -f manifests/20-sandboxwarmpool.yaml
for _ in $(seq 1 120); do
READY=$(kubectl -n "$NS" get sandboxes -l agents.x-k8s.io/warm-pool-sandbox \