feat(example): add optional gVisor, nodeSelector, NetworkPolicy and provider keys (#21)

Add opt-in hardening to the agents-as-a-service example:
- commented runtimeClassName/nodeSelector fields for sandboxes and Deployments
- optional sandbox NetworkPolicy plus exposure-variant policies
- optional dsh-provider-keys-extra Secret for extra provider keys
- docs/optional-hardening.md with verification, caveats and rollback

The base walkthrough stays portable; no gateway code, routes or auth changed.

Co-authored-by: SREDevOpsAgents <agents@sredevops.org>

examples/dsh-agents-as-a-service/manifests/00-prereqs.yaml

@@ -1,14 +1,31 @@
-# DSH "Agents as a Service" — prerequisites: namespace + shared provider key.
+# DSH "Agents as a Service" — prerequisites: namespace + shared provider keys.
 #
 # The dsh-provider-keys Secret is deliberately NOT committed with a usable
 # value. DEEPSEEK_API_KEY reaches every tenant sandbox (injected via envFrom
 # on the SandboxTemplate, stamped before any user exists — a warm-pool
 # constraint), so a real key committed here would be shared by every install
 # of this example. Generate a fresh value per install (README step 1;
-# scripts/e2e-kind.sh uses a throwaway):
+# scripts/e2e-kind.sh uses a throwaway). Use an upsert: this manifest already
+# creates the empty Secret, so a plain `kubectl create` would fail:
 #
 #   kubectl -n dsh-demo create secret generic dsh-provider-keys \
-#     --from-literal=DEEPSEEK_API_KEY="sk-..."
+#     --from-literal=DEEPSEEK_API_KEY="sk-..." \
+#     --dry-run=client -o yaml | kubectl apply -f -
+#
+# Optional: additional providers (OpenRouter, Gemini, NVIDIA, OpenAI, ...) go
+# in a second, optional Secret. The SandboxTemplate references it with
+# `optional: true`, so sandboxes start even when it does not exist, and every
+# key in it becomes an environment variable. Create it BEFORE the warm pool,
+# because pod env is stamped at pod creation:
+#
+#   kubectl -n dsh-demo create secret generic dsh-provider-keys-extra \
+#     --from-literal=OPENROUTER_API_KEY="sk-or-..." \
+#     --from-literal=GEMINI_API_KEY="..." \
+#     --from-literal=NVIDIA_API_KEY="nvapi-..." \
+#     --dry-run=client -o yaml | kubectl apply -f -
+#
+# Supported env names, precedence, shared-key caveats and the pod-refresh
+# procedure are in docs/optional-hardening.md.
 #
 # All the infrastructure in this example works without a real key; only
 # actual model calls need one.

examples/dsh-agents-as-a-service/manifests/10-sandboxtemplate.yaml

@@ -1,6 +1,6 @@
 # The reusable DSH Web blueprint. Warm pools stamp pod env BEFORE any user
-# exists, so per-user secrets cannot live in pod env; the shared provider key
-# is injected via envFrom, and claims that would bypass the warm pool are
+# exists, so per-user secrets cannot live in pod env; the shared provider keys
+# are injected via envFrom, and claims that would bypass the warm pool are
 # rejected outright (see 40-claim-rejected.yaml).
 #
 # A stripped-down single-tenant variant of this template (no warm pool, no
@@ -21,16 +21,31 @@ spec:
   # The gateway routes each user to their DSH Web instance by this name.
   service: true
 
-  # The example gateway does not manage NetworkPolicy, so it is left
-  # unmanaged here. A production platform would use
-  # networkPolicyManagement: Managed and restrict ingress to its gateway on
-  # :3080 — with the shared provider key, that NetworkPolicy is the per-user
-  # isolation boundary.
+  # The example leaves NetworkPolicy to an optional, opt-in manifest
+  # (manifests/optional/network-policy/): the base walkthrough should not
+  # require a policy-capable CNI, and the controller's Managed secure default
+  # only allows the sandbox-router, which this example does not deploy. If you
+  # switch to Managed, define spec.networkPolicy explicitly or the gateway will
+  # be blocked. See docs/optional-hardening.md.
   networkPolicyManagement: Unmanaged
 
   podTemplate:
     spec:
-      # DSH Web is multi-arch (amd64 + arm64); no arch nodeSelector needed.
+      # DSH Web is multi-arch (amd64 + arm64); no arch nodeSelector is needed.
+      #
+      # OPTIONAL: run the sandbox under gVisor. Requires a RuntimeClass named
+      # "gvisor" and nodes that can run it; gVisor breaks `kubectl port-forward`
+      # into the sandbox (use the gateway). Existing Sandboxes snapshot their
+      # pod template, so enabling it later requires patching/recreating them.
+      # See docs/optional-hardening.md.
+      # runtimeClassName: gvisor
+      #
+      # OPTIONAL: pin sandboxes to a dedicated node pool (for example a gVisor
+      # pool). Replace this placeholder with a label your nodes actually carry;
+      # add a matching toleration if the pool is tainted. Active
+      # cluster-specific scheduling belongs in a local patch, not here.
+      # nodeSelector:
+      #   dsh.example.com/node-pool: dsh
       restartPolicy: "OnFailure"
       securityContext:
         runAsUser: 1000   # the 'node' user in the DSH image
@@ -57,6 +72,13 @@ spec:
           envFrom:
             - secretRef:
                 name: dsh-provider-keys
+            # Optional extra provider keys (OpenRouter, Gemini, NVIDIA, ...).
+            # The Secret may be absent; every key in it becomes an env var.
+            # Create it before the warm pool, because pod env is stamped at
+            # pod creation. See docs/optional-hardening.md.
+            - secretRef:
+                name: dsh-provider-keys-extra
+                optional: true
           env:
             # Empty => DSH derives links from the request Host, which the
             # gateway forwards (plus X-Forwarded-*). Point this at the

examples/dsh-agents-as-a-service/manifests/50-gateway.yaml

@@ -84,6 +84,15 @@ spec:
         app: dsh-gateway
     spec:
       serviceAccountName: dsh-gateway
+      # OPTIONAL: run the gateway under gVisor. Requires a RuntimeClass named
+      # "gvisor" and nodes that can run it. See docs/optional-hardening.md.
+      # runtimeClassName: gvisor
+      #
+      # OPTIONAL: pin the gateway to a dedicated node pool. Replace the
+      # placeholder with a label your nodes carry; add a toleration for a
+      # tainted pool. Keep active cluster-specific scheduling in a local patch.
+      # nodeSelector:
+      #   dsh.example.com/node-pool: dsh
       securityContext:
         runAsNonRoot: true
         runAsUser: 10000 # matches USER in gateway/Dockerfile

examples/dsh-agents-as-a-service/manifests/exposure/README.md

@@ -16,6 +16,19 @@ Applying both gives the gateway two entry points, and the SSO variant is only
 safe when the gateway is reachable **exclusively** through oauth2-proxy. Do not
 mix them.
 
+Each variant also ships a NetworkPolicy, applied by the same directory apply:
+
+- `ingress/61-networkpolicy.yaml` allows only Traefik pods to reach the gateway
+  on `8080`.
+- `gateway-api-sso/75-networkpolicies.yaml` allows only oauth2-proxy to reach
+  the gateway, only the Gateway implementation (Traefik) to reach oauth2-proxy,
+  and limits oauth2-proxy egress to DNS, public HTTPS (GitHub) and the gateway.
+
+The policies assume Traefik's `app.kubernetes.io/name: traefik` pod label.
+Adjust the selector for another implementation or a `hostNetwork` install; see
+[optional hardening](../../docs/optional-hardening.md) for the probe caveat and
+verification commands.
+
 ## Applying a variant
 
 Run these commands from the example root
@@ -62,6 +75,9 @@ Secrets imperatively (commands below) before applying the directory.
    `Secure` cookie is never sent over HTTP, which silently breaks login.
 4. Do not attach buffering or compression middlewares to this router: they
    break WebSocket upgrades and streaming.
+5. `61-networkpolicy.yaml` is part of this variant: it allows only Traefik pods
+   to reach the gateway on `8080`. Adjust its selector for another ingress
+   controller or `hostNetwork`; if kubelet probes fail, see the hardening doc.
 
 Verify: `curl -sI https://<host>/healthz` returns `200`.
 
@@ -70,6 +86,7 @@ Verify: `curl -sI https://<host>/healthz` returns `200`.
 `70-gateway.yaml` defines the TLS `Gateway`, `71-httproute.yaml` sends every
 request to oauth2-proxy, and `72-oauth2-proxy.yaml` authenticates the browser
 against GitHub and proxies to the gateway with `X-Forwarded-User` set.
+`75-networkpolicies.yaml` makes this the only network path to the gateway.
 
 Prerequisites:
 
@@ -111,7 +128,7 @@ kubectl apply -f manifests/exposure/gateway-api-sso/
 
 `AUTH_PROXY_HEADER` is opt-in and deliberately unset in the base gateway
 manifest: the gateway ignores identity headers until it is set. Enabling SSO
-takes **both** sides:
+takes all of these:
 
 1. `AUTH_PROXY_HEADER=X-Forwarded-User` on `deploy/dsh-gateway` (step 3
    above). Without it, oauth2-proxy's login has no effect on the gateway and
@@ -122,6 +139,10 @@ takes **both** sides:
    other path to the Service) while SSO is enabled. A client that can reach
    `dsh-gateway:8080` directly can set `X-Forwarded-User` itself and
    impersonate any user.
+3. Keep `75-networkpolicies.yaml` with the variant. It enforces the boundary in
+   the network layer (gateway ← oauth2-proxy ← Traefik, and limited oauth2-proxy
+   egress). A custom exposure without an equivalent policy has no network
+   enforcement.
 
 Also terminate TLS in front of the gateway (the Gateway listener does) and
 leave `COOKIE_SECURE=true` so the session cookie never travels in clear text.

examples/dsh-agents-as-a-service/manifests/exposure/gateway-api-sso/72-oauth2-proxy.yaml

@@ -16,6 +16,12 @@
 #      apply ../ingress/ (or anything else that reaches the gateway). A
 #      client that can reach dsh-gateway:8080 can spoof X-Forwarded-User and
 #      impersonate any user.
+#
+# 75-networkpolicies.yaml enforces that boundary at the network layer: the
+# gateway accepts traffic only from oauth2-proxy pods, oauth2-proxy only from
+# the Gateway implementation, and oauth2-proxy egress is limited to DNS,
+# public HTTPS and the gateway. Apply it with this variant.
+#
 # TLS terminates at the Gateway listener (70-gateway.yaml); keep
 # --cookie-secure=true and COOKIE_SECURE=true on the gateway.
 #
@@ -42,6 +48,11 @@ spec:
         app: oauth2-proxy
         app.kubernetes.io/part-of: dsh-demo
     spec:
+      # OPTIONAL: run the proxy under gVisor and/or pin it to a node pool.
+      # See ../../../docs/optional-hardening.md.
+      # runtimeClassName: gvisor
+      # nodeSelector:
+      #   dsh.example.com/node-pool: dsh
       securityContext:
         runAsNonRoot: true
         runAsUser: 2000

examples/dsh-agents-as-a-service/manifests/exposure/gateway-api-sso/75-networkpolicies.yaml

@@ -0,0 +1,120 @@
+# OPTIONAL with the SSO exposure variant: enforce the trust boundary at the
+# network layer.
+#
+# The chain is Traefik (Gateway API) -> oauth2-proxy -> dsh-gateway. These
+# policies make it the ONLY chain:
+#   - the gateway accepts traffic only from oauth2-proxy pods;
+#   - oauth2-proxy accepts traffic only from the Gateway implementation;
+#   - oauth2-proxy egress is limited to DNS, public HTTPS (GitHub) and the
+#     gateway.
+# The gateway's own egress (Kubernetes API, sandbox pods) is intentionally not
+# restricted here: the API-server endpoint is cluster-specific. See
+# ../../../docs/optional-hardening.md for that recipe.
+#
+# This manifest assumes Traefik, matching gatewayClassName: traefik in
+# 70-gateway.yaml. Adjust the app.kubernetes.io/name selector for another
+# implementation. NetworkPolicy enforcement of kubelet probes is CNI-dependent;
+# if the gateway or proxy fails its probes, allow the node CIDR on the probe
+# port or remove the ingress policy.
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: dsh-gateway-ingress
+  namespace: dsh-demo
+spec:
+  podSelector:
+    matchLabels:
+      app: dsh-gateway
+  policyTypes:
+    - Ingress
+  ingress:
+    - from:
+        - podSelector:
+            matchLabels:
+              app: oauth2-proxy
+      ports:
+        - protocol: TCP
+          port: 8080
+---
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: oauth2-proxy-ingress
+  namespace: dsh-demo
+spec:
+  podSelector:
+    matchLabels:
+      app: oauth2-proxy
+  policyTypes:
+    - Ingress
+  ingress:
+    - from:
+        - namespaceSelector: {}
+          podSelector:
+            matchLabels:
+              app.kubernetes.io/name: traefik
+      ports:
+        - protocol: TCP
+          port: 4180
+---
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: oauth2-proxy-egress
+  namespace: dsh-demo
+spec:
+  podSelector:
+    matchLabels:
+      app: oauth2-proxy
+  policyTypes:
+    - Egress
+  egress:
+    # Cluster DNS.
+    - to:
+        - namespaceSelector:
+            matchLabels:
+              kubernetes.io/metadata.name: kube-system
+          podSelector:
+            matchLabels:
+              k8s-app: kube-dns
+      ports:
+        - protocol: UDP
+          port: 53
+        - protocol: TCP
+          port: 53
+    # NodeLocal DNSCache.
+    - to:
+        - ipBlock:
+            cidr: 169.254.20.10/32
+      ports:
+        - protocol: UDP
+          port: 53
+        - protocol: TCP
+          port: 53
+    # The gateway backend.
+    - to:
+        - podSelector:
+            matchLabels:
+              app: dsh-gateway
+      ports:
+        - protocol: TCP
+          port: 8080
+    # GitHub OAuth endpoints are public HTTPS. Private ranges and the metadata
+    # server stay blocked.
+    - to:
+        - ipBlock:
+            cidr: 0.0.0.0/0
+            except:
+              - 10.0.0.0/8
+              - 172.16.0.0/12
+              - 192.168.0.0/16
+              - 169.254.0.0/16
+              - 100.64.0.0/10
+        - ipBlock:
+            cidr: "::/0"
+            except:
+              - "fc00::/7"
+              - "fe80::/10"
+      ports:
+        - protocol: TCP
+          port: 443

examples/dsh-agents-as-a-service/manifests/exposure/ingress/60-ingress.yaml

@@ -23,6 +23,10 @@
 # middlewares (e.g. compress, buffering) to this router — they break streaming
 # and upgrades. `path: /` Prefix covers the landing page and every per-user
 # proxy path, so no per-sandbox Ingress is needed.
+#
+# 61-networkpolicy.yaml is part of this variant: it restricts the gateway's
+# ingress to Traefik pods. Adjust its selector if your ingress controller uses
+# a different label or hostNetwork.
 apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:

examples/dsh-agents-as-a-service/manifests/exposure/ingress/61-networkpolicy.yaml

@@ -0,0 +1,31 @@
+# OPTIONAL with the Ingress exposure variant: restrict the gateway's ingress
+# to the ingress controller.
+#
+# This manifest assumes Traefik, matching this variant's
+# `ingressClassName: traefik`. Traefik's Helm chart labels its pods
+# app.kubernetes.io/name=traefik; adjust the selector for another controller or
+# for a hostNetwork install (where the source is the node IP, not a pod IP).
+#
+# NetworkPolicy enforcement of kubelet probe traffic is CNI-dependent. If the
+# gateway starts failing its /healthz probes, allow the node CIDR on port 8080
+# or remove this policy. See ../../../docs/optional-hardening.md.
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: dsh-gateway-ingress
+  namespace: dsh-demo
+spec:
+  podSelector:
+    matchLabels:
+      app: dsh-gateway
+  policyTypes:
+    - Ingress
+  ingress:
+    - from:
+        - namespaceSelector: {}
+          podSelector:
+            matchLabels:
+              app.kubernetes.io/name: traefik
+      ports:
+        - protocol: TCP
+          port: 8080

examples/dsh-agents-as-a-service/manifests/optional/README.md

@@ -0,0 +1,37 @@
+# Optional manifests
+
+Opt-in manifests that are **not** part of the top-level walkthrough.
+`kubectl apply -f manifests/` is non-recursive, so it does not apply this
+directory; apply a subdirectory explicitly.
+
+## network-policy/
+
+A Kubernetes `NetworkPolicy` for the DSH sandboxes:
+
+- ingress only from the gateway pods on TCP `3080` (HTTP + WebSocket);
+- egress only to DNS and public HTTPS (provider APIs, package mirrors,
+  git-over-HTTPS).
+
+Apply and delete it explicitly:
+
+```bash
+kubectl apply -f manifests/optional/network-policy/
+kubectl delete -f manifests/optional/network-policy/
+```
+
+It requires a CNI that enforces `NetworkPolicy`. The selector uses the
+agent-sandbox controller's pod tracking label, so the policy applies to
+existing warm-pool and claimed sandboxes without recreation.
+
+## gVisor, nodeSelector and provider keys
+
+These are configured in the main manifests or with an extra Secret:
+
+- **gVisor:** uncomment `runtimeClassName: gvisor` in the SandboxTemplate and
+  the Deployments before applying, or patch an existing install.
+- **nodeSelector:** uncomment the field and replace the placeholder node label.
+- **Provider keys:** create the optional `dsh-provider-keys-extra` Secret; the
+  base `dsh-provider-keys` Secret remains the one required key.
+
+Full prerequisites, exact commands, verification and caveats:
+[optional hardening](../../docs/optional-hardening.md).

examples/dsh-agents-as-a-service/manifests/optional/network-policy/10-sandbox-networkpolicy.yaml

@@ -0,0 +1,80 @@
+# OPTIONAL, opt-in tenant network policy for the DSH sandboxes.
+#
+# It is NOT part of the top-level walkthrough. Apply and delete it explicitly:
+#   kubectl apply  -f manifests/optional/network-policy/
+#   kubectl delete -f manifests/optional/network-policy/
+#
+# The selector uses the agent-sandbox controller's pod tracking label, which is
+# present on every Sandbox pod (including warm-pool spares), so the policy
+# applies to existing sandboxes without recreating them. The SandboxTemplate
+# stays networkPolicyManagement: Unmanaged. Do not also switch it to Managed
+# unless you migrate these rules into spec.networkPolicy; NetworkPolicies are
+# additive and the two configurations would be confusing.
+#
+# Requires a CNI that enforces NetworkPolicy. See docs/optional-hardening.md
+# for the connectivity matrix, DNS variants, kubelet-probe caveat and rollback.
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: dsh-sandbox
+  namespace: dsh-demo
+spec:
+  podSelector:
+    matchExpressions:
+      - key: agents.x-k8s.io/sandbox-name-hash
+        operator: Exists
+  policyTypes:
+    - Ingress
+    - Egress
+  ingress:
+    # The gateway is the only intended client of a sandbox (HTTP + WebSocket).
+    - from:
+        - podSelector:
+            matchLabels:
+              app: dsh-gateway
+      ports:
+        - protocol: TCP
+          port: 3080
+  egress:
+    # Cluster DNS. CoreDNS/kube-dns pods carry k8s-app=kube-dns.
+    - to:
+        - namespaceSelector:
+            matchLabels:
+              kubernetes.io/metadata.name: kube-system
+          podSelector:
+            matchLabels:
+              k8s-app: kube-dns
+      ports:
+        - protocol: UDP
+          port: 53
+        - protocol: TCP
+          port: 53
+    # NodeLocal DNSCache, for clusters that do not send DNS to the kube-dns
+    # Service. 169.254.20.10 is its standard link-local address.
+    - to:
+        - ipBlock:
+            cidr: 169.254.20.10/32
+      ports:
+        - protocol: UDP
+          port: 53
+        - protocol: TCP
+          port: 53
+    # Public HTTPS only: provider APIs, package mirrors, git-over-HTTPS.
+    # Private ranges, CGNAT and the metadata server stay blocked.
+    - to:
+        - ipBlock:
+            cidr: 0.0.0.0/0
+            except:
+              - 10.0.0.0/8
+              - 172.16.0.0/12
+              - 192.168.0.0/16
+              - 169.254.0.0/16
+              - 100.64.0.0/10
+        - ipBlock:
+            cidr: "::/0"
+            except:
+              - "fc00::/7"
+              - "fe80::/10"
+      ports:
+        - protocol: TCP
+          port: 443