1
0
mirror of https://github.com/sredevopsorg/deepseek-kubernetes-sandbox.git synced 2026-11-03 01:13:24 +00:00
Files
deepseek-kubernetes-sandbox/examples/minimal-single-tenant
SREDevOps.org Agent a09d4ac340 fix: ship bubblewrap for DSH's command sandbox, keep agents usable without it (#35)
* fix(example): keep agents usable when DSH's inner sandbox has no backend

DSH confines bash commands itself (bubblewrap, else Landlock) and fails closed
with SANDBOX_UNAVAILABLE when no runner is usable. In this hardened pod -
non-root, capabilities dropped, seccomp RuntimeDefault - bubblewrap cannot
start (it needs CAP_SYS_ADMIN) and the pod exposes no Landlock runner, so the
shipped workspace-write default made every command fail:

  sandbox mode "workspace-write" is requested but no sandbox backend is usable

The DSH base bundle at the pinned ref documents DSH_PERMISSION_MODE as the
explicit deployment override, so set it to danger-full-access in both sandbox
templates. The per-user pod is the isolation boundary in this example (its own
pod, PVC and Service; non-root; all capabilities dropped), so a working agent
is worth more than an inner file policy that cannot be enforced. Note that
danger-full-access also turns approval prompts off.

Trade-off documented rather than hidden: an agent command can now read its own
DSH_HOME (including .credentials.yaml) and write anywhere in the container.
Hosts where a runner works should keep workspace-write or read-only.

- main-image docs: DSH_PERMISSION_MODE row plus a note on the runners and the
  fail-closed behavior
- example README: limits bullet and a troubleshooting row with the in-pod
  backend check and how to restore a confined mode
- architecture: tenant-isolation bullet extended
- skill runbook/troubleshooting: operation guidance; never grant CAP_SYS_ADMIN
  or seccompProfile: Unconfined to "fix" this without an explicit decision

Verified: make check (14 manifests), make lint, YAML assertions on both
templates, and the override expression at the pinned dsh-v0.2.1-rc.5 base
bundle (mode: !!js process.env.DSH_PERMISSION_MODE ?? 'workspace-write').

* fix(image): install bubblewrap for DSH's command sandbox

DSH's Linux sandbox runner chain is bubblewrap, then Landlock, and it fails
closed with SANDBOX_UNAVAILABLE when neither works. The image shipped neither,
so a sandboxed agent had no way to execute confined commands.

Add bubblewrap to the runtime stage. Debian ships it unsetuid (0.12.0), so it
still needs unprivileged user namespaces plus the namespace/mount syscalls: in a
hardened pod (all capabilities dropped, RuntimeDefault seccomp) it usually
cannot create one, and DSH then falls back to Landlock or, failing that, to the
DSH_PERMISSION_MODE=danger-full-access override the templates already set.
Where the host allows an unprivileged user namespace, the first-choice runner
now exists, so the confined modes can be restored.

Docs corrected to match: bubblewrap is no longer "absent"; the blocker is the
unsetuid binary's namespace requirement, not a missing CAP_SYS_ADMIN grant.

Verified: make check, make lint; the package exists in the pinned base
(node:24-trixie-slim, bubblewrap 0.12.0-1~deb13u1); a throwaway container with
the pod's flags (--user 1000 --cap-drop ALL --security-opt no-new-privileges)
still cannot create a namespace, which is why the override stays. No local image
build was run.

* ci(image): build and smoke-test the main image on PRs that touch it

The Dockerfile and the pins in versions.env could only be validated by a merge
to main or by running `make image` by hand; neither gives a PR-time signal.

Add ci-image.yaml: on pull requests that change Dockerfile, .dockerignore or
versions.env (and on manual dispatch) it builds linux/amd64 natively with the
pins sourced from versions.env - no registry login, no push - loads the image
locally and smoke-tests it:

- the packaged CLI runs and reports the pinned version
- the container is the non-root node user (1000) and owns /workspace + DSH_HOME
- the bubblewrap command-sandbox runner is present
- the OCI version label matches versions.env
- a container serves /favicon.svg (the same unauthenticated probe the shipped
  manifests use), with logs filtered so the launch token is never printed

Path-filtered like the other on-demand checks, so it is not a required status
check; the published multi-arch build stays in build-main.yaml and release.yaml.

Docs updated in the same change: AGENTS.md repository map and testing table,
docs/gitflow.md checks section, docs/main-image.md build section.

Verified: actionlint 1.7.12 and yamllint 1.38.0 (CI versions) clean over the
whole repository, plus make check and make lint.

---------

Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
2026-10-30 08:36:21 -03:00
..

Minimal single-tenant DSH Web

The smallest Kubernetes deployment of DeepSeek Harness (DSH) Web: one SandboxTemplate, a one-replica warm pool and one claim. It is the "just one DSH Web instance" quickstart.

For the full platform — warm pool sizing, per-user claims, a browser-facing gateway with cookie sessions, idle-suspend and wake-on-connect — see examples/dsh-agents-as-a-service/. This example deliberately has no gateway: you reach DSH through a kubectl port-forward and its launch-token URL.

Prerequisites

  • A Kubernetes cluster with a default StorageClass (kind and GKE both work).

  • The agent-sandbox controller with extensions (API v1beta1) installed. The version is pinned as AGENT_SANDBOX_VERSION in versions.env:

    # from the repository root:
    . versions.env
    kubectl apply -f "https://github.com/kubernetes-sigs/agent-sandbox/releases/download/${AGENT_SANDBOX_VERSION}/sandbox-with-extensions.yaml"
    kubectl -n agent-sandbox-system rollout status deployment --timeout=180s
    
  • A DeepSeek API key for real model calls. The pod starts without one; only model requests fail. Optionally create dsh-provider-keys-extra for OpenRouter, Gemini, NVIDIA or other provider keys (the template references it with optional: true).

Quickstart

Run these from this example's directory (examples/minimal-single-tenant/).

# 1. Namespace + shared provider keys. They are injected into the sandbox via
#    envFrom and are never committed.
kubectl create namespace dsh-demo
kubectl -n dsh-demo create secret generic dsh-provider-keys \
  --from-literal=DEEPSEEK_API_KEY="sk-..."

# Optional: extra providers, injected through a second Secret. Create it before
# the warm pool so the pods are stamped with the keys.
kubectl -n dsh-demo create secret generic dsh-provider-keys-extra \
  --from-literal=OPENROUTER_API_KEY="sk-or-..." \
  --from-literal=GEMINI_API_KEY="..." \
  --from-literal=NVIDIA_API_KEY="nvapi-..."

# 2. The template itself (image pinned in the manifest).
kubectl apply -f manifests/sandbox-template.yaml

# 3. A template only describes pods; a warm pool turns it into running
#    Sandboxes and a claim adopts one. One replica is enough here.
kubectl apply -f - <<'EOF'
apiVersion: extensions.agents.x-k8s.io/v1beta1
kind: SandboxWarmPool
metadata:
  name: dsh-pool
  namespace: dsh-demo
spec:
  replicas: 1
  sandboxTemplateRef:
    name: dsh-web-template
---
apiVersion: extensions.agents.x-k8s.io/v1beta1
kind: SandboxClaim
metadata:
  name: dsh
  namespace: dsh-demo
spec:
  warmPoolRef:
    name: dsh-pool
EOF

kubectl -n dsh-demo wait sandboxclaim/dsh --for=condition=Ready --timeout=180s

Talk to the instance

DSH gates / behind a launch token it prints only in its startup logs. Open the printed URL once; DSH then sets its own auth cookie and the UI loads normally.

SB=$(kubectl -n dsh-demo get sandboxclaim dsh -o jsonpath='{.status.sandbox.name}')
kubectl -n dsh-demo logs "$SB" | grep -m1 '?token='
# dsh web: http://<pod-ip>:3080/?token=...

kubectl -n dsh-demo port-forward "pod/$SB" 18081:3080
# open http://localhost:18081/?token=... from the log line

Port note: DSH Web listens on 3080 in-cluster. If 3080 is already in use on your workstation, forward to another local port (here 18081), never 3080. localhost is always in DSH's trusted-host list, so the forwarded request is accepted.

gVisor caveat: if you uncomment runtimeClassName: gvisor in the template, kubectl port-forward no longer works. Use the multi-tenant gateway example or a reachable Service instead; see optional hardening.

Network policy and optional hardening

The template omits networkPolicyManagement, so the agent-sandbox controller uses its Managed secure default: ingress only from the sandbox-router, public egress, and no internal cluster DNS. kubectl port-forward is local to the pod and still works (except under gVisor).

gVisor, an optional nodeSelector, extra provider keys and a stricter sandbox NetworkPolicy are documented in optional hardening.

Probes and trusted hosts

Two settings in manifests/sandbox-template.yaml are easy to get wrong:

  • Probe path. / returns 401 without the launch-token cookie, so a probe there crash-loops the pod. The template probes /favicon.svg (served unauthenticated) and gates liveness/readiness behind a startupProbe, which gives a cold start (plugin activation, or the first resume) up to 120s before a restart.
  • Host allow-list. DSH rejects unknown Host headers. localhost and 127.0.0.1 are always trusted; if you later front the sandbox with an Ingress or proxy, set DSH_TRUSTED_HOSTS to that public host (comma or space separated). The full example explains both exposure variants in manifests/exposure/README.md.

The template also sets fsGroupChangePolicy: OnRootMismatch. DSH's credentials plugin requires .credentials.yaml to be 0600, while the kubelet's default fsGroup policy rewrites it to 0660 on every pod start — without this policy a recreated or resumed sandbox crash-loops. The rationale is spelled out in 10-sandboxtemplate.yaml.

Suspend, resume and clean up

# Suspend: the pod is deleted, the PVC (state) and Service are kept.
kubectl -n dsh-demo patch sandbox "$SB" --type merge -p '{"spec":{"operatingMode":"Suspended"}}'

# Resume: a new pod reattaches the same PVC; conversations survive.
kubectl -n dsh-demo patch sandbox "$SB" --type merge -p '{"spec":{"operatingMode":"Running"}}'
kubectl -n dsh-demo wait sandbox "$SB" --for=condition=Ready --timeout=180s

# Delete everything this quickstart created.
kubectl -n dsh-demo delete sandboxclaim dsh
kubectl delete namespace dsh-demo

Deleting the claim cascades to the sandbox and its PVC.