mirror of
https://github.com/sredevopsorg/deepseek-kubernetes-sandbox.git
synced 2026-11-03 01:13:24 +00:00
* fix(example): keep agents usable when DSH's inner sandbox has no backend DSH confines bash commands itself (bubblewrap, else Landlock) and fails closed with SANDBOX_UNAVAILABLE when no runner is usable. In this hardened pod - non-root, capabilities dropped, seccomp RuntimeDefault - bubblewrap cannot start (it needs CAP_SYS_ADMIN) and the pod exposes no Landlock runner, so the shipped workspace-write default made every command fail: sandbox mode "workspace-write" is requested but no sandbox backend is usable The DSH base bundle at the pinned ref documents DSH_PERMISSION_MODE as the explicit deployment override, so set it to danger-full-access in both sandbox templates. The per-user pod is the isolation boundary in this example (its own pod, PVC and Service; non-root; all capabilities dropped), so a working agent is worth more than an inner file policy that cannot be enforced. Note that danger-full-access also turns approval prompts off. Trade-off documented rather than hidden: an agent command can now read its own DSH_HOME (including .credentials.yaml) and write anywhere in the container. Hosts where a runner works should keep workspace-write or read-only. - main-image docs: DSH_PERMISSION_MODE row plus a note on the runners and the fail-closed behavior - example README: limits bullet and a troubleshooting row with the in-pod backend check and how to restore a confined mode - architecture: tenant-isolation bullet extended - skill runbook/troubleshooting: operation guidance; never grant CAP_SYS_ADMIN or seccompProfile: Unconfined to "fix" this without an explicit decision Verified: make check (14 manifests), make lint, YAML assertions on both templates, and the override expression at the pinned dsh-v0.2.1-rc.5 base bundle (mode: !!js process.env.DSH_PERMISSION_MODE ?? 'workspace-write'). * fix(image): install bubblewrap for DSH's command sandbox DSH's Linux sandbox runner chain is bubblewrap, then Landlock, and it fails closed with SANDBOX_UNAVAILABLE when neither works. The image shipped neither, so a sandboxed agent had no way to execute confined commands. Add bubblewrap to the runtime stage. Debian ships it unsetuid (0.12.0), so it still needs unprivileged user namespaces plus the namespace/mount syscalls: in a hardened pod (all capabilities dropped, RuntimeDefault seccomp) it usually cannot create one, and DSH then falls back to Landlock or, failing that, to the DSH_PERMISSION_MODE=danger-full-access override the templates already set. Where the host allows an unprivileged user namespace, the first-choice runner now exists, so the confined modes can be restored. Docs corrected to match: bubblewrap is no longer "absent"; the blocker is the unsetuid binary's namespace requirement, not a missing CAP_SYS_ADMIN grant. Verified: make check, make lint; the package exists in the pinned base (node:24-trixie-slim, bubblewrap 0.12.0-1~deb13u1); a throwaway container with the pod's flags (--user 1000 --cap-drop ALL --security-opt no-new-privileges) still cannot create a namespace, which is why the override stays. No local image build was run. * ci(image): build and smoke-test the main image on PRs that touch it The Dockerfile and the pins in versions.env could only be validated by a merge to main or by running `make image` by hand; neither gives a PR-time signal. Add ci-image.yaml: on pull requests that change Dockerfile, .dockerignore or versions.env (and on manual dispatch) it builds linux/amd64 natively with the pins sourced from versions.env - no registry login, no push - loads the image locally and smoke-tests it: - the packaged CLI runs and reports the pinned version - the container is the non-root node user (1000) and owns /workspace + DSH_HOME - the bubblewrap command-sandbox runner is present - the OCI version label matches versions.env - a container serves /favicon.svg (the same unauthenticated probe the shipped manifests use), with logs filtered so the launch token is never printed Path-filtered like the other on-demand checks, so it is not a required status check; the published multi-arch build stays in build-main.yaml and release.yaml. Docs updated in the same change: AGENTS.md repository map and testing table, docs/gitflow.md checks section, docs/main-image.md build section. Verified: actionlint 1.7.12 and yamllint 1.38.0 (CI versions) clean over the whole repository, plus make check and make lint. --------- Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
185 lines
9.0 KiB
YAML
185 lines
9.0 KiB
YAML
# The reusable DSH Web blueprint. Warm pools stamp pod env BEFORE any user
|
|
# exists, so per-user secrets cannot live in pod env; the shared provider keys
|
|
# are injected via envFrom, and claims that would bypass the warm pool are
|
|
# rejected outright (see 40-claim-rejected.yaml).
|
|
#
|
|
# A stripped-down single-tenant variant of this template (no warm pool, no
|
|
# policy enforcement) lives in examples/minimal-single-tenant/.
|
|
apiVersion: extensions.agents.x-k8s.io/v1beta1
|
|
kind: SandboxTemplate
|
|
metadata:
|
|
name: dsh-web
|
|
namespace: dsh-demo
|
|
spec:
|
|
# Claims must not inject env or volumeClaimTemplates: with Disallowed, such
|
|
# claims are REJECTED instead of silently taking the cold (non-warm-pool)
|
|
# provisioning path. Demonstrated in step 6.
|
|
envVarsInjectionPolicy: Disallowed
|
|
volumeClaimTemplatesPolicy: Disallowed
|
|
|
|
# Per-sandbox headless Service -> stable DNS (sandbox.status.serviceFQDN).
|
|
# The gateway routes each user to their DSH Web instance by this name.
|
|
service: true
|
|
|
|
# The example leaves NetworkPolicy to an optional, opt-in manifest
|
|
# (manifests/optional/network-policy/): the base walkthrough should not
|
|
# require a policy-capable CNI, and the controller's Managed secure default
|
|
# only allows the sandbox-router, which this example does not deploy. If you
|
|
# switch to Managed, define spec.networkPolicy explicitly or the gateway will
|
|
# be blocked. See docs/optional-hardening.md.
|
|
networkPolicyManagement: Unmanaged
|
|
|
|
# OPTIONAL: replace a Failed sandbox Pod instead of parking the sandbox.
|
|
# The default, Retain, leaves the Failed Pod in place and surfaces
|
|
# Finished=True on the Sandbox until a human intervenes. Recreate deletes the
|
|
# Failed Pod and starts a replacement with the same Sandbox identity and
|
|
# Sandbox-owned PVCs, backing off (5s doubling to 5m) on repeated failures.
|
|
# Requires agent-sandbox v1.0.8+; existing Sandboxes keep their own stamped
|
|
# policy, so patch them separately. See docs/optional-hardening.md.
|
|
# failedPodPolicy: Recreate
|
|
|
|
podTemplate:
|
|
spec:
|
|
# DSH Web is multi-arch (amd64 + arm64); no arch nodeSelector is needed.
|
|
#
|
|
# OPTIONAL: run the sandbox under gVisor. Requires a RuntimeClass named
|
|
# "gvisor" and nodes that can run it; gVisor breaks `kubectl port-forward`
|
|
# into the sandbox (use the gateway). Existing Sandboxes snapshot their
|
|
# pod template, so enabling it later requires patching/recreating them.
|
|
# See docs/optional-hardening.md.
|
|
# runtimeClassName: gvisor
|
|
#
|
|
# OPTIONAL: pin sandboxes to a dedicated node pool (for example a gVisor
|
|
# pool). Replace this placeholder with a label your nodes actually carry;
|
|
# add a matching toleration if the pool is tainted. Active
|
|
# cluster-specific scheduling belongs in a local patch, not here.
|
|
# nodeSelector:
|
|
# dsh.example.com/node-pool: dsh
|
|
restartPolicy: "OnFailure"
|
|
securityContext:
|
|
runAsUser: 1000 # the 'node' user in the DSH image
|
|
runAsGroup: 1000
|
|
fsGroup: 1000
|
|
runAsNonRoot: true
|
|
# DSH's credentials plugin refuses to start when .credentials.yaml has
|
|
# any group/other permission bit (it creates the file 0600). The
|
|
# default kubelet fsGroup policy ("Always") recursively ORs 0660 into
|
|
# every file each time the pod starts, so a resumed sandbox comes back
|
|
# with 0660 and crash-loops before the gateway can proxy to it.
|
|
# OnRootMismatch skips that walk once the volume root already carries
|
|
# the fsGroup ownership/mode applied on the first mount.
|
|
# Keep in sync with the minimal variant:
|
|
# examples/minimal-single-tenant/manifests/sandbox-template.yaml.
|
|
fsGroupChangePolicy: OnRootMismatch
|
|
containers:
|
|
- name: dsh-web
|
|
image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.5
|
|
imagePullPolicy: IfNotPresent
|
|
ports:
|
|
- name: web
|
|
containerPort: 3080
|
|
envFrom:
|
|
- secretRef:
|
|
name: dsh-provider-keys
|
|
# Optional extra provider keys (OpenRouter, Gemini, NVIDIA, ...).
|
|
# The Secret may be absent; every key in it becomes an env var.
|
|
# Create it before the warm pool, because pod env is stamped at
|
|
# pod creation. See docs/optional-hardening.md.
|
|
- secretRef:
|
|
name: dsh-provider-keys-extra
|
|
optional: true
|
|
env:
|
|
# Empty => DSH derives links from the request Host, which the
|
|
# gateway forwards (plus X-Forwarded-*). Point this at the
|
|
# gateway base URL if you want hard-coded absolute links instead.
|
|
- name: DSH_PUBLIC_URL
|
|
value: ""
|
|
# DSH Host/Origin-fences /api. The gateway forwards the browser's
|
|
# Host, so DSH must trust the public gateway host. Empty is right
|
|
# for port-forward/localhost; for either exposure variant set this
|
|
# to the public host (e.g. dsh.example.com for
|
|
# manifests/exposure/ingress/60-ingress.yaml, or
|
|
# dsh-demo.example.com for the gateway-api-sso/ HTTPRoute).
|
|
- name: DSH_TRUSTED_HOSTS
|
|
value: ""
|
|
# DSH confines every bash command with its own file sandbox
|
|
# (bubblewrap, else Landlock). Neither backend is usable inside this
|
|
# pod: bubblewrap ships in the image but is unsetuid, so it needs an
|
|
# unprivileged user namespace that the pod's RuntimeDefault seccomp
|
|
# and dropped capabilities do not grant, and the runtime/kernel
|
|
# surface here exposes no Landlock runner. The executor therefore
|
|
# fails closed with SANDBOX_UNAVAILABLE and every command fails. The
|
|
# pod is the isolation boundary in this example (one pod
|
|
# per user, non-root, capabilities dropped, seccomp RuntimeDefault,
|
|
# a private PVC), so trade DSH's inner file policy for a working
|
|
# agent. Set this to workspace-write (or read-only) on hosts where a
|
|
# backend is available; see the README troubleshooting row.
|
|
- name: DSH_PERMISSION_MODE
|
|
value: danger-full-access
|
|
# Probes target /favicon.svg: DSH gates / behind its launch-token
|
|
# cookie and answers 401 to an unauthenticated GET, which would
|
|
# otherwise fail readiness and then get the pod killed by liveness.
|
|
# /favicon.svg is served without auth and is the cheapest liveness
|
|
# signal that proves the HTTP server is actually serving requests.
|
|
#
|
|
# Gate liveness/readiness behind a startup probe: a cold start on a
|
|
# small node takes tens of seconds to load plugins and start serving
|
|
# (a resumed sandbox starts cold too). Without it the kubelet can kill
|
|
# DSH mid-startup, and a SIGTERM'd DSH exits 0, which the sandbox
|
|
# controller records as finished. 24 x 5s = 120s, matching the
|
|
# gateway's WAKE_TIMEOUT.
|
|
startupProbe:
|
|
httpGet:
|
|
path: /favicon.svg
|
|
port: 3080
|
|
periodSeconds: 5
|
|
failureThreshold: 24
|
|
readinessProbe:
|
|
httpGet:
|
|
path: /favicon.svg
|
|
port: 3080
|
|
initialDelaySeconds: 5
|
|
periodSeconds: 5
|
|
failureThreshold: 30
|
|
livenessProbe:
|
|
httpGet:
|
|
path: /favicon.svg
|
|
port: 3080
|
|
initialDelaySeconds: 15
|
|
periodSeconds: 15
|
|
failureThreshold: 3
|
|
# The container only needs to serve HTTP; drop every capability and
|
|
# disallow privilege escalation. DSH writes only to its PVC-backed
|
|
# directories (DSH_HOME and the agent workspace).
|
|
securityContext:
|
|
allowPrivilegeEscalation: false
|
|
capabilities:
|
|
drop: ["ALL"]
|
|
seccompProfile:
|
|
type: RuntimeDefault
|
|
resources:
|
|
requests: {cpu: 250m, memory: 512Mi}
|
|
limits: {cpu: "1", memory: 1Gi}
|
|
volumeMounts:
|
|
- name: dsh-home
|
|
mountPath: /home/node/.dsh # DSH_HOME: sessions, config, state
|
|
# The agent workspace (the image's WORKDIR) is PVC-backed too, but
|
|
# under its own subPath so DSH state - including .credentials.yaml -
|
|
# never shows up as workspace data. Kubelet creates workspace/ on
|
|
# first mount with the volume's fsGroup permissions (0775, setgid
|
|
# inherited from the volume root), so UID 1000 can write; no init
|
|
# container is needed. If the kubelet is killed mid-creation the
|
|
# directory stays 0755: see Troubleshooting in the README.
|
|
- name: dsh-home
|
|
mountPath: /workspace
|
|
subPath: workspace
|
|
volumeClaimTemplates:
|
|
- metadata:
|
|
name: dsh-home
|
|
spec:
|
|
accessModes: ["ReadWriteOnce"]
|
|
resources:
|
|
requests:
|
|
storage: 5Gi
|
|
# storageClassName omitted => cluster default (kind & GKE both work)
|