mirror of
https://github.com/sredevopsorg/deepseek-kubernetes-sandbox.git
synced 2026-11-03 01:13:24 +00:00
fix: ship bubblewrap for DSH's command sandbox, keep agents usable without it (#35)
* fix(example): keep agents usable when DSH's inner sandbox has no backend DSH confines bash commands itself (bubblewrap, else Landlock) and fails closed with SANDBOX_UNAVAILABLE when no runner is usable. In this hardened pod - non-root, capabilities dropped, seccomp RuntimeDefault - bubblewrap cannot start (it needs CAP_SYS_ADMIN) and the pod exposes no Landlock runner, so the shipped workspace-write default made every command fail: sandbox mode "workspace-write" is requested but no sandbox backend is usable The DSH base bundle at the pinned ref documents DSH_PERMISSION_MODE as the explicit deployment override, so set it to danger-full-access in both sandbox templates. The per-user pod is the isolation boundary in this example (its own pod, PVC and Service; non-root; all capabilities dropped), so a working agent is worth more than an inner file policy that cannot be enforced. Note that danger-full-access also turns approval prompts off. Trade-off documented rather than hidden: an agent command can now read its own DSH_HOME (including .credentials.yaml) and write anywhere in the container. Hosts where a runner works should keep workspace-write or read-only. - main-image docs: DSH_PERMISSION_MODE row plus a note on the runners and the fail-closed behavior - example README: limits bullet and a troubleshooting row with the in-pod backend check and how to restore a confined mode - architecture: tenant-isolation bullet extended - skill runbook/troubleshooting: operation guidance; never grant CAP_SYS_ADMIN or seccompProfile: Unconfined to "fix" this without an explicit decision Verified: make check (14 manifests), make lint, YAML assertions on both templates, and the override expression at the pinned dsh-v0.2.1-rc.5 base bundle (mode: !!js process.env.DSH_PERMISSION_MODE ?? 'workspace-write'). * fix(image): install bubblewrap for DSH's command sandbox DSH's Linux sandbox runner chain is bubblewrap, then Landlock, and it fails closed with SANDBOX_UNAVAILABLE when neither works. The image shipped neither, so a sandboxed agent had no way to execute confined commands. Add bubblewrap to the runtime stage. Debian ships it unsetuid (0.12.0), so it still needs unprivileged user namespaces plus the namespace/mount syscalls: in a hardened pod (all capabilities dropped, RuntimeDefault seccomp) it usually cannot create one, and DSH then falls back to Landlock or, failing that, to the DSH_PERMISSION_MODE=danger-full-access override the templates already set. Where the host allows an unprivileged user namespace, the first-choice runner now exists, so the confined modes can be restored. Docs corrected to match: bubblewrap is no longer "absent"; the blocker is the unsetuid binary's namespace requirement, not a missing CAP_SYS_ADMIN grant. Verified: make check, make lint; the package exists in the pinned base (node:24-trixie-slim, bubblewrap 0.12.0-1~deb13u1); a throwaway container with the pod's flags (--user 1000 --cap-drop ALL --security-opt no-new-privileges) still cannot create a namespace, which is why the override stays. No local image build was run. * ci(image): build and smoke-test the main image on PRs that touch it The Dockerfile and the pins in versions.env could only be validated by a merge to main or by running `make image` by hand; neither gives a PR-time signal. Add ci-image.yaml: on pull requests that change Dockerfile, .dockerignore or versions.env (and on manual dispatch) it builds linux/amd64 natively with the pins sourced from versions.env - no registry login, no push - loads the image locally and smoke-tests it: - the packaged CLI runs and reports the pinned version - the container is the non-root node user (1000) and owns /workspace + DSH_HOME - the bubblewrap command-sandbox runner is present - the OCI version label matches versions.env - a container serves /favicon.svg (the same unauthenticated probe the shipped manifests use), with logs filtered so the launch token is never printed Path-filtered like the other on-demand checks, so it is not a required status check; the published multi-arch build stays in build-main.yaml and release.yaml. Docs updated in the same change: AGENTS.md repository map and testing table, docs/gitflow.md checks section, docs/main-image.md build section. Verified: actionlint 1.7.12 and yamllint 1.38.0 (CI versions) clean over the whole repository, plus make check and make lint. --------- Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
This commit is contained in:
committed by
GitHub
parent
367294b230
commit
a09d4ac340
@@ -588,6 +588,7 @@ is non-recursive, so the optional manifest is not applied by the base step.
|
||||
| Step 3: `pod "$SB" not found` after deploying the gateway | The claim was suspended by the idle sweeper. Claims created by the gateway carry `aaas.example.com/managed-by=dsh-gateway`; hand-applied claims (steps 1–7) do not and are never swept. Resume the sandbox: `kubectl -n dsh-demo patch sandbox "$SB" --type merge -p '{"spec":{"operatingMode":"Running"}}'` |
|
||||
| Resume crash-loops: `credentials-local: … is readable beyond its owner (mode 660)` | The pod template lacks `fsGroupChangePolicy: OnRootMismatch`, so the kubelet's default fsGroup policy rewrote `.credentials.yaml` to `0660`; DSH requires `0600` and refuses to start. New installs get the policy from the template. For an existing sandbox: suspend it, run a throwaway pod on `dsh-home-$SB` to `chmod 600 /home/node/.dsh/.credentials.yaml`, patch the sandbox's `podTemplate` with the policy, then resume |
|
||||
| Agents get `Permission denied` writing to `/workspace` on a fresh or resumed pod | The `/workspace` mount is the `dsh-home` PVC's `workspace/` subPath, and the kubelet is known to leave a `subPath` directory at `0755` if it is interrupted between creating and chmodding it ([kubernetes#142464](https://github.com/kubernetes/kubernetes/issues/142464)); later mounts never correct it. Suspend the sandbox, run a throwaway pod on `dsh-home-$SB` mounted at `/data`, `chmod 2775 /data/workspace` (do **not** chown/chmod recursively — that rewrites `.credentials.yaml`), then resume. If you recreate the sandbox instead, its current `/workspace` contents go with the PVC |
|
||||
| Every bash command fails: `SANDBOX_UNAVAILABLE`, `sandbox mode "workspace-write" is requested but no sandbox backend is usable` | DSH's own command sandbox found neither backend: `bubblewrap` ships in the image but is unsetuid, so it needs an unprivileged user namespace this hardened pod does not grant, and no Landlock runner exists on this kernel/runtime — the confined executor fails closed. The template sets `DSH_PERMISSION_MODE=danger-full-access` to keep agents usable inside the pod boundary. To restore confinement, confirm a backend inside the pod (`bwrap --ro-bind / / true`; `grep -o landlock /sys/kernel/security/lsm`), set `DSH_PERMISSION_MODE=workspace-write` (or `read-only`), and recreate the pod — the value is stamped at creation. A session that explicitly chose a mode keeps that choice |
|
||||
| SSO: GitHub login succeeds but the gateway asks for a token / returns `401` to `/recreate` | `AUTH_PROXY_HEADER` is not set on `deploy/dsh-gateway` (SSO is opt-in). Set `AUTH_PROXY_HEADER=X-Forwarded-User` and confirm the request actually reaches the gateway through oauth2-proxy |
|
||||
| SSO: gateway treats a caller as another user / bad `X-Forwarded-User` accepted | Trust boundary violation: something other than oauth2-proxy can reach `dsh-gateway:8080`. Remove the second path (Ingress/Gateway/port-forward), keep the Service ClusterIP, and only re-enable SSO once the proxy is the sole entry point. Never put a trusted identity header in front of a service that is directly reachable |
|
||||
| SSO: oauth2-proxy `cookie_secret must be 16, 24, or 32 bytes ... but is 44 bytes` | The cookie secret was generated as standard base64. Use URL-safe unpadded base64 (see `73-oauth2-proxy-secret.yaml.example`) |
|
||||
@@ -622,6 +623,19 @@ Deliberately **not** production code:
|
||||
to oauth2-proxy and oauth2-proxy egress to the gateway/DNS/public HTTPS; a
|
||||
custom exposure without that policy has no signature, mTLS or network
|
||||
enforcement.
|
||||
- **DSH's inner file sandbox is off here.** The template sets
|
||||
`DSH_PERMISSION_MODE=danger-full-access` because neither backend of DSH's own
|
||||
command sandbox is usable in this pod: `bubblewrap` is installed in the image
|
||||
but is unsetuid, so it needs an unprivileged user namespace that the pod's
|
||||
seccomp/capability surface does not grant (`capabilities.drop: ["ALL"]`), and
|
||||
the kernel/runtime exposes no Landlock runner. The executor therefore fails
|
||||
closed with `SANDBOX_UNAVAILABLE` and every bash command fails. The per-user
|
||||
pod remains the boundary — non-root,
|
||||
capabilities dropped, `seccompProfile: RuntimeDefault`, its own PVC and Service
|
||||
— but an agent command can now read its own `DSH_HOME` (including
|
||||
`.credentials.yaml`) and write anywhere in the container. On hosts where a
|
||||
backend works, set `workspace-write` (or `read-only`) and verify before relying
|
||||
on the inner policy.
|
||||
- **Isolation is opt-in.** gVisor, the sandbox NetworkPolicy and the exposure
|
||||
policies are opt-in; without them, sandboxes share the host kernel and can
|
||||
reach the cluster network. The base example is a demo, not a hardened
|
||||
|
||||
Reference in New Issue
Block a user