mirror of
https://github.com/sredevopsorg/deepseek-kubernetes-sandbox.git
synced 2026-11-03 01:13:24 +00:00
fix: ship bubblewrap for DSH's command sandbox, keep agents usable without it (#35)
* fix(example): keep agents usable when DSH's inner sandbox has no backend DSH confines bash commands itself (bubblewrap, else Landlock) and fails closed with SANDBOX_UNAVAILABLE when no runner is usable. In this hardened pod - non-root, capabilities dropped, seccomp RuntimeDefault - bubblewrap cannot start (it needs CAP_SYS_ADMIN) and the pod exposes no Landlock runner, so the shipped workspace-write default made every command fail: sandbox mode "workspace-write" is requested but no sandbox backend is usable The DSH base bundle at the pinned ref documents DSH_PERMISSION_MODE as the explicit deployment override, so set it to danger-full-access in both sandbox templates. The per-user pod is the isolation boundary in this example (its own pod, PVC and Service; non-root; all capabilities dropped), so a working agent is worth more than an inner file policy that cannot be enforced. Note that danger-full-access also turns approval prompts off. Trade-off documented rather than hidden: an agent command can now read its own DSH_HOME (including .credentials.yaml) and write anywhere in the container. Hosts where a runner works should keep workspace-write or read-only. - main-image docs: DSH_PERMISSION_MODE row plus a note on the runners and the fail-closed behavior - example README: limits bullet and a troubleshooting row with the in-pod backend check and how to restore a confined mode - architecture: tenant-isolation bullet extended - skill runbook/troubleshooting: operation guidance; never grant CAP_SYS_ADMIN or seccompProfile: Unconfined to "fix" this without an explicit decision Verified: make check (14 manifests), make lint, YAML assertions on both templates, and the override expression at the pinned dsh-v0.2.1-rc.5 base bundle (mode: !!js process.env.DSH_PERMISSION_MODE ?? 'workspace-write'). * fix(image): install bubblewrap for DSH's command sandbox DSH's Linux sandbox runner chain is bubblewrap, then Landlock, and it fails closed with SANDBOX_UNAVAILABLE when neither works. The image shipped neither, so a sandboxed agent had no way to execute confined commands. Add bubblewrap to the runtime stage. Debian ships it unsetuid (0.12.0), so it still needs unprivileged user namespaces plus the namespace/mount syscalls: in a hardened pod (all capabilities dropped, RuntimeDefault seccomp) it usually cannot create one, and DSH then falls back to Landlock or, failing that, to the DSH_PERMISSION_MODE=danger-full-access override the templates already set. Where the host allows an unprivileged user namespace, the first-choice runner now exists, so the confined modes can be restored. Docs corrected to match: bubblewrap is no longer "absent"; the blocker is the unsetuid binary's namespace requirement, not a missing CAP_SYS_ADMIN grant. Verified: make check, make lint; the package exists in the pinned base (node:24-trixie-slim, bubblewrap 0.12.0-1~deb13u1); a throwaway container with the pod's flags (--user 1000 --cap-drop ALL --security-opt no-new-privileges) still cannot create a namespace, which is why the override stays. No local image build was run. * ci(image): build and smoke-test the main image on PRs that touch it The Dockerfile and the pins in versions.env could only be validated by a merge to main or by running `make image` by hand; neither gives a PR-time signal. Add ci-image.yaml: on pull requests that change Dockerfile, .dockerignore or versions.env (and on manual dispatch) it builds linux/amd64 natively with the pins sourced from versions.env - no registry login, no push - loads the image locally and smoke-tests it: - the packaged CLI runs and reports the pinned version - the container is the non-root node user (1000) and owns /workspace + DSH_HOME - the bubblewrap command-sandbox runner is present - the OCI version label matches versions.env - a container serves /favicon.svg (the same unauthenticated probe the shipped manifests use), with logs filtered so the launch token is never printed Path-filtered like the other on-demand checks, so it is not a required status check; the published multi-arch build stays in build-main.yaml and release.yaml. Docs updated in the same change: AGENTS.md repository map and testing table, docs/gitflow.md checks section, docs/main-image.md build section. Verified: actionlint 1.7.12 and yamllint 1.38.0 (CI versions) clean over the whole repository, plus make check and make lint. --------- Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
This commit is contained in:
committed by
GitHub
parent
367294b230
commit
a09d4ac340
@@ -123,6 +123,16 @@ kubectl -n "$NS" patch sandboxwarmpool "$POOL" --type merge -p '{"spec":{"replic
|
||||
`0600`; the template uses `fsGroupChangePolicy: OnRootMismatch`. If a resumed
|
||||
sandbox crash-loops with a mode-660 error, suspend it, fix the file with a
|
||||
throwaway pod, patch the policy, then resume.
|
||||
- Agent files live at `/workspace`, the same PVC mounted with
|
||||
`subPath: workspace`; it survives suspend/resume and dies with the PVC.
|
||||
- DSH confines bash with its own file sandbox (`bubblewrap`, else Landlock) and
|
||||
fails closed with `SANDBOX_UNAVAILABLE` when neither works. The image ships
|
||||
`bubblewrap`, but it is unsetuid and needs an unprivileged user namespace that
|
||||
this hardened pod (all capabilities dropped, RuntimeDefault seccomp) does not
|
||||
grant; that plus no Landlock runner is why the template sets
|
||||
`DSH_PERMISSION_MODE=danger-full-access` and treats the pod as the boundary.
|
||||
Never "fix" it by granting `CAP_SYS_ADMIN` or `seccompProfile: Unconfined`
|
||||
without an explicit decision.
|
||||
|
||||
## 8. Gateway layer
|
||||
|
||||
|
||||
@@ -18,6 +18,7 @@ events, then compare desired vs observed state. Keep output bounded. Never
|
||||
| `ImagePullBackOff` | describe events | Wrong image/tag, private GHCR package, missing pull secret; on kind, load the image under the exact manifest ref. |
|
||||
| `CrashLoopBackOff`, probe 401 | logs, `describe` | Probe path is `/`; use `/favicon.svg`. |
|
||||
| Restart with mode-660 error | logs mention `.credentials.yaml` | kubelet fsGroup widened the file; suspend, `chmod 600` from a throwaway pod, patch `fsGroupChangePolicy: OnRootMismatch`, resume. |
|
||||
| Every bash command fails with `SANDBOX_UNAVAILABLE` | DSH tool result names the requested mode and backends | DSH's own command sandbox has no usable runner: `bubblewrap` ships in the image but is unsetuid and needs an unprivileged user namespace this hardened pod does not grant, and no Landlock runner is available. The example template sets `DSH_PERMISSION_MODE=danger-full-access`; do not grant capabilities to restore it unless the operator explicitly accepts that downgrade. |
|
||||
| PVC `Pending` | `kubectl describe pvc`, StorageClass | No default provisioner, access mode, zone binding. |
|
||||
| Launch token not found | `kubectl logs <pod> --tail=2000 | grep 'dsh web:'` | Pod not ready yet, or logs rotated. Do not guess tokens; wait for readiness. |
|
||||
| Sandbox reports `Finished=True` | pod phase, `kubectl get events` | The backing Pod reached `Failed` (eviction, spot reclaim) and `failedPodPolicy` defaults to `Retain`. Delete the Pod to force a replacement, or set `Recreate` (v1.0.8+) on the Sandbox/template; the claim and PVCs survive either way. |
|
||||
|
||||
126
.github/workflows/ci-image.yaml
vendored
Normal file
126
.github/workflows/ci-image.yaml
vendored
Normal file
@@ -0,0 +1,126 @@
|
||||
# Build-only check for the main DSH Web image: linux/amd64, no registry login,
|
||||
# no push. It proves the Dockerfile still builds from the pinned versions.env and
|
||||
# that the resulting image can actually run DSH Web. The published multi-arch
|
||||
# build stays in build-main.yaml (merge to main) and release.yaml (tags).
|
||||
#
|
||||
# Like ci-python.yaml and ci-manifests.yaml this is path-filtered, so it is an
|
||||
# on-demand check rather than a required status check.
|
||||
name: CI image
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
paths:
|
||||
- Dockerfile
|
||||
- .dockerignore
|
||||
- versions.env
|
||||
- .github/workflows/ci-image.yaml
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
concurrency:
|
||||
group: ci-image-${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
build-amd64:
|
||||
name: build-amd64
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 120
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Read versions.env
|
||||
id: meta
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
set -a
|
||||
# shellcheck disable=SC1091
|
||||
source versions.env
|
||||
set +a
|
||||
identity="$DSH_VERSION"
|
||||
[ -z "${RELEASE_REVISION:-}" ] || identity="$DSH_VERSION-$RELEASE_REVISION"
|
||||
{
|
||||
echo "identity=$identity"
|
||||
echo "dsh_version=$DSH_VERSION"
|
||||
echo "pnpm_version=$PNPM_VERSION"
|
||||
echo "release_revision=${RELEASE_REVISION:-}"
|
||||
} >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
|
||||
|
||||
- name: Build linux/amd64 (load locally, never push)
|
||||
uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7.4.0
|
||||
with:
|
||||
context: .
|
||||
file: Dockerfile
|
||||
platforms: linux/amd64
|
||||
push: false
|
||||
load: true
|
||||
tags: dsh-web:ci
|
||||
build-args: |
|
||||
DSH_VERSION=${{ steps.meta.outputs.dsh_version }}
|
||||
RELEASE_REVISION=${{ steps.meta.outputs.release_revision }}
|
||||
PNPM_VERSION=${{ steps.meta.outputs.pnpm_version }}
|
||||
VCS_REF=${{ github.sha }}
|
||||
labels: |
|
||||
org.opencontainers.image.source=https://github.com/${{ github.repository }}
|
||||
org.opencontainers.image.version=${{ steps.meta.outputs.identity }}
|
||||
org.opencontainers.image.revision=${{ github.sha }}
|
||||
cache-from: type=gha,scope=ci-image-linux-amd64
|
||||
cache-to: type=gha,mode=max,scope=ci-image-linux-amd64
|
||||
|
||||
- name: Smoke test the image
|
||||
env:
|
||||
IDENTITY: ${{ steps.meta.outputs.identity }}
|
||||
DSH_VERSION: ${{ steps.meta.outputs.dsh_version }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
image=dsh-web:ci
|
||||
|
||||
# The packaged CLI runs and reports the version pinned in versions.env.
|
||||
version="$(docker run --rm --entrypoint node "$image" /app/lib/bin.js --version)"
|
||||
printf 'DSH CLI: %s\n' "$version"
|
||||
grep -q "${DSH_VERSION#dsh-v}" <<<"$version"
|
||||
|
||||
# The container runs as the non-root node user and its writable
|
||||
# directories exist and are owned by that user.
|
||||
[ "$(docker run --rm --entrypoint id "$image" -u)" = "1000" ]
|
||||
docker run --rm --entrypoint sh "$image" -c \
|
||||
'test -d /workspace && test -d "$DSH_HOME" && [ "$(stat -c %u /workspace)" = 1000 ]'
|
||||
|
||||
# The command-sandbox runner this image is expected to ship.
|
||||
docker run --rm --entrypoint bwrap "$image" --version
|
||||
|
||||
# Labels come from the build args rather than a hard-coded value.
|
||||
label="$(docker inspect -f '{{ index .Config.Labels "org.opencontainers.image.version" }}' "$image")"
|
||||
printf 'image version label: %s\n' "$label"
|
||||
[ "$label" = "$IDENTITY" ]
|
||||
|
||||
- name: Serve DSH Web
|
||||
run: |
|
||||
set -euo pipefail
|
||||
docker run -d --name dsh-ci -p 18080:3080 \
|
||||
-e DSH_TRUSTED_HOSTS=127.0.0.1:18080 dsh-web:ci >/dev/null
|
||||
trap 'docker rm -f dsh-ci >/dev/null 2>&1 || true' EXIT
|
||||
|
||||
# /favicon.svg is unauthenticated and served by the web layer, which is
|
||||
# why the shipped probes use it too.
|
||||
for second in $(seq 1 60); do
|
||||
if curl -fsS -o /dev/null http://127.0.0.1:18080/favicon.svg; then
|
||||
printf 'DSH Web served /favicon.svg after ~%ss\n' "$((second * 2))"
|
||||
exit 0
|
||||
fi
|
||||
sleep 2
|
||||
done
|
||||
|
||||
echo "::error::DSH Web did not serve /favicon.svg within 120s"
|
||||
# Pod logs can hold the DSH launch token: never dump them verbatim.
|
||||
docker logs dsh-ci 2>&1 | grep -v 'token=' | tail -50 || true
|
||||
exit 1
|
||||
@@ -49,7 +49,7 @@ before changing anything; it encodes the conventions the CI enforces.
|
||||
| examples/minimal-single-tenant/ | The just-one-DSH-Web quickstart |
|
||||
| scripts/ | check-versions.sh, check-examples.sh, check-manifests.py, set-version.sh |
|
||||
| .dsh/skills/ | Project-local DSH skills (auto-discovered); dsh-agent-sandbox-platform covers platform operation, MCP limits and extension recipes |
|
||||
| .github/workflows/ | ci.yaml (lint, dependency-review), ci-python.yaml, ci-manifests.yaml, reusable _build-image.yaml, build-*, release.yaml, e2e-kind.yaml, pr-title.yaml, codeql.yml |
|
||||
| .github/workflows/ | ci.yaml (lint, dependency-review), ci-python.yaml, ci-manifests.yaml, ci-image.yaml (amd64 build + smoke test on Dockerfile/versions.env changes), reusable _build-image.yaml, build-*, release.yaml, e2e-kind.yaml, pr-title.yaml, codeql.yml |
|
||||
|
||||
## Commands
|
||||
|
||||
@@ -74,7 +74,7 @@ needs kind plus the agent-sandbox CRDs (see the example README).
|
||||
| Example manifest | make check; kubectl apply --dry-run=server in CI |
|
||||
| Optional hardening (gVisor, nodeSelector, NetworkPolicy, provider keys) | make check; server dry-run in CI; manual verification on a capable cluster |
|
||||
| Gateway code | make test; make lint; add or update tests |
|
||||
| Dockerfile / versions.env | make check; make image (amd64) and a container smoke test |
|
||||
| Dockerfile / versions.env | make check; the CI `build-amd64` job (`ci-image.yaml`) builds and smoke-tests the image on the PR; run `make image` locally only when you need the image |
|
||||
| Workflow | actionlint in CI; keep actions SHA-pinned; no hard-coded versions |
|
||||
| New example | make check; all files described in docs/examples.md |
|
||||
| Project skill (.dsh/skills/) | make check; frontmatter and size check; shellcheck skill scripts; load it with the skill tool |
|
||||
|
||||
@@ -79,6 +79,11 @@ ENV NODE_ENV=production \
|
||||
|
||||
# git/ripgrep/jq/yq/curl/gh are deliberate agent tooling parity: a sandboxed DSH
|
||||
# instance expects the usual developer CLI. tini reaps signals for a clean stop.
|
||||
# bubblewrap is DSH's first-choice command-sandbox runner on Linux. It ships
|
||||
# unsetuid, so it needs unprivileged user namespaces plus the namespace/mount
|
||||
# syscalls; a hardened pod (all capabilities dropped, RuntimeDefault seccomp)
|
||||
# usually cannot create one, which is why DSH then falls back to Landlock and,
|
||||
# when that is unavailable too, to DSH_PERMISSION_MODE=danger-full-access.
|
||||
RUN apt-get update \
|
||||
&& apt-get install -y --no-install-recommends \
|
||||
ca-certificates \
|
||||
@@ -90,6 +95,7 @@ RUN apt-get update \
|
||||
wget \
|
||||
tini \
|
||||
gh \
|
||||
bubblewrap \
|
||||
&& rm -rf /var/lib/apt/lists/* \
|
||||
&& curl -LsSf https://astral.sh/uv/install.sh | sh \
|
||||
&& install -d -o node -g node -m 0755 /workspace "$DSH_HOME" \
|
||||
|
||||
@@ -49,16 +49,25 @@ branch; the PR is squash-merged, so individual branch commits are never part of
|
||||
| `CodeQL` | `codeql.yml` | Static analysis for Python and GitHub Actions. |
|
||||
| `dependency-review` | `ci.yaml` | Dependency changes introduced by the PR (job `dependency-review`; runs on pull requests only). |
|
||||
|
||||
`python` and `manifests` use native `on.pull_request.paths` filtering, so PRs that
|
||||
do not touch the gateway or the manifests skip them. The gateway image build and
|
||||
`/healthz` are no longer smoke-tested in a dedicated job: `/healthz` is covered by
|
||||
the gateway unit tests, the Dockerfile is linted by `lint` (hadolint), and the
|
||||
multi-arch image build is published by `build-gateway.yaml` on merge.
|
||||
`python`, `manifests` and `build-amd64` use native `on.pull_request.paths`
|
||||
filtering, so PRs that do not touch the gateway, the manifests or the image
|
||||
definition skip them. The gateway image build and `/healthz` are no longer
|
||||
smoke-tested in a dedicated job: `/healthz` is covered by the gateway unit tests,
|
||||
the Dockerfile is linted by `lint` (hadolint), and the multi-arch image build is
|
||||
published by `build-gateway.yaml` on merge.
|
||||
|
||||
`ci-image.yaml` (`build-amd64`) is the PR-side check for the main image: when
|
||||
`Dockerfile`, `.dockerignore` or `versions.env` change it builds `linux/amd64`
|
||||
with the pins from `versions.env` — no registry login, no push — and smoke-tests
|
||||
the result (`--version`, non-root UID, `/workspace` + `DSH_HOME`, the
|
||||
`bubblewrap` runner, the version label, and `/favicon.svg` served from a running
|
||||
container). The published multi-arch build still happens only in
|
||||
`build-main.yaml` on merge.
|
||||
|
||||
Because a skipped path-filtered check is not a passing check, only the always-on
|
||||
checks (`lint`, `pr-title`, `CodeQL`, `dependency-review`) should be enforced as
|
||||
required status checks; `python` and `manifests` are on-demand checks that skip
|
||||
when unrelated to the change.
|
||||
required status checks; `python`, `manifests` and `build-amd64` are on-demand
|
||||
checks that skip when unrelated to the change.
|
||||
|
||||
The image build workflows (`_build-image.yaml`, `build-main.yaml`,
|
||||
`build-gateway.yaml`) and `e2e-kind.yaml` are **not** required on every PR: the
|
||||
|
||||
@@ -18,7 +18,8 @@ The Dockerfile is a two-stage build:
|
||||
CLI with `pnpm pack`, and installs its production dependencies with npm.
|
||||
2. **Runtime stage** (`node:24-trixie-slim`) copies `/opt/dsh` to `/app`, adds the
|
||||
CLI tools DSH agents commonly use (`git`, `ripgrep`, `jq`, `yq`, `curl`, `wget`,
|
||||
`gh`, `tini`, `uv`), and starts `dsh web` through a small entrypoint.
|
||||
`gh`, `tini`, `uv`), installs `bubblewrap` as DSH's preferred command-sandbox
|
||||
runner, and starts `dsh web` through a small entrypoint.
|
||||
|
||||
## Build it
|
||||
|
||||
@@ -27,6 +28,9 @@ make image # canonical: builds the DSH Web image, no push
|
||||
```
|
||||
|
||||
`make image` reads [versions.env](../versions.env) and passes the pins as build args.
|
||||
Pull requests that touch `Dockerfile`, `.dockerignore` or `versions.env` get the
|
||||
same build for `linux/amd64` from the [`ci-image.yaml`](../.github/workflows/ci-image.yaml)
|
||||
workflow — no registry login, no push — followed by a container smoke test.
|
||||
To reproduce the equivalent by hand:
|
||||
|
||||
```bash
|
||||
@@ -76,10 +80,24 @@ The entrypoint is `tini -- /usr/local/bin/dsh-entrypoint`:
|
||||
| `DSH_TRUSTED_HOSTS` | empty | Extra `Host`/`Origin` allow-list entries, comma- or space-separated. `localhost`/`127.0.0.1` are always trusted. Required when a proxy forwards the browser's `Host`. |
|
||||
| `DEEPSEEK_API_KEY` | empty | Provider credential read by DSH. Inject it with `-e`/`envFrom`; never bake it into the image. Only model calls fail when it is missing. |
|
||||
| `DSH_HOME` | `/home/node/.dsh` | DSH state directory (conversations, sessions, config, credentials). Mount a volume here to persist it. |
|
||||
| `DSH_PERMISSION_MODE` | `workspace-write` | Deployment default for DSH's file policy: `read-only`, `workspace-write`, or `danger-full-access` (which also turns approval prompts off). The confined modes need `bubblewrap` or Landlock inside the container; where neither is usable every bash command fails closed with `SANDBOX_UNAVAILABLE`. See the note below. |
|
||||
|
||||
The image also sets `NODE_ENV=production`, `HOME=/home/node`, and a `PATH` that
|
||||
includes `/app/node_modules/.bin`.
|
||||
|
||||
DSH confines bash commands itself (a second layer inside whatever container
|
||||
sandbox you already run). Its Linux runners are `bubblewrap` and then Landlock,
|
||||
and it fails closed rather than running unconfined. The image ships
|
||||
`bubblewrap`, but Debian's binary is unsetuid: it needs **unprivileged user
|
||||
namespaces** plus the namespace/mount syscalls. A hardened pod — non-root,
|
||||
`capabilities.drop: ["ALL"]`, `seccompProfile: RuntimeDefault` — usually cannot
|
||||
create one, and it may expose no Landlock runner either, in which case every
|
||||
command fails with `SANDBOX_UNAVAILABLE`. Set
|
||||
`DSH_PERMISSION_MODE=danger-full-access` only when the surrounding pod or VM is
|
||||
the isolation boundary you rely on; keep the confined modes when the host
|
||||
provides a working runner (check `grep -o landlock /sys/kernel/security/lsm` and
|
||||
`bwrap --ro-bind / / true` inside the container first).
|
||||
|
||||
### Provider credentials
|
||||
|
||||
DSH resolves provider API keys from the environment when no stored credential
|
||||
|
||||
@@ -588,6 +588,7 @@ is non-recursive, so the optional manifest is not applied by the base step.
|
||||
| Step 3: `pod "$SB" not found` after deploying the gateway | The claim was suspended by the idle sweeper. Claims created by the gateway carry `aaas.example.com/managed-by=dsh-gateway`; hand-applied claims (steps 1–7) do not and are never swept. Resume the sandbox: `kubectl -n dsh-demo patch sandbox "$SB" --type merge -p '{"spec":{"operatingMode":"Running"}}'` |
|
||||
| Resume crash-loops: `credentials-local: … is readable beyond its owner (mode 660)` | The pod template lacks `fsGroupChangePolicy: OnRootMismatch`, so the kubelet's default fsGroup policy rewrote `.credentials.yaml` to `0660`; DSH requires `0600` and refuses to start. New installs get the policy from the template. For an existing sandbox: suspend it, run a throwaway pod on `dsh-home-$SB` to `chmod 600 /home/node/.dsh/.credentials.yaml`, patch the sandbox's `podTemplate` with the policy, then resume |
|
||||
| Agents get `Permission denied` writing to `/workspace` on a fresh or resumed pod | The `/workspace` mount is the `dsh-home` PVC's `workspace/` subPath, and the kubelet is known to leave a `subPath` directory at `0755` if it is interrupted between creating and chmodding it ([kubernetes#142464](https://github.com/kubernetes/kubernetes/issues/142464)); later mounts never correct it. Suspend the sandbox, run a throwaway pod on `dsh-home-$SB` mounted at `/data`, `chmod 2775 /data/workspace` (do **not** chown/chmod recursively — that rewrites `.credentials.yaml`), then resume. If you recreate the sandbox instead, its current `/workspace` contents go with the PVC |
|
||||
| Every bash command fails: `SANDBOX_UNAVAILABLE`, `sandbox mode "workspace-write" is requested but no sandbox backend is usable` | DSH's own command sandbox found neither backend: `bubblewrap` ships in the image but is unsetuid, so it needs an unprivileged user namespace this hardened pod does not grant, and no Landlock runner exists on this kernel/runtime — the confined executor fails closed. The template sets `DSH_PERMISSION_MODE=danger-full-access` to keep agents usable inside the pod boundary. To restore confinement, confirm a backend inside the pod (`bwrap --ro-bind / / true`; `grep -o landlock /sys/kernel/security/lsm`), set `DSH_PERMISSION_MODE=workspace-write` (or `read-only`), and recreate the pod — the value is stamped at creation. A session that explicitly chose a mode keeps that choice |
|
||||
| SSO: GitHub login succeeds but the gateway asks for a token / returns `401` to `/recreate` | `AUTH_PROXY_HEADER` is not set on `deploy/dsh-gateway` (SSO is opt-in). Set `AUTH_PROXY_HEADER=X-Forwarded-User` and confirm the request actually reaches the gateway through oauth2-proxy |
|
||||
| SSO: gateway treats a caller as another user / bad `X-Forwarded-User` accepted | Trust boundary violation: something other than oauth2-proxy can reach `dsh-gateway:8080`. Remove the second path (Ingress/Gateway/port-forward), keep the Service ClusterIP, and only re-enable SSO once the proxy is the sole entry point. Never put a trusted identity header in front of a service that is directly reachable |
|
||||
| SSO: oauth2-proxy `cookie_secret must be 16, 24, or 32 bytes ... but is 44 bytes` | The cookie secret was generated as standard base64. Use URL-safe unpadded base64 (see `73-oauth2-proxy-secret.yaml.example`) |
|
||||
@@ -622,6 +623,19 @@ Deliberately **not** production code:
|
||||
to oauth2-proxy and oauth2-proxy egress to the gateway/DNS/public HTTPS; a
|
||||
custom exposure without that policy has no signature, mTLS or network
|
||||
enforcement.
|
||||
- **DSH's inner file sandbox is off here.** The template sets
|
||||
`DSH_PERMISSION_MODE=danger-full-access` because neither backend of DSH's own
|
||||
command sandbox is usable in this pod: `bubblewrap` is installed in the image
|
||||
but is unsetuid, so it needs an unprivileged user namespace that the pod's
|
||||
seccomp/capability surface does not grant (`capabilities.drop: ["ALL"]`), and
|
||||
the kernel/runtime exposes no Landlock runner. The executor therefore fails
|
||||
closed with `SANDBOX_UNAVAILABLE` and every bash command fails. The per-user
|
||||
pod remains the boundary — non-root,
|
||||
capabilities dropped, `seccompProfile: RuntimeDefault`, its own PVC and Service
|
||||
— but an agent command can now read its own `DSH_HOME` (including
|
||||
`.credentials.yaml`) and write anywhere in the container. On hosts where a
|
||||
backend works, set `workspace-write` (or `read-only`) and verify before relying
|
||||
on the inner policy.
|
||||
- **Isolation is opt-in.** gVisor, the sandbox NetworkPolicy and the exposure
|
||||
policies are opt-in; without them, sandboxes share the host kernel and can
|
||||
reach the cluster network. The base example is a demo, not a hardened
|
||||
|
||||
@@ -309,6 +309,12 @@ Notes and limits:
|
||||
`dsh-provider-keys-extra`). `manifests/optional/network-policy/` adds an opt-in sandbox
|
||||
NetworkPolicy, and gVisor/`nodeSelector` are opt-in PodSpec fields. Add per-tenant
|
||||
credentials before calling it multi-tenant-grade.
|
||||
- **DSH's inner file sandbox is disabled by deployment override.** The template sets
|
||||
`DSH_PERMISSION_MODE=danger-full-access` because the in-container backends are unusable here:
|
||||
`bubblewrap` ships in the image but is unsetuid and cannot create an unprivileged user namespace
|
||||
under the pod's seccomp/capability surface, and no Landlock runner is available; the confined
|
||||
executor would otherwise fail closed on every command. The pod remains the boundary, so an
|
||||
agent can read its own `DSH_HOME`. Hosts with a working backend can keep a confined mode.
|
||||
- **Optional hardening is opt-in.** gVisor, node selection, the sandbox/exposure NetworkPolicies,
|
||||
extra provider keys and failed-Pod recovery are documented in
|
||||
[optional hardening](optional-hardening.md). The base manifests stay portable (no gVisor
|
||||
|
||||
@@ -102,6 +102,20 @@ spec:
|
||||
# dsh-demo.example.com for the gateway-api-sso/ HTTPRoute).
|
||||
- name: DSH_TRUSTED_HOSTS
|
||||
value: ""
|
||||
# DSH confines every bash command with its own file sandbox
|
||||
# (bubblewrap, else Landlock). Neither backend is usable inside this
|
||||
# pod: bubblewrap ships in the image but is unsetuid, so it needs an
|
||||
# unprivileged user namespace that the pod's RuntimeDefault seccomp
|
||||
# and dropped capabilities do not grant, and the runtime/kernel
|
||||
# surface here exposes no Landlock runner. The executor therefore
|
||||
# fails closed with SANDBOX_UNAVAILABLE and every command fails. The
|
||||
# pod is the isolation boundary in this example (one pod
|
||||
# per user, non-root, capabilities dropped, seccomp RuntimeDefault,
|
||||
# a private PVC), so trade DSH's inner file policy for a working
|
||||
# agent. Set this to workspace-write (or read-only) on hosts where a
|
||||
# backend is available; see the README troubleshooting row.
|
||||
- name: DSH_PERMISSION_MODE
|
||||
value: danger-full-access
|
||||
# Probes target /favicon.svg: DSH gates / behind its launch-token
|
||||
# cookie and answers 401 to an unauthenticated GET, which would
|
||||
# otherwise fail readiness and then get the pod killed by liveness.
|
||||
|
||||
@@ -81,6 +81,14 @@ spec:
|
||||
# examples/dsh-agents-as-a-service/manifests/exposure/README.md.
|
||||
- name: DSH_TRUSTED_HOSTS
|
||||
value: ""
|
||||
# DSH's own bash file sandbox (bubblewrap, else Landlock) cannot be
|
||||
# enforced in this hardened pod - bubblewrap is installed but is
|
||||
# unsetuid and needs an unprivileged user namespace - and fails
|
||||
# closed with SANDBOX_UNAVAILABLE. The pod is the boundary here;
|
||||
# keep the deployment override in sync with the multi-tenant example
|
||||
# and see its README troubleshooting row to restore a confined mode.
|
||||
- name: DSH_PERMISSION_MODE
|
||||
value: danger-full-access
|
||||
volumeMounts:
|
||||
- name: dsh-home
|
||||
mountPath: /home/node/.dsh # DSH_HOME
|
||||
|
||||
Reference in New Issue
Block a user