mirror of
https://github.com/sredevopsorg/deepseek-kubernetes-sandbox.git
synced 2026-11-03 01:13:24 +00:00
* fix(example): keep agents usable when DSH's inner sandbox has no backend DSH confines bash commands itself (bubblewrap, else Landlock) and fails closed with SANDBOX_UNAVAILABLE when no runner is usable. In this hardened pod - non-root, capabilities dropped, seccomp RuntimeDefault - bubblewrap cannot start (it needs CAP_SYS_ADMIN) and the pod exposes no Landlock runner, so the shipped workspace-write default made every command fail: sandbox mode "workspace-write" is requested but no sandbox backend is usable The DSH base bundle at the pinned ref documents DSH_PERMISSION_MODE as the explicit deployment override, so set it to danger-full-access in both sandbox templates. The per-user pod is the isolation boundary in this example (its own pod, PVC and Service; non-root; all capabilities dropped), so a working agent is worth more than an inner file policy that cannot be enforced. Note that danger-full-access also turns approval prompts off. Trade-off documented rather than hidden: an agent command can now read its own DSH_HOME (including .credentials.yaml) and write anywhere in the container. Hosts where a runner works should keep workspace-write or read-only. - main-image docs: DSH_PERMISSION_MODE row plus a note on the runners and the fail-closed behavior - example README: limits bullet and a troubleshooting row with the in-pod backend check and how to restore a confined mode - architecture: tenant-isolation bullet extended - skill runbook/troubleshooting: operation guidance; never grant CAP_SYS_ADMIN or seccompProfile: Unconfined to "fix" this without an explicit decision Verified: make check (14 manifests), make lint, YAML assertions on both templates, and the override expression at the pinned dsh-v0.2.1-rc.5 base bundle (mode: !!js process.env.DSH_PERMISSION_MODE ?? 'workspace-write'). * fix(image): install bubblewrap for DSH's command sandbox DSH's Linux sandbox runner chain is bubblewrap, then Landlock, and it fails closed with SANDBOX_UNAVAILABLE when neither works. The image shipped neither, so a sandboxed agent had no way to execute confined commands. Add bubblewrap to the runtime stage. Debian ships it unsetuid (0.12.0), so it still needs unprivileged user namespaces plus the namespace/mount syscalls: in a hardened pod (all capabilities dropped, RuntimeDefault seccomp) it usually cannot create one, and DSH then falls back to Landlock or, failing that, to the DSH_PERMISSION_MODE=danger-full-access override the templates already set. Where the host allows an unprivileged user namespace, the first-choice runner now exists, so the confined modes can be restored. Docs corrected to match: bubblewrap is no longer "absent"; the blocker is the unsetuid binary's namespace requirement, not a missing CAP_SYS_ADMIN grant. Verified: make check, make lint; the package exists in the pinned base (node:24-trixie-slim, bubblewrap 0.12.0-1~deb13u1); a throwaway container with the pod's flags (--user 1000 --cap-drop ALL --security-opt no-new-privileges) still cannot create a namespace, which is why the override stays. No local image build was run. * ci(image): build and smoke-test the main image on PRs that touch it The Dockerfile and the pins in versions.env could only be validated by a merge to main or by running `make image` by hand; neither gives a PR-time signal. Add ci-image.yaml: on pull requests that change Dockerfile, .dockerignore or versions.env (and on manual dispatch) it builds linux/amd64 natively with the pins sourced from versions.env - no registry login, no push - loads the image locally and smoke-tests it: - the packaged CLI runs and reports the pinned version - the container is the non-root node user (1000) and owns /workspace + DSH_HOME - the bubblewrap command-sandbox runner is present - the OCI version label matches versions.env - a container serves /favicon.svg (the same unauthenticated probe the shipped manifests use), with logs filtered so the launch token is never printed Path-filtered like the other on-demand checks, so it is not a required status check; the published multi-arch build stays in build-main.yaml and release.yaml. Docs updated in the same change: AGENTS.md repository map and testing table, docs/gitflow.md checks section, docs/main-image.md build section. Verified: actionlint 1.7.12 and yamllint 1.38.0 (CI versions) clean over the whole repository, plus make check and make lint. --------- Co-authored-by: SREDevOpsAgents <agents@sredevops.org>
212 lines
10 KiB
Markdown
212 lines
10 KiB
Markdown
# The main DSH Web image
|
|
|
|
This repository's primary artifact is a container image that packages
|
|
[DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) Web
|
|
(`dsh web`) for Kubernetes and plain Docker. The [Dockerfile](../Dockerfile) at the
|
|
repository root builds it; every version pin it uses lives in
|
|
[versions.env](../versions.env).
|
|
|
|
The image is image-first: it is the product, and the examples under
|
|
[`examples/`](../examples/) are self-contained implementations that consume it.
|
|
|
|
## What the build does
|
|
|
|
The Dockerfile is a two-stage build:
|
|
|
|
1. **Build stage** (`node:24-trixie`) clones DSH at the pinned git ref, installs
|
|
dependencies with pnpm, runs the production build, packages the `@deepseek-ai/dsh`
|
|
CLI with `pnpm pack`, and installs its production dependencies with npm.
|
|
2. **Runtime stage** (`node:24-trixie-slim`) copies `/opt/dsh` to `/app`, adds the
|
|
CLI tools DSH agents commonly use (`git`, `ripgrep`, `jq`, `yq`, `curl`, `wget`,
|
|
`gh`, `tini`, `uv`), installs `bubblewrap` as DSH's preferred command-sandbox
|
|
runner, and starts `dsh web` through a small entrypoint.
|
|
|
|
## Build it
|
|
|
|
```bash
|
|
make image # canonical: builds the DSH Web image, no push
|
|
```
|
|
|
|
`make image` reads [versions.env](../versions.env) and passes the pins as build args.
|
|
Pull requests that touch `Dockerfile`, `.dockerignore` or `versions.env` get the
|
|
same build for `linux/amd64` from the [`ci-image.yaml`](../.github/workflows/ci-image.yaml)
|
|
workflow — no registry login, no push — followed by a container smoke test.
|
|
To reproduce the equivalent by hand:
|
|
|
|
```bash
|
|
docker build \
|
|
--build-arg DSH_VERSION=dsh-v0.2.1-rc.5 \
|
|
--build-arg RELEASE_REVISION= \
|
|
--build-arg PNPM_VERSION=11.28.5 \
|
|
--build-arg VCS_REF="$(git rev-parse --short HEAD)" \
|
|
-t ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.5 .
|
|
```
|
|
|
|
### Build args
|
|
|
|
| Arg | Default / source | Purpose |
|
|
| --- | --- | --- |
|
|
| `DSH_REPO` | `https://github.com/deepseek-ai/deepseek-harness.git` | Upstream DSH repository to clone. |
|
|
| `DSH_VERSION` | `DSH_VERSION` in [versions.env](../versions.env) | Upstream DSH git tag/ref to package; recorded as `org.opencontainers.image.version`. |
|
|
| `RELEASE_REVISION` | `RELEASE_REVISION` in [versions.env](../versions.env) | Repository revision suffix (`rN`); the release identity is `DSH_VERSION-RELEASE_REVISION`, recorded in the `io.sredevopsorg.release.revision` label. See [versioning](versioning.md). |
|
|
| `PNPM_VERSION` | `PNPM_VERSION` in [versions.env](../versions.env) | pnpm version activated with corepack in both stages. |
|
|
| `VCS_REF` | `git rev-parse --short HEAD` (via `make image`) | Source revision recorded for traceability. |
|
|
| `NODE_BUILD_IMAGE` | `docker.io/node:24-trixie` | Build-stage base image. |
|
|
| `NODE_RUNTIME_IMAGE` | `docker.io/node:24-trixie-slim` | Runtime-stage base image. |
|
|
|
|
Never hard-code a version outside [versions.env](../versions.env):
|
|
`./scripts/check-versions.sh` (and `make check`) fails when a pin drifts.
|
|
|
|
## Runtime contract
|
|
|
|
### Entrypoint
|
|
|
|
The entrypoint is `tini -- /usr/local/bin/dsh-entrypoint`:
|
|
|
|
1. `tini` runs as PID 1 and reaps children/forwards signals.
|
|
2. The shell entrypoint applies defaults, derives a bind address when `DSH_HOST` is
|
|
empty (the pod/container's first non-loopback IPv4), builds the
|
|
`dsh web` command line, and `exec`s it so signals reach the Node process.
|
|
3. `localhost` and `127.0.0.1` are always added to DSH's trusted-host list; each
|
|
`DSH_TRUSTED_HOSTS` entry is added as an extra `--trusted-host`.
|
|
|
|
### Environment variables
|
|
|
|
| Variable | Default | Purpose |
|
|
| --- | --- | --- |
|
|
| `DSH_PORT` | `3080` | Port DSH Web listens on inside the container. |
|
|
| `DSH_HOST` | auto-detected non-loopback IPv4 | Bind address. Leave empty in Kubernetes so the pod IP is used automatically. |
|
|
| `DSH_PUBLIC_URL` | `http://localhost:${DSH_PORT}` | Public base URL used to build links and the launch-token URL printed in the logs. Set it when the container is published on a different host port or behind a proxy. |
|
|
| `DSH_TRUSTED_HOSTS` | empty | Extra `Host`/`Origin` allow-list entries, comma- or space-separated. `localhost`/`127.0.0.1` are always trusted. Required when a proxy forwards the browser's `Host`. |
|
|
| `DEEPSEEK_API_KEY` | empty | Provider credential read by DSH. Inject it with `-e`/`envFrom`; never bake it into the image. Only model calls fail when it is missing. |
|
|
| `DSH_HOME` | `/home/node/.dsh` | DSH state directory (conversations, sessions, config, credentials). Mount a volume here to persist it. |
|
|
| `DSH_PERMISSION_MODE` | `workspace-write` | Deployment default for DSH's file policy: `read-only`, `workspace-write`, or `danger-full-access` (which also turns approval prompts off). The confined modes need `bubblewrap` or Landlock inside the container; where neither is usable every bash command fails closed with `SANDBOX_UNAVAILABLE`. See the note below. |
|
|
|
|
The image also sets `NODE_ENV=production`, `HOME=/home/node`, and a `PATH` that
|
|
includes `/app/node_modules/.bin`.
|
|
|
|
DSH confines bash commands itself (a second layer inside whatever container
|
|
sandbox you already run). Its Linux runners are `bubblewrap` and then Landlock,
|
|
and it fails closed rather than running unconfined. The image ships
|
|
`bubblewrap`, but Debian's binary is unsetuid: it needs **unprivileged user
|
|
namespaces** plus the namespace/mount syscalls. A hardened pod — non-root,
|
|
`capabilities.drop: ["ALL"]`, `seccompProfile: RuntimeDefault` — usually cannot
|
|
create one, and it may expose no Landlock runner either, in which case every
|
|
command fails with `SANDBOX_UNAVAILABLE`. Set
|
|
`DSH_PERMISSION_MODE=danger-full-access` only when the surrounding pod or VM is
|
|
the isolation boundary you rely on; keep the confined modes when the host
|
|
provides a working runner (check `grep -o landlock /sys/kernel/security/lsm` and
|
|
`bwrap --ro-bind / / true` inside the container first).
|
|
|
|
### Provider credentials
|
|
|
|
DSH resolves provider API keys from the environment when no stored credential
|
|
exists. The packaged provider catalog maps each provider to its standard
|
|
environment name. Commonly used names:
|
|
|
|
`DEEPSEEK_API_KEY`, `OPENROUTER_API_KEY`, `GEMINI_API_KEY`, `NVIDIA_API_KEY`,
|
|
`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GROQ_API_KEY`, `XAI_API_KEY`,
|
|
`MISTRAL_API_KEY`, `TOGETHER_API_KEY`, `CEREBRAS_API_KEY`,
|
|
`AI_GATEWAY_API_KEY`, `HF_TOKEN`.
|
|
|
|
Use `GEMINI_API_KEY` for Google Gemini (not `GOOGLE_API_KEY`), `HF_TOKEN` for
|
|
Hugging Face, and `AI_GATEWAY_API_KEY` for the Vercel AI Gateway. Inject keys
|
|
with `-e`/`--env-file` in Docker or a Secret + `envFrom` in Kubernetes; never
|
|
bake them into the image.
|
|
|
|
A key saved through DSH's credential store (`$DSH_HOME/.credentials.yaml`) wins
|
|
over the environment. The
|
|
[agents-as-a-service example](../examples/dsh-agents-as-a-service/docs/optional-hardening.md)
|
|
documents the optional `dsh-provider-keys-extra` Secret and the pod-refresh
|
|
procedure.
|
|
|
|
### Ports, volumes and signals
|
|
|
|
| Item | Value | Notes |
|
|
| --- | --- | --- |
|
|
| Exposed port | `3080/tcp` | `EXPOSE 3080`; one port serves the UI, assets, API, SSE and WebSocket traffic. |
|
|
| State volume | `/home/node/.dsh` | Declared with `VOLUME`; use a named volume (Docker) or PVC (Kubernetes). |
|
|
| Working directory | `/workspace` | The agent workspace. Mount a volume here if agents should keep files; the agents-as-a-service example mounts the same PVC's `workspace` subPath here. |
|
|
| Stop signal | `SIGTERM` | `tini` forwards it to `dsh web`; allow a graceful shutdown period. |
|
|
| User | `node`, UID/GID `1000` | The process is non-root. A PVC mounted at `DSH_HOME` must be writable by UID 1000. |
|
|
|
|
### Health probes
|
|
|
|
`/` is **not** a health endpoint: DSH returns `401` until a browser completes the
|
|
launch-token handshake. Probe `/favicon.svg` instead — it is unauthenticated and
|
|
served by the web layer, so it proves the HTTP server is up without triggering auth.
|
|
|
|
```yaml
|
|
startupProbe:
|
|
httpGet: {path: /favicon.svg, port: 3080}
|
|
periodSeconds: 5
|
|
failureThreshold: 24
|
|
readinessProbe:
|
|
httpGet: {path: /favicon.svg, port: 3080}
|
|
periodSeconds: 5
|
|
livenessProbe:
|
|
httpGet: {path: /favicon.svg, port: 3080}
|
|
periodSeconds: 15
|
|
```
|
|
|
|
The shipped templates use exactly this shape; see
|
|
[`examples/minimal-single-tenant/manifests/sandbox-template.yaml`](../examples/minimal-single-tenant/manifests/sandbox-template.yaml).
|
|
A `startupProbe` is important because plugin activation (or the first start of a
|
|
large sandbox) can exceed a default liveness budget.
|
|
|
|
### Launch token
|
|
|
|
DSH prints its one-time launch URL to the logs:
|
|
|
|
```
|
|
dsh web: http://<host>:${DSH_PORT}/?token=...
|
|
```
|
|
|
|
Open that URL once to obtain DSH's auth cookie. After that the UI works at the bare
|
|
URL; without the cookie `/` answers `401` by design. The
|
|
[agents-as-a-service gateway](../examples/dsh-agents-as-a-service/gateway/) performs
|
|
this handshake server-side so browsers never see the token.
|
|
|
|
## Published image names and tags
|
|
|
|
Registry layout: the main image is the repository itself, and every example
|
|
component is a nested package below it:
|
|
|
|
```
|
|
ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox # main DSH Web image
|
|
ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway # example component
|
|
```
|
|
|
|
The main image is published multi-arch (`linux/amd64`, `linux/arm64`) to
|
|
GitHub Container Registry:
|
|
|
|
| Tag | Meaning |
|
|
| --- | --- |
|
|
| `dsh-v0.2.1-rc.5` | Current release identity. **Immutable**: once published it never moves. |
|
|
| `latest` | Most recent release; points at the same digest as the current identity. |
|
|
| `main` | Development channel rebuilt from every push to `main`; not a release, not for production. |
|
|
|
|
Tags published before the release-identity scheme remain published but are not
|
|
updated; new deployments use the current identity.
|
|
|
|
Release identities, channels and rollback rules are defined in
|
|
[versioning](versioning.md). Tags are immutable, so production deployments should
|
|
pin a digest:
|
|
|
|
```bash
|
|
docker buildx imagetools inspect ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.5
|
|
# ... pin the digest in your deployment manifests:
|
|
# image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox@sha256:<digest>
|
|
```
|
|
|
|
## Verifying an image
|
|
|
|
```bash
|
|
docker inspect ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.5 \
|
|
--format '{{json .Config.Labels}}'
|
|
# org.opencontainers.image.title / .version / .source
|
|
```
|
|
|
|
Release builds also publish SBOM and provenance attestations from the release
|
|
workflow (see [SECURITY.md](../SECURITY.md) and [gitflow](gitflow.md)).
|