mirror of
https://github.com/sredevopsorg/deepseek-kubernetes-sandbox.git
synced 2026-11-03 01:13:24 +00:00
Compare commits
5 Commits
dsh-v0.2.1
...
dsh-v0.2.1
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
bed8e5675a | ||
|
|
14c3fb840e | ||
|
|
96d3030a49 | ||
|
|
2b43d5dfd5 | ||
|
|
97abf75dfe |
@@ -104,8 +104,9 @@ Verified from the upstream server:
|
|||||||
`shutdown_after_seconds` defaults to 300 s (`null` disables it).
|
`shutdown_after_seconds` defaults to 300 s (`null` disables it).
|
||||||
- **Output:** servers before v1.0.8 return unbounded `execute_command` output.
|
- **Output:** servers before v1.0.8 return unbounded `execute_command` output.
|
||||||
v1.0.8 adds `max_output_chars` (128-300,000, default 30,000) and truncation
|
v1.0.8 adds `max_output_chars` (128-300,000, default 30,000) and truncation
|
||||||
metadata. Never pass the new argument to an older server; redirect verbose
|
metadata. The repository pins v1.0.8 (`AGENT_SANDBOX_VERSION`), so the
|
||||||
output and read a file instead.
|
documented deployment is bounded; never pass the new argument to an older
|
||||||
|
server, and on one redirect verbose output and read a file instead.
|
||||||
|
|
||||||
Deployment, connection matrix and DSH row:
|
Deployment, connection matrix and DSH row:
|
||||||
[MCP deployment](references/mcp-deployment.md). Limits and cleanup:
|
[MCP deployment](references/mcp-deployment.md). Limits and cleanup:
|
||||||
|
|||||||
@@ -74,6 +74,10 @@ DSH template.
|
|||||||
Never pass `max_output_chars` to a pre-v1.0.8 server. On older servers, redirect
|
Never pass `max_output_chars` to a pre-v1.0.8 server. On older servers, redirect
|
||||||
verbose output to a file and `download_file` it, or pipe through `tail`/`head`.
|
verbose output to a file and `download_file` it, or pipe through `tail`/`head`.
|
||||||
|
|
||||||
|
The repository pins `AGENT_SANDBOX_VERSION=v1.0.8`, so the right-hand column is
|
||||||
|
what the documented platform and a same-tag MCP build provide; keep the
|
||||||
|
before-column for clusters still on v1.0.7.
|
||||||
|
|
||||||
## Safety and cleanup
|
## Safety and cleanup
|
||||||
|
|
||||||
- No auth: anyone reaching `:8000` gets every tool and the service account's
|
- No auth: anyone reaching `:8000` gets every tool and the service account's
|
||||||
|
|||||||
@@ -20,6 +20,7 @@ events, then compare desired vs observed state. Keep output bounded. Never
|
|||||||
| Restart with mode-660 error | logs mention `.credentials.yaml` | kubelet fsGroup widened the file; suspend, `chmod 600` from a throwaway pod, patch `fsGroupChangePolicy: OnRootMismatch`, resume. |
|
| Restart with mode-660 error | logs mention `.credentials.yaml` | kubelet fsGroup widened the file; suspend, `chmod 600` from a throwaway pod, patch `fsGroupChangePolicy: OnRootMismatch`, resume. |
|
||||||
| PVC `Pending` | `kubectl describe pvc`, StorageClass | No default provisioner, access mode, zone binding. |
|
| PVC `Pending` | `kubectl describe pvc`, StorageClass | No default provisioner, access mode, zone binding. |
|
||||||
| Launch token not found | `kubectl logs <pod> --tail=2000 | grep 'dsh web:'` | Pod not ready yet, or logs rotated. Do not guess tokens; wait for readiness. |
|
| Launch token not found | `kubectl logs <pod> --tail=2000 | grep 'dsh web:'` | Pod not ready yet, or logs rotated. Do not guess tokens; wait for readiness. |
|
||||||
|
| Sandbox reports `Finished=True` | pod phase, `kubectl get events` | The backing Pod reached `Failed` (eviction, spot reclaim) and `failedPodPolicy` defaults to `Retain`. Delete the Pod to force a replacement, or set `Recreate` (v1.0.8+) on the Sandbox/template; the claim and PVCs survive either way. |
|
||||||
| Hand-applied sandbox disappeared | claim annotations, gateway logs | Idle sweeper only manages claims with `aaas.example.com/managed-by=dsh-gateway`; if it was managed, recreate/resume. |
|
| Hand-applied sandbox disappeared | claim annotations, gateway logs | Idle sweeper only manages claims with `aaas.example.com/managed-by=dsh-gateway`; if it was managed, recreate/resume. |
|
||||||
| Gateway 401 | gateway logs, cookie flags | Missing/expired cookie, DSH handshake failed, or `COOKIE_SECURE=true` over HTTP. |
|
| Gateway 401 | gateway logs, cookie flags | Missing/expired cookie, DSH handshake failed, or `COOKIE_SECURE=true` over HTTP. |
|
||||||
| Gateway 502/503 | sandbox Ready, gateway logs | Wake timeout, stale DNS, token not yet in logs; check `WAKE_TIMEOUT` and RBAC `pods/log`. |
|
| Gateway 502/503 | sandbox Ready, gateway logs | Wake timeout, stale DNS, token not yet in logs; check `WAKE_TIMEOUT` and RBAC `pods/log`. |
|
||||||
|
|||||||
@@ -20,8 +20,8 @@ TOGETHER_API_KEY=
|
|||||||
CEREBRAS_API_KEY=
|
CEREBRAS_API_KEY=
|
||||||
AI_GATEWAY_API_KEY=
|
AI_GATEWAY_API_KEY=
|
||||||
HF_TOKEN=
|
HF_TOKEN=
|
||||||
DSH_VERSION=dsh-v0.2.1-rc.3
|
DSH_VERSION=dsh-v0.2.1-rc.5
|
||||||
RELEASE_REVISION=r4
|
RELEASE_REVISION=
|
||||||
PNPM_VERSION=11.28.5
|
PNPM_VERSION=11.28.5
|
||||||
# Container port, and the host port published by "make run".
|
# Container port, and the host port published by "make run".
|
||||||
DSH_PORT=3080
|
DSH_PORT=3080
|
||||||
|
|||||||
2
.github/ISSUE_TEMPLATE/bug_report.yml
vendored
2
.github/ISSUE_TEMPLATE/bug_report.yml
vendored
@@ -28,7 +28,7 @@ body:
|
|||||||
id: image-tag
|
id: image-tag
|
||||||
attributes:
|
attributes:
|
||||||
label: Image or release
|
label: Image or release
|
||||||
description: The pinned tag or digest, for example dsh-v0.2.1-rc.3-r4.
|
description: The pinned tag or digest, for example dsh-v0.2.1-rc.5.
|
||||||
- type: input
|
- type: input
|
||||||
id: cluster
|
id: cluster
|
||||||
attributes:
|
attributes:
|
||||||
|
|||||||
24
.github/workflows/_build-image.yaml
vendored
24
.github/workflows/_build-image.yaml
vendored
@@ -60,8 +60,17 @@ jobs:
|
|||||||
run: |
|
run: |
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
pair=$(printf '%s' "${{ matrix.platform }}" | tr '/' '-')
|
pair=$(printf '%s' "${{ matrix.platform }}" | tr '/' '-')
|
||||||
echo "PLATFORM_PAIR=$pair" >> "$GITHUB_ENV"
|
# This reusable workflow runs once for the main image and once for the
|
||||||
echo "CACHE_SCOPE=$(printf '%s' "$IMAGE" | tr '/' '-')" >> "$GITHUB_ENV"
|
# gateway in the same release run. Artifact names must include a
|
||||||
|
# unique token for the image: the repository path is a prefix of the
|
||||||
|
# gateway path, so a path-derived pattern would match both. Use a
|
||||||
|
# short hash of the full image name.
|
||||||
|
image_hash=$(printf '%s' "$IMAGE" | sha256sum | cut -c1-12)
|
||||||
|
{
|
||||||
|
echo "PLATFORM_PAIR=$pair"
|
||||||
|
echo "CACHE_SCOPE=$(printf '%s' "$IMAGE" | tr '/' '-')"
|
||||||
|
echo "ARTIFACT_PREFIX=$image_hash"
|
||||||
|
} >> "$GITHUB_ENV"
|
||||||
|
|
||||||
- name: Set up Docker Buildx
|
- name: Set up Docker Buildx
|
||||||
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
|
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
|
||||||
@@ -101,7 +110,7 @@ jobs:
|
|||||||
- name: Upload digest
|
- name: Upload digest
|
||||||
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
|
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
|
||||||
with:
|
with:
|
||||||
name: digests-${{ env.PLATFORM_PAIR }}
|
name: digests-${{ env.ARTIFACT_PREFIX }}-${{ env.PLATFORM_PAIR }}
|
||||||
path: /tmp/digests/*
|
path: /tmp/digests/*
|
||||||
if-no-files-found: error
|
if-no-files-found: error
|
||||||
retention-days: 1
|
retention-days: 1
|
||||||
@@ -112,11 +121,18 @@ jobs:
|
|||||||
runs-on: ubuntu-latest
|
runs-on: ubuntu-latest
|
||||||
timeout-minutes: 30
|
timeout-minutes: 30
|
||||||
steps:
|
steps:
|
||||||
|
- name: Prepare environment
|
||||||
|
env:
|
||||||
|
IMAGE: ${{ inputs.image }}
|
||||||
|
run: |
|
||||||
|
set -euo pipefail
|
||||||
|
echo "ARTIFACT_PREFIX=$(printf '%s' "$IMAGE" | sha256sum | cut -c1-12)" >> "$GITHUB_ENV"
|
||||||
|
|
||||||
- name: Download digests
|
- name: Download digests
|
||||||
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
|
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
|
||||||
with:
|
with:
|
||||||
path: /tmp/digests
|
path: /tmp/digests
|
||||||
pattern: digests-*
|
pattern: digests-${{ env.ARTIFACT_PREFIX }}-*
|
||||||
merge-multiple: true
|
merge-multiple: true
|
||||||
|
|
||||||
- name: Set up Docker Buildx
|
- name: Set up Docker Buildx
|
||||||
|
|||||||
3
.github/workflows/release.yaml
vendored
3
.github/workflows/release.yaml
vendored
@@ -205,6 +205,9 @@ jobs:
|
|||||||
needs: [meta, build-main, build-gateway]
|
needs: [meta, build-main, build-gateway]
|
||||||
runs-on: ubuntu-latest
|
runs-on: ubuntu-latest
|
||||||
timeout-minutes: 20
|
timeout-minutes: 20
|
||||||
|
permissions:
|
||||||
|
contents: write
|
||||||
|
pull-requests: write
|
||||||
steps:
|
steps:
|
||||||
- name: Checkout
|
- name: Checkout
|
||||||
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
||||||
|
|||||||
@@ -83,10 +83,10 @@ needs kind plus the agent-sandbox CRDs (see the example README).
|
|||||||
|
|
||||||
- Release identity = DSH_VERSION, optionally with a repository revision suffix
|
- Release identity = DSH_VERSION, optionally with a repository revision suffix
|
||||||
for re-releases at the same DSH ref (current identity:
|
for re-releases at the same DSH ref (current identity:
|
||||||
dsh-v0.2.1-rc.3-r4).
|
dsh-v0.2.1-rc.5).
|
||||||
- The current identity is dsh-v0.2.1-rc.3-r4 (stored in versions.env). The
|
- The current identity is dsh-v0.2.1-rc.5 (stored in versions.env). It is the
|
||||||
dsh-v0.2.1-rc.3 image tag already exists and tags are immutable, which is why
|
first release at this DSH ref, so it carries no revision suffix; a
|
||||||
the first release under this scheme carries -r2.
|
repository-only change at the same ref publishes -r2.
|
||||||
- Channels: release tags are immutable; latest follows the newest release;
|
- Channels: release tags are immutable; latest follows the newest release;
|
||||||
main is the development channel for both image packages.
|
main is the development channel for both image packages.
|
||||||
- A broken release is fixed by publishing -r(N+1), never by re-tagging.
|
- A broken release is fixed by publishing -r(N+1), never by re-tagging.
|
||||||
|
|||||||
12
CHANGELOG.md
12
CHANGELOG.md
@@ -3,6 +3,18 @@
|
|||||||
All notable changes to this repository are documented here.
|
All notable changes to this repository are documented here.
|
||||||
Generated by git-cliff from Conventional Commits; do not edit this file by hand.
|
Generated by git-cliff from Conventional Commits; do not edit this file by hand.
|
||||||
|
|
||||||
|
## [dsh-v0.2.1-rc.3-r4] - 2026-10-26
|
||||||
|
|
||||||
|
### Added
|
||||||
|
|
||||||
|
- **skills:** Add agent-sandbox platform skill (#20) (bf24306)
|
||||||
|
- **example:** Add optional gVisor, nodeSelector, NetworkPolicy and provider keys (#21) (3e9ba38)
|
||||||
|
|
||||||
|
### Maintenance
|
||||||
|
|
||||||
|
- **changelog:** Refresh for dsh-v0.2.1-rc.3-r3 (#19) (8681412)
|
||||||
|
- **release:** Dsh-v0.2.1-rc.3-r4 (#22) (e2f1fe5)
|
||||||
|
|
||||||
## [dsh-v0.2.1-rc.3-r3] - 2026-10-25
|
## [dsh-v0.2.1-rc.3-r3] - 2026-10-25
|
||||||
|
|
||||||
### Added
|
### Added
|
||||||
|
|||||||
@@ -20,7 +20,7 @@ ARG NODE_RUNTIME_IMAGE=docker.io/node:24-trixie-slim
|
|||||||
FROM $NODE_BUILD_IMAGE AS build
|
FROM $NODE_BUILD_IMAGE AS build
|
||||||
|
|
||||||
ARG DSH_REPO=https://github.com/deepseek-ai/deepseek-harness.git
|
ARG DSH_REPO=https://github.com/deepseek-ai/deepseek-harness.git
|
||||||
ARG DSH_VERSION=dsh-v0.2.1-rc.3
|
ARG DSH_VERSION=dsh-v0.2.1-rc.5
|
||||||
ARG PNPM_VERSION=11.28.5
|
ARG PNPM_VERSION=11.28.5
|
||||||
|
|
||||||
ENV CI=true \
|
ENV CI=true \
|
||||||
@@ -57,8 +57,8 @@ RUN cd /opt/dsh \
|
|||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
FROM $NODE_RUNTIME_IMAGE AS runtime
|
FROM $NODE_RUNTIME_IMAGE AS runtime
|
||||||
|
|
||||||
ARG DSH_VERSION=dsh-v0.2.1-rc.3
|
ARG DSH_VERSION=dsh-v0.2.1-rc.5
|
||||||
ARG RELEASE_REVISION=r4
|
ARG RELEASE_REVISION=
|
||||||
ARG PNPM_VERSION=11.28.5
|
ARG PNPM_VERSION=11.28.5
|
||||||
ARG VCS_REF=unknown
|
ARG VCS_REF=unknown
|
||||||
|
|
||||||
|
|||||||
21
README.md
21
README.md
@@ -20,7 +20,7 @@ docker run -d --name dsh-web \
|
|||||||
-p 3080:3080 \
|
-p 3080:3080 \
|
||||||
-v dsh-home:/home/node/.dsh \
|
-v dsh-home:/home/node/.dsh \
|
||||||
-e DEEPSEEK_API_KEY \
|
-e DEEPSEEK_API_KEY \
|
||||||
ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.3-r4
|
ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.5
|
||||||
```
|
```
|
||||||
|
|
||||||
DSH Web is **not usable at a bare `/`**. Without its launch-token cookie it
|
DSH Web is **not usable at a bare `/`**. Without its launch-token cookie it
|
||||||
@@ -45,7 +45,7 @@ docker run -d --name dsh-web -p 18081:3080 \
|
|||||||
-e DSH_PUBLIC_URL=http://localhost:18081 \
|
-e DSH_PUBLIC_URL=http://localhost:18081 \
|
||||||
-v dsh-home:/home/node/.dsh \
|
-v dsh-home:/home/node/.dsh \
|
||||||
-e DEEPSEEK_API_KEY \
|
-e DEEPSEEK_API_KEY \
|
||||||
ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.3-r4
|
ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.5
|
||||||
# then open http://localhost:18081/?token=...
|
# then open http://localhost:18081/?token=...
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -94,21 +94,23 @@ ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway
|
|||||||
|
|
||||||
| Tag | Contents | Meaning |
|
| Tag | Contents | Meaning |
|
||||||
| --- | --- | --- |
|
| --- | --- | --- |
|
||||||
| `dsh-v0.2.1-rc.3-r4` | DSH Web, port `3080` | **Current release identity.** Immutable: once published it never moves. |
|
| `dsh-v0.2.1-rc.5` | DSH Web, port `3080` | **Current release identity.** Immutable: once published it never moves. |
|
||||||
| `latest` | DSH Web | Most recent release; same digest as the current identity. |
|
| `latest` | DSH Web | Most recent release; same digest as the current identity. |
|
||||||
| `main` | DSH Web | Development channel rebuilt from every push to `main`. Not a release; not for production. |
|
| `main` | DSH Web | Development channel rebuilt from every push to `main`. Not a release; not for production. |
|
||||||
| `dsh-v0.2.1-rc.3` | DSH Web | **Legacy tag** published before the release-identity scheme. It is not updated. New deployments use `dsh-v0.2.1-rc.3-r4`. |
|
|
||||||
|
Tags published before the release-identity scheme remain on GHCR but are not
|
||||||
|
updated; new deployments use the current identity.
|
||||||
|
|
||||||
The example gateway is published under its nested path with the same release
|
The example gateway is published under its nested path with the same release
|
||||||
identity, e.g.
|
identity, e.g.
|
||||||
`ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway:dsh-v0.2.1-rc.3-r4`.
|
`ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway:dsh-v0.2.1-rc.5`.
|
||||||
|
|
||||||
Release identities, channels and rollback rules are explained in
|
Release identities, channels and rollback rules are explained in
|
||||||
[docs/versioning.md](docs/versioning.md). Production deployments should pin a
|
[docs/versioning.md](docs/versioning.md). Production deployments should pin a
|
||||||
digest:
|
digest:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker buildx imagetools inspect ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.3-r4
|
docker buildx imagetools inspect ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.5
|
||||||
# image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox@sha256:<digest>
|
# image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox@sha256:<digest>
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -189,10 +191,9 @@ required checks and releases: [docs/gitflow.md](docs/gitflow.md).
|
|||||||
|
|
||||||
The release identity is the upstream DSH ref plus an optional repository
|
The release identity is the upstream DSH ref plus an optional repository
|
||||||
revision: `DSH_VERSION` and `RELEASE_REVISION` in [versions.env](versions.env).
|
revision: `DSH_VERSION` and `RELEASE_REVISION` in [versions.env](versions.env).
|
||||||
The current identity is **`dsh-v0.2.1-rc.3-r4`** — the first release under the
|
The current identity is **`dsh-v0.2.1-rc.5`** — the first release at this
|
||||||
mirrored-DSH scheme at the same upstream DSH ref as the legacy
|
upstream DSH ref, so it carries no repository revision. Tags are immutable; a
|
||||||
`dsh-v0.2.1-rc.3` tag. Tags are immutable; a broken release is fixed by
|
broken release is fixed by publishing `-r(N+1)`, never by re-tagging.
|
||||||
publishing `-r(N+1)`, never by re-tagging.
|
|
||||||
|
|
||||||
`CHANGELOG.md` is generated by [git-cliff](https://git-cliff.org) from
|
`CHANGELOG.md` is generated by [git-cliff](https://git-cliff.org) from
|
||||||
Conventional Commits and is never edited by hand. See
|
Conventional Commits and is never edited by hand. See
|
||||||
|
|||||||
@@ -15,7 +15,7 @@ gets a DSH Web container running before anything else.
|
|||||||
| [main-image.md](main-image.md) | The primary artifact: build args, runtime environment, entrypoint, probes, volumes/ports/signals, non-root contract, image names, tags and digests. |
|
| [main-image.md](main-image.md) | The primary artifact: build args, runtime environment, entrypoint, probes, volumes/ports/signals, non-root contract, image names, tags and digests. |
|
||||||
| [examples.md](examples.md) | The example contract: required layout, manifest rules, exposure variants, components and image naming, scripts, docs, and how to add a new example. |
|
| [examples.md](examples.md) | The example contract: required layout, manifest rules, exposure variants, components and image naming, scripts, docs, and how to add a new example. |
|
||||||
| [gitflow.md](gitflow.md) | Branches, protected `main`, squash merges, Conventional PR titles, required CI checks, release branches, hotfixes and the one-time GitHub settings checklist. |
|
| [gitflow.md](gitflow.md) | Branches, protected `main`, squash merges, Conventional PR titles, required CI checks, release branches, hotfixes and the one-time GitHub settings checklist. |
|
||||||
| [versioning.md](versioning.md) | Release identity (`DSH ref` + optional `-rN`), why the current identity is `dsh-v0.2.1-rc.3-r4`, channels, compatibility matrix, bumping pins, the release checklist, rollback, git-cliff and digest pinning. |
|
| [versioning.md](versioning.md) | Release identity (`DSH ref` + optional `-rN`), the current identity `dsh-v0.2.1-rc.5`, why identities are immutable, channels, compatibility matrix, bumping pins, the release checklist, rollback, git-cliff and digest pinning. |
|
||||||
|
|
||||||
## Project skills
|
## Project skills
|
||||||
|
|
||||||
|
|||||||
@@ -31,11 +31,11 @@ To reproduce the equivalent by hand:
|
|||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker build \
|
docker build \
|
||||||
--build-arg DSH_VERSION=dsh-v0.2.1-rc.3 \
|
--build-arg DSH_VERSION=dsh-v0.2.1-rc.5 \
|
||||||
--build-arg RELEASE_REVISION=r4 \
|
--build-arg RELEASE_REVISION= \
|
||||||
--build-arg PNPM_VERSION=11.28.5 \
|
--build-arg PNPM_VERSION=11.28.5 \
|
||||||
--build-arg VCS_REF="$(git rev-parse --short HEAD)" \
|
--build-arg VCS_REF="$(git rev-parse --short HEAD)" \
|
||||||
-t ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.3-r4 .
|
-t ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.5 .
|
||||||
```
|
```
|
||||||
|
|
||||||
### Build args
|
### Build args
|
||||||
@@ -164,17 +164,19 @@ GitHub Container Registry:
|
|||||||
|
|
||||||
| Tag | Meaning |
|
| Tag | Meaning |
|
||||||
| --- | --- |
|
| --- | --- |
|
||||||
| `dsh-v0.2.1-rc.3-r4` | Current release identity. **Immutable**: once published it never moves. |
|
| `dsh-v0.2.1-rc.5` | Current release identity. **Immutable**: once published it never moves. |
|
||||||
| `latest` | Most recent release; points at the same digest as the current identity. |
|
| `latest` | Most recent release; points at the same digest as the current identity. |
|
||||||
| `main` | Development channel rebuilt from every push to `main`; not a release, not for production. |
|
| `main` | Development channel rebuilt from every push to `main`; not a release, not for production. |
|
||||||
| `dsh-v0.2.1-rc.3` | Legacy tag published before the release-identity scheme existed. Do not use it for new deployments; see the migration table in the [root README](../README.md). |
|
|
||||||
|
Tags published before the release-identity scheme remain published but are not
|
||||||
|
updated; new deployments use the current identity.
|
||||||
|
|
||||||
Release identities, channels and rollback rules are defined in
|
Release identities, channels and rollback rules are defined in
|
||||||
[versioning](versioning.md). Tags are immutable, so production deployments should
|
[versioning](versioning.md). Tags are immutable, so production deployments should
|
||||||
pin a digest:
|
pin a digest:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker buildx imagetools inspect ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.3-r4
|
docker buildx imagetools inspect ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.5
|
||||||
# ... pin the digest in your deployment manifests:
|
# ... pin the digest in your deployment manifests:
|
||||||
# image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox@sha256:<digest>
|
# image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox@sha256:<digest>
|
||||||
```
|
```
|
||||||
@@ -182,7 +184,7 @@ docker buildx imagetools inspect ghcr.io/sredevopsorg/deepseek-kubernetes-sandbo
|
|||||||
## Verifying an image
|
## Verifying an image
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker inspect ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.3-r4 \
|
docker inspect ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.5 \
|
||||||
--format '{{json .Config.Labels}}'
|
--format '{{json .Config.Labels}}'
|
||||||
# org.opencontainers.image.title / .version / .source
|
# org.opencontainers.image.title / .version / .source
|
||||||
```
|
```
|
||||||
|
|||||||
@@ -15,34 +15,35 @@ release identity = DSH_VERSION + optional "-" + RELEASE_REVISION
|
|||||||
|
|
||||||
| Piece | Meaning |
|
| Piece | Meaning |
|
||||||
| --- | --- |
|
| --- | --- |
|
||||||
| `DSH_VERSION` | The upstream DeepSeek Harness git ref being packaged, e.g. `dsh-v0.2.1-rc.3`. It mirrors upstream's own tag, which is what "mirrored-DSH scheme" means. |
|
| `DSH_VERSION` | The upstream DeepSeek Harness git ref being packaged, e.g. `dsh-v0.2.1-rc.5`. It mirrors upstream's own tag, which is what "mirrored-DSH scheme" means. |
|
||||||
| `RELEASE_REVISION` | An optional repository revision at that DSH ref: empty for the first release, `rN` (`N >= 2`) for a repository-only re-release. |
|
| `RELEASE_REVISION` | An optional repository revision at that DSH ref: empty for the first release, `rN` (`N >= 2`) for a repository-only re-release. |
|
||||||
|
|
||||||
Examples:
|
Examples:
|
||||||
|
|
||||||
- `DSH_VERSION=dsh-v0.2.1-rc.3`, `RELEASE_REVISION=` → identity `dsh-v0.2.1-rc.3`
|
- `DSH_VERSION=dsh-v0.2.1-rc.5`, `RELEASE_REVISION=` → identity `dsh-v0.2.1-rc.5`
|
||||||
(first release at that upstream ref).
|
(first release at that upstream ref).
|
||||||
- `DSH_VERSION=dsh-v0.2.1-rc.3`, `RELEASE_REVISION=r4` → identity
|
- `DSH_VERSION=<ref>`, `RELEASE_REVISION=rN` → identity `<ref>-rN` (for example
|
||||||
`dsh-v0.2.1-rc.3-r4` (documentation, CI or gateway changes on top of the same
|
documentation, CI or gateway changes on top of an already-released ref; no DSH
|
||||||
upstream DSH; no DSH upgrade).
|
upgrade). Use `r2` for the first re-release: revision 1 is the plain tag.
|
||||||
|
|
||||||
A DSH upgrade produces a new base identity (for example the next upstream ref
|
A DSH upgrade produces a new base identity (for example the next upstream ref
|
||||||
with an empty revision); a repository-only change at the same DSH ref increments
|
with an empty revision); a repository-only change at the same DSH ref increments
|
||||||
`rN`.
|
`rN`.
|
||||||
|
|
||||||
## Why the current identity is `dsh-v0.2.1-rc.3-r4`
|
## Why release identities are immutable
|
||||||
|
|
||||||
The image tag `dsh-v0.2.1-rc.3` was already published to GHCR for an earlier
|
**Container tags are immutable** in this project: re-pushing a tag would silently
|
||||||
repository state. **Container tags are immutable** in this project: re-pushing a
|
change what an existing deployment runs. A DSH upgrade therefore produces a new
|
||||||
tag would silently change what an existing deployment runs. The mirrored-DSH
|
base identity with an empty revision; a repository-only change at an
|
||||||
scheme therefore starts its first identity at `-r2`: the plain tag is
|
already-released ref publishes `-rN`, where `r2` is the first re-release (the
|
||||||
conceptually revision 1 and is taken, so the first publishable repository
|
plain tag is conceptually revision 1 and is taken).
|
||||||
revision at the same upstream ref is `r2`.
|
|
||||||
|
|
||||||
The identity is stored in [versions.env](../versions.env) and consumed everywhere:
|
The current identity is `dsh-v0.2.1-rc.5`, the first release at this upstream
|
||||||
the Dockerfile label, the example manifests, the Makefile and the release
|
ref, so it carries no revision suffix. The identity is stored in
|
||||||
workflow. The legacy `dsh-v0.2.1-rc.3` tag remains published but is not updated;
|
[versions.env](../versions.env) and consumed everywhere: the Dockerfile label,
|
||||||
new deployments use `dsh-v0.2.1-rc.3-r4`.
|
the example manifests, the Makefile and the release workflow. Tags published
|
||||||
|
before the release-identity scheme remain published but are not updated; new
|
||||||
|
deployments use the current identity.
|
||||||
|
|
||||||
The gateway image moved from the repository-level `:gateway` tag to a nested
|
The gateway image moved from the repository-level `:gateway` tag to a nested
|
||||||
package (`<repository>/dsh-agents-as-a-service/gateway`). The release workflow
|
package (`<repository>/dsh-agents-as-a-service/gateway`). The release workflow
|
||||||
@@ -54,7 +55,7 @@ should pin the nested package.
|
|||||||
|
|
||||||
| Channel | Image tag | Meaning |
|
| Channel | Image tag | Meaning |
|
||||||
| --- | --- | --- |
|
| --- | --- | --- |
|
||||||
| Release | `dsh-v0.2.1-rc.3-r4` | The current immutable release identity. What production should pin (preferably by digest). |
|
| Release | `dsh-v0.2.1-rc.5` | The current immutable release identity. What production should pin (preferably by digest). |
|
||||||
| `latest` | `latest` | Alias for the most recent release; moves only when a new release is published. |
|
| `latest` | `latest` | Alias for the most recent release; moves only when a new release is published. |
|
||||||
| `main` | `main` | Development channel, rebuilt from every push to `main`. Never treat it as a release; it exists so `main` can be tested. |
|
| `main` | `main` | Development channel, rebuilt from every push to `main`. Never treat it as a release; it exists so `main` can be tested. |
|
||||||
|
|
||||||
@@ -68,10 +69,10 @@ Values from [versions.env](../versions.env) at the current identity:
|
|||||||
| Component | Pin | Source |
|
| Component | Pin | Source |
|
||||||
| --- | --- | --- |
|
| --- | --- | --- |
|
||||||
| Upstream DSH source | `DSH_REPO` | [versions.env](../versions.env) |
|
| Upstream DSH source | `DSH_REPO` | [versions.env](../versions.env) |
|
||||||
| DSH ref | `dsh-v0.2.1-rc.3` | `DSH_VERSION` |
|
| DSH ref | `dsh-v0.2.1-rc.5` | `DSH_VERSION` |
|
||||||
| Repository revision | `r4` | `RELEASE_REVISION` |
|
| Repository revision | `(empty)` | `RELEASE_REVISION` |
|
||||||
| pnpm | `11.28.5` | `PNPM_VERSION` |
|
| pnpm | `11.28.5` | `PNPM_VERSION` |
|
||||||
| agent-sandbox (CRDs) | `v1.0.7` | `AGENT_SANDBOX_VERSION`, used by the examples and CI dry-run |
|
| agent-sandbox (CRDs) | `v1.0.8` | `AGENT_SANDBOX_VERSION`, used by the examples and CI dry-run |
|
||||||
| Kubernetes Gateway API | `v1.2.1` | `GATEWAY_API_VERSION`, used by the SSO exposure variant |
|
| Kubernetes Gateway API | `v1.2.1` | `GATEWAY_API_VERSION`, used by the SSO exposure variant |
|
||||||
| oauth2-proxy | `quay.io/oauth2-proxy/oauth2-proxy:v7.15.5` | `OAUTH2_PROXY_IMAGE` |
|
| oauth2-proxy | `quay.io/oauth2-proxy/oauth2-proxy:v7.15.5` | `OAUTH2_PROXY_IMAGE` |
|
||||||
| Main image | `ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox` | `GHCR_IMAGE` |
|
| Main image | `ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox` | `GHCR_IMAGE` |
|
||||||
@@ -153,7 +154,7 @@ Release identities are mutable only in theory; deployments should still pin the
|
|||||||
resolved digest for reproducibility and supply-chain safety:
|
resolved digest for reproducibility and supply-chain safety:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker buildx imagetools inspect ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.3-r4
|
docker buildx imagetools inspect ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.5
|
||||||
# then, in the deployment:
|
# then, in the deployment:
|
||||||
# image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox@sha256:<digest>
|
# image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox@sha256:<digest>
|
||||||
```
|
```
|
||||||
|
|||||||
@@ -83,7 +83,7 @@ What it demonstrates, feature by feature:
|
|||||||
- agent-sandbox with extensions installed:
|
- agent-sandbox with extensions installed:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
VERSION=v1.0.7 # AGENT_SANDBOX_VERSION in versions.env
|
VERSION=v1.0.8 # AGENT_SANDBOX_VERSION in versions.env
|
||||||
kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/download/$VERSION/sandbox-with-extensions.yaml
|
kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/download/$VERSION/sandbox-with-extensions.yaml
|
||||||
kubectl -n agent-sandbox-system rollout status deployment --timeout=180s
|
kubectl -n agent-sandbox-system rollout status deployment --timeout=180s
|
||||||
```
|
```
|
||||||
@@ -244,7 +244,7 @@ kubectl delete -f manifests/40-claim-rejected.yaml
|
|||||||
```bash
|
```bash
|
||||||
# Published image (on kind, load it into the cluster; on a real cluster the
|
# Published image (on kind, load it into the cluster; on a real cluster the
|
||||||
# kubelet pulls it from GHCR):
|
# kubelet pulls it from GHCR):
|
||||||
GW=ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway:dsh-v0.2.1-rc.3-r4
|
GW=ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway:dsh-v0.2.1-rc.5
|
||||||
docker pull "$GW"
|
docker pull "$GW"
|
||||||
kind load docker-image "$GW" --name <kind-cluster>
|
kind load docker-image "$GW" --name <kind-cluster>
|
||||||
|
|
||||||
@@ -541,8 +541,8 @@ gateway remains.
|
|||||||
|
|
||||||
## Optional hardening
|
## Optional hardening
|
||||||
|
|
||||||
The base walkthrough is deliberately portable. Four opt-in layers are documented
|
The base walkthrough is deliberately portable. Four opt-in isolation layers are
|
||||||
in [optional hardening](docs/optional-hardening.md):
|
documented in [optional hardening](docs/optional-hardening.md):
|
||||||
|
|
||||||
- **gVisor** — uncomment `runtimeClassName: gvisor` before applying, or patch an
|
- **gVisor** — uncomment `runtimeClassName: gvisor` before applying, or patch an
|
||||||
existing install. Requires a `gvisor` RuntimeClass and capable nodes.
|
existing install. Requires a `gvisor` RuntimeClass and capable nodes.
|
||||||
@@ -554,6 +554,9 @@ in [optional hardening](docs/optional-hardening.md):
|
|||||||
`kubectl apply -f manifests/optional/network-policy/` allows only
|
`kubectl apply -f manifests/optional/network-policy/` allows only
|
||||||
gateway→sandbox on `3080` and sandbox→DNS/public `443`.
|
gateway→sandbox on `3080` and sandbox→DNS/public `443`.
|
||||||
- **Provider keys** — create `dsh-provider-keys-extra` as shown in step 1.
|
- **Provider keys** — create `dsh-provider-keys-extra` as shown in step 1.
|
||||||
|
- **Failed-Pod recovery** — uncomment `failedPodPolicy: Recreate` so the
|
||||||
|
controller replaces a `Failed` sandbox Pod (node eviction) instead of parking
|
||||||
|
the sandbox at `Finished=True`. Requires agent-sandbox v1.0.8+.
|
||||||
|
|
||||||
Apply each independently and re-run the walkthrough. `kubectl apply -f manifests/`
|
Apply each independently and re-run the walkthrough. `kubectl apply -f manifests/`
|
||||||
is non-recursive, so the optional manifest is not applied by the base step.
|
is non-recursive, so the optional manifest is not applied by the base step.
|
||||||
@@ -586,6 +589,7 @@ is non-recursive, so the optional manifest is not applied by the base step.
|
|||||||
| Sandbox cannot reach a model API after applying the optional NetworkPolicy | The policy allows DNS and public TCP `443` only; a proxy, custom base URL, port `80` or non-443 endpoint needs an explicit egress rule |
|
| Sandbox cannot reach a model API after applying the optional NetworkPolicy | The policy allows DNS and public TCP `443` only; a proxy, custom base URL, port `80` or non-443 endpoint needs an explicit egress rule |
|
||||||
| Gateway cannot reach a sandbox after applying an exposure NetworkPolicy | Check that the CNI enforces NetworkPolicy and that the Traefik/oauth2-proxy pod labels match; kubelet probes may also need the node CIDR |
|
| Gateway cannot reach a sandbox after applying an exposure NetworkPolicy | Check that the CNI enforces NetworkPolicy and that the Traefik/oauth2-proxy pod labels match; kubelet probes may also need the node CIDR |
|
||||||
| New provider key not visible in a sandbox | Pod env is stamped at creation: delete the warm-pool spares and recreate claimed pods. A key saved in DSH's credential store also wins over env |
|
| New provider key not visible in a sandbox | Pod env is stamped at creation: delete the warm-pool spares and recreate claimed pods. A key saved in DSH's credential store also wins over env |
|
||||||
|
| Sandbox reports `Finished=True` and every gateway request fails | The sandbox Pod reached phase `Failed` (node eviction, spot reclaim) and the default `Retain` keeps it. Delete the Pod so the controller recreates it, or set `failedPodPolicy: Recreate` (agent-sandbox v1.0.8+; see optional hardening) |
|
||||||
|
|
||||||
## Limitations
|
## Limitations
|
||||||
|
|
||||||
@@ -649,4 +653,5 @@ delete the Secrets with the namespace and revoke the OAuth App in GitHub.
|
|||||||
security and RBAC, failure modes, evolution path.
|
security and RBAC, failure modes, evolution path.
|
||||||
- [Exposure README](manifests/exposure/README.md) — the two variants in detail.
|
- [Exposure README](manifests/exposure/README.md) — the two variants in detail.
|
||||||
- [Optional hardening](docs/optional-hardening.md) — gVisor, nodeSelector,
|
- [Optional hardening](docs/optional-hardening.md) — gVisor, nodeSelector,
|
||||||
NetworkPolicy and extra provider keys, with verification and rollback.
|
NetworkPolicy, extra provider keys and failed-Pod recovery, with verification
|
||||||
|
and rollback.
|
||||||
|
|||||||
@@ -6,7 +6,7 @@ Kubernetes, and DSH Web fit together, and why the design looks the way it does.
|
|||||||
|
|
||||||
Scope: one Kubernetes namespace (`dsh-demo`), one shared DSH Web image, one gateway deployment,
|
Scope: one Kubernetes namespace (`dsh-demo`), one shared DSH Web image, one gateway deployment,
|
||||||
and one DSH Web pod + PVC per user. The cluster is assumed to run
|
and one DSH Web pod + PVC per user. The cluster is assumed to run
|
||||||
[agent-sandbox](https://github.com/kubernetes-sigs/agent-sandbox) v1.0.7 with the extensions
|
[agent-sandbox](https://github.com/kubernetes-sigs/agent-sandbox) v1.0.8 with the extensions
|
||||||
(`SandboxTemplate`, `SandboxWarmPool`, `SandboxClaim`).
|
(`SandboxTemplate`, `SandboxWarmPool`, `SandboxClaim`).
|
||||||
|
|
||||||
## Container view
|
## Container view
|
||||||
@@ -301,10 +301,11 @@ Notes and limits:
|
|||||||
`dsh-provider-keys-extra`). `manifests/optional/network-policy/` adds an opt-in sandbox
|
`dsh-provider-keys-extra`). `manifests/optional/network-policy/` adds an opt-in sandbox
|
||||||
NetworkPolicy, and gVisor/`nodeSelector` are opt-in PodSpec fields. Add per-tenant
|
NetworkPolicy, and gVisor/`nodeSelector` are opt-in PodSpec fields. Add per-tenant
|
||||||
credentials before calling it multi-tenant-grade.
|
credentials before calling it multi-tenant-grade.
|
||||||
- **Optional hardening is opt-in.** gVisor, node selection, the sandbox/exposure NetworkPolicies
|
- **Optional hardening is opt-in.** gVisor, node selection, the sandbox/exposure NetworkPolicies,
|
||||||
and extra provider keys are documented in [optional hardening](optional-hardening.md). The
|
extra provider keys and failed-Pod recovery are documented in
|
||||||
base manifests stay portable (no gVisor RuntimeClass, no active node labels, no
|
[optional hardening](optional-hardening.md). The base manifests stay portable (no gVisor
|
||||||
NetworkPolicy and no extra Secret required).
|
RuntimeClass, no active node labels, no NetworkPolicy, no extra Secret required, and the
|
||||||
|
controller's default `failedPodPolicy: Retain`).
|
||||||
- **No gateway rate limiting.** Signup can create claims until the cluster or warm pool is
|
- **No gateway rate limiting.** Signup can create claims until the cluster or warm pool is
|
||||||
exhausted; put limits at the ingress for real deployments.
|
exhausted; put limits at the ingress for real deployments.
|
||||||
|
|
||||||
@@ -323,6 +324,7 @@ Notes and limits:
|
|||||||
| gVisor + `kubectl port-forward` | Forward refuses or returns nothing | Port-forward is incompatible with gVisor; use the gateway or a reachable Service |
|
| gVisor + `kubectl port-forward` | Forward refuses or returns nothing | Port-forward is incompatible with gVisor; use the gateway or a reachable Service |
|
||||||
| Provider key added but missing in the pod | Requests still fail with `MISSING_CREDENTIAL` | Pod env is stamped at creation; recreate spares/pods. A stored credential wins over env |
|
| Provider key added but missing in the pod | Requests still fail with `MISSING_CREDENTIAL` | Pod env is stamped at creation; recreate spares/pods. A stored credential wins over env |
|
||||||
| `nodeSelector` matches no node | Pod `Pending`, `0/N nodes are available` | Fix the label, pool capacity or taint/toleration; RuntimeClass scheduling may also constrain it |
|
| `nodeSelector` matches no node | Pod `Pending`, `0/N nodes are available` | Fix the label, pool capacity or taint/toleration; RuntimeClass scheduling may also constrain it |
|
||||||
|
| Sandbox Pod reaches phase `Failed` (node eviction, spot reclaim) | Sandbox reports `Finished=True`; the gateway keeps failing against a dead Pod | Set `failedPodPolicy: Recreate` (agent-sandbox v1.0.8+) so the controller replaces the Pod with the same identity and PVCs; otherwise delete the Pod by hand. See the hardening doc |
|
||||||
|
|
||||||
## Upstream feature request: remove log scraping
|
## Upstream feature request: remove log scraping
|
||||||
|
|
||||||
|
|||||||
@@ -1,18 +1,19 @@
|
|||||||
# Optional hardening: gVisor, nodeSelector, NetworkPolicy and provider keys
|
# Optional hardening: gVisor, nodeSelector, NetworkPolicy, provider keys and failed-Pod recovery
|
||||||
|
|
||||||
The base example is deliberately portable: it runs on kind and GKE without a
|
The base example is deliberately portable: it runs on kind and GKE without a
|
||||||
gVisor runtime, a tainted node pool or a NetworkPolicy-capable CNI, and it
|
gVisor runtime, a tainted node pool or a NetworkPolicy-capable CNI, and it
|
||||||
boots with only the required `dsh-provider-keys` Secret. This document covers
|
boots with only the required `dsh-provider-keys` Secret. This document covers
|
||||||
four **opt-in** layers:
|
four **opt-in** isolation layers, plus one availability setting:
|
||||||
|
|
||||||
| Layer | Where it lives | What it changes |
|
| Setting | Where it lives | What it changes |
|
||||||
| --- | --- | --- |
|
| --- | --- | --- |
|
||||||
| gVisor | commented `runtimeClassName` in the SandboxTemplates and Deployments | puts DSH and/or the gateway in a userspace kernel |
|
| gVisor | commented `runtimeClassName` in the SandboxTemplates and Deployments | puts DSH and/or the gateway in a userspace kernel |
|
||||||
| nodeSelector | commented `nodeSelector` in the same files | pins pods to a chosen node pool |
|
| nodeSelector | commented `nodeSelector` in the same files | pins pods to a chosen node pool |
|
||||||
| NetworkPolicy | `manifests/optional/network-policy/`, `manifests/exposure/*/` | limits sandbox, gateway and proxy connectivity |
|
| NetworkPolicy | `manifests/optional/network-policy/`, `manifests/exposure/*/` | limits sandbox, gateway and proxy connectivity |
|
||||||
| Provider keys | optional `dsh-provider-keys-extra` Secret | injects extra provider API keys into every sandbox |
|
| Provider keys | optional `dsh-provider-keys-extra` Secret | injects extra provider API keys into every sandbox |
|
||||||
|
| Failed-Pod recovery | commented `failedPodPolicy` in the SandboxTemplates | replaces a Failed sandbox Pod instead of parking the sandbox |
|
||||||
|
|
||||||
Each layer is independent. Enable only what the cluster supports and test the
|
Each setting is independent. Enable only what the cluster supports and test the
|
||||||
walkthrough after each change.
|
walkthrough after each change.
|
||||||
|
|
||||||
## 1. gVisor
|
## 1. gVisor
|
||||||
@@ -412,7 +413,7 @@ Secrets; when the same key exists twice, the later `envFrom` entry wins.
|
|||||||
|
|
||||||
### Supported names (curated)
|
### Supported names (curated)
|
||||||
|
|
||||||
These are the names DSH v0.2.1-rc.3 resolves through its packaged provider
|
These are the names the pinned DSH release resolves through its packaged provider
|
||||||
catalog; the catalog has more entries, these are the common ones:
|
catalog; the catalog has more entries, these are the common ones:
|
||||||
|
|
||||||
| Provider | Environment variable |
|
| Provider | Environment variable |
|
||||||
@@ -473,7 +474,130 @@ kubectl -n dsh-demo exec "$POD" -- sh -c '
|
|||||||
- The optional Secret is created imperatively; the repository never contains a
|
- The optional Secret is created imperatively; the repository never contains a
|
||||||
real key. Keep it out of Git and out of shell history where possible.
|
real key. Keep it out of Git and out of shell history where possible.
|
||||||
|
|
||||||
## 5. Verification summary
|
## 5. Failed-Pod recovery
|
||||||
|
|
||||||
|
### What it changes
|
||||||
|
|
||||||
|
When a sandbox Pod reaches phase `Failed`, the controller's default policy,
|
||||||
|
`Retain`, leaves the Pod in place and surfaces `Finished=True` on the Sandbox.
|
||||||
|
The Sandbox, its claim and its PVC survive, but nothing runs: the gateway keeps
|
||||||
|
proxying to a dead Pod and every request fails until someone deletes the Pod or
|
||||||
|
the claim.
|
||||||
|
|
||||||
|
agent-sandbox v1.0.8 (the pin in [versions.env](../../../versions.env)) adds
|
||||||
|
`spec.failedPodPolicy` to `Sandbox` and to `SandboxTemplate`:
|
||||||
|
|
||||||
|
| Value | Behavior |
|
||||||
|
| --- | --- |
|
||||||
|
| `Retain` (default) | The Failed Pod stays and the Sandbox reports `Finished=True`; the evidence stays available for debugging. |
|
||||||
|
| `Recreate` | The controller deletes the Failed Pod and starts a replacement; the Sandbox identity and its Sandbox-owned PVCs are retained. |
|
||||||
|
|
||||||
|
Repeated failures back off like a Deployment - 5 s, doubling to 5 min between
|
||||||
|
creations - so a crash loop cannot hammer the API server. The backoff resets
|
||||||
|
once the replacement Pod has been continuously `Running` for 10 minutes, and it
|
||||||
|
lives in memory only, so a controller restart clears it.
|
||||||
|
|
||||||
|
The base templates ship it commented out, for two reasons:
|
||||||
|
|
||||||
|
- A deterministic failure (bad image, bad Secret, a wrong probe) would recreate
|
||||||
|
forever while hiding the cause.
|
||||||
|
- The templates use `restartPolicy: OnFailure`, so the kubelet restarts a
|
||||||
|
crashing container in place and the Pod does not reach phase `Failed` anyway.
|
||||||
|
This policy matters for terminal failures - most commonly node eviction
|
||||||
|
(`Evicted`), when a spot/preemptible node is reclaimed or the node hits memory
|
||||||
|
pressure.
|
||||||
|
|
||||||
|
No gateway change is needed: the handshake cache is keyed by Pod UID, so a
|
||||||
|
replacement Pod re-handshakes on the next request, and the existing `401` retry
|
||||||
|
covers containers restarted in place.
|
||||||
|
|
||||||
|
### Enable it
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. The blueprint for future sandboxes.
|
||||||
|
kubectl -n dsh-demo patch sandboxtemplate dsh-web --type merge \
|
||||||
|
-p '{"spec":{"failedPodPolicy":"Recreate"}}'
|
||||||
|
|
||||||
|
# 2. Warm-pool spares were stamped at creation: delete them and let the pool
|
||||||
|
# replenish from the updated template.
|
||||||
|
kubectl -n dsh-demo delete sandboxes -l agents.x-k8s.io/warm-pool-sandbox
|
||||||
|
|
||||||
|
# 3. Claimed sandboxes keep their own spec: patch them in place.
|
||||||
|
for sb in $(kubectl -n dsh-demo get sandboxes \
|
||||||
|
-l '!agents.x-k8s.io/warm-pool-sandbox' -o jsonpath='{.items[*].metadata.name}'); do
|
||||||
|
kubectl -n dsh-demo patch sandbox "$sb" --type merge \
|
||||||
|
-p '{"spec":{"failedPodPolicy":"Recreate"}}'
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
For a fresh install, uncomment `failedPodPolicy: Recreate` in the template
|
||||||
|
instead.
|
||||||
|
|
||||||
|
### Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
SB=$(kubectl -n dsh-demo get sandboxclaim dsh-alice -o jsonpath='{.status.sandbox.name}')
|
||||||
|
kubectl -n dsh-demo get sandbox "$SB" -o jsonpath='{.spec.failedPodPolicy}{"\n"}'
|
||||||
|
# Recreate
|
||||||
|
```
|
||||||
|
|
||||||
|
A failure drill that never touches a user sandbox - a `Sandbox` with
|
||||||
|
`restartPolicy: Never` and a container that exits non-zero (swap the image if
|
||||||
|
Docker Hub is unreachable):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl -n dsh-demo apply -f - <<'EOF'
|
||||||
|
apiVersion: agents.x-k8s.io/v1beta1
|
||||||
|
kind: Sandbox
|
||||||
|
metadata:
|
||||||
|
name: failed-pod-drill
|
||||||
|
spec:
|
||||||
|
failedPodPolicy: Recreate
|
||||||
|
podTemplate:
|
||||||
|
spec:
|
||||||
|
restartPolicy: Never
|
||||||
|
containers:
|
||||||
|
- name: drill
|
||||||
|
image: busybox:1.36
|
||||||
|
command: ["sh", "-c", "exit 1"]
|
||||||
|
EOF
|
||||||
|
|
||||||
|
kubectl -n dsh-demo get pods -w
|
||||||
|
# The drill Pod reaches phase Failed; a replacement appears within ~5 s, then
|
||||||
|
# 10 s, 20 s, ... as the container keeps exiting non-zero.
|
||||||
|
kubectl -n dsh-demo delete sandbox failed-pod-drill
|
||||||
|
```
|
||||||
|
|
||||||
|
Repeat the drill with `failedPodPolicy: Retain` (or omit the field) to see the
|
||||||
|
default: the Pod stays `Failed` and the Sandbox condition `Finished` becomes
|
||||||
|
`True`.
|
||||||
|
|
||||||
|
### Caveats
|
||||||
|
|
||||||
|
- `Recreate` deletes the Failed Pod, so its logs go with it. Capture
|
||||||
|
`kubectl get events` or switch back to `Retain` when debugging.
|
||||||
|
- The replacement is a cold start: DSH boots again and issues a new launch
|
||||||
|
token. The workspace comes back from the PVC; in-flight sessions do not.
|
||||||
|
- The field needs agent-sandbox v1.0.8+. A v1.0.7 cluster rejects it as an
|
||||||
|
unknown field.
|
||||||
|
- It applies to sandbox Pods only. The gateway and oauth2-proxy are plain
|
||||||
|
Deployments, and their ReplicaSets already replace Failed Pods.
|
||||||
|
|
||||||
|
### Roll back
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl -n dsh-demo patch sandboxtemplate dsh-web --type merge \
|
||||||
|
-p '{"spec":{"failedPodPolicy":"Retain"}}'
|
||||||
|
for sb in $(kubectl -n dsh-demo get sandboxes -o jsonpath='{.items[*].metadata.name}'); do
|
||||||
|
kubectl -n dsh-demo patch sandbox "$sb" --type merge \
|
||||||
|
-p '{"spec":{"failedPodPolicy":"Retain"}}'
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
Warm-pool spares pick the default up as they are claimed and replenished, or
|
||||||
|
delete them as in step 2 above.
|
||||||
|
|
||||||
|
## 6. Verification summary
|
||||||
|
|
||||||
Required locally for any change to this example:
|
Required locally for any change to this example:
|
||||||
|
|
||||||
@@ -497,8 +621,10 @@ Manual checks, on a cluster with the relevant CNI/runtime:
|
|||||||
- Exposure policies: browser path works; a non-proxy pod cannot reach the
|
- Exposure policies: browser path works; a non-proxy pod cannot reach the
|
||||||
gateway; probes stay healthy.
|
gateway; probes stay healthy.
|
||||||
- Provider keys: set/unset output (never the values) and the refresh procedure.
|
- Provider keys: set/unset output (never the values) and the refresh procedure.
|
||||||
|
- Failed-Pod recovery: `spec.failedPodPolicy` on the Sandbox, and the drill
|
||||||
|
above replaces a `Failed` Pod while keeping the claim and PVC.
|
||||||
|
|
||||||
## 6. Rollback summary
|
## 7. Rollback summary
|
||||||
|
|
||||||
| Layer | Rollback |
|
| Layer | Rollback |
|
||||||
| --- | --- |
|
| --- | --- |
|
||||||
@@ -507,6 +633,7 @@ Manual checks, on a cluster with the relevant CNI/runtime:
|
|||||||
| gVisor | patch `runtimeClassName: null`, recreate pods |
|
| gVisor | patch `runtimeClassName: null`, recreate pods |
|
||||||
| nodeSelector | patch `nodeSelector: null`, recreate pods |
|
| nodeSelector | patch `nodeSelector: null`, recreate pods |
|
||||||
| Provider keys | delete the extra Secret or a key, recreate pods |
|
| Provider keys | delete the extra Secret or a key, recreate pods |
|
||||||
|
| Failed-Pod recovery | patch `failedPodPolicy: Retain`, delete warm-pool spares |
|
||||||
|
|
||||||
No PVC or workspace data is touched by any rollback. Deleting a `SandboxClaim`
|
No PVC or workspace data is touched by any rollback. Deleting a `SandboxClaim`
|
||||||
is different: it cascades to the Sandbox **and** its PVC, so do not use claim
|
is different: it cascades to the Sandbox **and** its PVC, so do not use claim
|
||||||
|
|||||||
@@ -22,7 +22,7 @@ It does four things:
|
|||||||
The DSH version pin lives in [`versions.env`](../../../versions.env)
|
The DSH version pin lives in [`versions.env`](../../../versions.env)
|
||||||
(`DSH_VERSION` plus the optional repository `RELEASE_REVISION`); the example
|
(`DSH_VERSION` plus the optional repository `RELEASE_REVISION`); the example
|
||||||
manifests and published images use the release identity built from it (currently
|
manifests and published images use the release identity built from it (currently
|
||||||
`dsh-v0.2.1-rc.3-r4`). DSH binds `:3080`, serves `/`, gates `/api` behind a
|
`dsh-v0.2.1-rc.5`). DSH binds `:3080`, serves `/`, gates `/api` behind a
|
||||||
`dsh-auth-*` cookie and requires a WebSocket at `/api/remote.mux`.
|
`dsh-auth-*` cookie and requires a WebSocket at `/api/remote.mux`.
|
||||||
|
|
||||||
## Flow
|
## Flow
|
||||||
|
|||||||
@@ -29,6 +29,15 @@ spec:
|
|||||||
# be blocked. See docs/optional-hardening.md.
|
# be blocked. See docs/optional-hardening.md.
|
||||||
networkPolicyManagement: Unmanaged
|
networkPolicyManagement: Unmanaged
|
||||||
|
|
||||||
|
# OPTIONAL: replace a Failed sandbox Pod instead of parking the sandbox.
|
||||||
|
# The default, Retain, leaves the Failed Pod in place and surfaces
|
||||||
|
# Finished=True on the Sandbox until a human intervenes. Recreate deletes the
|
||||||
|
# Failed Pod and starts a replacement with the same Sandbox identity and
|
||||||
|
# Sandbox-owned PVCs, backing off (5s doubling to 5m) on repeated failures.
|
||||||
|
# Requires agent-sandbox v1.0.8+; existing Sandboxes keep their own stamped
|
||||||
|
# policy, so patch them separately. See docs/optional-hardening.md.
|
||||||
|
# failedPodPolicy: Recreate
|
||||||
|
|
||||||
podTemplate:
|
podTemplate:
|
||||||
spec:
|
spec:
|
||||||
# DSH Web is multi-arch (amd64 + arm64); no arch nodeSelector is needed.
|
# DSH Web is multi-arch (amd64 + arm64); no arch nodeSelector is needed.
|
||||||
@@ -64,7 +73,7 @@ spec:
|
|||||||
fsGroupChangePolicy: OnRootMismatch
|
fsGroupChangePolicy: OnRootMismatch
|
||||||
containers:
|
containers:
|
||||||
- name: dsh-web
|
- name: dsh-web
|
||||||
image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.3-r4
|
image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.5
|
||||||
imagePullPolicy: IfNotPresent
|
imagePullPolicy: IfNotPresent
|
||||||
ports:
|
ports:
|
||||||
- name: web
|
- name: web
|
||||||
|
|||||||
@@ -101,7 +101,7 @@ spec:
|
|||||||
type: RuntimeDefault
|
type: RuntimeDefault
|
||||||
containers:
|
containers:
|
||||||
- name: gateway
|
- name: gateway
|
||||||
image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway:dsh-v0.2.1-rc.3-r4
|
image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway:dsh-v0.2.1-rc.5
|
||||||
imagePullPolicy: IfNotPresent
|
imagePullPolicy: IfNotPresent
|
||||||
securityContext:
|
securityContext:
|
||||||
allowPrivilegeEscalation: false
|
allowPrivilegeEscalation: false
|
||||||
|
|||||||
@@ -30,6 +30,9 @@ These are configured in the main manifests or with an extra Secret:
|
|||||||
- **gVisor:** uncomment `runtimeClassName: gvisor` in the SandboxTemplate and
|
- **gVisor:** uncomment `runtimeClassName: gvisor` in the SandboxTemplate and
|
||||||
the Deployments before applying, or patch an existing install.
|
the Deployments before applying, or patch an existing install.
|
||||||
- **nodeSelector:** uncomment the field and replace the placeholder node label.
|
- **nodeSelector:** uncomment the field and replace the placeholder node label.
|
||||||
|
- **Failed-Pod recovery:** uncomment `failedPodPolicy: Recreate` in the
|
||||||
|
SandboxTemplate (agent-sandbox v1.0.8+) so a Failed Pod is replaced instead
|
||||||
|
of parked.
|
||||||
- **Provider keys:** create the optional `dsh-provider-keys-extra` Secret; the
|
- **Provider keys:** create the optional `dsh-provider-keys-extra` Secret; the
|
||||||
base `dsh-provider-keys` Secret remains the one required key.
|
base `dsh-provider-keys` Secret remains the one required key.
|
||||||
|
|
||||||
|
|||||||
@@ -13,7 +13,7 @@
|
|||||||
# - Optional: Secret dsh-provider-keys-extra with additional provider keys
|
# - Optional: Secret dsh-provider-keys-extra with additional provider keys
|
||||||
# (OPENROUTER_API_KEY, GEMINI_API_KEY, NVIDIA_API_KEY, ...). It is
|
# (OPENROUTER_API_KEY, GEMINI_API_KEY, NVIDIA_API_KEY, ...). It is
|
||||||
# referenced with `optional: true`, so it may be absent.
|
# referenced with `optional: true`, so it may be absent.
|
||||||
# The agent-sandbox controller (v1.0.7, API v1beta1) must be installed.
|
# The agent-sandbox controller (v1.0.8, API v1beta1) must be installed.
|
||||||
apiVersion: extensions.agents.x-k8s.io/v1beta1
|
apiVersion: extensions.agents.x-k8s.io/v1beta1
|
||||||
kind: SandboxTemplate
|
kind: SandboxTemplate
|
||||||
metadata:
|
metadata:
|
||||||
@@ -26,6 +26,10 @@ spec:
|
|||||||
# quickstart reaches the sandbox through `kubectl port-forward`, which is
|
# quickstart reaches the sandbox through `kubectl port-forward`, which is
|
||||||
# local to the pod. The multi-tenant example documents opt-in policies in
|
# local to the pod. The multi-tenant example documents opt-in policies in
|
||||||
# examples/dsh-agents-as-a-service/docs/optional-hardening.md.
|
# examples/dsh-agents-as-a-service/docs/optional-hardening.md.
|
||||||
|
#
|
||||||
|
# OPTIONAL: recreate a Failed sandbox Pod instead of retaining it
|
||||||
|
# (agent-sandbox v1.0.8+). Documented with the multi-tenant example.
|
||||||
|
# failedPodPolicy: Recreate
|
||||||
podTemplate:
|
podTemplate:
|
||||||
spec:
|
spec:
|
||||||
# OPTIONAL: run under gVisor. gVisor breaks `kubectl port-forward`, so
|
# OPTIONAL: run under gVisor. gVisor breaks `kubectl port-forward`, so
|
||||||
@@ -52,7 +56,7 @@ spec:
|
|||||||
fsGroupChangePolicy: OnRootMismatch
|
fsGroupChangePolicy: OnRootMismatch
|
||||||
containers:
|
containers:
|
||||||
- name: dsh-web
|
- name: dsh-web
|
||||||
image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.3-r4
|
image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.5
|
||||||
imagePullPolicy: IfNotPresent
|
imagePullPolicy: IfNotPresent
|
||||||
ports:
|
ports:
|
||||||
- name: web
|
- name: web
|
||||||
|
|||||||
@@ -3,7 +3,7 @@
|
|||||||
# "make check".
|
# "make check".
|
||||||
#
|
#
|
||||||
# ./scripts/set-version.sh dsh-v0.2.2-rc.1 # new DSH ref, no revision
|
# ./scripts/set-version.sh dsh-v0.2.2-rc.1 # new DSH ref, no revision
|
||||||
# ./scripts/set-version.sh dsh-v0.2.1-rc.3 r3 # same ref, new repo revision
|
# ./scripts/set-version.sh dsh-v0.2.1-rc.5 r3 # same ref, new repo revision
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
cd "$(dirname "$0")/.."
|
cd "$(dirname "$0")/.."
|
||||||
|
|
||||||
|
|||||||
12
versions.env
12
versions.env
@@ -5,18 +5,18 @@
|
|||||||
|
|
||||||
# Upstream DeepSeek Harness source, and the ref this repository packages.
|
# Upstream DeepSeek Harness source, and the ref this repository packages.
|
||||||
DSH_REPO=https://github.com/deepseek-ai/deepseek-harness.git
|
DSH_REPO=https://github.com/deepseek-ai/deepseek-harness.git
|
||||||
DSH_VERSION=dsh-v0.2.1-rc.3
|
DSH_VERSION=dsh-v0.2.1-rc.5
|
||||||
PNPM_VERSION=11.28.5
|
PNPM_VERSION=11.28.5
|
||||||
|
|
||||||
# Release identity = DSH_VERSION plus an optional repository revision.
|
# Release identity = DSH_VERSION plus an optional repository revision.
|
||||||
# An empty RELEASE_REVISION is the first release at this DSH ref; rN (N >= 2)
|
# An empty RELEASE_REVISION is the first release at this DSH ref; rN (N >= 2)
|
||||||
# marks a repository-only re-release at the same DSH ref. The dsh-v0.2.1-rc.3
|
# marks a repository-only re-release at the same DSH ref. This is the first
|
||||||
# image tag is already published on GHCR and tags are immutable, so the current
|
# release at dsh-v0.2.1-rc.5, so the identity is the plain tag; a later
|
||||||
# scheme started at -r2; the current identity is dsh-v0.2.1-rc.3-r4.
|
# repository-only change at the same ref becomes -r2.
|
||||||
RELEASE_REVISION=r4
|
RELEASE_REVISION=
|
||||||
|
|
||||||
# Control-plane versions used by the examples and their manifests.
|
# Control-plane versions used by the examples and their manifests.
|
||||||
AGENT_SANDBOX_VERSION=v1.0.7
|
AGENT_SANDBOX_VERSION=v1.0.8
|
||||||
GATEWAY_API_VERSION=v1.2.1
|
GATEWAY_API_VERSION=v1.2.1
|
||||||
OAUTH2_PROXY_IMAGE=quay.io/oauth2-proxy/oauth2-proxy:v7.15.5
|
OAUTH2_PROXY_IMAGE=quay.io/oauth2-proxy/oauth2-proxy:v7.15.5
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user