mirror of
https://github.com/sredevopsorg/deepseek-kubernetes-sandbox.git
synced 2026-11-03 01:13:24 +00:00
Compare commits
18 Commits
v0.0.1
...
dsh-v0.2.1
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
e2f1fe567a | ||
|
|
3e9ba38fc1 | ||
|
|
bf24306421 | ||
|
|
8681412be4 | ||
|
|
958cbe25a5 | ||
|
|
027f74f8e9 | ||
|
|
d627526e3e | ||
|
|
ecba8b96a6 | ||
|
|
6a8fe54abc | ||
|
|
6876871110 | ||
|
|
3b70f179ac | ||
|
|
53e55fc851 | ||
|
|
ed1398dfd1 | ||
|
|
73a65d98d2 | ||
|
|
3f46279b39 | ||
|
|
2b47afa22b | ||
|
|
6d97f6193b | ||
|
|
7d3faf07dd |
155
.dockerignore
155
.dockerignore
@@ -1,149 +1,6 @@
|
|||||||
# Logs
|
# The main image build clones DeepSeek Harness from git and copies nothing
|
||||||
logs
|
# from this repository, so the build context is intentionally reduced to the
|
||||||
*.log
|
# Dockerfile alone. This keeps builds fast and prevents the rest of the repo
|
||||||
npm-debug.log*
|
# (examples, docs, git history) from being sent to the builder.
|
||||||
yarn-debug.log*
|
*
|
||||||
yarn-error.log*
|
!Dockerfile
|
||||||
lerna-debug.log*
|
|
||||||
|
|
||||||
# Diagnostic reports (https://nodejs.org/api/report.html)
|
|
||||||
report.[0-9]*.[0-9]*.[0-9]*.[0-9]*.json
|
|
||||||
|
|
||||||
# Runtime data
|
|
||||||
pids
|
|
||||||
*.pid
|
|
||||||
*.seed
|
|
||||||
*.pid.lock
|
|
||||||
|
|
||||||
# Directory for instrumented libs generated by jscoverage/JSCover
|
|
||||||
lib-cov
|
|
||||||
|
|
||||||
# Coverage directory used by tools like istanbul
|
|
||||||
coverage
|
|
||||||
*.lcov
|
|
||||||
|
|
||||||
# nyc test coverage
|
|
||||||
.nyc_output
|
|
||||||
|
|
||||||
# Grunt intermediate storage (https://gruntjs.com/creating-plugins#storing-task-files)
|
|
||||||
.grunt
|
|
||||||
|
|
||||||
# Bower dependency directory (https://bower.io/)
|
|
||||||
bower_components
|
|
||||||
|
|
||||||
# node-waf configuration
|
|
||||||
.lock-wscript
|
|
||||||
|
|
||||||
# Compiled binary addons (https://nodejs.org/api/addons.html)
|
|
||||||
build/Release
|
|
||||||
|
|
||||||
# Dependency directories
|
|
||||||
node_modules/
|
|
||||||
jspm_packages/
|
|
||||||
|
|
||||||
# Snowpack dependency directory (https://snowpack.dev/)
|
|
||||||
web_modules/
|
|
||||||
|
|
||||||
# TypeScript cache
|
|
||||||
*.tsbuildinfo
|
|
||||||
|
|
||||||
# Optional npm cache directory
|
|
||||||
.npm
|
|
||||||
|
|
||||||
# Optional eslint cache
|
|
||||||
.eslintcache
|
|
||||||
|
|
||||||
# Optional stylelint cache
|
|
||||||
.stylelintcache
|
|
||||||
|
|
||||||
# Optional REPL history
|
|
||||||
.node_repl_history
|
|
||||||
|
|
||||||
# Output of 'npm pack'
|
|
||||||
*.tgz
|
|
||||||
|
|
||||||
# Yarn Integrity file
|
|
||||||
.yarn-integrity
|
|
||||||
|
|
||||||
# dotenv environment variable files
|
|
||||||
.env
|
|
||||||
.env.*
|
|
||||||
!.env.example
|
|
||||||
|
|
||||||
# parcel-bundler cache (https://parceljs.org/)
|
|
||||||
.cache
|
|
||||||
.parcel-cache
|
|
||||||
|
|
||||||
# Next.js build output
|
|
||||||
.next
|
|
||||||
out
|
|
||||||
|
|
||||||
# Nuxt.js build / generate output
|
|
||||||
.nuxt
|
|
||||||
dist
|
|
||||||
.output
|
|
||||||
|
|
||||||
# Gatsby files
|
|
||||||
.cache/
|
|
||||||
# Comment in the public line in if your project uses Gatsby and not Next.js
|
|
||||||
# https://nextjs.org/blog/next-9-1#public-directory-support
|
|
||||||
# public
|
|
||||||
|
|
||||||
# vuepress build output
|
|
||||||
.vuepress/dist
|
|
||||||
|
|
||||||
# vuepress v2.x temp directory
|
|
||||||
.temp
|
|
||||||
|
|
||||||
# Sveltekit cache directory
|
|
||||||
.svelte-kit/
|
|
||||||
|
|
||||||
# vitepress build output
|
|
||||||
**/.vitepress/dist
|
|
||||||
|
|
||||||
# vitepress cache directory
|
|
||||||
**/.vitepress/cache
|
|
||||||
|
|
||||||
# Docusaurus cache and generated files
|
|
||||||
.docusaurus
|
|
||||||
|
|
||||||
# Serverless directories
|
|
||||||
.serverless/
|
|
||||||
|
|
||||||
# FuseBox cache
|
|
||||||
.fusebox/
|
|
||||||
|
|
||||||
# DynamoDB Local files
|
|
||||||
.dynamodb/
|
|
||||||
|
|
||||||
# Firebase cache directory
|
|
||||||
.firebase/
|
|
||||||
|
|
||||||
# TernJS port file
|
|
||||||
.tern-port
|
|
||||||
|
|
||||||
# Stores Visual Studio Code versions used for testing Visual Studio Code extensions
|
|
||||||
.vscode-test
|
|
||||||
|
|
||||||
# pnpm
|
|
||||||
.pnpm-store
|
|
||||||
|
|
||||||
# yarn v3
|
|
||||||
.pnp.*
|
|
||||||
.yarn/*
|
|
||||||
!.yarn/patches
|
|
||||||
!.yarn/plugins
|
|
||||||
!.yarn/releases
|
|
||||||
!.yarn/sdks
|
|
||||||
!.yarn/versions
|
|
||||||
|
|
||||||
# Vite files
|
|
||||||
vite.config.js.timestamp-*
|
|
||||||
vite.config.ts.timestamp-*
|
|
||||||
.vite/
|
|
||||||
|
|
||||||
# Astro generated types
|
|
||||||
.astro/
|
|
||||||
|
|
||||||
# Turbo Repo
|
|
||||||
.turbo
|
|
||||||
|
|||||||
155
.dsh/skills/dsh-agent-sandbox-platform/SKILL.md
Normal file
155
.dsh/skills/dsh-agent-sandbox-platform/SKILL.md
Normal file
@@ -0,0 +1,155 @@
|
|||||||
|
---
|
||||||
|
name: dsh-agent-sandbox-platform
|
||||||
|
description: >-
|
||||||
|
Operate and extend this repository's DeepSeek Harness-on-agent-sandbox platform:
|
||||||
|
warm pools, claims, sandbox suspend/resume, the gateway, and the optional
|
||||||
|
agent-sandbox MCP server, with version-aware safety and a read-only preflight.
|
||||||
|
whenToUse: >-
|
||||||
|
Use when working in deepseek-kubernetes-sandbox or with Kubernetes agent-sandbox:
|
||||||
|
inspect or provision sandboxes, debug warm pools or claims, use
|
||||||
|
mcp__*__create_sandbox/execute_command, connect the agent-sandbox MCP server, or
|
||||||
|
add a manifest, gateway route, exposure variant, example, or version bump here.
|
||||||
|
metadata:
|
||||||
|
author: sredevopsorg
|
||||||
|
version: "1.0.0"
|
||||||
|
---
|
||||||
|
|
||||||
|
# DSH agent-sandbox platform
|
||||||
|
|
||||||
|
This repository runs DeepSeek Harness Web (DSH) as one sandbox per user on
|
||||||
|
[agent-sandbox](https://github.com/kubernetes-sigs/agent-sandbox), with an
|
||||||
|
optional official MCP server for sandbox execution. For generic cluster work
|
||||||
|
load `kubernetes-kubectl-operations`; for profile MCP rows load
|
||||||
|
`cordis-plugin-development`.
|
||||||
|
|
||||||
|
## 0. Facts before action
|
||||||
|
|
||||||
|
1. Read `AGENTS.md`, `versions.env`, the example README/architecture. The
|
||||||
|
control-plane pin is `AGENT_SANDBOX_VERSION`; release identity is
|
||||||
|
`DSH_VERSION` + `RELEASE_REVISION`.
|
||||||
|
2. Establish context, user and namespace; never assume `dsh-demo`.
|
||||||
|
3. Run `scripts/platform-preflight.sh --namespace <ns>` (read-only).
|
||||||
|
4. If `mcp__*__create_sandbox`-style tools are absent, use `kubectl`; do not
|
||||||
|
install or enable a server on your own.
|
||||||
|
|
||||||
|
## 1. Non-negotiables
|
||||||
|
|
||||||
|
- **Read-only by default.** Change a cluster only when the user asked for that
|
||||||
|
exact change; show the command and rollback first.
|
||||||
|
- **Scope every command** with `--context` and `-n`. `--all`/`-A` mutations and
|
||||||
|
deleting namespaces, PVCs, PVs, CRDs, or cluster-scoped objects need an
|
||||||
|
explicit, named confirmation.
|
||||||
|
- **Secrets and logs stay private.** Never print Secret data, kubeconfigs, or
|
||||||
|
node logs. Pod logs can contain the DSH launch token.
|
||||||
|
- **The MCP server has no authentication.** It is a full-access management
|
||||||
|
endpoint; never expose it. A DSH profile row affects every session.
|
||||||
|
- **Clean up what you create.** Delete MCP-created sandboxes in the same
|
||||||
|
session; never leave orphaned claims running.
|
||||||
|
- **Bound output.** Use `--tail`, `--since`, selectors, JSONPath, and
|
||||||
|
head/tail excerpts instead of whole manifests or huge logs.
|
||||||
|
|
||||||
|
## 2. Choose the path
|
||||||
|
|
||||||
|
| Need | Path |
|
||||||
|
| --- | --- |
|
||||||
|
| Inspect or change claims, pools, sandboxes, PVCs, gateway | `kubectl` + [platform runbook](references/platform-runbook.md) |
|
||||||
|
| Run code or move files in a sandbox | agent-sandbox MCP, only for a runtime-capable template ([MCP reference](references/mcp-server.md)) |
|
||||||
|
| Add or change a repository feature | [repo extension guide](references/extending-the-repo.md) |
|
||||||
|
|
||||||
|
The MCP server cannot manage this example's DSH claims: its tools are scoped to
|
||||||
|
claims created by the same MCP session. Use `kubectl` for the platform.
|
||||||
|
|
||||||
|
## 3. Operate the platform
|
||||||
|
|
||||||
|
Resolve user -> claim -> sandbox -> pod:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
SB=$(kubectl -n "$NS" get sandboxclaim "$CLAIM" -o jsonpath='{.status.sandbox.name}')
|
||||||
|
kubectl -n "$NS" get sandbox "$SB" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}{" "}{.status.conditions[?(@.type=="Ready")].reason}{"\n"}{.status.serviceFQDN}{"\n"}'
|
||||||
|
POD=$(kubectl -n "$NS" get sandbox "$SB" -o jsonpath='{.metadata.annotations.agents\.x-k8s\.io/pod-name}'); POD=${POD:-$SB}
|
||||||
|
```
|
||||||
|
|
||||||
|
- Read runtime state from the Sandbox, not the claim: `spec.operatingMode`,
|
||||||
|
`Suspended`, `Ready`. The claim tracks binding, not liveness.
|
||||||
|
- Suspend = patch `operatingMode: Suspended` (pod deleted, PVC + Service kept);
|
||||||
|
resume = `Running` + wait for `Ready`. The gateway wakes on connect.
|
||||||
|
- Warm-pool spares carry `agents.x-k8s.io/warm-pool-sandbox`; adopted sandboxes
|
||||||
|
lose it. A pool with no spares makes signup cold and slow.
|
||||||
|
- The DSH launch token appears once in pod logs; hand the user the `?token=` URL,
|
||||||
|
never the token in reports. Probe `/favicon.svg`.
|
||||||
|
- The idle sweeper only suspends claims the gateway created.
|
||||||
|
|
||||||
|
Full commands, security boundaries and failure matrix:
|
||||||
|
[platform runbook](references/platform-runbook.md).
|
||||||
|
|
||||||
|
## 4. Use the agent-sandbox MCP server within its limits
|
||||||
|
|
||||||
|
Verified from the upstream server:
|
||||||
|
|
||||||
|
- Tools: `create_sandbox`, `delete_sandbox`, `execute_command`, `upload_file`,
|
||||||
|
`download_file`, `list_files`, `file_exists`, `get_sandbox_status`; resource
|
||||||
|
`sandboxes://{namespace}`.
|
||||||
|
- **Session-scoped:** every tool and the resource filter claims by
|
||||||
|
`K8S_SANDBOX_SESSION_ID_LABEL_KEY=<MCP session id>`. It cannot read, exec in,
|
||||||
|
or delete DSH platform claims or claims from a prior/reconnected session.
|
||||||
|
- **Streamable HTTP only:** stdio has no MCP session id and fails. A DSH row must
|
||||||
|
use `transport: streamable-http`; health endpoints are `/healthz` and `/readyz`
|
||||||
|
at the root, the MCP endpoint is `/mcp`.
|
||||||
|
- **Runtime required:** connection types target a sandbox runtime on 8888, or
|
||||||
|
sandboxd on 8080/9090. The DSH template listens only on 3080, so
|
||||||
|
`execute_command` cannot work against it. A sandboxd sidecar executes in its
|
||||||
|
own container, not DSH's.
|
||||||
|
- **Timeouts:** `create_sandbox` waits up to 180 s, but the DSH MCP client's
|
||||||
|
default call timeout is 60 s; raise `toolCallTimeoutMs`.
|
||||||
|
`shutdown_after_seconds` defaults to 300 s (`null` disables it).
|
||||||
|
- **Output:** servers before v1.0.8 return unbounded `execute_command` output.
|
||||||
|
v1.0.8 adds `max_output_chars` (128-300,000, default 30,000) and truncation
|
||||||
|
metadata. Never pass the new argument to an older server; redirect verbose
|
||||||
|
output and read a file instead.
|
||||||
|
|
||||||
|
Deployment, connection matrix and DSH row:
|
||||||
|
[MCP deployment](references/mcp-deployment.md). Limits and cleanup:
|
||||||
|
[MCP reference](references/mcp-server.md).
|
||||||
|
|
||||||
|
## 5. Extend the project
|
||||||
|
|
||||||
|
Read `docs/examples.md` and the testing table in `AGENTS.md` before editing.
|
||||||
|
|
||||||
|
- **Manifest step:** numbered `manifests/NN-*.yaml`, update the README step, run
|
||||||
|
`make check`; CI dry-runs it.
|
||||||
|
- **Gateway:** edit `gateway/gateway.py` and `gateway/test_gateway.py`, update
|
||||||
|
the gateway README API table and architecture doc, run `make test` and
|
||||||
|
`make lint`. Keep the HTTP/WebSocket contract stable.
|
||||||
|
- **Exposure variant:** one variant under `manifests/exposure/<variant>/`,
|
||||||
|
update the exposure README and trusted-host/cookie settings. Variants are
|
||||||
|
alternatives, never a stack.
|
||||||
|
- **New example:** follow `docs/examples.md`, add the index entry, keep scripts
|
||||||
|
executable/idempotent/self-cleaning, run `make check`.
|
||||||
|
- **Version bump:** edit `versions.env` only, update prerequisites, run
|
||||||
|
`make check` and the e2e. Never mix a bump with an unrelated feature.
|
||||||
|
- **MCP integration:** pin the build to the cluster's tag, least-privilege SA,
|
||||||
|
ClusterIP + auth proxy, raised DSH timeout, no secrets in the repo.
|
||||||
|
- **Optional hardening:** gVisor, `nodeSelector`, NetworkPolicy and extra
|
||||||
|
provider keys stay opt-in; the sandbox policy lives under
|
||||||
|
`manifests/optional/`. All four are documented in
|
||||||
|
`examples/dsh-agents-as-a-service/docs/optional-hardening.md`.
|
||||||
|
- **Docs:** update the affected doc in the same change.
|
||||||
|
|
||||||
|
Details and review checklist:
|
||||||
|
[repo extension guide](references/extending-the-repo.md).
|
||||||
|
|
||||||
|
## 6. Report and clean up
|
||||||
|
|
||||||
|
State context/namespace, bounded evidence, conclusion, exact change, rollback,
|
||||||
|
and resources deleted. Disable any MCP row enabled only for the task. Do not
|
||||||
|
leave sandboxes, port-forwards, or temporary files behind.
|
||||||
|
|
||||||
|
## References
|
||||||
|
|
||||||
|
- [Platform runbook](references/platform-runbook.md)
|
||||||
|
- [MCP deployment](references/mcp-deployment.md)
|
||||||
|
- [MCP server reference](references/mcp-server.md)
|
||||||
|
- [Repo extension guide](references/extending-the-repo.md)
|
||||||
|
- [Troubleshooting](references/troubleshooting.md)
|
||||||
|
- Upstream MCP server README:
|
||||||
|
https://github.com/kubernetes-sigs/agent-sandbox/blob/main/clients/integrations/mcp-server/README.md
|
||||||
@@ -0,0 +1,132 @@
|
|||||||
|
# Extending the repository
|
||||||
|
|
||||||
|
Read `AGENTS.md`, `docs/examples.md`, and the example README before editing.
|
||||||
|
This repository is image-first: the Dockerfile is the primary artifact, every
|
||||||
|
implementation is a self-contained example under `examples/<name>/`, and
|
||||||
|
`versions.env` is the only source of pins. Documentation is part of the change.
|
||||||
|
|
||||||
|
## Invariants
|
||||||
|
|
||||||
|
- Never hard-code a version, image tag, or registry path; edit `versions.env`.
|
||||||
|
For the DSH release identity use `make set-version`, never ad-hoc `sed`.
|
||||||
|
- Never commit a secret. Secret-shaped manifests are `*.yaml.example` with
|
||||||
|
placeholders; keys are documented only in `.env.sample`.
|
||||||
|
- Manifests are numbered `NN-*.yaml` and apply in order with plain `kubectl`.
|
||||||
|
Exposure variants live under `manifests/exposure/<variant>/`, are documented
|
||||||
|
by one README, and are mutually exclusive.
|
||||||
|
- Keep the gateway HTTP/WebSocket contract stable unless the change is explicitly
|
||||||
|
about it.
|
||||||
|
- Do not edit `CHANGELOG.md` (generated by git-cliff) or rewrite git history.
|
||||||
|
- Keep diffs scoped; do not mix a version bump with an unrelated feature.
|
||||||
|
|
||||||
|
## Change matrix
|
||||||
|
|
||||||
|
| Change | Edit | Minimum verification |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| Documentation only | `*.md`, `docs/` | `make check`; relative links resolve |
|
||||||
|
| Example manifest | `examples/<name>/manifests/NN-*.yaml`, README step | `make check`; server-side dry-run in CI |
|
||||||
|
| Optional hardening | `manifests/optional/`, exposure NPs, example `docs/optional-hardening.md` | `make check`; manual verification on a capable cluster |
|
||||||
|
| Gateway behavior/route | `gateway/gateway.py`, `gateway/test_gateway.py`, gateway README, architecture | `make test`, `make lint`; update tests and API docs |
|
||||||
|
| Exposure variant | `manifests/exposure/<variant>/`, exposure README, example README | `make check`; apply exactly one variant |
|
||||||
|
| New example | full `examples/<name>/` layout, `examples/README.md` | `make check`; CI dry-run; optional e2e |
|
||||||
|
| Version bump | `versions.env`, affected docs/manifests | `make check`; CI installs the pin; e2e on a cluster |
|
||||||
|
| MCP integration | deployment/RBAC, DSH row, docs | read-only `/readyz`; no secret in repo |
|
||||||
|
| Workflow | `.github/workflows/` | actionlint; SHA-pinned actions; no hard-coded versions |
|
||||||
|
|
||||||
|
## Recipes
|
||||||
|
|
||||||
|
### Add or change a resource step
|
||||||
|
|
||||||
|
1. Keep numbering stable; add `NN-*.yaml` at the end unless order matters.
|
||||||
|
2. Update the matching numbered step in the example README.
|
||||||
|
3. Keep cluster-specific state (node affinity, tuned replica counts) out of the
|
||||||
|
shipped manifest; document it as a local patch.
|
||||||
|
4. Run `make check`. CI additionally parses every manifest and runs a
|
||||||
|
server-side dry-run against the agent-sandbox CRDs.
|
||||||
|
|
||||||
|
### Change the gateway
|
||||||
|
|
||||||
|
The gateway is a single-file Starlette app. A change usually touches:
|
||||||
|
|
||||||
|
- `gateway/gateway.py` (route, auth, proxy, sweeper);
|
||||||
|
- `gateway/test_gateway.py` (unit tests; no cluster required);
|
||||||
|
- `gateway/README.md` (API table, configuration, auth semantics);
|
||||||
|
- `examples/dsh-agents-as-a-service/docs/architecture.md` when flows change;
|
||||||
|
- `examples/dsh-agents-as-a-service/README.md` when the walkthrough changes.
|
||||||
|
|
||||||
|
Run `make test` and `make lint`. Preserve session-cookie auth, same-origin rules
|
||||||
|
for cookie mutations and WebSocket upgrades, and the "waking"/`502`/`503`
|
||||||
|
semantics unless the change is explicitly about them. Do not expose a trusted
|
||||||
|
identity header unless the Service is reachable only through the proxy.
|
||||||
|
|
||||||
|
### Add an exposure variant
|
||||||
|
|
||||||
|
Create `manifests/exposure/<variant>/NN-*.yaml`, update
|
||||||
|
`manifests/exposure/README.md` and the example README, and state the trust
|
||||||
|
boundary. Variants are alternatives: never apply two paths to the same gateway.
|
||||||
|
Set `DSH_TRUSTED_HOSTS` in the template for the public host and
|
||||||
|
`COOKIE_SECURE=true` for HTTPS.
|
||||||
|
|
||||||
|
### Add an example
|
||||||
|
|
||||||
|
Follow `docs/examples.md`: `README.md`, `manifests/NN-*.yaml`, optional
|
||||||
|
`manifests/exposure/` with a README, optional components with Dockerfile +
|
||||||
|
README + tests, and executable, idempotent, self-cleaning scripts. Add the
|
||||||
|
literal `examples/<name>/` entry to `examples/README.md`; `make check` enforces
|
||||||
|
the layout. Publish component images under
|
||||||
|
`<registry>/<owner>/<repo>/<example>/<component>`.
|
||||||
|
|
||||||
|
### Bump a version
|
||||||
|
|
||||||
|
- **DSH release identity:** `make set-version DSH_VERSION=<ref>
|
||||||
|
[RELEASE_REVISION=rN]`, then `make check`, then build and smoke-test the image.
|
||||||
|
Tags are immutable; fix a bad release with `-r(N+1)`.
|
||||||
|
- **agent-sandbox:** edit `AGENT_SANDBOX_VERSION` in `versions.env`, update the
|
||||||
|
prerequisites in the example README, and note behavior changes (connection
|
||||||
|
types, MCP output limits, CRD fields). Install the new release YAML on a kind
|
||||||
|
cluster, then run `make check` and `make e2e`. Do not bump it inside an
|
||||||
|
unrelated PR.
|
||||||
|
|
||||||
|
### Integrate the agent-sandbox MCP server
|
||||||
|
|
||||||
|
Build from the cluster's pinned tag, give it a dedicated namespace and
|
||||||
|
least-privilege ServiceAccount, keep it ClusterIP behind an authenticating
|
||||||
|
proxy, and document how to reach `/readyz`. Add a DSH profile row only with user
|
||||||
|
approval; a row affects every session. Never commit its credentials or a
|
||||||
|
`cluster-admin` binding.
|
||||||
|
|
||||||
|
### Documentation
|
||||||
|
|
||||||
|
Update the doc whose behavior changed in the same commit: root README for the
|
||||||
|
front door, `docs/` for repository-wide contracts, the example README for that
|
||||||
|
walkthrough, the gateway README for API/config/auth. Skill docs live in
|
||||||
|
`.dsh/skills/`; keep `SKILL.md` under ~8 KB and put depth in `references/`.
|
||||||
|
`scripts/check-versions.sh` scans every `*.md` for stale `dsh-v…` tokens, so
|
||||||
|
refer to "the release identity in `versions.env`" rather than copying a tag.
|
||||||
|
|
||||||
|
## Verification and review
|
||||||
|
|
||||||
|
Run at least `make check`; add `make test` and `make lint` for gateway/script
|
||||||
|
changes. For manifests, use `kubectl apply --dry-run=server` against a cluster
|
||||||
|
with the CRDs. For behavior changes, run the example or `make e2e`.
|
||||||
|
|
||||||
|
Before reporting, review:
|
||||||
|
|
||||||
|
- [ ] Diff is scoped; no generated files or unrelated changes.
|
||||||
|
- [ ] Docs and examples match the new behavior.
|
||||||
|
- [ ] Pins come only from `versions.env`; no stale version tokens.
|
||||||
|
- [ ] No secrets or usable credentials; secret-shaped files are `.example`.
|
||||||
|
- [ ] Manifests parse; exposure variants remain mutually exclusive.
|
||||||
|
- [ ] Gateway tests and public contract unchanged or intentionally updated.
|
||||||
|
- [ ] Skills: frontmatter valid, body within budget, scripts shellcheck-clean.
|
||||||
|
- [ ] `CHANGELOG.md` untouched; branch/commit follow `docs/gitflow.md`.
|
||||||
|
|
||||||
|
## Safe follow-ups enabled by this skill
|
||||||
|
|
||||||
|
- A read-only preflight job or scheduled report using
|
||||||
|
`scripts/platform-preflight.sh`.
|
||||||
|
- An MCP readiness check in the example e2e (health endpoints only).
|
||||||
|
- A separate runtime warm pool for untrusted execution, leaving the DSH template
|
||||||
|
unchanged.
|
||||||
|
- A combined DSH + sandboxd image only with an explicit maintenance/exec design
|
||||||
|
and a documented trust boundary.
|
||||||
@@ -0,0 +1,94 @@
|
|||||||
|
# Deploying the agent-sandbox MCP server
|
||||||
|
|
||||||
|
Read [the MCP reference](mcp-server.md) first. This file covers the connection
|
||||||
|
matrix, a safe deployment shape, and the DSH profile row. The server is
|
||||||
|
unauthenticated; its service account is the security boundary.
|
||||||
|
|
||||||
|
## Connection matrix
|
||||||
|
|
||||||
|
| `K8S_SANDBOX_CONNECTION__TYPE` | Required env | Use when |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `gateway` | `GATEWAY_NAME`, `GATEWAY_NAMESPACE=default`, `GATEWAY_READY_TIMEOUT=180`, `SERVER_PORT=8888` | Gateway API + implementation exist |
|
||||||
|
| `direct` | `API_URL` (router URL), `SERVER_PORT=8888` | sandbox-router is deployed |
|
||||||
|
| `in-cluster` | `USE_POD_IP=false` (DNS) or `true` (`status.podIPs`), `SERVER_PORT=8888` | Bypass the router from inside the cluster |
|
||||||
|
| `sandboxd-in-cluster` (v1.0.8+) | `MODE=in-cluster-service` or `in-cluster-pod-ip`, `REST_PORT=8080`, `GRPC_PORT=9090` | Template runs sandboxd |
|
||||||
|
|
||||||
|
Also set `K8S_SANDBOX_PROBE_NAMESPACE` to a namespace the server's service
|
||||||
|
account can list claims in, and optionally
|
||||||
|
`K8S_SANDBOX_SESSION_ID_LABEL_KEY` (default
|
||||||
|
`mcp.k8s-agent-sandbox/session-id`).
|
||||||
|
|
||||||
|
For `sandboxd-in-cluster` service mode the template needs `spec.service: true`.
|
||||||
|
The default managed NetworkPolicy admits only sandbox-router, so add an ingress
|
||||||
|
rule for the MCP server on 8080/9090. Supplying `spec.networkPolicy` replaces the
|
||||||
|
secure defaults: also add the egress rules the sandbox needs, because an omitted
|
||||||
|
egress list denies all outbound traffic. RBAC for that mode is
|
||||||
|
`sandboxclaims` create/get/list/watch/delete and `sandboxes` get/list/watch; no
|
||||||
|
`pods/portforward`.
|
||||||
|
|
||||||
|
## Build and deploy (sketch)
|
||||||
|
|
||||||
|
No official image reference is documented in the upstream README; pin what you
|
||||||
|
build.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# AGENT_SANDBOX_VERSION comes from versions.env; never hard-code it.
|
||||||
|
tmp="$(mktemp -d)"; trap 'rm -rf "$tmp"' EXIT
|
||||||
|
git clone --depth 1 --branch "$AGENT_SANDBOX_VERSION" \
|
||||||
|
https://github.com/kubernetes-sigs/agent-sandbox.git "$tmp"
|
||||||
|
docker build -t "$MCP_IMAGE" "$tmp/clients/integrations/mcp-server"
|
||||||
|
```
|
||||||
|
|
||||||
|
Kubernetes shape:
|
||||||
|
|
||||||
|
- Dedicated namespace and ServiceAccount; a namespace-scoped Role granting only
|
||||||
|
`sandboxclaims` create/get/list/watch/delete and `sandboxes` get/list/watch.
|
||||||
|
No Secrets, no `pods/exec`, no cluster-admin.
|
||||||
|
- Deployment: one replica, non-root, `readOnlyRootFilesystem`, dropped
|
||||||
|
capabilities, `K8S_SANDBOX_CONNECTION__TYPE=in-cluster`,
|
||||||
|
`K8S_SANDBOX_PROBE_NAMESPACE`, resource limits.
|
||||||
|
- Probes: `GET /healthz` (liveness; never calls the API) and `GET /readyz`
|
||||||
|
(readiness; lists claims; use `periodSeconds: 15`, not aggressive polling).
|
||||||
|
- ClusterIP Service on `8000`. Add a NetworkPolicy that allows only the intended
|
||||||
|
client namespace/pod. Never expose it through Ingress/LoadBalancer without an
|
||||||
|
authenticating proxy. The service account is the real boundary.
|
||||||
|
|
||||||
|
## Connect it to DSH
|
||||||
|
|
||||||
|
Add a profile MCP row (through the Plugins page/`plugin_manager`; see
|
||||||
|
`cordis-plugin-development`). A row affects every session and needs approval.
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
- insert:
|
||||||
|
- id: mcp-agent-sandbox
|
||||||
|
name: '@deepseek-ai/dsh-mcp-client'
|
||||||
|
config:
|
||||||
|
serverName: agent-sandbox
|
||||||
|
transport: streamable-http
|
||||||
|
url: http://agent-sandbox-mcp.example.svc.cluster.local:8000/mcp
|
||||||
|
toolCallTimeoutMs: 240000
|
||||||
|
failOnStartupError: false
|
||||||
|
```
|
||||||
|
|
||||||
|
- `serverName` must match `[A-Za-z0-9_-]{1,32}`; tools appear as
|
||||||
|
`mcp__<serverName>__<tool>`. Check the actual names in the tool list; do not
|
||||||
|
assume hyphens/underscores.
|
||||||
|
- Raise `toolCallTimeoutMs` above `sandbox_ready_timeout` with headroom; the DSH
|
||||||
|
default is 60 s and would cancel a 180 s creation.
|
||||||
|
- If the server is behind an authenticating proxy, pass its token via the row's
|
||||||
|
`headers` and an environment expression; never commit it.
|
||||||
|
- To reach a locally port-forwarded server, use a loopback URL; the MCP session
|
||||||
|
still needs streamable HTTP.
|
||||||
|
- Remove the row when it was enabled only for one task.
|
||||||
|
|
||||||
|
## Deployment safety checklist
|
||||||
|
|
||||||
|
- [ ] Build tag equals the cluster's `AGENT_SANDBOX_VERSION`.
|
||||||
|
- [ ] Dedicated namespace/SA; RBAC limited to claims and sandboxes.
|
||||||
|
- [ ] Service is ClusterIP; no public route without authentication.
|
||||||
|
- [ ] NetworkPolicy admits only the DSH client and needed sandbox traffic.
|
||||||
|
- [ ] `/readyz` succeeds with `K8S_SANDBOX_PROBE_NAMESPACE` set to a listable
|
||||||
|
namespace.
|
||||||
|
- [ ] No Secret data in manifests, logs, or the repository.
|
||||||
|
- [ ] Cleanup path documented: delete session-owned claims or wait for the
|
||||||
|
shutdown timeout.
|
||||||
103
.dsh/skills/dsh-agent-sandbox-platform/references/mcp-server.md
Normal file
103
.dsh/skills/dsh-agent-sandbox-platform/references/mcp-server.md
Normal file
@@ -0,0 +1,103 @@
|
|||||||
|
# agent-sandbox MCP server reference
|
||||||
|
|
||||||
|
Upstream: [clients/integrations/mcp-server](https://github.com/kubernetes-sigs/agent-sandbox/tree/main/clients/integrations/mcp-server).
|
||||||
|
It is a Python/FastAPI + FastMCP service (streamable HTTP on `:8000`, MCP
|
||||||
|
endpoint `/mcp`) that provisions sandboxes from a warm pool and runs commands or
|
||||||
|
file operations through a sandbox runtime. It ships with **no authentication**.
|
||||||
|
|
||||||
|
Build the server from the same tag as the cluster: read `AGENT_SANDBOX_VERSION`
|
||||||
|
in `versions.env`, and check the installed controller image. Do not build from
|
||||||
|
`main` against an older cluster; tool schemas and connection types differ.
|
||||||
|
|
||||||
|
## Tool surface
|
||||||
|
|
||||||
|
| Tool | Arguments (defaults) | Returns / notes |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `create_sandbox` | `warmpool`, `namespace`, `sandbox_ready_timeout=180`, `labels`, `shutdown_after_seconds=300` (`null` disables), `pod_labels`, `pod_annotations` | `sandbox_claim_name`; adds the MCP session label |
|
||||||
|
| `delete_sandbox` | `sandbox_claim_name`, `namespace` | Deletes the session-owned claim; PVC cascades |
|
||||||
|
| `execute_command` | `sandbox_claim_name`, `namespace`, `command`, `timeout=60` (max 600), `max_output_chars` only on v1.0.8+ (128-300,000, default 30,000) | `exit_code`, `stdout`, `stderr`; v1.0.8 adds `*_total_chars` and `*_truncated` |
|
||||||
|
| `upload_file` | `sandbox_claim_name`, `namespace`, `path`, `content`, `binary=false`, `timeout=60` | Bytes written; base64 when `binary=true` |
|
||||||
|
| `download_file` | `sandbox_claim_name`, `namespace`, `path`, `binary=false`, `timeout=60` | Content + bytes read |
|
||||||
|
| `list_files` | `sandbox_claim_name`, `namespace`, `path`, `timeout=60`, `max_entries=1000` (max 10,000) | Entries (`name`, `size`, `type`, `mod_time`), `total_entries`, `truncated` |
|
||||||
|
| `file_exists` | `sandbox_claim_name`, `namespace`, `path`, `timeout=60` | `exists`; a missing path is not an error |
|
||||||
|
| `get_sandbox_status` | `sandbox_claim_name`, `namespace` | `SandboxReady` / `SandboxNotReady` / `SandboxNotFound`, `ready`, `message` |
|
||||||
|
|
||||||
|
Resource: `sandboxes://{namespace}` lists sandboxes for the current MCP session.
|
||||||
|
|
||||||
|
## Session scoping is the most important behavior
|
||||||
|
|
||||||
|
Every tool, including `get_sandbox_status`, calls `get_sandbox()` ->
|
||||||
|
`ensure_session_owns()`, which lists sandboxes with the label selector
|
||||||
|
`<K8S_SANDBOX_SESSION_ID_LABEL_KEY>=<MCP session id>`. The resource uses the same
|
||||||
|
selector. Consequences:
|
||||||
|
|
||||||
|
- The server can only see/exec/delete claims **it created in the same MCP
|
||||||
|
session**. It cannot inspect or manage DSH platform claims created by the
|
||||||
|
gateway or `kubectl`, or claims from another session.
|
||||||
|
- On a DSH MCP reconnect, the MCP session changes; earlier claims become
|
||||||
|
invisible to the tools. They keep running until `shutdown_after_seconds`
|
||||||
|
expires, or until you remove them by label with a confirmed `kubectl delete`.
|
||||||
|
- A "Sandbox claim '<name>' is not found in namespace '<ns>'" error usually
|
||||||
|
means wrong namespace **or** the claim belongs to a different session.
|
||||||
|
- Use `kubectl` for global inventory; use the MCP only for sandboxes it created
|
||||||
|
in this task.
|
||||||
|
|
||||||
|
Session identity requires a session-providing transport. With stdio,
|
||||||
|
`ctx.session_id` is `None` and creation fails. Connect DSH over
|
||||||
|
`streamable-http`.
|
||||||
|
|
||||||
|
## Runtime prerequisite
|
||||||
|
|
||||||
|
The server talks to a sandbox runtime, not to arbitrary pods:
|
||||||
|
|
||||||
|
- `gateway`, `direct`, `in-cluster`: runtime HTTP on `SERVER_PORT=8888`
|
||||||
|
(sandbox-router API or direct pod runtime).
|
||||||
|
- `sandboxd-in-cluster` (v1.0.8+): sandboxd REST `8080`, gRPC `9090`.
|
||||||
|
|
||||||
|
The shipped DSH template runs only DSH Web on `3080`, so no connection type can
|
||||||
|
exec into it. A sandboxd **sidecar** executes commands in the sidecar's own
|
||||||
|
container, not in the DSH container; shared volumes share files, not binaries or
|
||||||
|
processes. To exec inside DSH, build a combined image (sandboxd injected into the
|
||||||
|
DSH image) or use `kubectl exec` with explicit RBAC and confirmation. For
|
||||||
|
untrusted code, deploy a separate runtime warm pool rather than modifying the
|
||||||
|
DSH template.
|
||||||
|
|
||||||
|
## Version differences
|
||||||
|
|
||||||
|
| Area | Before v1.0.8 | v1.0.8+ |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| Connection types | `gateway`, `direct`, `in-cluster` | Adds `sandboxd-in-cluster` |
|
||||||
|
| `execute_command` output | Unbounded | Bounded by `max_output_chars`; four metadata fields |
|
||||||
|
| Failed pod recovery | `Retain` only | `spec.failedPodPolicy: Recreate` |
|
||||||
|
| Router install | `kubectl apply -f sandbox-router/deploy/` | `kubectl apply -k sandbox-router/deploy/`; projected token |
|
||||||
|
|
||||||
|
Never pass `max_output_chars` to a pre-v1.0.8 server. On older servers, redirect
|
||||||
|
verbose output to a file and `download_file` it, or pipe through `tail`/`head`.
|
||||||
|
|
||||||
|
## Safety and cleanup
|
||||||
|
|
||||||
|
- No auth: anyone reaching `:8000` gets every tool and the service account's
|
||||||
|
access. Keep it ClusterIP, authenticated, and namespace-scoped.
|
||||||
|
- Treat commands as untrusted code: they can read files and environment
|
||||||
|
variables inside the sandbox, including provider keys in a combined image.
|
||||||
|
- `delete_sandbox` cascades the claim, sandbox, and PVC: confirm the target and
|
||||||
|
namespace. Never delete a namespace or PVC as cleanup.
|
||||||
|
- Clean up in the same session that created the sandbox. If the session is gone,
|
||||||
|
list claims by the session label and delete them explicitly (requires write
|
||||||
|
RBAC); otherwise rely on the shutdown timeout.
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
| Symptom | Cause / fix |
|
||||||
|
| --- | --- |
|
||||||
|
| Tools missing | Row not enabled, wrong URL, connection failed; check logs and `/readyz` |
|
||||||
|
| 404 on connect | URL is missing `/mcp` or points at the wrong port |
|
||||||
|
| `ctx.session_id is None` | Connected over stdio; use streamable HTTP |
|
||||||
|
| `Sandbox claim ... is not found` | Wrong namespace, or claim created by another MCP session |
|
||||||
|
| Creation times out at ~60 s | MCP session alive but DSH cancelled the call; raise `toolCallTimeoutMs` |
|
||||||
|
| `execute_command` connection error | No runtime on 8888/8080/9090, or wrong connection type; the DSH template is unsupported |
|
||||||
|
| Output flood | Pre-v1.0.8 server; bound the command output yourself |
|
||||||
|
| Empty sandbox list | The resource is session-scoped; use `kubectl` for global inventory |
|
||||||
|
|
||||||
|
Deployment, connection matrix, RBAC, and the DSH row:
|
||||||
|
[MCP deployment](mcp-deployment.md).
|
||||||
@@ -0,0 +1,181 @@
|
|||||||
|
# Platform runbook
|
||||||
|
|
||||||
|
Scope: the resource layer and gateway in
|
||||||
|
`examples/dsh-agents-as-a-service/`, one namespace, one DSH Web sandbox per
|
||||||
|
user. Read the example README and `AGENTS.md` first. All commands are read-only
|
||||||
|
unless a section is explicitly a change; confirm before changing anything.
|
||||||
|
|
||||||
|
## 1. Context, version, CRDs
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl config current-context
|
||||||
|
kubectl auth whoami
|
||||||
|
kubectl config view --minify -o jsonpath='{..namespace}{"\n"}'
|
||||||
|
kubectl get crd | grep -E 'agents\.x-k8s\.io'
|
||||||
|
kubectl -n agent-sandbox-system get deploy agent-sandbox-controller \
|
||||||
|
-o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
|
||||||
|
```
|
||||||
|
|
||||||
|
Compare the controller image tag with `AGENT_SANDBOX_VERSION` in
|
||||||
|
`versions.env`. The example needs the extension CRDs plus core `sandboxes`;
|
||||||
|
a missing CRD is a prerequisite failure, not an application bug.
|
||||||
|
|
||||||
|
## 2. Resource model
|
||||||
|
|
||||||
|
| Object | Role | Key fields |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `SandboxTemplate` | Blueprint; pinned image, probes, security context, PVC template | `envVarsInjectionPolicy`, `volumeClaimTemplatesPolicy`, `service`, `podTemplate` |
|
||||||
|
| `SandboxWarmPool` | Pre-warmed spares from one template | `spec.replicas`, `sandboxTemplateRef` |
|
||||||
|
| `SandboxClaim` | One tenant/user request; adopts a spare | `spec.warmPoolRef`, `additionalPodMetadata`, `status.sandbox.name` |
|
||||||
|
| `Sandbox` | Adopted or directly created pod + Service + PVC | `spec.operatingMode`, `status.conditions`, `status.serviceFQDN`, `status.podIPs` |
|
||||||
|
| `Pod` | The DSH Web process | `:3080`, PVC at `/home/node/.dsh` |
|
||||||
|
| `PersistentVolumeClaim` | Per-sandbox state | `dsh-home-<sandbox>` |
|
||||||
|
| `Service` | Stable per-sandbox DNS | `<sandbox>`, `status.serviceFQDN` |
|
||||||
|
|
||||||
|
Delete a claim to cascade-delete its sandbox and PVC (account deletion). Never
|
||||||
|
delete a PVC or namespace without an explicit, named confirmation.
|
||||||
|
|
||||||
|
## 3. Read-only inspection
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl -n "$NS" get sandboxtemplates,sandboxwarmpools,sandboxclaims,sandboxes -o wide
|
||||||
|
kubectl -n "$NS" get pods,pvc,svc -o wide
|
||||||
|
kubectl -n "$NS" get events --sort-by=.lastTimestamp | tail -30
|
||||||
|
kubectl -n "$NS" get sandboxclaim "$CLAIM" -o yaml
|
||||||
|
kubectl -n "$NS" get sandbox "$SB" -o yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
Bounded logs: `kubectl -n "$NS" logs "$POD" --tail=200` (add `--previous`
|
||||||
|
only after a restart).
|
||||||
|
|
||||||
|
## 4. Claim -> sandbox -> pod
|
||||||
|
|
||||||
|
```bash
|
||||||
|
SB=$(kubectl -n "$NS" get sandboxclaim "$CLAIM" -o jsonpath='{.status.sandbox.name}')
|
||||||
|
POD=$(kubectl -n "$NS" get sandbox "$SB" -o jsonpath='{.metadata.annotations.agents\.x-k8s\.io/pod-name}')
|
||||||
|
POD=${POD:-$SB}
|
||||||
|
kubectl -n "$NS" get sandbox "$SB" -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}{"\n"}{end}'
|
||||||
|
kubectl -n "$NS" get sandbox "$SB" -o jsonpath='{.spec.operatingMode}{" "}{.status.serviceFQDN}{" "}{.status.podIPs}{"\n"}'
|
||||||
|
```
|
||||||
|
|
||||||
|
Rules:
|
||||||
|
|
||||||
|
- Resolve user -> sandbox only through `claim.status.sandbox.name`; the sandbox
|
||||||
|
keeps its pool-generated name, not the claim name.
|
||||||
|
- A claim tracks binding; for runtime state read `Sandbox.spec.operatingMode`
|
||||||
|
and the `Suspended`/`Ready` conditions. A suspended sandbox is not serving
|
||||||
|
even when the claim still exists.
|
||||||
|
- A missing `claim.status.sandbox.name` means the claim did not bind: inspect
|
||||||
|
the Ready condition `reason` and events. The pod is normally named after the
|
||||||
|
sandbox; use the annotation when present.
|
||||||
|
|
||||||
|
## 5. Suspend, resume, wake
|
||||||
|
|
||||||
|
Suspend keeps the PVC and Service, deletes the pod:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl -n "$NS" patch sandbox "$SB" --type merge \
|
||||||
|
-p '{"spec":{"operatingMode":"Suspended"}}'
|
||||||
|
kubectl -n "$NS" wait --for=delete "pod/$POD" --timeout=120s
|
||||||
|
kubectl -n "$NS" get pvc "dsh-home-$SB"
|
||||||
|
kubectl -n "$NS" get svc "$SB"
|
||||||
|
```
|
||||||
|
|
||||||
|
Resume and verify state:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl -n "$NS" patch sandbox "$SB" --type merge \
|
||||||
|
-p '{"spec":{"operatingMode":"Running"}}'
|
||||||
|
kubectl -n "$NS" wait sandbox "$SB" --for=condition=Ready --timeout=180s
|
||||||
|
kubectl -n "$NS" exec "$SB" -- cat /home/node/.dsh/marker.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
The gateway wakes a suspended sandbox on the first request. The idle sweeper
|
||||||
|
only suspends claims it created (`aaas.example.com/managed-by=dsh-gateway`);
|
||||||
|
hand-applied claims are never swept.
|
||||||
|
|
||||||
|
## 6. Warm pool
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl -n "$NS" get sandboxwarmpool "$POOL" -o yaml
|
||||||
|
kubectl -n "$NS" get sandboxes -l agents.x-k8s.io/warm-pool-sandbox -o wide
|
||||||
|
kubectl -n "$NS" get sandboxwarmpool "$POOL" -o jsonpath='{.spec.replicas}{"\n"}'
|
||||||
|
```
|
||||||
|
|
||||||
|
Adopted sandboxes lose the warm-pool label; count current spares against
|
||||||
|
`spec.replicas`. Zero spares or `spec.replicas: 0` means cold, slow signup.
|
||||||
|
Scale only when asked:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl -n "$NS" patch sandboxwarmpool "$POOL" --type merge -p '{"spec":{"replicas":2}}'
|
||||||
|
```
|
||||||
|
|
||||||
|
## 7. DSH runtime specifics
|
||||||
|
|
||||||
|
- DSH Web listens on `3080` in-cluster. Port-forward to a free local port such
|
||||||
|
as `18081`, never `3080` if a local DSH already uses it.
|
||||||
|
- The launch token is printed once as `dsh web: http://<ip>:3080/?token=...`.
|
||||||
|
Treat it as a credential: do not paste it into reports; hand the user the URL.
|
||||||
|
- `/` returns `401` without the token. Probes must target `/favicon.svg`.
|
||||||
|
- `DSH_TRUSTED_HOSTS` must include any public host that a proxy forwards; DSH
|
||||||
|
fences on `Host`/`Origin`.
|
||||||
|
- State lives at `/home/node/.dsh`. DSH requires `.credentials.yaml` to be
|
||||||
|
`0600`; the template uses `fsGroupChangePolicy: OnRootMismatch`. If a resumed
|
||||||
|
sandbox crash-loops with a mode-660 error, suspend it, fix the file with a
|
||||||
|
throwaway pod, patch the policy, then resume.
|
||||||
|
|
||||||
|
## 8. Gateway layer
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl -n "$NS" get deploy,svc,pods -l app=dsh-gateway -o wide
|
||||||
|
kubectl -n "$NS" get endpoints dsh-gateway
|
||||||
|
kubectl -n "$NS" logs deploy/dsh-gateway --tail=100
|
||||||
|
```
|
||||||
|
|
||||||
|
The gateway is a ClusterIP Service on `8080`; it proxies under `/u/{user}/` and
|
||||||
|
derives state (`Provisioning`, `Waking`, `Ready`, `Suspending`, `Suspended`).
|
||||||
|
It performs the DSH launch-token handshake server-side. With SSO enabled
|
||||||
|
(`AUTH_PROXY_HEADER`), the Service must be reachable only through the
|
||||||
|
authenticating proxy; never add a second path or port-forward it on a shared
|
||||||
|
host. A resource-only install has no gateway: that is a warning, not a failure.
|
||||||
|
|
||||||
|
## 9. Security boundaries
|
||||||
|
|
||||||
|
- Provider keys (`dsh-provider-keys` plus the optional
|
||||||
|
`dsh-provider-keys-extra`) are shared by all sandboxes; the example is not
|
||||||
|
tenant-isolated.
|
||||||
|
- gVisor, `nodeSelector`, the sandbox/exposure NetworkPolicies and extra
|
||||||
|
provider keys are opt-in, not active by default. A NetworkPolicy only helps
|
||||||
|
with a CNI that enforces it; see
|
||||||
|
[optional hardening](../../../../examples/dsh-agents-as-a-service/docs/optional-hardening.md).
|
||||||
|
- Secret data, kubeconfigs, and pod logs are sensitive; RBAC may allow Secret
|
||||||
|
access, policy does not. Keep gateway RBAC namespace-scoped; it needs pod logs
|
||||||
|
to find the launch token.
|
||||||
|
- TLS terminates outside the gateway; set `COOKIE_SECURE=true` for HTTPS.
|
||||||
|
|
||||||
|
## 10. Failure matrix
|
||||||
|
|
||||||
|
| Symptom | First evidence | Likely cause / fix |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| Claim not Ready, no sandbox | Ready condition reason, events | Empty pool, image pull, quota, unschedulable |
|
||||||
|
| `EnvVarsInjectionRejected` | claim Ready reason | Claim carries env/volume templates; template policy rejects it |
|
||||||
|
| Pod Pending | `kubectl describe pod` | Requests/taints/affinity, unbound PVC |
|
||||||
|
| CrashLoopBackOff | logs, `--previous` | Startup failure, bad config, probe 401, fsGroup mode 660 |
|
||||||
|
| ImagePullBackOff | describe events | Wrong tag, private registry, missing pull secret |
|
||||||
|
| DSH 401 in browser | gateway logs | Launch-token handshake failed; check gateway RBAC `pods/log` |
|
||||||
|
| 502/503 through gateway | sandbox Ready, gateway logs | Wake timeout, DNS stale, token not in logs yet |
|
||||||
|
| Warm pool empty | sandboxes label count | Scale the pool or wait for replenishment |
|
||||||
|
| Suspended but expected running | `operatingMode` | Resume it; the gateway only wakes on a request |
|
||||||
|
| WS closes `1011` | gateway logs | DSH rejected the upstream Host/authority; check trusted hosts |
|
||||||
|
|
||||||
|
## 11. Cleanup and reporting
|
||||||
|
|
||||||
|
Delete only claims/sandboxes you created, by name:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl -n "$NS" delete sandboxclaim "$CLAIM" --dry-run=server
|
||||||
|
kubectl -n "$NS" delete sandboxclaim "$CLAIM"
|
||||||
|
```
|
||||||
|
|
||||||
|
Report context/namespace, evidence, conclusion, change, rollback, and what was
|
||||||
|
deleted. Leave the namespace, pools, templates, and unrelated claims alone.
|
||||||
@@ -0,0 +1,71 @@
|
|||||||
|
# Troubleshooting
|
||||||
|
|
||||||
|
Start with evidence, not fixes: resolve context/namespace, read conditions and
|
||||||
|
events, then compare desired vs observed state. Keep output bounded. Never
|
||||||
|
"fix" an access error by broadening credentials.
|
||||||
|
|
||||||
|
## Platform
|
||||||
|
|
||||||
|
| Symptom | First evidence | Likely cause / fix |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `Forbidden` on every call | `kubectl auth whoami`, `kubectl auth can-i --list -n <ns>` | Wrong context or RBAC. Do not retry with broader credentials; ask for the right kubeconfig. |
|
||||||
|
| CRDs not found | `kubectl get crd | grep agents.x-k8s.io` | agent-sandbox extensions not installed. Install the pinned release YAML; that is a prerequisite, not a code change. |
|
||||||
|
| Claim created, no sandbox | `claim.status.conditions[?(@.type=="Ready")]`, events | Empty pool, image pull, quota, unschedulable; or policy rejection. |
|
||||||
|
| Claim rejected | Ready reason `EnvVarsInjectionRejected` | Claim carries `env`/`volumeClaimTemplates` while the template disallows injection. Remove them; do not bypass the pool. |
|
||||||
|
| User is suspended but the claim exists | `Sandbox.spec.operatingMode`, `Suspended` condition | Resume with `operatingMode: Running`; a gateway request would wake it. |
|
||||||
|
| Warm pool exhausted | `kubectl get sandboxes -l agents.x-k8s.io/warm-pool-sandbox` | Scale `spec.replicas` (with confirmation) or accept slow cold provisioning. |
|
||||||
|
| Pod `Pending` | `kubectl describe pod` | Requests/limits, taints, affinity, unbound PVC, storage class. |
|
||||||
|
| `ImagePullBackOff` | describe events | Wrong image/tag, private GHCR package, missing pull secret; on kind, load the image under the exact manifest ref. |
|
||||||
|
| `CrashLoopBackOff`, probe 401 | logs, `describe` | Probe path is `/`; use `/favicon.svg`. |
|
||||||
|
| Restart with mode-660 error | logs mention `.credentials.yaml` | kubelet fsGroup widened the file; suspend, `chmod 600` from a throwaway pod, patch `fsGroupChangePolicy: OnRootMismatch`, resume. |
|
||||||
|
| PVC `Pending` | `kubectl describe pvc`, StorageClass | No default provisioner, access mode, zone binding. |
|
||||||
|
| Launch token not found | `kubectl logs <pod> --tail=2000 | grep 'dsh web:'` | Pod not ready yet, or logs rotated. Do not guess tokens; wait for readiness. |
|
||||||
|
| Hand-applied sandbox disappeared | claim annotations, gateway logs | Idle sweeper only manages claims with `aaas.example.com/managed-by=dsh-gateway`; if it was managed, recreate/resume. |
|
||||||
|
| Gateway 401 | gateway logs, cookie flags | Missing/expired cookie, DSH handshake failed, or `COOKIE_SECURE=true` over HTTP. |
|
||||||
|
| Gateway 502/503 | sandbox Ready, gateway logs | Wake timeout, stale DNS, token not yet in logs; check `WAKE_TIMEOUT` and RBAC `pods/log`. |
|
||||||
|
| WebSocket closes 1011 | gateway logs, `Host`/Origin | DSH rejected the upstream authority; check `DSH_TRUSTED_HOSTS` and the gateway's pod-IP dial path. |
|
||||||
|
| WebSocket closes 4401 | cookie, Origin | Missing gateway auth or cross-origin upgrade; send a matching `Origin`. |
|
||||||
|
| SSO header ignored | gateway env | `AUTH_PROXY_HEADER` unset; setting it requires the proxy to be the only path to the ClusterIP Service. |
|
||||||
|
| Someone is treated as another user | service paths | Trust-boundary violation: the gateway is directly reachable. Remove the extra path and disable SSO until fixed. |
|
||||||
|
|
||||||
|
## Agent-sandbox MCP
|
||||||
|
|
||||||
|
| Symptom | First evidence | Likely cause / fix |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| No `mcp__*__create_sandbox` tools | tool list, DSH logs | Server not deployed or row not enabled. Use `kubectl`; connect only with approval. |
|
||||||
|
| 404 on connect | curl `/mcp`, `/healthz` | Wrong path/port; MCP endpoint is `/mcp`, health is at the root. |
|
||||||
|
| `/readyz` 503 | response body, server logs | API unreachable or namespace `K8S_SANDBOX_PROBE_NAMESPACE` not listable; fix RBAC/config. |
|
||||||
|
| `ctx.session_id is None` | server error | Connected over stdio; use streamable HTTP. |
|
||||||
|
| `Sandbox claim ... is not found` | server error, `kubectl get claim` | Wrong namespace, or the claim belongs to another MCP session. List with `kubectl`; use the MCP only for its own claims. |
|
||||||
|
| Creation cancelled at ~60 s | MCP call log | DSH `toolCallTimeoutMs` is 60 s while creation waits up to 180 s; raise the timeout. |
|
||||||
|
| `execute_command` connection error | server logs, template ports | No runtime on 8888/8080/9090. The DSH template only listens on 3080; use a runtime template or a combined image. |
|
||||||
|
| Output truncated / flooded | tool result | Pre-v1.0.8 is unbounded; v1.0.8 bounds output. Redirect verbose output to a file and read/download it; never pass `max_output_chars` to an old server. |
|
||||||
|
| Orphaned claims after reconnect | `kubectl get claims -l mcp.k8s-agent-sandbox/session-id` | MCP sessions are transient; wait for `shutdown_after_seconds` or delete by label with explicit confirmation. |
|
||||||
|
| `Forbidden` from the MCP server | server logs | Its ServiceAccount lacks claim/sandbox verbs, or RBAC is in another namespace; fix the Role, not the kubeconfig. |
|
||||||
|
| Sandbox expired | `get_sandbox_status` -> `SandboxNotFound` | `shutdown_after_seconds` elapsed; create a new sandbox and keep the task inside the budget. |
|
||||||
|
|
||||||
|
## Repository checks and CI
|
||||||
|
|
||||||
|
| Symptom | Cause / fix |
|
||||||
|
| --- | --- |
|
||||||
|
| `check-versions: stale version reference in docs: dsh-v...` | A copied release tag in a Markdown file. Refer to the identity in `versions.env`; run `make set-version` only for a real bump. |
|
||||||
|
| `check-versions: hard-coded version outside versions.env` | A tag in `.github/workflows` or the Makefile; move it to `versions.env`. |
|
||||||
|
| `check-examples` layout error | Missing `README.md`/`manifests/`, stray YAML at the example root, missing exposure README, non-executable script, or missing `examples/README.md` entry. |
|
||||||
|
| Manifest dry-run fails in CI | Invalid YAML or CRD schema; parse locally and run `kubectl apply --dry-run=server` against the pinned CRDs. |
|
||||||
|
| `shellcheck` failure | Quote expansions, use arrays for kubectl args, avoid unused variables; run shellcheck locally. |
|
||||||
|
| Lychee link failure | A relative/absolute link does not resolve; the link checker runs offline on the listed docs. |
|
||||||
|
| PR-title check failure | Use Conventional Commits: `feat(scope): ...`, `fix(scope): ...`, `docs(scope): ...`. |
|
||||||
|
| Changelog conflict | Never edit `CHANGELOG.md`; it is generated from commits. |
|
||||||
|
|
||||||
|
## Do not
|
||||||
|
|
||||||
|
- Do not use `--force --grace-period=0` to clear a stuck resource before
|
||||||
|
understanding why it is stuck.
|
||||||
|
- Do not delete namespaces, PVCs, PVs, CRDs, or cluster-scoped objects as a
|
||||||
|
cleanup shortcut.
|
||||||
|
- Do not read Secret data, kubeconfigs, or node logs to "debug" a permission
|
||||||
|
problem, and do not paste pod logs containing the DSH launch token into a
|
||||||
|
report.
|
||||||
|
- Do not expose the unauthenticated MCP server or enable `AUTH_PROXY_HEADER`
|
||||||
|
while another path to the gateway exists.
|
||||||
|
- Do not retry a `Forbidden` with a stronger kubeconfig; fix the RBAC scope.
|
||||||
298
.dsh/skills/dsh-agent-sandbox-platform/scripts/platform-preflight.sh
Executable file
298
.dsh/skills/dsh-agent-sandbox-platform/scripts/platform-preflight.sh
Executable file
@@ -0,0 +1,298 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Read-only preflight for the DSH agents-as-a-service platform.
|
||||||
|
#
|
||||||
|
# Checks the kubectl target, agent-sandbox CRDs/version, warm pools, claims,
|
||||||
|
# sandboxes, gateway and (optionally) an agent-sandbox MCP server. It never
|
||||||
|
# mutates the cluster, never reads Secret data, and never port-forwards.
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# platform-preflight.sh [--context NAME] [--namespace NS] [--pool NAME]
|
||||||
|
# [--template NAME] [--mcp-url URL] [--help]
|
||||||
|
#
|
||||||
|
# Exit codes: 0 = no failures, 1 = at least one failure, 2 = usage error.
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
REPO_ROOT="$(git -C "$SCRIPT_DIR" rev-parse --show-toplevel 2>/dev/null || true)"
|
||||||
|
|
||||||
|
CTX=""
|
||||||
|
NS="${NAMESPACE:-dsh-demo}"
|
||||||
|
POOL=""
|
||||||
|
TEMPLATE=""
|
||||||
|
MCP_URL=""
|
||||||
|
|
||||||
|
PASS=0
|
||||||
|
WARN=0
|
||||||
|
FAIL=0
|
||||||
|
|
||||||
|
usage() {
|
||||||
|
cat <<'EOF'
|
||||||
|
Read-only preflight for the DSH agents-as-a-service platform.
|
||||||
|
|
||||||
|
Options:
|
||||||
|
--context NAME kubectl context (default: current context)
|
||||||
|
--namespace NS namespace to inspect (default: $NAMESPACE or dsh-demo)
|
||||||
|
--pool NAME check only this warm pool
|
||||||
|
--template NAME check only this SandboxTemplate
|
||||||
|
--mcp-url URL also check an MCP server's /healthz and /readyz
|
||||||
|
--help show this help
|
||||||
|
|
||||||
|
The script makes no changes and never reads Secret data.
|
||||||
|
EOF
|
||||||
|
}
|
||||||
|
|
||||||
|
while [ "$#" -gt 0 ]; do
|
||||||
|
case "$1" in
|
||||||
|
--context) CTX="${2:-}"; shift 2 ;;
|
||||||
|
--namespace) NS="${2:-}"; shift 2 ;;
|
||||||
|
--pool) POOL="${2:-}"; shift 2 ;;
|
||||||
|
--template) TEMPLATE="${2:-}"; shift 2 ;;
|
||||||
|
--mcp-url) MCP_URL="${2:-}"; shift 2 ;;
|
||||||
|
--help|-h) usage; exit 0 ;;
|
||||||
|
*) echo "unknown argument: $1" >&2; usage >&2; exit 2 ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
|
||||||
|
if [ -z "$NS" ]; then
|
||||||
|
echo "namespace must not be empty" >&2
|
||||||
|
exit 2
|
||||||
|
fi
|
||||||
|
|
||||||
|
section() { printf '\n== %s ==\n' "$1"; }
|
||||||
|
pass() { PASS=$((PASS + 1)); printf 'PASS %s\n' "$1"; }
|
||||||
|
warn() { WARN=$((WARN + 1)); printf 'WARN %s\n' "$1"; }
|
||||||
|
fail() { FAIL=$((FAIL + 1)); printf 'FAIL %s\n' "$1"; }
|
||||||
|
info() { printf 'INFO %s\n' "$1"; }
|
||||||
|
|
||||||
|
command -v kubectl >/dev/null 2>&1 || {
|
||||||
|
echo "FAIL kubectl is not installed" >&2
|
||||||
|
exit 1
|
||||||
|
}
|
||||||
|
|
||||||
|
if [ -n "$CTX" ]; then
|
||||||
|
if ! kubectl config get-contexts -o name 2>/dev/null | grep -Fxq "$CTX"; then
|
||||||
|
echo "FAIL context '$CTX' is not in the kubeconfig" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
KUBECTL=(kubectl --context "$CTX" --request-timeout=15s)
|
||||||
|
else
|
||||||
|
KUBECTL=(kubectl --request-timeout=15s)
|
||||||
|
fi
|
||||||
|
|
||||||
|
k() { "${KUBECTL[@]}" "$@"; }
|
||||||
|
|
||||||
|
section "Target"
|
||||||
|
if current="$(k config current-context 2>/dev/null)"; then
|
||||||
|
info "context: $current"
|
||||||
|
pass "kubeconfig is readable"
|
||||||
|
else
|
||||||
|
fail "cannot read the current kubectl context"
|
||||||
|
fi
|
||||||
|
if user="$(k auth whoami -o jsonpath='{.status.userInfo.username}' 2>/dev/null)" && [ -n "$user" ]; then
|
||||||
|
info "identity: $user"
|
||||||
|
else
|
||||||
|
fallback_user="$(k config view --minify -o jsonpath='{.users[0].name}' 2>/dev/null || true)"
|
||||||
|
info "identity: ${fallback_user:-unknown}"
|
||||||
|
fi
|
||||||
|
if k get namespace "$NS" -o name >/dev/null 2>&1; then
|
||||||
|
pass "namespace $NS exists"
|
||||||
|
else
|
||||||
|
fail "namespace $NS is missing or not readable"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ -n "$REPO_ROOT" ] && [ -f "$REPO_ROOT/versions.env" ]; then
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
. "$REPO_ROOT/versions.env"
|
||||||
|
info "expected agent-sandbox pin: ${AGENT_SANDBOX_VERSION:-unknown}"
|
||||||
|
else
|
||||||
|
warn "versions.env not found; skipping the pin comparison"
|
||||||
|
fi
|
||||||
|
|
||||||
|
section "Capabilities (informational; never widen them)"
|
||||||
|
for check in "create sandboxclaims" "delete sandboxclaims" "create pods/exec"; do
|
||||||
|
verb="${check%% *}"
|
||||||
|
resource="${check#* }"
|
||||||
|
if [ "$(k auth can-i "$verb" "$resource" -n "$NS" 2>/dev/null || echo unknown)" = "yes" ]; then
|
||||||
|
info "can $verb $resource in $NS: yes"
|
||||||
|
else
|
||||||
|
info "can $verb $resource in $NS: no"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
if [ "$(k auth can-i get secrets -n "$NS" 2>/dev/null || echo unknown)" = "yes" ]; then
|
||||||
|
warn "RBAC allows reading Secrets; policy still forbids printing Secret data"
|
||||||
|
else
|
||||||
|
info "can get secrets in $NS: no"
|
||||||
|
fi
|
||||||
|
|
||||||
|
section "Agent Sandbox CRDs"
|
||||||
|
for crd in \
|
||||||
|
sandboxtemplates.extensions.agents.x-k8s.io \
|
||||||
|
sandboxwarmpools.extensions.agents.x-k8s.io \
|
||||||
|
sandboxclaims.extensions.agents.x-k8s.io \
|
||||||
|
sandboxes.agents.x-k8s.io; do
|
||||||
|
if k get crd "$crd" -o name >/dev/null 2>&1; then
|
||||||
|
pass "CRD $crd"
|
||||||
|
else
|
||||||
|
fail "CRD $crd is missing or not readable"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
section "Controller"
|
||||||
|
controller_ns="agent-sandbox-system"
|
||||||
|
controller_image="$(k -n "$controller_ns" get deployment agent-sandbox-controller \
|
||||||
|
-o jsonpath='{.spec.template.spec.containers[0].image}' 2>/dev/null || true)"
|
||||||
|
if [ -n "$controller_image" ]; then
|
||||||
|
info "controller image: $controller_image"
|
||||||
|
controller_tag="${controller_image##*:}"
|
||||||
|
if [ -n "${AGENT_SANDBOX_VERSION:-}" ] && [ "$controller_tag" != "$AGENT_SANDBOX_VERSION" ]; then
|
||||||
|
warn "controller tag '$controller_tag' differs from AGENT_SANDBOX_VERSION '$AGENT_SANDBOX_VERSION'"
|
||||||
|
else
|
||||||
|
pass "controller version matches versions.env"
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
warn "controller deployment not found in $controller_ns; cannot verify the installed version"
|
||||||
|
fi
|
||||||
|
|
||||||
|
section "Templates"
|
||||||
|
templates="$(k -n "$NS" get sandboxtemplates \
|
||||||
|
-o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' 2>/dev/null || true)"
|
||||||
|
if [ -z "$templates" ]; then
|
||||||
|
warn "no SandboxTemplates in $NS"
|
||||||
|
fi
|
||||||
|
if [ -n "$TEMPLATE" ] && ! printf '%s\n' "$templates" | grep -Fxq "$TEMPLATE"; then
|
||||||
|
fail "SandboxTemplate $TEMPLATE not found in $NS"
|
||||||
|
fi
|
||||||
|
for tpl in $templates; do
|
||||||
|
[ -n "$TEMPLATE" ] && [ "$tpl" != "$TEMPLATE" ] && continue
|
||||||
|
desc="$(k -n "$NS" get sandboxtemplate "$tpl" \
|
||||||
|
-o jsonpath='{range .spec.podTemplate.spec.containers[*]}{.name}{":"}{range .ports[*]}{.containerPort}{","}{end}{";"}{end}' \
|
||||||
|
2>/dev/null || true)"
|
||||||
|
info "template $tpl containers: ${desc:-<none>}"
|
||||||
|
case "$desc" in
|
||||||
|
*sandboxd*|*:8080,*|*:9090,*)
|
||||||
|
pass "template $tpl exposes a sandboxd runtime (MCP sandboxd-in-cluster)" ;;
|
||||||
|
*:8888,*)
|
||||||
|
pass "template $tpl exposes a runtime HTTP port (MCP gateway/direct/in-cluster)" ;;
|
||||||
|
*:3080,*)
|
||||||
|
warn "template $tpl is DSH Web only (:3080); MCP execute_command cannot run against it" ;;
|
||||||
|
*)
|
||||||
|
warn "template $tpl exposes no known sandbox runtime port" ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
|
||||||
|
section "Warm pools"
|
||||||
|
pools="$(k -n "$NS" get sandboxwarmpools \
|
||||||
|
-o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' 2>/dev/null || true)"
|
||||||
|
if [ -z "$pools" ]; then
|
||||||
|
warn "no SandboxWarmPools in $NS; claims will provision cold"
|
||||||
|
fi
|
||||||
|
if [ -n "$POOL" ] && ! printf '%s\n' "$pools" | grep -Fxq "$POOL"; then
|
||||||
|
fail "SandboxWarmPool $POOL not found in $NS"
|
||||||
|
fi
|
||||||
|
for pool in $pools; do
|
||||||
|
[ -n "$POOL" ] && [ "$pool" != "$POOL" ] && continue
|
||||||
|
replicas="$(k -n "$NS" get sandboxwarmpool "$pool" -o jsonpath='{.spec.replicas}' 2>/dev/null || true)"
|
||||||
|
selector="$(k -n "$NS" get sandboxwarmpool "$pool" -o jsonpath='{.status.selector}' 2>/dev/null || true)"
|
||||||
|
if [ -n "$selector" ]; then
|
||||||
|
ready="$(k -n "$NS" get sandboxes -l "$selector" \
|
||||||
|
-o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' \
|
||||||
|
2>/dev/null | grep -c '^True$' || true)"
|
||||||
|
else
|
||||||
|
ready="$(k -n "$NS" get sandboxes -l agents.x-k8s.io/warm-pool-sandbox \
|
||||||
|
-o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' \
|
||||||
|
2>/dev/null | grep -c '^True$' || true)"
|
||||||
|
fi
|
||||||
|
replicas="${replicas:-0}"
|
||||||
|
info "pool $pool: $ready/$replicas warm spares Ready"
|
||||||
|
if [ "$replicas" -eq 0 ]; then
|
||||||
|
warn "pool $pool has spec.replicas=0; signup will cold-provision"
|
||||||
|
elif [ "$ready" -lt "$replicas" ]; then
|
||||||
|
warn "pool $pool is below its desired spare count"
|
||||||
|
else
|
||||||
|
pass "pool $pool is at capacity"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
section "Claims"
|
||||||
|
claims="$(k -n "$NS" get sandboxclaims \
|
||||||
|
-o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.conditions[?(@.type=="Ready")].status}{"\t"}{.status.conditions[?(@.type=="Ready")].reason}{"\t"}{.status.sandbox.name}{"\n"}{end}' \
|
||||||
|
2>/dev/null || true)"
|
||||||
|
if [ -z "$claims" ]; then
|
||||||
|
warn "no SandboxClaims in $NS"
|
||||||
|
else
|
||||||
|
while IFS=$'\t' read -r name ready reason sandbox; do
|
||||||
|
[ -n "$name" ] || continue
|
||||||
|
if [ "$ready" != "True" ]; then
|
||||||
|
warn "claim $name is not Ready (status='${ready:-unknown}', reason='${reason:-unknown}')"
|
||||||
|
elif [ -z "$sandbox" ]; then
|
||||||
|
warn "claim $name is Ready but has no bound sandbox"
|
||||||
|
else
|
||||||
|
pass "claim $name -> sandbox $sandbox"
|
||||||
|
fi
|
||||||
|
done <<< "$claims"
|
||||||
|
fi
|
||||||
|
|
||||||
|
section "Sandboxes"
|
||||||
|
sandboxes="$(k -n "$NS" get sandboxes \
|
||||||
|
-o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.operatingMode}{"\t"}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' \
|
||||||
|
2>/dev/null || true)"
|
||||||
|
if [ -z "$sandboxes" ]; then
|
||||||
|
warn "no Sandboxes in $NS"
|
||||||
|
else
|
||||||
|
while IFS=$'\t' read -r name mode ready; do
|
||||||
|
[ -n "$name" ] || continue
|
||||||
|
info "sandbox $name mode=${mode:-Running} ready=${ready:-unknown}"
|
||||||
|
done <<< "$sandboxes"
|
||||||
|
fi
|
||||||
|
|
||||||
|
section "Storage and Services"
|
||||||
|
pvc_count="$(k -n "$NS" get pvc -o name 2>/dev/null | wc -l | tr -d ' ')"
|
||||||
|
svc_count="$(k -n "$NS" get service -o name 2>/dev/null | wc -l | tr -d ' ')"
|
||||||
|
info "persistent volume claims: ${pvc_count:-0}"
|
||||||
|
info "services: ${svc_count:-0}"
|
||||||
|
[ "${pvc_count:-0}" -gt 0 ] || warn "no PVCs in $NS"
|
||||||
|
[ "${svc_count:-0}" -gt 0 ] || warn "no Services in $NS"
|
||||||
|
|
||||||
|
section "Gateway"
|
||||||
|
if k -n "$NS" get deployment dsh-gateway -o name >/dev/null 2>&1; then
|
||||||
|
pass "gateway deployment dsh-gateway exists"
|
||||||
|
if k -n "$NS" get service dsh-gateway -o name >/dev/null 2>&1; then
|
||||||
|
endpoints="$(k -n "$NS" get endpoints dsh-gateway \
|
||||||
|
-o jsonpath='{range .subsets[*].addresses[*]}{.ip}{" "}{end}' 2>/dev/null || true)"
|
||||||
|
if [ -n "$endpoints" ]; then
|
||||||
|
pass "gateway endpoints: $endpoints"
|
||||||
|
else
|
||||||
|
warn "gateway Service has no ready endpoints"
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
warn "gateway Service dsh-gateway is missing"
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
warn "no dsh-gateway deployment; this is a resource-layer-only install"
|
||||||
|
fi
|
||||||
|
|
||||||
|
section "MCP server"
|
||||||
|
if [ -n "$MCP_URL" ]; then
|
||||||
|
if ! command -v curl >/dev/null 2>&1; then
|
||||||
|
fail "curl is required for --mcp-url"
|
||||||
|
else
|
||||||
|
base="${MCP_URL%/}"
|
||||||
|
base="${base%/mcp}"
|
||||||
|
for endpoint in healthz readyz; do
|
||||||
|
body="$(curl -fsS --max-time 5 "$base/$endpoint" 2>/dev/null | head -c 300 || true)"
|
||||||
|
if [ -n "$body" ]; then
|
||||||
|
pass "MCP /$endpoint: $body"
|
||||||
|
else
|
||||||
|
fail "MCP /$endpoint is unreachable at $base/$endpoint"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
info "no --mcp-url supplied; skipping server health checks"
|
||||||
|
fi
|
||||||
|
|
||||||
|
printf '\n== Summary ==\nPASS=%d WARN=%d FAIL=%d\n' "$PASS" "$WARN" "$FAIL"
|
||||||
|
if [ "$FAIL" -gt 0 ]; then
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
exit 0
|
||||||
15
.editorconfig
Normal file
15
.editorconfig
Normal file
@@ -0,0 +1,15 @@
|
|||||||
|
root = true
|
||||||
|
|
||||||
|
[*]
|
||||||
|
charset = utf-8
|
||||||
|
end_of_line = lf
|
||||||
|
insert_final_newline = true
|
||||||
|
trim_trailing_whitespace = true
|
||||||
|
indent_style = space
|
||||||
|
indent_size = 2
|
||||||
|
|
||||||
|
[*.py]
|
||||||
|
indent_size = 4
|
||||||
|
|
||||||
|
[Makefile]
|
||||||
|
indent_style = tab
|
||||||
47
.env.sample
47
.env.sample
@@ -1,29 +1,50 @@
|
|||||||
# Copy to .env and fill in. Never commit a real key.
|
# Copy to .env and fill in. Never commit a real key.
|
||||||
|
#
|
||||||
|
# The version values below are documentation only: versions.env is the single
|
||||||
|
# source of truth, and scripts/check-versions.sh fails if they drift.
|
||||||
|
|
||||||
# --- DSH Web image (Dockerfile / build.sh) ---
|
# --- Main DSH Web image (root Dockerfile) ---
|
||||||
DEEPSEEK_API_KEY=
|
DEEPSEEK_API_KEY=
|
||||||
DSH_VERSION=dsh-v0.2.1-rc.2
|
# Optional additional provider keys consumed by DSH. For Docker, pass them with
|
||||||
|
# -e; the Kubernetes examples inject them through the optional
|
||||||
|
# dsh-provider-keys-extra Secret.
|
||||||
|
OPENROUTER_API_KEY=
|
||||||
|
GEMINI_API_KEY=
|
||||||
|
NVIDIA_API_KEY=
|
||||||
|
OPENAI_API_KEY=
|
||||||
|
ANTHROPIC_API_KEY=
|
||||||
|
GROQ_API_KEY=
|
||||||
|
XAI_API_KEY=
|
||||||
|
MISTRAL_API_KEY=
|
||||||
|
TOGETHER_API_KEY=
|
||||||
|
CEREBRAS_API_KEY=
|
||||||
|
AI_GATEWAY_API_KEY=
|
||||||
|
HF_TOKEN=
|
||||||
|
DSH_VERSION=dsh-v0.2.1-rc.3
|
||||||
|
RELEASE_REVISION=r4
|
||||||
PNPM_VERSION=11.28.5
|
PNPM_VERSION=11.28.5
|
||||||
# DSH Web port inside the container, and the local port published by
|
# Container port, and the host port published by "make run".
|
||||||
# build.sh / docker run. The gateway also uses it as the upstream port.
|
|
||||||
DSH_PORT=3080
|
DSH_PORT=3080
|
||||||
# Comma/space separated extra Host allow-list entries for DSH (for an
|
# Comma/space separated extra Host allow-list entries for DSH (for a proxy or
|
||||||
# Ingress/public hostname). localhost and 127.0.0.1 are always trusted.
|
# public hostname). localhost and 127.0.0.1 are always trusted.
|
||||||
DSH_TRUSTED_HOSTS=
|
DSH_TRUSTED_HOSTS=
|
||||||
|
|
||||||
# --- Gateway (examples/dsh-agents-as-a-service) ---
|
# --- agents-as-a-service example ---
|
||||||
GATEWAY_IMAGE=ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway
|
|
||||||
NAMESPACE=dsh-demo
|
NAMESPACE=dsh-demo
|
||||||
POOL=dsh-pool
|
POOL=dsh-pool
|
||||||
# Seconds a sandbox may stay idle before the sweeper suspends it.
|
# Seconds a sandbox may stay idle before the sweeper suspends it, and the max
|
||||||
IDLE_TIMEOUT=60
|
# time a proxied request waits for a suspended sandbox to become Ready.
|
||||||
# Seconds a waking request may wait for the sandbox to become Ready.
|
IDLE_TIMEOUT=360
|
||||||
WAKE_TIMEOUT=120
|
WAKE_TIMEOUT=240
|
||||||
DSH_CONTAINER=dsh-web
|
DSH_CONTAINER=dsh-web
|
||||||
DSH_LOG_TAIL_LINES=2000
|
DSH_LOG_TAIL_LINES=2000
|
||||||
COOKIE_NAME=dsh_gw_session
|
COOKIE_NAME=dsh_gw_session
|
||||||
# Set to true only behind HTTPS; secure cookies are not sent over plain HTTP.
|
# Set to true only behind HTTPS; secure cookies are not sent over plain HTTP
|
||||||
|
# (localhost counts as a secure context in browsers).
|
||||||
COOKIE_SECURE=false
|
COOKIE_SECURE=false
|
||||||
COOKIE_MAX_AGE=604800
|
COOKIE_MAX_AGE=604800
|
||||||
MAX_BODY_MB=32
|
MAX_BODY_MB=32
|
||||||
HTTP_TIMEOUT=300
|
HTTP_TIMEOUT=300
|
||||||
|
# SSO is opt-in. Set this only when the gateway Service is reachable
|
||||||
|
# exclusively through oauth2-proxy (see the example exposure docs).
|
||||||
|
# AUTH_PROXY_HEADER=X-Forwarded-User
|
||||||
|
|||||||
42
.github/ISSUE_TEMPLATE/bug_report.yml
vendored
Normal file
42
.github/ISSUE_TEMPLATE/bug_report.yml
vendored
Normal file
@@ -0,0 +1,42 @@
|
|||||||
|
name: Bug report
|
||||||
|
description: Report a reproducible problem with the image, an example or the tooling.
|
||||||
|
labels: ["bug"]
|
||||||
|
body:
|
||||||
|
- type: markdown
|
||||||
|
attributes:
|
||||||
|
value: |
|
||||||
|
Do not include API keys, tokens or other secrets. Report security
|
||||||
|
issues privately via SECURITY.md instead.
|
||||||
|
- type: textarea
|
||||||
|
id: what-happened
|
||||||
|
attributes:
|
||||||
|
label: What happened
|
||||||
|
description: The observed behavior and the expected behavior.
|
||||||
|
validations:
|
||||||
|
required: true
|
||||||
|
- type: textarea
|
||||||
|
id: reproduce
|
||||||
|
attributes:
|
||||||
|
label: Reproduction steps
|
||||||
|
description: Commands, manifests and versions. Redact secrets.
|
||||||
|
placeholder: |
|
||||||
|
make image
|
||||||
|
docker run ...
|
||||||
|
validations:
|
||||||
|
required: true
|
||||||
|
- type: input
|
||||||
|
id: image-tag
|
||||||
|
attributes:
|
||||||
|
label: Image or release
|
||||||
|
description: The pinned tag or digest, for example dsh-v0.2.1-rc.3-r4.
|
||||||
|
- type: input
|
||||||
|
id: cluster
|
||||||
|
attributes:
|
||||||
|
label: Environment
|
||||||
|
description: Kubernetes distribution/version, kind version, agent-sandbox version.
|
||||||
|
- type: textarea
|
||||||
|
id: logs
|
||||||
|
attributes:
|
||||||
|
label: Relevant logs
|
||||||
|
description: Paste the smallest useful excerpt. Remove tokens and API keys.
|
||||||
|
render: shell
|
||||||
8
.github/ISSUE_TEMPLATE/config.yml
vendored
Normal file
8
.github/ISSUE_TEMPLATE/config.yml
vendored
Normal file
@@ -0,0 +1,8 @@
|
|||||||
|
blank_issues_enabled: false
|
||||||
|
contact_links:
|
||||||
|
- name: Security vulnerability
|
||||||
|
url: https://github.com/sredevopsorg/deepseek-kubernetes-sandbox/security/advisories/new
|
||||||
|
about: Report security issues privately; do not open a public issue.
|
||||||
|
- name: Documentation
|
||||||
|
url: https://github.com/sredevopsorg/deepseek-kubernetes-sandbox/tree/main/docs
|
||||||
|
about: Read the repository docs before opening an issue.
|
||||||
35
.github/ISSUE_TEMPLATE/feature_request.yml
vendored
Normal file
35
.github/ISSUE_TEMPLATE/feature_request.yml
vendored
Normal file
@@ -0,0 +1,35 @@
|
|||||||
|
name: Feature request
|
||||||
|
description: Propose a new example, image capability or workflow improvement.
|
||||||
|
labels: ["enhancement"]
|
||||||
|
body:
|
||||||
|
- type: textarea
|
||||||
|
id: problem
|
||||||
|
attributes:
|
||||||
|
label: Problem or use case
|
||||||
|
description: What are you trying to do, and what blocks you today?
|
||||||
|
validations:
|
||||||
|
required: true
|
||||||
|
- type: textarea
|
||||||
|
id: proposal
|
||||||
|
attributes:
|
||||||
|
label: Proposed solution
|
||||||
|
description: Describe the shape you have in mind. For a new example, say which implementation it demonstrates.
|
||||||
|
validations:
|
||||||
|
required: true
|
||||||
|
- type: dropdown
|
||||||
|
id: area
|
||||||
|
attributes:
|
||||||
|
label: Area
|
||||||
|
options:
|
||||||
|
- Main DSH Web image
|
||||||
|
- Existing example
|
||||||
|
- New example
|
||||||
|
- CI / release automation
|
||||||
|
- Documentation
|
||||||
|
- Other
|
||||||
|
validations:
|
||||||
|
required: true
|
||||||
|
- type: textarea
|
||||||
|
id: alternatives
|
||||||
|
attributes:
|
||||||
|
label: Alternatives considered
|
||||||
36
.github/dependabot.yml
vendored
Normal file
36
.github/dependabot.yml
vendored
Normal file
@@ -0,0 +1,36 @@
|
|||||||
|
version: 2
|
||||||
|
|
||||||
|
updates:
|
||||||
|
- package-ecosystem: github-actions
|
||||||
|
directory: /
|
||||||
|
schedule:
|
||||||
|
interval: weekly
|
||||||
|
groups:
|
||||||
|
actions:
|
||||||
|
patterns: ["*"]
|
||||||
|
commit-message:
|
||||||
|
prefix: "chore: "
|
||||||
|
|
||||||
|
- package-ecosystem: pip
|
||||||
|
directory: /examples/dsh-agents-as-a-service/gateway
|
||||||
|
schedule:
|
||||||
|
interval: weekly
|
||||||
|
groups:
|
||||||
|
gateway-python:
|
||||||
|
patterns: ["*"]
|
||||||
|
commit-message:
|
||||||
|
prefix: "chore: "
|
||||||
|
|
||||||
|
- package-ecosystem: docker
|
||||||
|
directory: /
|
||||||
|
schedule:
|
||||||
|
interval: weekly
|
||||||
|
commit-message:
|
||||||
|
prefix: "chore: "
|
||||||
|
|
||||||
|
- package-ecosystem: docker
|
||||||
|
directory: /examples/dsh-agents-as-a-service/gateway
|
||||||
|
schedule:
|
||||||
|
interval: weekly
|
||||||
|
commit-message:
|
||||||
|
prefix: "chore: "
|
||||||
31
.github/pull_request_template.md
vendored
Normal file
31
.github/pull_request_template.md
vendored
Normal file
@@ -0,0 +1,31 @@
|
|||||||
|
## Summary
|
||||||
|
|
||||||
|
<!-- What changes, and why? Link the issue: Closes #123 -->
|
||||||
|
|
||||||
|
## Type of change
|
||||||
|
|
||||||
|
- [ ] feat: new capability or example
|
||||||
|
- [ ] fix: bug fix
|
||||||
|
- [ ] docs: documentation only
|
||||||
|
- [ ] refactor: no behavior change
|
||||||
|
- [ ] build / ci: tooling, images, workflows
|
||||||
|
- [ ] chore: maintenance
|
||||||
|
|
||||||
|
## Breaking change
|
||||||
|
|
||||||
|
- [ ] Yes, and the migration is documented in the description
|
||||||
|
- [ ] No
|
||||||
|
|
||||||
|
## Checklist
|
||||||
|
|
||||||
|
- [ ] make check passes (versions, example layout, manifest YAML)
|
||||||
|
- [ ] make test passes (gateway unit tests)
|
||||||
|
- [ ] make lint passes (ruff, shellcheck; yamllint/actionlint where installed)
|
||||||
|
- [ ] Docs updated for the change (root README, docs/, example README)
|
||||||
|
- [ ] No secrets or usable credentials committed
|
||||||
|
- [ ] CHANGELOG.md was not edited by hand (generated by git-cliff)
|
||||||
|
- [ ] Version pins changed only through versions.env / make set-version
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
<!-- Commands run, clusters used, results. Mention if kind e2e was skipped. -->
|
||||||
160
.github/workflows/_build-image.yaml
vendored
Normal file
160
.github/workflows/_build-image.yaml
vendored
Normal file
@@ -0,0 +1,160 @@
|
|||||||
|
# Reusable multi-arch image build: one workflow, called by build-main,
|
||||||
|
# build-gateway and release. Builds natively per platform (no QEMU), pushes by
|
||||||
|
# digest, then merges the manifest list with SBOM and provenance attestations.
|
||||||
|
name: Reusable multi-arch image build
|
||||||
|
|
||||||
|
on:
|
||||||
|
workflow_call:
|
||||||
|
inputs:
|
||||||
|
image:
|
||||||
|
description: Fully qualified image name without a tag
|
||||||
|
required: true
|
||||||
|
type: string
|
||||||
|
version:
|
||||||
|
description: Version recorded in the OCI labels
|
||||||
|
required: true
|
||||||
|
type: string
|
||||||
|
context:
|
||||||
|
description: Docker build context
|
||||||
|
required: true
|
||||||
|
type: string
|
||||||
|
dockerfile:
|
||||||
|
description: Dockerfile path, relative to the context
|
||||||
|
required: false
|
||||||
|
default: Dockerfile
|
||||||
|
type: string
|
||||||
|
build-args:
|
||||||
|
description: Newline-separated build args
|
||||||
|
required: false
|
||||||
|
default: ""
|
||||||
|
type: string
|
||||||
|
tags:
|
||||||
|
description: Newline-separated fully qualified tags to publish
|
||||||
|
required: true
|
||||||
|
type: string
|
||||||
|
|
||||||
|
permissions:
|
||||||
|
contents: read
|
||||||
|
packages: write
|
||||||
|
id-token: write
|
||||||
|
attestations: write
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
build:
|
||||||
|
name: Build ${{ matrix.platform }}
|
||||||
|
runs-on: ${{ matrix.platform == 'linux/amd64' && 'ubuntu-latest' || 'ubuntu-24.04-arm' }}
|
||||||
|
timeout-minutes: 120
|
||||||
|
strategy:
|
||||||
|
fail-fast: false
|
||||||
|
matrix:
|
||||||
|
platform:
|
||||||
|
- linux/amd64
|
||||||
|
- linux/arm64
|
||||||
|
steps:
|
||||||
|
- name: Checkout
|
||||||
|
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
||||||
|
|
||||||
|
- name: Prepare environment
|
||||||
|
env:
|
||||||
|
IMAGE: ${{ inputs.image }}
|
||||||
|
run: |
|
||||||
|
set -euo pipefail
|
||||||
|
pair=$(printf '%s' "${{ matrix.platform }}" | tr '/' '-')
|
||||||
|
echo "PLATFORM_PAIR=$pair" >> "$GITHUB_ENV"
|
||||||
|
echo "CACHE_SCOPE=$(printf '%s' "$IMAGE" | tr '/' '-')" >> "$GITHUB_ENV"
|
||||||
|
|
||||||
|
- name: Set up Docker Buildx
|
||||||
|
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
|
||||||
|
|
||||||
|
- name: Login to GitHub Container Registry
|
||||||
|
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
|
||||||
|
with:
|
||||||
|
registry: ghcr.io
|
||||||
|
username: ${{ github.actor }}
|
||||||
|
password: ${{ secrets.GITHUB_TOKEN }}
|
||||||
|
|
||||||
|
- name: Build and push by digest
|
||||||
|
id: build
|
||||||
|
uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7.4.0
|
||||||
|
with:
|
||||||
|
context: ${{ inputs.context }}
|
||||||
|
file: ${{ inputs.context }}/${{ inputs.dockerfile }}
|
||||||
|
platforms: ${{ matrix.platform }}
|
||||||
|
build-args: ${{ inputs.build-args }}
|
||||||
|
labels: |
|
||||||
|
org.opencontainers.image.source=https://github.com/${{ github.repository }}
|
||||||
|
org.opencontainers.image.version=${{ inputs.version }}
|
||||||
|
org.opencontainers.image.revision=${{ github.sha }}
|
||||||
|
outputs: type=image,name=${{ inputs.image }},push-by-digest=true,name-canonical=true,push=true,oci-mediatypes=true
|
||||||
|
sbom: true
|
||||||
|
provenance: mode=max
|
||||||
|
cache-from: type=gha,scope=${{ env.CACHE_SCOPE }}-${{ matrix.platform }}
|
||||||
|
cache-to: type=gha,mode=max,scope=${{ env.CACHE_SCOPE }}-${{ matrix.platform }}
|
||||||
|
|
||||||
|
- name: Export digest
|
||||||
|
run: |
|
||||||
|
set -euo pipefail
|
||||||
|
mkdir -p /tmp/digests
|
||||||
|
digest="${{ steps.build.outputs.digest }}"
|
||||||
|
touch "/tmp/digests/${digest#sha256:}"
|
||||||
|
|
||||||
|
- name: Upload digest
|
||||||
|
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
|
||||||
|
with:
|
||||||
|
name: digests-${{ env.PLATFORM_PAIR }}
|
||||||
|
path: /tmp/digests/*
|
||||||
|
if-no-files-found: error
|
||||||
|
retention-days: 1
|
||||||
|
|
||||||
|
merge:
|
||||||
|
name: Merge manifest list
|
||||||
|
needs: build
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
timeout-minutes: 30
|
||||||
|
steps:
|
||||||
|
- name: Download digests
|
||||||
|
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
|
||||||
|
with:
|
||||||
|
path: /tmp/digests
|
||||||
|
pattern: digests-*
|
||||||
|
merge-multiple: true
|
||||||
|
|
||||||
|
- name: Set up Docker Buildx
|
||||||
|
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
|
||||||
|
|
||||||
|
- name: Login to GitHub Container Registry
|
||||||
|
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
|
||||||
|
with:
|
||||||
|
registry: ghcr.io
|
||||||
|
username: ${{ github.actor }}
|
||||||
|
password: ${{ secrets.GITHUB_TOKEN }}
|
||||||
|
|
||||||
|
- name: Create and push manifest list
|
||||||
|
working-directory: /tmp/digests
|
||||||
|
env:
|
||||||
|
IMAGE: ${{ inputs.image }}
|
||||||
|
TAGS: ${{ inputs.tags }}
|
||||||
|
run: |
|
||||||
|
set -euo pipefail
|
||||||
|
args=()
|
||||||
|
while IFS= read -r tag; do
|
||||||
|
if [ -n "$tag" ]; then
|
||||||
|
args+=("-t" "$tag")
|
||||||
|
fi
|
||||||
|
done <<< "$TAGS"
|
||||||
|
refs=()
|
||||||
|
for digest in *; do
|
||||||
|
refs+=("$IMAGE@sha256:$digest")
|
||||||
|
done
|
||||||
|
docker buildx imagetools create "${args[@]}" "${refs[@]}"
|
||||||
|
|
||||||
|
- name: Inspect image
|
||||||
|
env:
|
||||||
|
IMAGE: ${{ inputs.image }}
|
||||||
|
TAGS: ${{ inputs.tags }}
|
||||||
|
run: |
|
||||||
|
set -euo pipefail
|
||||||
|
first=$(printf '%s\n' "$TAGS" | head -n1)
|
||||||
|
docker buildx imagetools inspect "$first"
|
||||||
|
echo "### Published ${{ inputs.image }}" >> "$GITHUB_STEP_SUMMARY"
|
||||||
|
printf '%s\n' "$TAGS" | sed 's/^/- /' >> "$GITHUB_STEP_SUMMARY"
|
||||||
70
.github/workflows/build-gateway.yaml
vendored
Normal file
70
.github/workflows/build-gateway.yaml
vendored
Normal file
@@ -0,0 +1,70 @@
|
|||||||
|
# Development channel for the example gateway image: a merge that touches the
|
||||||
|
# gateway publishes ghcr.io/<owner>/<repo>/dsh-agents-as-a-service/gateway:main.
|
||||||
|
name: Build gateway image
|
||||||
|
|
||||||
|
on:
|
||||||
|
workflow_dispatch:
|
||||||
|
push:
|
||||||
|
branches: [main]
|
||||||
|
paths:
|
||||||
|
- examples/dsh-agents-as-a-service/gateway/**
|
||||||
|
- versions.env
|
||||||
|
- .github/workflows/build-gateway.yaml
|
||||||
|
- .github/workflows/_build-image.yaml
|
||||||
|
|
||||||
|
permissions:
|
||||||
|
contents: read
|
||||||
|
packages: write
|
||||||
|
id-token: write
|
||||||
|
attestations: write
|
||||||
|
|
||||||
|
concurrency:
|
||||||
|
group: build-gateway-${{ github.ref }}
|
||||||
|
cancel-in-progress: true
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
meta:
|
||||||
|
name: Read versions
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
timeout-minutes: 5
|
||||||
|
outputs:
|
||||||
|
identity: ${{ steps.meta.outputs.identity }}
|
||||||
|
gateway_image: ${{ steps.meta.outputs.gateway_image }}
|
||||||
|
steps:
|
||||||
|
- name: Checkout
|
||||||
|
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
||||||
|
with:
|
||||||
|
persist-credentials: false
|
||||||
|
- name: Read versions.env
|
||||||
|
id: meta
|
||||||
|
shell: bash
|
||||||
|
run: |
|
||||||
|
set -euo pipefail
|
||||||
|
set -a
|
||||||
|
# shellcheck disable=SC1091
|
||||||
|
source versions.env
|
||||||
|
set +a
|
||||||
|
|
||||||
|
identity="$DSH_VERSION"
|
||||||
|
[ -z "$RELEASE_REVISION" ] || identity="$DSH_VERSION-$RELEASE_REVISION"
|
||||||
|
{
|
||||||
|
echo "identity=$identity"
|
||||||
|
echo "gateway_image=$GATEWAY_IMAGE"
|
||||||
|
} >> "$GITHUB_OUTPUT"
|
||||||
|
|
||||||
|
build:
|
||||||
|
name: Build and push gateway image
|
||||||
|
needs: meta
|
||||||
|
uses: ./.github/workflows/_build-image.yaml
|
||||||
|
permissions:
|
||||||
|
contents: read
|
||||||
|
packages: write
|
||||||
|
id-token: write
|
||||||
|
attestations: write
|
||||||
|
with:
|
||||||
|
image: ${{ needs.meta.outputs.gateway_image }}
|
||||||
|
version: ${{ needs.meta.outputs.identity }}
|
||||||
|
context: examples/dsh-agents-as-a-service/gateway
|
||||||
|
dockerfile: Dockerfile
|
||||||
|
tags: |
|
||||||
|
${{ needs.meta.outputs.gateway_image }}:main
|
||||||
84
.github/workflows/build-main.yaml
vendored
Normal file
84
.github/workflows/build-main.yaml
vendored
Normal file
@@ -0,0 +1,84 @@
|
|||||||
|
# Development channel for the main DSH Web image: every merge to main that
|
||||||
|
# touches the image definition publishes ghcr.io/<owner>/<repo>:main.
|
||||||
|
# Releases are published by release.yaml with immutable version tags.
|
||||||
|
name: Build main image
|
||||||
|
|
||||||
|
on:
|
||||||
|
workflow_dispatch:
|
||||||
|
push:
|
||||||
|
branches: [main]
|
||||||
|
paths:
|
||||||
|
- Dockerfile
|
||||||
|
- versions.env
|
||||||
|
- .github/workflows/build-main.yaml
|
||||||
|
- .github/workflows/_build-image.yaml
|
||||||
|
|
||||||
|
permissions:
|
||||||
|
contents: read
|
||||||
|
packages: write
|
||||||
|
id-token: write
|
||||||
|
attestations: write
|
||||||
|
|
||||||
|
concurrency:
|
||||||
|
group: build-main-${{ github.ref }}
|
||||||
|
cancel-in-progress: true
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
meta:
|
||||||
|
name: Read versions
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
timeout-minutes: 5
|
||||||
|
outputs:
|
||||||
|
identity: ${{ steps.meta.outputs.identity }}
|
||||||
|
dsh_version: ${{ steps.meta.outputs.dsh_version }}
|
||||||
|
pnpm_version: ${{ steps.meta.outputs.pnpm_version }}
|
||||||
|
release_revision: ${{ steps.meta.outputs.release_revision }}
|
||||||
|
ghcr_image: ${{ steps.meta.outputs.ghcr_image }}
|
||||||
|
gateway_image: ${{ steps.meta.outputs.gateway_image }}
|
||||||
|
steps:
|
||||||
|
- name: Checkout
|
||||||
|
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
||||||
|
with:
|
||||||
|
persist-credentials: false
|
||||||
|
|
||||||
|
- name: Read versions.env
|
||||||
|
id: meta
|
||||||
|
shell: bash
|
||||||
|
run: |
|
||||||
|
set -euo pipefail
|
||||||
|
set -a
|
||||||
|
# shellcheck disable=SC1091
|
||||||
|
source versions.env
|
||||||
|
set +a
|
||||||
|
identity="$DSH_VERSION"
|
||||||
|
[ -z "$RELEASE_REVISION" ] || identity="$DSH_VERSION-$RELEASE_REVISION"
|
||||||
|
{
|
||||||
|
echo "identity=$identity"
|
||||||
|
echo "dsh_version=$DSH_VERSION"
|
||||||
|
echo "pnpm_version=$PNPM_VERSION"
|
||||||
|
echo "release_revision=$RELEASE_REVISION"
|
||||||
|
echo "ghcr_image=$GHCR_IMAGE"
|
||||||
|
echo "gateway_image=$GATEWAY_IMAGE"
|
||||||
|
} >> "$GITHUB_OUTPUT"
|
||||||
|
|
||||||
|
build:
|
||||||
|
name: Build and push main image
|
||||||
|
needs: meta
|
||||||
|
uses: ./.github/workflows/_build-image.yaml
|
||||||
|
permissions:
|
||||||
|
contents: read
|
||||||
|
packages: write
|
||||||
|
id-token: write
|
||||||
|
attestations: write
|
||||||
|
with:
|
||||||
|
image: ${{ needs.meta.outputs.ghcr_image }}
|
||||||
|
version: ${{ needs.meta.outputs.identity }}
|
||||||
|
context: .
|
||||||
|
dockerfile: Dockerfile
|
||||||
|
build-args: |
|
||||||
|
DSH_VERSION=${{ needs.meta.outputs.dsh_version }}
|
||||||
|
RELEASE_REVISION=${{ needs.meta.outputs.release_revision }}
|
||||||
|
PNPM_VERSION=${{ needs.meta.outputs.pnpm_version }}
|
||||||
|
VCS_REF=${{ github.sha }}
|
||||||
|
tags: |
|
||||||
|
${{ needs.meta.outputs.ghcr_image }}:main
|
||||||
257
.github/workflows/ci.yaml
vendored
Normal file
257
.github/workflows/ci.yaml
vendored
Normal file
@@ -0,0 +1,257 @@
|
|||||||
|
# Continuous integration: lint, unit tests, manifest dry-run, gateway image
|
||||||
|
# smoke test and dependency review. Job names are the required branch-protection
|
||||||
|
# checks documented in docs/gitflow.md.
|
||||||
|
name: CI
|
||||||
|
|
||||||
|
on:
|
||||||
|
pull_request:
|
||||||
|
push:
|
||||||
|
branches: [main]
|
||||||
|
workflow_dispatch:
|
||||||
|
|
||||||
|
permissions:
|
||||||
|
contents: read
|
||||||
|
|
||||||
|
concurrency:
|
||||||
|
group: ci-${{ github.workflow }}-${{ github.ref }}
|
||||||
|
cancel-in-progress: true
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
lint:
|
||||||
|
name: lint
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
timeout-minutes: 20
|
||||||
|
steps:
|
||||||
|
- name: Checkout
|
||||||
|
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
||||||
|
with:
|
||||||
|
persist-credentials: false
|
||||||
|
- name: Set up Python
|
||||||
|
uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
|
||||||
|
with:
|
||||||
|
python-version: "3.12"
|
||||||
|
- name: Install Python linters
|
||||||
|
run: |
|
||||||
|
set -euo pipefail
|
||||||
|
python -m pip install --upgrade pip
|
||||||
|
pip install "yamllint==1.38.0" "pyyaml==6.0.3"
|
||||||
|
- name: Install actionlint
|
||||||
|
env:
|
||||||
|
ACTIONLINT_VERSION: v1.7.12
|
||||||
|
run: |
|
||||||
|
set -euo pipefail
|
||||||
|
version="${ACTIONLINT_VERSION#v}"
|
||||||
|
curl -fsSL -o /tmp/actionlint.tar.gz "https://github.com/rhysd/actionlint/releases/download/$ACTIONLINT_VERSION/actionlint_${version}_linux_amd64.tar.gz"
|
||||||
|
tar -xzf /tmp/actionlint.tar.gz -C /tmp actionlint
|
||||||
|
install -m 0755 /tmp/actionlint /usr/local/bin/actionlint
|
||||||
|
actionlint -version
|
||||||
|
- name: Install hadolint
|
||||||
|
env:
|
||||||
|
HADOLINT_VERSION: v2.12.0
|
||||||
|
run: |
|
||||||
|
set -euo pipefail
|
||||||
|
curl -fsSL -o /tmp/hadolint "https://github.com/hadolint/hadolint/releases/download/$HADOLINT_VERSION/hadolint-Linux-x86_64"
|
||||||
|
install -m 0755 /tmp/hadolint /usr/local/bin/hadolint
|
||||||
|
hadolint --version
|
||||||
|
- name: Repository checks
|
||||||
|
run: |
|
||||||
|
set -euo pipefail
|
||||||
|
./scripts/check-versions.sh
|
||||||
|
./scripts/check-examples.sh
|
||||||
|
python scripts/check-manifests.py
|
||||||
|
- name: Shellcheck
|
||||||
|
run: shellcheck -x build.sh scripts/*.sh examples/*/scripts/*.sh
|
||||||
|
- name: Actionlint
|
||||||
|
run: actionlint -color
|
||||||
|
- name: Yamllint
|
||||||
|
run: yamllint .
|
||||||
|
- name: Hadolint
|
||||||
|
run: |
|
||||||
|
set -euo pipefail
|
||||||
|
hadolint Dockerfile
|
||||||
|
hadolint examples/dsh-agents-as-a-service/gateway/Dockerfile
|
||||||
|
- name: Documentation links
|
||||||
|
uses: lycheeverse/lychee-action@e7477775783ea5526144ba13e8db5eec57747ce8 # v2
|
||||||
|
with:
|
||||||
|
args: >-
|
||||||
|
--offline --no-progress --verbose
|
||||||
|
./README.md ./CONTRIBUTING.md ./SECURITY.md ./AGENTS.md ./docs ./examples
|
||||||
|
|
||||||
|
python:
|
||||||
|
name: python
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
timeout-minutes: 20
|
||||||
|
defaults:
|
||||||
|
run:
|
||||||
|
working-directory: examples/dsh-agents-as-a-service/gateway
|
||||||
|
steps:
|
||||||
|
- name: Checkout
|
||||||
|
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
||||||
|
with:
|
||||||
|
persist-credentials: false
|
||||||
|
- name: Set up Python
|
||||||
|
uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
|
||||||
|
with:
|
||||||
|
python-version: "3.12"
|
||||||
|
cache: pip
|
||||||
|
cache-dependency-path: |
|
||||||
|
examples/dsh-agents-as-a-service/gateway/requirements.txt
|
||||||
|
examples/dsh-agents-as-a-service/gateway/requirements-dev.txt
|
||||||
|
- name: Install dependencies
|
||||||
|
run: |
|
||||||
|
set -euo pipefail
|
||||||
|
python -m pip install --upgrade pip
|
||||||
|
pip install -r requirements.txt -r requirements-dev.txt
|
||||||
|
- name: Lint
|
||||||
|
run: |
|
||||||
|
set -euo pipefail
|
||||||
|
python -m ruff check .
|
||||||
|
python -m ruff format --check .
|
||||||
|
- name: Unit tests with coverage
|
||||||
|
run: >-
|
||||||
|
python -m pytest -q
|
||||||
|
--cov=. --cov-report=xml:coverage.xml --cov-report=term
|
||||||
|
--junitxml=junit.xml
|
||||||
|
- name: Coverage summary
|
||||||
|
run: |
|
||||||
|
set -euo pipefail
|
||||||
|
{
|
||||||
|
echo "### Gateway test coverage"
|
||||||
|
python -m coverage report
|
||||||
|
} >> "$GITHUB_STEP_SUMMARY"
|
||||||
|
- name: Upload test artefacts
|
||||||
|
if: always()
|
||||||
|
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
|
||||||
|
with:
|
||||||
|
name: gateway-test-results
|
||||||
|
path: |
|
||||||
|
examples/dsh-agents-as-a-service/gateway/junit.xml
|
||||||
|
examples/dsh-agents-as-a-service/gateway/coverage.xml
|
||||||
|
if-no-files-found: warn
|
||||||
|
retention-days: 7
|
||||||
|
|
||||||
|
manifests:
|
||||||
|
name: manifests
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
timeout-minutes: 25
|
||||||
|
steps:
|
||||||
|
- name: Checkout
|
||||||
|
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
||||||
|
with:
|
||||||
|
persist-credentials: false
|
||||||
|
- name: Create kind cluster
|
||||||
|
uses: helm/kind-action@1544676c07bb3570fcb70166beaa07c4528fd4aa # v1.15.1
|
||||||
|
with:
|
||||||
|
version: v0.33.0
|
||||||
|
cluster_name: dsh-ci
|
||||||
|
- name: Read versions.env
|
||||||
|
run: |
|
||||||
|
set -euo pipefail
|
||||||
|
while IFS= read -r line; do
|
||||||
|
case "$line" in ''|'#'*) continue ;; esac
|
||||||
|
echo "$line" >> "$GITHUB_ENV"
|
||||||
|
done < versions.env
|
||||||
|
- name: Install agent-sandbox and Gateway API CRDs
|
||||||
|
run: |
|
||||||
|
set -euo pipefail
|
||||||
|
kubectl apply -f "https://github.com/kubernetes-sigs/agent-sandbox/releases/download/$AGENT_SANDBOX_VERSION/sandbox-with-extensions.yaml"
|
||||||
|
kubectl wait --for=condition=Established crd/sandboxtemplates.extensions.agents.x-k8s.io --timeout=180s
|
||||||
|
kubectl wait --for=condition=Established crd/sandboxwarmpools.extensions.agents.x-k8s.io --timeout=180s
|
||||||
|
kubectl wait --for=condition=Established crd/sandboxclaims.extensions.agents.x-k8s.io --timeout=180s
|
||||||
|
kubectl wait --for=condition=Established crd/sandboxes.agents.x-k8s.io --timeout=180s
|
||||||
|
kubectl apply -f "https://github.com/kubernetes-sigs/gateway-api/releases/download/$GATEWAY_API_VERSION/standard-install.yaml"
|
||||||
|
kubectl wait --for=condition=Established crd/gateways.gateway.networking.k8s.io --timeout=180s
|
||||||
|
kubectl wait --for=condition=Established crd/httproutes.gateway.networking.k8s.io --timeout=180s
|
||||||
|
- name: Set up Python
|
||||||
|
uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
|
||||||
|
with:
|
||||||
|
python-version: "3.12"
|
||||||
|
- name: Install check dependencies
|
||||||
|
run: pip install "pyyaml==6.0.3"
|
||||||
|
- name: Validate manifest shape
|
||||||
|
run: python scripts/check-manifests.py
|
||||||
|
- name: Validate manifests (server dry-run)
|
||||||
|
run: |
|
||||||
|
set -euo pipefail
|
||||||
|
kubectl create namespace dsh-demo
|
||||||
|
shopt -s globstar nullglob
|
||||||
|
count=0
|
||||||
|
for file in examples/*/manifests/**/*.yaml; do
|
||||||
|
case "$file" in *.yaml.example) continue ;; esac
|
||||||
|
echo "== $file =="
|
||||||
|
kubectl apply --dry-run=server -f "$file"
|
||||||
|
count=$((count + 1))
|
||||||
|
done
|
||||||
|
if [ "$count" -eq 0 ]; then
|
||||||
|
echo "no manifests were validated" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo "validated $count manifests"
|
||||||
|
|
||||||
|
gateway-image:
|
||||||
|
name: gateway-image
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
timeout-minutes: 20
|
||||||
|
steps:
|
||||||
|
- name: Checkout
|
||||||
|
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
||||||
|
with:
|
||||||
|
persist-credentials: false
|
||||||
|
- name: Build gateway image
|
||||||
|
run: docker build -t dsh-aaas-gateway:ci examples/dsh-agents-as-a-service/gateway/
|
||||||
|
- name: Smoke-test /healthz without a cluster
|
||||||
|
run: |
|
||||||
|
set -euo pipefail
|
||||||
|
cat > /tmp/kubeconfig <<'EOF'
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Config
|
||||||
|
clusters:
|
||||||
|
- name: dummy
|
||||||
|
cluster:
|
||||||
|
server: https://127.0.0.1:6443
|
||||||
|
insecure-skip-tls-verify: true
|
||||||
|
contexts:
|
||||||
|
- name: dummy
|
||||||
|
context:
|
||||||
|
cluster: dummy
|
||||||
|
user: dummy
|
||||||
|
users:
|
||||||
|
- name: dummy
|
||||||
|
user: {}
|
||||||
|
current-context: dummy
|
||||||
|
EOF
|
||||||
|
docker run -d --name gateway-smoke \
|
||||||
|
-p 18080:8080 \
|
||||||
|
-e NAMESPACE=dsh-demo \
|
||||||
|
-e POOL=dsh-pool \
|
||||||
|
-e KUBECONFIG=/tmp/kubeconfig \
|
||||||
|
-v /tmp/kubeconfig:/tmp/kubeconfig:ro \
|
||||||
|
dsh-aaas-gateway:ci
|
||||||
|
trap 'docker logs gateway-smoke || true; docker rm -f gateway-smoke || true' EXIT
|
||||||
|
code=""
|
||||||
|
for _ in $(seq 1 30); do
|
||||||
|
code=$(curl -s -o /dev/null -w '%{http_code}' http://localhost:18080/healthz || true)
|
||||||
|
[ "$code" = "200" ] && break
|
||||||
|
sleep 1
|
||||||
|
done
|
||||||
|
if [ "$code" != "200" ]; then
|
||||||
|
echo "gateway /healthz returned '$code' instead of 200" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo "gateway image healthy: GET /healthz -> 200"
|
||||||
|
|
||||||
|
dependency-review:
|
||||||
|
name: dependency-review
|
||||||
|
if: github.event_name == 'pull_request'
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
timeout-minutes: 10
|
||||||
|
permissions:
|
||||||
|
contents: read
|
||||||
|
pull-requests: write
|
||||||
|
steps:
|
||||||
|
- name: Checkout
|
||||||
|
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
||||||
|
with:
|
||||||
|
persist-credentials: false
|
||||||
|
- name: Dependency review
|
||||||
|
uses: actions/dependency-review-action@a1d282b36b6f3519aa1f3fc636f609c47dddb294 # v4
|
||||||
6
.github/workflows/codeql.yml
vendored
6
.github/workflows/codeql.yml
vendored
@@ -57,7 +57,7 @@ jobs:
|
|||||||
# your codebase is analyzed, see https://docs.github.com/en/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/codeql-code-scanning-for-compiled-languages
|
# your codebase is analyzed, see https://docs.github.com/en/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/codeql-code-scanning-for-compiled-languages
|
||||||
steps:
|
steps:
|
||||||
- name: Checkout repository
|
- name: Checkout repository
|
||||||
uses: actions/checkout@v7
|
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
||||||
|
|
||||||
# Add any setup steps before running the `github/codeql-action/init` action.
|
# Add any setup steps before running the `github/codeql-action/init` action.
|
||||||
# This includes steps like installing compilers or runtimes (`actions/setup-node`
|
# This includes steps like installing compilers or runtimes (`actions/setup-node`
|
||||||
@@ -67,7 +67,7 @@ jobs:
|
|||||||
|
|
||||||
# Initializes the CodeQL tools for scanning.
|
# Initializes the CodeQL tools for scanning.
|
||||||
- name: Initialize CodeQL
|
- name: Initialize CodeQL
|
||||||
uses: github/codeql-action/init@v4
|
uses: github/codeql-action/init@9b3254085494b55a0f7c4c2b3d2d52cc712ce370 # v4
|
||||||
with:
|
with:
|
||||||
languages: ${{ matrix.language }}
|
languages: ${{ matrix.language }}
|
||||||
build-mode: ${{ matrix.build-mode }}
|
build-mode: ${{ matrix.build-mode }}
|
||||||
@@ -96,6 +96,6 @@ jobs:
|
|||||||
exit 1
|
exit 1
|
||||||
|
|
||||||
- name: Perform CodeQL Analysis
|
- name: Perform CodeQL Analysis
|
||||||
uses: github/codeql-action/analyze@v4
|
uses: github/codeql-action/analyze@9b3254085494b55a0f7c4c2b3d2d52cc712ce370 # v4
|
||||||
with:
|
with:
|
||||||
category: "/language:${{matrix.language}}"
|
category: "/language:${{matrix.language}}"
|
||||||
|
|||||||
78
.github/workflows/e2e-kind.yaml
vendored
Normal file
78
.github/workflows/e2e-kind.yaml
vendored
Normal file
@@ -0,0 +1,78 @@
|
|||||||
|
# Weekly end-to-end run of the agents-as-a-service example against a real kind
|
||||||
|
# cluster with agent-sandbox installed. Scheduled (not PR-blocking) so upstream
|
||||||
|
# controller churn does not block merges; also available on demand.
|
||||||
|
name: E2E (kind)
|
||||||
|
|
||||||
|
on:
|
||||||
|
schedule:
|
||||||
|
- cron: "0 6 * * 1"
|
||||||
|
workflow_dispatch:
|
||||||
|
|
||||||
|
permissions:
|
||||||
|
contents: read
|
||||||
|
|
||||||
|
concurrency:
|
||||||
|
group: e2e-kind
|
||||||
|
cancel-in-progress: false
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
e2e:
|
||||||
|
name: Example walkthrough on kind
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
timeout-minutes: 45
|
||||||
|
steps:
|
||||||
|
- name: Checkout
|
||||||
|
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
||||||
|
with:
|
||||||
|
persist-credentials: false
|
||||||
|
- name: Create kind cluster
|
||||||
|
uses: helm/kind-action@1544676c07bb3570fcb70166beaa07c4528fd4aa # v1.15.1
|
||||||
|
with:
|
||||||
|
version: v0.33.0
|
||||||
|
cluster_name: dsh-e2e
|
||||||
|
- name: Read versions.env
|
||||||
|
run: |
|
||||||
|
set -euo pipefail
|
||||||
|
while IFS= read -r line; do
|
||||||
|
case "$line" in ''|'#'*) continue ;; esac
|
||||||
|
echo "$line" >> "$GITHUB_ENV"
|
||||||
|
done < versions.env
|
||||||
|
- name: Install agent-sandbox
|
||||||
|
run: |
|
||||||
|
set -euo pipefail
|
||||||
|
kubectl apply -f "https://github.com/kubernetes-sigs/agent-sandbox/releases/download/$AGENT_SANDBOX_VERSION/sandbox-with-extensions.yaml"
|
||||||
|
kubectl -n agent-sandbox-system rollout status deployment --timeout=180s
|
||||||
|
kubectl wait --for=condition=Established crd/sandboxtemplates.extensions.agents.x-k8s.io --timeout=180s
|
||||||
|
kubectl wait --for=condition=Established crd/sandboxwarmpools.extensions.agents.x-k8s.io --timeout=180s
|
||||||
|
kubectl wait --for=condition=Established crd/sandboxclaims.extensions.agents.x-k8s.io --timeout=180s
|
||||||
|
kubectl wait --for=condition=Established crd/sandboxes.agents.x-k8s.io --timeout=180s
|
||||||
|
- name: Set up Python (WebSocket checks)
|
||||||
|
uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
|
||||||
|
with:
|
||||||
|
python-version: "3.12"
|
||||||
|
- name: Install WebSocket client
|
||||||
|
run: pip install "websockets==17.3"
|
||||||
|
- name: Run the example walkthrough
|
||||||
|
env:
|
||||||
|
KIND_CLUSTER: dsh-e2e
|
||||||
|
run: examples/dsh-agents-as-a-service/scripts/e2e-kind.sh
|
||||||
|
- name: Collect cluster state
|
||||||
|
if: always()
|
||||||
|
run: |
|
||||||
|
set -euo pipefail
|
||||||
|
{
|
||||||
|
echo "== resources =="
|
||||||
|
kubectl get all,pvc -n dsh-demo -o wide 2>&1 || true
|
||||||
|
echo "== events =="
|
||||||
|
kubectl get events -n dsh-demo --sort-by=.lastTimestamp 2>&1 || true
|
||||||
|
echo "== gateway logs =="
|
||||||
|
kubectl -n dsh-demo logs deploy/dsh-gateway --tail=200 2>&1 || true
|
||||||
|
} > /tmp/e2e-cluster.txt
|
||||||
|
- name: Upload cluster state
|
||||||
|
if: always()
|
||||||
|
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
|
||||||
|
with:
|
||||||
|
name: e2e-cluster-state
|
||||||
|
path: /tmp/e2e-cluster.txt
|
||||||
|
if-no-files-found: warn
|
||||||
|
retention-days: 7
|
||||||
277
.github/workflows/multi-build-gateway.yaml
vendored
277
.github/workflows/multi-build-gateway.yaml
vendored
@@ -1,277 +0,0 @@
|
|||||||
# This workflow builds the multi-arch gateway image (Starlette ASGI, served by
|
|
||||||
# uvicorn on :8080) and pushes it to GHCR as
|
|
||||||
# ghcr.io/<owner>/<repo>:gateway. The DSH Web image is built by multi-build.yaml.
|
|
||||||
# It uses Docker Buildx with native AMD64 and ARM64 runners, without QEMU.
|
|
||||||
name: Gateway Build multi arch Docker Image with separate Github Runners
|
|
||||||
|
|
||||||
on:
|
|
||||||
workflow_dispatch:
|
|
||||||
push:
|
|
||||||
branches:
|
|
||||||
- main
|
|
||||||
paths:
|
|
||||||
- 'examples/dsh-agents-as-a-service/gateway/**'
|
|
||||||
- '.github/workflows/multi-build-gateway.yaml'
|
|
||||||
env:
|
|
||||||
# ghcr.io/<owner>/<repo>; this workflow always publishes the `:gateway` tag.
|
|
||||||
GHCR_IMAGE: ghcr.io/${{ github.repository }}
|
|
||||||
permissions:
|
|
||||||
# Global permissions for the workflow, which can be overridden at the job level
|
|
||||||
contents: read
|
|
||||||
|
|
||||||
concurrency:
|
|
||||||
# This concurrency group ensures that only one job in the group runs at a time.
|
|
||||||
# If a new job is triggered, the previous one will be canceled.
|
|
||||||
group: ${{ github.workflow }}-${{ github.ref }}
|
|
||||||
cancel-in-progress: true
|
|
||||||
|
|
||||||
jobs:
|
|
||||||
# The build job builds the Docker image for each platform specified in the matrix.
|
|
||||||
build:
|
|
||||||
strategy:
|
|
||||||
fail-fast: false
|
|
||||||
matrix:
|
|
||||||
platform:
|
|
||||||
- linux/amd64
|
|
||||||
- linux/arm64
|
|
||||||
# The matrix includes two platforms: linux/amd64 and linux/arm64.
|
|
||||||
# The build job will run for each platform in the matrix.
|
|
||||||
|
|
||||||
permissions:
|
|
||||||
# Permissions for the build job, which can be overridden at the step level
|
|
||||||
# The permissions are set to allow the job to write to the GitHub Container Registry (GHCR) and read from the repository.
|
|
||||||
attestations: write
|
|
||||||
actions: read
|
|
||||||
checks: write
|
|
||||||
contents: write
|
|
||||||
deployments: none
|
|
||||||
id-token: write
|
|
||||||
issues: read
|
|
||||||
discussions: read
|
|
||||||
packages: write
|
|
||||||
pages: none
|
|
||||||
pull-requests: read
|
|
||||||
repository-projects: read
|
|
||||||
security-events: read
|
|
||||||
statuses: read
|
|
||||||
|
|
||||||
runs-on: ${{ matrix.platform == 'linux/amd64' && 'ubuntu-latest' || matrix.platform == 'linux/arm64' && 'ubuntu-24.04-arm' }}
|
|
||||||
# The job runs on different runners based on the platform.
|
|
||||||
# For linux/amd64, it runs on the latest Ubuntu runner.
|
|
||||||
# For linux/arm64, it runs on an Ubuntu 24.04 ARM runner.
|
|
||||||
# The runner is selected based on the platform specified in the matrix.
|
|
||||||
|
|
||||||
name: Build Docker image for ${{ matrix.platform }}
|
|
||||||
|
|
||||||
steps:
|
|
||||||
-
|
|
||||||
name: Prepare environment for current platform
|
|
||||||
# This step sets up the environment for the current platform being built.
|
|
||||||
# It replaces the '/' character in the platform name with '-' and sets it as an environment variable.
|
|
||||||
# This is useful for naming artifacts and other resources that cannot contain '/'.
|
|
||||||
# The environment variable PLATFORMS_PAIR will be used later in the workflow.
|
|
||||||
id: prepare
|
|
||||||
run: |
|
|
||||||
platform=${{ matrix.platform }}
|
|
||||||
echo "PLATFORM_PAIR=${platform//\//-}" >> $GITHUB_ENV
|
|
||||||
|
|
||||||
- name: Checkout
|
|
||||||
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
||||||
# This step checks out the code from the repository.
|
|
||||||
# It uses the actions/checkout action to clone the repository into the runner's workspace.
|
|
||||||
|
|
||||||
- name: Docker meta default
|
|
||||||
# This step generates metadata for the Docker image.
|
|
||||||
# It uses the docker/metadata-action to create metadata based on the repository information.
|
|
||||||
# The metadata includes information such as the image name, tags, and labels.
|
|
||||||
# The metadata will be used later in the workflow to build and push the Docker image.
|
|
||||||
id: meta
|
|
||||||
uses: docker/metadata-action@dc802804100637a589fabce1cb79ff13a1411302 # v6.2.0
|
|
||||||
with:
|
|
||||||
images: ${{ env.GHCR_IMAGE }}
|
|
||||||
tags: |
|
|
||||||
type=raw,value=gateway
|
|
||||||
|
|
||||||
|
|
||||||
- name: Set up Docker Context for Buildx
|
|
||||||
# This step sets up a Docker context for Buildx.
|
|
||||||
# It creates a new context named "builders" that will be used for building the Docker image.
|
|
||||||
# The context allows Buildx to use the Docker daemon for building images.
|
|
||||||
id: buildx-context
|
|
||||||
run: |
|
|
||||||
docker context create builders
|
|
||||||
|
|
||||||
- name: Set up Docker Buildx
|
|
||||||
# This step sets up Docker Buildx, which is a Docker CLI plugin for extended build capabilities with BuildKit.
|
|
||||||
# It uses the docker/setup-buildx-action to configure Buildx with the specified context and platforms.
|
|
||||||
# The platforms are specified in the matrix and will be used for building the Docker image.
|
|
||||||
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
|
|
||||||
with:
|
|
||||||
endpoint: builders
|
|
||||||
platforms: ${{ matrix.platform }}
|
|
||||||
|
|
||||||
- name: Login to GitHub Container Registry
|
|
||||||
# This step logs in to the GitHub Container Registry (GHCR) using the docker/login-action.
|
|
||||||
# It uses the GitHub actor's username and the GITHUB_TOKEN secret for authentication.
|
|
||||||
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
|
|
||||||
with:
|
|
||||||
registry: ghcr.io
|
|
||||||
username: ${{ github.actor }}
|
|
||||||
password: ${{ secrets.GITHUB_TOKEN }}
|
|
||||||
|
|
||||||
|
|
||||||
- name: Build and push by digest
|
|
||||||
# This step builds and pushes the Docker image using Buildx.
|
|
||||||
# It uses the docker/build-push-action to build the image with the specified context and platforms.
|
|
||||||
# The image is built with the labels and annotations generated in the previous steps.
|
|
||||||
# The outputs are configured to push the image by digest, which allows for better caching and versioning.
|
|
||||||
# The cache-from and cache-to options are used to enable caching for the build process.
|
|
||||||
# The cache is stored in GitHub Actions cache and is scoped to the repository, branch, and platform.
|
|
||||||
id: build
|
|
||||||
uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7.4.0
|
|
||||||
env:
|
|
||||||
DOCKER_BUILDKIT: 1
|
|
||||||
with:
|
|
||||||
context: ./examples/dsh-agents-as-a-service/gateway/
|
|
||||||
platforms: ${{ matrix.platform }}
|
|
||||||
labels: ${{ steps.meta.outputs.labels }}
|
|
||||||
annotations: ${{ steps.meta.outputs.annotations }}
|
|
||||||
outputs: type=image,name=${{ env.GHCR_IMAGE }},push-by-digest=true,name-canonical=true,push=true,oci-mediatypes=true
|
|
||||||
cache-from: type=gha,scope=${{ github.repository }}-${{ github.ref_name }}-${{ matrix.platform }}
|
|
||||||
cache-to: type=gha,scope=${{ github.repository }}-${{ github.ref_name }}-${{ matrix.platform }}
|
|
||||||
|
|
||||||
|
|
||||||
- name: Export digest
|
|
||||||
# This step exports the digest of the built image to a file.
|
|
||||||
# It creates a directory in /tmp/digests and saves the digest of the image to a file.
|
|
||||||
# The digest is obtained from the output of the build step.
|
|
||||||
# The digest is used to uniquely identify the built image and can be used for further processing or verification.
|
|
||||||
run: |
|
|
||||||
mkdir -p /tmp/digests
|
|
||||||
digest="${{ steps.build.outputs.digest }}"
|
|
||||||
touch "/tmp/digests/${digest#sha256:}"
|
|
||||||
|
|
||||||
- name: Upload digest
|
|
||||||
# This step uploads the digest file to the GitHub Actions artifact storage.
|
|
||||||
# It uses the actions/upload-artifact action to upload the file created in the previous step.
|
|
||||||
# The artifact is named digests-${{ env.PLATFORM_PAIR }}, where PLATFORM_PAIR is the platform name with '/' replaced by '-'.
|
|
||||||
# The artifact is retained for 1 day, and if no files are found, it will throw an error.
|
|
||||||
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
|
|
||||||
with:
|
|
||||||
name: digests-${{ env.PLATFORM_PAIR }}
|
|
||||||
path: /tmp/digests/*
|
|
||||||
if-no-files-found: error
|
|
||||||
retention-days: 1
|
|
||||||
|
|
||||||
|
|
||||||
merge:
|
|
||||||
# This job merges the Docker manifests for the different platforms built in the previous job.
|
|
||||||
name: Merge Docker manifests
|
|
||||||
runs-on: ubuntu-latest
|
|
||||||
permissions:
|
|
||||||
attestations: write
|
|
||||||
actions: read
|
|
||||||
checks: read
|
|
||||||
contents: read
|
|
||||||
deployments: none
|
|
||||||
id-token: write
|
|
||||||
issues: read
|
|
||||||
discussions: read
|
|
||||||
packages: write
|
|
||||||
pages: none
|
|
||||||
pull-requests: read
|
|
||||||
repository-projects: read
|
|
||||||
security-events: read
|
|
||||||
statuses: read
|
|
||||||
|
|
||||||
needs:
|
|
||||||
- build
|
|
||||||
# This job depends on the build job to complete before it starts.
|
|
||||||
# It ensures that the Docker images for all platforms are built before merging the manifests.
|
|
||||||
steps:
|
|
||||||
- name: Download digests
|
|
||||||
# This step downloads the digest files uploaded in the build job.
|
|
||||||
# It uses the actions/download-artifact action to download the artifacts with the pattern digests-*.
|
|
||||||
# The downloaded files are merged into the /tmp/digests directory.
|
|
||||||
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
|
|
||||||
with:
|
|
||||||
path: /tmp/digests
|
|
||||||
pattern: digests-*
|
|
||||||
merge-multiple: true
|
|
||||||
|
|
||||||
|
|
||||||
- name: Docker meta
|
|
||||||
# This step generates metadata for the Docker image.
|
|
||||||
# It uses the docker/metadata-action to create metadata based on the repository information.
|
|
||||||
# The metadata includes information such as the image name, tags, and labels.
|
|
||||||
id: meta
|
|
||||||
uses: docker/metadata-action@dc802804100637a589fabce1cb79ff13a1411302 # v6.2.0
|
|
||||||
with:
|
|
||||||
images: ${{ env.GHCR_IMAGE }}
|
|
||||||
annotations: |
|
|
||||||
type=org.opencontainers.image.description,value=${{ github.event.repository.description || 'No description provided' }}
|
|
||||||
tags: |
|
|
||||||
type=raw,value=gateway
|
|
||||||
|
|
||||||
- name: Set up Docker Buildx
|
|
||||||
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
|
|
||||||
# This step sets up Docker Buildx, which is a Docker CLI plugin for extended build capabilities with BuildKit.
|
|
||||||
with:
|
|
||||||
driver-opts: |
|
|
||||||
network=host
|
|
||||||
|
|
||||||
- name: Login to GitHub Container Registry
|
|
||||||
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
|
|
||||||
# This step logs in to the GitHub Container Registry (GHCR) using the docker/login-action.
|
|
||||||
# It uses the GitHub actor's username and the GITHUB_TOKEN secret for authentication.
|
|
||||||
# The login is necessary to push the merged manifest list to GHCR.
|
|
||||||
with:
|
|
||||||
registry: ghcr.io
|
|
||||||
username: ${{ github.actor }}
|
|
||||||
password: ${{ secrets.GITHUB_TOKEN }}
|
|
||||||
|
|
||||||
- name: Get execution timestamp with RFC3339 format
|
|
||||||
# This step gets the current execution timestamp in RFC3339 format.
|
|
||||||
# It uses the date command to get the current UTC time and formats it as a string.
|
|
||||||
# The timestamp is used for annotating the Docker manifest list.
|
|
||||||
id: timestamp
|
|
||||||
run: |
|
|
||||||
echo "timestamp=$(date -u +"%Y-%m-%dT%H:%M:%SZ")" >> $GITHUB_OUTPUT
|
|
||||||
|
|
||||||
- name: Create manifest list and pushs
|
|
||||||
# This step creates a manifest list for the Docker images built for different platforms.
|
|
||||||
# It uses the docker buildx imagetools create command to create the manifest list.
|
|
||||||
# The manifest list is annotated with metadata such as description, creation timestamp, and source URL.
|
|
||||||
# The annotations are obtained from the metadata generated in the previous steps.
|
|
||||||
# The manifest list is pushed to the GitHub Container Registry (GHCR) with the specified tags.
|
|
||||||
working-directory: /tmp/digests
|
|
||||||
id: manifest-annotate
|
|
||||||
continue-on-error: true
|
|
||||||
run: |
|
|
||||||
docker buildx imagetools create \
|
|
||||||
$(jq -cr '.tags | map("-t " + .) | join(" ")' <<< "$DOCKER_METADATA_OUTPUT_JSON") \
|
|
||||||
--annotation='index:org.opencontainers.image.description=${{ github.event.repository.description }}' \
|
|
||||||
--annotation='index:org.opencontainers.image.created=${{ steps.timestamp.outputs.timestamp }}' \
|
|
||||||
--annotation='index:org.opencontainers.image.url=${{ github.event.repository.url }}' \
|
|
||||||
--annotation='index:org.opencontainers.image.source=${{ github.event.repository.url }}' \
|
|
||||||
$(printf '${{ env.GHCR_IMAGE }}@sha256:%s ' *)
|
|
||||||
|
|
||||||
|
|
||||||
- name: Create manifest list and push without annotations
|
|
||||||
# This step creates a manifest list for the Docker images built for different platforms.
|
|
||||||
# It uses the docker buildx imagetools create command to create the manifest list.
|
|
||||||
# The manifest list is created without annotations if the previous step fails.
|
|
||||||
# The manifest list is pushed to the GitHub Container Registry (GHCR) with the specified tags.
|
|
||||||
if: steps.manifest-annotate.outcome == 'failure'
|
|
||||||
working-directory: /tmp/digests
|
|
||||||
run: |
|
|
||||||
docker buildx imagetools create $(jq -cr '.tags | map("-t " + .) | join(" ")' <<< "$DOCKER_METADATA_OUTPUT_JSON") \
|
|
||||||
$(printf '${{ env.GHCR_IMAGE }}@sha256:%s ' *)
|
|
||||||
|
|
||||||
- name: Inspect image
|
|
||||||
# This step inspects the created manifest list to verify its contents.
|
|
||||||
# It uses the docker buildx imagetools inspect command to display information about the manifest list.
|
|
||||||
# The inspection output will show the platforms and tags associated with the manifest list.
|
|
||||||
id: inspect
|
|
||||||
run: |
|
|
||||||
docker buildx imagetools inspect '${{ env.GHCR_IMAGE }}:${{ steps.meta.outputs.version }}'
|
|
||||||
295
.github/workflows/multi-build.yaml
vendored
295
.github/workflows/multi-build.yaml
vendored
@@ -1,295 +0,0 @@
|
|||||||
# This workflow builds a multi-arch Docker image using GitHub Actions and separated Github Runners with native support for ARM64 and AMD64 architectures, without using QEMU emulation.
|
|
||||||
# It uses Docker Buildx to build and push the image to GitHub Container Registry (GHCR).
|
|
||||||
name: Build multi arch Docker Image with separate Github Runners
|
|
||||||
|
|
||||||
on:
|
|
||||||
workflow_dispatch:
|
|
||||||
inputs:
|
|
||||||
dsh_version:
|
|
||||||
description: "DSH git ref to build (tag or branch)"
|
|
||||||
required: true
|
|
||||||
default: dsh-v0.2.1-rc.2
|
|
||||||
type: string
|
|
||||||
pnpm_version:
|
|
||||||
description: "pnpm version used to build DSH"
|
|
||||||
required: true
|
|
||||||
default: "11.28.5"
|
|
||||||
type: string
|
|
||||||
push:
|
|
||||||
branches:
|
|
||||||
- main
|
|
||||||
paths:
|
|
||||||
- 'Dockerfile'
|
|
||||||
- '.github/workflows/multi-build.yaml'
|
|
||||||
env:
|
|
||||||
# The name of the Docker image to be built and pushed to GHCR
|
|
||||||
# The image name is derived from the GitHub repository name and the GitHub Container Registry (GHCR) URL.
|
|
||||||
# The image name will be in the format: ghcr.io/<owner>/<repo>
|
|
||||||
GHCR_IMAGE: ghcr.io/${{ github.repository }}
|
|
||||||
# workflow_dispatch inputs win; pushes to main fall back to the pinned
|
|
||||||
# defaults. Both values flow into the Docker build args and the image tags.
|
|
||||||
DSH_VERSION: ${{ inputs.dsh_version || 'dsh-v0.2.1-rc.2' }}
|
|
||||||
PNPM_VERSION: ${{ inputs.pnpm_version || '11.28.5' }}
|
|
||||||
permissions:
|
|
||||||
# Global permissions for the workflow, which can be overridden at the job level
|
|
||||||
contents: read
|
|
||||||
|
|
||||||
concurrency:
|
|
||||||
# This concurrency group ensures that only one job in the group runs at a time.
|
|
||||||
# If a new job is triggered, the previous one will be canceled.
|
|
||||||
group: ${{ github.workflow }}-${{ github.ref }}
|
|
||||||
cancel-in-progress: true
|
|
||||||
|
|
||||||
jobs:
|
|
||||||
# The build job builds the Docker image for each platform specified in the matrix.
|
|
||||||
build:
|
|
||||||
strategy:
|
|
||||||
fail-fast: false
|
|
||||||
matrix:
|
|
||||||
platform:
|
|
||||||
- linux/amd64
|
|
||||||
- linux/arm64
|
|
||||||
# The matrix includes two platforms: linux/amd64 and linux/arm64.
|
|
||||||
# The build job will run for each platform in the matrix.
|
|
||||||
|
|
||||||
permissions:
|
|
||||||
# Permissions for the build job, which can be overridden at the step level
|
|
||||||
# The permissions are set to allow the job to write to the GitHub Container Registry (GHCR) and read from the repository.
|
|
||||||
attestations: write
|
|
||||||
actions: read
|
|
||||||
checks: write
|
|
||||||
contents: write
|
|
||||||
deployments: none
|
|
||||||
id-token: write
|
|
||||||
issues: read
|
|
||||||
discussions: read
|
|
||||||
packages: write
|
|
||||||
pages: none
|
|
||||||
pull-requests: read
|
|
||||||
repository-projects: read
|
|
||||||
security-events: read
|
|
||||||
statuses: read
|
|
||||||
|
|
||||||
runs-on: ${{ matrix.platform == 'linux/amd64' && 'ubuntu-latest' || matrix.platform == 'linux/arm64' && 'ubuntu-24.04-arm' }}
|
|
||||||
# The job runs on different runners based on the platform.
|
|
||||||
# For linux/amd64, it runs on the latest Ubuntu runner.
|
|
||||||
# For linux/arm64, it runs on an Ubuntu 24.04 ARM runner.
|
|
||||||
# The runner is selected based on the platform specified in the matrix.
|
|
||||||
|
|
||||||
name: Build Docker image for ${{ matrix.platform }}
|
|
||||||
|
|
||||||
steps:
|
|
||||||
-
|
|
||||||
name: Prepare environment for current platform
|
|
||||||
# This step sets up the environment for the current platform being built.
|
|
||||||
# It replaces the '/' character in the platform name with '-' and sets it as an environment variable.
|
|
||||||
# This is useful for naming artifacts and other resources that cannot contain '/'.
|
|
||||||
# The environment variable PLATFORMS_PAIR will be used later in the workflow.
|
|
||||||
id: prepare
|
|
||||||
run: |
|
|
||||||
platform=${{ matrix.platform }}
|
|
||||||
echo "PLATFORM_PAIR=${platform//\//-}" >> $GITHUB_ENV
|
|
||||||
|
|
||||||
- name: Checkout
|
|
||||||
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
||||||
# This step checks out the code from the repository.
|
|
||||||
# It uses the actions/checkout action to clone the repository into the runner's workspace.
|
|
||||||
|
|
||||||
- name: Docker meta default
|
|
||||||
# This step generates metadata for the Docker image.
|
|
||||||
# It uses the docker/metadata-action to create metadata based on the repository information.
|
|
||||||
# The metadata includes information such as the image name, tags, and labels.
|
|
||||||
# The metadata will be used later in the workflow to build and push the Docker image.
|
|
||||||
id: meta
|
|
||||||
uses: docker/metadata-action@dc802804100637a589fabce1cb79ff13a1411302 # v6.2.0
|
|
||||||
with:
|
|
||||||
images: ${{ env.GHCR_IMAGE }}
|
|
||||||
|
|
||||||
|
|
||||||
- name: Set up Docker Context for Buildx
|
|
||||||
# This step sets up a Docker context for Buildx.
|
|
||||||
# It creates a new context named "builders" that will be used for building the Docker image.
|
|
||||||
# The context allows Buildx to use the Docker daemon for building images.
|
|
||||||
id: buildx-context
|
|
||||||
run: |
|
|
||||||
docker context create builders
|
|
||||||
|
|
||||||
- name: Set up Docker Buildx
|
|
||||||
# This step sets up Docker Buildx, which is a Docker CLI plugin for extended build capabilities with BuildKit.
|
|
||||||
# It uses the docker/setup-buildx-action to configure Buildx with the specified context and platforms.
|
|
||||||
# The platforms are specified in the matrix and will be used for building the Docker image.
|
|
||||||
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
|
|
||||||
with:
|
|
||||||
endpoint: builders
|
|
||||||
platforms: ${{ matrix.platform }}
|
|
||||||
|
|
||||||
- name: Login to GitHub Container Registry
|
|
||||||
# This step logs in to the GitHub Container Registry (GHCR) using the docker/login-action.
|
|
||||||
# It uses the GitHub actor's username and the GITHUB_TOKEN secret for authentication.
|
|
||||||
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
|
|
||||||
with:
|
|
||||||
registry: ghcr.io
|
|
||||||
username: ${{ github.actor }}
|
|
||||||
password: ${{ secrets.GITHUB_TOKEN }}
|
|
||||||
|
|
||||||
|
|
||||||
- name: Build and push by digest
|
|
||||||
# This step builds and pushes the Docker image using Buildx.
|
|
||||||
# It uses the docker/build-push-action to build the image with the specified context and platforms.
|
|
||||||
# The image is built with the labels and annotations generated in the previous steps.
|
|
||||||
# The outputs are configured to push the image by digest, which allows for better caching and versioning.
|
|
||||||
# The cache-from and cache-to options are used to enable caching for the build process.
|
|
||||||
# The cache is stored in GitHub Actions cache and is scoped to the repository, branch, and platform.
|
|
||||||
id: build
|
|
||||||
uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7.4.0
|
|
||||||
env:
|
|
||||||
DOCKER_BUILDKIT: 1
|
|
||||||
DSH_PORT: 3080
|
|
||||||
with:
|
|
||||||
context: .
|
|
||||||
platforms: ${{ matrix.platform }}
|
|
||||||
labels: ${{ steps.meta.outputs.labels }}
|
|
||||||
build-args: |
|
|
||||||
DSH_VERSION=${{ env.DSH_VERSION }}
|
|
||||||
PNPM_VERSION=${{ env.PNPM_VERSION }}
|
|
||||||
annotations: ${{ steps.meta.outputs.annotations }}
|
|
||||||
outputs: type=image,name=${{ env.GHCR_IMAGE }},push-by-digest=true,name-canonical=true,push=true,oci-mediatypes=true
|
|
||||||
cache-from: type=gha,scope=${{ github.repository }}-${{ github.ref_name }}-${{ matrix.platform }}
|
|
||||||
cache-to: type=gha,scope=${{ github.repository }}-${{ github.ref_name }}-${{ matrix.platform }}
|
|
||||||
|
|
||||||
|
|
||||||
- name: Export digest
|
|
||||||
# This step exports the digest of the built image to a file.
|
|
||||||
# It creates a directory in /tmp/digests and saves the digest of the image to a file.
|
|
||||||
# The digest is obtained from the output of the build step.
|
|
||||||
# The digest is used to uniquely identify the built image and can be used for further processing or verification.
|
|
||||||
run: |
|
|
||||||
mkdir -p /tmp/digests
|
|
||||||
digest="${{ steps.build.outputs.digest }}"
|
|
||||||
touch "/tmp/digests/${digest#sha256:}"
|
|
||||||
|
|
||||||
- name: Upload digest
|
|
||||||
# This step uploads the digest file to the GitHub Actions artifact storage.
|
|
||||||
# It uses the actions/upload-artifact action to upload the file created in the previous step.
|
|
||||||
# The artifact is named digests-${{ env.PLATFORM_PAIR }}, where PLATFORM_PAIR is the platform name with '/' replaced by '-'.
|
|
||||||
# The artifact is retained for 1 day, and if no files are found, it will throw an error.
|
|
||||||
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
|
|
||||||
with:
|
|
||||||
name: digests-${{ env.PLATFORM_PAIR }}
|
|
||||||
path: /tmp/digests/*
|
|
||||||
if-no-files-found: error
|
|
||||||
retention-days: 1
|
|
||||||
|
|
||||||
|
|
||||||
merge:
|
|
||||||
# This job merges the Docker manifests for the different platforms built in the previous job.
|
|
||||||
name: Merge Docker manifests
|
|
||||||
runs-on: ubuntu-latest
|
|
||||||
permissions:
|
|
||||||
attestations: write
|
|
||||||
actions: read
|
|
||||||
checks: read
|
|
||||||
contents: read
|
|
||||||
deployments: none
|
|
||||||
id-token: write
|
|
||||||
issues: read
|
|
||||||
discussions: read
|
|
||||||
packages: write
|
|
||||||
pages: none
|
|
||||||
pull-requests: read
|
|
||||||
repository-projects: read
|
|
||||||
security-events: read
|
|
||||||
statuses: read
|
|
||||||
|
|
||||||
needs:
|
|
||||||
- build
|
|
||||||
# This job depends on the build job to complete before it starts.
|
|
||||||
# It ensures that the Docker images for all platforms are built before merging the manifests.
|
|
||||||
steps:
|
|
||||||
- name: Download digests
|
|
||||||
# This step downloads the digest files uploaded in the build job.
|
|
||||||
# It uses the actions/download-artifact action to download the artifacts with the pattern digests-*.
|
|
||||||
# The downloaded files are merged into the /tmp/digests directory.
|
|
||||||
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
|
|
||||||
with:
|
|
||||||
path: /tmp/digests
|
|
||||||
pattern: digests-*
|
|
||||||
merge-multiple: true
|
|
||||||
|
|
||||||
|
|
||||||
- name: Docker meta
|
|
||||||
# This step generates metadata for the Docker image.
|
|
||||||
# It uses the docker/metadata-action to create metadata based on the repository information.
|
|
||||||
# The metadata includes information such as the image name, tags, and labels.
|
|
||||||
id: meta
|
|
||||||
uses: docker/metadata-action@dc802804100637a589fabce1cb79ff13a1411302 # v6.2.0
|
|
||||||
with:
|
|
||||||
images: ${{ env.GHCR_IMAGE }}
|
|
||||||
annotations: |
|
|
||||||
type=org.opencontainers.image.description,value=${{ github.event.repository.description || 'No description provided' }}
|
|
||||||
tags: |
|
|
||||||
type=raw,value=main,enable=${{ github.ref_name == 'main' }}
|
|
||||||
type=raw,value=latest,enable=${{ github.ref_name == 'main' }}
|
|
||||||
type=raw,value=${{ env.DSH_VERSION }},enable=${{ github.ref_name == 'main' || github.event_name == 'workflow_dispatch' }}
|
|
||||||
|
|
||||||
- name: Set up Docker Buildx
|
|
||||||
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
|
|
||||||
# This step sets up Docker Buildx, which is a Docker CLI plugin for extended build capabilities with BuildKit.
|
|
||||||
with:
|
|
||||||
driver-opts: |
|
|
||||||
network=host
|
|
||||||
|
|
||||||
- name: Login to GitHub Container Registry
|
|
||||||
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
|
|
||||||
# This step logs in to the GitHub Container Registry (GHCR) using the docker/login-action.
|
|
||||||
# It uses the GitHub actor's username and the GITHUB_TOKEN secret for authentication.
|
|
||||||
# The login is necessary to push the merged manifest list to GHCR.
|
|
||||||
with:
|
|
||||||
registry: ghcr.io
|
|
||||||
username: ${{ github.actor }}
|
|
||||||
password: ${{ secrets.GITHUB_TOKEN }}
|
|
||||||
|
|
||||||
- name: Get execution timestamp with RFC3339 format
|
|
||||||
# This step gets the current execution timestamp in RFC3339 format.
|
|
||||||
# It uses the date command to get the current UTC time and formats it as a string.
|
|
||||||
# The timestamp is used for annotating the Docker manifest list.
|
|
||||||
id: timestamp
|
|
||||||
run: |
|
|
||||||
echo "timestamp=$(date -u +"%Y-%m-%dT%H:%M:%SZ")" >> $GITHUB_OUTPUT
|
|
||||||
|
|
||||||
- name: Create manifest list and pushs
|
|
||||||
# This step creates a manifest list for the Docker images built for different platforms.
|
|
||||||
# It uses the docker buildx imagetools create command to create the manifest list.
|
|
||||||
# The manifest list is annotated with metadata such as description, creation timestamp, and source URL.
|
|
||||||
# The annotations are obtained from the metadata generated in the previous steps.
|
|
||||||
# The manifest list is pushed to the GitHub Container Registry (GHCR) with the specified tags.
|
|
||||||
working-directory: /tmp/digests
|
|
||||||
id: manifest-annotate
|
|
||||||
continue-on-error: true
|
|
||||||
run: |
|
|
||||||
docker buildx imagetools create \
|
|
||||||
$(jq -cr '.tags | map("-t " + .) | join(" ")' <<< "$DOCKER_METADATA_OUTPUT_JSON") \
|
|
||||||
--annotation='index:org.opencontainers.image.description=${{ github.event.repository.description }}' \
|
|
||||||
--annotation='index:org.opencontainers.image.created=${{ steps.timestamp.outputs.timestamp }}' \
|
|
||||||
--annotation='index:org.opencontainers.image.url=${{ github.event.repository.url }}' \
|
|
||||||
--annotation='index:org.opencontainers.image.source=${{ github.event.repository.url }}' \
|
|
||||||
$(printf '${{ env.GHCR_IMAGE }}@sha256:%s ' *)
|
|
||||||
|
|
||||||
- name: Create manifest list and push without annotations
|
|
||||||
# This step creates a manifest list for the Docker images built for different platforms.
|
|
||||||
# It uses the docker buildx imagetools create command to create the manifest list.
|
|
||||||
# The manifest list is created without annotations if the previous step fails.
|
|
||||||
# The manifest list is pushed to the GitHub Container Registry (GHCR) with the specified tags.
|
|
||||||
if: steps.manifest-annotate.outcome == 'failure'
|
|
||||||
working-directory: /tmp/digests
|
|
||||||
run: |
|
|
||||||
docker buildx imagetools create $(jq -cr '.tags | map("-t " + .) | join(" ")' <<< "$DOCKER_METADATA_OUTPUT_JSON") \
|
|
||||||
$(printf '${{ env.GHCR_IMAGE }}@sha256:%s ' *)
|
|
||||||
|
|
||||||
- name: Inspect image
|
|
||||||
# This step inspects the created manifest list to verify its contents.
|
|
||||||
# It uses the docker buildx imagetools inspect command to display information about the manifest list.
|
|
||||||
# The inspection output will show the platforms and tags associated with the manifest list.
|
|
||||||
id: inspect
|
|
||||||
run: |
|
|
||||||
docker buildx imagetools inspect '${{ env.GHCR_IMAGE }}:${{ steps.meta.outputs.version }}'
|
|
||||||
38
.github/workflows/pr-title.yaml
vendored
Normal file
38
.github/workflows/pr-title.yaml
vendored
Normal file
@@ -0,0 +1,38 @@
|
|||||||
|
# Enforce Conventional Commit pull request titles so git-cliff can group
|
||||||
|
# release notes. The PR is squash-merged, so the title becomes the commit.
|
||||||
|
name: PR title
|
||||||
|
|
||||||
|
on:
|
||||||
|
pull_request:
|
||||||
|
types: [opened, edited, synchronize, reopened]
|
||||||
|
|
||||||
|
permissions:
|
||||||
|
contents: read
|
||||||
|
pull-requests: read
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
lint:
|
||||||
|
name: pr-title
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
timeout-minutes: 5
|
||||||
|
steps:
|
||||||
|
- name: Validate title
|
||||||
|
uses: amannn/action-semantic-pull-request@48f256284bd46cdaab1048c3721360e808335d50 # v6.1.1
|
||||||
|
env:
|
||||||
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||||
|
with:
|
||||||
|
types: |
|
||||||
|
feat
|
||||||
|
fix
|
||||||
|
docs
|
||||||
|
refactor
|
||||||
|
perf
|
||||||
|
test
|
||||||
|
build
|
||||||
|
ci
|
||||||
|
chore
|
||||||
|
revert
|
||||||
|
requireScope: false
|
||||||
|
subjectPattern: ^(?![A-Z]).+$
|
||||||
|
subjectPatternError: |
|
||||||
|
The PR title subject must start in lowercase and must not end with a period.
|
||||||
268
.github/workflows/release.yaml
vendored
Normal file
268
.github/workflows/release.yaml
vendored
Normal file
@@ -0,0 +1,268 @@
|
|||||||
|
# Release pipeline. Pushing a tag that matches the identity in versions.env
|
||||||
|
# publishes both images (version + latest), creates the GitHub Release from
|
||||||
|
# git-cliff, and opens a PR that refreshes CHANGELOG.md on main.
|
||||||
|
#
|
||||||
|
# The tag must be dsh-vX.Y.Z or dsh-vX.Y.Z-rN and must match versions.env;
|
||||||
|
# tags are immutable, so a broken release is fixed with a new -rN revision.
|
||||||
|
name: Release
|
||||||
|
|
||||||
|
on:
|
||||||
|
push:
|
||||||
|
tags:
|
||||||
|
- "dsh-v*"
|
||||||
|
workflow_dispatch:
|
||||||
|
inputs:
|
||||||
|
tag:
|
||||||
|
description: Existing tag to release (must match versions.env)
|
||||||
|
required: true
|
||||||
|
type: string
|
||||||
|
publish_legacy_gateway_alias:
|
||||||
|
description: Also publish the legacy :gateway alias (one release only)
|
||||||
|
required: false
|
||||||
|
default: true
|
||||||
|
type: boolean
|
||||||
|
|
||||||
|
permissions:
|
||||||
|
contents: write
|
||||||
|
packages: write
|
||||||
|
id-token: write
|
||||||
|
attestations: write
|
||||||
|
|
||||||
|
concurrency:
|
||||||
|
group: release-${{ github.ref }}
|
||||||
|
cancel-in-progress: false
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
meta:
|
||||||
|
name: Validate release identity
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
timeout-minutes: 15
|
||||||
|
outputs:
|
||||||
|
identity: ${{ steps.meta.outputs.identity }}
|
||||||
|
dsh_version: ${{ steps.meta.outputs.dsh_version }}
|
||||||
|
pnpm_version: ${{ steps.meta.outputs.pnpm_version }}
|
||||||
|
release_revision: ${{ steps.meta.outputs.release_revision }}
|
||||||
|
ghcr_image: ${{ steps.meta.outputs.ghcr_image }}
|
||||||
|
gateway_image: ${{ steps.meta.outputs.gateway_image }}
|
||||||
|
tag: ${{ steps.meta.outputs.tag }}
|
||||||
|
steps:
|
||||||
|
- name: Checkout
|
||||||
|
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
||||||
|
with:
|
||||||
|
fetch-depth: 0
|
||||||
|
persist-credentials: false
|
||||||
|
|
||||||
|
- name: Validate tag against versions.env
|
||||||
|
id: meta
|
||||||
|
shell: bash
|
||||||
|
run: |
|
||||||
|
set -euo pipefail
|
||||||
|
set -a
|
||||||
|
# shellcheck disable=SC1091
|
||||||
|
source versions.env
|
||||||
|
set +a
|
||||||
|
tag="${{ inputs.tag || github.ref_name }}"
|
||||||
|
identity="$DSH_VERSION"
|
||||||
|
[ -z "$RELEASE_REVISION" ] || identity="$DSH_VERSION-$RELEASE_REVISION"
|
||||||
|
if [ "$tag" != "$identity" ]; then
|
||||||
|
echo "tag '$tag' does not match versions.env identity '$identity'" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
if git rev-parse -q --verify "refs/tags/$tag" >/dev/null; then
|
||||||
|
git fetch origin main '+refs/heads/release/*:refs/remotes/origin/release/*'
|
||||||
|
commit=$(git rev-list -n1 "refs/tags/$tag")
|
||||||
|
allowed=0
|
||||||
|
for ref in origin/main origin/release/*; do
|
||||||
|
if git rev-parse -q --verify "$ref" >/dev/null && git merge-base --is-ancestor "$commit" "$ref"; then
|
||||||
|
allowed=1
|
||||||
|
break
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
if [ "$allowed" -ne 1 ]; then
|
||||||
|
echo "tag commit $commit is not on main or a release branch" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
echo "tag '$tag' does not exist in this repository" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
if ! git ls-remote --tags --exit-code "$DSH_REPO" "$DSH_VERSION" >/dev/null; then
|
||||||
|
echo "upstream ref '$DSH_VERSION' not found in $DSH_REPO" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
{
|
||||||
|
echo "identity=$identity"
|
||||||
|
echo "dsh_version=$DSH_VERSION"
|
||||||
|
echo "pnpm_version=$PNPM_VERSION"
|
||||||
|
echo "release_revision=$RELEASE_REVISION"
|
||||||
|
echo "ghcr_image=$GHCR_IMAGE"
|
||||||
|
echo "gateway_image=$GATEWAY_IMAGE"
|
||||||
|
echo "tag=$tag"
|
||||||
|
} >> "$GITHUB_OUTPUT"
|
||||||
|
|
||||||
|
- name: Verify repository pins
|
||||||
|
run: ./scripts/check-versions.sh
|
||||||
|
|
||||||
|
tests:
|
||||||
|
name: Unit tests and lint
|
||||||
|
needs: meta
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
timeout-minutes: 15
|
||||||
|
defaults:
|
||||||
|
run:
|
||||||
|
working-directory: examples/dsh-agents-as-a-service/gateway
|
||||||
|
steps:
|
||||||
|
- name: Checkout
|
||||||
|
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
||||||
|
with:
|
||||||
|
persist-credentials: false
|
||||||
|
- name: Set up Python
|
||||||
|
uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
|
||||||
|
with:
|
||||||
|
python-version: "3.12"
|
||||||
|
cache: pip
|
||||||
|
cache-dependency-path: |
|
||||||
|
examples/dsh-agents-as-a-service/gateway/requirements.txt
|
||||||
|
examples/dsh-agents-as-a-service/gateway/requirements-dev.txt
|
||||||
|
- name: Install dependencies
|
||||||
|
run: |
|
||||||
|
set -euo pipefail
|
||||||
|
python -m pip install --upgrade pip
|
||||||
|
pip install -r requirements.txt -r requirements-dev.txt
|
||||||
|
- name: Lint
|
||||||
|
run: |
|
||||||
|
set -euo pipefail
|
||||||
|
python -m ruff check .
|
||||||
|
python -m ruff format --check .
|
||||||
|
- name: Unit tests
|
||||||
|
run: python -m pytest -q
|
||||||
|
|
||||||
|
build-main:
|
||||||
|
name: Publish main image
|
||||||
|
needs: [meta, tests]
|
||||||
|
uses: ./.github/workflows/_build-image.yaml
|
||||||
|
permissions:
|
||||||
|
contents: read
|
||||||
|
packages: write
|
||||||
|
id-token: write
|
||||||
|
attestations: write
|
||||||
|
with:
|
||||||
|
image: ${{ needs.meta.outputs.ghcr_image }}
|
||||||
|
version: ${{ needs.meta.outputs.identity }}
|
||||||
|
context: .
|
||||||
|
dockerfile: Dockerfile
|
||||||
|
build-args: |
|
||||||
|
DSH_VERSION=${{ needs.meta.outputs.dsh_version }}
|
||||||
|
RELEASE_REVISION=${{ needs.meta.outputs.release_revision }}
|
||||||
|
PNPM_VERSION=${{ needs.meta.outputs.pnpm_version }}
|
||||||
|
VCS_REF=${{ github.sha }}
|
||||||
|
tags: |
|
||||||
|
${{ needs.meta.outputs.ghcr_image }}:${{ needs.meta.outputs.identity }}
|
||||||
|
${{ needs.meta.outputs.ghcr_image }}:latest
|
||||||
|
|
||||||
|
build-gateway:
|
||||||
|
name: Publish gateway image
|
||||||
|
needs: [meta, tests]
|
||||||
|
uses: ./.github/workflows/_build-image.yaml
|
||||||
|
permissions:
|
||||||
|
contents: read
|
||||||
|
packages: write
|
||||||
|
id-token: write
|
||||||
|
attestations: write
|
||||||
|
with:
|
||||||
|
image: ${{ needs.meta.outputs.gateway_image }}
|
||||||
|
version: ${{ needs.meta.outputs.identity }}
|
||||||
|
context: examples/dsh-agents-as-a-service/gateway
|
||||||
|
dockerfile: Dockerfile
|
||||||
|
tags: |
|
||||||
|
${{ needs.meta.outputs.gateway_image }}:${{ needs.meta.outputs.identity }}
|
||||||
|
${{ needs.meta.outputs.gateway_image }}:latest
|
||||||
|
|
||||||
|
legacy-gateway-alias:
|
||||||
|
name: Publish legacy :gateway alias (temporary)
|
||||||
|
needs: [meta, build-gateway]
|
||||||
|
if: github.event_name != 'workflow_dispatch' || inputs.publish_legacy_gateway_alias
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
timeout-minutes: 15
|
||||||
|
steps:
|
||||||
|
- name: Login to GitHub Container Registry
|
||||||
|
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
|
||||||
|
with:
|
||||||
|
registry: ghcr.io
|
||||||
|
username: ${{ github.actor }}
|
||||||
|
password: ${{ secrets.GITHUB_TOKEN }}
|
||||||
|
- name: Retag gateway as the legacy package tag
|
||||||
|
env:
|
||||||
|
SOURCE: ${{ needs.meta.outputs.gateway_image }}:${{ needs.meta.outputs.identity }}
|
||||||
|
LEGACY: ${{ needs.meta.outputs.ghcr_image }}:gateway
|
||||||
|
run: |
|
||||||
|
set -euo pipefail
|
||||||
|
docker buildx imagetools create -t "$LEGACY" "$SOURCE"
|
||||||
|
echo "Deprecated alias $LEGACY now points at $SOURCE" >> "$GITHUB_STEP_SUMMARY"
|
||||||
|
|
||||||
|
publish-release:
|
||||||
|
name: Changelog and GitHub Release
|
||||||
|
needs: [meta, build-main, build-gateway]
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
timeout-minutes: 20
|
||||||
|
steps:
|
||||||
|
- name: Checkout
|
||||||
|
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
||||||
|
with:
|
||||||
|
fetch-depth: 0
|
||||||
|
|
||||||
|
- name: Install git-cliff
|
||||||
|
env:
|
||||||
|
GIT_CLIFF_VERSION: v2.14.2
|
||||||
|
run: |
|
||||||
|
set -euo pipefail
|
||||||
|
version="${GIT_CLIFF_VERSION#v}"
|
||||||
|
curl -fsSL -o /tmp/git-cliff.tar.gz "https://github.com/orhun/git-cliff/releases/download/$GIT_CLIFF_VERSION/git-cliff-$version-x86_64-unknown-linux-gnu.tar.gz"
|
||||||
|
tar -xzf /tmp/git-cliff.tar.gz -C /tmp
|
||||||
|
install -m 0755 "/tmp/git-cliff-$version/git-cliff" /usr/local/bin/git-cliff
|
||||||
|
git-cliff --version
|
||||||
|
|
||||||
|
- name: Generate CHANGELOG and release notes
|
||||||
|
env:
|
||||||
|
IDENTITY: ${{ needs.meta.outputs.identity }}
|
||||||
|
run: |
|
||||||
|
set -euo pipefail
|
||||||
|
git-cliff --tag "$IDENTITY" --output CHANGELOG.md
|
||||||
|
git-cliff --tag "$IDENTITY" --latest --strip header --output /tmp/release-notes.md
|
||||||
|
echo "### Release $IDENTITY" >> "$GITHUB_STEP_SUMMARY"
|
||||||
|
head -n 40 /tmp/release-notes.md >> "$GITHUB_STEP_SUMMARY"
|
||||||
|
|
||||||
|
- name: Create GitHub Release
|
||||||
|
env:
|
||||||
|
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||||
|
IDENTITY: ${{ needs.meta.outputs.identity }}
|
||||||
|
run: |
|
||||||
|
set -euo pipefail
|
||||||
|
if gh release view "$IDENTITY" >/dev/null 2>&1; then
|
||||||
|
echo "release $IDENTITY already exists; skipping" >&2
|
||||||
|
else
|
||||||
|
gh release create "$IDENTITY" --title "$IDENTITY" --notes-file /tmp/release-notes.md --verify-tag
|
||||||
|
fi
|
||||||
|
|
||||||
|
- name: Open CHANGELOG pull request
|
||||||
|
env:
|
||||||
|
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||||
|
IDENTITY: ${{ needs.meta.outputs.identity }}
|
||||||
|
run: |
|
||||||
|
set -euo pipefail
|
||||||
|
git config user.name "github-actions[bot]"
|
||||||
|
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
|
||||||
|
if git diff --quiet -- CHANGELOG.md; then
|
||||||
|
echo "CHANGELOG.md already up to date"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
branch="chore/changelog-$IDENTITY"
|
||||||
|
git switch -c "$branch"
|
||||||
|
git add CHANGELOG.md
|
||||||
|
git commit -m "chore(changelog): refresh for $IDENTITY"
|
||||||
|
git push origin "$branch"
|
||||||
|
gh pr create \
|
||||||
|
--base main \
|
||||||
|
--head "$branch" \
|
||||||
|
--title "chore(changelog): refresh for $IDENTITY" \
|
||||||
|
--body "Generated by git-cliff for $IDENTITY. Do not edit CHANGELOG.md by hand."
|
||||||
137
.github/workflows/validate.yaml
vendored
137
.github/workflows/validate.yaml
vendored
@@ -1,137 +0,0 @@
|
|||||||
name: Validate
|
|
||||||
|
|
||||||
# Validates the Kubernetes manifests against the real agent-sandbox CRD
|
|
||||||
# schema (server-side dry-run on a throwaway kind cluster), runs the gateway
|
|
||||||
# unit tests + lint, and builds/smoke-tests the gateway image without a
|
|
||||||
# cluster. Runs on every PR and on pushes to main.
|
|
||||||
on:
|
|
||||||
pull_request:
|
|
||||||
push:
|
|
||||||
branches: [main]
|
|
||||||
|
|
||||||
permissions:
|
|
||||||
contents: read
|
|
||||||
|
|
||||||
jobs:
|
|
||||||
validate-manifests:
|
|
||||||
runs-on: ubuntu-latest
|
|
||||||
steps:
|
|
||||||
- name: Checkout
|
|
||||||
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
||||||
|
|
||||||
- name: Create kind cluster
|
|
||||||
uses: helm/kind-action@1544676c07bb3570fcb70166beaa07c4528fd4aa # v1.15.1
|
|
||||||
with:
|
|
||||||
version: v0.33.0
|
|
||||||
cluster_name: dsh-validate
|
|
||||||
|
|
||||||
- name: Install agent-sandbox (extensions)
|
|
||||||
run: |
|
|
||||||
VERSION=v1.0.7
|
|
||||||
kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/download/${VERSION}/sandbox-with-extensions.yaml
|
|
||||||
kubectl wait --for=condition=Established crd/sandboxtemplates.extensions.agents.x-k8s.io --timeout=180s
|
|
||||||
kubectl wait --for=condition=Established crd/sandboxwarmpools.extensions.agents.x-k8s.io --timeout=180s
|
|
||||||
kubectl wait --for=condition=Established crd/sandboxclaims.extensions.agents.x-k8s.io --timeout=180s
|
|
||||||
kubectl wait --for=condition=Established crd/sandboxes.agents.x-k8s.io --timeout=180s
|
|
||||||
|
|
||||||
- name: Validate manifests (server dry-run)
|
|
||||||
run: |
|
|
||||||
set -euo pipefail
|
|
||||||
kubectl create namespace dsh-demo
|
|
||||||
for f in examples/dsh-agents-as-a-service/*.yaml; do
|
|
||||||
echo "== $f =="
|
|
||||||
kubectl apply --dry-run=server -f "$f"
|
|
||||||
done
|
|
||||||
kubectl apply --dry-run=server -f deploy/k8s/sandbox-template.yaml
|
|
||||||
echo "all manifests validated"
|
|
||||||
|
|
||||||
gateway-tests:
|
|
||||||
# The gateway lives in examples/dsh-agents-as-a-service/gateway/. Its
|
|
||||||
# runtime pins and dev pins are kept separate so the image only ships
|
|
||||||
# requirements.txt while CI installs both files.
|
|
||||||
name: Gateway unit tests and lint
|
|
||||||
runs-on: ubuntu-latest
|
|
||||||
defaults:
|
|
||||||
run:
|
|
||||||
working-directory: examples/dsh-agents-as-a-service/gateway
|
|
||||||
steps:
|
|
||||||
- name: Checkout
|
|
||||||
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
||||||
|
|
||||||
- name: Set up Python
|
|
||||||
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
|
|
||||||
with:
|
|
||||||
python-version: "3.12"
|
|
||||||
cache: pip
|
|
||||||
cache-dependency-path: |
|
|
||||||
examples/dsh-agents-as-a-service/gateway/requirements.txt
|
|
||||||
examples/dsh-agents-as-a-service/gateway/requirements-dev.txt
|
|
||||||
|
|
||||||
- name: Install dependencies
|
|
||||||
run: |
|
|
||||||
python -m pip install --upgrade pip
|
|
||||||
pip install -r requirements.txt -r requirements-dev.txt
|
|
||||||
|
|
||||||
- name: Lint (ruff)
|
|
||||||
run: python -m ruff check .
|
|
||||||
|
|
||||||
- name: Run gateway unit tests
|
|
||||||
run: python -m pytest -v
|
|
||||||
|
|
||||||
gateway-image:
|
|
||||||
# No cluster and no real credentials: the gateway must start and answer
|
|
||||||
# /healthz with a dummy kubeconfig, which catches broken imports, missing
|
|
||||||
# runtime deps, and an unresponsive ASGI entrypoint.
|
|
||||||
name: Build and smoke-test gateway image
|
|
||||||
runs-on: ubuntu-latest
|
|
||||||
steps:
|
|
||||||
- name: Checkout
|
|
||||||
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
||||||
|
|
||||||
- name: Build gateway image
|
|
||||||
run: |
|
|
||||||
docker build \
|
|
||||||
-t dsh-aaas-gateway:ci \
|
|
||||||
examples/dsh-agents-as-a-service/gateway/
|
|
||||||
|
|
||||||
- name: Smoke-test /healthz without a cluster
|
|
||||||
run: |
|
|
||||||
set -euo pipefail
|
|
||||||
cat > /tmp/kubeconfig <<'EOF'
|
|
||||||
apiVersion: v1
|
|
||||||
kind: Config
|
|
||||||
clusters:
|
|
||||||
- name: dummy
|
|
||||||
cluster:
|
|
||||||
server: https://127.0.0.1:6443
|
|
||||||
insecure-skip-tls-verify: true
|
|
||||||
contexts:
|
|
||||||
- name: dummy
|
|
||||||
context:
|
|
||||||
cluster: dummy
|
|
||||||
user: dummy
|
|
||||||
users:
|
|
||||||
- name: dummy
|
|
||||||
user: {}
|
|
||||||
current-context: dummy
|
|
||||||
EOF
|
|
||||||
docker run -d --name gateway-smoke \
|
|
||||||
-p 18080:8080 \
|
|
||||||
-e NAMESPACE=dsh-demo \
|
|
||||||
-e POOL=dsh-pool \
|
|
||||||
-e KUBECONFIG=/tmp/kubeconfig \
|
|
||||||
-v /tmp/kubeconfig:/tmp/kubeconfig:ro \
|
|
||||||
dsh-aaas-gateway:ci
|
|
||||||
trap 'docker logs gateway-smoke || true; docker rm -f gateway-smoke || true' EXIT
|
|
||||||
|
|
||||||
code=""
|
|
||||||
for _ in $(seq 1 30); do
|
|
||||||
code=$(curl -s -o /dev/null -w '%{http_code}' http://localhost:18080/healthz || true)
|
|
||||||
[ "$code" = "200" ] && break
|
|
||||||
sleep 1
|
|
||||||
done
|
|
||||||
if [ "$code" != "200" ]; then
|
|
||||||
echo "gateway /healthz returned '$code' instead of 200" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
echo "gateway image healthy: GET /healthz -> 200"
|
|
||||||
5
.gitignore
vendored
5
.gitignore
vendored
@@ -159,3 +159,8 @@ __pycache__/
|
|||||||
htmlcov/
|
htmlcov/
|
||||||
.venv/
|
.venv/
|
||||||
venv/
|
venv/
|
||||||
|
|
||||||
|
# Local repository tooling
|
||||||
|
.uvcache/
|
||||||
|
coverage.xml
|
||||||
|
|
||||||
|
|||||||
11
.hadolint.yaml
Normal file
11
.hadolint.yaml
Normal file
@@ -0,0 +1,11 @@
|
|||||||
|
# Container lint policy. Base images are supplied as build args defaulting to
|
||||||
|
# pinned tags, and the image is rebuilt on every DSH release, so unpinned
|
||||||
|
# packages inside a pinned base image are intentional.
|
||||||
|
failure-threshold: warning
|
||||||
|
ignored:
|
||||||
|
- DL3008 # apt-get packages are intentionally not version-pinned
|
||||||
|
- DL3018 # apk packages are intentionally not version-pinned
|
||||||
|
- DL3006 # FROM uses build args (NODE_BUILD_IMAGE/NODE_RUNTIME_IMAGE) from versions.env
|
||||||
|
- DL3003 # the DSH packaging step deliberately cds inside a single layer
|
||||||
|
- DL3059 # consecutive RUN layers are kept for cache reuse and readability
|
||||||
|
- DL4006 # the upstream uv installer pipes curl to sh by design
|
||||||
13
.lychee.toml
Normal file
13
.lychee.toml
Normal file
@@ -0,0 +1,13 @@
|
|||||||
|
# Link checker for README/docs. Placeholders used by the examples are excluded
|
||||||
|
# on purpose; real URLs must resolve.
|
||||||
|
verbose = "info"
|
||||||
|
no_progress = true
|
||||||
|
max_retries = 2
|
||||||
|
exclude = [
|
||||||
|
"localhost",
|
||||||
|
"127\\.0\\.0\\.1",
|
||||||
|
"example\\.com",
|
||||||
|
"your-github-org",
|
||||||
|
"mailto:",
|
||||||
|
"ghcr\\.io/.*@sha256",
|
||||||
|
]
|
||||||
19
.yamllint.yml
Normal file
19
.yamllint.yml
Normal file
@@ -0,0 +1,19 @@
|
|||||||
|
extends: default
|
||||||
|
|
||||||
|
rules:
|
||||||
|
document-start: disable
|
||||||
|
line-length:
|
||||||
|
max: 200
|
||||||
|
level: error
|
||||||
|
truthy:
|
||||||
|
check-keys: false
|
||||||
|
comments-indentation: disable
|
||||||
|
comments:
|
||||||
|
min-spaces-from-content: 1
|
||||||
|
|
||||||
|
ignore: |
|
||||||
|
.venv/
|
||||||
|
.uvcache/
|
||||||
|
node_modules/
|
||||||
|
# Upstream CodeQL boilerplate: long documentation comments, linted by actionlint.
|
||||||
|
.github/workflows/codeql.yml
|
||||||
148
AGENTS.md
Normal file
148
AGENTS.md
Normal file
@@ -0,0 +1,148 @@
|
|||||||
|
# AGENTS.md
|
||||||
|
|
||||||
|
Guidance for AI coding agents (and humans) working in this repository. Read it
|
||||||
|
before changing anything; it encodes the conventions the CI enforces.
|
||||||
|
|
||||||
|
## What this repository is
|
||||||
|
|
||||||
|
- **The root is the primary artifact.** The Dockerfile packages DeepSeek
|
||||||
|
Harness Web (dsh web) as a slim, non-root, multi-arch image. Everything else
|
||||||
|
exists to support it or to demonstrate an implementation on top of it.
|
||||||
|
- **Every implementation is a self-contained example** under examples/<name>/.
|
||||||
|
There is exactly one example for the agents-as-a-service platform; do not
|
||||||
|
split it into parallel directories.
|
||||||
|
- **versions.env is the single source of truth** for the DSH ref, pnpm, the
|
||||||
|
control-plane versions and the image names.
|
||||||
|
- **Docs describe the shipped behavior**, not intentions. If you change a
|
||||||
|
manifest, port, env var or workflow, update the docs in the same change.
|
||||||
|
- **CHANGELOG.md is generated** by git-cliff from Conventional Commits. Never
|
||||||
|
edit it by hand.
|
||||||
|
|
||||||
|
## Golden rules
|
||||||
|
|
||||||
|
1. Keep the Dockerfile working for both linux/amd64 and linux/arm64; it must
|
||||||
|
stay the first-class artifact.
|
||||||
|
2. Never hard-code a version, image tag or registry path. Edit versions.env and
|
||||||
|
run make set-version when bumping.
|
||||||
|
3. Never commit a real secret or usable credential. Secret-shaped YAML lives in
|
||||||
|
*.yaml.example files; keys are documented in .env.sample only.
|
||||||
|
4. Every manifest under examples/*/manifests/ must parse and pass a server-side
|
||||||
|
dry-run in CI. Keep manifests numbered (NN-*.yaml), and keep exposure
|
||||||
|
variants under manifests/exposure/<variant>/ documented by the single
|
||||||
|
manifests/exposure/README.md (variants are mutually exclusive).
|
||||||
|
5. Do not edit CHANGELOG.md, and do not move or delete published tags.
|
||||||
|
Releasing is documented in docs/versioning.md.
|
||||||
|
6. Keep the gateway HTTP/WebSocket contract stable unless the change is
|
||||||
|
explicitly about it; clients and examples depend on it.
|
||||||
|
|
||||||
|
## Repository map
|
||||||
|
|
||||||
|
| Path | What lives there |
|
||||||
|
| --- | --- |
|
||||||
|
| Dockerfile | Main DSH Web image (build + runtime stages, entrypoint) |
|
||||||
|
| versions.env | Every pinned version and image name |
|
||||||
|
| Makefile | Canonical local entrypoint |
|
||||||
|
| build.sh | Deprecated shim that forwards to make |
|
||||||
|
| docs/ | main-image.md, examples.md, gitflow.md, versioning.md |
|
||||||
|
| examples/ | One implementation per directory (contract in docs/examples.md) |
|
||||||
|
| examples/dsh-agents-as-a-service/ | The multi-tenant platform example (manifests, gateway, docs, scripts) |
|
||||||
|
| examples/minimal-single-tenant/ | The just-one-DSH-Web quickstart |
|
||||||
|
| scripts/ | check-versions.sh, check-examples.sh, check-manifests.py, set-version.sh |
|
||||||
|
| .dsh/skills/ | Project-local DSH skills (auto-discovered); dsh-agent-sandbox-platform covers platform operation, MCP limits and extension recipes |
|
||||||
|
| .github/workflows/ | ci.yaml, reusable _build-image.yaml, build-*, release.yaml, e2e-kind.yaml, pr-title.yaml, codeql.yml |
|
||||||
|
|
||||||
|
## Commands
|
||||||
|
|
||||||
|
make check # versions, example layout and manifest YAML (run before every commit)
|
||||||
|
make test # gateway unit tests (109+ tests)
|
||||||
|
make lint # ruff + shellcheck (+ yamllint/actionlint when installed)
|
||||||
|
make image # build the main DSH Web image (slow: clones and builds DSH)
|
||||||
|
make gateway # build the example gateway image (fast)
|
||||||
|
make e2e # full kind walkthrough; needs a cluster with agent-sandbox
|
||||||
|
make changelog # regenerate CHANGELOG.md with git-cliff (release only)
|
||||||
|
make release-dry-run
|
||||||
|
make set-version DSH_VERSION=dsh-vX.Y.Z RELEASE_REVISION=rN
|
||||||
|
|
||||||
|
No cluster is needed for check, test, lint, gateway or image. Only make e2e
|
||||||
|
needs kind plus the agent-sandbox CRDs (see the example README).
|
||||||
|
|
||||||
|
## Testing expectations per change
|
||||||
|
|
||||||
|
| Change | Minimum verification |
|
||||||
|
| --- | --- |
|
||||||
|
| Documentation only | make check; verify relative links |
|
||||||
|
| Example manifest | make check; kubectl apply --dry-run=server in CI |
|
||||||
|
| Optional hardening (gVisor, nodeSelector, NetworkPolicy, provider keys) | make check; server dry-run in CI; manual verification on a capable cluster |
|
||||||
|
| Gateway code | make test; make lint; add or update tests |
|
||||||
|
| Dockerfile / versions.env | make check; make image (amd64) and a container smoke test |
|
||||||
|
| Workflow | actionlint in CI; keep actions SHA-pinned; no hard-coded versions |
|
||||||
|
| New example | make check; all files described in docs/examples.md |
|
||||||
|
| Project skill (.dsh/skills/) | make check; frontmatter and size check; shellcheck skill scripts; load it with the skill tool |
|
||||||
|
|
||||||
|
## Versioning and releases
|
||||||
|
|
||||||
|
- Release identity = DSH_VERSION, optionally with a repository revision suffix
|
||||||
|
for re-releases at the same DSH ref (current identity:
|
||||||
|
dsh-v0.2.1-rc.3-r4).
|
||||||
|
- The current identity is dsh-v0.2.1-rc.3-r4 (stored in versions.env). The
|
||||||
|
dsh-v0.2.1-rc.3 image tag already exists and tags are immutable, which is why
|
||||||
|
the first release under this scheme carries -r2.
|
||||||
|
- Channels: release tags are immutable; latest follows the newest release;
|
||||||
|
main is the development channel for both image packages.
|
||||||
|
- A broken release is fixed by publishing -r(N+1), never by re-tagging.
|
||||||
|
- The full procedure, compatibility table and rollback rules are in
|
||||||
|
docs/versioning.md.
|
||||||
|
|
||||||
|
## Branch and PR conventions
|
||||||
|
|
||||||
|
- Trunk-based with release branches, documented in docs/gitflow.md.
|
||||||
|
- Branch names: feat/, fix/, docs/, chore/, refactor/, test/.
|
||||||
|
- Conventional Commit subjects and PR titles (feat(scope): ..., fix(scope): ...).
|
||||||
|
- PRs are squash-merged; main stays linear and protected.
|
||||||
|
- Required checks: lint, python, manifests, gateway-image, pr-title, CodeQL,
|
||||||
|
dependency-review.
|
||||||
|
- Keep diffs scoped; update docs with the code; never mix a version bump with
|
||||||
|
unrelated changes.
|
||||||
|
|
||||||
|
## Adding an example
|
||||||
|
|
||||||
|
1. Read docs/examples.md and examples/README.md.
|
||||||
|
2. Create examples/<name>/ with README.md, manifests/NN-*.yaml, optional
|
||||||
|
manifests/exposure/<variant>/ (plus README), optional components with their
|
||||||
|
own Dockerfile/README/tests, and scripts/ (executable, idempotent,
|
||||||
|
self-cleaning).
|
||||||
|
3. Publish component images under <registry>/<owner>/<repo>/<example>/<component>.
|
||||||
|
4. Add the example to examples/README.md; scripts/check-examples.sh enforces
|
||||||
|
the layout.
|
||||||
|
|
||||||
|
## Security notes
|
||||||
|
|
||||||
|
- AUTH_PROXY_HEADER is opt-in. Only enable it when the gateway Service is
|
||||||
|
reachable exclusively through the authenticating proxy; otherwise any client
|
||||||
|
can spoof X-Forwarded-User.
|
||||||
|
- The example gateway reads the DSH launch token from pod logs. Treat pod logs
|
||||||
|
as sensitive and keep its RBAC namespace-scoped.
|
||||||
|
- The example shares provider keys (dsh-provider-keys plus the optional
|
||||||
|
dsh-provider-keys-extra) across sandboxes; do not describe it as
|
||||||
|
tenant-isolated.
|
||||||
|
- gVisor, nodeSelector and the NetworkPolicies are opt-in and are not active in
|
||||||
|
the base manifests; do not describe the base example as hardened.
|
||||||
|
- TLS terminates outside the gateway; set COOKIE_SECURE=true for HTTPS and keep
|
||||||
|
the base manifest safe for local HTTP.
|
||||||
|
|
||||||
|
## Ask before
|
||||||
|
|
||||||
|
- Changing release workflows, tag formats or published image names.
|
||||||
|
- Editing the gateway's public routes or auth semantics.
|
||||||
|
- Adding a new top-level tool, service or dependency to the main image.
|
||||||
|
- Rewriting generated files (CHANGELOG.md) or git history.
|
||||||
|
|
||||||
|
## Where to look first
|
||||||
|
|
||||||
|
- [docs/main-image.md](docs/main-image.md) - container contract and tags
|
||||||
|
- [docs/examples.md](docs/examples.md) - example authoring rules
|
||||||
|
- [docs/gitflow.md](docs/gitflow.md) - branches, PRs, required checks
|
||||||
|
- [docs/versioning.md](docs/versioning.md) - versioning, changelog, releases
|
||||||
|
- [.dsh/skills/dsh-agent-sandbox-platform/SKILL.md](.dsh/skills/dsh-agent-sandbox-platform/SKILL.md) - platform operations, MCP limits, extension recipes
|
||||||
|
- [CONTRIBUTING.md](CONTRIBUTING.md) - development setup
|
||||||
|
- [SECURITY.md](SECURITY.md) - vulnerability reporting and threat notes
|
||||||
55
CHANGELOG.md
Normal file
55
CHANGELOG.md
Normal file
@@ -0,0 +1,55 @@
|
|||||||
|
# Changelog
|
||||||
|
|
||||||
|
All notable changes to this repository are documented here.
|
||||||
|
Generated by git-cliff from Conventional Commits; do not edit this file by hand.
|
||||||
|
|
||||||
|
## [dsh-v0.2.1-rc.3-r3] - 2026-10-25
|
||||||
|
|
||||||
|
### Added
|
||||||
|
|
||||||
|
- **gateway-api:** Expose dsh-gateway externally via Traefik Gateway API (#12) (7d3faf0)
|
||||||
|
- **gateway-api:** GitHub SSO via oauth2-proxy in front of dsh-gateway (#13) (3f46279)
|
||||||
|
- **gateway:** SSO landing auto-connects and offers recreate sandbox (#15) (ed1398d)
|
||||||
|
- **repo:** Repo reorganization (#16) (3b70f17)
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
|
||||||
|
- **Dockerfile:** Update runtime dependencies and enable corepack for pnpm (6d97f61)
|
||||||
|
- **gateway-api:** Correct oauth2-proxy cookie-secret generation (#14) (73a65d9)
|
||||||
|
|
||||||
|
### Maintenance
|
||||||
|
|
||||||
|
- Bump the actions group across 1 directory with 3 updates (#17) (6a8fe54)
|
||||||
|
|
||||||
|
### Other
|
||||||
|
|
||||||
|
- Dockerfile extra tools (2b47afa)
|
||||||
|
- Modify gateway.yaml for image policy and timeout values (53e55fc)
|
||||||
|
- Add commit message prefix for Dependabot updates (6876871)
|
||||||
|
- Versioning update (#18) (ecba8b9)
|
||||||
|
- Refactor environment variable loading in build workflow (d627526)
|
||||||
|
- Refactor environment variable loading in build-gateway.yaml (027f74f)
|
||||||
|
- Refactor release workflow for environment variable sourcing (958cbe2)
|
||||||
|
|
||||||
|
## [v0.0.1] - 2026-10-21
|
||||||
|
|
||||||
|
### Other
|
||||||
|
|
||||||
|
- Initial commit (f3d4312)
|
||||||
|
- Update Dockerfile, README, and Kubernetes manifests for DSH Web deployment (#1) (0d66cba)
|
||||||
|
- Add DeepSeek Harness Web agents-as-a-service platform pattern (#2) (98cb547)
|
||||||
|
- Update action versions and cluster version (d9cb265)
|
||||||
|
- Add Python to CodeQL workflow languages (b0f992f)
|
||||||
|
- Add multi-arch Docker image build workflow with GitHub Actions (de11d34)
|
||||||
|
- Add step to create manifest list for Docker images (c9be929)
|
||||||
|
- Make DSH agents-as-a-service browser-usable; fix probe crash-loop (#3) (6301c21)
|
||||||
|
- Fix idle sweeper suspending claims the gateway does not own (#4) (29eefd6)
|
||||||
|
- Fix DSH crash-loop after suspend/resume (fsGroup chmod) (#5) (a8ef211)
|
||||||
|
- Add a startup probe so slow DSH cold starts are not killed (#6) (c193ed2)
|
||||||
|
- Normalise the pod model returned by CoreV1Api (#7) (9dea4a7)
|
||||||
|
- Forward the upstream query byte-for-byte (#8) (6876406)
|
||||||
|
- Do not read-time-out proxied upstream streams (#9) (163abae)
|
||||||
|
- Dial the sandbox pod IP for WebSocket upgrades (#10) (abe5b4d)
|
||||||
|
- Handshake under the pod-IP authority for WebSocket upgrades (#11) (0a61e57)
|
||||||
|
|
||||||
|
|
||||||
46
CODE_OF_CONDUCT.md
Normal file
46
CODE_OF_CONDUCT.md
Normal file
@@ -0,0 +1,46 @@
|
|||||||
|
# Code of Conduct
|
||||||
|
|
||||||
|
This project adopts the Contributor Covenant, version 2.1:
|
||||||
|
https://www.contributor-covenant.org/version/2/1/code_of_conduct/
|
||||||
|
|
||||||
|
## Our standards
|
||||||
|
|
||||||
|
We are committed to a welcoming, harassment-free experience for everyone,
|
||||||
|
regardless of age, body size, visible or invisible disability, ethnicity, sex
|
||||||
|
characteristics, gender identity and expression, level of experience,
|
||||||
|
education, socio-economic status, nationality, personal appearance, race,
|
||||||
|
religion, or sexual identity and orientation.
|
||||||
|
|
||||||
|
Examples of behavior that contributes to a positive environment:
|
||||||
|
|
||||||
|
- Demonstrating empathy and kindness toward other people
|
||||||
|
- Being respectful of differing opinions, viewpoints, and experiences
|
||||||
|
- Giving and gracefully accepting constructive feedback
|
||||||
|
- Accepting responsibility and apologizing to those affected by our mistakes
|
||||||
|
- Focusing on what is best for the community and the project
|
||||||
|
|
||||||
|
Examples of unacceptable behavior:
|
||||||
|
|
||||||
|
- Sexualized language or imagery, and sexual attention or advances
|
||||||
|
- Trolling, insulting or derogatory comments, and personal or political attacks
|
||||||
|
- Public or private harassment
|
||||||
|
- Publishing others' private information without explicit permission
|
||||||
|
- Other conduct which could reasonably be considered inappropriate in a
|
||||||
|
professional setting
|
||||||
|
|
||||||
|
## Enforcement
|
||||||
|
|
||||||
|
Maintainers are responsible for clarifying and enforcing these standards and
|
||||||
|
may remove, edit, or reject comments, commits, code, issues, and other
|
||||||
|
contributions that do not align with this Code of Conduct.
|
||||||
|
|
||||||
|
Report abusive, harassing, or otherwise unacceptable behavior to the
|
||||||
|
maintainers through a private GitHub Security Advisory or by contacting the
|
||||||
|
repository owner on GitHub. All complaints will be reviewed and investigated
|
||||||
|
promptly and fairly. Maintainers must respect the privacy and security of the
|
||||||
|
reporter.
|
||||||
|
|
||||||
|
## Attribution
|
||||||
|
|
||||||
|
This Code of Conduct is adapted from the Contributor Covenant, version 2.1,
|
||||||
|
available at https://www.contributor-covenant.org/version/2/1/code_of_conduct/
|
||||||
88
CONTRIBUTING.md
Normal file
88
CONTRIBUTING.md
Normal file
@@ -0,0 +1,88 @@
|
|||||||
|
# Contributing
|
||||||
|
|
||||||
|
Thanks for improving deepseek-kubernetes-sandbox. This repository packages
|
||||||
|
DeepSeek Harness Web as a container and ships self-contained implementation
|
||||||
|
examples around it. The root is the container; every implementation lives
|
||||||
|
under examples/.
|
||||||
|
|
||||||
|
## Repository layout
|
||||||
|
|
||||||
|
| Path | Purpose |
|
||||||
|
| --- | --- |
|
||||||
|
| Dockerfile | The primary artifact: the DSH Web image |
|
||||||
|
| versions.env | Single source of truth for every pinned version |
|
||||||
|
| Makefile | Canonical local entrypoint (build, test, lint, check, e2e) |
|
||||||
|
| docs/ | Repository-wide docs: main image, example contract, gitflow, versioning |
|
||||||
|
| examples/ | One self-contained implementation per directory |
|
||||||
|
| scripts/ | Repository checks (versions, example layout, manifests, version bump) |
|
||||||
|
| .github/ | CI, releases, dependency updates, templates |
|
||||||
|
|
||||||
|
## Requirements
|
||||||
|
|
||||||
|
- Docker (for image builds)
|
||||||
|
- Python 3.12+ (gateway tests and repository checks)
|
||||||
|
- make and git
|
||||||
|
- kubectl and kind only for the end-to-end example
|
||||||
|
|
||||||
|
## Local workflow
|
||||||
|
|
||||||
|
git switch -c feat/<short-slug>
|
||||||
|
make check # versions, example layout, manifest YAML
|
||||||
|
make test # gateway unit tests
|
||||||
|
make lint # ruff, shellcheck (+ yamllint/actionlint when installed)
|
||||||
|
make image # build the main DSH Web image
|
||||||
|
make gateway # build the example gateway image
|
||||||
|
make e2e # full kind walkthrough (optional, slow)
|
||||||
|
|
||||||
|
Install the gateway dev dependencies once:
|
||||||
|
|
||||||
|
python -m pip install -r examples/dsh-agents-as-a-service/gateway/requirements-dev.txt
|
||||||
|
|
||||||
|
## Pull requests
|
||||||
|
|
||||||
|
- Use a Conventional Commit title, for example feat(gateway): add wake metrics
|
||||||
|
or docs(examples): clarify SSO trust boundary. PR titles are linted.
|
||||||
|
- Keep the diff scoped. Squash merge into main; history stays linear.
|
||||||
|
- Required checks: lint, python, manifests, gateway-image, pr-title, CodeQL,
|
||||||
|
and dependency-review. Branch protection is described in docs/gitflow.md.
|
||||||
|
- Update the docs that the change affects; docs are part of the change, not a
|
||||||
|
follow-up.
|
||||||
|
- Never commit a real secret. Secret-shaped YAML belongs in a *.yaml.example
|
||||||
|
file, and versions live in versions.env only.
|
||||||
|
|
||||||
|
## Changing versions
|
||||||
|
|
||||||
|
make set-version DSH_VERSION=dsh-vX.Y.Z RELEASE_REVISION=r2
|
||||||
|
make check
|
||||||
|
|
||||||
|
CHANGELOG.md is generated by git-cliff from Conventional Commits. Do not edit
|
||||||
|
it by hand; the release workflow regenerates it.
|
||||||
|
|
||||||
|
## Adding an example
|
||||||
|
|
||||||
|
Read docs/examples.md, then add examples/<name>/ with:
|
||||||
|
|
||||||
|
- README.md (what it demonstrates, prerequisites, steps, teardown)
|
||||||
|
- manifests/NN-*.yaml (numbered, plain kubectl)
|
||||||
|
- optional manifests/exposure/<variant>/ with its own README.md
|
||||||
|
- optional components (a Dockerfile plus README and tests per component)
|
||||||
|
- scripts/ with executable, idempotent, self-cleaning scripts
|
||||||
|
- docs/ for deeper material
|
||||||
|
|
||||||
|
List it in examples/README.md. scripts/check-examples.sh enforces this layout.
|
||||||
|
|
||||||
|
## Documentation
|
||||||
|
|
||||||
|
- Root README.md is the front door: main image first, then examples.
|
||||||
|
- docs/ holds repository-wide material; example-specific material belongs in
|
||||||
|
the example directory.
|
||||||
|
|
||||||
|
## Branches and releases
|
||||||
|
|
||||||
|
See docs/gitflow.md for the branch model and docs/versioning.md for the
|
||||||
|
release identity, channels and rollback rules.
|
||||||
|
|
||||||
|
## Security
|
||||||
|
|
||||||
|
See SECURITY.md. Report vulnerabilities privately through GitHub Security
|
||||||
|
Advisories; do not open a public issue.
|
||||||
59
Dockerfile
59
Dockerfile
@@ -1,26 +1,37 @@
|
|||||||
# syntax=docker/dockerfile:1.7
|
# syntax=docker/dockerfile:1.7
|
||||||
|
#
|
||||||
|
# The primary artifact of this repository: DeepSeek Harness Web (dsh web),
|
||||||
|
# packaged as a slim, non-root, multi-arch runtime image.
|
||||||
|
#
|
||||||
|
# Build args (all pinned in versions.env, verified by scripts/check-versions.sh):
|
||||||
|
# DSH_REPO upstream git remote to package
|
||||||
|
# DSH_VERSION upstream git ref to build (tag or branch)
|
||||||
|
# RELEASE_REVISION optional repository revision label (r2, r3, ...)
|
||||||
|
# PNPM_VERSION pnpm used to build DSH
|
||||||
|
# VCS_REF this repository's commit, recorded in OCI labels
|
||||||
|
# NODE_BUILD_IMAGE node image used to build DSH
|
||||||
|
# NODE_RUNTIME_IMAGE node image used at runtime
|
||||||
ARG NODE_BUILD_IMAGE=docker.io/node:24-trixie
|
ARG NODE_BUILD_IMAGE=docker.io/node:24-trixie
|
||||||
ARG NODE_RUNTIME_IMAGE=docker.io/node:24-trixie-slim
|
ARG NODE_RUNTIME_IMAGE=docker.io/node:24-trixie-slim
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Build stage
|
# Build stage: clone DSH at the pinned ref, build, and package the CLI.
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
FROM ${NODE_BUILD_IMAGE} AS build
|
FROM $NODE_BUILD_IMAGE AS build
|
||||||
|
|
||||||
ARG DSH_REPO=https://github.com/deepseek-ai/deepseek-harness.git
|
ARG DSH_REPO=https://github.com/deepseek-ai/deepseek-harness.git
|
||||||
ARG DSH_VERSION=dsh-v0.2.1-rc.2
|
ARG DSH_VERSION=dsh-v0.2.1-rc.3
|
||||||
ARG PNPM_VERSION=11.28.5
|
ARG PNPM_VERSION=11.28.5
|
||||||
|
|
||||||
ENV CI=true \
|
ENV CI=true \
|
||||||
NODE_OPTIONS=--max-old-space-size=4096
|
NODE_OPTIONS=--max-old-space-size=4096
|
||||||
|
|
||||||
RUN corepack enable \
|
RUN corepack enable \
|
||||||
&& corepack prepare "pnpm@${PNPM_VERSION}" --activate
|
&& corepack prepare "pnpm@$PNPM_VERSION" --activate
|
||||||
|
|
||||||
WORKDIR /src
|
WORKDIR /src
|
||||||
|
|
||||||
RUN git clone --depth 1 --branch "${DSH_VERSION}" "${DSH_REPO}" .
|
RUN git clone --depth 1 --branch "$DSH_VERSION" "$DSH_REPO" .
|
||||||
|
|
||||||
RUN pnpm install --frozen-lockfile
|
RUN pnpm install --frozen-lockfile
|
||||||
|
|
||||||
@@ -44,29 +55,47 @@ RUN cd /opt/dsh \
|
|||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Runtime stage
|
# Runtime stage
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
FROM ${NODE_RUNTIME_IMAGE} AS runtime
|
FROM $NODE_RUNTIME_IMAGE AS runtime
|
||||||
|
|
||||||
ARG DSH_VERSION
|
ARG DSH_VERSION=dsh-v0.2.1-rc.3
|
||||||
|
ARG RELEASE_REVISION=r4
|
||||||
|
ARG PNPM_VERSION=11.28.5
|
||||||
|
ARG VCS_REF=unknown
|
||||||
|
|
||||||
LABEL org.opencontainers.image.title="DeepSeek Harness Web packaged by SREDevOps.org" \
|
LABEL org.opencontainers.image.title="DeepSeek Harness Web packaged by SREDevOps.org" \
|
||||||
org.opencontainers.image.version="${DSH_VERSION}" \
|
org.opencontainers.image.description="DeepSeek Harness Web (dsh web) as a slim, non-root, Kubernetes-ready image" \
|
||||||
org.opencontainers.image.source="https://github.com/sredevopsorg/deepseek-kubernetes-sandbox"
|
org.opencontainers.image.source="https://github.com/sredevopsorg/deepseek-kubernetes-sandbox" \
|
||||||
|
org.opencontainers.image.version="$DSH_VERSION" \
|
||||||
|
org.opencontainers.image.revision="$VCS_REF" \
|
||||||
|
org.opencontainers.image.licenses="MIT" \
|
||||||
|
io.sredevopsorg.dsh.version="$DSH_VERSION" \
|
||||||
|
io.sredevopsorg.release.revision="$RELEASE_REVISION"
|
||||||
|
|
||||||
ENV NODE_ENV=production \
|
ENV NODE_ENV=production \
|
||||||
HOME=/home/node \
|
HOME=/home/node \
|
||||||
DSH_HOME=/home/node/.dsh \
|
DSH_HOME=/home/node/.dsh \
|
||||||
DSH_PORT=3080 \
|
DSH_PORT=3080 \
|
||||||
PATH="/app/node_modules/.bin:${PATH}"
|
PATH="/app/node_modules/.bin:$PATH"
|
||||||
|
|
||||||
# git/ripgrep are useful for DSH agent tools; tini handles signals.
|
# git/ripgrep/jq/yq/curl/gh are deliberate agent tooling parity: a sandboxed DSH
|
||||||
|
# instance expects the usual developer CLI. tini reaps signals for a clean stop.
|
||||||
RUN apt-get update \
|
RUN apt-get update \
|
||||||
&& apt-get install -y --no-install-recommends \
|
&& apt-get install -y --no-install-recommends \
|
||||||
ca-certificates \
|
ca-certificates \
|
||||||
git \
|
git \
|
||||||
ripgrep \
|
ripgrep \
|
||||||
|
jq \
|
||||||
|
yq \
|
||||||
|
curl \
|
||||||
|
wget \
|
||||||
tini \
|
tini \
|
||||||
|
gh \
|
||||||
&& rm -rf /var/lib/apt/lists/* \
|
&& rm -rf /var/lib/apt/lists/* \
|
||||||
&& install -d -o node -g node -m 0755 /workspace "${DSH_HOME}"
|
&& curl -LsSf https://astral.sh/uv/install.sh | sh \
|
||||||
|
&& install -d -o node -g node -m 0755 /workspace "$DSH_HOME" \
|
||||||
|
&& corepack enable \
|
||||||
|
&& corepack prepare "pnpm@$PNPM_VERSION" --activate \
|
||||||
|
&& pnpm --version
|
||||||
|
|
||||||
COPY --from=build --chown=node:node /opt/dsh /app
|
COPY --from=build --chown=node:node /opt/dsh /app
|
||||||
|
|
||||||
@@ -88,7 +117,7 @@ if [ -n "$DSH_PUBLIC_URL" ]; then
|
|||||||
set -- --public-url "$DSH_PUBLIC_URL" "$@"
|
set -- --public-url "$DSH_PUBLIC_URL" "$@"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
# Additional trusted hosts (e.g. the public gateway host behind an Ingress).
|
# Additional trusted hosts (e.g. the public gateway host behind a proxy).
|
||||||
# DSH Host/Origin-fences /api: without this, requests forwarded with the
|
# DSH Host/Origin-fences /api: without this, requests forwarded with the
|
||||||
# browser's Host header are rejected. Accepts comma- or space-separated names;
|
# browser's Host header are rejected. Accepts comma- or space-separated names;
|
||||||
# localhost/127.0.0.1 are always trusted below.
|
# localhost/127.0.0.1 are always trusted below.
|
||||||
@@ -117,4 +146,4 @@ USER node
|
|||||||
EXPOSE 3080
|
EXPOSE 3080
|
||||||
VOLUME ["/home/node/.dsh"]
|
VOLUME ["/home/node/.dsh"]
|
||||||
STOPSIGNAL SIGTERM
|
STOPSIGNAL SIGTERM
|
||||||
ENTRYPOINT ["/usr/bin/tini", "--", "/usr/local/bin/dsh-entrypoint"]
|
ENTRYPOINT ["/usr/bin/tini", "--", "/usr/local/bin/dsh-entrypoint"]
|
||||||
|
|||||||
21
LICENSE
Normal file
21
LICENSE
Normal file
@@ -0,0 +1,21 @@
|
|||||||
|
MIT License
|
||||||
|
|
||||||
|
Copyright (c) 2026 Nicolás Georger
|
||||||
|
|
||||||
|
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||||
|
of this software and associated documentation files (the "Software"), to deal
|
||||||
|
in the Software without restriction, including without limitation the rights
|
||||||
|
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||||
|
copies of the Software, and to permit persons to whom the Software is
|
||||||
|
furnished to do so, subject to the following conditions:
|
||||||
|
|
||||||
|
The above copyright notice and this permission notice shall be included in all
|
||||||
|
copies or substantial portions of the Software.
|
||||||
|
|
||||||
|
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||||
|
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||||
|
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||||
|
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||||
|
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||||
|
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||||
|
SOFTWARE.
|
||||||
85
Makefile
85
Makefile
@@ -1,37 +1,82 @@
|
|||||||
# Local build and test helpers. Every variable can be overridden, e.g.
|
# Canonical local entrypoint. Includes versions.env so there is exactly one
|
||||||
# make image DSH_VERSION=dsh-v0.2.1-rc.2 IMAGE=dsh-web:dev
|
# place to bump a version (run "make check" to verify pins across the repo).
|
||||||
# make test
|
#
|
||||||
DSH_VERSION ?= dsh-v0.2.1-rc.2
|
# make image build the main DSH Web image
|
||||||
PNPM_VERSION ?= 11.28.5
|
# make gateway build the example gateway image
|
||||||
IMAGE ?= dsh-web:0.2.1-rc.2
|
# make test gateway unit tests
|
||||||
GATEWAY_IMAGE ?= ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway
|
# make lint ruff + shellcheck (+ yamllint/actionlint when installed)
|
||||||
GATEWAY_DIR := examples/dsh-agents-as-a-service/gateway
|
# make check versions, example layout and manifest YAML checks
|
||||||
PYTHON ?= python3
|
# make e2e full example walkthrough on a kind cluster
|
||||||
DSH_PORT ?= 3080
|
# make changelog regenerate CHANGELOG.md with git-cliff
|
||||||
|
include versions.env
|
||||||
|
|
||||||
.PHONY: help image gateway run test lint
|
RELEASE_IDENTITY := $(DSH_VERSION)$(if $(RELEASE_REVISION),-$(RELEASE_REVISION))
|
||||||
|
IMAGE ?= $(GHCR_IMAGE):$(RELEASE_IDENTITY)
|
||||||
|
GATEWAY_LOCAL_IMAGE ?= dsh-aaas-gateway:dev
|
||||||
|
GATEWAY_DIR := examples/dsh-agents-as-a-service/gateway
|
||||||
|
EXAMPLE_DIR := examples/dsh-agents-as-a-service
|
||||||
|
PYTHON ?= $(if $(wildcard $(CURDIR)/.venv/bin/python),$(CURDIR)/.venv/bin/python,python3)
|
||||||
|
DSH_PORT ?= 3080
|
||||||
|
VCS_REF ?= $(shell git rev-parse --short HEAD 2>/dev/null || echo unknown)
|
||||||
|
|
||||||
|
.PHONY: help image run gateway test cov lint check e2e changelog release-dry-run set-version clean
|
||||||
|
|
||||||
help: ## Show available targets
|
help: ## Show available targets
|
||||||
@grep -E '^[a-z-]+:.*?## ' $(MAKEFILE_LIST) | sed 's/:.*## /\t/'
|
@grep -E '^[a-z-]+:.*?## ' $(MAKEFILE_LIST) | sed 's/:.*## / /'
|
||||||
|
|
||||||
image: ## Build the DSH Web image
|
image: ## Build the main DSH Web image (no push)
|
||||||
docker build \
|
docker build \
|
||||||
--build-arg DSH_VERSION=$(DSH_VERSION) \
|
--build-arg DSH_VERSION=$(DSH_VERSION) \
|
||||||
|
--build-arg RELEASE_REVISION=$(RELEASE_REVISION) \
|
||||||
--build-arg PNPM_VERSION=$(PNPM_VERSION) \
|
--build-arg PNPM_VERSION=$(PNPM_VERSION) \
|
||||||
|
--build-arg VCS_REF=$(VCS_REF) \
|
||||||
-t $(IMAGE) .
|
-t $(IMAGE) .
|
||||||
|
|
||||||
gateway: ## Build the gateway image
|
run: image ## Build and run the main image (needs DEEPSEEK_API_KEY)
|
||||||
docker build -t $(GATEWAY_IMAGE) $(GATEWAY_DIR)/
|
|
||||||
|
|
||||||
run: image ## Build and run the DSH Web image
|
|
||||||
docker run --rm -it \
|
docker run --rm -it \
|
||||||
-p $(DSH_PORT):3080 \
|
-p $(DSH_PORT):3080 \
|
||||||
-v dsh-home:/home/node/.dsh \
|
-v dsh-home:/home/node/.dsh \
|
||||||
-e DEEPSEEK_API_KEY \
|
-e DEEPSEEK_API_KEY \
|
||||||
$(IMAGE)
|
$(IMAGE)
|
||||||
|
|
||||||
test: ## Run the gateway unit tests (needs requirements + requirements-dev installed)
|
gateway: ## Build the example gateway image locally
|
||||||
|
docker build -t $(GATEWAY_LOCAL_IMAGE) $(GATEWAY_DIR)/
|
||||||
|
|
||||||
|
test: ## Gateway unit tests
|
||||||
cd $(GATEWAY_DIR) && $(PYTHON) -m pytest -q
|
cd $(GATEWAY_DIR) && $(PYTHON) -m pytest -q
|
||||||
|
|
||||||
lint: ## Lint the gateway with ruff
|
cov: ## Gateway unit tests with coverage (needs pytest-cov)
|
||||||
cd $(GATEWAY_DIR) && ruff check .
|
cd $(GATEWAY_DIR) && $(PYTHON) -m pytest -q --cov=. --cov-report=term-missing --cov-report=xml:coverage.xml
|
||||||
|
|
||||||
|
lint: ## Lint Python, shell, YAML and Dockerfiles (skips missing tools locally)
|
||||||
|
cd $(GATEWAY_DIR) && $(PYTHON) -m ruff check .
|
||||||
|
cd $(GATEWAY_DIR) && $(PYTHON) -m ruff format --check .
|
||||||
|
shellcheck build.sh scripts/*.sh $(EXAMPLE_DIR)/scripts/*.sh
|
||||||
|
@command -v yamllint >/dev/null 2>&1 && yamllint . || echo "yamllint not installed; CI enforces it"
|
||||||
|
@command -v actionlint >/dev/null 2>&1 && actionlint || echo "actionlint not installed; CI enforces it"
|
||||||
|
|
||||||
|
check: ## Verify version pins, example layout and manifest YAML
|
||||||
|
./scripts/check-versions.sh
|
||||||
|
./scripts/check-examples.sh
|
||||||
|
$(PYTHON) scripts/check-manifests.py
|
||||||
|
|
||||||
|
e2e: ## Run the full example walkthrough on the current kind cluster
|
||||||
|
$(EXAMPLE_DIR)/scripts/e2e-kind.sh
|
||||||
|
|
||||||
|
changelog: ## Regenerate CHANGELOG.md from Conventional Commits
|
||||||
|
@command -v git-cliff >/dev/null 2>&1 || { echo "install git-cliff: https://git-cliff.org" >&2; exit 1; }
|
||||||
|
git-cliff --output CHANGELOG.md
|
||||||
|
|
||||||
|
release-dry-run: check ## Validate the release identity and preview release notes
|
||||||
|
@echo "release identity: $(RELEASE_IDENTITY)"
|
||||||
|
@echo "main image: $(GHCR_IMAGE):$(RELEASE_IDENTITY)"
|
||||||
|
@echo "gateway image: $(GATEWAY_IMAGE):$(RELEASE_IDENTITY)"
|
||||||
|
@if git rev-parse -q --verify "refs/tags/$(RELEASE_IDENTITY)" >/dev/null; then echo "ERROR: tag already exists" >&2; exit 1; fi
|
||||||
|
@command -v git-cliff >/dev/null 2>&1 && git-cliff --unreleased --tag $(RELEASE_IDENTITY) || echo "git-cliff not installed; skipping notes preview"
|
||||||
|
|
||||||
|
set-version: ## Bump pins: make set-version DSH_VERSION=dsh-vX [RELEASE_REVISION=rN]
|
||||||
|
@test -n "$(DSH_VERSION)" || { echo "DSH_VERSION is required" >&2; exit 1; }
|
||||||
|
./scripts/set-version.sh "$(DSH_VERSION)" "$(RELEASE_REVISION)"
|
||||||
|
|
||||||
|
clean: ## Remove local build/test caches
|
||||||
|
rm -rf .pytest_cache .ruff_cache $(GATEWAY_DIR)/.pytest_cache $(GATEWAY_DIR)/.ruff_cache $(GATEWAY_DIR)/__pycache__
|
||||||
|
|||||||
343
README.md
343
README.md
@@ -1,180 +1,233 @@
|
|||||||
# deepseek-kubernetes-sandbox
|
# deepseek-kubernetes-sandbox
|
||||||
|
|
||||||
Packages [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) Web (`dsh web`) into
|
**DeepSeek Harness Web, packaged as a container.** This repository's primary
|
||||||
a multi-arch, Kubernetes-ready container image, and ships the
|
artifact is a slim, non-root, multi-arch image of
|
||||||
[agents-as-a-service](examples/dsh-agents-as-a-service/) platform pattern — running a personal
|
[DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) Web
|
||||||
DSH Web instance per user on [kubernetes-sigs/agent-sandbox](https://github.com/kubernetes-sigs/agent-sandbox)
|
(`dsh web`), ready for Docker and Kubernetes. Everything else exists to support
|
||||||
behind a small browser-usable gateway.
|
that image or to demonstrate an implementation on top of it.
|
||||||
|
|
||||||
## What's in this repo
|
The flagship implementation is
|
||||||
|
[`examples/dsh-agents-as-a-service/`](examples/dsh-agents-as-a-service/): the
|
||||||
|
"agents-as-a-service" platform pattern with one DSH Web instance per user,
|
||||||
|
provisioned from a warm pool on
|
||||||
|
[kubernetes-sigs/agent-sandbox](https://github.com/kubernetes-sigs/agent-sandbox),
|
||||||
|
suspended when idle, and fronted by a small browser-usable gateway.
|
||||||
|
|
||||||
| Path | Purpose |
|
## Quick start
|
||||||
| --- | --- |
|
|
||||||
| [`Dockerfile`](Dockerfile) | Multi-stage build that clones DSH, `pnpm pack`s the CLI, and produces a slim runtime image (`:3080`) |
|
|
||||||
| [`build.sh`](build.sh), [`Makefile`](Makefile) | Local image builds: DSH Web and gateway, plus test/lint targets |
|
|
||||||
| [`examples/dsh-agents-as-a-service/`](examples/dsh-agents-as-a-service/) | The full multi-tenant platform pattern (warm pool, per-user claims, gateway, smoke test) |
|
|
||||||
| [`examples/dsh-agents-as-a-service/gateway/`](examples/dsh-agents-as-a-service/gateway/) | Starlette ASGI gateway: signup, cookie sessions, `/u/{user}/**` HTTP + WebSocket proxy, idle sweeper |
|
|
||||||
| [`deploy/k8s/sandbox-template.yaml`](deploy/k8s/sandbox-template.yaml) | Minimal single-tenant `SandboxTemplate` |
|
|
||||||
| [`docs/architecture.md`](docs/architecture.md) | Container and sequence diagrams, design decisions, security notes |
|
|
||||||
| [`.github/workflows/`](.github/workflows/) | CI: manifests dry-run, gateway tests/lint, multi-arch image builds |
|
|
||||||
|
|
||||||
## Published images
|
|
||||||
|
|
||||||
Everything is published to GitHub Container Registry as
|
|
||||||
`ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox`:
|
|
||||||
|
|
||||||
| Tag | Contents | Port |
|
|
||||||
| --- | --- | --- |
|
|
||||||
| `dsh-v0.2.1-rc.2` (also `main`, `latest`) | DSH Web image | `3080` |
|
|
||||||
| `gateway` | Starlette gateway image | `8080` |
|
|
||||||
|
|
||||||
The DSH Web tag matches the upstream DSH git tag it was built from.
|
|
||||||
|
|
||||||
## Quick start (DSH Web image)
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker run -d --name dsh-web \
|
docker run -d --name dsh-web \
|
||||||
-p 3080:3080 \
|
-p 3080:3080 \
|
||||||
-v dsh-home:/home/node/.dsh \
|
-v dsh-home:/home/node/.dsh \
|
||||||
-e DEEPSEEK_API_KEY \
|
-e DEEPSEEK_API_KEY \
|
||||||
ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.2
|
ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.3-r4
|
||||||
```
|
```
|
||||||
|
|
||||||
**DSH Web is not usable at a bare `/`.** Without its launch-token cookie it answers `401`, and
|
DSH Web is **not usable at a bare `/`**. Without its launch-token cookie it
|
||||||
the token is only printed in the startup logs:
|
answers `401`, and the token is printed only in the startup logs:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker logs dsh-web 2>&1 | grep -m1 '?token='
|
docker logs dsh-web 2>&1 | grep -m1 '?token='
|
||||||
# dsh web: http://localhost:3080/?token=...
|
# dsh web: http://localhost:3080/?token=...
|
||||||
```
|
```
|
||||||
|
|
||||||
Open the printed `?token=...` URL once — DSH sets its auth cookie and the UI loads normally
|
Open the printed `?token=...` URL once: DSH sets its auth cookie and the UI loads
|
||||||
afterwards. Hitting `/` directly (or after the cookie expires) returns `401` by design; that is
|
normally afterwards. Hitting `/` directly (or after the cookie expires) returns
|
||||||
why Kubernetes probes use `/favicon.svg` (see below) and why the gateway performs this
|
`401` by design.
|
||||||
handshake for the browser.
|
|
||||||
|
|
||||||
> **Different host port?** If `3080` is already taken on your workstation, map another port and
|
### Host port and public URL
|
||||||
> set the public URL so the printed token URL matches: `-p 18081:3080 -e
|
|
||||||
> DSH_PUBLIC_URL=http://localhost:18081`, then open `http://localhost:18081/?token=...`.
|
|
||||||
|
|
||||||
## Quick start (Kubernetes)
|
If `3080` is taken locally, map another port and set the public URL so the quoted
|
||||||
|
token URL matches:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
kubectl create namespace dsh-demo --dry-run=client -o yaml | kubectl apply -f -
|
docker run -d --name dsh-web -p 18081:3080 \
|
||||||
kubectl -n dsh-demo create secret generic dsh-provider-keys \
|
-e DSH_PUBLIC_URL=http://localhost:18081 \
|
||||||
--from-literal=DEEPSEEK_API_KEY="sk-..."
|
-v dsh-home:/home/node/.dsh \
|
||||||
kubectl apply -f deploy/k8s/sandbox-template.yaml
|
-e DEEPSEEK_API_KEY \
|
||||||
|
ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.3-r4
|
||||||
# A template needs a warm pool and a claim to become a running Sandbox.
|
# then open http://localhost:18081/?token=...
|
||||||
kubectl apply -f - <<'EOF'
|
|
||||||
apiVersion: extensions.agents.x-k8s.io/v1beta1
|
|
||||||
kind: SandboxWarmPool
|
|
||||||
metadata:
|
|
||||||
name: dsh-pool
|
|
||||||
namespace: dsh-demo
|
|
||||||
spec:
|
|
||||||
replicas: 1
|
|
||||||
sandboxTemplateRef:
|
|
||||||
name: dsh-web-template
|
|
||||||
---
|
|
||||||
apiVersion: extensions.agents.x-k8s.io/v1beta1
|
|
||||||
kind: SandboxClaim
|
|
||||||
metadata:
|
|
||||||
name: dsh
|
|
||||||
namespace: dsh-demo
|
|
||||||
spec:
|
|
||||||
warmPoolRef:
|
|
||||||
name: dsh-pool
|
|
||||||
EOF
|
|
||||||
kubectl -n dsh-demo wait sandboxclaim/dsh --for=condition=Ready --timeout=180s
|
|
||||||
```
|
```
|
||||||
|
|
||||||
Then reach the instance through its launch-token URL:
|
### Hosts, probes and the 401
|
||||||
|
|
||||||
|
- **Probes must not hit `/`.** It returns `401` without the launch token, so a
|
||||||
|
readiness/liveness probe on `/` crash-loops the pod. Probe `/favicon.svg`
|
||||||
|
instead: it is unauthenticated. The shipped manifests use it.
|
||||||
|
- **Host allow-listing.** DSH rejects unknown `Host` headers. `localhost` and
|
||||||
|
`127.0.0.1` are always trusted; add any other public hostname to
|
||||||
|
`DSH_TRUSTED_HOSTS` (comma or space separated). A gateway or Ingress that
|
||||||
|
forwards the browser's `Host` needs its hostname listed there.
|
||||||
|
- **State lives in `/home/node/.dsh`.** Mount a volume (Docker) or PVC
|
||||||
|
(Kubernetes) to keep conversations, sessions and config across restarts.
|
||||||
|
|
||||||
|
### Runtime environment
|
||||||
|
|
||||||
|
| Variable | Default | Purpose |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `DEEPSEEK_API_KEY` | empty | Provider credential consumed by DSH. Inject it; never bake it into the image or commit it. |
|
||||||
|
| `DSH_PORT` | `3080` | Port DSH Web listens on inside the container. |
|
||||||
|
| `DSH_HOST` | auto-detected pod/container IP | Bind address. Leave empty in Kubernetes. |
|
||||||
|
| `DSH_PUBLIC_URL` | `http://localhost:${DSH_PORT}` | Public base URL used in generated links and the launch-token URL in the logs. |
|
||||||
|
| `DSH_TRUSTED_HOSTS` | empty | Extra trusted `Host`/`Origin` entries, comma/space separated. |
|
||||||
|
| `DSH_HOME` | `/home/node/.dsh` | DSH state directory (mount a volume here). |
|
||||||
|
|
||||||
|
DSH also resolves the standard per-provider environment names —
|
||||||
|
`OPENROUTER_API_KEY`, `GEMINI_API_KEY`, `NVIDIA_API_KEY`, `OPENAI_API_KEY`,
|
||||||
|
`ANTHROPIC_API_KEY`, `GROQ_API_KEY`, `XAI_API_KEY`, `HF_TOKEN`, and others —
|
||||||
|
through its packaged provider catalog. Inject them the same way; a key saved in
|
||||||
|
DSH's credential store wins over the environment. See
|
||||||
|
[provider credentials](docs/main-image.md#provider-credentials).
|
||||||
|
|
||||||
|
Full contract, build args, probes, volumes and signals:
|
||||||
|
[docs/main-image.md](docs/main-image.md).
|
||||||
|
|
||||||
|
## Published images
|
||||||
|
|
||||||
|
Published to GitHub Container Registry. The main image is the repository image;
|
||||||
|
example components are nested packages below it:
|
||||||
|
|
||||||
|
```
|
||||||
|
ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox # DSH Web (main image)
|
||||||
|
ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway # example gateway
|
||||||
|
```
|
||||||
|
|
||||||
|
| Tag | Contents | Meaning |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `dsh-v0.2.1-rc.3-r4` | DSH Web, port `3080` | **Current release identity.** Immutable: once published it never moves. |
|
||||||
|
| `latest` | DSH Web | Most recent release; same digest as the current identity. |
|
||||||
|
| `main` | DSH Web | Development channel rebuilt from every push to `main`. Not a release; not for production. |
|
||||||
|
| `dsh-v0.2.1-rc.3` | DSH Web | **Legacy tag** published before the release-identity scheme. It is not updated. New deployments use `dsh-v0.2.1-rc.3-r4`. |
|
||||||
|
|
||||||
|
The example gateway is published under its nested path with the same release
|
||||||
|
identity, e.g.
|
||||||
|
`ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway:dsh-v0.2.1-rc.3-r4`.
|
||||||
|
|
||||||
|
Release identities, channels and rollback rules are explained in
|
||||||
|
[docs/versioning.md](docs/versioning.md). Production deployments should pin a
|
||||||
|
digest:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
SB=$(kubectl -n dsh-demo get sandboxclaim dsh -o jsonpath='{.status.sandbox.name}')
|
docker buildx imagetools inspect ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.3-r4
|
||||||
kubectl -n dsh-demo logs "$SB" | grep -m1 '?token='
|
# image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox@sha256:<digest>
|
||||||
kubectl -n dsh-demo port-forward "pod/$SB" 18081:3080
|
|
||||||
# open http://localhost:18081/?token=... from the log line
|
|
||||||
```
|
```
|
||||||
|
|
||||||
Two things bite first-time setups:
|
## Build from source
|
||||||
|
|
||||||
- **Probes must not hit `/`.** DSH returns `401` there without the launch token, so a
|
|
||||||
readiness/liveness probe on `/` crash-loops the pod. The shipped templates use
|
|
||||||
`/favicon.svg`, which is unauthenticated; keep that path in custom templates.
|
|
||||||
- **Host allow-listing.** DSH rejects unknown `Host` headers. `localhost` and `127.0.0.1` are
|
|
||||||
always trusted; for any other hostname add it to the template's `DSH_TRUSTED_HOSTS` (comma or
|
|
||||||
space separated). The gateway forwards the browser's `Host`, so the public gateway hostname
|
|
||||||
must be listed there.
|
|
||||||
|
|
||||||
## The multi-user platform pattern
|
|
||||||
|
|
||||||
`examples/dsh-agents-as-a-service/` is a port of the upstream
|
|
||||||
[hermes-agents-as-a-service](https://github.com/kubernetes-sigs/agent-sandbox/tree/main/examples/hermes-agents-as-a-service)
|
|
||||||
example to DSH Web. It demonstrates the platform shape — one `SandboxTemplate`, one warm pool,
|
|
||||||
one tiny `SandboxClaim` per user, and `operatingMode` as the cost dial — plus a gateway that
|
|
||||||
turns it into a usable service.
|
|
||||||
|
|
||||||
**Browser flow:** open the gateway landing page, sign up, and get an `HttpOnly` session cookie.
|
|
||||||
The browser then browses `http://<gateway>/u/<user>/`, which proxies HTTP **and WebSocket** to
|
|
||||||
that user's DSH Web. The gateway reads the DSH launch token from the pod logs, performs the
|
|
||||||
token handshake server-side, and injects the resulting DSH cookie on every proxied request — the
|
|
||||||
browser never sees the token. Suspended instances are woken transparently on the next request;
|
|
||||||
idle ones are suspended again after `IDLE_TIMEOUT`.
|
|
||||||
|
|
||||||
See the [example README](examples/dsh-agents-as-a-service/README.md) for the full walkthrough
|
|
||||||
and [docs/architecture.md](docs/architecture.md) for the diagrams and trade-offs.
|
|
||||||
|
|
||||||
### Mechanism mapping: Hermes → DSH
|
|
||||||
|
|
||||||
| Hermes example mechanism | Role | Adaptation for DSH |
|
|
||||||
| :--- | :--- | :--- |
|
|
||||||
| `SandboxTemplate` | Reusable Pod blueprint | Image `dsh-web`, port `3080`, probes on `/favicon.svg` |
|
|
||||||
| `volumeClaimTemplates` | Auto-create a PVC per user | Mount `/home/node/.dsh` (DSH state) |
|
|
||||||
| `SandboxWarmPool` | Pre-warmed instances | Keep as-is; size for expected concurrency |
|
|
||||||
| `SandboxClaim` + `warmPoolRef` | "Registration" adopts a pooled instance | Keep exactly as-is; one claim per user |
|
|
||||||
| `service: true` | Stable DNS per Sandbox | Enabled, so the gateway routes to each user's DSH Web |
|
|
||||||
| `envVarsInjectionPolicy: Disallowed` | Reject claims that would cold-start | `DEEPSEEK_API_KEY` injected via `envFrom` on the template (shared platform key) |
|
|
||||||
| `operatingMode` | Cost switch: Running / Suspended | Reused directly — suspend idle instances to save cost |
|
|
||||||
|
|
||||||
### Key differences from Hermes
|
|
||||||
|
|
||||||
- **Multi-arch** — the DSH image is built for amd64 and arm64 (no amd64 nodeSelector).
|
|
||||||
- **Single port** — DSH Web serves everything on `3080`; no API/dashboard port split.
|
|
||||||
- **One secret** — only `DEEPSEEK_API_KEY`; no dashboard/session/api-server credentials.
|
|
||||||
- **Host forwarding** — the gateway forwards `Host` + `X-Forwarded-*` (and keeps
|
|
||||||
`DSH_PUBLIC_URL` empty) so DSH generates reachable links; add the public hostname to
|
|
||||||
`DSH_TRUSTED_HOSTS`.
|
|
||||||
- **Browser sessions + WebSocket** — the gateway is Starlette ASGI, so it proxies WebSocket
|
|
||||||
upgrades as well as HTTP/SSE, and authenticates browsers with an `HttpOnly` cookie instead of
|
|
||||||
exposing per-user bearer tokens in URLs.
|
|
||||||
|
|
||||||
## Build it yourself
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
./build.sh # build the DSH Web image, then run it interactively
|
make image # build the main DSH Web image (clones and builds upstream DSH)
|
||||||
./build.sh web # build the DSH Web image only
|
make run # build it, then run it interactively (needs DEEPSEEK_API_KEY)
|
||||||
./build.sh gateway # build the gateway image
|
|
||||||
|
|
||||||
make image # same as ./build.sh web
|
|
||||||
make gateway # gateway image
|
|
||||||
make test # gateway unit tests (pytest)
|
|
||||||
make lint # gateway lint (ruff)
|
|
||||||
```
|
```
|
||||||
|
|
||||||
Or directly:
|
The pins (`DSH_VERSION`, `RELEASE_REVISION`, `PNPM_VERSION`) come from
|
||||||
|
[versions.env](versions.env) — never hard-code them
|
||||||
|
(`make check` verifies). [`build.sh`](build.sh) is a deprecated shim that
|
||||||
|
forwards to `make`; use the Makefile targets.
|
||||||
|
|
||||||
```bash
|
## Examples
|
||||||
docker build \
|
|
||||||
--build-arg DSH_VERSION=dsh-v0.2.1-rc.2 \
|
|
||||||
--build-arg PNPM_VERSION=11.28.5 \
|
|
||||||
-t dsh-web:0.2.1-rc.2 .
|
|
||||||
|
|
||||||
docker build -t ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway \
|
Self-contained implementations that consume the image. Index and contract:
|
||||||
examples/dsh-agents-as-a-service/gateway/
|
[`examples/README.md`](examples/README.md) and
|
||||||
```
|
[docs/examples.md](docs/examples.md).
|
||||||
|
|
||||||
See [`Dockerfile`](Dockerfile) for build args (`DSH_REPO`, `DSH_VERSION`, `PNPM_VERSION`) and
|
| Example | What it demonstrates |
|
||||||
the runtime image contract (non-root `node` user, `DSH_HOME=/home/node/.dsh`, `EXPOSE 3080`).
|
| --- | --- |
|
||||||
`.env.sample` documents the runtime variables for both images.
|
| [`examples/dsh-agents-as-a-service/`](examples/dsh-agents-as-a-service/) | Full multi-tenant platform: SandboxTemplate with per-user PVCs, warm pool, one claim per user, suspend-when-idle, wake-on-return, browser gateway with signup + `HttpOnly` sessions + HTTP/WebSocket proxy, Ingress or Gateway API + GitHub SSO exposure, and opt-in gVisor/NetworkPolicy/provider-key hardening. Start with the [walkthrough](examples/dsh-agents-as-a-service/README.md) and the [architecture](examples/dsh-agents-as-a-service/docs/architecture.md). |
|
||||||
|
| [`examples/minimal-single-tenant/`](examples/minimal-single-tenant/) | The smallest useful setup: one `SandboxTemplate` and one `SandboxClaim`, reached through `kubectl port-forward` + the launch-token URL. Manifest: [`manifests/sandbox-template.yaml`](examples/minimal-single-tenant/manifests/sandbox-template.yaml). |
|
||||||
|
|
||||||
|
The gateway is a component of the multi-tenant example and has its own
|
||||||
|
[README](examples/dsh-agents-as-a-service/gateway/README.md) (API, configuration,
|
||||||
|
auth/SSO, limitations). The example also documents opt-in gVisor,
|
||||||
|
`nodeSelector`, NetworkPolicy and extra provider-key hardening in
|
||||||
|
[optional hardening](examples/dsh-agents-as-a-service/docs/optional-hardening.md).
|
||||||
|
|
||||||
|
## Agent skills
|
||||||
|
|
||||||
|
DSH discovers project skills from `.dsh/skills/<name>/SKILL.md`. The repository
|
||||||
|
ships [`dsh-agent-sandbox-platform`](.dsh/skills/dsh-agent-sandbox-platform/SKILL.md),
|
||||||
|
which covers operating and extending the platform, the optional
|
||||||
|
[agent-sandbox MCP server](https://github.com/kubernetes-sigs/agent-sandbox/blob/main/clients/integrations/mcp-server/README.md),
|
||||||
|
and a read-only cluster preflight. Runbooks, the MCP reference, the extension
|
||||||
|
guide and troubleshooting notes live beside `SKILL.md`.
|
||||||
|
|
||||||
|
## Repository layout
|
||||||
|
|
||||||
|
| Path | Purpose |
|
||||||
|
| --- | --- |
|
||||||
|
| [`Dockerfile`](Dockerfile) | Primary artifact: the multi-stage DSH Web image (port `3080`). |
|
||||||
|
| [`versions.env`](versions.env) | Single source of truth for the DSH ref, repository revision, pnpm/control-plane versions and image names. |
|
||||||
|
| [`Makefile`](Makefile), [`build.sh`](build.sh) | Canonical local entrypoint; `build.sh` is a deprecated shim. |
|
||||||
|
| [`docs/`](docs/README.md) | Repository docs: image contract, example contract, gitflow, versioning. |
|
||||||
|
| [`examples/`](examples/README.md) | One self-contained implementation per directory. |
|
||||||
|
| [`examples/dsh-agents-as-a-service/`](examples/dsh-agents-as-a-service/) | Multi-tenant platform example: `manifests/`, `gateway/`, `docs/`, `scripts/`. |
|
||||||
|
| [`examples/minimal-single-tenant/`](examples/minimal-single-tenant/) | Single-sandbox quickstart. |
|
||||||
|
| [`scripts/`](scripts/) | Repository checks: versions, example layout, manifest YAML, version bump. |
|
||||||
|
| [`.dsh/skills/`](.dsh/skills/) | Project-local DSH skills, auto-discovered from the git root. |
|
||||||
|
| [`.github/`](.github/) | CI, release automation, dependency updates, PR/issue templates. |
|
||||||
|
| [`.env.sample`](.env.sample) | Documented local configuration (copy to `.env`; never commit it). |
|
||||||
|
|
||||||
|
## Development
|
||||||
|
|
||||||
|
The Makefile is the canonical entrypoint. No cluster is needed for `check`,
|
||||||
|
`test`, `lint`, `gateway` or `image`:
|
||||||
|
|
||||||
|
| Target | What it does |
|
||||||
|
| --- | --- |
|
||||||
|
| `make check` | Version pins, example layout and manifest YAML checks. Run before every commit. |
|
||||||
|
| `make test` | Gateway unit tests (`pytest`). |
|
||||||
|
| `make lint` | `ruff` + `shellcheck` (+ `yamllint`/`actionlint` when installed). |
|
||||||
|
| `make image` | Build the main DSH Web image. |
|
||||||
|
| `make gateway` | Build the example gateway image locally (`dsh-aaas-gateway:dev`). |
|
||||||
|
| `make e2e` | Full kind walkthrough of the multi-tenant example (needs a cluster with agent-sandbox). |
|
||||||
|
| `make changelog` | Regenerate `CHANGELOG.md` with git-cliff (release only). |
|
||||||
|
| `make set-version` | Bump pins in `versions.env`; e.g. `make set-version DSH_VERSION=<ref> RELEASE_REVISION=rN`. |
|
||||||
|
|
||||||
|
Local setup and the PR workflow: [CONTRIBUTING.md](CONTRIBUTING.md); branches,
|
||||||
|
required checks and releases: [docs/gitflow.md](docs/gitflow.md).
|
||||||
|
|
||||||
|
## Versioning and releases
|
||||||
|
|
||||||
|
The release identity is the upstream DSH ref plus an optional repository
|
||||||
|
revision: `DSH_VERSION` and `RELEASE_REVISION` in [versions.env](versions.env).
|
||||||
|
The current identity is **`dsh-v0.2.1-rc.3-r4`** — the first release under the
|
||||||
|
mirrored-DSH scheme at the same upstream DSH ref as the legacy
|
||||||
|
`dsh-v0.2.1-rc.3` tag. Tags are immutable; a broken release is fixed by
|
||||||
|
publishing `-r(N+1)`, never by re-tagging.
|
||||||
|
|
||||||
|
`CHANGELOG.md` is generated by [git-cliff](https://git-cliff.org) from
|
||||||
|
Conventional Commits and is never edited by hand. See
|
||||||
|
[`CHANGELOG.md`](CHANGELOG.md), [docs/versioning.md](docs/versioning.md) and
|
||||||
|
[cliff.toml](cliff.toml).
|
||||||
|
|
||||||
|
## Migrating from the previous layout
|
||||||
|
|
||||||
|
If you are coming from an older checkout, these paths moved. Nothing else changed
|
||||||
|
semantically; the content was split into the example it belongs to.
|
||||||
|
|
||||||
|
| Old path | New path |
|
||||||
|
| --- | --- |
|
||||||
|
| `gateway-api/dsh-demo/` (Gateway API + oauth2-proxy / GitHub SSO) | `examples/dsh-agents-as-a-service/manifests/exposure/gateway-api-sso/` (now `70-gateway.yaml` ... `74-tls-secret.yaml.example`) |
|
||||||
|
| `examples/dsh-agents-as-a-service/60-ingress.yaml` | `examples/dsh-agents-as-a-service/manifests/exposure/ingress/60-ingress.yaml` |
|
||||||
|
| `examples/dsh-agents-as-a-service/NN-*.yaml` (`00-` ... `50-`) | `examples/dsh-agents-as-a-service/manifests/NN-*.yaml` |
|
||||||
|
| `examples/dsh-agents-as-a-service/run-test-kind.sh` | `examples/dsh-agents-as-a-service/scripts/e2e-kind.sh` |
|
||||||
|
| `deploy/k8s/sandbox-template.yaml` | `examples/minimal-single-tenant/manifests/sandbox-template.yaml` |
|
||||||
|
| `docs/architecture.md` | `examples/dsh-agents-as-a-service/docs/architecture.md` |
|
||||||
|
|
||||||
|
The SSO walkthrough that lived in `gateway-api/dsh-demo/README.md` is merged
|
||||||
|
into the [multi-tenant example README](examples/dsh-agents-as-a-service/README.md)
|
||||||
|
(step 9) and the [exposure README](examples/dsh-agents-as-a-service/manifests/exposure/README.md).
|
||||||
|
|
||||||
|
## Contributing
|
||||||
|
|
||||||
|
- [CONTRIBUTING.md](CONTRIBUTING.md) — local setup, PR workflow, what a change needs.
|
||||||
|
- [AGENTS.md](AGENTS.md) — repository conventions for AI coding agents (and humans) and the CI commands.
|
||||||
|
- [SECURITY.md](SECURITY.md) — private vulnerability reporting and project threat notes.
|
||||||
|
- [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md) — community standards.
|
||||||
|
- [docs/gitflow.md](docs/gitflow.md) — branches, protected `main`, required checks, release process.
|
||||||
|
|
||||||
|
Do not commit secrets; secret-shaped YAML lives in `*.yaml.example` files, and
|
||||||
|
version pins change only through `versions.env` / `make set-version`.
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
[MIT](LICENSE) © Nicolás Georger.
|
||||||
|
|||||||
48
SECURITY.md
Normal file
48
SECURITY.md
Normal file
@@ -0,0 +1,48 @@
|
|||||||
|
# Security policy
|
||||||
|
|
||||||
|
## Supported versions
|
||||||
|
|
||||||
|
Only the latest published release is supported. Image tags are immutable;
|
||||||
|
fixes ship as a new release (see docs/versioning.md).
|
||||||
|
|
||||||
|
## Reporting a vulnerability
|
||||||
|
|
||||||
|
Report vulnerabilities privately through GitHub Security Advisories:
|
||||||
|
|
||||||
|
https://github.com/sredevopsorg/deepseek-kubernetes-sandbox/security/advisories/new
|
||||||
|
|
||||||
|
Do not open a public issue, and do not include live credentials in a report.
|
||||||
|
You can expect an acknowledgement within a few days and a coordinated
|
||||||
|
disclosure once a fix is available.
|
||||||
|
|
||||||
|
## Threat notes specific to this project
|
||||||
|
|
||||||
|
- **Launch token in pod logs.** The example gateway reads DSH's launch token
|
||||||
|
from pod logs (pods/log). Anyone with access to those logs can impersonate a
|
||||||
|
tenant. Keep the gateway's RBAC namespace-scoped and treat pod logs as
|
||||||
|
sensitive. Replacing log scraping with an upstream token API is tracked in
|
||||||
|
the example architecture document.
|
||||||
|
- **SSO header trust boundary.** AUTH_PROXY_HEADER makes the gateway trust an
|
||||||
|
identity header (for example X-Forwarded-User). Enable it only when the
|
||||||
|
gateway Service is reachable exclusively through the authenticating proxy;
|
||||||
|
otherwise any client can spoof the header. The SSO exposure variant ships a
|
||||||
|
NetworkPolicy that restricts gateway access to oauth2-proxy pods; a custom
|
||||||
|
exposure needs an equivalent.
|
||||||
|
- **Shared provider keys.** The example injects DEEPSEEK_API_KEY and any keys in
|
||||||
|
the optional dsh-provider-keys-extra Secret into every tenant sandbox (a
|
||||||
|
warm-pool constraint). Treat tenants as mutually trusting until per-tenant
|
||||||
|
credentials or an egress proxy exist.
|
||||||
|
- **Optional isolation is opt-in.** gVisor, nodeSelector, the sandbox/exposure
|
||||||
|
NetworkPolicies and extra provider keys are opt-in. Without a
|
||||||
|
NetworkPolicy-capable CNI or a gVisor runtime, the base example is not
|
||||||
|
tenant-isolated; see examples/dsh-agents-as-a-service/docs/optional-hardening.md.
|
||||||
|
- **TLS is external.** Terminate TLS at an Ingress or Gateway and set
|
||||||
|
COOKIE_SECURE=true. Plain HTTP outside localhost exposes session cookies.
|
||||||
|
- **No secrets in the repository.** The repository only ships empty Secret
|
||||||
|
manifests and *.yaml.example files.
|
||||||
|
|
||||||
|
## Dependency and supply chain
|
||||||
|
|
||||||
|
GitHub Actions are pinned by commit SHA. The CodeQL workflow scans Python and
|
||||||
|
Actions. The release workflow publishes images with SBOM and provenance
|
||||||
|
attestations.
|
||||||
69
build.sh
69
build.sh
@@ -1,68 +1,19 @@
|
|||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
# Build the DSH Web and/or gateway images locally.
|
# Deprecated compatibility shim: "make" is the canonical entrypoint.
|
||||||
#
|
#
|
||||||
# Usage:
|
# ./build.sh -> make run (build the main image, then run it)
|
||||||
# ./build.sh # build the DSH Web image, then run it interactively
|
# ./build.sh web -> make image
|
||||||
# ./build.sh web # build the DSH Web image only
|
# ./build.sh gateway -> make gateway
|
||||||
# ./build.sh gateway # build the gateway image (examples/.../gateway/)
|
|
||||||
#
|
#
|
||||||
# Overridable environment:
|
# Use the Makefile targets directly for everything else (test, lint, check, e2e).
|
||||||
# DSH_VERSION DSH git ref to build (default dsh-v0.2.1-rc.2)
|
|
||||||
# PNPM_VERSION pnpm version (default 11.28.5)
|
|
||||||
# IMAGE DSH Web image tag (default dsh-web:<version>)
|
|
||||||
# DSH_PORT host port for docker run (default 3080)
|
|
||||||
# GATEWAY_IMAGE gateway image tag (default ghcr.io/...:gateway)
|
|
||||||
# RUN 0 to build without running (default 1)
|
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
cd "$(dirname "$0")"
|
cd "$(dirname "$0")"
|
||||||
|
|
||||||
DSH_VERSION="${DSH_VERSION:-dsh-v0.2.1-rc.2}"
|
echo "build.sh is deprecated; use 'make image', 'make run' or 'make gateway'." >&2
|
||||||
PNPM_VERSION="${PNPM_VERSION:-11.28.5}"
|
|
||||||
IMAGE="${IMAGE:-dsh-web:${DSH_VERSION#dsh-v}}"
|
|
||||||
GATEWAY_IMAGE="${GATEWAY_IMAGE:-ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway}"
|
|
||||||
DSH_PORT="${DSH_PORT:-3080}"
|
|
||||||
RUN="${RUN:-1}"
|
|
||||||
GATEWAY_DIR=examples/dsh-agents-as-a-service/gateway
|
|
||||||
|
|
||||||
build_web() {
|
|
||||||
docker build \
|
|
||||||
--build-arg "DSH_VERSION=${DSH_VERSION}" \
|
|
||||||
--build-arg "PNPM_VERSION=${PNPM_VERSION}" \
|
|
||||||
-t "${IMAGE}" \
|
|
||||||
"$@" \
|
|
||||||
.
|
|
||||||
}
|
|
||||||
|
|
||||||
build_gateway() {
|
|
||||||
docker build -t "${GATEWAY_IMAGE}" "$@" "${GATEWAY_DIR}/"
|
|
||||||
}
|
|
||||||
|
|
||||||
usage() {
|
|
||||||
echo "usage: $0 [run|web|gateway] [extra docker build args...]" >&2
|
|
||||||
exit 2
|
|
||||||
}
|
|
||||||
|
|
||||||
case "${1:-run}" in
|
case "${1:-run}" in
|
||||||
run)
|
run) exec make run ;;
|
||||||
shift
|
web) exec make image ;;
|
||||||
build_web "$@"
|
gateway) exec make gateway ;;
|
||||||
if [ "${RUN}" = "1" ]; then
|
*) echo "usage: $0 [run|web|gateway]" >&2; exit 2 ;;
|
||||||
docker run --rm -it \
|
|
||||||
-p "${DSH_PORT}:3080" \
|
|
||||||
-v dsh-home:/home/node/.dsh \
|
|
||||||
-e DEEPSEEK_API_KEY \
|
|
||||||
"${IMAGE}"
|
|
||||||
fi
|
|
||||||
;;
|
|
||||||
web)
|
|
||||||
shift
|
|
||||||
build_web "$@"
|
|
||||||
;;
|
|
||||||
gateway)
|
|
||||||
shift
|
|
||||||
build_gateway "$@"
|
|
||||||
;;
|
|
||||||
*)
|
|
||||||
usage
|
|
||||||
;;
|
|
||||||
esac
|
esac
|
||||||
|
|||||||
49
cliff.toml
Normal file
49
cliff.toml
Normal file
@@ -0,0 +1,49 @@
|
|||||||
|
# git-cliff configuration: release notes and CHANGELOG.md are generated from
|
||||||
|
# Conventional Commits. See docs/versioning.md for the release procedure.
|
||||||
|
|
||||||
|
[changelog]
|
||||||
|
header = """
|
||||||
|
# Changelog
|
||||||
|
|
||||||
|
All notable changes to this repository are documented here.
|
||||||
|
Generated by git-cliff from Conventional Commits; do not edit this file by hand.
|
||||||
|
|
||||||
|
"""
|
||||||
|
body = """
|
||||||
|
{% if version %}\
|
||||||
|
## [{{ version }}] - {{ timestamp | date(format="%Y-%m-%d") }}
|
||||||
|
{% else %}\
|
||||||
|
## [Unreleased]
|
||||||
|
{% endif %}\
|
||||||
|
{% for group, commits in commits | group_by(attribute="group") %}
|
||||||
|
### {{ group | upper_first }}
|
||||||
|
{% for commit in commits %}
|
||||||
|
- {% if commit.scope %}**{{ commit.scope }}:** {% endif %}{{ commit.message | split(pat="\n") | first | upper_first }} ({{ commit.id | truncate(length=7, end="") }})
|
||||||
|
{%- endfor %}
|
||||||
|
{% endfor %}\n
|
||||||
|
"""
|
||||||
|
footer = ""
|
||||||
|
trim = true
|
||||||
|
|
||||||
|
[git]
|
||||||
|
conventional_commits = true
|
||||||
|
filter_unconventional = false
|
||||||
|
split_commits = false
|
||||||
|
commit_parsers = [
|
||||||
|
{ message = "^feat", group = "Added" },
|
||||||
|
{ message = "^fix", group = "Fixed" },
|
||||||
|
{ message = "^perf", group = "Performance" },
|
||||||
|
{ message = "^refactor", group = "Changed" },
|
||||||
|
{ message = "^docs", group = "Documentation" },
|
||||||
|
{ message = "^build", group = "Build" },
|
||||||
|
{ message = "^ci", group = "CI" },
|
||||||
|
{ message = "^test", group = "Tests" },
|
||||||
|
{ message = "^chore", group = "Maintenance" },
|
||||||
|
{ message = "^revert", group = "Reverted" },
|
||||||
|
{ message = ".*", group = "Other" },
|
||||||
|
]
|
||||||
|
protect_breaking_commits = false
|
||||||
|
filter_commits = false
|
||||||
|
# Release tags are dsh-vX.Y.Z[-rN]; the single legacy pre-scheme tag is v0.0.1.
|
||||||
|
tag_pattern = "^(v|dsh-v).*$"
|
||||||
|
sort_commits = "oldest"
|
||||||
47
docs/README.md
Normal file
47
docs/README.md
Normal file
@@ -0,0 +1,47 @@
|
|||||||
|
# Repository documentation
|
||||||
|
|
||||||
|
Repository-wide documentation lives here. Example-specific material lives with
|
||||||
|
its example under [`examples/<name>/docs/`](../examples/) — see the
|
||||||
|
[agents-as-a-service architecture](../examples/dsh-agents-as-a-service/docs/architecture.md)
|
||||||
|
for the deep-dive that used to live in this directory.
|
||||||
|
|
||||||
|
If you are new, read the [root README](../README.md) first: it is image-first and
|
||||||
|
gets a DSH Web container running before anything else.
|
||||||
|
|
||||||
|
## Index
|
||||||
|
|
||||||
|
| Document | Read it for |
|
||||||
|
| --- | --- |
|
||||||
|
| [main-image.md](main-image.md) | The primary artifact: build args, runtime environment, entrypoint, probes, volumes/ports/signals, non-root contract, image names, tags and digests. |
|
||||||
|
| [examples.md](examples.md) | The example contract: required layout, manifest rules, exposure variants, components and image naming, scripts, docs, and how to add a new example. |
|
||||||
|
| [gitflow.md](gitflow.md) | Branches, protected `main`, squash merges, Conventional PR titles, required CI checks, release branches, hotfixes and the one-time GitHub settings checklist. |
|
||||||
|
| [versioning.md](versioning.md) | Release identity (`DSH ref` + optional `-rN`), why the current identity is `dsh-v0.2.1-rc.3-r4`, channels, compatibility matrix, bumping pins, the release checklist, rollback, git-cliff and digest pinning. |
|
||||||
|
|
||||||
|
## Project skills
|
||||||
|
|
||||||
|
DSH discovers project skills from `<git root>/.dsh/skills/<name>/SKILL.md`
|
||||||
|
(the nearest `.git` ancestor). This repository ships one:
|
||||||
|
|
||||||
|
| Skill | Read it for |
|
||||||
|
| --- | --- |
|
||||||
|
| [dsh-agent-sandbox-platform](../.dsh/skills/dsh-agent-sandbox-platform/SKILL.md) | Operating and extending the agents-as-a-service platform, agent-sandbox MCP limits and deployment, and a read-only preflight. |
|
||||||
|
|
||||||
|
## Example documentation
|
||||||
|
|
||||||
|
| Document | Read it for |
|
||||||
|
| --- | --- |
|
||||||
|
| [examples/README.md](../examples/README.md) | The index of implementations in this repository. |
|
||||||
|
| [dsh-agents-as-a-service README](../examples/dsh-agents-as-a-service/README.md) | The multi-tenant platform walkthrough: warm pool, claims, suspend/resume, gateway, exposure and troubleshooting. |
|
||||||
|
| [dsh-agents-as-a-service architecture](../examples/dsh-agents-as-a-service/docs/architecture.md) | Container/sequence diagrams, ADRs, security and RBAC, failure modes and the evolution path. |
|
||||||
|
| [gateway README](../examples/dsh-agents-as-a-service/gateway/README.md) | The gateway's HTTP/WebSocket API, configuration, auth (including SSO), and limitations. |
|
||||||
|
|
||||||
|
## Project policies
|
||||||
|
|
||||||
|
| Document | Read it for |
|
||||||
|
| --- | --- |
|
||||||
|
| [AGENTS.md](../AGENTS.md) | Repository conventions for AI agents and humans, and the commands the CI runs. |
|
||||||
|
| [CONTRIBUTING.md](../CONTRIBUTING.md) | Local setup, the change workflow and what a pull request needs. |
|
||||||
|
| [SECURITY.md](../SECURITY.md) | Vulnerability reporting and project-specific threat notes (pod logs, SSO header trust, shared provider key, TLS). |
|
||||||
|
| [CODE_OF_CONDUCT.md](../CODE_OF_CONDUCT.md) | Community standards. |
|
||||||
|
| [CHANGELOG.md](../CHANGELOG.md) | Generated release history (do not edit by hand). |
|
||||||
|
| [LICENSE](../LICENSE) | MIT. |
|
||||||
159
docs/examples.md
Normal file
159
docs/examples.md
Normal file
@@ -0,0 +1,159 @@
|
|||||||
|
# Example contract
|
||||||
|
|
||||||
|
Every implementation in this repository is a **self-contained example** under
|
||||||
|
[`examples/<name>/`](../examples/). Examples demonstrate how to run the main
|
||||||
|
DSH Web image (and, when needed, auxiliary components such as the
|
||||||
|
agents-as-a-service gateway) on Kubernetes. They are reference material, not
|
||||||
|
libraries: each one is applied with plain `kubectl`, documented end to end, and
|
||||||
|
deletable in one command.
|
||||||
|
|
||||||
|
[`examples/README.md`](../examples/README.md) is the index. If an example is not
|
||||||
|
listed there, it does not exist as far as `make check` is concerned.
|
||||||
|
|
||||||
|
## Required layout
|
||||||
|
|
||||||
|
```
|
||||||
|
examples/<name>/
|
||||||
|
+-- README.md # required: what it demonstrates, prerequisites,
|
||||||
|
| # ordered steps, troubleshooting, cleanup
|
||||||
|
+-- manifests/ # required
|
||||||
|
| +-- NN-*.yaml # numbered top-level manifests, plain kubectl
|
||||||
|
| +-- optional/ # optional opt-in manifests, applied by path
|
||||||
|
| +-- README.md # required when optional/ exists
|
||||||
|
| +-- <feature>/NN-*.yaml # e.g. network-policy/
|
||||||
|
| +-- exposure/ # optional exposure variants
|
||||||
|
| +-- README.md # required when exposure/ exists
|
||||||
|
| +-- <variant>/NN-*.yaml # e.g. ingress/, gateway-api-sso/
|
||||||
|
+-- scripts/ # optional
|
||||||
|
| +-- *.sh # executable, idempotent, self-cleaning
|
||||||
|
+-- docs/ # optional deeper material (architecture, ADRs)
|
||||||
|
+-- <component>/ # optional published component
|
||||||
|
+-- Dockerfile
|
||||||
|
+-- README.md # required whenever a Dockerfile exists
|
||||||
|
+-- tests...
|
||||||
|
```
|
||||||
|
|
||||||
|
### Rules the CI enforces
|
||||||
|
|
||||||
|
[scripts/check-examples.sh](../scripts/check-examples.sh) fails the build when:
|
||||||
|
|
||||||
|
- `examples/README.md` is missing or does not mention `examples/<name>/` for every
|
||||||
|
example directory;
|
||||||
|
- an example has no `README.md` or no `manifests/` directory;
|
||||||
|
- `manifests/` contains no top-level `*.yaml` files;
|
||||||
|
- a `*.yaml` file sits at the example root instead of under `manifests/`;
|
||||||
|
- `manifests/exposure/` exists without a `manifests/exposure/README.md` describing
|
||||||
|
the variants;
|
||||||
|
- `manifests/optional/` exists without a `manifests/optional/README.md`
|
||||||
|
describing the opt-in manifests;
|
||||||
|
- a subdirectory with a `Dockerfile` has no `README.md`;
|
||||||
|
- a file in `scripts/` is not executable.
|
||||||
|
|
||||||
|
[scripts/check-manifests.py](../scripts/check-manifests.py) additionally parses
|
||||||
|
every `manifests/**/*.yaml` and requires `apiVersion`, `kind` and
|
||||||
|
`metadata.name`. CI also runs a server-side dry-run against the real
|
||||||
|
agent-sandbox CRDs on a throwaway kind cluster.
|
||||||
|
|
||||||
|
Run both locally with:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
make check
|
||||||
|
```
|
||||||
|
|
||||||
|
## Manifests
|
||||||
|
|
||||||
|
- **Numbered and ordered.** `NN-*.yaml` files are applied in filename order and
|
||||||
|
read as a walkthrough: `00-prereqs.yaml`, `10-sandboxtemplate.yaml`,
|
||||||
|
`20-sandboxwarmpool.yaml`, ... Keep the numbering stable; it is referenced from
|
||||||
|
the example README.
|
||||||
|
- **Plain kubectl.** No Helm, Kustomize, or custom installer. `kubectl apply -f`
|
||||||
|
against a namespace the example creates is the whole interface.
|
||||||
|
- **Safe by default.** Never ship a usable Secret. Secret-shaped manifests are
|
||||||
|
`*.yaml.example` files with obvious placeholders and an inline creation
|
||||||
|
command; real values are generated per install.
|
||||||
|
- **Cluster-specific state stays out.** Node affinity, replica counts tuned for a
|
||||||
|
cluster, and similar fields belong in a local patch, not in the shipped
|
||||||
|
manifest. Optional scheduler/runtime hints may ship **commented out** with
|
||||||
|
placeholder labels; they must not be active.
|
||||||
|
- **Optional manifests are opt-in.** Put them under
|
||||||
|
`manifests/optional/<feature>/`, document them in
|
||||||
|
`manifests/optional/README.md`, and apply them by path. The top-level
|
||||||
|
`kubectl apply -f manifests/` is non-recursive and does not apply them.
|
||||||
|
- **Placeholders are explicit.** Hostnames, GitHub orgs and TLS material use
|
||||||
|
values like `dsh.example.com` / `your-github-org` with a TODO comment.
|
||||||
|
|
||||||
|
## Exposure variants
|
||||||
|
|
||||||
|
An example that can be reached from outside the cluster puts each option under
|
||||||
|
`manifests/exposure/<variant>/` and describes the trade-offs in
|
||||||
|
`manifests/exposure/README.md`. Variants are alternatives, not a stack: apply one.
|
||||||
|
`dsh-agents-as-a-service` currently ships:
|
||||||
|
|
||||||
|
| Variant | Path | What it gives you |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| Ingress | `manifests/exposure/ingress/` | Traefik-style Ingress + TLS; the gateway does its own session auth. |
|
||||||
|
| Gateway API + SSO | `manifests/exposure/gateway-api-sso/` | Gateway API `Gateway`/`HTTPRoute` in front of oauth2-proxy (GitHub OAuth); the gateway inherits `X-Forwarded-User` via `AUTH_PROXY_HEADER`. |
|
||||||
|
|
||||||
|
## Components
|
||||||
|
|
||||||
|
A component is a program the example needs beyond the main image: today only the
|
||||||
|
agents-as-a-service gateway. A component lives in its own subdirectory with:
|
||||||
|
|
||||||
|
- `Dockerfile` — its image build;
|
||||||
|
- `README.md` — purpose, API/config reference, local development, limitations;
|
||||||
|
- tests — unit tests that run without a cluster, plus whatever else applies.
|
||||||
|
|
||||||
|
Component images are published as **nested packages below the repository image**:
|
||||||
|
|
||||||
|
```
|
||||||
|
<registry>/<owner>/<repo>/<example>/<component>
|
||||||
|
```
|
||||||
|
|
||||||
|
so the gateway is
|
||||||
|
`ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway`.
|
||||||
|
Never create an unrelated top-level package; `check-versions.sh` verifies the
|
||||||
|
nesting.
|
||||||
|
|
||||||
|
## Scripts
|
||||||
|
|
||||||
|
`scripts/*.sh` must be:
|
||||||
|
|
||||||
|
- **executable** (the CI checks the mode bit);
|
||||||
|
- **idempotent** — safe to re-run, cleaning up before creating;
|
||||||
|
- **self-cleaning** — a trap removes the namespace, port-forwards and temp files,
|
||||||
|
even on failure;
|
||||||
|
- **non-interactive** — usable in CI, with environment-variable knobs instead of
|
||||||
|
prompts.
|
||||||
|
|
||||||
|
`examples/dsh-agents-as-a-service/scripts/e2e-kind.sh` is the reference
|
||||||
|
implementation: it runs the README walkthrough and deletes everything it created.
|
||||||
|
|
||||||
|
## Documentation
|
||||||
|
|
||||||
|
- The example `README.md` is the walkthrough. Keep it runnable: every command
|
||||||
|
block should work when pasted in order.
|
||||||
|
- Deeper material (architecture, sequence diagrams, ADRs) belongs in the
|
||||||
|
example's `docs/`, not in the repository-level [docs/](../docs/).
|
||||||
|
- Repository-wide docs describe the container and the repo rules; example docs
|
||||||
|
describe one implementation.
|
||||||
|
|
||||||
|
## Adding a new example
|
||||||
|
|
||||||
|
1. Create `examples/<name>/` with the layout above.
|
||||||
|
2. Write `README.md` first: what it demonstrates, prerequisites, numbered steps,
|
||||||
|
troubleshooting, limitations, cleanup.
|
||||||
|
3. Add `manifests/NN-*.yaml`; keep secrets as `*.yaml.example`.
|
||||||
|
4. If a component needs an image, publish it under
|
||||||
|
`<registry>/<owner>/<repo>/<example>/<component>`.
|
||||||
|
5. Add an entry to [`examples/README.md`](../examples/README.md) using the literal
|
||||||
|
`examples/<name>/` path.
|
||||||
|
6. Run `make check` and, if you added a component, `make test` and `make lint`.
|
||||||
|
7. If the example is expensive to run (kind, images, cluster CRDs), ship a
|
||||||
|
`scripts/e2e-kind.sh` and wire it into CI.
|
||||||
|
|
||||||
|
## Current examples
|
||||||
|
|
||||||
|
| Example | Demonstrates |
|
||||||
|
| --- | --- |
|
||||||
|
| [`examples/dsh-agents-as-a-service/`](../examples/dsh-agents-as-a-service/) | The full multi-tenant platform pattern: SandboxTemplate, warm pool, per-user claims, suspend/resume, and the browser gateway. |
|
||||||
|
| [`examples/minimal-single-tenant/`](../examples/minimal-single-tenant/) | A single DSH Web sandbox: one SandboxTemplate and a claim. |
|
||||||
128
docs/gitflow.md
Normal file
128
docs/gitflow.md
Normal file
@@ -0,0 +1,128 @@
|
|||||||
|
# Gitflow: branches, pull requests and CI
|
||||||
|
|
||||||
|
This repository uses **trunk-based development with release branches**. `main` is
|
||||||
|
always releasable; feature work happens on short-lived branches; releases are cut
|
||||||
|
from `main` onto a release branch and tagged with an immutable release identity.
|
||||||
|
See [versioning](versioning.md) for what the tags mean.
|
||||||
|
|
||||||
|
## Branches
|
||||||
|
|
||||||
|
| Branch | Branched from | Merged into | Purpose |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| `main` | — | — | The integration trunk. Protected, linear history, always releasable. |
|
||||||
|
| `feat/<slug>` | `main` | `main` (squash) | New capability or example. |
|
||||||
|
| `fix/<slug>` | `main` | `main` (squash) | Bug fix. |
|
||||||
|
| `docs/<slug>` | `main` | `main` (squash) | Documentation only. |
|
||||||
|
| `refactor/<slug>`, `chore/<slug>`, `test/<slug>` | `main` | `main` (squash) | Non-behavioral rework, maintenance, tests. |
|
||||||
|
| `release/<identity>` | `main` at the release commit | — | Release stabilization and tagging. Only release-blocking fixes land here. |
|
||||||
|
| `hotfix/<slug>` | The release branch (or the tagged commit) | `main` and the release branch | Urgent production fix. |
|
||||||
|
|
||||||
|
Keep branches short-lived. Rebase on `main` rather than merging `main` into the
|
||||||
|
branch; the PR is squash-merged, so individual branch commits are never part of
|
||||||
|
`main` history.
|
||||||
|
|
||||||
|
## Pull requests
|
||||||
|
|
||||||
|
1. **One scoped change per PR.** Do not mix a version bump with unrelated work,
|
||||||
|
and update the docs the change touches in the same PR.
|
||||||
|
2. **Conventional PR title.** The title becomes the squash commit message, so it
|
||||||
|
is linted by the `pr-title` check: `feat(scope): ...`, `fix(scope): ...`,
|
||||||
|
`docs(scope): ...`, `build: ...`, `ci: ...`, `refactor: ...`, `test: ...`,
|
||||||
|
`chore: ...`. Breaking changes use `feat(scope)!: ...` or a
|
||||||
|
`BREAKING CHANGE:` footer.
|
||||||
|
3. **CI must be green.** All required checks below must pass before merge.
|
||||||
|
4. **Squash merge only.** `main` stays linear and protected; merged PRs are one
|
||||||
|
commit each.
|
||||||
|
5. **No direct pushes to `main`** or to a release branch. Both are protected.
|
||||||
|
6. **Review for behavior changes.** Infrastructure, workflow, release and
|
||||||
|
gateway-contract changes need a maintainer review; trivial docs changes still
|
||||||
|
need the checks.
|
||||||
|
|
||||||
|
## Required checks
|
||||||
|
|
||||||
|
| Check | Workflow / job | What it verifies |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `lint` | `ci.yaml` | `scripts/check-versions.sh` (version pins and markdown tokens), ruff on the gateway, shellcheck on shell scripts, yamllint/actionlint when available. |
|
||||||
|
| `python` | `ci.yaml` | Gateway unit tests (`pytest`). |
|
||||||
|
| `manifests` | `ci.yaml` | `scripts/check-examples.sh`, `scripts/check-manifests.py` and a server-side dry-run of every example manifest against the agent-sandbox CRDs. |
|
||||||
|
| `gateway-image` | `ci.yaml` | Builds the gateway image and smoke-tests `/healthz` without a cluster. |
|
||||||
|
| `pr-title` | `pr-title.yaml` | Conventional Commit PR title. |
|
||||||
|
| `CodeQL` | `codeql.yml` | Static analysis for Python and GitHub Actions. |
|
||||||
|
| `dependency-review` | `ci.yaml` | Dependency changes introduced by the PR (job `dependency-review`; runs on pull requests only). |
|
||||||
|
|
||||||
|
The image build workflows (`_build-image.yaml`, `build-main.yaml`,
|
||||||
|
`build-gateway.yaml`) and `e2e-kind.yaml` are **not** required on every PR: the
|
||||||
|
image builds run where they are needed, and the kind end-to-end suite is reserved
|
||||||
|
for release validation and manual dispatch because it is slow.
|
||||||
|
|
||||||
|
## Releases
|
||||||
|
|
||||||
|
Releases are cut from `main`:
|
||||||
|
|
||||||
|
1. Confirm `main` is green and `make check`, `make test`, `make lint` pass.
|
||||||
|
2. Bump [versions.env](../versions.env) if the DSH ref changed, using
|
||||||
|
`make set-version`; otherwise keep the same `DSH_VERSION` and bump
|
||||||
|
`RELEASE_REVISION`.
|
||||||
|
3. Create `release/<identity>` from the release commit.
|
||||||
|
4. Run `make release-dry-run` and review the generated notes.
|
||||||
|
5. Create and push the annotated `dsh-*` release tag. The `release.yaml`
|
||||||
|
workflow validates that the tag matches [versions.env](../versions.env), builds
|
||||||
|
and pushes the multi-arch images (version + latest), creates the GitHub
|
||||||
|
Release from the git-cliff notes, and opens a `chore(changelog)` PR that
|
||||||
|
refreshes `CHANGELOG.md` on `main`.
|
||||||
|
6. Merge the changelog PR and any release-branch-only fixes back to `main`.
|
||||||
|
|
||||||
|
Full procedure: [versioning](versioning.md).
|
||||||
|
|
||||||
|
## Hotfixes
|
||||||
|
|
||||||
|
A released tag is **never** moved or re-tagged — tags and image identities are
|
||||||
|
immutable. To fix a broken release:
|
||||||
|
|
||||||
|
1. Branch `hotfix/<slug>` from the release tag (or its release branch).
|
||||||
|
2. Make the minimal fix, add a test, open a PR; the same required checks apply.
|
||||||
|
3. Publish a new identity with a higher `RELEASE_REVISION` (`-r(N+1)`). Do not
|
||||||
|
reuse or delete the broken tag.
|
||||||
|
4. Merge the fix back into `main` (or cherry-pick it) so the trunk is fixed too.
|
||||||
|
|
||||||
|
Rollback is also a re-release, never a retag: see
|
||||||
|
[versioning](versioning.md#rollback).
|
||||||
|
|
||||||
|
## Tag protection
|
||||||
|
|
||||||
|
All release tags match `dsh-*` and are protected:
|
||||||
|
|
||||||
|
- only maintainers (the release workflow) may create them;
|
||||||
|
- they cannot be deleted or force-updated;
|
||||||
|
- they are annotated and point at the release commit on a release branch.
|
||||||
|
|
||||||
|
Image tags derive from the same identity and are equally immutable; the mutable
|
||||||
|
`latest` and `main` image channels always point at a release or a trunk build
|
||||||
|
respectively.
|
||||||
|
|
||||||
|
## One-time GitHub settings checklist
|
||||||
|
|
||||||
|
Configure these on the repository before relying on the workflow:
|
||||||
|
|
||||||
|
- [ ] **Branches → main**: require a pull request; require the checks `lint`,
|
||||||
|
`python`, `manifests`, `gateway-image`, `pr-title`, `CodeQL`,
|
||||||
|
`dependency-review`; require branches to be up to date; require conversation
|
||||||
|
resolution; restrict direct pushes; allow only squash merges and enable
|
||||||
|
"Automatically delete head branches".
|
||||||
|
- [ ] **Branches → release/***: same protections, plus allow the release workflow
|
||||||
|
to push tags.
|
||||||
|
- [ ] **Tags → dsh-***: restrict creation/update/deletion to maintainers; a tag
|
||||||
|
push triggers the release workflow.
|
||||||
|
- [ ] **Actions → General**: allow GitHub Actions; set workflow permissions to
|
||||||
|
read-only by default; allow `GITHUB_TOKEN` to create pull requests only
|
||||||
|
where required.
|
||||||
|
- [ ] **Actions → Runners**: make the arm64 runners (`ubuntu-24.04-arm`) available
|
||||||
|
to the public/private repository as appropriate.
|
||||||
|
- [ ] **Packages**: grant the release workflow `packages: write` (workflow
|
||||||
|
permissions do this) and protect the
|
||||||
|
`ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox` packages from deletion.
|
||||||
|
- [ ] **Security**: enable CodeQL default setup **or** keep the advanced
|
||||||
|
`codeql.yml` workflow (not both), enable Dependabot alerts and
|
||||||
|
dependency review.
|
||||||
|
- [ ] **Secrets/Variables**: none are needed for image publishing; the workflows
|
||||||
|
use `GITHUB_TOKEN`.
|
||||||
191
docs/main-image.md
Normal file
191
docs/main-image.md
Normal file
@@ -0,0 +1,191 @@
|
|||||||
|
# The main DSH Web image
|
||||||
|
|
||||||
|
This repository's primary artifact is a container image that packages
|
||||||
|
[DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) Web
|
||||||
|
(`dsh web`) for Kubernetes and plain Docker. The [Dockerfile](../Dockerfile) at the
|
||||||
|
repository root builds it; every version pin it uses lives in
|
||||||
|
[versions.env](../versions.env).
|
||||||
|
|
||||||
|
The image is image-first: it is the product, and the examples under
|
||||||
|
[`examples/`](../examples/) are self-contained implementations that consume it.
|
||||||
|
|
||||||
|
## What the build does
|
||||||
|
|
||||||
|
The Dockerfile is a two-stage build:
|
||||||
|
|
||||||
|
1. **Build stage** (`node:24-trixie`) clones DSH at the pinned git ref, installs
|
||||||
|
dependencies with pnpm, runs the production build, packages the `@deepseek-ai/dsh`
|
||||||
|
CLI with `pnpm pack`, and installs its production dependencies with npm.
|
||||||
|
2. **Runtime stage** (`node:24-trixie-slim`) copies `/opt/dsh` to `/app`, adds the
|
||||||
|
CLI tools DSH agents commonly use (`git`, `ripgrep`, `jq`, `yq`, `curl`, `wget`,
|
||||||
|
`gh`, `tini`, `uv`), and starts `dsh web` through a small entrypoint.
|
||||||
|
|
||||||
|
## Build it
|
||||||
|
|
||||||
|
```bash
|
||||||
|
make image # canonical: builds the DSH Web image, no push
|
||||||
|
```
|
||||||
|
|
||||||
|
`make image` reads [versions.env](../versions.env) and passes the pins as build args.
|
||||||
|
To reproduce the equivalent by hand:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker build \
|
||||||
|
--build-arg DSH_VERSION=dsh-v0.2.1-rc.3 \
|
||||||
|
--build-arg RELEASE_REVISION=r4 \
|
||||||
|
--build-arg PNPM_VERSION=11.28.5 \
|
||||||
|
--build-arg VCS_REF="$(git rev-parse --short HEAD)" \
|
||||||
|
-t ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.3-r4 .
|
||||||
|
```
|
||||||
|
|
||||||
|
### Build args
|
||||||
|
|
||||||
|
| Arg | Default / source | Purpose |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `DSH_REPO` | `https://github.com/deepseek-ai/deepseek-harness.git` | Upstream DSH repository to clone. |
|
||||||
|
| `DSH_VERSION` | `DSH_VERSION` in [versions.env](../versions.env) | Upstream DSH git tag/ref to package; recorded as `org.opencontainers.image.version`. |
|
||||||
|
| `RELEASE_REVISION` | `RELEASE_REVISION` in [versions.env](../versions.env) | Repository revision suffix (`rN`); the release identity is `DSH_VERSION-RELEASE_REVISION`, recorded in the `io.sredevopsorg.release.revision` label. See [versioning](versioning.md). |
|
||||||
|
| `PNPM_VERSION` | `PNPM_VERSION` in [versions.env](../versions.env) | pnpm version activated with corepack in both stages. |
|
||||||
|
| `VCS_REF` | `git rev-parse --short HEAD` (via `make image`) | Source revision recorded for traceability. |
|
||||||
|
| `NODE_BUILD_IMAGE` | `docker.io/node:24-trixie` | Build-stage base image. |
|
||||||
|
| `NODE_RUNTIME_IMAGE` | `docker.io/node:24-trixie-slim` | Runtime-stage base image. |
|
||||||
|
|
||||||
|
Never hard-code a version outside [versions.env](../versions.env):
|
||||||
|
`./scripts/check-versions.sh` (and `make check`) fails when a pin drifts.
|
||||||
|
|
||||||
|
## Runtime contract
|
||||||
|
|
||||||
|
### Entrypoint
|
||||||
|
|
||||||
|
The entrypoint is `tini -- /usr/local/bin/dsh-entrypoint`:
|
||||||
|
|
||||||
|
1. `tini` runs as PID 1 and reaps children/forwards signals.
|
||||||
|
2. The shell entrypoint applies defaults, derives a bind address when `DSH_HOST` is
|
||||||
|
empty (the pod/container's first non-loopback IPv4), builds the
|
||||||
|
`dsh web` command line, and `exec`s it so signals reach the Node process.
|
||||||
|
3. `localhost` and `127.0.0.1` are always added to DSH's trusted-host list; each
|
||||||
|
`DSH_TRUSTED_HOSTS` entry is added as an extra `--trusted-host`.
|
||||||
|
|
||||||
|
### Environment variables
|
||||||
|
|
||||||
|
| Variable | Default | Purpose |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `DSH_PORT` | `3080` | Port DSH Web listens on inside the container. |
|
||||||
|
| `DSH_HOST` | auto-detected non-loopback IPv4 | Bind address. Leave empty in Kubernetes so the pod IP is used automatically. |
|
||||||
|
| `DSH_PUBLIC_URL` | `http://localhost:${DSH_PORT}` | Public base URL used to build links and the launch-token URL printed in the logs. Set it when the container is published on a different host port or behind a proxy. |
|
||||||
|
| `DSH_TRUSTED_HOSTS` | empty | Extra `Host`/`Origin` allow-list entries, comma- or space-separated. `localhost`/`127.0.0.1` are always trusted. Required when a proxy forwards the browser's `Host`. |
|
||||||
|
| `DEEPSEEK_API_KEY` | empty | Provider credential read by DSH. Inject it with `-e`/`envFrom`; never bake it into the image. Only model calls fail when it is missing. |
|
||||||
|
| `DSH_HOME` | `/home/node/.dsh` | DSH state directory (conversations, sessions, config, credentials). Mount a volume here to persist it. |
|
||||||
|
|
||||||
|
The image also sets `NODE_ENV=production`, `HOME=/home/node`, and a `PATH` that
|
||||||
|
includes `/app/node_modules/.bin`.
|
||||||
|
|
||||||
|
### Provider credentials
|
||||||
|
|
||||||
|
DSH resolves provider API keys from the environment when no stored credential
|
||||||
|
exists. The packaged provider catalog maps each provider to its standard
|
||||||
|
environment name. Commonly used names:
|
||||||
|
|
||||||
|
`DEEPSEEK_API_KEY`, `OPENROUTER_API_KEY`, `GEMINI_API_KEY`, `NVIDIA_API_KEY`,
|
||||||
|
`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GROQ_API_KEY`, `XAI_API_KEY`,
|
||||||
|
`MISTRAL_API_KEY`, `TOGETHER_API_KEY`, `CEREBRAS_API_KEY`,
|
||||||
|
`AI_GATEWAY_API_KEY`, `HF_TOKEN`.
|
||||||
|
|
||||||
|
Use `GEMINI_API_KEY` for Google Gemini (not `GOOGLE_API_KEY`), `HF_TOKEN` for
|
||||||
|
Hugging Face, and `AI_GATEWAY_API_KEY` for the Vercel AI Gateway. Inject keys
|
||||||
|
with `-e`/`--env-file` in Docker or a Secret + `envFrom` in Kubernetes; never
|
||||||
|
bake them into the image.
|
||||||
|
|
||||||
|
A key saved through DSH's credential store (`$DSH_HOME/.credentials.yaml`) wins
|
||||||
|
over the environment. The
|
||||||
|
[agents-as-a-service example](../examples/dsh-agents-as-a-service/docs/optional-hardening.md)
|
||||||
|
documents the optional `dsh-provider-keys-extra` Secret and the pod-refresh
|
||||||
|
procedure.
|
||||||
|
|
||||||
|
### Ports, volumes and signals
|
||||||
|
|
||||||
|
| Item | Value | Notes |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| Exposed port | `3080/tcp` | `EXPOSE 3080`; one port serves the UI, assets, API, SSE and WebSocket traffic. |
|
||||||
|
| State volume | `/home/node/.dsh` | Declared with `VOLUME`; use a named volume (Docker) or PVC (Kubernetes). |
|
||||||
|
| Working directory | `/workspace` | The agent workspace; mount a volume here if agents should keep files. |
|
||||||
|
| Stop signal | `SIGTERM` | `tini` forwards it to `dsh web`; allow a graceful shutdown period. |
|
||||||
|
| User | `node`, UID/GID `1000` | The process is non-root. A PVC mounted at `DSH_HOME` must be writable by UID 1000. |
|
||||||
|
|
||||||
|
### Health probes
|
||||||
|
|
||||||
|
`/` is **not** a health endpoint: DSH returns `401` until a browser completes the
|
||||||
|
launch-token handshake. Probe `/favicon.svg` instead — it is unauthenticated and
|
||||||
|
served by the web layer, so it proves the HTTP server is up without triggering auth.
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
startupProbe:
|
||||||
|
httpGet: {path: /favicon.svg, port: 3080}
|
||||||
|
periodSeconds: 5
|
||||||
|
failureThreshold: 24
|
||||||
|
readinessProbe:
|
||||||
|
httpGet: {path: /favicon.svg, port: 3080}
|
||||||
|
periodSeconds: 5
|
||||||
|
livenessProbe:
|
||||||
|
httpGet: {path: /favicon.svg, port: 3080}
|
||||||
|
periodSeconds: 15
|
||||||
|
```
|
||||||
|
|
||||||
|
The shipped templates use exactly this shape; see
|
||||||
|
[`examples/minimal-single-tenant/manifests/sandbox-template.yaml`](../examples/minimal-single-tenant/manifests/sandbox-template.yaml).
|
||||||
|
A `startupProbe` is important because plugin activation (or the first start of a
|
||||||
|
large sandbox) can exceed a default liveness budget.
|
||||||
|
|
||||||
|
### Launch token
|
||||||
|
|
||||||
|
DSH prints its one-time launch URL to the logs:
|
||||||
|
|
||||||
|
```
|
||||||
|
dsh web: http://<host>:${DSH_PORT}/?token=...
|
||||||
|
```
|
||||||
|
|
||||||
|
Open that URL once to obtain DSH's auth cookie. After that the UI works at the bare
|
||||||
|
URL; without the cookie `/` answers `401` by design. The
|
||||||
|
[agents-as-a-service gateway](../examples/dsh-agents-as-a-service/gateway/) performs
|
||||||
|
this handshake server-side so browsers never see the token.
|
||||||
|
|
||||||
|
## Published image names and tags
|
||||||
|
|
||||||
|
Registry layout: the main image is the repository itself, and every example
|
||||||
|
component is a nested package below it:
|
||||||
|
|
||||||
|
```
|
||||||
|
ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox # main DSH Web image
|
||||||
|
ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway # example component
|
||||||
|
```
|
||||||
|
|
||||||
|
The main image is published multi-arch (`linux/amd64`, `linux/arm64`) to
|
||||||
|
GitHub Container Registry:
|
||||||
|
|
||||||
|
| Tag | Meaning |
|
||||||
|
| --- | --- |
|
||||||
|
| `dsh-v0.2.1-rc.3-r4` | Current release identity. **Immutable**: once published it never moves. |
|
||||||
|
| `latest` | Most recent release; points at the same digest as the current identity. |
|
||||||
|
| `main` | Development channel rebuilt from every push to `main`; not a release, not for production. |
|
||||||
|
| `dsh-v0.2.1-rc.3` | Legacy tag published before the release-identity scheme existed. Do not use it for new deployments; see the migration table in the [root README](../README.md). |
|
||||||
|
|
||||||
|
Release identities, channels and rollback rules are defined in
|
||||||
|
[versioning](versioning.md). Tags are immutable, so production deployments should
|
||||||
|
pin a digest:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker buildx imagetools inspect ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.3-r4
|
||||||
|
# ... pin the digest in your deployment manifests:
|
||||||
|
# image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox@sha256:<digest>
|
||||||
|
```
|
||||||
|
|
||||||
|
## Verifying an image
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker inspect ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.3-r4 \
|
||||||
|
--format '{{json .Config.Labels}}'
|
||||||
|
# org.opencontainers.image.title / .version / .source
|
||||||
|
```
|
||||||
|
|
||||||
|
Release builds also publish SBOM and provenance attestations from the release
|
||||||
|
workflow (see [SECURITY.md](../SECURITY.md) and [gitflow](gitflow.md)).
|
||||||
163
docs/versioning.md
Normal file
163
docs/versioning.md
Normal file
@@ -0,0 +1,163 @@
|
|||||||
|
# Versioning, releases and changelog
|
||||||
|
|
||||||
|
[versions.env](../versions.env) is the **single source of truth** for the DSH ref,
|
||||||
|
the repository revision, the pnpm version, the control-plane component versions
|
||||||
|
and the image names. Nothing else may hard-code a version:
|
||||||
|
[scripts/check-versions.sh](../scripts/check-versions.sh) fails when a pin drifts,
|
||||||
|
and it also verifies that prose mentions only the current release identity or its
|
||||||
|
base DSH tag.
|
||||||
|
|
||||||
|
## Release identity
|
||||||
|
|
||||||
|
```
|
||||||
|
release identity = DSH_VERSION + optional "-" + RELEASE_REVISION
|
||||||
|
```
|
||||||
|
|
||||||
|
| Piece | Meaning |
|
||||||
|
| --- | --- |
|
||||||
|
| `DSH_VERSION` | The upstream DeepSeek Harness git ref being packaged, e.g. `dsh-v0.2.1-rc.3`. It mirrors upstream's own tag, which is what "mirrored-DSH scheme" means. |
|
||||||
|
| `RELEASE_REVISION` | An optional repository revision at that DSH ref: empty for the first release, `rN` (`N >= 2`) for a repository-only re-release. |
|
||||||
|
|
||||||
|
Examples:
|
||||||
|
|
||||||
|
- `DSH_VERSION=dsh-v0.2.1-rc.3`, `RELEASE_REVISION=` → identity `dsh-v0.2.1-rc.3`
|
||||||
|
(first release at that upstream ref).
|
||||||
|
- `DSH_VERSION=dsh-v0.2.1-rc.3`, `RELEASE_REVISION=r4` → identity
|
||||||
|
`dsh-v0.2.1-rc.3-r4` (documentation, CI or gateway changes on top of the same
|
||||||
|
upstream DSH; no DSH upgrade).
|
||||||
|
|
||||||
|
A DSH upgrade produces a new base identity (for example the next upstream ref
|
||||||
|
with an empty revision); a repository-only change at the same DSH ref increments
|
||||||
|
`rN`.
|
||||||
|
|
||||||
|
## Why the current identity is `dsh-v0.2.1-rc.3-r4`
|
||||||
|
|
||||||
|
The image tag `dsh-v0.2.1-rc.3` was already published to GHCR for an earlier
|
||||||
|
repository state. **Container tags are immutable** in this project: re-pushing a
|
||||||
|
tag would silently change what an existing deployment runs. The mirrored-DSH
|
||||||
|
scheme therefore starts its first identity at `-r2`: the plain tag is
|
||||||
|
conceptually revision 1 and is taken, so the first publishable repository
|
||||||
|
revision at the same upstream ref is `r2`.
|
||||||
|
|
||||||
|
The identity is stored in [versions.env](../versions.env) and consumed everywhere:
|
||||||
|
the Dockerfile label, the example manifests, the Makefile and the release
|
||||||
|
workflow. The legacy `dsh-v0.2.1-rc.3` tag remains published but is not updated;
|
||||||
|
new deployments use `dsh-v0.2.1-rc.3-r4`.
|
||||||
|
|
||||||
|
The gateway image moved from the repository-level `:gateway` tag to a nested
|
||||||
|
package (`<repository>/dsh-agents-as-a-service/gateway`). The release workflow
|
||||||
|
also republishes the legacy `<repository>:gateway` tag as a deprecated
|
||||||
|
compatibility alias; it may be removed in a later release, so deployments
|
||||||
|
should pin the nested package.
|
||||||
|
|
||||||
|
## Channels
|
||||||
|
|
||||||
|
| Channel | Image tag | Meaning |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| Release | `dsh-v0.2.1-rc.3-r4` | The current immutable release identity. What production should pin (preferably by digest). |
|
||||||
|
| `latest` | `latest` | Alias for the most recent release; moves only when a new release is published. |
|
||||||
|
| `main` | `main` | Development channel, rebuilt from every push to `main`. Never treat it as a release; it exists so `main` can be tested. |
|
||||||
|
|
||||||
|
The same channels exist for every published package, including the nested
|
||||||
|
example component image.
|
||||||
|
|
||||||
|
## Compatibility matrix
|
||||||
|
|
||||||
|
Values from [versions.env](../versions.env) at the current identity:
|
||||||
|
|
||||||
|
| Component | Pin | Source |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| Upstream DSH source | `DSH_REPO` | [versions.env](../versions.env) |
|
||||||
|
| DSH ref | `dsh-v0.2.1-rc.3` | `DSH_VERSION` |
|
||||||
|
| Repository revision | `r4` | `RELEASE_REVISION` |
|
||||||
|
| pnpm | `11.28.5` | `PNPM_VERSION` |
|
||||||
|
| agent-sandbox (CRDs) | `v1.0.7` | `AGENT_SANDBOX_VERSION`, used by the examples and CI dry-run |
|
||||||
|
| Kubernetes Gateway API | `v1.2.1` | `GATEWAY_API_VERSION`, used by the SSO exposure variant |
|
||||||
|
| oauth2-proxy | `quay.io/oauth2-proxy/oauth2-proxy:v7.15.5` | `OAUTH2_PROXY_IMAGE` |
|
||||||
|
| Main image | `ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox` | `GHCR_IMAGE` |
|
||||||
|
| Gateway image | `ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway` | `GATEWAY_IMAGE`, nested below `GHCR_IMAGE` |
|
||||||
|
|
||||||
|
Example component images are always nested below the main package:
|
||||||
|
`<GHCR_IMAGE>/<example>/<component>`. `check-versions.sh` rejects an unrelated
|
||||||
|
top-level package.
|
||||||
|
|
||||||
|
## Bumping versions
|
||||||
|
|
||||||
|
```bash
|
||||||
|
make set-version DSH_VERSION=dsh-vX.Y.Z RELEASE_REVISION=rN
|
||||||
|
make check
|
||||||
|
```
|
||||||
|
|
||||||
|
`set-version` updates [versions.env](../versions.env) through
|
||||||
|
[scripts/set-version.sh](../scripts/set-version.sh) and leaves every consuming
|
||||||
|
file to derive from it. Never edit the Dockerfile, manifests, Makefile or
|
||||||
|
workflows with a literal version.
|
||||||
|
|
||||||
|
- New upstream DSH ref: change `DSH_VERSION` and clear `RELEASE_REVISION`.
|
||||||
|
- Repository-only change at the same DSH ref: keep `DSH_VERSION`, set
|
||||||
|
`RELEASE_REVISION=r(N+1)`.
|
||||||
|
|
||||||
|
## Release checklist
|
||||||
|
|
||||||
|
1. `main` is green and `make check`, `make test`, `make lint` pass locally.
|
||||||
|
2. `make image` and `make gateway` build; smoke-test the images.
|
||||||
|
3. Bump the pins with `make set-version` (commit that change through a PR).
|
||||||
|
4. `make release-dry-run` — verifies the identity, prints the image names, checks
|
||||||
|
the tag does not already exist, and previews the generated notes.
|
||||||
|
5. Create and push the annotated `dsh-*` release tag (or dispatch
|
||||||
|
`release.yaml` with an existing tag). The workflow validates that the tag
|
||||||
|
matches [versions.env](../versions.env).
|
||||||
|
6. The workflow builds and pushes multi-arch images for the main image and the
|
||||||
|
example components (the identity tag plus `latest`), creates the GitHub
|
||||||
|
Release from the git-cliff notes, records the published digests, and opens a
|
||||||
|
`chore(changelog)` PR that refreshes `CHANGELOG.md` on `main`.
|
||||||
|
7. Merge the changelog PR. Verify:
|
||||||
|
`docker buildx imagetools inspect <image>:<identity>` shows both platforms,
|
||||||
|
and the GitHub Release references the tag and the digest.
|
||||||
|
8. Announce the identity and the digest; deployments pin the digest.
|
||||||
|
|
||||||
|
## Rollback
|
||||||
|
|
||||||
|
Tags and release identities are **immutable**, so a bad release is never re-tagged
|
||||||
|
or deleted. Fix forward:
|
||||||
|
|
||||||
|
1. Fix the problem on a `hotfix/<slug>` branch (see [gitflow](gitflow.md)).
|
||||||
|
2. Publish a new identity with the next revision, `-r(N+1)`, at the same
|
||||||
|
`DSH_VERSION`.
|
||||||
|
3. Point deployments at the new digest; keep the broken tag for audit history.
|
||||||
|
|
||||||
|
Emergency rollback to the previous release is also a re-deploy of a **previous
|
||||||
|
digest**, not a move of `latest`.
|
||||||
|
|
||||||
|
## Changelog
|
||||||
|
|
||||||
|
`CHANGELOG.md` is generated by [git-cliff](https://git-cliff.org) from
|
||||||
|
Conventional Commits, configured in [cliff.toml](../cliff.toml):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
make changelog # regenerate CHANGELOG.md locally
|
||||||
|
```
|
||||||
|
|
||||||
|
- Do not edit `CHANGELOG.md` by hand; the release workflow regenerates it and opens a PR to refresh it on `main`.
|
||||||
|
- Commit messages and PR titles must be Conventional Commits so they are grouped
|
||||||
|
correctly (Added / Fixed / Changed / Documentation / Build / CI / Tests /
|
||||||
|
Maintenance).
|
||||||
|
- The changelog groups commits since the previous `dsh-*` tag; release notes use
|
||||||
|
the same rendering.
|
||||||
|
- The changelog is excluded from the version-token check because it records
|
||||||
|
historical identities on purpose.
|
||||||
|
|
||||||
|
## Digest pinning
|
||||||
|
|
||||||
|
Release identities are mutable only in theory; deployments should still pin the
|
||||||
|
resolved digest for reproducibility and supply-chain safety:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker buildx imagetools inspect ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.3-r4
|
||||||
|
# then, in the deployment:
|
||||||
|
# image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox@sha256:<digest>
|
||||||
|
```
|
||||||
|
|
||||||
|
The example manifests use the identity tag so they stay readable and upgrade with
|
||||||
|
`versions.env`; production overlays should pin the digest. Release images carry
|
||||||
|
SBOM and provenance attestations (see [SECURITY.md](../SECURITY.md)).
|
||||||
55
examples/README.md
Normal file
55
examples/README.md
Normal file
@@ -0,0 +1,55 @@
|
|||||||
|
# Examples
|
||||||
|
|
||||||
|
Every implementation in this repository is a self-contained example: one
|
||||||
|
directory, plain `kubectl`, an end-to-end README, and a cleanup command.
|
||||||
|
Examples consume the [main DSH Web image](../docs/main-image.md); they are
|
||||||
|
reference material, not libraries.
|
||||||
|
|
||||||
|
| Example | What it demonstrates | Needs |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| [`examples/dsh-agents-as-a-service/`](dsh-agents-as-a-service/) | The full multi-tenant platform pattern: one `SandboxTemplate` with per-user PVCs, a warm pool, one ~10-line `SandboxClaim` per user, suspend-when-idle / wake-on-return, and a browser gateway with signup, `HttpOnly` sessions, an HTTP + WebSocket proxy and two exposure variants (Ingress, or Gateway API + GitHub SSO). | agent-sandbox with extensions; kind for the e2e script. |
|
||||||
|
| [`examples/minimal-single-tenant/`](minimal-single-tenant/) | The smallest useful setup: a single `SandboxTemplate` and one `SandboxClaim`, no gateway — reach DSH Web with a launch-token URL through `kubectl port-forward`. | agent-sandbox with extensions. |
|
||||||
|
|
||||||
|
The multi-tenant example also documents opt-in gVisor, `nodeSelector`,
|
||||||
|
NetworkPolicy and extra provider-key hardening in
|
||||||
|
[optional hardening](dsh-agents-as-a-service/docs/optional-hardening.md).
|
||||||
|
|
||||||
|
## The example contract (short version)
|
||||||
|
|
||||||
|
Full rules: [docs/examples.md](../docs/examples.md). `scripts/check-examples.sh`
|
||||||
|
enforces the layout, and `make check` runs it.
|
||||||
|
|
||||||
|
```
|
||||||
|
examples/<name>/
|
||||||
|
+-- README.md # walkthrough: prerequisites, steps, troubleshooting, cleanup
|
||||||
|
+-- manifests/ # NN-*.yaml, applied with plain kubectl
|
||||||
|
| +-- optional/<feature>/ # optional opt-in manifests...
|
||||||
|
| +-- optional/README.md # ...required whenever optional/ exists
|
||||||
|
| +-- exposure/<variant>/ # optional per-variant manifests...
|
||||||
|
| +-- exposure/README.md # ...required whenever exposure/ exists
|
||||||
|
+-- scripts/ # optional executable, idempotent, self-cleaning *.sh
|
||||||
|
+-- docs/ # optional architecture/ADR material
|
||||||
|
+-- <component>/ # optional component with Dockerfile + README + tests
|
||||||
|
```
|
||||||
|
|
||||||
|
- Secrets never ship populated: use `*.yaml.example` files with placeholders.
|
||||||
|
- Manifests are numbered (`00-`, `10-`, ...) and read as an ordered walkthrough.
|
||||||
|
- Component images are published nested below the main image:
|
||||||
|
`<registry>/<owner>/<repo>/<example>/<component>`.
|
||||||
|
- Every example must be listed above with its literal `examples/<name>/` path.
|
||||||
|
- Scripts under `scripts/` must be executable, idempotent and clean up after
|
||||||
|
themselves.
|
||||||
|
|
||||||
|
## Running an example
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# e.g. the multi-tenant platform example
|
||||||
|
kubectl apply -f examples/dsh-agents-as-a-service/manifests/
|
||||||
|
|
||||||
|
# or run the whole walkthrough (kind + agent-sandbox required) and let it clean up
|
||||||
|
examples/dsh-agents-as-a-service/scripts/e2e-kind.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
`kubectl apply -f manifests/` is non-recursive: `optional/` and `exposure/`
|
||||||
|
subdirectories are applied explicitly. Each example README has the exact
|
||||||
|
prerequisites, optional hardening, exposure options and teardown.
|
||||||
@@ -1,27 +0,0 @@
|
|||||||
# DSH "Agents as a Service" — prerequisites: namespace + shared provider key.
|
|
||||||
#
|
|
||||||
# The dsh-provider-keys Secret is deliberately NOT committed with a usable
|
|
||||||
# value. DEEPSEEK_API_KEY reaches every tenant sandbox (injected via envFrom
|
|
||||||
# on the SandboxTemplate, stamped before any user exists — a warm-pool
|
|
||||||
# constraint), so a real key committed here would be shared by every install
|
|
||||||
# of this example. Generate a fresh value per install (README step 1;
|
|
||||||
# run-test-kind.sh uses a throwaway):
|
|
||||||
#
|
|
||||||
# kubectl -n dsh-demo create secret generic dsh-provider-keys \
|
|
||||||
# --from-literal=DEEPSEEK_API_KEY="sk-..."
|
|
||||||
#
|
|
||||||
# All the infrastructure in this example works without a real key; only
|
|
||||||
# actual model calls need one.
|
|
||||||
apiVersion: v1
|
|
||||||
kind: Namespace
|
|
||||||
metadata:
|
|
||||||
name: dsh-demo
|
|
||||||
---
|
|
||||||
apiVersion: v1
|
|
||||||
kind: Secret
|
|
||||||
metadata:
|
|
||||||
name: dsh-provider-keys
|
|
||||||
namespace: dsh-demo
|
|
||||||
type: Opaque
|
|
||||||
# stringData:
|
|
||||||
# DEEPSEEK_API_KEY: "sk-your-key-here"
|
|
||||||
@@ -2,20 +2,45 @@
|
|||||||
|
|
||||||
This example distills the upstream
|
This example distills the upstream
|
||||||
[hermes-agents-as-a-service](https://github.com/kubernetes-sigs/agent-sandbox/tree/main/examples/hermes-agents-as-a-service)
|
[hermes-agents-as-a-service](https://github.com/kubernetes-sigs/agent-sandbox/tree/main/examples/hermes-agents-as-a-service)
|
||||||
platform pattern into pure agent-sandbox resources + kubectl, running **DeepSeek Harness Web
|
platform pattern into pure agent-sandbox resources + kubectl, running **DeepSeek
|
||||||
(DSH)** instead of Hermes. Every user of such a platform gets a personal, long-lived DSH Web
|
Harness Web (DSH)** instead of Hermes. Every user of such a platform gets a
|
||||||
instance that is **provisioned in ~2 seconds from a warm pool, suspended when idle to save
|
personal, long-lived DSH Web instance that is **provisioned in ~2 seconds from a
|
||||||
cost, and woken transparently when the user returns** — with all state (conversations,
|
warm pool, suspended when idle to save cost, and woken transparently when the
|
||||||
sessions, config) surviving on a per-user PVC at `/home/node/.dsh`.
|
user returns** — with all state (conversations, sessions, config) surviving on a
|
||||||
|
per-user PVC at `/home/node/.dsh`.
|
||||||
|
|
||||||
It ships in two layers:
|
It ships in three layers:
|
||||||
|
|
||||||
1. **The resource layer** (steps 1–7): pure manifests + kubectl, so every mechanism is visible.
|
1. **The resource layer** (steps 1–7): pure manifests + kubectl, so every
|
||||||
2. **The service layer** (step 8): a small included **gateway** (`gateway/`) that makes the
|
mechanism is visible.
|
||||||
platform browser-usable — a signup page/API, `HttpOnly` cookie sessions, a `/u/{user}/**`
|
2. **The service layer** (step 8): a small included **gateway** (`gateway/`) that
|
||||||
proxy for HTTP **and WebSocket** that transparently **wakes suspended instances on
|
makes the platform browser-usable — a signup page/API, `HttpOnly` cookie
|
||||||
connect**, and an idle sweeper that suspends them again. The gateway performs the DSH
|
sessions, a `/u/{user}/**` proxy for HTTP **and WebSocket** that transparently
|
||||||
launch-token handshake server-side, so the browser never sees the token.
|
**wakes suspended instances on connect**, and an idle sweeper that suspends
|
||||||
|
them again. The gateway performs the DSH launch-token handshake server-side,
|
||||||
|
so the browser never sees the token.
|
||||||
|
3. **The exposure layer** (step 9): choose exactly one way in — a plain Traefik
|
||||||
|
Ingress, or Gateway API + oauth2-proxy (GitHub SSO).
|
||||||
|
|
||||||
|
All manifests live under [`manifests/`](manifests/):
|
||||||
|
|
||||||
|
```
|
||||||
|
manifests/
|
||||||
|
+-- 00-prereqs.yaml # namespace + empty provider-key Secret
|
||||||
|
+-- 10-sandboxtemplate.yaml # the DSH Web blueprint
|
||||||
|
+-- 20-sandboxwarmpool.yaml # 2 pre-warmed spares
|
||||||
|
+-- 30-claim-alice.yaml # one user
|
||||||
|
+-- 40-claim-rejected.yaml # negative demo: env injection rejected
|
||||||
|
+-- 50-gateway.yaml # Deployment/RBAC/Service for the gateway
|
||||||
|
+-- optional/
|
||||||
|
| +-- README.md # opt-in manifests, applied by path
|
||||||
|
| +-- network-policy/10-sandbox-networkpolicy.yaml
|
||||||
|
+-- exposure/
|
||||||
|
+-- README.md # the two exposure variants, when to use which
|
||||||
|
+-- ingress/60-ingress.yaml
|
||||||
|
+-- ingress/61-networkpolicy.yaml
|
||||||
|
+-- gateway-api-sso/70-gateway.yaml ... 75-networkpolicies.yaml
|
||||||
|
```
|
||||||
|
|
||||||
What it demonstrates, feature by feature:
|
What it demonstrates, feature by feature:
|
||||||
|
|
||||||
@@ -34,20 +59,23 @@ What it demonstrates, feature by feature:
|
|||||||
|
|
||||||
- **Multi-arch**: the DSH image is built for amd64 and arm64, so there is no
|
- **Multi-arch**: the DSH image is built for amd64 and arm64, so there is no
|
||||||
`kubernetes.io/arch: amd64` nodeSelector.
|
`kubernetes.io/arch: amd64` nodeSelector.
|
||||||
- **Single port**: DSH Web serves everything on `3080`; the gateway proxies to one upstream
|
- **Single port**: DSH Web serves everything on `3080`; the gateway proxies to one
|
||||||
instead of splitting `/v1/*` (API) from the dashboard.
|
upstream instead of splitting `/v1/*` (API) from the dashboard.
|
||||||
- **One shared secret**: only `DEEPSEEK_API_KEY` is injected (via `envFrom`); there is no
|
- **Shared provider keys**: `DEEPSEEK_API_KEY` plus any keys in the optional
|
||||||
dashboard basic-auth, session-cookie secret, or API-server key.
|
`dsh-provider-keys-extra` Secret are injected via `envFrom` into every
|
||||||
- **Launch-token handshake**: DSH gates `/` behind a token it prints only in its logs. The
|
sandbox. There is no per-tenant key, dashboard basic-auth, session-cookie
|
||||||
gateway reads it, performs the handshake, and keeps the resulting auth cookie server-side;
|
secret, or API-server key.
|
||||||
the browser authenticates to the gateway with its own cookie. See
|
- **Launch-token handshake**: DSH gates `/` behind a token it prints only in its
|
||||||
[How the DSH handshake works](#how-the-dsh-handshake-works).
|
logs. The gateway reads it, performs the handshake, and keeps the resulting
|
||||||
- **Host allow-list**: DSH rejects unknown `Host` headers. The gateway forwards the browser's
|
auth cookie server-side; the browser authenticates to the gateway with its own
|
||||||
`Host`, so the public hostname must be added to `DSH_TRUSTED_HOSTS` in the template.
|
cookie. See [How the DSH handshake works](#how-the-dsh-handshake-works).
|
||||||
- **Probes target `/favicon.svg`**: `/` returns `401` without the launch token, so a probe on
|
- **Host allow-list**: DSH rejects unknown `Host` headers. The gateway forwards
|
||||||
`/` crash-loops the pod. `/favicon.svg` is unauthenticated, and a `startupProbe` gates
|
the browser's `Host`, so the public hostname must be added to
|
||||||
liveness/readiness so a slow cold start (plugin activation, or a resumed sandbox) is not
|
`DSH_TRUSTED_HOSTS` in the template.
|
||||||
restarted mid-boot.
|
- **Probes target `/favicon.svg`**: `/` returns `401` without the launch token, so
|
||||||
|
a probe on `/` crash-loops the pod. `/favicon.svg` is unauthenticated, and a
|
||||||
|
`startupProbe` gates liveness/readiness so a slow cold start (plugin
|
||||||
|
activation, or a resumed sandbox) is not restarted mid-boot.
|
||||||
|
|
||||||
## Prerequisites
|
## Prerequisites
|
||||||
|
|
||||||
@@ -55,43 +83,74 @@ What it demonstrates, feature by feature:
|
|||||||
- agent-sandbox with extensions installed:
|
- agent-sandbox with extensions installed:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
VERSION=v1.0.7
|
VERSION=v1.0.7 # AGENT_SANDBOX_VERSION in versions.env
|
||||||
kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/download/${VERSION}/sandbox-with-extensions.yaml
|
kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/download/$VERSION/sandbox-with-extensions.yaml
|
||||||
kubectl -n agent-sandbox-system rollout status deployment --timeout=180s
|
kubectl -n agent-sandbox-system rollout status deployment --timeout=180s
|
||||||
```
|
```
|
||||||
|
|
||||||
Or run everything below in one shot with `./run-test-kind.sh` (expects a kind cluster + the
|
For step 9's SSO variant you additionally need the Gateway API CRDs and a
|
||||||
manifest above already applied; it creates and deletes the namespace itself).
|
Gateway implementation (`gatewayClassName: traefik` in the manifests); see the
|
||||||
|
[exposure README](manifests/exposure/README.md).
|
||||||
|
|
||||||
|
- Optional: run the whole resource + gateway walkthrough in one shot with
|
||||||
|
[`scripts/e2e-kind.sh`](scripts/e2e-kind.sh). It expects a kind cluster with
|
||||||
|
agent-sandbox installed, then creates and deletes the `dsh-demo` namespace
|
||||||
|
itself. It stops after the gateway layer (step 8); exposure is not applied.
|
||||||
|
|
||||||
## 1. Install the platform resources
|
## 1. Install the platform resources
|
||||||
|
|
||||||
First the namespace, then the shared provider key. The key reaches **every** tenant sandbox
|
First the namespace, then the shared provider keys. The keys reach **every**
|
||||||
(a warm-pool constraint: pod env is stamped before any user exists), so it is **never
|
tenant sandbox (a warm-pool constraint: pod env is stamped before any user
|
||||||
committed** — generate a fresh value per install:
|
exists), so they are **never committed** — generate fresh values per install.
|
||||||
|
`00-prereqs.yaml` already creates the empty `dsh-provider-keys` Secret, so use
|
||||||
|
an upsert:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
kubectl apply -f 00-prereqs.yaml
|
kubectl apply -f manifests/00-prereqs.yaml
|
||||||
kubectl -n dsh-demo create secret generic dsh-provider-keys \
|
kubectl -n dsh-demo create secret generic dsh-provider-keys \
|
||||||
--from-literal=DEEPSEEK_API_KEY="sk-..."
|
--from-literal=DEEPSEEK_API_KEY="sk-..." \
|
||||||
kubectl apply -f 10-sandboxtemplate.yaml -f 20-sandboxwarmpool.yaml
|
--dry-run=client -o yaml | kubectl apply -f -
|
||||||
```
|
```
|
||||||
|
|
||||||
Watch the two warm spares boot (first time pulls the image — subsequent operations never pay
|
Optional: additional providers (OpenRouter, Gemini, NVIDIA, OpenAI, Anthropic,
|
||||||
that again; that is the point of the pool):
|
...) go in a second Secret. It is referenced with `optional: true`, so the
|
||||||
|
sandboxes start without it, and every key becomes an env var:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl -n dsh-demo create secret generic dsh-provider-keys-extra \
|
||||||
|
--from-literal=OPENROUTER_API_KEY="sk-or-..." \
|
||||||
|
--from-literal=GEMINI_API_KEY="..." \
|
||||||
|
--from-literal=NVIDIA_API_KEY="nvapi-..." \
|
||||||
|
--dry-run=client -o yaml | kubectl apply -f -
|
||||||
|
```
|
||||||
|
|
||||||
|
Create both Secrets **before** the template/warm pool. A later Secret change
|
||||||
|
does not update running pods: delete the warm-pool spares and recreate claimed
|
||||||
|
pods to pick up new keys. Keys are shared by all tenants; a key saved in DSH's
|
||||||
|
credential store wins over env. See
|
||||||
|
[optional hardening](docs/optional-hardening.md) for the full env-name list and
|
||||||
|
the refresh procedure.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl apply -f manifests/10-sandboxtemplate.yaml -f manifests/20-sandboxwarmpool.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
Watch the two warm spares boot (first time pulls the image — subsequent operations
|
||||||
|
never pay that again; that is the point of the pool):
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
kubectl -n dsh-demo get sandboxes -l agents.x-k8s.io/warm-pool-sandbox -w
|
kubectl -n dsh-demo get sandboxes -l agents.x-k8s.io/warm-pool-sandbox -w
|
||||||
```
|
```
|
||||||
|
|
||||||
Wait until both show `READY: True`. Each spare already has its PVC bound and DSH Web serving
|
Wait until both show `READY: True`. Each spare already has its PVC bound and DSH
|
||||||
on port `3080`.
|
Web serving on port `3080`.
|
||||||
|
|
||||||
## 2. "Signup": claim an instance (~2s)
|
## 2. "Signup": claim an instance (~2s)
|
||||||
|
|
||||||
A user signing up is one small claim — `warmPoolRef` plus a pod label:
|
A user signing up is one small claim — `warmPoolRef` plus a pod label:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
kubectl apply -f 30-claim-alice.yaml
|
kubectl apply -f manifests/30-claim-alice.yaml
|
||||||
kubectl -n dsh-demo wait sandboxclaim dsh-alice --for=condition=Ready --timeout=60s
|
kubectl -n dsh-demo wait sandboxclaim dsh-alice --for=condition=Ready --timeout=60s
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -105,20 +164,21 @@ kubectl -n dsh-demo get sandboxes --show-labels
|
|||||||
|
|
||||||
Three things worth noticing:
|
Three things worth noticing:
|
||||||
|
|
||||||
- **The sandbox keeps its pool-generated name** (`dsh-pool-…`), not the claim's name —
|
- **The sandbox keeps its pool-generated name** (`dsh-pool-…`), not the claim's
|
||||||
adoption re-labels an existing warm sandbox rather than creating one. A platform must always
|
name — adoption re-labels an existing warm sandbox rather than creating one. A
|
||||||
resolve user → sandbox through `claim.status.sandbox.name`.
|
platform must always resolve user → sandbox through `claim.status.sandbox.name`.
|
||||||
- The `agents.x-k8s.io/warm-pool-sandbox` label is **gone** from the adopted sandbox (it left
|
- The `agents.x-k8s.io/warm-pool-sandbox` label is **gone** from the adopted
|
||||||
the pool's inventory), and the pool has already started replenishing back to 2 spares.
|
sandbox (it left the pool's inventory), and the pool has already started
|
||||||
|
replenishing back to 2 spares.
|
||||||
- The pod carries the claim's `additionalPodMetadata` label
|
- The pod carries the claim's `additionalPodMetadata` label
|
||||||
(`sandbox.users.io/dsh-user=alice`), and
|
(`sandbox.users.io/dsh-user=alice`), and
|
||||||
`kubectl -n dsh-demo get sandbox $SB -o jsonpath='{.status.serviceFQDN}'` shows the stable
|
`kubectl -n dsh-demo get sandbox $SB -o jsonpath='{.status.serviceFQDN}'` shows
|
||||||
DNS name the gateway proxies to.
|
the stable DNS name the gateway proxies to.
|
||||||
|
|
||||||
## 3. Talk to the instance
|
## 3. Talk to the instance
|
||||||
|
|
||||||
DSH Web needs its launch token on first contact — a bare `/` returns `401`. The token is in the
|
DSH Web needs its launch token on first contact — a bare `/` returns `401`. The
|
||||||
pod logs:
|
token is in the pod logs:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
kubectl -n dsh-demo logs "$SB" | grep -m1 '?token='
|
kubectl -n dsh-demo logs "$SB" | grep -m1 '?token='
|
||||||
@@ -127,16 +187,16 @@ kubectl -n dsh-demo port-forward "pod/$SB" 18081:3080
|
|||||||
# open http://localhost:18081/?token=... (from the log line)
|
# open http://localhost:18081/?token=... (from the log line)
|
||||||
```
|
```
|
||||||
|
|
||||||
> **Local port note:** DSH Web listens on `3080` **in-cluster**. If `3080` is already in use
|
> **Local port note:** DSH Web listens on `3080` **in-cluster**. If `3080` is
|
||||||
> on your workstation (e.g. by a local DSH), forward to a different local port (here `18081`),
|
> already in use on your workstation (e.g. by a local DSH), forward to a
|
||||||
> never `3080`. `localhost` is always in DSH's trusted-host list, so the forwarded request is
|
> different local port (here `18081`), never `3080`. `localhost` is always in
|
||||||
> accepted.
|
> DSH's trusted-host list, so the forwarded request is accepted.
|
||||||
|
|
||||||
> **Why the pod stays up:** this claim was applied by hand, so it carries no gateway ownership
|
> **Why the pod stays up:** this claim was applied by hand, so it carries no
|
||||||
> annotation (`aaas.example.com/managed-by`). The gateway's idle sweeper only suspends claims it
|
> gateway ownership annotation (`aaas.example.com/managed-by`). The gateway's idle
|
||||||
> created through `POST /users`; the resource-layer claims from these manifests are never
|
> sweeper only suspends claims it created through `POST /users`; the resource-layer
|
||||||
> touched. You can deploy the gateway (step 8) and still keep a sandbox you are port-forwarding
|
> claims from these manifests are never touched. You can deploy the gateway
|
||||||
> to.
|
> (step 8) and still keep a sandbox you are port-forwarding to.
|
||||||
|
|
||||||
## 4. Write state on the PVC
|
## 4. Write state on the PVC
|
||||||
|
|
||||||
@@ -161,20 +221,20 @@ kubectl -n dsh-demo wait sandbox "$SB" --for=condition=Ready --timeout=180s
|
|||||||
kubectl -n dsh-demo exec "$SB" -- cat /home/node/.dsh/marker.txt # "remember me"
|
kubectl -n dsh-demo exec "$SB" -- cat /home/node/.dsh/marker.txt # "remember me"
|
||||||
```
|
```
|
||||||
|
|
||||||
> **Why resume works:** the pod template sets `fsGroupChangePolicy: OnRootMismatch`. DSH requires
|
> **Why resume works:** the pod template sets `fsGroupChangePolicy: OnRootMismatch`.
|
||||||
> its `.credentials.yaml` to be `0600`, while the kubelet's default fsGroup policy widens every
|
> DSH requires its `.credentials.yaml` to be `0600`, while the kubelet's default
|
||||||
> file to `0660` on each pod start — a resumed sandbox would crash-loop. See
|
> fsGroup policy widens every file to `0660` on each pod start — a resumed sandbox
|
||||||
> [Troubleshooting](#troubleshooting) for the one-time fix on sandboxes created before this
|
> would crash-loop. See [Troubleshooting](#troubleshooting) for the one-time fix
|
||||||
> setting existed.
|
> on sandboxes created before this setting existed.
|
||||||
|
|
||||||
## 7. Injection policy rejects env-carrying claims
|
## 7. Injection policy rejects env-carrying claims
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
kubectl apply -f 40-claim-rejected.yaml
|
kubectl apply -f manifests/40-claim-rejected.yaml
|
||||||
kubectl -n dsh-demo get sandboxclaim dsh-mallory \
|
kubectl -n dsh-demo get sandboxclaim dsh-mallory \
|
||||||
-o jsonpath='{.status.conditions[?(@.type=="Ready")].reason}'
|
-o jsonpath='{.status.conditions[?(@.type=="Ready")].reason}'
|
||||||
# EnvVarsInjectionRejected — the claim never bound a sandbox
|
# EnvVarsInjectionRejected — the claim never bound a sandbox
|
||||||
kubectl delete -f 40-claim-rejected.yaml
|
kubectl delete -f manifests/40-claim-rejected.yaml
|
||||||
```
|
```
|
||||||
|
|
||||||
## 8. The gateway: signup, browser proxy, idle-suspend, wake-on-connect
|
## 8. The gateway: signup, browser proxy, idle-suspend, wake-on-connect
|
||||||
@@ -182,59 +242,72 @@ kubectl delete -f 40-claim-rejected.yaml
|
|||||||
### Deploy it
|
### Deploy it
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Published image (on kind, load it into the cluster; on a real cluster it is
|
# Published image (on kind, load it into the cluster; on a real cluster the
|
||||||
# pulled from GHCR):
|
# kubelet pulls it from GHCR):
|
||||||
docker pull ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway
|
GW=ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway:dsh-v0.2.1-rc.3-r4
|
||||||
kind load docker-image ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway --name <kind-cluster>
|
docker pull "$GW"
|
||||||
|
kind load docker-image "$GW" --name <kind-cluster>
|
||||||
|
|
||||||
# ... or build it locally and tag it exactly as 50-gateway.yaml expects:
|
# ... or build it locally and tag it exactly as manifests/50-gateway.yaml
|
||||||
docker build -t ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway gateway/
|
# expects (make gateway builds dsh-aaas-gateway:dev):
|
||||||
kind load docker-image ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway --name <kind-cluster>
|
make gateway
|
||||||
|
docker tag dsh-aaas-gateway:dev "$GW"
|
||||||
|
kind load docker-image "$GW" --name <kind-cluster>
|
||||||
|
|
||||||
kubectl apply -f 50-gateway.yaml
|
kubectl apply -f manifests/50-gateway.yaml
|
||||||
kubectl -n dsh-demo rollout status deploy/dsh-gateway --timeout=120s
|
kubectl -n dsh-demo rollout status deploy/dsh-gateway --timeout=120s
|
||||||
kubectl -n dsh-demo port-forward svc/dsh-gateway 18080:8080
|
kubectl -n dsh-demo port-forward svc/dsh-gateway 18080:8080
|
||||||
```
|
```
|
||||||
|
|
||||||
> The gateway binds `8080`; the local port-forward uses `18080` and never `3080`.
|
> The gateway binds `8080`; the local port-forward uses `18080` and never `3080`.
|
||||||
>
|
>
|
||||||
> **Re-deploying after a new image?** The workflow publishes the mutable `:gateway` tag and the
|
> **Image pull policy.** `manifests/50-gateway.yaml` uses
|
||||||
> Deployment uses `imagePullPolicy: IfNotPresent`, so a node that already cached the tag keeps
|
> `imagePullPolicy: IfNotPresent` with the release identity, so a locally built
|
||||||
> the old image across `kubectl apply`/`rollout restart`. Pin the digest for a reproducible
|
> image must be loaded into the cluster under exactly that name (kind:
|
||||||
> rollout (the workflow's `Inspect image` step and `docker buildx imagetools inspect` both show
|
> `kind load`). A node that already cached a different tag is the usual cause of
|
||||||
> it):
|
> "my new build is not running"; pin the digest for a reproducible rollout:
|
||||||
>
|
>
|
||||||
> ```bash
|
> ```bash
|
||||||
> kubectl -n dsh-demo set image deploy/dsh-gateway \
|
> kubectl -n dsh-demo set image deploy/dsh-gateway \
|
||||||
> gateway=ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox@sha256:<digest>
|
> gateway=ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway@sha256:<digest>
|
||||||
> kubectl -n dsh-demo rollout status deploy/dsh-gateway --timeout=120s
|
> kubectl -n dsh-demo rollout status deploy/dsh-gateway --timeout=120s
|
||||||
> ```
|
> ```
|
||||||
>
|
>
|
||||||
> **Applying over an existing install?** Run `kubectl diff -f 10-sandboxtemplate.yaml`
|
> **Applying over an existing install?** Run
|
||||||
> first. Cluster-specific fields such as `nodeAffinity` and the warm-pool replica count are
|
> `kubectl diff -f manifests/10-sandboxtemplate.yaml` first. Cluster-specific
|
||||||
> intentionally not in the repo manifests; preserve them with a targeted patch instead of a
|
> fields such as `nodeAffinity` and the warm-pool replica count are intentionally
|
||||||
|
> not in the repo manifests; preserve them with a targeted patch instead of a
|
||||||
> blind apply. Moving the probes to `/favicon.svg` is the crash-loop fix.
|
> blind apply. Moving the probes to `/favicon.svg` is the crash-loop fix.
|
||||||
>
|
>
|
||||||
> The template also sets `fsGroupChangePolicy: OnRootMismatch` (see the note in step 6).
|
> The template also sets `fsGroupChangePolicy: OnRootMismatch` (see the note in
|
||||||
> Existing `Sandbox` objects snapshot their pod template, so a template apply does not update
|
> step 6). Existing `Sandbox` objects snapshot their pod template, so a template
|
||||||
> them: patch each sandbox's `spec.podTemplate.spec.securityContext.fsGroupChangePolicy` and
|
> apply does not update them: patch each sandbox's
|
||||||
|
> `spec.podTemplate.spec.securityContext.fsGroupChangePolicy` and
|
||||||
> `chmod 600` its `.credentials.yaml`, or recreate it.
|
> `chmod 600` its `.credentials.yaml`, or recreate it.
|
||||||
|
>
|
||||||
|
> The optional gVisor/`nodeSelector` fields are pod-spec changes too: existing
|
||||||
|
> Sandboxes snapshot them, and warm-pool spares are not replaced by a template
|
||||||
|
> edit. See [optional hardening](docs/optional-hardening.md) for the patch and
|
||||||
|
> recreation commands. A provider-key Secret change likewise needs pod
|
||||||
|
> recreation to take effect.
|
||||||
|
|
||||||
### Browser flow
|
### Browser flow
|
||||||
|
|
||||||
Open <http://localhost:18080>. The landing page has a one-field signup form. Enter `bob`, and
|
Open <http://localhost:18080>. The landing page has a one-field signup form.
|
||||||
the gateway:
|
Enter `bob`, and the gateway:
|
||||||
|
|
||||||
1. creates the `SandboxClaim` over the warm pool,
|
1. creates the `SandboxClaim` over the warm pool,
|
||||||
2. waits for it to become `Ready`,
|
2. waits for it to become `Ready`,
|
||||||
3. sets an `HttpOnly`, `SameSite=Lax` cookie `dsh_gw_session=<user>:<token>` on the browser,
|
3. sets an `HttpOnly`, `SameSite=Lax` cookie `dsh_gw_session=<user>:<token>` on
|
||||||
|
the browser,
|
||||||
4. redirects to `/u/bob/`, where the DSH Web UI loads through the gateway.
|
4. redirects to `/u/bob/`, where the DSH Web UI loads through the gateway.
|
||||||
|
|
||||||
From then on every request under `/u/bob/` (assets, API calls, SSE, WebSocket) is proxied to
|
From then on every request under `/u/bob/` (assets, API calls, SSE, WebSocket) is
|
||||||
bob's sandbox. The DSH launch token and DSH's own auth cookie stay in the gateway; the browser
|
proxied to bob's sandbox. The DSH launch token and DSH's own auth cookie stay in
|
||||||
only holds the gateway cookie.
|
the gateway; the browser only holds the gateway cookie.
|
||||||
|
|
||||||
The same flow with `curl`, no JavaScript (the session cookie is the only credential):
|
The same flow with `curl`, no JavaScript (the session cookie is the only
|
||||||
|
credential):
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
JAR=$(mktemp)
|
JAR=$(mktemp)
|
||||||
@@ -252,48 +325,53 @@ curl -s -b "$JAR" -H 'Origin: http://localhost:18080' -X DELETE http://localhost
|
|||||||
```
|
```
|
||||||
|
|
||||||
API clients that prefer bearer tokens can use `POST /users` with a JSON body
|
API clients that prefer bearer tokens can use `POST /users` with a JSON body
|
||||||
(`{"user":"bob"}`) and the returned one-time token, `POST /login` to exchange a token for a
|
(`{"user":"bob"}`) and the returned one-time token, `POST /login` to exchange a
|
||||||
cookie, and `Authorization: Bearer <token>` on subsequent calls.
|
token for a cookie, and `Authorization: Bearer <token>` on subsequent calls.
|
||||||
|
|
||||||
### Gateway API
|
### Gateway API
|
||||||
|
|
||||||
| Route | Purpose |
|
| Route | Purpose |
|
||||||
| --- | --- |
|
| --- | --- |
|
||||||
| `GET /healthz` | Liveness/readiness; no cluster access |
|
| `GET /healthz` | Liveness/readiness; no cluster access |
|
||||||
| `GET /` | Landing page: signup, token login, logout |
|
| `GET /` | Landing page: signup, token login, logout. With SSO it auto-provisions and connects instead. |
|
||||||
| `POST /users` | Signup. JSON or form; `201` (JSON), `202` (waking), `303` (browser redirect), `409` if user exists; sets the session cookie |
|
| `POST /users` | Signup. JSON or form; `201` (JSON), `202` (waking), `303` (browser redirect), `409` if user exists; sets the session cookie |
|
||||||
| `POST /login` | Exchange a bearer token for the session cookie |
|
| `POST /login` | Exchange a bearer token for the session cookie |
|
||||||
| `POST /logout` | Clear the session cookie |
|
| `POST /logout` | Clear the session cookie |
|
||||||
|
| `POST /recreate` | SSO only: delete the authenticated user's claim (wiping workspace data) and provision a fresh one; `303` to `/u/{user}/`. Cookie-authenticated calls must be same-origin. |
|
||||||
| `GET /users/{user}` | Derived state (`Ready`/`Waking`/`Suspended`) |
|
| `GET /users/{user}` | Derived state (`Ready`/`Waking`/`Suspended`) |
|
||||||
| `DELETE /users/{user}` | Delete claim (cascades sandbox + PVC) |
|
| `DELETE /users/{user}` | Delete claim (cascades sandbox + PVC) |
|
||||||
| `GET /u/{user}` | `308` redirect to `/u/{user}/` |
|
| `GET /u/{user}` | `308` redirect to `/u/{user}/` |
|
||||||
| `GET /u/{user}/**` | HTTP/SSE proxy to the user's DSH Web |
|
| `GET /u/{user}/**` | HTTP/SSE proxy to the user's DSH Web |
|
||||||
| `WS /u/{user}/**` | WebSocket proxy to the user's DSH Web |
|
| `WS /u/{user}/**` | WebSocket proxy to the user's DSH Web |
|
||||||
|
|
||||||
|
The full API reference, configuration table, auth semantics and failure matrix
|
||||||
|
live in the [gateway README](gateway/README.md).
|
||||||
|
|
||||||
### How the DSH handshake works
|
### How the DSH handshake works
|
||||||
|
|
||||||
DSH does not expose an API for its launch token; it prints a URL containing `?token=…` in the
|
DSH does not expose an API for its launch token; it prints a URL containing
|
||||||
pod logs. For each `(user, pod UID, Host)` the gateway:
|
`?token=…` in the pod logs. For each `(user, pod UID, Host)` the gateway:
|
||||||
|
|
||||||
1. resolves the claim → sandbox → pod (annotation `agents.x-k8s.io/pod-name`, or the sandbox
|
1. resolves the claim → sandbox → pod (annotation `agents.x-k8s.io/pod-name`, or
|
||||||
name as pod name),
|
the sandbox name as pod name),
|
||||||
2. reads the pod's logs (last `DSH_LOG_TAIL_LINES` lines) and parses the most recent
|
2. reads the pod's logs (last `DSH_LOG_TAIL_LINES` lines) and parses the most
|
||||||
`dsh web: …?token=…` line,
|
recent `dsh web: …?token=…` line,
|
||||||
3. requests `http://<serviceFQDN>:3080/?token=…` with the browser's `Host`,
|
3. requests `http://<serviceFQDN>:3080/?token=…` with the browser's `Host`,
|
||||||
4. captures DSH's `dsh-auth-*` `Set-Cookie` response and caches it,
|
4. captures DSH's `dsh-auth-*` `Set-Cookie` response and caches it,
|
||||||
5. injects that cookie (and strips the browser's `Cookie`/`Authorization` and any `?token=`)
|
5. injects that cookie (and strips the browser's `Cookie`/`Authorization` and any
|
||||||
on every proxied HTTP and WebSocket request,
|
`?token=`) on every proxied HTTP and WebSocket request,
|
||||||
6. refreshes the handshake once if DSH answers `401`, and drops the cached cookie when the
|
6. refreshes the handshake once if DSH answers `401`, and drops the cached cookie
|
||||||
sandbox suspends or is deleted.
|
when the sandbox suspends or is deleted.
|
||||||
|
|
||||||
This is why the gateway's RBAC includes `pods` `get`/`list` and `pods/log` `get`. It is also
|
This is why the gateway's RBAC includes `pods` `get`/`list` and `pods/log`
|
||||||
the most fragile part of the design; see [Limitations](#limitations).
|
`get`. It is also the most fragile part of the design; see
|
||||||
|
[Limitations](#limitations).
|
||||||
|
|
||||||
### DSH_TRUSTED_HOSTS
|
### DSH_TRUSTED_HOSTS
|
||||||
|
|
||||||
DSH rejects requests whose `Host` is not allow-listed (`localhost` and `127.0.0.1` are always
|
DSH rejects requests whose `Host` is not allow-listed (`localhost` and
|
||||||
trusted). Because the gateway forwards the browser's `Host`, set the public hostname in the
|
`127.0.0.1` are always trusted). Because the gateway forwards the browser's
|
||||||
template:
|
`Host`, set the public hostname in the template:
|
||||||
|
|
||||||
```yaml
|
```yaml
|
||||||
env:
|
env:
|
||||||
@@ -301,36 +379,184 @@ env:
|
|||||||
value: "dsh.example.com" # comma or space separated; empty for port-forward only
|
value: "dsh.example.com" # comma or space separated; empty for port-forward only
|
||||||
```
|
```
|
||||||
|
|
||||||
The same applies to a local `docker run`: the printed token URL uses `DSH_PUBLIC_URL` if set,
|
The same applies to a local `docker run`: the printed token URL uses
|
||||||
otherwise `http://localhost:${DSH_PORT}`.
|
`DSH_PUBLIC_URL` if set, otherwise `http://localhost:<DSH_PORT>`.
|
||||||
|
|
||||||
### WebSocket
|
### WebSocket
|
||||||
|
|
||||||
The gateway is a Starlette ASGI app and proxies WebSocket upgrades with a bidirectional pump, so
|
The gateway is a Starlette ASGI app and proxies WebSocket upgrades with a
|
||||||
the DSH web GUI is fully interactive. Requirements:
|
bidirectional pump, so the DSH web GUI is fully interactive. Requirements:
|
||||||
|
|
||||||
- the ingress/load balancer in front of the gateway must pass `Upgrade`/`Connection` headers;
|
- the ingress/load balancer in front of the gateway must pass `Upgrade`/`Connection`
|
||||||
`60-ingress.yaml` is a starting point (Traefik handles upgrades by default),
|
headers; [`manifests/exposure/ingress/60-ingress.yaml`](manifests/exposure/ingress/60-ingress.yaml)
|
||||||
- idle suspension is disabled for users with in-flight proxied requests or open WebSockets, so
|
is a starting point (Traefik handles upgrades by default),
|
||||||
a long-running chat stream is not cut off mid-turn,
|
- idle suspension is disabled for users with in-flight proxied requests or open
|
||||||
- the upstream upgrade dials the sandbox **pod IP**, forwards no `Host`/`Origin`, and performs
|
WebSockets, so a long-running chat stream is not cut off mid-turn,
|
||||||
the launch-token handshake under that same `pod-IP:3080` authority. DSH fences on `Host`
|
- the upstream upgrade dials the sandbox **pod IP**, forwards no `Host`/`Origin`,
|
||||||
(only its bind address, loopback, or an allow-listed host passes) and binds its auth cookie to
|
and performs the launch-token handshake under that same `pod-IP:3080` authority.
|
||||||
the request authority (`payload.authority === authority`), while `websockets` derives `Host`
|
DSH fences on `Host` (only its bind address, loopback, or an allow-listed host
|
||||||
from the URI and cannot be overridden. A browser-host cookie or forwarded origin makes the
|
passes) and binds its auth cookie to the request authority
|
||||||
upgrade fail (`401`/`403`, surfaced as close `1011`), which the client retries forever as
|
(`payload.authority === authority`), while `websockets` derives `Host` from the
|
||||||
`[connection] connection lost, retry #N`,
|
URI and cannot be overridden. A browser-host cookie or forwarded origin makes
|
||||||
- missing/invalid gateway auth is rejected right after the upgrade with close code `4401`
|
the upgrade fail (`401`/`403`, surfaced as close `1011`), which the client
|
||||||
(WebSocket auth has no HTTP `401` path); cross-origin or Origin-less upgrades with the cookie
|
retries forever as `[connection] connection lost, retry #N`,
|
||||||
are rejected the same way (CSWSH protection), so clients must send a matching `Origin`;
|
- missing/invalid gateway auth is rejected right after the upgrade with close code
|
||||||
upstream bridging failures close with `1011`.
|
`4401` (WebSocket auth has no HTTP `401` path); cross-origin or Origin-less
|
||||||
|
upgrades with the cookie are rejected the same way (CSWSH protection), so
|
||||||
|
clients must send a matching `Origin`; upstream bridging failures close with
|
||||||
|
`1011`.
|
||||||
|
|
||||||
### Path routing
|
### Path routing
|
||||||
|
|
||||||
Each user's DSH Web is served under `/u/<user>/`. DSH injects `<base href="./">` and uses
|
Each user's DSH Web is served under `/u/<user>/`. DSH injects `<base href="./">`
|
||||||
relative asset/API URLs, so the UI works under a path prefix without a per-user hostname or
|
and uses relative asset/API URLs, so the UI works under a path prefix without a
|
||||||
wildcard DNS. The gateway also rewrites absolute-path `Location` headers into `/u/<user>/…` and
|
per-user hostname or wildcard DNS. The gateway also rewrites absolute-path
|
||||||
redirects `/u/<user>` to the trailing-slash form with `308` so relative URLs resolve correctly.
|
`Location` headers into `/u/<user>/…` and redirects `/u/<user>` to the
|
||||||
|
trailing-slash form with `308` so relative URLs resolve correctly.
|
||||||
|
|
||||||
|
## 9. Expose it: choose exactly one variant
|
||||||
|
|
||||||
|
The base manifests expose nothing outside the cluster (`dsh-gateway` is a
|
||||||
|
ClusterIP Service on port `8080`). Pick **one** variant — they are alternatives,
|
||||||
|
not a stack. Details, commands and the switch procedure are in the
|
||||||
|
[exposure README](manifests/exposure/README.md).
|
||||||
|
|
||||||
|
> Apply the base resources first, then the variant **by path**. Do not use
|
||||||
|
> `kubectl apply -R`: it would apply both variants and create two paths to the
|
||||||
|
> same gateway, which is unsafe once SSO is enabled.
|
||||||
|
|
||||||
|
Both variants also ship NetworkPolicies: `ingress/61-networkpolicy.yaml` limits
|
||||||
|
gateway ingress to Traefik pods, and `gateway-api-sso/75-networkpolicies.yaml`
|
||||||
|
limits the gateway to oauth2-proxy and oauth2-proxy egress to DNS, public HTTPS
|
||||||
|
and the gateway. They are part of the variant; see the
|
||||||
|
[exposure README](manifests/exposure/README.md) and
|
||||||
|
[optional hardening](docs/optional-hardening.md).
|
||||||
|
|
||||||
|
### Option A — plain Ingress (gateway's own session auth)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Replace the dsh.example.com host and TLS Secret (or drop the tls: block for
|
||||||
|
# plain HTTP), then:
|
||||||
|
kubectl apply -f manifests/exposure/ingress/
|
||||||
|
```
|
||||||
|
|
||||||
|
1. Set `DSH_TRUSTED_HOSTS` in `manifests/10-sandboxtemplate.yaml` to the same
|
||||||
|
public host (the gateway forwards the browser's `Host`) and re-apply the
|
||||||
|
template.
|
||||||
|
2. Set `COOKIE_SECURE=true` on the gateway once the Ingress serves HTTPS:
|
||||||
|
`kubectl -n dsh-demo set env deploy/dsh-gateway COOKIE_SECURE=true`.
|
||||||
|
(The base manifest keeps `false` so the plain-HTTP port-forward demo works.)
|
||||||
|
3. Verify: `curl -sI https://<host>/healthz` returns `200`; sign up in a browser
|
||||||
|
at `https://<host>/`.
|
||||||
|
4. `manifests/exposure/ingress/61-networkpolicy.yaml` is part of this variant:
|
||||||
|
it allows only Traefik pods to reach the gateway on `8080`. Adjust its
|
||||||
|
selector if your ingress controller uses a different pod label or
|
||||||
|
`hostNetwork`; if kubelet probes fail, see the hardening doc.
|
||||||
|
|
||||||
|
Keep the router free of buffering/compression middlewares: they break WebSocket
|
||||||
|
upgrades and streaming.
|
||||||
|
|
||||||
|
### Option B — Gateway API + GitHub SSO (oauth2-proxy)
|
||||||
|
|
||||||
|
`manifests/exposure/gateway-api-sso/` puts a TLS `Gateway` + `HTTPRoute` in
|
||||||
|
front of **oauth2-proxy**, which authenticates the browser against GitHub
|
||||||
|
(restricted to a configured organization) and forwards to the gateway with
|
||||||
|
`X-Forwarded-User` set. The gateway trusts that header only when
|
||||||
|
`AUTH_PROXY_HEADER` is set (SSO is **opt-in**).
|
||||||
|
|
||||||
|
Prerequisites:
|
||||||
|
|
||||||
|
- Gateway API CRDs and a Gateway implementation; the manifests assume
|
||||||
|
`gatewayClassName: traefik` (the CRD version is pinned as
|
||||||
|
`GATEWAY_API_VERSION` in `versions.env`).
|
||||||
|
- A GitHub OAuth App whose homepage/callback match the public FQDN
|
||||||
|
(`https://<host>/` and `https://<host>/oauth2/callback`).
|
||||||
|
- The oauth2-proxy credentials Secret (template:
|
||||||
|
`73-oauth2-proxy-secret.yaml.example`) and a TLS Secret for the Gateway
|
||||||
|
listener (template: `74-tls-secret.yaml.example`).
|
||||||
|
|
||||||
|
Steps:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. Real Secrets; the committed *.yaml.example files are templates only.
|
||||||
|
kubectl -n dsh-demo create secret generic oauth2-proxy \
|
||||||
|
--from-literal=client-id='<GitHub OAuth App client ID>' \
|
||||||
|
--from-literal=client-secret='<GitHub OAuth App client secret>' \
|
||||||
|
--from-literal=cookie-secret="$(python3 -c 'import secrets; print(secrets.token_urlsafe(32))')"
|
||||||
|
kubectl -n dsh-demo create secret tls dsh-demo-tls --cert=tls.crt --key=tls.key
|
||||||
|
|
||||||
|
# 2. Replace dsh-demo.example.com (70/71/72) and your-github-org (72).
|
||||||
|
|
||||||
|
# 3. Enable SSO on the gateway and trust the public host in the template.
|
||||||
|
kubectl -n dsh-demo set env deploy/dsh-gateway \
|
||||||
|
AUTH_PROXY_HEADER=X-Forwarded-User COOKIE_SECURE=true
|
||||||
|
# ... and set DSH_TRUSTED_HOSTS=dsh-demo.example.com in
|
||||||
|
# manifests/10-sandboxtemplate.yaml, then re-apply the template.
|
||||||
|
|
||||||
|
# 4. Apply the variant.
|
||||||
|
kubectl apply -f manifests/exposure/gateway-api-sso/
|
||||||
|
```
|
||||||
|
|
||||||
|
Verification:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl get gateway dsh-gateway -n dsh-demo -o yaml # Programmed=True
|
||||||
|
kubectl get httproute -n dsh-demo -o wide # backend oauth2-proxy:4180, Accepted
|
||||||
|
kubectl get deploy oauth2-proxy -n dsh-demo # 1/1 Ready
|
||||||
|
kubectl get endpoints oauth2-proxy -n dsh-demo # backend present
|
||||||
|
kubectl -n dsh-demo logs deploy/oauth2-proxy --tail=50 # no config errors
|
||||||
|
```
|
||||||
|
|
||||||
|
Browser check: open `https://<host>/` — you are redirected to GitHub, and after
|
||||||
|
signing in as a member of the allowed organization the gateway auto-provisions
|
||||||
|
your sandbox (the first page can show `Provisioning`) and connects you. A
|
||||||
|
non-member is denied before reaching the gateway. GitHub logins are lowercase
|
||||||
|
DNS-1123 labels, so they map 1:1 onto the gateway's `user` validation. The SSO
|
||||||
|
landing page also offers **Recreate sandbox** (`POST /recreate`), which deletes
|
||||||
|
the claim — wiping the workspace PVC — and provisions a fresh one.
|
||||||
|
|
||||||
|
Sign-out: the gateway `/logout` clears `dsh_gw_session`; oauth2-proxy's session
|
||||||
|
can be cleared at `/oauth2/sign_out`.
|
||||||
|
|
||||||
|
> **SSO trust boundary — read this before enabling it.** After SSO is on, the
|
||||||
|
> `dsh-gateway` Service must be reachable **only** through oauth2-proxy (it is
|
||||||
|
> ClusterIP). The variant's `75-networkpolicies.yaml` restricts pod traffic
|
||||||
|
> accordingly: the gateway accepts traffic only from oauth2-proxy pods,
|
||||||
|
> oauth2-proxy only from the Gateway implementation, and oauth2-proxy egress is
|
||||||
|
> limited to DNS, public HTTPS and the gateway. Anyone who can reach
|
||||||
|
> `dsh-gateway:8080` directly can still send `X-Forwarded-User: <victim>` and be
|
||||||
|
> treated as that user, so never add a second path (the Ingress variant, a
|
||||||
|
> shared-host port-forward, a permissive policy). To turn SSO off, remove the
|
||||||
|
> variable (`kubectl -n dsh-demo set env deploy/dsh-gateway AUTH_PROXY_HEADER-`)
|
||||||
|
> and delete the SSO variant; the gateway falls back to bearer-token/session
|
||||||
|
> auth.
|
||||||
|
|
||||||
|
### Switching variants
|
||||||
|
|
||||||
|
Delete the variant you had before applying the other one
|
||||||
|
(`kubectl delete -f manifests/exposure/ingress/`), revert
|
||||||
|
`AUTH_PROXY_HEADER`/`COOKIE_SECURE`, and verify with
|
||||||
|
`kubectl -n dsh-demo get ingress,gateway,httproute` that exactly one path to the
|
||||||
|
gateway remains.
|
||||||
|
|
||||||
|
## Optional hardening
|
||||||
|
|
||||||
|
The base walkthrough is deliberately portable. Four opt-in layers are documented
|
||||||
|
in [optional hardening](docs/optional-hardening.md):
|
||||||
|
|
||||||
|
- **gVisor** — uncomment `runtimeClassName: gvisor` before applying, or patch an
|
||||||
|
existing install. Requires a `gvisor` RuntimeClass and capable nodes.
|
||||||
|
`kubectl port-forward` does not work under gVisor; the gateway path does.
|
||||||
|
- **nodeSelector** — uncomment and replace the placeholder node label to pin
|
||||||
|
sandboxes and the gateway to a dedicated pool; add a toleration for tainted
|
||||||
|
pools.
|
||||||
|
- **Sandbox NetworkPolicy** —
|
||||||
|
`kubectl apply -f manifests/optional/network-policy/` allows only
|
||||||
|
gateway→sandbox on `3080` and sandbox→DNS/public `443`.
|
||||||
|
- **Provider keys** — create `dsh-provider-keys-extra` as shown in step 1.
|
||||||
|
|
||||||
|
Apply each independently and re-run the walkthrough. `kubectl apply -f manifests/`
|
||||||
|
is non-recursive, so the optional manifest is not applied by the base step.
|
||||||
|
|
||||||
## Troubleshooting
|
## Troubleshooting
|
||||||
|
|
||||||
@@ -346,36 +572,81 @@ redirects `/u/<user>` to the trailing-slash form with `308` so relative URLs res
|
|||||||
| Console spams `[connection] connection lost, retry #N` | The upstream WS upgrade is being rejected, surfaced as close `1011`. The gateway dials the sandbox **pod IP**, forwards no `Host`/`Origin`, and handshakes under the same `pod-IP:3080` authority DSH binds its cookie to; an older gateway image, or a sandbox with an empty `status.podIPs`, hits this loop |
|
| Console spams `[connection] connection lost, retry #N` | The upstream WS upgrade is being rejected, surfaced as close `1011`. The gateway dials the sandbox **pod IP**, forwards no `Host`/`Origin`, and handshakes under the same `pod-IP:3080` authority DSH binds its cookie to; an older gateway image, or a sandbox with an empty `status.podIPs`, hits this loop |
|
||||||
| `manifest.webmanifest` returns `401` in the console | Browsers fetch the Web App Manifest without credentials, so the gateway's session cookie is absent and the request fails closed. The UI is unaffected; only the PWA install prompt is lost |
|
| `manifest.webmanifest` returns `401` in the console | Browsers fetch the Web App Manifest without credentials, so the gateway's session cookie is absent and the request fails closed. The UI is unaffected; only the PWA install prompt is lost |
|
||||||
| Signup is slow / `202` then `Waking` | Warm pool exhausted (`kubectl get sandboxes -l agents.x-k8s.io/warm-pool-sandbox`); scale the pool |
|
| Signup is slow / `202` then `Waking` | Warm pool exhausted (`kubectl get sandboxes -l agents.x-k8s.io/warm-pool-sandbox`); scale the pool |
|
||||||
| `403` from the Kubernetes API in gateway logs | The gateway ServiceAccount is missing claim/sandbox/pod-log verbs (see `50-gateway.yaml`) |
|
| `403` from the Kubernetes API in gateway logs | The gateway ServiceAccount is missing claim/sandbox/pod-log verbs (see `manifests/50-gateway.yaml`) |
|
||||||
| Step 3: `pod "$SB" not found` after deploying the gateway | The claim was suspended by the idle sweeper. Claims created by the gateway carry `aaas.example.com/managed-by=dsh-gateway`; hand-applied claims (steps 1–7) do not and are never swept. Upgrade the gateway to a build that honors the marker, or resume the sandbox: `kubectl -n dsh-demo patch sandbox "$SB" --type merge -p '{"spec":{"operatingMode":"Running"}}'` |
|
| Gateway pod `ErrImagePull` / `ImagePullBackOff` (especially on kind) | The manifest uses `imagePullPolicy: IfNotPresent` with the release identity. Build with `make gateway`, retag to exactly the manifest's image, `kind load docker-image <that-ref>`, or set the Deployment image to your local tag / pinned digest. On a real cluster, make sure the GHCR package is readable (public or a pull secret) |
|
||||||
| Resume crash-loops: `credentials-local: … is readable beyond its owner (mode 660)` | The pod template lacks `fsGroupChangePolicy: OnRootMismatch`, so the kubelet's default fsGroup policy rewrote `.credentials.yaml` to `0660`; DSH requires `0600` and refuses to start. New installs get the policy from the template. For an existing sandbox: suspend it, run a throwaway pod on `dsh-home-$SB` to `chmod 600 /home/node/.dsh/.credentials.yaml`, patch the sandbox's `podTemplate` with the policy, then resume. |
|
| Step 3: `pod "$SB" not found` after deploying the gateway | The claim was suspended by the idle sweeper. Claims created by the gateway carry `aaas.example.com/managed-by=dsh-gateway`; hand-applied claims (steps 1–7) do not and are never swept. Resume the sandbox: `kubectl -n dsh-demo patch sandbox "$SB" --type merge -p '{"spec":{"operatingMode":"Running"}}'` |
|
||||||
|
| Resume crash-loops: `credentials-local: … is readable beyond its owner (mode 660)` | The pod template lacks `fsGroupChangePolicy: OnRootMismatch`, so the kubelet's default fsGroup policy rewrote `.credentials.yaml` to `0660`; DSH requires `0600` and refuses to start. New installs get the policy from the template. For an existing sandbox: suspend it, run a throwaway pod on `dsh-home-$SB` to `chmod 600 /home/node/.dsh/.credentials.yaml`, patch the sandbox's `podTemplate` with the policy, then resume |
|
||||||
|
| SSO: GitHub login succeeds but the gateway asks for a token / returns `401` to `/recreate` | `AUTH_PROXY_HEADER` is not set on `deploy/dsh-gateway` (SSO is opt-in). Set `AUTH_PROXY_HEADER=X-Forwarded-User` and confirm the request actually reaches the gateway through oauth2-proxy |
|
||||||
|
| SSO: gateway treats a caller as another user / bad `X-Forwarded-User` accepted | Trust boundary violation: something other than oauth2-proxy can reach `dsh-gateway:8080`. Remove the second path (Ingress/Gateway/port-forward), keep the Service ClusterIP, and only re-enable SSO once the proxy is the sole entry point. Never put a trusted identity header in front of a service that is directly reachable |
|
||||||
|
| SSO: oauth2-proxy `cookie_secret must be 16, 24, or 32 bytes ... but is 44 bytes` | The cookie secret was generated as standard base64. Use URL-safe unpadded base64 (see `73-oauth2-proxy-secret.yaml.example`) |
|
||||||
|
| SSO: redirect loop (`ERR_TOO_MANY_REDIRECTS`) | oauth2-proxy `--redirect-url` / callback does not match the public FQDN, or TLS is not terminating at the Gateway listener; fix the OAuth App URLs and the manifest placeholders |
|
||||||
|
| SSO: DSH rejects the public host behind the proxy | Add the public hostname to `DSH_TRUSTED_HOSTS` in `manifests/10-sandboxtemplate.yaml`; the gateway forwards the browser's `Host` |
|
||||||
|
| Pods `Pending` after setting `nodeSelector` | No node matches the label, the pool is full, or a tainted pool needs a toleration; inspect `kubectl describe pod` and the node labels |
|
||||||
|
| gVisor: `kubectl port-forward` fails or the connection is refused | Port-forward is incompatible with gVisor; reach DSH through the gateway (or a reachable Service/Sandbox Router) |
|
||||||
|
| Sandbox cannot reach a model API after applying the optional NetworkPolicy | The policy allows DNS and public TCP `443` only; a proxy, custom base URL, port `80` or non-443 endpoint needs an explicit egress rule |
|
||||||
|
| Gateway cannot reach a sandbox after applying an exposure NetworkPolicy | Check that the CNI enforces NetworkPolicy and that the Traefik/oauth2-proxy pod labels match; kubelet probes may also need the node CIDR |
|
||||||
|
| New provider key not visible in a sandbox | Pod env is stamped at creation: delete the warm-pool spares and recreate claimed pods. A key saved in DSH's credential store also wins over env |
|
||||||
|
|
||||||
## Limitations
|
## Limitations
|
||||||
|
|
||||||
Deliberately **not** production code:
|
Deliberately **not** production code:
|
||||||
|
|
||||||
- **Single replica.** Idle clocks, in-flight tracking, and the handshake cache are in memory.
|
- **Single replica.** Idle clocks, in-flight tracking, and the handshake cache
|
||||||
Running two gateway replicas would double-suspend or lose wake tracking; scale up only after
|
are in memory. Running two gateway replicas would double-suspend or lose wake
|
||||||
moving that state to a shared store.
|
tracking; scale up only after moving that state to a shared store.
|
||||||
- **Log scraping.** The launch token is recovered from pod logs. Log rotation, a restarted pod,
|
- **Log scraping.** The launch token is recovered from pod logs. Log rotation, a
|
||||||
or a changed DSH log format can break the handshake until the next `401` refresh.
|
restarted pod, or a changed DSH log format can break the handshake until the
|
||||||
- **Shared provider key.** `DEEPSEEK_API_KEY` is shared by all tenants via the template. Real
|
next `401` refresh.
|
||||||
multi-tenancy needs per-tenant credentials or an egress proxy.
|
- **Shared provider keys.** `DEEPSEEK_API_KEY` and any keys in the optional
|
||||||
- **No TLS in the gateway.** Terminate TLS at an Ingress/reverse proxy and set
|
`dsh-provider-keys-extra` Secret are shared by all tenants via the template.
|
||||||
|
Tenants can override a key through DSH's stored credential file on their PVC,
|
||||||
|
but the platform does not manage that. Real multi-tenancy needs per-tenant
|
||||||
|
credentials or an egress proxy.
|
||||||
|
- **No TLS in the gateway.** Terminate TLS at an Ingress or Gateway and set
|
||||||
`COOKIE_SECURE=true`.
|
`COOKIE_SECURE=true`.
|
||||||
- **In-memory user directory.** Users are claims + annotations; there is no database, password
|
- **SSO trust is hop-based.** `AUTH_PROXY_HEADER` trusts whoever sets the header.
|
||||||
reset, or admin API. Tokens are stored only as SHA-256 hashes in claim annotations.
|
The SSO variant ships `75-networkpolicies.yaml`, which restricts the gateway
|
||||||
- **Bearer tokens in URLs.** Avoid putting the signup token in a URL/log; the browser flow uses
|
to oauth2-proxy and oauth2-proxy egress to the gateway/DNS/public HTTPS; a
|
||||||
the `HttpOnly` cookie instead.
|
custom exposure without that policy has no signature, mTLS or network
|
||||||
|
enforcement.
|
||||||
|
- **Isolation is opt-in.** gVisor, the sandbox NetworkPolicy and the exposure
|
||||||
|
policies are opt-in; without them, sandboxes share the host kernel and can
|
||||||
|
reach the cluster network. The base example is a demo, not a hardened
|
||||||
|
multi-tenant platform.
|
||||||
|
- **OAuth org is the only authorization.** Any GitHub user in the configured
|
||||||
|
organization gets an auto-provisioned sandbox; there is no per-user allow-list,
|
||||||
|
quota, or admin API.
|
||||||
|
- **In-memory user directory.** Users are claims + annotations; there is no
|
||||||
|
database, password reset, or admin API. Tokens are stored only as SHA-256
|
||||||
|
hashes in claim annotations.
|
||||||
|
- **Bearer tokens in URLs.** Avoid putting the signup token in a URL/log; the
|
||||||
|
browser flow uses the `HttpOnly` cookie instead.
|
||||||
|
|
||||||
## Cost model
|
## Cost model
|
||||||
|
|
||||||
The same economics as the upstream pattern: suspend-when-idle plus warm pools turn an
|
The same economics as the upstream pattern: suspend-when-idle plus warm pools turn
|
||||||
always-on instance into a sub-second cold-start pause per return-after-absence, at a fraction
|
an always-on instance into a sub-second cold-start pause per return-after-absence,
|
||||||
of the cost.
|
at a fraction of the cost.
|
||||||
|
|
||||||
## Cleanup
|
## Cleanup
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
# Delete an exposure variant first if you applied one (optional; namespace
|
||||||
|
# deletion below removes everything):
|
||||||
|
kubectl delete -f manifests/exposure/ingress/ # or gateway-api-sso/
|
||||||
|
# Then the whole example:
|
||||||
kubectl delete namespace dsh-demo
|
kubectl delete namespace dsh-demo
|
||||||
```
|
```
|
||||||
|
|
||||||
|
If you created the oauth2-proxy/TLS Secrets or the GitHub OAuth App for step 9,
|
||||||
|
delete the Secrets with the namespace and revoke the OAuth App in GitHub.
|
||||||
|
|
||||||
|
## More
|
||||||
|
|
||||||
|
- [Gateway README](gateway/README.md) — API, configuration, auth/SSO, failure
|
||||||
|
semantics, limitations.
|
||||||
|
- [Architecture](docs/architecture.md) — container/sequence diagrams, ADRs,
|
||||||
|
security and RBAC, failure modes, evolution path.
|
||||||
|
- [Exposure README](manifests/exposure/README.md) — the two variants in detail.
|
||||||
|
- [Optional hardening](docs/optional-hardening.md) — gVisor, nodeSelector,
|
||||||
|
NetworkPolicy and extra provider keys, with verification and rollback.
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
# Architecture
|
# Architecture
|
||||||
|
|
||||||
This document describes the browser-usable **agents-as-a-service** platform in
|
This document describes the browser-usable **agents-as-a-service** platform in this example
|
||||||
[`examples/dsh-agents-as-a-service/`](../examples/dsh-agents-as-a-service/): how the gateway,
|
([README](../README.md), [manifests](../manifests/), [gateway](../gateway/)): how the gateway,
|
||||||
Kubernetes, and DSH Web fit together, and why the design looks the way it does.
|
Kubernetes, and DSH Web fit together, and why the design looks the way it does.
|
||||||
|
|
||||||
Scope: one Kubernetes namespace (`dsh-demo`), one shared DSH Web image, one gateway deployment,
|
Scope: one Kubernetes namespace (`dsh-demo`), one shared DSH Web image, one gateway deployment,
|
||||||
@@ -49,6 +49,31 @@ Responsibilities:
|
|||||||
| **DSH Web pod** | One `dsh web` process per user, state on the per-user PVC, listening on `3080`. |
|
| **DSH Web pod** | One `dsh web` process per user, state on the per-user PVC, listening on `3080`. |
|
||||||
| **Warm pool** | Keeps `N` Ready sandboxes so signup is a label flip instead of a cold start. |
|
| **Warm pool** | Keeps `N` Ready sandboxes so signup is a label flip instead of a cold start. |
|
||||||
|
|
||||||
|
## Exposure variants
|
||||||
|
|
||||||
|
The base manifests deploy the gateway as a ClusterIP Service (`dsh-gateway:8080`) and expose
|
||||||
|
nothing outside the cluster. Two **mutually exclusive** variants live under
|
||||||
|
[`manifests/exposure/`](../manifests/exposure/):
|
||||||
|
|
||||||
|
| Variant | Entry point | Authentication | Notes |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| [Ingress](../manifests/exposure/ingress/60-ingress.yaml) | Traefik `Ingress` + optional TLS | the gateway's own signup/session cookie | simplest; no external IdP; set `COOKIE_SECURE=true` with HTTPS |
|
||||||
|
| [Gateway API + SSO](../manifests/exposure/gateway-api-sso/70-gateway.yaml) | Gateway API `Gateway`/`HTTPRoute` terminating TLS | GitHub OAuth via [oauth2-proxy](../manifests/exposure/gateway-api-sso/72-oauth2-proxy.yaml), identity forwarded as `X-Forwarded-User` | requires `AUTH_PROXY_HEADER` on the gateway; the trust boundary applies |
|
||||||
|
|
||||||
|
They are alternatives, not a stack: applying both creates a second path to the same gateway
|
||||||
|
and defeats SSO. When SSO is enabled, the Service must stay ClusterIP and be reachable only
|
||||||
|
through oauth2-proxy — otherwise any client can send `X-Forwarded-User` itself and be treated
|
||||||
|
as that user. The gateway ignores the header unless `AUTH_PROXY_HEADER` is set. See the
|
||||||
|
[exposure README](../manifests/exposure/README.md) and the
|
||||||
|
[gateway README](../gateway/README.md#sso-opt-in).
|
||||||
|
|
||||||
|
Each variant also ships a NetworkPolicy: `ingress/61-networkpolicy.yaml` limits gateway ingress
|
||||||
|
to Traefik pods, and `gateway-api-sso/75-networkpolicies.yaml` limits the gateway to
|
||||||
|
oauth2-proxy, oauth2-proxy ingress to the Gateway implementation, and oauth2-proxy egress to
|
||||||
|
DNS/public HTTPS/the gateway. The SSO policy turns the trust boundary into pod-traffic
|
||||||
|
enforcement; the caveats (Traefik label, `hostNetwork`, kubelet probes) are in
|
||||||
|
[optional hardening](optional-hardening.md).
|
||||||
|
|
||||||
## Flows
|
## Flows
|
||||||
|
|
||||||
### Signup (browser)
|
### Signup (browser)
|
||||||
@@ -247,7 +272,7 @@ sequenceDiagram
|
|||||||
|
|
||||||
## Security and RBAC
|
## Security and RBAC
|
||||||
|
|
||||||
The gateway runs with a namespace-scoped ServiceAccount (`50-gateway.yaml`):
|
The gateway runs with a namespace-scoped ServiceAccount (`manifests/50-gateway.yaml`):
|
||||||
|
|
||||||
| API group | Resource | Verbs | Why |
|
| API group | Resource | Verbs | Why |
|
||||||
| --- | --- | --- | --- |
|
| --- | --- | --- | --- |
|
||||||
@@ -272,8 +297,14 @@ Notes and limits:
|
|||||||
- **TLS is external.** The gateway serves plain HTTP; terminate TLS at the Ingress and set
|
- **TLS is external.** The gateway serves plain HTTP; terminate TLS at the Ingress and set
|
||||||
`COOKIE_SECURE=true`. Without TLS, session cookies are replayable on the network.
|
`COOKIE_SECURE=true`. Without TLS, session cookies are replayable on the network.
|
||||||
- **Tenant isolation** is pod + PVC separation; all tenants share one namespace, one DSH binary,
|
- **Tenant isolation** is pod + PVC separation; all tenants share one namespace, one DSH binary,
|
||||||
and one `DEEPSEEK_API_KEY`. Add NetworkPolicies and per-tenant credentials before calling it
|
and the shared provider keys (`dsh-provider-keys` plus the optional
|
||||||
multi-tenant-grade.
|
`dsh-provider-keys-extra`). `manifests/optional/network-policy/` adds an opt-in sandbox
|
||||||
|
NetworkPolicy, and gVisor/`nodeSelector` are opt-in PodSpec fields. Add per-tenant
|
||||||
|
credentials before calling it multi-tenant-grade.
|
||||||
|
- **Optional hardening is opt-in.** gVisor, node selection, the sandbox/exposure NetworkPolicies
|
||||||
|
and extra provider keys are documented in [optional hardening](optional-hardening.md). The
|
||||||
|
base manifests stay portable (no gVisor RuntimeClass, no active node labels, no
|
||||||
|
NetworkPolicy and no extra Secret required).
|
||||||
- **No gateway rate limiting.** Signup can create claims until the cluster or warm pool is
|
- **No gateway rate limiting.** Signup can create claims until the cluster or warm pool is
|
||||||
exhausted; put limits at the ingress for real deployments.
|
exhausted; put limits at the ingress for real deployments.
|
||||||
|
|
||||||
@@ -288,6 +319,10 @@ Notes and limits:
|
|||||||
| DSH auth cookie expired mid-session | Proxy `401` once, then recovers | Gateway refreshes the handshake once |
|
| DSH auth cookie expired mid-session | Proxy `401` once, then recovers | Gateway refreshes the handshake once |
|
||||||
| Upstream pod name changed after resume | Cached cookie keyed by old pod UID is useless | Cache is invalidated on suspend; next request re-handshakes |
|
| Upstream pod name changed after resume | Cached cookie keyed by old pod UID is useless | Cache is invalidated on suspend; next request re-handshakes |
|
||||||
| Two gateway replicas | Double suspend/wake races | Keep `replicas: 1` until state is externalized |
|
| Two gateway replicas | Double suspend/wake races | Keep `replicas: 1` until state is externalized |
|
||||||
|
| Optional NetworkPolicy blocks DNS/probes | Sandbox 502/503, pod restarts, failed `getent` | Check the CNI, DNS selector/NodeLocal address and kubelet-probe traffic; see the hardening doc |
|
||||||
|
| gVisor + `kubectl port-forward` | Forward refuses or returns nothing | Port-forward is incompatible with gVisor; use the gateway or a reachable Service |
|
||||||
|
| Provider key added but missing in the pod | Requests still fail with `MISSING_CREDENTIAL` | Pod env is stamped at creation; recreate spares/pods. A stored credential wins over env |
|
||||||
|
| `nodeSelector` matches no node | Pod `Pending`, `0/N nodes are available` | Fix the label, pool capacity or taint/toleration; RuntimeClass scheduling may also constrain it |
|
||||||
|
|
||||||
## Upstream feature request: remove log scraping
|
## Upstream feature request: remove log scraping
|
||||||
|
|
||||||
513
examples/dsh-agents-as-a-service/docs/optional-hardening.md
Normal file
513
examples/dsh-agents-as-a-service/docs/optional-hardening.md
Normal file
@@ -0,0 +1,513 @@
|
|||||||
|
# Optional hardening: gVisor, nodeSelector, NetworkPolicy and provider keys
|
||||||
|
|
||||||
|
The base example is deliberately portable: it runs on kind and GKE without a
|
||||||
|
gVisor runtime, a tainted node pool or a NetworkPolicy-capable CNI, and it
|
||||||
|
boots with only the required `dsh-provider-keys` Secret. This document covers
|
||||||
|
four **opt-in** layers:
|
||||||
|
|
||||||
|
| Layer | Where it lives | What it changes |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| gVisor | commented `runtimeClassName` in the SandboxTemplates and Deployments | puts DSH and/or the gateway in a userspace kernel |
|
||||||
|
| nodeSelector | commented `nodeSelector` in the same files | pins pods to a chosen node pool |
|
||||||
|
| NetworkPolicy | `manifests/optional/network-policy/`, `manifests/exposure/*/` | limits sandbox, gateway and proxy connectivity |
|
||||||
|
| Provider keys | optional `dsh-provider-keys-extra` Secret | injects extra provider API keys into every sandbox |
|
||||||
|
|
||||||
|
Each layer is independent. Enable only what the cluster supports and test the
|
||||||
|
walkthrough after each change.
|
||||||
|
|
||||||
|
## 1. gVisor
|
||||||
|
|
||||||
|
### What it changes
|
||||||
|
|
||||||
|
The DSH sandbox and the gateway (and optionally oauth2-proxy) run under the
|
||||||
|
`gvisor` RuntimeClass. `agent-sandbox` passes `runtimeClassName` straight to the
|
||||||
|
PodSpec; no CRD change is involved.
|
||||||
|
|
||||||
|
### Prerequisites
|
||||||
|
|
||||||
|
- A `RuntimeClass` named `gvisor`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl get runtimeclass gvisor -o yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
- Nodes capable of running it (`runsc` installed and registered with the
|
||||||
|
container runtime). How you get there is cluster-specific:
|
||||||
|
- GKE Sandbox: create a node pool with `--sandbox=type=gvisor`; GKE labels
|
||||||
|
the nodes `sandbox.gke.io/runtime: gvisor`, taints them
|
||||||
|
`sandbox.gke.io/runtime=gvisor:NoSchedule`, and its `gvisor` RuntimeClass
|
||||||
|
applies the matching affinity and toleration automatically.
|
||||||
|
- Self-managed: install gVisor, create the `gvisor` RuntimeClass with
|
||||||
|
`handler: runsc`, and label/taint a dedicated node pool yourself.
|
||||||
|
- A scheduling plan: a gVisor RuntimeClass without capable nodes makes pods
|
||||||
|
fail to start. Use the RuntimeClass `scheduling` fields or the optional
|
||||||
|
`nodeSelector`/toleration described in section 2.
|
||||||
|
|
||||||
|
### Enable it
|
||||||
|
|
||||||
|
**Fresh install:** uncomment `runtimeClassName: gvisor` before applying:
|
||||||
|
|
||||||
|
- [`manifests/10-sandboxtemplate.yaml`](../manifests/10-sandboxtemplate.yaml)
|
||||||
|
- [`manifests/50-gateway.yaml`](../manifests/50-gateway.yaml)
|
||||||
|
- [`manifests/exposure/gateway-api-sso/72-oauth2-proxy.yaml`](../manifests/exposure/gateway-api-sso/72-oauth2-proxy.yaml)
|
||||||
|
(only with the SSO variant)
|
||||||
|
- [minimal sandbox-template.yaml](../../minimal-single-tenant/manifests/sandbox-template.yaml)
|
||||||
|
|
||||||
|
**Existing install:** a template edit does not update existing Sandboxes, and
|
||||||
|
warm-pool spares use `OnReplenish`, so patch explicitly. Plan a maintenance
|
||||||
|
window: claimed users are disconnected when their pod is recreated.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. The blueprint for future sandboxes.
|
||||||
|
kubectl -n dsh-demo patch sandboxtemplate dsh-web --type merge \
|
||||||
|
-p '{"spec":{"podTemplate":{"spec":{"runtimeClassName":"gvisor"}}}}'
|
||||||
|
|
||||||
|
# 2. Warm-pool spares: delete them; the pool replenishes from the template.
|
||||||
|
kubectl -n dsh-demo delete sandboxes -l agents.x-k8s.io/warm-pool-sandbox
|
||||||
|
|
||||||
|
# 3. Claimed sandboxes: snapshot their pod template, then suspend/resume.
|
||||||
|
for sb in $(kubectl -n dsh-demo get sandboxes \
|
||||||
|
-l '!agents.x-k8s.io/warm-pool-sandbox' -o jsonpath='{.items[*].metadata.name}'); do
|
||||||
|
kubectl -n dsh-demo patch sandbox "$sb" --type merge \
|
||||||
|
-p '{"spec":{"podTemplate":{"spec":{"runtimeClassName":"gvisor"}}}}'
|
||||||
|
kubectl -n dsh-demo patch sandbox "$sb" --type merge \
|
||||||
|
-p '{"spec":{"operatingMode":"Suspended"}}'
|
||||||
|
kubectl -n dsh-demo patch sandbox "$sb" --type merge \
|
||||||
|
-p '{"spec":{"operatingMode":"Running"}}'
|
||||||
|
done
|
||||||
|
|
||||||
|
# 4. The gateway (and oauth2-proxy when the SSO variant is installed).
|
||||||
|
kubectl -n dsh-demo patch deploy dsh-gateway --type merge \
|
||||||
|
-p '{"spec":{"template":{"spec":{"runtimeClassName":"gvisor"}}}}'
|
||||||
|
kubectl -n dsh-demo rollout status deploy/dsh-gateway --timeout=180s
|
||||||
|
# kubectl -n dsh-demo patch deploy oauth2-proxy --type merge \
|
||||||
|
# -p '{"spec":{"template":{"spec":{"runtimeClassName":"gvisor"}}}}'
|
||||||
|
```
|
||||||
|
|
||||||
|
If the pool's nodes are tainted and the RuntimeClass does not already add a
|
||||||
|
toleration, add the matching `tolerations` block (section 2) in the same patch
|
||||||
|
or in the manifests.
|
||||||
|
|
||||||
|
### Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
SB=$(kubectl -n dsh-demo get sandboxclaim dsh-alice -o jsonpath='{.status.sandbox.name}')
|
||||||
|
POD=$(kubectl -n dsh-demo get sandbox "$SB" \
|
||||||
|
-o jsonpath='{.metadata.annotations.agents\.x-k8s\.io/pod-name}'); POD=${POD:-$SB}
|
||||||
|
|
||||||
|
kubectl -n dsh-demo get pod "$POD" -o jsonpath='{.spec.runtimeClassName}{"\n"}'
|
||||||
|
# gvisor
|
||||||
|
kubectl -n dsh-demo exec "$POD" -- uname -r
|
||||||
|
kubectl -n dsh-demo get node "$(kubectl -n dsh-demo get pod "$POD" -o jsonpath='{.spec.nodeName}')" \
|
||||||
|
-o jsonpath='{.status.nodeInfo.kernelVersion}{"\n"}'
|
||||||
|
```
|
||||||
|
|
||||||
|
The pod's `runtimeClassName` is authoritative. The kernel release reported by
|
||||||
|
`uname -r` inside a gVisor pod is the sandbox kernel, not the host kernel; on
|
||||||
|
GKE it looks like `4.19.0-gvisor`.
|
||||||
|
|
||||||
|
### Caveats
|
||||||
|
|
||||||
|
- **`kubectl port-forward` does not work with gVisor.** The multi-tenant
|
||||||
|
gateway path (HTTP/WebSocket to the pod) works; the minimal example's
|
||||||
|
port-forward quickstart does not. Use the gateway example or a reachable
|
||||||
|
Service (NodePort/LoadBalancer/Sandbox Router) instead.
|
||||||
|
- gVisor has performance overhead and does not implement every syscall. Test
|
||||||
|
DSH and the agent tools you rely on (`git`, `gh`, `uv`, language toolchains).
|
||||||
|
- Node scheduling is the hard part. A `gvisor` RuntimeClass on a cluster with
|
||||||
|
no capable nodes leaves pods unschedulable or failing at container start.
|
||||||
|
- Existing Sandboxes are snapshots. The loop above is required for every
|
||||||
|
claimed sandbox; deleting the `SandboxClaim` would also delete its PVC and
|
||||||
|
workspace data, so do not use claim re-creation as a shortcut.
|
||||||
|
- A resumed sandbox starts cold; the startup probe gives it up to 120s.
|
||||||
|
|
||||||
|
### Roll back
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl -n dsh-demo patch sandboxtemplate dsh-web --type merge \
|
||||||
|
-p '{"spec":{"podTemplate":{"spec":{"runtimeClassName":null}}}}'
|
||||||
|
for sb in $(kubectl -n dsh-demo get sandboxes -o jsonpath='{.items[*].metadata.name}'); do
|
||||||
|
kubectl -n dsh-demo patch sandbox "$sb" --type merge \
|
||||||
|
-p '{"spec":{"podTemplate":{"spec":{"runtimeClassName":null}}}}'
|
||||||
|
done
|
||||||
|
kubectl -n dsh-demo delete sandboxes -l agents.x-k8s.io/warm-pool-sandbox
|
||||||
|
kubectl -n dsh-demo patch deploy dsh-gateway --type merge \
|
||||||
|
-p '{"spec":{"template":{"spec":{"runtimeClassName":null}}}}'
|
||||||
|
# kubectl -n dsh-demo patch deploy oauth2-proxy --type merge \
|
||||||
|
# -p '{"spec":{"template":{"spec":{"runtimeClassName":null}}}}'
|
||||||
|
```
|
||||||
|
|
||||||
|
Recreate the pods (delete spares, suspend/resume claimed sandboxes, let the
|
||||||
|
Deployment roll out) so the cleared field takes effect.
|
||||||
|
|
||||||
|
## 2. nodeSelector
|
||||||
|
|
||||||
|
### What it changes
|
||||||
|
|
||||||
|
Pods get a `nodeSelector` so they land only on nodes carrying the chosen label.
|
||||||
|
The same field is available in both SandboxTemplates and both Deployments;
|
||||||
|
the shipped value is **commented out** because active cluster-specific
|
||||||
|
scheduling does not belong in the repository.
|
||||||
|
|
||||||
|
### Choose a label
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl get nodes --show-labels
|
||||||
|
```
|
||||||
|
|
||||||
|
- **GKE Sandbox:** the nodes carry `sandbox.gke.io/runtime: gvisor` and are
|
||||||
|
tainted `sandbox.gke.io/runtime=gvisor:NoSchedule`. With
|
||||||
|
`runtimeClassName: gvisor`, GKE already schedules the pod onto those nodes;
|
||||||
|
add a `nodeSelector` only when you need a specific sandbox pool or
|
||||||
|
ComputeClass (`cloud.google.com/compute-class: <name>`).
|
||||||
|
- **Self-managed:** label the pool you want, for example
|
||||||
|
`dsh.example.com/node-pool: gvisor`, and use that label.
|
||||||
|
- **General pinning:** any stable node-pool label works. Keep the real key and
|
||||||
|
value in a local patch; the manifests use `dsh.example.com/node-pool` as an
|
||||||
|
obvious placeholder.
|
||||||
|
|
||||||
|
### Enable it
|
||||||
|
|
||||||
|
Uncomment and replace the placeholder before a fresh apply:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
nodeSelector:
|
||||||
|
dsh.example.com/node-pool: gvisor
|
||||||
|
```
|
||||||
|
|
||||||
|
For an existing install, patch the template, each claimed Sandbox and the
|
||||||
|
Deployments, then recreate pods exactly as in section 1:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl -n dsh-demo patch sandboxtemplate dsh-web --type merge \
|
||||||
|
-p '{"spec":{"podTemplate":{"spec":{"nodeSelector":{"dsh.example.com/node-pool":"gvisor"}}}}}'
|
||||||
|
# ... same warm-pool delete / claimed-sandbox suspend-resume / Deployment
|
||||||
|
# patch and rollout as in section 1, with nodeSelector instead of
|
||||||
|
# runtimeClassName.
|
||||||
|
```
|
||||||
|
|
||||||
|
If the selected nodes are tainted, add a matching toleration. This is the
|
||||||
|
companion block; do **not** uncomment it unless the pool is actually tainted:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
tolerations:
|
||||||
|
- key: dsh.example.com/node-pool
|
||||||
|
operator: Equal
|
||||||
|
value: gvisor
|
||||||
|
effect: NoSchedule
|
||||||
|
```
|
||||||
|
|
||||||
|
GKE Sandbox's taint is `sandbox.gke.io/runtime=gvisor:NoSchedule`; the GKE
|
||||||
|
`gvisor` RuntimeClass adds that toleration automatically, so a manual copy is
|
||||||
|
only needed when you schedule without the RuntimeClass or with a different
|
||||||
|
pool.
|
||||||
|
|
||||||
|
### Verify and troubleshoot
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl -n dsh-demo get pod "$POD" -o wide
|
||||||
|
kubectl -n dsh-demo get pod "$POD" -o jsonpath='{.spec.nodeSelector}{"\n"}'
|
||||||
|
kubectl -n dsh-demo describe pod "$POD" | sed -n '/Events:/,$p'
|
||||||
|
```
|
||||||
|
|
||||||
|
- `Pending` with `0/N nodes are available`: the label is wrong, the pool is
|
||||||
|
full, or the pod needs a toleration for a taint.
|
||||||
|
- A pod-level `nodeSelector` is merged with the RuntimeClass
|
||||||
|
`scheduling.nodeSelector`. If both are set and conflict, no node matches.
|
||||||
|
- Labels changed after scheduling do not move running pods. Patch/recreate.
|
||||||
|
- A `nodeSelector` that excludes every gVisor node is a valid way to keep the
|
||||||
|
gateway on a normal pool while the sandboxes run under gVisor.
|
||||||
|
|
||||||
|
### Roll back
|
||||||
|
|
||||||
|
Remove the field from the manifests, or patch it to `null`, then recreate the
|
||||||
|
pods:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl -n dsh-demo patch sandboxtemplate dsh-web --type merge \
|
||||||
|
-p '{"spec":{"podTemplate":{"spec":{"nodeSelector":null}}}}'
|
||||||
|
# ... same per-Sandbox and Deployment loop as section 1.
|
||||||
|
```
|
||||||
|
|
||||||
|
## 3. NetworkPolicy
|
||||||
|
|
||||||
|
The base example does **not** manage NetworkPolicy: the SandboxTemplate is
|
||||||
|
`networkPolicyManagement: Unmanaged`, and the gateway has no policy. That keeps
|
||||||
|
the walkthrough portable and independent of the sandbox-router; the opt-in
|
||||||
|
policies below are standard Kubernetes `NetworkPolicy` objects.
|
||||||
|
|
||||||
|
### Optional sandbox policy
|
||||||
|
|
||||||
|
[`manifests/optional/network-policy/10-sandbox-networkpolicy.yaml`](../manifests/optional/network-policy/10-sandbox-networkpolicy.yaml)
|
||||||
|
selects every Sandbox pod through the agent-sandbox tracking label
|
||||||
|
`agents.x-k8s.io/sandbox-name-hash`. It applies to existing warm-pool and
|
||||||
|
claimed sandboxes without recreating them.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl apply -f manifests/optional/network-policy/
|
||||||
|
kubectl delete -f manifests/optional/network-policy/ # rollback
|
||||||
|
```
|
||||||
|
|
||||||
|
What it allows:
|
||||||
|
|
||||||
|
| Direction | Allowed | Blocked |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| Ingress | gateway pods (`app: dsh-gateway`) on TCP `3080` | everything else, including other sandboxes and cluster Services |
|
||||||
|
| Egress | UDP/TCP `53` to kube-dns and NodeLocal DNSCache; TCP `443` to public IPs | RFC1918, CGNAT, link-local/metadata, cluster Services/API, plain HTTP, other ports |
|
||||||
|
|
||||||
|
Verify from inside the pods (NetworkPolicy does not affect `kubectl exec`
|
||||||
|
itself, only the traffic the commands generate):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
SB=$(kubectl -n dsh-demo get sandboxclaim dsh-alice -o jsonpath='{.status.sandbox.name}')
|
||||||
|
POD=$(kubectl -n dsh-demo get sandbox "$SB" \
|
||||||
|
-o jsonpath='{.metadata.annotations.agents\.x-k8s\.io/pod-name}'); POD=${POD:-$SB}
|
||||||
|
|
||||||
|
# Allowed: gateway -> sandbox.
|
||||||
|
kubectl -n dsh-demo exec deploy/dsh-gateway -- python -c \
|
||||||
|
"import urllib.request; print(urllib.request.urlopen('http://$POD:3080/favicon.svg', timeout=5).status)"
|
||||||
|
|
||||||
|
# Allowed: DNS and public HTTPS from the sandbox.
|
||||||
|
kubectl -n dsh-demo exec "$POD" -- getent hosts api.deepseek.com
|
||||||
|
kubectl -n dsh-demo exec "$POD" -- curl -fsS -o /dev/null -w '%{http_code}\n' https://example.com
|
||||||
|
|
||||||
|
# Denied: another sandbox cannot reach this one on 3080 (expect a timeout).
|
||||||
|
OTHER_SB=$(kubectl -n dsh-demo get sandboxclaim dsh-bob -o jsonpath='{.status.sandbox.name}')
|
||||||
|
OTHER_POD=$(kubectl -n dsh-demo get sandbox "$OTHER_SB" \
|
||||||
|
-o jsonpath='{.metadata.annotations.agents\.x-k8s\.io/pod-name}'); OTHER_POD=${OTHER_POD:-$OTHER_SB}
|
||||||
|
kubectl -n dsh-demo exec "$OTHER_POD" -- curl --max-time 3 -sS "http://$POD:3080/favicon.svg"
|
||||||
|
```
|
||||||
|
|
||||||
|
Caveats:
|
||||||
|
|
||||||
|
- **The CNI must enforce NetworkPolicy.** If it does not, the file applies but
|
||||||
|
has no effect.
|
||||||
|
- DNS: the default assumes CoreDNS/kube-dns pods labelled `k8s-app: kube-dns`
|
||||||
|
and covers NodeLocal DNSCache at `169.254.20.10`. A different DNS layout
|
||||||
|
needs a selector/`ipBlock` edit.
|
||||||
|
- **Kubelet probes** from the node are CNI-dependent. If a pod starts failing
|
||||||
|
its probes, allow the node CIDR on the probe port or remove the ingress
|
||||||
|
policy.
|
||||||
|
- The policy blocks cluster Services and the Kubernetes API from the sandbox.
|
||||||
|
DSH does not need them, but an agent tool that does needs an explicit rule.
|
||||||
|
- It allows any public host on TCP `443`, not an FQDN allowlist. It is network
|
||||||
|
isolation, not data-loss prevention.
|
||||||
|
- Custom provider base URLs, an HTTP proxy, port `80`, SSH `git` (port `22`) or
|
||||||
|
a non-443 provider endpoint need explicit egress additions.
|
||||||
|
- Do not combine this standalone policy with `networkPolicyManagement: Managed`
|
||||||
|
unless you migrate the rules into `spec.networkPolicy`. NetworkPolicies are
|
||||||
|
additive; the controller's Managed secure default would also be present.
|
||||||
|
- `kubectl port-forward` still works for debugging (the connection is local to
|
||||||
|
the pod), except under gVisor.
|
||||||
|
|
||||||
|
### Exposure-variant policies
|
||||||
|
|
||||||
|
Each exposure variant ships its own policy. They are part of the variant, not a
|
||||||
|
separate opt-in step:
|
||||||
|
|
||||||
|
- [`manifests/exposure/ingress/61-networkpolicy.yaml`](../manifests/exposure/ingress/61-networkpolicy.yaml)
|
||||||
|
allows only Traefik pods (`app.kubernetes.io/name: traefik`) to reach the
|
||||||
|
gateway on `8080`.
|
||||||
|
- [`manifests/exposure/gateway-api-sso/75-networkpolicies.yaml`](../manifests/exposure/gateway-api-sso/75-networkpolicies.yaml)
|
||||||
|
allows only oauth2-proxy to reach the gateway, only the Gateway
|
||||||
|
implementation (Traefik) to reach oauth2-proxy, and limits oauth2-proxy
|
||||||
|
egress to DNS, public HTTPS (GitHub) and the gateway.
|
||||||
|
|
||||||
|
The SSO policy is what turns the trust boundary in the
|
||||||
|
[exposure README](../manifests/exposure/README.md) into network enforcement.
|
||||||
|
The `dsh-gateway` Service must still stay ClusterIP; do not apply both exposure
|
||||||
|
variants.
|
||||||
|
|
||||||
|
Verify the negative case from a sandbox pod (it is not the proxy):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl -n dsh-demo exec "$POD" -- curl --max-time 3 -sS http://dsh-gateway:8080/healthz
|
||||||
|
# expected: timeout/connection failure
|
||||||
|
```
|
||||||
|
|
||||||
|
Then verify the real browser path through the Ingress or Gateway, and confirm
|
||||||
|
the gateway/proxy pods are `Ready`. If Traefik uses `hostNetwork` or a
|
||||||
|
different pod label, replace the `podSelector` with the correct selector or an
|
||||||
|
`ipBlock`. As with the sandbox policy, kubelet probes can be affected on some
|
||||||
|
CNIs.
|
||||||
|
|
||||||
|
### Gateway egress (documented, not shipped)
|
||||||
|
|
||||||
|
The gateway needs DNS, the Kubernetes API, and TCP `3080` to sandbox pods. The
|
||||||
|
API server endpoint is cluster-specific (Service CIDR or node IP/port), so an
|
||||||
|
egress policy is **not** shipped. If your cluster has a stable API-server CIDR
|
||||||
|
and port, add an egress policy like this (replace the placeholders; verify the
|
||||||
|
port, commonly `443` or `6443`):
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
apiVersion: networking.k8s.io/v1
|
||||||
|
kind: NetworkPolicy
|
||||||
|
metadata:
|
||||||
|
name: dsh-gateway-egress
|
||||||
|
namespace: dsh-demo
|
||||||
|
spec:
|
||||||
|
podSelector:
|
||||||
|
matchLabels:
|
||||||
|
app: dsh-gateway
|
||||||
|
policyTypes:
|
||||||
|
- Egress
|
||||||
|
egress:
|
||||||
|
- to:
|
||||||
|
- namespaceSelector:
|
||||||
|
matchLabels:
|
||||||
|
kubernetes.io/metadata.name: kube-system
|
||||||
|
podSelector:
|
||||||
|
matchLabels:
|
||||||
|
k8s-app: kube-dns
|
||||||
|
ports:
|
||||||
|
- protocol: UDP
|
||||||
|
port: 53
|
||||||
|
- protocol: TCP
|
||||||
|
port: 53
|
||||||
|
- to:
|
||||||
|
- ipBlock:
|
||||||
|
cidr: 10.96.0.1/32 # placeholder: API-server Service/cluster endpoint
|
||||||
|
ports:
|
||||||
|
- protocol: TCP
|
||||||
|
port: 443
|
||||||
|
- to:
|
||||||
|
- podSelector:
|
||||||
|
matchExpressions:
|
||||||
|
- key: agents.x-k8s.io/sandbox-name-hash
|
||||||
|
operator: Exists
|
||||||
|
ports:
|
||||||
|
- protocol: TCP
|
||||||
|
port: 3080
|
||||||
|
```
|
||||||
|
|
||||||
|
Cilium users can express the API server as `toEntities: kube-apiserver` (or
|
||||||
|
`toServices`) instead of a CIDR. Do not apply the sample blindly: a wrong CIDR
|
||||||
|
or port silently breaks signup/proxy.
|
||||||
|
|
||||||
|
## 4. Optional provider keys
|
||||||
|
|
||||||
|
### How injection works
|
||||||
|
|
||||||
|
The SandboxTemplate injects two Secrets via `envFrom`:
|
||||||
|
|
||||||
|
- `dsh-provider-keys` — required by the walkthrough; holds
|
||||||
|
`DEEPSEEK_API_KEY`.
|
||||||
|
- `dsh-provider-keys-extra` — optional; referenced with `optional: true`, so a
|
||||||
|
missing Secret does not block sandbox startup. Every key in it becomes an
|
||||||
|
environment variable in every sandbox.
|
||||||
|
|
||||||
|
Create the extra Secret before the warm pool (pod env is stamped at creation):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl -n dsh-demo create secret generic dsh-provider-keys-extra \
|
||||||
|
--from-literal=OPENROUTER_API_KEY="sk-or-..." \
|
||||||
|
--from-literal=GEMINI_API_KEY="..." \
|
||||||
|
--from-literal=NVIDIA_API_KEY="nvapi-..." \
|
||||||
|
--dry-run=client -o yaml | kubectl apply -f -
|
||||||
|
```
|
||||||
|
|
||||||
|
Use the exact env-var names (uppercase, underscores) — invalid Secret keys are
|
||||||
|
skipped by the kubelet with an event. Do not duplicate a key across the two
|
||||||
|
Secrets; when the same key exists twice, the later `envFrom` entry wins.
|
||||||
|
|
||||||
|
### Supported names (curated)
|
||||||
|
|
||||||
|
These are the names DSH v0.2.1-rc.3 resolves through its packaged provider
|
||||||
|
catalog; the catalog has more entries, these are the common ones:
|
||||||
|
|
||||||
|
| Provider | Environment variable |
|
||||||
|
| --- | --- |
|
||||||
|
| DeepSeek | `DEEPSEEK_API_KEY` |
|
||||||
|
| OpenRouter | `OPENROUTER_API_KEY` |
|
||||||
|
| Google Gemini | `GEMINI_API_KEY` |
|
||||||
|
| NVIDIA | `NVIDIA_API_KEY` |
|
||||||
|
| OpenAI | `OPENAI_API_KEY` |
|
||||||
|
| Anthropic | `ANTHROPIC_API_KEY` |
|
||||||
|
| Groq | `GROQ_API_KEY` |
|
||||||
|
| xAI | `XAI_API_KEY` |
|
||||||
|
| Mistral | `MISTRAL_API_KEY` |
|
||||||
|
| Together | `TOGETHER_API_KEY` |
|
||||||
|
| Cerebras | `CEREBRAS_API_KEY` |
|
||||||
|
| Vercel AI Gateway | `AI_GATEWAY_API_KEY` |
|
||||||
|
| Hugging Face | `HF_TOKEN` |
|
||||||
|
|
||||||
|
Aliases to avoid: `GOOGLE_API_KEY`, `HUGGINGFACE_API_KEY` and
|
||||||
|
`VERCEL_AI_GATEWAY_API_KEY`. The packaged catalog looks for `GEMINI_API_KEY`,
|
||||||
|
`HF_TOKEN` and `AI_GATEWAY_API_KEY` instead.
|
||||||
|
|
||||||
|
### Picking up a key change
|
||||||
|
|
||||||
|
A Secret update does not change running pods. The base key and extra keys are
|
||||||
|
both snapshotted at container start:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Warm-pool spares: delete them so the pool recreates them from the template.
|
||||||
|
kubectl -n dsh-demo delete sandboxes -l agents.x-k8s.io/warm-pool-sandbox
|
||||||
|
|
||||||
|
# Running claimed sandboxes: delete the pod (the Sandbox controller recreates
|
||||||
|
# it) or suspend/resume the sandbox.
|
||||||
|
POD=... # resolve it as in section 3
|
||||||
|
kubectl -n dsh-demo delete pod "$POD"
|
||||||
|
```
|
||||||
|
|
||||||
|
New claims get the new values automatically. Confirm names without printing
|
||||||
|
secret values:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl -n dsh-demo exec "$POD" -- sh -c '
|
||||||
|
for k in DEEPSEEK_API_KEY OPENROUTER_API_KEY GEMINI_API_KEY NVIDIA_API_KEY; do
|
||||||
|
if [ -n "$(printenv "$k" || true)" ]; then echo "$k=set"; else echo "$k=unset"; fi
|
||||||
|
done'
|
||||||
|
```
|
||||||
|
|
||||||
|
### Precedence and shared-key limits
|
||||||
|
|
||||||
|
- DSH's stored credential file (`$DSH_HOME/.credentials.yaml`, mode `0600`)
|
||||||
|
**wins over the environment**. If a tenant saved a key through the DSH UI,
|
||||||
|
changing the Secret will not override it.
|
||||||
|
- Both Secrets are shared by **every** tenant, and any agent can read its own
|
||||||
|
environment. This is a warm-pool constraint, not tenant isolation. A tenant
|
||||||
|
can override its own key through the DSH credential store on its PVC, but the
|
||||||
|
platform does not manage that; real multi-tenancy needs per-tenant
|
||||||
|
credentials or an egress proxy.
|
||||||
|
- The optional Secret is created imperatively; the repository never contains a
|
||||||
|
real key. Keep it out of Git and out of shell history where possible.
|
||||||
|
|
||||||
|
## 5. Verification summary
|
||||||
|
|
||||||
|
Required locally for any change to this example:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
make check # versions, example layout, manifest YAML shape
|
||||||
|
yamllint .
|
||||||
|
make test # gateway unit tests (gateway code is untouched)
|
||||||
|
```
|
||||||
|
|
||||||
|
CI additionally parses every manifest and runs `kubectl apply
|
||||||
|
--dry-run=server` against the real agent-sandbox CRDs. The weekly kind e2e
|
||||||
|
runs the base walkthrough; it also proves the optional extra Secret may be
|
||||||
|
absent.
|
||||||
|
|
||||||
|
Manual checks, on a cluster with the relevant CNI/runtime:
|
||||||
|
|
||||||
|
- gVisor: pod `runtimeClassName`, `uname -r` differs from the node kernel,
|
||||||
|
gateway proxy works, port-forward limitation acknowledged.
|
||||||
|
- nodeSelector: pods land on the intended nodes; no `Pending` events.
|
||||||
|
- Sandbox policy: the allow/deny matrix above.
|
||||||
|
- Exposure policies: browser path works; a non-proxy pod cannot reach the
|
||||||
|
gateway; probes stay healthy.
|
||||||
|
- Provider keys: set/unset output (never the values) and the refresh procedure.
|
||||||
|
|
||||||
|
## 6. Rollback summary
|
||||||
|
|
||||||
|
| Layer | Rollback |
|
||||||
|
| --- | --- |
|
||||||
|
| Sandbox policy | `kubectl delete -f manifests/optional/network-policy/` |
|
||||||
|
| Exposure policy | delete the exposure variant directory |
|
||||||
|
| gVisor | patch `runtimeClassName: null`, recreate pods |
|
||||||
|
| nodeSelector | patch `nodeSelector: null`, recreate pods |
|
||||||
|
| Provider keys | delete the extra Secret or a key, recreate pods |
|
||||||
|
|
||||||
|
No PVC or workspace data is touched by any rollback. Deleting a `SandboxClaim`
|
||||||
|
is different: it cascades to the Sandbox **and** its PVC, so do not use claim
|
||||||
|
deletion to apply a pod-spec change.
|
||||||
11
examples/dsh-agents-as-a-service/gateway/.dockerignore
Normal file
11
examples/dsh-agents-as-a-service/gateway/.dockerignore
Normal file
@@ -0,0 +1,11 @@
|
|||||||
|
# Runtime image only needs the app and its pinned runtime requirements.
|
||||||
|
__pycache__/
|
||||||
|
*.py[cod]
|
||||||
|
.pytest_cache/
|
||||||
|
.ruff_cache/
|
||||||
|
.venv/
|
||||||
|
test_gateway.py
|
||||||
|
requirements-dev.txt
|
||||||
|
pyproject.toml
|
||||||
|
README.md
|
||||||
|
coverage.xml
|
||||||
@@ -4,7 +4,7 @@ A single-file, browser-usable control + data plane for
|
|||||||
[DeepSeek Harness Web (DSH)](https://github.com/deepseek-ai/deepseek-harness)
|
[DeepSeek Harness Web (DSH)](https://github.com/deepseek-ai/deepseek-harness)
|
||||||
sandboxes running behind the `agent-sandbox` CRDs.
|
sandboxes running behind the `agent-sandbox` CRDs.
|
||||||
|
|
||||||
It does three things:
|
It does four things:
|
||||||
|
|
||||||
1. **Users** — creates a `SandboxClaim` from a warm pool, mints a bearer token
|
1. **Users** — creates a `SandboxClaim` from a warm pool, mints a bearer token
|
||||||
(only its SHA-256 is stored as a claim annotation), reports derived state,
|
(only its SHA-256 is stored as a claim annotation), reports derived state,
|
||||||
@@ -15,10 +15,15 @@ It does three things:
|
|||||||
3. **Reverse proxy** — proxies HTTP (including SSE) and WebSocket traffic to
|
3. **Reverse proxy** — proxies HTTP (including SSE) and WebSocket traffic to
|
||||||
the sandbox's DSH Web server on `:3080`, including DSH's own cookie
|
the sandbox's DSH Web server on `:3080`, including DSH's own cookie
|
||||||
handshake.
|
handshake.
|
||||||
|
4. **Optional SSO** — when `AUTH_PROXY_HEADER` is set, trusts the identity
|
||||||
|
header an authenticating reverse proxy (for example oauth2-proxy) forwards,
|
||||||
|
auto-provisions that user, and exposes `POST /recreate` for a clean restart.
|
||||||
|
|
||||||
DSH is pinned to `dsh-v0.2.1-rc.2` in the example manifests. It binds `:3080`,
|
The DSH version pin lives in [`versions.env`](../../../versions.env)
|
||||||
serves `/`, gates `/api` behind a `dsh-auth-*` cookie and requires a WebSocket
|
(`DSH_VERSION` plus the optional repository `RELEASE_REVISION`); the example
|
||||||
at `/api/remote.mux`.
|
manifests and published images use the release identity built from it (currently
|
||||||
|
`dsh-v0.2.1-rc.3-r4`). DSH binds `:3080`, serves `/`, gates `/api` behind a
|
||||||
|
`dsh-auth-*` cookie and requires a WebSocket at `/api/remote.mux`.
|
||||||
|
|
||||||
## Flow
|
## Flow
|
||||||
|
|
||||||
@@ -58,10 +63,11 @@ token. A failed handshake returns `503` with `Retry-After: 10`.
|
|||||||
| Method | Path | Auth | Purpose |
|
| Method | Path | Auth | Purpose |
|
||||||
| --- | --- | --- | --- |
|
| --- | --- | --- | --- |
|
||||||
| `GET` | `/healthz` | none | Liveness/readiness. Never touches the cluster. |
|
| `GET` | `/healthz` | none | Liveness/readiness. Never touches the cluster. |
|
||||||
| `GET` | `/` | none | Landing page with signup + token login forms. |
|
| `GET` | `/` | none | Landing page with signup + token login forms. With SSO it auto-provisions the proxy-identified user and redirects to their workspace. |
|
||||||
| `POST` | `/users` | none | Signup. JSON body `{"user": "<dns-1123 label>"}` or form field `user`. JSON returns `201` when Ready, `202` when accepted but still Waking; form returns `303` to `/u/<user>/`. Both set the session cookie. `409` if the user exists, `400` if invalid, `413` if the body is too large. |
|
| `POST` | `/users` | none | Signup. JSON body `{"user": "<dns-1123 label>"}` or form field `user`. JSON returns `201` when Ready, `202` when accepted but still Waking; form returns `303` to `/u/<user>/`. Both set the session cookie. `409` if the user exists, `400` if invalid, `413` if the body is too large. |
|
||||||
| `POST` | `/login` | none | Form (`user`, `token`) returns `303` to `/u/<user>/`; JSON returns `200`. Sets the session cookie. `401` on a bad token, `404` for an unknown user. |
|
| `POST` | `/login` | none | Form (`user`, `token`) returns `303` to `/u/<user>/`; JSON returns `200`. Sets the session cookie. `401` on a bad token, `404` for an unknown user. |
|
||||||
| `POST` | `/logout` | none | Clears the session cookie and `303`s to `/`. |
|
| `POST` | `/logout` | none | Clears the session cookie and `303`s to `/`. |
|
||||||
|
| `POST` | `/recreate` | trusted proxy header | **SSO only.** Deletes the authenticated user's claim (cascading sandbox + PVC, so workspace data is wiped), waits out finalizers, provisions a fresh claim and `303`s to `/u/<user>/`. `401` without a valid identity header; cookie-authenticated calls must be same-origin. |
|
||||||
| `GET` | `/users/{user}` | bearer or cookie | Derived state: `Provisioning`, `Waking`, `Ready`, `Suspending`, `Suspended`. |
|
| `GET` | `/users/{user}` | bearer or cookie | Derived state: `Provisioning`, `Waking`, `Ready`, `Suspending`, `Suspended`. |
|
||||||
| `DELETE` | `/users/{user}` | bearer or cookie | Delete the claim (sandbox + PVC cascade). Cookie-auth mutations require a same-origin `Origin`/`Referer`. |
|
| `DELETE` | `/users/{user}` | bearer or cookie | Delete the claim (sandbox + PVC cascade). Cookie-auth mutations require a same-origin `Origin`/`Referer`. |
|
||||||
| `GET` | `/u/{user}` | — | `308` redirect to `/u/{user}/`. |
|
| `GET` | `/u/{user}` | — | `308` redirect to `/u/{user}/`. |
|
||||||
@@ -86,6 +92,41 @@ token. A failed handshake returns `503` with `Retry-After: 10`.
|
|||||||
absolute paths are rewritten under `/u/<user>/`, and any `token` query
|
absolute paths are rewritten under `/u/<user>/`, and any `token` query
|
||||||
parameter is stripped.
|
parameter is stripped.
|
||||||
|
|
||||||
|
### SSO (opt-in)
|
||||||
|
|
||||||
|
By default the gateway is the only authenticator: users sign up with a token or
|
||||||
|
the session cookie, and the gateway ignores all identity headers. Set
|
||||||
|
`AUTH_PROXY_HEADER` to enable SSO behind an authenticating reverse proxy such as
|
||||||
|
oauth2-proxy:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl -n dsh-demo set env deploy/dsh-gateway AUTH_PROXY_HEADER=X-Forwarded-User
|
||||||
|
```
|
||||||
|
|
||||||
|
When set, the gateway treats a request carrying that header as authenticated as
|
||||||
|
that user, provided the value is a valid DNS-1123 label; the header is never
|
||||||
|
trusted when the variable is unset. The landing page then auto-provisions the
|
||||||
|
user (creates their claim idempotently, mints a rotated session token) and
|
||||||
|
redirects to `/u/<user>/`, and `POST /recreate` becomes available for a clean
|
||||||
|
restart. The base manifests leave it unset on purpose; the
|
||||||
|
[SSO exposure variant](../manifests/exposure/gateway-api-sso/) documents how to
|
||||||
|
enable it on the Deployment (`kubectl set env`). That variant also ships
|
||||||
|
`75-networkpolicies.yaml`, which restricts the gateway to oauth2-proxy and
|
||||||
|
oauth2-proxy's egress to the gateway, DNS and public HTTPS.
|
||||||
|
|
||||||
|
> **Trust boundary — this is the security-critical part.** A trusted identity
|
||||||
|
> header is only safe when **nothing else can reach the gateway**. The gateway
|
||||||
|
> Service must stay ClusterIP and be reachable exclusively through the
|
||||||
|
> authenticating proxy; if a client can reach `dsh-gateway:8080` directly (a
|
||||||
|
> second Ingress/Gateway/Route, a port-forward on a shared host, a broad
|
||||||
|
> NetworkPolicy), it can send `X-Forwarded-User` itself and impersonate any user.
|
||||||
|
> The variant's `75-networkpolicies.yaml` restricts pod traffic to the proxy; a
|
||||||
|
> custom exposure without an equivalent policy has no network enforcement.
|
||||||
|
> Treat `AUTH_PROXY_HEADER` as "the proxy already authenticated this request",
|
||||||
|
> not as an authentication mechanism of its own. Disable SSO by removing the
|
||||||
|
> variable: the header is then ignored and bearer-token/session auth applies
|
||||||
|
> again.
|
||||||
|
|
||||||
### Proxy failure semantics
|
### Proxy failure semantics
|
||||||
|
|
||||||
| Situation | Result |
|
| Situation | Result |
|
||||||
@@ -123,6 +164,7 @@ can also be injected into `create_app` for tests.
|
|||||||
| `MAX_BODY_MB` | `32` | Maximum buffered request body (and `413` threshold). |
|
| `MAX_BODY_MB` | `32` | Maximum buffered request body (and `413` threshold). |
|
||||||
| `HTTP_TIMEOUT` | `300` | Upstream connect/read timeout in seconds. Proxied streams (SSE, long responses) keep the connect/write budget but have **no read timeout**, so an idle event stream is not cut every five minutes. |
|
| `HTTP_TIMEOUT` | `300` | Upstream connect/read timeout in seconds. Proxied streams (SSE, long responses) keep the connect/write budget but have **no read timeout**, so an idle event stream is not cut every five minutes. |
|
||||||
| `SWEEP_INTERVAL` | `15` | Seconds between idle sweeps. |
|
| `SWEEP_INTERVAL` | `15` | Seconds between idle sweeps. |
|
||||||
|
| `AUTH_PROXY_HEADER` | unset | Opt-in SSO: name of the trusted identity header set by an authenticating reverse proxy (e.g. `X-Forwarded-User`). Unset disables SSO and the header is never trusted. Enable only when the Service is reachable exclusively through that proxy (see [SSO](#sso-opt-in)). |
|
||||||
|
|
||||||
### Required RBAC
|
### Required RBAC
|
||||||
|
|
||||||
@@ -147,12 +189,21 @@ python -m ruff check .
|
|||||||
uvicorn gateway:create_app --factory --host 0.0.0.0 --port 8080
|
uvicorn gateway:create_app --factory --host 0.0.0.0 --port 8080
|
||||||
```
|
```
|
||||||
|
|
||||||
Container image:
|
Container image, from the repository root:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker build -t dsh-gateway examples/dsh-agents-as-a-service/gateway
|
make gateway # builds dsh-aaas-gateway:dev from this directory
|
||||||
```
|
```
|
||||||
|
|
||||||
|
The equivalent direct build is
|
||||||
|
`docker build -t dsh-aaas-gateway:dev examples/dsh-agents-as-a-service/gateway`.
|
||||||
|
For kind, retag it to the image reference in
|
||||||
|
`examples/dsh-agents-as-a-service/manifests/50-gateway.yaml` and `kind load` it.
|
||||||
|
Published images are nested below the main image as
|
||||||
|
`<GHCR_IMAGE>/dsh-agents-as-a-service/gateway:<identity>`, where the identity
|
||||||
|
comes from [`versions.env`](../../../versions.env) — see
|
||||||
|
[`docs/versioning.md`](../../../docs/versioning.md).
|
||||||
|
|
||||||
The image runs as UID `10000`, sets `PYTHONDONTWRITEBYTECODE=1` and
|
The image runs as UID `10000`, sets `PYTHONDONTWRITEBYTECODE=1` and
|
||||||
`HOME=/tmp` (read-only-rootfs compatible), listens on `8080`, and disables
|
`HOME=/tmp` (read-only-rootfs compatible), listens on `8080`, and disables
|
||||||
uvicorn's access log so bearer tokens in URLs are not logged.
|
uvicorn's access log so bearer tokens in URLs are not logged.
|
||||||
@@ -182,3 +233,9 @@ uvicorn's access log so bearer tokens in URLs are not logged.
|
|||||||
re-runs the handshake once and replays the request.
|
re-runs the handshake once and replays the request.
|
||||||
* **Only DSH on `:3080` is proxied.** There is no dashboard/API-port split and
|
* **Only DSH on `:3080` is proxied.** There is no dashboard/API-port split and
|
||||||
no platform API key injection.
|
no platform API key injection.
|
||||||
|
* **SSO trusts a header.** With `AUTH_PROXY_HEADER` set, any client that can
|
||||||
|
reach the gateway Service directly can spoof the identity header. The gateway
|
||||||
|
cannot enforce the trust boundary itself: keep the Service ClusterIP and
|
||||||
|
reachable only through the authenticating proxy. The SSO exposure variant
|
||||||
|
ships a NetworkPolicy that restricts pod traffic to the proxy; without it
|
||||||
|
there is no mTLS or signed identity assertion.
|
||||||
|
|||||||
@@ -195,9 +195,7 @@ def derive_state(sandbox: dict | None) -> str:
|
|||||||
|
|
||||||
def token_matches(claim: dict, token: str | None) -> bool:
|
def token_matches(claim: dict, token: str | None) -> bool:
|
||||||
"""Timing-safe check of a user token against the stored SHA-256."""
|
"""Timing-safe check of a user token against the stored SHA-256."""
|
||||||
want = (claim.get("metadata", {}).get("annotations") or {}).get(
|
want = (claim.get("metadata", {}).get("annotations") or {}).get(TOKEN_ANNOTATION, "")
|
||||||
TOKEN_ANNOTATION, ""
|
|
||||||
)
|
|
||||||
if not token or not want:
|
if not token or not want:
|
||||||
return False
|
return False
|
||||||
got = hashlib.sha256(token.encode()).hexdigest()
|
got = hashlib.sha256(token.encode()).hexdigest()
|
||||||
@@ -331,6 +329,11 @@ class Settings:
|
|||||||
# Connection retries while a Ready pod is still binding its port.
|
# Connection retries while a Ready pod is still binding its port.
|
||||||
upstream_attempts: int = 3
|
upstream_attempts: int = 3
|
||||||
upstream_retry_delay: float = 1.0
|
upstream_retry_delay: float = 1.0
|
||||||
|
# Optional identity header (e.g. "X-Forwarded-User") set by an authenticating
|
||||||
|
# reverse proxy such as oauth2-proxy. When configured, requests carrying a
|
||||||
|
# valid user in this header are treated as authenticated (SSO); unset (None)
|
||||||
|
# disables SSO entirely and the header is never trusted.
|
||||||
|
auth_proxy_header: str | None = None
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def from_env(cls) -> Settings:
|
def from_env(cls) -> Settings:
|
||||||
@@ -348,6 +351,7 @@ class Settings:
|
|||||||
max_body_bytes=_env_int("MAX_BODY_MB", 32) * 1024 * 1024,
|
max_body_bytes=_env_int("MAX_BODY_MB", 32) * 1024 * 1024,
|
||||||
http_timeout=_env_float("HTTP_TIMEOUT", 300.0),
|
http_timeout=_env_float("HTTP_TIMEOUT", 300.0),
|
||||||
sweep_interval=_env_float("SWEEP_INTERVAL", 15.0),
|
sweep_interval=_env_float("SWEEP_INTERVAL", 15.0),
|
||||||
|
auth_proxy_header=os.environ.get("AUTH_PROXY_HEADER") or None,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -496,13 +500,23 @@ class Kubernetes:
|
|||||||
{"spec": {"operatingMode": mode}},
|
{"spec": {"operatingMode": mode}},
|
||||||
)
|
)
|
||||||
|
|
||||||
|
async def patch_token(self, user: str, token_hash: str) -> None:
|
||||||
|
await self._call(
|
||||||
|
"crd",
|
||||||
|
"patch_namespaced_custom_object",
|
||||||
|
GROUP,
|
||||||
|
VERSION,
|
||||||
|
self.namespace,
|
||||||
|
"sandboxclaims",
|
||||||
|
claim_name(user),
|
||||||
|
{"metadata": {"annotations": {TOKEN_ANNOTATION: token_hash}}},
|
||||||
|
)
|
||||||
|
|
||||||
# -- pods and logs -----------------------------------------------------
|
# -- pods and logs -----------------------------------------------------
|
||||||
|
|
||||||
async def get_pod(self, name: str) -> dict | None:
|
async def get_pod(self, name: str) -> dict | None:
|
||||||
try:
|
try:
|
||||||
pod = await self._call(
|
pod = await self._call("core", "read_namespaced_pod", name, self.namespace)
|
||||||
"core", "read_namespaced_pod", name, self.namespace
|
|
||||||
)
|
|
||||||
except ApiException as exc:
|
except ApiException as exc:
|
||||||
if exc.status == 404:
|
if exc.status == 404:
|
||||||
return None
|
return None
|
||||||
@@ -512,9 +526,7 @@ class Kubernetes:
|
|||||||
# so callers can keep using ``pod.get(...)``.
|
# so callers can keep using ``pod.get(...)``.
|
||||||
return k8s_client.ApiClient().sanitize_for_serialization(pod)
|
return k8s_client.ApiClient().sanitize_for_serialization(pod)
|
||||||
|
|
||||||
async def read_pod_logs(
|
async def read_pod_logs(self, name: str, container: str, tail_lines: int) -> str:
|
||||||
self, name: str, container: str, tail_lines: int
|
|
||||||
) -> str:
|
|
||||||
result = await self._call(
|
result = await self._call(
|
||||||
"core",
|
"core",
|
||||||
"read_namespaced_pod_log",
|
"read_namespaced_pod_log",
|
||||||
@@ -571,17 +583,66 @@ class Gateway:
|
|||||||
return token, user
|
return token, user
|
||||||
return None, None
|
return None, None
|
||||||
|
|
||||||
|
def remote_user(self, headers) -> str | None:
|
||||||
|
"""The user asserted by a trusted reverse proxy, when SSO is enabled.
|
||||||
|
|
||||||
|
Honored only when ``auth_proxy_header`` is configured, and only when the
|
||||||
|
value is a valid dns-1123 label — a spoofed or malformed value is never
|
||||||
|
trusted.
|
||||||
|
"""
|
||||||
|
name = self.settings.auth_proxy_header
|
||||||
|
if not name:
|
||||||
|
return None
|
||||||
|
raw = (headers.get(name) or "").strip()
|
||||||
|
return raw if valid_user(raw) else None
|
||||||
|
|
||||||
def _authorized(
|
def _authorized(
|
||||||
self,
|
self,
|
||||||
claim: dict,
|
claim: dict,
|
||||||
token: str | None,
|
token: str | None,
|
||||||
cookie_user: str | None,
|
cookie_user: str | None,
|
||||||
user: str,
|
user: str,
|
||||||
|
remote: str | None = None,
|
||||||
) -> bool:
|
) -> bool:
|
||||||
|
if remote == user:
|
||||||
|
return True # identity asserted by the authenticating proxy
|
||||||
if cookie_user is not None and cookie_user != user:
|
if cookie_user is not None and cookie_user != user:
|
||||||
return False # a session for another user never authorizes this one
|
return False # a session for another user never authorizes this one
|
||||||
return token_matches(claim, token)
|
return token_matches(claim, token)
|
||||||
|
|
||||||
|
async def _sso_session(self, remote: str, headers, cookies, response: Response) -> dict | None:
|
||||||
|
"""Provision ``remote`` idempotently and mint/refresh a session token.
|
||||||
|
|
||||||
|
The authenticating proxy has already verified ``remote``, so it is
|
||||||
|
trusted. A brand-new user is provisioned (claim created, token minted);
|
||||||
|
an existing user whose session cookie is missing or no longer matches
|
||||||
|
gets a rotated token so a fresh cookie can be issued. A still-valid
|
||||||
|
session is reused untouched. Provisioning is non-blocking: the sandbox
|
||||||
|
is awaited by the proxy path (``_resolve_sandbox``), so the landing page
|
||||||
|
can report "Provisioning" instead of stalling for ``WAKE_TIMEOUT``.
|
||||||
|
"""
|
||||||
|
claim = await self.k8s.get_claim(remote)
|
||||||
|
token, cookie_user = self.credentials(headers, cookies)
|
||||||
|
if claim is not None and cookie_user == remote and token_matches(claim, token):
|
||||||
|
return claim
|
||||||
|
token = secrets.token_urlsafe(32)
|
||||||
|
token_hash = hashlib.sha256(token.encode()).hexdigest()
|
||||||
|
is_new = claim is None
|
||||||
|
if is_new:
|
||||||
|
try:
|
||||||
|
await self.k8s.create_claim(remote, token_hash, self.settings.pool)
|
||||||
|
except UserExists:
|
||||||
|
# Lost a concurrent signup race: rotate the existing token.
|
||||||
|
is_new = False
|
||||||
|
await self.k8s.patch_token(remote, token_hash)
|
||||||
|
else:
|
||||||
|
await self._touch(remote)
|
||||||
|
else:
|
||||||
|
await self.k8s.patch_token(remote, token_hash)
|
||||||
|
await self._touch(remote)
|
||||||
|
self.set_session(response, remote, token)
|
||||||
|
return await self.k8s.get_claim(remote) if is_new else claim
|
||||||
|
|
||||||
@staticmethod
|
@staticmethod
|
||||||
def _mutation_from_cookie(headers, method: str, cookie_user: str | None) -> bool:
|
def _mutation_from_cookie(headers, method: str, cookie_user: str | None) -> bool:
|
||||||
return (
|
return (
|
||||||
@@ -613,9 +674,7 @@ class Gateway:
|
|||||||
while True:
|
while True:
|
||||||
async with self.state.lock:
|
async with self.state.lock:
|
||||||
if not self.state.maintenance.get(user):
|
if not self.state.maintenance.get(user):
|
||||||
self.state.in_flight[user] = (
|
self.state.in_flight[user] = self.state.in_flight.get(user, 0) + 1
|
||||||
self.state.in_flight.get(user, 0) + 1
|
|
||||||
)
|
|
||||||
self.state.last_activity[user] = time.time()
|
self.state.last_activity[user] = time.time()
|
||||||
return True
|
return True
|
||||||
if time.monotonic() > deadline:
|
if time.monotonic() > deadline:
|
||||||
@@ -624,9 +683,7 @@ class Gateway:
|
|||||||
|
|
||||||
async def _release(self, user: str) -> None:
|
async def _release(self, user: str) -> None:
|
||||||
async with self.state.lock:
|
async with self.state.lock:
|
||||||
self.state.in_flight[user] = max(
|
self.state.in_flight[user] = max(self.state.in_flight.get(user, 1) - 1, 0)
|
||||||
self.state.in_flight.get(user, 1) - 1, 0
|
|
||||||
)
|
|
||||||
self.state.last_activity[user] = time.time()
|
self.state.last_activity[user] = time.time()
|
||||||
|
|
||||||
# -- sandbox / handshake helpers --------------------------------------
|
# -- sandbox / handshake helpers --------------------------------------
|
||||||
@@ -655,9 +712,7 @@ class Gateway:
|
|||||||
# Wake-on-connect: safe in ANY state, including mid-suspension.
|
# Wake-on-connect: safe in ANY state, including mid-suspension.
|
||||||
self._clear_handshake(user)
|
self._clear_handshake(user)
|
||||||
try:
|
try:
|
||||||
await self.k8s.patch_operating_mode(
|
await self.k8s.patch_operating_mode(sandbox["metadata"]["name"], "Running")
|
||||||
sandbox["metadata"]["name"], "Running"
|
|
||||||
)
|
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
raise SandboxUnavailable("could not wake the sandbox") from exc
|
raise SandboxUnavailable("could not wake the sandbox") from exc
|
||||||
sandbox = await self.wait_ready(user, self.settings.wake_timeout)
|
sandbox = await self.wait_ready(user, self.settings.wake_timeout)
|
||||||
@@ -671,10 +726,7 @@ class Gateway:
|
|||||||
|
|
||||||
def _pod_name(self, sandbox: dict) -> str:
|
def _pod_name(self, sandbox: dict) -> str:
|
||||||
annotations = sandbox.get("metadata", {}).get("annotations") or {}
|
annotations = sandbox.get("metadata", {}).get("annotations") or {}
|
||||||
return (
|
return annotations.get(POD_NAME_ANNOTATION) or sandbox["metadata"]["name"]
|
||||||
annotations.get(POD_NAME_ANNOTATION)
|
|
||||||
or sandbox["metadata"]["name"]
|
|
||||||
)
|
|
||||||
|
|
||||||
@staticmethod
|
@staticmethod
|
||||||
def _pod_ip(sandbox: dict) -> str | None:
|
def _pod_ip(sandbox: dict) -> str | None:
|
||||||
@@ -718,16 +770,11 @@ class Gateway:
|
|||||||
self.settings.dsh_log_tail_lines,
|
self.settings.dsh_log_tail_lines,
|
||||||
)
|
)
|
||||||
except ApiException as exc:
|
except ApiException as exc:
|
||||||
raise HandshakeError(
|
raise HandshakeError(f"could not read logs for {pod_name}: {exc}") from exc
|
||||||
f"could not read logs for {pod_name}: {exc}"
|
|
||||||
) from exc
|
|
||||||
launch_token = parse_dsh_launch_token(logs)
|
launch_token = parse_dsh_launch_token(logs)
|
||||||
if not launch_token:
|
if not launch_token:
|
||||||
raise HandshakeError("DSH launch token not found in pod logs")
|
raise HandshakeError("DSH launch token not found in pod logs")
|
||||||
url = (
|
url = f"http://{fqdn}:{self.settings.dsh_port}/?token={quote(launch_token, safe='')}"
|
||||||
f"http://{fqdn}:{self.settings.dsh_port}/"
|
|
||||||
f"?token={quote(launch_token, safe='')}"
|
|
||||||
)
|
|
||||||
headers = {
|
headers = {
|
||||||
"Host": host,
|
"Host": host,
|
||||||
"X-Forwarded-Host": host,
|
"X-Forwarded-Host": host,
|
||||||
@@ -743,9 +790,7 @@ class Gateway:
|
|||||||
except httpx.HTTPError as exc:
|
except httpx.HTTPError as exc:
|
||||||
raise HandshakeError(f"DSH handshake failed: {exc}") from exc
|
raise HandshakeError(f"DSH handshake failed: {exc}") from exc
|
||||||
if response.status_code >= 400:
|
if response.status_code >= 400:
|
||||||
raise HandshakeError(
|
raise HandshakeError(f"DSH handshake returned {response.status_code}")
|
||||||
f"DSH handshake returned {response.status_code}"
|
|
||||||
)
|
|
||||||
cookie = extract_dsh_cookies(response.headers.get_list("set-cookie"))
|
cookie = extract_dsh_cookies(response.headers.get_list("set-cookie"))
|
||||||
if not cookie:
|
if not cookie:
|
||||||
raise HandshakeError("DSH handshake set no dsh-auth cookie")
|
raise HandshakeError("DSH handshake set no dsh-auth cookie")
|
||||||
@@ -767,7 +812,63 @@ class Gateway:
|
|||||||
async def healthz(self, request: Request) -> JSONResponse:
|
async def healthz(self, request: Request) -> JSONResponse:
|
||||||
return JSONResponse({"status": "ok"})
|
return JSONResponse({"status": "ok"})
|
||||||
|
|
||||||
async def landing(self, request: Request) -> HTMLResponse:
|
@staticmethod
|
||||||
|
def _sso_page(user: str, state: str) -> str:
|
||||||
|
"""Minimal SSO landing: connect to the sandbox, or offer to recreate it.
|
||||||
|
|
||||||
|
No signup/token forms — the authenticating proxy has already identified
|
||||||
|
the user. A "Provisioning" sandbox refreshes itself until a pod exists;
|
||||||
|
any other state auto-connects to ``/u/<user>/`` and offers the single
|
||||||
|
"Recreate sandbox" action.
|
||||||
|
"""
|
||||||
|
safe_user = html.escape(user)
|
||||||
|
path = quote(user)
|
||||||
|
if state == "Provisioning":
|
||||||
|
status = "Provisioning"
|
||||||
|
body = (
|
||||||
|
'<meta http-equiv="refresh" content="3">'
|
||||||
|
"<p>Provisioning your sandbox… this can take a minute.</p>"
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
status = html.escape(state)
|
||||||
|
confirm = (
|
||||||
|
"onsubmit=\"return confirm('Delete this sandbox and start fresh? "
|
||||||
|
"Your workspace data will be wiped.')\""
|
||||||
|
)
|
||||||
|
body = (
|
||||||
|
f'<meta http-equiv="refresh" content="1;url=/u/{path}/">'
|
||||||
|
"<p>Connecting to your sandbox…</p>"
|
||||||
|
f'<form method="post" action="/recreate" {confirm}>'
|
||||||
|
'<button type="submit">Recreate sandbox</button></form>'
|
||||||
|
)
|
||||||
|
return (
|
||||||
|
'<!doctype html><html lang="en"><head><meta charset="utf-8">'
|
||||||
|
'<meta name="viewport" content="width=device-width, initial-scale=1">'
|
||||||
|
"<title>DSH sandbox</title><style>"
|
||||||
|
"body{font-family:system-ui,sans-serif;max-width:42rem;margin:3rem auto;"
|
||||||
|
"padding:0 1rem;line-height:1.5}"
|
||||||
|
"form{margin:1rem 0}button{padding:.4rem 1rem}"
|
||||||
|
"</style></head><body>"
|
||||||
|
"<h1>DeepSeek Harness sandbox</h1>"
|
||||||
|
f"<p>Signed in as <strong>{safe_user}</strong>.</p>"
|
||||||
|
f"<p>Sandbox status: <strong>{status}</strong>.</p>"
|
||||||
|
f"{body}"
|
||||||
|
"</body></html>"
|
||||||
|
)
|
||||||
|
|
||||||
|
async def landing(self, request: Request) -> Response:
|
||||||
|
remote = self.remote_user(request.headers)
|
||||||
|
if remote:
|
||||||
|
# SSO: auto-provision + connect. A temporary response captures the
|
||||||
|
# session cookie minted by _sso_session; the real page is rendered
|
||||||
|
# once the sandbox state is known.
|
||||||
|
response = HTMLResponse()
|
||||||
|
claim = await self._sso_session(remote, request.headers, request.cookies, response)
|
||||||
|
sandbox = await self._sandbox_of(claim) if claim else None
|
||||||
|
page = HTMLResponse(self._sso_page(remote, derive_state(sandbox)))
|
||||||
|
if "set-cookie" in response.headers:
|
||||||
|
page.headers["set-cookie"] = response.headers["set-cookie"]
|
||||||
|
return page
|
||||||
token, user = self.credentials(request.headers, request.cookies)
|
token, user = self.credentials(request.headers, request.cookies)
|
||||||
session = ""
|
session = ""
|
||||||
if user and token:
|
if user and token:
|
||||||
@@ -826,14 +927,10 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
|
|||||||
@staticmethod
|
@staticmethod
|
||||||
def _wants_form(request: Request) -> bool:
|
def _wants_form(request: Request) -> bool:
|
||||||
ctype = request.headers.get("content-type", "").lower()
|
ctype = request.headers.get("content-type", "").lower()
|
||||||
return ctype.startswith(
|
return ctype.startswith(("application/x-www-form-urlencoded", "multipart/form-data"))
|
||||||
("application/x-www-form-urlencoded", "multipart/form-data")
|
|
||||||
)
|
|
||||||
|
|
||||||
async def _read_payload(self, request: Request) -> dict:
|
async def _read_payload(self, request: Request) -> dict:
|
||||||
if request.headers.get("content-type", "").lower().startswith(
|
if request.headers.get("content-type", "").lower().startswith("application/json"):
|
||||||
"application/json"
|
|
||||||
):
|
|
||||||
data = await self._read_body(request)
|
data = await self._read_body(request)
|
||||||
try:
|
try:
|
||||||
payload = json.loads(data)
|
payload = json.loads(data)
|
||||||
@@ -870,9 +967,7 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
|
|||||||
return self._error(413, "request body too large")
|
return self._error(413, "request body too large")
|
||||||
user = payload.get("user", "")
|
user = payload.get("user", "")
|
||||||
if not valid_user(user):
|
if not valid_user(user):
|
||||||
return self._error(
|
return self._error(400, "body must be {'user': '<dns-1123 label>'}")
|
||||||
400, "body must be {'user': '<dns-1123 label>'}"
|
|
||||||
)
|
|
||||||
if await self.k8s.get_claim(user) is not None:
|
if await self.k8s.get_claim(user) is not None:
|
||||||
return self._error(409, "user exists")
|
return self._error(409, "user exists")
|
||||||
token = secrets.token_urlsafe(32)
|
token = secrets.token_urlsafe(32)
|
||||||
@@ -932,7 +1027,7 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
|
|||||||
if claim is None:
|
if claim is None:
|
||||||
return self._error(404, "not found")
|
return self._error(404, "not found")
|
||||||
token, cookie_user = self.credentials(request.headers, request.cookies)
|
token, cookie_user = self.credentials(request.headers, request.cookies)
|
||||||
if not self._authorized(claim, token, cookie_user, user):
|
if not self._authorized(claim, token, cookie_user, user, self.remote_user(request.headers)):
|
||||||
return self._error(401, "unauthorized")
|
return self._error(401, "unauthorized")
|
||||||
sandbox = await self._sandbox_of(claim)
|
sandbox = await self._sandbox_of(claim)
|
||||||
return JSONResponse(
|
return JSONResponse(
|
||||||
@@ -951,7 +1046,7 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
|
|||||||
if claim is None:
|
if claim is None:
|
||||||
return self._error(404, "not found")
|
return self._error(404, "not found")
|
||||||
token, cookie_user = self.credentials(request.headers, request.cookies)
|
token, cookie_user = self.credentials(request.headers, request.cookies)
|
||||||
if not self._authorized(claim, token, cookie_user, user):
|
if not self._authorized(claim, token, cookie_user, user, self.remote_user(request.headers)):
|
||||||
return self._error(401, "unauthorized")
|
return self._error(401, "unauthorized")
|
||||||
if self._mutation_from_cookie(request.headers, "DELETE", cookie_user):
|
if self._mutation_from_cookie(request.headers, "DELETE", cookie_user):
|
||||||
return self._error(403, "cross-origin request rejected")
|
return self._error(403, "cross-origin request rejected")
|
||||||
@@ -960,9 +1055,37 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
|
|||||||
async with self.state.lock:
|
async with self.state.lock:
|
||||||
self.state.last_activity.pop(user, None)
|
self.state.last_activity.pop(user, None)
|
||||||
self.state.in_flight.pop(user, None)
|
self.state.in_flight.pop(user, None)
|
||||||
return JSONResponse(
|
return JSONResponse({"user": user, "note": "claim deleted; sandbox and PVC cascade"})
|
||||||
{"user": user, "note": "claim deleted; sandbox and PVC cascade"}
|
|
||||||
)
|
async def recreate(self, request: Request) -> Response:
|
||||||
|
"""Delete and re-provision the SSO user's sandbox for a clean start.
|
||||||
|
|
||||||
|
SSO-only: the identity comes from the trusted proxy header. Deleting the
|
||||||
|
claim cascades to the sandbox pod and its PVC, wiping user state; we poll
|
||||||
|
until the claim is fully gone (finalizers) before recreating to avoid a
|
||||||
|
name-collision 409, then reconnect.
|
||||||
|
"""
|
||||||
|
remote = self.remote_user(request.headers)
|
||||||
|
if not remote:
|
||||||
|
return self._error(401, "unauthorized")
|
||||||
|
_token, cookie_user = self.credentials(request.headers, request.cookies)
|
||||||
|
if cookie_user is not None and cookie_user != remote:
|
||||||
|
return self._error(401, "unauthorized")
|
||||||
|
if self._mutation_from_cookie(request.headers, "POST", cookie_user):
|
||||||
|
return self._error(403, "cross-origin request rejected")
|
||||||
|
await self.k8s.delete_claim(remote)
|
||||||
|
self._clear_handshake(remote)
|
||||||
|
async with self.state.lock:
|
||||||
|
self.state.last_activity.pop(remote, None)
|
||||||
|
self.state.in_flight.pop(remote, None)
|
||||||
|
# Wait out any finalizer so the claim name is reusable.
|
||||||
|
for _ in range(30):
|
||||||
|
if await self.k8s.get_claim(remote) is None:
|
||||||
|
break
|
||||||
|
await asyncio.sleep(1)
|
||||||
|
response = RedirectResponse(f"/u/{remote}/", status_code=303)
|
||||||
|
await self._sso_session(remote, request.headers, request.cookies, response)
|
||||||
|
return response
|
||||||
|
|
||||||
# -- HTTP proxy --------------------------------------------------------
|
# -- HTTP proxy --------------------------------------------------------
|
||||||
|
|
||||||
@@ -970,41 +1093,38 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
|
|||||||
user = request.path_params["user"]
|
user = request.path_params["user"]
|
||||||
if not valid_user(user):
|
if not valid_user(user):
|
||||||
return self._error(404, "unknown user")
|
return self._error(404, "unknown user")
|
||||||
|
remote = self.remote_user(request.headers)
|
||||||
claim = await self.k8s.get_claim(user)
|
claim = await self.k8s.get_claim(user)
|
||||||
if claim is None:
|
if claim is None:
|
||||||
|
if remote == user:
|
||||||
|
# SSO first hit straight at a deep link: let landing bootstrap.
|
||||||
|
return RedirectResponse("/", status_code=303)
|
||||||
return self._error(404, "unknown user")
|
return self._error(404, "unknown user")
|
||||||
token, cookie_user = self.credentials(request.headers, request.cookies)
|
token, cookie_user = self.credentials(request.headers, request.cookies)
|
||||||
if not self._authorized(claim, token, cookie_user, user):
|
if remote == user and not (cookie_user == user and token_matches(claim, token)):
|
||||||
|
# SSO user without a valid session cookie: bootstrap, then continue.
|
||||||
|
return RedirectResponse("/", status_code=303)
|
||||||
|
if not self._authorized(claim, token, cookie_user, user, remote):
|
||||||
return self._error(401, "unauthorized")
|
return self._error(401, "unauthorized")
|
||||||
if self._mutation_from_cookie(request.headers, request.method, cookie_user):
|
if self._mutation_from_cookie(request.headers, request.method, cookie_user):
|
||||||
return self._error(403, "cross-origin request rejected")
|
return self._error(403, "cross-origin request rejected")
|
||||||
|
|
||||||
if not await self._acquire(user):
|
if not await self._acquire(user):
|
||||||
return self._error(
|
return self._error(503, "gateway busy, retry shortly", {"Retry-After": "5"})
|
||||||
503, "gateway busy, retry shortly", {"Retry-After": "5"}
|
|
||||||
)
|
|
||||||
streaming = False
|
streaming = False
|
||||||
try:
|
try:
|
||||||
sandbox, fqdn = await self._resolve_sandbox(user, claim)
|
sandbox, fqdn = await self._resolve_sandbox(user, claim)
|
||||||
host = request.headers.get("host", "")
|
host = request.headers.get("host", "")
|
||||||
cookie, _, was_cached = await self.dsh_cookie(
|
cookie, _, was_cached = await self.dsh_cookie(user, sandbox, fqdn, host)
|
||||||
user, sandbox, fqdn, host
|
|
||||||
)
|
|
||||||
body = await self._read_body(request)
|
body = await self._read_body(request)
|
||||||
|
|
||||||
upstream = await self._send_upstream(
|
upstream = await self._send_upstream(request, user, fqdn, cookie, body)
|
||||||
request, user, fqdn, cookie, body
|
|
||||||
)
|
|
||||||
if upstream.status_code == 401 and was_cached:
|
if upstream.status_code == 401 and was_cached:
|
||||||
# The cached DSH cookie expired (pod restart, logout, ...):
|
# The cached DSH cookie expired (pod restart, logout, ...):
|
||||||
# handshake once more and replay the (already buffered) request.
|
# handshake once more and replay the (already buffered) request.
|
||||||
await upstream.aclose()
|
await upstream.aclose()
|
||||||
cookie, _, _ = await self.dsh_cookie(
|
cookie, _, _ = await self.dsh_cookie(user, sandbox, fqdn, host, force=True)
|
||||||
user, sandbox, fqdn, host, force=True
|
upstream = await self._send_upstream(request, user, fqdn, cookie, body)
|
||||||
)
|
|
||||||
upstream = await self._send_upstream(
|
|
||||||
request, user, fqdn, cookie, body
|
|
||||||
)
|
|
||||||
|
|
||||||
response = StreamingResponse(
|
response = StreamingResponse(
|
||||||
self._relay(user, upstream),
|
self._relay(user, upstream),
|
||||||
@@ -1044,8 +1164,7 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
|
|||||||
headers = {
|
headers = {
|
||||||
key: value
|
key: value
|
||||||
for key, value in request.headers.items()
|
for key, value in request.headers.items()
|
||||||
if key.lower() not in OWNED_REQUEST_HEADERS
|
if key.lower() not in OWNED_REQUEST_HEADERS and key.lower() != "content-length"
|
||||||
and key.lower() != "content-length"
|
|
||||||
}
|
}
|
||||||
host = request.headers.get("host", "")
|
host = request.headers.get("host", "")
|
||||||
headers["host"] = host
|
headers["host"] = host
|
||||||
@@ -1057,9 +1176,7 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
|
|||||||
else:
|
else:
|
||||||
headers["x-forwarded-for"] = ""
|
headers["x-forwarded-for"] = ""
|
||||||
port = host.rsplit(":", 1)[1] if ":" in host else ""
|
port = host.rsplit(":", 1)[1] if ":" in host else ""
|
||||||
headers["x-forwarded-port"] = port or (
|
headers["x-forwarded-port"] = port or ("443" if request.url.scheme == "https" else "80")
|
||||||
"443" if request.url.scheme == "https" else "80"
|
|
||||||
)
|
|
||||||
|
|
||||||
last_error: Exception | None = None
|
last_error: Exception | None = None
|
||||||
for attempt in range(self.settings.upstream_attempts):
|
for attempt in range(self.settings.upstream_attempts):
|
||||||
@@ -1143,7 +1260,9 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
|
|||||||
if claim is None:
|
if claim is None:
|
||||||
return await self._ws_reject(websocket, 4401)
|
return await self._ws_reject(websocket, 4401)
|
||||||
token, cookie_user = self.credentials(websocket.headers, websocket.cookies)
|
token, cookie_user = self.credentials(websocket.headers, websocket.cookies)
|
||||||
if not self._authorized(claim, token, cookie_user, user):
|
if not self._authorized(
|
||||||
|
claim, token, cookie_user, user, self.remote_user(websocket.headers)
|
||||||
|
):
|
||||||
return await self._ws_reject(websocket, 4401)
|
return await self._ws_reject(websocket, 4401)
|
||||||
if cookie_user is not None and not same_origin(websocket.headers):
|
if cookie_user is not None and not same_origin(websocket.headers):
|
||||||
return await self._ws_reject(websocket, 4401)
|
return await self._ws_reject(websocket, 4401)
|
||||||
@@ -1167,9 +1286,7 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
|
|||||||
if ":" in upstream_host and not upstream_host.startswith("["):
|
if ":" in upstream_host and not upstream_host.startswith("["):
|
||||||
upstream_host = f"[{upstream_host}]" # IPv6 literal
|
upstream_host = f"[{upstream_host}]" # IPv6 literal
|
||||||
authority = f"{upstream_host}:{self.settings.dsh_port}"
|
authority = f"{upstream_host}:{self.settings.dsh_port}"
|
||||||
cookie, _, _ = await self.dsh_cookie(
|
cookie, _, _ = await self.dsh_cookie(user, sandbox, fqdn, authority)
|
||||||
user, sandbox, fqdn, authority
|
|
||||||
)
|
|
||||||
url = f"ws://{authority}/{path}"
|
url = f"ws://{authority}/{path}"
|
||||||
query = filtered_query(websocket.url.query)
|
query = filtered_query(websocket.url.query)
|
||||||
if query:
|
if query:
|
||||||
@@ -1188,9 +1305,7 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
|
|||||||
headers["x-forwarded-for"] = websocket.client.host
|
headers["x-forwarded-for"] = websocket.client.host
|
||||||
subprotocols = [
|
subprotocols = [
|
||||||
part.strip()
|
part.strip()
|
||||||
for part in websocket.headers.get(
|
for part in websocket.headers.get("sec-websocket-protocol", "").split(",")
|
||||||
"sec-websocket-protocol", ""
|
|
||||||
).split(",")
|
|
||||||
if part.strip()
|
if part.strip()
|
||||||
]
|
]
|
||||||
upstream_cm = self.ws_connect(
|
upstream_cm = self.ws_connect(
|
||||||
@@ -1206,16 +1321,12 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
|
|||||||
return await self._ws_reject(websocket, 1011)
|
return await self._ws_reject(websocket, 1011)
|
||||||
|
|
||||||
try:
|
try:
|
||||||
await websocket.accept(
|
await websocket.accept(subprotocol=getattr(upstream, "subprotocol", None))
|
||||||
subprotocol=getattr(upstream, "subprotocol", None)
|
|
||||||
)
|
|
||||||
except RuntimeError:
|
except RuntimeError:
|
||||||
pass
|
pass
|
||||||
await self._pump(websocket, upstream)
|
await self._pump(websocket, upstream)
|
||||||
with contextlib.suppress(Exception):
|
with contextlib.suppress(Exception):
|
||||||
await websocket.close(
|
await websocket.close(code=self._upstream_close_code(upstream))
|
||||||
code=self._upstream_close_code(upstream)
|
|
||||||
)
|
|
||||||
finally:
|
finally:
|
||||||
if upstream_cm is not None:
|
if upstream_cm is not None:
|
||||||
with contextlib.suppress(Exception):
|
with contextlib.suppress(Exception):
|
||||||
@@ -1255,9 +1366,7 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
|
|||||||
asyncio.create_task(client_to_upstream()),
|
asyncio.create_task(client_to_upstream()),
|
||||||
asyncio.create_task(upstream_to_client()),
|
asyncio.create_task(upstream_to_client()),
|
||||||
]
|
]
|
||||||
done, pending = await asyncio.wait(
|
done, pending = await asyncio.wait(tasks, return_when=asyncio.FIRST_COMPLETED)
|
||||||
tasks, return_when=asyncio.FIRST_COMPLETED
|
|
||||||
)
|
|
||||||
for task in pending:
|
for task in pending:
|
||||||
task.cancel()
|
task.cancel()
|
||||||
await asyncio.gather(*done, *pending, return_exceptions=True)
|
await asyncio.gather(*done, *pending, return_exceptions=True)
|
||||||
@@ -1284,17 +1393,13 @@ against the JSON API. The browser session is an HttpOnly cookie.</p>
|
|||||||
continue
|
continue
|
||||||
if self.state.in_flight.get(user, 0) > 0:
|
if self.state.in_flight.get(user, 0) > 0:
|
||||||
continue # requests in flight => the user is active
|
continue # requests in flight => the user is active
|
||||||
idle = time.time() - self.state.last_activity.setdefault(
|
idle = time.time() - self.state.last_activity.setdefault(user, time.time())
|
||||||
user, time.time()
|
|
||||||
)
|
|
||||||
if idle < self.settings.idle_timeout:
|
if idle < self.settings.idle_timeout:
|
||||||
continue
|
continue
|
||||||
self.state.maintenance[user] = True
|
self.state.maintenance[user] = True
|
||||||
try:
|
try:
|
||||||
print(f"suspending {user} (idle {int(idle)}s)", flush=True)
|
print(f"suspending {user} (idle {int(idle)}s)", flush=True)
|
||||||
await self.k8s.patch_operating_mode(
|
await self.k8s.patch_operating_mode(sandbox["metadata"]["name"], "Suspended")
|
||||||
sandbox["metadata"]["name"], "Suspended"
|
|
||||||
)
|
|
||||||
self._clear_handshake(user)
|
self._clear_handshake(user)
|
||||||
finally:
|
finally:
|
||||||
async with self.state.lock:
|
async with self.state.lock:
|
||||||
@@ -1345,18 +1450,15 @@ def create_app(
|
|||||||
k8s = k8s if k8s is not None else Kubernetes(settings.namespace)
|
k8s = k8s if k8s is not None else Kubernetes(settings.namespace)
|
||||||
owns_http_client = http_client is None
|
owns_http_client = http_client is None
|
||||||
if http_client is None:
|
if http_client is None:
|
||||||
http_client = httpx.AsyncClient(
|
http_client = httpx.AsyncClient(follow_redirects=False, timeout=settings.http_timeout)
|
||||||
follow_redirects=False, timeout=settings.http_timeout
|
gateway = Gateway(settings, k8s, http_client, ws_connect or websockets.connect)
|
||||||
)
|
|
||||||
gateway = Gateway(
|
|
||||||
settings, k8s, http_client, ws_connect or websockets.connect
|
|
||||||
)
|
|
||||||
routes = [
|
routes = [
|
||||||
Route("/healthz", gateway.healthz),
|
Route("/healthz", gateway.healthz),
|
||||||
Route("/", gateway.landing),
|
Route("/", gateway.landing),
|
||||||
Route("/users", gateway.signup, methods=["POST"]),
|
Route("/users", gateway.signup, methods=["POST"]),
|
||||||
Route("/login", gateway.login, methods=["POST"]),
|
Route("/login", gateway.login, methods=["POST"]),
|
||||||
Route("/logout", gateway.logout, methods=["POST"]),
|
Route("/logout", gateway.logout, methods=["POST"]),
|
||||||
|
Route("/recreate", gateway.recreate, methods=["POST"]),
|
||||||
Route("/users/{user}", gateway.get_user),
|
Route("/users/{user}", gateway.get_user),
|
||||||
Route("/users/{user}", gateway.delete_user, methods=["DELETE"]),
|
Route("/users/{user}", gateway.delete_user, methods=["DELETE"]),
|
||||||
Route("/u/{user}", gateway.redirect_to_workspace),
|
Route("/u/{user}", gateway.redirect_to_workspace),
|
||||||
|
|||||||
@@ -1,4 +1,5 @@
|
|||||||
-r requirements.txt
|
-r requirements.txt
|
||||||
pytest==9.1.1
|
pytest==9.1.1
|
||||||
pytest-asyncio==1.4.0
|
pytest-asyncio==1.4.0
|
||||||
|
pytest-cov==7.1.0
|
||||||
ruff==0.17.1
|
ruff==0.17.1
|
||||||
|
|||||||
@@ -45,9 +45,7 @@ FQDN = "dsh-alice-sbx.test-ns.svc.cluster.local"
|
|||||||
|
|
||||||
def dsh_response(status=200, content=b"", headers=None):
|
def dsh_response(status=200, content=b"", headers=None):
|
||||||
"""A MockTransport response that is a real stream (``aiter_raw`` works)."""
|
"""A MockTransport response that is a real stream (``aiter_raw`` works)."""
|
||||||
return httpx.Response(
|
return httpx.Response(status, headers=headers, stream=httpx.ByteStream(content))
|
||||||
status, headers=headers, stream=httpx.ByteStream(content)
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
# --- fakes -----------------------------------------------------------------
|
# --- fakes -----------------------------------------------------------------
|
||||||
@@ -62,6 +60,7 @@ class FakeK8s:
|
|||||||
self.created = []
|
self.created = []
|
||||||
self.deleted = []
|
self.deleted = []
|
||||||
self.patches = []
|
self.patches = []
|
||||||
|
self.token_patches = []
|
||||||
self.conflict_on_create = False
|
self.conflict_on_create = False
|
||||||
self.get_claim_error = None
|
self.get_claim_error = None
|
||||||
self.patch_error = None
|
self.patch_error = None
|
||||||
@@ -92,6 +91,15 @@ class FakeK8s:
|
|||||||
raise self.patch_error
|
raise self.patch_error
|
||||||
self.patches.append((name, mode))
|
self.patches.append((name, mode))
|
||||||
|
|
||||||
|
async def patch_token(self, user, token_hash):
|
||||||
|
if self.patch_error is not None:
|
||||||
|
raise self.patch_error
|
||||||
|
self.token_patches.append((user, token_hash))
|
||||||
|
claim = self.claims.get(user)
|
||||||
|
if claim is not None:
|
||||||
|
annotations = claim.setdefault("metadata", {}).setdefault("annotations", {})
|
||||||
|
annotations["aaas.example.com/token-sha256"] = token_hash
|
||||||
|
|
||||||
async def get_pod(self, name):
|
async def get_pod(self, name):
|
||||||
return self.pods.get(name)
|
return self.pods.get(name)
|
||||||
|
|
||||||
@@ -106,9 +114,7 @@ class FakeUpstream:
|
|||||||
self.requests = []
|
self.requests = []
|
||||||
self.handshake_requests = []
|
self.handshake_requests = []
|
||||||
self.handshake_status = 303
|
self.handshake_status = 303
|
||||||
self.handshake_cookies = [
|
self.handshake_cookies = ["dsh-auth-a=one; Path=/; HttpOnly; SameSite=Strict"]
|
||||||
"dsh-auth-a=one; Path=/; HttpOnly; SameSite=Strict"
|
|
||||||
]
|
|
||||||
self.proxy_handler = None
|
self.proxy_handler = None
|
||||||
self.proxy_requests = []
|
self.proxy_requests = []
|
||||||
|
|
||||||
@@ -128,9 +134,7 @@ class FakeUpstream:
|
|||||||
self.proxy_requests.append(request)
|
self.proxy_requests.append(request)
|
||||||
if self.proxy_handler is not None:
|
if self.proxy_handler is not None:
|
||||||
return self.proxy_handler(request)
|
return self.proxy_handler(request)
|
||||||
return dsh_response(
|
return dsh_response(200, b"hello from DSH", {"content-type": "text/plain"})
|
||||||
200, b"hello from DSH", {"content-type": "text/plain"}
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
class FakeWS:
|
class FakeWS:
|
||||||
@@ -195,9 +199,7 @@ def add_user(k8s, user="alice", token=TOKEN, ready=True, pod_name=None, suffix="
|
|||||||
"metadata": {
|
"metadata": {
|
||||||
"name": claim_name(user),
|
"name": claim_name(user),
|
||||||
"annotations": {
|
"annotations": {
|
||||||
"aaas.example.com/token-sha256": hashlib.sha256(
|
"aaas.example.com/token-sha256": hashlib.sha256(token.encode()).hexdigest()
|
||||||
token.encode()
|
|
||||||
).hexdigest()
|
|
||||||
},
|
},
|
||||||
},
|
},
|
||||||
"status": {"sandbox": {"name": sandbox_name}},
|
"status": {"sandbox": {"name": sandbox_name}},
|
||||||
@@ -212,10 +214,7 @@ def add_user(k8s, user="alice", token=TOKEN, ready=True, pod_name=None, suffix="
|
|||||||
},
|
},
|
||||||
}
|
}
|
||||||
k8s.pods[pod_name] = {"metadata": {"name": pod_name, "uid": POD_UID}}
|
k8s.pods[pod_name] = {"metadata": {"name": pod_name, "uid": POD_UID}}
|
||||||
k8s.logs[pod_name] = (
|
k8s.logs[pod_name] = f"booting\ndsh web: http://10.0.0.7:3080/?token={TOKEN}-launch\n"
|
||||||
"booting\n"
|
|
||||||
f"dsh web: http://10.0.0.7:3080/?token={TOKEN}-launch\n"
|
|
||||||
)
|
|
||||||
return sandbox_name, pod_name
|
return sandbox_name, pod_name
|
||||||
|
|
||||||
|
|
||||||
@@ -231,9 +230,7 @@ def env():
|
|||||||
)
|
)
|
||||||
k8s = FakeK8s()
|
k8s = FakeK8s()
|
||||||
upstream = FakeUpstream()
|
upstream = FakeUpstream()
|
||||||
http_client = httpx.AsyncClient(
|
http_client = httpx.AsyncClient(transport=httpx.MockTransport(upstream), timeout=5)
|
||||||
transport=httpx.MockTransport(upstream), timeout=5
|
|
||||||
)
|
|
||||||
ws_connect = FakeWSConnect()
|
ws_connect = FakeWSConnect()
|
||||||
app = create_app(
|
app = create_app(
|
||||||
settings=settings,
|
settings=settings,
|
||||||
@@ -263,9 +260,7 @@ def asgi_client(env, cookies=None):
|
|||||||
|
|
||||||
|
|
||||||
def stale_handshake(env):
|
def stale_handshake(env):
|
||||||
env.gateway.state.handshake[("alice", POD_UID, "gateway.test")] = (
|
env.gateway.state.handshake[("alice", POD_UID, "gateway.test")] = "dsh-auth-a=stale"
|
||||||
"dsh-auth-a=stale"
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
# --- pure helpers ----------------------------------------------------------
|
# --- pure helpers ----------------------------------------------------------
|
||||||
@@ -407,9 +402,7 @@ def test_token_matches_is_timing_safe_hash_check():
|
|||||||
claim = {
|
claim = {
|
||||||
"metadata": {
|
"metadata": {
|
||||||
"annotations": {
|
"annotations": {
|
||||||
"aaas.example.com/token-sha256": hashlib.sha256(
|
"aaas.example.com/token-sha256": hashlib.sha256(TOKEN.encode()).hexdigest()
|
||||||
TOKEN.encode()
|
|
||||||
).hexdigest()
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -431,6 +424,7 @@ def test_settings_from_env(monkeypatch):
|
|||||||
monkeypatch.setenv("COOKIE_MAX_AGE", "100")
|
monkeypatch.setenv("COOKIE_MAX_AGE", "100")
|
||||||
monkeypatch.setenv("MAX_BODY_MB", "2")
|
monkeypatch.setenv("MAX_BODY_MB", "2")
|
||||||
monkeypatch.setenv("HTTP_TIMEOUT", "12.5")
|
monkeypatch.setenv("HTTP_TIMEOUT", "12.5")
|
||||||
|
monkeypatch.setenv("AUTH_PROXY_HEADER", "X-Forwarded-User")
|
||||||
settings = Settings.from_env()
|
settings = Settings.from_env()
|
||||||
assert settings.namespace == "ns-x"
|
assert settings.namespace == "ns-x"
|
||||||
assert settings.pool == "pool-x"
|
assert settings.pool == "pool-x"
|
||||||
@@ -443,6 +437,12 @@ def test_settings_from_env(monkeypatch):
|
|||||||
assert settings.cookie_max_age == 100
|
assert settings.cookie_max_age == 100
|
||||||
assert settings.max_body_bytes == 2 * 1024 * 1024
|
assert settings.max_body_bytes == 2 * 1024 * 1024
|
||||||
assert settings.http_timeout == 12.5
|
assert settings.http_timeout == 12.5
|
||||||
|
assert settings.auth_proxy_header == "X-Forwarded-User"
|
||||||
|
|
||||||
|
|
||||||
|
def test_settings_auth_proxy_header_defaults_to_none(monkeypatch):
|
||||||
|
monkeypatch.delenv("AUTH_PROXY_HEADER", raising=False)
|
||||||
|
assert Settings.from_env().auth_proxy_header is None
|
||||||
|
|
||||||
|
|
||||||
# --- Kubernetes adapter ----------------------------------------------------
|
# --- Kubernetes adapter ----------------------------------------------------
|
||||||
@@ -467,14 +467,9 @@ async def test_kubernetes_create_claim_body(monkeypatch):
|
|||||||
body = args[4]
|
body = args[4]
|
||||||
assert body["metadata"]["name"] == "dsh-alice"
|
assert body["metadata"]["name"] == "dsh-alice"
|
||||||
assert body["metadata"]["annotations"]["aaas.example.com/token-sha256"] == "hash"
|
assert body["metadata"]["annotations"]["aaas.example.com/token-sha256"] == "hash"
|
||||||
assert (
|
assert body["metadata"]["annotations"]["aaas.example.com/managed-by"] == "dsh-gateway"
|
||||||
body["metadata"]["annotations"]["aaas.example.com/managed-by"]
|
|
||||||
== "dsh-gateway"
|
|
||||||
)
|
|
||||||
assert body["spec"]["warmPoolRef"]["name"] == "test-pool"
|
assert body["spec"]["warmPoolRef"]["name"] == "test-pool"
|
||||||
assert body["spec"]["additionalPodMetadata"]["labels"][
|
assert body["spec"]["additionalPodMetadata"]["labels"]["sandbox.users.io/dsh-user"] == "alice"
|
||||||
"sandbox.users.io/dsh-user"
|
|
||||||
] == "alice"
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
@@ -606,9 +601,7 @@ async def test_landing_page_has_signup_and_login_forms(env):
|
|||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
async def test_landing_page_shows_session_when_cookie_present(env):
|
async def test_landing_page_shows_session_when_cookie_present(env):
|
||||||
async with asgi_client(
|
async with asgi_client(env, cookies={"dsh_gw_session": f"alice:{TOKEN}"}) as client:
|
||||||
env, cookies={"dsh_gw_session": f"alice:{TOKEN}"}
|
|
||||||
) as client:
|
|
||||||
response = await client.get("/")
|
response = await client.get("/")
|
||||||
assert "Signed in as" in response.text
|
assert "Signed in as" in response.text
|
||||||
assert "/u/alice/" in response.text
|
assert "/u/alice/" in response.text
|
||||||
@@ -626,9 +619,7 @@ async def test_workspace_redirect_adds_trailing_slash(env):
|
|||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
@pytest.mark.parametrize(
|
@pytest.mark.parametrize("bad_user", ["Alice", "-alice", "alice-", "", "a_b", 123, None, "a" * 60])
|
||||||
"bad_user", ["Alice", "-alice", "alice-", "", "a_b", 123, None, "a" * 60]
|
|
||||||
)
|
|
||||||
async def test_signup_rejects_invalid_dns1123_label(env, bad_user):
|
async def test_signup_rejects_invalid_dns1123_label(env, bad_user):
|
||||||
async with asgi_client(env) as client:
|
async with asgi_client(env) as client:
|
||||||
response = await client.post("/users", json={"user": bad_user})
|
response = await client.post("/users", json={"user": bad_user})
|
||||||
@@ -762,9 +753,7 @@ async def test_login_json_and_failures(env):
|
|||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
async def test_logout_clears_cookie(env):
|
async def test_logout_clears_cookie(env):
|
||||||
async with asgi_client(
|
async with asgi_client(env, cookies={"dsh_gw_session": f"alice:{TOKEN}"}) as client:
|
||||||
env, cookies={"dsh_gw_session": f"alice:{TOKEN}"}
|
|
||||||
) as client:
|
|
||||||
response = await client.post("/logout")
|
response = await client.post("/logout")
|
||||||
assert response.status_code == 303
|
assert response.status_code == 303
|
||||||
assert response.headers["location"] == "/"
|
assert response.headers["location"] == "/"
|
||||||
@@ -779,12 +768,8 @@ async def test_logout_clears_cookie(env):
|
|||||||
async def test_get_user_requires_valid_bearer_or_cookie(env):
|
async def test_get_user_requires_valid_bearer_or_cookie(env):
|
||||||
add_user(env.k8s)
|
add_user(env.k8s)
|
||||||
async with asgi_client(env) as client:
|
async with asgi_client(env) as client:
|
||||||
forbidden = await client.get(
|
forbidden = await client.get("/users/alice", headers={"Authorization": "Bearer wrong"})
|
||||||
"/users/alice", headers={"Authorization": "Bearer wrong"}
|
bearer = await client.get("/users/alice", headers={"Authorization": f"Bearer {TOKEN}"})
|
||||||
)
|
|
||||||
bearer = await client.get(
|
|
||||||
"/users/alice", headers={"Authorization": f"Bearer {TOKEN}"}
|
|
||||||
)
|
|
||||||
cookie = await client.get(
|
cookie = await client.get(
|
||||||
"/users/alice",
|
"/users/alice",
|
||||||
headers={"Cookie": f"dsh_gw_session=alice:{TOKEN}"},
|
headers={"Cookie": f"dsh_gw_session=alice:{TOKEN}"},
|
||||||
@@ -820,9 +805,7 @@ async def test_invalid_user_in_path_is_404(env, path):
|
|||||||
async def test_delete_user_requires_auth(env):
|
async def test_delete_user_requires_auth(env):
|
||||||
add_user(env.k8s)
|
add_user(env.k8s)
|
||||||
async with asgi_client(env) as client:
|
async with asgi_client(env) as client:
|
||||||
response = await client.delete(
|
response = await client.delete("/users/alice", headers={"Authorization": "Bearer wrong"})
|
||||||
"/users/alice", headers={"Authorization": "Bearer wrong"}
|
|
||||||
)
|
|
||||||
assert response.status_code == 401
|
assert response.status_code == 401
|
||||||
assert env.k8s.deleted == []
|
assert env.k8s.deleted == []
|
||||||
|
|
||||||
@@ -834,12 +817,8 @@ async def test_delete_with_cookie_requires_same_origin(env):
|
|||||||
cookies = {"dsh_gw_session": f"alice:{TOKEN}"}
|
cookies = {"dsh_gw_session": f"alice:{TOKEN}"}
|
||||||
async with asgi_client(env, cookies=cookies) as client:
|
async with asgi_client(env, cookies=cookies) as client:
|
||||||
missing = await client.delete("/users/alice")
|
missing = await client.delete("/users/alice")
|
||||||
cross = await client.delete(
|
cross = await client.delete("/users/alice", headers={"Origin": "http://evil.test"})
|
||||||
"/users/alice", headers={"Origin": "http://evil.test"}
|
good = await client.delete("/users/alice", headers={"Origin": "http://gateway.test"})
|
||||||
)
|
|
||||||
good = await client.delete(
|
|
||||||
"/users/alice", headers={"Origin": "http://gateway.test"}
|
|
||||||
)
|
|
||||||
assert missing.status_code == 403
|
assert missing.status_code == 403
|
||||||
assert cross.status_code == 403
|
assert cross.status_code == 403
|
||||||
assert good.status_code == 200
|
assert good.status_code == 200
|
||||||
@@ -851,9 +830,7 @@ async def test_delete_with_cookie_requires_same_origin(env):
|
|||||||
async def test_delete_with_bearer_does_not_need_origin(env):
|
async def test_delete_with_bearer_does_not_need_origin(env):
|
||||||
add_user(env.k8s)
|
add_user(env.k8s)
|
||||||
async with asgi_client(env) as client:
|
async with asgi_client(env) as client:
|
||||||
response = await client.delete(
|
response = await client.delete("/users/alice", headers={"Authorization": f"Bearer {TOKEN}"})
|
||||||
"/users/alice", headers={"Authorization": f"Bearer {TOKEN}"}
|
|
||||||
)
|
|
||||||
assert response.status_code == 200
|
assert response.status_code == 200
|
||||||
|
|
||||||
|
|
||||||
@@ -862,9 +839,7 @@ async def test_delete_missing_claim_is_404(env):
|
|||||||
add_user(env.k8s)
|
add_user(env.k8s)
|
||||||
env.k8s.claims.clear()
|
env.k8s.claims.clear()
|
||||||
async with asgi_client(env) as client:
|
async with asgi_client(env) as client:
|
||||||
response = await client.delete(
|
response = await client.delete("/users/alice", headers={"Authorization": f"Bearer {TOKEN}"})
|
||||||
"/users/alice", headers={"Authorization": f"Bearer {TOKEN}"}
|
|
||||||
)
|
|
||||||
assert response.status_code == 404
|
assert response.status_code == 404
|
||||||
|
|
||||||
|
|
||||||
@@ -939,9 +914,9 @@ async def test_missing_cookie_in_handshake_is_503(env):
|
|||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
async def test_pod_annotation_overrides_sandbox_name(env):
|
async def test_pod_annotation_overrides_sandbox_name(env):
|
||||||
sandbox_name, pod_name = add_user(env.k8s)
|
sandbox_name, pod_name = add_user(env.k8s)
|
||||||
env.k8s.sandboxes[sandbox_name]["metadata"]["annotations"][
|
env.k8s.sandboxes[sandbox_name]["metadata"]["annotations"]["agents.x-k8s.io/pod-name"] = (
|
||||||
"agents.x-k8s.io/pod-name"
|
"custom-pod"
|
||||||
] = "custom-pod"
|
)
|
||||||
env.k8s.pods["custom-pod"] = {"metadata": {"uid": "other-uid"}}
|
env.k8s.pods["custom-pod"] = {"metadata": {"uid": "other-uid"}}
|
||||||
env.k8s.logs["custom-pod"] = env.k8s.logs[pod_name]
|
env.k8s.logs["custom-pod"] = env.k8s.logs[pod_name]
|
||||||
async with asgi_client(env, cookies={"dsh_gw_session": f"alice:{TOKEN}"}) as client:
|
async with asgi_client(env, cookies={"dsh_gw_session": f"alice:{TOKEN}"}) as client:
|
||||||
@@ -1063,9 +1038,7 @@ async def test_proxy_upstream_connection_error_is_502(env):
|
|||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
async def test_proxy_body_limit_returns_413(env):
|
async def test_proxy_body_limit_returns_413(env):
|
||||||
add_user(env.k8s)
|
add_user(env.k8s)
|
||||||
env.gateway.settings = Settings(
|
env.gateway.settings = Settings(**{**env.settings.__dict__, "max_body_bytes": 4})
|
||||||
**{**env.settings.__dict__, "max_body_bytes": 4}
|
|
||||||
)
|
|
||||||
async with asgi_client(env) as client:
|
async with asgi_client(env) as client:
|
||||||
response = await client.post(
|
response = await client.post(
|
||||||
"/u/alice/api/echo",
|
"/u/alice/api/echo",
|
||||||
@@ -1175,9 +1148,7 @@ async def test_proxied_requests_use_the_stream_timeout(env, monkeypatch):
|
|||||||
"server": ("gateway.test", 80),
|
"server": ("gateway.test", 80),
|
||||||
"path_params": {"user": "alice", "path": "events"},
|
"path_params": {"user": "alice", "path": "events"},
|
||||||
}
|
}
|
||||||
response = await env.gateway._send_upstream(
|
response = await env.gateway._send_upstream(Request(scope), "alice", FQDN, "dsh-auth-a=1", b"")
|
||||||
Request(scope), "alice", FQDN, "dsh-auth-a=1", b""
|
|
||||||
)
|
|
||||||
await response.aclose()
|
await response.aclose()
|
||||||
|
|
||||||
assert captured["timeout"].read is None
|
assert captured["timeout"].read is None
|
||||||
@@ -1340,9 +1311,7 @@ async def test_sweeper_skips_claim_it_does_not_own(env):
|
|||||||
assert "alice" not in env.gateway.state.last_activity
|
assert "alice" not in env.gateway.state.last_activity
|
||||||
|
|
||||||
# ...and does not suspend it even when it looks long-idle.
|
# ...and does not suspend it even when it looks long-idle.
|
||||||
env.gateway.state.last_activity["alice"] = (
|
env.gateway.state.last_activity["alice"] = time.time() - env.settings.idle_timeout - 10
|
||||||
time.time() - env.settings.idle_timeout - 10
|
|
||||||
)
|
|
||||||
await env.gateway.sweep_once()
|
await env.gateway.sweep_once()
|
||||||
assert env.k8s.patches == []
|
assert env.k8s.patches == []
|
||||||
|
|
||||||
@@ -1351,9 +1320,7 @@ async def test_sweeper_skips_claim_it_does_not_own(env):
|
|||||||
async def test_sweeper_suspends_idle_ready_sandbox(env):
|
async def test_sweeper_suspends_idle_ready_sandbox(env):
|
||||||
env.k8s.claims["alice"] = ready_claim()
|
env.k8s.claims["alice"] = ready_claim()
|
||||||
env.k8s.sandboxes["dsh-alice-sbx"] = ready_sandbox()
|
env.k8s.sandboxes["dsh-alice-sbx"] = ready_sandbox()
|
||||||
env.gateway.state.last_activity["alice"] = (
|
env.gateway.state.last_activity["alice"] = time.time() - env.settings.idle_timeout - 10
|
||||||
time.time() - env.settings.idle_timeout - 10
|
|
||||||
)
|
|
||||||
env.gateway.state.handshake[("alice", POD_UID, "gateway.test")] = "dsh-auth-a=x"
|
env.gateway.state.handshake[("alice", POD_UID, "gateway.test")] = "dsh-auth-a=x"
|
||||||
|
|
||||||
await env.gateway.sweep_once()
|
await env.gateway.sweep_once()
|
||||||
@@ -1413,3 +1380,146 @@ async def test_sweeper_survives_transient_api_error(env):
|
|||||||
|
|
||||||
# The per-user maintenance flag must be released even on failure.
|
# The per-user maintenance flag must be released even on failure.
|
||||||
assert env.gateway.state.maintenance["alice"] is False
|
assert env.gateway.state.maintenance["alice"] is False
|
||||||
|
|
||||||
|
|
||||||
|
# --- SSO (reverse-proxy identity) ------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def enable_sso(env, header="X-Forwarded-User"):
|
||||||
|
"""Flip an env into SSO mode: the gateway trusts the named identity header."""
|
||||||
|
env.settings = Settings(**{**env.settings.__dict__, "auth_proxy_header": header})
|
||||||
|
env.gateway.settings = env.settings
|
||||||
|
return env
|
||||||
|
|
||||||
|
|
||||||
|
def test_remote_user_none_when_sso_disabled(env):
|
||||||
|
assert env.gateway.remote_user({"X-Forwarded-User": "alice"}) is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_remote_user_returns_valid_and_rejects_invalid(env):
|
||||||
|
enable_sso(env)
|
||||||
|
g = env.gateway
|
||||||
|
assert g.remote_user({"X-Forwarded-User": "alice"}) == "alice"
|
||||||
|
assert g.remote_user({"X-Forwarded-User": " bob "}) == "bob"
|
||||||
|
assert g.remote_user({"X-Forwarded-User": "UPPER"}) is None
|
||||||
|
assert g.remote_user({"X-Forwarded-User": "bad name"}) is None
|
||||||
|
assert g.remote_user({"X-Forwarded-User": ""}) is None
|
||||||
|
assert g.remote_user({}) is None
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_sso_landing_new_user_shows_provisioning(env):
|
||||||
|
enable_sso(env)
|
||||||
|
async with asgi_client(env) as client:
|
||||||
|
response = await client.get("/", headers={"X-Forwarded-User": "alice"})
|
||||||
|
assert response.status_code == 200
|
||||||
|
assert "text/html" in response.headers["content-type"]
|
||||||
|
assert "Signed in as <strong>alice</strong>" in response.text
|
||||||
|
assert "Provisioning" in response.text
|
||||||
|
assert "Recreate sandbox" not in response.text
|
||||||
|
assert re_cookie_user(response.headers["set-cookie"]) == "alice"
|
||||||
|
assert [c[0] for c in env.k8s.created] == ["alice"]
|
||||||
|
assert env.k8s.token_patches == []
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_sso_landing_ready_connects_and_offers_recreate_only(env):
|
||||||
|
enable_sso(env)
|
||||||
|
add_user(env.k8s)
|
||||||
|
async with asgi_client(env, cookies={"dsh_gw_session": f"alice:{TOKEN}"}) as client:
|
||||||
|
response = await client.get("/", headers={"X-Forwarded-User": "alice"})
|
||||||
|
assert response.status_code == 200
|
||||||
|
assert "Signed in as <strong>alice</strong>" in response.text
|
||||||
|
assert "Recreate sandbox" in response.text
|
||||||
|
# No signup/token forms for SSO users.
|
||||||
|
assert 'action="/users"' not in response.text
|
||||||
|
assert 'action="/login"' not in response.text
|
||||||
|
# Auto-connects to the workspace.
|
||||||
|
assert "url=/u/alice/" in response.text
|
||||||
|
assert env.k8s.created == []
|
||||||
|
assert env.k8s.token_patches == []
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_sso_landing_rotates_token_when_session_missing(env):
|
||||||
|
enable_sso(env)
|
||||||
|
add_user(env.k8s)
|
||||||
|
async with asgi_client(env) as client:
|
||||||
|
response = await client.get("/", headers={"X-Forwarded-User": "alice"})
|
||||||
|
assert response.status_code == 200
|
||||||
|
assert "Recreate sandbox" in response.text
|
||||||
|
assert re_cookie_user(response.headers["set-cookie"]) == "alice"
|
||||||
|
assert [p[0] for p in env.k8s.token_patches] == ["alice"]
|
||||||
|
assert env.k8s.created == []
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_recreate_deletes_and_reprovisions(env):
|
||||||
|
enable_sso(env)
|
||||||
|
add_user(env.k8s)
|
||||||
|
async with asgi_client(env) as client:
|
||||||
|
response = await client.post("/recreate", headers={"X-Forwarded-User": "alice"})
|
||||||
|
assert response.status_code == 303
|
||||||
|
assert response.headers["location"] == "/u/alice/"
|
||||||
|
assert env.k8s.deleted == ["alice"]
|
||||||
|
assert [c[0] for c in env.k8s.created] == ["alice"]
|
||||||
|
assert re_cookie_user(response.headers["set-cookie"]) == "alice"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_recreate_requires_sso(env):
|
||||||
|
add_user(env.k8s)
|
||||||
|
async with asgi_client(env) as client:
|
||||||
|
response = await client.post("/recreate", headers={"X-Forwarded-User": "alice"})
|
||||||
|
assert response.status_code == 401
|
||||||
|
assert env.k8s.deleted == []
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_recreate_rejects_cross_origin(env):
|
||||||
|
enable_sso(env)
|
||||||
|
add_user(env.k8s)
|
||||||
|
async with asgi_client(env, cookies={"dsh_gw_session": f"alice:{TOKEN}"}) as client:
|
||||||
|
response = await client.post(
|
||||||
|
"/recreate",
|
||||||
|
headers={"X-Forwarded-User": "alice", "Origin": "http://evil.test"},
|
||||||
|
)
|
||||||
|
assert response.status_code == 403
|
||||||
|
assert env.k8s.deleted == []
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_sso_get_user_authorizes_via_header(env):
|
||||||
|
enable_sso(env)
|
||||||
|
add_user(env.k8s)
|
||||||
|
async with asgi_client(env) as client:
|
||||||
|
response = await client.get("/users/alice", headers={"X-Forwarded-User": "alice"})
|
||||||
|
assert response.status_code == 200
|
||||||
|
assert response.json()["user"] == "alice"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_sso_proxy_redirects_to_bootstrap_without_session(env):
|
||||||
|
enable_sso(env)
|
||||||
|
add_user(env.k8s)
|
||||||
|
async with asgi_client(env) as client:
|
||||||
|
response = await client.get("/u/alice/", headers={"X-Forwarded-User": "alice"})
|
||||||
|
assert response.status_code == 303
|
||||||
|
assert response.headers["location"] == "/"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_sso_proxy_redirects_to_bootstrap_when_claim_missing(env):
|
||||||
|
enable_sso(env)
|
||||||
|
async with asgi_client(env) as client:
|
||||||
|
response = await client.get("/u/alice/", headers={"X-Forwarded-User": "alice"})
|
||||||
|
assert response.status_code == 303
|
||||||
|
assert response.headers["location"] == "/"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_spoofed_header_ignored_when_sso_disabled(env):
|
||||||
|
add_user(env.k8s)
|
||||||
|
async with asgi_client(env) as client:
|
||||||
|
response = await client.get("/u/alice/", headers={"X-Forwarded-User": "alice"})
|
||||||
|
assert response.status_code == 401
|
||||||
|
|||||||
44
examples/dsh-agents-as-a-service/manifests/00-prereqs.yaml
Normal file
44
examples/dsh-agents-as-a-service/manifests/00-prereqs.yaml
Normal file
@@ -0,0 +1,44 @@
|
|||||||
|
# DSH "Agents as a Service" — prerequisites: namespace + shared provider keys.
|
||||||
|
#
|
||||||
|
# The dsh-provider-keys Secret is deliberately NOT committed with a usable
|
||||||
|
# value. DEEPSEEK_API_KEY reaches every tenant sandbox (injected via envFrom
|
||||||
|
# on the SandboxTemplate, stamped before any user exists — a warm-pool
|
||||||
|
# constraint), so a real key committed here would be shared by every install
|
||||||
|
# of this example. Generate a fresh value per install (README step 1;
|
||||||
|
# scripts/e2e-kind.sh uses a throwaway). Use an upsert: this manifest already
|
||||||
|
# creates the empty Secret, so a plain `kubectl create` would fail:
|
||||||
|
#
|
||||||
|
# kubectl -n dsh-demo create secret generic dsh-provider-keys \
|
||||||
|
# --from-literal=DEEPSEEK_API_KEY="sk-..." \
|
||||||
|
# --dry-run=client -o yaml | kubectl apply -f -
|
||||||
|
#
|
||||||
|
# Optional: additional providers (OpenRouter, Gemini, NVIDIA, OpenAI, ...) go
|
||||||
|
# in a second, optional Secret. The SandboxTemplate references it with
|
||||||
|
# `optional: true`, so sandboxes start even when it does not exist, and every
|
||||||
|
# key in it becomes an environment variable. Create it BEFORE the warm pool,
|
||||||
|
# because pod env is stamped at pod creation:
|
||||||
|
#
|
||||||
|
# kubectl -n dsh-demo create secret generic dsh-provider-keys-extra \
|
||||||
|
# --from-literal=OPENROUTER_API_KEY="sk-or-..." \
|
||||||
|
# --from-literal=GEMINI_API_KEY="..." \
|
||||||
|
# --from-literal=NVIDIA_API_KEY="nvapi-..." \
|
||||||
|
# --dry-run=client -o yaml | kubectl apply -f -
|
||||||
|
#
|
||||||
|
# Supported env names, precedence, shared-key caveats and the pod-refresh
|
||||||
|
# procedure are in docs/optional-hardening.md.
|
||||||
|
#
|
||||||
|
# All the infrastructure in this example works without a real key; only
|
||||||
|
# actual model calls need one.
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Namespace
|
||||||
|
metadata:
|
||||||
|
name: dsh-demo
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Secret
|
||||||
|
metadata:
|
||||||
|
name: dsh-provider-keys
|
||||||
|
namespace: dsh-demo
|
||||||
|
type: Opaque
|
||||||
|
# stringData:
|
||||||
|
# DEEPSEEK_API_KEY: "sk-your-key-here"
|
||||||
@@ -1,7 +1,10 @@
|
|||||||
# The reusable DSH Web blueprint. Warm pools stamp pod env BEFORE any user
|
# The reusable DSH Web blueprint. Warm pools stamp pod env BEFORE any user
|
||||||
# exists, so per-user secrets cannot live in pod env; the shared provider key
|
# exists, so per-user secrets cannot live in pod env; the shared provider keys
|
||||||
# is injected via envFrom, and claims that would bypass the warm pool are
|
# are injected via envFrom, and claims that would bypass the warm pool are
|
||||||
# rejected outright (see 40-claim-rejected.yaml).
|
# rejected outright (see 40-claim-rejected.yaml).
|
||||||
|
#
|
||||||
|
# A stripped-down single-tenant variant of this template (no warm pool, no
|
||||||
|
# policy enforcement) lives in examples/minimal-single-tenant/.
|
||||||
apiVersion: extensions.agents.x-k8s.io/v1beta1
|
apiVersion: extensions.agents.x-k8s.io/v1beta1
|
||||||
kind: SandboxTemplate
|
kind: SandboxTemplate
|
||||||
metadata:
|
metadata:
|
||||||
@@ -18,16 +21,31 @@ spec:
|
|||||||
# The gateway routes each user to their DSH Web instance by this name.
|
# The gateway routes each user to their DSH Web instance by this name.
|
||||||
service: true
|
service: true
|
||||||
|
|
||||||
# No gateway pod scopes ingress in this standalone demo, so the
|
# The example leaves NetworkPolicy to an optional, opt-in manifest
|
||||||
# NetworkPolicy is left unmanaged. A production platform would use
|
# (manifests/optional/network-policy/): the base walkthrough should not
|
||||||
# networkPolicyManagement: Managed and restrict ingress to its gateway on
|
# require a policy-capable CNI, and the controller's Managed secure default
|
||||||
# :3080 — with the shared provider key, that NetworkPolicy is the per-user
|
# only allows the sandbox-router, which this example does not deploy. If you
|
||||||
# isolation boundary.
|
# switch to Managed, define spec.networkPolicy explicitly or the gateway will
|
||||||
|
# be blocked. See docs/optional-hardening.md.
|
||||||
networkPolicyManagement: Unmanaged
|
networkPolicyManagement: Unmanaged
|
||||||
|
|
||||||
podTemplate:
|
podTemplate:
|
||||||
spec:
|
spec:
|
||||||
# DSH Web is multi-arch (amd64 + arm64); no arch nodeSelector needed.
|
# DSH Web is multi-arch (amd64 + arm64); no arch nodeSelector is needed.
|
||||||
|
#
|
||||||
|
# OPTIONAL: run the sandbox under gVisor. Requires a RuntimeClass named
|
||||||
|
# "gvisor" and nodes that can run it; gVisor breaks `kubectl port-forward`
|
||||||
|
# into the sandbox (use the gateway). Existing Sandboxes snapshot their
|
||||||
|
# pod template, so enabling it later requires patching/recreating them.
|
||||||
|
# See docs/optional-hardening.md.
|
||||||
|
# runtimeClassName: gvisor
|
||||||
|
#
|
||||||
|
# OPTIONAL: pin sandboxes to a dedicated node pool (for example a gVisor
|
||||||
|
# pool). Replace this placeholder with a label your nodes actually carry;
|
||||||
|
# add a matching toleration if the pool is tainted. Active
|
||||||
|
# cluster-specific scheduling belongs in a local patch, not here.
|
||||||
|
# nodeSelector:
|
||||||
|
# dsh.example.com/node-pool: dsh
|
||||||
restartPolicy: "OnFailure"
|
restartPolicy: "OnFailure"
|
||||||
securityContext:
|
securityContext:
|
||||||
runAsUser: 1000 # the 'node' user in the DSH image
|
runAsUser: 1000 # the 'node' user in the DSH image
|
||||||
@@ -41,11 +59,12 @@ spec:
|
|||||||
# with 0660 and crash-loops before the gateway can proxy to it.
|
# with 0660 and crash-loops before the gateway can proxy to it.
|
||||||
# OnRootMismatch skips that walk once the volume root already carries
|
# OnRootMismatch skips that walk once the volume root already carries
|
||||||
# the fsGroup ownership/mode applied on the first mount.
|
# the fsGroup ownership/mode applied on the first mount.
|
||||||
# Keep in sync with deploy/k8s/sandbox-template.yaml.
|
# Keep in sync with the minimal variant:
|
||||||
|
# examples/minimal-single-tenant/manifests/sandbox-template.yaml.
|
||||||
fsGroupChangePolicy: OnRootMismatch
|
fsGroupChangePolicy: OnRootMismatch
|
||||||
containers:
|
containers:
|
||||||
- name: dsh-web
|
- name: dsh-web
|
||||||
image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.2
|
image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.3-r4
|
||||||
imagePullPolicy: IfNotPresent
|
imagePullPolicy: IfNotPresent
|
||||||
ports:
|
ports:
|
||||||
- name: web
|
- name: web
|
||||||
@@ -53,6 +72,13 @@ spec:
|
|||||||
envFrom:
|
envFrom:
|
||||||
- secretRef:
|
- secretRef:
|
||||||
name: dsh-provider-keys
|
name: dsh-provider-keys
|
||||||
|
# Optional extra provider keys (OpenRouter, Gemini, NVIDIA, ...).
|
||||||
|
# The Secret may be absent; every key in it becomes an env var.
|
||||||
|
# Create it before the warm pool, because pod env is stamped at
|
||||||
|
# pod creation. See docs/optional-hardening.md.
|
||||||
|
- secretRef:
|
||||||
|
name: dsh-provider-keys-extra
|
||||||
|
optional: true
|
||||||
env:
|
env:
|
||||||
# Empty => DSH derives links from the request Host, which the
|
# Empty => DSH derives links from the request Host, which the
|
||||||
# gateway forwards (plus X-Forwarded-*). Point this at the
|
# gateway forwards (plus X-Forwarded-*). Point this at the
|
||||||
@@ -61,8 +87,10 @@ spec:
|
|||||||
value: ""
|
value: ""
|
||||||
# DSH Host/Origin-fences /api. The gateway forwards the browser's
|
# DSH Host/Origin-fences /api. The gateway forwards the browser's
|
||||||
# Host, so DSH must trust the public gateway host. Empty is right
|
# Host, so DSH must trust the public gateway host. Empty is right
|
||||||
# for port-forward/localhost; for the optional Ingress set this to
|
# for port-forward/localhost; for either exposure variant set this
|
||||||
# the same host as 60-ingress.yaml (e.g. dsh.example.com).
|
# to the public host (e.g. dsh.example.com for
|
||||||
|
# manifests/exposure/ingress/60-ingress.yaml, or
|
||||||
|
# dsh-demo.example.com for the gateway-api-sso/ HTTPRoute).
|
||||||
- name: DSH_TRUSTED_HOSTS
|
- name: DSH_TRUSTED_HOSTS
|
||||||
value: ""
|
value: ""
|
||||||
# Probes target /favicon.svg: DSH gates / behind its launch-token
|
# Probes target /favicon.svg: DSH gates / behind its launch-token
|
||||||
@@ -6,11 +6,19 @@
|
|||||||
# POST /users, POST /login, ... session cookie auth
|
# POST /users, POST /login, ... session cookie auth
|
||||||
# /u/<user>/... HTTP + WebSocket proxy to the user's DSH
|
# /u/<user>/... HTTP + WebSocket proxy to the user's DSH
|
||||||
#
|
#
|
||||||
# Image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway
|
# Image: the release image published for this example, pinned by tag.
|
||||||
# For local kind development you can build and load a local tag instead, then
|
# imagePullPolicy: IfNotPresent lets a local kind build loaded under the same
|
||||||
# change `image` below to it:
|
# tag be used without a registry round-trip (scripts/e2e-kind.sh builds
|
||||||
# docker build -t dsh-aaas-gateway:demo gateway/
|
# dsh-aaas-gateway:e2e and overrides this image).
|
||||||
# kind load docker-image dsh-aaas-gateway:demo
|
#
|
||||||
|
# SSO IS OPT-IN: this manifest runs the gateway with its own bearer/session
|
||||||
|
# auth only, so AUTH_PROXY_HEADER is intentionally NOT set (see the env
|
||||||
|
# block). To put GitHub OAuth in front of it, use
|
||||||
|
# manifests/exposure/gateway-api-sso/.
|
||||||
|
#
|
||||||
|
# This Service is ClusterIP and is the only intended entry point: expose it
|
||||||
|
# through exactly ONE of the documented variants in manifests/exposure/
|
||||||
|
# (ingress/ or gateway-api-sso/), never both and never directly.
|
||||||
apiVersion: v1
|
apiVersion: v1
|
||||||
kind: ServiceAccount
|
kind: ServiceAccount
|
||||||
metadata:
|
metadata:
|
||||||
@@ -76,6 +84,15 @@ spec:
|
|||||||
app: dsh-gateway
|
app: dsh-gateway
|
||||||
spec:
|
spec:
|
||||||
serviceAccountName: dsh-gateway
|
serviceAccountName: dsh-gateway
|
||||||
|
# OPTIONAL: run the gateway under gVisor. Requires a RuntimeClass named
|
||||||
|
# "gvisor" and nodes that can run it. See docs/optional-hardening.md.
|
||||||
|
# runtimeClassName: gvisor
|
||||||
|
#
|
||||||
|
# OPTIONAL: pin the gateway to a dedicated node pool. Replace the
|
||||||
|
# placeholder with a label your nodes carry; add a toleration for a
|
||||||
|
# tainted pool. Keep active cluster-specific scheduling in a local patch.
|
||||||
|
# nodeSelector:
|
||||||
|
# dsh.example.com/node-pool: dsh
|
||||||
securityContext:
|
securityContext:
|
||||||
runAsNonRoot: true
|
runAsNonRoot: true
|
||||||
runAsUser: 10000 # matches USER in gateway/Dockerfile
|
runAsUser: 10000 # matches USER in gateway/Dockerfile
|
||||||
@@ -84,7 +101,7 @@ spec:
|
|||||||
type: RuntimeDefault
|
type: RuntimeDefault
|
||||||
containers:
|
containers:
|
||||||
- name: gateway
|
- name: gateway
|
||||||
image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway
|
image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway:dsh-v0.2.1-rc.3-r4
|
||||||
imagePullPolicy: IfNotPresent
|
imagePullPolicy: IfNotPresent
|
||||||
securityContext:
|
securityContext:
|
||||||
allowPrivilegeEscalation: false
|
allowPrivilegeEscalation: false
|
||||||
@@ -105,20 +122,36 @@ spec:
|
|||||||
# gateway created are swept: a claim applied by hand from the
|
# gateway created are swept: a claim applied by hand from the
|
||||||
# resource-layer manifests carries no aaas.example.com/managed-by
|
# resource-layer manifests carries no aaas.example.com/managed-by
|
||||||
# annotation and is never suspended.
|
# annotation and is never suspended.
|
||||||
|
#
|
||||||
|
# These are the demo values (360/240). scripts/e2e-kind.sh
|
||||||
|
# overrides both to 60/120 with `kubectl set env` so the smoke
|
||||||
|
# test does not wait six minutes for the sweeper.
|
||||||
- name: IDLE_TIMEOUT
|
- name: IDLE_TIMEOUT
|
||||||
value: "60"
|
value: "360"
|
||||||
- name: WAKE_TIMEOUT
|
- name: WAKE_TIMEOUT
|
||||||
value: "120"
|
value: "240"
|
||||||
# Container name inside the per-user sandbox pods whose logs carry
|
# Container name inside the per-user sandbox pods whose logs carry
|
||||||
# the DSH launch token.
|
# the DSH launch token.
|
||||||
- name: DSH_CONTAINER
|
- name: DSH_CONTAINER
|
||||||
value: dsh-web
|
value: dsh-web
|
||||||
# Cookie dsh_gw_session is HttpOnly + SameSite=Lax. Set to "true"
|
# Cookie dsh_gw_session is HttpOnly + SameSite=Lax. "false" keeps
|
||||||
# when this Service is reached over HTTPS (e.g. the optional
|
# the plain-HTTP demo usable through port-forward: a Secure cookie
|
||||||
# 60-ingress.yaml with TLS); "false" keeps the plain-HTTP demo
|
# is never sent over HTTP, so login would silently fail.
|
||||||
# usable via port-forward.
|
# HTTPS deployments (the ingress/ variant with TLS, or
|
||||||
|
# gateway-api-sso/, which terminates TLS at the Gateway) MUST set
|
||||||
|
# this to "true".
|
||||||
- name: COOKIE_SECURE
|
- name: COOKIE_SECURE
|
||||||
value: "false"
|
value: "false"
|
||||||
|
# SSO IS OPT-IN AND DISABLED HERE. With AUTH_PROXY_HEADER unset the
|
||||||
|
# gateway trusts only its own bearer/session auth and ignores every
|
||||||
|
# identity header. Enable it ONLY behind
|
||||||
|
# manifests/exposure/gateway-api-sso/, where oauth2-proxy is the
|
||||||
|
# sole route to this Service: set it to X-Forwarded-User, keep the
|
||||||
|
# Service ClusterIP and reachable only through the proxy, and
|
||||||
|
# terminate TLS in front of the gateway. Otherwise anyone who can
|
||||||
|
# reach :8080 can impersonate any user by sending that header.
|
||||||
|
# - name: AUTH_PROXY_HEADER
|
||||||
|
# value: X-Forwarded-User
|
||||||
# Reject proxied request bodies larger than this (413).
|
# Reject proxied request bodies larger than this (413).
|
||||||
- name: MAX_BODY_MB
|
- name: MAX_BODY_MB
|
||||||
value: "32"
|
value: "32"
|
||||||
156
examples/dsh-agents-as-a-service/manifests/exposure/README.md
Normal file
156
examples/dsh-agents-as-a-service/manifests/exposure/README.md
Normal file
@@ -0,0 +1,156 @@
|
|||||||
|
# Exposing the gateway
|
||||||
|
|
||||||
|
The base manifests deploy the gateway as a ClusterIP Service
|
||||||
|
(`dsh-gateway:8080`) and expose nothing outside the cluster. WebSocket
|
||||||
|
upgrades on `/u/<user>/**` are part of the contract, so the entry point must
|
||||||
|
forward `Upgrade`/`Connection` headers (Traefik does by default).
|
||||||
|
|
||||||
|
There are two **mutually exclusive** exposure variants. Pick exactly one:
|
||||||
|
|
||||||
|
| Variant | Entry point | Authentication | TLS |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| [`ingress/60-ingress.yaml`](ingress/60-ingress.yaml) | Traefik `Ingress` | the gateway's own signup/session cookie | optional, recommended |
|
||||||
|
| [`gateway-api-sso/`](gateway-api-sso/72-oauth2-proxy.yaml) | Gateway API `Gateway` + `HTTPRoute` | GitHub OAuth via oauth2-proxy, then the gateway cookie | required (the Gateway listener terminates it) |
|
||||||
|
|
||||||
|
Applying both gives the gateway two entry points, and the SSO variant is only
|
||||||
|
safe when the gateway is reachable **exclusively** through oauth2-proxy. Do not
|
||||||
|
mix them.
|
||||||
|
|
||||||
|
Each variant also ships a NetworkPolicy, applied by the same directory apply:
|
||||||
|
|
||||||
|
- `ingress/61-networkpolicy.yaml` allows only Traefik pods to reach the gateway
|
||||||
|
on `8080`.
|
||||||
|
- `gateway-api-sso/75-networkpolicies.yaml` allows only oauth2-proxy to reach
|
||||||
|
the gateway, only the Gateway implementation (Traefik) to reach oauth2-proxy,
|
||||||
|
and limits oauth2-proxy egress to DNS, public HTTPS (GitHub) and the gateway.
|
||||||
|
|
||||||
|
The policies assume Traefik's `app.kubernetes.io/name: traefik` pod label.
|
||||||
|
Adjust the selector for another implementation or a `hostNetwork` install; see
|
||||||
|
[optional hardening](../../docs/optional-hardening.md) for the probe caveat and
|
||||||
|
verification commands.
|
||||||
|
|
||||||
|
## Applying a variant
|
||||||
|
|
||||||
|
Run these commands from the example root
|
||||||
|
(`examples/dsh-agents-as-a-service/`), the same working directory the example
|
||||||
|
README uses.
|
||||||
|
|
||||||
|
`kubectl apply -f manifests/` is **not recursive**: it applies only the
|
||||||
|
top-level numbered files and does not descend into `exposure/`. Apply the base
|
||||||
|
resources first, then the chosen variant explicitly:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Base resources: namespace, template, warm pool, claims, gateway.
|
||||||
|
kubectl apply -f manifests/
|
||||||
|
|
||||||
|
# Plain Ingress exposure:
|
||||||
|
kubectl apply -f manifests/exposure/ingress/
|
||||||
|
|
||||||
|
# ... or SSO exposure (create the two Secrets and replace the example.com
|
||||||
|
# placeholders first — see below):
|
||||||
|
kubectl apply -f manifests/exposure/gateway-api-sso/
|
||||||
|
```
|
||||||
|
|
||||||
|
**Never use `kubectl apply -R`** (recursive): it would apply both variants at
|
||||||
|
once and hand out a second, unauthenticated path to the same gateway. Apply the
|
||||||
|
variant directory on purpose, or the individual files by name.
|
||||||
|
|
||||||
|
`kubectl` only reads `.yaml`, `.yml` and `.json` files from a directory, so
|
||||||
|
the `*.yaml.example` secret templates are skipped automatically; create the real
|
||||||
|
Secrets imperatively (commands below) before applying the directory.
|
||||||
|
|
||||||
|
## `ingress/`: plain Ingress, no SSO
|
||||||
|
|
||||||
|
`60-ingress.yaml` routes a host through the cluster's Traefik
|
||||||
|
(`ingressClassName: traefik`; k3s ships one) to the gateway Service.
|
||||||
|
|
||||||
|
1. Replace the `dsh.example.com` host and the `dsh-example-tls` Secret name
|
||||||
|
with real values, or delete the `tls:` block to serve plain HTTP.
|
||||||
|
2. Set `DSH_TRUSTED_HOSTS` in the SandboxTemplate to the same public host:
|
||||||
|
the gateway forwards the browser's `Host`, and DSH rejects unknown hosts on
|
||||||
|
`/api` (see `../10-sandboxtemplate.yaml`).
|
||||||
|
3. Set `COOKIE_SECURE=true` on the gateway once the Ingress serves HTTPS:
|
||||||
|
`kubectl -n dsh-demo set env deploy/dsh-gateway COOKIE_SECURE=true`. The
|
||||||
|
base manifest keeps `false` so the plain-HTTP port-forward demo works; a
|
||||||
|
`Secure` cookie is never sent over HTTP, which silently breaks login.
|
||||||
|
4. Do not attach buffering or compression middlewares to this router: they
|
||||||
|
break WebSocket upgrades and streaming.
|
||||||
|
5. `61-networkpolicy.yaml` is part of this variant: it allows only Traefik pods
|
||||||
|
to reach the gateway on `8080`. Adjust its selector for another ingress
|
||||||
|
controller or `hostNetwork`; if kubelet probes fail, see the hardening doc.
|
||||||
|
|
||||||
|
Verify: `curl -sI https://<host>/healthz` returns `200`.
|
||||||
|
|
||||||
|
## `gateway-api-sso/`: TLS + GitHub OAuth
|
||||||
|
|
||||||
|
`70-gateway.yaml` defines the TLS `Gateway`, `71-httproute.yaml` sends every
|
||||||
|
request to oauth2-proxy, and `72-oauth2-proxy.yaml` authenticates the browser
|
||||||
|
against GitHub and proxies to the gateway with `X-Forwarded-User` set.
|
||||||
|
`75-networkpolicies.yaml` makes this the only network path to the gateway.
|
||||||
|
|
||||||
|
Prerequisites:
|
||||||
|
|
||||||
|
- Gateway API CRDs and a Gateway implementation installed;
|
||||||
|
`gatewayClassName: traefik` matches the Traefik Gateway provider. The CRD
|
||||||
|
version is pinned in `versions.env` (`GATEWAY_API_VERSION`).
|
||||||
|
- The OAuth App credentials and a cookie secret, from
|
||||||
|
`73-oauth2-proxy-secret.yaml.example`.
|
||||||
|
- A TLS certificate/key for the public host, from
|
||||||
|
`74-tls-secret.yaml.example`.
|
||||||
|
- oauth2-proxy is pinned in the manifest to
|
||||||
|
`quay.io/oauth2-proxy/oauth2-proxy:v7.15.5` (`versions.env`:
|
||||||
|
`OAUTH2_PROXY_IMAGE`).
|
||||||
|
|
||||||
|
Steps:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. Real Secrets (the committed files are *.yaml.example templates).
|
||||||
|
kubectl -n dsh-demo create secret generic oauth2-proxy \
|
||||||
|
--from-literal=client-id='<GitHub OAuth App client ID>' \
|
||||||
|
--from-literal=client-secret='<GitHub OAuth App client secret>' \
|
||||||
|
--from-literal=cookie-secret="$(python3 -c 'import secrets; print(secrets.token_urlsafe(32))')"
|
||||||
|
kubectl -n dsh-demo create secret tls dsh-demo-tls --cert=tls.crt --key=tls.key
|
||||||
|
|
||||||
|
# 2. Replace dsh-demo.example.com (70/71/72) and your-github-org (72) with
|
||||||
|
# the real public host and allowed GitHub organization.
|
||||||
|
|
||||||
|
# 3. Enable SSO on the gateway and trust the public host in the template.
|
||||||
|
kubectl -n dsh-demo set env deploy/dsh-gateway \
|
||||||
|
AUTH_PROXY_HEADER=X-Forwarded-User COOKIE_SECURE=true
|
||||||
|
# then set DSH_TRUSTED_HOSTS=dsh-demo.example.com in the SandboxTemplate
|
||||||
|
# (see ../10-sandboxtemplate.yaml) and re-apply it.
|
||||||
|
|
||||||
|
# 4. Apply the variant.
|
||||||
|
kubectl apply -f manifests/exposure/gateway-api-sso/
|
||||||
|
```
|
||||||
|
|
||||||
|
### SSO trust boundary
|
||||||
|
|
||||||
|
`AUTH_PROXY_HEADER` is opt-in and deliberately unset in the base gateway
|
||||||
|
manifest: the gateway ignores identity headers until it is set. Enabling SSO
|
||||||
|
takes all of these:
|
||||||
|
|
||||||
|
1. `AUTH_PROXY_HEADER=X-Forwarded-User` on `deploy/dsh-gateway` (step 3
|
||||||
|
above). Without it, oauth2-proxy's login has no effect on the gateway and
|
||||||
|
users are still asked for the gateway's own token.
|
||||||
|
2. The gateway Service must be reachable **only through oauth2-proxy**: keep it
|
||||||
|
ClusterIP, route all browser traffic through the `HTTPRoute` →
|
||||||
|
oauth2-proxy → `dsh-gateway` chain, and never apply `ingress/` (or any
|
||||||
|
other path to the Service) while SSO is enabled. A client that can reach
|
||||||
|
`dsh-gateway:8080` directly can set `X-Forwarded-User` itself and
|
||||||
|
impersonate any user.
|
||||||
|
3. Keep `75-networkpolicies.yaml` with the variant. It enforces the boundary in
|
||||||
|
the network layer (gateway ← oauth2-proxy ← Traefik, and limited oauth2-proxy
|
||||||
|
egress). A custom exposure without an equivalent policy has no network
|
||||||
|
enforcement.
|
||||||
|
|
||||||
|
Also terminate TLS in front of the gateway (the Gateway listener does) and
|
||||||
|
leave `COOKIE_SECURE=true` so the session cookie never travels in clear text.
|
||||||
|
|
||||||
|
## Switching variants
|
||||||
|
|
||||||
|
Delete the variant you had before applying the other one, e.g.
|
||||||
|
`kubectl delete -f manifests/exposure/ingress/`, and revert
|
||||||
|
`AUTH_PROXY_HEADER`/`COOKIE_SECURE` on the gateway. Verify with
|
||||||
|
`kubectl -n dsh-demo get ingress,gateway,httproute` that exactly one path to
|
||||||
|
the gateway remains.
|
||||||
@@ -0,0 +1,25 @@
|
|||||||
|
# Browser entry point (TLS) for the gateway-api-sso variant: terminates TLS
|
||||||
|
# for dsh-demo.example.com and routes through 71-httproute.yaml to
|
||||||
|
# oauth2-proxy, which authenticates the browser before the gateway.
|
||||||
|
# Mutually exclusive with ../../ingress/ (see ../../README.md). The
|
||||||
|
# "dsh-demo-tls" Secret comes from 74-tls-secret.yaml.example.
|
||||||
|
apiVersion: gateway.networking.k8s.io/v1
|
||||||
|
kind: Gateway
|
||||||
|
metadata:
|
||||||
|
name: dsh-gateway
|
||||||
|
namespace: dsh-demo
|
||||||
|
labels:
|
||||||
|
app.kubernetes.io/part-of: dsh-demo
|
||||||
|
spec:
|
||||||
|
gatewayClassName: traefik
|
||||||
|
listeners:
|
||||||
|
- name: https
|
||||||
|
port: 8443
|
||||||
|
protocol: HTTPS
|
||||||
|
tls:
|
||||||
|
mode: Terminate
|
||||||
|
certificateRefs:
|
||||||
|
- name: dsh-demo-tls
|
||||||
|
allowedRoutes:
|
||||||
|
namespaces:
|
||||||
|
from: Same
|
||||||
@@ -0,0 +1,23 @@
|
|||||||
|
apiVersion: gateway.networking.k8s.io/v1
|
||||||
|
kind: HTTPRoute
|
||||||
|
metadata:
|
||||||
|
name: dsh-gateway
|
||||||
|
namespace: dsh-demo
|
||||||
|
labels:
|
||||||
|
app.kubernetes.io/part-of: dsh-demo
|
||||||
|
spec:
|
||||||
|
parentRefs:
|
||||||
|
- name: dsh-gateway
|
||||||
|
namespace: dsh-demo
|
||||||
|
sectionName: https
|
||||||
|
hostnames:
|
||||||
|
# TODO: replace with the actual public FQDN before applying.
|
||||||
|
- dsh-demo.example.com
|
||||||
|
rules:
|
||||||
|
- backendRefs:
|
||||||
|
# Route through oauth2-proxy (GitHub OAuth) first; it authenticates the
|
||||||
|
# browser and forwards to dsh-gateway:8080 with X-Forwarded-User set.
|
||||||
|
# The gateway only trusts that header when AUTH_PROXY_HEADER is set on
|
||||||
|
# its Deployment — see 72-oauth2-proxy.yaml and ../README.md.
|
||||||
|
- name: oauth2-proxy
|
||||||
|
port: 4180
|
||||||
@@ -0,0 +1,145 @@
|
|||||||
|
# oauth2-proxy — GitHub OAuth login in front of the dsh-gateway.
|
||||||
|
#
|
||||||
|
# Runs as a reverse proxy: it authenticates the browser against GitHub
|
||||||
|
# (restricted to a configured organization), then forwards the request to
|
||||||
|
# the dsh-gateway Service with the authenticated identity in X-Forwarded-User
|
||||||
|
# (plus X-Auth-Request-* via --set-xauthrequest). The gateway trusts that
|
||||||
|
# header only when AUTH_PROXY_HEADER is set on its Deployment.
|
||||||
|
#
|
||||||
|
# ENABLING SSO TAKES BOTH SIDES (see ../../../exposure/README.md):
|
||||||
|
# 1. Set AUTH_PROXY_HEADER=X-Forwarded-User on deploy/dsh-gateway, otherwise
|
||||||
|
# the gateway ignores the header and the login is cosmetic:
|
||||||
|
# kubectl -n dsh-demo set env deploy/dsh-gateway \
|
||||||
|
# AUTH_PROXY_HEADER=X-Forwarded-User COOKIE_SECURE=true
|
||||||
|
# 2. Make dsh-gateway reachable ONLY through this proxy: keep the Service
|
||||||
|
# ClusterIP, route browser traffic through 71-httproute.yaml, and do not
|
||||||
|
# apply ../ingress/ (or anything else that reaches the gateway). A
|
||||||
|
# client that can reach dsh-gateway:8080 can spoof X-Forwarded-User and
|
||||||
|
# impersonate any user.
|
||||||
|
#
|
||||||
|
# 75-networkpolicies.yaml enforces that boundary at the network layer: the
|
||||||
|
# gateway accepts traffic only from oauth2-proxy pods, oauth2-proxy only from
|
||||||
|
# the Gateway implementation, and oauth2-proxy egress is limited to DNS,
|
||||||
|
# public HTTPS and the gateway. Apply it with this variant.
|
||||||
|
#
|
||||||
|
# TLS terminates at the Gateway listener (70-gateway.yaml); keep
|
||||||
|
# --cookie-secure=true and COOKIE_SECURE=true on the gateway.
|
||||||
|
#
|
||||||
|
# The image is pinned to quay.io/oauth2-proxy/oauth2-proxy:v7.15.5
|
||||||
|
# (versions.env: OAUTH2_PROXY_IMAGE). Credentials come from the
|
||||||
|
# "oauth2-proxy" Secret (see 73-oauth2-proxy-secret.yaml.example); this
|
||||||
|
# manifest never carries secrets.
|
||||||
|
apiVersion: apps/v1
|
||||||
|
kind: Deployment
|
||||||
|
metadata:
|
||||||
|
name: oauth2-proxy
|
||||||
|
namespace: dsh-demo
|
||||||
|
labels:
|
||||||
|
app: oauth2-proxy
|
||||||
|
app.kubernetes.io/part-of: dsh-demo
|
||||||
|
spec:
|
||||||
|
replicas: 1
|
||||||
|
selector:
|
||||||
|
matchLabels:
|
||||||
|
app: oauth2-proxy
|
||||||
|
template:
|
||||||
|
metadata:
|
||||||
|
labels:
|
||||||
|
app: oauth2-proxy
|
||||||
|
app.kubernetes.io/part-of: dsh-demo
|
||||||
|
spec:
|
||||||
|
# OPTIONAL: run the proxy under gVisor and/or pin it to a node pool.
|
||||||
|
# See ../../../docs/optional-hardening.md.
|
||||||
|
# runtimeClassName: gvisor
|
||||||
|
# nodeSelector:
|
||||||
|
# dsh.example.com/node-pool: dsh
|
||||||
|
securityContext:
|
||||||
|
runAsNonRoot: true
|
||||||
|
runAsUser: 2000
|
||||||
|
runAsGroup: 2000
|
||||||
|
seccompProfile:
|
||||||
|
type: RuntimeDefault
|
||||||
|
containers:
|
||||||
|
- name: oauth2-proxy
|
||||||
|
image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.5
|
||||||
|
imagePullPolicy: IfNotPresent
|
||||||
|
securityContext:
|
||||||
|
allowPrivilegeEscalation: false
|
||||||
|
readOnlyRootFilesystem: true
|
||||||
|
capabilities:
|
||||||
|
drop: ["ALL"]
|
||||||
|
args:
|
||||||
|
- --http-address=0.0.0.0:4180
|
||||||
|
- --provider=github
|
||||||
|
# TODO: replace with the GitHub organization allowed to sign in.
|
||||||
|
- --github-org=your-github-org
|
||||||
|
- --email-domain=*
|
||||||
|
# Reverse-proxy target: the gateway backend Service.
|
||||||
|
- --upstream=http://dsh-gateway:8080
|
||||||
|
# Public callback (TLS terminates at the Gateway listener).
|
||||||
|
# TODO: replace with the actual public FQDN.
|
||||||
|
- --redirect-url=https://dsh-demo.example.com/oauth2/callback
|
||||||
|
- --cookie-secure=true
|
||||||
|
- --cookie-name=_oauth2_proxy
|
||||||
|
# Also expose X-Auth-Request-User/Email/Groups upstream.
|
||||||
|
- --set-xauthrequest
|
||||||
|
# Single provider: skip the provider-selection page.
|
||||||
|
- --skip-provider-button
|
||||||
|
env:
|
||||||
|
- name: OAUTH2_PROXY_CLIENT_ID
|
||||||
|
valueFrom:
|
||||||
|
secretKeyRef:
|
||||||
|
name: oauth2-proxy
|
||||||
|
key: client-id
|
||||||
|
- name: OAUTH2_PROXY_CLIENT_SECRET
|
||||||
|
valueFrom:
|
||||||
|
secretKeyRef:
|
||||||
|
name: oauth2-proxy
|
||||||
|
key: client-secret
|
||||||
|
- name: OAUTH2_PROXY_COOKIE_SECRET
|
||||||
|
valueFrom:
|
||||||
|
secretKeyRef:
|
||||||
|
name: oauth2-proxy
|
||||||
|
key: cookie-secret
|
||||||
|
ports:
|
||||||
|
- name: http
|
||||||
|
containerPort: 4180
|
||||||
|
livenessProbe:
|
||||||
|
httpGet:
|
||||||
|
path: /ping
|
||||||
|
port: http
|
||||||
|
initialDelaySeconds: 5
|
||||||
|
periodSeconds: 10
|
||||||
|
timeoutSeconds: 1
|
||||||
|
failureThreshold: 3
|
||||||
|
readinessProbe:
|
||||||
|
httpGet:
|
||||||
|
path: /ready
|
||||||
|
port: http
|
||||||
|
initialDelaySeconds: 5
|
||||||
|
periodSeconds: 10
|
||||||
|
timeoutSeconds: 5
|
||||||
|
failureThreshold: 3
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
cpu: 10m
|
||||||
|
memory: 32Mi
|
||||||
|
limits:
|
||||||
|
cpu: 200m
|
||||||
|
memory: 128Mi
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Service
|
||||||
|
metadata:
|
||||||
|
name: oauth2-proxy
|
||||||
|
namespace: dsh-demo
|
||||||
|
labels:
|
||||||
|
app: oauth2-proxy
|
||||||
|
app.kubernetes.io/part-of: dsh-demo
|
||||||
|
spec:
|
||||||
|
selector:
|
||||||
|
app: oauth2-proxy
|
||||||
|
ports:
|
||||||
|
- name: http
|
||||||
|
port: 4180
|
||||||
|
targetPort: http
|
||||||
@@ -0,0 +1,39 @@
|
|||||||
|
# EXAMPLE ONLY — do not apply as-is.
|
||||||
|
#
|
||||||
|
# oauth2-proxy reads its GitHub OAuth credentials and cookie-signing secret from
|
||||||
|
# this Secret. Populate real values and apply:
|
||||||
|
#
|
||||||
|
# kubectl -n dsh-demo create secret generic oauth2-proxy \
|
||||||
|
# --from-literal=client-id='<GitHub OAuth App client ID>' \
|
||||||
|
# --from-literal=client-secret='<GitHub OAuth App client secret>' \
|
||||||
|
# --from-literal=cookie-secret="$(python3 -c 'import secrets; print(secrets.token_urlsafe(32))')"
|
||||||
|
#
|
||||||
|
# The cookie secret must decode (base64) to exactly 16, 24, or 32 bytes.
|
||||||
|
# oauth2-proxy base64-decodes it, so it must be URL-safe, unpadded base64.
|
||||||
|
# Generate it with either of:
|
||||||
|
#
|
||||||
|
# python3 -c 'import secrets; print(secrets.token_urlsafe(32))'
|
||||||
|
# openssl rand -base64 32 | tr -- '+/' '-_' | tr -d '='
|
||||||
|
#
|
||||||
|
# Do NOT use plain `openssl rand -base64 32`: that yields 44 characters of
|
||||||
|
# standard base64 (with "=" and possibly "+"/"/"), which oauth2-proxy cannot
|
||||||
|
# decode and therefore treats as 44 raw bytes, failing with "cookie_secret must
|
||||||
|
# be 16, 24, or 32 bytes ... but is 44 bytes".
|
||||||
|
#
|
||||||
|
# GitHub OAuth App (https://github.com/settings/developers):
|
||||||
|
# Homepage URL https://dsh-demo.example.com
|
||||||
|
# Authorization callback https://dsh-demo.example.com/oauth2/callback
|
||||||
|
# (replace dsh-demo.example.com with the actual public FQDN)
|
||||||
|
#
|
||||||
|
# NOTE: pass the base64 value as-is via --from-literal (kubectl stores the
|
||||||
|
# literal; it does not base64-encode it a second time).
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Secret
|
||||||
|
metadata:
|
||||||
|
name: oauth2-proxy
|
||||||
|
namespace: dsh-demo
|
||||||
|
type: Opaque
|
||||||
|
stringData:
|
||||||
|
client-id: "<github-oauth-app-client-id>"
|
||||||
|
client-secret: "<github-oauth-app-client-secret>"
|
||||||
|
cookie-secret: "<urlsafe-unpadded-base64-of-32-random-bytes>"
|
||||||
@@ -0,0 +1,17 @@
|
|||||||
|
# EXAMPLE ONLY — do not apply as-is.
|
||||||
|
#
|
||||||
|
# Copy to tls-secret.yaml and populate with real base64-encoded PEM material,
|
||||||
|
# then apply. The secret must live in the dsh-demo namespace and be named
|
||||||
|
# "dsh-demo-tls" (referenced by 70-gateway.yaml).
|
||||||
|
#
|
||||||
|
# kubectl -n dsh-demo create secret tls dsh-demo-tls \
|
||||||
|
# --cert=tls.crt --key=tls.key --dry-run=client -o yaml > tls-secret.yaml
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Secret
|
||||||
|
metadata:
|
||||||
|
name: dsh-demo-tls
|
||||||
|
namespace: dsh-demo
|
||||||
|
type: kubernetes.io/tls
|
||||||
|
data:
|
||||||
|
tls.crt: <base64-encoded-certificate>
|
||||||
|
tls.key: <base64-encoded-private-key>
|
||||||
@@ -0,0 +1,120 @@
|
|||||||
|
# OPTIONAL with the SSO exposure variant: enforce the trust boundary at the
|
||||||
|
# network layer.
|
||||||
|
#
|
||||||
|
# The chain is Traefik (Gateway API) -> oauth2-proxy -> dsh-gateway. These
|
||||||
|
# policies make it the ONLY chain:
|
||||||
|
# - the gateway accepts traffic only from oauth2-proxy pods;
|
||||||
|
# - oauth2-proxy accepts traffic only from the Gateway implementation;
|
||||||
|
# - oauth2-proxy egress is limited to DNS, public HTTPS (GitHub) and the
|
||||||
|
# gateway.
|
||||||
|
# The gateway's own egress (Kubernetes API, sandbox pods) is intentionally not
|
||||||
|
# restricted here: the API-server endpoint is cluster-specific. See
|
||||||
|
# ../../../docs/optional-hardening.md for that recipe.
|
||||||
|
#
|
||||||
|
# This manifest assumes Traefik, matching gatewayClassName: traefik in
|
||||||
|
# 70-gateway.yaml. Adjust the app.kubernetes.io/name selector for another
|
||||||
|
# implementation. NetworkPolicy enforcement of kubelet probes is CNI-dependent;
|
||||||
|
# if the gateway or proxy fails its probes, allow the node CIDR on the probe
|
||||||
|
# port or remove the ingress policy.
|
||||||
|
apiVersion: networking.k8s.io/v1
|
||||||
|
kind: NetworkPolicy
|
||||||
|
metadata:
|
||||||
|
name: dsh-gateway-ingress
|
||||||
|
namespace: dsh-demo
|
||||||
|
spec:
|
||||||
|
podSelector:
|
||||||
|
matchLabels:
|
||||||
|
app: dsh-gateway
|
||||||
|
policyTypes:
|
||||||
|
- Ingress
|
||||||
|
ingress:
|
||||||
|
- from:
|
||||||
|
- podSelector:
|
||||||
|
matchLabels:
|
||||||
|
app: oauth2-proxy
|
||||||
|
ports:
|
||||||
|
- protocol: TCP
|
||||||
|
port: 8080
|
||||||
|
---
|
||||||
|
apiVersion: networking.k8s.io/v1
|
||||||
|
kind: NetworkPolicy
|
||||||
|
metadata:
|
||||||
|
name: oauth2-proxy-ingress
|
||||||
|
namespace: dsh-demo
|
||||||
|
spec:
|
||||||
|
podSelector:
|
||||||
|
matchLabels:
|
||||||
|
app: oauth2-proxy
|
||||||
|
policyTypes:
|
||||||
|
- Ingress
|
||||||
|
ingress:
|
||||||
|
- from:
|
||||||
|
- namespaceSelector: {}
|
||||||
|
podSelector:
|
||||||
|
matchLabels:
|
||||||
|
app.kubernetes.io/name: traefik
|
||||||
|
ports:
|
||||||
|
- protocol: TCP
|
||||||
|
port: 4180
|
||||||
|
---
|
||||||
|
apiVersion: networking.k8s.io/v1
|
||||||
|
kind: NetworkPolicy
|
||||||
|
metadata:
|
||||||
|
name: oauth2-proxy-egress
|
||||||
|
namespace: dsh-demo
|
||||||
|
spec:
|
||||||
|
podSelector:
|
||||||
|
matchLabels:
|
||||||
|
app: oauth2-proxy
|
||||||
|
policyTypes:
|
||||||
|
- Egress
|
||||||
|
egress:
|
||||||
|
# Cluster DNS.
|
||||||
|
- to:
|
||||||
|
- namespaceSelector:
|
||||||
|
matchLabels:
|
||||||
|
kubernetes.io/metadata.name: kube-system
|
||||||
|
podSelector:
|
||||||
|
matchLabels:
|
||||||
|
k8s-app: kube-dns
|
||||||
|
ports:
|
||||||
|
- protocol: UDP
|
||||||
|
port: 53
|
||||||
|
- protocol: TCP
|
||||||
|
port: 53
|
||||||
|
# NodeLocal DNSCache.
|
||||||
|
- to:
|
||||||
|
- ipBlock:
|
||||||
|
cidr: 169.254.20.10/32
|
||||||
|
ports:
|
||||||
|
- protocol: UDP
|
||||||
|
port: 53
|
||||||
|
- protocol: TCP
|
||||||
|
port: 53
|
||||||
|
# The gateway backend.
|
||||||
|
- to:
|
||||||
|
- podSelector:
|
||||||
|
matchLabels:
|
||||||
|
app: dsh-gateway
|
||||||
|
ports:
|
||||||
|
- protocol: TCP
|
||||||
|
port: 8080
|
||||||
|
# GitHub OAuth endpoints are public HTTPS. Private ranges and the metadata
|
||||||
|
# server stay blocked.
|
||||||
|
- to:
|
||||||
|
- ipBlock:
|
||||||
|
cidr: 0.0.0.0/0
|
||||||
|
except:
|
||||||
|
- 10.0.0.0/8
|
||||||
|
- 172.16.0.0/12
|
||||||
|
- 192.168.0.0/16
|
||||||
|
- 169.254.0.0/16
|
||||||
|
- 100.64.0.0/10
|
||||||
|
- ipBlock:
|
||||||
|
cidr: "::/0"
|
||||||
|
except:
|
||||||
|
- "fc00::/7"
|
||||||
|
- "fe80::/10"
|
||||||
|
ports:
|
||||||
|
- protocol: TCP
|
||||||
|
port: 443
|
||||||
@@ -1,15 +1,21 @@
|
|||||||
# OPTIONAL browser entry point: expose the gateway through Traefik.
|
# OPTIONAL browser entry point: expose the gateway through Traefik.
|
||||||
#
|
#
|
||||||
# Requires 50-gateway.yaml (Service dsh-gateway:8080) and a Traefik install
|
# This is the "no SSO" exposure variant and is MUTUALLY EXCLUSIVE with
|
||||||
# with an IngressClass named "traefik" (k3s ships one). Replace the placeholder
|
# ../gateway-api-sso/ — apply exactly one of them (see ../README.md).
|
||||||
# host below with a name that resolves to your ingress address, and either
|
#
|
||||||
# supply a real TLS Secret or delete the `tls:` block for plain HTTP.
|
# Requires the base manifests (Service dsh-gateway:8080 from
|
||||||
|
# manifests/50-gateway.yaml) and a Traefik install with an IngressClass named
|
||||||
|
# "traefik" (k3s ships one). Replace the placeholder host below with a name
|
||||||
|
# that resolves to your ingress address, and either supply a real TLS Secret or
|
||||||
|
# delete the `tls:` block for plain HTTP.
|
||||||
#
|
#
|
||||||
# Two things must match this host:
|
# Two things must match this host:
|
||||||
# 1. SandboxTemplate (10-sandboxtemplate.yaml): set DSH_TRUSTED_HOSTS to
|
# 1. SandboxTemplate (manifests/10-sandboxtemplate.yaml): set
|
||||||
# this host so DSH accepts the forwarded Host/Origin on /api.
|
# DSH_TRUSTED_HOSTS to this host so DSH accepts the forwarded Host/Origin
|
||||||
# 2. Gateway Deployment (50-gateway.yaml): set COOKIE_SECURE to "true"
|
# on /api.
|
||||||
# once this Ingress serves HTTPS, so the session cookie is Secure.
|
# 2. Gateway Deployment (manifests/50-gateway.yaml): set COOKIE_SECURE to
|
||||||
|
# "true" once this Ingress serves HTTPS, so the session cookie is Secure.
|
||||||
|
# The base manifest sets "false" for the plain-HTTP port-forward demo.
|
||||||
#
|
#
|
||||||
# WebSocket support is REQUIRED: the browser talks to /u/<user>/... and DSH
|
# WebSocket support is REQUIRED: the browser talks to /u/<user>/... and DSH
|
||||||
# uses WebSockets for live session traffic. Traefik upgrades WS/SSE
|
# uses WebSockets for live session traffic. Traefik upgrades WS/SSE
|
||||||
@@ -17,6 +23,10 @@
|
|||||||
# middlewares (e.g. compress, buffering) to this router — they break streaming
|
# middlewares (e.g. compress, buffering) to this router — they break streaming
|
||||||
# and upgrades. `path: /` Prefix covers the landing page and every per-user
|
# and upgrades. `path: /` Prefix covers the landing page and every per-user
|
||||||
# proxy path, so no per-sandbox Ingress is needed.
|
# proxy path, so no per-sandbox Ingress is needed.
|
||||||
|
#
|
||||||
|
# 61-networkpolicy.yaml is part of this variant: it restricts the gateway's
|
||||||
|
# ingress to Traefik pods. Adjust its selector if your ingress controller uses
|
||||||
|
# a different label or hostNetwork.
|
||||||
apiVersion: networking.k8s.io/v1
|
apiVersion: networking.k8s.io/v1
|
||||||
kind: Ingress
|
kind: Ingress
|
||||||
metadata:
|
metadata:
|
||||||
@@ -0,0 +1,31 @@
|
|||||||
|
# OPTIONAL with the Ingress exposure variant: restrict the gateway's ingress
|
||||||
|
# to the ingress controller.
|
||||||
|
#
|
||||||
|
# This manifest assumes Traefik, matching this variant's
|
||||||
|
# `ingressClassName: traefik`. Traefik's Helm chart labels its pods
|
||||||
|
# app.kubernetes.io/name=traefik; adjust the selector for another controller or
|
||||||
|
# for a hostNetwork install (where the source is the node IP, not a pod IP).
|
||||||
|
#
|
||||||
|
# NetworkPolicy enforcement of kubelet probe traffic is CNI-dependent. If the
|
||||||
|
# gateway starts failing its /healthz probes, allow the node CIDR on port 8080
|
||||||
|
# or remove this policy. See ../../../docs/optional-hardening.md.
|
||||||
|
apiVersion: networking.k8s.io/v1
|
||||||
|
kind: NetworkPolicy
|
||||||
|
metadata:
|
||||||
|
name: dsh-gateway-ingress
|
||||||
|
namespace: dsh-demo
|
||||||
|
spec:
|
||||||
|
podSelector:
|
||||||
|
matchLabels:
|
||||||
|
app: dsh-gateway
|
||||||
|
policyTypes:
|
||||||
|
- Ingress
|
||||||
|
ingress:
|
||||||
|
- from:
|
||||||
|
- namespaceSelector: {}
|
||||||
|
podSelector:
|
||||||
|
matchLabels:
|
||||||
|
app.kubernetes.io/name: traefik
|
||||||
|
ports:
|
||||||
|
- protocol: TCP
|
||||||
|
port: 8080
|
||||||
@@ -0,0 +1,37 @@
|
|||||||
|
# Optional manifests
|
||||||
|
|
||||||
|
Opt-in manifests that are **not** part of the top-level walkthrough.
|
||||||
|
`kubectl apply -f manifests/` is non-recursive, so it does not apply this
|
||||||
|
directory; apply a subdirectory explicitly.
|
||||||
|
|
||||||
|
## network-policy/
|
||||||
|
|
||||||
|
A Kubernetes `NetworkPolicy` for the DSH sandboxes:
|
||||||
|
|
||||||
|
- ingress only from the gateway pods on TCP `3080` (HTTP + WebSocket);
|
||||||
|
- egress only to DNS and public HTTPS (provider APIs, package mirrors,
|
||||||
|
git-over-HTTPS).
|
||||||
|
|
||||||
|
Apply and delete it explicitly:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl apply -f manifests/optional/network-policy/
|
||||||
|
kubectl delete -f manifests/optional/network-policy/
|
||||||
|
```
|
||||||
|
|
||||||
|
It requires a CNI that enforces `NetworkPolicy`. The selector uses the
|
||||||
|
agent-sandbox controller's pod tracking label, so the policy applies to
|
||||||
|
existing warm-pool and claimed sandboxes without recreation.
|
||||||
|
|
||||||
|
## gVisor, nodeSelector and provider keys
|
||||||
|
|
||||||
|
These are configured in the main manifests or with an extra Secret:
|
||||||
|
|
||||||
|
- **gVisor:** uncomment `runtimeClassName: gvisor` in the SandboxTemplate and
|
||||||
|
the Deployments before applying, or patch an existing install.
|
||||||
|
- **nodeSelector:** uncomment the field and replace the placeholder node label.
|
||||||
|
- **Provider keys:** create the optional `dsh-provider-keys-extra` Secret; the
|
||||||
|
base `dsh-provider-keys` Secret remains the one required key.
|
||||||
|
|
||||||
|
Full prerequisites, exact commands, verification and caveats:
|
||||||
|
[optional hardening](../../docs/optional-hardening.md).
|
||||||
@@ -0,0 +1,80 @@
|
|||||||
|
# OPTIONAL, opt-in tenant network policy for the DSH sandboxes.
|
||||||
|
#
|
||||||
|
# It is NOT part of the top-level walkthrough. Apply and delete it explicitly:
|
||||||
|
# kubectl apply -f manifests/optional/network-policy/
|
||||||
|
# kubectl delete -f manifests/optional/network-policy/
|
||||||
|
#
|
||||||
|
# The selector uses the agent-sandbox controller's pod tracking label, which is
|
||||||
|
# present on every Sandbox pod (including warm-pool spares), so the policy
|
||||||
|
# applies to existing sandboxes without recreating them. The SandboxTemplate
|
||||||
|
# stays networkPolicyManagement: Unmanaged. Do not also switch it to Managed
|
||||||
|
# unless you migrate these rules into spec.networkPolicy; NetworkPolicies are
|
||||||
|
# additive and the two configurations would be confusing.
|
||||||
|
#
|
||||||
|
# Requires a CNI that enforces NetworkPolicy. See docs/optional-hardening.md
|
||||||
|
# for the connectivity matrix, DNS variants, kubelet-probe caveat and rollback.
|
||||||
|
apiVersion: networking.k8s.io/v1
|
||||||
|
kind: NetworkPolicy
|
||||||
|
metadata:
|
||||||
|
name: dsh-sandbox
|
||||||
|
namespace: dsh-demo
|
||||||
|
spec:
|
||||||
|
podSelector:
|
||||||
|
matchExpressions:
|
||||||
|
- key: agents.x-k8s.io/sandbox-name-hash
|
||||||
|
operator: Exists
|
||||||
|
policyTypes:
|
||||||
|
- Ingress
|
||||||
|
- Egress
|
||||||
|
ingress:
|
||||||
|
# The gateway is the only intended client of a sandbox (HTTP + WebSocket).
|
||||||
|
- from:
|
||||||
|
- podSelector:
|
||||||
|
matchLabels:
|
||||||
|
app: dsh-gateway
|
||||||
|
ports:
|
||||||
|
- protocol: TCP
|
||||||
|
port: 3080
|
||||||
|
egress:
|
||||||
|
# Cluster DNS. CoreDNS/kube-dns pods carry k8s-app=kube-dns.
|
||||||
|
- to:
|
||||||
|
- namespaceSelector:
|
||||||
|
matchLabels:
|
||||||
|
kubernetes.io/metadata.name: kube-system
|
||||||
|
podSelector:
|
||||||
|
matchLabels:
|
||||||
|
k8s-app: kube-dns
|
||||||
|
ports:
|
||||||
|
- protocol: UDP
|
||||||
|
port: 53
|
||||||
|
- protocol: TCP
|
||||||
|
port: 53
|
||||||
|
# NodeLocal DNSCache, for clusters that do not send DNS to the kube-dns
|
||||||
|
# Service. 169.254.20.10 is its standard link-local address.
|
||||||
|
- to:
|
||||||
|
- ipBlock:
|
||||||
|
cidr: 169.254.20.10/32
|
||||||
|
ports:
|
||||||
|
- protocol: UDP
|
||||||
|
port: 53
|
||||||
|
- protocol: TCP
|
||||||
|
port: 53
|
||||||
|
# Public HTTPS only: provider APIs, package mirrors, git-over-HTTPS.
|
||||||
|
# Private ranges, CGNAT and the metadata server stay blocked.
|
||||||
|
- to:
|
||||||
|
- ipBlock:
|
||||||
|
cidr: 0.0.0.0/0
|
||||||
|
except:
|
||||||
|
- 10.0.0.0/8
|
||||||
|
- 172.16.0.0/12
|
||||||
|
- 192.168.0.0/16
|
||||||
|
- 169.254.0.0/16
|
||||||
|
- 100.64.0.0/10
|
||||||
|
- ipBlock:
|
||||||
|
cidr: "::/0"
|
||||||
|
except:
|
||||||
|
- "fc00::/7"
|
||||||
|
- "fe80::/10"
|
||||||
|
ports:
|
||||||
|
- protocol: TCP
|
||||||
|
port: 443
|
||||||
@@ -1,13 +1,37 @@
|
|||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
# Smoke test for the dsh-agents-as-a-service example: runs the full README
|
# End-to-end smoke test for the dsh-agents-as-a-service example: runs the
|
||||||
# walkthrough non-interactively against the current kubectl context.
|
# README walkthrough non-interactively against the current kubectl context.
|
||||||
# Expects agent-sandbox (with extensions) already installed.
|
#
|
||||||
|
# Prerequisites:
|
||||||
|
# - A kind cluster whose name matches the current kubectl context, or set
|
||||||
|
# KIND_CLUSTER=<name>.
|
||||||
|
# - agent-sandbox with extensions installed (API v1beta1). The release
|
||||||
|
# manifest is pinned by AGENT_SANDBOX_VERSION in versions.env:
|
||||||
|
# . ../../../versions.env
|
||||||
|
# kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/download/${AGENT_SANDBOX_VERSION}/sandbox-with-extensions.yaml
|
||||||
|
# - docker, to build the gateway image (skipped with SKIP_GATEWAY=1).
|
||||||
|
#
|
||||||
|
# Environment:
|
||||||
|
# SKIP_GATEWAY=1 run only the resource-layer checks (no gateway build/load,
|
||||||
|
# no signup/proxy/WebSocket/idle/wake checks)
|
||||||
|
# KIND_CLUSTER kind cluster name for "kind load docker-image"
|
||||||
|
# (default: derived from the current kubectl context)
|
||||||
|
# GW_LOCAL_IMAGE local gateway image tag
|
||||||
|
# (default: dsh-aaas-gateway:e2e)
|
||||||
|
#
|
||||||
|
# The gateway is built from the working tree (gateway/) and loaded into kind,
|
||||||
|
# so this tests the example as checked out, not the published image.
|
||||||
|
#
|
||||||
|
# The script is destructive only inside the dsh-demo namespace: it creates the
|
||||||
|
# namespace (and a throwaway provider key) and DELETES it on exit via the
|
||||||
|
# cleanup trap.
|
||||||
#
|
#
|
||||||
# NOTE: the only local port bound is the gateway port-forward on 18080.
|
# NOTE: the only local port bound is the gateway port-forward on 18080.
|
||||||
# The DSH Web container port (3080) is in-cluster only and is never bound
|
# The DSH Web container port (3080) is in-cluster only and is never bound
|
||||||
# on the local host.
|
# on the local host.
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
cd "$(dirname "$0")"
|
# Example root: paths below are manifests/... and gateway/... from here.
|
||||||
|
cd "$(dirname "$0")/.."
|
||||||
|
|
||||||
NS=dsh-demo
|
NS=dsh-demo
|
||||||
pass() { echo "PASS: $1"; }
|
pass() { echo "PASS: $1"; }
|
||||||
@@ -20,12 +44,14 @@ cleanup() {
|
|||||||
trap cleanup EXIT
|
trap cleanup EXIT
|
||||||
|
|
||||||
echo "=== 1. install prereqs (throwaway key) + template + warm pool"
|
echo "=== 1. install prereqs (throwaway key) + template + warm pool"
|
||||||
kubectl apply -f 00-prereqs.yaml
|
kubectl apply -f manifests/00-prereqs.yaml
|
||||||
# The provider key is never committed; generate a throwaway value.
|
# The provider key is never committed; generate a throwaway value.
|
||||||
kubectl -n "$NS" delete secret dsh-provider-keys --ignore-not-found >/dev/null
|
kubectl -n "$NS" delete secret dsh-provider-keys --ignore-not-found >/dev/null
|
||||||
kubectl -n "$NS" create secret generic dsh-provider-keys \
|
kubectl -n "$NS" create secret generic dsh-provider-keys \
|
||||||
--from-literal=DEEPSEEK_API_KEY="sk-test-$(openssl rand -hex 16)"
|
--from-literal=DEEPSEEK_API_KEY="sk-test-$(openssl rand -hex 16)"
|
||||||
kubectl apply -f 10-sandboxtemplate.yaml -f 20-sandboxwarmpool.yaml
|
# dsh-provider-keys-extra is intentionally absent: the template's optional
|
||||||
|
# envFrom reference must not block the warm pool.
|
||||||
|
kubectl apply -f manifests/10-sandboxtemplate.yaml -f manifests/20-sandboxwarmpool.yaml
|
||||||
for _ in $(seq 1 120); do
|
for _ in $(seq 1 120); do
|
||||||
READY=$(kubectl -n "$NS" get sandboxes -l agents.x-k8s.io/warm-pool-sandbox \
|
READY=$(kubectl -n "$NS" get sandboxes -l agents.x-k8s.io/warm-pool-sandbox \
|
||||||
-o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' | grep -c True || true)
|
-o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' | grep -c True || true)
|
||||||
@@ -37,7 +63,7 @@ pass "warm pool: 2 spares Ready"
|
|||||||
|
|
||||||
echo "=== 2. signup via claim (warm adoption)"
|
echo "=== 2. signup via claim (warm adoption)"
|
||||||
T0=$(date +%s)
|
T0=$(date +%s)
|
||||||
kubectl apply -f 30-claim-alice.yaml
|
kubectl apply -f manifests/30-claim-alice.yaml
|
||||||
kubectl -n "$NS" wait sandboxclaim dsh-alice --for=condition=Ready --timeout=60s
|
kubectl -n "$NS" wait sandboxclaim dsh-alice --for=condition=Ready --timeout=60s
|
||||||
ELAPSED=$(( $(date +%s) - T0 ))
|
ELAPSED=$(( $(date +%s) - T0 ))
|
||||||
SB=$(kubectl -n "$NS" get sandboxclaim dsh-alice -o jsonpath='{.status.sandbox.name}')
|
SB=$(kubectl -n "$NS" get sandboxclaim dsh-alice -o jsonpath='{.status.sandbox.name}')
|
||||||
@@ -65,7 +91,7 @@ MARKER=$(kubectl -n "$NS" exec "$POD" -- cat /home/node/.dsh/marker.txt)
|
|||||||
pass "resumed: state survived on the reattached PVC"
|
pass "resumed: state survived on the reattached PVC"
|
||||||
|
|
||||||
echo "=== 6. injection policy rejects env-carrying claims"
|
echo "=== 6. injection policy rejects env-carrying claims"
|
||||||
kubectl apply -f 40-claim-rejected.yaml
|
kubectl apply -f manifests/40-claim-rejected.yaml
|
||||||
# Wait for the POSITIVE rejection signal, not merely "not bound yet".
|
# Wait for the POSITIVE rejection signal, not merely "not bound yet".
|
||||||
REASON=""
|
REASON=""
|
||||||
for _ in $(seq 1 12); do
|
for _ in $(seq 1 12); do
|
||||||
@@ -78,19 +104,25 @@ done
|
|||||||
|| fail "claim was not rejected (Ready reason='$REASON')"
|
|| fail "claim was not rejected (Ready reason='$REASON')"
|
||||||
BOUND=$(kubectl -n "$NS" get sandboxclaim dsh-mallory -o jsonpath='{.status.sandbox.name}' 2>/dev/null || true)
|
BOUND=$(kubectl -n "$NS" get sandboxclaim dsh-mallory -o jsonpath='{.status.sandbox.name}' 2>/dev/null || true)
|
||||||
[ -z "$BOUND" ] || fail "env-carrying claim unexpectedly bound sandbox '$BOUND'"
|
[ -z "$BOUND" ] || fail "env-carrying claim unexpectedly bound sandbox '$BOUND'"
|
||||||
kubectl delete -f 40-claim-rejected.yaml >/dev/null
|
kubectl delete -f manifests/40-claim-rejected.yaml >/dev/null
|
||||||
pass "claim rejected with EnvVarsInjectionRejected and never bound"
|
pass "claim rejected with EnvVarsInjectionRejected and never bound"
|
||||||
|
|
||||||
echo "=== 7. gateway: browser signup, proxy, idle-suspend, wake-on-connect"
|
echo "=== 7. gateway: browser signup, proxy, idle-suspend, wake-on-connect"
|
||||||
if command -v docker >/dev/null && command -v kind >/dev/null; then
|
if command -v docker >/dev/null && command -v kind >/dev/null; then
|
||||||
KIND_CLUSTER="${KIND_CLUSTER:-$(kubectl config current-context | sed 's/^kind-//')}"
|
KIND_CLUSTER="${KIND_CLUSTER:-$(kubectl config current-context | sed 's/^kind-//')}"
|
||||||
# Tag exactly what 50-gateway.yaml references so the smoke test exercises
|
GW_LOCAL_IMAGE="${GW_LOCAL_IMAGE:-dsh-aaas-gateway:e2e}"
|
||||||
# the shipped manifest unchanged (imagePullPolicy: IfNotPresent uses the
|
# Test the working tree, not the published image: build the gateway from
|
||||||
# image loaded into kind).
|
# gateway/, load it into kind, then point the Deployment at that tag. The
|
||||||
GW_IMAGE=ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:gateway
|
# shipped manifest pins the release image; imagePullPolicy: IfNotPresent
|
||||||
docker build -q -t "$GW_IMAGE" gateway/ >/dev/null
|
# makes the node use the loaded one.
|
||||||
kind load docker-image "$GW_IMAGE" --name "$KIND_CLUSTER" >/dev/null
|
docker build -q -t "$GW_LOCAL_IMAGE" gateway/ >/dev/null
|
||||||
kubectl apply -f 50-gateway.yaml
|
kind load docker-image "$GW_LOCAL_IMAGE" --name "$KIND_CLUSTER" >/dev/null
|
||||||
|
kubectl apply -f manifests/50-gateway.yaml
|
||||||
|
kubectl -n "$NS" set image deploy/dsh-gateway gateway="$GW_LOCAL_IMAGE"
|
||||||
|
# Override the demo timeouts from manifests/50-gateway.yaml (360/240) with the
|
||||||
|
# gateway's working-tree defaults, so idle-suspend and wake-on-connect are
|
||||||
|
# exercised in seconds instead of minutes.
|
||||||
|
kubectl -n "$NS" set env deploy/dsh-gateway IDLE_TIMEOUT=60 WAKE_TIMEOUT=120
|
||||||
kubectl -n "$NS" rollout status deploy/dsh-gateway --timeout=120s
|
kubectl -n "$NS" rollout status deploy/dsh-gateway --timeout=120s
|
||||||
# Local port-forward uses 18080 (NOT 3080, which stays in-cluster).
|
# Local port-forward uses 18080 (NOT 3080, which stays in-cluster).
|
||||||
kubectl -n "$NS" port-forward svc/dsh-gateway 18080:8080 >/dev/null 2>&1 &
|
kubectl -n "$NS" port-forward svc/dsh-gateway 18080:8080 >/dev/null 2>&1 &
|
||||||
152
examples/minimal-single-tenant/README.md
Normal file
152
examples/minimal-single-tenant/README.md
Normal file
@@ -0,0 +1,152 @@
|
|||||||
|
# Minimal single-tenant DSH Web
|
||||||
|
|
||||||
|
The smallest Kubernetes deployment of DeepSeek Harness (DSH) Web: one
|
||||||
|
`SandboxTemplate`, a one-replica warm pool and one claim. It is the "just one
|
||||||
|
DSH Web instance" quickstart.
|
||||||
|
|
||||||
|
For the full platform — warm pool sizing, per-user claims, a browser-facing
|
||||||
|
gateway with cookie sessions, idle-suspend and wake-on-connect — see
|
||||||
|
[`examples/dsh-agents-as-a-service/`](../dsh-agents-as-a-service/README.md).
|
||||||
|
This example deliberately has no gateway: you reach DSH through a
|
||||||
|
`kubectl port-forward` and its launch-token URL.
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
|
||||||
|
- A Kubernetes cluster with a default StorageClass (kind and GKE both work).
|
||||||
|
- The agent-sandbox controller with extensions (API `v1beta1`) installed.
|
||||||
|
The version is pinned as `AGENT_SANDBOX_VERSION` in
|
||||||
|
[`versions.env`](../../versions.env):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# from the repository root:
|
||||||
|
. versions.env
|
||||||
|
kubectl apply -f "https://github.com/kubernetes-sigs/agent-sandbox/releases/download/${AGENT_SANDBOX_VERSION}/sandbox-with-extensions.yaml"
|
||||||
|
kubectl -n agent-sandbox-system rollout status deployment --timeout=180s
|
||||||
|
```
|
||||||
|
|
||||||
|
- A DeepSeek API key for real model calls. The pod starts without one; only
|
||||||
|
model requests fail. Optionally create `dsh-provider-keys-extra` for
|
||||||
|
OpenRouter, Gemini, NVIDIA or other provider keys (the template references it
|
||||||
|
with `optional: true`).
|
||||||
|
|
||||||
|
## Quickstart
|
||||||
|
|
||||||
|
Run these from this example's directory (`examples/minimal-single-tenant/`).
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. Namespace + shared provider keys. They are injected into the sandbox via
|
||||||
|
# envFrom and are never committed.
|
||||||
|
kubectl create namespace dsh-demo
|
||||||
|
kubectl -n dsh-demo create secret generic dsh-provider-keys \
|
||||||
|
--from-literal=DEEPSEEK_API_KEY="sk-..."
|
||||||
|
|
||||||
|
# Optional: extra providers, injected through a second Secret. Create it before
|
||||||
|
# the warm pool so the pods are stamped with the keys.
|
||||||
|
kubectl -n dsh-demo create secret generic dsh-provider-keys-extra \
|
||||||
|
--from-literal=OPENROUTER_API_KEY="sk-or-..." \
|
||||||
|
--from-literal=GEMINI_API_KEY="..." \
|
||||||
|
--from-literal=NVIDIA_API_KEY="nvapi-..."
|
||||||
|
|
||||||
|
# 2. The template itself (image pinned in the manifest).
|
||||||
|
kubectl apply -f manifests/sandbox-template.yaml
|
||||||
|
|
||||||
|
# 3. A template only describes pods; a warm pool turns it into running
|
||||||
|
# Sandboxes and a claim adopts one. One replica is enough here.
|
||||||
|
kubectl apply -f - <<'EOF'
|
||||||
|
apiVersion: extensions.agents.x-k8s.io/v1beta1
|
||||||
|
kind: SandboxWarmPool
|
||||||
|
metadata:
|
||||||
|
name: dsh-pool
|
||||||
|
namespace: dsh-demo
|
||||||
|
spec:
|
||||||
|
replicas: 1
|
||||||
|
sandboxTemplateRef:
|
||||||
|
name: dsh-web-template
|
||||||
|
---
|
||||||
|
apiVersion: extensions.agents.x-k8s.io/v1beta1
|
||||||
|
kind: SandboxClaim
|
||||||
|
metadata:
|
||||||
|
name: dsh
|
||||||
|
namespace: dsh-demo
|
||||||
|
spec:
|
||||||
|
warmPoolRef:
|
||||||
|
name: dsh-pool
|
||||||
|
EOF
|
||||||
|
|
||||||
|
kubectl -n dsh-demo wait sandboxclaim/dsh --for=condition=Ready --timeout=180s
|
||||||
|
```
|
||||||
|
|
||||||
|
## Talk to the instance
|
||||||
|
|
||||||
|
DSH gates `/` behind a launch token it prints only in its startup logs. Open
|
||||||
|
the printed URL once; DSH then sets its own auth cookie and the UI loads
|
||||||
|
normally.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
SB=$(kubectl -n dsh-demo get sandboxclaim dsh -o jsonpath='{.status.sandbox.name}')
|
||||||
|
kubectl -n dsh-demo logs "$SB" | grep -m1 '?token='
|
||||||
|
# dsh web: http://<pod-ip>:3080/?token=...
|
||||||
|
|
||||||
|
kubectl -n dsh-demo port-forward "pod/$SB" 18081:3080
|
||||||
|
# open http://localhost:18081/?token=... from the log line
|
||||||
|
```
|
||||||
|
|
||||||
|
> **Port note:** DSH Web listens on `3080` in-cluster. If `3080` is already in
|
||||||
|
> use on your workstation, forward to another local port (here `18081`), never
|
||||||
|
> `3080`. `localhost` is always in DSH's trusted-host list, so the forwarded
|
||||||
|
> request is accepted.
|
||||||
|
>
|
||||||
|
> **gVisor caveat:** if you uncomment `runtimeClassName: gvisor` in the
|
||||||
|
> template, `kubectl port-forward` no longer works. Use the multi-tenant gateway
|
||||||
|
> example or a reachable Service instead; see
|
||||||
|
> [optional hardening](../dsh-agents-as-a-service/docs/optional-hardening.md).
|
||||||
|
|
||||||
|
## Network policy and optional hardening
|
||||||
|
|
||||||
|
The template omits `networkPolicyManagement`, so the agent-sandbox controller
|
||||||
|
uses its Managed secure default: ingress only from the sandbox-router, public
|
||||||
|
egress, and no internal cluster DNS. `kubectl port-forward` is local to the pod
|
||||||
|
and still works (except under gVisor).
|
||||||
|
|
||||||
|
gVisor, an optional `nodeSelector`, extra provider keys and a stricter sandbox
|
||||||
|
NetworkPolicy are documented in
|
||||||
|
[optional hardening](../dsh-agents-as-a-service/docs/optional-hardening.md).
|
||||||
|
|
||||||
|
## Probes and trusted hosts
|
||||||
|
|
||||||
|
Two settings in `manifests/sandbox-template.yaml` are easy to get wrong:
|
||||||
|
|
||||||
|
- **Probe path.** `/` returns `401` without the launch-token cookie, so a probe
|
||||||
|
there crash-loops the pod. The template probes `/favicon.svg` (served
|
||||||
|
unauthenticated) and gates liveness/readiness behind a `startupProbe`, which
|
||||||
|
gives a cold start (plugin activation, or the first resume) up to 120s before
|
||||||
|
a restart.
|
||||||
|
- **Host allow-list.** DSH rejects unknown `Host` headers. `localhost` and
|
||||||
|
`127.0.0.1` are always trusted; if you later front the sandbox with an
|
||||||
|
Ingress or proxy, set `DSH_TRUSTED_HOSTS` to that public host (comma or space
|
||||||
|
separated). The full example explains both exposure variants in
|
||||||
|
[`manifests/exposure/README.md`](../dsh-agents-as-a-service/manifests/exposure/README.md).
|
||||||
|
|
||||||
|
The template also sets `fsGroupChangePolicy: OnRootMismatch`. DSH's
|
||||||
|
credentials plugin requires `.credentials.yaml` to be `0600`, while the
|
||||||
|
kubelet's default fsGroup policy rewrites it to `0660` on every pod start —
|
||||||
|
without this policy a recreated or resumed sandbox crash-loops. The rationale
|
||||||
|
is spelled out in
|
||||||
|
[`10-sandboxtemplate.yaml`](../dsh-agents-as-a-service/manifests/10-sandboxtemplate.yaml).
|
||||||
|
|
||||||
|
## Suspend, resume and clean up
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Suspend: the pod is deleted, the PVC (state) and Service are kept.
|
||||||
|
kubectl -n dsh-demo patch sandbox "$SB" --type merge -p '{"spec":{"operatingMode":"Suspended"}}'
|
||||||
|
|
||||||
|
# Resume: a new pod reattaches the same PVC; conversations survive.
|
||||||
|
kubectl -n dsh-demo patch sandbox "$SB" --type merge -p '{"spec":{"operatingMode":"Running"}}'
|
||||||
|
kubectl -n dsh-demo wait sandbox "$SB" --for=condition=Ready --timeout=180s
|
||||||
|
|
||||||
|
# Delete everything this quickstart created.
|
||||||
|
kubectl -n dsh-demo delete sandboxclaim dsh
|
||||||
|
kubectl delete namespace dsh-demo
|
||||||
|
```
|
||||||
|
|
||||||
|
Deleting the claim cascades to the sandbox and its PVC.
|
||||||
@@ -4,11 +4,15 @@
|
|||||||
# multi-tenant platform (warm pool, per-user claims, gateway, idle-suspend),
|
# multi-tenant platform (warm pool, per-user claims, gateway, idle-suspend),
|
||||||
# see examples/dsh-agents-as-a-service/.
|
# see examples/dsh-agents-as-a-service/.
|
||||||
#
|
#
|
||||||
# Prerequisites (create these first, or apply 00-prereqs.yaml from the
|
# Claims reference this template directly (spec.sandboxTemplateRef): there is
|
||||||
# examples directory):
|
# no warm pool and no per-user platform here. See the README for the full
|
||||||
|
# quickstart; in short the prerequisites are:
|
||||||
# - Namespace dsh-demo.
|
# - Namespace dsh-demo.
|
||||||
# - Secret dsh-provider-keys in dsh-demo with key DEEPSEEK_API_KEY
|
# - Secret dsh-provider-keys in dsh-demo with key DEEPSEEK_API_KEY
|
||||||
# (injected via envFrom; the pod starts without it, only model calls fail).
|
# (injected via envFrom; the pod starts without it, only model calls fail).
|
||||||
|
# - Optional: Secret dsh-provider-keys-extra with additional provider keys
|
||||||
|
# (OPENROUTER_API_KEY, GEMINI_API_KEY, NVIDIA_API_KEY, ...). It is
|
||||||
|
# referenced with `optional: true`, so it may be absent.
|
||||||
# The agent-sandbox controller (v1.0.7, API v1beta1) must be installed.
|
# The agent-sandbox controller (v1.0.7, API v1beta1) must be installed.
|
||||||
apiVersion: extensions.agents.x-k8s.io/v1beta1
|
apiVersion: extensions.agents.x-k8s.io/v1beta1
|
||||||
kind: SandboxTemplate
|
kind: SandboxTemplate
|
||||||
@@ -17,8 +21,22 @@ metadata:
|
|||||||
namespace: dsh-demo
|
namespace: dsh-demo
|
||||||
spec:
|
spec:
|
||||||
service: true # headless Service per Sandbox -> stable DNS (status.serviceFQDN)
|
service: true # headless Service per Sandbox -> stable DNS (status.serviceFQDN)
|
||||||
|
# networkPolicyManagement is omitted, so the controller uses its Managed
|
||||||
|
# secure default (ingress only from the sandbox-router, public egress). This
|
||||||
|
# quickstart reaches the sandbox through `kubectl port-forward`, which is
|
||||||
|
# local to the pod. The multi-tenant example documents opt-in policies in
|
||||||
|
# examples/dsh-agents-as-a-service/docs/optional-hardening.md.
|
||||||
podTemplate:
|
podTemplate:
|
||||||
spec:
|
spec:
|
||||||
|
# OPTIONAL: run under gVisor. gVisor breaks `kubectl port-forward`, so
|
||||||
|
# this quickstart's only access path will not work; use the multi-tenant
|
||||||
|
# gateway example or a Service/NodePort instead.
|
||||||
|
# runtimeClassName: gvisor
|
||||||
|
#
|
||||||
|
# OPTIONAL: pin to a dedicated node pool. Replace the placeholder with a
|
||||||
|
# node label; add a matching toleration if the pool is tainted.
|
||||||
|
# nodeSelector:
|
||||||
|
# dsh.example.com/node-pool: dsh
|
||||||
securityContext:
|
securityContext:
|
||||||
runAsNonRoot: true
|
runAsNonRoot: true
|
||||||
runAsUser: 1000 # Matches the UID of the 'node' user
|
runAsUser: 1000 # Matches the UID of the 'node' user
|
||||||
@@ -26,16 +44,15 @@ spec:
|
|||||||
fsGroup: 1000
|
fsGroup: 1000
|
||||||
# DSH's credentials plugin refuses to start when .credentials.yaml has
|
# DSH's credentials plugin refuses to start when .credentials.yaml has
|
||||||
# any group/other permission bit (it creates the file 0600). The
|
# any group/other permission bit (it creates the file 0600). The
|
||||||
# default kubelet fsGroup policy ("Always") recursively ORs 0660 into
|
# default kubelet fsGroup policy ("Always") rewrites it to 0660 on every
|
||||||
# every file each time the pod starts, so a recreated/resumed sandbox
|
# pod start, so a recreated or resumed sandbox crash-loops; OnRootMismatch
|
||||||
# comes back with 0660 and crash-loops. OnRootMismatch skips that walk
|
# avoids that walk once the volume root already carries the ownership.
|
||||||
# once the volume root already has the fsGroup ownership/mode from the
|
# Rationale in full at
|
||||||
# first mount. Keep in sync with
|
# examples/dsh-agents-as-a-service/manifests/10-sandboxtemplate.yaml.
|
||||||
# examples/dsh-agents-as-a-service/10-sandboxtemplate.yaml.
|
|
||||||
fsGroupChangePolicy: OnRootMismatch
|
fsGroupChangePolicy: OnRootMismatch
|
||||||
containers:
|
containers:
|
||||||
- name: dsh-web
|
- name: dsh-web
|
||||||
image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.2
|
image: ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox:dsh-v0.2.1-rc.3-r4
|
||||||
imagePullPolicy: IfNotPresent
|
imagePullPolicy: IfNotPresent
|
||||||
ports:
|
ports:
|
||||||
- name: web
|
- name: web
|
||||||
@@ -43,6 +60,10 @@ spec:
|
|||||||
envFrom:
|
envFrom:
|
||||||
- secretRef:
|
- secretRef:
|
||||||
name: dsh-provider-keys # Secret with key DEEPSEEK_API_KEY
|
name: dsh-provider-keys # Secret with key DEEPSEEK_API_KEY
|
||||||
|
# Optional extra provider keys; an absent Secret is safe.
|
||||||
|
- secretRef:
|
||||||
|
name: dsh-provider-keys-extra
|
||||||
|
optional: true
|
||||||
env:
|
env:
|
||||||
# Empty => DSH derives links from the request Host. Keep it empty
|
# Empty => DSH derives links from the request Host. Keep it empty
|
||||||
# when fronting the sandbox with a proxy/gateway.
|
# when fronting the sandbox with a proxy/gateway.
|
||||||
@@ -50,8 +71,10 @@ spec:
|
|||||||
value: ""
|
value: ""
|
||||||
# DSH Host/Origin-fences /api. Empty trusts only localhost, which
|
# DSH Host/Origin-fences /api. Empty trusts only localhost, which
|
||||||
# is right for `kubectl port-forward`. Set this to the public host
|
# is right for `kubectl port-forward`. Set this to the public host
|
||||||
# (e.g. dsh.example.com) if you expose DSH behind an Ingress, so
|
# (e.g. dsh.example.com) if you front DSH with a proxy or Ingress,
|
||||||
# requests forwarded with the browser's Host are accepted.
|
# so requests forwarded with the browser's Host are accepted. The
|
||||||
|
# full example documents both exposure variants in
|
||||||
|
# examples/dsh-agents-as-a-service/manifests/exposure/README.md.
|
||||||
- name: DSH_TRUSTED_HOSTS
|
- name: DSH_TRUSTED_HOSTS
|
||||||
value: ""
|
value: ""
|
||||||
volumeMounts:
|
volumeMounts:
|
||||||
60
scripts/check-examples.sh
Executable file
60
scripts/check-examples.sh
Executable file
@@ -0,0 +1,60 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Enforce the example contract documented in docs/examples.md.
|
||||||
|
# Run by "make check" locally and by the CI lint job.
|
||||||
|
set -euo pipefail
|
||||||
|
cd "$(dirname "$0")/.."
|
||||||
|
|
||||||
|
index=examples/README.md
|
||||||
|
fail=0
|
||||||
|
err() { echo "check-examples: $*" >&2; fail=1; }
|
||||||
|
|
||||||
|
[ -f "$index" ] || err "missing $index"
|
||||||
|
|
||||||
|
for dir in examples/*/; do
|
||||||
|
name="$(basename "$dir")"
|
||||||
|
[ "$name" = "_template" ] && continue
|
||||||
|
|
||||||
|
[ -f "${dir}README.md" ] || err "$name: missing README.md"
|
||||||
|
[ -d "${dir}manifests" ] || err "$name: missing manifests/"
|
||||||
|
|
||||||
|
if [ -d "${dir}manifests" ]; then
|
||||||
|
count=$(find "${dir}manifests" -maxdepth 1 -type f -name '*.yaml' | wc -l | tr -d ' ')
|
||||||
|
[ "$count" -gt 0 ] || err "$name: manifests/ has no top-level *.yaml files"
|
||||||
|
fi
|
||||||
|
|
||||||
|
stray=$(find "$dir" -maxdepth 1 -type f -name '*.yaml' | wc -l | tr -d ' ')
|
||||||
|
[ "$stray" -eq 0 ] || err "$name: manifests belong under manifests/, not at the example root"
|
||||||
|
|
||||||
|
if [ -d "${dir}manifests/exposure" ] && [ ! -f "${dir}manifests/exposure/README.md" ]; then
|
||||||
|
err "$name: manifests/exposure/ requires a README.md describing the variants"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ -d "${dir}manifests/optional" ] && [ ! -f "${dir}manifests/optional/README.md" ]; then
|
||||||
|
err "$name: manifests/optional/ requires a README.md describing the opt-in manifests"
|
||||||
|
fi
|
||||||
|
|
||||||
|
for component in "$dir"*/; do
|
||||||
|
[ -d "$component" ] || continue
|
||||||
|
case "$(basename "$component")" in
|
||||||
|
manifests|scripts|docs) continue ;;
|
||||||
|
esac
|
||||||
|
if [ -f "${component}Dockerfile" ] && [ ! -f "${component}README.md" ]; then
|
||||||
|
err "$name/$(basename "$component"): component with a Dockerfile needs a README.md"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
if ! grep -Fq "examples/$name/" "$index"; then
|
||||||
|
err "$name: not listed in examples/README.md"
|
||||||
|
fi
|
||||||
|
|
||||||
|
for script in "$dir"scripts/*.sh; do
|
||||||
|
[ -e "$script" ] || continue
|
||||||
|
[ -x "$script" ] || err "$script: must be executable"
|
||||||
|
done
|
||||||
|
done
|
||||||
|
|
||||||
|
if [ "$fail" -ne 0 ]; then
|
||||||
|
echo "check-examples: FAILED" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo "check-examples: ok"
|
||||||
47
scripts/check-manifests.py
Normal file
47
scripts/check-manifests.py
Normal file
@@ -0,0 +1,47 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Parse every example manifest and assert the minimal Kubernetes shape."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
try:
|
||||||
|
import yaml
|
||||||
|
except ModuleNotFoundError: # pragma: no cover - environment guard
|
||||||
|
sys.exit("check-manifests: PyYAML is required (python -m pip install pyyaml)")
|
||||||
|
|
||||||
|
ROOT = Path(__file__).resolve().parent.parent
|
||||||
|
FILES = sorted(ROOT.glob("examples/*/manifests/**/*.yaml"))
|
||||||
|
|
||||||
|
if not FILES:
|
||||||
|
sys.exit("check-manifests: no manifests found under examples/*/manifests/")
|
||||||
|
|
||||||
|
errors: list[str] = []
|
||||||
|
documents = 0
|
||||||
|
for path in FILES:
|
||||||
|
relative = path.relative_to(ROOT)
|
||||||
|
try:
|
||||||
|
loaded = list(yaml.safe_load_all(path.read_text(encoding="utf-8")))
|
||||||
|
except yaml.YAMLError as exc:
|
||||||
|
errors.append(f"{relative}: invalid YAML: {exc}")
|
||||||
|
continue
|
||||||
|
for position, document in enumerate(loaded, start=1):
|
||||||
|
if document is None:
|
||||||
|
continue
|
||||||
|
documents += 1
|
||||||
|
if not isinstance(document, dict):
|
||||||
|
errors.append(f"{relative} document {position}: not a mapping")
|
||||||
|
continue
|
||||||
|
missing = [key for key in ("apiVersion", "kind", "metadata") if key not in document]
|
||||||
|
if missing:
|
||||||
|
errors.append(f"{relative} document {position}: missing {', '.join(missing)}")
|
||||||
|
continue
|
||||||
|
metadata = document["metadata"]
|
||||||
|
if not isinstance(metadata, dict) or not metadata.get("name"):
|
||||||
|
errors.append(f"{relative} document {position}: metadata.name is required")
|
||||||
|
|
||||||
|
if errors:
|
||||||
|
print("\n".join(errors), file=sys.stderr)
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
print(f"check-manifests: ok ({len(FILES)} files, {documents} documents)")
|
||||||
70
scripts/check-versions.sh
Executable file
70
scripts/check-versions.sh
Executable file
@@ -0,0 +1,70 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Verify that every version pin in the repository agrees with versions.env.
|
||||||
|
# Run by "make check" locally and by the CI lint job.
|
||||||
|
set -euo pipefail
|
||||||
|
cd "$(dirname "$0")/.."
|
||||||
|
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
. ./versions.env
|
||||||
|
|
||||||
|
IDENTITY="$DSH_VERSION"
|
||||||
|
if [ -n "${RELEASE_REVISION:-}" ]; then
|
||||||
|
IDENTITY="$DSH_VERSION-$RELEASE_REVISION"
|
||||||
|
fi
|
||||||
|
|
||||||
|
fail=0
|
||||||
|
err() { echo "check-versions: $*" >&2; fail=1; }
|
||||||
|
expect() { # <file> <literal>
|
||||||
|
if ! grep -Fq -- "$2" "$1"; then
|
||||||
|
err "$1: expected to contain: $2"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# Main Dockerfile build args
|
||||||
|
expect Dockerfile "ARG DSH_VERSION=$DSH_VERSION"
|
||||||
|
expect Dockerfile "ARG PNPM_VERSION=$PNPM_VERSION"
|
||||||
|
expect Dockerfile "ARG RELEASE_REVISION=$RELEASE_REVISION"
|
||||||
|
|
||||||
|
# Local configuration mirrored for humans
|
||||||
|
expect .env.sample "DSH_VERSION=$DSH_VERSION"
|
||||||
|
expect .env.sample "PNPM_VERSION=$PNPM_VERSION"
|
||||||
|
expect .env.sample "RELEASE_REVISION=$RELEASE_REVISION"
|
||||||
|
|
||||||
|
# Image references in the example manifests
|
||||||
|
for f in \
|
||||||
|
examples/dsh-agents-as-a-service/manifests/10-sandboxtemplate.yaml \
|
||||||
|
examples/minimal-single-tenant/manifests/sandbox-template.yaml; do
|
||||||
|
expect "$f" "image: $GHCR_IMAGE:$IDENTITY"
|
||||||
|
done
|
||||||
|
expect examples/dsh-agents-as-a-service/manifests/50-gateway.yaml "image: $GATEWAY_IMAGE:$IDENTITY"
|
||||||
|
|
||||||
|
# The gateway image must live below the repository image (one package per
|
||||||
|
# example component, no unrelated top-level packages).
|
||||||
|
case "$GATEWAY_IMAGE" in
|
||||||
|
"$GHCR_IMAGE"/*) ;;
|
||||||
|
*) err "GATEWAY_IMAGE ($GATEWAY_IMAGE) must be nested below GHCR_IMAGE ($GHCR_IMAGE)" ;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
# Build tooling must read versions.env instead of hard-coding a version.
|
||||||
|
# (scripts/ is excluded: set-version.sh documents usage examples in comments.)
|
||||||
|
while IFS= read -r hit; do
|
||||||
|
[ -n "$hit" ] || continue
|
||||||
|
err "hard-coded version outside versions.env: $hit"
|
||||||
|
done < <(grep -rnE 'dsh-v[0-9]' .github/workflows Makefile 2>/dev/null || true)
|
||||||
|
|
||||||
|
# Prose may mention versions, but only the current identity or its base tag.
|
||||||
|
while IFS= read -r token; do
|
||||||
|
[ -n "$token" ] || continue
|
||||||
|
case "$token" in
|
||||||
|
"$IDENTITY"|"$DSH_VERSION") ;;
|
||||||
|
*) err "stale version reference in docs: $token" ;;
|
||||||
|
esac
|
||||||
|
done < <(grep -rhoE 'dsh-v[0-9][A-Za-z0-9._-]*' \
|
||||||
|
--include='*.md' --exclude-dir=.git --exclude-dir=.venv --exclude-dir=.uvcache \
|
||||||
|
--exclude=CHANGELOG.md . | sort -u)
|
||||||
|
|
||||||
|
if [ "$fail" -ne 0 ]; then
|
||||||
|
echo "check-versions: FAILED" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo "check-versions: ok ($IDENTITY)"
|
||||||
64
scripts/set-version.sh
Executable file
64
scripts/set-version.sh
Executable file
@@ -0,0 +1,64 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Rewrite every version pin in one pass, then leave the tree ready for
|
||||||
|
# "make check".
|
||||||
|
#
|
||||||
|
# ./scripts/set-version.sh dsh-v0.2.2-rc.1 # new DSH ref, no revision
|
||||||
|
# ./scripts/set-version.sh dsh-v0.2.1-rc.3 r3 # same ref, new repo revision
|
||||||
|
set -euo pipefail
|
||||||
|
cd "$(dirname "$0")/.."
|
||||||
|
|
||||||
|
new_version="${1:-}"
|
||||||
|
new_revision="${2:-}"
|
||||||
|
|
||||||
|
case "$new_version" in
|
||||||
|
dsh-v[0-9]*) ;;
|
||||||
|
*) echo "usage: $0 <dsh-vX.Y.Z> [rN]" >&2; exit 2 ;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
if [ -n "$new_revision" ]; then
|
||||||
|
case "$new_revision" in
|
||||||
|
r[2-9]|r[1-9][0-9]*) ;;
|
||||||
|
*) echo "revision must look like r2, r3, ..." >&2; exit 2 ;;
|
||||||
|
esac
|
||||||
|
fi
|
||||||
|
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
old_version="$(. ./versions.env; printf '%s' "$DSH_VERSION")"
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
old_revision="$(. ./versions.env; printf '%s' "${RELEASE_REVISION:-}")"
|
||||||
|
|
||||||
|
old_identity="$old_version"
|
||||||
|
[ -z "$old_revision" ] || old_identity="$old_version-$old_revision"
|
||||||
|
new_identity="$new_version"
|
||||||
|
[ -z "$new_revision" ] || new_identity="$new_version-$new_revision"
|
||||||
|
|
||||||
|
if [ "$old_identity" = "$new_identity" ]; then
|
||||||
|
echo "set-version: already at $new_identity"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Replace the identity first (it contains the base version), then the base
|
||||||
|
# version itself. CHANGELOG.md is generated from tags, so it is left alone.
|
||||||
|
while IFS= read -r file; do
|
||||||
|
[ -f "$file" ] || continue
|
||||||
|
case "$file" in
|
||||||
|
.env.sample|Dockerfile|Makefile|*.md|*.yaml|*.yml|*.sh|*.py|*.toml|*.env) ;;
|
||||||
|
*) continue ;;
|
||||||
|
esac
|
||||||
|
sed -i "s/$old_identity/$new_identity/g; s/$old_version/$new_version/g" "$file"
|
||||||
|
done < <(git ls-files)
|
||||||
|
|
||||||
|
# versions.env is authoritative: write both keys explicitly so an empty
|
||||||
|
# revision is stored as an empty value.
|
||||||
|
sed -i "s/^DSH_VERSION=.*/DSH_VERSION=$new_version/" versions.env
|
||||||
|
sed -i "s/^RELEASE_REVISION=.*/RELEASE_REVISION=$new_revision/" versions.env
|
||||||
|
|
||||||
|
# Values stored separately from the full identity: Dockerfile ARG defaults and
|
||||||
|
# the human-facing sample cannot be reached by the identity replacement above.
|
||||||
|
sed -i "s/^ARG DSH_VERSION=.*/ARG DSH_VERSION=$new_version/" Dockerfile
|
||||||
|
sed -i "s/^ARG RELEASE_REVISION=.*/ARG RELEASE_REVISION=$new_revision/" Dockerfile
|
||||||
|
sed -i "s/^DSH_VERSION=.*/DSH_VERSION=$new_version/" .env.sample
|
||||||
|
sed -i "s/^RELEASE_REVISION=.*/RELEASE_REVISION=$new_revision/" .env.sample
|
||||||
|
|
||||||
|
echo "set-version: $old_identity -> $new_identity"
|
||||||
|
echo "set-version: run 'make check' and review the diff"
|
||||||
26
versions.env
Normal file
26
versions.env
Normal file
@@ -0,0 +1,26 @@
|
|||||||
|
# Single source of truth for every pinned version and image name in this
|
||||||
|
# repository. Consumed by the Makefile, build.sh, scripts/*.sh and the GitHub
|
||||||
|
# Actions workflows. Never hard-code these values elsewhere: run "make check"
|
||||||
|
# (scripts/check-versions.sh) to verify every pin.
|
||||||
|
|
||||||
|
# Upstream DeepSeek Harness source, and the ref this repository packages.
|
||||||
|
DSH_REPO=https://github.com/deepseek-ai/deepseek-harness.git
|
||||||
|
DSH_VERSION=dsh-v0.2.1-rc.3
|
||||||
|
PNPM_VERSION=11.28.5
|
||||||
|
|
||||||
|
# Release identity = DSH_VERSION plus an optional repository revision.
|
||||||
|
# An empty RELEASE_REVISION is the first release at this DSH ref; rN (N >= 2)
|
||||||
|
# marks a repository-only re-release at the same DSH ref. The dsh-v0.2.1-rc.3
|
||||||
|
# image tag is already published on GHCR and tags are immutable, so the current
|
||||||
|
# scheme started at -r2; the current identity is dsh-v0.2.1-rc.3-r4.
|
||||||
|
RELEASE_REVISION=r4
|
||||||
|
|
||||||
|
# Control-plane versions used by the examples and their manifests.
|
||||||
|
AGENT_SANDBOX_VERSION=v1.0.7
|
||||||
|
GATEWAY_API_VERSION=v1.2.1
|
||||||
|
OAUTH2_PROXY_IMAGE=quay.io/oauth2-proxy/oauth2-proxy:v7.15.5
|
||||||
|
|
||||||
|
# Registry layout: the main image is the repository itself, and every example
|
||||||
|
# component is a nested package below it (<registry>/<owner>/<repo>/<example>/<component>).
|
||||||
|
GHCR_IMAGE=ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox
|
||||||
|
GATEWAY_IMAGE=ghcr.io/sredevopsorg/deepseek-kubernetes-sandbox/dsh-agents-as-a-service/gateway
|
||||||
Reference in New Issue
Block a user