# crafting-table Polyglot dev/build/audit container — the build farm for the Sulkta ecosystem. ## What this is A single Docker container with every toolchain we work with, fronted by a FastAPI HTTP API + async job runner. Used as a reliable place to compile / test / audit any Sulkta repo regardless of where the caller is — agents, Claude sessions, ad-hoc curl, scheduled cron. Eventual surface (v0.1 full): HTTP API + MCP server + project registry + job runner + structured findings + email digest + autonomous patch loop through clawdforge. Spec: `Sulkta-Coop/openclaw-workspace/memory/spec-crafting-table.md` (LAN-only). ## Status — v0.1 complete (10 of 10) - [x] Step 1: Dockerfile + per-language smoke - [x] Step 2: SQLite ledger + project registry - [x] Step 3: HTTP API skeleton (FastAPI, port 8810) - [x] Step 4: Job runner core (asyncio worker pool, git worktree, subprocess) - [x] Step 5: Per-language parsers (Rust / Python / Go / TS first) - [x] Step 6: Findings extraction + storage - [x] Step 7: MCP server (stdio JSON-RPC, 8 tools) — see [mcp/README.md](mcp/README.md) - [x] Step 8: Email digest scheduler - [x] Step 9: Autonomous patch loop (clawdforge integration → unified diff → worktree apply → verify recipe → push branch → Gitea PR) - [x] Step 10: Production recipes — clawdforge, cauldron, tradecraft (see [examples/recipes/](examples/recipes/)) ## Toolchains in v0.1 | Lang | Versions / extras | |----------|--------------------------------------------------------------------| | Python | 3.11 (Debian default) + uv, pipx, pip-audit, ruff, mypy, pytest, semgrep | | Node | 22.11.0 LTS + npm, pnpm, tsx, eslint, typescript | | Bun | latest (rolling) | | Go | 1.22.10 + govulncheck, staticcheck | | Rust | stable (rustup) + clippy, rustfmt, cargo-audit, cargo-deny | | Ruby | 3.1 (Debian default) + bundler, bundler-audit, rubocop | | PHP | 8.2 (Debian default) + composer, phpstan, phpunit | | JDK | 17 (default) + 21 (Temurin, alongside via `JAVA_HOME_21`) | | Maven | 3.x (Debian) | | Gradle | 8.10 | | .NET | 8.0 SDK | | Swift | 5.9.2 (Ubuntu 22.04 tarball — works on Debian bookworm) | | Kotlin | 1.9.25 (compiler) | | C/C++ | clang + lld + cmake + ninja + valgrind | | Bash | bash + shellcheck + bats + shfmt | | Generic | git, jq, yq, ripgrep, fd, gh-cli, curl, wget | ## HTTP API surface LAN-only. Every request needs `Authorization: Bearer `. The default LAN allowlist is `10/8`, `172.16/12`, `192.168/16`, `127/8`, `::1/128`; override via `CRAFTING_LAN_CIDRS`. | Method | Path | Who | What | |--------|-----------------------------------|--------|--------------------------------| | GET | `/healthz` | LAN | Liveness + runner stats | | POST | `/admin/tokens` | admin | Mint a new bearer | | GET | `/admin/tokens` | admin | List tokens | | DELETE | `/admin/tokens/{name}` | admin | Revoke a token | | POST | `/projects` | any | Register (becomes owner) | | GET | `/projects` | any | List own (or all if admin) | | GET | `/projects/{name}` | owner | Detail (404 if not visible) | | PUT | `/projects/{name}` | owner | Update | | DELETE | `/projects/{name}` | owner | Remove (cascades jobs+findings)| | POST | `/projects/{name}/jobs` | owner | Enqueue a recipe run | | GET | `/jobs?project=&status=&limit=` | any | List own (or all if admin) | | GET | `/jobs/{id}` | owner | State + last 200 log lines | | GET | `/jobs/{id}/log` | owner | Full log (file stream) | | GET | `/jobs/{id}/findings` | owner | Structured findings (see Findings) | | POST | `/jobs/{id}/patches` | owner | Trigger an auto-patch attempt (wave 3) | | GET | `/patches?project=&status=&limit=`| any | List own patch attempts | | GET | `/patches/{id}` | owner | Patch attempt detail | Cross-token access returns **404, not 403** — same existence-leak guard as clawdforge sessions. ## Quickstart After build + first boot, the admin bearer is written to `/data/admin-bearer.txt` (chmod 600 inside the container — readable from the bind-mounted appdata path on the host). ```bash ADMIN=$(cat /mnt/user/appdata/crafting-table/data/admin-bearer.txt) # Mint a project-scoped token TOKEN=$(curl -s http://192.168.0.5:8810/admin/tokens \ -H "Authorization: Bearer $ADMIN" \ -H "content-type: application/json" \ -d '{"name":"clawdforge","is_admin":false,"ip_cidrs":[]}' | jq -r .bearer) # Register the project curl -s http://192.168.0.5:8810/projects \ -H "Authorization: Bearer $TOKEN" \ -H "content-type: application/json" \ -d '{ "name": "clawdforge", "git_url": "http://192.168.0.5:3001/Sulkta-Coop/clawdforge.git", "default_branch": "main", "languages": ["python", "rust"], "subprojects": [ {"path": "clients/python", "language": "python", "test": "pytest tests/", "lint": "ruff check .", "audit": "pip-audit", "timeout_secs": 600}, {"path": "clients/rust", "language": "rust", "build": "cargo build --release", "test": "cargo test --all", "audit": "cargo audit", "timeout_secs": 1800} ] }' # Kick off a test job JOB=$(curl -s http://192.168.0.5:8810/projects/clawdforge/jobs \ -H "Authorization: Bearer $TOKEN" \ -H "content-type: application/json" \ -d '{"recipe":"test","subproject":"clients/python"}' | jq -r .job_id) # Poll status watch -n2 "curl -s http://192.168.0.5:8810/jobs/$JOB \ -H 'Authorization: Bearer $TOKEN' | jq '.job.status, .log_tail[-5:]'" # Stream the full log curl http://192.168.0.5:8810/jobs/$JOB/log \ -H "Authorization: Bearer $TOKEN" ``` ## Build + smoke ```bash docker network inspect sulkta >/dev/null 2>&1 || docker network create sulkta docker compose build # Run the per-toolchain hello-world smoke docker compose run --rm crafting-table /usr/local/bin/smoke.sh # expect: "=== ALL TOOLCHAINS GREEN ===" # Bring up the API docker compose up -d curl http://192.168.0.5:8810/healthz ``` ## Image notes - Base: `debian:bookworm-slim`. Swift uses the upstream Ubuntu 22.04 tarball which links against bookworm's libicu/libstdc++ baseline. - Runs as non-root user `crafter` (uid 1000) with passwordless sudo. The API server runs as `crafter`. Recipe commands run as `crafter` too — never elevated to root. - Volume mount points (production): - `/data` — SQLite ledger, admin bearer file, per-job logs - `/workspace` — bare clones + per-job worktrees - `/caches` — cargo / maven / gradle / npm / pip / bun caches - Network: external `sulkta` bridge (same one clawdforge + cauldron use). Create with `docker network create sulkta` if missing. - Image size baseline is large (8-15 GB expected). Per spec: that's fine. ## Architecture notes ### Recipe security Recipe commands run via `/bin/sh -c` so any shell metachar works. This is **by design** — admins set them. Recipes like `cargo build && cargo test` work as expected; `; rm -rf /` would too if an admin set it. The container sandbox + `crafter` user are the safety net, not the recipe parser. ### Workspace strategy - `/workspace//.cache/` — bare clone of the upstream - `/workspace///` — git worktree pointing at the requested branch+sha, removed after the job ends - Periodic gc (1h timer) prunes worktrees older than 24h and runs `git gc --prune=now` on bare clones quiet for >7d ### Recipe immutability Every job snapshots the project's recipe at run-time (`recipe_snapshot_json`). Editing a project's recipe doesn't retcon the view of in-flight or already-finished jobs. ### Process group + timeout Recipe subprocesses spawn in their own process group via `start_new_session=True`. On timeout we `os.killpg(pgid, SIGTERM)` and grace for 10s before escalating to `SIGKILL`. Without process-group kill, multi-process recipes (cargo build spawning rustc, etc.) could leave orphans that hold the stdout pipe open. ### Restart resilience Jobs marked `running` but stranded by a process crash are NOT auto-resumed. On startup the runner sweeps them to `failed` with `exit_code=-1` and appends a synthetic log line `[crafting-table] runner restart, job orphaned`. Callers can re-enqueue if they want. ### SQLite + WAL Single host, single process — SQLite is plenty. WAL mode (`PRAGMA journal_mode=WAL` + `synchronous=NORMAL`) gives many readers + one writer without lock contention. The runner is the only mutator of `jobs`/ `findings`; HTTP workers mostly read. ## Layout ``` . ├── Dockerfile # monolith image: toolchains + Python app ├── compose.yml # build + run wiring ├── smoke.sh # per-language hello-world test ├── crafting_table/ │ ├── server.py # FastAPI app + endpoints │ ├── db.py # SQLite + migrations │ ├── auth.py # bearer + IP allowlist │ ├── runner.py # async job pool + subprocess exec │ ├── workspace.py # bare clone + worktree materialization + gc │ ├── models.py # Pydantic schemas │ ├── digest.py # email digest scheduler │ ├── patcher.py # autonomous patch loop (clawdforge → diff → verify → PR) │ ├── parsers/ # per-language Finding extractors │ └── config.py # env-driven config ├── tests/ # pytest suite (143 tests) ├── mcp/ # crafting-table-mcp — MCP stdio bridge (separate pip install) ├── examples/ │ ├── recipes/ # production recipes — clawdforge, cauldron, tradecraft │ └── register-all.sh # bulk-register helper ├── pyproject.toml ├── requirements.txt └── .env.example ``` ## Tests ```bash python3 -m venv .venv && source .venv/bin/activate pip install -e '.[test]' pytest tests/ ``` ## Findings After every job, the runner reads the captured log and hands it to a per-language parser. The parser turns native tool output (clippy JSON, ruff JSON, govulncheck NDJSON, eslint JSON, tsc human errors, etc.) into structured rows in the `findings` table. ### Parsers in v0.1 | Language | Recipes parsed | Tool output expected | |-------------|------------------------------|------------------------------------------------------| | rust | audit, lint, test, build | `cargo audit --json`, `cargo clippy --message-format=json`, `cargo test` (human) | | python | audit, lint, test, build | `pip-audit -f json`, `ruff --output-format=json`, `mypy --output=json`, `pytest --tb=line` | | go | audit, lint, build, test | `govulncheck -json`, `go vet -json` | | typescript | lint, build, test, audit | `eslint -f json`, `tsc --noEmit` (stderr) | | javascript | (alias of typescript) | same | | _other_ | (any) | falls back to `GenericParser` — emits one `recipe_fail` row when exit_code != 0, else nothing | Resolution order in the registry: exact `language` match (parsers self-gate on recipe via `Parser.matches`), then `GenericParser`. Adding a new language is a single file in `crafting_table/parsers/` plus an entry in `PARSERS`. ### Finding kinds - `lint` — clippy / ruff / mypy / eslint / tsc / go vet diagnostic. - `cve` — vulnerability from cargo-audit / pip-audit / govulncheck. Carries `code` = advisory id (RUSTSEC-..., GO-..., PYSEC-...) and a `suggested_fix` of the form "bump to " when patched versions are known. - `test_fail` — failed test name extracted from `cargo test` / `pytest` output. - `recipe_fail` — fallback when no language-specific parser fired and the recipe exited non-zero. `code` = `exit_`, message names the recipe. ### Fingerprints + dedup Every finding row carries a 16-char `fingerprint` hash over `kind|file|line|code` (NOT the message — tool wording drifts). The same lint reappearing across nightly runs produces the same fingerprint, so a later wave can dedup digest output and surface only "new since last run." ### Consuming findings ``` GET /jobs/{job_id}/findings → {"ok": true, "findings": [ {"id": 1, "job_id": "...", "kind": "lint", "severity": "warn", "file": "src/app.py", "line": 3, "code": "F401", "message": "...", "suggested_fix": "...", "fingerprint": "...", "raw_json": "{...}", "created_at": ...}, ... ]} ``` Authorization is project-token-scoped (same model as `/jobs/{id}`). The matching `job` row's `findings_count` mirrors the array length so callers can decide whether to fetch the detail. ## Digest Daily 06:00 PT email digest. One message per project per day; aggregates the last 24h of jobs per recipient and sends via SMTP relay (Lucy postfix). Set the SMTP block in `.env` to enable — leaving `CRAFTING_SMTP_HOST` unset keeps the scheduler off and logs `digest disabled — CRAFTING_SMTP_HOST not set` at startup. The `/digests` and `/admin/digest/run-now` endpoints still work in dry-run mode regardless. ```bash CRAFTING_SMTP_HOST=postfix.sulkta.com CRAFTING_SMTP_PORT=587 CRAFTING_SMTP_USER=crafting-table@sulkta.com CRAFTING_SMTP_PASS=... CRAFTING_SMTP_FROM=crafting-table@sulkta.com CRAFTING_SMTP_TLS=1 ``` Each project's `notify.email` + `notify.on` fields control delivery: | `notify.on` event | When it fires | |---------------------|--------------------------------------------------------| | `audit_pass` | a passing audit job | | `audit_fail` | a failing audit job | | `test_fail` | a failing test job | | `lint_warn` | a lint job with warning-severity findings | | `cve_found` | any job whose findings include a `cve` | | `patch_drafted` | (wave 3 / step 9) auto-patch was drafted | | `nightly_summary` | catch-all — show ALL jobs in the project's section | Empty `notify.on` defaults to `["audit_fail", "cve_found", "patch_drafted"]`. Empty `notify.email` silently excludes the project. Manual trigger from the LAN admin token: ```bash # Render today's digest without sending curl -sH "Authorization: Bearer $ADMIN" \ -X POST http://192.168.0.5:8810/admin/digest/run-now \ -d '{"dry_run": true}' | jq . # Render an arbitrary date as JSON curl -sH "Authorization: Bearer $ADMIN" \ http://192.168.0.5:8810/digests/2026-04-29 | jq . ``` Idempotency: `digest_runs` table holds `UNIQUE(date, project_name)`, so the 06:00 loop is safe to re-fire on the same day — only the first call sends. ## Autonomous patch loop Wave 3 wires crafting-table into clawdforge so a project with `notify.auto_patch=true` gets an automatic patch attempt on every actionable finding (lint with file/line; cve with a known fix). Lifecycle: 1. Runner finishes a job + parsers populate findings. 2. Post-job hook fires: pulls the highest-severity actionable finding, reads ±20 lines of context from the worktree. 3. Patcher opens a clawdforge session (`POST /sessions`), sends one turn with the finding + source context + project metadata, expects `{"diff": ..., "explanation": ..., "confidence": ...}` back. 4. Diff applied to a fresh worktree on `crafting-table/auto/-`. Apply failure → status `apply_failed`. 5. Recipe re-runs against the patched worktree (the **verify** step). Fail → `verify_failed`. 6. Pass → commit + push + open Gitea PR. Status `pr_opened`. 7. clawdforge session always closed. Configuration (env vars): ``` CRAFTING_CLAWDFORGE_URL=http://192.168.0.5:8800 CRAFTING_CLAWDFORGE_TOKEN=cf_... CRAFTING_GITEA_URL=http://192.168.0.5:3001 CRAFTING_GITEA_TOKEN= CRAFTING_PATCHER_MAX_ATTEMPTS=3 CRAFTING_PATCHER_BRANCH_PREFIX=crafting-table/auto/ ``` If any of the four required vars is missing, the patcher stays disabled and `POST /jobs/{id}/patches` returns 503. The runner hook silently no-ops in that case so existing job flow is unaffected. **Verification cost matters.** The verify step re-runs the failing recipe on the patched worktree — for projects with multi-minute builds this DOUBLES the latency. Set `notify.auto_patch=true` only for projects where the audit/test recipe is <5min, OR accept the latency. v0.2 candidate: "fast verify" mode that re-runs only the specific lint that fired. `patch_attempts` table holds every attempt with `UNIQUE(finding_id, attempt_number)`; the loop early-exits at `max_attempts_per_finding` (default 3). No auto-merge; PRs land for human review. Manual trigger: ```bash curl -sH "Authorization: Bearer $TOKEN" \ -X POST http://192.168.0.5:8810/jobs/$JOB/patches \ -d '{"finding_id": 42}' | jq . # → {"ok": true, "attempt": {"status": "pr_opened", "pr_url": "...", ...}} ``` ## First production recipes Three recipes ship in `examples/recipes/`: | Recipe | Subprojects | Schedule (audit) | auto_patch | |---------------|-------------|------------------|------------| | `clawdforge` | 14 (one per SDK + root) | nightly 02:00 | **true** | | `cauldron` | 1 (Flask app, `.`) | nightly 02:00 | **true** | | `tradecraft` | 1 (`.`) | nightly 02:00 | **false** (manual review) | Each ships with a placeholder `REPLACE_WITH_GITEA_TOKEN` in `git_url`; `examples/register-all.sh` substitutes `$GITEA_TOKEN` at register time so no real token ever lands in the repo. Smoke procedure (post-deploy): ``` 1. docker compose up -d 2. TOKEN=$(cat /mnt/user/appdata/crafting-table/data/admin-bearer.txt) 3. CRAFTING_TABLE_TOKEN=$TOKEN GITEA_TOKEN= bash examples/register-all.sh 4. curl -H "Authorization: Bearer $TOKEN" http://192.168.0.5:8810/projects \ → expect 3 projects (clawdforge, cauldron, tradecraft) 5. curl -X POST -H "Authorization: Bearer $TOKEN" \ http://192.168.0.5:8810/projects/clawdforge/jobs \ -d '{"recipe":"test","subproject":"clients/python"}' → expect job_id 6. Poll GET /jobs/{job_id} until status terminal → expect succeeded ``` Per-recipe smoke status (today, pre-deploy): - `clawdforge` — 14 subprojects; `clients/python` & `clients/typescript` & `clients/go` & `clients/rust` known clean from existing CI; ruby / php / kotlin / java / csharp / swift compile-cleanly today but toolchain availability inside the crafting-table image is what step 1 smoke verified. Bash subproject's `test/run.sh` may not exist (manual check needed post-deploy). - `cauldron` — single Flask subproject; pip-audit & pytest known to run cleanly from the cauldron repo's own CI history. - `tradecraft` — single subproject; auto_patch is **off** by design (production app, manual PR review only). ## MCP bridge The `mcp/` subdirectory ships a self-contained `crafting-table-mcp` Python package that exposes the HTTP API to MCP-aware clients (Claude Desktop, Claude Code, Cursor, Zed, custom agents). See [mcp/README.md](mcp/README.md) for the tool surface, installation, and configuration. Quickstart: ```bash pip install -e mcp export CRAFTING_TABLE_BASE_URL=http://192.168.0.5:8810 export CRAFTING_TABLE_TOKEN=ct_... crafting-table-mcp # stdio JSON-RPC server ``` ## License MIT