crafting-table/README.md

crafting-table

Polyglot dev/build/audit container. HTTP API + async job runner + autonomous patch loop + email digest.

A single Docker image carries every toolchain in the matrix below. You hand it a git URL + a per-language recipe (build / test / lint / audit), it materializes a worktree, runs the recipe, parses the tool output into structured findings, and stores everything in a SQLite ledger. Optional patch loop drafts a unified diff via an external Claude-style agent, verifies it by re-running the recipe, and opens a PR.

Use it as a replacement for ad-hoc per-repo build environments. One image, one runner, every language.

Toolchains

Lang	Versions / extras
Python	3.11 (Debian default) + uv, pipx, pip-audit, ruff, mypy, pytest, semgrep
Node	22.11.0 LTS + npm, pnpm, tsx, eslint, typescript
Bun	latest (rolling)
Go	1.25.9 + govulncheck, staticcheck
Rust	stable (rustup) + clippy, rustfmt
Ruby	3.1 (Debian default) + bundler, bundler-audit, rubocop
PHP	8.2 (Debian default) + composer, phpstan, phpunit
JDK	17 (default) + 21 (Temurin, alongside via JAVA_HOME_21)
Maven	3.x (Debian)
Gradle	8.10
.NET	8.0 SDK
Swift	5.9.2 (Ubuntu 22.04 tarball — works on Debian bookworm)
Kotlin	1.9.25 (compiler)
C/C++	clang + lld + cmake + ninja + valgrind
Bash	bash + shellcheck + bats + shfmt
Nix	single-user install with cache.nixos.org + cache.iog.io substituters
Generic	git, jq, yq, ripgrep, fd, gh-cli, curl, wget

cargo-audit and cargo-deny are NOT baked into the image — both flaked during build (libgit2-sys C bindings + GitHub release download flakiness). Install at runtime with cargo install cargo-audit cargo-deny if you need them.

HTTP API

LAN-only. Every request needs Authorization: Bearer <token>. Default IP allowlist is 10/8, 172.16/12, 192.168/16, 127/8, ::1/128; override via CRAFTING_LAN_CIDRS.

Method	Path	Who	What
GET	/healthz	LAN	Liveness + runner stats
POST	/admin/tokens	admin	Mint a new bearer
GET	/admin/tokens	admin	List tokens
DELETE	/admin/tokens/{name}	admin	Revoke a token
POST	/projects	any	Register (becomes owner)
GET	/projects	any	List own (or all if admin)
GET	/projects/{name}	owner	Detail (404 if not visible)
PUT	/projects/{name}	owner	Update
DELETE	/projects/{name}	owner	Remove (cascades jobs+findings)
POST	/projects/{name}/jobs	owner	Enqueue a recipe run
GET	/jobs?project=&status=&limit=	any	List own (or all if admin)
GET	/jobs/{id}	owner	State + last 200 log lines
GET	/jobs/{id}/log	owner	Full log (file stream)
GET	/jobs/{id}/findings	owner	Structured findings
POST	/jobs/{id}/patches	owner	Trigger an auto-patch attempt
GET	/patches?project=&status=&limit=	any	List own patch attempts
GET	/patches/{id}	owner	Patch attempt detail

Cross-token access returns 404, not 403 — existence-leak guard.

Quickstart

After build + first boot, the admin bearer is written to /data/admin-bearer.txt (chmod 600 inside the container — readable from the bind-mounted host path).

HOST=http://localhost:8810
ADMIN=$(cat ./data/admin-bearer.txt)

# Mint a project-scoped token
TOKEN=$(curl -s "$HOST/admin/tokens" \
  -H "Authorization: Bearer $ADMIN" \
  -H "content-type: application/json" \
  -d '{"name":"alpha","is_admin":false,"ip_cidrs":[]}' | jq -r .bearer)

# Register a project
curl -s "$HOST/projects" \
  -H "Authorization: Bearer $TOKEN" \
  -H "content-type: application/json" \
  -d '{
    "name": "alpha",
    "git_url": "http://git.example.com/org/alpha.git",
    "default_branch": "main",
    "languages": ["python", "rust"],
    "subprojects": [
      {"path": "clients/python", "language": "python",
       "test": "pytest tests/", "lint": "ruff check .", "audit": "pip-audit",
       "timeout_secs": 600},
      {"path": "clients/rust", "language": "rust",
       "build": "cargo build --release", "test": "cargo test --all",
       "timeout_secs": 1800}
    ]
  }'

# Kick off a test job
JOB=$(curl -s "$HOST/projects/alpha/jobs" \
  -H "Authorization: Bearer $TOKEN" \
  -H "content-type: application/json" \
  -d '{"recipe":"test","subproject":"clients/python"}' | jq -r .job_id)

# Poll status
curl -s "$HOST/jobs/$JOB" -H "Authorization: Bearer $TOKEN" | jq '.job.status, .log_tail[-5:]'

# Stream the full log
curl "$HOST/jobs/$JOB/log" -H "Authorization: Bearer $TOKEN"

Build + smoke

docker compose build

# Per-toolchain hello-world
docker compose run --rm crafting-table /usr/local/bin/smoke.sh
# expect: "=== ALL TOOLCHAINS GREEN ==="

# Bring up the API
docker compose up -d
curl http://localhost:8810/healthz

Image notes

• Base: debian:bookworm-slim. Swift uses the upstream Ubuntu 22.04 tarball which links against bookworm's libicu/libstdc++ baseline.
• Runs as non-root user crafter (uid 1000) with passwordless sudo. The API server runs as crafter. Recipe commands run as crafter too — never elevated to root.
• Volume mount points:
  • /data — SQLite ledger, admin bearer file, per-job logs
  • /workspace — bare clones + per-job worktrees
  • /caches — cargo / maven / gradle / npm / pip / bun caches
  • /nix — pre-seeded single-user Nix store (Plutarch / IOG flakes)
• Image size baseline is large (8-15 GB). By design.

Architecture

Recipe security

Recipe commands run via /bin/sh -c so any shell metachar works. By design — admins set them. Recipes like cargo build && cargo test work as expected; ; rm -rf / would too if an admin set it. The container sandbox + crafter user are the safety net, not the recipe parser.

Workspace

• /workspace/<project>/.cache/ — bare clone of the upstream
• /workspace/<project>/<job_id>/ — git worktree pointing at the requested branch+sha, removed after the job ends
• Periodic gc (1h timer) prunes worktrees older than 24h and runs git gc --prune=now on bare clones quiet for >7d

Recipe immutability

Every job snapshots the project's recipe at run-time (recipe_snapshot_json). Editing a project's recipe doesn't retcon the view of in-flight or finished jobs.

Process group + timeout

Recipe subprocesses spawn in their own process group via start_new_session=True. On timeout we os.killpg(pgid, SIGTERM) and grace for 10s before escalating to SIGKILL. Without process-group kill, multi-process recipes (cargo build spawning rustc, etc.) leave orphans that hold the stdout pipe open.

Restart resilience

Jobs marked running but stranded by a process crash are NOT auto-resumed. On startup the runner sweeps them to failed with exit_code=-1 and a synthetic log line [crafting-table] runner restart, job orphaned. Callers can re-enqueue if they want.

SQLite + WAL

Single host, single process. WAL mode (PRAGMA journal_mode=WAL + synchronous=NORMAL) gives many readers + one writer without lock contention. The runner is the only mutator of jobs/findings; HTTP workers mostly read.

Layout

.
├── Dockerfile           # monolith image: toolchains + Python app
├── compose.yml          # build + run wiring
├── smoke.sh             # per-language hello-world test
├── crafting_table/
│   ├── server.py        # FastAPI app + endpoints
│   ├── db.py            # SQLite + migrations
│   ├── auth.py          # bearer + IP allowlist
│   ├── runner.py        # async job pool + subprocess exec
│   ├── workspace.py     # bare clone + worktree materialization + gc
│   ├── models.py        # Pydantic schemas
│   ├── digest.py        # email digest scheduler
│   ├── patcher.py       # autonomous patch loop
│   ├── parsers/         # per-language Finding extractors
│   └── config.py        # env-driven config
├── tests/               # pytest suite
├── mcp/                 # crafting-table-mcp — MCP stdio bridge (separate pip install)
├── examples/
│   ├── recipes/         # example recipe JSONs
│   └── register-all.sh  # bulk-register helper
├── pyproject.toml
├── requirements.txt
└── .env.example

Tests

python3 -m venv .venv && source .venv/bin/activate
pip install -e '.[test]'
pytest tests/

Findings

After every job the runner reads the captured log and hands it to a per-language parser. The parser turns native tool output (clippy JSON, ruff JSON, govulncheck NDJSON, eslint JSON, tsc human errors, etc.) into structured rows in the findings table.

Parsers

Language	Recipes parsed	Tool output expected
rust	audit, lint, test, build	cargo audit --json, cargo clippy --message-format=json, cargo test (human)
python	audit, lint, test, build	pip-audit -f json, ruff --output-format=json, mypy --output=json, pytest --tb=line
go	audit, lint, build, test	govulncheck -json, go vet -json
typescript	lint, build, test, audit	eslint -f json, tsc --noEmit (stderr)
javascript	(alias of typescript)	same
other	(any)	falls back to GenericParser — emits one recipe_fail row when exit_code != 0, else nothing

Resolution order: exact language match (parsers self-gate on recipe via Parser.matches), then GenericParser. Adding a new language is a single file in crafting_table/parsers/ plus an entry in PARSERS.

Finding kinds

• lint — clippy / ruff / mypy / eslint / tsc / go vet diagnostic.
• cve — vulnerability from cargo-audit / pip-audit / govulncheck. Carries code = advisory id (RUSTSEC-..., GO-..., PYSEC-...) and a suggested_fix of the form "bump to " when patched versions are known.
• test_fail — failed test name extracted from cargo test / pytest output.
• recipe_fail — fallback when no language-specific parser fired and the recipe exited non-zero. code = exit_<n>, message names the recipe.

Fingerprints + dedup

Every finding row carries a 16-char fingerprint hash over kind|file|line|code (NOT the message — tool wording drifts). The same lint reappearing across nightly runs produces the same fingerprint, so later passes can dedup digest output and surface only "new since last run."

Consuming findings

GET /jobs/{job_id}/findings
  → {"ok": true, "findings": [
      {"id": 1, "job_id": "...", "kind": "lint", "severity": "warn",
       "file": "src/app.py", "line": 3, "code": "F401",
       "message": "...", "suggested_fix": "...", "fingerprint": "...",
       "raw_json": "{...}", "created_at": ...},
      ...
    ]}

Authorization is project-token-scoped (same as /jobs/{id}). The matching job row's findings_count mirrors the array length so callers can decide whether to fetch the detail.

Digest

Daily 06:00 PT email digest. One message per project per day; aggregates the last 24h of jobs per recipient and sends via SMTP.

Set the SMTP block in .env to enable — leaving CRAFTING_SMTP_HOST unset keeps the scheduler off and logs digest disabled — CRAFTING_SMTP_HOST not set at startup. The /digests and /admin/digest/run-now endpoints still work in dry-run mode regardless.

CRAFTING_SMTP_HOST=smtp.example.com
CRAFTING_SMTP_PORT=587
CRAFTING_SMTP_USER=crafting-table@example.com
CRAFTING_SMTP_PASS=...
CRAFTING_SMTP_FROM=crafting-table@example.com
CRAFTING_SMTP_TLS=1

Each project's notify.email + notify.on fields control delivery:

notify.on event	When it fires
audit_pass	a passing audit job
audit_fail	a failing audit job
test_fail	a failing test job
lint_warn	a lint job with warning-severity findings
cve_found	any job whose findings include a cve
patch_drafted	an auto-patch was drafted
nightly_summary	catch-all — show ALL jobs in the project's section

Empty notify.on defaults to ["audit_fail", "cve_found", "patch_drafted"]. Empty notify.email silently excludes the project.

Manual trigger from the LAN admin token:

# Render today's digest without sending
curl -sH "Authorization: Bearer $ADMIN" \
  -X POST http://localhost:8810/admin/digest/run-now \
  -d '{"dry_run": true}' | jq .

# Render an arbitrary date as JSON
curl -sH "Authorization: Bearer $ADMIN" \
  http://localhost:8810/digests/2026-04-29 | jq .

Idempotency: digest_runs table holds UNIQUE(date, project_name), so the 06:00 loop is safe to re-fire on the same day — only the first call sends.

Autonomous patch loop

Wires crafting-table into an external Claude-agent host (default: clawdforge) so a project with notify.auto_patch=true gets an automatic patch attempt on every actionable finding (lint with file/line; cve with a known fix). Lifecycle:

1. Runner finishes a job + parsers populate findings.
2. Post-job hook fires: pulls the highest-severity actionable finding, reads ±20 lines of context from the worktree.
3. Patcher opens an agent session, sends one turn with the finding + source context + project metadata, expects {"diff": ..., "explanation": ..., "confidence": ...} back.
4. Diff applied to a fresh worktree on crafting-table/auto/<job_id>-<finding_id>. Apply failure → status apply_failed.
5. Recipe re-runs against the patched worktree (the verify step). Fail → verify_failed.
6. Pass → commit + push + open Gitea PR. Status pr_opened.
7. Agent session always closed.

Configuration (env vars):

CRAFTING_CLAWDFORGE_URL=http://clawdforge.internal:8800
CRAFTING_CLAWDFORGE_TOKEN=cf_...
CRAFTING_GITEA_URL=http://gitea.internal:3000
CRAFTING_GITEA_TOKEN=<gitea PAT>
CRAFTING_PATCHER_MAX_ATTEMPTS=3
CRAFTING_PATCHER_BRANCH_PREFIX=crafting-table/auto/

If any of the four required vars is missing, the patcher stays disabled and POST /jobs/{id}/patches returns 503. The runner hook silently no-ops.

Verification cost matters. The verify step re-runs the failing recipe on the patched worktree — for projects with multi-minute builds this DOUBLES the latency. Set notify.auto_patch=true only for projects where the audit/test recipe is <5min, OR accept the latency. v0.2 candidate: "fast verify" mode that re-runs only the specific lint that fired.

patch_attempts table holds every attempt with UNIQUE(finding_id, attempt_number); the loop early-exits at max_attempts_per_finding (default 3). No auto-merge; PRs land for human review.

Manual trigger:

curl -sH "Authorization: Bearer $TOKEN" \
  -X POST http://localhost:8810/jobs/$JOB/patches \
  -d '{"finding_id": 42}' | jq .
# → {"ok": true, "attempt": {"status": "pr_opened", "pr_url": "...", ...}}

Example recipes

Three example recipes ship in examples/recipes/:

Recipe	Subprojects	auto_patch
alpha	14 (one per SDK + root)	true
beta	1 (Flask app, .)	true
gamma	1 (.)	false (manual review)

Each ships with a placeholder REPLACE_WITH_GITEA_TOKEN in git_url and REPLACE_WITH_GIT_HOST for the host. examples/register-all.sh substitutes $GITEA_TOKEN + $GIT_HOST at register time so no real token or host ever lands in the repo.

Smoke procedure (post-deploy):

1. docker compose up -d
2. TOKEN=$(cat ./data/admin-bearer.txt)
3. CRAFTING_TABLE_TOKEN=$TOKEN GITEA_TOKEN=<your-pat> \
     GIT_HOST=git.example.com bash examples/register-all.sh
4. curl -H "Authorization: Bearer $TOKEN" http://localhost:8810/projects
5. curl -X POST -H "Authorization: Bearer $TOKEN" \
     http://localhost:8810/projects/alpha/jobs \
     -d '{"recipe":"test","subproject":"clients/python"}'
6. Poll GET /jobs/{job_id} until status terminal

MCP bridge

mcp/ ships a self-contained crafting-table-mcp Python package that exposes the HTTP API to MCP-aware clients (Claude Desktop, Claude Code, Cursor, Zed, etc.). See mcp/README.md.

pip install -e mcp
export CRAFTING_TABLE_BASE_URL=http://localhost:8810
export CRAFTING_TABLE_TOKEN=ct_...
crafting-table-mcp  # stdio JSON-RPC server

License

MIT — see LICENSE.