crafting-table/README.md

crafting-table

Polyglot dev/build/audit container — the build farm for the Sulkta ecosystem.

What this is

A single Docker container with every toolchain we work with, fronted by a FastAPI HTTP API + async job runner. Used as a reliable place to compile / test / audit any Sulkta repo regardless of where the caller is — agents, Claude sessions, ad-hoc curl, scheduled cron.

Eventual surface (v0.1 full): HTTP API + MCP server + project registry + job runner + structured findings + email digest + autonomous patch loop through clawdforge.

Spec: Sulkta-Coop/openclaw-workspace/memory/spec-crafting-table.md (LAN-only).

Status — v0.1 step 7 of 10

• Step 1: Dockerfile + per-language smoke
• Step 2: SQLite ledger + project registry
• Step 3: HTTP API skeleton (FastAPI, port 8810)
• Step 4: Job runner core (asyncio worker pool, git worktree, subprocess)
• Step 5: Per-language parsers (Rust / Python / Go / TS first)
• Step 6: Findings extraction + storage
• Step 7: MCP server (stdio JSON-RPC, 8 tools) — see mcp/README.md
• Step 8: Email digest scheduler
• Step 9: Autonomous patch loop (clawdforge integration)
• Step 10: Production recipes — clawdforge, cauldron, tradecraft

Toolchains in v0.1

Lang	Versions / extras
Python	3.11 (Debian default) + uv, pipx, pip-audit, ruff, mypy, pytest, semgrep
Node	22.11.0 LTS + npm, pnpm, tsx, eslint, typescript
Bun	latest (rolling)
Go	1.22.10 + govulncheck, staticcheck
Rust	stable (rustup) + clippy, rustfmt, cargo-audit, cargo-deny
Ruby	3.1 (Debian default) + bundler, bundler-audit, rubocop
PHP	8.2 (Debian default) + composer, phpstan, phpunit
JDK	17 (default) + 21 (Temurin, alongside via JAVA_HOME_21)
Maven	3.x (Debian)
Gradle	8.10
.NET	8.0 SDK
Swift	5.9.2 (Ubuntu 22.04 tarball — works on Debian bookworm)
Kotlin	1.9.25 (compiler)
C/C++	clang + lld + cmake + ninja + valgrind
Bash	bash + shellcheck + bats + shfmt
Generic	git, jq, yq, ripgrep, fd, gh-cli, curl, wget

HTTP API surface

LAN-only. Every request needs Authorization: Bearer <token>. The default LAN allowlist is 10/8, 172.16/12, 192.168/16, 127/8, ::1/128; override via CRAFTING_LAN_CIDRS.

Method	Path	Who	What
GET	/healthz	LAN	Liveness + runner stats
POST	/admin/tokens	admin	Mint a new bearer
GET	/admin/tokens	admin	List tokens
DELETE	/admin/tokens/{name}	admin	Revoke a token
POST	/projects	any	Register (becomes owner)
GET	/projects	any	List own (or all if admin)
GET	/projects/{name}	owner	Detail (404 if not visible)
PUT	/projects/{name}	owner	Update
DELETE	/projects/{name}	owner	Remove (cascades jobs+findings)
POST	/projects/{name}/jobs	owner	Enqueue a recipe run
GET	/jobs?project=&status=&limit=	any	List own (or all if admin)
GET	/jobs/{id}	owner	State + last 200 log lines
GET	/jobs/{id}/log	owner	Full log (file stream)
GET	/jobs/{id}/findings	owner	Structured findings (see Findings)

Cross-token access returns 404, not 403 — same existence-leak guard as clawdforge sessions.

Quickstart

After build + first boot, the admin bearer is written to /data/admin-bearer.txt (chmod 600 inside the container — readable from the bind-mounted appdata path on the host).

ADMIN=$(cat /mnt/user/appdata/crafting-table/data/admin-bearer.txt)

# Mint a project-scoped token
TOKEN=$(curl -s http://192.168.0.5:8810/admin/tokens \
  -H "Authorization: Bearer $ADMIN" \
  -H "content-type: application/json" \
  -d '{"name":"clawdforge","is_admin":false,"ip_cidrs":[]}' | jq -r .bearer)

# Register the project
curl -s http://192.168.0.5:8810/projects \
  -H "Authorization: Bearer $TOKEN" \
  -H "content-type: application/json" \
  -d '{
    "name": "clawdforge",
    "git_url": "http://192.168.0.5:3001/Sulkta-Coop/clawdforge.git",
    "default_branch": "main",
    "languages": ["python", "rust"],
    "subprojects": [
      {"path": "clients/python", "language": "python",
       "test": "pytest tests/", "lint": "ruff check .", "audit": "pip-audit",
       "timeout_secs": 600},
      {"path": "clients/rust", "language": "rust",
       "build": "cargo build --release", "test": "cargo test --all",
       "audit": "cargo audit", "timeout_secs": 1800}
    ]
  }'

# Kick off a test job
JOB=$(curl -s http://192.168.0.5:8810/projects/clawdforge/jobs \
  -H "Authorization: Bearer $TOKEN" \
  -H "content-type: application/json" \
  -d '{"recipe":"test","subproject":"clients/python"}' | jq -r .job_id)

# Poll status
watch -n2 "curl -s http://192.168.0.5:8810/jobs/$JOB \
  -H 'Authorization: Bearer $TOKEN' | jq '.job.status, .log_tail[-5:]'"

# Stream the full log
curl http://192.168.0.5:8810/jobs/$JOB/log \
  -H "Authorization: Bearer $TOKEN"

Build + smoke

docker network inspect sulkta >/dev/null 2>&1 || docker network create sulkta
docker compose build

# Run the per-toolchain hello-world smoke
docker compose run --rm crafting-table /usr/local/bin/smoke.sh
# expect: "=== ALL TOOLCHAINS GREEN ==="

# Bring up the API
docker compose up -d
curl http://192.168.0.5:8810/healthz

Image notes

• Base: debian:bookworm-slim. Swift uses the upstream Ubuntu 22.04 tarball which links against bookworm's libicu/libstdc++ baseline.
• Runs as non-root user crafter (uid 1000) with passwordless sudo. The API server runs as crafter. Recipe commands run as crafter too — never elevated to root.
• Volume mount points (production):
  • /data — SQLite ledger, admin bearer file, per-job logs
  • /workspace — bare clones + per-job worktrees
  • /caches — cargo / maven / gradle / npm / pip / bun caches
• Network: external sulkta bridge (same one clawdforge + cauldron use). Create with docker network create sulkta if missing.
• Image size baseline is large (8-15 GB expected). Per spec: that's fine.

Architecture notes

Recipe security

Recipe commands run via /bin/sh -c so any shell metachar works. This is by design — admins set them. Recipes like cargo build && cargo test work as expected; ; rm -rf / would too if an admin set it. The container sandbox + crafter user are the safety net, not the recipe parser.

Workspace strategy

• /workspace/<project>/.cache/ — bare clone of the upstream
• /workspace/<project>/<job_id>/ — git worktree pointing at the requested branch+sha, removed after the job ends
• Periodic gc (1h timer) prunes worktrees older than 24h and runs git gc --prune=now on bare clones quiet for >7d

Recipe immutability

Every job snapshots the project's recipe at run-time (recipe_snapshot_json). Editing a project's recipe doesn't retcon the view of in-flight or already-finished jobs.

Process group + timeout

Recipe subprocesses spawn in their own process group via start_new_session=True. On timeout we os.killpg(pgid, SIGTERM) and grace for 10s before escalating to SIGKILL. Without process-group kill, multi-process recipes (cargo build spawning rustc, etc.) could leave orphans that hold the stdout pipe open.

Restart resilience

Jobs marked running but stranded by a process crash are NOT auto-resumed. On startup the runner sweeps them to failed with exit_code=-1 and appends a synthetic log line [crafting-table] runner restart, job orphaned. Callers can re-enqueue if they want.

SQLite + WAL

Single host, single process — SQLite is plenty. WAL mode (PRAGMA journal_mode=WAL + synchronous=NORMAL) gives many readers + one writer without lock contention. The runner is the only mutator of jobs/ findings; HTTP workers mostly read.

Layout

.
├── Dockerfile           # monolith image: toolchains + Python app
├── compose.yml          # build + run wiring
├── smoke.sh             # per-language hello-world test
├── crafting_table/
│   ├── server.py        # FastAPI app + endpoints
│   ├── db.py            # SQLite + migrations
│   ├── auth.py          # bearer + IP allowlist
│   ├── runner.py        # async job pool + subprocess exec
│   ├── workspace.py     # bare clone + worktree materialization + gc
│   ├── models.py        # Pydantic schemas
│   └── config.py        # env-driven config
├── tests/               # pytest suite (~60 tests)
├── mcp/                 # crafting-table-mcp — MCP stdio bridge (separate pip install)
├── pyproject.toml
├── requirements.txt
└── .env.example

Tests

python3 -m venv .venv && source .venv/bin/activate
pip install -e '.[test]'
pytest tests/

Findings

After every job, the runner reads the captured log and hands it to a per-language parser. The parser turns native tool output (clippy JSON, ruff JSON, govulncheck NDJSON, eslint JSON, tsc human errors, etc.) into structured rows in the findings table.

Parsers in v0.1

Language	Recipes parsed	Tool output expected
rust	audit, lint, test, build	cargo audit --json, cargo clippy --message-format=json, cargo test (human)
python	audit, lint, test, build	pip-audit -f json, ruff --output-format=json, mypy --output=json, pytest --tb=line
go	audit, lint, build, test	govulncheck -json, go vet -json
typescript	lint, build, test, audit	eslint -f json, tsc --noEmit (stderr)
javascript	(alias of typescript)	same
other	(any)	falls back to GenericParser — emits one recipe_fail row when exit_code != 0, else nothing

Resolution order in the registry: exact language match (parsers self-gate on recipe via Parser.matches), then GenericParser. Adding a new language is a single file in crafting_table/parsers/ plus an entry in PARSERS.

Finding kinds

• lint — clippy / ruff / mypy / eslint / tsc / go vet diagnostic.
• cve — vulnerability from cargo-audit / pip-audit / govulncheck. Carries code = advisory id (RUSTSEC-..., GO-..., PYSEC-...) and a suggested_fix of the form "bump to " when patched versions are known.
• test_fail — failed test name extracted from cargo test / pytest output.
• recipe_fail — fallback when no language-specific parser fired and the recipe exited non-zero. code = exit_<n>, message names the recipe.

Fingerprints + dedup

Every finding row carries a 16-char fingerprint hash over kind|file|line|code (NOT the message — tool wording drifts). The same lint reappearing across nightly runs produces the same fingerprint, so a later wave can dedup digest output and surface only "new since last run."

Consuming findings

GET /jobs/{job_id}/findings
  → {"ok": true, "findings": [
      {"id": 1, "job_id": "...", "kind": "lint", "severity": "warn",
       "file": "src/app.py", "line": 3, "code": "F401",
       "message": "...", "suggested_fix": "...", "fingerprint": "...",
       "raw_json": "{...}", "created_at": ...},
      ...
    ]}

Authorization is project-token-scoped (same model as /jobs/{id}). The matching job row's findings_count mirrors the array length so callers can decide whether to fetch the detail.

Digest

Daily 06:00 PT email digest. One message per project per day; aggregates the last 24h of jobs per recipient and sends via SMTP relay (Lucy postfix).

Set the SMTP block in .env to enable — leaving CRAFTING_SMTP_HOST unset keeps the scheduler off and logs digest disabled — CRAFTING_SMTP_HOST not set at startup. The /digests and /admin/digest/run-now endpoints still work in dry-run mode regardless.

CRAFTING_SMTP_HOST=postfix.sulkta.com
CRAFTING_SMTP_PORT=587
CRAFTING_SMTP_USER=crafting-table@sulkta.com
CRAFTING_SMTP_PASS=...
CRAFTING_SMTP_FROM=crafting-table@sulkta.com
CRAFTING_SMTP_TLS=1

Each project's notify.email + notify.on fields control delivery:

notify.on event	When it fires
audit_pass	a passing audit job
audit_fail	a failing audit job
test_fail	a failing test job
lint_warn	a lint job with warning-severity findings
cve_found	any job whose findings include a cve
patch_drafted	(wave 3 / step 9) auto-patch was drafted
nightly_summary	catch-all — show ALL jobs in the project's section

Empty notify.on defaults to ["audit_fail", "cve_found", "patch_drafted"]. Empty notify.email silently excludes the project.

Manual trigger from the LAN admin token:

# Render today's digest without sending
curl -sH "Authorization: Bearer $ADMIN" \
  -X POST http://192.168.0.5:8810/admin/digest/run-now \
  -d '{"dry_run": true}' | jq .

# Render an arbitrary date as JSON
curl -sH "Authorization: Bearer $ADMIN" \
  http://192.168.0.5:8810/digests/2026-04-29 | jq .

Idempotency: digest_runs table holds UNIQUE(date, project_name), so the 06:00 loop is safe to re-fire on the same day — only the first call sends.

MCP bridge

The mcp/ subdirectory ships a self-contained crafting-table-mcp Python package that exposes the HTTP API to MCP-aware clients (Claude Desktop, Claude Code, Cursor, Zed, custom agents). See mcp/README.md for the tool surface, installation, and configuration.

Quickstart:

pip install -e mcp
export CRAFTING_TABLE_BASE_URL=http://192.168.0.5:8810
export CRAFTING_TABLE_TOKEN=ct_...
crafting-table-mcp  # stdio JSON-RPC server

License

MIT