crafting-table/README.md

# crafting-table

Polyglot dev/build/audit container — the build farm for the Sulkta ecosystem.

## What this is

A single Docker container with every toolchain we work with, fronted by a
FastAPI HTTP API + async job runner. Used as a reliable place to compile /
test / audit any Sulkta repo regardless of where the caller is — agents,
Claude sessions, ad-hoc curl, scheduled cron.

Eventual surface (v0.1 full): HTTP API + MCP server + project registry +
job runner + structured findings + email digest + autonomous patch loop
through clawdforge.

Spec: `Sulkta-Coop/openclaw-workspace/memory/spec-crafting-table.md` (LAN-only).

## Status — v0.1 complete (10 of 10)

- [x] Step 1: Dockerfile + per-language smoke
- [x] Step 2: SQLite ledger + project registry
- [x] Step 3: HTTP API skeleton (FastAPI, port 8810)
- [x] Step 4: Job runner core (asyncio worker pool, git worktree, subprocess)
- [x] Step 5: Per-language parsers (Rust / Python / Go / TS first)
- [x] Step 6: Findings extraction + storage
- [x] Step 7: MCP server (stdio JSON-RPC, 8 tools) — see [mcp/README.md](mcp/README.md)
- [x] Step 8: Email digest scheduler
- [x] Step 9: Autonomous patch loop (clawdforge integration → unified diff → worktree apply → verify recipe → push branch → Gitea PR)
- [x] Step 10: Production recipes — clawdforge, cauldron, tradecraft (see [examples/recipes/](examples/recipes/))

## Toolchains in v0.1

| Lang     | Versions / extras                                                  |
|----------|--------------------------------------------------------------------|
| Python   | 3.11 (Debian default) + uv, pipx, pip-audit, ruff, mypy, pytest, semgrep |
| Node     | 22.11.0 LTS + npm, pnpm, tsx, eslint, typescript                    |
| Bun      | latest (rolling)                                                    |
| Go       | 1.22.10 + govulncheck, staticcheck                                  |
| Rust     | stable (rustup) + clippy, rustfmt, cargo-audit, cargo-deny          |
| Ruby     | 3.1 (Debian default) + bundler, bundler-audit, rubocop              |
| PHP      | 8.2 (Debian default) + composer, phpstan, phpunit                   |
| JDK      | 17 (default) + 21 (Temurin, alongside via `JAVA_HOME_21`)           |
| Maven    | 3.x (Debian)                                                        |
| Gradle   | 8.10                                                                |
| .NET     | 8.0 SDK                                                             |
| Swift    | 5.9.2 (Ubuntu 22.04 tarball — works on Debian bookworm)             |
| Kotlin   | 1.9.25 (compiler)                                                   |
| C/C++    | clang + lld + cmake + ninja + valgrind                              |
| Bash     | bash + shellcheck + bats + shfmt                                    |
| Generic  | git, jq, yq, ripgrep, fd, gh-cli, curl, wget                        |

## HTTP API surface

LAN-only. Every request needs `Authorization: Bearer <token>`. The default
LAN allowlist is `10/8`, `172.16/12`, `192.168/16`, `127/8`, `::1/128`;
override via `CRAFTING_LAN_CIDRS`.

| Method | Path                              | Who    | What                           |
|--------|-----------------------------------|--------|--------------------------------|
| GET    | `/healthz`                        | LAN    | Liveness + runner stats        |
| POST   | `/admin/tokens`                   | admin  | Mint a new bearer              |
| GET    | `/admin/tokens`                   | admin  | List tokens                    |
| DELETE | `/admin/tokens/{name}`            | admin  | Revoke a token                 |
| POST   | `/projects`                       | any    | Register (becomes owner)       |
| GET    | `/projects`                       | any    | List own (or all if admin)     |
| GET    | `/projects/{name}`                | owner  | Detail (404 if not visible)    |
| PUT    | `/projects/{name}`                | owner  | Update                         |
| DELETE | `/projects/{name}`                | owner  | Remove (cascades jobs+findings)|
| POST   | `/projects/{name}/jobs`           | owner  | Enqueue a recipe run           |
| GET    | `/jobs?project=&status=&limit=`   | any    | List own (or all if admin)     |
| GET    | `/jobs/{id}`                      | owner  | State + last 200 log lines     |
| GET    | `/jobs/{id}/log`                  | owner  | Full log (file stream)         |
| GET    | `/jobs/{id}/findings`             | owner  | Structured findings (see Findings) |
| POST   | `/jobs/{id}/patches`              | owner  | Trigger an auto-patch attempt (wave 3) |
| GET    | `/patches?project=&status=&limit=`| any    | List own patch attempts        |
| GET    | `/patches/{id}`                   | owner  | Patch attempt detail           |

Cross-token access returns **404, not 403** — same existence-leak guard as
clawdforge sessions.

## Quickstart

After build + first boot, the admin bearer is written to
`/data/admin-bearer.txt` (chmod 600 inside the container — readable from
the bind-mounted appdata path on the host).

```bash
ADMIN=$(cat /mnt/user/appdata/crafting-table/data/admin-bearer.txt)

# Mint a project-scoped token
TOKEN=$(curl -s http://192.168.0.5:8810/admin/tokens \
  -H "Authorization: Bearer $ADMIN" \
  -H "content-type: application/json" \
  -d '{"name":"clawdforge","is_admin":false,"ip_cidrs":[]}' | jq -r .bearer)

# Register the project
curl -s http://192.168.0.5:8810/projects \
  -H "Authorization: Bearer $TOKEN" \
  -H "content-type: application/json" \
  -d '{
    "name": "clawdforge",
    "git_url": "http://192.168.0.5:3001/Sulkta-Coop/clawdforge.git",
    "default_branch": "main",
    "languages": ["python", "rust"],
    "subprojects": [
      {"path": "clients/python", "language": "python",
       "test": "pytest tests/", "lint": "ruff check .", "audit": "pip-audit",
       "timeout_secs": 600},
      {"path": "clients/rust", "language": "rust",
       "build": "cargo build --release", "test": "cargo test --all",
       "audit": "cargo audit", "timeout_secs": 1800}
    ]
  }'

# Kick off a test job
JOB=$(curl -s http://192.168.0.5:8810/projects/clawdforge/jobs \
  -H "Authorization: Bearer $TOKEN" \
  -H "content-type: application/json" \
  -d '{"recipe":"test","subproject":"clients/python"}' | jq -r .job_id)

# Poll status
watch -n2 "curl -s http://192.168.0.5:8810/jobs/$JOB \
  -H 'Authorization: Bearer $TOKEN' | jq '.job.status, .log_tail[-5:]'"

# Stream the full log
curl http://192.168.0.5:8810/jobs/$JOB/log \
  -H "Authorization: Bearer $TOKEN"
```

## Build + smoke

```bash
docker network inspect sulkta >/dev/null 2>&1 || docker network create sulkta
docker compose build

# Run the per-toolchain hello-world smoke
docker compose run --rm crafting-table /usr/local/bin/smoke.sh
# expect: "=== ALL TOOLCHAINS GREEN ==="

# Bring up the API
docker compose up -d
curl http://192.168.0.5:8810/healthz
```

## Image notes

- Base: `debian:bookworm-slim`. Swift uses the upstream Ubuntu 22.04 tarball
  which links against bookworm's libicu/libstdc++ baseline.
- Runs as non-root user `crafter` (uid 1000) with passwordless sudo. The
  API server runs as `crafter`. Recipe commands run as `crafter` too —
  never elevated to root.
- Volume mount points (production):
  - `/data` — SQLite ledger, admin bearer file, per-job logs
  - `/workspace` — bare clones + per-job worktrees
  - `/caches` — cargo / maven / gradle / npm / pip / bun caches
- Network: external `sulkta` bridge (same one clawdforge + cauldron use).
  Create with `docker network create sulkta` if missing.
- Image size baseline is large (8-15 GB expected). Per spec: that's fine.

## Architecture notes

### Recipe security
Recipe commands run via `/bin/sh -c` so any shell metachar works. This is
**by design** — admins set them. Recipes like `cargo build && cargo test`
work as expected; `; rm -rf /` would too if an admin set it. The container
sandbox + `crafter` user are the safety net, not the recipe parser.

### Workspace strategy
- `/workspace/<project>/.cache/` — bare clone of the upstream
- `/workspace/<project>/<job_id>/` — git worktree pointing at the requested
  branch+sha, removed after the job ends
- Periodic gc (1h timer) prunes worktrees older than 24h and runs
  `git gc --prune=now` on bare clones quiet for >7d

### Recipe immutability
Every job snapshots the project's recipe at run-time
(`recipe_snapshot_json`). Editing a project's recipe doesn't retcon the
view of in-flight or already-finished jobs.

### Process group + timeout
Recipe subprocesses spawn in their own process group via
`start_new_session=True`. On timeout we `os.killpg(pgid, SIGTERM)` and grace
for 10s before escalating to `SIGKILL`. Without process-group kill,
multi-process recipes (cargo build spawning rustc, etc.) could leave
orphans that hold the stdout pipe open.

### Restart resilience
Jobs marked `running` but stranded by a process crash are NOT auto-resumed.
On startup the runner sweeps them to `failed` with `exit_code=-1` and
appends a synthetic log line `[crafting-table] runner restart, job
orphaned`. Callers can re-enqueue if they want.

### SQLite + WAL
Single host, single process — SQLite is plenty. WAL mode (`PRAGMA
journal_mode=WAL` + `synchronous=NORMAL`) gives many readers + one writer
without lock contention. The runner is the only mutator of `jobs`/
`findings`; HTTP workers mostly read.

## Layout

```
.
├── Dockerfile           # monolith image: toolchains + Python app
├── compose.yml          # build + run wiring
├── smoke.sh             # per-language hello-world test
├── crafting_table/
│   ├── server.py        # FastAPI app + endpoints
│   ├── db.py            # SQLite + migrations
│   ├── auth.py          # bearer + IP allowlist
│   ├── runner.py        # async job pool + subprocess exec
│   ├── workspace.py     # bare clone + worktree materialization + gc
│   ├── models.py        # Pydantic schemas
│   ├── digest.py        # email digest scheduler
│   ├── patcher.py       # autonomous patch loop (clawdforge → diff → verify → PR)
│   ├── parsers/         # per-language Finding extractors
│   └── config.py        # env-driven config
├── tests/               # pytest suite (143 tests)
├── mcp/                 # crafting-table-mcp — MCP stdio bridge (separate pip install)
├── examples/
│   ├── recipes/         # production recipes — clawdforge, cauldron, tradecraft
│   └── register-all.sh  # bulk-register helper
├── pyproject.toml
├── requirements.txt
└── .env.example
```

## Tests

```bash
python3 -m venv .venv && source .venv/bin/activate
pip install -e '.[test]'
pytest tests/
```

## Findings

After every job, the runner reads the captured log and hands it to a
per-language parser. The parser turns native tool output (clippy JSON,
ruff JSON, govulncheck NDJSON, eslint JSON, tsc human errors, etc.) into
structured rows in the `findings` table.

### Parsers in v0.1

| Language    | Recipes parsed              | Tool output expected                                |
|-------------|------------------------------|------------------------------------------------------|
| rust        | audit, lint, test, build     | `cargo audit --json`, `cargo clippy --message-format=json`, `cargo test` (human) |
| python      | audit, lint, test, build     | `pip-audit -f json`, `ruff --output-format=json`, `mypy --output=json`, `pytest --tb=line` |
| go          | audit, lint, build, test     | `govulncheck -json`, `go vet -json`                  |
| typescript  | lint, build, test, audit     | `eslint -f json`, `tsc --noEmit` (stderr)            |
| javascript  | (alias of typescript)        | same                                                 |
| _other_     | (any)                        | falls back to `GenericParser` — emits one `recipe_fail` row when exit_code != 0, else nothing |

Resolution order in the registry: exact `language` match (parsers
self-gate on recipe via `Parser.matches`), then `GenericParser`. Adding a
new language is a single file in `crafting_table/parsers/` plus an entry
in `PARSERS`.

### Finding kinds

- `lint` — clippy / ruff / mypy / eslint / tsc / go vet diagnostic.
- `cve` — vulnerability from cargo-audit / pip-audit / govulncheck. Carries
  `code` = advisory id (RUSTSEC-..., GO-..., PYSEC-...) and a
  `suggested_fix` of the form "bump <pkg> to <version>" when patched
  versions are known.
- `test_fail` — failed test name extracted from `cargo test` /
  `pytest` output.
- `recipe_fail` — fallback when no language-specific parser fired and the
  recipe exited non-zero. `code` = `exit_<n>`, message names the recipe.

### Fingerprints + dedup

Every finding row carries a 16-char `fingerprint` hash over
`kind|file|line|code` (NOT the message — tool wording drifts). The same
lint reappearing across nightly runs produces the same fingerprint, so a
later wave can dedup digest output and surface only "new since last run."

### Consuming findings

```
GET /jobs/{job_id}/findings
  → {"ok": true, "findings": [
      {"id": 1, "job_id": "...", "kind": "lint", "severity": "warn",
       "file": "src/app.py", "line": 3, "code": "F401",
       "message": "...", "suggested_fix": "...", "fingerprint": "...",
       "raw_json": "{...}", "created_at": ...},
      ...
    ]}
```

Authorization is project-token-scoped (same model as `/jobs/{id}`). The
matching `job` row's `findings_count` mirrors the array length so
callers can decide whether to fetch the detail.

## Digest

Daily 06:00 PT email digest. One message per project per day; aggregates the
last 24h of jobs per recipient and sends via SMTP relay (Lucy postfix).

Set the SMTP block in `.env` to enable — leaving `CRAFTING_SMTP_HOST` unset
keeps the scheduler off and logs `digest disabled — CRAFTING_SMTP_HOST not set`
at startup. The `/digests` and `/admin/digest/run-now` endpoints still work
in dry-run mode regardless.

```bash
CRAFTING_SMTP_HOST=postfix.sulkta.com
CRAFTING_SMTP_PORT=587
CRAFTING_SMTP_USER=crafting-table@sulkta.com
CRAFTING_SMTP_PASS=...
CRAFTING_SMTP_FROM=crafting-table@sulkta.com
CRAFTING_SMTP_TLS=1
```

Each project's `notify.email` + `notify.on` fields control delivery:

| `notify.on` event   | When it fires                                          |
|---------------------|--------------------------------------------------------|
| `audit_pass`        | a passing audit job                                    |
| `audit_fail`        | a failing audit job                                    |
| `test_fail`         | a failing test job                                     |
| `lint_warn`         | a lint job with warning-severity findings              |
| `cve_found`         | any job whose findings include a `cve`                 |
| `patch_drafted`     | (wave 3 / step 9) auto-patch was drafted               |
| `nightly_summary`   | catch-all — show ALL jobs in the project's section     |

Empty `notify.on` defaults to `["audit_fail", "cve_found", "patch_drafted"]`.
Empty `notify.email` silently excludes the project.

Manual trigger from the LAN admin token:

```bash
# Render today's digest without sending
curl -sH "Authorization: Bearer $ADMIN" \
  -X POST http://192.168.0.5:8810/admin/digest/run-now \
  -d '{"dry_run": true}' | jq .

# Render an arbitrary date as JSON
curl -sH "Authorization: Bearer $ADMIN" \
  http://192.168.0.5:8810/digests/2026-04-29 | jq .
```

Idempotency: `digest_runs` table holds `UNIQUE(date, project_name)`, so the
06:00 loop is safe to re-fire on the same day — only the first call sends.

## Autonomous patch loop

Wave 3 wires crafting-table into clawdforge so a project with
`notify.auto_patch=true` gets an automatic patch attempt on every
actionable finding (lint with file/line; cve with a known fix). Lifecycle:

1. Runner finishes a job + parsers populate findings.
2. Post-job hook fires: pulls the highest-severity actionable finding,
   reads ±20 lines of context from the worktree.
3. Patcher opens a clawdforge session (`POST /sessions`), sends one
   turn with the finding + source context + project metadata, expects
   `{"diff": ..., "explanation": ..., "confidence": ...}` back.
4. Diff applied to a fresh worktree on `crafting-table/auto/<job_id>-<finding_id>`.
   Apply failure → status `apply_failed`.
5. Recipe re-runs against the patched worktree (the **verify** step).
   Fail → `verify_failed`.
6. Pass → commit + push + open Gitea PR. Status `pr_opened`.
7. clawdforge session always closed.

Configuration (env vars):

```
CRAFTING_CLAWDFORGE_URL=http://192.168.0.5:8800
CRAFTING_CLAWDFORGE_TOKEN=cf_...
CRAFTING_GITEA_URL=http://192.168.0.5:3001
CRAFTING_GITEA_TOKEN=<gitea PAT>
CRAFTING_PATCHER_MAX_ATTEMPTS=3
CRAFTING_PATCHER_BRANCH_PREFIX=crafting-table/auto/
```

If any of the four required vars is missing, the patcher stays disabled
and `POST /jobs/{id}/patches` returns 503. The runner hook silently no-ops
in that case so existing job flow is unaffected.

**Verification cost matters.** The verify step re-runs the failing recipe
on the patched worktree — for projects with multi-minute builds this
DOUBLES the latency. Set `notify.auto_patch=true` only for projects where
the audit/test recipe is <5min, OR accept the latency. v0.2 candidate:
"fast verify" mode that re-runs only the specific lint that fired.

`patch_attempts` table holds every attempt with `UNIQUE(finding_id, attempt_number)`;
the loop early-exits at `max_attempts_per_finding` (default 3). No
auto-merge; PRs land for human review.

Manual trigger:

```bash
curl -sH "Authorization: Bearer $TOKEN" \
  -X POST http://192.168.0.5:8810/jobs/$JOB/patches \
  -d '{"finding_id": 42}' | jq .
# → {"ok": true, "attempt": {"status": "pr_opened", "pr_url": "...", ...}}
```

## First production recipes

Three recipes ship in `examples/recipes/`:

| Recipe        | Subprojects | Schedule (audit) | auto_patch |
|---------------|-------------|------------------|------------|
| `clawdforge`  | 14 (one per SDK + root) | nightly 02:00 | **true** |
| `cauldron`    | 1 (Flask app, `.`)      | nightly 02:00 | **true** |
| `tradecraft`  | 1 (`.`)                 | nightly 02:00 | **false** (manual review) |

Each ships with a placeholder `REPLACE_WITH_GITEA_TOKEN` in `git_url`;
`examples/register-all.sh` substitutes `$GITEA_TOKEN` at register time so
no real token ever lands in the repo.

Smoke procedure (post-deploy):

```
1. docker compose up -d
2. TOKEN=$(cat /mnt/user/appdata/crafting-table/data/admin-bearer.txt)
3. CRAFTING_TABLE_TOKEN=$TOKEN GITEA_TOKEN=<your-pat> bash examples/register-all.sh
4. curl -H "Authorization: Bearer $TOKEN" http://192.168.0.5:8810/projects \
     → expect 3 projects (clawdforge, cauldron, tradecraft)
5. curl -X POST -H "Authorization: Bearer $TOKEN" \
     http://192.168.0.5:8810/projects/clawdforge/jobs \
     -d '{"recipe":"test","subproject":"clients/python"}'
     → expect job_id
6. Poll GET /jobs/{job_id} until status terminal → expect succeeded
```

Per-recipe smoke status (today, pre-deploy):

- `clawdforge` — 14 subprojects; `clients/python` & `clients/typescript`
  & `clients/go` & `clients/rust` known clean from existing CI; ruby /
  php / kotlin / java / csharp / swift compile-cleanly today but
  toolchain availability inside the crafting-table image is what step 1
  smoke verified. Bash subproject's `test/run.sh` may not exist (manual
  check needed post-deploy).
- `cauldron` — single Flask subproject; pip-audit & pytest known to run
  cleanly from the cauldron repo's own CI history.
- `tradecraft` — single subproject; auto_patch is **off** by design
  (production app, manual PR review only).

## MCP bridge

The `mcp/` subdirectory ships a self-contained `crafting-table-mcp` Python
package that exposes the HTTP API to MCP-aware clients (Claude Desktop, Claude
Code, Cursor, Zed, custom agents). See [mcp/README.md](mcp/README.md) for the
tool surface, installation, and configuration.

Quickstart:

```bash
pip install -e mcp
export CRAFTING_TABLE_BASE_URL=http://192.168.0.5:8810
export CRAFTING_TABLE_TOKEN=ct_...
crafting-table-mcp  # stdio JSON-RPC server
```

## License

MIT