Sign-out now lives in the account.sulkta.com bar (hits Authentik's
default-invalidation-flow). Cauldron already had no per-app sign-out
link in the nav, so this is just the bar embed.
Wires cauldron up to the unified Sulkta bug tracker per
memory/spec-bugs-unified-sdk.md (Phases 1-7 shipped 2026-05-02).
What's included:
- Vendored bugs-sulkta-py at cauldron/vendor/bugs_sulkta (4 stdlib-only
files copied verbatim from Sulkta-Coop/bugs-sulkta-py main). Same
vendoring approach as TC's backend/api/bugs_sulkta — Docker BuildKit
can't reach LAN Gitea, so the package ships in the source tree.
- BUGS_API_KEY + BUGS_BASE_URL env (config.py). Empty key = page renders
"not configured" placeholder; POSTs return 503. Lets dev runs skip
provisioning a key.
- New routes (server.py): GET /bugs (page), GET /api/bugs (list),
POST /api/bugs (create). All session-auth. Per-service key returns
every cauldron report; we filter client-side by user_email so each
household member sees only their own. Admins get a "show all" toggle.
- bugs.html template in mythic-witch style: subject + message + kind
+ severity form, filed-reports list with status glyphs (📂 open
🔨 in-progress ✅ resolved ⛔ wontfix), relative timestamps.
- _base.html nav: 🐞 bugs link between discover and me.
- Server-side auto-fill: user_email/user_name from session, page_url
from referrer, user_agent from request headers.
Defaults are dev-safe — no env change required for the LAN soak. When
Cobb mints the key with:
docker exec bugs-sulkta bugs-sulkta-cli keys create \
--service=cauldron --scopes=read,write,update \
--description="cauldron prod"
…drop it into BUGS_API_KEY and the page lights up.
Cobb requested all the small ones land before LAN testing.
discover.html CVE-NEW3-2 (LOW): switched recipe card image from a CSS
background-image:url('${_esc(url)}') to a plain <img class="img" src=...>
element. Recipe image_url is scraped from JSON-LD on third-party pages —
a malicious page could return an image_url crafted to close the CSS
url(...) string and inject layout-breaking CSS. With <img src=...> the
URL stays in HTML-attribute context end-to-end where _esc is sufficient.
Also adds defense-in-depth: validate URL parses as http(s) before
rendering, fall through to placeholder otherwise, and set
referrerpolicy=no-referrer so we don't leak our path to image hosts.
CSS for .dcard .img widened with object-fit:cover so img and div both
center/cover correctly.
server.py CVE-NEW3-3 (LOW): _origin_of() now lowercases scheme AND
host (urlparse only does scheme), and drops scheme-default ports
(:80/:443) so `https://x.com:443` matches `https://x.com`. Closes a
false-reject path on browsers that preserve case in Origin headers,
or non-canonical CAULDRON_BASE_URL values. Not a bypass — false-reject
robustness only — but cheap to fix and operationally important.
server.py CODE3-3 (LOW): _safe_next() now allows `%` in the path
charset so percent-encoded paths (e.g. /recipes/spaghetti%20bol)
don't silently land at /me. Defense-in-depth: also percent-decode
the path and reject if the decoded form contains `..` traversal or
`//` (encoded forms of the same patterns the front-of-function
reject already).
server.py INFO3-2: auth_callback now does session.clear() before
setting session["user"]. Capture+revalidate `next` BEFORE the clear
so we don't drop our own redirect target. Drops every pre-auth key
on login — defense-in-depth against session-state contamination if
anything else ever lands in pre-auth session.
.env.example INFO3-1: added CAULDRON_ADMIN_SUBS, CAULDRON_BASE_URL,
CAULDRON_BEHIND_TLS, CAULDRON_TRUSTED_PROXIES with comments
explaining what each one gates. Defaults are the safe-LAN set.
Holding for public deploy — Cobb running LAN tests for a few days.
INFO3-3 (rate limit) intentionally NOT addressed in code: the audit
notes this as architecturally a proxy-layer concern (rackham vhost),
not in-process. Rolled into the public-deploy commit when the vhost
work lands.
INFO3-4 (security primitive test coverage) deferred — separate test-
sweep PR, doesn't block deploy.
dedupe_recipes.py CODE3-1 (HIGH): mirror the consolidate apply path's
stale-404 tolerance into dedupe. Pair-based clustering can emit
overlapping pairs — (A,B) approved+deleted; later (A,C) tries to
delete A again. Mealie returns 404; the prior code logged this as
an error and bumped error_count for a desired-end-state-achieved
operation. Now treats 404 as already-handled and continues.
discover_recipes.py CVE-NEW3-1 (MED): added allow_redirects=False to
the fallback _rq.get(url, ...) call. is_public_url validates the
original host as public-IP-space, but requests' default redirect-
following would chase a 30x to 127.0.0.1 / 169.254.x — letting a
malicious recipe-page server redirect a scrape worker at internal
LAN services or cloud metadata. The recipe_scrapers primary path
has its own internal request chain that's a documented residual
(per is_public_url's docstring). Closes the easier of the two paths.
server.py CODE3-2 (MED): /api/discover/search ?limit / ?offset
swap raw int() for the existing _opt_int helper that's defined
20 lines up. Mirrors the wave-2 fix on /api/recipes and /me-recipes
that the prior pass installed everywhere except this endpoint.
3rd-pass audit (against HEAD 291fea0) verdict: codebase is in a
defensible production-deploy state. Remaining LOW/INFO items
(Origin RFC normalization, CSS-injection in discover image_url,
session.clear() on login, .env.example freshness) are robustness-
class rather than security-class and don't gate deploy.
server.py + config.py CVE-NEW-6 (MED): introduces CAULDRON_TRUSTED_PROXIES
CIDR-list env var and a _StripUntrustedForwardedHeaders WSGI middleware
that wraps the outside of ProxyFix. Behavior:
- empty list (default, dev/LAN) → ProxyFix is NOT enabled, AND
incoming X-Forwarded-* / Forwarded headers are stripped from
every request. Even on a LAN HTTP deploy nobody can spoof
request.is_secure / request.host downstream.
- non-empty list (e.g. 192.168.50.1/32 for rackham over WireGuard)
→ strip middleware drops X-Forwarded-* unless REMOTE_ADDR is in
a listed CIDR; ProxyFix then trusts the surviving headers.
Closes the audit's concern that the prior `if cfg.behind_tls: ProxyFix`
trusted X-Forwarded-* from any peer that could reach :7790 — sibling
containers on the sulkta docker network could spoof scheme/host since
gunicorn binds 0.0.0.0:7790 and Docker bridges resolve container DNS
internally. The trust anchor is now the peer IP, not just the hostname
the request claims to come from.
Decoupling trusted_proxies from behind_tls also handles deploy shapes
where TLS is terminated by something that doesn't forward X-Forwarded-*
(SSL passthrough, etc).
server.py CODE-4 (MED, doc-only): added docstring on
current_household_id() declaring it the canonical hid source for
session-auth routes. Admin-bearer endpoints legitimately derive hid
from started_by_sub (the bearer is the trust anchor); session-auth
endpoints must never accept hid from request body. No code change —
the current code already follows this convention; the docstring
prevents future drift.
enrich_recipes.py CODE-5 (MED): added an explicit progress heartbeat
between forge.enrich_recipe() and forge.verify_allergens() so a
slow allergen-verification pass on a complex recipe can't push
last_progress_at past db.fail_stuck_enrich_jobs's 15-min stale gate.
Without this, two ~3-4-min Sonnet calls back-to-back could straddle
the gate and a still-alive job would be incorrectly reaped at the
next worker restart.
Continuing through the 2nd-pass audit findings.
dedupe_recipes.py CODE-2 (HIGH): _cluster_by_name dropped union-find
single-link agglomerative for the same pair-accumulator pattern
consolidate_foods._cluster already uses. Single-link chained weak
similarities through the recipe corpus the same way it did with foods,
producing one giant cluster on a 250+ corpus that Sonnet would refuse.
Now emits one 2-recipe pair per (i,j) above NAME_THRESHOLD.
consolidate_foods.py + dedupe_recipes.py CODE-1 (HIGH): added
in-loop cancel-poll to both _cluster passes. Polls cancel_check every
5K pair-comparisons so a user-initiated cancel can abort cleanly mid-
scan instead of waiting tens of seconds. Run-walk callers also
re-check _cancelled() right after clustering returns and bail.
server.py CVE-NEW-5 (MED): /login skips OIDC re-init when
session.get('user') exists. Closes the login-CSRF surface where a
malicious cross-origin link `<a href=…/login?next=/poison>` would
re-trigger the OIDC handshake on a logged-in user and silently change
their post-login landing.
server.py CVE-NEW-7 (LOW): Permissions-Policy now sends both
`interest-cohort=()` (FLoC, Chrome ≤94) and `browsing-topics=()`
(Topics API, Chrome ≥115) for opt-out across the lineage.
server.py CVE-NEW-8 (LOW): /auth/callback OAuthError branch no longer
echoes Authentik's raw error string to auth_retry.html. Detail is
logged server-side; users see a generic retry message. Closes the
information-disclosure on Authentik error codes.
server.py CODE-9/CODE-10 (LOW): wrapped int() of ?page= and
?per_page= in try/except so garbage args land on safe defaults
instead of surfacing ValueError as a 500.
Deferred to the rackham-vhost commit:
- CVE-NEW-6 (cauldron bind / ProxyFix trusted-peer filter) needs
paired Apache vhost config (RequestHeader unset X-Forwarded-*)
before the docker-side change is safe; landing it solo would
either break the LAN deploy or leave a half-broken trust chain.
Both findings are net-new — first audit ran before the CSRF guard existed,
and the picks→onclick interpolation predates the discover/scrape work that
expanded slug shape. Verified by grep+read.
server.py CVE-NEW-1 (CRIT): @before_request CSRF guard used
`origin.startswith(cfg.base_url)`. With CAULDRON_BASE_URL=https://cauldron.sulkta.com,
an attacker-registered `cauldron.sulkta.com.evil.com` produces an
Origin header that startswith the configured base — guard passes,
A1 fix trivially defeated. Replaced with parsed-origin equality
(scheme+netloc, byte-exact at the netloc boundary). Pre-parse the
expected origin once at app boot rather than per-request.
server.py + picks.html CVE-NEW-2 (CRIT): /api/picks/<slug> took
slug from URL path with NO validation against the household's
recipe index. picks.html then interpolated slug into
`onclick="removePick('{{ slug }}', this)"` — Jinja escapes `'` to
`'` but HTML attribute decoding returns bare `'` to the JS
engine, so a slug like `x'),alert(1);//` round-trips DB → template
→ JS execution for every household member who loads /picks. Two-
layer fix:
- add_pick now requires slug ∈ db.find_indexed_recipe(hid, slug),
returning 404 on miss. Also closes prompt-injection-via-poison-
slug into the planner. Indexed name is trusted over client-
supplied name (defense in depth on the name field too).
- picks.html switches to a delegated click listener reading
slug from the parent <li>'s data-slug attribute. Slug never
lands inside a JS string literal in HTML.
server.py CVE-NEW-3 (HIGH): _safe_next() helper centralizes the
post-login redirect validation. Applied at BOTH /login stash time
AND /auth/callback consumption time so a future writer of
session['post_login_next'] can't bypass. Strict path charset
[A-Za-z0-9_./-], rejects scheme/netloc, rejects `//`, `/\`.
Captured by 2026-05-02 CVE audit (memory/cauldron-cve-audit.md). Code-only
change; deploy is gated until current discover scrape pipeline drains.
config.py: add CAULDRON_BASE_URL + CAULDRON_BEHIND_TLS env-driven fields.
server.py:
- SESSION_COOKIE_SECURE flips on when behind_tls (CVE-D1)
- PERMANENT_SESSION_LIFETIME=14d + session.permanent=True at login,
SESSION_REFRESH_EACH_REQUEST sliding expiry (CVE-D2)
- MAX_CONTENT_LENGTH=1 MiB request-body cap (CVE-G1)
- ProxyFix wrapped when behind_tls so request.is_secure / .host reflect
rackham's TLS endpoint, not the OpenVPN-internal :7790 (CVE-J6)
- @before_request CSRF Origin/Referer guard, exempts GET-class methods
and Bearer-auth callers, only enforced when CAULDRON_BASE_URL is set
so LAN dev still works (CVE-A1)
- @after_request security headers: X-Frame-Options DENY, nosniff,
Referrer-Policy same-origin, Permissions-Policy interest-cohort=(),
CSP (self + inline for now), HSTS only when behind_tls (CVE-E1)
- /healthz trimmed to public {"ok": bool} liveness probe; detailed
upstream status moved to bearer-gated /api/admin/healthz so error
strings don't leak internal LAN topology (CVE-E3)
- _const_eq replaced with hmac.compare_digest — drops the early-return
length check that itself was a side channel for probing bearer
length (CVE-A3)
Apache vhost + actual public deploy still gated. Defaults are dev-safe:
no env vars set → no behavior change vs. current LAN HTTP deploy.
Same recipe URL with vs without trailing slash was producing duplicate
discover corpus rows because UNIQUE(source_url) is byte-exact:
https://www.tasteofhome.com/recipes/falafel → id 7
https://www.tasteofhome.com/recipes/falafel/ → id 3 (manually pasted)
Caught 2026-05-02 when Cobb pasted his first 4 with trailing slashes,
then a follow-up listing-page extractor stripped them, producing 1:1
dupes. rstrip('/') in insert_discovered_recipe normalizes at the
persistence layer so all callers get the dedup for free.
Existing data manually fixed: deleted dupes 7,8,5; stripped trailing
slashes off rows 3,4,6 to canonical form. Corpus now clean (4 rows).
Three audit-day UX fixes after Cobb's big-screen review:
WIDE SCREEN
- main max-width 920px → 1500px so the layout actually uses 4K real estate
- recipe-grid gets a 4-column breakpoint at 1400px (was 3 at 1100px)
- discover already auto-fills via repeat(auto-fill, minmax(280px, 1fr)) —
spreads naturally on wider viewports
RECIPE PHOTOS on /recipes
- _index_row_to_card now emits image_url derived from
raw.id + raw.image + cfg.mealie_public_url, pointing at Mealie's
/api/media/recipes/{id}/images/min-original.webp endpoint
- raw.image (which Mealie bumps on every update) is appended as
?v=<image-stamp> for cache-busting
- new .recipe-card .rimg style: 16:10 aspect ratio, object-fit cover,
placeholder 🍴 fallback when no image
- _recipe_card.html (server-rendered first page) and recipes.html
(AJAX-rendered subsequent pages) both render thumbnails consistently
ADMIN-ONLY VISIBILITY (Cobb 2026-05-02)
- new CAULDRON_ADMIN_SUBS env var → cfg.admin_subs (CSV of authentik
subjects). Empty default = nobody is admin (safe-fail).
- @app.context_processor injects is_admin globally for templates
- _base.html nav: /discover tab gated by {% if is_admin %}
- me.html: consolidate + discover tool blocks gated by {% if is_admin %}.
Sterilize / dedupe / enrich stay visible to everyone (Cobb's scope was
consolidate + discover only)
- new @require_admin decorator (returns 404, not 403, to not advertise
the route's existence) applied to all 13 consolidate/discover routes:
pages + api endpoints. URL-typing non-admins now blocked, not just
hidden in UI
Tested AST. Deploy: cauldron uses up -d --build (no source bind mount).
CVE audit (memory/cauldron-cve/00-deps.md, 2026-05-02): 13 known CVEs
across these four packages in the deployed versions. Verified each
against cauldron's actual code path — most not directly exploitable in
current usage (no JWE decrypt, no key=None JWS, no EC crypto, no
.netrc, sessions in Flask not Authlib cache). The 9.1-CVSS Authlib
JWS bypass (CVE-2026-27962) requires a code path cauldron doesn't
take, but the library is 8+ versions stale and the bump is mandatory
before any public exposure.
Authlib jumps the most (1.3.2 → 1.6.11). High-level OAuth/OIDC API
is stable across this range — OAuth(app), register(...),
authorize_access_token(), userinfo() all unchanged. Smoke-test the
OIDC round-trip after deploy.
Six audit-driven fixes from 2026-05-02 punch list at memory/cauldron-codebase-audit.md.
CRITICAL
- F-1 routes: SSRF guard on /api/discover/scrape-start. Every URL is
validated via discover_recipes.is_public_url() — parses host, rejects
IP literals in private/loopback/link-local/multicast/reserved ranges,
resolves hostnames via getaddrinfo and rejects if any A/AAAA is private.
Defense-in-depth: _scrape_one re-validates before fetch in case any
future caller bypasses the route. Rejected URLs are returned in the
response payload so the user knows which were skipped.
- F-6 domain: prompt-injection mitigation on enrich_recipe + verify_allergens.
New apply_allergen_safety_override() in forge.py runs regex pattern-
matching against the raw ingredient text for the SIX anaphylaxis-class
allergens (peanuts, nuts, shellfish, fish, eggs, sesame, dairy). On
match, force contains.<allergen>=TRUE regardless of Sonnet output. False
positives are recoverable; undetected anaphylaxis is not. Pork/soy/
gluten not auto-overridden (religious/dietary or too-common).
HIGH
- F-2 routes: /api/discover/reject swapped from global status flip to
per-household scope. New migration 039 cauldron_discover_skips
(discover_id, household_id, skipped_by_sub, skipped_at) join table.
list_discovered_recipes default view filters out caller-household
skips; ?status=skipped surfaces them for unskip. Different households
have different tastes.
- F-3a routes: /login?next= same-origin validation. Reject anything that
doesn't start with `/`, AND reject `//evil.example` protocol-relative
redirects. One-line fix.
- F-10 domain: Sterilizer.apply_recipe ingredient-count guard. Refuse to
apply if Mealie's current recipeIngredient length differs from the
preview's proposals length. Python's zip would silently truncate;
user edits made during the 60-300s Sonnet window now raise
RuntimeError instead of getting clobbered. Bulk runner already catches
RuntimeError per-recipe, marks proposal stale.
- F-15 domain: aggregator qty=None safety net. Ingredients with no
quantity now go to a separate no_qty_items list instead of being
silently coerced to 0.0 (which then failed the `any(qty for ...)`
truthiness check and dropped the food off the shopping list). If no
other line was emitted, write a "qty unspecified" placeholder so the
food APPEARS on the list. If a sized line WAS emitted, append a
"+ N ingredient(s) with no quantity" note.
ALSO (one-liners called out in the punch list)
- Migration 029 DROP INDEX gets IF EXISTS — prevents boot-brick on
partial-failure retry.
- Flavor B prefix prompt rule — Sonnet now told to keep `lib:`/`disc:`
prefix verbatim; prevents intermittent 502s on the panel just shipped.
- list_discover_eligible_for_group switched from LEFT JOIN to NOT EXISTS
subqueries — fixes F-5 data (LIMIT-shrink from cross-group import
multiplication) and adds the per-household skip filter cleanly.
All edits AST-verified. Allergen regex tested with peanut/fish/clean
inputs — flips correctly, preserves Sonnet TRUEs, no over-broad coverage.
Mediums + lows from the audit are tracked in
memory/cauldron-codebase-audit.md and deferred until Cobb hits them
during dogfood.
The suggestion panel now folds enriched-but-unimported Discover entries
into the same Hecate-driven pool as library recipes. Pinning a discover
suggestion auto-imports it to the household's Mealie library and adds
the resulting slug to cauldron_meal_picks.
- db: list_discover_eligible_for_group(mealie_group_id, limit=80) — joins
cauldron_discovered_recipes against cauldron_discover_imports to find
enriched rows no household in the caller's group has imported yet
- forge.suggest_recipes: accepts a per-entry `source` field; when any
entry is source='discover' the prompt gains an explicit hint so Hecate
treats discover entries as "something new to try" with a soft cap on
proportion (~half max unless library exhausted)
- /api/plan/suggest: builds a unified pool with prefixed IDs (lib:<slug>
vs disc:<id>) so library and discover entries can coexist in Sonnet's
validation map; decorates each suggestion with kind+image+meta_summary+
hecate_quip; discover entries also carry source_url and source_host
- /api/plan/suggest/pin: new dispatch on body.kind:
- kind=library (default — back-compat with existing Flavor A callers):
same as before
- kind=discover: looks up the discover row, short-circuits to cached
Mealie slug if THIS household already imported it, else
mealie.import_from_url() + record_discover_import() + add to picks.
Returns {kind, mealie_slug, imported, added_to_picks}
- plan.html: card render is kind-aware. Discover entries get a
"📬 from Discover · <host>" footer instead of last-planned recall, an
"import + pin" button label, and use data-* attributes for the click
payload so the JS doesn't need to template-interpolate slugs into
onclick handlers
current_household_id() short-circuited on member-row presence and never
called sync, so the mealie_group_id column added by migration 035 stayed
NULL on existing households. Now: if the row is missing group_id,
re-run sync_user_household once to backfill from who_am_i. After backfill
the check is a cheap row read.
Audit-driven cleanup of the multi-household design:
- migration 035: cauldron_households.mealie_group_id (was missing — blocked
any cross-household reasoning)
- migration 036: cauldron_recipe_index.mealie_group_id (additive; current
per-household keying preserved, the column lets future code key by group
without a destructive PK rebuild)
- migration 037: drop dead cauldron_food_mapping (replaced by
cauldron_food_metadata in foundation reset Step 2 / commit f74a627
2026-04-30; zero callers in the codebase)
- migration 038: cauldron_discover_imports(discover_id, household_id,
mealie_slug, imported_by_sub, imported_at) — per-household provenance.
Replaces the global cauldron_discovered_recipes.status='imported' flag
that incorrectly hid the import button from every user once anyone
imported.
Code:
- sync_user_household reads who_am_i()["groupId"] (and nested group.id)
and persists it on cauldron_households.mealie_group_id
- upsert_household accepts mealie_group_id with COALESCE-on-update
- new helpers: record_discover_import, get_discover_imports_for_group,
discover_imported_by_household, get_household
- /api/discover/import: per-household idempotency (returns cached slug if
this household already imported), records to the new join table; no
longer flips global status='imported'
- /api/discover/search: decorates each row with imported_in_my_group +
imported_by_my_household + mealie_slug + imported_by_household_name
- discover.html: card render uses imported_in_my_group; shows
"✓ in your library as <slug>" when this household imported, or
"✓ shared from <Other Household> as <slug>" when another household in
the group imported (recipe is group-shared via Mealie's group-scope
read so re-importing would create a duplicate)
The imported badge now correctly surfaces "imported by another household
in your group" rather than hiding the row entirely.
scrape_me(url) no longer accepts wild_mode in 15.x. Two-stage fallback:
1. scrape_me(url) — site-specific scraper (best quality)
2. fetch html + scrape_html(html, org_url, supported_only=False) —
generic JSON-LD/microdata pass for unsupported sites
Caught at first dogfood test against allrecipes; previous code raised
"unexpected keyword argument 'wild_mode'" before issuing any HTTP.
Append-only schema for Discover v0.1. Two tables:
cauldron_discovered_recipes — corpus of recipes scraped from external
sources (allrecipes, BBC Good Food, smitten kitchen, etc) before any
household has imported them. status walks raw → enriched → imported
| rejected. source_url is capped at 768 chars to stay under InnoDB's
3072-byte unique-key ceiling (768 × 4-byte utf8mb4 = 3072). FULLTEXT
on (name, description) for the discover-page search bar.
cauldron_discover_jobs — daemon state for the scrape runner, same
pattern as sterilize/enrich/foods-consolidate (state-poll cancel,
last_progress_at watchdog, terminal-only finalize).
Migrations only — no Python helpers, no scraper, no UI yet. Lands when
the next cauldron container restart picks them up.
Hecate looks through the household's library and surfaces 1-6 recipes
that fit the picker profiles but haven't been served in 90+ days (or
ever). Server-side filters drop recently-planned + already-on-this-week
+ already-on-picks-list + allergen-conflict before Sonnet sees the pool,
keeping the prompt focused. Each suggestion comes with a 1-5 fit score
and a one-line reason in Hecate's voice. Pin → adds to the user's picks
so the next /plan/generate will naturally pull it in. Skip → hides the
card.
Endpoints:
POST /api/plan/suggest body {count?: 1-6, week?: ISO}
POST /api/plan/suggest/pin body {recipe_slug}
Layered on top of 37d7d60 (parser fix, committed not deployed). Both
should land together once enrich job 3 drains.
appends extra prose
Job 3 hit one recipe (healthy-chicken-stir-fry-with-vegetables) where
Sonnet returned a valid JSON object + appended prose afterward, in
violation of 'no prose' rule. json.loads choked with 'Extra data:
line 54'. _parse_json_blob now falls back to JSONDecoder.raw_decode
which extracts the first complete JSON value and ignores anything
after — any trailing notes / fence remnants / inline commentary get
silently dropped.
Plain json.loads is still tried first (fastest path on clean output).
The fallback only kicks in for malformed-but-recoverable cases.
hecate's quip + macro confidence + chain-of-thought macro estimation
Eight new fields packed in before Cobb's force re-enrich rerun:
(1) active_minutes + hands_off_minutes — split estimated_minutes into
"you must be present" vs "rise/marinade/braise unattended". Lets
weeknight planning match real available active time. A 4-hour beef
stew might be 30 active + 210 hands-off — fine for Sunday but
misleading as "240 minutes" on a busy weeknight.
(2) equipment[] — required appliances/modes from a fixed enum:
oven, stovetop, grill, instant-pot, slow-cooker, sheet-pan,
cast-iron, air-fryer, food-processor, blender, mixer, smoker,
sous-vide, no-cook. Foundation for "no oven this week" filters.
(3) flavor_profile[] — 2-5 dominant flavor notes (spicy, sweet,
savory, umami, tangy, smoky, herby, citrusy, rich, fresh, bitter,
earthy). Lets the planner enforce variety so we don't get five
spicy nights in a row.
(4) kid_friendly_score 1-5 — separate from comfort_tier. Cobb has
Leia and Luna; this is real signal. 1=adults-only (capers, blue
cheese, super spicy), 5=kids beg for it (mac and cheese, pancakes).
(5) fiber_g + sodium_mg per serving — extends the macro coverage.
Sodium for heart-health weeks, fiber for gut weeks.
(6) cost_per_serving_estimate — rough USD per serving (2026 prices).
Bean bowl $2, salmon for two $8-12, fancy roast $15+. Foundation
for budget-week preference. Set null for too-volatile items.
(7) occasion_fit[] — when is this dish AT HOME? weeknight, weekend,
brunch, date-night, party, picnic, camping, holiday, game-day,
kids-birthday, quiet-night-in.
(8) hecate_quip — one-line voice description in Hecate's mythic-witch
tone. Pure flavor for tooltips/detail pages. ~10-20 words. Examples:
"Pure midwinter comfort — asks for a fire and quiet."
"Sharp and bright like a Tuesday morning resolution."
MACRO QUALITY (Cobb's separate ask):
- Prompt now instructs Sonnet to compute macros chain-of-thought:
list each major ingredient, approximate its per-serving contribution
in grams, sum, output. Plus a sanity check: protein×4 + carbs×4 +
fat×9 ≈ calories. Not free precision but better than guess-the-total.
- New macros_confidence field (low/medium/high) — Sonnet rates own
certainty so users know when to trust the numbers and the planner
can avoid summing low-confidence macros over a 7-day budget.
- Future commit: deterministic USDA-FDC backfill into cauldron_food_
metadata for true ingredient-by-ingredient sums. Same shape as the
existing density backfill we did in Step 2.
Plan generator pool prompt expanded — every recipe entry now carries
inline:
active=Nm+offhand=Mm
eq:oven,stovetop
flavor:savory/herby/rich
kid=4
~$5/svg
for:weeknight/date-night
fiber=Ng / sodium=Nmg
ENRICH_VERSION 4→5. Cobb's force re-enrich gets all of these on the
existing 222 recipes in one walk.
Hecate's weekly reading + 2nd-pass allergen verification
Five new context dimensions for the planner agent (and one quality
fix), bumping ENRICH_VERSION 3→4 so existing meta gets refreshed on
the next walk:
(A) Per-user fit score. Computed at plan-gen time from picker_profiles
× meta — no extra Sonnet calls. _compute_fit_score scores each
recipe 1-5 per household member based on cuisine match, protein
match, comfort_tier match, and tag overlap with that user's
historical picks. Renders in the recipe pool prompt as
'fit:cobb=5,abby=2,leia=3'. Gives Sonnet per-user signal to
bias AI-chosen slots toward whoever's home that week, AND makes
pick rationale explainable.
(B) Pairings per recipe. New meta fields:
pairings.serves_well_with: ["crusty bread","green salad",...]
pairings.drinks: ["pinot noir","iced tea",...]
Sonnet enriches both during the main pass. Foundation for future
"auto-suggest matching side" UX in multi-meal slots.
(C) Leftover potential 1-5. Per-recipe score: 1=eat-now-only (crispy
things, fresh salads), 5=actually BETTER as leftovers (stews,
braises, lasagna). Lets the planner thread Sunday's slow-cooked
pot roast into Monday's lunch slots intentionally.
(D) Mood scores. Per-recipe 1-5 ratings on:
cozy / summer_fresh / energizing / comfort
Independent dimensions — a recipe can score high on multiple.
Foundation for future weather/mood-aware planning ("rainy week
→ bias cozy>=4").
(E) Hecate's weekly reading. New TEXT column hecate_reading on
cauldron_meal_plans (migration 032). The plan-gen prompt asks
Hecate to write a 1-paragraph narrative voice description of the
week — "this week leans into the brisk turn of the season, three
hearty one-pots front-load your weekday energy, salmon Wednesday
for Cobb's gym push..." Confident, wise, theatrical voice. Pure
flavor, no functional impact, but makes Hecate FEEL like an
advisor not an opaque function. Plus the planner system prompt
now opens with the Hecate persona ("You are Hecate, Greek-
mythology witch goddess of crossroads, herbs, and magic — and
the family's meal planner"). Output schema gains a "reading"
field alongside "slots".
ALLERGEN VERIFICATION (the quality fix Cobb explicitly asked for):
- Sonnet sometimes flags pork=true on a sweet potato recipe via the
conservative-default rule. Cobb wants CLEAN data.
- New forge.verify_allergens — second Sonnet pass after main enrich
with a strict prompt: "name the SPECIFIC ingredient triggering each
allergen flag, or set FALSE." For non-anaphylaxis exclusions like
pork, set FALSE unless an actual pork ingredient is named. For
ANAPHYLAXIS allergens, conservative TRUE still applies.
- Cost: ~3s/recipe extra. Wired into enrich_recipes.run_enrich after
initial enrich; failure is non-fatal (falls through to original).
- Eliminates the pork-on-sweet-potatoes class of false positives.
Code restructure: forge.generate_plan now returns {slots, reading}
instead of just slots. _extract_plan_payload pulls both. Server-side
generate + regenerate paths unwrap and persist reading via the new
db.set_plan_hecate_reading. Plan template renders the reading in a
purple-bordered serif callout above the day grid.
Schema:
- migration 032 adds hecate_reading TEXT to cauldron_meal_plans.
- Cauldron_recipe_meta gets new fields persisted in meta_json (no
schema change there — JSON column already accommodates).
Greek goddess of magic, crossroads (decisions/planning), and herbs.
Three-for-three thematic fit for a meal planner. Internal field names
(sonnet_decision etc) unchanged — those are data, not user-facing.
Big plan-generator addition: each plan can now span multiple meal types,
not just dinner. Default stays dinner-only for back-compat; opt-in via
checkboxes on /plan.
Schema (migrations 028-031):
- cauldron_meal_plan_slots gains meal_type ENUM('breakfast','lunch',
'dinner','snack','dessert','side') NOT NULL DEFAULT 'dinner'.
- Old UNIQUE key (plan_id, day) → (plan_id, day, meal_type) so a
Monday can have breakfast AND lunch AND dinner slots.
- cauldron_meal_plans gains meal_types_json (which meals to plan
for that week — list of strings, defaults to ["dinner"]).
Forge:
- generate_plan accepts meal_types list. Output schema gains meal_type
per slot. Validates expected_total = slots * len(meal_types) and
rejects duplicate (day, meal_type) pairs.
- _build_plan_prompt renders MEAL TYPES TO PLAN block, instructing
Sonnet to match recipe meta.meal_type to slot type (breakfast slot
→ recipe whose meta tags it as breakfast). Falls back gracefully
when the pool is thin for a particular meal type.
Server:
- /api/plan/generate + regenerate accept body.meal_types, persist via
db.set_plan_meal_types.
- plan_view decodes meal_types_json into plan["meal_types_list"] and
builds plan["meal_types_label"] for the readout.
UI (/plan):
- New checkbox row at the top of the pref-block: 🍳 breakfast / 🥪 lunch
/ 🍽️ dinner. Defaults to whatever's persisted (or just dinner).
- Day cards now group multiple meal_type slots per day with small
meal-type tags above each recipe row. Single-meal plans render the
same way they always did (no tag shown when only one meal_type).
- readMealTypes() in JS reads checkboxes and ships in the body.
DB:
- save_plan_slots accepts meal_type per slot, defaults to 'dinner'.
- list_plan_slots orders by day then meal_type via MEAL_ORDER.
==
UX rename: "claude" / "sonnet" → "Sage" across all user-visible copy.
Sage doubles as kitchen-herb (theme fit) and wise advisor (planner
role). The internal field name `sonnet_decision` on consolidate +
dedupe proposals is unchanged (it's a data field, not user-facing).
Renames touched plan, consolidate, dedupe_recipes, list, me,
enrich_recipes, sterilize templates. Cobb can swap to Mim or his own
name later — easy global s/sage/whatever/g.
==
The /list 'clear' button removed earlier today (b4cb48b) — not
re-introduced.
(1) /list had a 'clear' button that just unchecked all the localStorage
shopping checkboxes. Cobb correctly flagged it as confusing — looks
like a destructive action ('clear what?'). It was just a UX nicety
for re-using the same shopping list, not a plan-touching operation.
The plan-level reset already lives on /plan via a properly-scoped
button. Removing the /list 'clear' so there's exactly one reset
concept in the app: /plan → ⌫ reset week.
(2) forge.enrich_recipe was hitting clawdforge with timeout_secs=90
which proved too short for one recipe (peppery-barbecue-glazed-shrimp
type long ingredient list). Bumped to 180s to match the sterilizer's
post-fix timeout. Walks now have more slack before timing out a single
recipe.
Three connected features so the planner is a real tool, not just a
single-week dashboard:
1. RESET. New POST /api/plan/reset wipes a plan back to blank — clears
slots + generated_at + preference_prompt + daily_targets_json +
exclusions_json. Hard-guards on lock state: locked plans return 409
"locked weeks are immutable history". This is the historical-
preservation guarantee — once a week is locked (via /api/plan/lock
OR auto-lock at week-rollover), it can never be touched again.
That's how 'what did we eat May 15 six years ago' stays answerable
forever.
2. WEEK NAVIGATION. /plan now accepts ?week=YYYY-MM-DD (snapped to
that week's Monday). Defaults to the current week as before. New
page-head nav: ← prev week / ⊕ this week (only when off-current) /
next week → / week-of-X span label. /list also accepts ?week= so
the shopping list view follows the same pattern.
3. HISTORICAL BROWSING. Past weeks render their plan slots as before —
just locked + immutable. Their preference_prompt + macros +
exclusions render in the readouts so you can see WHY that week
looked the way it did. The data was already preserved; this just
surfaces it through the existing /plan UI.
API + template changes:
- _resolve_week helper picks target Monday from body['week'] or today
- /api/plan/{lock,generate,regenerate,reset} all accept body['week']
- Plan view passes prev_week/next_week/current_week + is_*_week flags
+ week_end (Monday + 6) for the date-range label
- Template: PLAN_WEEK js constant threads the active week into every
mutation API call so prev/next nav can act on the displayed week,
not always today's
- Reset button styled red ('btn-danger'), only shown on unlocked
generated plans, confirms before firing
- 'view list →' link now passes ?week= so it stays in-week
DB:
- reset_plan(plan_id) wipes UNLOCKED plan state in one transaction.
Returns False (no-op) if the plan is locked — caller sees 409.
No schema changes — just leverages the per-week (household, week_start)
row uniqueness we already had.
The next-tier of structured planning constraints, building on the
free-form preference textarea. Now the household can set hard rules
like '2200 cal/day, 150g protein, no dairy or shellfish' and the
planner enforces them.
Schema (migrations 026 + 027):
- daily_targets_json on cauldron_meal_plans: {calories?, protein_g?,
carbs_g?, fat_g?} per-day budget. Sonnet sums per-recipe macros
across the 7-day plan and aims for ±15% of the weekly total.
- exclusions_json on cauldron_meal_plans: list of contains.* keys
(subset of {dairy, gluten, nuts, peanuts, eggs, shellfish, fish,
soy, sesame, pork}). Hard filter on AI-chosen slots; picks still
apply but get flagged with the conflict in their reason field.
DB:
- set_plan_targets_and_exclusions normalizes inputs (drops zeros,
validates against allowed enum values, lowercases).
- The plan_view path decodes the JSON columns server-side and adds
display labels (`targets_label`, `exclusions_label`) and the parsed
dict/list (`targets_dict`, `exclusions_list`) for the template.
Forge:
- generate_plan accepts daily_targets dict + exclusions list.
- New DAILY MACRO TARGETS prompt block — instructs Sonnet to sum
per-serving macros across all 7 slots and land within ±15% of
weekly totals; tradeoff slots when needed.
- New STRICT EXCLUSIONS prompt block — recipes whose has: list
intersects exclusions are forbidden in AI-chosen slots. Picks that
conflict still appear (explicit user choice) but get flagged in
the slot reason ("contains dairy — household pick").
Server:
- /api/plan/generate accepts {preference, targets, exclusions}.
Persists all three before kicking off Sonnet so re-rolls reuse them.
- /api/plan/regenerate same — empty body reuses persisted constraints.
UI (/plan):
- New <details> section "numeric targets + allergen exclusions
(optional)" tucked under the existing vibe textarea + presets.
- 4 numeric inputs (cal/protein/carbs/fat per day) with quick-set
preset chips: balanced 2200 / protein lean 2400 / carb load 2600 /
cut 1800 / clear.
- 10 allergen checkbox chips with red-pill styling when checked
(uses CSS :has(input:checked) — modern browsers).
- Hydration: persisted targets pre-fill the inputs, persisted
exclusions pre-check the boxes.
- After generation, three readouts above the action buttons show
active vibe / macros / exclusions.
Cobb's exact ask "2200 cal/day, protein and carb balanced, no dairy"
now maps to: targets={calories: 2200, protein_g: 150, carbs_g: 250}
+ exclusions=[dairy] + free-form preference (optional). All persist.
Re-rolls iterate within those constraints automatically.
Tier-1 data additions for the planner — turning the AI from a title-
matching guesser into a structured-data consumer. ENRICH_VERSION bumped
2→3 so existing meta gets refreshed with the new fields on next walk.
(A) Cook history. db.household_recipe_history aggregates recipe slug
→ {last_planned_date, count_30d, count_long} from cauldron_meal_
plan_slots over a 180-day window. The plan generator's pool prompt
now renders each recipe with rotation context: "last:8w-ago 0×/30d
1×/180d". New planner rule: ROTATION — demote recipes shown 2+
times in 30d unless they're picks; never repeat the same slug
within the 7-day plan. New planner rule: VARIETY — don't fill 5
of 7 slots with the same primary_protein or cuisine.
(B) Per-serving macros in enrichment. forge.enrich_recipe now asks
Sonnet for calories, protein_g, carbs_g, fat_g per serving (rough
USDA-grade estimates from ingredient list + yields). Renders into
the pool prompt as "~480cal protein=32g carbs=45g fat=18g". Lets
"high protein week" become a quantitative filter instead of a
title-keyword match.
(C) Allergen booleans. New contains.* block in enrichment:
{dairy, gluten, nuts, peanuts, eggs, shellfish, fish, soy, sesame,
pork} — bool per allergen, conservatively defaulting to TRUE when
uncertain since false negatives can hurt people. Pool prompt
renders as "has:dairy,gluten,eggs". Foundation for upcoming
"no dairy this week" exclusion-list UI on /plan.
(D) Picker profiles. db.household_picker_profiles unions current
cauldron_meal_picks + historical meal_plan_slots.picker_subs over
365 days, joins with cauldron_recipe_meta, aggregates per-user:
{display_name, total_picks, cuisines, proteins, comfort_tiers,
tags} — top-N counters each. Plan generator includes a new
PICKER PROFILES block in the prompt:
- cobb (sub=cobb@sulkta.com, 24 picks):
cuisines=[asian:6, mexican:4, italian:3] ·
proteins=[chicken:8, beef:5, fish:2] ·
tags=[weeknight:11, high-protein:9, spicy:7]
Sonnet uses these to bias AI-chosen slots toward each member's
actual demonstrated taste — golden signal that's been sitting in
the database the whole time. Picks still override profile bias.
Cost: cook history is a single SQL aggregate (free, sub-100ms). New
macro+allergen fields fold into the existing ~5s/recipe Sonnet call
with maybe 30 more output tokens. Picker profiles are 2-3 SQL queries
totaling sub-200ms even at scale. No new network round-trips.
Net effect once Cobb runs /enrich-recipes against ENRICH_VERSION 3:
plan generator has structured macros + allergen flags + cook-history
rotation context + per-user preferences to work with. The free-form
preference textarea ("high protein, no dairy") becomes a real query
against actual data, not just a Sonnet vibe-prompt.
The 'fancy data fun' Cobb wanted: pre-compute structured metadata for
every recipe so the plan generator can match preferences to actual
recipe characteristics, not just match keywords on names.
Sonnet returns per recipe:
- tags[]: curated descriptors (high-protein, weeknight, one-pan,
leftovers-good, kid-friendly, etc — picks 3-8 that genuinely apply)
- cuisine, complexity (easy/medium/involved), estimated_minutes
- meal_type (breakfast/lunch/dinner/snack/dessert/side/sauce/drink)
- primary_protein (chicken/beef/pork/fish/seafood/tofu/...)
- primary_carb (rice/pasta/bread/potato/tortilla/quinoa/...)
- veg_forward (veg-forward/mixed/meat-forward)
- comfort_tier (weeknight-easy/hearty-comfort/fancy-occasion/...)
- season_fit[] + summary one-liner + best_for short phrase
Schema:
- Migration 024: cauldron_recipe_meta keyed by (household_id, recipe_slug),
meta_json + enrich_version (bumping the version invalidates the cache
and forces re-walk). One row per Mealie recipe Cobb owns.
- Migration 025: cauldron_enrich_jobs — job runner state. No
proposals/review needed since metadata is purely additive.
Forge:
- enrich_recipe(recipe) builds a compact prompt with name + description
+ ingredients + steps (capped at 2000 chars total) + yields, asks
Sonnet for the structured blob. _extract_recipe_meta validates and
coerces types.
Module enrich_recipes.py:
- Daemon thread runner, walks all household recipes, skips already-
enriched at current ENRICH_VERSION (idempotent), respects external
cancel + stuck-job recovery. Skips cross-household recipes (Lake
Elsinore stuff visible but not enrichable).
Plan generator hookup:
- /api/plan/generate + regenerate now pulls cauldron_recipe_meta and
splices it into the recipe pool prompt. Each pool line goes from:
- chicken-stir-fry: Chicken Stir Fry [asian]
to:
- chicken-stir-fry: Chicken Stir Fry [asian · easy · 30min ·
protein:chicken · carb:rice · high-protein/weeknight/one-pan]
quick weeknight stir-fry with leftover-friendly portions
Sonnet now has rich attributes to actually match a 'high protein
week' or 'comfort food' or 'quick' preference against, instead of
guessing from titles.
Endpoints:
- /enrich-recipes UI page (progress bar + start + force re-enrich +
cancel; no review/approve since meta is additive)
- /api/recipes/enrich-{start,status,cancel} session-authed
- /api/admin/recipes/enrich-start bearer-authed for kayos kick-off
Cost (one-time): ~5s/recipe × 226 = ~20 min walk. Subsequent runs
only process new/changed recipes.
The killer-feature one Cobb wanted on /plan — a free-form textarea
above the generate button so the household can bias the week's plan
toward "high protein low carb" / "carb load" / "light recovery" /
etc. Plus 8 preset chips that one-click-fill the textarea with common
templates.
Schema:
- Migration 023 adds preference_prompt VARCHAR(1000) to
cauldron_meal_plans (idempotent ALTER ADD COLUMN IF NOT EXISTS)
- Persisted per-week, per-household. Survives re-rolls so the same
preference applies unless explicitly changed.
Forge:
- generate_plan accepts optional preference kwarg
- _build_plan_prompt splices in HOUSEHOLD PREFERENCE block when set:
"BIAS your AI-chosen slots toward recipes from the pool that match
it... preference does NOT override picks — every pick still appears.
It DOES change which other recipes you choose to fill the rest."
- Examples in the prompt cover diet, occasion, shopping constraints,
vibe categories so Sonnet has full coverage of what users might mean.
Server:
- POST /api/plan/generate accepts {preference: "..."} body, persists
to plan row before kicking off Sonnet
- POST /api/plan/regenerate (re-roll): if body has preference, persist
+ use it; else reuse the persisted preference. Lets re-rolls iterate
on the same vibe.
UI (/plan):
- Textarea visible when no slots yet, with maxlength=1000 and a
placeholder showing example prompts
- 8 preset chips: 🥩 high protein / 🍞 carb load / 🥗 light & lean /
🌱 vegetarian / 🍲 comfort food / ⚡ quick / 💧 recovery / 🌍 global —
click fills the textarea with a curated phrase Cobb can edit
- After generation: read-only "vibe this week:" callout above the
action buttons so the household can see what was used
- Locked plan also shows the preference (audit / nostalgia)
- Generate + re-roll JS now sends JSON body with preference
Cost: ~30 extra tokens in the prompt when preference is set, ~0 when
unset. No additional Sonnet calls.
Job 1's 131 clusters included a 50+ food megacluster ('2% milk', 'acai
berry', 'acai berry juice', 'achiote oil', 'aleppo pepper', 'all
purpose flour', ...) that Sonnet correctly rejected as a false positive.
Cause: single-link agglomerative chains weak similarities — A~B and
B~C unite A and C even though A and C aren't actually similar.
Switched to pair-based: emit one 2-food candidate per (i, j) above
threshold, no clustering. Eliminates the megacluster shape entirely.
Sonnet decisions are cleaner on uniform 2-row pairs, UI cards are
uniform, and Mealie's merge endpoint is per-pair anyway.
Trade-off: a true 3-way dupe (A=B=C) now produces 3 separate pairs
(A,B) (B,C) (A,C) that each go through review. Net effect after
approval: same merges happen. Apply path defensively catches the
404 case — once (A,B) merges, the (A,C) pair has stale A and Mealie
returns 404; treat as already-handled, not an error.
For ~3000 foods this is ~4M comparisons in pure Python (a few seconds).
Job 1's data still applies cleanly — 10 historical merges + 121
keep-distinct decisions stay where they are. Future runs use the new
shape.
New bulk job at /dedupe-recipes (and /api/recipes/dedupe-* + admin
variants). Mirrors the consolidate_foods pattern but for recipes
themselves rather than the foods table:
Walk:
- Pull all household recipes via list_recipes (paginated, ~250 for Cobb)
- Cluster by name token_set_ratio ≥ 85
- For each multi-recipe cluster, fetch full bodies + build a summary
(slug, name, source_url, ingredient_summary, step_count, yields)
- Ask Sonnet via forge.recipe_dedupe_decision: are these the same
dish? canonical_slug + delete_slugs + reason
- Persist proposal
Apply:
- Per approved proposal: DELETE each delete_slug via Mealie API
- Mark applied / record error per cluster
Schema: migrations 021+022 (cauldron_recipe_dedupe_jobs +
cauldron_recipe_dedupe_proposals). Same state machine: running →
review → applying → done/failed/cancelled. Same daemon-thread runner
with cancel-respect + stuck-recovery.
Sonnet integration:
- recipe_dedupe_decision prompt is conservative-by-default. duplicates=
false on the slightest doubt (different ingredient sets, suggestive
name differences, etc). Picks canonical = cleanest name + most
complete data + lex-older slug as tiebreaker.
Mealie integration:
- mealie.delete_recipe(slug) → DELETE /api/recipes/<slug>. Permanent.
Permission-scoped per-household (cross-household will 403).
UI:
- /dedupe-recipes — same shape as /consolidate but with side-by-side
recipe cells (canonical marked ★ KEEP, deletes marked × DELETE in
red). Source URLs link out so user can sanity-check.
- DEFAULT TO NOT-APPROVED — recipe deletion is destructive, user must
opt in per cluster. Bulk "approve all dupes" is one click but the
apply confirm explicitly counts how many recipes will die.
- Linked from /me alongside sterilize + consolidate.
Cobb confirmed earlier: "we can't lose recipe data" — answered by
(1) conservative Sonnet decisions, (2) opt-in default, (3) explicit
permanent-deletion confirm, (4) same-pattern logging + DB audit trail
on every attempt.
Bug: my anti-zombie guard from 4707e6a was too strict — WHERE clause
required state IN ('running','applying') to update. But the normal
flow goes running→review→applying→done. Once a job entered review,
NO state transition could fire — including the legitimate apply
sequence triggered by user clicking "apply selected".
Symptom Cobb hit: clicked apply on job 6, the daemon thread did the
work (11 of 13 proposals applied cleanly to Mealie), but the row
stayed at state='review' so the UI never moved off the review screen.
The 11 successful applies are real — Mealie has the updated
recipeIngredient food links. The bookkeeping just didn't follow.
Fix: change WHERE clause from a positive whitelist (running/applying)
to a negative blocklist (NOT IN done/failed/cancelled). This still
prevents the original failure mode (daemon overwriting a user-cancelled
job) — terminal states still can't be overwritten — but lets review
transition to applying when the user approves.
Same fix applied to finalize_consolidate_job since it copy-pasted the
same too-strict guard.
Two failures surfaced by job 6 with the bigger prompt (full Mealie food
catalog ~50KB + recipe context with steps + spell-cleanup rules):
1. quinoa-chili-with-sweet-potatoes: 180s timeout. The bigger prompt
means Sonnet has more to chew through per call. Bumped _parse_batch
timeout 180s → 300s. Recipes with many ingredients now get more
slack before clawdforge gives up.
2. salmon-sushi-bake: "unexpected response shape" — Sonnet returned
the JSON as a STRING rather than a parsed dict (depends on size +
how clawdforge unwraps the response). _parse_batch was strictly
requiring isinstance(result, dict) and rejecting strings outright,
leaving the recipe in error state with valid JSON visible in the
error message. Added defensive string→dict parsing (with optional
code-fence stripping) mirroring the pattern already used by
forge._extract_food_info.
Both errored recipes from job 6 can now be re-run cleanly. Apply path
unchanged — defensive food.id preservation from 6bcf79e still in
effect.
Three improvements driven by Cobb's review of the fan-out output:
1. Recipe context. _parse_batch now accepts an optional recipe_context
dict carrying recipe_name, recipe_description, and recipe_steps.
preview_recipe builds the context from the Mealie recipe and passes
it through. The Sonnet prompt has new USE RECIPE CONTEXT WHEN
AMBIGUOUS rules: "1 cup flour" is ambiguous (AP / bread / cake);
the cooking steps usually disambiguate ("knead until elastic" →
bread flour, "sift with cocoa powder" + cake recipe → cake flour).
Step text capped to 3000 chars so the user prompt stays modest;
defaults to all-purpose flour when steps don't disambiguate.
Brand/style hints in the description carry through too.
2. Spell + grammar cleanup. New SPELL/GRAMMAR CLEANUP rules in the
prompt: silently fix typos in food and note ("tomatos" → "tomatoes",
"chopped finly" → "chopped finely", "heavy cram" → "heavy cream").
Normalize spacing. Critically: preserve EVERY semantic value —
numeric quantities verbatim, every prep state, brand, color. When
uncertain whether something is a typo or intentional ("yellow
squash" is a real food, not a typo), keep it. Original strings
stay in originalText for audit / rollback.
3. Defensive food.id preservation in apply_recipe. Three new safeguards
protect against Sonnet hallucinations dropping live recipe data:
a) If Sonnet returns a single all-null parsed item but the original
Mealie row had a real food.id, pass the original through
verbatim. (Sonnet probably parse-failed; never blank a real link.)
b) When Sonnet returns a food name that we can't resolve in Mealie's
catalog AND the original had a food.id, preserve the original
link rather than emit food=null.
c) When Sonnet explicitly returns food=null on the first child of
an ingredient that originally had a food.id, treat that as a
misread and preserve the original. Real section headers — where
the original was ALREADY foodless — still pass through cleanly.
Net effect: no apply path can drop a recipe's existing food
reference. Sonnet can ADD food links (good), CHANGE them (good),
or fail to parse (we keep what was there). It cannot remove them.
The is_new_food field also benefits from recipe context — Sonnet has
more evidence to set is_new_food=false (matched a known canonical)
when the steps confirm the ingredient identity.
Migration 020 drops cauldron_pick_points (game system stripped from
/plan 2026-04-30 — all award_pick_points calls were already removed
from server.py). The DB methods household_scoreboard, household_streak,
and award_pick_points are deleted now too — they were unused dead code
since the strip. delete_plan_slots no longer DELETEs from the dropped
table (kept it minimal: just wipe the meal_plan_slots).
DEFERRED from Step 4: dropping cauldron_foods. The 229 Sonnet-curated
density rows + 2462 USDA seed rows are still useful raw material for
a fuzzy backfill into cauldron_food_metadata (only 128 of Mealie's
2895 foods got densities matched on the first exact-name pass). Until
we run a fuzzy backfill, holding onto cauldron_foods as a cold archive
is cheaper than losing the data. Will revisit once the metadata
catalog is more complete.
Recovery from 020: revert this migration, restore the CREATE TABLE
position-014 entry from db.py history. The points data was never
meaningfully populated (jobs 1+3 award attempts were folded into the
strip), so loss is essentially zero.
New bulk job that scans Mealie's foods table for duplicate-feeling
clusters, asks Sonnet to pick the canonical survivor + flag the rest
as merge candidates, and uses Mealie's PUT /api/foods/merge to
consolidate. After each successful merge, alias_additions get pushed
onto the survivor so Mealie's CRF/NLP parser fuzzy-matches the
discarded variant names from then on.
Architecture mirrors bulk_sterilize.py:
- Migrations 018+019 add cauldron_consolidate_jobs +
cauldron_consolidate_proposals (state machine: running → review →
applying → done/failed/cancelled)
- New consolidate_foods.py — daemon-thread runner with cancel-respect
and stuck-job recovery
- /api/foods/consolidate-{start,status,jobs,apply,cancel} for session
users + /api/admin/foods/consolidate-{start,jobs,cancel} for kayos
Sonnet integration:
- forge.cluster_decision(foods) → returns {merge, canonical_id,
canonical_name, discard_ids, alias_additions, reason}
- Conservative-by-default: when in doubt Sonnet returns merge=false
(the "olive oil vs olive" false-positive case from the prompt)
- Alias rules in the prompt explain why we want discarded names to
travel back to the survivor as aliases (parser future-proofing)
Mealie integration:
- mealie.merge_foods(from_id, to_id) → PUT /api/foods/merge
- mealie.update_food(food_id, body) → for pushing aliases onto the
survivor after merges land
- Apply path catches 403/permission errors and surfaces them as the
per-cluster apply_error (cross-household merge attempts will fail
here, same way as the sterilize cross-household path)
Clustering:
- rapidfuzz token_set_ratio ≥ 88 (slightly stricter than Mealie's
parser threshold of 85 to reduce false-positive clusters)
- Single-link agglomerative — O(n²) but Cobb's ~3000 foods = ~9M
comparisons, runs in seconds
- Singleton clusters (no merge candidates) are dropped, not stored
UI:
- /consolidate — same shape as /sterilize: progress bar → review grid
→ apply button. Cards show member chips with the canonical marked
★, discards marked × in red, alias_additions listed in green, plus
Sonnet's one-line reasoning. Mergeable approved by default; user
toggles individual clusters off if they disagree.
- Linked from /me → tools section, alongside bulk sterilize.
Total: ~600 LoC across 6 files. Foundation for the "Mealie owns
canonical names" architectural rule is now actually enforceable —
cobb runs this once, his foods table gets cleaned up, and Sonnet's
catalog-aware parser (Step 1) starts matching aliases for free.
System MEALIE_API_TOKEN is 401 (legacy 'Cauldron' token from Mealie
mint long ago, since rotated/expired). The backfill tries it first;
on 401 falls back to any stored per-user token via the new
db.first_usable_mealie_token helper. Cobb's user is admin in Mealie
and his token returns the full group catalog, so the backfill works
without minting a fresh system token.
Aligns cauldron's data layer with the architectural rule "Mealie owns
canonical food names; cauldron only owns cooking metadata Mealie can't
store". The old parallel name catalog (cauldron_foods, 2462 noisy USDA
rows + ~229 Sonnet-curated names) was always going to drift from
Mealie's foods table over time. Now metadata follows Mealie's UUID, so
when Cobb merges or renames a food in Mealie the density+unit_class
travels with it automatically.
Schema:
- New table cauldron_food_metadata (migration 017): primary key is
mealie_food_id VARCHAR(64); columns are density_g_per_ml,
common_size_g, default_unit_class, category, source enum (seed /
claude / manual), notes JSON.
- cauldron_foods table stays untouched in this step (Step 4 drops it
after the backfill ledger has been verified in production).
Code:
- cauldron/foods.py rewritten:
- get_metadata_by_food_id(db, mealie_food_id) — primary read
- upsert_metadata(...) — write keyed by mealie_food_id
- fetch_and_persist(db, mealie_food_id, food_name, forge) — Sonnet
fallback, persists keyed by id
- backfill_seed_from_legacy(db, mealie) — one-time migration helper
called at app boot when metadata table is empty. Walks Mealie's
foods, looks up each in legacy cauldron_foods by name/plural/alias,
copies density into the new table keyed by Mealie's id. Returns
{matched, missed, total_mealie} stats.
- Legacy shims (search_food, upsert_claude_food, load_seed_if_empty)
kept as no-ops so server boot doesn't break before full cutover.
- cauldron/aggregator.py:
- Ingredient.mealie_food_id new optional field
- aggregate() now keys by mealie_food_id when present, falls back to
normalized name. Verified with rice-from-3-recipes synthetic:
same id → consolidates to "2.25 lb rice" single line as before.
- foods_lookup callable signature changed to (name, food_id) — id is
primary, name is for display + Sonnet fallback.
- cauldron/server.py:
- /list view captures Mealie's food.id from each recipe ingredient
and threads it through the Ingredient. foods_lookup now does an
id-keyed cauldron_food_metadata read; on miss with a known id,
calls forge.fetch_food_info and persists. When food.id is missing
(ingredient still in note form, no Mealie row linked), returns
None and aggregator falls back to name grouping.
- Boot: replaces the USDA seed loader with a one-time backfill of
legacy cauldron_foods → cauldron_food_metadata via the system
Mealie token. Runs only when metadata table is empty.
Net effect: rice in 3 recipes that all link to the same Mealie food
row now group by UUID, not by lowercased name. When Mealie's foods
get cleaned up (Step 3 consolidator), cauldron's metadata follows
because the ids are stable. Foundation for the consolidator is now
in place.
Sonnet was picking food names blindly. We then tried to match those
back to Mealie's catalog post-hoc. When Sonnet's natural pick didn't
match Mealie's exact convention, we'd create a duplicate row instead
of reusing the existing one. Lucky alignment with the seed kept the
dupe rate low, but the architecture had no real "Mealie is source of
truth" guarantee.
This change makes that guarantee explicit:
1. Sterilizer now lazy-loads Mealie's full food catalog on first
_parse_batch call (one fetch per Sterilizer instance, so a bulk
job pulls 2895 rows once and reuses across all 226 recipe parses).
Uses the underlying mealie._get with per_page=2000 + page-walk for
defensive coverage of really large catalogs.
2. STERILIZE_SYSTEM is now STERILIZE_SYSTEM_TEMPLATE — a string with
a {foods} placeholder. _system_prompt() splices in a bullet list of
every Mealie food (name, plural, aliases) at runtime.
3. New CATALOG RULES in the prompt instruct Sonnet to:
- Match against name / pluralName / aliases first, return canonical
name verbatim with is_new_food=false
- Strip prep modifiers into note
- Singularize plurals when canonical is singular
- Treat brand variations as canonical+note ("Heinz ketchup" →
food: "ketchup", note: "Heinz")
- Set is_new_food=true ONLY when no reasonable catalog match exists,
since adding aliases to fix mismatches later is way easier than
cleaning duplicate food rows after the fact
4. New is_new_food field on IngredientParse and per-item schema. Will
eventually drive an "alias suggestion" UI, but for v1 just gives
us telemetry on how often Sonnet falls back to inventing names.
Net effect for the family-internal goal: zero duplicate food creates
from convention mismatches, smarter parses that respect the catalog
Cobb spent time curating, foundation laid for the Step 2 re-key
migration where cauldron_food_metadata gets keyed by Mealie food.id.
Cobb spotted job 4's proposals weren't actually doing useful work —
"Toppings (Cinnamon Butter, Marshmallows, Ground Cinnamon, Butter, Etc)"
came back unchanged because the prompt was rigidly 1-in-1-out and
treated the whole compound line as a section header. Same for
"salt and ground black pepper to taste" — should be 2 separate
shopping list items but the parser kept them as one note.
Three changes:
1. STERILIZE_SYSTEM rewritten to allow fan-out. New return shape is
list-of-lists: outer list mirrors input length, each inner list
has 1 item (normal case) or N items (fan-out). Explicit fan-out
rules cover the two patterns Cobb cares about:
- "salt and pepper" / "X and Y to taste" → 2 items
- "Toppings (a, b, c, etc)" / "Optional: A, B, C" → N items,
wrapper word dropped, filler ("etc") skipped
Plus a heuristic against accidentally splitting compound food
names ("salt and vinegar chips", "macaroni and cheese" → keep).
2. _parse_batch + IngredientProposal + apply_recipe all updated for
the new shape. IngredientProposal.parsed → parsed_items: list.
apply_recipe iterates each child:
- First child inherits the original Mealie row's id/refId/
originalText so existing references stay live
- Additional children are fresh dicts; Mealie generates ids
on save when none provided
Backward-compat fallback in apply_recipe accepts the legacy
single-parsed shape so any in-flight job 4 proposals still apply
cleanly.
3. /sterilize UI was→becomes table now:
- Renders one row per parsed child (rowspan'd "was" cell when
fanning out, with a "→³" superscript marker on the arrow)
- Drops identity rows (1→1 case where parse matches original
verbatim) so the diff shows only ACTUAL changes — fixes Cobb's
"this doesn't look sterilized at all" complaint where every
diff was identical
- Cards with all-identity proposals show "no changes proposed
(all ingredients already matched)" instead of an empty table
Job 4's stored proposals use the legacy 1→1 shape so won't show fan-out
until re-walked. Recommend cancelling job 4 and starting a fresh job 5
with the new prompt to see the toppings line break out properly.
Two small adds:
(1) auth/callback no longer 500s on transient JWKS-fetch failures or
on a stale state cookie from a prior failed callback. Catches three
distinct failure modes and renders templates/auth_retry.html with
a 503 / 400 status:
- RequestsConnectionError + Timeout → "couldn't reach the auth
server — usually a momentary DNS or network blip"
- MismatchingStateError → "that login link expired (you probably
retried after a blip). hit login again to start fresh"
- OAuthError → "auth handshake failed: <detail>"
All three paths pop _state_cauldron_authlib so the next /login
starts with a fresh state token. Today's incident: cauldron
container hit a brief auth.sulkta.com DNS resolution failure
during OIDC callback, threw a stack trace, then the user's retry
failed differently because of the leftover state cookie.
(2) /api/admin/sterilize/bulk-start (require_bearer) accepts
{"started_by_sub": "user@example.com"} body. Resolves that user's
household, decrypts their stored Mealie token, spawns the same
preview thread the session-authed endpoint does. The job appears
in /sterilize for the user as if they'd started it themselves.
Plus admin-status (GET /api/admin/sterilize/jobs/<id>) and
admin-cancel (POST /api/admin/sterilize/bulk-cancel/<id>) for
operator symmetry. Existing user-session endpoints unchanged.
Net: kayos can kick off + cancel + monitor sterilize jobs from the
outside without piggybacking on Cobb's browser cookie, and a brief
DNS hiccup on the LAN no longer turns into a "your login is broken"
experience for the user.
Job 3 surfaced the bug — when I set state=cancelled in the DB, the
daemon thread kept running and finalize() at the end overwrote it
with 'done'. User cancellations were getting silently undone.
Two changes:
1. Runners (run_bulk_preview, run_bulk_apply) now check the job's
current state at the top of every iteration via the new lightweight
db.get_sterilize_job_state. If the state has moved to a terminal
value (cancelled, failed, done) externally, the loop returns
without finalizing.
2. db.finalize_sterilize_job now refuses to overwrite a non-running
state — added "AND state IN ('running','applying')" to the WHERE
clause. Belt-and-suspenders for the same problem: even if a runner
races past the state check and limps to its finalize call, the DB
itself won't let the cancellation be replaced.
Net: hitting cancel via the UI button (or a DB update) now actually
stops the runner mid-flight. Polling roundtrip per recipe is one
SELECT — negligible vs the multi-second clawdforge call that
dominates each iteration.
Mealie's group model spans multiple households (Hayes House group has
Redondo + Lake Elsinore — Cobb's family + Bay/mom share recipes
read-only across households). Members can list/read every recipe in
the group, but write access is per-household. Trying to sterilize a
foreign-household recipe returns 403 on the PUT; the food/unit creates
that ran first end up as orphan rows in the user's own household.
Walk path now resolves /api/users/self.householdId once (cached on the
Mealie client), and skips any recipe whose top-level householdId differs.
No proposal row is created; the recipe just counts as "skipped" along-
side already-clean ones.
Apply path adds the same defensive check (covers job 1's existing
proposals from before this fix landed) and translates any remaining
403 to a friendly "skipped: recipe belongs to a different household —
sterilize from that household's account" message instead of dumping
Mealie's raw permission-denied JSON into the UI.
Net: Cobb sterilizes Redondo recipes from his account; Bay or mom
each get their own walk-and-apply scoped to Lake Elsinore when they
sign in. Same architecture works for new households joining Sulkta —
each member's bulk job is automatically scoped to the household they
belong to. Zero cross-pollution.
Job 1's bulk run apply'd 184 recipes and 182 of them failed with the
same error: POST /api/foods -> 400 UNIQUE constraint failed:
ingredient_foods.name, ingredient_foods.group_id. Cause: Mealie's
name_normalized strips punctuation/whitespace/case more aggressively
than our local _build_name_index's plain .lower(), so the cache misses,
the create_food fires blindly, and Mealie's UNIQUE constraint kills
the call. Whole-recipe apply was wrapped in try/except at the bulk
runner so the recipe got marked errored — but applied_at was still
set to NOW(), making the rerun think we'd already tried. We had, but
the recipe's still unparsed.
Two fixes:
1. sterilizer._resolve_food / _resolve_unit replace the inline
create-on-miss block. Order: local cache → Mealie search-endpoint
tie-break → create. On any UNIQUE-flavored 400 from create, fall
back to one more search to adopt whatever Mealie has under the
normalized form. Mealie's search endpoint applies its own
name_normalized internally so we don't have to mirror its rules.
_search_for_match takes "foods" or "units" and looks for an exact
case-insensitive match against name or pluralName, with a fallback
to "trust Mealie's ranker" when there's exactly one hit.
2. db.mark_proposal_applied no longer sets applied_at on error. On
success: applied_at=NOW(), apply_error=NULL. On error: applied_at
stays NULL, apply_error gets the message. list_approved_unapplied_
proposals keys off applied_at IS NULL, so a rerun naturally retries
only the failed recipes.
Net effect: rerun can now successfully apply the 182 failed recipes
without re-walking them, and won't waste calls on the 2 that did go
through.
Mealie's "Parse" button on every recipe is per-recipe; clicking through
226 of them is the toil Cobb explicitly hates. The bulk sterilizer wraps
cauldron's existing per-recipe Sterilizer (clawdforge → Sonnet, parses
free-form ingredient strings into structured form) into a household-wide
job with progress tracking, batch human review, and one-shot apply.
Flow:
1. POST /api/sterilize/bulk-start — creates cauldron_sterilize_jobs row,
spawns a daemon thread that walks every recipe in the user's
household. Recipes whose ingredients all already have food.id are
skipped (no point re-parsing Cobb's manual cleanup). For each
recipe needing work, Sonnet returns a structured proposal that gets
persisted to cauldron_sterilize_proposals.
2. GET /api/sterilize/bulk-status — polled every 2s by the UI for
{state, processed_count, skipped_count, error_count, current_slug}.
3. After the walk completes, state moves to 'review'. UI loads
/api/sterilize/bulk-jobs/<id>/proposals and renders one card per
recipe with a was→becomes diff per ingredient. User toggles
approve/skip per recipe.
4. POST /api/sterilize/bulk-apply/<id> with {approved_slugs: [...]}.
A second daemon thread iterates approved proposals, calls
Sterilizer.apply_recipe (which resolves food/unit IDs in Mealie,
creating any missing rows, then PUT /api/recipes/<slug>).
Job state machine: running → review → applying → done (or 'cancelled' /
'failed' along the way). At app startup, fail_stuck_sterilize_jobs()
recovers any 'running' / 'applying' rows older than 10 min with no
progress — covers the case where the daemon thread's gunicorn worker
died mid-job. New job state lives in DB; thread is just a runner.
Concurrency: the start endpoint blocks if the household already has a
running/applying job. Sterilize calls cost clawdforge time and parallel
jobs would race on Mealie writes, so one-at-a-time per household is the
right ceiling.
UI lives at /sterilize. Linked from /me → "tools → bulk sterilize" since
it's a one-off admin action, not part of the daily flow. Mobile-aware,
diff cards expand on click. Disabled "apply" button when no recipes
are selected; preview-error rows can't be approved.
DB: two new tables (migrations 015+016). cauldron_sterilize_jobs tracks
overall job state with last_progress_at for the stuck-job recovery;
cauldron_sterilize_proposals holds per-recipe JSON proposals with the
approval flag and the final apply_at/apply_error.
Two changes:
1. foods catalog grows organically. Switch the canonical seed from the
noisy USDA dump (2462 rows of "'s, classic chicken noodle soup")
to the Sonnet-curated cut (229 clean rows). search_food() is now
exact + case-insensitive — Mealie's parser already canonicalizes
food names household-side, so cauldron just needs to look them up
verbatim. On miss, the /list view calls forge.fetch_food_info() to
ask Sonnet for {density_g_per_ml, default_unit_class, common_size_g,
category}, persists the row with source='claude', and the household's
actual kitchen catalog builds itself out as Abby uses it.
Killer case verified end-to-end: "2 cups + 50g + 1.25 lb rice"
collapses to a single "2.25 lb rice" line on the shopping list once
rice has a density row.
2. Game system stripped from /plan. Scoreboard panel, streak banner,
"first to lock takes the week" / "🏆 you locked this one in" copy
all gone. award_pick_points calls in /api/plan/generate +
/api/plan/regenerate stopped firing. household_scoreboard /
household_streak DB methods kept as dead code; cauldron_pick_points
table left in place — non-destructive, easy to revive later if
gamification comes back. Goal: get the base flow (pick → plan →
list) working for Abby first, layer features on after.
- migrations 012 + 013: cauldron_meal_plan_slots + cauldron_pick_points
- db: list_plan_slots, save_plan_slots, delete_plan_slots, mark_plan_generated,
clear_plan_generated, award_pick_points, enrich_plan_with_slots; scoreboard
extended with points (sum from pick_points) and weeks_locked alias
- forge.generate_plan: sonnet prompt builds 7-day plan respecting picks,
validates slot count + day uniqueness + slug-in-pool, fills picker_subs
from ground-truth picks (model output is advisory)
- POST /api/plan/generate: race-safe (existing slots → 409 with plan),
lock-aware (locked → 409), idempotent
- POST /api/plan/regenerate: re-roll for the original generator, gated by
ownership + lock; wipes slots + pick_points then re-runs generate
- plan.html: generate CTA + 7 day cards with picker chips + AI reason +
re-roll button (generator-only, pre-lock); scoreboard now shows points + wins
- /list: pulls plan slots, queries Mealie for ingredients, runs aggregator,
renders 48px-tall checkbox shopping list with localStorage state per plan_id
- tests: 13 new tests across forge.generate_plan + /api/plan/generate routes
+ /list view + scoreboard SQL inspection. conftest+_testenv stub
pymysql/oidc/foods at import time so tests run against module-level app
without a live DB. Both pytest and `unittest discover` paths green (27/27).
Defers: bulk sterilizer admin (A1), foods dedupe (A2), Mealie shopping-list-
export (button rendered but disabled). 7-slot count is fixed at the
endpoint (no UI for slot-count selection yet).
Spec: memory/spec-cauldron-v0.3.md
