scrape_me(url) no longer accepts wild_mode in 15.x. Two-stage fallback:
1. scrape_me(url) — site-specific scraper (best quality)
2. fetch html + scrape_html(html, org_url, supported_only=False) —
generic JSON-LD/microdata pass for unsupported sites
Caught at first dogfood test against allrecipes; previous code raised
"unexpected keyword argument 'wild_mode'" before issuing any HTTP.
Append-only schema for Discover v0.1. Two tables:
cauldron_discovered_recipes — corpus of recipes scraped from external
sources (allrecipes, BBC Good Food, smitten kitchen, etc) before any
household has imported them. status walks raw → enriched → imported
| rejected. source_url is capped at 768 chars to stay under InnoDB's
3072-byte unique-key ceiling (768 × 4-byte utf8mb4 = 3072). FULLTEXT
on (name, description) for the discover-page search bar.
cauldron_discover_jobs — daemon state for the scrape runner, same
pattern as sterilize/enrich/foods-consolidate (state-poll cancel,
last_progress_at watchdog, terminal-only finalize).
Migrations only — no Python helpers, no scraper, no UI yet. Lands when
the next cauldron container restart picks them up.
Hecate looks through the household's library and surfaces 1-6 recipes
that fit the picker profiles but haven't been served in 90+ days (or
ever). Server-side filters drop recently-planned + already-on-this-week
+ already-on-picks-list + allergen-conflict before Sonnet sees the pool,
keeping the prompt focused. Each suggestion comes with a 1-5 fit score
and a one-line reason in Hecate's voice. Pin → adds to the user's picks
so the next /plan/generate will naturally pull it in. Skip → hides the
card.
Endpoints:
POST /api/plan/suggest body {count?: 1-6, week?: ISO}
POST /api/plan/suggest/pin body {recipe_slug}
Layered on top of 37d7d60 (parser fix, committed not deployed). Both
should land together once enrich job 3 drains.
appends extra prose
Job 3 hit one recipe (healthy-chicken-stir-fry-with-vegetables) where
Sonnet returned a valid JSON object + appended prose afterward, in
violation of 'no prose' rule. json.loads choked with 'Extra data:
line 54'. _parse_json_blob now falls back to JSONDecoder.raw_decode
which extracts the first complete JSON value and ignores anything
after — any trailing notes / fence remnants / inline commentary get
silently dropped.
Plain json.loads is still tried first (fastest path on clean output).
The fallback only kicks in for malformed-but-recoverable cases.
hecate's quip + macro confidence + chain-of-thought macro estimation
Eight new fields packed in before Cobb's force re-enrich rerun:
(1) active_minutes + hands_off_minutes — split estimated_minutes into
"you must be present" vs "rise/marinade/braise unattended". Lets
weeknight planning match real available active time. A 4-hour beef
stew might be 30 active + 210 hands-off — fine for Sunday but
misleading as "240 minutes" on a busy weeknight.
(2) equipment[] — required appliances/modes from a fixed enum:
oven, stovetop, grill, instant-pot, slow-cooker, sheet-pan,
cast-iron, air-fryer, food-processor, blender, mixer, smoker,
sous-vide, no-cook. Foundation for "no oven this week" filters.
(3) flavor_profile[] — 2-5 dominant flavor notes (spicy, sweet,
savory, umami, tangy, smoky, herby, citrusy, rich, fresh, bitter,
earthy). Lets the planner enforce variety so we don't get five
spicy nights in a row.
(4) kid_friendly_score 1-5 — separate from comfort_tier. Cobb has
Leia and Luna; this is real signal. 1=adults-only (capers, blue
cheese, super spicy), 5=kids beg for it (mac and cheese, pancakes).
(5) fiber_g + sodium_mg per serving — extends the macro coverage.
Sodium for heart-health weeks, fiber for gut weeks.
(6) cost_per_serving_estimate — rough USD per serving (2026 prices).
Bean bowl $2, salmon for two $8-12, fancy roast $15+. Foundation
for budget-week preference. Set null for too-volatile items.
(7) occasion_fit[] — when is this dish AT HOME? weeknight, weekend,
brunch, date-night, party, picnic, camping, holiday, game-day,
kids-birthday, quiet-night-in.
(8) hecate_quip — one-line voice description in Hecate's mythic-witch
tone. Pure flavor for tooltips/detail pages. ~10-20 words. Examples:
"Pure midwinter comfort — asks for a fire and quiet."
"Sharp and bright like a Tuesday morning resolution."
MACRO QUALITY (Cobb's separate ask):
- Prompt now instructs Sonnet to compute macros chain-of-thought:
list each major ingredient, approximate its per-serving contribution
in grams, sum, output. Plus a sanity check: protein×4 + carbs×4 +
fat×9 ≈ calories. Not free precision but better than guess-the-total.
- New macros_confidence field (low/medium/high) — Sonnet rates own
certainty so users know when to trust the numbers and the planner
can avoid summing low-confidence macros over a 7-day budget.
- Future commit: deterministic USDA-FDC backfill into cauldron_food_
metadata for true ingredient-by-ingredient sums. Same shape as the
existing density backfill we did in Step 2.
Plan generator pool prompt expanded — every recipe entry now carries
inline:
active=Nm+offhand=Mm
eq:oven,stovetop
flavor:savory/herby/rich
kid=4
~$5/svg
for:weeknight/date-night
fiber=Ng / sodium=Nmg
ENRICH_VERSION 4→5. Cobb's force re-enrich gets all of these on the
existing 222 recipes in one walk.
Hecate's weekly reading + 2nd-pass allergen verification
Five new context dimensions for the planner agent (and one quality
fix), bumping ENRICH_VERSION 3→4 so existing meta gets refreshed on
the next walk:
(A) Per-user fit score. Computed at plan-gen time from picker_profiles
× meta — no extra Sonnet calls. _compute_fit_score scores each
recipe 1-5 per household member based on cuisine match, protein
match, comfort_tier match, and tag overlap with that user's
historical picks. Renders in the recipe pool prompt as
'fit:cobb=5,abby=2,leia=3'. Gives Sonnet per-user signal to
bias AI-chosen slots toward whoever's home that week, AND makes
pick rationale explainable.
(B) Pairings per recipe. New meta fields:
pairings.serves_well_with: ["crusty bread","green salad",...]
pairings.drinks: ["pinot noir","iced tea",...]
Sonnet enriches both during the main pass. Foundation for future
"auto-suggest matching side" UX in multi-meal slots.
(C) Leftover potential 1-5. Per-recipe score: 1=eat-now-only (crispy
things, fresh salads), 5=actually BETTER as leftovers (stews,
braises, lasagna). Lets the planner thread Sunday's slow-cooked
pot roast into Monday's lunch slots intentionally.
(D) Mood scores. Per-recipe 1-5 ratings on:
cozy / summer_fresh / energizing / comfort
Independent dimensions — a recipe can score high on multiple.
Foundation for future weather/mood-aware planning ("rainy week
→ bias cozy>=4").
(E) Hecate's weekly reading. New TEXT column hecate_reading on
cauldron_meal_plans (migration 032). The plan-gen prompt asks
Hecate to write a 1-paragraph narrative voice description of the
week — "this week leans into the brisk turn of the season, three
hearty one-pots front-load your weekday energy, salmon Wednesday
for Cobb's gym push..." Confident, wise, theatrical voice. Pure
flavor, no functional impact, but makes Hecate FEEL like an
advisor not an opaque function. Plus the planner system prompt
now opens with the Hecate persona ("You are Hecate, Greek-
mythology witch goddess of crossroads, herbs, and magic — and
the family's meal planner"). Output schema gains a "reading"
field alongside "slots".
ALLERGEN VERIFICATION (the quality fix Cobb explicitly asked for):
- Sonnet sometimes flags pork=true on a sweet potato recipe via the
conservative-default rule. Cobb wants CLEAN data.
- New forge.verify_allergens — second Sonnet pass after main enrich
with a strict prompt: "name the SPECIFIC ingredient triggering each
allergen flag, or set FALSE." For non-anaphylaxis exclusions like
pork, set FALSE unless an actual pork ingredient is named. For
ANAPHYLAXIS allergens, conservative TRUE still applies.
- Cost: ~3s/recipe extra. Wired into enrich_recipes.run_enrich after
initial enrich; failure is non-fatal (falls through to original).
- Eliminates the pork-on-sweet-potatoes class of false positives.
Code restructure: forge.generate_plan now returns {slots, reading}
instead of just slots. _extract_plan_payload pulls both. Server-side
generate + regenerate paths unwrap and persist reading via the new
db.set_plan_hecate_reading. Plan template renders the reading in a
purple-bordered serif callout above the day grid.
Schema:
- migration 032 adds hecate_reading TEXT to cauldron_meal_plans.
- Cauldron_recipe_meta gets new fields persisted in meta_json (no
schema change there — JSON column already accommodates).
Greek goddess of magic, crossroads (decisions/planning), and herbs.
Three-for-three thematic fit for a meal planner. Internal field names
(sonnet_decision etc) unchanged — those are data, not user-facing.
Big plan-generator addition: each plan can now span multiple meal types,
not just dinner. Default stays dinner-only for back-compat; opt-in via
checkboxes on /plan.
Schema (migrations 028-031):
- cauldron_meal_plan_slots gains meal_type ENUM('breakfast','lunch',
'dinner','snack','dessert','side') NOT NULL DEFAULT 'dinner'.
- Old UNIQUE key (plan_id, day) → (plan_id, day, meal_type) so a
Monday can have breakfast AND lunch AND dinner slots.
- cauldron_meal_plans gains meal_types_json (which meals to plan
for that week — list of strings, defaults to ["dinner"]).
Forge:
- generate_plan accepts meal_types list. Output schema gains meal_type
per slot. Validates expected_total = slots * len(meal_types) and
rejects duplicate (day, meal_type) pairs.
- _build_plan_prompt renders MEAL TYPES TO PLAN block, instructing
Sonnet to match recipe meta.meal_type to slot type (breakfast slot
→ recipe whose meta tags it as breakfast). Falls back gracefully
when the pool is thin for a particular meal type.
Server:
- /api/plan/generate + regenerate accept body.meal_types, persist via
db.set_plan_meal_types.
- plan_view decodes meal_types_json into plan["meal_types_list"] and
builds plan["meal_types_label"] for the readout.
UI (/plan):
- New checkbox row at the top of the pref-block: 🍳 breakfast / 🥪 lunch
/ 🍽️ dinner. Defaults to whatever's persisted (or just dinner).
- Day cards now group multiple meal_type slots per day with small
meal-type tags above each recipe row. Single-meal plans render the
same way they always did (no tag shown when only one meal_type).
- readMealTypes() in JS reads checkboxes and ships in the body.
DB:
- save_plan_slots accepts meal_type per slot, defaults to 'dinner'.
- list_plan_slots orders by day then meal_type via MEAL_ORDER.
==
UX rename: "claude" / "sonnet" → "Sage" across all user-visible copy.
Sage doubles as kitchen-herb (theme fit) and wise advisor (planner
role). The internal field name `sonnet_decision` on consolidate +
dedupe proposals is unchanged (it's a data field, not user-facing).
Renames touched plan, consolidate, dedupe_recipes, list, me,
enrich_recipes, sterilize templates. Cobb can swap to Mim or his own
name later — easy global s/sage/whatever/g.
==
The /list 'clear' button removed earlier today (b4cb48b) — not
re-introduced.
(1) /list had a 'clear' button that just unchecked all the localStorage
shopping checkboxes. Cobb correctly flagged it as confusing — looks
like a destructive action ('clear what?'). It was just a UX nicety
for re-using the same shopping list, not a plan-touching operation.
The plan-level reset already lives on /plan via a properly-scoped
button. Removing the /list 'clear' so there's exactly one reset
concept in the app: /plan → ⌫ reset week.
(2) forge.enrich_recipe was hitting clawdforge with timeout_secs=90
which proved too short for one recipe (peppery-barbecue-glazed-shrimp
type long ingredient list). Bumped to 180s to match the sterilizer's
post-fix timeout. Walks now have more slack before timing out a single
recipe.
Three connected features so the planner is a real tool, not just a
single-week dashboard:
1. RESET. New POST /api/plan/reset wipes a plan back to blank — clears
slots + generated_at + preference_prompt + daily_targets_json +
exclusions_json. Hard-guards on lock state: locked plans return 409
"locked weeks are immutable history". This is the historical-
preservation guarantee — once a week is locked (via /api/plan/lock
OR auto-lock at week-rollover), it can never be touched again.
That's how 'what did we eat May 15 six years ago' stays answerable
forever.
2. WEEK NAVIGATION. /plan now accepts ?week=YYYY-MM-DD (snapped to
that week's Monday). Defaults to the current week as before. New
page-head nav: ← prev week / ⊕ this week (only when off-current) /
next week → / week-of-X span label. /list also accepts ?week= so
the shopping list view follows the same pattern.
3. HISTORICAL BROWSING. Past weeks render their plan slots as before —
just locked + immutable. Their preference_prompt + macros +
exclusions render in the readouts so you can see WHY that week
looked the way it did. The data was already preserved; this just
surfaces it through the existing /plan UI.
API + template changes:
- _resolve_week helper picks target Monday from body['week'] or today
- /api/plan/{lock,generate,regenerate,reset} all accept body['week']
- Plan view passes prev_week/next_week/current_week + is_*_week flags
+ week_end (Monday + 6) for the date-range label
- Template: PLAN_WEEK js constant threads the active week into every
mutation API call so prev/next nav can act on the displayed week,
not always today's
- Reset button styled red ('btn-danger'), only shown on unlocked
generated plans, confirms before firing
- 'view list →' link now passes ?week= so it stays in-week
DB:
- reset_plan(plan_id) wipes UNLOCKED plan state in one transaction.
Returns False (no-op) if the plan is locked — caller sees 409.
No schema changes — just leverages the per-week (household, week_start)
row uniqueness we already had.
The next-tier of structured planning constraints, building on the
free-form preference textarea. Now the household can set hard rules
like '2200 cal/day, 150g protein, no dairy or shellfish' and the
planner enforces them.
Schema (migrations 026 + 027):
- daily_targets_json on cauldron_meal_plans: {calories?, protein_g?,
carbs_g?, fat_g?} per-day budget. Sonnet sums per-recipe macros
across the 7-day plan and aims for ±15% of the weekly total.
- exclusions_json on cauldron_meal_plans: list of contains.* keys
(subset of {dairy, gluten, nuts, peanuts, eggs, shellfish, fish,
soy, sesame, pork}). Hard filter on AI-chosen slots; picks still
apply but get flagged with the conflict in their reason field.
DB:
- set_plan_targets_and_exclusions normalizes inputs (drops zeros,
validates against allowed enum values, lowercases).
- The plan_view path decodes the JSON columns server-side and adds
display labels (`targets_label`, `exclusions_label`) and the parsed
dict/list (`targets_dict`, `exclusions_list`) for the template.
Forge:
- generate_plan accepts daily_targets dict + exclusions list.
- New DAILY MACRO TARGETS prompt block — instructs Sonnet to sum
per-serving macros across all 7 slots and land within ±15% of
weekly totals; tradeoff slots when needed.
- New STRICT EXCLUSIONS prompt block — recipes whose has: list
intersects exclusions are forbidden in AI-chosen slots. Picks that
conflict still appear (explicit user choice) but get flagged in
the slot reason ("contains dairy — household pick").
Server:
- /api/plan/generate accepts {preference, targets, exclusions}.
Persists all three before kicking off Sonnet so re-rolls reuse them.
- /api/plan/regenerate same — empty body reuses persisted constraints.
UI (/plan):
- New <details> section "numeric targets + allergen exclusions
(optional)" tucked under the existing vibe textarea + presets.
- 4 numeric inputs (cal/protein/carbs/fat per day) with quick-set
preset chips: balanced 2200 / protein lean 2400 / carb load 2600 /
cut 1800 / clear.
- 10 allergen checkbox chips with red-pill styling when checked
(uses CSS :has(input:checked) — modern browsers).
- Hydration: persisted targets pre-fill the inputs, persisted
exclusions pre-check the boxes.
- After generation, three readouts above the action buttons show
active vibe / macros / exclusions.
Cobb's exact ask "2200 cal/day, protein and carb balanced, no dairy"
now maps to: targets={calories: 2200, protein_g: 150, carbs_g: 250}
+ exclusions=[dairy] + free-form preference (optional). All persist.
Re-rolls iterate within those constraints automatically.
Tier-1 data additions for the planner — turning the AI from a title-
matching guesser into a structured-data consumer. ENRICH_VERSION bumped
2→3 so existing meta gets refreshed with the new fields on next walk.
(A) Cook history. db.household_recipe_history aggregates recipe slug
→ {last_planned_date, count_30d, count_long} from cauldron_meal_
plan_slots over a 180-day window. The plan generator's pool prompt
now renders each recipe with rotation context: "last:8w-ago 0×/30d
1×/180d". New planner rule: ROTATION — demote recipes shown 2+
times in 30d unless they're picks; never repeat the same slug
within the 7-day plan. New planner rule: VARIETY — don't fill 5
of 7 slots with the same primary_protein or cuisine.
(B) Per-serving macros in enrichment. forge.enrich_recipe now asks
Sonnet for calories, protein_g, carbs_g, fat_g per serving (rough
USDA-grade estimates from ingredient list + yields). Renders into
the pool prompt as "~480cal protein=32g carbs=45g fat=18g". Lets
"high protein week" become a quantitative filter instead of a
title-keyword match.
(C) Allergen booleans. New contains.* block in enrichment:
{dairy, gluten, nuts, peanuts, eggs, shellfish, fish, soy, sesame,
pork} — bool per allergen, conservatively defaulting to TRUE when
uncertain since false negatives can hurt people. Pool prompt
renders as "has:dairy,gluten,eggs". Foundation for upcoming
"no dairy this week" exclusion-list UI on /plan.
(D) Picker profiles. db.household_picker_profiles unions current
cauldron_meal_picks + historical meal_plan_slots.picker_subs over
365 days, joins with cauldron_recipe_meta, aggregates per-user:
{display_name, total_picks, cuisines, proteins, comfort_tiers,
tags} — top-N counters each. Plan generator includes a new
PICKER PROFILES block in the prompt:
- cobb (sub=cobb@sulkta.com, 24 picks):
cuisines=[asian:6, mexican:4, italian:3] ·
proteins=[chicken:8, beef:5, fish:2] ·
tags=[weeknight:11, high-protein:9, spicy:7]
Sonnet uses these to bias AI-chosen slots toward each member's
actual demonstrated taste — golden signal that's been sitting in
the database the whole time. Picks still override profile bias.
Cost: cook history is a single SQL aggregate (free, sub-100ms). New
macro+allergen fields fold into the existing ~5s/recipe Sonnet call
with maybe 30 more output tokens. Picker profiles are 2-3 SQL queries
totaling sub-200ms even at scale. No new network round-trips.
Net effect once Cobb runs /enrich-recipes against ENRICH_VERSION 3:
plan generator has structured macros + allergen flags + cook-history
rotation context + per-user preferences to work with. The free-form
preference textarea ("high protein, no dairy") becomes a real query
against actual data, not just a Sonnet vibe-prompt.
The 'fancy data fun' Cobb wanted: pre-compute structured metadata for
every recipe so the plan generator can match preferences to actual
recipe characteristics, not just match keywords on names.
Sonnet returns per recipe:
- tags[]: curated descriptors (high-protein, weeknight, one-pan,
leftovers-good, kid-friendly, etc — picks 3-8 that genuinely apply)
- cuisine, complexity (easy/medium/involved), estimated_minutes
- meal_type (breakfast/lunch/dinner/snack/dessert/side/sauce/drink)
- primary_protein (chicken/beef/pork/fish/seafood/tofu/...)
- primary_carb (rice/pasta/bread/potato/tortilla/quinoa/...)
- veg_forward (veg-forward/mixed/meat-forward)
- comfort_tier (weeknight-easy/hearty-comfort/fancy-occasion/...)
- season_fit[] + summary one-liner + best_for short phrase
Schema:
- Migration 024: cauldron_recipe_meta keyed by (household_id, recipe_slug),
meta_json + enrich_version (bumping the version invalidates the cache
and forces re-walk). One row per Mealie recipe Cobb owns.
- Migration 025: cauldron_enrich_jobs — job runner state. No
proposals/review needed since metadata is purely additive.
Forge:
- enrich_recipe(recipe) builds a compact prompt with name + description
+ ingredients + steps (capped at 2000 chars total) + yields, asks
Sonnet for the structured blob. _extract_recipe_meta validates and
coerces types.
Module enrich_recipes.py:
- Daemon thread runner, walks all household recipes, skips already-
enriched at current ENRICH_VERSION (idempotent), respects external
cancel + stuck-job recovery. Skips cross-household recipes (Lake
Elsinore stuff visible but not enrichable).
Plan generator hookup:
- /api/plan/generate + regenerate now pulls cauldron_recipe_meta and
splices it into the recipe pool prompt. Each pool line goes from:
- chicken-stir-fry: Chicken Stir Fry [asian]
to:
- chicken-stir-fry: Chicken Stir Fry [asian · easy · 30min ·
protein:chicken · carb:rice · high-protein/weeknight/one-pan]
quick weeknight stir-fry with leftover-friendly portions
Sonnet now has rich attributes to actually match a 'high protein
week' or 'comfort food' or 'quick' preference against, instead of
guessing from titles.
Endpoints:
- /enrich-recipes UI page (progress bar + start + force re-enrich +
cancel; no review/approve since meta is additive)
- /api/recipes/enrich-{start,status,cancel} session-authed
- /api/admin/recipes/enrich-start bearer-authed for kayos kick-off
Cost (one-time): ~5s/recipe × 226 = ~20 min walk. Subsequent runs
only process new/changed recipes.
The killer-feature one Cobb wanted on /plan — a free-form textarea
above the generate button so the household can bias the week's plan
toward "high protein low carb" / "carb load" / "light recovery" /
etc. Plus 8 preset chips that one-click-fill the textarea with common
templates.
Schema:
- Migration 023 adds preference_prompt VARCHAR(1000) to
cauldron_meal_plans (idempotent ALTER ADD COLUMN IF NOT EXISTS)
- Persisted per-week, per-household. Survives re-rolls so the same
preference applies unless explicitly changed.
Forge:
- generate_plan accepts optional preference kwarg
- _build_plan_prompt splices in HOUSEHOLD PREFERENCE block when set:
"BIAS your AI-chosen slots toward recipes from the pool that match
it... preference does NOT override picks — every pick still appears.
It DOES change which other recipes you choose to fill the rest."
- Examples in the prompt cover diet, occasion, shopping constraints,
vibe categories so Sonnet has full coverage of what users might mean.
Server:
- POST /api/plan/generate accepts {preference: "..."} body, persists
to plan row before kicking off Sonnet
- POST /api/plan/regenerate (re-roll): if body has preference, persist
+ use it; else reuse the persisted preference. Lets re-rolls iterate
on the same vibe.
UI (/plan):
- Textarea visible when no slots yet, with maxlength=1000 and a
placeholder showing example prompts
- 8 preset chips: 🥩 high protein / 🍞 carb load / 🥗 light & lean /
🌱 vegetarian / 🍲 comfort food / ⚡ quick / 💧 recovery / 🌍 global —
click fills the textarea with a curated phrase Cobb can edit
- After generation: read-only "vibe this week:" callout above the
action buttons so the household can see what was used
- Locked plan also shows the preference (audit / nostalgia)
- Generate + re-roll JS now sends JSON body with preference
Cost: ~30 extra tokens in the prompt when preference is set, ~0 when
unset. No additional Sonnet calls.
Job 1's 131 clusters included a 50+ food megacluster ('2% milk', 'acai
berry', 'acai berry juice', 'achiote oil', 'aleppo pepper', 'all
purpose flour', ...) that Sonnet correctly rejected as a false positive.
Cause: single-link agglomerative chains weak similarities — A~B and
B~C unite A and C even though A and C aren't actually similar.
Switched to pair-based: emit one 2-food candidate per (i, j) above
threshold, no clustering. Eliminates the megacluster shape entirely.
Sonnet decisions are cleaner on uniform 2-row pairs, UI cards are
uniform, and Mealie's merge endpoint is per-pair anyway.
Trade-off: a true 3-way dupe (A=B=C) now produces 3 separate pairs
(A,B) (B,C) (A,C) that each go through review. Net effect after
approval: same merges happen. Apply path defensively catches the
404 case — once (A,B) merges, the (A,C) pair has stale A and Mealie
returns 404; treat as already-handled, not an error.
For ~3000 foods this is ~4M comparisons in pure Python (a few seconds).
Job 1's data still applies cleanly — 10 historical merges + 121
keep-distinct decisions stay where they are. Future runs use the new
shape.
New bulk job at /dedupe-recipes (and /api/recipes/dedupe-* + admin
variants). Mirrors the consolidate_foods pattern but for recipes
themselves rather than the foods table:
Walk:
- Pull all household recipes via list_recipes (paginated, ~250 for Cobb)
- Cluster by name token_set_ratio ≥ 85
- For each multi-recipe cluster, fetch full bodies + build a summary
(slug, name, source_url, ingredient_summary, step_count, yields)
- Ask Sonnet via forge.recipe_dedupe_decision: are these the same
dish? canonical_slug + delete_slugs + reason
- Persist proposal
Apply:
- Per approved proposal: DELETE each delete_slug via Mealie API
- Mark applied / record error per cluster
Schema: migrations 021+022 (cauldron_recipe_dedupe_jobs +
cauldron_recipe_dedupe_proposals). Same state machine: running →
review → applying → done/failed/cancelled. Same daemon-thread runner
with cancel-respect + stuck-recovery.
Sonnet integration:
- recipe_dedupe_decision prompt is conservative-by-default. duplicates=
false on the slightest doubt (different ingredient sets, suggestive
name differences, etc). Picks canonical = cleanest name + most
complete data + lex-older slug as tiebreaker.
Mealie integration:
- mealie.delete_recipe(slug) → DELETE /api/recipes/<slug>. Permanent.
Permission-scoped per-household (cross-household will 403).
UI:
- /dedupe-recipes — same shape as /consolidate but with side-by-side
recipe cells (canonical marked ★ KEEP, deletes marked × DELETE in
red). Source URLs link out so user can sanity-check.
- DEFAULT TO NOT-APPROVED — recipe deletion is destructive, user must
opt in per cluster. Bulk "approve all dupes" is one click but the
apply confirm explicitly counts how many recipes will die.
- Linked from /me alongside sterilize + consolidate.
Cobb confirmed earlier: "we can't lose recipe data" — answered by
(1) conservative Sonnet decisions, (2) opt-in default, (3) explicit
permanent-deletion confirm, (4) same-pattern logging + DB audit trail
on every attempt.
Bug: my anti-zombie guard from 4707e6a was too strict — WHERE clause
required state IN ('running','applying') to update. But the normal
flow goes running→review→applying→done. Once a job entered review,
NO state transition could fire — including the legitimate apply
sequence triggered by user clicking "apply selected".
Symptom Cobb hit: clicked apply on job 6, the daemon thread did the
work (11 of 13 proposals applied cleanly to Mealie), but the row
stayed at state='review' so the UI never moved off the review screen.
The 11 successful applies are real — Mealie has the updated
recipeIngredient food links. The bookkeeping just didn't follow.
Fix: change WHERE clause from a positive whitelist (running/applying)
to a negative blocklist (NOT IN done/failed/cancelled). This still
prevents the original failure mode (daemon overwriting a user-cancelled
job) — terminal states still can't be overwritten — but lets review
transition to applying when the user approves.
Same fix applied to finalize_consolidate_job since it copy-pasted the
same too-strict guard.
Two failures surfaced by job 6 with the bigger prompt (full Mealie food
catalog ~50KB + recipe context with steps + spell-cleanup rules):
1. quinoa-chili-with-sweet-potatoes: 180s timeout. The bigger prompt
means Sonnet has more to chew through per call. Bumped _parse_batch
timeout 180s → 300s. Recipes with many ingredients now get more
slack before clawdforge gives up.
2. salmon-sushi-bake: "unexpected response shape" — Sonnet returned
the JSON as a STRING rather than a parsed dict (depends on size +
how clawdforge unwraps the response). _parse_batch was strictly
requiring isinstance(result, dict) and rejecting strings outright,
leaving the recipe in error state with valid JSON visible in the
error message. Added defensive string→dict parsing (with optional
code-fence stripping) mirroring the pattern already used by
forge._extract_food_info.
Both errored recipes from job 6 can now be re-run cleanly. Apply path
unchanged — defensive food.id preservation from 6bcf79e still in
effect.
Three improvements driven by Cobb's review of the fan-out output:
1. Recipe context. _parse_batch now accepts an optional recipe_context
dict carrying recipe_name, recipe_description, and recipe_steps.
preview_recipe builds the context from the Mealie recipe and passes
it through. The Sonnet prompt has new USE RECIPE CONTEXT WHEN
AMBIGUOUS rules: "1 cup flour" is ambiguous (AP / bread / cake);
the cooking steps usually disambiguate ("knead until elastic" →
bread flour, "sift with cocoa powder" + cake recipe → cake flour).
Step text capped to 3000 chars so the user prompt stays modest;
defaults to all-purpose flour when steps don't disambiguate.
Brand/style hints in the description carry through too.
2. Spell + grammar cleanup. New SPELL/GRAMMAR CLEANUP rules in the
prompt: silently fix typos in food and note ("tomatos" → "tomatoes",
"chopped finly" → "chopped finely", "heavy cram" → "heavy cream").
Normalize spacing. Critically: preserve EVERY semantic value —
numeric quantities verbatim, every prep state, brand, color. When
uncertain whether something is a typo or intentional ("yellow
squash" is a real food, not a typo), keep it. Original strings
stay in originalText for audit / rollback.
3. Defensive food.id preservation in apply_recipe. Three new safeguards
protect against Sonnet hallucinations dropping live recipe data:
a) If Sonnet returns a single all-null parsed item but the original
Mealie row had a real food.id, pass the original through
verbatim. (Sonnet probably parse-failed; never blank a real link.)
b) When Sonnet returns a food name that we can't resolve in Mealie's
catalog AND the original had a food.id, preserve the original
link rather than emit food=null.
c) When Sonnet explicitly returns food=null on the first child of
an ingredient that originally had a food.id, treat that as a
misread and preserve the original. Real section headers — where
the original was ALREADY foodless — still pass through cleanly.
Net effect: no apply path can drop a recipe's existing food
reference. Sonnet can ADD food links (good), CHANGE them (good),
or fail to parse (we keep what was there). It cannot remove them.
The is_new_food field also benefits from recipe context — Sonnet has
more evidence to set is_new_food=false (matched a known canonical)
when the steps confirm the ingredient identity.
Migration 020 drops cauldron_pick_points (game system stripped from
/plan 2026-04-30 — all award_pick_points calls were already removed
from server.py). The DB methods household_scoreboard, household_streak,
and award_pick_points are deleted now too — they were unused dead code
since the strip. delete_plan_slots no longer DELETEs from the dropped
table (kept it minimal: just wipe the meal_plan_slots).
DEFERRED from Step 4: dropping cauldron_foods. The 229 Sonnet-curated
density rows + 2462 USDA seed rows are still useful raw material for
a fuzzy backfill into cauldron_food_metadata (only 128 of Mealie's
2895 foods got densities matched on the first exact-name pass). Until
we run a fuzzy backfill, holding onto cauldron_foods as a cold archive
is cheaper than losing the data. Will revisit once the metadata
catalog is more complete.
Recovery from 020: revert this migration, restore the CREATE TABLE
position-014 entry from db.py history. The points data was never
meaningfully populated (jobs 1+3 award attempts were folded into the
strip), so loss is essentially zero.
New bulk job that scans Mealie's foods table for duplicate-feeling
clusters, asks Sonnet to pick the canonical survivor + flag the rest
as merge candidates, and uses Mealie's PUT /api/foods/merge to
consolidate. After each successful merge, alias_additions get pushed
onto the survivor so Mealie's CRF/NLP parser fuzzy-matches the
discarded variant names from then on.
Architecture mirrors bulk_sterilize.py:
- Migrations 018+019 add cauldron_consolidate_jobs +
cauldron_consolidate_proposals (state machine: running → review →
applying → done/failed/cancelled)
- New consolidate_foods.py — daemon-thread runner with cancel-respect
and stuck-job recovery
- /api/foods/consolidate-{start,status,jobs,apply,cancel} for session
users + /api/admin/foods/consolidate-{start,jobs,cancel} for kayos
Sonnet integration:
- forge.cluster_decision(foods) → returns {merge, canonical_id,
canonical_name, discard_ids, alias_additions, reason}
- Conservative-by-default: when in doubt Sonnet returns merge=false
(the "olive oil vs olive" false-positive case from the prompt)
- Alias rules in the prompt explain why we want discarded names to
travel back to the survivor as aliases (parser future-proofing)
Mealie integration:
- mealie.merge_foods(from_id, to_id) → PUT /api/foods/merge
- mealie.update_food(food_id, body) → for pushing aliases onto the
survivor after merges land
- Apply path catches 403/permission errors and surfaces them as the
per-cluster apply_error (cross-household merge attempts will fail
here, same way as the sterilize cross-household path)
Clustering:
- rapidfuzz token_set_ratio ≥ 88 (slightly stricter than Mealie's
parser threshold of 85 to reduce false-positive clusters)
- Single-link agglomerative — O(n²) but Cobb's ~3000 foods = ~9M
comparisons, runs in seconds
- Singleton clusters (no merge candidates) are dropped, not stored
UI:
- /consolidate — same shape as /sterilize: progress bar → review grid
→ apply button. Cards show member chips with the canonical marked
★, discards marked × in red, alias_additions listed in green, plus
Sonnet's one-line reasoning. Mergeable approved by default; user
toggles individual clusters off if they disagree.
- Linked from /me → tools section, alongside bulk sterilize.
Total: ~600 LoC across 6 files. Foundation for the "Mealie owns
canonical names" architectural rule is now actually enforceable —
cobb runs this once, his foods table gets cleaned up, and Sonnet's
catalog-aware parser (Step 1) starts matching aliases for free.
System MEALIE_API_TOKEN is 401 (legacy 'Cauldron' token from Mealie
mint long ago, since rotated/expired). The backfill tries it first;
on 401 falls back to any stored per-user token via the new
db.first_usable_mealie_token helper. Cobb's user is admin in Mealie
and his token returns the full group catalog, so the backfill works
without minting a fresh system token.
Aligns cauldron's data layer with the architectural rule "Mealie owns
canonical food names; cauldron only owns cooking metadata Mealie can't
store". The old parallel name catalog (cauldron_foods, 2462 noisy USDA
rows + ~229 Sonnet-curated names) was always going to drift from
Mealie's foods table over time. Now metadata follows Mealie's UUID, so
when Cobb merges or renames a food in Mealie the density+unit_class
travels with it automatically.
Schema:
- New table cauldron_food_metadata (migration 017): primary key is
mealie_food_id VARCHAR(64); columns are density_g_per_ml,
common_size_g, default_unit_class, category, source enum (seed /
claude / manual), notes JSON.
- cauldron_foods table stays untouched in this step (Step 4 drops it
after the backfill ledger has been verified in production).
Code:
- cauldron/foods.py rewritten:
- get_metadata_by_food_id(db, mealie_food_id) — primary read
- upsert_metadata(...) — write keyed by mealie_food_id
- fetch_and_persist(db, mealie_food_id, food_name, forge) — Sonnet
fallback, persists keyed by id
- backfill_seed_from_legacy(db, mealie) — one-time migration helper
called at app boot when metadata table is empty. Walks Mealie's
foods, looks up each in legacy cauldron_foods by name/plural/alias,
copies density into the new table keyed by Mealie's id. Returns
{matched, missed, total_mealie} stats.
- Legacy shims (search_food, upsert_claude_food, load_seed_if_empty)
kept as no-ops so server boot doesn't break before full cutover.
- cauldron/aggregator.py:
- Ingredient.mealie_food_id new optional field
- aggregate() now keys by mealie_food_id when present, falls back to
normalized name. Verified with rice-from-3-recipes synthetic:
same id → consolidates to "2.25 lb rice" single line as before.
- foods_lookup callable signature changed to (name, food_id) — id is
primary, name is for display + Sonnet fallback.
- cauldron/server.py:
- /list view captures Mealie's food.id from each recipe ingredient
and threads it through the Ingredient. foods_lookup now does an
id-keyed cauldron_food_metadata read; on miss with a known id,
calls forge.fetch_food_info and persists. When food.id is missing
(ingredient still in note form, no Mealie row linked), returns
None and aggregator falls back to name grouping.
- Boot: replaces the USDA seed loader with a one-time backfill of
legacy cauldron_foods → cauldron_food_metadata via the system
Mealie token. Runs only when metadata table is empty.
Net effect: rice in 3 recipes that all link to the same Mealie food
row now group by UUID, not by lowercased name. When Mealie's foods
get cleaned up (Step 3 consolidator), cauldron's metadata follows
because the ids are stable. Foundation for the consolidator is now
in place.
Sonnet was picking food names blindly. We then tried to match those
back to Mealie's catalog post-hoc. When Sonnet's natural pick didn't
match Mealie's exact convention, we'd create a duplicate row instead
of reusing the existing one. Lucky alignment with the seed kept the
dupe rate low, but the architecture had no real "Mealie is source of
truth" guarantee.
This change makes that guarantee explicit:
1. Sterilizer now lazy-loads Mealie's full food catalog on first
_parse_batch call (one fetch per Sterilizer instance, so a bulk
job pulls 2895 rows once and reuses across all 226 recipe parses).
Uses the underlying mealie._get with per_page=2000 + page-walk for
defensive coverage of really large catalogs.
2. STERILIZE_SYSTEM is now STERILIZE_SYSTEM_TEMPLATE — a string with
a {foods} placeholder. _system_prompt() splices in a bullet list of
every Mealie food (name, plural, aliases) at runtime.
3. New CATALOG RULES in the prompt instruct Sonnet to:
- Match against name / pluralName / aliases first, return canonical
name verbatim with is_new_food=false
- Strip prep modifiers into note
- Singularize plurals when canonical is singular
- Treat brand variations as canonical+note ("Heinz ketchup" →
food: "ketchup", note: "Heinz")
- Set is_new_food=true ONLY when no reasonable catalog match exists,
since adding aliases to fix mismatches later is way easier than
cleaning duplicate food rows after the fact
4. New is_new_food field on IngredientParse and per-item schema. Will
eventually drive an "alias suggestion" UI, but for v1 just gives
us telemetry on how often Sonnet falls back to inventing names.
Net effect for the family-internal goal: zero duplicate food creates
from convention mismatches, smarter parses that respect the catalog
Cobb spent time curating, foundation laid for the Step 2 re-key
migration where cauldron_food_metadata gets keyed by Mealie food.id.
Cobb spotted job 4's proposals weren't actually doing useful work —
"Toppings (Cinnamon Butter, Marshmallows, Ground Cinnamon, Butter, Etc)"
came back unchanged because the prompt was rigidly 1-in-1-out and
treated the whole compound line as a section header. Same for
"salt and ground black pepper to taste" — should be 2 separate
shopping list items but the parser kept them as one note.
Three changes:
1. STERILIZE_SYSTEM rewritten to allow fan-out. New return shape is
list-of-lists: outer list mirrors input length, each inner list
has 1 item (normal case) or N items (fan-out). Explicit fan-out
rules cover the two patterns Cobb cares about:
- "salt and pepper" / "X and Y to taste" → 2 items
- "Toppings (a, b, c, etc)" / "Optional: A, B, C" → N items,
wrapper word dropped, filler ("etc") skipped
Plus a heuristic against accidentally splitting compound food
names ("salt and vinegar chips", "macaroni and cheese" → keep).
2. _parse_batch + IngredientProposal + apply_recipe all updated for
the new shape. IngredientProposal.parsed → parsed_items: list.
apply_recipe iterates each child:
- First child inherits the original Mealie row's id/refId/
originalText so existing references stay live
- Additional children are fresh dicts; Mealie generates ids
on save when none provided
Backward-compat fallback in apply_recipe accepts the legacy
single-parsed shape so any in-flight job 4 proposals still apply
cleanly.
3. /sterilize UI was→becomes table now:
- Renders one row per parsed child (rowspan'd "was" cell when
fanning out, with a "→³" superscript marker on the arrow)
- Drops identity rows (1→1 case where parse matches original
verbatim) so the diff shows only ACTUAL changes — fixes Cobb's
"this doesn't look sterilized at all" complaint where every
diff was identical
- Cards with all-identity proposals show "no changes proposed
(all ingredients already matched)" instead of an empty table
Job 4's stored proposals use the legacy 1→1 shape so won't show fan-out
until re-walked. Recommend cancelling job 4 and starting a fresh job 5
with the new prompt to see the toppings line break out properly.
Two small adds:
(1) auth/callback no longer 500s on transient JWKS-fetch failures or
on a stale state cookie from a prior failed callback. Catches three
distinct failure modes and renders templates/auth_retry.html with
a 503 / 400 status:
- RequestsConnectionError + Timeout → "couldn't reach the auth
server — usually a momentary DNS or network blip"
- MismatchingStateError → "that login link expired (you probably
retried after a blip). hit login again to start fresh"
- OAuthError → "auth handshake failed: <detail>"
All three paths pop _state_cauldron_authlib so the next /login
starts with a fresh state token. Today's incident: cauldron
container hit a brief auth.sulkta.com DNS resolution failure
during OIDC callback, threw a stack trace, then the user's retry
failed differently because of the leftover state cookie.
(2) /api/admin/sterilize/bulk-start (require_bearer) accepts
{"started_by_sub": "user@example.com"} body. Resolves that user's
household, decrypts their stored Mealie token, spawns the same
preview thread the session-authed endpoint does. The job appears
in /sterilize for the user as if they'd started it themselves.
Plus admin-status (GET /api/admin/sterilize/jobs/<id>) and
admin-cancel (POST /api/admin/sterilize/bulk-cancel/<id>) for
operator symmetry. Existing user-session endpoints unchanged.
Net: kayos can kick off + cancel + monitor sterilize jobs from the
outside without piggybacking on Cobb's browser cookie, and a brief
DNS hiccup on the LAN no longer turns into a "your login is broken"
experience for the user.
Job 3 surfaced the bug — when I set state=cancelled in the DB, the
daemon thread kept running and finalize() at the end overwrote it
with 'done'. User cancellations were getting silently undone.
Two changes:
1. Runners (run_bulk_preview, run_bulk_apply) now check the job's
current state at the top of every iteration via the new lightweight
db.get_sterilize_job_state. If the state has moved to a terminal
value (cancelled, failed, done) externally, the loop returns
without finalizing.
2. db.finalize_sterilize_job now refuses to overwrite a non-running
state — added "AND state IN ('running','applying')" to the WHERE
clause. Belt-and-suspenders for the same problem: even if a runner
races past the state check and limps to its finalize call, the DB
itself won't let the cancellation be replaced.
Net: hitting cancel via the UI button (or a DB update) now actually
stops the runner mid-flight. Polling roundtrip per recipe is one
SELECT — negligible vs the multi-second clawdforge call that
dominates each iteration.
Mealie's group model spans multiple households (Hayes House group has
Redondo + Lake Elsinore — Cobb's family + Bay/mom share recipes
read-only across households). Members can list/read every recipe in
the group, but write access is per-household. Trying to sterilize a
foreign-household recipe returns 403 on the PUT; the food/unit creates
that ran first end up as orphan rows in the user's own household.
Walk path now resolves /api/users/self.householdId once (cached on the
Mealie client), and skips any recipe whose top-level householdId differs.
No proposal row is created; the recipe just counts as "skipped" along-
side already-clean ones.
Apply path adds the same defensive check (covers job 1's existing
proposals from before this fix landed) and translates any remaining
403 to a friendly "skipped: recipe belongs to a different household —
sterilize from that household's account" message instead of dumping
Mealie's raw permission-denied JSON into the UI.
Net: Cobb sterilizes Redondo recipes from his account; Bay or mom
each get their own walk-and-apply scoped to Lake Elsinore when they
sign in. Same architecture works for new households joining Sulkta —
each member's bulk job is automatically scoped to the household they
belong to. Zero cross-pollution.
Job 1's bulk run apply'd 184 recipes and 182 of them failed with the
same error: POST /api/foods -> 400 UNIQUE constraint failed:
ingredient_foods.name, ingredient_foods.group_id. Cause: Mealie's
name_normalized strips punctuation/whitespace/case more aggressively
than our local _build_name_index's plain .lower(), so the cache misses,
the create_food fires blindly, and Mealie's UNIQUE constraint kills
the call. Whole-recipe apply was wrapped in try/except at the bulk
runner so the recipe got marked errored — but applied_at was still
set to NOW(), making the rerun think we'd already tried. We had, but
the recipe's still unparsed.
Two fixes:
1. sterilizer._resolve_food / _resolve_unit replace the inline
create-on-miss block. Order: local cache → Mealie search-endpoint
tie-break → create. On any UNIQUE-flavored 400 from create, fall
back to one more search to adopt whatever Mealie has under the
normalized form. Mealie's search endpoint applies its own
name_normalized internally so we don't have to mirror its rules.
_search_for_match takes "foods" or "units" and looks for an exact
case-insensitive match against name or pluralName, with a fallback
to "trust Mealie's ranker" when there's exactly one hit.
2. db.mark_proposal_applied no longer sets applied_at on error. On
success: applied_at=NOW(), apply_error=NULL. On error: applied_at
stays NULL, apply_error gets the message. list_approved_unapplied_
proposals keys off applied_at IS NULL, so a rerun naturally retries
only the failed recipes.
Net effect: rerun can now successfully apply the 182 failed recipes
without re-walking them, and won't waste calls on the 2 that did go
through.
Mealie's "Parse" button on every recipe is per-recipe; clicking through
226 of them is the toil Cobb explicitly hates. The bulk sterilizer wraps
cauldron's existing per-recipe Sterilizer (clawdforge → Sonnet, parses
free-form ingredient strings into structured form) into a household-wide
job with progress tracking, batch human review, and one-shot apply.
Flow:
1. POST /api/sterilize/bulk-start — creates cauldron_sterilize_jobs row,
spawns a daemon thread that walks every recipe in the user's
household. Recipes whose ingredients all already have food.id are
skipped (no point re-parsing Cobb's manual cleanup). For each
recipe needing work, Sonnet returns a structured proposal that gets
persisted to cauldron_sterilize_proposals.
2. GET /api/sterilize/bulk-status — polled every 2s by the UI for
{state, processed_count, skipped_count, error_count, current_slug}.
3. After the walk completes, state moves to 'review'. UI loads
/api/sterilize/bulk-jobs/<id>/proposals and renders one card per
recipe with a was→becomes diff per ingredient. User toggles
approve/skip per recipe.
4. POST /api/sterilize/bulk-apply/<id> with {approved_slugs: [...]}.
A second daemon thread iterates approved proposals, calls
Sterilizer.apply_recipe (which resolves food/unit IDs in Mealie,
creating any missing rows, then PUT /api/recipes/<slug>).
Job state machine: running → review → applying → done (or 'cancelled' /
'failed' along the way). At app startup, fail_stuck_sterilize_jobs()
recovers any 'running' / 'applying' rows older than 10 min with no
progress — covers the case where the daemon thread's gunicorn worker
died mid-job. New job state lives in DB; thread is just a runner.
Concurrency: the start endpoint blocks if the household already has a
running/applying job. Sterilize calls cost clawdforge time and parallel
jobs would race on Mealie writes, so one-at-a-time per household is the
right ceiling.
UI lives at /sterilize. Linked from /me → "tools → bulk sterilize" since
it's a one-off admin action, not part of the daily flow. Mobile-aware,
diff cards expand on click. Disabled "apply" button when no recipes
are selected; preview-error rows can't be approved.
DB: two new tables (migrations 015+016). cauldron_sterilize_jobs tracks
overall job state with last_progress_at for the stuck-job recovery;
cauldron_sterilize_proposals holds per-recipe JSON proposals with the
approval flag and the final apply_at/apply_error.
Two changes:
1. foods catalog grows organically. Switch the canonical seed from the
noisy USDA dump (2462 rows of "'s, classic chicken noodle soup")
to the Sonnet-curated cut (229 clean rows). search_food() is now
exact + case-insensitive — Mealie's parser already canonicalizes
food names household-side, so cauldron just needs to look them up
verbatim. On miss, the /list view calls forge.fetch_food_info() to
ask Sonnet for {density_g_per_ml, default_unit_class, common_size_g,
category}, persists the row with source='claude', and the household's
actual kitchen catalog builds itself out as Abby uses it.
Killer case verified end-to-end: "2 cups + 50g + 1.25 lb rice"
collapses to a single "2.25 lb rice" line on the shopping list once
rice has a density row.
2. Game system stripped from /plan. Scoreboard panel, streak banner,
"first to lock takes the week" / "🏆 you locked this one in" copy
all gone. award_pick_points calls in /api/plan/generate +
/api/plan/regenerate stopped firing. household_scoreboard /
household_streak DB methods kept as dead code; cauldron_pick_points
table left in place — non-destructive, easy to revive later if
gamification comes back. Goal: get the base flow (pick → plan →
list) working for Abby first, layer features on after.
- migrations 012 + 013: cauldron_meal_plan_slots + cauldron_pick_points
- db: list_plan_slots, save_plan_slots, delete_plan_slots, mark_plan_generated,
clear_plan_generated, award_pick_points, enrich_plan_with_slots; scoreboard
extended with points (sum from pick_points) and weeks_locked alias
- forge.generate_plan: sonnet prompt builds 7-day plan respecting picks,
validates slot count + day uniqueness + slug-in-pool, fills picker_subs
from ground-truth picks (model output is advisory)
- POST /api/plan/generate: race-safe (existing slots → 409 with plan),
lock-aware (locked → 409), idempotent
- POST /api/plan/regenerate: re-roll for the original generator, gated by
ownership + lock; wipes slots + pick_points then re-runs generate
- plan.html: generate CTA + 7 day cards with picker chips + AI reason +
re-roll button (generator-only, pre-lock); scoreboard now shows points + wins
- /list: pulls plan slots, queries Mealie for ingredients, runs aggregator,
renders 48px-tall checkbox shopping list with localStorage state per plan_id
- tests: 13 new tests across forge.generate_plan + /api/plan/generate routes
+ /list view + scoreboard SQL inspection. conftest+_testenv stub
pymysql/oidc/foods at import time so tests run against module-level app
without a live DB. Both pytest and `unittest discover` paths green (27/27).
Defers: bulk sterilizer admin (A1), foods dedupe (A2), Mealie shopping-list-
export (button rendered but disabled). 7-slot count is fixed at the
endpoint (no UI for slot-count selection yet).
Spec: memory/spec-cauldron-v0.3.md
Pure-Python module + 14 unit tests proving the centerpiece works:
test_rice_mixed:
in: [(2 cup, rice), (1.25 lb, rice)]
out: 2.25 lb rice (one line, properly mass+volume combined via density)
test_butter_mixed:
in: [(0.5 cup, butter), (4 oz, butter)]
out: ~227g butter (~8oz / 0.5 lb)
test_three_recipes:
feeds 9 ingredients across 3 recipes through the aggregator;
rice (cup + lb) collapses, garlic (cloves) sums, eggs count, salt as 'pinch'
bucketed as to-taste. All on one shopping list.
Algorithm in cauldron/aggregator.py:
1. Bucket ingredients by canonical food (foods_lookup callable injected — no DB coupling)
2. Within each food, classify each unit (mass / volume / count / vague / unknown)
3. CASE 1: only one unit class present → simple sum, display in canonical store-friendly unit
4. CASE 2: mass + volume (the killer) → use density_g_per_ml to combine to grams
5. CASE 3: count + (mass | volume) → use common_size_g to convert count to grams
6. CASE 4: anything that can't reconcile (no density, mixed unknown) → split into 1 line per class with is_split=True
7. vague (pinch, dash, to taste) → annotate as 'plus to-taste'
8. unknown units → emit verbatim with the original text
Display: store-friendly unit picker:
<30g → grams
<500g → ounces (nearest 0.5)
<2kg → pounds (nearest 0.25)
>2kg → big pounds
The aggregator is dependency-injection-friendly — foods_lookup(name) is
the only external call. Tests pass a stub dict; production will pass
foods.search_food(db, name). Decouples math from data quality.
Tests run via:
python3 -m unittest discover -s tests -v
Phase A foundation. Cobb 2026-04-29: 'go big or go home' on density-table
aggregator — this commit lands the schema + seed data so the aggregator
engine has something to look up against in step 2.
DB:
- migration 010: cauldron_foods (canonical_name PK, density_g_per_ml,
default_unit_class enum mass/volume/count/mixed, common_size_g,
category, usda_fdc_id, source enum)
- migration 011: cauldron_food_mapping (per-household Mealie food_id →
cauldron canonical food_id, used by aggregator + foods-dedupe later)
Seed data:
- scripts/build_foods_seed.py — extractor that walks USDA SR Legacy
foodPortions, derives density g/ml from cup/tbsp/tsp/fl-oz/ml/etc
measurements (handles SR Legacy's quirk of putting unit in 'modifier'
with measureUnit.name='undetermined'), filters out babyfood / branded
/ fast-food / alcoholic-beverage clutter, normalizes names, categorizes
via longest-keyword-wins
- cauldron/data/foods_seed_usda.json — 2,462 foods with density values
derived from USDA. 636KB, ships in the image.
- cauldron/data/README.md — regen instructions + known issues / iteration
plan (next pass: claude-curated cleanup → ~500-800 high-relevance entries
+ count-based foods like egg/onion that USDA doesn't cover)
Loader (cauldron/foods.py):
- load_seed_if_empty(db) called on app startup right after migrate().
Idempotent — won't reload if table is non-empty.
- reload_seed(db) for forced reloads (INSERT IGNORE).
- search_food(db, name) helper for the aggregator + UI.
Categories present in seed:
produce-vegetable: 300, spice: 256, dairy: 207, condiment: 197,
legume: 189, meat: 166, beverage: 153, baking: 129, produce-fruit: 128,
oil-fat: 126, nut-seed: 115, grain: 89, other: 407
The 407 'other' bucket and the verbose USDA names ('mayonnaise, reduced
fat, with olive oil') will get cleaned up via clawdforge in step 3.
For now the aggregator can already do the math against this seed; the
unit-conversion engine is the next commit.
Cobb: 'searching recipes is a bit off. lets make that way way more on
point. need to be the google of recipe searching.'
Architecture:
- New cauldron_recipe_index table mirrors enough of Mealie's recipe shape
to fuzzy-rank locally without round-tripping. Migrations 008+009.
- Refresh on first /recipes load + every 5 minutes + on-demand button.
Single page-200 pull from Mealie covers Cobb's 226 recipes in one trip.
- recipe_index.py — flatten_recipe(), refresh_household_index(),
search_index().
Search algorithm (rapidfuzz):
- Multi-field weighted: name×1.00, tags×0.85, cats×0.80, foods×0.70,
ings×0.55, description×0.45 (max-of wins, not sum, to avoid noise spike)
- Three scorers per field: WRatio (overall), partial_token_set_ratio
(handles 'spag bol' → 'Spaghetti Bolognese'), token_set_ratio
(order-independent)
- Substring-of-query in title bonus +20
- Floor 50 to filter junk
- Top-80 returned
API:
- /api/recipes.json now uses local index for both search and browse
- /recipes route same — first-page server-render from index
- POST /api/index/refresh — manual refresh button (admin-y)
- ?q=... → ranked fuzzy results, paginated
- no q → ordered browse from index, paginated, has_next via lookahead
Performance:
- Local index query: ~5ms for browse
- Search across 226 rows × 6 fields × 3 scorers: ~60ms
- Should feel instant compared to Mealie's network round-trip
Cobb caught it. Mealie's web URL format is:
https://recipes.sulkta.com/g/hayes-house/r/dairy-free-bread-recipe
Was emitting /recipe/<slug> which 404s.
Added _user_group_slug() helper that pulls .group.slug from /api/users/self
(handles dict-or-string-or-fallback shapes for cross-version compat),
threads the proper URL through both:
- /api/recipes/<slug>.json (used by the modal)
- /recipes/<slug> server-rendered detail (used as fallback for direct links)
Falls back to the old /recipe/<slug> path if we can't resolve a group
slug (won't break, will just 404 in Mealie if that ever happens).
Cobb: 'new page resets all filters. bad flow.' Right.
Modal pattern:
- Click any .recipe-card anywhere → fetches /api/recipes/<slug>.json,
opens a full-screen-on-mobile / centered-on-desktop modal
- The underlying recipes page (search query, sort chip, category chip,
scroll position) stays put
- Browser back button closes the modal (history.pushState + popstate
listener)
- /recipes/<slug> still works server-side as fallback for direct links
and shared URLs (no auto-modal-open on page load)
Implementation:
- New endpoint GET /api/recipes/<slug>.json — same data the detail page
used, plus a public_url field for the 'in mealie ↗' deep link
- Modal HTML lives in _base.html so it's available everywhere
- Click delegate at document level catches .recipe-card clicks app-wide;
Mushroom button click bubbles back out (skipped) so pin toggle still
works without opening the modal
- Modal contains: title bar (sticky), scrollable body (recipe meta pills,
description, ingredients ticked list, numbered instructions), sticky
footer with pin button + 'in mealie' link
- Pin button in modal syncs the matching card's picked state on close so
the grid reflects the change without a refetch
Mobile:
- Modal slides up from bottom, takes full viewport-1 of height
- Close button is 40px circle, top-right
- backdrop-blur 6px, body scroll-locked while open
- Tap outside the modal card → close (e.target === backdrop check)
- Esc key closes
- Cmd/Ctrl/Shift/middle-click still opens in new tab (don't intercept)
Desktop:
- Modal centered, max-width 720px, max-height 86vh
- Backdrop has 30px padding so the card has breathing room
The /plan crash:
AttributeError: 'str' object has no attribute 'get'
in sync_user_household — Mealie's user-self response sometimes returns
household as a slug-string (newer versions) instead of a dict.
Fix: defensive — accept dict OR str, also fall back to top-level
householdId / householdSlug fields. Doesn't crash if all are missing.
Mobile + UX pass (Cobb 2026-04-29 — 'two kids screaming at the store'):
- Mushroom pick button bumped 30px → 48px (Apple HIG min tap target),
inner SVG 16 → 24px, kept opacity transition. Stays one-handed-friendly.
- Recipe cards: padding 14px → 18px, min-height 88px, font-size 1.08em
→ 1.2em on the title. Bigger comfortable tap area.
- Recipe grid: 1col by default, 2col at 720px+, 3col at 1100px+ (was
560/900). Phones get single column, the bigger card.
- Search bar: 14px → 16px font (kills iOS auto-zoom on focus), 8px → 10px
padding. Now sticky at top of /recipes with backdrop-blur.
- Pill chip rows for sort + category — horizontally scrollable, hidden
scrollbar, pill rounded shape, active state in purple-deep bg.
- Sort options: newest, recent (last_made), a-z, updated. Default = newest.
- Category chips: pulled live from Mealie's
/api/organizers/categories, top 14 shown. 'all' chip clears filter.
Mealie client: list_recipes() now accepts orderBy + orderDirection +
categories[] + tags[] params. New list_categories() helper.
Server: _sort_to_order() maps our sort keys to Mealie's orderBy. Recipes
+ json endpoints both honor sort + cat query params and pass through to
the AJAX paginator.
DB:
- migration 006 — cauldron_households (mirrors Mealie household), members
- migration 007 — cauldron_meal_plans (per household per week, lock state)
- new helpers: upsert_household, add_household_member, get_user_household_id,
list_household_member_subs, get_or_create_plan, lock_plan,
auto_lock_past_unlocked_plans, household_scoreboard, household_streak,
list_household_pick_slugs, list_household_picks_with_pickers
Backend:
- sync_user_household() — pulls Mealie's /api/users/self, upserts the
household row, ensures membership. Fires on /connect-mealie POST and
lazy on every /me / /picks / /plan / /recipes load.
- current_household_id() helper used by all user-facing routes
- /recipes + /api/recipes.json now mark items.picked=True if ANYONE in the
household has pinned (shared pool, not per-user)
- /picks now renders household-pooled view with attribution (who pinned what)
- /plan replaces stub: shows current week's lock state + lock button +
scoreboard + streak. POST /api/plan/lock locks this week's plan with
reason='user'. Past unlocked weeks auto-lock on read with reason='auto'.
- /list still stub (v0.3 territory)
UI:
- picks.html: each pick shows '🍄 pinned by <name(s)>' attribution row,
unpin button only on your own picks, household_size in lede
- plan.html: NEW. Lock state pill, lock button (open weeks only), scoreboard
table with rank/wins/last_win, streak ribbon ('🔥 abby on a 3-week run')
if streak >= 2
- me.html: shows household name + member count
Locking semantics match Cobb's pick:
- (c) both — user-lock = scoreboard win, auto-lock past calendar = no win.
Auto-lock fires lazily on /plan view (no cron needed for v0.2).
- Picks pool = (a) shared across household.
- Game shape = medium (locked-by + tally + streak counter, room for richer
badges/notifs in v0.4).
Cobb pass: more mobile friendly + infinite scroll + auto-search +
add-to-AI-plan picks list + mushroom vibes.
DB:
- New migration 005 — cauldron_meal_picks (authentik_sub, recipe_slug,
recipe_name, added_at). Per-user wishlist for the v0.3 AI meal-plan
generator. add/remove/list helpers in DB.
Backend:
- GET /api/recipes.json?page=N&q=Q — paginated + searchable, returns
{items, page, total, total_pages, next}. Each item is annotated with
picked: bool from the user's pick set.
- POST /api/picks/<slug> — add (slug + optional name body)
- DEL /api/picks/<slug> — remove
- GET /api/picks.json — list current picks
- GET /picks — picks page (replaces empty stub)
- Mealie.list_recipes() now accepts search=...
Frontend:
- recipes.html rebuilt:
- sticky search bar with 250ms debounce, hits /api/recipes.json?q=
- IntersectionObserver-driven infinite scroll, loads page+1 when the
sentinel comes into view (200px rootMargin, AbortController for
in-flight cancel on new search)
- per-card mushroom toggle button (top-right) — POST/DELETE to picks
with optimistic UI flip + rollback on failure
- picked cards get a left purple-glow stripe + tinted background
- _recipe_card.html partial — first-page server-render shares markup with
JS-rendered subsequent cards (mushroom SVG inline, same shape)
- recipe_detail.html — '🍄 pin for ai plan' button toggles state in place
- picks.html — list of current picks with remove button + v0.3 explainer
- Topbar nav: dropped /home, added /picks
Mushroom vibes:
- Hand-rolled SVG toadstool (purple cap, bone stem, dark spots) used as
the pick toggle icon — it's the gesture itself
- Same mushroom tiled into the body bg pattern at ~5% opacity in the
bottom-right of the 160px sigil tile, alongside the existing pentagram
- Mushroom emoji on the detail page button + picks page nudge
Mobile pass:
- Topbar nav scrolls horizontally on narrow screens, brand-sub hidden
under 720px, larger tap targets on cards, font-size pulled in slightly
- Recipe grid: 1 col <560, 2 col 560-900, 3 col 900+
- Page-head + button + card padding all tightened on small screens
Cobb: 'more gothic somehow.' Pushed in concrete elements:
- Drop cap on h1 page titles (first letter 1.5em in witch-purple with deep
glow; lowercase rest) — illuminated-manuscript feel
- h1 also lowercase + serif italic for the .accent span (blood-wine color)
- Drop cap on recipe-card .rname (first letter in blood-wine, jumps to
glow on hover)
- Panels now have gilt 18px corner brackets (top-left + top-right) via
::before/::after — very thin gothic frame, brightens on hover
- Inset 1px highlight on top edge for stone-and-glaze depth
- Three panel variants: .green (gilt brackets stay gold), .purple, .blood
- Panel h2 prefixed with ✦ in blood color (decorative bullet)
- New CSS vars: --blood (#7a2c4a aged bordeaux), --blood-bright (#a83d62),
--gilt (#8a6e3a aged brass), --gilt-bright (#c0a058)
- Brand mark Cauldron now flanked by ✦ in blood color
- Crumb breadcrumbs prefixed with ❧ (heart-petal) in gilt
- Lede italic with leading curly quote in gilt
- HR is now a custom SVG flourish divider (gilt curlicue with blood center
dot) instead of a plain line
- Top-of-page-head 60px gilt fade rule
- Body bg replaced pentagram tile with a more atmospheric cross-and-vine
glyph plus a subtle full-page corner vignette darkening (NOT cWHO scanlines,
just radial darken)
- Brand letter-spacing bumped to .22em for that ceremonial feel
Cobb feedback: pull back from cWHO terminal-coded look, blend Abby's gothic
witch with sulkta.com polish. 'burn tokens till we nail it.'
Style direction:
- Polished dark base like sulkta.com — soft purple/green radial glows on
near-black, faint witchy pentagram-circle SVG bg pattern at 5% opacity
- NO scanlines, NO CRT vignette overlay (cWHO is too terminal for this)
- Inter for body (sulkta.com), Cinzel SERIF for h1/h2/brand/recipe-card
titles (gothic flourish), JetBrains Mono for code/labels/uppercase chips
- Soft glow shadows (rgba box-shadow) instead of hard cWHO 3px offsets
- Rounded corners 4-6px throughout
- Smooth fade-in animations on .panel and .page-head
- Pills with subtle background tint (sulkta pill style)
- KV labels in mono uppercase purple — kept the gothic occult-tag feel
- Recipe cards lift on hover with purple glow shadow
Templates extracted from server.py to cauldron/templates/:
- _base.html — full layout shell, topbar, scanlines REMOVED, animation
- me.html — uses {extends '_base.html'}
- connect.html — same
- recipes.html — NEW, paginated grid view
- recipe_detail.html — NEW, full recipe with ingredients + instructions
- stub.html — NEW, placeholder for /plan and /list (v0.3)
Routes added:
- GET /recipes — user-tier: list via current_user_mealie()
- GET /recipes/<slug> — user-tier: detail view
- GET /plan, /list — stubs so nav doesn't 404
Server:
- render_template_string → render_template (proper Jinja file lookup)
- Stripped inline _PALETTE_CSS / ME_TEMPLATE / CONNECT_TEMPLATE constants
- Added current_user_mealie() helper to all user-facing routes
Cauldron's container can't resolve 'recipes.sulkta.com' from inside the
sulkta+sulkta-db-net bridges (Lucy's split-horizon doesn't propagate to
container DNS). Symptom: 500 on /connect-mealie POST when validating the
pasted token.
Fix: take the LAN-internal HTTP path direct to mealie. Mealie shares
OpenVPN-rack2's netns, listening on 9000 inside that netns. Both cauldron
and OpenVPN-rack2 are on sulkta-db-net (172.30.1.0/24), so cauldron talks
to 'http://OpenVPN-rack2:9000' via Docker's internal DNS — bypasses
Apache/HTTPS termination on Rackham entirely.
The public URL stays in the UI (so the connect-mealie page deep-link to
mint a token still goes to https://recipes.sulkta.com via the user's
browser, which DOES resolve it).
Also tightened Mealie._get/_put/_post to wrap requests.RequestException
into MealieError so connection failures don't 500 callers.
Cobb redirect: break away from forest-only, go darker. Halloween / witch /
underground / occult feels.
Palette flipped:
Black #050505 deep, #0a0a0e crypt, #110a1a vault
Purple #1a0d24 eggplant, #3d1f5a witch, #5a2d8c violet, #b878ff hex
Green #1a2611 moss, #2a3a1d swamp, #5a8c3a poison, #88c060 toxic, #9bff5a acid
Bone #d8c8a8 main text, #c9b27c warn
Fonts replaced:
Display Cinzel (sharp Roman caps, occult/witch trial vibe)
Body JetBrains Mono (terminal underground)
Brand Nosifer (the dripping-blood Halloween display face — used only
on the .brand tagline as a single bright accent)
Hard edges everywhere: no border-radius (square corners), no smooth
gradients on elements (only on the body bg as deep purple/green vignette
into black), 3px solid drop-shadows on button hover (the brutalist offset),
sharp 1px borders.
Specifically:
- h1 in toxic green (Cinzel 900, uppercase, letter-spaced) with a 0/24px
acid-green outer glow
- h2 in hex purple, Cinzel 700, uppercase, with a 1px witch-purple
underline
- Panel left-border alternates 3px violet → poison-green stripes
- KV labels in hex purple uppercase, occult-tag style
- Buttons rectangular Cinzel uppercase, 3px hard offset shadow on hover
- Brand mark in Nosifer (dripping)
- Code blocks on eggplant bg with hex purple text
- Selection color: violet on bone
Body bg = pitch black (#050505) with two radial vignettes — witch-purple
top-left, swamp-green bottom-right — bleeding into the black like
candlelight under a coven door.
Cobb course-corrected — wants more purple. Palette now:
Greens forest #1f2d1f, panel #2d3a2a, swamp #3a4a35, meadow #6b8e5a/#88a87a
Purples deep #2a1f3a, amethyst #4a2d5e, heather #6b4a8a, light #9b78c4
Cream #f0e6cc text, #ddd4ba lede, #c9b27c warn
Where purples land:
- h1 in heather (was meadow); h2 stays meadow for hierarchy
- Background a subtle linear-gradient toward #251f30 in the lower right
- Panel left-border accent in heather
- Links default to light heather; underline in deep amethyst
- kv dt labels in heather small-caps for the magic-grimoire feel
- Code blocks on deep purple bg
- Form input focus ring in light heather
- New .btn-purple variant for primary purple actions
- HR uses a gradient image-border heather→swamp→fade
- Brand tagline in heather italic small-caps
Greens still own structure (panels, body bg base, secondary headers,
buttons primary action). Purples are the magic.
- New ME_TEMPLATE — palette-locked, shows user identity + Mealie connection
status + connect/disconnect actions + sign out
- /me.json kept for programmatic callers
- Extracted _PALETTE_CSS shared between /me and /connect-mealie templates
(forest #1f2d1f bg, panels #2d3a2a, meadow accents #6b8e5a/#88a87a,
parchment text #f0e6cc/#ddd4ba, Cormorant Garamond serif headers)
- /me also fetches the Mealie /api/users/self for the connected user so
the page can show 'logged in as <username>, admin: yes/no'
- Connect page polished with cancel button + autocomplete=off on the token
input
Strict palette: no purple, no neon. As locked.
