Commit graph

66 commits

Author SHA1 Message Date
0bd4e7bd19 chore(deps): update rust crate clap to v4.6.5
Some checks failed
renovate/stability-days Updates have met minimum release age requirement
gitleaks / scan (push) Failing after 1s
gitleaks / scan (pull_request) Failing after 1s
2026-08-04 07:02:37 +00:00
c0f75c1365 Merge pull request 'chore(deps): update rust crate tokio-util to v0.7.19' (#19) from renovate/tokio-util-0.x-lockfile into main
Some checks failed
gitleaks / scan (push) Failing after 12m8s
2026-07-24 23:52:22 -07:00
9c7c950842 chore(deps): update rust crate tokio-util to v0.7.19
Some checks failed
renovate/stability-days Updates have met minimum release age requirement
gitleaks / scan (push) Failing after 1s
gitleaks / scan (pull_request) Failing after 1s
2026-07-25 06:52:15 +00:00
b3610c87ab Merge pull request 'chore(deps): update rust crate clap to v4.6.4' (#18) from renovate/clap-4.x-lockfile into main
Some checks are pending
gitleaks / scan (push) Waiting to run
2026-07-24 23:52:05 -07:00
686b0359ad chore(deps): update rust crate clap to v4.6.4
Some checks failed
renovate/stability-days Updates have met minimum release age requirement
gitleaks / scan (push) Failing after 1s
gitleaks / scan (pull_request) Failing after 1s
2026-07-25 06:51:57 +00:00
004f682b50 Merge pull request 'chore(deps): update rust crate clap to v4.6.3' (#17) from renovate/clap-4.x-lockfile into main
Some checks failed
gitleaks / scan (push) Failing after 1s
2026-07-23 23:51:16 -07:00
29dd84c946 chore(deps): update rust crate clap to v4.6.3
Some checks failed
renovate/stability-days Updates have met minimum release age requirement
gitleaks / scan (push) Failing after 1s
gitleaks / scan (pull_request) Failing after 1s
2026-07-24 06:51:08 +00:00
f826346934 Merge pull request 'chore(deps): update rust crate thiserror to v2.0.19' (#16) from renovate/thiserror-2.x-lockfile into main
Some checks failed
gitleaks / scan (push) Failing after 1s
2026-07-22 23:53:56 -07:00
f46ca5c159 chore(deps): update rust crate thiserror to v2.0.19
Some checks failed
renovate/stability-days Updates have met minimum release age requirement
gitleaks / scan (push) Failing after 1s
gitleaks / scan (pull_request) Failing after 1s
2026-07-23 06:53:49 +00:00
a23705c882 Merge pull request 'chore(deps): update rust crate serde_json to v1.0.151' (#15) from renovate/serde_json-1.x-lockfile into main
Some checks are pending
gitleaks / scan (push) Waiting to run
2026-07-22 23:53:46 -07:00
812a99e99c chore(deps): update rust crate serde_json to v1.0.151
Some checks failed
renovate/stability-days Updates have met minimum release age requirement
gitleaks / scan (push) Failing after 1s
gitleaks / scan (pull_request) Failing after 1s
2026-07-23 06:53:39 +00:00
856a2a4a5c Merge pull request 'chore(deps): update rust crate serde to v1.0.229' (#14) from renovate/serde-1.x-lockfile into main
Some checks failed
gitleaks / scan (push) Failing after 2s
2026-07-21 23:56:04 -07:00
2f0de58dd0 chore(deps): update rust crate serde to v1.0.229
Some checks failed
renovate/stability-days Updates have met minimum release age requirement
gitleaks / scan (push) Failing after 1s
gitleaks / scan (pull_request) Failing after 1s
2026-07-22 06:55:57 +00:00
92a155dc85 Merge pull request 'chore(deps): update rust crate anyhow to v1.0.104' (#13) from renovate/anyhow-1.x-lockfile into main
Some checks are pending
gitleaks / scan (push) Waiting to run
2026-07-21 23:55:54 -07:00
43368c0c45 chore(deps): update rust crate anyhow to v1.0.104
Some checks failed
renovate/stability-days Updates have met minimum release age requirement
gitleaks / scan (push) Failing after 1s
gitleaks / scan (pull_request) Failing after 1s
2026-07-22 06:55:42 +00:00
43e2895018 Merge pull request 'chore(deps): update rust crate clap to v4.6.2' (#12) from renovate/clap-4.x-lockfile into main
Some checks failed
gitleaks / scan (push) Failing after 8s
2026-07-18 23:48:46 -07:00
a51373dbae chore(deps): update rust crate clap to v4.6.2
Some checks failed
renovate/stability-days Updates have met minimum release age requirement
gitleaks / scan (push) Failing after 3s
gitleaks / scan (pull_request) Failing after 3s
2026-07-19 06:48:37 +00:00
Sulkta
d645cbc8ed skald: public release prep — AGPL-3.0, generic config, scrubbed identifiers 2026-06-28 21:38:36 -07:00
3fa9f181ac Merge pull request 'chore(deps): update rust crate uuid to v1.23.4' (#6) from renovate/uuid-1.x-lockfile into main 2026-06-27 23:45:11 -07:00
3d7d4a6de0 chore(deps): update rust crate uuid to v1.23.4 2026-06-28 01:28:02 +00:00
Sulkta
d5dd66df0b Merge pull request 'chore(deps): update rust crate serde_json to v1.0.150' (#5) from renovate/serde_json-1.x-lockfile into main 2026-06-27 17:07:14 -07:00
Sulkta
799a703648 Merge pull request 'chore(deps): update rust crate regex to v1.12.4' (#4) from renovate/regex-1.x-lockfile into main 2026-06-27 17:07:08 -07:00
Sulkta
320fc26cc8 Merge pull request 'chore(deps): update rust crate chrono to v0.4.45' (#2) from renovate/chrono-0.x-lockfile into main 2026-06-27 17:07:00 -07:00
Sulkta
35b44fc828 Merge pull request 'chore(deps): update rust crate anyhow to v1.0.103' (#1) from renovate/anyhow-1.x-lockfile into main 2026-06-27 17:06:54 -07:00
76914a50f7 chore(deps): update rust crate serde_json to v1.0.150 2026-06-27 21:37:26 +00:00
80152e0373 chore(deps): update rust crate regex to v1.12.4 2026-06-27 21:37:17 +00:00
d02d1bef9f chore(deps): update rust crate chrono to v0.4.45 2026-06-27 20:46:42 +00:00
c7b92d1c06 chore(deps): update rust crate anyhow to v1.0.103 2026-06-27 20:46:35 +00:00
Sulkta
7932e53a45 ci: add gitleaks workflow (Sulkta canonical) 2026-05-27 22:14:56 -07:00
Sulkta
564233cbff Public-flip audit: env-driven paths, scrub audit-ticket prefixes, terser README
the host bind paths + LAN host pins replaced with env defaults. Repository URLs
→ git.sulkta.com. Audit-changelog scaffolding stripped from inline comments
(technical reasoning preserved). README sheds marketing scaffolding. AI-speak
in load-bearing prompts/SOULs left alone — that IS the product.
2026-05-27 11:42:58 -07:00
Sulkta
566ca05ea8 dedup: length guard against corrupted output + --chapter retry
One chapter in a dedup run had the model emit the chapter twice
(18.7k -> 32.6k chars). A surgical dedup only nudges length, so
the pass now rejects any output wildly off the input length —
marks that chapter's run failed, leaves the prose untouched, and
moves on rather than silently corrupting it. New --chapter flag
narrows a run to one chapter so a skipped one can be retried.
2026-05-15 15:39:47 -07:00
Sulkta
8dde7afeda forge: dedup pass — the fix half of the audit loop
Adds `skald dedup --story <id>`: reads a story's most recent
prose-audit findings and walks every chapter, handing the author
the chapter prose + the findings with instructions to rephrase
ONLY the flagged repetitions (each recurrence made distinct) and
fix flagged continuity errors — everything else stays verbatim.
A surgical dedup, not a rewrite. Overwrites body_md, clears
body_md_tts so the chapter is re-prepped before narration. High
effort (prose-craft). Migration 0011 adds the 'dedup' run kind.

Completes the QC loop: audit (find) -> dedup (fix) -> re-audit.
2026-05-15 14:49:08 -07:00
Sulkta
8bf4137687 forge: prose audit runs at high effort, not max
A max-effort pass over a whole-book input (quarter-million chars)
runs an impractical hour-plus and was timing out. Finding repeated
passages is comparison work, not deep reasoning — high effort does
it well in minutes at a fraction of the cost.
2026-05-15 14:25:59 -07:00
Sulkta
f963c6cf03 forge: raise timeouts for the max-effort whole-book audit pass
The prose audit reads a whole story at max effort and genuinely
runs past 1800s on a 50k-word book — it was timing out. Bump the
audit pass server-side timeout to 3600s and the shared reqwest
client ceiling to 7200s (matching clawdforge's new cap). Other
passes keep their own shorter timeout_secs.
2026-05-15 11:54:50 -07:00
Sulkta
bc03f52c3e forge: prose-quality audit pass + anti-repetition directives
Adds `skald audit --story <id>`: a whole-story QC pass that reads
every chapter end to end and flags repetition, template tics,
self-restatement and continuity drift — the gate before a story
goes to narration, where repetition a silent reader skims is
glaring read aloud. Runs at max effort (real reasoning work,
worth the spend); findings land in audit_findings and print.

Also hardens the gen + cleanup directives to hunt repetition at
the source: re-phrase recurring motifs fresh, no stacked template
anaphora, dialogue echoed verbatim at most once.

Migration 0010: 'prose_audit' generation_runs.kind, 'repetition'
audit_findings.area.
2026-05-15 11:19:04 -07:00
Sulkta
f70cfe1f0b web: audiobook player — stitched-file playback with chapter seek
Adds GET /stories/{id}/listen: one <audio> element over a story's
stitched audiobook file plus a clickable chapter list. Clicking a
chapter seeks; the chapter under the playhead highlights as it
plays. Chapter offsets are summed from each chapter's latest
succeeded narration_run duration — the same order the file was
stitched. One small inline script, the web UI's first JS.

New stories.audiobook_path column (migration 0009) holds the
served path; the story page shows a "listen" action when set.
2026-05-15 07:30:56 -07:00
Sulkta
61f1bdd40a narrate: single-voice prep drops voice tags; GC superseded renders
Two fixes:

- narrate_prep in single-voice mode (empty character roster) was
  still handed the multi-voice directive, so the model invented
  [voice:<slug>] tags from character names in the prose. The
  narrate path neutralised them by falling back to the narrator,
  but it was log spam and a leak of intent. Single-voice now gets
  directive + house-system variants that forbid voice tags
  outright, and the user-prompt task line matches.

- Every narrate run wrote a fresh ~80MB WAV and never reclaimed
  the previous one, so re-renders piled up stale files. A
  successful render now deletes the WAVs of prior renders of the
  same chapter and nulls their output_path. Render history rows
  are kept; only the dead file pointer is cleared. Best-effort —
  cleanup failure never fails the render.
2026-05-15 07:02:10 -07:00
Sulkta
c863543580 forge: high effort for prose-craft passes, max only for audit
gen, cleanup, narrate_prep and rewrite drop from max to high
effort. Audit keeps max — it is the one pass doing real reasoning
(canon drift, timeline gaps, retcons) rather than prose-craft, so
it is worth the frontier spend. Prose-craft is "good enough" at
high. This also keeps the all-Opus skald pattern under the
$200/month claude -p cap landing next month.
2026-05-14 22:32:52 -07:00
Sulkta
b5576f4ec5 forge: rewrite pass — re-author prose in an author's voice
New Forge::rewrite + PassKind::Rewrite. An author re-authors
existing chapter prose entirely in their voice — sentence rhythm,
word choice, paragraph shape all become theirs — while canon
(names, dates, places, events, order, technical facts) is preserved
exactly. Not editing; re-authoring. SystemMode::Replace, max effort.

skald rewrite --chapter <uuid> [--author slug] overwrites body_md
with the rewritten version. The pre-rewrite prose is stashed in the
new chapters.body_md_original column on first rewrite (migration
0008, idempotent) so the original is never lost. body_md_tts is
cleared — it was annotated against the old prose and must be
regenerated by a fresh prepare-narration.

prepare-narration gains --single-voice: skips the character speaker
roster so no [voice:X] dialogue tags are inserted, only beat
markers. Right for one-voice narration.

Migration 0008 also extends generation_runs.kind to allow 'rewrite'.
2026-05-14 21:35:20 -07:00
Sulkta
1e7905d1e1 narrate: route tortoise_* voices to TORTOISE_URL
Engine dispatch was a binary kokoro-vs-everything-else split that
sent tortoise voices to F5. Now three branches:
  kokoro_*    → KOKORO_URL    (the host default :7794)
  tortoise_*  → TORTOISE_URL  (the host default :7795)
  *           → F5_TTS_URL    (the host default :7792)

substitute_speaker_voices also runs for tortoise voices, so
multi-voice [voice:slug] tags survive in the prose. Tortoise voices
without a per-character mapping fall back to the narrator voice
server-side (tortoise_server.py logs the fallback).
2026-05-14 09:46:16 -07:00
Sulkta
01ec9ffd0e engines: import f5-tts + kokoro + tortoise sidecars into the tree
The python FastAPI sidecars have lived ad-hoc at /srv/appdata/
<engine>/build/ on the host without version control. Bringing them into
the skald repo so the engine code travels with the cross-engine
routing it depends on.

This commit lands the VANILLA version of each engine on main:

  engines/f5-tts/    SWivid F5-TTS (CC-BY-NC weights flagged)
  engines/kokoro/    hexgrad Kokoro-82M (Apache 2.0 top to bottom)
  engines/tortoise/  neonbjb Tortoise-TTS (Apache 2.0 top to bottom)

Engine-specific kludges (question doubling, GPU coordination,
pause-duration tuning) get layered on engine/* branches per the
README. Main stays the safe-to-read baseline.
2026-05-14 09:40:01 -07:00
Sulkta
d2ec84af21 narrate: pronunciation overrides for proper nouns
skald narrate pre-processes body_md_tts with word-boundary regex
substitutions from pronunciation_overrides where phoneme_format =
'respelling'. Story-scoped overrides win over global; longer words
substitute first (so 'Bryukhanov' wins over a hypothetical 'Bry'
override). Case is preserved on the first letter so 'Pripyat' at
sentence start stays capitalised after respelling to 'Prip-yat'.

Seeded Coast-Down with 52 entries: Russian/Ukrainian surnames
(Dyatlov, Akimov, Bryukhanov, Stolyarchuk, Yuvchenko, Khmel, etc.),
first names (Pyotr, Lyudmilla, Anatoly, Vasily, Sasha, Aleksandr,
Leonid), patronymics (Stepanovich, Fyodorovich, Mykolaivna,
Hryhorivna), and places (Pripyat, Chernobyl, Kyiv, Kopachi,
Lubyanka). Plus the operational acronyms NIKIET, RBMK, AZ-5, SIUR,
SIUB, ChNPP, MSCh.

Other phoneme_format values (ipa, arpabet) are no-ops for now —
Kokoro's misaki tokenizer doesn't expose a stable lexicon-injection
API across the HTTP boundary in v0.1. Future: pass IPA forms in a
new server-side request field and inject into the pipeline's g2p
lexicon for more accurate phonetics.
2026-05-14 08:56:17 -07:00
Sulkta
21c5db19a1 multi-voice: per-character dialogue rendering
Schema: characters.voice_id + characters.slug (migration 0007).
voice_id is FK to voices(id); slug is the stable lowercase token
the narrate_prep pass uses inside [voice:slug]...[/voice].

Forge::narrate_prep takes &[CharacterSpeaker]. System prompt
expanded to instruct the author to wrap dialogue lines in voice
tags based on a roster supplied in the user prompt (slug + name +
short hint from key_facts). Unattributed dialogue stays unwrapped
and inherits the narrator voice.

skald narrate substitutes [voice:<character-slug>] →
[voice:<kokoro-voice-name>] right before sending to Kokoro, using
characters.voice_id JOIN voices.reference_path as the map. Slugs
with no voice or no character row fall back to the narrator voice
defensively (logged as warn).

kokoro_server.py v0.4: splitter recognises [voice:X]...[/voice]
blocks at the paragraph level. Each text node carries an optional
voice attribution; renderer feeds it to Kokoro per-segment. Outside
voice blocks the request's default voice is used. voices_used is
reported back so callers can verify multi-voice actually ran.

Only kokoro-routed renders pre-process voice tags; F5 paths leave
the tags in place (F5 multi-voice not implemented). Defensive
fallback: orphan/unclosed [/voice] markers are silently absorbed
rather than failing the render.
2026-05-14 08:35:33 -07:00
Sulkta
14ffd6d79a migration 0005: idempotent ADD COLUMN IF NOT EXISTS
Caught when redeploying after the 0006 patch: the live DB had
migration 5 stamped with a stale checksum + the column already
present, so neither re-apply nor checksum-only-fix worked cleanly.
Making 0005 idempotent fixes both paths.
2026-05-13 20:32:41 -07:00
Sulkta
ac1490c28a migration 0006: extend generation_runs.kind to allow narrate_prep
Migration 0005 added the chapters.body_md_tts column but missed
this check constraint update — caught at runtime when
prepare-narration tried to insert kind='narrate_prep'.

Postgres doesn't ALTER CHECK in place; we drop + re-add.
2026-05-13 20:28:59 -07:00
Sulkta
1f1d63dd1b narrate: body_md_tts column + narrate_prep pass + Kokoro routing
Two new things working together:

1. Migration 0005 adds chapters.body_md_tts (nullable). Narrate path
   prefers it over body_md when present — that's the annotated-for-
   audiobook variant. Falls back to body_md if not set.

2. New Forge::narrate_prep pass: author (or House) annotates prose
   with [breath] / [pause:Xs] / [scene] beat markers AND occasional
   humanizing narrator stumbles (em-dash repetition, self-correction,
   hesitation — sparingly, 1-3 per chapter). Apart from stumbles, the
   prose is verbatim. Author voice threads through.

3. New CLI: 'skald prepare-narration --chapter <uuid> [--author slug]
   [--overwrite]'. Records as generation_runs row kind=narrate_prep.

4. skald narrate now routes by voice.source — kokoro_* voices hit
   KOKORO_URL (Apache 2.0 stack, audiobook-tuned with the v0.2 render-
   and-stitch server), everything else hits F5_TTS_URL (voice-cloning
   path). Voice DB row carries source as the dispatch key.

Why no new tag for narrator stumbles: em-dash repetition and self-
correction are just prose patterns Kokoro reads correctly because of
its punctuation cues. No new server-side machinery.
2026-05-13 20:24:38 -07:00
Sulkta
3c159d8d75 forge: skald owns chapter numbers, not the LLM
Caught on the 2026-05-13 Coast-Down 10-chapter Orson run: the LLM
labeled two chapters 'Chapter 1' instead of 9 and 10, and
ON CONFLICT (story_id, n) DO UPDATE silently overwrote them.
8 visible chapters from 10 successful gen+cleanup passes; 27k
words of work, ~6k buried. The audit caught the symptom but the
data damage was already done.

Fix:
- continue_story::run computes next_n from MAX(chapters.n) before
  the batch loop; each iteration's authoritative n is next_n,
  incremented after success.
- forge::generate + cleanup take chapter_n: Option<i32>. The gen
  prompt is now 'Write Chapter N. Begin with: ## Chapter N — ...'
  instead of the vague 'Write the next chapter.'
- We still parse_chapter() the LLM output but only to extract the
  title; if the LLM-returned n disagrees with ours, we log a warn
  and use the authoritative N at INSERT time.

The (story_id, n) unique constraint stays — it's now a defensive
catch for skald bugs, not the LLM's free-spirited numbering.
2026-05-13 18:29:22 -07:00
Sulkta
dbd41424ac web: chapter audio player + render button
Chapter view now shows a narration card between title and prose
with three states:
  - succeeded → HTML5 <audio> + voice + duration + download link
  - running   → 'rendering…' banner with relative start time
  - none/failed → 'Render audio' POST button (spawns background
                  tokio task calling narrate::run)

ServeDir mounted at /audio serves WAVs from the f5-tts bind-mount
read-only. Range requests work, so 16-min chapters seek cleanly.

Deploy needs: compose mount /srv/appdata/f5-tts/audio:/audio:ro
on skald (already staged in /srv/appdata/skald/compose.yml on
the host).
2026-05-13 17:08:43 -07:00
Sulkta
1f2dd40105 narrate: F5-TTS HTTP client + skald narrate CLI
skald-core::narrate ships a thin reqwest client + voice DB access
(get_by_name, get_default, get_by_id). The boundary is the f5-tts
container's purpose-built FastAPI sidecar (python lives there
because torch + transformers + safetensors do); skald never touches
python.

CLI: skald narrate --chapter <uuid> [--voice slug] [--speed 1.0].
Voice resolution: --voice flag → story.preferred_voice_id → system
default. Persists narration_runs row (engine='f5-tts', engine_version
pinned, status: running → succeeded|failed). Output path stored is
the f5-tts container's view (/audio/<story>-<n>-<run>.wav); web
playback wiring deferred.
2026-05-13 16:45:04 -07:00
Sulkta
a291f44a10 forge: always --effort max + multi-chapter batch (cap 20)
forge.rs threads Effort::Max on gen + cleanup. Audit + summarize stay
default — they're structured-output / tool-shaped tasks where extended
thinking doesn't help. Bumps subprocess timeout from 600s to 1800s so
max-effort prose-craft has the wall clock it needs.

continue_story::run takes a chapter_count param; loops gen+cleanup per
chapter with each iteration's just-written prose appended to context.
Audit fires once at end against the combined batch vs parent canon.
Cap is 20 (~5h wall clock, ~$600 at max effort — beyond that is
operationally absurd).

CLI: 'skald continue --chapters N'. Web: numeric field on both new-
story and continue forms, 1..=20, defaults to 1.

Vendored clawdforge SDK refreshed for the Effort enum.
2026-05-13 14:26:53 -07:00