diff --git a/MILESTONES.md b/MILESTONES.md index d154516..7baa524 100644 --- a/MILESTONES.md +++ b/MILESTONES.md @@ -7,11 +7,13 @@ - [x] GPL-3.0 license headers - [ ] crafting-table build target produces a static aarch64 sidecar binary -## M1 — sidecar resolve +## M1 — sidecar resolve (three-tier) - [ ] reads `{"op":"resolve","id":""}` from stdin -- [ ] calls rustypipe → returns `{"streams":[…],"title":"…","duration_s":N}` -- [ ] handles age-restricted + region-restricted as typed errors, not panics +- [ ] **Tier 1:** rustypipe → `{"streams":[…],"title":"…","duration_s":N,"source":"rustypipe"}` +- [ ] **Tier 2:** on Tier-1 failure, shell out to `yt-dlp -j ` → same JSON shape with `"source":"yt-dlp"` +- [ ] **Tier 3:** new op `{"op":"rip","id":"","dest":"/storage/.kodi/temp/torttube/."}` invoked by addon on Tier-1+2 stream failures or 403-mid-play; yt-dlp downloads file, sidecar returns local path +- [ ] typed errors for age-restricted / region-restricted / private (not panics) - [ ] sig decoding verified against a known-good video - [ ] DASH manifest URL or per-itag direct stream URL — whichever inputstream.adaptive prefers @@ -49,10 +51,16 @@ - [ ] `addon.zip` ships with platform detect via `xbmc.getCondVisibility('system.platform.linux.raspberrypi')` - [ ] one-shot install path documented for LibreELEC `/storage/.kodi/` -## Upstream PR targets (parallel, opportunistic) +## Upstream PR work (parallel lane — every bug evaluated for "fix it upstream?") -Pick one as we hit it organically — don't gate milestones on these. +This isn't a separate milestone, it's a posture. Every sidecar bug we +diagnose: ask "is this rustypipe / NPE / yt-dlp's bug?" — if yes, fix lands +upstream too. Log every filed PR in [docs/upstream.md](docs/upstream.md). -- NPE #1357 (JDoc) — smallest, just to land a credible PR +Opening shortlist: + +- rustypipe PR #77 — review + help land +- NPE #1357 (JDoc) — smallest credible PR, opens the relationship - NPE #1444 (typed errors) — clean signal/contract change -- rustypipe — file issues + PRs on codeberg as we hit gaps +- NPE #1360 (refactor link handlers, "help wanted") +- rustypipe ↔ NPE port — anything one project has that the other lacks diff --git a/README.md b/README.md index 945b5a6..e192839 100644 --- a/README.md +++ b/README.md @@ -11,32 +11,57 @@ required account-linking for the upstream addon. ``` Kodi (LibreELEC, RPi) └── plugin.video.torttube [Python addon — UI, browse, settings] - └── torttube-sidecar [Rust binary — Innertube extraction, - SponsorBlock fetch, JSON-over-stdio] - ├── rustypipe [YouTube Innertube client + sig decode] - └── sponsorblock [REST API client, SHA-256 prefix lookup] + └── torttube-sidecar [Rust binary — JSON-over-stdio] + ├── rustypipe [Tier 1: native Rust Innertube] + ├── yt-dlp subprocess [Tier 2: fallback resolve] + ├── yt-dlp rip-to-temp [Tier 3: download to /storage/.kodi/temp, + │ play local file when streams die mid-play] + └── sponsorblock [REST client, SHA-256 prefix lookup] ``` Kodi addons are Python — the engine layer (n-param sig decoding, Innertube, SponsorBlock hashing) lives in a Rust sidecar so we get a single maintained extraction surface and clean aarch64/armv7 cross-compiles. +**Three-tier resolve** because YouTube actively fights every extractor: + +1. **rustypipe (Rust)** — preferred. Fast, in-process, no Python dep on the RPi. +2. **yt-dlp subprocess** — fallback when rustypipe sig-decoding falls behind YouTube's deobfuscator changes. yt-dlp updates weekly; we shell out, parse `-j` JSON. +3. **Rip-to-temp** — last resort when stream URLs 403 mid-playback (poToken expiry, cookie session mismatch). yt-dlp downloads to `/storage/.kodi/temp/torttube/.`, Kodi plays the local file. Temp dir has size cap + age cleanup. + ## Status M0 scaffold. Nothing playable yet — see [MILESTONES.md](MILESTONES.md). -## Upstream contributions +## Upstream — we fight with the FOSS extractor ecosystem, not next to it -NewPipeExtractor (Java) is the canonical reference for Innertube behaviour. -When we hit a corner case RustyPipe doesn't cover we look at NPE for the -fix, then either port it to RustyPipe (Rust PR) or fix NPE directly (Java -PR). Either way upstream lands a real improvement. +YouTube's anti-scraping changes hit every extractor: NewPipe, yt-dlp, Invidious, +rustypipe. Every fix we make in our sidecar gets evaluated for "is this +upstreamable?" — if yes, the fix lands at the upstream project, not just here. + +Active lanes: + +- **rustypipe** (Rust, codeberg.org/ThetaDev/rustypipe) — maintenance has slowed. + Open PR #77 "Some fixes" is unmerged as of 2026-05-23. We will either help land + it (review + ping maintainer) or fork to `Sulkta-Coop/rustypipe` if upstream + stays quiet. Forking is the worst case, not the first move. +- **NewPipeExtractor** (Java, github.com/TeamNewPipe/NewPipeExtractor) — actively + maintained, 177 open issues. We use it as the reference implementation for + Innertube behaviour. PRs to NPE land in Rust here via rustypipe, and vice + versa. +- **yt-dlp** (Python, github.com/yt-dlp/yt-dlp) — the gold standard. We're more + consumers than contributors here, but if our rip-to-temp tier surfaces a + specific extractor bug we file it. Issues we're watching: - [NPE #1339](https://github.com/TeamNewPipe/NewPipeExtractor/issues/1339) — n-parameter deobfuscation - [NPE #1444](https://github.com/TeamNewPipe/NewPipeExtractor/issues/1444) — distinguish unavailable vs unextractable - [NPE #1360](https://github.com/TeamNewPipe/NewPipeExtractor/issues/1360) — refactor link handlers (help wanted) - [NPE #1357](https://github.com/TeamNewPipe/NewPipeExtractor/issues/1357) — JDoc checks in PR pipeline (good first issue) +- [rustypipe PR #77](https://codeberg.org/ThetaDev/rustypipe/pulls/77) — open as of 2026-05-23, unmerged + +Contribution log lives at [docs/upstream.md](docs/upstream.md) — every PR we +file lands there with its outcome. ## License diff --git a/docs/upstream.md b/docs/upstream.md new file mode 100644 index 0000000..2102ce8 --- /dev/null +++ b/docs/upstream.md @@ -0,0 +1,33 @@ +# Upstream contributions log + +One row per PR/issue we file. Outcomes recorded honestly — merged, closed, +abandoned, whatever happened. + +## Active + +_(none yet — opens with M1 development)_ + +## Filed + +| Date | Project | PR/Issue | Title | Status | Outcome | +|------|---------|----------|-------|--------|---------| +| _empty_ | | | | | | + +## Watching (not ours, but relevant to torttube) + +| Project | PR/Issue | Title | Why we care | +|---------|----------|-------|-------------| +| rustypipe | [PR #77](https://codeberg.org/ThetaDev/rustypipe/pulls/77) | "Some fixes" | Open 2026-05-23, unmerged. If maintainer stays quiet we may need to help land it or fork. | +| NPE | [#1339](https://github.com/TeamNewPipe/NewPipeExtractor/issues/1339) | n-parameter deobfuscation broken | Core to playback. Fix here = fix in our Tier-1. | +| NPE | [#1444](https://github.com/TeamNewPipe/NewPipeExtractor/issues/1444) | Distinguish unavailable vs unextractable | Clean typed-error PR target. | +| NPE | [#1360](https://github.com/TeamNewPipe/NewPipeExtractor/issues/1360) | Refactor link handlers | "help wanted" label. | +| NPE | [#1357](https://github.com/TeamNewPipe/NewPipeExtractor/issues/1357) | JDoc checks in PR pipeline | "good first issue" — smallest credible first PR. | + +## Posture + +- Every sidecar bug we trace ends with one question: **is the root cause + in rustypipe / NPE / yt-dlp?** If yes, fix it there first; pull the fix + into our sidecar via dep bump or backport. +- We are not pretending to be on the rustypipe / NPE / yt-dlp maintainer + level. We're a downstream consumer that does its share. +- A merged PR > a forked patch. A fork is the last resort, not the first move.