The previous decoder treated each %XX as an isolated code point via
`out.push(v as char)`. For UTF-8 multi-byte sequences (e.g. %E2%9C%93
for ✓) that produced three garbage chars at U+00E2 / U+009C / U+0093
instead of the proper U+2713. YT cipher strings are typically ASCII-
only so this was latent, but the function was named generically and
nothing in the type system prevented a non-ASCII input from reaching it.
`url::form_urlencoded::parse` is the canonical &-separated query-pair
parser — handles %-decode as UTF-8, handles + → space, and the url
crate is already a transitive dep. parse_cipher_string collapses to
one line; the bespoke 20-line decoder goes.
Second pass through the cruft inventory. All deletes — no behavior change.
Method enum + downloader trait:
* Method::Head / Put / Delete dropped (no caller). Match arms in
request.rs::as_str() and default_impl.rs::execute() collapse to
just Get / Post.
* Request::head builder dropped.
* Downloader trait's get / get_localized / head / post default
methods dropped. Every caller went through execute() directly
anyway; the convenience wrappers carried 4 dead arms each.
Parsing module:
* bootstrap_visitor_data — pub fn, no caller.
* discover_web_client_version + CACHED_WEB_CLIENT_VERSION +
reset_web_client_version_cache — entire sw.js live-version
discovery pipeline, never wired up by any caller. The cache was
never populated, so web_client_version() always returned the
hardcoded constant. Collapsed to just returning the constant.
* Drops once_cell::Lazy, parking_lot::RwLock around the version
cache (consent flag still uses RwLock), Regex import, serde_json::Value
import, downloader/exceptions/Request/InnertubeClientRequestInfo
imports — all only kept alive by the deleted code.
stream_helper:
* get_web_embedded_player_response — pub fn, no caller.
js/player_manager + extractor:
* player_manager::player_hash — pub fn, no caller. Was only kept
alive by its own definition.
* extractor::extract_player_hash — pub fn, only called by the now-
dead player_hash. Test removed alongside.
Stale comments:
* itag.rs:1 header claimed 53 entries; ITAG_TABLE has 57 and the
test at line 179 already asserts it.
* js/mod.rs:12-13 claimed the submodules were 'crate-private
plumbing' but they're declared pub mod. Tightened the comment to
explain the integration-test dependency that keeps them public.
Net delete: ~170 LOC of dead surface across 9 files.
Release builds were emitting unused_imports for MediaFormat, which is
now only referenced inside a #[cfg(test)] block (after dropping the
_suppress_unused stub in the previous commit).
Round-2 cruft audit punch list — mechanical deletes, no behavior change.
Whole modules deleted (no wrapper consumer):
* youtube/playlist_extractor.rs (297 LOC) — full playlist extraction
* youtube/linkhandler/playlist.rs (81 LOC) — playlist URL parser
* youtube/suggestion_extractor.rs (91 LOC) — search-as-you-type
* tests/stream_phase4_offline.rs (186 LOC) — tautological test
Dead pub fns + enum variants + constants:
* WEB_REMIX_* constants (3) + WEB_MUSIC_ANALYTICS_* constants (3)
* InnertubeClientRequestInfo::of_web_music_analytics_charts_client
factory + its charts_client_omits_platform_and_screen test
* SearchFilter::Music{Songs,Videos,Albums,Playlists,Artists} variants
(5 of 9 cases) + uses_music_endpoint helper + the search_extractor
'music search not implemented' reject branch
* Two #[allow(dead_code)] _suppress_unused stub fns and the imports
they were keeping alive (std::sync::Arc in js/extractor.rs,
NetworkError in stream_extractor.rs)
Renamed:
* search_extractor::test_helpers -> renderer_helpers. Mis-named:
it's production code called from channel.rs, not a test fixture.
potoken/ kept and documented as the designed Phase-5 extension point
for YouTube bot-detection — wrapper's Android side hasn't registered
a real provider yet, but the trait + global slot stay so when YT
forces po_token universally the integration is one Kotlin patch away,
not a Rust-side rewrite.
~580 LOC removed from production. Wrapper does not need to change.
Two adversarial bugs surfaced by the round-2 audit on this crate.
extract_video_id recursion (linkhandler/stream.rs)
/attribution_link?u=<inner> recursed on the inner URL with no depth
guard. The comment claimed 'only one level deep' but the call was
plain recursion — a pasted URL whose u= param decodes to another
/attribution_link would recurse until the JVM stack blew. Wrap the
recursion in extract_video_id_inner with an explicit depth counter
capped at MAX_ATTRIBUTION_DEPTH = 1.
ReqwestDownloader body cap (downloader/default_impl.rs)
resp.text() read the entire response body into a String with no
upper bound. Player.js is ~1.5 MB, watch HTML ~3 MB, channel
responses well under 1 MB. A hostile redirect target (or compromised
host) could blast multi-GB and OOM-kill the Android process — there
is no headroom on a 1 GB JVM heap ceiling.
Cap at 32 MB. Two-stage check: bail fast on a known Content-Length
that exceeds the cap, and use Read::take(MAX+1) on the stream so we
detect overrun rather than silently truncate. Switched the final
decode to from_utf8_lossy so a single mojibake byte doesn't drop the
whole response (same fix shape as the wrapper's read_capped_body).
These three modules were ported from NewPipeExtractor in Phase 1 as
part of the spine. Nothing in the YT extractor (channel/search/stream/
playlist/linkhandler) imports them, and the strawcore wrapper crate
that consumes us doesn't re-export them either. Per the round-2
audit's cruft inventory, this is ~195 LOC of dead surface shipping
to every Android APK.
* page.rs — Page continuation carrier; continuation tokens flow
through the codebase as plain Strings.
* metainfo.rs — NPE MetaInfo info-card struct; no extractor
populates it.
* service.rs — StreamingService trait + ServiceInfo + LinkType;
zero impls exist anywhere.
Wrapper does not need to change — none of the pub use re-exports
crossed the crate boundary.
Channels on the newer pageHeaderRenderer layout (most channels with a
2024+ refreshed header — WTYP, etc.) were getting empty avatars and
banners since the parse_channel_browse only extracted those from the
older c4TabbedHeaderRenderer branch.
Two fixes layered:
1. parse_page_header_avatar() — walks the deep ViewModel nest:
header.content.pageHeaderViewModel.image
.decoratedAvatarViewModel.avatar.avatarViewModel.image.sources[]
Falls back to a couple of shallower nestings YT has used on this
path historically. Returns ImageSet sorted by height ascending so
.last() still picks the largest source.
2. metadata.channelMetadataRenderer.avatar.thumbnails[] backfill.
Set whether the header is c4Tabbed or pageHeader, and the most
reliable single avatar source. Used only when both header branches
came back empty so we don't override a higher-quality header avatar.
Description-from-metadata extraction folded into the same metadata
walk to avoid the JSON tree twice.
Found via emulator smoke that channelInfo was returning empty
recent_videos list, breaking the subscriptions feed.
Two root causes:
1. First browse of a channel by browseId lands on the HOME tab in
2026 YT, not Videos. Home uses sectionListRenderer, not the
richGridRenderer my parser expected. The Videos tab in the
response carries an empty content block (you need a SECOND
browse with the params token to populate it).
2. Channel video items on the Videos tab migrated from
videoRenderer to lockupViewModel (YT made the switch ~2024).
My old parser only handled videoRenderer.
Fix:
* fetch_channel_browse now does TWO browses — first for Home
(header + metadata), second with params='EgZ2aWRlb3PyBgQKAjoA'
for the Videos tab. Same magic constant NPE uses (audit Track
A §2.4).
* parse_videos_tab handles BOTH videoRenderer (legacy/fallback)
AND lockupViewModel (current). lockupViewModel parse extracts:
- contentId → video ID
- metadata.lockupMetadataViewModel.title.content → title
- metadataRows[].metadataParts[].text.content → view-count
('1.1m views') + relative-age ('2 years ago') + uploader
- contentImage.thumbnailViewModel.overlays[]
.thumbnailBottomOverlayViewModel.badges[]
.thumbnailBadgeViewModel.text → duration ('3:14:08')
- contentImage.thumbnailViewModel.image.sources[] → thumbnails
* parse_videos_continuation pulls the continuation token from the
Videos tab grid for pagination.
Second browse is best-effort: if it fails, recent_videos stays
empty and the channel header still populates from the first.
Verified the YT response shape by probing live channel
UCwwtUfy0-CqN50HfaFDzL0w (NCS Spektrem) — got 30+ lockup-style
video items with the expected fields.
Caught during the cargo-ndk cross-compile — strawcore-core was
emitting its own libstrawcore_core.so (~306 KB per ABI) into Straw's
jniLibs. That .so is never loaded by Android; the wrapper crate's
libstrawcore.so is the only entry point.
rlib only is what consumer crates need.
Straw's wrapper crate already owns the name 'strawcore' (and that name
is baked into the Android .so file + Kotlin's System.loadLibrary call).
Renaming this extractor crate to 'strawcore-core' resolves the cargo
package-name collision so both can live in the same workspace dep tree.
Repo name on Gitea stays Sulkta-Coop/strawcore.
Mirrors NPE PoTokenProvider.java + PoTokenResult.java; defines the
host-injection surface for BotGuard attestation. The Rust crate stays
out of the BotGuard business — embedders (Straw on Android, future
Sulkta CLI via Browserless, etc.) supply their own impl.
src/youtube/potoken/mod.rs
* PoTokenResult { player_request_po_token, streaming_data_po_token,
visitor_data } + ::new + ::single constructors
* PoTokenError (Unavailable, MintFailed) — FIX vs NPE: split 'declined'
(Ok(None)) from 'errored' (Err) so callers can react differently
* trait PoTokenProvider with 4 client-scoped methods; default impl
returns Ok(None) so embedders can override just what they support
* set_po_token_provider / clear_po_token_provider / po_token_provider
static registration via RwLock<Option<Arc<dyn PoTokenProvider>>>
src/youtube/potoken/noop.rs
* NoopPoTokenProvider — safe default
src/youtube/stream_extractor.rs
* resolve_po_token via options-first-then-provider helper
(options_or_provider)
* Android branch: pulls player_request_po_token + visitor_data into
/player body, streams streaming_data_po_token through to URL &pot=
* iOS branch: same shape, gated on fetch_ios_client AND non-empty
provider result
Kotlin side (PoTokenWebView lift into Straw via UniFFI's foreign-trait
bridge) is separate work — strawcore just owns the contract.
Tests: 77 lib unit pass (+4 since Phase 4) + 7 Phase 2 offline + 7
Phase 4 offline = 91 green.
Port NewPipeExtractor's JS pipeline: player.js fetch + cache, sig and
nsig function extraction, deobfuscation, sticky-error caching.
src/youtube/js/
* runtime.rs — rquickjs wrapper (mirrors utils/JavaScript.java)
compile_or_throw + run(snippet, name, parameter)
* lexer.rs — match_to_closing_brace via the `ress` JS scanner
(NPE's lexer is derived from the same crate
upstream)
* extractor.rs — iframe_api → embed page fallback for player.js
URL, regex-driven hash extraction, clean-and-fetch
* signature.rs — 6 sig fn name regexes (front-most-recent),
deobf-function-body via lexer w/ regex fallback,
helper-object + global-string-array extraction,
signatureTimestamp, snippet assembler
* nsig.rs — 8 nsig fn name regexes (incl. array-indirection),
body via lexer w/ regex fallback, fixupFunction
early-return strip
* player_manager.rs — orchestrator + sticky-error cache mirroring
YoutubeJavaScriptPlayerManager
PORT DEVIATIONS from NPE (each flagged in code):
* dropped the 6th sig fn name regex (used Java backref \2; Rust's
`regex` crate is backtracking-free, so we substitute a loose form
that NPE itself half-broke per audit Track B §2.1)
* dropped the Java atomic group `(?>...)` from helper-object regex —
Rust's NFA is already linear-time
* nsig fixup substitutes `(?:"undefined"|'undefined')` for the
\1 backref; harmless loosening
* sig and nsig assembled snippets prepend `var` — QuickJS rejects
bare-assignment to undeclared identifiers; NPE relied on Rhino's
non-strict mode
Tests:
* 43 lib unit tests (up from 7 in Phase 1)
* 7 Phase 2 offline integration tests against a hand-crafted
minified synthetic player.js — exercises the full sig pipeline
(build_deobfuscator → runtime::run) and nsig fixup_function
* 7 Phase 1 live smoke tests still green
57/57 total green.
