youtube: S3 narrowed-$fields feed metadata + S7 player.js byte-release
Some checks failed
gitleaks / scan (push) Failing after 2s
Some checks failed
gitleaks / scan (push) Failing after 2s
Post-outage-review batch (Fable adversarially audited: SHIP-WITH-FIXES; the one
MED is folded below).
S3 — cheaper stream_metadata feed-enrich. On the anonymous reel path, ask
reel_item_watch for only playabilityStatus+videoDetails via a narrowed nested
$fields selector (~3-6 KB vs the full ~150-500 KB reel body). BEST-EFFORT with a
HARD fallback to the unchanged full fetch_android, so feed enrichment can never
regress: the narrowed response is trusted only when videoDetails.videoId equals
the requested id — a bare is_some() would accept a gutted videoDetails:{} or wave
through a videoId-stripped decoy, so the id binding also subsumes the decoy check
for this path (audit MED). Full extraction path untouched; po_token path skips
the narrowing.
S7 — reclaim the raw ~1.5 MB player.js once the hot-path snippets (signature
timestamp + nsig) are built, keeping player_url as the installed-generation
sentinel. memo_verdict is rekeyed off player_url (was player_code): under the
old lockstep invariant the two were equivalent, and decoupling is REQUIRED so a
byte-released generation is still "installed" for memo purposes. Release is
single-site, build-success-only, under the state lock; a build FAILURE keeps the
bytes so sibling artifacts still extract; the dead-in-android sig path re-fetches
on demand via ensure_player_code.
Also fixes three leftovers from the 2026-07-29 outage flip (23ab7ad) that the
fdroid CI never caught (it builds the APK but runs no cargo test/clippy): a test
still asserting the old visionOS-true default (cargo test was red on main), the
stale struct doc, and a derivable_impls lint (kept the manual impl explicit +
#[allow] so the load-bearing false default stays greppable). 161 tests, clippy clean.
This commit is contained in:
parent
23ab7ade41
commit
a21a6e14ab
3 changed files with 412 additions and 17 deletions
|
|
@ -3,7 +3,11 @@
|
|||
// sole public class in the JS subsystem).
|
||||
//
|
||||
// Cache layout:
|
||||
// * cached_player_code — process-lifetime, until invalidate()
|
||||
// * cached_player_code — the raw ~1.5 MB player.js. Retained only
|
||||
// until the hot-path snippets (signature timestamp + nsig) are built,
|
||||
// then DROPPED to reclaim native heap (S7); `player_url` remains as the
|
||||
// installed-generation sentinel. Re-fetched on demand if a later,
|
||||
// un-built artifact (the dead-in-android sig path) ever needs the bytes.
|
||||
// * cached_signature_timestamp
|
||||
// * cached_sig_snippet — assembled JS, ready for runtime::run
|
||||
// * cached_nsig_name + snippet
|
||||
|
|
@ -122,15 +126,20 @@ fn memo_verdict(state: &ManagerState, memo: &Option<FailMemo>) -> MemoVerdict {
|
|||
return MemoVerdict::Proceed;
|
||||
};
|
||||
let live = m.at.elapsed() < FAILURE_COOLDOWN;
|
||||
let installed_same = state.player_code.is_some()
|
||||
&& state.player_url.as_deref() == Some(m.player_url.as_str());
|
||||
// Installed-generation sentinel: `player_url` marks WHICH player.js is
|
||||
// installed, set/cleared in lockstep with install/invalidate and keyed the
|
||||
// same way the memos are. It is DECOUPLED from `player_code` (the raw bytes),
|
||||
// which S7 drops once the hot-path snippets are built — so a byte-released
|
||||
// player.js is still "installed" for memo purposes. (ensure_player_code keeps
|
||||
// keying off `player_code` because IT needs the bytes to build from.)
|
||||
let installed_same = state.player_url.as_deref() == Some(m.player_url.as_str());
|
||||
if installed_same {
|
||||
if live {
|
||||
MemoVerdict::Replay(m.error.clone())
|
||||
} else {
|
||||
MemoVerdict::Probe(m.player_url.clone())
|
||||
}
|
||||
} else if state.player_code.is_some() {
|
||||
} else if state.player_url.is_some() {
|
||||
// A different player.js has been installed since the failure —
|
||||
// retry this artifact against it.
|
||||
MemoVerdict::Proceed
|
||||
|
|
@ -141,6 +150,28 @@ fn memo_verdict(state: &ManagerState, memo: &Option<FailMemo>) -> MemoVerdict {
|
|||
}
|
||||
}
|
||||
|
||||
/// S7: reclaim the raw ~1.5 MB player.js once the hot-path artifacts are built.
|
||||
///
|
||||
/// The bytes exist only to build the derived snippets. The android-primary flow
|
||||
/// needs exactly two — the signature timestamp and the nsig deobfuscator — and
|
||||
/// both are always built per open (timestamp by `stream_info_with`, nsig by the
|
||||
/// first throttled format). Once both are present, every later call hits the
|
||||
/// snippet/timestamp cache and never touches the bytes again, so we drop them
|
||||
/// and keep `player_url` as the installed-generation sentinel.
|
||||
///
|
||||
/// Deliberately keyed on {timestamp, nsig} and NOT on the sig snippet: the sig
|
||||
/// path is dead in the android-primary flow (formats carry direct URLs), so
|
||||
/// requiring it would pin the bytes forever. If that dead path IS ever hit after
|
||||
/// a release, `ensure_player_code` (which keys off `player_code`) simply
|
||||
/// re-fetches — the audit-sanctioned worst case. Called ONLY on a clean build
|
||||
/// success; a build FAILURE keeps the bytes so sibling artifacts can still
|
||||
/// extract from them.
|
||||
fn release_player_code_if_built(state: &mut ManagerState) {
|
||||
if state.signature_timestamp.is_some() && state.nsig_snippet.is_some() {
|
||||
state.player_code = None;
|
||||
}
|
||||
}
|
||||
|
||||
/// Outcome of ensure_player_code.
|
||||
enum Ensured {
|
||||
/// player_code is installed; `fresh` = this call downloaded it.
|
||||
|
|
@ -152,6 +183,13 @@ enum Ensured {
|
|||
|
||||
#[derive(Default)]
|
||||
struct ManagerState {
|
||||
// `player_url` is the installed-GENERATION sentinel — Some iff a player.js
|
||||
// has been discovered+installed and its derived snippets are the current
|
||||
// generation. `player_code` is the raw ~1.5 MB bytes, dropped by
|
||||
// `release_player_code_if_built` once the hot-path snippets exist (S7), so
|
||||
// it can be None while `player_url` is Some. memo/rotation logic keys off
|
||||
// `player_url`; only `ensure_player_code` (which needs to build) keys off
|
||||
// `player_code`, re-fetching when the bytes are gone.
|
||||
player_url: Option<String>,
|
||||
player_code: Option<String>,
|
||||
|
||||
|
|
@ -409,6 +447,9 @@ impl PlayerManager {
|
|||
// A successful build proves any lingering memo for this
|
||||
// artifact records a superseded player.js.
|
||||
*which.memo_mut(&mut state) = None;
|
||||
// S7: once the hot-path snippets exist, the raw player.js
|
||||
// bytes are dead weight — release them (keeping player_url).
|
||||
release_player_code_if_built(&mut state);
|
||||
Ok((v, fresh))
|
||||
}
|
||||
Err(e) => {
|
||||
|
|
@ -539,6 +580,13 @@ impl PlayerManager {
|
|||
fn state_lock_is_free(&self) -> bool {
|
||||
self.state.try_lock().is_some()
|
||||
}
|
||||
|
||||
/// Test hook: are the raw player.js bytes still retained? (S7 drops them
|
||||
/// once the hot-path snippets are built, keeping `player_url`.)
|
||||
#[cfg(test)]
|
||||
fn player_code_is_retained(&self) -> bool {
|
||||
self.state.lock().player_code.is_some()
|
||||
}
|
||||
}
|
||||
|
||||
impl Default for PlayerManager {
|
||||
|
|
@ -868,4 +916,80 @@ mod tests {
|
|||
let out = worker.join().unwrap().unwrap();
|
||||
assert_eq!(out, "https://x/?n=cba");
|
||||
}
|
||||
|
||||
// ---- S7 player_code byte-release lifecycle ----------------------------
|
||||
|
||||
#[test]
|
||||
fn player_code_released_after_hot_artifacts_built_but_stays_installed() {
|
||||
let _g = GLOBAL_DOWNLOADER_LOCK.lock();
|
||||
let stub = Arc::new(StubDownloader::new("aaaaaaaa", PLAYER_GOOD));
|
||||
install(&stub);
|
||||
let mgr = PlayerManager::new();
|
||||
|
||||
// Timestamp built first (mirrors stream_info_with's worker). Only ONE of
|
||||
// the two hot artifacts exists → bytes MUST still be retained so nsig can
|
||||
// build without a re-download.
|
||||
assert_eq!(mgr.signature_timestamp("vid").unwrap(), 20244);
|
||||
assert_eq!(stub.count(), 2, "discovery + body download");
|
||||
assert!(
|
||||
mgr.player_code_is_retained(),
|
||||
"bytes kept until BOTH hot snippets exist"
|
||||
);
|
||||
|
||||
// nsig built from the SAME retained bytes (no new network) → now both
|
||||
// hot artifacts exist → bytes released, but the generation stays
|
||||
// installed (player_url intact).
|
||||
let out = mgr
|
||||
.url_with_throttling_parameter_deobfuscated("vid", "https://x/?n=abc")
|
||||
.unwrap();
|
||||
assert_eq!(out, "https://x/?n=cba");
|
||||
assert_eq!(stub.count(), 2, "nsig built from retained bytes, no refetch");
|
||||
assert!(
|
||||
!mgr.player_code_is_retained(),
|
||||
"bytes dropped once timestamp + nsig are built"
|
||||
);
|
||||
assert!(
|
||||
mgr.player_url().is_some(),
|
||||
"installed-generation sentinel survives the byte release"
|
||||
);
|
||||
|
||||
// Warm re-request of BOTH hot artifacts after the release: pure cache
|
||||
// hits, zero further network — the release is invisible to the hot path.
|
||||
assert_eq!(mgr.signature_timestamp("vid").unwrap(), 20244);
|
||||
let out = mgr
|
||||
.url_with_throttling_parameter_deobfuscated("vid", "https://x/?n=xyz")
|
||||
.unwrap();
|
||||
assert_eq!(out, "https://x/?n=zyx");
|
||||
assert_eq!(stub.count(), 2, "warm path never re-downloads after release");
|
||||
assert!(!mgr.player_code_is_retained());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn build_failure_keeps_player_code_for_sibling_extraction() {
|
||||
let _g = GLOBAL_DOWNLOADER_LOCK.lock();
|
||||
// nsig build fails on this player.js; the signature timestamp extracts.
|
||||
let stub = Arc::new(StubDownloader::new("aaaaaaaa", PLAYER_NSIG_BROKEN));
|
||||
install(&stub);
|
||||
let mgr = PlayerManager::new();
|
||||
|
||||
// nsig build FAILS → memo armed, bytes must be KEPT (never released on a
|
||||
// failure) so the sibling timestamp artifact can still extract.
|
||||
assert!(mgr
|
||||
.url_with_throttling_parameter_deobfuscated("vid", "https://x/?n=AAA")
|
||||
.is_err());
|
||||
assert_eq!(stub.count(), 2);
|
||||
assert!(
|
||||
mgr.player_code_is_retained(),
|
||||
"a build failure must NOT drop the bytes"
|
||||
);
|
||||
|
||||
// Sibling timestamp extracts from the retained bytes — no re-download —
|
||||
// and because nsig never built, the release condition stays unmet.
|
||||
assert_eq!(mgr.signature_timestamp("vid").unwrap(), 19999);
|
||||
assert_eq!(stub.count(), 2, "sibling built from retained bytes");
|
||||
assert!(
|
||||
mgr.player_code_is_retained(),
|
||||
"release requires the nsig snippet, which never built here"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -59,12 +59,14 @@ pub struct ExtractOptions {
|
|||
/// videos where ANDROID/IOS cap at 360p. Its formats are merged in
|
||||
/// (android-preferred on itag ties); a failure changes nothing.
|
||||
///
|
||||
/// Defaults to `true` (see `impl Default`), matching upstream NPE, which
|
||||
/// runs the visionOS fetch UNCONDITIONALLY in `onFetchPage`. The shipped app
|
||||
/// path (`stream_info` → `stream_info_with(.., ExtractOptions::default())`)
|
||||
/// therefore fetches visionOS on every open. Set to `false` per-call to skip
|
||||
/// it (e.g. a latency-sensitive path). The cheap `stream_metadata` path does
|
||||
/// not build streams and never fetches visionOS regardless.
|
||||
/// Defaults to `false` (see `impl Default`) as of the 2026-07-29 playback
|
||||
/// outage fix. Upstream NPE runs the visionOS fetch UNCONDITIONALLY in
|
||||
/// `onFetchPage`, but its adaptive video-only + HLS output has no working
|
||||
/// playback path in the Straw app yet, so the shipped path (`stream_info` →
|
||||
/// `stream_info_with(.., ExtractOptions::default())`) keeps it OFF. Set to
|
||||
/// `true` per-call to opt in (e.g. once the app's adaptive path is fixed and
|
||||
/// verified via the NPE differential harness). The cheap `stream_metadata`
|
||||
/// path does not build streams and never fetches visionOS regardless.
|
||||
pub fetch_visionos_client: bool,
|
||||
pub android_streaming_pot: Option<String>,
|
||||
pub ios_streaming_pot: Option<String>,
|
||||
|
|
@ -74,6 +76,12 @@ pub struct ExtractOptions {
|
|||
pub ios_player_request_pot: Option<String>,
|
||||
}
|
||||
|
||||
// Explicit manual impl (not a derive) so the load-bearing `fetch_visionos_client:
|
||||
// false` default is spelled out and greppable — a silent flip of exactly this
|
||||
// value caused the 2026-07-29 playback outage. clippy flags it as derivable only
|
||||
// because every field currently equals its type default; keep it explicit on
|
||||
// purpose so re-enabling visionOS is a deliberate, visible edit here.
|
||||
#[allow(clippy::derivable_impls)]
|
||||
impl Default for ExtractOptions {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
|
|
@ -350,7 +358,7 @@ pub fn stream_metadata(video_id: &str) -> Result<StreamInfo, ExtractionError> {
|
|||
});
|
||||
|
||||
let android_cpn = generate_content_playback_nonce();
|
||||
let player_response = fetch_android(
|
||||
let player_response = fetch_android_metadata(
|
||||
video_id,
|
||||
&localization,
|
||||
&content_country,
|
||||
|
|
@ -429,6 +437,69 @@ fn fetch_android(
|
|||
}
|
||||
}
|
||||
|
||||
/// Metadata-scoped Android fetch for the cheap `stream_metadata` feed-enrich
|
||||
/// path — do NOT use on the full extraction path, which needs `streamingData`.
|
||||
///
|
||||
/// On the anonymous reel path (the default — no PoTokenProvider registered) this
|
||||
/// tries the NARROWED-`$fields` reel variant, which asks the endpoint for only
|
||||
/// `playabilityStatus` + `videoDetails` (~3-6 KB vs the full ~150-500 KB reel
|
||||
/// body). It is BEST-EFFORT with a HARD full-fetch fallback: `reel_item_watch`
|
||||
/// may not honor the nested `$fields` selector, so if the narrowed request errors
|
||||
/// (e.g. a 400 rejecting the selector) OR returns a shape MISSING `videoDetails`
|
||||
/// (endpoint ignored it into an error/empty response), we fall back to the full
|
||||
/// `fetch_android`, which is byte-for-byte today's behavior. Net: narrowed
|
||||
/// honored → bytes saved; narrowed not honored → identical to today. Feed
|
||||
/// enrichment can never regress from this optimization.
|
||||
///
|
||||
/// The po_token `/player` path (non-default) has no `$fields` narrowing to apply,
|
||||
/// so it goes straight to the shared `fetch_android`.
|
||||
fn fetch_android_metadata(
|
||||
video_id: &str,
|
||||
localization: &Localization,
|
||||
content_country: &ContentCountry,
|
||||
cpn: &str,
|
||||
po_token: Option<&str>,
|
||||
visitor_data: Option<&str>,
|
||||
) -> Result<Value, ExtractionError> {
|
||||
// Only the anonymous reel path carries a $fields selector to narrow.
|
||||
if po_token.is_none() {
|
||||
if let Ok(r) = stream_helper::get_android_reel_player_response_metadata(
|
||||
video_id,
|
||||
localization,
|
||||
content_country,
|
||||
cpn,
|
||||
) {
|
||||
let unwrapped = unwrap_reel_player_response(r);
|
||||
// Guard: only trust the narrowed response if its videoDetails is
|
||||
// bound to the REQUESTED video — i.e. `videoDetails.videoId` is
|
||||
// present AND equals `video_id`. A bare `is_some()` would accept a
|
||||
// present-but-gutted `videoDetails: {}` (endpoint half-honored the
|
||||
// nested selector) → silently-empty title/views/duration, and would
|
||||
// also wave through a videoId-stripped decoy (the downstream decoy
|
||||
// check treats a missing videoId as valid). Binding to the id
|
||||
// subsumes the decoy check for this path and rejects every
|
||||
// not-fully-honored shape into the full-fetch fallback.
|
||||
let honored = unwrapped
|
||||
.get("videoDetails")
|
||||
.and_then(|vd| vd.get("videoId"))
|
||||
.and_then(|id| id.as_str())
|
||||
== Some(video_id);
|
||||
if honored {
|
||||
return Ok(unwrapped);
|
||||
}
|
||||
}
|
||||
}
|
||||
// Fallback (and the po_token path): the full, unchanged fetch.
|
||||
fetch_android(
|
||||
video_id,
|
||||
localization,
|
||||
content_country,
|
||||
cpn,
|
||||
po_token,
|
||||
visitor_data,
|
||||
)
|
||||
}
|
||||
|
||||
/// Unwrap the reel endpoint's one-level `playerResponse` nesting, MOVING the
|
||||
/// (~150-500 KB) subtree out instead of deep-cloning it. `r` is owned and dead
|
||||
/// after this call, so `serde_json::Map::remove` returns the owned subtree with
|
||||
|
|
@ -1254,10 +1325,12 @@ mod tests {
|
|||
}
|
||||
|
||||
#[test]
|
||||
fn default_options_fetch_visionos() {
|
||||
// Fix 1: the shipped app path (stream_info → ExtractOptions::default())
|
||||
// must fetch visionOS on every open.
|
||||
assert!(ExtractOptions::default().fetch_visionos_client);
|
||||
fn default_options_visionos_off() {
|
||||
// 2026-07-29 outage fix: the shipped app path (stream_info →
|
||||
// ExtractOptions::default()) must keep visionOS OFF — its adaptive/HLS
|
||||
// output has no working playback path in the app yet. Re-enable only
|
||||
// after the app handles it and the NPE harness verifies the shape.
|
||||
assert!(!ExtractOptions::default().fetch_visionos_client);
|
||||
}
|
||||
|
||||
// ---- I-3 xtags audioTrackType -----------------------------------------
|
||||
|
|
@ -1328,4 +1401,156 @@ mod tests {
|
|||
assert_eq!(a.track_type, None);
|
||||
assert!(a.is_descriptive);
|
||||
}
|
||||
|
||||
// ---- S3 narrowed-$fields feed-enrich fetch + HARD fallback ------------
|
||||
//
|
||||
// These touch the process-global Downloader, so they serialize on
|
||||
// DOWNLOADER_TEST_LOCK (shared with the player_manager / stream_helper
|
||||
// stub tests). They drive `fetch_android_metadata` DIRECTLY with an
|
||||
// explicit `po_token: None` so they exercise the anonymous narrowed-reel
|
||||
// path without depending on the process-global PoTokenProvider state that
|
||||
// the potoken tests mutate.
|
||||
|
||||
use crate::downloader::request::Request;
|
||||
use crate::downloader::response::Response;
|
||||
use crate::downloader::Downloader;
|
||||
use crate::exceptions::NetworkError;
|
||||
use crate::newpipe::DOWNLOADER_TEST_LOCK;
|
||||
use std::sync::atomic::{AtomicUsize, Ordering};
|
||||
use std::sync::Arc;
|
||||
|
||||
/// Serves the reel endpoint two ways, distinguished by the `$fields`
|
||||
/// selector in the URL: the NARROWED metadata request (selector contains a
|
||||
/// nested `playerResponse.` path) vs the FULL request (`$fields=playerResponse`).
|
||||
/// Counts each so tests can assert whether the fallback fired. `visitor_id`
|
||||
/// is declined (best-effort None) so the reel proceeds without it.
|
||||
struct ReelStub {
|
||||
narrowed_body: String,
|
||||
full_body: String,
|
||||
narrowed_hits: AtomicUsize,
|
||||
full_hits: AtomicUsize,
|
||||
}
|
||||
|
||||
impl ReelStub {
|
||||
fn new(narrowed_body: &str, full_body: &str) -> Self {
|
||||
Self {
|
||||
narrowed_body: narrowed_body.to_string(),
|
||||
full_body: full_body.to_string(),
|
||||
narrowed_hits: AtomicUsize::new(0),
|
||||
full_hits: AtomicUsize::new(0),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl Downloader for ReelStub {
|
||||
fn execute(&self, request: Request) -> Result<Response, NetworkError> {
|
||||
let url = request.url().to_string();
|
||||
if url.contains("visitor_id") {
|
||||
// Best-effort path: decline → reel proceeds without visitorData.
|
||||
return Err(NetworkError::Transport("reel-stub: no visitor".into()));
|
||||
}
|
||||
if url.contains("reel_item_watch") {
|
||||
// The nested dot-selector (`playerResponse.playabilityStatus,...`)
|
||||
// only appears on the narrowed metadata request; the full request
|
||||
// ends `&$fields=playerResponse` with no dot.
|
||||
let (body, counter) = if url.contains("playerResponse.") {
|
||||
(&self.narrowed_body, &self.narrowed_hits)
|
||||
} else {
|
||||
(&self.full_body, &self.full_hits)
|
||||
};
|
||||
counter.fetch_add(1, Ordering::SeqCst);
|
||||
return Ok(Response::new(200, "OK", Default::default(), body.clone(), url));
|
||||
}
|
||||
Err(NetworkError::Transport("reel-stub: unexpected request".into()))
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn feed_metadata_falls_back_to_full_reel_when_narrowed_lacks_videodetails() {
|
||||
let _g = DOWNLOADER_TEST_LOCK.lock();
|
||||
stream_helper::reset_visitor_data_cache();
|
||||
// Narrowed response has playabilityStatus but NO videoDetails — the
|
||||
// shape the endpoint would return if it ignored/rejected the nested
|
||||
// `$fields` selector into an error/empty body. The full response carries
|
||||
// the real videoDetails + streamingData.
|
||||
let stub = Arc::new(ReelStub::new(
|
||||
r#"{"playerResponse":{"playabilityStatus":{"status":"OK"}}}"#,
|
||||
r#"{"playerResponse":{"playabilityStatus":{"status":"OK"},"videoDetails":{"videoId":"vid","title":"FULL","lengthSeconds":"100","viewCount":"5"},"streamingData":{"formats":[]}}}"#,
|
||||
));
|
||||
NewPipe::init(stub.clone() as Arc<dyn Downloader>);
|
||||
|
||||
let loc = NewPipe::preferred_localization();
|
||||
let cc = NewPipe::preferred_content_country();
|
||||
let out = fetch_android_metadata("vid", &loc, &cc, "cpn", None, None).unwrap();
|
||||
|
||||
// The narrowed request was tried, found videoDetails-less, and the full
|
||||
// fetch was used as the safety fallback — feed enrich never regresses.
|
||||
assert_eq!(stub.narrowed_hits.load(Ordering::SeqCst), 1, "narrowed tried");
|
||||
assert_eq!(stub.full_hits.load(Ordering::SeqCst), 1, "fell back to full");
|
||||
// The value returned is the FULL response (unwrapped playerResponse).
|
||||
assert_eq!(
|
||||
out.get("videoDetails").and_then(|v| v.get("title")).and_then(|v| v.as_str()),
|
||||
Some("FULL")
|
||||
);
|
||||
assert!(out.get("streamingData").is_some());
|
||||
stream_helper::reset_visitor_data_cache();
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn feed_metadata_uses_narrowed_response_when_videodetails_present() {
|
||||
let _g = DOWNLOADER_TEST_LOCK.lock();
|
||||
stream_helper::reset_visitor_data_cache();
|
||||
// Narrowed response DOES carry videoDetails → the endpoint honored the
|
||||
// selector; use it directly, NO full-fetch fallback (this is the
|
||||
// bytes-saved happy path).
|
||||
let stub = Arc::new(ReelStub::new(
|
||||
r#"{"playerResponse":{"playabilityStatus":{"status":"OK"},"videoDetails":{"videoId":"vid","title":"NARROWED","lengthSeconds":"42","viewCount":"7"}}}"#,
|
||||
r#"{"playerResponse":{"playabilityStatus":{"status":"OK"},"videoDetails":{"videoId":"vid","title":"FULL"},"streamingData":{"formats":[]}}}"#,
|
||||
));
|
||||
NewPipe::init(stub.clone() as Arc<dyn Downloader>);
|
||||
|
||||
let loc = NewPipe::preferred_localization();
|
||||
let cc = NewPipe::preferred_content_country();
|
||||
let out = fetch_android_metadata("vid", &loc, &cc, "cpn", None, None).unwrap();
|
||||
|
||||
assert_eq!(stub.narrowed_hits.load(Ordering::SeqCst), 1, "narrowed used");
|
||||
assert_eq!(stub.full_hits.load(Ordering::SeqCst), 0, "no fallback needed");
|
||||
assert_eq!(
|
||||
out.get("videoDetails").and_then(|v| v.get("title")).and_then(|v| v.as_str()),
|
||||
Some("NARROWED")
|
||||
);
|
||||
// Narrowed body has no streamingData — proof we did NOT silently fall
|
||||
// back to the full response.
|
||||
assert!(out.get("streamingData").is_none());
|
||||
stream_helper::reset_visitor_data_cache();
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn feed_metadata_falls_back_when_narrowed_videoid_mismatches() {
|
||||
let _g = DOWNLOADER_TEST_LOCK.lock();
|
||||
stream_helper::reset_visitor_data_cache();
|
||||
// Narrowed response carries a videoDetails, but its videoId is NOT the
|
||||
// requested one (a gutted/decoy shape the endpoint could return if it
|
||||
// only half-honored the nested selector). The videoId-binding guard must
|
||||
// REJECT it — a bare is_some() would have accepted this empty/decoy
|
||||
// metadata — and fall through to the full fetch.
|
||||
let stub = Arc::new(ReelStub::new(
|
||||
r#"{"playerResponse":{"playabilityStatus":{"status":"OK"},"videoDetails":{"videoId":"DECOY","title":"WRONG"}}}"#,
|
||||
r#"{"playerResponse":{"playabilityStatus":{"status":"OK"},"videoDetails":{"videoId":"vid","title":"FULL","lengthSeconds":"100","viewCount":"5"},"streamingData":{"formats":[]}}}"#,
|
||||
));
|
||||
NewPipe::init(stub.clone() as Arc<dyn Downloader>);
|
||||
|
||||
let loc = NewPipe::preferred_localization();
|
||||
let cc = NewPipe::preferred_content_country();
|
||||
let out = fetch_android_metadata("vid", &loc, &cc, "cpn", None, None).unwrap();
|
||||
|
||||
assert_eq!(stub.narrowed_hits.load(Ordering::SeqCst), 1, "narrowed tried");
|
||||
assert_eq!(stub.full_hits.load(Ordering::SeqCst), 1, "mismatch -> fell back to full");
|
||||
assert_eq!(
|
||||
out.get("videoDetails").and_then(|v| v.get("title")).and_then(|v| v.as_str()),
|
||||
Some("FULL"),
|
||||
"returned the full (correct-id) response, not the decoy"
|
||||
);
|
||||
stream_helper::reset_visitor_data_cache();
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -276,12 +276,58 @@ pub fn get_android_player_response(
|
|||
|
||||
/// ANDROID `/reel/reel_item_watch` fallback — used when no poToken is
|
||||
/// available. Returns a `playerResponse`-shaped JSON wrapped inside the
|
||||
/// reel response.
|
||||
/// reel response. Requests the FULL `playerResponse` (`$fields=playerResponse`)
|
||||
/// — the stream-extraction path needs `streamingData` and everything else.
|
||||
pub fn get_android_reel_player_response(
|
||||
video_id: &str,
|
||||
localization: &Localization,
|
||||
content_country: &ContentCountry,
|
||||
cpn: &str,
|
||||
) -> Result<Value, ExtractionError> {
|
||||
reel_player_response_with_fields(video_id, localization, content_country, cpn, "playerResponse")
|
||||
}
|
||||
|
||||
/// Metadata-scoped variant of [`get_android_reel_player_response`] for the cheap
|
||||
/// `stream_metadata` feed-enrich path. Requests a NARROWED `$fields` selector so
|
||||
/// only `playabilityStatus` + `videoDetails` come back (~3-6 KB) instead of the
|
||||
/// full ~150-500 KB reel body carrying every adaptive format's ~1.5 KB URL — the
|
||||
/// feed path reads nothing outside those two subtrees (S3).
|
||||
///
|
||||
/// BEST-EFFORT, with a full-fetch fallback owned by the caller
|
||||
/// (`stream_extractor::fetch_android_metadata`): the `reel_item_watch` endpoint's
|
||||
/// acceptance of the NESTED `$fields` selector (`playerResponse.videoDetails`) is
|
||||
/// not guaranteed. If it honors the selector we save the bytes; if it rejects it
|
||||
/// (400 → `Err` here) or returns a shape without `videoDetails`, the caller falls
|
||||
/// back to the full reel fetch, so feed enrichment can NEVER regress. The
|
||||
/// dot-form nested selector matches the spelling upstream/`get_web_metadata_player_response`
|
||||
/// already use against the /player endpoint (`videoDetails.thumbnail.thumbnails`),
|
||||
/// so the syntax itself is proven — only this endpoint's honoring of it is not.
|
||||
pub fn get_android_reel_player_response_metadata(
|
||||
video_id: &str,
|
||||
localization: &Localization,
|
||||
content_country: &ContentCountry,
|
||||
cpn: &str,
|
||||
) -> Result<Value, ExtractionError> {
|
||||
reel_player_response_with_fields(
|
||||
video_id,
|
||||
localization,
|
||||
content_country,
|
||||
cpn,
|
||||
"playerResponse.playabilityStatus,playerResponse.videoDetails",
|
||||
)
|
||||
}
|
||||
|
||||
/// Shared body of the two reel helpers. `fields` is the `$fields` selector: the
|
||||
/// full path passes `"playerResponse"` (byte-for-byte today's request), the
|
||||
/// metadata path passes the narrowed nested selector. Everything else — the
|
||||
/// best-effort visitorData, the envelope, the `playerRequest`/body shape, the
|
||||
/// headers — is identical across both.
|
||||
fn reel_player_response_with_fields(
|
||||
video_id: &str,
|
||||
localization: &Localization,
|
||||
content_country: &ContentCountry,
|
||||
cpn: &str,
|
||||
fields: &str,
|
||||
) -> Result<Value, ExtractionError> {
|
||||
let mut info = InnertubeClientRequestInfo::of_android_client();
|
||||
let ua = android_user_agent(content_country);
|
||||
|
|
@ -310,7 +356,7 @@ pub fn get_android_reel_player_response(
|
|||
);
|
||||
add_player_body_fields(&mut body, video_id, cpn);
|
||||
let url = format!(
|
||||
"{YOUTUBEI_V1_GAPIS_URL}reel/reel_item_watch{DISABLE_PRETTY_PRINT_PARAM}&t={t}&id={video_id}&$fields=playerResponse",
|
||||
"{YOUTUBEI_V1_GAPIS_URL}reel/reel_item_watch{DISABLE_PRETTY_PRINT_PARAM}&t={t}&id={video_id}&$fields={fields}",
|
||||
t = generate_content_playback_nonce()
|
||||
);
|
||||
post_youtube(&url, &Value::Object(body), mobile_post_headers(&ua))
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue