From 3740ec8f75f4501530d277a34fe2fa7f760d3854 Mon Sep 17 00:00:00 2001 From: Sulkta Date: Sun, 28 Jun 2026 12:06:47 -0700 Subject: [PATCH] docs: rewrite README for public release --- README.md | 114 +++++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 108 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 44b34df..e1a6d43 100644 --- a/README.md +++ b/README.md @@ -1,17 +1,119 @@ # strawcore -Rust port of [NewPipeExtractor](https://github.com/TeamNewPipe/NewPipeExtractor) (v0.26.2), YouTube-only. Plugs into [Straw](https://git.sulkta.com/Sulkta-OSS/straw) via UniFFI. +A Rust library for extracting YouTube stream URLs, metadata, search results, and +channel listings — without the official Data API and without an API key. It is a +Rust port of [NewPipeExtractor](https://github.com/TeamNewPipe/NewPipeExtractor) +(tracking v0.26.2), scoped to YouTube. -`rustypipe` regex-parses YouTube's `player.js` and reimplements the signature deobfuscator in Rust — every YT player rotation breaks it. NPE embeds a JS engine and executes the function live, which survives rotations. strawcore mirrors that architecture on QuickJS via rquickjs. +strawcore speaks YouTube's internal **InnerTube** endpoints directly and resolves +the playback-URL signature the same way NewPipeExtractor does: by running +YouTube's own `player.js` in an embedded JavaScript engine. The crate ships as a +plain Rust `rlib`, so it can be embedded in a desktop app, a server, a CLI, or +bridged into a mobile app over FFI. -## Build + test +## Why an embedded JS engine + +Most lightweight extractors regex-scrape `player.js` and re-implement the +signature/`n`-parameter deobfuscator in their host language. That re-implementation +breaks every time YouTube rotates the player. NewPipeExtractor instead embeds a JS +engine and *executes* the deobfuscation function live, which survives rotations. +strawcore mirrors that architecture using [QuickJS](https://bellard.org/quickjs/) +via [`rquickjs`](https://crates.io/crates/rquickjs), so the same resilience carries +over to Rust. + +## Status + +Early (`0.1`). YouTube only. The public surface covers: + +- **Streams** — resolve a video ID to its audio/video stream URLs, formats, and metadata. +- **Search** — query with an optional content-type filter, plus continuation (pagination). +- **Channels** — resolve a channel ID / `@handle` / legacy URL to channel info and its video list, plus continuation. +- **Localization** — preferred language + content country are honored on InnerTube requests. + +## Installation + +Not yet published to crates.io. Add it as a git dependency: + +```toml +[dependencies] +strawcore-core = { git = "https://git.sulkta.com/Sulkta-OSS/strawcore" } +``` + +The library crate is imported as `strawcore_core`. + +## Quick start + +You provide an HTTP client by implementing the `Downloader` trait, or use the +bundled `reqwest`-based default. Install it once into the process-global service +singleton, then call the extractors. + +```rust +use std::sync::Arc; +use strawcore_core::downloader::ReqwestDownloader; +use strawcore_core::youtube::{stream_extractor, search_extractor, channel}; +use strawcore_core::youtube::linkhandler::search::SearchFilter; +use strawcore_core::youtube::linkhandler::channel::ChannelIdentifier; +use strawcore_core::NewPipe; + +fn main() -> Result<(), Box> { + // Install a downloader (here, the bundled reqwest client). + NewPipe::init(Arc::new(ReqwestDownloader::new()?)); + + // Resolve a video's streams + metadata. + let info = stream_extractor::stream_info("dQw4w9WgXcQ")?; + println!("{} — {} streams", info.name, info.video_streams.len()); + + // Search for videos. + let results = search_extractor::search("rust programming", SearchFilter::Videos)?; + println!("{} results", results.videos.len()); + + // Look up a channel by @handle. + let ch = channel::channel_info(ChannelIdentifier::Handle("@rustlang".into()))?; + println!("channel: {}", ch.name); + + Ok(()) +} +``` + +### Bring your own HTTP client + +`Downloader` is a single-method trait (`execute(Request) -> Result`). +Implement it over whatever HTTP stack you already have and pass your instance to +`NewPipe::init`, instead of the bundled `ReqwestDownloader`. + +### PoTokens + +Some YouTube requests can require a proof-of-origin token (`po_token`), which is +minted by YouTube's BotGuard challenge and needs a real browser/WebView to solve. +strawcore does not mint these itself; it exposes a `PoTokenProvider` trait so an +embedder can supply one (for example by driving a headless browser). The default +provider declines, which is fine for the code paths that do not require a token. + +## Build and test ```bash cargo build -cargo test --lib # offline unit tests -cargo test --features online-tests # full smoke incl. live httpbin.org +cargo test --lib # offline unit tests +cargo test --features online-tests # also runs tests that hit the network ``` +Network-dependent integration tests are gated behind the `online-tests` feature +so offline and CI runs stay green. + ## License -GPL-3.0-or-later. NPE is GPL-3.0; this port inherits. +[GPL-3.0-or-later](LICENSE). NewPipeExtractor is GPL-3.0-licensed; this port +inherits that license. + +## Acknowledgements + +strawcore is a port of [NewPipeExtractor](https://github.com/TeamNewPipe/NewPipeExtractor) +by the NewPipe team, and follows its extraction architecture closely. All credit +for the original design and the hard-won InnerTube/player reverse-engineering goes +to that project. + +## Contributing + +Issues and pull requests are welcome. Please keep changes focused, run +`cargo fmt`, `cargo clippy`, and `cargo test --lib` before opening a PR, and add +tests for new extraction logic where practical.