119 lines
4.6 KiB
Markdown
119 lines
4.6 KiB
Markdown
# strawcore
|
|
|
|
A Rust library for extracting YouTube stream URLs, metadata, search results, and
|
|
channel listings — without the official Data API and without an API key. It is a
|
|
Rust port of [NewPipeExtractor](https://github.com/TeamNewPipe/NewPipeExtractor)
|
|
(tracking v0.26.2), scoped to YouTube.
|
|
|
|
strawcore speaks YouTube's internal **InnerTube** endpoints directly and resolves
|
|
the playback-URL signature the same way NewPipeExtractor does: by running
|
|
YouTube's own `player.js` in an embedded JavaScript engine. The crate ships as a
|
|
plain Rust `rlib`, so it can be embedded in a desktop app, a server, a CLI, or
|
|
bridged into a mobile app over FFI.
|
|
|
|
## Why an embedded JS engine
|
|
|
|
Most lightweight extractors regex-scrape `player.js` and re-implement the
|
|
signature/`n`-parameter deobfuscator in their host language. That re-implementation
|
|
breaks every time YouTube rotates the player. NewPipeExtractor instead embeds a JS
|
|
engine and *executes* the deobfuscation function live, which survives rotations.
|
|
strawcore mirrors that architecture using [QuickJS](https://bellard.org/quickjs/)
|
|
via [`rquickjs`](https://crates.io/crates/rquickjs), so the same resilience carries
|
|
over to Rust.
|
|
|
|
## Status
|
|
|
|
Early (`0.1`). YouTube only. The public surface covers:
|
|
|
|
- **Streams** — resolve a video ID to its audio/video stream URLs, formats, and metadata.
|
|
- **Search** — query with an optional content-type filter, plus continuation (pagination).
|
|
- **Channels** — resolve a channel ID / `@handle` / legacy URL to channel info and its video list, plus continuation.
|
|
- **Localization** — preferred language + content country are honored on InnerTube requests.
|
|
|
|
## Installation
|
|
|
|
Not yet published to crates.io. Add it as a git dependency:
|
|
|
|
```toml
|
|
[dependencies]
|
|
strawcore-core = { git = "https://git.sulkta.com/Sulkta-OSS/strawcore" }
|
|
```
|
|
|
|
The library crate is imported as `strawcore_core`.
|
|
|
|
## Quick start
|
|
|
|
You provide an HTTP client by implementing the `Downloader` trait, or use the
|
|
bundled `reqwest`-based default. Install it once into the process-global service
|
|
singleton, then call the extractors.
|
|
|
|
```rust
|
|
use std::sync::Arc;
|
|
use strawcore_core::downloader::ReqwestDownloader;
|
|
use strawcore_core::youtube::{stream_extractor, search_extractor, channel};
|
|
use strawcore_core::youtube::linkhandler::search::SearchFilter;
|
|
use strawcore_core::youtube::linkhandler::channel::ChannelIdentifier;
|
|
use strawcore_core::NewPipe;
|
|
|
|
fn main() -> Result<(), Box<dyn std::error::Error>> {
|
|
// Install a downloader (here, the bundled reqwest client).
|
|
NewPipe::init(Arc::new(ReqwestDownloader::new()?));
|
|
|
|
// Resolve a video's streams + metadata.
|
|
let info = stream_extractor::stream_info("dQw4w9WgXcQ")?;
|
|
println!("{} — {} streams", info.name, info.video_streams.len());
|
|
|
|
// Search for videos.
|
|
let results = search_extractor::search("rust programming", SearchFilter::Videos)?;
|
|
println!("{} results", results.videos.len());
|
|
|
|
// Look up a channel by @handle.
|
|
let ch = channel::channel_info(ChannelIdentifier::Handle("@rustlang".into()))?;
|
|
println!("channel: {}", ch.name);
|
|
|
|
Ok(())
|
|
}
|
|
```
|
|
|
|
### Bring your own HTTP client
|
|
|
|
`Downloader` is a single-method trait (`execute(Request) -> Result<Response, _>`).
|
|
Implement it over whatever HTTP stack you already have and pass your instance to
|
|
`NewPipe::init`, instead of the bundled `ReqwestDownloader`.
|
|
|
|
### PoTokens
|
|
|
|
Some YouTube requests can require a proof-of-origin token (`po_token`), which is
|
|
minted by YouTube's BotGuard challenge and needs a real browser/WebView to solve.
|
|
strawcore does not mint these itself; it exposes a `PoTokenProvider` trait so an
|
|
embedder can supply one (for example by driving a headless browser). The default
|
|
provider declines, which is fine for the code paths that do not require a token.
|
|
|
|
## Build and test
|
|
|
|
```bash
|
|
cargo build
|
|
cargo test --lib # offline unit tests
|
|
cargo test --features online-tests # also runs tests that hit the network
|
|
```
|
|
|
|
Network-dependent integration tests are gated behind the `online-tests` feature
|
|
so offline and CI runs stay green.
|
|
|
|
## License
|
|
|
|
[GPL-3.0-or-later](LICENSE). NewPipeExtractor is GPL-3.0-licensed; this port
|
|
inherits that license.
|
|
|
|
## Acknowledgements
|
|
|
|
strawcore is a port of [NewPipeExtractor](https://github.com/TeamNewPipe/NewPipeExtractor)
|
|
by the NewPipe team, and follows its extraction architecture closely. All credit
|
|
for the original design and the hard-won InnerTube/player reverse-engineering goes
|
|
to that project.
|
|
|
|
## Contributing
|
|
|
|
Issues and pull requests are welcome. Please keep changes focused, run
|
|
`cargo fmt`, `cargo clippy`, and `cargo test --lib` before opening a PR, and add
|
|
tests for new extraction logic where practical.
|