diff --git a/Cargo.toml b/Cargo.toml index eec0cb4..53da6be 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -81,7 +81,7 @@ rustypipe-downloader = { path = "./downloader", version = "0.2.1", default-featu [features] default = ["default-tls"] -rss = ["quick-xml"] +rss = ["dep:quick-xml"] # Reqwest TLS options default-tls = ["reqwest/default-tls"] diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md new file mode 100644 index 0000000..5cfc9b6 --- /dev/null +++ b/DEVELOPMENT.md @@ -0,0 +1,26 @@ +## Development + +**Requirements:** + +- Current version of stable Rust +- [`just`](https://github.com/casey/just) task runner +- [`nextest`](https://nexte.st) test runner +- [`pre-commit`](https://pre-commit.com/) +- yq (YAML processor) + +### Tasks + +**Testing** + +- `just test` Run unit+integration tests +- `just unittest` Run unit tests +- `just testyt` Run YouTube integration tests +- `just testintl` Run YouTube integration tests for all supported languages (this takes + a long time and is therefore not run in CI) +- `YT_LANG=de just testyt` Run YouTube integration tests for a specific language + +**Tools** + +- `just testfiles` Download missing testfiles for unit tests +- `just report2yaml` Convert RustyPipe reports into a more readable yaml format + (requires `yq`) diff --git a/README.md b/README.md index 5701a81..d52bb1c 100644 --- a/README.md +++ b/README.md @@ -41,6 +41,25 @@ RustyPipe is a fully featured Rust client for the public YouTube / YouTube Music ## Getting started +The RustyPipe library works as follows: at first you have to instantiate a RustyPipe +client. You can either create it with default options or use the `RustyPipe::builder()` +to customize it. + +For fetching data you have to start with a new RustyPipe query object (`rp.query()`). +The query object holds options for an individual query (e.g. content language or +country). You can adjust these options with setter methods. Finally call your query +method to fetch the data you need. + +All query methods are async, you need the tokio runtime to execute them. + +```rust ignore +let rp = RustyPipe::new(); +let rp = RustyPipe::builder().storage_dir("/app/data").build().unwrap(); +let channel = rp.query().lang(Language::De).channel_videos("UCl2mFZoRqjw_ELax4Yisf6w").await.unwrap(); +``` + +Here are a few examples to get you started: + ### Cargo.toml ```toml @@ -162,29 +181,74 @@ Subscribers: 1780000 ... ``` -## Development +## Cache storage -**Requirements:** +The RustyPipe cache holds the current version numbers for all clients, the JavaScript +code used to deobfuscate video URLs and the authentication token/cookies. Never share +the contents of the cache if you are using authentication. -- Current version of stable Rust -- [`just`](https://github.com/casey/just) task runner -- [`nextest`](https://nexte.st) test runner -- [`pre-commit`](https://pre-commit.com/) -- yq (YAML processor) +By default the cache is written to a JSON file named `rustypipe_cache.json` in the +current working directory. This path can be changed with the `storage_dir` option of the +RustyPipeBuilder. The RustyPipe CLI stores its cache in the userdata folder. The full +path on Linux is `~/.local/share/rustypipe/rustypipe_cache.json`. -### Tasks +You can integrate your own cache storage backend (e.g. database storage) by implementing +the `CacheStorage` trait. -**Testing** +## Reports -- `just test` Run unit+integration tests -- `just unittest` Run unit tests -- `just testyt` Run YouTube integration tests -- `just testintl` Run YouTube integration tests for all supported languages (this takes - a long time and is therefore not run in CI) -- `YT_LANG=de just testyt` Run YouTube integration tests for a specific language +RustyPipe has a builtin error reporting system. If a YouTube response cannot be +deserialized or parsed, the original response data along with some request metadata is +written to a JSON file in the folder `rustypipe_reports`, located in RustyPipe's storage +directory (current folder by default, `~/.local/share/rustypipe` for the CLI). -**Tools** +When submitting a bug report to the RustyPipe project, you can share this report to help +resolve the issue. -- `just testfiles` Download missing testfiles for unit tests -- `just report2yaml` Convert RustyPipe reports into a more readable yaml format - (requires `yq`) +RustyPipe reports come in 3 severity levels: + +- DBG (no error occurred, report creation was enabled by the `RustyPipeQuery::report` + query option) +- WRN (parts of the response could not be deserialized/parsed, response data may be + incomplete) +- ERR (entire response could not be deserialized/parsed, RustyPipe returned an error) + +## Authentication + +RustyPipe supports authentication in with your YouTube account. There are 2 supported +authentication methods: OAuth and cookies. + +To execute a query with authentication, use the `.authenticated()` query option. This +option is enabled by default for methods that require authentication like the user data +methods. RustyPipe may automatically use authentication if available in case a video is +age-restricted or the user is IP-banned by YouTube. If you absolutely dont want to use +authentication, set the `.unauthenticated()` query option. + +### OAuth + +OAuth is the authentication method used by the YouTube TV client. It is more +user-friendly than extracting cookies, however it only works with the TV client. This +means that you can only fetch videos and not access any user data. + +To login using OAuth, you first have to get a new device code using the +`rp.user_auth_get_code()` function. You can then enter the code on +https://google.com/device and log in with your Google account. After generating the +code, you can call the `rp.user_auth_wait_for_login()` function which waits until the +user has logged in and stores the authentication token in the cache. + +### Cookies + +Authenticating with cookies allows you to use the functionality of the YouTube/YouTube +Music Desktop client. You can fetch your subscribed channels, playlists and your music +collection. You can also fetch videos using the Desktop client, including private +videos, as long as you have access to them. + +To authenticate with cookies you have to log into YouTube in a fresh browser session +(open Incognito/Private mode). Then extract the cookies from the developer tools or by +using browser plugins like "Get cookies.txt LOCALLY" +([Firefox](https://addons.mozilla.org/de/firefox/addon/get-cookies-txt-locally/)) +([Chromium](https://chromewebstore.google.com/detail/get-cookiestxt-locally/cclelndahbckbenkjhflpdbgdldlbecc)). + +You can then add the cookies to your RustyPipe client using the +`rp.()user_auth_set_cookie` or `user_auth_set_cookie_txt` function. The cookies are +stored in the cache. To log out, use the function `user_auth_remove_cookie`. diff --git a/cli/README.md b/cli/README.md index bd6c898..a731749 100644 --- a/cli/README.md +++ b/cli/README.md @@ -18,13 +18,13 @@ the associated metadata. It can fetch channels, playlists, albums and videos. **Usage:** `rustypipe get UC2TXq_t06Hjdr2g_KdKpHQg` - `-l`, `--limit` Limit the number of list items to fetch -- ``-t, --tab` Channel tab (options: **videos**, shorts, live, playlists, info) +- `-t`, `--tab` Channel tab (options: **videos**, shorts, live, playlists, info) - `-m, --music` Use the YouTube Music API - `--rss`Fetch the RSS feed of a channel - `--comments` Get comments (options: top, latest) - `--lyrics` Get the lyrics for YTM tracks - `--player` Get the player data instead of the video details when fetching videos -- `-c, --client-type` YT clients used to fetch player data (options: desktop, tv, +- `-c`, `--client-type` YT clients used to fetch player data (options: desktop, tv, tv-embed, android, ios; if multiple clients are specified, they are attempted in order) @@ -47,7 +47,7 @@ when searching YTM or individual channels. - `--date` Filter results by upload date (options: hour, day, week, month, year) - `--order` Sort search results (options: rating, date, views) - `--channel` Channel ID for searching channel videos -- `-m, --music` Search YouTube Music in the given category (options: all, tracks, +- `-m`, `--music` Search YouTube Music in the given category (options: all, tracks, videos, artists, albums, playlists-ytm, playlists-community) ## `dl`: Download videos @@ -66,7 +66,7 @@ videos can be downloaded in parallel for improved performance. - `-r`, `--resolution` Video resolution (e.g. 720, 1080). Set to 0 for audio-only - `-a`, `--audio` Download only the audio track and write track metadata + album cover - `-p`, `--parallel` Number of videos downloaded in parallel (default: 8) -- `-m, --music` Use YouTube Music for downloading playlists +- `-m`, `--music` Use YouTube Music for downloading playlists - `-l`, `--limit` Limit the number of videos to download (default: 1000) - `-c`, `--client-type` YT clients used to fetch player data (options: desktop, tv, tv-embed, android, ios; if multiple clients are specified, they are attempted in @@ -78,6 +78,52 @@ videos can be downloaded in parallel for improved performance. You can use the vdata command to get a new visitor data cookie. This feature may come in handy for testing and reproducing A/B tests. +## `releases` Get YouTube Music new releases + +Get a list of new albums or music videos on YouTube Music + +**Usage:** `rustypipe releases` or `rustypipe releases --videos` + +## `charts`: Get YouTube Music charts + +Get a list of the most popular tracks and artists for a given country + +**Usage:** `rustypipe charts DE` + +## `history`: Get YouTube playback history + +Get a list of recently played videos or tracks + +### Options + +- `-l`, `--limit` Limit the number of list items to fetch +- `--search` Search the playback history (unavailable on YouTube Music) +- `-m`, `--music` Get the YouTube Music playback history + +## `subscriptions`: Get subscribed channels + +You can use the RustyPipe CLI to get a list of the channels you subscribed to. With the +`--format` flag you can export then in different formats, including OPML and NewPipe +JSON. + +With the `--feed` option you can output a list of the latest videos from your +subscription feed instead. + +### Options + +- `-l`, `--limit` Limit the number of list items to fetch +- `-m`, `--music` Get a list of subscribed YouTube Music artists +- `--feed` Output YouTube Music subscription feed + +## `playlists`, `albums`, `tracks`: Get your YouTube library + +Fetch a list of all the items you stored in your YouTube/YouTube Music profile. + +### Options + +- `-l`, `--limit` Limit the number of list items to fetch +- `-m`, `--music` (only for playlists): Get your YouTube Music playlists + ## Global options - **Proxy:** RustyPipe respects the environment variables `HTTP_PROXY`, `HTTPS_PROXY` @@ -85,7 +131,17 @@ handy for testing and reproducing A/B tests. - **Logging:** You can change the log level with the `RUST_LOG` environment variable, it is set to `info` by default - **Visitor data:** A custom visitor data cookie can be used with the `--vdata` flag -- `--report` +- **Authentication:** Use the commands `rustypipe login` and `rustypipe login --cookie` + to log into your Google account using either OAuth or YouTube cookies. With the + `--auth` flag you can use authentication for any request. +- `--lang` Change the YouTube content language +- `--country` Change the YouTube content country +- `--report` Generate a report on every request and store it in a `rustypipe_reports` + folder in the current directory +- `--cache-file` Change the RustyPipe cache file location (Default: + `~/.local/share/rustypipe/rustypipe_cache.json`) +- `--report-dir` Change the RustyPipe report directory location (Default: + `~/.local/share/rustypipe/rustypipe_reports`) ### Output format diff --git a/cli/src/main.rs b/cli/src/main.rs index fee4670..12e23fa 100644 --- a/cli/src/main.rs +++ b/cli/src/main.rs @@ -288,7 +288,7 @@ enum Commands { #[clap(long)] pretty: bool, }, - /// Get YouTube music charts + /// Get YouTube Music charts Charts { /// Chart country country: Option, diff --git a/downloader/src/lib.rs b/downloader/src/lib.rs index 5cb6ae4..339af94 100644 --- a/downloader/src/lib.rs +++ b/downloader/src/lib.rs @@ -529,7 +529,7 @@ impl Downloader { self.query(DownloadVideo::from_entity(video)) } - /// Download a video from a [`TrackItem`] (YouTube music album/playlist item) + /// Download a video from a [`TrackItem`] (YouTube Music album/playlist item) /// /// Providing an entity has the advantage that the download path can be determined before the video /// is fetched, so already downloaded videos get skipped right away. diff --git a/renovate.json b/renovate.json index 0aac422..2c4d77d 100644 --- a/renovate.json +++ b/renovate.json @@ -1,9 +1,6 @@ { "$schema": "https://docs.renovatebot.com/renovate-schema.json", - "extends": [ - "config:best-practices", - ":semanticCommitTypeAll(chore)" - ], + "extends": ["config:best-practices"], "semanticCommits": "enabled", "automerge": true, "automergeStrategy": "squash", diff --git a/src/client/mod.rs b/src/client/mod.rs index f439d48..1185aa3 100644 --- a/src/client/mod.rs +++ b/src/client/mod.rs @@ -225,7 +225,7 @@ struct OauthCodeRequest { /// The login process works as follows: /// 1. Obtain a user code and show it to the user /// 2. The user opens the login page under , enters the code and logs in with his account -/// 3. The application has to check periodically if the login has succeeded using [`RustyPipe::oauth_login`] or [`RustyPipe::oauth_wait_for_login`] +/// 3. The application has to check periodically if the login has succeeded using [`RustyPipe::user_auth_login`] or [`RustyPipe::user_auth_wait_for_login`] /// 4. If the login is successful, the application receives a valid access/refresh token pair which can be used to access YouTube #[derive(Debug, Deserialize)] pub struct OauthDeviceCode { @@ -1261,7 +1261,7 @@ impl RustyPipe { /// /// Returns `false` if the user has not logged in yet, in this case repeat /// the login attempt after a few seconds. - /// The function [`RustyPipe::oauth_wait_for_login`] does this automatically. + /// The function [`RustyPipe::user_auth_wait_for_login`] does this automatically. pub async fn user_auth_login(&self, code: &OauthDeviceCode) -> Result { tracing::debug!("OAuth login attempt (user_code: {})", code.user_code); diff --git a/src/client/music_details.rs b/src/client/music_details.rs index 7ae7311..1eb4e51 100644 --- a/src/client/music_details.rs +++ b/src/client/music_details.rs @@ -37,7 +37,7 @@ struct QRadio<'a> { } impl RustyPipeQuery { - /// Get the metadata of a YouTube music track + /// Get the metadata of a YouTube Music track #[tracing::instrument(skip(self), level = "error")] pub async fn music_details + Debug>( &self, @@ -61,7 +61,7 @@ impl RustyPipeQuery { .await } - /// Get the lyrics of a YouTube music track + /// Get the lyrics of a YouTube Music track /// /// The `lyrics_id` has to be obtained using [`RustyPipeQuery::music_details`]. #[tracing::instrument(skip(self), level = "error")] diff --git a/src/client/music_search.rs b/src/client/music_search.rs index b9344f5..e213517 100644 --- a/src/client/music_search.rs +++ b/src/client/music_search.rs @@ -57,7 +57,7 @@ impl RustyPipeQuery { .await } - /// Search YouTube music and return items of all types + /// Search YouTube Music and return items of all types pub async fn music_search_main>( &self, query: S, diff --git a/src/client/player.rs b/src/client/player.rs index d0da7cc..b1e808c 100644 --- a/src/client/player.rs +++ b/src/client/player.rs @@ -81,9 +81,6 @@ impl RustyPipeQuery { /// /// The clients are used in the given order. If a client cannot fetch the requested video, /// an attempt is made with the next one. - /// - /// If an age-restricted video is detected, it will automatically use the [`ClientType::TvHtml5Embed`] - /// since it is the only one that can circumvent age restrictions. pub async fn player_from_clients + Debug>( &self, video_id: S, diff --git a/src/error.rs b/src/error.rs index f41cd8a..92460c6 100644 --- a/src/error.rs +++ b/src/error.rs @@ -82,9 +82,6 @@ pub enum ExtractionError { #[non_exhaustive] pub enum UnavailabilityReason { /// Video/Channel is age restricted. - /// - /// Video age restriction may be circumvented with the - /// [`ClientType::TvHtml5Embed`](crate::client::ClientType::TvHtml5Embed) client. AgeRestricted, /// Video was deleted or censored Deleted, diff --git a/src/model/convert.rs b/src/model/convert.rs index 5610014..2843fd9 100644 --- a/src/model/convert.rs +++ b/src/model/convert.rs @@ -4,7 +4,7 @@ use super::{ VideoItem, YouTubeItem, }; -/// Trait for casting generic YouTube/YouTube music items to a specific kind. +/// Trait for casting generic YouTube/YouTube Music items to a specific kind. /// /// Returns [`None`] if the item does not match. pub trait FromYtItem: Sized { diff --git a/src/model/mod.rs b/src/model/mod.rs index 8b60e34..e265595 100644 --- a/src/model/mod.rs +++ b/src/model/mod.rs @@ -1180,7 +1180,7 @@ pub struct AlbumId { pub name: String, } -/// YouTube music playlist object +/// YouTube Music playlist object #[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] #[non_exhaustive] pub struct MusicPlaylist { @@ -1204,7 +1204,7 @@ pub struct MusicPlaylist { pub related_playlists: Paginator, } -/// YouTube music album object +/// YouTube Music album object #[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] #[non_exhaustive] pub struct MusicAlbum { @@ -1234,7 +1234,7 @@ pub struct MusicAlbum { pub variants: Vec, } -/// YouTube music artist object +/// YouTube Music artist object #[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] #[non_exhaustive] pub struct MusicArtist { diff --git a/src/util/mod.rs b/src/util/mod.rs index 8f3e782..fe294c7 100644 --- a/src/util/mod.rs +++ b/src/util/mod.rs @@ -355,7 +355,7 @@ impl TryRemove for Vec { } /// Check if a channel name equals "YouTube Music" -/// (the author of original YouTube music playlists) +/// (the author of original YouTube Music playlists) pub(crate) fn is_ytm(text: &TextComponent) -> bool { if let TextComponent::Text { text, .. } = text { text.starts_with("YouTube")