docs: update README

Cargo.toml

@@ -81,7 +81,7 @@ rustypipe-downloader = { path = "./downloader", version = "0.2.1", default-featu
 [features]
 default = ["default-tls"]
 
-rss = ["quick-xml"]
+rss = ["dep:quick-xml"]
 
 # Reqwest TLS options
 default-tls = ["reqwest/default-tls"]

DEVELOPMENT.md

@@ -0,0 +1,26 @@
+## Development
+
+**Requirements:**
+
+- Current version of stable Rust
+- [`just`](https://github.com/casey/just) task runner
+- [`nextest`](https://nexte.st) test runner
+- [`pre-commit`](https://pre-commit.com/)
+- yq (YAML processor)
+
+### Tasks
+
+**Testing**
+
+- `just test` Run unit+integration tests
+- `just unittest` Run unit tests
+- `just testyt` Run YouTube integration tests
+- `just testintl` Run YouTube integration tests for all supported languages (this takes
+  a long time and is therefore not run in CI)
+- `YT_LANG=de just testyt` Run YouTube integration tests for a specific language
+
+**Tools**
+
+- `just testfiles` Download missing testfiles for unit tests
+- `just report2yaml` Convert RustyPipe reports into a more readable yaml format
+  (requires `yq`)

README.md

@@ -41,6 +41,25 @@ RustyPipe is a fully featured Rust client for the public YouTube / YouTube Music
 
 ## Getting started
 
+The RustyPipe library works as follows: at first you have to instantiate a RustyPipe
+client. You can either create it with default options or use the `RustyPipe::builder()`
+to customize it.
+
+For fetching data you have to start with a new RustyPipe query object (`rp.query()`).
+The query object holds options for an individual query (e.g. content language or
+country). You can adjust these options with setter methods. Finally call your query
+method to fetch the data you need.
+
+All query methods are async, you need the tokio runtime to execute them.
+
+```rust ignore
+let rp = RustyPipe::new();
+let rp = RustyPipe::builder().storage_dir("/app/data").build().unwrap();
+let channel = rp.query().lang(Language::De).channel_videos("UCl2mFZoRqjw_ELax4Yisf6w").await.unwrap();
+```
+
+Here are a few examples to get you started:
+
 ### Cargo.toml
 
 ```toml
@@ -162,29 +181,74 @@ Subscribers: 1780000
 ...
 ```
 
-## Development
+## Cache storage
 
-**Requirements:**
+The RustyPipe cache holds the current version numbers for all clients, the JavaScript
+code used to deobfuscate video URLs and the authentication token/cookies. Never share
+the contents of the cache if you are using authentication.
 
-- Current version of stable Rust
+By default the cache is written to a JSON file named `rustypipe_cache.json` in the
-- [`just`](https://github.com/casey/just) task runner
+current working directory. This path can be changed with the `storage_dir` option of the
-- [`nextest`](https://nexte.st) test runner
+RustyPipeBuilder. The RustyPipe CLI stores its cache in the userdata folder. The full
-- [`pre-commit`](https://pre-commit.com/)
+path on Linux is `~/.local/share/rustypipe/rustypipe_cache.json`.
-- yq (YAML processor)
 
-### Tasks
+You can integrate your own cache storage backend (e.g. database storage) by implementing
+the `CacheStorage` trait.
 
-**Testing**
+## Reports
 
-- `just test` Run unit+integration tests
+RustyPipe has a builtin error reporting system. If a YouTube response cannot be
-- `just unittest` Run unit tests
+deserialized or parsed, the original response data along with some request metadata is
-- `just testyt` Run YouTube integration tests
+written to a JSON file in the folder `rustypipe_reports`, located in RustyPipe's storage
-- `just testintl` Run YouTube integration tests for all supported languages (this takes
+directory (current folder by default, `~/.local/share/rustypipe` for the CLI).
-  a long time and is therefore not run in CI)
-- `YT_LANG=de just testyt` Run YouTube integration tests for a specific language
 
-**Tools**
+When submitting a bug report to the RustyPipe project, you can share this report to help
+resolve the issue.
 
-- `just testfiles` Download missing testfiles for unit tests
+RustyPipe reports come in 3 severity levels:
-- `just report2yaml` Convert RustyPipe reports into a more readable yaml format
+
-  (requires `yq`)
+- DBG (no error occurred, report creation was enabled by the `RustyPipeQuery::report`
+  query option)
+- WRN (parts of the response could not be deserialized/parsed, response data may be
+  incomplete)
+- ERR (entire response could not be deserialized/parsed, RustyPipe returned an error)
+
+## Authentication
+
+RustyPipe supports authentication in with your YouTube account. There are 2 supported
+authentication methods: OAuth and cookies.
+
+To execute a query with authentication, use the `.authenticated()` query option. This
+option is enabled by default for methods that require authentication like the user data
+methods. RustyPipe may automatically use authentication if available in case a video is
+age-restricted or the user is IP-banned by YouTube. If you absolutely dont want to use
+authentication, set the `.unauthenticated()` query option.
+
+### OAuth
+
+OAuth is the authentication method used by the YouTube TV client. It is more
+user-friendly than extracting cookies, however it only works with the TV client. This
+means that you can only fetch videos and not access any user data.
+
+To login using OAuth, you first have to get a new device code using the
+`rp.user_auth_get_code()` function. You can then enter the code on
+https://google.com/device and log in with your Google account. After generating the
+code, you can call the `rp.user_auth_wait_for_login()` function which waits until the
+user has logged in and stores the authentication token in the cache.
+
+### Cookies
+
+Authenticating with cookies allows you to use the functionality of the YouTube/YouTube
+Music Desktop client. You can fetch your subscribed channels, playlists and your music
+collection. You can also fetch videos using the Desktop client, including private
+videos, as long as you have access to them.
+
+To authenticate with cookies you have to log into YouTube in a fresh browser session
+(open Incognito/Private mode). Then extract the cookies from the developer tools or by
+using browser plugins like "Get cookies.txt LOCALLY"
+([Firefox](https://addons.mozilla.org/de/firefox/addon/get-cookies-txt-locally/))
+([Chromium](https://chromewebstore.google.com/detail/get-cookiestxt-locally/cclelndahbckbenkjhflpdbgdldlbecc)).
+
+You can then add the cookies to your RustyPipe client using the
+`rp.()user_auth_set_cookie` or `user_auth_set_cookie_txt` function. The cookies are
+stored in the cache. To log out, use the function `user_auth_remove_cookie`.

cli/README.md

@@ -18,13 +18,13 @@ the associated metadata. It can fetch channels, playlists, albums and videos.
 **Usage:** `rustypipe get UC2TXq_t06Hjdr2g_KdKpHQg`
 
 - `-l`, `--limit` Limit the number of list items to fetch
-- ``-t, --tab` Channel tab (options: **videos**, shorts, live, playlists, info)
+- `-t`, `--tab` Channel tab (options: **videos**, shorts, live, playlists, info)
 - `-m, --music` Use the YouTube Music API
 - `--rss`Fetch the RSS feed of a channel
 - `--comments` Get comments (options: top, latest)
 - `--lyrics` Get the lyrics for YTM tracks
 - `--player` Get the player data instead of the video details when fetching videos
-- `-c, --client-type` YT clients used to fetch player data (options: desktop, tv,
+- `-c`, `--client-type` YT clients used to fetch player data (options: desktop, tv,
   tv-embed, android, ios; if multiple clients are specified, they are attempted in
   order)
 
@@ -47,7 +47,7 @@ when searching YTM or individual channels.
 - `--date` Filter results by upload date (options: hour, day, week, month, year)
 - `--order` Sort search results (options: rating, date, views)
 - `--channel` Channel ID for searching channel videos
-- `-m, --music` Search YouTube Music in the given category (options: all, tracks,
+- `-m`, `--music` Search YouTube Music in the given category (options: all, tracks,
   videos, artists, albums, playlists-ytm, playlists-community)
 
 ## `dl`: Download videos
@@ -66,7 +66,7 @@ videos can be downloaded in parallel for improved performance.
 - `-r`, `--resolution` Video resolution (e.g. 720, 1080). Set to 0 for audio-only
 - `-a`, `--audio` Download only the audio track and write track metadata + album cover
 - `-p`, `--parallel` Number of videos downloaded in parallel (default: 8)
-- `-m, --music` Use YouTube Music for downloading playlists
+- `-m`, `--music` Use YouTube Music for downloading playlists
 - `-l`, `--limit` Limit the number of videos to download (default: 1000)
 - `-c`, `--client-type` YT clients used to fetch player data (options: desktop, tv,
   tv-embed, android, ios; if multiple clients are specified, they are attempted in
@@ -78,6 +78,52 @@ videos can be downloaded in parallel for improved performance.
 You can use the vdata command to get a new visitor data cookie. This feature may come in
 handy for testing and reproducing A/B tests.
 
+## `releases` Get YouTube Music new releases
+
+Get a list of new albums or music videos on YouTube Music
+
+**Usage:** `rustypipe releases` or `rustypipe releases --videos`
+
+## `charts`: Get YouTube Music charts
+
+Get a list of the most popular tracks and artists for a given country
+
+**Usage:** `rustypipe charts DE`
+
+## `history`: Get YouTube playback history
+
+Get a list of recently played videos or tracks
+
+### Options
+
+- `-l`, `--limit` Limit the number of list items to fetch
+- `--search` Search the playback history (unavailable on YouTube Music)
+- `-m`, `--music` Get the YouTube Music playback history
+
+## `subscriptions`: Get subscribed channels
+
+You can use the RustyPipe CLI to get a list of the channels you subscribed to. With the
+`--format` flag you can export then in different formats, including OPML and NewPipe
+JSON.
+
+With the `--feed` option you can output a list of the latest videos from your
+subscription feed instead.
+
+### Options
+
+- `-l`, `--limit` Limit the number of list items to fetch
+- `-m`, `--music` Get a list of subscribed YouTube Music artists
+- `--feed` Output YouTube Music subscription feed
+
+## `playlists`, `albums`, `tracks`: Get your YouTube library
+
+Fetch a list of all the items you stored in your YouTube/YouTube Music profile.
+
+### Options
+
+- `-l`, `--limit` Limit the number of list items to fetch
+- `-m`, `--music` (only for playlists): Get your YouTube Music playlists
+
 ## Global options
 
 - **Proxy:** RustyPipe respects the environment variables `HTTP_PROXY`, `HTTPS_PROXY`
@@ -85,7 +131,17 @@ handy for testing and reproducing A/B tests.
 - **Logging:** You can change the log level with the `RUST_LOG` environment variable, it
   is set to `info` by default
 - **Visitor data:** A custom visitor data cookie can be used with the `--vdata` flag
-- `--report`
+- **Authentication:** Use the commands `rustypipe login` and `rustypipe login --cookie`
+  to log into your Google account using either OAuth or YouTube cookies. With the
+  `--auth` flag you can use authentication for any request.
+- `--lang` Change the YouTube content language
+- `--country` Change the YouTube content country
+- `--report` Generate a report on every request and store it in a `rustypipe_reports`
+  folder in the current directory
+- `--cache-file` Change the RustyPipe cache file location (Default:
+  `~/.local/share/rustypipe/rustypipe_cache.json`)
+- `--report-dir` Change the RustyPipe report directory location (Default:
+  `~/.local/share/rustypipe/rustypipe_reports`)
 
 ### Output format
 

cli/src/main.rs

@@ -288,7 +288,7 @@ enum Commands {
         #[clap(long)]
         pretty: bool,
     },
-    /// Get YouTube music charts
+    /// Get YouTube Music charts
     Charts {
         /// Chart country
         country: Option<String>,

downloader/src/lib.rs

@@ -529,7 +529,7 @@ impl Downloader {
         self.query(DownloadVideo::from_entity(video))
     }
 
-    /// Download a video from a [`TrackItem`] (YouTube music album/playlist item)
+    /// Download a video from a [`TrackItem`] (YouTube Music album/playlist item)
     ///
     /// Providing an entity has the advantage that the download path can be determined before the video
     /// is fetched, so already downloaded videos get skipped right away.

renovate.json

@@ -1,9 +1,6 @@
 {
   "$schema": "https://docs.renovatebot.com/renovate-schema.json",
-  "extends": [
+  "extends": ["config:best-practices"],
-    "config:best-practices",
-    ":semanticCommitTypeAll(chore)"
-  ],
   "semanticCommits": "enabled",
   "automerge": true,
   "automergeStrategy": "squash",

src/client/mod.rs

@@ -225,7 +225,7 @@ struct OauthCodeRequest {
 /// The login process works as follows:
 /// 1. Obtain a user code and show it to the user
 /// 2. The user opens the login page under <https://google.com/device>, enters the code and logs in with his account
-/// 3. The application has to check periodically if the login has succeeded using [`RustyPipe::oauth_login`] or [`RustyPipe::oauth_wait_for_login`]
+/// 3. The application has to check periodically if the login has succeeded using [`RustyPipe::user_auth_login`] or [`RustyPipe::user_auth_wait_for_login`]
 /// 4. If the login is successful, the application receives a valid access/refresh token pair which can be used to access YouTube
 #[derive(Debug, Deserialize)]
 pub struct OauthDeviceCode {
@@ -1261,7 +1261,7 @@ impl RustyPipe {
     ///
     /// Returns `false` if the user has not logged in yet, in this case repeat
     /// the login attempt after a few seconds.
-    /// The function [`RustyPipe::oauth_wait_for_login`] does this automatically.
+    /// The function [`RustyPipe::user_auth_wait_for_login`] does this automatically.
     pub async fn user_auth_login(&self, code: &OauthDeviceCode) -> Result<bool, Error> {
         tracing::debug!("OAuth login attempt (user_code: {})", code.user_code);
 

src/client/music_details.rs

@@ -37,7 +37,7 @@ struct QRadio<'a> {
 }
 
 impl RustyPipeQuery {
-    /// Get the metadata of a YouTube music track
+    /// Get the metadata of a YouTube Music track
     #[tracing::instrument(skip(self), level = "error")]
     pub async fn music_details<S: AsRef<str> + Debug>(
         &self,
@@ -61,7 +61,7 @@ impl RustyPipeQuery {
         .await
     }
 
-    /// Get the lyrics of a YouTube music track
+    /// Get the lyrics of a YouTube Music track
     ///
     /// The `lyrics_id` has to be obtained using [`RustyPipeQuery::music_details`].
     #[tracing::instrument(skip(self), level = "error")]

src/client/music_search.rs

@@ -57,7 +57,7 @@ impl RustyPipeQuery {
         .await
     }
 
-    /// Search YouTube music and return items of all types
+    /// Search YouTube Music and return items of all types
     pub async fn music_search_main<S: AsRef<str>>(
         &self,
         query: S,

src/client/player.rs

@@ -81,9 +81,6 @@ impl RustyPipeQuery {
     ///
     /// The clients are used in the given order. If a client cannot fetch the requested video,
     /// an attempt is made with the next one.
-    ///
-    /// If an age-restricted video is detected, it will automatically use the [`ClientType::TvHtml5Embed`]
-    /// since it is the only one that can circumvent age restrictions.
     pub async fn player_from_clients<S: AsRef<str> + Debug>(
         &self,
         video_id: S,

src/error.rs

@@ -82,9 +82,6 @@ pub enum ExtractionError {
 #[non_exhaustive]
 pub enum UnavailabilityReason {
     /// Video/Channel is age restricted.
-    ///
-    /// Video age restriction may be circumvented with the
-    /// [`ClientType::TvHtml5Embed`](crate::client::ClientType::TvHtml5Embed) client.
     AgeRestricted,
     /// Video was deleted or censored
     Deleted,

src/model/convert.rs

@@ -4,7 +4,7 @@ use super::{
     VideoItem, YouTubeItem,
 };
 
-/// Trait for casting generic YouTube/YouTube music items to a specific kind.
+/// Trait for casting generic YouTube/YouTube Music items to a specific kind.
 ///
 /// Returns [`None`] if the item does not match.
 pub trait FromYtItem: Sized {

src/model/mod.rs

@@ -1180,7 +1180,7 @@ pub struct AlbumId {
     pub name: String,
 }
 
-/// YouTube music playlist object
+/// YouTube Music playlist object
 #[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)]
 #[non_exhaustive]
 pub struct MusicPlaylist {
@@ -1204,7 +1204,7 @@ pub struct MusicPlaylist {
     pub related_playlists: Paginator<MusicPlaylistItem>,
 }
 
-/// YouTube music album object
+/// YouTube Music album object
 #[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)]
 #[non_exhaustive]
 pub struct MusicAlbum {
@@ -1234,7 +1234,7 @@ pub struct MusicAlbum {
     pub variants: Vec<AlbumItem>,
 }
 
-/// YouTube music artist object
+/// YouTube Music artist object
 #[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)]
 #[non_exhaustive]
 pub struct MusicArtist {

src/util/mod.rs

@@ -355,7 +355,7 @@ impl<T> TryRemove<T> for Vec<T> {
 }
 
 /// Check if a channel name equals "YouTube Music"
-/// (the author of original YouTube music playlists)
+/// (the author of original YouTube Music playlists)
 pub(crate) fn is_ytm(text: &TextComponent) -> bool {
     if let TextComponent::Text { text, .. } = text {
         text.starts_with("YouTube")