docs: update README
This commit is contained in:
ThetaDev
parent
dee8a99e7a
commit
0432477451
15 changed files with 184 additions and 47 deletions
|
|
@ -81,7 +81,7 @@ rustypipe-downloader = { path = "./downloader", version = "0.2.1", default-featu
|
|||
[features]
|
||||
default = ["default-tls"]
|
||||
|
||||
rss = ["quick-xml"]
|
||||
rss = ["dep:quick-xml"]
|
||||
|
||||
# Reqwest TLS options
|
||||
default-tls = ["reqwest/default-tls"]
|
||||
|
|
|
|||
26
DEVELOPMENT.md
Normal file
26
DEVELOPMENT.md
Normal file
|
|
@ -0,0 +1,26 @@
|
|||
## Development
|
||||
|
||||
**Requirements:**
|
||||
|
||||
- Current version of stable Rust
|
||||
- [`just`](https://github.com/casey/just) task runner
|
||||
- [`nextest`](https://nexte.st) test runner
|
||||
- [`pre-commit`](https://pre-commit.com/)
|
||||
- yq (YAML processor)
|
||||
|
||||
### Tasks
|
||||
|
||||
**Testing**
|
||||
|
||||
- `just test` Run unit+integration tests
|
||||
- `just unittest` Run unit tests
|
||||
- `just testyt` Run YouTube integration tests
|
||||
- `just testintl` Run YouTube integration tests for all supported languages (this takes
|
||||
a long time and is therefore not run in CI)
|
||||
- `YT_LANG=de just testyt` Run YouTube integration tests for a specific language
|
||||
|
||||
**Tools**
|
||||
|
||||
- `just testfiles` Download missing testfiles for unit tests
|
||||
- `just report2yaml` Convert RustyPipe reports into a more readable yaml format
|
||||
(requires `yq`)
|
||||
102
README.md
102
README.md
|
|
@ -41,6 +41,25 @@ RustyPipe is a fully featured Rust client for the public YouTube / YouTube Music
|
|||
|
||||
## Getting started
|
||||
|
||||
The RustyPipe library works as follows: at first you have to instantiate a RustyPipe
|
||||
client. You can either create it with default options or use the `RustyPipe::builder()`
|
||||
to customize it.
|
||||
|
||||
For fetching data you have to start with a new RustyPipe query object (`rp.query()`).
|
||||
The query object holds options for an individual query (e.g. content language or
|
||||
country). You can adjust these options with setter methods. Finally call your query
|
||||
method to fetch the data you need.
|
||||
|
||||
All query methods are async, you need the tokio runtime to execute them.
|
||||
|
||||
```rust ignore
|
||||
let rp = RustyPipe::new();
|
||||
let rp = RustyPipe::builder().storage_dir("/app/data").build().unwrap();
|
||||
let channel = rp.query().lang(Language::De).channel_videos("UCl2mFZoRqjw_ELax4Yisf6w").await.unwrap();
|
||||
```
|
||||
|
||||
Here are a few examples to get you started:
|
||||
|
||||
### Cargo.toml
|
||||
|
||||
```toml
|
||||
|
|
@ -162,29 +181,74 @@ Subscribers: 1780000
|
|||
...
|
||||
```
|
||||
|
||||
## Development
|
||||
## Cache storage
|
||||
|
||||
**Requirements:**
|
||||
The RustyPipe cache holds the current version numbers for all clients, the JavaScript
|
||||
code used to deobfuscate video URLs and the authentication token/cookies. Never share
|
||||
the contents of the cache if you are using authentication.
|
||||
|
||||
- Current version of stable Rust
|
||||
- [`just`](https://github.com/casey/just) task runner
|
||||
- [`nextest`](https://nexte.st) test runner
|
||||
- [`pre-commit`](https://pre-commit.com/)
|
||||
- yq (YAML processor)
|
||||
By default the cache is written to a JSON file named `rustypipe_cache.json` in the
|
||||
current working directory. This path can be changed with the `storage_dir` option of the
|
||||
RustyPipeBuilder. The RustyPipe CLI stores its cache in the userdata folder. The full
|
||||
path on Linux is `~/.local/share/rustypipe/rustypipe_cache.json`.
|
||||
|
||||
### Tasks
|
||||
You can integrate your own cache storage backend (e.g. database storage) by implementing
|
||||
the `CacheStorage` trait.
|
||||
|
||||
**Testing**
|
||||
## Reports
|
||||
|
||||
- `just test` Run unit+integration tests
|
||||
- `just unittest` Run unit tests
|
||||
- `just testyt` Run YouTube integration tests
|
||||
- `just testintl` Run YouTube integration tests for all supported languages (this takes
|
||||
a long time and is therefore not run in CI)
|
||||
- `YT_LANG=de just testyt` Run YouTube integration tests for a specific language
|
||||
RustyPipe has a builtin error reporting system. If a YouTube response cannot be
|
||||
deserialized or parsed, the original response data along with some request metadata is
|
||||
written to a JSON file in the folder `rustypipe_reports`, located in RustyPipe's storage
|
||||
directory (current folder by default, `~/.local/share/rustypipe` for the CLI).
|
||||
|
||||
**Tools**
|
||||
When submitting a bug report to the RustyPipe project, you can share this report to help
|
||||
resolve the issue.
|
||||
|
||||
- `just testfiles` Download missing testfiles for unit tests
|
||||
- `just report2yaml` Convert RustyPipe reports into a more readable yaml format
|
||||
(requires `yq`)
|
||||
RustyPipe reports come in 3 severity levels:
|
||||
|
||||
- DBG (no error occurred, report creation was enabled by the `RustyPipeQuery::report`
|
||||
query option)
|
||||
- WRN (parts of the response could not be deserialized/parsed, response data may be
|
||||
incomplete)
|
||||
- ERR (entire response could not be deserialized/parsed, RustyPipe returned an error)
|
||||
|
||||
## Authentication
|
||||
|
||||
RustyPipe supports authentication in with your YouTube account. There are 2 supported
|
||||
authentication methods: OAuth and cookies.
|
||||
|
||||
To execute a query with authentication, use the `.authenticated()` query option. This
|
||||
option is enabled by default for methods that require authentication like the user data
|
||||
methods. RustyPipe may automatically use authentication if available in case a video is
|
||||
age-restricted or the user is IP-banned by YouTube. If you absolutely dont want to use
|
||||
authentication, set the `.unauthenticated()` query option.
|
||||
|
||||
### OAuth
|
||||
|
||||
OAuth is the authentication method used by the YouTube TV client. It is more
|
||||
user-friendly than extracting cookies, however it only works with the TV client. This
|
||||
means that you can only fetch videos and not access any user data.
|
||||
|
||||
To login using OAuth, you first have to get a new device code using the
|
||||
`rp.user_auth_get_code()` function. You can then enter the code on
|
||||
https://google.com/device and log in with your Google account. After generating the
|
||||
code, you can call the `rp.user_auth_wait_for_login()` function which waits until the
|
||||
user has logged in and stores the authentication token in the cache.
|
||||
|
||||
### Cookies
|
||||
|
||||
Authenticating with cookies allows you to use the functionality of the YouTube/YouTube
|
||||
Music Desktop client. You can fetch your subscribed channels, playlists and your music
|
||||
collection. You can also fetch videos using the Desktop client, including private
|
||||
videos, as long as you have access to them.
|
||||
|
||||
To authenticate with cookies you have to log into YouTube in a fresh browser session
|
||||
(open Incognito/Private mode). Then extract the cookies from the developer tools or by
|
||||
using browser plugins like "Get cookies.txt LOCALLY"
|
||||
([Firefox](https://addons.mozilla.org/de/firefox/addon/get-cookies-txt-locally/))
|
||||
([Chromium](https://chromewebstore.google.com/detail/get-cookiestxt-locally/cclelndahbckbenkjhflpdbgdldlbecc)).
|
||||
|
||||
You can then add the cookies to your RustyPipe client using the
|
||||
`rp.()user_auth_set_cookie` or `user_auth_set_cookie_txt` function. The cookies are
|
||||
stored in the cache. To log out, use the function `user_auth_remove_cookie`.
|
||||
|
|
|
|||
|
|
@ -18,13 +18,13 @@ the associated metadata. It can fetch channels, playlists, albums and videos.
|
|||
**Usage:** `rustypipe get UC2TXq_t06Hjdr2g_KdKpHQg`
|
||||
|
||||
- `-l`, `--limit` Limit the number of list items to fetch
|
||||
- ``-t, --tab` Channel tab (options: **videos**, shorts, live, playlists, info)
|
||||
- `-t`, `--tab` Channel tab (options: **videos**, shorts, live, playlists, info)
|
||||
- `-m, --music` Use the YouTube Music API
|
||||
- `--rss`Fetch the RSS feed of a channel
|
||||
- `--comments` Get comments (options: top, latest)
|
||||
- `--lyrics` Get the lyrics for YTM tracks
|
||||
- `--player` Get the player data instead of the video details when fetching videos
|
||||
- `-c, --client-type` YT clients used to fetch player data (options: desktop, tv,
|
||||
- `-c`, `--client-type` YT clients used to fetch player data (options: desktop, tv,
|
||||
tv-embed, android, ios; if multiple clients are specified, they are attempted in
|
||||
order)
|
||||
|
||||
|
|
@ -47,7 +47,7 @@ when searching YTM or individual channels.
|
|||
- `--date` Filter results by upload date (options: hour, day, week, month, year)
|
||||
- `--order` Sort search results (options: rating, date, views)
|
||||
- `--channel` Channel ID for searching channel videos
|
||||
- `-m, --music` Search YouTube Music in the given category (options: all, tracks,
|
||||
- `-m`, `--music` Search YouTube Music in the given category (options: all, tracks,
|
||||
videos, artists, albums, playlists-ytm, playlists-community)
|
||||
|
||||
## `dl`: Download videos
|
||||
|
|
@ -66,7 +66,7 @@ videos can be downloaded in parallel for improved performance.
|
|||
- `-r`, `--resolution` Video resolution (e.g. 720, 1080). Set to 0 for audio-only
|
||||
- `-a`, `--audio` Download only the audio track and write track metadata + album cover
|
||||
- `-p`, `--parallel` Number of videos downloaded in parallel (default: 8)
|
||||
- `-m, --music` Use YouTube Music for downloading playlists
|
||||
- `-m`, `--music` Use YouTube Music for downloading playlists
|
||||
- `-l`, `--limit` Limit the number of videos to download (default: 1000)
|
||||
- `-c`, `--client-type` YT clients used to fetch player data (options: desktop, tv,
|
||||
tv-embed, android, ios; if multiple clients are specified, they are attempted in
|
||||
|
|
@ -78,6 +78,52 @@ videos can be downloaded in parallel for improved performance.
|
|||
You can use the vdata command to get a new visitor data cookie. This feature may come in
|
||||
handy for testing and reproducing A/B tests.
|
||||
|
||||
## `releases` Get YouTube Music new releases
|
||||
|
||||
Get a list of new albums or music videos on YouTube Music
|
||||
|
||||
**Usage:** `rustypipe releases` or `rustypipe releases --videos`
|
||||
|
||||
## `charts`: Get YouTube Music charts
|
||||
|
||||
Get a list of the most popular tracks and artists for a given country
|
||||
|
||||
**Usage:** `rustypipe charts DE`
|
||||
|
||||
## `history`: Get YouTube playback history
|
||||
|
||||
Get a list of recently played videos or tracks
|
||||
|
||||
### Options
|
||||
|
||||
- `-l`, `--limit` Limit the number of list items to fetch
|
||||
- `--search` Search the playback history (unavailable on YouTube Music)
|
||||
- `-m`, `--music` Get the YouTube Music playback history
|
||||
|
||||
## `subscriptions`: Get subscribed channels
|
||||
|
||||
You can use the RustyPipe CLI to get a list of the channels you subscribed to. With the
|
||||
`--format` flag you can export then in different formats, including OPML and NewPipe
|
||||
JSON.
|
||||
|
||||
With the `--feed` option you can output a list of the latest videos from your
|
||||
subscription feed instead.
|
||||
|
||||
### Options
|
||||
|
||||
- `-l`, `--limit` Limit the number of list items to fetch
|
||||
- `-m`, `--music` Get a list of subscribed YouTube Music artists
|
||||
- `--feed` Output YouTube Music subscription feed
|
||||
|
||||
## `playlists`, `albums`, `tracks`: Get your YouTube library
|
||||
|
||||
Fetch a list of all the items you stored in your YouTube/YouTube Music profile.
|
||||
|
||||
### Options
|
||||
|
||||
- `-l`, `--limit` Limit the number of list items to fetch
|
||||
- `-m`, `--music` (only for playlists): Get your YouTube Music playlists
|
||||
|
||||
## Global options
|
||||
|
||||
- **Proxy:** RustyPipe respects the environment variables `HTTP_PROXY`, `HTTPS_PROXY`
|
||||
|
|
@ -85,7 +131,17 @@ handy for testing and reproducing A/B tests.
|
|||
- **Logging:** You can change the log level with the `RUST_LOG` environment variable, it
|
||||
is set to `info` by default
|
||||
- **Visitor data:** A custom visitor data cookie can be used with the `--vdata` flag
|
||||
- `--report`
|
||||
- **Authentication:** Use the commands `rustypipe login` and `rustypipe login --cookie`
|
||||
to log into your Google account using either OAuth or YouTube cookies. With the
|
||||
`--auth` flag you can use authentication for any request.
|
||||
- `--lang` Change the YouTube content language
|
||||
- `--country` Change the YouTube content country
|
||||
- `--report` Generate a report on every request and store it in a `rustypipe_reports`
|
||||
folder in the current directory
|
||||
- `--cache-file` Change the RustyPipe cache file location (Default:
|
||||
`~/.local/share/rustypipe/rustypipe_cache.json`)
|
||||
- `--report-dir` Change the RustyPipe report directory location (Default:
|
||||
`~/.local/share/rustypipe/rustypipe_reports`)
|
||||
|
||||
### Output format
|
||||
|
||||
|
|
|
|||
|
|
@ -288,7 +288,7 @@ enum Commands {
|
|||
#[clap(long)]
|
||||
pretty: bool,
|
||||
},
|
||||
/// Get YouTube music charts
|
||||
/// Get YouTube Music charts
|
||||
Charts {
|
||||
/// Chart country
|
||||
country: Option<String>,
|
||||
|
|
|
|||
|
|
@ -529,7 +529,7 @@ impl Downloader {
|
|||
self.query(DownloadVideo::from_entity(video))
|
||||
}
|
||||
|
||||
/// Download a video from a [`TrackItem`] (YouTube music album/playlist item)
|
||||
/// Download a video from a [`TrackItem`] (YouTube Music album/playlist item)
|
||||
///
|
||||
/// Providing an entity has the advantage that the download path can be determined before the video
|
||||
/// is fetched, so already downloaded videos get skipped right away.
|
||||
|
|
|
|||
|
|
@ -1,9 +1,6 @@
|
|||
{
|
||||
"$schema": "https://docs.renovatebot.com/renovate-schema.json",
|
||||
"extends": [
|
||||
"config:best-practices",
|
||||
":semanticCommitTypeAll(chore)"
|
||||
],
|
||||
"extends": ["config:best-practices"],
|
||||
"semanticCommits": "enabled",
|
||||
"automerge": true,
|
||||
"automergeStrategy": "squash",
|
||||
|
|
|
|||
|
|
@ -225,7 +225,7 @@ struct OauthCodeRequest {
|
|||
/// The login process works as follows:
|
||||
/// 1. Obtain a user code and show it to the user
|
||||
/// 2. The user opens the login page under <https://google.com/device>, enters the code and logs in with his account
|
||||
/// 3. The application has to check periodically if the login has succeeded using [`RustyPipe::oauth_login`] or [`RustyPipe::oauth_wait_for_login`]
|
||||
/// 3. The application has to check periodically if the login has succeeded using [`RustyPipe::user_auth_login`] or [`RustyPipe::user_auth_wait_for_login`]
|
||||
/// 4. If the login is successful, the application receives a valid access/refresh token pair which can be used to access YouTube
|
||||
#[derive(Debug, Deserialize)]
|
||||
pub struct OauthDeviceCode {
|
||||
|
|
@ -1261,7 +1261,7 @@ impl RustyPipe {
|
|||
///
|
||||
/// Returns `false` if the user has not logged in yet, in this case repeat
|
||||
/// the login attempt after a few seconds.
|
||||
/// The function [`RustyPipe::oauth_wait_for_login`] does this automatically.
|
||||
/// The function [`RustyPipe::user_auth_wait_for_login`] does this automatically.
|
||||
pub async fn user_auth_login(&self, code: &OauthDeviceCode) -> Result<bool, Error> {
|
||||
tracing::debug!("OAuth login attempt (user_code: {})", code.user_code);
|
||||
|
||||
|
|
|
|||
|
|
@ -37,7 +37,7 @@ struct QRadio<'a> {
|
|||
}
|
||||
|
||||
impl RustyPipeQuery {
|
||||
/// Get the metadata of a YouTube music track
|
||||
/// Get the metadata of a YouTube Music track
|
||||
#[tracing::instrument(skip(self), level = "error")]
|
||||
pub async fn music_details<S: AsRef<str> + Debug>(
|
||||
&self,
|
||||
|
|
@ -61,7 +61,7 @@ impl RustyPipeQuery {
|
|||
.await
|
||||
}
|
||||
|
||||
/// Get the lyrics of a YouTube music track
|
||||
/// Get the lyrics of a YouTube Music track
|
||||
///
|
||||
/// The `lyrics_id` has to be obtained using [`RustyPipeQuery::music_details`].
|
||||
#[tracing::instrument(skip(self), level = "error")]
|
||||
|
|
|
|||
|
|
@ -57,7 +57,7 @@ impl RustyPipeQuery {
|
|||
.await
|
||||
}
|
||||
|
||||
/// Search YouTube music and return items of all types
|
||||
/// Search YouTube Music and return items of all types
|
||||
pub async fn music_search_main<S: AsRef<str>>(
|
||||
&self,
|
||||
query: S,
|
||||
|
|
|
|||
|
|
@ -81,9 +81,6 @@ impl RustyPipeQuery {
|
|||
///
|
||||
/// The clients are used in the given order. If a client cannot fetch the requested video,
|
||||
/// an attempt is made with the next one.
|
||||
///
|
||||
/// If an age-restricted video is detected, it will automatically use the [`ClientType::TvHtml5Embed`]
|
||||
/// since it is the only one that can circumvent age restrictions.
|
||||
pub async fn player_from_clients<S: AsRef<str> + Debug>(
|
||||
&self,
|
||||
video_id: S,
|
||||
|
|
|
|||
|
|
@ -82,9 +82,6 @@ pub enum ExtractionError {
|
|||
#[non_exhaustive]
|
||||
pub enum UnavailabilityReason {
|
||||
/// Video/Channel is age restricted.
|
||||
///
|
||||
/// Video age restriction may be circumvented with the
|
||||
/// [`ClientType::TvHtml5Embed`](crate::client::ClientType::TvHtml5Embed) client.
|
||||
AgeRestricted,
|
||||
/// Video was deleted or censored
|
||||
Deleted,
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@ use super::{
|
|||
VideoItem, YouTubeItem,
|
||||
};
|
||||
|
||||
/// Trait for casting generic YouTube/YouTube music items to a specific kind.
|
||||
/// Trait for casting generic YouTube/YouTube Music items to a specific kind.
|
||||
///
|
||||
/// Returns [`None`] if the item does not match.
|
||||
pub trait FromYtItem: Sized {
|
||||
|
|
|
|||
|
|
@ -1180,7 +1180,7 @@ pub struct AlbumId {
|
|||
pub name: String,
|
||||
}
|
||||
|
||||
/// YouTube music playlist object
|
||||
/// YouTube Music playlist object
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)]
|
||||
#[non_exhaustive]
|
||||
pub struct MusicPlaylist {
|
||||
|
|
@ -1204,7 +1204,7 @@ pub struct MusicPlaylist {
|
|||
pub related_playlists: Paginator<MusicPlaylistItem>,
|
||||
}
|
||||
|
||||
/// YouTube music album object
|
||||
/// YouTube Music album object
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)]
|
||||
#[non_exhaustive]
|
||||
pub struct MusicAlbum {
|
||||
|
|
@ -1234,7 +1234,7 @@ pub struct MusicAlbum {
|
|||
pub variants: Vec<AlbumItem>,
|
||||
}
|
||||
|
||||
/// YouTube music artist object
|
||||
/// YouTube Music artist object
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)]
|
||||
#[non_exhaustive]
|
||||
pub struct MusicArtist {
|
||||
|
|
|
|||
|
|
@ -355,7 +355,7 @@ impl<T> TryRemove<T> for Vec<T> {
|
|||
}
|
||||
|
||||
/// Check if a channel name equals "YouTube Music"
|
||||
/// (the author of original YouTube music playlists)
|
||||
/// (the author of original YouTube Music playlists)
|
||||
pub(crate) fn is_ytm(text: &TextComponent) -> bool {
|
||||
if let TextComponent::Text { text, .. } = text {
|
||||
text.starts_with("YouTube")
|
||||
|
|
|
|||
Reference in a new issue