Pairs with strawcore's new extractor instrumentation to make "Send logs to Kayos"
actually diagnosable. Today's bot-wall outage produced a log dump that was 90%
framework noise, one mangled strawcore line, and zero failure signal.
- LogDump: split the scrubber into a FULL profile (share-sheet export + on-screen
error strings — unchanged, still over-redacts) and a lighter DOGFOOD profile
(the logs.sulkta.com ingest path, our own infra) that KEEPS bare YouTube ids
(video/channel/playlist/youtu.be — the triage signal) while still scrubbing
every real credential: signed googlevideo URLs, bearer/cookie/token headers,
sig/pot/n/cpn params, URL queries, emails, high-entropy tokens (visitorData),
and IPs. LogShipper uses the dogfood profile.
- Fixed the IPv6-compressed regex that mangled Rust module paths
(`strawcore::stream` -> `strawcor<ip>stream`) and, as a bonus, a latent
false-negative where `::1` was never scrubbed.
- Always-log extraction + playback failures (strawLogI) so a user-visible failure
always lands in the ring.
- Wrapper (rust/strawcore): one catch-all WARN in run_extract that traces EVERY
extraction failure system-wide, and raise android_logger Info->Debug so the
extractor's DEBUG tier is live in the dogfood build.
- Added a JVM unit test for the scrubber (not yet wired into CI — the repo has no
test source set; follow-up).
- Thumbnails + channel icons stay cached: pin an explicit 256MB Coil disk
cache + sized memory cache via SingletonImageLoader.Factory. Coil's
default disk cap is 2% of the device's free space, so on a storage-tight
phone the subs feed (most image-heavy screen) thrashed it and
re-downloaded thumbnails on every visit.
- SponsorBlock + Return-YouTube-Dislike clients moved Kotlin -> Rust
(strawcore net.rs: fetchSponsorSegments / fetchRydVotes). SponsorBlock
keeps its privacy-preserving SHA-256 hash-prefix lookup. Kotlin is now a
thin shim mapping the FFI records onto the SbSegment/RydVotes domain
types; behavior identical. Migration #2 of "all backend -> Rust".
- Fix crash: extract_channel_id sliced the channel URL by a length derived
from a lowercased copy of itself; to_lowercase() can change byte length
on non-ASCII, so a non-ASCII URL tail could panic across the FFI and
abort the app on a feed refresh. Now matches the prefix case-insensitively
against the original with length + char-boundary guards.
- Fix autoplay hijack: advancing to the next video resolves over ~500ms; if
you manually start a different video meanwhile, autoplay would replace
your choice with the stale next-up. Added a staleness fence.
Verified: cargo check/test/clippy on the wrapper, full Android
compileDebugKotlin green, adversarial FFI pre-push audit passed.
Two changes:
1. Launcher name is now just 'Straw', not 'Straw debug' — past the
debug-branding phase. Kept the .debug applicationId suffix (package
stays com.sulkta.straw.debug) on purpose so fdroid updates install in
place and the in-app auto-updater keeps working; dropping the suffix
would change the package id and force a reinstall that wipes everyone's
subs/history. That's a separate, deliberate release-track cutover.
2. Stream-selection logic moved out of Kotlin (resolveStreamPlayback) into
the Rust strawcore wrapper as resolve_playback(StreamInfo, max_height)
-> ResolvedStreams. The app keeps a thin shim that supplies the
resolution cap (Settings.maxResolution) and attaches SponsorBlock
segments. Byte-for-byte behavior parity with the old Kotlin picker:
highest-bitrate stream at/under the cap, lowest-height fallback when
nothing fits, first-element-wins on ties (matching Kotlin
maxByOrNull/minByOrNull, not Rust's last-on-ties max_by_key), and
isNotBlank() handling for the DASH/HLS URLs. First step of moving all
backend logic to Rust; the picking lives at the FFI boundary because it
depends on an app setting, keeping strawcore-core a pure extractor.
Wrapper cargo check + clippy clean (no new warnings); FFI surface adds
ResolvedStreams + resolvePlayback, bindings regen at build.
Wire the new strawcore continuation fetchers through UniFFI and add
load-more-on-scroll to the Channel and Search screens — previously both
loaded only page 1 and stopped.
FFI (rust/strawcore): search() now returns Page{items, continuation};
channelInfo carries videos_continuation; new searchContinuation() and
channelVideosContinuation() suspend funs map the core ContinuationPage.
Channel + Search ViewModels: loadMore() fetches the next page, dedups by
url, advances the token, and stops when the token runs out or a page
yields zero net-new items (guards a looping continuation). Result-set
swaps (channel switch / new submit / cache preview) cancel the in-flight
page, and a token fence inside the state update prevents a stale page
being spliced into a replaced list. Screens add a near-end LazyColumn
trigger (rememberLazyListState + derivedStateOf) and a footer spinner.
Strawcore — new channel_feed_rss(channel_url) and subscription_feed
(bulk fan-out 50x via tokio buffer_unordered). Fetches the YouTube
Atom RSS at /feeds/videos.xml?channel_id=UCxxx. Each call is
~50-150ms vs ~500ms for the InnerTube channel_info page-scrape.
Deps added to strawcore wrapper Cargo.toml: reqwest (rustls-tls),
quick-xml, futures. reqwest dedupes against strawcore-core's
existing reqwest dep.
App — SubscriptionFeedViewModel.fetchChannelInto swapped to
channel_feed_rss. Parallelism cranked 12 -> 50 since each fetch is
lightweight now. perChannelMax dropped 30 -> 15 (the RSS upstream
cap is 15). RSS doesn't carry duration / viewCount / avatar — those
backfill on tap-through via the existing streamInfo path. Avatar
opportunistic-refresh dropped from this path (lazy-load on
ChannelScreen open is enough).
Hide-shorts content filter — new util/ContentFilter.kt with
looksLikeShort() (URL /shorts/ match OR title contains
'#shorts'/'#short'). Settings toggle defaults off. Filter applies
at row-emit in SubsPane, SearchScreen, ChannelScreen. Paid +
age-restricted stubs in place for vc=57 when strawcore-core gets
the flags.
Expected refresh time on 50 subs: ~30s sequential -> ~1s parallel-50
RSS.
Three round-3 Opus audits ran on vc=37. NO new CRITs (round-2 work
held) but real new HIGHs — several were vc=37 own-goals.
HIGH
R3-1 recordAllWatches dropped import on capacity=0. Old: when
watches store hit MAX_WATCHES (50), capacity=0, the whole
import was discarded silently. New: build fresh import list
capped at MAX_WATCHES, then combine + take(MAX_WATCHES) so
imports always land (truncating oldest current entries).
Also: skip SP write when next === before (no-op import on
already-saturated store no longer thrashes disk).
New recordAllSearches with same shape — round-3 CVE MED-6:
importHistory was per-row recordSearch.
R3-2 / CVE-2 SubscriptionsStore.addAll counter race. The vc=36
size-delta fix snapshot `cur = _subs.value` BEFORE
updateAndGet, so a concurrent toggle inflated `added`. New:
AtomicInteger reset at the start of each lambda re-run,
counted by checking each ref against the pre-image inside
the CAS. Exactly the additions THIS call made.
R3-3 refresh() empty-channels didn't cancel inFlight. Cancel
moved to the top of refresh() unconditionally so a refresh
on the prior sub set is killed before the empty branch
clears + wipes disk.
clearInMemoryCache also cancels inFlight — without it, a
cache-disable flip during a refresh could see fetchChannelInto
re-populate the just-cleared map.
R3-4 Non-atomic `_ui.value = it.copy(...)` at init hydrate path
and clearInMemoryCache. Replaced with `_ui.update {}` for
atomicity vs concurrent refresh writes. init's
lastFetchedAt write now uses maxOf so it never regresses
past a fresh refresh value.
CVE-1 state.error rendered raw UniFFI/Rust error strings to UI
— NetworkError::Recaptcha { url } embeds full signed
googlevideo URL. User screenshots a "reCAPTCHA at <URL>"
banner → leak. All four VMs (Channel/Detail/Feed/Search)
now scrub via LogDump.scrubLine before storing.
CVE-3 pruneCacheToSubs in init can clobber concurrent
fetchChannelInto writes. init's putAll → putIfAbsent so
a fresh entry from a parallel refresh isn't overwritten
with disk-stale data.
CVE-4 SIGNED_PARAM_RE over-redacted short tokens (`\bn=`
matched `n=42` counters from any wrapped lib). Split into
SIGNED_PARAM_LONG_RE (signature/sparams/lsig/cpn/expire/
pot/sig/key — match anywhere) and SIGNED_PARAM_SHORT_RE
(n/mn/ms/mo/pl/ip/ei — require `[?&]` immediately before).
Func-HIGH-1 refresh() swallowed CancellationException as a
user-visible error. Spam-tapping Refresh produced a
"refresh failed: StandaloneCoroutineCancelled" banner.
Re-throw CancellationException; catch only real errors.
MED
R3-5 reactiveFilter did N `.lowercase()` allocations per
keystroke. Switched to contains(ignoreCase = true) — zero
allocations.
CVE-MED-5 FileProvider cache-path was "." (whole cacheDir,
including SettingsImport workdirs). Narrowed to "logs/";
LogDump.capture now writes to cacheDir/logs/ to match.
CVE-MED-7 Downloader.Request.setTitle was the raw title
(bidi-override / control chars possible). Switched to
safeTitle.
CVE-MED-8 Rust hello_from_rust value-log scrubbed to name_len.
Func-LOW-4 recordAllWatches skip-write-on-no-change (`next !==
before`).
Deferred to a follow-up (not user-facing this ship):
R3-MED-6 — Settings setMaxResolution/setThemeMode/setCacheEnabled
not atomic via updateAndGet. Inconsistent with toggle()
but the Switch UI throttles enough that no real race.
R3-MED-8 — Minibar play-button reads live controller.isPlaying
instead of listener-tracked. One-frame oscillation on
super-fast double-tap.
R3-LOW — collectAsState vs collectAsStateWithLifecycle drift.
Func-LOW-6 — refreshIfStale isActive check is TOCTOU on a
non-existent multi-threaded call surface (LaunchedEffect
+ button are both Main).
Replaces the rustypipe-backed extraction with calls into the new
NPE-port crate. The UniFFI surface Kotlin sees is unchanged:
suspend fun search(query: String): List<SearchItem>
suspend fun streamInfo(input: String): StreamInfo
suspend fun channelInfo(input: String): ChannelInfo
fun initLogging() // also wires the strawcore-core Downloader
fun helloFromRust(name: String): String
rust/strawcore/
* Cargo.toml — dropped rustypipe + rquickjs-sys direct dep;
added strawcore-core path dep (../../../strawcore)
* src/error.rs — From<strawcore_core::ExtractionError>, mapping
ContentUnavailable variants to typed
StrawcoreError cases (AgeRestricted, GeoRestricted,
Private, RequiresLogin) instead of bucketing all
to Extractor
* src/runtime.rs — Once-guarded ReqwestDownloader init via
NewPipe::init_full
* src/search.rs — search() spawn_blocks core search_extractor::search
against SearchFilter::Videos
* src/stream.rs — stream_info() resolves URL → video_id via
strawcore_core::linkhandler::stream, then
spawn_blocks core stream_extractor::stream_info,
then maps StreamInfo → wrapper DTOs (combined/
video_only/audio_only/dash/hls)
* src/channel.rs — channel_info() parses input via
strawcore_core::linkhandler::channel (handle /
custom-url / legacy-user resolution lives in
core), then spawn_blocks core channel::channel_info
Build verified: wrapper compiles linking strawcore-core, uniffi-bindgen
generates Kotlin bindings with the same suspend fun + data class
surface Kotlin already consumes. Android NDK cross-compile + APK + on-
device smoke pending (needs build-host container).
This commits onto rollback/vc18-back-to-NPE — the existing Kotlin code
still calls NewPipeExtractor directly. Switching the Kotlin side to
consume the rust wrapper is a separate cutover.
channel_info(url) UniFFI suspend fn. ChannelViewModel +
SubscriptionFeedViewModel both swap. NewPipeExtractor (Java) is OUT —
zero org.schabi.newpipe classes in the APK now.
Cleanup:
- NewPipeDownloader.kt deleted (was the OkHttp adapter)
- Thumbnails.kt deleted (rustypipe returns full URLs)
- NewPipe.init() dropped from StrawApp.onCreate
- libs.newpipe.extractor removed from build.gradle.kts
- STRAW_USER_AGENT + strawHttpClient() now live in net/Http.kt
- RydClient + SponsorBlockClient + PlayerScreen + PlaybackService all
read from net/Http.kt instead of the extractor package
rustypipe API quirks beat:
- channel_videos(id) is the right method (channel() doesn't exist)
- ChannelInfo struct = basic metadata; Channel<T> wrapper carries
name/avatar/banner + .content is the paginator of videos
- description is String (not Option), subscriber_count is Option<u64>
End state: strawApp Kotlin is ~UI + thin glue to strawcore. The Rust
core handles search / streamInfo / channel / channel_videos via UniFFI
suspend fns. Tokio + reqwest + rustls + rquickjs all packed in
libstrawcore.so (~6MB per ABI). APK 40MB total.
stream_info(url) UniFFI suspend fn replaces NewPipeExtractor's
StreamInfo.getInfo() for both VideoDetailViewModel and PlayerViewModel.
One Rust round-trip drives the detail screen render AND the player's
resolve(). The VideoDetailUiState.info field cached on detail load is
reused by the Download dialog so we don't refetch.
Deferred to U-3.5:
- like_count (rustypipe's player() doesn't surface engagement data;
a separate query is needed)
- related (player() doesn't include 'up next'; comes from a separate
endpoint). Kotlin gets empty list for now — RelatedRow handles it.
Type quirks vs my initial guesses (caught by cargo check):
- details.duration is u32, not Option<u32>
- channel is split into channel_id + channel_name, not a struct
- like_count doesn't exist at this query depth
- VideoFormat::Webm (lowercase mb), VideoCodec::Avc1 (not H264)
- video_only is a separate vec (video_only_streams), not a bool flag
