From 156b9e3684ab67470a16bfbe1f7869cad40a36b0 Mon Sep 17 00:00:00 2001 From: Duncan Tourolle Date: Sat, 22 Aug 2026 10:19:58 +0200 Subject: [PATCH] fix(playback): ask the renderer what it can decode, in one place MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four bugs, one cause. "What can this device decode" was answered in five places, four of which assumed the webview was decoding: - the device profile's direct-play codecs (cfg per platform, inline) - the transcoding targets (hardcoded "h264,hevc") - the direct-play audio narrowing (webview list, all platforms) - the client-side audio override (webview list, all platforms) - get_video_stream_url's VideoCodec (hardcoded "h264") On Android the decoder is ExoPlayer, so four of those were simply wrong there, and the costs were invisible without a device: - dts is in the tablet's own codec list, gets stripped from the profile, and is then forced to transcode by a rule about a renderer that is not playing it. - An hevc source whose *audio* is eac3 had its **picture fully re-encoded**. The server's own transcoding URL got this right — VideoCodec=h264,hevc, TranscodeReasons=AudioCodecNotSupported, video copied — but the moment a quality change or track switch re-opened the stream through our builder, the hardcoded h264 turned a cheap audio remux into a full transcode. That is a quality change silently making playback more expensive, on the exact path a viewer uses when playback is already struggling. `renderer_codecs()` and `renderer_can_decode_audio()` are now the single source, and all five sites read them. On the webview path every value resolves exactly as before, so desktop behaviour is unchanged by construction; on Android the profile becomes the device's own. The list is also what lets the server *copy* rather than re-encode: naming every codec the renderer can decode is what turns a transcode into a passthrough when the source is already playable. That is the whole of "use the best format available". Also corrects this branch's headline number where it is asserted — the architecture doc, the desktop-native-video spec and the spike. The measured 85% Android direct-play rate used a profile containing ac3/eac3; the device it was later verified on reports neither, so eac3 content correctly transcodes there. It is a ceiling for an ExoPlayer-appropriate profile, not what the app achieves, and realising any of it depends on this change. Left in place with the caveat rather than deleted, because the measurement is real — it just measures something narrower than it was quoted as measuring. Unverified: this changes what Android negotiates and has not been exercised on the tablet yet. Desktop is unchanged by construction but also unre-tested. --- docs/architecture/01-rust-backend.md | 20 +++-- docs/specs/desktop-native-video.md | 18 ++++- docs/specs/linux-native-video-spike.md | 4 + src-tauri/src/commands/player/mod.rs | 1 - src-tauri/src/repository/device_profile.rs | 91 +++++++++++++++++++++- src-tauri/src/repository/online.rs | 72 ++++++++--------- 6 files changed, 156 insertions(+), 50 deletions(-) diff --git a/docs/architecture/01-rust-backend.md b/docs/architecture/01-rust-backend.md index 75997c7c..04952047 100644 --- a/docs/architecture/01-rust-backend.md +++ b/docs/architecture/01-rust-backend.md @@ -770,11 +770,21 @@ a free passthrough as a server-side re-encode. > | Android / ExoPlayer (`h264,hevc,vp8,vp9,av1,mpeg4` + `ac3,eac3`, 6ch) | 34/40 — **85%** | > > The library is ~80% hevc (`hevc+eac3` alone is a third of it), which is why the -> two diverge so hard. **The payoff is overwhelmingly Android**, where 85% of -> plays previously burned a transcode nobody needed. Linux stays near 7% until -> libmpv decodes the picture — the h264-only profile is a WebKitGTK constraint, -> not a JellyTau choice, and is what `linux-native-video-spike.md` exists to -> remove. A reviewer should not expect this code to fix Linux on its own. +> two diverge so hard. +> +> **Read that 85% as a ceiling, not a result.** It was measured with a profile +> containing `ac3,eac3`. The Android device this was later run on reports neither +> in its `MediaCodecList` — no Dolby licence, which is normal for a tablet — so +> eac3 content, about a third of the sampled library, correctly transcodes there. +> What any given device achieves depends on its own codec list, and on the +> profile being derived from the renderer at all (DR-233), which it was not when +> the figure was taken. +> +> **The payoff is still overwhelmingly Android**, because that is where a real +> decoder is already doing the work. Linux stays near 7% until libmpv decodes the +> picture — the h264-only profile is a WebKitGTK constraint, not a JellyTau +> choice, and is what `linux-native-video-spike.md` exists to remove. A reviewer +> should not expect this code to fix Linux on its own. #### The quality ladder per source diff --git a/docs/specs/desktop-native-video.md b/docs/specs/desktop-native-video.md index a8c2ead5..643b8363 100644 --- a/docs/specs/desktop-native-video.md +++ b/docs/specs/desktop-native-video.md @@ -51,8 +51,18 @@ server: | Android / ExoPlayer — `h264,hevc,vp8,vp9,av1,mpeg4` + `ac3,eac3`, 6ch | **85%** | The sampled library is ~80% hevc. **Those rows differ only by which component -decodes.** Moving the picture to mpv is what lets the desktop row claim what the -machine can actually do, and that — not the compositing — is the product. +decodes.** + +Moving the picture to mpv is what lets the desktop row claim what the machine +can actually do, and that — not the compositing — is the product. + +> **The 85% is a ceiling, not a shipped result.** It was measured with a profile +> containing `ac3,eac3`. The Android device later used for verification reports +> neither in its `MediaCodecList` — no Dolby licence, normal for a tablet — so +> eac3 content, about a third of the sampled library, correctly transcodes there. +> Realising any of this depends on DR-233, deriving the profile from the renderer +> rather than from the platform, which is why that requirement is load-bearing +> and not tidy-up. ### One desktop video path, not two @@ -210,7 +220,7 @@ applies to the video path — but the multichannel bound still does, since a 5.1 track direct-played into a 2-channel sink is silence or inaudible dialogue. Both constraints stay, sourced from the renderer rather than assumed. -**This converts 7% into ~85%**, and it is also the change most able to break +**This is what converts the 7% figure upward** (toward, not necessarily to, the 85% ceiling — see the caveat above), and it is also the change most able to break playback silently — so it lands after compositing is proven, covered by the DR-227 override tests. @@ -346,7 +356,7 @@ and shrinks to the surface. does not, the multichannel bound survives both. The DR-233 table as a table-driven test. - **Rust, pure:** `PlaybackInfo` fixtures that transcode under the webview - profile and direct-play under the mpv profile — the 7%→85% conversion as a unit + profile and direct-play under the mpv profile — the direct-play conversion as a unit test, not only as a measurement. - **Rust:** teardown ordering — callback unregistered before context freed, freed before GL context destroyed. Structure it so the ordering is assertable without diff --git a/docs/specs/linux-native-video-spike.md b/docs/specs/linux-native-video-spike.md index b2030be4..560e0e3c 100644 --- a/docs/specs/linux-native-video-spike.md +++ b/docs/specs/linux-native-video-spike.md @@ -315,6 +315,10 @@ anything. | Linux / WebKitGTK — `h264` only, 2ch | **7%** | | Android / ExoPlayer — `h264,hevc,vp8,vp9,av1,mpeg4` + `ac3,eac3`, 6ch | **85%** | + **The 85% is a ceiling, not a shipped result** — it was measured with a + profile containing `ac3,eac3`, which the Android device later used for + verification does not support. + The library sampled is ~80% hevc. Linux sits at 7% **solely because the WebKitGTK profile can only claim h264** — not because of anything about the server or the negotiation. mpv decodes hevc, so widening the Linux device diff --git a/src-tauri/src/commands/player/mod.rs b/src-tauri/src/commands/player/mod.rs index a0a569a0..afa868bc 100644 --- a/src-tauri/src/commands/player/mod.rs +++ b/src-tauri/src/commands/player/mod.rs @@ -1768,7 +1768,6 @@ pub async fn player_set_stream_quality( }); } - // Native backend (Android/ExoPlayer): stop, repoint the queue entry at the // new URL, and reload — mirroring `VideoSeekStrategy::BackendReloadStream`. // The re-opened stream begins at zero (an HLS playlist cannot carry a start diff --git a/src-tauri/src/repository/device_profile.rs b/src-tauri/src/repository/device_profile.rs index 14b00cd5..7d16beff 100644 --- a/src-tauri/src/repository/device_profile.rs +++ b/src-tauri/src/repository/device_profile.rs @@ -212,6 +212,16 @@ pub fn subtitle_supports_external_delivery(codec: Option<&str>) -> bool { /// /// TRACES: UR-004 | DR-148 | UT-142 pub fn video_audio_codecs(detected: &str) -> String { + // Where the video renderer decodes the audio itself (ExoPlayer), the + // platform list *is* the answer and narrowing it to the webview's throws + // away codecs the device genuinely plays — dts, on the tablet this was + // found on. TRACES: UR-004, UR-080 | DR-233 + #[cfg(target_os = "android")] + { + return detected.to_string(); + } + + #[allow(unreachable_code)] let kept: Vec<&str> = detected .split(',') .filter_map(|codec| { @@ -233,6 +243,83 @@ pub fn video_audio_codecs(detected: &str) -> String { } } +/// What the renderer that will actually decode video on this platform can play. +/// +/// Returns `(video_codecs, audio_codecs)` as Jellyfin-style comma lists. +/// +/// This exists because the answer was previously derived in four places and +/// hardcoded in a fifth, each of them assuming the *webview* was decoding: +/// the device profile, the transcoding targets, the direct-play audio +/// narrowing, the client-side audio override, and `get_video_stream_url`'s +/// `VideoCodec`. On Android the decoder is ExoPlayer, so every one of those was +/// wrong there — the observed cost being an hevc source re-encoded to h264 +/// because its *audio* was eac3, and dts forced to transcode though the device +/// decodes it. +/// +/// One source, so the copies cannot disagree again. +/// +/// TRACES: UR-004, UR-080 | DR-233 +pub fn renderer_codecs() -> (String, String) { + #[cfg(target_os = "android")] + { + // ExoPlayer, and the device itself answers via MediaCodecList. + crate::player::get_detected_codecs() + .map(|(video, audio, _channels)| (video, audio)) + .unwrap_or_else(|| { + log::warn!( + "[DeviceProfile] Codec detection not complete, using conservative defaults" + ); + ("h264,hevc".to_string(), "aac,mp3".to_string()) + }) + } + + // Linux desktop draws video in the WebKitGTK HTML5