dtourolle/jellytau
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 37 Wiki Activity

fix(playback): ask the renderer what it can decode, in one place

Browse Source 
Four bugs, one cause. "What can this device decode" was answered in five
places, four of which assumed the webview was decoding:

  - the device profile's direct-play codecs      (cfg per platform, inline)
  - the transcoding targets                       (hardcoded "h264,hevc")
  - the direct-play audio narrowing               (webview list, all platforms)
  - the client-side audio override                (webview list, all platforms)
  - get_video_stream_url's VideoCodec             (hardcoded "h264")

On Android the decoder is ExoPlayer, so four of those were simply wrong there,
and the costs were invisible without a device:

  - dts is in the tablet's own codec list, gets stripped from the profile, and
    is then forced to transcode by a rule about a renderer that is not playing
    it.
  - An hevc source whose *audio* is eac3 had its **picture fully re-encoded**.
    The server's own transcoding URL got this right — VideoCodec=h264,hevc,
    TranscodeReasons=AudioCodecNotSupported, video copied — but the moment a
    quality change or track switch re-opened the stream through our builder,
    the hardcoded h264 turned a cheap audio remux into a full transcode. That
    is a quality change silently making playback more expensive, on the exact
    path a viewer uses when playback is already struggling.

`renderer_codecs()` and `renderer_can_decode_audio()` are now the single
source, and all five sites read them. On the webview path every value resolves
exactly as before, so desktop behaviour is unchanged by construction; on
Android the profile becomes the device's own.

The list is also what lets the server *copy* rather than re-encode: naming
every codec the renderer can decode is what turns a transcode into a
passthrough when the source is already playable. That is the whole of "use the
best format available".

Also corrects this branch's headline number where it is asserted — the
architecture doc, the desktop-native-video spec and the spike. The measured 85%
Android direct-play rate used a profile containing ac3/eac3; the device it was
later verified on reports neither, so eac3 content correctly transcodes there.
It is a ceiling for an ExoPlayer-appropriate profile, not what the app achieves,
and realising any of it depends on this change. Left in place with the caveat
rather than deleted, because the measurement is real — it just measures
something narrower than it was quoted as measuring.

Unverified: this changes what Android negotiates and has not been exercised on
the tablet yet. Desktop is unchanged by construction but also unre-tested.
This commit is contained in:
dtourolle
2026-08-22 13:45:03 +02:00
parent 4f6cf22419
commit 156b9e3684
6 changed files with 156 additions and 50 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/architecture/01-rust-backend.md
+15 -5
View File
@@ -770,11 +770,21 @@ a free passthrough as a server-side re-encode.
> | Android / ExoPlayer (`h264,hevc,vp8,vp9,av1,mpeg4` + `ac3,eac3`, 6ch) | 34/40 — **85%** |
>
> The library is ~80% hevc (`hevc+eac3` alone is a third of it), which is why the
> two diverge so hard. **The payoff is overwhelmingly Android**, where 85% of
> plays previously burned a transcode nobody needed. Linux stays near 7% until
> libmpv decodes the picture — the h264-only profile is a WebKitGTK constraint,
> not a JellyTau choice, and is what `linux-native-video-spike.md` exists to
> remove. A reviewer should not expect this code to fix Linux on its own.
> two diverge so hard.
>
> **Read that 85% as a ceiling, not a result.** It was measured with a profile
> containing `ac3,eac3`. The Android device this was later run on reports neither
> in its `MediaCodecList` — no Dolby licence, which is normal for a tablet — so
> eac3 content, about a third of the sampled library, correctly transcodes there.
> What any given device achieves depends on its own codec list, and on the
> profile being derived from the renderer at all (DR-233), which it was not when
> the figure was taken.
>
> **The payoff is still overwhelmingly Android**, because that is where a real
> decoder is already doing the work. Linux stays near 7% until libmpv decodes the
> picture — the h264-only profile is a WebKitGTK constraint, not a JellyTau
> choice, and is what `linux-native-video-spike.md` exists to remove. A reviewer
> should not expect this code to fix Linux on its own.
#### The quality ladder per source
docs/specs/desktop-native-video.md
+14 -4
View File
@@ -51,8 +51,18 @@ server:
| Android / ExoPlayer — `h264,hevc,vp8,vp9,av1,mpeg4` + `ac3,eac3`, 6ch | **85%** |
The sampled library is ~80% hevc. **Those rows differ only by which component
decodes.** Moving the picture to mpv is what lets the desktop row claim what the
machine can actually do, and that — not the compositing — is the product.
decodes.**
Moving the picture to mpv is what lets the desktop row claim what the machine
can actually do, and that — not the compositing — is the product.
> **The 85% is a ceiling, not a shipped result.** It was measured with a profile
> containing `ac3,eac3`. The Android device later used for verification reports
> neither in its `MediaCodecList` — no Dolby licence, normal for a tablet — so
> eac3 content, about a third of the sampled library, correctly transcodes there.
> Realising any of this depends on DR-233, deriving the profile from the renderer
> rather than from the platform, which is why that requirement is load-bearing
> and not tidy-up.
### One desktop video path, not two
@@ -210,7 +220,7 @@ applies to the video path — but the multichannel bound still does, since a 5.1
track direct-played into a 2-channel sink is silence or inaudible dialogue. Both
constraints stay, sourced from the renderer rather than assumed.
**This converts 7% into ~85%**, and it is also the change most able to break
**This is what converts the 7% figure upward** (toward, not necessarily to, the 85% ceiling — see the caveat above), and it is also the change most able to break
playback silently — so it lands after compositing is proven, covered by the
DR-227 override tests.
@@ -346,7 +356,7 @@ and shrinks to the surface.
does not, the multichannel bound survives both. The DR-233 table as a
table-driven test.
- **Rust, pure:** `PlaybackInfo` fixtures that transcode under the webview
profile and direct-play under the mpv profile — the 7%→85% conversion as a unit
profile and direct-play under the mpv profile — the direct-play conversion as a unit
test, not only as a measurement.
- **Rust:** teardown ordering — callback unregistered before context freed, freed
before GL context destroyed. Structure it so the ordering is assertable without
docs/specs/linux-native-video-spike.md
+4
View File
@@ -315,6 +315,10 @@ anything.
| Linux / WebKitGTK — `h264` only, 2ch | **7%** |
| Android / ExoPlayer — `h264,hevc,vp8,vp9,av1,mpeg4` + `ac3,eac3`, 6ch | **85%** |
**The 85% is a ceiling, not a shipped result** — it was measured with a
profile containing `ac3,eac3`, which the Android device later used for
verification does not support.
The library sampled is ~80% hevc. Linux sits at 7% **solely because the
WebKitGTK profile can only claim h264** — not because of anything about the
server or the negotiation. mpv decodes hevc, so widening the Linux device
src-tauri/src/commands/player/mod.rs
-1
View File
@@ -1768,7 +1768,6 @@ pub async fn player_set_stream_quality(
});
}
// Native backend (Android/ExoPlayer): stop, repoint the queue entry at the
// new URL, and reload — mirroring `VideoSeekStrategy::BackendReloadStream`.
// The re-opened stream begins at zero (an HLS playlist cannot carry a start
src-tauri/src/repository/device_profile.rs
+90 -1
View File
@@ -212,6 +212,16 @@ pub fn subtitle_supports_external_delivery(codec: Option<&str>) -> bool {
///
/// TRACES: UR-004 | DR-148 | UT-142
pub fn video_audio_codecs(detected: &str) -> String {
// Where the video renderer decodes the audio itself (ExoPlayer), the
// platform list *is* the answer and narrowing it to the webview's throws
// away codecs the device genuinely plays — dts, on the tablet this was
// found on. TRACES: UR-004, UR-080 | DR-233
#[cfg(target_os = "android")]
{
return detected.to_string();
}
#[allow(unreachable_code)]
let kept: Vec<&str> = detected
.split(',')
.filter_map(|codec| {
@@ -233,6 +243,83 @@ pub fn video_audio_codecs(detected: &str) -> String {
}
}
/// What the renderer that will actually decode video on this platform can play.
///
/// Returns `(video_codecs, audio_codecs)` as Jellyfin-style comma lists.
///
/// This exists because the answer was previously derived in four places and
/// hardcoded in a fifth, each of them assuming the *webview* was decoding:
/// the device profile, the transcoding targets, the direct-play audio
/// narrowing, the client-side audio override, and `get_video_stream_url`'s
/// `VideoCodec`. On Android the decoder is ExoPlayer, so every one of those was
/// wrong there — the observed cost being an hevc source re-encoded to h264
/// because its *audio* was eac3, and dts forced to transcode though the device
/// decodes it.
///
/// One source, so the copies cannot disagree again.
///
/// TRACES: UR-004, UR-080 | DR-233
pub fn renderer_codecs() -> (String, String) {
#[cfg(target_os = "android")]
{
// ExoPlayer, and the device itself answers via MediaCodecList.
crate::player::get_detected_codecs()
.map(|(video, audio, _channels)| (video, audio))
.unwrap_or_else(|| {
log::warn!(
"[DeviceProfile] Codec detection not complete, using conservative defaults"
);
("h264,hevc".to_string(), "aac,mp3".to_string())
})
}
// Linux desktop draws video in the WebKitGTK HTML5 <video> element, which
// cannot reliably decode HEVC/AV1/VP9. Claim only what it decodes, so
// Jellyfin transcodes the rest to h264 HLS. (Audio-only playback goes
// through MPV and is unaffected — that is a different renderer and a
// different profile.)
//
// When mpv draws the picture here this stops being a platform constant and
// becomes a question about the active renderer — which is the whole point of
// returning it from a function rather than a `cfg` block.
#[cfg(all(not(target_os = "android"), target_os = "linux"))]
{
("h264".to_string(), "aac,mp3,opus,vorbis,flac".to_string())
}
#[cfg(all(not(target_os = "android"), not(target_os = "linux")))]
{
(
"h264,hevc,vp8,vp9,av1,mpeg4".to_string(),
"aac,mp3,opus,vorbis,flac".to_string(),
)
}
}
/// Whether the renderer that decodes *video* on this platform can also decode
/// this audio codec.
///
/// On a webview platform this is the webview's narrow list, because the element
/// decodes both halves. On Android it is the device's own list: ExoPlayer plays
/// the audio, so judging it against the webview's capabilities transcodes files
/// that would have played.
///
/// TRACES: UR-004, UR-080 | DR-233
pub fn renderer_can_decode_audio(codec: &str) -> bool {
let codec = codec.trim();
#[cfg(target_os = "android")]
{
let (_video, audio) = renderer_codecs();
return audio
.split(',')
.any(|supported| supported.trim().eq_ignore_ascii_case(codec));
}
#[cfg(not(target_os = "android"))]
{
webview_can_decode_audio(codec)
}
}
/// Whether the webview `<video>` element can decode this audio codec.
///
/// TRACES: UR-004 | DR-149 | UT-148
@@ -260,7 +347,9 @@ pub fn webview_can_decode_audio(codec: &str) -> bool {
/// TRACES: UR-004 | DR-149 | UT-148
pub fn audio_forces_transcode(streams: &[(Option<&str>, bool)]) -> bool {
match served_audio_codec(streams) {
Some(codec) => !webview_can_decode_audio(codec),
// The renderer that will decode it, not always the webview — see
// `renderer_can_decode_audio`. TRACES: UR-004, UR-080 | DR-233
Some(codec) => !renderer_can_decode_audio(codec),
// No audio at all, or a codec the server did not name: leave it alone.
None => false,
}
src-tauri/src/repository/online.rs
+33 -39
View File
@@ -611,13 +611,29 @@ impl OnlineRepository {
self.stop_transcode(&previous).await;
}
// Build an HLS transcode URL. VideoCodec lists h264 first so the server
// transcodes HEVC/10-bit/unsupported sources to h264 the WebView can decode.
// Build an HLS transcode URL, naming every codec this renderer can
// decode rather than only h264.
//
// The list is what lets the server *copy* the video stream instead of
// re-encoding it. Hardcoding h264 meant an hevc source whose only
// problem was its audio — eac3 on a device with no Dolby licence — got
// its picture fully re-encoded to satisfy a sound problem. The server's
// own transcoding URL already did the right thing (`h264,hevc`, video
// copied, `TranscodeReasons=AudioCodecNotSupported`); this builder,
// which takes over whenever a quality change or track switch re-opens
// the stream, quietly did not — so changing quality turned a cheap
// remux into a full transcode.
//
// On the webview path this still resolves to "h264" alone, so nothing
// changes there.
//
// TRACES: UR-004, UR-080 | DR-233
let (renderer_video_codecs, _) = super::device_profile::renderer_codecs();
let mut params = vec![
("api_key", self.access_token.clone()),
("DeviceId", DEVICE_ID.to_string()),
("PlaySessionId", play_session_id),
("VideoCodec", "h264".to_string()),
("VideoCodec", renderer_video_codecs),
("AudioCodec", "aac".to_string()),
("MaxStreamingBitrate", max_bitrate.to_string()),
("VideoBitrate", video_bitrate.to_string()),
@@ -769,30 +785,11 @@ impl OnlineRepository {
) -> Result<(NegotiatedSource, String), RepoError> {
let endpoint = format!("/Items/{}/PlaybackInfo", urlencoding::encode(item_id));
// Get detected codecs from Android MediaCodecList or use platform defaults
#[cfg(target_os = "android")]
let (video_codecs, audio_codecs) = crate::player::get_detected_codecs()
.map(|(video, audio, _channels)| (video, audio))
.unwrap_or_else(|| {
warn!("[DeviceProfile] Codec detection not complete, using conservative defaults");
("h264,hevc".to_string(), "aac,mp3".to_string())
});
// Linux desktop plays video through the WebKitGTK HTML5 <video> element,
// which cannot reliably decode HEVC/AV1/VP9. Advertise only codecs the
// WebView can decode so Jellyfin transcodes anything else to h264 HLS.
// (Audio-only files still direct-play via MPV; these codecs are what
// both renderers handle, and the audio profile keeps them in full while
// the video profile is narrowed below.)
#[cfg(all(not(target_os = "android"), target_os = "linux"))]
let (video_codecs, audio_codecs) =
("h264".to_string(), "aac,mp3,opus,vorbis,flac".to_string());
#[cfg(all(not(target_os = "android"), not(target_os = "linux")))]
let (video_codecs, audio_codecs) = (
"h264,hevc,vp8,vp9,av1,mpeg4".to_string(),
"aac,mp3,opus,vorbis,flac".to_string(),
);
// What the renderer that will decode this can play. One source, shared
// with the transcode URL builder and the client-side audio override, so
// the profile we advertise and the stream we then ask for cannot
// disagree. TRACES: UR-004, UR-080 | DR-233
let (video_codecs, audio_codecs) = super::device_profile::renderer_codecs();
// Video plays in a webview <video> element on every platform, which
// decodes a narrower audio set than the platform does — so the video
@@ -864,19 +861,16 @@ impl OnlineRepository {
container: "ts".to_string(),
// The server may only transcode *to* something this renderer
// can decode. This said "h264,hevc" unconditionally while the
// direct-play profile above claims h264 alone on the webview
// path — a straight contradiction: it tells the server "I
// cannot play hevc, so re-encode it" and then "re-encoding it
// to hevc is fine". When the server took that option the
// webview got a stream it could not decode, which presents as
// video stuck on its first frame rather than as an error.
// direct-play profile claims h264 alone on the webview path —
// a straight contradiction: it tells the server "I cannot
// play hevc, so re-encode it" and then "re-encoding it to
// hevc is fine". When the server took that option the webview
// got a stream it could not decode, which presents as video
// stuck on its first frame rather than as an error.
//
// Derived from the same codec list as direct play, so the two
// halves of the profile cannot disagree again. Capped at the
// two codecs a Jellyfin server actually encodes, so widening
// the decode list never asks it for an av1 encode.
//
// TRACES: UR-004, UR-080 | DR-233
// Capped at the two codecs a Jellyfin server actually
// encodes, so a wider decode list never asks it for an av1
// encode. TRACES: UR-004, UR-080 | DR-233
video_codec: Some(
if video_codecs.contains("hevc") {
"h264,hevc"