jellytau/docs/specs/playback-backend-unification.md

Spec: Playback backend unification — findings and strategy

Status: Accepted (analysis; no code changes) Requirements: IR-004, UR-031, UR-032, UR-033 — revises the "Platform Playback Backend Parity" issue in requirements.md UX spec: n/a Supersedes / revises: informed the Android native-video and audio-parity work (both since shipped — see 05-platform-backends.md) and windows-native-audio-backend.md, still open

Summary

This spec records the outcome of an investigation into unifying JellyTau's playback backends (Linux/MPV, Android/ExoPlayer, Windows/webview) onto a single engine with hardware acceleration everywhere. The conclusion is that video cannot be unified onto a native engine, and should not be attempted. Audio can be, and that is where the remaining specs direct effort.

No code changes follow from this spec directly. It exists so the decision is written down with its evidence, and so a future session does not re-run the same investigation.

Motivation

The requirements doc carries a "Platform Playback Backend Parity" issue noting that audio settings work on Linux but not Android, and proposing eventual convergence. The natural next question — "should we just run one engine everywhere?" — needed answering before spending effort on per-backend patches.

The investigation also surfaced that several statements in requirements.md and in code comments are factually wrong. Those corrections are part of the deliverable.

Findings

1. The current architecture is not what the docs describe

Platform	Audio	Video
Linux	MPV (native, audio-only)	webview <video> + hls.js
Android	ExoPlayer (native)	webview <video> + hls.js
Windows	webview <audio>	webview <video> + hls.js

Two surprises:

• MPV never decodes video. mpv_backend.rs sets video = no and audio-display = no at construction. Linux video has always been the webview. Correspondingly, player_play_item deliberately does not load into MPV on Linux (it calls set_current_item, which only updates the queue).
• Android video is also the webview. createAdapter() in src/lib/player/adapters/index.ts hardcodes const effectiveKind = "html5" and does void backendKind, discarding the use_html5_element signal that get_player_status computes in Rust. NativePlayerAdapter is dead code, and ExoPlayer's SurfaceView path in JellyTauPlayer.kt is unreachable.

So video is already unified — on HTML5, everywhere, by accident of that hardcode — and on the path without hardware decoding on Android.

2. Native video cannot be composited with a Tauri webview

This is the load-bearing finding. It is not an mpv limitation; it defeats every candidate engine identically:

• mpv: tauri-plugin-libmpv's own platform table reads Linux ⚠️ "Experimental. Window embedding is not working."
• GStreamer (wry discussion #284, 2024): "Gstreamer was rendering above the surface and covering all html elements."
• libVLC (tauri discussion #6343, 2024): "I had to render the webview in a child window though because vlc kept rendering on top of it."

Root cause, from Tauri maintainer amrbashir (tauri#9220, 2024-03-30):

"we are limited to using Webkit2GTK on Linux and that requires a GTK window. While possible to add a GTK widget as a child X11 window inside raw X11 window, this is however a bit hacky and it is not possible on Wayland at all."

WebKitGTK, WebView2, and Android WebView each draw into their own compositor surface. A native video surface is either entirely above or entirely below the webview; it cannot interleave with HTML. Every working example in the ecosystem is the same hack — a separate child window position-synced to a getBoundingClientRect() div — which breaks on resize, scroll, and any UI drawn over the video. For JellyTau that means the controls, subtitle overlay, and mini-player.

The most recent comment on tauri#6343 (2026-05-23) confirms it is still unsolved:

"I'm faking it and the window is not truly embedded, basically when the parent moves or resizes I reset the position and size of the libmpv window to align it with an HTML div."

The principle to carry forward: audio can unify on a native engine; video cannot, because video needs a surface and the webview owns the surface.

Re-opened on Linux (2026-08-21). This finding's general form has since been falsified on Android — native video now composites behind a transparent Tauri WebView and ships on by default (see 05-platform-backends.md). The evidence above is also entirely about foreign-window embedding; mpv's render API, drawing into a GL context we own inside Tauri's own GTK tree, was never tested. linux-native-video-spike.md tests that one claim on Linux. Findings 3-6 below are untouched by it - in particular finding 3, which is an independent disqualifier a green spike would not clear.

3. mpv would regress streaming quality

mpv has no adaptive bitrate. It delegates HLS to FFmpeg's demuxer, which selects one variant at open time and never adapts; mpv#3548 (2016) requested ABR and it never landed. --hls-bitrate is a static picker defaulting to max.

The webview path already has real ABR via hls.js. Moving video to mpv would be a downgrade on every platform — no graceful degradation on weak networks, and quality changes requiring teardown and reload.

Premise in doubt (2026-08-21). "The webview path already has real ABR" was not verified against the URLs this app actually builds. get_video_stream_url requests a single rendition (one VideoBitrate, one MaxHeight), the frontend has no level-handling code (hls.levels, LEVEL_SWITCH, currentLevel appear nowhere), and this repo implements a quality switch by re-opening the stream — all of which point to a single-variant playlist, i.e. no ABR to lose. The decisive test is counting #EXT-X-STREAM-INF lines in a real master.m3u8; it needs a live server and has not been run. See linux-native-video-spike.md.

4. Crossfade is architecturally blocked on mpv

mpv's audio chain is single-stream. FFmpeg's acrossfade is an N→A filter requiring two input streams, so there is no second input to feed it. Real crossfade needs two libmpv instances with manually ramped volumes. Upstream maintainer response (mpv#4512, closed three minutes after opening):

"No. I also find crossfading stupid and complex, so the likeliness of that happening is low."

GStreamer could do it via audiomixer. mpv cannot, at any reasonable cost.

5. Engine comparison summary

Criterion	mpv	GStreamer	libVLC
Webview compositing	❌ Linux broken	❌ same wall	❌ same wall
Adaptive bitrate HLS	❌ none	✅ adaptivedemux2	✅ adaptive module
Rust bindings	⚠️ libmpv2 active; our pin is dead	✅ gstreamer-rs excellent	❌ vlc-rs abandoned (2018)
Windows cross-MSVC	⚠️ prebuilt DLL	❌ pkg-config vs cargo-xwin	❌ no better
Android packaging	✅ Maven AAR (used by Findroid)	⚠️ Cerbero/NDK, painful	✅ mature AAR
ASS/SSA subtitles	✅ libass built in	✅ libass	✅ libass
Crossfade	❌ impossible	✅ audiomixer	⚠️ unclear

Every candidate fails the first row, which is the disqualifying one.

6. Two further options ruled out

Webview <audio>/<video> everywhere (i.e. delete the native audio backends too) is dead on Android: navigator.mediaSession is deliberately compiled out of Android WebView (Chromium CL 2613133003), so lockscreen/media-notification control would be impossible. Chromium has also never shipped audioTracks. It remains fine for Windows video, which is what we already do.

FFmpeg-direct / Rust-native (ffmpeg-next, rsmpeg, Symphonia) is not close: the safe bindings do not expose hardware decode at all, ffmpeg-next is self-declared maintenance-only, and Symphonia lacks HE-AAC and gapless AAC. This is a multi-person-year path to reach parity with what we already have.

7. If libmpv is ever revisited on Android

Recorded so the next investigation starts from evidence rather than repeating the search. The dev.jdtech.mpv:libmpv AAR — maintained by Findroid's author, i.e. another Jellyfin Android client — was inspected directly:

• libmpv.so exports the full 54-function mpv_* C API with zero Java_ symbols; JNI is a separate optional ~19 KB libplayer.so. So it is drivable from Rust without a Java shim. (This is precisely what disqualifies libVLC, whose Android video path hard-requires a Java AWindow jobject.)
• ~23 MB/ABI, versus libVLC's ~46 MB/ABI.
• 🔴 The published AAR is built --enable-gpl --enable-version3 — it is GPLv3, not LGPL. Fine for us (see libmpv2-migration.md), but it would be a hard constraint for anyone shipping closed source, and an LGPL rebuild would be your own build to own.
• Top unverified risk if anyone tries this: whether libmpv2-sys can cross-compile for aarch64-linux-android against that prebuilt .so. No working example of libmpv2 on Android was found.

None of this changes the verdict — the cost is the MediaSession/foreground-service rewrite, not the bindings.

Decision

1. Do not unify video onto a native engine. Video stays in the webview with hls.js on all platforms. This is not a compromise — it is the configuration that falls out of the compositing constraint, and it is the only one that gives us ABR for free.
2. Android native video is worth a bounded spike anyway — not for unification, but because ExoPlayer's SurfaceView path already exists and would restore hardware decode plus ASS/SSA subtitles. See 05-platform-backends.md.
3. Audio parity is the real gap and is achievable without touching any of the above. See 05-platform-backends.md and windows-native-audio-backend.md.
4. Migrate the dead libmpv pin regardless of any of this. See libmpv2-migration.md.

Corrections to existing docs

These are factual errors found during the investigation. Fixing them is in scope for this spec.

Location	Says	Actually
requirements.md UR-031 (line ~44)	"Done (Linux only)"	Not implemented on any platform.
requirements.md DR-034 (line ~196)	"Done (Linux only)"	Not implemented anywhere — mpv_backend.rs has a bare // TODO: Implement crossfade via MPV audio filters if needed. Architecturally blocked on mpv (finding 4).
requirements.md parity matrix	Crossfade ✅ Linux / ❌ Android	❌ / ❌
requirements.md parity matrix	(no EQ row)	EQ is also Linux-only — build_af_filter/eq_filter_entries exist only in mpv_backend.rs. Same root cause, same fix.
nativeAdapter.ts:11-14	Native Android video "blocked upstream by tauri#10152"	tauri#10152 is a stale feature request, dead since 2024-07-01. The capability shipped in tauri commit 27d01834 (2024-09-02). Not a blocker.

Layer assignment

No new logic. The one boundary observation worth recording:

Logic / responsibility	Layer	Why it belongs there
Which video backend a platform uses (use_html5_element)	Rust	Already correctly computed in get_player_status. The frontend currently discards it — that is the bug, not the design. Restoring it means the frontend consumes a backend decision rather than making its own.

Out of scope

• Any code change. This spec is analysis; the sibling specs carry the work.
• iOS/macOS. Not current targets.
• Replacing hls.js.

Acceptance criteria

• requirements.md DR-034 status corrected; parity matrix updated (crossfade ❌/❌, EQ row added).
• Stale tauri#10152 comment in nativeAdapter.ts corrected.
• The four sibling specs exist and are linked from here.

Testing

n/a — documentation only.

TRACES

No new code. Requirement text changes only; DR-034's status line is the one substantive edit.

Notes for the implementer

• The evidence above was gathered in July 2026. The compositing constraint has been stable since 2021 (wry#284) and is maintainer-declared unfixable, so it is unlikely to change soon — but if someone revisits this, tauri#6343 and wry#284 are the threads to re-read first.
• A parallel Claude session may be active in this repo — git diff before "repairing" unexpected changes.