jellytau/docs/specs/playback-backend-unification.md

# Spec: Playback backend unification — findings and strategy

**Status:** Accepted (analysis; no code changes)
**Requirements:** IR-004, UR-031, UR-032, UR-033 — revises the "Platform Playback Backend Parity" issue in requirements.md
**UX spec:** n/a
**Supersedes / revises:** informed the Android native-video and audio-parity
work (both since shipped — see
[05-platform-backends.md](../architecture/05-platform-backends.md)) and
[windows-native-audio-backend.md](windows-native-audio-backend.md), still open

## Summary

This spec records the outcome of an investigation into unifying JellyTau's
playback backends (Linux/MPV, Android/ExoPlayer, Windows/webview) onto a single
engine with hardware acceleration everywhere. **The conclusion is that video
cannot be unified onto a native engine, and should not be attempted.** Audio
*can* be, and that is where the remaining specs direct effort.

No code changes follow from this spec directly. It exists so the decision is
written down with its evidence, and so a future session does not re-run the same
investigation.

## Motivation

The requirements doc carries a "Platform Playback Backend Parity" issue noting
that audio settings work on Linux but not Android, and proposing eventual
convergence. The natural next question — "should we just run one engine
everywhere?" — needed answering before spending effort on per-backend patches.

The investigation also surfaced that several statements in requirements.md and in
code comments are factually wrong. Those corrections are part of the deliverable.

## Findings

### 1. The current architecture is not what the docs describe

| Platform | Audio | Video |
|----------|-------|-------|
| Linux | MPV (native, **audio-only**) | webview `<video>` + hls.js |
| Android | ExoPlayer (native) | **webview `<video>` + hls.js** |
| Windows | webview `<audio>` | webview `<video>` + hls.js |

Two surprises:

- **MPV never decodes video.** `mpv_backend.rs` sets `video = no` and
  `audio-display = no` at construction. Linux video has always been the webview.
  Correspondingly, `player_play_item` deliberately does *not* load into MPV on
  Linux (it calls `set_current_item`, which only updates the queue).
- **Android video is also the webview.** `createAdapter()` in
  `src/lib/player/adapters/index.ts` hardcodes `const effectiveKind = "html5"`
  and does `void backendKind`, discarding the `use_html5_element` signal that
  `get_player_status` computes in Rust. `NativePlayerAdapter` is dead code, and
  ExoPlayer's `SurfaceView` path in `JellyTauPlayer.kt` is unreachable.

So video is *already* unified — on HTML5, everywhere, by accident of that
hardcode — and on the path without hardware decoding on Android.

### 2. Native video cannot be composited with a Tauri webview

This is the load-bearing finding. It is **not** an mpv limitation; it defeats
every candidate engine identically:

- **mpv**: `tauri-plugin-libmpv`'s own platform table reads Linux ⚠️
  *"Experimental. Window embedding is not working."*
- **GStreamer** (wry discussion #284, 2024): *"Gstreamer was rendering above the
  surface and covering all html elements."*
- **libVLC** (tauri discussion #6343, 2024): *"I had to render the webview in a
  child window though because vlc kept rendering on top of it."*

Root cause, from Tauri maintainer amrbashir (tauri#9220, 2024-03-30):

> "we are limited to using Webkit2GTK on Linux and that requires a GTK window.
> While possible to add a GTK widget as a child X11 window inside raw X11 window,
> this is however a bit hacky and **it is not possible on Wayland at all**."

WebKitGTK, WebView2, and Android WebView each draw into their own compositor
surface. A native video surface is either entirely above or entirely below the
webview; it cannot interleave with HTML. Every working example in the ecosystem
is the same hack — a separate child window position-synced to a
`getBoundingClientRect()` div — which breaks on resize, scroll, and any UI drawn
over the video. For JellyTau that means the controls, subtitle overlay, and
mini-player.

The most recent comment on tauri#6343 (2026-05-23) confirms it is still unsolved:

> "I'm faking it and the window is not truly embedded, basically when the parent
> moves or resizes I reset the position and size of the libmpv window to align it
> with an HTML div."

**The principle to carry forward: audio can unify on a native engine; video
cannot, because video needs a surface and the webview owns the surface.**

> **Re-opened on Linux (2026-08-21).** This finding's general form has since been
> falsified on Android — native video now composites behind a transparent Tauri
> WebView and ships on by default (see
> [05-platform-backends.md](../architecture/05-platform-backends.md#native-video-compositing-android)).
> The evidence above is also entirely about *foreign-window embedding*; mpv's
> render API, drawing into a GL context we own inside Tauri's own GTK tree, was
> never tested. [linux-native-video-spike.md](linux-native-video-spike.md) tests
> that one claim on Linux. **Findings 3-6 below are untouched by it** - in
> particular finding 3, which is an independent disqualifier a green spike would
> not clear.

### 3. mpv would regress streaming quality

mpv has **no adaptive bitrate**. It delegates HLS to FFmpeg's demuxer, which
selects one variant at open time and never adapts; mpv#3548 (2016) requested ABR
and it never landed. `--hls-bitrate` is a static picker defaulting to `max`.

The webview path already has real ABR via hls.js. Moving video to mpv would be a
**downgrade** on every platform — no graceful degradation on weak networks, and
quality changes requiring teardown and reload.

> **Premise in doubt (2026-08-21).** "The webview path already has real ABR"
> was not verified against the URLs this app actually builds.
> `get_video_stream_url` requests a *single* rendition (one `VideoBitrate`, one
> `MaxHeight`), the frontend has **no** level-handling code (`hls.levels`,
> `LEVEL_SWITCH`, `currentLevel` appear nowhere), and this repo implements a
> quality switch by *re-opening the stream* — all of which point to a
> single-variant playlist, i.e. no ABR to lose. The decisive test is counting
> `#EXT-X-STREAM-INF` lines in a real `master.m3u8`; it needs a live server and
> has not been run. See
> [linux-native-video-spike.md](linux-native-video-spike.md).

### 4. Crossfade is architecturally blocked on mpv

mpv's audio chain is single-stream. FFmpeg's `acrossfade` is an `N→A` filter
requiring two input streams, so there is no second input to feed it. Real
crossfade needs **two libmpv instances** with manually ramped volumes. Upstream
maintainer response (mpv#4512, closed three minutes after opening):

> "No. I also find crossfading stupid and complex, so the likeliness of that
> happening is low."

GStreamer *could* do it via `audiomixer`. mpv cannot, at any reasonable cost.

### 5. Engine comparison summary

| Criterion | mpv | GStreamer | libVLC |
|-----------|-----|-----------|--------|
| Webview compositing | ❌ Linux broken | ❌ same wall | ❌ same wall |
| Adaptive bitrate HLS | ❌ none | ✅ adaptivedemux2 | ✅ adaptive module |
| Rust bindings | ⚠️ `libmpv2` active; our pin is dead | ✅ `gstreamer-rs` excellent | ❌ `vlc-rs` abandoned (2018) |
| Windows cross-MSVC | ⚠️ prebuilt DLL | ❌ pkg-config vs cargo-xwin | ❌ no better |
| Android packaging | ✅ Maven AAR (used by Findroid) | ⚠️ Cerbero/NDK, painful | ✅ mature AAR |
| ASS/SSA subtitles | ✅ libass built in | ✅ libass | ✅ libass |
| Crossfade | ❌ impossible | ✅ `audiomixer` | ⚠️ unclear |

Every candidate fails the first row, which is the disqualifying one.

### 6. Two further options ruled out

**Webview `<audio>`/`<video>` everywhere** (i.e. delete the native audio backends
too) is dead on Android: `navigator.mediaSession` is *deliberately compiled out*
of Android WebView (Chromium CL 2613133003), so lockscreen/media-notification
control would be impossible. Chromium has also never shipped `audioTracks`. It
remains fine for Windows *video*, which is what we already do.

**FFmpeg-direct / Rust-native** (`ffmpeg-next`, `rsmpeg`, Symphonia) is not
close: the safe bindings do not expose hardware decode at all, `ffmpeg-next` is
self-declared maintenance-only, and Symphonia lacks HE-AAC and gapless AAC. This
is a multi-person-year path to reach parity with what we already have.

### 7. If libmpv is ever revisited on Android

Recorded so the next investigation starts from evidence rather than repeating the
search. The `dev.jdtech.mpv:libmpv` AAR — maintained by Findroid's author, i.e.
another Jellyfin Android client — was inspected directly:

- `libmpv.so` exports the full 54-function `mpv_*` C API with **zero `Java_`
  symbols**; JNI is a separate optional ~19 KB `libplayer.so`. So it is drivable
  from Rust without a Java shim. (This is precisely what disqualifies libVLC,
  whose Android video path hard-requires a Java `AWindow` jobject.)
- ~23 MB/ABI, versus libVLC's ~46 MB/ABI.
- 🔴 **The published AAR is built `--enable-gpl --enable-version3` — it is
  GPLv3**, not LGPL. Fine for us (see [libmpv2-migration.md](libmpv2-migration.md)),
  but it would be a hard constraint for anyone shipping closed source, and an
  LGPL rebuild would be your own build to own.
- Top unverified risk if anyone tries this: whether `libmpv2-sys` can
  cross-compile for `aarch64-linux-android` against that prebuilt `.so`. No
  working example of `libmpv2` on Android was found.

None of this changes the verdict — the cost is the MediaSession/foreground-service
rewrite, not the bindings.

## Decision

1. **Do not unify video onto a native engine.** Video stays in the webview with
   hls.js on all platforms. This is not a compromise — it is the configuration
   that falls out of the compositing constraint, and it is the only one that
   gives us ABR for free.
2. **Android native video is worth a bounded spike anyway** — not for
   unification, but because ExoPlayer's `SurfaceView` path already exists and
   would restore hardware decode plus ASS/SSA subtitles. See
   [05-platform-backends.md](../architecture/05-platform-backends.md#native-video-compositing-android).
3. **Audio parity is the real gap** and is achievable without touching any of the
   above. See [05-platform-backends.md](../architecture/05-platform-backends.md)
   and [windows-native-audio-backend.md](windows-native-audio-backend.md).
4. **Migrate the dead libmpv pin** regardless of any of this. See
   [libmpv2-migration.md](libmpv2-migration.md).

## Corrections to existing docs

These are factual errors found during the investigation. Fixing them is in scope
for this spec.

| Location | Says | Actually |
|----------|------|----------|
| `requirements.md` UR-031 (line ~44) | "Done (Linux only)" | Not implemented on any platform. |
| `requirements.md` DR-034 (line ~196) | "Done (Linux only)" | Not implemented anywhere — `mpv_backend.rs` has a bare `// TODO: Implement crossfade via MPV audio filters if needed`. Architecturally blocked on mpv (finding 4). |
| `requirements.md` parity matrix | Crossfade ✅ Linux / ❌ Android | ❌ / ❌ |
| `requirements.md` parity matrix | (no EQ row) | EQ is also Linux-only — `build_af_filter`/`eq_filter_entries` exist only in `mpv_backend.rs`. Same root cause, same fix. |
| `nativeAdapter.ts:11-14` | Native Android video "blocked upstream by tauri#10152" | tauri#10152 is a stale *feature request*, dead since 2024-07-01. The capability shipped in tauri commit `27d01834` (2024-09-02). Not a blocker. |

## Layer assignment

No new logic. The one boundary observation worth recording:

| Logic / responsibility | Layer | Why it belongs there |
|------------------------|-------|----------------------|
| Which video backend a platform uses (`use_html5_element`) | Rust | Already correctly computed in `get_player_status`. The frontend currently *discards* it — that is the bug, not the design. Restoring it means the frontend consumes a backend decision rather than making its own. |

## Out of scope

- Any code change. This spec is analysis; the sibling specs carry the work.
- iOS/macOS. Not current targets.
- Replacing hls.js.

## Acceptance criteria

- [ ] `requirements.md` DR-034 status corrected; parity matrix updated (crossfade ❌/❌, EQ row added).
- [ ] Stale tauri#10152 comment in `nativeAdapter.ts` corrected.
- [ ] The four sibling specs exist and are linked from here.

## Testing

n/a — documentation only.

## TRACES

No new code. Requirement text changes only; DR-034's status line is the one
substantive edit.

## Notes for the implementer

- The evidence above was gathered in July 2026. The compositing constraint has
  been stable since 2021 (wry#284) and is maintainer-declared unfixable, so it is
  unlikely to change soon — but if someone revisits this, tauri#6343 and wry#284
  are the threads to re-read first.
- A parallel Claude session may be active in this repo — `git diff` before
  "repairing" unexpected changes.