dtourolle
32043a2152
A spec was a promise; sixteen of them had become descriptions of code that already shipped, sitting beside four that describe work still outstanding, with nothing in the file telling the two apart. Half the statuses were also wrong — audio-equalizer read "Accepted" with the EQ live on both platforms, the native video spec said the flag stays off after the default was flipped on. The shipped designs move into docs/architecture, which is the maintained description of the build, and the spec files go. Git history keeps the originals; what a future change still needs is carried across: - 01-rust-backend: favourites rewritten (the old section named a file that no longer exists and called shipped buttons "planned"), domain vocabulary owned by Rust (SearchScope, exclusions, the bitrate ladder), background workers - 02-svelte-frontend: app shell and chrome, library mosaic, series/episode navigation, downloaded browse, safe-area insets, native-video store, logging - 03-data-flow: locally-indexed search - 05-platform-backends: audio settings on ExoPlayer, the equalizer's band vocabulary, native video compositing, the background-audio handoff - 06-downloads-and-offline: one storage model, offline catalog visibility - 09-security: path confinement and input binding docs/specs/README.md now says what the directory is for and where each shipped design went. Deferred work the specs recorded is kept beside the code it concerns rather than lost: season-bounded autoplay, the two dead search commands, why indexing is a full crawl. requirements.md had fourteen stale statuses — Android audio parity still read "Linux only", DR-150 still said the native-video default was off, DR-190 was Proposed after DR-196 implemented it, and five tooling requirements were Proposed after landing. Three unbuilt specs suggested requirement ids that have since been allocated to other work; each now carries a warning.
230 lines
11 KiB
Markdown
230 lines
11 KiB
Markdown
# Spec: Playback backend unification — findings and strategy
|
|
|
|
**Status:** Accepted (analysis; no code changes)
|
|
**Requirements:** IR-004, UR-031, UR-032, UR-033 — revises the "Platform Playback Backend Parity" issue in requirements.md
|
|
**UX spec:** n/a
|
|
**Supersedes / revises:** informed the Android native-video and audio-parity
|
|
work (both since shipped — see
|
|
[05-platform-backends.md](../architecture/05-platform-backends.md)) and
|
|
[windows-native-audio-backend.md](windows-native-audio-backend.md), still open
|
|
|
|
## Summary
|
|
|
|
This spec records the outcome of an investigation into unifying JellyTau's
|
|
playback backends (Linux/MPV, Android/ExoPlayer, Windows/webview) onto a single
|
|
engine with hardware acceleration everywhere. **The conclusion is that video
|
|
cannot be unified onto a native engine, and should not be attempted.** Audio
|
|
*can* be, and that is where the remaining specs direct effort.
|
|
|
|
No code changes follow from this spec directly. It exists so the decision is
|
|
written down with its evidence, and so a future session does not re-run the same
|
|
investigation.
|
|
|
|
## Motivation
|
|
|
|
The requirements doc carries a "Platform Playback Backend Parity" issue noting
|
|
that audio settings work on Linux but not Android, and proposing eventual
|
|
convergence. The natural next question — "should we just run one engine
|
|
everywhere?" — needed answering before spending effort on per-backend patches.
|
|
|
|
The investigation also surfaced that several statements in requirements.md and in
|
|
code comments are factually wrong. Those corrections are part of the deliverable.
|
|
|
|
## Findings
|
|
|
|
### 1. The current architecture is not what the docs describe
|
|
|
|
| Platform | Audio | Video |
|
|
|----------|-------|-------|
|
|
| Linux | MPV (native, **audio-only**) | webview `<video>` + hls.js |
|
|
| Android | ExoPlayer (native) | **webview `<video>` + hls.js** |
|
|
| Windows | webview `<audio>` | webview `<video>` + hls.js |
|
|
|
|
Two surprises:
|
|
|
|
- **MPV never decodes video.** `mpv_backend.rs` sets `video = no` and
|
|
`audio-display = no` at construction. Linux video has always been the webview.
|
|
Correspondingly, `player_play_item` deliberately does *not* load into MPV on
|
|
Linux (it calls `set_current_item`, which only updates the queue).
|
|
- **Android video is also the webview.** `createAdapter()` in
|
|
`src/lib/player/adapters/index.ts` hardcodes `const effectiveKind = "html5"`
|
|
and does `void backendKind`, discarding the `use_html5_element` signal that
|
|
`get_player_status` computes in Rust. `NativePlayerAdapter` is dead code, and
|
|
ExoPlayer's `SurfaceView` path in `JellyTauPlayer.kt` is unreachable.
|
|
|
|
So video is *already* unified — on HTML5, everywhere, by accident of that
|
|
hardcode — and on the path without hardware decoding on Android.
|
|
|
|
### 2. Native video cannot be composited with a Tauri webview
|
|
|
|
This is the load-bearing finding. It is **not** an mpv limitation; it defeats
|
|
every candidate engine identically:
|
|
|
|
- **mpv**: `tauri-plugin-libmpv`'s own platform table reads Linux ⚠️
|
|
*"Experimental. Window embedding is not working."*
|
|
- **GStreamer** (wry discussion #284, 2024): *"Gstreamer was rendering above the
|
|
surface and covering all html elements."*
|
|
- **libVLC** (tauri discussion #6343, 2024): *"I had to render the webview in a
|
|
child window though because vlc kept rendering on top of it."*
|
|
|
|
Root cause, from Tauri maintainer amrbashir (tauri#9220, 2024-03-30):
|
|
|
|
> "we are limited to using Webkit2GTK on Linux and that requires a GTK window.
|
|
> While possible to add a GTK widget as a child X11 window inside raw X11 window,
|
|
> this is however a bit hacky and **it is not possible on Wayland at all**."
|
|
|
|
WebKitGTK, WebView2, and Android WebView each draw into their own compositor
|
|
surface. A native video surface is either entirely above or entirely below the
|
|
webview; it cannot interleave with HTML. Every working example in the ecosystem
|
|
is the same hack — a separate child window position-synced to a
|
|
`getBoundingClientRect()` div — which breaks on resize, scroll, and any UI drawn
|
|
over the video. For JellyTau that means the controls, subtitle overlay, and
|
|
mini-player.
|
|
|
|
The most recent comment on tauri#6343 (2026-05-23) confirms it is still unsolved:
|
|
|
|
> "I'm faking it and the window is not truly embedded, basically when the parent
|
|
> moves or resizes I reset the position and size of the libmpv window to align it
|
|
> with an HTML div."
|
|
|
|
**The principle to carry forward: audio can unify on a native engine; video
|
|
cannot, because video needs a surface and the webview owns the surface.**
|
|
|
|
### 3. mpv would regress streaming quality
|
|
|
|
mpv has **no adaptive bitrate**. It delegates HLS to FFmpeg's demuxer, which
|
|
selects one variant at open time and never adapts; mpv#3548 (2016) requested ABR
|
|
and it never landed. `--hls-bitrate` is a static picker defaulting to `max`.
|
|
|
|
The webview path already has real ABR via hls.js. Moving video to mpv would be a
|
|
**downgrade** on every platform — no graceful degradation on weak networks, and
|
|
quality changes requiring teardown and reload.
|
|
|
|
### 4. Crossfade is architecturally blocked on mpv
|
|
|
|
mpv's audio chain is single-stream. FFmpeg's `acrossfade` is an `N→A` filter
|
|
requiring two input streams, so there is no second input to feed it. Real
|
|
crossfade needs **two libmpv instances** with manually ramped volumes. Upstream
|
|
maintainer response (mpv#4512, closed three minutes after opening):
|
|
|
|
> "No. I also find crossfading stupid and complex, so the likeliness of that
|
|
> happening is low."
|
|
|
|
GStreamer *could* do it via `audiomixer`. mpv cannot, at any reasonable cost.
|
|
|
|
### 5. Engine comparison summary
|
|
|
|
| Criterion | mpv | GStreamer | libVLC |
|
|
|-----------|-----|-----------|--------|
|
|
| Webview compositing | ❌ Linux broken | ❌ same wall | ❌ same wall |
|
|
| Adaptive bitrate HLS | ❌ none | ✅ adaptivedemux2 | ✅ adaptive module |
|
|
| Rust bindings | ⚠️ `libmpv2` active; our pin is dead | ✅ `gstreamer-rs` excellent | ❌ `vlc-rs` abandoned (2018) |
|
|
| Windows cross-MSVC | ⚠️ prebuilt DLL | ❌ pkg-config vs cargo-xwin | ❌ no better |
|
|
| Android packaging | ✅ Maven AAR (used by Findroid) | ⚠️ Cerbero/NDK, painful | ✅ mature AAR |
|
|
| ASS/SSA subtitles | ✅ libass built in | ✅ libass | ✅ libass |
|
|
| Crossfade | ❌ impossible | ✅ `audiomixer` | ⚠️ unclear |
|
|
|
|
Every candidate fails the first row, which is the disqualifying one.
|
|
|
|
### 6. Two further options ruled out
|
|
|
|
**Webview `<audio>`/`<video>` everywhere** (i.e. delete the native audio backends
|
|
too) is dead on Android: `navigator.mediaSession` is *deliberately compiled out*
|
|
of Android WebView (Chromium CL 2613133003), so lockscreen/media-notification
|
|
control would be impossible. Chromium has also never shipped `audioTracks`. It
|
|
remains fine for Windows *video*, which is what we already do.
|
|
|
|
**FFmpeg-direct / Rust-native** (`ffmpeg-next`, `rsmpeg`, Symphonia) is not
|
|
close: the safe bindings do not expose hardware decode at all, `ffmpeg-next` is
|
|
self-declared maintenance-only, and Symphonia lacks HE-AAC and gapless AAC. This
|
|
is a multi-person-year path to reach parity with what we already have.
|
|
|
|
### 7. If libmpv is ever revisited on Android
|
|
|
|
Recorded so the next investigation starts from evidence rather than repeating the
|
|
search. The `dev.jdtech.mpv:libmpv` AAR — maintained by Findroid's author, i.e.
|
|
another Jellyfin Android client — was inspected directly:
|
|
|
|
- `libmpv.so` exports the full 54-function `mpv_*` C API with **zero `Java_`
|
|
symbols**; JNI is a separate optional ~19 KB `libplayer.so`. So it is drivable
|
|
from Rust without a Java shim. (This is precisely what disqualifies libVLC,
|
|
whose Android video path hard-requires a Java `AWindow` jobject.)
|
|
- ~23 MB/ABI, versus libVLC's ~46 MB/ABI.
|
|
- 🔴 **The published AAR is built `--enable-gpl --enable-version3` — it is
|
|
GPLv3**, not LGPL. Fine for us (see [libmpv2-migration.md](libmpv2-migration.md)),
|
|
but it would be a hard constraint for anyone shipping closed source, and an
|
|
LGPL rebuild would be your own build to own.
|
|
- Top unverified risk if anyone tries this: whether `libmpv2-sys` can
|
|
cross-compile for `aarch64-linux-android` against that prebuilt `.so`. No
|
|
working example of `libmpv2` on Android was found.
|
|
|
|
None of this changes the verdict — the cost is the MediaSession/foreground-service
|
|
rewrite, not the bindings.
|
|
|
|
## Decision
|
|
|
|
1. **Do not unify video onto a native engine.** Video stays in the webview with
|
|
hls.js on all platforms. This is not a compromise — it is the configuration
|
|
that falls out of the compositing constraint, and it is the only one that
|
|
gives us ABR for free.
|
|
2. **Android native video is worth a bounded spike anyway** — not for
|
|
unification, but because ExoPlayer's `SurfaceView` path already exists and
|
|
would restore hardware decode plus ASS/SSA subtitles. See
|
|
[05-platform-backends.md](../architecture/05-platform-backends.md#native-video-compositing-android).
|
|
3. **Audio parity is the real gap** and is achievable without touching any of the
|
|
above. See [05-platform-backends.md](../architecture/05-platform-backends.md)
|
|
and [windows-native-audio-backend.md](windows-native-audio-backend.md).
|
|
4. **Migrate the dead libmpv pin** regardless of any of this. See
|
|
[libmpv2-migration.md](libmpv2-migration.md).
|
|
|
|
## Corrections to existing docs
|
|
|
|
These are factual errors found during the investigation. Fixing them is in scope
|
|
for this spec.
|
|
|
|
| Location | Says | Actually |
|
|
|----------|------|----------|
|
|
| `requirements.md` UR-031 (line ~44) | "Done (Linux only)" | Not implemented on any platform. |
|
|
| `requirements.md` DR-034 (line ~196) | "Done (Linux only)" | Not implemented anywhere — `mpv_backend.rs` has a bare `// TODO: Implement crossfade via MPV audio filters if needed`. Architecturally blocked on mpv (finding 4). |
|
|
| `requirements.md` parity matrix | Crossfade ✅ Linux / ❌ Android | ❌ / ❌ |
|
|
| `requirements.md` parity matrix | (no EQ row) | EQ is also Linux-only — `build_af_filter`/`eq_filter_entries` exist only in `mpv_backend.rs`. Same root cause, same fix. |
|
|
| `nativeAdapter.ts:11-14` | Native Android video "blocked upstream by tauri#10152" | tauri#10152 is a stale *feature request*, dead since 2024-07-01. The capability shipped in tauri commit `27d01834` (2024-09-02). Not a blocker. |
|
|
|
|
## Layer assignment
|
|
|
|
No new logic. The one boundary observation worth recording:
|
|
|
|
| Logic / responsibility | Layer | Why it belongs there |
|
|
|------------------------|-------|----------------------|
|
|
| Which video backend a platform uses (`use_html5_element`) | Rust | Already correctly computed in `get_player_status`. The frontend currently *discards* it — that is the bug, not the design. Restoring it means the frontend consumes a backend decision rather than making its own. |
|
|
|
|
## Out of scope
|
|
|
|
- Any code change. This spec is analysis; the sibling specs carry the work.
|
|
- iOS/macOS. Not current targets.
|
|
- Replacing hls.js.
|
|
|
|
## Acceptance criteria
|
|
|
|
- [ ] `requirements.md` DR-034 status corrected; parity matrix updated (crossfade ❌/❌, EQ row added).
|
|
- [ ] Stale tauri#10152 comment in `nativeAdapter.ts` corrected.
|
|
- [ ] The four sibling specs exist and are linked from here.
|
|
|
|
## Testing
|
|
|
|
n/a — documentation only.
|
|
|
|
## TRACES
|
|
|
|
No new code. Requirement text changes only; DR-034's status line is the one
|
|
substantive edit.
|
|
|
|
## Notes for the implementer
|
|
|
|
- The evidence above was gathered in July 2026. The compositing constraint has
|
|
been stable since 2021 (wry#284) and is maintainer-declared unfixable, so it is
|
|
unlikely to change soon — but if someone revisits this, tauri#6343 and wry#284
|
|
are the threads to re-read first.
|
|
- A parallel Claude session may be active in this repo — `git diff` before
|
|
"repairing" unexpected changes.
|