docs(player): backend unification findings + correct false parity claims
Investigation into unifying the playback backends (Linux/MPV, Android/ExoPlayer, Windows/webview) onto one engine with hardware acceleration. Conclusion: video cannot be unified onto a native engine; audio can. The blocker is not mpv-specific. WebKitGTK, WebView2 and Android WebView each draw into their own compositor surface, so a native video surface sits either entirely above or entirely below the webview and cannot interleave with HTML. GStreamer and libVLC fail identically. mpv would additionally regress streaming: it has no adaptive bitrate, while the current hls.js path does. Six specs added: - playback-backend-unification: the analysis and decision, with evidence - android-audio-settings-parity: set_audio_settings on ExoPlayerBackend - android-native-video-spike: timeboxed test of SurfaceView compositing - windows-native-audio-backend: replace the webview <audio> shim with libmpv - libmpv2-migration: dead libmpv git pin -> libmpv2, plus a LICENSE file - playback-docs-corrections: the requirement-status fixes applied here Corrections to requirements.md, all verified against source: - UR-031/DR-034 claimed crossfade was "Done (Linux only)". It is implemented nowhere (mpv_backend.rs has a bare TODO) and is architecturally blocked on mpv, whose single-stream audio chain cannot feed acrossfade's two inputs. - Parity matrix listed crossfade as a Linux/Android gap; it is neither. - The matrix omitted the equalizer, which has the same Linux-only shape. - The suggested ConcatenatingMediaSource is deprecated in current Media3. nativeAdapter.ts cited tauri#10152 as an upstream blocker for native Android video. That issue is a stale feature request, dead since 2024-07-01; the capability shipped in tauri 27d01834 (2024-09-02), and the related black-screen bug was fixed in wry 0.39.4 (we ship 0.55.x). What is genuinely unproven is SurfaceView-behind-WebView compositing, which the spike now tracks.
This commit is contained in:
@@ -0,0 +1,209 @@
|
||||
# Spec: Windows native audio backend
|
||||
|
||||
**Status:** Proposed
|
||||
**Requirements:** UR-003, UR-027, UR-032, UR-033 → DR-030, DR-035, DR-036; new IR-030
|
||||
**UX spec:** n/a — Settings › Audio already renders the controls
|
||||
**Supersedes / revises:** acts on the "audio can unify, video cannot" conclusion in [playback-backend-unification.md](playback-backend-unification.md)
|
||||
|
||||
## Summary
|
||||
|
||||
Give Windows a real native audio backend instead of the current webview
|
||||
`<audio>` shim. Windows is the only platform where audio playback has no decoder
|
||||
of its own: `WebviewAudioBackend` hands a URL to a frontend `<audio>` element and
|
||||
relays transport commands. It cannot set volume, cannot apply any audio setting,
|
||||
and reports state only via DOM events.
|
||||
|
||||
Audio needs no rendering surface, so **none of the webview-compositing problems
|
||||
that block unified video apply here.** This is the cleanest available win.
|
||||
|
||||
## Motivation
|
||||
|
||||
`WebviewAudioBackend` was a deliberate stopgap ("audio-only playback for
|
||||
platforms without a native audio backend"), and it works — but it has a hard
|
||||
functional gap. From `webview_audio_backend.rs`:
|
||||
|
||||
```rust
|
||||
fn set_volume(&mut self, volume: f32) -> Result<(), PlayerError> {
|
||||
// ...stores locally only; there is no ControlCommand action for volume
|
||||
}
|
||||
```
|
||||
|
||||
So volume changes never reach the element; the frontend has to observe the player
|
||||
store and apply volume itself. `set_audio_settings` likewise stores values that
|
||||
nothing consumes — EQ, normalization, and gapless are all inert on Windows.
|
||||
|
||||
Meanwhile the backend-unification investigation established that a native *audio*
|
||||
engine is unproblematic on Windows specifically: `tauri-plugin-libmpv` lists
|
||||
Windows as its **fully tested** platform (in contrast to Linux, where embedding
|
||||
is broken — but that is a *video surface* problem, which audio does not have).
|
||||
|
||||
## Layer assignment
|
||||
|
||||
| Logic / responsibility | Layer | Why it belongs there |
|
||||
|------------------------|-------|----------------------|
|
||||
| Decoding and playing the audio stream | Rust | Playback is domain logic; every other platform already decodes in Rust or a native player. The webview shim is the anomaly. |
|
||||
| Applying `AudioSettings` (EQ/normalize/gapless) | Rust | Same `AudioSettings` contract as MPV/ExoPlayer; band layout and presets stay canonical in `settings.rs`. |
|
||||
| Position/state reporting | Rust | Restores the project's core principle — the player is the authoritative source of state. Today Windows inverts this: the DOM element is authoritative and Rust mirrors it. |
|
||||
| Volume | Rust | Currently broken precisely because it is split across the boundary. |
|
||||
| Rendering the player UI | Frontend | Unchanged. |
|
||||
|
||||
The strongest argument for this change is the third row. CLAUDE.md states
|
||||
playback state is one-directional with the player authoritative; on Windows that
|
||||
is currently false, and the `player_report_*` round-trip exists to paper over it.
|
||||
|
||||
## Design
|
||||
|
||||
### Engine choice
|
||||
|
||||
Two viable options; **libmpv is recommended** for consistency with the Linux
|
||||
audio backend.
|
||||
|
||||
| | libmpv | GStreamer |
|
||||
|---|---|---|
|
||||
| Windows status | ✅ `tauri-plugin-libmpv` reports fully tested | ✅ works, but… |
|
||||
| Rust bindings | `libmpv2` 6.0.0, active | `gstreamer-rs` 0.25.x, excellent |
|
||||
| Cross-MSVC from Linux | ⚠️ needs prebuilt DLL + import lib | ❌ `gstreamer-sys` uses pkg-config, fights `cargo-xwin` |
|
||||
| Code reuse | ✅ `MpvBackend` logic is directly reusable | ❌ a second engine to learn |
|
||||
| Crossfade capable | ❌ single-stream chain | ✅ `audiomixer` |
|
||||
|
||||
libmpv wins on reuse: `MpvBackend`'s `set_audio_settings` — the `af` lavfi graph
|
||||
built by `build_af_filter`, `eq_filter_entries`, `normalize_filter_entry` — is
|
||||
platform-independent and would apply unchanged.
|
||||
|
||||
The one reason to prefer GStreamer is crossfade (UR-031), which mpv structurally
|
||||
cannot do. If crossfade becomes a priority, revisit; it would then argue for
|
||||
GStreamer on *both* Linux and Windows, which is a much larger change.
|
||||
|
||||
### Structure
|
||||
|
||||
Rename the cfg gate so `MpvBackend` is no longer Linux-only:
|
||||
|
||||
```rust
|
||||
// src-tauri/src/player/mod.rs
|
||||
#[cfg(any(target_os = "linux", target_os = "windows"))]
|
||||
pub mod mpv_backend;
|
||||
```
|
||||
|
||||
`MpvBackend::new` needs one platform-specific branch: `detect_audio_system()`
|
||||
currently probes `pactl`/`pw-cli`/`/proc/asound/cards` to pick an `ao`. On
|
||||
Windows the equivalent is `wasapi` (mpv's default), so the detection is a
|
||||
`#[cfg]` returning `"wasapi"` — no probing needed.
|
||||
|
||||
Everything else — the event loop, the 250ms position thread, the seek-suppression
|
||||
window, the `af` filter graph — is unchanged.
|
||||
|
||||
`WebviewAudioBackend` stays for other targets (macOS and anything else hitting
|
||||
the `not(any(...))` arm) and as the fallback if libmpv fails to initialize. The
|
||||
existing `emit_backend_init_failed` path already handles that gracefully.
|
||||
|
||||
### Build
|
||||
|
||||
`libmpv2-sys` is well-suited to cross-compilation: no pkg-config, vendored
|
||||
headers, pregenerated bindings (no libclang). It emits `cargo:rustc-link-lib=mpv`
|
||||
unconditionally, so the build must supply a linkable import library for
|
||||
`x86_64-pc-windows-msvc`.
|
||||
|
||||
Keep the `build_libmpv` feature **off** — its Unix path shells out to mpv-build
|
||||
and explicitly rejects cross-compilation.
|
||||
|
||||
🔴 Per CLAUDE.md, the prebuilt libmpv **must be added to the builder image**
|
||||
(`Dockerfile.builder` → rebuild + push via `scripts/build-builder-image.sh`), not
|
||||
installed at CI job time. `libmpv-2.dll` must also be bundled into the NSIS
|
||||
installer via `tauri.conf.json`'s resources.
|
||||
|
||||
### Verified build mechanics
|
||||
|
||||
The cross-compile path was tested hands-on from Linux (July 2026), not inferred:
|
||||
|
||||
- Neither shinchiro nor zhongfly ships an `mpv.def` or MSVC `mpv.lib` — only a
|
||||
MinGW `libmpv.dll.a`. (Several online sources claim otherwise; they are wrong.)
|
||||
- An MSVC-style import lib can be generated locally with LLVM tools only:
|
||||
`llvm-readobj --coff-exports libmpv-2.dll` → synthesize `mpv.def` →
|
||||
`llvm-dlltool -m i386:x86-64 -d mpv.def -l mpv.lib`. `llvm-lib /def:` produces a
|
||||
byte-identical result.
|
||||
- A real `lld-link` link against that import lib **succeeds**, and the resulting
|
||||
import table resolves `mpv_client_api_version` from `libmpv-2.dll`. `lld-link`
|
||||
is the linker `cargo-xwin` uses, so this is the load-bearing step.
|
||||
- Linking directly against the shipped MinGW `libmpv.dll.a` **also** succeeds, so
|
||||
def-generation may be skippable — but that relies on lld's GNU-archive
|
||||
tolerance rather than a documented contract. Keep `llvm-dlltool` as the
|
||||
fallback.
|
||||
- MinGW origin is not an ABI problem: libmpv exports a pure C ABI, and the x86-64
|
||||
Windows calling convention is platform-defined. The upstream note that MSVC
|
||||
cannot *build* mpv is frequently misread as "MSVC cannot *link* libmpv" — that
|
||||
is not what it says.
|
||||
- 🔴 Never free/realloc across the DLL boundary — use `mpv_free`.
|
||||
|
||||
Build wiring is ordinary: `cargo:rustc-link-lib=dylib=mpv` plus
|
||||
`cargo:rustc-link-search`. Nothing about libmpv conflicts with `cargo-xwin`.
|
||||
|
||||
### Size and shipping
|
||||
|
||||
Measured uncompressed: **93 MiB** (zhongfly `mpv-dev-lgpl-x86_64`) vs **112 MiB**
|
||||
(shinchiro, full GPL build); ~26–30 MB compressed in the `.7z`.
|
||||
|
||||
**Ship the zhongfly LGPL build** — smaller, and there is no reason to pull the
|
||||
GPL variant in for an audio-only use.
|
||||
|
||||
Import-table inspection confirms **no companion DLLs are needed**: every
|
||||
dependency is a system DLL (`KERNEL32`, `USER32`, `d2d1`, `DWrite`, `OPENGL32`,
|
||||
`vulkan-1`, UCRT `api-ms-win-*`). One file to bundle.
|
||||
|
||||
93 MiB is still substantial against a Tauri app's usual few MB. Since we use mpv
|
||||
audio-only, investigate whether a pruned build (no video decoders, no libplacebo)
|
||||
is worth producing for the builder image — but treat that as an optimization,
|
||||
not a blocker.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- Windows *video*. Stays in WebView2 + hls.js — it works and has ABR.
|
||||
- Crossfade (UR-031/DR-034) — not implemented anywhere; needs its own spec.
|
||||
- Replacing `WebviewAudioBackend` for macOS.
|
||||
- MPRIS/SMTC media-key integration — worth a follow-up, not this spec.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- [ ] Windows build produces a `MpvBackend`-backed player; `backend-init-failed` is emitted (not a crash) if libmpv is unavailable.
|
||||
- [ ] Volume control works from the UI — the current hard gap.
|
||||
- [ ] EQ, normalization, and gapless audibly take effect on Windows.
|
||||
- [ ] Position/state originate in Rust; the `<audio>` element is no longer in the audio path.
|
||||
- [ ] Seek, next/previous, and queue advance work; sleep timer stops playback.
|
||||
- [ ] `libmpv-2.dll` ships in the NSIS installer and the app runs on a clean Windows VM with no mpv installed.
|
||||
- [ ] Builder image carries the Windows libmpv artefacts; **no toolchain install added to any CI step**.
|
||||
- [ ] `bun run check`, `bun run test`, `bun run check:boundary` pass.
|
||||
- [ ] `cargo fmt` clean, `cargo clippy` clean, `bun run test:rust` passes.
|
||||
- [ ] New requirement-implementing code carries `// TRACES:` comments.
|
||||
|
||||
## Testing
|
||||
|
||||
**Rust**: the existing `mpv_backend_test.rs` and the `build_af_filter` /
|
||||
`normalize_filter_entry` / `eq_filter_entries` unit tests already cover the
|
||||
filter-graph logic and are platform-independent — they should pass unchanged
|
||||
under a Windows `cargo check`/test. Add a test asserting `detect_audio_system()`
|
||||
returns `wasapi` under `cfg(windows)`.
|
||||
|
||||
**Manual, on Windows**: volume, EQ preset change, normalization toggle, gapless
|
||||
between two tracks, seek, queue advance, sleep timer. Then the packaging test —
|
||||
install the NSIS output on a clean VM and confirm it launches and plays.
|
||||
|
||||
Per CLAUDE.md, the volume gap is a *bug fix*: write a failing test for
|
||||
"`set_volume` reaches the backend" before implementing.
|
||||
|
||||
## TRACES
|
||||
|
||||
- Windows `MpvBackend` construction in `create_player_backend` → `// TRACES: UR-003 | IR-030`
|
||||
- `detect_audio_system` Windows branch → `IR-030`
|
||||
- Existing `set_audio_settings` gains Windows coverage → `UR-027, UR-032, UR-033 | DR-030, DR-035, DR-036`
|
||||
- Allocate **IR-030** in `requirements.md` ("libmpv integration for Windows audio playback").
|
||||
|
||||
## Notes for the implementer
|
||||
|
||||
- Do this **after** [libmpv2-migration.md](libmpv2-migration.md) — porting the
|
||||
current dead `libmpv` git pin to a second platform would double the migration
|
||||
work.
|
||||
- `libmpv2` has broken its API in every major release (4.0 removed command
|
||||
helpers, 5.0 removed `mpv_node`, 6.0 changed `RenderContext` ownership). Pin an
|
||||
exact version.
|
||||
- Only the `render`-feature parts of `libmpv2` concern video; audio-only use does
|
||||
not need it, and disabling the default `render` feature may shrink the build.
|
||||
- A parallel Claude session may be active — `git diff` first.
|
||||
Reference in New Issue
Block a user