jellytau/docs/specs/windows-native-audio-backend.md

# Spec: Windows native audio backend

**Status:** Partially implemented — code and packaging done, **unverified on
Windows hardware**. `MpvBackend` is the Windows audio backend (`ao=wasapi`);
the builder image carries zhongfly's pinned LGPL `libmpv-2.dll` plus a generated
MSVC `mpv.lib`, and the NSIS installer ships the DLL beside `jellytau.exe` with
its licence texts ([build-windows.md](../build/build-windows.md)). Done *before*
the libmpv2 migration after all: the current `libmpv` pin cross-links fine, and
the migration now has one call site to keep safe (`mpv_command`, DR-298) rather
than a crate API to port twice. **Left:** every acceptance criterion that needs a
Windows machine — audible volume/EQ/normalization/gapless, seek/queue/sleep
timer, and the clean-VM install test.
**Requirements:** UR-003, UR-027, UR-032, UR-033 → DR-030, DR-035, DR-036;
⚠️ the suggested id **IR-030 has since been allocated** to the scheduled catalog
crawl — allocate a fresh id (IR-033 or later) on implementation
**UX spec:** n/a — Settings › Audio already renders the controls
**Supersedes / revises:** acts on the "audio can unify, video cannot" conclusion in [playback-backend-unification.md](playback-backend-unification.md)

## Summary

Give Windows a real native audio backend instead of the current webview
`<audio>` shim. Windows is the only platform where audio playback has no decoder
of its own: `WebviewAudioBackend` hands a URL to a frontend `<audio>` element and
relays transport commands. It cannot set volume, cannot apply any audio setting,
and reports state only via DOM events.

Audio needs no rendering surface, so **none of the webview-compositing problems
that block unified video apply here.** This is the cleanest available win.

## Motivation

`WebviewAudioBackend` was a deliberate stopgap ("audio-only playback for
platforms without a native audio backend"), and it works — but it has a hard
functional gap. From `webview_audio_backend.rs`:

```rust
fn set_volume(&mut self, volume: f32) -> Result<(), PlayerError> {
    // ...stores locally only; there is no ControlCommand action for volume
}
```

So volume changes never reach the element; the frontend has to observe the player
store and apply volume itself. `set_audio_settings` likewise stores values that
nothing consumes — EQ, normalization, and gapless are all inert on Windows.

Meanwhile the backend-unification investigation established that a native *audio*
engine is unproblematic on Windows specifically: `tauri-plugin-libmpv` lists
Windows as its **fully tested** platform (in contrast to Linux, where embedding
is broken — but that is a *video surface* problem, which audio does not have).

## Layer assignment

| Logic / responsibility | Layer | Why it belongs there |
|------------------------|-------|----------------------|
| Decoding and playing the audio stream | Rust | Playback is domain logic; every other platform already decodes in Rust or a native player. The webview shim is the anomaly. |
| Applying `AudioSettings` (EQ/normalize/gapless) | Rust | Same `AudioSettings` contract as MPV/ExoPlayer; band layout and presets stay canonical in `settings.rs`. |
| Position/state reporting | Rust | Restores the project's core principle — the player is the authoritative source of state. Today Windows inverts this: the DOM element is authoritative and Rust mirrors it. |
| Volume | Rust | Currently broken precisely because it is split across the boundary. |
| Rendering the player UI | Frontend | Unchanged. |

The strongest argument for this change is the third row. CLAUDE.md states
playback state is one-directional with the player authoritative; on Windows that
is currently false, and the `player_report_*` round-trip exists to paper over it.

## Design

### Engine choice

Two viable options; **libmpv is recommended** for consistency with the Linux
audio backend.

| | libmpv | GStreamer |
|---|---|---|
| Windows status | ✅ `tauri-plugin-libmpv` reports fully tested | ✅ works, but… |
| Rust bindings | `libmpv2` 6.0.0, active | `gstreamer-rs` 0.25.x, excellent |
| Cross-MSVC from Linux | ⚠️ needs prebuilt DLL + import lib | ❌ `gstreamer-sys` uses pkg-config, fights `cargo-xwin` |
| Code reuse | ✅ `MpvBackend` logic is directly reusable | ❌ a second engine to learn |
| Crossfade capable | ❌ single-stream chain | ✅ `audiomixer` |

libmpv wins on reuse: `MpvBackend`'s `set_audio_settings` — the `af` lavfi graph
built by `build_af_filter`, `eq_filter_entries`, `normalize_filter_entry` — is
platform-independent and would apply unchanged.

The one reason to prefer GStreamer is crossfade (UR-031), which mpv structurally
cannot do. If crossfade becomes a priority, revisit; it would then argue for
GStreamer on *both* Linux and Windows, which is a much larger change.

### Structure

Rename the cfg gate so `MpvBackend` is no longer Linux-only:

```rust
// src-tauri/src/player/mod.rs
#[cfg(any(target_os = "linux", target_os = "windows"))]
pub mod mpv_backend;
```

`MpvBackend::new` needs one platform-specific branch: `detect_audio_system()`
currently probes `pactl`/`pw-cli`/`/proc/asound/cards` to pick an `ao`. On
Windows the equivalent is `wasapi` (mpv's default), so the detection is a
`#[cfg]` returning `"wasapi"` — no probing needed.

Everything else — the event loop, the 250ms position thread, the seek-suppression
window, the `af` filter graph — is unchanged.

`WebviewAudioBackend` stays for other targets (macOS and anything else hitting
the `not(any(...))` arm) and as the fallback if libmpv fails to initialize. The
existing `emit_backend_init_failed` path already handles that gracefully.

### Build

`libmpv2-sys` is well-suited to cross-compilation: no pkg-config, vendored
headers, pregenerated bindings (no libclang). It emits `cargo:rustc-link-lib=mpv`
unconditionally, so the build must supply a linkable import library for
`x86_64-pc-windows-msvc`.

Keep the `build_libmpv` feature **off** — its Unix path shells out to mpv-build
and explicitly rejects cross-compilation.

🔴 Per CLAUDE.md, the prebuilt libmpv **must be added to the builder image**
(`Dockerfile.builder` → rebuild + push via `scripts/build-builder-image.sh`), not
installed at CI job time. `libmpv-2.dll` must also be bundled into the NSIS
installer via `tauri.conf.json`'s resources.

### Verified build mechanics

The cross-compile path was tested hands-on from Linux (July 2026), not inferred:

- Neither shinchiro nor zhongfly ships an `mpv.def` or MSVC `mpv.lib` — only a
  MinGW `libmpv.dll.a`. (Several online sources claim otherwise; they are wrong.)
- An MSVC-style import lib can be generated locally with LLVM tools only:
  `llvm-readobj --coff-exports libmpv-2.dll` → synthesize `mpv.def` →
  `llvm-dlltool -m i386:x86-64 -d mpv.def -l mpv.lib`. `llvm-lib /def:` produces a
  byte-identical result.
- A real `lld-link` link against that import lib **succeeds**, and the resulting
  import table resolves `mpv_client_api_version` from `libmpv-2.dll`. `lld-link`
  is the linker `cargo-xwin` uses, so this is the load-bearing step.
- Linking directly against the shipped MinGW `libmpv.dll.a` **also** succeeds, so
  def-generation may be skippable — but that relies on lld's GNU-archive
  tolerance rather than a documented contract. Keep `llvm-dlltool` as the
  fallback.
- MinGW origin is not an ABI problem: libmpv exports a pure C ABI, and the x86-64
  Windows calling convention is platform-defined. The upstream note that MSVC
  cannot *build* mpv is frequently misread as "MSVC cannot *link* libmpv" — that
  is not what it says.
- 🔴 Never free/realloc across the DLL boundary — use `mpv_free`.

Build wiring is ordinary: `cargo:rustc-link-lib=dylib=mpv` plus
`cargo:rustc-link-search`. Nothing about libmpv conflicts with `cargo-xwin`.

### Size and shipping

Measured uncompressed: **93 MiB** (zhongfly `mpv-dev-lgpl-x86_64`) vs **112 MiB**
(shinchiro, full GPL build); ~26–30 MB compressed in the `.7z`.

**Ship the zhongfly LGPL build** — smaller, and there is no reason to pull the
GPL variant in for an audio-only use.

Import-table inspection confirms **no companion DLLs are needed**: every
dependency is a system DLL (`KERNEL32`, `USER32`, `d2d1`, `DWrite`, `OPENGL32`,
`vulkan-1`, UCRT `api-ms-win-*`). One file to bundle.

93 MiB is still substantial against a Tauri app's usual few MB. Since we use mpv
audio-only, investigate whether a pruned build (no video decoders, no libplacebo)
is worth producing for the builder image — but treat that as an optimization,
not a blocker.

## Out of scope

- Windows *video*. Stays in WebView2 + hls.js — it works and has ABR.
- Crossfade (UR-031/DR-034) — not implemented anywhere; needs its own spec.
- Replacing `WebviewAudioBackend` for macOS.
- MPRIS/SMTC media-key integration — worth a follow-up, not this spec.

## Acceptance criteria

- [ ] Windows build produces a `MpvBackend`-backed player; `backend-init-failed` is emitted (not a crash) if libmpv is unavailable.
- [ ] Volume control works from the UI — the current hard gap.
- [ ] EQ, normalization, and gapless audibly take effect on Windows.
- [ ] Position/state originate in Rust; the `<audio>` element is no longer in the audio path.
- [ ] Seek, next/previous, and queue advance work; sleep timer stops playback.
- [ ] `libmpv-2.dll` ships in the NSIS installer and the app runs on a clean Windows VM with no mpv installed.
- [ ] Builder image carries the Windows libmpv artefacts; **no toolchain install added to any CI step**.
- [ ] `bun run check`, `bun run test`, `bun run check:boundary` pass.
- [ ] `cargo fmt` clean, `cargo clippy` clean, `bun run test:rust` passes.
- [ ] New requirement-implementing code carries `// TRACES:` comments.

## Testing

**Rust**: the existing `mpv_backend_test.rs` and the `build_af_filter` /
`normalize_filter_entry` / `eq_filter_entries` unit tests already cover the
filter-graph logic and are platform-independent — they should pass unchanged
under a Windows `cargo check`/test. Add a test asserting `detect_audio_system()`
returns `wasapi` under `cfg(windows)`.

**Manual, on Windows**: volume, EQ preset change, normalization toggle, gapless
between two tracks, seek, queue advance, sleep timer. Then the packaging test —
install the NSIS output on a clean VM and confirm it launches and plays.

Per CLAUDE.md, the volume gap is a *bug fix*: write a failing test for
"`set_volume` reaches the backend" before implementing.

## TRACES

- Windows `MpvBackend` construction in `create_player_backend` → `// TRACES: UR-003 | IR-030`
- `detect_audio_system` Windows branch → `IR-030`
- Existing `set_audio_settings` gains Windows coverage → `UR-027, UR-032, UR-033 | DR-030, DR-035, DR-036`
- Allocate **IR-030** in `requirements.md` ("libmpv integration for Windows audio playback").

## Notes for the implementer

- Do this **after** [libmpv2-migration.md](libmpv2-migration.md) — porting the
  current dead `libmpv` git pin to a second platform would double the migration
  work.
- `libmpv2` has broken its API in every major release (4.0 removed command
  helpers, 5.0 removed `mpv_node`, 6.0 changed `RenderContext` ownership). Pin an
  exact version.
- Only the `render`-feature parts of `libmpv2` concern video; audio-only use does
  not need it, and disabling the default `render` feature may shrink the build.
- A parallel Claude session may be active — `git diff` first.