jellytau/docs/specs/windows-native-audio-backend.md

Spec: Windows native audio backend

Status: Proposed Requirements: UR-003, UR-027, UR-032, UR-033 → DR-030, DR-035, DR-036; new IR-030 UX spec: n/a — Settings › Audio already renders the controls Supersedes / revises: acts on the "audio can unify, video cannot" conclusion in playback-backend-unification.md

Summary

Give Windows a real native audio backend instead of the current webview <audio> shim. Windows is the only platform where audio playback has no decoder of its own: WebviewAudioBackend hands a URL to a frontend <audio> element and relays transport commands. It cannot set volume, cannot apply any audio setting, and reports state only via DOM events.

Audio needs no rendering surface, so none of the webview-compositing problems that block unified video apply here. This is the cleanest available win.

Motivation

WebviewAudioBackend was a deliberate stopgap ("audio-only playback for platforms without a native audio backend"), and it works — but it has a hard functional gap. From webview_audio_backend.rs:

fn set_volume(&mut self, volume: f32) -> Result<(), PlayerError> {
    // ...stores locally only; there is no ControlCommand action for volume
}

So volume changes never reach the element; the frontend has to observe the player store and apply volume itself. set_audio_settings likewise stores values that nothing consumes — EQ, normalization, and gapless are all inert on Windows.

Meanwhile the backend-unification investigation established that a native audio engine is unproblematic on Windows specifically: tauri-plugin-libmpv lists Windows as its fully tested platform (in contrast to Linux, where embedding is broken — but that is a video surface problem, which audio does not have).

Layer assignment

Logic / responsibility	Layer	Why it belongs there
Decoding and playing the audio stream	Rust	Playback is domain logic; every other platform already decodes in Rust or a native player. The webview shim is the anomaly.
Applying AudioSettings (EQ/normalize/gapless)	Rust	Same AudioSettings contract as MPV/ExoPlayer; band layout and presets stay canonical in settings.rs.
Position/state reporting	Rust	Restores the project's core principle — the player is the authoritative source of state. Today Windows inverts this: the DOM element is authoritative and Rust mirrors it.
Volume	Rust	Currently broken precisely because it is split across the boundary.
Rendering the player UI	Frontend	Unchanged.

The strongest argument for this change is the third row. CLAUDE.md states playback state is one-directional with the player authoritative; on Windows that is currently false, and the player_report_* round-trip exists to paper over it.

Design

Engine choice

Two viable options; libmpv is recommended for consistency with the Linux audio backend.

	libmpv	GStreamer
Windows status	✅ tauri-plugin-libmpv reports fully tested	✅ works, but…
Rust bindings	libmpv2 6.0.0, active	gstreamer-rs 0.25.x, excellent
Cross-MSVC from Linux	⚠️ needs prebuilt DLL + import lib	❌ gstreamer-sys uses pkg-config, fights cargo-xwin
Code reuse	✅ MpvBackend logic is directly reusable	❌ a second engine to learn
Crossfade capable	❌ single-stream chain	✅ audiomixer

libmpv wins on reuse: MpvBackend's set_audio_settings — the af lavfi graph built by build_af_filter, eq_filter_entries, normalize_filter_entry — is platform-independent and would apply unchanged.

The one reason to prefer GStreamer is crossfade (UR-031), which mpv structurally cannot do. If crossfade becomes a priority, revisit; it would then argue for GStreamer on both Linux and Windows, which is a much larger change.

Structure

Rename the cfg gate so MpvBackend is no longer Linux-only:

// src-tauri/src/player/mod.rs
#[cfg(any(target_os = "linux", target_os = "windows"))]
pub mod mpv_backend;

MpvBackend::new needs one platform-specific branch: detect_audio_system() currently probes pactl/pw-cli//proc/asound/cards to pick an ao. On Windows the equivalent is wasapi (mpv's default), so the detection is a #[cfg] returning "wasapi" — no probing needed.

Everything else — the event loop, the 250ms position thread, the seek-suppression window, the af filter graph — is unchanged.

WebviewAudioBackend stays for other targets (macOS and anything else hitting the not(any(...)) arm) and as the fallback if libmpv fails to initialize. The existing emit_backend_init_failed path already handles that gracefully.

Build

libmpv2-sys is well-suited to cross-compilation: no pkg-config, vendored headers, pregenerated bindings (no libclang). It emits cargo:rustc-link-lib=mpv unconditionally, so the build must supply a linkable import library for x86_64-pc-windows-msvc.

Keep the build_libmpv feature off — its Unix path shells out to mpv-build and explicitly rejects cross-compilation.

🔴 Per CLAUDE.md, the prebuilt libmpv must be added to the builder image (Dockerfile.builder → rebuild + push via scripts/build-builder-image.sh), not installed at CI job time. libmpv-2.dll must also be bundled into the NSIS installer via tauri.conf.json's resources.

Verified build mechanics

The cross-compile path was tested hands-on from Linux (July 2026), not inferred:

• Neither shinchiro nor zhongfly ships an mpv.def or MSVC mpv.lib — only a MinGW libmpv.dll.a. (Several online sources claim otherwise; they are wrong.)
• An MSVC-style import lib can be generated locally with LLVM tools only: llvm-readobj --coff-exports libmpv-2.dll → synthesize mpv.def → llvm-dlltool -m i386:x86-64 -d mpv.def -l mpv.lib. llvm-lib /def: produces a byte-identical result.
• A real lld-link link against that import lib succeeds, and the resulting import table resolves mpv_client_api_version from libmpv-2.dll. lld-link is the linker cargo-xwin uses, so this is the load-bearing step.
• Linking directly against the shipped MinGW libmpv.dll.a also succeeds, so def-generation may be skippable — but that relies on lld's GNU-archive tolerance rather than a documented contract. Keep llvm-dlltool as the fallback.
• MinGW origin is not an ABI problem: libmpv exports a pure C ABI, and the x86-64 Windows calling convention is platform-defined. The upstream note that MSVC cannot build mpv is frequently misread as "MSVC cannot link libmpv" — that is not what it says.
• 🔴 Never free/realloc across the DLL boundary — use mpv_free.

Build wiring is ordinary: cargo:rustc-link-lib=dylib=mpv plus cargo:rustc-link-search. Nothing about libmpv conflicts with cargo-xwin.

Size and shipping

Measured uncompressed: 93 MiB (zhongfly mpv-dev-lgpl-x86_64) vs 112 MiB (shinchiro, full GPL build); ~26–30 MB compressed in the .7z.

Ship the zhongfly LGPL build — smaller, and there is no reason to pull the GPL variant in for an audio-only use.

Import-table inspection confirms no companion DLLs are needed: every dependency is a system DLL (KERNEL32, USER32, d2d1, DWrite, OPENGL32, vulkan-1, UCRT api-ms-win-*). One file to bundle.

93 MiB is still substantial against a Tauri app's usual few MB. Since we use mpv audio-only, investigate whether a pruned build (no video decoders, no libplacebo) is worth producing for the builder image — but treat that as an optimization, not a blocker.

Out of scope

• Windows video. Stays in WebView2 + hls.js — it works and has ABR.
• Crossfade (UR-031/DR-034) — not implemented anywhere; needs its own spec.
• Replacing WebviewAudioBackend for macOS.
• MPRIS/SMTC media-key integration — worth a follow-up, not this spec.

Acceptance criteria

• Windows build produces a MpvBackend-backed player; backend-init-failed is emitted (not a crash) if libmpv is unavailable.
• Volume control works from the UI — the current hard gap.
• EQ, normalization, and gapless audibly take effect on Windows.
• Position/state originate in Rust; the <audio> element is no longer in the audio path.
• Seek, next/previous, and queue advance work; sleep timer stops playback.
• libmpv-2.dll ships in the NSIS installer and the app runs on a clean Windows VM with no mpv installed.
• Builder image carries the Windows libmpv artefacts; no toolchain install added to any CI step.
• bun run check, bun run test, bun run check:boundary pass.
• cargo fmt clean, cargo clippy clean, bun run test:rust passes.
• New requirement-implementing code carries // TRACES: comments.

Testing

Rust: the existing mpv_backend_test.rs and the build_af_filter / normalize_filter_entry / eq_filter_entries unit tests already cover the filter-graph logic and are platform-independent — they should pass unchanged under a Windows cargo check/test. Add a test asserting detect_audio_system() returns wasapi under cfg(windows).

Manual, on Windows: volume, EQ preset change, normalization toggle, gapless between two tracks, seek, queue advance, sleep timer. Then the packaging test — install the NSIS output on a clean VM and confirm it launches and plays.

Per CLAUDE.md, the volume gap is a bug fix: write a failing test for "set_volume reaches the backend" before implementing.

TRACES

• Windows MpvBackend construction in create_player_backend → // TRACES: UR-003 | IR-030
• detect_audio_system Windows branch → IR-030
• Existing set_audio_settings gains Windows coverage → UR-027, UR-032, UR-033 | DR-030, DR-035, DR-036
• Allocate IR-030 in requirements.md ("libmpv integration for Windows audio playback").

Notes for the implementer

• Do this after libmpv2-migration.md — porting the current dead libmpv git pin to a second platform would double the migration work.
• libmpv2 has broken its API in every major release (4.0 removed command helpers, 5.0 removed mpv_node, 6.0 changed RenderContext ownership). Pin an exact version.
• Only the render-feature parts of libmpv2 concern video; audio-only use does not need it, and disabling the default render feature may shrink the build.
• A parallel Claude session may be active — git diff first.