feat(windows): mpv draws Windows video into the app window

DR-237's video half. mpv now renders Windows video the way
tauri-plugin-libmpv does on Windows: MpvBackend is handed the main
window's HWND and sets it as `wid` before mpv initialises, so mpv draws
as a child of the app window beneath the transparent WebView2, with
vo=gpu-next,gpu. osc, default bindings, VO keyboard and cursor handling
are off, so the Svelte controls drawn over the picture are the only ones.

- video_output() decides per platform (UT-274): Window(hwnd) on Windows,
  RenderApi on Linux, Off without native video. Windows with no handle
  draws nothing rather than letting mpv open a top-level window.
- native_video::enabled() is true on Windows as well, so the frontend
  takes the native path there: NativePlayerAdapter, and the page clears
  its background while video is on screen.
- enableNativeVideoCompositing() no longer logs a missing Android bridge
  as an error on the desktop, where there is no bridge to have.

Verified: every option, wid included, is accepted by the real libmpv on
Linux and by the shipped Windows DLL under wine; the Windows unit suite
passes under wine (949). Not yet seen on real Windows hardware.
This commit is contained in:
2026-09-24 22:42:42 -04:00
parent 4daf172834
commit bb3ab1edd7
7 changed files with 240 additions and 31 deletions
+10 -4
View File
@@ -7,9 +7,15 @@ job / SMTC lockscreen), but it runs and plays media.
## How playback works on Windows ## How playback works on Windows
- **Video** — renders through the webview HTML5 `<video>` element (hls.js); on - **Video** — **mpv**, drawing into the app's own window: `MpvBackend` is
Windows that is WebView2 (Chromium/Edge), which plays HLS + h264 fine. mpv handed the main window's HWND as `wid` before mpv initialises and renders with
takes this over in DR-237's video phase; Linux already made that move. `vo=gpu-next,gpu` as a child of it, beneath the transparent WebView2 whose page
clears its background while a video is on screen (`data-native-video`). mpv's
own controller, key bindings and cursor handling are off — the Svelte controls
drawn over it are the only ones. This is the arrangement tauri-plugin-libmpv
ships on Windows (same zhongfly LGPL DLL). **Not yet seen on real Windows
hardware** — if the picture ends up *over* the controls, the z-order of mpv's
child window is the first thing to check.
- **Audio** — **libmpv**, the same `MpvBackend` Linux uses, with `ao=wasapi`. - **Audio** — **libmpv**, the same `MpvBackend` Linux uses, with `ao=wasapi`.
Volume, EQ, normalization and gapless all go through mpv's filter graph as on Volume, EQ, normalization and gapless all go through mpv's filter graph as on
Linux. `libmpv-2.dll` ships beside `jellytau.exe` in the installer. Linux. `libmpv-2.dll` ships beside `jellytau.exe` in the installer.
@@ -87,4 +93,4 @@ Outputs:
1. SMTC (lockscreen / media keys) — not wired on Windows. 1. SMTC (lockscreen / media keys) — not wired on Windows.
2. Code signing — the installer is unsigned. 2. Code signing — the installer is unsigned.
3. Video through mpv (DR-237) — needs a WebView2-side surface. 3. First run of mpv video on real Windows hardware (DR-237).
+1
View File
@@ -869,6 +869,7 @@ Internal architecture, components, and application logic.
| UT-271 | Native video is on for Linux whatever `JELLYTAU_NATIVE_VIDEO` says, including unset and explicit "off" values, and off where mpv is not the video renderer | DR-235 | Done | | UT-271 | Native video is on for Linux whatever `JELLYTAU_NATIVE_VIDEO` says, including unset and explicit "off" values, and off where mpv is not the video renderer | DR-235 | Done |
| UT-272 | No platform reports a webview video fallback, Linux reports native video, and the player status on Linux never sends video to the `<video>` element | DR-235 | Done | | UT-272 | No platform reports a webview video fallback, Linux reports native video, and the player status on Linux never sends video to the `<video>` element | DR-235 | Done |
| UT-273 | `player_play_item`, `get_player_status` and `player_get_capabilities` answer "does a native renderer draw video here" from one function, so the backend is loaded with video exactly where the frontend is told not to use a `<video>` element — on Windows, where mpv now plays audio, the film's soundtrack is not decoded twice | DR-237 | Done | | UT-273 | `player_play_item`, `get_player_status` and `player_get_capabilities` answer "does a native renderer draw video here" from one function, so the backend is loaded with video exactly where the frontend is told not to use a `<video>` element — on Windows, where mpv now plays audio, the film's soundtrack is not decoded twice | DR-237 | Done |
| UT-274 | mpv's video output is decided per platform: Windows renders into the app window's HWND (`wid`, set before initialisation) with `vo=gpu-next,gpu` and mpv's own controller, bindings and cursor handling off; Windows with no handle draws nothing rather than opening a window of its own; Linux uses the render API; no native video means `video=no`. Every option is accepted by the real libmpv, on Linux and by the shipped Windows DLL | DR-237 | Done |
### Integration Tests ### Integration Tests
| Test ID | Test Description | Traces To | Status | | Test ID | Test Description | Traces To | Status |
+1
View File
@@ -150,6 +150,7 @@ pub fn run_engine(url: &str, engine: Engine) -> u32 {
None, None,
std::sync::Arc::new(tokio::sync::Mutex::new(None)), std::sync::Arc::new(tokio::sync::Mutex::new(None)),
std::sync::Arc::new(crate::playback_reporting::throttle::EventThrottler::new()), std::sync::Arc::new(crate::playback_reporting::throttle::EventThrottler::new()),
None,
) )
.expect("could not create the legacy backend"), .expect("could not create the legacy backend"),
crate::player::media_player::Capabilities::mpv(), crate::player::media_player::Capabilities::mpv(),
+19 -1
View File
@@ -699,7 +699,25 @@ fn create_player_backend(
#[cfg(any(target_os = "linux", target_os = "windows"))] #[cfg(any(target_os = "linux", target_os = "windows"))]
{ {
info!("Initializing MPV backend"); info!("Initializing MPV backend");
match MpvBackend::new(Some(_event_emitter), playback_reporter, position_throttler) { // Windows: mpv draws video into the main window itself (DR-237), so it
// needs the HWND before it initialises. Linux draws through the render
// API into a GTK surface attached later, and needs no handle.
#[cfg(target_os = "windows")]
let video_window = {
use tauri::Manager;
app_handle
.get_webview_window("main")
.and_then(|w| w.hwnd().ok())
.map(|hwnd| hwnd.0 as i64)
};
#[cfg(not(target_os = "windows"))]
let video_window = None;
match MpvBackend::new(
Some(_event_emitter),
playback_reporter,
position_throttler,
video_window,
) {
Ok(backend) => { Ok(backend) => {
info!("Successfully initialized MPV backend"); info!("Successfully initialized MPV backend");
Box::new(backend) Box::new(backend)
+180 -13
View File
@@ -149,12 +149,73 @@ pub fn registered_handle() -> *mut libmpv_sys::mpv_handle {
.unwrap_or(std::ptr::null_mut()) .unwrap_or(std::ptr::null_mut())
} }
/// How mpv shows video on this platform.
///
/// TRACES: UR-080 | DR-231, DR-237
#[derive(Debug, Clone, PartialEq, Eq)]
pub(crate) enum VideoOutput {
/// No picture: audio-only playback, or nowhere to draw.
Off,
/// Linux: frames through the render API into the GTK surface beneath the
/// webview (`video_surface`).
RenderApi,
/// Windows: mpv renders as a child of the app's own window (`wid`, set
/// before initialisation), beneath the transparent WebView2 — the
/// arrangement tauri-plugin-libmpv ships on Windows.
Window(i64),
}
impl VideoOutput {
/// Runtime options for this output. `wid` is not among them: it only takes
/// effect before initialisation, so the constructor sets it separately.
pub(crate) fn options(&self) -> Vec<(&'static str, String)> {
match self {
VideoOutput::Off => vec![("video", "no".to_string())],
VideoOutput::RenderApi => vec![("vo", "libmpv".to_string())],
VideoOutput::Window(_) => [
// libplacebo's renderer, with the classic one as fallback for a
// build or GPU that lacks it.
("vo", "gpu-next,gpu"),
// mpv is a surface here, not a player: the app's controls are
// drawn over it, so its own controller and bindings must not
// answer clicks, keys or the cursor.
("osc", "no"),
("input-default-bindings", "no"),
("input-vo-keyboard", "no"),
("input-cursor", "no"),
("cursor-autohide", "no"),
]
.into_iter()
.map(|(k, v)| (k, v.to_string()))
.collect(),
}
}
}
/// Decide the video output from whether native video is on, the platform, and
/// the app window's handle (Windows only).
///
/// TRACES: UR-080 | DR-231, DR-237 | UT-274
pub(crate) fn video_output(native: bool, is_windows: bool, window: Option<i64>) -> VideoOutput {
match (native, is_windows, window) {
(false, _, _) => VideoOutput::Off,
(true, true, Some(wid)) => VideoOutput::Window(wid),
// No handle: mpv would open a top-level window of its own.
(true, true, None) => VideoOutput::Off,
(true, false, _) => VideoOutput::RenderApi,
}
}
impl MpvBackend { impl MpvBackend {
/// Create a new MPV backend /// Create a new MPV backend
///
/// `video_window` is the app window's native handle (an HWND), which mpv
/// draws video into on Windows; `None` elsewhere.
pub fn new( pub fn new(
event_emitter: Option<Arc<dyn PlayerEventEmitter>>, event_emitter: Option<Arc<dyn PlayerEventEmitter>>,
playback_reporter: Arc<TokioMutex<Option<PlaybackReporter>>>, playback_reporter: Arc<TokioMutex<Option<PlaybackReporter>>>,
position_throttler: Arc<EventThrottler>, position_throttler: Arc<EventThrottler>,
video_window: Option<i64>,
) -> Result<Self, PlayerError> { ) -> Result<Self, PlayerError> {
info!("[MpvBackend] Initializing MPV backend..."); info!("[MpvBackend] Initializing MPV backend...");
@@ -166,7 +227,22 @@ impl MpvBackend {
libc::setlocale(libc::LC_NUMERIC, c_locale.as_ptr()); libc::setlocale(libc::LC_NUMERIC, c_locale.as_ptr());
} }
let mpv = Mpv::new().map_err(|e| PlayerError { let output = video_output(
super::native_video::enabled(),
cfg!(target_os = "windows"),
video_window,
);
if super::native_video::enabled() && output == VideoOutput::Off {
error!("[MpvBackend] no window handle to draw video into; video will have no picture");
}
// `wid` only takes effect before initialisation. TRACES: UR-080 | DR-237
let mpv = Mpv::with_initializer(|init| {
if let VideoOutput::Window(wid) = output {
init.set_property("wid", wid)?;
}
Ok(())
})
.map_err(|e| PlayerError {
message: format!("Failed to initialize MPV: {:?}", e), message: format!("Failed to initialize MPV: {:?}", e),
})?; })?;
// TRACES: UR-012 | DR-299 // TRACES: UR-012 | DR-299
@@ -203,24 +279,21 @@ impl MpvBackend {
// //
// Linux video went through the webview until DR-235, and decoding it // Linux video went through the webview until DR-235, and decoding it
// here too would have burned a core for a picture nobody saw — hence // here too would have burned a core for a picture nobody saw — hence
// `video: no`. With native video, mpv needs both the decoder *and* // `video: no`. With native video, mpv needs the decoder *and* an output
// `vo=libmpv` — the render API only works through that output, and the // that draws where the app wants it: the render API on Linux (the default
// default would try to open a window of its own. // would open a window of its own), the app's window on Windows.
// //
// Set at construction because mpv resolves the video output when it // Set at construction because mpv resolves the video output when it
// initialises; flipping it later does not re-open one. // initialises; flipping it later does not re-open one.
// //
// TRACES: UR-080 | DR-231, DR-235 // TRACES: UR-080 | DR-231, DR-235, DR-237
if super::native_video::enabled() { for (name, value) in output.options() {
mpv.set_property("vo", "libmpv").map_err(|e| PlayerError { mpv.set_property(name, value.as_str())
message: format!("Failed to select the libmpv video output: {:?}", e), .map_err(|e| PlayerError {
})?; message: format!("Failed to set {name}={value}: {:?}", e),
info!("[MpvBackend] native video enabled (vo=libmpv)");
} else {
mpv.set_property("video", "no").map_err(|e| PlayerError {
message: format!("Failed to configure MPV video: {:?}", e),
})?; })?;
} }
info!("[MpvBackend] video output: {:?}", output);
// Set volume to 100% (we'll control via MPV's volume property) // Set volume to 100% (we'll control via MPV's volume property)
mpv.set_property("volume", 100i64) mpv.set_property("volume", 100i64)
@@ -1023,3 +1096,97 @@ mod af_filter_tests {
assert!(eq_pos < norm_pos, "normalizer runs after EQ: {af}"); assert!(eq_pos < norm_pos, "normalizer runs after EQ: {af}");
} }
} }
#[cfg(test)]
mod video_output_tests {
use super::{video_output, VideoOutput};
/// Windows draws into the app's own window: mpv is handed its HWND before
/// initialising and renders as a child of it, beneath the transparent
/// WebView2.
///
/// TRACES: UR-080 | DR-237 | UT-274
#[test]
fn windows_native_video_renders_into_the_app_window() {
assert_eq!(
video_output(true, true, Some(0x1234)),
VideoOutput::Window(0x1234)
);
}
/// Without a handle mpv would open a top-level window of its own, a second
/// window floating beside the app. No picture is the honest failure.
///
/// TRACES: UR-080 | DR-237 | UT-274
#[test]
fn windows_without_a_window_handle_draws_nothing() {
assert_eq!(video_output(true, true, None), VideoOutput::Off);
}
/// Linux keeps the render API the GTK surface draws from.
///
/// TRACES: UR-080 | DR-231 | UT-274
#[test]
fn linux_native_video_uses_the_render_api() {
assert_eq!(video_output(true, false, None), VideoOutput::RenderApi);
}
/// TRACES: UR-080 | DR-231 | UT-274
#[test]
fn no_native_video_decodes_no_picture() {
assert_eq!(video_output(false, true, Some(1)), VideoOutput::Off);
assert_eq!(video_output(false, false, None), VideoOutput::Off);
}
/// In the app's window mpv must not act as a player of its own: its
/// on-screen controller and key/mouse bindings would compete with the
/// Svelte controls drawn over it.
///
/// TRACES: UR-080 | DR-237 | UT-274
#[test]
fn a_window_output_hands_all_input_to_the_app() {
let opts = VideoOutput::Window(7).options();
for (k, v) in [
("vo", "gpu-next,gpu"),
("osc", "no"),
("input-default-bindings", "no"),
("input-vo-keyboard", "no"),
("input-cursor", "no"),
("cursor-autohide", "no"),
] {
assert!(
opts.iter().any(|(ok, ov)| *ok == k && ov == v),
"missing {k}={v} in {opts:?}"
);
}
assert_eq!(
VideoOutput::Off.options(),
vec![("video", "no".to_string())]
);
}
/// Every option the outputs set is one this libmpv accepts, `wid` included —
/// against the real library, so a misspelt or removed option fails here
/// rather than as a player that will not start on a user's machine. Runs on
/// the Windows DLL too (under wine in the cross-build).
///
/// TRACES: UR-080 | DR-237 | UT-274
#[test]
fn libmpv_accepts_every_video_output_option() {
let mpv = libmpv::Mpv::with_initializer(|init| {
init.set_property("wid", 0i64)?;
Ok(())
})
.expect("libmpv must accept wid before initialisation");
for output in [
VideoOutput::Off,
VideoOutput::RenderApi,
VideoOutput::Window(0),
] {
for (name, value) in output.options() {
mpv.set_property(name, value.as_str())
.unwrap_or_else(|e| panic!("libmpv rejected {name}={value}: {e:?}"));
}
}
}
}
+11 -12
View File
@@ -16,31 +16,30 @@
/// Whether mpv should decode and draw video in this process. /// Whether mpv should decode and draw video in this process.
/// ///
/// True wherever mpv is the desktop video renderer — Linux, since DR-235 /// True wherever mpv is the desktop video renderer: Linux since DR-235 phase 1,
/// phase 1. There is no opt-out: the webview `<video>` path is no longer a Linux /// Windows since DR-237. There is no opt-out and no webview fallback — the
/// video renderer, so "off" would leave nothing drawing the picture. It was the /// webview `<video>` path is gone, so "off" would leave nothing drawing the
/// `JELLYTAU_NATIVE_VIDEO` opt-in while the render path was being proven; the /// picture. It was the `JELLYTAU_NATIVE_VIDEO` opt-in while the render path was
/// variable is now ignored. Windows joins in DR-237, and only then does the /// being proven; the variable is ignored.
/// webview path go (phase 3).
/// ///
/// TRACES: UR-080 | DR-231, DR-235 /// TRACES: UR-080 | DR-231, DR-235
pub fn enabled() -> bool { pub fn enabled() -> bool {
// On Android ExoPlayer draws video and `use_html5_element` is false for // On Android ExoPlayer draws video and `use_html5_element` is false for
// entirely separate reasons. // entirely separate reasons.
cfg!(all(target_os = "linux", not(target_os = "android"))) cfg!(any(target_os = "linux", target_os = "windows"))
} }
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use super::*; use super::*;
/// mpv draws video on Linux with nothing to opt into, and the retired /// mpv draws video on Linux and Windows with nothing to opt into, and the retired
/// variable cannot opt back out: with the webview path gone from Linux, "off" /// variable cannot opt back out: with the webview path gone, "off"
/// would configure mpv for audio only with nothing else to draw the picture. /// would configure mpv for audio only with nothing else to draw the picture.
/// ///
/// TRACES: UR-080 | DR-235 | UT-271 /// TRACES: UR-080 | DR-235, DR-237 | UT-271
#[test] #[test]
fn linux_always_renders_video_natively() { fn desktop_always_renders_video_natively() {
let restore = std::env::var("JELLYTAU_NATIVE_VIDEO").ok(); let restore = std::env::var("JELLYTAU_NATIVE_VIDEO").ok();
for value in [None, Some("0"), Some("false"), Some("1")] { for value in [None, Some("0"), Some("false"), Some("1")] {
@@ -50,7 +49,7 @@ mod tests {
} }
assert_eq!( assert_eq!(
enabled(), enabled(),
cfg!(all(target_os = "linux", not(target_os = "android"))), cfg!(any(target_os = "linux", target_os = "windows")),
"JELLYTAU_NATIVE_VIDEO={value:?} must not decide the renderer" "JELLYTAU_NATIVE_VIDEO={value:?} must not decide the renderer"
); );
} }
+17
View File
@@ -24,9 +24,18 @@
import { nativeVideoActive } from "$lib/stores/nativeVideo"; import { nativeVideoActive } from "$lib/stores/nativeVideo";
import { createLogger } from "$lib/utils/logger"; import { createLogger } from "$lib/utils/logger";
import { platform } from "@tauri-apps/plugin-os";
const log = createLogger("videoSurface"); const log = createLogger("videoSurface");
function isAndroid(): boolean {
try {
return platform() === "android";
} catch {
return false;
}
}
interface AndroidVideoSurfaceBridge { interface AndroidVideoSurfaceBridge {
setTransparent(transparent: boolean): void; setTransparent(transparent: boolean): void;
isSupported(): boolean; isSupported(): boolean;
@@ -69,6 +78,14 @@ export function enableNativeVideoCompositing(): void {
// would see through the app to the home screen. // would see through the app to the home screen.
nativeVideoActive.set(true); nativeVideoActive.set(true);
const androidVideoSurface = bridge(); const androidVideoSurface = bridge();
// Only Android has a webview widget to make transparent. On the desktop the
// window is created transparent and mpv draws beneath it, so the page layer
// above is all there is — an absent bridge there is correct, not a fault.
// TRACES: UR-080 | DR-237
if (!androidVideoSurface && !isAndroid()) {
nativeVideoActive.set(true);
return;
}
if (!androidVideoSurface) { if (!androidVideoSurface) {
// Say so loudly. Every bridge call in this file is optional-chained, so a // Say so loudly. Every bridge call in this file is optional-chained, so a
// missing bridge is silent — and a silently-skipped setTransparent(true) is // missing bridge is silent — and a silently-skipped setTransparent(true) is