feat(video): mpv plays video on Linux, composited under the webview
DR-231 works. Video and audio, drawn by mpv into a framebuffer we own and
blitted into the default vbox's draw handler with `gdk_cairo_draw_from_gl()`.
The widget tree Tauri built is untouched, so nothing here can be invalidated by
a Tauri upgrade that assumes its own layout.
That settles finding 2 of playback-backend-unification.md on Linux by
demonstration rather than argument, and completes the half of G1 the spike could
not test.
Three pieces had to land together, none of which existed before:
- mpv was configured with `video: no` and no video output, so it had never
decoded a frame in this app. Now `vo=libmpv` when native video is on.
- `player_play_item` skipped loading into the native backend on Linux behind a
`#[cfg(not(target_os = "linux"))]`, because the webview always played video
there. With the webview no longer loading it, that guard meant *nothing*
played — no picture and no audio, which reads as a broken stream rather than
as a file nobody was given.
- The webview paints its own opaque background. Android clears it through a
Kotlin bridge from `enableNativeVideoCompositing()`; the CSS half of that
already ran on Linux, so only `transparent: true` on the window was missing.
Until it was, the frame was rendered correctly and covered by white.
Frame pacing is polled, not pushed. The tick callback asks mpv `has_frame()` and
draws only when the answer is yes. Both neighbouring designs were tried and both
fail, in ways that point at the wrong culprit:
- Waiting on mpv's update callback before rendering *deadlocks*: mpv does not
progress until the client renders, so if the client waits to be told, the
two hold each other. The file loads, one frame appears, and everything
stops.
- Rendering every frame-clock tick and reporting a swap each time claims a
presentation far more often than one happened. It plays, and judders badly —
which reads as a GPU or compositing limit, exactly as the spike warned.
The update callback survives as a hint and does the least it safely can from an
mpv thread: set an `AtomicBool`. It must not touch GTK — `idle_add_local*`
requires the caller to own the main context and panics from there — and it must
not hold the `Rc<RefCell<..>>` state, which is not `Send`.
Two memory-safety fixes in this file's own short history, both worth recording
because neither announced itself:
- The callback context was handed over with `Rc::into_raw` (a pointer to the
Rc's *contents*) and read back as `*const Rc<..>`, reinterpreting a RefCell
as an Rc and corrupting its refcount on the first clone. mpv invokes the
callback immediately, so this happened before anything drew. The symptom was
the process ending quietly with status 0.
- The surface was attached before the player backend was constructed, so the
mpv handle it needs had not been registered yet and it found null every
time.
Teardown (DR-232) is confirmed working on a real run: callback unregistered,
render context freed, GL objects released with the context still current, boxed
callback state reclaimed only after mpv can no longer reach it — no crash.
Still behind JELLYTAU_NATIVE_VIDEO=1 and off by default. Known open: whether
exiting the player stops mpv (reported, evidence ambiguous, needs re-checking
now the picture works), hardware decode (DR-236), and deleting the webview video
path (DR-235).
This commit is contained in:
@@ -725,18 +725,30 @@ pub async fn player_play_item(
|
||||
}
|
||||
|
||||
let controller = player.0.lock().await;
|
||||
// On Linux, video plays in the WebKitGTK HTML5 <video> element (see
|
||||
// get_player_status -> use_html5_element). The MPV backend has no embedded
|
||||
// window, so loading the stream into it would only start a redundant decode
|
||||
// (and the frontend would immediately stop it). Only load into the native
|
||||
// backend on platforms that actually render video through it (e.g. Android).
|
||||
#[cfg(not(target_os = "linux"))]
|
||||
controller
|
||||
.play_item(media_item)
|
||||
.map_err(|e| e.to_string())?;
|
||||
#[cfg(target_os = "linux")]
|
||||
{
|
||||
// Keep the queue in sync for UI/remote-transfer without starting MPV.
|
||||
// Who gets the stream depends on who is going to *render* it, which is a
|
||||
// runtime question, not a platform constant.
|
||||
//
|
||||
// Historically Linux video was always the webview's (`use_html5_element`),
|
||||
// so handing the file to MPV as well would only have started a redundant
|
||||
// decode with no window to show it in — hence a `#[cfg(not(linux))]` guard
|
||||
// and a queue-only path here. With mpv drawing the picture that inverts:
|
||||
// the webview is no longer loading anything, so if this does not load the
|
||||
// file, *nothing does*. The symptom is total silence — no picture and no
|
||||
// audio — which reads like a broken stream rather than a stream nobody was
|
||||
// given.
|
||||
//
|
||||
// This is the fifth place in this cycle where a renderer's capability was
|
||||
// written as a compile-time platform fact. Same fix as the others: ask.
|
||||
//
|
||||
// TRACES: UR-080 | DR-231, DR-235
|
||||
let renders_natively = cfg!(not(target_os = "linux")) || crate::player::native_video::enabled();
|
||||
if renders_natively {
|
||||
controller
|
||||
.play_item(media_item)
|
||||
.map_err(|e| e.to_string())?;
|
||||
} else {
|
||||
// The webview will play it; keep the queue in sync for the UI and for a
|
||||
// remote transfer without starting a second decode.
|
||||
controller
|
||||
.set_current_item(media_item)
|
||||
.map_err(|e| e.to_string())?;
|
||||
|
||||
@@ -166,9 +166,17 @@ impl MpvRenderContext {
|
||||
get_proc_address_ctx: ptr::null_mut(),
|
||||
};
|
||||
let mut api_type = CString::new("opengl").ok()?;
|
||||
// `advanced_control` lets mpv tell us when a frame is ready instead of
|
||||
// us guessing; it is what makes the update callback meaningful (DR-233).
|
||||
let mut advanced: c_int = 1;
|
||||
// Advanced control is deliberately OFF.
|
||||
//
|
||||
// With it on, mpv expects the client to drive rendering to a stricter
|
||||
// contract than a GTK draw handler can promise — it will wait on us, and
|
||||
// if we in turn wait on its update callback, neither side proceeds. That
|
||||
// deadlock presents as a file that loads, renders one frame, and then
|
||||
// sits there with no audio and a spinner.
|
||||
//
|
||||
// Off, mpv is tolerant of being rendered on the toolkit's schedule,
|
||||
// which is what the frame clock gives us.
|
||||
let mut advanced: c_int = 0;
|
||||
|
||||
let mut params = [
|
||||
libmpv_sys::mpv_render_param {
|
||||
@@ -223,6 +231,29 @@ impl MpvRenderContext {
|
||||
libmpv_sys::mpv_render_context_set_update_callback(self.ctx, callback, ctx);
|
||||
}
|
||||
|
||||
/// Whether mpv has a new frame waiting.
|
||||
///
|
||||
/// Asked of mpv directly rather than inferred from its update callback, and
|
||||
/// that distinction is the whole of frame pacing here:
|
||||
///
|
||||
/// - Waiting only on the callback deadlocks — mpv will not progress until
|
||||
/// the client renders, so if the client will not render until mpv says
|
||||
/// so, neither moves. That presents as a file that loads, shows one
|
||||
/// frame, and then sits silent.
|
||||
/// - Rendering on *every* frame-clock tick regardless is the opposite
|
||||
/// error: `report_swap` then claims a presentation far more often than
|
||||
/// real frames exist, mpv has nothing coherent to time against, and
|
||||
/// playback judders badly.
|
||||
///
|
||||
/// Polling is neither. It runs on the main thread, costs a single atomic
|
||||
/// read inside mpv, and answers the only question that matters.
|
||||
///
|
||||
/// TRACES: UR-080 | DR-233
|
||||
pub unsafe fn has_frame(&self) -> bool {
|
||||
let flags = libmpv_sys::mpv_render_context_update(self.ctx);
|
||||
(flags & libmpv_sys::mpv_render_update_flag_MPV_RENDER_UPDATE_FRAME as u64) != 0
|
||||
}
|
||||
|
||||
/// Render the current frame at `width` x `height`, returning the texture id
|
||||
/// holding it. The GL context must be current.
|
||||
///
|
||||
|
||||
@@ -31,6 +31,8 @@
|
||||
use std::cell::RefCell;
|
||||
use std::ffi::c_void;
|
||||
use std::rc::Rc;
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
use std::sync::Arc;
|
||||
|
||||
use gtk::prelude::*;
|
||||
use gtk::{gdk, glib};
|
||||
@@ -47,9 +49,19 @@ struct SurfaceState {
|
||||
gl: Option<gdk::GLContext>,
|
||||
render: Option<MpvRenderContext>,
|
||||
mpv: *mut libmpv_sys::mpv_handle,
|
||||
/// The widget to repaint when mpv reports a frame. Held here because the
|
||||
/// update callback arrives on an mpv thread with nothing but this state.
|
||||
widget: gtk::Box,
|
||||
/// Set by mpv's update callback (on an mpv thread), cleared by the frame
|
||||
/// clock (on the main thread). The whole cross-thread contract.
|
||||
frame_ready: Arc<AtomicBool>,
|
||||
/// The boxed clone of `frame_ready` handed to mpv, reclaimed on teardown.
|
||||
/// Null when no callback is registered.
|
||||
callback_ctx: *mut Arc<AtomicBool>,
|
||||
// One-shot diagnostic latches; see `draw`.
|
||||
logged_first_draw: bool,
|
||||
logged_first_frame: bool,
|
||||
logged_no_gl: bool,
|
||||
logged_no_window: bool,
|
||||
logged_no_size: bool,
|
||||
logged_render_fail: bool,
|
||||
}
|
||||
|
||||
impl SurfaceState {
|
||||
@@ -65,8 +77,16 @@ impl SurfaceState {
|
||||
if let Some(gl) = &self.gl {
|
||||
gl.make_current();
|
||||
}
|
||||
// Unregisters the callback before freeing the context.
|
||||
unsafe { render.destroy() };
|
||||
}
|
||||
// Only now is it safe to reclaim what the callback was holding: mpv can
|
||||
// no longer reach it. Freeing it first would be the use-after-free this
|
||||
// ordering exists to prevent.
|
||||
if !self.callback_ctx.is_null() {
|
||||
unsafe { drop(Box::from_raw(self.callback_ctx)) };
|
||||
self.callback_ctx = std::ptr::null_mut();
|
||||
}
|
||||
self.gl = None;
|
||||
}
|
||||
}
|
||||
@@ -100,18 +120,20 @@ unsafe extern "C" fn on_mpv_update(ctx: *mut c_void) {
|
||||
if ctx.is_null() {
|
||||
return;
|
||||
}
|
||||
let state = &*(ctx as *const Rc<RefCell<SurfaceState>>);
|
||||
let state = state.clone();
|
||||
// Hop to the main loop: GTK is not thread-safe and mpv is not on its thread.
|
||||
// Runs on an *mpv* thread. It therefore does exactly one thing that is safe
|
||||
// to do from there: set an atomic flag.
|
||||
//
|
||||
// Asking for a repaint is the entire job. Without it a decoded frame simply
|
||||
// waits — the picture would then only advance when something else happened
|
||||
// to invalidate the widget, which looks like severe stutter rather than like
|
||||
// a missing call.
|
||||
glib::idle_add_local_once(move || {
|
||||
let Ok(s) = state.try_borrow() else { return };
|
||||
s.widget.queue_draw();
|
||||
});
|
||||
// It must not touch GTK, and specifically must not schedule work with
|
||||
// `idle_add_local*`, which requires the calling thread to own the default
|
||||
// main context — from here that panics with "default main context already
|
||||
// acquired by another thread". Nor can it hold the `Rc<RefCell<..>>` state:
|
||||
// an `Rc` is not `Send`, and cloning one from two threads races its
|
||||
// refcount.
|
||||
//
|
||||
// The frame clock on the widget picks the flag up on the main thread. See
|
||||
// `install_frame_clock`.
|
||||
let flag = &*(ctx as *const Arc<AtomicBool>);
|
||||
flag.store(true, Ordering::Release);
|
||||
}
|
||||
|
||||
/// Start drawing mpv's video underneath the webview.
|
||||
@@ -132,7 +154,14 @@ pub fn attach(vbox: >k::Box, mpv: *mut libmpv_sys::mpv_handle) -> bool {
|
||||
gl: None,
|
||||
render: None,
|
||||
mpv,
|
||||
widget: vbox.clone(),
|
||||
frame_ready: Arc::new(AtomicBool::new(false)),
|
||||
callback_ctx: std::ptr::null_mut(),
|
||||
logged_first_draw: false,
|
||||
logged_first_frame: false,
|
||||
logged_no_gl: false,
|
||||
logged_no_window: false,
|
||||
logged_no_size: false,
|
||||
logged_render_fail: false,
|
||||
}));
|
||||
|
||||
let mut handlers = Vec::new();
|
||||
@@ -154,6 +183,48 @@ pub fn attach(vbox: >k::Box, mpv: *mut libmpv_sys::mpv_handle) -> bool {
|
||||
unrealize_state.borrow_mut().teardown();
|
||||
}));
|
||||
|
||||
// Drive the render loop from the widget's frame clock, on the main thread,
|
||||
// rendering only when mpv actually has a frame.
|
||||
//
|
||||
// Both nearby mistakes were made and are worth naming, because each has a
|
||||
// symptom that points somewhere else:
|
||||
//
|
||||
// - Waiting on mpv's update callback before rendering deadlocks. mpv does
|
||||
// not progress until the client renders. The file loads, one frame
|
||||
// appears, and everything stops — no picture, no audio, a spinner that
|
||||
// never clears. It reads as a broken stream.
|
||||
// - Rendering unconditionally every tick and reporting a swap each time
|
||||
// tells mpv a frame reached the screen far more often than one did. It
|
||||
// plays, and judders badly. It reads as a GPU or compositing limit.
|
||||
//
|
||||
// Polling `has_frame` each tick is neither.
|
||||
//
|
||||
// The frame clock only ticks while the widget is mapped, so this costs
|
||||
// nothing when the window is hidden.
|
||||
//
|
||||
// TRACES: UR-080 | DR-233
|
||||
let tick_state = state.clone();
|
||||
vbox.add_tick_callback(move |widget, _clock| {
|
||||
// Ask mpv, on the main thread, whether there is anything new. The
|
||||
// update callback's flag is only a hint that something *may* have
|
||||
// happened; `has_frame` is the authority, and asking it here is what
|
||||
// keeps this from either deadlocking or over-presenting.
|
||||
let ready = match tick_state.try_borrow() {
|
||||
Ok(s) => {
|
||||
s.frame_ready.swap(false, Ordering::AcqRel);
|
||||
match s.render.as_ref() {
|
||||
Some(render) => unsafe { render.has_frame() },
|
||||
None => false,
|
||||
}
|
||||
}
|
||||
Err(_) => false,
|
||||
};
|
||||
if ready {
|
||||
widget.queue_draw();
|
||||
}
|
||||
glib::ControlFlow::Continue
|
||||
});
|
||||
|
||||
let draw_state = state.clone();
|
||||
handlers.push(vbox.connect_draw(move |widget, cr| {
|
||||
draw(widget, cr, &draw_state);
|
||||
@@ -220,14 +291,19 @@ fn init_gl(widget: >k::Box, state: &Rc<RefCell<SurfaceState>>) -> Result<(), S
|
||||
let mut render =
|
||||
unsafe { MpvRenderContext::new(mpv) }.ok_or("mpv render context creation failed")?;
|
||||
|
||||
// The callback holds a pointer to the shared state, so that Rc must outlive
|
||||
// it. `destroy` unregisters before freeing, which is what makes it safe.
|
||||
let ctx_ptr = Rc::into_raw(state.clone()) as *mut c_void;
|
||||
unsafe { render.set_update_callback(Some(on_mpv_update), ctx_ptr) };
|
||||
// The callback needs an owned handle that outlives this function, so a
|
||||
// clone of the flag is boxed and leaked. `Arc<AtomicBool>` rather than the
|
||||
// state itself: it is the only thing that may cross to an mpv thread. The
|
||||
// pointer is kept so teardown can reclaim it — after the callback is
|
||||
// unregistered, never before.
|
||||
let flag = state.borrow().frame_ready.clone();
|
||||
let ctx_box: *mut Arc<AtomicBool> = Box::into_raw(Box::new(flag));
|
||||
unsafe { render.set_update_callback(Some(on_mpv_update), ctx_box as *mut c_void) };
|
||||
|
||||
let mut s = state.borrow_mut();
|
||||
s.gl = Some(gl);
|
||||
s.render = Some(render);
|
||||
s.callback_ctx = ctx_box;
|
||||
info!("[VideoSurface] GL and render context ready");
|
||||
Ok(())
|
||||
}
|
||||
@@ -239,13 +315,33 @@ fn init_gl(widget: >k::Box, state: &Rc<RefCell<SurfaceState>>) -> Result<(), S
|
||||
/// mean "draw nothing this pass" rather than an error — the webview then paints
|
||||
/// over an untouched background, which is exactly the pre-native appearance.
|
||||
fn draw(widget: >k::Box, cr: >k::cairo::Context, state: &Rc<RefCell<SurfaceState>>) {
|
||||
// Report each way of doing nothing exactly once. Without this the whole
|
||||
// path is invisible: a draw handler that never runs, one that bails on a
|
||||
// zero allocation, and one that renders perfectly all look identical from
|
||||
// outside — and mpv stalls if frames are never consumed, so "no audio and
|
||||
// it hangs" is a plausible symptom of *any* of them.
|
||||
fn once(flag: &mut bool, msg: &str) {
|
||||
if !*flag {
|
||||
*flag = true;
|
||||
warn!("[VideoSurface] not drawing: {msg}");
|
||||
}
|
||||
}
|
||||
|
||||
let Ok(mut s) = state.try_borrow_mut() else {
|
||||
return;
|
||||
};
|
||||
if !s.logged_first_draw {
|
||||
s.logged_first_draw = true;
|
||||
info!("[VideoSurface] draw handler running");
|
||||
}
|
||||
let Some(gl) = s.gl.clone() else {
|
||||
let f = &mut s.logged_no_gl;
|
||||
once(f, "no GL context");
|
||||
return;
|
||||
};
|
||||
let Some(window) = widget.window() else {
|
||||
let f = &mut s.logged_no_window;
|
||||
once(f, "widget has no GdkWindow");
|
||||
return;
|
||||
};
|
||||
|
||||
@@ -253,17 +349,27 @@ fn draw(widget: >k::Box, cr: >k::cairo::Context, state: &Rc<RefCell<SurfaceS
|
||||
let width = widget.allocated_width() * scale;
|
||||
let height = widget.allocated_height() * scale;
|
||||
if width <= 0 || height <= 0 {
|
||||
let f = &mut s.logged_no_size;
|
||||
once(f, "zero allocation");
|
||||
return;
|
||||
}
|
||||
|
||||
gl.make_current();
|
||||
|
||||
let Some(render) = s.render.as_mut() else {
|
||||
return;
|
||||
};
|
||||
let Some(texture) = (unsafe { render.render(width, height) }) else {
|
||||
// Render and end the mutable borrow before touching the latches again.
|
||||
let rendered = match s.render.as_mut() {
|
||||
Some(render) => unsafe { render.render(width, height) },
|
||||
None => return,
|
||||
};
|
||||
let Some(texture) = rendered else {
|
||||
let f = &mut s.logged_render_fail;
|
||||
once(f, "mpv render produced no texture");
|
||||
return;
|
||||
};
|
||||
if !s.logged_first_frame {
|
||||
s.logged_first_frame = true;
|
||||
info!("[VideoSurface] first frame rendered ({width}x{height}, texture {texture})");
|
||||
}
|
||||
|
||||
unsafe {
|
||||
cr.draw_from_gl(
|
||||
@@ -278,6 +384,8 @@ fn draw(widget: >k::Box, cr: >k::cairo::Context, state: &Rc<RefCell<SurfaceS
|
||||
);
|
||||
// Tell mpv the frame reached the screen. Without this it has nothing to
|
||||
// pace against — see DR-233.
|
||||
render.report_swap();
|
||||
if let Some(render) = s.render.as_ref() {
|
||||
render.report_swap();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -17,7 +17,8 @@
|
||||
"height": 800,
|
||||
"minWidth": 800,
|
||||
"minHeight": 600,
|
||||
"resizable": true
|
||||
"resizable": true,
|
||||
"transparent": true
|
||||
}
|
||||
],
|
||||
"security": {
|
||||
|
||||
Reference in New Issue
Block a user