feat(video): mpv plays video on Linux, composited under the webview

DR-231 works. Video and audio, drawn by mpv into a framebuffer we own and
blitted into the default vbox's draw handler with `gdk_cairo_draw_from_gl()`.
The widget tree Tauri built is untouched, so nothing here can be invalidated by
a Tauri upgrade that assumes its own layout.

That settles finding 2 of playback-backend-unification.md on Linux by
demonstration rather than argument, and completes the half of G1 the spike could
not test.

Three pieces had to land together, none of which existed before:

  - mpv was configured with `video: no` and no video output, so it had never
    decoded a frame in this app. Now `vo=libmpv` when native video is on.
  - `player_play_item` skipped loading into the native backend on Linux behind a
    `#[cfg(not(target_os = "linux"))]`, because the webview always played video
    there. With the webview no longer loading it, that guard meant *nothing*
    played — no picture and no audio, which reads as a broken stream rather than
    as a file nobody was given.
  - The webview paints its own opaque background. Android clears it through a
    Kotlin bridge from `enableNativeVideoCompositing()`; the CSS half of that
    already ran on Linux, so only `transparent: true` on the window was missing.
    Until it was, the frame was rendered correctly and covered by white.

Frame pacing is polled, not pushed. The tick callback asks mpv `has_frame()` and
draws only when the answer is yes. Both neighbouring designs were tried and both
fail, in ways that point at the wrong culprit:

  - Waiting on mpv's update callback before rendering *deadlocks*: mpv does not
    progress until the client renders, so if the client waits to be told, the
    two hold each other. The file loads, one frame appears, and everything
    stops.
  - Rendering every frame-clock tick and reporting a swap each time claims a
    presentation far more often than one happened. It plays, and judders badly —
    which reads as a GPU or compositing limit, exactly as the spike warned.

The update callback survives as a hint and does the least it safely can from an
mpv thread: set an `AtomicBool`. It must not touch GTK — `idle_add_local*`
requires the caller to own the main context and panics from there — and it must
not hold the `Rc<RefCell<..>>` state, which is not `Send`.

Two memory-safety fixes in this file's own short history, both worth recording
because neither announced itself:

  - The callback context was handed over with `Rc::into_raw` (a pointer to the
    Rc's *contents*) and read back as `*const Rc<..>`, reinterpreting a RefCell
    as an Rc and corrupting its refcount on the first clone. mpv invokes the
    callback immediately, so this happened before anything drew. The symptom was
    the process ending quietly with status 0.
  - The surface was attached before the player backend was constructed, so the
    mpv handle it needs had not been registered yet and it found null every
    time.

Teardown (DR-232) is confirmed working on a real run: callback unregistered,
render context freed, GL objects released with the context still current, boxed
callback state reclaimed only after mpv can no longer reach it — no crash.

Still behind JELLYTAU_NATIVE_VIDEO=1 and off by default. Known open: whether
exiting the player stops mpv (reported, evidence ambiguous, needs re-checking
now the picture works), hardware decode (DR-236), and deleting the webview video
path (DR-235).
This commit is contained in:
2026-08-22 14:22:58 +02:00
parent 2f637d4775
commit d3ecd8ee91
5 changed files with 3577 additions and 3404 deletions
+132 -24
View File
@@ -31,6 +31,8 @@
use std::cell::RefCell;
use std::ffi::c_void;
use std::rc::Rc;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::Arc;
use gtk::prelude::*;
use gtk::{gdk, glib};
@@ -47,9 +49,19 @@ struct SurfaceState {
gl: Option<gdk::GLContext>,
render: Option<MpvRenderContext>,
mpv: *mut libmpv_sys::mpv_handle,
/// The widget to repaint when mpv reports a frame. Held here because the
/// update callback arrives on an mpv thread with nothing but this state.
widget: gtk::Box,
/// Set by mpv's update callback (on an mpv thread), cleared by the frame
/// clock (on the main thread). The whole cross-thread contract.
frame_ready: Arc<AtomicBool>,
/// The boxed clone of `frame_ready` handed to mpv, reclaimed on teardown.
/// Null when no callback is registered.
callback_ctx: *mut Arc<AtomicBool>,
// One-shot diagnostic latches; see `draw`.
logged_first_draw: bool,
logged_first_frame: bool,
logged_no_gl: bool,
logged_no_window: bool,
logged_no_size: bool,
logged_render_fail: bool,
}
impl SurfaceState {
@@ -65,8 +77,16 @@ impl SurfaceState {
if let Some(gl) = &self.gl {
gl.make_current();
}
// Unregisters the callback before freeing the context.
unsafe { render.destroy() };
}
// Only now is it safe to reclaim what the callback was holding: mpv can
// no longer reach it. Freeing it first would be the use-after-free this
// ordering exists to prevent.
if !self.callback_ctx.is_null() {
unsafe { drop(Box::from_raw(self.callback_ctx)) };
self.callback_ctx = std::ptr::null_mut();
}
self.gl = None;
}
}
@@ -100,18 +120,20 @@ unsafe extern "C" fn on_mpv_update(ctx: *mut c_void) {
if ctx.is_null() {
return;
}
let state = &*(ctx as *const Rc<RefCell<SurfaceState>>);
let state = state.clone();
// Hop to the main loop: GTK is not thread-safe and mpv is not on its thread.
// Runs on an *mpv* thread. It therefore does exactly one thing that is safe
// to do from there: set an atomic flag.
//
// Asking for a repaint is the entire job. Without it a decoded frame simply
// waits — the picture would then only advance when something else happened
// to invalidate the widget, which looks like severe stutter rather than like
// a missing call.
glib::idle_add_local_once(move || {
let Ok(s) = state.try_borrow() else { return };
s.widget.queue_draw();
});
// It must not touch GTK, and specifically must not schedule work with
// `idle_add_local*`, which requires the calling thread to own the default
// main context — from here that panics with "default main context already
// acquired by another thread". Nor can it hold the `Rc<RefCell<..>>` state:
// an `Rc` is not `Send`, and cloning one from two threads races its
// refcount.
//
// The frame clock on the widget picks the flag up on the main thread. See
// `install_frame_clock`.
let flag = &*(ctx as *const Arc<AtomicBool>);
flag.store(true, Ordering::Release);
}
/// Start drawing mpv's video underneath the webview.
@@ -132,7 +154,14 @@ pub fn attach(vbox: &gtk::Box, mpv: *mut libmpv_sys::mpv_handle) -> bool {
gl: None,
render: None,
mpv,
widget: vbox.clone(),
frame_ready: Arc::new(AtomicBool::new(false)),
callback_ctx: std::ptr::null_mut(),
logged_first_draw: false,
logged_first_frame: false,
logged_no_gl: false,
logged_no_window: false,
logged_no_size: false,
logged_render_fail: false,
}));
let mut handlers = Vec::new();
@@ -154,6 +183,48 @@ pub fn attach(vbox: &gtk::Box, mpv: *mut libmpv_sys::mpv_handle) -> bool {
unrealize_state.borrow_mut().teardown();
}));
// Drive the render loop from the widget's frame clock, on the main thread,
// rendering only when mpv actually has a frame.
//
// Both nearby mistakes were made and are worth naming, because each has a
// symptom that points somewhere else:
//
// - Waiting on mpv's update callback before rendering deadlocks. mpv does
// not progress until the client renders. The file loads, one frame
// appears, and everything stops — no picture, no audio, a spinner that
// never clears. It reads as a broken stream.
// - Rendering unconditionally every tick and reporting a swap each time
// tells mpv a frame reached the screen far more often than one did. It
// plays, and judders badly. It reads as a GPU or compositing limit.
//
// Polling `has_frame` each tick is neither.
//
// The frame clock only ticks while the widget is mapped, so this costs
// nothing when the window is hidden.
//
// TRACES: UR-080 | DR-233
let tick_state = state.clone();
vbox.add_tick_callback(move |widget, _clock| {
// Ask mpv, on the main thread, whether there is anything new. The
// update callback's flag is only a hint that something *may* have
// happened; `has_frame` is the authority, and asking it here is what
// keeps this from either deadlocking or over-presenting.
let ready = match tick_state.try_borrow() {
Ok(s) => {
s.frame_ready.swap(false, Ordering::AcqRel);
match s.render.as_ref() {
Some(render) => unsafe { render.has_frame() },
None => false,
}
}
Err(_) => false,
};
if ready {
widget.queue_draw();
}
glib::ControlFlow::Continue
});
let draw_state = state.clone();
handlers.push(vbox.connect_draw(move |widget, cr| {
draw(widget, cr, &draw_state);
@@ -220,14 +291,19 @@ fn init_gl(widget: &gtk::Box, state: &Rc<RefCell<SurfaceState>>) -> Result<(), S
let mut render =
unsafe { MpvRenderContext::new(mpv) }.ok_or("mpv render context creation failed")?;
// The callback holds a pointer to the shared state, so that Rc must outlive
// it. `destroy` unregisters before freeing, which is what makes it safe.
let ctx_ptr = Rc::into_raw(state.clone()) as *mut c_void;
unsafe { render.set_update_callback(Some(on_mpv_update), ctx_ptr) };
// The callback needs an owned handle that outlives this function, so a
// clone of the flag is boxed and leaked. `Arc<AtomicBool>` rather than the
// state itself: it is the only thing that may cross to an mpv thread. The
// pointer is kept so teardown can reclaim it — after the callback is
// unregistered, never before.
let flag = state.borrow().frame_ready.clone();
let ctx_box: *mut Arc<AtomicBool> = Box::into_raw(Box::new(flag));
unsafe { render.set_update_callback(Some(on_mpv_update), ctx_box as *mut c_void) };
let mut s = state.borrow_mut();
s.gl = Some(gl);
s.render = Some(render);
s.callback_ctx = ctx_box;
info!("[VideoSurface] GL and render context ready");
Ok(())
}
@@ -239,13 +315,33 @@ fn init_gl(widget: &gtk::Box, state: &Rc<RefCell<SurfaceState>>) -> Result<(), S
/// mean "draw nothing this pass" rather than an error — the webview then paints
/// over an untouched background, which is exactly the pre-native appearance.
fn draw(widget: &gtk::Box, cr: &gtk::cairo::Context, state: &Rc<RefCell<SurfaceState>>) {
// Report each way of doing nothing exactly once. Without this the whole
// path is invisible: a draw handler that never runs, one that bails on a
// zero allocation, and one that renders perfectly all look identical from
// outside — and mpv stalls if frames are never consumed, so "no audio and
// it hangs" is a plausible symptom of *any* of them.
fn once(flag: &mut bool, msg: &str) {
if !*flag {
*flag = true;
warn!("[VideoSurface] not drawing: {msg}");
}
}
let Ok(mut s) = state.try_borrow_mut() else {
return;
};
if !s.logged_first_draw {
s.logged_first_draw = true;
info!("[VideoSurface] draw handler running");
}
let Some(gl) = s.gl.clone() else {
let f = &mut s.logged_no_gl;
once(f, "no GL context");
return;
};
let Some(window) = widget.window() else {
let f = &mut s.logged_no_window;
once(f, "widget has no GdkWindow");
return;
};
@@ -253,17 +349,27 @@ fn draw(widget: &gtk::Box, cr: &gtk::cairo::Context, state: &Rc<RefCell<SurfaceS
let width = widget.allocated_width() * scale;
let height = widget.allocated_height() * scale;
if width <= 0 || height <= 0 {
let f = &mut s.logged_no_size;
once(f, "zero allocation");
return;
}
gl.make_current();
let Some(render) = s.render.as_mut() else {
return;
};
let Some(texture) = (unsafe { render.render(width, height) }) else {
// Render and end the mutable borrow before touching the latches again.
let rendered = match s.render.as_mut() {
Some(render) => unsafe { render.render(width, height) },
None => return,
};
let Some(texture) = rendered else {
let f = &mut s.logged_render_fail;
once(f, "mpv render produced no texture");
return;
};
if !s.logged_first_frame {
s.logged_first_frame = true;
info!("[VideoSurface] first frame rendered ({width}x{height}, texture {texture})");
}
unsafe {
cr.draw_from_gl(
@@ -278,6 +384,8 @@ fn draw(widget: &gtk::Box, cr: &gtk::cairo::Context, state: &Rc<RefCell<SurfaceS
);
// Tell mpv the frame reached the screen. Without this it has nothing to
// pace against — see DR-233.
render.report_swap();
if let Some(render) = s.render.as_ref() {
render.report_swap();
}
}
}