dtourolle/jellytau
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 43 Wiki Activity

feat(player): native video on Linux, and one contract for every player (v0.11.0)

Browse Source 
mpv now decodes video on Linux, drawn into a framebuffer we own and blitted
into the default vbox's draw handler. Tauri's widget tree is untouched, so an
upgrade that assumes its own layout cannot invalidate this. Direct play means
the original file, hardware decoding, and no server transcode at all — where
previously every desktop video was re-encoded to h264 for the browser engine,
whatever the file actually was. Off by default: JELLYTAU_NATIVE_VIDEO=1.

That settles finding 2 of playback-backend-unification.md — "native video
cannot be composited with a Tauri webview" — by demonstration rather than
argument, on X11 and Wayland both.

Turning it on exposed nine defects, none of them mpv's. Each was the same
mistake in a different place: a capability written down as a compile-time fact
about the platform, or a state asserted instead of confirmed.

  DR-238/246  a seek routed by the stream's container rather than by what the
              engine could do with it - correct only while one player handled
              those streams, silent the moment another did
  DR-239      a property handled but never observed, so the play/pause button
              waited for an event that could not arrive
  DR-240      fullscreen expanding the document while the window stayed put
  DR-241      a seek issued before the engine had a file, failed, and discarded
              - which is why resume began at zero
  DR-247      a Linux-only gate outliving the caller that made it Linux-only,
              breaking the Android build outright
  DR-250      a stop aimed at whichever renderer bookkeeping believed was in
              charge, missing the one actually making sound
  DR-251      a duration of zero believed, leaving the seek bar no scale
  DR-252      a junk float converted to a Duration, panicking the backend the
              instant a length-less stream appeared

So the MediaPlayer contract (DR-242 … DR-247): `open` carries a start position,
so no caller sequences load-then-seek and none can race an engine's load;
`seek` states a destination and leaves in-place-versus-re-open to the engine;
`snapshot` is one coherent read; and `Phase::Opening` names the window where
intent used to be lost. One conformance suite runs against every engine —
FakePlayer and mpv under cargo test, ExoPlayer instrumented on a device — so an
engine is either correct or visibly failing.

Two of the nine were introduced during this work and caught on hardware, not by
any suite: an over-broad capability that grouped ExoPlayer with mpv, and the
Duration panic. The suites test engines that behave. That is recorded in
docs/native-player-verification.md, which asks for the exact action sequences
that found them.

Verified: all automated gates, conformance (mpv 9/9, legacy 8/9 by design,
ExoPlayer 7/7 on device), and manual desktop and Android passes on real
hardware.

Known open and deliberately shipped: resume reads local progress and never the
server's; the background-audio handoff still declares a state swap it does not
confirm (the symptom is now impossible, the race is not); and `bun run
android:dev` builds an APK carrying the release application id, whose failure
message advises an uninstall that would destroy app data. Fix that last one
before anyone else builds for Android.

Squashed from worktree-linux-native-video, which keeps the per-defect history.
This commit is contained in:
dtourolle
2026-08-23 10:51:45 +02:00
parent 5fede123e7
commit 11d9d760d8
87 changed files with 15968 additions and 7508 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src-tauri/src/player/mpv_render.rs
+405
View File
@@ -0,0 +1,405 @@
//! mpv's render API, driven into an OpenGL framebuffer we own.
//!
//! This is the half of native video that is not GTK: create a render context
//! over the mpv handle the audio backend already drives, render a frame into a
//! texture, and hand that texture id back for the toolkit to composite.
//!
//! Kept apart from `video_surface` deliberately — everything here is portable
//! across the platforms this app targets, while the surface that consumes it is
//! not. Windows reuses this file unchanged (DR-237).
//!
//! TRACES: UR-080 | DR-231, DR-232, IR-033
use std::ffi::{c_void, CStr, CString};
use std::os::raw::{c_char, c_int};
use std::ptr;
use log::{error, info, warn};
/// GL entry points, resolved once.
///
/// Only the handful needed to own a framebuffer; mpv resolves everything else
/// it needs through [`get_proc_address`].
struct Gl {
gen_framebuffers: unsafe extern "C" fn(c_int, *mut u32),
delete_framebuffers: unsafe extern "C" fn(c_int, *const u32),
bind_framebuffer: unsafe extern "C" fn(u32, u32),
framebuffer_texture_2d: unsafe extern "C" fn(u32, u32, u32, u32, c_int),
gen_textures: unsafe extern "C" fn(c_int, *mut u32),
delete_textures: unsafe extern "C" fn(c_int, *const u32),
bind_texture: unsafe extern "C" fn(u32, u32),
tex_image_2d:
unsafe extern "C" fn(u32, c_int, c_int, c_int, c_int, c_int, u32, u32, *const c_void),
tex_parameteri: unsafe extern "C" fn(u32, u32, c_int),
check_framebuffer_status: unsafe extern "C" fn(u32) -> u32,
}
const GL_TEXTURE_2D: u32 = 0x0DE1;
const GL_FRAMEBUFFER: u32 = 0x8D40;
const GL_COLOR_ATTACHMENT0: u32 = 0x8CE0;
const GL_RGBA: u32 = 0x1908;
const GL_RGBA8: c_int = 0x8058;
const GL_UNSIGNED_BYTE: u32 = 0x1401;
const GL_LINEAR: c_int = 0x2601;
const GL_TEXTURE_MIN_FILTER: u32 = 0x2801;
const GL_TEXTURE_MAG_FILTER: u32 = 0x2800;
const GL_FRAMEBUFFER_COMPLETE: u32 = 0x8CD5;
/// Resolve a GL symbol the way libepoxy actually exports it.
///
/// **This is the trap that cost the spike a debugging cycle.** libepoxy does not
/// export `glFoo` as a function. It exports `epoxy_glFoo` as a *data* symbol
/// holding a lazily-resolving function pointer. So the address `dlsym` returns
/// is the address *of the pointer*, not of any code: returning it makes mpv jump
/// into non-executable data and take SIGSEGV/SEGV_ACCERR on the very first GL
/// call. The value must be read *out of* that location.
///
/// The `epoxy` crate does this correctly and is unusable here — its
/// `gl_generator` dependency pulls a yanked `xml-rs`.
///
/// TRACES: UR-080 | IR-033
unsafe fn resolve(name: &str) -> *mut c_void {
let epoxy_name = match CString::new(format!("epoxy_{name}")) {
Ok(n) => n,
Err(_) => return ptr::null_mut(),
};
let slot = libc::dlsym(libc::RTLD_DEFAULT, epoxy_name.as_ptr());
if !slot.is_null() {
// The symbol holds the function pointer; return what is stored there.
return *(slot as *mut *mut c_void);
}
// Fall back to a plain symbol, for a GL stack that is not behind epoxy.
match CString::new(name) {
Ok(n) => libc::dlsym(libc::RTLD_DEFAULT, n.as_ptr()),
Err(_) => ptr::null_mut(),
}
}
/// What mpv calls to find GL entry points. Same rule as [`resolve`].
unsafe extern "C" fn get_proc_address(_ctx: *mut c_void, name: *const c_char) -> *mut c_void {
if name.is_null() {
return ptr::null_mut();
}
match CStr::from_ptr(name).to_str() {
Ok(n) => resolve(n),
Err(_) => ptr::null_mut(),
}
}
macro_rules! load {
($name:literal) => {{
let p = resolve($name);
if p.is_null() {
error!("[MpvRender] GL symbol not found: {}", $name);
return None;
}
std::mem::transmute(p)
}};
}
impl Gl {
/// Resolve every entry point, or none — a partially-loaded table would fail
/// later at a call site with no context.
///
/// The transmutes are unannotated on purpose: each target type is declared
/// once on the struct field above, and repeating it at the call site would
/// be two places to get the same signature wrong.
#[allow(clippy::missing_transmute_annotations)]
unsafe fn load() -> Option<Self> {
Some(Gl {
gen_framebuffers: load!("glGenFramebuffers"),
delete_framebuffers: load!("glDeleteFramebuffers"),
bind_framebuffer: load!("glBindFramebuffer"),
framebuffer_texture_2d: load!("glFramebufferTexture2D"),
gen_textures: load!("glGenTextures"),
delete_textures: load!("glDeleteTextures"),
bind_texture: load!("glBindTexture"),
tex_image_2d: load!("glTexImage2D"),
tex_parameteri: load!("glTexParameteri"),
check_framebuffer_status: load!("glCheckFramebufferStatus"),
})
}
}
/// A colour-renderable framebuffer mpv draws into, sized to the widget.
struct Target {
fbo: u32,
texture: u32,
width: i32,
height: i32,
}
/// mpv's render context plus the framebuffer it draws into.
///
/// # Lifetime (DR-232)
///
/// The render context must not outlive the GL context it was created against.
/// `Drop` unregisters mpv's update callback *before* freeing the context, so a
/// callback cannot land on a freed pointer, and frees the GL objects while the
/// caller still has the context current. The caller is responsible for making
/// the GL context current around both creation and drop — see `video_surface`.
///
/// This is DR-184 on Android restated: a surface outliving its player. The spike
/// had no defence at all and saw one unexplained SIGSEGV in a decoder thread.
pub struct MpvRenderContext {
ctx: *mut libmpv_sys::mpv_render_context,
gl: Gl,
target: Option<Target>,
}
// The render context is driven only from the GTK main thread; the update
// callback merely schedules a redraw and touches nothing here.
unsafe impl Send for MpvRenderContext {}
impl MpvRenderContext {
/// Create a render context over an existing mpv handle.
///
/// The GL context must already be current on this thread.
///
/// TRACES: UR-080 | DR-231, IR-033
pub unsafe fn new(mpv: *mut libmpv_sys::mpv_handle) -> Option<Self> {
let gl = Gl::load()?;
let mut init = libmpv_sys::mpv_opengl_init_params {
get_proc_address: Some(get_proc_address),
get_proc_address_ctx: ptr::null_mut(),
};
let mut api_type = CString::new("opengl").ok()?;
// Advanced control is deliberately OFF.
//
// With it on, mpv expects the client to drive rendering to a stricter
// contract than a GTK draw handler can promise — it will wait on us, and
// if we in turn wait on its update callback, neither side proceeds. That
// deadlock presents as a file that loads, renders one frame, and then
// sits there with no audio and a spinner.
//
// Off, mpv is tolerant of being rendered on the toolkit's schedule,
// which is what the frame clock gives us.
let mut advanced: c_int = 0;
let mut params = [
libmpv_sys::mpv_render_param {
type_: libmpv_sys::mpv_render_param_type_MPV_RENDER_PARAM_API_TYPE,
data: api_type.as_ptr() as *mut c_void,
},
libmpv_sys::mpv_render_param {
type_: libmpv_sys::mpv_render_param_type_MPV_RENDER_PARAM_OPENGL_INIT_PARAMS,
data: &mut init as *mut _ as *mut c_void,
},
libmpv_sys::mpv_render_param {
type_: libmpv_sys::mpv_render_param_type_MPV_RENDER_PARAM_ADVANCED_CONTROL,
data: &mut advanced as *mut _ as *mut c_void,
},
libmpv_sys::mpv_render_param {
type_: 0,
data: ptr::null_mut(),
},
];
let mut ctx: *mut libmpv_sys::mpv_render_context = ptr::null_mut();
let rc = libmpv_sys::mpv_render_context_create(&mut ctx, mpv, params.as_mut_ptr());
// Keep the CString alive until after the call.
let _ = &mut api_type;
if rc < 0 || ctx.is_null() {
error!("[MpvRender] mpv_render_context_create failed: {rc}");
return None;
}
info!("[MpvRender] render context created");
Some(MpvRenderContext {
ctx,
gl,
target: None,
})
}
/// Ask to be told when a new frame is ready.
///
/// Paired with [`report_swap`](Self::report_swap): without both, mpv has
/// nothing to time against. The symptom is misleading — playback looks fine
/// in a window and judders at fullscreen, which reads as a compositing or
/// GPU limit and is neither (DR-233).
///
/// TRACES: UR-080 | DR-233
pub unsafe fn set_update_callback(
&mut self,
callback: libmpv_sys::mpv_render_update_fn,
ctx: *mut c_void,
) {
libmpv_sys::mpv_render_context_set_update_callback(self.ctx, callback, ctx);
}
/// Whether mpv has a new frame waiting.
///
/// Asked of mpv directly rather than inferred from its update callback, and
/// that distinction is the whole of frame pacing here:
///
/// - Waiting only on the callback deadlocks — mpv will not progress until
/// the client renders, so if the client will not render until mpv says
/// so, neither moves. That presents as a file that loads, shows one
/// frame, and then sits silent.
/// - Rendering on *every* frame-clock tick regardless is the opposite
/// error: `report_swap` then claims a presentation far more often than
/// real frames exist, mpv has nothing coherent to time against, and
/// playback judders badly.
///
/// Polling is neither. It runs on the main thread, costs a single atomic
/// read inside mpv, and answers the only question that matters.
///
/// TRACES: UR-080 | DR-233
pub unsafe fn has_frame(&self) -> bool {
let flags = libmpv_sys::mpv_render_context_update(self.ctx);
(flags & libmpv_sys::mpv_render_update_flag_MPV_RENDER_UPDATE_FRAME as u64) != 0
}
/// Render the current frame at `width` x `height`, returning the texture id
/// holding it. The GL context must be current.
///
/// TRACES: UR-080 | DR-231
pub unsafe fn render(&mut self, width: i32, height: i32) -> Option<u32> {
if width <= 0 || height <= 0 {
return None;
}
self.ensure_target(width, height)?;
let target = self.target.as_ref()?;
let mut fbo = libmpv_sys::mpv_opengl_fbo {
fbo: target.fbo as c_int,
w: width as c_int,
h: height as c_int,
internal_format: 0,
};
// GTK's cairo surface has its origin at the top left; mpv defaults to
// OpenGL's bottom-left. Without this the picture is drawn upside down —
// which looks like a broken decode rather than a coordinate convention.
let mut flip: c_int = 1;
let mut params = [
libmpv_sys::mpv_render_param {
type_: libmpv_sys::mpv_render_param_type_MPV_RENDER_PARAM_OPENGL_FBO,
data: &mut fbo as *mut _ as *mut c_void,
},
libmpv_sys::mpv_render_param {
type_: libmpv_sys::mpv_render_param_type_MPV_RENDER_PARAM_FLIP_Y,
data: &mut flip as *mut _ as *mut c_void,
},
libmpv_sys::mpv_render_param {
type_: 0,
data: ptr::null_mut(),
},
];
let rc = libmpv_sys::mpv_render_context_render(self.ctx, params.as_mut_ptr());
if rc < 0 {
warn!("[MpvRender] render failed: {rc}");
return None;
}
Some(target.texture)
}
/// Tell mpv the frame reached the screen. See [`set_update_callback`].
///
/// TRACES: UR-080 | DR-233
pub unsafe fn report_swap(&self) {
libmpv_sys::mpv_render_context_report_swap(self.ctx);
}
/// Create or resize the framebuffer. Reused across frames — reallocating per
/// frame would churn GPU memory at the display rate.
unsafe fn ensure_target(&mut self, width: i32, height: i32) -> Option<()> {
if let Some(t) = &self.target {
if t.width == width && t.height == height {
return Some(());
}
}
self.drop_target();
let gl = &self.gl;
let mut texture: u32 = 0;
(gl.gen_textures)(1, &mut texture);
(gl.bind_texture)(GL_TEXTURE_2D, texture);
(gl.tex_image_2d)(
GL_TEXTURE_2D,
0,
GL_RGBA8,
width,
height,
0,
GL_RGBA,
GL_UNSIGNED_BYTE,
ptr::null(),
);
(gl.tex_parameteri)(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
(gl.tex_parameteri)(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
(gl.bind_texture)(GL_TEXTURE_2D, 0);
let mut fbo: u32 = 0;
(gl.gen_framebuffers)(1, &mut fbo);
(gl.bind_framebuffer)(GL_FRAMEBUFFER, fbo);
(gl.framebuffer_texture_2d)(
GL_FRAMEBUFFER,
GL_COLOR_ATTACHMENT0,
GL_TEXTURE_2D,
texture,
0,
);
let status = (gl.check_framebuffer_status)(GL_FRAMEBUFFER);
(gl.bind_framebuffer)(GL_FRAMEBUFFER, 0);
if status != GL_FRAMEBUFFER_COMPLETE {
error!("[MpvRender] framebuffer incomplete: 0x{status:x}");
(gl.delete_framebuffers)(1, &fbo);
(gl.delete_textures)(1, &texture);
return None;
}
self.target = Some(Target {
fbo,
texture,
width,
height,
});
Some(())
}
unsafe fn drop_target(&mut self) {
if let Some(t) = self.target.take() {
(self.gl.delete_framebuffers)(1, &t.fbo);
(self.gl.delete_textures)(1, &t.texture);
}
}
/// Free everything, with the GL context current.
///
/// Explicit rather than left to `Drop` because the ordering matters and the
/// caller is the only one that can guarantee the GL context is current. See
/// DR-232.
pub unsafe fn destroy(mut self) {
// Unregister first: a callback arriving after the free would be a use
// after free, and it is scheduled from mpv's own threads.
libmpv_sys::mpv_render_context_set_update_callback(self.ctx, None, ptr::null_mut());
self.drop_target();
libmpv_sys::mpv_render_context_free(self.ctx);
self.ctx = ptr::null_mut();
info!("[MpvRender] render context freed");
std::mem::forget(self);
}
}
impl Drop for MpvRenderContext {
fn drop(&mut self) {
if !self.ctx.is_null() {
// Reached only if `destroy` was not called — the GL context may not
// be current, so the GL objects are deliberately leaked rather than
// deleted against whatever context happens to be bound. Freeing the
// render context is still safe and is the part that matters.
warn!("[MpvRender] dropped without destroy(); GL objects leaked deliberately");
unsafe {
libmpv_sys::mpv_render_context_set_update_callback(self.ctx, None, ptr::null_mut());
libmpv_sys::mpv_render_context_free(self.ctx);
}
}
}
}