jellytau/docs/specs/linux-native-video-spike.md

Spec: Linux native video — bounded compositing spike

Status: Run 2026-08-21 — compositing works; G5 carries an open crash. The compositing claim it set out to test is falsified on Linux. See "Result". This file stays open until the implementation spec exists. ABR is resolved — the playlist carries one EXT-X-STREAM-INF, so finding 3 is false and there is no adaptation for mpv to lose. The remaining blocker is the unexplained SIGSEGV under G5, which is a lifetime problem, not a compositing one. Requirements: none allocated. This spike produces a decision record, not product code — same shape as playback-backend-unification.md, which is Accepted with no requirement ids of its own. Ids are allocated by the implementation spec that follows a green result. UX spec: n/a Supersedes / revises: re-opens finding 2 of playback-backend-unification.md on Linux only. Its findings 3, 4, 5 and 6 stand unchallenged and are not in scope here.

Destination on completion: 05-platform-backends.md — a "Native Video Compositing (Linux)" section alongside the existing Android one. The durable half is the mechanism and the two traps below; the gates and phases are disposable.

Summary

Test one falsifiable claim: a native video surface cannot be composited with a Tauri webview on Linux. The claim is load-bearing — it is why Linux video goes through an h264 HLS transcode into a WebKitGTK <video> element instead of decoding directly in the mpv instance we already run. The spike renders one mpv frame beneath the webview, on both X11 and Wayland, and stops. It ships no product code and flips no defaults.

A green result does not authorise native video on Linux; it authorises writing the spec that would.

Motivation

playback-backend-unification.md finding 2 concluded that native video cannot be composited with a Tauri webview, on evidence from tauri-plugin-libmpv's platform table, wry#284, tauri#6343, and a Tauri maintainer's 2024 statement that a GTK widget as a child X11 window is "a bit hacky and it is not possible on Wayland at all."

Two things have changed since that was written, and one thing was never tested.

1. The general claim has already been falsified on one platform — by us. Android now renders ExoPlayer video on a TextureView at index 0 behind a transparent Tauri WebView, with the Svelte controls drawn over it, on by default. See 05-platform-backends.md. That is exactly the composition finding 2 said was impossible, shipped. What survives of the finding is a narrower, WebKitGTK-specific claim — which is worth testing on its own terms rather than inheriting.

2. A Tauri app now ships Linux native mpv as an active platform. MaxVideoPlayer (354 commits) embeds libmpv via EGL + X11 child window / Wayland subsurface, with Linux and macOS active and Windows only planned — the inverse of the plugin matrix finding 2 sampled. Its existence does not prove our case works, but it does mean the Wayland half of the maintainer quote is out of date.

3. The render API was never tested. Every source in finding 2 describes foreign-window embedding: --wid, child windows, a second toplevel position-synced to a getBoundingClientRect() div. That is a different mechanism from mpv's render API, where we own the GL context and mpv draws into an FBO we hand it (mpv_render_context_create / mpv_render_context_render, with an upstream GTK example). Tauri v2 exposes WebviewWindow::gtk_window() and default_vbox(), so the target is a widget inside Tauri's own GTK tree — not a foreign window, not a second toplevel, and therefore not the thing that was found broken.

The prize is direct play: no h264 transcode, hardware decode, libass subtitles, and no server CPU burned on every Linux play.

The blocker a green spike does not clear

🔴 Read this before treating a green result as a green light.

Finding 3 of the unification spec stands: mpv has no adaptive bitrate. It delegates HLS to FFmpeg's demuxer, which picks one variant at open and never adapts. The webview path has real ABR via hls.js. Compositing is necessary for native video on Linux; it is not sufficient.

There is a plausible answer, and this spike exists partly to make it testable: ABR only matters on the transcode path. A direct-played file has no variant ladder to adapt between — the adaptation the server offers is the transcode. So "mpv when the stream is direct-play, HTML5 + hls.js when the server transcodes" would sidestep finding 3 rather than fight it, and it maps onto a decision Rust already makes when it builds the stream URL.

That is a hypothesis, not a conclusion. It is out of scope here. Record it in the spike's decision note so the follow-up spec starts from it.

Layer assignment

The spike introduces no product logic. The table below is the assignment the follow-up would inherit, written now so a green result cannot drift into frontend decisions during implementation.

Logic / responsibility	Layer	Why it belongs there
Which backend renders video on this platform (use_html5_element, supports_native_video)	Rust	Already there — get_player_status in commands/player/mod.rs computes it from a cfg!. The spike would widen that cfg!, not relocate the decision. The frontend already consumes it via createAdapter.
Whether this stream is direct-play or transcoded, and therefore whether mpv or hls.js renders it	Rust	Domain. It depends on Jellyfin's PlaybackInfo response, container/codec support, and the bitrate cap — all of which change when Jellyfin's API or our quality ladder changes. The frontend must never re-derive it from a URL shape.
Creating, sizing, and destroying the GL surface; the mpv render context	Rust	Owns the backend and the GTK window handle. There is no presentation decision in it.
Where controls, subtitles, and the mini-player sit above the video, and the letterbox/poster treatment	Frontend	Pure presentation; changes only if the UI is redesigned. Precisely the split the Android path already uses.
Reserving the video rectangle in layout and marking the shell transparent	Frontend	Presentation. nativeVideo.ts + the [data-native-video="active"] rule in app.css already do this for Android and are platform-agnostic.

Borderline row, stated with its tie-breaker: "is the surface currently attached?" reads like view state, but the Android work found that a surface left in the hierarchy outlives its player (DR-184). Attachment is backend lifecycle → Rust, with the frontend told about it, not asked.

Design

A throwaway branch. No merge to master except the decision note.

What gets built

One #[cfg(target_os = "linux")] experiment behind a feature flag, in a scratch binary or an ignored test — not in MpvBackend's constructor path:

1. From app.get_webview_window(...), take gtk_window() and default_vbox().
2. Reparent the webview into a gtk::Overlay: GLArea as the main child, the webview as the overlay child.
3. Set the webview background to fully transparent (wry does this when "transparent": true; verify it reaches webkit_web_view_set_background_color).
4. In the GLArea's render signal, drive mpv_render_context_render with MPV_RENDER_PARAM_OPENGL_FBO pointing at the FBO GTK bound for us.
5. Play one local file. Draw an opaque HTML element over the video area.

video = no and audio-display = no are set in mpv_backend.rs:135-141; the spike overrides them on its own Mpv handle rather than editing that path.

Bindings

The current pin is libmpv = { git = "…/libmpv-rs", branch = "master" } — the dead pin libmpv2-migration.md exists to replace. The render API lives in libmpv2-sys (mpv_render_context_render); the safe wrapper was only ever a PR against the old crate. Use libmpv2-sys raw FFI directly in the spike. Do not block the spike on the migration, and do not let the spike half-perform it — if the spike goes green the migration becomes a hard prerequisite of the implementation, which is the ordering windows-native-audio-backend.md already sits in.

IPC

None. The spike crosses no boundary. If it goes green, the follow-up changes only the value of the existing useHtml5Element / supportsNativeVideo fields — no new wire shapes, no bindings.ts regeneration.

Gates

Each is pass/fail with a named failure. Stop at the first red and write it up — a red result is a successful spike.

#	Question	Fails if
G1	Can a custom GTK widget join Tauri's widget tree and survive the window's lifetime?	default_vbox() is absent/unusable, or reparenting the webview breaks input or crashes.
G2	Does the webview still paint, with a transparent backdrop, over that widget?	The backdrop renders opaque black (wry#1540) or the webview stops repainting (tauri#12800). This is the highest-risk gate.
G3	Does mpv render a frame into our FBO?	The render context refuses GTK's context, or frames land in the wrong buffer.
G4	Does HTML drawn over the video area actually appear over it?	Video covers the controls — the exact failure wry#284 and tauri#6343 report. Without this, the whole thing is worthless: our controls, subtitles and mini-player all sit over the video.
G5	Does it survive resize, fullscreen, and SPA navigation away and back?	Flicker on resize, or a surface that outlives its route.
G6	Does it hold on both X11 and Wayland?	Either session backend fails. Wayland is the one the 2024 maintainer quote says is impossible — test it first, not last.

G6 is not a nice-to-have. A result that only holds on X11 is red for a project shipping to current desktops.

Time box

If G1–G4 are not all green, stop and write the result up. The value of this spike is a dated, method-specific answer — including "still no, and here is the mechanism" — not a working player.

Result (2026-08-21)

Run on GNOME, kernel 7.1.8, libmpv 2.5.0 (mpv 0.41.0), GTK 3.24.52, WebKitGTK 2.52.6, wry 0.53.5 — the versions src-tauri/Cargo.lock resolves. Spike source: a ~250-line standalone crate using wry + gtk + libmpv2-sys raw FFI, driving mpv's render API with an update callback, frame-gated repaints and report_swap.

Gate	Result	Observed mechanism
G1 widget in GTK tree	🟡 partial	GtkOverlay with GtkGLArea as main child and the wry webview as overlay child works, built directly. Tauri's own default_vbox() was not exercised — see below.
G2 webview paints transparently over it	✅ green	with_transparent(true) alone. No window-level transparency was used or needed.
G3 mpv renders into our FBO	✅ green	vo=libmpv + mpv_render_context_create with MPV_RENDER_PARAM_OPENGL_FBO into the FBO GTK binds.
G4 HTML over video	✅ green	Opaque panel and a translucent control bar both drew over moving video.
G5 resize / drag / fullscreen	🟡 green on appearance, suspect underneath	No flicker, gap or misalignment, and smooth once frame pacing was correct (trap 3). But the only crash observed came from the only session where fullscreen was exercised — see "What is still open".
G6 X11 and Wayland	✅ green	Identical on both; GDK_BACKEND flipped between runs.

Finding 2 of playback-backend-unification.md is false on Linux when tested by the render API rather than by foreign-window embedding. Wayland — the half the 2024 maintainer quote called impossible — is green.

Better than the gate asked for: the translucent bar composited alpha against the video, not merely opaque-over. Scrims, gradient fades and subtitle backdrops therefore work, which is most of how a player UI actually looks. mpv also painted the letterbox bars black on its own — the Android equivalent was a shipped defect (DR-194).

Three traps, each of which cost a debugging cycle

Carry these into the implementation; each produced a failure that looked like a platform limitation and was not.

1. LC_NUMERIC must be reset after gtk::init(), not before. mpv refuses to start under a non-C numeric locale. mpv_backend.rs already handles this, but it has no GTK init in front of it; on this path gtk::init() applies the user's locale afterwards and mpv_create returns null.
2. libepoxy exports GL entry points as data symbols. There is no glFoo function to resolve — there is epoxy_glFoo, a variable holding a lazily resolving function pointer. get_proc_address must return the pointer stored at that symbol; returning the symbol's own address makes mpv jump into non-executable data and take SIGSEGV/SEGV_ACCERR on the first GL call. The epoxy crate does this correctly but is unusable — its gl_generator dependency pulls a yanked xml-rs.
3. Frame pacing is not optional, and its symptom is misleading. Driving queue_render() off the widget's frame clock on every tick, without calling mpv_render_context_report_swap after each render, leaves mpv with nothing to time against. Playback looks fine in a window and judders at fullscreen — which reads as a compositing or GPU limit and is neither. The fix is to register mpv_render_context_set_update_callback, redraw only when it says a frame is ready, and report the swap afterwards. Fullscreen was smooth immediately once both were in place.

Hardware decode through the render API

Tested by asking mpv what it actually selected (hwdec-current), not what it was asked for. All three ran 20s clean at a steady 30 fps.

hwdec	hwdec-current	Note
vaapi	no	Did not engage on this box — silently fell back to software. vainfo is not installed, so the libva driver for the Iris Xe iGPU is likely absent. No render-API error; this looks like a missing driver package, not a compositing limit.
auto	nvdec-copy	Hardware decode does work through the render API, on the discrete RTX 3050. Copy-back rather than zero-copy interop.
no	no	Software. Clean baseline.

The load-bearing result is the middle row: hardware decode is compatible with mpv's render API, so the direct-play prize is real and not traded away for software decoding. Which decoder to prefer is an implementation question — on a hybrid Intel+NVIDIA laptop auto reached for the discrete GPU in copy-back mode, which is the least efficient hardware path. An implementation should evaluate zero-copy VA-API on the iGPU (after confirming the driver is installed) before accepting auto.

hwdec=auto-safe probes Vulkan video decode, which this GPU does not support. It logs two Failed setup for format vulkan / no frame! pairs at start-up and then settles on nvdec-copy — the same place auto lands. A first reading of these logs mistook the start-up pair for a per-frame flood; it is not. Every run, clean or crashed, contains exactly two. auto-safe is not implicated in anything.

What is still open

• The Tauri half of G1. The spike built its own GtkOverlay. The app must instead reach WebviewWindow::gtk_window() / default_vbox() and reparent Tauri's existing webview into an overlay. Low risk — the same widgets, one extra reparent — but unproven, and it is the only place Tauri-specific behaviour could still bite.

• ✅ ABR — resolved. Finding 3's premise is false. Finding 3 said mpv would regress streaming quality because "the webview path already has real ABR via hls.js". Three pieces of evidence in this repo suggested that is not true of the URLs we actually build:

  1. get_video_stream_url (repository/online.rs) requests a single rendition — one VideoBitrate, one MaxStreamingBitrate, one MaxHeight. Jellyfin transcodes to what it is asked for; it does not build a ladder.
  2. The frontend contains no level-handling code at all — no hls.levels, no LEVEL_SWITCH, no currentLevel. The abrEwma* options in VideoPlayer.svelte are default tuning with nothing to act on. hls.js is serving as an HLS demuxer (WebKitGTK cannot play HLS natively), not as an adaptation engine.
  3. That function's own comment describes a quality switch as rebuilding the URL — "every path that re-opens a stream (quality switch, transcoded seek, audio-track switch)". Manual selection by stream re-open is what you build when there is no adaptation, and mpv can do the same thing.

  The decisive test has now been run (2026-08-21, against the development server, Jellyfin 10.11.5):

  curl -s ".../Videos/<itemId>/master.m3u8?…&TranscodingProtocol=hls&…" \
    | grep -c EXT-X-STREAM-INF
  1
  

  One line. The playlist carries a single EXT-X-STREAM-INF plus an EXT-X-IMAGE-STREAM-INF trickplay entry, which is not a rendition. Jellyfin builds the master playlist from the rendition the request asked for; it does not publish a ladder. So there is no ABR to lose, and this blocker is closed — hls.js is serving as an HLS demuxer, exactly as (2) above supposed, and mpv gives up nothing by replacing it.

  Recorded as DR-229 (Won't Do) rather than deleted, because it is a measurement: a server that does publish a ladder would change the answer, and the re-negotiation path is the hook that work would build on.

  The direct-play path now exists. It did not when this spike was written — every video play went through the HLS transcode endpoint. Backend-owned stream selection (DR-225 … DR-230) built it: Rust negotiates direct play / direct stream / transcode and hands every backend one StreamSelection carrying the URL, the transport and the chosen rendition. That is the contract this implementation consumes — mpv is a consumer of a decision already made, not a place to re-derive it.

  It also sizes the prize precisely. Measured over the same server, 40 items through a real negotiation per profile:

  Profile	Direct play
  Linux / WebKitGTK — h264 only, 2ch	7%
  Android / ExoPlayer — h264,hevc,vp8,vp9,av1,mpeg4 + ac3,eac3, 6ch	85%

  The 85% is a ceiling, not a shipped result — it was measured with a profile containing ac3,eac3, which the Android device later used for verification does not support.

  The library sampled is ~80% hevc. Linux sits at 7% solely because the WebKitGTK profile can only claim h264 — not because of anything about the server or the negotiation. mpv decodes hevc, so widening the Linux device profile once mpv renders the picture is what converts that 7% toward the Android figure. That conversion is the actual product of this work; the compositing proven above is the mechanism that permits it.

• 🔴 One unexplained SIGSEGV. A ~180s run died in a decoder thread (libavcodec -> av_log -> libmpv's log handler -> libc). No Tauri, wry, WebKitGTK, GTK or GL frame appears anywhere in the stack, so the fault is on the mpv/ffmpeg side of the process rather than in the compositing seam.

  Three hypotheses were tested and none reproduced it:

  Hypothesis	Test	Result
  hwdec=auto-safe's Vulkan failures	300s soak on auto-safe	Survived. Also based on a misreading — the failures are 2 per run at start-up, not per-frame. Dead.
  Fullscreen transitions recreating the GL context under mpv's render context	240s soak, ~120 automated transitions	Survived, no core dumped.
  Continuous resize thrashing the GL framebuffer	240s soak, ~2000 resizes	Survived, no core dumped.

  The crash is therefore unexplained. It was observed exactly once, in the only session a human interacted with, and did not recur in ~13 minutes of targeted stress across the three most plausible causes. It is recorded here rather than dismissed precisely because nothing explains it: an intermittent fault that nobody can reproduce is worse to inherit than a deterministic one, not better.

  The underlying concern stands regardless of which test eventually reproduces it. A SIGSEGV in an unrelated thread is characteristic of memory corruption, and this spike never calls mpv_render_context_free and never tears down on unrealize — it has no defence against the GL context being recreated beneath the render context. That is DR-184 on Android restated: a surface outliving its player. An implementation must bind the two lifetimes together whether or not this particular crash is ever explained.

  Therefore G5 is recorded green on appearance only, and this crash is the single largest piece of unfinished business in the spike. Do not read the green gates above as "safe to build on" until it is explained or a long soak clears it.

• Long-run stability, seeking, track switching, HDR, and multi-window were not exercised at all.

Out of scope

• Any change to the shipping Linux video path. experimentalNativeVideo in adapters/index.ts is a suppressor, never a promoter; the spike must not change that.
• Adaptive bitrate. See "The blocker a green spike does not clear".
• Windows and macOS — different mechanisms, and Windows is the easier case, not the endangered one. See below.
• Android. Already shipped; it is the precedent, not the target.
• Crossfade, the libmpv2 migration, and the audio-parity work.

Why Windows is unaffected, and cheaper

Nothing here can regress Windows. use_html5_element is already a per-platform cfg! in get_player_status — Android native, everything else HTML5 — so divergent video paths are the existing design rather than something this introduces. Windows keeps <video> + hls.js whatever this spike returns.

The mechanism does not port: default_vbox(), GtkOverlay and GtkGLArea are GTK3/WebKitGTK concepts. But the question is already answered more favourably there. Both mpv plugins list Windows as fully tested and Linux as broken, because WebView2 honours a transparent background — the "native surface beneath a transparent webview" approach that fails on WebKitGTK is the one that works on Windows. That asymmetry is why windows-native-audio-backend.md can call Windows "the cleanest available win".

Windows' cost is packaging, not compositing: the build cross-compiles with MSVC + cargo-xwin, so libmpv arrives as a bundled prebuilt DLL (the ⚠️ in finding 5's comparison table). That cost is already committed for audio. Once the DLL ships to replace WebviewAudioBackend, Windows video is largely a follow-on.

Sequencing, if native video is ever pursued on both:

1. libmpv2-migration.md — prerequisite for either.
2. windows-native-audio-backend.md — already specced; lands the DLL and a real Windows backend.
3. Windows native video — cheap once 2 exists, and does not need this spike.
4. Linux native video — needs this spike, and runs independently of 1–3.

Does this add a backend?

No — and the trajectory is convergence, not proliferation.

create_player_backend in lib.rs already selects between four PlayerBackend impls by cfg!: MpvBackend (Linux), ExoPlayerBackend (Android), WebviewAudioBackend (Windows and anything else), and NullBackend as the graceful-init fallback. The HTML5 video path is not among them — it is a frontend adapter reporting through player_report_*, not a PlayerBackend.

This spike adds none of these. MpvBackend already exists and already runs on Linux; it merely sets video = no at construction. Giving it video widens an existing backend rather than introducing an engine.

Following the sequence above, the count goes down: replacing WebviewAudioBackend with mpv on Windows leaves two native engines — mpv (Linux + Windows) and ExoPlayer (Android) — with native video riding on both.

Two is the floor, for a reason worth stating so nobody re-litigates it: Android cannot drop ExoPlayer even if libmpv runs there, because the foreground service, MediaSessionCompat and lockscreen control are built on it (finding 7 puts the cost at that rewrite, not at the bindings). The HTML5 path does not go away either — it is the transcode/ABR route and the fallback.

The trait surface converges too: ExoPlayerBackend already implements the video-surface lifecycle for Android native compositing, so teaching MpvBackend video follows a path already walked rather than opening a second one.

• Adopting tauri-plugin-libmpv or tauri-plugin-mpv as dependencies. Both report Linux window embedding as not working and are small projects (20 and ~70 commits); read them, do not depend on them.

Acceptance criteria

The deliverable is a decision, not a feature.

• Each of G1–G6 recorded green/red with the observed mechanism, not just the verdict.
• X11 and Wayland results reported separately, each naming the compositor and WebKitGTK version tested.
• The direct-play/transcode ABR hypothesis recorded as open, with whatever the spike learned about it.
• docs/specs/README.md updated — this spec listed, and its row moved or deleted per the result.
• The Linux claim in the createAdapter doc comment (adapters/index.ts:12-13) corrected either way: if red, cite this spike instead of asserting it; if green, it is wrong and must be rewritten.
• On red: finding 2 of playback-backend-unification.md gains a dated note naming the render-API method as also tested, and this file is deleted. The verdict lives in the design-authority spec, not in a second file that contradicts nothing.
• On green: an implementation spec exists, allocating ids from UR-077 / IR-033 / DR-216 (re-check requirements.md — the README's "next free DR-215" is stale, DR-215 landed), and it must answer ABR before being accepted.
• No spike code on master. If any lands, the standard gates apply: bun run check, bun run test, bun run check:boundary, cargo fmt, cargo clippy, bun run test:rust.

Testing

No automated tests. A compositing result is a visual, per-session-backend observation and cannot be asserted in cargo test or vitest — pretending otherwise would produce a test that passes on a headless runner and tells us nothing.

Capture a screenshot per gate. G4 specifically: an opaque HTML element over the video area, photographed showing the video behind it.

If it goes green, the implementation spec inherits the testable surface the Android work already established — nativeVideoLayers.test.ts asserts the app.css selector list and the data-native-video contract, and both are platform-agnostic.

TRACES

None. No requirement-implementing code is produced. The implementation spec that follows a green result allocates from DR-216 and tags there.

Notes for the implementer

• Read playback-backend-unification.md first, in full. This spike disputes exactly one of its six findings, on one platform, by one method it did not try. Everything else in it is still binding — particularly finding 3.
• Test Wayland first. It is the gate most likely to be red and the one that makes the rest moot.
• The frontend plumbing already exists from the Android work: createAdapter, NativePlayerAdapter, nativeVideo.ts, videoSurface.ts, and the [data-native-video="active"] rule. A green spike is far cheaper to implement than it would have been a year ago — which is itself part of why the question is worth re-asking.
• The Android record in 05-platform-backends.md lists six shipped defects from getting this right on one platform. Expect the Linux equivalents (the surface outliving its player, the shell painting over it, unpainted letterbox bars) rather than rediscovering them.
• A parallel Claude session may be active in this repo — git diff before "repairing" unexpected changes.