jellytau/docs/specs/desktop-native-video.md

Spec: Desktop native video — mpv renders the picture, everywhere

Status: Proposed Requirements: UR-080 (new) → DR-231 … DR-237 (new); IR-033 (new) UX spec: n/a — nothing about the player's appearance changes. What changes is what is behind the controls. Supersedes / revises: consumes and closes linux-native-video-spike.md, whose gates authorised exactly this spec and nothing more. Settles finding 2 of playback-backend-unification.md on the desktop; finding 3 was already settled by DR-229. Absorbs the video half of what windows-native-audio-backend.md leaves open. Depends on: backend-owned stream selection (DR-225 … DR-230), the branch below this one. mpv is a consumer of StreamSelection, never a second place to decide what to play.

Destination on completion: 05-platform-backends.md — a "Native Video Compositing (Desktop)" section beside the existing Android one, which this mirrors; and 01-rust-backend.md — the device profile becomes renderer-dependent, beside the stream-selection section. The spike is deleted in the same commit, its three traps and its hardware-decode table folded in; they are the durable half.

Summary

mpv decodes and draws video on every desktop platform, composited beneath the transparent webview, exactly as Android already does with ExoPlayer. The HTML5 <video> path and hls.js are then deleted, not merely bypassed.

The user-visible change is that most video stops being re-encoded by the server before it can be watched. The change for whoever maintains this is that video goes from three renderers to two.

Motivation

The transcode is a decoder constraint, not a rendering one

Desktop video goes through an h264 HLS transcode because the picture is drawn by a WebKitGTK <video> element, and that element decodes little else. The device profile therefore claims h264 alone. That is not a statement about the machine — the same machine runs mpv, which decodes essentially everything in the library — it is a statement about which widget is holding the frame.

DR-228 made the cost measurable. Over 40 items negotiated against the development server:

Profile	Direct play
Desktop / WebKitGTK — h264 only, 2ch	7%
Android / ExoPlayer — h264,hevc,vp8,vp9,av1,mpeg4 + ac3,eac3, 6ch	85%

The sampled library is ~80% hevc. Those rows differ only by which component decodes.

Moving the picture to mpv is what lets the desktop row claim what the machine can actually do, and that — not the compositing — is the product.

The 85% is a ceiling, not a shipped result. It was measured with a profile containing ac3,eac3. The Android device later used for verification reports neither in its MediaCodecList — no Dolby licence, normal for a tablet — so eac3 content, about a third of the sampled library, correctly transcodes there. Realising any of this depends on DR-234, deriving the profile from the renderer rather than from the platform, which is why that requirement is load-bearing and not tidy-up.

One desktop video path, not two

This is why the spec covers Windows rather than stopping at Linux.

Today video has three renderers: ExoPlayer, the WebKitGTK <video> element, and (on Android, via the opt-out) that same element again. A Linux-only version of this work would make it four, permanently: mpv on Linux, HTML5 on Windows, ExoPlayer on Android, plus hls.js underneath the HTML5 one. Every seek strategy, every track switch, every quality change, every lifecycle bug would then have one more place to be got right — and the HTML5 path would survive indefinitely because something would still need it.

Finishing the job removes that: mpv on desktop, ExoPlayer on Android, and hls.js, html5Adapter.ts, videoLoaderFor and the webview video element all go. The maintenance win is the reason Windows is in this spec and not in a follow-up that never gets written.

Three blockers are gone

1. Compositing works, including Wayland. The spike ran all six gates; the 2024 "not possible on Wayland at all" claim is out of date when the render API is used instead of foreign-window embedding.
2. There is no ABR to lose. DR-229: the server's master playlist carries one EXT-X-STREAM-INF. hls.js was demuxing, not adapting.
3. A direct-play path exists. It did not when the spike was written. DR-228 built it; DR-230 proved the contract is player-agnostic.

And on Windows specifically, tauri-plugin-libmpv lists Windows as its fully tested platform — the inverse of the Linux situation the spike had to disprove. The embedding difficulty was always WebKitGTK-specific.

Layer assignment

Logic / responsibility	Layer	Why it belongs there
Which codecs this device can decode	Rust	Domain: it is the input to Jellyfin's PlaybackInfo negotiation. It stops being a property of the platform and becomes a property of the renderer in use — see "The structural change".
Which backend renders video	Rust	Rust already owns this (use_html5_element / VideoBackend). It stops being a cfg! constant and becomes a runtime fact.
What stream to play (direct / remux / transcode, transport, ceiling)	Rust — already decided	DR-225. mpv consumes StreamSelection. Re-deriving any of it in a new backend would be the defect DR-225 exists to remove, restated.
Creating the GL surface, reparenting the webview, owning the render context	Rust (platform layer)	Native window and GL-context lifetime. Not presentation, and not expressible above the IPC boundary at all.
Render-context ↔ GL-context lifetime binding	Rust	A correctness invariant over native resources. DR-232.
Frame pacing (update callback, report_swap)	Rust	Timing against the compositor; mpv's own contract.
Hardware-decode selection	Rust	A capability question about the machine, answered from what mpv reports it actually selected.
Z-order of controls over video, overlay chrome, letterbox colour	Frontend / mpv	Presentation. Controls already draw over a transparent webview on Android; mpv paints its own letterbox bars (better than the Android equivalent, which shipped DR-194 as a defect).
Whether the surface is visible right now	Frontend	nativeVideoActive already exists and toggles data-native-video. Unchanged.

The structural change

Everything above is routine except one row, and it carries the whole benefit.

video_codecs in build_device_profile is a compile-time constant per platform:

#[cfg(all(not(target_os = "android"), target_os = "linux"))]
let (video_codecs, audio_codecs) = ("h264".to_string(), "aac,mp3,opus,…");

That is correct only while a build has exactly one video renderer. It must be derived from which renderer will decode this stream, which is runtime state.

It looks like configuration and is not: it is the input that decides whether the server re-encodes, it changes when Jellyfin's API or our renderer changes, and getting it wrong fails silently — a claimed codec the renderer cannot decode is a black picture or silence, which is DR-148 and DR-228's audio override already.

Write this against "the active video renderer", never cfg!(target_os). It is the single piece that must not be Linux-shaped, because phase 2 reuses it unchanged.

Design

Backend and compositing (DR-231, IR-033)

An MpvVideoBackend beside the existing MpvBackend (audio). The mpv side — render context, FBO, update callback, hwdec — is shared; only the surface differs per platform:

Platform	Surface	Status
Linux (X11 + Wayland)	gdk_cairo_draw_from_gl() in the default vbox's draw handler, over a GdkGLContext on its GdkWindow. No reparenting — see below	Render path proven by the spike; the overlay approach it used is rejected
Windows	Native HWND child beneath a transparent WebView2	Phase 2

vo=libmpv plus mpv_render_context_create with MPV_RENDER_PARAM_OPENGL_FBO. Webview transparency via with_transparent(true) — no window-level transparency; the spike showed it is neither used nor needed.

G1's untested half failed, and the design changed because of it.

Reparenting Tauri's webview into a GtkOverlay attaches cleanly and then aborts the process on the first click. tauri-runtime-wry connects a button-press handler to the webview that walks a hard-coded path:

webview.parent()            // "This one should be GtkBox"
       .parent()            // ...and this one the GtkWindow
       .downcast::<gtk::Window>().unwrap()

An overlay makes that chain webview → GtkOverlay → GtkBox, the downcast fails, and the panic is non-unwinding so it kills the app. Nothing in configuration avoids it: on Linux attach_resize_handler is called unconditionally (the Windows equivalent is guarded by is_decorated()), and the decoration check that would make the handler inert runs after the unwrap.

So the webview is not moved at all. mpv draws into the default vbox's own draw handler instead, via gdk_cairo_draw_from_gl() over a GdkGLContext created on that widget's GdkWindow. GTK3 draws a container before its children, so the webview composites on top for free — the same z-order the overlay was for, without touching the widget tree Tauri walks.

That is strictly better than the overlay it replaces: no reparent, no extra widget, and the arrangement cannot be broken by a Tauri upgrade that assumes its own layout. It is also why "the surface attached successfully" is not the gate — a click is.

Three traps from the spike, each of which cost a debugging cycle and each of which looks like a platform limitation and is not:

1. LC_NUMERIC must be reset after gtk::init(). mpv refuses to start under a non-C numeric locale. mpv_backend.rs already handles this but has no GTK init in front of it; here gtk::init() applies the user's locale afterwards and mpv_create returns null.
2. libepoxy exports GL entry points as data symbols. There is no glFoo function — there is epoxy_glFoo, a variable holding a lazily-resolving pointer. get_proc_address must return the pointer stored at that symbol; returning the symbol's own address makes mpv jump into non-executable data and take SIGSEGV on the first GL call. The epoxy crate does this correctly but is unusable — its gl_generator dependency pulls a yanked xml-rs.
3. Frame pacing is not optional and its symptom misleads. See DR-233.

Render-context lifetime (DR-232) — the crash defence

The spike's one unexplained SIGSEGV landed in a decoder thread with no Tauri, GTK or GL frame in the stack, and three plausible causes failed to reproduce it across ~13 minutes of targeted stress.

What is not unexplained is that the spike had no defence: it never calls mpv_render_context_free and never tears down on unrealize, so nothing stopped the GL context being recreated beneath the render context. That is DR-184 on Android restated — a surface outliving its player.

Built as a requirement in its own right, not as a fix for a crash we cannot yet reproduce:

• Render context created on realize, freed on unrealize, same thread, before the GL context goes away.
• The update callback is unregistered before the context is freed, so a callback cannot land on a freed context.
• Playback teardown and surface teardown are ordered, not racing.

If the crash recurs after this, it is a different bug and the likeliest cause is out of the search space. If it does not, we needed this anyway.

Frame pacing (DR-233)

Register mpv_render_context_set_update_callback; redraw only when it reports a frame ready; call mpv_render_context_report_swap after each render.

Recorded because the failure mode is a trap: driving queue_render() off the frame clock every tick without reporting the swap leaves mpv nothing to time against. It looks fine in a window and judders at fullscreen, which reads as a compositing or GPU limit and is neither.

Renderer-dependent device profile (DR-234)

build_device_profile takes the active video renderer and derives the codec lists from it:

Renderer	Video codecs	Audio (video direct play)	Channels
mpv (desktop native)	h264,hevc,vp8,vp9,av1,mpeg4	platform list incl. ac3,eac3 where the sink can voice it	from the audio route
WebKitGTK <video>	h264	webview-decodable set only	2
ExoPlayer (Android)	unchanged	unchanged	unchanged

The existing video_audio_codecs() narrowing exists because the webview decodes a narrower audio set than the platform. With mpv decoding, that no longer applies to the video path — but the multichannel bound still does, since a 5.1 track direct-played into a 2-channel sink is silence or inaudible dialogue. Both constraints stay, sourced from the renderer rather than assumed.

This is what converts the 7% figure upward (toward, not necessarily to, the 85% ceiling — see the caveat above), and it is also the change most able to break playback silently — so it lands after compositing is proven, covered by the DR-228 override tests.

Deleting the webview video path (DR-235)

get_player_status stops reporting use_html5_element: true on desktop; supports_native_video becomes true there.

Deletion is staged, because a path cannot be removed while a shipped platform still needs it:

Phase	Linux	Windows	HTML5 video path
1	mpv	HTML5	alive — Windows needs it
2	mpv	mpv	alive but unreached
3	mpv	mpv	deleted, with hls.js

Phase 3 is a real phase with its own acceptance criterion, not a "later". The whole maintenance argument for including Windows collapses if the fork survives.

Android keeps ExoPlayer and keeps the webview as its documented opt-out; the <audio> element and the background-audio handoff are untouched throughout.

What happens when mpv fails to initialise. With no HTML5 path there is no silent fallback, and inventing one resurrects what we deleted. The graceful-backend-init principle applies as written: fall back to the no-op backend, emit backend-init-failed, and surface a real error rather than a black rectangle. An honest failure beats a hidden downgrade to the transcode we are trying to stop paying for.

Hardware decode (DR-236)

The spike established the load-bearing fact: hardware decode works through the render API (hwdec-current reported nvdec-copy on the discrete GPU), so the direct-play prize is not traded for software decoding.

Policy is decided from what mpv reports it selected, never from what it was asked for:

• Prefer zero-copy VA-API on the integrated GPU where the driver is present.
• auto reached for the discrete GPU in copy-back mode on a hybrid Intel+NVIDIA laptop — the least efficient hardware path — so auto is a fallback, not the default.
• vaapi silently fell back to software on the spike box because vainfo was absent. A missing driver must be detected and logged, not mistaken for a compositing limit.
• Log hwdec-current at start-up; knowing what was actually chosen is the whole diagnostic value.

Windows: what phase 2 actually costs (DR-237)

Not hidden, because it is the part most likely to be underestimated:

• The surface is different code. WebView2 in an HWND, not GTK. A transparent WebView2 over a native child window is a solved arrangement, but DR-231's Linux surface does not transfer. Everything else does.
• libmpv is currently a Linux-only dependency, and Windows is cross-compiled from Linux via x86_64-pc-windows-msvc + cargo-xwin. Phase 2 must source a Windows libmpv (DLL + import library) into that cross-build and ship the DLL in the NSIS bundle.
• LGPL obligations follow the DLL. DR-216 already records them for Linux: keep the linkage dynamic, ship libmpv's licence text with any bundle carrying it. The Windows bundle inherits both.
• bun run test:rust and CI must still build. Per the CI rule, any tool this needs goes into the builder image and is pushed — never installed at job time.

Windows also gains a native audio decoder as a side effect, which is what windows-native-audio-backend.md wants and cannot currently have. If that spec lands first, phase 2 inherits its build work and shrinks to the surface.

Out of scope

• Android. Unchanged in every respect.
• macOS. Not a shipped target. If it becomes one it joins phase 2's shape.
• Audio backends. mpv already plays audio on Linux; this adds a video renderer beside it. Windows audio is its own spec.
• HDR, tone mapping, multi-window. Not exercised by the spike at all.
• Re-deciding what stream to play. DR-225 owns that. If this spec finds itself choosing a URL, something has gone wrong.

Acceptance criteria

Phase 1 — Linux

• Tauri's own webview reparents into the overlay (the untested half of G1), on X11 and Wayland.
• Video plays, seeks and switches audio track in mpv, with the Svelte controls composited over it and alpha blending intact.
• The render context is freed on unrealize and the update callback unregistered before the free; a test demonstrates the ordering.
• A direct-play negotiation returns DirectPlay for an hevc source that today returns Transcode, and it plays.
• Direct-play rate over the same 40-item sample rises from 7% toward the Android figure. Record the number.
• mpv init failure emits backend-init-failed and surfaces an error rather than falling back to a transcode.
• hwdec-current is logged and is not copy-back where zero-copy is available.
• A soak covering seek, track switch and fullscreen runs clean for an agreed duration. The spike's SIGSEGV is why this is a criterion.

Phase 2 — Windows

• libmpv links in the cargo-xwin cross-build; the DLL and its licence ship in the NSIS bundle; any new tool lives in the builder image, not in a CI step.
• Video plays composited under a transparent WebView2.
• The device profile, lifetime and hwdec code are reused, not reimplemented — a reviewer confirms no cfg!(target_os = "linux") guards them.

Phase 3 — deletion

• use_html5_element is false on every desktop platform.
• hls.js is gone from package.json; html5Adapter.ts, videoLoaderFor and the <video> element are deleted; Android's opt-out and the background-audio <audio> path still work.

Throughout

• bun run check, bun run test, bun run format:check, bun run lint pass.
• cargo fmt clean, cargo clippy -D warnings clean, bun run test:rust passes.
• bun run check:boundary passes, and a reviewer confirms no stream decision was reconstructed in the new backend.
• bindings.ts regenerated from Rust.
• bun run traces:validate passes; coverage stays ≥ the CI ratchet.
• The spike and this spec are folded into 05-platform-backends.md and both deleted in the same commit.

Testing

• Rust, pure: the device profile per renderer — mpv claims hevc, the webview does not, the multichannel bound survives both. The DR-234 table as a table-driven test.
• Rust, pure: PlaybackInfo fixtures that transcode under the webview profile and direct-play under the mpv profile — the direct-play conversion as a unit test, not only as a measurement.
• Rust: teardown ordering — callback unregistered before context freed, freed before GL context destroyed. Structure it so the ordering is assertable without a live GL context.
• Frontend: no desktop path selects an HTML5 video adapter. After phase 3, the adapter does not exist and the test goes with it.
• Manual / soak: the criterion above. The spike's automated fullscreen and resize soaks are reusable and already written.

TRACES

Piece	Tag
mpv video backend + compositing	UR-080 | DR-231, IR-033
Render-context lifetime binding	UR-080 | DR-232
Frame pacing	UR-080 | DR-233
Renderer-dependent device profile	UR-080, UR-070 | DR-234
Webview video path removed	UR-080 | DR-235
Hardware-decode policy	UR-080 | DR-236
Windows surface + cross-build	UR-080 | DR-237

Notes for the implementer

• Read the spike before writing a line. Its three traps and its hardware-decode table are the most valuable things in this directory, and each cost a debugging cycle to find.
• mpv consumes StreamSelection; it does not decide. The transport is on the queue item (DR-230). If you are parsing a URL, stop.
• Guard nothing on cfg!(target_os = "linux") that phase 2 will need. That is the one avoidable mistake here.
• The Android backend is the reference for the shape of this — transparent webview over a native surface at index 0. Read 05-platform-backends.md's Android section for what shipped and what its defects were (DR-184 surface lifetime, DR-194 letterbox).
• Do not call sync/blocking APIs from mpv event callbacks that can re-enter the player or hold a lock. The existing deadlock gotchas apply.
• A parallel Claude session may be active in this repo — git diff before "repairing" unexpected changes.
• This branch is stacked on backend-owned stream selection. Rebase when that merges rather than merging master into it.