dtourolle/jellytau
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases 39 Wiki Activity
Files
7a99751cdf3830d776dd6c19be32e789fa9ce142
jellytau/specs/desktop-native-video.html
T

622 lines
46 KiB
HTML
Raw Blame History

<!DOCTYPE HTML>
<html lang="en" class="navy" dir="ltr">
<head>
<!-- Book generated using mdBook -->
<meta charset="UTF-8">
<title>Desktop Native Video - JellyTau Documentation</title>
<!-- Custom HTML head -->
<meta name="description" content="Requirements, traceability, and architecture for the JellyTau Jellyfin client.">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="theme-color" content="#ffffff">
<link rel="icon" href="../favicon.svg">
<link rel="shortcut icon" href="../favicon.png">
<link rel="stylesheet" href="../css/variables.css">
<link rel="stylesheet" href="../css/general.css">
<link rel="stylesheet" href="../css/chrome.css">
<link rel="stylesheet" href="../css/print.css" media="print">
<!-- Fonts -->
<link rel="stylesheet" href="../FontAwesome/css/font-awesome.css">
<link rel="stylesheet" href="../fonts/fonts.css">
<!-- Highlight.js Stylesheets -->
<link rel="stylesheet" href="../highlight.css">
<link rel="stylesheet" href="../tomorrow-night.css">
<link rel="stylesheet" href="../ayu-highlight.css">
<!-- Custom theme stylesheets -->
</head>
<body class="sidebar-visible no-js">
<div id="body-container">
<!-- Provide site root to javascript -->
<script>
var path_to_root = "../";
var default_theme = window.matchMedia("(prefers-color-scheme: dark)").matches ? "navy" : "navy";
</script>
<!-- Work around some values being stored in localStorage wrapped in quotes -->
<script>
try {
var theme = localStorage.getItem('mdbook-theme');
var sidebar = localStorage.getItem('mdbook-sidebar');
if (theme.startsWith('"') && theme.endsWith('"')) {
localStorage.setItem('mdbook-theme', theme.slice(1, theme.length - 1));
}
if (sidebar.startsWith('"') && sidebar.endsWith('"')) {
localStorage.setItem('mdbook-sidebar', sidebar.slice(1, sidebar.length - 1));
}
} catch (e) { }
</script>
<!-- Set the theme before any content is loaded, prevents flash -->
<script>
var theme;
try { theme = localStorage.getItem('mdbook-theme'); } catch(e) { }
if (theme === null || theme === undefined) { theme = default_theme; }
var html = document.querySelector('html');
html.classList.remove('navy')
html.classList.add(theme);
var body = document.querySelector('body');
body.classList.remove('no-js')
body.classList.add('js');
</script>
<input type="checkbox" id="sidebar-toggle-anchor" class="hidden">
<!-- Hide / unhide sidebar before it is displayed -->
<script>
var body = document.querySelector('body');
var sidebar = null;
var sidebar_toggle = document.getElementById("sidebar-toggle-anchor");
if (document.body.clientWidth >= 1080) {
try { sidebar = localStorage.getItem('mdbook-sidebar'); } catch(e) { }
sidebar = sidebar || 'visible';
} else {
sidebar = 'hidden';
}
sidebar_toggle.checked = sidebar === 'visible';
body.classList.remove('sidebar-visible');
body.classList.add("sidebar-" + sidebar);
</script>
<nav id="sidebar" class="sidebar" aria-label="Table of contents">
<div class="sidebar-scrollbox">
<ol class="chapter"><li class="chapter-item expanded affix "><a href="../index.html">Introduction</a></li><li class="chapter-item expanded affix "><li class="part-title">Requirements & Traceability</li><li class="chapter-item expanded "><a href="../requirements.html"><strong aria-hidden="true">1.</strong> Requirements Specification</a></li><li class="chapter-item expanded "><a href="../traceability.html"><strong aria-hidden="true">2.</strong> Traceability Matrix</a></li><li class="chapter-item expanded "><a href="../traceability-ci.html"><strong aria-hidden="true">3.</strong> Traceability CI</a></li><li class="chapter-item expanded "><a href="../traces-quick-ref.html"><strong aria-hidden="true">4.</strong> Traces Quick Reference</a></li><li class="chapter-item expanded affix "><li class="part-title">Architecture</li><li class="chapter-item expanded "><a href="../architecture/index.html"><strong aria-hidden="true">5.</strong> Overview</a><a class="toggle"><div>❱</div></a></li><li><ol class="section"><li class="chapter-item "><a href="../architecture/01-rust-backend.html"><strong aria-hidden="true">5.1.</strong> Rust Backend</a></li><li class="chapter-item "><a href="../architecture/02-svelte-frontend.html"><strong aria-hidden="true">5.2.</strong> Svelte Frontend</a></li><li class="chapter-item "><a href="../architecture/03-data-flow.html"><strong aria-hidden="true">5.3.</strong> Data Flow</a></li><li class="chapter-item "><a href="../architecture/04-type-sync-and-threading.html"><strong aria-hidden="true">5.4.</strong> Type Sync & Threading</a></li><li class="chapter-item "><a href="../architecture/05-platform-backends.html"><strong aria-hidden="true">5.5.</strong> Platform Backends</a></li><li class="chapter-item "><a href="../architecture/06-downloads-and-offline.html"><strong aria-hidden="true">5.6.</strong> Downloads & Offline</a></li><li class="chapter-item "><a href="../architecture/07-connectivity.html"><strong aria-hidden="true">5.7.</strong> Connectivity</a></li><li class="chapter-item "><a href="../architecture/08-database-design.html"><strong aria-hidden="true">5.8.</strong> Database Design</a></li><li class="chapter-item "><a href="../architecture/09-security.html"><strong aria-hidden="true">5.9.</strong> Security</a></li></ol></li><li class="chapter-item expanded "><li class="part-title">UX</li><li class="chapter-item expanded "><a href="../ux-flows.html"><strong aria-hidden="true">6.</strong> UX Flows</a></li><li class="chapter-item expanded affix "><li class="part-title">Specs — Pending Work</li><li class="chapter-item expanded "><a href="../specs/index.html"><strong aria-hidden="true">7.</strong> Specs Index</a></li><li class="chapter-item expanded "><a href="../specs/SPEC-TEMPLATE.html"><strong aria-hidden="true">8.</strong> Spec Template</a></li><li class="chapter-item expanded "><a href="../specs/SPEC-REVIEW-CHECKLIST.html"><strong aria-hidden="true">9.</strong> Spec Review Checklist</a></li><li class="chapter-item expanded "><a href="../specs/playback-backend-unification.html"><strong aria-hidden="true">10.</strong> Playback Backend Unification</a></li><li class="chapter-item expanded "><a href="../specs/linux-native-video-spike.html"><strong aria-hidden="true">11.</strong> Linux Native Video Spike</a></li><li class="chapter-item expanded "><a href="../specs/backend-owned-stream-selection.html"><strong aria-hidden="true">12.</strong> Backend-Owned Stream Selection</a></li><li class="chapter-item expanded "><a href="../specs/player-facade-enforcement.html"><strong aria-hidden="true">13.</strong> Player Facade Enforcement</a></li><li class="chapter-item expanded "><a href="../specs/windows-native-audio-backend.html"><strong aria-hidden="true">14.</strong> Windows Native Audio Backend</a></li><li class="chapter-item expanded "><a href="../specs/libmpv2-migration.html"><strong aria-hidden="true">15.</strong> libmpv2 Migration</a></li><li class="chapter-item expanded "><a href="../specs/read-through-media-cache.html"><strong aria-hidden="true">16.</strong> Read-Through Media Cache</a></li><li class="chapter-item expanded "><a href="../specs/scoped-search.html"><strong aria-hidden="true">17.</strong> Scoped Search</a></li><li class="chapter-item expanded "><a href="../specs/scoped-search-boundary.html"><strong aria-hidden="true">18.</strong> Scoped Search Boundary</a></li><li class="chapter-item expanded "><a href="../specs/scoped-search-boundary-implementation.html"><strong aria-hidden="true">19.</strong> Scoped Search Boundary — Implementation</a></li><li class="chapter-item expanded "><a href="../specs/frontend-domain-model.html"><strong aria-hidden="true">20.</strong> Frontend Domain Model</a></li><li class="chapter-item expanded "><a href="../specs/desktop-native-video.html" class="active"><strong aria-hidden="true">21.</strong> Desktop Native Video</a></li><li class="chapter-item expanded "><a href="../specs/build-provenance.html"><strong aria-hidden="true">22.</strong> Build Provenance</a></li><li class="chapter-item expanded affix "><li class="part-title">Build & Release</li><li class="chapter-item expanded "><a href="../build/build-release.html"><strong aria-hidden="true">23.</strong> Build & Release</a></li><li class="chapter-item expanded "><a href="../release-checklist.html"><strong aria-hidden="true">24.</strong> Release Checklist</a></li><li class="chapter-item expanded "><a href="../native-player-verification.html"><strong aria-hidden="true">25.</strong> Native Player Verification</a></li><li class="chapter-item expanded "><a href="../build/build-desktop-packages.html"><strong aria-hidden="true">26.</strong> Desktop Packaging</a></li><li class="chapter-item expanded "><a href="../build/build-windows.html"><strong aria-hidden="true">27.</strong> Windows Build</a></li><li class="chapter-item expanded "><a href="../defect-windows.html"><strong aria-hidden="true">28.</strong> Defect Windows</a></li><li class="chapter-item expanded "><a href="../build/docker.html"><strong aria-hidden="true">29.</strong> Docker</a></li><li class="chapter-item expanded "><a href="../build/build-builder-image.html"><strong aria-hidden="true">30.</strong> Builder Image</a></li><li class="chapter-item expanded affix "><li class="spacer"></li><li class="chapter-item expanded affix "><a href="../api-redirect.html">Rust API Reference (rustdoc)</a></li></ol>
</div>
<div id="sidebar-resize-handle" class="sidebar-resize-handle">
<div class="sidebar-resize-indicator"></div>
</div>
</nav>
<!-- Track and set sidebar scroll position -->
<script>
var sidebarScrollbox = document.querySelector('#sidebar .sidebar-scrollbox');
sidebarScrollbox.addEventListener('click', function(e) {
if (e.target.tagName === 'A') {
sessionStorage.setItem('sidebar-scroll', sidebarScrollbox.scrollTop);
}
}, { passive: true });
var sidebarScrollTop = sessionStorage.getItem('sidebar-scroll');
sessionStorage.removeItem('sidebar-scroll');
if (sidebarScrollTop) {
// preserve sidebar scroll position when navigating via links within sidebar
sidebarScrollbox.scrollTop = sidebarScrollTop;
} else {
// scroll sidebar to current active section when navigating via "next/previous chapter" buttons
var activeSection = document.querySelector('#sidebar .active');
if (activeSection) {
activeSection.scrollIntoView({ block: 'center' });
}
}
</script>
<div id="page-wrapper" class="page-wrapper">
<div class="page">
<div id="menu-bar-hover-placeholder"></div>
<div id="menu-bar" class="menu-bar sticky">
<div class="left-buttons">
<label id="sidebar-toggle" class="icon-button" for="sidebar-toggle-anchor" title="Toggle Table of Contents" aria-label="Toggle Table of Contents" aria-controls="sidebar">
<i class="fa fa-bars"></i>
</label>
<button id="theme-toggle" class="icon-button" type="button" title="Change theme" aria-label="Change theme" aria-haspopup="true" aria-expanded="false" aria-controls="theme-list">
<i class="fa fa-paint-brush"></i>
</button>
<ul id="theme-list" class="theme-popup" aria-label="Themes" role="menu">
<li role="none"><button role="menuitem" class="theme" id="light">Light</button></li>
<li role="none"><button role="menuitem" class="theme" id="rust">Rust</button></li>
<li role="none"><button role="menuitem" class="theme" id="coal">Coal</button></li>
<li role="none"><button role="menuitem" class="theme" id="navy">Navy</button></li>
<li role="none"><button role="menuitem" class="theme" id="ayu">Ayu</button></li>
</ul>
<button id="search-toggle" class="icon-button" type="button" title="Search. (Shortkey: s)" aria-label="Toggle Searchbar" aria-expanded="false" aria-keyshortcuts="S" aria-controls="searchbar">
<i class="fa fa-search"></i>
</button>
</div>
<h1 class="menu-title">JellyTau Documentation</h1>
<div class="right-buttons">
<a href="../print.html" title="Print this book" aria-label="Print this book">
<i id="print-button" class="fa fa-print"></i>
</a>
<a href="https://gitea.tourolle.paris/dtourolle/jellytau" title="Git repository" aria-label="Git repository">
<i id="git-repository-button" class="fa fa-github"></i>
</a>
<a href="https://gitea.tourolle.paris/dtourolle/jellytau/_edit/master/docs/../docs/specs/desktop-native-video.md" title="Suggest an edit" aria-label="Suggest an edit">
<i id="git-edit-button" class="fa fa-edit"></i>
</a>
</div>
</div>
<div id="search-wrapper" class="hidden">
<form id="searchbar-outer" class="searchbar-outer">
<input type="search" id="searchbar" name="searchbar" placeholder="Search this book ..." aria-controls="searchresults-outer" aria-describedby="searchresults-header">
</form>
<div id="searchresults-outer" class="searchresults-outer hidden">
<div id="searchresults-header" class="searchresults-header"></div>
<ul id="searchresults">
</ul>
</div>
</div>
<!-- Apply ARIA attributes after the sidebar and the sidebar toggle button are added to the DOM -->
<script>
document.getElementById('sidebar-toggle').setAttribute('aria-expanded', sidebar === 'visible');
document.getElementById('sidebar').setAttribute('aria-hidden', sidebar !== 'visible');
Array.from(document.querySelectorAll('#sidebar a')).forEach(function(link) {
link.setAttribute('tabIndex', sidebar === 'visible' ? 0 : -1);
});
</script>
<div id="content" class="content">
<main>
<h1 id="spec-desktop-native-video--mpv-renders-the-picture-everywhere"><a class="header" href="#spec-desktop-native-video--mpv-renders-the-picture-everywhere">Spec: Desktop native video — mpv renders the picture, everywhere</a></h1>
<p><strong>Status:</strong> Proposed
<strong>Requirements:</strong> UR-080 (new) → DR-231 … DR-237 (new); IR-033 (new)
<strong>UX spec:</strong> n/a — nothing about the player's appearance changes. What changes is
what is behind the controls.
<strong>Supersedes / revises:</strong> consumes and closes
<a href="linux-native-video-spike.html">linux-native-video-spike.md</a>, whose gates
authorised exactly this spec and nothing more. Settles finding 2 of
<a href="playback-backend-unification.html">playback-backend-unification.md</a> on the
desktop; finding 3 was already settled by DR-229. Absorbs the video half of what
<a href="windows-native-audio-backend.html">windows-native-audio-backend.md</a> leaves open.
<strong>Depends on:</strong> backend-owned stream selection (DR-225 … DR-230), the branch
below this one. mpv is a <em>consumer</em> of <code>StreamSelection</code>, never a second place to
decide what to play.</p>
<p><strong>Destination on completion:</strong>
<a href="../architecture/05-platform-backends.html">05-platform-backends.md</a> — a "Native
Video Compositing (Desktop)" section beside the existing Android one, which this
mirrors; and <a href="../architecture/01-rust-backend.html">01-rust-backend.md</a> — the
device profile becomes renderer-dependent, beside the stream-selection section.
<strong>The spike is deleted in the same commit</strong>, its three traps and its
hardware-decode table folded in; they are the durable half.</p>
<h2 id="summary"><a class="header" href="#summary">Summary</a></h2>
<p>mpv decodes and draws video on <strong>every desktop platform</strong>, composited beneath the
transparent webview, exactly as Android already does with ExoPlayer. The HTML5
<code>&lt;video&gt;</code> path and hls.js are then <strong>deleted</strong>, not merely bypassed.</p>
<p>The user-visible change is that most video stops being re-encoded by the server
before it can be watched. The change for whoever maintains this is that video
goes from three renderers to two.</p>
<h2 id="motivation"><a class="header" href="#motivation">Motivation</a></h2>
<h3 id="the-transcode-is-a-decoder-constraint-not-a-rendering-one"><a class="header" href="#the-transcode-is-a-decoder-constraint-not-a-rendering-one">The transcode is a decoder constraint, not a rendering one</a></h3>
<p>Desktop video goes through an h264 HLS transcode because the picture is drawn by
a WebKitGTK <code>&lt;video&gt;</code> element, and that element decodes little else. The device
profile therefore claims <code>h264</code> alone. That is not a statement about the machine
— the same machine runs mpv, which decodes essentially everything in the library
— it is a statement about which widget is holding the frame.</p>
<p>DR-228 made the cost measurable. Over 40 items negotiated against the development
server:</p>
<div class="table-wrapper"><table><thead><tr><th>Profile</th><th>Direct play</th></tr></thead><tbody>
<tr><td>Desktop / WebKitGTK — <code>h264</code> only, 2ch</td><td><strong>7%</strong></td></tr>
<tr><td>Android / ExoPlayer — <code>h264,hevc,vp8,vp9,av1,mpeg4</code> + <code>ac3,eac3</code>, 6ch</td><td><strong>85%</strong></td></tr>
</tbody></table>
</div>
<p>The sampled library is ~80% hevc. <strong>Those rows differ only by which component
decodes.</strong></p>
<p>Moving the picture to mpv is what lets the desktop row claim what the machine
can actually do, and that — not the compositing — is the product.</p>
<blockquote>
<p><strong>The 85% is a ceiling, not a shipped result.</strong> It was measured with a profile
containing <code>ac3,eac3</code>. The Android device later used for verification reports
neither in its <code>MediaCodecList</code> — no Dolby licence, normal for a tablet — so
eac3 content, about a third of the sampled library, correctly transcodes there.
Realising any of this depends on DR-234, deriving the profile from the renderer
rather than from the platform, which is why that requirement is load-bearing
and not tidy-up.</p>
</blockquote>
<h3 id="one-desktop-video-path-not-two"><a class="header" href="#one-desktop-video-path-not-two">One desktop video path, not two</a></h3>
<p>This is why the spec covers Windows rather than stopping at Linux.</p>
<p>Today video has <strong>three</strong> renderers: ExoPlayer, the WebKitGTK <code>&lt;video&gt;</code> element,
and (on Android, via the opt-out) that same element again. A Linux-only version
of this work would make it four, permanently: mpv on Linux, HTML5 on Windows,
ExoPlayer on Android, plus hls.js underneath the HTML5 one. Every seek strategy,
every track switch, every quality change, every lifecycle bug would then have one
more place to be got right — and the HTML5 path would survive indefinitely
because <em>something</em> would still need it.</p>
<p>Finishing the job removes that: <strong>mpv on desktop, ExoPlayer on Android</strong>, and
<code>hls.js</code>, <code>html5Adapter.ts</code>, <code>videoLoaderFor</code> and the webview video element all
go. The maintenance win is the reason Windows is in this spec and not in a
follow-up that never gets written.</p>
<h3 id="three-blockers-are-gone"><a class="header" href="#three-blockers-are-gone">Three blockers are gone</a></h3>
<ol>
<li><strong>Compositing works, including Wayland.</strong> The spike ran all six gates; the
2024 "not possible on Wayland at all" claim is out of date when the render API
is used instead of foreign-window embedding.</li>
<li><strong>There is no ABR to lose.</strong> DR-229: the server's master playlist carries one
<code>EXT-X-STREAM-INF</code>. hls.js was demuxing, not adapting.</li>
<li><strong>A direct-play path exists.</strong> It did not when the spike was written. DR-228
built it; DR-230 proved the contract is player-agnostic.</li>
</ol>
<p>And on Windows specifically, <code>tauri-plugin-libmpv</code> lists Windows as its <strong>fully
tested</strong> platform — the inverse of the Linux situation the spike had to
disprove. The embedding difficulty was always WebKitGTK-specific.</p>
<h2 id="layer-assignment"><a class="header" href="#layer-assignment">Layer assignment</a></h2>
<div class="table-wrapper"><table><thead><tr><th>Logic / responsibility</th><th>Layer</th><th>Why it belongs there</th></tr></thead><tbody>
<tr><td><strong>Which codecs this device can decode</strong></td><td><strong>Rust</strong></td><td>Domain: it is the input to Jellyfin's <code>PlaybackInfo</code> negotiation. It stops being a property of the <em>platform</em> and becomes a property of <em>the renderer in use</em> — see "The structural change".</td></tr>
<tr><td>Which backend renders video</td><td><strong>Rust</strong></td><td>Rust already owns this (<code>use_html5_element</code> / <code>VideoBackend</code>). It stops being a <code>cfg!</code> constant and becomes a runtime fact.</td></tr>
<tr><td>What stream to play (direct / remux / transcode, transport, ceiling)</td><td><strong>Rust — already decided</strong></td><td>DR-225. mpv consumes <code>StreamSelection</code>. Re-deriving any of it in a new backend would be the defect DR-225 exists to remove, restated.</td></tr>
<tr><td>Creating the GL surface, reparenting the webview, owning the render context</td><td><strong>Rust (platform layer)</strong></td><td>Native window and GL-context lifetime. Not presentation, and not expressible above the IPC boundary at all.</td></tr>
<tr><td>Render-context ↔ GL-context lifetime binding</td><td><strong>Rust</strong></td><td>A correctness invariant over native resources. DR-232.</td></tr>
<tr><td>Frame pacing (update callback, <code>report_swap</code>)</td><td><strong>Rust</strong></td><td>Timing against the compositor; mpv's own contract.</td></tr>
<tr><td>Hardware-decode selection</td><td><strong>Rust</strong></td><td>A capability question about the machine, answered from what mpv reports it actually selected.</td></tr>
<tr><td>Z-order of controls over video, overlay chrome, letterbox colour</td><td><strong>Frontend / mpv</strong></td><td>Presentation. Controls already draw over a transparent webview on Android; mpv paints its own letterbox bars (better than the Android equivalent, which shipped DR-194 as a defect).</td></tr>
<tr><td>Whether the surface is visible right now</td><td><strong>Frontend</strong></td><td><code>nativeVideoActive</code> already exists and toggles <code>data-native-video</code>. Unchanged.</td></tr>
</tbody></table>
</div>
<h3 id="the-structural-change"><a class="header" href="#the-structural-change">The structural change</a></h3>
<p>Everything above is routine except one row, and it carries the whole benefit.</p>
<p><code>video_codecs</code> in <code>build_device_profile</code> is a <strong>compile-time constant per
platform</strong>:</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span>#[cfg(all(not(target_os = "android"), target_os = "linux"))]
let (video_codecs, audio_codecs) = ("h264".to_string(), "aac,mp3,opus,…");
<span class="boring">}</span></code></pre></pre>
<p>That is correct only while a build has exactly one video renderer. It must be
derived from <strong>which renderer will decode this stream</strong>, which is runtime state.</p>
<p>It looks like configuration and is not: it is the input that decides whether the
server re-encodes, it changes when Jellyfin's API or our renderer changes, and
getting it wrong fails <em>silently</em> — a claimed codec the renderer cannot decode is
a black picture or silence, which is DR-148 and DR-228's audio override already.</p>
<p><strong>Write this against "the active video renderer", never <code>cfg!(target_os)</code>.</strong> It
is the single piece that must not be Linux-shaped, because phase 2 reuses it
unchanged.</p>
<h2 id="design"><a class="header" href="#design">Design</a></h2>
<h3 id="backend-and-compositing-dr-231-ir-033"><a class="header" href="#backend-and-compositing-dr-231-ir-033">Backend and compositing (DR-231, IR-033)</a></h3>
<p>An <code>MpvVideoBackend</code> beside the existing <code>MpvBackend</code> (audio). The mpv side —
render context, FBO, update callback, hwdec — is <strong>shared</strong>; only the surface
differs per platform:</p>
<div class="table-wrapper"><table><thead><tr><th>Platform</th><th>Surface</th><th>Status</th></tr></thead><tbody>
<tr><td>Linux (X11 + Wayland)</td><td><code>gdk_cairo_draw_from_gl()</code> in the default vbox's <code>draw</code> handler, over a <code>GdkGLContext</code> on its <code>GdkWindow</code>. No reparenting — see below</td><td>Render path proven by the spike; the <em>overlay</em> approach it used is rejected</td></tr>
<tr><td>Windows</td><td>Native HWND child beneath a transparent WebView2</td><td>Phase 2</td></tr>
</tbody></table>
</div>
<p><code>vo=libmpv</code> plus <code>mpv_render_context_create</code> with <code>MPV_RENDER_PARAM_OPENGL_FBO</code>.
Webview transparency via <code>with_transparent(true)</code> — no window-level transparency;
the spike showed it is neither used nor needed.</p>
<p><strong>G1's untested half failed, and the design changed because of it.</strong></p>
<p>Reparenting Tauri's webview into a <code>GtkOverlay</code> attaches cleanly and then aborts
the process on the first click. <code>tauri-runtime-wry</code> connects a
button-press handler to the webview that walks a hard-coded path:</p>
<pre><pre class="playground"><code class="language-rust"><span class="boring">#![allow(unused)]
</span><span class="boring">fn main() {
</span>webview.parent() // "This one should be GtkBox"
.parent() // ...and this one the GtkWindow
.downcast::&lt;gtk::Window&gt;().unwrap()
<span class="boring">}</span></code></pre></pre>
<p>An overlay makes that chain <code>webview → GtkOverlay → GtkBox</code>, the downcast fails,
and the panic is non-unwinding so it kills the app. Nothing in configuration
avoids it: on Linux <code>attach_resize_handler</code> is called <strong>unconditionally</strong> (the
Windows equivalent is guarded by <code>is_decorated()</code>), and the decoration check that
would make the handler inert runs <em>after</em> the unwrap.</p>
<p><strong>So the webview is not moved at all.</strong> mpv draws into the <em>default vbox's own
<code>draw</code> handler</em> instead, via <code>gdk_cairo_draw_from_gl()</code> over a <code>GdkGLContext</code>
created on that widget's <code>GdkWindow</code>. GTK3 draws a container before its children,
so the webview composites on top for free — the same z-order the overlay was for,
without touching the widget tree Tauri walks.</p>
<p>That is strictly better than the overlay it replaces: no reparent, no extra
widget, and the arrangement cannot be broken by a Tauri upgrade that assumes its
own layout. It is also why "the surface attached successfully" is not the gate —
a click is.</p>
<p>Three traps from the spike, each of which cost a debugging cycle and each of
which looks like a platform limitation and is not:</p>
<ol>
<li><strong><code>LC_NUMERIC</code> must be reset <em>after</em> <code>gtk::init()</code>.</strong> mpv refuses to start
under a non-C numeric locale. <code>mpv_backend.rs</code> already handles this but has no
GTK init in front of it; here <code>gtk::init()</code> applies the user's locale
afterwards and <code>mpv_create</code> returns null.</li>
<li><strong>libepoxy exports GL entry points as <em>data</em> symbols.</strong> There is no <code>glFoo</code>
function — there is <code>epoxy_glFoo</code>, a variable holding a lazily-resolving
pointer. <code>get_proc_address</code> must return the pointer <strong>stored at</strong> that symbol;
returning the symbol's own address makes mpv jump into non-executable data and
take SIGSEGV on the first GL call. The <code>epoxy</code> crate does this correctly but is
unusable — its <code>gl_generator</code> dependency pulls a yanked <code>xml-rs</code>.</li>
<li><strong>Frame pacing is not optional and its symptom misleads.</strong> See DR-233.</li>
</ol>
<h3 id="render-context-lifetime-dr-232--the-crash-defence"><a class="header" href="#render-context-lifetime-dr-232--the-crash-defence">Render-context lifetime (DR-232) — the crash defence</a></h3>
<p>The spike's one unexplained SIGSEGV landed in a <em>decoder</em> thread with no Tauri,
GTK or GL frame in the stack, and three plausible causes failed to reproduce it
across ~13 minutes of targeted stress.</p>
<p>What is <strong>not</strong> unexplained is that the spike had no defence: it never calls
<code>mpv_render_context_free</code> and never tears down on <code>unrealize</code>, so nothing stopped
the GL context being recreated beneath the render context. That is DR-184 on
Android restated — a surface outliving its player.</p>
<p>Built as a requirement in its own right, not as a fix for a crash we cannot yet
reproduce:</p>
<ul>
<li>Render context created on <code>realize</code>, freed on <code>unrealize</code>, same thread, before
the GL context goes away.</li>
<li>The update callback is unregistered <strong>before</strong> the context is freed, so a
callback cannot land on a freed context.</li>
<li>Playback teardown and surface teardown are ordered, not racing.</li>
</ul>
<p>If the crash recurs after this, it is a different bug and the likeliest cause is
out of the search space. If it does not, we needed this anyway.</p>
<h3 id="frame-pacing-dr-233"><a class="header" href="#frame-pacing-dr-233">Frame pacing (DR-233)</a></h3>
<p>Register <code>mpv_render_context_set_update_callback</code>; redraw only when it reports a
frame ready; call <code>mpv_render_context_report_swap</code> after each render.</p>
<p>Recorded because the failure mode is a trap: driving <code>queue_render()</code> off the
frame clock every tick without reporting the swap leaves mpv nothing to time
against. It looks fine in a window and <strong>judders at fullscreen</strong>, which reads as
a compositing or GPU limit and is neither.</p>
<h3 id="renderer-dependent-device-profile-dr-234"><a class="header" href="#renderer-dependent-device-profile-dr-234">Renderer-dependent device profile (DR-234)</a></h3>
<p><code>build_device_profile</code> takes the active video renderer and derives the codec
lists from it:</p>
<div class="table-wrapper"><table><thead><tr><th>Renderer</th><th>Video codecs</th><th>Audio (video direct play)</th><th>Channels</th></tr></thead><tbody>
<tr><td>mpv (desktop native)</td><td><code>h264,hevc,vp8,vp9,av1,mpeg4</code></td><td>platform list incl. <code>ac3,eac3</code> where the sink can voice it</td><td>from the audio route</td></tr>
<tr><td>WebKitGTK <code>&lt;video&gt;</code></td><td><code>h264</code></td><td>webview-decodable set only</td><td>2</td></tr>
<tr><td>ExoPlayer (Android)</td><td>unchanged</td><td>unchanged</td><td>unchanged</td></tr>
</tbody></table>
</div>
<p>The existing <code>video_audio_codecs()</code> narrowing exists because <em>the webview decodes
a narrower audio set than the platform</em>. With mpv decoding, that no longer
applies to the video path — but the multichannel bound still does, since a 5.1
track direct-played into a 2-channel sink is silence or inaudible dialogue. Both
constraints stay, sourced from the renderer rather than assumed.</p>
<p><strong>This is what converts the 7% figure upward</strong> (toward, not necessarily to, the 85% ceiling — see the caveat above), and it is also the change most able to break
playback silently — so it lands after compositing is proven, covered by the
DR-228 override tests.</p>
<h3 id="deleting-the-webview-video-path-dr-235"><a class="header" href="#deleting-the-webview-video-path-dr-235">Deleting the webview video path (DR-235)</a></h3>
<p><code>get_player_status</code> stops reporting <code>use_html5_element: true</code> on desktop;
<code>supports_native_video</code> becomes true there.</p>
<p>Deletion is staged, because a path cannot be removed while a shipped platform
still needs it:</p>
<div class="table-wrapper"><table><thead><tr><th>Phase</th><th>Linux</th><th>Windows</th><th>HTML5 video path</th></tr></thead><tbody>
<tr><td>1</td><td>mpv</td><td>HTML5</td><td>alive — Windows needs it</td></tr>
<tr><td>2</td><td>mpv</td><td>mpv</td><td>alive but unreached</td></tr>
<tr><td>3</td><td>mpv</td><td>mpv</td><td><strong>deleted</strong>, with hls.js</td></tr>
</tbody></table>
</div>
<p>Phase 3 is a real phase with its own acceptance criterion, not a "later". The
whole maintenance argument for including Windows collapses if the fork survives.</p>
<p>Android keeps ExoPlayer and keeps the webview as its documented opt-out; the
<code>&lt;audio&gt;</code> element and the background-audio handoff are untouched throughout.</p>
<p><strong>What happens when mpv fails to initialise.</strong> With no HTML5 path there is no
silent fallback, and inventing one resurrects what we deleted. The
graceful-backend-init principle applies as written: fall back to the no-op
backend, emit <code>backend-init-failed</code>, and surface a real error rather than a black
rectangle. An honest failure beats a hidden downgrade to the transcode we are
trying to stop paying for.</p>
<h3 id="hardware-decode-dr-236"><a class="header" href="#hardware-decode-dr-236">Hardware decode (DR-236)</a></h3>
<p>The spike established the load-bearing fact: <strong>hardware decode works through the
render API</strong> (<code>hwdec-current</code> reported <code>nvdec-copy</code> on the discrete GPU), so the
direct-play prize is not traded for software decoding.</p>
<p>Policy is decided from what mpv reports it <em>selected</em>, never from what it was
asked for:</p>
<ul>
<li>Prefer zero-copy VA-API on the integrated GPU where the driver is present.</li>
<li><code>auto</code> reached for the discrete GPU in <strong>copy-back</strong> mode on a hybrid
Intel+NVIDIA laptop — the least efficient hardware path — so <code>auto</code> is a
fallback, not the default.</li>
<li><code>vaapi</code> silently fell back to software on the spike box because <code>vainfo</code> was
absent. A missing driver must be detected and logged, not mistaken for a
compositing limit.</li>
<li>Log <code>hwdec-current</code> at start-up; knowing what was actually chosen is the whole
diagnostic value.</li>
</ul>
<h3 id="windows-what-phase-2-actually-costs-dr-237"><a class="header" href="#windows-what-phase-2-actually-costs-dr-237">Windows: what phase 2 actually costs (DR-237)</a></h3>
<p>Not hidden, because it is the part most likely to be underestimated:</p>
<ul>
<li><strong>The surface is different code.</strong> WebView2 in an HWND, not GTK. A transparent
WebView2 over a native child window is a solved arrangement, but DR-231's
Linux surface does not transfer. Everything else does.</li>
<li><strong>libmpv is currently a Linux-only dependency</strong>, and Windows is
<strong>cross-compiled from Linux</strong> via <code>x86_64-pc-windows-msvc</code> + <code>cargo-xwin</code>. Phase
2 must source a Windows libmpv (DLL + import library) into that cross-build and
ship the DLL in the NSIS bundle.</li>
<li><strong>LGPL obligations follow the DLL.</strong> DR-216 already records them for Linux:
keep the linkage dynamic, ship libmpv's licence text with any bundle carrying
it. The Windows bundle inherits both.</li>
<li><strong><code>bun run test:rust</code> and CI must still build.</strong> Per the CI rule, any tool this
needs goes into the builder image and is pushed — never installed at job time.</li>
</ul>
<p>Windows also gains a native <em>audio</em> decoder as a side effect, which is what
<a href="windows-native-audio-backend.html">windows-native-audio-backend.md</a> wants and
cannot currently have. If that spec lands first, phase 2 inherits its build work
and shrinks to the surface.</p>
<h2 id="out-of-scope"><a class="header" href="#out-of-scope">Out of scope</a></h2>
<ul>
<li><strong>Android.</strong> Unchanged in every respect.</li>
<li><strong>macOS.</strong> Not a shipped target. If it becomes one it joins phase 2's shape.</li>
<li><strong>Audio backends.</strong> mpv already plays audio on Linux; this adds a video
renderer beside it. Windows audio is its own spec.</li>
<li><strong>HDR, tone mapping, multi-window.</strong> Not exercised by the spike at all.</li>
<li><strong>Re-deciding what stream to play.</strong> DR-225 owns that. If this spec finds
itself choosing a URL, something has gone wrong.</li>
</ul>
<h2 id="acceptance-criteria"><a class="header" href="#acceptance-criteria">Acceptance criteria</a></h2>
<p><strong>Phase 1 — Linux</strong></p>
<ul>
<li><input disabled="" type="checkbox"/>
Tauri's own webview reparents into the overlay (the untested half of G1),
on X11 <strong>and</strong> Wayland.</li>
<li><input disabled="" type="checkbox"/>
Video plays, seeks and switches audio track in mpv, with the Svelte
controls composited over it and alpha blending intact.</li>
<li><input disabled="" type="checkbox"/>
The render context is freed on <code>unrealize</code> and the update callback
unregistered before the free; a test demonstrates the ordering.</li>
<li><input disabled="" type="checkbox"/>
A direct-play negotiation returns <code>DirectPlay</code> for an hevc source that
today returns <code>Transcode</code>, and it plays.</li>
<li><input disabled="" type="checkbox"/>
Direct-play rate over the same 40-item sample rises from 7% toward the
Android figure. <strong>Record the number.</strong></li>
<li><input disabled="" type="checkbox"/>
mpv init failure emits <code>backend-init-failed</code> and surfaces an error rather
than falling back to a transcode.</li>
<li><input disabled="" type="checkbox"/>
<code>hwdec-current</code> is logged and is not copy-back where zero-copy is available.</li>
<li><input disabled="" type="checkbox"/>
A soak covering seek, track switch and fullscreen runs clean for an agreed
duration. <strong>The spike's SIGSEGV is why this is a criterion.</strong></li>
</ul>
<p><strong>Phase 2 — Windows</strong></p>
<ul>
<li><input disabled="" type="checkbox"/>
libmpv links in the <code>cargo-xwin</code> cross-build; the DLL and its licence ship
in the NSIS bundle; any new tool lives in the builder image, not in a CI step.</li>
<li><input disabled="" type="checkbox"/>
Video plays composited under a transparent WebView2.</li>
<li><input disabled="" type="checkbox"/>
The device profile, lifetime and hwdec code are <strong>reused, not
reimplemented</strong> — a reviewer confirms no <code>cfg!(target_os = "linux")</code> guards
them.</li>
</ul>
<p><strong>Phase 3 — deletion</strong></p>
<ul>
<li><input disabled="" type="checkbox"/>
<code>use_html5_element</code> is false on every desktop platform.</li>
<li><input disabled="" type="checkbox"/>
<code>hls.js</code> is gone from <code>package.json</code>; <code>html5Adapter.ts</code>, <code>videoLoaderFor</code>
and the <code>&lt;video&gt;</code> element are deleted; Android's opt-out and the
background-audio <code>&lt;audio&gt;</code> path still work.</li>
</ul>
<p><strong>Throughout</strong></p>
<ul>
<li><input disabled="" type="checkbox"/>
<code>bun run check</code>, <code>bun run test</code>, <code>bun run format:check</code>, <code>bun run lint</code> pass.</li>
<li><input disabled="" type="checkbox"/>
<code>cargo fmt</code> clean, <code>cargo clippy -D warnings</code> clean, <code>bun run test:rust</code> passes.</li>
<li><input disabled="" type="checkbox"/>
<code>bun run check:boundary</code> passes, and a reviewer confirms no stream decision
was reconstructed in the new backend.</li>
<li><input disabled="" type="checkbox"/>
<code>bindings.ts</code> regenerated from Rust.</li>
<li><input disabled="" type="checkbox"/>
<code>bun run traces:validate</code> passes; coverage stays ≥ the CI ratchet.</li>
<li><input disabled="" type="checkbox"/>
The spike and this spec are folded into
<a href="../architecture/05-platform-backends.html">05-platform-backends.md</a> and both
deleted in the same commit.</li>
</ul>
<h2 id="testing"><a class="header" href="#testing">Testing</a></h2>
<ul>
<li><strong>Rust, pure:</strong> the device profile per renderer — mpv claims hevc, the webview
does not, the multichannel bound survives both. The DR-234 table as a
table-driven test.</li>
<li><strong>Rust, pure:</strong> <code>PlaybackInfo</code> fixtures that transcode under the webview
profile and direct-play under the mpv profile — the direct-play conversion as a unit
test, not only as a measurement.</li>
<li><strong>Rust:</strong> teardown ordering — callback unregistered before context freed, freed
before GL context destroyed. Structure it so the ordering is assertable without
a live GL context.</li>
<li><strong>Frontend:</strong> no desktop path selects an HTML5 video adapter. After phase 3,
the adapter does not exist and the test goes with it.</li>
<li><strong>Manual / soak:</strong> the criterion above. The spike's automated fullscreen and
resize soaks are reusable and already written.</li>
</ul>
<h2 id="traces"><a class="header" href="#traces">TRACES</a></h2>
<div class="table-wrapper"><table><thead><tr><th>Piece</th><th>Tag</th></tr></thead><tbody>
<tr><td>mpv video backend + compositing</td><td><code>UR-080 | DR-231, IR-033</code></td></tr>
<tr><td>Render-context lifetime binding</td><td><code>UR-080 | DR-232</code></td></tr>
<tr><td>Frame pacing</td><td><code>UR-080 | DR-233</code></td></tr>
<tr><td>Renderer-dependent device profile</td><td><code>UR-080, UR-070 | DR-234</code></td></tr>
<tr><td>Webview video path removed</td><td><code>UR-080 | DR-235</code></td></tr>
<tr><td>Hardware-decode policy</td><td><code>UR-080 | DR-236</code></td></tr>
<tr><td>Windows surface + cross-build</td><td><code>UR-080 | DR-237</code></td></tr>
</tbody></table>
</div>
<h2 id="notes-for-the-implementer"><a class="header" href="#notes-for-the-implementer">Notes for the implementer</a></h2>
<ul>
<li><strong>Read the spike before writing a line.</strong> Its three traps and its
hardware-decode table are the most valuable things in this directory, and each
cost a debugging cycle to find.</li>
<li><strong>mpv consumes <code>StreamSelection</code>; it does not decide.</strong> The transport is on the
queue item (DR-230). If you are parsing a URL, stop.</li>
<li><strong>Guard nothing on <code>cfg!(target_os = "linux")</code> that phase 2 will need.</strong> That is
the one avoidable mistake here.</li>
<li>The Android backend is the reference for the <em>shape</em> of this — transparent
webview over a native surface at index 0. Read <code>05-platform-backends.md</code>'s
Android section for what shipped and what its defects were (DR-184 surface
lifetime, DR-194 letterbox).</li>
<li>Do not call sync/blocking APIs from mpv event callbacks that can re-enter the
player or hold a lock. The existing deadlock gotchas apply.</li>
<li>A parallel Claude session may be active in this repo — <code>git diff</code> before
"repairing" unexpected changes.</li>
<li>This branch is stacked on backend-owned stream selection. Rebase when that
merges rather than merging master into it.</li>
</ul>
</main>
<nav class="nav-wrapper" aria-label="Page navigation">
<!-- Mobile navigation buttons -->
<a rel="prev" href="../specs/frontend-domain-model.html" class="mobile-nav-chapters previous" title="Previous chapter" aria-label="Previous chapter" aria-keyshortcuts="Left">
<i class="fa fa-angle-left"></i>
</a>
<a rel="next prefetch" href="../specs/build-provenance.html" class="mobile-nav-chapters next" title="Next chapter" aria-label="Next chapter" aria-keyshortcuts="Right">
<i class="fa fa-angle-right"></i>
</a>
<div style="clear: both"></div>
</nav>
</div>
</div>
<nav class="nav-wide-wrapper" aria-label="Page navigation">
<a rel="prev" href="../specs/frontend-domain-model.html" class="nav-chapters previous" title="Previous chapter" aria-label="Previous chapter" aria-keyshortcuts="Left">
<i class="fa fa-angle-left"></i>
</a>
<a rel="next prefetch" href="../specs/build-provenance.html" class="nav-chapters next" title="Next chapter" aria-label="Next chapter" aria-keyshortcuts="Right">
<i class="fa fa-angle-right"></i>
</a>
</nav>
</div>
<script>
window.playground_copyable = true;
</script>
<script src="../elasticlunr.min.js"></script>
<script src="../mark.min.js"></script>
<script src="../searcher.js"></script>
<script src="../clipboard.min.js"></script>
<script src="../highlight.js"></script>
<script src="../book.js"></script>
<!-- Custom JS scripts -->
</div>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink