dtourolle/jellytau
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases 38 Wiki Activity

docs(specs): mpv draws desktop video, and the webview path goes

Browse Source 
The spike proved compositing works on Linux, including Wayland, and left two
blockers. One is now closed: DR-228 measured a single EXT-X-STREAM-INF in the
server's master playlist, so there is no adaptive bitrate for mpv to lose and
finding 3 of playback-backend-unification.md is false. The spike is updated to
record that. The other — an unexplained SIGSEGV in a decoder thread — is carried
into the spec as DR-231 rather than chased: the spike had no render-context
teardown at all, which is DR-184 on Android restated, and removing the likeliest
cause is worth doing whether or not it was the cause.

The spec targets every desktop platform rather than Linux alone, because the
maintenance argument runs the other way. Video has three renderers today. A
Linux-only version makes it four, permanently — mpv on Linux, HTML5 on Windows,
ExoPlayer on Android, hls.js underneath — and the webview path then survives
indefinitely because something still needs it. Finishing the job leaves mpv on
desktop and ExoPlayer on Android, and hls.js, html5Adapter.ts, videoLoaderFor
and the <video> element are deleted in a phase that has its own acceptance
criterion so it cannot quietly become "later".

The load-bearing change is DR-233: the device profile stops being a
compile-time platform constant and becomes a property of the renderer that will
decode the stream. The measured 7% desktop direct-play rate and Android's 85%
differ by nothing except which component decodes, so that one change is what
converts the former toward the latter. It looks like configuration and is not —
it decides whether the server re-encodes, and it fails silently when wrong.

Windows is costed rather than waved at: the surface is genuinely different code
(WebView2 in an HWND, not GTK), but everything else is shared, so nothing may be
guarded on cfg!(target_os = "linux"). The real cost is build — libmpv is a
Linux-only dependency while Windows cross-compiles via cargo-xwin, so a Windows
libmpv must reach that build and ship in the NSIS bundle under the LGPL terms
DR-216 already records.

Allocates UR-080, DR-230..236, IR-033. No product code yet.
This commit is contained in:
dtourolle
2026-08-22 13:45:03 +02:00
parent 109700b949
commit 7cc392d78f
5 changed files with 459 additions and 34 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/requirements.md
+20 -16
View File
@@ -89,6 +89,7 @@ For a narrative overview of the system design, see
| UR-077 | The app can update itself, or tell the user how. Somebody who installed an AppImage or ran the Windows installer had no upgrade path at all: nothing in the app ever mentioned that a newer version existed, and the release notes were the only announcement. On Linux and Windows the app checks a signed manifest, offers the new version with its notes, and installs and relaunches on request — the signature check is the point, since it is what stops a substituted download from being installed by the app itself. Android cannot do this (an app may not overwrite its own APK; that is the package installer's job) and is given the honest alternative, a link to the releases page, rather than a button that would throw | Medium | Done |
| UR-078 | JellyTau keeps a record of what it did, and can hand it over. The app forgot everything the moment it exited: the backend logged to stdout only — which a user launching from a desktop icon never sees, and which on Android is not logcat, so the Rust half was invisible on the platform carrying the hardest bugs. A crash left nothing at all. Logs are now written to a size-capped rotating file, a panic is recorded before the process dies, the frontend's messages land in the same timeline as the backend's, and Settings exports the lot as one file to attach to a bug report. Nothing is transmitted anywhere — the user attaches it themselves, which is also what keeps this from being telemetry. Access tokens and passwords never reach the file | Medium | Done |
| UR-079 | The app decides *what stream to play* and says so. Playing a video used to mean asking the server to re-encode it, always — a decision made nowhere, written down nowhere, and re-derived downstream by whoever needed it: the player worked out whether it had been handed a playlist by looking for `.m3u8` in the URL. So a viewer paid for a transcode of a file their device could have played untouched, and the app could not tell them which it was. Now one negotiation produces one self-describing answer — direct play, remux, or transcode; over a playlist, a plain HTTP file, or a local one — and every renderer consumes that same answer instead of guessing from a string. On Android, where the player decodes almost everything the library holds, this stops around 85% of plays from starting a transcode nobody needed | Medium | Done |
| UR-080 | Video on the desktop plays as itself. The picture was drawn by a webview `<video>` element, which decodes little beyond h264 — so the app told the server it could accept only h264, and the server re-encoded almost everything before sending it. That was never a statement about the machine: the same machine already runs mpv for audio, which decodes essentially the whole library. Measured against a real library, 93% of desktop playback was a transcode nobody needed, against 15% on Android where a real decoder does the work. mpv now draws the picture, the app claims what it can genuinely decode, and video is sent as it was stored wherever that is possible — sparing the server the work, the network the bitrate, and the picture a generation of re-encoding | Medium | Proposed |
| UR-074 | Video streaming can be held to a **bandwidth budget the viewer sets**, rather than spent at whatever rate the server would otherwise send. A ceiling chosen once — from the source's own bitrate down to a rung that still plays on a poor connection — governs every video the app opens, live TV included, and survives a restart, so a metered connection is not quietly drained by the next thing played. A single video can be moved to a different ceiling from the player, resuming where it was, without disturbing that default | Medium | Done |
---
@@ -133,6 +134,7 @@ External system integrations and platform-specific implementations.
| IR-030 | Scheduled full-catalog crawl of every library (`Recursive=true`, paged) feeding the local index, driven by a Rust background task and the `ConnectivityMonitor` reconnect signal rather than by the frontend | Storage | UR-065 | Implemented |
| IR-031 | Android `WindowInsets` bridge: an `OnApplyWindowInsetsListener` on the decor view reports `systemBars() | displayCutout()` in CSS pixels, pushed into the WebView as `jt-inset` CSS custom properties plus a `jellytau-insets-changed` event, and pullable via the `AndroidInsets` JS bridge | Platform | UR-066 | Done (pending device verification) |
| IR-032 | Whole-file background download of the item being played, reusing the existing resumable download worker and the Range-capable `/Videos/{id}/stream.mp4` endpoint; plus per-platform read-through caching hooks (ExoPlayer `CacheDataSource`, mpv `stream-record`) for direct-play sessions only | Storage | UR-071 | Proposed |
| IR-033 | libmpv render-API integration for video: `vo=libmpv` driving an OpenGL FBO bound by the host toolkit, with GL entry points resolved through libepoxy. Note that libepoxy exports them as *data* symbols — there is no `glFoo` function, only an `epoxy_glFoo` variable holding a lazily-resolving pointer — so `get_proc_address` must return the pointer stored **at** that symbol; returning the symbol's own address makes mpv jump into non-executable data and take SIGSEGV on the first GL call. The `epoxy` crate resolves this correctly but is unusable, its `gl_generator` dependency pulling a yanked `xml-rs` | Playback | UR-080 | Proposed |
> **Where a UR is met by a different mechanism than its IR anticipated.** Several
> integration requirements were written when libmpv was expected to be the single
@@ -416,16 +418,20 @@ Internal architecture, components, and application logic.
| DR-221 | The release path is exercised before a tag exists. Nothing in `build-and-test.yml` runs `tauri build` — only a tag does — so a whole class of breakage was invisible until release day, and two instances of it were sitting on master at once. Tauri refuses to build when a plugin's Rust crate and npm package differ by minor version, which the updater and logging work had introduced (`tauri-plugin-log 2.8.0` against `@tauri-apps/plugin-log 2.9.0`) while `cargo check`, clippy, the tests and `svelte-check` all passed; both sides are now pinned exactly rather than by caret, since a caret is what let them separate, and CI runs `tauri info` to compare them without building. The AppImage target had never once been built: linuxdeploy carries a `strip` too old to parse the `.relr.dyn` section modern toolchains emit, so bundling failed on every library — and Ubuntu 23.10+ links with `-z pack-relative-relocs` by default, so the builder image fails the same way a modern Arch host does. `NO_STRIP=true` is linuxdeploy's documented escape hatch; the cost is a larger, unstripped bundle. Both were found by building the target locally before tagging rather than by publishing a release that could not build | Tooling | - | Done |
| DR-222 | Build tooling matches the package manager the project declares. `scripts/build-android.sh` ran `npm install` on its clean-build path — in a bun project, where `packageManager` says bun and `bun.lock` is the committed lockfile. npm ignores that lockfile, re-resolves the whole tree from package.json, and writes a `package-lock.json` that `.gitignore` then hides. That is not a style preference: the JS halves of the Tauri plugins are pinned exactly against Cargo.lock because the CLI refuses to build when a plugin's crate and package differ by minor version, and a silent re-resolve is precisely how they drift apart. It survived because clean builds are rare — the shape shared by nearly every defect found preparing v0.10.0, where the code running on every commit was healthy and the code running on a release, a tag or a clean build had no guard at all. `scripts/check-tooling.sh` fails on any npm/yarn/pnpm invocation or foreign lockfile | Tooling | - | Done |
| DR-223 | The Android JavaVM and Application are published into `ndk_context` by this crate, not by a transitive dependency. Seven call sites (five in credentials.rs, two in lib.rs) read that process-global to reach JNI, and nothing here ever set it — `tao` did, three levels below anything this project names in Cargo.toml. tao 0.35.3 moved those pointers into a private struct and stopped publishing them, so the Tauri 2.11 upgrade made the first credential read abort the process on every launch: `PANIC ... android context was not initialized`. Our code had not changed; an undocumented side effect of the windowing layer had gone. The invariant is now owned here rather than assumed: `JNI_OnLoad` captures the JavaVM as the shared library loads, and the Application is resolved lazily via `ActivityThread.currentApplication()` and pinned as a global reference for the process lifetime — the Application rather than the Activity, since that is what `SecureStorage.initialize()` immediately reduces its argument to. Failure degrades to the encrypted-file credential path and is logged, rather than aborting. Found only by installing on a device: nothing in CI runs the app | Security | UR-012 | Done |
<<<<<<< HEAD
| DR-224 | Backgrounding the app obeys the background-audio toggle on every renderer. The toggle (UR-040) was built for the WebView `<video>` path, where losing visibility kills the decode: it chose between handing off to a native audio stream and letting playback stop. Native video then became the default renderer (DR-188), and on that path playback runs through ExoPlayer inside a `MediaSessionService` — a foreground media service whose purpose is to keep playing while the app is hidden. Nothing paused it and nothing in the codebase paused on background, so locking the screen kept the audio going whether or not the toggle was on: the toggle governed a handoff that no longer had a gap to bridge, and users got background playback they never asked for. The decision now lives in Rust (`player/background_policy.rs`) and both renderers obey it: a video with the toggle off pauses, with the toggle on hands off to audio, music is never paused by backgrounding, and picture-in-picture keeps playing because the window is still on screen (UR-041). It takes no renderer parameter on purpose — the split between the two paths is what produced the defect | Player | UR-040 | Done |
=======
| DR-224 | `StreamSelection` replaces the bare URL returned for playback: URL, `Transport` (hls / progressive / localFile), `PlaybackKind` (directPlay / directStream / transcode), the negotiated `Rendition`, the ladder this source can offer, and a `needs_transcoding` flag derived in Rust so "which kinds count as transcoding" is answered once. Both enums are serde-tagged (`{"type":"hls"}`) so the frontend matches a discriminant rather than comparing text. The field that mattered most is `transport`: `VideoPlayer.svelte` chose its loader with `url.includes(".m3u8")` in two places, a domain fact reconstructed in the presentation layer — the same class of error as leaking item-type taxonomy, and one that fails silently in both directions (a progressive file served from a path containing the substring gets an HLS loader; a playlist served from one without it does not). The paths that never negotiate — a downloaded file, a live channel — get the same shape from Rust (`media_local_selection`, `LiveStreamInfo.transport`) rather than having the page assemble one, so there is no second place where a transport is decided | Playback | UR-079 | Done |
| DR-225 | The bandwidth ceiling is two-level: a durable device default (Settings, persisted, restored at startup) and a per-playback override the in-player picker sets. The picker's own documentation had called it a "this film, this connection" control since it was written, but it was implemented by writing the process-wide default — so dropping one awkward film to 2 Mbps silently capped every video played afterwards for the rest of the process, while the Settings screen still displayed the old value and nothing in the UI admitted the change. The override is cleared whenever playback moves to a new item, which is what keeps it from surviving into an autoplayed next episode where nobody would reopen the picker. `effective_streaming_quality()` is the single resolution point; every URL builder and the `PlaybackInfo` negotiation go through it, because a negotiation that authorises a direct play the URL builder then constrains (or the reverse) leaks the cap | Playback | UR-074, UR-079 | Done |
| DR-226 | The quality picker is filled from what *this* media source can offer, not from the fixed eight-rung enum. Rust marks each rung `exceeds_source` when its ceiling is at or above the source's own bitrate — such a rung produces the same bytes as `Original`, so offering it is another way to spell one choice — and the frontend simply does not draw those. `Original` is never marked (it *is* the source) and a source whose bitrate the server does not report (the sampled library has `avi` files with none) marks nothing redundant, keeping every rung offered, which is the safe direction. The picker also shows what the server is actually doing with the stream, which only became knowable once `PlaybackKind` existed. Labels and detail lines come from Rust beside the numbers they describe, so a relabelled rung cannot drift out of step with what it does | UI | UR-070, UR-079 | Done |
| DR-227 | Direct play and direct stream are negotiated rather than assumed away. `get_video_stream_url` always built an HLS transcode URL, so every video play burned server CPU even when the file would have played untouched. The decision now comes from `PlaybackInfo` under the device profile and the ceiling in force, with two client-side overrides applied on top because the server's answer is right about the *file* and wrong about what this app will do with it: undecodable audio (Jellyfin 10.11.5 honours a DirectPlayProfile's container and video codec but ignores its audio codec, so it offers direct play for an E-AC-3 track the webview renders in silence) and a viewer-pinned audio track the source file does not default to. Measured against the development server over a 400-item sample: **85% direct play on the Android profile, 7% on the Linux one** — the library is ~80% hevc and WebKitGTK can only claim h264, so the Linux figure is a property of the renderer, not of this code, and is what `linux-native-video-spike.md` exists to change. A direct *stream* is a remux and is deliberately not counted as transcoding | Playback | UR-079 | Done |
| DR-228 | Mid-playback re-negotiation on throughput was scoped and **dropped on measurement**. The premise — that hls.js gives this app real adaptive bitrate and mpv would lose it — does not hold: a master playlist from the development server carries exactly one `EXT-X-STREAM-INF`, because Jellyfin builds it from the single rendition the request asked for rather than publishing a ladder. There is no adaptation to preserve, so "adapt mid-stream" collapses into "pick well at open", which is what DR-225 and DR-226 already are. Recorded rather than deleted because the conclusion is a measurement, not an opinion, and a server that does publish a ladder would change it — the DR-224 re-negotiation path is the hook that work would build on | Playback | UR-079 | Won't Do |
| DR-229 | Every player backend consumes the same selection, proving the contract is player-agnostic rather than HTML5-shaped. The queue item carries the negotiated `transport`, so `player_seek_video` picks its seek strategy from the backend's own decision instead of the last `stream_url.contains(".m3u8")` in the codebase; items queued by a path that never negotiated (audio tracks, direct URLs) carry `None` and fall back to `needs_transcoding`, which is exact rather than a guess because every transcode this app requests is HLS (DR-140). The webview adapter's bridge carries the whole selection rather than a URL, so the component's HLS effect reads a tag instead of searching a string, and the background-audio handoff states the transport it is moving to (progressive mp3 out, HLS back) rather than leaving it to be inferred | Playback | UR-003, UR-004, UR-079 | Done |
>>>>>>> 83dc8c70 (feat(playback): let Rust decide what stream to play, and say so)
| DR-225 | `StreamSelection` replaces the bare URL returned for playback: URL, `Transport` (hls / progressive / localFile), `PlaybackKind` (directPlay / directStream / transcode), the negotiated `Rendition`, the ladder this source can offer, and a `needs_transcoding` flag derived in Rust so "which kinds count as transcoding" is answered once. Both enums are serde-tagged (`{"type":"hls"}`) so the frontend matches a discriminant rather than comparing text. The field that mattered most is `transport`: `VideoPlayer.svelte` chose its loader with `url.includes(".m3u8")` in two places, a domain fact reconstructed in the presentation layer — the same class of error as leaking item-type taxonomy, and one that fails silently in both directions (a progressive file served from a path containing the substring gets an HLS loader; a playlist served from one without it does not). The paths that never negotiate — a downloaded file, a live channel — get the same shape from Rust (`media_local_selection`, `LiveStreamInfo.transport`) rather than having the page assemble one, so there is no second place where a transport is decided | Playback | UR-079 | Done |
| DR-226 | The bandwidth ceiling is two-level: a durable device default (Settings, persisted, restored at startup) and a per-playback override the in-player picker sets. The picker's own documentation had called it a "this film, this connection" control since it was written, but it was implemented by writing the process-wide default — so dropping one awkward film to 2 Mbps silently capped every video played afterwards for the rest of the process, while the Settings screen still displayed the old value and nothing in the UI admitted the change. The override is cleared whenever playback moves to a new item, which is what keeps it from surviving into an autoplayed next episode where nobody would reopen the picker. `effective_streaming_quality()` is the single resolution point; every URL builder and the `PlaybackInfo` negotiation go through it, because a negotiation that authorises a direct play the URL builder then constrains (or the reverse) leaks the cap | Playback | UR-074, UR-079 | Done |
| DR-227 | The quality picker is filled from what *this* media source can offer, not from the fixed eight-rung enum. Rust marks each rung `exceeds_source` when its ceiling is at or above the source's own bitrate — such a rung produces the same bytes as `Original`, so offering it is another way to spell one choice — and the frontend simply does not draw those. `Original` is never marked (it *is* the source) and a source whose bitrate the server does not report (the sampled library has `avi` files with none) marks nothing redundant, keeping every rung offered, which is the safe direction. The picker also shows what the server is actually doing with the stream, which only became knowable once `PlaybackKind` existed. Labels and detail lines come from Rust beside the numbers they describe, so a relabelled rung cannot drift out of step with what it does | UI | UR-070, UR-079 | Done |
| DR-228 | Direct play and direct stream are negotiated rather than assumed away. `get_video_stream_url` always built an HLS transcode URL, so every video play burned server CPU even when the file would have played untouched. The decision now comes from `PlaybackInfo` under the device profile and the ceiling in force, with two client-side overrides applied on top because the server's answer is right about the *file* and wrong about what this app will do with it: undecodable audio (Jellyfin 10.11.5 honours a DirectPlayProfile's container and video codec but ignores its audio codec, so it offers direct play for an E-AC-3 track the webview renders in silence) and a viewer-pinned audio track the source file does not default to. Measured against the development server over a 400-item sample: **85% direct play on the Android profile, 7% on the Linux one** — the library is ~80% hevc and WebKitGTK can only claim h264, so the Linux figure is a property of the renderer, not of this code, and is what `linux-native-video-spike.md` exists to change. A direct *stream* is a remux and is deliberately not counted as transcoding | Playback | UR-079 | Done |
| DR-229 | Mid-playback re-negotiation on throughput was scoped and **dropped on measurement**. The premise — that hls.js gives this app real adaptive bitrate and mpv would lose it — does not hold: a master playlist from the development server carries exactly one `EXT-X-STREAM-INF`, because Jellyfin builds it from the single rendition the request asked for rather than publishing a ladder. There is no adaptation to preserve, so "adapt mid-stream" collapses into "pick well at open", which is what DR-225 and DR-226 already are. Recorded rather than deleted because the conclusion is a measurement, not an opinion, and a server that does publish a ladder would change it — the DR-224 re-negotiation path is the hook that work would build on | Playback | UR-079 | Won't Do |
| DR-230 | Every player backend consumes the same selection, proving the contract is player-agnostic rather than HTML5-shaped. The queue item carries the negotiated `transport`, so `player_seek_video` picks its seek strategy from the backend's own decision instead of the last `stream_url.contains(".m3u8")` in the codebase; items queued by a path that never negotiated (audio tracks, direct URLs) carry `None` and fall back to `needs_transcoding`, which is exact rather than a guess because every transcode this app requests is HLS (DR-140). The webview adapter's bridge carries the whole selection rather than a URL, so the component's HLS effect reads a tag instead of searching a string, and the background-audio handoff states the transport it is moving to (progressive mp3 out, HLS back) rather than leaving it to be inferred | Playback | UR-003, UR-004, UR-079 | Done |
| DR-231 | An mpv video backend that composites beneath the transparent webview, the desktop counterpart of the Android TextureView arrangement. mpv renders through its **render API** into an FBO the toolkit binds (`vo=libmpv` + `mpv_render_context_create` with `MPV_RENDER_PARAM_OPENGL_FBO`), rather than by embedding a foreign window — which is what the 2024 "not possible on Wayland at all" conclusion was about and why it does not apply. On Linux that is a `GtkOverlay` with a `GtkGLArea` as main child and Tauri's own webview reparented as the overlay child; the mpv half is shared and only the surface differs per platform. Webview transparency alone suffices — no window-level transparency is used or needed | Playback | UR-080 | Proposed |
| DR-232 | The mpv render context's lifetime is bound to the GL context it draws into: created on `realize`, freed on `unrealize`, on the same thread, with the update callback unregistered *before* the free so a callback cannot land on a freed context. This is DR-184 on Android restated — a surface outliving its player — and it is a requirement in its own right rather than a fix for a specific crash. The spike observed one SIGSEGV in a decoder thread that three targeted soaks failed to reproduce; what is not in doubt is that the spike never called `mpv_render_context_free` and never tore down on `unrealize`, so nothing defended against the GL context being recreated underneath. Removing the likeliest cause is worth doing whether or not it was the cause | Playback | UR-080 | Proposed |
| DR-233 | Frame pacing goes through mpv's update callback, with `mpv_render_context_report_swap` after each render. Recorded as a requirement because the failure mode misleads: driving the widget's frame clock every tick without reporting the swap leaves mpv with nothing to time against, which looks fine in a window and **judders at fullscreen** — reading as a compositing or GPU limit and being neither | Playback | UR-080 | Proposed |
| DR-234 | The device profile is derived from the **renderer that will decode the stream**, not from a compile-time platform constant. `video_codecs` was `#[cfg(target_os)]`, which is correct only while a build has one video renderer; once mpv and the webview element coexist it must be runtime state. This is the change that converts the measured 7% desktop direct-play rate toward the 85% the Android profile achieves on the same library, because the two differ by nothing except which component decodes. It looks like configuration and is not — it is the input that decides whether the server re-encodes, and getting it wrong fails silently, a claimed codec the renderer cannot decode being a black picture or silence (DR-148, and DR-227's audio override). The webview's narrower *audio* set stops applying to the video path once mpv decodes it, while the multichannel bound still does, since a 5.1 track direct-played into a two-channel sink is silence or inaudible dialogue | Repository | UR-080, UR-070 | Proposed |
| DR-235 | The webview video path is deleted, not merely bypassed. Staged, because a path cannot be removed while a shipped platform still needs it: Linux moves to mpv first, Windows follows, and only then do `hls.js`, `html5Adapter.ts`, `videoLoaderFor` and the `<video>` element go. The staging is the point — a Linux-only version would leave the fork alive permanently, taking video from three renderers to four and giving every seek strategy, track switch and lifecycle bug one more place to be got right. Android keeps ExoPlayer and keeps the webview as its documented opt-out; the background-audio `<audio>` path is untouched. With no HTML5 fallback left, a failed mpv init emits `backend-init-failed` and surfaces a real error rather than silently degrading to the transcode this work exists to stop paying for | Playback | UR-080 | Proposed |
| DR-236 | Hardware-decode policy is decided from what mpv reports it **selected** (`hwdec-current`), never from what it was asked for. The spike established that hardware decode works through the render API at all — the load-bearing result, since it means direct play is not bought with software decoding — but also that `auto` reached for the discrete GPU in copy-back mode on a hybrid Intel+NVIDIA laptop, the least efficient hardware path, and that `vaapi` fell back to software silently because the libva driver was absent. So zero-copy VA-API on the integrated GPU is preferred where the driver is present, `auto` is a fallback rather than the default, and a missing driver is detected and logged rather than mistaken for a compositing limit | Playback | UR-080 | Proposed |
| DR-237 | Windows reaches the same mpv path, reusing everything except the surface. The surface is genuinely different code — a native child window beneath a transparent WebView2, not GTK — but the render context, lifetime discipline, frame pacing, device profile and hwdec policy are shared, which is why none of them may be guarded on `cfg!(target_os = "linux")`. The cost is mostly build, not video: `libmpv` is currently a Linux-only dependency while Windows is cross-compiled from Linux via `x86_64-pc-windows-msvc` + `cargo-xwin`, so a Windows libmpv must reach that cross-build and its DLL must ship in the NSIS bundle, carrying the LGPL obligations DR-216 already records — dynamic linkage, licence text shipped alongside. Windows gains a native audio decoder as a side effect, which is what the long-blocked Windows audio work wants and cannot otherwise have | Playback | UR-080 | Proposed |
| DR-198 | The webview runs under a real Content-Security-Policy, and the asset protocol is scoped to the one directory it still serves. `csp` was `null`, which disables CSP entirely: any script that reached the web layer — through a future `{@html}`, a dependency, or a devtools paste — would have inherited the whole IPC surface, and with it the user's session. `script-src 'self'` (Tauri injects a nonce for SvelteKit's inline bootstrap script at build time, so no `'unsafe-inline'` is needed) plus `object-src`/`frame-src 'none'` and `base-uri 'self'` is the part that is genuinely restrictive. `img-src`/`media-src`/`connect-src` cannot be: the Jellyfin origin is typed in by the user at run time and is commonly plain `http` on a LAN, so they allow `http:`/`https:` — a wide grant for *data*, but one that still bars `file:`, `filesystem:` and scripting schemes, and leaves `script-src` untouched. `style-src` keeps `'unsafe-inline'` because Svelte compiles `style="…"` attributes (including `app.html`'s `display: contents` wrapper) into markup; this is safe only while no `<style>` element survives into `index.html`, since a nonce there would make Tauri's injection outrank — and therefore void — `'unsafe-inline'`. `worker-src blob:` and `media-src blob:` are hls.js: it demuxes in a worker built from a blob and attaches MSE through `URL.createObjectURL`. `asset:` and `http://asset.localhost` are the same protocol under the two naming schemes `convertFileSrc` emits (custom scheme on Linux/macOS, `http` host on Windows/Android); `ipc:`/`http://ipc.localhost` is the invoke transport, which would otherwise be blocked by `connect-src`. A run-time CSP naming the server origin exactly was rejected: Tauri computes the header from immutable config when it serves the HTML, so it would mean rebuilding config and reloading the webview on every server change, for a policy the user can already point anywhere. The asset-protocol scope narrows from `$APPDATA/**` to `$APPDATA/thumbnails/**` — since DR-137 moved downloaded media to the loopback server, `imageCache` is the only `convertFileSrc` caller left, so the database and the encrypted-token fallback file no longer sit inside the grant | Security | UR-012, UR-071 | Done |
---
@@ -513,7 +519,8 @@ Internal architecture, components, and application logic.
| UR-076 | - | DR-209 |
| UR-077 | - | DR-217 |
| UR-078 | - | DR-218 |
| UR-079 | - | DR-224, DR-225, DR-226, DR-227, DR-228, DR-229 |
| UR-079 | - | DR-225, DR-226, DR-227, DR-228, DR-229, DR-230 |
| UR-080 | IR-033 | DR-231, DR-232, DR-233, DR-234, DR-235, DR-236, DR-237 |
---
@@ -727,13 +734,10 @@ Internal architecture, components, and application logic.
| UT-208 | The update decision: each numeric version field is compared in order, the installed version is not offered to itself, a leading `v` is tolerated because that is how the tags are written, a pre-release sorts below the release of the same number so 0.9.2-rc1 is not offered to somebody on 0.9.2, a missing patch field reads as zero rather than NaN, mobile reports link-only while desktop reports install, and absent release notes normalise to null rather than undefined | DR-217 | Done |
| UT-209 | Redaction and forwarding. Rust: every credential shape reduces to `[REDACTED]` while the host, username and neighbouring parameters survive; redaction is idempotent, leaves ordinary lines alone, does not fire on the word "token" in prose, and does not panic on multi-byte input; a server URL keeps only scheme and host and drops an embedded `user:pass@`; an unparseable level falls back to info rather than failing at startup. Frontend: info and above forward while debug does not, a message the level filter suppressed is not forwarded, a throwing forwarder neither propagates nor prevents the console write, and an `Error` renders as name and message rather than the `{}` that `JSON.stringify` produces | DR-218 | Done |
| UT-210 | Cosmetic-commit detection for release notes: a `chore(format)`, `chore(deps)` or `style` subject is skipped when deriving a range's changed files, while `fix`, `feat`, `ci`, `docs`, a bare `chore:` and `chore(release):` are kept; and the word "format" appearing later in a subject ("fix(duration): format times over 24 hours") does not make a real fix look cosmetic | DR-219 | Done |
<<<<<<< HEAD
| UT-211 | The background decision: a video with the toggle off pauses (the reported defect, where the media service kept playing regardless), a video with it on hands off to audio, music keeps playing whatever the toggle says because it has no picture to lose, picture-in-picture keeps playing in every combination since the window is still visible, and the answer does not vary by renderer | DR-224 | Done |
=======
| UT-211 | The stream-selection contract. `Transport` and `PlaybackKind` each serialise to exactly the tag the frontend matches (`{"type":"hls"}`, `{"type":"directPlay"}`, …) and round-trip; nested `StreamSelection` fields are camelCase on the wire including `playbackKind`, `mediaSourceId` and `maxBitrate`; only `Transcode` counts as transcoding, so a direct stream does not; a local file is a direct play over a local transport with no ladder. The ladder: every rung at or above a 1.12 Mbps source is marked redundant while the three that constrain it are not, `Original` is never marked for any bitrate including zero and unknown, an unreported source bitrate keeps all eight rungs offered, a 40 Mbps source marks none, and each option carries the ladder's own label and detail | DR-224, DR-226 | Done |
| UT-212 | The direct-play negotiation, one test per branch, against `PlaybackInfo` fixtures whose shapes were all observed on a live server: a supported source direct-plays; a remuxable one direct-streams and reports itself as *not* transcoding; an unsupported codec transcodes; undecodable audio overrides the server's direct-play offer (silent picture is worse than a transcode); a pinned audio track forces a transcode; a ceiling below the source bitrate transcodes even though the codec is fine, and the ladder agrees that rung constrains it; direct play wins over direct stream when both are offered. Plus the ceiling: a per-playback override governs the stream being opened without disturbing the durable default the Settings screen shows, and dropping it returns to that default | DR-225, DR-227 | Done |
| UT-213 | The loader comes from the transport, never the URL. hls.js is attached for `hls` when available and the element's own loader when not; progressive and local files load directly; the element's `src` is emptied only when hls.js drives it. The two cases that fail against a substring check, and the reason the field exists: a `progressive` stream whose URL contains `.m3u8` is *not* given an HLS loader, and an `hls` stream whose URL contains no `.m3u8` *is*. Both failed against the pre-DR-224 implementation before the fix landed | DR-224 | Done |
>>>>>>> 83dc8c70 (feat(playback): let Rust decide what stream to play, and say so)
| UT-212 | The stream-selection contract. `Transport` and `PlaybackKind` each serialise to exactly the tag the frontend matches (`{"type":"hls"}`, `{"type":"directPlay"}`, …) and round-trip; nested `StreamSelection` fields are camelCase on the wire including `playbackKind`, `mediaSourceId` and `maxBitrate`; only `Transcode` counts as transcoding, so a direct stream does not; a local file is a direct play over a local transport with no ladder. The ladder: every rung at or above a 1.12 Mbps source is marked redundant while the three that constrain it are not, `Original` is never marked for any bitrate including zero and unknown, an unreported source bitrate keeps all eight rungs offered, a 40 Mbps source marks none, and each option carries the ladder's own label and detail | DR-224, DR-226 | Done |
| UT-213 | The direct-play negotiation, one test per branch, against `PlaybackInfo` fixtures whose shapes were all observed on a live server: a supported source direct-plays; a remuxable one direct-streams and reports itself as *not* transcoding; an unsupported codec transcodes; undecodable audio overrides the server's direct-play offer (silent picture is worse than a transcode); a pinned audio track forces a transcode; a ceiling below the source bitrate transcodes even though the codec is fine, and the ladder agrees that rung constrains it; direct play wins over direct stream when both are offered. Plus the ceiling: a per-playback override governs the stream being opened without disturbing the durable default the Settings screen shows, and dropping it returns to that default | DR-225, DR-227 | Done |
| UT-214 | The loader comes from the transport, never the URL. hls.js is attached for `hls` when available and the element's own loader when not; progressive and local files load directly; the element's `src` is emptied only when hls.js drives it. The two cases that fail against a substring check, and the reason the field exists: a `progressive` stream whose URL contains `.m3u8` is *not* given an HLS loader, and an `hls` stream whose URL contains no `.m3u8` *is*. Both failed against the pre-DR-224 implementation before the fix landed | DR-224 | Done |
### Integration Tests
docs/specs/README.md
+1
View File
@@ -44,6 +44,7 @@ taken by other work; each carries a ⚠️ note at the top.
| Spec | Blocked on / note |
|---|---|
| [desktop-native-video.md](desktop-native-video.md) | mpv draws video on every desktop platform, then the webview `<video>` path and hls.js are deleted. Converts a measured 7% direct-play rate toward Android's 85%. Stacked on backend-owned stream selection. |
| [build-provenance.md](build-provenance.md) | `build.rs` is still bare. ⚠️ suggested id DR-093 is taken. |
| [player-facade-enforcement.md](player-facade-enforcement.md) | ~60 `commands.player*` sites still outside the facade; no lint rule. ⚠️ suggested id DR-095 is taken. |
| [windows-native-audio-backend.md](windows-native-audio-backend.md) | Blocked on the libmpv2 swap. ⚠️ suggested id IR-030 is taken. |
docs/specs/desktop-native-video.md
+389
View File
@@ -0,0 +1,389 @@
# Spec: Desktop native video — mpv renders the picture, everywhere
**Status:** Proposed
**Requirements:** UR-080 (new) → DR-230 … DR-236 (new); IR-033 (new)
**UX spec:** n/a — nothing about the player's appearance changes. What changes is
what is behind the controls.
**Supersedes / revises:** consumes and closes
[linux-native-video-spike.md](linux-native-video-spike.md), whose gates
authorised exactly this spec and nothing more. Settles finding 2 of
[playback-backend-unification.md](playback-backend-unification.md) on the
desktop; finding 3 was already settled by DR-228. Absorbs the video half of what
[windows-native-audio-backend.md](windows-native-audio-backend.md) leaves open.
**Depends on:** backend-owned stream selection (DR-224 … DR-229), the branch
below this one. mpv is a *consumer* of `StreamSelection`, never a second place to
decide what to play.
**Destination on completion:**
[05-platform-backends.md](../architecture/05-platform-backends.md) — a "Native
Video Compositing (Desktop)" section beside the existing Android one, which this
mirrors; and [01-rust-backend.md](../architecture/01-rust-backend.md) — the
device profile becomes renderer-dependent, beside the stream-selection section.
**The spike is deleted in the same commit**, its three traps and its
hardware-decode table folded in; they are the durable half.
## Summary
mpv decodes and draws video on **every desktop platform**, composited beneath the
transparent webview, exactly as Android already does with ExoPlayer. The HTML5
`<video>` path and hls.js are then **deleted**, not merely bypassed.
The user-visible change is that most video stops being re-encoded by the server
before it can be watched. The change for whoever maintains this is that video
goes from three renderers to two.
## Motivation
### The transcode is a decoder constraint, not a rendering one
Desktop video goes through an h264 HLS transcode because the picture is drawn by
a WebKitGTK `<video>` element, and that element decodes little else. The device
profile therefore claims `h264` alone. That is not a statement about the machine
— the same machine runs mpv, which decodes essentially everything in the library
— it is a statement about which widget is holding the frame.
DR-227 made the cost measurable. Over 40 items negotiated against the development
server:
| Profile | Direct play |
|---|---|
| Desktop / WebKitGTK — `h264` only, 2ch | **7%** |
| Android / ExoPlayer — `h264,hevc,vp8,vp9,av1,mpeg4` + `ac3,eac3`, 6ch | **85%** |
The sampled library is ~80% hevc. **Those rows differ only by which component
decodes.** Moving the picture to mpv is what lets the desktop row claim what the
machine can actually do, and that — not the compositing — is the product.
### One desktop video path, not two
This is why the spec covers Windows rather than stopping at Linux.
Today video has **three** renderers: ExoPlayer, the WebKitGTK `<video>` element,
and (on Android, via the opt-out) that same element again. A Linux-only version
of this work would make it four, permanently: mpv on Linux, HTML5 on Windows,
ExoPlayer on Android, plus hls.js underneath the HTML5 one. Every seek strategy,
every track switch, every quality change, every lifecycle bug would then have one
more place to be got right — and the HTML5 path would survive indefinitely
because *something* would still need it.
Finishing the job removes that: **mpv on desktop, ExoPlayer on Android**, and
`hls.js`, `html5Adapter.ts`, `videoLoaderFor` and the webview video element all
go. The maintenance win is the reason Windows is in this spec and not in a
follow-up that never gets written.
### Three blockers are gone
1. **Compositing works, including Wayland.** The spike ran all six gates; the
2024 "not possible on Wayland at all" claim is out of date when the render API
is used instead of foreign-window embedding.
2. **There is no ABR to lose.** DR-228: the server's master playlist carries one
`EXT-X-STREAM-INF`. hls.js was demuxing, not adapting.
3. **A direct-play path exists.** It did not when the spike was written. DR-227
built it; DR-229 proved the contract is player-agnostic.
And on Windows specifically, `tauri-plugin-libmpv` lists Windows as its **fully
tested** platform — the inverse of the Linux situation the spike had to
disprove. The embedding difficulty was always WebKitGTK-specific.
## Layer assignment
| Logic / responsibility | Layer | Why it belongs there |
|---|---|---|
| **Which codecs this device can decode** | **Rust** | Domain: it is the input to Jellyfin's `PlaybackInfo` negotiation. It stops being a property of the *platform* and becomes a property of *the renderer in use* — see "The structural change". |
| Which backend renders video | **Rust** | Rust already owns this (`use_html5_element` / `VideoBackend`). It stops being a `cfg!` constant and becomes a runtime fact. |
| What stream to play (direct / remux / transcode, transport, ceiling) | **Rust — already decided** | DR-224. mpv consumes `StreamSelection`. Re-deriving any of it in a new backend would be the defect DR-224 exists to remove, restated. |
| Creating the GL surface, reparenting the webview, owning the render context | **Rust (platform layer)** | Native window and GL-context lifetime. Not presentation, and not expressible above the IPC boundary at all. |
| Render-context ↔ GL-context lifetime binding | **Rust** | A correctness invariant over native resources. DR-231. |
| Frame pacing (update callback, `report_swap`) | **Rust** | Timing against the compositor; mpv's own contract. |
| Hardware-decode selection | **Rust** | A capability question about the machine, answered from what mpv reports it actually selected. |
| Z-order of controls over video, overlay chrome, letterbox colour | **Frontend / mpv** | Presentation. Controls already draw over a transparent webview on Android; mpv paints its own letterbox bars (better than the Android equivalent, which shipped DR-194 as a defect). |
| Whether the surface is visible right now | **Frontend** | `nativeVideoActive` already exists and toggles `data-native-video`. Unchanged. |
### The structural change
Everything above is routine except one row, and it carries the whole benefit.
`video_codecs` in `build_device_profile` is a **compile-time constant per
platform**:
```rust
#[cfg(all(not(target_os = "android"), target_os = "linux"))]
let (video_codecs, audio_codecs) = ("h264".to_string(), "aac,mp3,opus,…");
```
That is correct only while a build has exactly one video renderer. It must be
derived from **which renderer will decode this stream**, which is runtime state.
It looks like configuration and is not: it is the input that decides whether the
server re-encodes, it changes when Jellyfin's API or our renderer changes, and
getting it wrong fails *silently* — a claimed codec the renderer cannot decode is
a black picture or silence, which is DR-148 and DR-227's audio override already.
**Write this against "the active video renderer", never `cfg!(target_os)`.** It
is the single piece that must not be Linux-shaped, because phase 2 reuses it
unchanged.
## Design
### Backend and compositing (DR-230, IR-033)
An `MpvVideoBackend` beside the existing `MpvBackend` (audio). The mpv side —
render context, FBO, update callback, hwdec — is **shared**; only the surface
differs per platform:
| Platform | Surface | Status |
|---|---|---|
| Linux (X11 + Wayland) | `GtkOverlay`: `GtkGLArea` main child, Tauri's webview reparented as overlay child, via `WebviewWindow::gtk_window()` / `default_vbox()` | Proven by the spike, bar the reparent |
| Windows | Native HWND child beneath a transparent WebView2 | Phase 2 |
`vo=libmpv` plus `mpv_render_context_create` with `MPV_RENDER_PARAM_OPENGL_FBO`.
Webview transparency via `with_transparent(true)` — no window-level transparency;
the spike showed it is neither used nor needed.
**The untested half of G1 is the reparent.** The spike built its own overlay; the
app must adopt Tauri's. Same widgets, one extra step, but it is the only place
Tauri-specific behaviour can still bite — so it is the first gate, proven before
anything is built on it.
Three traps from the spike, each of which cost a debugging cycle and each of
which looks like a platform limitation and is not:
1. **`LC_NUMERIC` must be reset *after* `gtk::init()`.** mpv refuses to start
under a non-C numeric locale. `mpv_backend.rs` already handles this but has no
GTK init in front of it; here `gtk::init()` applies the user's locale
afterwards and `mpv_create` returns null.
2. **libepoxy exports GL entry points as *data* symbols.** There is no `glFoo`
function — there is `epoxy_glFoo`, a variable holding a lazily-resolving
pointer. `get_proc_address` must return the pointer **stored at** that symbol;
returning the symbol's own address makes mpv jump into non-executable data and
take SIGSEGV on the first GL call. The `epoxy` crate does this correctly but is
unusable — its `gl_generator` dependency pulls a yanked `xml-rs`.
3. **Frame pacing is not optional and its symptom misleads.** See DR-232.
### Render-context lifetime (DR-231) — the crash defence
The spike's one unexplained SIGSEGV landed in a *decoder* thread with no Tauri,
GTK or GL frame in the stack, and three plausible causes failed to reproduce it
across ~13 minutes of targeted stress.
What is **not** unexplained is that the spike had no defence: it never calls
`mpv_render_context_free` and never tears down on `unrealize`, so nothing stopped
the GL context being recreated beneath the render context. That is DR-184 on
Android restated — a surface outliving its player.
Built as a requirement in its own right, not as a fix for a crash we cannot yet
reproduce:
- Render context created on `realize`, freed on `unrealize`, same thread, before
the GL context goes away.
- The update callback is unregistered **before** the context is freed, so a
callback cannot land on a freed context.
- Playback teardown and surface teardown are ordered, not racing.
If the crash recurs after this, it is a different bug and the likeliest cause is
out of the search space. If it does not, we needed this anyway.
### Frame pacing (DR-232)
Register `mpv_render_context_set_update_callback`; redraw only when it reports a
frame ready; call `mpv_render_context_report_swap` after each render.
Recorded because the failure mode is a trap: driving `queue_render()` off the
frame clock every tick without reporting the swap leaves mpv nothing to time
against. It looks fine in a window and **judders at fullscreen**, which reads as
a compositing or GPU limit and is neither.
### Renderer-dependent device profile (DR-233)
`build_device_profile` takes the active video renderer and derives the codec
lists from it:
| Renderer | Video codecs | Audio (video direct play) | Channels |
|---|---|---|---|
| mpv (desktop native) | `h264,hevc,vp8,vp9,av1,mpeg4` | platform list incl. `ac3,eac3` where the sink can voice it | from the audio route |
| WebKitGTK `<video>` | `h264` | webview-decodable set only | 2 |
| ExoPlayer (Android) | unchanged | unchanged | unchanged |
The existing `video_audio_codecs()` narrowing exists because *the webview decodes
a narrower audio set than the platform*. With mpv decoding, that no longer
applies to the video path — but the multichannel bound still does, since a 5.1
track direct-played into a 2-channel sink is silence or inaudible dialogue. Both
constraints stay, sourced from the renderer rather than assumed.
**This converts 7% into ~85%**, and it is also the change most able to break
playback silently — so it lands after compositing is proven, covered by the
DR-227 override tests.
### Deleting the webview video path (DR-234)
`get_player_status` stops reporting `use_html5_element: true` on desktop;
`supports_native_video` becomes true there.
Deletion is staged, because a path cannot be removed while a shipped platform
still needs it:
| Phase | Linux | Windows | HTML5 video path |
|---|---|---|---|
| 1 | mpv | HTML5 | alive — Windows needs it |
| 2 | mpv | mpv | alive but unreached |
| 3 | mpv | mpv | **deleted**, with hls.js |
Phase 3 is a real phase with its own acceptance criterion, not a "later". The
whole maintenance argument for including Windows collapses if the fork survives.
Android keeps ExoPlayer and keeps the webview as its documented opt-out; the
`<audio>` element and the background-audio handoff are untouched throughout.
**What happens when mpv fails to initialise.** With no HTML5 path there is no
silent fallback, and inventing one resurrects what we deleted. The
graceful-backend-init principle applies as written: fall back to the no-op
backend, emit `backend-init-failed`, and surface a real error rather than a black
rectangle. An honest failure beats a hidden downgrade to the transcode we are
trying to stop paying for.
### Hardware decode (DR-235)
The spike established the load-bearing fact: **hardware decode works through the
render API** (`hwdec-current` reported `nvdec-copy` on the discrete GPU), so the
direct-play prize is not traded for software decoding.
Policy is decided from what mpv reports it *selected*, never from what it was
asked for:
- Prefer zero-copy VA-API on the integrated GPU where the driver is present.
- `auto` reached for the discrete GPU in **copy-back** mode on a hybrid
Intel+NVIDIA laptop — the least efficient hardware path — so `auto` is a
fallback, not the default.
- `vaapi` silently fell back to software on the spike box because `vainfo` was
absent. A missing driver must be detected and logged, not mistaken for a
compositing limit.
- Log `hwdec-current` at start-up; knowing what was actually chosen is the whole
diagnostic value.
### Windows: what phase 2 actually costs (DR-236)
Not hidden, because it is the part most likely to be underestimated:
- **The surface is different code.** WebView2 in an HWND, not GTK. A transparent
WebView2 over a native child window is a solved arrangement, but DR-230's
Linux surface does not transfer. Everything else does.
- **libmpv is currently a Linux-only dependency**, and Windows is
**cross-compiled from Linux** via `x86_64-pc-windows-msvc` + `cargo-xwin`. Phase
2 must source a Windows libmpv (DLL + import library) into that cross-build and
ship the DLL in the NSIS bundle.
- **LGPL obligations follow the DLL.** DR-216 already records them for Linux:
keep the linkage dynamic, ship libmpv's licence text with any bundle carrying
it. The Windows bundle inherits both.
- **`bun run test:rust` and CI must still build.** Per the CI rule, any tool this
needs goes into the builder image and is pushed — never installed at job time.
Windows also gains a native *audio* decoder as a side effect, which is what
[windows-native-audio-backend.md](windows-native-audio-backend.md) wants and
cannot currently have. If that spec lands first, phase 2 inherits its build work
and shrinks to the surface.
## Out of scope
- **Android.** Unchanged in every respect.
- **macOS.** Not a shipped target. If it becomes one it joins phase 2's shape.
- **Audio backends.** mpv already plays audio on Linux; this adds a video
renderer beside it. Windows audio is its own spec.
- **HDR, tone mapping, multi-window.** Not exercised by the spike at all.
- **Re-deciding what stream to play.** DR-224 owns that. If this spec finds
itself choosing a URL, something has gone wrong.
## Acceptance criteria
**Phase 1 — Linux**
- [ ] Tauri's own webview reparents into the overlay (the untested half of G1),
on X11 **and** Wayland.
- [ ] Video plays, seeks and switches audio track in mpv, with the Svelte
controls composited over it and alpha blending intact.
- [ ] The render context is freed on `unrealize` and the update callback
unregistered before the free; a test demonstrates the ordering.
- [ ] A direct-play negotiation returns `DirectPlay` for an hevc source that
today returns `Transcode`, and it plays.
- [ ] Direct-play rate over the same 40-item sample rises from 7% toward the
Android figure. **Record the number.**
- [ ] mpv init failure emits `backend-init-failed` and surfaces an error rather
than falling back to a transcode.
- [ ] `hwdec-current` is logged and is not copy-back where zero-copy is available.
- [ ] A soak covering seek, track switch and fullscreen runs clean for an agreed
duration. **The spike's SIGSEGV is why this is a criterion.**
**Phase 2 — Windows**
- [ ] libmpv links in the `cargo-xwin` cross-build; the DLL and its licence ship
in the NSIS bundle; any new tool lives in the builder image, not in a CI step.
- [ ] Video plays composited under a transparent WebView2.
- [ ] The device profile, lifetime and hwdec code are **reused, not
reimplemented** — a reviewer confirms no `cfg!(target_os = "linux")` guards
them.
**Phase 3 — deletion**
- [ ] `use_html5_element` is false on every desktop platform.
- [ ] `hls.js` is gone from `package.json`; `html5Adapter.ts`, `videoLoaderFor`
and the `<video>` element are deleted; Android's opt-out and the
background-audio `<audio>` path still work.
**Throughout**
- [ ] `bun run check`, `bun run test`, `bun run format:check`, `bun run lint` pass.
- [ ] `cargo fmt` clean, `cargo clippy -D warnings` clean, `bun run test:rust` passes.
- [ ] `bun run check:boundary` passes, and a reviewer confirms no stream decision
was reconstructed in the new backend.
- [ ] `bindings.ts` regenerated from Rust.
- [ ] `bun run traces:validate` passes; coverage stays ≥ the CI ratchet.
- [ ] The spike and this spec are folded into
[05-platform-backends.md](../architecture/05-platform-backends.md) and both
deleted in the same commit.
## Testing
- **Rust, pure:** the device profile per renderer — mpv claims hevc, the webview
does not, the multichannel bound survives both. The DR-233 table as a
table-driven test.
- **Rust, pure:** `PlaybackInfo` fixtures that transcode under the webview
profile and direct-play under the mpv profile — the 7%→85% conversion as a unit
test, not only as a measurement.
- **Rust:** teardown ordering — callback unregistered before context freed, freed
before GL context destroyed. Structure it so the ordering is assertable without
a live GL context.
- **Frontend:** no desktop path selects an HTML5 video adapter. After phase 3,
the adapter does not exist and the test goes with it.
- **Manual / soak:** the criterion above. The spike's automated fullscreen and
resize soaks are reusable and already written.
## TRACES
| Piece | Tag |
|---|---|
| mpv video backend + compositing | `UR-080 \| DR-230, IR-033` |
| Render-context lifetime binding | `UR-080 \| DR-231` |
| Frame pacing | `UR-080 \| DR-232` |
| Renderer-dependent device profile | `UR-080, UR-070 \| DR-233` |
| Webview video path removed | `UR-080 \| DR-234` |
| Hardware-decode policy | `UR-080 \| DR-235` |
| Windows surface + cross-build | `UR-080 \| DR-236` |
## Notes for the implementer
- **Read the spike before writing a line.** Its three traps and its
hardware-decode table are the most valuable things in this directory, and each
cost a debugging cycle to find.
- **mpv consumes `StreamSelection`; it does not decide.** The transport is on the
queue item (DR-229). If you are parsing a URL, stop.
- **Guard nothing on `cfg!(target_os = "linux")` that phase 2 will need.** That is
the one avoidable mistake here.
- The Android backend is the reference for the *shape* of this — transparent
webview over a native surface at index 0. Read `05-platform-backends.md`'s
Android section for what shipped and what its defects were (DR-184 surface
lifetime, DR-194 letterbox).
- Do not call sync/blocking APIs from mpv event callbacks that can re-enter the
player or hold a lock. The existing deadlock gotchas apply.
- A parallel Claude session may be active in this repo — `git diff` before
"repairing" unexpected changes.
- This branch is stacked on backend-owned stream selection. Rebase when that
merges rather than merging master into it.
docs/specs/linux-native-video-spike.md
+48 -18
View File
@@ -2,7 +2,10 @@
**Status:** **Run 2026-08-21 — compositing works; G5 carries an open crash.**
The compositing claim it set out to test is falsified on Linux. See "Result".
This file stays open until the implementation spec exists; ABR is unresolved.
This file stays open until the implementation spec exists. **ABR is resolved** —
the playlist carries one `EXT-X-STREAM-INF`, so finding 3 is false and there is
no adaptation for mpv to lose. The remaining blocker is the unexplained SIGSEGV
under G5, which is a lifetime problem, not a compositing one.
**Requirements:** none allocated. This spike produces a decision record, not
product code — same shape as
[playback-backend-unification.md](playback-backend-unification.md), which is
@@ -258,10 +261,10 @@ anything.
Tauri's existing webview into an overlay. Low risk — the same widgets, one
extra reparent — but unproven, and it is the only place Tauri-specific
behaviour could still bite.
- 🔴 **ABR — finding 3's premise is in doubt.** Finding 3 says mpv would regress
streaming quality because "the webview path already has real ABR via hls.js".
Three pieces of evidence in this repo suggest that is **not true of the URLs we
actually build**:
- ✅ **ABR — resolved. Finding 3's premise is false.** Finding 3 said mpv would
regress streaming quality because "the webview path already has real ABR via
hls.js". Three pieces of evidence in this repo suggested that is **not true of
the URLs we actually build**:
1. `get_video_stream_url` (`repository/online.rs`) requests a *single*
rendition — one `VideoBitrate`, one `MaxStreamingBitrate`, one `MaxHeight`.
@@ -276,21 +279,48 @@ anything.
audio-track switch)". Manual selection by stream re-open is what you build
when there is no adaptation, and mpv can do the same thing.
**The decisive test has not been run** and needs a live server plus an API key:
count `#EXT-X-STREAM-INF` lines in a real `master.m3u8`. One line means there
is no ABR to lose and this blocker disappears. More than one means finding 3
stands and the work below applies.
**The decisive test has now been run** (2026-08-21, against the development
server, Jellyfin 10.11.5):
If ABR does turn out to be real, it belongs in **Rust**, not in mpv, and there
are three designs in increasing cost: pick the variant at open; re-open at a
new bitrate on sustained throughput drops (this is the quality-switch path the
app already has, so it is nearly free); or run a local proxy serving mpv a
synthesized single-variant playlist while swapping renditions underneath. The
middle option is almost certainly sufficient.
```
curl -s ".../Videos/<itemId>/master.m3u8?…&TranscodingProtocol=hls&…" \
| grep -c EXT-X-STREAM-INF
1
```
Either way the **direct-play path still does not exist** — every video play
currently goes through the HLS transcode endpoint. Building it is the real
project; the compositing work proven above is the smaller half.
**One line.** The playlist carries a single `EXT-X-STREAM-INF` plus an
`EXT-X-IMAGE-STREAM-INF` trickplay entry, which is not a rendition. Jellyfin
builds the master playlist from the rendition the request asked for; it does
not publish a ladder. So **there is no ABR to lose, and this blocker is
closed** — hls.js is serving as an HLS demuxer, exactly as (2) above supposed,
and mpv gives up nothing by replacing it.
Recorded as DR-228 (Won't Do) rather than deleted, because it is a
measurement: a server that *does* publish a ladder would change the answer, and
the re-negotiation path is the hook that work would build on.
**The direct-play path now exists.** It did not when this spike was written —
every video play went through the HLS transcode endpoint. Backend-owned stream
selection (DR-224 … DR-229) built it: Rust negotiates direct play / direct
stream / transcode and hands every backend one `StreamSelection` carrying the
URL, the transport and the chosen rendition. **That is the contract this
implementation consumes** — mpv is a consumer of a decision already made, not a
place to re-derive it.
It also sizes the prize precisely. Measured over the same server, 40 items
through a real negotiation per profile:
| Profile | Direct play |
|---|---|
| Linux / WebKitGTK — `h264` only, 2ch | **7%** |
| Android / ExoPlayer — `h264,hevc,vp8,vp9,av1,mpeg4` + `ac3,eac3`, 6ch | **85%** |
The library sampled is ~80% hevc. Linux sits at 7% **solely because the
WebKitGTK profile can only claim h264** — not because of anything about the
server or the negotiation. mpv decodes hevc, so widening the Linux device
profile once mpv renders the picture is what converts that 7% toward the
Android figure. That conversion is the actual product of this work; the
compositing proven above is the mechanism that permits it.
- 🔴 **One unexplained SIGSEGV.** A ~180s
run died in a *decoder* thread (libavcodec -> `av_log` -> libmpv's log handler
-> libc). No Tauri, wry, WebKitGTK, GTK or GL frame appears anywhere in the