Compare commits

...
Author SHA1 Message Date
dtourolle 2e3a864ef0 Merge branch 'master' into chore/infra-hardening
Traceability Validation / Check Requirement Traces (pull_request) Successful in 11s
2026-08-21 17:44:27 +00:00
dtourolle 1d56517f07 docs(specs): add backend-owned stream selection
Traceability Validation / Check Requirement Traces (pull_request) Successful in 11s
Rust becomes the single owner of which stream to play — direct play or
transcode, at what ceiling, over what transport — and hands every player
backend a self-describing StreamSelection instead of a bare URL. mpv,
ExoPlayer and the HTML5/hls.js path all consume one decision rather than
three places re-deriving it.

The motivating leak is concrete. VideoPlayer.svelte determines transport with
`currentStreamUrl.includes(".m3u8")`, in two places, for a URL Rust
constructed and therefore already knows the shape of. That is the boundary
rule in miniature: not item-type taxonomy, but the same error of
reconstructing a domain fact in the presentation layer because the wire shape
did not carry it. A tagged Transport enum deletes it.

The design line, which ExoPlayer forces: Rust decides *what stream*, the
player decides *how to deliver it*. ExoPlayer has genuine adaptive track
selection; this spec must not reimplement or fight it. Rust only adapts where
the player cannot (mpv) and the server actually offers a ladder.

Six phases, and phase 1 stands alone as pure ownership movement with no
behaviour change. Phase 4 (direct-play negotiation) is what removes the
transcode and unblocks the Linux native-video work. Phase 5 (adaptation) is
gated on counting EXT-X-STREAM-INF entries in a real playlist — the
acceptance criteria require that count be recorded before it is either
started or dropped.

Takes DR-121 from read-through-media-cache.md, which specced Rust-owned
quality reporting but never built it; that spec keeps its capture half.
2026-08-21 19:41:32 +02:00
dtourolle 99d96163d8 docs(specs): correct the spike's crash and ABR findings
Three corrections to the Linux native-video spike, each of which reverses
something recorded earlier in the same session:

- The crash is unexplained. It was first blamed on hwdec=auto-safe's Vulkan
  failures, on a misreading of the logs — those are two per run at start-up,
  not per-frame, and every clean run has the same two. A 300s soak on
  auto-safe survived. So did 240s of automated fullscreen toggling (~120
  transitions) and 240s of continuous resizing (~2000 reallocations). Three
  hypotheses, none reproduced. Recorded rather than dismissed: an
  intermittent fault nobody can reproduce is worse to inherit than a
  deterministic one.

- G5 drops to amber. It looked and felt smooth, but the only SIGSEGV observed
  came from the only session in which fullscreen was exercised, and the spike
  has no lifecycle handling at all — it never frees the render context. An
  implementation must bind that to the GL context's lifetime regardless of
  what caused this crash, because Android already paid for that lesson as
  DR-184.

- Finding 3's premise is in doubt. "The webview path already has real ABR via
  hls.js" was never checked against the URLs this app builds:
  get_video_stream_url requests a single rendition, the frontend has no
  level-handling code at all, and a quality switch is implemented by
  re-opening the stream. If the playlist is single-variant there is no
  adaptation to lose. The decisive test needs a live server and is recorded
  as unrun.

Also records hardware decode working through the render API (nvdec-copy
engaged), and that hwdec=vaapi silently fell back to software on this box.
2026-08-21 19:41:21 +02:00
dtourolle eda6e36d3d Merge pull request 'Infrastructure hardening: CI enforcement, supply chain, updater, diagnostics' (#14) from chore/infra-hardening into master
Publish Documentation / Build & publish docs to gitea-pages (push) Canceled after 0s
🏗️ Build and Test JellyTau / Run Tests (push) Successful in 15m0s
🏗️ Build and Test JellyTau / Supply Chain (push) Successful in 38s
Traceability Validation / Check Requirement Traces (push) Successful in 13s
🏗️ Build and Test JellyTau / Android Compile Check (push) Successful in 4m5s
Reviewed-on: #14
2026-08-21 17:33:53 +00:00
dtourolle f11f5eddd5 docs(specs): record the Linux native-video compositing spike
🏗️ Build and Test JellyTau / Run Tests (pull_request) Successful in 15m3s
🏗️ Build and Test JellyTau / Supply Chain (pull_request) Successful in 38s
Traceability Validation / Check Requirement Traces (pull_request) Successful in 20s
🏗️ Build and Test JellyTau / Android Compile Check (pull_request) Successful in 4m11s
Adds the spike write-up and re-opens finding 2 of
playback-backend-unification.md, which concluded that video cannot unify
on a native engine because the webview owns the surface.

That finding's general form has since been falsified on Android, where
native video composites behind a transparent WebView and ships on by
default. The evidence behind it was also entirely about foreign-window
embedding -- mpv's render API, drawing into a GL context we own inside
Tauri's own GTK tree, was never tested. The spike tests that one claim
and comes back green on Linux for both X11 and Wayland, bar the Tauri
default_vbox() half of G1.

Findings 3-6 are deliberately left standing. Finding 3 in particular --
mpv has no adaptive bitrate -- is an independent disqualifier that a
green compositing result does not clear, and the spike says so rather
than reading as a green light.

The next-free-id line moves to UR-079 / IR-033 / DR-219: this branch
allocated UR-077 and UR-078 for the updater and diagnostics work, and
DR-215 through DR-218 with them, after that line was last written.

Authored in a parallel session working in the same checkout; committed
here so it travels with the rest of the branch.
2026-08-21 19:06:13 +02:00
5 changed files with 739 additions and 2 deletions
+2
View File
@@ -32,6 +32,8 @@
- [Spec Template](specs/SPEC-TEMPLATE.md) - [Spec Template](specs/SPEC-TEMPLATE.md)
- [Spec Review Checklist](specs/SPEC-REVIEW-CHECKLIST.md) - [Spec Review Checklist](specs/SPEC-REVIEW-CHECKLIST.md)
- [Playback Backend Unification](specs/playback-backend-unification.md) - [Playback Backend Unification](specs/playback-backend-unification.md)
- [Linux Native Video Spike](specs/linux-native-video-spike.md)
- [Backend-Owned Stream Selection](specs/backend-owned-stream-selection.md)
- [Player Facade Enforcement](specs/player-facade-enforcement.md) - [Player Facade Enforcement](specs/player-facade-enforcement.md)
- [Windows Native Audio Backend](specs/windows-native-audio-backend.md) - [Windows Native Audio Backend](specs/windows-native-audio-backend.md)
- [libmpv2 Migration](specs/libmpv2-migration.md) - [libmpv2 Migration](specs/libmpv2-migration.md)
+4 -2
View File
@@ -27,8 +27,8 @@ know how something *works*, read
| Design authority | No code of its own — it records a decision later specs act on. | | Design authority | No code of its own — it records a decision later specs act on. |
**Next free requirement ids** (always re-check **Next free requirement ids** (always re-check
[requirements.md](../requirements.md) before allocating): **UR-077**, [requirements.md](../requirements.md) before allocating): **UR-079**,
**IR-033**, **DR-215**. Three specs below suggested ids that have since been **IR-033**, **DR-219**. Three specs below suggested ids that have since been
taken by other work; each carries a ⚠️ note at the top. taken by other work; each carries a ⚠️ note at the top.
## Partially implemented ## Partially implemented
@@ -44,9 +44,11 @@ taken by other work; each carries a ⚠️ note at the top.
| Spec | Blocked on / note | | Spec | Blocked on / note |
|---|---| |---|---|
| [backend-owned-stream-selection.md](backend-owned-stream-selection.md) | Rust owns direct-play-vs-transcode, transport and quality; players consume one `StreamSelection`. Phase 1 (delete the `.m3u8` sniff) stands alone. Unblocks Linux native video. |
| [build-provenance.md](build-provenance.md) | `build.rs` is still bare. ⚠️ suggested id DR-093 is taken. | | [build-provenance.md](build-provenance.md) | `build.rs` is still bare. ⚠️ suggested id DR-093 is taken. |
| [player-facade-enforcement.md](player-facade-enforcement.md) | ~60 `commands.player*` sites still outside the facade; no lint rule. ⚠️ suggested id DR-095 is taken. | | [player-facade-enforcement.md](player-facade-enforcement.md) | ~60 `commands.player*` sites still outside the facade; no lint rule. ⚠️ suggested id DR-095 is taken. |
| [windows-native-audio-backend.md](windows-native-audio-backend.md) | Blocked on the libmpv2 swap. ⚠️ suggested id IR-030 is taken. | | [windows-native-audio-backend.md](windows-native-audio-backend.md) | Blocked on the libmpv2 swap. ⚠️ suggested id IR-030 is taken. |
| [linux-native-video-spike.md](linux-native-video-spike.md) | **Spike run 2026-08-21: compositing works on Linux, X11 and Wayland.** G1-G6 green bar the Tauri `default_vbox()` half of G1. Needs an implementation spec that answers adaptive bitrate. |
## Design authority ## Design authority
@@ -0,0 +1,242 @@
# Spec: Backend-owned stream selection
**Status:** Proposed
**Requirements:** UR-079 (new) → DR-219 … DR-224 (new); **implements and extends
DR-121**, currently allocated to
[read-through-media-cache.md](read-through-media-cache.md) and not started.
Re-check `requirements.md` before allocating — the ids moved twice while this was
being written (`DR` max was 215, then 218).
**UX spec:** the quality selector in `VideoPlayer.svelte` already exists; this
changes what fills it, not how it looks.
**Supersedes / revises:** takes DR-121 out of
[read-through-media-cache.md](read-through-media-cache.md), which should keep
only its capture/eviction half. Unblocks
[linux-native-video-spike.md](linux-native-video-spike.md).
**Destination on completion:**
[01-rust-backend.md](../architecture/01-rust-backend.md) — extends the
"Streaming quality ladder" section; and
[03-data-flow.md](../architecture/03-data-flow.md) — playback initiation. The
durable half is the layer line and the `StreamSelection` contract; phases and
acceptance criteria are disposable.
## Summary
Make Rust the single owner of *which stream to play* — direct play or transcode,
at what ceiling, over what transport — and hand every player backend a
self-describing selection instead of a bare URL. mpv, ExoPlayer and the HTML5
`<video>`/hls.js path all become consumers of the same decision rather than three
places that re-derive it.
Nothing about how playback *looks* changes. What changes is that the frontend
stops inferring transport from a URL string, and that direct play becomes
possible at all.
## Motivation
Four concrete problems, all the same shape.
**1. The frontend sniffs transport out of the URL.**
[VideoPlayer.svelte:569](../../src/lib/components/player/VideoPlayer.svelte#L569):
```ts
const isHlsStream = currentStreamUrl.includes(".m3u8");
```
and again inline at line 2364. Rust *built* that URL and knows exactly what it
is; the frontend re-derives it by substring match. Change the endpoint, add a DASH
path, serve a progressive file, and this silently picks wrong. This is the
boundary rule in miniature — not item-type taxonomy, but the same error: a
domain fact reconstructed in the presentation layer because the wire shape did
not carry it.
**2. There is no direct-play path.** `get_video_stream_url` always builds an HLS
transcode URL (`TranscodingProtocol=hls`, `VideoCodec=h264` first). Every video
play burns server CPU, even when the file would play untouched. This is the cost
the Linux native-video work exists to remove, and it cannot be removed without a
decision that does not currently exist anywhere in the codebase.
**3. Quality is a process-wide global.** `streaming_quality()` /
`set_streaming_quality()` in `repository/online.rs` read and write a static.
It is not per-session or per-item, so it cannot express "this 4K remux needs a
ceiling, that podcast does not", and two concurrent playbacks would share one
setting.
**4. Rust cannot say what qualities *this* media source supports.** The selector
is populated from a fixed enum rather than from what the source actually offers.
DR-121 already names this; it has not been built.
### The prior question
Finding 3 of [playback-backend-unification.md](playback-backend-unification.md)
holds that hls.js gives us real adaptive bitrate and mpv would lose it. Evidence
in this repo suggests **there is no ABR today**: a single rendition is requested,
no level-handling code exists anywhere in the frontend, and a quality switch is
implemented by re-opening the stream.
**Run this before sizing the adaptation work.** It needs a live server:
```
curl -s "https://<server>/Videos/<itemId>/master.m3u8?api_key=<key>&…" \
| grep -c EXT-X-STREAM-INF
```
`1` → there is no adaptation to preserve, and the adaptation half of this spec
collapses to "pick well at open". `>1` → finding 3 stands and DR-223 applies.
**Everything else in this spec is worth doing either way** — the ownership
problems above are independent of the answer.
## Layer assignment
| Logic / responsibility | Layer | Why it belongs there |
|---|---|---|
| Direct play vs direct stream vs transcode | Rust | Depends on Jellyfin's `PlaybackInfo`, container/codec support and the device profile. Changes when Jellyfin's API or our profile changes → domain, by the litmus test. |
| Transport of the chosen stream (HLS / progressive / local file) | Rust | Rust constructs the URL; it is the only place that *knows* rather than infers. Today the frontend guesses from `.m3u8`. |
| Which qualities this media source can offer | Rust | Derived from the source's own streams and the quality→transcode-parameter mapping that `get_video_download_url` already holds. DR-121. |
| The quality ceiling in force, per playback session | Rust | Domain state that outlives any one view and must survive a backend swap or a mode transfer. Currently a process-wide static. |
| Deciding to re-negotiate mid-playback (if adaptation is needed) | Rust | It performs the HTTP and already derives reachability from real traffic via `ConnectivityMonitor`. Throughput estimation is the same pattern on the same data — a side-channel probe would repeat the mistake that principle exists to prevent. |
| Frame-level delivery *within* the selected stream, including a player's own ABR | **Player** | ExoPlayer has genuine adaptive selection; if Rust hands it a multi-variant playlist it should use it. Rust chooses *what to request*, never how a player paces bytes. See "The line". |
| Rendering the selector, showing the current quality, ordering the list | Frontend | Pure presentation over a backend-supplied list. |
| Poster, letterbox, controls, overlay z-order | Frontend | Unchanged. |
### The line
**Rust decides *what stream*. The player decides *how to deliver it*.**
This matters most for ExoPlayer, which already does real adaptive track selection
over HLS. This spec must not reimplement that or fight it — if a multi-variant
playlist reaches ExoPlayer, ExoPlayer adapts and Rust stays out of the way. The
same restraint applies to any future backend that gains the capability. Rust only
steps in where the player has no such ability (mpv) *and* the server actually
offers a ladder.
Borderline row, with its tie-breaker: "which media source of a multi-source item"
looks like a user choice, and its *presentation* is. The default and the
constraint set are domain → **Rust**, per the borderline-defaults-to-Rust rule.
## Design
### The contract
One self-describing selection replaces the bare URL. Nested fields are
camelCase over the wire (`#[serde(rename_all = "camelCase")]`); the enums are
tagged so the frontend matches a tag instead of parsing a string.
```rust
#[derive(Serialize, Type)]
#[serde(rename_all = "camelCase")]
pub struct StreamSelection {
pub url: String,
pub transport: Transport,
pub playback_kind: PlaybackKind,
/// The negotiated rendition; None when direct-playing the source as-is.
pub rendition: Option<Rendition>,
/// What this media source can offer — fills the selector (DR-121).
pub available: Vec<QualityOption>,
}
#[derive(Serialize, Type)]
#[serde(tag = "type", rename_all = "camelCase")]
pub enum Transport { Hls, Progressive, LocalFile }
#[derive(Serialize, Type)]
#[serde(tag = "type", rename_all = "camelCase")]
pub enum PlaybackKind { DirectPlay, DirectStream, Transcode }
```
`Transport` is the field that deletes the `.m3u8` sniff. The frontend picks
hls.js on `Hls` and the element's own loader otherwise — a tag match, not a
substring search.
### Re-negotiation
Rust emits `stream-selection-changed` (kebab-case, per convention) carrying a new
`StreamSelection` plus the position to resume at. The existing
`playerSetStreamQuality` response already has exactly the right shape — a tagged
`strategy` that tells the caller who reloads, with the backend handling native
itself and handing HTML5 a URL for `reloadSource`
([index.ts:198](../../src/lib/player/index.ts#L198)). **Extend that; do not
invent a second mechanism.** It is the one piece of this that is already right.
Note the existing wart to preserve or fix deliberately, not accidentally:
tauri-specta keeps those response fields snake_case (`new_url`), and the facade
comments say so.
### Phases
1. **DR-219** `StreamSelection` + `Transport`; delete the `.m3u8` sniff. No
behaviour change — pure ownership move, and independently shippable.
2. **DR-220** Per-session quality ceiling replacing the `online.rs` static.
3. **DR-221** `available` populated from the media source (DR-121's substance).
4. **DR-222** Direct-play/direct-stream negotiation via `PlaybackInfo`. This is
the phase that unlocks native video and removes the transcode.
5. **DR-223** Adaptation, **only if the playlist check says a ladder exists**.
Cheapest sufficient design: re-negotiate on sustained throughput drop, reusing
the phase-1 re-negotiation path. A local proxy synthesizing a single-variant
playlist is a last resort, not a starting point.
6. **DR-224** ExoPlayer and mpv consume `StreamSelection` unchanged, proving the
contract is player-agnostic rather than HTML5-shaped.
Phases 14 stand on their own merits with no dependency on the ladder question.
## Out of scope
- Rendering, compositing, and the Linux native-video work itself. This spec
unblocks [linux-native-video-spike.md](linux-native-video-spike.md); it does
not contain it.
- Replacing hls.js. It stays as the HLS loader for the webview path.
- Reimplementing or overriding ExoPlayer's own adaptive selection. See "The line".
- The download/capture half of [read-through-media-cache.md](read-through-media-cache.md)
(DR-122, DR-124, DR-125), which keeps its own spec.
- Audio. The same argument applies, but video is where the transcode cost is.
## Acceptance criteria
- [ ] The `.m3u8` substring check is gone from `VideoPlayer.svelte` (both sites)
and transport comes from the tagged enum.
- [ ] `bun run check`, `bun run test`, `bun run format:check`, `bun run lint` pass.
- [ ] `cargo fmt` clean, `cargo clippy -D warnings` clean, `bun run test:rust` passes.
- [ ] `bun run check:boundary` passes — and the reviewer confirms by reading that
no transport/kind decision was reconstructed in `src/`, since the tripwire
only catches item-type array literals.
- [ ] `bindings.ts` regenerated from Rust, not hand-edited.
- [ ] New code carries `// TRACES:` comments; `bun run traces:validate` passes and
coverage stays ≥ the CI ratchet.
- [ ] The `EXT-X-STREAM-INF` count is recorded in this spec before DR-223 is
started or dropped.
- [ ] DR-121 is removed from `read-through-media-cache.md` with a pointer here.
## Testing
- Rust: `PlaybackInfo` fixtures → expected `PlaybackKind`, one per branch
(supported container direct-plays; unsupported codec transcodes; a ceiling
below the source bitrate transcodes even when the codec is fine).
- Rust: `Transport` round-trips through serde with the tag the frontend matches.
- Frontend: adapter selection driven by `transport`, including the case a URL
ending `.m3u8` is served as `Progressive` — that test fails on today's code,
which is the point.
- Extend `tauriIntegration.test.ts` for the new command params (camelCase rule).
- No test asserts a URL substring.
## TRACES
| Piece | Tag |
|---|---|
| `StreamSelection` / `Transport` | `UR-079 \| DR-219` |
| Per-session ceiling | `UR-074 \| DR-220` |
| `available` from media source | `UR-079 \| DR-221, DR-121` |
| Direct-play negotiation | `UR-079 \| DR-222` |
| Adaptation, if built | `UR-079 \| DR-223` |
| ExoPlayer/mpv consumers | `UR-003, UR-004 \| DR-224` |
## Notes for the implementer
- **Phase 1 is worth doing on its own**, even if everything after it is dropped.
It removes a real leak and costs almost nothing.
- Do not frame any phase as "no Rust changes required" — that framing is what
produced the leak `scoped-search-boundary.md` records.
- `ConnectivityMonitor` is the precedent for DR-223: derive network facts from
real traffic, never from a side-channel poller.
- A parallel Claude session may be active in this repo — `git diff` before
"repairing" unexpected changes. Requirement ids in particular moved twice
during the writing of this spec.
+469
View File
@@ -0,0 +1,469 @@
# Spec: Linux native video — bounded compositing spike
**Status:** **Run 2026-08-21 — compositing works; G5 carries an open crash.**
The compositing claim it set out to test is falsified on Linux. See "Result".
This file stays open until the implementation spec exists; ABR is unresolved.
**Requirements:** none allocated. This spike produces a decision record, not
product code — same shape as
[playback-backend-unification.md](playback-backend-unification.md), which is
Accepted with no requirement ids of its own. Ids are allocated by the
*implementation* spec that follows a green result.
**UX spec:** n/a
**Supersedes / revises:** re-opens finding 2 of
[playback-backend-unification.md](playback-backend-unification.md) on Linux only.
Its findings 3, 4, 5 and 6 stand unchallenged and are **not** in scope here.
**Destination on completion:**
[05-platform-backends.md](../architecture/05-platform-backends.md) — a "Native
Video Compositing (Linux)" section alongside the existing Android one. The
durable half is the mechanism and the two traps below; the gates and phases are
disposable.
## Summary
Test one falsifiable claim: *a native video surface cannot be composited with a
Tauri webview on Linux.* The claim is load-bearing — it is why Linux video goes
through an h264 HLS transcode into a WebKitGTK `<video>` element instead of
decoding directly in the mpv instance we already run. The spike renders one mpv
frame beneath the webview, on both X11 and Wayland, and stops. It ships no
product code and flips no defaults.
A green result does **not** authorise native video on Linux; it authorises
writing the spec that would.
## Motivation
[playback-backend-unification.md](playback-backend-unification.md) finding 2
concluded that native video cannot be composited with a Tauri webview, on
evidence from `tauri-plugin-libmpv`'s platform table, wry#284, tauri#6343, and a
Tauri maintainer's 2024 statement that a GTK widget as a child X11 window is
"a bit hacky and it is not possible on Wayland at all."
Two things have changed since that was written, and one thing was never tested.
**1. The general claim has already been falsified on one platform — by us.**
Android now renders ExoPlayer video on a TextureView at index 0 *behind a
transparent Tauri WebView*, with the Svelte controls drawn over it, on by
default. See
[05-platform-backends.md](../architecture/05-platform-backends.md#native-video-compositing-android).
That is exactly the composition finding 2 said was impossible, shipped. What
survives of the finding is a narrower, WebKitGTK-specific claim — which is worth
testing on its own terms rather than inheriting.
**2. A Tauri app now ships Linux native mpv as an active platform.**
[MaxVideoPlayer](https://github.com/MaxMB15/MaxVideoPlayer) (354 commits) embeds
libmpv via **EGL + X11 child window / Wayland subsurface**, with Linux and macOS
active and Windows only planned — the inverse of the plugin matrix finding 2
sampled. Its existence does not prove our case works, but it does mean the
Wayland half of the maintainer quote is out of date.
**3. The render API was never tested.** Every source in finding 2 describes
*foreign-window embedding*: `--wid`, child windows, a second toplevel
position-synced to a `getBoundingClientRect()` div. That is a different mechanism
from mpv's render API, where **we** own the GL context and mpv draws into an FBO
we hand it (`mpv_render_context_create` / `mpv_render_context_render`, with an
upstream [GTK example](https://github.com/mpv-player/mpv-examples/pull/44/files)).
Tauri v2 exposes `WebviewWindow::gtk_window()` and `default_vbox()`, so the
target is a widget inside Tauri's own GTK tree — not a foreign window, not a
second toplevel, and therefore not the thing that was found broken.
The prize is direct play: no h264 transcode, hardware decode, libass subtitles,
and no server CPU burned on every Linux play.
## The blocker a green spike does not clear
🔴 **Read this before treating a green result as a green light.**
Finding 3 of the unification spec stands: **mpv has no adaptive bitrate.** It
delegates HLS to FFmpeg's demuxer, which picks one variant at open and never
adapts. The webview path has real ABR via hls.js. Compositing is necessary for
native video on Linux; it is not sufficient.
There is a plausible answer, and this spike exists partly to make it testable:
**ABR only matters on the transcode path.** A direct-played file has no variant
ladder to adapt between — the adaptation the server offers *is* the transcode.
So "mpv when the stream is direct-play, HTML5 + hls.js when the server
transcodes" would sidestep finding 3 rather than fight it, and it maps onto a
decision Rust already makes when it builds the stream URL.
That is a **hypothesis, not a conclusion.** It is out of scope here. Record it in
the spike's decision note so the follow-up spec starts from it.
## Layer assignment
The spike introduces no product logic. The table below is the assignment the
*follow-up* would inherit, written now so a green result cannot drift into
frontend decisions during implementation.
| Logic / responsibility | Layer | Why it belongs there |
|------------------------|-------|----------------------|
| Which backend renders video on this platform (`use_html5_element`, `supports_native_video`) | Rust | Already there — `get_player_status` in `commands/player/mod.rs` computes it from a `cfg!`. The spike would widen that `cfg!`, not relocate the decision. The frontend already consumes it via `createAdapter`. |
| Whether *this stream* is direct-play or transcoded, and therefore whether mpv or hls.js renders it | Rust | Domain. It depends on Jellyfin's `PlaybackInfo` response, container/codec support, and the bitrate cap — all of which change when Jellyfin's API or our quality ladder changes. The frontend must never re-derive it from a URL shape. |
| Creating, sizing, and destroying the GL surface; the mpv render context | Rust | Owns the backend and the GTK window handle. There is no presentation decision in it. |
| Where controls, subtitles, and the mini-player sit above the video, and the letterbox/poster treatment | Frontend | Pure presentation; changes only if the UI is redesigned. Precisely the split the Android path already uses. |
| Reserving the video rectangle in layout and marking the shell transparent | Frontend | Presentation. `nativeVideo.ts` + the `[data-native-video="active"]` rule in `app.css` already do this for Android and are platform-agnostic. |
Borderline row, stated with its tie-breaker: *"is the surface currently
attached?"* reads like view state, but the Android work found that a surface left
in the hierarchy outlives its player (DR-184). Attachment is backend lifecycle →
**Rust**, with the frontend told about it, not asked.
## Design
A throwaway branch. No merge to `master` except the decision note.
### What gets built
One `#[cfg(target_os = "linux")]` experiment behind a feature flag, in a scratch
binary or an ignored test — **not** in `MpvBackend`'s constructor path:
1. From `app.get_webview_window(...)`, take `gtk_window()` and `default_vbox()`.
2. Reparent the webview into a `gtk::Overlay`: `GLArea` as the main child, the
webview as the overlay child.
3. Set the webview background to fully transparent (wry does this when
`"transparent": true`; verify it reaches `webkit_web_view_set_background_color`).
4. In the `GLArea`'s `render` signal, drive
`mpv_render_context_render` with `MPV_RENDER_PARAM_OPENGL_FBO` pointing at the
FBO GTK bound for us.
5. Play one local file. Draw an opaque HTML element over the video area.
`video = no` and `audio-display = no` are set in
[mpv_backend.rs:135-141](../../src-tauri/src/player/mpv_backend.rs#L135-L141);
the spike overrides them on its own `Mpv` handle rather than editing that path.
### Bindings
The current pin is `libmpv = { git = "…/libmpv-rs", branch = "master" }` — the
dead pin [libmpv2-migration.md](libmpv2-migration.md) exists to replace. The
render API lives in `libmpv2-sys` (`mpv_render_context_render`); the safe wrapper
was only ever a PR against the old crate. **Use `libmpv2-sys` raw FFI directly in
the spike.** Do not block the spike on the migration, and do not let the spike
half-perform it — if the spike goes green the migration becomes a hard
prerequisite of the implementation, which is the ordering
[windows-native-audio-backend.md](windows-native-audio-backend.md) already sits
in.
### IPC
None. The spike crosses no boundary. If it goes green, the follow-up changes only
the *value* of the existing `useHtml5Element` / `supportsNativeVideo` fields — no
new wire shapes, no `bindings.ts` regeneration.
## Gates
Each is pass/fail with a named failure. Stop at the first red and write it up —
a red result is a successful spike.
| # | Question | Fails if |
|---|---|---|
| G1 | Can a custom GTK widget join Tauri's widget tree and survive the window's lifetime? | `default_vbox()` is absent/unusable, or reparenting the webview breaks input or crashes. |
| G2 | Does the webview still paint, with a transparent backdrop, over that widget? | The backdrop renders opaque black ([wry#1540](https://github.com/tauri-apps/wry/issues/1540)) or the webview stops repainting ([tauri#12800](https://github.com/tauri-apps/tauri/issues/12800)). **This is the highest-risk gate.** |
| G3 | Does mpv render a frame into our FBO? | The render context refuses GTK's context, or frames land in the wrong buffer. |
| G4 | Does HTML drawn over the video area actually appear over it? | Video covers the controls — the exact failure wry#284 and tauri#6343 report. Without this, the whole thing is worthless: our controls, subtitles and mini-player all sit over the video. |
| G5 | Does it survive resize, fullscreen, and SPA navigation away and back? | Flicker on resize, or a surface that outlives its route. |
| G6 | Does it hold on **both** X11 and Wayland? | Either session backend fails. Wayland is the one the 2024 maintainer quote says is impossible — test it first, not last. |
G6 is not a nice-to-have. A result that only holds on X11 is red for a project
shipping to current desktops.
### Time box
If G1G4 are not all green, stop and write the result up. The value of this spike
is a dated, method-specific answer — including "still no, and here is the
mechanism" — not a working player.
## Result (2026-08-21)
Run on GNOME, kernel 7.1.8, libmpv 2.5.0 (mpv 0.41.0), GTK 3.24.52, WebKitGTK
2.52.6, wry 0.53.5 — the versions `src-tauri/Cargo.lock` resolves. Spike source:
a ~250-line standalone crate using wry + gtk + `libmpv2-sys` raw FFI, driving
mpv's render API with an update callback, frame-gated repaints and
`report_swap`.
| Gate | Result | Observed mechanism |
|---|---|---|
| G1 widget in GTK tree | 🟡 **partial** | `GtkOverlay` with `GtkGLArea` as main child and the wry webview as overlay child works, built directly. **Tauri's own `default_vbox()` was not exercised** — see below. |
| G2 webview paints transparently over it | ✅ green | `with_transparent(true)` alone. No window-level transparency was used or needed. |
| G3 mpv renders into our FBO | ✅ green | `vo=libmpv` + `mpv_render_context_create` with `MPV_RENDER_PARAM_OPENGL_FBO` into the FBO GTK binds. |
| G4 HTML over video | ✅ green | Opaque panel and a translucent control bar both drew over moving video. |
| G5 resize / drag / fullscreen | 🟡 **green on appearance, suspect underneath** | No flicker, gap or misalignment, and smooth once frame pacing was correct (trap 3). But the only crash observed came from the only session where fullscreen was exercised — see "What is still open". |
| G6 X11 **and** Wayland | ✅ green | Identical on both; `GDK_BACKEND` flipped between runs. |
**Finding 2 of [playback-backend-unification.md](playback-backend-unification.md)
is false on Linux** when tested by the render API rather than by foreign-window
embedding. Wayland — the half the 2024 maintainer quote called impossible — is
green.
Better than the gate asked for: the translucent bar composited *alpha* against
the video, not merely opaque-over. Scrims, gradient fades and subtitle backdrops
therefore work, which is most of how a player UI actually looks. mpv also painted
the letterbox bars black on its own — the Android equivalent was a shipped defect
(DR-194).
### Three traps, each of which cost a debugging cycle
Carry these into the implementation; each produced a failure that looked like a
platform limitation and was not.
1. **`LC_NUMERIC` must be reset *after* `gtk::init()`, not before.** mpv refuses
to start under a non-C numeric locale. `mpv_backend.rs` already handles this,
but it has no GTK init in front of it; on this path `gtk::init()` applies the
user's locale afterwards and `mpv_create` returns null.
2. **libepoxy exports GL entry points as *data* symbols.** There is no `glFoo`
function to resolve — there is `epoxy_glFoo`, a variable holding a lazily
resolving function pointer. `get_proc_address` must return the pointer *stored
at* that symbol; returning the symbol's own address makes mpv jump into
non-executable data and take SIGSEGV/SEGV_ACCERR on the first GL call. The
`epoxy` crate does this correctly but is unusable — its `gl_generator`
dependency pulls a yanked `xml-rs`.
3. **Frame pacing is not optional, and its symptom is misleading.** Driving
`queue_render()` off the widget's frame clock on every tick, without calling
`mpv_render_context_report_swap` after each render, leaves mpv with nothing to
time against. Playback looks fine in a window and **judders at fullscreen**
which reads as a compositing or GPU limit and is neither. The fix is to
register `mpv_render_context_set_update_callback`, redraw only when it says a
frame is ready, and report the swap afterwards. Fullscreen was smooth
immediately once both were in place.
### Hardware decode through the render API
Tested by asking mpv what it actually selected (`hwdec-current`), not what it was
asked for. All three ran 20s clean at a steady 30 fps.
| `hwdec` | `hwdec-current` | Note |
|---|---|---|
| `vaapi` | `no` | **Did not engage** on this box — silently fell back to software. `vainfo` is not installed, so the libva driver for the Iris Xe iGPU is likely absent. No render-API error; this looks like a missing driver package, not a compositing limit. |
| `auto` | `nvdec-copy` | Hardware decode **does** work through the render API, on the discrete RTX 3050. Copy-back rather than zero-copy interop. |
| `no` | `no` | Software. Clean baseline. |
The load-bearing result is the middle row: **hardware decode is compatible with
mpv's render API**, so the direct-play prize is real and not traded away for
software decoding. Which decoder to prefer is an implementation question — on a
hybrid Intel+NVIDIA laptop `auto` reached for the discrete GPU in copy-back mode,
which is the least efficient hardware path. An implementation should evaluate
zero-copy VA-API on the iGPU (after confirming the driver is installed) before
accepting `auto`.
`hwdec=auto-safe` probes Vulkan video decode, which this GPU does not support.
It logs two `Failed setup for format vulkan` / `no frame!` pairs at start-up and
then settles on `nvdec-copy` — the same place `auto` lands. A first reading of
these logs mistook the start-up pair for a per-frame flood; **it is not**. Every
run, clean or crashed, contains exactly two. `auto-safe` is not implicated in
anything.
### What is still open
- **The Tauri half of G1.** The spike built its own `GtkOverlay`. The app must
instead reach `WebviewWindow::gtk_window()` / `default_vbox()` and reparent
Tauri's existing webview into an overlay. Low risk — the same widgets, one
extra reparent — but unproven, and it is the only place Tauri-specific
behaviour could still bite.
- 🔴 **ABR — finding 3's premise is in doubt.** Finding 3 says mpv would regress
streaming quality because "the webview path already has real ABR via hls.js".
Three pieces of evidence in this repo suggest that is **not true of the URLs we
actually build**:
1. `get_video_stream_url` (`repository/online.rs`) requests a *single*
rendition — one `VideoBitrate`, one `MaxStreamingBitrate`, one `MaxHeight`.
Jellyfin transcodes to what it is asked for; it does not build a ladder.
2. The frontend contains **no level-handling code at all** — no `hls.levels`,
no `LEVEL_SWITCH`, no `currentLevel`. The `abrEwma*` options in
`VideoPlayer.svelte` are default tuning with nothing to act on. hls.js is
serving as an HLS *demuxer* (WebKitGTK cannot play HLS natively), not as an
adaptation engine.
3. That function's own comment describes a quality switch as **rebuilding the
URL** — "every path that re-opens a stream (quality switch, transcoded seek,
audio-track switch)". Manual selection by stream re-open is what you build
when there is no adaptation, and mpv can do the same thing.
**The decisive test has not been run** and needs a live server plus an API key:
count `#EXT-X-STREAM-INF` lines in a real `master.m3u8`. One line means there
is no ABR to lose and this blocker disappears. More than one means finding 3
stands and the work below applies.
If ABR does turn out to be real, it belongs in **Rust**, not in mpv, and there
are three designs in increasing cost: pick the variant at open; re-open at a
new bitrate on sustained throughput drops (this is the quality-switch path the
app already has, so it is nearly free); or run a local proxy serving mpv a
synthesized single-variant playlist while swapping renditions underneath. The
middle option is almost certainly sufficient.
Either way the **direct-play path still does not exist** — every video play
currently goes through the HLS transcode endpoint. Building it is the real
project; the compositing work proven above is the smaller half.
- 🔴 **One unexplained SIGSEGV.** A ~180s
run died in a *decoder* thread (libavcodec -> `av_log` -> libmpv's log handler
-> libc). No Tauri, wry, WebKitGTK, GTK or GL frame appears anywhere in the
stack, so the fault is on the mpv/ffmpeg side of the process rather than in the
compositing seam.
Three hypotheses were tested and **none reproduced it**:
| Hypothesis | Test | Result |
|---|---|---|
| `hwdec=auto-safe`'s Vulkan failures | 300s soak on `auto-safe` | Survived. Also based on a misreading — the failures are 2 per run at start-up, not per-frame. Dead. |
| Fullscreen transitions recreating the GL context under mpv's render context | 240s soak, ~120 automated transitions | Survived, no core dumped. |
| Continuous resize thrashing the GL framebuffer | 240s soak, ~2000 resizes | Survived, no core dumped. |
**The crash is therefore unexplained.** It was observed exactly once, in the
only session a human interacted with, and did not recur in ~13 minutes of
targeted stress across the three most plausible causes. It is recorded here
rather than dismissed precisely because nothing explains it: an intermittent
fault that nobody can reproduce is worse to inherit than a deterministic one,
not better.
The underlying concern stands regardless of which test eventually reproduces
it. A SIGSEGV in an unrelated thread is characteristic of memory corruption,
and this spike never calls `mpv_render_context_free` and never tears down on
`unrealize` — it has no defence against the GL context being recreated beneath
the render context. That is DR-184 on Android restated: a surface outliving its
player. An implementation must bind the two lifetimes together whether or not
this particular crash is ever explained.
**Therefore G5 is recorded green on appearance only**, and this crash is the
single largest piece of unfinished business in the spike. Do not read the green
gates above as "safe to build on" until it is explained or a long soak clears
it.
- Long-run stability, seeking, track switching, HDR, and multi-window were not
exercised at all.
## Out of scope
- Any change to the shipping Linux video path. `experimentalNativeVideo` in
`adapters/index.ts` is a **suppressor, never a promoter**; the spike must not
change that.
- Adaptive bitrate. See "The blocker a green spike does not clear".
- Windows and macOS — different mechanisms, and **Windows is the easier case, not
the endangered one**. See below.
- Android. Already shipped; it is the precedent, not the target.
- Crossfade, the libmpv2 migration, and the audio-parity work.
### Why Windows is unaffected, and cheaper
Nothing here can regress Windows. `use_html5_element` is already a per-platform
`cfg!` in `get_player_status` — Android native, everything else HTML5 — so
divergent video paths are the existing design rather than something this
introduces. Windows keeps `<video>` + hls.js whatever this spike returns.
The mechanism does not port: `default_vbox()`, `GtkOverlay` and `GtkGLArea` are
GTK3/WebKitGTK concepts. But the *question* is already answered more favourably
there. Both mpv plugins list Windows as **fully tested** and Linux as broken,
because WebView2 honours a transparent background — the "native surface beneath a
transparent webview" approach that fails on WebKitGTK is the one that works on
Windows. That asymmetry is why
[windows-native-audio-backend.md](windows-native-audio-backend.md) can call
Windows "the cleanest available win".
Windows' cost is packaging, not compositing: the build cross-compiles with MSVC +
`cargo-xwin`, so libmpv arrives as a bundled prebuilt DLL (the ⚠️ in finding 5's
comparison table). That cost is already committed for *audio*. Once the DLL ships
to replace `WebviewAudioBackend`, Windows video is largely a follow-on.
Sequencing, if native video is ever pursued on both:
1. [libmpv2-migration.md](libmpv2-migration.md) — prerequisite for either.
2. [windows-native-audio-backend.md](windows-native-audio-backend.md) — already
specced; lands the DLL and a real Windows backend.
3. Windows native video — cheap once 2 exists, and does not need this spike.
4. Linux native video — needs this spike, and runs independently of 13.
### Does this add a backend?
No — and the trajectory is convergence, not proliferation.
`create_player_backend` in `lib.rs` already selects between four
`PlayerBackend` impls by `cfg!`: `MpvBackend` (Linux), `ExoPlayerBackend`
(Android), `WebviewAudioBackend` (Windows and anything else), and `NullBackend`
as the graceful-init fallback. The HTML5 video path is not among them — it is a
frontend adapter reporting through `player_report_*`, not a `PlayerBackend`.
This spike adds none of these. `MpvBackend` already exists and already runs on
Linux; it merely sets `video = no` at construction. Giving it video widens an
existing backend rather than introducing an engine.
Following the sequence above, the count goes **down**: replacing
`WebviewAudioBackend` with mpv on Windows leaves two native engines — mpv
(Linux + Windows) and ExoPlayer (Android) — with native video riding on both.
Two is the floor, for a reason worth stating so nobody re-litigates it: Android
cannot drop ExoPlayer even if libmpv runs there, because the foreground service,
`MediaSessionCompat` and lockscreen control are built on it (finding 7 puts the
cost at that rewrite, not at the bindings). The HTML5 path does not go away
either — it is the transcode/ABR route and the fallback.
The trait surface converges too: `ExoPlayerBackend` already implements the
video-surface lifecycle for Android native compositing, so teaching `MpvBackend`
video follows a path already walked rather than opening a second one.
- Adopting `tauri-plugin-libmpv` or `tauri-plugin-mpv` as dependencies. Both
report Linux window embedding as not working and are small projects
(20 and ~70 commits); read them, do not depend on them.
## Acceptance criteria
The deliverable is a decision, not a feature.
- [ ] Each of G1G6 recorded green/red **with the observed mechanism**, not just
the verdict.
- [ ] X11 and Wayland results reported separately, each naming the compositor
and WebKitGTK version tested.
- [ ] The direct-play/transcode ABR hypothesis recorded as open, with whatever
the spike learned about it.
- [ ] `docs/specs/README.md` updated — this spec listed, and its row moved or
deleted per the result.
- [ ] The Linux claim in the `createAdapter` doc comment
([adapters/index.ts:12-13](../../src/lib/player/adapters/index.ts#L12-L13))
corrected either way: if red, cite this spike instead of asserting it; if
green, it is wrong and must be rewritten.
- [ ] On **red**: finding 2 of
[playback-backend-unification.md](playback-backend-unification.md) gains a
dated note naming the render-API method as also tested, and this file is
deleted. The verdict lives in the design-authority spec, not in a second
file that contradicts nothing.
- [ ] On **green**: an implementation spec exists, allocating ids from
**UR-077 / IR-033 / DR-216** (re-check `requirements.md` — the README's
"next free DR-215" is stale, DR-215 landed), and it must answer ABR before
being accepted.
- [ ] No spike code on `master`. If any lands, the standard gates apply:
`bun run check`, `bun run test`, `bun run check:boundary`, `cargo fmt`,
`cargo clippy`, `bun run test:rust`.
## Testing
No automated tests. A compositing result is a visual, per-session-backend
observation and cannot be asserted in `cargo test` or vitest — pretending
otherwise would produce a test that passes on a headless runner and tells us
nothing.
Capture a screenshot per gate. G4 specifically: an opaque HTML element over the
video area, photographed showing the video *behind* it.
If it goes green, the implementation spec inherits the testable surface the
Android work already established — `nativeVideoLayers.test.ts` asserts the
`app.css` selector list and the `data-native-video` contract, and both are
platform-agnostic.
## TRACES
None. No requirement-implementing code is produced. The implementation spec that
follows a green result allocates from DR-216 and tags there.
## Notes for the implementer
- **Read [playback-backend-unification.md](playback-backend-unification.md)
first, in full.** This spike disputes exactly one of its six findings, on one
platform, by one method it did not try. Everything else in it is still binding
— particularly finding 3.
- Test **Wayland first**. It is the gate most likely to be red and the one that
makes the rest moot.
- The frontend plumbing already exists from the Android work: `createAdapter`,
`NativePlayerAdapter`, `nativeVideo.ts`, `videoSurface.ts`, and the
`[data-native-video="active"]` rule. A green spike is far cheaper to implement
than it would have been a year ago — which is itself part of why the question
is worth re-asking.
- The Android record in
[05-platform-backends.md](../architecture/05-platform-backends.md#native-video-compositing-android)
lists six shipped defects from getting this right on one platform. Expect the
Linux equivalents (the surface outliving its player, the shell painting over
it, unpainted letterbox bars) rather than rediscovering them.
- A parallel Claude session may be active in this repo — `git diff` before
"repairing" unexpected changes.
@@ -90,6 +90,17 @@ The most recent comment on tauri#6343 (2026-05-23) confirms it is still unsolved
**The principle to carry forward: audio can unify on a native engine; video **The principle to carry forward: audio can unify on a native engine; video
cannot, because video needs a surface and the webview owns the surface.** cannot, because video needs a surface and the webview owns the surface.**
> **Re-opened on Linux (2026-08-21).** This finding's general form has since been
> falsified on Android — native video now composites behind a transparent Tauri
> WebView and ships on by default (see
> [05-platform-backends.md](../architecture/05-platform-backends.md#native-video-compositing-android)).
> The evidence above is also entirely about *foreign-window embedding*; mpv's
> render API, drawing into a GL context we own inside Tauri's own GTK tree, was
> never tested. [linux-native-video-spike.md](linux-native-video-spike.md) tests
> that one claim on Linux. **Findings 3-6 below are untouched by it** - in
> particular finding 3, which is an independent disqualifier a green spike would
> not clear.
### 3. mpv would regress streaming quality ### 3. mpv would regress streaming quality
mpv has **no adaptive bitrate**. It delegates HLS to FFmpeg's demuxer, which mpv has **no adaptive bitrate**. It delegates HLS to FFmpeg's demuxer, which
@@ -100,6 +111,17 @@ The webview path already has real ABR via hls.js. Moving video to mpv would be a
**downgrade** on every platform — no graceful degradation on weak networks, and **downgrade** on every platform — no graceful degradation on weak networks, and
quality changes requiring teardown and reload. quality changes requiring teardown and reload.
> **Premise in doubt (2026-08-21).** "The webview path already has real ABR"
> was not verified against the URLs this app actually builds.
> `get_video_stream_url` requests a *single* rendition (one `VideoBitrate`, one
> `MaxHeight`), the frontend has **no** level-handling code (`hls.levels`,
> `LEVEL_SWITCH`, `currentLevel` appear nowhere), and this repo implements a
> quality switch by *re-opening the stream* — all of which point to a
> single-variant playlist, i.e. no ABR to lose. The decisive test is counting
> `#EXT-X-STREAM-INF` lines in a real `master.m3u8`; it needs a live server and
> has not been run. See
> [linux-native-video-spike.md](linux-native-video-spike.md).
### 4. Crossfade is architecturally blocked on mpv ### 4. Crossfade is architecturally blocked on mpv
mpv's audio chain is single-stream. FFmpeg's `acrossfade` is an `N→A` filter mpv's audio chain is single-stream. FFmpeg's `acrossfade` is an `N→A` filter