dtourolle/jellytau
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 38 Wiki Activity

feat(player): native video on Linux, and one contract for every player (v0.11.0)

Browse Source 
mpv now decodes video on Linux, drawn into a framebuffer we own and blitted
into the default vbox's draw handler. Tauri's widget tree is untouched, so an
upgrade that assumes its own layout cannot invalidate this. Direct play means
the original file, hardware decoding, and no server transcode at all — where
previously every desktop video was re-encoded to h264 for the browser engine,
whatever the file actually was. Off by default: JELLYTAU_NATIVE_VIDEO=1.

That settles finding 2 of playback-backend-unification.md — "native video
cannot be composited with a Tauri webview" — by demonstration rather than
argument, on X11 and Wayland both.

Turning it on exposed nine defects, none of them mpv's. Each was the same
mistake in a different place: a capability written down as a compile-time fact
about the platform, or a state asserted instead of confirmed.

  DR-238/246  a seek routed by the stream's container rather than by what the
              engine could do with it - correct only while one player handled
              those streams, silent the moment another did
  DR-239      a property handled but never observed, so the play/pause button
              waited for an event that could not arrive
  DR-240      fullscreen expanding the document while the window stayed put
  DR-241      a seek issued before the engine had a file, failed, and discarded
              - which is why resume began at zero
  DR-247      a Linux-only gate outliving the caller that made it Linux-only,
              breaking the Android build outright
  DR-250      a stop aimed at whichever renderer bookkeeping believed was in
              charge, missing the one actually making sound
  DR-251      a duration of zero believed, leaving the seek bar no scale
  DR-252      a junk float converted to a Duration, panicking the backend the
              instant a length-less stream appeared

So the MediaPlayer contract (DR-242 … DR-247): `open` carries a start position,
so no caller sequences load-then-seek and none can race an engine's load;
`seek` states a destination and leaves in-place-versus-re-open to the engine;
`snapshot` is one coherent read; and `Phase::Opening` names the window where
intent used to be lost. One conformance suite runs against every engine —
FakePlayer and mpv under cargo test, ExoPlayer instrumented on a device — so an
engine is either correct or visibly failing.

Two of the nine were introduced during this work and caught on hardware, not by
any suite: an over-broad capability that grouped ExoPlayer with mpv, and the
Duration panic. The suites test engines that behave. That is recorded in
docs/native-player-verification.md, which asks for the exact action sequences
that found them.

Verified: all automated gates, conformance (mpv 9/9, legacy 8/9 by design,
ExoPlayer 7/7 on device), and manual desktop and Android passes on real
hardware.

Known open and deliberately shipped: resume reads local progress and never the
server's; the background-audio handoff still declares a state swap it does not
confirm (the symptom is now impossible, the race is not); and `bun run
android:dev` builds an APK carrying the release application id, whose failure
message advises an uninstall that would destroy app data. Fix that last one
before anyone else builds for Android.

Squashed from worktree-linux-native-video, which keeps the per-defect history.
This commit is contained in:
dtourolle
2026-08-23 10:51:45 +02:00
parent 5fede123e7
commit 11d9d760d8
87 changed files with 15968 additions and 7508 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/specs/README.md
+4 -4
View File
@@ -28,7 +28,7 @@ know how something *works*, read
**Next free requirement ids** (always re-check
[requirements.md](../requirements.md) before allocating): **UR-079**,
**IR-033**, **DR-225**. Three specs below suggested ids that have since been
**IR-033**, **DR-232**. Three specs below suggested ids that have since been
taken by other work; each carries a ⚠️ note at the top.
## Partially implemented
@@ -37,18 +37,18 @@ taken by other work; each carries a ⚠️ note at the top.
|---|---|---|
| [frontend-domain-model.md](frontend-domain-model.md) | Catalog surface: `MediaKind`, `from_jellyfin` isolated, ticks → ms | `primaryImageTag` → `imageId` (~30 sites); player/session/reporting tick math; `stream.type` |
| [libmpv2-migration.md](libmpv2-migration.md) | `LICENSE` | The `libmpv` → `libmpv2` crate swap |
| [read-through-media-cache.md](read-through-media-cache.md) | DR-126…128, DR-133…138 — cache entries *are* download rows; local playback of downloads | DR-121/122/124/125 — the player quality selector and the read-through capture |
| [read-through-media-cache.md](read-through-media-cache.md) | DR-126…128, DR-133…138 — cache entries *are* download rows; local playback of downloads | DR-122/124/125 — the read-through capture. DR-121 shipped as backend-owned stream selection and left this spec |
| [scoped-search-boundary-implementation.md](scoped-search-boundary-implementation.md) | Stage 1: `SearchScope` owned by Rust (DR-063…067) | Stage 2: result-side grouping (`GROUP_ITEM_TYPES` still in `searchScope.ts`) |
## Not started
| Spec | Blocked on / note |
|---|---|
| [backend-owned-stream-selection.md](backend-owned-stream-selection.md) | Rust owns direct-play-vs-transcode, transport and quality; players consume one `StreamSelection`. Phase 1 (delete the `.m3u8` sniff) stands alone. Unblocks Linux native video. |
| [desktop-native-video.md](desktop-native-video.md) | mpv draws video on every desktop platform, then the webview `<video>` path and hls.js are deleted. Converts a measured 7% direct-play rate toward Android's 85%. Stacked on backend-owned stream selection. |
| [build-provenance.md](build-provenance.md) | `build.rs` is still bare. ⚠️ suggested id DR-093 is taken. |
| [player-facade-enforcement.md](player-facade-enforcement.md) | ~60 `commands.player*` sites still outside the facade; no lint rule. ⚠️ suggested id DR-095 is taken. |
| [windows-native-audio-backend.md](windows-native-audio-backend.md) | Blocked on the libmpv2 swap. ⚠️ suggested id IR-030 is taken. |
| [linux-native-video-spike.md](linux-native-video-spike.md) | **Spike run 2026-08-21: compositing works on Linux, X11 and Wayland.** G1-G6 green bar the Tauri `default_vbox()` half of G1. Needs an implementation spec that answers adaptive bitrate. |
| [linux-native-video-spike.md](linux-native-video-spike.md) | **Spike run 2026-08-21: compositing works on Linux, X11 and Wayland.** G1-G6 green bar the Tauri `default_vbox()` half of G1. The adaptive-bitrate question it was waiting on is **answered**: the server publishes one `EXT-X-STREAM-INF`, so there is no ladder for mpv to lose (DR-229). `StreamSelection` (DR-225) is the contract to consume. |
## Design authority
docs/specs/backend-owned-stream-selection.md
-242
View File
@@ -1,242 +0,0 @@
# Spec: Backend-owned stream selection
**Status:** Proposed
**Requirements:** UR-079 (new) → DR-219 … DR-224 (new); **implements and extends
DR-121**, currently allocated to
[read-through-media-cache.md](read-through-media-cache.md) and not started.
Re-check `requirements.md` before allocating — the ids moved twice while this was
being written (`DR` max was 215, then 218).
**UX spec:** the quality selector in `VideoPlayer.svelte` already exists; this
changes what fills it, not how it looks.
**Supersedes / revises:** takes DR-121 out of
[read-through-media-cache.md](read-through-media-cache.md), which should keep
only its capture/eviction half. Unblocks
[linux-native-video-spike.md](linux-native-video-spike.md).
**Destination on completion:**
[01-rust-backend.md](../architecture/01-rust-backend.md) — extends the
"Streaming quality ladder" section; and
[03-data-flow.md](../architecture/03-data-flow.md) — playback initiation. The
durable half is the layer line and the `StreamSelection` contract; phases and
acceptance criteria are disposable.
## Summary
Make Rust the single owner of *which stream to play* — direct play or transcode,
at what ceiling, over what transport — and hand every player backend a
self-describing selection instead of a bare URL. mpv, ExoPlayer and the HTML5
`<video>`/hls.js path all become consumers of the same decision rather than three
places that re-derive it.
Nothing about how playback *looks* changes. What changes is that the frontend
stops inferring transport from a URL string, and that direct play becomes
possible at all.
## Motivation
Four concrete problems, all the same shape.
**1. The frontend sniffs transport out of the URL.**
[VideoPlayer.svelte:569](../../src/lib/components/player/VideoPlayer.svelte#L569):
```ts
const isHlsStream = currentStreamUrl.includes(".m3u8");
```
and again inline at line 2364. Rust *built* that URL and knows exactly what it
is; the frontend re-derives it by substring match. Change the endpoint, add a DASH
path, serve a progressive file, and this silently picks wrong. This is the
boundary rule in miniature — not item-type taxonomy, but the same error: a
domain fact reconstructed in the presentation layer because the wire shape did
not carry it.
**2. There is no direct-play path.** `get_video_stream_url` always builds an HLS
transcode URL (`TranscodingProtocol=hls`, `VideoCodec=h264` first). Every video
play burns server CPU, even when the file would play untouched. This is the cost
the Linux native-video work exists to remove, and it cannot be removed without a
decision that does not currently exist anywhere in the codebase.
**3. Quality is a process-wide global.** `streaming_quality()` /
`set_streaming_quality()` in `repository/online.rs` read and write a static.
It is not per-session or per-item, so it cannot express "this 4K remux needs a
ceiling, that podcast does not", and two concurrent playbacks would share one
setting.
**4. Rust cannot say what qualities *this* media source supports.** The selector
is populated from a fixed enum rather than from what the source actually offers.
DR-121 already names this; it has not been built.
### The prior question
Finding 3 of [playback-backend-unification.md](playback-backend-unification.md)
holds that hls.js gives us real adaptive bitrate and mpv would lose it. Evidence
in this repo suggests **there is no ABR today**: a single rendition is requested,
no level-handling code exists anywhere in the frontend, and a quality switch is
implemented by re-opening the stream.
**Run this before sizing the adaptation work.** It needs a live server:
```
curl -s "https://<server>/Videos/<itemId>/master.m3u8?api_key=<key>&…" \
| grep -c EXT-X-STREAM-INF
```
`1` → there is no adaptation to preserve, and the adaptation half of this spec
collapses to "pick well at open". `>1` → finding 3 stands and DR-223 applies.
**Everything else in this spec is worth doing either way** — the ownership
problems above are independent of the answer.
## Layer assignment
| Logic / responsibility | Layer | Why it belongs there |
|---|---|---|
| Direct play vs direct stream vs transcode | Rust | Depends on Jellyfin's `PlaybackInfo`, container/codec support and the device profile. Changes when Jellyfin's API or our profile changes → domain, by the litmus test. |
| Transport of the chosen stream (HLS / progressive / local file) | Rust | Rust constructs the URL; it is the only place that *knows* rather than infers. Today the frontend guesses from `.m3u8`. |
| Which qualities this media source can offer | Rust | Derived from the source's own streams and the quality→transcode-parameter mapping that `get_video_download_url` already holds. DR-121. |
| The quality ceiling in force, per playback session | Rust | Domain state that outlives any one view and must survive a backend swap or a mode transfer. Currently a process-wide static. |
| Deciding to re-negotiate mid-playback (if adaptation is needed) | Rust | It performs the HTTP and already derives reachability from real traffic via `ConnectivityMonitor`. Throughput estimation is the same pattern on the same data — a side-channel probe would repeat the mistake that principle exists to prevent. |
| Frame-level delivery *within* the selected stream, including a player's own ABR | **Player** | ExoPlayer has genuine adaptive selection; if Rust hands it a multi-variant playlist it should use it. Rust chooses *what to request*, never how a player paces bytes. See "The line". |
| Rendering the selector, showing the current quality, ordering the list | Frontend | Pure presentation over a backend-supplied list. |
| Poster, letterbox, controls, overlay z-order | Frontend | Unchanged. |
### The line
**Rust decides *what stream*. The player decides *how to deliver it*.**
This matters most for ExoPlayer, which already does real adaptive track selection
over HLS. This spec must not reimplement that or fight it — if a multi-variant
playlist reaches ExoPlayer, ExoPlayer adapts and Rust stays out of the way. The
same restraint applies to any future backend that gains the capability. Rust only
steps in where the player has no such ability (mpv) *and* the server actually
offers a ladder.
Borderline row, with its tie-breaker: "which media source of a multi-source item"
looks like a user choice, and its *presentation* is. The default and the
constraint set are domain → **Rust**, per the borderline-defaults-to-Rust rule.
## Design
### The contract
One self-describing selection replaces the bare URL. Nested fields are
camelCase over the wire (`#[serde(rename_all = "camelCase")]`); the enums are
tagged so the frontend matches a tag instead of parsing a string.
```rust
#[derive(Serialize, Type)]
#[serde(rename_all = "camelCase")]
pub struct StreamSelection {
pub url: String,
pub transport: Transport,
pub playback_kind: PlaybackKind,
/// The negotiated rendition; None when direct-playing the source as-is.
pub rendition: Option<Rendition>,
/// What this media source can offer — fills the selector (DR-121).
pub available: Vec<QualityOption>,
}
#[derive(Serialize, Type)]
#[serde(tag = "type", rename_all = "camelCase")]
pub enum Transport { Hls, Progressive, LocalFile }
#[derive(Serialize, Type)]
#[serde(tag = "type", rename_all = "camelCase")]
pub enum PlaybackKind { DirectPlay, DirectStream, Transcode }
```
`Transport` is the field that deletes the `.m3u8` sniff. The frontend picks
hls.js on `Hls` and the element's own loader otherwise — a tag match, not a
substring search.
### Re-negotiation
Rust emits `stream-selection-changed` (kebab-case, per convention) carrying a new
`StreamSelection` plus the position to resume at. The existing
`playerSetStreamQuality` response already has exactly the right shape — a tagged
`strategy` that tells the caller who reloads, with the backend handling native
itself and handing HTML5 a URL for `reloadSource`
([index.ts:198](../../src/lib/player/index.ts#L198)). **Extend that; do not
invent a second mechanism.** It is the one piece of this that is already right.
Note the existing wart to preserve or fix deliberately, not accidentally:
tauri-specta keeps those response fields snake_case (`new_url`), and the facade
comments say so.
### Phases
1. **DR-219** `StreamSelection` + `Transport`; delete the `.m3u8` sniff. No
behaviour change — pure ownership move, and independently shippable.
2. **DR-220** Per-session quality ceiling replacing the `online.rs` static.
3. **DR-221** `available` populated from the media source (DR-121's substance).
4. **DR-222** Direct-play/direct-stream negotiation via `PlaybackInfo`. This is
the phase that unlocks native video and removes the transcode.
5. **DR-223** Adaptation, **only if the playlist check says a ladder exists**.
Cheapest sufficient design: re-negotiate on sustained throughput drop, reusing
the phase-1 re-negotiation path. A local proxy synthesizing a single-variant
playlist is a last resort, not a starting point.
6. **DR-224** ExoPlayer and mpv consume `StreamSelection` unchanged, proving the
contract is player-agnostic rather than HTML5-shaped.
Phases 1–4 stand on their own merits with no dependency on the ladder question.
## Out of scope
- Rendering, compositing, and the Linux native-video work itself. This spec
unblocks [linux-native-video-spike.md](linux-native-video-spike.md); it does
not contain it.
- Replacing hls.js. It stays as the HLS loader for the webview path.
- Reimplementing or overriding ExoPlayer's own adaptive selection. See "The line".
- The download/capture half of [read-through-media-cache.md](read-through-media-cache.md)
(DR-122, DR-124, DR-125), which keeps its own spec.
- Audio. The same argument applies, but video is where the transcode cost is.
## Acceptance criteria
- [ ] The `.m3u8` substring check is gone from `VideoPlayer.svelte` (both sites)
and transport comes from the tagged enum.
- [ ] `bun run check`, `bun run test`, `bun run format:check`, `bun run lint` pass.
- [ ] `cargo fmt` clean, `cargo clippy -D warnings` clean, `bun run test:rust` passes.
- [ ] `bun run check:boundary` passes — and the reviewer confirms by reading that
no transport/kind decision was reconstructed in `src/`, since the tripwire
only catches item-type array literals.
- [ ] `bindings.ts` regenerated from Rust, not hand-edited.
- [ ] New code carries `// TRACES:` comments; `bun run traces:validate` passes and
coverage stays ≥ the CI ratchet.
- [ ] The `EXT-X-STREAM-INF` count is recorded in this spec before DR-223 is
started or dropped.
- [ ] DR-121 is removed from `read-through-media-cache.md` with a pointer here.
## Testing
- Rust: `PlaybackInfo` fixtures → expected `PlaybackKind`, one per branch
(supported container direct-plays; unsupported codec transcodes; a ceiling
below the source bitrate transcodes even when the codec is fine).
- Rust: `Transport` round-trips through serde with the tag the frontend matches.
- Frontend: adapter selection driven by `transport`, including the case a URL
ending `.m3u8` is served as `Progressive` — that test fails on today's code,
which is the point.
- Extend `tauriIntegration.test.ts` for the new command params (camelCase rule).
- No test asserts a URL substring.
## TRACES
| Piece | Tag |
|---|---|
| `StreamSelection` / `Transport` | `UR-079 \| DR-219` |
| Per-session ceiling | `UR-074 \| DR-220` |
| `available` from media source | `UR-079 \| DR-221, DR-121` |
| Direct-play negotiation | `UR-079 \| DR-222` |
| Adaptation, if built | `UR-079 \| DR-223` |
| ExoPlayer/mpv consumers | `UR-003, UR-004 \| DR-224` |
## Notes for the implementer
- **Phase 1 is worth doing on its own**, even if everything after it is dropped.
It removes a real leak and costs almost nothing.
- Do not frame any phase as "no Rust changes required" — that framing is what
produced the leak `scoped-search-boundary.md` records.
- `ConnectivityMonitor` is the precedent for DR-223: derive network facts from
real traffic, never from a side-channel poller.
- A parallel Claude session may be active in this repo — `git diff` before
"repairing" unexpected changes. Requirement ids in particular moved twice
during the writing of this spec.
docs/specs/desktop-native-video.md
+423
View File
@@ -0,0 +1,423 @@
# Spec: Desktop native video — mpv renders the picture, everywhere
**Status:** Proposed
**Requirements:** UR-080 (new) → DR-231 … DR-237 (new); IR-033 (new)
**UX spec:** n/a — nothing about the player's appearance changes. What changes is
what is behind the controls.
**Supersedes / revises:** consumes and closes
[linux-native-video-spike.md](linux-native-video-spike.md), whose gates
authorised exactly this spec and nothing more. Settles finding 2 of
[playback-backend-unification.md](playback-backend-unification.md) on the
desktop; finding 3 was already settled by DR-229. Absorbs the video half of what
[windows-native-audio-backend.md](windows-native-audio-backend.md) leaves open.
**Depends on:** backend-owned stream selection (DR-225 … DR-230), the branch
below this one. mpv is a *consumer* of `StreamSelection`, never a second place to
decide what to play.
**Destination on completion:**
[05-platform-backends.md](../architecture/05-platform-backends.md) — a "Native
Video Compositing (Desktop)" section beside the existing Android one, which this
mirrors; and [01-rust-backend.md](../architecture/01-rust-backend.md) — the
device profile becomes renderer-dependent, beside the stream-selection section.
**The spike is deleted in the same commit**, its three traps and its
hardware-decode table folded in; they are the durable half.
## Summary
mpv decodes and draws video on **every desktop platform**, composited beneath the
transparent webview, exactly as Android already does with ExoPlayer. The HTML5
`<video>` path and hls.js are then **deleted**, not merely bypassed.
The user-visible change is that most video stops being re-encoded by the server
before it can be watched. The change for whoever maintains this is that video
goes from three renderers to two.
## Motivation
### The transcode is a decoder constraint, not a rendering one
Desktop video goes through an h264 HLS transcode because the picture is drawn by
a WebKitGTK `<video>` element, and that element decodes little else. The device
profile therefore claims `h264` alone. That is not a statement about the machine
— the same machine runs mpv, which decodes essentially everything in the library
— it is a statement about which widget is holding the frame.
DR-228 made the cost measurable. Over 40 items negotiated against the development
server:
| Profile | Direct play |
|---|---|
| Desktop / WebKitGTK — `h264` only, 2ch | **7%** |
| Android / ExoPlayer — `h264,hevc,vp8,vp9,av1,mpeg4` + `ac3,eac3`, 6ch | **85%** |
The sampled library is ~80% hevc. **Those rows differ only by which component
decodes.**
Moving the picture to mpv is what lets the desktop row claim what the machine
can actually do, and that — not the compositing — is the product.
> **The 85% is a ceiling, not a shipped result.** It was measured with a profile
> containing `ac3,eac3`. The Android device later used for verification reports
> neither in its `MediaCodecList` — no Dolby licence, normal for a tablet — so
> eac3 content, about a third of the sampled library, correctly transcodes there.
> Realising any of this depends on DR-234, deriving the profile from the renderer
> rather than from the platform, which is why that requirement is load-bearing
> and not tidy-up.
### One desktop video path, not two
This is why the spec covers Windows rather than stopping at Linux.
Today video has **three** renderers: ExoPlayer, the WebKitGTK `<video>` element,
and (on Android, via the opt-out) that same element again. A Linux-only version
of this work would make it four, permanently: mpv on Linux, HTML5 on Windows,
ExoPlayer on Android, plus hls.js underneath the HTML5 one. Every seek strategy,
every track switch, every quality change, every lifecycle bug would then have one
more place to be got right — and the HTML5 path would survive indefinitely
because *something* would still need it.
Finishing the job removes that: **mpv on desktop, ExoPlayer on Android**, and
`hls.js`, `html5Adapter.ts`, `videoLoaderFor` and the webview video element all
go. The maintenance win is the reason Windows is in this spec and not in a
follow-up that never gets written.
### Three blockers are gone
1. **Compositing works, including Wayland.** The spike ran all six gates; the
2024 "not possible on Wayland at all" claim is out of date when the render API
is used instead of foreign-window embedding.
2. **There is no ABR to lose.** DR-229: the server's master playlist carries one
`EXT-X-STREAM-INF`. hls.js was demuxing, not adapting.
3. **A direct-play path exists.** It did not when the spike was written. DR-228
built it; DR-230 proved the contract is player-agnostic.
And on Windows specifically, `tauri-plugin-libmpv` lists Windows as its **fully
tested** platform — the inverse of the Linux situation the spike had to
disprove. The embedding difficulty was always WebKitGTK-specific.
## Layer assignment
| Logic / responsibility | Layer | Why it belongs there |
|---|---|---|
| **Which codecs this device can decode** | **Rust** | Domain: it is the input to Jellyfin's `PlaybackInfo` negotiation. It stops being a property of the *platform* and becomes a property of *the renderer in use* — see "The structural change". |
| Which backend renders video | **Rust** | Rust already owns this (`use_html5_element` / `VideoBackend`). It stops being a `cfg!` constant and becomes a runtime fact. |
| What stream to play (direct / remux / transcode, transport, ceiling) | **Rust — already decided** | DR-225. mpv consumes `StreamSelection`. Re-deriving any of it in a new backend would be the defect DR-225 exists to remove, restated. |
| Creating the GL surface, reparenting the webview, owning the render context | **Rust (platform layer)** | Native window and GL-context lifetime. Not presentation, and not expressible above the IPC boundary at all. |
| Render-context ↔ GL-context lifetime binding | **Rust** | A correctness invariant over native resources. DR-232. |
| Frame pacing (update callback, `report_swap`) | **Rust** | Timing against the compositor; mpv's own contract. |
| Hardware-decode selection | **Rust** | A capability question about the machine, answered from what mpv reports it actually selected. |
| Z-order of controls over video, overlay chrome, letterbox colour | **Frontend / mpv** | Presentation. Controls already draw over a transparent webview on Android; mpv paints its own letterbox bars (better than the Android equivalent, which shipped DR-194 as a defect). |
| Whether the surface is visible right now | **Frontend** | `nativeVideoActive` already exists and toggles `data-native-video`. Unchanged. |
### The structural change
Everything above is routine except one row, and it carries the whole benefit.
`video_codecs` in `build_device_profile` is a **compile-time constant per
platform**:
```rust
#[cfg(all(not(target_os = "android"), target_os = "linux"))]
let (video_codecs, audio_codecs) = ("h264".to_string(), "aac,mp3,opus,…");
```
That is correct only while a build has exactly one video renderer. It must be
derived from **which renderer will decode this stream**, which is runtime state.
It looks like configuration and is not: it is the input that decides whether the
server re-encodes, it changes when Jellyfin's API or our renderer changes, and
getting it wrong fails *silently* — a claimed codec the renderer cannot decode is
a black picture or silence, which is DR-148 and DR-228's audio override already.
**Write this against "the active video renderer", never `cfg!(target_os)`.** It
is the single piece that must not be Linux-shaped, because phase 2 reuses it
unchanged.
## Design
### Backend and compositing (DR-231, IR-033)
An `MpvVideoBackend` beside the existing `MpvBackend` (audio). The mpv side —
render context, FBO, update callback, hwdec — is **shared**; only the surface
differs per platform:
| Platform | Surface | Status |
|---|---|---|
| Linux (X11 + Wayland) | `gdk_cairo_draw_from_gl()` in the default vbox's `draw` handler, over a `GdkGLContext` on its `GdkWindow`. No reparenting — see below | Render path proven by the spike; the *overlay* approach it used is rejected |
| Windows | Native HWND child beneath a transparent WebView2 | Phase 2 |
`vo=libmpv` plus `mpv_render_context_create` with `MPV_RENDER_PARAM_OPENGL_FBO`.
Webview transparency via `with_transparent(true)` — no window-level transparency;
the spike showed it is neither used nor needed.
**G1's untested half failed, and the design changed because of it.**
Reparenting Tauri's webview into a `GtkOverlay` attaches cleanly and then aborts
the process on the first click. `tauri-runtime-wry` connects a
button-press handler to the webview that walks a hard-coded path:
```rust
webview.parent() // "This one should be GtkBox"
.parent() // ...and this one the GtkWindow
.downcast::<gtk::Window>().unwrap()
```
An overlay makes that chain `webview → GtkOverlay → GtkBox`, the downcast fails,
and the panic is non-unwinding so it kills the app. Nothing in configuration
avoids it: on Linux `attach_resize_handler` is called **unconditionally** (the
Windows equivalent is guarded by `is_decorated()`), and the decoration check that
would make the handler inert runs *after* the unwrap.
**So the webview is not moved at all.** mpv draws into the *default vbox's own
`draw` handler* instead, via `gdk_cairo_draw_from_gl()` over a `GdkGLContext`
created on that widget's `GdkWindow`. GTK3 draws a container before its children,
so the webview composites on top for free — the same z-order the overlay was for,
without touching the widget tree Tauri walks.
That is strictly better than the overlay it replaces: no reparent, no extra
widget, and the arrangement cannot be broken by a Tauri upgrade that assumes its
own layout. It is also why "the surface attached successfully" is not the gate —
a click is.
Three traps from the spike, each of which cost a debugging cycle and each of
which looks like a platform limitation and is not:
1. **`LC_NUMERIC` must be reset *after* `gtk::init()`.** mpv refuses to start
under a non-C numeric locale. `mpv_backend.rs` already handles this but has no
GTK init in front of it; here `gtk::init()` applies the user's locale
afterwards and `mpv_create` returns null.
2. **libepoxy exports GL entry points as *data* symbols.** There is no `glFoo`
function — there is `epoxy_glFoo`, a variable holding a lazily-resolving
pointer. `get_proc_address` must return the pointer **stored at** that symbol;
returning the symbol's own address makes mpv jump into non-executable data and
take SIGSEGV on the first GL call. The `epoxy` crate does this correctly but is
unusable — its `gl_generator` dependency pulls a yanked `xml-rs`.
3. **Frame pacing is not optional and its symptom misleads.** See DR-233.
### Render-context lifetime (DR-232) — the crash defence
The spike's one unexplained SIGSEGV landed in a *decoder* thread with no Tauri,
GTK or GL frame in the stack, and three plausible causes failed to reproduce it
across ~13 minutes of targeted stress.
What is **not** unexplained is that the spike had no defence: it never calls
`mpv_render_context_free` and never tears down on `unrealize`, so nothing stopped
the GL context being recreated beneath the render context. That is DR-184 on
Android restated — a surface outliving its player.
Built as a requirement in its own right, not as a fix for a crash we cannot yet
reproduce:
- Render context created on `realize`, freed on `unrealize`, same thread, before
the GL context goes away.
- The update callback is unregistered **before** the context is freed, so a
callback cannot land on a freed context.
- Playback teardown and surface teardown are ordered, not racing.
If the crash recurs after this, it is a different bug and the likeliest cause is
out of the search space. If it does not, we needed this anyway.
### Frame pacing (DR-233)
Register `mpv_render_context_set_update_callback`; redraw only when it reports a
frame ready; call `mpv_render_context_report_swap` after each render.
Recorded because the failure mode is a trap: driving `queue_render()` off the
frame clock every tick without reporting the swap leaves mpv nothing to time
against. It looks fine in a window and **judders at fullscreen**, which reads as
a compositing or GPU limit and is neither.
### Renderer-dependent device profile (DR-234)
`build_device_profile` takes the active video renderer and derives the codec
lists from it:
| Renderer | Video codecs | Audio (video direct play) | Channels |
|---|---|---|---|
| mpv (desktop native) | `h264,hevc,vp8,vp9,av1,mpeg4` | platform list incl. `ac3,eac3` where the sink can voice it | from the audio route |
| WebKitGTK `<video>` | `h264` | webview-decodable set only | 2 |
| ExoPlayer (Android) | unchanged | unchanged | unchanged |
The existing `video_audio_codecs()` narrowing exists because *the webview decodes
a narrower audio set than the platform*. With mpv decoding, that no longer
applies to the video path — but the multichannel bound still does, since a 5.1
track direct-played into a 2-channel sink is silence or inaudible dialogue. Both
constraints stay, sourced from the renderer rather than assumed.
**This is what converts the 7% figure upward** (toward, not necessarily to, the 85% ceiling — see the caveat above), and it is also the change most able to break
playback silently — so it lands after compositing is proven, covered by the
DR-228 override tests.
### Deleting the webview video path (DR-235)
`get_player_status` stops reporting `use_html5_element: true` on desktop;
`supports_native_video` becomes true there.
Deletion is staged, because a path cannot be removed while a shipped platform
still needs it:
| Phase | Linux | Windows | HTML5 video path |
|---|---|---|---|
| 1 | mpv | HTML5 | alive — Windows needs it |
| 2 | mpv | mpv | alive but unreached |
| 3 | mpv | mpv | **deleted**, with hls.js |
Phase 3 is a real phase with its own acceptance criterion, not a "later". The
whole maintenance argument for including Windows collapses if the fork survives.
Android keeps ExoPlayer and keeps the webview as its documented opt-out; the
`<audio>` element and the background-audio handoff are untouched throughout.
**What happens when mpv fails to initialise.** With no HTML5 path there is no
silent fallback, and inventing one resurrects what we deleted. The
graceful-backend-init principle applies as written: fall back to the no-op
backend, emit `backend-init-failed`, and surface a real error rather than a black
rectangle. An honest failure beats a hidden downgrade to the transcode we are
trying to stop paying for.
### Hardware decode (DR-236)
The spike established the load-bearing fact: **hardware decode works through the
render API** (`hwdec-current` reported `nvdec-copy` on the discrete GPU), so the
direct-play prize is not traded for software decoding.
Policy is decided from what mpv reports it *selected*, never from what it was
asked for:
- Prefer zero-copy VA-API on the integrated GPU where the driver is present.
- `auto` reached for the discrete GPU in **copy-back** mode on a hybrid
Intel+NVIDIA laptop — the least efficient hardware path — so `auto` is a
fallback, not the default.
- `vaapi` silently fell back to software on the spike box because `vainfo` was
absent. A missing driver must be detected and logged, not mistaken for a
compositing limit.
- Log `hwdec-current` at start-up; knowing what was actually chosen is the whole
diagnostic value.
### Windows: what phase 2 actually costs (DR-237)
Not hidden, because it is the part most likely to be underestimated:
- **The surface is different code.** WebView2 in an HWND, not GTK. A transparent
WebView2 over a native child window is a solved arrangement, but DR-231's
Linux surface does not transfer. Everything else does.
- **libmpv is currently a Linux-only dependency**, and Windows is
**cross-compiled from Linux** via `x86_64-pc-windows-msvc` + `cargo-xwin`. Phase
2 must source a Windows libmpv (DLL + import library) into that cross-build and
ship the DLL in the NSIS bundle.
- **LGPL obligations follow the DLL.** DR-216 already records them for Linux:
keep the linkage dynamic, ship libmpv's licence text with any bundle carrying
it. The Windows bundle inherits both.
- **`bun run test:rust` and CI must still build.** Per the CI rule, any tool this
needs goes into the builder image and is pushed — never installed at job time.
Windows also gains a native *audio* decoder as a side effect, which is what
[windows-native-audio-backend.md](windows-native-audio-backend.md) wants and
cannot currently have. If that spec lands first, phase 2 inherits its build work
and shrinks to the surface.
## Out of scope
- **Android.** Unchanged in every respect.
- **macOS.** Not a shipped target. If it becomes one it joins phase 2's shape.
- **Audio backends.** mpv already plays audio on Linux; this adds a video
renderer beside it. Windows audio is its own spec.
- **HDR, tone mapping, multi-window.** Not exercised by the spike at all.
- **Re-deciding what stream to play.** DR-225 owns that. If this spec finds
itself choosing a URL, something has gone wrong.
## Acceptance criteria
**Phase 1 — Linux**
- [ ] Tauri's own webview reparents into the overlay (the untested half of G1),
on X11 **and** Wayland.
- [ ] Video plays, seeks and switches audio track in mpv, with the Svelte
controls composited over it and alpha blending intact.
- [ ] The render context is freed on `unrealize` and the update callback
unregistered before the free; a test demonstrates the ordering.
- [ ] A direct-play negotiation returns `DirectPlay` for an hevc source that
today returns `Transcode`, and it plays.
- [ ] Direct-play rate over the same 40-item sample rises from 7% toward the
Android figure. **Record the number.**
- [ ] mpv init failure emits `backend-init-failed` and surfaces an error rather
than falling back to a transcode.
- [ ] `hwdec-current` is logged and is not copy-back where zero-copy is available.
- [ ] A soak covering seek, track switch and fullscreen runs clean for an agreed
duration. **The spike's SIGSEGV is why this is a criterion.**
**Phase 2 — Windows**
- [ ] libmpv links in the `cargo-xwin` cross-build; the DLL and its licence ship
in the NSIS bundle; any new tool lives in the builder image, not in a CI step.
- [ ] Video plays composited under a transparent WebView2.
- [ ] The device profile, lifetime and hwdec code are **reused, not
reimplemented** — a reviewer confirms no `cfg!(target_os = "linux")` guards
them.
**Phase 3 — deletion**
- [ ] `use_html5_element` is false on every desktop platform.
- [ ] `hls.js` is gone from `package.json`; `html5Adapter.ts`, `videoLoaderFor`
and the `<video>` element are deleted; Android's opt-out and the
background-audio `<audio>` path still work.
**Throughout**
- [ ] `bun run check`, `bun run test`, `bun run format:check`, `bun run lint` pass.
- [ ] `cargo fmt` clean, `cargo clippy -D warnings` clean, `bun run test:rust` passes.
- [ ] `bun run check:boundary` passes, and a reviewer confirms no stream decision
was reconstructed in the new backend.
- [ ] `bindings.ts` regenerated from Rust.
- [ ] `bun run traces:validate` passes; coverage stays ≥ the CI ratchet.
- [ ] The spike and this spec are folded into
[05-platform-backends.md](../architecture/05-platform-backends.md) and both
deleted in the same commit.
## Testing
- **Rust, pure:** the device profile per renderer — mpv claims hevc, the webview
does not, the multichannel bound survives both. The DR-234 table as a
table-driven test.
- **Rust, pure:** `PlaybackInfo` fixtures that transcode under the webview
profile and direct-play under the mpv profile — the direct-play conversion as a unit
test, not only as a measurement.
- **Rust:** teardown ordering — callback unregistered before context freed, freed
before GL context destroyed. Structure it so the ordering is assertable without
a live GL context.
- **Frontend:** no desktop path selects an HTML5 video adapter. After phase 3,
the adapter does not exist and the test goes with it.
- **Manual / soak:** the criterion above. The spike's automated fullscreen and
resize soaks are reusable and already written.
## TRACES
| Piece | Tag |
|---|---|
| mpv video backend + compositing | `UR-080 \| DR-231, IR-033` |
| Render-context lifetime binding | `UR-080 \| DR-232` |
| Frame pacing | `UR-080 \| DR-233` |
| Renderer-dependent device profile | `UR-080, UR-070 \| DR-234` |
| Webview video path removed | `UR-080 \| DR-235` |
| Hardware-decode policy | `UR-080 \| DR-236` |
| Windows surface + cross-build | `UR-080 \| DR-237` |
## Notes for the implementer
- **Read the spike before writing a line.** Its three traps and its
hardware-decode table are the most valuable things in this directory, and each
cost a debugging cycle to find.
- **mpv consumes `StreamSelection`; it does not decide.** The transport is on the
queue item (DR-230). If you are parsing a URL, stop.
- **Guard nothing on `cfg!(target_os = "linux")` that phase 2 will need.** That is
the one avoidable mistake here.
- The Android backend is the reference for the *shape* of this — transparent
webview over a native surface at index 0. Read `05-platform-backends.md`'s
Android section for what shipped and what its defects were (DR-184 surface
lifetime, DR-194 letterbox).
- Do not call sync/blocking APIs from mpv event callbacks that can re-enter the
player or hold a lock. The existing deadlock gotchas apply.
- A parallel Claude session may be active in this repo — `git diff` before
"repairing" unexpected changes.
- This branch is stacked on backend-owned stream selection. Rebase when that
merges rather than merging master into it.
docs/specs/linux-native-video-spike.md
+52 -18
View File
@@ -2,7 +2,10 @@
**Status:** **Run 2026-08-21 — compositing works; G5 carries an open crash.**
The compositing claim it set out to test is falsified on Linux. See "Result".
This file stays open until the implementation spec exists; ABR is unresolved.
This file stays open until the implementation spec exists. **ABR is resolved** —
the playlist carries one `EXT-X-STREAM-INF`, so finding 3 is false and there is
no adaptation for mpv to lose. The remaining blocker is the unexplained SIGSEGV
under G5, which is a lifetime problem, not a compositing one.
**Requirements:** none allocated. This spike produces a decision record, not
product code — same shape as
[playback-backend-unification.md](playback-backend-unification.md), which is
@@ -258,10 +261,10 @@ anything.
Tauri's existing webview into an overlay. Low risk — the same widgets, one
extra reparent — but unproven, and it is the only place Tauri-specific
behaviour could still bite.
- 🔴 **ABR — finding 3's premise is in doubt.** Finding 3 says mpv would regress
streaming quality because "the webview path already has real ABR via hls.js".
Three pieces of evidence in this repo suggest that is **not true of the URLs we
actually build**:
- ✅ **ABR — resolved. Finding 3's premise is false.** Finding 3 said mpv would
regress streaming quality because "the webview path already has real ABR via
hls.js". Three pieces of evidence in this repo suggested that is **not true of
the URLs we actually build**:
1. `get_video_stream_url` (`repository/online.rs`) requests a *single*
rendition — one `VideoBitrate`, one `MaxStreamingBitrate`, one `MaxHeight`.
@@ -276,21 +279,52 @@ anything.
audio-track switch)". Manual selection by stream re-open is what you build
when there is no adaptation, and mpv can do the same thing.
**The decisive test has not been run** and needs a live server plus an API key:
count `#EXT-X-STREAM-INF` lines in a real `master.m3u8`. One line means there
is no ABR to lose and this blocker disappears. More than one means finding 3
stands and the work below applies.
**The decisive test has now been run** (2026-08-21, against the development
server, Jellyfin 10.11.5):
If ABR does turn out to be real, it belongs in **Rust**, not in mpv, and there
are three designs in increasing cost: pick the variant at open; re-open at a
new bitrate on sustained throughput drops (this is the quality-switch path the
app already has, so it is nearly free); or run a local proxy serving mpv a
synthesized single-variant playlist while swapping renditions underneath. The
middle option is almost certainly sufficient.
```
curl -s ".../Videos/<itemId>/master.m3u8?…&TranscodingProtocol=hls&…" \
| grep -c EXT-X-STREAM-INF
1
```
Either way the **direct-play path still does not exist** — every video play
currently goes through the HLS transcode endpoint. Building it is the real
project; the compositing work proven above is the smaller half.
**One line.** The playlist carries a single `EXT-X-STREAM-INF` plus an
`EXT-X-IMAGE-STREAM-INF` trickplay entry, which is not a rendition. Jellyfin
builds the master playlist from the rendition the request asked for; it does
not publish a ladder. So **there is no ABR to lose, and this blocker is
closed** — hls.js is serving as an HLS demuxer, exactly as (2) above supposed,
and mpv gives up nothing by replacing it.
Recorded as DR-229 (Won't Do) rather than deleted, because it is a
measurement: a server that *does* publish a ladder would change the answer, and
the re-negotiation path is the hook that work would build on.
**The direct-play path now exists.** It did not when this spike was written —
every video play went through the HLS transcode endpoint. Backend-owned stream
selection (DR-225 … DR-230) built it: Rust negotiates direct play / direct
stream / transcode and hands every backend one `StreamSelection` carrying the
URL, the transport and the chosen rendition. **That is the contract this
implementation consumes** — mpv is a consumer of a decision already made, not a
place to re-derive it.
It also sizes the prize precisely. Measured over the same server, 40 items
through a real negotiation per profile:
| Profile | Direct play |
|---|---|
| Linux / WebKitGTK — `h264` only, 2ch | **7%** |
| Android / ExoPlayer — `h264,hevc,vp8,vp9,av1,mpeg4` + `ac3,eac3`, 6ch | **85%** |
**The 85% is a ceiling, not a shipped result** — it was measured with a
profile containing `ac3,eac3`, which the Android device later used for
verification does not support.
The library sampled is ~80% hevc. Linux sits at 7% **solely because the
WebKitGTK profile can only claim h264** — not because of anything about the
server or the negotiation. mpv decodes hevc, so widening the Linux device
profile once mpv renders the picture is what converts that 7% toward the
Android figure. That conversion is the actual product of this work; the
compositing proven above is the mechanism that permits it.
- 🔴 **One unexplained SIGSEGV.** A ~180s
run died in a *decoder* thread (libavcodec -> `av_log` -> libmpv's log handler
-> libc). No Tauri, wry, WebKitGTK, GTK or GL frame appears anywhere in the
docs/specs/media-player-controller.md
+324
View File
@@ -0,0 +1,324 @@
# Spec: MediaPlayer — one controller API, three interchangeable engines
**Status:** **Partially implemented.** DR-242 … DR-247 have shipped: the
contract, `FakePlayer` and the conformance suite, `MpvPlayer`, the standalone
runner, `LegacyPlayer`, the controller port, the capability-driven seek
strategy, and ExoPlayer conformance on a device. What is left is DR-248 (the
webview as an engine) and DR-249 (deleting `PlayerBackend` and the frontend
playback-state flags).
**Requirements:** UR-081 (new) → DR-242 … DR-249 (new); IR-034. Re-check
`requirements.md` before allocating — ids moved several times while this was
written.
**UX spec:** n/a — no user-visible change is intended. That is the point.
**Supersedes / revises:** absorbs `determine_video_seek_strategy`
(`player/seek.rs`, DR-238) into the engines. Revises the backend half of
[playback-backend-unification.md](playback-backend-unification.md).
**Destination on completion:**
[01-rust-backend.md](../architecture/01-rust-backend.md) — replaces the player
state-machine section; and
[05-platform-backends.md](../architecture/05-platform-backends.md) — the engines
become implementations of a stated contract rather than three separate designs.
## Summary
Replace the `PlayerBackend` trait with a `MediaPlayer` contract that expresses
**intent** ("present this item, starting here") rather than **device operations**
("load", then "seek"). MPV, ExoPlayer and the webview element implement it; a
`FakePlayer` implements it for tests; and one conformance suite runs against
every implementation so a backend is either correct or visibly failing.
No user-visible behaviour changes. What changes is that playback logic stops
being written three times in the command layer.
## Motivation
A day of debugging Linux native video produced four defects (DR-238 … DR-241).
Every one of them traces to the same missing seam, not to mpv:
| Defect | What it looked like | What it was |
|---|---|---|
| DR-241 | "Resume is broken", "I cannot skip" | `loadfile` is async, so a seek issued straight after a load fails and was discarded. The trait has no way to say *open at a position*, so every caller does load-then-seek and each races independently. |
| DR-238 | Transcoded seeks silently did nothing | `use_html5` was doing double duty as "who renders" **and** "how do I seek", decided in the command layer by a truth table. |
| DR-239 | Play/pause control never moved | `PropertyChange { name: "pause" }` was handled but never observed. Nothing in the contract required an engine to report its own state. |
| DR-240 | Fullscreen left the picture at window size | `requestFullscreen()` moves the document; whoever owns the pixels has to be told separately. |
The shape is consistent: **the same intent implemented in several places, each
with its own timing and its own idea of the rules.** Resume worked through the
adapter (which seeks after `File loaded`) and failed through the command (which
seeks immediately). Two callers, one intent, two behaviours.
Supporting evidence for the diagnosis:
- `commands/player/mod.rs` is **3,561 lines** and is where "stop → rebuild URL →
update queue → load → seek" lives. That is playback orchestration in the IPC
layer.
- `player_play_item` needed a `#[cfg(not(target_os = "linux"))]` guard, i.e. a
platform decision in a command handler.
- The frontend carries `didStartNativePlayback`, `didStopBackendEarly`,
`hasPerformedInitialSeek`, `lastAppliedInitialPosition` — playback state in the
UI, which contradicts the one-directional rule in CLAUDE.md.
### Why an abstraction, and not more fixes
Each defect above was individually cheap to patch, and patching them is what
produced a regression: routing transcoded seeks to a reload path turned "seek
does nothing" into "seek jumps to zero", because the reload path's own seek was
broken in the same way. **Symptom fixes in this area compound.**
## The background-audio handoff is an unconfirmed state swap
Diagnosed on a device, 2026-08-23, and the likeliest explanation for "audio
keeps playing after I leave the player" — the report this whole line of work
started from.
`enter_background_audio` and `exit_background_audio` in `PlayerController` are
pure bookkeeping: they flip a boolean and set or clear a base offset. Neither
confirms that the audio stream actually opened, nor that the webview `<video>`
actually came back. `exit_background_audio`'s own doc comment says the element
"becomes the player again once it reloads" — a future event nothing waits for,
while the flag declares the swap complete the moment it is called.
The sequence that exposes it:
1. Background audio is enabled.
2. The app is backgrounded — `enter_background_audio(pos)`, audio stream opens.
3. The app is foregrounded — `exit_background_audio()` sets the flag back, so
the controller believes the video element owns playback again.
4. The player is exited *before the element has reloaded*. The stop is aimed at
an element that does not exist yet; the audio stream is still running.
5. The mini player sees a live audio session and adopts it — which is why the
symptom is a **movie appearing as an audio track**, and why it is
intermittent rather than reliable.
Duration reporting `0.0` on Android widens the window: the reload is slower and
less certain to land at the right position.
**This is the same defect class as DR-238 … DR-241: state asserted rather than
confirmed.** It is what `Phase::Opening` and `MpvPlayer`'s open generation
exist for — a handoff *is* an open in flight, and a `close` during one has to
cancel it rather than race it. The handoff is not modelled as an open at all
today; it is two booleans and an offset.
The fix therefore belongs with this contract rather than beside it: route the
handoff through `open`/`close` so the swap has a phase, and so leaving the
player during one cancels the thing that is actually playing instead of the
thing the controller believes is playing. `close_during_open_never_plays`
already states the required behaviour and passes on all four engines — the gap
is that the handoff never reaches an engine as an open.
## Layer assignment
| Logic / responsibility | Layer | Why it belongs there |
|---|---|---|
| Presenting an item at a position, in one operation | **Engine** (`MediaPlayer`) | Only the engine knows when its pipeline can accept a position. Expressing it as caller-sequenced load-then-seek exports a race the engine is the only one able to close. |
| Whether *this* stream can be seeked in place, or must be re-opened | **Engine** | A property of the engine × transport pair: hls.js seeks a VOD playlist, mpv's HLS demuxer cannot make Jellyfin transcode from a new offset. Today this is a truth table in a command handler that has to guess for engines it does not own. |
| Reporting position, phase, duration, active tracks | **Engine** | The player is the authoritative source of playback state (CLAUDE.md). An engine that does not report is not implementing the contract — DR-239 was exactly this. |
| Choosing *which* stream to open (direct play vs transcode, ceiling, transport) | **Rust, above the engine** | Domain: depends on Jellyfin's `PlaybackInfo`, codec support, quality ceiling. See [backend-owned-stream-selection.md](backend-owned-stream-selection.md). The engine is handed a `StreamSelection`; it never negotiates one. |
| Queue, autoplay, session, playback reporting | **`PlayerController`** | Policy across items. Unchanged — but it talks to one contract instead of branching per platform. |
| Which engine this platform uses | **Rust, at construction** | Already correct today; stays a single `cfg` at the composition root rather than `cfg`s scattered through command handlers. |
| Rendering surfaces, controls, fullscreen chrome | **Frontend / platform** | Presentation. The engine reports *what* is playing; it does not own the window. |
Borderline row and its tie-breaker: "should a transcoded seek re-open the
stream?" reads like domain policy. It is **engine** capability — the *decision*
is "seek to T", and how to achieve it is the engine's business. If it were
policy, every new engine would require editing a shared truth table, which is
precisely the coupling DR-238 came from.
## Design
### The contract
```rust
/// Anything that can present media: MpvPlayer, ExoPlayer, WebviewPlayer, FakePlayer.
pub trait MediaPlayer: Send {
/// Present `req.selection`, beginning at `req.start`.
///
/// One operation, deliberately. `open` is where a start position is
/// *expressible*, so no caller has to sequence load-then-seek and no caller
/// can race the engine's own load. An engine that cannot start at an offset
/// natively must absorb that internally (defer until loaded, or re-open) —
/// it is the only layer that knows when it is able to.
fn open(&mut self, req: OpenRequest) -> Result<(), PlayerError>;
fn play(&mut self) -> Result<(), PlayerError>;
fn pause(&mut self) -> Result<(), PlayerError>;
/// Stop and release the current item. Must be idempotent, and must leave the
/// engine producing no audio — DR-2xx exists because "stopped" and "silent"
/// were not the same thing.
fn close(&mut self) -> Result<(), PlayerError>;
/// Seek to an absolute position on the item's timeline.
///
/// The engine decides in-place vs re-open. Callers never choose.
fn seek(&mut self, to: Duration) -> Result<(), PlayerError>;
fn set_volume(&mut self, volume: Volume) -> Result<(), PlayerError>;
fn set_rate(&mut self, rate: f64) -> Result<(), PlayerError>;
fn select_audio_track(&mut self, index: Option<i32>) -> Result<(), PlayerError>;
fn select_subtitle_track(&mut self, index: Option<i32>) -> Result<(), PlayerError>;
/// One coherent read of everything the UI consumes.
fn snapshot(&self) -> PlaybackSnapshot;
/// Engine capabilities, so callers can adapt without naming engines.
fn capabilities(&self) -> Capabilities;
}
```
```rust
pub struct OpenRequest {
pub media: MediaItem,
pub selection: StreamSelection, // url + transport + playback kind
pub start: Duration, // Duration::ZERO for "from the beginning"
pub audio_track: Option<i32>,
pub subtitle_track: Option<i32>,
pub autoplay: bool,
}
pub struct PlaybackSnapshot {
pub phase: Phase,
pub position: Duration,
pub duration: Option<Duration>,
pub seekable: bool,
pub volume: Volume,
pub rate: f64,
pub audio_track: Option<i32>,
pub subtitle_track: Option<i32>,
}
/// `Opening` is the state today's code cannot express, and the direct cause of
/// DR-241: a seek arriving with nothing loaded had no phase to be rejected or
/// queued against, so it was simply lost.
pub enum Phase { Idle, Opening, Ready, Playing, Paused, Ended, Failed(String) }
```
Engines emit `PlayerEvent` for phase, position, track and error changes. Emitting
is part of the contract, and the conformance suite asserts it — an engine that
stays silent fails, which is what would have caught DR-239 the day it landed.
### What this deletes
- `determine_video_seek_strategy` and `VideoSeekStrategy` — replaced by
`seek()` + `capabilities()`. The command layer stops deciding how engines seek.
- The reload orchestration in `player_seek_video` — moves inside the engines that
need it.
- `#[cfg(target_os = "linux")]` branches in command handlers.
- Frontend playback-state flags, which become reads of `snapshot()`.
### IPC
No new commands. Existing ones keep their names and shapes; they become thin
delegations. `PlayerStatus` gains nothing the frontend does not already receive.
Regenerate `bindings.ts` only if `PlaybackSnapshot` is exposed directly — prefer
mapping it onto the existing `PlayerStatus` so this stays invisible at the wire.
## Testing
This is the half that makes the abstraction worth having, and it is the reason to
do it rather than keep patching.
### 1. A conformance suite, run against every engine
One set of tests, parameterised over implementations. Any `MediaPlayer` must pass
it; a new engine is "done" when it does.
```
conformance::run(&mut engine, fixture) covering:
open(start = ZERO) -> phase Ready|Playing, position ~0
open(start = 10min) -> position within tolerance of 10min, NEVER 0 [DR-241]
seek while Opening -> honoured once Ready, not discarded [DR-241]
seek on a transcoded stream -> position lands, by whatever means [DR-238]
pause / play -> phase changes AND an event is emitted [DR-239]
close -> phase Idle, silent, idempotent
close during Opening -> no playback ever starts [audio-on-exit]
volume / rate / track select -> reflected in snapshot()
```
The `open(start = 10min)` and `seek while Opening` cases are the ones that fail
on today's code. They are written first, and they are the acceptance criterion.
### 2. `FakePlayer`
A deterministic in-memory implementation with a controllable clock. Lets
`PlayerController`, autoplay, queue, sleep-timer and session logic be tested with
no mpv, no device, no network — most of which is currently only reachable through
a real engine.
### 3. Per-engine runs
| Engine | Where | Note |
|---|---|---|
| `FakePlayer` | `cargo test` | Always. |
| `MpvPlayer` | `cargo test`, Linux | libmpv is already in the builder image (the Linux build links it), so **no CI toolchain install** — see CLAUDE.md. Needs a tiny local fixture file; generate it in-test rather than committing media. |
| `ExoPlayer` | instrumented, on device | Not in the standard CI job. Run via `scripts/` on a connected device; record results in the PR. |
| `WebviewPlayer` | vitest | Against a stubbed element, as `html5Adapter` is tested today. |
An engine that cannot run in CI still has the same suite; it is just run by hand.
That is the point of writing it once.
## Migration
Strangler, not a rewrite. Each step ships independently and leaves the app working.
1. **DR-242** Define `MediaPlayer`, `OpenRequest`, `PlaybackSnapshot`, `Phase`,
`Capabilities`. No implementations. Compiles alongside `PlayerBackend`.
2. **DR-243** `FakePlayer` + the conformance suite. The suite fails against
nothing yet — it is the specification.
3. **DR-244** `MpvPlayer` implementing `MediaPlayer`, wrapping today's
`MpvBackend` internals. Make conformance pass, including `open(start)`.
4. **DR-245** `PlayerController` talks to `MediaPlayer`. `PlayerBackend` retained
behind an adapter so the other engines keep working.
5. **DR-246** Move seek strategy and reload orchestration out of
`commands/player/mod.rs` into the engines; delete `seek.rs`'s truth table.
**Shipped with a deviation.** The engine cannot own this outright:
re-negotiating a stream needs the repository, which sits *above* the engine.
So the engine *declares* `seeks_transcoded_in_place` and the caller acts on
it. That removes the defect — nobody guesses on another component's behalf,
and adding an engine no longer means editing a shared table — without
pretending an engine can reach upward. `determine_video_seek_strategy`
survives as a correctly-typed decision over declared abilities rather than
being deleted; the defect was its *input*, not its existence.
6. **DR-247** `ExoPlayerPlayer`; conformance on device.
7. **DR-248** `WebviewPlayer`; retire the adapter shim.
8. **DR-249** Delete `PlayerBackend` and the frontend playback-state flags.
Steps 1–3 are pure addition and risk nothing. Step 5 is where today's defect
classes actually die.
## Out of scope
- Stream selection (which URL, which quality) — that is
[backend-owned-stream-selection.md](backend-owned-stream-selection.md), and
this spec consumes its `StreamSelection` rather than duplicating it.
- Rendering surfaces and compositing.
- Any user-visible behaviour change. If one appears, it is a bug in the migration.
- Replacing hls.js or changing the transcode path.
## Acceptance criteria
- [ ] The conformance suite exists and `open(start = 10min)` fails against the
pre-migration mpv path — proving it reproduces DR-241 — then passes.
- [ ] `FakePlayer` lets at least one controller-level test run with no engine.
- [ ] `determine_video_seek_strategy` is deleted, not merely bypassed.
- [ ] No `cfg(target_os = ...)` remains in `commands/player/`.
- [ ] `bun run check`, `bun run test`, `bun run format:check`, `bun run lint` pass.
- [ ] `cargo fmt`, `cargo clippy -D warnings`, `bun run test:rust` pass.
- [ ] `bun run check:boundary` passes.
- [ ] `// TRACES:` on new code; `bun run traces:validate` passes; coverage stays
at or above the CI ratchet.
- [ ] Manual: resume, skip on a transcoded item, pause/play, and exit-while-playing
verified on Linux **and** Android before `PlayerBackend` is deleted.
## Notes for the implementer
- **Write the conformance suite before the second engine**, or it will encode
whatever the first engine happens to do.
- `close()` must mean *silent*. The bug that motivated this spec had `stop` being
called, reported, and audible afterwards.
- Do not let `Capabilities` grow into engine sniffing. If a caller branches on
the engine's identity, the contract is missing something — add it there.
- A parallel Claude session may be active in this repo — `git diff` before
"repairing" unexpected changes.
docs/specs/read-through-media-cache.md
+22 -26
View File
@@ -4,15 +4,20 @@
(DR-126, DR-127 — a cache entry *is* a `downloads` row with a shorter life, and
eviction only reclaims the temporary tier), local playback of downloaded media
(DR-128), and the one-path/one-row invariants that followed (DR-133 … DR-138).
DR-123 is in progress. Still open: the **player quality selector** and the
read-through capture itself — DR-121, DR-122, DR-124, DR-125. The separate
settings-level bitrate cap (DR-162, shipped —
[01-rust-backend.md](../architecture/01-rust-backend.md#streaming-quality-ladder))
covers a *settings-level*
ceiling (DR-162), which serves part of UR-070 but is not the per-playback
selector specified here.
**Requirements:** UR-070, UR-071 → DR-121, DR-122, DR-123, DR-124, DR-125; IR-032
**UX spec:** player quality selector — needs a `ux-flows.md` section before build
DR-123 is in progress. Still open: the read-through capture itself — DR-122,
DR-124, DR-125.
**DR-121 has shipped and left this spec.** The player quality selector, the
per-playback bitrate ceiling, and the backend-owned stream decision it needed
were built as *backend-owned stream selection* (DR-225 … DR-228) and are
described in
[01-rust-backend.md](../architecture/01-rust-backend.md#stream-selection) and
[03-data-flow.md](../architecture/03-data-flow.md#video-stream-selection-flow).
The settings-level ceiling (DR-162) is the same section. What remains here is the
*capture* half only — this spec no longer specifies anything about choosing a
bitrate.
**Requirements:** UR-070, UR-071 → DR-122, DR-123, DR-124, DR-125; IR-032
**Related:** the locally-indexed search and downloaded-browse work, both
shipped — see
[03-data-flow.md](../architecture/03-data-flow.md) and
@@ -70,24 +75,16 @@ frontend stores the user's *choice*; Rust decides what that choice resolves to.
## Design
### DR-121 — Bitrate selection in the player
### DR-121 — moved out (shipped)
The player exposes the qualities Rust reports for the current item. Changing it
re-negotiates the stream URL at the new quality and resumes at the current
position. This is a deliberate, user-initiated interruption — a brief rebuffer is
expected and acceptable, unlike the involuntary swap the earlier design would
have needed.
Bitrate selection in the player shipped as DR-225 … DR-228; see
[01-rust-backend.md](../architecture/01-rust-backend.md#stream-selection).
Constraints that must not be broken:
- On Linux, video playback must keep using the HLS `master.m3u8` URL. CLAUDE.md
records that returning `stream.mp4` means transcoded playback never starts.
A quality change re-negotiates *within* HLS.
- The quality→transcode-parameter mapping already exists in
`get_video_download_url` ([online.rs:1702-1717](../../src-tauri/src/repository/online.rs#L1702-L1717)).
Playback must call into the same mapping. Two copies of that table will drift.
- Track selection (audio/subtitle) already survives a stream re-negotiation
elsewhere in the player; a quality change must preserve it too.
The one constraint here that the capture work still has to respect: a quality
change re-negotiates **within HLS**. Returning a progressive `stream.mp4` for a
transcode means playback never starts, because the server encodes the whole file
before serving a byte (DR-140). That is why DR-122 below abandons a capture on a
quality change rather than trying to splice one.
### DR-122 — The playback path is ephemeral
@@ -212,7 +209,6 @@ codec taxonomy in `src/`; the selector's remembered choice is a view preference.
| Piece | Tag |
|---|---|
| Quality selector + re-negotiation | `// TRACES: UR-070 \| DR-121` |
| Ephemeral playback / capture abandonment | `// TRACES: UR-070 \| DR-122` |
| Independent whole-file download + local video playback fix | `// TRACES: UR-071 \| DR-123, IR-032` |
| ExoPlayer cache / mpv stream-record / keepability | `// TRACES: UR-071 \| DR-124` |