feat(player): native video on Linux, and one contract for every player (v0.11.0)

mpv now decodes video on Linux, drawn into a framebuffer we own and blitted
into the default vbox's draw handler. Tauri's widget tree is untouched, so an
upgrade that assumes its own layout cannot invalidate this. Direct play means
the original file, hardware decoding, and no server transcode at all — where
previously every desktop video was re-encoded to h264 for the browser engine,
whatever the file actually was. Off by default: JELLYTAU_NATIVE_VIDEO=1.

That settles finding 2 of playback-backend-unification.md — "native video
cannot be composited with a Tauri webview" — by demonstration rather than
argument, on X11 and Wayland both.

Turning it on exposed nine defects, none of them mpv's. Each was the same
mistake in a different place: a capability written down as a compile-time fact
about the platform, or a state asserted instead of confirmed.

  DR-238/246  a seek routed by the stream's container rather than by what the
              engine could do with it - correct only while one player handled
              those streams, silent the moment another did
  DR-239      a property handled but never observed, so the play/pause button
              waited for an event that could not arrive
  DR-240      fullscreen expanding the document while the window stayed put
  DR-241      a seek issued before the engine had a file, failed, and discarded
              - which is why resume began at zero
  DR-247      a Linux-only gate outliving the caller that made it Linux-only,
              breaking the Android build outright
  DR-250      a stop aimed at whichever renderer bookkeeping believed was in
              charge, missing the one actually making sound
  DR-251      a duration of zero believed, leaving the seek bar no scale
  DR-252      a junk float converted to a Duration, panicking the backend the
              instant a length-less stream appeared

So the MediaPlayer contract (DR-242 … DR-247): `open` carries a start position,
so no caller sequences load-then-seek and none can race an engine's load;
`seek` states a destination and leaves in-place-versus-re-open to the engine;
`snapshot` is one coherent read; and `Phase::Opening` names the window where
intent used to be lost. One conformance suite runs against every engine —
FakePlayer and mpv under cargo test, ExoPlayer instrumented on a device — so an
engine is either correct or visibly failing.

Two of the nine were introduced during this work and caught on hardware, not by
any suite: an over-broad capability that grouped ExoPlayer with mpv, and the
Duration panic. The suites test engines that behave. That is recorded in
docs/native-player-verification.md, which asks for the exact action sequences
that found them.

Verified: all automated gates, conformance (mpv 9/9, legacy 8/9 by design,
ExoPlayer 7/7 on device), and manual desktop and Android passes on real
hardware.

Known open and deliberately shipped: resume reads local progress and never the
server's; the background-audio handoff still declares a state swap it does not
confirm (the symptom is now impossible, the race is not); and `bun run
android:dev` builds an APK carrying the release application id, whose failure
message advises an uninstall that would destroy app data. Fix that last one
before anyone else builds for Android.

Squashed from worktree-linux-native-video, which keeps the per-defect history.
This commit is contained in:
2026-08-23 10:51:45 +02:00
parent 5fede123e7
commit 11d9d760d8
87 changed files with 15968 additions and 7508 deletions
+154 -1
View File
@@ -673,8 +673,161 @@ device profile. Sending it there — not just on the transcode URL — is what m
the cap real: a stream the server decides to *direct play* is served at the
source file's own bitrate, and no URL parameter afterwards can reduce it.
#### Two levels of ceiling
**Location**: `src-tauri/src/repository/online.rs` (TRACES: UR-074, UR-079 | DR-226)
There are two, and they are not the same thing:
| | Set by | Lives until | Read via |
|---|---|---|---|
| **Device default** | Settings (`player_set_video_settings`) | Persisted; restored at startup | `streaming_quality()` |
| **Per-playback override** | The in-player picker (`player_set_stream_quality`) | The next item starts playing | `playback_quality_override()` |
`effective_streaming_quality()` resolves the pair — override first, else default —
and **is the only thing stream construction may read**. Every URL builder and the
`PlaybackInfo` negotiation go through it, for the reason the process-wide static
existed in the first place: if the negotiation and the URL builder disagree, the
cap leaks — the negotiation authorises a direct play the builder then never gets
to constrain, or the reverse.
> The override exists because a single global cannot express "this 4K remux needs
> a ceiling, that podcast does not". The picker had documented itself as a "this
> film, this connection" control since it was written, but was implemented by
> writing the *default* — so dropping one awkward film to 2 Mbps silently capped
> every video played afterwards for the rest of the process, with Settings still
> showing the old value. It is cleared on every `player_play_item` /
> `player_play_queue` / `player_play_tracks`, which is what stops it surviving
> into an autoplayed next episode where nobody would reopen the picker.
### Stream selection
**Location**: `src-tauri/src/repository/stream_selection.rs`,
`OnlineRepository::get_stream_selection` (TRACES: UR-070, UR-079 | DR-225, DR-227, DR-228)
**Rust decides *what stream*. The player decides *how to deliver it*.** That line
is the whole design. A backend with genuine adaptive selection (ExoPlayer over a
multi-variant playlist) is left to do it; Rust chooses what to request and never
paces bytes.
`get_stream_selection` returns one self-describing `StreamSelection` in place of
the bare URL `get_video_stream_url` used to hand out:
| Field | Carries |
|---|---|
| `url` | What to open |
| `transport` | `Hls` / `Progressive` / `LocalFile` — how to fetch it |
| `playback_kind` | `DirectPlay` / `DirectStream` / `Transcode` — what the server is doing to the source |
| `rendition` | The negotiated ceiling and codecs; `None` for a direct play, which *is* the source |
| `available` | The quality ladder as it applies to this media source (DR-227) |
| `needs_transcoding` | Derived from `playback_kind`, so the rule is answered once |
Both enums are serde-tagged (`{"type":"hls"}`) so the frontend matches a
discriminant rather than comparing text.
> **Why `transport` exists.** `VideoPlayer.svelte` chose its loader with
> `url.includes(".m3u8")`, in two places. Rust *built* that URL and knows exactly
> what it is; re-deriving it downstream by substring match is a domain fact
> reconstructed in the presentation layer — the same class of error as leaking
> item-type taxonomy, and one that fails silently in **both** directions: a
> progressive file served from a path containing the substring gets an HLS
> loader, and a playlist served from a path without it does not.
>
> The paths that never negotiate get the same shape from Rust rather than letting
> a caller assemble one — `media_local_selection` for a downloaded file,
> `LiveStreamInfo.transport` for a live channel — so there is no second place
> where a transport is decided.
#### The playback-kind decision
`decide_playback_kind` is a free function and pure, so every branch is testable
from `PlaybackInfo` fixtures without a server. Order matters — the two
client-side overrides come first, because each describes a case where the
server's answer is right about the *file* and wrong about what this app will do
with it:
1. **Undecodable audio → `Transcode`.** Jellyfin 10.11.5 honours a
DirectPlayProfile's container and video codec but *ignores its audio codec*,
so it offers direct play for an E-AC-3 track the webview renders in silence.
A silent direct play is worse than a transcode.
2. **A pinned audio track → `Transcode`.** Not a defect in the server's answer, a
different question: the file has one default track and the viewer asked for
another.
3. Otherwise `supports_direct_play``DirectPlay`, else `supports_direct_stream`
`DirectStream`, else `Transcode`.
A direct **stream** is a remux — codecs copied, container repackaged. It is cheap
and is deliberately *not* counted as transcoding; conflating the two would report
a free passthrough as a server-side re-encode.
> **What this is worth, measured.** Against the development server (Jellyfin
> 10.11.5), 400 items sampled for codec mix and 40 put through a real negotiation
> per profile:
>
> | Profile | Direct play |
> |---|---|
> | Linux / WebKitGTK (`h264` only, 2ch) | 3/40 — **7%** |
> | Android / ExoPlayer (`h264,hevc,vp8,vp9,av1,mpeg4` + `ac3,eac3`, 6ch) | 34/40 — **85%** |
>
> The library is ~80% hevc (`hevc+eac3` alone is a third of it), which is why the
> two diverge so hard.
>
> **Read that 85% as a ceiling, not a result.** It was measured with a profile
> containing `ac3,eac3`. The Android device this was later run on reports neither
> in its `MediaCodecList` — no Dolby licence, which is normal for a tablet — so
> eac3 content, about a third of the sampled library, correctly transcodes there.
> What any given device achieves depends on its own codec list, and on the
> profile being derived from the renderer at all (DR-234), which it was not when
> the figure was taken.
>
> **The payoff is still overwhelmingly Android**, because that is where a real
> decoder is already doing the work. Linux stays near 7% until libmpv decodes the
> picture — the h264-only profile is a WebKitGTK constraint, not a JellyTau
> choice, and is what `linux-native-video-spike.md` exists to remove. A reviewer
> should not expect this code to fix Linux on its own.
#### The quality ladder per source
`quality_options_for_source(source_bitrate)` returns every rung, each marked with
`exceeds_source`: true when that rung's ceiling is at or above what the source
itself carries, so selecting it produces the same bytes as `Original`. The
frontend draws the list and drops the redundant rungs; it does not decide which
they are.
- `Original` is never marked — it *is* the source.
- An unreported source bitrate (some containers have none; the sampled library
has `avi` files with no bitrate at all) marks **nothing** redundant, keeping
every rung offered. That is the safe direction: the viewer keeps every choice.
#### No adaptive ladder to preserve
**TRACES: UR-079 | DR-229 (Won't Do)**
Mid-playback re-negotiation on throughput was scoped and dropped on measurement.
A master playlist from this server carries exactly **one** `EXT-X-STREAM-INF`:
Jellyfin builds it from the single rendition the request asked for rather than
publishing a ladder. So there is no adaptation for hls.js to be preserving and
none that mpv would lose — the claim that there was is recorded in
`playback-backend-unification.md` and does not hold. "Adapt mid-stream" collapses
into "pick well at open", which is what the two levels of ceiling and the
per-source ladder already are.
Kept here because it is a measurement, not an opinion: a server that *does*
publish a ladder would change the answer, and the re-negotiation path below is
the hook that work would build on.
#### Re-negotiation
One mechanism, not two. `player_seek_video`, `player_switch_audio_track` and
`player_set_stream_quality` all return a tagged `strategy` saying who reloads —
the backend handles a native backend itself and hands the webview a
`StreamSelection` for `reloadSource`. Note the wire wart: tauri-specta keeps
these response fields snake_case (`seek_offset`), while the `strategy` tag itself
is camelCase.
The frontend names a variant and nothing else; the labels the picker shows are
served over IPC by `player_get_streaming_qualities`.
served over IPC — from `available` on the selection, or
`player_get_streaming_qualities` for the Settings list.
## Background workers
+39
View File
@@ -802,6 +802,45 @@ by exactly the inset.
Unlike `addJavascriptInterface`, the inset push only writes CSS properties, so it
can safely be re-sent on resume.
## Stream Transport
**Location**: `src/lib/player/streamTransport.ts`
**TRACES**: UR-079 | DR-225 | UT-214
`videoLoaderFor(selection, capabilities)` picks the loader for the webview
`<video>` element — `hlsjs`, `nativeHls`, or `direct` — from the backend's tagged
`selection.transport`. `elementSrcFor` is its template companion: the element's
`src` is emptied only when hls.js is driving it.
The split is the point. **The transport is the stream's property and comes from
Rust; whether a given loader exists is the browser's, and is the only thing
decided here.**
> This replaced `currentStreamUrl.includes(".m3u8")`, which appeared twice in
> `VideoPlayer.svelte` — once in the HLS `$effect` and once inline in the
> template's `src`. Rust builds that URL and knows what it is; re-deriving it
> here by substring match was a domain fact reconstructed in the presentation
> layer, and it fails silently in both directions. The two tests that pin it are
> the ones that failed against the old implementation: a `progressive` stream
> whose URL contains `.m3u8` must **not** get an HLS loader, and an `hls` stream
> whose URL contains no `.m3u8` must.
>
> Logic lives in a plain `.ts` module rather than in the component for the usual
> reason — it is testable there. Same pattern as `episodeStrip.ts`.
`VideoPlayer` holds a `currentSelection`, not a URL string; `currentStreamUrl` is
derived from it. A reload replaces the selection **wholesale** (the adapter's
bridge takes a `StreamSelection`, not a URL), so transport and URL can never
drift apart. The background-audio handoff states the transport it is moving to —
progressive mp3 out, HLS back — via `selectionAt()`, rather than leaving it to be
inferred.
The quality picker is filled from `selection.available` (DR-227): rungs the
backend marked `exceedsSource` are not drawn, because they produce the same bytes
as `Original`. Nothing is optimistically assigned when the viewer picks a rung —
what the menu shows comes from the selection the backend hands back, since a
ceiling above the source bitrate *is* the source.
## Native Video Store
**Location**: `src/lib/stores/nativeVideo.ts`
+49
View File
@@ -132,6 +132,55 @@ sequenceDiagram
Note over Store: UI updates reactively
```
## Video Stream Selection Flow
**TRACES: UR-070, UR-079 | DR-225, DR-227, DR-228**
Before a video plays, Rust decides *what stream* — direct play, remux or
transcode, over which transport — and hands the player one self-describing
`StreamSelection`. The page no longer inspects the URL to work any of this out.
```mermaid
sequenceDiagram
participant Page as player/[id]/+page.svelte
participant Repo as HybridRepository
participant Online as OnlineRepository
participant Server as Jellyfin
participant VP as VideoPlayer.svelte
Page->>Repo: playerLocalMediaPath(id)
alt a completed download exists
Page->>Repo: mediaLocalSelection(path)
Note over Page: LocalFile / DirectPlay, no ladder —<br/>nothing about a file on disk re-negotiates
else stream from the server
Page->>Repo: getStreamSelection(id, mediaSourceId)
Repo->>Online: get_stream_selection()
Online->>Online: effective_streaming_quality()
Note over Online: per-playback override, else device default
Online->>Server: POST /Items/{id}/PlaybackInfo<br/>(device profile + ceiling)
Server-->>Online: MediaSource {supportsDirectPlay,<br/>supportsDirectStream, transcodingUrl, bitrate}
Online->>Online: decide_playback_kind()
alt Transcode
Online->>Online: adopt/stop prior play session,<br/>build HLS URL
Note over Online: Transport::Hls
else DirectPlay / DirectStream
Online->>Online: /Videos/{id}/stream?static=true
Note over Online: Transport::Progressive,<br/>rendition = None (it IS the source)
end
Online->>Online: quality_options_for_source(bitrate)
Online-->>Page: StreamSelection
end
Page->>VP: selection
VP->>VP: videoLoaderFor(selection, caps)
Note over VP: hls.js / native HLS / direct —<br/>from the tag, never from the URL
```
The selection travels with the stream from then on. A reload — a quality change,
an audio-track switch, a transcoded seek — returns a *new* selection through the
same tagged `strategy` response, so transport and URL can never disagree; and the
queue item carries the transport so `player_seek_video` picks its seek strategy
from the backend's decision rather than from the URL string.
## Playback Mode Transfer Flow
```mermaid