feat(player): native video on Linux, and one contract for every player (v0.11.0)
mpv now decodes video on Linux, drawn into a framebuffer we own and blitted
into the default vbox's draw handler. Tauri's widget tree is untouched, so an
upgrade that assumes its own layout cannot invalidate this. Direct play means
the original file, hardware decoding, and no server transcode at all — where
previously every desktop video was re-encoded to h264 for the browser engine,
whatever the file actually was. Off by default: JELLYTAU_NATIVE_VIDEO=1.
That settles finding 2 of playback-backend-unification.md — "native video
cannot be composited with a Tauri webview" — by demonstration rather than
argument, on X11 and Wayland both.
Turning it on exposed nine defects, none of them mpv's. Each was the same
mistake in a different place: a capability written down as a compile-time fact
about the platform, or a state asserted instead of confirmed.
DR-238/246 a seek routed by the stream's container rather than by what the
engine could do with it - correct only while one player handled
those streams, silent the moment another did
DR-239 a property handled but never observed, so the play/pause button
waited for an event that could not arrive
DR-240 fullscreen expanding the document while the window stayed put
DR-241 a seek issued before the engine had a file, failed, and discarded
- which is why resume began at zero
DR-247 a Linux-only gate outliving the caller that made it Linux-only,
breaking the Android build outright
DR-250 a stop aimed at whichever renderer bookkeeping believed was in
charge, missing the one actually making sound
DR-251 a duration of zero believed, leaving the seek bar no scale
DR-252 a junk float converted to a Duration, panicking the backend the
instant a length-less stream appeared
So the MediaPlayer contract (DR-242 … DR-247): `open` carries a start position,
so no caller sequences load-then-seek and none can race an engine's load;
`seek` states a destination and leaves in-place-versus-re-open to the engine;
`snapshot` is one coherent read; and `Phase::Opening` names the window where
intent used to be lost. One conformance suite runs against every engine —
FakePlayer and mpv under cargo test, ExoPlayer instrumented on a device — so an
engine is either correct or visibly failing.
Two of the nine were introduced during this work and caught on hardware, not by
any suite: an over-broad capability that grouped ExoPlayer with mpv, and the
Duration panic. The suites test engines that behave. That is recorded in
docs/native-player-verification.md, which asks for the exact action sequences
that found them.
Verified: all automated gates, conformance (mpv 9/9, legacy 8/9 by design,
ExoPlayer 7/7 on device), and manual desktop and Android passes on real
hardware.
Known open and deliberately shipped: resume reads local progress and never the
server's; the background-audio handoff still declares a state swap it does not
confirm (the symptom is now impossible, the race is not); and `bun run
android:dev` builds an APK carrying the release application id, whose failure
message advises an uninstall that would destroy app data. Fix that last one
before anyone else builds for Android.
Squashed from worktree-linux-native-video, which keeps the per-defect history.
This commit is contained in:
@@ -673,8 +673,161 @@ device profile. Sending it there — not just on the transcode URL — is what m
|
||||
the cap real: a stream the server decides to *direct play* is served at the
|
||||
source file's own bitrate, and no URL parameter afterwards can reduce it.
|
||||
|
||||
#### Two levels of ceiling
|
||||
|
||||
**Location**: `src-tauri/src/repository/online.rs` (TRACES: UR-074, UR-079 | DR-226)
|
||||
|
||||
There are two, and they are not the same thing:
|
||||
|
||||
| | Set by | Lives until | Read via |
|
||||
|---|---|---|---|
|
||||
| **Device default** | Settings (`player_set_video_settings`) | Persisted; restored at startup | `streaming_quality()` |
|
||||
| **Per-playback override** | The in-player picker (`player_set_stream_quality`) | The next item starts playing | `playback_quality_override()` |
|
||||
|
||||
`effective_streaming_quality()` resolves the pair — override first, else default —
|
||||
and **is the only thing stream construction may read**. Every URL builder and the
|
||||
`PlaybackInfo` negotiation go through it, for the reason the process-wide static
|
||||
existed in the first place: if the negotiation and the URL builder disagree, the
|
||||
cap leaks — the negotiation authorises a direct play the builder then never gets
|
||||
to constrain, or the reverse.
|
||||
|
||||
> The override exists because a single global cannot express "this 4K remux needs
|
||||
> a ceiling, that podcast does not". The picker had documented itself as a "this
|
||||
> film, this connection" control since it was written, but was implemented by
|
||||
> writing the *default* — so dropping one awkward film to 2 Mbps silently capped
|
||||
> every video played afterwards for the rest of the process, with Settings still
|
||||
> showing the old value. It is cleared on every `player_play_item` /
|
||||
> `player_play_queue` / `player_play_tracks`, which is what stops it surviving
|
||||
> into an autoplayed next episode where nobody would reopen the picker.
|
||||
|
||||
### Stream selection
|
||||
|
||||
**Location**: `src-tauri/src/repository/stream_selection.rs`,
|
||||
`OnlineRepository::get_stream_selection` (TRACES: UR-070, UR-079 | DR-225, DR-227, DR-228)
|
||||
|
||||
**Rust decides *what stream*. The player decides *how to deliver it*.** That line
|
||||
is the whole design. A backend with genuine adaptive selection (ExoPlayer over a
|
||||
multi-variant playlist) is left to do it; Rust chooses what to request and never
|
||||
paces bytes.
|
||||
|
||||
`get_stream_selection` returns one self-describing `StreamSelection` in place of
|
||||
the bare URL `get_video_stream_url` used to hand out:
|
||||
|
||||
| Field | Carries |
|
||||
|---|---|
|
||||
| `url` | What to open |
|
||||
| `transport` | `Hls` / `Progressive` / `LocalFile` — how to fetch it |
|
||||
| `playback_kind` | `DirectPlay` / `DirectStream` / `Transcode` — what the server is doing to the source |
|
||||
| `rendition` | The negotiated ceiling and codecs; `None` for a direct play, which *is* the source |
|
||||
| `available` | The quality ladder as it applies to this media source (DR-227) |
|
||||
| `needs_transcoding` | Derived from `playback_kind`, so the rule is answered once |
|
||||
|
||||
Both enums are serde-tagged (`{"type":"hls"}`) so the frontend matches a
|
||||
discriminant rather than comparing text.
|
||||
|
||||
> **Why `transport` exists.** `VideoPlayer.svelte` chose its loader with
|
||||
> `url.includes(".m3u8")`, in two places. Rust *built* that URL and knows exactly
|
||||
> what it is; re-deriving it downstream by substring match is a domain fact
|
||||
> reconstructed in the presentation layer — the same class of error as leaking
|
||||
> item-type taxonomy, and one that fails silently in **both** directions: a
|
||||
> progressive file served from a path containing the substring gets an HLS
|
||||
> loader, and a playlist served from a path without it does not.
|
||||
>
|
||||
> The paths that never negotiate get the same shape from Rust rather than letting
|
||||
> a caller assemble one — `media_local_selection` for a downloaded file,
|
||||
> `LiveStreamInfo.transport` for a live channel — so there is no second place
|
||||
> where a transport is decided.
|
||||
|
||||
#### The playback-kind decision
|
||||
|
||||
`decide_playback_kind` is a free function and pure, so every branch is testable
|
||||
from `PlaybackInfo` fixtures without a server. Order matters — the two
|
||||
client-side overrides come first, because each describes a case where the
|
||||
server's answer is right about the *file* and wrong about what this app will do
|
||||
with it:
|
||||
|
||||
1. **Undecodable audio → `Transcode`.** Jellyfin 10.11.5 honours a
|
||||
DirectPlayProfile's container and video codec but *ignores its audio codec*,
|
||||
so it offers direct play for an E-AC-3 track the webview renders in silence.
|
||||
A silent direct play is worse than a transcode.
|
||||
2. **A pinned audio track → `Transcode`.** Not a defect in the server's answer, a
|
||||
different question: the file has one default track and the viewer asked for
|
||||
another.
|
||||
3. Otherwise `supports_direct_play` → `DirectPlay`, else `supports_direct_stream`
|
||||
→ `DirectStream`, else `Transcode`.
|
||||
|
||||
A direct **stream** is a remux — codecs copied, container repackaged. It is cheap
|
||||
and is deliberately *not* counted as transcoding; conflating the two would report
|
||||
a free passthrough as a server-side re-encode.
|
||||
|
||||
> **What this is worth, measured.** Against the development server (Jellyfin
|
||||
> 10.11.5), 400 items sampled for codec mix and 40 put through a real negotiation
|
||||
> per profile:
|
||||
>
|
||||
> | Profile | Direct play |
|
||||
> |---|---|
|
||||
> | Linux / WebKitGTK (`h264` only, 2ch) | 3/40 — **7%** |
|
||||
> | Android / ExoPlayer (`h264,hevc,vp8,vp9,av1,mpeg4` + `ac3,eac3`, 6ch) | 34/40 — **85%** |
|
||||
>
|
||||
> The library is ~80% hevc (`hevc+eac3` alone is a third of it), which is why the
|
||||
> two diverge so hard.
|
||||
>
|
||||
> **Read that 85% as a ceiling, not a result.** It was measured with a profile
|
||||
> containing `ac3,eac3`. The Android device this was later run on reports neither
|
||||
> in its `MediaCodecList` — no Dolby licence, which is normal for a tablet — so
|
||||
> eac3 content, about a third of the sampled library, correctly transcodes there.
|
||||
> What any given device achieves depends on its own codec list, and on the
|
||||
> profile being derived from the renderer at all (DR-234), which it was not when
|
||||
> the figure was taken.
|
||||
>
|
||||
> **The payoff is still overwhelmingly Android**, because that is where a real
|
||||
> decoder is already doing the work. Linux stays near 7% until libmpv decodes the
|
||||
> picture — the h264-only profile is a WebKitGTK constraint, not a JellyTau
|
||||
> choice, and is what `linux-native-video-spike.md` exists to remove. A reviewer
|
||||
> should not expect this code to fix Linux on its own.
|
||||
|
||||
#### The quality ladder per source
|
||||
|
||||
`quality_options_for_source(source_bitrate)` returns every rung, each marked with
|
||||
`exceeds_source`: true when that rung's ceiling is at or above what the source
|
||||
itself carries, so selecting it produces the same bytes as `Original`. The
|
||||
frontend draws the list and drops the redundant rungs; it does not decide which
|
||||
they are.
|
||||
|
||||
- `Original` is never marked — it *is* the source.
|
||||
- An unreported source bitrate (some containers have none; the sampled library
|
||||
has `avi` files with no bitrate at all) marks **nothing** redundant, keeping
|
||||
every rung offered. That is the safe direction: the viewer keeps every choice.
|
||||
|
||||
#### No adaptive ladder to preserve
|
||||
|
||||
**TRACES: UR-079 | DR-229 (Won't Do)**
|
||||
|
||||
Mid-playback re-negotiation on throughput was scoped and dropped on measurement.
|
||||
A master playlist from this server carries exactly **one** `EXT-X-STREAM-INF`:
|
||||
Jellyfin builds it from the single rendition the request asked for rather than
|
||||
publishing a ladder. So there is no adaptation for hls.js to be preserving and
|
||||
none that mpv would lose — the claim that there was is recorded in
|
||||
`playback-backend-unification.md` and does not hold. "Adapt mid-stream" collapses
|
||||
into "pick well at open", which is what the two levels of ceiling and the
|
||||
per-source ladder already are.
|
||||
|
||||
Kept here because it is a measurement, not an opinion: a server that *does*
|
||||
publish a ladder would change the answer, and the re-negotiation path below is
|
||||
the hook that work would build on.
|
||||
|
||||
#### Re-negotiation
|
||||
|
||||
One mechanism, not two. `player_seek_video`, `player_switch_audio_track` and
|
||||
`player_set_stream_quality` all return a tagged `strategy` saying who reloads —
|
||||
the backend handles a native backend itself and hands the webview a
|
||||
`StreamSelection` for `reloadSource`. Note the wire wart: tauri-specta keeps
|
||||
these response fields snake_case (`seek_offset`), while the `strategy` tag itself
|
||||
is camelCase.
|
||||
|
||||
The frontend names a variant and nothing else; the labels the picker shows are
|
||||
served over IPC by `player_get_streaming_qualities`.
|
||||
served over IPC — from `available` on the selection, or
|
||||
`player_get_streaming_qualities` for the Settings list.
|
||||
|
||||
## Background workers
|
||||
|
||||
|
||||
@@ -802,6 +802,45 @@ by exactly the inset.
|
||||
Unlike `addJavascriptInterface`, the inset push only writes CSS properties, so it
|
||||
can safely be re-sent on resume.
|
||||
|
||||
## Stream Transport
|
||||
|
||||
**Location**: `src/lib/player/streamTransport.ts`
|
||||
**TRACES**: UR-079 | DR-225 | UT-214
|
||||
|
||||
`videoLoaderFor(selection, capabilities)` picks the loader for the webview
|
||||
`<video>` element — `hlsjs`, `nativeHls`, or `direct` — from the backend's tagged
|
||||
`selection.transport`. `elementSrcFor` is its template companion: the element's
|
||||
`src` is emptied only when hls.js is driving it.
|
||||
|
||||
The split is the point. **The transport is the stream's property and comes from
|
||||
Rust; whether a given loader exists is the browser's, and is the only thing
|
||||
decided here.**
|
||||
|
||||
> This replaced `currentStreamUrl.includes(".m3u8")`, which appeared twice in
|
||||
> `VideoPlayer.svelte` — once in the HLS `$effect` and once inline in the
|
||||
> template's `src`. Rust builds that URL and knows what it is; re-deriving it
|
||||
> here by substring match was a domain fact reconstructed in the presentation
|
||||
> layer, and it fails silently in both directions. The two tests that pin it are
|
||||
> the ones that failed against the old implementation: a `progressive` stream
|
||||
> whose URL contains `.m3u8` must **not** get an HLS loader, and an `hls` stream
|
||||
> whose URL contains no `.m3u8` must.
|
||||
>
|
||||
> Logic lives in a plain `.ts` module rather than in the component for the usual
|
||||
> reason — it is testable there. Same pattern as `episodeStrip.ts`.
|
||||
|
||||
`VideoPlayer` holds a `currentSelection`, not a URL string; `currentStreamUrl` is
|
||||
derived from it. A reload replaces the selection **wholesale** (the adapter's
|
||||
bridge takes a `StreamSelection`, not a URL), so transport and URL can never
|
||||
drift apart. The background-audio handoff states the transport it is moving to —
|
||||
progressive mp3 out, HLS back — via `selectionAt()`, rather than leaving it to be
|
||||
inferred.
|
||||
|
||||
The quality picker is filled from `selection.available` (DR-227): rungs the
|
||||
backend marked `exceedsSource` are not drawn, because they produce the same bytes
|
||||
as `Original`. Nothing is optimistically assigned when the viewer picks a rung —
|
||||
what the menu shows comes from the selection the backend hands back, since a
|
||||
ceiling above the source bitrate *is* the source.
|
||||
|
||||
## Native Video Store
|
||||
|
||||
**Location**: `src/lib/stores/nativeVideo.ts`
|
||||
|
||||
@@ -132,6 +132,55 @@ sequenceDiagram
|
||||
Note over Store: UI updates reactively
|
||||
```
|
||||
|
||||
## Video Stream Selection Flow
|
||||
|
||||
**TRACES: UR-070, UR-079 | DR-225, DR-227, DR-228**
|
||||
|
||||
Before a video plays, Rust decides *what stream* — direct play, remux or
|
||||
transcode, over which transport — and hands the player one self-describing
|
||||
`StreamSelection`. The page no longer inspects the URL to work any of this out.
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Page as player/[id]/+page.svelte
|
||||
participant Repo as HybridRepository
|
||||
participant Online as OnlineRepository
|
||||
participant Server as Jellyfin
|
||||
participant VP as VideoPlayer.svelte
|
||||
|
||||
Page->>Repo: playerLocalMediaPath(id)
|
||||
alt a completed download exists
|
||||
Page->>Repo: mediaLocalSelection(path)
|
||||
Note over Page: LocalFile / DirectPlay, no ladder —<br/>nothing about a file on disk re-negotiates
|
||||
else stream from the server
|
||||
Page->>Repo: getStreamSelection(id, mediaSourceId)
|
||||
Repo->>Online: get_stream_selection()
|
||||
Online->>Online: effective_streaming_quality()
|
||||
Note over Online: per-playback override, else device default
|
||||
Online->>Server: POST /Items/{id}/PlaybackInfo<br/>(device profile + ceiling)
|
||||
Server-->>Online: MediaSource {supportsDirectPlay,<br/>supportsDirectStream, transcodingUrl, bitrate}
|
||||
Online->>Online: decide_playback_kind()
|
||||
alt Transcode
|
||||
Online->>Online: adopt/stop prior play session,<br/>build HLS URL
|
||||
Note over Online: Transport::Hls
|
||||
else DirectPlay / DirectStream
|
||||
Online->>Online: /Videos/{id}/stream?static=true
|
||||
Note over Online: Transport::Progressive,<br/>rendition = None (it IS the source)
|
||||
end
|
||||
Online->>Online: quality_options_for_source(bitrate)
|
||||
Online-->>Page: StreamSelection
|
||||
end
|
||||
Page->>VP: selection
|
||||
VP->>VP: videoLoaderFor(selection, caps)
|
||||
Note over VP: hls.js / native HLS / direct —<br/>from the tag, never from the URL
|
||||
```
|
||||
|
||||
The selection travels with the stream from then on. A reload — a quality change,
|
||||
an audio-track switch, a transcoded seek — returns a *new* selection through the
|
||||
same tagged `strategy` response, so transport and URL can never disagree; and the
|
||||
queue item carries the transport so `player_seek_video` picks its seek strategy
|
||||
from the backend's decision rather than from the URL string.
|
||||
|
||||
## Playback Mode Transfer Flow
|
||||
|
||||
```mermaid
|
||||
|
||||
Reference in New Issue
Block a user