feat(playback): let Rust decide what stream to play, and say so
Playing a video meant asking the server to re-encode it, always. That
decision was made nowhere and written down nowhere, so whoever needed it
re-derived it downstream — the player worked out whether it had been handed
a playlist by looking for ".m3u8" in the URL, in two places. A viewer paid
for a transcode of a file their device could have played untouched, and the
app could not tell them which it was.
One negotiation now produces one self-describing StreamSelection — direct
play, remux or transcode; over a playlist, a plain HTTP file, or a local one
— and every renderer consumes that same answer.
Measured against the development server (Jellyfin 10.11.5), 400 items
sampled for codec mix and 40 put through a real PlaybackInfo negotiation
per profile:
Linux / WebKitGTK (h264 only, 2ch) 3/40 — 7% direct play
Android / ExoPlayer (hevc, ac3/eac3, 6ch) 34/40 — 85% direct play
The library is ~80% hevc, which is why the two diverge so hard. The payoff
is overwhelmingly Android, where 85% of plays were starting a transcode
nobody needed. Linux stays near 7% until libmpv decodes the picture — the
h264-only profile is a WebKitGTK constraint, not a JellyTau choice.
DR-219 StreamSelection: url + tagged Transport (hls/progressive/localFile)
+ PlaybackKind (directPlay/directStream/transcode) + the negotiated
rendition + this source's ladder + a needs_transcoding flag derived
in Rust so the rule is answered once. Both enums are serde-tagged
so the frontend matches a discriminant, not a substring. The paths
that never negotiate get the same shape from Rust rather than
assembling one — media_local_selection for a downloaded file,
LiveStreamInfo.transport for a live channel — so there is no second
place where a transport is decided.
DR-220 The ceiling becomes two levels: a durable device default (Settings,
persisted) and a per-playback override the in-player picker sets.
The picker had called itself a "this film, this connection" control
since it was written but wrote the process-wide default, so dropping
one awkward film to 2 Mbps silently capped every video played
afterwards for the rest of the process, with Settings still showing
the old value. The override is cleared whenever playback moves to a
new item, which stops it surviving into an autoplayed next episode.
effective_streaming_quality() is the single resolution point.
DR-221 The quality picker is filled from what this media source can offer.
Rust marks a rung exceeds_source when its ceiling is at or above the
source's own bitrate — such a rung is another way to spell Original
— and the frontend does not draw those. Original is never marked; a
source whose bitrate the server does not report marks nothing, which
keeps every rung offered.
DR-222 Direct play and direct stream are negotiated, with two client-side
overrides on top because the server's answer is right about the file
and wrong about what this app will do with it: undecodable audio
(Jellyfin 10.11.5 honours a DirectPlayProfile's container and video
codec but ignores its audio codec, so it offers direct play for an
E-AC-3 track the webview renders in silence) and a viewer-pinned
audio track the file does not default to. A direct stream is a remux
and is deliberately not counted as transcoding.
DR-223 Dropped on measurement, not deferred. A master playlist from this
server carries exactly one EXT-X-STREAM-INF: Jellyfin builds it from
the single rendition the request asked for rather than publishing a
ladder. So there is no adaptation for hls.js to be preserving and
none mpv would lose — the claim that there was, in
playback-backend-unification.md, does not hold. Recorded rather than
deleted because it is a measurement: a server that does publish a
ladder would change the answer.
DR-224 Every backend consumes the same selection. The queue item carries
the transport, so player_seek_video picks its seek strategy from the
backend's decision instead of the last stream_url.contains(".m3u8")
in the codebase. Items queued by a path that never negotiated carry
None and fall back to needs_transcoding, which is exact rather than
a guess because every transcode this app requests is HLS (DR-140).
The frontend loader decision moves to streamTransport.ts so it can be
tested: the two cases that pin it are the ones that failed against the old
implementation — a progressive stream whose URL contains ".m3u8" must not
get an HLS loader, and an HLS stream whose URL contains none must.
Also verified the URL the direct-play branch builds actually serves playable
bytes: 206, video/mp4, valid ISO-BMFF, and a mid-file range works, so
seeking a direct play works.
The spec is folded into docs/architecture/{01,02,03} and deleted, per the
rule that docs/specs holds only work that has not shipped. DR-121 leaves
read-through-media-cache.md with a pointer; that spec keeps its capture half.
Not verified: real playback on a device. Direct play changes what actually
gets played, and neither fixtures nor curl prove the WebKitGTK and ExoPlayer
paths render it.
This commit is contained in:
@@ -673,8 +673,151 @@ device profile. Sending it there — not just on the transcode URL — is what m
|
||||
the cap real: a stream the server decides to *direct play* is served at the
|
||||
source file's own bitrate, and no URL parameter afterwards can reduce it.
|
||||
|
||||
#### Two levels of ceiling
|
||||
|
||||
**Location**: `src-tauri/src/repository/online.rs` (TRACES: UR-074, UR-079 | DR-225)
|
||||
|
||||
There are two, and they are not the same thing:
|
||||
|
||||
| | Set by | Lives until | Read via |
|
||||
|---|---|---|---|
|
||||
| **Device default** | Settings (`player_set_video_settings`) | Persisted; restored at startup | `streaming_quality()` |
|
||||
| **Per-playback override** | The in-player picker (`player_set_stream_quality`) | The next item starts playing | `playback_quality_override()` |
|
||||
|
||||
`effective_streaming_quality()` resolves the pair — override first, else default —
|
||||
and **is the only thing stream construction may read**. Every URL builder and the
|
||||
`PlaybackInfo` negotiation go through it, for the reason the process-wide static
|
||||
existed in the first place: if the negotiation and the URL builder disagree, the
|
||||
cap leaks — the negotiation authorises a direct play the builder then never gets
|
||||
to constrain, or the reverse.
|
||||
|
||||
> The override exists because a single global cannot express "this 4K remux needs
|
||||
> a ceiling, that podcast does not". The picker had documented itself as a "this
|
||||
> film, this connection" control since it was written, but was implemented by
|
||||
> writing the *default* — so dropping one awkward film to 2 Mbps silently capped
|
||||
> every video played afterwards for the rest of the process, with Settings still
|
||||
> showing the old value. It is cleared on every `player_play_item` /
|
||||
> `player_play_queue` / `player_play_tracks`, which is what stops it surviving
|
||||
> into an autoplayed next episode where nobody would reopen the picker.
|
||||
|
||||
### Stream selection
|
||||
|
||||
**Location**: `src-tauri/src/repository/stream_selection.rs`,
|
||||
`OnlineRepository::get_stream_selection` (TRACES: UR-070, UR-079 | DR-224, DR-226, DR-227)
|
||||
|
||||
**Rust decides *what stream*. The player decides *how to deliver it*.** That line
|
||||
is the whole design. A backend with genuine adaptive selection (ExoPlayer over a
|
||||
multi-variant playlist) is left to do it; Rust chooses what to request and never
|
||||
paces bytes.
|
||||
|
||||
`get_stream_selection` returns one self-describing `StreamSelection` in place of
|
||||
the bare URL `get_video_stream_url` used to hand out:
|
||||
|
||||
| Field | Carries |
|
||||
|---|---|
|
||||
| `url` | What to open |
|
||||
| `transport` | `Hls` / `Progressive` / `LocalFile` — how to fetch it |
|
||||
| `playback_kind` | `DirectPlay` / `DirectStream` / `Transcode` — what the server is doing to the source |
|
||||
| `rendition` | The negotiated ceiling and codecs; `None` for a direct play, which *is* the source |
|
||||
| `available` | The quality ladder as it applies to this media source (DR-226) |
|
||||
| `needs_transcoding` | Derived from `playback_kind`, so the rule is answered once |
|
||||
|
||||
Both enums are serde-tagged (`{"type":"hls"}`) so the frontend matches a
|
||||
discriminant rather than comparing text.
|
||||
|
||||
> **Why `transport` exists.** `VideoPlayer.svelte` chose its loader with
|
||||
> `url.includes(".m3u8")`, in two places. Rust *built* that URL and knows exactly
|
||||
> what it is; re-deriving it downstream by substring match is a domain fact
|
||||
> reconstructed in the presentation layer — the same class of error as leaking
|
||||
> item-type taxonomy, and one that fails silently in **both** directions: a
|
||||
> progressive file served from a path containing the substring gets an HLS
|
||||
> loader, and a playlist served from a path without it does not.
|
||||
>
|
||||
> The paths that never negotiate get the same shape from Rust rather than letting
|
||||
> a caller assemble one — `media_local_selection` for a downloaded file,
|
||||
> `LiveStreamInfo.transport` for a live channel — so there is no second place
|
||||
> where a transport is decided.
|
||||
|
||||
#### The playback-kind decision
|
||||
|
||||
`decide_playback_kind` is a free function and pure, so every branch is testable
|
||||
from `PlaybackInfo` fixtures without a server. Order matters — the two
|
||||
client-side overrides come first, because each describes a case where the
|
||||
server's answer is right about the *file* and wrong about what this app will do
|
||||
with it:
|
||||
|
||||
1. **Undecodable audio → `Transcode`.** Jellyfin 10.11.5 honours a
|
||||
DirectPlayProfile's container and video codec but *ignores its audio codec*,
|
||||
so it offers direct play for an E-AC-3 track the webview renders in silence.
|
||||
A silent direct play is worse than a transcode.
|
||||
2. **A pinned audio track → `Transcode`.** Not a defect in the server's answer, a
|
||||
different question: the file has one default track and the viewer asked for
|
||||
another.
|
||||
3. Otherwise `supports_direct_play` → `DirectPlay`, else `supports_direct_stream`
|
||||
→ `DirectStream`, else `Transcode`.
|
||||
|
||||
A direct **stream** is a remux — codecs copied, container repackaged. It is cheap
|
||||
and is deliberately *not* counted as transcoding; conflating the two would report
|
||||
a free passthrough as a server-side re-encode.
|
||||
|
||||
> **What this is worth, measured.** Against the development server (Jellyfin
|
||||
> 10.11.5), 400 items sampled for codec mix and 40 put through a real negotiation
|
||||
> per profile:
|
||||
>
|
||||
> | Profile | Direct play |
|
||||
> |---|---|
|
||||
> | Linux / WebKitGTK (`h264` only, 2ch) | 3/40 — **7%** |
|
||||
> | Android / ExoPlayer (`h264,hevc,vp8,vp9,av1,mpeg4` + `ac3,eac3`, 6ch) | 34/40 — **85%** |
|
||||
>
|
||||
> The library is ~80% hevc (`hevc+eac3` alone is a third of it), which is why the
|
||||
> two diverge so hard. **The payoff is overwhelmingly Android**, where 85% of
|
||||
> plays previously burned a transcode nobody needed. Linux stays near 7% until
|
||||
> libmpv decodes the picture — the h264-only profile is a WebKitGTK constraint,
|
||||
> not a JellyTau choice, and is what `linux-native-video-spike.md` exists to
|
||||
> remove. A reviewer should not expect this code to fix Linux on its own.
|
||||
|
||||
#### The quality ladder per source
|
||||
|
||||
`quality_options_for_source(source_bitrate)` returns every rung, each marked with
|
||||
`exceeds_source`: true when that rung's ceiling is at or above what the source
|
||||
itself carries, so selecting it produces the same bytes as `Original`. The
|
||||
frontend draws the list and drops the redundant rungs; it does not decide which
|
||||
they are.
|
||||
|
||||
- `Original` is never marked — it *is* the source.
|
||||
- An unreported source bitrate (some containers have none; the sampled library
|
||||
has `avi` files with no bitrate at all) marks **nothing** redundant, keeping
|
||||
every rung offered. That is the safe direction: the viewer keeps every choice.
|
||||
|
||||
#### No adaptive ladder to preserve
|
||||
|
||||
**TRACES: UR-079 | DR-228 (Won't Do)**
|
||||
|
||||
Mid-playback re-negotiation on throughput was scoped and dropped on measurement.
|
||||
A master playlist from this server carries exactly **one** `EXT-X-STREAM-INF`:
|
||||
Jellyfin builds it from the single rendition the request asked for rather than
|
||||
publishing a ladder. So there is no adaptation for hls.js to be preserving and
|
||||
none that mpv would lose — the claim that there was is recorded in
|
||||
`playback-backend-unification.md` and does not hold. "Adapt mid-stream" collapses
|
||||
into "pick well at open", which is what the two levels of ceiling and the
|
||||
per-source ladder already are.
|
||||
|
||||
Kept here because it is a measurement, not an opinion: a server that *does*
|
||||
publish a ladder would change the answer, and the re-negotiation path below is
|
||||
the hook that work would build on.
|
||||
|
||||
#### Re-negotiation
|
||||
|
||||
One mechanism, not two. `player_seek_video`, `player_switch_audio_track` and
|
||||
`player_set_stream_quality` all return a tagged `strategy` saying who reloads —
|
||||
the backend handles a native backend itself and hands the webview a
|
||||
`StreamSelection` for `reloadSource`. Note the wire wart: tauri-specta keeps
|
||||
these response fields snake_case (`seek_offset`), while the `strategy` tag itself
|
||||
is camelCase.
|
||||
|
||||
The frontend names a variant and nothing else; the labels the picker shows are
|
||||
served over IPC by `player_get_streaming_qualities`.
|
||||
served over IPC — from `available` on the selection, or
|
||||
`player_get_streaming_qualities` for the Settings list.
|
||||
|
||||
## Background workers
|
||||
|
||||
|
||||
@@ -802,6 +802,45 @@ by exactly the inset.
|
||||
Unlike `addJavascriptInterface`, the inset push only writes CSS properties, so it
|
||||
can safely be re-sent on resume.
|
||||
|
||||
## Stream Transport
|
||||
|
||||
**Location**: `src/lib/player/streamTransport.ts`
|
||||
**TRACES**: UR-079 | DR-224 | UT-213
|
||||
|
||||
`videoLoaderFor(selection, capabilities)` picks the loader for the webview
|
||||
`<video>` element — `hlsjs`, `nativeHls`, or `direct` — from the backend's tagged
|
||||
`selection.transport`. `elementSrcFor` is its template companion: the element's
|
||||
`src` is emptied only when hls.js is driving it.
|
||||
|
||||
The split is the point. **The transport is the stream's property and comes from
|
||||
Rust; whether a given loader exists is the browser's, and is the only thing
|
||||
decided here.**
|
||||
|
||||
> This replaced `currentStreamUrl.includes(".m3u8")`, which appeared twice in
|
||||
> `VideoPlayer.svelte` — once in the HLS `$effect` and once inline in the
|
||||
> template's `src`. Rust builds that URL and knows what it is; re-deriving it
|
||||
> here by substring match was a domain fact reconstructed in the presentation
|
||||
> layer, and it fails silently in both directions. The two tests that pin it are
|
||||
> the ones that failed against the old implementation: a `progressive` stream
|
||||
> whose URL contains `.m3u8` must **not** get an HLS loader, and an `hls` stream
|
||||
> whose URL contains no `.m3u8` must.
|
||||
>
|
||||
> Logic lives in a plain `.ts` module rather than in the component for the usual
|
||||
> reason — it is testable there. Same pattern as `episodeStrip.ts`.
|
||||
|
||||
`VideoPlayer` holds a `currentSelection`, not a URL string; `currentStreamUrl` is
|
||||
derived from it. A reload replaces the selection **wholesale** (the adapter's
|
||||
bridge takes a `StreamSelection`, not a URL), so transport and URL can never
|
||||
drift apart. The background-audio handoff states the transport it is moving to —
|
||||
progressive mp3 out, HLS back — via `selectionAt()`, rather than leaving it to be
|
||||
inferred.
|
||||
|
||||
The quality picker is filled from `selection.available` (DR-226): rungs the
|
||||
backend marked `exceedsSource` are not drawn, because they produce the same bytes
|
||||
as `Original`. Nothing is optimistically assigned when the viewer picks a rung —
|
||||
what the menu shows comes from the selection the backend hands back, since a
|
||||
ceiling above the source bitrate *is* the source.
|
||||
|
||||
## Native Video Store
|
||||
|
||||
**Location**: `src/lib/stores/nativeVideo.ts`
|
||||
|
||||
@@ -132,6 +132,55 @@ sequenceDiagram
|
||||
Note over Store: UI updates reactively
|
||||
```
|
||||
|
||||
## Video Stream Selection Flow
|
||||
|
||||
**TRACES: UR-070, UR-079 | DR-224, DR-226, DR-227**
|
||||
|
||||
Before a video plays, Rust decides *what stream* — direct play, remux or
|
||||
transcode, over which transport — and hands the player one self-describing
|
||||
`StreamSelection`. The page no longer inspects the URL to work any of this out.
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Page as player/[id]/+page.svelte
|
||||
participant Repo as HybridRepository
|
||||
participant Online as OnlineRepository
|
||||
participant Server as Jellyfin
|
||||
participant VP as VideoPlayer.svelte
|
||||
|
||||
Page->>Repo: playerLocalMediaPath(id)
|
||||
alt a completed download exists
|
||||
Page->>Repo: mediaLocalSelection(path)
|
||||
Note over Page: LocalFile / DirectPlay, no ladder —<br/>nothing about a file on disk re-negotiates
|
||||
else stream from the server
|
||||
Page->>Repo: getStreamSelection(id, mediaSourceId)
|
||||
Repo->>Online: get_stream_selection()
|
||||
Online->>Online: effective_streaming_quality()
|
||||
Note over Online: per-playback override, else device default
|
||||
Online->>Server: POST /Items/{id}/PlaybackInfo<br/>(device profile + ceiling)
|
||||
Server-->>Online: MediaSource {supportsDirectPlay,<br/>supportsDirectStream, transcodingUrl, bitrate}
|
||||
Online->>Online: decide_playback_kind()
|
||||
alt Transcode
|
||||
Online->>Online: adopt/stop prior play session,<br/>build HLS URL
|
||||
Note over Online: Transport::Hls
|
||||
else DirectPlay / DirectStream
|
||||
Online->>Online: /Videos/{id}/stream?static=true
|
||||
Note over Online: Transport::Progressive,<br/>rendition = None (it IS the source)
|
||||
end
|
||||
Online->>Online: quality_options_for_source(bitrate)
|
||||
Online-->>Page: StreamSelection
|
||||
end
|
||||
Page->>VP: selection
|
||||
VP->>VP: videoLoaderFor(selection, caps)
|
||||
Note over VP: hls.js / native HLS / direct —<br/>from the tag, never from the URL
|
||||
```
|
||||
|
||||
The selection travels with the stream from then on. A reload — a quality change,
|
||||
an audio-track switch, a transcoded seek — returns a *new* selection through the
|
||||
same tagged `strategy` response, so transport and URL can never disagree; and the
|
||||
queue item carries the transport so `player_seek_video` picks its seek strategy
|
||||
from the backend's decision rather than from the URL string.
|
||||
|
||||
## Playback Mode Transfer Flow
|
||||
|
||||
```mermaid
|
||||
|
||||
Reference in New Issue
Block a user