From 83dc8c70280932d23de2aa23e5380fba5c585a95 Mon Sep 17 00:00:00 2001 From: Duncan Tourolle Date: Fri, 21 Aug 2026 22:33:58 +0200 Subject: [PATCH] feat(playback): let Rust decide what stream to play, and say so MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Playing a video meant asking the server to re-encode it, always. That decision was made nowhere and written down nowhere, so whoever needed it re-derived it downstream — the player worked out whether it had been handed a playlist by looking for ".m3u8" in the URL, in two places. A viewer paid for a transcode of a file their device could have played untouched, and the app could not tell them which it was. One negotiation now produces one self-describing StreamSelection — direct play, remux or transcode; over a playlist, a plain HTTP file, or a local one — and every renderer consumes that same answer. Measured against the development server (Jellyfin 10.11.5), 400 items sampled for codec mix and 40 put through a real PlaybackInfo negotiation per profile: Linux / WebKitGTK (h264 only, 2ch) 3/40 — 7% direct play Android / ExoPlayer (hevc, ac3/eac3, 6ch) 34/40 — 85% direct play The library is ~80% hevc, which is why the two diverge so hard. The payoff is overwhelmingly Android, where 85% of plays were starting a transcode nobody needed. Linux stays near 7% until libmpv decodes the picture — the h264-only profile is a WebKitGTK constraint, not a JellyTau choice. DR-219 StreamSelection: url + tagged Transport (hls/progressive/localFile) + PlaybackKind (directPlay/directStream/transcode) + the negotiated rendition + this source's ladder + a needs_transcoding flag derived in Rust so the rule is answered once. Both enums are serde-tagged so the frontend matches a discriminant, not a substring. The paths that never negotiate get the same shape from Rust rather than assembling one — media_local_selection for a downloaded file, LiveStreamInfo.transport for a live channel — so there is no second place where a transport is decided. DR-220 The ceiling becomes two levels: a durable device default (Settings, persisted) and a per-playback override the in-player picker sets. The picker had called itself a "this film, this connection" control since it was written but wrote the process-wide default, so dropping one awkward film to 2 Mbps silently capped every video played afterwards for the rest of the process, with Settings still showing the old value. The override is cleared whenever playback moves to a new item, which stops it surviving into an autoplayed next episode. effective_streaming_quality() is the single resolution point. DR-221 The quality picker is filled from what this media source can offer. Rust marks a rung exceeds_source when its ceiling is at or above the source's own bitrate — such a rung is another way to spell Original — and the frontend does not draw those. Original is never marked; a source whose bitrate the server does not report marks nothing, which keeps every rung offered. DR-222 Direct play and direct stream are negotiated, with two client-side overrides on top because the server's answer is right about the file and wrong about what this app will do with it: undecodable audio (Jellyfin 10.11.5 honours a DirectPlayProfile's container and video codec but ignores its audio codec, so it offers direct play for an E-AC-3 track the webview renders in silence) and a viewer-pinned audio track the file does not default to. A direct stream is a remux and is deliberately not counted as transcoding. DR-223 Dropped on measurement, not deferred. A master playlist from this server carries exactly one EXT-X-STREAM-INF: Jellyfin builds it from the single rendition the request asked for rather than publishing a ladder. So there is no adaptation for hls.js to be preserving and none mpv would lose — the claim that there was, in playback-backend-unification.md, does not hold. Recorded rather than deleted because it is a measurement: a server that does publish a ladder would change the answer. DR-224 Every backend consumes the same selection. The queue item carries the transport, so player_seek_video picks its seek strategy from the backend's decision instead of the last stream_url.contains(".m3u8") in the codebase. Items queued by a path that never negotiated carry None and fall back to needs_transcoding, which is exact rather than a guess because every transcode this app requests is HLS (DR-140). The frontend loader decision moves to streamTransport.ts so it can be tested: the two cases that pin it are the ones that failed against the old implementation — a progressive stream whose URL contains ".m3u8" must not get an HLS loader, and an HLS stream whose URL contains none must. Also verified the URL the direct-play branch builds actually serves playable bytes: 206, video/mp4, valid ISO-BMFF, and a mid-file range works, so seeking a direct play works. The spec is folded into docs/architecture/{01,02,03} and deleted, per the rule that docs/specs holds only work that has not shipped. DR-121 leaves read-through-media-cache.md with a pointer; that spec keeps its capture half. Not verified: real playback on a device. Direct play changes what actually gets played, and neither fixtures nor curl prove the WebKitGTK and ExoPlayer paths render it. --- .gitea/workflows/build-and-test.yml | 2 +- docs-site/SUMMARY.md | 1 - docs/architecture/01-rust-backend.md | 145 +- docs/architecture/02-svelte-frontend.md | 39 + docs/architecture/03-data-flow.md | 49 + docs/requirements.md | 11 + docs/specs/README.md | 7 +- docs/specs/backend-owned-stream-selection.md | 242 - docs/specs/read-through-media-cache.md | 48 +- docs/traceability.md | 12917 +++++++++------- src-tauri/src/commands/player/mod.rs | 194 +- src-tauri/src/commands/player/queue.rs | 4 + src-tauri/src/commands/repository.rs | 31 +- src-tauri/src/commands/storage/mod.rs | 24 + src-tauri/src/lib.rs | 4 + src-tauri/src/playback_mode/mod.rs | 4 + src-tauri/src/player/backend.rs | 6 + src-tauri/src/player/media.rs | 24 + src-tauri/src/player/mod.rs | 18 + src-tauri/src/player/queue.rs | 2 + src-tauri/src/player/session.rs | 4 + src-tauri/src/player/state.rs | 2 + src-tauri/src/player/stream_end.rs | 8 + src-tauri/src/repository/hybrid.rs | 18 + src-tauri/src/repository/mod.rs | 3 + src-tauri/src/repository/online.rs | 1023 +- src-tauri/src/repository/stream_selection.rs | 416 + src-tauri/src/repository/types.rs | 9 + src/lib/api/bindings.ts | 265 +- src/lib/api/repository-client.ts | 30 +- .../player/VideoPlayer.nativeReveal.test.ts | 23 +- .../VideoPlayer.scrubRegression.test.ts | 19 +- src/lib/components/player/VideoPlayer.svelte | 223 +- .../player/VideoPlayer.tapSurface.test.ts | 19 +- .../player/VideoPlayer.touchScrub.test.ts | 19 +- src/lib/player/adapters/html5Adapter.test.ts | 33 +- src/lib/player/adapters/html5Adapter.ts | 63 +- src/lib/player/adapters/nativeAdapter.test.ts | 19 +- src/lib/player/adapters/nativeAdapter.ts | 3 +- src/lib/player/adapters/types.ts | 28 +- .../player/adapters/webviewAudioAdapter.ts | 5 +- src/lib/player/index.ts | 29 +- src/lib/player/streamTransport.test.ts | 93 + src/lib/player/streamTransport.ts | 68 + src/routes/player/[id]/+page.svelte | 120 +- 45 files changed, 9733 insertions(+), 6581 deletions(-) delete mode 100644 docs/specs/backend-owned-stream-selection.md create mode 100644 src-tauri/src/repository/stream_selection.rs create mode 100644 src/lib/player/streamTransport.test.ts create mode 100644 src/lib/player/streamTransport.ts diff --git a/.gitea/workflows/build-and-test.yml b/.gitea/workflows/build-and-test.yml index 6b3a856c..f3907ac3 100644 --- a/.gitea/workflows/build-and-test.yml +++ b/.gitea/workflows/build-and-test.yml @@ -109,7 +109,7 @@ jobs: # at "warn" until its class is cleared and it can be promoted to "error". # Lower this as you clear them. Never raise it to make a build pass. - name: Lint - run: bun run lint -- --max-warnings=159 + run: bun run lint -- --max-warnings=158 - name: Check TypeScript run: | diff --git a/docs-site/SUMMARY.md b/docs-site/SUMMARY.md index 45a207be..ce86b824 100644 --- a/docs-site/SUMMARY.md +++ b/docs-site/SUMMARY.md @@ -33,7 +33,6 @@ - [Spec Review Checklist](specs/SPEC-REVIEW-CHECKLIST.md) - [Playback Backend Unification](specs/playback-backend-unification.md) - [Linux Native Video Spike](specs/linux-native-video-spike.md) -- [Backend-Owned Stream Selection](specs/backend-owned-stream-selection.md) - [Player Facade Enforcement](specs/player-facade-enforcement.md) - [Windows Native Audio Backend](specs/windows-native-audio-backend.md) - [libmpv2 Migration](specs/libmpv2-migration.md) diff --git a/docs/architecture/01-rust-backend.md b/docs/architecture/01-rust-backend.md index 8486aeb0..75997c7c 100644 --- a/docs/architecture/01-rust-backend.md +++ b/docs/architecture/01-rust-backend.md @@ -673,8 +673,151 @@ device profile. Sending it there — not just on the transcode URL — is what m the cap real: a stream the server decides to *direct play* is served at the source file's own bitrate, and no URL parameter afterwards can reduce it. +#### Two levels of ceiling + +**Location**: `src-tauri/src/repository/online.rs` (TRACES: UR-074, UR-079 | DR-225) + +There are two, and they are not the same thing: + +| | Set by | Lives until | Read via | +|---|---|---|---| +| **Device default** | Settings (`player_set_video_settings`) | Persisted; restored at startup | `streaming_quality()` | +| **Per-playback override** | The in-player picker (`player_set_stream_quality`) | The next item starts playing | `playback_quality_override()` | + +`effective_streaming_quality()` resolves the pair — override first, else default — +and **is the only thing stream construction may read**. Every URL builder and the +`PlaybackInfo` negotiation go through it, for the reason the process-wide static +existed in the first place: if the negotiation and the URL builder disagree, the +cap leaks — the negotiation authorises a direct play the builder then never gets +to constrain, or the reverse. + +> The override exists because a single global cannot express "this 4K remux needs +> a ceiling, that podcast does not". The picker had documented itself as a "this +> film, this connection" control since it was written, but was implemented by +> writing the *default* — so dropping one awkward film to 2 Mbps silently capped +> every video played afterwards for the rest of the process, with Settings still +> showing the old value. It is cleared on every `player_play_item` / +> `player_play_queue` / `player_play_tracks`, which is what stops it surviving +> into an autoplayed next episode where nobody would reopen the picker. + +### Stream selection + +**Location**: `src-tauri/src/repository/stream_selection.rs`, +`OnlineRepository::get_stream_selection` (TRACES: UR-070, UR-079 | DR-224, DR-226, DR-227) + +**Rust decides *what stream*. The player decides *how to deliver it*.** That line +is the whole design. A backend with genuine adaptive selection (ExoPlayer over a +multi-variant playlist) is left to do it; Rust chooses what to request and never +paces bytes. + +`get_stream_selection` returns one self-describing `StreamSelection` in place of +the bare URL `get_video_stream_url` used to hand out: + +| Field | Carries | +|---|---| +| `url` | What to open | +| `transport` | `Hls` / `Progressive` / `LocalFile` — how to fetch it | +| `playback_kind` | `DirectPlay` / `DirectStream` / `Transcode` — what the server is doing to the source | +| `rendition` | The negotiated ceiling and codecs; `None` for a direct play, which *is* the source | +| `available` | The quality ladder as it applies to this media source (DR-226) | +| `needs_transcoding` | Derived from `playback_kind`, so the rule is answered once | + +Both enums are serde-tagged (`{"type":"hls"}`) so the frontend matches a +discriminant rather than comparing text. + +> **Why `transport` exists.** `VideoPlayer.svelte` chose its loader with +> `url.includes(".m3u8")`, in two places. Rust *built* that URL and knows exactly +> what it is; re-deriving it downstream by substring match is a domain fact +> reconstructed in the presentation layer — the same class of error as leaking +> item-type taxonomy, and one that fails silently in **both** directions: a +> progressive file served from a path containing the substring gets an HLS +> loader, and a playlist served from a path without it does not. +> +> The paths that never negotiate get the same shape from Rust rather than letting +> a caller assemble one — `media_local_selection` for a downloaded file, +> `LiveStreamInfo.transport` for a live channel — so there is no second place +> where a transport is decided. + +#### The playback-kind decision + +`decide_playback_kind` is a free function and pure, so every branch is testable +from `PlaybackInfo` fixtures without a server. Order matters — the two +client-side overrides come first, because each describes a case where the +server's answer is right about the *file* and wrong about what this app will do +with it: + +1. **Undecodable audio → `Transcode`.** Jellyfin 10.11.5 honours a + DirectPlayProfile's container and video codec but *ignores its audio codec*, + so it offers direct play for an E-AC-3 track the webview renders in silence. + A silent direct play is worse than a transcode. +2. **A pinned audio track → `Transcode`.** Not a defect in the server's answer, a + different question: the file has one default track and the viewer asked for + another. +3. Otherwise `supports_direct_play` → `DirectPlay`, else `supports_direct_stream` + → `DirectStream`, else `Transcode`. + +A direct **stream** is a remux — codecs copied, container repackaged. It is cheap +and is deliberately *not* counted as transcoding; conflating the two would report +a free passthrough as a server-side re-encode. + +> **What this is worth, measured.** Against the development server (Jellyfin +> 10.11.5), 400 items sampled for codec mix and 40 put through a real negotiation +> per profile: +> +> | Profile | Direct play | +> |---|---| +> | Linux / WebKitGTK (`h264` only, 2ch) | 3/40 — **7%** | +> | Android / ExoPlayer (`h264,hevc,vp8,vp9,av1,mpeg4` + `ac3,eac3`, 6ch) | 34/40 — **85%** | +> +> The library is ~80% hevc (`hevc+eac3` alone is a third of it), which is why the +> two diverge so hard. **The payoff is overwhelmingly Android**, where 85% of +> plays previously burned a transcode nobody needed. Linux stays near 7% until +> libmpv decodes the picture — the h264-only profile is a WebKitGTK constraint, +> not a JellyTau choice, and is what `linux-native-video-spike.md` exists to +> remove. A reviewer should not expect this code to fix Linux on its own. + +#### The quality ladder per source + +`quality_options_for_source(source_bitrate)` returns every rung, each marked with +`exceeds_source`: true when that rung's ceiling is at or above what the source +itself carries, so selecting it produces the same bytes as `Original`. The +frontend draws the list and drops the redundant rungs; it does not decide which +they are. + +- `Original` is never marked — it *is* the source. +- An unreported source bitrate (some containers have none; the sampled library + has `avi` files with no bitrate at all) marks **nothing** redundant, keeping + every rung offered. That is the safe direction: the viewer keeps every choice. + +#### No adaptive ladder to preserve + +**TRACES: UR-079 | DR-228 (Won't Do)** + +Mid-playback re-negotiation on throughput was scoped and dropped on measurement. +A master playlist from this server carries exactly **one** `EXT-X-STREAM-INF`: +Jellyfin builds it from the single rendition the request asked for rather than +publishing a ladder. So there is no adaptation for hls.js to be preserving and +none that mpv would lose — the claim that there was is recorded in +`playback-backend-unification.md` and does not hold. "Adapt mid-stream" collapses +into "pick well at open", which is what the two levels of ceiling and the +per-source ladder already are. + +Kept here because it is a measurement, not an opinion: a server that *does* +publish a ladder would change the answer, and the re-negotiation path below is +the hook that work would build on. + +#### Re-negotiation + +One mechanism, not two. `player_seek_video`, `player_switch_audio_track` and +`player_set_stream_quality` all return a tagged `strategy` saying who reloads — +the backend handles a native backend itself and hands the webview a +`StreamSelection` for `reloadSource`. Note the wire wart: tauri-specta keeps +these response fields snake_case (`seek_offset`), while the `strategy` tag itself +is camelCase. + The frontend names a variant and nothing else; the labels the picker shows are -served over IPC by `player_get_streaming_qualities`. +served over IPC — from `available` on the selection, or +`player_get_streaming_qualities` for the Settings list. ## Background workers diff --git a/docs/architecture/02-svelte-frontend.md b/docs/architecture/02-svelte-frontend.md index 539253bb..2e0d5afa 100644 --- a/docs/architecture/02-svelte-frontend.md +++ b/docs/architecture/02-svelte-frontend.md @@ -802,6 +802,45 @@ by exactly the inset. Unlike `addJavascriptInterface`, the inset push only writes CSS properties, so it can safely be re-sent on resume. +## Stream Transport + +**Location**: `src/lib/player/streamTransport.ts` +**TRACES**: UR-079 | DR-224 | UT-213 + +`videoLoaderFor(selection, capabilities)` picks the loader for the webview +`