feat(playback): let Rust decide what stream to play, and say so

Playing a video meant asking the server to re-encode it, always. That decision was made nowhere and written down nowhere, so whoever needed it re-derived it downstream — the player worked out whether it had been handed a playlist by looking for ".m3u8" in the URL, in two places. A viewer paid for a transcode of a file their device could have played untouched, and the app could not tell them which it was. One negotiation now produces one self-describing StreamSelection — direct play, remux or transcode; over a playlist, a plain HTTP file, or a local one — and every renderer consumes that same answer. Measured against the development server (Jellyfin 10.11.5), 400 items sampled for codec mix and 40 put through a real PlaybackInfo negotiation per profile: Linux / WebKitGTK (h264 only, 2ch) 3/40 — 7% direct play Android / ExoPlayer (hevc, ac3/eac3, 6ch) 34/40 — 85% direct play The library is ~80% hevc, which is why the two diverge so hard. The payoff is overwhelmingly Android, where 85% of plays were starting a transcode nobody needed. Linux stays near 7% until libmpv decodes the picture — the h264-only profile is a WebKitGTK constraint, not a JellyTau choice. DR-219 StreamSelection: url + tagged Transport (hls/progressive/localFile) + PlaybackKind (directPlay/directStream/transcode) + the negotiated rendition + this source's ladder + a needs_transcoding flag derived in Rust so the rule is answered once. Both enums are serde-tagged so the frontend matches a discriminant, not a substring. The paths that never negotiate get the same shape from Rust rather than assembling one — media_local_selection for a downloaded file, LiveStreamInfo.transport for a live channel — so there is no second place where a transport is decided. DR-220 The ceiling becomes two levels: a durable device default (Settings, persisted) and a per-playback override the in-player picker sets. The picker had called itself a "this film, this connection" control since it was written but wrote the process-wide default, so dropping one awkward film to 2 Mbps silently capped every video played afterwards for the rest of the process, with Settings still showing the old value. The override is cleared whenever playback moves to a new item, which stops it surviving into an autoplayed next episode. effective_streaming_quality() is the single resolution point. DR-221 The quality picker is filled from what this media source can offer. Rust marks a rung exceeds_source when its ceiling is at or above the source's own bitrate — such a rung is another way to spell Original — and the frontend does not draw those. Original is never marked; a source whose bitrate the server does not report marks nothing, which keeps every rung offered. DR-222 Direct play and direct stream are negotiated, with two client-side overrides on top because the server's answer is right about the file and wrong about what this app will do with it: undecodable audio (Jellyfin 10.11.5 honours a DirectPlayProfile's container and video codec but ignores its audio codec, so it offers direct play for an E-AC-3 track the webview renders in silence) and a viewer-pinned audio track the file does not default to. A direct stream is a remux and is deliberately not counted as transcoding. DR-223 Dropped on measurement, not deferred. A master playlist from this server carries exactly one EXT-X-STREAM-INF: Jellyfin builds it from the single rendition the request asked for rather than publishing a ladder. So there is no adaptation for hls.js to be preserving and none mpv would lose — the claim that there was, in playback-backend-unification.md, does not hold. Recorded rather than deleted because it is a measurement: a server that does publish a ladder would change the answer. DR-224 Every backend consumes the same selection. The queue item carries the transport, so player_seek_video picks its seek strategy from the backend's decision instead of the last stream_url.contains(".m3u8") in the codebase. Items queued by a path that never negotiated carry None and fall back to needs_transcoding, which is exact rather than a guess because every transcode this app requests is HLS (DR-140). The frontend loader decision moves to streamTransport.ts so it can be tested: the two cases that pin it are the ones that failed against the old implementation — a progressive stream whose URL contains ".m3u8" must not get an HLS loader, and an HLS stream whose URL contains none must. Also verified the URL the direct-play branch builds actually serves playable bytes: 206, video/mp4, valid ISO-BMFF, and a mid-file range works, so seeking a direct play works. The spec is folded into docs/architecture/{01,02,03} and deleted, per the rule that docs/specs holds only work that has not shipped. DR-121 leaves read-through-media-cache.md with a pointer; that spec keeps its capture half. Not verified: real playback on a device. Direct play changes what actually gets played, and neither fixtures nor curl prove the WebKitGTK and ExoPlayer paths render it.
2026-08-22 13:45:03 +02:00
parent 5fede123e7
commit 109700b949
45 changed files with 9744 additions and 6581 deletions
@@ -120,7 +120,7 @@ async playerSeek(position: number) : Promise<PlayerStatus> {
 * 
 * This command analyzes the current video stream and automatically chooses
 * the best seeking strategy:
- * - HLS streams (.m3u8): Use native seeking
+ * - HLS streams: Use native seeking
 * - Direct play streams: Use native seeking
 * - Transcoded non-HLS: Request new stream URL from server starting at seek position
 * 
@@ -263,13 +263,17 @@ async playerGetStreamingQualities() : Promise<([StreamingQuality, string, string
 * two-sided split: HTML5 gets the URL back and reloads its own element, while a
 * native backend is reloaded here.
 * 
- * The change applies to this playback *and* to everything started afterwards
- * (it sets the process-wide ceiling), but it is deliberately **not** persisted:
- * the in-player picker is a "this film, this connection" control, and the
- * durable default belongs to Settings. `player_set_video_settings` is the one
- * that writes to the database.
+ * The change applies to **this playback only**. The in-player picker is a
+ * "this film, this connection" control and its doc has always said so, but it
+ * used to be implemented by writing the process-wide ceiling — so choosing
+ * 2 Mbps to get one awkward film moving silently capped every video played
+ * afterwards for the rest of the process, with the Settings screen still
+ * showing the old value and nothing in the UI admitting the change. It now
+ * sets a per-playback override that the next item clears; the durable default
+ * belongs to Settings, and `player_set_video_settings` is the one that writes
+ * to the database.
 * 
- * TRACES: UR-074 | DR-162
+ * TRACES: UR-074, UR-079 | DR-162, DR-225
 */
 async playerSetStreamQuality(repositoryHandle: string, quality: StreamingQuality, useHtml5: boolean, currentPosition: number | null, mediaSourceId: string | null, audioStreamIndex: number | null) : Promise<StreamQualityResponse> {
    return await TAURI_INVOKE("player_set_stream_quality", { repositoryHandle, quality, useHtml5, currentPosition, mediaSourceId, audioStreamIndex });
@@ -1033,6 +1037,22 @@ async markDownloadFailed(downloadId: number, errorMessage: string) : Promise<nul
 async mediaLocalUrl(path: string) : Promise<string> {
    return await TAURI_INVOKE("media_local_url", { path });
 },
+/**
+ * The stream selection for a downloaded file.
+ * 
+ * The local-playback counterpart to `repository_get_stream_selection`. A file
+ * on disk needs no negotiation — it is a direct play over a local transport,
+ * with no quality ladder, because nothing about it can be re-negotiated — but
+ * the *frontend must not be the one to say so*. It gets the same
+ * [`StreamSelection`] shape as a streamed source so the player has one contract
+ * to consume rather than two, and so no caller has to infer a transport from a
+ * loopback URL.
+ * 
+ * TRACES: UR-071, UR-079 | DR-224
+ */
+async mediaLocalSelection(path: string) : Promise<StreamSelection> {
+    return await TAURI_INVOKE("media_local_selection", { path });
+},
 /**
 * Start downloading a file immediately
 * This command actually downloads the file using the worker
@@ -1618,6 +1638,24 @@ async repositoryGetPlaybackInfo(handle: string, itemId: string) : Promise<Playba
 async repositoryGetVideoStreamUrl(handle: string, itemId: string, mediaSourceId: string | null, audioStreamIndex: number | null) : Promise<string> {
    return await TAURI_INVOKE("repository_get_video_stream_url", { handle, itemId, mediaSourceId, audioStreamIndex });
 },
+/**
+ * Decide what stream to play for a video, and describe it.
+ * 
+ * Replaces `repository_get_video_stream_url` for playback. The returned
+ * [`StreamSelection`] carries the transport explicitly, so the frontend picks
+ * its loader from a tagged enum instead of testing the URL for `.m3u8`; and it
+ * carries the quality ladder as it applies to *this* source, so the picker can
+ * stop offering rungs that produce the same bytes as Original.
+ * 
+ * No start-position parameter, for the same reason as the URL builder: a
+ * position on an HLS playlist is copied onto every segment URI and the server
+ * rejects each with `400` (DR-181). Callers resume by seeking after load.
+ * 
+ * TRACES: UR-070, UR-079 | DR-224, DR-226, DR-227 | UT-212
+ */
+async repositoryGetStreamSelection(handle: string, itemId: string, mediaSourceId: string | null, audioStreamIndex: number | null) : Promise<StreamSelection> {
+    return await TAURI_INVOKE("repository_get_stream_selection", { handle, itemId, mediaSourceId, audioStreamIndex });
+},
 /**
 * Get audio stream URL for a track
 */
@@ -1965,7 +2003,7 @@ export type AudioTrackSwitchResponse =
 /**
 * HTML5 needs to reload stream with new audio track
 */
-{ strategy: "reloadStream"; new_url: string; position: number }
+{ strategy: "reloadStream"; selection: StreamSelection; position: number }
 /**
 * Authentication result
 */
@@ -2329,7 +2367,18 @@ excludedItemIds?: string[] }
 * streamed; the server returns a transcoding URL (already absolute) plus a
 * `live_stream_id` that can later be used to close the stream.
 */
-export type LiveStreamInfo = { streamUrl: string; playSessionId: string | null; liveStreamId: string | null; mediaSourceId: string | null }
+export type LiveStreamInfo = { streamUrl: string; playSessionId: string | null; liveStreamId: string | null; mediaSourceId: string | null; 
+/**
+ * How to open `stream_url`.
+ * 
+ * A live channel is always an HLS transcode — the server has to repackage a
+ * broadcast mux into something a browser can play, and there is no static
+ * file to direct-play. Saying so here means the player page never has to
+ * work it out from the URL, which is the whole of DR-224.
+ * 
+ * TRACES: UR-079 | DR-224
+ */
+transport: Transport }
 /**
 * An LMS multi-room sync group, as returned by JellyLMS `/JellyLms/SyncGroups`.
 * 
@@ -2568,6 +2617,18 @@ videoCodec: string;
 * Whether the video requires server-side transcoding
 */
 needsTranscoding: boolean; 
+/**
+ * How this item's stream is fetched, as the backend decided it.
+ * 
+ * Carried on the queue item so a later seek/reload does not have to guess.
+ * `None` for items queued by a path that never negotiated (audio tracks,
+ * direct URLs) and for anything queued before this field existed, where the
+ * caller falls back to `needs_transcoding` — every transcode this app
+ * requests is HLS (DR-140), so that fallback is exact rather than a guess.
+ * 
+ * TRACES: UR-003, UR-004, UR-079 | DR-224, DR-229
+ */
+transport?: Transport | null; 
 /**
 * Optional now-playing metadata. Used by the background-audio handoff so the
 * lockscreen/miniplayer show the item (title/subtitle/artwork). Defaulted so
@@ -2680,6 +2741,32 @@ supportsNativeVideo: boolean }
 * Playback information
 */
 export type PlaybackInfo = { mediaSourceId: string; playSessionId: string; streamUrl: string; directPlay: boolean; needsTranscoding: boolean }
+/**
+ * What the server is doing to the source to produce this stream.
+ * 
+ * Distinct from [`Transport`] because the two are genuinely independent: a
+ * direct-streamed remux and a transcode can both arrive over HLS, and a direct
+ * play can arrive progressively or as a local file. Keeping them apart is what
+ * lets the UI say "this is not costing the server anything" without inferring
+ * it from a URL shape.
+ * 
+ * TRACES: UR-079 | DR-227
+ */
+export type PlaybackKind = 
+/**
+ * The source file is served untouched. No server CPU, no quality loss.
+ */
+{ type: "directPlay" } | 
+/**
+ * The container is repackaged but the codecs are copied — cheap, and
+ * visually identical to the source.
+ */
+{ type: "directStream" } | 
+/**
+ * The server is re-encoding. The only case where a bitrate ceiling can
+ * actually be honoured, and the only one that costs the server real work.
+ */
+{ type: "transcode" }
 /**
 * Playback mode - local device, remote session, or idle
 */
@@ -2779,6 +2866,18 @@ videoCodec?: string | null;
 * Whether the video requires server-side transcoding
 */
 needsTranscoding?: boolean; 
+/**
+ * How this item's stream is fetched, as the backend decided it.
+ * 
+ * Carried on the queue item so a later seek/reload does not have to guess.
+ * `None` for items queued by a path that never negotiated (audio tracks,
+ * direct URLs) and for anything queued before this field existed, where the
+ * caller falls back to `needs_transcoding` — every transcode this app
+ * requests is HLS (DR-140), so that fallback is exact rather than a guess.
+ * 
+ * TRACES: UR-003, UR-004, UR-079 | DR-224, DR-229
+ */
+transport?: Transport | null; 
 /**
 * Video width in pixels
 */
@@ -3061,6 +3160,38 @@ alreadyDownloaded: number;
 * Number of tracks skipped (no jellyfin ID or other reasons)
 */
 skipped: number }
+/**
+ * One rung of the quality picker, as it applies to *this* media source.
+ * 
+ * The picker used to be filled from the fixed [`StreamingQuality::ALL`] ladder,
+ * which meant offering "20 Mbps" for a 1.1 Mbps podcast — eight rungs, six of
+ * them indistinguishable from Original. `exceeds_source` is what lets the
+ * frontend render that honestly without knowing anything about bitrates.
+ * 
+ * TRACES: UR-070, UR-079 | DR-226, DR-121
+ */
+export type QualityOption = { quality: StreamingQuality; 
+/**
+ * Human label ("8 Mbps"). Lives in Rust beside the number it describes.
+ */
+label: string; 
+/**
+ * Secondary line ("1080p").
+ */
+detail: string; 
+/**
+ * True when this rung's ceiling is at or above what the source itself
+ * carries, so selecting it yields the same stream as `Original`.
+ * 
+ * The frontend renders these differently (or hides them); it does not
+ * decide which they are.
+ */
+exceedsSource: boolean; 
+/**
+ * The source's own bitrate, when the server reported one. Presentation
+ * only — the picker shows "Original (6.7 Mbps)" rather than a bare word.
+ */
+sourceBitrate: number | null }
 /**
 * Response for queue queries
 */
@@ -3069,6 +3200,36 @@ export type QueueStatus = { items: PlayerMediaItem[]; currentIndex: number | nul
 * Remote session status for UI updates
 */
 export type RemoteSessionStatus = { position: number; duration: number | null; isPlaying: boolean; nowPlayingItem: NowPlayingItem | null }
+/**
+ * The rendition actually negotiated — what the viewer is receiving right now.
+ * 
+ * `None` on a [`StreamSelection`] when the source is being direct-played as-is:
+ * there is no *chosen* rendition in that case, only the file itself, and
+ * reporting the ceiling that happened to be set would misdescribe it.
+ * 
+ * TRACES: UR-079 | DR-224, DR-225
+ */
+export type Rendition = { 
+/**
+ * The rung of the ladder this stream was built against.
+ */
+quality: StreamingQuality; 
+/**
+ * Total bits per second the stream may use, when a ceiling applies.
+ */
+maxBitrate: number | null; 
+/**
+ * Resolution ceiling, when one applies. `None` preserves the source's.
+ */
+maxHeight: number | null; 
+/**
+ * Video codec the server was asked to produce.
+ */
+videoCodec: string | null; 
+/**
+ * Audio codec the server was asked to produce.
+ */
+audioCodec: string | null }
 /**
 * Repeat mode for the queue
 * 
@@ -3186,9 +3347,59 @@ export type StreamQualityResponse =
 */
 { strategy: "native"; position: number } | 
 /**
- * HTML5 must reload its element with this URL.
+ * HTML5 must reload its element with this selection.
 */
-{ strategy: "reloadStream"; new_url: string; position: number }
+{ strategy: "reloadStream"; selection: StreamSelection; position: number }
+/**
+ * Everything a player backend needs to open a stream, and everything the UI
+ * needs to describe it.
+ * 
+ * Replaces the bare `String` URL that `get_video_stream_url` used to return.
+ * 
+ * TRACES: UR-079 | DR-224, DR-226, DR-227
+ */
+export type StreamSelection = { 
+/**
+ * The URL (or loopback URL) to open.
+ */
+url: string; 
+/**
+ * How to fetch it. Replaces the `.m3u8` substring check.
+ */
+transport: Transport; 
+/**
+ * What the server is doing to the source to produce it.
+ */
+playbackKind: PlaybackKind; 
+/**
+ * The negotiated rendition; `None` when direct-playing the source as-is.
+ */
+rendition: Rendition | null; 
+/**
+ * What this media source can offer, for the quality picker (DR-226).
+ */
+available: QualityOption[]; 
+/**
+ * The media source this selection is for, so a later re-open (quality
+ * change, audio-track switch, transcoded seek) targets the same one.
+ */
+mediaSourceId: string | null; 
+/**
+ * The transcode identity the server keyed this job by, when there is one.
+ */
+playSessionId: string | null; 
+/**
+ * Whether the server is spending encoder time on this stream.
+ * 
+ * Derived from [`playback_kind`](Self::playback_kind) rather than left for
+ * the frontend to compute: "which kinds count as transcoding" is a domain
+ * rule, and a direct *stream* is a remux that must not be counted. The
+ * queue's long-standing `needs_transcoding` flag and the seek strategy both
+ * read this, so there is one answer rather than three.
+ * 
+ * TRACES: UR-079 | DR-224, DR-227
+ */
+needsTranscoding: boolean }
 /**
 * A ceiling on how much bandwidth a *video* stream may consume.
 * 
@@ -3269,6 +3480,36 @@ itemName: string | null }
 * Statistics about the thumbnail cache
 */
 export type ThumbnailCacheStats = { totalSizeBytes: number; itemCount: number; limitBytes: number }
+/**
+ * How the bytes of a chosen stream are fetched.
+ * 
+ * This field exists to delete a substring search. The frontend previously
+ * decided which loader to attach by testing `url.contains(".m3u8")`, which is a
+ * domain fact reconstructed in the presentation layer — the same class of leak
+ * as the item-type taxonomy that `check:boundary` guards, and one that breaks
+ * silently the moment a server serves a playlist from a path that does not end
+ * in `.m3u8`, or serves a progressive file from one that does.
+ * 
+ * Tagged (`{"type":"hls"}`) rather than a bare string so the frontend matches a
+ * discriminant instead of comparing text.
+ * 
+ * TRACES: UR-079 | DR-224
+ */
+export type Transport = 
+/**
+ * An HLS playlist. The webview attaches hls.js (or Safari's native loader);
+ * ExoPlayer uses its HLS media source.
+ */
+{ type: "hls" } | 
+/**
+ * A single progressive HTTP resource, seekable by byte range.
+ */
+{ type: "progressive" } | 
+/**
+ * A file already on disk — a completed download, or the loopback media
+ * server standing in front of one.
+ */
+{ type: "localFile" }
 /**
 * User information
 */
@@ -3316,7 +3557,7 @@ export type VideoSeekResponse =
 /**
 * Reload stream from new position (transcoded non-HLS)
 */
-{ strategy: "reloadStream"; new_url: string; seek_offset: number }
+{ strategy: "reloadStream"; selection: StreamSelection; seek_offset: number }
 /**
 * Video playback settings
 */
@@ -3,7 +3,7 @@
 // NO direct HTTP calls - everything routes through Rust backend

 import { commands } from "./bindings";
-import type { JRayActor, DownloadDiskUsage, SearchScope } from "./bindings";
+import type { DownloadDiskUsage, JRayActor, SearchScope, StreamSelection } from "./bindings";
 import type { QualityPreset } from "./quality-presets";
 import type {
  Library,
@@ -247,6 +247,34 @@ export class RepositoryClient {
    );
  }

+  /**
+   * Decide what stream to play, and describe it.
+   *
+   * The playback counterpart to {@link getVideoStreamUrl}, which returns only a
+   * URL and therefore forces its caller to work out the rest. This returns the
+   * transport (so the player picks a loader from a tagged enum rather than by
+   * searching the URL for `.m3u8`), the playback kind (direct play / direct
+   * stream / transcode), and the quality ladder as it applies to this source.
+   *
+   * No position parameter, for the same reason as {@link getVideoStreamUrl}: a
+   * start position on an HLS playlist makes Jellyfin reject every segment behind
+   * it with `400` (DR-181). Resume by seeking once loaded.
+   *
+   * TRACES: UR-070, UR-079 | DR-224, DR-226, DR-227 | UT-212
+   */
+  async getStreamSelection(
+    itemId: string,
+    mediaSourceId?: string | null,
+    audioStreamIndex?: number | null,
+  ): Promise<StreamSelection> {
+    return commands.repositoryGetStreamSelection(
+      this.ensureHandle(),
+      itemId,
+      mediaSourceId ?? null,
+      audioStreamIndex ?? null,
+    );
+  }
+
  /**
   * Audio-only stream URL for a video item, for the background-audio handoff.
   * The server extracts just the audio track — no video is decoded on-device.