feat(playback): let Rust decide what stream to play, and say so

Playing a video meant asking the server to re-encode it, always. That decision was made nowhere and written down nowhere, so whoever needed it re-derived it downstream — the player worked out whether it had been handed a playlist by looking for ".m3u8" in the URL, in two places. A viewer paid for a transcode of a file their device could have played untouched, and the app could not tell them which it was. One negotiation now produces one self-describing StreamSelection — direct play, remux or transcode; over a playlist, a plain HTTP file, or a local one — and every renderer consumes that same answer. Measured against the development server (Jellyfin 10.11.5), 400 items sampled for codec mix and 40 put through a real PlaybackInfo negotiation per profile: Linux / WebKitGTK (h264 only, 2ch) 3/40 — 7% direct play Android / ExoPlayer (hevc, ac3/eac3, 6ch) 34/40 — 85% direct play The library is ~80% hevc, which is why the two diverge so hard. The payoff is overwhelmingly Android, where 85% of plays were starting a transcode nobody needed. Linux stays near 7% until libmpv decodes the picture — the h264-only profile is a WebKitGTK constraint, not a JellyTau choice. DR-219 StreamSelection: url + tagged Transport (hls/progressive/localFile) + PlaybackKind (directPlay/directStream/transcode) + the negotiated rendition + this source's ladder + a needs_transcoding flag derived in Rust so the rule is answered once. Both enums are serde-tagged so the frontend matches a discriminant, not a substring. The paths that never negotiate get the same shape from Rust rather than assembling one — media_local_selection for a downloaded file, LiveStreamInfo.transport for a live channel — so there is no second place where a transport is decided. DR-220 The ceiling becomes two levels: a durable device default (Settings, persisted) and a per-playback override the in-player picker sets. The picker had called itself a "this film, this connection" control since it was written but wrote the process-wide default, so dropping one awkward film to 2 Mbps silently capped every video played afterwards for the rest of the process, with Settings still showing the old value. The override is cleared whenever playback moves to a new item, which stops it surviving into an autoplayed next episode. effective_streaming_quality() is the single resolution point. DR-221 The quality picker is filled from what this media source can offer. Rust marks a rung exceeds_source when its ceiling is at or above the source's own bitrate — such a rung is another way to spell Original — and the frontend does not draw those. Original is never marked; a source whose bitrate the server does not report marks nothing, which keeps every rung offered. DR-222 Direct play and direct stream are negotiated, with two client-side overrides on top because the server's answer is right about the file and wrong about what this app will do with it: undecodable audio (Jellyfin 10.11.5 honours a DirectPlayProfile's container and video codec but ignores its audio codec, so it offers direct play for an E-AC-3 track the webview renders in silence) and a viewer-pinned audio track the file does not default to. A direct stream is a remux and is deliberately not counted as transcoding. DR-223 Dropped on measurement, not deferred. A master playlist from this server carries exactly one EXT-X-STREAM-INF: Jellyfin builds it from the single rendition the request asked for rather than publishing a ladder. So there is no adaptation for hls.js to be preserving and none mpv would lose — the claim that there was, in playback-backend-unification.md, does not hold. Recorded rather than deleted because it is a measurement: a server that does publish a ladder would change the answer. DR-224 Every backend consumes the same selection. The queue item carries the transport, so player_seek_video picks its seek strategy from the backend's decision instead of the last stream_url.contains(".m3u8") in the codebase. Items queued by a path that never negotiated carry None and fall back to needs_transcoding, which is exact rather than a guess because every transcode this app requests is HLS (DR-140). The frontend loader decision moves to streamTransport.ts so it can be tested: the two cases that pin it are the ones that failed against the old implementation — a progressive stream whose URL contains ".m3u8" must not get an HLS loader, and an HLS stream whose URL contains none must. Also verified the URL the direct-play branch builds actually serves playable bytes: 206, video/mp4, valid ISO-BMFF, and a mid-file range works, so seeking a direct play works. The spec is folded into docs/architecture/{01,02,03} and deleted, per the rule that docs/specs holds only work that has not shipped. DR-121 leaves read-through-media-cache.md with a pointer; that spec keeps its capture half. Not verified: real playback on a device. Direct play changes what actually gets played, and neither fixtures nor curl prove the WebKitGTK and ExoPlayer paths render it.
2026-08-22 13:45:03 +02:00
parent 5fede123e7
commit 109700b949
45 changed files with 9744 additions and 6581 deletions
@@ -97,6 +97,24 @@ impl HybridRepository {
            .await
    }

+    /// Decide what stream to play and describe it fully — the DR-224 contract.
+    ///
+    /// Online-only for the same reason as `get_video_stream_url`: an offline
+    /// item is a file on disk, and the caller builds
+    /// [`StreamSelection::local_file`] for it rather than negotiating anything.
+    ///
+    /// TRACES: UR-070, UR-079 | DR-224, DR-226, DR-227
+    pub async fn get_stream_selection(
+        &self,
+        item_id: &str,
+        media_source_id: Option<&str>,
+        audio_stream_index: Option<i32>,
+    ) -> Result<super::StreamSelection, RepoError> {
+        self.online
+            .get_stream_selection(item_id, media_source_id, audio_stream_index)
+            .await
+    }
+
    /// Get an audio-only stream URL for a video item (background-audio handoff).
    /// Online-only, like `get_video_stream_url`.
    ///
@@ -5,11 +5,14 @@ pub mod hybrid;
 pub mod offline;
 pub mod online;
 pub mod series_progress;
+/// Backend-owned stream selection (UR-079 / DR-224).
+pub mod stream_selection;
 pub mod types;

 pub use hybrid::HybridRepository;
 pub use offline::OfflineRepository;
 pub use online::{JRayActor, OnlineRepository};
+pub use stream_selection::{StreamSelection, Transport};
 pub use types::*;

 use async_trait::async_trait;
@@ -0,0 +1,416 @@
+//! What stream to play, decided in Rust and handed to a player whole.
+//!
+//! Every player backend — mpv, ExoPlayer, the webview `<video>`/hls.js path —
+//! used to receive a bare URL and re-derive the rest: the frontend decided
+//! "is this HLS?" by looking for `.m3u8` in the string, and nothing anywhere
+//! carried *why* a stream was transcoded or what else the source could have
+//! offered. This module is the replacement contract: one self-describing
+//! [`StreamSelection`] that says what the stream is, how to fetch it, and what
+//! the alternatives were.
+//!
+//! The division of labour it encodes — **Rust decides *what stream*, the player
+//! decides *how to deliver it*** — is the point. A multi-variant playlist handed
+//! to ExoPlayer is still ExoPlayer's to adapt over; Rust never paces bytes.
+//!
+//! TRACES: UR-079 | DR-224
+
+use serde::{Deserialize, Serialize};
+
+use crate::settings::StreamingQuality;
+
+/// How the bytes of a chosen stream are fetched.
+///
+/// This field exists to delete a substring search. The frontend previously
+/// decided which loader to attach by testing `url.contains(".m3u8")`, which is a
+/// domain fact reconstructed in the presentation layer — the same class of leak
+/// as the item-type taxonomy that `check:boundary` guards, and one that breaks
+/// silently the moment a server serves a playlist from a path that does not end
+/// in `.m3u8`, or serves a progressive file from one that does.
+///
+/// Tagged (`{"type":"hls"}`) rather than a bare string so the frontend matches a
+/// discriminant instead of comparing text.
+///
+/// TRACES: UR-079 | DR-224
+#[derive(specta::Type, Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(tag = "type", rename_all = "camelCase")]
+pub enum Transport {
+    /// An HLS playlist. The webview attaches hls.js (or Safari's native loader);
+    /// ExoPlayer uses its HLS media source.
+    Hls,
+    /// A single progressive HTTP resource, seekable by byte range.
+    Progressive,
+    /// A file already on disk — a completed download, or the loopback media
+    /// server standing in front of one.
+    LocalFile,
+}
+
+/// What the server is doing to the source to produce this stream.
+///
+/// Distinct from [`Transport`] because the two are genuinely independent: a
+/// direct-streamed remux and a transcode can both arrive over HLS, and a direct
+/// play can arrive progressively or as a local file. Keeping them apart is what
+/// lets the UI say "this is not costing the server anything" without inferring
+/// it from a URL shape.
+///
+/// TRACES: UR-079 | DR-227
+#[derive(specta::Type, Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(tag = "type", rename_all = "camelCase")]
+pub enum PlaybackKind {
+    /// The source file is served untouched. No server CPU, no quality loss.
+    DirectPlay,
+    /// The container is repackaged but the codecs are copied — cheap, and
+    /// visually identical to the source.
+    DirectStream,
+    /// The server is re-encoding. The only case where a bitrate ceiling can
+    /// actually be honoured, and the only one that costs the server real work.
+    Transcode,
+}
+
+impl PlaybackKind {
+    /// Whether the server is spending encoder time on this stream.
+    ///
+    /// The queue carries a `needs_transcoding` flag that predates this enum and
+    /// that several seek/reload paths still branch on; this keeps the two from
+    /// drifting by making one derive from the other.
+    ///
+    /// TRACES: UR-079 | DR-227
+    pub fn needs_transcoding(&self) -> bool {
+        matches!(self, PlaybackKind::Transcode)
+    }
+}
+
+/// The rendition actually negotiated — what the viewer is receiving right now.
+///
+/// `None` on a [`StreamSelection`] when the source is being direct-played as-is:
+/// there is no *chosen* rendition in that case, only the file itself, and
+/// reporting the ceiling that happened to be set would misdescribe it.
+///
+/// TRACES: UR-079 | DR-224, DR-225
+#[derive(specta::Type, Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub struct Rendition {
+    /// The rung of the ladder this stream was built against.
+    pub quality: StreamingQuality,
+    /// Total bits per second the stream may use, when a ceiling applies.
+    pub max_bitrate: Option<u64>,
+    /// Resolution ceiling, when one applies. `None` preserves the source's.
+    pub max_height: Option<u32>,
+    /// Video codec the server was asked to produce.
+    pub video_codec: Option<String>,
+    /// Audio codec the server was asked to produce.
+    pub audio_codec: Option<String>,
+}
+
+/// One rung of the quality picker, as it applies to *this* media source.
+///
+/// The picker used to be filled from the fixed [`StreamingQuality::ALL`] ladder,
+/// which meant offering "20 Mbps" for a 1.1 Mbps podcast — eight rungs, six of
+/// them indistinguishable from Original. `exceeds_source` is what lets the
+/// frontend render that honestly without knowing anything about bitrates.
+///
+/// TRACES: UR-070, UR-079 | DR-226, DR-121
+#[derive(specta::Type, Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub struct QualityOption {
+    pub quality: StreamingQuality,
+    /// Human label ("8 Mbps"). Lives in Rust beside the number it describes.
+    pub label: String,
+    /// Secondary line ("1080p").
+    pub detail: String,
+    /// True when this rung's ceiling is at or above what the source itself
+    /// carries, so selecting it yields the same stream as `Original`.
+    ///
+    /// The frontend renders these differently (or hides them); it does not
+    /// decide which they are.
+    pub exceeds_source: bool,
+    /// The source's own bitrate, when the server reported one. Presentation
+    /// only — the picker shows "Original (6.7 Mbps)" rather than a bare word.
+    pub source_bitrate: Option<u64>,
+}
+
+/// Everything a player backend needs to open a stream, and everything the UI
+/// needs to describe it.
+///
+/// Replaces the bare `String` URL that `get_video_stream_url` used to return.
+///
+/// TRACES: UR-079 | DR-224, DR-226, DR-227
+#[derive(specta::Type, Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub struct StreamSelection {
+    /// The URL (or loopback URL) to open.
+    pub url: String,
+    /// How to fetch it. Replaces the `.m3u8` substring check.
+    pub transport: Transport,
+    /// What the server is doing to the source to produce it.
+    pub playback_kind: PlaybackKind,
+    /// The negotiated rendition; `None` when direct-playing the source as-is.
+    pub rendition: Option<Rendition>,
+    /// What this media source can offer, for the quality picker (DR-226).
+    pub available: Vec<QualityOption>,
+    /// The media source this selection is for, so a later re-open (quality
+    /// change, audio-track switch, transcoded seek) targets the same one.
+    pub media_source_id: Option<String>,
+    /// The transcode identity the server keyed this job by, when there is one.
+    pub play_session_id: Option<String>,
+    /// Whether the server is spending encoder time on this stream.
+    ///
+    /// Derived from [`playback_kind`](Self::playback_kind) rather than left for
+    /// the frontend to compute: "which kinds count as transcoding" is a domain
+    /// rule, and a direct *stream* is a remux that must not be counted. The
+    /// queue's long-standing `needs_transcoding` flag and the seek strategy both
+    /// read this, so there is one answer rather than three.
+    ///
+    /// TRACES: UR-079 | DR-224, DR-227
+    pub needs_transcoding: bool,
+}
+
+impl StreamSelection {
+    /// A selection for a file already on disk.
+    ///
+    /// A downloaded file is a direct play by definition — the bytes are the
+    /// source's — and offering a quality ladder over it would be a lie, since
+    /// nothing about a local file can be re-negotiated.
+    ///
+    /// TRACES: UR-071, UR-079 | DR-224
+    pub fn local_file(url: impl Into<String>) -> Self {
+        Self {
+            url: url.into(),
+            transport: Transport::LocalFile,
+            playback_kind: PlaybackKind::DirectPlay,
+            rendition: None,
+            available: Vec::new(),
+            media_source_id: None,
+            play_session_id: None,
+            needs_transcoding: false,
+        }
+    }
+}
+
+/// Build the quality ladder as it applies to a source of a known bitrate.
+///
+/// Every rung is returned — the picker stays a fixed, predictable list rather
+/// than one that changes length per item — but each is marked with whether it
+/// would actually constrain *this* source. A rung whose ceiling is at or above
+/// the source bitrate produces the same bytes as `Original`, so presenting it as
+/// a distinct choice is noise.
+///
+/// `source_bitrate` is `None` when the server did not report one (it is absent
+/// for some containers — the sampled library has `avi` files with no bitrate at
+/// all). In that case nothing can be judged redundant and every rung is offered,
+/// which is the safe direction: the viewer keeps every choice they had before.
+///
+/// TRACES: UR-070, UR-079 | DR-226, DR-121 | UT-211
+pub fn quality_options_for_source(source_bitrate: Option<u64>) -> Vec<QualityOption> {
+    StreamingQuality::ALL
+        .iter()
+        .map(|quality| QualityOption {
+            quality: *quality,
+            label: quality.label().to_string(),
+            detail: quality.detail().to_string(),
+            exceeds_source: match (quality.max_bitrate(), source_bitrate) {
+                // `Original` is the source; it never "exceeds" it.
+                (None, _) => false,
+                // Nothing known about the source — judge nothing redundant.
+                (Some(_), None) => false,
+                (Some(cap), Some(source)) => cap >= source,
+            },
+            source_bitrate,
+        })
+        .collect()
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    /// The tag the frontend matches on has to be exactly what it expects, and
+    /// it is a *string in TypeScript* — nothing but a test keeps the two in step.
+    ///
+    /// TRACES: UR-079 | DR-224 | UT-211
+    #[test]
+    fn test_transport_serialises_with_the_tag_the_frontend_matches() {
+        let cases = [
+            (Transport::Hls, r#"{"type":"hls"}"#),
+            (Transport::Progressive, r#"{"type":"progressive"}"#),
+            (Transport::LocalFile, r#"{"type":"localFile"}"#),
+        ];
+        for (transport, expected) in cases {
+            let json = serde_json::to_string(&transport).expect("serialises");
+            assert_eq!(json, expected, "wire shape of {transport:?}");
+            let back: Transport = serde_json::from_str(&json).expect("round-trips");
+            assert_eq!(back, transport);
+        }
+    }
+
+    /// TRACES: UR-079 | DR-227 | UT-211
+    #[test]
+    fn test_playback_kind_serialises_with_the_tag_the_frontend_matches() {
+        let cases = [
+            (PlaybackKind::DirectPlay, r#"{"type":"directPlay"}"#),
+            (PlaybackKind::DirectStream, r#"{"type":"directStream"}"#),
+            (PlaybackKind::Transcode, r#"{"type":"transcode"}"#),
+        ];
+        for (kind, expected) in cases {
+            let json = serde_json::to_string(&kind).expect("serialises");
+            assert_eq!(json, expected, "wire shape of {kind:?}");
+            let back: PlaybackKind = serde_json::from_str(&json).expect("round-trips");
+            assert_eq!(back, kind);
+        }
+    }
+
+    /// Only a transcode costs the server encoder time. A direct *stream* is a
+    /// remux — cheap, and not what `needs_transcoding` has ever meant.
+    ///
+    /// TRACES: UR-079 | DR-227 | UT-211
+    #[test]
+    fn test_only_transcode_counts_as_transcoding() {
+        assert!(PlaybackKind::Transcode.needs_transcoding());
+        assert!(!PlaybackKind::DirectStream.needs_transcoding());
+        assert!(!PlaybackKind::DirectPlay.needs_transcoding());
+    }
+
+    /// A local file is a direct play over a local transport, with no ladder:
+    /// nothing about a file on disk can be re-negotiated.
+    ///
+    /// TRACES: UR-071, UR-079 | DR-224 | UT-211
+    #[test]
+    fn test_local_file_selection_offers_no_ladder() {
+        let selection = StreamSelection::local_file("http://127.0.0.1:9000/media/x.mkv");
+        assert_eq!(selection.transport, Transport::LocalFile);
+        assert_eq!(selection.playback_kind, PlaybackKind::DirectPlay);
+        assert!(selection.rendition.is_none());
+        assert!(selection.available.is_empty());
+        assert!(!selection.needs_transcoding);
+    }
+
+    /// The camelCase rule applies to nested struct fields too, and
+    /// `playbackKind` is the one the frontend branches on.
+    ///
+    /// TRACES: UR-079 | DR-224 | UT-211
+    #[test]
+    fn test_stream_selection_fields_are_camel_case_on_the_wire() {
+        let selection = StreamSelection {
+            url: "https://example/master.m3u8".to_string(),
+            transport: Transport::Hls,
+            playback_kind: PlaybackKind::Transcode,
+            rendition: Some(Rendition {
+                quality: StreamingQuality::Mbps8,
+                max_bitrate: Some(8_000_000),
+                max_height: Some(1080),
+                video_codec: Some("h264".to_string()),
+                audio_codec: Some("aac".to_string()),
+            }),
+            available: Vec::new(),
+            media_source_id: Some("src-1".to_string()),
+            play_session_id: Some("sess-1".to_string()),
+            needs_transcoding: true,
+        };
+        let json = serde_json::to_string(&selection).expect("serialises");
+        assert!(
+            json.contains(r#""playbackKind":{"type":"transcode"}"#),
+            "{json}"
+        );
+        assert!(json.contains(r#""transport":{"type":"hls"}"#), "{json}");
+        assert!(json.contains(r#""mediaSourceId":"src-1""#), "{json}");
+        assert!(json.contains(r#""playSessionId":"sess-1""#), "{json}");
+        assert!(json.contains(r#""maxBitrate":8000000"#), "{json}");
+        assert!(json.contains(r#""maxHeight":1080"#), "{json}");
+        assert!(json.contains(r#""needsTranscoding":true"#), "{json}");
+    }
+
+    /// The measured library has 1.1 Mbps sources in it. Offering those a choice
+    /// of 20, 10, 8, 4 and 2 Mbps is offering five ways to spell "Original".
+    ///
+    /// TRACES: UR-070, UR-079 | DR-226, DR-121 | UT-211
+    #[test]
+    fn test_rungs_above_the_source_bitrate_are_marked_redundant() {
+        let options = quality_options_for_source(Some(1_122_137));
+        let redundant: Vec<_> = options
+            .iter()
+            .filter(|o| o.exceeds_source)
+            .map(|o| o.quality)
+            .collect();
+        assert_eq!(
+            redundant,
+            vec![
+                StreamingQuality::Mbps20,
+                StreamingQuality::Mbps10,
+                StreamingQuality::Mbps8,
+                StreamingQuality::Mbps4,
+                StreamingQuality::Mbps2,
+            ],
+            "every rung at or above a 1.12 Mbps source is the source"
+        );
+
+        // The rungs that genuinely constrain it are not marked.
+        let constraining: Vec<_> = options
+            .iter()
+            .filter(|o| !o.exceeds_source)
+            .map(|o| o.quality)
+            .collect();
+        assert_eq!(
+            constraining,
+            vec![
+                StreamingQuality::Original,
+                StreamingQuality::Mbps1,
+                StreamingQuality::Kbps720,
+            ]
+        );
+    }
+
+    /// `Original` is the source, so it is never "above" it — not even for a
+    /// source whose bitrate is unknown or zero.
+    ///
+    /// TRACES: UR-070, UR-079 | DR-226 | UT-211
+    #[test]
+    fn test_original_is_never_marked_as_exceeding_the_source() {
+        for bitrate in [None, Some(0), Some(1), Some(50_000_000)] {
+            let options = quality_options_for_source(bitrate);
+            let original = options
+                .iter()
+                .find(|o| o.quality == StreamingQuality::Original)
+                .expect("Original is always offered");
+            assert!(!original.exceeds_source, "bitrate {bitrate:?}");
+        }
+    }
+
+    /// An `avi` with no reported bitrate must not lose the picker. Judging
+    /// nothing redundant is the safe direction — the viewer keeps every choice.
+    ///
+    /// TRACES: UR-070, UR-079 | DR-226 | UT-211
+    #[test]
+    fn test_an_unknown_source_bitrate_keeps_every_rung_offered() {
+        let options = quality_options_for_source(None);
+        assert_eq!(options.len(), StreamingQuality::ALL.len());
+        assert!(
+            options.iter().all(|o| !o.exceeds_source),
+            "nothing can be judged redundant without a source bitrate"
+        );
+        assert!(options.iter().all(|o| o.source_bitrate.is_none()));
+    }
+
+    /// A 4K remux constrains at every rung — the ladder is fully meaningful.
+    ///
+    /// TRACES: UR-070, UR-079 | DR-226 | UT-211
+    #[test]
+    fn test_a_source_above_the_ladder_marks_nothing_redundant() {
+        let options = quality_options_for_source(Some(40_000_000));
+        assert!(options.iter().all(|o| !o.exceeds_source));
+    }
+
+    /// The picker's text comes from Rust, beside the numbers it describes, so a
+    /// relabelled rung cannot drift out of step with what it does.
+    ///
+    /// TRACES: UR-070, UR-079 | DR-226 | UT-211
+    #[test]
+    fn test_options_carry_the_ladder_labels() {
+        let options = quality_options_for_source(Some(6_652_961));
+        assert_eq!(options.len(), StreamingQuality::ALL.len());
+        for (option, quality) in options.iter().zip(StreamingQuality::ALL) {
+            assert_eq!(option.quality, quality);
+            assert_eq!(option.label, quality.label());
+            assert_eq!(option.detail, quality.detail());
+            assert_eq!(option.source_bitrate, Some(6_652_961));
+        }
+    }
+}
@@ -469,6 +469,15 @@ pub struct LiveStreamInfo {
    pub play_session_id: Option<String>,
    pub live_stream_id: Option<String>,
    pub media_source_id: Option<String>,
+    /// How to open `stream_url`.
+    ///
+    /// A live channel is always an HLS transcode — the server has to repackage a
+    /// broadcast mux into something a browser can play, and there is no static
+    /// file to direct-play. Saying so here means the player page never has to
+    /// work it out from the URL, which is the whole of DR-224.
+    ///
+    /// TRACES: UR-079 | DR-224
+    pub transport: super::stream_selection::Transport,
 }

 /// Genre