Files
jellytau/src-tauri/src/player/media.rs
T
dtourolle 109700b949 feat(playback): let Rust decide what stream to play, and say so
Playing a video meant asking the server to re-encode it, always. That
decision was made nowhere and written down nowhere, so whoever needed it
re-derived it downstream — the player worked out whether it had been handed
a playlist by looking for ".m3u8" in the URL, in two places. A viewer paid
for a transcode of a file their device could have played untouched, and the
app could not tell them which it was.

One negotiation now produces one self-describing StreamSelection — direct
play, remux or transcode; over a playlist, a plain HTTP file, or a local one
— and every renderer consumes that same answer.

Measured against the development server (Jellyfin 10.11.5), 400 items
sampled for codec mix and 40 put through a real PlaybackInfo negotiation
per profile:

  Linux / WebKitGTK (h264 only, 2ch)          3/40 —  7% direct play
  Android / ExoPlayer (hevc, ac3/eac3, 6ch)  34/40 — 85% direct play

The library is ~80% hevc, which is why the two diverge so hard. The payoff
is overwhelmingly Android, where 85% of plays were starting a transcode
nobody needed. Linux stays near 7% until libmpv decodes the picture — the
h264-only profile is a WebKitGTK constraint, not a JellyTau choice.

DR-219  StreamSelection: url + tagged Transport (hls/progressive/localFile)
        + PlaybackKind (directPlay/directStream/transcode) + the negotiated
        rendition + this source's ladder + a needs_transcoding flag derived
        in Rust so the rule is answered once. Both enums are serde-tagged
        so the frontend matches a discriminant, not a substring. The paths
        that never negotiate get the same shape from Rust rather than
        assembling one — media_local_selection for a downloaded file,
        LiveStreamInfo.transport for a live channel — so there is no second
        place where a transport is decided.

DR-220  The ceiling becomes two levels: a durable device default (Settings,
        persisted) and a per-playback override the in-player picker sets.
        The picker had called itself a "this film, this connection" control
        since it was written but wrote the process-wide default, so dropping
        one awkward film to 2 Mbps silently capped every video played
        afterwards for the rest of the process, with Settings still showing
        the old value. The override is cleared whenever playback moves to a
        new item, which stops it surviving into an autoplayed next episode.
        effective_streaming_quality() is the single resolution point.

DR-221  The quality picker is filled from what this media source can offer.
        Rust marks a rung exceeds_source when its ceiling is at or above the
        source's own bitrate — such a rung is another way to spell Original
        — and the frontend does not draw those. Original is never marked; a
        source whose bitrate the server does not report marks nothing, which
        keeps every rung offered.

DR-222  Direct play and direct stream are negotiated, with two client-side
        overrides on top because the server's answer is right about the file
        and wrong about what this app will do with it: undecodable audio
        (Jellyfin 10.11.5 honours a DirectPlayProfile's container and video
        codec but ignores its audio codec, so it offers direct play for an
        E-AC-3 track the webview renders in silence) and a viewer-pinned
        audio track the file does not default to. A direct stream is a remux
        and is deliberately not counted as transcoding.

DR-223  Dropped on measurement, not deferred. A master playlist from this
        server carries exactly one EXT-X-STREAM-INF: Jellyfin builds it from
        the single rendition the request asked for rather than publishing a
        ladder. So there is no adaptation for hls.js to be preserving and
        none mpv would lose — the claim that there was, in
        playback-backend-unification.md, does not hold. Recorded rather than
        deleted because it is a measurement: a server that does publish a
        ladder would change the answer.

DR-224  Every backend consumes the same selection. The queue item carries
        the transport, so player_seek_video picks its seek strategy from the
        backend's decision instead of the last stream_url.contains(".m3u8")
        in the codebase. Items queued by a path that never negotiated carry
        None and fall back to needs_transcoding, which is exact rather than
        a guess because every transcode this app requests is HLS (DR-140).

The frontend loader decision moves to streamTransport.ts so it can be
tested: the two cases that pin it are the ones that failed against the old
implementation — a progressive stream whose URL contains ".m3u8" must not
get an HLS loader, and an HLS stream whose URL contains none must.

Also verified the URL the direct-play branch builds actually serves playable
bytes: 206, video/mp4, valid ISO-BMFF, and a mid-file range works, so
seeking a direct play works.

The spec is folded into docs/architecture/{01,02,03} and deleted, per the
rule that docs/specs holds only work that has not shipped. DR-121 leaves
read-through-media-cache.md with a pointer; that spec keeps its capture half.

Not verified: real playback on a device. Direct play changes what actually
gets played, and neither fixtures nor curl prove the WebKitGTK and ExoPlayer
paths render it.
2026-08-22 13:45:03 +02:00

605 lines
19 KiB
Rust

use serde::{Deserialize, Serialize};
use std::path::PathBuf;
/// Context for the current queue - where did the queue items come from?
/// This is used for remote playback transfer to send album/playlist context.
#[derive(specta::Type, Debug, Clone, Serialize, Deserialize, PartialEq, Default)]
#[serde(tag = "type", rename_all = "lowercase")]
pub enum QueueContext {
/// Playing from a specific album
Album {
album_id: String,
album_name: String,
},
/// Playing from a specific playlist
Playlist {
playlist_id: String,
playlist_name: String,
},
/// Custom queue (search results, manual queue, etc.)
/// Will create a temporary playlist on remote transfer
#[default]
Custom,
}
/// Represents a subtitle track
///
/// 🔴 **Do not add `#[serde(rename_all = "camelCase")]` here.** This is the one
/// struct in the player that deliberately keeps snake_case on the wire, because
/// the *same* serialization feeds two consumers that both spell `mime_type`:
///
/// * the JNI boundary — `player/android/mod.rs` serializes `MediaItem::subtitles`
/// with `serde_json` and hands the string to `JellyTauPlayer.loadWithMetadata`,
/// whose parser reads `url`, `language`, `label` and `optString("mime_type")`;
/// * the IPC boundary — `PlayItemRequest::subtitles` deserializes this same type
/// from the frontend, and the generated binding (`SubtitleTrack` in
/// `bindings.ts`) therefore also declares `mime_type`.
///
/// Renaming would not break the build and would not fail the IPC: Kotlin's
/// `optString` would just fall back to its default MIME type for every track, so
/// the failure would be silent. UT-146 asserts the serialized keys.
///
/// TRACES: UR-020 | IR-016, JA-008 | UT-146
#[derive(specta::Type, Debug, Clone, Serialize, Deserialize, PartialEq)]
pub struct SubtitleTrack {
/// Stream index in the media source
pub index: i32,
/// Subtitle URL
pub url: String,
/// Language code (e.g., "eng", "spa")
pub language: Option<String>,
/// Display title
pub label: Option<String>,
/// MIME type (e.g., "text/vtt", "application/x-subrip").
/// Snake_case on purpose — see the note on the struct.
pub mime_type: String,
}
/// Represents a media item that can be played
///
/// TRACES: UR-003, UR-004 | DR-002
#[derive(specta::Type, Debug, Clone, Serialize, Deserialize, PartialEq)]
#[serde(rename_all = "camelCase")]
#[specta(rename = "PlayerMediaItem")]
pub struct MediaItem {
/// Unique identifier
pub id: String,
/// Display title
pub title: String,
/// Name (alias for title - for frontend compatibility)
#[serde(default, skip_serializing_if = "Option::is_none")]
pub name: Option<String>,
/// Artist name(s) for audio
pub artist: Option<String>,
/// Album name for audio
pub album: Option<String>,
/// Album name (alias - for frontend compatibility)
#[serde(default, skip_serializing_if = "Option::is_none")]
pub album_name: Option<String>,
/// Album ID (Jellyfin ID) for remote transfer context
#[serde(default)]
pub album_id: Option<String>,
/// Artist items with IDs for clickable links
#[serde(default, skip_serializing_if = "Option::is_none")]
pub artist_items: Option<Vec<crate::repository::types::ArtistItem>>,
/// Artists as array of strings (fallback when artist_items not available)
#[serde(default, skip_serializing_if = "Option::is_none")]
pub artists: Option<Vec<String>>,
/// Primary image tag for artwork.
///
/// Legacy Jellyfin name; being replaced by `image_id` (same value). Dual-carried
/// while the frontend migrates (docs/specs/frontend-domain-model.md).
#[serde(default, skip_serializing_if = "Option::is_none")]
pub primary_image_tag: Option<String>,
/// Neutral image identifier the frontend resolves to a URL — replaces
/// `primary_image_tag`.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub image_id: Option<String>,
/// Item type (Audio, Movie, Episode, etc.)
#[serde(rename = "type", default, skip_serializing_if = "Option::is_none")]
pub item_type: Option<String>,
/// Playlist ID (Jellyfin ID) for remote transfer context
#[serde(default)]
pub playlist_id: Option<String>,
/// Duration in seconds
pub duration: Option<f64>,
/// URL or path to artwork image
pub artwork_url: Option<String>,
/// Type of media
pub media_type: MediaType,
/// Source of the media
pub source: MediaSource,
/// Video codec (e.g., "h264", "hevc") for video media
#[serde(default)]
pub video_codec: Option<String>,
/// Whether the video requires server-side transcoding
#[serde(default)]
pub needs_transcoding: bool,
/// How this item's stream is fetched, as the backend decided it.
///
/// Carried on the queue item so a later seek/reload does not have to guess.
/// `None` for items queued by a path that never negotiated (audio tracks,
/// direct URLs) and for anything queued before this field existed, where the
/// caller falls back to `needs_transcoding` — every transcode this app
/// requests is HLS (DR-140), so that fallback is exact rather than a guess.
///
/// TRACES: UR-003, UR-004, UR-079 | DR-224, DR-229
#[serde(default)]
pub transport: Option<crate::repository::Transport>,
/// Video width in pixels
#[serde(default)]
pub video_width: Option<u32>,
/// Video height in pixels
#[serde(default)]
pub video_height: Option<u32>,
/// Available subtitle tracks
#[serde(default)]
pub subtitles: Vec<SubtitleTrack>,
/// Series ID (for TV show episodes) - used for series audio preferences
#[serde(default)]
pub series_id: Option<String>,
/// Server ID - used for series audio preferences
#[serde(default)]
pub server_id: Option<String>,
}
#[derive(specta::Type, Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "lowercase")]
pub enum MediaType {
Audio,
Video,
}
/// TRACES: UR-002, UR-003, UR-004, UR-011 | DR-003
#[derive(specta::Type, Debug, Clone, Serialize, Deserialize, PartialEq)]
#[serde(tag = "type", rename_all = "lowercase")]
#[specta(rename = "PlayerMediaSource")]
pub enum MediaSource {
/// Streaming from Jellyfin server
Remote {
stream_url: String,
jellyfin_item_id: String,
},
/// Downloaded/cached locally
Local {
file_path: PathBuf,
/// Original Jellyfin ID for sync-back
jellyfin_item_id: Option<String>,
},
/// Direct URL (e.g., channel plugins)
DirectUrl { url: String },
}
impl MediaItem {
/// Get the Jellyfin item ID if available
pub fn jellyfin_id(&self) -> Option<&str> {
match &self.source {
MediaSource::Remote {
jellyfin_item_id, ..
} => Some(jellyfin_item_id),
MediaSource::Local {
jellyfin_item_id, ..
} => jellyfin_item_id.as_deref(),
MediaSource::DirectUrl { .. } => None,
}
}
/// Get the playback URL or file path
///
/// Only available on Android where ExoPlayer needs direct URL access
#[cfg(target_os = "android")]
pub fn playback_url(&self) -> String {
match &self.source {
MediaSource::Remote { stream_url, .. } => stream_url.clone(),
MediaSource::Local { file_path, .. } => file_path.to_string_lossy().to_string(),
MediaSource::DirectUrl { url } => url.clone(),
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::path::PathBuf;
#[test]
fn test_queue_context_album() {
let context = QueueContext::Album {
album_id: "album-123".to_string(),
album_name: "Test Album".to_string(),
};
assert!(matches!(context, QueueContext::Album { .. }));
}
#[test]
fn test_queue_context_playlist() {
let context = QueueContext::Playlist {
playlist_id: "playlist-456".to_string(),
playlist_name: "Test Playlist".to_string(),
};
assert!(matches!(context, QueueContext::Playlist { .. }));
}
#[test]
fn test_queue_context_custom() {
let context = QueueContext::Custom;
assert!(matches!(context, QueueContext::Custom));
}
#[test]
fn test_queue_context_serialization() {
let context = QueueContext::Album {
album_id: "alb-001".to_string(),
album_name: "Album 001".to_string(),
};
let json = serde_json::to_string(&context);
assert!(json.is_ok());
let serialized = json.unwrap();
assert!(serialized.contains("album"));
}
#[test]
fn test_queue_context_default() {
let context = QueueContext::default();
assert!(matches!(context, QueueContext::Custom));
}
#[test]
fn test_queue_context_clone() {
let context = QueueContext::Album {
album_id: "clone-alb".to_string(),
album_name: "Clone Album".to_string(),
};
let cloned = context.clone();
assert_eq!(context, cloned);
}
#[test]
fn test_subtitle_track_creation() {
let track = SubtitleTrack {
index: 0,
url: "https://example.com/subs.vtt".to_string(),
language: Some("eng".to_string()),
label: Some("English".to_string()),
mime_type: "text/vtt".to_string(),
};
assert_eq!(track.index, 0);
assert_eq!(track.language, Some("eng".to_string()));
}
#[test]
fn test_subtitle_track_without_language() {
let track = SubtitleTrack {
index: 1,
url: "https://example.com/subs.srt".to_string(),
language: None,
label: Some("Subtitles".to_string()),
mime_type: "application/x-subrip".to_string(),
};
assert!(track.language.is_none());
assert!(track.label.is_some());
}
#[test]
fn test_subtitle_track_serialization() {
let track = SubtitleTrack {
index: 0,
url: "url.vtt".to_string(),
language: Some("eng".to_string()),
label: None,
mime_type: "text/vtt".to_string(),
};
let json = serde_json::to_string(&track);
assert!(json.is_ok());
let serialized = json.unwrap();
assert!(serialized.contains("0"));
assert!(serialized.contains("eng"));
}
#[test]
fn test_media_type_audio() {
let media_type = MediaType::Audio;
let json = serde_json::to_string(&media_type).unwrap();
assert!(json.contains("audio"));
}
#[test]
fn test_media_type_video() {
let media_type = MediaType::Video;
let json = serde_json::to_string(&media_type).unwrap();
assert!(json.contains("video"));
}
#[test]
fn test_media_source_remote() {
let source = MediaSource::Remote {
stream_url: "https://server.com/video.mp4".to_string(),
jellyfin_item_id: "item-123".to_string(),
};
assert!(matches!(source, MediaSource::Remote { .. }));
}
#[test]
fn test_media_source_local() {
let source = MediaSource::Local {
file_path: PathBuf::from("/path/to/video.mp4"),
jellyfin_item_id: Some("item-456".to_string()),
};
assert!(matches!(source, MediaSource::Local { .. }));
}
#[test]
fn test_media_source_local_without_jellyfin_id() {
let source = MediaSource::Local {
file_path: PathBuf::from("/downloads/audio.mp3"),
jellyfin_item_id: None,
};
assert!(matches!(source, MediaSource::Local { .. }));
}
#[test]
fn test_media_source_direct_url() {
let source = MediaSource::DirectUrl {
url: "https://external.com/stream".to_string(),
};
assert!(matches!(source, MediaSource::DirectUrl { .. }));
}
#[test]
fn test_media_source_serialization() {
let source = MediaSource::DirectUrl {
url: "https://example.com/stream".to_string(),
};
let json = serde_json::to_string(&source);
assert!(json.is_ok());
let serialized = json.unwrap();
assert!(serialized.contains("directurl"));
}
#[test]
fn test_media_item_creation_minimal() {
let item = MediaItem {
// Audio and direct-URL items never negotiate a transport.
transport: None,
id: "item-1".to_string(),
title: "Test Item".to_string(),
name: None,
artist: None,
album: None,
album_name: None,
album_id: None,
artist_items: None,
artists: None,
primary_image_tag: None,
image_id: None,
item_type: None,
playlist_id: None,
duration: None,
artwork_url: None,
media_type: MediaType::Video,
source: MediaSource::DirectUrl {
url: "https://example.com/video".to_string(),
},
video_codec: None,
needs_transcoding: false,
video_width: None,
video_height: None,
subtitles: vec![],
series_id: None,
server_id: None,
};
assert_eq!(item.id, "item-1");
assert_eq!(item.title, "Test Item");
assert!(!item.needs_transcoding);
}
#[test]
fn test_media_item_jellyfin_id() {
let item = MediaItem {
// Audio and direct-URL items never negotiate a transport.
transport: None,
id: "item-2".to_string(),
title: "Test".to_string(),
name: None,
artist: None,
album: None,
album_name: None,
album_id: None,
artist_items: None,
artists: None,
primary_image_tag: None,
image_id: None,
item_type: None,
playlist_id: None,
duration: None,
artwork_url: None,
media_type: MediaType::Audio,
source: MediaSource::Remote {
stream_url: "https://server/stream".to_string(),
jellyfin_item_id: "jf-id-123".to_string(),
},
video_codec: None,
needs_transcoding: false,
video_width: None,
video_height: None,
subtitles: vec![],
series_id: None,
server_id: None,
};
assert_eq!(item.jellyfin_id(), Some("jf-id-123"));
}
#[test]
fn test_media_item_jellyfin_id_local() {
let item = MediaItem {
// Audio and direct-URL items never negotiate a transport.
transport: None,
id: "item-3".to_string(),
title: "Local".to_string(),
name: None,
artist: None,
album: None,
album_name: None,
album_id: None,
artist_items: None,
artists: None,
primary_image_tag: None,
image_id: None,
item_type: None,
playlist_id: None,
duration: None,
artwork_url: None,
media_type: MediaType::Video,
source: MediaSource::Local {
file_path: PathBuf::from("/local/video.mp4"),
jellyfin_item_id: Some("jf-local".to_string()),
},
video_codec: None,
needs_transcoding: false,
video_width: None,
video_height: None,
subtitles: vec![],
series_id: None,
server_id: None,
};
assert_eq!(item.jellyfin_id(), Some("jf-local"));
}
#[test]
fn test_media_item_jellyfin_id_direct_url() {
let item = MediaItem {
// Audio and direct-URL items never negotiate a transport.
transport: None,
id: "item-4".to_string(),
title: "Direct".to_string(),
name: None,
artist: None,
album: None,
album_name: None,
album_id: None,
artist_items: None,
artists: None,
primary_image_tag: None,
image_id: None,
item_type: None,
playlist_id: None,
duration: None,
artwork_url: None,
media_type: MediaType::Video,
source: MediaSource::DirectUrl {
url: "https://external.com/media".to_string(),
},
video_codec: None,
needs_transcoding: false,
video_width: None,
video_height: None,
subtitles: vec![],
series_id: None,
server_id: None,
};
assert_eq!(item.jellyfin_id(), None);
}
#[test]
fn test_media_item_with_subtitles() {
let sub = SubtitleTrack {
index: 0,
url: "subs.vtt".to_string(),
language: Some("eng".to_string()),
label: Some("English".to_string()),
mime_type: "text/vtt".to_string(),
};
let item = MediaItem {
// Audio and direct-URL items never negotiate a transport.
transport: None,
id: "item-subs".to_string(),
title: "With Subs".to_string(),
name: None,
artist: None,
album: None,
album_name: None,
album_id: None,
artist_items: None,
artists: None,
primary_image_tag: None,
image_id: None,
item_type: None,
playlist_id: None,
duration: None,
artwork_url: None,
media_type: MediaType::Video,
source: MediaSource::DirectUrl {
url: "video.mp4".to_string(),
},
video_codec: None,
needs_transcoding: false,
video_width: None,
video_height: None,
subtitles: vec![sub],
series_id: None,
server_id: None,
};
assert_eq!(item.subtitles.len(), 1);
assert_eq!(item.subtitles[0].language, Some("eng".to_string()));
}
#[test]
fn test_media_item_serialization() {
let item = MediaItem {
// Audio and direct-URL items never negotiate a transport.
transport: None,
id: "serial-item".to_string(),
title: "Serial Test".to_string(),
name: Some("Name".to_string()),
artist: None,
album: None,
album_name: None,
album_id: None,
artist_items: None,
artists: None,
primary_image_tag: None,
image_id: None,
item_type: Some("Movie".to_string()),
playlist_id: None,
duration: Some(120.0),
artwork_url: None,
media_type: MediaType::Video,
source: MediaSource::DirectUrl {
url: "https://example.com/movie.mp4".to_string(),
},
video_codec: Some("h264".to_string()),
needs_transcoding: false,
video_width: Some(1920),
video_height: Some(1080),
subtitles: vec![],
series_id: None,
server_id: None,
};
let json = serde_json::to_string(&item);
assert!(json.is_ok());
let serialized = json.unwrap();
assert!(serialized.contains("serial-item"));
assert!(serialized.contains("Serial Test"));
}
}