dtourolle/jellytau
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 39 Wiki Activity

feat(playback): let Rust decide what stream to play, and say so

Browse Source 
Playing a video meant asking the server to re-encode it, always. That
decision was made nowhere and written down nowhere, so whoever needed it
re-derived it downstream — the player worked out whether it had been handed
a playlist by looking for ".m3u8" in the URL, in two places. A viewer paid
for a transcode of a file their device could have played untouched, and the
app could not tell them which it was.

One negotiation now produces one self-describing StreamSelection — direct
play, remux or transcode; over a playlist, a plain HTTP file, or a local one
— and every renderer consumes that same answer.

Measured against the development server (Jellyfin 10.11.5), 400 items
sampled for codec mix and 40 put through a real PlaybackInfo negotiation
per profile:

  Linux / WebKitGTK (h264 only, 2ch)          3/40 —  7% direct play
  Android / ExoPlayer (hevc, ac3/eac3, 6ch)  34/40 — 85% direct play

The library is ~80% hevc, which is why the two diverge so hard. The payoff
is overwhelmingly Android, where 85% of plays were starting a transcode
nobody needed. Linux stays near 7% until libmpv decodes the picture — the
h264-only profile is a WebKitGTK constraint, not a JellyTau choice.

DR-219  StreamSelection: url + tagged Transport (hls/progressive/localFile)
        + PlaybackKind (directPlay/directStream/transcode) + the negotiated
        rendition + this source's ladder + a needs_transcoding flag derived
        in Rust so the rule is answered once. Both enums are serde-tagged
        so the frontend matches a discriminant, not a substring. The paths
        that never negotiate get the same shape from Rust rather than
        assembling one — media_local_selection for a downloaded file,
        LiveStreamInfo.transport for a live channel — so there is no second
        place where a transport is decided.

DR-220  The ceiling becomes two levels: a durable device default (Settings,
        persisted) and a per-playback override the in-player picker sets.
        The picker had called itself a "this film, this connection" control
        since it was written but wrote the process-wide default, so dropping
        one awkward film to 2 Mbps silently capped every video played
        afterwards for the rest of the process, with Settings still showing
        the old value. The override is cleared whenever playback moves to a
        new item, which stops it surviving into an autoplayed next episode.
        effective_streaming_quality() is the single resolution point.

DR-221  The quality picker is filled from what this media source can offer.
        Rust marks a rung exceeds_source when its ceiling is at or above the
        source's own bitrate — such a rung is another way to spell Original
        — and the frontend does not draw those. Original is never marked; a
        source whose bitrate the server does not report marks nothing, which
        keeps every rung offered.

DR-222  Direct play and direct stream are negotiated, with two client-side
        overrides on top because the server's answer is right about the file
        and wrong about what this app will do with it: undecodable audio
        (Jellyfin 10.11.5 honours a DirectPlayProfile's container and video
        codec but ignores its audio codec, so it offers direct play for an
        E-AC-3 track the webview renders in silence) and a viewer-pinned
        audio track the file does not default to. A direct stream is a remux
        and is deliberately not counted as transcoding.

DR-223  Dropped on measurement, not deferred. A master playlist from this
        server carries exactly one EXT-X-STREAM-INF: Jellyfin builds it from
        the single rendition the request asked for rather than publishing a
        ladder. So there is no adaptation for hls.js to be preserving and
        none mpv would lose — the claim that there was, in
        playback-backend-unification.md, does not hold. Recorded rather than
        deleted because it is a measurement: a server that does publish a
        ladder would change the answer.

DR-224  Every backend consumes the same selection. The queue item carries
        the transport, so player_seek_video picks its seek strategy from the
        backend's decision instead of the last stream_url.contains(".m3u8")
        in the codebase. Items queued by a path that never negotiated carry
        None and fall back to needs_transcoding, which is exact rather than
        a guess because every transcode this app requests is HLS (DR-140).

The frontend loader decision moves to streamTransport.ts so it can be
tested: the two cases that pin it are the ones that failed against the old
implementation — a progressive stream whose URL contains ".m3u8" must not
get an HLS loader, and an HLS stream whose URL contains none must.

Also verified the URL the direct-play branch builds actually serves playable
bytes: 206, video/mp4, valid ISO-BMFF, and a mid-file range works, so
seeking a direct play works.

The spec is folded into docs/architecture/{01,02,03} and deleted, per the
rule that docs/specs holds only work that has not shipped. DR-121 leaves
read-through-media-cache.md with a pointer; that spec keeps its capture half.

Not verified: real playback on a device. Direct play changes what actually
gets played, and neither fixtures nor curl prove the WebKitGTK and ExoPlayer
paths render it.
This commit is contained in:
dtourolle
2026-08-22 13:45:03 +02:00
parent 5fede123e7
commit 109700b949
45 changed files with 9744 additions and 6581 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src-tauri/src/commands/player/mod.rs
+133 -61
View File
@@ -30,7 +30,7 @@ use crate::player::{
};
use crate::repository::{
types::{GetItemsOptions, ImageOptions, ImageType},
MediaRepository,
MediaRepository, StreamSelection,
};
use crate::settings::VideoSettings;
use crate::storage::db_service::{DatabaseService, Query, QueryParam};
@@ -179,6 +179,18 @@ pub struct PlayItemRequest {
pub video_codec: String,
/// Whether the video requires server-side transcoding
pub needs_transcoding: bool,
/// How this item's stream is fetched, as the backend decided it.
///
/// Carried on the queue item so a later seek/reload does not have to guess.
/// `None` for items queued by a path that never negotiated (audio tracks,
/// direct URLs) and for anything queued before this field existed, where the
/// caller falls back to `needs_transcoding` — every transcode this app
/// requests is HLS (DR-140), so that fallback is exact rather than a guess.
///
/// TRACES: UR-003, UR-004, UR-079 | DR-224, DR-229
#[serde(default)]
pub transport: Option<crate::repository::Transport>,
/// Optional now-playing metadata. Used by the background-audio handoff so the
/// lockscreen/miniplayer show the item (title/subtitle/artwork). Defaulted so
/// existing video-only callers need not send them.
@@ -317,9 +329,15 @@ pub enum VideoSeekResponse {
},
/// Reload stream from new position (transcoded non-HLS)
ReloadStream {
/// New stream URL starting at seek position
new_url: String,
/// Position offset to track (for display purposes)
/// What to open, and how — transport included, so the frontend picks
/// its loader from a tagged enum rather than by searching the URL for
/// `.m3u8`. TRACES: UR-079 | DR-224
selection: StreamSelection,
/// `seek_offset` carries the position to RESUME AT, not a base to add to
/// the element's clock. The reloaded stream starts at the item's zero —
/// a position on an HLS playlist makes the server 400 every segment
/// behind it (DR-181) — so the adapter reaches the position by seeking
/// the element and leaves the transcode offset at zero.
seek_offset: f64,
},
}
@@ -335,8 +353,8 @@ pub enum AudioTrackSwitchResponse {
},
/// HTML5 needs to reload stream with new audio track
ReloadStream {
/// New stream URL with selected audio track
new_url: String,
/// What to open, and how. TRACES: UR-079 | DR-224
selection: StreamSelection,
/// Current position to resume from
position: f64,
},
@@ -356,10 +374,13 @@ pub enum StreamQualityResponse {
/// Position playback resumed at.
position: f64,
},
/// HTML5 must reload its element with this URL.
/// HTML5 must reload its element with this selection.
ReloadStream {
/// New stream URL, already transcoded to the requested ceiling.
new_url: String,
/// What to open, and how — already negotiated against the requested
/// ceiling. Carries `available` too, so a picker opened after a quality
/// change still describes the source correctly.
/// TRACES: UR-070, UR-079 | DR-224, DR-226
selection: StreamSelection,
/// Position to resume from.
position: f64,
},
@@ -415,6 +436,8 @@ pub(super) async fn create_media_item(
source,
video_codec: Some(req.video_codec),
needs_transcoding: req.needs_transcoding,
// The caller's negotiated transport, when it had one. TRACES: UR-079 | DR-229
transport: req.transport,
video_width: None, // Not available from video-only request
video_height: None, // Not available from video-only request
// Sideloaded subtitles, in the order the frontend sent them — that order
@@ -663,6 +686,14 @@ pub async fn player_play_item(
item.title, item.stream_url
);
// A ceiling chosen from the in-player picker belongs to the playback it was
// chosen for. Starting a different item returns to the device default —
// otherwise "2 Mbps, just for this one film" quietly governs the rest of the
// session, which is the defect DR-225 exists to close.
//
// TRACES: UR-074, UR-079 | DR-225
crate::repository::online::clear_playback_quality_override();
// Create media item, checking for local download first
let media_item = create_media_item(item, Some(&db)).await?;
@@ -762,6 +793,8 @@ pub async fn player_enter_background_audio(
// create_media_item() because that hardcodes MediaType::Video; background
// audio must be Audio so no video decode is started.
let media_item = MediaItem {
// Audio and direct-URL items never negotiate a transport.
transport: None,
id: item.id.clone(),
title: item.title.clone(),
name: Some(item.title.clone()),
@@ -935,6 +968,14 @@ pub async fn player_play_queue(
request.shuffle
);
// A ceiling chosen from the in-player picker belongs to the playback it was
// chosen for. Starting a different item returns to the device default —
// otherwise "2 Mbps, just for this one film" quietly governs the rest of the
// session, which is the defect DR-225 exists to close.
//
// TRACES: UR-074, UR-079 | DR-225
crate::repository::online::clear_playback_quality_override();
// Handle shuffle first
if request.shuffle {
let controller = player.0.lock().await;
@@ -1347,7 +1388,7 @@ pub async fn player_seek(
///
/// This command analyzes the current video stream and automatically chooses
/// the best seeking strategy:
/// - HLS streams (.m3u8): Use native seeking
/// - HLS streams: Use native seeking
/// - Direct play streams: Use native seeking
/// - Transcoded non-HLS: Request new stream URL from server starting at seek position
///
@@ -1377,7 +1418,7 @@ pub async fn player_seek_video(
// Get current playing item to analyze stream characteristics
// Clone what we need to avoid holding locks across await points
let (needs_transcoding, jellyfin_item_id, stream_url, is_local) = {
let (needs_transcoding, jellyfin_item_id, is_local, transport) = {
let controller = player.0.lock().await;
let queue_arc = controller.queue();
let queue = queue_arc.lock().map_err(|e| e.to_string())?;
@@ -1393,18 +1434,27 @@ pub async fn player_seek_video(
.ok_or("Current video has no Jellyfin ID")?
.to_string();
let (stream_url, is_local_file) = match &current_item.source {
MediaSource::Remote { stream_url, .. } => (stream_url.clone(), false),
MediaSource::Local { .. } => (String::new(), true),
MediaSource::DirectUrl { url } => (url.clone(), false),
};
// The URL itself is no longer read here: the seek strategy now comes
// from the item's own `transport`, not from inspecting the string.
let is_local_file = matches!(current_item.source, MediaSource::Local { .. });
let needs_trans = current_item.needs_transcoding;
(needs_trans, jellyfin_id, stream_url, is_local_file)
let transport = current_item.transport;
(needs_trans, jellyfin_id, is_local_file, transport)
}; // Locks are dropped here
// Determine seek strategy using the testable helper function
let is_hls = stream_url.contains(".m3u8");
// The transport comes from the backend's own decision, not from searching
// the URL for `.m3u8` — Rust built that URL and knows what it is. Items
// queued without one fall back to `needs_transcoding`, which is exact:
// every transcode this app requests is HLS (DR-140).
//
// TRACES: UR-004, UR-079 | DR-224, DR-229
let is_hls = match transport {
Some(crate::repository::Transport::Hls) => true,
Some(crate::repository::Transport::Progressive)
| Some(crate::repository::Transport::LocalFile) => false,
None => needs_transcoding,
};
let strategy = determine_video_seek_strategy(is_local, is_hls, needs_transcoding, use_html5);
info!("[player_seek_video] Stream analysis: is_local={}, is_hls={}, needs_transcoding={}, use_html5={}, strategy={:?}",
@@ -1428,29 +1478,22 @@ pub async fn player_seek_video(
// Transcoded non-HLS with HTML5 - frontend handles stream reload
info!("[player_seek_video] HTML5 reload stream - requesting new stream URL");
let new_url = repository
.get_video_stream_url(
let selection = repository
.get_stream_selection(
&jellyfin_item_id,
media_source_id.as_deref(),
audio_stream_index,
)
.await
.map_err(|e| format!("Failed to get video stream URL: {:?}", e))?;
.map_err(|e| format!("Failed to select a stream: {:?}", e))?;
info!(
"[player_seek_video] Got new stream URL for position {}",
position
"[player_seek_video] Selected {:?} over {:?} for position {}",
selection.playback_kind, selection.transport, position
);
// `seek_offset` carries the position to RESUME AT, not a base to add
// to the element's clock. The reloaded stream starts at the item's
// zero — a position on an HLS playlist makes the server 400 every
// segment behind it (DR-181) — so the adapter reaches the position by
// seeking the element and leaves the transcode offset at zero. The
// field keeps its name only because renaming it means regenerating
// the specta bindings; `reloadSource` documents the contract.
Ok(VideoSeekResponse::ReloadStream {
new_url,
selection,
seek_offset: position,
})
}
@@ -1458,16 +1501,17 @@ pub async fn player_seek_video(
// Transcoded non-HLS with native backend - backend handles stream reload
info!("[player_seek_video] Backend reload stream - requesting new stream URL");
let new_url = repository
.get_video_stream_url(
let selection = repository
.get_stream_selection(
&jellyfin_item_id,
media_source_id.as_deref(),
audio_stream_index,
)
.await
.map_err(|e| format!("Failed to get video stream URL: {:?}", e))?;
.map_err(|e| format!("Failed to select a stream: {:?}", e))?;
let new_url = selection.url.clone();
info!("[player_seek_video] Got new stream URL, handling reload internally");
info!("[player_seek_video] Got new selection, handling reload internally");
// Stop current playback
{
@@ -1568,20 +1612,25 @@ pub async fn player_switch_audio_track(
.to_string()
};
// Get new stream URL with selected audio track. It starts at zero — an
// HLS playlist cannot carry a position (DR-181) — and `position` below
// tells the frontend where to seek the reloaded element back to.
let new_url = repository
.get_video_stream_url(
// Select a stream carrying the chosen audio track. It starts at zero —
// an HLS playlist cannot carry a position (DR-181) — and `position`
// below tells the frontend where to seek the reloaded element back to.
//
// Pinning a track is itself a reason the source cannot be direct-played:
// the file has one default track and the viewer asked for another, so
// the negotiation returns a transcode. That decision lives in
// `decide_playback_kind`, not here.
let selection = repository
.get_stream_selection(
&jellyfin_item_id,
media_source_id.as_deref(),
Some(stream_index),
)
.await
.map_err(|e| format!("Failed to get video stream URL: {:?}", e))?;
.map_err(|e| format!("Failed to select a stream: {:?}", e))?;
Ok(AudioTrackSwitchResponse::ReloadStream {
new_url,
selection,
position: current_position.unwrap_or(0.0),
})
} else {
@@ -1604,23 +1653,26 @@ pub async fn player_switch_audio_track(
/// two-sided split: HTML5 gets the URL back and reloads its own element, while a
/// native backend is reloaded here.
///
/// The change applies to this playback *and* to everything started afterwards
/// (it sets the process-wide ceiling), but it is deliberately **not** persisted:
/// the in-player picker is a "this film, this connection" control, and the
/// durable default belongs to Settings. `player_set_video_settings` is the one
/// that writes to the database.
/// The change applies to **this playback only**. The in-player picker is a
/// "this film, this connection" control and its doc has always said so, but it
/// used to be implemented by writing the process-wide ceiling — so choosing
/// 2 Mbps to get one awkward film moving silently capped every video played
/// afterwards for the rest of the process, with the Settings screen still
/// showing the old value and nothing in the UI admitting the change. It now
/// sets a per-playback override that the next item clears; the durable default
/// belongs to Settings, and `player_set_video_settings` is the one that writes
/// to the database.
///
/// TRACES: UR-074 | DR-162
/// TRACES: UR-074, UR-079 | DR-162, DR-225
#[tauri::command]
#[specta::specta]
// Three of the nine arguments are Tauri `State<'_, _>` injections, not caller
// Two of the eight arguments are Tauri `State<'_, _>` injections, not caller
// input. Folding the rest into a struct would change the IPC contract and the
// generated TypeScript for no readability gain.
#[allow(clippy::too_many_arguments)]
pub async fn player_set_stream_quality(
player: State<'_, PlayerStateWrapper>,
repository_manager: State<'_, super::repository::RepositoryManagerWrapper>,
video_settings: State<'_, VideoSettingsWrapper>,
repository_handle: String,
quality: crate::settings::StreamingQuality,
use_html5: bool,
@@ -1657,25 +1709,31 @@ pub async fn player_set_stream_quality(
.to_string()
};
// Set the ceiling *before* building the URL — the builder reads it.
crate::repository::online::set_streaming_quality(quality);
{
let mut settings = video_settings.0.lock().map_err(|e| e.to_string())?;
settings.streaming_quality = quality;
}
// Set the ceiling *before* negotiating — the negotiation and every URL
// builder resolve through `effective_streaming_quality`, and they have to
// agree or the cap leaks (a negotiation authorising a direct play the URL
// builder then never gets to constrain).
//
// Deliberately the *override*, not the device default: see the doc above.
// TRACES: UR-074, UR-079 | DR-225
crate::repository::online::set_playback_quality_override(quality);
let position = current_position.unwrap_or(0.0);
let new_url = repository
.get_video_stream_url(
let selection = repository
.get_stream_selection(
&jellyfin_item_id,
media_source_id.as_deref(),
audio_stream_index,
)
.await
.map_err(|e| format!("Failed to get video stream URL: {:?}", e))?;
.map_err(|e| format!("Failed to select a stream: {:?}", e))?;
let new_url = selection.url.clone();
if use_html5 {
return Ok(StreamQualityResponse::ReloadStream { new_url, position });
return Ok(StreamQualityResponse::ReloadStream {
selection,
position,
});
}
// Native backend (Android/ExoPlayer): stop, repoint the queue entry at the
@@ -2208,6 +2266,8 @@ pub async fn player_play_album_track(
let primary_image_tag_for_url = track.primary_image_tag.clone();
let media_item = MediaItem {
// Audio and direct-URL items never negotiate a transport.
transport: None,
id: track.id.clone(),
title: track.name.clone(),
name: Some(track.name.clone()), // Frontend compatibility
@@ -2353,6 +2413,14 @@ pub async fn player_play_tracks(
repository_handle: String,
request: PlayTracksRequest,
) -> Result<PlayerStatus, String> {
// A ceiling chosen from the in-player picker belongs to the playback it was
// chosen for. Starting a different item returns to the device default —
// otherwise "2 Mbps, just for this one film" quietly governs the rest of the
// session, which is the defect DR-225 exists to close.
//
// TRACES: UR-074, UR-079 | DR-225
crate::repository::online::clear_playback_quality_override();
info!(
"player_play_tracks called: {} tracks, start_index={}, shuffle={}",
request.track_ids.len(),
@@ -2404,6 +2472,8 @@ pub async fn player_play_tracks(
// Transform to MediaItem with frontend-compatible fields
let primary_image_tag_for_url = track.primary_image_tag.clone();
let media_item = MediaItem {
// Audio and direct-URL items never negotiate a transport.
transport: None,
id: track.id.clone(),
title: track.name.clone(),
name: Some(track.name.clone()), // Frontend compatibility
@@ -3210,6 +3280,8 @@ mod tests {
let db = DatabaseWrapper(Mutex::new(database));
let make_item = |id: &str| MediaItem {
// Audio and direct-URL items never negotiate a transport.
transport: None,
id: id.to_string(),
title: id.to_string(),
name: None,
src-tauri/src/commands/player/queue.rs
+4
View File
@@ -198,6 +198,8 @@ pub async fn player_add_track_by_id(
// Build MediaItem with artwork URL from repository and frontend-compatible fields
let primary_image_tag_for_url = track.primary_image_tag.clone();
let media_item = MediaItem {
// Audio and direct-URL items never negotiate a transport.
transport: None,
id: track.id.clone(),
title: track.name.clone(),
name: Some(track.name.clone()), // Frontend compatibility
@@ -317,6 +319,8 @@ pub async fn player_add_tracks_by_ids(
// Build MediaItem with artwork URL from repository and frontend-compatible fields
let primary_image_tag_for_url = track.primary_image_tag.clone();
let media_item = MediaItem {
// Audio and direct-URL items never negotiate a transport.
transport: None,
id: track.id.clone(),
title: track.name.clone(),
name: Some(track.name.clone()), // Frontend compatibility
src-tauri/src/commands/repository.rs
+30 -1
View File
@@ -16,7 +16,7 @@ use crate::domain::rank_search_results;
use crate::jellyfin::HttpClient;
use crate::repository::{
series_progress, types::*, HybridRepository, MediaRepository, OfflineRepository,
OnlineRepository,
OnlineRepository, StreamSelection,
};
/// Repository handle manager
@@ -606,6 +606,35 @@ pub async fn repository_get_video_stream_url(
.map_err(|e| format!("{:?}", e))
}
/// Decide what stream to play for a video, and describe it.
///
/// Replaces `repository_get_video_stream_url` for playback. The returned
/// [`StreamSelection`] carries the transport explicitly, so the frontend picks
/// its loader from a tagged enum instead of testing the URL for `.m3u8`; and it
/// carries the quality ladder as it applies to *this* source, so the picker can
/// stop offering rungs that produce the same bytes as Original.
///
/// No start-position parameter, for the same reason as the URL builder: a
/// position on an HLS playlist is copied onto every segment URI and the server
/// rejects each with `400` (DR-181). Callers resume by seeking after load.
///
/// TRACES: UR-070, UR-079 | DR-224, DR-226, DR-227 | UT-212
#[tauri::command]
#[specta::specta]
pub async fn repository_get_stream_selection(
manager: State<'_, RepositoryManagerWrapper>,
handle: String,
item_id: String,
media_source_id: Option<String>,
audio_stream_index: Option<i32>,
) -> Result<StreamSelection, String> {
let repo = manager.0.get(&handle).ok_or("Repository not found")?;
repo.as_ref()
.get_stream_selection(&item_id, media_source_id.as_deref(), audio_stream_index)
.await
.map_err(|e| format!("{:?}", e))
}
/// Get an audio-only stream URL for a *video* item (background-audio handoff).
///
/// TRACES: UR-040 | JA-032 | UT-061
src-tauri/src/commands/storage/mod.rs
+24
View File
@@ -104,6 +104,30 @@ pub fn media_local_url(
.ok_or_else(|| "Local media server is not running".to_string())
}
/// The stream selection for a downloaded file.
///
/// The local-playback counterpart to `repository_get_stream_selection`. A file
/// on disk needs no negotiation — it is a direct play over a local transport,
/// with no quality ladder, because nothing about it can be re-negotiated — but
/// the *frontend must not be the one to say so*. It gets the same
/// [`StreamSelection`] shape as a streamed source so the player has one contract
/// to consume rather than two, and so no caller has to infer a transport from a
/// loopback URL.
///
/// TRACES: UR-071, UR-079 | DR-224
#[tauri::command]
#[specta::specta]
pub fn media_local_selection(
server: State<crate::media_server::MediaServerWrapper>,
path: String,
) -> Result<crate::repository::StreamSelection, String> {
server
.0
.as_ref()
.map(|s| crate::repository::StreamSelection::local_file(s.url_for(&path)))
.ok_or_else(|| "Local media server is not running".to_string())
}
/// Get storage directory path (parent directory of the database file)
#[tauri::command]
#[specta::specta]