feat(player): native video on Linux, and one contract for every player (v0.11.0)

mpv now decodes video on Linux, drawn into a framebuffer we own and blitted
into the default vbox's draw handler. Tauri's widget tree is untouched, so an
upgrade that assumes its own layout cannot invalidate this. Direct play means
the original file, hardware decoding, and no server transcode at all — where
previously every desktop video was re-encoded to h264 for the browser engine,
whatever the file actually was. Off by default: JELLYTAU_NATIVE_VIDEO=1.

That settles finding 2 of playback-backend-unification.md — "native video
cannot be composited with a Tauri webview" — by demonstration rather than
argument, on X11 and Wayland both.

Turning it on exposed nine defects, none of them mpv's. Each was the same
mistake in a different place: a capability written down as a compile-time fact
about the platform, or a state asserted instead of confirmed.

  DR-238/246  a seek routed by the stream's container rather than by what the
              engine could do with it - correct only while one player handled
              those streams, silent the moment another did
  DR-239      a property handled but never observed, so the play/pause button
              waited for an event that could not arrive
  DR-240      fullscreen expanding the document while the window stayed put
  DR-241      a seek issued before the engine had a file, failed, and discarded
              - which is why resume began at zero
  DR-247      a Linux-only gate outliving the caller that made it Linux-only,
              breaking the Android build outright
  DR-250      a stop aimed at whichever renderer bookkeeping believed was in
              charge, missing the one actually making sound
  DR-251      a duration of zero believed, leaving the seek bar no scale
  DR-252      a junk float converted to a Duration, panicking the backend the
              instant a length-less stream appeared

So the MediaPlayer contract (DR-242 … DR-247): `open` carries a start position,
so no caller sequences load-then-seek and none can race an engine's load;
`seek` states a destination and leaves in-place-versus-re-open to the engine;
`snapshot` is one coherent read; and `Phase::Opening` names the window where
intent used to be lost. One conformance suite runs against every engine —
FakePlayer and mpv under cargo test, ExoPlayer instrumented on a device — so an
engine is either correct or visibly failing.

Two of the nine were introduced during this work and caught on hardware, not by
any suite: an over-broad capability that grouped ExoPlayer with mpv, and the
Duration panic. The suites test engines that behave. That is recorded in
docs/native-player-verification.md, which asks for the exact action sequences
that found them.

Verified: all automated gates, conformance (mpv 9/9, legacy 8/9 by design,
ExoPlayer 7/7 on device), and manual desktop and Android passes on real
hardware.

Known open and deliberately shipped: resume reads local progress and never the
server's; the background-audio handoff still declares a state swap it does not
confirm (the symptom is now impossible, the race is not); and `bun run
android:dev` builds an APK carrying the release application id, whose failure
message advises an uninstall that would destroy app data. Fix that last one
before anyone else builds for Android.

Squashed from worktree-linux-native-video, which keeps the per-defect history.
This commit is contained in:
2026-08-23 10:51:45 +02:00
parent 5fede123e7
commit 11d9d760d8
87 changed files with 15968 additions and 7508 deletions
+218 -82
View File
@@ -30,7 +30,7 @@ use crate::player::{
};
use crate::repository::{
types::{GetItemsOptions, ImageOptions, ImageType},
MediaRepository,
MediaRepository, StreamSelection,
};
use crate::settings::VideoSettings;
use crate::storage::db_service::{DatabaseService, Query, QueryParam};
@@ -179,6 +179,18 @@ pub struct PlayItemRequest {
pub video_codec: String,
/// Whether the video requires server-side transcoding
pub needs_transcoding: bool,
/// How this item's stream is fetched, as the backend decided it.
///
/// Carried on the queue item so a later seek/reload does not have to guess.
/// `None` for items queued by a path that never negotiated (audio tracks,
/// direct URLs) and for anything queued before this field existed, where the
/// caller falls back to `needs_transcoding` — every transcode this app
/// requests is HLS (DR-140), so that fallback is exact rather than a guess.
///
/// TRACES: UR-003, UR-004, UR-079 | DR-225, DR-230
#[serde(default)]
pub transport: Option<crate::repository::Transport>,
/// Optional now-playing metadata. Used by the background-audio handoff so the
/// lockscreen/miniplayer show the item (title/subtitle/artwork). Defaulted so
/// existing video-only callers need not send them.
@@ -317,9 +329,15 @@ pub enum VideoSeekResponse {
},
/// Reload stream from new position (transcoded non-HLS)
ReloadStream {
/// New stream URL starting at seek position
new_url: String,
/// Position offset to track (for display purposes)
/// What to open, and how — transport included, so the frontend picks
/// its loader from a tagged enum rather than by searching the URL for
/// `.m3u8`. TRACES: UR-079 | DR-225
selection: StreamSelection,
/// `seek_offset` carries the position to RESUME AT, not a base to add to
/// the element's clock. The reloaded stream starts at the item's zero —
/// a position on an HLS playlist makes the server 400 every segment
/// behind it (DR-181) — so the adapter reaches the position by seeking
/// the element and leaves the transcode offset at zero.
seek_offset: f64,
},
}
@@ -335,8 +353,8 @@ pub enum AudioTrackSwitchResponse {
},
/// HTML5 needs to reload stream with new audio track
ReloadStream {
/// New stream URL with selected audio track
new_url: String,
/// What to open, and how. TRACES: UR-079 | DR-225
selection: StreamSelection,
/// Current position to resume from
position: f64,
},
@@ -351,15 +369,31 @@ pub enum AudioTrackSwitchResponse {
#[derive(specta::Type, Debug, Serialize)]
#[serde(tag = "strategy", rename_all = "camelCase")]
pub enum StreamQualityResponse {
/// The native backend was reloaded here; nothing left for the frontend.
/// The native backend was reloaded here; nothing left for the frontend to
/// *do* — but it still has to be told what was negotiated.
///
/// This carried only a position at first, which left the picker on Android
/// pinned to the rendition of the *first* stream: the UI derives the rung in
/// force from the selection it holds, nothing replaced that selection on the
/// native path, and a transcode always has a rendition — so the fallback
/// that would have used the requested value was never reached. The stream
/// changed and the menu did not.
///
/// TRACES: UR-074, UR-079 | DR-226, DR-227
Native {
/// What the backend actually opened, so the UI reflects it rather than
/// assuming the request was honoured verbatim.
selection: StreamSelection,
/// Position playback resumed at.
position: f64,
},
/// HTML5 must reload its element with this URL.
/// HTML5 must reload its element with this selection.
ReloadStream {
/// New stream URL, already transcoded to the requested ceiling.
new_url: String,
/// What to open, and how — already negotiated against the requested
/// ceiling. Carries `available` too, so a picker opened after a quality
/// change still describes the source correctly.
/// TRACES: UR-070, UR-079 | DR-225, DR-227
selection: StreamSelection,
/// Position to resume from.
position: f64,
},
@@ -415,6 +449,8 @@ pub(super) async fn create_media_item(
source,
video_codec: Some(req.video_codec),
needs_transcoding: req.needs_transcoding,
// The caller's negotiated transport, when it had one. TRACES: UR-079 | DR-230
transport: req.transport,
video_width: None, // Not available from video-only request
video_height: None, // Not available from video-only request
// Sideloaded subtitles, in the order the frontend sent them — that order
@@ -663,6 +699,14 @@ pub async fn player_play_item(
item.title, item.stream_url
);
// A ceiling chosen from the in-player picker belongs to the playback it was
// chosen for. Starting a different item returns to the device default —
// otherwise "2 Mbps, just for this one film" quietly governs the rest of the
// session, which is the defect DR-226 exists to close.
//
// TRACES: UR-074, UR-079 | DR-226
crate::repository::online::clear_playback_quality_override();
// Create media item, checking for local download first
let media_item = create_media_item(item, Some(&db)).await?;
@@ -681,18 +725,30 @@ pub async fn player_play_item(
}
let controller = player.0.lock().await;
// On Linux, video plays in the WebKitGTK HTML5 <video> element (see
// get_player_status -> use_html5_element). The MPV backend has no embedded
// window, so loading the stream into it would only start a redundant decode
// (and the frontend would immediately stop it). Only load into the native
// backend on platforms that actually render video through it (e.g. Android).
#[cfg(not(target_os = "linux"))]
controller
.play_item(media_item)
.map_err(|e| e.to_string())?;
#[cfg(target_os = "linux")]
{
// Keep the queue in sync for UI/remote-transfer without starting MPV.
// Who gets the stream depends on who is going to *render* it, which is a
// runtime question, not a platform constant.
//
// Historically Linux video was always the webview's (`use_html5_element`),
// so handing the file to MPV as well would only have started a redundant
// decode with no window to show it in — hence a `#[cfg(not(linux))]` guard
// and a queue-only path here. With mpv drawing the picture that inverts:
// the webview is no longer loading anything, so if this does not load the
// file, *nothing does*. The symptom is total silence — no picture and no
// audio — which reads like a broken stream rather than a stream nobody was
// given.
//
// This is the fifth place in this cycle where a renderer's capability was
// written as a compile-time platform fact. Same fix as the others: ask.
//
// TRACES: UR-080 | DR-231, DR-235
let renders_natively = cfg!(not(target_os = "linux")) || crate::player::native_video::enabled();
if renders_natively {
controller
.play_item(media_item)
.map_err(|e| e.to_string())?;
} else {
// The webview will play it; keep the queue in sync for the UI and for a
// remote transfer without starting a second decode.
controller
.set_current_item(media_item)
.map_err(|e| e.to_string())?;
@@ -762,6 +818,8 @@ pub async fn player_enter_background_audio(
// create_media_item() because that hardcodes MediaType::Video; background
// audio must be Audio so no video decode is started.
let media_item = MediaItem {
// Audio and direct-URL items never negotiate a transport.
transport: None,
id: item.id.clone(),
title: item.title.clone(),
name: Some(item.title.clone()),
@@ -852,7 +910,7 @@ pub async fn player_enter_background_audio(
/// playing there is nothing to pause, and an error would make the frontend
/// handle a case that is not a failure.
///
/// TRACES: UR-040, UR-041 | DR-224 | UT-211
/// TRACES: UR-040, UR-041 | DR-225 | UT-212
#[tauri::command]
#[specta::specta]
pub async fn player_background_action(
@@ -935,6 +993,14 @@ pub async fn player_play_queue(
request.shuffle
);
// A ceiling chosen from the in-player picker belongs to the playback it was
// chosen for. Starting a different item returns to the device default —
// otherwise "2 Mbps, just for this one film" quietly governs the rest of the
// session, which is the defect DR-226 exists to close.
//
// TRACES: UR-074, UR-079 | DR-226
crate::repository::online::clear_playback_quality_override();
// Handle shuffle first
if request.shuffle {
let controller = player.0.lock().await;
@@ -1115,6 +1181,13 @@ pub async fn player_stop(
// Check if we're in remote mode
let mode = playback_mode.0.get_mode();
// Stopping is a state transition worth seeing in a log. Native video is
// what made its absence matter: the webview <video> stopped implicitly when
// the component unmounted, so nothing ever had to call this — and "never
// called" and "called but the backend kept playing" look identical from
// outside without it.
info!("[player_stop] called (mode: {:?})", mode);
if let crate::playback_mode::PlaybackMode::Remote { session_id } = mode {
// Send stop command to remote session - clone client before await
let client = {
@@ -1347,7 +1420,7 @@ pub async fn player_seek(
///
/// This command analyzes the current video stream and automatically chooses
/// the best seeking strategy:
/// - HLS streams (.m3u8): Use native seeking
/// - HLS streams: Use native seeking
/// - Direct play streams: Use native seeking
/// - Transcoded non-HLS: Request new stream URL from server starting at seek position
///
@@ -1377,7 +1450,7 @@ pub async fn player_seek_video(
// Get current playing item to analyze stream characteristics
// Clone what we need to avoid holding locks across await points
let (needs_transcoding, jellyfin_item_id, stream_url, is_local) = {
let (needs_transcoding, jellyfin_item_id, is_local) = {
let controller = player.0.lock().await;
let queue_arc = controller.queue();
let queue = queue_arc.lock().map_err(|e| e.to_string())?;
@@ -1393,22 +1466,34 @@ pub async fn player_seek_video(
.ok_or("Current video has no Jellyfin ID")?
.to_string();
let (stream_url, is_local_file) = match &current_item.source {
MediaSource::Remote { stream_url, .. } => (stream_url.clone(), false),
MediaSource::Local { .. } => (String::new(), true),
MediaSource::DirectUrl { url } => (url.clone(), false),
};
// Neither the URL nor the item's transport is read here any more. The
// strategy turns on whether the *engine* can seek a transcode in place,
// which it declares for itself — so the container the stream happens to
// arrive in stopped being a proxy for anything (DR-246).
let is_local_file = matches!(current_item.source, MediaSource::Local { .. });
let needs_trans = current_item.needs_transcoding;
(needs_trans, jellyfin_id, stream_url, is_local_file)
(current_item.needs_transcoding, jellyfin_id, is_local_file)
}; // Locks are dropped here
// Determine seek strategy using the testable helper function
let is_hls = stream_url.contains(".m3u8");
let strategy = determine_video_seek_strategy(is_local, is_hls, needs_transcoding, use_html5);
// Whether a transcode can be seeked in place is asked of the engine that is
// rendering, not guessed from the URL's shape or from who is rendering.
// TRACES: UR-040, UR-079 | DR-238, DR-246
let seeks_transcoded_in_place = {
let controller = player.0.lock().await;
controller.capabilities().seeks_transcoded_in_place
};
let strategy = determine_video_seek_strategy(
is_local,
seeks_transcoded_in_place,
needs_transcoding,
use_html5,
);
info!("[player_seek_video] Stream analysis: is_local={}, is_hls={}, needs_transcoding={}, use_html5={}, strategy={:?}",
is_local, is_hls, needs_transcoding, use_html5, strategy);
info!(
"[player_seek_video] Stream analysis: is_local={}, seeks_transcoded_in_place={}, \
needs_transcoding={}, use_html5={}, strategy={:?}",
is_local, seeks_transcoded_in_place, needs_transcoding, use_html5, strategy
);
match strategy {
VideoSeekStrategy::LocalNativeSeek | VideoSeekStrategy::BackendNativeSeek => {
@@ -1428,29 +1513,22 @@ pub async fn player_seek_video(
// Transcoded non-HLS with HTML5 - frontend handles stream reload
info!("[player_seek_video] HTML5 reload stream - requesting new stream URL");
let new_url = repository
.get_video_stream_url(
let selection = repository
.get_stream_selection(
&jellyfin_item_id,
media_source_id.as_deref(),
audio_stream_index,
)
.await
.map_err(|e| format!("Failed to get video stream URL: {:?}", e))?;
.map_err(|e| format!("Failed to select a stream: {:?}", e))?;
info!(
"[player_seek_video] Got new stream URL for position {}",
position
"[player_seek_video] Selected {:?} over {:?} for position {}",
selection.playback_kind, selection.transport, position
);
// `seek_offset` carries the position to RESUME AT, not a base to add
// to the element's clock. The reloaded stream starts at the item's
// zero — a position on an HLS playlist makes the server 400 every
// segment behind it (DR-181) — so the adapter reaches the position by
// seeking the element and leaves the transcode offset at zero. The
// field keeps its name only because renaming it means regenerating
// the specta bindings; `reloadSource` documents the contract.
Ok(VideoSeekResponse::ReloadStream {
new_url,
selection,
seek_offset: position,
})
}
@@ -1458,16 +1536,17 @@ pub async fn player_seek_video(
// Transcoded non-HLS with native backend - backend handles stream reload
info!("[player_seek_video] Backend reload stream - requesting new stream URL");
let new_url = repository
.get_video_stream_url(
let selection = repository
.get_stream_selection(
&jellyfin_item_id,
media_source_id.as_deref(),
audio_stream_index,
)
.await
.map_err(|e| format!("Failed to get video stream URL: {:?}", e))?;
.map_err(|e| format!("Failed to select a stream: {:?}", e))?;
let new_url = selection.url.clone();
info!("[player_seek_video] Got new stream URL, handling reload internally");
info!("[player_seek_video] Got new selection, handling reload internally");
// Stop current playback
{
@@ -1568,20 +1647,25 @@ pub async fn player_switch_audio_track(
.to_string()
};
// Get new stream URL with selected audio track. It starts at zero — an
// HLS playlist cannot carry a position (DR-181) — and `position` below
// tells the frontend where to seek the reloaded element back to.
let new_url = repository
.get_video_stream_url(
// Select a stream carrying the chosen audio track. It starts at zero —
// an HLS playlist cannot carry a position (DR-181) — and `position`
// below tells the frontend where to seek the reloaded element back to.
//
// Pinning a track is itself a reason the source cannot be direct-played:
// the file has one default track and the viewer asked for another, so
// the negotiation returns a transcode. That decision lives in
// `decide_playback_kind`, not here.
let selection = repository
.get_stream_selection(
&jellyfin_item_id,
media_source_id.as_deref(),
Some(stream_index),
)
.await
.map_err(|e| format!("Failed to get video stream URL: {:?}", e))?;
.map_err(|e| format!("Failed to select a stream: {:?}", e))?;
Ok(AudioTrackSwitchResponse::ReloadStream {
new_url,
selection,
position: current_position.unwrap_or(0.0),
})
} else {
@@ -1604,23 +1688,26 @@ pub async fn player_switch_audio_track(
/// two-sided split: HTML5 gets the URL back and reloads its own element, while a
/// native backend is reloaded here.
///
/// The change applies to this playback *and* to everything started afterwards
/// (it sets the process-wide ceiling), but it is deliberately **not** persisted:
/// the in-player picker is a "this film, this connection" control, and the
/// durable default belongs to Settings. `player_set_video_settings` is the one
/// that writes to the database.
/// The change applies to **this playback only**. The in-player picker is a
/// "this film, this connection" control and its doc has always said so, but it
/// used to be implemented by writing the process-wide ceiling — so choosing
/// 2 Mbps to get one awkward film moving silently capped every video played
/// afterwards for the rest of the process, with the Settings screen still
/// showing the old value and nothing in the UI admitting the change. It now
/// sets a per-playback override that the next item clears; the durable default
/// belongs to Settings, and `player_set_video_settings` is the one that writes
/// to the database.
///
/// TRACES: UR-074 | DR-162
/// TRACES: UR-074, UR-079 | DR-162, DR-226
#[tauri::command]
#[specta::specta]
// Three of the nine arguments are Tauri `State<'_, _>` injections, not caller
// Two of the eight arguments are Tauri `State<'_, _>` injections, not caller
// input. Folding the rest into a struct would change the IPC contract and the
// generated TypeScript for no readability gain.
#[allow(clippy::too_many_arguments)]
pub async fn player_set_stream_quality(
player: State<'_, PlayerStateWrapper>,
repository_manager: State<'_, super::repository::RepositoryManagerWrapper>,
video_settings: State<'_, VideoSettingsWrapper>,
repository_handle: String,
quality: crate::settings::StreamingQuality,
use_html5: bool,
@@ -1657,25 +1744,50 @@ pub async fn player_set_stream_quality(
.to_string()
};
// Set the ceiling *before* building the URL — the builder reads it.
crate::repository::online::set_streaming_quality(quality);
{
let mut settings = video_settings.0.lock().map_err(|e| e.to_string())?;
settings.streaming_quality = quality;
}
// Set the ceiling *before* negotiating — the negotiation and every URL
// builder resolve through `effective_streaming_quality`, and they have to
// agree or the cap leaks (a negotiation authorising a direct play the URL
// builder then never gets to constrain).
//
// Deliberately the *override*, not the device default: see the doc above.
// TRACES: UR-074, UR-079 | DR-226
crate::repository::online::set_playback_quality_override(quality);
let position = current_position.unwrap_or(0.0);
let new_url = repository
.get_video_stream_url(
// Where to resume. `current_position` is the *element's* clock, which only
// the webview path has — on a native backend there is no `<video>` and the
// frontend correctly sends null, so trusting it there resumed every quality
// change from zero.
//
// The player is the authority on position (it is the authority on all
// playback state); asking the DOM for it and falling back to 0 inverted
// that. Fall back to what the controller reports instead.
//
// TRACES: UR-005, UR-074 | DR-226
// The guard is bound inside the arm's block so it is dropped before the
// reload below takes the same lock. This codebase has been bitten by a
// MutexGuard living longer than the expression that produced it.
let position = match current_position {
Some(p) => p,
None => {
let controller = player.0.lock().await;
controller.absolute_position()
}
};
let selection = repository
.get_stream_selection(
&jellyfin_item_id,
media_source_id.as_deref(),
audio_stream_index,
)
.await
.map_err(|e| format!("Failed to get video stream URL: {:?}", e))?;
.map_err(|e| format!("Failed to select a stream: {:?}", e))?;
let new_url = selection.url.clone();
if use_html5 {
return Ok(StreamQualityResponse::ReloadStream { new_url, position });
return Ok(StreamQualityResponse::ReloadStream {
selection,
position,
});
}
// Native backend (Android/ExoPlayer): stop, repoint the queue entry at the
@@ -1705,7 +1817,10 @@ pub async fn player_set_stream_quality(
}
}
Ok(StreamQualityResponse::Native { position })
Ok(StreamQualityResponse::Native {
selection,
position,
})
}
/// Set the active audio track on a native backend directly.
@@ -1993,7 +2108,9 @@ pub async fn player_get_capabilities() -> Result<PlaybackCapabilities, String> {
Ok(PlaybackCapabilities {
uses_webview_audio: !native_audio,
supports_native_video: cfg!(target_os = "android"),
// TRACES: UR-080 | DR-235
supports_native_video: cfg!(target_os = "android")
|| crate::player::native_video::enabled(),
})
}
@@ -2002,6 +2119,11 @@ pub(super) fn get_player_status(controller: &PlayerController) -> PlayerStatus {
let (backend, use_html5_element) = if cfg!(target_os = "android") {
// Android uses ExoPlayer native backend
(VideoBackend::Native, false)
} else if crate::player::native_video::enabled() {
// mpv draws the picture on this desktop; the frontend must not also
// load it into a <video> element or the stream decodes twice and the
// two fight over the audio. TRACES: UR-080 | DR-235
(VideoBackend::Native, false)
} else {
// Linux and other platforms use HTML5 video element in frontend
(VideoBackend::Html5, true)
@@ -2208,6 +2330,8 @@ pub async fn player_play_album_track(
let primary_image_tag_for_url = track.primary_image_tag.clone();
let media_item = MediaItem {
// Audio and direct-URL items never negotiate a transport.
transport: None,
id: track.id.clone(),
title: track.name.clone(),
name: Some(track.name.clone()), // Frontend compatibility
@@ -2353,6 +2477,14 @@ pub async fn player_play_tracks(
repository_handle: String,
request: PlayTracksRequest,
) -> Result<PlayerStatus, String> {
// A ceiling chosen from the in-player picker belongs to the playback it was
// chosen for. Starting a different item returns to the device default —
// otherwise "2 Mbps, just for this one film" quietly governs the rest of the
// session, which is the defect DR-226 exists to close.
//
// TRACES: UR-074, UR-079 | DR-226
crate::repository::online::clear_playback_quality_override();
info!(
"player_play_tracks called: {} tracks, start_index={}, shuffle={}",
request.track_ids.len(),
@@ -2404,6 +2536,8 @@ pub async fn player_play_tracks(
// Transform to MediaItem with frontend-compatible fields
let primary_image_tag_for_url = track.primary_image_tag.clone();
let media_item = MediaItem {
// Audio and direct-URL items never negotiate a transport.
transport: None,
id: track.id.clone(),
title: track.name.clone(),
name: Some(track.name.clone()), // Frontend compatibility
@@ -3210,6 +3344,8 @@ mod tests {
let db = DatabaseWrapper(Mutex::new(database));
let make_item = |id: &str| MediaItem {
// Audio and direct-URL items never negotiate a transport.
transport: None,
id: id.to_string(),
title: id.to_string(),
name: None,
+4
View File
@@ -198,6 +198,8 @@ pub async fn player_add_track_by_id(
// Build MediaItem with artwork URL from repository and frontend-compatible fields
let primary_image_tag_for_url = track.primary_image_tag.clone();
let media_item = MediaItem {
// Audio and direct-URL items never negotiate a transport.
transport: None,
id: track.id.clone(),
title: track.name.clone(),
name: Some(track.name.clone()), // Frontend compatibility
@@ -317,6 +319,8 @@ pub async fn player_add_tracks_by_ids(
// Build MediaItem with artwork URL from repository and frontend-compatible fields
let primary_image_tag_for_url = track.primary_image_tag.clone();
let media_item = MediaItem {
// Audio and direct-URL items never negotiate a transport.
transport: None,
id: track.id.clone(),
title: track.name.clone(),
name: Some(track.name.clone()), // Frontend compatibility