dtourolle/jellytau
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 40 Wiki Activity

feat(player): native video on Linux, and one contract for every player (v0.11.0)

Browse Source 
mpv now decodes video on Linux, drawn into a framebuffer we own and blitted
into the default vbox's draw handler. Tauri's widget tree is untouched, so an
upgrade that assumes its own layout cannot invalidate this. Direct play means
the original file, hardware decoding, and no server transcode at all — where
previously every desktop video was re-encoded to h264 for the browser engine,
whatever the file actually was. Off by default: JELLYTAU_NATIVE_VIDEO=1.

That settles finding 2 of playback-backend-unification.md — "native video
cannot be composited with a Tauri webview" — by demonstration rather than
argument, on X11 and Wayland both.

Turning it on exposed nine defects, none of them mpv's. Each was the same
mistake in a different place: a capability written down as a compile-time fact
about the platform, or a state asserted instead of confirmed.

  DR-238/246  a seek routed by the stream's container rather than by what the
              engine could do with it - correct only while one player handled
              those streams, silent the moment another did
  DR-239      a property handled but never observed, so the play/pause button
              waited for an event that could not arrive
  DR-240      fullscreen expanding the document while the window stayed put
  DR-241      a seek issued before the engine had a file, failed, and discarded
              - which is why resume began at zero
  DR-247      a Linux-only gate outliving the caller that made it Linux-only,
              breaking the Android build outright
  DR-250      a stop aimed at whichever renderer bookkeeping believed was in
              charge, missing the one actually making sound
  DR-251      a duration of zero believed, leaving the seek bar no scale
  DR-252      a junk float converted to a Duration, panicking the backend the
              instant a length-less stream appeared

So the MediaPlayer contract (DR-242 … DR-247): `open` carries a start position,
so no caller sequences load-then-seek and none can race an engine's load;
`seek` states a destination and leaves in-place-versus-re-open to the engine;
`snapshot` is one coherent read; and `Phase::Opening` names the window where
intent used to be lost. One conformance suite runs against every engine —
FakePlayer and mpv under cargo test, ExoPlayer instrumented on a device — so an
engine is either correct or visibly failing.

Two of the nine were introduced during this work and caught on hardware, not by
any suite: an over-broad capability that grouped ExoPlayer with mpv, and the
Duration panic. The suites test engines that behave. That is recorded in
docs/native-player-verification.md, which asks for the exact action sequences
that found them.

Verified: all automated gates, conformance (mpv 9/9, legacy 8/9 by design,
ExoPlayer 7/7 on device), and manual desktop and Android passes on real
hardware.

Known open and deliberately shipped: resume reads local progress and never the
server's; the background-audio handoff still declares a state swap it does not
confirm (the symptom is now impossible, the race is not); and `bun run
android:dev` builds an APK carrying the release application id, whose failure
message advises an uninstall that would destroy app data. Fix that last one
before anyone else builds for Android.

Squashed from worktree-linux-native-video, which keeps the per-defect history.
This commit is contained in:
dtourolle
2026-08-23 10:51:45 +02:00
parent 5fede123e7
commit 11d9d760d8
87 changed files with 15968 additions and 7508 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src-tauri/src/player/mod.rs
+187 -19
View File
@@ -5,8 +5,18 @@
pub mod autoplay;
pub mod backend;
pub mod background_policy;
#[cfg(any(test, feature = "conformance"))]
pub mod conformance;
pub mod events;
#[cfg(any(test, feature = "conformance"))]
pub mod fake_player;
#[cfg(test)]
mod fake_player_conformance;
pub mod legacy_player;
pub mod media;
pub mod media_player;
#[cfg(target_os = "linux")]
pub mod mpv_player;
pub mod queue;
pub mod seek;
pub mod session;
@@ -24,16 +34,38 @@ pub mod android;
#[cfg(target_os = "linux")]
pub mod mpv_backend;
/// Whether this process renders video natively — one answer, three consumers
/// (UR-080 / DR-231, DR-235).
pub mod native_video;
/// mpv's render API into a framebuffer we own (UR-080 / DR-231, IR-033).
///
/// Deliberately *not* GTK-gated beyond the platform that currently builds it:
/// everything here is the portable half, and Windows reuses it unchanged behind
/// its own surface.
#[cfg(target_os = "linux")]
pub mod mpv_render;
/// The native video surface mpv renders into (UR-080 / DR-231).
///
/// Linux-gated because the *surface* is GTK. Everything around it — the render
/// context, its lifetime, frame pacing, the device profile — is not.
#[cfg(target_os = "linux")]
pub mod video_surface;
// Platforms with no native audio backend (e.g. Windows) render audio-only
// playback through a webview <audio> element, mirroring how all video renders.
#[cfg(not(any(target_os = "linux", target_os = "android")))]
pub mod webview_audio_backend;
// Re-export commonly used types
use crate::repository::stream_selection::StreamSelection;
pub use autoplay::{AutoplayDecision, AutoplaySettings};
pub use backend::{NullBackend, PlayerBackend, PlayerError};
pub use events::{PlayerEventEmitter, PlayerStatusEvent, TauriEventEmitter};
pub use legacy_player::LegacyPlayer;
pub use media::{MediaItem, MediaSource, MediaType, QueueContext, SubtitleTrack};
pub use media_player::{MediaPlayer, OpenRequest, Phase};
pub use queue::{QueueManager, RepeatMode};
pub use seek::{determine_video_seek_strategy, VideoSeekStrategy};
pub use session::{MediaSessionManager, MediaSessionType};
@@ -208,7 +240,9 @@ use crate::utils::conversions::seconds_to_ticks;
/// Central player controller that coordinates playback
pub struct PlayerController {
backend: Arc<Mutex<Box<dyn PlayerBackend>>>,
/// The engine. One contract, so the controller stops branching on which
/// platform it is running on — see docs/specs/media-player-controller.md.
backend: Arc<Mutex<Box<dyn MediaPlayer>>>,
queue: Arc<Mutex<QueueManager>>,
jellyfin_client: Arc<Mutex<Option<JellyfinClient>>>,
muted: bool,
@@ -307,7 +341,7 @@ pub struct PlayerController {
impl PlayerController {
pub fn new(
backend: Box<dyn PlayerBackend>,
backend: Box<dyn MediaPlayer>,
playback_reporter: Arc<TokioMutex<Option<PlaybackReporter>>>,
position_throttler: Arc<EventThrottler>,
) -> Self {
@@ -485,7 +519,12 @@ impl PlayerController {
/// Used on platforms where video is rendered outside the native backend
/// (Linux WebKitGTK HTML5 <video>): the queue/UI state must reflect the
/// item, but MPV must not start a redundant decode for it.
#[cfg(target_os = "linux")]
///
/// Not gated to Linux. Its caller stopped being a `#[cfg]` branch and became
/// a runtime question — "does this renderer draw the picture?" — so the
/// `else` arm is compiled on every platform even where it never runs. The
/// gate outliving its caller broke the Android build outright, which went
/// unnoticed because nothing built for Android afterwards.
pub fn set_current_item(&self, item: MediaItem) -> Result<(), PlayerError> {
debug!(
"[PlayerController] set_current_item (no backend load): {}",
@@ -538,8 +577,16 @@ impl PlayerController {
*self.html5_playing.lock_safe() = None;
let mut backend = self.backend.lock_safe();
backend.load(item)?;
backend.play()?;
// One operation: the engine is handed the item and where to begin, so
// there is no window between them for a position to be lost in.
backend.open(OpenRequest::new(
item.clone(),
StreamSelection::for_queued_item(
item.playable_url(),
item.transport,
item.needs_transcoding,
),
))?;
drop(backend);
// A different item is loading; the last one's reported position must not
@@ -713,7 +760,7 @@ impl PlayerController {
return Ok(());
}
let mut backend = self.backend.lock_safe();
if backend.state().is_playing() {
if backend.snapshot().phase.is_active() {
backend.pause()
} else {
backend.play()
@@ -738,10 +785,32 @@ impl PlayerController {
let position = self.absolute_position();
let mut backend = self.backend.lock_safe();
backend.stop()?;
backend.close()?;
drop(backend);
self.clear_reported_time();
// Stopping means *nothing is playing*, from any renderer — not "the
// thing we currently believe owns playback has been asked to stop".
//
// A background-audio handoff swaps which renderer that is, and the swap
// is bookkeeping that can be mid-flight: `exit_background_audio` marks
// the webview element the player again the moment it is called, while
// the element has not reloaded yet. A stop aimed at what the flags say
// is playing therefore misses the audio stream that actually is, and it
// resurfaces in the mini player as an audio track.
//
// Clearing the handoff here is the other half of that: a stop that
// leaves the base offset and the active flag behind lets the next
// position read be interpreted against a handoff that no longer exists.
//
// TRACES: UR-040, UR-005 | DR-250
if self.is_background_audio_active() {
debug!("[PlayerController] stop: clearing an active background-audio handoff");
}
*self.background_audio_active.lock_safe() = false;
self.set_background_audio_base(0.0);
*self.html5_playing.lock_safe() = None;
if let Some(jellyfin_id) = jellyfin_id {
self.report_stopped_at(jellyfin_id, position);
}
@@ -819,7 +888,7 @@ impl PlayerController {
// If we're more than 3 seconds in, restart current track
{
let backend = self.backend.lock_safe();
if backend.position() > 3.0 {
if backend.snapshot().position.as_secs_f64() > 3.0 {
debug!("[PlayerController] previous: restarting current track (position > 3s)");
drop(backend);
return self.seek(0.0);
@@ -851,7 +920,7 @@ impl PlayerController {
/// timeline and is what every caller outside the player itself means.
pub fn seek(&self, position: f64) -> Result<(), PlayerError> {
let mut backend = self.backend.lock_safe();
backend.seek(position)
backend.seek(Duration::from_secs_f64(position.max(0.0)))
}
/// Seek to an **absolute** position on the item's own timeline.
@@ -902,23 +971,48 @@ impl PlayerController {
/// Set the active audio track by stream index
pub fn set_audio_track(&self, stream_index: i32) -> Result<(), PlayerError> {
let mut backend = self.backend.lock_safe();
backend.set_audio_track(stream_index)
backend.select_audio_track(Some(stream_index))
}
/// Set the active subtitle track by stream index (None to disable subtitles)
pub fn set_subtitle_track(&self, stream_index: Option<i32>) -> Result<(), PlayerError> {
let mut backend = self.backend.lock_safe();
backend.set_subtitle_track(stream_index)
backend.select_subtitle_track(stream_index)
}
/// Get current state
pub fn state(&self) -> PlayerState {
self.backend.lock_safe().state()
let phase = self.backend.lock_safe().snapshot().phase;
let media = self.queue.lock_safe().current().cloned();
match (phase, media) {
(Phase::Playing, Some(media)) => PlayerState::Playing {
media,
position: self.position(),
duration: self.duration().unwrap_or(0.0),
},
(Phase::Paused, Some(media)) => PlayerState::Paused {
media,
position: self.position(),
duration: self.duration().unwrap_or(0.0),
},
(Phase::Opening, Some(media)) => PlayerState::Loading { media },
(Phase::Failed(error), media) => PlayerState::Error { media, error },
// Ready without an item, or anything terminal, reads as idle: the
// queue is what says whether there is something to resume.
_ => PlayerState::Idle,
}
}
/// What the engine currently rendering can do.
///
/// TRACES: UR-081 | DR-246
pub fn capabilities(&self) -> crate::player::media_player::Capabilities {
self.backend.lock_safe().capabilities()
}
/// Get current position
pub fn position(&self) -> f64 {
self.backend.lock_safe().position()
self.backend.lock_safe().snapshot().position.as_secs_f64()
}
/// The position on the **item's own timeline**, whatever is rendering it.
@@ -944,7 +1038,7 @@ impl PlayerController {
///
/// TRACES: UR-040, UR-005, UR-025 | DR-178 | UT-176, UT-177
pub fn absolute_position(&self) -> f64 {
let native = self.backend.lock_safe().position().max(0.0);
let native = self.backend.lock_safe().snapshot().position.as_secs_f64();
let reported = self.reported_time.lock_safe().last_position();
let base = if self.is_background_audio_active() {
*self.background_audio_base.lock_safe()
@@ -1008,10 +1102,34 @@ impl PlayerController {
///
/// TRACES: UR-005 | DR-178
pub fn duration(&self) -> Option<f64> {
// Zero is not a duration, it is an engine saying it does not know yet.
//
// ExoPlayer reports `C.TIME_UNSET` until it has resolved one, and
// `JellyTauPlayer.getDuration()` maps that to `0.0` — so the engine
// answers `Some(0.0)`, every "unknown duration" fallback below is
// skipped, and the seek bar is left with no scale. That presents as
// scrubbing being broken rather than as a duration that never arrived.
//
// The item usually knows: the catalog carried a runtime long before
// anything started decoding.
//
// TRACES: UR-005, UR-040 | DR-251
let usable = |d: f64| (d > 0.0).then_some(d);
self.backend
.lock_safe()
.duration()
.or_else(|| self.observed_duration())
.snapshot()
.duration
.map(|d| d.as_secs_f64())
.and_then(usable)
.or_else(|| self.observed_duration().and_then(usable))
.or_else(|| {
self.queue
.lock_safe()
.current()
.and_then(|item| item.duration)
.and_then(usable)
})
}
/// Get queue reference
@@ -1066,7 +1184,7 @@ impl PlayerController {
/// Get current volume (0.0 - 1.0)
pub fn volume(&self) -> f32 {
self.backend.lock_safe().volume()
self.backend.lock_safe().snapshot().volume
}
/// Check if muted
@@ -1167,7 +1285,7 @@ impl PlayerController {
drop(timer);
// Stop the backend
if let Err(e) = backend.lock_safe().stop() {
if let Err(e) = backend.lock_safe().close() {
error!("[SleepTimer] Failed to stop playback: {}", e);
}
continue;
@@ -1897,6 +2015,8 @@ impl PlayerController {
.map_err(|e| format!("Failed to build audio-only URL for next episode: {}", e))?;
let media_item = MediaItem {
// Audio and direct-URL items never negotiate a transport.
transport: None,
id: next.id.clone(),
title: next.name.clone(),
name: Some(next.name.clone()),
@@ -2180,7 +2300,10 @@ impl Default for PlayerController {
let playback_reporter = Arc::new(TokioMutex::new(None));
let position_throttler = Arc::new(EventThrottler::new());
Self::new(
Box::new(NullBackend::new()),
Box::new(LegacyPlayer::new(
NullBackend::new(),
crate::player::media_player::Capabilities::mpv(),
)),
playback_reporter,
position_throttler,
)
@@ -2189,6 +2312,35 @@ impl Default for PlayerController {
#[cfg(test)]
mod tests {
/// A duration the engine does not know must fall back to the one the item
/// carries, and zero must count as "does not know".
///
/// ExoPlayer reports `C.TIME_UNSET` for a duration it has not resolved;
/// `JellyTauPlayer.getDuration()` maps that to `0.0`, so the engine answers
/// `Some(0.0)` rather than `None` and every "unknown duration" fallback is
/// skipped. The seek bar then has no scale, which presents as scrubbing
/// being dead rather than as a missing duration.
///
/// TRACES: UR-005, UR-040 | DR-251 | UT-221
#[test]
fn test_duration_falls_back_to_the_item_when_the_engine_does_not_know() {
let controller = PlayerController::default();
let mut item = MediaItem::sample("item-1", "https://example.invalid/a.mp4");
item.duration = Some(1800.0);
{
let queue_arc = controller.queue();
let mut queue = queue_arc.lock_safe();
queue.set_queue(vec![item], 0);
}
assert_eq!(
controller.duration(),
Some(1800.0),
"an engine that cannot report a duration should not erase the one the item carries"
);
}
use super::*;
/// Test emitter that captures events for asserting the HTML5 report methods
@@ -2580,6 +2732,8 @@ mod tests {
fn create_test_items(count: usize) -> Vec<MediaItem> {
(0..count)
.map(|i| MediaItem {
// Audio and direct-URL items never negotiate a transport.
transport: None,
id: format!("item_{}", i),
title: format!("Track {}", i + 1),
name: Some(format!("Track {}", i + 1)),
@@ -3792,6 +3946,8 @@ mod tests {
// Queue holds the episode that just finished playing
let episode = MediaItem {
// Audio and direct-URL items never negotiate a transport.
transport: None,
media_type: MediaType::Video,
source: MediaSource::Remote {
stream_url: "http://example.com/ep1.mkv".to_string(),
@@ -3827,6 +3983,8 @@ mod tests {
// Mirrors what player_enter_background_audio builds: the episode as AUDIO.
let episode = MediaItem {
// Audio and direct-URL items never negotiate a transport.
transport: None,
item_type: Some("Episode".to_string()),
media_type: MediaType::Audio, // audio-only handoff, not Video
series_id: Some("series1".to_string()),
@@ -3943,6 +4101,8 @@ mod tests {
// Currently playing: ep2 handed off to audio-only background playback.
let episode = MediaItem {
// Audio and direct-URL items never negotiate a transport.
transport: None,
id: "ep2".to_string(),
item_type: Some("Episode".to_string()),
media_type: MediaType::Audio,
@@ -3978,6 +4138,8 @@ mod tests {
/// URL carrying the handoff position.
fn audio_only_episode(runtime_seconds: f64) -> MediaItem {
MediaItem {
// Audio and direct-URL items never negotiate a transport.
transport: None,
id: "ep2".to_string(),
item_type: Some("Episode".to_string()),
media_type: MediaType::Audio,
@@ -3998,6 +4160,8 @@ mod tests {
/// handoff point.
fn local_audio_only_episode(runtime_seconds: f64) -> MediaItem {
MediaItem {
// Audio and direct-URL items never negotiate a transport.
transport: None,
source: MediaSource::Local {
file_path: std::path::PathBuf::from("/downloads/ep2.mkv"),
jellyfin_item_id: Some("ep2".to_string()),
@@ -4682,6 +4846,8 @@ mod tests {
controller.set_repository(Arc::new(MockEpisodeRepo::season(3)));
let episode = MediaItem {
// Audio and direct-URL items never negotiate a transport.
transport: None,
id: "ep2".to_string(),
item_type: Some("Episode".to_string()),
media_type: MediaType::Video,
@@ -4717,6 +4883,8 @@ mod tests {
let controller = PlayerController::default();
let episode = MediaItem {
// Audio and direct-URL items never negotiate a transport.
transport: None,
media_type: MediaType::Video,
source: MediaSource::Remote {
stream_url: "http://example.com/ep1.mkv".to_string(),