dtourolle/jellytau
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases 37 Wiki Activity

feat(player): native video on Linux, and one contract for every player (v0.11.0)

Browse Source 
mpv now decodes video on Linux, drawn into a framebuffer we own and blitted
into the default vbox's draw handler. Tauri's widget tree is untouched, so an
upgrade that assumes its own layout cannot invalidate this. Direct play means
the original file, hardware decoding, and no server transcode at all — where
previously every desktop video was re-encoded to h264 for the browser engine,
whatever the file actually was. Off by default: JELLYTAU_NATIVE_VIDEO=1.

That settles finding 2 of playback-backend-unification.md — "native video
cannot be composited with a Tauri webview" — by demonstration rather than
argument, on X11 and Wayland both.

Turning it on exposed nine defects, none of them mpv's. Each was the same
mistake in a different place: a capability written down as a compile-time fact
about the platform, or a state asserted instead of confirmed.

  DR-238/246  a seek routed by the stream's container rather than by what the
              engine could do with it - correct only while one player handled
              those streams, silent the moment another did
  DR-239      a property handled but never observed, so the play/pause button
              waited for an event that could not arrive
  DR-240      fullscreen expanding the document while the window stayed put
  DR-241      a seek issued before the engine had a file, failed, and discarded
              - which is why resume began at zero
  DR-247      a Linux-only gate outliving the caller that made it Linux-only,
              breaking the Android build outright
  DR-250      a stop aimed at whichever renderer bookkeeping believed was in
              charge, missing the one actually making sound
  DR-251      a duration of zero believed, leaving the seek bar no scale
  DR-252      a junk float converted to a Duration, panicking the backend the
              instant a length-less stream appeared

So the MediaPlayer contract (DR-242 … DR-247): `open` carries a start position,
so no caller sequences load-then-seek and none can race an engine's load;
`seek` states a destination and leaves in-place-versus-re-open to the engine;
`snapshot` is one coherent read; and `Phase::Opening` names the window where
intent used to be lost. One conformance suite runs against every engine —
FakePlayer and mpv under cargo test, ExoPlayer instrumented on a device — so an
engine is either correct or visibly failing.

Two of the nine were introduced during this work and caught on hardware, not by
any suite: an over-broad capability that grouped ExoPlayer with mpv, and the
Duration panic. The suites test engines that behave. That is recorded in
docs/native-player-verification.md, which asks for the exact action sequences
that found them.

Verified: all automated gates, conformance (mpv 9/9, legacy 8/9 by design,
ExoPlayer 7/7 on device), and manual desktop and Android passes on real
hardware.

Known open and deliberately shipped: resume reads local progress and never the
server's; the background-audio handoff still declares a state swap it does not
confirm (the symptom is now impossible, the race is not); and `bun run
android:dev` builds an APK carrying the release application id, whose failure
message advises an uninstall that would destroy app data. Fix that last one
before anyone else builds for Android.

Squashed from worktree-linux-native-video, which keeps the per-defect history.
This commit is contained in:
dtourolle
2026-08-23 10:51:45 +02:00
parent 5fede123e7
commit 11d9d760d8
87 changed files with 15968 additions and 7508 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/lib/api/bindings.ts
+266 -15
View File
@@ -39,7 +39,7 @@ async playerPlayItem(item: PlayItemRequest) : Promise<PlayerStatus> {
* playing there is nothing to pause, and an error would make the frontend
* handle a case that is not a failure.
*
* TRACES: UR-040, UR-041 | DR-224 | UT-211
* TRACES: UR-040, UR-041 | DR-225 | UT-212
*/
async playerBackgroundAction(backgroundAudioArmed: boolean, inPictureInPicture: boolean) : Promise<BackgroundAction> {
return await TAURI_INVOKE("player_background_action", { backgroundAudioArmed, inPictureInPicture });
@@ -120,7 +120,7 @@ async playerSeek(position: number) : Promise<PlayerStatus> {
*
* This command analyzes the current video stream and automatically chooses
* the best seeking strategy:
* - HLS streams (.m3u8): Use native seeking
* - HLS streams: Use native seeking
* - Direct play streams: Use native seeking
* - Transcoded non-HLS: Request new stream URL from server starting at seek position
*
@@ -263,13 +263,17 @@ async playerGetStreamingQualities() : Promise<([StreamingQuality, string, string
* two-sided split: HTML5 gets the URL back and reloads its own element, while a
* native backend is reloaded here.
*
* The change applies to this playback *and* to everything started afterwards
* (it sets the process-wide ceiling), but it is deliberately **not** persisted:
* the in-player picker is a "this film, this connection" control, and the
* durable default belongs to Settings. `player_set_video_settings` is the one
* that writes to the database.
* The change applies to **this playback only**. The in-player picker is a
* "this film, this connection" control and its doc has always said so, but it
* used to be implemented by writing the process-wide ceiling — so choosing
* 2 Mbps to get one awkward film moving silently capped every video played
* afterwards for the rest of the process, with the Settings screen still
* showing the old value and nothing in the UI admitting the change. It now
* sets a per-playback override that the next item clears; the durable default
* belongs to Settings, and `player_set_video_settings` is the one that writes
* to the database.
*
* TRACES: UR-074 | DR-162
* TRACES: UR-074, UR-079 | DR-162, DR-226
*/
async playerSetStreamQuality(repositoryHandle: string, quality: StreamingQuality, useHtml5: boolean, currentPosition: number | null, mediaSourceId: string | null, audioStreamIndex: number | null) : Promise<StreamQualityResponse> {
return await TAURI_INVOKE("player_set_stream_quality", { repositoryHandle, quality, useHtml5, currentPosition, mediaSourceId, audioStreamIndex });
@@ -1033,6 +1037,22 @@ async markDownloadFailed(downloadId: number, errorMessage: string) : Promise<nul
async mediaLocalUrl(path: string) : Promise<string> {
return await TAURI_INVOKE("media_local_url", { path });
},
/**
* The stream selection for a downloaded file.
*
* The local-playback counterpart to `repository_get_stream_selection`. A file
* on disk needs no negotiation — it is a direct play over a local transport,
* with no quality ladder, because nothing about it can be re-negotiated — but
* the *frontend must not be the one to say so*. It gets the same
* [`StreamSelection`] shape as a streamed source so the player has one contract
* to consume rather than two, and so no caller has to infer a transport from a
* loopback URL.
*
* TRACES: UR-071, UR-079 | DR-225
*/
async mediaLocalSelection(path: string) : Promise<StreamSelection> {
return await TAURI_INVOKE("media_local_selection", { path });
},
/**
* Start downloading a file immediately
* This command actually downloads the file using the worker
@@ -1618,6 +1638,24 @@ async repositoryGetPlaybackInfo(handle: string, itemId: string) : Promise<Playba
async repositoryGetVideoStreamUrl(handle: string, itemId: string, mediaSourceId: string | null, audioStreamIndex: number | null) : Promise<string> {
return await TAURI_INVOKE("repository_get_video_stream_url", { handle, itemId, mediaSourceId, audioStreamIndex });
},
/**
* Decide what stream to play for a video, and describe it.
*
* Replaces `repository_get_video_stream_url` for playback. The returned
* [`StreamSelection`] carries the transport explicitly, so the frontend picks
* its loader from a tagged enum instead of testing the URL for `.m3u8`; and it
* carries the quality ladder as it applies to *this* source, so the picker can
* stop offering rungs that produce the same bytes as Original.
*
* No start-position parameter, for the same reason as the URL builder: a
* position on an HLS playlist is copied onto every segment URI and the server
* rejects each with `400` (DR-181). Callers resume by seeking after load.
*
* TRACES: UR-070, UR-079 | DR-225, DR-227, DR-228 | UT-213
*/
async repositoryGetStreamSelection(handle: string, itemId: string, mediaSourceId: string | null, audioStreamIndex: number | null) : Promise<StreamSelection> {
return await TAURI_INVOKE("repository_get_stream_selection", { handle, itemId, mediaSourceId, audioStreamIndex });
},
/**
* Get audio stream URL for a track
*/
@@ -1965,7 +2003,7 @@ export type AudioTrackSwitchResponse =
/**
* HTML5 needs to reload stream with new audio track
*/
{ strategy: "reloadStream"; new_url: string; position: number }
{ strategy: "reloadStream"; selection: StreamSelection; position: number }
/**
* Authentication result
*/
@@ -2329,7 +2367,18 @@ excludedItemIds?: string[] }
* streamed; the server returns a transcoding URL (already absolute) plus a
* `live_stream_id` that can later be used to close the stream.
*/
export type LiveStreamInfo = { streamUrl: string; playSessionId: string | null; liveStreamId: string | null; mediaSourceId: string | null }
export type LiveStreamInfo = { streamUrl: string; playSessionId: string | null; liveStreamId: string | null; mediaSourceId: string | null;
/**
* How to open `stream_url`.
*
* A live channel is always an HLS transcode — the server has to repackage a
* broadcast mux into something a browser can play, and there is no static
* file to direct-play. Saying so here means the player page never has to
* work it out from the URL, which is the whole of DR-225.
*
* TRACES: UR-079 | DR-225
*/
transport: Transport }
/**
* An LMS multi-room sync group, as returned by JellyLMS `/JellyLms/SyncGroups`.
*
@@ -2568,6 +2617,18 @@ videoCodec: string;
* Whether the video requires server-side transcoding
*/
needsTranscoding: boolean;
/**
* How this item's stream is fetched, as the backend decided it.
*
* Carried on the queue item so a later seek/reload does not have to guess.
* `None` for items queued by a path that never negotiated (audio tracks,
* direct URLs) and for anything queued before this field existed, where the
* caller falls back to `needs_transcoding` — every transcode this app
* requests is HLS (DR-140), so that fallback is exact rather than a guess.
*
* TRACES: UR-003, UR-004, UR-079 | DR-225, DR-230
*/
transport?: Transport | null;
/**
* Optional now-playing metadata. Used by the background-audio handoff so the
* lockscreen/miniplayer show the item (title/subtitle/artwork). Defaulted so
@@ -2680,6 +2741,32 @@ supportsNativeVideo: boolean }
* Playback information
*/
export type PlaybackInfo = { mediaSourceId: string; playSessionId: string; streamUrl: string; directPlay: boolean; needsTranscoding: boolean }
/**
* What the server is doing to the source to produce this stream.
*
* Distinct from [`Transport`] because the two are genuinely independent: a
* direct-streamed remux and a transcode can both arrive over HLS, and a direct
* play can arrive progressively or as a local file. Keeping them apart is what
* lets the UI say "this is not costing the server anything" without inferring
* it from a URL shape.
*
* TRACES: UR-079 | DR-228
*/
export type PlaybackKind =
/**
* The source file is served untouched. No server CPU, no quality loss.
*/
{ type: "directPlay" } |
/**
* The container is repackaged but the codecs are copied — cheap, and
* visually identical to the source.
*/
{ type: "directStream" } |
/**
* The server is re-encoding. The only case where a bitrate ceiling can
* actually be honoured, and the only one that costs the server real work.
*/
{ type: "transcode" }
/**
* Playback mode - local device, remote session, or idle
*/
@@ -2779,6 +2866,18 @@ videoCodec?: string | null;
* Whether the video requires server-side transcoding
*/
needsTranscoding?: boolean;
/**
* How this item's stream is fetched, as the backend decided it.
*
* Carried on the queue item so a later seek/reload does not have to guess.
* `None` for items queued by a path that never negotiated (audio tracks,
* direct URLs) and for anything queued before this field existed, where the
* caller falls back to `needs_transcoding` — every transcode this app
* requests is HLS (DR-140), so that fallback is exact rather than a guess.
*
* TRACES: UR-003, UR-004, UR-079 | DR-225, DR-230
*/
transport?: Transport | null;
/**
* Video width in pixels
*/
@@ -3061,6 +3160,38 @@ alreadyDownloaded: number;
* Number of tracks skipped (no jellyfin ID or other reasons)
*/
skipped: number }
/**
* One rung of the quality picker, as it applies to *this* media source.
*
* The picker used to be filled from the fixed [`StreamingQuality::ALL`] ladder,
* which meant offering "20 Mbps" for a 1.1 Mbps podcast — eight rungs, six of
* them indistinguishable from Original. `exceeds_source` is what lets the
* frontend render that honestly without knowing anything about bitrates.
*
* TRACES: UR-070, UR-079 | DR-227, DR-121
*/
export type QualityOption = { quality: StreamingQuality;
/**
* Human label ("8 Mbps"). Lives in Rust beside the number it describes.
*/
label: string;
/**
* Secondary line ("1080p").
*/
detail: string;
/**
* True when this rung's ceiling is at or above what the source itself
* carries, so selecting it yields the same stream as `Original`.
*
* The frontend renders these differently (or hides them); it does not
* decide which they are.
*/
exceedsSource: boolean;
/**
* The source's own bitrate, when the server reported one. Presentation
* only — the picker shows "Original (6.7 Mbps)" rather than a bare word.
*/
sourceBitrate: number | null }
/**
* Response for queue queries
*/
@@ -3069,6 +3200,36 @@ export type QueueStatus = { items: PlayerMediaItem[]; currentIndex: number | nul
* Remote session status for UI updates
*/
export type RemoteSessionStatus = { position: number; duration: number | null; isPlaying: boolean; nowPlayingItem: NowPlayingItem | null }
/**
* The rendition actually negotiated — what the viewer is receiving right now.
*
* `None` on a [`StreamSelection`] when the source is being direct-played as-is:
* there is no *chosen* rendition in that case, only the file itself, and
* reporting the ceiling that happened to be set would misdescribe it.
*
* TRACES: UR-079 | DR-225, DR-226
*/
export type Rendition = {
/**
* The rung of the ladder this stream was built against.
*/
quality: StreamingQuality;
/**
* Total bits per second the stream may use, when a ceiling applies.
*/
maxBitrate: number | null;
/**
* Resolution ceiling, when one applies. `None` preserves the source's.
*/
maxHeight: number | null;
/**
* Video codec the server was asked to produce.
*/
videoCodec: string | null;
/**
* Audio codec the server was asked to produce.
*/
audioCodec: string | null }
/**
* Repeat mode for the queue
*
@@ -3182,13 +3343,73 @@ export type StreamKind = "audio" | "video" | "subtitle" |
*/
export type StreamQualityResponse =
/**
* The native backend was reloaded here; nothing left for the frontend.
* The native backend was reloaded here; nothing left for the frontend to
* *do* — but it still has to be told what was negotiated.
*
* This carried only a position at first, which left the picker on Android
* pinned to the rendition of the *first* stream: the UI derives the rung in
* force from the selection it holds, nothing replaced that selection on the
* native path, and a transcode always has a rendition — so the fallback
* that would have used the requested value was never reached. The stream
* changed and the menu did not.
*
* TRACES: UR-074, UR-079 | DR-226, DR-227
*/
{ strategy: "native"; position: number } |
{ strategy: "native"; selection: StreamSelection; position: number } |
/**
* HTML5 must reload its element with this URL.
* HTML5 must reload its element with this selection.
*/
{ strategy: "reloadStream"; new_url: string; position: number }
{ strategy: "reloadStream"; selection: StreamSelection; position: number }
/**
* Everything a player backend needs to open a stream, and everything the UI
* needs to describe it.
*
* Replaces the bare `String` URL that `get_video_stream_url` used to return.
*
* TRACES: UR-079 | DR-225, DR-227, DR-228
*/
export type StreamSelection = {
/**
* The URL (or loopback URL) to open.
*/
url: string;
/**
* How to fetch it. Replaces the `.m3u8` substring check.
*/
transport: Transport;
/**
* What the server is doing to the source to produce it.
*/
playbackKind: PlaybackKind;
/**
* The negotiated rendition; `None` when direct-playing the source as-is.
*/
rendition: Rendition | null;
/**
* What this media source can offer, for the quality picker (DR-227).
*/
available: QualityOption[];
/**
* The media source this selection is for, so a later re-open (quality
* change, audio-track switch, transcoded seek) targets the same one.
*/
mediaSourceId: string | null;
/**
* The transcode identity the server keyed this job by, when there is one.
*/
playSessionId: string | null;
/**
* Whether the server is spending encoder time on this stream.
*
* Derived from [`playback_kind`](Self::playback_kind) rather than left for
* the frontend to compute: "which kinds count as transcoding" is a domain
* rule, and a direct *stream* is a remux that must not be counted. The
* queue's long-standing `needs_transcoding` flag and the seek strategy both
* read this, so there is one answer rather than three.
*
* TRACES: UR-079 | DR-225, DR-228
*/
needsTranscoding: boolean }
/**
* A ceiling on how much bandwidth a *video* stream may consume.
*
@@ -3269,6 +3490,36 @@ itemName: string | null }
* Statistics about the thumbnail cache
*/
export type ThumbnailCacheStats = { totalSizeBytes: number; itemCount: number; limitBytes: number }
/**
* How the bytes of a chosen stream are fetched.
*
* This field exists to delete a substring search. The frontend previously
* decided which loader to attach by testing `url.contains(".m3u8")`, which is a
* domain fact reconstructed in the presentation layer — the same class of leak
* as the item-type taxonomy that `check:boundary` guards, and one that breaks
* silently the moment a server serves a playlist from a path that does not end
* in `.m3u8`, or serves a progressive file from one that does.
*
* Tagged (`{"type":"hls"}`) rather than a bare string so the frontend matches a
* discriminant instead of comparing text.
*
* TRACES: UR-079 | DR-225
*/
export type Transport =
/**
* An HLS playlist. The webview attaches hls.js (or Safari's native loader);
* ExoPlayer uses its HLS media source.
*/
{ type: "hls" } |
/**
* A single progressive HTTP resource, seekable by byte range.
*/
{ type: "progressive" } |
/**
* A file already on disk — a completed download, or the loopback media
* server standing in front of one.
*/
{ type: "localFile" }
/**
* User information
*/
@@ -3316,7 +3567,7 @@ export type VideoSeekResponse =
/**
* Reload stream from new position (transcoded non-HLS)
*/
{ strategy: "reloadStream"; new_url: string; seek_offset: number }
{ strategy: "reloadStream"; selection: StreamSelection; seek_offset: number }
/**
* Video playback settings
*/