dtourolle/jellytau
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases 38 Wiki Activity

feat(playback): let Rust decide what stream to play, and say so

Browse Source 
Playing a video meant asking the server to re-encode it, always. That
decision was made nowhere and written down nowhere, so whoever needed it
re-derived it downstream — the player worked out whether it had been handed
a playlist by looking for ".m3u8" in the URL, in two places. A viewer paid
for a transcode of a file their device could have played untouched, and the
app could not tell them which it was.

One negotiation now produces one self-describing StreamSelection — direct
play, remux or transcode; over a playlist, a plain HTTP file, or a local one
— and every renderer consumes that same answer.

Measured against the development server (Jellyfin 10.11.5), 400 items
sampled for codec mix and 40 put through a real PlaybackInfo negotiation
per profile:

  Linux / WebKitGTK (h264 only, 2ch)          3/40 —  7% direct play
  Android / ExoPlayer (hevc, ac3/eac3, 6ch)  34/40 — 85% direct play

The library is ~80% hevc, which is why the two diverge so hard. The payoff
is overwhelmingly Android, where 85% of plays were starting a transcode
nobody needed. Linux stays near 7% until libmpv decodes the picture — the
h264-only profile is a WebKitGTK constraint, not a JellyTau choice.

DR-219  StreamSelection: url + tagged Transport (hls/progressive/localFile)
        + PlaybackKind (directPlay/directStream/transcode) + the negotiated
        rendition + this source's ladder + a needs_transcoding flag derived
        in Rust so the rule is answered once. Both enums are serde-tagged
        so the frontend matches a discriminant, not a substring. The paths
        that never negotiate get the same shape from Rust rather than
        assembling one — media_local_selection for a downloaded file,
        LiveStreamInfo.transport for a live channel — so there is no second
        place where a transport is decided.

DR-220  The ceiling becomes two levels: a durable device default (Settings,
        persisted) and a per-playback override the in-player picker sets.
        The picker had called itself a "this film, this connection" control
        since it was written but wrote the process-wide default, so dropping
        one awkward film to 2 Mbps silently capped every video played
        afterwards for the rest of the process, with Settings still showing
        the old value. The override is cleared whenever playback moves to a
        new item, which stops it surviving into an autoplayed next episode.
        effective_streaming_quality() is the single resolution point.

DR-221  The quality picker is filled from what this media source can offer.
        Rust marks a rung exceeds_source when its ceiling is at or above the
        source's own bitrate — such a rung is another way to spell Original
        — and the frontend does not draw those. Original is never marked; a
        source whose bitrate the server does not report marks nothing, which
        keeps every rung offered.

DR-222  Direct play and direct stream are negotiated, with two client-side
        overrides on top because the server's answer is right about the file
        and wrong about what this app will do with it: undecodable audio
        (Jellyfin 10.11.5 honours a DirectPlayProfile's container and video
        codec but ignores its audio codec, so it offers direct play for an
        E-AC-3 track the webview renders in silence) and a viewer-pinned
        audio track the file does not default to. A direct stream is a remux
        and is deliberately not counted as transcoding.

DR-223  Dropped on measurement, not deferred. A master playlist from this
        server carries exactly one EXT-X-STREAM-INF: Jellyfin builds it from
        the single rendition the request asked for rather than publishing a
        ladder. So there is no adaptation for hls.js to be preserving and
        none mpv would lose — the claim that there was, in
        playback-backend-unification.md, does not hold. Recorded rather than
        deleted because it is a measurement: a server that does publish a
        ladder would change the answer.

DR-224  Every backend consumes the same selection. The queue item carries
        the transport, so player_seek_video picks its seek strategy from the
        backend's decision instead of the last stream_url.contains(".m3u8")
        in the codebase. Items queued by a path that never negotiated carry
        None and fall back to needs_transcoding, which is exact rather than
        a guess because every transcode this app requests is HLS (DR-140).

The frontend loader decision moves to streamTransport.ts so it can be
tested: the two cases that pin it are the ones that failed against the old
implementation — a progressive stream whose URL contains ".m3u8" must not
get an HLS loader, and an HLS stream whose URL contains none must.

Also verified the URL the direct-play branch builds actually serves playable
bytes: 206, video/mp4, valid ISO-BMFF, and a mid-file range works, so
seeking a direct play works.

The spec is folded into docs/architecture/{01,02,03} and deleted, per the
rule that docs/specs holds only work that has not shipped. DR-121 leaves
read-through-media-cache.md with a pointer; that spec keeps its capture half.

Not verified: real playback on a device. Direct play changes what actually
gets played, and neither fixtures nor curl prove the WebKitGTK and ExoPlayer
paths render it.
This commit is contained in:
dtourolle
2026-08-22 13:45:03 +02:00
parent 5fede123e7
commit 109700b949
45 changed files with 9744 additions and 6581 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/lib/components/player/VideoPlayer.svelte
+172 -56
View File
@@ -4,7 +4,12 @@
import { get } from "svelte/store";
import { goto } from "$app/navigation";
import { commands } from "$lib/api/bindings";
import type { JRayActor, StreamingQuality, BackgroundAction } from "$lib/api/bindings";
import type {
JRayActor,
StreamingQuality,
BackgroundAction,
StreamSelection,
} from "$lib/api/bindings";
import { listen } from "@tauri-apps/api/event";
import Hls from "hls.js";
import type { MediaItem } from "$lib/api/types";
@@ -77,12 +82,21 @@
type BackgroundAudioState,
} from "./backgroundAudioHandoff";
import { createLogger } from "$lib/utils/logger";
import { elementSrcFor, videoLoaderFor } from "$lib/player/streamTransport";
const log = createLogger("VideoPlayer");
interface Props {
media: MediaItem | null;
streamUrl: string;
/**
* What to play, as the backend decided it: URL, transport, playback kind and
* the quality ladder for this source. Replaces the bare `streamUrl` string,
* which forced this component to re-derive the transport by searching for
* `.m3u8`.
*
* TRACES: UR-079 | DR-224, DR-226
*/
selection: StreamSelection;
mediaSourceId?: string; // Media source ID for subtitle URLs
initialPosition?: number; // Position in seconds to seek to after load (for resume)
needsTranscoding?: boolean; // Whether content needs transcoding (HEVC/10-bit) - affects seeking behavior
@@ -103,7 +117,7 @@
let {
media,
streamUrl,
selection,
mediaSourceId,
initialPosition,
needsTranscoding = false,
@@ -179,7 +193,12 @@
// Capture only the initial streamUrl prop; later prop changes are applied via
// the $effect below (untrack keeps this a one-time snapshot, matching
// reportMediaId above and silencing state_referenced_locally).
let currentStreamUrl = $state(untrack(() => streamUrl));
// The selection currently loaded. Starts from the prop and is replaced
// wholesale by a reload (quality change, audio-track switch, transcoded seek)
// so transport and URL can never disagree.
// TRACES: UR-079 | DR-224
let currentSelection = $state<StreamSelection>(untrack(() => selection));
const currentStreamUrl = $derived(currentSelection.url);
let hasReportedStart = $state(false);
let progressInterval: ReturnType<typeof setInterval> | null = null;
let isMediaReady = $state(false); // Track if media is ready to play (implements Loading state from DR-001)
@@ -250,14 +269,31 @@
}
}
/**
* A selection identical to the one loaded, but pointing at a different URL.
*
* Used by the paths that swap the stream without re-negotiating — the
* background-audio handoff and its return. Each states the transport it is
* moving to rather than letting it be inferred, which is the whole point of
* DR-224: the audio handoff really is a progressive mp3, and the rebuilt
* video stream really is an HLS transcode, and neither is knowable from the
* URL text.
*
* TRACES: UR-040, UR-079 | DR-224
*/
function selectionAt(url: string, transport: StreamSelection["transport"]): StreamSelection {
// A re-opened stream is a new transcode job; the old session id is stale.
return { ...currentSelection, url, transport, playSessionId: null };
}
const adapterBridge: Html5ElementBridge = {
getElement: () => videoElement,
getSeekOffset: () => seekOffset,
setSeekOffset: (o) => {
seekOffset = o;
},
setStreamUrl: (u) => {
currentStreamUrl = u;
setStreamSelection: (sel) => {
currentSelection = sel;
},
destroyHls: tearDownHls,
getMediaSourceId: () => mediaSourceId ?? null,
@@ -275,9 +311,46 @@
// Rust — the frontend never encodes what a step means.
// TRACES: UR-074 | DR-162
let showQualityMenu = $state(false);
let streamingQualities = $state<[StreamingQuality, string, string][]>([]);
let selectedQuality = $state<StreamingQuality>("original");
let changingQuality = $state(false);
/**
* The device's durable default, shown when the stream is a direct play and so
* has no rendition of its own to report. Read once from Settings.
*/
let defaultQuality = $state<StreamingQuality>("original");
/**
* The rungs to offer for the stream that is playing, straight from the
* backend (DR-226). Rungs whose ceiling is at or above the source bitrate are
* dropped: they produce the same bytes as Original, so listing five of them is
* five ways to spell one choice. Rust decides which those are — this only
* decides not to draw them.
*
* `Original` is always kept; it is the source, never redundant with it.
*
* TRACES: UR-070, UR-079 | DR-226, DR-121
*/
const qualityOptions = $derived(
currentSelection.available.filter((o) => !o.exceedsSource || o.quality === "original"),
);
/**
* The rung in force. A transcode reports the rendition it was built against;
* a direct play has none, because it *is* the source — so it reads as
* Original rather than as whatever ceiling happens to be set.
*/
const selectedQuality = $derived<StreamingQuality>(
currentSelection.rendition?.quality ??
(currentSelection.playbackKind.type === "transcode" ? defaultQuality : "original"),
);
/** Human line for what the server is doing with this stream. */
const playbackKindLabel = $derived(
currentSelection.playbackKind.type === "directPlay"
? "Direct play — the original file"
: currentSelection.playbackKind.type === "directStream"
? "Direct stream — repackaged, not re-encoded"
: "Transcoding on the server",
);
// Track duration from video element (for when media item doesn't have runTimeTicks)
let videoDuration = $state(0);
@@ -450,9 +523,9 @@
// Update stream URL when prop changes (from parent component, not from internal seeks)
$effect(() => {
// Only reset when the streamUrl prop actually changes from parent
if (streamUrl !== lastStreamUrlProp) {
lastStreamUrlProp = streamUrl;
currentStreamUrl = streamUrl;
if (selection.url !== lastStreamUrlProp) {
lastStreamUrlProp = selection.url;
currentSelection = selection;
seekOffset = 0;
isMediaReady = false; // Reset to loading state when stream URL changes
hasPerformedInitialSeek = false; // Reset so new video can seek to initial position
@@ -567,9 +640,14 @@
return;
}
const isHlsStream = currentStreamUrl.includes(".m3u8");
// The loader comes from the backend's tagged transport, never from the URL.
// TRACES: UR-079 | DR-224 | UT-213
const loader = videoLoaderFor(currentSelection, {
hlsJsSupported: Hls.isSupported(),
nativeHlsSupported: !!videoElement.canPlayType("application/vnd.apple.mpegurl"),
});
if (isHlsStream && Hls.isSupported()) {
if (loader === "hlsjs") {
// Clean up existing HLS instance if any - CRITICAL for preventing dual audio
if (hls) {
log.debug("Cleaning up existing HLS instance");
@@ -724,13 +802,13 @@
videoElement.pause();
}
};
} else if (isHlsStream && videoElement.canPlayType("application/vnd.apple.mpegurl")) {
// Native HLS support (Safari)
} else if (loader === "nativeHls") {
// The element parses the playlist itself (Safari/WebKit).
log.debug("Using native HLS support");
videoElement.src = currentStreamUrl;
} else {
// Not an HLS stream, use regular video element
log.debug("Using regular video element for non-HLS stream");
// Progressive or local: the element loads the URL directly.
log.debug("Using regular video element", currentSelection.transport.type);
}
});
@@ -801,21 +879,25 @@
});
});
// Populate the quality menu. Deliberately its own *synchronous* onMount that
// fires the load without awaiting it: an await inside the main onMount below
// flips the component into HTML5 mode and breaks native seeking, and nothing
// about playback waits on this list.
// The quality *ladder* now arrives with the stream selection (DR-226), so all
// this still needs is the device default, for the case where the stream is a
// direct play and has no rendition of its own.
//
// TRACES: UR-074 | DR-162
// Deliberately its own *synchronous* onMount that fires the load without
// awaiting it: an await inside the main onMount below flips the component into
// HTML5 mode and breaks native seeking, and nothing about playback waits on
// this value.
//
// TRACES: UR-074, UR-079 | DR-162, DR-226
onMount(() => {
Promise.all([commands.playerGetStreamingQualities(), commands.playerGetVideoSettings()])
.then(([qualities, settings]) => {
streamingQualities = qualities;
commands
.playerGetVideoSettings()
.then((settings) => {
// Optional on the wire (serde default) — absent means uncapped.
selectedQuality = settings.streamingQuality ?? "original";
defaultQuality = settings.streamingQuality ?? "original";
})
.catch((err) => {
log.warn("Failed to load streaming qualities:", err);
log.warn("Failed to load the default streaming quality:", err);
});
});
@@ -878,6 +960,9 @@
id: media.id,
videoCodec: needsTranscoding ? "hevc" : "h264",
needsTranscoding: needsTranscoding,
// Carry the negotiated transport onto the queue item so a later seek
// reads it instead of falling back. TRACES: UR-079 | DR-229
transport: currentSelection.transport,
// Order matters: player_set_subtitle_track(n) is a position in this
// array. Previously this array was built and then dropped, so
// ExoPlayer got a MediaItem with no subtitles at all.
@@ -944,7 +1029,9 @@
const host = createRustReportHost(media.id, {
onEnded: () => notifyEnded(),
onStreamUrlChanged: (u) => {
currentStreamUrl = u;
// Rust re-opened the same stream (a transcoded seek): the
// transport is unchanged, only the job behind it.
currentSelection = selectionAt(u, currentSelection.transport);
},
});
playerAdapter = createAdapter({
@@ -1885,8 +1972,9 @@
pendingForegroundPlay = plan.shouldPlay;
// Determine the target URL + how the element/offset should be positioned.
let targetUrl: string;
// Determine the target stream + how the element/offset should be
// positioned.
let targetSelection: StreamSelection;
if (needsTranscoding && onSeek) {
// Transcoded HLS is rebuilt rather than seeked in place, but the rebuilt
// stream starts at the BEGINNING of the item, not at `pos`: a start
@@ -1897,13 +1985,16 @@
// that really did start there; leaving it would now display `pos` while
// playing the opening titles.
// TRACES: UR-040, UR-004 | DR-181
targetUrl = await onSeek(pos, selectedAudioTrackIndex ?? undefined);
// Every transcode this app requests is HLS (DR-140).
targetSelection = selectionAt(await onSeek(pos, selectedAudioTrackIndex ?? undefined), {
type: "hls",
});
seekOffset = 0;
currentTime = pos;
pendingForegroundSeek = pos;
} else {
// Direct stream: reload the original URL and seek the element to pos.
targetUrl = streamUrl;
// Direct stream: reload the original selection and seek to pos.
targetSelection = selection;
seekOffset = 0;
pendingForegroundSeek = pos;
}
@@ -1923,18 +2014,21 @@
// than re-fetched.
//
// TRACES: UR-040, UR-003 | DR-196
currentStreamUrl = targetUrl;
currentSelection = targetSelection;
await commands.playerPlayItem({
streamUrl: targetUrl,
streamUrl: targetSelection.url,
title: media.name,
id: media.id,
videoCodec: needsTranscoding ? "hevc" : "h264",
needsTranscoding,
// TRACES: UR-079 | DR-229
transport: targetSelection.transport,
subtitles: nativeSubtitleTracks(sentSubtitleTracks),
});
didStartNativePlayback = true;
await playerAdapter?.load(targetUrl, {
await playerAdapter?.load(targetSelection.url, {
mediaId: media.id,
selection: targetSelection,
mediaSourceId: mediaSourceId ?? null,
needsTranscoding,
initialPosition: plan.position,
@@ -1961,9 +2055,9 @@
// blank it first, then set it on the next microtask so Svelte sees a real
// transition. Without this, assigning the same value is a no-op and the
// player stays stuck on the loading spinner (HLS never re-initialises).
currentStreamUrl = "";
currentSelection = selectionAt("", targetSelection.transport);
await Promise.resolve();
currentStreamUrl = targetUrl;
currentSelection = targetSelection;
} catch (err) {
log.error("Background-audio return failed:", err);
}
@@ -2285,33 +2379,41 @@
*
* The backend owns everything about how that happens — it decides whether the
* caller reloads (HTML5) or it reloads the native backend itself — so this
* only supplies the position to resume at and reverts the selection if the
* switch fails.
* only supplies the position to resume at.
*
* TRACES: UR-074 | DR-162
* The change applies to this playback alone; the durable Settings default is
* untouched (DR-225). Nothing is optimistically assigned here: what the picker
* shows comes from the selection the backend hands back, because what you get
* is not always what you asked for — a ceiling above the source bitrate is the
* source, and claiming otherwise is the kind of lie the old picker told.
*
* TRACES: UR-074, UR-079 | DR-162, DR-225, DR-226
*/
async function selectQuality(quality: StreamingQuality) {
showQualityMenu = false;
if (quality === selectedQuality || changingQuality) return;
const previous = selectedQuality;
selectedQuality = quality;
changingQuality = true;
try {
stopTimeUpdates();
await playerController.setStreamQuality(
const negotiated = await playerController.setStreamQuality(
quality,
videoElement ? videoElement.currentTime + seekOffset : null,
mediaSourceId ?? null,
selectedAudioTrackIndex,
);
// The HTML5 path reloads through the adapter, which already set the new
// selection via the bridge. The native path reloads inside Rust and
// returns nothing, so record what was asked for as the ceiling in force.
if (!negotiated) {
defaultQuality = quality;
}
if (videoElement && !videoElement.paused) {
startTimeUpdates();
}
log.debug("Streaming quality changed:", quality);
} catch (err) {
log.error("Failed to change streaming quality:", err);
selectedQuality = previous;
} finally {
changingQuality = false;
}
@@ -2416,7 +2518,10 @@
<!-- HTML5 video for desktop/non-Android platforms -->
<video
bind:this={videoElement}
src={currentStreamUrl.includes(".m3u8") && Hls.isSupported() ? "" : currentStreamUrl}
src={elementSrcFor(currentSelection, {
hlsJsSupported: Hls.isSupported(),
nativeHlsSupported: true,
})}
crossorigin={videoCrossOrigin}
class={videoFitClass()}
class:invisible={!isMediaReady}
@@ -2756,8 +2861,11 @@
</div>
{/if}
<!-- Streaming quality (bandwidth ceiling). TRACES: UR-074 | DR-162 -->
{#if streamingQualities.length > 0}
<!--
Streaming quality (bandwidth ceiling), populated from what this media
source can actually offer. TRACES: UR-070, UR-074 | DR-162, DR-226
-->
{#if qualityOptions.length > 1}
<div class="relative">
<button
onclick={toggleQualityMenu}
@@ -2778,22 +2886,30 @@
class="absolute bottom-full right-0 mb-2 bg-black/90 backdrop-blur-sm rounded-lg shadow-xl min-w-[220px] max-h-[300px] overflow-y-auto"
>
<div class="p-2">
<div class="text-white text-sm font-semibold px-3 py-2 border-b border-white/20">
Quality
<div class="px-3 py-2 border-b border-white/20">
<div class="text-white text-sm font-semibold">Quality</div>
<!--
What the server is actually doing. Only knowable now that
the backend reports it. TRACES: UR-079 | DR-227
-->
<div class="text-xs text-gray-400 mt-0.5">{playbackKindLabel}</div>
</div>
{#each streamingQualities as [quality, label, detail]}
{#each qualityOptions as option (option.quality)}
<button
onclick={() => selectQuality(quality)}
onclick={() => selectQuality(option.quality)}
class="w-full text-left px-3 py-2 text-white hover:bg-white/10 rounded transition-colors flex items-center justify-between {selectedQuality ===
quality
option.quality
? 'bg-white/20'
: ''}"
>
<div class="flex flex-col">
<span class="text-sm">{label}</span>
<span class="text-xs text-gray-400">{detail}</span>
<span class="text-sm">{option.label}</span>
<span class="text-xs text-gray-400">
{option.detail}{#if option.quality === "original" && option.sourceBitrate}
&middot; {(option.sourceBitrate / 1_000_000).toFixed(1)} Mbps{/if}
</span>
</div>
{#if selectedQuality === quality}
{#if selectedQuality === option.quality}
<svg
class="w-4 h-4 text-[var(--color-jellyfin)]"
fill="currentColor"