Files
jellytau/src/lib/player/adapters/webviewAudioAdapter.ts
T
dtourolle 109700b949 feat(playback): let Rust decide what stream to play, and say so
Playing a video meant asking the server to re-encode it, always. That
decision was made nowhere and written down nowhere, so whoever needed it
re-derived it downstream — the player worked out whether it had been handed
a playlist by looking for ".m3u8" in the URL, in two places. A viewer paid
for a transcode of a file their device could have played untouched, and the
app could not tell them which it was.

One negotiation now produces one self-describing StreamSelection — direct
play, remux or transcode; over a playlist, a plain HTTP file, or a local one
— and every renderer consumes that same answer.

Measured against the development server (Jellyfin 10.11.5), 400 items
sampled for codec mix and 40 put through a real PlaybackInfo negotiation
per profile:

  Linux / WebKitGTK (h264 only, 2ch)          3/40 —  7% direct play
  Android / ExoPlayer (hevc, ac3/eac3, 6ch)  34/40 — 85% direct play

The library is ~80% hevc, which is why the two diverge so hard. The payoff
is overwhelmingly Android, where 85% of plays were starting a transcode
nobody needed. Linux stays near 7% until libmpv decodes the picture — the
h264-only profile is a WebKitGTK constraint, not a JellyTau choice.

DR-219  StreamSelection: url + tagged Transport (hls/progressive/localFile)
        + PlaybackKind (directPlay/directStream/transcode) + the negotiated
        rendition + this source's ladder + a needs_transcoding flag derived
        in Rust so the rule is answered once. Both enums are serde-tagged
        so the frontend matches a discriminant, not a substring. The paths
        that never negotiate get the same shape from Rust rather than
        assembling one — media_local_selection for a downloaded file,
        LiveStreamInfo.transport for a live channel — so there is no second
        place where a transport is decided.

DR-220  The ceiling becomes two levels: a durable device default (Settings,
        persisted) and a per-playback override the in-player picker sets.
        The picker had called itself a "this film, this connection" control
        since it was written but wrote the process-wide default, so dropping
        one awkward film to 2 Mbps silently capped every video played
        afterwards for the rest of the process, with Settings still showing
        the old value. The override is cleared whenever playback moves to a
        new item, which stops it surviving into an autoplayed next episode.
        effective_streaming_quality() is the single resolution point.

DR-221  The quality picker is filled from what this media source can offer.
        Rust marks a rung exceeds_source when its ceiling is at or above the
        source's own bitrate — such a rung is another way to spell Original
        — and the frontend does not draw those. Original is never marked; a
        source whose bitrate the server does not report marks nothing, which
        keeps every rung offered.

DR-222  Direct play and direct stream are negotiated, with two client-side
        overrides on top because the server's answer is right about the file
        and wrong about what this app will do with it: undecodable audio
        (Jellyfin 10.11.5 honours a DirectPlayProfile's container and video
        codec but ignores its audio codec, so it offers direct play for an
        E-AC-3 track the webview renders in silence) and a viewer-pinned
        audio track the file does not default to. A direct stream is a remux
        and is deliberately not counted as transcoding.

DR-223  Dropped on measurement, not deferred. A master playlist from this
        server carries exactly one EXT-X-STREAM-INF: Jellyfin builds it from
        the single rendition the request asked for rather than publishing a
        ladder. So there is no adaptation for hls.js to be preserving and
        none mpv would lose — the claim that there was, in
        playback-backend-unification.md, does not hold. Recorded rather than
        deleted because it is a measurement: a server that does publish a
        ladder would change the answer.

DR-224  Every backend consumes the same selection. The queue item carries
        the transport, so player_seek_video picks its seek strategy from the
        backend's decision instead of the last stream_url.contains(".m3u8")
        in the codebase. Items queued by a path that never negotiated carry
        None and fall back to needs_transcoding, which is exact rather than
        a guess because every transcode this app requests is HLS (DR-140).

The frontend loader decision moves to streamTransport.ts so it can be
tested: the two cases that pin it are the ones that failed against the old
implementation — a progressive stream whose URL contains ".m3u8" must not
get an HLS loader, and an HLS stream whose URL contains none must.

Also verified the URL the direct-play branch builds actually serves playable
bytes: 206, video/mp4, valid ISO-BMFF, and a mid-file range works, so
seeking a direct play works.

The spec is folded into docs/architecture/{01,02,03} and deleted, per the
rule that docs/specs holds only work that has not shipped. DR-121 leaves
read-through-media-cache.md with a pointer; that spec keeps its capture half.

Not verified: real playback on a device. Direct play changes what actually
gets played, and neither fixtures nor curl prove the WebKitGTK and ExoPlayer
paths render it.
2026-08-22 13:45:03 +02:00

147 lines
4.7 KiB
TypeScript

import type { StreamSelection } from "$lib/api/bindings";
/**
* Webview audio adapter — plays audio-only media through a hidden `<audio>`
* element on platforms with no native audio backend (currently Windows).
*
* All *video* already renders through the webview `<video>` element on every
* platform; libmpv/ExoPlayer only drive audio-only playback. On Windows there is
* no native audio backend, so the Rust `WebviewAudioBackend` hands the stream URL
* to the frontend via a `webview_audio_load` event and drives play/pause/seek
* through `control_command`. This adapter owns the `<audio>` element that plays
* it and reports state/position/duration/ended back to Rust through the same
* `player_report_*` round-trip the HTML5 video adapter uses (via {@link AdapterHost}).
*
* It implements the {@link PlayerAdapter} surface so it can be registered with
* `playerController.setActiveAdapter` — but only the methods `handleControlCommand`
* actually routes (`play`, `pause`, `seekElement`) carry audio-specific logic;
* the video-only members (subtitles, transcode reload) are inert stubs.
*
* TRACES: UR-003, UR-005 | DR-004
*/
import type { AdapterHost, PlayerAdapter, PlayerLoadOptions } from "./types";
export class WebviewAudioAdapter implements PlayerAdapter {
readonly kind = "html5" as const;
private audio: HTMLAudioElement;
private host: AdapterHost;
private endedFired = false;
constructor(audio: HTMLAudioElement, host: AdapterHost) {
this.audio = audio;
this.host = host;
this.wire();
}
private wire(): void {
const a = this.audio;
a.addEventListener("loadedmetadata", () => {
this.host.onMediaLoaded(Number.isFinite(a.duration) ? a.duration : 0);
});
a.addEventListener("timeupdate", () => {
this.host.onPosition(a.currentTime, Number.isFinite(a.duration) ? a.duration : 0);
});
a.addEventListener("playing", () => this.host.onState("playing"));
a.addEventListener("pause", () => {
// A pause fired at the natural end is part of "ended"; don't report paused.
if (!a.ended) this.host.onState("paused");
});
a.addEventListener("waiting", () => this.host.onBuffering(true));
a.addEventListener("canplay", () => this.host.onReady());
a.addEventListener("ended", () => {
if (this.endedFired) return;
this.endedFired = true;
this.host.onState("stopped");
this.host.onEnded();
});
a.addEventListener("error", () => {
const err = a.error;
this.host.onError(err ? `audio error code ${err.code}` : "unknown audio error");
});
}
/** Load `url` at `initialPosition` and (by default) begin playing. */
async load(url: string, options: PlayerLoadOptions): Promise<void> {
this.endedFired = false;
this.host.onState("loading");
this.host.onStreamUrlChanged(url);
this.audio.src = url;
this.audio.load();
if (options.initialPosition > 0) {
// Seek once metadata is ready so currentTime sticks.
const seekWhenReady = () => {
this.audio.currentTime = options.initialPosition;
this.audio.removeEventListener("loadedmetadata", seekWhenReady);
};
this.audio.addEventListener("loadedmetadata", seekWhenReady);
}
await this.play();
}
async play(): Promise<void> {
try {
await this.audio.play();
} catch (e) {
this.host.onError(`play() rejected: ${String(e)}`);
}
}
async pause(): Promise<void> {
this.audio.pause();
}
async toggle(): Promise<boolean> {
if (this.audio.paused) {
await this.play();
return true;
}
await this.pause();
return false;
}
async seekElement(positionSeconds: number, _offset: number): Promise<void> {
this.audio.currentTime = positionSeconds;
}
/** No transcode-reload concept for direct audio; treat as a fresh load. */
async reloadSource(selection: StreamSelection, offset: number): Promise<void> {
await this.load(selection.url, {
mediaId: "",
mediaSourceId: null,
needsTranscoding: false,
initialPosition: offset,
isLive: false,
audioTrackIndex: null,
knownDuration: 0,
subtitleTracks: [],
});
}
attach(_element: HTMLVideoElement | null): void {
// The audio element is owned by the controller, not attached here.
}
setVolume(volume: number): void {
this.audio.volume = Math.max(0, Math.min(1, volume));
}
setMuted(muted: boolean): void {
this.audio.muted = muted;
}
async selectSubtitle(_streamIndex: number | null, _arrayIndex?: number): Promise<void> {
// No subtitles for audio-only playback.
}
getPosition(): number {
return this.audio.currentTime;
}
async dispose(): Promise<void> {
this.audio.pause();
this.audio.removeAttribute("src");
this.audio.load();
}
}