dtourolle/jellytau
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 52 Wiki Activity

refactor(player): delete the webview video path; mpv selects its own tracks

Browse Source 
DR-235 phase 3. Every video renderer is native now: mpv on Linux and
Windows, ExoPlayer on Android, all drawing behind the transparent
webview. The HTML5 <video> path is gone, not bypassed:

- Frontend: hls.js, Html5PlayerAdapter and its compatibility shim, the
  createAdapter factory, streamTransport, hlsRecovery, timeTracking,
  videoFit, the <video>/<track> markup and every element handler in
  VideoPlayer (3277 -> 2144 lines), the experimentalNativeVideo store
  and its Settings toggle, webviewVideoFallback/supportsNativeVideo, and
  the setHtml5VideoState PiP bridge call. NativePlayerAdapter is the one
  video adapter; webview audio gets its own adapter kind.
- Rust: use_html5 dropped from player_seek_video,
  player_switch_audio_track and player_set_stream_quality with the
  Html5* strategies and ReloadStream responses; use_html5_element and
  VideoBackend dropped from PlayerStatus; player_play_item always loads
  the backend (set_current_item removed); Capabilities::webview removed;
  the WebKitGTK GStreamer/VAAPI setup (and its gst-inspect spawn) removed.
- Android: the HTML5 video state in PictureInPictureManager and
  ScreenWakeManager, and the bridge method feeding it.
- CSP: connect-src loses http:/https: and worker-src loses blob: -
  both existed for hls.js; with it gone they were only an exfiltration
  channel and a blob worker for injected script. A test now keeps them
  out.

mpv takes over what the <video> element did (mpv_tracks, UT-275):
subtitles are the WebVTT list the play request carries, queued on
sub-files and selected by position in that list, starting off; audio
tracks are selected by position in the file; sid/aid are reset before
each load. Without this, Linux video had no subtitle selection and a
direct-play audio switch failed since mpv became its renderer.

Verified: Rust 948 passing, and the same 948 cross-compiled for Windows
under wine against the shipped DLL (track tests included); frontend
1111 passing; aarch64 debug APK builds. Lint warnings 158 -> 146, CI
ratchet tightened to match. Not yet seen on Windows hardware.
This commit is contained in:
dtourolle
2026-09-24 23:11:17 -04:00
parent bb3ab1edd7
commit 1677f5f299
69 changed files with 4014 additions and 7330 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/architecture/02-svelte-frontend.md
+14 -40
View File
@@ -791,36 +791,15 @@ by exactly the inset.
Unlike `addJavascriptInterface`, the inset push only writes CSS properties, so it
can safely be re-sent on resume.
## Stream Transport
## Stream Selection in the Player
**Location**: `src/lib/player/streamTransport.ts`
**TRACES**: UR-079 | DR-225 | UT-214
`videoLoaderFor(selection, capabilities)` picks the loader for the webview
`<video>` element — `hlsjs`, `nativeHls`, or `direct` — from the backend's tagged
`selection.transport`. `elementSrcFor` is its template companion: the element's
`src` is emptied only when hls.js is driving it.
The split is the point. **The transport is the stream's property and comes from
Rust; whether a given loader exists is the browser's, and is the only thing
decided here.**
> This replaced `currentStreamUrl.includes(".m3u8")`, which appeared twice in
> `VideoPlayer.svelte` — once in the HLS `$effect` and once inline in the
> template's `src`. Rust builds that URL and knows what it is; re-deriving it
> here by substring match was a domain fact reconstructed in the presentation
> layer, and it fails silently in both directions. The two tests that pin it are
> the ones that failed against the old implementation: a `progressive` stream
> whose URL contains `.m3u8` must **not** get an HLS loader, and an `hls` stream
> whose URL contains no `.m3u8` must.
>
> Logic lives in a plain `.ts` module rather than in the component for the usual
> reason — it is testable there. Same pattern as `episodeStrip.ts`.
**TRACES**: UR-079 | DR-225, DR-227
`VideoPlayer` holds a `currentSelection`, not a URL string; `currentStreamUrl` is
derived from it. A reload replaces the selection **wholesale** (the adapter's
bridge takes a `StreamSelection`, not a URL), so transport and URL can never
drift apart. The background-audio handoff states the transport it is moving to —
derived from it. A reload replaces the selection **wholesale**, so transport and
URL can never drift apart — the transport is the stream's property and comes from
Rust, never from a substring match on the URL (which is what `.m3u8` sniffing in
the component once did). The background-audio handoff states the transport it is moving to —
progressive mp3 out, HLS back — via `selectionAt()`, rather than leaving it to be
inferred.
@@ -833,21 +812,16 @@ ceiling above the source bitrate *is* the source.
## Native Video Store
**Location**: `src/lib/stores/nativeVideo.ts`
**TRACES**: UR-003, UR-004 | DR-188
**TRACES**: UR-003, UR-004 | DR-188, DR-235
Two separate concerns live here, deliberately:
`nativeVideoActive` — whether a native surface is on screen *right now*.
Setting it toggles `data-native-video` on `<html>`, which is what the CSS in
`app.css` keys off to clear the app's opaque backgrounds so the picture behind
the webview shows. The backgrounds must come back the moment the player unmounts.
- `experimentalNativeVideo` — the user-facing opt-in flag, **defaulting to on**.
Rust already decides *which backend this platform has* (`useHtml5Element` from
`player_play_item`); this flag only *suppresses* that decision. It never turns
native on where Rust says HTML5. An explicit stored choice wins in both
directions, so someone who opted out is not re-enabled by a default flip —
hence the `null` check rather than a bare `=== "true"`.
- `nativeVideoActive` — whether a native surface is on screen *right now*.
Setting it toggles `data-native-video` on `<html>`, which is what the CSS in
`app.css` keys off to clear the app's opaque backgrounds. It is deliberately
**not** derived from the flag: the backgrounds must come back the moment the
player unmounts.
It used to hold `experimentalNativeVideo`, a stored switch that could force video
back to the webview `<video>` element. That element is gone (DR-235), and the
switch with it.
See [05-platform-backends.md](05-platform-backends.md#native-video-compositing-android)
for what is behind the WebView.
docs/architecture/03-data-flow.md
+2 -2
View File
@@ -213,8 +213,8 @@ sequenceDiagram
Online-->>Page: StreamSelection
end
Page->>VP: selection
VP->>VP: videoLoaderFor(selection, caps)
Note over VP: hls.js / native HLS / direct —<br/>from the tag, never from the URL
VP->>VP: player_play_item(selection.url, transport)
Note over VP: the native player opens it —<br/>transport from the tag, never from the URL
```
The selection travels with the stream from then on. A reload — a quality change,
docs/architecture/05-platform-backends.md
+76 -68
View File
@@ -90,57 +90,75 @@ flowchart LR
**Important**: The command is `player_get_queue` (returns `QueueStatus` with `hasNext`/`hasPrevious`). There is no `player_get_queue_status` command.
## HTML5 Video Adapter (webview-rendered video)
## Video is always native (no webview `<video>`)
**Location**: `src/lib/player/html5Adapter.ts`, `src/lib/player/index.ts`, report commands in
`src-tauri/src/commands/player/timers.rs`
**TRACES**: UR-080 | DR-231, DR-235, DR-237
Video on **Windows** is rendered by an HTML5 `<video>`/HLS element **inside the webview**. Neither
other platform uses this path for video: Android draws in ExoPlayer (see *The webview is not a video
renderer on Android* below), and Linux draws in mpv beneath the webview (DR-235 phase 1 — see
[desktop-native-video.md](../specs/desktop-native-video.md) until it ships on Windows too). Where the
element is used, no native backend renders or observes it. The `<video>` is therefore
the real player, living outside Rust's reach.
Every video renderer is a native player drawing **behind** the transparent
webview, with the Svelte controls composited over it: ExoPlayer on Android, mpv on
Linux and Windows. There is no webview `<video>` element, no hls.js, and no
HTML5 adapter; the frontend has one video adapter, `NativePlayerAdapter`, and the
backend performs every seek, track switch and quality change itself.
To keep the `PlayerController` the single source of truth (matching the audio path), the HTML5 element
is treated as **a dumb output device that reports back into Rust**, rather than an independent state
authority:
Why the webview path was deleted rather than kept as a fallback:
```mermaid
flowchart LR
subgraph Webview["Webview"]
Video["HTML5 <video> / HLS.js"]
Adapter["html5Adapter.ts<br/>(reports DOM events)"]
end
subgraph Backend["Rust"]
Cmds["player_report_state<br/>player_report_position<br/>player_report_media_loaded"]
Controller["PlayerController"]
Emitter["TauriEventEmitter"]
end
subgraph Frontend["Frontend"]
Events["playerEvents.ts"]
Store["player store"]
end
- **The transcode was a decoder constraint.** The `<video>` element decodes
little beyond h264, so the desktop profile could only claim h264 and the server
re-encoded almost everything (7% direct play on a real library, against 85% for
the same library through ExoPlayer). The machine was never the limit — mpv was
already running for audio. (The Linux/Windows profile still claims h264 until
DR-234 widens it; see [desktop-native-video.md](../specs/desktop-native-video.md).)
- **Each renderer is another place for every bug.** Three video renderers meant
every seek strategy, track switch and lifecycle fix had three places to be got
right; the webview path was also where the renderer choice itself went wrong
(silent Linux video, a Windows soundtrack decoded twice — DR-237).
- **A fallback that decodes less is not a fallback.** Android showed it first
(DR-293): an original-file download plays silent in the webview. With no
webview path there is no silent downgrade to fall into; a failed mpv init
emits `backend-init-failed` instead.
Video -->|DOM events| Adapter --> Cmds --> Controller --> Emitter --> Events --> Store
```
What survives of the webview reporting pipeline is audio-only:
`WebviewAudioBackend` plays through a hidden `<audio>` element and reports back
through the `player_report_*` commands (`rustReportHost.ts`), for a desktop with
no mpv. No shipped platform uses it.
**Key points:**
- The adapter re-emits the *same* `PlayerStatusEvent`s (`StateChanged`, `PositionUpdate`, `MediaLoaded`)
the native backends emit, so `playerEvents.ts` needs **no** HTML5-specific branch — HTML5 is just
another event source feeding the existing pipeline.
- Position reports are throttled (~250ms) to match the MPV cadence and avoid flooding IPC from the
60fps RAF loop.
- **Boundary rule**: UI components never touch the report commands or `videoElement` state directly.
Playback *control* goes through the unified facade `src/lib/player/index.ts` (`playerController`);
HTML5 *state reporting* goes through `html5Adapter.ts`. This restores the documented invariant
("frontend only displays state and invokes commands") for the video path.
## Native video on the desktop (mpv)
**TRACES**: UR-080 | DR-231 … DR-237, DR-298, DR-299
The mpv half is shared; only the surface differs per platform
(`mpv_backend::video_output`):
| Platform | Output | Where the picture goes |
|---|---|---|
| Linux | `vo=libmpv` (render API) | An FBO drawn in the main window's own `GtkBox` `draw` handler, which GTK paints *before* its children — so beneath the webview, with no widget reparenting (`video_surface.rs`; a `GtkOverlay` aborts the process on the first click, see that module) |
| Windows | `vo=gpu-next,gpu`, `wid=<HWND>` | mpv renders as a child of the app window, beneath the transparent WebView2 — the arrangement tauri-plugin-libmpv ships. `wid` only takes effect before initialisation, so `MpvBackend::new` takes the handle; with no handle it draws nothing rather than open a window of its own. mpv's controller, bindings and cursor handling are off |
In both, the page clears its opaque backgrounds while a video is on screen
(`data-native-video`, the same CSS Android uses).
**Tracks** (`mpv_tracks.rs`): subtitles are the WebVTT list the play request
carries, queued on `sub-files` before the load and selected **by position in
that list** — the same meaning ExoPlayer gives `player_set_subtitle_track`.
Selection starts off (the menu opens on "Off") and `sid`/`aid` are reset before
every load, so a choice made for one item cannot leak into the next. Audio tracks
are selected by position in the file; a transcode carries one track and is
re-opened instead.
**Two rules for every libmpv handle** (`mpv_command.rs`):
- Commands go through `mpv_command::command`, an argv built for `mpv_command`,
never the pinned crate's `Mpv::command`, which joins its arguments into a
string that mpv parses — `;` chains a second command, so a track title in a
downloaded file's path could run `run …` (DR-298).
- Every handle is hardened before its first load: `tls-verify=yes` (mpv's
default is *no*, and its URLs carry the `ApiKey`) and `ytdl=no` (DR-299).
## MpvBackend (Linux)
**Location**: `src-tauri/src/player/mpv/`
The MPV backend uses libmpv for audio playback on Linux. Since MPV handles are not `Send`, all operations occur on a dedicated thread.
The MPV backend uses libmpv for audio **and video** playback on Linux and Windows (video: see *Native video on the desktop* above). Since MPV handles are not `Send`, all operations occur on a dedicated thread.
```mermaid
flowchart TB
@@ -320,18 +338,12 @@ remux). It costs minutes of CPU and twice the disk per film, needs a pipeline
state of its own, and does nothing for streaming. Decoding at playback fixes both
paths with no extra step.
### The webview is not a video renderer on Android
### Why the webview could not stay a fallback on Android
ExoPlayer is Android's only video renderer. The HTML5 path used to be reachable
through the `experimentalNativeVideo` setting (a *suppressor* of Rust's native
choice), but the webview decodes none of the codecs above — so with the original
file now downloaded as-is (DR-293), turning native video off would play every such
download as a silent film. Rust reports `webview_video_fallback` in
`PlaybackCapabilities`: **false on Android**, true only beside mpv native video on
Linux, where the webview is still the tested fallback. The frontend offers the
switch and honours a stored "off" only when it is true (`nativeVideoWanted` in
`stores/nativeVideo.ts`), so a user who once switched it off on Android is not
stranded on the silent path.
The webview decodes none of the codecs above — so with the original file now
downloaded as-is (DR-293), the old `experimentalNativeVideo` switch would have
played every such download as a silent film. That is what first removed the
webview video path on Android; DR-235 then removed it everywhere.
### The equalizer, and where its vocabulary lives
@@ -360,8 +372,8 @@ chain of unbounded length.
Keeping a video's **audio** alive when the app is backgrounded or the screen
locks, while video decode stops. Two verified facts drive the whole design:
1. An Android WebView `<video>` **does not** keep playing audio once the app is
backgrounded — the system throttles the WebView and media pauses.
1. Nothing in the WebView keeps playing once the app is backgrounded — the
system throttles it.
2. Keeping audio alive in the background requires a **native foreground media
service**, which already exists for music (`JellyTauPlaybackService` +
`JellyTauPlayer` + `MediaSessionCompat`).
@@ -391,18 +403,14 @@ sequenceDiagram
Details that were each a shipped defect:
- **Position is absolute.** Transcoded HLS tracks time as
`videoElement.currentTime + seekOffset` (the element resets to 0 after each
transcode reload). `computeHandoffPosition` sums both terms; using the element
time alone rewinds by the offset.
- **Position is absolute** — the position on the item's timeline the player
reports, never an offset within a re-opened transcode (`computeHandoffPosition`).
- **A downloaded episode takes no base URL and an ordinary seek** (DR-180); a
stream takes the base and no seek; a handoff at 0:00 takes neither.
- **The return must restart the renderer that is actually on screen** (DR-196).
The two paths resume by different means — the webview `<video>` reloads off its
stream URL, watched by an `$effect`; ExoPlayer owns no element and nothing
watches the URL for it, so it needs an explicit re-issue. Doing only the URL
assignment restarted nothing on the native path and left a black screen with a
play button that did nothing.
- **The return must re-issue the load** (DR-196). The native player owns no
element and nothing watches the stream URL for it, so reassigning the URL — how
the deleted webview path came back — restarted nothing and left a black screen
with a play button that did nothing.
- **`wasPlaying` is captured on the way out** so play/pause survives the round
trip, and the handoff does not silently rewind (DR-203).
- **Mutually exclusive with PiP.** Toggle on → `setAutoEnterEnabled(false)`;
@@ -423,10 +431,10 @@ non-Android platform.
**TRACES**: UR-003, UR-004 | DR-150 … DR-152, DR-182 … DR-196
Android can render video on the **native ExoPlayer surface behind a transparent
Tauri WebView**, with the Svelte controls drawn over it. This is on by default;
the HTML5 `<video>` path remains the fallback and is not being removed. The
default has been flipped and reverted twice and each revert has a named cause —
the per-defect record is in `requirements.md` (DR-150 … DR-196).
Tauri WebView**, with the Svelte controls drawn over it. It is the only video
path (DR-235). Before that it was an opt-in whose default was flipped and
reverted twice, each revert with a named cause — the per-defect record is in
`requirements.md` (DR-150 … DR-196).
```mermaid
flowchart TB
@@ -456,7 +464,7 @@ Load-bearing details, each of which was a shipped defect:
- **The app shell stops painting over the surface** (DR-185). `app.css` clears
its opaque backgrounds off `[data-native-video]`; before that, a CSS rule
targeted an attribute nothing ever set, so the fix looked applied and was not.
- **The poster card can lift on a path with no `<video>` element** (DR-182) — the
- **The poster card lifts without a `<video>` element** (DR-182) — the
native reveal fires on a `playing` state or a position tick carrying a position
or duration, and on nothing else.
- **Letterbox bars are painted**, not left holding whatever was last in the
docs/architecture/09-security.md
+6 -6
View File
@@ -68,8 +68,8 @@ style-src 'self' 'unsafe-inline';
font-src 'self' data:;
img-src 'self' data: blob: asset: http://asset.localhost http: https:;
media-src 'self' blob: asset: http://asset.localhost http://127.0.0.1:* http: https:;
connect-src 'self' ipc: http://ipc.localhost http: https:;
worker-src 'self' blob:;
connect-src 'self' ipc: http://ipc.localhost;
worker-src 'self';
object-src 'none'; frame-src 'none'; base-uri 'self'; form-action 'self'; frame-ancestors 'none'
```
@@ -79,13 +79,13 @@ object-src 'none'; frame-src 'none'; base-uri 'self'; form-action 'self'; frame-
| `script-src 'self'` | The genuinely restrictive half. Bundled JS only; Tauri's build-time nonce covers the one inline `<script>` in `index.html`. Adding `'unsafe-inline'` here would silently do nothing anyway — a nonce in a directive voids it. |
| `style-src 'self' 'unsafe-inline'` | Svelte compiles `style="…"` attributes into markup, including `app.html`'s `display: contents` wrapper, and CSP treats a style *attribute* as inline. Safe only while no `<style>` **element** survives into `index.html`: Tauri would nonce it, and the nonce would then void `'unsafe-inline'`. The production build extracts all CSS to files, so it currently has none. |
| `img-src` | Thumbnails come from two places: the asset protocol (`asset://localhost/…` on Linux/macOS, `http://asset.localhost/…` on Windows/Android — the same protocol, named differently by `convertFileSrc`) and, on a cache miss, straight from the Jellyfin server. `data:`/`blob:` cover inline and generated images. |
| `media-src` | `<video>`/`<audio>` sources: HLS transcodes and progressive streams from the server, the token-guarded loopback media server on `http://127.0.0.1:<random port>` (DR-137), and `blob:` for the MSE object URL hls.js attaches. |
| `connect-src` | `ipc:` / `http://ipc.localhost` is Tauri's `invoke` transport (custom scheme on Linux/macOS, `http` host on Windows/Android) — without it every command is blocked. `http:`/`https:` is hls.js fetching manifests and segments; ordinary API traffic goes through Rust and is not subject to CSP. |
| `worker-src 'self' blob:` | hls.js runs its demuxer in a worker built from a blob (`enableWorker: true`). Without `blob:` it falls back to main-thread demuxing — playback survives but costs more CPU. |
| `media-src` | `<audio>` sources for the webview audio backend (a desktop without mpv; no shipped platform): streams from the server and the token-guarded loopback media server on `http://127.0.0.1:<random port>` (DR-137). Video never plays in the webview (DR-235). |
| `connect-src` | `ipc:` / `http://ipc.localhost` is Tauri's `invoke` transport (custom scheme on Linux/macOS, `http` host on Windows/Android) — without it every command is blocked. Nothing else: all network traffic goes through Rust. It allowed any `http:`/`https:` host while hls.js fetched manifests and segments in the page; with hls.js gone (DR-235) that grant was only an exfiltration channel for injected script, so it went too. |
| `worker-src 'self'` | No blob workers since hls.js (whose demuxer ran in one) was removed (DR-235). |
| `object-src`, `frame-src` = `'none'` | No plugins, no iframes; both are classic injection sinks. |
| `base-uri 'self'`, `form-action 'self'`, `frame-ancestors 'none'` | Block `<base>` hijacking, form exfiltration and framing. `frame-ancestors` is only honoured when the policy is delivered as a header, which is platform-dependent; it is harmless where it is not. |
**`img-src`/`media-src`/`connect-src` are deliberately permissive.** The Jellyfin
**`img-src`/`media-src` are deliberately permissive.** The Jellyfin
origin is typed in by the user at run time and is routinely plain `http` on a
LAN, so it cannot be enumerated at build time. `http: https:` is a wide grant for
*data* — but it still bars `file:`, `filesystem:` and scripting schemes, and it
docs/architecture/README.md
+1 -2
View File
@@ -99,7 +99,7 @@ Each major subsystem is documented in its own file in this directory:
| [02 - Svelte Frontend](02-svelte-frontend.md) | Store structure, music library navigation, playback reporting, repository architecture, playback mode system, database service abstraction, component hierarchy, MiniPlayer, sleep timer, auto-play, navigation guard, playlist management UI, library mosaic, series/episode navigation, downloaded browse, safe-area insets, native-video store, logging |
| [03 - Data Flow](03-data-flow.md) | Repository query flow (cache-first), locally-indexed search, playback initiation, playback mode transfer, queue navigation, volume control |
| [04 - Type Sync & Threading](04-type-sync-and-threading.md) | Rust/TypeScript type synchronization, Tauri v2 IPC parameter naming convention, thread safety patterns |
| [05 - Platform Backends](05-platform-backends.md) | Player events system, HTML5 video adapter, MpvBackend (Linux), ExoPlayerBackend (Android) incl. audio settings parity, **native video compositing**, MediaSession & remote volume, album art caching, backend initialization |
| [05 - Platform Backends](05-platform-backends.md) | Player events system, MpvBackend (Linux), ExoPlayerBackend (Android) incl. audio settings parity, **native video compositing**, MediaSession & remote volume, album art caching, backend initialization |
| [06 - Downloads & Offline](06-downloads-and-offline.md) | Download manager, download worker, smart caching engine, **one storage model (cache entries are downloads)**, offline catalog visibility, download/offline commands, player integration, frontend store, UI components |
| [07 - Connectivity](07-connectivity.md) | HTTP client with retry logic, connectivity monitor, network resilience architecture |
| [08 - Database Design](08-database-design.md) | Entity relationships, all table definitions (servers, users, libraries, items, user_data, downloads, media_streams, sync_queue, thumbnails, playlists), key queries, data flow diagrams, storage estimates |
@@ -176,7 +176,6 @@ src/lib/
│ └── sessions.ts # SessionsApi (remote session control)
├── player/ # Unified player boundary (frontend)
│ ├── index.ts # playerController facade — the only write-side entry point for playback
│ └── html5Adapter.ts # Reports webview <video> DOM events back into Rust (player_report_*)
├── services/
│ ├── playerEvents.ts # Tauri event listener for player events
│ └── playbackReporting.ts # Thin wrapper (~50 lines)