dtourolle/jellytau
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 52 Wiki Activity

refactor(player): delete the webview video path; mpv selects its own tracks

Browse Source 
DR-235 phase 3. Every video renderer is native now: mpv on Linux and
Windows, ExoPlayer on Android, all drawing behind the transparent
webview. The HTML5 <video> path is gone, not bypassed:

- Frontend: hls.js, Html5PlayerAdapter and its compatibility shim, the
  createAdapter factory, streamTransport, hlsRecovery, timeTracking,
  videoFit, the <video>/<track> markup and every element handler in
  VideoPlayer (3277 -> 2144 lines), the experimentalNativeVideo store
  and its Settings toggle, webviewVideoFallback/supportsNativeVideo, and
  the setHtml5VideoState PiP bridge call. NativePlayerAdapter is the one
  video adapter; webview audio gets its own adapter kind.
- Rust: use_html5 dropped from player_seek_video,
  player_switch_audio_track and player_set_stream_quality with the
  Html5* strategies and ReloadStream responses; use_html5_element and
  VideoBackend dropped from PlayerStatus; player_play_item always loads
  the backend (set_current_item removed); Capabilities::webview removed;
  the WebKitGTK GStreamer/VAAPI setup (and its gst-inspect spawn) removed.
- Android: the HTML5 video state in PictureInPictureManager and
  ScreenWakeManager, and the bridge method feeding it.
- CSP: connect-src loses http:/https: and worker-src loses blob: -
  both existed for hls.js; with it gone they were only an exfiltration
  channel and a blob worker for injected script. A test now keeps them
  out.

mpv takes over what the <video> element did (mpv_tracks, UT-275):
subtitles are the WebVTT list the play request carries, queued on
sub-files and selected by position in that list, starting off; audio
tracks are selected by position in the file; sid/aid are reset before
each load. Without this, Linux video had no subtitle selection and a
direct-play audio switch failed since mpv became its renderer.

Verified: Rust 948 passing, and the same 948 cross-compiled for Windows
under wine against the shipped DLL (track tests included); frontend
1111 passing; aarch64 debug APK builds. Lint warnings 158 -> 146, CI
ratchet tightened to match. Not yet seen on Windows hardware.
This commit is contained in:
dtourolle
2026-09-24 23:11:17 -04:00
parent bb3ab1edd7
commit 1677f5f299
69 changed files with 4014 additions and 7330 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/architecture/05-platform-backends.md
+76 -68
View File
@@ -90,57 +90,75 @@ flowchart LR
**Important**: The command is `player_get_queue` (returns `QueueStatus` with `hasNext`/`hasPrevious`). There is no `player_get_queue_status` command.
## HTML5 Video Adapter (webview-rendered video)
## Video is always native (no webview `<video>`)
**Location**: `src/lib/player/html5Adapter.ts`, `src/lib/player/index.ts`, report commands in
`src-tauri/src/commands/player/timers.rs`
**TRACES**: UR-080 | DR-231, DR-235, DR-237
Video on **Windows** is rendered by an HTML5 `<video>`/HLS element **inside the webview**. Neither
other platform uses this path for video: Android draws in ExoPlayer (see *The webview is not a video
renderer on Android* below), and Linux draws in mpv beneath the webview (DR-235 phase 1 — see
[desktop-native-video.md](../specs/desktop-native-video.md) until it ships on Windows too). Where the
element is used, no native backend renders or observes it. The `<video>` is therefore
the real player, living outside Rust's reach.
Every video renderer is a native player drawing **behind** the transparent
webview, with the Svelte controls composited over it: ExoPlayer on Android, mpv on
Linux and Windows. There is no webview `<video>` element, no hls.js, and no
HTML5 adapter; the frontend has one video adapter, `NativePlayerAdapter`, and the
backend performs every seek, track switch and quality change itself.
To keep the `PlayerController` the single source of truth (matching the audio path), the HTML5 element
is treated as **a dumb output device that reports back into Rust**, rather than an independent state
authority:
Why the webview path was deleted rather than kept as a fallback:
```mermaid
flowchart LR
subgraph Webview["Webview"]
Video["HTML5 <video> / HLS.js"]
Adapter["html5Adapter.ts<br/>(reports DOM events)"]
end
subgraph Backend["Rust"]
Cmds["player_report_state<br/>player_report_position<br/>player_report_media_loaded"]
Controller["PlayerController"]
Emitter["TauriEventEmitter"]
end
subgraph Frontend["Frontend"]
Events["playerEvents.ts"]
Store["player store"]
end
- **The transcode was a decoder constraint.** The `<video>` element decodes
little beyond h264, so the desktop profile could only claim h264 and the server
re-encoded almost everything (7% direct play on a real library, against 85% for
the same library through ExoPlayer). The machine was never the limit — mpv was
already running for audio. (The Linux/Windows profile still claims h264 until
DR-234 widens it; see [desktop-native-video.md](../specs/desktop-native-video.md).)
- **Each renderer is another place for every bug.** Three video renderers meant
every seek strategy, track switch and lifecycle fix had three places to be got
right; the webview path was also where the renderer choice itself went wrong
(silent Linux video, a Windows soundtrack decoded twice — DR-237).
- **A fallback that decodes less is not a fallback.** Android showed it first
(DR-293): an original-file download plays silent in the webview. With no
webview path there is no silent downgrade to fall into; a failed mpv init
emits `backend-init-failed` instead.
Video -->|DOM events| Adapter --> Cmds --> Controller --> Emitter --> Events --> Store
```
What survives of the webview reporting pipeline is audio-only:
`WebviewAudioBackend` plays through a hidden `<audio>` element and reports back
through the `player_report_*` commands (`rustReportHost.ts`), for a desktop with
no mpv. No shipped platform uses it.
**Key points:**
- The adapter re-emits the *same* `PlayerStatusEvent`s (`StateChanged`, `PositionUpdate`, `MediaLoaded`)
the native backends emit, so `playerEvents.ts` needs **no** HTML5-specific branch — HTML5 is just
another event source feeding the existing pipeline.
- Position reports are throttled (~250ms) to match the MPV cadence and avoid flooding IPC from the
60fps RAF loop.
- **Boundary rule**: UI components never touch the report commands or `videoElement` state directly.
Playback *control* goes through the unified facade `src/lib/player/index.ts` (`playerController`);
HTML5 *state reporting* goes through `html5Adapter.ts`. This restores the documented invariant
("frontend only displays state and invokes commands") for the video path.
## Native video on the desktop (mpv)
**TRACES**: UR-080 | DR-231 … DR-237, DR-298, DR-299
The mpv half is shared; only the surface differs per platform
(`mpv_backend::video_output`):
| Platform | Output | Where the picture goes |
|---|---|---|
| Linux | `vo=libmpv` (render API) | An FBO drawn in the main window's own `GtkBox` `draw` handler, which GTK paints *before* its children — so beneath the webview, with no widget reparenting (`video_surface.rs`; a `GtkOverlay` aborts the process on the first click, see that module) |
| Windows | `vo=gpu-next,gpu`, `wid=<HWND>` | mpv renders as a child of the app window, beneath the transparent WebView2 — the arrangement tauri-plugin-libmpv ships. `wid` only takes effect before initialisation, so `MpvBackend::new` takes the handle; with no handle it draws nothing rather than open a window of its own. mpv's controller, bindings and cursor handling are off |
In both, the page clears its opaque backgrounds while a video is on screen
(`data-native-video`, the same CSS Android uses).
**Tracks** (`mpv_tracks.rs`): subtitles are the WebVTT list the play request
carries, queued on `sub-files` before the load and selected **by position in
that list** — the same meaning ExoPlayer gives `player_set_subtitle_track`.
Selection starts off (the menu opens on "Off") and `sid`/`aid` are reset before
every load, so a choice made for one item cannot leak into the next. Audio tracks
are selected by position in the file; a transcode carries one track and is
re-opened instead.
**Two rules for every libmpv handle** (`mpv_command.rs`):
- Commands go through `mpv_command::command`, an argv built for `mpv_command`,
never the pinned crate's `Mpv::command`, which joins its arguments into a
string that mpv parses — `;` chains a second command, so a track title in a
downloaded file's path could run `run …` (DR-298).
- Every handle is hardened before its first load: `tls-verify=yes` (mpv's
default is *no*, and its URLs carry the `ApiKey`) and `ytdl=no` (DR-299).
## MpvBackend (Linux)
**Location**: `src-tauri/src/player/mpv/`
The MPV backend uses libmpv for audio playback on Linux. Since MPV handles are not `Send`, all operations occur on a dedicated thread.
The MPV backend uses libmpv for audio **and video** playback on Linux and Windows (video: see *Native video on the desktop* above). Since MPV handles are not `Send`, all operations occur on a dedicated thread.
```mermaid
flowchart TB
@@ -320,18 +338,12 @@ remux). It costs minutes of CPU and twice the disk per film, needs a pipeline
state of its own, and does nothing for streaming. Decoding at playback fixes both
paths with no extra step.
### The webview is not a video renderer on Android
### Why the webview could not stay a fallback on Android
ExoPlayer is Android's only video renderer. The HTML5 path used to be reachable
through the `experimentalNativeVideo` setting (a *suppressor* of Rust's native
choice), but the webview decodes none of the codecs above — so with the original
file now downloaded as-is (DR-293), turning native video off would play every such
download as a silent film. Rust reports `webview_video_fallback` in
`PlaybackCapabilities`: **false on Android**, true only beside mpv native video on
Linux, where the webview is still the tested fallback. The frontend offers the
switch and honours a stored "off" only when it is true (`nativeVideoWanted` in
`stores/nativeVideo.ts`), so a user who once switched it off on Android is not
stranded on the silent path.
The webview decodes none of the codecs above — so with the original file now
downloaded as-is (DR-293), the old `experimentalNativeVideo` switch would have
played every such download as a silent film. That is what first removed the
webview video path on Android; DR-235 then removed it everywhere.
### The equalizer, and where its vocabulary lives
@@ -360,8 +372,8 @@ chain of unbounded length.
Keeping a video's **audio** alive when the app is backgrounded or the screen
locks, while video decode stops. Two verified facts drive the whole design:
1. An Android WebView `<video>` **does not** keep playing audio once the app is
backgrounded — the system throttles the WebView and media pauses.
1. Nothing in the WebView keeps playing once the app is backgrounded — the
system throttles it.
2. Keeping audio alive in the background requires a **native foreground media
service**, which already exists for music (`JellyTauPlaybackService` +
`JellyTauPlayer` + `MediaSessionCompat`).
@@ -391,18 +403,14 @@ sequenceDiagram
Details that were each a shipped defect:
- **Position is absolute.** Transcoded HLS tracks time as
`videoElement.currentTime + seekOffset` (the element resets to 0 after each
transcode reload). `computeHandoffPosition` sums both terms; using the element
time alone rewinds by the offset.
- **Position is absolute** — the position on the item's timeline the player
reports, never an offset within a re-opened transcode (`computeHandoffPosition`).
- **A downloaded episode takes no base URL and an ordinary seek** (DR-180); a
stream takes the base and no seek; a handoff at 0:00 takes neither.
- **The return must restart the renderer that is actually on screen** (DR-196).
The two paths resume by different means — the webview `<video>` reloads off its
stream URL, watched by an `$effect`; ExoPlayer owns no element and nothing
watches the URL for it, so it needs an explicit re-issue. Doing only the URL
assignment restarted nothing on the native path and left a black screen with a
play button that did nothing.
- **The return must re-issue the load** (DR-196). The native player owns no
element and nothing watches the stream URL for it, so reassigning the URL — how
the deleted webview path came back — restarted nothing and left a black screen
with a play button that did nothing.
- **`wasPlaying` is captured on the way out** so play/pause survives the round
trip, and the handoff does not silently rewind (DR-203).
- **Mutually exclusive with PiP.** Toggle on → `setAutoEnterEnabled(false)`;
@@ -423,10 +431,10 @@ non-Android platform.
**TRACES**: UR-003, UR-004 | DR-150 … DR-152, DR-182 … DR-196
Android can render video on the **native ExoPlayer surface behind a transparent
Tauri WebView**, with the Svelte controls drawn over it. This is on by default;
the HTML5 `<video>` path remains the fallback and is not being removed. The
default has been flipped and reverted twice and each revert has a named cause —
the per-defect record is in `requirements.md` (DR-150 … DR-196).
Tauri WebView**, with the Svelte controls drawn over it. It is the only video
path (DR-235). Before that it was an opt-in whose default was flipped and
reverted twice, each revert with a named cause — the per-defect record is in
`requirements.md` (DR-150 … DR-196).
```mermaid
flowchart TB
@@ -456,7 +464,7 @@ Load-bearing details, each of which was a shipped defect:
- **The app shell stops painting over the surface** (DR-185). `app.css` clears
its opaque backgrounds off `[data-native-video]`; before that, a CSS rule
targeted an attribute nothing ever set, so the fix looked applied and was not.
- **The poster card can lift on a path with no `<video>` element** (DR-182) — the
- **The poster card lifts without a `<video>` element** (DR-182) — the
native reveal fires on a `playing` state or a position tick carrying a position
or duration, and on nothing else.
- **Letterbox bars are painted**, not left holding whatever was last in the