From 11d9d760d8b13983bddd94d4d05bd6462c5b1bb5 Mon Sep 17 00:00:00 2001 From: Duncan Tourolle Date: Sun, 23 Aug 2026 10:51:45 +0200 Subject: [PATCH] feat(player): native video on Linux, and one contract for every player (v0.11.0) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit mpv now decodes video on Linux, drawn into a framebuffer we own and blitted into the default vbox's draw handler. Tauri's widget tree is untouched, so an upgrade that assumes its own layout cannot invalidate this. Direct play means the original file, hardware decoding, and no server transcode at all — where previously every desktop video was re-encoded to h264 for the browser engine, whatever the file actually was. Off by default: JELLYTAU_NATIVE_VIDEO=1. That settles finding 2 of playback-backend-unification.md — "native video cannot be composited with a Tauri webview" — by demonstration rather than argument, on X11 and Wayland both. Turning it on exposed nine defects, none of them mpv's. Each was the same mistake in a different place: a capability written down as a compile-time fact about the platform, or a state asserted instead of confirmed. DR-238/246 a seek routed by the stream's container rather than by what the engine could do with it - correct only while one player handled those streams, silent the moment another did DR-239 a property handled but never observed, so the play/pause button waited for an event that could not arrive DR-240 fullscreen expanding the document while the window stayed put DR-241 a seek issued before the engine had a file, failed, and discarded - which is why resume began at zero DR-247 a Linux-only gate outliving the caller that made it Linux-only, breaking the Android build outright DR-250 a stop aimed at whichever renderer bookkeeping believed was in charge, missing the one actually making sound DR-251 a duration of zero believed, leaving the seek bar no scale DR-252 a junk float converted to a Duration, panicking the backend the instant a length-less stream appeared So the MediaPlayer contract (DR-242 … DR-247): `open` carries a start position, so no caller sequences load-then-seek and none can race an engine's load; `seek` states a destination and leaves in-place-versus-re-open to the engine; `snapshot` is one coherent read; and `Phase::Opening` names the window where intent used to be lost. One conformance suite runs against every engine — FakePlayer and mpv under cargo test, ExoPlayer instrumented on a device — so an engine is either correct or visibly failing. Two of the nine were introduced during this work and caught on hardware, not by any suite: an over-broad capability that grouped ExoPlayer with mpv, and the Duration panic. The suites test engines that behave. That is recorded in docs/native-player-verification.md, which asks for the exact action sequences that found them. Verified: all automated gates, conformance (mpv 9/9, legacy 8/9 by design, ExoPlayer 7/7 on device), and manual desktop and Android passes on real hardware. Known open and deliberately shipped: resume reads local progress and never the server's; the background-audio handoff still declares a state swap it does not confirm (the symptom is now impossible, the race is not); and `bun run android:dev` builds an APK carrying the release application id, whose failure message advises an uninstall that would destroy app data. Fix that last one before anyone else builds for Android. Squashed from worktree-linux-native-video, which keeps the per-defect history. --- .gitea/workflows/build-and-test.yml | 2 +- CHANGELOG.md | 91 + docs-site/SUMMARY.md | 3 +- docs/architecture/01-rust-backend.md | 155 +- docs/architecture/02-svelte-frontend.md | 39 + docs/architecture/03-data-flow.md | 49 + docs/native-player-verification.md | 191 + docs/requirements.md | 44 + docs/specs/README.md | 8 +- docs/specs/backend-owned-stream-selection.md | 242 - docs/specs/desktop-native-video.md | 423 + docs/specs/linux-native-video-spike.md | 70 +- docs/specs/media-player-controller.md | 324 + docs/specs/read-through-media-cache.md | 48 +- docs/traceability.md | 14377 +++++++++------- package.json | 6 +- scripts/sync-android-sources.sh | 13 + scripts/test-player-conformance.sh | 60 + src-tauri/Cargo.lock | 4 +- src-tauri/Cargo.toml | 40 +- src-tauri/android/app/build.gradle.kts | 4 + .../jellytau/player/PlayerConformanceTest.kt | 252 + .../jellytau/player/JellyTauPlayer.kt | 22 +- src-tauri/src/bin/player_conformance.rs | 37 + src-tauri/src/commands/player/mod.rs | 300 +- src-tauri/src/commands/player/queue.rs | 4 + src-tauri/src/commands/repository.rs | 31 +- src-tauri/src/commands/storage/mod.rs | 24 + src-tauri/src/conformance_runner.rs | 171 + src-tauri/src/lib.rs | 94 +- src-tauri/src/playback_mode/mod.rs | 4 + src-tauri/src/player/backend.rs | 56 + src-tauri/src/player/conformance.rs | 289 + src-tauri/src/player/fake_player.rs | 232 + .../src/player/fake_player_conformance.rs | 56 + src-tauri/src/player/legacy_player.rs | 154 + src-tauri/src/player/media.rs | 79 + src-tauri/src/player/media_player.rs | 328 + src-tauri/src/player/mod.rs | 206 +- src-tauri/src/player/mpv_backend.rs | 159 +- src-tauri/src/player/mpv_backend_test.rs | 46 + src-tauri/src/player/mpv_player.rs | 383 + src-tauri/src/player/mpv_render.rs | 405 + src-tauri/src/player/native_video.rs | 78 + src-tauri/src/player/queue.rs | 2 + src-tauri/src/player/seek.rs | 101 +- src-tauri/src/player/session.rs | 4 + src-tauri/src/player/state.rs | 2 + src-tauri/src/player/stream_end.rs | 8 + src-tauri/src/player/video_surface.rs | 412 + src-tauri/src/repository/device_profile.rs | 91 +- src-tauri/src/repository/hybrid.rs | 18 + src-tauri/src/repository/mod.rs | 3 + src-tauri/src/repository/online.rs | 1045 +- src-tauri/src/repository/stream_selection.rs | 455 + src-tauri/src/repository/types.rs | 9 + src-tauri/tauri.conf.json | 5 +- src/lib/api/bindings.ts | 281 +- src/lib/api/repository-client.ts | 30 +- .../library/EpisodeFocusView.svelte | 15 +- src/lib/components/player/Queue.svelte | 9 +- .../player/VideoPlayer.nativeReveal.test.ts | 23 +- .../VideoPlayer.scrubRegression.test.ts | 19 +- src/lib/components/player/VideoPlayer.svelte | 338 +- .../player/VideoPlayer.tapSurface.test.ts | 19 +- .../player/VideoPlayer.touchScrub.test.ts | 19 +- .../player/fullscreenTarget.test.ts | 15 + src/lib/components/player/fullscreenTarget.ts | 35 + src/lib/components/player/videoFit.test.ts | 38 +- src/lib/components/player/videoFit.ts | 29 - src/lib/player/adapters/html5Adapter.test.ts | 33 +- src/lib/player/adapters/html5Adapter.ts | 63 +- src/lib/player/adapters/nativeAdapter.test.ts | 19 +- src/lib/player/adapters/nativeAdapter.ts | 3 +- src/lib/player/adapters/types.ts | 28 +- .../player/adapters/webviewAudioAdapter.ts | 5 +- src/lib/player/index.ts | 30 +- src/lib/player/localSource.test.ts | 73 +- src/lib/player/localSource.ts | 50 - src/lib/player/streamTransport.test.ts | 93 + src/lib/player/streamTransport.ts | 88 + src/lib/stores/auth.ts | 44 + src/lib/stores/auth.waitForRepository.test.ts | 117 + src/lib/utils/duration.test.ts | 22 +- src/lib/utils/duration.ts | 38 +- src/routes/library/[id]/+page.svelte | 15 +- src/routes/player/[id]/+page.svelte | 127 +- 87 files changed, 15968 insertions(+), 7508 deletions(-) create mode 100644 docs/native-player-verification.md delete mode 100644 docs/specs/backend-owned-stream-selection.md create mode 100644 docs/specs/desktop-native-video.md create mode 100644 docs/specs/media-player-controller.md create mode 100755 scripts/test-player-conformance.sh create mode 100644 src-tauri/android/src/androidTest/java/com/dtourolle/jellytau/player/PlayerConformanceTest.kt create mode 100644 src-tauri/src/bin/player_conformance.rs create mode 100644 src-tauri/src/conformance_runner.rs create mode 100644 src-tauri/src/player/conformance.rs create mode 100644 src-tauri/src/player/fake_player.rs create mode 100644 src-tauri/src/player/fake_player_conformance.rs create mode 100644 src-tauri/src/player/legacy_player.rs create mode 100644 src-tauri/src/player/media_player.rs create mode 100644 src-tauri/src/player/mpv_player.rs create mode 100644 src-tauri/src/player/mpv_render.rs create mode 100644 src-tauri/src/player/native_video.rs create mode 100644 src-tauri/src/player/video_surface.rs create mode 100644 src-tauri/src/repository/stream_selection.rs create mode 100644 src/lib/components/player/fullscreenTarget.test.ts create mode 100644 src/lib/components/player/fullscreenTarget.ts create mode 100644 src/lib/player/streamTransport.test.ts create mode 100644 src/lib/player/streamTransport.ts create mode 100644 src/lib/stores/auth.waitForRepository.test.ts diff --git a/.gitea/workflows/build-and-test.yml b/.gitea/workflows/build-and-test.yml index e25cbecc..c5356e7a 100644 --- a/.gitea/workflows/build-and-test.yml +++ b/.gitea/workflows/build-and-test.yml @@ -109,7 +109,7 @@ jobs: # at "warn" until its class is cleared and it can be promoted to "error". # Lower this as you clear them. Never raise it to make a build pass. - name: Lint - run: bun run lint -- --max-warnings=159 + run: bun run lint -- --max-warnings=158 - name: Check TypeScript run: | diff --git a/CHANGELOG.md b/CHANGELOG.md index 757e83bf..45461db8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,97 @@ generated trace matrix lives in [docs/traceability.md](docs/traceability.md). For how long each fixed defect had been shipping before it was found, see [docs/defect-windows.md](docs/defect-windows.md). +## v0.11.0 + +Video can play through the native renderer on Linux, and the machinery every +platform's playback goes through was rebuilt around one contract. Nine defects +fell out of doing it — each one a capability the code had written down as a +fact about the platform rather than asking the thing that would know. + +### ✨ Changes + +- **Video can decode natively on Linux, without the server re-encoding it.** + Until now every video played on the desktop was transcoded by Jellyfin to + h264 and handed to the browser engine, whatever the file actually was — so the + server burned CPU on every play, and quality was capped by that conversion. + mpv can now draw the picture directly, composited beneath the interface so the + controls, subtitles and overlays still sit on top of it. Direct play means the + original file, hardware decoding, and no server work at all. This is off by + default while it settles: set `JELLYTAU_NATIVE_VIDEO=1` to try it. The browser + path is untouched and remains what you get otherwise. (UR-080 → DR-231 … + DR-237) + +- **Playback speaks one language across every player.** Linux, Android and + Windows each drove their engine through a different set of calls, and a rule + learned on one did not reach the others — which is why several of the fixes + below existed on one platform and not another. All three now go through a + single contract, and one suite of behaviours runs against every engine, + including ExoPlayer on a real device. An engine is either correct or visibly + failing. Nothing about this is visible while it works, which is the point. + (UR-081 → DR-242 … DR-247) + +### 🐛 Fixes + +- **Resuming a film starts where you left it, instead of at the beginning.** + Asking a player to open a file and asking it to start at a position were two + separate steps, and the second was issued before the first had finished — so + it failed, was discarded, and playback began at zero. It affected resume and + any skip on a stream the server was converting. The position is now part of + opening the file, so there is no gap for it to fall into. (DR-241) + +- **Skipping works on films the server is converting.** A skip was routed by the + *shape* of the stream rather than by what the player could do with it. That + happened to be right while one particular player handled those streams and + became wrong the moment another did — after which skipping simply did nothing, + silently. Players now say what they can do and are asked. (DR-238, DR-246) + +- **The play and pause button follows the player again.** The code that reacted + to pausing was never subscribed to the event it was waiting for, so the button + stayed where it was while playback did something else. (DR-239) + +- **Fullscreen fills the screen.** It expanded the page rather than the window, + which was invisible while the picture was drawn inside the page and obvious as + soon as it was not. (DR-240) + +- **The seek bar knows how long the film is.** A player that had not yet worked + out the duration reported zero, and zero was believed — leaving the bar with + no scale and nothing to drag against, even though the length had been known + since the library listed it. (DR-251) + +- **Leaving the player stops the sound.** The stop was aimed at whichever + renderer the app believed was in charge. Enabling background audio hands over + to a different one, so afterwards the app stopped something that was no longer + playing and the film carried on as an audio track in the mini player. Closing + now stops everything, regardless of who was in charge. (DR-250) + +- **Coming back from background audio no longer leaves a black screen.** The + stream that plays while the app is hidden has no fixed length, and the value a + player uses to say so is a very large negative number. Converting it crashed + the playback engine outright, which looked like a dead player with no + controls. (DR-252) + +- **Android builds again.** A rule that only applied to Linux stayed attached to + code that had stopped being Linux-only, and the Android build had not compiled + since. (DR-247) + +### 🧹 Under the hood + +- The conformance suite can be run on its own: `bun run test:player` for the + desktop engines, `bun run test:player:android` for ExoPlayer on a connected + device. Both build a test fixture rather than carrying media in the + repository. + +- [docs/native-player-verification.md](docs/native-player-verification.md) + records what to check before a release, including the exact sequences that + found two of the defects above — both of which passed every automated test. + +### Known limitations + +- Resume reads progress saved on the device, not from the server, so a fresh + install or a second device will not offer to resume something watched + elsewhere. +- Native video on Linux is opt-in and is not yet the default. + ## v0.10.1 A single fix, for something that had been quietly overriding a choice you made. diff --git a/docs-site/SUMMARY.md b/docs-site/SUMMARY.md index 45a207be..bdaebfa1 100644 --- a/docs-site/SUMMARY.md +++ b/docs-site/SUMMARY.md @@ -33,7 +33,6 @@ - [Spec Review Checklist](specs/SPEC-REVIEW-CHECKLIST.md) - [Playback Backend Unification](specs/playback-backend-unification.md) - [Linux Native Video Spike](specs/linux-native-video-spike.md) -- [Backend-Owned Stream Selection](specs/backend-owned-stream-selection.md) - [Player Facade Enforcement](specs/player-facade-enforcement.md) - [Windows Native Audio Backend](specs/windows-native-audio-backend.md) - [libmpv2 Migration](specs/libmpv2-migration.md) @@ -42,12 +41,14 @@ - [Scoped Search Boundary](specs/scoped-search-boundary.md) - [Scoped Search Boundary — Implementation](specs/scoped-search-boundary-implementation.md) - [Frontend Domain Model](specs/frontend-domain-model.md) +- [Desktop Native Video](specs/desktop-native-video.md) - [Build Provenance](specs/build-provenance.md) # Build & Release - [Build & Release](build/build-release.md) - [Release Checklist](release-checklist.md) +- [Native Player Verification](native-player-verification.md) - [Desktop Packaging](build/build-desktop-packages.md) - [Windows Build](build/build-windows.md) - [Defect Windows](defect-windows.md) diff --git a/docs/architecture/01-rust-backend.md b/docs/architecture/01-rust-backend.md index 8486aeb0..cbd7b93b 100644 --- a/docs/architecture/01-rust-backend.md +++ b/docs/architecture/01-rust-backend.md @@ -673,8 +673,161 @@ device profile. Sending it there — not just on the transcode URL — is what m the cap real: a stream the server decides to *direct play* is served at the source file's own bitrate, and no URL parameter afterwards can reduce it. +#### Two levels of ceiling + +**Location**: `src-tauri/src/repository/online.rs` (TRACES: UR-074, UR-079 | DR-226) + +There are two, and they are not the same thing: + +| | Set by | Lives until | Read via | +|---|---|---|---| +| **Device default** | Settings (`player_set_video_settings`) | Persisted; restored at startup | `streaming_quality()` | +| **Per-playback override** | The in-player picker (`player_set_stream_quality`) | The next item starts playing | `playback_quality_override()` | + +`effective_streaming_quality()` resolves the pair — override first, else default — +and **is the only thing stream construction may read**. Every URL builder and the +`PlaybackInfo` negotiation go through it, for the reason the process-wide static +existed in the first place: if the negotiation and the URL builder disagree, the +cap leaks — the negotiation authorises a direct play the builder then never gets +to constrain, or the reverse. + +> The override exists because a single global cannot express "this 4K remux needs +> a ceiling, that podcast does not". The picker had documented itself as a "this +> film, this connection" control since it was written, but was implemented by +> writing the *default* — so dropping one awkward film to 2 Mbps silently capped +> every video played afterwards for the rest of the process, with Settings still +> showing the old value. It is cleared on every `player_play_item` / +> `player_play_queue` / `player_play_tracks`, which is what stops it surviving +> into an autoplayed next episode where nobody would reopen the picker. + +### Stream selection + +**Location**: `src-tauri/src/repository/stream_selection.rs`, +`OnlineRepository::get_stream_selection` (TRACES: UR-070, UR-079 | DR-225, DR-227, DR-228) + +**Rust decides *what stream*. The player decides *how to deliver it*.** That line +is the whole design. A backend with genuine adaptive selection (ExoPlayer over a +multi-variant playlist) is left to do it; Rust chooses what to request and never +paces bytes. + +`get_stream_selection` returns one self-describing `StreamSelection` in place of +the bare URL `get_video_stream_url` used to hand out: + +| Field | Carries | +|---|---| +| `url` | What to open | +| `transport` | `Hls` / `Progressive` / `LocalFile` — how to fetch it | +| `playback_kind` | `DirectPlay` / `DirectStream` / `Transcode` — what the server is doing to the source | +| `rendition` | The negotiated ceiling and codecs; `None` for a direct play, which *is* the source | +| `available` | The quality ladder as it applies to this media source (DR-227) | +| `needs_transcoding` | Derived from `playback_kind`, so the rule is answered once | + +Both enums are serde-tagged (`{"type":"hls"}`) so the frontend matches a +discriminant rather than comparing text. + +> **Why `transport` exists.** `VideoPlayer.svelte` chose its loader with +> `url.includes(".m3u8")`, in two places. Rust *built* that URL and knows exactly +> what it is; re-deriving it downstream by substring match is a domain fact +> reconstructed in the presentation layer — the same class of error as leaking +> item-type taxonomy, and one that fails silently in **both** directions: a +> progressive file served from a path containing the substring gets an HLS +> loader, and a playlist served from a path without it does not. +> +> The paths that never negotiate get the same shape from Rust rather than letting +> a caller assemble one — `media_local_selection` for a downloaded file, +> `LiveStreamInfo.transport` for a live channel — so there is no second place +> where a transport is decided. + +#### The playback-kind decision + +`decide_playback_kind` is a free function and pure, so every branch is testable +from `PlaybackInfo` fixtures without a server. Order matters — the two +client-side overrides come first, because each describes a case where the +server's answer is right about the *file* and wrong about what this app will do +with it: + +1. **Undecodable audio → `Transcode`.** Jellyfin 10.11.5 honours a + DirectPlayProfile's container and video codec but *ignores its audio codec*, + so it offers direct play for an E-AC-3 track the webview renders in silence. + A silent direct play is worse than a transcode. +2. **A pinned audio track → `Transcode`.** Not a defect in the server's answer, a + different question: the file has one default track and the viewer asked for + another. +3. Otherwise `supports_direct_play` → `DirectPlay`, else `supports_direct_stream` + → `DirectStream`, else `Transcode`. + +A direct **stream** is a remux — codecs copied, container repackaged. It is cheap +and is deliberately *not* counted as transcoding; conflating the two would report +a free passthrough as a server-side re-encode. + +> **What this is worth, measured.** Against the development server (Jellyfin +> 10.11.5), 400 items sampled for codec mix and 40 put through a real negotiation +> per profile: +> +> | Profile | Direct play | +> |---|---| +> | Linux / WebKitGTK (`h264` only, 2ch) | 3/40 — **7%** | +> | Android / ExoPlayer (`h264,hevc,vp8,vp9,av1,mpeg4` + `ac3,eac3`, 6ch) | 34/40 — **85%** | +> +> The library is ~80% hevc (`hevc+eac3` alone is a third of it), which is why the +> two diverge so hard. +> +> **Read that 85% as a ceiling, not a result.** It was measured with a profile +> containing `ac3,eac3`. The Android device this was later run on reports neither +> in its `MediaCodecList` — no Dolby licence, which is normal for a tablet — so +> eac3 content, about a third of the sampled library, correctly transcodes there. +> What any given device achieves depends on its own codec list, and on the +> profile being derived from the renderer at all (DR-234), which it was not when +> the figure was taken. +> +> **The payoff is still overwhelmingly Android**, because that is where a real +> decoder is already doing the work. Linux stays near 7% until libmpv decodes the +> picture — the h264-only profile is a WebKitGTK constraint, not a JellyTau +> choice, and is what `linux-native-video-spike.md` exists to remove. A reviewer +> should not expect this code to fix Linux on its own. + +#### The quality ladder per source + +`quality_options_for_source(source_bitrate)` returns every rung, each marked with +`exceeds_source`: true when that rung's ceiling is at or above what the source +itself carries, so selecting it produces the same bytes as `Original`. The +frontend draws the list and drops the redundant rungs; it does not decide which +they are. + +- `Original` is never marked — it *is* the source. +- An unreported source bitrate (some containers have none; the sampled library + has `avi` files with no bitrate at all) marks **nothing** redundant, keeping + every rung offered. That is the safe direction: the viewer keeps every choice. + +#### No adaptive ladder to preserve + +**TRACES: UR-079 | DR-229 (Won't Do)** + +Mid-playback re-negotiation on throughput was scoped and dropped on measurement. +A master playlist from this server carries exactly **one** `EXT-X-STREAM-INF`: +Jellyfin builds it from the single rendition the request asked for rather than +publishing a ladder. So there is no adaptation for hls.js to be preserving and +none that mpv would lose — the claim that there was is recorded in +`playback-backend-unification.md` and does not hold. "Adapt mid-stream" collapses +into "pick well at open", which is what the two levels of ceiling and the +per-source ladder already are. + +Kept here because it is a measurement, not an opinion: a server that *does* +publish a ladder would change the answer, and the re-negotiation path below is +the hook that work would build on. + +#### Re-negotiation + +One mechanism, not two. `player_seek_video`, `player_switch_audio_track` and +`player_set_stream_quality` all return a tagged `strategy` saying who reloads — +the backend handles a native backend itself and hands the webview a +`StreamSelection` for `reloadSource`. Note the wire wart: tauri-specta keeps +these response fields snake_case (`seek_offset`), while the `strategy` tag itself +is camelCase. + The frontend names a variant and nothing else; the labels the picker shows are -served over IPC by `player_get_streaming_qualities`. +served over IPC — from `available` on the selection, or +`player_get_streaming_qualities` for the Settings list. ## Background workers diff --git a/docs/architecture/02-svelte-frontend.md b/docs/architecture/02-svelte-frontend.md index 539253bb..93d2a8e0 100644 --- a/docs/architecture/02-svelte-frontend.md +++ b/docs/architecture/02-svelte-frontend.md @@ -802,6 +802,45 @@ by exactly the inset. Unlike `addJavascriptInterface`, the inset push only writes CSS properties, so it can safely be re-sent on resume. +## Stream Transport + +**Location**: `src/lib/player/streamTransport.ts` +**TRACES**: UR-079 | DR-225 | UT-214 + +`videoLoaderFor(selection, capabilities)` picks the loader for the webview +`