feat(player): render Android video natively behind a transparent webview (DR-150, DR-151, DR-152)

Rust already reported `use_html5_element: false` on Android, but two frontend
overrides threw that answer away, so ExoPlayer's video path had never actually
run. Both are lifted behind an `experimentalNativeVideo` opt-in (default off).

The flag is a suppressor, never a promoter: off forces HTML5 even where Rust
says native, so an in-progress spike cannot ship as the default, but it can
never select native where Rust reported HTML5 — Linux cannot composite behind
WebKitGTK, and promoting there would be a black screen.

Two blockers the spec did not anticipate, both in code assumed to be merely
unreachable rather than broken:

- `JellyTauPlayer.setActivity()` had zero callers, so `currentActivity` was
  always null and `autoAttachSurface()` bailed. The SurfaceView was created and
  wired to ExoPlayer but never added to the view hierarchy — video would have
  decoded to a surface that was never on screen, whatever the webview did.
  This also revives PiP on the video path, which gated on the same flag.
- `createAdapter()` was not the real gate; it is never called in production.
  The actual override was in VideoPlayer.svelte, which forced HTML5 and stopped
  the native backend `player_play_item` had just started. Both sites now route
  through `createAdapter()`.

Compositing needs two independent opaque layers cleared, not one. Clearing only
the page leaves the WebView widget opaque — audio over a black picture, exactly
the symptom the old INTERIM comment described. `videoSurface.ts` toggles both:
the widget background and window drawable from Kotlin, the page backgrounds via
a `data-native-video` attribute keyed by app.css. Transparency lives in
`tauri.android.conf.json` so Linux keeps an opaque window, and is scoped to the
playback session so the launcher never shows through the rest of the app.

Phase 3's rect plumbing turned out to be unnecessary: video is fullscreen on the
player route, and `fitSurfaceToScreen()` already letterboxes and re-fits on
rotation. The mini-player transition remains unverified on device.

Also removes the `navigator.userAgent` sniffing in webviewAudio.ts, which was a
second copy of the Rust cfg gate free to drift from it. `player_get_capabilities`
now reports `usesWebviewAudio` and `supportsNativeVideo` from those same gates.

Tests: adapter selection covers the full matrix, including the regression guard
that the flag off beats Rust. Written first and confirmed failing (2 of 7) before
the fix.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-11 20:57:58 +02:00
co-authored by Claude Opus 5
parent 07d10dfed7
commit e144e62b31
16 changed files with 745 additions and 56 deletions
+4
View File
@@ -312,6 +312,9 @@ Internal architecture, components, and application logic.
| DR-146 | The no-audio-track fallback picks a track the renderer can actually play. When ExoPlayer selected no audio track, the recovery forced group 0 / track 0 unconditionally — but the most likely reason nothing was selected is that this very track cannot be decoded on this device, so the override reinstated the silence it was meant to fix. It now scans the groups for the first `isTrackSupported` track and overrides to that, and clears `setTrackTypeDisabled(TRACK_TYPE_AUDIO)` because audio may equally have been off at the type level, which an override alone does not undo. When no group holds a supported track the condition is logged as an error — the server was expected to transcode — rather than leaving a silent video with no explanation in the log | Playback | UR-004 | Done |
| DR-148 | The video direct-play profile advertises only what the **webview** can decode. The audio codec list comes from `MediaCodecList`, which describes ExoPlayer — but video does not play through ExoPlayer on either platform: Android force-renders every video in the webview `<video>` element (the interim override in `VideoPlayer.svelte`, because the native SurfaceView sits behind an opaque webview) and Linux always has. Chromium and WebKit decode a far narrower set than the platform does, and the gap is widest on devices whose vendor licenses Dolby: a phone shipping `/vendor/etc/media_codecs_dolby_audio.xml` reports `ac3,eac3`, so Jellyfin direct-played an E-AC-3 track with `static=true` and the webview built a video decoder and no audio decoder at all — full picture, no sound. The defect is triggered by *capability*, not the lack of it, which is why it reproduced on one Motorola while a Fairphone and an Honor tablet played the same file on the same build: a device without the Dolby decoder never claims the codec, so the server transcodes to AAC and it plays. `video_audio_codecs` narrows the platform list to the webview-decodable set (`aac,mp3,opus,vorbis,flac`) for the video direct-play profile *only* — the audio-only profile keeps the full list, since that playback really is the native player's and narrowing it would transcode music that plays perfectly well. A list with nothing decodable still claims `aac` rather than going out empty, because a profile that claims nothing invites the server to give up instead of transcoding. The video codec list is deliberately untouched: HEVC direct-plays through the webview correctly, so the constraint is specific to audio | Playback | UR-004 | Done |
| DR-149 | The client decides whether its own renderer can decode the audio, rather than trusting the server's negotiation. Advertising a webview-shaped profile (DR-148) turned out to be necessary but not sufficient: Jellyfin 10.11.5 enforces a `DirectPlayProfile`'s `Container` and `VideoCodec` — excluding either returns `SupportsDirectPlay: false` with `TranscodeReasons=ContainerNotSupported` / `VideoCodecNotSupported` — but **ignores its `AudioCodec`**, offering an E-AC-3 track for direct play against a profile listing only `aac,flac,mp3,opus,vorbis`. Neither a `VideoAudio` `CodecProfile` forbidding the codec nor a `MaxAudioChannels: 2` against a 6-channel track changes the answer, so no profile the client can send fixes it and the picture plays silent. The negotiated source's audio is therefore checked locally against what the webview decodes, and an undecodable track forces the existing h264/aac HLS transcode URL regardless of the server saying direct play is fine — `direct_play` and `needs_transcoding` are corrected to match, so the frontend and the reporting path agree with the URL actually used. The track judged is the one the server would serve: the default, or the first when nothing is marked default, since a supported track further down the list is not the one that plays. A source with no audio streams, or a stream whose codec the server did not name, is left alone — forcing a transcode on a guess spends server CPU on files that already play | Playback | UR-004 | Done |
| DR-150 | Android video renders on the native ExoPlayer surface behind a transparent WebView, behind the `experimentalNativeVideo` opt-in. Rust already reported `use_html5_element: false` on Android, but two frontend overrides discarded it — `createAdapter()` hardcoded `"html5"`, and `VideoPlayer.svelte` forced `useHtml5Element = true` and stopped the native backend `player_play_item` had just started. The flag is a **suppressor, never a promoter**: off forces HTML5 even where Rust says native, so an in-progress spike cannot ship as the default, but it can never select native where Rust reported HTML5 (Linux cannot composite behind WebKitGTK, so promoting there is a black screen). Compositing requires clearing two independent opaque layers, and clearing only one leaves audio over a black picture — the WebView widget background and window drawable from Kotlin (`AndroidVideoSurface.setTransparent`), and the page's `html`/`body` and app-shell background from CSS (`data-native-video`). Transparency is declared in `tauri.android.conf.json` rather than the base config, because a transparent window on Linux has nothing behind it, and is toggled per playback session rather than set once, because a permanently transparent window shows the launcher through the rest of the app | Playback | UR-003, UR-004 | In Progress |
| DR-151 | The player's video SurfaceView actually reaches the view hierarchy. `JellyTauPlayer.setActivity()` had zero callers, so `currentActivity` was always null and `autoAttachSurface()` returned at "Cannot attach surface - no Activity reference". The surface was created and handed to ExoPlayer but never added to the content view, so native video decoded to a surface that was never on screen — independent of any webview transparency. `MainActivity.onCreate` now supplies the reference, which also revives PiP on the video path: `canEnterPip()` gates on `isVideoSurfaceAttached()`, which had been permanently false | Playback | UR-003, UR-041 | Done |
| DR-152 | Platform playback facilities are reported by Rust, not sniffed from the user agent. `webviewAudio.ts` re-derived "does this platform have a native audio backend" by matching `navigator.userAgent` against `android`/`linux` — a second copy of the `cfg!` gate the backends are compiled under, free to drift from it. `player_get_capabilities` now returns `usesWebviewAudio` and `supportsNativeVideo` from the same cfg gates, and the frontend consumes them; the settings toggle for native video is hidden entirely where the platform cannot support it | Player | UR-003, UR-005 | Done |
| DR-143 | Flipping the offline downloaded-only gate actually re-queries the listing. The gate (DR-078) is a process-wide flag in Rust consulted only *while a query runs*, but no library surface re-queried when its inputs changed: `useServerReachabilityReload` fires only on the offline → **online** transition, and `GenericMediaListPage`, `GenericGenreBrowser` and the favourites page never even called its `checkServerReachability`. So going offline left the full server catalog on screen under a now-closed gate, and toggling "Show all server media" only greyed cards — `MediaCard.isServerOnly` is a pure frontend derivation that updates instantly — without adding or removing a single row. The filter therefore read as "shows everything until I filter, then greys some of it" while the backend gate was correct and simply never exercised. `catalogFilterVersion` is the refetch signal: `pushCatalogVisibility` now awaits `set_show_server_catalog` and bumps the version only **after** the backend accepts the new flag, since a reload racing the push would re-query under the old gate and undo itself. A failed push clears `lastIncludeCatalog` instead of latching it, so the next identical transition is retried rather than skipped as a no-op and left permanently disagreeing with the backend. `useOfflineFilterReload` subscribes pages to that signal, skipping the value they already loaded under; it is wired into both generic list components and the movies/music/tv/favourites landing pages and the `/library/[id]` detail page | UI | UR-052 | Done |
| DR-135 | A download's media type comes from the item, not a default. `download_item` — the path a media card uses to queue an item while offline — never records `media_type`, and the reconnect resolver read that NULL as `'audio'`, so a **movie** queued from a card had its URL resolved by `get_audio_stream_url`. The file that landed on disk was an audio-only transcode, which is why a "downloaded" film could never play offline no matter how the path or protocol was fixed. The resolver now falls back to the item's own `item_type` (`VIDEO_ITEM_TYPES` in Rust, so the frontend never learns which types are video) and only defaults to audio when the item is not cached locally. An explicit `media_type` on the row still wins | Downloads | UR-071, UR-052 | Done |
| DR-136 | Rows already downloaded under the audio default are repaired, not just prevented. They are identifiable after the fact — no `media_type`, but a video item — so on reconnect they are reset to `pending` with their audio URL cleared and re-resolved by DR-135's corrected logic, overwriting the audio file in place. Without this the fix is invisible to anyone who had already queued a film: the row still reads "downloaded" and still fails to play. Rows carrying an explicit `media_type` and genuine audio downloads are left untouched | Downloads | UR-071 | Done |
@@ -543,6 +546,7 @@ Internal architecture, components, and application logic.
| UT-140 | `useOfflineFilterReload` skips the value a page already loaded under and reloads on each later change | DR-143 | Done |
| UT-141 | The advertised channel cap: an unknown or zero reading falls back to stereo, a real route keeps its channels, an absurd driver reading is capped at 7.1, and mono is taken at its word | DR-141 | Done |
| UT-148 | Forcing a transcode from the client: an undecodable default track forces one, a decodable track does not, the default track decides rather than the first, the first decides when nothing is marked default, and neither an audio-less source nor an unnamed codec is second-guessed | DR-149 | Done |
| UT-149 | `createAdapter` returns the native adapter only when Rust reports native AND `experimentalNativeVideo` is on; the flag off forces HTML5 even when Rust says native, and the flag on never promotes a platform Rust reported as HTML5 | DR-150 | Done |
| UT-142 | The audio codecs offered for video direct play: a Dolby device's real `MediaCodecList` output drops `ac3`/`eac3`, AMR and raw PCM are dropped too, a fully-supported list is passed through untouched, a list with nothing decodable still claims `aac`, and stray spacing or casing does not decide whether the user gets sound | DR-148 | Done |
| UT-143 | Subtitle URLs resolve to plain strings before they reach the markup (never a Promise), unresolvable tracks are dropped, a stale selection collapses to "Off", and a server-default track is never auto-selected | UR-020, DR-023 | Done |
| UT-144 | VideoPlayer actually renders `<track kind="subtitles">` children carrying `data-stream-index`, with no `default` attribute and no async `getSubtitleUrl()` bound to `src` | UR-020, DR-023 | Done |
+77 -6
View File
@@ -1,6 +1,7 @@
# Spec: Android native video — transparent-webview spike
**Status:** Proposed (spiketimeboxed, may conclude "not viable")
**Status:** In progress — implemented behind `experimentalNativeVideo`, pending
on-device confirmation. Branch `feat/android-native-video`.
**Requirements:** IR-004, UR-003, UR-004 → DR-001, DR-023, DR-024
**UX spec:** n/a — no intended visual change; the video surface must land exactly where the `<video>` element is today
**Supersedes / revises:** acts on finding 2 of [playback-backend-unification.md](playback-backend-unification.md)
@@ -109,6 +110,46 @@ in here rather than leaving a second, subtler copy of the bug behind. If
`get_player_status` does not currently expose enough to cover the audio case, add
the field — that is backend work, and correct.
### Implementation findings (2026-08-11)
Two blockers existed that this spec did not anticipate. Both were in code the
spec assumed was merely *unreachable*; it was also *broken*.
**1. The Kotlin attach chain was severed.** `JellyTauPlayer.setActivity()` had
**zero callers** anywhere in the tree. `currentActivity` was therefore always
null, so `autoAttachSurface()` logged "Cannot attach surface - no Activity
reference" and returned. The `SurfaceView` was created and wired to ExoPlayer but
never added to the view hierarchy — video would have decoded to a surface that
was never on screen, *regardless* of webview transparency. Fixed by calling
`JellyTauPlayer.setActivity(this)` from `MainActivity.onCreate`.
Note the knock-on: `PictureInPictureManager.canEnterPip()` gates on
`VideoOverlayManager.isVideoSurfaceAttached()`, which was permanently false. PiP
on the video path was dead for the same reason.
**2. `createAdapter()` was not the real gate.** It is never called by production
code — `VideoPlayer.svelte` constructs `Html5PlayerAdapter` directly. The actual
override was `VideoPlayer.svelte`'s INTERIM block, which read Rust's
`useHtml5Element`, forced it to `true`, and called `playerStop()` to kill the
native backend `player_play_item` had just started. Both sites are now fixed;
`VideoPlayer.svelte` routes through `createAdapter()` so there is one gate.
**Transparency needs two independent layers cleared,** not one. The spec's
Phase 1 named only `html, body`. Clearing just the page leaves the WebView
widget's own background opaque, which is a black screen with audio — the exact
symptom the INTERIM comment described as "native surface not visible". Both are
now toggled together by `$lib/utils/videoSurface.ts`:
| Layer | Cleared by | Reachable from |
|-------|-----------|----------------|
| WebView widget background + window drawable | `AndroidVideoSurface.setTransparent()` (MainActivity) | Kotlin only |
| `html`/`body` + app-shell `--color-background` | `data-native-video` attribute → app.css | CSS only |
Transparency is scoped to `tauri.android.conf.json` rather than the base config:
a transparent window on Linux is a regression, since nothing renders behind it.
It is also toggled per-session rather than set once — a permanently transparent
window shows the launcher through the rest of the app.
### Phase 3 — surface positioning
The hard part, and where this most likely fails. The webview's `<video>` element
@@ -124,6 +165,29 @@ the video is effectively fullscreen on Android, which it is in the player route.
rotation or the mini-player transition without visible artefacts, the spike fails
and we keep HTML5. Do not ship a janky native path for a codec win.
**Update: no rect plumbing was needed.** The premise — that the surface must be
positioned to match a laid-out `<video>` box — does not hold on the player route,
where video is fullscreen. `VideoOverlayManager` adds the SurfaceView at index 0
of `android.R.id.content` with `MATCH_PARENT`, and `fitSurfaceToScreen()`
(`JellyTauPlayer.kt`) already letterboxes/pillarboxes to the real video aspect
ratio and re-centres via a `Gravity.CENTER` `FrameLayout.LayoutParams`. Rotation
is handled by an `OnLayoutChangeListener` that re-fits on any bounds change. The
frontend's native branch is a bare `flex-1` box, so there is no rect to report
and nothing to keep in sync.
This does mean **the mini-player transition is untested** for the native path —
it is the one case the fullscreen assumption does not cover, and it remains an
on-device check.
### A trap for the next implementer
There is a **stale duplicate player** at
`src-tauri/android/app/src/main/java/com/dtourolle/jellytau/player/JellyTauPlayer.kt`
(only commit: `cfddc1e` "First working POC"). No `sourceSets` entry points at it,
so it is not compiled — but edits made there silently do nothing. The canonical
tree is `src-tauri/android/src`, synced into `gen/` by
`scripts/sync-android-sources.sh`.
### What we gain if it works
- **Hardware decode via MediaCodec**`CodecDetector.kt` already reports
@@ -145,9 +209,9 @@ and we keep HTML5. Do not ship a janky native path for a codec win.
The spike is **complete** when one of these is true:
**Success path**
- [ ] Transparent WebView confirmed working on a physical device.
- [ ] `experimentalNativeVideo` off → behaviour byte-identical to today.
- [ ] `webviewAudio.ts` no longer inspects `navigator.userAgent`; the platform's audio backend is read from Rust.
- [x] Transparent WebView confirmed working on a physical device (reported by the maintainer; the config that enables it is now committed in `tauri.android.conf.json`).
- [x] `experimentalNativeVideo` off → behaviour byte-identical to today. Guarded by `adapterSelection.test.ts`, which asserts the flag-off case forces HTML5 even when Rust reports native.
- [x] `webviewAudio.ts` no longer inspects `navigator.userAgent`; the platform's audio backend is read from Rust (`player_get_capabilities``usesWebviewAudio`).
- [ ] `experimentalNativeVideo` on → video plays via ExoPlayer/MediaCodec, correctly positioned, with working seek, audio-track switch, and subtitle selection through the existing `PlayerAdapter` contract.
- [ ] No artefacts on rotation, background/foreground, or mini-player transition.
- [ ] `adb shell dumpsys media.metrics` (or logcat) confirms a hardware decoder is in use.
@@ -159,8 +223,15 @@ The spike is **complete** when one of these is true:
- [ ] `nativeAdapter.ts:11-14` no longer cites tauri#10152.
Either way:
- [ ] `bun run check`, `bun run test`, `bun run check:boundary` pass.
- [ ] `cargo fmt` / `cargo clippy` clean; `bun run test:rust` passes.
- [x] `bun run check` (0 errors), `bun run test` (892 passed), `bun run check:boundary` pass.
- [x] `cargo fmt` / `cargo clippy` clean (no new warnings); `cargo test` passes (603 lib + 7 doc).
> Note: this environment has no host WebKitGTK dev packages, no Android SDK and
> no `bun`, so all of the above were run inside the CI builder image
> (`gitea.tourolle.paris/dtourolle/jellytau-builder:latest`). On Fedora the bind
> mount needs `:z` for SELinux, and `scripts/build-android.sh` hardcodes
> `ANDROID_HOME="$HOME/Android/Sdk"`, so the image's SDK at `/opt/android-sdk`
> must be symlinked there rather than passed by env var.
## Testing