feat(player): render Android video natively behind a transparent webview (DR-150, DR-151, DR-152)

Rust already reported `use_html5_element: false` on Android, but two frontend
overrides threw that answer away, so ExoPlayer's video path had never actually
run. Both are lifted behind an `experimentalNativeVideo` opt-in (default off).

The flag is a suppressor, never a promoter: off forces HTML5 even where Rust
says native, so an in-progress spike cannot ship as the default, but it can
never select native where Rust reported HTML5 — Linux cannot composite behind
WebKitGTK, and promoting there would be a black screen.

Two blockers the spec did not anticipate, both in code assumed to be merely
unreachable rather than broken:

- `JellyTauPlayer.setActivity()` had zero callers, so `currentActivity` was
  always null and `autoAttachSurface()` bailed. The SurfaceView was created and
  wired to ExoPlayer but never added to the view hierarchy — video would have
  decoded to a surface that was never on screen, whatever the webview did.
  This also revives PiP on the video path, which gated on the same flag.
- `createAdapter()` was not the real gate; it is never called in production.
  The actual override was in VideoPlayer.svelte, which forced HTML5 and stopped
  the native backend `player_play_item` had just started. Both sites now route
  through `createAdapter()`.

Compositing needs two independent opaque layers cleared, not one. Clearing only
the page leaves the WebView widget opaque — audio over a black picture, exactly
the symptom the old INTERIM comment described. `videoSurface.ts` toggles both:
the widget background and window drawable from Kotlin, the page backgrounds via
a `data-native-video` attribute keyed by app.css. Transparency lives in
`tauri.android.conf.json` so Linux keeps an opaque window, and is scoped to the
playback session so the launcher never shows through the rest of the app.

Phase 3's rect plumbing turned out to be unnecessary: video is fullscreen on the
player route, and `fitSurfaceToScreen()` already letterboxes and re-fits on
rotation. The mini-player transition remains unverified on device.

Also removes the `navigator.userAgent` sniffing in webviewAudio.ts, which was a
second copy of the Rust cfg gate free to drift from it. `player_get_capabilities`
now reports `usesWebviewAudio` and `supportsNativeVideo` from those same gates.

Tests: adapter selection covers the full matrix, including the regression guard
that the flag off beats Rust. Written first and confirmed failing (2 of 7) before
the fix.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-11 20:57:58 +02:00
co-authored by Claude Opus 5
parent 07d10dfed7
commit e144e62b31
16 changed files with 745 additions and 56 deletions
+77 -6
View File
@@ -1,6 +1,7 @@
# Spec: Android native video — transparent-webview spike
**Status:** Proposed (spiketimeboxed, may conclude "not viable")
**Status:** In progress — implemented behind `experimentalNativeVideo`, pending
on-device confirmation. Branch `feat/android-native-video`.
**Requirements:** IR-004, UR-003, UR-004 → DR-001, DR-023, DR-024
**UX spec:** n/a — no intended visual change; the video surface must land exactly where the `<video>` element is today
**Supersedes / revises:** acts on finding 2 of [playback-backend-unification.md](playback-backend-unification.md)
@@ -109,6 +110,46 @@ in here rather than leaving a second, subtler copy of the bug behind. If
`get_player_status` does not currently expose enough to cover the audio case, add
the field — that is backend work, and correct.
### Implementation findings (2026-08-11)
Two blockers existed that this spec did not anticipate. Both were in code the
spec assumed was merely *unreachable*; it was also *broken*.
**1. The Kotlin attach chain was severed.** `JellyTauPlayer.setActivity()` had
**zero callers** anywhere in the tree. `currentActivity` was therefore always
null, so `autoAttachSurface()` logged "Cannot attach surface - no Activity
reference" and returned. The `SurfaceView` was created and wired to ExoPlayer but
never added to the view hierarchy — video would have decoded to a surface that
was never on screen, *regardless* of webview transparency. Fixed by calling
`JellyTauPlayer.setActivity(this)` from `MainActivity.onCreate`.
Note the knock-on: `PictureInPictureManager.canEnterPip()` gates on
`VideoOverlayManager.isVideoSurfaceAttached()`, which was permanently false. PiP
on the video path was dead for the same reason.
**2. `createAdapter()` was not the real gate.** It is never called by production
code — `VideoPlayer.svelte` constructs `Html5PlayerAdapter` directly. The actual
override was `VideoPlayer.svelte`'s INTERIM block, which read Rust's
`useHtml5Element`, forced it to `true`, and called `playerStop()` to kill the
native backend `player_play_item` had just started. Both sites are now fixed;
`VideoPlayer.svelte` routes through `createAdapter()` so there is one gate.
**Transparency needs two independent layers cleared,** not one. The spec's
Phase 1 named only `html, body`. Clearing just the page leaves the WebView
widget's own background opaque, which is a black screen with audio — the exact
symptom the INTERIM comment described as "native surface not visible". Both are
now toggled together by `$lib/utils/videoSurface.ts`:
| Layer | Cleared by | Reachable from |
|-------|-----------|----------------|
| WebView widget background + window drawable | `AndroidVideoSurface.setTransparent()` (MainActivity) | Kotlin only |
| `html`/`body` + app-shell `--color-background` | `data-native-video` attribute → app.css | CSS only |
Transparency is scoped to `tauri.android.conf.json` rather than the base config:
a transparent window on Linux is a regression, since nothing renders behind it.
It is also toggled per-session rather than set once — a permanently transparent
window shows the launcher through the rest of the app.
### Phase 3 — surface positioning
The hard part, and where this most likely fails. The webview's `<video>` element
@@ -124,6 +165,29 @@ the video is effectively fullscreen on Android, which it is in the player route.
rotation or the mini-player transition without visible artefacts, the spike fails
and we keep HTML5. Do not ship a janky native path for a codec win.
**Update: no rect plumbing was needed.** The premise — that the surface must be
positioned to match a laid-out `<video>` box — does not hold on the player route,
where video is fullscreen. `VideoOverlayManager` adds the SurfaceView at index 0
of `android.R.id.content` with `MATCH_PARENT`, and `fitSurfaceToScreen()`
(`JellyTauPlayer.kt`) already letterboxes/pillarboxes to the real video aspect
ratio and re-centres via a `Gravity.CENTER` `FrameLayout.LayoutParams`. Rotation
is handled by an `OnLayoutChangeListener` that re-fits on any bounds change. The
frontend's native branch is a bare `flex-1` box, so there is no rect to report
and nothing to keep in sync.
This does mean **the mini-player transition is untested** for the native path —
it is the one case the fullscreen assumption does not cover, and it remains an
on-device check.
### A trap for the next implementer
There is a **stale duplicate player** at
`src-tauri/android/app/src/main/java/com/dtourolle/jellytau/player/JellyTauPlayer.kt`
(only commit: `cfddc1e` "First working POC"). No `sourceSets` entry points at it,
so it is not compiled — but edits made there silently do nothing. The canonical
tree is `src-tauri/android/src`, synced into `gen/` by
`scripts/sync-android-sources.sh`.
### What we gain if it works
- **Hardware decode via MediaCodec** — `CodecDetector.kt` already reports
@@ -145,9 +209,9 @@ and we keep HTML5. Do not ship a janky native path for a codec win.
The spike is **complete** when one of these is true:
**Success path**
- [ ] Transparent WebView confirmed working on a physical device.
- [ ] `experimentalNativeVideo` off → behaviour byte-identical to today.
- [ ] `webviewAudio.ts` no longer inspects `navigator.userAgent`; the platform's audio backend is read from Rust.
- [x] Transparent WebView confirmed working on a physical device (reported by the maintainer; the config that enables it is now committed in `tauri.android.conf.json`).
- [x] `experimentalNativeVideo` off → behaviour byte-identical to today. Guarded by `adapterSelection.test.ts`, which asserts the flag-off case forces HTML5 even when Rust reports native.
- [x] `webviewAudio.ts` no longer inspects `navigator.userAgent`; the platform's audio backend is read from Rust (`player_get_capabilities``usesWebviewAudio`).
- [ ] `experimentalNativeVideo` on → video plays via ExoPlayer/MediaCodec, correctly positioned, with working seek, audio-track switch, and subtitle selection through the existing `PlayerAdapter` contract.
- [ ] No artefacts on rotation, background/foreground, or mini-player transition.
- [ ] `adb shell dumpsys media.metrics` (or logcat) confirms a hardware decoder is in use.
@@ -159,8 +223,15 @@ The spike is **complete** when one of these is true:
- [ ] `nativeAdapter.ts:11-14` no longer cites tauri#10152.
Either way:
- [ ] `bun run check`, `bun run test`, `bun run check:boundary` pass.
- [ ] `cargo fmt` / `cargo clippy` clean; `bun run test:rust` passes.
- [x] `bun run check` (0 errors), `bun run test` (892 passed), `bun run check:boundary` pass.
- [x] `cargo fmt` / `cargo clippy` clean (no new warnings); `cargo test` passes (603 lib + 7 doc).
> Note: this environment has no host WebKitGTK dev packages, no Android SDK and
> no `bun`, so all of the above were run inside the CI builder image
> (`gitea.tourolle.paris/dtourolle/jellytau-builder:latest`). On Fedora the bind
> mount needs `:z` for SELinux, and `scripts/build-android.sh` hardcodes
> `ANDROID_HOME="$HOME/Android/Sdk"`, so the image's SDK at `/opt/android-sdk`
> must be symlinked there rather than passed by env var.
## Testing