feat(player): render Android video natively behind a transparent webview (DR-150, DR-151, DR-152)
Rust already reported `use_html5_element: false` on Android, but two frontend overrides threw that answer away, so ExoPlayer's video path had never actually run. Both are lifted behind an `experimentalNativeVideo` opt-in (default off). The flag is a suppressor, never a promoter: off forces HTML5 even where Rust says native, so an in-progress spike cannot ship as the default, but it can never select native where Rust reported HTML5 — Linux cannot composite behind WebKitGTK, and promoting there would be a black screen. Two blockers the spec did not anticipate, both in code assumed to be merely unreachable rather than broken: - `JellyTauPlayer.setActivity()` had zero callers, so `currentActivity` was always null and `autoAttachSurface()` bailed. The SurfaceView was created and wired to ExoPlayer but never added to the view hierarchy — video would have decoded to a surface that was never on screen, whatever the webview did. This also revives PiP on the video path, which gated on the same flag. - `createAdapter()` was not the real gate; it is never called in production. The actual override was in VideoPlayer.svelte, which forced HTML5 and stopped the native backend `player_play_item` had just started. Both sites now route through `createAdapter()`. Compositing needs two independent opaque layers cleared, not one. Clearing only the page leaves the WebView widget opaque — audio over a black picture, exactly the symptom the old INTERIM comment described. `videoSurface.ts` toggles both: the widget background and window drawable from Kotlin, the page backgrounds via a `data-native-video` attribute keyed by app.css. Transparency lives in `tauri.android.conf.json` so Linux keeps an opaque window, and is scoped to the playback session so the launcher never shows through the rest of the app. Phase 3's rect plumbing turned out to be unnecessary: video is fullscreen on the player route, and `fitSurfaceToScreen()` already letterboxes and re-fits on rotation. The mini-player transition remains unverified on device. Also removes the `navigator.userAgent` sniffing in webviewAudio.ts, which was a second copy of the Rust cfg gate free to drift from it. `player_get_capabilities` now reports `usesWebviewAudio` and `supportsNativeVideo` from those same gates. Tests: adapter selection covers the full matrix, including the regression guard that the flag off beats Rust. Written first and confirmed failing (2 of 7) before the fix. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -1,6 +1,7 @@
|
||||
# Spec: Android native video — transparent-webview spike
|
||||
|
||||
**Status:** Proposed (spike — timeboxed, may conclude "not viable")
|
||||
**Status:** In progress — implemented behind `experimentalNativeVideo`, pending
|
||||
on-device confirmation. Branch `feat/android-native-video`.
|
||||
**Requirements:** IR-004, UR-003, UR-004 → DR-001, DR-023, DR-024
|
||||
**UX spec:** n/a — no intended visual change; the video surface must land exactly where the `<video>` element is today
|
||||
**Supersedes / revises:** acts on finding 2 of [playback-backend-unification.md](playback-backend-unification.md)
|
||||
@@ -109,6 +110,46 @@ in here rather than leaving a second, subtler copy of the bug behind. If
|
||||
`get_player_status` does not currently expose enough to cover the audio case, add
|
||||
the field — that is backend work, and correct.
|
||||
|
||||
### Implementation findings (2026-08-11)
|
||||
|
||||
Two blockers existed that this spec did not anticipate. Both were in code the
|
||||
spec assumed was merely *unreachable*; it was also *broken*.
|
||||
|
||||
**1. The Kotlin attach chain was severed.** `JellyTauPlayer.setActivity()` had
|
||||
**zero callers** anywhere in the tree. `currentActivity` was therefore always
|
||||
null, so `autoAttachSurface()` logged "Cannot attach surface - no Activity
|
||||
reference" and returned. The `SurfaceView` was created and wired to ExoPlayer but
|
||||
never added to the view hierarchy — video would have decoded to a surface that
|
||||
was never on screen, *regardless* of webview transparency. Fixed by calling
|
||||
`JellyTauPlayer.setActivity(this)` from `MainActivity.onCreate`.
|
||||
|
||||
Note the knock-on: `PictureInPictureManager.canEnterPip()` gates on
|
||||
`VideoOverlayManager.isVideoSurfaceAttached()`, which was permanently false. PiP
|
||||
on the video path was dead for the same reason.
|
||||
|
||||
**2. `createAdapter()` was not the real gate.** It is never called by production
|
||||
code — `VideoPlayer.svelte` constructs `Html5PlayerAdapter` directly. The actual
|
||||
override was `VideoPlayer.svelte`'s INTERIM block, which read Rust's
|
||||
`useHtml5Element`, forced it to `true`, and called `playerStop()` to kill the
|
||||
native backend `player_play_item` had just started. Both sites are now fixed;
|
||||
`VideoPlayer.svelte` routes through `createAdapter()` so there is one gate.
|
||||
|
||||
**Transparency needs two independent layers cleared,** not one. The spec's
|
||||
Phase 1 named only `html, body`. Clearing just the page leaves the WebView
|
||||
widget's own background opaque, which is a black screen with audio — the exact
|
||||
symptom the INTERIM comment described as "native surface not visible". Both are
|
||||
now toggled together by `$lib/utils/videoSurface.ts`:
|
||||
|
||||
| Layer | Cleared by | Reachable from |
|
||||
|-------|-----------|----------------|
|
||||
| WebView widget background + window drawable | `AndroidVideoSurface.setTransparent()` (MainActivity) | Kotlin only |
|
||||
| `html`/`body` + app-shell `--color-background` | `data-native-video` attribute → app.css | CSS only |
|
||||
|
||||
Transparency is scoped to `tauri.android.conf.json` rather than the base config:
|
||||
a transparent window on Linux is a regression, since nothing renders behind it.
|
||||
It is also toggled per-session rather than set once — a permanently transparent
|
||||
window shows the launcher through the rest of the app.
|
||||
|
||||
### Phase 3 — surface positioning
|
||||
|
||||
The hard part, and where this most likely fails. The webview's `<video>` element
|
||||
@@ -124,6 +165,29 @@ the video is effectively fullscreen on Android, which it is in the player route.
|
||||
rotation or the mini-player transition without visible artefacts, the spike fails
|
||||
and we keep HTML5. Do not ship a janky native path for a codec win.
|
||||
|
||||
**Update: no rect plumbing was needed.** The premise — that the surface must be
|
||||
positioned to match a laid-out `<video>` box — does not hold on the player route,
|
||||
where video is fullscreen. `VideoOverlayManager` adds the SurfaceView at index 0
|
||||
of `android.R.id.content` with `MATCH_PARENT`, and `fitSurfaceToScreen()`
|
||||
(`JellyTauPlayer.kt`) already letterboxes/pillarboxes to the real video aspect
|
||||
ratio and re-centres via a `Gravity.CENTER` `FrameLayout.LayoutParams`. Rotation
|
||||
is handled by an `OnLayoutChangeListener` that re-fits on any bounds change. The
|
||||
frontend's native branch is a bare `flex-1` box, so there is no rect to report
|
||||
and nothing to keep in sync.
|
||||
|
||||
This does mean **the mini-player transition is untested** for the native path —
|
||||
it is the one case the fullscreen assumption does not cover, and it remains an
|
||||
on-device check.
|
||||
|
||||
### A trap for the next implementer
|
||||
|
||||
There is a **stale duplicate player** at
|
||||
`src-tauri/android/app/src/main/java/com/dtourolle/jellytau/player/JellyTauPlayer.kt`
|
||||
(only commit: `cfddc1e` "First working POC"). No `sourceSets` entry points at it,
|
||||
so it is not compiled — but edits made there silently do nothing. The canonical
|
||||
tree is `src-tauri/android/src`, synced into `gen/` by
|
||||
`scripts/sync-android-sources.sh`.
|
||||
|
||||
### What we gain if it works
|
||||
|
||||
- **Hardware decode via MediaCodec** — `CodecDetector.kt` already reports
|
||||
@@ -145,9 +209,9 @@ and we keep HTML5. Do not ship a janky native path for a codec win.
|
||||
The spike is **complete** when one of these is true:
|
||||
|
||||
**Success path**
|
||||
- [ ] Transparent WebView confirmed working on a physical device.
|
||||
- [ ] `experimentalNativeVideo` off → behaviour byte-identical to today.
|
||||
- [ ] `webviewAudio.ts` no longer inspects `navigator.userAgent`; the platform's audio backend is read from Rust.
|
||||
- [x] Transparent WebView confirmed working on a physical device (reported by the maintainer; the config that enables it is now committed in `tauri.android.conf.json`).
|
||||
- [x] `experimentalNativeVideo` off → behaviour byte-identical to today. Guarded by `adapterSelection.test.ts`, which asserts the flag-off case forces HTML5 even when Rust reports native.
|
||||
- [x] `webviewAudio.ts` no longer inspects `navigator.userAgent`; the platform's audio backend is read from Rust (`player_get_capabilities` → `usesWebviewAudio`).
|
||||
- [ ] `experimentalNativeVideo` on → video plays via ExoPlayer/MediaCodec, correctly positioned, with working seek, audio-track switch, and subtitle selection through the existing `PlayerAdapter` contract.
|
||||
- [ ] No artefacts on rotation, background/foreground, or mini-player transition.
|
||||
- [ ] `adb shell dumpsys media.metrics` (or logcat) confirms a hardware decoder is in use.
|
||||
@@ -159,8 +223,15 @@ The spike is **complete** when one of these is true:
|
||||
- [ ] `nativeAdapter.ts:11-14` no longer cites tauri#10152.
|
||||
|
||||
Either way:
|
||||
- [ ] `bun run check`, `bun run test`, `bun run check:boundary` pass.
|
||||
- [ ] `cargo fmt` / `cargo clippy` clean; `bun run test:rust` passes.
|
||||
- [x] `bun run check` (0 errors), `bun run test` (892 passed), `bun run check:boundary` pass.
|
||||
- [x] `cargo fmt` / `cargo clippy` clean (no new warnings); `cargo test` passes (603 lib + 7 doc).
|
||||
|
||||
> Note: this environment has no host WebKitGTK dev packages, no Android SDK and
|
||||
> no `bun`, so all of the above were run inside the CI builder image
|
||||
> (`gitea.tourolle.paris/dtourolle/jellytau-builder:latest`). On Fedora the bind
|
||||
> mount needs `:z` for SELinux, and `scripts/build-android.sh` hardcodes
|
||||
> `ANDROID_HOME="$HOME/Android/Sdk"`, so the image's SDK at `/opt/android-sdk`
|
||||
> must be symlinked there rather than passed by env var.
|
||||
|
||||
## Testing
|
||||
|
||||
|
||||
Reference in New Issue
Block a user