jellytau/docs/specs/android-native-video-spike.md

# Spec: Android native video — transparent-webview spike

**Status:** Spike succeeded — native video confirmed working on a physical
device (2026-08-11) with `experimentalNativeVideo` on. Shipped behind that flag,
default off. Branch `feat/android-native-video`.

**The spike's central question is answered: yes.** A `SurfaceView` *can* be
composited behind a transparent Tauri WebView on Android. Nothing upstream
blocked it and nothing upstream demonstrated it — this is, as far as the issue
trackers show, the first working instance. The remaining flag is about test
coverage and the unverified cases below, not about viability.
**Requirements:** IR-004, UR-003, UR-004 → DR-001, DR-023, DR-024
**UX spec:** n/a — no intended visual change; the video surface must land exactly where the `<video>` element is today
**Supersedes / revises:** acts on finding 2 of [playback-backend-unification.md](playback-backend-unification.md)

## Summary

Test whether ExoPlayer's existing `SurfaceView` video path can be composited
behind a transparent Tauri WebView on Android. If it works, Android regains
hardware video decoding (MediaCodec) and libass-quality ASS/SSA subtitles, both
of which the current webview path lacks. If it does not, we document why and
delete the dead code.

This is a **spike**, not a feature commitment. The deliverable is a yes/no answer
with evidence, plus either a working path behind a flag or a removal.

## Motivation

`createAdapter()` hardcodes `const effectiveKind = "html5"` and does
`void backendKind`, discarding the `use_html5_element` value Rust computes in
`get_player_status`. As a result:

- `NativePlayerAdapter` is dead code.
- `JellyTauPlayer.kt`'s `getOrCreateSurfaceView()` — which already calls
  `setZOrderMediaOverlay(false)` and wires `setVideoSurfaceHolder` — is
  unreachable.
- Android video decodes in the WebView instead of via MediaCodec, despite
  `CodecDetector.kt` going to the trouble of reporting hardware codec
  capabilities back to Rust for DeviceProfile generation.

The code comment in `nativeAdapter.ts:11-14` justifies this by citing
tauri#10152 as an upstream blocker. **That justification is stale.**

### Why the blocker no longer holds

- tauri#10152 is open but **dead since 2024-07-01**, and it is a *feature
  request* ("Support transparent webviews on mobile"), not a bug report about
  compositing.
- The capability shipped in tauri commit `27d01834` (2024-09-02) — a clippy
  cleanup that moved `transparent()` out of the desktop-gated impl block, fencing
  only the tao call behind `#[cfg(desktop)]`. Because it landed as unrelated
  cleanup, nobody closed the issue.
- The black/white-screen reports (tauri#8381, tauri#9408) were a real but
  *different* bug: a broken JNI signature for `setBackgroundColor`, fixed in
  **wry 0.39.4** (PR #1237). We ship wry 0.55.x.
- Current wry calls `setBackgroundColor(0)` unconditionally on Android when
  transparency is requested.

### The honest caveat

**Nobody has demonstrated SurfaceView-behind-WebView on Tauri Android.** A search
of both `tauri-apps/tauri` and `tauri-apps/wry` issues for `surfaceview` returns
zero results, and the one native-video Tauri plugin
(`YeonV/tauri-plugin-videoplayer`) sidesteps compositing by launching a separate
fullscreen Activity. Nothing upstream blocks this; nothing upstream proves it.
Hence: spike, not feature.

Note this is the *Android* question only. The equivalent Linux compositing
problem is maintainer-declared unfixable and is **not** in scope — see the
unification spec.

## Layer assignment

| Logic / responsibility | Layer | Why it belongs there |
|------------------------|-------|----------------------|
| Which video backend this platform uses | Rust (existing) | `get_player_status` already computes `use_html5_element`. The frontend must *consume* it, not decide it. Restoring that is the point of the spike. |
| Surface creation, z-ordering, `setVideoSurfaceHolder` lifecycle | Kotlin | Android platform mechanics; already written in `JellyTauPlayer.kt`. |
| Seek/audio-track *strategy* | Rust (existing) | Already returned by `player_seek_video` / `player_switch_audio_track`; `NativePlayerAdapter` executes the chosen primitive. Unchanged — this is exactly what the `PlayerAdapter` contract was built for. |
| Positioning the surface under the video viewport | Frontend | Pure presentation/layout. **This is the risk area** — see Design. |

## Design

### Phase 1 — prove compositing (no app changes)

Before touching the adapter factory, verify the primitive works at all:

1. Set `"transparent": true` in `tauri.conf.json` for the Android build, plus
   `html, body { background: transparent; }`.
2. Confirm the WebView is genuinely transparent (a native view behind it is
   visible) and that the app does not regress to a black/white screen.

If this fails, stop — everything downstream is moot, and the finding is that
Tauri Android transparency is still broken in practice despite the shipped fix.

### Phase 2 — un-hardcode the factory

```ts
// src/lib/player/adapters/index.ts
export function createAdapter({ backendKind, host, bridge }: CreateAdapterArgs): PlayerAdapter {
  return backendKind === "native"
    ? new NativePlayerAdapter(host)
    : new Html5PlayerAdapter(host, bridge);
}
```

`backendKind` comes from `get_player_status` (`VideoBackend::Native` on Android).
Gate behind a setting — `experimentalNativeVideo`, default **off** — so a broken
spike cannot ship as a regression. Rust already owns this decision; the flag only
suppresses it.

**Also in scope: remove the user-agent sniffing in
`src/lib/services/webviewAudio.ts:30-41`.** It re-derives which audio backend the
platform has from `navigator.userAgent` ("matching the Rust cfg gate", per its own
comment) — the frontend deciding a backend fact it should be told. Same root cause
as the hardcode above, same fix: consume the value Rust already computes. Fold it
in here rather than leaving a second, subtler copy of the bug behind. If
`get_player_status` does not currently expose enough to cover the audio case, add
the field — that is backend work, and correct.

### Implementation findings (2026-08-11)

Two blockers existed that this spec did not anticipate. Both were in code the
spec assumed was merely *unreachable*; it was also *broken*.

**1. The Kotlin attach chain was severed.** `JellyTauPlayer.setActivity()` had
**zero callers** anywhere in the tree. `currentActivity` was therefore always
null, so `autoAttachSurface()` logged "Cannot attach surface - no Activity
reference" and returned. The `SurfaceView` was created and wired to ExoPlayer but
never added to the view hierarchy — video would have decoded to a surface that
was never on screen, *regardless* of webview transparency. Fixed by calling
`JellyTauPlayer.setActivity(this)` from `MainActivity.onCreate`.

Note the knock-on: `PictureInPictureManager.canEnterPip()` gates on
`VideoOverlayManager.isVideoSurfaceAttached()`, which was permanently false. PiP
on the video path was dead for the same reason.

**2. `createAdapter()` was not the real gate.** It is never called by production
code — `VideoPlayer.svelte` constructs `Html5PlayerAdapter` directly. The actual
override was `VideoPlayer.svelte`'s INTERIM block, which read Rust's
`useHtml5Element`, forced it to `true`, and called `playerStop()` to kill the
native backend `player_play_item` had just started. Both sites are now fixed;
`VideoPlayer.svelte` routes through `createAdapter()` so there is one gate.

**Transparency needs two independent layers cleared,** not one. The spec's
Phase 1 named only `html, body`. Clearing just the page leaves the WebView
widget's own background opaque, which is a black screen with audio — the exact
symptom the INTERIM comment described as "native surface not visible". Both are
now toggled together by `$lib/utils/videoSurface.ts`:

| Layer | Cleared by | Reachable from |
|-------|-----------|----------------|
| WebView widget background + window drawable | `AndroidVideoSurface.setTransparent()` (MainActivity) | Kotlin only |
| `html`/`body` + app-shell `--color-background` | `data-native-video` attribute → app.css | CSS only |

Transparency is scoped to `tauri.android.conf.json` rather than the base config:
a transparent window on Linux is a regression, since nothing renders behind it.
It is also toggled per-session rather than set once — a permanently transparent
window shows the launcher through the rest of the app.

### Phase 3 — surface positioning

The hard part, and where this most likely fails. The webview's `<video>` element
occupies a laid-out box; the `SurfaceView` must be positioned to match it, and
kept matched through scroll, rotation, and mini-player transitions.

Approach: the video view reports its `getBoundingClientRect()` to Rust, which
forwards the rect to Kotlin to position the `SurfaceView`. This is the same
"faking it" technique the ecosystem uses on desktop — acceptable here *only if*
the video is effectively fullscreen on Android, which it is in the player route.

**Explicit failure criterion**: if the surface cannot be kept aligned during
rotation or the mini-player transition without visible artefacts, the spike fails
and we keep HTML5. Do not ship a janky native path for a codec win.

**Update: no rect plumbing was needed.** The premise — that the surface must be
positioned to match a laid-out `<video>` box — does not hold on the player route,
where video is fullscreen. `VideoOverlayManager` adds the SurfaceView at index 0
of `android.R.id.content` with `MATCH_PARENT`, and `fitSurfaceToScreen()`
(`JellyTauPlayer.kt`) already letterboxes/pillarboxes to the real video aspect
ratio and re-centres via a `Gravity.CENTER` `FrameLayout.LayoutParams`. Rotation
is handled by an `OnLayoutChangeListener` that re-fits on any bounds change. The
frontend's native branch is a bare `flex-1` box, so there is no rect to report
and nothing to keep in sync.

Fullscreen playback is confirmed working on device. But this reasoning rests
entirely on the fullscreen assumption, so **the mini-player transition is the
known gap** — it is the one case where the surface is *not* fullscreen, and
therefore the one case where the "no rect plumbing needed" conclusion could
still turn out to be wrong. If artefacts appear there, the fix is the rect
reporting this section originally proposed, scoped to that transition alone.

### A trap for the next implementer

There is a **stale duplicate player** at
`src-tauri/android/app/src/main/java/com/dtourolle/jellytau/player/JellyTauPlayer.kt`
(only commit: `cfddc1e` "First working POC"). No `sourceSets` entry points at it,
so it is not compiled — but edits made there silently do nothing. The canonical
tree is `src-tauri/android/src`, synced into `gen/` by
`scripts/sync-android-sources.sh`.

### What we gain if it works

- **Hardware decode via MediaCodec** — `CodecDetector.kt` already reports
  capabilities; the DeviceProfile would finally match what actually plays.
- **ASS/SSA subtitles** are *not* automatic. ExoPlayer cannot render them; that
  would require libmpv, which is a separate and much larger decision (see the
  unification spec's engine comparison). Scope this spike to hardware decode
  only, and do not claim subtitle improvements from it.

## Out of scope

- Linux native video. Maintainer-declared unfixable on WebKitGTK/Wayland.
- Replacing ExoPlayer with libmpv on Android.
- Windows native video.
- Removing the HTML5 path. It stays as the default and the fallback.

## Acceptance criteria

The spike is **complete** when one of these is true:

**Success path**
- [x] Transparent WebView confirmed working on a physical device (reported by the maintainer; the config that enables it is now committed in `tauri.android.conf.json`).
- [x] `experimentalNativeVideo` off → behaviour byte-identical to today. Guarded by `adapterSelection.test.ts`, which asserts the flag-off case forces HTML5 even when Rust reports native.
- [x] `webviewAudio.ts` no longer inspects `navigator.userAgent`; the platform's audio backend is read from Rust (`player_get_capabilities` → `usesWebviewAudio`).
- [x] `experimentalNativeVideo` on → video plays via ExoPlayer, correctly positioned, on a physical device (2026-08-11). The surface reaches the hierarchy and is visible through the transparent WebView — the whole point of the spike.
- [ ] Seek, audio-track switch and subtitle selection exercised through `NativePlayerAdapter`. Playback is confirmed; these individual controls are not yet each verified on the native path.
- [ ] No artefacts on rotation, background/foreground, or **mini-player transition** — the last is the one case the fullscreen assumption does not cover, so it is the likeliest place to find a problem.
- [ ] `adb shell dumpsys media.metrics` (or logcat) confirms a hardware decoder is in use. Plausible but unmeasured — do not claim the MediaCodec win until this is read.
- [ ] Measured battery/thermal or CPU improvement over the HTML5 path on the same clip.

**Failure path**
- [ ] The blocking behaviour is documented in this spec with evidence.
- [ ] `NativePlayerAdapter` and the unreachable `SurfaceView` code are deleted, or explicitly retained with a *correct* comment.
- [ ] `nativeAdapter.ts:11-14` no longer cites tauri#10152.

Either way:
- [x] `bun run check` (0 errors), `bun run test` (892 passed), `bun run check:boundary` pass.
- [x] `cargo fmt` / `cargo clippy` clean (no new warnings); `cargo test` passes (603 lib + 7 doc).

> Note: this environment has no host WebKitGTK dev packages, no Android SDK and
> no `bun`, so all of the above were run inside the CI builder image
> (`gitea.tourolle.paris/dtourolle/jellytau-builder:latest`). On Fedora the bind
> mount needs `:z` for SELinux, and `scripts/build-android.sh` hardcodes
> `ANDROID_HOME="$HOME/Android/Sdk"`, so the image's SDK at `/opt/android-sdk`
> must be symlinked there rather than passed by env var.

## Testing

Adapter-selection logic is pure and testable without a device: assert
`createAdapter` returns `NativePlayerAdapter` for `backendKind: "native"` with
the flag on, and `Html5PlayerAdapter` in every other combination — including that
the flag off forces HTML5 even when Rust says native. That last case is the
regression guard.

Everything else is manual on-device; there is no meaningful way to unit-test
surface compositing. Test on at least two devices — compositing behaviour varies
by OEM and Android version.

Per CLAUDE.md, if the spike turns into a bug fix (e.g. seek breaks under the
native adapter), write the failing test first.

## TRACES

- `createAdapter` → `// TRACES: UR-003, UR-004 | DR-023, DR-024`
- Adapter-selection tests → `UT-xxx`
- No new requirement IDs; this spike either satisfies existing IR-004 expectations or documents why it cannot.

## Notes for the implementer

- **Do not skip Phase 1.** If transparency does not work, phases 2 and 3 are
  wasted effort.
- `VideoPlayer.svelte` has a documented hazard: no lifecycle calls after an
  `await` in `onMount` — it flips to HTML5 mode and breaks Android seek. The
  adapter swap touches exactly this code path.
- tauri-specta tagged responses keep Rust field names (`new_url`, not `newUrl`).
- Android source edits go in `src-tauri/android/src`, then run
  `scripts/sync-android-sources.sh`.
- A parallel Claude session may be active — `git diff` first.