jellytau/docs/specs/android-native-video-spike.md

# Spec: Android native video — transparent-webview spike

**Status:** Proposed (spike — timeboxed, may conclude "not viable")
**Requirements:** IR-004, UR-003, UR-004 → DR-001, DR-023, DR-024
**UX spec:** n/a — no intended visual change; the video surface must land exactly where the `<video>` element is today
**Supersedes / revises:** acts on finding 2 of [playback-backend-unification.md](playback-backend-unification.md)

## Summary

Test whether ExoPlayer's existing `SurfaceView` video path can be composited
behind a transparent Tauri WebView on Android. If it works, Android regains
hardware video decoding (MediaCodec) and libass-quality ASS/SSA subtitles, both
of which the current webview path lacks. If it does not, we document why and
delete the dead code.

This is a **spike**, not a feature commitment. The deliverable is a yes/no answer
with evidence, plus either a working path behind a flag or a removal.

## Motivation

`createAdapter()` hardcodes `const effectiveKind = "html5"` and does
`void backendKind`, discarding the `use_html5_element` value Rust computes in
`get_player_status`. As a result:

- `NativePlayerAdapter` is dead code.
- `JellyTauPlayer.kt`'s `getOrCreateSurfaceView()` — which already calls
  `setZOrderMediaOverlay(false)` and wires `setVideoSurfaceHolder` — is
  unreachable.
- Android video decodes in the WebView instead of via MediaCodec, despite
  `CodecDetector.kt` going to the trouble of reporting hardware codec
  capabilities back to Rust for DeviceProfile generation.

The code comment in `nativeAdapter.ts:11-14` justifies this by citing
tauri#10152 as an upstream blocker. **That justification is stale.**

### Why the blocker no longer holds

- tauri#10152 is open but **dead since 2024-07-01**, and it is a *feature
  request* ("Support transparent webviews on mobile"), not a bug report about
  compositing.
- The capability shipped in tauri commit `27d01834` (2024-09-02) — a clippy
  cleanup that moved `transparent()` out of the desktop-gated impl block, fencing
  only the tao call behind `#[cfg(desktop)]`. Because it landed as unrelated
  cleanup, nobody closed the issue.
- The black/white-screen reports (tauri#8381, tauri#9408) were a real but
  *different* bug: a broken JNI signature for `setBackgroundColor`, fixed in
  **wry 0.39.4** (PR #1237). We ship wry 0.55.x.
- Current wry calls `setBackgroundColor(0)` unconditionally on Android when
  transparency is requested.

### The honest caveat

**Nobody has demonstrated SurfaceView-behind-WebView on Tauri Android.** A search
of both `tauri-apps/tauri` and `tauri-apps/wry` issues for `surfaceview` returns
zero results, and the one native-video Tauri plugin
(`YeonV/tauri-plugin-videoplayer`) sidesteps compositing by launching a separate
fullscreen Activity. Nothing upstream blocks this; nothing upstream proves it.
Hence: spike, not feature.

Note this is the *Android* question only. The equivalent Linux compositing
problem is maintainer-declared unfixable and is **not** in scope — see the
unification spec.

## Layer assignment

| Logic / responsibility | Layer | Why it belongs there |
|------------------------|-------|----------------------|
| Which video backend this platform uses | Rust (existing) | `get_player_status` already computes `use_html5_element`. The frontend must *consume* it, not decide it. Restoring that is the point of the spike. |
| Surface creation, z-ordering, `setVideoSurfaceHolder` lifecycle | Kotlin | Android platform mechanics; already written in `JellyTauPlayer.kt`. |
| Seek/audio-track *strategy* | Rust (existing) | Already returned by `player_seek_video` / `player_switch_audio_track`; `NativePlayerAdapter` executes the chosen primitive. Unchanged — this is exactly what the `PlayerAdapter` contract was built for. |
| Positioning the surface under the video viewport | Frontend | Pure presentation/layout. **This is the risk area** — see Design. |

## Design

### Phase 1 — prove compositing (no app changes)

Before touching the adapter factory, verify the primitive works at all:

1. Set `"transparent": true` in `tauri.conf.json` for the Android build, plus
   `html, body { background: transparent; }`.
2. Confirm the WebView is genuinely transparent (a native view behind it is
   visible) and that the app does not regress to a black/white screen.

If this fails, stop — everything downstream is moot, and the finding is that
Tauri Android transparency is still broken in practice despite the shipped fix.

### Phase 2 — un-hardcode the factory

```ts
// src/lib/player/adapters/index.ts
export function createAdapter({ backendKind, host, bridge }: CreateAdapterArgs): PlayerAdapter {
  return backendKind === "native"
    ? new NativePlayerAdapter(host)
    : new Html5PlayerAdapter(host, bridge);
}
```

`backendKind` comes from `get_player_status` (`VideoBackend::Native` on Android).
Gate behind a setting — `experimentalNativeVideo`, default **off** — so a broken
spike cannot ship as a regression. Rust already owns this decision; the flag only
suppresses it.

**Also in scope: remove the user-agent sniffing in
`src/lib/services/webviewAudio.ts:30-41`.** It re-derives which audio backend the
platform has from `navigator.userAgent` ("matching the Rust cfg gate", per its own
comment) — the frontend deciding a backend fact it should be told. Same root cause
as the hardcode above, same fix: consume the value Rust already computes. Fold it
in here rather than leaving a second, subtler copy of the bug behind. If
`get_player_status` does not currently expose enough to cover the audio case, add
the field — that is backend work, and correct.

### Phase 3 — surface positioning

The hard part, and where this most likely fails. The webview's `<video>` element
occupies a laid-out box; the `SurfaceView` must be positioned to match it, and
kept matched through scroll, rotation, and mini-player transitions.

Approach: the video view reports its `getBoundingClientRect()` to Rust, which
forwards the rect to Kotlin to position the `SurfaceView`. This is the same
"faking it" technique the ecosystem uses on desktop — acceptable here *only if*
the video is effectively fullscreen on Android, which it is in the player route.

**Explicit failure criterion**: if the surface cannot be kept aligned during
rotation or the mini-player transition without visible artefacts, the spike fails
and we keep HTML5. Do not ship a janky native path for a codec win.

### What we gain if it works

- **Hardware decode via MediaCodec** — `CodecDetector.kt` already reports
  capabilities; the DeviceProfile would finally match what actually plays.
- **ASS/SSA subtitles** are *not* automatic. ExoPlayer cannot render them; that
  would require libmpv, which is a separate and much larger decision (see the
  unification spec's engine comparison). Scope this spike to hardware decode
  only, and do not claim subtitle improvements from it.

## Out of scope

- Linux native video. Maintainer-declared unfixable on WebKitGTK/Wayland.
- Replacing ExoPlayer with libmpv on Android.
- Windows native video.
- Removing the HTML5 path. It stays as the default and the fallback.

## Acceptance criteria

The spike is **complete** when one of these is true:

**Success path**
- [ ] Transparent WebView confirmed working on a physical device.
- [ ] `experimentalNativeVideo` off → behaviour byte-identical to today.
- [ ] `webviewAudio.ts` no longer inspects `navigator.userAgent`; the platform's audio backend is read from Rust.
- [ ] `experimentalNativeVideo` on → video plays via ExoPlayer/MediaCodec, correctly positioned, with working seek, audio-track switch, and subtitle selection through the existing `PlayerAdapter` contract.
- [ ] No artefacts on rotation, background/foreground, or mini-player transition.
- [ ] `adb shell dumpsys media.metrics` (or logcat) confirms a hardware decoder is in use.
- [ ] Measured battery/thermal or CPU improvement over the HTML5 path on the same clip.

**Failure path**
- [ ] The blocking behaviour is documented in this spec with evidence.
- [ ] `NativePlayerAdapter` and the unreachable `SurfaceView` code are deleted, or explicitly retained with a *correct* comment.
- [ ] `nativeAdapter.ts:11-14` no longer cites tauri#10152.

Either way:
- [ ] `bun run check`, `bun run test`, `bun run check:boundary` pass.
- [ ] `cargo fmt` / `cargo clippy` clean; `bun run test:rust` passes.

## Testing

Adapter-selection logic is pure and testable without a device: assert
`createAdapter` returns `NativePlayerAdapter` for `backendKind: "native"` with
the flag on, and `Html5PlayerAdapter` in every other combination — including that
the flag off forces HTML5 even when Rust says native. That last case is the
regression guard.

Everything else is manual on-device; there is no meaningful way to unit-test
surface compositing. Test on at least two devices — compositing behaviour varies
by OEM and Android version.

Per CLAUDE.md, if the spike turns into a bug fix (e.g. seek breaks under the
native adapter), write the failing test first.

## TRACES

- `createAdapter` → `// TRACES: UR-003, UR-004 | DR-023, DR-024`
- Adapter-selection tests → `UT-xxx`
- No new requirement IDs; this spike either satisfies existing IR-004 expectations or documents why it cannot.

## Notes for the implementer

- **Do not skip Phase 1.** If transparency does not work, phases 2 and 3 are
  wasted effort.
- `VideoPlayer.svelte` has a documented hazard: no lifecycle calls after an
  `await` in `onMount` — it flips to HTML5 mode and breaks Android seek. The
  adapter swap touches exactly this code path.
- tauri-specta tagged responses keep Rust field names (`new_url`, not `newUrl`).
- Android source edits go in `src-tauri/android/src`, then run
  `scripts/sync-android-sources.sh`.
- A parallel Claude session may be active — `git diff` first.