jellytau/docs/specs/android-native-video-spike.md

Spec: Android native video — transparent-webview spike

Status: Proposed (spike — timeboxed, may conclude "not viable") Requirements: IR-004, UR-003, UR-004 → DR-001, DR-023, DR-024 UX spec: n/a — no intended visual change; the video surface must land exactly where the <video> element is today Supersedes / revises: acts on finding 2 of playback-backend-unification.md

Summary

Test whether ExoPlayer's existing SurfaceView video path can be composited behind a transparent Tauri WebView on Android. If it works, Android regains hardware video decoding (MediaCodec) and libass-quality ASS/SSA subtitles, both of which the current webview path lacks. If it does not, we document why and delete the dead code.

This is a spike, not a feature commitment. The deliverable is a yes/no answer with evidence, plus either a working path behind a flag or a removal.

Motivation

createAdapter() hardcodes const effectiveKind = "html5" and does void backendKind, discarding the use_html5_element value Rust computes in get_player_status. As a result:

• NativePlayerAdapter is dead code.
• JellyTauPlayer.kt's getOrCreateSurfaceView() — which already calls setZOrderMediaOverlay(false) and wires setVideoSurfaceHolder — is unreachable.
• Android video decodes in the WebView instead of via MediaCodec, despite CodecDetector.kt going to the trouble of reporting hardware codec capabilities back to Rust for DeviceProfile generation.

The code comment in nativeAdapter.ts:11-14 justifies this by citing tauri#10152 as an upstream blocker. That justification is stale.

Why the blocker no longer holds

• tauri#10152 is open but dead since 2024-07-01, and it is a feature request ("Support transparent webviews on mobile"), not a bug report about compositing.
• The capability shipped in tauri commit 27d01834 (2024-09-02) — a clippy cleanup that moved transparent() out of the desktop-gated impl block, fencing only the tao call behind #[cfg(desktop)]. Because it landed as unrelated cleanup, nobody closed the issue.
• The black/white-screen reports (tauri#8381, tauri#9408) were a real but different bug: a broken JNI signature for setBackgroundColor, fixed in wry 0.39.4 (PR #1237). We ship wry 0.55.x.
• Current wry calls setBackgroundColor(0) unconditionally on Android when transparency is requested.

The honest caveat

Nobody has demonstrated SurfaceView-behind-WebView on Tauri Android. A search of both tauri-apps/tauri and tauri-apps/wry issues for surfaceview returns zero results, and the one native-video Tauri plugin (YeonV/tauri-plugin-videoplayer) sidesteps compositing by launching a separate fullscreen Activity. Nothing upstream blocks this; nothing upstream proves it. Hence: spike, not feature.

Note this is the Android question only. The equivalent Linux compositing problem is maintainer-declared unfixable and is not in scope — see the unification spec.

Layer assignment

Logic / responsibility	Layer	Why it belongs there
Which video backend this platform uses	Rust (existing)	get_player_status already computes use_html5_element. The frontend must consume it, not decide it. Restoring that is the point of the spike.
Surface creation, z-ordering, setVideoSurfaceHolder lifecycle	Kotlin	Android platform mechanics; already written in JellyTauPlayer.kt.
Seek/audio-track strategy	Rust (existing)	Already returned by player_seek_video / player_switch_audio_track; NativePlayerAdapter executes the chosen primitive. Unchanged — this is exactly what the PlayerAdapter contract was built for.
Positioning the surface under the video viewport	Frontend	Pure presentation/layout. This is the risk area — see Design.

Design

Phase 1 — prove compositing (no app changes)

Before touching the adapter factory, verify the primitive works at all:

1. Set "transparent": true in tauri.conf.json for the Android build, plus html, body { background: transparent; }.
2. Confirm the WebView is genuinely transparent (a native view behind it is visible) and that the app does not regress to a black/white screen.

If this fails, stop — everything downstream is moot, and the finding is that Tauri Android transparency is still broken in practice despite the shipped fix.

Phase 2 — un-hardcode the factory

// src/lib/player/adapters/index.ts
export function createAdapter({ backendKind, host, bridge }: CreateAdapterArgs): PlayerAdapter {
  return backendKind === "native"
    ? new NativePlayerAdapter(host)
    : new Html5PlayerAdapter(host, bridge);
}

backendKind comes from get_player_status (VideoBackend::Native on Android). Gate behind a setting — experimentalNativeVideo, default off — so a broken spike cannot ship as a regression. Rust already owns this decision; the flag only suppresses it.

Also in scope: remove the user-agent sniffing in src/lib/services/webviewAudio.ts:30-41. It re-derives which audio backend the platform has from navigator.userAgent ("matching the Rust cfg gate", per its own comment) — the frontend deciding a backend fact it should be told. Same root cause as the hardcode above, same fix: consume the value Rust already computes. Fold it in here rather than leaving a second, subtler copy of the bug behind. If get_player_status does not currently expose enough to cover the audio case, add the field — that is backend work, and correct.

Phase 3 — surface positioning

The hard part, and where this most likely fails. The webview's <video> element occupies a laid-out box; the SurfaceView must be positioned to match it, and kept matched through scroll, rotation, and mini-player transitions.

Approach: the video view reports its getBoundingClientRect() to Rust, which forwards the rect to Kotlin to position the SurfaceView. This is the same "faking it" technique the ecosystem uses on desktop — acceptable here only if the video is effectively fullscreen on Android, which it is in the player route.

Explicit failure criterion: if the surface cannot be kept aligned during rotation or the mini-player transition without visible artefacts, the spike fails and we keep HTML5. Do not ship a janky native path for a codec win.

What we gain if it works

• Hardware decode via MediaCodec — CodecDetector.kt already reports capabilities; the DeviceProfile would finally match what actually plays.
• ASS/SSA subtitles are not automatic. ExoPlayer cannot render them; that would require libmpv, which is a separate and much larger decision (see the unification spec's engine comparison). Scope this spike to hardware decode only, and do not claim subtitle improvements from it.

Out of scope

• Linux native video. Maintainer-declared unfixable on WebKitGTK/Wayland.
• Replacing ExoPlayer with libmpv on Android.
• Windows native video.
• Removing the HTML5 path. It stays as the default and the fallback.

Acceptance criteria

The spike is complete when one of these is true:

Success path

• Transparent WebView confirmed working on a physical device.
• experimentalNativeVideo off → behaviour byte-identical to today.
• webviewAudio.ts no longer inspects navigator.userAgent; the platform's audio backend is read from Rust.
• experimentalNativeVideo on → video plays via ExoPlayer/MediaCodec, correctly positioned, with working seek, audio-track switch, and subtitle selection through the existing PlayerAdapter contract.
• No artefacts on rotation, background/foreground, or mini-player transition.
• adb shell dumpsys media.metrics (or logcat) confirms a hardware decoder is in use.
• Measured battery/thermal or CPU improvement over the HTML5 path on the same clip.

Failure path

• The blocking behaviour is documented in this spec with evidence.
• NativePlayerAdapter and the unreachable SurfaceView code are deleted, or explicitly retained with a correct comment.
• nativeAdapter.ts:11-14 no longer cites tauri#10152.

Either way:

• bun run check, bun run test, bun run check:boundary pass.
• cargo fmt / cargo clippy clean; bun run test:rust passes.

Testing

Adapter-selection logic is pure and testable without a device: assert createAdapter returns NativePlayerAdapter for backendKind: "native" with the flag on, and Html5PlayerAdapter in every other combination — including that the flag off forces HTML5 even when Rust says native. That last case is the regression guard.

Everything else is manual on-device; there is no meaningful way to unit-test surface compositing. Test on at least two devices — compositing behaviour varies by OEM and Android version.

Per CLAUDE.md, if the spike turns into a bug fix (e.g. seek breaks under the native adapter), write the failing test first.

TRACES

• createAdapter → // TRACES: UR-003, UR-004 | DR-023, DR-024
• Adapter-selection tests → UT-xxx
• No new requirement IDs; this spike either satisfies existing IR-004 expectations or documents why it cannot.

Notes for the implementer

• Do not skip Phase 1. If transparency does not work, phases 2 and 3 are wasted effort.
• VideoPlayer.svelte has a documented hazard: no lifecycle calls after an await in onMount — it flips to HTML5 mode and breaks Android seek. The adapter swap touches exactly this code path.
• tauri-specta tagged responses keep Rust field names (new_url, not newUrl).
• Android source edits go in src-tauri/android/src, then run scripts/sync-android-sources.sh.
• A parallel Claude session may be active — git diff first.