jellytau/docs/specs/android-native-video-spike.md

Spec: Android native video — transparent-webview spike

Status: In progress — implemented behind experimentalNativeVideo, pending on-device confirmation. Branch feat/android-native-video. Requirements: IR-004, UR-003, UR-004 → DR-001, DR-023, DR-024 UX spec: n/a — no intended visual change; the video surface must land exactly where the <video> element is today Supersedes / revises: acts on finding 2 of playback-backend-unification.md

Summary

Test whether ExoPlayer's existing SurfaceView video path can be composited behind a transparent Tauri WebView on Android. If it works, Android regains hardware video decoding (MediaCodec) and libass-quality ASS/SSA subtitles, both of which the current webview path lacks. If it does not, we document why and delete the dead code.

This is a spike, not a feature commitment. The deliverable is a yes/no answer with evidence, plus either a working path behind a flag or a removal.

Motivation

createAdapter() hardcodes const effectiveKind = "html5" and does void backendKind, discarding the use_html5_element value Rust computes in get_player_status. As a result:

• NativePlayerAdapter is dead code.
• JellyTauPlayer.kt's getOrCreateSurfaceView() — which already calls setZOrderMediaOverlay(false) and wires setVideoSurfaceHolder — is unreachable.
• Android video decodes in the WebView instead of via MediaCodec, despite CodecDetector.kt going to the trouble of reporting hardware codec capabilities back to Rust for DeviceProfile generation.

The code comment in nativeAdapter.ts:11-14 justifies this by citing tauri#10152 as an upstream blocker. That justification is stale.

Why the blocker no longer holds

• tauri#10152 is open but dead since 2024-07-01, and it is a feature request ("Support transparent webviews on mobile"), not a bug report about compositing.
• The capability shipped in tauri commit 27d01834 (2024-09-02) — a clippy cleanup that moved transparent() out of the desktop-gated impl block, fencing only the tao call behind #[cfg(desktop)]. Because it landed as unrelated cleanup, nobody closed the issue.
• The black/white-screen reports (tauri#8381, tauri#9408) were a real but different bug: a broken JNI signature for setBackgroundColor, fixed in wry 0.39.4 (PR #1237). We ship wry 0.55.x.
• Current wry calls setBackgroundColor(0) unconditionally on Android when transparency is requested.

The honest caveat

Nobody has demonstrated SurfaceView-behind-WebView on Tauri Android. A search of both tauri-apps/tauri and tauri-apps/wry issues for surfaceview returns zero results, and the one native-video Tauri plugin (YeonV/tauri-plugin-videoplayer) sidesteps compositing by launching a separate fullscreen Activity. Nothing upstream blocks this; nothing upstream proves it. Hence: spike, not feature.

Note this is the Android question only. The equivalent Linux compositing problem is maintainer-declared unfixable and is not in scope — see the unification spec.

Layer assignment

Logic / responsibility	Layer	Why it belongs there
Which video backend this platform uses	Rust (existing)	get_player_status already computes use_html5_element. The frontend must consume it, not decide it. Restoring that is the point of the spike.
Surface creation, z-ordering, setVideoSurfaceHolder lifecycle	Kotlin	Android platform mechanics; already written in JellyTauPlayer.kt.
Seek/audio-track strategy	Rust (existing)	Already returned by player_seek_video / player_switch_audio_track; NativePlayerAdapter executes the chosen primitive. Unchanged — this is exactly what the PlayerAdapter contract was built for.
Positioning the surface under the video viewport	Frontend	Pure presentation/layout. This is the risk area — see Design.

Design

Phase 1 — prove compositing (no app changes)

Before touching the adapter factory, verify the primitive works at all:

1. Set "transparent": true in tauri.conf.json for the Android build, plus html, body { background: transparent; }.
2. Confirm the WebView is genuinely transparent (a native view behind it is visible) and that the app does not regress to a black/white screen.

If this fails, stop — everything downstream is moot, and the finding is that Tauri Android transparency is still broken in practice despite the shipped fix.

Phase 2 — un-hardcode the factory

// src/lib/player/adapters/index.ts
export function createAdapter({ backendKind, host, bridge }: CreateAdapterArgs): PlayerAdapter {
  return backendKind === "native"
    ? new NativePlayerAdapter(host)
    : new Html5PlayerAdapter(host, bridge);
}

backendKind comes from get_player_status (VideoBackend::Native on Android). Gate behind a setting — experimentalNativeVideo, default off — so a broken spike cannot ship as a regression. Rust already owns this decision; the flag only suppresses it.

Also in scope: remove the user-agent sniffing in src/lib/services/webviewAudio.ts:30-41. It re-derives which audio backend the platform has from navigator.userAgent ("matching the Rust cfg gate", per its own comment) — the frontend deciding a backend fact it should be told. Same root cause as the hardcode above, same fix: consume the value Rust already computes. Fold it in here rather than leaving a second, subtler copy of the bug behind. If get_player_status does not currently expose enough to cover the audio case, add the field — that is backend work, and correct.

Implementation findings (2026-08-11)

Two blockers existed that this spec did not anticipate. Both were in code the spec assumed was merely unreachable; it was also broken.

1. The Kotlin attach chain was severed. JellyTauPlayer.setActivity() had zero callers anywhere in the tree. currentActivity was therefore always null, so autoAttachSurface() logged "Cannot attach surface - no Activity reference" and returned. The SurfaceView was created and wired to ExoPlayer but never added to the view hierarchy — video would have decoded to a surface that was never on screen, regardless of webview transparency. Fixed by calling JellyTauPlayer.setActivity(this) from MainActivity.onCreate.

Note the knock-on: PictureInPictureManager.canEnterPip() gates on VideoOverlayManager.isVideoSurfaceAttached(), which was permanently false. PiP on the video path was dead for the same reason.

2. createAdapter() was not the real gate. It is never called by production code — VideoPlayer.svelte constructs Html5PlayerAdapter directly. The actual override was VideoPlayer.svelte's INTERIM block, which read Rust's useHtml5Element, forced it to true, and called playerStop() to kill the native backend player_play_item had just started. Both sites are now fixed; VideoPlayer.svelte routes through createAdapter() so there is one gate.

Transparency needs two independent layers cleared, not one. The spec's Phase 1 named only html, body. Clearing just the page leaves the WebView widget's own background opaque, which is a black screen with audio — the exact symptom the INTERIM comment described as "native surface not visible". Both are now toggled together by $lib/utils/videoSurface.ts:

Layer	Cleared by	Reachable from
WebView widget background + window drawable	AndroidVideoSurface.setTransparent() (MainActivity)	Kotlin only
html/body + app-shell --color-background	data-native-video attribute → app.css	CSS only

Transparency is scoped to tauri.android.conf.json rather than the base config: a transparent window on Linux is a regression, since nothing renders behind it. It is also toggled per-session rather than set once — a permanently transparent window shows the launcher through the rest of the app.

Phase 3 — surface positioning

The hard part, and where this most likely fails. The webview's <video> element occupies a laid-out box; the SurfaceView must be positioned to match it, and kept matched through scroll, rotation, and mini-player transitions.

Approach: the video view reports its getBoundingClientRect() to Rust, which forwards the rect to Kotlin to position the SurfaceView. This is the same "faking it" technique the ecosystem uses on desktop — acceptable here only if the video is effectively fullscreen on Android, which it is in the player route.

Explicit failure criterion: if the surface cannot be kept aligned during rotation or the mini-player transition without visible artefacts, the spike fails and we keep HTML5. Do not ship a janky native path for a codec win.

Update: no rect plumbing was needed. The premise — that the surface must be positioned to match a laid-out <video> box — does not hold on the player route, where video is fullscreen. VideoOverlayManager adds the SurfaceView at index 0 of android.R.id.content with MATCH_PARENT, and fitSurfaceToScreen() (JellyTauPlayer.kt) already letterboxes/pillarboxes to the real video aspect ratio and re-centres via a Gravity.CENTER FrameLayout.LayoutParams. Rotation is handled by an OnLayoutChangeListener that re-fits on any bounds change. The frontend's native branch is a bare flex-1 box, so there is no rect to report and nothing to keep in sync.

This does mean the mini-player transition is untested for the native path — it is the one case the fullscreen assumption does not cover, and it remains an on-device check.

A trap for the next implementer

There is a stale duplicate player at src-tauri/android/app/src/main/java/com/dtourolle/jellytau/player/JellyTauPlayer.kt (only commit: cfddc1e "First working POC"). No sourceSets entry points at it, so it is not compiled — but edits made there silently do nothing. The canonical tree is src-tauri/android/src, synced into gen/ by scripts/sync-android-sources.sh.

What we gain if it works

• Hardware decode via MediaCodec — CodecDetector.kt already reports capabilities; the DeviceProfile would finally match what actually plays.
• ASS/SSA subtitles are not automatic. ExoPlayer cannot render them; that would require libmpv, which is a separate and much larger decision (see the unification spec's engine comparison). Scope this spike to hardware decode only, and do not claim subtitle improvements from it.

Out of scope

• Linux native video. Maintainer-declared unfixable on WebKitGTK/Wayland.
• Replacing ExoPlayer with libmpv on Android.
• Windows native video.
• Removing the HTML5 path. It stays as the default and the fallback.

Acceptance criteria

The spike is complete when one of these is true:

Success path

• Transparent WebView confirmed working on a physical device (reported by the maintainer; the config that enables it is now committed in tauri.android.conf.json).
• experimentalNativeVideo off → behaviour byte-identical to today. Guarded by adapterSelection.test.ts, which asserts the flag-off case forces HTML5 even when Rust reports native.
• webviewAudio.ts no longer inspects navigator.userAgent; the platform's audio backend is read from Rust (player_get_capabilities → usesWebviewAudio).
• experimentalNativeVideo on → video plays via ExoPlayer/MediaCodec, correctly positioned, with working seek, audio-track switch, and subtitle selection through the existing PlayerAdapter contract.
• No artefacts on rotation, background/foreground, or mini-player transition.
• adb shell dumpsys media.metrics (or logcat) confirms a hardware decoder is in use.
• Measured battery/thermal or CPU improvement over the HTML5 path on the same clip.

Failure path

• The blocking behaviour is documented in this spec with evidence.
• NativePlayerAdapter and the unreachable SurfaceView code are deleted, or explicitly retained with a correct comment.
• nativeAdapter.ts:11-14 no longer cites tauri#10152.

Either way:

• bun run check (0 errors), bun run test (892 passed), bun run check:boundary pass.
• cargo fmt / cargo clippy clean (no new warnings); cargo test passes (603 lib + 7 doc).

Note: this environment has no host WebKitGTK dev packages, no Android SDK and no bun, so all of the above were run inside the CI builder image (gitea.tourolle.paris/dtourolle/jellytau-builder:latest). On Fedora the bind mount needs :z for SELinux, and scripts/build-android.sh hardcodes ANDROID_HOME="$HOME/Android/Sdk", so the image's SDK at /opt/android-sdk must be symlinked there rather than passed by env var.

Testing

Adapter-selection logic is pure and testable without a device: assert createAdapter returns NativePlayerAdapter for backendKind: "native" with the flag on, and Html5PlayerAdapter in every other combination — including that the flag off forces HTML5 even when Rust says native. That last case is the regression guard.

Everything else is manual on-device; there is no meaningful way to unit-test surface compositing. Test on at least two devices — compositing behaviour varies by OEM and Android version.

Per CLAUDE.md, if the spike turns into a bug fix (e.g. seek breaks under the native adapter), write the failing test first.

TRACES

• createAdapter → // TRACES: UR-003, UR-004 | DR-023, DR-024
• Adapter-selection tests → UT-xxx
• No new requirement IDs; this spike either satisfies existing IR-004 expectations or documents why it cannot.

Notes for the implementer

• Do not skip Phase 1. If transparency does not work, phases 2 and 3 are wasted effort.
• VideoPlayer.svelte has a documented hazard: no lifecycle calls after an await in onMount — it flips to HTML5 mode and breaks Android seek. The adapter swap touches exactly this code path.
• tauri-specta tagged responses keep Rust field names (new_url, not newUrl).
• Android source edits go in src-tauri/android/src, then run scripts/sync-android-sources.sh.
• A parallel Claude session may be active — git diff first.