jellytau/docs/specs/android-native-video-spike.md

Spec: Android native video — transparent-webview spike

Status: Spike succeeded (2026-08-11); shipped behind experimentalNativeVideo, default off. Flipping that default shipped audio with no picture and was reverted (DR-172). Three defects behind that have since been fixed — DR-182 (nothing on the native path could lift the poster overlay), DR-183 (the JS bridges raced the page load), DR-184 (the SurfaceView was never detached). Branch fix/android-native-video-visible. The default stays off until the device criteria below are green.

The spike's central question is answered: yes. A SurfaceView can be composited behind a transparent Tauri WebView on Android. Nothing upstream blocked it and nothing upstream demonstrated it — this is, as far as the issue trackers show, the first working instance. The remaining flag is about test coverage and the unverified cases below, not about viability. Requirements: IR-004, UR-003, UR-004, UR-041 → DR-001, DR-004, DR-150, DR-151, DR-152 Note: the original draft cited DR-023/DR-024 here. Those are the subtitle and audio-track selection UI requirements — unrelated to this work. The IDs actually implemented are DR-150 (native rendering behind the flag), DR-151 (the severed SurfaceView attach chain) and DR-152 (capabilities reported by Rust). UX spec: n/a — no intended visual change; the video surface must land exactly where the <video> element is today Supersedes / revises: acts on finding 2 of playback-backend-unification.md

Summary

Test whether ExoPlayer's existing SurfaceView video path can be composited behind a transparent Tauri WebView on Android. If it works, Android regains hardware video decoding (MediaCodec) and libass-quality ASS/SSA subtitles, both of which the current webview path lacks. If it does not, we document why and delete the dead code.

This is a spike, not a feature commitment. The deliverable is a yes/no answer with evidence, plus either a working path behind a flag or a removal.

Motivation

createAdapter() hardcodes const effectiveKind = "html5" and does void backendKind, discarding the use_html5_element value Rust computes in get_player_status. As a result:

• NativePlayerAdapter is dead code.
• JellyTauPlayer.kt's getOrCreateSurfaceView() — which already calls setZOrderMediaOverlay(false) and wires setVideoSurfaceHolder — is unreachable.
• Android video decodes in the WebView instead of via MediaCodec, despite CodecDetector.kt going to the trouble of reporting hardware codec capabilities back to Rust for DeviceProfile generation.

The code comment in nativeAdapter.ts:11-14 justifies this by citing tauri#10152 as an upstream blocker. That justification is stale.

Why the blocker no longer holds

• tauri#10152 is open but dead since 2024-07-01, and it is a feature request ("Support transparent webviews on mobile"), not a bug report about compositing.
• The capability shipped in tauri commit 27d01834 (2024-09-02) — a clippy cleanup that moved transparent() out of the desktop-gated impl block, fencing only the tao call behind #[cfg(desktop)]. Because it landed as unrelated cleanup, nobody closed the issue.
• The black/white-screen reports (tauri#8381, tauri#9408) were a real but different bug: a broken JNI signature for setBackgroundColor, fixed in wry 0.39.4 (PR #1237). We ship wry 0.55.x.
• Current wry calls setBackgroundColor(0) unconditionally on Android when transparency is requested.

The honest caveat

Nobody has demonstrated SurfaceView-behind-WebView on Tauri Android. A search of both tauri-apps/tauri and tauri-apps/wry issues for surfaceview returns zero results, and the one native-video Tauri plugin (YeonV/tauri-plugin-videoplayer) sidesteps compositing by launching a separate fullscreen Activity. Nothing upstream blocks this; nothing upstream proves it. Hence: spike, not feature.

Note this is the Android question only. The equivalent Linux compositing problem is maintainer-declared unfixable and is not in scope — see the unification spec.

Layer assignment

Logic / responsibility	Layer	Why it belongs there
Which video backend this platform uses	Rust (existing)	get_player_status already computes use_html5_element. The frontend must consume it, not decide it. Restoring that is the point of the spike.
Surface creation, z-ordering, setVideoSurfaceHolder lifecycle	Kotlin	Android platform mechanics; already written in JellyTauPlayer.kt.
Seek/audio-track strategy	Rust (existing)	Already returned by player_seek_video / player_switch_audio_track; NativePlayerAdapter executes the chosen primitive. Unchanged — this is exactly what the PlayerAdapter contract was built for.
Positioning the surface under the video viewport	Frontend	Pure presentation/layout. This is the risk area — see Design.

Design

Phase 1 — prove compositing (no app changes)

Before touching the adapter factory, verify the primitive works at all:

1. Set "transparent": true in tauri.conf.json for the Android build, plus html, body { background: transparent; }.
2. Confirm the WebView is genuinely transparent (a native view behind it is visible) and that the app does not regress to a black/white screen.

If this fails, stop — everything downstream is moot, and the finding is that Tauri Android transparency is still broken in practice despite the shipped fix.

Phase 2 — un-hardcode the factory

// src/lib/player/adapters/index.ts
export function createAdapter({ backendKind, host, bridge }: CreateAdapterArgs): PlayerAdapter {
  return backendKind === "native"
    ? new NativePlayerAdapter(host)
    : new Html5PlayerAdapter(host, bridge);
}

backendKind comes from get_player_status (VideoBackend::Native on Android). Gate behind a setting — experimentalNativeVideo, default off — so a broken spike cannot ship as a regression. Rust already owns this decision; the flag only suppresses it.

Also in scope: remove the user-agent sniffing in src/lib/services/webviewAudio.ts:30-41. It re-derives which audio backend the platform has from navigator.userAgent ("matching the Rust cfg gate", per its own comment) — the frontend deciding a backend fact it should be told. Same root cause as the hardcode above, same fix: consume the value Rust already computes. Fold it in here rather than leaving a second, subtler copy of the bug behind. If get_player_status does not currently expose enough to cover the audio case, add the field — that is backend work, and correct.

Implementation findings (2026-08-11)

Two blockers existed that this spec did not anticipate. Both were in code the spec assumed was merely unreachable; it was also broken.

1. The Kotlin attach chain was severed. JellyTauPlayer.setActivity() had zero callers anywhere in the tree. currentActivity was therefore always null, so autoAttachSurface() logged "Cannot attach surface - no Activity reference" and returned. The SurfaceView was created and wired to ExoPlayer but never added to the view hierarchy — video would have decoded to a surface that was never on screen, regardless of webview transparency. Fixed by calling JellyTauPlayer.setActivity(this) from MainActivity.onCreate.

Note the knock-on: PictureInPictureManager.canEnterPip() gates on VideoOverlayManager.isVideoSurfaceAttached(), which was permanently false. PiP on the video path was dead for the same reason.

2. createAdapter() was not the real gate. It is never called by production code — VideoPlayer.svelte constructs Html5PlayerAdapter directly. The actual override was VideoPlayer.svelte's INTERIM block, which read Rust's useHtml5Element, forced it to true, and called playerStop() to kill the native backend player_play_item had just started. Both sites are now fixed; VideoPlayer.svelte routes through createAdapter() so there is one gate.

Transparency needs two independent layers cleared, not one. The spec's Phase 1 named only html, body. Clearing just the page leaves the WebView widget's own background opaque, which is a black screen with audio — the exact symptom the INTERIM comment described as "native surface not visible". Both are now toggled together by $lib/utils/videoSurface.ts:

Layer	Cleared by	Reachable from
WebView widget background + window drawable	AndroidVideoSurface.setTransparent() (MainActivity)	Kotlin only
html/body + app-shell --color-background	data-native-video attribute → app.css	CSS only

Transparency is scoped to tauri.android.conf.json rather than the base config: a transparent window on Linux is a regression, since nothing renders behind it. It is also toggled per-session rather than set once — a permanently transparent window shows the launcher through the rest of the app.

Phase 3 — surface positioning

The hard part, and where this most likely fails. The webview's <video> element occupies a laid-out box; the SurfaceView must be positioned to match it, and kept matched through scroll, rotation, and mini-player transitions.

Approach: the video view reports its getBoundingClientRect() to Rust, which forwards the rect to Kotlin to position the SurfaceView. This is the same "faking it" technique the ecosystem uses on desktop — acceptable here only if the video is effectively fullscreen on Android, which it is in the player route.

Explicit failure criterion: if the surface cannot be kept aligned during rotation or the mini-player transition without visible artefacts, the spike fails and we keep HTML5. Do not ship a janky native path for a codec win.

Update: no rect plumbing was needed. The premise — that the surface must be positioned to match a laid-out <video> box — does not hold on the player route, where video is fullscreen. VideoOverlayManager adds the SurfaceView at index 0 of android.R.id.content with MATCH_PARENT, and fitSurfaceToScreen() (JellyTauPlayer.kt) already letterboxes/pillarboxes to the real video aspect ratio and re-centres via a Gravity.CENTER FrameLayout.LayoutParams. Rotation is handled by an OnLayoutChangeListener that re-fits on any bounds change. The frontend's native branch is a bare flex-1 box, so there is no rect to report and nothing to keep in sync.

Fullscreen playback is confirmed working on device. But this reasoning rests entirely on the fullscreen assumption, so the mini-player transition is the known gap — it is the one case where the surface is not fullscreen, and therefore the one case where the "no rect plumbing needed" conclusion could still turn out to be wrong. If artefacts appear there, the fix is the rect reporting this section originally proposed, scoped to that transition alone.

A trap for the next implementer

There is a stale duplicate player at src-tauri/android/app/src/main/java/com/dtourolle/jellytau/player/JellyTauPlayer.kt (only commit: cfddc1e "First working POC"). No sourceSets entry points at it, so it is not compiled — but edits made there silently do nothing. The canonical tree is src-tauri/android/src, synced into gen/ by scripts/sync-android-sources.sh.

What we gain if it works

• Hardware decode via MediaCodec — CodecDetector.kt already reports capabilities; the DeviceProfile would finally match what actually plays.
• ASS/SSA subtitles are not automatic. ExoPlayer cannot render them; that would require libmpv, which is a separate and much larger decision (see the unification spec's engine comparison). Scope this spike to hardware decode only, and do not claim subtitle improvements from it.

Out of scope

• Linux native video. Maintainer-declared unfixable on WebKitGTK/Wayland.
• Replacing ExoPlayer with libmpv on Android.
• Windows native video.
• Removing the HTML5 path. It stays as the default and the fallback.

Acceptance criteria

The spike is complete when one of these is true:

Success path

• Transparent WebView confirmed working on a physical device (reported by the maintainer; the config that enables it is now committed in tauri.android.conf.json).
• experimentalNativeVideo off → behaviour byte-identical to today. Guarded by adapterSelection.test.ts, which asserts the flag-off case forces HTML5 even when Rust reports native.
• webviewAudio.ts no longer inspects navigator.userAgent; the platform's audio backend is read from Rust (player_get_capabilities → usesWebviewAudio).
• experimentalNativeVideo on → video plays via ExoPlayer, correctly positioned, on a physical device (2026-08-11). The surface reaches the hierarchy and is visible through the transparent WebView — the whole point of the spike.
• The poster/title card comes down on the native path. It never could: every markMediaReady() call site is a <video> element event and the native branch renders no element, so an opaque bg-black overlay covered the ExoPlayer surface for the whole session. See DR-182; guarded by mediaReady.test.ts (UT-184) and VideoPlayer.nativeReveal.test.ts (UT-185), the latter written failing first.
• The AndroidVideoSurface bridge is installed before the page that calls it loads, via WryActivity.onWebViewCreate instead of a 500 ms tree walk, and a missing bridge now logs an error instead of no-oping. See DR-183.
• The SurfaceView is detached when video stops, instead of accumulating one leaked view per native video. See DR-184.
• Seek, audio-track switch and subtitle selection exercised through NativePlayerAdapter. Playback is confirmed; these individual controls are not yet each verified on the native path.
• No artefacts on rotation, background/foreground, or mini-player transition — the last is the one case the fullscreen assumption does not cover, so it is the likeliest place to find a problem.
• adb shell dumpsys media.metrics (or logcat) confirms a hardware decoder is in use. Plausible but unmeasured — do not claim the MediaCodec win until this is read.
• Measured battery/thermal or CPU improvement over the HTML5 path on the same clip.

Failure path

• The blocking behaviour is documented in this spec with evidence.
• NativePlayerAdapter and the unreachable SurfaceView code are deleted, or explicitly retained with a correct comment.
• nativeAdapter.ts:11-14 no longer cites tauri#10152.

Either way:

• bun run check (0 errors), bun run test (997 passed), bun run check:boundary pass.
• cargo fmt / cargo clippy clean (no new warnings); cargo test passes (603 lib + 7 doc).

Why the 2026-08-11 verification and DR-172 do not contradict each other

The spike was reported working on device; the same path then shipped as audio with no picture. Both are consistent with DR-182: the poster overlay is drawn only while isMediaReady is false, and the native path has no way to set it, so what the surface shows depends entirely on whether that overlay is on screen — not on whether compositing works. Any run that reached the player through a path leaving isMediaReady already true (a handoff return, a re-render, a session that had previously played on the HTML5 path) shows video; a cold start into the native path never does. That is also why DR-172 read the symptom as a compositing failure: on screen the two are identical, and the one piece of evidence separating them — WebView transparent = true never being logged — points at DR-183 rather than at the compositing itself.

This reasoning is not yet device-confirmed. It explains the reports and is backed by the code, but the criteria above are what settle it.

Note: this environment has no host WebKitGTK dev packages, no Android SDK and no bun, so all of the above were run inside the CI builder image (gitea.tourolle.paris/dtourolle/jellytau-builder:latest). On Fedora the bind mount needs :z for SELinux, and scripts/build-android.sh hardcodes ANDROID_HOME="$HOME/Android/Sdk", so the image's SDK at /opt/android-sdk must be symlinked there rather than passed by env var.

Testing

Adapter-selection logic is pure and testable without a device: assert createAdapter returns NativePlayerAdapter for backendKind: "native" with the flag on, and Html5PlayerAdapter in every other combination — including that the flag off forces HTML5 even when Rust says native. That last case is the regression guard.

Everything else is manual on-device; there is no meaningful way to unit-test surface compositing. Test on at least two devices — compositing behaviour varies by OEM and Android version.

Per CLAUDE.md, if the spike turns into a bug fix (e.g. seek breaks under the native adapter), write the failing test first.

TRACES

• createAdapter → // TRACES: UR-003, UR-004 | DR-004, DR-150 | UT-149
• Adapter-selection tests → UT-xxx
• No new requirement IDs; this spike either satisfies existing IR-004 expectations or documents why it cannot.

Notes for the implementer

• Do not skip Phase 1. If transparency does not work, phases 2 and 3 are wasted effort.
• VideoPlayer.svelte has a documented hazard: no lifecycle calls after an await in onMount — it flips to HTML5 mode and breaks Android seek. The adapter swap touches exactly this code path.
• tauri-specta tagged responses keep Rust field names (new_url, not newUrl).
• Android source edits go in src-tauri/android/src, then run scripts/sync-android-sources.sh.
• A parallel Claude session may be active — git diff first.