dtourolle/jellytau
1
0
Fork 0
Code Issues Pull Requests Actions 8 Packages Projects Releases 37 Wiki Activity

feat(player): render Android video natively behind a transparent webview (DR-150, DR-151, DR-152)

Browse Source 
Rust already reported `use_html5_element: false` on Android, but two frontend
overrides threw that answer away, so ExoPlayer's video path had never actually
run. Both are lifted behind an `experimentalNativeVideo` opt-in (default off).

The flag is a suppressor, never a promoter: off forces HTML5 even where Rust
says native, so an in-progress spike cannot ship as the default, but it can
never select native where Rust reported HTML5 — Linux cannot composite behind
WebKitGTK, and promoting there would be a black screen.

Two blockers the spec did not anticipate, both in code assumed to be merely
unreachable rather than broken:

- `JellyTauPlayer.setActivity()` had zero callers, so `currentActivity` was
  always null and `autoAttachSurface()` bailed. The SurfaceView was created and
  wired to ExoPlayer but never added to the view hierarchy — video would have
  decoded to a surface that was never on screen, whatever the webview did.
  This also revives PiP on the video path, which gated on the same flag.
- `createAdapter()` was not the real gate; it is never called in production.
  The actual override was in VideoPlayer.svelte, which forced HTML5 and stopped
  the native backend `player_play_item` had just started. Both sites now route
  through `createAdapter()`.

Compositing needs two independent opaque layers cleared, not one. Clearing only
the page leaves the WebView widget opaque — audio over a black picture, exactly
the symptom the old INTERIM comment described. `videoSurface.ts` toggles both:
the widget background and window drawable from Kotlin, the page backgrounds via
a `data-native-video` attribute keyed by app.css. Transparency lives in
`tauri.android.conf.json` so Linux keeps an opaque window, and is scoped to the
playback session so the launcher never shows through the rest of the app.

Phase 3's rect plumbing turned out to be unnecessary: video is fullscreen on the
player route, and `fitSurfaceToScreen()` already letterboxes and re-fits on
rotation. The mini-player transition remains unverified on device.

Also removes the `navigator.userAgent` sniffing in webviewAudio.ts, which was a
second copy of the Rust cfg gate free to drift from it. `player_get_capabilities`
now reports `usesWebviewAudio` and `supportsNativeVideo` from those same gates.

Tests: adapter selection covers the full matrix, including the regression guard
that the flag off beats Rust. Written first and confirmed failing (2 of 7) before
the fix.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
dtourolle
2026-08-11 20:57:58 +02:00
co-authored by Claude Opus 5
parent 07d10dfed7
commit e144e62b31
16 changed files with 745 additions and 56 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/lib/components/player/VideoPlayer.svelte
+55 -15
View File
@@ -26,8 +26,18 @@
import { playbackPosition, playerState } from "$lib/stores/player";
import * as html5Adapter from "$lib/player/html5Adapter";
import { playerController } from "$lib/player";
import { Html5PlayerAdapter, type Html5ElementBridge } from "$lib/player/adapters";
import {
createAdapter,
Html5PlayerAdapter,
type PlayerAdapter,
type Html5ElementBridge,
} from "$lib/player/adapters";
import { createRustReportHost } from "$lib/player/adapters/rustReportHost";
import { experimentalNativeVideo } from "$lib/stores/nativeVideo";
import {
enableNativeVideoCompositing,
disableNativeVideoCompositing,
} from "$lib/utils/videoSurface";
import { isPipSupported, enterPip, setAutoEnterEnabled } from "$lib/utils/pictureInPicture";
import {
createTapGestureState,
@@ -166,7 +176,10 @@
// VideoPlayer supplies a narrow bridge for the element/HLS-coupled parts and
// registers the adapter with the facade so control intents — from UI OR from a
// backend control event (lockscreen/remote/sleep) — reach this element.
let playerAdapter: Html5PlayerAdapter | null = null;
// Widened from Html5PlayerAdapter: the native path registers a
// NativePlayerAdapter here. Element-coupled work is guarded by
// `useHtml5Element`, not by narrowing this type.
let playerAdapter: PlayerAdapter | null = null;
function tearDownHls() {
if (hls) {
@@ -645,15 +658,16 @@
backendChosen = true;
console.log(`[VideoPlayer] Backend: ${response.backend}, useHtml5Element: ${useHtml5Element}`);
// INTERIM (until the video-player API refactor lands): always render
// through the webview HTML5 element, including Android. The native
// ExoPlayer SurfaceView sits behind an opaque webview and has never
// actually been visible (an init bug kept the app on the HTML5 path
// since the POC), so true native mode plays audio behind a frozen
// picture. Stop the native backend and let the webview own playback,
// matching Linux behavior and avoiding dual audio.
if (!useHtml5Element) {
console.warn("[VideoPlayer] Native video backend reported - overriding to HTML5 rendering (native surface not visible through webview)");
// Rust reported a native backend (Android/ExoPlayer). Honour it only if
// the user opted into the experimental native path; otherwise fall back
// to the webview element, which is what shipped by default.
//
// The flag is a suppressor, never a promoter — see createAdapter(). When
// it is off we must also stop the native backend that player_play_item
// just started, or ExoPlayer and the <video> element both decode the
// same stream and the audio doubles.
if (!useHtml5Element && !$experimentalNativeVideo) {
console.log("[VideoPlayer] Native backend available but experimentalNativeVideo is off - using HTML5");
useHtml5Element = true;
try {
await commands.playerStop();
@@ -661,6 +675,14 @@
} catch (err) {
console.warn("[VideoPlayer] Failed to stop native backend:", err);
}
} else if (!useHtml5Element) {
// Native path: clear the opaque layers between the viewport and the
// ExoPlayer SurfaceView (webview widget background + page background).
// Paired with disableNativeVideoCompositing() in the teardown path —
// leaving this on renders the rest of the app over a transparent
// window.
console.log("[VideoPlayer] Using native ExoPlayer video surface");
enableNativeVideoCompositing();
}
// If using HTML5 element for non-transcoded content, stop the backend player
@@ -679,14 +701,24 @@
didStartNativePlayback = true; // Track that we need to stop backend on unmount
}
// Register the HTML5 player adapter with the facade so control intents
// (UI or backend lockscreen/remote/sleep events) route to this element.
if (useHtml5Element) {
// Register the adapter with the facade so control intents (UI, or a
// backend lockscreen/remote/sleep event) route to whatever is actually
// rendering. Both paths need one: the native adapter forwards control
// intents to ExoPlayer over IPC.
{
const host = createRustReportHost(media.id, {
onEnded: () => notifyEnded(),
onStreamUrlChanged: (u) => { currentStreamUrl = u; },
});
playerAdapter = new Html5PlayerAdapter(host, adapterBridge);
playerAdapter = createAdapter({
backendKind: useHtml5Element ? "html5" : "native",
host,
bridge: adapterBridge,
// useHtml5Element is already the resolved decision above, so the
// flag has had its say; pass it through for the invariant check.
experimentalNativeVideo: $experimentalNativeVideo,
});
// No-op for the native adapter, which owns no DOM element.
playerAdapter.attach(videoElement);
playerController.setActiveAdapter(playerAdapter);
}
@@ -780,6 +812,14 @@
});
onDestroy(async () => {
// FIRST, and synchronously: restore the opaque webview/page backgrounds.
//
// This callback is async, so anything after an `await` may run a frame or
// more later. Leaving the window transparent for even that long shows the
// launcher/wallpaper through the app as the player unwinds. Unconditional
// and idempotent — a no-op when compositing was never enabled.
disableNativeVideoCompositing();
// Stop RAF loop
stopTimeUpdates();