docs(player): backend unification findings + correct false parity claims
Investigation into unifying the playback backends (Linux/MPV, Android/ExoPlayer, Windows/webview) onto one engine with hardware acceleration. Conclusion: video cannot be unified onto a native engine; audio can. The blocker is not mpv-specific. WebKitGTK, WebView2 and Android WebView each draw into their own compositor surface, so a native video surface sits either entirely above or entirely below the webview and cannot interleave with HTML. GStreamer and libVLC fail identically. mpv would additionally regress streaming: it has no adaptive bitrate, while the current hls.js path does. Six specs added: - playback-backend-unification: the analysis and decision, with evidence - android-audio-settings-parity: set_audio_settings on ExoPlayerBackend - android-native-video-spike: timeboxed test of SurfaceView compositing - windows-native-audio-backend: replace the webview <audio> shim with libmpv - libmpv2-migration: dead libmpv git pin -> libmpv2, plus a LICENSE file - playback-docs-corrections: the requirement-status fixes applied here Corrections to requirements.md, all verified against source: - UR-031/DR-034 claimed crossfade was "Done (Linux only)". It is implemented nowhere (mpv_backend.rs has a bare TODO) and is architecturally blocked on mpv, whose single-stream audio chain cannot feed acrossfade's two inputs. - Parity matrix listed crossfade as a Linux/Android gap; it is neither. - The matrix omitted the equalizer, which has the same Linux-only shape. - The suggested ConcatenatingMediaSource is deprecated in current Media3. nativeAdapter.ts cited tauri#10152 as an upstream blocker for native Android video. That issue is a stale feature request, dead since 2024-07-01; the capability shipped in tauri 27d01834 (2024-09-02), and the related black-screen bug was fixed in wry 0.39.4 (we ship 0.55.x). What is genuinely unproven is SurfaceView-behind-WebView compositing, which the spike now tracks.
This commit is contained in:
@@ -0,0 +1,196 @@
|
||||
# Spec: Android native video — transparent-webview spike
|
||||
|
||||
**Status:** Proposed (spike — timeboxed, may conclude "not viable")
|
||||
**Requirements:** IR-004, UR-003, UR-004 → DR-001, DR-023, DR-024
|
||||
**UX spec:** n/a — no intended visual change; the video surface must land exactly where the `<video>` element is today
|
||||
**Supersedes / revises:** acts on finding 2 of [playback-backend-unification.md](playback-backend-unification.md)
|
||||
|
||||
## Summary
|
||||
|
||||
Test whether ExoPlayer's existing `SurfaceView` video path can be composited
|
||||
behind a transparent Tauri WebView on Android. If it works, Android regains
|
||||
hardware video decoding (MediaCodec) and libass-quality ASS/SSA subtitles, both
|
||||
of which the current webview path lacks. If it does not, we document why and
|
||||
delete the dead code.
|
||||
|
||||
This is a **spike**, not a feature commitment. The deliverable is a yes/no answer
|
||||
with evidence, plus either a working path behind a flag or a removal.
|
||||
|
||||
## Motivation
|
||||
|
||||
`createAdapter()` hardcodes `const effectiveKind = "html5"` and does
|
||||
`void backendKind`, discarding the `use_html5_element` value Rust computes in
|
||||
`get_player_status`. As a result:
|
||||
|
||||
- `NativePlayerAdapter` is dead code.
|
||||
- `JellyTauPlayer.kt`'s `getOrCreateSurfaceView()` — which already calls
|
||||
`setZOrderMediaOverlay(false)` and wires `setVideoSurfaceHolder` — is
|
||||
unreachable.
|
||||
- Android video decodes in the WebView instead of via MediaCodec, despite
|
||||
`CodecDetector.kt` going to the trouble of reporting hardware codec
|
||||
capabilities back to Rust for DeviceProfile generation.
|
||||
|
||||
The code comment in `nativeAdapter.ts:11-14` justifies this by citing
|
||||
tauri#10152 as an upstream blocker. **That justification is stale.**
|
||||
|
||||
### Why the blocker no longer holds
|
||||
|
||||
- tauri#10152 is open but **dead since 2024-07-01**, and it is a *feature
|
||||
request* ("Support transparent webviews on mobile"), not a bug report about
|
||||
compositing.
|
||||
- The capability shipped in tauri commit `27d01834` (2024-09-02) — a clippy
|
||||
cleanup that moved `transparent()` out of the desktop-gated impl block, fencing
|
||||
only the tao call behind `#[cfg(desktop)]`. Because it landed as unrelated
|
||||
cleanup, nobody closed the issue.
|
||||
- The black/white-screen reports (tauri#8381, tauri#9408) were a real but
|
||||
*different* bug: a broken JNI signature for `setBackgroundColor`, fixed in
|
||||
**wry 0.39.4** (PR #1237). We ship wry 0.55.x.
|
||||
- Current wry calls `setBackgroundColor(0)` unconditionally on Android when
|
||||
transparency is requested.
|
||||
|
||||
### The honest caveat
|
||||
|
||||
**Nobody has demonstrated SurfaceView-behind-WebView on Tauri Android.** A search
|
||||
of both `tauri-apps/tauri` and `tauri-apps/wry` issues for `surfaceview` returns
|
||||
zero results, and the one native-video Tauri plugin
|
||||
(`YeonV/tauri-plugin-videoplayer`) sidesteps compositing by launching a separate
|
||||
fullscreen Activity. Nothing upstream blocks this; nothing upstream proves it.
|
||||
Hence: spike, not feature.
|
||||
|
||||
Note this is the *Android* question only. The equivalent Linux compositing
|
||||
problem is maintainer-declared unfixable and is **not** in scope — see the
|
||||
unification spec.
|
||||
|
||||
## Layer assignment
|
||||
|
||||
| Logic / responsibility | Layer | Why it belongs there |
|
||||
|------------------------|-------|----------------------|
|
||||
| Which video backend this platform uses | Rust (existing) | `get_player_status` already computes `use_html5_element`. The frontend must *consume* it, not decide it. Restoring that is the point of the spike. |
|
||||
| Surface creation, z-ordering, `setVideoSurfaceHolder` lifecycle | Kotlin | Android platform mechanics; already written in `JellyTauPlayer.kt`. |
|
||||
| Seek/audio-track *strategy* | Rust (existing) | Already returned by `player_seek_video` / `player_switch_audio_track`; `NativePlayerAdapter` executes the chosen primitive. Unchanged — this is exactly what the `PlayerAdapter` contract was built for. |
|
||||
| Positioning the surface under the video viewport | Frontend | Pure presentation/layout. **This is the risk area** — see Design. |
|
||||
|
||||
## Design
|
||||
|
||||
### Phase 1 — prove compositing (no app changes)
|
||||
|
||||
Before touching the adapter factory, verify the primitive works at all:
|
||||
|
||||
1. Set `"transparent": true` in `tauri.conf.json` for the Android build, plus
|
||||
`html, body { background: transparent; }`.
|
||||
2. Confirm the WebView is genuinely transparent (a native view behind it is
|
||||
visible) and that the app does not regress to a black/white screen.
|
||||
|
||||
If this fails, stop — everything downstream is moot, and the finding is that
|
||||
Tauri Android transparency is still broken in practice despite the shipped fix.
|
||||
|
||||
### Phase 2 — un-hardcode the factory
|
||||
|
||||
```ts
|
||||
// src/lib/player/adapters/index.ts
|
||||
export function createAdapter({ backendKind, host, bridge }: CreateAdapterArgs): PlayerAdapter {
|
||||
return backendKind === "native"
|
||||
? new NativePlayerAdapter(host)
|
||||
: new Html5PlayerAdapter(host, bridge);
|
||||
}
|
||||
```
|
||||
|
||||
`backendKind` comes from `get_player_status` (`VideoBackend::Native` on Android).
|
||||
Gate behind a setting — `experimentalNativeVideo`, default **off** — so a broken
|
||||
spike cannot ship as a regression. Rust already owns this decision; the flag only
|
||||
suppresses it.
|
||||
|
||||
**Also in scope: remove the user-agent sniffing in
|
||||
`src/lib/services/webviewAudio.ts:30-41`.** It re-derives which audio backend the
|
||||
platform has from `navigator.userAgent` ("matching the Rust cfg gate", per its own
|
||||
comment) — the frontend deciding a backend fact it should be told. Same root cause
|
||||
as the hardcode above, same fix: consume the value Rust already computes. Fold it
|
||||
in here rather than leaving a second, subtler copy of the bug behind. If
|
||||
`get_player_status` does not currently expose enough to cover the audio case, add
|
||||
the field — that is backend work, and correct.
|
||||
|
||||
### Phase 3 — surface positioning
|
||||
|
||||
The hard part, and where this most likely fails. The webview's `<video>` element
|
||||
occupies a laid-out box; the `SurfaceView` must be positioned to match it, and
|
||||
kept matched through scroll, rotation, and mini-player transitions.
|
||||
|
||||
Approach: the video view reports its `getBoundingClientRect()` to Rust, which
|
||||
forwards the rect to Kotlin to position the `SurfaceView`. This is the same
|
||||
"faking it" technique the ecosystem uses on desktop — acceptable here *only if*
|
||||
the video is effectively fullscreen on Android, which it is in the player route.
|
||||
|
||||
**Explicit failure criterion**: if the surface cannot be kept aligned during
|
||||
rotation or the mini-player transition without visible artefacts, the spike fails
|
||||
and we keep HTML5. Do not ship a janky native path for a codec win.
|
||||
|
||||
### What we gain if it works
|
||||
|
||||
- **Hardware decode via MediaCodec** — `CodecDetector.kt` already reports
|
||||
capabilities; the DeviceProfile would finally match what actually plays.
|
||||
- **ASS/SSA subtitles** are *not* automatic. ExoPlayer cannot render them; that
|
||||
would require libmpv, which is a separate and much larger decision (see the
|
||||
unification spec's engine comparison). Scope this spike to hardware decode
|
||||
only, and do not claim subtitle improvements from it.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- Linux native video. Maintainer-declared unfixable on WebKitGTK/Wayland.
|
||||
- Replacing ExoPlayer with libmpv on Android.
|
||||
- Windows native video.
|
||||
- Removing the HTML5 path. It stays as the default and the fallback.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
The spike is **complete** when one of these is true:
|
||||
|
||||
**Success path**
|
||||
- [ ] Transparent WebView confirmed working on a physical device.
|
||||
- [ ] `experimentalNativeVideo` off → behaviour byte-identical to today.
|
||||
- [ ] `webviewAudio.ts` no longer inspects `navigator.userAgent`; the platform's audio backend is read from Rust.
|
||||
- [ ] `experimentalNativeVideo` on → video plays via ExoPlayer/MediaCodec, correctly positioned, with working seek, audio-track switch, and subtitle selection through the existing `PlayerAdapter` contract.
|
||||
- [ ] No artefacts on rotation, background/foreground, or mini-player transition.
|
||||
- [ ] `adb shell dumpsys media.metrics` (or logcat) confirms a hardware decoder is in use.
|
||||
- [ ] Measured battery/thermal or CPU improvement over the HTML5 path on the same clip.
|
||||
|
||||
**Failure path**
|
||||
- [ ] The blocking behaviour is documented in this spec with evidence.
|
||||
- [ ] `NativePlayerAdapter` and the unreachable `SurfaceView` code are deleted, or explicitly retained with a *correct* comment.
|
||||
- [ ] `nativeAdapter.ts:11-14` no longer cites tauri#10152.
|
||||
|
||||
Either way:
|
||||
- [ ] `bun run check`, `bun run test`, `bun run check:boundary` pass.
|
||||
- [ ] `cargo fmt` / `cargo clippy` clean; `bun run test:rust` passes.
|
||||
|
||||
## Testing
|
||||
|
||||
Adapter-selection logic is pure and testable without a device: assert
|
||||
`createAdapter` returns `NativePlayerAdapter` for `backendKind: "native"` with
|
||||
the flag on, and `Html5PlayerAdapter` in every other combination — including that
|
||||
the flag off forces HTML5 even when Rust says native. That last case is the
|
||||
regression guard.
|
||||
|
||||
Everything else is manual on-device; there is no meaningful way to unit-test
|
||||
surface compositing. Test on at least two devices — compositing behaviour varies
|
||||
by OEM and Android version.
|
||||
|
||||
Per CLAUDE.md, if the spike turns into a bug fix (e.g. seek breaks under the
|
||||
native adapter), write the failing test first.
|
||||
|
||||
## TRACES
|
||||
|
||||
- `createAdapter` → `// TRACES: UR-003, UR-004 | DR-023, DR-024`
|
||||
- Adapter-selection tests → `UT-xxx`
|
||||
- No new requirement IDs; this spike either satisfies existing IR-004 expectations or documents why it cannot.
|
||||
|
||||
## Notes for the implementer
|
||||
|
||||
- **Do not skip Phase 1.** If transparency does not work, phases 2 and 3 are
|
||||
wasted effort.
|
||||
- `VideoPlayer.svelte` has a documented hazard: no lifecycle calls after an
|
||||
`await` in `onMount` — it flips to HTML5 mode and breaks Android seek. The
|
||||
adapter swap touches exactly this code path.
|
||||
- tauri-specta tagged responses keep Rust field names (`new_url`, not `newUrl`).
|
||||
- Android source edits go in `src-tauri/android/src`, then run
|
||||
`scripts/sync-android-sources.sh`.
|
||||
- A parallel Claude session may be active — `git diff` first.
|
||||
Reference in New Issue
Block a user