DR-235 phase 3. Every video renderer is native now: mpv on Linux and Windows, ExoPlayer on Android, all drawing behind the transparent webview. The HTML5 <video> path is gone, not bypassed: - Frontend: hls.js, Html5PlayerAdapter and its compatibility shim, the createAdapter factory, streamTransport, hlsRecovery, timeTracking, videoFit, the <video>/<track> markup and every element handler in VideoPlayer (3277 -> 2144 lines), the experimentalNativeVideo store and its Settings toggle, webviewVideoFallback/supportsNativeVideo, and the setHtml5VideoState PiP bridge call. NativePlayerAdapter is the one video adapter; webview audio gets its own adapter kind. - Rust: use_html5 dropped from player_seek_video, player_switch_audio_track and player_set_stream_quality with the Html5* strategies and ReloadStream responses; use_html5_element and VideoBackend dropped from PlayerStatus; player_play_item always loads the backend (set_current_item removed); Capabilities::webview removed; the WebKitGTK GStreamer/VAAPI setup (and its gst-inspect spawn) removed. - Android: the HTML5 video state in PictureInPictureManager and ScreenWakeManager, and the bridge method feeding it. - CSP: connect-src loses http:/https: and worker-src loses blob: - both existed for hls.js; with it gone they were only an exfiltration channel and a blob worker for injected script. A test now keeps them out. mpv takes over what the <video> element did (mpv_tracks, UT-275): subtitles are the WebVTT list the play request carries, queued on sub-files and selected by position in that list, starting off; audio tracks are selected by position in the file; sid/aid are reset before each load. Without this, Linux video had no subtitle selection and a direct-play audio switch failed since mpv became its renderer. Verified: Rust 948 passing, and the same 948 cross-compiled for Windows under wine against the shipped DLL (track tests included); frontend 1111 passing; aarch64 debug APK builds. Lint warnings 158 -> 146, CI ratchet tightened to match. Not yet seen on Windows hardware.
762 lines
30 KiB
Markdown
762 lines
30 KiB
Markdown
# Platform-Specific Player Backends
|
||
|
||
## Player Events System
|
||
|
||
**Location**: `src-tauri/src/player/events.rs`
|
||
|
||
The player uses a push-based event system to notify the frontend of state changes:
|
||
|
||
```rust
|
||
pub enum PlayerStatusEvent {
|
||
/// Playback position updated (emitted periodically during playback)
|
||
PositionUpdate { position: f64, duration: f64 },
|
||
|
||
/// Player state changed
|
||
StateChanged { state: String, media_id: Option<String> },
|
||
|
||
/// Media has finished loading and is ready to play
|
||
MediaLoaded { duration: f64 },
|
||
|
||
/// Playback has ended naturally
|
||
PlaybackEnded,
|
||
|
||
/// Buffering state changed
|
||
Buffering { percent: u8 },
|
||
|
||
/// An error occurred during playback
|
||
Error { message: String, recoverable: bool },
|
||
|
||
/// Volume changed
|
||
VolumeChanged { volume: f32, muted: bool },
|
||
|
||
/// Sleep timer state changed
|
||
SleepTimerChanged {
|
||
mode: SleepTimerMode,
|
||
remaining_seconds: u32,
|
||
},
|
||
|
||
/// Show next episode popup with countdown
|
||
ShowNextEpisodePopup {
|
||
current_episode: MediaItem,
|
||
next_episode: MediaItem,
|
||
countdown_seconds: u32,
|
||
auto_advance: bool,
|
||
},
|
||
|
||
/// Countdown tick (emitted every second during autoplay countdown)
|
||
CountdownTick { remaining_seconds: u32 },
|
||
|
||
/// Queue changed (items added, removed, reordered, or playback mode changed)
|
||
QueueChanged {
|
||
items: Vec<MediaItem>,
|
||
current_index: Option<usize>,
|
||
shuffle: bool,
|
||
repeat: RepeatMode,
|
||
has_next: bool,
|
||
has_previous: bool,
|
||
},
|
||
|
||
/// Media session changed (activity context changed: Audio/Movie/TvShow/Idle)
|
||
SessionChanged { session: MediaSessionType },
|
||
}
|
||
```
|
||
|
||
Events are emitted via Tauri's event system:
|
||
|
||
```mermaid
|
||
flowchart LR
|
||
subgraph Backend["Player Backend"]
|
||
MPV["MPV/ExoPlayer"]
|
||
end
|
||
|
||
subgraph EventSystem["Event System"]
|
||
Emitter["TauriEventEmitter<br/>emit()"]
|
||
Bus["Tauri Event Bus<br/>'player-event'"]
|
||
end
|
||
|
||
subgraph Frontend["Frontend"]
|
||
Listener["playerEvents.ts<br/>Frontend Listener"]
|
||
Store["Player Store Update<br/>(position, state, etc)"]
|
||
end
|
||
|
||
MPV --> Emitter --> Bus --> Listener --> Store
|
||
```
|
||
|
||
**Frontend Listener** (`src/lib/services/playerEvents.ts`):
|
||
- Listens for `player-event` Tauri events
|
||
- Updates player/queue stores based on event type
|
||
- Auto-advances to next track on `PlaybackEnded`
|
||
- On `StateChanged` events, calls `invoke("player_get_queue")` to update `appState.hasNext`/`hasPrevious` -- this enables MiniPlayer skip button state
|
||
|
||
**Important**: The command is `player_get_queue` (returns `QueueStatus` with `hasNext`/`hasPrevious`). There is no `player_get_queue_status` command.
|
||
|
||
## Video is always native (no webview `<video>`)
|
||
|
||
**TRACES**: UR-080 | DR-231, DR-235, DR-237
|
||
|
||
Every video renderer is a native player drawing **behind** the transparent
|
||
webview, with the Svelte controls composited over it: ExoPlayer on Android, mpv on
|
||
Linux and Windows. There is no webview `<video>` element, no hls.js, and no
|
||
HTML5 adapter; the frontend has one video adapter, `NativePlayerAdapter`, and the
|
||
backend performs every seek, track switch and quality change itself.
|
||
|
||
Why the webview path was deleted rather than kept as a fallback:
|
||
|
||
- **The transcode was a decoder constraint.** The `<video>` element decodes
|
||
little beyond h264, so the desktop profile could only claim h264 and the server
|
||
re-encoded almost everything (7% direct play on a real library, against 85% for
|
||
the same library through ExoPlayer). The machine was never the limit — mpv was
|
||
already running for audio. (The Linux/Windows profile still claims h264 until
|
||
DR-234 widens it; see [desktop-native-video.md](../specs/desktop-native-video.md).)
|
||
- **Each renderer is another place for every bug.** Three video renderers meant
|
||
every seek strategy, track switch and lifecycle fix had three places to be got
|
||
right; the webview path was also where the renderer choice itself went wrong
|
||
(silent Linux video, a Windows soundtrack decoded twice — DR-237).
|
||
- **A fallback that decodes less is not a fallback.** Android showed it first
|
||
(DR-293): an original-file download plays silent in the webview. With no
|
||
webview path there is no silent downgrade to fall into; a failed mpv init
|
||
emits `backend-init-failed` instead.
|
||
|
||
What survives of the webview reporting pipeline is audio-only:
|
||
`WebviewAudioBackend` plays through a hidden `<audio>` element and reports back
|
||
through the `player_report_*` commands (`rustReportHost.ts`), for a desktop with
|
||
no mpv. No shipped platform uses it.
|
||
|
||
## Native video on the desktop (mpv)
|
||
|
||
**TRACES**: UR-080 | DR-231 … DR-237, DR-298, DR-299
|
||
|
||
The mpv half is shared; only the surface differs per platform
|
||
(`mpv_backend::video_output`):
|
||
|
||
| Platform | Output | Where the picture goes |
|
||
|---|---|---|
|
||
| Linux | `vo=libmpv` (render API) | An FBO drawn in the main window's own `GtkBox` `draw` handler, which GTK paints *before* its children — so beneath the webview, with no widget reparenting (`video_surface.rs`; a `GtkOverlay` aborts the process on the first click, see that module) |
|
||
| Windows | `vo=gpu-next,gpu`, `wid=<HWND>` | mpv renders as a child of the app window, beneath the transparent WebView2 — the arrangement tauri-plugin-libmpv ships. `wid` only takes effect before initialisation, so `MpvBackend::new` takes the handle; with no handle it draws nothing rather than open a window of its own. mpv's controller, bindings and cursor handling are off |
|
||
|
||
In both, the page clears its opaque backgrounds while a video is on screen
|
||
(`data-native-video`, the same CSS Android uses).
|
||
|
||
**Tracks** (`mpv_tracks.rs`): subtitles are the WebVTT list the play request
|
||
carries, queued on `sub-files` before the load and selected **by position in
|
||
that list** — the same meaning ExoPlayer gives `player_set_subtitle_track`.
|
||
Selection starts off (the menu opens on "Off") and `sid`/`aid` are reset before
|
||
every load, so a choice made for one item cannot leak into the next. Audio tracks
|
||
are selected by position in the file; a transcode carries one track and is
|
||
re-opened instead.
|
||
|
||
**Two rules for every libmpv handle** (`mpv_command.rs`):
|
||
|
||
- Commands go through `mpv_command::command`, an argv built for `mpv_command`,
|
||
never the pinned crate's `Mpv::command`, which joins its arguments into a
|
||
string that mpv parses — `;` chains a second command, so a track title in a
|
||
downloaded file's path could run `run …` (DR-298).
|
||
- Every handle is hardened before its first load: `tls-verify=yes` (mpv's
|
||
default is *no*, and its URLs carry the `ApiKey`) and `ytdl=no` (DR-299).
|
||
|
||
## MpvBackend (Linux)
|
||
|
||
**Location**: `src-tauri/src/player/mpv/`
|
||
|
||
The MPV backend uses libmpv for audio **and video** playback on Linux and Windows (video: see *Native video on the desktop* above). Since MPV handles are not `Send`, all operations occur on a dedicated thread.
|
||
|
||
```mermaid
|
||
flowchart TB
|
||
subgraph MainThread["Main Thread"]
|
||
MpvBackend["MpvBackend<br/>- command_tx<br/>- shared_state<br/>- shutdown"]
|
||
Commands["Commands:<br/>Load, Play, Pause<br/>Stop, Seek, SetVolume"]
|
||
end
|
||
|
||
subgraph EventLoopThread["MPV Event Loop Thread"]
|
||
EventLoop["event_loop.rs<br/>- MPV Handle<br/>- command_rx<br/>- Event Emitter"]
|
||
TauriEmitter["TauriEventEmitter"]
|
||
end
|
||
|
||
MpvBackend -->|"MpvCommand"| EventLoop
|
||
MpvBackend <-->|"Arc<Mutex<>>"| EventLoop
|
||
EventLoop -->|"Events"| TauriEmitter
|
||
TauriEmitter --> FrontendStore["Frontend Store"]
|
||
```
|
||
|
||
**Key Components:**
|
||
|
||
```rust
|
||
// Command enum sent to event loop thread
|
||
pub enum MpvCommand {
|
||
Load { url: String, media: MediaItem },
|
||
Play,
|
||
Pause,
|
||
Stop,
|
||
Seek(f64),
|
||
SetVolume(f32),
|
||
Quit,
|
||
}
|
||
|
||
// Shared state between main thread and event loop
|
||
pub struct MpvSharedState {
|
||
pub state: PlayerState,
|
||
pub position: f64,
|
||
pub duration: Option<f64>,
|
||
pub volume: f32,
|
||
pub is_loaded: bool,
|
||
pub current_media: Option<MediaItem>,
|
||
}
|
||
```
|
||
|
||
**Event Loop** (`event_loop.rs`):
|
||
- Initializes MPV with audio-only config (`vo=null`, `video=false`)
|
||
- Observes properties: `time-pos`, `duration`, `pause`, `volume`
|
||
- Emits position updates every 250ms during playback
|
||
- Processes commands from channel (non-blocking)
|
||
- Handles MPV events: `FileLoaded`, `EndFile`, `PropertyChange`
|
||
|
||
## ExoPlayerBackend (Android)
|
||
|
||
**Location**: `src-tauri/src/player/android/` and Kotlin sources
|
||
|
||
The ExoPlayer backend uses Android's Media3/ExoPlayer library via JNI.
|
||
|
||
```mermaid
|
||
flowchart TB
|
||
subgraph RustNative["Rust (Native)"]
|
||
ExoBackend["ExoPlayerBackend<br/>- player_ref<br/>- shared_state"]
|
||
NativeFuncs["JNI Callbacks<br/>nativeOnPosition...<br/>nativeOnState...<br/>nativeOnMediaLoaded<br/>nativeOnPlaybackEnd"]
|
||
TauriEmitter2["TauriEventEmitter"]
|
||
end
|
||
|
||
subgraph KotlinJVM["Kotlin (JVM)"]
|
||
JellyTauPlayer["JellyTauPlayer<br/>- ExoPlayer<br/>- Player.Listener"]
|
||
end
|
||
|
||
ExoBackend -->|"JNI Calls"| JellyTauPlayer
|
||
JellyTauPlayer -->|"Callbacks"| NativeFuncs
|
||
NativeFuncs --> TauriEmitter2
|
||
TauriEmitter2 --> FrontendStore2["Frontend Store"]
|
||
```
|
||
|
||
**Kotlin Player** (`JellyTauPlayer.kt`):
|
||
```kotlin
|
||
class JellyTauPlayer(context: Context) {
|
||
private val exoPlayer: ExoPlayer
|
||
private var positionUpdateJob: Job?
|
||
|
||
// Methods callable from Rust via JNI
|
||
fun load(url: String, mediaId: String)
|
||
fun play()
|
||
fun pause()
|
||
fun stop()
|
||
fun seek(positionSeconds: Double)
|
||
fun setVolume(volume: Float)
|
||
|
||
// Native callbacks to Rust
|
||
private external fun nativeOnPositionUpdate(position: Double, duration: Double)
|
||
private external fun nativeOnStateChanged(state: String, mediaId: String?)
|
||
private external fun nativeOnMediaLoaded(duration: Double)
|
||
private external fun nativeOnPlaybackEnded()
|
||
}
|
||
```
|
||
|
||
**JNI Callbacks** (Rust):
|
||
```rust
|
||
#[no_mangle]
|
||
pub extern "system" fn Java_com_dtourolle_jellytau_player_JellyTauPlayer_nativeOnPositionUpdate(
|
||
_env: JNIEnv, _class: JClass, position: jdouble, duration: jdouble
|
||
) {
|
||
// Update shared state
|
||
// Emit PlayerStatusEvent::PositionUpdate
|
||
}
|
||
```
|
||
|
||
### Audio settings on ExoPlayer
|
||
|
||
**TRACES**: UR-027, UR-032, UR-033 | DR-030, DR-035, DR-036
|
||
|
||
`PlayerBackend` declares `set_audio_settings` with a default `Ok(())` body. For a
|
||
long time `ExoPlayerBackend` took that default, so Settings › Audio rendered
|
||
controls that silently did nothing on Android — the parity gap recorded in
|
||
[requirements.md](../requirements.md#platform-playback-backend-parity-linux-vs-android),
|
||
now closed.
|
||
|
||
The settings cross to Kotlin as **JSON over JNI**, not as a wide signature, so new
|
||
fields do not change the method signature — the same approach `load()` uses for
|
||
subtitles:
|
||
|
||
```rust
|
||
fn set_audio_settings(&mut self, settings: &AudioSettings) -> Result<(), PlayerError> {
|
||
let json = audio_settings_jni_payload(settings)?;
|
||
env.call_method(&self.player_ref, "setAudioSettings", "(Ljava/lang/String;)V", …)?;
|
||
// Store the sanitised form, so audio_settings() reflects what was applied.
|
||
self.shared_state.lock_safe().audio_settings =
|
||
settings.clone().with_crossfade_clamped().with_equalizer_normalised();
|
||
}
|
||
```
|
||
|
||
Kotlin owns the *mechanics* — attaching `AudioEffect`s to the audio session — while
|
||
the canonical band layout and preset curves stay in Rust:
|
||
|
||
| Feature | Android mechanism | Notes |
|
||
|---------|-------------------|-------|
|
||
| Gapless | `pauseAtEndOfMediaItems` | |
|
||
| Volume normalization | `LoudnessEnhancer` | A gain stage — approximate next to MPV's `dynaudnorm` |
|
||
| Equalizer | `android.media.audiofx.Equalizer` | The canonical 10 bands are resampled onto the device's own band centres |
|
||
| Crossfade | — | Unimplemented on **every** platform (DR-034), architecturally blocked on MPV. Building it on Android alone would invert the parity gap |
|
||
|
||
Two things are deliberately still open: the effects are **not yet verified on a
|
||
physical device** (`AudioEffect` availability and band layouts are device-specific),
|
||
and the trait default is still a silent `Ok(())` rather than an error, so a backend
|
||
that omits the method still reports success. Flipping that default waits on the
|
||
device verification.
|
||
|
||
### Licensed audio codecs: the FFmpeg extension
|
||
|
||
**TRACES**: UR-004, UR-071 | DR-293
|
||
|
||
Android does not ship AC-3, E-AC-3, DTS or TrueHD decoders — they are licensed
|
||
codecs, present only where a vendor paid for them. The ROD2-W09 test tablet has a
|
||
vendor DTS decoder and no AC-3/E-AC-3 at all. ExoPlayer has no decoders of its
|
||
own, so on such a device those tracks are undecodable, and before this every film
|
||
with Dolby audio was re-encoded by the server — for streaming *and* for download.
|
||
|
||
`JellyTauPlayer` builds ExoPlayer with `DefaultRenderersFactory` in
|
||
`EXTENSION_RENDERER_MODE_ON`: the platform's decoders are tried first (a vendor DTS
|
||
decoder stays in charge where there is one) and the FFmpeg audio renderer takes
|
||
what they cannot decode. `CodecDetector` reports the extension's codecs beside the
|
||
`MediaCodecList` ones, asking `FfmpegLibrary.supportsFormat` per MIME type rather
|
||
than assuming, so a build whose native library failed to load reports only what
|
||
the platform decodes. Rust's device profile and download policy read that list,
|
||
which is what keeps "what we tell the server" and "what actually decodes" in step.
|
||
|
||
The decoder is `org.jellyfin.media3:media3-ffmpeg-decoder` — Jellyfin's build of
|
||
media3's FFmpeg extension, versioned `<media3 version>+N`. **Bump it in the same
|
||
commit as media3.** It is GPL-3.0: the distributed APK carries those terms, the
|
||
source stays MIT (see `THIRD_PARTY_NOTICES.md`). Its JNI methods are covered by the
|
||
AAR's own consumer rules and by `-keep class androidx.media3.** { *; }` in
|
||
`proguard-jellytau.pro`, which also keeps the renderer ExoPlayer loads reflectively.
|
||
|
||
**Rejected:** re-encoding a download's audio on the device after it lands (a
|
||
remux). It costs minutes of CPU and twice the disk per film, needs a pipeline
|
||
state of its own, and does nothing for streaming. Decoding at playback fixes both
|
||
paths with no extra step.
|
||
|
||
### Why the webview could not stay a fallback on Android
|
||
|
||
The webview decodes none of the codecs above — so with the original file now
|
||
downloaded as-is (DR-293), the old `experimentalNativeVideo` switch would have
|
||
played every such download as a silent film. That is what first removed the
|
||
webview video path on Android; DR-235 then removed it everywhere.
|
||
|
||
### The equalizer, and where its vocabulary lives
|
||
|
||
**TRACES**: UR-027 | DR-030, IR-020
|
||
|
||
The canonical band layout (`EQ_BANDS`) and the preset curves live in
|
||
`settings.rs`, **not** in either backend and not in the UI: a preset *is* a gain
|
||
curve defined by the band layout, and the layout is a property of the audio
|
||
engine rather than of the picker that renders it. Presets are Flat, Rock, Pop,
|
||
Jazz, Classical, Bass Boost, Treble Boost and Vocal, all conservative (within
|
||
±8 dB) so they stack safely with volume normalization.
|
||
|
||
| Platform | Mechanism |
|
||
|----------|-----------|
|
||
| Linux | One ffmpeg two-pole peaking `equalizer` filter per band, composed by `build_af_filter` into MPV's `af` property alongside the normalization filter: `equalizer=f=31:width_type=o:width=1:g=5` |
|
||
| Android | `android.media.audiofx.Equalizer`, with the canonical 10 bands **resampled onto whatever band centres the device actually has** |
|
||
|
||
Gains are normalised (`with_equalizer_normalised`) before use, and bands beyond
|
||
`EQ_BANDS` are ignored, so a malformed settings payload cannot produce a filter
|
||
chain of unbounded length.
|
||
|
||
## Background Audio Handoff (Android)
|
||
|
||
**TRACES**: UR-040 | IR-025, DR-051, DR-052, DR-178 … DR-180, DR-196, DR-203
|
||
|
||
Keeping a video's **audio** alive when the app is backgrounded or the screen
|
||
locks, while video decode stops. Two verified facts drive the whole design:
|
||
|
||
1. Nothing in the WebView keeps playing once the app is backgrounded — the
|
||
system throttles it.
|
||
2. Keeping audio alive in the background requires a **native foreground media
|
||
service**, which already exists for music (`JellyTauPlaybackService` +
|
||
`JellyTauPlayer` + `MediaSessionCompat`).
|
||
|
||
So this is a **handoff**, not "keep the WebView alive": on background, tear down
|
||
the current renderer and play the same item audio-only through the native
|
||
service; on foreground, hand back. In the project's one-directional playback
|
||
model this is a change of *which player is authoritative*, and the position must
|
||
transfer cleanly across it.
|
||
|
||
```mermaid
|
||
sequenceDiagram
|
||
participant App as App backgrounded
|
||
participant FE as VideoPlayer
|
||
participant Rust as player_enter/exit_background_audio
|
||
participant Exo as Native audio service
|
||
|
||
App-->>FE: jellytau-background (DOM CustomEvent)
|
||
FE->>Rust: enter(item, position, audioStreamIndex)
|
||
Rust->>Exo: play audio-only at position
|
||
Note over Exo: lockscreen + notification, existing MediaSession
|
||
App-->>FE: jellytau-foreground
|
||
FE->>Rust: exit() -> final position
|
||
Rust-->>FE: position
|
||
FE->>FE: restart the renderer that is on screen
|
||
```
|
||
|
||
Details that were each a shipped defect:
|
||
|
||
- **Position is absolute** — the position on the item's timeline the player
|
||
reports, never an offset within a re-opened transcode (`computeHandoffPosition`).
|
||
- **A downloaded episode takes no base URL and an ordinary seek** (DR-180); a
|
||
stream takes the base and no seek; a handoff at 0:00 takes neither.
|
||
- **The return must re-issue the load** (DR-196). The native player owns no
|
||
element and nothing watches the stream URL for it, so reassigning the URL — how
|
||
the deleted webview path came back — restarted nothing and left a black screen
|
||
with a play button that did nothing.
|
||
- **`wasPlaying` is captured on the way out** so play/pause survives the round
|
||
trip, and the handoff does not silently rewind (DR-203).
|
||
- **Mutually exclusive with PiP.** Toggle on → `setAutoEnterEnabled(false)`;
|
||
toggle off → PiP on background, the status quo. The frontend re-asserts the
|
||
value whenever the toggle changes and on unmount, so a stale setting cannot
|
||
leak into the next player.
|
||
- The pure arithmetic and state transitions live in
|
||
`backgroundAudioHandoff.ts`, free of Svelte and the DOM, so they are testable
|
||
without mounting the player.
|
||
|
||
Native signals background/foreground to the frontend as DOM CustomEvents
|
||
(`jellytau-background` / `jellytau-foreground`); the frontend carries the toggle
|
||
state to native through the `AndroidBackgroundAudio` bridge. No-op on every
|
||
non-Android platform.
|
||
|
||
## Native Video Compositing (Android)
|
||
|
||
**TRACES**: UR-003, UR-004 | DR-150 … DR-152, DR-182 … DR-196
|
||
|
||
Android can render video on the **native ExoPlayer surface behind a transparent
|
||
Tauri WebView**, with the Svelte controls drawn over it. It is the only video
|
||
path (DR-235). Before that it was an opt-in whose default was flipped and
|
||
reverted twice, each revert with a named cause — the per-defect record is in
|
||
`requirements.md` (DR-150 … DR-196).
|
||
|
||
```mermaid
|
||
flowchart TB
|
||
subgraph Window["One Android window"]
|
||
Texture["TextureView (index 0)<br/>ExoPlayer video"]
|
||
WebView["Tauri WebView (above)<br/>transparent, Svelte controls"]
|
||
end
|
||
Rust["ExoPlayerBackend"] -->|JNI| Player["JellyTauPlayer"]
|
||
Player --> Texture
|
||
MainActivity -->|"setTransparent(true)"| WebView
|
||
VideoOverlayManager -->|"attach / detach"| Texture
|
||
```
|
||
|
||
Load-bearing details, each of which was a shipped defect:
|
||
|
||
- **TextureView, not SurfaceView** (DR-192). A SurfaceView renders on its own
|
||
layer *outside* the app window and punches a transparent hole through it;
|
||
everything drawn above that hole — for us the whole UI — depends on that
|
||
composition path, which Android's own documentation says does not reliably
|
||
work. A TextureView makes "behind" ordinary view z-order within one window.
|
||
- **Attached at index 0** by `VideoOverlayManager`, and **detached when the video
|
||
goes** (DR-184) — a surface left in the hierarchy outlives its player.
|
||
- **Bridges are installed before the page that uses them** (DR-183).
|
||
`addJavascriptInterface` must run once per WebView instance and a call that
|
||
lands after the page has loaded never reaches it, so `setTransparent(true)`
|
||
could be dropped entirely.
|
||
- **The app shell stops painting over the surface** (DR-185). `app.css` clears
|
||
its opaque backgrounds off `[data-native-video]`; before that, a CSS rule
|
||
targeted an attribute nothing ever set, so the fix looked applied and was not.
|
||
- **The poster card lifts without a `<video>` element** (DR-182) — the
|
||
native reveal fires on a `playing` state or a position tick carrying a position
|
||
or duration, and on nothing else.
|
||
- **Letterbox bars are painted**, not left holding whatever was last in the
|
||
framebuffer (DR-194).
|
||
- There is deliberately **no audio-focus bridge**: manual focus requests from the
|
||
WebView competed with Chromium's `AudioFocusDelegate` and with ExoPlayer, and
|
||
the resulting `AUDIOFOCUS_LOSS` paused playback.
|
||
|
||
Related Kotlin pieces in the same window: `PictureInPictureManager` (DR-160/161),
|
||
`ScreenWakeManager` (DR-202 — Android counts its display timeout from touch
|
||
events, which a playing video does not generate), `ImmersiveModeBridge` and
|
||
`WindowInsetsBridge` (IR-031/DR-112 — see
|
||
[02-svelte-frontend.md](02-svelte-frontend.md#safe-area-insets)).
|
||
|
||
## Android MediaSession & Remote Volume Control
|
||
|
||
**Location**: `JellyTauPlaybackService.kt`
|
||
|
||
JellyTau uses a dual MediaSession architecture for Android to support both Media3 playback controls and remote volume control:
|
||
|
||
```mermaid
|
||
flowchart TB
|
||
subgraph Service["JellyTauPlaybackService"]
|
||
MediaSession["Media3 MediaSession<br/>- Lockscreen controls<br/>- Media notifications<br/>- Play/Pause/Next/Previous"]
|
||
|
||
MediaSessionCompat["MediaSessionCompat<br/>- Remote volume control<br/>- Hardware button interception"]
|
||
|
||
VolumeProvider["VolumeProviderCompat<br/>- onSetVolumeTo()<br/>- onAdjustVolume()"]
|
||
|
||
MediaSessionCompat --> VolumeProvider
|
||
end
|
||
|
||
subgraph Hardware["System"]
|
||
VolumeButtons["Hardware Volume Buttons"]
|
||
Lockscreen["Lockscreen Controls"]
|
||
Notification["Media Notification"]
|
||
end
|
||
|
||
subgraph Rust["Rust Backend"]
|
||
JNI["JNI Callbacks<br/>nativeOnRemoteVolumeChange()"]
|
||
PlaybackMode["PlaybackModeManager<br/>send_remote_volume_command()"]
|
||
JellyfinAPI["Jellyfin API<br/>session_set_volume()"]
|
||
end
|
||
|
||
VolumeButtons --> VolumeProvider
|
||
Lockscreen --> MediaSession
|
||
Notification --> MediaSession
|
||
|
||
VolumeProvider --> JNI
|
||
JNI --> PlaybackMode
|
||
PlaybackMode --> JellyfinAPI
|
||
```
|
||
|
||
**Architecture Rationale:**
|
||
|
||
JellyTau maintains both MediaSession types because they serve different purposes:
|
||
|
||
1. **Media3 MediaSession**: Handles lockscreen/notification playback controls (play/pause/next/previous)
|
||
2. **MediaSessionCompat**: Intercepts hardware volume button presses for remote playback control
|
||
|
||
When in remote playback mode (controlling a Jellyfin session on another device):
|
||
- Volume buttons are routed through `VolumeProviderCompat`
|
||
- Volume changes are sent to the remote session via Jellyfin API
|
||
- System volume UI shows the remote session's volume level
|
||
|
||
**Remote Volume Flow:**
|
||
|
||
```mermaid
|
||
sequenceDiagram
|
||
participant User
|
||
participant VolumeButton as Hardware Volume Button
|
||
participant VolumeProvider as VolumeProviderCompat
|
||
participant JNI as nativeOnRemoteVolumeChange
|
||
participant PlaybackMode as PlaybackModeManager
|
||
participant Jellyfin as Jellyfin Server
|
||
participant RemoteSession as Remote Session (TV/Browser)
|
||
|
||
User->>VolumeButton: Press Volume Up
|
||
VolumeButton->>VolumeProvider: onAdjustVolume(ADJUST_RAISE)
|
||
VolumeProvider->>VolumeProvider: remoteVolumeLevel += 2
|
||
VolumeProvider->>VolumeProvider: currentVolume = remoteVolumeLevel
|
||
VolumeProvider->>JNI: nativeOnRemoteVolumeChange("VolumeUp", level)
|
||
JNI->>PlaybackMode: send_remote_volume_command("VolumeUp", level)
|
||
PlaybackMode->>Jellyfin: POST /Sessions/{id}/Command/VolumeUp
|
||
Jellyfin->>RemoteSession: Set volume to new level
|
||
RemoteSession-->>User: Volume changes on TV/Browser
|
||
```
|
||
|
||
**Key Implementation Details:**
|
||
|
||
**Enabling Remote Volume** (`enableRemoteVolume()`):
|
||
```kotlin
|
||
fun enableRemoteVolume(initialVolume: Int) {
|
||
volumeProvider = object : VolumeProviderCompat(
|
||
VolumeProviderCompat.VOLUME_CONTROL_ABSOLUTE,
|
||
100, // Max volume
|
||
initialVolume
|
||
) {
|
||
override fun onSetVolumeTo(volume: Int) {
|
||
remoteVolumeLevel = volume.coerceIn(0, 100)
|
||
nativeOnRemoteVolumeChange("SetVolume", remoteVolumeLevel)
|
||
}
|
||
|
||
override fun onAdjustVolume(direction: Int) {
|
||
when (direction) {
|
||
AudioManager.ADJUST_RAISE -> {
|
||
remoteVolumeLevel = (remoteVolumeLevel + 2).coerceAtMost(100)
|
||
nativeOnRemoteVolumeChange("VolumeUp", remoteVolumeLevel)
|
||
currentVolume = remoteVolumeLevel
|
||
}
|
||
AudioManager.ADJUST_LOWER -> {
|
||
remoteVolumeLevel = (remoteVolumeLevel - 2).coerceAtLeast(0)
|
||
nativeOnRemoteVolumeChange("VolumeDown", remoteVolumeLevel)
|
||
currentVolume = remoteVolumeLevel
|
||
}
|
||
}
|
||
}
|
||
}
|
||
|
||
mediaSessionCompat.setPlaybackToRemote(volumeProvider)
|
||
}
|
||
```
|
||
|
||
**Disabling Remote Volume** (`disableRemoteVolume()`):
|
||
```kotlin
|
||
fun disableRemoteVolume() {
|
||
mediaSessionCompat.setPlaybackToLocal(AudioManager.STREAM_MUSIC)
|
||
volumeProvider = null
|
||
}
|
||
```
|
||
|
||
**Rust Integration** (`src-tauri/src/player/android/mod.rs`):
|
||
```rust
|
||
/// Enable remote volume control on Android
|
||
pub fn enable_remote_volume(initial_volume: i32) -> Result<(), String> {
|
||
start_playback_service()?;
|
||
let service_instance = get_playback_service_instance()?;
|
||
env.call_method(&service_instance, "enableRemoteVolume", "(I)V",
|
||
&[JValue::Int(initial_volume)])?;
|
||
Ok(())
|
||
}
|
||
```
|
||
|
||
**Dependencies** (`src-tauri/android/build.gradle.kts`):
|
||
```kotlin
|
||
dependencies {
|
||
implementation("androidx.media3:media3-session:1.5.1") // Media3 MediaSession
|
||
implementation("androidx.media:media:1.7.0") // MediaSessionCompat
|
||
}
|
||
```
|
||
|
||
**Integration with Playback Mode:**
|
||
|
||
Remote volume is automatically enabled/disabled during playback mode transfers:
|
||
|
||
```rust
|
||
// In PlaybackModeManager::transfer_to_remote()
|
||
#[cfg(target_os = "android")]
|
||
{
|
||
if let Err(e) = crate::player::enable_remote_volume(50) {
|
||
log::warn!("Failed to enable remote volume: {}", e);
|
||
}
|
||
}
|
||
|
||
// In PlaybackModeManager::transfer_to_local()
|
||
#[cfg(target_os = "android")]
|
||
{
|
||
if let Err(e) = crate::player::disable_remote_volume() {
|
||
log::warn!("Failed to disable remote volume: {}", e);
|
||
}
|
||
}
|
||
```
|
||
|
||
## Android Album Art Caching
|
||
|
||
**Location**: `src-tauri/android/src/main/java/com/dtourolle/jellytau/player/AlbumArtCache.kt`
|
||
|
||
Album art caching provides efficient bitmap storage for lock screen notifications with automatic LRU eviction and memory management.
|
||
|
||
```mermaid
|
||
flowchart TB
|
||
subgraph JellyTauPlayer["JellyTauPlayer.kt"]
|
||
LoadMedia["loadWithMetadata()<br/>- Store artworkUrl<br/>- Launch async download"]
|
||
AsyncDownload["Coroutine<br/>- Non-blocking<br/>- Dispatchers.IO"]
|
||
end
|
||
|
||
subgraph Cache["AlbumArtCache.kt"]
|
||
MemoryCache["LruCache<String, Bitmap><br/>- 1/8 of heap<br/>- ~12-16MB typical<br/>- 50-100 albums capacity"]
|
||
Download["Download & Scale<br/>- 512x512 max<br/>- Exponential backoff"]
|
||
ErrorHandle["Error Handling<br/>- Graceful fallback<br/>- Auto-retry"]
|
||
end
|
||
|
||
subgraph Service["JellyTauPlaybackService.kt"]
|
||
UpdateMeta["updateMediaMetadata()<br/>- Accept Bitmap parameter<br/>- Add METADATA_KEY_ALBUM_ART"]
|
||
Notification["Notification<br/>- setLargeIcon()<br/>- Lock screen display"]
|
||
end
|
||
|
||
LoadMedia --> AsyncDownload
|
||
AsyncDownload --> MemoryCache
|
||
MemoryCache --> Download
|
||
Download --> ErrorHandle
|
||
AsyncDownload --> UpdateMeta
|
||
UpdateMeta --> Notification
|
||
```
|
||
|
||
**AlbumArtCache Singleton:**
|
||
|
||
```kotlin
|
||
class AlbumArtCache(context: Context) {
|
||
private val memoryCache = object : LruCache<String, Bitmap>(cacheSize) {
|
||
override fun sizeOf(key: String, bitmap: Bitmap): Int {
|
||
return bitmap.byteCount / 1024 // Size in KB
|
||
}
|
||
}
|
||
|
||
suspend fun getArtwork(url: String): Bitmap? {
|
||
memoryCache.get(url)?.let { return it }
|
||
return downloadAndCache(url)
|
||
}
|
||
|
||
private suspend fun downloadAndCache(url: String): Bitmap? =
|
||
withContext(Dispatchers.IO) {
|
||
// HTTP download with 5s timeout
|
||
// Scale to 512x512 max
|
||
// Auto-evict LRU if needed
|
||
}
|
||
}
|
||
```
|
||
|
||
**Integration Flow:**
|
||
|
||
1. **Track Load** (`loadWithMetadata()`):
|
||
- Store artwork URL in `currentArtworkUrl`
|
||
- Reset bitmap to null
|
||
- Start playback immediately (non-blocking)
|
||
|
||
2. **Async Download** (Background Coroutine):
|
||
- Check cache: instant hit if available
|
||
- Network miss: download, scale, cache
|
||
- Auto-retry on network failure with exponential backoff
|
||
- Graceful fallback if artwork unavailable
|
||
|
||
3. **Notification Update**:
|
||
- Pass bitmap to `updatePlaybackServiceNotification()`
|
||
- Add to `MediaMetadataCompat` with `METADATA_KEY_ALBUM_ART`
|
||
- Display as large icon in notification
|
||
- Show on lock screen
|
||
|
||
**Memory Management:**
|
||
|
||
| Metric | Value |
|
||
|--------|-------|
|
||
| Cache Size | 1/8 of heap (12-16MB typical) |
|
||
| Max Resolution | 512x512 pixels |
|
||
| Capacity | ~50-100 album arts |
|
||
| Eviction Policy | LRU (Least Recently Used) |
|
||
| Lifetime | In-memory only (app session) |
|
||
| Network Timeout | 5 seconds per download |
|
||
|
||
**Performance Characteristics:**
|
||
|
||
- **Cache Hit**: ~1ms (in-memory retrieval)
|
||
- **Cache Miss**: ~200-500ms (download + scale)
|
||
- **Playback Impact**: Zero (async downloads)
|
||
- **Memory Overhead**: Max 16MB (auto-eviction)
|
||
- **Error Recovery**: Automatic with exponential backoff
|
||
|
||
## Backend Initialization
|
||
|
||
**Location**: `src-tauri/src/lib.rs`
|
||
|
||
Backend selection is platform-specific:
|
||
|
||
```rust
|
||
fn create_player_backend(app_handle: tauri::AppHandle) -> Box<dyn PlayerBackend> {
|
||
let event_emitter = Arc::new(TauriEventEmitter::new(app_handle));
|
||
|
||
#[cfg(target_os = "linux")]
|
||
{
|
||
match MpvBackend::new(event_emitter.clone()) {
|
||
Ok(backend) => return Box::new(backend),
|
||
Err(e) => eprintln!("MPV init failed: {}", e),
|
||
}
|
||
}
|
||
|
||
#[cfg(target_os = "android")]
|
||
{
|
||
// ExoPlayer requires Activity context, initialized separately
|
||
}
|
||
|
||
// Fallback
|
||
Box::new(NullBackend::new())
|
||
}
|
||
```
|