Security (DR-298, DR-299): - The pinned libmpv crate's Mpv::command joins its arguments and calls mpv_command_string, which parses `;` as a command separator. Stream URLs carry server-controlled ids and TranscodingUrl, and a download's file:// path carries its track title, so a crafted title could run any mpv command, `run` included. Every call now goes through mpv_command::command, an argv built for mpv_command. The same parse broke loadfile for every downloaded title containing a space. - mpv's tls-verify defaults to no, and its URLs carry the ApiKey. Every handle is now hardened with tls-verify=yes and ytdl=no before its first loadfile, and fails construction if it cannot be. Linux video (DR-235 phase 1): - native_video::enabled() is unconditional on Linux; the JELLYTAU_NATIVE_VIDEO opt-in is retired. No platform reports a webview video fallback, so the Settings switch no longer appears. Windows keeps the webview element until mpv reaches it (DR-237). - The Linux device profile is unchanged (still h264, DR-234), so this ships the configuration that was tested under the env var.
754 lines
29 KiB
Markdown
754 lines
29 KiB
Markdown
# Platform-Specific Player Backends
|
||
|
||
## Player Events System
|
||
|
||
**Location**: `src-tauri/src/player/events.rs`
|
||
|
||
The player uses a push-based event system to notify the frontend of state changes:
|
||
|
||
```rust
|
||
pub enum PlayerStatusEvent {
|
||
/// Playback position updated (emitted periodically during playback)
|
||
PositionUpdate { position: f64, duration: f64 },
|
||
|
||
/// Player state changed
|
||
StateChanged { state: String, media_id: Option<String> },
|
||
|
||
/// Media has finished loading and is ready to play
|
||
MediaLoaded { duration: f64 },
|
||
|
||
/// Playback has ended naturally
|
||
PlaybackEnded,
|
||
|
||
/// Buffering state changed
|
||
Buffering { percent: u8 },
|
||
|
||
/// An error occurred during playback
|
||
Error { message: String, recoverable: bool },
|
||
|
||
/// Volume changed
|
||
VolumeChanged { volume: f32, muted: bool },
|
||
|
||
/// Sleep timer state changed
|
||
SleepTimerChanged {
|
||
mode: SleepTimerMode,
|
||
remaining_seconds: u32,
|
||
},
|
||
|
||
/// Show next episode popup with countdown
|
||
ShowNextEpisodePopup {
|
||
current_episode: MediaItem,
|
||
next_episode: MediaItem,
|
||
countdown_seconds: u32,
|
||
auto_advance: bool,
|
||
},
|
||
|
||
/// Countdown tick (emitted every second during autoplay countdown)
|
||
CountdownTick { remaining_seconds: u32 },
|
||
|
||
/// Queue changed (items added, removed, reordered, or playback mode changed)
|
||
QueueChanged {
|
||
items: Vec<MediaItem>,
|
||
current_index: Option<usize>,
|
||
shuffle: bool,
|
||
repeat: RepeatMode,
|
||
has_next: bool,
|
||
has_previous: bool,
|
||
},
|
||
|
||
/// Media session changed (activity context changed: Audio/Movie/TvShow/Idle)
|
||
SessionChanged { session: MediaSessionType },
|
||
}
|
||
```
|
||
|
||
Events are emitted via Tauri's event system:
|
||
|
||
```mermaid
|
||
flowchart LR
|
||
subgraph Backend["Player Backend"]
|
||
MPV["MPV/ExoPlayer"]
|
||
end
|
||
|
||
subgraph EventSystem["Event System"]
|
||
Emitter["TauriEventEmitter<br/>emit()"]
|
||
Bus["Tauri Event Bus<br/>'player-event'"]
|
||
end
|
||
|
||
subgraph Frontend["Frontend"]
|
||
Listener["playerEvents.ts<br/>Frontend Listener"]
|
||
Store["Player Store Update<br/>(position, state, etc)"]
|
||
end
|
||
|
||
MPV --> Emitter --> Bus --> Listener --> Store
|
||
```
|
||
|
||
**Frontend Listener** (`src/lib/services/playerEvents.ts`):
|
||
- Listens for `player-event` Tauri events
|
||
- Updates player/queue stores based on event type
|
||
- Auto-advances to next track on `PlaybackEnded`
|
||
- On `StateChanged` events, calls `invoke("player_get_queue")` to update `appState.hasNext`/`hasPrevious` -- this enables MiniPlayer skip button state
|
||
|
||
**Important**: The command is `player_get_queue` (returns `QueueStatus` with `hasNext`/`hasPrevious`). There is no `player_get_queue_status` command.
|
||
|
||
## HTML5 Video Adapter (webview-rendered video)
|
||
|
||
**Location**: `src/lib/player/html5Adapter.ts`, `src/lib/player/index.ts`, report commands in
|
||
`src-tauri/src/commands/player/timers.rs`
|
||
|
||
Video on **Windows** is rendered by an HTML5 `<video>`/HLS element **inside the webview**. Neither
|
||
other platform uses this path for video: Android draws in ExoPlayer (see *The webview is not a video
|
||
renderer on Android* below), and Linux draws in mpv beneath the webview (DR-235 phase 1 — see
|
||
[desktop-native-video.md](../specs/desktop-native-video.md) until it ships on Windows too). Where the
|
||
element is used, no native backend renders or observes it. The `<video>` is therefore
|
||
the real player, living outside Rust's reach.
|
||
|
||
To keep the `PlayerController` the single source of truth (matching the audio path), the HTML5 element
|
||
is treated as **a dumb output device that reports back into Rust**, rather than an independent state
|
||
authority:
|
||
|
||
```mermaid
|
||
flowchart LR
|
||
subgraph Webview["Webview"]
|
||
Video["HTML5 <video> / HLS.js"]
|
||
Adapter["html5Adapter.ts<br/>(reports DOM events)"]
|
||
end
|
||
subgraph Backend["Rust"]
|
||
Cmds["player_report_state<br/>player_report_position<br/>player_report_media_loaded"]
|
||
Controller["PlayerController"]
|
||
Emitter["TauriEventEmitter"]
|
||
end
|
||
subgraph Frontend["Frontend"]
|
||
Events["playerEvents.ts"]
|
||
Store["player store"]
|
||
end
|
||
|
||
Video -->|DOM events| Adapter --> Cmds --> Controller --> Emitter --> Events --> Store
|
||
```
|
||
|
||
**Key points:**
|
||
- The adapter re-emits the *same* `PlayerStatusEvent`s (`StateChanged`, `PositionUpdate`, `MediaLoaded`)
|
||
the native backends emit, so `playerEvents.ts` needs **no** HTML5-specific branch — HTML5 is just
|
||
another event source feeding the existing pipeline.
|
||
- Position reports are throttled (~250ms) to match the MPV cadence and avoid flooding IPC from the
|
||
60fps RAF loop.
|
||
- **Boundary rule**: UI components never touch the report commands or `videoElement` state directly.
|
||
Playback *control* goes through the unified facade `src/lib/player/index.ts` (`playerController`);
|
||
HTML5 *state reporting* goes through `html5Adapter.ts`. This restores the documented invariant
|
||
("frontend only displays state and invokes commands") for the video path.
|
||
|
||
## MpvBackend (Linux)
|
||
|
||
**Location**: `src-tauri/src/player/mpv/`
|
||
|
||
The MPV backend uses libmpv for audio playback on Linux. Since MPV handles are not `Send`, all operations occur on a dedicated thread.
|
||
|
||
```mermaid
|
||
flowchart TB
|
||
subgraph MainThread["Main Thread"]
|
||
MpvBackend["MpvBackend<br/>- command_tx<br/>- shared_state<br/>- shutdown"]
|
||
Commands["Commands:<br/>Load, Play, Pause<br/>Stop, Seek, SetVolume"]
|
||
end
|
||
|
||
subgraph EventLoopThread["MPV Event Loop Thread"]
|
||
EventLoop["event_loop.rs<br/>- MPV Handle<br/>- command_rx<br/>- Event Emitter"]
|
||
TauriEmitter["TauriEventEmitter"]
|
||
end
|
||
|
||
MpvBackend -->|"MpvCommand"| EventLoop
|
||
MpvBackend <-->|"Arc<Mutex<>>"| EventLoop
|
||
EventLoop -->|"Events"| TauriEmitter
|
||
TauriEmitter --> FrontendStore["Frontend Store"]
|
||
```
|
||
|
||
**Key Components:**
|
||
|
||
```rust
|
||
// Command enum sent to event loop thread
|
||
pub enum MpvCommand {
|
||
Load { url: String, media: MediaItem },
|
||
Play,
|
||
Pause,
|
||
Stop,
|
||
Seek(f64),
|
||
SetVolume(f32),
|
||
Quit,
|
||
}
|
||
|
||
// Shared state between main thread and event loop
|
||
pub struct MpvSharedState {
|
||
pub state: PlayerState,
|
||
pub position: f64,
|
||
pub duration: Option<f64>,
|
||
pub volume: f32,
|
||
pub is_loaded: bool,
|
||
pub current_media: Option<MediaItem>,
|
||
}
|
||
```
|
||
|
||
**Event Loop** (`event_loop.rs`):
|
||
- Initializes MPV with audio-only config (`vo=null`, `video=false`)
|
||
- Observes properties: `time-pos`, `duration`, `pause`, `volume`
|
||
- Emits position updates every 250ms during playback
|
||
- Processes commands from channel (non-blocking)
|
||
- Handles MPV events: `FileLoaded`, `EndFile`, `PropertyChange`
|
||
|
||
## ExoPlayerBackend (Android)
|
||
|
||
**Location**: `src-tauri/src/player/android/` and Kotlin sources
|
||
|
||
The ExoPlayer backend uses Android's Media3/ExoPlayer library via JNI.
|
||
|
||
```mermaid
|
||
flowchart TB
|
||
subgraph RustNative["Rust (Native)"]
|
||
ExoBackend["ExoPlayerBackend<br/>- player_ref<br/>- shared_state"]
|
||
NativeFuncs["JNI Callbacks<br/>nativeOnPosition...<br/>nativeOnState...<br/>nativeOnMediaLoaded<br/>nativeOnPlaybackEnd"]
|
||
TauriEmitter2["TauriEventEmitter"]
|
||
end
|
||
|
||
subgraph KotlinJVM["Kotlin (JVM)"]
|
||
JellyTauPlayer["JellyTauPlayer<br/>- ExoPlayer<br/>- Player.Listener"]
|
||
end
|
||
|
||
ExoBackend -->|"JNI Calls"| JellyTauPlayer
|
||
JellyTauPlayer -->|"Callbacks"| NativeFuncs
|
||
NativeFuncs --> TauriEmitter2
|
||
TauriEmitter2 --> FrontendStore2["Frontend Store"]
|
||
```
|
||
|
||
**Kotlin Player** (`JellyTauPlayer.kt`):
|
||
```kotlin
|
||
class JellyTauPlayer(context: Context) {
|
||
private val exoPlayer: ExoPlayer
|
||
private var positionUpdateJob: Job?
|
||
|
||
// Methods callable from Rust via JNI
|
||
fun load(url: String, mediaId: String)
|
||
fun play()
|
||
fun pause()
|
||
fun stop()
|
||
fun seek(positionSeconds: Double)
|
||
fun setVolume(volume: Float)
|
||
|
||
// Native callbacks to Rust
|
||
private external fun nativeOnPositionUpdate(position: Double, duration: Double)
|
||
private external fun nativeOnStateChanged(state: String, mediaId: String?)
|
||
private external fun nativeOnMediaLoaded(duration: Double)
|
||
private external fun nativeOnPlaybackEnded()
|
||
}
|
||
```
|
||
|
||
**JNI Callbacks** (Rust):
|
||
```rust
|
||
#[no_mangle]
|
||
pub extern "system" fn Java_com_dtourolle_jellytau_player_JellyTauPlayer_nativeOnPositionUpdate(
|
||
_env: JNIEnv, _class: JClass, position: jdouble, duration: jdouble
|
||
) {
|
||
// Update shared state
|
||
// Emit PlayerStatusEvent::PositionUpdate
|
||
}
|
||
```
|
||
|
||
### Audio settings on ExoPlayer
|
||
|
||
**TRACES**: UR-027, UR-032, UR-033 | DR-030, DR-035, DR-036
|
||
|
||
`PlayerBackend` declares `set_audio_settings` with a default `Ok(())` body. For a
|
||
long time `ExoPlayerBackend` took that default, so Settings › Audio rendered
|
||
controls that silently did nothing on Android — the parity gap recorded in
|
||
[requirements.md](../requirements.md#platform-playback-backend-parity-linux-vs-android),
|
||
now closed.
|
||
|
||
The settings cross to Kotlin as **JSON over JNI**, not as a wide signature, so new
|
||
fields do not change the method signature — the same approach `load()` uses for
|
||
subtitles:
|
||
|
||
```rust
|
||
fn set_audio_settings(&mut self, settings: &AudioSettings) -> Result<(), PlayerError> {
|
||
let json = audio_settings_jni_payload(settings)?;
|
||
env.call_method(&self.player_ref, "setAudioSettings", "(Ljava/lang/String;)V", …)?;
|
||
// Store the sanitised form, so audio_settings() reflects what was applied.
|
||
self.shared_state.lock_safe().audio_settings =
|
||
settings.clone().with_crossfade_clamped().with_equalizer_normalised();
|
||
}
|
||
```
|
||
|
||
Kotlin owns the *mechanics* — attaching `AudioEffect`s to the audio session — while
|
||
the canonical band layout and preset curves stay in Rust:
|
||
|
||
| Feature | Android mechanism | Notes |
|
||
|---------|-------------------|-------|
|
||
| Gapless | `pauseAtEndOfMediaItems` | |
|
||
| Volume normalization | `LoudnessEnhancer` | A gain stage — approximate next to MPV's `dynaudnorm` |
|
||
| Equalizer | `android.media.audiofx.Equalizer` | The canonical 10 bands are resampled onto the device's own band centres |
|
||
| Crossfade | — | Unimplemented on **every** platform (DR-034), architecturally blocked on MPV. Building it on Android alone would invert the parity gap |
|
||
|
||
Two things are deliberately still open: the effects are **not yet verified on a
|
||
physical device** (`AudioEffect` availability and band layouts are device-specific),
|
||
and the trait default is still a silent `Ok(())` rather than an error, so a backend
|
||
that omits the method still reports success. Flipping that default waits on the
|
||
device verification.
|
||
|
||
### Licensed audio codecs: the FFmpeg extension
|
||
|
||
**TRACES**: UR-004, UR-071 | DR-293
|
||
|
||
Android does not ship AC-3, E-AC-3, DTS or TrueHD decoders — they are licensed
|
||
codecs, present only where a vendor paid for them. The ROD2-W09 test tablet has a
|
||
vendor DTS decoder and no AC-3/E-AC-3 at all. ExoPlayer has no decoders of its
|
||
own, so on such a device those tracks are undecodable, and before this every film
|
||
with Dolby audio was re-encoded by the server — for streaming *and* for download.
|
||
|
||
`JellyTauPlayer` builds ExoPlayer with `DefaultRenderersFactory` in
|
||
`EXTENSION_RENDERER_MODE_ON`: the platform's decoders are tried first (a vendor DTS
|
||
decoder stays in charge where there is one) and the FFmpeg audio renderer takes
|
||
what they cannot decode. `CodecDetector` reports the extension's codecs beside the
|
||
`MediaCodecList` ones, asking `FfmpegLibrary.supportsFormat` per MIME type rather
|
||
than assuming, so a build whose native library failed to load reports only what
|
||
the platform decodes. Rust's device profile and download policy read that list,
|
||
which is what keeps "what we tell the server" and "what actually decodes" in step.
|
||
|
||
The decoder is `org.jellyfin.media3:media3-ffmpeg-decoder` — Jellyfin's build of
|
||
media3's FFmpeg extension, versioned `<media3 version>+N`. **Bump it in the same
|
||
commit as media3.** It is GPL-3.0: the distributed APK carries those terms, the
|
||
source stays MIT (see `THIRD_PARTY_NOTICES.md`). Its JNI methods are covered by the
|
||
AAR's own consumer rules and by `-keep class androidx.media3.** { *; }` in
|
||
`proguard-jellytau.pro`, which also keeps the renderer ExoPlayer loads reflectively.
|
||
|
||
**Rejected:** re-encoding a download's audio on the device after it lands (a
|
||
remux). It costs minutes of CPU and twice the disk per film, needs a pipeline
|
||
state of its own, and does nothing for streaming. Decoding at playback fixes both
|
||
paths with no extra step.
|
||
|
||
### The webview is not a video renderer on Android
|
||
|
||
ExoPlayer is Android's only video renderer. The HTML5 path used to be reachable
|
||
through the `experimentalNativeVideo` setting (a *suppressor* of Rust's native
|
||
choice), but the webview decodes none of the codecs above — so with the original
|
||
file now downloaded as-is (DR-293), turning native video off would play every such
|
||
download as a silent film. Rust reports `webview_video_fallback` in
|
||
`PlaybackCapabilities`: **false on Android**, true only beside mpv native video on
|
||
Linux, where the webview is still the tested fallback. The frontend offers the
|
||
switch and honours a stored "off" only when it is true (`nativeVideoWanted` in
|
||
`stores/nativeVideo.ts`), so a user who once switched it off on Android is not
|
||
stranded on the silent path.
|
||
|
||
### The equalizer, and where its vocabulary lives
|
||
|
||
**TRACES**: UR-027 | DR-030, IR-020
|
||
|
||
The canonical band layout (`EQ_BANDS`) and the preset curves live in
|
||
`settings.rs`, **not** in either backend and not in the UI: a preset *is* a gain
|
||
curve defined by the band layout, and the layout is a property of the audio
|
||
engine rather than of the picker that renders it. Presets are Flat, Rock, Pop,
|
||
Jazz, Classical, Bass Boost, Treble Boost and Vocal, all conservative (within
|
||
±8 dB) so they stack safely with volume normalization.
|
||
|
||
| Platform | Mechanism |
|
||
|----------|-----------|
|
||
| Linux | One ffmpeg two-pole peaking `equalizer` filter per band, composed by `build_af_filter` into MPV's `af` property alongside the normalization filter: `equalizer=f=31:width_type=o:width=1:g=5` |
|
||
| Android | `android.media.audiofx.Equalizer`, with the canonical 10 bands **resampled onto whatever band centres the device actually has** |
|
||
|
||
Gains are normalised (`with_equalizer_normalised`) before use, and bands beyond
|
||
`EQ_BANDS` are ignored, so a malformed settings payload cannot produce a filter
|
||
chain of unbounded length.
|
||
|
||
## Background Audio Handoff (Android)
|
||
|
||
**TRACES**: UR-040 | IR-025, DR-051, DR-052, DR-178 … DR-180, DR-196, DR-203
|
||
|
||
Keeping a video's **audio** alive when the app is backgrounded or the screen
|
||
locks, while video decode stops. Two verified facts drive the whole design:
|
||
|
||
1. An Android WebView `<video>` **does not** keep playing audio once the app is
|
||
backgrounded — the system throttles the WebView and media pauses.
|
||
2. Keeping audio alive in the background requires a **native foreground media
|
||
service**, which already exists for music (`JellyTauPlaybackService` +
|
||
`JellyTauPlayer` + `MediaSessionCompat`).
|
||
|
||
So this is a **handoff**, not "keep the WebView alive": on background, tear down
|
||
the current renderer and play the same item audio-only through the native
|
||
service; on foreground, hand back. In the project's one-directional playback
|
||
model this is a change of *which player is authoritative*, and the position must
|
||
transfer cleanly across it.
|
||
|
||
```mermaid
|
||
sequenceDiagram
|
||
participant App as App backgrounded
|
||
participant FE as VideoPlayer
|
||
participant Rust as player_enter/exit_background_audio
|
||
participant Exo as Native audio service
|
||
|
||
App-->>FE: jellytau-background (DOM CustomEvent)
|
||
FE->>Rust: enter(item, position, audioStreamIndex)
|
||
Rust->>Exo: play audio-only at position
|
||
Note over Exo: lockscreen + notification, existing MediaSession
|
||
App-->>FE: jellytau-foreground
|
||
FE->>Rust: exit() -> final position
|
||
Rust-->>FE: position
|
||
FE->>FE: restart the renderer that is on screen
|
||
```
|
||
|
||
Details that were each a shipped defect:
|
||
|
||
- **Position is absolute.** Transcoded HLS tracks time as
|
||
`videoElement.currentTime + seekOffset` (the element resets to 0 after each
|
||
transcode reload). `computeHandoffPosition` sums both terms; using the element
|
||
time alone rewinds by the offset.
|
||
- **A downloaded episode takes no base URL and an ordinary seek** (DR-180); a
|
||
stream takes the base and no seek; a handoff at 0:00 takes neither.
|
||
- **The return must restart the renderer that is actually on screen** (DR-196).
|
||
The two paths resume by different means — the webview `<video>` reloads off its
|
||
stream URL, watched by an `$effect`; ExoPlayer owns no element and nothing
|
||
watches the URL for it, so it needs an explicit re-issue. Doing only the URL
|
||
assignment restarted nothing on the native path and left a black screen with a
|
||
play button that did nothing.
|
||
- **`wasPlaying` is captured on the way out** so play/pause survives the round
|
||
trip, and the handoff does not silently rewind (DR-203).
|
||
- **Mutually exclusive with PiP.** Toggle on → `setAutoEnterEnabled(false)`;
|
||
toggle off → PiP on background, the status quo. The frontend re-asserts the
|
||
value whenever the toggle changes and on unmount, so a stale setting cannot
|
||
leak into the next player.
|
||
- The pure arithmetic and state transitions live in
|
||
`backgroundAudioHandoff.ts`, free of Svelte and the DOM, so they are testable
|
||
without mounting the player.
|
||
|
||
Native signals background/foreground to the frontend as DOM CustomEvents
|
||
(`jellytau-background` / `jellytau-foreground`); the frontend carries the toggle
|
||
state to native through the `AndroidBackgroundAudio` bridge. No-op on every
|
||
non-Android platform.
|
||
|
||
## Native Video Compositing (Android)
|
||
|
||
**TRACES**: UR-003, UR-004 | DR-150 … DR-152, DR-182 … DR-196
|
||
|
||
Android can render video on the **native ExoPlayer surface behind a transparent
|
||
Tauri WebView**, with the Svelte controls drawn over it. This is on by default;
|
||
the HTML5 `<video>` path remains the fallback and is not being removed. The
|
||
default has been flipped and reverted twice and each revert has a named cause —
|
||
the per-defect record is in `requirements.md` (DR-150 … DR-196).
|
||
|
||
```mermaid
|
||
flowchart TB
|
||
subgraph Window["One Android window"]
|
||
Texture["TextureView (index 0)<br/>ExoPlayer video"]
|
||
WebView["Tauri WebView (above)<br/>transparent, Svelte controls"]
|
||
end
|
||
Rust["ExoPlayerBackend"] -->|JNI| Player["JellyTauPlayer"]
|
||
Player --> Texture
|
||
MainActivity -->|"setTransparent(true)"| WebView
|
||
VideoOverlayManager -->|"attach / detach"| Texture
|
||
```
|
||
|
||
Load-bearing details, each of which was a shipped defect:
|
||
|
||
- **TextureView, not SurfaceView** (DR-192). A SurfaceView renders on its own
|
||
layer *outside* the app window and punches a transparent hole through it;
|
||
everything drawn above that hole — for us the whole UI — depends on that
|
||
composition path, which Android's own documentation says does not reliably
|
||
work. A TextureView makes "behind" ordinary view z-order within one window.
|
||
- **Attached at index 0** by `VideoOverlayManager`, and **detached when the video
|
||
goes** (DR-184) — a surface left in the hierarchy outlives its player.
|
||
- **Bridges are installed before the page that uses them** (DR-183).
|
||
`addJavascriptInterface` must run once per WebView instance and a call that
|
||
lands after the page has loaded never reaches it, so `setTransparent(true)`
|
||
could be dropped entirely.
|
||
- **The app shell stops painting over the surface** (DR-185). `app.css` clears
|
||
its opaque backgrounds off `[data-native-video]`; before that, a CSS rule
|
||
targeted an attribute nothing ever set, so the fix looked applied and was not.
|
||
- **The poster card can lift on a path with no `<video>` element** (DR-182) — the
|
||
native reveal fires on a `playing` state or a position tick carrying a position
|
||
or duration, and on nothing else.
|
||
- **Letterbox bars are painted**, not left holding whatever was last in the
|
||
framebuffer (DR-194).
|
||
- There is deliberately **no audio-focus bridge**: manual focus requests from the
|
||
WebView competed with Chromium's `AudioFocusDelegate` and with ExoPlayer, and
|
||
the resulting `AUDIOFOCUS_LOSS` paused playback.
|
||
|
||
Related Kotlin pieces in the same window: `PictureInPictureManager` (DR-160/161),
|
||
`ScreenWakeManager` (DR-202 — Android counts its display timeout from touch
|
||
events, which a playing video does not generate), `ImmersiveModeBridge` and
|
||
`WindowInsetsBridge` (IR-031/DR-112 — see
|
||
[02-svelte-frontend.md](02-svelte-frontend.md#safe-area-insets)).
|
||
|
||
## Android MediaSession & Remote Volume Control
|
||
|
||
**Location**: `JellyTauPlaybackService.kt`
|
||
|
||
JellyTau uses a dual MediaSession architecture for Android to support both Media3 playback controls and remote volume control:
|
||
|
||
```mermaid
|
||
flowchart TB
|
||
subgraph Service["JellyTauPlaybackService"]
|
||
MediaSession["Media3 MediaSession<br/>- Lockscreen controls<br/>- Media notifications<br/>- Play/Pause/Next/Previous"]
|
||
|
||
MediaSessionCompat["MediaSessionCompat<br/>- Remote volume control<br/>- Hardware button interception"]
|
||
|
||
VolumeProvider["VolumeProviderCompat<br/>- onSetVolumeTo()<br/>- onAdjustVolume()"]
|
||
|
||
MediaSessionCompat --> VolumeProvider
|
||
end
|
||
|
||
subgraph Hardware["System"]
|
||
VolumeButtons["Hardware Volume Buttons"]
|
||
Lockscreen["Lockscreen Controls"]
|
||
Notification["Media Notification"]
|
||
end
|
||
|
||
subgraph Rust["Rust Backend"]
|
||
JNI["JNI Callbacks<br/>nativeOnRemoteVolumeChange()"]
|
||
PlaybackMode["PlaybackModeManager<br/>send_remote_volume_command()"]
|
||
JellyfinAPI["Jellyfin API<br/>session_set_volume()"]
|
||
end
|
||
|
||
VolumeButtons --> VolumeProvider
|
||
Lockscreen --> MediaSession
|
||
Notification --> MediaSession
|
||
|
||
VolumeProvider --> JNI
|
||
JNI --> PlaybackMode
|
||
PlaybackMode --> JellyfinAPI
|
||
```
|
||
|
||
**Architecture Rationale:**
|
||
|
||
JellyTau maintains both MediaSession types because they serve different purposes:
|
||
|
||
1. **Media3 MediaSession**: Handles lockscreen/notification playback controls (play/pause/next/previous)
|
||
2. **MediaSessionCompat**: Intercepts hardware volume button presses for remote playback control
|
||
|
||
When in remote playback mode (controlling a Jellyfin session on another device):
|
||
- Volume buttons are routed through `VolumeProviderCompat`
|
||
- Volume changes are sent to the remote session via Jellyfin API
|
||
- System volume UI shows the remote session's volume level
|
||
|
||
**Remote Volume Flow:**
|
||
|
||
```mermaid
|
||
sequenceDiagram
|
||
participant User
|
||
participant VolumeButton as Hardware Volume Button
|
||
participant VolumeProvider as VolumeProviderCompat
|
||
participant JNI as nativeOnRemoteVolumeChange
|
||
participant PlaybackMode as PlaybackModeManager
|
||
participant Jellyfin as Jellyfin Server
|
||
participant RemoteSession as Remote Session (TV/Browser)
|
||
|
||
User->>VolumeButton: Press Volume Up
|
||
VolumeButton->>VolumeProvider: onAdjustVolume(ADJUST_RAISE)
|
||
VolumeProvider->>VolumeProvider: remoteVolumeLevel += 2
|
||
VolumeProvider->>VolumeProvider: currentVolume = remoteVolumeLevel
|
||
VolumeProvider->>JNI: nativeOnRemoteVolumeChange("VolumeUp", level)
|
||
JNI->>PlaybackMode: send_remote_volume_command("VolumeUp", level)
|
||
PlaybackMode->>Jellyfin: POST /Sessions/{id}/Command/VolumeUp
|
||
Jellyfin->>RemoteSession: Set volume to new level
|
||
RemoteSession-->>User: Volume changes on TV/Browser
|
||
```
|
||
|
||
**Key Implementation Details:**
|
||
|
||
**Enabling Remote Volume** (`enableRemoteVolume()`):
|
||
```kotlin
|
||
fun enableRemoteVolume(initialVolume: Int) {
|
||
volumeProvider = object : VolumeProviderCompat(
|
||
VolumeProviderCompat.VOLUME_CONTROL_ABSOLUTE,
|
||
100, // Max volume
|
||
initialVolume
|
||
) {
|
||
override fun onSetVolumeTo(volume: Int) {
|
||
remoteVolumeLevel = volume.coerceIn(0, 100)
|
||
nativeOnRemoteVolumeChange("SetVolume", remoteVolumeLevel)
|
||
}
|
||
|
||
override fun onAdjustVolume(direction: Int) {
|
||
when (direction) {
|
||
AudioManager.ADJUST_RAISE -> {
|
||
remoteVolumeLevel = (remoteVolumeLevel + 2).coerceAtMost(100)
|
||
nativeOnRemoteVolumeChange("VolumeUp", remoteVolumeLevel)
|
||
currentVolume = remoteVolumeLevel
|
||
}
|
||
AudioManager.ADJUST_LOWER -> {
|
||
remoteVolumeLevel = (remoteVolumeLevel - 2).coerceAtLeast(0)
|
||
nativeOnRemoteVolumeChange("VolumeDown", remoteVolumeLevel)
|
||
currentVolume = remoteVolumeLevel
|
||
}
|
||
}
|
||
}
|
||
}
|
||
|
||
mediaSessionCompat.setPlaybackToRemote(volumeProvider)
|
||
}
|
||
```
|
||
|
||
**Disabling Remote Volume** (`disableRemoteVolume()`):
|
||
```kotlin
|
||
fun disableRemoteVolume() {
|
||
mediaSessionCompat.setPlaybackToLocal(AudioManager.STREAM_MUSIC)
|
||
volumeProvider = null
|
||
}
|
||
```
|
||
|
||
**Rust Integration** (`src-tauri/src/player/android/mod.rs`):
|
||
```rust
|
||
/// Enable remote volume control on Android
|
||
pub fn enable_remote_volume(initial_volume: i32) -> Result<(), String> {
|
||
start_playback_service()?;
|
||
let service_instance = get_playback_service_instance()?;
|
||
env.call_method(&service_instance, "enableRemoteVolume", "(I)V",
|
||
&[JValue::Int(initial_volume)])?;
|
||
Ok(())
|
||
}
|
||
```
|
||
|
||
**Dependencies** (`src-tauri/android/build.gradle.kts`):
|
||
```kotlin
|
||
dependencies {
|
||
implementation("androidx.media3:media3-session:1.5.1") // Media3 MediaSession
|
||
implementation("androidx.media:media:1.7.0") // MediaSessionCompat
|
||
}
|
||
```
|
||
|
||
**Integration with Playback Mode:**
|
||
|
||
Remote volume is automatically enabled/disabled during playback mode transfers:
|
||
|
||
```rust
|
||
// In PlaybackModeManager::transfer_to_remote()
|
||
#[cfg(target_os = "android")]
|
||
{
|
||
if let Err(e) = crate::player::enable_remote_volume(50) {
|
||
log::warn!("Failed to enable remote volume: {}", e);
|
||
}
|
||
}
|
||
|
||
// In PlaybackModeManager::transfer_to_local()
|
||
#[cfg(target_os = "android")]
|
||
{
|
||
if let Err(e) = crate::player::disable_remote_volume() {
|
||
log::warn!("Failed to disable remote volume: {}", e);
|
||
}
|
||
}
|
||
```
|
||
|
||
## Android Album Art Caching
|
||
|
||
**Location**: `src-tauri/android/src/main/java/com/dtourolle/jellytau/player/AlbumArtCache.kt`
|
||
|
||
Album art caching provides efficient bitmap storage for lock screen notifications with automatic LRU eviction and memory management.
|
||
|
||
```mermaid
|
||
flowchart TB
|
||
subgraph JellyTauPlayer["JellyTauPlayer.kt"]
|
||
LoadMedia["loadWithMetadata()<br/>- Store artworkUrl<br/>- Launch async download"]
|
||
AsyncDownload["Coroutine<br/>- Non-blocking<br/>- Dispatchers.IO"]
|
||
end
|
||
|
||
subgraph Cache["AlbumArtCache.kt"]
|
||
MemoryCache["LruCache<String, Bitmap><br/>- 1/8 of heap<br/>- ~12-16MB typical<br/>- 50-100 albums capacity"]
|
||
Download["Download & Scale<br/>- 512x512 max<br/>- Exponential backoff"]
|
||
ErrorHandle["Error Handling<br/>- Graceful fallback<br/>- Auto-retry"]
|
||
end
|
||
|
||
subgraph Service["JellyTauPlaybackService.kt"]
|
||
UpdateMeta["updateMediaMetadata()<br/>- Accept Bitmap parameter<br/>- Add METADATA_KEY_ALBUM_ART"]
|
||
Notification["Notification<br/>- setLargeIcon()<br/>- Lock screen display"]
|
||
end
|
||
|
||
LoadMedia --> AsyncDownload
|
||
AsyncDownload --> MemoryCache
|
||
MemoryCache --> Download
|
||
Download --> ErrorHandle
|
||
AsyncDownload --> UpdateMeta
|
||
UpdateMeta --> Notification
|
||
```
|
||
|
||
**AlbumArtCache Singleton:**
|
||
|
||
```kotlin
|
||
class AlbumArtCache(context: Context) {
|
||
private val memoryCache = object : LruCache<String, Bitmap>(cacheSize) {
|
||
override fun sizeOf(key: String, bitmap: Bitmap): Int {
|
||
return bitmap.byteCount / 1024 // Size in KB
|
||
}
|
||
}
|
||
|
||
suspend fun getArtwork(url: String): Bitmap? {
|
||
memoryCache.get(url)?.let { return it }
|
||
return downloadAndCache(url)
|
||
}
|
||
|
||
private suspend fun downloadAndCache(url: String): Bitmap? =
|
||
withContext(Dispatchers.IO) {
|
||
// HTTP download with 5s timeout
|
||
// Scale to 512x512 max
|
||
// Auto-evict LRU if needed
|
||
}
|
||
}
|
||
```
|
||
|
||
**Integration Flow:**
|
||
|
||
1. **Track Load** (`loadWithMetadata()`):
|
||
- Store artwork URL in `currentArtworkUrl`
|
||
- Reset bitmap to null
|
||
- Start playback immediately (non-blocking)
|
||
|
||
2. **Async Download** (Background Coroutine):
|
||
- Check cache: instant hit if available
|
||
- Network miss: download, scale, cache
|
||
- Auto-retry on network failure with exponential backoff
|
||
- Graceful fallback if artwork unavailable
|
||
|
||
3. **Notification Update**:
|
||
- Pass bitmap to `updatePlaybackServiceNotification()`
|
||
- Add to `MediaMetadataCompat` with `METADATA_KEY_ALBUM_ART`
|
||
- Display as large icon in notification
|
||
- Show on lock screen
|
||
|
||
**Memory Management:**
|
||
|
||
| Metric | Value |
|
||
|--------|-------|
|
||
| Cache Size | 1/8 of heap (12-16MB typical) |
|
||
| Max Resolution | 512x512 pixels |
|
||
| Capacity | ~50-100 album arts |
|
||
| Eviction Policy | LRU (Least Recently Used) |
|
||
| Lifetime | In-memory only (app session) |
|
||
| Network Timeout | 5 seconds per download |
|
||
|
||
**Performance Characteristics:**
|
||
|
||
- **Cache Hit**: ~1ms (in-memory retrieval)
|
||
- **Cache Miss**: ~200-500ms (download + scale)
|
||
- **Playback Impact**: Zero (async downloads)
|
||
- **Memory Overhead**: Max 16MB (auto-eviction)
|
||
- **Error Recovery**: Automatic with exponential backoff
|
||
|
||
## Backend Initialization
|
||
|
||
**Location**: `src-tauri/src/lib.rs`
|
||
|
||
Backend selection is platform-specific:
|
||
|
||
```rust
|
||
fn create_player_backend(app_handle: tauri::AppHandle) -> Box<dyn PlayerBackend> {
|
||
let event_emitter = Arc::new(TauriEventEmitter::new(app_handle));
|
||
|
||
#[cfg(target_os = "linux")]
|
||
{
|
||
match MpvBackend::new(event_emitter.clone()) {
|
||
Ok(backend) => return Box::new(backend),
|
||
Err(e) => eprintln!("MPV init failed: {}", e),
|
||
}
|
||
}
|
||
|
||
#[cfg(target_os = "android")]
|
||
{
|
||
// ExoPlayer requires Activity context, initialized separately
|
||
}
|
||
|
||
// Fallback
|
||
Box::new(NullBackend::new())
|
||
}
|
||
```
|