Android ships no AC-3, E-AC-3, DTS or TrueHD decoders; they are licensed codecs, present only where a vendor paid for them. The ROD2-W09 tablet has a vendor DTS decoder and no AC-3/E-AC-3 at all. So every film with Dolby audio was re-encoded by the server, for streaming and for download alike, and a transcoded download has no Content-Length and ignores Range: ~1 MB/s, restarting from byte zero on every network blip. ExoPlayer now carries Jellyfin's media3 FFmpeg audio decoder in extension mode ON (platform decoders first, FFmpeg for what they lack), and CodecDetector reports its codecs so the device profile and the download policy agree with what actually decodes. The download policy judges audio against the renderer that will play the file (renderer_can_decode_audio) instead of the webview's list, so Android downloads are always the direct copy — a 910 MB E-AC-3 5.1 episode downloaded in 94 s and played offline. The webview video path is removed on Android: it decodes none of these codecs, so a stored "native video off" would play every original-file download silent. Rust reports webview_video_fallback (false on Android, true only beside mpv native video on Linux); Settings offers the switch and the player honours it only then. Linux keeps the fallback and, with it, the server transcode for undecodable audio. The decoder is GPL-3.0; the distributed APK carries its terms and the source stays MIT (THIRD_PARTY_NOTICES.md). The on-device remux spec this replaces is folded into 05-platform-backends.md and deleted. DR-293, UT-259, UT-262.
753 lines
29 KiB
Markdown
753 lines
29 KiB
Markdown
# Platform-Specific Player Backends
|
||
|
||
## Player Events System
|
||
|
||
**Location**: `src-tauri/src/player/events.rs`
|
||
|
||
The player uses a push-based event system to notify the frontend of state changes:
|
||
|
||
```rust
|
||
pub enum PlayerStatusEvent {
|
||
/// Playback position updated (emitted periodically during playback)
|
||
PositionUpdate { position: f64, duration: f64 },
|
||
|
||
/// Player state changed
|
||
StateChanged { state: String, media_id: Option<String> },
|
||
|
||
/// Media has finished loading and is ready to play
|
||
MediaLoaded { duration: f64 },
|
||
|
||
/// Playback has ended naturally
|
||
PlaybackEnded,
|
||
|
||
/// Buffering state changed
|
||
Buffering { percent: u8 },
|
||
|
||
/// An error occurred during playback
|
||
Error { message: String, recoverable: bool },
|
||
|
||
/// Volume changed
|
||
VolumeChanged { volume: f32, muted: bool },
|
||
|
||
/// Sleep timer state changed
|
||
SleepTimerChanged {
|
||
mode: SleepTimerMode,
|
||
remaining_seconds: u32,
|
||
},
|
||
|
||
/// Show next episode popup with countdown
|
||
ShowNextEpisodePopup {
|
||
current_episode: MediaItem,
|
||
next_episode: MediaItem,
|
||
countdown_seconds: u32,
|
||
auto_advance: bool,
|
||
},
|
||
|
||
/// Countdown tick (emitted every second during autoplay countdown)
|
||
CountdownTick { remaining_seconds: u32 },
|
||
|
||
/// Queue changed (items added, removed, reordered, or playback mode changed)
|
||
QueueChanged {
|
||
items: Vec<MediaItem>,
|
||
current_index: Option<usize>,
|
||
shuffle: bool,
|
||
repeat: RepeatMode,
|
||
has_next: bool,
|
||
has_previous: bool,
|
||
},
|
||
|
||
/// Media session changed (activity context changed: Audio/Movie/TvShow/Idle)
|
||
SessionChanged { session: MediaSessionType },
|
||
}
|
||
```
|
||
|
||
Events are emitted via Tauri's event system:
|
||
|
||
```mermaid
|
||
flowchart LR
|
||
subgraph Backend["Player Backend"]
|
||
MPV["MPV/ExoPlayer"]
|
||
end
|
||
|
||
subgraph EventSystem["Event System"]
|
||
Emitter["TauriEventEmitter<br/>emit()"]
|
||
Bus["Tauri Event Bus<br/>'player-event'"]
|
||
end
|
||
|
||
subgraph Frontend["Frontend"]
|
||
Listener["playerEvents.ts<br/>Frontend Listener"]
|
||
Store["Player Store Update<br/>(position, state, etc)"]
|
||
end
|
||
|
||
MPV --> Emitter --> Bus --> Listener --> Store
|
||
```
|
||
|
||
**Frontend Listener** (`src/lib/services/playerEvents.ts`):
|
||
- Listens for `player-event` Tauri events
|
||
- Updates player/queue stores based on event type
|
||
- Auto-advances to next track on `PlaybackEnded`
|
||
- On `StateChanged` events, calls `invoke("player_get_queue")` to update `appState.hasNext`/`hasPrevious` -- this enables MiniPlayer skip button state
|
||
|
||
**Important**: The command is `player_get_queue` (returns `QueueStatus` with `hasNext`/`hasPrevious`). There is no `player_get_queue_status` command.
|
||
|
||
## HTML5 Video Adapter (webview-rendered video)
|
||
|
||
**Location**: `src/lib/player/html5Adapter.ts`, `src/lib/player/index.ts`, report commands in
|
||
`src-tauri/src/commands/player/timers.rs`
|
||
|
||
Video on desktop (Linux WebKitGTK) is rendered by an HTML5 `<video>`/HLS element **inside the
|
||
webview**. Android no longer uses this path for video — see *The webview is not a video renderer on
|
||
Android* below. libmpv is initialized audio-only (`vo=null`,
|
||
`video=false`), so the native backend cannot render or observe this element. The `<video>` is therefore
|
||
the real player, living outside Rust's reach.
|
||
|
||
To keep the `PlayerController` the single source of truth (matching the audio path), the HTML5 element
|
||
is treated as **a dumb output device that reports back into Rust**, rather than an independent state
|
||
authority:
|
||
|
||
```mermaid
|
||
flowchart LR
|
||
subgraph Webview["Webview"]
|
||
Video["HTML5 <video> / HLS.js"]
|
||
Adapter["html5Adapter.ts<br/>(reports DOM events)"]
|
||
end
|
||
subgraph Backend["Rust"]
|
||
Cmds["player_report_state<br/>player_report_position<br/>player_report_media_loaded"]
|
||
Controller["PlayerController"]
|
||
Emitter["TauriEventEmitter"]
|
||
end
|
||
subgraph Frontend["Frontend"]
|
||
Events["playerEvents.ts"]
|
||
Store["player store"]
|
||
end
|
||
|
||
Video -->|DOM events| Adapter --> Cmds --> Controller --> Emitter --> Events --> Store
|
||
```
|
||
|
||
**Key points:**
|
||
- The adapter re-emits the *same* `PlayerStatusEvent`s (`StateChanged`, `PositionUpdate`, `MediaLoaded`)
|
||
the native backends emit, so `playerEvents.ts` needs **no** HTML5-specific branch — HTML5 is just
|
||
another event source feeding the existing pipeline.
|
||
- Position reports are throttled (~250ms) to match the MPV cadence and avoid flooding IPC from the
|
||
60fps RAF loop.
|
||
- **Boundary rule**: UI components never touch the report commands or `videoElement` state directly.
|
||
Playback *control* goes through the unified facade `src/lib/player/index.ts` (`playerController`);
|
||
HTML5 *state reporting* goes through `html5Adapter.ts`. This restores the documented invariant
|
||
("frontend only displays state and invokes commands") for the video path.
|
||
|
||
## MpvBackend (Linux)
|
||
|
||
**Location**: `src-tauri/src/player/mpv/`
|
||
|
||
The MPV backend uses libmpv for audio playback on Linux. Since MPV handles are not `Send`, all operations occur on a dedicated thread.
|
||
|
||
```mermaid
|
||
flowchart TB
|
||
subgraph MainThread["Main Thread"]
|
||
MpvBackend["MpvBackend<br/>- command_tx<br/>- shared_state<br/>- shutdown"]
|
||
Commands["Commands:<br/>Load, Play, Pause<br/>Stop, Seek, SetVolume"]
|
||
end
|
||
|
||
subgraph EventLoopThread["MPV Event Loop Thread"]
|
||
EventLoop["event_loop.rs<br/>- MPV Handle<br/>- command_rx<br/>- Event Emitter"]
|
||
TauriEmitter["TauriEventEmitter"]
|
||
end
|
||
|
||
MpvBackend -->|"MpvCommand"| EventLoop
|
||
MpvBackend <-->|"Arc<Mutex<>>"| EventLoop
|
||
EventLoop -->|"Events"| TauriEmitter
|
||
TauriEmitter --> FrontendStore["Frontend Store"]
|
||
```
|
||
|
||
**Key Components:**
|
||
|
||
```rust
|
||
// Command enum sent to event loop thread
|
||
pub enum MpvCommand {
|
||
Load { url: String, media: MediaItem },
|
||
Play,
|
||
Pause,
|
||
Stop,
|
||
Seek(f64),
|
||
SetVolume(f32),
|
||
Quit,
|
||
}
|
||
|
||
// Shared state between main thread and event loop
|
||
pub struct MpvSharedState {
|
||
pub state: PlayerState,
|
||
pub position: f64,
|
||
pub duration: Option<f64>,
|
||
pub volume: f32,
|
||
pub is_loaded: bool,
|
||
pub current_media: Option<MediaItem>,
|
||
}
|
||
```
|
||
|
||
**Event Loop** (`event_loop.rs`):
|
||
- Initializes MPV with audio-only config (`vo=null`, `video=false`)
|
||
- Observes properties: `time-pos`, `duration`, `pause`, `volume`
|
||
- Emits position updates every 250ms during playback
|
||
- Processes commands from channel (non-blocking)
|
||
- Handles MPV events: `FileLoaded`, `EndFile`, `PropertyChange`
|
||
|
||
## ExoPlayerBackend (Android)
|
||
|
||
**Location**: `src-tauri/src/player/android/` and Kotlin sources
|
||
|
||
The ExoPlayer backend uses Android's Media3/ExoPlayer library via JNI.
|
||
|
||
```mermaid
|
||
flowchart TB
|
||
subgraph RustNative["Rust (Native)"]
|
||
ExoBackend["ExoPlayerBackend<br/>- player_ref<br/>- shared_state"]
|
||
NativeFuncs["JNI Callbacks<br/>nativeOnPosition...<br/>nativeOnState...<br/>nativeOnMediaLoaded<br/>nativeOnPlaybackEnd"]
|
||
TauriEmitter2["TauriEventEmitter"]
|
||
end
|
||
|
||
subgraph KotlinJVM["Kotlin (JVM)"]
|
||
JellyTauPlayer["JellyTauPlayer<br/>- ExoPlayer<br/>- Player.Listener"]
|
||
end
|
||
|
||
ExoBackend -->|"JNI Calls"| JellyTauPlayer
|
||
JellyTauPlayer -->|"Callbacks"| NativeFuncs
|
||
NativeFuncs --> TauriEmitter2
|
||
TauriEmitter2 --> FrontendStore2["Frontend Store"]
|
||
```
|
||
|
||
**Kotlin Player** (`JellyTauPlayer.kt`):
|
||
```kotlin
|
||
class JellyTauPlayer(context: Context) {
|
||
private val exoPlayer: ExoPlayer
|
||
private var positionUpdateJob: Job?
|
||
|
||
// Methods callable from Rust via JNI
|
||
fun load(url: String, mediaId: String)
|
||
fun play()
|
||
fun pause()
|
||
fun stop()
|
||
fun seek(positionSeconds: Double)
|
||
fun setVolume(volume: Float)
|
||
|
||
// Native callbacks to Rust
|
||
private external fun nativeOnPositionUpdate(position: Double, duration: Double)
|
||
private external fun nativeOnStateChanged(state: String, mediaId: String?)
|
||
private external fun nativeOnMediaLoaded(duration: Double)
|
||
private external fun nativeOnPlaybackEnded()
|
||
}
|
||
```
|
||
|
||
**JNI Callbacks** (Rust):
|
||
```rust
|
||
#[no_mangle]
|
||
pub extern "system" fn Java_com_dtourolle_jellytau_player_JellyTauPlayer_nativeOnPositionUpdate(
|
||
_env: JNIEnv, _class: JClass, position: jdouble, duration: jdouble
|
||
) {
|
||
// Update shared state
|
||
// Emit PlayerStatusEvent::PositionUpdate
|
||
}
|
||
```
|
||
|
||
### Audio settings on ExoPlayer
|
||
|
||
**TRACES**: UR-027, UR-032, UR-033 | DR-030, DR-035, DR-036
|
||
|
||
`PlayerBackend` declares `set_audio_settings` with a default `Ok(())` body. For a
|
||
long time `ExoPlayerBackend` took that default, so Settings › Audio rendered
|
||
controls that silently did nothing on Android — the parity gap recorded in
|
||
[requirements.md](../requirements.md#platform-playback-backend-parity-linux-vs-android),
|
||
now closed.
|
||
|
||
The settings cross to Kotlin as **JSON over JNI**, not as a wide signature, so new
|
||
fields do not change the method signature — the same approach `load()` uses for
|
||
subtitles:
|
||
|
||
```rust
|
||
fn set_audio_settings(&mut self, settings: &AudioSettings) -> Result<(), PlayerError> {
|
||
let json = audio_settings_jni_payload(settings)?;
|
||
env.call_method(&self.player_ref, "setAudioSettings", "(Ljava/lang/String;)V", …)?;
|
||
// Store the sanitised form, so audio_settings() reflects what was applied.
|
||
self.shared_state.lock_safe().audio_settings =
|
||
settings.clone().with_crossfade_clamped().with_equalizer_normalised();
|
||
}
|
||
```
|
||
|
||
Kotlin owns the *mechanics* — attaching `AudioEffect`s to the audio session — while
|
||
the canonical band layout and preset curves stay in Rust:
|
||
|
||
| Feature | Android mechanism | Notes |
|
||
|---------|-------------------|-------|
|
||
| Gapless | `pauseAtEndOfMediaItems` | |
|
||
| Volume normalization | `LoudnessEnhancer` | A gain stage — approximate next to MPV's `dynaudnorm` |
|
||
| Equalizer | `android.media.audiofx.Equalizer` | The canonical 10 bands are resampled onto the device's own band centres |
|
||
| Crossfade | — | Unimplemented on **every** platform (DR-034), architecturally blocked on MPV. Building it on Android alone would invert the parity gap |
|
||
|
||
Two things are deliberately still open: the effects are **not yet verified on a
|
||
physical device** (`AudioEffect` availability and band layouts are device-specific),
|
||
and the trait default is still a silent `Ok(())` rather than an error, so a backend
|
||
that omits the method still reports success. Flipping that default waits on the
|
||
device verification.
|
||
|
||
### Licensed audio codecs: the FFmpeg extension
|
||
|
||
**TRACES**: UR-004, UR-071 | DR-293
|
||
|
||
Android does not ship AC-3, E-AC-3, DTS or TrueHD decoders — they are licensed
|
||
codecs, present only where a vendor paid for them. The ROD2-W09 test tablet has a
|
||
vendor DTS decoder and no AC-3/E-AC-3 at all. ExoPlayer has no decoders of its
|
||
own, so on such a device those tracks are undecodable, and before this every film
|
||
with Dolby audio was re-encoded by the server — for streaming *and* for download.
|
||
|
||
`JellyTauPlayer` builds ExoPlayer with `DefaultRenderersFactory` in
|
||
`EXTENSION_RENDERER_MODE_ON`: the platform's decoders are tried first (a vendor DTS
|
||
decoder stays in charge where there is one) and the FFmpeg audio renderer takes
|
||
what they cannot decode. `CodecDetector` reports the extension's codecs beside the
|
||
`MediaCodecList` ones, asking `FfmpegLibrary.supportsFormat` per MIME type rather
|
||
than assuming, so a build whose native library failed to load reports only what
|
||
the platform decodes. Rust's device profile and download policy read that list,
|
||
which is what keeps "what we tell the server" and "what actually decodes" in step.
|
||
|
||
The decoder is `org.jellyfin.media3:media3-ffmpeg-decoder` — Jellyfin's build of
|
||
media3's FFmpeg extension, versioned `<media3 version>+N`. **Bump it in the same
|
||
commit as media3.** It is GPL-3.0: the distributed APK carries those terms, the
|
||
source stays MIT (see `THIRD_PARTY_NOTICES.md`). Its JNI methods are covered by the
|
||
AAR's own consumer rules and by `-keep class androidx.media3.** { *; }` in
|
||
`proguard-jellytau.pro`, which also keeps the renderer ExoPlayer loads reflectively.
|
||
|
||
**Rejected:** re-encoding a download's audio on the device after it lands (a
|
||
remux). It costs minutes of CPU and twice the disk per film, needs a pipeline
|
||
state of its own, and does nothing for streaming. Decoding at playback fixes both
|
||
paths with no extra step.
|
||
|
||
### The webview is not a video renderer on Android
|
||
|
||
ExoPlayer is Android's only video renderer. The HTML5 path used to be reachable
|
||
through the `experimentalNativeVideo` setting (a *suppressor* of Rust's native
|
||
choice), but the webview decodes none of the codecs above — so with the original
|
||
file now downloaded as-is (DR-293), turning native video off would play every such
|
||
download as a silent film. Rust reports `webview_video_fallback` in
|
||
`PlaybackCapabilities`: **false on Android**, true only beside mpv native video on
|
||
Linux, where the webview is still the tested fallback. The frontend offers the
|
||
switch and honours a stored "off" only when it is true (`nativeVideoWanted` in
|
||
`stores/nativeVideo.ts`), so a user who once switched it off on Android is not
|
||
stranded on the silent path.
|
||
|
||
### The equalizer, and where its vocabulary lives
|
||
|
||
**TRACES**: UR-027 | DR-030, IR-020
|
||
|
||
The canonical band layout (`EQ_BANDS`) and the preset curves live in
|
||
`settings.rs`, **not** in either backend and not in the UI: a preset *is* a gain
|
||
curve defined by the band layout, and the layout is a property of the audio
|
||
engine rather than of the picker that renders it. Presets are Flat, Rock, Pop,
|
||
Jazz, Classical, Bass Boost, Treble Boost and Vocal, all conservative (within
|
||
±8 dB) so they stack safely with volume normalization.
|
||
|
||
| Platform | Mechanism |
|
||
|----------|-----------|
|
||
| Linux | One ffmpeg two-pole peaking `equalizer` filter per band, composed by `build_af_filter` into MPV's `af` property alongside the normalization filter: `equalizer=f=31:width_type=o:width=1:g=5` |
|
||
| Android | `android.media.audiofx.Equalizer`, with the canonical 10 bands **resampled onto whatever band centres the device actually has** |
|
||
|
||
Gains are normalised (`with_equalizer_normalised`) before use, and bands beyond
|
||
`EQ_BANDS` are ignored, so a malformed settings payload cannot produce a filter
|
||
chain of unbounded length.
|
||
|
||
## Background Audio Handoff (Android)
|
||
|
||
**TRACES**: UR-040 | IR-025, DR-051, DR-052, DR-178 … DR-180, DR-196, DR-203
|
||
|
||
Keeping a video's **audio** alive when the app is backgrounded or the screen
|
||
locks, while video decode stops. Two verified facts drive the whole design:
|
||
|
||
1. An Android WebView `<video>` **does not** keep playing audio once the app is
|
||
backgrounded — the system throttles the WebView and media pauses.
|
||
2. Keeping audio alive in the background requires a **native foreground media
|
||
service**, which already exists for music (`JellyTauPlaybackService` +
|
||
`JellyTauPlayer` + `MediaSessionCompat`).
|
||
|
||
So this is a **handoff**, not "keep the WebView alive": on background, tear down
|
||
the current renderer and play the same item audio-only through the native
|
||
service; on foreground, hand back. In the project's one-directional playback
|
||
model this is a change of *which player is authoritative*, and the position must
|
||
transfer cleanly across it.
|
||
|
||
```mermaid
|
||
sequenceDiagram
|
||
participant App as App backgrounded
|
||
participant FE as VideoPlayer
|
||
participant Rust as player_enter/exit_background_audio
|
||
participant Exo as Native audio service
|
||
|
||
App-->>FE: jellytau-background (DOM CustomEvent)
|
||
FE->>Rust: enter(item, position, audioStreamIndex)
|
||
Rust->>Exo: play audio-only at position
|
||
Note over Exo: lockscreen + notification, existing MediaSession
|
||
App-->>FE: jellytau-foreground
|
||
FE->>Rust: exit() -> final position
|
||
Rust-->>FE: position
|
||
FE->>FE: restart the renderer that is on screen
|
||
```
|
||
|
||
Details that were each a shipped defect:
|
||
|
||
- **Position is absolute.** Transcoded HLS tracks time as
|
||
`videoElement.currentTime + seekOffset` (the element resets to 0 after each
|
||
transcode reload). `computeHandoffPosition` sums both terms; using the element
|
||
time alone rewinds by the offset.
|
||
- **A downloaded episode takes no base URL and an ordinary seek** (DR-180); a
|
||
stream takes the base and no seek; a handoff at 0:00 takes neither.
|
||
- **The return must restart the renderer that is actually on screen** (DR-196).
|
||
The two paths resume by different means — the webview `<video>` reloads off its
|
||
stream URL, watched by an `$effect`; ExoPlayer owns no element and nothing
|
||
watches the URL for it, so it needs an explicit re-issue. Doing only the URL
|
||
assignment restarted nothing on the native path and left a black screen with a
|
||
play button that did nothing.
|
||
- **`wasPlaying` is captured on the way out** so play/pause survives the round
|
||
trip, and the handoff does not silently rewind (DR-203).
|
||
- **Mutually exclusive with PiP.** Toggle on → `setAutoEnterEnabled(false)`;
|
||
toggle off → PiP on background, the status quo. The frontend re-asserts the
|
||
value whenever the toggle changes and on unmount, so a stale setting cannot
|
||
leak into the next player.
|
||
- The pure arithmetic and state transitions live in
|
||
`backgroundAudioHandoff.ts`, free of Svelte and the DOM, so they are testable
|
||
without mounting the player.
|
||
|
||
Native signals background/foreground to the frontend as DOM CustomEvents
|
||
(`jellytau-background` / `jellytau-foreground`); the frontend carries the toggle
|
||
state to native through the `AndroidBackgroundAudio` bridge. No-op on every
|
||
non-Android platform.
|
||
|
||
## Native Video Compositing (Android)
|
||
|
||
**TRACES**: UR-003, UR-004 | DR-150 … DR-152, DR-182 … DR-196
|
||
|
||
Android can render video on the **native ExoPlayer surface behind a transparent
|
||
Tauri WebView**, with the Svelte controls drawn over it. This is on by default;
|
||
the HTML5 `<video>` path remains the fallback and is not being removed. The
|
||
default has been flipped and reverted twice and each revert has a named cause —
|
||
the per-defect record is in `requirements.md` (DR-150 … DR-196).
|
||
|
||
```mermaid
|
||
flowchart TB
|
||
subgraph Window["One Android window"]
|
||
Texture["TextureView (index 0)<br/>ExoPlayer video"]
|
||
WebView["Tauri WebView (above)<br/>transparent, Svelte controls"]
|
||
end
|
||
Rust["ExoPlayerBackend"] -->|JNI| Player["JellyTauPlayer"]
|
||
Player --> Texture
|
||
MainActivity -->|"setTransparent(true)"| WebView
|
||
VideoOverlayManager -->|"attach / detach"| Texture
|
||
```
|
||
|
||
Load-bearing details, each of which was a shipped defect:
|
||
|
||
- **TextureView, not SurfaceView** (DR-192). A SurfaceView renders on its own
|
||
layer *outside* the app window and punches a transparent hole through it;
|
||
everything drawn above that hole — for us the whole UI — depends on that
|
||
composition path, which Android's own documentation says does not reliably
|
||
work. A TextureView makes "behind" ordinary view z-order within one window.
|
||
- **Attached at index 0** by `VideoOverlayManager`, and **detached when the video
|
||
goes** (DR-184) — a surface left in the hierarchy outlives its player.
|
||
- **Bridges are installed before the page that uses them** (DR-183).
|
||
`addJavascriptInterface` must run once per WebView instance and a call that
|
||
lands after the page has loaded never reaches it, so `setTransparent(true)`
|
||
could be dropped entirely.
|
||
- **The app shell stops painting over the surface** (DR-185). `app.css` clears
|
||
its opaque backgrounds off `[data-native-video]`; before that, a CSS rule
|
||
targeted an attribute nothing ever set, so the fix looked applied and was not.
|
||
- **The poster card can lift on a path with no `<video>` element** (DR-182) — the
|
||
native reveal fires on a `playing` state or a position tick carrying a position
|
||
or duration, and on nothing else.
|
||
- **Letterbox bars are painted**, not left holding whatever was last in the
|
||
framebuffer (DR-194).
|
||
- There is deliberately **no audio-focus bridge**: manual focus requests from the
|
||
WebView competed with Chromium's `AudioFocusDelegate` and with ExoPlayer, and
|
||
the resulting `AUDIOFOCUS_LOSS` paused playback.
|
||
|
||
Related Kotlin pieces in the same window: `PictureInPictureManager` (DR-160/161),
|
||
`ScreenWakeManager` (DR-202 — Android counts its display timeout from touch
|
||
events, which a playing video does not generate), `ImmersiveModeBridge` and
|
||
`WindowInsetsBridge` (IR-031/DR-112 — see
|
||
[02-svelte-frontend.md](02-svelte-frontend.md#safe-area-insets)).
|
||
|
||
## Android MediaSession & Remote Volume Control
|
||
|
||
**Location**: `JellyTauPlaybackService.kt`
|
||
|
||
JellyTau uses a dual MediaSession architecture for Android to support both Media3 playback controls and remote volume control:
|
||
|
||
```mermaid
|
||
flowchart TB
|
||
subgraph Service["JellyTauPlaybackService"]
|
||
MediaSession["Media3 MediaSession<br/>- Lockscreen controls<br/>- Media notifications<br/>- Play/Pause/Next/Previous"]
|
||
|
||
MediaSessionCompat["MediaSessionCompat<br/>- Remote volume control<br/>- Hardware button interception"]
|
||
|
||
VolumeProvider["VolumeProviderCompat<br/>- onSetVolumeTo()<br/>- onAdjustVolume()"]
|
||
|
||
MediaSessionCompat --> VolumeProvider
|
||
end
|
||
|
||
subgraph Hardware["System"]
|
||
VolumeButtons["Hardware Volume Buttons"]
|
||
Lockscreen["Lockscreen Controls"]
|
||
Notification["Media Notification"]
|
||
end
|
||
|
||
subgraph Rust["Rust Backend"]
|
||
JNI["JNI Callbacks<br/>nativeOnRemoteVolumeChange()"]
|
||
PlaybackMode["PlaybackModeManager<br/>send_remote_volume_command()"]
|
||
JellyfinAPI["Jellyfin API<br/>session_set_volume()"]
|
||
end
|
||
|
||
VolumeButtons --> VolumeProvider
|
||
Lockscreen --> MediaSession
|
||
Notification --> MediaSession
|
||
|
||
VolumeProvider --> JNI
|
||
JNI --> PlaybackMode
|
||
PlaybackMode --> JellyfinAPI
|
||
```
|
||
|
||
**Architecture Rationale:**
|
||
|
||
JellyTau maintains both MediaSession types because they serve different purposes:
|
||
|
||
1. **Media3 MediaSession**: Handles lockscreen/notification playback controls (play/pause/next/previous)
|
||
2. **MediaSessionCompat**: Intercepts hardware volume button presses for remote playback control
|
||
|
||
When in remote playback mode (controlling a Jellyfin session on another device):
|
||
- Volume buttons are routed through `VolumeProviderCompat`
|
||
- Volume changes are sent to the remote session via Jellyfin API
|
||
- System volume UI shows the remote session's volume level
|
||
|
||
**Remote Volume Flow:**
|
||
|
||
```mermaid
|
||
sequenceDiagram
|
||
participant User
|
||
participant VolumeButton as Hardware Volume Button
|
||
participant VolumeProvider as VolumeProviderCompat
|
||
participant JNI as nativeOnRemoteVolumeChange
|
||
participant PlaybackMode as PlaybackModeManager
|
||
participant Jellyfin as Jellyfin Server
|
||
participant RemoteSession as Remote Session (TV/Browser)
|
||
|
||
User->>VolumeButton: Press Volume Up
|
||
VolumeButton->>VolumeProvider: onAdjustVolume(ADJUST_RAISE)
|
||
VolumeProvider->>VolumeProvider: remoteVolumeLevel += 2
|
||
VolumeProvider->>VolumeProvider: currentVolume = remoteVolumeLevel
|
||
VolumeProvider->>JNI: nativeOnRemoteVolumeChange("VolumeUp", level)
|
||
JNI->>PlaybackMode: send_remote_volume_command("VolumeUp", level)
|
||
PlaybackMode->>Jellyfin: POST /Sessions/{id}/Command/VolumeUp
|
||
Jellyfin->>RemoteSession: Set volume to new level
|
||
RemoteSession-->>User: Volume changes on TV/Browser
|
||
```
|
||
|
||
**Key Implementation Details:**
|
||
|
||
**Enabling Remote Volume** (`enableRemoteVolume()`):
|
||
```kotlin
|
||
fun enableRemoteVolume(initialVolume: Int) {
|
||
volumeProvider = object : VolumeProviderCompat(
|
||
VolumeProviderCompat.VOLUME_CONTROL_ABSOLUTE,
|
||
100, // Max volume
|
||
initialVolume
|
||
) {
|
||
override fun onSetVolumeTo(volume: Int) {
|
||
remoteVolumeLevel = volume.coerceIn(0, 100)
|
||
nativeOnRemoteVolumeChange("SetVolume", remoteVolumeLevel)
|
||
}
|
||
|
||
override fun onAdjustVolume(direction: Int) {
|
||
when (direction) {
|
||
AudioManager.ADJUST_RAISE -> {
|
||
remoteVolumeLevel = (remoteVolumeLevel + 2).coerceAtMost(100)
|
||
nativeOnRemoteVolumeChange("VolumeUp", remoteVolumeLevel)
|
||
currentVolume = remoteVolumeLevel
|
||
}
|
||
AudioManager.ADJUST_LOWER -> {
|
||
remoteVolumeLevel = (remoteVolumeLevel - 2).coerceAtLeast(0)
|
||
nativeOnRemoteVolumeChange("VolumeDown", remoteVolumeLevel)
|
||
currentVolume = remoteVolumeLevel
|
||
}
|
||
}
|
||
}
|
||
}
|
||
|
||
mediaSessionCompat.setPlaybackToRemote(volumeProvider)
|
||
}
|
||
```
|
||
|
||
**Disabling Remote Volume** (`disableRemoteVolume()`):
|
||
```kotlin
|
||
fun disableRemoteVolume() {
|
||
mediaSessionCompat.setPlaybackToLocal(AudioManager.STREAM_MUSIC)
|
||
volumeProvider = null
|
||
}
|
||
```
|
||
|
||
**Rust Integration** (`src-tauri/src/player/android/mod.rs`):
|
||
```rust
|
||
/// Enable remote volume control on Android
|
||
pub fn enable_remote_volume(initial_volume: i32) -> Result<(), String> {
|
||
start_playback_service()?;
|
||
let service_instance = get_playback_service_instance()?;
|
||
env.call_method(&service_instance, "enableRemoteVolume", "(I)V",
|
||
&[JValue::Int(initial_volume)])?;
|
||
Ok(())
|
||
}
|
||
```
|
||
|
||
**Dependencies** (`src-tauri/android/build.gradle.kts`):
|
||
```kotlin
|
||
dependencies {
|
||
implementation("androidx.media3:media3-session:1.5.1") // Media3 MediaSession
|
||
implementation("androidx.media:media:1.7.0") // MediaSessionCompat
|
||
}
|
||
```
|
||
|
||
**Integration with Playback Mode:**
|
||
|
||
Remote volume is automatically enabled/disabled during playback mode transfers:
|
||
|
||
```rust
|
||
// In PlaybackModeManager::transfer_to_remote()
|
||
#[cfg(target_os = "android")]
|
||
{
|
||
if let Err(e) = crate::player::enable_remote_volume(50) {
|
||
log::warn!("Failed to enable remote volume: {}", e);
|
||
}
|
||
}
|
||
|
||
// In PlaybackModeManager::transfer_to_local()
|
||
#[cfg(target_os = "android")]
|
||
{
|
||
if let Err(e) = crate::player::disable_remote_volume() {
|
||
log::warn!("Failed to disable remote volume: {}", e);
|
||
}
|
||
}
|
||
```
|
||
|
||
## Android Album Art Caching
|
||
|
||
**Location**: `src-tauri/android/src/main/java/com/dtourolle/jellytau/player/AlbumArtCache.kt`
|
||
|
||
Album art caching provides efficient bitmap storage for lock screen notifications with automatic LRU eviction and memory management.
|
||
|
||
```mermaid
|
||
flowchart TB
|
||
subgraph JellyTauPlayer["JellyTauPlayer.kt"]
|
||
LoadMedia["loadWithMetadata()<br/>- Store artworkUrl<br/>- Launch async download"]
|
||
AsyncDownload["Coroutine<br/>- Non-blocking<br/>- Dispatchers.IO"]
|
||
end
|
||
|
||
subgraph Cache["AlbumArtCache.kt"]
|
||
MemoryCache["LruCache<String, Bitmap><br/>- 1/8 of heap<br/>- ~12-16MB typical<br/>- 50-100 albums capacity"]
|
||
Download["Download & Scale<br/>- 512x512 max<br/>- Exponential backoff"]
|
||
ErrorHandle["Error Handling<br/>- Graceful fallback<br/>- Auto-retry"]
|
||
end
|
||
|
||
subgraph Service["JellyTauPlaybackService.kt"]
|
||
UpdateMeta["updateMediaMetadata()<br/>- Accept Bitmap parameter<br/>- Add METADATA_KEY_ALBUM_ART"]
|
||
Notification["Notification<br/>- setLargeIcon()<br/>- Lock screen display"]
|
||
end
|
||
|
||
LoadMedia --> AsyncDownload
|
||
AsyncDownload --> MemoryCache
|
||
MemoryCache --> Download
|
||
Download --> ErrorHandle
|
||
AsyncDownload --> UpdateMeta
|
||
UpdateMeta --> Notification
|
||
```
|
||
|
||
**AlbumArtCache Singleton:**
|
||
|
||
```kotlin
|
||
class AlbumArtCache(context: Context) {
|
||
private val memoryCache = object : LruCache<String, Bitmap>(cacheSize) {
|
||
override fun sizeOf(key: String, bitmap: Bitmap): Int {
|
||
return bitmap.byteCount / 1024 // Size in KB
|
||
}
|
||
}
|
||
|
||
suspend fun getArtwork(url: String): Bitmap? {
|
||
memoryCache.get(url)?.let { return it }
|
||
return downloadAndCache(url)
|
||
}
|
||
|
||
private suspend fun downloadAndCache(url: String): Bitmap? =
|
||
withContext(Dispatchers.IO) {
|
||
// HTTP download with 5s timeout
|
||
// Scale to 512x512 max
|
||
// Auto-evict LRU if needed
|
||
}
|
||
}
|
||
```
|
||
|
||
**Integration Flow:**
|
||
|
||
1. **Track Load** (`loadWithMetadata()`):
|
||
- Store artwork URL in `currentArtworkUrl`
|
||
- Reset bitmap to null
|
||
- Start playback immediately (non-blocking)
|
||
|
||
2. **Async Download** (Background Coroutine):
|
||
- Check cache: instant hit if available
|
||
- Network miss: download, scale, cache
|
||
- Auto-retry on network failure with exponential backoff
|
||
- Graceful fallback if artwork unavailable
|
||
|
||
3. **Notification Update**:
|
||
- Pass bitmap to `updatePlaybackServiceNotification()`
|
||
- Add to `MediaMetadataCompat` with `METADATA_KEY_ALBUM_ART`
|
||
- Display as large icon in notification
|
||
- Show on lock screen
|
||
|
||
**Memory Management:**
|
||
|
||
| Metric | Value |
|
||
|--------|-------|
|
||
| Cache Size | 1/8 of heap (12-16MB typical) |
|
||
| Max Resolution | 512x512 pixels |
|
||
| Capacity | ~50-100 album arts |
|
||
| Eviction Policy | LRU (Least Recently Used) |
|
||
| Lifetime | In-memory only (app session) |
|
||
| Network Timeout | 5 seconds per download |
|
||
|
||
**Performance Characteristics:**
|
||
|
||
- **Cache Hit**: ~1ms (in-memory retrieval)
|
||
- **Cache Miss**: ~200-500ms (download + scale)
|
||
- **Playback Impact**: Zero (async downloads)
|
||
- **Memory Overhead**: Max 16MB (auto-eviction)
|
||
- **Error Recovery**: Automatic with exponential backoff
|
||
|
||
## Backend Initialization
|
||
|
||
**Location**: `src-tauri/src/lib.rs`
|
||
|
||
Backend selection is platform-specific:
|
||
|
||
```rust
|
||
fn create_player_backend(app_handle: tauri::AppHandle) -> Box<dyn PlayerBackend> {
|
||
let event_emitter = Arc::new(TauriEventEmitter::new(app_handle));
|
||
|
||
#[cfg(target_os = "linux")]
|
||
{
|
||
match MpvBackend::new(event_emitter.clone()) {
|
||
Ok(backend) => return Box::new(backend),
|
||
Err(e) => eprintln!("MPV init failed: {}", e),
|
||
}
|
||
}
|
||
|
||
#[cfg(target_os = "android")]
|
||
{
|
||
// ExoPlayer requires Activity context, initialized separately
|
||
}
|
||
|
||
// Fallback
|
||
Box::new(NullBackend::new())
|
||
}
|
||
```
|