jellytau/docs/specs/video-background-audio.md

Spec: Background audio for video playback (Android)

Status: Draft Scope: Android only (v1). Linux noted as future work. Branch base: android-picture-in-picture Requirements: UR-040 → IR-025, JA-032, DR-051, DR-052 (see requirements.md). Tests: UT-059, UT-060, UT-061, IT-013.

Summary

Add a per-player toggle that lets the audio of a video keep playing when the app is backgrounded or the screen is locked, while video decoding stops. When the app returns to the foreground, video decoding resumes from the current audio position.

This is the audio-first counterpart to the existing Picture-in-Picture feature (which keeps the whole video decoding in a floating window). The two are mutually exclusive: enabling background audio suppresses auto-PiP.

Motivation

Users watching talk-heavy content (podcasts-as-video, lectures, music videos, concert films) want to lock the phone or switch apps and keep listening without draining battery on video decode or needing a visible floating window.

Background: how playback actually works here

Two facts drive the entire design (verified in code, not assumed):

1. Video renders through the HTML5 <video> element in the WebView on both platforms. The native ExoPlayer video surface path is disabled — see the INTERIM override in VideoPlayer.svelte around the playerPlayItem response handling (useHtml5Element is forced true, native backend is stopped). So "video decoding" == the WebView <video> element, and the WebView is what Android suspends on background.

2. An Android WebView <video> element does not keep playing audio when the app is backgrounded / locked. The system throttles the WebView and media pauses. Keeping audio alive in the background requires a native foreground media service, which already exists for music: JellyTauPlaybackService + JellyTauPlayer (ExoPlayer) + MediaSessionCompat.

Therefore the design is a handoff, not "keep the WebView alive": on background, stop the WebView <video> and start audio-only playback of the same item through the existing native ExoPlayer audio service; on foreground, hand back to the WebView <video>.

This also aligns with the project's one-directional playback rule (CLAUDE.md → "Playback state is one-directional"): the currently-authoritative player (WebView element or native audio service) drives position; the UI and MediaSession consume it. The handoff is a change of which player is authoritative, and must transfer position cleanly.

User-facing behavior

The toggle

• A toggle button in the video player controls (next to the existing PiP / fullscreen buttons in VideoPlayer.svelte).
• Icon: headphones / "audio-only" glyph. Two visual states (on/off).
• Visible only when isPipSupported()-equivalent conditions hold — i.e. Android with a native audio service available. Hidden on Linux in v1.
• State is a UI preference on the player. Consider persisting the last choice per user (see Open Questions) — v1 may default OFF each session.

When toggle is ON and the app goes to background / screen locks

1. Auto-PiP is suppressed (see "Interaction with PiP").
2. The WebView <video> is paused and its decode stopped (release the media source so the decoder is freed, not merely pause()).
3. Native audio-only playback of the same item starts at the current position, through JellyTauPlaybackService (foreground notification + lockscreen controls via the existing MediaSessionCompat).
4. Lockscreen / notification shows the item with play/pause/seek, driven by the native player (existing music behavior — reused, not rebuilt).

When toggle is ON and the app returns to foreground

1. Native audio playback stops; its final position is captured.
2. WebView <video> reloads/resumes at that position and continues as normal audiovisual playback.
3. Playback state (playing/paused) is preserved across the handoff.

When toggle is OFF (default)

Current behavior is unchanged: backgrounding video auto-enters PiP (onUserLeaveHint → PictureInPictureManager.enterPip).

Interaction with PiP

The toggle chooses one behavior or the other:

• Toggle ON → call AndroidPictureInPicture.setAutoEnterEnabled(false) (the bridge already exists, pictureInPicture.ts → setAutoEnterEnabled). Background → audio handoff instead of PiP.
• Toggle OFF → setAutoEnterEnabled(true). Background → PiP (status quo).

The frontend must also call setAutoEnterEnabled(false) on unmount if it left it enabled, and re-assert the correct value whenever the toggle changes, so a stale setting can't leak into the next player.

Note: canEnterPip() today requires isPlayingVideo() on the native ExoPlayer, but video plays via the WebView, so native isPlayingVideo() is false during normal playback. Confirm during implementation how auto-PiP is actually triggering today (it may rely on a different signal), because the background-audio handoff needs the same "is a local video active" signal to know it should fire. This is a load-bearing unknown — resolve it first (Phase 0).

Technical design

The audio-only stream

Jellyfin can transcode/stream a video item as audio-only. Add a repository method (mirroring get_video_stream_url and get_audio_stream_url) that returns an audio-only stream URL for a video item at a given audio-stream index — so the currently-selected audio track (selectedAudioTrackIndex in the player) carries over. Prefer direct-play of the audio stream where the container/codec allows; transcode to a broadly-supported audio codec otherwise.

Position semantics must match between the WebView <video> timeline and the audio stream (account for the transcoded-HLS seekOffset model already in the player — see the seekOffset handling in VideoPlayer.svelte).

Backend command surface (Rust)

New/extended #[tauri::command]s in src-tauri/src/commands/player/ (follow the camelCase param rule and Result<T, String> convention):

• player_enter_background_audio(item_id, position_seconds, audio_stream_index) — stop WebView authority, start native audio-only playback at position; makes the native player authoritative. Emits state via the existing player-event channel so MediaSession/UI stay consumers.
• player_exit_background_audio() -> position_seconds — stop native audio, return final position for the WebView to resume from; restores WebView authority.

Reuse existing player_play_* / player_stop plumbing where possible rather than adding a parallel path.

Android native

• Reuse JellyTauPlaybackService + JellyTauPlayer audio path (MediaSessionCompat, foreground notification, audio-becoming-noisy, etc. — all already implemented for music).
• Add a bridge method (alongside AndroidPictureInPicture) or reuse an existing one so the frontend can signal "prepare for background audio handoff" tied to the Activity lifecycle (onPause/onStop/onUserLeaveHint).
• On onUserLeaveHint / screen-off with background-audio enabled: do not enter PiP; instead trigger the handoff command.
• Respect the deadlock gotchas in CLAUDE.md (no sync/blocking calls from player event callbacks; bind locked AutoplayDecision to a let before matching).

Frontend (VideoPlayer.svelte)

• Add toggle state + button. On change, call setAutoEnterEnabled(!on).
• Listen for Android lifecycle background/foreground signals (via a bridge event or existing visibility hooks) and:
  • background + ON → player_enter_background_audio(...), pause + tear down the <video>/HLS decode (reuse the existing HLS teardown sequence to avoid dual audio).
  • foreground + ON → player_exit_background_audio(), reload <video> at the returned position, restore play/pause state.
• Follow the native-mode pitfall (memory: videoplayer-native-mode-pitfalls): no lifecycle calls after an await in onMount. Keep the handoff logic out of that window.
• Dual-audio is the key regression risk: at every handoff exactly one of {WebView <video>, native ExoPlayer} produces audio. Tear the other down before starting the next, mirroring the existing HLS cleanup discipline.

Phasing

• Phase 0 — De-risk (do first):
  • Confirm what actually triggers today's auto-PiP given video is on the WebView (resolve the canEnterPip/isPlayingVideo question).
  • Spike: obtain an audio-only stream URL for a video item and play it through the native audio service; measure position accuracy and that WebView audio is fully silenced (no dual audio).
• Phase 1 — Backend: repository audio-only-URL method + the two player commands + events.
• Phase 2 — Native: lifecycle wiring, PiP suppression, handoff trigger.
• Phase 3 — Frontend: toggle UI, lifecycle listeners, handoff calls, teardown discipline.
• Phase 4 — Polish: persist toggle preference, subtitle/audio-track carry-over, edge cases (calls, headphone unplug, autoplay-next during background audio).

Testing

• Rust: unit tests for the audio-only URL builder and the two commands (cargo test, bun run test:rust).
• IPC param-naming integration tests for any new commands (bun run test -- tauriIntegration.test.ts).
• Frontend: bun run check, bun run test, plus a VideoPlayer logic test for the handoff state machine (mirror the existing VideoPlayer.logic.test.ts).
• Manual on-device matrix:
  • toggle ON: home button → audio continues, video stops decoding; return → video resumes at position; playing/paused preserved.
  • toggle ON: screen lock → audio continues; lockscreen controls work; unlock → resumes.
  • toggle OFF: background → PiP (unchanged).
  • No dual audio at any transition. No audio leak after leaving the player.
  • Transcoded (HEVC/10-bit) item — verify position with seekOffset.
  • Autoplay-next fires correctly if an episode ends during background audio.

Open questions

1. Persist the toggle per user/series, or default OFF each session? (Recommend: remember last choice; series-level like the audio-track preference is a nice-to-have.)
2. Autoplay-next during background audio — should the next episode start as audio-only and stay audio until foreground, or pause at episode end? (Recommend: continue as audio-only.)
3. Subtitles are irrelevant in audio-only mode but must restore on foreground — confirm they survive the <video> teardown/reload.
4. Exact Android lifecycle signal for "screen locked" vs "app backgrounded" — onUserLeaveHint covers Home but not lock; may need a screen-off receiver.

Non-goals (v1)

• Linux background audio (desktop windows keep running unfocused; low value).
• Replacing or removing PiP — it stays as the toggle-OFF behavior.
• Re-enabling the native ExoPlayer video surface path.