jellytau/docs/specs/video-background-audio.md

# Spec: Background audio for video playback (Android)

**Status:** Draft
**Scope:** Android only (v1). Linux noted as future work.
**Branch base:** `android-picture-in-picture`
**Requirements:** UR-040 → IR-025, JA-032, DR-051, DR-052 (see
[requirements.md](../requirements.md)). Tests: UT-059, UT-060, UT-061, IT-013.

## Summary

Add a per-player toggle that lets the **audio** of a video keep playing when the
app is backgrounded or the screen is locked, while **video decoding stops**.
When the app returns to the foreground, video decoding resumes from the current
audio position.

This is the audio-first counterpart to the existing Picture-in-Picture feature
(which keeps the *whole video* decoding in a floating window). The two are
mutually exclusive: enabling background audio suppresses auto-PiP.

## Motivation

Users watching talk-heavy content (podcasts-as-video, lectures, music videos,
concert films) want to lock the phone or switch apps and keep listening without
draining battery on video decode or needing a visible floating window.

## Background: how playback actually works here

Two facts drive the entire design (verified in code, not assumed):

1. **Video renders through the HTML5 `<video>` element in the WebView on both
   platforms.** The native ExoPlayer *video* surface path is disabled — see the
   INTERIM override in
   [VideoPlayer.svelte](../../src/lib/components/player/VideoPlayer.svelte)
   around the `playerPlayItem` response handling (`useHtml5Element` is forced
   `true`, native backend is stopped). So "video decoding" == the WebView
   `<video>` element, and the WebView is what Android suspends on background.

2. **An Android WebView `<video>` element does not keep playing audio when the
   app is backgrounded / locked.** The system throttles the WebView and media
   pauses. Keeping audio alive in the background requires a **native foreground
   media service**, which already exists for music:
   [`JellyTauPlaybackService`](../../src-tauri/android/src/main/java/com/dtourolle/jellytau/player/JellyTauPlaybackService.kt)
   +
   [`JellyTauPlayer`](../../src-tauri/android/src/main/java/com/dtourolle/jellytau/player/JellyTauPlayer.kt)
   (ExoPlayer) + `MediaSessionCompat`.

**Therefore the design is a handoff**, not "keep the WebView alive": on
background, stop the WebView `<video>` and start audio-only playback of the same
item through the existing native ExoPlayer audio service; on foreground, hand
back to the WebView `<video>`.

This also aligns with the project's one-directional playback rule
(`CLAUDE.md` → "Playback state is one-directional"): the currently-authoritative
player (WebView element **or** native audio service) drives position; the UI and
MediaSession consume it. The handoff is a change of *which* player is
authoritative, and must transfer position cleanly.

## User-facing behavior

### The toggle

- A toggle button in the video player controls (next to the existing PiP /
  fullscreen buttons in
  [VideoPlayer.svelte](../../src/lib/components/player/VideoPlayer.svelte)).
- Icon: headphones / "audio-only" glyph. Two visual states (on/off).
- **Visible only when** `isPipSupported()`-equivalent conditions hold — i.e.
  Android with a native audio service available. Hidden on Linux in v1.
- State is a UI preference on the player. Consider persisting the last choice
  per user (see Open Questions) — v1 may default OFF each session.

### When toggle is ON and the app goes to background / screen locks

1. Auto-PiP is suppressed (see "Interaction with PiP").
2. The WebView `<video>` is paused and its decode stopped (release the media
   source so the decoder is freed, not merely `pause()`).
3. Native audio-only playback of the same item starts at the current position,
   through `JellyTauPlaybackService` (foreground notification + lockscreen
   controls via the existing `MediaSessionCompat`).
4. Lockscreen / notification shows the item with play/pause/seek, driven by the
   native player (existing music behavior — reused, not rebuilt).

### When toggle is ON and the app returns to foreground

1. Native audio playback stops; its final position is captured.
2. WebView `<video>` reloads/resumes at that position and continues as normal
   audiovisual playback.
3. Playback state (playing/paused) is preserved across the handoff.

### When toggle is OFF (default)

Current behavior is unchanged: backgrounding video auto-enters PiP
(`onUserLeaveHint` → `PictureInPictureManager.enterPip`).

## Interaction with PiP

The toggle chooses one behavior or the other:

- Toggle **ON** → call `AndroidPictureInPicture.setAutoEnterEnabled(false)` (the
  bridge already exists,
  [pictureInPicture.ts](../../src/lib/utils/pictureInPicture.ts) →
  `setAutoEnterEnabled`). Background → audio handoff instead of PiP.
- Toggle **OFF** → `setAutoEnterEnabled(true)`. Background → PiP (status quo).

The frontend must also call `setAutoEnterEnabled(false)` on unmount if it left
it enabled, and re-assert the correct value whenever the toggle changes, so a
stale setting can't leak into the next player.

> Note: `canEnterPip()` today requires `isPlayingVideo()` on the *native*
> ExoPlayer, but video plays via the WebView, so native `isPlayingVideo()` is
> false during normal playback. Confirm during implementation how auto-PiP is
> actually triggering today (it may rely on a different signal), because the
> background-audio handoff needs the same "is a local video active" signal to
> know it should fire. **This is a load-bearing unknown — resolve it first
> (Phase 0).**

## Technical design

### The audio-only stream

Jellyfin can transcode/stream a video item as audio-only. Add a repository
method (mirroring
[`get_video_stream_url`](../../src-tauri/src/repository/online.rs) and
[`get_audio_stream_url`](../../src-tauri/src/repository/mod.rs)) that returns an
**audio-only stream URL for a video item** at a given audio-stream index — so
the currently-selected audio track (`selectedAudioTrackIndex` in the player)
carries over. Prefer direct-play of the audio stream where the container/codec
allows; transcode to a broadly-supported audio codec otherwise.

Position semantics must match between the WebView `<video>` timeline and the
audio stream (account for the transcoded-HLS `seekOffset` model already in the
player — see the `seekOffset` handling in `VideoPlayer.svelte`).

### Backend command surface (Rust)

New/extended `#[tauri::command]`s in `src-tauri/src/commands/player/` (follow the
camelCase param rule and `Result<T, String>` convention):

- `player_enter_background_audio(item_id, position_seconds, audio_stream_index)`
  — stop WebView authority, start native audio-only playback at position; makes
  the native player authoritative. Emits state via the existing player-event
  channel so MediaSession/UI stay consumers.
- `player_exit_background_audio() -> position_seconds` — stop native audio,
  return final position for the WebView to resume from; restores WebView
  authority.

Reuse existing `player_play_*` / `player_stop` plumbing where possible rather
than adding a parallel path.

### Android native

- Reuse `JellyTauPlaybackService` + `JellyTauPlayer` audio path
  (`MediaSessionCompat`, foreground notification, audio-becoming-noisy, etc. —
  all already implemented for music).
- Add a bridge method (alongside `AndroidPictureInPicture`) or reuse an existing
  one so the frontend can signal "prepare for background audio handoff" tied to
  the Activity lifecycle (`onPause`/`onStop`/`onUserLeaveHint`).
- On `onUserLeaveHint` / screen-off with background-audio enabled: **do not**
  enter PiP; instead trigger the handoff command.
- Respect the deadlock gotchas in `CLAUDE.md` (no sync/blocking calls from
  player event callbacks; bind locked `AutoplayDecision` to a `let` before
  matching).

### Frontend (VideoPlayer.svelte)

- Add toggle state + button. On change, call `setAutoEnterEnabled(!on)`.
- Listen for Android lifecycle background/foreground signals (via a bridge event
  or existing visibility hooks) and:
  - background + ON → `player_enter_background_audio(...)`, pause + tear down the
    `<video>`/HLS decode (reuse the existing HLS teardown sequence to avoid dual
    audio).
  - foreground + ON → `player_exit_background_audio()`, reload `<video>` at the
    returned position, restore play/pause state.
- **Follow the native-mode pitfall** (memory:
  `videoplayer-native-mode-pitfalls`): no lifecycle calls after an `await` in
  `onMount`. Keep the handoff logic out of that window.
- Dual-audio is the key regression risk: at every handoff exactly one of
  {WebView `<video>`, native ExoPlayer} produces audio. Tear the other down
  *before* starting the next, mirroring the existing HLS cleanup discipline.

## Phasing

- **Phase 0 — De-risk (do first):**
  - Confirm what actually triggers today's auto-PiP given video is on the
    WebView (resolve the `canEnterPip`/`isPlayingVideo` question).
  - Spike: obtain an audio-only stream URL for a video item and play it through
    the native audio service; measure position accuracy and that WebView audio
    is fully silenced (no dual audio).
- **Phase 1 — Backend:** repository audio-only-URL method + the two player
  commands + events.
- **Phase 2 — Native:** lifecycle wiring, PiP suppression, handoff trigger.
- **Phase 3 — Frontend:** toggle UI, lifecycle listeners, handoff calls,
  teardown discipline.
- **Phase 4 — Polish:** persist toggle preference, subtitle/audio-track
  carry-over, edge cases (calls, headphone unplug, autoplay-next during
  background audio).

## Testing

- Rust: unit tests for the audio-only URL builder and the two commands
  (`cargo test`, `bun run test:rust`).
- IPC param-naming integration tests for any new commands
  (`bun run test -- tauriIntegration.test.ts`).
- Frontend: `bun run check`, `bun run test`, plus a VideoPlayer logic test for
  the handoff state machine (mirror the existing
  `VideoPlayer.logic.test.ts`).
- Manual on-device matrix:
  - toggle ON: home button → audio continues, video stops decoding; return →
    video resumes at position; playing/paused preserved.
  - toggle ON: screen lock → audio continues; lockscreen controls work; unlock →
    resumes.
  - toggle OFF: background → PiP (unchanged).
  - No dual audio at any transition. No audio leak after leaving the player.
  - Transcoded (HEVC/10-bit) item — verify position with `seekOffset`.
  - Autoplay-next fires correctly if an episode ends during background audio.

## Open questions

1. **Persist the toggle per user/series, or default OFF each session?**
   (Recommend: remember last choice; series-level like the audio-track
   preference is a nice-to-have.)
2. **Autoplay-next during background audio** — should the next episode start as
   audio-only and stay audio until foreground, or pause at episode end? (Recommend:
   continue as audio-only.)
3. **Subtitles** are irrelevant in audio-only mode but must restore on
   foreground — confirm they survive the `<video>` teardown/reload.
4. Exact **Android lifecycle signal** for "screen locked" vs "app backgrounded"
   — `onUserLeaveHint` covers Home but not lock; may need a screen-off receiver.

## Non-goals (v1)

- Linux background audio (desktop windows keep running unfocused; low value).
- Replacing or removing PiP — it stays as the toggle-OFF behavior.
- Re-enabling the native ExoPlayer *video* surface path.