docs: add mdBook docs-site, publish workflow, and release-notes tooling
Add a docs-site (mdBook) with a Gitea publish-docs workflow, a release-notes generator script (release:notes) that turns a commit range's TRACES into grouped notes, the background-audio feature spec, and CLAUDE.md. Ignore docs-site build artifacts.
This commit is contained in:
@@ -0,0 +1,233 @@
|
||||
# Spec: Background audio for video playback (Android)
|
||||
|
||||
**Status:** Draft
|
||||
**Scope:** Android only (v1). Linux noted as future work.
|
||||
**Branch base:** `android-picture-in-picture`
|
||||
**Requirements:** UR-040 → IR-025, JA-032, DR-051, DR-052 (see
|
||||
[requirements.md](../requirements.md)). Tests: UT-059, UT-060, UT-061, IT-013.
|
||||
|
||||
## Summary
|
||||
|
||||
Add a per-player toggle that lets the **audio** of a video keep playing when the
|
||||
app is backgrounded or the screen is locked, while **video decoding stops**.
|
||||
When the app returns to the foreground, video decoding resumes from the current
|
||||
audio position.
|
||||
|
||||
This is the audio-first counterpart to the existing Picture-in-Picture feature
|
||||
(which keeps the *whole video* decoding in a floating window). The two are
|
||||
mutually exclusive: enabling background audio suppresses auto-PiP.
|
||||
|
||||
## Motivation
|
||||
|
||||
Users watching talk-heavy content (podcasts-as-video, lectures, music videos,
|
||||
concert films) want to lock the phone or switch apps and keep listening without
|
||||
draining battery on video decode or needing a visible floating window.
|
||||
|
||||
## Background: how playback actually works here
|
||||
|
||||
Two facts drive the entire design (verified in code, not assumed):
|
||||
|
||||
1. **Video renders through the HTML5 `<video>` element in the WebView on both
|
||||
platforms.** The native ExoPlayer *video* surface path is disabled — see the
|
||||
INTERIM override in
|
||||
[VideoPlayer.svelte](../../src/lib/components/player/VideoPlayer.svelte)
|
||||
around the `playerPlayItem` response handling (`useHtml5Element` is forced
|
||||
`true`, native backend is stopped). So "video decoding" == the WebView
|
||||
`<video>` element, and the WebView is what Android suspends on background.
|
||||
|
||||
2. **An Android WebView `<video>` element does not keep playing audio when the
|
||||
app is backgrounded / locked.** The system throttles the WebView and media
|
||||
pauses. Keeping audio alive in the background requires a **native foreground
|
||||
media service**, which already exists for music:
|
||||
[`JellyTauPlaybackService`](../../src-tauri/android/src/main/java/com/dtourolle/jellytau/player/JellyTauPlaybackService.kt)
|
||||
+
|
||||
[`JellyTauPlayer`](../../src-tauri/android/src/main/java/com/dtourolle/jellytau/player/JellyTauPlayer.kt)
|
||||
(ExoPlayer) + `MediaSessionCompat`.
|
||||
|
||||
**Therefore the design is a handoff**, not "keep the WebView alive": on
|
||||
background, stop the WebView `<video>` and start audio-only playback of the same
|
||||
item through the existing native ExoPlayer audio service; on foreground, hand
|
||||
back to the WebView `<video>`.
|
||||
|
||||
This also aligns with the project's one-directional playback rule
|
||||
(`CLAUDE.md` → "Playback state is one-directional"): the currently-authoritative
|
||||
player (WebView element **or** native audio service) drives position; the UI and
|
||||
MediaSession consume it. The handoff is a change of *which* player is
|
||||
authoritative, and must transfer position cleanly.
|
||||
|
||||
## User-facing behavior
|
||||
|
||||
### The toggle
|
||||
|
||||
- A toggle button in the video player controls (next to the existing PiP /
|
||||
fullscreen buttons in
|
||||
[VideoPlayer.svelte](../../src/lib/components/player/VideoPlayer.svelte)).
|
||||
- Icon: headphones / "audio-only" glyph. Two visual states (on/off).
|
||||
- **Visible only when** `isPipSupported()`-equivalent conditions hold — i.e.
|
||||
Android with a native audio service available. Hidden on Linux in v1.
|
||||
- State is a UI preference on the player. Consider persisting the last choice
|
||||
per user (see Open Questions) — v1 may default OFF each session.
|
||||
|
||||
### When toggle is ON and the app goes to background / screen locks
|
||||
|
||||
1. Auto-PiP is suppressed (see "Interaction with PiP").
|
||||
2. The WebView `<video>` is paused and its decode stopped (release the media
|
||||
source so the decoder is freed, not merely `pause()`).
|
||||
3. Native audio-only playback of the same item starts at the current position,
|
||||
through `JellyTauPlaybackService` (foreground notification + lockscreen
|
||||
controls via the existing `MediaSessionCompat`).
|
||||
4. Lockscreen / notification shows the item with play/pause/seek, driven by the
|
||||
native player (existing music behavior — reused, not rebuilt).
|
||||
|
||||
### When toggle is ON and the app returns to foreground
|
||||
|
||||
1. Native audio playback stops; its final position is captured.
|
||||
2. WebView `<video>` reloads/resumes at that position and continues as normal
|
||||
audiovisual playback.
|
||||
3. Playback state (playing/paused) is preserved across the handoff.
|
||||
|
||||
### When toggle is OFF (default)
|
||||
|
||||
Current behavior is unchanged: backgrounding video auto-enters PiP
|
||||
(`onUserLeaveHint` → `PictureInPictureManager.enterPip`).
|
||||
|
||||
## Interaction with PiP
|
||||
|
||||
The toggle chooses one behavior or the other:
|
||||
|
||||
- Toggle **ON** → call `AndroidPictureInPicture.setAutoEnterEnabled(false)` (the
|
||||
bridge already exists,
|
||||
[pictureInPicture.ts](../../src/lib/utils/pictureInPicture.ts) →
|
||||
`setAutoEnterEnabled`). Background → audio handoff instead of PiP.
|
||||
- Toggle **OFF** → `setAutoEnterEnabled(true)`. Background → PiP (status quo).
|
||||
|
||||
The frontend must also call `setAutoEnterEnabled(false)` on unmount if it left
|
||||
it enabled, and re-assert the correct value whenever the toggle changes, so a
|
||||
stale setting can't leak into the next player.
|
||||
|
||||
> Note: `canEnterPip()` today requires `isPlayingVideo()` on the *native*
|
||||
> ExoPlayer, but video plays via the WebView, so native `isPlayingVideo()` is
|
||||
> false during normal playback. Confirm during implementation how auto-PiP is
|
||||
> actually triggering today (it may rely on a different signal), because the
|
||||
> background-audio handoff needs the same "is a local video active" signal to
|
||||
> know it should fire. **This is a load-bearing unknown — resolve it first
|
||||
> (Phase 0).**
|
||||
|
||||
## Technical design
|
||||
|
||||
### The audio-only stream
|
||||
|
||||
Jellyfin can transcode/stream a video item as audio-only. Add a repository
|
||||
method (mirroring
|
||||
[`get_video_stream_url`](../../src-tauri/src/repository/online.rs) and
|
||||
[`get_audio_stream_url`](../../src-tauri/src/repository/mod.rs)) that returns an
|
||||
**audio-only stream URL for a video item** at a given audio-stream index — so
|
||||
the currently-selected audio track (`selectedAudioTrackIndex` in the player)
|
||||
carries over. Prefer direct-play of the audio stream where the container/codec
|
||||
allows; transcode to a broadly-supported audio codec otherwise.
|
||||
|
||||
Position semantics must match between the WebView `<video>` timeline and the
|
||||
audio stream (account for the transcoded-HLS `seekOffset` model already in the
|
||||
player — see the `seekOffset` handling in `VideoPlayer.svelte`).
|
||||
|
||||
### Backend command surface (Rust)
|
||||
|
||||
New/extended `#[tauri::command]`s in `src-tauri/src/commands/player/` (follow the
|
||||
camelCase param rule and `Result<T, String>` convention):
|
||||
|
||||
- `player_enter_background_audio(item_id, position_seconds, audio_stream_index)`
|
||||
— stop WebView authority, start native audio-only playback at position; makes
|
||||
the native player authoritative. Emits state via the existing player-event
|
||||
channel so MediaSession/UI stay consumers.
|
||||
- `player_exit_background_audio() -> position_seconds` — stop native audio,
|
||||
return final position for the WebView to resume from; restores WebView
|
||||
authority.
|
||||
|
||||
Reuse existing `player_play_*` / `player_stop` plumbing where possible rather
|
||||
than adding a parallel path.
|
||||
|
||||
### Android native
|
||||
|
||||
- Reuse `JellyTauPlaybackService` + `JellyTauPlayer` audio path
|
||||
(`MediaSessionCompat`, foreground notification, audio-becoming-noisy, etc. —
|
||||
all already implemented for music).
|
||||
- Add a bridge method (alongside `AndroidPictureInPicture`) or reuse an existing
|
||||
one so the frontend can signal "prepare for background audio handoff" tied to
|
||||
the Activity lifecycle (`onPause`/`onStop`/`onUserLeaveHint`).
|
||||
- On `onUserLeaveHint` / screen-off with background-audio enabled: **do not**
|
||||
enter PiP; instead trigger the handoff command.
|
||||
- Respect the deadlock gotchas in `CLAUDE.md` (no sync/blocking calls from
|
||||
player event callbacks; bind locked `AutoplayDecision` to a `let` before
|
||||
matching).
|
||||
|
||||
### Frontend (VideoPlayer.svelte)
|
||||
|
||||
- Add toggle state + button. On change, call `setAutoEnterEnabled(!on)`.
|
||||
- Listen for Android lifecycle background/foreground signals (via a bridge event
|
||||
or existing visibility hooks) and:
|
||||
- background + ON → `player_enter_background_audio(...)`, pause + tear down the
|
||||
`<video>`/HLS decode (reuse the existing HLS teardown sequence to avoid dual
|
||||
audio).
|
||||
- foreground + ON → `player_exit_background_audio()`, reload `<video>` at the
|
||||
returned position, restore play/pause state.
|
||||
- **Follow the native-mode pitfall** (memory:
|
||||
`videoplayer-native-mode-pitfalls`): no lifecycle calls after an `await` in
|
||||
`onMount`. Keep the handoff logic out of that window.
|
||||
- Dual-audio is the key regression risk: at every handoff exactly one of
|
||||
{WebView `<video>`, native ExoPlayer} produces audio. Tear the other down
|
||||
*before* starting the next, mirroring the existing HLS cleanup discipline.
|
||||
|
||||
## Phasing
|
||||
|
||||
- **Phase 0 — De-risk (do first):**
|
||||
- Confirm what actually triggers today's auto-PiP given video is on the
|
||||
WebView (resolve the `canEnterPip`/`isPlayingVideo` question).
|
||||
- Spike: obtain an audio-only stream URL for a video item and play it through
|
||||
the native audio service; measure position accuracy and that WebView audio
|
||||
is fully silenced (no dual audio).
|
||||
- **Phase 1 — Backend:** repository audio-only-URL method + the two player
|
||||
commands + events.
|
||||
- **Phase 2 — Native:** lifecycle wiring, PiP suppression, handoff trigger.
|
||||
- **Phase 3 — Frontend:** toggle UI, lifecycle listeners, handoff calls,
|
||||
teardown discipline.
|
||||
- **Phase 4 — Polish:** persist toggle preference, subtitle/audio-track
|
||||
carry-over, edge cases (calls, headphone unplug, autoplay-next during
|
||||
background audio).
|
||||
|
||||
## Testing
|
||||
|
||||
- Rust: unit tests for the audio-only URL builder and the two commands
|
||||
(`cargo test`, `bun run test:rust`).
|
||||
- IPC param-naming integration tests for any new commands
|
||||
(`bun run test -- tauriIntegration.test.ts`).
|
||||
- Frontend: `bun run check`, `bun run test`, plus a VideoPlayer logic test for
|
||||
the handoff state machine (mirror the existing
|
||||
`VideoPlayer.logic.test.ts`).
|
||||
- Manual on-device matrix:
|
||||
- toggle ON: home button → audio continues, video stops decoding; return →
|
||||
video resumes at position; playing/paused preserved.
|
||||
- toggle ON: screen lock → audio continues; lockscreen controls work; unlock →
|
||||
resumes.
|
||||
- toggle OFF: background → PiP (unchanged).
|
||||
- No dual audio at any transition. No audio leak after leaving the player.
|
||||
- Transcoded (HEVC/10-bit) item — verify position with `seekOffset`.
|
||||
- Autoplay-next fires correctly if an episode ends during background audio.
|
||||
|
||||
## Open questions
|
||||
|
||||
1. **Persist the toggle per user/series, or default OFF each session?**
|
||||
(Recommend: remember last choice; series-level like the audio-track
|
||||
preference is a nice-to-have.)
|
||||
2. **Autoplay-next during background audio** — should the next episode start as
|
||||
audio-only and stay audio until foreground, or pause at episode end? (Recommend:
|
||||
continue as audio-only.)
|
||||
3. **Subtitles** are irrelevant in audio-only mode but must restore on
|
||||
foreground — confirm they survive the `<video>` teardown/reload.
|
||||
4. Exact **Android lifecycle signal** for "screen locked" vs "app backgrounded"
|
||||
— `onUserLeaveHint` covers Home but not lock; may need a screen-off receiver.
|
||||
|
||||
## Non-goals (v1)
|
||||
|
||||
- Linux background audio (desktop windows keep running unfocused; low value).
|
||||
- Replacing or removing PiP — it stays as the toggle-OFF behavior.
|
||||
- Re-enabling the native ExoPlayer *video* surface path.
|
||||
Reference in New Issue
Block a user