dtourolle/jellytau
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases 38 Wiki Activity

feat(playback): let Rust decide what stream to play, and say so

Browse Source 
Playing a video meant asking the server to re-encode it, always. That
decision was made nowhere and written down nowhere, so whoever needed it
re-derived it downstream — the player worked out whether it had been handed
a playlist by looking for ".m3u8" in the URL, in two places. A viewer paid
for a transcode of a file their device could have played untouched, and the
app could not tell them which it was.

One negotiation now produces one self-describing StreamSelection — direct
play, remux or transcode; over a playlist, a plain HTTP file, or a local one
— and every renderer consumes that same answer.

Measured against the development server (Jellyfin 10.11.5), 400 items
sampled for codec mix and 40 put through a real PlaybackInfo negotiation
per profile:

  Linux / WebKitGTK (h264 only, 2ch)          3/40 —  7% direct play
  Android / ExoPlayer (hevc, ac3/eac3, 6ch)  34/40 — 85% direct play

The library is ~80% hevc, which is why the two diverge so hard. The payoff
is overwhelmingly Android, where 85% of plays were starting a transcode
nobody needed. Linux stays near 7% until libmpv decodes the picture — the
h264-only profile is a WebKitGTK constraint, not a JellyTau choice.

DR-219  StreamSelection: url + tagged Transport (hls/progressive/localFile)
        + PlaybackKind (directPlay/directStream/transcode) + the negotiated
        rendition + this source's ladder + a needs_transcoding flag derived
        in Rust so the rule is answered once. Both enums are serde-tagged
        so the frontend matches a discriminant, not a substring. The paths
        that never negotiate get the same shape from Rust rather than
        assembling one — media_local_selection for a downloaded file,
        LiveStreamInfo.transport for a live channel — so there is no second
        place where a transport is decided.

DR-220  The ceiling becomes two levels: a durable device default (Settings,
        persisted) and a per-playback override the in-player picker sets.
        The picker had called itself a "this film, this connection" control
        since it was written but wrote the process-wide default, so dropping
        one awkward film to 2 Mbps silently capped every video played
        afterwards for the rest of the process, with Settings still showing
        the old value. The override is cleared whenever playback moves to a
        new item, which stops it surviving into an autoplayed next episode.
        effective_streaming_quality() is the single resolution point.

DR-221  The quality picker is filled from what this media source can offer.
        Rust marks a rung exceeds_source when its ceiling is at or above the
        source's own bitrate — such a rung is another way to spell Original
        — and the frontend does not draw those. Original is never marked; a
        source whose bitrate the server does not report marks nothing, which
        keeps every rung offered.

DR-222  Direct play and direct stream are negotiated, with two client-side
        overrides on top because the server's answer is right about the file
        and wrong about what this app will do with it: undecodable audio
        (Jellyfin 10.11.5 honours a DirectPlayProfile's container and video
        codec but ignores its audio codec, so it offers direct play for an
        E-AC-3 track the webview renders in silence) and a viewer-pinned
        audio track the file does not default to. A direct stream is a remux
        and is deliberately not counted as transcoding.

DR-223  Dropped on measurement, not deferred. A master playlist from this
        server carries exactly one EXT-X-STREAM-INF: Jellyfin builds it from
        the single rendition the request asked for rather than publishing a
        ladder. So there is no adaptation for hls.js to be preserving and
        none mpv would lose — the claim that there was, in
        playback-backend-unification.md, does not hold. Recorded rather than
        deleted because it is a measurement: a server that does publish a
        ladder would change the answer.

DR-224  Every backend consumes the same selection. The queue item carries
        the transport, so player_seek_video picks its seek strategy from the
        backend's decision instead of the last stream_url.contains(".m3u8")
        in the codebase. Items queued by a path that never negotiated carry
        None and fall back to needs_transcoding, which is exact rather than
        a guess because every transcode this app requests is HLS (DR-140).

The frontend loader decision moves to streamTransport.ts so it can be
tested: the two cases that pin it are the ones that failed against the old
implementation — a progressive stream whose URL contains ".m3u8" must not
get an HLS loader, and an HLS stream whose URL contains none must.

Also verified the URL the direct-play branch builds actually serves playable
bytes: 206, video/mp4, valid ISO-BMFF, and a mid-file range works, so
seeking a direct play works.

The spec is folded into docs/architecture/{01,02,03} and deleted, per the
rule that docs/specs holds only work that has not shipped. DR-121 leaves
read-through-media-cache.md with a pointer; that spec keeps its capture half.

Not verified: real playback on a device. Direct play changes what actually
gets played, and neither fixtures nor curl prove the WebKitGTK and ExoPlayer
paths render it.
This commit is contained in:
dtourolle
2026-08-22 13:45:03 +02:00
parent 5fede123e7
commit 109700b949
45 changed files with 9744 additions and 6581 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/specs/README.md
+3 -4
View File
@@ -28,7 +28,7 @@ know how something *works*, read
**Next free requirement ids** (always re-check
[requirements.md](../requirements.md) before allocating): **UR-079**,
**IR-033**, **DR-225**. Three specs below suggested ids that have since been
**IR-033**, **DR-231**. Three specs below suggested ids that have since been
taken by other work; each carries a ⚠️ note at the top.
## Partially implemented
@@ -37,18 +37,17 @@ taken by other work; each carries a ⚠️ note at the top.
|---|---|---|
| [frontend-domain-model.md](frontend-domain-model.md) | Catalog surface: `MediaKind`, `from_jellyfin` isolated, ticks → ms | `primaryImageTag` → `imageId` (~30 sites); player/session/reporting tick math; `stream.type` |
| [libmpv2-migration.md](libmpv2-migration.md) | `LICENSE` | The `libmpv` → `libmpv2` crate swap |
| [read-through-media-cache.md](read-through-media-cache.md) | DR-126…128, DR-133…138 — cache entries *are* download rows; local playback of downloads | DR-121/122/124/125 — the player quality selector and the read-through capture |
| [read-through-media-cache.md](read-through-media-cache.md) | DR-126…128, DR-133…138 — cache entries *are* download rows; local playback of downloads | DR-122/124/125 — the read-through capture. DR-121 shipped as backend-owned stream selection and left this spec |
| [scoped-search-boundary-implementation.md](scoped-search-boundary-implementation.md) | Stage 1: `SearchScope` owned by Rust (DR-063…067) | Stage 2: result-side grouping (`GROUP_ITEM_TYPES` still in `searchScope.ts`) |
## Not started
| Spec | Blocked on / note |
|---|---|
| [backend-owned-stream-selection.md](backend-owned-stream-selection.md) | Rust owns direct-play-vs-transcode, transport and quality; players consume one `StreamSelection`. Phase 1 (delete the `.m3u8` sniff) stands alone. Unblocks Linux native video. |
| [build-provenance.md](build-provenance.md) | `build.rs` is still bare. ⚠️ suggested id DR-093 is taken. |
| [player-facade-enforcement.md](player-facade-enforcement.md) | ~60 `commands.player*` sites still outside the facade; no lint rule. ⚠️ suggested id DR-095 is taken. |
| [windows-native-audio-backend.md](windows-native-audio-backend.md) | Blocked on the libmpv2 swap. ⚠️ suggested id IR-030 is taken. |
| [linux-native-video-spike.md](linux-native-video-spike.md) | **Spike run 2026-08-21: compositing works on Linux, X11 and Wayland.** G1-G6 green bar the Tauri `default_vbox()` half of G1. Needs an implementation spec that answers adaptive bitrate. |
| [linux-native-video-spike.md](linux-native-video-spike.md) | **Spike run 2026-08-21: compositing works on Linux, X11 and Wayland.** G1-G6 green bar the Tauri `default_vbox()` half of G1. The adaptive-bitrate question it was waiting on is **answered**: the server publishes one `EXT-X-STREAM-INF`, so there is no ladder for mpv to lose (DR-228). `StreamSelection` (DR-224) is the contract to consume. |
## Design authority
docs/specs/backend-owned-stream-selection.md
-242
View File
@@ -1,242 +0,0 @@
# Spec: Backend-owned stream selection
**Status:** Proposed
**Requirements:** UR-079 (new) → DR-219 … DR-224 (new); **implements and extends
DR-121**, currently allocated to
[read-through-media-cache.md](read-through-media-cache.md) and not started.
Re-check `requirements.md` before allocating — the ids moved twice while this was
being written (`DR` max was 215, then 218).
**UX spec:** the quality selector in `VideoPlayer.svelte` already exists; this
changes what fills it, not how it looks.
**Supersedes / revises:** takes DR-121 out of
[read-through-media-cache.md](read-through-media-cache.md), which should keep
only its capture/eviction half. Unblocks
[linux-native-video-spike.md](linux-native-video-spike.md).
**Destination on completion:**
[01-rust-backend.md](../architecture/01-rust-backend.md) — extends the
"Streaming quality ladder" section; and
[03-data-flow.md](../architecture/03-data-flow.md) — playback initiation. The
durable half is the layer line and the `StreamSelection` contract; phases and
acceptance criteria are disposable.
## Summary
Make Rust the single owner of *which stream to play* — direct play or transcode,
at what ceiling, over what transport — and hand every player backend a
self-describing selection instead of a bare URL. mpv, ExoPlayer and the HTML5
`<video>`/hls.js path all become consumers of the same decision rather than three
places that re-derive it.
Nothing about how playback *looks* changes. What changes is that the frontend
stops inferring transport from a URL string, and that direct play becomes
possible at all.
## Motivation
Four concrete problems, all the same shape.
**1. The frontend sniffs transport out of the URL.**
[VideoPlayer.svelte:569](../../src/lib/components/player/VideoPlayer.svelte#L569):
```ts
const isHlsStream = currentStreamUrl.includes(".m3u8");
```
and again inline at line 2364. Rust *built* that URL and knows exactly what it
is; the frontend re-derives it by substring match. Change the endpoint, add a DASH
path, serve a progressive file, and this silently picks wrong. This is the
boundary rule in miniature — not item-type taxonomy, but the same error: a
domain fact reconstructed in the presentation layer because the wire shape did
not carry it.
**2. There is no direct-play path.** `get_video_stream_url` always builds an HLS
transcode URL (`TranscodingProtocol=hls`, `VideoCodec=h264` first). Every video
play burns server CPU, even when the file would play untouched. This is the cost
the Linux native-video work exists to remove, and it cannot be removed without a
decision that does not currently exist anywhere in the codebase.
**3. Quality is a process-wide global.** `streaming_quality()` /
`set_streaming_quality()` in `repository/online.rs` read and write a static.
It is not per-session or per-item, so it cannot express "this 4K remux needs a
ceiling, that podcast does not", and two concurrent playbacks would share one
setting.
**4. Rust cannot say what qualities *this* media source supports.** The selector
is populated from a fixed enum rather than from what the source actually offers.
DR-121 already names this; it has not been built.
### The prior question
Finding 3 of [playback-backend-unification.md](playback-backend-unification.md)
holds that hls.js gives us real adaptive bitrate and mpv would lose it. Evidence
in this repo suggests **there is no ABR today**: a single rendition is requested,
no level-handling code exists anywhere in the frontend, and a quality switch is
implemented by re-opening the stream.
**Run this before sizing the adaptation work.** It needs a live server:
```
curl -s "https://<server>/Videos/<itemId>/master.m3u8?api_key=<key>&…" \
| grep -c EXT-X-STREAM-INF
```
`1` → there is no adaptation to preserve, and the adaptation half of this spec
collapses to "pick well at open". `>1` → finding 3 stands and DR-223 applies.
**Everything else in this spec is worth doing either way** — the ownership
problems above are independent of the answer.
## Layer assignment
| Logic / responsibility | Layer | Why it belongs there |
|---|---|---|
| Direct play vs direct stream vs transcode | Rust | Depends on Jellyfin's `PlaybackInfo`, container/codec support and the device profile. Changes when Jellyfin's API or our profile changes → domain, by the litmus test. |
| Transport of the chosen stream (HLS / progressive / local file) | Rust | Rust constructs the URL; it is the only place that *knows* rather than infers. Today the frontend guesses from `.m3u8`. |
| Which qualities this media source can offer | Rust | Derived from the source's own streams and the quality→transcode-parameter mapping that `get_video_download_url` already holds. DR-121. |
| The quality ceiling in force, per playback session | Rust | Domain state that outlives any one view and must survive a backend swap or a mode transfer. Currently a process-wide static. |
| Deciding to re-negotiate mid-playback (if adaptation is needed) | Rust | It performs the HTTP and already derives reachability from real traffic via `ConnectivityMonitor`. Throughput estimation is the same pattern on the same data — a side-channel probe would repeat the mistake that principle exists to prevent. |
| Frame-level delivery *within* the selected stream, including a player's own ABR | **Player** | ExoPlayer has genuine adaptive selection; if Rust hands it a multi-variant playlist it should use it. Rust chooses *what to request*, never how a player paces bytes. See "The line". |
| Rendering the selector, showing the current quality, ordering the list | Frontend | Pure presentation over a backend-supplied list. |
| Poster, letterbox, controls, overlay z-order | Frontend | Unchanged. |
### The line
**Rust decides *what stream*. The player decides *how to deliver it*.**
This matters most for ExoPlayer, which already does real adaptive track selection
over HLS. This spec must not reimplement that or fight it — if a multi-variant
playlist reaches ExoPlayer, ExoPlayer adapts and Rust stays out of the way. The
same restraint applies to any future backend that gains the capability. Rust only
steps in where the player has no such ability (mpv) *and* the server actually
offers a ladder.
Borderline row, with its tie-breaker: "which media source of a multi-source item"
looks like a user choice, and its *presentation* is. The default and the
constraint set are domain → **Rust**, per the borderline-defaults-to-Rust rule.
## Design
### The contract
One self-describing selection replaces the bare URL. Nested fields are
camelCase over the wire (`#[serde(rename_all = "camelCase")]`); the enums are
tagged so the frontend matches a tag instead of parsing a string.
```rust
#[derive(Serialize, Type)]
#[serde(rename_all = "camelCase")]
pub struct StreamSelection {
pub url: String,
pub transport: Transport,
pub playback_kind: PlaybackKind,
/// The negotiated rendition; None when direct-playing the source as-is.
pub rendition: Option<Rendition>,
/// What this media source can offer — fills the selector (DR-121).
pub available: Vec<QualityOption>,
}
#[derive(Serialize, Type)]
#[serde(tag = "type", rename_all = "camelCase")]
pub enum Transport { Hls, Progressive, LocalFile }
#[derive(Serialize, Type)]
#[serde(tag = "type", rename_all = "camelCase")]
pub enum PlaybackKind { DirectPlay, DirectStream, Transcode }
```
`Transport` is the field that deletes the `.m3u8` sniff. The frontend picks
hls.js on `Hls` and the element's own loader otherwise — a tag match, not a
substring search.
### Re-negotiation
Rust emits `stream-selection-changed` (kebab-case, per convention) carrying a new
`StreamSelection` plus the position to resume at. The existing
`playerSetStreamQuality` response already has exactly the right shape — a tagged
`strategy` that tells the caller who reloads, with the backend handling native
itself and handing HTML5 a URL for `reloadSource`
([index.ts:198](../../src/lib/player/index.ts#L198)). **Extend that; do not
invent a second mechanism.** It is the one piece of this that is already right.
Note the existing wart to preserve or fix deliberately, not accidentally:
tauri-specta keeps those response fields snake_case (`new_url`), and the facade
comments say so.
### Phases
1. **DR-219** `StreamSelection` + `Transport`; delete the `.m3u8` sniff. No
behaviour change — pure ownership move, and independently shippable.
2. **DR-220** Per-session quality ceiling replacing the `online.rs` static.
3. **DR-221** `available` populated from the media source (DR-121's substance).
4. **DR-222** Direct-play/direct-stream negotiation via `PlaybackInfo`. This is
the phase that unlocks native video and removes the transcode.
5. **DR-223** Adaptation, **only if the playlist check says a ladder exists**.
Cheapest sufficient design: re-negotiate on sustained throughput drop, reusing
the phase-1 re-negotiation path. A local proxy synthesizing a single-variant
playlist is a last resort, not a starting point.
6. **DR-224** ExoPlayer and mpv consume `StreamSelection` unchanged, proving the
contract is player-agnostic rather than HTML5-shaped.
Phases 1–4 stand on their own merits with no dependency on the ladder question.
## Out of scope
- Rendering, compositing, and the Linux native-video work itself. This spec
unblocks [linux-native-video-spike.md](linux-native-video-spike.md); it does
not contain it.
- Replacing hls.js. It stays as the HLS loader for the webview path.
- Reimplementing or overriding ExoPlayer's own adaptive selection. See "The line".
- The download/capture half of [read-through-media-cache.md](read-through-media-cache.md)
(DR-122, DR-124, DR-125), which keeps its own spec.
- Audio. The same argument applies, but video is where the transcode cost is.
## Acceptance criteria
- [ ] The `.m3u8` substring check is gone from `VideoPlayer.svelte` (both sites)
and transport comes from the tagged enum.
- [ ] `bun run check`, `bun run test`, `bun run format:check`, `bun run lint` pass.
- [ ] `cargo fmt` clean, `cargo clippy -D warnings` clean, `bun run test:rust` passes.
- [ ] `bun run check:boundary` passes — and the reviewer confirms by reading that
no transport/kind decision was reconstructed in `src/`, since the tripwire
only catches item-type array literals.
- [ ] `bindings.ts` regenerated from Rust, not hand-edited.
- [ ] New code carries `// TRACES:` comments; `bun run traces:validate` passes and
coverage stays ≥ the CI ratchet.
- [ ] The `EXT-X-STREAM-INF` count is recorded in this spec before DR-223 is
started or dropped.
- [ ] DR-121 is removed from `read-through-media-cache.md` with a pointer here.
## Testing
- Rust: `PlaybackInfo` fixtures → expected `PlaybackKind`, one per branch
(supported container direct-plays; unsupported codec transcodes; a ceiling
below the source bitrate transcodes even when the codec is fine).
- Rust: `Transport` round-trips through serde with the tag the frontend matches.
- Frontend: adapter selection driven by `transport`, including the case a URL
ending `.m3u8` is served as `Progressive` — that test fails on today's code,
which is the point.
- Extend `tauriIntegration.test.ts` for the new command params (camelCase rule).
- No test asserts a URL substring.
## TRACES
| Piece | Tag |
|---|---|
| `StreamSelection` / `Transport` | `UR-079 \| DR-219` |
| Per-session ceiling | `UR-074 \| DR-220` |
| `available` from media source | `UR-079 \| DR-221, DR-121` |
| Direct-play negotiation | `UR-079 \| DR-222` |
| Adaptation, if built | `UR-079 \| DR-223` |
| ExoPlayer/mpv consumers | `UR-003, UR-004 \| DR-224` |
## Notes for the implementer
- **Phase 1 is worth doing on its own**, even if everything after it is dropped.
It removes a real leak and costs almost nothing.
- Do not frame any phase as "no Rust changes required" — that framing is what
produced the leak `scoped-search-boundary.md` records.
- `ConnectivityMonitor` is the precedent for DR-223: derive network facts from
real traffic, never from a side-channel poller.
- A parallel Claude session may be active in this repo — `git diff` before
"repairing" unexpected changes. Requirement ids in particular moved twice
during the writing of this spec.
docs/specs/read-through-media-cache.md
+22 -26
View File
@@ -4,15 +4,20 @@
(DR-126, DR-127 — a cache entry *is* a `downloads` row with a shorter life, and
eviction only reclaims the temporary tier), local playback of downloaded media
(DR-128), and the one-path/one-row invariants that followed (DR-133 … DR-138).
DR-123 is in progress. Still open: the **player quality selector** and the
read-through capture itself — DR-121, DR-122, DR-124, DR-125. The separate
settings-level bitrate cap (DR-162, shipped —
[01-rust-backend.md](../architecture/01-rust-backend.md#streaming-quality-ladder))
covers a *settings-level*
ceiling (DR-162), which serves part of UR-070 but is not the per-playback
selector specified here.
**Requirements:** UR-070, UR-071 → DR-121, DR-122, DR-123, DR-124, DR-125; IR-032
**UX spec:** player quality selector — needs a `ux-flows.md` section before build
DR-123 is in progress. Still open: the read-through capture itself — DR-122,
DR-124, DR-125.
**DR-121 has shipped and left this spec.** The player quality selector, the
per-playback bitrate ceiling, and the backend-owned stream decision it needed
were built as *backend-owned stream selection* (DR-224 … DR-227) and are
described in
[01-rust-backend.md](../architecture/01-rust-backend.md#stream-selection) and
[03-data-flow.md](../architecture/03-data-flow.md#video-stream-selection-flow).
The settings-level ceiling (DR-162) is the same section. What remains here is the
*capture* half only — this spec no longer specifies anything about choosing a
bitrate.
**Requirements:** UR-070, UR-071 → DR-122, DR-123, DR-124, DR-125; IR-032
**Related:** the locally-indexed search and downloaded-browse work, both
shipped — see
[03-data-flow.md](../architecture/03-data-flow.md) and
@@ -70,24 +75,16 @@ frontend stores the user's *choice*; Rust decides what that choice resolves to.
## Design
### DR-121 — Bitrate selection in the player
### DR-121 — moved out (shipped)
The player exposes the qualities Rust reports for the current item. Changing it
re-negotiates the stream URL at the new quality and resumes at the current
position. This is a deliberate, user-initiated interruption — a brief rebuffer is
expected and acceptable, unlike the involuntary swap the earlier design would
have needed.
Bitrate selection in the player shipped as DR-224 … DR-227; see
[01-rust-backend.md](../architecture/01-rust-backend.md#stream-selection).
Constraints that must not be broken:
- On Linux, video playback must keep using the HLS `master.m3u8` URL. CLAUDE.md
records that returning `stream.mp4` means transcoded playback never starts.
A quality change re-negotiates *within* HLS.
- The quality→transcode-parameter mapping already exists in
`get_video_download_url` ([online.rs:1702-1717](../../src-tauri/src/repository/online.rs#L1702-L1717)).
Playback must call into the same mapping. Two copies of that table will drift.
- Track selection (audio/subtitle) already survives a stream re-negotiation
elsewhere in the player; a quality change must preserve it too.
The one constraint here that the capture work still has to respect: a quality
change re-negotiates **within HLS**. Returning a progressive `stream.mp4` for a
transcode means playback never starts, because the server encodes the whole file
before serving a byte (DR-140). That is why DR-122 below abandons a capture on a
quality change rather than trying to splice one.
### DR-122 — The playback path is ephemeral
@@ -212,7 +209,6 @@ codec taxonomy in `src/`; the selector's remembered choice is a view preference.
| Piece | Tag |
|---|---|
| Quality selector + re-negotiation | `// TRACES: UR-070 \| DR-121` |
| Ephemeral playback / capture abandonment | `// TRACES: UR-070 \| DR-122` |
| Independent whole-file download + local video playback fix | `// TRACES: UR-071 \| DR-123, IR-032` |
| ExoPlayer cache / mpv stream-record / keepability | `// TRACES: UR-071 \| DR-124` |