|
|
|
@@ -1,16 +1,16 @@
|
|
|
|
|
# Spec: Desktop native video — mpv renders the picture, everywhere
|
|
|
|
|
|
|
|
|
|
**Status:** Proposed
|
|
|
|
|
**Requirements:** UR-080 (new) → DR-230 … DR-236 (new); IR-033 (new)
|
|
|
|
|
**Requirements:** UR-080 (new) → DR-231 … DR-237 (new); IR-033 (new)
|
|
|
|
|
**UX spec:** n/a — nothing about the player's appearance changes. What changes is
|
|
|
|
|
what is behind the controls.
|
|
|
|
|
**Supersedes / revises:** consumes and closes
|
|
|
|
|
[linux-native-video-spike.md](linux-native-video-spike.md), whose gates
|
|
|
|
|
authorised exactly this spec and nothing more. Settles finding 2 of
|
|
|
|
|
[playback-backend-unification.md](playback-backend-unification.md) on the
|
|
|
|
|
desktop; finding 3 was already settled by DR-228. Absorbs the video half of what
|
|
|
|
|
desktop; finding 3 was already settled by DR-229. Absorbs the video half of what
|
|
|
|
|
[windows-native-audio-backend.md](windows-native-audio-backend.md) leaves open.
|
|
|
|
|
**Depends on:** backend-owned stream selection (DR-224 … DR-229), the branch
|
|
|
|
|
**Depends on:** backend-owned stream selection (DR-225 … DR-230), the branch
|
|
|
|
|
below this one. mpv is a *consumer* of `StreamSelection`, never a second place to
|
|
|
|
|
decide what to play.
|
|
|
|
|
|
|
|
|
@@ -42,7 +42,7 @@ profile therefore claims `h264` alone. That is not a statement about the machine
|
|
|
|
|
— the same machine runs mpv, which decodes essentially everything in the library
|
|
|
|
|
— it is a statement about which widget is holding the frame.
|
|
|
|
|
|
|
|
|
|
DR-227 made the cost measurable. Over 40 items negotiated against the development
|
|
|
|
|
DR-228 made the cost measurable. Over 40 items negotiated against the development
|
|
|
|
|
server:
|
|
|
|
|
|
|
|
|
|
| Profile | Direct play |
|
|
|
|
@@ -60,7 +60,7 @@ can actually do, and that — not the compositing — is the product.
|
|
|
|
|
> containing `ac3,eac3`. The Android device later used for verification reports
|
|
|
|
|
> neither in its `MediaCodecList` — no Dolby licence, normal for a tablet — so
|
|
|
|
|
> eac3 content, about a third of the sampled library, correctly transcodes there.
|
|
|
|
|
> Realising any of this depends on DR-233, deriving the profile from the renderer
|
|
|
|
|
> Realising any of this depends on DR-234, deriving the profile from the renderer
|
|
|
|
|
> rather than from the platform, which is why that requirement is load-bearing
|
|
|
|
|
> and not tidy-up.
|
|
|
|
|
|
|
|
|
@@ -86,10 +86,10 @@ follow-up that never gets written.
|
|
|
|
|
1. **Compositing works, including Wayland.** The spike ran all six gates; the
|
|
|
|
|
2024 "not possible on Wayland at all" claim is out of date when the render API
|
|
|
|
|
is used instead of foreign-window embedding.
|
|
|
|
|
2. **There is no ABR to lose.** DR-228: the server's master playlist carries one
|
|
|
|
|
2. **There is no ABR to lose.** DR-229: the server's master playlist carries one
|
|
|
|
|
`EXT-X-STREAM-INF`. hls.js was demuxing, not adapting.
|
|
|
|
|
3. **A direct-play path exists.** It did not when the spike was written. DR-227
|
|
|
|
|
built it; DR-229 proved the contract is player-agnostic.
|
|
|
|
|
3. **A direct-play path exists.** It did not when the spike was written. DR-228
|
|
|
|
|
built it; DR-230 proved the contract is player-agnostic.
|
|
|
|
|
|
|
|
|
|
And on Windows specifically, `tauri-plugin-libmpv` lists Windows as its **fully
|
|
|
|
|
tested** platform — the inverse of the Linux situation the spike had to
|
|
|
|
@@ -101,9 +101,9 @@ disprove. The embedding difficulty was always WebKitGTK-specific.
|
|
|
|
|
|---|---|---|
|
|
|
|
|
| **Which codecs this device can decode** | **Rust** | Domain: it is the input to Jellyfin's `PlaybackInfo` negotiation. It stops being a property of the *platform* and becomes a property of *the renderer in use* — see "The structural change". |
|
|
|
|
|
| Which backend renders video | **Rust** | Rust already owns this (`use_html5_element` / `VideoBackend`). It stops being a `cfg!` constant and becomes a runtime fact. |
|
|
|
|
|
| What stream to play (direct / remux / transcode, transport, ceiling) | **Rust — already decided** | DR-224. mpv consumes `StreamSelection`. Re-deriving any of it in a new backend would be the defect DR-224 exists to remove, restated. |
|
|
|
|
|
| What stream to play (direct / remux / transcode, transport, ceiling) | **Rust — already decided** | DR-225. mpv consumes `StreamSelection`. Re-deriving any of it in a new backend would be the defect DR-225 exists to remove, restated. |
|
|
|
|
|
| Creating the GL surface, reparenting the webview, owning the render context | **Rust (platform layer)** | Native window and GL-context lifetime. Not presentation, and not expressible above the IPC boundary at all. |
|
|
|
|
|
| Render-context ↔ GL-context lifetime binding | **Rust** | A correctness invariant over native resources. DR-231. |
|
|
|
|
|
| Render-context ↔ GL-context lifetime binding | **Rust** | A correctness invariant over native resources. DR-232. |
|
|
|
|
|
| Frame pacing (update callback, `report_swap`) | **Rust** | Timing against the compositor; mpv's own contract. |
|
|
|
|
|
| Hardware-decode selection | **Rust** | A capability question about the machine, answered from what mpv reports it actually selected. |
|
|
|
|
|
| Z-order of controls over video, overlay chrome, letterbox colour | **Frontend / mpv** | Presentation. Controls already draw over a transparent webview on Android; mpv paints its own letterbox bars (better than the Android equivalent, which shipped DR-194 as a defect). |
|
|
|
|
@@ -127,7 +127,7 @@ derived from **which renderer will decode this stream**, which is runtime state.
|
|
|
|
|
It looks like configuration and is not: it is the input that decides whether the
|
|
|
|
|
server re-encodes, it changes when Jellyfin's API or our renderer changes, and
|
|
|
|
|
getting it wrong fails *silently* — a claimed codec the renderer cannot decode is
|
|
|
|
|
a black picture or silence, which is DR-148 and DR-227's audio override already.
|
|
|
|
|
a black picture or silence, which is DR-148 and DR-228's audio override already.
|
|
|
|
|
|
|
|
|
|
**Write this against "the active video renderer", never `cfg!(target_os)`.** It
|
|
|
|
|
is the single piece that must not be Linux-shaped, because phase 2 reuses it
|
|
|
|
@@ -135,7 +135,7 @@ unchanged.
|
|
|
|
|
|
|
|
|
|
## Design
|
|
|
|
|
|
|
|
|
|
### Backend and compositing (DR-230, IR-033)
|
|
|
|
|
### Backend and compositing (DR-231, IR-033)
|
|
|
|
|
|
|
|
|
|
An `MpvVideoBackend` beside the existing `MpvBackend` (audio). The mpv side —
|
|
|
|
|
render context, FBO, update callback, hwdec — is **shared**; only the surface
|
|
|
|
@@ -168,9 +168,9 @@ which looks like a platform limitation and is not:
|
|
|
|
|
returning the symbol's own address makes mpv jump into non-executable data and
|
|
|
|
|
take SIGSEGV on the first GL call. The `epoxy` crate does this correctly but is
|
|
|
|
|
unusable — its `gl_generator` dependency pulls a yanked `xml-rs`.
|
|
|
|
|
3. **Frame pacing is not optional and its symptom misleads.** See DR-232.
|
|
|
|
|
3. **Frame pacing is not optional and its symptom misleads.** See DR-233.
|
|
|
|
|
|
|
|
|
|
### Render-context lifetime (DR-231) — the crash defence
|
|
|
|
|
### Render-context lifetime (DR-232) — the crash defence
|
|
|
|
|
|
|
|
|
|
The spike's one unexplained SIGSEGV landed in a *decoder* thread with no Tauri,
|
|
|
|
|
GTK or GL frame in the stack, and three plausible causes failed to reproduce it
|
|
|
|
@@ -193,7 +193,7 @@ reproduce:
|
|
|
|
|
If the crash recurs after this, it is a different bug and the likeliest cause is
|
|
|
|
|
out of the search space. If it does not, we needed this anyway.
|
|
|
|
|
|
|
|
|
|
### Frame pacing (DR-232)
|
|
|
|
|
### Frame pacing (DR-233)
|
|
|
|
|
|
|
|
|
|
Register `mpv_render_context_set_update_callback`; redraw only when it reports a
|
|
|
|
|
frame ready; call `mpv_render_context_report_swap` after each render.
|
|
|
|
@@ -203,7 +203,7 @@ frame clock every tick without reporting the swap leaves mpv nothing to time
|
|
|
|
|
against. It looks fine in a window and **judders at fullscreen**, which reads as
|
|
|
|
|
a compositing or GPU limit and is neither.
|
|
|
|
|
|
|
|
|
|
### Renderer-dependent device profile (DR-233)
|
|
|
|
|
### Renderer-dependent device profile (DR-234)
|
|
|
|
|
|
|
|
|
|
`build_device_profile` takes the active video renderer and derives the codec
|
|
|
|
|
lists from it:
|
|
|
|
@@ -222,9 +222,9 @@ constraints stay, sourced from the renderer rather than assumed.
|
|
|
|
|
|
|
|
|
|
**This is what converts the 7% figure upward** (toward, not necessarily to, the 85% ceiling — see the caveat above), and it is also the change most able to break
|
|
|
|
|
playback silently — so it lands after compositing is proven, covered by the
|
|
|
|
|
DR-227 override tests.
|
|
|
|
|
DR-228 override tests.
|
|
|
|
|
|
|
|
|
|
### Deleting the webview video path (DR-234)
|
|
|
|
|
### Deleting the webview video path (DR-235)
|
|
|
|
|
|
|
|
|
|
`get_player_status` stops reporting `use_html5_element: true` on desktop;
|
|
|
|
|
`supports_native_video` becomes true there.
|
|
|
|
@@ -251,7 +251,7 @@ backend, emit `backend-init-failed`, and surface a real error rather than a blac
|
|
|
|
|
rectangle. An honest failure beats a hidden downgrade to the transcode we are
|
|
|
|
|
trying to stop paying for.
|
|
|
|
|
|
|
|
|
|
### Hardware decode (DR-235)
|
|
|
|
|
### Hardware decode (DR-236)
|
|
|
|
|
|
|
|
|
|
The spike established the load-bearing fact: **hardware decode works through the
|
|
|
|
|
render API** (`hwdec-current` reported `nvdec-copy` on the discrete GPU), so the
|
|
|
|
@@ -270,12 +270,12 @@ asked for:
|
|
|
|
|
- Log `hwdec-current` at start-up; knowing what was actually chosen is the whole
|
|
|
|
|
diagnostic value.
|
|
|
|
|
|
|
|
|
|
### Windows: what phase 2 actually costs (DR-236)
|
|
|
|
|
### Windows: what phase 2 actually costs (DR-237)
|
|
|
|
|
|
|
|
|
|
Not hidden, because it is the part most likely to be underestimated:
|
|
|
|
|
|
|
|
|
|
- **The surface is different code.** WebView2 in an HWND, not GTK. A transparent
|
|
|
|
|
WebView2 over a native child window is a solved arrangement, but DR-230's
|
|
|
|
|
WebView2 over a native child window is a solved arrangement, but DR-231's
|
|
|
|
|
Linux surface does not transfer. Everything else does.
|
|
|
|
|
- **libmpv is currently a Linux-only dependency**, and Windows is
|
|
|
|
|
**cross-compiled from Linux** via `x86_64-pc-windows-msvc` + `cargo-xwin`. Phase
|
|
|
|
@@ -299,7 +299,7 @@ and shrinks to the surface.
|
|
|
|
|
- **Audio backends.** mpv already plays audio on Linux; this adds a video
|
|
|
|
|
renderer beside it. Windows audio is its own spec.
|
|
|
|
|
- **HDR, tone mapping, multi-window.** Not exercised by the spike at all.
|
|
|
|
|
- **Re-deciding what stream to play.** DR-224 owns that. If this spec finds
|
|
|
|
|
- **Re-deciding what stream to play.** DR-225 owns that. If this spec finds
|
|
|
|
|
itself choosing a URL, something has gone wrong.
|
|
|
|
|
|
|
|
|
|
## Acceptance criteria
|
|
|
|
@@ -353,7 +353,7 @@ and shrinks to the surface.
|
|
|
|
|
## Testing
|
|
|
|
|
|
|
|
|
|
- **Rust, pure:** the device profile per renderer — mpv claims hevc, the webview
|
|
|
|
|
does not, the multichannel bound survives both. The DR-233 table as a
|
|
|
|
|
does not, the multichannel bound survives both. The DR-234 table as a
|
|
|
|
|
table-driven test.
|
|
|
|
|
- **Rust, pure:** `PlaybackInfo` fixtures that transcode under the webview
|
|
|
|
|
profile and direct-play under the mpv profile — the direct-play conversion as a unit
|
|
|
|
@@ -370,13 +370,13 @@ and shrinks to the surface.
|
|
|
|
|
|
|
|
|
|
| Piece | Tag |
|
|
|
|
|
|---|---|
|
|
|
|
|
| mpv video backend + compositing | `UR-080 \| DR-230, IR-033` |
|
|
|
|
|
| Render-context lifetime binding | `UR-080 \| DR-231` |
|
|
|
|
|
| Frame pacing | `UR-080 \| DR-232` |
|
|
|
|
|
| Renderer-dependent device profile | `UR-080, UR-070 \| DR-233` |
|
|
|
|
|
| Webview video path removed | `UR-080 \| DR-234` |
|
|
|
|
|
| Hardware-decode policy | `UR-080 \| DR-235` |
|
|
|
|
|
| Windows surface + cross-build | `UR-080 \| DR-236` |
|
|
|
|
|
| mpv video backend + compositing | `UR-080 \| DR-231, IR-033` |
|
|
|
|
|
| Render-context lifetime binding | `UR-080 \| DR-232` |
|
|
|
|
|
| Frame pacing | `UR-080 \| DR-233` |
|
|
|
|
|
| Renderer-dependent device profile | `UR-080, UR-070 \| DR-234` |
|
|
|
|
|
| Webview video path removed | `UR-080 \| DR-235` |
|
|
|
|
|
| Hardware-decode policy | `UR-080 \| DR-236` |
|
|
|
|
|
| Windows surface + cross-build | `UR-080 \| DR-237` |
|
|
|
|
|
|
|
|
|
|
## Notes for the implementer
|
|
|
|
|
|
|
|
|
@@ -384,7 +384,7 @@ and shrinks to the surface.
|
|
|
|
|
hardware-decode table are the most valuable things in this directory, and each
|
|
|
|
|
cost a debugging cycle to find.
|
|
|
|
|
- **mpv consumes `StreamSelection`; it does not decide.** The transport is on the
|
|
|
|
|
queue item (DR-229). If you are parsing a URL, stop.
|
|
|
|
|
queue item (DR-230). If you are parsing a URL, stop.
|
|
|
|
|
- **Guard nothing on `cfg!(target_os = "linux")` that phase 2 will need.** That is
|
|
|
|
|
the one avoidable mistake here.
|
|
|
|
|
- The Android backend is the reference for the *shape* of this — transparent
|
|
|
|
|
