feat(audio): align a fetched manifest to the local file before storing

The signature had a producer and a reader but no consumer, so nothing
ever fingerprinted anything. `ManifestAligner` runs on the fetch path,
before the windows are stored.

A local alignment supersedes the server's offset. The server has never
seen this file — its offset is a runtime-difference inference at best,
while the local comparison is against the media the windows will actually
be drawn over. It also needs no round trip, so no signature leaves the
instance. This is what jRay's spec already meant by matching being a
consumer concern: the server never rewrites a manifest, so one stored
manifest serves every trim of the same cut.

The offset has two terms and only one is in the server's pseudocode. Both
windows are centred on their own file's midpoint, so unequal runtimes
start them at different absolute times; a release with 40 s of extra head
material recovers 20 s from the slide and 20 s from the anchor
difference. Using the slide alone is wrong by half the runtime difference
on every shifted release.

Degradation, never failure. Signatures off, no manifest signature, media
under the window, a `v2:` producer, a missing binary, a decode error —
each applies the server's offset rather than refusing, because a
signature is an enhancement to cut matching and must never break a fetch.

"Un-comparable" and "does not match" are kept distinct, which a test
caught: `Compare` returns null for both, and conflating them would report
a 90-second extra as content disagreeing with its own manifest. A genuine
disagreement is stored anyway — the audio may legitimately differ, a
different language track being the obvious case — and surfaced as a
caveat that outranks the tier's, since it is the stronger statement.

The applied offset, score, slide and the local file's own signature are
written beside the truth file: the offset is otherwise unrecoverable once
the windows are shifted, and the stored signature lets a later fetch
align without decoding again. Provenance is never injected into the truth
file, so the bytes served back stay the producer's (JR-004).

`docs/audio-alignment.md` documents the mechanism end to end.

TRACES: JR-047 | SR-003
This commit is contained in:
2026-07-31 16:52:21 +02:00
parent e8ce779ad3
commit 1e247c4c7d
12 changed files with 981 additions and 29 deletions
@@ -0,0 +1,42 @@
namespace Jellyfin.Plugin.JRay.Models;
/// <summary>
/// Which comparison produced the offset that was applied to a fetched manifest.
/// </summary>
/// <remarks>
/// Recorded because the two are not equally strong evidence. The server has
/// never seen the local file, so its offset is at best a runtime-difference
/// inference; a local alignment compares the manifest's own audio signature
/// against the file the windows will actually be drawn over.
/// </remarks>
public enum AlignmentSource
{
/// <summary>
/// The server's offset was applied — no local alignment was possible.
/// </summary>
/// <remarks>
/// Either signatures are switched off, the item is under the 120 s window,
/// the manifest carried no signature, or the decode failed. A signature is
/// an enhancement, so every one of those degrades to this rather than
/// failing the fetch.
/// </remarks>
Server = 0,
/// <summary>
/// A local audio alignment was computed and its offset was applied.
/// </summary>
Local = 1,
/// <summary>
/// A local alignment was attempted and the two signatures did not match at
/// any tier; the server's offset was applied and the disagreement recorded.
/// </summary>
/// <remarks>
/// Deliberately not a failure. The audio may legitimately differ — a
/// different language track, a heavy re-encode — and a signature must never
/// be able to break a fetch. But it is the strongest available hint that a
/// manifest describes different content, so it is surfaced as a caveat
/// rather than discarded.
/// </remarks>
LocalMismatch = 2,
}
@@ -0,0 +1,74 @@
using System.Text.Json.Serialization;
using Jellyfin.Plugin.JRay.Configuration;
namespace Jellyfin.Plugin.JRay.Models;
/// <summary>
/// How a fetched manifest was aligned to the local file, recorded alongside the
/// truth data.
/// </summary>
/// <remarks>
/// The offset is applied once, at store time (JR-030), after which the stored
/// windows look native and nothing would say they had been shifted. This block
/// is what makes that reconstructable: which comparison produced the offset, how
/// strong it was, and the local file's own signature, so a later fetch can
/// re-align without decoding the media again.
/// </remarks>
// TRACES: JR-047 | SR-003
public class TruthAlignment
{
/// <summary>Gets or sets which comparison produced the applied offset.</summary>
[JsonPropertyName("source")]
public AlignmentSource Source { get; set; }
/// <summary>Gets or sets the tier that was applied.</summary>
[JsonPropertyName("tier")]
public MatchTier Tier { get; set; }
/// <summary>Gets or sets the offset applied to every window, in seconds.</summary>
[JsonPropertyName("offset_sec")]
public double OffsetSec { get; set; }
/// <summary>
/// Gets or sets the local alignment's score, or <c>null</c> when none was
/// computed.
/// </summary>
[JsonPropertyName("score")]
public double? Score { get; set; }
/// <summary>
/// Gets or sets the recovered slide between the two analysis windows, in
/// frames, or <c>null</c> when no local alignment was computed.
/// </summary>
/// <remarks>
/// Only part of <see cref="OffsetSec"/>: the rest comes from the two windows
/// being anchored at different points when the runtimes differ.
/// </remarks>
[JsonPropertyName("offset_frames")]
public int? OffsetFrames { get; set; }
/// <summary>
/// Gets or sets the local file's own <c>v1:</c> audio signature, or
/// <c>null</c> when none was computed.
/// </summary>
/// <remarks>
/// Kept so a later fetch can align against a new manifest without running
/// FFmpeg over the media again — the expensive half of the operation, and
/// the reason signatures are opt-in. It never leaves the instance: it is
/// stored beside the truth file, and contribution strips provenance
/// entirely (JR-034).
/// </remarks>
[JsonPropertyName("local_signature")]
public string? LocalSignature { get; set; }
/// <summary>
/// Gets or sets the offset the server claimed, retained even when a local
/// alignment superseded it.
/// </summary>
[JsonPropertyName("server_offset_sec")]
public double ServerOffsetSec { get; set; }
/// <summary>Gets or sets the tier the server claimed.</summary>
[JsonPropertyName("server_tier")]
public MatchTier ServerTier { get; set; }
}
@@ -74,6 +74,18 @@ public class TruthProvenance
[JsonPropertyName("offset_sec")]
public double OffsetSec { get; set; }
/// <summary>
/// Gets or sets how the applied offset was arrived at, or null for local
/// sources and for fetches made before alignment was recorded.
/// </summary>
/// <remarks>
/// <see cref="OffsetSec"/> says what was applied; this says why, and keeps
/// the local file's own signature so a later fetch can re-align without
/// decoding the media again (JR-047).
/// </remarks>
[JsonPropertyName("alignment")]
public TruthAlignment? Alignment { get; set; }
/// <summary>
/// Gets or sets a human-readable caveat to surface with the overlay, or
/// null when the claim needs none.