feat(audio): align a fetched manifest to the local file before storing
The signature had a producer and a reader but no consumer, so nothing ever fingerprinted anything. `ManifestAligner` runs on the fetch path, before the windows are stored. A local alignment supersedes the server's offset. The server has never seen this file — its offset is a runtime-difference inference at best, while the local comparison is against the media the windows will actually be drawn over. It also needs no round trip, so no signature leaves the instance. This is what jRay's spec already meant by matching being a consumer concern: the server never rewrites a manifest, so one stored manifest serves every trim of the same cut. The offset has two terms and only one is in the server's pseudocode. Both windows are centred on their own file's midpoint, so unequal runtimes start them at different absolute times; a release with 40 s of extra head material recovers 20 s from the slide and 20 s from the anchor difference. Using the slide alone is wrong by half the runtime difference on every shifted release. Degradation, never failure. Signatures off, no manifest signature, media under the window, a `v2:` producer, a missing binary, a decode error — each applies the server's offset rather than refusing, because a signature is an enhancement to cut matching and must never break a fetch. "Un-comparable" and "does not match" are kept distinct, which a test caught: `Compare` returns null for both, and conflating them would report a 90-second extra as content disagreeing with its own manifest. A genuine disagreement is stored anyway — the audio may legitimately differ, a different language track being the obvious case — and surfaced as a caveat that outranks the tier's, since it is the stronger statement. The applied offset, score, slide and the local file's own signature are written beside the truth file: the offset is otherwise unrecoverable once the windows are shifted, and the stored signature lets a later fetch align without decoding again. Provenance is never injected into the truth file, so the bytes served back stay the producer's (JR-004). `docs/audio-alignment.md` documents the mechanism end to end. TRACES: JR-047 | SR-003
This commit is contained in:
@@ -0,0 +1,205 @@
|
||||
using System;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
using Jellyfin.Plugin.JRay.Configuration;
|
||||
using Jellyfin.Plugin.JRay.Models;
|
||||
using Microsoft.Extensions.Logging;
|
||||
|
||||
namespace Jellyfin.Plugin.JRay.Services;
|
||||
|
||||
/// <summary>
|
||||
/// Aligns a freshly fetched manifest to the local file before its windows are
|
||||
/// stored.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <b>Why the client re-derives an offset the server already sent.</b> The
|
||||
/// server has never seen the local file. Its offset is a claim about a runtime
|
||||
/// it was told, so it can only ever be a runtime-difference inference. A local
|
||||
/// alignment compares the manifest's own audio signature against the file the
|
||||
/// windows will actually be drawn over, which is the authoritative comparison —
|
||||
/// and it needs no round trip, so no signature leaves the instance. Where the
|
||||
/// two disagree, the local one wins.
|
||||
/// <para>
|
||||
/// This is what jRay's specification means by matching being "a consumer
|
||||
/// concern": the server never rewrites a manifest, so one stored manifest serves
|
||||
/// every trim of the same cut, and each client shifts it to its own timebase
|
||||
/// (JR-030).
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Degradation, not failure.</b> Signatures switched off, an item under the
|
||||
/// 120 s window, a manifest with no signature, a missing FFmpeg binary, a decode
|
||||
/// error, or two signatures that simply do not match — every one of these falls
|
||||
/// back to the server's offset. A signature is an enhancement to cut matching; a
|
||||
/// missing one costs a tier and must never be able to break a fetch.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
// TRACES: JR-047 | SR-003
|
||||
public class ManifestAligner
|
||||
{
|
||||
private readonly AudioSignatureService _signatures;
|
||||
private readonly ILogger<ManifestAligner> _logger;
|
||||
|
||||
/// <summary>
|
||||
/// Initializes a new instance of the <see cref="ManifestAligner"/> class.
|
||||
/// </summary>
|
||||
/// <param name="signatures">Computes the local file's audio signature.</param>
|
||||
/// <param name="logger">Logger.</param>
|
||||
public ManifestAligner(AudioSignatureService signatures, ILogger<ManifestAligner> logger)
|
||||
{
|
||||
_signatures = signatures;
|
||||
_logger = logger;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Decides which offset to apply, given a local signature that has already
|
||||
/// been computed.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Split from <see cref="AlignAsync"/> so the decision is testable without
|
||||
/// an FFmpeg binary or a media file: everything interesting happens here,
|
||||
/// and the caller only supplies the two strings.
|
||||
/// </remarks>
|
||||
/// <param name="localSignature">The local file's signature, or <c>null</c>.</param>
|
||||
/// <param name="manifestSignature">The manifest's signature, or <c>null</c>.</param>
|
||||
/// <param name="localRuntimeSec">Local file runtime, in seconds.</param>
|
||||
/// <param name="manifestRuntimeSec">Runtime the manifest records, in seconds.</param>
|
||||
/// <param name="serverTier">The tier the server reported.</param>
|
||||
/// <param name="serverOffsetSec">The offset the server reported.</param>
|
||||
/// <returns>What to apply, and how it was decided.</returns>
|
||||
public static TruthAlignment Resolve(
|
||||
string? localSignature,
|
||||
string? manifestSignature,
|
||||
double localRuntimeSec,
|
||||
double manifestRuntimeSec,
|
||||
MatchTier serverTier,
|
||||
double serverOffsetSec)
|
||||
{
|
||||
var alignment = new TruthAlignment
|
||||
{
|
||||
Source = AlignmentSource.Server,
|
||||
Tier = serverTier,
|
||||
OffsetSec = serverOffsetSec,
|
||||
ServerTier = serverTier,
|
||||
ServerOffsetSec = serverOffsetSec,
|
||||
LocalSignature = localSignature,
|
||||
};
|
||||
|
||||
// "No alignment was possible" and "the audio does not match" are very
|
||||
// different things to tell someone, and `Compare` returns null for both.
|
||||
// A comparison is only possible when both items clear the 120 s window
|
||||
// (JR-044) and both signatures parse as v1 (JR-045) — a `v2:` signature
|
||||
// from a future producer is un-comparable, not a mismatch. Separating
|
||||
// them here is what keeps a 90-second extra from being reported as
|
||||
// content that disagrees with its own manifest.
|
||||
var comparable = localSignature is not null
|
||||
&& manifestSignature is not null
|
||||
&& localRuntimeSec >= AudioSignature.WindowSec
|
||||
&& manifestRuntimeSec >= AudioSignature.WindowSec
|
||||
&& AudioSignatureMatcher.TryParseFrames(localSignature) is not null
|
||||
&& AudioSignatureMatcher.TryParseFrames(manifestSignature) is not null;
|
||||
|
||||
if (!comparable)
|
||||
{
|
||||
return alignment;
|
||||
}
|
||||
|
||||
var match = AudioSignatureMatcher.Compare(
|
||||
localSignature, manifestSignature, localRuntimeSec, manifestRuntimeSec);
|
||||
|
||||
if (match is null)
|
||||
{
|
||||
// Both sides had a signature and they did not align at any tier. The
|
||||
// server's offset still applies — the audio may legitimately differ,
|
||||
// and a signature must not break a fetch — but this is the strongest
|
||||
// available hint that the manifest describes different content, so
|
||||
// it is recorded rather than silently discarded.
|
||||
alignment.Source = AlignmentSource.LocalMismatch;
|
||||
return alignment;
|
||||
}
|
||||
|
||||
alignment.Source = AlignmentSource.Local;
|
||||
alignment.Tier = match.Value.Tier;
|
||||
alignment.OffsetSec = match.Value.OffsetSec;
|
||||
alignment.Score = match.Value.Score;
|
||||
alignment.OffsetFrames = match.Value.OffsetFrames;
|
||||
return alignment;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Computes the local file's signature if it can, then resolves the
|
||||
/// alignment.
|
||||
/// </summary>
|
||||
/// <param name="mediaPath">Path of the local media file.</param>
|
||||
/// <param name="localRuntimeSec">Local file runtime, in seconds.</param>
|
||||
/// <param name="manifest">The fetched manifest.</param>
|
||||
/// <param name="serverTier">The tier the server reported.</param>
|
||||
/// <param name="serverOffsetSec">The offset the server reported.</param>
|
||||
/// <param name="computeSignatures">Whether signatures are enabled in configuration.</param>
|
||||
/// <param name="cancellationToken">Cancellation token.</param>
|
||||
/// <returns>What to apply, and how it was decided.</returns>
|
||||
public async Task<TruthAlignment> AlignAsync(
|
||||
string mediaPath,
|
||||
double localRuntimeSec,
|
||||
Jmanifest manifest,
|
||||
MatchTier serverTier,
|
||||
double serverOffsetSec,
|
||||
bool computeSignatures,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(manifest);
|
||||
|
||||
var manifestSignature = manifest.Cut?.AudioSignature;
|
||||
string? localSignature = null;
|
||||
|
||||
// The decode is the expensive half, so it is skipped outright when it
|
||||
// could not change the answer: no manifest signature to compare against
|
||||
// means no local alignment is possible.
|
||||
if (computeSignatures && manifestSignature is not null && !string.IsNullOrEmpty(mediaPath))
|
||||
{
|
||||
localSignature = await _signatures
|
||||
.ComputeAsync(mediaPath, localRuntimeSec, cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
}
|
||||
|
||||
var alignment = Resolve(
|
||||
localSignature,
|
||||
manifestSignature,
|
||||
localRuntimeSec,
|
||||
manifest.Cut?.RuntimeSec ?? 0.0,
|
||||
serverTier,
|
||||
serverOffsetSec);
|
||||
|
||||
switch (alignment.Source)
|
||||
{
|
||||
case AlignmentSource.Local:
|
||||
_logger.LogInformation(
|
||||
"Local audio alignment for {Path}: score {Score:F3}, {Frames} frames, offset {Offset:F3}s at tier {Tier} (server said {ServerOffset:F3}s at {ServerTier})",
|
||||
mediaPath,
|
||||
alignment.Score,
|
||||
alignment.OffsetFrames,
|
||||
alignment.OffsetSec,
|
||||
alignment.Tier,
|
||||
alignment.ServerOffsetSec,
|
||||
alignment.ServerTier);
|
||||
break;
|
||||
|
||||
case AlignmentSource.LocalMismatch:
|
||||
_logger.LogWarning(
|
||||
"Audio signatures for {Path} did not align with the fetched manifest; applying the server's {Offset:F3}s at {Tier}. This may be a different cut, a different language track, or a heavy re-encode.",
|
||||
mediaPath,
|
||||
alignment.OffsetSec,
|
||||
alignment.Tier);
|
||||
break;
|
||||
|
||||
default:
|
||||
_logger.LogDebug(
|
||||
"No local alignment for {Path}; applying the server's {Offset:F3}s at {Tier}",
|
||||
mediaPath,
|
||||
alignment.OffsetSec,
|
||||
alignment.Tier);
|
||||
break;
|
||||
}
|
||||
|
||||
return alignment;
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user