jRay/docs/audio-alignment.md

Audio signature and alignment — how it works

How a manifest fetched from a public server is checked against, and shifted onto, your own copy of a film.

Requirements: JR-042 … JR-045, JR-047. Construction is owned by JRay-public-server/SPEC.md §3; this document is the mechanism end to end.

---

The problem

A manifest says "Peter Capaldi is on screen from 42:10 to 42:38". Your file may not agree, because releases of the same film are trimmed differently — a distributor logo here, a longer certificate card there. Thirty seconds of extra head material makes every window in the manifest wrong by thirty seconds.

The old defence was the runtime tier: accept a manifest only if the runtimes agree within ±2 s. That rejects exactly the releases it should be fixing, and accepts anything that happens to be the same length.

A content-derived signature answers the question the exchange actually needs — do these timings apply to this media? — and, when the answer is "yes but shifted", says by how much.

Deliberately, it does not answer what file is this? There is no file hash anywhere in the system; the tier was withdrawn on legal grounds. The signature identifies a cut, so two different encodes of the same edit agree.

---

The signature

120 seconds from the centre of the media — the head and tail are the least content-specific parts of a release, being logos and credits.

decode centre window   runtime/2 ± 60 s
downmix                mono
resample               11025 Hz
STFT                   4096-sample frame, 1024-sample hop (~93 ms), Hann
band                   300–3000 Hz, 32 logarithmically spaced bands
per frame              peak band index (5 bits) + energy class (2 bits)
                       -> one byte, bit 7 always clear
result                 1288 bytes, base64, prefixed "v1:"

Peak bins are used because they survive lossy re-encoding, loudness normalisation and channel-layout differences, where absolute magnitudes do not. Same principle as Chromaprint/AcoustID, but self-contained: no external service is queried, so no lookup leaks which titles an instance holds.

Bit 7 being always clear is not decoration. It is what makes an arbitrary byte string not a valid signature, which is what the server validates on upload, and what stops the field being usable as a payload channel. The length is fixed by the construction rather than merely bounded, for the same reason — a caller cannot choose it, so it cannot become a variable-size container.

Why the parameters are written down twice

The specification's prose does not determine a byte stream. Two people implementing "32 log-spaced bands, take the peak" will disagree on at least:

• float or double — the fixture has frames whose two strongest bands are within 1.3% of each other, so float is not sufficient;
• periodic Hann (/N) or symmetric (/(N-1));
• whether a band's value is the mean or the sum of its magnitudes (sum favours wide high bands over narrow low ones);
• which way an argmax breaks ties;
• whether the frame count is 1 + (n - 4096)/1024 or something that wobbles with the resampler tail.

Every one of those is pinned at the top of AudioSignature.cs and matched in the C++ producer. A signature that differs in any parameter simply does not match, which defeats the entire point of having one.

Two implementations, proven equal

The extraction pipeline (C++, IR-004) computes this for files it processes locally. The plugin (C#, JR-042) computes it for the files the pipeline never sees. Both must agree exactly.

That is a checked claim, not an aspiration. fixtures/audio/ holds three files byte-identical to the extraction repo's copies:

File	What it pins
jray_audio_v1_tone.flac	120 s of tones stepping through all 32 bands, amplitudes walking a golden-ratio sequence so all four energy classes appear
jray_audio_v1_golden.json	The expected signature, the band→FFT-bin table, and checksums of the decoded PCM
make_fixture.py	Regenerates the media from plain arithmetic — no numpy, ports to any language in ~20 lines

The binding check regenerates the fixture PCM from make_fixture.py's arithmetic and verifies it against the recorded s16le/f32le checksums before making any DSP claim. So it runs on a CI host with no codec at all, and a decode divergence stays distinguishable from a DSP one. The two tests that drive real FFmpeg self-skip without a binary — a check that skips is not a check, so it is never the only cover for a claim.

---

Matching, and the offset

Two signatures are compared by sliding one against the other:

for offset in -600 .. +600 frames:        # ±56 s
    score(offset) = fraction of overlapping frames whose peak band matches
best = argmax score

Score	Meaning
≥ 0.85	Same cut. audio tier. The offset applies
0.60 – 0.85	Possibly the same cut, degraded audio. loose tier, surfaced as a caveat
< 0.60	Different content. No match

Only the peak band is scored. The energy class is the coarser and less re-encoding-stable of the two fields, and the specification's rule names the peak bin alone.

Speed-differing releases (a PAL 4% speed-up) are not a constant offset and are correctly rejected by the score threshold rather than mis-aligned.

The offset has two terms

This is the part that is easy to get wrong. Both windows are centred on their own file's midpoint, so when the runtimes differ the two windows do not start at the same point in the content:

offset = (local_window_start - manifest_window_start)  +  slide × 1024/11025
         └────────── anchor difference ──────────┘        └──── recovered ────┘

A release carrying 40 s of extra head material recovers 20 s from each term. Using the slide alone would be wrong by half the runtime difference on every shifted release. The specification's pseudocode describes only the slide, because it is written from the server's position, where both signatures are being compared against one stored manifest.

One parameter that is not from the specification

An alignment must overlap by at least 64 frames (~6 s) before its score counts. Without a floor the extreme offsets compare a handful of frames, where a chance agreement scores 1.0 and beats the true alignment. It never binds on the real case: two full-length signatures still overlap by 688 frames at the widest offset. It is marked as a local addition in the code.

---

What happens when a manifest arrives

ManifestController.FetchItem → ManifestAligner.AlignAsync, before anything is stored:

1. Server returns a manifest, a tier, and an offset.
2. Does the manifest carry cut.audio_signature?      no  -> use the server's offset
   Are signatures enabled in configuration?          no  -> use the server's offset
3. Decode this file's centre window, compute its signature.
4. Compare the two.
      match          -> apply the LOCAL offset and tier
      no match       -> apply the server's offset, record the disagreement
      not comparable -> apply the server's offset
5. Apply the offset to every window, once, at store time.
6. Write the truth file, and the alignment beside it.

Why the local answer wins

The server has never seen your file. Its offset is a claim about a runtime it was told — at best a runtime-difference inference. A local alignment compares the manifest's own signature against the media the windows will actually be drawn over, which is the authoritative comparison. It also needs no round trip, so no signature ever leaves the instance.

This is what the specification means by matching being "a consumer concern": the server never rewrites a manifest, so one stored manifest serves every trim of the same cut, and each client shifts it onto its own timebase.

Degradation, never failure

A signature is an enhancement to cut matching. A missing one costs a tier and must never be able to break a fetch. Every one of these stores the manifest on the server's terms:

• signatures switched off in configuration (they are opt-in — the decode costs a second or two of I/O per item);
• the item is under 120 s, so the window underflows and there is no signature to compute (JR-044);
• the manifest carried no signature;
• the manifest's signature is v2: from a future producer — refused, not parsed (JR-045), because scoring an unknown DSP chain as v1 would be a confident wrong answer where declining is a correct one;
• no FFmpeg binary, no audio stream, or a decode error.

"Un-comparable" and "does not match" are different

The distinction matters more than it looks. A 90-second extra is not content that disagrees with its manifest — it is content that could not be compared. Reporting the first as the second would show a user a scary warning about the wrong thing.

A genuine mismatch — both signatures present, both valid, both items long enough, and the best alignment still below 0.60 — is the strongest available hint that a manifest describes different content. It is still not a failure: the audio may legitimately differ, a different language track being the obvious case. So the manifest is stored on the server's terms and the disagreement is surfaced as a caveat, which outranks the tier's own caveat because it is the stronger statement.

---

What gets stored

The offset is applied once, at store time, so the stored windows are always in your file's own timebase and no read path needs offset awareness (JR-030). That makes the offset unrecoverable afterwards — the windows look native — which is why the alignment is recorded beside the truth file:

"alignment": {
  "source": "Local",
  "tier": "Audio",
  "offset_sec": 40.0,
  "score": 0.97,
  "offset_frames": 215,
  "local_signature": "v1:AAAAA…",
  "server_offset_sec": 0.0,
  "server_tier": "Runtime"
}

local_signature is kept so a later fetch aligns for free — the decode is the expensive half and the reason signatures are opt-in. It never leaves the instance: provenance is stored beside the truth file, not inside it, and contribution strips provenance entirely (JR-034).

The truth file itself is untouched by any of this. Injecting fields would mean the bytes served back are not the bytes the producer wrote, which is the property JR-004 turns on.

---

Where the code is

Concern	File
The DSP	Services/AudioSignature.cs
The decode	Services/AudioSignatureService.cs
Reading and matching	Services/AudioSignatureMatcher.cs
The fetch-path decision	Services/ManifestAligner.cs
What is recorded	Models/TruthAlignment.cs
Server-side validation	JRay-public-server/src/validate.rs, validate_audio_signature
C++ producer	scene-actor-extraction/src/audio_signature.{hpp,cpp}

Tests: AudioSignatureTests (UT-038 … UT-044), AudioSignatureMatcherTests (UT-045 … UT-052), ManifestAlignerTests (UT-053 … UT-057).

Verifying the cross-repo claim by hand

# C# side
cd jRay && dotnet test Jellyfin.Plugin.JRay.Tests/Jellyfin.Plugin.JRay.Tests.csproj

# C++ side, against the same fixture
cd scene-actor-extraction/build && python3 -c "
import json, sae_audio
g = json.load(open('../tests/fixtures/audio/jray_audio_v1_golden.json'))
print(sae_audio.compute_signature('../tests/fixtures/audio/jray_audio_v1_tone.flac') == g['signature'])
"

# and that the fixtures really are the same bytes
md5sum jRay/Jellyfin.Plugin.JRay.Tests/fixtures/audio/jray_audio_v1_golden.json \
       scene-actor-extraction/tests/fixtures/audio/jray_audio_v1_golden.json

Not built yet

• Matching against a server's catalogue. POST /manifests/search and the audio tier server-side are UR-009, still in progress. The sequencing is deliberate — accumulate signatures first, enable matching once coverage is useful. Nothing here depends on it: alignment works from the signature the manifest already carries.
• Contribution. Uploads do not yet attach a signature (JR-034).
• Re-using the stored signature. It is written but not yet read back on a second fetch, so today every fetch decodes.