# Audio signature and alignment — how it works How a manifest fetched from a public server is checked against, and shifted onto, your own copy of a film. Requirements: `JR-042` … `JR-045`, `JR-047`. Construction is owned by [`JRay-public-server/SPEC.md` §3](../../JRay-public-server/SPEC.md); this document is the mechanism end to end. --- ## The problem A manifest says "Peter Capaldi is on screen from 42:10 to 42:38". Your file may not agree, because releases of the same film are trimmed differently — a distributor logo here, a longer certificate card there. Thirty seconds of extra head material makes every window in the manifest wrong by thirty seconds. The old defence was the runtime tier: accept a manifest only if the runtimes agree within ±2 s. That rejects exactly the releases it should be fixing, and accepts anything that happens to be the same length. A **content-derived signature** answers the question the exchange actually needs — *do these timings apply to this media?* — and, when the answer is "yes but shifted", says by how much. Deliberately, it does not answer *what file is this?* There is no file hash anywhere in the system; the tier was withdrawn on legal grounds. The signature identifies a **cut**, so two different encodes of the same edit agree. --- ## The signature 120 seconds from the **centre** of the media — the head and tail are the least content-specific parts of a release, being logos and credits. ``` decode centre window runtime/2 ± 60 s downmix mono resample 11025 Hz STFT 4096-sample frame, 1024-sample hop (~93 ms), Hann band 300–3000 Hz, 32 logarithmically spaced bands per frame peak band index (5 bits) + energy class (2 bits) -> one byte, bit 7 always clear result 1288 bytes, base64, prefixed "v1:" ``` Peak bins are used because they survive lossy re-encoding, loudness normalisation and channel-layout differences, where absolute magnitudes do not. Same principle as Chromaprint/AcoustID, but self-contained: no external service is queried, so no lookup leaks which titles an instance holds. Bit 7 being always clear is not decoration. It is what makes an arbitrary byte string *not* a valid signature, which is what the server validates on upload, and what stops the field being usable as a payload channel. The length is fixed by the construction rather than merely bounded, for the same reason — a caller cannot choose it, so it cannot become a variable-size container. ### Why the parameters are written down twice The specification's prose does not determine a byte stream. Two people implementing "32 log-spaced bands, take the peak" will disagree on at least: - `float` or `double` — the fixture has frames whose two strongest bands are within 1.3% of each other, so `float` is not sufficient; - periodic Hann (`/N`) or symmetric (`/(N-1)`); - whether a band's value is the **mean** or the **sum** of its magnitudes (sum favours wide high bands over narrow low ones); - which way an `argmax` breaks ties; - whether the frame count is `1 + (n - 4096)/1024` or something that wobbles with the resampler tail. Every one of those is pinned at the top of [`AudioSignature.cs`](../Jellyfin.Plugin.JRay/Services/AudioSignature.cs) and matched in the C++ producer. A signature that differs in any parameter simply does not match, which defeats the entire point of having one. ### Two implementations, proven equal The extraction pipeline (C++, `IR-004`) computes this for files it processes locally. The plugin (C#, `JR-042`) computes it for the files the pipeline never sees. Both must agree exactly. That is a *checked* claim, not an aspiration. `fixtures/audio/` holds three files byte-identical to the extraction repo's copies: | File | What it pins | |---|---| | `jray_audio_v1_tone.flac` | 120 s of tones stepping through all 32 bands, amplitudes walking a golden-ratio sequence so all four energy classes appear | | `jray_audio_v1_golden.json` | The expected signature, the band→FFT-bin table, and checksums of the decoded PCM | | `make_fixture.py` | Regenerates the media from plain arithmetic — no numpy, ports to any language in ~20 lines | The binding check regenerates the fixture PCM from `make_fixture.py`'s arithmetic and verifies it against the recorded `s16le`/`f32le` checksums **before** making any DSP claim. So it runs on a CI host with no codec at all, and a decode divergence stays distinguishable from a DSP one. The two tests that drive real FFmpeg self-skip without a binary — a check that skips is not a check, so it is never the only cover for a claim. --- ## Matching, and the offset Two signatures are compared by sliding one against the other: ``` for offset in -600 .. +600 frames: # ±56 s score(offset) = fraction of overlapping frames whose peak band matches best = argmax score ``` | Score | Meaning | |---|---| | `≥ 0.85` | Same cut. `audio` tier. The offset applies | | `0.60 – 0.85` | Possibly the same cut, degraded audio. `loose` tier, surfaced as a caveat | | `< 0.60` | Different content. No match | Only the **peak band** is scored. The energy class is the coarser and less re-encoding-stable of the two fields, and the specification's rule names the peak bin alone. Speed-differing releases (a PAL 4% speed-up) are not a constant offset and are correctly rejected by the score threshold rather than mis-aligned. ### The offset has two terms This is the part that is easy to get wrong. Both windows are centred on **their own file's** midpoint, so when the runtimes differ the two windows do not start at the same point in the content: ``` offset = (local_window_start - manifest_window_start) + slide × 1024/11025 └────────── anchor difference ──────────┘ └──── recovered ────┘ ``` A release carrying 40 s of extra head material recovers **20 s from each term**. Using the slide alone would be wrong by half the runtime difference on every shifted release. The specification's pseudocode describes only the slide, because it is written from the server's position, where both signatures are being compared against one stored manifest. ### One parameter that is not from the specification An alignment must overlap by at least **64 frames** (~6 s) before its score counts. Without a floor the extreme offsets compare a handful of frames, where a chance agreement scores 1.0 and beats the true alignment. It never binds on the real case: two full-length signatures still overlap by 688 frames at the widest offset. It is marked as a local addition in the code. --- ## What happens when a manifest arrives `ManifestController.FetchItem` → `ManifestAligner.AlignAsync`, before anything is stored: ``` 1. Server returns a manifest, a tier, and an offset. 2. Does the manifest carry cut.audio_signature? no -> use the server's offset Are signatures enabled in configuration? no -> use the server's offset 3. Decode this file's centre window, compute its signature. 4. Compare the two. match -> apply the LOCAL offset and tier no match -> apply the server's offset, record the disagreement not comparable -> apply the server's offset 5. Apply the offset to every window, once, at store time. 6. Write the truth file, and the alignment beside it. ``` ### Why the local answer wins **The server has never seen your file.** Its offset is a claim about a runtime it was told — at best a runtime-difference inference. A local alignment compares the manifest's own signature against the media the windows will actually be drawn over, which is the authoritative comparison. It also needs no round trip, so no signature ever leaves the instance. This is what the specification means by matching being "a consumer concern": the server never rewrites a manifest, so **one stored manifest serves every trim of the same cut**, and each client shifts it onto its own timebase. ### Degradation, never failure A signature is an enhancement to cut matching. A missing one costs a tier and **must never be able to break a fetch**. Every one of these stores the manifest on the server's terms: - signatures switched off in configuration (they are opt-in — the decode costs a second or two of I/O per item); - the item is under 120 s, so the window underflows and there is no signature to compute (`JR-044`); - the manifest carried no signature; - the manifest's signature is `v2:` from a future producer — **refused, not parsed** (`JR-045`), because scoring an unknown DSP chain as v1 would be a confident wrong answer where declining is a correct one; - no FFmpeg binary, no audio stream, or a decode error. ### "Un-comparable" and "does not match" are different The distinction matters more than it looks. A 90-second extra is not content that disagrees with its manifest — it is content that could not be compared. Reporting the first as the second would show a user a scary warning about the wrong thing. A genuine mismatch — both signatures present, both valid, both items long enough, and the best alignment still below 0.60 — is the strongest available hint that a manifest describes different content. It is **still not a failure**: the audio may legitimately differ, a different language track being the obvious case. So the manifest is stored on the server's terms and the disagreement is surfaced as a caveat, which outranks the tier's own caveat because it is the stronger statement. --- ## What gets stored The offset is applied **once**, at store time, so the stored windows are always in your file's own timebase and no read path needs offset awareness (`JR-030`). That makes the offset unrecoverable afterwards — the windows look native — which is why the alignment is recorded beside the truth file: ```json "alignment": { "source": "Local", "tier": "Audio", "offset_sec": 40.0, "score": 0.97, "offset_frames": 215, "local_signature": "v1:AAAAA…", "server_offset_sec": 0.0, "server_tier": "Runtime" } ``` `local_signature` is kept so a **later fetch aligns for free** — the decode is the expensive half and the reason signatures are opt-in. It never leaves the instance: provenance is stored beside the truth file, not inside it, and contribution strips provenance entirely (`JR-034`). The truth file itself is untouched by any of this. Injecting fields would mean the bytes served back are not the bytes the producer wrote, which is the property `JR-004` turns on. --- ## Where the code is | Concern | File | |---|---| | The DSP | [`Services/AudioSignature.cs`](../Jellyfin.Plugin.JRay/Services/AudioSignature.cs) | | The decode | [`Services/AudioSignatureService.cs`](../Jellyfin.Plugin.JRay/Services/AudioSignatureService.cs) | | Reading and matching | [`Services/AudioSignatureMatcher.cs`](../Jellyfin.Plugin.JRay/Services/AudioSignatureMatcher.cs) | | The fetch-path decision | [`Services/ManifestAligner.cs`](../Jellyfin.Plugin.JRay/Services/ManifestAligner.cs) | | What is recorded | [`Models/TruthAlignment.cs`](../Jellyfin.Plugin.JRay/Models/TruthAlignment.cs) | | Server-side validation | `JRay-public-server/src/validate.rs`, `validate_audio_signature` | | C++ producer | `scene-actor-extraction/src/audio_signature.{hpp,cpp}` | Tests: `AudioSignatureTests` (UT-038 … UT-044), `AudioSignatureMatcherTests` (UT-045 … UT-052), `ManifestAlignerTests` (UT-053 … UT-057). ## Verifying the cross-repo claim by hand ```sh # C# side cd jRay && dotnet test Jellyfin.Plugin.JRay.Tests/Jellyfin.Plugin.JRay.Tests.csproj # C++ side, against the same fixture cd scene-actor-extraction/build && python3 -c " import json, sae_audio g = json.load(open('../tests/fixtures/audio/jray_audio_v1_golden.json')) print(sae_audio.compute_signature('../tests/fixtures/audio/jray_audio_v1_tone.flac') == g['signature']) " # and that the fixtures really are the same bytes md5sum jRay/Jellyfin.Plugin.JRay.Tests/fixtures/audio/jray_audio_v1_golden.json \ scene-actor-extraction/tests/fixtures/audio/jray_audio_v1_golden.json ``` ## Not built yet - **Matching against a server's catalogue.** `POST /manifests/search` and the `audio` tier server-side are `UR-009`, still in progress. The sequencing is deliberate — accumulate signatures first, enable matching once coverage is useful. Nothing here depends on it: alignment works from the signature the manifest already carries. - **Contribution.** Uploads do not yet attach a signature (`JR-034`). - **Re-using the stored signature.** It is written but not yet read back on a second fetch, so today every fetch decodes.