jRay/docs/audio-alignment.md

# Audio signature and alignment — how it works

How a manifest fetched from a public server is checked against, and shifted onto,
your own copy of a film.

Requirements: `JR-042` … `JR-045`, `JR-047`. Construction is owned by
[`JRay-public-server/SPEC.md` §3](../../JRay-public-server/SPEC.md); this document
is the mechanism end to end.

---

## The problem

A manifest says "Peter Capaldi is on screen from 42:10 to 42:38". Your file may
not agree, because releases of the same film are trimmed differently — a
distributor logo here, a longer certificate card there. Thirty seconds of extra
head material makes every window in the manifest wrong by thirty seconds.

The old defence was the runtime tier: accept a manifest only if the runtimes
agree within ±2 s. That rejects exactly the releases it should be fixing, and
accepts anything that happens to be the same length.

A **content-derived signature** answers the question the exchange actually needs
— *do these timings apply to this media?* — and, when the answer is "yes but
shifted", says by how much.

Deliberately, it does not answer *what file is this?* There is no file hash
anywhere in the system; the tier was withdrawn on legal grounds. The signature
identifies a **cut**, so two different encodes of the same edit agree.

---

## The signature

120 seconds from the **centre** of the media — the head and tail are the least
content-specific parts of a release, being logos and credits.

```
decode centre window   runtime/2 ± 60 s
downmix                mono
resample               11025 Hz
STFT                   4096-sample frame, 1024-sample hop (~93 ms), Hann
band                   300–3000 Hz, 32 logarithmically spaced bands
per frame              peak band index (5 bits) + energy class (2 bits)
                       -> one byte, bit 7 always clear
result                 1288 bytes, base64, prefixed "v1:"
```

Peak bins are used because they survive lossy re-encoding, loudness
normalisation and channel-layout differences, where absolute magnitudes do not.
Same principle as Chromaprint/AcoustID, but self-contained: no external service
is queried, so no lookup leaks which titles an instance holds.

Bit 7 being always clear is not decoration. It is what makes an arbitrary byte
string *not* a valid signature, which is what the server validates on upload, and
what stops the field being usable as a payload channel. The length is fixed by
the construction rather than merely bounded, for the same reason — a caller
cannot choose it, so it cannot become a variable-size container.

### Why the parameters are written down twice

The specification's prose does not determine a byte stream. Two people
implementing "32 log-spaced bands, take the peak" will disagree on at least:

- `float` or `double` — the fixture has frames whose two strongest bands are
  within 1.3% of each other, so `float` is not sufficient;
- periodic Hann (`/N`) or symmetric (`/(N-1)`);
- whether a band's value is the **mean** or the **sum** of its magnitudes (sum
  favours wide high bands over narrow low ones);
- which way an `argmax` breaks ties;
- whether the frame count is `1 + (n - 4096)/1024` or something that wobbles
  with the resampler tail.

Every one of those is pinned at the top of
[`AudioSignature.cs`](../Jellyfin.Plugin.JRay/Services/AudioSignature.cs) and
matched in the C++ producer. A signature that differs in any parameter simply
does not match, which defeats the entire point of having one.

### Two implementations, proven equal

The extraction pipeline (C++, `IR-004`) computes this for files it processes
locally. The plugin (C#, `JR-042`) computes it for the files the pipeline never
sees. Both must agree exactly.

That is a *checked* claim, not an aspiration. `fixtures/audio/` holds three files
byte-identical to the extraction repo's copies:

| File | What it pins |
|---|---|
| `jray_audio_v1_tone.flac` | 120 s of tones stepping through all 32 bands, amplitudes walking a golden-ratio sequence so all four energy classes appear |
| `jray_audio_v1_golden.json` | The expected signature, the band→FFT-bin table, and checksums of the decoded PCM |
| `make_fixture.py` | Regenerates the media from plain arithmetic — no numpy, ports to any language in ~20 lines |

The binding check regenerates the fixture PCM from `make_fixture.py`'s arithmetic
and verifies it against the recorded `s16le`/`f32le` checksums **before** making
any DSP claim. So it runs on a CI host with no codec at all, and a decode
divergence stays distinguishable from a DSP one. The two tests that drive real
FFmpeg self-skip without a binary — a check that skips is not a check, so it is
never the only cover for a claim.

---

## Matching, and the offset

Two signatures are compared by sliding one against the other:

```
for offset in -600 .. +600 frames:        # ±56 s
    score(offset) = fraction of overlapping frames whose peak band matches
best = argmax score
```

| Score | Meaning |
|---|---|
| `≥ 0.85` | Same cut. `audio` tier. The offset applies |
| `0.60 – 0.85` | Possibly the same cut, degraded audio. `loose` tier, surfaced as a caveat |
| `< 0.60` | Different content. No match |

Only the **peak band** is scored. The energy class is the coarser and less
re-encoding-stable of the two fields, and the specification's rule names the peak
bin alone.

Speed-differing releases (a PAL 4% speed-up) are not a constant offset and are
correctly rejected by the score threshold rather than mis-aligned.

### The offset has two terms

This is the part that is easy to get wrong. Both windows are centred on **their
own file's** midpoint, so when the runtimes differ the two windows do not start
at the same point in the content:

```
offset = (local_window_start - manifest_window_start)  +  slide × 1024/11025
         └────────── anchor difference ──────────┘        └──── recovered ────┘
```

A release carrying 40 s of extra head material recovers **20 s from each term**.
Using the slide alone would be wrong by half the runtime difference on every
shifted release. The specification's pseudocode describes only the slide, because
it is written from the server's position, where both signatures are being
compared against one stored manifest.

### One parameter that is not from the specification

An alignment must overlap by at least **64 frames** (~6 s) before its score
counts. Without a floor the extreme offsets compare a handful of frames, where a
chance agreement scores 1.0 and beats the true alignment. It never binds on the
real case: two full-length signatures still overlap by 688 frames at the widest
offset. It is marked as a local addition in the code.

---

## What happens when a manifest arrives

`ManifestController.FetchItem` → `ManifestAligner.AlignAsync`, before anything is
stored:

```
1. Server returns a manifest, a tier, and an offset.
2. Does the manifest carry cut.audio_signature?      no  -> use the server's offset
   Are signatures enabled in configuration?          no  -> use the server's offset
3. Decode this file's centre window, compute its signature.
4. Compare the two.
      match          -> apply the LOCAL offset and tier
      no match       -> apply the server's offset, record the disagreement
      not comparable -> apply the server's offset
5. Apply the offset to every window, once, at store time.
6. Write the truth file, and the alignment beside it.
```

### Why the local answer wins

**The server has never seen your file.** Its offset is a claim about a runtime it
was told — at best a runtime-difference inference. A local alignment compares the
manifest's own signature against the media the windows will actually be drawn
over, which is the authoritative comparison. It also needs no round trip, so no
signature ever leaves the instance.

This is what the specification means by matching being "a consumer concern": the
server never rewrites a manifest, so **one stored manifest serves every trim of
the same cut**, and each client shifts it onto its own timebase.

### Degradation, never failure

A signature is an enhancement to cut matching. A missing one costs a tier and
**must never be able to break a fetch**. Every one of these stores the manifest
on the server's terms:

- signatures switched off in configuration (they are opt-in — the decode costs a
  second or two of I/O per item);
- the item is under 120 s, so the window underflows and there is no signature to
  compute (`JR-044`);
- the manifest carried no signature;
- the manifest's signature is `v2:` from a future producer — **refused, not
  parsed** (`JR-045`), because scoring an unknown DSP chain as v1 would be a
  confident wrong answer where declining is a correct one;
- no FFmpeg binary, no audio stream, or a decode error.

### "Un-comparable" and "does not match" are different

The distinction matters more than it looks. A 90-second extra is not content that
disagrees with its manifest — it is content that could not be compared. Reporting
the first as the second would show a user a scary warning about the wrong thing.

A genuine mismatch — both signatures present, both valid, both items long enough,
and the best alignment still below 0.60 — is the strongest available hint that a
manifest describes different content. It is **still not a failure**: the audio may
legitimately differ, a different language track being the obvious case. So the
manifest is stored on the server's terms and the disagreement is surfaced as a
caveat, which outranks the tier's own caveat because it is the stronger
statement.

---

## What gets stored

The offset is applied **once**, at store time, so the stored windows are always
in your file's own timebase and no read path needs offset awareness (`JR-030`).
That makes the offset unrecoverable afterwards — the windows look native — which
is why the alignment is recorded beside the truth file:

```json
"alignment": {
  "source": "Local",
  "tier": "Audio",
  "offset_sec": 40.0,
  "score": 0.97,
  "offset_frames": 215,
  "local_signature": "v1:AAAAA…",
  "server_offset_sec": 0.0,
  "server_tier": "Runtime"
}
```

`local_signature` is kept so a **later fetch aligns for free** — the decode is
the expensive half and the reason signatures are opt-in. It never leaves the
instance: provenance is stored beside the truth file, not inside it, and
contribution strips provenance entirely (`JR-034`).

The truth file itself is untouched by any of this. Injecting fields would mean
the bytes served back are not the bytes the producer wrote, which is the property
`JR-004` turns on.

---

## Where the code is

| Concern | File |
|---|---|
| The DSP | [`Services/AudioSignature.cs`](../Jellyfin.Plugin.JRay/Services/AudioSignature.cs) |
| The decode | [`Services/AudioSignatureService.cs`](../Jellyfin.Plugin.JRay/Services/AudioSignatureService.cs) |
| Reading and matching | [`Services/AudioSignatureMatcher.cs`](../Jellyfin.Plugin.JRay/Services/AudioSignatureMatcher.cs) |
| The fetch-path decision | [`Services/ManifestAligner.cs`](../Jellyfin.Plugin.JRay/Services/ManifestAligner.cs) |
| What is recorded | [`Models/TruthAlignment.cs`](../Jellyfin.Plugin.JRay/Models/TruthAlignment.cs) |
| Server-side validation | `JRay-public-server/src/validate.rs`, `validate_audio_signature` |
| C++ producer | `scene-actor-extraction/src/audio_signature.{hpp,cpp}` |

Tests: `AudioSignatureTests` (UT-038 … UT-044), `AudioSignatureMatcherTests`
(UT-045 … UT-052), `ManifestAlignerTests` (UT-053 … UT-057).

## Verifying the cross-repo claim by hand

```sh
# C# side
cd jRay && dotnet test Jellyfin.Plugin.JRay.Tests/Jellyfin.Plugin.JRay.Tests.csproj

# C++ side, against the same fixture
cd scene-actor-extraction/build && python3 -c "
import json, sae_audio
g = json.load(open('../tests/fixtures/audio/jray_audio_v1_golden.json'))
print(sae_audio.compute_signature('../tests/fixtures/audio/jray_audio_v1_tone.flac') == g['signature'])
"

# and that the fixtures really are the same bytes
md5sum jRay/Jellyfin.Plugin.JRay.Tests/fixtures/audio/jray_audio_v1_golden.json \
       scene-actor-extraction/tests/fixtures/audio/jray_audio_v1_golden.json
```

## Not built yet

- **Matching against a server's catalogue.** `POST /manifests/search` and the
  `audio` tier server-side are `UR-009`, still in progress. The sequencing is
  deliberate — accumulate signatures first, enable matching once coverage is
  useful. Nothing here depends on it: alignment works from the signature the
  manifest already carries.
- **Contribution.** Uploads do not yet attach a signature (`JR-034`).
- **Re-using the stored signature.** It is written but not yet read back on a
  second fetch, so today every fetch decodes.