Add the v1 audio signature to the pipeline (IR-004, IR-005, IR-007, IR-008)
Implements the content-derived spectral-peak signature from JRay-public-server/SPEC.md §3 so a truth file is self-identifying: 120 s window centred on the media midpoint, mono at 11025 Hz, 4096/1024 Hann STFT, 32 log-spaced bins over 300-3000 Hz, one byte per frame (5-bit peak band + 2-bit energy class), base64, `v1:` prefix. Audio decode is a second stream from the FFmpeg libraries the pipeline already links for video; libswresample is added to the existing ffmpeg_libs interface target. The FFT is written out rather than pulled from a library for the same reason the plugin vendors one: the output has to be bit-identical across two languages, so a dependency whose version could change the numerics is a liability. The server spec fixes the geometry but not enough to reproduce a byte stream — Hann periodicity, band aggregation, the energy-class definition, tie-breaking and the base64 alphabet are all unconstrained by it. Those are pinned in audio_signature.hpp and mirrored in the golden fixture, so the plugin can be implemented from the fixture alone. IR-005: tests/fixtures/audio/ carries a deterministic 120 s tone (FLAC — lossless, so identical PCM to the WAV make_fixture.py emits, and 3.5x smaller in git) plus the signature it must produce, the decoded-PCM checksum and the full parameter contract. That directory is the artefact shared with the plugin repo; the PCM checksum is separate from the signature so a codec-level difference is distinguishable from a DSP one. IR-007: media under 120 s emits no signature. Same for a file with no audio stream or one that will not open — UR-9 is an enhancement and must never be able to break a fetch. Verified against an independent Python reference implementation: same bytes. All 32 bands and all 4 energy classes appear in the golden vector, and the window-centring test wraps the fixture in 90 s of silence either side and requires the golden value back. Not wired into the truth-file output yet — that is the schema_version bump under IR-002/IR-003 and is deliberately out of scope here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5
+194
@@ -0,0 +1,194 @@
|
||||
{
|
||||
"_": "Golden vector for the JRay v1 audio signature (JRay-public-server SPEC.md \u00a73). Shared verbatim between scene-actor-extraction (C++) and the jRay Jellyfin plugin (C#) so the two implementations can be proven bit-identical. IR-004, IR-005, IR-007, IR-008.",
|
||||
"version": "v1",
|
||||
"signature": "v1:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAeHx8eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh8fHzk5OTk5OTk5OTk5OTk5OTk5OTk5OTk5OTk5OTk5V1dXV1dXV1dXV1dXV1dXV1dXV1dXV1dXV1dXV1dXV1dXV1dycnJycnJycnJycnJycnJycnJycnJycnJycnJycnMPDgwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKytFRUVFRUVFRUVFRUVFRUVFRUVFRUVFRUVFRUVFRWNjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2Njfn5+fn5+fn5+fn5+fn5+fn5+fn5+fn5+fn5+fn5/GxoZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGTc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3UlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSU1JsbGxsbGxsbGxsbGxsbGxsbGxsbGxsbGxsbGxsbAoLCwoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCwsLJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSVDQ0NDQ0NDQ0NDQ0NDQ0NDQ0NDQ0NDQ0NDQ0NDQ0NDQ0NDQ15eXl5eXl5eXl5eXl5eXl5eXl5eXl5eXl5eXl5eX19eeXl5eXl5eXl5eXl5eXl5eXl5eXl5eXl5eXl5eXkXFxcXFxcXFxcXFxcXFxcXFxcXFxcXFxcXFxcXFxcXFxcXFzExMTExMTExMTExMTExMTExMTExMTExMTExMTExT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09qampqampqampqampqampqampqampqampqampqamsHBwUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyM+Pj4+Pj4+Pj4+Pj4+Pj4+Pj4+Pj4+Pj4+Pj4+Pj4/PlhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYd3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3ExMRERERERERERERERERERERERERERERERERERERES8vLy8vLy8vLy8vLy8vLy8vLy8vLy8vLy8vLy8vLy8vLy8vSkpKSkpKSkpKSkpKSkpKSkpKSkpKSkpKSkpKSkpLS0plZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZQMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDHR0dHR0dHR0dHR0dHR0dHR0dHR0dHR0dHR0dHR0eHh44ODg4ODg4ODg4ODg4ODg4ODg4ODg4ODg4ODg4OFdXV1ZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWV1dXcXFxcXFxcXFxcXFxcXFxcXFxcXFxcXFxcXFxcXEPDw8PDw8PDw8PDw8PDw8PDw8PDw8PDw8PDw8PDw8PDw8PDyoqKioqKioqKioqKioqKioqKioqKioqKioqKioqKysqRERERERERERERERERERERERERERERERERERERERjY2NjY2NjY2NjYw==",
|
||||
"frame_count": 1288,
|
||||
"media": {
|
||||
"file": "jray_audio_v1_tone.flac",
|
||||
"generator": "make_fixture.py",
|
||||
"container": "FLAC (lossless \u2014 decodes to exactly the PCM make_fixture.py emits)",
|
||||
"duration_sec": 120.0,
|
||||
"sample_rate": 11025,
|
||||
"channels": 1,
|
||||
"sample_format": "s16",
|
||||
"sha256": "912ecd426cd426dccb37753e0249694227619c701cb9f533502b37da0fbe8096",
|
||||
"bytes": 585142
|
||||
},
|
||||
"decoded_window": {
|
||||
"_": "Checksums of the 120 s centre window after downmix to mono and resample to 11025 Hz, i.e. exactly the stream `ffmpeg -ss <mid-60> -t 120 -i <file> -vn -ac 1 -ar 11025 -f f32le -` produces. Check these first: a mismatch here is a decode problem, not a DSP one.",
|
||||
"samples": 1323000,
|
||||
"f32le_fnv1a64": "0x1ef7899cd4d12662",
|
||||
"s16le_fnv1a64": "0xf824fa56f125c0dc"
|
||||
},
|
||||
"params": {
|
||||
"window_sec": 120.0,
|
||||
"window_centre": "runtime/2, i.e. samples from runtime/2 - 60 s; truncated to exactly 1323000 samples",
|
||||
"min_duration_sec": 120.0,
|
||||
"min_duration_rule": "IR-007 \u2014 below this emit NO signature and apply no sync offset",
|
||||
"sample_rate": 11025,
|
||||
"channels": 1,
|
||||
"arithmetic": "IEEE-754 double throughout; float32 is not sufficient",
|
||||
"sample_scale": "s16 * (1/32768), FFmpeg's native s16->flt",
|
||||
"frame_size": 4096,
|
||||
"hop_size": 1024,
|
||||
"frame_count_rule": "1 + (n_samples - 4096) / 1024, integer division; whole frames only",
|
||||
"window_fn": "Hann, PERIODIC: w[n] = 0.5 * (1 - cos(2*pi*n/4096))",
|
||||
"transform": "radix-2 DIT complex FFT over the 4096 real samples (imag=0), no normalisation",
|
||||
"magnitude": "sqrt(re^2 + im^2), linear",
|
||||
"band_lo_hz": 300.0,
|
||||
"band_hi_hz": 3000.0,
|
||||
"num_bands": 32,
|
||||
"band_edges": "edge[b] = 300 * (3000/300)^(b/32), b = 0..32",
|
||||
"band_bins": "band b owns FFT bins [k_lo[b], k_lo[b+1]) with k_lo[b] = ceil(edge[b] * 4096 / 11025); see band_fft_bins",
|
||||
"band_value": "MEAN of the linear magnitudes in the band (not sum, not max)",
|
||||
"peak_bin": "argmax over the 32 band values; ties resolve to the LOWEST index",
|
||||
"energy_metric": "E = mean magnitude over all FFT bins 112..1114, i.e. the whole 300-3000 Hz band",
|
||||
"energy_reference": "upper median of E over all frames: sorted[n/2], no averaging of the two middle values",
|
||||
"energy_ratio": "r = log10((E + 1e-12) / (E_ref + 1e-12))",
|
||||
"energy_class_edges": [
|
||||
-0.6,
|
||||
-0.2,
|
||||
0.2
|
||||
],
|
||||
"energy_class": "0 if r < -0.6, 1 if r < -0.2, 2 if r < 0.2, else 3",
|
||||
"byte_layout": "bit7 = 0 (reserved), bits6..2 = 5-bit band index, bits1..0 = 2-bit energy class; byte = (band << 2) | class",
|
||||
"base64": "standard alphabet A-Za-z0-9+/ with '=' padding",
|
||||
"prefix": "v1:"
|
||||
},
|
||||
"band_fft_bins": [
|
||||
[
|
||||
112,
|
||||
120
|
||||
],
|
||||
[
|
||||
120,
|
||||
129
|
||||
],
|
||||
[
|
||||
129,
|
||||
139
|
||||
],
|
||||
[
|
||||
139,
|
||||
149
|
||||
],
|
||||
[
|
||||
149,
|
||||
160
|
||||
],
|
||||
[
|
||||
160,
|
||||
172
|
||||
],
|
||||
[
|
||||
172,
|
||||
185
|
||||
],
|
||||
[
|
||||
185,
|
||||
199
|
||||
],
|
||||
[
|
||||
199,
|
||||
213
|
||||
],
|
||||
[
|
||||
213,
|
||||
229
|
||||
],
|
||||
[
|
||||
229,
|
||||
246
|
||||
],
|
||||
[
|
||||
246,
|
||||
265
|
||||
],
|
||||
[
|
||||
265,
|
||||
285
|
||||
],
|
||||
[
|
||||
285,
|
||||
306
|
||||
],
|
||||
[
|
||||
306,
|
||||
328
|
||||
],
|
||||
[
|
||||
328,
|
||||
353
|
||||
],
|
||||
[
|
||||
353,
|
||||
379
|
||||
],
|
||||
[
|
||||
379,
|
||||
408
|
||||
],
|
||||
[
|
||||
408,
|
||||
438
|
||||
],
|
||||
[
|
||||
438,
|
||||
471
|
||||
],
|
||||
[
|
||||
471,
|
||||
506
|
||||
],
|
||||
[
|
||||
506,
|
||||
543
|
||||
],
|
||||
[
|
||||
543,
|
||||
584
|
||||
],
|
||||
[
|
||||
584,
|
||||
627
|
||||
],
|
||||
[
|
||||
627,
|
||||
674
|
||||
],
|
||||
[
|
||||
674,
|
||||
724
|
||||
],
|
||||
[
|
||||
724,
|
||||
778
|
||||
],
|
||||
[
|
||||
778,
|
||||
836
|
||||
],
|
||||
[
|
||||
836,
|
||||
899
|
||||
],
|
||||
[
|
||||
899,
|
||||
966
|
||||
],
|
||||
[
|
||||
966,
|
||||
1038
|
||||
],
|
||||
[
|
||||
1038,
|
||||
1115
|
||||
]
|
||||
],
|
||||
"notes": [
|
||||
"The server spec fixes the window, rate, STFT geometry, band and the 5+2 bit packing. Everything under params beyond that (Hann periodicity, band aggregation, the energy-class definition, tie-breaking, base64 alphabet) is pinned HERE for v1 \u2014 the spec does not constrain it, and two implementations that guess differently produce non-matching signatures.",
|
||||
"Decision margins on this fixture: the two strongest bands are within 1.3% on the closest frame, and the closest frame to an energy-class edge is 3.6e-3 away in log10. Both are many orders of magnitude above double-precision FFT differences, so any two correct double- precision implementations agree; a float32 implementation is not guaranteed to.",
|
||||
"Coverage: all 32 bands and all 4 energy classes appear in the golden signature.",
|
||||
"Robustness observed on this fixture: identical peak-bin sequence after a stereo/44100 Hz round trip and after AAC 128 kbit/s re-encoding."
|
||||
]
|
||||
}
|
||||
Reference in New Issue
Block a user