Add the v1 audio signature to the pipeline (IR-004, IR-005, IR-007, IR-008)

Implements the content-derived spectral-peak signature from
JRay-public-server/SPEC.md §3 so a truth file is self-identifying: 120 s
window centred on the media midpoint, mono at 11025 Hz, 4096/1024 Hann
STFT, 32 log-spaced bins over 300-3000 Hz, one byte per frame (5-bit peak
band + 2-bit energy class), base64, `v1:` prefix.

Audio decode is a second stream from the FFmpeg libraries the pipeline
already links for video; libswresample is added to the existing
ffmpeg_libs interface target. The FFT is written out rather than pulled
from a library for the same reason the plugin vendors one: the output has
to be bit-identical across two languages, so a dependency whose version
could change the numerics is a liability.

The server spec fixes the geometry but not enough to reproduce a byte
stream — Hann periodicity, band aggregation, the energy-class definition,
tie-breaking and the base64 alphabet are all unconstrained by it. Those
are pinned in audio_signature.hpp and mirrored in the golden fixture, so
the plugin can be implemented from the fixture alone.

IR-005: tests/fixtures/audio/ carries a deterministic 120 s tone (FLAC —
lossless, so identical PCM to the WAV make_fixture.py emits, and 3.5x
smaller in git) plus the signature it must produce, the decoded-PCM
checksum and the full parameter contract. That directory is the artefact
shared with the plugin repo; the PCM checksum is separate from the
signature so a codec-level difference is distinguishable from a DSP one.

IR-007: media under 120 s emits no signature. Same for a file with no
audio stream or one that will not open — UR-9 is an enhancement and must
never be able to break a fetch.

Verified against an independent Python reference implementation: same
bytes. All 32 bands and all 4 energy classes appear in the golden vector,
and the window-centring test wraps the fixture in 90 s of silence either
side and requires the golden value back.

Not wired into the truth-file output yet — that is the schema_version
bump under IR-002/IR-003 and is deliberately out of scope here.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-30 18:25:37 +02:00
co-authored by Claude Opus 5
parent 43d2c976c3
commit 45ef7c1916
9 changed files with 1283 additions and 9 deletions
+9 -1
View File
@@ -21,21 +21,29 @@ add_executable(sae_tests
test_face_utils.cpp
test_track_gallery.cpp
test_face_tracker.cpp
test_audio_signature.cpp
${CMAKE_SOURCE_DIR}/src/backends/gemm_backend.cpp
${CMAKE_SOURCE_DIR}/src/gallery/gallery_store.cpp
${CMAKE_SOURCE_DIR}/src/audio_signature.cpp
)
target_include_directories(sae_tests PRIVATE ${CMAKE_SOURCE_DIR}/src)
# SAE_GEMM_CPU: build the CPU reference GEMM regardless of the main backend.
# SAE_MODELS_DIR: config.hpp (pulled in by track_gallery.hpp) bakes model paths.
# SAE_TEST_FIXTURES_DIR: the audio golden vector is read from the source tree,
# not copied, so the file the plugin repo shares is the file under test.
target_compile_definitions(sae_tests PRIVATE
SAE_GEMM_CPU
SAE_MODELS_DIR="${SAE_MODELS_DIR}")
SAE_MODELS_DIR="${SAE_MODELS_DIR}"
SAE_TEST_FIXTURES_DIR="${CMAKE_CURRENT_SOURCE_DIR}/fixtures")
# gallery_store.cpp + gallery_calibration.hpp use nlohmann/json and HDF5
# (galleries are HDF5-native, see src/gallery/gallery_store.cpp); face_utils.hpp
# and the calibration GEMM pull in OpenCV (calib3d/imgproc/core) via types.hpp.
# ffmpeg_libs: audio_signature.cpp decodes the golden fixture (avformat/avcodec/
# avutil/swresample). Still GPU-free — the audio path is pure CPU.
target_link_libraries(sae_tests PRIVATE
Catch2::Catch2WithMain
nlohmann_json::nlohmann_json
ffmpeg_libs
${OpenCV_LIBS}
${HDF5_CXX_LIBRARIES})
target_include_directories(sae_tests PRIVATE ${HDF5_INCLUDE_DIRS})