Add the v1 audio signature to the pipeline (IR-004, IR-005, IR-007, IR-008)

Implements the content-derived spectral-peak signature from
JRay-public-server/SPEC.md §3 so a truth file is self-identifying: 120 s
window centred on the media midpoint, mono at 11025 Hz, 4096/1024 Hann
STFT, 32 log-spaced bins over 300-3000 Hz, one byte per frame (5-bit peak
band + 2-bit energy class), base64, `v1:` prefix.

Audio decode is a second stream from the FFmpeg libraries the pipeline
already links for video; libswresample is added to the existing
ffmpeg_libs interface target. The FFT is written out rather than pulled
from a library for the same reason the plugin vendors one: the output has
to be bit-identical across two languages, so a dependency whose version
could change the numerics is a liability.

The server spec fixes the geometry but not enough to reproduce a byte
stream — Hann periodicity, band aggregation, the energy-class definition,
tie-breaking and the base64 alphabet are all unconstrained by it. Those
are pinned in audio_signature.hpp and mirrored in the golden fixture, so
the plugin can be implemented from the fixture alone.

IR-005: tests/fixtures/audio/ carries a deterministic 120 s tone (FLAC —
lossless, so identical PCM to the WAV make_fixture.py emits, and 3.5x
smaller in git) plus the signature it must produce, the decoded-PCM
checksum and the full parameter contract. That directory is the artefact
shared with the plugin repo; the PCM checksum is separate from the
signature so a codec-level difference is distinguishable from a DSP one.

IR-007: media under 120 s emits no signature. Same for a file with no
audio stream or one that will not open — UR-9 is an enhancement and must
never be able to break a fetch.

Verified against an independent Python reference implementation: same
bytes. All 32 bands and all 4 energy classes appear in the golden vector,
and the window-centring test wraps the fixture in 90 s of silence either
side and requires the golden value back.

Not wired into the truth-file output yet — that is the schema_version
bump under IR-002/IR-003 and is deliberately out of scope here.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-30 18:25:37 +02:00
co-authored by Claude Opus 5
parent 43d2c976c3
commit 45ef7c1916
9 changed files with 1283 additions and 9 deletions
+17 -8
View File
@@ -196,23 +196,31 @@ endif()
# FFmpeg (hwaccel video decode: CUDA/VAAPI, runtime-detected + swscale colour
# conversion). Hwaccel support is built into libavcodec/libavutil; no extra
# libraries are needed here.
# libswresample is the audio side of the same dependency — downmix + resample
# for the audio signature (IR-004, src/audio_signature.cpp). Not a new project
# dependency: it ships with the libav* set already required above.
find_package(PkgConfig REQUIRED)
pkg_check_modules(AVFORMAT REQUIRED libavformat)
pkg_check_modules(AVCODEC REQUIRED libavcodec)
pkg_check_modules(AVUTIL REQUIRED libavutil)
pkg_check_modules(SWSCALE REQUIRED libswscale)
pkg_check_modules(AVFORMAT REQUIRED libavformat)
pkg_check_modules(AVCODEC REQUIRED libavcodec)
pkg_check_modules(AVUTIL REQUIRED libavutil)
pkg_check_modules(SWSCALE REQUIRED libswscale)
pkg_check_modules(SWRESAMPLE REQUIRED libswresample)
add_library(ffmpeg_libs INTERFACE)
target_compile_options(ffmpeg_libs INTERFACE
${AVFORMAT_CFLAGS_OTHER} ${AVCODEC_CFLAGS_OTHER}
${AVUTIL_CFLAGS_OTHER} ${SWSCALE_CFLAGS_OTHER})
${AVUTIL_CFLAGS_OTHER} ${SWSCALE_CFLAGS_OTHER}
${SWRESAMPLE_CFLAGS_OTHER})
target_include_directories(ffmpeg_libs INTERFACE
${AVFORMAT_INCLUDE_DIRS} ${AVCODEC_INCLUDE_DIRS}
${AVUTIL_INCLUDE_DIRS} ${SWSCALE_INCLUDE_DIRS})
${AVUTIL_INCLUDE_DIRS} ${SWSCALE_INCLUDE_DIRS}
${SWRESAMPLE_INCLUDE_DIRS})
target_link_libraries(ffmpeg_libs INTERFACE
${AVFORMAT_LIBRARIES} ${AVCODEC_LIBRARIES}
${AVUTIL_LIBRARIES} ${SWSCALE_LIBRARIES})
message(STATUS "FFmpeg: avformat=${AVFORMAT_VERSION} avcodec=${AVCODEC_VERSION}")
${AVUTIL_LIBRARIES} ${SWSCALE_LIBRARIES}
${SWRESAMPLE_LIBRARIES})
message(STATUS "FFmpeg: avformat=${AVFORMAT_VERSION} avcodec=${AVCODEC_VERSION} "
"swresample=${SWRESAMPLE_VERSION}")
# nlohmann/json (gallery + output serialisation)
include(FetchContent)
@@ -249,6 +257,7 @@ find_package(HDF5 REQUIRED COMPONENTS CXX)
add_library(sae_gallery STATIC
src/gallery/gallery_store.cpp
src/gallery/gallery_builder.cpp
src/audio_signature.cpp # IR-004 — content-derived audio signature
)
set_target_properties(sae_gallery PROPERTIES POSITION_INDEPENDENT_CODE ON)
target_include_directories(sae_gallery PUBLIC src ${HDF5_INCLUDE_DIRS})