Claude
7db40f430d
A gallery is only valid for the embedder that produced its vectors. Cosine
similarities across models are meaningless but *look* plausible, so the mistake
is silent and every measurement taken afterwards is suspect. Stamp the embedder
identity into the gallery at build; verify it at every load.
The stamp is the model file's basename plus the SHA-256 of its bytes (plus
embed_dim). The hash decides, the name explains. A name alone is a promise
rather than a fact — models get re-exported and overwritten in place under an
unchanged filename, which is exactly the case where the weights differ and
nothing else does. A hash alone is correct but unactionable in an error message.
SHA-256 is derived from the artefact, needs no registry kept current, and costs
~0.1s for a 250MB ONNX, memoised per process.
Mismatch is a hard error in every mode, with no bypass, naming both sides.
Unstamped legacy galleries warn loudly and proceed: unknown is not known-bad,
and hard-failing every pre-existing gallery would turn the check into something
people disable rather than trust. --require-gallery-stamp (or
SAE_REQUIRE_GALLERY_STAMP=1, which propagates to subprocesses) promotes that to
a hard error — the mode measurement work should run in. scripts/stamp_gallery.py
re-binds an existing gallery with no re-embedding, so "warn" is a cheap state to
leave rather than a permanent one.
Embedding dumps carry the same stamp: a replay has no live embedder, so the dump
is the embedder as far as the gallery is concerned. Derived galleries inherit
their source's stamp; --merge and the JSON gallery merge check before writing,
since one file holding two embedding spaces cannot be untangled afterwards.
Verified in: scene_analyze, scene_preview, the sae_kpn matcher binding,
replay.py, optimize.py (once per film at startup, before the first evaluation),
movienet_eval.py and both merge paths.
Stamp logic lives in src/gallery/embedder_stamp.{hpp,cpp} and its Python twin
scripts/sae_stamp.py, kept dependency-light so replay subprocesses do not pay
sae_gallery's requests/Pillow import to ask whether two models match.
Tests: 12 new cases in test_gallery_store.cpp covering the comparison logic,
both round trips, and the SHA-256 vectors that guarantee the C++ and hashlib
stamps agree. No ONNX or GPU required.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
47 lines
1.8 KiB
CMake
47 lines
1.8 KiB
CMake
# ── Unit tests ────────────────────────────────────────────────────────────────
|
|
# Pure, GPU-free, model-free tests. The similarity tests compile the GEMM backend
|
|
# directly with SAE_GEMM_CPU so the suite builds and runs on a machine without a
|
|
# GPU regardless of the main build's SAE_GEMM_BACKEND selection.
|
|
|
|
find_package(Catch2 3 QUIET)
|
|
if(NOT Catch2_FOUND)
|
|
include(FetchContent)
|
|
FetchContent_Declare(
|
|
Catch2
|
|
GIT_REPOSITORY https://github.com/catchorg/Catch2.git
|
|
GIT_TAG v3.5.3
|
|
)
|
|
FetchContent_MakeAvailable(Catch2)
|
|
endif()
|
|
|
|
add_executable(sae_tests
|
|
test_similarity.cpp
|
|
test_calibration.cpp
|
|
test_gallery_store.cpp
|
|
test_face_utils.cpp
|
|
test_track_gallery.cpp
|
|
test_face_tracker.cpp
|
|
${CMAKE_SOURCE_DIR}/src/backends/gemm_backend.cpp
|
|
${CMAKE_SOURCE_DIR}/src/gallery/gallery_store.cpp
|
|
${CMAKE_SOURCE_DIR}/src/gallery/embedder_stamp.cpp
|
|
)
|
|
target_include_directories(sae_tests PRIVATE ${CMAKE_SOURCE_DIR}/src)
|
|
# SAE_GEMM_CPU: build the CPU reference GEMM regardless of the main backend.
|
|
# SAE_MODELS_DIR: config.hpp (pulled in by track_gallery.hpp) bakes model paths.
|
|
target_compile_definitions(sae_tests PRIVATE
|
|
SAE_GEMM_CPU
|
|
SAE_MODELS_DIR="${SAE_MODELS_DIR}")
|
|
# gallery_store.cpp + gallery_calibration.hpp use nlohmann/json and HDF5
|
|
# (galleries are HDF5-native, see src/gallery/gallery_store.cpp); face_utils.hpp
|
|
# and the calibration GEMM pull in OpenCV (calib3d/imgproc/core) via types.hpp.
|
|
target_link_libraries(sae_tests PRIVATE
|
|
Catch2::Catch2WithMain
|
|
nlohmann_json::nlohmann_json
|
|
${OpenCV_LIBS}
|
|
${HDF5_CXX_LIBRARIES})
|
|
target_include_directories(sae_tests PRIVATE ${HDF5_INCLUDE_DIRS})
|
|
|
|
include(CTest)
|
|
include(Catch)
|
|
catch_discover_tests(sae_tests)
|