dtourolle/scene-actor-extraction
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

GR-004: bind galleries to the embedder that built them

Browse Source 
A gallery is only valid for the embedder that produced its vectors. Cosine
similarities across models are meaningless but *look* plausible, so the mistake
is silent and every measurement taken afterwards is suspect. Stamp the embedder
identity into the gallery at build; verify it at every load.

The stamp is the model file's basename plus the SHA-256 of its bytes (plus
embed_dim). The hash decides, the name explains. A name alone is a promise
rather than a fact — models get re-exported and overwritten in place under an
unchanged filename, which is exactly the case where the weights differ and
nothing else does. A hash alone is correct but unactionable in an error message.
SHA-256 is derived from the artefact, needs no registry kept current, and costs
~0.1s for a 250MB ONNX, memoised per process.

Mismatch is a hard error in every mode, with no bypass, naming both sides.

Unstamped legacy galleries warn loudly and proceed: unknown is not known-bad,
and hard-failing every pre-existing gallery would turn the check into something
people disable rather than trust. --require-gallery-stamp (or
SAE_REQUIRE_GALLERY_STAMP=1, which propagates to subprocesses) promotes that to
a hard error — the mode measurement work should run in. scripts/stamp_gallery.py
re-binds an existing gallery with no re-embedding, so "warn" is a cheap state to
leave rather than a permanent one.

Embedding dumps carry the same stamp: a replay has no live embedder, so the dump
is the embedder as far as the gallery is concerned. Derived galleries inherit
their source's stamp; --merge and the JSON gallery merge check before writing,
since one file holding two embedding spaces cannot be untangled afterwards.

Verified in: scene_analyze, scene_preview, the sae_kpn matcher binding,
replay.py, optimize.py (once per film at startup, before the first evaluation),
movienet_eval.py and both merge paths.

Stamp logic lives in src/gallery/embedder_stamp.{hpp,cpp} and its Python twin
scripts/sae_stamp.py, kept dependency-light so replay subprocesses do not pay
sae_gallery's requests/Pillow import to ask whether two models match.

Tests: 12 new cases in test_gallery_store.cpp covering the comparison logic,
both round trips, and the SHA-256 vectors that guarantee the C++ and hashlib
stamps agree. No ONNX or GPU required.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Claude
2026-07-30 18:35:46 +02:00
parent 43d2c976c3
commit 7db40f430d
30 changed files with 1392 additions and 41 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/CMakeLists.txt
+1
View File
@@ -23,6 +23,7 @@ add_executable(sae_tests
test_face_tracker.cpp
${CMAKE_SOURCE_DIR}/src/backends/gemm_backend.cpp
${CMAKE_SOURCE_DIR}/src/gallery/gallery_store.cpp
${CMAKE_SOURCE_DIR}/src/gallery/embedder_stamp.cpp
)
target_include_directories(sae_tests PRIVATE ${CMAKE_SOURCE_DIR}/src)
# SAE_GEMM_CPU: build the CPU reference GEMM regardless of the main backend.
tests/test_gallery_store.cpp
+251 -1
View File
@@ -1,13 +1,17 @@
// Unit tests for gallery (de)serialisation: HDF5 round-trip fidelity (the only
// format save_gallery writes), legacy JSON read back-compat (optional field
// defaults, the legacy "jellyfin_person_id" fallback). GPU-free, model-free.
// defaults, the legacy "jellyfin_person_id" fallback), and the GR-004 embedder
// stamp. GPU-free, model-free — the stamp tests exercise the comparison logic
// with synthetic stamps and never load an ONNX, so they run on CI's Intel N100.
#include <catch2/catch_test_macros.hpp>
#include "gallery/embedder_stamp.hpp"
#include "gallery/gallery_store.hpp"
#include "types.hpp"
#include <cstdio>
#include <fstream>
#include <stdexcept>
#include <string>
#include <nlohmann/json.hpp>
@@ -27,6 +31,22 @@ Embedding make_embedding(float base) {
return e;
}
// A stamp built by hand — no ONNX is read, so these tests never need a model.
EmbedderStamp stamp(const std::string& name, const std::string& sha, int32_t dim = 512) {
EmbedderStamp s;
s.model_name = name;
s.model_sha256 = sha;
s.embed_dim = dim;
return s;
}
const std::string kShaA(64, 'a');
const std::string kShaB(64, 'b');
bool mentions(const std::string& haystack, const std::string& needle) {
return haystack.find(needle) != std::string::npos;
}
} // namespace
TEST_CASE("gallery save/load round-trips actors and embeddings", "[gallery]") {
@@ -153,3 +173,233 @@ TEST_CASE("load_gallery reads the legacy jellyfin_person_id key", "[gallery]") {
TEST_CASE("load_gallery throws on a missing file", "[gallery]") {
CHECK_THROWS(load_gallery("/nonexistent/path/gallery.json"));
}
// ── GR-004: gallery ↔ embedder binding ───────────────────────────────────────
// Verification plan row GR-004/T1: "Mismatched embedder → hard startup error;
// error names both sides." The comparison is a pure function over two stamps, so
// none of this needs a GPU, an ONNX, or even a file.
/// TRACES: GR-004 | SR-001
TEST_CASE("gallery save/load round-trips the embedder stamp", "[gallery][GR-004]") {
ActorGallery g;
ActorGallery::Actor a;
a.name = "Stamped Actor";
a.embeddings = {make_embedding(0.3f)};
g.actors.push_back(a);
g.embedder = stamp("LVFace-B_Glint360K.onnx", kShaA);
TempFile tf("gallery_stamped.h5");
save_gallery(tf.path, g);
ActorGallery loaded = load_gallery(tf.path);
CHECK(loaded.embedder.model_name == "LVFace-B_Glint360K.onnx");
CHECK(loaded.embedder.model_sha256 == kShaA);
CHECK(loaded.embedder.embed_dim == 512);
CHECK_FALSE(loaded.embedder.empty());
}
/// TRACES: GR-004 | SR-001
TEST_CASE("a gallery written without a stamp loads as unstamped", "[gallery][GR-004]") {
// The back-compat case: pre-GR-004 files have no /embedder group at all. The
// absence must survive the round trip as an absence — a stamp naming no model
// would read as "checked and fine" to every consumer.
ActorGallery g;
ActorGallery::Actor a;
a.name = "Legacy Actor";
a.embeddings = {make_embedding(0.f)};
g.actors.push_back(a);
TempFile tf("gallery_unstamped.h5");
save_gallery(tf.path, g);
ActorGallery loaded = load_gallery(tf.path);
CHECK(loaded.embedder.empty());
}
/// TRACES: GR-004 | SR-001
TEST_CASE("legacy JSON galleries carry an optional embedder stamp", "[gallery][GR-004]") {
nlohmann::json j;
j["embedder"] = {{"model_name", "arcface_w600k_r50.onnx"},
{"model_sha256", kShaB},
{"embed_dim", 512}};
j["actors"] = nlohmann::json::array();
nlohmann::json ja;
ja["name"] = "JSON Actor";
ja["embeddings"] = nlohmann::json::array();
ja["embeddings"].push_back(std::vector<float>(512, 0.1f));
j["actors"].push_back(ja);
TempFile tf("gallery_json_stamp.json");
{ std::ofstream out(tf.path); out << j.dump(); }
ActorGallery g = load_gallery(tf.path);
CHECK(g.embedder.model_name == "arcface_w600k_r50.onnx");
CHECK(g.embedder.model_sha256 == kShaB);
}
/// TRACES: GR-004 | SR-001
TEST_CASE("matching embedder stamps pass", "[gallery][GR-004]") {
auto chk = compare_embedder_stamps(stamp("model.onnx", kShaA),
stamp("model.onnx", kShaA));
CHECK(chk.verdict == StampVerdict::match);
CHECK_FALSE(chk.fatal(false));
CHECK_FALSE(chk.fatal(true)); // a proven match is never fatal, even in strict mode
CHECK_NOTHROW(enforce_embedder_stamp(stamp("model.onnx", kShaA),
stamp("model.onnx", kShaA),
"g.h5", "model.onnx", true));
}
/// TRACES: GR-004 | SR-001
TEST_CASE("the hash decides, not the filename", "[gallery][GR-004]") {
// Same bytes under a different filename is the SAME model — a renamed or
// relocated file must not be treated as a different one.
auto same = compare_embedder_stamps(stamp("lvface.onnx", kShaA),
stamp("LVFace-B_Glint360K.onnx", kShaA));
CHECK(same.verdict == StampVerdict::match);
// Different bytes under the SAME filename is a DIFFERENT model — this is the
// in-place re-export a name-only stamp would miss entirely, and the reason the
// stamp carries a hash at all.
auto differ = compare_embedder_stamps(stamp("model.onnx", kShaA),
stamp("model.onnx", kShaB));
CHECK(differ.verdict == StampVerdict::mismatch);
}
/// TRACES: GR-004 | SR-001
TEST_CASE("mismatched embedder is fatal and names both sides", "[gallery][GR-004]") {
const auto built = stamp("LVFace-B_Glint360K.onnx", kShaA);
const auto loaded = stamp("arcface_w600k_r50.onnx", kShaB);
auto chk = compare_embedder_stamps(built, loaded, "cast.h5", "models/r50.onnx");
REQUIRE(chk.verdict == StampVerdict::mismatch);
CHECK(chk.fatal(false)); // no bypass: a mismatch is fatal in every mode
CHECK(chk.fatal(true));
// Both sides must be identifiable from the message alone.
CHECK(mentions(chk.message, "LVFace-B_Glint360K.onnx"));
CHECK(mentions(chk.message, "arcface_w600k_r50.onnx"));
CHECK(mentions(chk.message, kShaA));
CHECK(mentions(chk.message, kShaB));
CHECK(mentions(chk.message, "cast.h5"));
CHECK(mentions(chk.message, "models/r50.onnx"));
// ...and it must reach the caller as an error, not a log line.
CHECK_THROWS_AS(enforce_embedder_stamp(built, loaded, "cast.h5",
"models/r50.onnx", false),
std::runtime_error);
try {
enforce_embedder_stamp(built, loaded, "cast.h5", "models/r50.onnx", false);
FAIL("mismatch must throw");
} catch (const std::runtime_error& e) {
const std::string what = e.what();
CHECK(mentions(what, "LVFace-B_Glint360K.onnx"));
CHECK(mentions(what, "arcface_w600k_r50.onnx"));
}
}
/// TRACES: GR-004 | SR-001
TEST_CASE("differing embedding width is a mismatch", "[gallery][GR-004]") {
auto chk = compare_embedder_stamps(stamp("a.onnx", kShaA, 512),
stamp("a.onnx", kShaA, 256));
CHECK(chk.verdict == StampVerdict::mismatch);
CHECK(mentions(chk.message, "512"));
CHECK(mentions(chk.message, "256"));
}
/// TRACES: GR-004 | SR-001
TEST_CASE("an unstamped gallery warns by default and fails under strict",
"[gallery][GR-004]") {
// Decision recorded in src/gallery/embedder_stamp.hpp: unstamped is UNKNOWN,
// not known-bad, and every pre-GR-004 gallery is unstamped. Hard-failing them
// all would make the check something people disable rather than trust; so it
// warns loudly, names the risk, and is promotable to fatal for measurement runs.
EmbedderStamp none;
auto chk = compare_embedder_stamps(none, stamp("model.onnx", kShaA), "old.h5");
REQUIRE(chk.verdict == StampVerdict::unstamped);
CHECK_FALSE(chk.fatal(false));
CHECK(chk.fatal(true));
CHECK(mentions(chk.message, "old.h5"));
CHECK(mentions(chk.message, "model.onnx")); // the loaded side is still named
CHECK(mentions(chk.message, "UNKNOWN")); // ...and the gallery side is honest
CHECK_NOTHROW(enforce_embedder_stamp(none, stamp("model.onnx", kShaA),
"old.h5", "model.onnx", false));
CHECK_THROWS_AS(enforce_embedder_stamp(none, stamp("model.onnx", kShaA),
"old.h5", "model.onnx", true),
std::runtime_error);
}
/// TRACES: GR-004 | SR-001
TEST_CASE("an unidentifiable embedder against a stamped gallery is not silent",
"[gallery][GR-004]") {
// e.g. a replay whose dump predates GR-004: we know what built the gallery but
// not what produced the vectors being fed in. Unverifiable, so it must not
// report success.
auto chk = compare_embedder_stamps(stamp("model.onnx", kShaA), EmbedderStamp{},
"g.h5", "old dump.h5");
CHECK(chk.verdict == StampVerdict::unknown_embedder);
CHECK_FALSE(chk.fatal(false));
CHECK(chk.fatal(true));
CHECK(mentions(chk.message, "model.onnx"));
CHECK(mentions(chk.message, "old dump.h5"));
}
/// TRACES: GR-004 | SR-001
TEST_CASE("name-only agreement is a weak match, not a clean pass", "[gallery][GR-004]") {
// A TRT deployment can run from a prebuilt .engine with the .onnx absent, so
// no hash is computable. Names agreeing is evidence, not proof.
auto weak = compare_embedder_stamps(stamp("model.onnx", kShaA),
stamp("model.onnx", ""));
CHECK(weak.verdict == StampVerdict::weak_match);
CHECK_FALSE(weak.fatal(false));
CHECK(weak.fatal(true));
// Names disagreeing with no hash available is still a mismatch — the weaker
// evidence is enough to convict, just not to acquit.
auto bad = compare_embedder_stamps(stamp("lvface.onnx", ""),
stamp("arcface.onnx", ""));
CHECK(bad.verdict == StampVerdict::mismatch);
CHECK(mentions(bad.message, "lvface.onnx"));
CHECK(mentions(bad.message, "arcface.onnx"));
}
/// TRACES: GR-004 | SR-001
TEST_CASE("sha256 matches the published vectors", "[gallery][GR-004]") {
// Pins the in-tree FIPS 180-4 implementation against the standard vectors.
// This is what guarantees the C++ stamp and the Python (hashlib) stamp in
// scripts/sae_gallery.py agree on the same model file — without it the two
// halves of GR-004 could silently diverge and every check would be a mismatch.
CHECK(sha256_hex("") ==
"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855");
CHECK(sha256_hex("abc") ==
"ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad");
CHECK(sha256_hex("abcdbcdecdefdefgefghfghighijhijkijkljklmklmnlmnomnopnopq") ==
"248d6a61d20638b8e5c026930c3e6039a33ce45964ff2167f6ecedd419db06c1");
// Multi-block input, exercising the length-padding path past 64 bytes.
CHECK(sha256_hex(std::string(1000, 'a')) ==
"41edece42d63e8d9bf515a9ba6932e1c20cbc9f5a5d134645adb5db1b9737ea3");
}
/// TRACES: GR-004 | SR-001
TEST_CASE("make_embedder_stamp hashes a real file and degrades gracefully",
"[gallery][GR-004]") {
// Stands in for an ONNX: the stamp does not care what the bytes mean.
TempFile tf("fake_model.onnx");
{ std::ofstream out(tf.path, std::ios::binary); out << "abc"; }
EmbedderStamp s = make_embedder_stamp(tf.path);
CHECK(s.model_sha256 ==
"ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad");
CHECK_FALSE(s.model_name.empty());
CHECK(s.model_name.find('/') == std::string::npos); // basename, not full path
// A model path that does not exist still yields a comparable name-only stamp
// rather than an empty one, which is what keeps engine-only deployments usable.
EmbedderStamp missing = make_embedder_stamp("/nonexistent/models/foo.onnx");
CHECK(missing.model_name == "foo.onnx");
CHECK(missing.model_sha256.empty());
CHECK_FALSE(missing.empty());
CHECK(make_embedder_stamp("").empty());
}