GR-004: bind galleries to the embedder that built them
A gallery is only valid for the embedder that produced its vectors. Cosine
similarities across models are meaningless but *look* plausible, so the mistake
is silent and every measurement taken afterwards is suspect. Stamp the embedder
identity into the gallery at build; verify it at every load.
The stamp is the model file's basename plus the SHA-256 of its bytes (plus
embed_dim). The hash decides, the name explains. A name alone is a promise
rather than a fact — models get re-exported and overwritten in place under an
unchanged filename, which is exactly the case where the weights differ and
nothing else does. A hash alone is correct but unactionable in an error message.
SHA-256 is derived from the artefact, needs no registry kept current, and costs
~0.1s for a 250MB ONNX, memoised per process.
Mismatch is a hard error in every mode, with no bypass, naming both sides.
Unstamped legacy galleries warn loudly and proceed: unknown is not known-bad,
and hard-failing every pre-existing gallery would turn the check into something
people disable rather than trust. --require-gallery-stamp (or
SAE_REQUIRE_GALLERY_STAMP=1, which propagates to subprocesses) promotes that to
a hard error — the mode measurement work should run in. scripts/stamp_gallery.py
re-binds an existing gallery with no re-embedding, so "warn" is a cheap state to
leave rather than a permanent one.
Embedding dumps carry the same stamp: a replay has no live embedder, so the dump
is the embedder as far as the gallery is concerned. Derived galleries inherit
their source's stamp; --merge and the JSON gallery merge check before writing,
since one file holding two embedding spaces cannot be untangled afterwards.
Verified in: scene_analyze, scene_preview, the sae_kpn matcher binding,
replay.py, optimize.py (once per film at startup, before the first evaluation),
movienet_eval.py and both merge paths.
Stamp logic lives in src/gallery/embedder_stamp.{hpp,cpp} and its Python twin
scripts/sae_stamp.py, kept dependency-light so replay subprocesses do not pay
sae_gallery's requests/Pillow import to ask whether two models match.
Tests: 12 new cases in test_gallery_store.cpp covering the comparison logic,
both round trips, and the SHA-256 vectors that guarantee the C++ and hashlib
stamps agree. No ONNX or GPU required.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
+51
-2
@@ -945,8 +945,57 @@ surfaced as a build report.
|
||||
different models are meaningless but *look* plausible — this fails silently and
|
||||
expensively otherwise.
|
||||
|
||||
**Gap:** named as step 4 of the `service-conversion.md` implementation plan;
|
||||
unbuilt. This is the highest-value small fix in the document.
|
||||
### The stamp
|
||||
|
||||
Two fields, written together: the model file's **basename** and the **SHA-256 of
|
||||
its bytes** (plus `embed_dim` as a cheap extra guard). Stored as the `/embedder`
|
||||
group in the gallery HDF5, and as an optional top-level `"embedder"` object in
|
||||
the legacy JSON format.
|
||||
|
||||
The hash *decides*; the name is what a human *reads*. Neither alone is enough. A
|
||||
name is a promise rather than a fact — models get re-exported, re-quantised and
|
||||
overwritten in place under an unchanged filename, which is exactly the case where
|
||||
the weights differ and nothing else does, so a name-only stamp is blind to the
|
||||
failure it exists to catch. A hash alone is correct but unactionable: *"expected
|
||||
3f2a…, got 9c1b…"* tells an operator nothing about what to do next. SHA-256 over
|
||||
the file is derived from the artefact rather than asserted about it, needs no
|
||||
registry kept up to date, and costs ~0.1 s for a 250 MB ONNX once per process.
|
||||
|
||||
### Verdicts
|
||||
|
||||
| Verdict | When | Default | Under strict mode |
|
||||
|---|---|---|---|
|
||||
| `match` | hashes agree | proceed | proceed |
|
||||
| `weak_match` | names agree, one side unhashable | **warn** | **error** |
|
||||
| `unstamped` | gallery predates GR-004 | **warn** | **error** |
|
||||
| `unknown_embedder` | gallery stamped, embedder unidentifiable | **warn** | **error** |
|
||||
| `mismatch` | proven different models | **error** | **error** |
|
||||
|
||||
**A mismatch is fatal in every mode, with no bypass**, and the message names both
|
||||
sides — what the gallery was built with and what is loaded.
|
||||
|
||||
The three "cannot prove it" verdicts warn loudly instead, because they describe an
|
||||
*unknown* state rather than a *known-bad* one, and because every gallery built
|
||||
before this requirement is unstamped. Hard-failing all of them would make the
|
||||
check something people route around rather than trust. Strict mode
|
||||
(`--require-gallery-stamp`, or `SAE_REQUIRE_GALLERY_STAMP=1`, which propagates to
|
||||
subprocesses) promotes them to errors — that is the mode measurement work runs in.
|
||||
`scripts/stamp_gallery.py` re-binds an existing gallery without re-embedding, so
|
||||
migration costs one command; that is what makes "warn" a temporary state rather
|
||||
than a permanent one.
|
||||
|
||||
### Scope of the check
|
||||
|
||||
Embedding **dumps** carry the same stamp (`embedder_model` / `embedder_sha256`
|
||||
root attributes, `scripts/optimizer/SCHEMA.md`): a replay has no live embedder, so
|
||||
the dump *is* the embedder as far as the gallery is concerned. Derived galleries
|
||||
(filter, cast-restrict) inherit their source's stamp; `--merge` and the JSON
|
||||
gallery merge check *before* writing, since a merged file holding two embedding
|
||||
spaces cannot be untangled afterwards by any later check.
|
||||
|
||||
**Gap:** none. Stamped in `gallery_builder.cpp` and the Python builders; verified
|
||||
in `scene_analyze`, `scene_preview`, the `sae_kpn` matcher binding, `replay.py`,
|
||||
`optimize.py`, `movienet_eval.py` and the merge paths.
|
||||
|
||||
## GR-006 … GR-009 — Provenance tiers and poisoning guard
|
||||
|
||||
|
||||
+15
-5
@@ -260,13 +260,23 @@ Context crops opt-in behind `--dump-unidentified-crops`.
|
||||
|
||||
# Gallery
|
||||
|
||||
## GR-004 — Model binding
|
||||
## GR-004 — Model binding — **DONE**
|
||||
|
||||
**Depends on:** nothing. **Startable immediately, highest value per line.**
|
||||
**Depended on:** nothing. Landed before any measurement work, as intended.
|
||||
|
||||
Stamp embedder identity into the gallery at build; verify at load in
|
||||
`scene_analyze`, `replay.py` and the optimizer. Mismatch is a hard error naming
|
||||
both sides.
|
||||
Stamp = model basename + SHA-256 of the ONNX, written as the `/embedder` group at
|
||||
build time (`gallery_builder.cpp`, `sae_gallery.save_gallery_hdf5`) and verified
|
||||
at load in `scene_analyze`, `scene_preview`, the `sae_kpn` matcher binding,
|
||||
`replay.py`, `optimize.py` and `movienet_eval.py`. Mismatch is a hard error naming
|
||||
both sides, with no bypass. Embedding dumps carry the same stamp, since a replay
|
||||
has no live embedder to check against.
|
||||
|
||||
Unstamped legacy galleries **warn loudly and proceed** rather than failing:
|
||||
unknown is not known-bad, and hard-failing every pre-existing gallery would turn
|
||||
the check into something people disable. `--require-gallery-stamp` /
|
||||
`SAE_REQUIRE_GALLERY_STAMP=1` promotes that to a hard error — measurement runs
|
||||
should set it. `scripts/stamp_gallery.py` re-binds an existing gallery without
|
||||
re-embedding, so the warning state is cheap to leave.
|
||||
|
||||
Cross-model similarities are meaningless but *look* plausible — this fails
|
||||
silently and expensively, and it would corrupt every measurement taken during the
|
||||
|
||||
@@ -87,7 +87,7 @@ Status: `Done` · `In Progress` · `Planned` · `TBD` · `Withdrawn`
|
||||
| GR-001 | Build gallery from Jellyfin library cast, TMDB profile fallback | SR-001, SR-005 | High | Done |
|
||||
| GR-002 | Incremental `--merge` refresh without re-embedding known actors | PR-003 | High | Done |
|
||||
| GR-003 | Report coverage: zero-image actors, under-referenced actors, dedup, calibration PDFs | SR-001 | Medium | Planned |
|
||||
| GR-004 | Stamp embedder identity into the gallery; **hard startup error** on mismatch | SR-001 | High | Planned |
|
||||
| GR-004 | Stamp embedder identity into the gallery; **hard startup error** on mismatch | SR-001 | High | Done |
|
||||
| GR-005 | Gallery data never leaves the instance | **SR-005** | High | Done |
|
||||
| GR-006 | Provenance tiers: baked / harvested / confirmed, distinguishable per embedding | SR-005 | High | Planned |
|
||||
| GR-007 | Persist harvested embeddings **flagged and reviewable**, never silently equal to baked | SR-005 | Medium | Planned |
|
||||
@@ -232,7 +232,7 @@ because it will be trusted.
|
||||
| IR-001/002 | T1 | Serialised output matches golden file | Zero-length window; actor with many windows |
|
||||
| IR-003 | T1 | Output written after deferred pass | Not at EOF |
|
||||
| IR-004/005 | **T1** | Signature matches golden vector bit-for-bit | **Media < 120 s → no signature**; identical result in both repos |
|
||||
| GR-004 | T1 | Mismatched embedder → hard startup error | Error names both sides |
|
||||
| GR-004 | T1 | Mismatched embedder → hard startup error | Error names both sides; **unstamped gallery warns, and errors under `SAE_REQUIRE_GALLERY_STAMP`**; same filename + different SHA-256 must still be a mismatch |
|
||||
| GR-008 | T1 | Outlier flagged among an actor's references | Injected poisoned embedding detected |
|
||||
| VR-009 | T1 | Posterior calibration holds | A 0.99 posterior is wrong ~1% of the time on held-out tracks |
|
||||
|
||||
|
||||
Reference in New Issue
Block a user