Claude
7db40f430d
A gallery is only valid for the embedder that produced its vectors. Cosine
similarities across models are meaningless but *look* plausible, so the mistake
is silent and every measurement taken afterwards is suspect. Stamp the embedder
identity into the gallery at build; verify it at every load.
The stamp is the model file's basename plus the SHA-256 of its bytes (plus
embed_dim). The hash decides, the name explains. A name alone is a promise
rather than a fact — models get re-exported and overwritten in place under an
unchanged filename, which is exactly the case where the weights differ and
nothing else does. A hash alone is correct but unactionable in an error message.
SHA-256 is derived from the artefact, needs no registry kept current, and costs
~0.1s for a 250MB ONNX, memoised per process.
Mismatch is a hard error in every mode, with no bypass, naming both sides.
Unstamped legacy galleries warn loudly and proceed: unknown is not known-bad,
and hard-failing every pre-existing gallery would turn the check into something
people disable rather than trust. --require-gallery-stamp (or
SAE_REQUIRE_GALLERY_STAMP=1, which propagates to subprocesses) promotes that to
a hard error — the mode measurement work should run in. scripts/stamp_gallery.py
re-binds an existing gallery with no re-embedding, so "warn" is a cheap state to
leave rather than a permanent one.
Embedding dumps carry the same stamp: a replay has no live embedder, so the dump
is the embedder as far as the gallery is concerned. Derived galleries inherit
their source's stamp; --merge and the JSON gallery merge check before writing,
since one file holding two embedding spaces cannot be untangled afterwards.
Verified in: scene_analyze, scene_preview, the sae_kpn matcher binding,
replay.py, optimize.py (once per film at startup, before the first evaluation),
movienet_eval.py and both merge paths.
Stamp logic lives in src/gallery/embedder_stamp.{hpp,cpp} and its Python twin
scripts/sae_stamp.py, kept dependency-light so replay subprocesses do not pay
sae_gallery's requests/Pillow import to ask whether two models match.
Tests: 12 new cases in test_gallery_store.cpp covering the comparison logic,
both round trips, and the SHA-256 vectors that guarantee the C++ and hashlib
stamps agree. No ONNX or GPU required.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
67 lines
3.3 KiB
Markdown
67 lines
3.3 KiB
Markdown
# Embedding-dump HDF5 schema (v1)
|
|
|
|
One file per analysed title. Captures the pipeline state at the `EmbeddedSceneFrame`
|
|
channel — i.e. after decode → detect → align → embed, but **before** tracking and
|
|
identity matching. Everything downstream (face tracker, identity matcher, scene
|
|
tracker/anneal) is cheap CPU math, so replaying from this file lets a parameter
|
|
sweep re-run the whole downstream tail thousands of times with no GPU and no video.
|
|
|
|
Written by the C++ dump sink (`--dump-embeddings out.h5`); read by
|
|
`scripts/optimizer/replay.py`.
|
|
|
|
## Layout
|
|
|
|
The dump is **flat/ragged**: all faces across all frames are concatenated into
|
|
per-face arrays, with a per-frame index table pointing into them. This avoids
|
|
variable-length HDF5 types and reads straight into numpy.
|
|
|
|
```
|
|
/ (root)
|
|
attrs:
|
|
schema_version : int = 1
|
|
movie : str (source video path)
|
|
sample_fps : float
|
|
embed_dim : int = 512
|
|
embedder_model : str basename of the embedding model (GR-004)
|
|
embedder_sha256: str SHA-256 of that model file (GR-004)
|
|
|
|
frames/ group — one row per sampled frame
|
|
timestamp_sec : float64 [F]
|
|
frame_idx : int64 [F]
|
|
is_cut : uint8 [F] (histogram intra-scene cut)
|
|
is_scene_boundary : uint8 [F] (TransNetV2 boundary; 0 if scene_detect off)
|
|
face_offset : int64 [F] start index into faces/* for this frame
|
|
face_count : int32 [F] number of faces in this frame
|
|
|
|
faces/ group — one row per detected face, concatenated
|
|
embedding : float32 [N, 512] L2-normalised ArcFace embedding
|
|
bbox : float32 [N, 4] x, y, w, h in original video pixels
|
|
landmarks : float32 [N, 10] 5 (x,y) pairs, SCRFD/ArcFace order
|
|
confidence : float32 [N] detector confidence
|
|
```
|
|
|
|
`F` = number of sampled frames, `N` = total faces (= sum of face_count).
|
|
Frame *i*'s faces are `faces/*[ face_offset[i] : face_offset[i]+face_count[i] ]`.
|
|
|
|
## Model binding (GR-004)
|
|
|
|
`embedder_model` / `embedder_sha256` record which embedder produced every vector
|
|
in `faces/embedding`. A replay has no live embedder, so **the dump is the embedder
|
|
as far as the gallery is concerned**: `replay.py` checks these two attributes
|
|
against the gallery's own `/embedder` stamp and refuses to run on a mismatch,
|
|
naming both sides. Cross-model cosines are meaningless but look plausible.
|
|
|
|
The attributes are additive, not a format break — `schema_version` stays 1. Dumps
|
|
written before GR-004 simply lack them, which reports as *unverifiable* (a loud
|
|
warning, or a hard error under `SAE_REQUIRE_GALLERY_STAMP=1`) rather than as a
|
|
pass. Re-dump to bind an old dump; there is no in-place migration, because unlike
|
|
a gallery nobody can assert after the fact which model produced a vector.
|
|
|
|
## Invariants
|
|
- `embedding` rows are unit-norm (cosine == dot product against the gallery).
|
|
- `face_offset[0] == 0`; `face_offset[i+1] == face_offset[i] + face_count[i]`.
|
|
- `bbox` is already mapped to original resolution (bbox_upscale applied at dump time),
|
|
matching what the identity matcher would emit.
|
|
- A frame with no faces has `face_count == 0` (still gets a row, so timestamps stay dense).
|
|
- EOF sentinel frames are NOT written.
|