dtourolle/scene-actor-extraction
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

GR-004: bind galleries to the embedder that built them

Browse Source 
A gallery is only valid for the embedder that produced its vectors. Cosine
similarities across models are meaningless but *look* plausible, so the mistake
is silent and every measurement taken afterwards is suspect. Stamp the embedder
identity into the gallery at build; verify it at every load.

The stamp is the model file's basename plus the SHA-256 of its bytes (plus
embed_dim). The hash decides, the name explains. A name alone is a promise
rather than a fact — models get re-exported and overwritten in place under an
unchanged filename, which is exactly the case where the weights differ and
nothing else does. A hash alone is correct but unactionable in an error message.
SHA-256 is derived from the artefact, needs no registry kept current, and costs
~0.1s for a 250MB ONNX, memoised per process.

Mismatch is a hard error in every mode, with no bypass, naming both sides.

Unstamped legacy galleries warn loudly and proceed: unknown is not known-bad,
and hard-failing every pre-existing gallery would turn the check into something
people disable rather than trust. --require-gallery-stamp (or
SAE_REQUIRE_GALLERY_STAMP=1, which propagates to subprocesses) promotes that to
a hard error — the mode measurement work should run in. scripts/stamp_gallery.py
re-binds an existing gallery with no re-embedding, so "warn" is a cheap state to
leave rather than a permanent one.

Embedding dumps carry the same stamp: a replay has no live embedder, so the dump
is the embedder as far as the gallery is concerned. Derived galleries inherit
their source's stamp; --merge and the JSON gallery merge check before writing,
since one file holding two embedding spaces cannot be untangled afterwards.

Verified in: scene_analyze, scene_preview, the sae_kpn matcher binding,
replay.py, optimize.py (once per film at startup, before the first evaluation),
movienet_eval.py and both merge paths.

Stamp logic lives in src/gallery/embedder_stamp.{hpp,cpp} and its Python twin
scripts/sae_stamp.py, kept dependency-light so replay subprocesses do not pay
sae_gallery's requests/Pillow import to ask whether two models match.

Tests: 12 new cases in test_gallery_store.cpp covering the comparison logic,
both round trips, and the SHA-256 vectors that guarantee the C++ and hashlib
stamps agree. No ONNX or GPU required.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Claude
2026-07-30 18:35:46 +02:00
parent 43d2c976c3
commit 7db40f430d
30 changed files with 1392 additions and 41 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/optimizer/SCHEMA.md
+16
View File
@@ -22,6 +22,8 @@ variable-length HDF5 types and reads straight into numpy.
movie : str (source video path)
sample_fps : float
embed_dim : int = 512
embedder_model : str basename of the embedding model (GR-004)
embedder_sha256: str SHA-256 of that model file (GR-004)
frames/ group — one row per sampled frame
timestamp_sec : float64 [F]
@@ -41,6 +43,20 @@ variable-length HDF5 types and reads straight into numpy.
`F` = number of sampled frames, `N` = total faces (= sum of face_count).
Frame *i*'s faces are `faces/*[ face_offset[i] : face_offset[i]+face_count[i] ]`.
## Model binding (GR-004)
`embedder_model` / `embedder_sha256` record which embedder produced every vector
in `faces/embedding`. A replay has no live embedder, so **the dump is the embedder
as far as the gallery is concerned**: `replay.py` checks these two attributes
against the gallery's own `/embedder` stamp and refuses to run on a mismatch,
naming both sides. Cross-model cosines are meaningless but look plausible.
The attributes are additive, not a format break — `schema_version` stays 1. Dumps
written before GR-004 simply lack them, which reports as *unverifiable* (a loud
warning, or a hard error under `SAE_REQUIRE_GALLERY_STAMP=1`) rather than as a
pass. Re-dump to bind an old dump; there is no in-place migration, because unlike
a gallery nobody can assert after the fact which model produced a vector.
## Invariants
- `embedding` rows are unit-norm (cosine == dot product against the gallery).
- `face_offset[0] == 0`; `face_offset[i+1] == face_offset[i] + face_count[i]`.
scripts/optimizer/fetch_missing_actors.py
+13 -2
View File
@@ -35,7 +35,9 @@ REPO = Path(__file__).resolve().parent.parent.parent
sys.path.insert(0, str(REPO / "scripts"))
import sae_env # noqa: E402 loads .env
from sae_tmdb import tmdb_get, tmdb_person_for_imdb, TMDB_IMG # noqa: E402
from sae_gallery import download_images, wikidata_image_urls # noqa: E402
from sae_embed_loader import resolve_arcface # noqa: E402
from sae_gallery import (download_images, embedder_stamp, # noqa: E402
enforce_embedder_stamp, wikidata_image_urls)
from sae_embed_loader import load_embedder # noqa: E402
@@ -57,6 +59,7 @@ def fetch(missing_path, out_path, token, build_dir, models_dir, arcface,
src = "TMDB + Wikidata fallback" if use_wikidata else "TMDB"
print(f"[fetch] {len(missing)} missing actors to resolve via {src}", file=sys.stderr)
embedder = load_embedder(build_dir, models_dir, arcface)
stamp = embedder_stamp(resolve_arcface(models_dir, arcface)) # TRACES: GR-004 | SR-001
img_root = Path(tempfile.mkdtemp(prefix="missing_gallery_"))
actors = []
@@ -103,7 +106,9 @@ def fetch(missing_path, out_path, token, build_dir, models_dir, arcface,
f"(wiki={n_via_wikidata}) no_tmdb={n_no_tmdb} no_img={n_no_img} "
f"no_face={n_no_face}", file=sys.stderr)
Path(out_path).write_text(json.dumps({"actors": actors}, indent=2))
# TRACES: GR-004 | SR-001 — the legacy JSON gallery carries the same stamp as
# the HDF5 one; src/gallery/gallery_store.cpp reads it from either.
Path(out_path).write_text(json.dumps({"embedder": stamp, "actors": actors}, indent=2))
n_emb = sum(len(a["embeddings"]) for a in actors)
print(f"\n[fetch] recovered {n_resolved}/{len(missing)} actors "
f"({n_via_wikidata} via Wikidata), {n_emb} embeddings → {out_path}",
@@ -115,6 +120,12 @@ def fetch(missing_path, out_path, token, build_dir, models_dir, arcface,
def merge(base_path, add_path, out_path):
base = json.loads(Path(base_path).read_text())
add = json.loads(Path(add_path).read_text())
# TRACES: GR-004 | SR-001 — merging two galleries from different models makes
# ONE file containing two incompatible embedding spaces. Nothing downstream can
# ever untangle that, so this is the one place the check must run before, not
# after, the write.
enforce_embedder_stamp(base.get("embedder"), add.get("embedder"),
str(base_path), str(add_path))
have = {a.get("imdb_id") for a in base["actors"] if a.get("imdb_id")}
added = [a for a in add["actors"] if a.get("imdb_id") not in have]
base["actors"].extend(added)
scripts/optimizer/optimize.py
+23
View File
@@ -34,6 +34,7 @@ from scipy.optimize import differential_evolution
REPO = Path(__file__).resolve().parent.parent.parent
sys.path.insert(0, str(REPO / "scripts" / "optimizer"))
sys.path.insert(0, str(REPO / "scripts" / "validation"))
sys.path.insert(0, str(REPO / "scripts"))
import json as _json
import os
@@ -56,6 +57,8 @@ DE_WORKERS = int(os.environ.get("DE_WORKERS", "1"))
from second_score import score_seconds # noqa: E402 uniform per-second TPI/FPI scoring
from sample_eval import load_gallery_keys # noqa: E402
from replay import dump_embedder_stamp # noqa: E402
from sae_stamp import EmbedderMismatch, verify_gallery_stamp # noqa: E402
_GAL_KEYS: dict = {} # gallery path → key set (fair-recall FN mask), loaded once
_REPLAY_TIMEOUT = 45 # seconds per film; a wedged replay is killed, not left to hang
@@ -180,7 +183,15 @@ def main():
p.add_argument("--seed", type=int, default=0)
p.add_argument("--trajectory", help="write every evaluation here (JSON lines)")
p.add_argument("--out", help="write best config + metrics")
# TRACES: GR-004 | SR-001
p.add_argument("--require-gallery-stamp", action="store_true",
help="unprovable gallery/dump model binding is a hard error, "
"not a warning (also via SAE_REQUIRE_GALLERY_STAMP=1)")
args = p.parse_args()
if args.require_gallery_stamp:
# Set the env var rather than threading a flag through cfg: replays run as
# subprocesses and inherit it, so strictness cannot be lost in the handoff.
os.environ["SAE_REQUIRE_GALLERY_STAMP"] = "1"
films = json.loads(Path(args.manifest).read_text())
for f in films:
@@ -188,6 +199,18 @@ def main():
if not Path(f["dump"]).exists():
sys.exit(f"[opt] missing dump for {f['name']}: {f['dump']}")
# TRACES: GR-004 | SR-001 — every (dump, gallery) pair is checked ONCE here,
# before the first evaluation. A DE sweep is thousands of replays; discovering
# a cross-model pair at the end (or never) means every number it produced was
# noise. Each replay subprocess re-checks its own pair anyway.
for f in films:
try:
verify_gallery_stamp(f["gallery"], stamp=dump_embedder_stamp(f["dump"]),
embedder_desc=f"embedding dump {Path(f['dump']).name}",
require_stamp=args.require_gallery_stamp)
except EmbedderMismatch as e:
sys.exit(f"[opt] {f['name']}: {e}")
names, bounds = [], []
int_knobs = {"track_max_frames_missing", "cut_inactive_max_frames"}
for spec in args.params:
scripts/optimizer/reembed_gallery.py
+9 -3
View File
@@ -27,8 +27,9 @@ from pathlib import Path
REPO = Path(__file__).resolve().parent.parent.parent
sys.path.insert(0, str(REPO / "scripts"))
from sae_embed_loader import load_embedder # noqa: E402
from sae_gallery import load_gallery_hdf5, save_gallery_hdf5 # noqa: E402
from sae_embed_loader import load_embedder, resolve_arcface # noqa: E402
from sae_gallery import (embedder_stamp, load_gallery_hdf5, # noqa: E402
save_gallery_hdf5)
def find_dir(images_root: Path, jellyfin_id: str, name: str) -> Path | None:
@@ -58,6 +59,11 @@ def main():
ref = load_gallery_hdf5(Path(args.ref))
images_root = Path(args.images)
embedder = load_embedder(args.build_dir, args.models_dir, args.arcface)
# TRACES: GR-004 | SR-001 — this script exists to produce a gallery in a
# DIFFERENT model's space from the reference. The output must therefore never
# inherit the reference's stamp; it carries the stamp of --arcface, which is
# the whole point of the bake-off being safe to run.
stamp = embedder_stamp(resolve_arcface(args.models_dir, args.arcface))
out_actors = []
n_ok = n_nodir = n_noemb = 0
@@ -84,7 +90,7 @@ def main():
print(f" [{i}/{total}] ok={n_ok} no_dir={n_nodir} no_emb={n_noemb}",
file=sys.stderr)
save_gallery_hdf5({"actors": out_actors}, Path(args.out))
save_gallery_hdf5({"actors": out_actors}, Path(args.out), stamp)
n_emb = sum(len(a["embeddings"]) for a in out_actors)
print(f"[reembed] {Path(args.arcface).stem}: {n_ok}/{total} actors, {n_emb} embeddings "
f"→ {args.out}", file=sys.stderr)
scripts/optimizer/replay.py
+33 -1
View File
@@ -25,6 +25,22 @@ import h5py
import numpy as np
REPO = Path(__file__).resolve().parent.parent.parent
sys.path.insert(0, str(REPO / "scripts"))
from sae_stamp import verify_gallery_stamp # noqa: E402
def dump_embedder_stamp(dump_path: str) -> dict:
"""The GR-004 embedder stamp recorded in an embedding dump.
A replay has no live embedder — the dump IS the embedder as far as the gallery
is concerned, so the dump's stamp is what the gallery must be checked against.
Dumps written before GR-004 have no attributes and yield an empty stamp, which
the check reports as unverifiable rather than silently accepting."""
with h5py.File(dump_path, "r") as f:
name = f.attrs.get("embedder_model", "")
sha = f.attrs.get("embedder_sha256", "")
dec = lambda v: v.decode() if isinstance(v, bytes) else ("" if v is None else str(v))
return {"model_name": dec(name), "model_sha256": dec(sha), "embed_dim": 512}
def load_frames(dump_path: str, min_conf: float = 0.0):
@@ -92,6 +108,15 @@ def replay(dump_path: str, gallery: str, cfg: dict, build_dir: str, stop: bool =
sys.path.insert(0, build_dir)
import sae_kpn
# TRACES: GR-004 | SR-001 — checked here, before any network is built, so a
# cross-model replay dies with one readable error instead of producing a
# plausible-looking score. add_identity_matcher re-checks it C++-side below;
# that is the backstop for any other caller of the binding.
stamp = dump_embedder_stamp(dump_path)
verify_gallery_stamp(gallery, stamp=stamp,
embedder_desc=f"embedding dump {Path(dump_path).name}",
require_stamp=bool(cfg.get("require_gallery_stamp", False)))
frames, movie, fps = load_frames(dump_path, min_conf=float(cfg.get("detector_conf", 0.0)))
net = sae_kpn.Network()
@@ -122,7 +147,8 @@ def replay(dump_path: str, gallery: str, cfg: dict, build_dir: str, stop: bool =
cap = len(frames) * 2 + 64
sae_kpn.add_node_python(net, "replay", source, [], ["EmbeddedSceneFrame"], cap)
sae_kpn.add_face_tracker(net, "tracker", cfg, cap)
sae_kpn.add_identity_matcher(net, "matcher", gallery, cfg, cap)
sae_kpn.add_identity_matcher(net, "matcher", gallery, cfg, cap,
stamp["model_name"], stamp["model_sha256"])
sae_kpn.add_scene_tracker(net, "scene", cfg, cap)
net.connect("replay", 0, "tracker", 0)
net.connect("tracker", 0, "matcher", 0)
@@ -221,11 +247,17 @@ def main():
# per-film gallery expansion: promotes pose-varied views of confidently-identified
# actors into an in-memory annex, recovering ~+4 recall at no precision cost.
p.add_argument("--expand-gallery", action="store_true")
# TRACES: GR-004 | SR-001 — promote an unprovable gallery/dump binding from a
# loud warning to a hard error. Measurement sweeps should set this (or
# SAE_REQUIRE_GALLERY_STAMP=1) so no number comes from an unbound pair.
p.add_argument("--require-gallery-stamp", action="store_true")
args = p.parse_args()
cfg = {k: getattr(args, k) for k in CFG_KEYS if getattr(args, k) is not None}
if args.expand_gallery:
cfg["expand_gallery"] = True
if args.require_gallery_stamp:
cfg["require_gallery_stamp"] = True
# stop=True: PyNode::stop() sets stop_flag_ before joining, so the source
# thread's run_loop actually exits. stop=False skips that, leaving stop_flag_
# false forever — the PyNode destructor's jthread.join() then blocks forever