GR-004: bind galleries to the embedder that built them
A gallery is only valid for the embedder that produced its vectors. Cosine
similarities across models are meaningless but *look* plausible, so the mistake
is silent and every measurement taken afterwards is suspect. Stamp the embedder
identity into the gallery at build; verify it at every load.
The stamp is the model file's basename plus the SHA-256 of its bytes (plus
embed_dim). The hash decides, the name explains. A name alone is a promise
rather than a fact — models get re-exported and overwritten in place under an
unchanged filename, which is exactly the case where the weights differ and
nothing else does. A hash alone is correct but unactionable in an error message.
SHA-256 is derived from the artefact, needs no registry kept current, and costs
~0.1s for a 250MB ONNX, memoised per process.
Mismatch is a hard error in every mode, with no bypass, naming both sides.
Unstamped legacy galleries warn loudly and proceed: unknown is not known-bad,
and hard-failing every pre-existing gallery would turn the check into something
people disable rather than trust. --require-gallery-stamp (or
SAE_REQUIRE_GALLERY_STAMP=1, which propagates to subprocesses) promotes that to
a hard error — the mode measurement work should run in. scripts/stamp_gallery.py
re-binds an existing gallery with no re-embedding, so "warn" is a cheap state to
leave rather than a permanent one.
Embedding dumps carry the same stamp: a replay has no live embedder, so the dump
is the embedder as far as the gallery is concerned. Derived galleries inherit
their source's stamp; --merge and the JSON gallery merge check before writing,
since one file holding two embedding spaces cannot be untangled afterwards.
Verified in: scene_analyze, scene_preview, the sae_kpn matcher binding,
replay.py, optimize.py (once per film at startup, before the first evaluation),
movienet_eval.py and both merge paths.
Stamp logic lives in src/gallery/embedder_stamp.{hpp,cpp} and its Python twin
scripts/sae_stamp.py, kept dependency-light so replay subprocesses do not pay
sae_gallery's requests/Pillow import to ask whether two models match.
Tests: 12 new cases in test_gallery_store.cpp covering the comparison logic,
both round trips, and the SHA-256 vectors that guarantee the C++ and hashlib
stamps agree. No ONNX or GPU required.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -34,6 +34,7 @@ from scipy.optimize import differential_evolution
|
||||
REPO = Path(__file__).resolve().parent.parent.parent
|
||||
sys.path.insert(0, str(REPO / "scripts" / "optimizer"))
|
||||
sys.path.insert(0, str(REPO / "scripts" / "validation"))
|
||||
sys.path.insert(0, str(REPO / "scripts"))
|
||||
|
||||
import json as _json
|
||||
import os
|
||||
@@ -56,6 +57,8 @@ DE_WORKERS = int(os.environ.get("DE_WORKERS", "1"))
|
||||
|
||||
from second_score import score_seconds # noqa: E402 uniform per-second TPI/FPI scoring
|
||||
from sample_eval import load_gallery_keys # noqa: E402
|
||||
from replay import dump_embedder_stamp # noqa: E402
|
||||
from sae_stamp import EmbedderMismatch, verify_gallery_stamp # noqa: E402
|
||||
|
||||
_GAL_KEYS: dict = {} # gallery path → key set (fair-recall FN mask), loaded once
|
||||
_REPLAY_TIMEOUT = 45 # seconds per film; a wedged replay is killed, not left to hang
|
||||
@@ -180,7 +183,15 @@ def main():
|
||||
p.add_argument("--seed", type=int, default=0)
|
||||
p.add_argument("--trajectory", help="write every evaluation here (JSON lines)")
|
||||
p.add_argument("--out", help="write best config + metrics")
|
||||
# TRACES: GR-004 | SR-001
|
||||
p.add_argument("--require-gallery-stamp", action="store_true",
|
||||
help="unprovable gallery/dump model binding is a hard error, "
|
||||
"not a warning (also via SAE_REQUIRE_GALLERY_STAMP=1)")
|
||||
args = p.parse_args()
|
||||
if args.require_gallery_stamp:
|
||||
# Set the env var rather than threading a flag through cfg: replays run as
|
||||
# subprocesses and inherit it, so strictness cannot be lost in the handoff.
|
||||
os.environ["SAE_REQUIRE_GALLERY_STAMP"] = "1"
|
||||
|
||||
films = json.loads(Path(args.manifest).read_text())
|
||||
for f in films:
|
||||
@@ -188,6 +199,18 @@ def main():
|
||||
if not Path(f["dump"]).exists():
|
||||
sys.exit(f"[opt] missing dump for {f['name']}: {f['dump']}")
|
||||
|
||||
# TRACES: GR-004 | SR-001 — every (dump, gallery) pair is checked ONCE here,
|
||||
# before the first evaluation. A DE sweep is thousands of replays; discovering
|
||||
# a cross-model pair at the end (or never) means every number it produced was
|
||||
# noise. Each replay subprocess re-checks its own pair anyway.
|
||||
for f in films:
|
||||
try:
|
||||
verify_gallery_stamp(f["gallery"], stamp=dump_embedder_stamp(f["dump"]),
|
||||
embedder_desc=f"embedding dump {Path(f['dump']).name}",
|
||||
require_stamp=args.require_gallery_stamp)
|
||||
except EmbedderMismatch as e:
|
||||
sys.exit(f"[opt] {f['name']}: {e}")
|
||||
|
||||
names, bounds = [], []
|
||||
int_knobs = {"track_max_frames_missing", "cut_inactive_max_frames"}
|
||||
for spec in args.params:
|
||||
|
||||
Reference in New Issue
Block a user