VR-010 — a dump made with one detector/embedder pair was byte-indistinguishable
from one made with another, except for the two attributes GR-004 added. Replayed
against a gallery from a different model, cosine similarities are meaningless
but look entirely plausible. The register states the principle directly: a
fixture whose provenance is unknown is worse than no fixture, because it will be
trusted.
Sixteen attributes now record everything that determines the dump's content:
detector model and thresholds, min_face_px, max_faces, cut_threshold,
dense_scale, bbox_upscale, start/end, track_assoc_min_prob, and scene_detect.
scene_detect is the one that matters most. is_scene_boundary is all-zero both
when the detector found nothing and when it never ran, and those mean completely
different things to a consumer — without the flag they are indistinguishable.
No schema_version bump: new root attributes are additive and replay.py already
reads attributes with a default, so older dumps stay readable and the committed
fixtures — which predate this — still load.
Also corrects SCHEMA.md, which claimed bbox was already mapped to original
resolution at dump time. It is not; the upscale is applied downstream in the
matcher, after the dump tap. Harmless while dense_scale is 1 and silently wrong
otherwise, so bbox_upscale is now recorded and the doc says what the code does.
Verified end to end: all sixteen attributes present and correct on a freshly
generated dump.
Suite: 92 cases, 6136 assertions.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
TRACES: VR-010, VR-001 | PR-002
Five HDF5 embedding dumps from bali/ — Road to Bali (1952) — 3.6 MB total,
generated at 5 fps with a 32 px minimum face. CI never calls a model, so
inference happens on a GPU host and CI replays these as data; everything
downstream of embedding is cheap CPU maths.
Public domain is the reason this corpus rather than a convenient one: derived
fixtures can be committed, where anything cut from a copyrighted title could not
live in the repository at all.
The set covers distinct behaviours rather than being five of the same thing:
bali_28 has 9 cuts, so it exercises shot/reverse-shot association (AR-007);
bali_46 is sparse at 140 faces over 385 frames, so it exercises gaps and
extinction (AR-013); bali_13 is the busiest at 4 faces per frame; bali_31 is
short at 29s. All five recorded zero drops.
Both pinned parameters are consequences of measurements, not defaults: 5 fps
because 1 fps over a 77s clip is 77 frames, too thin for an extinction window
measured in tens of seconds; 32 px because that is the VR-005 floor, and the
corpus is 480x360 so a stricter value would reject most of what is there.
make_fixtures.sh regenerates them. Reproducibility is the requirement — a
fixture whose provenance is unknown is worse than none, because it will be
trusted. These are byte-reproducible only because of AR-004: before node
outputs blocked rather than dropped, the same command produced different dumps
run to run.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
TRACES: VR-001 | PR-002
Adds TRACES tags to code that already satisfies a Done requirement, so coverage
reflects what exists rather than starting from zero:
AR-001 face detection, AR-005 ArcFace alignment, AR-023 calibration fit,
DP-001/DP-002 the single analysis core behind the CLI, IR-001 truth-file
emission, IR-006 the Jellyfin round trip, GR-001/GR-002 gallery build and
incremental merge, VR-001 the embedding dump, VR-002 replay through the real
nodes, VR-003 per-second scoring.
Only Done requirements are tagged. A tag on Planned work would inflate coverage
with fiction that looks plausible — the same failure family as a gate that
cannot fail, and harder to spot.
GR-005 (gallery never leaves the instance) stays untagged deliberately: it is a
prohibition satisfied by the absence of an egress path, so there is no unit that
decides it. Same shape as PR-005 in the system spec, which has no software row
for the same reason. A goal held only by prohibitions cannot be verified by
pointing at code.
Coverage 5/63 to 14/63. The three VR tags are reported as tagged-but-unexecuted
and excluded from the numerator, since their tier cannot run on the CI host —
tagging deliberately cannot raise the number on its own.
Suite still 64 cases, 3199 assertions.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
TRACES: AR-001, AR-005, AR-023, DP-001, DP-002, IR-001, IR-006, GR-001, GR-002, VR-001, VR-002, VR-003
GR-004 — a gallery built with one embedding model is meaningless with another.
Cosine similarities across models are garbage but look entirely plausible, so
this fails silently and expensively; every measurement taken against a
mismatched pair would have been quietly wrong.
The stamp is the model basename plus a SHA-256 of its bytes, with embed_dim as
a cheap extra guard. The hash decides and the name explains, because neither
works alone: a name is a promise rather than a fact — models get re-exported in
place under an unchanged filename, which is exactly the case where the weights
differ and nothing else does — while a bare hash mismatch tells an operator
nothing actionable.
Mismatch is fatal in every mode with no bypass. Unstamped only warns, because
unstamped is unknown rather than known-bad, and an error firing on every legacy
gallery trains people to reach for the bypass reflexively. scripts/stamp_gallery.py
binds an existing gallery in place with no re-embedding, so the warning is a
migration step rather than a permanent state; --require-gallery-stamp promotes
it to an error once a site has migrated.
Two gaps found that would have defeated the requirement outright:
- Embedding dumps carried no stamp, so a replay — which has no live embedder —
had nothing to check the gallery against. Dumps now carry embedder_model and
embedder_sha256 as root attributes. Additive; schema_version stays 1. This is
the same gap the dump audit identified independently.
- --merge produced one file holding two embedding spaces, which no later check
can untangle. Merge paths now verify before writing.
The stamp also survives identity_matcher's calibration write-back, which would
otherwise have stripped it on the first analysis run — the check would have
worked exactly once.
Conflicts resolved additively: both branches appended a source to sae_gallery
and to the test target, and both edited the GR-004 register row.
Merged suite: 64 cases, 3199 assertions, passing on CPU with no GPU.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
TRACES: GR-004, VR-001 | SR-001
jray-project is added at scripts/vendor/jray-project and the extractor is used
from there. Only two files are repo-local: traceability.toml, which carries
everything repo-specific, and the CI workflow that invokes the vendored gate.
The extractor is deliberately NOT copied in. One implementation, parameterised
by config — a second copy would drift from the first, and the tool already
proves it works unchanged against all three registers.
Enables system_spec so PR/SR orphan checking runs: previously uncheckable,
because the system spec lived outside every component's checkout.
Gate is green at 0.0% of 63 requirements, which is correct — nothing is tagged
yet. Eleven are flagged unverifiable on this CI host and excluded from the
numerator rather than counted.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A gallery is only valid for the embedder that produced its vectors. Cosine
similarities across models are meaningless but *look* plausible, so the mistake
is silent and every measurement taken afterwards is suspect. Stamp the embedder
identity into the gallery at build; verify it at every load.
The stamp is the model file's basename plus the SHA-256 of its bytes (plus
embed_dim). The hash decides, the name explains. A name alone is a promise
rather than a fact — models get re-exported and overwritten in place under an
unchanged filename, which is exactly the case where the weights differ and
nothing else does. A hash alone is correct but unactionable in an error message.
SHA-256 is derived from the artefact, needs no registry kept current, and costs
~0.1s for a 250MB ONNX, memoised per process.
Mismatch is a hard error in every mode, with no bypass, naming both sides.
Unstamped legacy galleries warn loudly and proceed: unknown is not known-bad,
and hard-failing every pre-existing gallery would turn the check into something
people disable rather than trust. --require-gallery-stamp (or
SAE_REQUIRE_GALLERY_STAMP=1, which propagates to subprocesses) promotes that to
a hard error — the mode measurement work should run in. scripts/stamp_gallery.py
re-binds an existing gallery with no re-embedding, so "warn" is a cheap state to
leave rather than a permanent one.
Embedding dumps carry the same stamp: a replay has no live embedder, so the dump
is the embedder as far as the gallery is concerned. Derived galleries inherit
their source's stamp; --merge and the JSON gallery merge check before writing,
since one file holding two embedding spaces cannot be untangled afterwards.
Verified in: scene_analyze, scene_preview, the sae_kpn matcher binding,
replay.py, optimize.py (once per film at startup, before the first evaluation),
movienet_eval.py and both merge paths.
Stamp logic lives in src/gallery/embedder_stamp.{hpp,cpp} and its Python twin
scripts/sae_stamp.py, kept dependency-light so replay subprocesses do not pay
sae_gallery's requests/Pillow import to ask whether two models match.
Tests: 12 new cases in test_gallery_store.cpp covering the comparison logic,
both round trips, and the SHA-256 vectors that guarantee the C++ and hashlib
stamps agree. No ONNX or GPU required.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
trtexec rejects --minShapes/--optShapes/--maxShapes for a fully static model
("Static model does not take explicit shapes"). TransNetV2's input is fixed at
1x100x27x48x3, so the shape comes from the model itself.
Gallery build now over-fetches TMDB/Wikidata candidates by a configurable
factor: near-duplicate stills (the same photo at different crops or
resolutions) are discarded after embedding, so downloading exactly
images_per_actor left actors short of that many *distinct* embeddings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
build_trt_engines.sh hardcoded 'input.1' for the ArcFace and SCRFD shape
profiles, which only matches arcface_w600k_{r50,mbf}. Building engines for
any other embedder failed with:
Cannot find input tensor with name "input.1" in the network inputs!
Input names differ per model: LVFace-B_Glint360K uses 'data', arcface_r18
uses 'input', arcface_w600k_{r50,mbf} use 'input.1'. This matters now that
LVFace-B is the default embedder (src/config.hpp), so ARCFACE_MODEL=<LVFace>
is the expected path.
Read the name from each model via onnxruntime at build time.
Replaces narrative claims with verified numbers across all report pages:
- Cross-model held-out validation (LVFace/mbf/r18, all 5 held-out
films): LVFace wins every film outright, not just "consistent with"
the training-set pick. r50 dropped from the detailed comparison
(gallery has ~30% fewer reference images per actor than the other
three models on identical source photos).
- Per-film training breakdown: LVFace does not win every training
film (mbf beats it on Lord of War); the 75.3% macro figure hides a
10.7pp spread.
- Gallery coverage computed per film (20.3%-78.6%) instead of one
flat 67%-missing average.
- Found and fixed a real scoring bug in optimize.py: a candidate
whose hardest film's replay timed out was averaged over survivors
instead of penalized, silently rewarding partial coverage. Affected
3 of 16 training combos; corrected throughout, and optimize.py now
scores an incomplete evaluation f1=0.0 instead of averaging over
whichever films happened to finish.
- Every FPI frame in the deep dive now comes from the proper montage
renderer (Onscreen/Offscreen panel, ghosts never drawn as boxes),
never the bare-box debug overlay used earlier.
- Every distinct out-of-cast name across all 9 films gets its own
frame at its first appearance (9 names, 4 films), not a
single-example spot check: 2 ground-truth gaps, 1 photograph
misread as a person, 6 genuine lookalike confusions.
- New methodology.md: the scene-level-vs-per-second scoring mismatch
that the rest of the report assumes, written out once.
- Cut the deadlock/gdb debugging narrative from the experiment log;
kept the one fact that matters (KPN's node/network split lets the
expensive GPU stage run once and the cheap stage replay against
cached embeddings).
- Plain declarative style throughout, no em dashes, no blog voice.
- switch report frames to the scene best/worst montage renderer
(Onscreen/Offscreen panels + TPI/FPI/FN legend): perfect-second hero,
wedding couple, funeral 19-of-20, polygraph bridging, crew-scene FN
ceiling, Robert Patrick ground-truth gap, rapid-cut double label,
Herbie Hancock on an in-fiction screen
- deep dive restructured: extinction bridging framed as designed
behavior with a measurable cost (debug overlay draws the boxes; the
shipped output is presence windows), plus the face-vs-presence
ceiling and two X-Ray-is-wrong exhibits
- Material polish: light/dark palette toggle, landing-page grid cards,
figure/caption CSS, how-to-read admonition; site_url set so 404 links
resolve under the Pages subpath
- README: perfect-second and screen-call frames committed (gitignore
exceptions), readme_example.jpg retired
- build_site.sh: stage_frame helper downscales montage frames to 1920px
and pulls any missing montage-frames packages
- experiment_charts.py generates 4 figures from experiments/ artifacts:
held-out per-film F1, 16-combo ranking, DE search landscape, and the
Downton detector-vs-tracker ghost timeline (replaces the blank
title-card screenshot)
- new frames: 19-correct wedding shot (success case), Many Saints
ghost-vs-unknown frame (three error classes in one image)
- rename rep4-optimizer-results.md -> model-bakeoff.md; rep4 kept only
as the on-disk artifact prefix, explained once
- repo file references are now links via https://REPOLINK/<path>
placeholders; build_site.sh pins them to the HEAD commit's raw URLs
and fails the build if a linked path doesn't exist at HEAD
- drop references to removed scripts (scene_score.py, score_config.py)
and to session-memory names; mark artifact-registry paths with their
pull commands
- commit readme_example.jpg + pipeline_topology.svg so README renders
on the plain Gitea repo view
- deploy_pages.sh: push built site/ to the gitea-pages branch
Splits the rep4 write-up's key findings into their own linkable pages:
- best-model.md: calibration curves first (discriminative power, independent
of any threshold), then F1 on the benchmark — LVFace-B Glint360K wins both.
- gallery-scope.md: whole vs. cast-restricted gallery, isolated from model and
expansion choice — restriction wins on every axis, but isn't a shipped
runtime feature yet.
- pose-expansion.md: the training-set expand_gallery effect, and the held-out
replication attempt that found it doesn't reproduce (5 films, 2 models,
after catching and fixing a replay-timeout truncation bug and a bbox
first-match-instead-of-best-match bug in the comparison harness itself). An
honest null result, with the methodology errors documented since they're
exactly the kind that manufacture a false "it works!" finding.
- lvface-deep-dive.md: the winning model's held-out generalization gap, its
two failure modes (frozen-bbox ghost tracks), and a verified case (cross-
checked against Jellyfin's independent cast metadata) where LVFace
correctly identified an actor that X-Ray's ground truth failed to credit.
Adds a "report-highlights" artifact-registry package (scripts/artifacts/
push_artifacts.sh, pull_artifacts.sh) for hand-picked illustrative frames that
aren't reproducible via the automated best/worst montage selection, and wires
pulling it into scripts/docs/build_site.sh.
Optimizer (scripts/optimizer/): replay.py runs the real C++ tracker/matcher/
scene_tracker chain over a dumped-embeddings HDF5 via sae_kpn, so a threshold
sweep never re-decodes video or re-embeds faces. optimize.py drives scipy's
differential_evolution over the knob space, with DE-level parallelism
(multiple population candidates evaluated concurrently via a ThreadPoolExecutor)
on top of per-film replay parallelism. second_score.py is the per-second X-Ray
scoring metric (TPI/FPI/FN, out-of-cast misID weighted 10x, fair recall masked
to gallery-known cast) that superseded an earlier scene-union metric.
dump_error_frames.py / dump_scene_montage.py extract annotated video frames
(bounding boxes, TPI/FPI/FN captions, onscreen-vs-offscreen split) for visual
review of a replay against ground truth. Gallery utilities: cast_restrict.py,
gallery_membership.py, fetch_missing_actors.py, reembed_gallery.py.
scripts/validation/: X-Ray ground-truth loading and provider-agnostic identity
matching (identity.py's keys_for — an actor is the union of every id we can
derive, since pipeline output and ground truth don't share one id space).
scripts/artifacts/: push/pull scripts for the Gitea generic package registry —
galleries, montage frames, and experiment data (manifests/trajectories/results)
are pushed there instead of committed, since none are needed to run the app,
only benchmarks. Versioned by git short-SHA.
scripts/docs/: MkDocs site build (build_site.sh) and the calibration-curve
comparison chart (calibration_chart.py, matplotlib, reads each gallery's
embedded calibration).
Gallery-building scripts (make_jellyfin_gallery.py, make_gallery.py,
filter_gallery.py, run_from_jellyfin.py, movienet_eval.py, movienet_prep.py,
sae_gallery.py) updated to read/write HDF5 galleries exclusively, matching the
engine-side format switch. run_from_jellyfin.py and the optimizer no longer
carry movie source paths in shared manifests (some source filenames include
scene-release tags) — resolved locally via a gitignored file-lut.json instead.
scene_gap_hist.py scans scene_analyze output JSONs and, for every actor,
computes the gap (next_scene_start - prev_scene_end) between consecutive
scenes, emitting a text histogram of the distribution. Used to inform the
anneal_sec default.
movienet_eval: replace the per-element dot() with numpy — actor references are
loaded once as an ndarray and scored with a single matmul, keeping a
whole-library gallery fast.
movienet_prep: count and report frames referenced by annotations but absent
from Image.zip instead of skipping them silently.
Add two cameo hunters that flag actors recognised in a title but absent from
its cast:
- cameo_jellyfin.py — pure-Jellyfin cast-membership check (no id cross-walk)
- cameo_hunt.py — TMDB filmography check (actor's combined_credits)
run_from_jellyfin.py now stamps the analysed title's Jellyfin item GUID into
the output JSON as top-level 'jellyfin_item_id' (scene_analyze can't know it),
which cameo_jellyfin.py uses to look up the cast in Jellyfin's own id space.
Document that field in the result-sink output schema header.
Consolidate copy-pasted logic across the gallery/run scripts into shared
modules:
- sae_env.py — zero-dependency .env loader (populates os.environ)
- sae_tmdb.py — TMDB API helpers (tmdb_get, person images, id lookups)
- sae_jellyfin.py— Jellyfin API helpers (jf_get, id/URL normalisation)
- sae_gallery.py — image download + gallery.json writing
make_gallery, make_jellyfin_gallery and filter_gallery now import these
instead of carrying their own near-identical copies.
dtourolleandClaude Opus 5