Bali was chosen because the TRECVID DVU set ships character mugshots, but
its reference crops are unusable at scale: median detected face 27 px
against a 69 px maximum, so every reference was upscaled 4x or more past
what the embedder was trained for (AR-011). A 66 px floor left 2 of 69
references; no threshold exists that both keeps the faces in distribution
and leaves enough of them to calibrate.
SuperHero is 69 px median and 241 px max. Its gallery builds at a 66 px
floor with 14 references over 5 characters, and calibrates on its own
(a=15.2867 b=-4.98633, 100% train accuracy) instead of borrowing constants.
Measured on the fused 17-minute film, one stream rather than per-scene
clips so presence windows cross real scene boundaries as SR-002 intends:
precision 1.00, recall 0.65, F1 0.79 — 13 true positives, 0 false
positives, 7 misses. Every out-of-gallery character was declined rather
than forced onto a nearest match. The misses are the short scenes (14 s,
38 s, 27 s), consistent with per-track accumulation needing sightings.
- build_gallery gains --min-face-px, filtering the *detected face* rather
than the crop. The DVU images are scene crops, not mugshots, so crop
dimensions say nothing about face scale. A poisoned reference is
permanent in a way a bad frame is not: it corrupts every future match
against that identity.
- scripts/fetch_dvu.sh fetches mugshots, scene graphs and segmentation for
any DVU film. NIST names the same film three different ways, so KG_DIR
and KG_FILE are overridable rather than derived. This exists as a script
because the first copy of this data was assembled ad hoc in /tmp and was
lost with it, taking the working gallery along.
- Replay fixtures move to the artifact registry: push/pull_artifacts.sh
gain a replay-fixtures target, and tests/fixtures/dumps/.gitignore keeps
them out of git. superhero.h5 is ~9 MB and regenerating it needs the
film, the models and a GPU — none of which CI has. The gallery ships
with the dumps, since a dump only replays against the gallery it was
produced with.
- AR-012 and AR-013 coverage is ported onto the new fixture rather than
dropped with the Bali cases: 12369 assertions, up from 7991, since the
film is an order of magnitude larger than the clips.
Suite: 15679 assertions, 101 test cases.
TRACES: AR-011, AR-012, AR-013 | VR-001, VR-005 | SR-002
AR-018 was marked Done while the promotion path still ran on the
constants it was meant to replace. track_gallery.hpp rejected a track
when buffer_spread (1 minus the minimum pairwise cosine) exceeded
expand_track_spread_max, and skipped a view when its raw gal_sim cleared
expand_novelty_sim. Both were bare cosines with no recorded EXCEPTION,
so both were defects under the AR-024 invariant rather than tagging gaps.
The calibrated band was real but unreachable. expand_band_lo/hi were
declared in Config and read nowhere, and set_band() had no callers, so
the gate always ran at the hardcoded 0.90/0.95 while --expand-novelty-sim
and --expand-spread-max stayed live flags.
The spread gate becomes store_coherence: the band's lower bound asked of
every pair in the store, in probability space, rather than a second
constant. admit() compares a newcomer only against its nearest existing
member, so a gradually drifting track chains A to B to C with every step
inside the band while A and C are strangers — the shape a track-ID
collision takes over a slow pan. The bound is re-asked pairwise before
anything reaches an actor's annex.
The novelty gate is deleted rather than converted. SPEC section AR-018
contrasts the band with expand_novelty_sim as the thing it replaces, and
AR-019 requires only that the band is satisfied. Novelty-seeking now
lives entirely in the eviction ordering, which ranks by similarity to the
actor's references instead of cutting at a constant, so there is nothing
left to tune but the two bounds.
BufEntry stored a raw cosine and the eviction loop compared two of them.
The map is monotonic so the ranking was never wrong, but it left a bare
cosine as a decision variable; it now stores the calibrated probability.
The [AR-018] Catch2 tag previously sat on the spread gate, reporting the
replaced mechanism as verification of its replacement. It now sits on the
band: both bounds asserted exactly, since they are inclusive and an
off-by-one there is invisible anywhere else; refusal counted on each
side; and the config bounds driven away from the shipped defaults so a
hardcoded fallback fails. The case that carries the invariant is "band
thresholds probability, not cosine" — under a calibration shifted by
0.10, cosine 0.84 is admitted and cosine 0.92 refused, the opposite of
their raw verdicts. A raw-cosine gate passes an identity-calibrated test
by accident and cannot pass that one. 15 cases, 38 assertions, passing.
scene_preview.cpp takes the flag rename because it would otherwise
reference deleted Config fields. It still does not compile, for reasons
predating this change: it also reads track_max_embed_dist and
track_max_frames_missing, retired by the earlier AR-024 tracker work, and
constructs FaceTrackerFunc with one argument where the registry and
calibration are now required.
Two notes for anyone reading the chain. The main.cpp flag rename and the
AR-018/AR-024 register rows landed in 35e7033, whose trailer names AR-004
only, so git log --grep=AR-018 will not surface them. And
docs/traceability.md is left uncommitted on purpose: regenerating it now
would bake in VR-013 rows for two experiment scripts that are not yet
committed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
TRACES: AR-018, AR-024 | SR-005
A track recognised on 318 of 385 frames was owned on none, so the truth file
named nobody while the matcher was accepting almost continuously.
The correlation discount was an annihilator rather than an attenuator. Weight
was 1 - P(same view), so once a track had one stored view every later frame of
that same face scored ~0.01 and the belief stopped moving. One observation just
over the accept threshold is logit(0.78) ~ 1.27, under the ownership bar — hence
recognised always, owned never.
Two changes, in the order they were found.
Correlated evidence is now attenuated by effective sample size,
n_eff = n / (1 + (n-1)·rho), each frame contributing the marginal gain. That has
the right shape at both ends: uncorrelated evidence accumulates linearly, and a
held pose converges on 1/rho rather than growing without bound. A constant floor
was tried first and rejected — it grows linearly forever, so a long shot could
out-argue genuinely varied evidence purely by lasting longer.
Combination is now weighted lazy-OR: P = 1 - (1-P_old)·(1-p)^w, stored as
log(1-P) so the update is additive and precision stays where it matters as P
approaches 1. Each frame is new evidence that this track is that actor, and the
belief is the probability that at least one sighting was right. It converges
faster than summing log-odds at the same effective count — 2.98 vs 2.53 after
two observations at p=0.78 — which is what a real clip needs.
Note that summing log-odds was already a correct sequential Bayesian update:
the matcher fits with prior 0.5, so logit(p) IS the per-frame log-likelihood
ratio and the running sum carries the prior forward. It was not wrong, it was
slow. What blocked ownership was the discount, not the combination rule.
Also fixes a real correctness bug: the observation count lived on the
discounter, which is shared by every track, so tracks pooled into one effective
sample and each was discounted by how many others happened to be on screen. It
is now a per-track parameter.
The registry's frame scope holds its lock for its lifetime and the mutex is not
recursive, so calling observe() inside a scope self-deadlocks. The pipeline
never does — separate nodes — but the test did, and hung rather than failing.
Documented at the call site.
Verified end to end: the same clip that produced zero actors now identifies
Bing Crosby and Dorothy Lamour with belief 0.97.
Suite: 96 cases, 6142 assertions.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
TRACES: AR-025 | SR-002
GR-003 — the calibration fit already computed per-actor dedup counts, how many
actors are eligible for positive pairs, and a 200-bin histogram of the intra and
inter distributions, then discarded all of it to stderr. Nothing persisted, so
nobody could audit whether a gallery was any good.
The report is written alongside the gallery at build time. That is the right
moment: the matcher fits the same sigmoid at analysis time, but by then the
answer is per-run and nobody is looking, whereas build time is when a gallery's
quality is actually decided.
What it surfaces, in order of usefulness:
- actors with no usable image — a silent recall ceiling, since the pipeline can
never name them and nothing else says why
- actors below the positive-pair threshold — not broken, so nothing complains;
they just quietly weaken every threshold downstream
- near-duplicate references removed, per actor and total
- the fitted calibration AND the two distributions behind it
That last one is the point. Every threshold in the pipeline is expressed in the
probability space this sigmoid defines, so if the distributions overlap heavily
the calibration is weak and every downstream decision inherits it — while the
gallery still looks fine from the outside.
The gallery-derived prior, intra/(intra+inter), is computed and reported but the
shipped default of 0.5 is deliberately left alone. The spec records these as
disagreeing; now the real value is visible, so the decision can be made on
evidence rather than argument.
Three tests: a zero-image actor is visible in the report, an under-referenced
actor is counted, and the report round-trips through JSON.
Suite: 95 cases, 6142 assertions.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
TRACES: GR-003 | SR-001
AR-019 — promotion may only borrow same-identity evidence from a span where
identity is certain, so every discontinuity signal now clears the buffers rather
than just the histogram cut.
is_scene_boundary was already named in the gate but never set by anything, so
that half of it was dead until AR-010 gave it a producer. It now does what the
spec always said. The third signal, an identity contradiction, needs no code
here: AR-015 closes a track whose belief swapped, so it can no longer promote.
Ownership now comes from the registry rather than a second tally. TrackGallery
was computing its own plurality vote over accepted frames, which meant two
different answers to "who is this track" could coexist in one run — and the
expansion one ignored the Bayesian accumulation entirely, weighting thirty
near-identical looks the same as thirty distinct ones. The local tally survives
only as a fallback for callers with no registry attached, which is the unit
tests and the replay harness.
Suite: 92 cases, 6133 assertions.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
TRACES: AR-019, AR-010, AR-015 | SR-005
AR-018 — an embedding joins a track's store only if its similarity to something
already there falls inside a band, rather than merely being far from the gallery.
Above the upper bound it is redundant: another look at a pose the store already
covers, teaching the annex nothing while costing a slot a novel view could have
used. Below the lower bound it is suspect: within one track every face is the
same person by construction, so an embedding unlike everything else on the track
is evidence that construction failed — a track-ID collision or a bad detection.
Admitting it is exactly how an actor's annex gets poisoned with someone else's
face.
The old gate had only the upper half of that idea, expressed as a raw cosine
against the gallery. Both bounds are now calibrated probabilities (AR-024), so
the same number means the same thing here as in association and evidence
weighting rather than three different things.
This catches track-ID collisions EARLIER than the spread gate did — at the door
rather than at promotion — so the buffer never becomes two-person in the first
place. The spread gate stays as a second line for a track that drifts gradually
instead of jumping. The existing test was asserting the mechanism rather than
the outcome, so it was rewritten to assert what actually matters: whichever gate
fires, the outsider must not reach the annex.
Rejections are counted. A store that admits nothing is as broken as one that
admits everything, and neither is visible otherwise.
Band defaults 0.90-0.95 are working values pending VR-007; the two bounds fail in
opposite directions and must be swept separately.
Suite: 92 cases, 6133 assertions.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
TRACES: AR-018, AR-024 | SR-005
Three requirements land together because they cannot be separated. The
cross-cut revival branch was the only user of cut_revive_sim, so retiring that
raw cosine forces the pool collapse, and collapsing the pool removes the only
caller of the constant. Splitting them would have produced an intermediate
commit whose only purpose was to be split.
AR-008 — FaceTrackerFunc no longer keeps its own tracks_/inactive_ maps; it
holds a shared_ptr<TrackRegistry> and operates on it directly. Two parallel
copies of track state could disagree, and every divergence would surface as a
wrong presence window with nothing to indicate it. There is now ONE candidate
pool: last_seen alone says whether IoU is meaningful. The park/revive path is
deleted outright — matching a dormant track is ordinary inter-frame
association, and continuity falls out of the embedding comparison the tracker
already did rather than being a mechanism of its own.
AR-007 — track_alpha becomes the base weight for ordinary frames only.
Association drops to embedding-only when position carries no information:
on is_cut or is_scene_boundary, because the viewpoint changed, and for a
dormant track, because time has passed since its box was last valid. The second
case matters as much as the first and had no equivalent before.
AR-024 — association cost is a calibrated probability, never a raw cosine. The
tracker takes the calibration belonging to the active embedder, the same
function object EvidenceDiscounter uses. track_max_embed_dist becomes
track_assoc_min_prob, which means the same thing for every model, gallery and
face size, where a bare cosine threshold did not.
Retired: track_max_embed_dist, cut_revive_sim, cut_inactive_max_frames, and
track_max_frames_missing — the last superseded by the registry's extinction
window. That one is worth naming: a frame count silently changed meaning with
sample_fps, so the same configuration behaved differently at 1 fps and 5 fps.
Extinction is in seconds and lives in one place.
Tests rewritten rather than deleted. The old cases asserted revival by raw
cosine; the same behaviours are now asserted through the registry — a face lost
across a cut and re-associated is the SAME track, one unbroken window, and a
face returning past the extinction window is not. Added the case AR-007 exists
for: two people swap screen positions across a cut while keeping their faces,
and identity must follow the embedding rather than the box.
Suite: 80 cases, 3250 assertions.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
TRACES: AR-007, AR-008, AR-024 | SR-002
Adds TRACES tags to code that already satisfies a Done requirement, so coverage
reflects what exists rather than starting from zero:
AR-001 face detection, AR-005 ArcFace alignment, AR-023 calibration fit,
DP-001/DP-002 the single analysis core behind the CLI, IR-001 truth-file
emission, IR-006 the Jellyfin round trip, GR-001/GR-002 gallery build and
incremental merge, VR-001 the embedding dump, VR-002 replay through the real
nodes, VR-003 per-second scoring.
Only Done requirements are tagged. A tag on Planned work would inflate coverage
with fiction that looks plausible — the same failure family as a gate that
cannot fail, and harder to spot.
GR-005 (gallery never leaves the instance) stays untagged deliberately: it is a
prohibition satisfied by the absence of an egress path, so there is no unit that
decides it. Same shape as PR-005 in the system spec, which has no software row
for the same reason. A goal held only by prohibitions cannot be verified by
pointing at code.
Coverage 5/63 to 14/63. The three VR tags are reported as tagged-but-unexecuted
and excluded from the numerator, since their tier cannot run on the CI host —
tagging deliberately cannot raise the number on its own.
Suite still 64 cases, 3199 assertions.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
TRACES: AR-001, AR-005, AR-023, DP-001, DP-002, IR-001, IR-006, GR-001, GR-002, VR-001, VR-002, VR-003
A gallery is only valid for the embedder that produced its vectors. Cosine
similarities across models are meaningless but *look* plausible, so the mistake
is silent and every measurement taken afterwards is suspect. Stamp the embedder
identity into the gallery at build; verify it at every load.
The stamp is the model file's basename plus the SHA-256 of its bytes (plus
embed_dim). The hash decides, the name explains. A name alone is a promise
rather than a fact — models get re-exported and overwritten in place under an
unchanged filename, which is exactly the case where the weights differ and
nothing else does. A hash alone is correct but unactionable in an error message.
SHA-256 is derived from the artefact, needs no registry kept current, and costs
~0.1s for a 250MB ONNX, memoised per process.
Mismatch is a hard error in every mode, with no bypass, naming both sides.
Unstamped legacy galleries warn loudly and proceed: unknown is not known-bad,
and hard-failing every pre-existing gallery would turn the check into something
people disable rather than trust. --require-gallery-stamp (or
SAE_REQUIRE_GALLERY_STAMP=1, which propagates to subprocesses) promotes that to
a hard error — the mode measurement work should run in. scripts/stamp_gallery.py
re-binds an existing gallery with no re-embedding, so "warn" is a cheap state to
leave rather than a permanent one.
Embedding dumps carry the same stamp: a replay has no live embedder, so the dump
is the embedder as far as the gallery is concerned. Derived galleries inherit
their source's stamp; --merge and the JSON gallery merge check before writing,
since one file holding two embedding spaces cannot be untangled afterwards.
Verified in: scene_analyze, scene_preview, the sae_kpn matcher binding,
replay.py, optimize.py (once per film at startup, before the first evaluation),
movienet_eval.py and both merge paths.
Stamp logic lives in src/gallery/embedder_stamp.{hpp,cpp} and its Python twin
scripts/sae_stamp.py, kept dependency-light so replay subprocesses do not pay
sae_gallery's requests/Pillow import to ask whether two models match.
Tests: 12 new cases in test_gallery_store.cpp covering the comparison logic,
both round trips, and the SHA-256 vectors that guarantee the C++ and hashlib
stamps agree. No ONNX or GPU required.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
New C++ sources:
- kpn_bindings.cpp (sae_kpn): assembles the real face_tracker/identity_matcher/
scene_tracker nodes inside a Python-driven KPN network via nanobind, for
offline threshold-sweep replay against dumped embeddings (scripts/optimizer/).
- track_gallery.hpp: per-film gallery expansion — promotes a confidently-
identified track's novel-pose reference views into an in-memory annex so
later frames/tracks of that actor at similar poses are recognised, without
touching the baked gallery.
- dump_embeddings.cpp: standalone exe that runs detect→embed only (no gallery,
no matching) and dumps per-frame face embeddings + metadata to HDF5, so a
parameter sweep can replay the expensive half once and vary tracking/matching
config freely downstream.
- scene_detector.hpp / scene_detector_node.hpp: TransNetV2-based shot-boundary
detection, opt-in alongside the always-on histogram cut detector.
- camera_position_change_detector_node.hpp, embedding_dump_node.hpp: supporting
nodes for the above.
Gallery format switches from JSON to HDF5 exclusively (JSON read-only kept for
back-compat): save_gallery always writes HDF5, and the fitted Platt-sigmoid
calibration (a, b, valid, hash) is now embedded directly in the gallery file
instead of a sidecar .calib_cache.json — identity_matcher reads it from the
loaded gallery and writes back only when the embeddings actually changed
(hash mismatch), skipping the O(n^2) refit otherwise.
Also includes: TensorRT inference backend support (ort_backend.cpp,
trt_backend.cpp), gemm_backend improvements, TransNetV2-based scene-boundary
detection wired through frame_source/face_tracker/main, and CMake build
target updates for the new sources.
Bumps the KPN submodule to feature/persistent-pipeline-reuse (push_blocking
backpressure, node_ptr/node_stats introspection, ObjectVariantNodeWrapper for
stateful functors) — needed by the optimizer's sae_kpn Python bindings.
dtourolle
