Deletes the Python ports of SCRFDDecoder, ArcFaceEmbedder, align_face,
enhance_for_retry and calibrate_gallery, and calls the shipped C++
instead. 297 lines removed, 108 added.
The ports existed because sae_embed only exposed embed(path), so a
caller could not embed a crop it had degraded. That gap is closed:
detect(), align_face(), enhance_for_retry(), embed_crop()/embed_crops()
and GalleryCalibration are bound now, so there is no longer a reason to
keep a second implementation of any of them.
The calibration is the one that mattered. A parallel copy of the sigmoid
is precisely where "always the calibrated probability, never a raw
cosine" (AR-024) breaks without anyone noticing — the copy goes on
returning plausible numbers after the original has moved. Scoring
through the binding makes the rule structural rather than remembered.
Verified against the committed run: same shape, FPI 0.0% at every size,
same operating point of 32 px. Absolute rates differ by 1-2 points
because this check sampled 100 actors / 574 crops against the original's
258 / 999, not because anything regressed.
Also: --providers and --batch are gone, since provider selection and
batching belong to the backend; embeds are chunked at its max_batch,
because the engine does not split an oversized request and a whole
gallery in one call asks CUDA for a multi-gigabyte buffer. DEDUP_SIM and
MIN_EMB_FOR_POSITIVE stay as mirrored constants — used only to report
the population the C++ fitted on, not to refit it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
TRACES: VR-005 | AR-024
Holds out one mugshot per actor, degrades that probe to each candidate
face size and matches it against a gallery held at native resolution,
reporting TPI/FPI per size. Replaces AR-002's 66x66 px working estimate
with a measurement. Needs no video and no ground truth beyond the
mugshot cache already on disk.
LVFace-B over 258 actors, 999 gallery embeddings, threshold 0.754:
px 12 16 20 24 32 40 48+
TPI 6.6% 46.5% 81.8% 93.4% 98.1% 99.2% 99.2%
FPI is 0.000 at every size — a face too small to identify degrades to
unidentified, never to a wrong name. rank-1 holds at >=99.6% from 24 px
up, so what fails first is the calibrated probability crossing
threshold, not the ranking.
Two limits on reading this. FPI grows with the number of actors
competing, so 258 understates it against a production library. And
detection and alignment run on the native image with only the resulting
112x112 crop degraded, so landmark error at small face sizes is excluded
by construction and the curve is an upper bound — VR-010 measures the
same question end to end, and lands well above these numbers.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
TRACES: VR-005 | AR-002
trtexec rejects --minShapes/--optShapes/--maxShapes for a fully static model
("Static model does not take explicit shapes"). TransNetV2's input is fixed at
1x100x27x48x3, so the shape comes from the model itself.
Gallery build now over-fetches TMDB/Wikidata candidates by a configurable
factor: near-duplicate stills (the same photo at different crops or
resolutions) are discarded after embedding, so downloading exactly
images_per_actor left actors short of that many *distinct* embeddings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
build_trt_engines.sh hardcoded 'input.1' for the ArcFace and SCRFD shape
profiles, which only matches arcface_w600k_{r50,mbf}. Building engines for
any other embedder failed with:
Cannot find input tensor with name "input.1" in the network inputs!
Input names differ per model: LVFace-B_Glint360K uses 'data', arcface_r18
uses 'input', arcface_w600k_{r50,mbf} use 'input.1'. This matters now that
LVFace-B is the default embedder (src/config.hpp), so ARCFACE_MODEL=<LVFace>
is the expected path.
Read the name from each model via onnxruntime at build time.
Replaces narrative claims with verified numbers across all report pages:
- Cross-model held-out validation (LVFace/mbf/r18, all 5 held-out
films): LVFace wins every film outright, not just "consistent with"
the training-set pick. r50 dropped from the detailed comparison
(gallery has ~30% fewer reference images per actor than the other
three models on identical source photos).
- Per-film training breakdown: LVFace does not win every training
film (mbf beats it on Lord of War); the 75.3% macro figure hides a
10.7pp spread.
- Gallery coverage computed per film (20.3%-78.6%) instead of one
flat 67%-missing average.
- Found and fixed a real scoring bug in optimize.py: a candidate
whose hardest film's replay timed out was averaged over survivors
instead of penalized, silently rewarding partial coverage. Affected
3 of 16 training combos; corrected throughout, and optimize.py now
scores an incomplete evaluation f1=0.0 instead of averaging over
whichever films happened to finish.
- Every FPI frame in the deep dive now comes from the proper montage
renderer (Onscreen/Offscreen panel, ghosts never drawn as boxes),
never the bare-box debug overlay used earlier.
- Every distinct out-of-cast name across all 9 films gets its own
frame at its first appearance (9 names, 4 films), not a
single-example spot check: 2 ground-truth gaps, 1 photograph
misread as a person, 6 genuine lookalike confusions.
- New methodology.md: the scene-level-vs-per-second scoring mismatch
that the rest of the report assumes, written out once.
- Cut the deadlock/gdb debugging narrative from the experiment log;
kept the one fact that matters (KPN's node/network split lets the
expensive GPU stage run once and the cheap stage replay against
cached embeddings).
- Plain declarative style throughout, no em dashes, no blog voice.
- switch report frames to the scene best/worst montage renderer
(Onscreen/Offscreen panels + TPI/FPI/FN legend): perfect-second hero,
wedding couple, funeral 19-of-20, polygraph bridging, crew-scene FN
ceiling, Robert Patrick ground-truth gap, rapid-cut double label,
Herbie Hancock on an in-fiction screen
- deep dive restructured: extinction bridging framed as designed
behavior with a measurable cost (debug overlay draws the boxes; the
shipped output is presence windows), plus the face-vs-presence
ceiling and two X-Ray-is-wrong exhibits
- Material polish: light/dark palette toggle, landing-page grid cards,
figure/caption CSS, how-to-read admonition; site_url set so 404 links
resolve under the Pages subpath
- README: perfect-second and screen-call frames committed (gitignore
exceptions), readme_example.jpg retired
- build_site.sh: stage_frame helper downscales montage frames to 1920px
and pulls any missing montage-frames packages
- experiment_charts.py generates 4 figures from experiments/ artifacts:
held-out per-film F1, 16-combo ranking, DE search landscape, and the
Downton detector-vs-tracker ghost timeline (replaces the blank
title-card screenshot)
- new frames: 19-correct wedding shot (success case), Many Saints
ghost-vs-unknown frame (three error classes in one image)
- rename rep4-optimizer-results.md -> model-bakeoff.md; rep4 kept only
as the on-disk artifact prefix, explained once
- repo file references are now links via https://REPOLINK/<path>
placeholders; build_site.sh pins them to the HEAD commit's raw URLs
and fails the build if a linked path doesn't exist at HEAD
- drop references to removed scripts (scene_score.py, score_config.py)
and to session-memory names; mark artifact-registry paths with their
pull commands
- commit readme_example.jpg + pipeline_topology.svg so README renders
on the plain Gitea repo view
- deploy_pages.sh: push built site/ to the gitea-pages branch
Splits the rep4 write-up's key findings into their own linkable pages:
- best-model.md: calibration curves first (discriminative power, independent
of any threshold), then F1 on the benchmark — LVFace-B Glint360K wins both.
- gallery-scope.md: whole vs. cast-restricted gallery, isolated from model and
expansion choice — restriction wins on every axis, but isn't a shipped
runtime feature yet.
- pose-expansion.md: the training-set expand_gallery effect, and the held-out
replication attempt that found it doesn't reproduce (5 films, 2 models,
after catching and fixing a replay-timeout truncation bug and a bbox
first-match-instead-of-best-match bug in the comparison harness itself). An
honest null result, with the methodology errors documented since they're
exactly the kind that manufacture a false "it works!" finding.
- lvface-deep-dive.md: the winning model's held-out generalization gap, its
two failure modes (frozen-bbox ghost tracks), and a verified case (cross-
checked against Jellyfin's independent cast metadata) where LVFace
correctly identified an actor that X-Ray's ground truth failed to credit.
Adds a "report-highlights" artifact-registry package (scripts/artifacts/
push_artifacts.sh, pull_artifacts.sh) for hand-picked illustrative frames that
aren't reproducible via the automated best/worst montage selection, and wires
pulling it into scripts/docs/build_site.sh.
Optimizer (scripts/optimizer/): replay.py runs the real C++ tracker/matcher/
scene_tracker chain over a dumped-embeddings HDF5 via sae_kpn, so a threshold
sweep never re-decodes video or re-embeds faces. optimize.py drives scipy's
differential_evolution over the knob space, with DE-level parallelism
(multiple population candidates evaluated concurrently via a ThreadPoolExecutor)
on top of per-film replay parallelism. second_score.py is the per-second X-Ray
scoring metric (TPI/FPI/FN, out-of-cast misID weighted 10x, fair recall masked
to gallery-known cast) that superseded an earlier scene-union metric.
dump_error_frames.py / dump_scene_montage.py extract annotated video frames
(bounding boxes, TPI/FPI/FN captions, onscreen-vs-offscreen split) for visual
review of a replay against ground truth. Gallery utilities: cast_restrict.py,
gallery_membership.py, fetch_missing_actors.py, reembed_gallery.py.
scripts/validation/: X-Ray ground-truth loading and provider-agnostic identity
matching (identity.py's keys_for — an actor is the union of every id we can
derive, since pipeline output and ground truth don't share one id space).
scripts/artifacts/: push/pull scripts for the Gitea generic package registry —
galleries, montage frames, and experiment data (manifests/trajectories/results)
are pushed there instead of committed, since none are needed to run the app,
only benchmarks. Versioned by git short-SHA.
scripts/docs/: MkDocs site build (build_site.sh) and the calibration-curve
comparison chart (calibration_chart.py, matplotlib, reads each gallery's
embedded calibration).
Gallery-building scripts (make_jellyfin_gallery.py, make_gallery.py,
filter_gallery.py, run_from_jellyfin.py, movienet_eval.py, movienet_prep.py,
sae_gallery.py) updated to read/write HDF5 galleries exclusively, matching the
engine-side format switch. run_from_jellyfin.py and the optimizer no longer
carry movie source paths in shared manifests (some source filenames include
scene-release tags) — resolved locally via a gitignored file-lut.json instead.
scene_gap_hist.py scans scene_analyze output JSONs and, for every actor,
computes the gap (next_scene_start - prev_scene_end) between consecutive
scenes, emitting a text histogram of the distribution. Used to inform the
anneal_sec default.
movienet_eval: replace the per-element dot() with numpy — actor references are
loaded once as an ndarray and scored with a single matmul, keeping a
whole-library gallery fast.
movienet_prep: count and report frames referenced by annotations but absent
from Image.zip instead of skipping them silently.
Add two cameo hunters that flag actors recognised in a title but absent from
its cast:
- cameo_jellyfin.py — pure-Jellyfin cast-membership check (no id cross-walk)
- cameo_hunt.py — TMDB filmography check (actor's combined_credits)
run_from_jellyfin.py now stamps the analysed title's Jellyfin item GUID into
the output JSON as top-level 'jellyfin_item_id' (scene_analyze can't know it),
which cameo_jellyfin.py uses to look up the cast in Jellyfin's own id space.
Document that field in the result-sink output schema header.
Consolidate copy-pasted logic across the gallery/run scripts into shared
modules:
- sae_env.py — zero-dependency .env loader (populates os.environ)
- sae_tmdb.py — TMDB API helpers (tmdb_get, person images, id lookups)
- sae_jellyfin.py— Jellyfin API helpers (jf_get, id/URL normalisation)
- sae_gallery.py — image download + gallery.json writing
make_gallery, make_jellyfin_gallery and filter_gallery now import these
instead of carrying their own near-identical copies.
dtourolleandClaude Opus 5