dtourolle
d98dc2855a
Bali was chosen because the TRECVID DVU set ships character mugshots, but its reference crops are unusable at scale: median detected face 27 px against a 69 px maximum, so every reference was upscaled 4x or more past what the embedder was trained for (AR-011). A 66 px floor left 2 of 69 references; no threshold exists that both keeps the faces in distribution and leaves enough of them to calibrate. SuperHero is 69 px median and 241 px max. Its gallery builds at a 66 px floor with 14 references over 5 characters, and calibrates on its own (a=15.2867 b=-4.98633, 100% train accuracy) instead of borrowing constants. Measured on the fused 17-minute film, one stream rather than per-scene clips so presence windows cross real scene boundaries as SR-002 intends: precision 1.00, recall 0.65, F1 0.79 — 13 true positives, 0 false positives, 7 misses. Every out-of-gallery character was declined rather than forced onto a nearest match. The misses are the short scenes (14 s, 38 s, 27 s), consistent with per-track accumulation needing sightings. - build_gallery gains --min-face-px, filtering the *detected face* rather than the crop. The DVU images are scene crops, not mugshots, so crop dimensions say nothing about face scale. A poisoned reference is permanent in a way a bad frame is not: it corrupts every future match against that identity. - scripts/fetch_dvu.sh fetches mugshots, scene graphs and segmentation for any DVU film. NIST names the same film three different ways, so KG_DIR and KG_FILE are overridable rather than derived. This exists as a script because the first copy of this data was assembled ad hoc in /tmp and was lost with it, taking the working gallery along. - Replay fixtures move to the artifact registry: push/pull_artifacts.sh gain a replay-fixtures target, and tests/fixtures/dumps/.gitignore keeps them out of git. superhero.h5 is ~9 MB and regenerating it needs the film, the models and a GPU — none of which CI has. The gallery ships with the dumps, since a dump only replays against the gallery it was produced with. - AR-012 and AR-013 coverage is ported onto the new fixture rather than dropped with the Bali cases: 12369 assertions, up from 7991, since the film is an order of magnitude larger than the clips. Suite: 15679 assertions, 101 test cases. TRACES: AR-011, AR-012, AR-013 | VR-001, VR-005 | SR-002
scripts/validation — per-scene actor-presence eval
Validates the pipeline's per-scene "who's on screen" output against external
ground truth, offline. Annealing (anneal_sec) means an actor's presence is only
defined after the whole file is merged into [start,end] windows, so we cannot
score live: process → write the pipeline JSON → sample timepoints → compare
predicted vs ground-truth presence sets → micro-sum TP/FP/FN → precision/recall/F1.
Ground-truth sources
| Source | Semantics | Fair to a face pipeline? | What it measures |
|---|---|---|---|
| MovieNet-PS | on-screen face presence per shot | yes — like-for-like | recognition accuracy |
| Amazon X-Ray (Zenodo) | cast-in-scene (incl. off-camera / non-speaking) | no — penalizes by design | coverage ceiling; recall gap = actors we structurally can't see |
- MovieNet is the honest recognition number.
- X-Ray is an upper bound: its recall gap tells you how much presence is off-camera cast a face detector can never reach — not a pipeline error.
X-Ray dataset: Zenodo DOI 10.5281/zenodo.17659734 (CC-BY-4.0). Per movie it ships
people.csv, scenes.csv, people_in_scenes.csv.
Usage
# against Amazon X-Ray CSVs for one title
python scripts/validation/sample_eval.py \
--pred "Scene in a Mall.json" \
--xray /data/xray/<movie_dir> \
--gallery gallery_arcface_w600k_r50.json \
--step 1.0
# against MovieNet-PS for one title
python scripts/validation/sample_eval.py \
--pred out.json \
--movienet /data/movienet --split Train_app10 --title tt0032138 \
--gallery gallery_arcface_w600k_r50.json
Sampling modes
--step Sregular grid every S s (default 1.0) — time-weighted headline number.--random NN uniform-random timepoints (for confidence intervals).--scene-anchoredone timepoint per GT scene midpoint — the literal X-Ray "did I get this scene's cast right?" question; neutralizes long-scene bias.
Ground truth is compared raw (annealing is not applied to GT).
Matching & masking
Identity is provider-agnostic (identity.py): each actor is the set of every key
we can derive — imdb:nm…, tmdb:…, jf:…, name:<normalized>. Predicted and GT
actors match iff their key-sets intersect, so an output carrying only tmdb/jellyfin
ids still joins X-Ray's nm ids via the normalized-name fallback.
Scoring is masked to gallery ∩ GT: a GT actor absent from the gallery is
ignored (not an FN), so we measure pipeline accuracy, not gallery coverage. Without
--gallery the mask falls back to GT ∩ pred keys. --no-mask disables it.
Exact id join via the tmdb→imdb crosswalk (recommended)
The gallery/pipeline output key actors by TMDB id (no nm…), while X-Ray and
MovieNet key on IMDb. They only overlap on the fuzzy name: key by default.
Build a cached tmdb→imdb table once and pass it with --crosswalk to turn the
name join into an exact id join:
# one-time: resolve every gallery tmdb id via TMDB /person/{id}/external_ids
python scripts/validation/tmdb_imdb_map.py \
--gallery gallery_arcface_w600k_r50.json \
--out scripts/validation/tmdb_imdb.json # TMDB_API_KEY from env/.env
# then score with exact ids
python scripts/validation/sample_eval.py --pred out.json --xray <dir> \
--gallery gallery_arcface_w600k_r50.json \
--crosswalk scripts/validation/tmdb_imdb.json
The table caches nulls (tmdb ids TMDB has no IMDb id for) and checkpoints, so a
re-run only resolves new ids. TMDB is authoritative for this crosswalk — there is
no clean free bulk tmdb_person ↔ nm file, so we query the API once and cache.
Minimum face size (VR-005)
min_face_size.py is a separate, self-contained study: it needs no video and no
ground truth, only the gallery mugshot cache. It holds out one image per actor,
degrades that probe to each candidate face size and matches it against a gallery
held at native resolution, reporting TPI/FPI per size — the measurement that
replaces AR-002's 66×66 px estimate.
python scripts/validation/min_face_size.py \
--images images --gallery gallery_lvface.h5 \
--arcface models/LVFace-B_Glint360K.onnx \
--actors 100 --out experiments/results/vr005_min_face_size
FPI grows with the number of actors competing, so a 100-actor run understates it
against a library of thousands: read FPI as relative across sizes, not as an
absolute rate. Re-run per --arcface model to see whether min_face_px should be
one constant or scale with the embedder (GR-004).
Files
sample_eval.py— CLI scorer.ground_truth.py—XRayGroundTruth,MovieNetGroundTruthloaders.identity.py— provider-agnostic match keys.tmdb_imdb_map.py— build/consult the cachedtmdb→imdbcrosswalk.min_face_size.py— VR-005 probe-size sweep (see above).test_sample_eval.py— self-contained tests (python scripts/validation/test_sample_eval.py).