dtourolle
22758da118
Records the reference film end to end: fetching the annotations and mugshots, fusing the scene clips into one stream, building the gallery at the 66 px face floor, and the measured result (precision 1.00, recall 0.65, F1 0.79). Three things are written down because each cost time to discover: - Why Bali was withdrawn. Its reference crops have a median detected face of 27 px against a 69 px maximum, so every reference was upscaled past what the embedder was trained for (AR-011). No threshold fixed it — at 66 px, 2 of 69 references survived. Any accuracy figure recorded against Bali measures upscaling artifacts as much as the pipeline. - Run it as one film. Per-scene clips defeat per-film gallery expansion (AR-019) and pay model load ten times over. - Check you are on the GPU. ORT's CUDA provider fails to load here and falls back to CPU silently, so a build-ort timing is a CPU number wearing a GPU label — a 15x error whose only symptom is a number with no baseline. TRACES: AR-011, AR-019 | VR-001, VR-005 | SR-002
7.2 KiB
Benchmark — SuperHero
The reference film for end-to-end accuracy. Replaces Road to Bali, which was withdrawn for the reason in Why not Road to Bali.
TRACES: AR-011, AR-012, AR-013 | VR-001, VR-005 | SR-002
The film
SuperHero, from the NIST TRECVID Deep Video Understanding development set. 14 films are asserted Creative Commons and need no data agreement; only the 5 KinoLorber test films are gated.
| Runtime | 1025.5 s (17.1 min), 10 scenes |
| Resolution | 640×360 |
| Ground truth | Per-scene presence, from the scene knowledge graphs |
| Gallery | 5 characters, 14 references |
The DVU set is what makes this workable: it ships character face crops cut from the film itself, so ground truth and gallery are both in character space and scoring needs no actor→character mapping.
Licence caveat. NIST links licence evidence for only 4 of the 14 films, and SuperHero is not one of them — its end credits carry no copyright or CC notice, list a "Temporary Musical Score" and a SAG cast, and it has no traceable online release. Fine for internal benchmarking; do not redistribute frames from it. Valkaama is the one film with an independently documented licence (CC BY-SA 3.0) if provenance ever has to be defended.
Reproducing it
# 1. Annotations, character mugshots, scene segmentation.
# NIST names the same film three different ways, hence the overrides.
KG_DIR=superHero KG_FILE=superhero scripts/fetch_dvu.sh SuperHero ../dvu-hero
# 2. Scene clips (movie.shots), then fuse them into one stream.
# Fusing matters — see "Run it as one film" below.
# SuperHero-1.webm … SuperHero-10.webm from
# <dataset>/movie.shots/, then:
ffmpeg -f concat -safe 0 -i concat.txt -c copy SuperHero_full.webm
# 3. Gallery, with the face-size floor that keeps references in distribution.
./build/build_gallery --root ../dvu-hero/root \
--output ../dvu-hero/hero66.h5 --min-face-px 66
# 4. Run, on the GPU path (see "Check you are on the GPU").
./build/scene_analyze --movie hero/SuperHero_full.webm \
--gallery ../dvu-hero/hero66.h5 \
--detector-engine trt_cache/scrfd.scrfd_500m_bnkps.640.fp16.engine \
--arcface-engine trt_cache/arcface.LVFace-B_Glint360K.b4.fp16.engine \
--fps 5 --min-face-px 32 --expand-gallery \
--output pred.json
Nothing here is in git: the clips are ~130 MB and the annotations are regenerable. Replay fixtures derived from the run ship through the artifact registry instead:
scripts/artifacts/push_artifacts.sh replay-fixtures
scripts/artifacts/pull_artifacts.sh replay-fixtures [version]
The gallery travels in the same archive as the dumps deliberately — a dump only replays meaningfully against the gallery it was produced with, and pairing one with a different gallery silently changes every identity decision in it.
Results
Measured on the fused film, gallery expansion on.
| Metric | Value |
|---|---|
| Precision | 1.00 |
| Recall | 0.65 |
| F1 | 0.79 |
| True positives | 13 |
| False positives | 0 |
| False negatives | 7 |
Six of ten scenes scored exactly right, including the three-character scenes 4 and 5.
Zero false positives is the result worth keeping. Every out-of-gallery character — Beast, Mighty Celestial, Ms. Johnson, Doctor, two Masked Persons — was declined rather than forced onto a nearest match. That is the calibrated probability (AR-024) doing its job, and it is the right failure direction for an X-Ray overlay: a miss is a gap, an invention is a lie.
The misses have a shape. Scenes 1, 2, 3 and 8 were missed, and 1–3 are the three shortest scenes in the film (14 s, 38 s, 27 s). That is consistent with per-track Bayesian accumulation (AR-025) needing enough sightings before belief crosses threshold. Scene 8 is 65 s and does not fit that story — it is the one to look at first when improving recall.
Running the same scenes as isolated clips did not do better, so cross-scene gallery expansion is not currently compensating for short scenes.
Run it as one film, not as clips
Per-scene clips defeat per-film gallery expansion (AR-019), which grows a temporary gallery from track continuity across the whole film and re-assesses unknown tracks at the end. Ten isolated clips give it nothing to work with, and pay model and gallery load ten times over.
Fusing also makes presence windows cross real scene boundaries, which is how SR-002's scene-scoped question is asked in production. Note the joins are artificial cuts — consecutive scenes were never contiguous footage — so presence bleeding across a boundary may be the join rather than a tracking fault.
Throughput
| Path | Realtime factor | Sampled fps | 17-min film |
|---|---|---|---|
build/ (TensorRT) |
2.0× | ~10 | ~8 min |
build-ort/ (ORT) |
0.54× | 2.7 | ~32 min |
Throughput varies strongly with face density; a sparse stretch measured 8× realtime, so quote the whole-film average, not a window.
Check you are on the GPU
ORT's CUDA execution provider fails to load on this machine and silently falls back to CPU:
Failed to load library libonnxruntime_providers_cuda.so:
undefined symbol: cudnnGetConvolutionBackwardDataAlgorithm_v7
That symbol was removed in cuDNN 9; the packaged ORT is built against cuDNN 8.
ORT logs this once at startup and then runs happily on CPU, so a build-ort
timing is a CPU number wearing a GPU label — a 15× error with no symptom other
than a figure you have no baseline for. Grep the log for Failed to load library before trusting any throughput measurement.
The TensorRT path (build/) needs prebuilt engines from
scripts/build_trt_engines.sh and reports what it loaded:
[TrtScrfd] loaded: … [TrtArcFace] loaded: … max_batch=4
[similarity] cuBLAS/CUDA engine: gallery resident on GPU
Why not Road to Bali
Bali was chosen because DVU ships character mugshots for it. It was withdrawn on face scale, measured on its own reference crops:
| Bali | SuperHero | |
|---|---|---|
| Median detected face | 27 px | 69 px |
| Maximum detected face | 69 px | 241 px |
| References ≥66 px | 2 of 69 | 14 of 27 |
The DVU images are scene crops, not mugshots, so the crop dimensions say nothing about face scale — the face has to be detected and measured. Bali's median reference was being upscaled roughly 4× to reach ArcFace's 112×112, and the worst 7×, which violates AR-011: every model gets the input it was trained for. A model run off-distribution returns confident, plausible, wrong output.
In a gallery that error is permanent. A bad frame costs one frame; a poisoned reference corrupts every future match against that identity.
No threshold rescued it. At 66 px only 2 of 69 references survived — the largest
face in the entire set is 69 px — so there was no cut that both kept references
in distribution and left enough of them to calibrate. SuperHero's gallery builds
at a 66 px floor and calibrates on its own (a=15.2867 b=-4.98633, 100 % train
accuracy) rather than borrowing constants.
Any accuracy figure recorded against Bali predates this and should be treated as measuring upscaling artifacts as much as the pipeline.