dtourolle
22758da118
Records the reference film end to end: fetching the annotations and mugshots, fusing the scene clips into one stream, building the gallery at the 66 px face floor, and the measured result (precision 1.00, recall 0.65, F1 0.79). Three things are written down because each cost time to discover: - Why Bali was withdrawn. Its reference crops have a median detected face of 27 px against a 69 px maximum, so every reference was upscaled past what the embedder was trained for (AR-011). No threshold fixed it — at 66 px, 2 of 69 references survived. Any accuracy figure recorded against Bali measures upscaling artifacts as much as the pipeline. - Run it as one film. Per-scene clips defeat per-film gallery expansion (AR-019) and pay model load ten times over. - Check you are on the GPU. ORT's CUDA provider fails to load here and falls back to CPU silently, so a build-ort timing is a CPU number wearing a GPU label — a 15x error whose only symptom is a number with no baseline. TRACES: AR-011, AR-019 | VR-001, VR-005 | SR-002
187 lines
7.2 KiB
Markdown
187 lines
7.2 KiB
Markdown
# Benchmark — SuperHero
|
||
|
||
The reference film for end-to-end accuracy. Replaces Road to Bali, which was
|
||
withdrawn for the reason in [Why not Road to Bali](#why-not-road-to-bali).
|
||
|
||
TRACES: AR-011, AR-012, AR-013 | VR-001, VR-005 | SR-002
|
||
|
||
---
|
||
|
||
## The film
|
||
|
||
SuperHero, from the [NIST TRECVID Deep Video Understanding development
|
||
set](https://www-nlpir.nist.gov/projects/trecvid/dvu/dvu.development.dataset/).
|
||
14 films are asserted Creative Commons and need no data agreement; only the 5
|
||
KinoLorber test films are gated.
|
||
|
||
| | |
|
||
|---|---|
|
||
| Runtime | 1025.5 s (17.1 min), 10 scenes |
|
||
| Resolution | 640×360 |
|
||
| Ground truth | Per-scene presence, from the scene knowledge graphs |
|
||
| Gallery | 5 characters, 14 references |
|
||
|
||
The DVU set is what makes this workable: it ships **character** face crops cut
|
||
from the film itself, so ground truth and gallery are both in character space
|
||
and scoring needs no actor→character mapping.
|
||
|
||
**Licence caveat.** NIST links licence evidence for only 4 of the 14 films, and
|
||
SuperHero is not one of them — its end credits carry no copyright or CC notice,
|
||
list a "Temporary Musical Score" and a SAG cast, and it has no traceable online
|
||
release. Fine for internal benchmarking; do not redistribute frames from it.
|
||
Valkaama is the one film with an independently documented licence (CC BY-SA 3.0)
|
||
if provenance ever has to be defended.
|
||
|
||
---
|
||
|
||
## Reproducing it
|
||
|
||
```sh
|
||
# 1. Annotations, character mugshots, scene segmentation.
|
||
# NIST names the same film three different ways, hence the overrides.
|
||
KG_DIR=superHero KG_FILE=superhero scripts/fetch_dvu.sh SuperHero ../dvu-hero
|
||
|
||
# 2. Scene clips (movie.shots), then fuse them into one stream.
|
||
# Fusing matters — see "Run it as one film" below.
|
||
# SuperHero-1.webm … SuperHero-10.webm from
|
||
# <dataset>/movie.shots/, then:
|
||
ffmpeg -f concat -safe 0 -i concat.txt -c copy SuperHero_full.webm
|
||
|
||
# 3. Gallery, with the face-size floor that keeps references in distribution.
|
||
./build/build_gallery --root ../dvu-hero/root \
|
||
--output ../dvu-hero/hero66.h5 --min-face-px 66
|
||
|
||
# 4. Run, on the GPU path (see "Check you are on the GPU").
|
||
./build/scene_analyze --movie hero/SuperHero_full.webm \
|
||
--gallery ../dvu-hero/hero66.h5 \
|
||
--detector-engine trt_cache/scrfd.scrfd_500m_bnkps.640.fp16.engine \
|
||
--arcface-engine trt_cache/arcface.LVFace-B_Glint360K.b4.fp16.engine \
|
||
--fps 5 --min-face-px 32 --expand-gallery \
|
||
--output pred.json
|
||
```
|
||
|
||
Nothing here is in git: the clips are ~130 MB and the annotations are
|
||
regenerable. Replay fixtures derived from the run ship through the artifact
|
||
registry instead:
|
||
|
||
```sh
|
||
scripts/artifacts/push_artifacts.sh replay-fixtures
|
||
scripts/artifacts/pull_artifacts.sh replay-fixtures [version]
|
||
```
|
||
|
||
The gallery travels in the same archive as the dumps deliberately — a dump only
|
||
replays meaningfully against the gallery it was produced with, and pairing one
|
||
with a different gallery silently changes every identity decision in it.
|
||
|
||
---
|
||
|
||
## Results
|
||
|
||
Measured on the fused film, gallery expansion on.
|
||
|
||
| Metric | Value |
|
||
|---|---|
|
||
| Precision | **1.00** |
|
||
| Recall | 0.65 |
|
||
| F1 | 0.79 |
|
||
| True positives | 13 |
|
||
| False positives | **0** |
|
||
| False negatives | 7 |
|
||
|
||
Six of ten scenes scored exactly right, including the three-character scenes 4
|
||
and 5.
|
||
|
||
**Zero false positives is the result worth keeping.** Every out-of-gallery
|
||
character — Beast, Mighty Celestial, Ms. Johnson, Doctor, two Masked Persons —
|
||
was declined rather than forced onto a nearest match. That is the calibrated
|
||
probability (AR-024) doing its job, and it is the right failure direction for an
|
||
X-Ray overlay: a miss is a gap, an invention is a lie.
|
||
|
||
**The misses have a shape.** Scenes 1, 2, 3 and 8 were missed, and 1–3 are the
|
||
three shortest scenes in the film (14 s, 38 s, 27 s). That is consistent with
|
||
per-track Bayesian accumulation (AR-025) needing enough sightings before belief
|
||
crosses threshold. Scene 8 is 65 s and does not fit that story — it is the one
|
||
to look at first when improving recall.
|
||
|
||
Running the same scenes as isolated clips did *not* do better, so cross-scene
|
||
gallery expansion is not currently compensating for short scenes.
|
||
|
||
### Run it as one film, not as clips
|
||
|
||
Per-scene clips defeat per-film gallery expansion (AR-019), which grows a
|
||
temporary gallery from track continuity across the whole film and re-assesses
|
||
unknown tracks at the end. Ten isolated clips give it nothing to work with, and
|
||
pay model and gallery load ten times over.
|
||
|
||
Fusing also makes presence windows cross real scene boundaries, which is how
|
||
SR-002's scene-scoped question is asked in production. Note the joins are
|
||
artificial cuts — consecutive scenes were never contiguous footage — so presence
|
||
bleeding across a boundary may be the join rather than a tracking fault.
|
||
|
||
---
|
||
|
||
## Throughput
|
||
|
||
| Path | Realtime factor | Sampled fps | 17-min film |
|
||
|---|---|---|---|
|
||
| `build/` (TensorRT) | **2.0×** | ~10 | ~8 min |
|
||
| `build-ort/` (ORT) | 0.54× | 2.7 | ~32 min |
|
||
|
||
Throughput varies strongly with face density; a sparse stretch measured 8×
|
||
realtime, so quote the whole-film average, not a window.
|
||
|
||
### Check you are on the GPU
|
||
|
||
ORT's CUDA execution provider fails to load on this machine and **silently falls
|
||
back to CPU**:
|
||
|
||
```
|
||
Failed to load library libonnxruntime_providers_cuda.so:
|
||
undefined symbol: cudnnGetConvolutionBackwardDataAlgorithm_v7
|
||
```
|
||
|
||
That symbol was removed in cuDNN 9; the packaged ORT is built against cuDNN 8.
|
||
ORT logs this once at startup and then runs happily on CPU, so a `build-ort`
|
||
timing is a CPU number wearing a GPU label — a 15× error with no symptom other
|
||
than a figure you have no baseline for. Grep the log for `Failed to load
|
||
library` before trusting any throughput measurement.
|
||
|
||
The TensorRT path (`build/`) needs prebuilt engines from
|
||
`scripts/build_trt_engines.sh` and reports what it loaded:
|
||
|
||
```
|
||
[TrtScrfd] loaded: … [TrtArcFace] loaded: … max_batch=4
|
||
[similarity] cuBLAS/CUDA engine: gallery resident on GPU
|
||
```
|
||
|
||
---
|
||
|
||
## Why not Road to Bali
|
||
|
||
Bali was chosen because DVU ships character mugshots for it. It was withdrawn on
|
||
**face scale**, measured on its own reference crops:
|
||
|
||
| | Bali | SuperHero |
|
||
|---|---|---|
|
||
| Median detected face | 27 px | **69 px** |
|
||
| Maximum detected face | 69 px | **241 px** |
|
||
| References ≥66 px | 2 of 69 | 14 of 27 |
|
||
|
||
The DVU images are scene crops, not mugshots, so the crop dimensions say nothing
|
||
about face scale — the face has to be detected and measured. Bali's median
|
||
reference was being upscaled roughly 4× to reach ArcFace's 112×112, and the
|
||
worst 7×, which violates AR-011: every model gets the input it was trained for.
|
||
A model run off-distribution returns confident, plausible, wrong output.
|
||
|
||
In a gallery that error is permanent. A bad frame costs one frame; a poisoned
|
||
reference corrupts every future match against that identity.
|
||
|
||
No threshold rescued it. At 66 px only 2 of 69 references survived — the largest
|
||
face in the entire set is 69 px — so there was no cut that both kept references
|
||
in distribution and left enough of them to calibrate. SuperHero's gallery builds
|
||
at a 66 px floor and calibrates on its own (`a=15.2867 b=-4.98633`, 100 % train
|
||
accuracy) rather than borrowing constants.
|
||
|
||
Any accuracy figure recorded against Bali predates this and should be treated as
|
||
measuring upscaling artifacts as much as the pipeline.
|