# Benchmark — SuperHero The reference film for end-to-end accuracy. Replaces Road to Bali, which was withdrawn for the reason in [Why not Road to Bali](#why-not-road-to-bali). TRACES: AR-011, AR-012, AR-013 | VR-001, VR-005 | SR-002 --- ## The film SuperHero, from the [NIST TRECVID Deep Video Understanding development set](https://www-nlpir.nist.gov/projects/trecvid/dvu/dvu.development.dataset/). 14 films are asserted Creative Commons and need no data agreement; only the 5 KinoLorber test films are gated. | | | |---|---| | Runtime | 1025.5 s (17.1 min), 10 scenes | | Resolution | 640×360 | | Ground truth | Per-scene presence, from the scene knowledge graphs | | Gallery | 5 characters, 14 references | The DVU set is what makes this workable: it ships **character** face crops cut from the film itself, so ground truth and gallery are both in character space and scoring needs no actor→character mapping. **Licence caveat.** NIST links licence evidence for only 4 of the 14 films, and SuperHero is not one of them — its end credits carry no copyright or CC notice, list a "Temporary Musical Score" and a SAG cast, and it has no traceable online release. Fine for internal benchmarking; do not redistribute frames from it. Valkaama is the one film with an independently documented licence (CC BY-SA 3.0) if provenance ever has to be defended. --- ## Reproducing it ```sh # 1. Annotations, character mugshots, scene segmentation. # NIST names the same film three different ways, hence the overrides. KG_DIR=superHero KG_FILE=superhero scripts/fetch_dvu.sh SuperHero ../dvu-hero # 2. Scene clips (movie.shots), then fuse them into one stream. # Fusing matters — see "Run it as one film" below. # SuperHero-1.webm … SuperHero-10.webm from # /movie.shots/, then: ffmpeg -f concat -safe 0 -i concat.txt -c copy SuperHero_full.webm # 3. Gallery, with the face-size floor that keeps references in distribution. ./build/build_gallery --root ../dvu-hero/root \ --output ../dvu-hero/hero66.h5 --min-face-px 66 # 4. Run, on the GPU path (see "Check you are on the GPU"). ./build/scene_analyze --movie hero/SuperHero_full.webm \ --gallery ../dvu-hero/hero66.h5 \ --detector-engine trt_cache/scrfd.scrfd_500m_bnkps.640.fp16.engine \ --arcface-engine trt_cache/arcface.LVFace-B_Glint360K.b4.fp16.engine \ --fps 5 --min-face-px 32 --expand-gallery \ --output pred.json ``` Nothing here is in git: the clips are ~130 MB and the annotations are regenerable. Replay fixtures derived from the run ship through the artifact registry instead: ```sh scripts/artifacts/push_artifacts.sh replay-fixtures scripts/artifacts/pull_artifacts.sh replay-fixtures [version] ``` The gallery travels in the same archive as the dumps deliberately — a dump only replays meaningfully against the gallery it was produced with, and pairing one with a different gallery silently changes every identity decision in it. --- ## Results Measured on the fused film, gallery expansion on. | Metric | Value | |---|---| | Precision | **1.00** | | Recall | 0.65 | | F1 | 0.79 | | True positives | 13 | | False positives | **0** | | False negatives | 7 | Six of ten scenes scored exactly right, including the three-character scenes 4 and 5. **Zero false positives is the result worth keeping.** Every out-of-gallery character — Beast, Mighty Celestial, Ms. Johnson, Doctor, two Masked Persons — was declined rather than forced onto a nearest match. That is the calibrated probability (AR-024) doing its job, and it is the right failure direction for an X-Ray overlay: a miss is a gap, an invention is a lie. **The misses have a shape.** Scenes 1, 2, 3 and 8 were missed, and 1–3 are the three shortest scenes in the film (14 s, 38 s, 27 s). That is consistent with per-track Bayesian accumulation (AR-025) needing enough sightings before belief crosses threshold. Scene 8 is 65 s and does not fit that story — it is the one to look at first when improving recall. Running the same scenes as isolated clips did *not* do better, so cross-scene gallery expansion is not currently compensating for short scenes. ### Run it as one film, not as clips Per-scene clips defeat per-film gallery expansion (AR-019), which grows a temporary gallery from track continuity across the whole film and re-assesses unknown tracks at the end. Ten isolated clips give it nothing to work with, and pay model and gallery load ten times over. Fusing also makes presence windows cross real scene boundaries, which is how SR-002's scene-scoped question is asked in production. Note the joins are artificial cuts — consecutive scenes were never contiguous footage — so presence bleeding across a boundary may be the join rather than a tracking fault. --- ## Throughput | Path | Realtime factor | Sampled fps | 17-min film | |---|---|---|---| | `build/` (TensorRT) | **8.25×** | 41.3 | **2.1 min** | | `build-ort/` (ORT) | 0.54× | 2.7 | ~32 min | TensorRT figure re-measured 2026-08-04 over the whole film at `--fps 5 --min-face-px 32 --expand-gallery`: 5129 frames, 1025.4 s of film in 124.2 s wall. Two runs agreed to 0.4% (124.2 s clean, 124.7 s under gdb). It supersedes an earlier 2.0×; that figure predates the current tree and was not re-derived here, so treat the gain as measured rather than explained. Throughput varies strongly with face density, and **a short window is not a sample of the film**. The opening 60 s benchmarks at 23.9× — decode there costs 4-6 ms/frame against a 12.35 ms whole-film mean (n=510), because seeking forward in VP8/WebM gets dearer the deeper you go, and there are few faces. Always quote the whole-film average. ### Where the time goes (VR-015) Measured over the whole film, 2026-08-04: | node | cpu_s | % of pipeline CPU | cpu/f | exec/f | stall/f | in% | out% | |---|---|---|---|---|---|---|---| | **embedder** | **91.0** | **60%** | 17.74 | 21.61 | 3.87 | 12 | 0 | | **face_detector** ▶ | 41.4 | 27% | 8.07 | 24.20 | **16.14** | **99** | **0** | | frame_source | 12.1 | 8% | 2.36 | 11.26 | 8.90 | — | 97 | | camera_pos | 3.2 | 2% | 0.63 | 0.64 | 0.01 | 97 | 99 | | face_aligner | 1.7 | 1% | 0.33 | 0.34 | 0.01 | 0 | 12 | | identity_matcher | 1.3 | 1% | 0.26 | 0.34 | 0.09 | 0 | 0 | | tracker / sink | 0.5 | <1% | — | — | — | 0 | 0 | **`face_detector` paces the run**: its input channel is 97.8% full while its output is 99.4% empty — everything upstream jammed, everything downstream starved. It occupies 5129 × 24.20 ms ≈ 124.1 s of a 124.2 s run, essentially 100% wall occupancy, yet only 33% of that is CPU. The other 16.14 ms/frame is device wait. **The embedder is the larger cost but not the constraint**: 60% of all pipeline CPU, 73% of wall as thread-busy. Whether that is real work or a spinning `cudaStreamSynchronize` is unresolved — see the sync caveat below, which is a one-line experiment. **`frame_source` is the trap this table exists to defuse.** It reports `exec/f = 11.26 ms` against `cpu/f = 2.36 ms`, and its output channel is 97% full: it is backpressured, not expensive. The old KPN `ema` reading made it look like the most costly node in the pipeline at 141.899 ms/frame. `--benchmark ` writes a per-node timing report and prints a table at shutdown. `hero/run_bench.sh` is `run_trt.sh` with it switched on: ```bash ./build/scene_analyze … --benchmark $H/bench_trt.json --output $H/pred_bench.json ``` **Do not read the `ema` column of the old KPN diagnostics block as a cost.** KPN times a node across `fire_once`, which wraps the functor *and* the push to the next channel, and a push parks when that channel is full (AR-004). A backpressured node therefore bills its waiting to itself. On this film that produced a genuinely inverted answer: ``` │ frame_source frames=5132 ema=141.899ms ← reported cost [frame_source] decode avg=16.6127ms fps=60.19 ← actual decode ``` The source is not expensive; it is idle, holding a frame nobody has taken yet. Optimising against that number means optimising the fastest node in the graph. The benchmark report separates the two: | Column | Meaning | Blind spot | |---|---|---| | `cpu_s`, `cpu%tot` | thread CPU time, and this node's share of all of it | a GPU wait looks like idleness | | `cpu/f` | CPU ms per frame — backpressure cannot inflate it | as above | | `exec/f` | wall ms per frame in the node, **including parked pushes** | overstates a blocked node | | `stall/f` | `exec/f − cpu/f`: parked, or waiting on a device | does not say which | | `in%`, `out%` | mean fill of the node's input and output channels | — | | `press` | `in% − out%`; **the node marked ▶ is pacing the run** | not a cost, an ordering | Read `press` first: work queues up in front of the bottleneck and starves everything after it, so the pacing node is the one with a full input and an empty output. Then read `cpu%run` to decide the repair — a saturated thread means the work itself must get cheaper, while an idle thread under pressure means the node is waiting on the GPU or the disk, where batch size and engine precision are the knobs and the C++ is not. Channel fills are sampled every 100 ms (`--benchmark-interval-ms`) because `current_fill` is instantaneous: by shutdown every channel has drained, so a single read at the end reports an idle pipeline no matter how congested it was. #### Check the GPU is not throttled before comparing anything **On this hardware, thermal state moves the result more than any code change we are likely to make.** The same binary measured **8.25× cool and 3.12× once heat-soaked** — a 2.6× swing — because the laptop RTX 3050 hits `SW Thermal Slowdown` and pins the SM clock to **210 MHz out of 2100**: ``` $ nvidia-smi -q -d PERFORMANCE | grep -E "SW Power Cap|SW Thermal" SW Power Cap : Active SW Thermal Slowdown : Active ``` A number recorded without its clock state is not comparable to any other number, and back-to-back full-film runs guarantee the later ones are throttled. `run_bench.sh` now records `nvidia-smi` either side of the run into `bench_gpu.txt`; check it before believing a regression. Let the GPU idle back to full clock between measurements, and never A/B two runs across a heat-soak. This one cost real time here: a 2.7× "regression" was attributed to a code change and reverted on that basis, when the change was innocent and the GPU had simply warmed up between the two measurements. #### `cpu_s` on a GPU node is mostly spin — measured CUDA's default sync policy (`cudaDeviceScheduleAuto`) spin-waits before it yields, so `cudaStreamSynchronize` charges the *calling thread's* CPU while the GPU works. A GPU-bound node therefore reports a large `cpu_s` and reads as CPU-bound. `SAE_CUDA_BLOCKING_SYNC=1` switches to a blocking wait. Measured over 300 s of film, four cases, identical otherwise: | case | realtime | total CPU | embedder CPU | |---|---|---|---| | baseline | 3.29× | 103 s | 66 s | | **`SAE_CUDA_BLOCKING_SYNC=1`** | 3.29× | **23 s** | **4 s** | | `SAE_CV_THREADS=1` | 3.30× | 101 s | 66 s | | both | 3.29× | 25 s | 5 s | **94% of the embedder's CPU was spin, not work**, and 78% of the pipeline's. Throughput is unchanged, so this is free CPU — which matters for a service sharing a box (DP-003) and makes `cpu_s` mean what it says. Prefer it for any run where the CPU numbers are being read. `SAE_CV_THREADS=1` does nothing measurable: the only OpenCV-heavy node is `face_aligner` at 1-2% of the pipeline, so the TBB arena is not worth removing and `warpAffine` is not worth replacing. **Caveat: measured with the GPU clamped at 210 MHz** (see below). A device at full clock spends less time in the sync, so the absolute spin figure will fall; the ranking should not. #### `cpu_s` counts one thread — mind the TBB arena OpenCV 5 here is built against TBB, and every OpenCV module links it, so `cv::parallel_for_` dispatches onto a TBB arena of `nproc − 1` workers (19 on the 20-core dev box; visible as `libtbb.so.12` frames in a thread dump). Since `CLOCK_THREAD_CPUTIME_ID` is per-thread, work a node fans out that way is billed to the TBB workers, **not** to the node. So a node using `warpAffine`, a histogram compare or a colour conversion reads cheaper in `cpu_s` than it really is, and the missing time appears in `stall/f`, where it looks identical to a GPU wait. `exec/f` does capture it — the functor does not return until the parallel region joins — so the tell is a node whose `exec/f` far exceeds its `cpu/f` **while its output channel is empty**: that is fan-out, not blocking. Worth knowing for its own sake, too: 9 KPN node threads plus 19 TBB workers plus the CUDA and NVDEC threads is heavy oversubscription on 20 cores. The JSON carries the same data plus the run's configuration, so two runs can be diffed directly — which is the point, when sweeping `--embed-batch`, `--fps` or an engine precision. ### Check you are on the GPU ORT's CUDA execution provider fails to load on this machine and **silently falls back to CPU**: ``` Failed to load library libonnxruntime_providers_cuda.so: undefined symbol: cudnnGetConvolutionBackwardDataAlgorithm_v7 ``` That symbol was removed in cuDNN 9; the packaged ORT is built against cuDNN 8. ORT logs this once at startup and then runs happily on CPU, so a `build-ort` timing is a CPU number wearing a GPU label — a 15× error with no symptom other than a figure you have no baseline for. Grep the log for `Failed to load library` before trusting any throughput measurement. The TensorRT path (`build/`) needs prebuilt engines from `scripts/build_trt_engines.sh` and reports what it loaded: ``` [TrtScrfd] loaded: … [TrtArcFace] loaded: … max_batch=4 [similarity] cuBLAS/CUDA engine: gallery resident on GPU ``` --- ## Why not Road to Bali Bali was chosen because DVU ships character mugshots for it. It was withdrawn on **face scale**, measured on its own reference crops: | | Bali | SuperHero | |---|---|---| | Median detected face | 27 px | **69 px** | | Maximum detected face | 69 px | **241 px** | | References ≥66 px | 2 of 69 | 14 of 27 | The DVU images are scene crops, not mugshots, so the crop dimensions say nothing about face scale — the face has to be detected and measured. Bali's median reference was being upscaled roughly 4× to reach ArcFace's 112×112, and the worst 7×, which violates AR-011: every model gets the input it was trained for. A model run off-distribution returns confident, plausible, wrong output. In a gallery that error is permanent. A bad frame costs one frame; a poisoned reference corrupts every future match against that identity. No threshold rescued it. At 66 px only 2 of 69 references survived — the largest face in the entire set is 69 px — so there was no cut that both kept references in distribution and left enough of them to calibrate. SuperHero's gallery builds at a 66 px floor and calibrates on its own (`a=15.2867 b=-4.98633`, 100 % train accuracy) rather than borrowing constants. Any accuracy figure recorded against Bali predates this and should be treated as measuring upscaling artifacts as much as the pipeline.