scene-actor-extraction/docs/benchmark.md

# Benchmark — SuperHero

The reference film for end-to-end accuracy. Replaces Road to Bali, which was
withdrawn for the reason in [Why not Road to Bali](#why-not-road-to-bali).

TRACES: AR-011, AR-012, AR-013 | VR-001, VR-005 | SR-002

---

## The film

SuperHero, from the [NIST TRECVID Deep Video Understanding development
set](https://www-nlpir.nist.gov/projects/trecvid/dvu/dvu.development.dataset/).
14 films are asserted Creative Commons and need no data agreement; only the 5
KinoLorber test films are gated.

| | |
|---|---|
| Runtime | 1025.5 s (17.1 min), 10 scenes |
| Resolution | 640×360 |
| Ground truth | Per-scene presence, from the scene knowledge graphs |
| Gallery | 5 characters, 14 references |

The DVU set is what makes this workable: it ships **character** face crops cut
from the film itself, so ground truth and gallery are both in character space
and scoring needs no actor→character mapping.

**Licence caveat.** NIST links licence evidence for only 4 of the 14 films, and
SuperHero is not one of them — its end credits carry no copyright or CC notice,
list a "Temporary Musical Score" and a SAG cast, and it has no traceable online
release. Fine for internal benchmarking; do not redistribute frames from it.
Valkaama is the one film with an independently documented licence (CC BY-SA 3.0)
if provenance ever has to be defended.

---

## Reproducing it

```sh
# 1. Annotations, character mugshots, scene segmentation.
#    NIST names the same film three different ways, hence the overrides.
KG_DIR=superHero KG_FILE=superhero scripts/fetch_dvu.sh SuperHero ../dvu-hero

# 2. Scene clips (movie.shots), then fuse them into one stream.
#    Fusing matters — see "Run it as one film" below.
#    SuperHero-1.webm … SuperHero-10.webm from
#    <dataset>/movie.shots/, then:
ffmpeg -f concat -safe 0 -i concat.txt -c copy SuperHero_full.webm

# 3. Gallery, with the face-size floor that keeps references in distribution.
./build/build_gallery --root ../dvu-hero/root \
    --output ../dvu-hero/hero66.h5 --min-face-px 66

# 4. Run, on the GPU path (see "Check you are on the GPU").
./build/scene_analyze --movie hero/SuperHero_full.webm \
    --gallery ../dvu-hero/hero66.h5 \
    --detector-engine trt_cache/scrfd.scrfd_500m_bnkps.640.fp16.engine \
    --arcface-engine  trt_cache/arcface.LVFace-B_Glint360K.b4.fp16.engine \
    --fps 5 --min-face-px 32 --expand-gallery \
    --output pred.json
```

Nothing here is in git: the clips are ~130 MB and the annotations are
regenerable. Replay fixtures derived from the run ship through the artifact
registry instead:

```sh
scripts/artifacts/push_artifacts.sh replay-fixtures
scripts/artifacts/pull_artifacts.sh replay-fixtures [version]
```

The gallery travels in the same archive as the dumps deliberately — a dump only
replays meaningfully against the gallery it was produced with, and pairing one
with a different gallery silently changes every identity decision in it.

---

## Results

Measured on the fused film, gallery expansion on.

| Metric | Value |
|---|---|
| Precision | **1.00** |
| Recall | 0.65 |
| F1 | 0.79 |
| True positives | 13 |
| False positives | **0** |
| False negatives | 7 |

Six of ten scenes scored exactly right, including the three-character scenes 4
and 5.

**Zero false positives is the result worth keeping.** Every out-of-gallery
character — Beast, Mighty Celestial, Ms. Johnson, Doctor, two Masked Persons —
was declined rather than forced onto a nearest match. That is the calibrated
probability (AR-024) doing its job, and it is the right failure direction for an
X-Ray overlay: a miss is a gap, an invention is a lie.

**The misses have a shape.** Scenes 1, 2, 3 and 8 were missed, and 1–3 are the
three shortest scenes in the film (14 s, 38 s, 27 s). That is consistent with
per-track Bayesian accumulation (AR-025) needing enough sightings before belief
crosses threshold. Scene 8 is 65 s and does not fit that story — it is the one
to look at first when improving recall.

Running the same scenes as isolated clips did *not* do better, so cross-scene
gallery expansion is not currently compensating for short scenes.

### Run it as one film, not as clips

Per-scene clips defeat per-film gallery expansion (AR-019), which grows a
temporary gallery from track continuity across the whole film and re-assesses
unknown tracks at the end. Ten isolated clips give it nothing to work with, and
pay model and gallery load ten times over.

Fusing also makes presence windows cross real scene boundaries, which is how
SR-002's scene-scoped question is asked in production. Note the joins are
artificial cuts — consecutive scenes were never contiguous footage — so presence
bleeding across a boundary may be the join rather than a tracking fault.

---

## Throughput

| Path | Realtime factor | Sampled fps | 17-min film |
|---|---|---|---|
| `build/` (TensorRT) | **8.25×** | 41.3 | **2.1 min** |
| `build-ort/` (ORT) | 0.54× | 2.7 | ~32 min |

TensorRT figure re-measured 2026-08-04 over the whole film at `--fps 5
--min-face-px 32 --expand-gallery`: 5129 frames, 1025.4 s of film in 124.2 s
wall. Two runs agreed to 0.4% (124.2 s clean, 124.7 s under gdb). It supersedes
an earlier 2.0×; that figure predates the current tree and was not re-derived
here, so treat the gain as measured rather than explained.

Throughput varies strongly with face density, and **a short window is not a
sample of the film**. The opening 60 s benchmarks at 23.9× — decode there costs
4-6 ms/frame against a 12.35 ms whole-film mean (n=510), because seeking forward
in VP8/WebM gets dearer the deeper you go, and there are few faces. Always quote
the whole-film average.

### Where the time goes (VR-015)

Measured over the whole film, 2026-08-04:

| node | cpu_s | % of pipeline CPU | cpu/f | exec/f | stall/f | in% | out% |
|---|---|---|---|---|---|---|---|
| **embedder** | **91.0** | **60%** | 17.74 | 21.61 | 3.87 | 12 | 0 |
| **face_detector** ▶ | 41.4 | 27% | 8.07 | 24.20 | **16.14** | **99** | **0** |
| frame_source | 12.1 | 8% | 2.36 | 11.26 | 8.90 | — | 97 |
| camera_pos | 3.2 | 2% | 0.63 | 0.64 | 0.01 | 97 | 99 |
| face_aligner | 1.7 | 1% | 0.33 | 0.34 | 0.01 | 0 | 12 |
| identity_matcher | 1.3 | 1% | 0.26 | 0.34 | 0.09 | 0 | 0 |
| tracker / sink | 0.5 | <1% | — | — | — | 0 | 0 |

**`face_detector` paces the run**: its input channel is 97.8% full while its
output is 99.4% empty — everything upstream jammed, everything downstream
starved. It occupies 5129 × 24.20 ms ≈ 124.1 s of a 124.2 s run, essentially
100% wall occupancy, yet only 33% of that is CPU. The other 16.14 ms/frame is
device wait.

**The embedder is the larger cost but not the constraint**: 60% of all pipeline
CPU, 73% of wall as thread-busy. Whether that is real work or a spinning
`cudaStreamSynchronize` is unresolved — see the sync caveat below, which is a
one-line experiment.

**`frame_source` is the trap this table exists to defuse.** It reports
`exec/f = 11.26 ms` against `cpu/f = 2.36 ms`, and its output channel is 97%
full: it is backpressured, not expensive. The old KPN `ema` reading made it look
like the most costly node in the pipeline at 141.899 ms/frame.


`--benchmark <path>` writes a per-node timing report and prints a table at
shutdown. `hero/run_bench.sh` is `run_trt.sh` with it switched on:

```bash
./build/scene_analyze … --benchmark $H/bench_trt.json --output $H/pred_bench.json
```

**Do not read the `ema` column of the old KPN diagnostics block as a cost.** KPN
times a node across `fire_once`, which wraps the functor *and* the push to the
next channel, and a push parks when that channel is full (AR-004). A
backpressured node therefore bills its waiting to itself. On this film that
produced a genuinely inverted answer:

```
│  frame_source  frames=5132  ema=141.899ms      ← reported cost
[frame_source] decode  avg=16.6127ms  fps=60.19  ← actual decode
```

The source is not expensive; it is idle, holding a frame nobody has taken yet.
Optimising against that number means optimising the fastest node in the graph.

The benchmark report separates the two:

| Column | Meaning | Blind spot |
|---|---|---|
| `cpu_s`, `cpu%tot` | thread CPU time, and this node's share of all of it | a GPU wait looks like idleness |
| `cpu/f` | CPU ms per frame — backpressure cannot inflate it | as above |
| `exec/f` | wall ms per frame in the node, **including parked pushes** | overstates a blocked node |
| `stall/f` | `exec/f − cpu/f`: parked, or waiting on a device | does not say which |
| `in%`, `out%` | mean fill of the node's input and output channels | — |
| `press` | `in% − out%`; **the node marked ▶ is pacing the run** | not a cost, an ordering |

Read `press` first: work queues up in front of the bottleneck and starves
everything after it, so the pacing node is the one with a full input and an empty
output. Then read `cpu%run` to decide the repair — a saturated thread means the
work itself must get cheaper, while an idle thread under pressure means the node
is waiting on the GPU or the disk, where batch size and engine precision are the
knobs and the C++ is not.

Channel fills are sampled every 100 ms (`--benchmark-interval-ms`) because
`current_fill` is instantaneous: by shutdown every channel has drained, so a
single read at the end reports an idle pipeline no matter how congested it was.

#### Check the GPU is not throttled before comparing anything

**On this hardware, thermal state moves the result more than any code change
we are likely to make.** The same binary measured **8.25× cool and 3.12× once
heat-soaked** — a 2.6× swing — because the laptop RTX 3050 hits `SW Thermal
Slowdown` and pins the SM clock to **210 MHz out of 2100**:

```
$ nvidia-smi -q -d PERFORMANCE | grep -E "SW Power Cap|SW Thermal"
        SW Power Cap        : Active
        SW Thermal Slowdown : Active
```

A number recorded without its clock state is not comparable to any other
number, and back-to-back full-film runs guarantee the later ones are throttled.
`run_bench.sh` now records `nvidia-smi` either side of the run into
`bench_gpu.txt`; check it before believing a regression. Let the GPU idle back
to full clock between measurements, and never A/B two runs across a heat-soak.

This one cost real time here: a 2.7× "regression" was attributed to a code
change and reverted on that basis, when the change was innocent and the GPU had
simply warmed up between the two measurements.

#### `cpu_s` on a GPU node is mostly spin — measured

CUDA's default sync policy (`cudaDeviceScheduleAuto`) spin-waits before it
yields, so `cudaStreamSynchronize` charges the *calling thread's* CPU while the
GPU works. A GPU-bound node therefore reports a large `cpu_s` and reads as
CPU-bound.

`SAE_CUDA_BLOCKING_SYNC=1` switches to a blocking wait. Measured over 300 s of
film, four cases, identical otherwise:

| case | realtime | total CPU | embedder CPU |
|---|---|---|---|
| baseline | 3.29× | 103 s | 66 s |
| **`SAE_CUDA_BLOCKING_SYNC=1`** | 3.29× | **23 s** | **4 s** |
| `SAE_CV_THREADS=1` | 3.30× | 101 s | 66 s |
| both | 3.29× | 25 s | 5 s |

**94% of the embedder's CPU was spin, not work**, and 78% of the pipeline's.
Throughput is unchanged, so this is free CPU — which matters for a service
sharing a box (DP-003) and makes `cpu_s` mean what it says. Prefer it for any
run where the CPU numbers are being read.

`SAE_CV_THREADS=1` does nothing measurable: the only OpenCV-heavy node is
`face_aligner` at 1-2% of the pipeline, so the TBB arena is not worth removing
and `warpAffine` is not worth replacing.

**Caveat: measured with the GPU clamped at 210 MHz** (see below). A device at
full clock spends less time in the sync, so the absolute spin figure will fall;
the ranking should not.

#### `cpu_s` counts one thread — mind the TBB arena

OpenCV 5 here is built against TBB, and every OpenCV module links it, so
`cv::parallel_for_` dispatches onto a TBB arena of `nproc − 1` workers (19 on the
20-core dev box; visible as `libtbb.so.12` frames in a thread dump). Since
`CLOCK_THREAD_CPUTIME_ID` is per-thread, work a node fans out that way is billed
to the TBB workers, **not** to the node.

So a node using `warpAffine`, a histogram compare or a colour conversion reads
cheaper in `cpu_s` than it really is, and the missing time appears in `stall/f`,
where it looks identical to a GPU wait. `exec/f` does capture it — the functor
does not return until the parallel region joins — so the tell is a node whose
`exec/f` far exceeds its `cpu/f` **while its output channel is empty**: that is
fan-out, not blocking.

Worth knowing for its own sake, too: 9 KPN node threads plus 19 TBB workers plus
the CUDA and NVDEC threads is heavy oversubscription on 20 cores.

The JSON carries the same data plus the run's configuration, so two runs can be
diffed directly — which is the point, when sweeping `--embed-batch`, `--fps` or
an engine precision.

### Check you are on the GPU

ORT's CUDA execution provider fails to load on this machine and **silently falls
back to CPU**:

```
Failed to load library libonnxruntime_providers_cuda.so:
  undefined symbol: cudnnGetConvolutionBackwardDataAlgorithm_v7
```

That symbol was removed in cuDNN 9; the packaged ORT is built against cuDNN 8.
ORT logs this once at startup and then runs happily on CPU, so a `build-ort`
timing is a CPU number wearing a GPU label — a 15× error with no symptom other
than a figure you have no baseline for. Grep the log for `Failed to load
library` before trusting any throughput measurement.

The TensorRT path (`build/`) needs prebuilt engines from
`scripts/build_trt_engines.sh` and reports what it loaded:

```
[TrtScrfd] loaded: … [TrtArcFace] loaded: … max_batch=4
[similarity] cuBLAS/CUDA engine: gallery resident on GPU
```

---

## Why not Road to Bali

Bali was chosen because DVU ships character mugshots for it. It was withdrawn on
**face scale**, measured on its own reference crops:

| | Bali | SuperHero |
|---|---|---|
| Median detected face | 27 px | **69 px** |
| Maximum detected face | 69 px | **241 px** |
| References ≥66 px | 2 of 69 | 14 of 27 |

The DVU images are scene crops, not mugshots, so the crop dimensions say nothing
about face scale — the face has to be detected and measured. Bali's median
reference was being upscaled roughly 4× to reach ArcFace's 112×112, and the
worst 7×, which violates AR-011: every model gets the input it was trained for.
A model run off-distribution returns confident, plausible, wrong output.

In a gallery that error is permanent. A bad frame costs one frame; a poisoned
reference corrupts every future match against that identity.

No threshold rescued it. At 66 px only 2 of 69 references survived — the largest
face in the entire set is 69 px — so there was no cut that both kept references
in distribution and left enough of them to calibrate. SuperHero's gallery builds
at a 66 px floor and calibrates on its own (`a=15.2867 b=-4.98633`, 100 % train
accuracy) rather than borrowing constants.

Any accuracy figure recorded against Bali predates this and should be treated as
measuring upscaling artifacts as much as the pipeline.