diff --git a/docs/benchmark.md b/docs/benchmark.md new file mode 100644 index 0000000..3beb8b8 --- /dev/null +++ b/docs/benchmark.md @@ -0,0 +1,186 @@ +# Benchmark — SuperHero + +The reference film for end-to-end accuracy. Replaces Road to Bali, which was +withdrawn for the reason in [Why not Road to Bali](#why-not-road-to-bali). + +TRACES: AR-011, AR-012, AR-013 | VR-001, VR-005 | SR-002 + +--- + +## The film + +SuperHero, from the [NIST TRECVID Deep Video Understanding development +set](https://www-nlpir.nist.gov/projects/trecvid/dvu/dvu.development.dataset/). +14 films are asserted Creative Commons and need no data agreement; only the 5 +KinoLorber test films are gated. + +| | | +|---|---| +| Runtime | 1025.5 s (17.1 min), 10 scenes | +| Resolution | 640×360 | +| Ground truth | Per-scene presence, from the scene knowledge graphs | +| Gallery | 5 characters, 14 references | + +The DVU set is what makes this workable: it ships **character** face crops cut +from the film itself, so ground truth and gallery are both in character space +and scoring needs no actor→character mapping. + +**Licence caveat.** NIST links licence evidence for only 4 of the 14 films, and +SuperHero is not one of them — its end credits carry no copyright or CC notice, +list a "Temporary Musical Score" and a SAG cast, and it has no traceable online +release. Fine for internal benchmarking; do not redistribute frames from it. +Valkaama is the one film with an independently documented licence (CC BY-SA 3.0) +if provenance ever has to be defended. + +--- + +## Reproducing it + +```sh +# 1. Annotations, character mugshots, scene segmentation. +# NIST names the same film three different ways, hence the overrides. +KG_DIR=superHero KG_FILE=superhero scripts/fetch_dvu.sh SuperHero ../dvu-hero + +# 2. Scene clips (movie.shots), then fuse them into one stream. +# Fusing matters — see "Run it as one film" below. +# SuperHero-1.webm … SuperHero-10.webm from +# /movie.shots/, then: +ffmpeg -f concat -safe 0 -i concat.txt -c copy SuperHero_full.webm + +# 3. Gallery, with the face-size floor that keeps references in distribution. +./build/build_gallery --root ../dvu-hero/root \ + --output ../dvu-hero/hero66.h5 --min-face-px 66 + +# 4. Run, on the GPU path (see "Check you are on the GPU"). +./build/scene_analyze --movie hero/SuperHero_full.webm \ + --gallery ../dvu-hero/hero66.h5 \ + --detector-engine trt_cache/scrfd.scrfd_500m_bnkps.640.fp16.engine \ + --arcface-engine trt_cache/arcface.LVFace-B_Glint360K.b4.fp16.engine \ + --fps 5 --min-face-px 32 --expand-gallery \ + --output pred.json +``` + +Nothing here is in git: the clips are ~130 MB and the annotations are +regenerable. Replay fixtures derived from the run ship through the artifact +registry instead: + +```sh +scripts/artifacts/push_artifacts.sh replay-fixtures +scripts/artifacts/pull_artifacts.sh replay-fixtures [version] +``` + +The gallery travels in the same archive as the dumps deliberately — a dump only +replays meaningfully against the gallery it was produced with, and pairing one +with a different gallery silently changes every identity decision in it. + +--- + +## Results + +Measured on the fused film, gallery expansion on. + +| Metric | Value | +|---|---| +| Precision | **1.00** | +| Recall | 0.65 | +| F1 | 0.79 | +| True positives | 13 | +| False positives | **0** | +| False negatives | 7 | + +Six of ten scenes scored exactly right, including the three-character scenes 4 +and 5. + +**Zero false positives is the result worth keeping.** Every out-of-gallery +character — Beast, Mighty Celestial, Ms. Johnson, Doctor, two Masked Persons — +was declined rather than forced onto a nearest match. That is the calibrated +probability (AR-024) doing its job, and it is the right failure direction for an +X-Ray overlay: a miss is a gap, an invention is a lie. + +**The misses have a shape.** Scenes 1, 2, 3 and 8 were missed, and 1–3 are the +three shortest scenes in the film (14 s, 38 s, 27 s). That is consistent with +per-track Bayesian accumulation (AR-025) needing enough sightings before belief +crosses threshold. Scene 8 is 65 s and does not fit that story — it is the one +to look at first when improving recall. + +Running the same scenes as isolated clips did *not* do better, so cross-scene +gallery expansion is not currently compensating for short scenes. + +### Run it as one film, not as clips + +Per-scene clips defeat per-film gallery expansion (AR-019), which grows a +temporary gallery from track continuity across the whole film and re-assesses +unknown tracks at the end. Ten isolated clips give it nothing to work with, and +pay model and gallery load ten times over. + +Fusing also makes presence windows cross real scene boundaries, which is how +SR-002's scene-scoped question is asked in production. Note the joins are +artificial cuts — consecutive scenes were never contiguous footage — so presence +bleeding across a boundary may be the join rather than a tracking fault. + +--- + +## Throughput + +| Path | Realtime factor | Sampled fps | 17-min film | +|---|---|---|---| +| `build/` (TensorRT) | **2.0×** | ~10 | ~8 min | +| `build-ort/` (ORT) | 0.54× | 2.7 | ~32 min | + +Throughput varies strongly with face density; a sparse stretch measured 8× +realtime, so quote the whole-film average, not a window. + +### Check you are on the GPU + +ORT's CUDA execution provider fails to load on this machine and **silently falls +back to CPU**: + +``` +Failed to load library libonnxruntime_providers_cuda.so: + undefined symbol: cudnnGetConvolutionBackwardDataAlgorithm_v7 +``` + +That symbol was removed in cuDNN 9; the packaged ORT is built against cuDNN 8. +ORT logs this once at startup and then runs happily on CPU, so a `build-ort` +timing is a CPU number wearing a GPU label — a 15× error with no symptom other +than a figure you have no baseline for. Grep the log for `Failed to load +library` before trusting any throughput measurement. + +The TensorRT path (`build/`) needs prebuilt engines from +`scripts/build_trt_engines.sh` and reports what it loaded: + +``` +[TrtScrfd] loaded: … [TrtArcFace] loaded: … max_batch=4 +[similarity] cuBLAS/CUDA engine: gallery resident on GPU +``` + +--- + +## Why not Road to Bali + +Bali was chosen because DVU ships character mugshots for it. It was withdrawn on +**face scale**, measured on its own reference crops: + +| | Bali | SuperHero | +|---|---|---| +| Median detected face | 27 px | **69 px** | +| Maximum detected face | 69 px | **241 px** | +| References ≥66 px | 2 of 69 | 14 of 27 | + +The DVU images are scene crops, not mugshots, so the crop dimensions say nothing +about face scale — the face has to be detected and measured. Bali's median +reference was being upscaled roughly 4× to reach ArcFace's 112×112, and the +worst 7×, which violates AR-011: every model gets the input it was trained for. +A model run off-distribution returns confident, plausible, wrong output. + +In a gallery that error is permanent. A bad frame costs one frame; a poisoned +reference corrupts every future match against that identity. + +No threshold rescued it. At 66 px only 2 of 69 references survived — the largest +face in the entire set is 69 px — so there was no cut that both kept references +in distribution and left enough of them to calibrate. SuperHero's gallery builds +at a 66 px floor and calibrates on its own (`a=15.2867 b=-4.98633`, 100 % train +accuracy) rather than borrowing constants. + +Any accuracy figure recorded against Bali predates this and should be treated as +measuring upscaling artifacts as much as the pipeline. diff --git a/mkdocs.yml b/mkdocs.yml index 2ec5120..0041ca7 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -35,6 +35,7 @@ extra_css: nav: - Home: index.md - How We Score Against X-Ray: methodology.md + - Benchmark — SuperHero: benchmark.md - Findings: - Best Model: best-model.md - Gallery Scope (Full vs. Limited): gallery-scope.md