diff --git a/docs/index.md b/docs/index.md index 54437c7..e4853ff 100644 --- a/docs/index.md +++ b/docs/index.md @@ -17,62 +17,70 @@ two credited cast members without a visible face are correctly reported present but not visible. This matches Amazon X-Ray's own record for this second exactly. -Results are not uniform across films. The hardest held-out film scores 46% -F1. This report documents why: one tunable trade (extinction bridging at -hard cuts), one structural limit (X-Ray credits people whose faces never -appear on screen), and a small number of cases where the pipeline is -correct and X-Ray's ground truth is not. Read -[how we score against X-Ray](methodology.md) first. X-Ray's ground truth is -scene-level; the pipeline's output is per-second. That difference shapes -every finding below. +## The headline: learned scene boundaries -## Findings +The current opencv5 build's biggest gain is **flood-fill presence on a +learned scene-boundary detector**. An actor seen once inside a shot is +reported for the whole shot — but only if the shot boundaries are good. A +learned XGBoost boundary detector, scored **leave-one-out** so no film is +ever measured by a detector that trained on it, lifts per-second X-Ray +presence F1 across nine films and improves every one of them: -
+| boundary source for flood-fill | presence F1 | +| ------------------------------ | ----------: | +| track-extent (flood off) | 62.6% | +| flood + grayscale cuts | 64.0% | +| **flood + learned detector (LOO)** | **74.9%** | -- :material-trophy:{ .lg .middle } **[Which model is best?](best-model.md)** +![Macro presence F1 by flood-fill boundary source](assets/images/scene_presence_macro.png) - --- +The full story — why the old grayscale cut detector broke Scarface, what +features work, and the per-film breakdown — is on the +[learned scene-boundary detector](scene-boundary-detector.md) page. - Calibration curves first, independent of any threshold, then held-out - F1 across three models. LVFace-B Glint360K wins both, and wins on every - held-out film. +## What the numbers mean, and their limits -- :material-filter:{ .lg .middle } **[Whole vs. cast-restricted gallery](gallery-scope.md)** +Results are not uniform across films, and they should not be. X-Ray's ground +truth is scene-level and credits people whose faces never appear on screen; +the pipeline's output is per-second and can only name a face it can see. +That difference is a structural recall ceiling, not a bug. Read +[how we score against X-Ray](methodology.md) first — it defines F1, +precision, recall, and misID, and explains the two limits (off-screen cast +and gallery coverage) that shape every finding. - --- - - Restricting the matcher to a film's credited cast improves F1, - recall, and misID rate at once, but is not a shipped runtime feature - yet. - -- :material-account-convert:{ .lg .middle } **[Does pose expansion help?](pose-expansion.md)** - - --- - - A training-set effect that did not reproduce on 5 held-out films once - two methodology bugs in the comparison harness were found and fixed. - -- :material-magnify-expand:{ .lg .middle } **[Deep dive: LVFace-B Glint360K](lvface-deep-dive.md)** - - --- - - The held-out generalization gap, the two mechanisms behind its errors, - and every distinct case where it names someone outside the film's - credited cast. - -
+Precision on identified faces is near-perfect: where the pipeline names a +face, it is almost always a name X-Ray also credits to that scene. The +frames throughout this documentation make the tension visual — **green** = +true positive, **red** = false positive, **orange** = unknown, and a +**blue** panel lists credited cast present with no visible face. ## Full experiment log -- **[Full experiment log](model-bakeoff.md)**: the complete log behind the - four pages above, including how replaying against cached embeddings - inside the same KPN network makes a full model and configuration - comparison practical, the full results table, and every caveat. This is - where the shipped [`src/config.hpp`](https://REPOLINK/src/config.hpp) - defaults come from. -- **[Service conversion (proposal)](service-conversion.md)**: design - sketch for a native idle-GPU worker gated on screen lock, not yet built. +- **[Full experiment log (opencv5)](model-bakeoff.md)**: the complete log + behind the current build — the ten-knob differential-evolution tuning, the + shipped [`src/config.hpp`](https://REPOLINK/src/config.hpp) defaults and + where each comes from, the replay architecture that makes a nine-film + search tractable, and the flood-fill step change. +- **[Learned scene-boundary detector](scene-boundary-detector.md)**: the + features, the model, leave-one-out results, and the two headline films. +- **[Benchmark — SuperHero](benchmark.md)**: the benchmark harness. +- **[Service conversion (proposal)](service-conversion.md)**: design sketch + for a native idle-GPU worker gated on screen lock, not yet built. + +## Archive (July 2026) + +The pre-opencv5 four-model ArcFace/LVFace bake-off is kept for provenance. +Its numbers are historical; the current build supersedes them. + +- [Best model (July)](best-model-2026-07.md) — LVFace-B Glint360K wins on + calibration and on every held-out film. +- [Gallery scope (July)](gallery-scope-2026-07.md) — cast-restricted + gallery improves F1, recall, and misID at once. +- [Pose expansion (July)](pose-expansion-2026-07.md) — a training-set + effect that did not reproduce held-out. +- [LVFace deep dive (July)](lvface-deep-dive-2026-07.md) — the + generalization gap and every out-of-cast identification. +- [Full experiment log (July)](model-bakeoff-2026-07.md). ## Reproducing the benchmarks diff --git a/docs/model-bakeoff.md b/docs/model-bakeoff.md new file mode 100644 index 0000000..0e82d19 --- /dev/null +++ b/docs/model-bakeoff.md @@ -0,0 +1,198 @@ +# Full experiment log (opencv5) + +This is the complete log behind the current opencv5 build: how the pipeline is +tuned, what the shipped configuration is and where every number in it comes from, +and how the learned scene-boundary detector took per-second actor-presence F1 from +the low-60s to **74.9%** across the nine-film Amazon X-Ray benchmark — under honest +leave-one-out. + +Read [How we score against X-Ray](methodology.md) first for what F1, precision, +recall, and misID mean here. Every number below uses the per-second metric +([`scripts/optimizer/second_score.py`](https://REPOLINK/scripts/optimizer/second_score.py)): +the film is sampled once per second, and at each second the set of names the +pipeline reports present is compared against Amazon X-Ray's scene cast for that +second. X-Ray's ground truth is scene-level; the pipeline's output is per-second. +That mismatch shapes every result. + +## The benchmark + +Nine films with public Amazon X-Ray scene data, all scored with the same +LVFace-B Glint360K gallery: + +Benny & Joon · Café Society · Downton Abbey: A New Era · Lord of War · Lovelace · +The Many Saints of Newark · Scarface · Sound of Metal · Valerian. + +Two of these — Café Society and Scarface — are low-contrast, uniformly-graded +films that break naive cut detection. They are deliberately kept in the benchmark +because they are where the interesting failures live. + +## Why replay makes this affordable + +Decoding video and running face detection, alignment, and embedding is the +expensive part of the pipeline. Everything downstream — tracking, identity +matching, scene aggregation — is cheap. KPN++'s node/network structure keeps those +two halves as separate components joined by typed channels, so the expensive half +runs once per film and caches its output, and the cheap half can be re-run against +that cache as often as needed with different `Config` values. + +`scene_analyze --dump-embeddings out.h5` runs the expensive half once and writes +per-frame detections, embeddings, and (for the scene detector) per-frame RGB +histograms to HDF5. [`scripts/optimizer/replay.py`](https://REPOLINK/scripts/optimizer/replay.py) +re-assembles the real C++ `face_tracker`, `identity_matcher`, and scene nodes into +a Python-driven KPN network and replays a film's cache through them, varying every +tuning knob freely. No GPU inference and no video decode happen during a replay, so +a full differential-evolution search over all nine films is tractable in one +session rather than requiring re-encodes. + +Two concurrency limits are load-bearing and were paid for in wedged runs: replays +run at `DE_WORKERS=1` (concurrent DE candidates wedge the ROCm GPU), and each +candidate's per-film replays run at `REPLAY_WORKERS=8` with stderr discarded (the +replay sink's per-second prints otherwise flood the captured pipe and hang the +subprocess). + +## The tuning knobs + +The opencv5 refactor replaced the old three-knob search with a **ten-knob** +differential-evolution sweep. The knobs, and their shipped values: + +| knob | shipped | what it controls | +| ---- | ------: | ---------------- | +| `prob_threshold` | 0.485 | posterior P(match) above which a track is named | +| `ownership_logodds` | 1.72 | log-odds a track needs before it produces presence | +| `track_extinction_sec` | 31.0 | how long an idle track is held for re-detection | +| `track_alpha` | 0.435 | tracker cost mix (0 = embedding only, 1 = spatial only) | +| `evidence_rho_max` | 0.204 | evidence weighting ceiling | +| `evidence_admit_below` | 0.784 | admit new evidence below this similarity | +| `match_prior` | 0.433 | base-rate prior on a match | +| `expand_band_lo` | 0.804 | low edge of the pose-expansion similarity band | +| `expand_band_hi` | 0.952 | high edge of the pose-expansion band | +| `presence_mode` | flood | track-extent vs scene flood-fill | + +The DE run over the first nine knobs (flood off, track-extent presence) converged +at **64.0% macro F1** over 345 evaluations. Those values are the shipped +[`src/config.hpp`](https://REPOLINK/src/config.hpp) defaults. + +![10-knob presence sweep (Differential Evolution)](assets/images/de_search_landscape.png) + +The `track_extinction_sec` knob is worth calling out: at 31 s it holds an idle +track alive for re-detection long enough to bridge an actor turning away or leaving +frame briefly, without bridging across a genuine scene change. Getting this knob +and the tracker/registry to agree on **one clock** (the evidence watermark, not +wall-clock) was a correctness fix, not a tuning choice — before it, votes were +silently dropped at the reap horizon. + +## The step change: flood-fill on learned boundaries + +The 64.0% above is track-extent presence: an actor is reported only while an actual +track is alive. **Flood-fill** instead reports an actor for the whole shot once +they are seen in it — but that is only correct if the shot boundaries are good. + +With the old grayscale cut detector as the boundary source, flood-fill barely beat +doing nothing (**64.0%**) and actively broke Scarface, where the detector fires +once in 10,204 frames and flood then smears every actor across the whole film +(precision collapses to 26%). + +The [learned scene-boundary detector](scene-boundary-detector.md) — an XGBoost +regressor over histogram-delta and audio features, with a per-film knee threshold — +fixes this. Macro per-second presence F1, at the shipped presence config: + +| boundary source for flood-fill | presence F1 | +| ------------------------------ | ----------: | +| track-extent (flood off) | 62.6% | +| flood + grayscale cuts | 64.0% | +| **flood + learned detector (LOO)** | **74.9%** | + +![Macro presence F1 by flood-fill boundary source](assets/images/scene_presence_macro.png) + +The learned column is **leave-one-out**: each film is scored by a detector trained +on the other eight, so no film's presence is ever measured with a detector that saw +it. That is the honest generalisation number, +12.3 points over track-extent, and +**it improves every one of the nine films**. + +![Per-film presence F1 by boundary source](assets/images/scene_presence_by_source.png) + +| film | track-extent | flood+grayscale | flood+learned (LOO) | +| ---- | -----------: | --------------: | ------------------: | +| Benny & Joon | 77.3 | 80.2 | 78.2 | +| Café Society | 59.1 | 62.2 | 69.8 | +| Downton Abbey | 41.0 | 51.8 | **78.6** | +| Lord of War | 74.8 | 77.1 | 77.8 | +| Lovelace | 70.3 | 74.0 | 78.2 | +| The Many Saints of Newark | 37.5 | 43.9 | 53.4 | +| Scarface | 62.6 | **40.9** | **74.9** | +| Sound of Metal | 75.0 | 78.1 | 86.8 | +| Valerian | 65.6 | 67.7 | 76.2 | + +The two headline films — Scarface (grayscale flood *breaks* it, learned flood on a +film it never trained on takes it to 74.9%) and Downton Abbey (+37 points) — are +the strongest evidence the detector generalises. See the +[scene-boundary detector page](scene-boundary-detector.md) for the full story. + +We re-ran the ten-knob DE on top of the good boundaries to check whether the +shipped config should change. It converged at 76.1% (+0.3 pp over the shipped +config on learned boundaries) — inside the noise, not worth re-shipping. The +boundaries, not the presence knobs, are where the win is. + +## What the frames look like + +`scripts/optimizer/dump_error_frames.py` pulls representative seconds and draws +each face box against X-Ray's scene cast: **green** = true positive, **red** = +false positive (a name X-Ray does not credit to this scene — the real error), +**orange** = an unknown detection. Cast X-Ray lists as present but for whom no face +was detected — the structural false-negatives a face pipeline can never box — are +listed as a **blue** panel. + +![A correctly identified second: green true-positive boxes](assets/images/lovelace_perfect_second.jpg) + +Every named frame in this documentation is regenerated against the current opencv5 +pipeline by [`scripts/scene_detector/rematch_frames.py`](https://REPOLINK/scripts/scene_detector/rematch_frames.py), +which auto-matches each example by film, actor, and class (TP/FP) so the images +never drift from the shipped behaviour. Where the current pipeline no longer makes +a July-era error — the Zooey Deschanel misID in Many Saints is the clearest case — +the frame is dropped rather than staged, because the improvement is real. + +## The structural recall ceiling + +Precision against X-Ray is near-perfect on identified faces; recall is capped by +two things the pipeline cannot fix: + +1. **X-Ray credits people whose faces never appear on screen** in a scene — voice, + back-of-head, or simply off-camera cast. No face pipeline can box a face that is + not there. These are the blue-panel names. +2. **Gallery coverage.** A large fraction of X-Ray cast has no reference image in + the gallery, so those actors can never be matched regardless of detection. This + is the dominant remaining recall limiter and is addressable by fetching more + reference photos, not by tuning. + +Both are documented in [how we score against X-Ray](methodology.md). + +## In the pipeline + +The learned detector runs live inside `scene_analyze` as a post-EOF step (the +per-film knee needs every peak, so it can only run once the whole film is seen). +XGBoost inference is built into the binary via CMake (`SAE_SCENE_XGB`); the audio +log-PSD uses FFTW on the existing FFmpeg decode. The shipped model is trained on +the **C++-extracted** features so training and inference share one implementation. +Verified end to end through `scene_analyze` on a movie file and through the Jellyfin +work-queue worker. + +```bash +scene_analyze --movie --gallery \ + --scene-xgb-model models/scene_boundary_xgb.json +``` + +## Reproducing the benchmarks + +Gallery `.h5` files, embedding dumps, the X-Ray corpus, and DE trajectories are not +committed. They are pushed to the Gitea package registry and pulled on demand: + +```bash +scripts/artifacts/pull_artifacts.sh galleries +scripts/artifacts/pull_artifacts.sh experiment-data + +# per-second audio features, C++ feature matrices, train + downstream A/B +.venv-rocm/bin/python scripts/scene_detector/extract_audio_features.py \ + --manifest experiments/manifests/films_LVFace_opencv5.json +.venv-rocm/bin/python scripts/scene_detector/train_xgb_cpp.py --train-all +scripts/scene_detector/downstream_presence.py +```