dtourolle
7556c836da
Add model-bakeoff.md for the opencv5 build: the ten-knob DE tuning and where each shipped config default comes from, the replay architecture, and the flood-fill-on-learned-boundaries step change (62.6% -> 74.9% presence F1, LOO). Rewrite index.md to lead with the learned scene-boundary result and point at the current pages, with the July four-model bake-off moved to an Archive section. Both pages build with no broken links.
199 lines
10 KiB
Markdown
199 lines
10 KiB
Markdown
# Full experiment log (opencv5)
|
|
|
|
This is the complete log behind the current opencv5 build: how the pipeline is
|
|
tuned, what the shipped configuration is and where every number in it comes from,
|
|
and how the learned scene-boundary detector took per-second actor-presence F1 from
|
|
the low-60s to **74.9%** across the nine-film Amazon X-Ray benchmark — under honest
|
|
leave-one-out.
|
|
|
|
Read [How we score against X-Ray](methodology.md) first for what F1, precision,
|
|
recall, and misID mean here. Every number below uses the per-second metric
|
|
([`scripts/optimizer/second_score.py`](https://REPOLINK/scripts/optimizer/second_score.py)):
|
|
the film is sampled once per second, and at each second the set of names the
|
|
pipeline reports present is compared against Amazon X-Ray's scene cast for that
|
|
second. X-Ray's ground truth is scene-level; the pipeline's output is per-second.
|
|
That mismatch shapes every result.
|
|
|
|
## The benchmark
|
|
|
|
Nine films with public Amazon X-Ray scene data, all scored with the same
|
|
LVFace-B Glint360K gallery:
|
|
|
|
Benny & Joon · Café Society · Downton Abbey: A New Era · Lord of War · Lovelace ·
|
|
The Many Saints of Newark · Scarface · Sound of Metal · Valerian.
|
|
|
|
Two of these — Café Society and Scarface — are low-contrast, uniformly-graded
|
|
films that break naive cut detection. They are deliberately kept in the benchmark
|
|
because they are where the interesting failures live.
|
|
|
|
## Why replay makes this affordable
|
|
|
|
Decoding video and running face detection, alignment, and embedding is the
|
|
expensive part of the pipeline. Everything downstream — tracking, identity
|
|
matching, scene aggregation — is cheap. KPN++'s node/network structure keeps those
|
|
two halves as separate components joined by typed channels, so the expensive half
|
|
runs once per film and caches its output, and the cheap half can be re-run against
|
|
that cache as often as needed with different `Config` values.
|
|
|
|
`scene_analyze --dump-embeddings out.h5` runs the expensive half once and writes
|
|
per-frame detections, embeddings, and (for the scene detector) per-frame RGB
|
|
histograms to HDF5. [`scripts/optimizer/replay.py`](https://REPOLINK/scripts/optimizer/replay.py)
|
|
re-assembles the real C++ `face_tracker`, `identity_matcher`, and scene nodes into
|
|
a Python-driven KPN network and replays a film's cache through them, varying every
|
|
tuning knob freely. No GPU inference and no video decode happen during a replay, so
|
|
a full differential-evolution search over all nine films is tractable in one
|
|
session rather than requiring re-encodes.
|
|
|
|
Two concurrency limits are load-bearing and were paid for in wedged runs: replays
|
|
run at `DE_WORKERS=1` (concurrent DE candidates wedge the ROCm GPU), and each
|
|
candidate's per-film replays run at `REPLAY_WORKERS=8` with stderr discarded (the
|
|
replay sink's per-second prints otherwise flood the captured pipe and hang the
|
|
subprocess).
|
|
|
|
## The tuning knobs
|
|
|
|
The opencv5 refactor replaced the old three-knob search with a **ten-knob**
|
|
differential-evolution sweep. The knobs, and their shipped values:
|
|
|
|
| knob | shipped | what it controls |
|
|
| ---- | ------: | ---------------- |
|
|
| `prob_threshold` | 0.485 | posterior P(match) above which a track is named |
|
|
| `ownership_logodds` | 1.72 | log-odds a track needs before it produces presence |
|
|
| `track_extinction_sec` | 31.0 | how long an idle track is held for re-detection |
|
|
| `track_alpha` | 0.435 | tracker cost mix (0 = embedding only, 1 = spatial only) |
|
|
| `evidence_rho_max` | 0.204 | evidence weighting ceiling |
|
|
| `evidence_admit_below` | 0.784 | admit new evidence below this similarity |
|
|
| `match_prior` | 0.433 | base-rate prior on a match |
|
|
| `expand_band_lo` | 0.804 | low edge of the pose-expansion similarity band |
|
|
| `expand_band_hi` | 0.952 | high edge of the pose-expansion band |
|
|
| `presence_mode` | flood | track-extent vs scene flood-fill |
|
|
|
|
The DE run over the first nine knobs (flood off, track-extent presence) converged
|
|
at **64.0% macro F1** over 345 evaluations. Those values are the shipped
|
|
[`src/config.hpp`](https://REPOLINK/src/config.hpp) defaults.
|
|
|
|
|
|
|
|
The `track_extinction_sec` knob is worth calling out: at 31 s it holds an idle
|
|
track alive for re-detection long enough to bridge an actor turning away or leaving
|
|
frame briefly, without bridging across a genuine scene change. Getting this knob
|
|
and the tracker/registry to agree on **one clock** (the evidence watermark, not
|
|
wall-clock) was a correctness fix, not a tuning choice — before it, votes were
|
|
silently dropped at the reap horizon.
|
|
|
|
## The step change: flood-fill on learned boundaries
|
|
|
|
The 64.0% above is track-extent presence: an actor is reported only while an actual
|
|
track is alive. **Flood-fill** instead reports an actor for the whole shot once
|
|
they are seen in it — but that is only correct if the shot boundaries are good.
|
|
|
|
With the old grayscale cut detector as the boundary source, flood-fill barely beat
|
|
doing nothing (**64.0%**) and actively broke Scarface, where the detector fires
|
|
once in 10,204 frames and flood then smears every actor across the whole film
|
|
(precision collapses to 26%).
|
|
|
|
The [learned scene-boundary detector](scene-boundary-detector.md) — an XGBoost
|
|
regressor over histogram-delta and audio features, with a per-film knee threshold —
|
|
fixes this. Macro per-second presence F1, at the shipped presence config:
|
|
|
|
| boundary source for flood-fill | presence F1 |
|
|
| ------------------------------ | ----------: |
|
|
| track-extent (flood off) | 62.6% |
|
|
| flood + grayscale cuts | 64.0% |
|
|
| **flood + learned detector (LOO)** | **74.9%** |
|
|
|
|
|
|
|
|
The learned column is **leave-one-out**: each film is scored by a detector trained
|
|
on the other eight, so no film's presence is ever measured with a detector that saw
|
|
it. That is the honest generalisation number, +12.3 points over track-extent, and
|
|
**it improves every one of the nine films**.
|
|
|
|
|
|
|
|
| film | track-extent | flood+grayscale | flood+learned (LOO) |
|
|
| ---- | -----------: | --------------: | ------------------: |
|
|
| Benny & Joon | 77.3 | 80.2 | 78.2 |
|
|
| Café Society | 59.1 | 62.2 | 69.8 |
|
|
| Downton Abbey | 41.0 | 51.8 | **78.6** |
|
|
| Lord of War | 74.8 | 77.1 | 77.8 |
|
|
| Lovelace | 70.3 | 74.0 | 78.2 |
|
|
| The Many Saints of Newark | 37.5 | 43.9 | 53.4 |
|
|
| Scarface | 62.6 | **40.9** | **74.9** |
|
|
| Sound of Metal | 75.0 | 78.1 | 86.8 |
|
|
| Valerian | 65.6 | 67.7 | 76.2 |
|
|
|
|
The two headline films — Scarface (grayscale flood *breaks* it, learned flood on a
|
|
film it never trained on takes it to 74.9%) and Downton Abbey (+37 points) — are
|
|
the strongest evidence the detector generalises. See the
|
|
[scene-boundary detector page](scene-boundary-detector.md) for the full story.
|
|
|
|
We re-ran the ten-knob DE on top of the good boundaries to check whether the
|
|
shipped config should change. It converged at 76.1% (+0.3 pp over the shipped
|
|
config on learned boundaries) — inside the noise, not worth re-shipping. The
|
|
boundaries, not the presence knobs, are where the win is.
|
|
|
|
## What the frames look like
|
|
|
|
`scripts/optimizer/dump_error_frames.py` pulls representative seconds and draws
|
|
each face box against X-Ray's scene cast: **green** = true positive, **red** =
|
|
false positive (a name X-Ray does not credit to this scene — the real error),
|
|
**orange** = an unknown detection. Cast X-Ray lists as present but for whom no face
|
|
was detected — the structural false-negatives a face pipeline can never box — are
|
|
listed as a **blue** panel.
|
|
|
|
|
|
|
|
Every named frame in this documentation is regenerated against the current opencv5
|
|
pipeline by [`scripts/scene_detector/rematch_frames.py`](https://REPOLINK/scripts/scene_detector/rematch_frames.py),
|
|
which auto-matches each example by film, actor, and class (TP/FP) so the images
|
|
never drift from the shipped behaviour. Where the current pipeline no longer makes
|
|
a July-era error — the Zooey Deschanel misID in Many Saints is the clearest case —
|
|
the frame is dropped rather than staged, because the improvement is real.
|
|
|
|
## The structural recall ceiling
|
|
|
|
Precision against X-Ray is near-perfect on identified faces; recall is capped by
|
|
two things the pipeline cannot fix:
|
|
|
|
1. **X-Ray credits people whose faces never appear on screen** in a scene — voice,
|
|
back-of-head, or simply off-camera cast. No face pipeline can box a face that is
|
|
not there. These are the blue-panel names.
|
|
2. **Gallery coverage.** A large fraction of X-Ray cast has no reference image in
|
|
the gallery, so those actors can never be matched regardless of detection. This
|
|
is the dominant remaining recall limiter and is addressable by fetching more
|
|
reference photos, not by tuning.
|
|
|
|
Both are documented in [how we score against X-Ray](methodology.md).
|
|
|
|
## In the pipeline
|
|
|
|
The learned detector runs live inside `scene_analyze` as a post-EOF step (the
|
|
per-film knee needs every peak, so it can only run once the whole film is seen).
|
|
XGBoost inference is built into the binary via CMake (`SAE_SCENE_XGB`); the audio
|
|
log-PSD uses FFTW on the existing FFmpeg decode. The shipped model is trained on
|
|
the **C++-extracted** features so training and inference share one implementation.
|
|
Verified end to end through `scene_analyze` on a movie file and through the Jellyfin
|
|
work-queue worker.
|
|
|
|
```bash
|
|
scene_analyze --movie <file> --gallery <gallery.h5> \
|
|
--scene-xgb-model models/scene_boundary_xgb.json
|
|
```
|
|
|
|
## Reproducing the benchmarks
|
|
|
|
Gallery `.h5` files, embedding dumps, the X-Ray corpus, and DE trajectories are not
|
|
committed. They are pushed to the Gitea package registry and pulled on demand:
|
|
|
|
```bash
|
|
scripts/artifacts/pull_artifacts.sh galleries
|
|
scripts/artifacts/pull_artifacts.sh experiment-data
|
|
|
|
# per-second audio features, C++ feature matrices, train + downstream A/B
|
|
.venv-rocm/bin/python scripts/scene_detector/extract_audio_features.py \
|
|
--manifest experiments/manifests/films_LVFace_opencv5.json
|
|
.venv-rocm/bin/python scripts/scene_detector/train_xgb_cpp.py --train-all
|
|
scripts/scene_detector/downstream_presence.py
|
|
```
|