scene-actor-extraction/docs/model-bakeoff.md

Full experiment log (opencv5)

This is the complete log behind the current opencv5 build: how the pipeline is tuned, what the shipped configuration is and where every number in it comes from, and how the learned scene-boundary detector took per-second actor-presence F1 from the low-60s to 74.9% across the nine-film Amazon X-Ray benchmark — under honest leave-one-out.

Read How we score against X-Ray first for what F1, precision, recall, and misID mean here. Every number below uses the per-second metric (scripts/optimizer/second_score.py): the film is sampled once per second, and at each second the set of names the pipeline reports present is compared against Amazon X-Ray's scene cast for that second. X-Ray's ground truth is scene-level; the pipeline's output is per-second. That mismatch shapes every result.

The benchmark

Nine films with public Amazon X-Ray scene data, all scored with the same LVFace-B Glint360K gallery:

Benny & Joon · Café Society · Downton Abbey: A New Era · Lord of War · Lovelace · The Many Saints of Newark · Scarface · Sound of Metal · Valerian.

Two of these — Café Society and Scarface — are low-contrast, uniformly-graded films that break naive cut detection. They are deliberately kept in the benchmark because they are where the interesting failures live.

Why replay makes this affordable

Decoding video and running face detection, alignment, and embedding is the expensive part of the pipeline. Everything downstream — tracking, identity matching, scene aggregation — is cheap. KPN++'s node/network structure keeps those two halves as separate components joined by typed channels, so the expensive half runs once per film and caches its output, and the cheap half can be re-run against that cache as often as needed with different Config values.

scene_analyze --dump-embeddings out.h5 runs the expensive half once and writes per-frame detections, embeddings, and (for the scene detector) per-frame RGB histograms to HDF5. scripts/optimizer/replay.py re-assembles the real C++ face_tracker, identity_matcher, and scene nodes into a Python-driven KPN network and replays a film's cache through them, varying every tuning knob freely. No GPU inference and no video decode happen during a replay, so a full differential-evolution search over all nine films is tractable in one session rather than requiring re-encodes.

Two concurrency limits are load-bearing and were paid for in wedged runs: replays run at DE_WORKERS=1 (concurrent DE candidates wedge the ROCm GPU), and each candidate's per-film replays run at REPLAY_WORKERS=8 with stderr discarded (the replay sink's per-second prints otherwise flood the captured pipe and hang the subprocess).

The tuning knobs

The opencv5 refactor replaced the old three-knob search with a ten-knob differential-evolution sweep. The knobs, and their shipped values:

knob	shipped	what it controls
prob_threshold	0.485	posterior P(match) above which a track is named
ownership_logodds	1.72	log-odds a track needs before it produces presence
track_extinction_sec	31.0	how long an idle track is held for re-detection
track_alpha	0.435	tracker cost mix (0 = embedding only, 1 = spatial only)
evidence_rho_max	0.204	evidence weighting ceiling
evidence_admit_below	0.784	admit new evidence below this similarity
match_prior	0.433	base-rate prior on a match
expand_band_lo	0.804	low edge of the pose-expansion similarity band
expand_band_hi	0.952	high edge of the pose-expansion band
presence_mode	flood	track-extent vs scene flood-fill

The DE run over the first nine knobs (flood off, track-extent presence) converged at 64.0% macro F1 over 345 evaluations. Those values are the shipped src/config.hpp defaults.

The track_extinction_sec knob is worth calling out: at 31 s it holds an idle track alive for re-detection long enough to bridge an actor turning away or leaving frame briefly, without bridging across a genuine scene change. Getting this knob and the tracker/registry to agree on one clock (the evidence watermark, not wall-clock) was a correctness fix, not a tuning choice — before it, votes were silently dropped at the reap horizon.

The step change: flood-fill on learned boundaries

The 64.0% above is track-extent presence: an actor is reported only while an actual track is alive. Flood-fill instead reports an actor for the whole shot once they are seen in it — but that is only correct if the shot boundaries are good.

With the old grayscale cut detector as the boundary source, flood-fill barely beat doing nothing (64.0%) and actively broke Scarface, where the detector fires once in 10,204 frames and flood then smears every actor across the whole film (precision collapses to 26%).

The learned scene-boundary detector — an XGBoost regressor over histogram-delta and audio features, with a per-film knee threshold — fixes this. Macro per-second presence F1, at the shipped presence config:

boundary source for flood-fill	presence F1
track-extent (flood off)	62.6%
flood + grayscale cuts	64.0%
flood + learned detector (LOO)	74.9%

The learned column is leave-one-out: each film is scored by a detector trained on the other eight, so no film's presence is ever measured with a detector that saw it. That is the honest generalisation number, +12.3 points over track-extent, and it improves every one of the nine films.

film	track-extent	flood+grayscale	flood+learned (LOO)
Benny & Joon	77.3	80.2	78.2
Café Society	59.1	62.2	69.8
Downton Abbey	41.0	51.8	78.6
Lord of War	74.8	77.1	77.8
Lovelace	70.3	74.0	78.2
The Many Saints of Newark	37.5	43.9	53.4
Scarface	62.6	40.9	74.9
Sound of Metal	75.0	78.1	86.8
Valerian	65.6	67.7	76.2

The two headline films — Scarface (grayscale flood breaks it, learned flood on a film it never trained on takes it to 74.9%) and Downton Abbey (+37 points) — are the strongest evidence the detector generalises. See the scene-boundary detector page for the full story.

We re-ran the ten-knob DE on top of the good boundaries to check whether the shipped config should change. It converged at 76.1% (+0.3 pp over the shipped config on learned boundaries) — inside the noise, not worth re-shipping. The boundaries, not the presence knobs, are where the win is.

What the frames look like

scripts/optimizer/dump_error_frames.py pulls representative seconds and draws each face box against X-Ray's scene cast: green = true positive, red = false positive (a name X-Ray does not credit to this scene — the real error), orange = an unknown detection. Cast X-Ray lists as present but for whom no face was detected — the structural false-negatives a face pipeline can never box — are listed as a blue panel.

Every named frame in this documentation is regenerated against the current opencv5 pipeline by scripts/scene_detector/rematch_frames.py, which auto-matches each example by film, actor, and class (TP/FP) so the images never drift from the shipped behaviour. Where the current pipeline no longer makes a July-era error — the Zooey Deschanel misID in Many Saints is the clearest case — the frame is dropped rather than staged, because the improvement is real.

The structural recall ceiling

Precision against X-Ray is near-perfect on identified faces; recall is capped by two things the pipeline cannot fix:

1. X-Ray credits people whose faces never appear on screen in a scene — voice, back-of-head, or simply off-camera cast. No face pipeline can box a face that is not there. These are the blue-panel names.
2. Gallery coverage. A large fraction of X-Ray cast has no reference image in the gallery, so those actors can never be matched regardless of detection. This is the dominant remaining recall limiter and is addressable by fetching more reference photos, not by tuning.

Both are documented in how we score against X-Ray.

In the pipeline

The learned detector runs live inside scene_analyze as a post-EOF step (the per-film knee needs every peak, so it can only run once the whole film is seen). XGBoost inference is built into the binary via CMake (SAE_SCENE_XGB); the audio log-PSD uses FFTW on the existing FFmpeg decode. The shipped model is trained on the C++-extracted features so training and inference share one implementation. Verified end to end through scene_analyze on a movie file and through the Jellyfin work-queue worker.

scene_analyze --movie <file> --gallery <gallery.h5> \
    --scene-xgb-model models/scene_boundary_xgb.json

Reproducing the benchmarks

Gallery .h5 files, embedding dumps, the X-Ray corpus, and DE trajectories are not committed. They are pushed to the Gitea package registry and pulled on demand:

scripts/artifacts/pull_artifacts.sh galleries
scripts/artifacts/pull_artifacts.sh experiment-data

# per-second audio features, C++ feature matrices, train + downstream A/B
.venv-rocm/bin/python scripts/scene_detector/extract_audio_features.py \
    --manifest experiments/manifests/films_LVFace_opencv5.json
.venv-rocm/bin/python scripts/scene_detector/train_xgb_cpp.py --train-all
scripts/scene_detector/downstream_presence.py