scene-actor-extraction/docs/model-bakeoff-2026-07.md

Archived (July 2026). This report covers the pre-opencv5 framework and the 4-model ArcFace/LVFace bake-off. It is superseded by the current experiment log for the opencv5 build. Kept for provenance; the numbers here are historical.

Full experiment log

This page reports how the pipeline performs across three questions: which embedding model is best, whether restricting the gallery to a film's credited cast helps, and whether promoting confidently identified poses into a per-film gallery annex helps. It also documents the replay architecture that made testing all three questions in one pass practical, and every caveat needed to trust the numbers.

Read How we score against X-Ray first for what F1, precision, recall, and misID mean in this report. All numbers below use the per-second metric (scripts/optimizer/second_score.py).

r50 (ArcFace w600k-R50) is excluded from the detailed comparison below. Its gallery was built with roughly 30% fewer reference images per actor than the other three models on the identical source photos (10808 vs 15055 total embeddings across the same 2418 actors), which confounds any direct comparison of its scores against the others. It remains in the calibration curve comparison, which does not depend on the training benchmark.

Why replay makes this affordable

Decoding video and running face detection, alignment, and embedding is the expensive part of this pipeline. Everything downstream of that (tracking, identity matching, scene aggregation) is cheap. KPN++'s node/network structure means those two stages are separate components connected by typed channels, so the expensive stage can run once per film, cache its output, and the cheap stage can be re-run against that cache as many times as needed with different Config values.

scene_analyze --dump-embeddings out.h5 runs the expensive half once per film and writes per-frame face detections and embeddings to HDF5 (scripts/optimizer/SCHEMA.md). scripts/optimizer/replay.py then re-assembles the real C++ face_tracker, identity_matcher, and scene_tracker nodes into a Python-driven KPN network and replays a film's cached embeddings through them, varying prob_threshold, anneal_sec, extinction_sec, and expand_gallery freely. No GPU inference and no video decode happen during a replay; each one completes in seconds. This is what makes a 512-evaluation differential-evolution search per model, per gallery mode, per expansion setting, tractable, and what made the full held-out validation across three models in this report possible in one session rather than requiring three full re-encodes of the benchmark set.

optimize.py runs differential_evolution over this replay function as its objective, with DE-level parallelism (multiple candidate configs evaluated concurrently, each spawning its own replay subprocesses) on top of it. The practical ceiling on this machine's GPU was 8 concurrent replay processes; 9 silently degraded every score to 0.0% (well-formed output, wrong numbers, not a crash), so optimize.py was run at REPLAY_WORKERS=4 DE_WORKERS=2.

Search space

popsize=10, maxiter=15 per combo (3 parameters, up to 512 evaluations, usually stopping earlier on DE's convergence tolerance). anneal_sec/extinction_sec bounds were widened from 1-30/1-15 to 1-60/1-60 partway through the sweep. r50's 4 combos finished before the widening and used the old, narrower bounds; this is one more reason r50 is excluded from direct comparison here.

Training films and held-out films

9 films have dumped embeddings across all 4 models. 4 were used for optimization:

• Café Society (62-cast)
• Lord of War (64-cast)
• Scarface (67-cast)
• Sound of Metal (14-cast)

5 were held out, never seen by any optimizer run:

• Benny & Joon
• Downton Abbey: A New Era
• Lovelace
• The Many Saints of Newark
• Valerian and the City of a Thousand Planets

Gallery coverage per film

The gallery has reference embeddings for 2418 actors, but coverage of any given film's credited cast varies widely. This was previously reported as one flat number (67% of X-Ray cast lacking a reference embedding, averaged across the whole benchmark); the per-film breakdown is:

film	cast credited	in gallery	coverage
Lord of War	64	13	20.3%
Scarface	67	15	22.4%
The Many Saints of Newark	48	13	27.1%
Café Society	62	17	27.4%
Lovelace	42	15	35.7%
Valerian and the City of a Thousand Planets	36	13	36.1%
Benny & Joon	23	12	52.2%
Downton Abbey: A New Era	36	22	61.1%
Sound of Metal	14	11	78.6%

Two training films (Lord of War, Scarface) have the worst coverage in the set, 20-22%. Their training-set F1 numbers below are partly capped by missing references, not purely by model quality. Downton Abbey has 61% coverage, the second-best in the benchmark, yet the worst held-out recall of any film (39.4%, LVFace). Its recall problem is not primarily a coverage problem; it is the extinction-bridging failure documented in the LVFace deep dive. Reproduce with scripts/docs/gallery_coverage_per_film.py.

Training results, 3 models × 2 gallery modes × 2 expansion settings

Ranked by F1. misid = FPI_misid, the count of true wrong-actor identifications (naming someone not in the film's cast at all), distinct from FPI, which also includes in-cast timing slips.

Each combo's row is its best full-coverage evaluation: the highest-F1 DE evaluation in which all 4 training films replayed without a timeout (see Dropped-film scoring below for why this qualifier is load-bearing and not the same as argmax F1 over the raw sweep).

combo	F1	P	R	TPI	FPI	misid	FN
LVFace-B_Glint360K_restricted_exp	78.3%	91.0%	68.9%	42830	3782	60	19492
LVFace-B_Glint360K_restricted_noexp	76.7%	91.5%	66.2%	41149	3400	59	21173
arcface_w600k_mbf_restricted_exp	76.2%	90.0%	66.2%	64328	7480	0	33234
arcface_r18_restricted_exp	75.5%	87.6%	66.5%	41399	5666	60	20923
LVFace-B_Glint360K_full_exp	75.3%	89.7%	65.4%	47757	3407	232	26966
arcface_w600k_mbf_restricted_noexp	75.0%	91.1%	63.9%	39752	3465	60	22570
arcface_r18_restricted_noexp	73.5%	91.3%	61.7%	38299	3220	60	24023
LVFace-B_Glint360K_full_noexp	72.3%	88.3%	61.8%	40363	3503	244	25850
arcface_w600k_mbf_full_exp	72.0%	87.7%	61.4%	39875	3729	240	26338
arcface_w600k_mbf_full_noexp	71.0%	93.2%	57.9%	41699	2472	56	33024
arcface_r18_full_exp	69.1%	87.6%	57.7%	37342	3119	242	28871
arcface_r18_full_noexp	66.6%	91.3%	53.1%	34314	2362	107	31899

The two clearest patterns: every model's best-scoring combo uses the restricted gallery, and LVFace leads within both gallery modes. full_exp (the shipped combination) is the best-scoring option that uses only features the running application currently supports; restriction is not wired into the application yet (see Whole vs. cast-restricted gallery).

A scoring bug worth recording: dropped-film evaluations

The numbers above are corrected ones. The raw rep4_best_*.json files, and an earlier version of this table, reported a different arcface_w600k_mbf_full_noexp row: 74.2% F1 at TPI 12645, a third the TPI of every sibling combo. That was not a better config; it was an artifact of how the optimizer aggregates.

optimize.py builds each candidate's score from only the films whose replay subprocess returned (per_film = [m for m in ex.map(_one, films) if m is not None]), then averages F1/precision/recall and sums TPI/FPI/misID over just those survivors. When a film's replay times out (the sweep ran near the 8-process concurrency ceiling, so this happened intermittently), that film silently drops from both. A candidate whose hardest film timed out is therefore scored on an easier subset, and differential evolution, maximizing that score, will happily converge onto exactly such a candidate. For mbf_full_noexp the reported winner was one of 7 evaluations (out of 512) whose TPI had collapsed to a partial-film subset; its median-coverage evaluations sit around 51686 TPI.

The fix here was to re-derive each combo's best row from its DE trajectory (experiments/trajectories/rep4_*.jsonl), keeping only evaluations within 30% of that combo's median TPI (full 4-film coverage) before taking the best F1. This needs no re-running, the honest best configuration was already in the sweep, just not the one argmax F1 selected. Three combos moved: mbf_full_noexp 74.2% → 71.0%, LVFace_full_noexp 72.4% → 72.3% (and its misID, 0 → 244, was itself a dropped-film artifact), mbf_restricted_exp 76.5% → 76.2%. The shipped LVFace full_exp winner was unaffected, its reported evaluation already had full coverage (TPI 47757 ≈ median). experiment_charts.py applies the same clean_best filter, so every figure on this page matches the corrected table. The underlying optimize.py aggregation is also being fixed so a dropped-film evaluation can never be selected as a winner again.

Per-film training breakdown

The 75.3% LVFace training figure is a macro average across 4 films, not a uniform result:

film	LVFace F1	mbf F1	r18 F1	best model
Café Society	68.1%	62.2%	60.1%	LVFace
Lord of War	75.6%	77.2%	75.6%	mbf
Scarface	71.5%	68.6%	64.1%	LVFace
Sound of Metal	78.8%	76.5%	71.6%	LVFace

LVFace does not win every training film. mbf scores higher on Lord of War (77.2% vs 75.6%). LVFace's own training-film range is 68.1% to 78.8%, a 10.7pp spread, smaller than the 37pp spread seen on held-out films but real. Reproduce with scripts/docs/run_holdout_all_models.py --films training.

Held-out validation, all 3 models

The training matrix above is training-set fit. Each model's own tuned full_exp config was replayed against the 5 held-out films, scored the same way:

film	LVFace F1	mbf F1	r18 F1
Benny & Joon	83.0%	78.5%	77.1%
Lovelace	77.5%	73.7%	72.2%
Valerian and the City of a Thousand Planets	74.1%	70.2%	71.0%
Downton Abbey: A New Era	56.2%	55.0%	53.0%
The Many Saints of Newark	46.3%	44.5%	42.1%
macro average	67.4%	64.4%	63.1%

LVFace scores highest on every one of the 5 held-out films; the ranking never flips. Total misIDs across the 5 films: LVFace 1032, mbf 2197, r18 1224. LVFace has less than half mbf's misID count while also scoring higher on every film. This directly confirms the model choice out of sample; it is not inferred from the training numbers alone. See the LVFace deep dive for frame-level detail on where and why LVFace still fails on the two worst films. Reproduce with scripts/docs/run_holdout_all_models.py.

Two effects in isolation: gallery scope and pose expansion

Averaging across the 3 compared models (r50 excluded) isolates each variable from model choice.

Gallery scope, averaged over both expansion settings and all 3 models (6 evaluations per row):

scope	F1	P	R	total misID
full	71.1%	89.6%	59.6%	1121
restricted	75.9%	90.4%	65.6%	299

Restriction improves every metric at once. This is not a precision/recall trade: +4.8pp F1, +6.0pp recall, and roughly a quarter the misIDs. Fewer candidates in the matcher's search space means fewer opportunities for a lookalike false match, and the recall gain shows this does not cost real detections. Restriction is currently an offline optimizer technique, not a runtime feature of the application; see Whole vs. cast-restricted gallery for what building it into the application would require.

Pose expansion (promoting a confidently identified track's novel-pose views into a per-film gallery annex, src/gallery/track_gallery.hpp):

scope	expansion	F1	R	misID
full	off	70.0%	57.6%	407
full	on	72.1%	61.5%	714
restricted	off	75.1%	63.9%	179
restricted	on	76.7%	67.2%	120

In restricted mode, expansion is a clean win: +1.6pp F1, +3.3pp recall, misID drops. The annex only competes against the film's own roughly 15-actor cast, so a new pose of a known actor is unlikely to be confused with someone else. In full mode, expansion buys +2.1pp F1 and +3.9pp recall but at a real cost: misID rises from 407 to 714 as the same new-pose view now competes against the full 2418-actor gallery, where a confidently learned pose is more likely to match the wrong person. On the full gallery it is a recall-vs-misID trade, not a free gain. This training-set effect did not reproduce on held-out data; see Does pose expansion help? for the full held-out test and the two methodology bugs caught while checking it.

Calibration curves

Each gallery carries a fitted Platt sigmoid P(match | sim) = σ(a·sim + b), stored directly in the gallery HDF5 (src/gallery/gallery_calibration.hpp). This measures discriminative power independent of whatever prob_threshold a given run used:

LVFace has the steepest curve (a=17.7 vs 15.3-16.2 for the ArcFace variants) and the lowest P=0.5 decision boundary (similarity 0.23 vs 0.27-0.31), separating same-actor from different-actor pairs more confidently at a lower similarity than any ArcFace variant tested, including r50. Generated by scripts/docs/calibration_chart.py.

Extinction and anneal window search

Every one of the 512 DE evaluations for the winning LVFace full_exp combo, plotted over the prob_threshold × extinction_sec plane:

Nearly everything scoring well sits at extinction_sec above 50, across a wide range of thresholds. Short extinction windows are uniformly weaker: under a strict threshold, there is no good configuration in that region of the search space. The optimizer converged with anneal_sec=59.2, extinction_sec=59.2, about 99% of the widened 60s bound, which raises an open question not resolved in this round: does performance keep improving past 60s, or does it plateau there. Not chased further this pass.

Caveats

• r50's 4 combos used the older, narrower search bounds (1-30/1-15 instead of 1-60/1-60) and are further confounded by its thinner gallery. Excluded from all comparisons above except calibration.
• The shipped defaults use full_exp (75.3% training F1), not the higher-scoring restricted_exp (78.3%), because cast restriction is not a runtime feature of the application yet.
• expand_gallery is mode-dependent, not a free win. Averaged across models on the full gallery it trades misIDs for recall (see the pose-expansion table). For LVFace specifically, though, full_exp beats full_noexp on every axis at once (F1 75.3 vs 72.3, precision 89.7 vs 88.3, recall 65.4 vs 61.8, misID 232 vs 244), so the shipped full_exp is a clean choice for this model, not an F1-vs-safety trade. (An earlier version of this page reported full_noexp at 72.4% with zero misIDs and higher precision, which made it look like the safer option; that was the dropped-film artifact described above, not a real property of the config.)
• Switching the default model is an operational change: any gallery built from a different model's embeddings must be rebuilt before the new default takes effect.

Reproduce

# 4-film training matrix, all 4 models × 2 gallery modes × 2 expansion settings
bash experiments/run_rep4_subprocess.sh

# single combo
SAE_EXPAND=1 REPLAY_WORKERS=4 DE_WORKERS=2 python3 scripts/optimizer/optimize.py \
    --manifest experiments/manifests/rep4_LVFace-B_Glint360K_full.json \
    --gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5 \
    --params prob_threshold:0.5:0.999 anneal_sec:1:60 extinction_sec:1:60 \
    --popsize 10 --maxiter 15 --trajectory traj.jsonl --out best.json

# held-out validation, all 3 models, 5 films
python3 scripts/docs/run_holdout_all_models.py --out docs_data/holdout_all_models.json

# per-film training breakdown, all 3 models, 4 films
python3 scripts/docs/run_holdout_all_models.py --films training --out docs_data/training_per_film.json

# gallery coverage per film
python3 scripts/docs/gallery_coverage_per_film.py --out docs_data/gallery_coverage_per_film.json

# regenerate this page's charts from experiments/ artifacts
python3 scripts/docs/experiment_charts.py --out-dir docs/assets/images

# one frame per distinct out-of-cast name across all 9 films (used in the deep dive)
python3 scripts/docs/first_fpi_frames.py

See also the session log experiments/SESSION_STATE.md.