scene-actor-extraction/docs/model-bakeoff.md

Model bake-off + threshold re-tune — experiment log (2026-07-18/19)

Follow-on to the prior optimizer round, which used an older, since-superseded scene-union metric. This round uses the per-second metric (scripts/optimizer/second_score.py) and answers three questions in one 16-run matrix: which embedding model is best, does cast-restriction cut misIDs, and does per-film gallery expansion help.

(This was the fourth optimizer campaign against the X-Ray corpus, so its on-disk artifacts carry an internal rep4_ prefix — experiments/results/rep4_best_*.json, experiments/trajectories/rep4_*.jsonl, and the manifests referenced below. The earlier campaigns used the superseded scene-union metric and were discarded.)

Why this experiment, and what it actually delivered

Four goals going in, and an honest read on each after held-out validation (see below):

1. Find the best default parameters to ship. Partially delivered. The DE optimum generalizes unevenly — strong on 3 of 5 held-out films, badly broken on 2 (one with a 974-count misID blowup). The tuned values are shipped anyway (see Caveats) because they still beat the old defaults on average, but this is not a settled, film-agnostic optimum.
2. Find the best default model. Delivered with more confidence. LVFace beat r50/r18/mbf across all 4 training combos, and nothing in held-out validation contradicts the model choice specifically — the held-out failures trace to extinction_sec/threshold interactions and gallery coverage, not the embedder.
3. Provide insight into how the application works. The strongest, most durable output. Found and fixed a real teardown deadlock bug (100% reproducible, not the assumed rare GPU flake), established a real concurrency ceiling (8 parallel replays, not more), and found a real parameter interaction (a strict prob_threshold "earns" a longer extinction window before it starts hurting).
4. Demonstrate limitations. Delivered, and reinforced hard by held-out validation — see the "Held-out validation" section below for concrete examples, including a screenshot of the matcher naming 15 actors, none correctly, on a completely blank title card.

TL;DR — what changed in src/config.hpp

knob	old default	new default	why
arcface_model	arcface_w600k_r50.onnx	LVFace-B_Glint360K.onnx	Best F1 in the full-gallery bake-off (75.3% vs r50's 68.5%). LVFace was worth its size.
prob_threshold	0.76	0.754	Re-tuned for LVFace + per-second metric.
extinction_sec	1.5	57.4	Reverses the earlier "short is better" finding — see below.
anneal_sec	10.0	35.5	Same reversal; previously thought insensitive.
expand_gallery	false	true	Helped recall on the full (unrestricted) gallery for the winning model — opposite of the earlier assumption.

These are the LVFace-B_Glint360K_full_exp winning values, applied to src/config.hpp — the best result that uses only features already live in the running app (full gallery, no cast restriction; see below for why restricted mode isn't applied even though it scored higher).

Why re-run at all

The prior round's scene-union metric hid out-of-cast false positives behind a gallery∩cast recall mask, and the earlier 9-film benchmark was scene-level (union over a whole X-Ray scene), not a fair per-timepoint comparison. SESSION_STATE flagged the old scene-metric bake-off numbers (R50≈LVFace≈MBF ~85%) as superseded. This round uses second_score.py: uniform per-second sampling, GT = X-Ray scene's cast at time t, pred = actors whose presence window covers t, FPI weighted 10× when the named actor isn't in the film's cast at all (true misID) vs. an in-cast timing slip. FN only counts gallery-known cast (fair recall — 67% of X-Ray cast have no reference embedding; see the prior round's gallery-coverage-gap analysis).

The deadlock that was blocking all of this

Every replay in this line of work goes through scripts/optimizer/replay.py, which runs the real C++ tracker/matcher/scene_tracker nodes inside a Python-assembled KPN network. Before this session, every subprocess replay timed out at 45s, 100% of the time — not the documented ~20-30% ROCm rocBLAS-GEMM driver flake, but a plain logic bug: replay.py's CLI called replay(net, ..., stop=False) to skip net.stop() (trying to dodge the GEMM deadlock), planning to os._exit(0) immediately after. But:

• PyNode::stop() (include/kpn/python/bindings.hpp in the KPN++ submodule) is the only code that sets stop_flag_ = true before joining the node's worker thread.
• The source node's run_loop() has while (!stop_flag_) as its only exit condition (it has no input channels, so it never sees a channel-closed signal either).
• Skipping stop() meant stop_flag_ never became true. When replay() returned, its local net went out of scope immediately, running ~PyNetwork → ~PyNode → thread_.join() synchronously inside replay()'s own call frame — before main() ever got control back to run os._exit(0).

Root-caused via gdb -p <pid> -batch -ex "thread apply all bt" on a hung process: the main thread was stuck in ~PyNode's jthread::join(); the worker thread was in an ordinary time.sleep() inside the Python source callback, waiting for a stop signal that was never sent. The two HSA kfd_wait_on_events threads visible in the same trace are normal ROCm runtime housekeeping, not evidence of a wedged GPU kernel.

Fix: replay.py now calls replay(..., stop=True) (the removed stop=False + os._exit workaround was actively harmful). Verified 3/3 clean runs at ~8s each (down from a guaranteed 45s timeout), and a full DE sweep producing real, sensible F1/precision/recall instead of flat 0.0%.

Concurrency tuning

With the deadlock fixed, scripts/optimizer/optimize.py was extended with DE-level parallelism — differential_evolution(..., workers=ThreadPoolExecutor.map) — so multiple population candidates evaluate concurrently, each spawning its own per-film replay subprocesses (REPLAY_WORKERS). Total concurrent GPU replay processes ≈ DE_WORKERS × REPLAY_WORKERS.

concurrent replays	result
3 (REPLAY_WORKERS=3, no DE parallelism)	baseline, GPU underutilised
6 (DE_WORKERS=2 × REPLAY_WORKERS=3)	clean, real scores, ~1 isolated timeout per run
8 (DE_WORKERS=2 × REPLAY_WORKERS=4, 4-film manifest)	clean, real scores
9 (DE_WORKERS=3 × REPLAY_WORKERS=3)	broken — every replay blew past the 45s timeout, all scores silently degraded to 0.0%

9 concurrent replays looks like valid output (well-formed JSON, a real number) while actually being garbage — a dangerous failure mode, not a crash. 8 concurrent is the practical ceiling on this GPU (gfx1100) for this workload. The matrix ran at REPLAY_WORKERS=4 DE_WORKERS=2.

Training films and validation set

9 films total have dumped embeddings across all 4 models. 4 were used for optimization, leaving 5 held out for validation:

• Lord of War (64-cast, "clean")
• Scarface (67-cast, "ensemble/lookalike")
• Sound of Metal (14-cast, "high gallery-coverage")
• Café Society (62-cast, added this round — similar ensemble size to Scarface but different genre/lighting; picked to add diversity, not genre-overlap, over Downton Abbey or The Many Saints of Newark)

Held out: Benny & Joon, Downton Abbey: A New Era, Lovelace, The Many Saints of Newark, Valerian and the City of a Thousand Planets. The training-set numbers below are training-set fit — see "Held-out validation" further down for the real generalization test.

Search space and DE settings

popsize=10, maxiter=15 (3 params → ≤480 evals/combo ceiling; DE's tol convergence usually stops earlier). anneal_sec/extinction_sec bounds were widened from 1–30/1–15 to 1–60/1–60 mid-run (see below) — the 4 arcface_w600k_r50 combos finished before the widening and still use the old, narrower bounds, so they are not directly comparable to the other 12 on those two params. Re-running r50 with the wider bounds was deferred (diminishing-returns judgment call, not yet done).

Results — all 16 combos (4 models × {full, restricted} × {expand, noexp})

Ranked by F1. misid = FPI_misid, count of true wrong-actor identifications (an actor named who isn't in the film's cast at all) — distinct from FPI, which includes in-cast timing slips.

combo	F1	P	R	TPI	FPI	misid	FN
LVFace-B_Glint360K_restricted_exp	78.3%	91.0%	68.9%	42830	3782	60	19492
LVFace-B_Glint360K_restricted_noexp	76.7%	91.5%	66.2%	41149	3400	59	21173
arcface_w600k_mbf_restricted_exp	76.5%	90.7%	66.3%	41270	4431	0	21052
arcface_r18_restricted_exp	75.5%	87.6%	66.5%	41399	5666	60	20923
LVFace-B_Glint360K_full_exp	75.3%	89.7%	65.4%	47757	3407	232	26966
arcface_w600k_mbf_restricted_noexp	75.0%	91.1%	63.9%	39752	3465	60	22570
arcface_w600k_mbf_full_noexp	74.2%	87.4%	64.4%	12645	1312	57	6985
arcface_r18_restricted_noexp	73.5%	91.3%	61.7%	38299	3220	60	24023
LVFace-B_Glint360K_full_noexp	72.4%	94.2%	58.9%	27077	1725	0	19506
arcface_w600k_mbf_full_exp	72.0%	87.7%	61.4%	39875	3729	240	26338
arcface_w600k_r50_full_noexp †	71.6%	96.7%	56.9%	22471	361	45	17012
arcface_w600k_r50_restricted_exp †	71.1%	96.5%	56.4%	34954	1119	15	27368
arcface_w600k_r50_restricted_noexp †	69.2%	97.9%	53.6%	21146	327	15	18337
arcface_r18_full_exp	69.1%	87.6%	57.7%	37342	3119	242	28871
arcface_w600k_r50_full_exp †	68.5%	94.0%	54.1%	34982	903	150	31231
arcface_r18_full_noexp	66.6%	91.3%	53.1%	34314	2362	107	31899

† old, narrower anneal/extinction bounds (see above) — not directly comparable to the other 12 on those two params.

The same 16 results as a picture — the two headline effects are visible without reading a single row: filled (restricted) dots stack the top of the ranking for every model color, and yellow (LVFace) leads within both scopes:

Calibration curves — discriminative power, independent of the threshold

Each model's gallery carries a fitted Platt sigmoid P(match | sim) = σ(a·sim + b) (embedded directly in the gallery HDF5, see src/gallery/gallery_calibration.hpp). Plotting all four side by side shows discriminative power directly, independent of whatever prob_threshold a particular run happened to use:

LVFace-B has both the steepest curve (a=17.7, vs. 15.3–16.2 for the ArcFace variants) and the lowest P=0.5 decision boundary (similarity 0.23 vs. 0.27–0.31) — it separates same-actor from different-actor pairs more confidently at a lower similarity, consistent with it winning the full-gallery F1 comparison below. Generated by scripts/docs/calibration_chart.py (requires each gallery to have been calibrated at least once — run any replay against it first).

Two effects in isolation: gallery scope, and pose expansion

The matrix crosses two independent variables — averaging across all 4 models isolates each one from model choice:

Gallery scope (whole 2418-actor gallery vs. restricted to the film's credited cast) — averaged over both expansion settings and all 4 models:

scope	F1	P	R	total misID (16 evals→8 each)
full	71.2%	91.1%	59.0%	1073
restricted	74.5%	92.2%	62.9%	329

Restriction wins outright on every axis — not a precision/recall trade, a clean win: +3.3pp F1, +3.9pp recall, and less than a third the total misIDs. Fewer candidates in the matcher's search space means fewer opportunities for a look-alike false match, and (per the recall gain) doesn't cost real detections. This is the single cleanest signal in the whole matrix — stronger than the model choice itself — which is exactly why cast-restriction becoming a real runtime feature (not just an optimizer trick) is the top item in Caveats below.

Pose expansion (promoting a confidently-identified track's novel-pose views into a per-film gallery annex — src/gallery/track_gallery.hpp) is smaller and interacts with scope rather than acting independently:

scope	expansion	F1	R	misID
full	off	71.2%	58.3%	209
full	on	71.2%	59.7%	864
restricted	off	73.6%	61.3%	194
restricted	on	75.4%	64.5%	135

In restricted mode, expansion is a clean win (+1.8pp F1, +3.2pp recall, misID actually drops) — the annex only ever competes against the film's own ~15-actor cast, so a "confidently identified, new pose" view is unlikely to be mistaken for someone else. In full mode, expansion buys essentially nothing on F1 (71.2% → 71.2%, recall +1.4pp) while quadrupling misIDs (209 → 864): a novel-pose view promoted into the annex now competes against the whole 2418-actor gallery, so a "confident" identity is confident against the wrong universe of candidates — the expansion mechanism is "learning" a pose correctly, but the enlarged evidence pool makes it easier for that learned pose to look like a plausible match for a different actor. Practical takeaway: gallery expansion should be paired with cast restriction, not used on the full gallery — the version currently shipped as default (full_exp, see TL;DR) sits in the worse of these four cells for this specific knob, even though it's the best available combo without cast-restriction support in the app yet (see Caveats).

What the data says

• LVFace was worth its size. It wins full-gallery mode outright (75.3% vs r50's 68.5%, r18's 69.1%, mbf's 72.0%) with the highest recall of any full-mode combo — the earlier scene-union-metric conclusion ("not worth it") doesn't survive the better metric.
• Cast-restriction is a consistent, broad win. Every model's best combo is restricted. It isn't just precision-safe: arcface_w600k_mbf_restricted_exp and LVFace-B_Glint360K_full_noexp both hit misid=0 — zero true wrong-actor identifications. But restriction is an offline optimizer technique, not a live app feature — it pre-filters each film's gallery to its Jellyfin-credited cast before the matcher ever runs; there's no runtime "restrict to this film's cast" switch in the app today. Implementing it for real is future work, tracked separately from this defaults update.
• Gallery expansion (expand_gallery) is mode-dependent. It helps on restricted galleries (smaller, so novel-pose promotion adds real signal) and on LVFace's full gallery, but hurts r50 and mbf in full mode (compare arcface_w600k_r50_full_exp 68.5% vs full_noexp 71.6%). Don't assume it's a free win — model- and mode-dependent.
• arcface_r18 (smallest/cheapest) is last across all 4 modes — model capacity matters here, this isn't just parameter-count padding.
• anneal_sec/extinction_sec kept pinning at the search ceiling. With the original 1–30/1–15 bounds, 3 of 4 r50 combos landed at ~93-98% of the upper bound. Widened to 1–60/1–60 mid-run (after the r50 combos had already finished) — every subsequent combo's best config landed at ~90%+ of the new ceiling too (e.g. the LVFace winner: ann=59.2, ext=59.2, both ~99% of 60). The likely mechanism: a strict prob_threshold "earns" a long extinction/anneal window — once false matches are rare, a long window just bridges real presence gaps (occlusion, turned face) instead of smearing false positives into later scenes, which is what made short windows look better under the old, laxer thresholds. Open question, not resolved: does this keep climbing past 60s, or does it actually plateau there? Decided not to chase further this round (diminishing-returns judgment call) — flag for a future sweep if it matters.

The ceiling-pinning is visible in the raw search itself. Every one of the 512 DE evaluations for the winning combo, plotted over the prob_threshold × extinction_sec plane:

The dark band hugging the top edge is the finding: nearly everything scoring well sits at extinction_sec ≥ 50, across a wide range of thresholds, and the population converged into a dense cloud around the optimum (threshold ~0.70–0.80, extinction pinned at the 60s bound). Short extinction windows (bottom half) are uniformly pale — under a strict threshold there is simply no good configuration down there. Generated by scripts/docs/experiment_charts.py from the DE trajectories (experiments/trajectories/*.jsonl, part of the experiment-data artifact package).

Held-out validation — the number that actually matters

The 16-combo matrix above is training-set fit. This is the real test: the shipped config (LVFace-B_Glint360K_full_exp — prob_threshold=0.754, anneal_sec=35.5, extinction_sec=57.4, expand_gallery=true) replayed against the 5 films never seen by the optimizer (Benny & Joon, Downton Abbey: A New Era, Lovelace, The Many Saints of Newark, Valerian and the City of a Thousand Planets), scored the same way.

film	F1	P	R	agree	TPI	FPI	misid	FN
Benny & Joon	83.0%	89.1%	77.7%	72.4%	15125	1846	0	4337
Lovelace	77.5%	90.3%	67.9%	72.1%	14990	1085	58	7085
Valerian and the City of a Thousand Planets	74.1%	97.1%	60.0%	58.8%	18663	548	0	12467
Downton Abbey: A New Era	56.2%	97.8%	39.4%	40.6%	52027	1173	0	80084
The Many Saints of Newark	46.3%	54.7%	40.1%	37.0%	15922	4394	974	23791
macro average (5 films)	67.4%	85.8%	57.0%	56.2%	116727	9046	1032	127764

67.4% held out vs. 75.3% on training — an ~8pp drop, and a much more informative number than the training-set F1 alone: a 37pp spread between best and worst film (83.0% vs 46.3%). The config does not generalize uniformly.

Two films are outright failure cases, and rendering bounding boxes + names on the extracted frames (replay.py --raw-out + dump_error_frames.py --raw, see Reproduce) turned what looked like a same-scene misidentification into something more precise and more damning:

• The Many Saints of Newark (mob-family drama, picked as a training-adjacent genre test) has 974 true misIDs — far more than any training combo saw at any setting. The annotated frame below shows the same mechanism as Downton Abbey, at smaller scale: "Jon Bernthal 100%", "Joey Diaz 100%", and "Billy Magnussen 100%" are all frozen boxes over empty background — a blurred wall, hanging plates — with no face in them at all. Only one real face in frame has a box, and it carries a second, colliding label ("Leslie Odom Jr." and "Michael Gandolfini" both at high confidence on the same box) — likely two tracks whose frozen bboxes happen to overlap.

  Frame many_saints/fpi/fpi_t03543.jpg from the montage-frames artifact package (scripts/artifacts/pull_artifacts.sh montage-frames Many_Saints_of_Newark).

• Downton Abbey: A New Era (large ensemble, 36-cast) has high precision (97.8%) but recall collapses to 39.4% (FN=80084, by far the largest of the 5). Its starkest failure happens where there is nothing to see at all: the film's hard cut into its closing credits, where the matcher kept reporting 15 actors — all wrong — for nearly a minute of faceless screen.

Both are the same mechanism, and it can be measured, not just screenshotted. Plotting the dump's own per-second face_count (detector output, independent of the tracker) against the number of actors the tracker reports, through Downton Abbey's cut to credits:

From the cut onward the detector sees zero faces — yet the tracker holds a perfectly flat plateau of 15 reported identities for 56 seconds, each with the exact same bbox, unchanged to the pixel (verified for Hugh Bonneville: (1743.2, 0.0, 171.3, 317.8) at every sampled second from 7222 through 7279+). The staircase on the right edge is the extinction window finally expiring, actor by actor. That plateau is SceneTrackerFunc's active_[actor_idx].last_bbox (src/nodes/scene_tracker_node.hpp) being re-emitted unchanged — the extinction state machine working exactly as coded, not a bug in the logic. The film cuts from a packed group shot straight into ~40+ seconds of blank titles/credits with zero faces, and extinction_sec=57.4 is comfortably long enough to bridge that entire gap without expiring, so the tracker faithfully keeps reporting "last known position" for a cast that is no longer on screen at all.

This reframes the "long extinction window wins" DE-search pattern (see above): it isn't unambiguously good. It buys recall by bridging real gaps (occlusion, turned face) in some films, but on others — specifically, hard cuts into long faceless footage — it manufactures a frozen-bbox ghost the tracker has no way to verify, precisely the failure mode the original short-extinction-window default (1.5s) was chosen to avoid. The training-set films apparently didn't have a long enough faceless stretch after a confirmed identity to expose this; the held-out set did.

Frames for all three films (benny_joon, many_saints, downton_abbey — one strong performer, two failure cases) are under experiments/results/holdout/frames/ (not committed — pull per film with scripts/artifacts/pull_artifacts.sh montage-frames <film-slug>), each with a manifest.json listing the bucket (best/fpi/fn), timestamp, and predicted vs. ground-truth actors for every dumped frame. Frames are annotated with bounding boxes + name/confidence (green = identified, orange = unknown), matching src/nodes/debug_renderer_node.hpp's colour convention. Generated by scripts/optimizer/dump_error_frames.py --raw <replay.py --raw-out output> (see Reproduce).

dump_error_frames.py --interval-sec 600 also supports a per-N-second sweep instead of the fixed best/fpi/fn buckets: one best (highest Jaccard) and one worst (lowest Jaccard) frame per 10-minute window across the whole film, e.g. experiments/results/holdout/frames/many_saints_intervals/ (13 windows × 2 = 26 frames for the ~2h Many Saints runtime) — a way to sample "how are we doing" evenly across a film's runtime rather than only at its most extreme seconds.

Caveats / what this is not

• r50's 4 combos used the old, narrower search bounds and aren't fully comparable to the other 12 on anneal_sec/extinction_sec.
• The applied defaults use full_exp, not the higher-scoring restricted_exp, because cast-restriction isn't a real runtime feature yet (see above). The 78.3% F1 number is not what the shipped defaults will produce — 75.3% is.
• full_exp is the best full-gallery combo, but not the safest. Per the isolated-effects analysis above, expand_gallery=true only cleanly pays off when paired with cast-restriction; on the full gallery it's flat on F1 while ~4x-ing misIDs (209→864, averaged across models). full_noexp scores lower (72.4% vs 75.3% for LVFace) but with zero true misIDs and higher precision (94.2% vs 89.7%). Kept full_exp as shipped since it's the highest-F1 option available without cast-restriction, but this is a real F1-vs-safety trade, not a strictly-better choice — worth revisiting if misID rate matters more than the last few points of F1 for a given deployment.
• Switching the default model is an operational change, not just a config tweak: any existing gallery built from r50 embeddings is incompatible with LVFace embeddings and needs rebuilding.

Reproduce

# 4-film matrix, all 4 models × 2 modes × 2 expansion settings
bash experiments/run_rep4_subprocess.sh

# single combo
SAE_EXPAND=1 REPLAY_WORKERS=4 DE_WORKERS=2 python3 scripts/optimizer/optimize.py \
    --manifest experiments/manifests/rep4_LVFace-B_Glint360K_full.json \
    --gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5 \
    --params prob_threshold:0.5:0.999 anneal_sec:1:60 extinction_sec:1:60 \
    --popsize 10 --maxiter 15 --trajectory traj.jsonl --out best.json

# replay the shipped config against a held-out film — --raw-out is needed to draw
# bboxes later (the merged pred.json has no per-frame bbox, only actor windows)
python3 scripts/optimizer/replay.py \
    --dump experiments/dumps/LVFace-B_Glint360K/dump_<slug>.h5 \
    --gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5 \
    --out pred.json --raw-out raw.jsonl --prob-threshold 0.754 --anneal-sec 35.54 \
    --extinction-sec 57.43 --expand-gallery

# regenerate the report's charts (16-combo ranking, DE landscape, held-out
# per-film F1, Downton ghost timeline) from the artifacts under experiments/
python3 scripts/docs/experiment_charts.py --out-dir docs/assets/images

# dump example frames (best-agreement / FPI / FN) for visual inspection, annotated
# with bounding boxes + names (--raw is optional; omit for unannotated frames)
python3 scripts/optimizer/dump_error_frames.py \
    --pred pred.json --raw raw.jsonl --xray experiments/xray/.../<xray_dir> \
    --movie "<path to source video>" \
    --gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5 \
    --out-dir experiments/results/holdout/frames/<name> --n-per-bucket 4

# or: one best + one worst frame per 10-minute window across the whole film
python3 scripts/optimizer/dump_error_frames.py \
    --pred pred.json --raw raw.jsonl --xray experiments/xray/.../<xray_dir> \
    --movie "<path to source video>" \
    --gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5 \
    --out-dir experiments/results/holdout/frames/<name>_intervals --interval-sec 600

See also: the prior optimizer round (superseded metric) and the session log experiments/SESSION_STATE.md.