scene-actor-extraction/docs/rep4-optimizer-results.md

# Rep4 model bake-off + threshold re-tune — experiment log (2026-07-18/19)

Follow-on to `docs/optimizer-experiments.md`, which used an older, since-superseded
scene-union metric. This round uses the **per-second** metric
(`scripts/optimizer/second_score.py`) and answers three questions in one 16-run
matrix: which embedding model is best, does cast-restriction cut misIDs, and does
per-film gallery expansion help.

## Why this experiment, and what it actually delivered

Four goals going in, and an honest read on each after held-out validation (see
below):

1. **Find the best default parameters to ship.** Partially delivered. The DE optimum
   generalizes *unevenly* — strong on 3 of 5 held-out films, badly broken on 2 (one
   with a 974-count misID blowup). The tuned values are shipped anyway (see
   Caveats) because they still beat the old defaults on average, but this is not a
   settled, film-agnostic optimum.
2. **Find the best default model.** Delivered with more confidence. LVFace beat
   r50/r18/mbf across all 4 training combos, and nothing in held-out validation
   contradicts the model choice specifically — the held-out failures trace to
   `extinction_sec`/threshold interactions and gallery coverage, not the embedder.
3. **Provide insight into how the application works.** The strongest, most durable
   output. Found and fixed a real teardown deadlock bug (100% reproducible, not the
   assumed rare GPU flake), established a real concurrency ceiling (8 parallel
   replays, not more), and found a real parameter interaction (a strict
   `prob_threshold` "earns" a longer extinction window before it starts hurting).
4. **Demonstrate limitations.** Delivered, and reinforced hard by held-out
   validation — see the "Held-out validation" section below for concrete examples,
   including a screenshot of the matcher naming 15 actors, none correctly, on a
   completely blank title card.

## TL;DR — what changed in `src/config.hpp`

| knob | old default | new default | why |
| ---- | ----------- | ----------- | --- |
| `arcface_model` | `arcface_w600k_r50.onnx` | **`LVFace-B_Glint360K.onnx`** | Best F1 in the full-gallery bake-off (75.3% vs r50's 68.5%). LVFace was worth its size. |
| `prob_threshold` | 0.76 | **0.754** | Re-tuned for LVFace + per-second metric. |
| `extinction_sec` | 1.5 | **57.4** | Reverses the earlier "short is better" finding — see below. |
| `anneal_sec` | 10.0 | **35.5** | Same reversal; previously thought insensitive. |
| `expand_gallery` | false | **true** | Helped recall on the full (unrestricted) gallery for the winning model — opposite of the earlier assumption. |

These are the **`LVFace-B_Glint360K_full_exp`** winning values — the best result that
uses only features already live in the running app (full gallery, no cast
restriction; see below for why restricted mode isn't applied even though it scored
higher).

## Why re-run at all

`docs/optimizer-experiments.md`'s scene-union metric hid out-of-cast false positives
behind a gallery∩cast recall mask, and the earlier 9-film benchmark was scene-level
(union over a whole X-Ray scene), not a fair per-timepoint comparison. SESSION_STATE
flagged the old scene-metric bake-off numbers (R50≈LVFace≈MBF ~85%) as superseded.
This round uses `second_score.py`: uniform per-second sampling, GT = X-Ray scene's
cast at time *t*, pred = actors whose presence window covers *t*, FPI weighted 10×
when the named actor isn't in the film's cast at all (true misID) vs. an in-cast
timing slip. FN only counts gallery-known cast (fair recall — 67% of X-Ray cast have
no reference embedding, see `gallery-coverage-gap` memory).

## The deadlock that was blocking all of this

Every replay in this line of work goes through `scripts/optimizer/replay.py`, which
runs the real C++ tracker/matcher/scene_tracker nodes inside a Python-assembled KPN
network. Before this session, every subprocess replay **timed out at 45s, 100% of
the time** — not the documented ~20-30% ROCm rocBLAS-GEMM driver flake, but a plain
logic bug: `replay.py`'s CLI called `replay(net, ..., stop=False)` to *skip*
`net.stop()` (trying to dodge the GEMM deadlock), planning to `os._exit(0)`
immediately after. But:

- `PyNode::stop()` (`external/KPN/include/kpn/python/bindings.hpp`) is the *only*
  code that sets `stop_flag_ = true` before joining the node's worker thread.
- The source node's `run_loop()` has `while (!stop_flag_)` as its only exit
  condition (it has no input channels, so it never sees a channel-closed signal
  either).
- Skipping `stop()` meant `stop_flag_` never became true. When `replay()` returned,
  its local `net` went out of scope immediately, running `~PyNetwork` → `~PyNode` →
  `thread_.join()` **synchronously inside `replay()`'s own call frame** — before
  `main()` ever got control back to run `os._exit(0)`.

Root-caused via `gdb -p <pid> -batch -ex "thread apply all bt"` on a hung process:
the main thread was stuck in `~PyNode`'s `jthread::join()`; the worker thread was in
an ordinary `time.sleep()` inside the Python source callback, waiting for a stop
signal that was never sent. The two HSA `kfd_wait_on_events` threads visible in the
same trace are normal ROCm runtime housekeeping, not evidence of a wedged GPU kernel.

**Fix:** `replay.py` now calls `replay(..., stop=True)` (the removed `stop=False` +
`os._exit` workaround was actively harmful). Verified 3/3 clean runs at ~8s each
(down from a guaranteed 45s timeout), and a full DE sweep producing real, sensible
F1/precision/recall instead of flat 0.0%.

## Concurrency tuning

With the deadlock fixed, `optimize.py` was extended with DE-level parallelism —
`differential_evolution(..., workers=ThreadPoolExecutor.map)` — so multiple
population candidates evaluate concurrently, each spawning its own per-film replay
subprocesses (`REPLAY_WORKERS`). Total concurrent GPU replay processes ≈
`DE_WORKERS × REPLAY_WORKERS`.

| concurrent replays | result |
| --- | --- |
| 3 (`REPLAY_WORKERS=3`, no DE parallelism) | baseline, GPU underutilised |
| 6 (`DE_WORKERS=2 × REPLAY_WORKERS=3`) | clean, real scores, ~1 isolated timeout per run |
| 8 (`DE_WORKERS=2 × REPLAY_WORKERS=4`, 4-film manifest) | clean, real scores |
| 9 (`DE_WORKERS=3 × REPLAY_WORKERS=3`) | **broken** — every replay blew past the 45s timeout, all scores silently degraded to 0.0% |

9 concurrent replays looks like valid output (well-formed JSON, a real number) while
actually being garbage — a dangerous failure mode, not a crash. **8 concurrent is the
practical ceiling** on this GPU (gfx1100) for this workload. The matrix ran at
`REPLAY_WORKERS=4 DE_WORKERS=2`.

## Training films (rep4) and validation set

9 films total have dumped embeddings across all 4 models. 4 were used for
optimization (rep4), leaving 5 held out for validation:

- **Lord of War** (64-cast, "clean")
- **Scarface** (67-cast, "ensemble/lookalike")
- **Sound of Metal** (14-cast, "high gallery-coverage")
- **Café Society** (62-cast, added this round — similar ensemble size to Scarface but
  different genre/lighting; picked to add diversity, not genre-overlap, over
  Downton Abbey or The Many Saints of Newark)

Held out: Benny & Joon, Downton Abbey: A New Era, Lovelace, The Many Saints of
Newark, Valerian and the City of a Thousand Planets. The rep4 numbers below are
training-set fit — see "Held-out validation" further down for the real
generalization test.

## Search space and DE settings

`popsize=10, maxiter=15` (3 params → ≤480 evals/combo ceiling; DE's `tol` convergence
usually stops earlier). `anneal_sec`/`extinction_sec` bounds were **widened from
1–30/1–15 to 1–60/1–60 mid-run** (see below) — the 4 `arcface_w600k_r50` combos
finished before the widening and still use the old, narrower bounds, so they are
**not directly comparable** to the other 12 on those two params. Re-running r50 with
the wider bounds was deferred (diminishing-returns judgment call, not yet done).

## Results — all 16 combos (4 models × {full, restricted} × {expand, noexp})

Ranked by F1. `misid` = FPI_misid, count of true wrong-actor identifications (an
actor named who isn't in the film's cast at all) — distinct from `FPI`, which
includes in-cast timing slips.

| combo | F1 | P | R | TPI | FPI | misid | FN |
|---|---|---|---|---|---|---|---|
| LVFace-B_Glint360K_restricted_exp | **78.3%** | 91.0% | 68.9% | 42830 | 3782 | 60 | 19492 |
| LVFace-B_Glint360K_restricted_noexp | 76.7% | 91.5% | 66.2% | 41149 | 3400 | 59 | 21173 |
| arcface_w600k_mbf_restricted_exp | 76.5% | 90.7% | 66.3% | 41270 | 4431 | **0** | 21052 |
| arcface_r18_restricted_exp | 75.5% | 87.6% | 66.5% | 41399 | 5666 | 60 | 20923 |
| **LVFace-B_Glint360K_full_exp** | **75.3%** | 89.7% | 65.4% | 47757 | 3407 | 232 | 26966 |
| arcface_w600k_mbf_restricted_noexp | 75.0% | 91.1% | 63.9% | 39752 | 3465 | 60 | 22570 |
| arcface_w600k_mbf_full_noexp | 74.2% | 87.4% | 64.4% | 12645 | 1312 | 57 | 6985 |
| arcface_r18_restricted_noexp | 73.5% | 91.3% | 61.7% | 38299 | 3220 | 60 | 24023 |
| LVFace-B_Glint360K_full_noexp | 72.4% | 94.2% | 58.9% | 27077 | 1725 | **0** | 19506 |
| arcface_w600k_mbf_full_exp | 72.0% | 87.7% | 61.4% | 39875 | 3729 | 240 | 26338 |
| arcface_w600k_r50_full_noexp † | 71.6% | 96.7% | 56.9% | 22471 | 361 | 45 | 17012 |
| arcface_w600k_r50_restricted_exp † | 71.1% | 96.5% | 56.4% | 34954 | 1119 | 15 | 27368 |
| arcface_w600k_r50_restricted_noexp † | 69.2% | 97.9% | 53.6% | 21146 | 327 | 15 | 18337 |
| arcface_r18_full_exp | 69.1% | 87.6% | 57.7% | 37342 | 3119 | 242 | 28871 |
| arcface_w600k_r50_full_exp † | 68.5% | 94.0% | 54.1% | 34982 | 903 | 150 | 31231 |
| arcface_r18_full_noexp | 66.6% | 91.3% | 53.1% | 34314 | 2362 | 107 | 31899 |

† old, narrower anneal/extinction bounds (see above) — not directly comparable to
the other 12 on those two params.

## Calibration curves — discriminative power, independent of the threshold

Each model's gallery carries a fitted Platt sigmoid `P(match | sim) = σ(a·sim + b)`
(embedded directly in the gallery HDF5, see `src/gallery/gallery_calibration.hpp`).
Plotting all four side by side shows discriminative power directly, independent of
whatever `prob_threshold` a particular run happened to use:

![Calibrated P(match|similarity) for all four models](assets/images/calibration_curves.png)

LVFace-B has both the steepest curve (`a=17.7`, vs. 15.3–16.2 for the ArcFace
variants) and the lowest P=0.5 decision boundary (similarity 0.23 vs. 0.27–0.31) —
it separates same-actor from different-actor pairs more confidently at a lower
similarity, consistent with it winning the full-gallery F1 comparison below.
Generated by `scripts/docs/calibration_chart.py` (requires each gallery to have
been calibrated at least once — run any replay against it first).

## Two effects in isolation: gallery scope, and pose expansion

The matrix crosses two independent variables — averaging across all 4 models
isolates each one from model choice:

**Gallery scope (whole 2418-actor gallery vs. restricted to the film's credited
cast)** — averaged over both expansion settings and all 4 models:

| scope | F1 | P | R | total misID (16 evals→8 each) |
|---|---|---|---|---|
| full | 71.2% | 91.1% | 59.0% | 1073 |
| **restricted** | **74.5%** | 92.2% | **62.9%** | **329** |

Restriction wins outright on every axis — not a precision/recall trade, a clean
win: **+3.3pp F1, +3.9pp recall, and less than a third the total misIDs.** Fewer
candidates in the matcher's search space means fewer opportunities for a
look-alike false match, and (per the recall gain) doesn't cost real detections.
This is the single cleanest signal in the whole matrix — stronger than the model
choice itself — which is exactly why cast-restriction becoming a real runtime
feature (not just an optimizer trick) is the top item in Caveats below.

**Pose expansion (promoting a confidently-identified track's novel-pose views into
a per-film gallery annex — `track_gallery.hpp`)** is smaller and interacts with
scope rather than acting independently:

| scope | expansion | F1 | R | misID |
|---|---|---|---|---|
| full | off | 71.2% | 58.3% | 209 |
| full | **on** | 71.2% | 59.7% | **864** |
| restricted | off | 73.6% | 61.3% | 194 |
| restricted | **on** | **75.4%** | **64.5%** | 135 |

In **restricted** mode, expansion is a clean win (+1.8pp F1, +3.2pp recall, misID
actually *drops*) — the annex only ever competes against the film's own ~15-actor
cast, so a "confidently identified, new pose" view is unlikely to be mistaken for
someone else. In **full** mode, expansion buys essentially nothing on F1 (71.2% →
71.2%, recall +1.4pp) while **quadrupling misIDs** (209 → 864): a novel-pose view
promoted into the annex now competes against the whole 2418-actor gallery, so a
"confident" identity is confident against the wrong universe of candidates — the
expansion mechanism is "learning" a pose correctly, but the enlarged evidence pool
makes it easier for that learned pose to look like a plausible match for a
different actor. **Practical takeaway: gallery expansion should be paired with
cast restriction, not used on the full gallery** — the version currently shipped
as default (`full_exp`, see TL;DR) sits in the worse of these four cells for this
specific knob, even though it's the best available combo without cast-restriction
support in the app yet (see Caveats).

## What the data says

- **LVFace was worth its size.** It wins full-gallery mode outright (75.3% vs r50's
  68.5%, r18's 69.1%, mbf's 72.0%) with the highest recall of any full-mode combo —
  the earlier scene-union-metric conclusion ("not worth it") doesn't survive the
  better metric.
- **Cast-restriction is a consistent, broad win.** Every model's best combo is
  `restricted`. It isn't just precision-safe: `arcface_w600k_mbf_restricted_exp` and
  `LVFace-B_Glint360K_full_noexp` both hit **misid=0** — zero true wrong-actor
  identifications. But restriction is an **offline optimizer technique, not a live
  app feature** — it pre-filters each film's gallery to its Jellyfin-credited cast
  before the matcher ever runs; there's no runtime "restrict to this film's cast"
  switch in the app today. Implementing it for real is future work, tracked
  separately from this defaults update.
- **Gallery expansion (`expand_gallery`) is mode-dependent.** It helps on
  `restricted` galleries (smaller, so novel-pose promotion adds real signal) and on
  LVFace's full gallery, but **hurts** r50 and mbf in full mode (compare
  `arcface_w600k_r50_full_exp` 68.5% vs `full_noexp` 71.6%). Don't assume it's a free
  win — model- and mode-dependent.
- **arcface_r18 (smallest/cheapest) is last across all 4 modes** — model capacity
  matters here, this isn't just parameter-count padding.
- **`anneal_sec`/`extinction_sec` kept pinning at the search ceiling.** With the
  original 1–30/1–15 bounds, 3 of 4 r50 combos landed at ~93-98% of the upper bound.
  Widened to 1–60/1–60 mid-run (after the r50 combos had already finished) — every
  subsequent combo's best config landed at ~90%+ of the *new* ceiling too (e.g. the
  LVFace winner: `ann=59.2, ext=59.2`, both ~99% of 60). The likely mechanism: a
  strict `prob_threshold` "earns" a long extinction/anneal window — once false
  matches are rare, a long window just bridges real presence gaps (occlusion, turned
  face) instead of smearing false positives into later scenes, which is what made
  short windows look better under the old, laxer thresholds. **Open question, not
  resolved**: does this keep climbing past 60s, or does it actually plateau there?
  Decided not to chase further this round (diminishing-returns judgment call) — flag
  for a future sweep if it matters.

## Held-out validation — the number that actually matters

The 16-combo matrix above is training-set fit. This is the real test: the shipped
config (`LVFace-B_Glint360K_full_exp` — `prob_threshold=0.754, anneal_sec=35.5,
extinction_sec=57.4, expand_gallery=true`) replayed against the **5 films never seen
by the optimizer** (Benny & Joon, Downton Abbey: A New Era, Lovelace, The Many
Saints of Newark, Valerian and the City of a Thousand Planets), scored the same way.

| film | F1 | P | R | agree | TPI | FPI | misid | FN |
|---|---|---|---|---|---|---|---|---|
| Benny & Joon | 83.0% | 89.1% | 77.7% | 72.4% | 15125 | 1846 | 0 | 4337 |
| Lovelace | 77.5% | 90.3% | 67.9% | 72.1% | 14990 | 1085 | 58 | 7085 |
| Valerian and the City of a Thousand Planets | 74.1% | 97.1% | 60.0% | 58.8% | 18663 | 548 | 0 | 12467 |
| Downton Abbey: A New Era | 56.2% | 97.8% | 39.4% | 40.6% | 52027 | 1173 | 0 | 80084 |
| **The Many Saints of Newark** | **46.3%** | **54.7%** | 40.1% | 37.0% | 15922 | 4394 | **974** | 23791 |
| **macro average (5 films)** | **67.4%** | 85.8% | 57.0% | 56.2% | 116727 | 9046 | 1032 | 127764 |

**67.4% held out vs. 75.3% on training** — an ~8pp drop, and a much more informative
number than the training-set F1 alone: a **37pp spread between best and worst film**
(83.0% vs 46.3%). The config does not generalize uniformly.

Two films are outright failure cases, and rendering bounding boxes + names on the
extracted frames (`replay.py --raw-out` + `dump_error_frames.py --raw`, see
Reproduce) turned what looked like a same-scene misidentification into something
more precise and more damning:

- **The Many Saints of Newark** (mob-family drama, picked as a training-adjacent
  genre test) has **974 true misIDs** — far more than any training combo saw at any
  setting. The annotated frame below shows the same mechanism as Downton Abbey,
  at smaller scale: **"Jon Bernthal 100%", "Joey Diaz 100%", and "Billy Magnussen
  100%" are all frozen boxes over empty background — a blurred wall, hanging plates —
  with no face in them at all.** Only one real face in frame has a box, and it
  carries a *second*, colliding label ("Leslie Odom Jr." and "Michael Gandolfini"
  both at high confidence on the same box) — likely two tracks whose frozen bboxes
  happen to overlap.

  ![Frozen ghost boxes over background, The Many Saints of Newark](assets/images/many_saints_ghost_fpi.jpg)
  *`experiments/results/holdout/frames/many_saints/fpi/fpi_t03543.jpg`*

- **Downton Abbey: A New Era** (large ensemble, 36-cast) has high precision (97.8%)
  but recall collapses to 39.4% (FN=80084, by far the largest of the 5). The frame
  below is the starkest evidence in this whole experiment: **the matcher named 15
  actors — all of them wrong — over a completely blank closing title card with no
  faces on screen at all.**

  ![15 ghost boxes over a blank title card, Downton Abbey: A New Era](assets/images/downton_abbey_ghost_fpi.jpg)
  *`experiments/results/holdout/frames/downton_abbey/fpi/fpi_t07242.jpg`*

Both are the same mechanism, verified directly against the HDF5 dump and the raw
per-frame stream (not just inferred from the screenshot): at the Downton Abbey
title card (t=7242), the dump's own `face_count` is **0 from t≈7240 onward** — no
detector output at all, confirmed independently of the pipeline. Yet all 15 "wrong"
actors are still marked visible, each with the *exact same bbox, unchanged to the
pixel*, repeated every single frame back to t=7222 (verified for Hugh Bonneville:
`(1743.2, 0.0, 171.3, 317.8)` at every sampled second from 7222 through 7279+). That
is `SceneTrackerFunc`'s `active_[actor_idx].last_bbox` (`scene_tracker_node.hpp`)
being re-emitted unchanged — **this is the extinction state machine working exactly
as coded**, not a bug in the logic. The film cuts from a packed group shot straight
into ~40+ seconds of blank titles/credits with zero faces, and `extinction_sec=57.4`
is comfortably long enough to bridge that entire gap without expiring, so the
tracker faithfully keeps reporting "last known position" for a cast that is no
longer on screen at all.

This reframes the "long extinction window wins" DE-search pattern (see above): it
isn't unambiguously good. It buys recall by bridging real gaps (occlusion, turned
face) in some films, but on others — specifically, hard cuts into long faceless
footage — it manufactures a frozen-bbox ghost the tracker has no way to verify,
precisely the failure mode the *original* short-extinction-window default (`1.5s`)
was chosen to avoid. The training-set films apparently didn't have a long enough
faceless stretch after a confirmed identity to expose this; the held-out set did.

Frames for all three films (`benny_joon`, `many_saints`, `downton_abbey` — one strong
performer, two failure cases) are under `experiments/results/holdout/frames/`, each
with a `manifest.json` listing the bucket (`best`/`fpi`/`fn`), timestamp, and
predicted vs. ground-truth actors for every dumped frame. Frames are annotated with
bounding boxes + name/confidence (green = identified, orange = unknown), matching
`debug_renderer_node.hpp`'s colour convention. Generated by
`scripts/optimizer/dump_error_frames.py --raw <replay.py --raw-out output>` (see
Reproduce).

`dump_error_frames.py --interval-sec 600` also supports a per-N-second sweep
instead of the fixed best/fpi/fn buckets: one best (highest Jaccard) and one worst
(lowest Jaccard) frame per 10-minute window across the whole film, e.g.
`experiments/results/holdout/frames/many_saints_intervals/` (13 windows × 2 = 26
frames for the ~2h Many Saints runtime) — a way to sample "how are we doing" evenly
across a film's runtime rather than only at its most extreme seconds.

## Caveats / what this is not

- **r50's 4 combos used the old, narrower search bounds** and aren't fully
  comparable to the other 12 on `anneal_sec`/`extinction_sec`.
- **The applied defaults use `full_exp`, not the higher-scoring `restricted_exp`**,
  because cast-restriction isn't a real runtime feature yet (see above). The
  78.3% F1 number is not what the shipped defaults will produce — 75.3% is.
- **`full_exp` is the best full-gallery combo, but not the safest.** Per the
  isolated-effects analysis above, `expand_gallery=true` only cleanly pays off
  when paired with cast-restriction; on the full gallery it's flat on F1 while
  ~4x-ing misIDs (209→864, averaged across models). `full_noexp` scores lower
  (72.4% vs 75.3% for LVFace) but with **zero** true misIDs and higher precision
  (94.2% vs 89.7%). Kept `full_exp` as shipped since it's the highest-F1 option
  available without cast-restriction, but this is a real F1-vs-safety trade, not
  a strictly-better choice — worth revisiting if misID rate matters more than
  the last few points of F1 for a given deployment.
- **Switching the default model is an operational change, not just a config tweak**:
  any existing gallery built from r50 embeddings is incompatible with LVFace
  embeddings and needs rebuilding.

## Reproduce

```bash
# 4-film matrix, all 4 models × 2 modes × 2 expansion settings
bash experiments/run_rep4_subprocess.sh

# single combo
SAE_EXPAND=1 REPLAY_WORKERS=4 DE_WORKERS=2 python3 scripts/optimizer/optimize.py \
    --manifest experiments/manifests/rep4_LVFace-B_Glint360K_full.json \
    --gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5 \
    --params prob_threshold:0.5:0.999 anneal_sec:1:60 extinction_sec:1:60 \
    --popsize 10 --maxiter 15 --trajectory traj.jsonl --out best.json

# replay the shipped config against a held-out film — --raw-out is needed to draw
# bboxes later (the merged pred.json has no per-frame bbox, only actor windows)
python3 scripts/optimizer/replay.py \
    --dump experiments/dumps/LVFace-B_Glint360K/dump_<slug>.h5 \
    --gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5 \
    --out pred.json --raw-out raw.jsonl --prob-threshold 0.754 --anneal-sec 35.54 \
    --extinction-sec 57.43 --expand-gallery

# dump example frames (best-agreement / FPI / FN) for visual inspection, annotated
# with bounding boxes + names (--raw is optional; omit for unannotated frames)
python3 scripts/optimizer/dump_error_frames.py \
    --pred pred.json --raw raw.jsonl --xray experiments/xray/.../<xray_dir> \
    --movie "<path to source video>" \
    --gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5 \
    --out-dir experiments/results/holdout/frames/<name> --n-per-bucket 4

# or: one best + one worst frame per 10-minute window across the whole film
python3 scripts/optimizer/dump_error_frames.py \
    --pred pred.json --raw raw.jsonl --xray experiments/xray/.../<xray_dir> \
    --movie "<path to source video>" \
    --gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5 \
    --out-dir experiments/results/holdout/frames/<name>_intervals --interval-sec 600
```

See also: `docs/optimizer-experiments.md` (prior round, superseded metric),
`experiments/SESSION_STATE.md`, and memory: kpn-python-replay-optimizer,
gallery-coverage-gap, xray-validation-results, per-scene-presence-eval-design.