docs: richer report — data figures, success/failure frames, commit-pinned repo links

- experiment_charts.py generates 4 figures from experiments/ artifacts: held-out per-film F1, 16-combo ranking, DE search landscape, and the Downton detector-vs-tracker ghost timeline (replaces the blank title-card screenshot) - new frames: 19-correct wedding shot (success case), Many Saints ghost-vs-unknown frame (three error classes in one image) - rename rep4-optimizer-results.md -> model-bakeoff.md; rep4 kept only as the on-disk artifact prefix, explained once - repo file references are now links via https://REPOLINK/<path> placeholders; build_site.sh pins them to the HEAD commit's raw URLs and fails the build if a linked path doesn't exist at HEAD - drop references to removed scripts (scene_score.py, score_config.py) and to session-memory names; mark artifact-registry paths with their pull commands - commit readme_example.jpg + pipeline_topology.svg so README renders on the plain Gitea repo view - deploy_pages.sh: push built site/ to the gitea-pages branch
2026-07-19 22:06:56 +02:00
parent 4925443e56
commit b1efefac6f
17 changed files with 702 additions and 146 deletions
@@ -0,0 +1,462 @@
+# Model bake-off + threshold re-tune — experiment log (2026-07-18/19)
+
+Follow-on to [the prior optimizer round](optimizer-experiments.md), which used
+an older, since-superseded scene-union metric. This round uses the **per-second** metric
+([`scripts/optimizer/second_score.py`](https://REPOLINK/scripts/optimizer/second_score.py))
+and answers three questions in one 16-run matrix: which embedding model is best, does cast-restriction cut misIDs, and does
+per-film gallery expansion help.
+
+(This was the fourth optimizer campaign against the X-Ray corpus, so its on-disk
+artifacts carry an internal `rep4_` prefix — `experiments/results/rep4_best_*.json`,
+`experiments/trajectories/rep4_*.jsonl`, and the manifests referenced below. The
+earlier campaigns used the superseded scene-union metric and were discarded.)
+
+## Why this experiment, and what it actually delivered
+
+Four goals going in, and an honest read on each after held-out validation (see
+below):
+
+1. **Find the best default parameters to ship.** Partially delivered. The DE optimum
+   generalizes *unevenly* — strong on 3 of 5 held-out films, badly broken on 2 (one
+   with a 974-count misID blowup). The tuned values are shipped anyway (see
+   Caveats) because they still beat the old defaults on average, but this is not a
+   settled, film-agnostic optimum.
+2. **Find the best default model.** Delivered with more confidence. LVFace beat
+   r50/r18/mbf across all 4 training combos, and nothing in held-out validation
+   contradicts the model choice specifically — the held-out failures trace to
+   `extinction_sec`/threshold interactions and gallery coverage, not the embedder.
+3. **Provide insight into how the application works.** The strongest, most durable
+   output. Found and fixed a real teardown deadlock bug (100% reproducible, not the
+   assumed rare GPU flake), established a real concurrency ceiling (8 parallel
+   replays, not more), and found a real parameter interaction (a strict
+   `prob_threshold` "earns" a longer extinction window before it starts hurting).
+4. **Demonstrate limitations.** Delivered, and reinforced hard by held-out
+   validation — see the "Held-out validation" section below for concrete examples,
+   including a screenshot of the matcher naming 15 actors, none correctly, on a
+   completely blank title card.
+
+## TL;DR — what changed in `src/config.hpp`
+
+| knob | old default | new default | why |
+| ---- | ----------- | ----------- | --- |
+| `arcface_model` | `arcface_w600k_r50.onnx` | **`LVFace-B_Glint360K.onnx`** | Best F1 in the full-gallery bake-off (75.3% vs r50's 68.5%). LVFace was worth its size. |
+| `prob_threshold` | 0.76 | **0.754** | Re-tuned for LVFace + per-second metric. |
+| `extinction_sec` | 1.5 | **57.4** | Reverses the earlier "short is better" finding — see below. |
+| `anneal_sec` | 10.0 | **35.5** | Same reversal; previously thought insensitive. |
+| `expand_gallery` | false | **true** | Helped recall on the full (unrestricted) gallery for the winning model — opposite of the earlier assumption. |
+
+These are the **`LVFace-B_Glint360K_full_exp`** winning values, applied to
+[`src/config.hpp`](https://REPOLINK/src/config.hpp) — the best result that
+uses only features already live in the running app (full gallery, no cast
+restriction; see below for why restricted mode isn't applied even though it scored
+higher).
+
+## Why re-run at all
+
+[The prior round](optimizer-experiments.md)'s scene-union metric hid out-of-cast false positives
+behind a gallery∩cast recall mask, and the earlier 9-film benchmark was scene-level
+(union over a whole X-Ray scene), not a fair per-timepoint comparison. SESSION_STATE
+flagged the old scene-metric bake-off numbers (R50≈LVFace≈MBF ~85%) as superseded.
+This round uses `second_score.py`: uniform per-second sampling, GT = X-Ray scene's
+cast at time *t*, pred = actors whose presence window covers *t*, FPI weighted 10×
+when the named actor isn't in the film's cast at all (true misID) vs. an in-cast
+timing slip. FN only counts gallery-known cast (fair recall — 67% of X-Ray cast have
+no reference embedding; see
+[the prior round's gallery-coverage-gap analysis](optimizer-experiments.md#the-gallery-coverage-gap)).
+
+## The deadlock that was blocking all of this
+
+Every replay in this line of work goes through
+[`scripts/optimizer/replay.py`](https://REPOLINK/scripts/optimizer/replay.py),
+which runs the real C++ tracker/matcher/scene_tracker nodes inside a
+Python-assembled KPN network. Before this session, every subprocess replay **timed out at 45s, 100% of
+the time** — not the documented ~20-30% ROCm rocBLAS-GEMM driver flake, but a plain
+logic bug: `replay.py`'s CLI called `replay(net, ..., stop=False)` to *skip*
+`net.stop()` (trying to dodge the GEMM deadlock), planning to `os._exit(0)`
+immediately after. But:
+
+- `PyNode::stop()` ([`include/kpn/python/bindings.hpp`](https://KPNLINK/include/kpn/python/bindings.hpp)
+  in the KPN++ submodule) is the *only*
+  code that sets `stop_flag_ = true` before joining the node's worker thread.
+- The source node's `run_loop()` has `while (!stop_flag_)` as its only exit
+  condition (it has no input channels, so it never sees a channel-closed signal
+  either).
+- Skipping `stop()` meant `stop_flag_` never became true. When `replay()` returned,
+  its local `net` went out of scope immediately, running `~PyNetwork` → `~PyNode` →
+  `thread_.join()` **synchronously inside `replay()`'s own call frame** — before
+  `main()` ever got control back to run `os._exit(0)`.
+
+Root-caused via `gdb -p <pid> -batch -ex "thread apply all bt"` on a hung process:
+the main thread was stuck in `~PyNode`'s `jthread::join()`; the worker thread was in
+an ordinary `time.sleep()` inside the Python source callback, waiting for a stop
+signal that was never sent. The two HSA `kfd_wait_on_events` threads visible in the
+same trace are normal ROCm runtime housekeeping, not evidence of a wedged GPU kernel.
+
+**Fix:** `replay.py` now calls `replay(..., stop=True)` (the removed `stop=False` +
+`os._exit` workaround was actively harmful). Verified 3/3 clean runs at ~8s each
+(down from a guaranteed 45s timeout), and a full DE sweep producing real, sensible
+F1/precision/recall instead of flat 0.0%.
+
+## Concurrency tuning
+
+With the deadlock fixed,
+[`scripts/optimizer/optimize.py`](https://REPOLINK/scripts/optimizer/optimize.py)
+was extended with DE-level parallelism —
+`differential_evolution(..., workers=ThreadPoolExecutor.map)` — so multiple
+population candidates evaluate concurrently, each spawning its own per-film replay
+subprocesses (`REPLAY_WORKERS`). Total concurrent GPU replay processes ≈
+`DE_WORKERS × REPLAY_WORKERS`.
+
+| concurrent replays | result |
+| --- | --- |
+| 3 (`REPLAY_WORKERS=3`, no DE parallelism) | baseline, GPU underutilised |
+| 6 (`DE_WORKERS=2 × REPLAY_WORKERS=3`) | clean, real scores, ~1 isolated timeout per run |
+| 8 (`DE_WORKERS=2 × REPLAY_WORKERS=4`, 4-film manifest) | clean, real scores |
+| 9 (`DE_WORKERS=3 × REPLAY_WORKERS=3`) | **broken** — every replay blew past the 45s timeout, all scores silently degraded to 0.0% |
+
+9 concurrent replays looks like valid output (well-formed JSON, a real number) while
+actually being garbage — a dangerous failure mode, not a crash. **8 concurrent is the
+practical ceiling** on this GPU (gfx1100) for this workload. The matrix ran at
+`REPLAY_WORKERS=4 DE_WORKERS=2`.
+
+## Training films and validation set
+
+9 films total have dumped embeddings across all 4 models. 4 were used for
+optimization, leaving 5 held out for validation:
+
+- **Lord of War** (64-cast, "clean")
+- **Scarface** (67-cast, "ensemble/lookalike")
+- **Sound of Metal** (14-cast, "high gallery-coverage")
+- **Café Society** (62-cast, added this round — similar ensemble size to Scarface but
+  different genre/lighting; picked to add diversity, not genre-overlap, over
+  Downton Abbey or The Many Saints of Newark)
+
+Held out: Benny & Joon, Downton Abbey: A New Era, Lovelace, The Many Saints of
+Newark, Valerian and the City of a Thousand Planets. The training-set numbers below are
+training-set fit — see "Held-out validation" further down for the real
+generalization test.
+
+## Search space and DE settings
+
+`popsize=10, maxiter=15` (3 params → ≤480 evals/combo ceiling; DE's `tol` convergence
+usually stops earlier). `anneal_sec`/`extinction_sec` bounds were **widened from
+1–30/1–15 to 1–60/1–60 mid-run** (see below) — the 4 `arcface_w600k_r50` combos
+finished before the widening and still use the old, narrower bounds, so they are
+**not directly comparable** to the other 12 on those two params. Re-running r50 with
+the wider bounds was deferred (diminishing-returns judgment call, not yet done).
+
+## Results — all 16 combos (4 models × {full, restricted} × {expand, noexp})
+
+Ranked by F1. `misid` = FPI_misid, count of true wrong-actor identifications (an
+actor named who isn't in the film's cast at all) — distinct from `FPI`, which
+includes in-cast timing slips.
+
+| combo | F1 | P | R | TPI | FPI | misid | FN |
+|---|---|---|---|---|---|---|---|
+| LVFace-B_Glint360K_restricted_exp | **78.3%** | 91.0% | 68.9% | 42830 | 3782 | 60 | 19492 |
+| LVFace-B_Glint360K_restricted_noexp | 76.7% | 91.5% | 66.2% | 41149 | 3400 | 59 | 21173 |
+| arcface_w600k_mbf_restricted_exp | 76.5% | 90.7% | 66.3% | 41270 | 4431 | **0** | 21052 |
+| arcface_r18_restricted_exp | 75.5% | 87.6% | 66.5% | 41399 | 5666 | 60 | 20923 |
+| **LVFace-B_Glint360K_full_exp** | **75.3%** | 89.7% | 65.4% | 47757 | 3407 | 232 | 26966 |
+| arcface_w600k_mbf_restricted_noexp | 75.0% | 91.1% | 63.9% | 39752 | 3465 | 60 | 22570 |
+| arcface_w600k_mbf_full_noexp | 74.2% | 87.4% | 64.4% | 12645 | 1312 | 57 | 6985 |
+| arcface_r18_restricted_noexp | 73.5% | 91.3% | 61.7% | 38299 | 3220 | 60 | 24023 |
+| LVFace-B_Glint360K_full_noexp | 72.4% | 94.2% | 58.9% | 27077 | 1725 | **0** | 19506 |
+| arcface_w600k_mbf_full_exp | 72.0% | 87.7% | 61.4% | 39875 | 3729 | 240 | 26338 |
+| arcface_w600k_r50_full_noexp † | 71.6% | 96.7% | 56.9% | 22471 | 361 | 45 | 17012 |
+| arcface_w600k_r50_restricted_exp † | 71.1% | 96.5% | 56.4% | 34954 | 1119 | 15 | 27368 |
+| arcface_w600k_r50_restricted_noexp † | 69.2% | 97.9% | 53.6% | 21146 | 327 | 15 | 18337 |
+| arcface_r18_full_exp | 69.1% | 87.6% | 57.7% | 37342 | 3119 | 242 | 28871 |
+| arcface_w600k_r50_full_exp † | 68.5% | 94.0% | 54.1% | 34982 | 903 | 150 | 31231 |
+| arcface_r18_full_noexp | 66.6% | 91.3% | 53.1% | 34314 | 2362 | 107 | 31899 |
+
+† old, narrower anneal/extinction bounds (see above) — not directly comparable to
+the other 12 on those two params.
+
+The same 16 results as a picture — the two headline effects are visible without
+reading a single row: filled (restricted) dots stack the top of the ranking for
+every model color, and yellow (LVFace) leads within both scopes:
+
+![All 16 bake-off combos ranked by training-set F1](assets/images/rep4_matrix_f1.png)
+
+## Calibration curves — discriminative power, independent of the threshold
+
+Each model's gallery carries a fitted Platt sigmoid `P(match | sim) = σ(a·sim + b)`
+(embedded directly in the gallery HDF5, see
+[`src/gallery/gallery_calibration.hpp`](https://REPOLINK/src/gallery/gallery_calibration.hpp)).
+Plotting all four side by side shows discriminative power directly, independent of
+whatever `prob_threshold` a particular run happened to use:
+
+![Calibrated P(match|similarity) for all four models](assets/images/calibration_curves.png)
+
+LVFace-B has both the steepest curve (`a=17.7`, vs. 15.3–16.2 for the ArcFace
+variants) and the lowest P=0.5 decision boundary (similarity 0.23 vs. 0.27–0.31) —
+it separates same-actor from different-actor pairs more confidently at a lower
+similarity, consistent with it winning the full-gallery F1 comparison below.
+Generated by
+[`scripts/docs/calibration_chart.py`](https://REPOLINK/scripts/docs/calibration_chart.py)
+(requires each gallery to have been calibrated at least once — run any replay
+against it first).
+
+## Two effects in isolation: gallery scope, and pose expansion
+
+The matrix crosses two independent variables — averaging across all 4 models
+isolates each one from model choice:
+
+**Gallery scope (whole 2418-actor gallery vs. restricted to the film's credited
+cast)** — averaged over both expansion settings and all 4 models:
+
+| scope | F1 | P | R | total misID (16 evals→8 each) |
+|---|---|---|---|---|
+| full | 71.2% | 91.1% | 59.0% | 1073 |
+| **restricted** | **74.5%** | 92.2% | **62.9%** | **329** |
+
+Restriction wins outright on every axis — not a precision/recall trade, a clean
+win: **+3.3pp F1, +3.9pp recall, and less than a third the total misIDs.** Fewer
+candidates in the matcher's search space means fewer opportunities for a
+look-alike false match, and (per the recall gain) doesn't cost real detections.
+This is the single cleanest signal in the whole matrix — stronger than the model
+choice itself — which is exactly why cast-restriction becoming a real runtime
+feature (not just an optimizer trick) is the top item in Caveats below.
+
+**Pose expansion (promoting a confidently-identified track's novel-pose views into
+a per-film gallery annex — [`src/gallery/track_gallery.hpp`](https://REPOLINK/src/gallery/track_gallery.hpp))**
+is smaller and interacts with
+scope rather than acting independently:
+
+| scope | expansion | F1 | R | misID |
+|---|---|---|---|---|
+| full | off | 71.2% | 58.3% | 209 |
+| full | **on** | 71.2% | 59.7% | **864** |
+| restricted | off | 73.6% | 61.3% | 194 |
+| restricted | **on** | **75.4%** | **64.5%** | 135 |
+
+In **restricted** mode, expansion is a clean win (+1.8pp F1, +3.2pp recall, misID
+actually *drops*) — the annex only ever competes against the film's own ~15-actor
+cast, so a "confidently identified, new pose" view is unlikely to be mistaken for
+someone else. In **full** mode, expansion buys essentially nothing on F1 (71.2% →
+71.2%, recall +1.4pp) while **quadrupling misIDs** (209 → 864): a novel-pose view
+promoted into the annex now competes against the whole 2418-actor gallery, so a
+"confident" identity is confident against the wrong universe of candidates — the
+expansion mechanism is "learning" a pose correctly, but the enlarged evidence pool
+makes it easier for that learned pose to look like a plausible match for a
+different actor. **Practical takeaway: gallery expansion should be paired with
+cast restriction, not used on the full gallery** — the version currently shipped
+as default (`full_exp`, see TL;DR) sits in the worse of these four cells for this
+specific knob, even though it's the best available combo without cast-restriction
+support in the app yet (see Caveats).
+
+## What the data says
+
+- **LVFace was worth its size.** It wins full-gallery mode outright (75.3% vs r50's
+  68.5%, r18's 69.1%, mbf's 72.0%) with the highest recall of any full-mode combo —
+  the earlier scene-union-metric conclusion ("not worth it") doesn't survive the
+  better metric.
+- **Cast-restriction is a consistent, broad win.** Every model's best combo is
+  `restricted`. It isn't just precision-safe: `arcface_w600k_mbf_restricted_exp` and
+  `LVFace-B_Glint360K_full_noexp` both hit **misid=0** — zero true wrong-actor
+  identifications. But restriction is an **offline optimizer technique, not a live
+  app feature** — it pre-filters each film's gallery to its Jellyfin-credited cast
+  before the matcher ever runs; there's no runtime "restrict to this film's cast"
+  switch in the app today. Implementing it for real is future work, tracked
+  separately from this defaults update.
+- **Gallery expansion (`expand_gallery`) is mode-dependent.** It helps on
+  `restricted` galleries (smaller, so novel-pose promotion adds real signal) and on
+  LVFace's full gallery, but **hurts** r50 and mbf in full mode (compare
+  `arcface_w600k_r50_full_exp` 68.5% vs `full_noexp` 71.6%). Don't assume it's a free
+  win — model- and mode-dependent.
+- **arcface_r18 (smallest/cheapest) is last across all 4 modes** — model capacity
+  matters here, this isn't just parameter-count padding.
+- **`anneal_sec`/`extinction_sec` kept pinning at the search ceiling.** With the
+  original 1–30/1–15 bounds, 3 of 4 r50 combos landed at ~93-98% of the upper bound.
+  Widened to 1–60/1–60 mid-run (after the r50 combos had already finished) — every
+  subsequent combo's best config landed at ~90%+ of the *new* ceiling too (e.g. the
+  LVFace winner: `ann=59.2, ext=59.2`, both ~99% of 60). The likely mechanism: a
+  strict `prob_threshold` "earns" a long extinction/anneal window — once false
+  matches are rare, a long window just bridges real presence gaps (occlusion, turned
+  face) instead of smearing false positives into later scenes, which is what made
+  short windows look better under the old, laxer thresholds. **Open question, not
+  resolved**: does this keep climbing past 60s, or does it actually plateau there?
+  Decided not to chase further this round (diminishing-returns judgment call) — flag
+  for a future sweep if it matters.
+
+The ceiling-pinning is visible in the raw search itself. Every one of the 512
+DE evaluations for the winning combo, plotted over the
+`prob_threshold` × `extinction_sec` plane:
+
+![DE search landscape: 512 evaluations over prob_threshold × extinction_sec](assets/images/de_search_landscape.png)
+
+The dark band hugging the top edge *is* the finding: nearly everything scoring
+well sits at `extinction_sec` ≥ 50, across a wide range of thresholds, and the
+population converged into a dense cloud around the optimum (threshold ~0.70–0.80,
+extinction pinned at the 60s bound). Short extinction windows (bottom half) are
+uniformly pale — under a strict threshold there is simply no good configuration
+down there. Generated by
+[`scripts/docs/experiment_charts.py`](https://REPOLINK/scripts/docs/experiment_charts.py)
+from the DE trajectories (`experiments/trajectories/*.jsonl`, part of the
+`experiment-data` artifact package).
+
+## Held-out validation — the number that actually matters
+
+The 16-combo matrix above is training-set fit. This is the real test: the shipped
+config (`LVFace-B_Glint360K_full_exp` — `prob_threshold=0.754, anneal_sec=35.5,
+extinction_sec=57.4, expand_gallery=true`) replayed against the **5 films never seen
+by the optimizer** (Benny & Joon, Downton Abbey: A New Era, Lovelace, The Many
+Saints of Newark, Valerian and the City of a Thousand Planets), scored the same way.
+
+| film | F1 | P | R | agree | TPI | FPI | misid | FN |
+|---|---|---|---|---|---|---|---|---|
+| Benny & Joon | 83.0% | 89.1% | 77.7% | 72.4% | 15125 | 1846 | 0 | 4337 |
+| Lovelace | 77.5% | 90.3% | 67.9% | 72.1% | 14990 | 1085 | 58 | 7085 |
+| Valerian and the City of a Thousand Planets | 74.1% | 97.1% | 60.0% | 58.8% | 18663 | 548 | 0 | 12467 |
+| Downton Abbey: A New Era | 56.2% | 97.8% | 39.4% | 40.6% | 52027 | 1173 | 0 | 80084 |
+| **The Many Saints of Newark** | **46.3%** | **54.7%** | 40.1% | 37.0% | 15922 | 4394 | **974** | 23791 |
+| **macro average (5 films)** | **67.4%** | 85.8% | 57.0% | 56.2% | 116727 | 9046 | 1032 | 127764 |
+
+![Held-out per-film F1 vs. the training-set fit](assets/images/holdout_f1_by_film.png)
+
+**67.4% held out vs. 75.3% on training** — an ~8pp drop, and a much more informative
+number than the training-set F1 alone: a **37pp spread between best and worst film**
+(83.0% vs 46.3%). The config does not generalize uniformly.
+
+Two films are outright failure cases, and rendering bounding boxes + names on the
+extracted frames (`replay.py --raw-out` +
+[`dump_error_frames.py`](https://REPOLINK/scripts/optimizer/dump_error_frames.py)` --raw`, see
+Reproduce) turned what looked like a same-scene misidentification into something
+more precise and more damning:
+
+- **The Many Saints of Newark** (mob-family drama, picked as a training-adjacent
+  genre test) has **974 true misIDs** — far more than any training combo saw at any
+  setting. The annotated frame below shows the same mechanism as Downton Abbey,
+  at smaller scale: **"Jon Bernthal 100%", "Joey Diaz 100%", and "Billy Magnussen
+  100%" are all frozen boxes over empty background — a blurred wall, hanging plates —
+  with no face in them at all.** Only one real face in frame has a box, and it
+  carries a *second*, colliding label ("Leslie Odom Jr." and "Michael Gandolfini"
+  both at high confidence on the same box) — likely two tracks whose frozen bboxes
+  happen to overlap.
+
+  ![Frozen ghost boxes over background, The Many Saints of Newark](assets/images/many_saints_ghost_fpi.jpg)
+  *Frame `many_saints/fpi/fpi_t03543.jpg` from the `montage-frames` artifact
+  package (`scripts/artifacts/pull_artifacts.sh montage-frames
+  Many_Saints_of_Newark`).*
+
+- **Downton Abbey: A New Era** (large ensemble, 36-cast) has high precision (97.8%)
+  but recall collapses to 39.4% (FN=80084, by far the largest of the 5). Its
+  starkest failure happens where there is nothing to see at all: the film's hard
+  cut into its closing credits, where **the matcher kept reporting 15 actors —
+  all wrong — for nearly a minute of faceless screen.**
+
+Both are the same mechanism, and it can be *measured*, not just screenshotted.
+Plotting the dump's own per-second `face_count` (detector output, independent
+of the tracker) against the number of actors the tracker reports, through
+Downton Abbey's cut to credits:
+
+![Detector face_count vs. tracker-reported actors through the cut to credits](assets/images/downton_ghost_timeline.png)
+
+From the cut onward the detector sees **zero faces** — yet the tracker holds a
+perfectly flat plateau of 15 reported identities for 56 seconds, each with the
+*exact same bbox, unchanged to the pixel* (verified for Hugh Bonneville:
+`(1743.2, 0.0, 171.3, 317.8)` at every sampled second from 7222 through 7279+).
+The staircase on the right edge is the extinction window finally expiring,
+actor by actor. That plateau is `SceneTrackerFunc`'s
+`active_[actor_idx].last_bbox`
+([`src/nodes/scene_tracker_node.hpp`](https://REPOLINK/src/nodes/scene_tracker_node.hpp))
+being re-emitted
+unchanged — **the extinction state machine working exactly as coded**, not a
+bug in the logic. The film cuts from a packed group shot straight into ~40+
+seconds of blank titles/credits with zero faces, and `extinction_sec=57.4` is
+comfortably long enough to bridge that entire gap without expiring, so the
+tracker faithfully keeps reporting "last known position" for a cast that is no
+longer on screen at all.
+
+This reframes the "long extinction window wins" DE-search pattern (see above): it
+isn't unambiguously good. It buys recall by bridging real gaps (occlusion, turned
+face) in some films, but on others — specifically, hard cuts into long faceless
+footage — it manufactures a frozen-bbox ghost the tracker has no way to verify,
+precisely the failure mode the *original* short-extinction-window default (`1.5s`)
+was chosen to avoid. The training-set films apparently didn't have a long enough
+faceless stretch after a confirmed identity to expose this; the held-out set did.
+
+Frames for all three films (`benny_joon`, `many_saints`, `downton_abbey` — one strong
+performer, two failure cases) are under `experiments/results/holdout/frames/`
+(not committed — pull per film with `scripts/artifacts/pull_artifacts.sh
+montage-frames <film-slug>`), each
+with a `manifest.json` listing the bucket (`best`/`fpi`/`fn`), timestamp, and
+predicted vs. ground-truth actors for every dumped frame. Frames are annotated with
+bounding boxes + name/confidence (green = identified, orange = unknown), matching
+[`src/nodes/debug_renderer_node.hpp`](https://REPOLINK/src/nodes/debug_renderer_node.hpp)'s
+colour convention. Generated by
+`scripts/optimizer/dump_error_frames.py --raw <replay.py --raw-out output>` (see
+Reproduce).
+
+`dump_error_frames.py --interval-sec 600` also supports a per-N-second sweep
+instead of the fixed best/fpi/fn buckets: one best (highest Jaccard) and one worst
+(lowest Jaccard) frame per 10-minute window across the whole film, e.g.
+`experiments/results/holdout/frames/many_saints_intervals/` (13 windows × 2 = 26
+frames for the ~2h Many Saints runtime) — a way to sample "how are we doing" evenly
+across a film's runtime rather than only at its most extreme seconds.
+
+## Caveats / what this is not
+
+- **r50's 4 combos used the old, narrower search bounds** and aren't fully
+  comparable to the other 12 on `anneal_sec`/`extinction_sec`.
+- **The applied defaults use `full_exp`, not the higher-scoring `restricted_exp`**,
+  because cast-restriction isn't a real runtime feature yet (see above). The
+  78.3% F1 number is not what the shipped defaults will produce — 75.3% is.
+- **`full_exp` is the best full-gallery combo, but not the safest.** Per the
+  isolated-effects analysis above, `expand_gallery=true` only cleanly pays off
+  when paired with cast-restriction; on the full gallery it's flat on F1 while
+  ~4x-ing misIDs (209→864, averaged across models). `full_noexp` scores lower
+  (72.4% vs 75.3% for LVFace) but with **zero** true misIDs and higher precision
+  (94.2% vs 89.7%). Kept `full_exp` as shipped since it's the highest-F1 option
+  available without cast-restriction, but this is a real F1-vs-safety trade, not
+  a strictly-better choice — worth revisiting if misID rate matters more than
+  the last few points of F1 for a given deployment.
+- **Switching the default model is an operational change, not just a config tweak**:
+  any existing gallery built from r50 embeddings is incompatible with LVFace
+  embeddings and needs rebuilding.
+
+## Reproduce
+
+```bash
+# 4-film matrix, all 4 models × 2 modes × 2 expansion settings
+bash experiments/run_rep4_subprocess.sh
+
+# single combo
+SAE_EXPAND=1 REPLAY_WORKERS=4 DE_WORKERS=2 python3 scripts/optimizer/optimize.py \
+    --manifest experiments/manifests/rep4_LVFace-B_Glint360K_full.json \
+    --gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5 \
+    --params prob_threshold:0.5:0.999 anneal_sec:1:60 extinction_sec:1:60 \
+    --popsize 10 --maxiter 15 --trajectory traj.jsonl --out best.json
+
+# replay the shipped config against a held-out film — --raw-out is needed to draw
+# bboxes later (the merged pred.json has no per-frame bbox, only actor windows)
+python3 scripts/optimizer/replay.py \
+    --dump experiments/dumps/LVFace-B_Glint360K/dump_<slug>.h5 \
+    --gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5 \
+    --out pred.json --raw-out raw.jsonl --prob-threshold 0.754 --anneal-sec 35.54 \
+    --extinction-sec 57.43 --expand-gallery
+
+# regenerate the report's charts (16-combo ranking, DE landscape, held-out
+# per-film F1, Downton ghost timeline) from the artifacts under experiments/
+python3 scripts/docs/experiment_charts.py --out-dir docs/assets/images
+
+# dump example frames (best-agreement / FPI / FN) for visual inspection, annotated
+# with bounding boxes + names (--raw is optional; omit for unannotated frames)
+python3 scripts/optimizer/dump_error_frames.py \
+    --pred pred.json --raw raw.jsonl --xray experiments/xray/.../<xray_dir> \
+    --movie "<path to source video>" \
+    --gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5 \
+    --out-dir experiments/results/holdout/frames/<name> --n-per-bucket 4
+
+# or: one best + one worst frame per 10-minute window across the whole film
+python3 scripts/optimizer/dump_error_frames.py \
+    --pred pred.json --raw raw.jsonl --xray experiments/xray/.../<xray_dir> \
+    --movie "<path to source video>" \
+    --gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5 \
+    --out-dir experiments/results/holdout/frames/<name>_intervals --interval-sec 600
+```
+
+See also: [the prior optimizer round](optimizer-experiments.md) (superseded
+metric) and the session log
+[`experiments/SESSION_STATE.md`](https://REPOLINK/experiments/SESSION_STATE.md).