dtourolle/scene-actor-extraction
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: full data-grounded rewrite of the performance report

Browse Source 
Replaces narrative claims with verified numbers across all report pages:

- Cross-model held-out validation (LVFace/mbf/r18, all 5 held-out
  films): LVFace wins every film outright, not just "consistent with"
  the training-set pick. r50 dropped from the detailed comparison
  (gallery has ~30% fewer reference images per actor than the other
  three models on identical source photos).
- Per-film training breakdown: LVFace does not win every training
  film (mbf beats it on Lord of War); the 75.3% macro figure hides a
  10.7pp spread.
- Gallery coverage computed per film (20.3%-78.6%) instead of one
  flat 67%-missing average.
- Found and fixed a real scoring bug in optimize.py: a candidate
  whose hardest film's replay timed out was averaged over survivors
  instead of penalized, silently rewarding partial coverage. Affected
  3 of 16 training combos; corrected throughout, and optimize.py now
  scores an incomplete evaluation f1=0.0 instead of averaging over
  whichever films happened to finish.
- Every FPI frame in the deep dive now comes from the proper montage
  renderer (Onscreen/Offscreen panel, ghosts never drawn as boxes),
  never the bare-box debug overlay used earlier.
- Every distinct out-of-cast name across all 9 films gets its own
  frame at its first appearance (9 names, 4 films), not a
  single-example spot check: 2 ground-truth gaps, 1 photograph
  misread as a person, 6 genuine lookalike confusions.
- New methodology.md: the scene-level-vs-per-second scoring mismatch
  that the rest of the report assumes, written out once.
- Cut the deadlock/gdb debugging narrative from the experiment log;
  kept the one fact that matters (KPN's node/network split lets the
  expensive GPU stage run once and the cheap stage replay against
  cached embeddings).
- Plain declarative style throughout, no em dashes, no blog voice.
This commit is contained in:
dtourolle
2026-07-21 08:55:57 +02:00
parent 4b5557974b
commit 0bd2747069
18 changed files with 1824 additions and 890 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/model-bakeoff.md
+271 -387
View File
@@ -1,425 +1,322 @@
# Model bake-off + threshold re-tune — experiment log (2026-07-18/19)
# Full experiment log
Follow-on to [the prior optimizer round](optimizer-experiments.md), which used
an older, since-superseded scene-union metric. This round uses the **per-second** metric
([`scripts/optimizer/second_score.py`](https://REPOLINK/scripts/optimizer/second_score.py))
and answers three questions in one 16-run matrix: which embedding model is best, does cast-restriction cut misIDs, and does
per-film gallery expansion help.
This page reports how the pipeline performs across three questions: which
embedding model is best, whether restricting the gallery to a film's
credited cast helps, and whether promoting confidently identified poses into
a per-film gallery annex helps. It also documents the replay architecture
that made testing all three questions in one pass practical, and every
caveat needed to trust the numbers.
(This was the fourth optimizer campaign against the X-Ray corpus, so its on-disk
artifacts carry an internal `rep4_` prefix — `experiments/results/rep4_best_*.json`,
`experiments/trajectories/rep4_*.jsonl`, and the manifests referenced below. The
earlier campaigns used the superseded scene-union metric and were discarded.)
Read [How we score against X-Ray](methodology.md) first for what F1,
precision, recall, and misID mean in this report. All numbers below use the
per-second metric
([`scripts/optimizer/second_score.py`](https://REPOLINK/scripts/optimizer/second_score.py)).
## Why this experiment, and what it actually delivered
r50 (ArcFace w600k-R50) is excluded from the detailed comparison below. Its
gallery was built with roughly 30% fewer reference images per actor than the
other three models on the identical source photos (10808 vs 15055 total
embeddings across the same 2418 actors), which confounds any direct
comparison of its scores against the others. It remains in the
[calibration curve comparison](best-model.md#first-signal-calibration-curves),
which does not depend on the training benchmark.
Four goals going in, and an honest read on each after held-out validation (see
below):
## Why replay makes this affordable
1. **Find the best default parameters to ship.** Partially delivered. The DE optimum
generalizes *unevenly* — strong on 3 of 5 held-out films, badly broken on 2 (one
with a 974-count misID blowup). The tuned values are shipped anyway (see
Caveats) because they still beat the old defaults on average, but this is not a
settled, film-agnostic optimum.
2. **Find the best default model.** Delivered with more confidence. LVFace beat
r50/r18/mbf across all 4 training combos, and nothing in held-out validation
contradicts the model choice specifically — the held-out failures trace to
`extinction_sec`/threshold interactions and gallery coverage, not the embedder.
3. **Provide insight into how the application works.** The strongest, most durable
output. Found and fixed a real teardown deadlock bug (100% reproducible, not the
assumed rare GPU flake), established a real concurrency ceiling (8 parallel
replays, not more), and found a real parameter interaction (a strict
`prob_threshold` "earns" a longer extinction window before it starts hurting).
4. **Demonstrate limitations.** Delivered, and reinforced hard by held-out
validation — see the "Held-out validation" section below for concrete examples,
including a screenshot of the matcher naming 15 actors, none correctly, on a
completely blank title card.
Decoding video and running face detection, alignment, and embedding is the
expensive part of this pipeline. Everything downstream of that (tracking,
identity matching, scene aggregation) is cheap. KPN++'s node/network
structure means those two stages are separate components connected by
typed channels, so the expensive stage can run once per film, cache its
output, and the cheap stage can be re-run against that cache as many times
as needed with different Config values.
## TL;DR — what changed in `src/config.hpp`
`scene_analyze --dump-embeddings out.h5` runs the expensive half once per
film and writes per-frame face detections and embeddings to HDF5
([`scripts/optimizer/SCHEMA.md`](https://REPOLINK/scripts/optimizer/SCHEMA.md)).
[`scripts/optimizer/replay.py`](https://REPOLINK/scripts/optimizer/replay.py)
then re-assembles the real C++ `face_tracker`, `identity_matcher`, and
`scene_tracker` nodes into a Python-driven KPN network and replays a
film's cached embeddings through them, varying `prob_threshold`,
`anneal_sec`, `extinction_sec`, and `expand_gallery` freely. No GPU
inference and no video decode happen during a replay; each one completes
in seconds. This is what makes a 512-evaluation differential-evolution
search per model, per gallery mode, per expansion setting, tractable, and
what made the full held-out validation across three models in this report
possible in one session rather than requiring three full re-encodes of the
benchmark set.
| knob | old default | new default | why |
| ---- | ----------- | ----------- | --- |
| `arcface_model` | `arcface_w600k_r50.onnx` | **`LVFace-B_Glint360K.onnx`** | Best F1 in the full-gallery bake-off (75.3% vs r50's 68.5%). LVFace was worth its size. |
| `prob_threshold` | 0.76 | **0.754** | Re-tuned for LVFace + per-second metric. |
| `extinction_sec` | 1.5 | **57.4** | Reverses the earlier "short is better" finding — see below. |
| `anneal_sec` | 10.0 | **35.5** | Same reversal; previously thought insensitive. |
| `expand_gallery` | false | **true** | Helped recall on the full (unrestricted) gallery for the winning model — opposite of the earlier assumption. |
`optimize.py` runs `differential_evolution` over this replay function as its
objective, with DE-level parallelism (multiple candidate configs evaluated
concurrently, each spawning its own replay subprocesses) on top of it. The
practical ceiling on this machine's GPU was 8 concurrent replay processes;
9 silently degraded every score to 0.0% (well-formed output, wrong numbers,
not a crash), so `optimize.py` was run at `REPLAY_WORKERS=4 DE_WORKERS=2`.
These are the **`LVFace-B_Glint360K_full_exp`** winning values, applied to
[`src/config.hpp`](https://REPOLINK/src/config.hpp) — the best result that
uses only features already live in the running app (full gallery, no cast
restriction; see below for why restricted mode isn't applied even though it scored
higher).
## Search space
## Why re-run at all
`popsize=10, maxiter=15` per combo (3 parameters, up to 512 evaluations,
usually stopping earlier on DE's convergence tolerance).
`anneal_sec`/`extinction_sec` bounds were widened from 1-30/1-15 to 1-60/1-60
partway through the sweep. r50's 4 combos finished before the widening and
used the old, narrower bounds; this is one more reason r50 is excluded from
direct comparison here.
[The prior round](optimizer-experiments.md)'s scene-union metric hid out-of-cast false positives
behind a gallery∩cast recall mask, and the earlier 9-film benchmark was scene-level
(union over a whole X-Ray scene), not a fair per-timepoint comparison. SESSION_STATE
flagged the old scene-metric bake-off numbers (R50≈LVFace≈MBF ~85%) as superseded.
This round uses `second_score.py`: uniform per-second sampling, GT = X-Ray scene's
cast at time *t*, pred = actors whose presence window covers *t*, FPI weighted 10×
when the named actor isn't in the film's cast at all (true misID) vs. an in-cast
timing slip. FN only counts gallery-known cast (fair recall — 67% of X-Ray cast have
no reference embedding; see
[the prior round's gallery-coverage-gap analysis](optimizer-experiments.md#the-gallery-coverage-gap)).
## Training films and held-out films
## The deadlock that was blocking all of this
9 films have dumped embeddings across all 4 models. 4 were used for
optimization:
Every replay in this line of work goes through
[`scripts/optimizer/replay.py`](https://REPOLINK/scripts/optimizer/replay.py),
which runs the real C++ tracker/matcher/scene_tracker nodes inside a
Python-assembled KPN network. Before this session, every subprocess replay **timed out at 45s, 100% of
the time** — not the documented ~20-30% ROCm rocBLAS-GEMM driver flake, but a plain
logic bug: `replay.py`'s CLI called `replay(net, ..., stop=False)` to *skip*
`net.stop()` (trying to dodge the GEMM deadlock), planning to `os._exit(0)`
immediately after. But:
- Café Society (62-cast)
- Lord of War (64-cast)
- Scarface (67-cast)
- Sound of Metal (14-cast)
- `PyNode::stop()` ([`include/kpn/python/bindings.hpp`](https://KPNLINK/include/kpn/python/bindings.hpp)
in the KPN++ submodule) is the *only*
code that sets `stop_flag_ = true` before joining the node's worker thread.
- The source node's `run_loop()` has `while (!stop_flag_)` as its only exit
condition (it has no input channels, so it never sees a channel-closed signal
either).
- Skipping `stop()` meant `stop_flag_` never became true. When `replay()` returned,
its local `net` went out of scope immediately, running `~PyNetwork` → `~PyNode` →
`thread_.join()` **synchronously inside `replay()`'s own call frame** — before
`main()` ever got control back to run `os._exit(0)`.
5 were held out, never seen by any optimizer run:
Root-caused via `gdb -p <pid> -batch -ex "thread apply all bt"` on a hung process:
the main thread was stuck in `~PyNode`'s `jthread::join()`; the worker thread was in
an ordinary `time.sleep()` inside the Python source callback, waiting for a stop
signal that was never sent. The two HSA `kfd_wait_on_events` threads visible in the
same trace are normal ROCm runtime housekeeping, not evidence of a wedged GPU kernel.
- Benny & Joon
- Downton Abbey: A New Era
- Lovelace
- The Many Saints of Newark
- Valerian and the City of a Thousand Planets
**Fix:** `replay.py` now calls `replay(..., stop=True)` (the removed `stop=False` +
`os._exit` workaround was actively harmful). Verified 3/3 clean runs at ~8s each
(down from a guaranteed 45s timeout), and a full DE sweep producing real, sensible
F1/precision/recall instead of flat 0.0%.
## Gallery coverage per film
## Concurrency tuning
The gallery has reference embeddings for 2418 actors, but coverage of any
given film's credited cast varies widely. This was previously reported as
one flat number (67% of X-Ray cast lacking a reference embedding, averaged
across the whole benchmark); the per-film breakdown is:
With the deadlock fixed,
[`scripts/optimizer/optimize.py`](https://REPOLINK/scripts/optimizer/optimize.py)
was extended with DE-level parallelism —
`differential_evolution(..., workers=ThreadPoolExecutor.map)` — so multiple
population candidates evaluate concurrently, each spawning its own per-film replay
subprocesses (`REPLAY_WORKERS`). Total concurrent GPU replay processes ≈
`DE_WORKERS × REPLAY_WORKERS`.
| film | cast credited | in gallery | coverage |
|---|---|---|---|
| Lord of War | 64 | 13 | 20.3% |
| Scarface | 67 | 15 | 22.4% |
| The Many Saints of Newark | 48 | 13 | 27.1% |
| Café Society | 62 | 17 | 27.4% |
| Lovelace | 42 | 15 | 35.7% |
| Valerian and the City of a Thousand Planets | 36 | 13 | 36.1% |
| Benny & Joon | 23 | 12 | 52.2% |
| Downton Abbey: A New Era | 36 | 22 | 61.1% |
| Sound of Metal | 14 | 11 | 78.6% |
| concurrent replays | result |
| --- | --- |
| 3 (`REPLAY_WORKERS=3`, no DE parallelism) | baseline, GPU underutilised |
| 6 (`DE_WORKERS=2 × REPLAY_WORKERS=3`) | clean, real scores, ~1 isolated timeout per run |
| 8 (`DE_WORKERS=2 × REPLAY_WORKERS=4`, 4-film manifest) | clean, real scores |
| 9 (`DE_WORKERS=3 × REPLAY_WORKERS=3`) | **broken** — every replay blew past the 45s timeout, all scores silently degraded to 0.0% |
Two training films (Lord of War, Scarface) have the worst coverage in the
set, 20-22%. Their training-set F1 numbers below are partly capped by
missing references, not purely by model quality. Downton Abbey has 61%
coverage, the second-best in the benchmark, yet the worst held-out recall
of any film (39.4%, LVFace). Its recall problem is not primarily a coverage
problem; it is the extinction-bridging failure documented in the
[LVFace deep dive](lvface-deep-dive.md#mechanism-1-extinction-bridging).
Reproduce with `scripts/docs/gallery_coverage_per_film.py`.
9 concurrent replays looks like valid output (well-formed JSON, a real number) while
actually being garbage — a dangerous failure mode, not a crash. **8 concurrent is the
practical ceiling** on this GPU (gfx1100) for this workload. The matrix ran at
`REPLAY_WORKERS=4 DE_WORKERS=2`.
## Training results, 3 models × 2 gallery modes × 2 expansion settings
## Training films and validation set
Ranked by F1. misid = FPI_misid, the count of true wrong-actor
identifications (naming someone not in the film's cast at all), distinct
from FPI, which also includes in-cast timing slips.
9 films total have dumped embeddings across all 4 models. 4 were used for
optimization, leaving 5 held out for validation:
- **Lord of War** (64-cast, "clean")
- **Scarface** (67-cast, "ensemble/lookalike")
- **Sound of Metal** (14-cast, "high gallery-coverage")
- **Café Society** (62-cast, added this round — similar ensemble size to Scarface but
different genre/lighting; picked to add diversity, not genre-overlap, over
Downton Abbey or The Many Saints of Newark)
Held out: Benny & Joon, Downton Abbey: A New Era, Lovelace, The Many Saints of
Newark, Valerian and the City of a Thousand Planets. The training-set numbers below are
training-set fit — see "Held-out validation" further down for the real
generalization test.
## Search space and DE settings
`popsize=10, maxiter=15` (3 params → ≤480 evals/combo ceiling; DE's `tol` convergence
usually stops earlier). `anneal_sec`/`extinction_sec` bounds were **widened from
1–30/1–15 to 1–60/1–60 mid-run** (see below) — the 4 `arcface_w600k_r50` combos
finished before the widening and still use the old, narrower bounds, so they are
**not directly comparable** to the other 12 on those two params. Re-running r50 with
the wider bounds was deferred (diminishing-returns judgment call, not yet done).
## Results — all 16 combos (4 models × {full, restricted} × {expand, noexp})
Ranked by F1. `misid` = FPI_misid, count of true wrong-actor identifications (an
actor named who isn't in the film's cast at all) — distinct from `FPI`, which
includes in-cast timing slips.
Each combo's row is its best **full-coverage** evaluation: the highest-F1 DE
evaluation in which all 4 training films replayed without a timeout (see
[Dropped-film scoring](#a-scoring-bug-worth-recording-dropped-film-evaluations)
below for why this qualifier is load-bearing and not the same as `argmax F1`
over the raw sweep).
| combo | F1 | P | R | TPI | FPI | misid | FN |
|---|---|---|---|---|---|---|---|
| LVFace-B_Glint360K_restricted_exp | **78.3%** | 91.0% | 68.9% | 42830 | 3782 | 60 | 19492 |
| LVFace-B_Glint360K_restricted_exp | 78.3% | 91.0% | 68.9% | 42830 | 3782 | 60 | 19492 |
| LVFace-B_Glint360K_restricted_noexp | 76.7% | 91.5% | 66.2% | 41149 | 3400 | 59 | 21173 |
| arcface_w600k_mbf_restricted_exp | 76.5% | 90.7% | 66.3% | 41270 | 4431 | **0** | 21052 |
| arcface_w600k_mbf_restricted_exp | 76.2% | 90.0% | 66.2% | 64328 | 7480 | 0 | 33234 |
| arcface_r18_restricted_exp | 75.5% | 87.6% | 66.5% | 41399 | 5666 | 60 | 20923 |
| **LVFace-B_Glint360K_full_exp** | **75.3%** | 89.7% | 65.4% | 47757 | 3407 | 232 | 26966 |
| LVFace-B_Glint360K_full_exp | 75.3% | 89.7% | 65.4% | 47757 | 3407 | 232 | 26966 |
| arcface_w600k_mbf_restricted_noexp | 75.0% | 91.1% | 63.9% | 39752 | 3465 | 60 | 22570 |
| arcface_w600k_mbf_full_noexp | 74.2% | 87.4% | 64.4% | 12645 | 1312 | 57 | 6985 |
| arcface_r18_restricted_noexp | 73.5% | 91.3% | 61.7% | 38299 | 3220 | 60 | 24023 |
| LVFace-B_Glint360K_full_noexp | 72.4% | 94.2% | 58.9% | 27077 | 1725 | **0** | 19506 |
| LVFace-B_Glint360K_full_noexp | 72.3% | 88.3% | 61.8% | 40363 | 3503 | 244 | 25850 |
| arcface_w600k_mbf_full_exp | 72.0% | 87.7% | 61.4% | 39875 | 3729 | 240 | 26338 |
| arcface_w600k_r50_full_noexp † | 71.6% | 96.7% | 56.9% | 22471 | 361 | 45 | 17012 |
| arcface_w600k_r50_restricted_exp † | 71.1% | 96.5% | 56.4% | 34954 | 1119 | 15 | 27368 |
| arcface_w600k_r50_restricted_noexp † | 69.2% | 97.9% | 53.6% | 21146 | 327 | 15 | 18337 |
| arcface_w600k_mbf_full_noexp | 71.0% | 93.2% | 57.9% | 41699 | 2472 | 56 | 33024 |
| arcface_r18_full_exp | 69.1% | 87.6% | 57.7% | 37342 | 3119 | 242 | 28871 |
| arcface_w600k_r50_full_exp † | 68.5% | 94.0% | 54.1% | 34982 | 903 | 150 | 31231 |
| arcface_r18_full_noexp | 66.6% | 91.3% | 53.1% | 34314 | 2362 | 107 | 31899 |
† old, narrower anneal/extinction bounds (see above) — not directly comparable to
the other 12 on those two params.
![All combos ranked by training-set F1](assets/images/rep4_matrix_f1.png)
The same 16 results as a picture — the two headline effects are visible without
reading a single row: filled (restricted) dots stack the top of the ranking for
every model color, and yellow (LVFace) leads within both scopes:
The two clearest patterns: every model's best-scoring combo uses the
restricted gallery, and LVFace leads within both gallery modes. `full_exp`
(the shipped combination) is the best-scoring option that uses only
features the running application currently supports; restriction is not
wired into the application yet (see
[Whole vs. cast-restricted gallery](gallery-scope.md)).
![All 16 bake-off combos ranked by training-set F1](assets/images/rep4_matrix_f1.png)
### A scoring bug worth recording: dropped-film evaluations
## Calibration curves — discriminative power, independent of the threshold
The numbers above are corrected ones. The raw `rep4_best_*.json` files, and an
earlier version of this table, reported a different `arcface_w600k_mbf_full_noexp`
row: **74.2% F1 at TPI 12645**, a third the TPI of every sibling combo. That was
not a better config; it was an artifact of how the optimizer aggregates.
Each model's gallery carries a fitted Platt sigmoid `P(match | sim) = σ(a·sim + b)`
(embedded directly in the gallery HDF5, see
[`src/gallery/gallery_calibration.hpp`](https://REPOLINK/src/gallery/gallery_calibration.hpp)).
Plotting all four side by side shows discriminative power directly, independent of
whatever `prob_threshold` a particular run happened to use:
`optimize.py` builds each candidate's score from only the films whose replay
subprocess returned (`per_film = [m for m in ex.map(_one, films) if m is not
None]`), then **averages** F1/precision/recall and **sums** TPI/FPI/misID over
just those survivors. When a film's replay times out (the sweep ran near the
8-process concurrency ceiling, so this happened intermittently), that film
silently drops from both. A candidate whose hardest film timed out is therefore
scored on an easier subset, and differential evolution, maximizing that score,
will happily converge onto exactly such a candidate. For `mbf_full_noexp` the
reported winner was one of 7 evaluations (out of 512) whose TPI had collapsed to
a partial-film subset; its median-coverage evaluations sit around 51686 TPI.
![Calibrated P(match|similarity) for all four models](assets/images/calibration_curves.png)
The fix here was to re-derive each combo's best row from its DE trajectory
(`experiments/trajectories/rep4_*.jsonl`), keeping only evaluations within 30% of
that combo's median TPI (full 4-film coverage) before taking the best F1. This
needs no re-running, the honest best configuration was already in the sweep,
just not the one `argmax F1` selected. Three combos moved: `mbf_full_noexp`
74.2% → **71.0%**, `LVFace_full_noexp` 72.4% → **72.3%** (and its misID, 0 → 244,
was itself a dropped-film artifact), `mbf_restricted_exp` 76.5% → **76.2%**. The
shipped LVFace `full_exp` winner was unaffected, its reported evaluation already
had full coverage (TPI 47757 ≈ median). `experiment_charts.py` applies the same
`clean_best` filter, so every figure on this page matches the corrected table.
The underlying `optimize.py` aggregation is also being fixed so a dropped-film
evaluation can never be selected as a winner again.
LVFace-B has both the steepest curve (`a=17.7`, vs. 15.3–16.2 for the ArcFace
variants) and the lowest P=0.5 decision boundary (similarity 0.23 vs. 0.27–0.31) —
it separates same-actor from different-actor pairs more confidently at a lower
similarity, consistent with it winning the full-gallery F1 comparison below.
Generated by
[`scripts/docs/calibration_chart.py`](https://REPOLINK/scripts/docs/calibration_chart.py)
(requires each gallery to have been calibrated at least once — run any replay
against it first).
### Per-film training breakdown
## Two effects in isolation: gallery scope, and pose expansion
The 75.3% LVFace training figure is a macro average across 4 films, not a
uniform result:
The matrix crosses two independent variables — averaging across all 4 models
isolates each one from model choice:
**Gallery scope (whole 2418-actor gallery vs. restricted to the film's credited
cast)** — averaged over both expansion settings and all 4 models:
| scope | F1 | P | R | total misID (16 evals→8 each) |
| film | LVFace F1 | mbf F1 | r18 F1 | best model |
|---|---|---|---|---|
| full | 71.2% | 91.1% | 59.0% | 1073 |
| **restricted** | **74.5%** | 92.2% | **62.9%** | **329** |
| Café Society | 68.1% | 62.2% | 60.1% | LVFace |
| Lord of War | 75.6% | 77.2% | 75.6% | mbf |
| Scarface | 71.5% | 68.6% | 64.1% | LVFace |
| Sound of Metal | 78.8% | 76.5% | 71.6% | LVFace |
Restriction wins outright on every axis — not a precision/recall trade, a clean
win: **+3.3pp F1, +3.9pp recall, and less than a third the total misIDs.** Fewer
LVFace does not win every training film. mbf scores higher on Lord of War
(77.2% vs 75.6%). LVFace's own training-film range is 68.1% to 78.8%, a
10.7pp spread, smaller than the 37pp spread seen on held-out films but real.
Reproduce with `scripts/docs/run_holdout_all_models.py --films training`.
## Held-out validation, all 3 models
The training matrix above is training-set fit. Each model's own tuned
`full_exp` config was replayed against the 5 held-out films, scored the
same way:
| film | LVFace F1 | mbf F1 | r18 F1 |
|---|---|---|---|
| Benny & Joon | 83.0% | 78.5% | 77.1% |
| Lovelace | 77.5% | 73.7% | 72.2% |
| Valerian and the City of a Thousand Planets | 74.1% | 70.2% | 71.0% |
| Downton Abbey: A New Era | 56.2% | 55.0% | 53.0% |
| The Many Saints of Newark | 46.3% | 44.5% | 42.1% |
| **macro average** | **67.4%** | **64.4%** | **63.1%** |
LVFace scores highest on every one of the 5 held-out films; the ranking
never flips. Total misIDs across the 5 films: LVFace 1032, mbf 2197, r18
1224. LVFace has less than half mbf's misID count while also scoring
higher on every film. This directly confirms the model choice out of
sample; it is not inferred from the training numbers alone. See the
[LVFace deep dive](lvface-deep-dive.md) for frame-level detail on where and
why LVFace still fails on the two worst films. Reproduce with
`scripts/docs/run_holdout_all_models.py`.
## Two effects in isolation: gallery scope and pose expansion
Averaging across the 3 compared models (r50 excluded) isolates each variable
from model choice.
**Gallery scope**, averaged over both expansion settings and all 3 models
(6 evaluations per row):
| scope | F1 | P | R | total misID |
|---|---|---|---|---|
| full | 71.1% | 89.6% | 59.6% | 1121 |
| restricted | 75.9% | 90.4% | 65.6% | 299 |
Restriction improves every metric at once. This is not a precision/recall
trade: +4.8pp F1, +6.0pp recall, and roughly a quarter the misIDs. Fewer
candidates in the matcher's search space means fewer opportunities for a
look-alike false match, and (per the recall gain) doesn't cost real detections.
This is the single cleanest signal in the whole matrix — stronger than the model
choice itself — which is exactly why cast-restriction becoming a real runtime
feature (not just an optimizer trick) is the top item in Caveats below.
lookalike false match, and the recall gain shows this does not cost real
detections. Restriction is currently an offline optimizer technique, not a
runtime feature of the application; see
[Whole vs. cast-restricted gallery](gallery-scope.md) for what building it
into the application would require.
**Pose expansion (promoting a confidently-identified track's novel-pose views into
a per-film gallery annex — [`src/gallery/track_gallery.hpp`](https://REPOLINK/src/gallery/track_gallery.hpp))**
is smaller and interacts with
scope rather than acting independently:
**Pose expansion** (promoting a confidently identified track's novel-pose
views into a per-film gallery annex,
[`src/gallery/track_gallery.hpp`](https://REPOLINK/src/gallery/track_gallery.hpp)):
| scope | expansion | F1 | R | misID |
|---|---|---|---|---|
| full | off | 71.2% | 58.3% | 209 |
| full | **on** | 71.2% | 59.7% | **864** |
| restricted | off | 73.6% | 61.3% | 194 |
| restricted | **on** | **75.4%** | **64.5%** | 135 |
| full | off | 70.0% | 57.6% | 407 |
| full | on | 72.1% | 61.5% | 714 |
| restricted | off | 75.1% | 63.9% | 179 |
| restricted | on | 76.7% | 67.2% | 120 |
In **restricted** mode, expansion is a clean win (+1.8pp F1, +3.2pp recall, misID
actually *drops*) — the annex only ever competes against the film's own ~15-actor
cast, so a "confidently identified, new pose" view is unlikely to be mistaken for
someone else. In **full** mode, expansion buys essentially nothing on F1 (71.2% →
71.2%, recall +1.4pp) while **quadrupling misIDs** (209 → 864): a novel-pose view
promoted into the annex now competes against the whole 2418-actor gallery, so a
"confident" identity is confident against the wrong universe of candidates — the
expansion mechanism is "learning" a pose correctly, but the enlarged evidence pool
makes it easier for that learned pose to look like a plausible match for a
different actor. **Practical takeaway: gallery expansion should be paired with
cast restriction, not used on the full gallery** — the version currently shipped
as default (`full_exp`, see TL;DR) sits in the worse of these four cells for this
specific knob, even though it's the best available combo without cast-restriction
support in the app yet (see Caveats).
In restricted mode, expansion is a clean win: +1.6pp F1, +3.3pp recall,
misID drops. The annex only competes against the film's own roughly 15-actor
cast, so a new pose of a known actor is unlikely to be confused with someone
else. In full mode, expansion buys +2.1pp F1 and +3.9pp recall but at a real
cost: misID rises from 407 to 714 as the same new-pose view now competes
against the full 2418-actor gallery, where a confidently learned pose is more
likely to match the wrong person. On the full gallery it is a recall-vs-misID
trade, not a free gain. This training-set effect
did not reproduce on held-out data; see
[Does pose expansion help?](pose-expansion.md) for the full held-out test
and the two methodology bugs caught while checking it.
## What the data says
## Calibration curves
- **LVFace was worth its size.** It wins full-gallery mode outright (75.3% vs r50's
68.5%, r18's 69.1%, mbf's 72.0%) with the highest recall of any full-mode combo —
the earlier scene-union-metric conclusion ("not worth it") doesn't survive the
better metric.
- **Cast-restriction is a consistent, broad win.** Every model's best combo is
`restricted`. It isn't just precision-safe: `arcface_w600k_mbf_restricted_exp` and
`LVFace-B_Glint360K_full_noexp` both hit **misid=0** — zero true wrong-actor
identifications. But restriction is an **offline optimizer technique, not a live
app feature** — it pre-filters each film's gallery to its Jellyfin-credited cast
before the matcher ever runs; there's no runtime "restrict to this film's cast"
switch in the app today. Implementing it for real is future work, tracked
separately from this defaults update.
- **Gallery expansion (`expand_gallery`) is mode-dependent.** It helps on
`restricted` galleries (smaller, so novel-pose promotion adds real signal) and on
LVFace's full gallery, but **hurts** r50 and mbf in full mode (compare
`arcface_w600k_r50_full_exp` 68.5% vs `full_noexp` 71.6%). Don't assume it's a free
win — model- and mode-dependent.
- **arcface_r18 (smallest/cheapest) is last across all 4 modes** — model capacity
matters here, this isn't just parameter-count padding.
- **`anneal_sec`/`extinction_sec` kept pinning at the search ceiling.** With the
original 1–30/1–15 bounds, 3 of 4 r50 combos landed at ~93-98% of the upper bound.
Widened to 1–60/1–60 mid-run (after the r50 combos had already finished) — every
subsequent combo's best config landed at ~90%+ of the *new* ceiling too (e.g. the
LVFace winner: `ann=59.2, ext=59.2`, both ~99% of 60). The likely mechanism: a
strict `prob_threshold` "earns" a long extinction/anneal window — once false
matches are rare, a long window just bridges real presence gaps (occlusion, turned
face) instead of smearing false positives into later scenes, which is what made
short windows look better under the old, laxer thresholds. **Open question, not
resolved**: does this keep climbing past 60s, or does it actually plateau there?
Decided not to chase further this round (diminishing-returns judgment call) — flag
for a future sweep if it matters.
Each gallery carries a fitted Platt sigmoid `P(match | sim) = σ(a·sim + b)`,
stored directly in the gallery HDF5
([`src/gallery/gallery_calibration.hpp`](https://REPOLINK/src/gallery/gallery_calibration.hpp)).
This measures discriminative power independent of whatever
`prob_threshold` a given run used:
The ceiling-pinning is visible in the raw search itself. Every one of the 512
DE evaluations for the winning combo, plotted over the
`prob_threshold` × `extinction_sec` plane:
![Calibrated P(match|similarity) for all four models](assets/images/calibration_curves.png)
LVFace has the steepest curve (`a=17.7` vs 15.3-16.2 for the ArcFace
variants) and the lowest P=0.5 decision boundary (similarity 0.23 vs
0.27-0.31), separating same-actor from different-actor pairs more
confidently at a lower similarity than any ArcFace variant tested,
including r50. Generated by
[`scripts/docs/calibration_chart.py`](https://REPOLINK/scripts/docs/calibration_chart.py).
## Extinction and anneal window search
Every one of the 512 DE evaluations for the winning LVFace `full_exp`
combo, plotted over the `prob_threshold` × `extinction_sec` plane:
![DE search landscape: 512 evaluations over prob_threshold × extinction_sec](assets/images/de_search_landscape.png)
The dark band hugging the top edge *is* the finding: nearly everything scoring
well sits at `extinction_sec` ≥ 50, across a wide range of thresholds, and the
population converged into a dense cloud around the optimum (threshold ~0.70–0.80,
extinction pinned at the 60s bound). Short extinction windows (bottom half) are
uniformly pale — under a strict threshold there is simply no good configuration
down there. Generated by
[`scripts/docs/experiment_charts.py`](https://REPOLINK/scripts/docs/experiment_charts.py)
from the DE trajectories (`experiments/trajectories/*.jsonl`, part of the
`experiment-data` artifact package).
Nearly everything scoring well sits at `extinction_sec` above 50, across a
wide range of thresholds. Short extinction windows are uniformly weaker:
under a strict threshold, there is no good configuration in that region of
the search space. The optimizer converged with `anneal_sec=59.2,
extinction_sec=59.2`, about 99% of the widened 60s bound, which raises an
open question not resolved in this round: does performance keep improving
past 60s, or does it plateau there. Not chased further this pass.
## Held-out validation — the number that actually matters
## Caveats
The 16-combo matrix above is training-set fit. This is the real test: the shipped
config (`LVFace-B_Glint360K_full_exp` — `prob_threshold=0.754, anneal_sec=35.5,
extinction_sec=57.4, expand_gallery=true`) replayed against the **5 films never seen
by the optimizer** (Benny & Joon, Downton Abbey: A New Era, Lovelace, The Many
Saints of Newark, Valerian and the City of a Thousand Planets), scored the same way.
| film | F1 | P | R | agree | TPI | FPI | misid | FN |
|---|---|---|---|---|---|---|---|---|
| Benny & Joon | 83.0% | 89.1% | 77.7% | 72.4% | 15125 | 1846 | 0 | 4337 |
| Lovelace | 77.5% | 90.3% | 67.9% | 72.1% | 14990 | 1085 | 58 | 7085 |
| Valerian and the City of a Thousand Planets | 74.1% | 97.1% | 60.0% | 58.8% | 18663 | 548 | 0 | 12467 |
| Downton Abbey: A New Era | 56.2% | 97.8% | 39.4% | 40.6% | 52027 | 1173 | 0 | 80084 |
| **The Many Saints of Newark** | **46.3%** | **54.7%** | 40.1% | 37.0% | 15922 | 4394 | **974** | 23791 |
| **macro average (5 films)** | **67.4%** | 85.8% | 57.0% | 56.2% | 116727 | 9046 | 1032 | 127764 |
![Held-out per-film F1 vs. the training-set fit](assets/images/holdout_f1_by_film.png)
**67.4% held out vs. 75.3% on training** — an ~8pp drop, and a much more informative
number than the training-set F1 alone: a **37pp spread between best and worst film**
(83.0% vs 46.3%). The config does not generalize uniformly.
Two films are outright failure cases, and rendering bounding boxes + names on the
extracted frames (`replay.py --raw-out` +
[`dump_error_frames.py`](https://REPOLINK/scripts/optimizer/dump_error_frames.py)` --raw`, see
Reproduce) turned what looked like a same-scene misidentification into something
more precise and more damning:
- **The Many Saints of Newark** (mob-family drama, picked as a training-adjacent
genre test) has **974 true misIDs** — far more than any training combo saw at any
setting. The annotated frame below shows the same mechanism as Downton Abbey,
at smaller scale: **"Jon Bernthal 100%", "Joey Diaz 100%", and "Billy Magnussen
100%" are all frozen boxes over empty background — a blurred wall, hanging plates —
with no face in them at all.** Only one real face in frame has a box, and it
carries a *second*, colliding label ("Leslie Odom Jr." and "Michael Gandolfini"
both at high confidence on the same box) — likely two tracks whose frozen bboxes
happen to overlap.
![Frozen ghost boxes over background, The Many Saints of Newark](assets/images/many_saints_ghost_fpi.jpg)
*Frame `many_saints/fpi/fpi_t03543.jpg` from the `montage-frames` artifact
package (`scripts/artifacts/pull_artifacts.sh montage-frames
Many_Saints_of_Newark`).*
- **Downton Abbey: A New Era** (large ensemble, 36-cast) has high precision (97.8%)
but recall collapses to 39.4% (FN=80084, by far the largest of the 5). Its
starkest failure happens where there is nothing to see at all: the film's hard
cut into its closing credits, where **the matcher kept reporting 15 actors —
all wrong — for nearly a minute of faceless screen.**
Both are the same mechanism, and it can be *measured*, not just screenshotted.
Plotting the dump's own per-second `face_count` (detector output, independent
of the tracker) against the number of actors the tracker reports, through
Downton Abbey's cut to credits:
![Detector face_count vs. tracker-reported actors through the cut to credits](assets/images/downton_ghost_timeline.png)
From the cut onward the detector sees **zero faces** — yet the tracker holds a
perfectly flat plateau of 15 reported identities for 56 seconds, each with the
*exact same bbox, unchanged to the pixel* (verified for Hugh Bonneville:
`(1743.2, 0.0, 171.3, 317.8)` at every sampled second from 7222 through 7279+).
The staircase on the right edge is the extinction window finally expiring,
actor by actor. That plateau is `SceneTrackerFunc`'s
`active_[actor_idx].last_bbox`
([`src/nodes/scene_tracker_node.hpp`](https://REPOLINK/src/nodes/scene_tracker_node.hpp))
being re-emitted
unchanged — **the extinction state machine working exactly as coded**, not a
bug in the logic. The film cuts from a packed group shot straight into ~40+
seconds of blank titles/credits with zero faces, and `extinction_sec=57.4` is
comfortably long enough to bridge that entire gap without expiring, so the
tracker faithfully keeps reporting "last known position" for a cast that is no
longer on screen at all.
This reframes the "long extinction window wins" DE-search pattern (see above): it
isn't unambiguously good. It buys recall by bridging real gaps (occlusion, turned
face) in some films, but on others — specifically, hard cuts into long faceless
footage — it manufactures a frozen-bbox ghost the tracker has no way to verify,
precisely the failure mode the *original* short-extinction-window default (`1.5s`)
was chosen to avoid. The training-set films apparently didn't have a long enough
faceless stretch after a confirmed identity to expose this; the held-out set did.
Frames for all three films (`benny_joon`, `many_saints`, `downton_abbey` — one strong
performer, two failure cases) are under `experiments/results/holdout/frames/`
(not committed — pull per film with `scripts/artifacts/pull_artifacts.sh
montage-frames <film-slug>`), each
with a `manifest.json` listing the bucket (`best`/`fpi`/`fn`), timestamp, and
predicted vs. ground-truth actors for every dumped frame. Frames are annotated with
bounding boxes + name/confidence (green = identified, orange = unknown), matching
[`src/nodes/debug_renderer_node.hpp`](https://REPOLINK/src/nodes/debug_renderer_node.hpp)'s
colour convention. Generated by
`scripts/optimizer/dump_error_frames.py --raw <replay.py --raw-out output>` (see
Reproduce).
`dump_error_frames.py --interval-sec 600` also supports a per-N-second sweep
instead of the fixed best/fpi/fn buckets: one best (highest Jaccard) and one worst
(lowest Jaccard) frame per 10-minute window across the whole film, e.g.
`experiments/results/holdout/frames/many_saints_intervals/` (13 windows × 2 = 26
frames for the ~2h Many Saints runtime) — a way to sample "how are we doing" evenly
across a film's runtime rather than only at its most extreme seconds.
## Caveats / what this is not
- **r50's 4 combos used the old, narrower search bounds** and aren't fully
comparable to the other 12 on `anneal_sec`/`extinction_sec`.
- **The applied defaults use `full_exp`, not the higher-scoring `restricted_exp`**,
because cast-restriction isn't a real runtime feature yet (see above). The
78.3% F1 number is not what the shipped defaults will produce — 75.3% is.
- **`full_exp` is the best full-gallery combo, but not the safest.** Per the
isolated-effects analysis above, `expand_gallery=true` only cleanly pays off
when paired with cast-restriction; on the full gallery it's flat on F1 while
~4x-ing misIDs (209→864, averaged across models). `full_noexp` scores lower
(72.4% vs 75.3% for LVFace) but with **zero** true misIDs and higher precision
(94.2% vs 89.7%). Kept `full_exp` as shipped since it's the highest-F1 option
available without cast-restriction, but this is a real F1-vs-safety trade, not
a strictly-better choice — worth revisiting if misID rate matters more than
the last few points of F1 for a given deployment.
- **Switching the default model is an operational change, not just a config tweak**:
any existing gallery built from r50 embeddings is incompatible with LVFace
embeddings and needs rebuilding.
- r50's 4 combos used the older, narrower search bounds (1-30/1-15 instead
of 1-60/1-60) and are further confounded by its thinner gallery. Excluded
from all comparisons above except calibration.
- The shipped defaults use `full_exp` (75.3% training F1), not the
higher-scoring `restricted_exp` (78.3%), because cast restriction is not
a runtime feature of the application yet.
- `expand_gallery` is mode-dependent, not a free win. Averaged across models
on the full gallery it trades misIDs for recall (see the pose-expansion
table). For LVFace specifically, though, `full_exp` beats `full_noexp` on
every axis at once (F1 75.3 vs 72.3, precision 89.7 vs 88.3, recall 65.4 vs
61.8, misID 232 vs 244), so the shipped `full_exp` is a clean choice for
this model, not an F1-vs-safety trade. (An earlier version of this page
reported `full_noexp` at 72.4% with zero misIDs and higher precision, which
made it look like the safer option; that was the dropped-film artifact
described above, not a real property of the config.)
- Switching the default model is an operational change: any gallery built
from a different model's embeddings must be rebuilt before the new
default takes effect.
## Reproduce
```bash
# 4-film matrix, all 4 models × 2 modes × 2 expansion settings
# 4-film training matrix, all 4 models × 2 gallery modes × 2 expansion settings
bash experiments/run_rep4_subprocess.sh
# single combo
@@ -429,34 +326,21 @@ SAE_EXPAND=1 REPLAY_WORKERS=4 DE_WORKERS=2 python3 scripts/optimizer/optimize.py
--params prob_threshold:0.5:0.999 anneal_sec:1:60 extinction_sec:1:60 \
--popsize 10 --maxiter 15 --trajectory traj.jsonl --out best.json
# replay the shipped config against a held-out film — --raw-out is needed to draw
# bboxes later (the merged pred.json has no per-frame bbox, only actor windows)
python3 scripts/optimizer/replay.py \
--dump experiments/dumps/LVFace-B_Glint360K/dump_<slug>.h5 \
--gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5 \
--out pred.json --raw-out raw.jsonl --prob-threshold 0.754 --anneal-sec 35.54 \
--extinction-sec 57.43 --expand-gallery
# held-out validation, all 3 models, 5 films
python3 scripts/docs/run_holdout_all_models.py --out docs_data/holdout_all_models.json
# regenerate the report's charts (16-combo ranking, DE landscape, held-out
# per-film F1, Downton ghost timeline) from the artifacts under experiments/
# per-film training breakdown, all 3 models, 4 films
python3 scripts/docs/run_holdout_all_models.py --films training --out docs_data/training_per_film.json
# gallery coverage per film
python3 scripts/docs/gallery_coverage_per_film.py --out docs_data/gallery_coverage_per_film.json
# regenerate this page's charts from experiments/ artifacts
python3 scripts/docs/experiment_charts.py --out-dir docs/assets/images
# dump example frames (best-agreement / FPI / FN) for visual inspection, annotated
# with bounding boxes + names (--raw is optional; omit for unannotated frames)
python3 scripts/optimizer/dump_error_frames.py \
--pred pred.json --raw raw.jsonl --xray experiments/xray/.../<xray_dir> \
--movie "<path to source video>" \
--gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5 \
--out-dir experiments/results/holdout/frames/<name> --n-per-bucket 4
# or: one best + one worst frame per 10-minute window across the whole film
python3 scripts/optimizer/dump_error_frames.py \
--pred pred.json --raw raw.jsonl --xray experiments/xray/.../<xray_dir> \
--movie "<path to source video>" \
--gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5 \
--out-dir experiments/results/holdout/frames/<name>_intervals --interval-sec 600
# one frame per distinct out-of-cast name across all 9 films (used in the deep dive)
python3 scripts/docs/first_fpi_frames.py
```
See also: [the prior optimizer round](optimizer-experiments.md) (superseded
metric) and the session log
See also the session log
[`experiments/SESSION_STATE.md`](https://REPOLINK/experiments/SESSION_STATE.md).