docs: full data-grounded rewrite of the performance report

Replaces narrative claims with verified numbers across all report pages:

- Cross-model held-out validation (LVFace/mbf/r18, all 5 held-out
  films): LVFace wins every film outright, not just "consistent with"
  the training-set pick. r50 dropped from the detailed comparison
  (gallery has ~30% fewer reference images per actor than the other
  three models on identical source photos).
- Per-film training breakdown: LVFace does not win every training
  film (mbf beats it on Lord of War); the 75.3% macro figure hides a
  10.7pp spread.
- Gallery coverage computed per film (20.3%-78.6%) instead of one
  flat 67%-missing average.
- Found and fixed a real scoring bug in optimize.py: a candidate
  whose hardest film's replay timed out was averaged over survivors
  instead of penalized, silently rewarding partial coverage. Affected
  3 of 16 training combos; corrected throughout, and optimize.py now
  scores an incomplete evaluation f1=0.0 instead of averaging over
  whichever films happened to finish.
- Every FPI frame in the deep dive now comes from the proper montage
  renderer (Onscreen/Offscreen panel, ghosts never drawn as boxes),
  never the bare-box debug overlay used earlier.
- Every distinct out-of-cast name across all 9 films gets its own
  frame at its first appearance (9 names, 4 films), not a
  single-example spot check: 2 ground-truth gaps, 1 photograph
  misread as a person, 6 genuine lookalike confusions.
- New methodology.md: the scene-level-vs-per-second scoring mismatch
  that the rest of the report assumes, written out once.
- Cut the deadlock/gdb debugging narrative from the experiment log;
  kept the one fact that matters (KPN's node/network split lets the
  expensive GPU stage run once and the cheap stage replay against
  cached embeddings).
- Plain declarative style throughout, no em dashes, no blog voice.
This commit is contained in:
2026-07-21 08:55:57 +02:00
parent 4b5557974b
commit 0bd2747069
18 changed files with 1824 additions and 890 deletions
+53 -49
View File
@@ -1,65 +1,69 @@
# Whole gallery vs. limited (cast-restricted) gallery
# Whole gallery vs. cast-restricted gallery
Two ways to run the matcher: **full** scores every detected face against the
entire library gallery (2418 actors across the 9-film benchmark set); **restricted**
pre-filters each film's gallery down to just its Jellyfin-credited cast (typically
~15 top-billed actors) before the matcher ever runs.
Two ways to run the matcher. Full mode scores every detected face against
the entire 2418-actor gallery. Restricted mode pre-filters each film's
gallery down to just its Jellyfin-credited cast (typically around 15
top-billed actors) before the matcher runs.
## The result
## Result
Averaged across all 4 models and both expansion settings, on the 4 bake-off training
films:
Averaged across the 3 compared models (r50 excluded, see
[the full experiment log](model-bakeoff.md)) and both expansion settings, on
the 4 training films:
| scope | F1 | P | R | total misID (8 evals) |
| scope | F1 | P | R | total misID |
|---|---|---|---|---|
| full | 71.2% | 91.1% | 59.0% | 1073 |
| **restricted** | **74.5%** | 92.2% | **62.9%** | **329** |
| full | 71.1% | 89.6% | 59.6% | 1121 |
| restricted | 75.9% | 90.4% | 65.6% | 299 |
This is not a precision/recall trade — restriction wins on every axis at once:
**+3.3pp F1, +3.9pp recall, and less than a third the total misIDs.** Fewer
Restriction improves every metric at once, not a precision/recall trade:
+4.8pp F1, +6.0pp recall, roughly a quarter the total misIDs. Fewer
candidates in the matcher's search space means fewer opportunities for a
look-alike false match (an actor who happens to share enough facial structure
with someone in the film, but isn't actually in it), and the recall gain shows
it isn't costing real detections to get there.
lookalike false match, and the recall gain shows this does not cost real
detections.
Per-model, every single model's best-scoring combo in the full 16-way matrix is
a `restricted` variant — visible directly in the ranking below (filled dots =
restricted, open = full; the filled dots cluster at the top for every color):
Every model's best-scoring combo in the training matrix uses the
restricted gallery:
![All 16 bake-off combos — filled dots (restricted) dominate the top](assets/images/rep4_matrix_f1.png)
![All combos ranked by training-set F1, filled dots are restricted](assets/images/rep4_matrix_f1.png)
See the full table in the
[bake-off experiment log](model-bakeoff.md). Two
combos hit **zero** true out-of-cast misidentifications:
`arcface_w600k_mbf_restricted_exp` (F1 76.5%) and, in full mode,
`LVFace-B_Glint360K_full_noexp` (F1 72.4%) — restriction isn't the only way to
reach misid=0, but it's the more reliable one.
See [the full experiment log](model-bakeoff.md) for the complete table. One
combo reaches zero true out-of-cast misidentifications,
`arcface_w600k_mbf_restricted_exp` (F1 76.2%), and it is a restricted one,
consistent with restriction, not expansion, being what suppresses cross-film
confusions.
## Why this isn't the shipped default
The restriction effect (+4.8pp averaged across models) is larger than the
model-choice effect: LVFace beats r18 by 6.2pp in full mode but beats mbf by
3.3pp. Restriction is the single strongest lever in the matrix.
Cast-restriction is implemented today only as an **offline optimizer technique**
## Why this is not the shipped default
Cast restriction is implemented today only as an offline optimizer
technique
([`scripts/optimizer/cast_restrict.py`](https://REPOLINK/scripts/optimizer/cast_restrict.py)):
it pre-builds a filtered gallery file
per film, using Jellyfin's own cast list, before the benchmark ever calls the
matcher. There's no runtime "restrict matching to this title's credited cast"
switch in the shipped application — `scene_analyze` always matches against
whatever single gallery file it's given.
it pre-builds a filtered gallery file per film using Jellyfin's cast list
before the benchmark calls the matcher. There is no runtime "restrict to
this title's credited cast" switch in the shipped application;
`scene_analyze` always matches against whatever single gallery file it is
given.
Building that as a real feature would need, at minimum:
Building this as a real feature requires:
- A live Jellyfin cast lookup at analysis time (the title is already known
[`scripts/run_from_jellyfin.py`](https://REPOLINK/scripts/run_from_jellyfin.py)
already does this same lookup for its own
`filter_gallery`-based restriction path, just not wired into `scene_analyze`
itself as a first-class option).
- A decision on the *fallback*: what happens to a real, uncredited cameo
(see the Germar Terrell Gardner case in the LVFace deep-dive) if the gallery
never includes them at all?
- Regenerating the restricted-gallery cache whenever the title's Jellyfin cast
list changes.
- A live Jellyfin cast lookup at analysis time. The title is already known,
and [`scripts/run_from_jellyfin.py`](https://REPOLINK/scripts/run_from_jellyfin.py)
already performs this lookup for its own `filter_gallery`-based
restriction path; it is not wired into `scene_analyze` as a first-class
option.
- A decision on the fallback case: what happens to a real, uncredited
cameo (see the Germar Terrell Gardner and Talia Balsam cases in the
[LVFace deep dive](lvface-deep-dive.md#where-lvface-beat-x-ray)) if the
restricted gallery never includes them at all.
- Regenerating the restricted-gallery cache whenever a title's Jellyfin
cast list changes.
This is why the shipped [`src/config.hpp`](https://REPOLINK/src/config.hpp)
defaults use the `full`-mode winner
(`LVFace-B_Glint360K_full_exp`, F1 75.3% training / 67.4% held-out macro) rather
than the higher-scoring `restricted_exp` (78.3%) — the 78.3% number describes a
capability the app doesn't have yet, not what actually ships.
The shipped [`src/config.hpp`](https://REPOLINK/src/config.hpp) defaults use
the full-mode winner (`LVFace-B_Glint360K_full_exp`, F1 75.3% training,
67.4% held-out macro) rather than the higher-scoring `restricted_exp`
(78.3%), because 78.3% describes a capability the application does not
have yet.