dtourolle
0bd2747069
Replaces narrative claims with verified numbers across all report pages: - Cross-model held-out validation (LVFace/mbf/r18, all 5 held-out films): LVFace wins every film outright, not just "consistent with" the training-set pick. r50 dropped from the detailed comparison (gallery has ~30% fewer reference images per actor than the other three models on identical source photos). - Per-film training breakdown: LVFace does not win every training film (mbf beats it on Lord of War); the 75.3% macro figure hides a 10.7pp spread. - Gallery coverage computed per film (20.3%-78.6%) instead of one flat 67%-missing average. - Found and fixed a real scoring bug in optimize.py: a candidate whose hardest film's replay timed out was averaged over survivors instead of penalized, silently rewarding partial coverage. Affected 3 of 16 training combos; corrected throughout, and optimize.py now scores an incomplete evaluation f1=0.0 instead of averaging over whichever films happened to finish. - Every FPI frame in the deep dive now comes from the proper montage renderer (Onscreen/Offscreen panel, ghosts never drawn as boxes), never the bare-box debug overlay used earlier. - Every distinct out-of-cast name across all 9 films gets its own frame at its first appearance (9 names, 4 films), not a single-example spot check: 2 ground-truth gaps, 1 photograph misread as a person, 6 genuine lookalike confusions. - New methodology.md: the scene-level-vs-per-second scoring mismatch that the rest of the report assumes, written out once. - Cut the deadlock/gdb debugging narrative from the experiment log; kept the one fact that matters (KPN's node/network split lets the expensive GPU stage run once and the cheap stage replay against cached embeddings). - Plain declarative style throughout, no em dashes, no blog voice.
3.1 KiB
Whole gallery vs. cast-restricted gallery
Two ways to run the matcher. Full mode scores every detected face against the entire 2418-actor gallery. Restricted mode pre-filters each film's gallery down to just its Jellyfin-credited cast (typically around 15 top-billed actors) before the matcher runs.
Result
Averaged across the 3 compared models (r50 excluded, see the full experiment log) and both expansion settings, on the 4 training films:
| scope | F1 | P | R | total misID |
|---|---|---|---|---|
| full | 71.1% | 89.6% | 59.6% | 1121 |
| restricted | 75.9% | 90.4% | 65.6% | 299 |
Restriction improves every metric at once, not a precision/recall trade: +4.8pp F1, +6.0pp recall, roughly a quarter the total misIDs. Fewer candidates in the matcher's search space means fewer opportunities for a lookalike false match, and the recall gain shows this does not cost real detections.
Every model's best-scoring combo in the training matrix uses the restricted gallery:
See the full experiment log for the complete table. One
combo reaches zero true out-of-cast misidentifications,
arcface_w600k_mbf_restricted_exp (F1 76.2%), and it is a restricted one,
consistent with restriction, not expansion, being what suppresses cross-film
confusions.
The restriction effect (+4.8pp averaged across models) is larger than the model-choice effect: LVFace beats r18 by 6.2pp in full mode but beats mbf by 3.3pp. Restriction is the single strongest lever in the matrix.
Why this is not the shipped default
Cast restriction is implemented today only as an offline optimizer
technique
(scripts/optimizer/cast_restrict.py):
it pre-builds a filtered gallery file per film using Jellyfin's cast list
before the benchmark calls the matcher. There is no runtime "restrict to
this title's credited cast" switch in the shipped application;
scene_analyze always matches against whatever single gallery file it is
given.
Building this as a real feature requires:
- A live Jellyfin cast lookup at analysis time. The title is already known,
and
scripts/run_from_jellyfin.pyalready performs this lookup for its ownfilter_gallery-based restriction path; it is not wired intoscene_analyzeas a first-class option. - A decision on the fallback case: what happens to a real, uncredited cameo (see the Germar Terrell Gardner and Talia Balsam cases in the LVFace deep dive) if the restricted gallery never includes them at all.
- Regenerating the restricted-gallery cache whenever a title's Jellyfin cast list changes.
The shipped src/config.hpp defaults use
the full-mode winner (LVFace-B_Glint360K_full_exp, F1 75.3% training,
67.4% held-out macro) rather than the higher-scoring restricted_exp
(78.3%), because 78.3% describes a capability the application does not
have yet.