docs: full data-grounded rewrite of the performance report
Replaces narrative claims with verified numbers across all report pages: - Cross-model held-out validation (LVFace/mbf/r18, all 5 held-out films): LVFace wins every film outright, not just "consistent with" the training-set pick. r50 dropped from the detailed comparison (gallery has ~30% fewer reference images per actor than the other three models on identical source photos). - Per-film training breakdown: LVFace does not win every training film (mbf beats it on Lord of War); the 75.3% macro figure hides a 10.7pp spread. - Gallery coverage computed per film (20.3%-78.6%) instead of one flat 67%-missing average. - Found and fixed a real scoring bug in optimize.py: a candidate whose hardest film's replay timed out was averaged over survivors instead of penalized, silently rewarding partial coverage. Affected 3 of 16 training combos; corrected throughout, and optimize.py now scores an incomplete evaluation f1=0.0 instead of averaging over whichever films happened to finish. - Every FPI frame in the deep dive now comes from the proper montage renderer (Onscreen/Offscreen panel, ghosts never drawn as boxes), never the bare-box debug overlay used earlier. - Every distinct out-of-cast name across all 9 films gets its own frame at its first appearance (9 names, 4 films), not a single-example spot check: 2 ground-truth gaps, 1 photograph misread as a person, 6 genuine lookalike confusions. - New methodology.md: the scene-level-vs-per-second scoring mismatch that the rest of the report assumes, written out once. - Cut the deadlock/gdb debugging narrative from the experiment log; kept the one fact that matters (KPN's node/network split lets the expensive GPU stage run once and the cheap stage replay against cached embeddings). - Plain declarative style throughout, no em dashes, no blog voice.
This commit is contained in:
+53
-49
@@ -1,65 +1,69 @@
|
||||
# Whole gallery vs. limited (cast-restricted) gallery
|
||||
# Whole gallery vs. cast-restricted gallery
|
||||
|
||||
Two ways to run the matcher: **full** scores every detected face against the
|
||||
entire library gallery (2418 actors across the 9-film benchmark set); **restricted**
|
||||
pre-filters each film's gallery down to just its Jellyfin-credited cast (typically
|
||||
~15 top-billed actors) before the matcher ever runs.
|
||||
Two ways to run the matcher. Full mode scores every detected face against
|
||||
the entire 2418-actor gallery. Restricted mode pre-filters each film's
|
||||
gallery down to just its Jellyfin-credited cast (typically around 15
|
||||
top-billed actors) before the matcher runs.
|
||||
|
||||
## The result
|
||||
## Result
|
||||
|
||||
Averaged across all 4 models and both expansion settings, on the 4 bake-off training
|
||||
films:
|
||||
Averaged across the 3 compared models (r50 excluded, see
|
||||
[the full experiment log](model-bakeoff.md)) and both expansion settings, on
|
||||
the 4 training films:
|
||||
|
||||
| scope | F1 | P | R | total misID (8 evals) |
|
||||
| scope | F1 | P | R | total misID |
|
||||
|---|---|---|---|---|
|
||||
| full | 71.2% | 91.1% | 59.0% | 1073 |
|
||||
| **restricted** | **74.5%** | 92.2% | **62.9%** | **329** |
|
||||
| full | 71.1% | 89.6% | 59.6% | 1121 |
|
||||
| restricted | 75.9% | 90.4% | 65.6% | 299 |
|
||||
|
||||
This is not a precision/recall trade — restriction wins on every axis at once:
|
||||
**+3.3pp F1, +3.9pp recall, and less than a third the total misIDs.** Fewer
|
||||
Restriction improves every metric at once, not a precision/recall trade:
|
||||
+4.8pp F1, +6.0pp recall, roughly a quarter the total misIDs. Fewer
|
||||
candidates in the matcher's search space means fewer opportunities for a
|
||||
look-alike false match (an actor who happens to share enough facial structure
|
||||
with someone in the film, but isn't actually in it), and the recall gain shows
|
||||
it isn't costing real detections to get there.
|
||||
lookalike false match, and the recall gain shows this does not cost real
|
||||
detections.
|
||||
|
||||
Per-model, every single model's best-scoring combo in the full 16-way matrix is
|
||||
a `restricted` variant — visible directly in the ranking below (filled dots =
|
||||
restricted, open = full; the filled dots cluster at the top for every color):
|
||||
Every model's best-scoring combo in the training matrix uses the
|
||||
restricted gallery:
|
||||
|
||||

|
||||

|
||||
|
||||
See the full table in the
|
||||
[bake-off experiment log](model-bakeoff.md). Two
|
||||
combos hit **zero** true out-of-cast misidentifications:
|
||||
`arcface_w600k_mbf_restricted_exp` (F1 76.5%) and, in full mode,
|
||||
`LVFace-B_Glint360K_full_noexp` (F1 72.4%) — restriction isn't the only way to
|
||||
reach misid=0, but it's the more reliable one.
|
||||
See [the full experiment log](model-bakeoff.md) for the complete table. One
|
||||
combo reaches zero true out-of-cast misidentifications,
|
||||
`arcface_w600k_mbf_restricted_exp` (F1 76.2%), and it is a restricted one,
|
||||
consistent with restriction, not expansion, being what suppresses cross-film
|
||||
confusions.
|
||||
|
||||
## Why this isn't the shipped default
|
||||
The restriction effect (+4.8pp averaged across models) is larger than the
|
||||
model-choice effect: LVFace beats r18 by 6.2pp in full mode but beats mbf by
|
||||
3.3pp. Restriction is the single strongest lever in the matrix.
|
||||
|
||||
Cast-restriction is implemented today only as an **offline optimizer technique**
|
||||
## Why this is not the shipped default
|
||||
|
||||
Cast restriction is implemented today only as an offline optimizer
|
||||
technique
|
||||
([`scripts/optimizer/cast_restrict.py`](https://REPOLINK/scripts/optimizer/cast_restrict.py)):
|
||||
it pre-builds a filtered gallery file
|
||||
per film, using Jellyfin's own cast list, before the benchmark ever calls the
|
||||
matcher. There's no runtime "restrict matching to this title's credited cast"
|
||||
switch in the shipped application — `scene_analyze` always matches against
|
||||
whatever single gallery file it's given.
|
||||
it pre-builds a filtered gallery file per film using Jellyfin's cast list
|
||||
before the benchmark calls the matcher. There is no runtime "restrict to
|
||||
this title's credited cast" switch in the shipped application;
|
||||
`scene_analyze` always matches against whatever single gallery file it is
|
||||
given.
|
||||
|
||||
Building that as a real feature would need, at minimum:
|
||||
Building this as a real feature requires:
|
||||
|
||||
- A live Jellyfin cast lookup at analysis time (the title is already known —
|
||||
[`scripts/run_from_jellyfin.py`](https://REPOLINK/scripts/run_from_jellyfin.py)
|
||||
already does this same lookup for its own
|
||||
`filter_gallery`-based restriction path, just not wired into `scene_analyze`
|
||||
itself as a first-class option).
|
||||
- A decision on the *fallback*: what happens to a real, uncredited cameo
|
||||
(see the Germar Terrell Gardner case in the LVFace deep-dive) if the gallery
|
||||
never includes them at all?
|
||||
- Regenerating the restricted-gallery cache whenever the title's Jellyfin cast
|
||||
list changes.
|
||||
- A live Jellyfin cast lookup at analysis time. The title is already known,
|
||||
and [`scripts/run_from_jellyfin.py`](https://REPOLINK/scripts/run_from_jellyfin.py)
|
||||
already performs this lookup for its own `filter_gallery`-based
|
||||
restriction path; it is not wired into `scene_analyze` as a first-class
|
||||
option.
|
||||
- A decision on the fallback case: what happens to a real, uncredited
|
||||
cameo (see the Germar Terrell Gardner and Talia Balsam cases in the
|
||||
[LVFace deep dive](lvface-deep-dive.md#where-lvface-beat-x-ray)) if the
|
||||
restricted gallery never includes them at all.
|
||||
- Regenerating the restricted-gallery cache whenever a title's Jellyfin
|
||||
cast list changes.
|
||||
|
||||
This is why the shipped [`src/config.hpp`](https://REPOLINK/src/config.hpp)
|
||||
defaults use the `full`-mode winner
|
||||
(`LVFace-B_Glint360K_full_exp`, F1 75.3% training / 67.4% held-out macro) rather
|
||||
than the higher-scoring `restricted_exp` (78.3%) — the 78.3% number describes a
|
||||
capability the app doesn't have yet, not what actually ships.
|
||||
The shipped [`src/config.hpp`](https://REPOLINK/src/config.hpp) defaults use
|
||||
the full-mode winner (`LVFace-B_Glint360K_full_exp`, F1 75.3% training,
|
||||
67.4% held-out macro) rather than the higher-scoring `restricted_exp`
|
||||
(78.3%), because 78.3% describes a capability the application does not
|
||||
have yet.
|
||||
|
||||
Reference in New Issue
Block a user