dtourolle
0bd2747069
Replaces narrative claims with verified numbers across all report pages: - Cross-model held-out validation (LVFace/mbf/r18, all 5 held-out films): LVFace wins every film outright, not just "consistent with" the training-set pick. r50 dropped from the detailed comparison (gallery has ~30% fewer reference images per actor than the other three models on identical source photos). - Per-film training breakdown: LVFace does not win every training film (mbf beats it on Lord of War); the 75.3% macro figure hides a 10.7pp spread. - Gallery coverage computed per film (20.3%-78.6%) instead of one flat 67%-missing average. - Found and fixed a real scoring bug in optimize.py: a candidate whose hardest film's replay timed out was averaged over survivors instead of penalized, silently rewarding partial coverage. Affected 3 of 16 training combos; corrected throughout, and optimize.py now scores an incomplete evaluation f1=0.0 instead of averaging over whichever films happened to finish. - Every FPI frame in the deep dive now comes from the proper montage renderer (Onscreen/Offscreen panel, ghosts never drawn as boxes), never the bare-box debug overlay used earlier. - Every distinct out-of-cast name across all 9 films gets its own frame at its first appearance (9 names, 4 films), not a single-example spot check: 2 ground-truth gaps, 1 photograph misread as a person, 6 genuine lookalike confusions. - New methodology.md: the scene-level-vs-per-second scoring mismatch that the rest of the report assumes, written out once. - Cut the deadlock/gdb debugging narrative from the experiment log; kept the one fact that matters (KPN's node/network split lets the expensive GPU stage run once and the cheap stage replay against cached embeddings). - Plain declarative style throughout, no em dashes, no blog voice.
104 lines
4.6 KiB
Markdown
104 lines
4.6 KiB
Markdown
# Which embedding model is best?
|
||
|
||
Three ArcFace variants (w600k-R50, R18, w600k-MBF) and LVFace-B (Glint360K,
|
||
455MB) were compared. r50 is excluded from the training/held-out comparison
|
||
below; its gallery has roughly 30% fewer reference images per actor than the
|
||
other three on the identical source photos, which confounds a direct score
|
||
comparison (see [the full experiment log](model-bakeoff.md) for detail). It
|
||
remains in the calibration comparison, which does not depend on the gallery
|
||
image count.
|
||
|
||
## First signal: calibration curves
|
||
|
||
Each gallery carries a fitted Platt sigmoid `P(match | cosine similarity) =
|
||
σ(a·sim + b)`, stored directly in the gallery HDF5
|
||
([`src/gallery/gallery_calibration.hpp`](https://REPOLINK/src/gallery/gallery_calibration.hpp)).
|
||
This is a property of the embedding space alone, computed from intra- and
|
||
inter-actor reference-image pairs with no tracking or scene logic involved,
|
||
so it is a clean first read on discriminative power before running a
|
||
benchmark.
|
||
|
||
|
||
|
||
| model | a (steepness) | boundary at P=0.5 |
|
||
|---|---|---|
|
||
| LVFace-B Glint360K | 17.7 | sim 0.228 |
|
||
| ArcFace w600k-MBF | 16.2 | sim 0.267 |
|
||
| ArcFace w600k-R50 | 15.4 | sim 0.301 |
|
||
| ArcFace R18 | 15.3 | sim 0.309 |
|
||
|
||
LVFace has both the steepest transition and the lowest decision boundary,
|
||
separating same-actor from different-actor reference pairs more confidently
|
||
at a lower similarity than any ArcFace variant.
|
||
|
||
## Second signal: held-out F1
|
||
|
||
Each model's own tuned `full_exp` config, replayed against the 5 films the
|
||
optimizer never saw and scored the same way:
|
||
|
||
| film | LVFace F1 | mbf F1 | r18 F1 |
|
||
|---|---|---|---|
|
||
| Benny & Joon | 83.0% | 78.5% | 77.1% |
|
||
| Lovelace | 77.5% | 73.7% | 72.2% |
|
||
| Valerian and the City of a Thousand Planets | 74.1% | 70.2% | 71.0% |
|
||
| Downton Abbey: A New Era | 56.2% | 55.0% | 53.0% |
|
||
| The Many Saints of Newark | 46.3% | 44.5% | 42.1% |
|
||
| **macro average** | **67.4%** | **64.4%** | **63.1%** |
|
||
|
||
LVFace scores highest on all 5 held-out films; the ranking never flips
|
||
between models. Total misID count across the 5 films: LVFace 1032, mbf
|
||
2197, r18 1224. LVFace has less than half mbf's misID total and still
|
||
scores higher on every film.
|
||
|
||
Held-out results are stronger evidence than training results, because
|
||
training numbers can reflect what the optimizer was tuned to fit rather
|
||
than general performance. On training data, the ordering is not as clean:
|
||
|
||
| film | LVFace F1 | mbf F1 | r18 F1 | best |
|
||
|---|---|---|---|---|
|
||
| Café Society | 68.1% | 62.2% | 60.1% | LVFace |
|
||
| Lord of War | 75.6% | 77.2% | 75.6% | mbf |
|
||
| Scarface | 71.5% | 68.6% | 64.1% | LVFace |
|
||
| Sound of Metal | 78.8% | 76.5% | 71.6% | LVFace |
|
||
|
||
mbf beats LVFace on Lord of War (77.2% vs 75.6%), the only film in either
|
||
table where LVFace does not score highest. LVFace's training-set macro
|
||
average (75.3%, see [the full experiment log](model-bakeoff.md)) is not a
|
||
uniform win across every film it contributes to; the held-out result, where
|
||
LVFace wins all 5 films outright, is the stronger claim.
|
||
|
||
This reverses an earlier, superseded benchmarking pass that used a
|
||
scene-union metric and found the three models statistically
|
||
indistinguishable (around 85% each), concluding LVFace was not worth its
|
||
size. That metric masked out-of-cast false positives behind a
|
||
gallery-intersect-cast recall filter; the per-second metric used here does
|
||
not.
|
||
|
||
## Full training-matrix picture
|
||
|
||
|
||
|
||
Best full-gallery combo per model (all three are `full_exp`), from the
|
||
training matrix in [the full experiment log](model-bakeoff.md):
|
||
|
||
| model | F1 | P | R | misID |
|
||
|---|---|---|---|---|
|
||
| LVFace-B Glint360K | 75.3% | 89.7% | 65.4% | 232 |
|
||
| ArcFace w600k-MBF | 72.0% | 87.7% | 61.4% | 240 |
|
||
| ArcFace R18 | 69.1% | 87.6% | 57.7% | 242 |
|
||
|
||
LVFace leads within both the restricted and full gallery modes, visible
|
||
directly in the chart above without reading the table. The three models'
|
||
misID counts on the full gallery are nearly identical (232/240/242); LVFace's
|
||
lead here is a precision-and-recall lead, not a misID one.
|
||
|
||
## Operational note
|
||
|
||
Switching the default embedder is not a config change alone; the gallery
|
||
is model-specific, since embeddings from different models are not
|
||
comparable. Any existing gallery built against a different model must be
|
||
rebuilt from source images before the new default takes effect.
|
||
[`scripts/optimizer/reembed_gallery.py`](https://REPOLINK/scripts/optimizer/reembed_gallery.py)
|
||
does this from a reference gallery's cached source images without
|
||
re-downloading anything.
|