Files

T

dtourolle 0bd2747069 docs: full data-grounded rewrite of the performance report

Replaces narrative claims with verified numbers across all report pages:

- Cross-model held-out validation (LVFace/mbf/r18, all 5 held-out
  films): LVFace wins every film outright, not just "consistent with"
  the training-set pick. r50 dropped from the detailed comparison
  (gallery has ~30% fewer reference images per actor than the other
  three models on identical source photos).
- Per-film training breakdown: LVFace does not win every training
  film (mbf beats it on Lord of War); the 75.3% macro figure hides a
  10.7pp spread.
- Gallery coverage computed per film (20.3%-78.6%) instead of one
  flat 67%-missing average.
- Found and fixed a real scoring bug in optimize.py: a candidate
  whose hardest film's replay timed out was averaged over survivors
  instead of penalized, silently rewarding partial coverage. Affected
  3 of 16 training combos; corrected throughout, and optimize.py now
  scores an incomplete evaluation f1=0.0 instead of averaging over
  whichever films happened to finish.
- Every FPI frame in the deep dive now comes from the proper montage
  renderer (Onscreen/Offscreen panel, ghosts never drawn as boxes),
  never the bare-box debug overlay used earlier.
- Every distinct out-of-cast name across all 9 films gets its own
  frame at its first appearance (9 names, 4 films), not a
  single-example spot check: 2 ground-truth gaps, 1 photograph
  misread as a person, 6 genuine lookalike confusions.
- New methodology.md: the scene-level-vs-per-second scoring mismatch
  that the rest of the report assumes, written out once.
- Cut the deadlock/gdb debugging narrative from the experiment log;
  kept the one fact that matters (KPN's node/network split lets the
  expensive GPU stage run once and the cheap stage replay against
  cached embeddings).
- Plain declarative style throughout, no em dashes, no blog voice.

2026-07-21 08:55:57 +02:00

4.6 KiB

Raw Permalink Blame History

Which embedding model is best?

Three ArcFace variants (w600k-R50, R18, w600k-MBF) and LVFace-B (Glint360K, 455MB) were compared. r50 is excluded from the training/held-out comparison below; its gallery has roughly 30% fewer reference images per actor than the other three on the identical source photos, which confounds a direct score comparison (see the full experiment log for detail). It remains in the calibration comparison, which does not depend on the gallery image count.

First signal: calibration curves

Each gallery carries a fitted Platt sigmoid P(match | cosine similarity) = σ(a·sim + b), stored directly in the gallery HDF5 (src/gallery/gallery_calibration.hpp). This is a property of the embedding space alone, computed from intra- and inter-actor reference-image pairs with no tracking or scene logic involved, so it is a clean first read on discriminative power before running a benchmark.

model	a (steepness)	boundary at P=0.5
LVFace-B Glint360K	17.7	sim 0.228
ArcFace w600k-MBF	16.2	sim 0.267
ArcFace w600k-R50	15.4	sim 0.301
ArcFace R18	15.3	sim 0.309

LVFace has both the steepest transition and the lowest decision boundary, separating same-actor from different-actor reference pairs more confidently at a lower similarity than any ArcFace variant.

Second signal: held-out F1

Each model's own tuned full_exp config, replayed against the 5 films the optimizer never saw and scored the same way:

film	LVFace F1	mbf F1	r18 F1
Benny & Joon	83.0%	78.5%	77.1%
Lovelace	77.5%	73.7%	72.2%
Valerian and the City of a Thousand Planets	74.1%	70.2%	71.0%
Downton Abbey: A New Era	56.2%	55.0%	53.0%
The Many Saints of Newark	46.3%	44.5%	42.1%
macro average	67.4%	64.4%	63.1%

LVFace scores highest on all 5 held-out films; the ranking never flips between models. Total misID count across the 5 films: LVFace 1032, mbf 2197, r18 1224. LVFace has less than half mbf's misID total and still scores higher on every film.

Held-out results are stronger evidence than training results, because training numbers can reflect what the optimizer was tuned to fit rather than general performance. On training data, the ordering is not as clean:

film	LVFace F1	mbf F1	r18 F1	best
Café Society	68.1%	62.2%	60.1%	LVFace
Lord of War	75.6%	77.2%	75.6%	mbf
Scarface	71.5%	68.6%	64.1%	LVFace
Sound of Metal	78.8%	76.5%	71.6%	LVFace

mbf beats LVFace on Lord of War (77.2% vs 75.6%), the only film in either table where LVFace does not score highest. LVFace's training-set macro average (75.3%, see the full experiment log) is not a uniform win across every film it contributes to; the held-out result, where LVFace wins all 5 films outright, is the stronger claim.

This reverses an earlier, superseded benchmarking pass that used a scene-union metric and found the three models statistically indistinguishable (around 85% each), concluding LVFace was not worth its size. That metric masked out-of-cast false positives behind a gallery-intersect-cast recall filter; the per-second metric used here does not.

Full training-matrix picture

Best full-gallery combo per model (all three are full_exp), from the training matrix in the full experiment log:

model	F1	P	R	misID
LVFace-B Glint360K	75.3%	89.7%	65.4%	232
ArcFace w600k-MBF	72.0%	87.7%	61.4%	240
ArcFace R18	69.1%	87.6%	57.7%	242

LVFace leads within both the restricted and full gallery modes, visible directly in the chart above without reading the table. The three models' misID counts on the full gallery are nearly identical (232/240/242); LVFace's lead here is a precision-and-recall lead, not a misID one.

Operational note

Switching the default embedder is not a config change alone; the gallery is model-specific, since embeddings from different models are not comparable. Any existing gallery built against a different model must be rebuilt from source images before the new default takes effect. scripts/optimizer/reembed_gallery.py does this from a reference gallery's cached source images without re-downloading anything.

4.6 KiB Raw Permalink Blame History Unescape Escape