+
+
+
+ Operational note
@@ -803,31 +825,35 @@
Which embedding model is best?
-Four candidates went into the bake-off: three ArcFace variants (w600k-R50,
-R18, w600k-MBF) and LVFace-B (Glint360K), a Vision-Transformer embedder that's
-a drop-in replacement for ArcFace's [N,3,112,112] input / 512-d output. The
-open question: is LVFace (455MB) actually better, or just the biggest?
+Three ArcFace variants (w600k-R50, R18, w600k-MBF) and LVFace-B (Glint360K,
+455MB) were compared. r50 is excluded from the training/held-out comparison
+below; its gallery has roughly 30% fewer reference images per actor than the
+other three on the identical source photos, which confounds a direct score
+comparison (see the full experiment log for detail). It
+remains in the calibration comparison, which does not depend on the gallery
+image count.
First signal: calibration curves
Each gallery carries a fitted Platt sigmoid P(match | cosine similarity) =
-σ(a·sim + b), embedded directly in the gallery's HDF5 file
-(src/gallery/gallery_calibration.hpp). This is a property of the embedding
-space alone — computed from intra/inter-actor reference-image pairs, no
-tracking or scene logic involved — so it's a clean first read on discriminative
-power before running a single benchmark.
+σ(a·sim + b), stored directly in the gallery HDF5
+(src/gallery/gallery_calibration.hpp).
+This is a property of the embedding space alone, computed from intra- and
+inter-actor reference-image pairs with no tracking or scene logic involved,
+so it is a clean first read on discriminative power before running a
+benchmark.
| model |
-a (steepness) |
+a (steepness) |
boundary at P=0.5 |
-| LVFace-B Glint360K |
-17.7 |
-sim 0.228 |
+LVFace-B Glint360K |
+17.7 |
+sim 0.228 |
| ArcFace w600k-MBF |
@@ -846,13 +872,123 @@ power before running a single benchmark.
-LVFace has both the steepest transition and the lowest decision boundary — it
-separates same-actor from different-actor reference pairs more confidently, at
-a lower similarity threshold, than any ArcFace variant. That's a genuine
-head start before the tracking/scoring pipeline is even involved.
-Second signal: F1 on the actual benchmark
-Best full-gallery (no cast-restriction) result per model, from the 16-combo
-bake-off matrix (full experiment log):
+LVFace has both the steepest transition and the lowest decision boundary,
+separating same-actor from different-actor reference pairs more confidently
+at a lower similarity than any ArcFace variant.
+Second signal: held-out F1
+Each model's own tuned full_exp config, replayed against the 5 films the
+optimizer never saw and scored the same way:
+
+
+
+| film |
+LVFace F1 |
+mbf F1 |
+r18 F1 |
+
+
+
+
+| Benny & Joon |
+83.0% |
+78.5% |
+77.1% |
+
+
+| Lovelace |
+77.5% |
+73.7% |
+72.2% |
+
+
+| Valerian and the City of a Thousand Planets |
+74.1% |
+70.2% |
+71.0% |
+
+
+| Downton Abbey: A New Era |
+56.2% |
+55.0% |
+53.0% |
+
+
+| The Many Saints of Newark |
+46.3% |
+44.5% |
+42.1% |
+
+
+| macro average |
+67.4% |
+64.4% |
+63.1% |
+
+
+
+LVFace scores highest on all 5 held-out films; the ranking never flips
+between models. Total misID count across the 5 films: LVFace 1032, mbf
+2197, r18 1224. LVFace has less than half mbf's misID total and still
+scores higher on every film.
+Held-out results are stronger evidence than training results, because
+training numbers can reflect what the optimizer was tuned to fit rather
+than general performance. On training data, the ordering is not as clean:
+
+
+
+| film |
+LVFace F1 |
+mbf F1 |
+r18 F1 |
+best |
+
+
+
+
+| Café Society |
+68.1% |
+62.2% |
+60.1% |
+LVFace |
+
+
+| Lord of War |
+75.6% |
+77.2% |
+75.6% |
+mbf |
+
+
+| Scarface |
+71.5% |
+68.6% |
+64.1% |
+LVFace |
+
+
+| Sound of Metal |
+78.8% |
+76.5% |
+71.6% |
+LVFace |
+
+
+
+mbf beats LVFace on Lord of War (77.2% vs 75.6%), the only film in either
+table where LVFace does not score highest. LVFace's training-set macro
+average (75.3%, see the full experiment log) is not a
+uniform win across every film it contributes to; the held-out result, where
+LVFace wins all 5 films outright, is the stronger claim.
+This reverses an earlier, superseded benchmarking pass that used a
+scene-union metric and found the three models statistically
+indistinguishable (around 85% each), concluding LVFace was not worth its
+size. That metric masked out-of-cast false positives behind a
+gallery-intersect-cast recall filter; the per-second metric used here does
+not.
+Full training-matrix picture
+
+Best full-gallery combo per model (all three are full_exp), from the
+training matrix in the full experiment log:
@@ -865,18 +1001,18 @@ bake-off matrix (full experiment log):
-| LVFace-B Glint360K |
-75.3% |
+LVFace-B Glint360K |
+75.3% |
89.7% |
-65.4% |
+65.4% |
232 |
| ArcFace w600k-MBF |
-74.2% |
-87.4% |
-64.4% |
-57 |
+72.0% |
+87.7% |
+61.4% |
+240 |
| ArcFace R18 |
@@ -885,36 +1021,18 @@ bake-off matrix (full experiment log):
57.7% |
242 |
-
-| ArcFace w600k-R50 |
-68.5% |
-94.0% |
-54.1% |
-150 |
-
-The full 16-combo picture makes the model ordering visible at a glance — LVFace
-(yellow) tops both the restricted and full columns, and R18 (green) props up
-the bottom of the full-gallery ranking:
-
-LVFace wins outright, with the highest recall of any full-mode combo. This
-reverses an earlier conclusion from a prior (superseded) benchmarking pass
-using a scene-union metric, which found the three models statistically
-indistinguishable (~85% each) and concluded LVFace wasn't worth its size — that
-metric hid out-of-cast false positives behind a gallery∩cast recall mask (see
-the prior optimizer round); the per-second metric
-used here does not.
-Held-out validation (5 films never seen by the optimizer) confirms LVFace's
-lead holds up out of sample — see the
-LVFace deep dive for the full breakdown, including
-where it fails.
-Caveat: model choice is an operational change
-Switching the default embedder isn't just flipping a config value — the
-gallery itself is model-specific (embeddings from different models aren't
-comparable), so any existing gallery built against ArcFace w600k-R50 needs to
-be rebuilt from source images against LVFace before the new default takes
-effect. scripts/optimizer/reembed_gallery.py
+
LVFace leads within both the restricted and full gallery modes, visible
+directly in the chart above without reading the table. The three models'
+misID counts on the full gallery are nearly identical (232/240/242); LVFace's
+lead here is a precision-and-recall lead, not a misID one.
+Operational note
+Switching the default embedder is not a config change alone; the gallery
+is model-specific, since embeddings from different models are not
+comparable. Any existing gallery built against a different model must be
+rebuilt from source images before the new default takes effect.
+scripts/optimizer/reembed_gallery.py
does this from a reference gallery's cached source images without
re-downloading anything.
@@ -952,7 +1070,7 @@ re-downloading anything.