> **Archived (July 2026).** This report covers the pre-opencv5 framework and the 4-model ArcFace/LVFace bake-off. It is superseded by the current [experiment log](model-bakeoff.md) for the opencv5 build. Kept for provenance; the numbers here are historical.

# Which embedding model is best?

Three ArcFace variants (w600k-R50, R18, w600k-MBF) and LVFace-B (Glint360K,
455MB) were compared. r50 is excluded from the training/held-out comparison
below; its gallery has roughly 30% fewer reference images per actor than the
other three on the identical source photos, which confounds a direct score
comparison (see [the full experiment log](model-bakeoff-2026-07.md) for detail). It
remains in the calibration comparison, which does not depend on the gallery
image count.

## First signal: calibration curves

Each gallery carries a fitted Platt sigmoid `P(match | cosine similarity) =
σ(a·sim + b)`, stored directly in the gallery HDF5
([`src/gallery/gallery_calibration.hpp`](https://REPOLINK/src/gallery/gallery_calibration.hpp)).
This is a property of the embedding space alone, computed from intra- and
inter-actor reference-image pairs with no tracking or scene logic involved,
so it is a clean first read on discriminative power before running a
benchmark.

![Calibrated P(match|similarity) for all four models](assets/images/calibration_curves.png)

| model | a (steepness) | boundary at P=0.5 |
|---|---|---|
| LVFace-B Glint360K | 17.7 | sim 0.228 |
| ArcFace w600k-MBF | 16.2 | sim 0.267 |
| ArcFace w600k-R50 | 15.4 | sim 0.301 |
| ArcFace R18 | 15.3 | sim 0.309 |

LVFace has both the steepest transition and the lowest decision boundary,
separating same-actor from different-actor reference pairs more confidently
at a lower similarity than any ArcFace variant.

## Second signal: held-out F1

Each model's own tuned `full_exp` config, replayed against the 5 films the
optimizer never saw and scored the same way:

| film | LVFace F1 | mbf F1 | r18 F1 |
|---|---|---|---|
| Benny & Joon | 83.0% | 78.5% | 77.1% |
| Lovelace | 77.5% | 73.7% | 72.2% |
| Valerian and the City of a Thousand Planets | 74.1% | 70.2% | 71.0% |
| Downton Abbey: A New Era | 56.2% | 55.0% | 53.0% |
| The Many Saints of Newark | 46.3% | 44.5% | 42.1% |
| **macro average** | **67.4%** | **64.4%** | **63.1%** |

LVFace scores highest on all 5 held-out films; the ranking never flips
between models. Total misID count across the 5 films: LVFace 1032, mbf
2197, r18 1224. LVFace has less than half mbf's misID total and still
scores higher on every film.

Held-out results are stronger evidence than training results, because
training numbers can reflect what the optimizer was tuned to fit rather
than general performance. On training data, the ordering is not as clean:

| film | LVFace F1 | mbf F1 | r18 F1 | best |
|---|---|---|---|---|
| Café Society | 68.1% | 62.2% | 60.1% | LVFace |
| Lord of War | 75.6% | 77.2% | 75.6% | mbf |
| Scarface | 71.5% | 68.6% | 64.1% | LVFace |
| Sound of Metal | 78.8% | 76.5% | 71.6% | LVFace |

mbf beats LVFace on Lord of War (77.2% vs 75.6%), the only film in either
table where LVFace does not score highest. LVFace's training-set macro
average (75.3%, see [the full experiment log](model-bakeoff-2026-07.md)) is not a
uniform win across every film it contributes to; the held-out result, where
LVFace wins all 5 films outright, is the stronger claim.

This reverses an earlier, superseded benchmarking pass that used a
scene-union metric and found the three models statistically
indistinguishable (around 85% each), concluding LVFace was not worth its
size. That metric masked out-of-cast false positives behind a
gallery-intersect-cast recall filter; the per-second metric used here does
not.

## Full training-matrix picture

![All 12 combos ranked by training-set F1](assets/images/rep4_matrix_f1.png)

Best full-gallery combo per model (all three are `full_exp`), from the
training matrix in [the full experiment log](model-bakeoff-2026-07.md):

| model | F1 | P | R | misID |
|---|---|---|---|---|
| LVFace-B Glint360K | 75.3% | 89.7% | 65.4% | 232 |
| ArcFace w600k-MBF | 72.0% | 87.7% | 61.4% | 240 |
| ArcFace R18 | 69.1% | 87.6% | 57.7% | 242 |

LVFace leads within both the restricted and full gallery modes, visible
directly in the chart above without reading the table. The three models'
misID counts on the full gallery are nearly identical (232/240/242); LVFace's
lead here is a precision-and-recall lead, not a misID one.

## Operational note

Switching the default embedder is not a config change alone; the gallery
is model-specific, since embeddings from different models are not
comparable. Any existing gallery built against a different model must be
rebuilt from source images before the new default takes effect.
[`scripts/optimizer/reembed_gallery.py`](https://REPOLINK/scripts/optimizer/reembed_gallery.py)
does this from a reference gallery's cached source images without
re-downloading anything.
