dtourolle
ea922356f1
The July report (4-model ArcFace/LVFace bake-off, pre-opencv5 framework, 3-film training + held-out validation) is superseded by the opencv5 build: single-model LVFace-B, a 6-knob DE sweep over all 9 films, flood-fill presence, and the registry/decode fixes. Rather than overwrite it, archive it date-suffixed and start the current report fresh. - Rename the six July result pages to *-2026-07.md, rewrite their intra-archive cross-links, and add an "Archived (July 2026)" banner to each. - mkdocs nav: current report at top, the July set under an Archive section. - New docs/methodology.md for the opencv5 run: corrects the withdrawn anneal_sec/extinction_sec presence bridging (windows are now [first_seen, last_seen], AR-012/013), documents the two presence modes (track_extent / flood), and records that every eval scores all 9 films. The current experiment log (model-bakeoff.md) and Home rewrite land once the DE sweep converges and the final optimum is known.
4.8 KiB
Archived (July 2026). This report covers the pre-opencv5 framework and the 4-model ArcFace/LVFace bake-off. It is superseded by the current experiment log for the opencv5 build. Kept for provenance; the numbers here are historical.
Which embedding model is best?
Three ArcFace variants (w600k-R50, R18, w600k-MBF) and LVFace-B (Glint360K, 455MB) were compared. r50 is excluded from the training/held-out comparison below; its gallery has roughly 30% fewer reference images per actor than the other three on the identical source photos, which confounds a direct score comparison (see the full experiment log for detail). It remains in the calibration comparison, which does not depend on the gallery image count.
First signal: calibration curves
Each gallery carries a fitted Platt sigmoid P(match | cosine similarity) = σ(a·sim + b), stored directly in the gallery HDF5
(src/gallery/gallery_calibration.hpp).
This is a property of the embedding space alone, computed from intra- and
inter-actor reference-image pairs with no tracking or scene logic involved,
so it is a clean first read on discriminative power before running a
benchmark.
| model | a (steepness) | boundary at P=0.5 |
|---|---|---|
| LVFace-B Glint360K | 17.7 | sim 0.228 |
| ArcFace w600k-MBF | 16.2 | sim 0.267 |
| ArcFace w600k-R50 | 15.4 | sim 0.301 |
| ArcFace R18 | 15.3 | sim 0.309 |
LVFace has both the steepest transition and the lowest decision boundary, separating same-actor from different-actor reference pairs more confidently at a lower similarity than any ArcFace variant.
Second signal: held-out F1
Each model's own tuned full_exp config, replayed against the 5 films the
optimizer never saw and scored the same way:
| film | LVFace F1 | mbf F1 | r18 F1 |
|---|---|---|---|
| Benny & Joon | 83.0% | 78.5% | 77.1% |
| Lovelace | 77.5% | 73.7% | 72.2% |
| Valerian and the City of a Thousand Planets | 74.1% | 70.2% | 71.0% |
| Downton Abbey: A New Era | 56.2% | 55.0% | 53.0% |
| The Many Saints of Newark | 46.3% | 44.5% | 42.1% |
| macro average | 67.4% | 64.4% | 63.1% |
LVFace scores highest on all 5 held-out films; the ranking never flips between models. Total misID count across the 5 films: LVFace 1032, mbf 2197, r18 1224. LVFace has less than half mbf's misID total and still scores higher on every film.
Held-out results are stronger evidence than training results, because training numbers can reflect what the optimizer was tuned to fit rather than general performance. On training data, the ordering is not as clean:
| film | LVFace F1 | mbf F1 | r18 F1 | best |
|---|---|---|---|---|
| Café Society | 68.1% | 62.2% | 60.1% | LVFace |
| Lord of War | 75.6% | 77.2% | 75.6% | mbf |
| Scarface | 71.5% | 68.6% | 64.1% | LVFace |
| Sound of Metal | 78.8% | 76.5% | 71.6% | LVFace |
mbf beats LVFace on Lord of War (77.2% vs 75.6%), the only film in either table where LVFace does not score highest. LVFace's training-set macro average (75.3%, see the full experiment log) is not a uniform win across every film it contributes to; the held-out result, where LVFace wins all 5 films outright, is the stronger claim.
This reverses an earlier, superseded benchmarking pass that used a scene-union metric and found the three models statistically indistinguishable (around 85% each), concluding LVFace was not worth its size. That metric masked out-of-cast false positives behind a gallery-intersect-cast recall filter; the per-second metric used here does not.
Full training-matrix picture
Best full-gallery combo per model (all three are full_exp), from the
training matrix in the full experiment log:
| model | F1 | P | R | misID |
|---|---|---|---|---|
| LVFace-B Glint360K | 75.3% | 89.7% | 65.4% | 232 |
| ArcFace w600k-MBF | 72.0% | 87.7% | 61.4% | 240 |
| ArcFace R18 | 69.1% | 87.6% | 57.7% | 242 |
LVFace leads within both the restricted and full gallery modes, visible directly in the chart above without reading the table. The three models' misID counts on the full gallery are nearly identical (232/240/242); LVFace's lead here is a precision-and-recall lead, not a misID one.
Operational note
Switching the default embedder is not a config change alone; the gallery
is model-specific, since embeddings from different models are not
comparable. Any existing gallery built against a different model must be
rebuilt from source images before the new default takes effect.
scripts/optimizer/reembed_gallery.py
does this from a reference gallery's cached source images without
re-downloading anything.