dtourolle
ea922356f1
The July report (4-model ArcFace/LVFace bake-off, pre-opencv5 framework, 3-film training + held-out validation) is superseded by the opencv5 build: single-model LVFace-B, a 6-knob DE sweep over all 9 films, flood-fill presence, and the registry/decode fixes. Rather than overwrite it, archive it date-suffixed and start the current report fresh. - Rename the six July result pages to *-2026-07.md, rewrite their intra-archive cross-links, and add an "Archived (July 2026)" banner to each. - mkdocs nav: current report at top, the July set under an Archive section. - New docs/methodology.md for the opencv5 run: corrects the withdrawn anneal_sec/extinction_sec presence bridging (windows are now [first_seen, last_seen], AR-012/013), documents the two presence modes (track_extent / flood), and records that every eval scores all 9 films. The current experiment log (model-bakeoff.md) and Home rewrite land once the DE sweep converges and the final optimum is known.
106 lines
4.8 KiB
Markdown
106 lines
4.8 KiB
Markdown
> **Archived (July 2026).** This report covers the pre-opencv5 framework and the 4-model ArcFace/LVFace bake-off. It is superseded by the current [experiment log](model-bakeoff.md) for the opencv5 build. Kept for provenance; the numbers here are historical.
|
||
|
||
# Which embedding model is best?
|
||
|
||
Three ArcFace variants (w600k-R50, R18, w600k-MBF) and LVFace-B (Glint360K,
|
||
455MB) were compared. r50 is excluded from the training/held-out comparison
|
||
below; its gallery has roughly 30% fewer reference images per actor than the
|
||
other three on the identical source photos, which confounds a direct score
|
||
comparison (see [the full experiment log](model-bakeoff-2026-07.md) for detail). It
|
||
remains in the calibration comparison, which does not depend on the gallery
|
||
image count.
|
||
|
||
## First signal: calibration curves
|
||
|
||
Each gallery carries a fitted Platt sigmoid `P(match | cosine similarity) =
|
||
σ(a·sim + b)`, stored directly in the gallery HDF5
|
||
([`src/gallery/gallery_calibration.hpp`](https://REPOLINK/src/gallery/gallery_calibration.hpp)).
|
||
This is a property of the embedding space alone, computed from intra- and
|
||
inter-actor reference-image pairs with no tracking or scene logic involved,
|
||
so it is a clean first read on discriminative power before running a
|
||
benchmark.
|
||
|
||
|
||
|
||
| model | a (steepness) | boundary at P=0.5 |
|
||
|---|---|---|
|
||
| LVFace-B Glint360K | 17.7 | sim 0.228 |
|
||
| ArcFace w600k-MBF | 16.2 | sim 0.267 |
|
||
| ArcFace w600k-R50 | 15.4 | sim 0.301 |
|
||
| ArcFace R18 | 15.3 | sim 0.309 |
|
||
|
||
LVFace has both the steepest transition and the lowest decision boundary,
|
||
separating same-actor from different-actor reference pairs more confidently
|
||
at a lower similarity than any ArcFace variant.
|
||
|
||
## Second signal: held-out F1
|
||
|
||
Each model's own tuned `full_exp` config, replayed against the 5 films the
|
||
optimizer never saw and scored the same way:
|
||
|
||
| film | LVFace F1 | mbf F1 | r18 F1 |
|
||
|---|---|---|---|
|
||
| Benny & Joon | 83.0% | 78.5% | 77.1% |
|
||
| Lovelace | 77.5% | 73.7% | 72.2% |
|
||
| Valerian and the City of a Thousand Planets | 74.1% | 70.2% | 71.0% |
|
||
| Downton Abbey: A New Era | 56.2% | 55.0% | 53.0% |
|
||
| The Many Saints of Newark | 46.3% | 44.5% | 42.1% |
|
||
| **macro average** | **67.4%** | **64.4%** | **63.1%** |
|
||
|
||
LVFace scores highest on all 5 held-out films; the ranking never flips
|
||
between models. Total misID count across the 5 films: LVFace 1032, mbf
|
||
2197, r18 1224. LVFace has less than half mbf's misID total and still
|
||
scores higher on every film.
|
||
|
||
Held-out results are stronger evidence than training results, because
|
||
training numbers can reflect what the optimizer was tuned to fit rather
|
||
than general performance. On training data, the ordering is not as clean:
|
||
|
||
| film | LVFace F1 | mbf F1 | r18 F1 | best |
|
||
|---|---|---|---|---|
|
||
| Café Society | 68.1% | 62.2% | 60.1% | LVFace |
|
||
| Lord of War | 75.6% | 77.2% | 75.6% | mbf |
|
||
| Scarface | 71.5% | 68.6% | 64.1% | LVFace |
|
||
| Sound of Metal | 78.8% | 76.5% | 71.6% | LVFace |
|
||
|
||
mbf beats LVFace on Lord of War (77.2% vs 75.6%), the only film in either
|
||
table where LVFace does not score highest. LVFace's training-set macro
|
||
average (75.3%, see [the full experiment log](model-bakeoff-2026-07.md)) is not a
|
||
uniform win across every film it contributes to; the held-out result, where
|
||
LVFace wins all 5 films outright, is the stronger claim.
|
||
|
||
This reverses an earlier, superseded benchmarking pass that used a
|
||
scene-union metric and found the three models statistically
|
||
indistinguishable (around 85% each), concluding LVFace was not worth its
|
||
size. That metric masked out-of-cast false positives behind a
|
||
gallery-intersect-cast recall filter; the per-second metric used here does
|
||
not.
|
||
|
||
## Full training-matrix picture
|
||
|
||
|
||
|
||
Best full-gallery combo per model (all three are `full_exp`), from the
|
||
training matrix in [the full experiment log](model-bakeoff-2026-07.md):
|
||
|
||
| model | F1 | P | R | misID |
|
||
|---|---|---|---|---|
|
||
| LVFace-B Glint360K | 75.3% | 89.7% | 65.4% | 232 |
|
||
| ArcFace w600k-MBF | 72.0% | 87.7% | 61.4% | 240 |
|
||
| ArcFace R18 | 69.1% | 87.6% | 57.7% | 242 |
|
||
|
||
LVFace leads within both the restricted and full gallery modes, visible
|
||
directly in the chart above without reading the table. The three models'
|
||
misID counts on the full gallery are nearly identical (232/240/242); LVFace's
|
||
lead here is a precision-and-recall lead, not a misID one.
|
||
|
||
## Operational note
|
||
|
||
Switching the default embedder is not a config change alone; the gallery
|
||
is model-specific, since embeddings from different models are not
|
||
comparable. Any existing gallery built against a different model must be
|
||
rebuilt from source images before the new default takes effect.
|
||
[`scripts/optimizer/reembed_gallery.py`](https://REPOLINK/scripts/optimizer/reembed_gallery.py)
|
||
does this from a reference gallery's cached source images without
|
||
re-downloading anything.
|