dtourolle
b1efefac6f
- experiment_charts.py generates 4 figures from experiments/ artifacts: held-out per-film F1, 16-combo ranking, DE search landscape, and the Downton detector-vs-tracker ghost timeline (replaces the blank title-card screenshot) - new frames: 19-correct wedding shot (success case), Many Saints ghost-vs-unknown frame (three error classes in one image) - rename rep4-optimizer-results.md -> model-bakeoff.md; rep4 kept only as the on-disk artifact prefix, explained once - repo file references are now links via https://REPOLINK/<path> placeholders; build_site.sh pins them to the HEAD commit's raw URLs and fails the build if a linked path doesn't exist at HEAD - drop references to removed scripts (scene_score.py, score_config.py) and to session-memory names; mark artifact-registry paths with their pull commands - commit readme_example.jpg + pipeline_topology.svg so README renders on the plain Gitea repo view - deploy_pages.sh: push built site/ to the gitea-pages branch
3.4 KiB
Which embedding model is best?
Four candidates went into the bake-off: three ArcFace variants (w600k-R50,
R18, w600k-MBF) and LVFace-B (Glint360K), a Vision-Transformer embedder that's
a drop-in replacement for ArcFace's [N,3,112,112] input / 512-d output. The
open question: is LVFace (455MB) actually better, or just the biggest?
First signal: calibration curves
Each gallery carries a fitted Platt sigmoid P(match | cosine similarity) = σ(a·sim + b), embedded directly in the gallery's HDF5 file
(src/gallery/gallery_calibration.hpp). This is a property of the embedding
space alone — computed from intra/inter-actor reference-image pairs, no
tracking or scene logic involved — so it's a clean first read on discriminative
power before running a single benchmark.
| model | a (steepness) |
boundary at P=0.5 |
|---|---|---|
| LVFace-B Glint360K | 17.7 | sim 0.228 |
| ArcFace w600k-MBF | 16.2 | sim 0.267 |
| ArcFace w600k-R50 | 15.4 | sim 0.301 |
| ArcFace R18 | 15.3 | sim 0.309 |
LVFace has both the steepest transition and the lowest decision boundary — it separates same-actor from different-actor reference pairs more confidently, at a lower similarity threshold, than any ArcFace variant. That's a genuine head start before the tracking/scoring pipeline is even involved.
Second signal: F1 on the actual benchmark
Best full-gallery (no cast-restriction) result per model, from the 16-combo bake-off matrix (full experiment log):
| model | F1 | P | R | misID |
|---|---|---|---|---|
| LVFace-B Glint360K | 75.3% | 89.7% | 65.4% | 232 |
| ArcFace w600k-MBF | 74.2% | 87.4% | 64.4% | 57 |
| ArcFace R18 | 69.1% | 87.6% | 57.7% | 242 |
| ArcFace w600k-R50 | 68.5% | 94.0% | 54.1% | 150 |
The full 16-combo picture makes the model ordering visible at a glance — LVFace (yellow) tops both the restricted and full columns, and R18 (green) props up the bottom of the full-gallery ranking:
LVFace wins outright, with the highest recall of any full-mode combo. This reverses an earlier conclusion from a prior (superseded) benchmarking pass using a scene-union metric, which found the three models statistically indistinguishable (~85% each) and concluded LVFace wasn't worth its size — that metric hid out-of-cast false positives behind a gallery∩cast recall mask (see the prior optimizer round); the per-second metric used here does not.
Held-out validation (5 films never seen by the optimizer) confirms LVFace's lead holds up out of sample — see the LVFace deep dive for the full breakdown, including where it fails.
Caveat: model choice is an operational change
Switching the default embedder isn't just flipping a config value — the
gallery itself is model-specific (embeddings from different models aren't
comparable), so any existing gallery built against ArcFace w600k-R50 needs to
be rebuilt from source images against LVFace before the new default takes
effect. scripts/optimizer/reembed_gallery.py
does this from a reference gallery's cached source images without
re-downloading anything.