docs: four focused findings pages (best model, gallery scope, expansion, deep dive)

Splits the rep4 write-up's key findings into their own linkable pages:
- best-model.md: calibration curves first (discriminative power, independent
  of any threshold), then F1 on the benchmark — LVFace-B Glint360K wins both.
- gallery-scope.md: whole vs. cast-restricted gallery, isolated from model and
  expansion choice — restriction wins on every axis, but isn't a shipped
  runtime feature yet.
- pose-expansion.md: the training-set expand_gallery effect, and the held-out
  replication attempt that found it doesn't reproduce (5 films, 2 models,
  after catching and fixing a replay-timeout truncation bug and a bbox
  first-match-instead-of-best-match bug in the comparison harness itself). An
  honest null result, with the methodology errors documented since they're
  exactly the kind that manufacture a false "it works!" finding.
- lvface-deep-dive.md: the winning model's held-out generalization gap, its
  two failure modes (frozen-bbox ghost tracks), and a verified case (cross-
  checked against Jellyfin's independent cast metadata) where LVFace
  correctly identified an actor that X-Ray's ground truth failed to credit.

Adds a "report-highlights" artifact-registry package (scripts/artifacts/
push_artifacts.sh, pull_artifacts.sh) for hand-picked illustrative frames that
aren't reproducible via the automated best/worst montage selection, and wires
pulling it into scripts/docs/build_site.sh.
This commit is contained in:
2026-07-19 19:40:19 +02:00
parent d340da755a
commit 4925443e56
9 changed files with 384 additions and 13 deletions
+61
View File
@@ -0,0 +1,61 @@
# Which embedding model is best?
Four candidates went into the bake-off: three ArcFace variants (w600k-R50,
R18, w600k-MBF) and LVFace-B (Glint360K), a Vision-Transformer embedder that's
a drop-in replacement for ArcFace's `[N,3,112,112]` input / 512-d output. The
open question: is LVFace (455MB) actually better, or just the biggest?
## First signal: calibration curves
Each gallery carries a fitted Platt sigmoid `P(match | cosine similarity) =
σ(a·sim + b)`, embedded directly in the gallery's HDF5 file
(`src/gallery/gallery_calibration.hpp`). This is a property of the embedding
space alone — computed from intra/inter-actor reference-image pairs, no
tracking or scene logic involved — so it's a clean first read on discriminative
power before running a single benchmark.
![Calibrated P(match|similarity) for all four models](assets/images/calibration_curves.png)
| model | `a` (steepness) | boundary at P=0.5 |
|---|---|---|
| **LVFace-B Glint360K** | **17.7** | **sim 0.228** |
| ArcFace w600k-MBF | 16.2 | sim 0.267 |
| ArcFace w600k-R50 | 15.4 | sim 0.301 |
| ArcFace R18 | 15.3 | sim 0.309 |
LVFace has both the steepest transition and the lowest decision boundary — it
separates same-actor from different-actor reference pairs more confidently, at
a *lower* similarity threshold, than any ArcFace variant. That's a genuine
head start before the tracking/scoring pipeline is even involved.
## Second signal: F1 on the actual benchmark
Best full-gallery (no cast-restriction) result per model, from the 16-combo
rep4 matrix (`rep4-optimizer-results.md`):
| model | F1 | P | R | misID |
|---|---|---|---|---|
| **LVFace-B Glint360K** | **75.3%** | 89.7% | **65.4%** | 232 |
| ArcFace w600k-MBF | 74.2% | 87.4% | 64.4% | 57 |
| ArcFace R18 | 69.1% | 87.6% | 57.7% | 242 |
| ArcFace w600k-R50 | 68.5% | 94.0% | 54.1% | 150 |
LVFace wins outright, with the highest recall of any full-mode combo. This
reverses an earlier conclusion from a prior (superseded) benchmarking pass
using a scene-union metric, which found the three models statistically
indistinguishable (~85% each) and concluded LVFace wasn't worth its size — that
metric hid out-of-cast false positives behind a gallery∩cast recall mask (see
`optimizer-experiments.md`); the per-second metric used here does not.
Held-out validation (5 films never seen by the optimizer) confirms LVFace's
lead holds up out of sample — see the deep-dive page for the full breakdown,
including where it fails.
## Caveat: model choice is an operational change
Switching the default embedder isn't just flipping a config value — the
gallery itself is model-specific (embeddings from different models aren't
comparable), so any existing gallery built against ArcFace w600k-R50 needs to
be rebuilt from source images against LVFace before the new default takes
effect. `scripts/optimizer/reembed_gallery.py` does this from a reference
gallery's cached source images without re-downloading anything.