docs: four focused findings pages (best model, gallery scope, expansion, deep dive)
Splits the rep4 write-up's key findings into their own linkable pages: - best-model.md: calibration curves first (discriminative power, independent of any threshold), then F1 on the benchmark — LVFace-B Glint360K wins both. - gallery-scope.md: whole vs. cast-restricted gallery, isolated from model and expansion choice — restriction wins on every axis, but isn't a shipped runtime feature yet. - pose-expansion.md: the training-set expand_gallery effect, and the held-out replication attempt that found it doesn't reproduce (5 films, 2 models, after catching and fixing a replay-timeout truncation bug and a bbox first-match-instead-of-best-match bug in the comparison harness itself). An honest null result, with the methodology errors documented since they're exactly the kind that manufacture a false "it works!" finding. - lvface-deep-dive.md: the winning model's held-out generalization gap, its two failure modes (frozen-bbox ghost tracks), and a verified case (cross- checked against Jellyfin's independent cast metadata) where LVFace correctly identified an actor that X-Ray's ground truth failed to credit. Adds a "report-highlights" artifact-registry package (scripts/artifacts/ push_artifacts.sh, pull_artifacts.sh) for hand-picked illustrative frames that aren't reproducible via the automated best/worst montage selection, and wires pulling it into scripts/docs/build_site.sh.
This commit is contained in:
@@ -0,0 +1,98 @@
|
||||
# Deep dive: LVFace-B Glint360K
|
||||
|
||||
LVFace won the model bake-off (see `best-model.md`) and is the shipped default
|
||||
embedder. This page is the honest accounting of how it actually performs —
|
||||
including where it's wrong, and one case where the ground truth itself is
|
||||
wrong and LVFace is right.
|
||||
|
||||
## Training vs. held-out: the generalization gap
|
||||
|
||||
The shipped config (`prob_threshold=0.754, anneal_sec=35.54,
|
||||
extinction_sec=57.43, expand_gallery=true`) was tuned against 4 films. Scored
|
||||
against the 5 films the optimizer never saw:
|
||||
|
||||
| film | F1 | P | R | TPI | FPI | misid | FN |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| Benny & Joon | 83.0% | 89.1% | 77.7% | 15125 | 1846 | 0 | 4337 |
|
||||
| Lovelace | 77.5% | 90.3% | 67.9% | 14990 | 1085 | 58 | 7085 |
|
||||
| Valerian and the City of a Thousand Planets | 74.1% | 97.1% | 60.0% | 18663 | 548 | 0 | 12467 |
|
||||
| Downton Abbey: A New Era | 56.2% | 97.8% | 39.4% | 52027 | 1173 | 0 | 80084 |
|
||||
| **The Many Saints of Newark** | **46.3%** | **54.7%** | 40.1% | 15922 | 4394 | **974** | 23791 |
|
||||
| **macro average** | **67.4%** | 85.8% | 57.0% | | | | |
|
||||
|
||||
**67.4% held-out vs. 75.3% on training** — an ~8pp drop, and a **37pp spread
|
||||
between the best and worst held-out film**. The config does not generalize
|
||||
uniformly; two films are outright failure cases, for two different reasons.
|
||||
|
||||
## Failure mode 1: frozen-bbox "ghost tracks"
|
||||
|
||||
Both Many Saints of Newark (974 misIDs) and Downton Abbey (FN=80084, the worst
|
||||
recall of the five) trace to the same root cause, verified directly against
|
||||
the raw per-frame stream and the HDF5 dump's own detection counts — not
|
||||
inferred from the score alone.
|
||||
|
||||
|
||||
|
||||
At this second, three of the four labeled boxes ("Jon Bernthal", "Joey Diaz",
|
||||
"Billy Magnussen") sit over empty background — a blurred wall, hanging
|
||||
plates — with no face in them. The real face in frame carries a second,
|
||||
colliding label from another frozen box.
|
||||
|
||||
|
||||
|
||||
This is the starkest case: **15 actors named, all wrong, over a completely
|
||||
blank closing title card.** Confirmed against the dump directly: `face_count`
|
||||
is 0 from this point onward (no detector output at all), yet the same 15
|
||||
identities keep appearing with the *exact same bounding box, unchanged to the
|
||||
pixel*, for 57+ consecutive seconds.
|
||||
|
||||
This is `SceneTrackerFunc::active_[actor_idx].last_bbox`
|
||||
(`src/nodes/scene_tracker_node.hpp`) being re-emitted unchanged — the
|
||||
extinction state machine working exactly as coded, not a bug. The film cuts
|
||||
from a packed group shot straight into 40+ seconds of blank titles/credits,
|
||||
and `extinction_sec=57.4` is comfortably long enough to bridge that entire gap
|
||||
without expiring, so the tracker faithfully reports "last known position" for
|
||||
a cast that is no longer on screen at all. `extinction_sec` was tuned toward
|
||||
long windows specifically because they bridge real gaps (occlusion, a turned
|
||||
face) in most training footage — this is the cost side of that trade,
|
||||
surfacing only when a film has a long enough faceless stretch to expose it.
|
||||
|
||||
## Failure mode 2: a genuine misID (for contrast)
|
||||
|
||||
Not every held-out failure is a ghost. This is a real face, correctly
|
||||
detected, confidently misidentified:
|
||||
|
||||
*(same many_saints_ghost_fpi.jpg frame above also shows Leslie Odom Jr.'s box
|
||||
carrying a second, colliding "Michael Gandolfini" label — two real tracks'
|
||||
frozen positions happening to overlap, not a detection error.)*
|
||||
|
||||
## Where LVFace beat X-Ray
|
||||
|
||||
Not every "misID" is actually wrong. `second_score.py` counts a name as a true
|
||||
out-of-cast misID whenever the named actor isn't in X-Ray's credited cast list
|
||||
for the film at all — but X-Ray's cast list is itself incomplete.
|
||||
|
||||
|
||||
|
||||
Germar Terrell Gardner — a real, clean, high-confidence detection — is counted
|
||||
as a misID here because he doesn't appear in X-Ray's `people.csv` for The Many
|
||||
Saints of Newark at all. But Jellyfin's independent cast metadata *does* credit
|
||||
him for this exact film (cross-checked via `experiments/manifests/
|
||||
jellyfin_casts.json`, a completely separate data source from X-Ray). This
|
||||
isn't a lookalike error or a gallery mixup — it's the pipeline correctly
|
||||
recognising a real cast member that one ground-truth source happened to omit.
|
||||
|
||||
This doesn't mean every flagged misID is secretly correct — Many Saints'
|
||||
974-count total is still overwhelmingly the frozen-bbox failure mode above,
|
||||
not uncredited-but-real cameos. But it's a reminder that the X-Ray corpus is a
|
||||
convenient, large-scale ground truth, not a perfect one, and the "misID" number
|
||||
in any of these tables has some irreducible noise floor from ground-truth gaps
|
||||
in the other direction too.
|
||||
|
||||
## Summary
|
||||
|
||||
LVFace is the right default: it wins the model comparison outright, and its
|
||||
failures are traceable, understood, and mostly attributable to one tunable
|
||||
knob (`extinction_sec`) rather than the embedder itself. The held-out
|
||||
generalization gap (75.3% → 67.4%) is real and should be treated as the honest
|
||||
expected performance, not the training-set number.
|
||||
Reference in New Issue
Block a user