scene-actor-extraction/docs/lvface-deep-dive.md

# Deep dive: LVFace-B Glint360K

LVFace won the model bake-off (see [Which model is best?](best-model.md)) and is
the shipped default embedder. This page is the honest accounting of how it
actually performs — what a good second looks like, where it's wrong and *why*,
and one case where the ground truth itself is wrong and LVFace is right.

## What good looks like

Before the failure analysis, the ceiling. This is a single sampled second from
Downton Abbey's wedding scene — a packed, hat-heavy, period-costume group shot,
about as hostile as ensemble framing gets:

![19 correct identifications in one wedding shot, Downton Abbey: A New Era](assets/images/downton_wedding_19_correct.jpg)
*Frame `downton_abbey/best/best_t00127.jpg` from the `montage-frames` artifact
package (`scripts/artifacts/pull_artifacts.sh montage-frames
Downton_Abbey__A_New_Era`) — green = identified, blue = detected but unknown.*

**Nineteen named faces in one frame, all nineteen correct** — Jim Carter half
behind a flower arrangement, Penelope Wilton at a three-quarter turn, Lesley
Nicol under a hat brim. The blue "unknown" boxes are the honest cases: faces the
detector found but the matcher declined to name rather than guess. The one miss
at this second is Maggie Smith — not on screen in this framing, but X-Ray marks
her present for the scene. That distinction (on-screen face vs. scene-level
ground truth) sets up everything below.

## Training vs. held-out: the generalization gap

The shipped config (`prob_threshold=0.754, anneal_sec=35.54,
extinction_sec=57.43, expand_gallery=true`) was tuned against 4 films. Scored
against the 5 films the optimizer never saw:

![Held-out per-film F1 vs. the training-set fit](assets/images/holdout_f1_by_film.png)

| film | F1 | P | R | TPI | FPI | misid | FN |
|---|---|---|---|---|---|---|---|
| Benny & Joon | 83.0% | 89.1% | 77.7% | 15125 | 1846 | 0 | 4337 |
| Lovelace | 77.5% | 90.3% | 67.9% | 14990 | 1085 | 58 | 7085 |
| Valerian and the City of a Thousand Planets | 74.1% | 97.1% | 60.0% | 18663 | 548 | 0 | 12467 |
| Downton Abbey: A New Era | 56.2% | 97.8% | 39.4% | 52027 | 1173 | 0 | 80084 |
| **The Many Saints of Newark** | **46.3%** | **54.7%** | 40.1% | 15922 | 4394 | **974** | 23791 |
| **macro average** | **67.4%** | 85.8% | 57.0% | | | | |

**67.4% held-out vs. 75.3% on training** — an ~8pp drop, and a **37pp spread
between the best and worst held-out film**. The config does not generalize
uniformly; two films are outright failure cases, for reasons that turn out to
be one mechanism.

## The failure mode: frozen-bbox "ghost tracks"

Both Many Saints of Newark (974 misIDs) and Downton Abbey (FN=80084, the worst
recall of the five) trace to the same root cause, verified directly against
the raw per-frame stream and the HDF5 dump's own detection counts — not
inferred from the score alone.

![Frozen ghost boxes over background, The Many Saints of Newark](assets/images/many_saints_ghost_fpi.jpg)

At this second, three of the four labeled boxes ("Jon Bernthal", "Joey Diaz",
"Billy Magnussen") sit over empty background — a blurred wall, hanging
plates — with no face in them. The real face in frame carries a second,
colliding label from another frozen box. And it isn't an isolated second — the
same signature recurs throughout the film:

![Ghost labels over a staircase while real faces stay honest unknowns](assets/images/many_saints_ghosts_vs_unknowns.jpg)
*Frame `many_saints_intervals/w002_worst_t01382.jpg`, same artifact package —
one frame, three distinct error classes.*

This frame is worth reading closely, because it separates three things that a
single aggregate F1 number smears together. The two green labels ("Jon Bernthal
100%", "Michela De Rossi 100%") float over a staircase and a policeman's back —
frozen boxes from a previous shot, reported at full confidence. Meanwhile the
two *real* frontal faces in frame get honest blue "unknown 0%" boxes (they're
uncredited day-players with no gallery reference — the
[gallery coverage gap](gallery-scope.md)), and three more people simply face
away from camera, invisible to any face detector but still "present" in X-Ray's
scene-level ground truth. Precision failure, gallery-coverage failure, and the
face-vs-presence ceiling — one frame.

### The mechanism, measured

The starkest case is Downton Abbey's hard cut from a packed group shot into a
long blank credits sequence. Plotting the detector's per-second `face_count`
(from the dump HDF5, independent of the tracker) against what the tracker
reports makes the failure legible at a glance:

![Detector vs. tracker through Downton Abbey's cut to credits](assets/images/downton_ghost_timeline.png)

From the cut onward the detector sees **zero faces for nearly a minute** — and
the tracker keeps reporting the last group shot's 15 identities the entire
time, each with the *exact same bounding box, unchanged to the pixel* (verified
for Hugh Bonneville: `(1743.2, 0.0, 171.3, 317.8)` at every sampled second for
57+ seconds). The staircase decay at the right edge is the extinction window
finally expiring, actor by actor.

This is `SceneTrackerFunc::active_[actor_idx].last_bbox`
([`src/nodes/scene_tracker_node.hpp`](https://REPOLINK/src/nodes/scene_tracker_node.hpp))
being re-emitted unchanged — the
extinction state machine working exactly as coded, not a bug. `extinction_sec`
was tuned to 57.4s specifically because long windows bridge real gaps
(occlusion, a turned face) in most footage; a hard cut into long faceless
footage is the one case where that same bridging manufactures ghosts, and the
training films never contained one long enough to punish it. The optimizer
"discovered" the plateau at the top of its search range for a reason that only
generalizes to films that never go faceless for a minute.

## Where LVFace beat X-Ray

Not every flagged "misID" is actually wrong.
[`scripts/optimizer/second_score.py`](https://REPOLINK/scripts/optimizer/second_score.py)
counts a name as a true out-of-cast misID whenever the named actor isn't in
X-Ray's credited cast list for the film at all — but X-Ray's cast list is
itself incomplete.

![LVFace correctly identifies Germar Terrell Gardner, uncredited by X-Ray](assets/images/germar_beats_xray.jpg)

Germar Terrell Gardner — a real, clean, high-confidence detection — is counted
as a misID here because he doesn't appear in X-Ray's `people.csv` for The Many
Saints of Newark at all. But Jellyfin's independent cast metadata *does* credit
him for this exact film (cross-checked via
`experiments/manifests/jellyfin_casts.json` from the `experiment-data` artifact
package, a completely separate data source from X-Ray). This isn't a lookalike error or a gallery mixup — it's the
pipeline correctly recognising a real cast member that one ground-truth source
happened to omit.

This doesn't mean every flagged misID is secretly correct — Many Saints'
974-count total is still overwhelmingly the frozen-bbox failure mode above,
not uncredited-but-real cameos. But it's a reminder that the X-Ray corpus is a
convenient, large-scale ground truth, not a perfect one, and the "misID" number
in any of these tables has some irreducible noise floor from ground-truth gaps
in the other direction too.

## Summary

LVFace is the right default: it wins the model comparison outright, it can name
19 faces correctly in a single hostile group shot, and its failures are
traceable, understood, and mostly attributable to one tunable knob
(`extinction_sec`) rather than the embedder itself. The held-out
generalization gap (75.3% → 67.4%) is real and should be treated as the honest
expected performance, not the training-set number.