scene-actor-extraction/docs/lvface-deep-dive.md

Deep dive: LVFace-B Glint360K

LVFace won the model bake-off (see Which model is best?) and is the shipped default embedder. This page is the honest accounting of how it actually performs — what a good second looks like, where it's wrong and why, and one case where the ground truth itself is wrong and LVFace is right.

What good looks like

Before the failure analysis, the ceiling. This is a single sampled second from Downton Abbey's wedding scene — a packed, hat-heavy, period-costume group shot, about as hostile as ensemble framing gets:

Frame downton_abbey/best/best_t00127.jpg from the montage-frames artifact package (scripts/artifacts/pull_artifacts.sh montage-frames Downton_Abbey__A_New_Era) — green = identified, blue = detected but unknown.

Nineteen named faces in one frame, all nineteen correct — Jim Carter half behind a flower arrangement, Penelope Wilton at a three-quarter turn, Lesley Nicol under a hat brim. The blue "unknown" boxes are the honest cases: faces the detector found but the matcher declined to name rather than guess. The one miss at this second is Maggie Smith — not on screen in this framing, but X-Ray marks her present for the scene. That distinction (on-screen face vs. scene-level ground truth) sets up everything below.

Training vs. held-out: the generalization gap

The shipped config (prob_threshold=0.754, anneal_sec=35.54, extinction_sec=57.43, expand_gallery=true) was tuned against 4 films. Scored against the 5 films the optimizer never saw:

film	F1	P	R	TPI	FPI	misid	FN
Benny & Joon	83.0%	89.1%	77.7%	15125	1846	0	4337
Lovelace	77.5%	90.3%	67.9%	14990	1085	58	7085
Valerian and the City of a Thousand Planets	74.1%	97.1%	60.0%	18663	548	0	12467
Downton Abbey: A New Era	56.2%	97.8%	39.4%	52027	1173	0	80084
The Many Saints of Newark	46.3%	54.7%	40.1%	15922	4394	974	23791
macro average	67.4%	85.8%	57.0%

67.4% held-out vs. 75.3% on training — an ~8pp drop, and a 37pp spread between the best and worst held-out film. The config does not generalize uniformly; two films are outright failure cases, for reasons that turn out to be one mechanism.

The failure mode: frozen-bbox "ghost tracks"

Both Many Saints of Newark (974 misIDs) and Downton Abbey (FN=80084, the worst recall of the five) trace to the same root cause, verified directly against the raw per-frame stream and the HDF5 dump's own detection counts — not inferred from the score alone.

At this second, three of the four labeled boxes ("Jon Bernthal", "Joey Diaz", "Billy Magnussen") sit over empty background — a blurred wall, hanging plates — with no face in them. The real face in frame carries a second, colliding label from another frozen box. And it isn't an isolated second — the same signature recurs throughout the film:

Frame many_saints_intervals/w002_worst_t01382.jpg, same artifact package — one frame, three distinct error classes.

This frame is worth reading closely, because it separates three things that a single aggregate F1 number smears together. The two green labels ("Jon Bernthal 100%", "Michela De Rossi 100%") float over a staircase and a policeman's back — frozen boxes from a previous shot, reported at full confidence. Meanwhile the two real frontal faces in frame get honest blue "unknown 0%" boxes (they're uncredited day-players with no gallery reference — the gallery coverage gap), and three more people simply face away from camera, invisible to any face detector but still "present" in X-Ray's scene-level ground truth. Precision failure, gallery-coverage failure, and the face-vs-presence ceiling — one frame.

The mechanism, measured

The starkest case is Downton Abbey's hard cut from a packed group shot into a long blank credits sequence. Plotting the detector's per-second face_count (from the dump HDF5, independent of the tracker) against what the tracker reports makes the failure legible at a glance:

From the cut onward the detector sees zero faces for nearly a minute — and the tracker keeps reporting the last group shot's 15 identities the entire time, each with the exact same bounding box, unchanged to the pixel (verified for Hugh Bonneville: (1743.2, 0.0, 171.3, 317.8) at every sampled second for 57+ seconds). The staircase decay at the right edge is the extinction window finally expiring, actor by actor.

This is SceneTrackerFunc::active_[actor_idx].last_bbox (src/nodes/scene_tracker_node.hpp) being re-emitted unchanged — the extinction state machine working exactly as coded, not a bug. extinction_sec was tuned to 57.4s specifically because long windows bridge real gaps (occlusion, a turned face) in most footage; a hard cut into long faceless footage is the one case where that same bridging manufactures ghosts, and the training films never contained one long enough to punish it. The optimizer "discovered" the plateau at the top of its search range for a reason that only generalizes to films that never go faceless for a minute.

Where LVFace beat X-Ray

Not every flagged "misID" is actually wrong. scripts/optimizer/second_score.py counts a name as a true out-of-cast misID whenever the named actor isn't in X-Ray's credited cast list for the film at all — but X-Ray's cast list is itself incomplete.

Germar Terrell Gardner — a real, clean, high-confidence detection — is counted as a misID here because he doesn't appear in X-Ray's people.csv for The Many Saints of Newark at all. But Jellyfin's independent cast metadata does credit him for this exact film (cross-checked via experiments/manifests/jellyfin_casts.json from the experiment-data artifact package, a completely separate data source from X-Ray). This isn't a lookalike error or a gallery mixup — it's the pipeline correctly recognising a real cast member that one ground-truth source happened to omit.

This doesn't mean every flagged misID is secretly correct — Many Saints' 974-count total is still overwhelmingly the frozen-bbox failure mode above, not uncredited-but-real cameos. But it's a reminder that the X-Ray corpus is a convenient, large-scale ground truth, not a perfect one, and the "misID" number in any of these tables has some irreducible noise floor from ground-truth gaps in the other direction too.

Summary

LVFace is the right default: it wins the model comparison outright, it can name 19 faces correctly in a single hostile group shot, and its failures are traceable, understood, and mostly attributable to one tunable knob (extinction_sec) rather than the embedder itself. The held-out generalization gap (75.3% → 67.4%) is real and should be treated as the honest expected performance, not the training-set number.