dtourolle/scene-actor-extraction
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: full data-grounded rewrite of the performance report

Browse Source 
Replaces narrative claims with verified numbers across all report pages:

- Cross-model held-out validation (LVFace/mbf/r18, all 5 held-out
  films): LVFace wins every film outright, not just "consistent with"
  the training-set pick. r50 dropped from the detailed comparison
  (gallery has ~30% fewer reference images per actor than the other
  three models on identical source photos).
- Per-film training breakdown: LVFace does not win every training
  film (mbf beats it on Lord of War); the 75.3% macro figure hides a
  10.7pp spread.
- Gallery coverage computed per film (20.3%-78.6%) instead of one
  flat 67%-missing average.
- Found and fixed a real scoring bug in optimize.py: a candidate
  whose hardest film's replay timed out was averaged over survivors
  instead of penalized, silently rewarding partial coverage. Affected
  3 of 16 training combos; corrected throughout, and optimize.py now
  scores an incomplete evaluation f1=0.0 instead of averaging over
  whichever films happened to finish.
- Every FPI frame in the deep dive now comes from the proper montage
  renderer (Onscreen/Offscreen panel, ghosts never drawn as boxes),
  never the bare-box debug overlay used earlier.
- Every distinct out-of-cast name across all 9 films gets its own
  frame at its first appearance (9 names, 4 films), not a
  single-example spot check: 2 ground-truth gaps, 1 photograph
  misread as a person, 6 genuine lookalike confusions.
- New methodology.md: the scene-level-vs-per-second scoring mismatch
  that the rest of the report assumes, written out once.
- Cut the deadlock/gdb debugging narrative from the experiment log;
  kept the one fact that matters (KPN's node/network split lets the
  expensive GPU stage run once and the cheap stage replay against
  cached embeddings).
- Plain declarative style throughout, no em dashes, no blog voice.
This commit is contained in:
dtourolle
2026-07-21 08:55:57 +02:00
parent 4b5557974b
commit 0bd2747069
18 changed files with 1824 additions and 890 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/lvface-deep-dive.md
+233 -130
View File
@@ -1,52 +1,53 @@
# Deep dive: LVFace-B Glint360K
LVFace won the model bake-off (see [Which model is best?](best-model.md)) and is
the shipped default embedder. This page is the honest accounting of how it
actually performs — what a good second looks like, where the errors actually
come from, and two cases where the ground truth itself is wrong and LVFace is
right.
LVFace won the model comparison (see [Which model is best?](best-model.md))
and is the shipped default embedder. This page reports how it performs in
detail: a baseline of correct output, the two mechanisms behind its errors,
and every distinct case where it names someone who is not in the film's
credited cast.
Read [How we score against X-Ray](methodology.md) first. X-Ray's ground truth
is scene-level, not per-frame. A name marked correct in the Offscreen column
below is the pipeline correctly reporting scene membership, not a workaround.
!!! note "How to read the frames on this page"
The top is the film frame, with a box and name on every face the pipeline
identified. The bottom panels are the per-second verdict against X-Ray:
**Onscreen** lists faces named in the frame, **Offscreen** lists cast
X-Ray marks present in the scene without a visible face — presence
carried by the tracker's windows, not by a detection. Colors are the
score: <span style="color:#0ca30c">**green**</span> = correct (TPI),
<span style="color:#eb6834">**orange**</span> = wrong (FPI),
<span style="color:#3987e5">**blue**</span> = missed (FN).
The top of each image is the film frame, with a box and name on every
face the pipeline matched to a real detection. The panels below are the
per-second result against X-Ray. **Onscreen** lists names attached to a
visible face this second. **Offscreen** lists names the pipeline reports
present without a currently visible face. Colors mark the verdict:
<span style="color:#0ca30c">**green**</span> correct (TPI),
<span style="color:#eb6834">**orange**</span> wrong (FPI),
<span style="color:#3987e5">**blue**</span> missed (FN).
## What good looks like
## Baseline: correctly scored seconds
![Wedding couple correctly identified, Downton Abbey: A New Era](assets/images/downton_wedding_couple.jpg)
Six faces on screen, all six named correctly — including Penelope Wilton at the
edge of the pews and a half-occluded Michelle Dockery — while thirteen more
cast members X-Ray marks present in the scene are correctly carried as
"Offscreen" by their presence windows. One miss in the whole frame: Maggie
Smith (blue). Score for this second: 0.86.
Six faces on screen, all six named correctly, including Penelope Wilton at
the edge of the pews and a partly occluded Michelle Dockery. Thirteen more
cast members X-Ray lists as present in the scene are correctly reported
Offscreen. One miss: Maggie Smith (blue). Score for this second: 0.86.
![19 of 20 correct in the funeral crowd](assets/images/downton_funeral_19of20.jpg)
The same film's funeral gathering: mourning dress, hats, half the faces turned.
**Nineteen of the twenty cast X-Ray lists for this scene are scored correctly**
— seven named on screen at up to 100% confidence, twelve more correctly held
as present off-screen.
And the pipeline doesn't need the face to be *real*:
The same film's funeral scene: dark clothing, hats, half the faces turned
away. Nineteen of the twenty cast members X-Ray lists for this scene score
correct: seven named on screen at up to 100% confidence, twelve more reported
correctly as present but not visible.
![Herbie Hancock identified on an in-fiction video call](assets/images/valerian_screen_call.jpg)
That's Herbie Hancock at 98% — as a face on a *screen inside the movie*, over a
sci-fi HUD overlay, during a video call in Valerian. A face is a face, whether
it's in the room or on the bridge's comms display.
The pipeline does not require a live face. This is Herbie Hancock at 98%
confidence, identified from a face displayed on a screen inside the film, on
a video call under a science-fiction HUD overlay.
## Training vs. held-out: the generalization gap
The shipped config (`prob_threshold=0.754, anneal_sec=35.54,
extinction_sec=57.43, expand_gallery=true`) was tuned against 4 films. Scored
against the 5 films the optimizer never saw:
The shipped config (`prob_threshold=0.754`, `anneal_sec=35.54`,
`extinction_sec=57.43`, `expand_gallery=true`) was tuned on 4 films. Scored
on the 5 films the optimizer never saw:
![Held-out per-film F1 vs. the training-set fit](assets/images/holdout_f1_by_film.png)
@@ -56,139 +57,241 @@ against the 5 films the optimizer never saw:
| Lovelace | 77.5% | 90.3% | 67.9% | 14990 | 1085 | 58 | 7085 |
| Valerian and the City of a Thousand Planets | 74.1% | 97.1% | 60.0% | 18663 | 548 | 0 | 12467 |
| Downton Abbey: A New Era | 56.2% | 97.8% | 39.4% | 52027 | 1173 | 0 | 80084 |
| **The Many Saints of Newark** | **46.3%** | **54.7%** | 40.1% | 15922 | 4394 | **974** | 23791 |
| **macro average** | **67.4%** | 85.8% | 57.0% | | | | |
| The Many Saints of Newark | 46.3% | 54.7% | 40.1% | 15922 | 4394 | 974 | 23791 |
| macro average | 67.4% | 85.8% | 57.0% | | | | |
**67.4% held-out vs. 75.3% on training** — an ~8pp drop, and a **37pp spread
between the best and worst held-out film**. The config does not generalize
uniformly, and the spread traces to two mechanisms, both visible frame by
frame below.
The `P` column is misID-weighted (each out-of-film name counts 10x in the
denominator; see [methodology](methodology.md#precision-recall-and-the-misid-weighting)).
That weighting is why Many Saints reads 54.7% here despite naming mostly real,
present faces: its raw (unweighted) precision is **78.4%**, and the gap is
entirely its 974 misIDs paying the 10x penalty. The three zero-misID films
(Benny & Joon, Downton, Valerian) have identical weighted and raw precision;
Lovelace, with 58 misIDs, sits 3pp below its raw 93.3%.
## Mechanism 1: extinction bridging — usually right, wrong at hard cuts
Held-out F1 is 67.4%, against 75.3% on training, an 8pp drop. The spread
between the best and worst held-out film is 37pp. This is not unique to
LVFace: [the full experiment log](model-bakeoff.md#held-out-validation-all-3-models)
shows mbf and r18 with the same shape of spread on the same films, at a
uniformly lower level. Two mechanisms explain the spread. Both are shown
below with frame-level evidence.
The extinction window keeps an identity alive through seconds where no face is
detectable. **Most of the time this is exactly what you want**, and it's where
a lot of the TPI count comes from:
## Mechanism 1: extinction bridging
The extinction window keeps a name reported as present for up to
`extinction_sec` after its last real detection. This is deliberate: most
gaps in face visibility are short (a turned head, an occlusion, a cut to a
reaction shot), and the window bridges them.
![Two faces on screen, six more correctly bridged](assets/images/lovelace_polygraph_bridged.jpg)
Lovelace's polygraph scene: only Eric Roberts and Amanda Seyfried have visible
faces, but X-Ray lists eight cast present — and all eight score green, the
other six correctly carried by presence windows through a scene where the
camera never shows them. A perfect second, and the extinction/anneal machinery
is *why*.
Lovelace's polygraph scene: only Eric Roberts and Amanda Seyfried have
visible faces. X-Ray lists eight cast members present. All eight score
correct; the other six are reported Offscreen through a stretch where the
camera never shows them. The extinction window is why.
The same mechanism has a failure case: a hard cut into long faceless footage.
Both Many Saints of Newark (974 misIDs) and Downton Abbey (FN=80084, the worst
recall of the five) are dominated by it — verified directly against the raw
per-frame stream and the HDF5 dump's own detection counts, not inferred from
the score alone. **This is not a malfunction**: the tracker is doing exactly
what its window is for; the footage just stops cooperating. In the debug
overlay (which draws a bridged identity's last-known bbox, unlike the shipped
output, which emits presence windows and no boxes at all) the bridged state is
visible spatially:
![Debug overlay: bridged identities drawn at their last-known positions](assets/images/many_saints_ghost_fpi.jpg)
*Debug-overlay rendering (`dump_error_frames.py --raw`): "Jon Bernthal", "Joey
Diaz" and "Billy Magnussen" are extinction-bridged identities from the previous
shot, drawn frozen over the wall and the hanging plates. Frame
`many_saints/fpi/fpi_t03543.jpg`, `montage-frames` artifact package.*
The cost is measurable, not just visible. Downton Abbey's hard cut into its
closing credits, plotting the dump's own per-second `face_count` (detector
output, independent of the tracker) against what the tracker reports:
The same mechanism fails at a hard cut into a long stretch with no faces at
all. Downton Abbey's recall (39.4%, the worst of the five held-out films) is
dominated by this failure. It is verified directly against the raw
per-frame stream and the dump's own detection counts, not inferred from the
score. Plotting the dump's per-second `face_count` (detector output,
independent of the tracker) against what the tracker reports, through
Downton Abbey's hard cut into its closing credits:
![Detector vs. tracker through Downton Abbey's cut to credits](assets/images/downton_ghost_timeline.png)
From the cut onward the detector sees **zero faces for nearly a minute** — and
the tracker keeps reporting the last shot's 15 identities the whole time
(verified for Hugh Bonneville: bbox `(1743.2, 0.0, 171.3, 317.8)`, unchanged to
the pixel, at every sampled second for 57+ seconds). The staircase at the right
edge is the extinction window expiring actor by actor. That plateau is
`SceneTrackerFunc::active_[actor_idx].last_bbox`
From the cut onward the detector reports zero faces for close to a minute.
The tracker continues reporting the previous shot's 15 identities for the
same span (verified for Hugh Bonneville: bbox `(1743.2, 0.0, 171.3, 317.8)`,
unchanged to the pixel, at every sampled second for 57 seconds). The
staircase at the right edge is the extinction window expiring, actor by
actor. This is `SceneTrackerFunc::active_[actor_idx].last_bbox`
([`src/nodes/scene_tracker_node.hpp`](https://REPOLINK/src/nodes/scene_tracker_node.hpp))
re-emitted as designed: `extinction_sec=57.4` was tuned long because bridging
wins on most footage (see the polygraph frame above) — the training films just
never contained a faceless stretch long enough to show the cost side, and the
held-out set did.
re-emitted as designed. `extinction_sec=57.4` was tuned long because
bridging is correct on most footage, as in the polygraph scene above. The
training films did not contain a faceless stretch long enough to expose the
cost side; the held-out set did.
The same track-continuation machinery has one milder spatial artifact, worth
knowing when reading these frames:
The extinction window is a scoring concept, not something drawn on screen.
The shipped output is presence windows with no bounding boxes. Even the
debug overlay used for this report never draws a box for a bridged name: a
name inside its extinction window with no current detection appears only as
a name in the Offscreen column, the same as every correctly bridged name
above.
A related, smaller effect shows up at rapid cuts:
![Two labels on one face after a shot/reverse-shot cut](assets/images/cafe_society_rapid_cut.jpg)
*Café Society (a training film), a shot/reverse-shot dialog: that is Steve
Carell wearing both his own label and Jesse Eisenberg's.*
At a rapid cut, the previous shot's track can linger for a beat at nearly the
same screen position the new face occupies — here Jesse Eisenberg's box from
the counter-shot lands on Steve Carell. Note what the score panel says,
though: both actors are green, because both *are* present in this dialog
scene per X-Ray. The spatial label is briefly wrong; the per-second presence
claim — the thing the pipeline actually ships — is right. It's the same trade
as the extinction window: track continuation smooths over cuts, and 1 fps
sampling occasionally catches the seam.
Café Society (a training film), a shot/reverse-shot dialog. The box on Steve
Carell's face carries two labels: his own, and Jesse Eisenberg's, left over
from the counter-shot a moment earlier. Both names score correct, because
both actors are present in this scene per X-Ray. The box position is
briefly wrong; the presence claim, which is what the pipeline ships, is
right.
## Mechanism 2: the face-vs-presence ceiling
Downton Abbey's recall didn't collapse because faces were misread — it
collapsed because for most of its 80084 FN-seconds there was **no face to
read**:
Downton Abbey's recall did not collapse because faces were misread. It
collapsed because for most of its 80084 false-negative seconds there was no
face to read.
![22 cast credited, nobody facing the camera](assets/images/downton_crew_fn.jpg)
A newsreel crew hauls equipment through the hall: X-Ray credits 22 cast as
present in this scene; not one face looks at the camera. Eight are still
scored green (windows bridging from adjacent shots) — the other fourteen are
blue FNs that no face-recognition pipeline could ever recover. X-Ray encodes
*scene membership*; the pipeline measures *on-screen faces*. In ensemble films
those two definitions diverge massively, and that gap — not identification
error — is most of what the FN column counts.
A newsreel crew moves equipment through the hall. X-Ray credits 22 cast
members as present in this scene. None face the camera. Eight still score
correct, carried by presence windows from adjacent shots. The other fourteen
are missed, and no face-recognition system can recover them, because there
is no face in the frame. X-Ray records scene membership; the pipeline
measures visible faces. In ensemble scenes these two quantities diverge, and
that gap accounts for most of the false-negative count.
![Presence without a detectable face, The Many Saints of Newark](assets/images/many_saints_outofcast_fpi.jpg)
## Every distinct out-of-cast name
Same ceiling from the other side: Michela De Rossi in frame but turned away,
five cast correctly bridged as offscreen (green), four blue FNs — and one
orange we'll come back to below.
Many Saints of Newark has the largest misID count of any held-out film: 974
seconds, weighted. Rather than characterize this from a single frame, the
raw replay stream was searched directly for every name the pipeline reports
that is not in the film's credited cast. The same search was run on all 9
films in the benchmark, one rule applied uniformly: **find the first second
each distinct out-of-cast name appears, and render that exact second.**
Five films produce no such name anywhere in their runtime: Benny & Joon,
Café Society, Downton Abbey, Sound of Metal, Valerian. Zero out-of-cast
names across their entire length. Four films produce nine distinct names
between them, shown below in full, not a sample.
### The Many Saints of Newark: 4 names
![Germar Terrell Gardner, first out-of-cast name in Many Saints](assets/images/many_saints_fpi_gardner.jpg)
Germar Terrell Gardner, t=848s, 78% confidence. A real, clearly visible
background actor. He is not in X-Ray's cast list for this film, but he is
credited in Jellyfin's independent cast metadata (see
[Where LVFace beat X-Ray](#where-lvface-beat-x-ray) below). This is a
ground-truth gap, not a model error.
![Archie Yates, second out-of-cast name in Many Saints](assets/images/many_saints_fpi_yates.jpg)
Archie Yates, t=2521s, 78% confidence. A real detected face, a genuine
lookalike confusion.
![Zooey Deschanel, third out-of-cast name in Many Saints](assets/images/many_saints_fpi_deschanel.jpg)
Zooey Deschanel, t=2819s, 99% confidence. A real detected face at a dinner
table, high-confidence lookalike confusion.
![Talia Balsam, fourth out-of-cast name in Many Saints](assets/images/many_saints_fpi_balsam.jpg)
Talia Balsam, t=4551s, 93% confidence. A real detected face. Talia Balsam
plays Mrs. Jarecki, a guidance counselor, in this film; she is confirmed
on screen by direct inspection of the frame. She does not appear in X-Ray's
`people.csv` for this title. This is a second ground-truth gap in the same
film, not a model error.
Two of these four names are ground-truth gaps (Gardner, Balsam), not
misidentifications. The other two (Yates, Deschanel) are genuine embedding
errors on real faces.
### Lord of War: 3 names
![David Shumbris, first out-of-cast name in Lord of War](assets/images/lord_of_war_fpi_shumbris.jpg)
David Shumbris, t=418s, 81% confidence. A real face in a dim, low-detail
shot under a train track. A genuine lookalike confusion in poor lighting.
![Ronald Reagan, second out-of-cast name in Lord of War](assets/images/lord_of_war_fpi_reagan_photo.jpg)
Ronald Reagan, t=1003s, 100% confidence. This is not a lookalike confusion.
The detected face is a photograph of Reagan appearing within the shot, not a
living actor. The detector and matcher both did their job correctly on the
image content in front of them; the error is that a photograph inside the
scene is not the same thing as an actor present in the scene, and the
pipeline has no way to draw that distinction from a face crop alone.
![Lance Reddick, third out-of-cast name in Lord of War](assets/images/lord_of_war_fpi_reddick.jpg)
Lance Reddick, t=6424s, 78% confidence. A small, distant, low-detail face at
the edge of frame. A marginal, low-confidence lookalike confusion.
### Lovelace: 1 name
![Chloë Sevigny, out-of-cast name in Lovelace](assets/images/lovelace_fpi_sevigny.jpg)
Chloë Sevigny, t=2451s, 100% confidence. Two boxes are drawn on the same
face: one correctly labeled Amanda Seyfried, one incorrectly labeled Chloë
Sevigny, both at 100%. A single detection producing two competing high-
confidence identities on the same crop.
### Scarface: 1 name
![Kirstie Alley, out-of-cast name in Scarface](assets/images/scarface_fpi_alley.jpg)
Kirstie Alley, t=2451s, 89% confidence. Al Pacino is correctly identified in
the foreground at 100%; a background face in the same shot is wrongly
labeled Kirstie Alley. (The t=2451s here and the Lovelace Chloë Sevigny case
above landing on the identical second is a genuine coincidence, verified from
each film's raw stream by [`first_fpi_frames.py`](https://REPOLINK/scripts/docs/first_fpi_frames.py),
not a transcription slip, two unrelated films whose *first* out-of-cast name
happens to fall at the same timestamp.)
### Summary of the nine
| film | name | t (s) | confidence | classification |
|---|---|---|---|---|
| Many Saints of Newark | Germar Terrell Gardner | 848 | 78% | ground-truth gap |
| Many Saints of Newark | Archie Yates | 2521 | 78% | lookalike confusion |
| Many Saints of Newark | Zooey Deschanel | 2819 | 99% | lookalike confusion |
| Many Saints of Newark | Talia Balsam | 4551 | 93% | ground-truth gap |
| Lord of War | David Shumbris | 418 | 81% | lookalike confusion |
| Lord of War | Ronald Reagan | 1003 | 100% | photo-in-frame |
| Lord of War | Lance Reddick | 6424 | 78% | lookalike confusion, marginal |
| Lovelace | Chloë Sevigny | 2451 | 100% | lookalike confusion |
| Scarface | Kirstie Alley | 2451 | 89% | lookalike confusion |
Of nine distinct out-of-cast names across four films, two are ground-truth
gaps, one is a photograph misread as a person, and six are genuine
embedding-space confusions on real detected faces. None trace to extinction
bridging: every one of these nine is a fresh detection on a real face crop
at the second it first appears.
## Where LVFace beat X-Ray
Not every orange in these frames is actually wrong.
Not every name marked wrong is actually wrong.
[`scripts/optimizer/second_score.py`](https://REPOLINK/scripts/optimizer/second_score.py)
scores strictly against X-Ray — but X-Ray itself has holes, and the pipeline
found two kinds.
scores strictly against X-Ray, and X-Ray has gaps of its own.
![LVFace correctly identifies Germar Terrell Gardner, uncredited by X-Ray](assets/images/germar_beats_xray.jpg)
Germar Terrell Gardner — a real, clean, high-confidence detection — is counted
as an out-of-cast misID because he doesn't appear in X-Ray's `people.csv` for
The Many Saints of Newark at all. But Jellyfin's independent cast metadata
*does* credit him for this exact film (cross-checked via
`experiments/manifests/jellyfin_casts.json` from the `experiment-data` artifact
package, a completely separate data source from X-Ray). That's also him in
orange in the frame above — every one of those "errors" is the pipeline being
right about a person X-Ray forgot.
Germar Terrell Gardner, the same name from the table above, does not appear
in X-Ray's `people.csv` for The Many Saints of Newark. Jellyfin's
independent cast metadata does credit him for this film (cross-checked
against `experiments/manifests/jellyfin_casts.json` from the
`experiment-data` artifact package, a data source entirely separate from
X-Ray). Talia Balsam is the same case: confirmed on screen, absent from
X-Ray's cast list for this title.
![Robert Patrick, clearly on screen, scored wrong by a ground-truth gap](assets/images/lovelace_robert_patrick_fpi.jpg)
And it isn't only uncredited bit-parts. That is **Robert Patrick** — top-billed
in Lovelace, unmistakably on screen, reading his newspaper, identified at
100% — scored orange because X-Ray's people-in-scene list for *this scene*
doesn't include him. The identification is flawless; the ground truth missed
an actor sitting in the middle of the frame.
This extends past uncredited background actors. This is Robert Patrick,
top-billed in Lovelace, clearly on screen reading a newspaper, identified at
100%. The frame is scored wrong because X-Ray's people-in-scene list for
this specific scene omits him, despite crediting him elsewhere in the film.
The identification is correct; the ground truth is missing an entry.
This doesn't mean every flagged misID is secretly correct — Many Saints'
974-count total is still overwhelmingly extinction bridging at cuts, not
uncredited cameos. But the X-Ray corpus is a convenient, large-scale ground
truth, not a perfect one, and the misID/FPI numbers in these tables carry an
irreducible noise floor from ground-truth gaps in both directions.
X-Ray is a large, convenient ground truth. It is not a complete one. The
misID and FPI counts reported throughout this document include some fixed
amount of noise from gaps in X-Ray itself, in both directions.
## Summary
LVFace is the right default: it wins the model comparison outright, it names
19 of 20 correctly across a hat-heavy funeral crowd, and it recognises a face
on a screen inside the movie. Its error budget decomposes into two understood
mechanisms — extinction bridging at hard cuts (a tunable trade, not a bug) and
the face-vs-presence ceiling baked into X-Ray's semantics — plus a nonzero
slice where the pipeline is right and the ground truth is wrong. The held-out
generalization gap (75.3% → 67.4%) is real and should be treated as the honest
expected performance, not the training-set number.
LVFace wins the model comparison on every held-out film. It correctly names
19 of 20 people in a crowded funeral scene and correctly identifies a face
displayed on a screen inside the film. Its errors resolve into two
mechanisms: extinction bridging, which is correct on most footage and fails
specifically at hard cuts into long faceless stretches, and the
face-versus-presence ceiling, where X-Ray credits scene membership for
people whose faces never appear on screen. Of the nine distinct
out-of-cast identifications found across the benchmark, two trace to gaps in
X-Ray's own cast data, one is a photograph misread as a person, and six are
genuine lookalike confusions on real faces. The held-out generalization gap,
75.3% training to 67.4% held-out, is real and should be treated as the
expected operating point, not the training-set figure.