dtourolle/scene-actor-extraction
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: richer report — data figures, success/failure frames, commit-pinned repo links

Browse Source 
- experiment_charts.py generates 4 figures from experiments/ artifacts:
  held-out per-film F1, 16-combo ranking, DE search landscape, and the
  Downton detector-vs-tracker ghost timeline (replaces the blank
  title-card screenshot)
- new frames: 19-correct wedding shot (success case), Many Saints
  ghost-vs-unknown frame (three error classes in one image)
- rename rep4-optimizer-results.md -> model-bakeoff.md; rep4 kept only
  as the on-disk artifact prefix, explained once
- repo file references are now links via https://REPOLINK/<path>
  placeholders; build_site.sh pins them to the HEAD commit's raw URLs
  and fails the build if a linked path doesn't exist at HEAD
- drop references to removed scripts (scene_score.py, score_config.py)
  and to session-memory names; mark artifact-registry paths with their
  pull commands
- commit readme_example.jpg + pipeline_topology.svg so README renders
  on the plain Gitea repo view
- deploy_pages.sh: push built site/ to the gitea-pages branch
This commit is contained in:
dtourolle
2026-07-19 22:06:56 +02:00
parent 4925443e56
commit b1efefac6f
17 changed files with 702 additions and 146 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/lvface-deep-dive.md
+82 -41
View File
@@ -1,9 +1,28 @@
# Deep dive: LVFace-B Glint360K
LVFace won the model bake-off (see `best-model.md`) and is the shipped default
embedder. This page is the honest accounting of how it actually performs —
including where it's wrong, and one case where the ground truth itself is
wrong and LVFace is right.
LVFace won the model bake-off (see [Which model is best?](best-model.md)) and is
the shipped default embedder. This page is the honest accounting of how it
actually performs — what a good second looks like, where it's wrong and *why*,
and one case where the ground truth itself is wrong and LVFace is right.
## What good looks like
Before the failure analysis, the ceiling. This is a single sampled second from
Downton Abbey's wedding scene — a packed, hat-heavy, period-costume group shot,
about as hostile as ensemble framing gets:
![19 correct identifications in one wedding shot, Downton Abbey: A New Era](assets/images/downton_wedding_19_correct.jpg)
*Frame `downton_abbey/best/best_t00127.jpg` from the `montage-frames` artifact
package (`scripts/artifacts/pull_artifacts.sh montage-frames
Downton_Abbey__A_New_Era`) — green = identified, blue = detected but unknown.*
**Nineteen named faces in one frame, all nineteen correct** — Jim Carter half
behind a flower arrangement, Penelope Wilton at a three-quarter turn, Lesley
Nicol under a hat brim. The blue "unknown" boxes are the honest cases: faces the
detector found but the matcher declined to name rather than guess. The one miss
at this second is Maggie Smith — not on screen in this framing, but X-Ray marks
her present for the scene. That distinction (on-screen face vs. scene-level
ground truth) sets up everything below.
## Training vs. held-out: the generalization gap
@@ -11,6 +30,8 @@ The shipped config (`prob_threshold=0.754, anneal_sec=35.54,
extinction_sec=57.43, expand_gallery=true`) was tuned against 4 films. Scored
against the 5 films the optimizer never saw:
![Held-out per-film F1 vs. the training-set fit](assets/images/holdout_f1_by_film.png)
| film | F1 | P | R | TPI | FPI | misid | FN |
|---|---|---|---|---|---|---|---|
| Benny & Joon | 83.0% | 89.1% | 77.7% | 15125 | 1846 | 0 | 4337 |
@@ -22,9 +43,10 @@ against the 5 films the optimizer never saw:
**67.4% held-out vs. 75.3% on training** — an ~8pp drop, and a **37pp spread
between the best and worst held-out film**. The config does not generalize
uniformly; two films are outright failure cases, for two different reasons.
uniformly; two films are outright failure cases, for reasons that turn out to
be one mechanism.
## Failure mode 1: frozen-bbox "ghost tracks"
## The failure mode: frozen-bbox "ghost tracks"
Both Many Saints of Newark (974 misIDs) and Downton Abbey (FN=80084, the worst
recall of the five) trace to the same root cause, verified directly against
@@ -36,51 +58,69 @@ inferred from the score alone.
At this second, three of the four labeled boxes ("Jon Bernthal", "Joey Diaz",
"Billy Magnussen") sit over empty background — a blurred wall, hanging
plates — with no face in them. The real face in frame carries a second,
colliding label from another frozen box.
colliding label from another frozen box. And it isn't an isolated second — the
same signature recurs throughout the film:
![15 ghost boxes over a blank title card, Downton Abbey: A New Era](assets/images/downton_abbey_ghost_fpi.jpg)
![Ghost labels over a staircase while real faces stay honest unknowns](assets/images/many_saints_ghosts_vs_unknowns.jpg)
*Frame `many_saints_intervals/w002_worst_t01382.jpg`, same artifact package —
one frame, three distinct error classes.*
This is the starkest case: **15 actors named, all wrong, over a completely
blank closing title card.** Confirmed against the dump directly: `face_count`
is 0 from this point onward (no detector output at all), yet the same 15
identities keep appearing with the *exact same bounding box, unchanged to the
pixel*, for 57+ consecutive seconds.
This frame is worth reading closely, because it separates three things that a
single aggregate F1 number smears together. The two green labels ("Jon Bernthal
100%", "Michela De Rossi 100%") float over a staircase and a policeman's back —
frozen boxes from a previous shot, reported at full confidence. Meanwhile the
two *real* frontal faces in frame get honest blue "unknown 0%" boxes (they're
uncredited day-players with no gallery reference — the
[gallery coverage gap](gallery-scope.md)), and three more people simply face
away from camera, invisible to any face detector but still "present" in X-Ray's
scene-level ground truth. Precision failure, gallery-coverage failure, and the
face-vs-presence ceiling — one frame.
### The mechanism, measured
The starkest case is Downton Abbey's hard cut from a packed group shot into a
long blank credits sequence. Plotting the detector's per-second `face_count`
(from the dump HDF5, independent of the tracker) against what the tracker
reports makes the failure legible at a glance:
![Detector vs. tracker through Downton Abbey's cut to credits](assets/images/downton_ghost_timeline.png)
From the cut onward the detector sees **zero faces for nearly a minute** — and
the tracker keeps reporting the last group shot's 15 identities the entire
time, each with the *exact same bounding box, unchanged to the pixel* (verified
for Hugh Bonneville: `(1743.2, 0.0, 171.3, 317.8)` at every sampled second for
57+ seconds). The staircase decay at the right edge is the extinction window
finally expiring, actor by actor.
This is `SceneTrackerFunc::active_[actor_idx].last_bbox`
(`src/nodes/scene_tracker_node.hpp`) being re-emitted unchanged — the
extinction state machine working exactly as coded, not a bug. The film cuts
from a packed group shot straight into 40+ seconds of blank titles/credits,
and `extinction_sec=57.4` is comfortably long enough to bridge that entire gap
without expiring, so the tracker faithfully reports "last known position" for
a cast that is no longer on screen at all. `extinction_sec` was tuned toward
long windows specifically because they bridge real gaps (occlusion, a turned
face) in most training footage — this is the cost side of that trade,
surfacing only when a film has a long enough faceless stretch to expose it.
## Failure mode 2: a genuine misID (for contrast)
Not every held-out failure is a ghost. This is a real face, correctly
detected, confidently misidentified:
*(same many_saints_ghost_fpi.jpg frame above also shows Leslie Odom Jr.'s box
carrying a second, colliding "Michael Gandolfini" label — two real tracks'
frozen positions happening to overlap, not a detection error.)*
([`src/nodes/scene_tracker_node.hpp`](https://REPOLINK/src/nodes/scene_tracker_node.hpp))
being re-emitted unchanged — the
extinction state machine working exactly as coded, not a bug. `extinction_sec`
was tuned to 57.4s specifically because long windows bridge real gaps
(occlusion, a turned face) in most footage; a hard cut into long faceless
footage is the one case where that same bridging manufactures ghosts, and the
training films never contained one long enough to punish it. The optimizer
"discovered" the plateau at the top of its search range for a reason that only
generalizes to films that never go faceless for a minute.
## Where LVFace beat X-Ray
Not every "misID" is actually wrong. `second_score.py` counts a name as a true
out-of-cast misID whenever the named actor isn't in X-Ray's credited cast list
for the film at all — but X-Ray's cast list is itself incomplete.
Not every flagged "misID" is actually wrong.
[`scripts/optimizer/second_score.py`](https://REPOLINK/scripts/optimizer/second_score.py)
counts a name as a true out-of-cast misID whenever the named actor isn't in
X-Ray's credited cast list for the film at all — but X-Ray's cast list is
itself incomplete.
![LVFace correctly identifies Germar Terrell Gardner, uncredited by X-Ray](assets/images/germar_beats_xray.jpg)
Germar Terrell Gardner — a real, clean, high-confidence detection — is counted
as a misID here because he doesn't appear in X-Ray's `people.csv` for The Many
Saints of Newark at all. But Jellyfin's independent cast metadata *does* credit
him for this exact film (cross-checked via `experiments/manifests/
jellyfin_casts.json`, a completely separate data source from X-Ray). This
isn't a lookalike error or a gallery mixup — it's the pipeline correctly
recognising a real cast member that one ground-truth source happened to omit.
him for this exact film (cross-checked via
`experiments/manifests/jellyfin_casts.json` from the `experiment-data` artifact
package, a completely separate data source from X-Ray). This isn't a lookalike error or a gallery mixup — it's the
pipeline correctly recognising a real cast member that one ground-truth source
happened to omit.
This doesn't mean every flagged misID is secretly correct — Many Saints'
974-count total is still overwhelmingly the frozen-bbox failure mode above,
@@ -91,8 +131,9 @@ in the other direction too.
## Summary
LVFace is the right default: it wins the model comparison outright, and its
failures are traceable, understood, and mostly attributable to one tunable
knob (`extinction_sec`) rather than the embedder itself. The held-out
LVFace is the right default: it wins the model comparison outright, it can name
19 faces correctly in a single hostile group shot, and its failures are
traceable, understood, and mostly attributable to one tunable knob
(`extinction_sec`) rather than the embedder itself. The held-out
generalization gap (75.3% → 67.4%) is real and should be treated as the honest
expected performance, not the training-set number.