dtourolle
0bd2747069
Replaces narrative claims with verified numbers across all report pages: - Cross-model held-out validation (LVFace/mbf/r18, all 5 held-out films): LVFace wins every film outright, not just "consistent with" the training-set pick. r50 dropped from the detailed comparison (gallery has ~30% fewer reference images per actor than the other three models on identical source photos). - Per-film training breakdown: LVFace does not win every training film (mbf beats it on Lord of War); the 75.3% macro figure hides a 10.7pp spread. - Gallery coverage computed per film (20.3%-78.6%) instead of one flat 67%-missing average. - Found and fixed a real scoring bug in optimize.py: a candidate whose hardest film's replay timed out was averaged over survivors instead of penalized, silently rewarding partial coverage. Affected 3 of 16 training combos; corrected throughout, and optimize.py now scores an incomplete evaluation f1=0.0 instead of averaging over whichever films happened to finish. - Every FPI frame in the deep dive now comes from the proper montage renderer (Onscreen/Offscreen panel, ghosts never drawn as boxes), never the bare-box debug overlay used earlier. - Every distinct out-of-cast name across all 9 films gets its own frame at its first appearance (9 names, 4 films), not a single-example spot check: 2 ground-truth gaps, 1 photograph misread as a person, 6 genuine lookalike confusions. - New methodology.md: the scene-level-vs-per-second scoring mismatch that the rest of the report assumes, written out once. - Cut the deadlock/gdb debugging narrative from the experiment log; kept the one fact that matters (KPN's node/network split lets the expensive GPU stage run once and the cheap stage replay against cached embeddings). - Plain declarative style throughout, no em dashes, no blog voice.
135 lines
6.2 KiB
Markdown
135 lines
6.2 KiB
Markdown
# How we score against X-Ray
|
||
|
||
Every number in this report, every F1 and misID count, comes from one
|
||
comparison. The comparison has a mismatch at its core that shapes nearly
|
||
every finding in this report: the ground truth is scene-level, the
|
||
pipeline's output is per-second, and the two do not mean the same thing.
|
||
This page documents that comparison once, so the findings pages can rely on
|
||
it without re-explaining it.
|
||
|
||
## What Amazon X-Ray records
|
||
|
||
X-Ray ships three tables per film: `scenes.csv` (a list of `[start, end]`
|
||
timespans), `people_in_scenes.csv` (which actors are credited in each
|
||
scene), and `people.csv` (actor identities). There is no per-frame or
|
||
per-second annotation anywhere in X-Ray. A scene might run 45 seconds, and
|
||
X-Ray records one cast list for the entire span, not "on screen from
|
||
second 12 to second 30."
|
||
|
||
To compare this against per-second predictions, `second_score.py` expands
|
||
every scene into per-second ground truth by copying the whole scene's cast
|
||
list onto every second inside it:
|
||
|
||
```python
|
||
for sn, (t0, t1) in spans.items():
|
||
cast = scene_cast.get(sn, [])
|
||
for t in range(int(t0), int(t1)):
|
||
timeline[t] = cast
|
||
```
|
||
|
||
That is the entire mechanism. If X-Ray credits five actors to a 30-second
|
||
scene, all five count as ground truth present for all 30 seconds, including
|
||
seconds where only one of them is on screen. This is not a simplification
|
||
introduced by the pipeline; it is the only reading of X-Ray's data that is
|
||
possible, because X-Ray itself does not record anything finer-grained.
|
||
|
||
## Why an offscreen name can be scored correct
|
||
|
||
A name listed under Offscreen with a correct (green) label is not the
|
||
pipeline guessing or padding its score. It is the pipeline correctly
|
||
answering the question X-Ray actually asks: is this actor part of this
|
||
scene. It answers that question using a presence window (`[start, end]`,
|
||
held open across cuts by `anneal_sec` and `extinction_sec`), which matches
|
||
X-Ray's scene-level semantics more closely than a raw per-frame detection
|
||
would.
|
||
|
||
A system that only reported "this actor is visible in this exact frame"
|
||
would score worse against X-Ray's scene-level ground truth, producing a
|
||
false negative every time the camera cuts away from a character who is
|
||
still present in the scene. Not because it is wrong about the world, but
|
||
because it would be answering a stricter, different question than the one
|
||
X-Ray's data supports. The presence-window design exists specifically to
|
||
answer X-Ray's actual question.
|
||
|
||
## What this resolves and what it does not
|
||
|
||
This resolves the semantic mismatch between a scene and an instant. It does
|
||
not resolve two other limitations, both discussed in the
|
||
[LVFace deep dive](lvface-deep-dive.md).
|
||
|
||
**The face-vs-presence ceiling.** X-Ray credits scene membership regardless
|
||
of whether a face is ever visible: background crew, characters shot from
|
||
behind, voice-only presence. No amount of bridging recovers a face that
|
||
never appears on screen. This is a hard ceiling on recall, not a defect.
|
||
|
||
**Extinction bridging can overshoot.** The same presence-window mechanism
|
||
that correctly answers "still in this scene" during a normal cut can also
|
||
bridge across a scene boundary it has no way to detect. A hard cut into a
|
||
different scene with no faces, such as closing credits, carries the
|
||
previous scene's identities forward until the window expires. This is the
|
||
mechanism behind Downton Abbey's recall collapse, documented in the deep
|
||
dive.
|
||
|
||
## Precision, recall, and the misID weighting
|
||
|
||
Per sampled second `t`:
|
||
|
||
**TPI** (true positive instances): actors both X-Ray and the pipeline agree
|
||
are present.
|
||
|
||
**FPI** (false positive instances): actors the pipeline reports that are
|
||
not in X-Ray's cast for this second. Split into two categories:
|
||
|
||
- **FPI_incast**: the actor is in the film's cast, just not credited to
|
||
this particular scene. A timing or boundary slip.
|
||
- **FPI_misid**: the actor is not in the film's cast at all. A genuine
|
||
wrong-identity error, weighted 10x in the precision objective, because
|
||
naming someone who is not even in the film is a categorically worse
|
||
error than a few seconds of scene-boundary slop.
|
||
|
||
!!! note "Every headline `P` and `F1` is misID-weighted"
|
||
|
||
The precision reported throughout this report, and therefore the F1
|
||
derived from it, puts each `FPI_misid` into the denominator **10 times**
|
||
(`precision = TPI / (TPI + FPI_incast + 10·FPI_misid)`,
|
||
[`second_score.py`](https://REPOLINK/scripts/optimizer/second_score.py)).
|
||
This is deliberate: the whole point is to punish naming an out-of-film
|
||
actor far harder than a scene-boundary slip. But it means the `P` column
|
||
is not raw precision, and a misID-heavy film's `P` is depressed
|
||
super-linearly. `second_score.py` also emits an unweighted `precision_raw`
|
||
(always ≥ the weighted `P`); where the gap matters, The Many Saints of
|
||
Newark, weighted `P` 54.7% vs. raw 78.4%, the [LVFace deep dive](lvface-deep-dive.md)
|
||
reports both. When comparing `P` across films, remember you are comparing a
|
||
quantity that penalizes misIDs, not just a hit rate.
|
||
|
||
**FN** (false negatives): actors X-Ray lists that the pipeline never
|
||
reports, counted only for actors who have a gallery reference embedding.
|
||
Across the 9-film benchmark, coverage of X-Ray's credited cast ranges from
|
||
20% to 79% by film (see
|
||
[the full experiment log](model-bakeoff.md#gallery-coverage-per-film)); an
|
||
actor with no reference photo can never be recognized regardless of model
|
||
quality, and counting them as a miss would penalize gallery coverage, not
|
||
recognition accuracy.
|
||
|
||
Two further numbers are reported alongside F1:
|
||
|
||
**agreement_rate**: mean per-second Jaccard overlap
|
||
(`|Pred ∩ GT| / |Pred ∪ GT|`), partial credit. Naming 2 of 3 present actors
|
||
scores 2/3, not 0.
|
||
|
||
**exact_match_rate**: the fraction of sampled seconds where the pipeline's
|
||
named set exactly equals X-Ray's, no partial credit. Far harsher, and
|
||
dominated by recall, since any single missed actor zeroes that second.
|
||
|
||
## Reproduce
|
||
|
||
```bash
|
||
python3 scripts/optimizer/second_score.py \
|
||
--pred pred.json --xray experiments/xray/.../<xray_dir> \
|
||
--gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5
|
||
```
|
||
|
||
See also [the full experiment log](model-bakeoff.md) for how `pred.json` is
|
||
produced, and the [LVFace deep dive](lvface-deep-dive.md) for what these
|
||
mechanisms look like frame by frame.
|