scene-actor-extraction/docs/methodology.md

How we score against X-Ray

Every number in this report, every F1 and misID count, comes from one comparison. The comparison has a mismatch at its core that shapes nearly every finding in this report: the ground truth is scene-level, the pipeline's output is per-second, and the two do not mean the same thing. This page documents that comparison once, so the findings pages can rely on it without re-explaining it.

What Amazon X-Ray records

X-Ray ships three tables per film: scenes.csv (a list of [start, end] timespans), people_in_scenes.csv (which actors are credited in each scene), and people.csv (actor identities). There is no per-frame or per-second annotation anywhere in X-Ray. A scene might run 45 seconds, and X-Ray records one cast list for the entire span, not "on screen from second 12 to second 30."

To compare this against per-second predictions, second_score.py expands every scene into per-second ground truth by copying the whole scene's cast list onto every second inside it:

for sn, (t0, t1) in spans.items():
    cast = scene_cast.get(sn, [])
    for t in range(int(t0), int(t1)):
        timeline[t] = cast

That is the entire mechanism. If X-Ray credits five actors to a 30-second scene, all five count as ground truth present for all 30 seconds, including seconds where only one of them is on screen. This is not a simplification introduced by the pipeline; it is the only reading of X-Ray's data that is possible, because X-Ray itself does not record anything finer-grained.

Why an offscreen name can be scored correct

A name listed under Offscreen with a correct (green) label is not the pipeline guessing or padding its score. It is the pipeline correctly answering the question X-Ray actually asks: is this actor part of this scene. It answers that question using a presence window ([start, end], held open across cuts by anneal_sec and extinction_sec), which matches X-Ray's scene-level semantics more closely than a raw per-frame detection would.

A system that only reported "this actor is visible in this exact frame" would score worse against X-Ray's scene-level ground truth, producing a false negative every time the camera cuts away from a character who is still present in the scene. Not because it is wrong about the world, but because it would be answering a stricter, different question than the one X-Ray's data supports. The presence-window design exists specifically to answer X-Ray's actual question.

What this resolves and what it does not

This resolves the semantic mismatch between a scene and an instant. It does not resolve two other limitations, both discussed in the LVFace deep dive.

The face-vs-presence ceiling. X-Ray credits scene membership regardless of whether a face is ever visible: background crew, characters shot from behind, voice-only presence. No amount of bridging recovers a face that never appears on screen. This is a hard ceiling on recall, not a defect.

Extinction bridging can overshoot. The same presence-window mechanism that correctly answers "still in this scene" during a normal cut can also bridge across a scene boundary it has no way to detect. A hard cut into a different scene with no faces, such as closing credits, carries the previous scene's identities forward until the window expires. This is the mechanism behind Downton Abbey's recall collapse, documented in the deep dive.

Precision, recall, and the misID weighting

Per sampled second t:

TPI (true positive instances): actors both X-Ray and the pipeline agree are present.

FPI (false positive instances): actors the pipeline reports that are not in X-Ray's cast for this second. Split into two categories:

• FPI_incast: the actor is in the film's cast, just not credited to this particular scene. A timing or boundary slip.
• FPI_misid: the actor is not in the film's cast at all. A genuine wrong-identity error, weighted 10x in the precision objective, because naming someone who is not even in the film is a categorically worse error than a few seconds of scene-boundary slop.

!!! note "Every headline P and F1 is misID-weighted"

The precision reported throughout this report, and therefore the F1
derived from it, puts each `FPI_misid` into the denominator **10 times**
(`precision = TPI / (TPI + FPI_incast + 10·FPI_misid)`,
[`second_score.py`](https://REPOLINK/scripts/optimizer/second_score.py)).
This is deliberate: the whole point is to punish naming an out-of-film
actor far harder than a scene-boundary slip. But it means the `P` column
is not raw precision, and a misID-heavy film's `P` is depressed
super-linearly. `second_score.py` also emits an unweighted `precision_raw`
(always ≥ the weighted `P`); where the gap matters, The Many Saints of
Newark, weighted `P` 54.7% vs. raw 78.4%, the [LVFace deep dive](lvface-deep-dive.md)
reports both. When comparing `P` across films, remember you are comparing a
quantity that penalizes misIDs, not just a hit rate.

FN (false negatives): actors X-Ray lists that the pipeline never reports, counted only for actors who have a gallery reference embedding. Across the 9-film benchmark, coverage of X-Ray's credited cast ranges from 20% to 79% by film (see the full experiment log); an actor with no reference photo can never be recognized regardless of model quality, and counting them as a miss would penalize gallery coverage, not recognition accuracy.

Two further numbers are reported alongside F1:

agreement_rate: mean per-second Jaccard overlap (|Pred ∩ GT| / |Pred ∪ GT|), partial credit. Naming 2 of 3 present actors scores 2/3, not 0.

exact_match_rate: the fraction of sampled seconds where the pipeline's named set exactly equals X-Ray's, no partial credit. Far harsher, and dominated by recall, since any single missed actor zeroes that second.

Reproduce

python3 scripts/optimizer/second_score.py \
    --pred pred.json --xray experiments/xray/.../<xray_dir> \
    --gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5

See also the full experiment log for how pred.json is produced, and the LVFace deep dive for what these mechanisms look like frame by frame.