scene-actor-extraction/docs/methodology-2026-07.md

> **Archived (July 2026).** This report covers the pre-opencv5 framework and the 4-model ArcFace/LVFace bake-off. It is superseded by the current [experiment log](model-bakeoff.md) for the opencv5 build. Kept for provenance; the numbers here are historical.

# How we score against X-Ray

Every number in this report, every F1 and misID count, comes from one
comparison. The comparison has a mismatch at its core that shapes nearly
every finding in this report: the ground truth is scene-level, the
pipeline's output is per-second, and the two do not mean the same thing.
This page documents that comparison once, so the findings pages can rely on
it without re-explaining it.

## What Amazon X-Ray records

X-Ray ships three tables per film: `scenes.csv` (a list of `[start, end]`
timespans), `people_in_scenes.csv` (which actors are credited in each
scene), and `people.csv` (actor identities). There is no per-frame or
per-second annotation anywhere in X-Ray. A scene might run 45 seconds, and
X-Ray records one cast list for the entire span, not "on screen from
second 12 to second 30."

To compare this against per-second predictions, `second_score.py` expands
every scene into per-second ground truth by copying the whole scene's cast
list onto every second inside it:

```python
for sn, (t0, t1) in spans.items():
    cast = scene_cast.get(sn, [])
    for t in range(int(t0), int(t1)):
        timeline[t] = cast
```

That is the entire mechanism. If X-Ray credits five actors to a 30-second
scene, all five count as ground truth present for all 30 seconds, including
seconds where only one of them is on screen. This is not a simplification
introduced by the pipeline; it is the only reading of X-Ray's data that is
possible, because X-Ray itself does not record anything finer-grained.

## Why an offscreen name can be scored correct

A name listed under Offscreen with a correct (green) label is not the
pipeline guessing or padding its score. It is the pipeline correctly
answering the question X-Ray actually asks: is this actor part of this
scene. It answers that question using a presence window (`[start, end]`,
held open across cuts by `anneal_sec` and `extinction_sec`), which matches
X-Ray's scene-level semantics more closely than a raw per-frame detection
would.

A system that only reported "this actor is visible in this exact frame"
would score worse against X-Ray's scene-level ground truth, producing a
false negative every time the camera cuts away from a character who is
still present in the scene. Not because it is wrong about the world, but
because it would be answering a stricter, different question than the one
X-Ray's data supports. The presence-window design exists specifically to
answer X-Ray's actual question.

## What this resolves and what it does not

This resolves the semantic mismatch between a scene and an instant. It does
not resolve two other limitations, both discussed in the
[LVFace deep dive](lvface-deep-dive-2026-07.md).

**The face-vs-presence ceiling.** X-Ray credits scene membership regardless
of whether a face is ever visible: background crew, characters shot from
behind, voice-only presence. No amount of bridging recovers a face that
never appears on screen. This is a hard ceiling on recall, not a defect.

**Extinction bridging can overshoot.** The same presence-window mechanism
that correctly answers "still in this scene" during a normal cut can also
bridge across a scene boundary it has no way to detect. A hard cut into a
different scene with no faces, such as closing credits, carries the
previous scene's identities forward until the window expires. This is the
mechanism behind Downton Abbey's recall collapse, documented in the deep
dive.

## Precision, recall, and the misID weighting

Per sampled second `t`:

**TPI** (true positive instances): actors both X-Ray and the pipeline agree
are present.

**FPI** (false positive instances): actors the pipeline reports that are
not in X-Ray's cast for this second. Split into two categories:

- **FPI_incast**: the actor is in the film's cast, just not credited to
  this particular scene. A timing or boundary slip.
- **FPI_misid**: the actor is not in the film's cast at all. A genuine
  wrong-identity error, weighted 10x in the precision objective, because
  naming someone who is not even in the film is a categorically worse
  error than a few seconds of scene-boundary slop.

!!! note "Every headline `P` and `F1` is misID-weighted"

    The precision reported throughout this report, and therefore the F1
    derived from it, puts each `FPI_misid` into the denominator **10 times**
    (`precision = TPI / (TPI + FPI_incast + 10·FPI_misid)`,
    [`second_score.py`](https://REPOLINK/scripts/optimizer/second_score.py)).
    This is deliberate: the whole point is to punish naming an out-of-film
    actor far harder than a scene-boundary slip. But it means the `P` column
    is not raw precision, and a misID-heavy film's `P` is depressed
    super-linearly. `second_score.py` also emits an unweighted `precision_raw`
    (always ≥ the weighted `P`); where the gap matters, The Many Saints of
    Newark, weighted `P` 54.7% vs. raw 78.4%, the [LVFace deep dive](lvface-deep-dive-2026-07.md)
    reports both. When comparing `P` across films, remember you are comparing a
    quantity that penalizes misIDs, not just a hit rate.

**FN** (false negatives): actors X-Ray lists that the pipeline never
reports, counted only for actors who have a gallery reference embedding.
Across the 9-film benchmark, coverage of X-Ray's credited cast ranges from
20% to 79% by film (see
[the full experiment log](model-bakeoff-2026-07.md#gallery-coverage-per-film)); an
actor with no reference photo can never be recognized regardless of model
quality, and counting them as a miss would penalize gallery coverage, not
recognition accuracy.

Two further numbers are reported alongside F1:

**agreement_rate**: mean per-second Jaccard overlap
(`|Pred ∩ GT| / |Pred ∪ GT|`), partial credit. Naming 2 of 3 present actors
scores 2/3, not 0.

**exact_match_rate**: the fraction of sampled seconds where the pipeline's
named set exactly equals X-Ray's, no partial credit. Far harsher, and
dominated by recall, since any single missed actor zeroes that second.

## Reproduce

```bash
python3 scripts/optimizer/second_score.py \
    --pred pred.json --xray experiments/xray/.../<xray_dir> \
    --gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5
```

See also [the full experiment log](model-bakeoff-2026-07.md) for how `pred.json` is
produced, and the [LVFace deep dive](lvface-deep-dive-2026-07.md) for what these
mechanisms look like frame by frame.