scene-actor-extraction/docs/methodology.md

# How we score against X-Ray

Every number in this report comes from one comparison, and that comparison
has a mismatch at its core: the ground truth is scene-level, the pipeline's
output is per-second, and the two do not mean the same thing. This page
documents the comparison once so the findings can rely on it.

## What Amazon X-Ray records

X-Ray ships three tables per film: `scenes.csv` (a list of `[start, end]`
timespans), `people_in_scenes.csv` (which actors are credited in each
scene), and `people.csv` (actor identities). There is no per-frame or
per-second annotation anywhere in X-Ray. A scene might run 45 seconds, and
X-Ray records one cast list for the entire span, not "on screen from second
12 to second 30."

To compare against per-second predictions, `second_score.py` expands every
scene into per-second ground truth by copying the whole scene's cast list
onto every second inside it:

```python
for sn, (t0, t1) in spans.items():
    cast = scene_cast.get(sn, [])
    for t in range(int(t0), int(t1)):
        timeline[t] = cast
```

If X-Ray credits five actors to a 30-second scene, all five count as ground
truth present for all 30 seconds, including seconds where only one is on
screen. This is not a simplification the pipeline introduces; it is the only
reading X-Ray's data supports, because X-Ray records nothing finer.

## How the pipeline reports presence

A presence claim is one actor owning one time window. How that window is
derived is a tunable choice — a knob the optimizer weighs — with two modes:

- **`track_extent` (default).** A claim is exactly `[first_seen, last_seen]`
  of a track the actor owned (AR-012), ending at the last sighting and never
  after (AR-013). There is no keep-alive: the withdrawn `anneal_sec` and the
  scene-tracker `extinction_sec` — which the July report's windows were held
  open by — are **gone**. A track that survives its own gaps needs no bridge;
  a gap after the final sighting is never claimed.
- **`flood`.** Each claim is snapped to the shot it sits in, so an actor seen
  once anywhere in a shot is reported for the whole shot
  `[prev_boundary, next_boundary]`. Boundaries come from TransNetV2 shot
  detection when available, otherwise from the always-on histogram cut
  detector (`is_cut`). This trades precision for recall against X-Ray's
  scene-level granularity, and the optimizer decides per run whether it pays.

Do not confuse the surviving `track_extinction_sec` with the withdrawn
scene `extinction_sec`: the former bounds how long a lost track stays
available for **re-association** (a tracking question), and never extends a
presence claim.

## The two limits this does not resolve

**The face-vs-presence ceiling.** X-Ray credits scene membership regardless
of whether a face is ever visible: background crew, characters shot from
behind, voice-only presence. No face pipeline can recover a face that never
appears, so recall against X-Ray is a structural ceiling, not a defect.

**Flood-fill can overshoot.** Snapping to a shot correctly answers "still in
this scene" through an intra-scene cut, but a shot boundary is not a scene
boundary: on a film with sparse cuts, flood-fill can carry an actor across a
long "shot" they only briefly appeared in. This is why flood-fill is a knob,
not a default — its value depends on the film's cut density.

## Precision, recall, and the misID weighting

Per sampled second `t`:

**TPI** (true positive instances): actors both X-Ray and the pipeline agree
are present.

**FPI** (false positive instances): actors the pipeline reports that are not
in X-Ray's cast for this second, split into:

- **FPI_incast**: the actor is in the film's cast, just not credited to this
  scene. A timing or boundary slip.
- **FPI_misid**: the actor is not in the film's cast at all. A genuine
  wrong-identity error, weighted **10×** in the precision objective, because
  naming someone not even in the film is categorically worse than a few
  seconds of scene-boundary slop.

!!! note "Every headline `P` and `F1` is misID-weighted"

    Precision puts each `FPI_misid` into the denominator 10 times
    (`precision = TPI / (TPI + FPI_incast + 10·FPI_misid)`,
    [`second_score.py`](https://REPOLINK/scripts/optimizer/second_score.py)).
    This deliberately punishes naming an out-of-film actor far harder than a
    boundary slip, so the `P` column is not raw precision and a misID-heavy
    film's `P` is depressed super-linearly.

**FN** (false negatives): actors X-Ray lists that the pipeline never reports,
counted **only** for actors who have a gallery reference embedding. An actor
with no reference photo can never be recognized, and counting them as a miss
would measure gallery coverage, not recognition accuracy.

Two further numbers accompany F1:

**agreement_rate**: mean per-second Jaccard overlap
(`|Pred ∩ GT| / |Pred ∪ GT|`) — partial credit, so naming 2 of 3 present
actors scores 2/3, not 0.

**exact_match_rate**: the fraction of seconds where the pipeline's named set
exactly equals X-Ray's — no partial credit, dominated by recall.

## The benchmark set

Unlike the July report — which trained on a 3-film subset and validated on
held-out films to keep evaluations fast — this run scores **all 9 films on
every evaluation**. The registry one-clock fix and uncapped dumps made
full-set replay affordable, so the reported optimum is tuned against the
complete set rather than a training subset.

## Reproduce

```bash
python3 scripts/optimizer/second_score.py \
    --pred pred.json --xray experiments/xray/.../<xray_dir> \
    --gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5
```

See the [full experiment log](model-bakeoff.md) for how `pred.json` is
produced and where the shipped `src/config.hpp` defaults come from.