# How we score against X-Ray Every number in this report comes from one comparison, and that comparison has a mismatch at its core: the ground truth is scene-level, the pipeline's output is per-second, and the two do not mean the same thing. This page documents the comparison once so the findings can rely on it. ## What Amazon X-Ray records X-Ray ships three tables per film: `scenes.csv` (a list of `[start, end]` timespans), `people_in_scenes.csv` (which actors are credited in each scene), and `people.csv` (actor identities). There is no per-frame or per-second annotation anywhere in X-Ray. A scene might run 45 seconds, and X-Ray records one cast list for the entire span, not "on screen from second 12 to second 30." To compare against per-second predictions, `second_score.py` expands every scene into per-second ground truth by copying the whole scene's cast list onto every second inside it: ```python for sn, (t0, t1) in spans.items(): cast = scene_cast.get(sn, []) for t in range(int(t0), int(t1)): timeline[t] = cast ``` If X-Ray credits five actors to a 30-second scene, all five count as ground truth present for all 30 seconds, including seconds where only one is on screen. This is not a simplification the pipeline introduces; it is the only reading X-Ray's data supports, because X-Ray records nothing finer. ## How the pipeline reports presence A presence claim is one actor owning one time window. How that window is derived is a tunable choice — a knob the optimizer weighs — with two modes: - **`track_extent` (default).** A claim is exactly `[first_seen, last_seen]` of a track the actor owned (AR-012), ending at the last sighting and never after (AR-013). There is no keep-alive: the withdrawn `anneal_sec` and the scene-tracker `extinction_sec` — which the July report's windows were held open by — are **gone**. A track that survives its own gaps needs no bridge; a gap after the final sighting is never claimed. - **`flood`.** Each claim is snapped to the shot it sits in, so an actor seen once anywhere in a shot is reported for the whole shot `[prev_boundary, next_boundary]`. Boundaries come from TransNetV2 shot detection when available, otherwise from the always-on histogram cut detector (`is_cut`). This trades precision for recall against X-Ray's scene-level granularity, and the optimizer decides per run whether it pays. Do not confuse the surviving `track_extinction_sec` with the withdrawn scene `extinction_sec`: the former bounds how long a lost track stays available for **re-association** (a tracking question), and never extends a presence claim. ## The two limits this does not resolve **The face-vs-presence ceiling.** X-Ray credits scene membership regardless of whether a face is ever visible: background crew, characters shot from behind, voice-only presence. No face pipeline can recover a face that never appears, so recall against X-Ray is a structural ceiling, not a defect. **Flood-fill can overshoot.** Snapping to a shot correctly answers "still in this scene" through an intra-scene cut, but a shot boundary is not a scene boundary: on a film with sparse cuts, flood-fill can carry an actor across a long "shot" they only briefly appeared in. This is why flood-fill is a knob, not a default — its value depends on the film's cut density. ## Precision, recall, and the misID weighting Per sampled second `t`: **TPI** (true positive instances): actors both X-Ray and the pipeline agree are present. **FPI** (false positive instances): actors the pipeline reports that are not in X-Ray's cast for this second, split into: - **FPI_incast**: the actor is in the film's cast, just not credited to this scene. A timing or boundary slip. - **FPI_misid**: the actor is not in the film's cast at all. A genuine wrong-identity error, weighted **10×** in the precision objective, because naming someone not even in the film is categorically worse than a few seconds of scene-boundary slop. !!! note "Every headline `P` and `F1` is misID-weighted" Precision puts each `FPI_misid` into the denominator 10 times (`precision = TPI / (TPI + FPI_incast + 10·FPI_misid)`, [`second_score.py`](https://REPOLINK/scripts/optimizer/second_score.py)). This deliberately punishes naming an out-of-film actor far harder than a boundary slip, so the `P` column is not raw precision and a misID-heavy film's `P` is depressed super-linearly. **FN** (false negatives): actors X-Ray lists that the pipeline never reports, counted **only** for actors who have a gallery reference embedding. An actor with no reference photo can never be recognized, and counting them as a miss would measure gallery coverage, not recognition accuracy. Two further numbers accompany F1: **agreement_rate**: mean per-second Jaccard overlap (`|Pred ∩ GT| / |Pred ∪ GT|`) — partial credit, so naming 2 of 3 present actors scores 2/3, not 0. **exact_match_rate**: the fraction of seconds where the pipeline's named set exactly equals X-Ray's — no partial credit, dominated by recall. ## The benchmark set Unlike the July report — which trained on a 3-film subset and validated on held-out films to keep evaluations fast — this run scores **all 9 films on every evaluation**. The registry one-clock fix and uncapped dumps made full-set replay affordable, so the reported optimum is tuned against the complete set rather than a training subset. ## Reproduce ```bash python3 scripts/optimizer/second_score.py \ --pred pred.json --xray experiments/xray/.../ \ --gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5 ``` See the [full experiment log](model-bakeoff.md) for how `pred.json` is produced and where the shipped `src/config.hpp` defaults come from.