# scripts/validation — per-scene actor-presence eval

Validates the pipeline's per-scene "who's on screen" output against external
ground truth, offline. Annealing (`anneal_sec`) means an actor's presence is only
defined *after* the whole file is merged into `[start,end]` windows, so we cannot
score live: process → write the pipeline JSON → **sample timepoints** → compare
predicted vs ground-truth presence sets → micro-sum TP/FP/FN → precision/recall/F1.

## Ground-truth sources

| Source | Semantics | Fair to a face pipeline? | What it measures |
| ------ | --------- | ------------------------ | ---------------- |
| **MovieNet-PS** | on-screen **face** presence per shot | yes — like-for-like | recognition accuracy |
| **Amazon X-Ray** (Zenodo) | **cast-in-scene** (incl. off-camera / non-speaking) | no — penalizes by design | coverage ceiling; recall gap = actors we structurally can't see |

- MovieNet is the honest recognition number.
- X-Ray is an upper bound: its recall gap tells you how much presence is off-camera
  cast a face detector can never reach — not a pipeline error.

X-Ray dataset: Zenodo DOI `10.5281/zenodo.17659734` (CC-BY-4.0). Per movie it ships
`people.csv`, `scenes.csv`, `people_in_scenes.csv`.

## Usage

```bash
# against Amazon X-Ray CSVs for one title
python scripts/validation/sample_eval.py \
    --pred "Scene in a Mall.json" \
    --xray /data/xray/<movie_dir> \
    --gallery gallery_arcface_w600k_r50.json \
    --step 1.0

# against MovieNet-PS for one title
python scripts/validation/sample_eval.py \
    --pred out.json \
    --movienet /data/movienet --split Train_app10 --title tt0032138 \
    --gallery gallery_arcface_w600k_r50.json
```

### Sampling modes
- `--step S`      regular grid every S s (default 1.0) — time-weighted headline number.
- `--random N`    N uniform-random timepoints (for confidence intervals).
- `--scene-anchored`  one timepoint per GT scene midpoint — the literal X-Ray
  "did I get this scene's cast right?" question; neutralizes long-scene bias.

Ground truth is compared **raw** (annealing is *not* applied to GT).

## Matching & masking

Identity is provider-agnostic (`identity.py`): each actor is the *set* of every key
we can derive — `imdb:nm…`, `tmdb:…`, `jf:…`, `name:<normalized>`. Predicted and GT
actors match iff their key-sets intersect, so an output carrying only tmdb/jellyfin
ids still joins X-Ray's `nm` ids via the normalized-name fallback.

Scoring is **masked to `gallery ∩ GT`**: a GT actor absent from the gallery is
ignored (not an FN), so we measure pipeline accuracy, not gallery coverage. Without
`--gallery` the mask falls back to `GT ∩ pred` keys. `--no-mask` disables it.

### Exact id join via the tmdb→imdb crosswalk (recommended)

The gallery/pipeline output key actors by **TMDB** id (no `nm…`), while X-Ray and
MovieNet key on **IMDb**. They only overlap on the fuzzy `name:` key by default.
Build a cached `tmdb→imdb` table once and pass it with `--crosswalk` to turn the
name join into an exact id join:

```bash
# one-time: resolve every gallery tmdb id via TMDB /person/{id}/external_ids
python scripts/validation/tmdb_imdb_map.py \
    --gallery gallery_arcface_w600k_r50.json \
    --out scripts/validation/tmdb_imdb.json      # TMDB_API_KEY from env/.env

# then score with exact ids
python scripts/validation/sample_eval.py --pred out.json --xray <dir> \
    --gallery gallery_arcface_w600k_r50.json \
    --crosswalk scripts/validation/tmdb_imdb.json
```

The table caches nulls (tmdb ids TMDB has no IMDb id for) and checkpoints, so a
re-run only resolves new ids. TMDB is authoritative for this crosswalk — there is
no clean free bulk `tmdb_person ↔ nm` file, so we query the API once and cache.

## Files
- `sample_eval.py`   — CLI scorer.
- `ground_truth.py`  — `XRayGroundTruth`, `MovieNetGroundTruth` loaders.
- `identity.py`      — provider-agnostic match keys.
- `tmdb_imdb_map.py` — build/consult the cached `tmdb→imdb` crosswalk.
- `test_sample_eval.py` — self-contained tests (`python scripts/validation/test_sample_eval.py`).