# scripts/validation — per-scene actor-presence eval

Validates the pipeline's per-scene "who's on screen" output against external
ground truth, offline. Annealing (`anneal_sec`) means an actor's presence is only
defined *after* the whole file is merged into `[start,end]` windows, so we cannot
score live: process → write the pipeline JSON → **sample timepoints** → compare
predicted vs ground-truth presence sets → micro-sum TP/FP/FN → precision/recall/F1.

## Ground-truth sources

| Source | Semantics | Fair to a face pipeline? | What it measures |
| ------ | --------- | ------------------------ | ---------------- |
| **MovieNet-PS** | on-screen **face** presence per shot | yes — like-for-like | recognition accuracy |
| **Amazon X-Ray** (Zenodo) | **cast-in-scene** (incl. off-camera / non-speaking) | no — penalizes by design | coverage ceiling; recall gap = actors we structurally can't see |

- MovieNet is the honest recognition number.
- X-Ray is an upper bound: its recall gap tells you how much presence is off-camera
  cast a face detector can never reach — not a pipeline error.

X-Ray dataset: Zenodo DOI `10.5281/zenodo.17659734` (CC-BY-4.0). Per movie it ships
`people.csv`, `scenes.csv`, `people_in_scenes.csv`.

## Usage

```bash
# against Amazon X-Ray CSVs for one title
python scripts/validation/sample_eval.py \
    --pred "Scene in a Mall.json" \
    --xray /data/xray/<movie_dir> \
    --gallery gallery_arcface_w600k_r50.json \
    --step 1.0

# against MovieNet-PS for one title
python scripts/validation/sample_eval.py \
    --pred out.json \
    --movienet /data/movienet --split Train_app10 --title tt0032138 \
    --gallery gallery_arcface_w600k_r50.json
```

### Sampling modes
- `--step S`      regular grid every S s (default 1.0) — time-weighted headline number.
- `--random N`    N uniform-random timepoints (for confidence intervals).
- `--scene-anchored`  one timepoint per GT scene midpoint — the literal X-Ray
  "did I get this scene's cast right?" question; neutralizes long-scene bias.

Ground truth is compared **raw** (annealing is *not* applied to GT).

## Matching & masking

Identity is provider-agnostic (`identity.py`): each actor is the *set* of every key
we can derive — `imdb:nm…`, `tmdb:…`, `jf:…`, `name:<normalized>`. Predicted and GT
actors match iff their key-sets intersect, so an output carrying only tmdb/jellyfin
ids still joins X-Ray's `nm` ids via the normalized-name fallback.

Scoring is **masked to `gallery ∩ GT`**: a GT actor absent from the gallery is
ignored (not an FN), so we measure pipeline accuracy, not gallery coverage. Without
`--gallery` the mask falls back to `GT ∩ pred` keys. `--no-mask` disables it.

### Exact id join via the tmdb→imdb crosswalk (recommended)

The gallery/pipeline output key actors by **TMDB** id (no `nm…`), while X-Ray and
MovieNet key on **IMDb**. They only overlap on the fuzzy `name:` key by default.
Build a cached `tmdb→imdb` table once and pass it with `--crosswalk` to turn the
name join into an exact id join:

```bash
# one-time: resolve every gallery tmdb id via TMDB /person/{id}/external_ids
python scripts/validation/tmdb_imdb_map.py \
    --gallery gallery_arcface_w600k_r50.json \
    --out scripts/validation/tmdb_imdb.json      # TMDB_API_KEY from env/.env

# then score with exact ids
python scripts/validation/sample_eval.py --pred out.json --xray <dir> \
    --gallery gallery_arcface_w600k_r50.json \
    --crosswalk scripts/validation/tmdb_imdb.json
```

The table caches nulls (tmdb ids TMDB has no IMDb id for) and checkpoints, so a
re-run only resolves new ids. TMDB is authoritative for this crosswalk — there is
no clean free bulk `tmdb_person ↔ nm` file, so we query the API once and cache.

## Minimum face size (VR-005)

`min_face_size.py` is a separate, self-contained study: it needs no video and no
ground truth, only the gallery mugshot cache. It holds out one image per actor,
degrades that probe to each candidate face size and matches it against a gallery
held at **native** resolution, reporting TPI/FPI per size — the measurement that
replaces AR-002's 66×66 px estimate.

```bash
python scripts/validation/min_face_size.py \
    --images images --gallery gallery_lvface.h5 \
    --arcface models/LVFace-B_Glint360K.onnx \
    --actors 100 --out experiments/results/vr005_min_face_size
```

FPI grows with the number of actors competing, so a 100-actor run understates it
against a library of thousands: read FPI as relative across sizes, not as an
absolute rate. Re-run per `--arcface` model to see whether `min_face_px` should be
one constant or scale with the embedder (GR-004).

## Files
- `sample_eval.py`   — CLI scorer.
- `ground_truth.py`  — `XRayGroundTruth`, `MovieNetGroundTruth` loaders.
- `identity.py`      — provider-agnostic match keys.
- `tmdb_imdb_map.py` — build/consult the cached `tmdb→imdb` crosswalk.
- `min_face_size.py` — VR-005 probe-size sweep (see above).
- `test_sample_eval.py` — self-contained tests (`python scripts/validation/test_sample_eval.py`).
