feat(tooling): X-Ray threshold optimizer, gallery utilities, artifact registry, docs build

Optimizer (scripts/optimizer/): replay.py runs the real C++ tracker/matcher/
scene_tracker chain over a dumped-embeddings HDF5 via sae_kpn, so a threshold
sweep never re-decodes video or re-embeds faces. optimize.py drives scipy's
differential_evolution over the knob space, with DE-level parallelism
(multiple population candidates evaluated concurrently via a ThreadPoolExecutor)
on top of per-film replay parallelism. second_score.py is the per-second X-Ray
scoring metric (TPI/FPI/FN, out-of-cast misID weighted 10x, fair recall masked
to gallery-known cast) that superseded an earlier scene-union metric.
dump_error_frames.py / dump_scene_montage.py extract annotated video frames
(bounding boxes, TPI/FPI/FN captions, onscreen-vs-offscreen split) for visual
review of a replay against ground truth. Gallery utilities: cast_restrict.py,
gallery_membership.py, fetch_missing_actors.py, reembed_gallery.py.

scripts/validation/: X-Ray ground-truth loading and provider-agnostic identity
matching (identity.py's keys_for — an actor is the union of every id we can
derive, since pipeline output and ground truth don't share one id space).

scripts/artifacts/: push/pull scripts for the Gitea generic package registry —
galleries, montage frames, and experiment data (manifests/trajectories/results)
are pushed there instead of committed, since none are needed to run the app,
only benchmarks. Versioned by git short-SHA.

scripts/docs/: MkDocs site build (build_site.sh) and the calibration-curve
comparison chart (calibration_chart.py, matplotlib, reads each gallery's
embedded calibration).

Gallery-building scripts (make_jellyfin_gallery.py, make_gallery.py,
filter_gallery.py, run_from_jellyfin.py, movienet_eval.py, movienet_prep.py,
sae_gallery.py) updated to read/write HDF5 galleries exclusively, matching the
engine-side format switch. run_from_jellyfin.py and the optimizer no longer
carry movie source paths in shared manifests (some source filenames include
scene-release tags) — resolved locally via a gitignored file-lut.json instead.

scripts/validation/README.md

@@ -0,0 +1,87 @@
+# scripts/validation — per-scene actor-presence eval
+
+Validates the pipeline's per-scene "who's on screen" output against external
+ground truth, offline. Annealing (`anneal_sec`) means an actor's presence is only
+defined *after* the whole file is merged into `[start,end]` windows, so we cannot
+score live: process → write the pipeline JSON → **sample timepoints** → compare
+predicted vs ground-truth presence sets → micro-sum TP/FP/FN → precision/recall/F1.
+
+## Ground-truth sources
+
+| Source | Semantics | Fair to a face pipeline? | What it measures |
+| ------ | --------- | ------------------------ | ---------------- |
+| **MovieNet-PS** | on-screen **face** presence per shot | yes — like-for-like | recognition accuracy |
+| **Amazon X-Ray** (Zenodo) | **cast-in-scene** (incl. off-camera / non-speaking) | no — penalizes by design | coverage ceiling; recall gap = actors we structurally can't see |
+
+- MovieNet is the honest recognition number.
+- X-Ray is an upper bound: its recall gap tells you how much presence is off-camera
+  cast a face detector can never reach — not a pipeline error.
+
+X-Ray dataset: Zenodo DOI `10.5281/zenodo.17659734` (CC-BY-4.0). Per movie it ships
+`people.csv`, `scenes.csv`, `people_in_scenes.csv`.
+
+## Usage
+
+```bash
+# against Amazon X-Ray CSVs for one title
+python scripts/validation/sample_eval.py \
+    --pred "Scene in a Mall.json" \
+    --xray /data/xray/<movie_dir> \
+    --gallery gallery_arcface_w600k_r50.json \
+    --step 1.0
+
+# against MovieNet-PS for one title
+python scripts/validation/sample_eval.py \
+    --pred out.json \
+    --movienet /data/movienet --split Train_app10 --title tt0032138 \
+    --gallery gallery_arcface_w600k_r50.json
+```
+
+### Sampling modes
+- `--step S`      regular grid every S s (default 1.0) — time-weighted headline number.
+- `--random N`    N uniform-random timepoints (for confidence intervals).
+- `--scene-anchored`  one timepoint per GT scene midpoint — the literal X-Ray
+  "did I get this scene's cast right?" question; neutralizes long-scene bias.
+
+Ground truth is compared **raw** (annealing is *not* applied to GT).
+
+## Matching & masking
+
+Identity is provider-agnostic (`identity.py`): each actor is the *set* of every key
+we can derive — `imdb:nm…`, `tmdb:…`, `jf:…`, `name:<normalized>`. Predicted and GT
+actors match iff their key-sets intersect, so an output carrying only tmdb/jellyfin
+ids still joins X-Ray's `nm` ids via the normalized-name fallback.
+
+Scoring is **masked to `gallery ∩ GT`**: a GT actor absent from the gallery is
+ignored (not an FN), so we measure pipeline accuracy, not gallery coverage. Without
+`--gallery` the mask falls back to `GT ∩ pred` keys. `--no-mask` disables it.
+
+### Exact id join via the tmdb→imdb crosswalk (recommended)
+
+The gallery/pipeline output key actors by **TMDB** id (no `nm…`), while X-Ray and
+MovieNet key on **IMDb**. They only overlap on the fuzzy `name:` key by default.
+Build a cached `tmdb→imdb` table once and pass it with `--crosswalk` to turn the
+name join into an exact id join:
+
+```bash
+# one-time: resolve every gallery tmdb id via TMDB /person/{id}/external_ids
+python scripts/validation/tmdb_imdb_map.py \
+    --gallery gallery_arcface_w600k_r50.json \
+    --out scripts/validation/tmdb_imdb.json      # TMDB_API_KEY from env/.env
+
+# then score with exact ids
+python scripts/validation/sample_eval.py --pred out.json --xray <dir> \
+    --gallery gallery_arcface_w600k_r50.json \
+    --crosswalk scripts/validation/tmdb_imdb.json
+```
+
+The table caches nulls (tmdb ids TMDB has no IMDb id for) and checkpoints, so a
+re-run only resolves new ids. TMDB is authoritative for this crosswalk — there is
+no clean free bulk `tmdb_person ↔ nm` file, so we query the API once and cache.
+
+## Files
+- `sample_eval.py`   — CLI scorer.
+- `ground_truth.py`  — `XRayGroundTruth`, `MovieNetGroundTruth` loaders.
+- `identity.py`      — provider-agnostic match keys.
+- `tmdb_imdb_map.py` — build/consult the cached `tmdb→imdb` crosswalk.
+- `test_sample_eval.py` — self-contained tests (`python scripts/validation/test_sample_eval.py`).

scripts/validation/ground_truth.py

@@ -0,0 +1,221 @@
+#!/usr/bin/env python3
+"""
+ground_truth.py — pluggable ground-truth loaders for per-scene presence eval.
+
+A ground truth is a timeline of "who is present when", exposed as:
+
+    GroundTruth.present_at(t: float) -> set[str]      # match-keys present at time t
+    GroundTruth.scene_windows()     -> list[(t0, t1)] # scene spans (for --scene-anchored)
+    GroundTruth.all_keys()          -> set[str]        # every actor the GT knows (for masking)
+
+"Match-keys" are provider-agnostic identity tokens (see identity.py): an actor is
+represented by *all* the keys we can derive (nm-id, tmdb-id, jellyfin-id, normalized
+name), so predicted and GT sets intersect if they agree on *any* shared id space.
+This matters because the pipeline output may carry only tmdb/jellyfin ids while
+X-Ray/MovieNet key on IMDb nm-ids — see [[per-scene-presence-eval-design]].
+
+Two sources implemented:
+  * XRayGroundTruth  — Zenodo scene-level Amazon X-Ray CSVs (cast-in-scene).
+  * MovieNetGroundTruth — MovieNet-PS per-shot face annotations (on-screen faces).
+"""
+
+from __future__ import annotations
+
+import csv
+import sys
+from bisect import bisect_right
+from pathlib import Path
+
+from identity import keys_for
+
+
+class GroundTruth:
+    """Base: a set of actors, each with presence intervals [(t0,t1), ...] in seconds."""
+
+    def __init__(self) -> None:
+        # actor_id (any stable local id) -> {"keys": set[str], "intervals": [(t0,t1)]}
+        self._actors: dict[str, dict] = {}
+        self._scene_spans: list[tuple[float, float]] = []
+
+    # -- construction helpers ------------------------------------------------
+    def _add_interval(self, actor_id: str, keys: set[str], t0: float, t1: float) -> None:
+        a = self._actors.setdefault(actor_id, {"keys": set(), "intervals": []})
+        a["keys"] |= keys
+        a["intervals"].append((float(t0), float(t1)))
+
+    def _finalize(self) -> None:
+        """Sort intervals and precompute a flat sorted start-array per actor."""
+        for a in self._actors.values():
+            a["intervals"].sort()
+            a["_starts"] = [iv[0] for iv in a["intervals"]]
+        self._scene_spans.sort()
+
+    # -- query API -----------------------------------------------------------
+    def present_at(self, t: float) -> set[frozenset[str]]:
+        """Set of actors present at t; each actor is its (frozen) key-set."""
+        out: set[frozenset[str]] = set()
+        for a in self._actors.values():
+            ivs = a["intervals"]
+            i = bisect_right(a["_starts"], t)  # first interval starting after t
+            # walk back over intervals that started at/before t
+            j = i - 1
+            while j >= 0:
+                t0, t1 = ivs[j]
+                if t1 >= t:
+                    out.add(frozenset(a["keys"]))
+                    break
+                # intervals sorted by start; an earlier one could still cover t,
+                # but since we only need membership, keep scanning a bounded window.
+                j -= 1
+                if i - j > 8:  # bound: overlapping intervals per actor are rare
+                    break
+        return out
+
+    def scene_windows(self) -> list[tuple[float, float]]:
+        return self._scene_spans
+
+    def all_keys(self) -> set[str]:
+        out: set[str] = set()
+        for a in self._actors.values():
+            out |= a["keys"]
+        return out
+
+    def summary(self) -> str:
+        n_iv = sum(len(a["intervals"]) for a in self._actors.values())
+        return (f"{len(self._actors)} actors, {n_iv} intervals, "
+                f"{len(self._scene_spans)} scenes")
+
+
+# ── Amazon X-Ray (Zenodo) ─────────────────────────────────────────────────────
+
+class XRayGroundTruth(GroundTruth):
+    """
+    Load one movie's X-Ray CSVs (Zenodo DOI 10.5281/zenodo.17659734).
+    Real schema (columns are milliseconds):
+      people.csv            name_id (nm...), person, character
+      scenes.csv            scene, start, end                  (ms)
+      people_in_scenes.csv  scene, start, end, name_id, timestamp   (ms)
+
+    Presence = whole scene span for every character listed in that scene.
+    Semantics: cast-in-scene (incl. off-camera) — a recall ceiling, not accuracy.
+    Columns are resolved case-insensitively so minor variants still load.
+    """
+
+    def __init__(self, movie_dir: str | Path) -> None:
+        super().__init__()
+        d = Path(movie_dir)
+        people = _read_csv(d / "people.csv")
+        scenes = _read_csv(d / "scenes.csv")
+        pis = _read_csv(d / "people_in_scenes.csv")
+
+        # nm-id -> actor name (for building match keys)
+        nm_col_p = _find_col(people, "name_id", "nm", "imdb")
+        name_col = _find_col(people, "person", "actor")  # actor name lives in "person"
+        id_to_name: dict[str, str] = {}
+        for row in people:
+            nm = (row.get(nm_col_p) or "").strip()
+            if nm:
+                id_to_name[nm] = (row.get(name_col) or "").strip()
+
+        # scene number -> (t0_sec, t1_sec)
+        scene_col_s = _find_col(scenes, "scene")
+        start_col = _find_col(scenes, "start")
+        end_col = _find_col(scenes, "end")
+        span: dict[str, tuple[float, float]] = {}
+        for row in scenes:
+            sn = (row.get(scene_col_s) or "").strip()
+            t0 = _ms_to_sec(row.get(start_col))
+            t1 = _ms_to_sec(row.get(end_col))
+            if sn and t0 is not None and t1 is not None:
+                span[sn] = (t0, t1)
+                self._scene_spans.append((t0, t1))
+
+        # scene number -> [nm ids present]
+        scene_col_pis = _find_col(pis, "scene")
+        nm_col_pis = _find_col(pis, "name_id", "nm", "imdb")
+        for row in pis:
+            sn = (row.get(scene_col_pis) or "").strip()
+            nm = (row.get(nm_col_pis) or "").strip()
+            if sn not in span or not nm:
+                continue
+            t0, t1 = span[sn]
+            self._add_interval(nm, keys_for(imdb_id=nm, name=id_to_name.get(nm)), t0, t1)
+
+        self._finalize()
+
+
+# ── MovieNet-PS ────────────────────────────────────────────────────────────────
+
+class MovieNetGroundTruth(GroundTruth):
+    """
+    Build presence from MovieNet-PS per-shot face annotations for a single title.
+
+    Input: the flat annotation list produced by movienet_prep.load_movienet_annotations
+    filtered to one movie (tt-id), plus a shot->time map. Because MovieNet frames are
+    named tt.../shot_XXXX_img_Y.jpg with no absolute timestamp, presence is expressed
+    in *shot index* units unless a fps/shot-duration map is supplied. For the sampler
+    we therefore sample at shot granularity (one timepoint per annotated shot).
+
+    Semantics: on-screen face presence per shot — like-for-like fair benchmark.
+    """
+
+    def __init__(self, annotations: list[dict], id_to_name: dict[str, str] | None = None,
+                 shot_seconds: float = 1.0) -> None:
+        super().__init__()
+        id_to_name = id_to_name or {}
+        # group by shot index; each annotated shot becomes a unit interval on a
+        # synthetic timeline (shot_index * shot_seconds).
+        shots: dict[int, set[str]] = {}
+        for ann in annotations:
+            shot = _shot_index(ann["img_path"])
+            if shot is None:
+                continue
+            shots.setdefault(shot, set()).add(ann["imdb_id"])
+
+        for shot, nm_ids in shots.items():
+            t0 = shot * shot_seconds
+            t1 = t0 + shot_seconds
+            self._scene_spans.append((t0, t1))
+            for nm in nm_ids:
+                self._add_interval(nm, keys_for(imdb_id=nm, name=id_to_name.get(nm)), t0, t1)
+
+        self._finalize()
+
+
+# ── small parsing helpers ──────────────────────────────────────────────────────
+
+def _read_csv(path: Path) -> list[dict]:
+    if not path.exists():
+        raise FileNotFoundError(f"expected X-Ray CSV not found: {path}")
+    with open(path, newline="", encoding="utf-8") as f:
+        return list(csv.DictReader(f))
+
+
+def _find_col(rows: list[dict], *keywords: str) -> str:
+    """Return the first column whose lowercased name contains all keywords of any
+    single keyword group. We try each keyword in order and accept the first hit."""
+    if not rows:
+        raise ValueError("empty CSV — cannot resolve columns")
+    cols = list(rows[0].keys())
+    low = {c: c.lower() for c in cols}
+    for kw in keywords:
+        for c in cols:
+            if kw in low[c]:
+                return c
+    raise KeyError(f"no column matching {keywords} in {cols}")
+
+
+def _ms_to_sec(v) -> float | None:
+    if v is None or str(v).strip() == "":
+        return None
+    try:
+        return float(v) / 1000.0
+    except ValueError:
+        return None
+
+
+def _shot_index(img_path: str) -> int | None:
+    # tt0032138/shot_0003_img_1.jpg -> 3
+    import re
+    m = re.search(r"shot_(\d+)", img_path)
+    return int(m.group(1)) if m else None

scripts/validation/identity.py

@@ -0,0 +1,54 @@
+#!/usr/bin/env python3
+"""
+identity.py — provider-agnostic match keys.
+
+The pipeline output and the ground truth may not share one id space: an output
+actor can carry only tmdb/jellyfin ids while X-Ray/MovieNet key on IMDb nm-ids.
+We represent each actor by the *set* of every key we can derive, and treat two
+actors as the same iff their key sets intersect. Namespacing each key by its
+provider prevents cross-provider collisions (e.g. an nm-number equalling a
+tmdb-number).
+"""
+
+from __future__ import annotations
+
+import re
+import unicodedata
+
+
+def norm_name(name: str | None) -> str | None:
+    """Lowercased, accent-stripped, punctuation-free name for fuzzy fallback match."""
+    if not name:
+        return None
+    s = unicodedata.normalize("NFKD", name)
+    s = "".join(c for c in s if not unicodedata.combining(c))
+    s = re.sub(r"[^a-z0-9 ]+", "", s.lower()).strip()
+    s = re.sub(r"\s+", " ", s)
+    return s or None
+
+
+def keys_for(imdb_id: str | None = None,
+             tmdb_id: str | None = None,
+             jellyfin_id: str | None = None,
+             name: str | None = None,
+             crosswalk=None) -> set[str]:
+    """All identity tokens for one actor. Empty strings are ignored.
+
+    If `crosswalk` (a CrosswalkTable) is given and no imdb_id is present, resolve
+    tmdb_id → imdb_id through it so a tmdb-only actor still gets an exact `imdb:`
+    key — turning the fuzzy name join into an exact id join. See tmdb_imdb_map.py.
+    """
+    keys: set[str] = set()
+    imdb = imdb_id.strip() if (imdb_id and imdb_id.strip()) else None
+    if not imdb and crosswalk is not None and tmdb_id:
+        imdb = crosswalk.imdb_for(tmdb_id)
+    if imdb:
+        keys.add(f"imdb:{imdb}")
+    if tmdb_id and str(tmdb_id).strip():
+        keys.add(f"tmdb:{str(tmdb_id).strip()}")
+    if jellyfin_id and jellyfin_id.strip():
+        keys.add(f"jf:{jellyfin_id.strip()}")
+    nn = norm_name(name)
+    if nn:
+        keys.add(f"name:{nn}")
+    return keys

scripts/validation/sample_eval.py

@@ -0,0 +1,313 @@
+#!/usr/bin/env python3
+"""
+sample_eval.py — offline per-scene presence eval by timepoint sampling.
+
+Annealing (anneal_sec) means an actor is "present" only after the whole file is
+merged into [start,end] windows, so we cannot score live: we process → write the
+pipeline JSON → sample timepoints → compare predicted vs ground-truth presence
+sets → micro-sum TP/FP/FN → precision / recall / F1.
+See [[per-scene-presence-eval-design]].
+
+Usage:
+  # against Amazon X-Ray CSVs (Zenodo)
+  python scripts/validation/sample_eval.py \
+      --pred "Scene in a Mall.json" \
+      --xray  /data/xray/tt0384766 \
+      --step 1.0
+
+  # against MovieNet-PS (needs the .mat split + a title tt-id)
+  python scripts/validation/sample_eval.py \
+      --pred out.json \
+      --movienet /data/movienet --split Train_app10 --title tt0032138
+
+Sampling:
+  --step S        regular grid every S seconds (default 1.0) — time-weighted headline
+  --random N      N uniform-random timepoints instead of a grid (for CIs)
+  --scene-anchored  one timepoint at each GT scene midpoint (X-Ray "per-scene" question)
+
+Masking: scoring is restricted to actors present in BOTH the pipeline gallery
+(--gallery) AND the ground truth. A GT actor absent from the gallery is ignored
+(not counted as a miss) so we measure pipeline accuracy, not gallery coverage.
+Pass --no-mask to disable.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import random
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+from identity import keys_for                       # noqa: E402
+from ground_truth import XRayGroundTruth, MovieNetGroundTruth  # noqa: E402
+from tmdb_imdb_map import CrosswalkTable             # noqa: E402
+
+
+# ── pipeline output → presence timeline ────────────────────────────────────────
+
+class Prediction:
+    """Pipeline output (minimal/standard schema) as per-actor presence windows."""
+
+    def __init__(self, path: str | Path, crosswalk=None) -> None:
+        with open(path) as f:
+            data = json.load(f)
+        self.movie = data.get("movie", "")
+        self.anneal_sec = data.get("anneal_sec")
+        self.actors: list[dict] = []
+        self._max_t = 0.0
+        for a in data.get("actors", []):
+            keys = keys_for(imdb_id=a.get("imdb_id"), tmdb_id=a.get("tmdb_id"),
+                            jellyfin_id=a.get("jellyfin_id"), name=a.get("name"),
+                            crosswalk=crosswalk)
+            windows = [(float(t0), float(t1)) for t0, t1 in a.get("scenes", [])]
+            for _, t1 in windows:
+                self._max_t = max(self._max_t, t1)
+            self.actors.append({"keys": keys, "windows": windows})
+
+    def present_at(self, t: float) -> set[frozenset[str]]:
+        """Set of actors present at t; each actor is its (frozen) key-set."""
+        out: set[frozenset[str]] = set()
+        for a in self.actors:
+            for t0, t1 in a["windows"]:
+                if t0 <= t <= t1:
+                    out.add(frozenset(a["keys"]))
+                    break
+        return out
+
+    def present_in_span(self, s0: float, s1: float) -> set[frozenset[str]]:
+        """Actors with ANY detection window overlapping [s0,s1].
+
+        Snaps detections to a scene grid: an actor seen anywhere inside a scene
+        counts as present for the whole scene. Isolates 'did we see this actor in
+        this scene at all' (coverage) from exact-timing recall."""
+        out: set[frozenset[str]] = set()
+        for a in self.actors:
+            for t0, t1 in a["windows"]:
+                if t0 <= s1 and t1 >= s0:  # interval overlap
+                    out.add(frozenset(a["keys"]))
+                    break
+        return out
+
+    def all_keys(self) -> set[str]:
+        out: set[str] = set()
+        for a in self.actors:
+            out |= a["keys"]
+        return out
+
+    @property
+    def max_t(self) -> float:
+        return self._max_t
+
+
+def load_gallery_keys(path: str | None, crosswalk=None) -> set[str] | None:
+    """Union of match keys for every actor in the gallery, for masking.
+
+    Accepts either the JSON gallery or the HDF5 fast-load gallery (.h5/.hdf5,
+    produced by json_to_hdf5_gallery.py) — the matcher reads HDF5, so this side must
+    too. HDF5 stores ids/names as parallel string datasets."""
+    if not path:
+        return None
+    out: set[str] = set()
+    if path.endswith(".h5") or path.endswith(".hdf5"):
+        import h5py
+        with h5py.File(path, "r") as f:
+            def col(name):
+                return [(v.decode() if isinstance(v, bytes) else str(v))
+                        for v in f[name][:]] if name in f else []
+            imdb, tmdb = col("imdb_id"), col("tmdb_id")
+            jf, name = col("jellyfin_id"), col("name")
+            for i in range(len(name)):
+                out |= keys_for(imdb_id=imdb[i] if i < len(imdb) else "",
+                                tmdb_id=tmdb[i] if i < len(tmdb) else "",
+                                jellyfin_id=jf[i] if i < len(jf) else "",
+                                name=name[i], crosswalk=crosswalk)
+        return out
+    with open(path) as f:
+        data = json.load(f)
+    for a in data.get("actors", []):
+        out |= keys_for(imdb_id=a.get("imdb_id"), tmdb_id=a.get("tmdb_id"),
+                        jellyfin_id=a.get("jellyfin_id"), name=a.get("name"),
+                        crosswalk=crosswalk)
+    return out
+
+
+# ── sampling ────────────────────────────────────────────────────────────────
+
+def sample_points(args, pred: Prediction, gt) -> list[float]:
+    if args.scene_anchored:
+        spans = gt.scene_windows()
+        if not spans:
+            sys.exit("[eval] --scene-anchored: ground truth has no scene spans")
+        return [(t0 + t1) / 2.0 for t0, t1 in spans]
+
+    end = args.end if args.end is not None else max(pred.max_t, _gt_end(gt))
+    if end <= 0:
+        sys.exit("[eval] could not determine timeline end; pass --end")
+
+    if args.random:
+        rng = random.Random(args.seed)
+        return sorted(rng.uniform(0.0, end) for _ in range(args.random))
+
+    n = int(end / args.step) + 1
+    return [i * args.step for i in range(n)]
+
+
+def _gt_end(gt) -> float:
+    spans = gt.scene_windows()
+    return max((t1 for _, t1 in spans), default=0.0)
+
+
+# ── scoring ────────────────────────────────────────────────────────────────
+
+def score(pred: Prediction, gt, points: list[float], mask: set[str] | None,
+          count_out_of_cast_fp: bool = False):
+    """Micro-sum TP/FP/FN over timepoints.
+
+    Each side is a set of actors, an actor being its key-set. Predicted actor P
+    matches GT actor G iff their key-sets intersect (any shared id/name). We match
+    greedily so each actor is used once, then:
+      TP = matched pairs, FP = unmatched predicted, FN = unmatched GT.
+
+    `mask` (gallery∩GT keys) restricts GT so X-Ray cast we can't recognise doesn't
+    inflate FN. By default predictions are masked the same way — which DROPS a
+    predicted actor who isn't in this film's cast (a cross-film misidentification),
+    hiding the pipeline's worst false positives.
+
+    Set count_out_of_cast_fp=True to keep ALL predictions: an actor named who is not
+    a present GT cast member counts as an FP, including out-of-cast confusions. This
+    is the honest, ship-relevant precision. GT is still masked for fair recall.
+    """
+    TP = FP = FN = 0
+    per_point = []
+
+    for t in points:
+        P = [set(a) for a in pred.present_at(t)]
+        G = [set(a) for a in gt.present_at(t)]
+        if mask is not None:
+            G = [a for a in G if a & mask]
+            if not count_out_of_cast_fp:
+                P = [a for a in P if a & mask]
+
+        tp = _match_count(P, G)
+        fp = len(P) - tp
+        fn = len(G) - tp
+        TP += tp
+        FP += fp
+        FN += fn
+        per_point.append((t, tp, fp, fn))
+
+    prec = TP / (TP + FP) if (TP + FP) else 0.0
+    rec = TP / (TP + FN) if (TP + FN) else 0.0
+    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) else 0.0
+    return {"TP": TP, "FP": FP, "FN": FN, "precision": prec,
+            "recall": rec, "f1": f1, "n_points": len(points),
+            "per_point": per_point}
+
+
+def _match_count(P: list[set[str]], G: list[set[str]]) -> int:
+    """Greedy 1:1 matching of predicted↔GT actors by key intersection."""
+    used = [False] * len(G)
+    matched = 0
+    for pa in P:
+        for j, ga in enumerate(G):
+            if not used[j] and pa & ga:
+                used[j] = True
+                matched += 1
+                break
+    return matched
+
+
+# ── main ────────────────────────────────────────────────────────────────────
+
+def main():
+    p = argparse.ArgumentParser(description=__doc__,
+                                formatter_class=argparse.RawDescriptionHelpFormatter)
+    p.add_argument("--pred", required=True, help="pipeline output JSON")
+    src = p.add_mutually_exclusive_group(required=True)
+    src.add_argument("--xray", help="dir with people.csv/scenes.csv/people_in_scenes.csv")
+    src.add_argument("--movienet", help="MovieNet-PS root (needs --split and --title)")
+    p.add_argument("--split", default="Train_app10", help="MovieNet annotation split")
+    p.add_argument("--title", help="MovieNet title tt-id to filter to")
+    p.add_argument("--gallery", help="gallery.json for masking (gallery ∩ GT)")
+    p.add_argument("--crosswalk", help="tmdb→imdb JSON (tmdb_imdb_map.py) for exact "
+                                       "id join when pred/gallery lack imdb_id")
+    p.add_argument("--no-mask", action="store_true", help="disable gallery∩GT masking")
+    p.add_argument("--step", type=float, default=1.0, help="regular grid step (s)")
+    p.add_argument("--random", type=int, help="sample N uniform-random timepoints")
+    p.add_argument("--scene-anchored", action="store_true",
+                   help="sample GT scene midpoints (one vote per scene)")
+    p.add_argument("--end", type=float, help="timeline end (s); default = max of pred/GT")
+    p.add_argument("--seed", type=int, default=0)
+    p.add_argument("--json-out", help="write full metrics (incl. per-point) here")
+    args = p.parse_args()
+
+    crosswalk = CrosswalkTable.load(args.crosswalk) if args.crosswalk else None
+    if crosswalk is not None:
+        print(f"[eval] crosswalk: {len(crosswalk)} tmdb→imdb entries", file=sys.stderr)
+
+    pred = Prediction(args.pred, crosswalk=crosswalk)
+    print(f"[eval] pred: {len(pred.actors)} actors, timeline≈{pred.max_t:.0f}s "
+          f"({pred.movie})", file=sys.stderr)
+
+    if args.xray:
+        gt = XRayGroundTruth(args.xray)
+    else:
+        if not args.title:
+            sys.exit("[eval] --movienet requires --title tt-id")
+        gt = _load_movienet(args.movienet, args.split, args.title, args.gallery)
+    print(f"[eval] GT: {gt.summary()}", file=sys.stderr)
+
+    mask = None
+    if not args.no_mask:
+        gkeys = load_gallery_keys(args.gallery, crosswalk=crosswalk)
+        gt_keys = gt.all_keys()
+        if gkeys is None:
+            # no gallery given → mask to GT ∩ pred key spaces so absent-from-gallery
+            # GT actors don't inflate FN. Fall back to GT keys the pred could name.
+            mask = gt_keys & pred.all_keys()
+            print("[eval] no --gallery; masking to GT∩pred keys "
+                  f"({len(mask)})", file=sys.stderr)
+        else:
+            mask = gkeys & gt_keys
+            print(f"[eval] mask = gallery∩GT ({len(mask)} keys)", file=sys.stderr)
+
+    points = sample_points(args, pred, gt)
+    print(f"[eval] sampling {len(points)} timepoints "
+          f"({'scene-anchored' if args.scene_anchored else 'random' if args.random else f'grid@{args.step}s'})",
+          file=sys.stderr)
+
+    m = score(pred, gt, points, mask)
+    print("\n── presence eval ─────────────────────────────")
+    print(f"  timepoints : {m['n_points']}")
+    print(f"  TP/FP/FN   : {m['TP']} / {m['FP']} / {m['FN']}")
+    print(f"  precision  : {m['precision']*100:.1f}%")
+    print(f"  recall     : {m['recall']*100:.1f}%")
+    print(f"  F1         : {m['f1']*100:.1f}%")
+
+    if args.json_out:
+        out = {k: v for k, v in m.items() if k != "per_point"}
+        out["per_point"] = [{"t": t, "tp": tp, "fp": fp, "fn": fn}
+                            for t, tp, fp, fn in m["per_point"]]
+        Path(args.json_out).write_text(json.dumps(out, indent=2))
+        print(f"[eval] wrote {args.json_out}", file=sys.stderr)
+
+
+def _load_movienet(root, split, title, gallery):
+    sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
+    from movienet_prep import load_movienet_annotations
+    anns = load_movienet_annotations(Path(root), split)
+    anns = [a for a in anns if a["img_path"].startswith(title)]
+    if not anns:
+        sys.exit(f"[eval] no MovieNet annotations for title {title} in {split}")
+    id_to_name = {}
+    if gallery:
+        for a in json.load(open(gallery)).get("actors", []):
+            if a.get("imdb_id"):
+                id_to_name[a["imdb_id"]] = a.get("name", "")
+    return MovieNetGroundTruth(anns, id_to_name)
+
+
+if __name__ == "__main__":
+    main()

scripts/validation/test_sample_eval.py

@@ -0,0 +1,148 @@
+#!/usr/bin/env python3
+"""
+Self-contained tests for the presence eval. Builds a synthetic X-Ray fixture and
+a pipeline-output JSON in a temp dir, then checks scoring, masking, name-fallback
+matching, and sampling modes. Run: python scripts/validation/test_sample_eval.py
+"""
+
+import csv
+import json
+import sys
+import tempfile
+from pathlib import Path
+
+HERE = Path(__file__).resolve().parent
+sys.path.insert(0, str(HERE))
+
+from identity import keys_for, norm_name          # noqa: E402
+from sample_eval import Prediction, score, sample_points, _match_count  # noqa: E402
+from ground_truth import XRayGroundTruth           # noqa: E402
+from tmdb_imdb_map import CrosswalkTable           # noqa: E402
+
+
+def _write_xray(d: Path):
+    d.mkdir(parents=True, exist_ok=True)
+    with open(d / "scenes.csv", "w", newline="") as f:
+        w = csv.writer(f); w.writerow(["scene", "start_ms", "end_ms"])
+        w.writerows([("1", 0, 60000), ("2", 300000, 340000)])
+    # real Zenodo X-Ray schema: people(name_id,person,character);
+    # people_in_scenes(scene,start,end,name_id,timestamp)
+    with open(d / "people.csv", "w", newline="") as f:
+        w = csv.writer(f); w.writerow(["name_id", "person", "character"])
+        w.writerows([("nm0330687", "Lauren Graham", "Lorelai"),
+                     ("nm0004754", "Alexis Bledel", "Rory"),
+                     ("nm0000001", "Ghost Actor",   "Ghost")])
+    with open(d / "people_in_scenes.csv", "w", newline="") as f:
+        w = csv.writer(f); w.writerow(["scene", "start", "end", "name_id", "timestamp"])
+        w.writerows([("1", 0, 60000, "nm0330687", 5000),
+                     ("1", 0, 60000, "nm0000001", 8000),
+                     ("2", 300000, 340000, "nm0004754", 305000)])
+
+
+def _write_pred(path: Path):
+    # pipeline output carries tmdb+name but NO nm ids -> name-fallback join to X-Ray
+    doc = {"schema_version": 1, "movie": "x", "sample_fps": 1.0, "anneal_sec": 10.0,
+           "actors": [
+               {"name": "Lauren Graham", "imdb_id": "", "tmdb_id": "16858",
+                "jellyfin_id": "a", "scenes": [[21.0, 31.0], [50.0, 59.0]]},
+               {"name": "Alexis Bledel", "imdb_id": "", "tmdb_id": "6279",
+                "jellyfin_id": "b", "scenes": [[301.0, 311.0]]},
+               {"name": "Edward Herrmann", "imdb_id": "", "tmdb_id": "52995",
+                "jellyfin_id": "c", "scenes": [[4.0, 56.0]]}]}
+    path.write_text(json.dumps(doc))
+
+
+class T:
+    n = 0
+    def check(self, cond, msg):
+        T.n += 1
+        assert cond, f"FAIL: {msg}"
+        print(f"  ok: {msg}")
+
+
+def main():
+    t = T()
+
+    # -- identity ---------------------------------------------------------------
+    t.check(norm_name("Zöe Saldaña!") == "zoe saldana", "accent/punct normalization")
+    t.check(keys_for(imdb_id="nm1", name="Jo Ann") == {"imdb:nm1", "name:jo ann"},
+            "keys_for builds namespaced tokens")
+    t.check(keys_for(imdb_id="") == set(), "empty ids dropped")
+
+    # -- crosswalk: tmdb-only actor gains an exact imdb: key --------------------
+    xw = CrosswalkTable({"16858": "nm0330687", "999": None})
+    k = keys_for(imdb_id="", tmdb_id="16858", name="Lauren Graham", crosswalk=xw)
+    t.check("imdb:nm0330687" in k, "crosswalk resolves tmdb→imdb key")
+    k_null = keys_for(tmdb_id="999", crosswalk=xw)
+    t.check(not any(x.startswith("imdb:") for x in k_null), "crosswalk null → no imdb key")
+    t.check(len(xw) == 1, "CrosswalkTable len counts non-null entries")
+
+    # -- matching ---------------------------------------------------------------
+    t.check(_match_count([{"name:jo"}, {"name:al"}], [{"imdb:x", "name:jo"}]) == 1,
+            "one match by shared name key")
+    t.check(_match_count([{"name:jo"}], [{"name:jo"}, {"name:jo"}]) == 1,
+            "greedy 1:1 uses each GT once")
+
+    with tempfile.TemporaryDirectory() as tmp:
+        tmp = Path(tmp)
+        _write_xray(tmp / "xray")
+        _write_pred(tmp / "pred.json")
+
+        pred = Prediction(tmp / "pred.json")
+        gt = XRayGroundTruth(tmp / "xray")
+
+        # -- presence lookups ---------------------------------------------------
+        t.check(len(pred.present_at(25)) == 2, "Graham+Herrmann present at 25s")
+        t.check(len(pred.present_at(305)) == 1, "only Bledel present at 305s")
+        t.check(len(gt.present_at(30)) == 2, "X-Ray scene1 has Graham+Ghost at 30s")
+        t.check(len(gt.present_at(320)) == 1, "X-Ray scene2 has Bledel at 320s")
+
+        # -- masking behaviour --------------------------------------------------
+        pts = [30.0]
+        # unmasked: at 30s pred={Graham,Herrmann}, gt={Graham,Ghost}
+        #   match Graham -> TP1; Herrmann unmatched -> FP1; Ghost unmatched -> FN1
+        m = score(pred, gt, pts, mask=None)
+        t.check((m["TP"], m["FP"], m["FN"]) == (1, 1, 1), "unmasked 30s = 1/1/1")
+
+        # masked to GT∩pred keys: Ghost & Herrmann are absent from the other side's
+        # key space, so both drop -> only Graham remains on both -> 1/0/0
+        gt_keys = gt.all_keys(); pred_keys = pred.all_keys()
+        mask = gt_keys & pred_keys
+        m2 = score(pred, gt, pts, mask=mask)
+        t.check((m2["TP"], m2["FP"], m2["FN"]) == (1, 0, 0),
+                "masked 30s drops off-gallery actors = 1/0/0")
+
+        # -- crosswalk end-to-end: exact imdb join, names garbled ---------------
+        # Rebuild pred with names that WON'T match X-Ray, but a crosswalk that maps
+        # their tmdb ids to the correct nm ids. Match must survive via imdb key.
+        garbled = {"schema_version": 1, "movie": "x", "actors": [
+            {"name": "WRONG NAME A", "imdb_id": "", "tmdb_id": "16858",
+             "jellyfin_id": "a", "scenes": [[21.0, 31.0]]},            # →nm0330687 Graham
+            {"name": "WRONG NAME B", "imdb_id": "", "tmdb_id": "6279",
+             "jellyfin_id": "b", "scenes": [[301.0, 311.0]]}]}          # →nm0004754 Bledel
+        (tmp / "garbled.json").write_text(json.dumps(garbled))
+        xw = CrosswalkTable({"16858": "nm0330687", "6279": "nm0004754"})
+        pred_g = Prediction(tmp / "garbled.json", crosswalk=xw)
+        # at 30s (scene1) Graham should match by imdb despite wrong name
+        m_g = score(pred_g, gt, [30.0], mask=None)
+        t.check(m_g["TP"] == 1, "crosswalk yields exact imdb match despite wrong names")
+        # without crosswalk, wrong names → no match at all
+        pred_bad = Prediction(tmp / "garbled.json")
+        m_bad = score(pred_bad, gt, [30.0], mask=None)
+        t.check(m_bad["TP"] == 0, "no crosswalk + wrong names → no match")
+
+        # -- sampling modes -----------------------------------------------------
+        class A:  # arg stub
+            scene_anchored = True; random = None; step = 1.0; end = None; seed = 0
+        sp = sample_points(A, pred, gt)
+        t.check(sp == [30.0, 320.0], "scene-anchored samples scene midpoints")
+
+        A.scene_anchored = False; A.end = 10.0; A.step = 2.0
+        grid = sample_points(A, pred, gt)
+        t.check(grid == [0.0, 2.0, 4.0, 6.0, 8.0, 10.0], "regular grid step")
+
+    print(f"\nALL {T.n} CHECKS PASSED")
+
+
+if __name__ == "__main__":
+    main()

scripts/validation/tmdb_imdb_map.py

@@ -0,0 +1,135 @@
+#!/usr/bin/env python3
+"""
+tmdb_imdb_map.py — build & consult a cached tmdb_person_id → imdb_id crosswalk.
+
+The gallery (and pipeline output) key actors by TMDB person id but carry no IMDb
+`nm…` id, while the X-Ray / MovieNet ground truth keys on IMDb. Rather than join on
+fuzzy names, we resolve tmdb→imdb once via TMDB's authoritative
+`/person/{id}/external_ids` endpoint and cache the result to JSON. The eval loaders
+consult this table to add an exact `imdb:` key alongside each `tmdb:` key.
+
+Table format (JSON): { "<tmdb_person_id>": "nm0000123" | null, ... }
+A null means "looked up, TMDB has no IMDb id" — cached so we don't re-query.
+
+Build / refresh the table:
+  python scripts/validation/tmdb_imdb_map.py \
+      --gallery gallery_arcface_w600k_r50.json \
+      --out scripts/validation/tmdb_imdb.json
+  # TMDB_API_KEY read from env / .env (via sae_env)
+
+Consult it from code:
+  from tmdb_imdb_map import CrosswalkTable
+  tbl = CrosswalkTable.load("scripts/validation/tmdb_imdb.json")
+  nm = tbl.imdb_for("35467")   # -> "nm..." or None
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import sys
+import time
+from pathlib import Path
+
+_HERE = Path(__file__).resolve().parent
+
+
+class CrosswalkTable:
+    """Read-only view over the cached tmdb→imdb JSON. Missing file → empty table."""
+
+    def __init__(self, mapping: dict[str, str | None]):
+        self._m = mapping
+
+    @classmethod
+    def load(cls, path: str | Path) -> "CrosswalkTable":
+        p = Path(path)
+        if not p.exists():
+            return cls({})
+        return cls(json.loads(p.read_text()))
+
+    def imdb_for(self, tmdb_id) -> str | None:
+        if tmdb_id is None:
+            return None
+        return self._m.get(str(tmdb_id))
+
+    def __len__(self) -> int:
+        return sum(1 for v in self._m.values() if v)
+
+
+# ── builder ─────────────────────────────────────────────────────────────────
+
+def _external_ids(tmdb_get, tmdb_person_id: str, token: str) -> str | None:
+    data = tmdb_get(f"/person/{tmdb_person_id}/external_ids", token)
+    imdb = data.get("imdb_id")
+    return imdb or None  # normalize "" → None
+
+
+def build(gallery_path: str, out_path: str, token: str, sleep: float = 0.0) -> None:
+    # import the existing TMDB helper (scripts/ is the parent dir)
+    sys.path.insert(0, str(_HERE.parent))
+    from sae_tmdb import tmdb_get
+
+    with open(gallery_path) as f:
+        actors = json.load(f).get("actors", [])
+    tmdb_ids = sorted({str(a["tmdb_id"]) for a in actors if a.get("tmdb_id")})
+    print(f"[map] gallery tmdb ids: {len(tmdb_ids)}", file=sys.stderr)
+
+    out = Path(out_path)
+    existing: dict[str, str | None] = {}
+    if out.exists():
+        existing = json.loads(out.read_text())
+        print(f"[map] resuming from {len(existing)} cached entries", file=sys.stderr)
+
+    todo = [t for t in tmdb_ids if t not in existing]
+    print(f"[map] to resolve: {len(todo)}", file=sys.stderr)
+
+    n_ok = n_none = n_err = 0
+    for i, tid in enumerate(todo, 1):
+        try:
+            nm = _external_ids(tmdb_get, tid, token)
+            existing[tid] = nm
+            n_ok += (nm is not None)
+            n_none += (nm is None)
+        except Exception as e:  # network/rate-limit/404 — record nothing, keep going
+            n_err += 1
+            print(f"\n[map] error on tmdb {tid}: {e}", file=sys.stderr)
+        if i % 25 == 0 or i == len(todo):
+            print(f"\r[map] {i}/{len(todo)} resolved "
+                  f"(imdb={n_ok} none={n_none} err={n_err})", end="", file=sys.stderr)
+            out.write_text(json.dumps(existing, indent=2))  # checkpoint
+        if sleep:
+            time.sleep(sleep)
+
+    out.write_text(json.dumps(existing, indent=2))
+    print(f"\n[map] wrote {out} — {sum(1 for v in existing.values() if v)} imdb ids",
+          file=sys.stderr)
+
+
+def main():
+    # load .env → os.environ (same convention as the gallery builders)
+    sys.path.insert(0, str(_HERE.parent))
+    try:
+        import sae_env  # noqa: F401  (side-effect import)
+    except Exception:
+        pass
+
+    p = argparse.ArgumentParser(description=__doc__,
+                                formatter_class=argparse.RawDescriptionHelpFormatter)
+    p.add_argument("--gallery", required=True, help="gallery.json (source of tmdb ids)")
+    p.add_argument("--out", default=str(_HERE / "tmdb_imdb.json"),
+                   help="output crosswalk JSON (default: scripts/validation/tmdb_imdb.json)")
+    p.add_argument("--tmdb-key", default=os.environ.get("TMDB_API_KEY"),
+                   help="TMDB v3 API key or v4 read token. Env: TMDB_API_KEY")
+    p.add_argument("--sleep", type=float, default=0.0,
+                   help="seconds between requests (TMDB has no hard limit; use if throttled)")
+    args = p.parse_args()
+
+    if not args.tmdb_key:
+        sys.exit("[map] no TMDB key — set TMDB_API_KEY or pass --tmdb-key")
+
+    build(args.gallery, args.out, args.tmdb_key, args.sleep)
+
+
+if __name__ == "__main__":
+    main()