feat(tooling): X-Ray threshold optimizer, gallery utilities, artifact registry, docs build
Optimizer (scripts/optimizer/): replay.py runs the real C++ tracker/matcher/ scene_tracker chain over a dumped-embeddings HDF5 via sae_kpn, so a threshold sweep never re-decodes video or re-embeds faces. optimize.py drives scipy's differential_evolution over the knob space, with DE-level parallelism (multiple population candidates evaluated concurrently via a ThreadPoolExecutor) on top of per-film replay parallelism. second_score.py is the per-second X-Ray scoring metric (TPI/FPI/FN, out-of-cast misID weighted 10x, fair recall masked to gallery-known cast) that superseded an earlier scene-union metric. dump_error_frames.py / dump_scene_montage.py extract annotated video frames (bounding boxes, TPI/FPI/FN captions, onscreen-vs-offscreen split) for visual review of a replay against ground truth. Gallery utilities: cast_restrict.py, gallery_membership.py, fetch_missing_actors.py, reembed_gallery.py. scripts/validation/: X-Ray ground-truth loading and provider-agnostic identity matching (identity.py's keys_for — an actor is the union of every id we can derive, since pipeline output and ground truth don't share one id space). scripts/artifacts/: push/pull scripts for the Gitea generic package registry — galleries, montage frames, and experiment data (manifests/trajectories/results) are pushed there instead of committed, since none are needed to run the app, only benchmarks. Versioned by git short-SHA. scripts/docs/: MkDocs site build (build_site.sh) and the calibration-curve comparison chart (calibration_chart.py, matplotlib, reads each gallery's embedded calibration). Gallery-building scripts (make_jellyfin_gallery.py, make_gallery.py, filter_gallery.py, run_from_jellyfin.py, movienet_eval.py, movienet_prep.py, sae_gallery.py) updated to read/write HDF5 galleries exclusively, matching the engine-side format switch. run_from_jellyfin.py and the optimizer no longer carry movie source paths in shared manifests (some source filenames include scene-release tags) — resolved locally via a gitignored file-lut.json instead.
This commit is contained in:
@@ -0,0 +1,87 @@
|
||||
# scripts/validation — per-scene actor-presence eval
|
||||
|
||||
Validates the pipeline's per-scene "who's on screen" output against external
|
||||
ground truth, offline. Annealing (`anneal_sec`) means an actor's presence is only
|
||||
defined *after* the whole file is merged into `[start,end]` windows, so we cannot
|
||||
score live: process → write the pipeline JSON → **sample timepoints** → compare
|
||||
predicted vs ground-truth presence sets → micro-sum TP/FP/FN → precision/recall/F1.
|
||||
|
||||
## Ground-truth sources
|
||||
|
||||
| Source | Semantics | Fair to a face pipeline? | What it measures |
|
||||
| ------ | --------- | ------------------------ | ---------------- |
|
||||
| **MovieNet-PS** | on-screen **face** presence per shot | yes — like-for-like | recognition accuracy |
|
||||
| **Amazon X-Ray** (Zenodo) | **cast-in-scene** (incl. off-camera / non-speaking) | no — penalizes by design | coverage ceiling; recall gap = actors we structurally can't see |
|
||||
|
||||
- MovieNet is the honest recognition number.
|
||||
- X-Ray is an upper bound: its recall gap tells you how much presence is off-camera
|
||||
cast a face detector can never reach — not a pipeline error.
|
||||
|
||||
X-Ray dataset: Zenodo DOI `10.5281/zenodo.17659734` (CC-BY-4.0). Per movie it ships
|
||||
`people.csv`, `scenes.csv`, `people_in_scenes.csv`.
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
# against Amazon X-Ray CSVs for one title
|
||||
python scripts/validation/sample_eval.py \
|
||||
--pred "Scene in a Mall.json" \
|
||||
--xray /data/xray/<movie_dir> \
|
||||
--gallery gallery_arcface_w600k_r50.json \
|
||||
--step 1.0
|
||||
|
||||
# against MovieNet-PS for one title
|
||||
python scripts/validation/sample_eval.py \
|
||||
--pred out.json \
|
||||
--movienet /data/movienet --split Train_app10 --title tt0032138 \
|
||||
--gallery gallery_arcface_w600k_r50.json
|
||||
```
|
||||
|
||||
### Sampling modes
|
||||
- `--step S` regular grid every S s (default 1.0) — time-weighted headline number.
|
||||
- `--random N` N uniform-random timepoints (for confidence intervals).
|
||||
- `--scene-anchored` one timepoint per GT scene midpoint — the literal X-Ray
|
||||
"did I get this scene's cast right?" question; neutralizes long-scene bias.
|
||||
|
||||
Ground truth is compared **raw** (annealing is *not* applied to GT).
|
||||
|
||||
## Matching & masking
|
||||
|
||||
Identity is provider-agnostic (`identity.py`): each actor is the *set* of every key
|
||||
we can derive — `imdb:nm…`, `tmdb:…`, `jf:…`, `name:<normalized>`. Predicted and GT
|
||||
actors match iff their key-sets intersect, so an output carrying only tmdb/jellyfin
|
||||
ids still joins X-Ray's `nm` ids via the normalized-name fallback.
|
||||
|
||||
Scoring is **masked to `gallery ∩ GT`**: a GT actor absent from the gallery is
|
||||
ignored (not an FN), so we measure pipeline accuracy, not gallery coverage. Without
|
||||
`--gallery` the mask falls back to `GT ∩ pred` keys. `--no-mask` disables it.
|
||||
|
||||
### Exact id join via the tmdb→imdb crosswalk (recommended)
|
||||
|
||||
The gallery/pipeline output key actors by **TMDB** id (no `nm…`), while X-Ray and
|
||||
MovieNet key on **IMDb**. They only overlap on the fuzzy `name:` key by default.
|
||||
Build a cached `tmdb→imdb` table once and pass it with `--crosswalk` to turn the
|
||||
name join into an exact id join:
|
||||
|
||||
```bash
|
||||
# one-time: resolve every gallery tmdb id via TMDB /person/{id}/external_ids
|
||||
python scripts/validation/tmdb_imdb_map.py \
|
||||
--gallery gallery_arcface_w600k_r50.json \
|
||||
--out scripts/validation/tmdb_imdb.json # TMDB_API_KEY from env/.env
|
||||
|
||||
# then score with exact ids
|
||||
python scripts/validation/sample_eval.py --pred out.json --xray <dir> \
|
||||
--gallery gallery_arcface_w600k_r50.json \
|
||||
--crosswalk scripts/validation/tmdb_imdb.json
|
||||
```
|
||||
|
||||
The table caches nulls (tmdb ids TMDB has no IMDb id for) and checkpoints, so a
|
||||
re-run only resolves new ids. TMDB is authoritative for this crosswalk — there is
|
||||
no clean free bulk `tmdb_person ↔ nm` file, so we query the API once and cache.
|
||||
|
||||
## Files
|
||||
- `sample_eval.py` — CLI scorer.
|
||||
- `ground_truth.py` — `XRayGroundTruth`, `MovieNetGroundTruth` loaders.
|
||||
- `identity.py` — provider-agnostic match keys.
|
||||
- `tmdb_imdb_map.py` — build/consult the cached `tmdb→imdb` crosswalk.
|
||||
- `test_sample_eval.py` — self-contained tests (`python scripts/validation/test_sample_eval.py`).
|
||||
Reference in New Issue
Block a user