dtourolleandClaude Opus 5
2a8ee3660b
sae_audio exposes the shipped signature to Python. It compiles audio_signature.cpp directly against FFmpeg rather than linking sae_gallery: the signature needs no model, no OpenCV and no HDF5, so a module that dragged those in would make `import sae_audio` depend on a GPU-capable build of a path that is pure CPU DSP. The point of binding rather than porting is that a fingerprint is only useful if every implementation agrees byte for byte. A numpy port would be a third implementation, and the one nobody checks against the golden vector. VR-014 then recovers a known trim from real film audio rather than from the synthetic tone: 40 random in-cap offsets, every one recovered to the nearest frame, worst error 46 ms against a 500 ms budget — and 46 ms is the quantisation floor, not a result, since offsets land on whole 92.88 ms frames. The soft spot is tier labelling rather than accuracy. Sub-frame misalignment drags the score down (0.94-0.99 near a frame boundary, 0.69-0.73 at half a frame), demoting 27 of 40 correct alignments to `loose`. Allowing +/-1 frame of slack in the score fixes it: all 40 back to `audio` at min 0.906, false matches unmoved at 0.12-0.16, for 81 ms of the budget. The module stops at the producer's edge. Sliding one signature against another is the consumer's algorithm (server SPEC §3, and the jRay plugin implements it), so a caller writing that slide in numpy is not duplicating anything this repo owns. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> TRACES: IR-004, IR-005 | VR-014 | UT-105, UT-106, UT-107, UT-108 | SR-003
scripts/validation — per-scene actor-presence eval
Validates the pipeline's per-scene "who's on screen" output against external
ground truth, offline. Annealing (anneal_sec) means an actor's presence is only
defined after the whole file is merged into [start,end] windows, so we cannot
score live: process → write the pipeline JSON → sample timepoints → compare
predicted vs ground-truth presence sets → micro-sum TP/FP/FN → precision/recall/F1.
Ground-truth sources
| Source | Semantics | Fair to a face pipeline? | What it measures |
|---|---|---|---|
| MovieNet-PS | on-screen face presence per shot | yes — like-for-like | recognition accuracy |
| Amazon X-Ray (Zenodo) | cast-in-scene (incl. off-camera / non-speaking) | no — penalizes by design | coverage ceiling; recall gap = actors we structurally can't see |
- MovieNet is the honest recognition number.
- X-Ray is an upper bound: its recall gap tells you how much presence is off-camera cast a face detector can never reach — not a pipeline error.
X-Ray dataset: Zenodo DOI 10.5281/zenodo.17659734 (CC-BY-4.0). Per movie it ships
people.csv, scenes.csv, people_in_scenes.csv.
Usage
# against Amazon X-Ray CSVs for one title
python scripts/validation/sample_eval.py \
--pred "Scene in a Mall.json" \
--xray /data/xray/<movie_dir> \
--gallery gallery_arcface_w600k_r50.json \
--step 1.0
# against MovieNet-PS for one title
python scripts/validation/sample_eval.py \
--pred out.json \
--movienet /data/movienet --split Train_app10 --title tt0032138 \
--gallery gallery_arcface_w600k_r50.json
Sampling modes
--step Sregular grid every S s (default 1.0) — time-weighted headline number.--random NN uniform-random timepoints (for confidence intervals).--scene-anchoredone timepoint per GT scene midpoint — the literal X-Ray "did I get this scene's cast right?" question; neutralizes long-scene bias.
Ground truth is compared raw (annealing is not applied to GT).
Matching & masking
Identity is provider-agnostic (identity.py): each actor is the set of every key
we can derive — imdb:nm…, tmdb:…, jf:…, name:<normalized>. Predicted and GT
actors match iff their key-sets intersect, so an output carrying only tmdb/jellyfin
ids still joins X-Ray's nm ids via the normalized-name fallback.
Scoring is masked to gallery ∩ GT: a GT actor absent from the gallery is
ignored (not an FN), so we measure pipeline accuracy, not gallery coverage. Without
--gallery the mask falls back to GT ∩ pred keys. --no-mask disables it.
Exact id join via the tmdb→imdb crosswalk (recommended)
The gallery/pipeline output key actors by TMDB id (no nm…), while X-Ray and
MovieNet key on IMDb. They only overlap on the fuzzy name: key by default.
Build a cached tmdb→imdb table once and pass it with --crosswalk to turn the
name join into an exact id join:
# one-time: resolve every gallery tmdb id via TMDB /person/{id}/external_ids
python scripts/validation/tmdb_imdb_map.py \
--gallery gallery_arcface_w600k_r50.json \
--out scripts/validation/tmdb_imdb.json # TMDB_API_KEY from env/.env
# then score with exact ids
python scripts/validation/sample_eval.py --pred out.json --xray <dir> \
--gallery gallery_arcface_w600k_r50.json \
--crosswalk scripts/validation/tmdb_imdb.json
The table caches nulls (tmdb ids TMDB has no IMDb id for) and checkpoints, so a
re-run only resolves new ids. TMDB is authoritative for this crosswalk — there is
no clean free bulk tmdb_person ↔ nm file, so we query the API once and cache.
Files
sample_eval.py— CLI scorer.ground_truth.py—XRayGroundTruth,MovieNetGroundTruthloaders.identity.py— provider-agnostic match keys.tmdb_imdb_map.py— build/consult the cachedtmdb→imdbcrosswalk.test_sample_eval.py— self-contained tests (python scripts/validation/test_sample_eval.py).