feat(tooling): X-Ray threshold optimizer, gallery utilities, artifact registry, docs build
Optimizer (scripts/optimizer/): replay.py runs the real C++ tracker/matcher/ scene_tracker chain over a dumped-embeddings HDF5 via sae_kpn, so a threshold sweep never re-decodes video or re-embeds faces. optimize.py drives scipy's differential_evolution over the knob space, with DE-level parallelism (multiple population candidates evaluated concurrently via a ThreadPoolExecutor) on top of per-film replay parallelism. second_score.py is the per-second X-Ray scoring metric (TPI/FPI/FN, out-of-cast misID weighted 10x, fair recall masked to gallery-known cast) that superseded an earlier scene-union metric. dump_error_frames.py / dump_scene_montage.py extract annotated video frames (bounding boxes, TPI/FPI/FN captions, onscreen-vs-offscreen split) for visual review of a replay against ground truth. Gallery utilities: cast_restrict.py, gallery_membership.py, fetch_missing_actors.py, reembed_gallery.py. scripts/validation/: X-Ray ground-truth loading and provider-agnostic identity matching (identity.py's keys_for — an actor is the union of every id we can derive, since pipeline output and ground truth don't share one id space). scripts/artifacts/: push/pull scripts for the Gitea generic package registry — galleries, montage frames, and experiment data (manifests/trajectories/results) are pushed there instead of committed, since none are needed to run the app, only benchmarks. Versioned by git short-SHA. scripts/docs/: MkDocs site build (build_site.sh) and the calibration-curve comparison chart (calibration_chart.py, matplotlib, reads each gallery's embedded calibration). Gallery-building scripts (make_jellyfin_gallery.py, make_gallery.py, filter_gallery.py, run_from_jellyfin.py, movienet_eval.py, movienet_prep.py, sae_gallery.py) updated to read/write HDF5 galleries exclusively, matching the engine-side format switch. run_from_jellyfin.py and the optimizer no longer carry movie source paths in shared manifests (some source filenames include scene-release tags) — resolved locally via a gitignored file-lut.json instead.
This commit is contained in:
@@ -0,0 +1,54 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
identity.py — provider-agnostic match keys.
|
||||
|
||||
The pipeline output and the ground truth may not share one id space: an output
|
||||
actor can carry only tmdb/jellyfin ids while X-Ray/MovieNet key on IMDb nm-ids.
|
||||
We represent each actor by the *set* of every key we can derive, and treat two
|
||||
actors as the same iff their key sets intersect. Namespacing each key by its
|
||||
provider prevents cross-provider collisions (e.g. an nm-number equalling a
|
||||
tmdb-number).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import unicodedata
|
||||
|
||||
|
||||
def norm_name(name: str | None) -> str | None:
|
||||
"""Lowercased, accent-stripped, punctuation-free name for fuzzy fallback match."""
|
||||
if not name:
|
||||
return None
|
||||
s = unicodedata.normalize("NFKD", name)
|
||||
s = "".join(c for c in s if not unicodedata.combining(c))
|
||||
s = re.sub(r"[^a-z0-9 ]+", "", s.lower()).strip()
|
||||
s = re.sub(r"\s+", " ", s)
|
||||
return s or None
|
||||
|
||||
|
||||
def keys_for(imdb_id: str | None = None,
|
||||
tmdb_id: str | None = None,
|
||||
jellyfin_id: str | None = None,
|
||||
name: str | None = None,
|
||||
crosswalk=None) -> set[str]:
|
||||
"""All identity tokens for one actor. Empty strings are ignored.
|
||||
|
||||
If `crosswalk` (a CrosswalkTable) is given and no imdb_id is present, resolve
|
||||
tmdb_id → imdb_id through it so a tmdb-only actor still gets an exact `imdb:`
|
||||
key — turning the fuzzy name join into an exact id join. See tmdb_imdb_map.py.
|
||||
"""
|
||||
keys: set[str] = set()
|
||||
imdb = imdb_id.strip() if (imdb_id and imdb_id.strip()) else None
|
||||
if not imdb and crosswalk is not None and tmdb_id:
|
||||
imdb = crosswalk.imdb_for(tmdb_id)
|
||||
if imdb:
|
||||
keys.add(f"imdb:{imdb}")
|
||||
if tmdb_id and str(tmdb_id).strip():
|
||||
keys.add(f"tmdb:{str(tmdb_id).strip()}")
|
||||
if jellyfin_id and jellyfin_id.strip():
|
||||
keys.add(f"jf:{jellyfin_id.strip()}")
|
||||
nn = norm_name(name)
|
||||
if nn:
|
||||
keys.add(f"name:{nn}")
|
||||
return keys
|
||||
Reference in New Issue
Block a user