feat(tooling): X-Ray threshold optimizer, gallery utilities, artifact registry, docs build

Optimizer (scripts/optimizer/): replay.py runs the real C++ tracker/matcher/
scene_tracker chain over a dumped-embeddings HDF5 via sae_kpn, so a threshold
sweep never re-decodes video or re-embeds faces. optimize.py drives scipy's
differential_evolution over the knob space, with DE-level parallelism
(multiple population candidates evaluated concurrently via a ThreadPoolExecutor)
on top of per-film replay parallelism. second_score.py is the per-second X-Ray
scoring metric (TPI/FPI/FN, out-of-cast misID weighted 10x, fair recall masked
to gallery-known cast) that superseded an earlier scene-union metric.
dump_error_frames.py / dump_scene_montage.py extract annotated video frames
(bounding boxes, TPI/FPI/FN captions, onscreen-vs-offscreen split) for visual
review of a replay against ground truth. Gallery utilities: cast_restrict.py,
gallery_membership.py, fetch_missing_actors.py, reembed_gallery.py.

scripts/validation/: X-Ray ground-truth loading and provider-agnostic identity
matching (identity.py's keys_for — an actor is the union of every id we can
derive, since pipeline output and ground truth don't share one id space).

scripts/artifacts/: push/pull scripts for the Gitea generic package registry —
galleries, montage frames, and experiment data (manifests/trajectories/results)
are pushed there instead of committed, since none are needed to run the app,
only benchmarks. Versioned by git short-SHA.

scripts/docs/: MkDocs site build (build_site.sh) and the calibration-curve
comparison chart (calibration_chart.py, matplotlib, reads each gallery's
embedded calibration).

Gallery-building scripts (make_jellyfin_gallery.py, make_gallery.py,
filter_gallery.py, run_from_jellyfin.py, movienet_eval.py, movienet_prep.py,
sae_gallery.py) updated to read/write HDF5 galleries exclusively, matching the
engine-side format switch. run_from_jellyfin.py and the optimizer no longer
carry movie source paths in shared manifests (some source filenames include
scene-release tags) — resolved locally via a gitignored file-lut.json instead.
This commit is contained in:
2026-07-19 19:06:48 +02:00
parent 26139ffe8a
commit 6f0ad83a55
31 changed files with 3411 additions and 47 deletions
+94
View File
@@ -0,0 +1,94 @@
#!/usr/bin/env python3
"""
gallery_membership.py — definitive per-film gallery coverage of X-Ray cast.
For each film, splits the X-Ray cast (people.csv) into those WITH a gallery reference
embedding and those WITHOUT. This is the model-independent foundation for honest
FP/FN rates: because every model's gallery is built from the SAME TMDB source images
(same actors), the membership list is identical across models — only the embedding
values differ. So FN can be measured over the recognisable denominator (in-gallery
cast) and out-of-cast misIDs (predicted actor not in the film at all) are well defined.
Outputs experiments/results/membership.json:
{ film: {
xray_cast: N, in_gallery: M, coverage: M/N,
in_gallery_names: [...], missing_names: [...] } }
Usage:
python scripts/optimizer/gallery_membership.py \
--manifest experiments/manifests/films.json \
--gallery gallery_arcface_w600k_r50.json \
--out experiments/results/membership.json
"""
from __future__ import annotations
import argparse
import csv
import json
import sys
from pathlib import Path
REPO = Path(__file__).resolve().parent.parent.parent
sys.path.insert(0, str(REPO / "scripts" / "validation"))
from identity import keys_for # noqa: E402
def gallery_keyset(gallery_path: str) -> set:
keys = set()
for a in json.loads(Path(gallery_path).read_text())["actors"]:
if not a.get("embeddings"):
continue # no embedding = not actually recognisable
keys |= keys_for(imdb_id=a.get("imdb_id"), tmdb_id=a.get("tmdb_id"),
jellyfin_id=a.get("jellyfin_id"), name=a.get("name"))
return keys
def film_cast(xray_dir: str) -> dict[str, str]:
"""nm_id → person name from a film's X-Ray people.csv."""
out = {}
with open(Path(xray_dir) / "people.csv", newline="", encoding="utf-8") as f:
for r in csv.DictReader(f):
nm = (r.get("name_id") or "").strip()
if nm:
out[nm] = (r.get("person") or "").strip()
return out
def main():
p = argparse.ArgumentParser(description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
p.add_argument("--manifest", required=True)
p.add_argument("--gallery", required=True)
p.add_argument("--out", required=True)
args = p.parse_args()
gkeys = gallery_keyset(args.gallery)
films = json.loads(Path(args.manifest).read_text())
report = {}
tot_cast = tot_in = 0
print(f"{'film':32s} {'cast':>5s} {'in-gal':>7s} {'cover':>6s}")
for f in films:
cast = film_cast(f["xray"])
in_g, miss = [], []
for nm, name in cast.items():
if keys_for(imdb_id=nm, name=name) & gkeys:
in_g.append(name)
else:
miss.append(name)
n, m = len(cast), len(in_g)
tot_cast += n; tot_in += m
report[f["name"]] = {"xray_cast": n, "in_gallery": m,
"coverage": round(m / n, 3) if n else 0.0,
"in_gallery_names": sorted(in_g),
"missing_names": sorted(miss)}
print(f"{f['name'][:32]:32s} {n:>5d} {m:>7d} {m/n*100 if n else 0:>5.0f}%")
print(f"{'TOTAL':32s} {tot_cast:>5d} {tot_in:>7d} {tot_in/tot_cast*100:>5.0f}%")
Path(args.out).parent.mkdir(parents=True, exist_ok=True)
Path(args.out).write_text(json.dumps(report, indent=2))
print(f"\n{args.out}")
if __name__ == "__main__":
main()