docs: full data-grounded rewrite of the performance report
Replaces narrative claims with verified numbers across all report pages: - Cross-model held-out validation (LVFace/mbf/r18, all 5 held-out films): LVFace wins every film outright, not just "consistent with" the training-set pick. r50 dropped from the detailed comparison (gallery has ~30% fewer reference images per actor than the other three models on identical source photos). - Per-film training breakdown: LVFace does not win every training film (mbf beats it on Lord of War); the 75.3% macro figure hides a 10.7pp spread. - Gallery coverage computed per film (20.3%-78.6%) instead of one flat 67%-missing average. - Found and fixed a real scoring bug in optimize.py: a candidate whose hardest film's replay timed out was averaged over survivors instead of penalized, silently rewarding partial coverage. Affected 3 of 16 training combos; corrected throughout, and optimize.py now scores an incomplete evaluation f1=0.0 instead of averaging over whichever films happened to finish. - Every FPI frame in the deep dive now comes from the proper montage renderer (Onscreen/Offscreen panel, ghosts never drawn as boxes), never the bare-box debug overlay used earlier. - Every distinct out-of-cast name across all 9 films gets its own frame at its first appearance (9 names, 4 films), not a single-example spot check: 2 ground-truth gaps, 1 photograph misread as a person, 6 genuine lookalike confusions. - New methodology.md: the scene-level-vs-per-second scoring mismatch that the rest of the report assumes, written out once. - Cut the deadlock/gdb debugging narrative from the experiment log; kept the one fact that matters (KPN's node/network split lets the expensive GPU stage run once and the cheap stage replay against cached embeddings). - Plain declarative style throughout, no em dashes, no blog voice.
This commit is contained in:
@@ -0,0 +1,62 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
gallery_coverage_per_film.py — fraction of each film's X-Ray credited cast
|
||||
that has a reference embedding in the gallery, computed per film rather than
|
||||
as a single benchmark-wide average.
|
||||
|
||||
Usage: python3 scripts/docs/gallery_coverage_per_film.py --out docs_data/gallery_coverage_per_film.json
|
||||
"""
|
||||
import argparse
|
||||
import csv
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import h5py
|
||||
|
||||
REPO = Path(__file__).resolve().parent.parent.parent
|
||||
sys.path.insert(0, str(REPO / "scripts" / "validation"))
|
||||
from identity import keys_for # noqa: E402
|
||||
|
||||
|
||||
def main():
|
||||
p = argparse.ArgumentParser()
|
||||
p.add_argument("--gallery", default=str(REPO / "experiments/galleries/gallery_LVFace-B_Glint360K.h5"))
|
||||
p.add_argument("--films", default=str(REPO / "experiments/manifests/films.json"))
|
||||
p.add_argument("--out", required=True)
|
||||
args = p.parse_args()
|
||||
|
||||
films = json.load(open(args.films))
|
||||
with h5py.File(args.gallery, "r") as f:
|
||||
names = [n.decode() if isinstance(n, bytes) else n for n in f["name"][:]]
|
||||
jids = [j.decode() if isinstance(j, bytes) else j for j in f["jellyfin_id"][:]]
|
||||
imdbs = [j.decode() if isinstance(j, bytes) else j for j in f["imdb_id"][:]]
|
||||
gallery_keys = set()
|
||||
for n, j, im in zip(names, jids, imdbs):
|
||||
gallery_keys |= keys_for(imdb_id=im, name=n, jellyfin_id=j)
|
||||
|
||||
out = []
|
||||
for film in films:
|
||||
xray_dir = REPO / film["xray"]
|
||||
id_to_name = {}
|
||||
with open(xray_dir / "people.csv", newline="", encoding="utf-8") as fh:
|
||||
for r in csv.DictReader(fh):
|
||||
nm = (r.get("name_id") or "").strip()
|
||||
if nm:
|
||||
id_to_name[nm] = (r.get("person") or "").strip()
|
||||
cast_keys = [keys_for(imdb_id=nm, name=name) for nm, name in id_to_name.items()]
|
||||
covered = sum(1 for ck in cast_keys if ck & gallery_keys)
|
||||
total = len(cast_keys)
|
||||
out.append({"film": film["name"], "cast_total": total, "covered": covered,
|
||||
"coverage_pct": round(covered / total * 100, 1) if total else 0.0})
|
||||
|
||||
out.sort(key=lambda x: x["coverage_pct"])
|
||||
Path(args.out).parent.mkdir(parents=True, exist_ok=True)
|
||||
json.dump(out, open(args.out, "w"), indent=1)
|
||||
for o in out:
|
||||
print(f"{o['film']:45s} {o['covered']:3d}/{o['cast_total']:3d} ({o['coverage_pct']}%)",
|
||||
file=sys.stderr)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user