diff --git a/docs/requirements.md b/docs/requirements.md index 6d2f909..1abe90e 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -102,7 +102,7 @@ Status: `Done` · `In Progress` · `Planned` · `TBD` · `Withdrawn` | VR-002 | Replay drives the **real** KPN nodes, not a reimplementation | PR-002 | High | Done | | VR-003 | Scoring: micro-F1 against X-Ray, precision/recall logged at every evaluation | PR-002 | High | Done | | VR-004 | Reproducible validation corpus with ground truth | PR-002 | High | Done | -| VR-005 | Minimum face size study — TPI/FPI vs probe size, gallery held at native res | PR-002 | Medium | Planned | +| VR-005 | Minimum face size study — TPI/FPI vs probe size, gallery held at native res | PR-002 | Medium | Done | | VR-006 | Re-tune `scene_threshold` once native-rate decode lands | PR-002 | Low | Planned | | VR-007 | Expansion band, clustering threshold, and deferred-pass ablation | PR-002 | Medium | Planned | | VR-008 | Gallery scaling benchmark — throughput vs gallery size | PR-002 | Medium | Planned | diff --git a/scripts/validation/README.md b/scripts/validation/README.md index 0f165bc..8f97150 100644 --- a/scripts/validation/README.md +++ b/scripts/validation/README.md @@ -79,9 +79,30 @@ The table caches nulls (tmdb ids TMDB has no IMDb id for) and checkpoints, so a re-run only resolves new ids. TMDB is authoritative for this crosswalk — there is no clean free bulk `tmdb_person ↔ nm` file, so we query the API once and cache. +## Minimum face size (VR-005) + +`min_face_size.py` is a separate, self-contained study: it needs no video and no +ground truth, only the gallery mugshot cache. It holds out one image per actor, +degrades that probe to each candidate face size and matches it against a gallery +held at **native** resolution, reporting TPI/FPI per size — the measurement that +replaces AR-002's 66×66 px estimate. + +```bash +python scripts/validation/min_face_size.py \ + --images images --gallery gallery_lvface.h5 \ + --arcface models/LVFace-B_Glint360K.onnx \ + --actors 100 --out experiments/results/vr005_min_face_size +``` + +FPI grows with the number of actors competing, so a 100-actor run understates it +against a library of thousands: read FPI as relative across sizes, not as an +absolute rate. Re-run per `--arcface` model to see whether `min_face_px` should be +one constant or scale with the embedder (GR-004). + ## Files - `sample_eval.py` — CLI scorer. - `ground_truth.py` — `XRayGroundTruth`, `MovieNetGroundTruth` loaders. - `identity.py` — provider-agnostic match keys. - `tmdb_imdb_map.py` — build/consult the cached `tmdb→imdb` crosswalk. +- `min_face_size.py` — VR-005 probe-size sweep (see above). - `test_sample_eval.py` — self-contained tests (`python scripts/validation/test_sample_eval.py`). diff --git a/scripts/validation/min_face_size.py b/scripts/validation/min_face_size.py new file mode 100644 index 0000000..5e62437 --- /dev/null +++ b/scripts/validation/min_face_size.py @@ -0,0 +1,1015 @@ +#!/usr/bin/env python3 +""" +min_face_size.py — VR-005: at what face size do embeddings stop identifying people? + +TRACES: VR-005 + +`min_face_px` is currently a working estimate (AR-002: 66x66 px in original video +resolution). This script replaces the guess with a measurement, using only gallery +mugshots already on disk — no video, no C++ changes. + +Protocol +-------- +1. Select ~100 gallery actors that have more than one mugshot. +2. Per actor hold out ONE image as the *probe*; that actor's remaining images stay + in the gallery at native resolution. +3. For each target size S, take the probe's native aligned 112x112 crop, downscale + it to SxS and upscale it back to 112x112, then embed. Detail is genuinely + destroyed and then the same warp the pipeline applies is re-applied on top — + which is what a face detected at SxS in a frame actually suffers. +4. Match each degraded probe against the whole gallery. +5. Record, per size, TPI (identified as the correct actor) and FPI (identified as + someone else). Everything else is an unidentified probe (TBI). + +The asymmetry is the point: **the gallery stays at native resolution and only the +probe degrades.** That is the production case — reference mugshots are clean, the +face coming out of the video is small. Degrading both sides would measure +something the pipeline never does. + +What this deliberately does NOT measure +--------------------------------------- +The cosine between the size-S embedding and the native embedding of the *same* +image. That is embedding *drift*, and it answers the wrong question: an embedding +can drift a long way and stay perfectly separable, or drift a little in a +direction that destroys separation. What matters is the decision the pipeline +makes — probe against a competing gallery — so that is what is recorded. + +CAVEAT — FPI IS RELATIVE, NOT ABSOLUTE +-------------------------------------- +False positives grow with the number of actors competing for the match. A +~100-actor gallery therefore *understates* the false-positive rate against a +production library of thousands. Read the FPI column as a relative curve across +sizes ("FPI is 4x worse at 32 px than at 64 px"), never as the rate you would see +in production. Re-run with `--actors` at production scale before setting a +threshold from an absolute FPI number. + +Decision rule +------------- +Per the repo invariant (CLAUDE.md: "always use the calibrated probability, never a +raw cosine"), identification goes through the same path as `identity_matcher_node`: +per-actor best-of-N cosine -> Platt sigmoid P(match) = sigma(a*sim + b + log-prior) +-> accept if P > `prob_threshold`. The sigmoid is fitted here by the same +histogram/gradient-descent procedure as `src/gallery/gallery_calibration.hpp`, +over the native gallery embeddings only (held-out probes are excluded, so the +calibration cannot see the images it will be scored on). + +Why an ONNX Runtime pipeline instead of the `sae_embed` module +-------------------------------------------------------------- +`sae_embed.FaceEmbedder` only exposes `embed(path)` — detect, align and embed in +one step — so it cannot embed a crop the caller has degraded. + + SUPERSEDED: sae_embed now binds the production stages directly — + detect(), align_face(), embed_crop() and GalleryCalibration — so the ports + below can be deleted and this driven off the shipped C++ instead. Do that + before extending them: a second implementation of the calibration is + exactly where the "always the calibrated probability, never a raw cosine" + rule gets broken silently. This script +therefore drives `scrfd_500m_bnkps.onnx` and the embedder ONNX directly, porting +`SCRFDDecoder` / `ArcFaceEmbedder` (src/backends/ort_backend.cpp), `align_face` / +`enhance_for_retry` (src/face_utils.hpp) and `calibrate_gallery` +(src/gallery/gallery_calibration.hpp). + +That means the ONNX-Runtime fp32 backend — the reference one +(`SAE_INFERENCE_BACKEND=ORT`), which loads the .onnx directly. A TensorRT fp16 +build is a *different realisation* of the same model and its embeddings are +measurably not the same vectors: on LVFace-B_Glint360K the stored TRT-fp16 gallery +agrees with an fp32 recompute of the same mugshot at only ~0.85 cosine, while +same-actor/different-actor separation is essentially unchanged (d' 5.3 vs 5.7). +Nothing here is invalidated by that — gallery and probes go through one session, +so the comparison is internally consistent — but the two embedding spaces are not +interchangeable, and `--verify-against ` will show ~0.85, not ~1.0, +against a TRT-built gallery. It reports the separation of both sets alongside the +agreement so the two causes are distinguishable: a broken port collapses +separation, a different backend does not. + +Secondary output (VR-005): running the sweep per `--arcface` model shows whether +`min_face_px` should be one constant at all, or should scale with the embedder — +which matters because the model is a build-time choice (GR-004). + +Usage +----- + python scripts/validation/min_face_size.py \ + --images images \ + --gallery gallery_lvface.h5 \ + --arcface models/LVFace-B_Glint360K.onnx \ + --actors 100 --seed 0 \ + --out experiments/results/vr005_min_face_size + +Writes .csv, .json and .png (plus .per_probe.csv with +--per-probe). +""" +from __future__ import annotations + +import argparse +import csv +import json +import random +import re +import sys +import time +from pathlib import Path + +import cv2 +import numpy as np + +REPO = Path(__file__).resolve().parent.parent.parent +sys.path.insert(0, str(REPO / "scripts")) + +# ArcFace 5-point reference landmarks in the 112x112 aligned frame. +# Mirrors kArcFaceRef in src/types.hpp. +ARCFACE_REF = np.array([ + [38.2946, 51.6963], + [73.5318, 51.5014], + [56.0252, 71.7366], + [41.5493, 92.3655], + [70.7299, 92.2041], +], dtype=np.float32) + +IMAGE_EXTS = (".jpg", ".jpeg", ".png", ".webp") +JELLYFIN_ID_RE = re.compile(r"^[0-9a-f]{32}$") + +# Interpolation used for the two halves of the degradation. Downscaling uses +# INTER_AREA (correct low-pass for shrinking, i.e. detail that a small detection +# genuinely never had); upscaling uses INTER_LINEAR, which is what warpAffine in +# align_face() uses when it blows a small detection up to 112x112. +INTERP = { + "area": cv2.INTER_AREA, + "linear": cv2.INTER_LINEAR, + "cubic": cv2.INTER_CUBIC, + "nearest": cv2.INTER_NEAREST, + "lanczos": cv2.INTER_LANCZOS4, +} + +# House chart palette, shared with scripts/docs/experiment_charts.py so figures +# across the report read as one set. +INK, MUTED, GRID, SURFACE = "#0b0b0b", "#898781", "#e1e0d9", "#fcfcfb" +BLUE, GREEN, RED, AMBER = "#2a78d6", "#008300", "#e34948", "#eda100" + + +# ── SCRFD detector (port of SCRFDDecoder, src/backends/ort_backend.cpp) ──────── + +class SCRFDDetector: + """SCRFD-with-keypoints decoder: letterbox to 640x640, decode strides 8/16/32 + (/64 for a 12-output model) at 2 anchors each, then NMS.""" + + INPUT_W = 640 + INPUT_H = 640 + STRIDES = (8, 16, 32, 64) + ANCHORS = 2 + + def __init__(self, model_path: str, providers: list[str], + conf: float = 0.5, nms: float = 0.4): + import onnxruntime as ort + opts = ort.SessionOptions() + opts.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL + self.sess = ort.InferenceSession(model_path, opts, providers=providers) + self.input_name = self.sess.get_inputs()[0].name + self.out_names = [o.name for o in self.sess.get_outputs()] + n_out = len(self.out_names) + if n_out % 3 != 0 or not (9 <= n_out <= 12): + raise SystemExit( + f"[scrfd] expected 9 or 12 outputs (kps-variant model), got {n_out}: " + f"{model_path}") + self.fmc = n_out // 3 + # Reject non-SCRFD models with the same output count (e.g. YuNet). + for gi, last in enumerate((1, 4, 10)): + for si in range(self.fmc): + shape = self.sess.get_outputs()[gi * self.fmc + si].shape + if not shape or shape[-1] != last: + raise SystemExit( + f"[scrfd] {model_path} does not look like InsightFace SCRFD: " + f"output '{self.out_names[gi * self.fmc + si]}' last-dim is " + f"{shape[-1] if shape else None}, expected {last}. " + f"Hint: pass scrfd_500m_bnkps.onnx, not yunet/*.onnx.") + self.conf = conf + self.nms = nms + + def detect(self, img: np.ndarray) -> list[dict]: + h, w = img.shape[:2] + scale = min(self.INPUT_W / w, self.INPUT_H / h) + new_w, new_h = int(round(w * scale)), int(round(h * scale)) + pad_x, pad_y = (self.INPUT_W - new_w) // 2, (self.INPUT_H - new_h) // 2 + + resized = cv2.resize(img, (new_w, new_h), interpolation=cv2.INTER_LINEAR) + letterboxed = np.full((self.INPUT_H, self.INPUT_W, 3), 114, dtype=img.dtype) + letterboxed[pad_y:pad_y + new_h, pad_x:pad_x + new_w] = resized + + blob = cv2.dnn.blobFromImage(letterboxed, 1.0 / 128.0, + (self.INPUT_W, self.INPUT_H), + (127.5, 127.5, 127.5), swapRB=True, crop=False) + outs = self.sess.run(self.out_names, {self.input_name: blob}) + + boxes, scores, kpss = [], [], [] + for si in range(self.fmc): + stride = self.STRIDES[si] + fw = self.INPUT_W // stride + s = np.asarray(outs[si]).reshape(-1) + b = np.asarray(outs[self.fmc + si]).reshape(-1, 4) + k = np.asarray(outs[self.fmc * 2 + si]).reshape(-1, 10) + + keep = np.nonzero(s >= self.conf)[0] + if keep.size == 0: + continue + # idx = (r * fw + c) * ANCHORS + a -> anchor centres + cell = keep // self.ANCHORS + cx = (cell % fw).astype(np.float32) * stride + cy = (cell // fw).astype(np.float32) * stride + + x1 = (cx - b[keep, 0] * stride - pad_x) / scale + y1 = (cy - b[keep, 1] * stride - pad_y) / scale + x2 = (cx + b[keep, 2] * stride - pad_x) / scale + y2 = (cy + b[keep, 3] * stride - pad_y) / scale + + kp = k[keep].reshape(-1, 5, 2) * stride + kp[:, :, 0] = (kp[:, :, 0] + cx[:, None] - pad_x) / scale + kp[:, :, 1] = (kp[:, :, 1] + cy[:, None] - pad_y) / scale + + boxes.append(np.stack([x1, y1, x2 - x1, y2 - y1], axis=1)) + scores.append(s[keep]) + kpss.append(kp) + + if not boxes: + return [] + boxes = np.concatenate(boxes).astype(np.float64) + scores = np.concatenate(scores).astype(np.float32) + kpss = np.concatenate(kpss).astype(np.float32) + + keep = cv2.dnn.NMSBoxes(boxes.tolist(), scores.tolist(), self.conf, self.nms) + if keep is None or len(keep) == 0: + return [] + keep = np.asarray(keep).reshape(-1) + + faces = [] + for i in keep: + x = max(0.0, float(boxes[i, 0])) + y = max(0.0, float(boxes[i, 1])) + faces.append({ + "bbox": (x, y, + min(float(boxes[i, 2]), w - x), + min(float(boxes[i, 3]), h - y)), + "confidence": float(scores[i]), + "landmarks": kpss[i].copy(), + }) + return faces + + +# ── Alignment (port of src/face_utils.hpp) ──────────────────────────────────── + +def align_face(img: np.ndarray, landmarks: np.ndarray) -> np.ndarray | None: + """112x112 BGR crop via the ArcFace 5-point similarity transform.""" + M, _ = cv2.estimateAffinePartial2D(landmarks.astype(np.float32), ARCFACE_REF, + method=cv2.RANSAC, ransacReprojThreshold=3.0) + if M is None: + return None + return cv2.warpAffine(img, M, (112, 112), flags=cv2.INTER_LINEAR, + borderMode=cv2.BORDER_CONSTANT, borderValue=(0, 0, 0)) + + +def enhance_for_retry(img: np.ndarray) -> np.ndarray: + """Border-replicate pad by 50% and CLAHE the luminance, so a detector that + found nothing gets a second try. Same as src/face_utils.hpp.""" + pad_x, pad_y = img.shape[1] // 4, img.shape[0] // 4 + padded = cv2.copyMakeBorder(img, pad_y, pad_y, pad_x, pad_x, cv2.BORDER_REPLICATE) + lab = cv2.cvtColor(padded, cv2.COLOR_BGR2Lab) + l, a, b = cv2.split(lab) + l = cv2.createCLAHE(clipLimit=2.0, tileGridSize=(8, 8)).apply(l) + return cv2.cvtColor(cv2.merge([l, a, b]), cv2.COLOR_Lab2BGR) + + +# ── Embedder (port of ArcFaceEmbedder, src/backends/ort_backend.cpp) ────────── + +class Embedder: + """112x112 BGR crops -> L2-normalised 512-d embeddings. + Input is BGR->RGB, scaled to [-1, 1] as (px - 127.5) / 128.""" + + def __init__(self, model_path: str, providers: list[str], batch: int = 16): + import onnxruntime as ort + opts = ort.SessionOptions() + opts.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL + self.sess = ort.InferenceSession(model_path, opts, providers=providers) + self.input_name = self.sess.get_inputs()[0].name + self.output_name = self.sess.get_outputs()[0].name + self.fp16 = "float16" in self.sess.get_inputs()[0].type + self.batch = max(1, self._probe_batch(batch)) + + def _probe_batch(self, batch: int) -> int: + """Some exports pin the batch dimension. Try a small batch once and fall + back to 1 rather than failing halfway through the sweep.""" + if batch <= 1: + return 1 + n = min(batch, 4) + dummy = np.zeros((n, 3, 112, 112), np.float16 if self.fp16 else np.float32) + try: + out = self.sess.run([self.output_name], {self.input_name: dummy})[0] + except Exception as e: # noqa: BLE001 — any ORT shape/type rejection + print(f"[embed] batching unsupported ({e}); falling back to batch=1", + file=sys.stderr) + return 1 + if out.shape[0] != n: + print(f"[embed] model returned {out.shape[0]} rows for a batch of {n}; " + f"falling back to batch=1", file=sys.stderr) + return 1 + return batch + + def embed(self, crops: list[np.ndarray]) -> np.ndarray: + if not crops: + return np.zeros((0, 512), np.float32) + out = np.empty((len(crops), 512), np.float32) + for i in range(0, len(crops), self.batch): + chunk = crops[i:i + self.batch] + rgbs = [cv2.cvtColor(c, cv2.COLOR_BGR2RGB) for c in chunk] + blob = cv2.dnn.blobFromImages(rgbs, 1.0 / 128.0, (112, 112), + (127.5, 127.5, 127.5), + swapRB=False, crop=False) + if self.fp16: + blob = blob.astype(np.float16) + raw = self.sess.run([self.output_name], {self.input_name: blob})[0] + out[i:i + len(chunk)] = np.asarray(raw, dtype=np.float32) + norms = np.linalg.norm(out, axis=1, keepdims=True) + return out / np.maximum(norms, 1e-6) + + +# ── Calibration (port of src/gallery/gallery_calibration.hpp) ───────────────── + +MIN_EMB_FOR_POSITIVE = 5 +DEDUP_SIM = 1.0 - 1e-7 +HIST_BINS = 200 + + +def _sigmoid(z: np.ndarray | float) -> np.ndarray | float: + return np.where(z >= 0, 1.0 / (1.0 + np.exp(-np.abs(z))), + np.exp(-np.abs(z)) / (1.0 + np.exp(-np.abs(z)))) + + +def calibrate_gallery(emb: np.ndarray, actor: np.ndarray) -> dict: + """Fit P(match) = sigma(a*sim + b) from intra-class (same actor, different + reference image) and inter-class pairs, exactly as calibrate_gallery() does: + per-actor dedup, actors with < 5 distinct embeddings contribute negatives + only, similarities bucketed into 200 bins, class-weighted gradient descent.""" + n_actors = int(actor.max()) + 1 if actor.size else 0 + + keep_rows, eligible = [], np.zeros(n_actors, bool) + for ai in range(n_actors): + rows = np.nonzero(actor == ai)[0] + kept: list[int] = [] + for r in rows: + if all(float(emb[r] @ emb[k]) <= DEDUP_SIM for k in kept): + kept.append(int(r)) + eligible[ai] = len(kept) >= MIN_EMB_FOR_POSITIVE + keep_rows.extend(kept) + + keep_rows = np.asarray(sorted(keep_rows), dtype=int) + e, a_idx = emb[keep_rows], actor[keep_rows] + n = len(keep_rows) + print(f"[calibration] dedup: {len(emb)} -> {n} embeddings " + f"({int(eligible.sum())}/{n_actors} actors have >= {MIN_EMB_FOR_POSITIVE} " + f"distinct embeddings, eligible for positive pairs)", file=sys.stderr) + if n < 2: + return {"a": 10.0, "b": -5.0, "valid": False} + + S = e @ e.T + iu, ju = np.triu_indices(n, k=1) + sims = S[iu, ju] + same = a_idx[iu] == a_idx[ju] + pos_mask = same & eligible[a_idx[iu]] + neg_mask = ~same + + bw = 2.0 / HIST_BINS + bin_idx = np.clip(((sims + 1.0) / bw).astype(int), 0, HIST_BINS - 1) + pos = np.bincount(bin_idx[pos_mask], minlength=HIST_BINS).astype(np.float64) + neg = np.bincount(bin_idx[neg_mask], minlength=HIST_BINS).astype(np.float64) + + n_pos, n_neg = pos.sum(), neg.sum() + if n_pos < 2 or n_neg < 1: + print(f"[calibration] insufficient pairs (+{n_pos:.0f}/-{n_neg:.0f}) — " + f"calibration skipped", file=sys.stderr) + return {"a": 10.0, "b": -5.0, "valid": False} + + total = n_pos + n_neg + w_pos, w_neg = total / (2.0 * n_pos), total / (2.0 * n_neg) + centers = -1.0 + (np.arange(HIST_BINS) + 0.5) * bw + + a, b = 10.0, -5.0 + lr, max_iter, tol = 0.05, 20000, 1e-7 + for _ in range(max_iter): + sig = _sigmoid(a * centers + b) + err = (sig - 1.0) * w_pos * pos + sig * w_neg * neg + da = float((err * centers).sum()) / total + db = float(err.sum()) / total + a -= lr * da + b -= lr * db + if da * da + db * db < tol * tol: + break + + sig = _sigmoid(a * centers + b) + correct = float(np.where(sig > 0.5, pos, neg).sum()) + boundary = (0.0 - b) / a + print(f"[calibration] sigmoid fitted: a={a:.4f} b={b:.4f} " + f"boundary(P=0.5)=sim{boundary:.4f} pairs={int(total)} " + f"(+{int(n_pos)}/-{int(n_neg)}) bins={HIST_BINS} " + f"train_acc={100.0 * correct / total:.2f}%", file=sys.stderr) + return {"a": float(a), "b": float(b), "valid": True} + + +def probability(sim, a: float, b: float, log_prior_odds: float = 0.0): + return _sigmoid(a * np.asarray(sim, dtype=np.float64) + b + log_prior_odds) + + +# ── Runtime / actor discovery ───────────────────────────────────────────────── + +def resolve_providers(requested: str) -> list[str]: + """Keep only providers this onnxruntime build actually has — asking for an + absent one is a hard error in recent versions, and CUDA is routinely absent.""" + import onnxruntime as ort + available = ort.get_available_providers() + keep = [p for p in (s.strip() for s in requested.split(",")) if p in available] + dropped = [p for p in (s.strip() for s in requested.split(",")) if p not in available] + if dropped: + print(f"[models] providers unavailable, skipping: {', '.join(dropped)} " + f"(have: {', '.join(available)})", file=sys.stderr) + return keep or ["CPUExecutionProvider"] + + +def normalise_name(name: str) -> str: + return re.sub(r"[^a-z0-9]+", "", name.lower()) + + +def discover_actors(images_root: Path) -> list[dict]: + """Enumerate the gallery-build image cache: /_/NN.jpg + (the layout make_jellyfin_gallery.py / reembed_gallery.py use).""" + actors = [] + for d in sorted(p for p in images_root.iterdir() if p.is_dir()): + imgs = sorted(p for p in d.iterdir() + if p.is_file() and p.suffix.lower() in IMAGE_EXTS) + if not imgs: + continue + head, _, tail = d.name.partition("_") + if JELLYFIN_ID_RE.match(head) and tail: + jellyfin_id, name = head, tail.replace("_", " ") + else: + jellyfin_id, name = "", d.name.replace("_", " ") + actors.append({"dir": d, "jellyfin_id": jellyfin_id, "name": name, + "images": imgs}) + return actors + + +def gallery_keys(gallery_path: Path) -> tuple[set[str], set[str]]: + """(jellyfin ids, normalised names) of the actors an existing gallery holds.""" + from sae_gallery import load_gallery_hdf5 + g = load_gallery_hdf5(gallery_path) + ids = {a.get("jellyfin_id", "") for a in g["actors"] if a.get("jellyfin_id")} + names = {normalise_name(a.get("name", "")) for a in g["actors"] if a.get("name")} + return ids, names + + +# ── Degradation ─────────────────────────────────────────────────────────────── + +def degrade(crop: np.ndarray, size: int, down: int, up: int) -> np.ndarray: + """Throw away everything a face detected at size x size never had, then warp + it back up to the 112x112 the embedder is fed.""" + if size == 112: + return crop + small = cv2.resize(crop, (size, size), interpolation=down) + return cv2.resize(small, (112, 112), interpolation=up) + + +# ── Reporting ───────────────────────────────────────────────────────────────── + +CAVEAT = ( + "CAVEAT: FPI grows with gallery size. This ran against {n_actors} actors, so it " + "UNDERSTATES the false-positive rate of a production library of thousands. Read " + "FPI as relative across sizes, not as an absolute rate." +) + + +def write_plot(rows: list[dict], out_png: Path, meta: dict) -> None: + import matplotlib + matplotlib.use("Agg") + import matplotlib.pyplot as plt + + plt.rcParams.update({ + "figure.facecolor": SURFACE, "axes.facecolor": SURFACE, + "savefig.facecolor": SURFACE, "text.color": INK, + "axes.edgecolor": MUTED, "axes.labelcolor": INK, + "xtick.color": MUTED, "ytick.color": MUTED, + "axes.grid": True, "grid.color": GRID, "grid.linewidth": 0.8, + "axes.spines.top": False, "axes.spines.right": False, + }) + + sizes = [r["size_px"] for r in rows] + fig, ax = plt.subplots(figsize=(9, 5.6)) + ax.plot(sizes, [100 * r["tpi_rate"] for r in rows], "-o", color=GREEN, + lw=2, label="TPI — identified, correct actor") + ax.plot(sizes, [100 * r["fpi_rate"] for r in rows], "-s", color=RED, + lw=2, label="FPI — identified, wrong actor") + ax.plot(sizes, [100 * r["unidentified_rate"] for r in rows], color=MUTED, + marker="^", lw=1.4, ls="--", label="unidentified (below P threshold)") + ax.plot(sizes, [100 * r["rank1_rate"] for r in rows], ":", color=BLUE, + lw=1.6, label="rank-1 correct (ignoring threshold)") + + op = meta.get("operating_point") + if op: + ax.axvline(op, color=AMBER, lw=1.6, ls="-.", zorder=1) + ax.annotate(f"operating point {op} px", xy=(op, 50), + xytext=(4, 0), textcoords="offset points", + color=AMBER, fontsize=9, rotation=90, va="center") + + ax.set_xlabel("probe face size before upscaling (px)") + ax.set_ylabel("% of probes") + ax.set_ylim(-2, 102) + ax.set_xticks(sizes) + ax.set_title(f"VR-005 — identification vs. probe face size\n" + f"{meta['model']}, {meta['n_actors']} actors, " + f"{meta['n_probes']} probes/size, gallery at native resolution", + fontsize=11, loc="left") + ax.legend(frameon=False, fontsize=9, loc="center left") + fig.text(0.01, 0.005, CAVEAT.format(n_actors=meta["n_actors"]), + fontsize=7.5, color=MUTED, wrap=True) + fig.tight_layout(rect=(0, 0.05, 1, 1)) + out_png.parent.mkdir(parents=True, exist_ok=True) + fig.savefig(out_png, dpi=150) + plt.close(fig) + + +def pick_operating_point(rows: list[dict], retention: float, fpi_slack: float) -> int | None: + """Smallest size that keeps `retention` of the undegraded (112 px control) + TPI rate and does not add more than `fpi_slack` absolute FPI over it. + A stated rule, not a magic number — change the rule, not the answer.""" + control = next((r for r in rows if r["size_px"] == 112), None) + if control is None or control["n_probes"] == 0: + return None + tpi_floor = retention * control["tpi_rate"] + fpi_ceil = control["fpi_rate"] + fpi_slack + ok = [r["size_px"] for r in rows + if r["tpi_rate"] >= tpi_floor and r["fpi_rate"] <= fpi_ceil] + return min(ok) if ok else None + + +# ── Main ────────────────────────────────────────────────────────────────────── + +def main() -> int: + p = argparse.ArgumentParser( + description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + p.add_argument("--images", required=True, + help="gallery image cache root (_/NN.jpg)") + p.add_argument("--gallery", default=None, + help="gallery .h5 — restricts the actor pool to its members") + p.add_argument("--out", default=str(REPO / "experiments/results/vr005_min_face_size"), + help="output path prefix (.csv/.json/.png are appended)") + p.add_argument("--per-probe", action="store_true", + help="also write .per_probe.csv, one row per probe per size") + + p.add_argument("--actors", type=int, default=100, help="actors to sample (default 100)") + p.add_argument("--min-images", type=int, default=2, + help="minimum mugshots for an actor to be eligible (default 2)") + p.add_argument("--probes-per-actor", type=int, default=1, + help="images held out per actor; 1 is the VR-005 protocol") + p.add_argument("--seed", type=int, default=0, help="actor/probe selection seed") + p.add_argument("--keep-duplicates", action="store_true", + help="keep mugshots that are the same photograph twice; by " + "default they are dropped, since a probe identical to a " + "gallery reference is identified for free at every size") + p.add_argument("--sizes", default="12,16,20,24,32,40,48,56,64,72,80,96,112", + help="comma-separated probe sizes; 112 is the undegraded control") + + p.add_argument("--models-dir", default=str(REPO / "models")) + p.add_argument("--arcface", default=None, + help="embedder ONNX (default /LVFace-B_Glint360K.onnx)") + p.add_argument("--detector", default=None, + help="SCRFD ONNX (default /scrfd_500m_bnkps.onnx)") + p.add_argument("--providers", default="CUDAExecutionProvider,CPUExecutionProvider", + help="onnxruntime execution providers, in preference order") + p.add_argument("--batch", type=int, default=16, help="embedder batch size") + p.add_argument("--conf", type=float, default=0.5, help="detector confidence") + p.add_argument("--nms", type=float, default=0.4, help="detector NMS IoU") + p.add_argument("--max-side", type=int, default=500, + help="downscale mugshots to this longest side before detection, " + "matching the gallery builders' embedder settings") + + p.add_argument("--prob-threshold", type=float, default=0.754, + help="accept if P(match) exceeds this (Config::prob_threshold)") + p.add_argument("--match-prior", type=float, default=0.5, + help="base-rate prior (Config::match_prior)") + p.add_argument("--calib-a", type=float, default=None, + help="override the fitted sigmoid scale instead of fitting") + p.add_argument("--calib-b", type=float, default=None, + help="override the fitted sigmoid bias instead of fitting") + + p.add_argument("--down-interp", default="area", choices=sorted(INTERP), + help="interpolation for the 112 -> S downscale (default area)") + p.add_argument("--up-interp", default="linear", choices=sorted(INTERP), + help="interpolation for the S -> 112 upscale (default linear, " + "as warpAffine uses in align_face)") + + p.add_argument("--tpi-retention", type=float, default=0.95, + help="operating point keeps this fraction of the control TPI rate") + p.add_argument("--fpi-slack", type=float, default=0.01, + help="operating point may add at most this absolute FPI over control") + p.add_argument("--verify-against", default=None, + help="gallery .h5 built from --images with the same model: report " + "agreement and separation of recomputed vs stored embeddings " + "(a TensorRT-built gallery will not agree; see the docstring)") + args = p.parse_args() + + if (args.calib_a is None) != (args.calib_b is None): + return err("--calib-a and --calib-b must be given together") + + models_dir = Path(args.models_dir) + arcface = Path(args.arcface) if args.arcface else models_dir / "LVFace-B_Glint360K.onnx" + detector = Path(args.detector) if args.detector else models_dir / "scrfd_500m_bnkps.onnx" + for path, what in ((arcface, "embedder"), (detector, "detector")): + if not path.is_file(): + return err(f"{what} model not found: {path}\n" + f"Run: bash scripts/download_models.sh") + + images_root = Path(args.images) + if not images_root.is_dir(): + return err(f"image cache not found: {images_root}") + + sizes = sorted({int(s) for s in args.sizes.split(",") if s.strip()}) + if not sizes: + return err("--sizes is empty") + if args.probes_per_actor < 1: + return err("--probes-per-actor must be >= 1") + + cv2.setRNGSeed(args.seed) # estimateAffinePartial2D's RANSAC draws from this + + # ── actor pool ──────────────────────────────────────────────────────────── + pool = discover_actors(images_root) + print(f"[select] {len(pool)} actor dirs with images under {images_root}", + file=sys.stderr) + if args.gallery: + ids, names = gallery_keys(Path(args.gallery)) + pool = [a for a in pool + if (a["jellyfin_id"] and a["jellyfin_id"] in ids) + or normalise_name(a["name"]) in names] + print(f"[select] {len(pool)} of them are in {args.gallery}", file=sys.stderr) + + need = max(args.min_images, args.probes_per_actor + 1) + eligible = [a for a in pool if len(a["images"]) >= need] + print(f"[select] {len(eligible)} have >= {need} mugshots", file=sys.stderr) + if len(eligible) < 2: + return err(f"need at least 2 actors with >= {need} mugshots; found " + f"{len(eligible)}. Build the image cache first " + f"(scripts/make_jellyfin_gallery.py) or lower --min-images.") + + rng = random.Random(args.seed) + selected = sorted(rng.sample(eligible, min(args.actors, len(eligible))), + key=lambda a: a["dir"].name) + if len(selected) < args.actors: + print(f"[select] WARNING: only {len(selected)} eligible actors, " + f"--actors {args.actors} requested. FPI is gallery-size dependent — " + f"see the caveat.", file=sys.stderr) + + # ── detect + align every mugshot of the selected actors, once ───────────── + providers = resolve_providers(args.providers) + det = SCRFDDetector(str(detector), providers, args.conf, args.nms) + emb_model = Embedder(str(arcface), providers, args.batch) + print(f"[models] detector={detector.name} embedder={arcface.name} " + f"providers={emb_model.sess.get_providers()} batch={emb_model.batch}", + file=sys.stderr) + + t0 = time.time() + crops: list[np.ndarray] = [] + rows: list[dict] = [] # parallel to crops: {actor, actor_idx, image} + actors: list[dict] = [] + n_nodetect = 0 + for a in selected: + actor_crops, actor_paths = [], [] + for img_path in a["images"]: + img = cv2.imread(str(img_path)) + if img is None: + n_nodetect += 1 + continue + if args.max_side > 0 and max(img.shape[:2]) > args.max_side: + s = args.max_side / max(img.shape[:2]) + img = cv2.resize(img, None, fx=s, fy=s, interpolation=cv2.INTER_AREA) + faces = det.detect(img) + if not faces: + enhanced = enhance_for_retry(img) + faces = det.detect(enhanced) + if faces: + img = enhanced + if not faces: + n_nodetect += 1 + continue + best = max(faces, key=lambda f: f["confidence"]) + crop = align_face(img, best["landmarks"]) + if crop is None: + n_nodetect += 1 + continue + actor_crops.append(crop) + actor_paths.append(img_path) + if len(actor_crops) < args.probes_per_actor + 1: + continue + ai = len(actors) + actors.append({"name": a["name"], "jellyfin_id": a["jellyfin_id"], + "dir": a["dir"].name, "n_images": len(actor_crops)}) + for crop, img_path in zip(actor_crops, actor_paths): + rows.append({"actor_idx": ai, "image": str(img_path)}) + crops.append(crop) + if len(actors) % 20 == 0: + print(f" [align] {len(actors)}/{len(selected)} actors, " + f"{len(crops)} crops", file=sys.stderr) + + if len(actors) < 2: + return err(f"only {len(actors)} actors survived detection/alignment — " + f"nothing to match against") + print(f"[align] {len(actors)} actors, {len(crops)} aligned crops, " + f"{n_nodetect} images skipped (no face / unreadable) in " + f"{time.time() - t0:.1f}s", file=sys.stderr) + + actor_of = np.array([r["actor_idx"] for r in rows], dtype=int) + + # ── embed everything at native resolution ──────────────────────────────── + t0 = time.time() + native = emb_model.embed(crops) + print(f"[embed] {len(crops)} native crops in {time.time() - t0:.1f}s", + file=sys.stderr) + + if args.verify_against: + verify_embeddings(Path(args.verify_against), rows, native, actor_of) + + # ── drop duplicate mugshots ─────────────────────────────────────────────── + # The cache holds the same photograph twice for some actors (two provider + # URLs, one picture). A probe that is identical to a gallery reference is + # identified for free at every size, which flatters the whole curve, so + # remove duplicates the same way calibrate_gallery does. + n_dup = 0 + if not args.keep_duplicates: + keep = np.ones(len(rows), bool) + for ai in range(len(actors)): + kept: list[int] = [] + for i in np.nonzero(actor_of == ai)[0]: + if any(float(native[i] @ native[k]) > DEDUP_SIM for k in kept): + keep[i] = False + else: + kept.append(int(i)) + n_dup = int((~keep).sum()) + + # An actor left with too few distinct mugshots to hold one out drops out. + counts = np.bincount(actor_of[keep], minlength=len(actors)) + drop_actor = counts < args.probes_per_actor + 1 + keep &= ~drop_actor[actor_of] + + remap = np.full(len(actors), -1, dtype=int) + remap[~drop_actor] = np.arange(int((~drop_actor).sum())) + actors = [a for a, d in zip(actors, drop_actor) if not d] + rows = [r for r, k in zip(rows, keep) if k] + crops = [c for c, k in zip(crops, keep) if k] + native = native[keep] + actor_of = remap[actor_of[keep]] + for r, ai in zip(rows, actor_of): + r["actor_idx"] = int(ai) + for ai, a in enumerate(actors): + a["n_images"] = int(np.sum(actor_of == ai)) + print(f"[dedup] dropped {n_dup} duplicate mugshots and " + f"{int(drop_actor.sum())} actors left with too few; " + f"{len(actors)} actors, {len(rows)} images remain", file=sys.stderr) + if len(actors) < 2: + return err("fewer than 2 actors survive de-duplication — the image " + "cache holds too few distinct mugshots") + + # ── hold out the probes ─────────────────────────────────────────────────── + is_probe = np.zeros(len(rows), bool) + for ai in range(len(actors)): + idx = np.nonzero(actor_of == ai)[0] + # Seeded per actor so the choice does not depend on iteration order. + r = random.Random(f"{args.seed}:{actors[ai]['dir']}") + for pick in r.sample(list(idx), args.probes_per_actor): + is_probe[pick] = True + probe_rows = np.nonzero(is_probe)[0] + gal_rows = np.nonzero(~is_probe)[0] + print(f"[holdout] {len(probe_rows)} probes held out, " + f"{len(gal_rows)} gallery embeddings remain", file=sys.stderr) + + gal_emb = native[gal_rows] + gal_actor = actor_of[gal_rows] + probe_actor = actor_of[probe_rows] + + # Per-actor column masks for the best-of-N scan (identity_matcher_node). + actor_cols = [np.nonzero(gal_actor == ai)[0] for ai in range(len(actors))] + have_refs = np.array([len(c) > 0 for c in actor_cols]) + if not have_refs.all(): + return err("an actor ended up with no gallery references left; " + "raise --min-images") + + # ── calibration ─────────────────────────────────────────────────────────── + if args.calib_a is not None: + cal = {"a": args.calib_a, "b": args.calib_b, "valid": True} + print(f"[calibration] using supplied a={cal['a']} b={cal['b']}", file=sys.stderr) + else: + cal = calibrate_gallery(gal_emb, gal_actor) + if not cal["valid"]: + return err( + "calibration could not be fitted, and this study will not fall back to a " + "raw cosine threshold (CLAUDE.md invariant). Use more actors with >= " + f"{MIN_EMB_FOR_POSITIVE} mugshots, or pass --calib-a/--calib-b from a " + "production gallery.") + log_prior_odds = float(np.log(args.match_prior / (1.0 - args.match_prior))) + + # ── sweep ───────────────────────────────────────────────────────────────── + down, up = INTERP[args.down_interp], INTERP[args.up_interp] + probe_crops = [crops[i] for i in probe_rows] + results, per_probe = [], [] + for size in sizes: + t0 = time.time() + degraded = [degrade(c, size, down, up) for c in probe_crops] + q = emb_model.embed(degraded) + + sims = q @ gal_emb.T # [n_probe, n_gal] + best_per_actor = np.stack([sims[:, cols].max(axis=1) for cols in actor_cols], + axis=1) # [n_probe, n_actor] + best_actor = best_per_actor.argmax(axis=1) + best_sim = best_per_actor.max(axis=1) + p_match = np.asarray(probability(best_sim, cal["a"], cal["b"], log_prior_odds)) + + accept = p_match > args.prob_threshold + correct = best_actor == probe_actor + tpi = int(np.sum(accept & correct)) + fpi = int(np.sum(accept & ~correct)) + unid = int(np.sum(~accept)) + n = len(probe_rows) + + results.append({ + "size_px": size, + "n_probes": n, + "tpi": tpi, "fpi": fpi, "unidentified": unid, + "tpi_rate": tpi / n, "fpi_rate": fpi / n, "unidentified_rate": unid / n, + "rank1_rate": float(np.mean(correct)), + "mean_best_sim": float(np.mean(best_sim)), + "mean_p_match": float(np.mean(p_match)), + "mean_sim_true_actor": float(np.mean( + best_per_actor[np.arange(n), probe_actor])), + }) + if args.per_probe: + for j in range(n): + per_probe.append({ + "size_px": size, + "probe_image": rows[probe_rows[j]]["image"], + "true_actor": actors[probe_actor[j]]["name"], + "matched_actor": actors[best_actor[j]]["name"], + "best_sim": float(best_sim[j]), + "p_match": float(p_match[j]), + "outcome": ("TPI" if accept[j] and correct[j] + else "FPI" if accept[j] else "unidentified"), + }) + print(f"[sweep] {size:3d}px TPI {tpi:4d} ({100 * tpi / n:5.1f}%) " + f"FPI {fpi:4d} ({100 * fpi / n:5.1f}%) " + f"unid {unid:4d} ({100 * unid / n:5.1f}%) " + f"rank1 {100 * np.mean(correct):5.1f}% " + f"[{time.time() - t0:.1f}s]", file=sys.stderr) + + op = pick_operating_point(results, args.tpi_retention, args.fpi_slack) + + # ── outputs ─────────────────────────────────────────────────────────────── + out = Path(args.out) + out.parent.mkdir(parents=True, exist_ok=True) + # Append rather than with_suffix() so a prefix containing a dot keeps its name. + csv_path = out.with_name(out.name + ".csv") + json_path = out.with_name(out.name + ".json") + png_path = out.with_name(out.name + ".png") + fields = list(results[0].keys()) + with open(csv_path, "w", newline="") as f: + w = csv.DictWriter(f, fieldnames=fields) + w.writeheader() + w.writerows(results) + + meta = { + "requirement": "VR-005", + "caveat": CAVEAT.format(n_actors=len(actors)), + "model": arcface.stem, + "detector": detector.stem, + "backend": f"onnxruntime {'/'.join(emb_model.sess.get_providers())} (fp32 ONNX; " + f"a TensorRT fp16 build is a different embedding space)", + "n_actors": len(actors), + "n_probes": len(probe_rows), + "n_gallery_embeddings": len(gal_rows), + "probes_per_actor": args.probes_per_actor, + "seed": args.seed, + "sizes": sizes, + "prob_threshold": args.prob_threshold, + "match_prior": args.match_prior, + "calibration": cal, + "calibration_source": "supplied" if args.calib_a is not None else "fitted", + "sim_boundary_at_threshold": float( + (np.log(args.prob_threshold / (1 - args.prob_threshold)) + - cal["b"] - log_prior_odds) / cal["a"]), + "down_interp": args.down_interp, + "up_interp": args.up_interp, + "max_side": args.max_side, + "duplicate_mugshots_dropped": n_dup, + "operating_point_rule": ( + f"smallest size retaining >= {args.tpi_retention:.0%} of the 112 px " + f"control TPI rate with <= +{args.fpi_slack:.1%} absolute FPI"), + "operating_point": op, + "images_skipped_no_face": n_nodetect, + "curve": results, + "actors": actors, + } + json_path.write_text(json.dumps(meta, indent=2) + "\n") + + if per_probe: + pp = out.with_name(out.name + ".per_probe.csv") + with open(pp, "w", newline="") as f: + w = csv.DictWriter(f, fieldnames=list(per_probe[0].keys())) + w.writeheader() + w.writerows(per_probe) + print(f"[out] {pp}", file=sys.stderr) + + write_plot(results, png_path, meta) + + # ── stdout report ───────────────────────────────────────────────────────── + print(f"\nVR-005 — minimum face size, {arcface.stem}") + print(f"{len(actors)} actors, {len(probe_rows)} probes/size, " + f"{len(gal_rows)} gallery embeddings at native resolution") + print(f"identify when P>{args.prob_threshold}, i.e. cosine above " + f"{meta['sim_boundary_at_threshold']:.4f} under the calibration fitted " + f"on this gallery\n") + print(f"{'size':>5} {'TPI':>8} {'FPI':>8} {'unid':>8} {'rank1':>8} {'mean sim':>9}") + for r in results: + print(f"{r['size_px']:>5} {100 * r['tpi_rate']:>7.1f}% " + f"{100 * r['fpi_rate']:>7.1f}% {100 * r['unidentified_rate']:>7.1f}% " + f"{100 * r['rank1_rate']:>7.1f}% {r['mean_best_sim']:>9.4f}") + print(f"\noperating point: {op if op else 'none of the swept sizes qualifies'}" + f" ({meta['operating_point_rule']})") + print(f"\n{meta['caveat']}") + print(f"\n[out] {csv_path}\n[out] {json_path}\n[out] {png_path}") + return 0 + + +def _separation(emb: np.ndarray, actor: np.ndarray) -> tuple[float, float, float]: + """(mean same-actor sim, mean different-actor sim, d') — the property that has + to survive for an embedding space to be usable, whatever its coordinates.""" + iu, ju = np.triu_indices(len(actor), k=1) + sims = (emb @ emb.T)[iu, ju] + same = actor[iu] == actor[ju] + pos = sims[same & (sims < 0.9999)] # drop duplicate source images + neg = sims[~same] + if pos.size < 2 or neg.size < 2: + return float("nan"), float("nan"), float("nan") + d = (pos.mean() - neg.mean()) / np.sqrt(0.5 * (pos.var() + neg.var())) + return float(pos.mean()), float(neg.mean()), float(d) + + +def verify_embeddings(gallery_path: Path, rows: list[dict], native: np.ndarray, + actor_of: np.ndarray) -> None: + """Cross-check this script's ONNX port against a gallery built by the C++ + pipeline from the same mugshots. + + Agreement is ~1.0 only if that gallery was built with the same backend. A + TensorRT fp16 build lands around 0.85 on LVFace-B while separating just as + well, so the separation figures — not the agreement — are what says whether + the port is sound.""" + from sae_gallery import load_gallery_hdf5 + g = load_gallery_hdf5(gallery_path) + stored: dict[tuple[str, str], np.ndarray] = {} + for a in g["actors"]: + key = a.get("jellyfin_id") or normalise_name(a.get("name", "")) + for e, src in zip(a.get("embeddings", []), a.get("source_images", [])): + if src: + stored[(key, src)] = np.asarray(e, np.float32) + + sims, paired_mine, paired_ref, paired_actor = [], [], [], [] + for i, r in enumerate(rows): + path = Path(r["image"]) + head, _, _ = path.parent.name.partition("_") + key = head if JELLYFIN_ID_RE.match(head) else normalise_name( + path.parent.name.replace("_", " ")) + ref = stored.get((key, path.name)) + if ref is None or ref.shape != native[i].shape: + continue + ref = ref / max(float(np.linalg.norm(ref)), 1e-6) + sims.append(float(native[i] @ ref)) + paired_mine.append(native[i]) + paired_ref.append(ref) + paired_actor.append(actor_of[i]) + if not sims: + print(f"[verify] no overlap with {gallery_path} — nothing checked", + file=sys.stderr) + return + + sims_arr = np.asarray(sims) + print(f"[verify] {len(sims)} embeddings vs {gallery_path.name}: " + f"mean cos={sims_arr.mean():.4f} min={sims_arr.min():.4f}", + file=sys.stderr) + act = np.asarray(paired_actor) + for label, mat in (("this script", np.asarray(paired_mine)), + ("stored gallery", np.asarray(paired_ref))): + pos, neg, d = _separation(mat, act) + print(f"[verify] {label:>14s}: same-actor {pos:.3f} " + f"different-actor {neg:.3f} d'={d:.2f}", file=sys.stderr) + if sims_arr.mean() < 0.99: + print("[verify] embeddings differ from the stored gallery. If d' is " + "comparable this is a backend difference (e.g. a TensorRT fp16 " + "build), not a broken port; the study is self-consistent either " + "way. If d' collapsed, the port is wrong.", file=sys.stderr) + + +def err(msg: str) -> int: + print(f"error: {msg}", file=sys.stderr) + return 2 + + +if __name__ == "__main__": + sys.exit(main())