dtourolleandClaude Opus 5
042e424961
Holds out one mugshot per actor, degrades that probe to each candidate
face size and matches it against a gallery held at native resolution,
reporting TPI/FPI per size. Replaces AR-002's 66x66 px working estimate
with a measurement. Needs no video and no ground truth beyond the
mugshot cache already on disk.
LVFace-B over 258 actors, 999 gallery embeddings, threshold 0.754:
px 12 16 20 24 32 40 48+
TPI 6.6% 46.5% 81.8% 93.4% 98.1% 99.2% 99.2%
FPI is 0.000 at every size — a face too small to identify degrades to
unidentified, never to a wrong name. rank-1 holds at >=99.6% from 24 px
up, so what fails first is the calibrated probability crossing
threshold, not the ranking.
Two limits on reading this. FPI grows with the number of actors
competing, so 258 understates it against a production library. And
detection and alignment run on the native image with only the resulting
112x112 crop degraded, so landmark error at small face sizes is excluded
by construction and the curve is an upper bound — VR-010 measures the
same question end to end, and lands well above these numbers.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
TRACES: VR-005 | AR-002
1016 lines
48 KiB
Python
1016 lines
48 KiB
Python
#!/usr/bin/env python3
|
|
"""
|
|
min_face_size.py — VR-005: at what face size do embeddings stop identifying people?
|
|
|
|
TRACES: VR-005
|
|
|
|
`min_face_px` is currently a working estimate (AR-002: 66x66 px in original video
|
|
resolution). This script replaces the guess with a measurement, using only gallery
|
|
mugshots already on disk — no video, no C++ changes.
|
|
|
|
Protocol
|
|
--------
|
|
1. Select ~100 gallery actors that have more than one mugshot.
|
|
2. Per actor hold out ONE image as the *probe*; that actor's remaining images stay
|
|
in the gallery at native resolution.
|
|
3. For each target size S, take the probe's native aligned 112x112 crop, downscale
|
|
it to SxS and upscale it back to 112x112, then embed. Detail is genuinely
|
|
destroyed and then the same warp the pipeline applies is re-applied on top —
|
|
which is what a face detected at SxS in a frame actually suffers.
|
|
4. Match each degraded probe against the whole gallery.
|
|
5. Record, per size, TPI (identified as the correct actor) and FPI (identified as
|
|
someone else). Everything else is an unidentified probe (TBI).
|
|
|
|
The asymmetry is the point: **the gallery stays at native resolution and only the
|
|
probe degrades.** That is the production case — reference mugshots are clean, the
|
|
face coming out of the video is small. Degrading both sides would measure
|
|
something the pipeline never does.
|
|
|
|
What this deliberately does NOT measure
|
|
---------------------------------------
|
|
The cosine between the size-S embedding and the native embedding of the *same*
|
|
image. That is embedding *drift*, and it answers the wrong question: an embedding
|
|
can drift a long way and stay perfectly separable, or drift a little in a
|
|
direction that destroys separation. What matters is the decision the pipeline
|
|
makes — probe against a competing gallery — so that is what is recorded.
|
|
|
|
CAVEAT — FPI IS RELATIVE, NOT ABSOLUTE
|
|
--------------------------------------
|
|
False positives grow with the number of actors competing for the match. A
|
|
~100-actor gallery therefore *understates* the false-positive rate against a
|
|
production library of thousands. Read the FPI column as a relative curve across
|
|
sizes ("FPI is 4x worse at 32 px than at 64 px"), never as the rate you would see
|
|
in production. Re-run with `--actors` at production scale before setting a
|
|
threshold from an absolute FPI number.
|
|
|
|
Decision rule
|
|
-------------
|
|
Per the repo invariant (CLAUDE.md: "always use the calibrated probability, never a
|
|
raw cosine"), identification goes through the same path as `identity_matcher_node`:
|
|
per-actor best-of-N cosine -> Platt sigmoid P(match) = sigma(a*sim + b + log-prior)
|
|
-> accept if P > `prob_threshold`. The sigmoid is fitted here by the same
|
|
histogram/gradient-descent procedure as `src/gallery/gallery_calibration.hpp`,
|
|
over the native gallery embeddings only (held-out probes are excluded, so the
|
|
calibration cannot see the images it will be scored on).
|
|
|
|
Why an ONNX Runtime pipeline instead of the `sae_embed` module
|
|
--------------------------------------------------------------
|
|
`sae_embed.FaceEmbedder` only exposes `embed(path)` — detect, align and embed in
|
|
one step — so it cannot embed a crop the caller has degraded.
|
|
|
|
SUPERSEDED: sae_embed now binds the production stages directly —
|
|
detect(), align_face(), embed_crop() and GalleryCalibration — so the ports
|
|
below can be deleted and this driven off the shipped C++ instead. Do that
|
|
before extending them: a second implementation of the calibration is
|
|
exactly where the "always the calibrated probability, never a raw cosine"
|
|
rule gets broken silently. This script
|
|
therefore drives `scrfd_500m_bnkps.onnx` and the embedder ONNX directly, porting
|
|
`SCRFDDecoder` / `ArcFaceEmbedder` (src/backends/ort_backend.cpp), `align_face` /
|
|
`enhance_for_retry` (src/face_utils.hpp) and `calibrate_gallery`
|
|
(src/gallery/gallery_calibration.hpp).
|
|
|
|
That means the ONNX-Runtime fp32 backend — the reference one
|
|
(`SAE_INFERENCE_BACKEND=ORT`), which loads the .onnx directly. A TensorRT fp16
|
|
build is a *different realisation* of the same model and its embeddings are
|
|
measurably not the same vectors: on LVFace-B_Glint360K the stored TRT-fp16 gallery
|
|
agrees with an fp32 recompute of the same mugshot at only ~0.85 cosine, while
|
|
same-actor/different-actor separation is essentially unchanged (d' 5.3 vs 5.7).
|
|
Nothing here is invalidated by that — gallery and probes go through one session,
|
|
so the comparison is internally consistent — but the two embedding spaces are not
|
|
interchangeable, and `--verify-against <gallery.h5>` will show ~0.85, not ~1.0,
|
|
against a TRT-built gallery. It reports the separation of both sets alongside the
|
|
agreement so the two causes are distinguishable: a broken port collapses
|
|
separation, a different backend does not.
|
|
|
|
Secondary output (VR-005): running the sweep per `--arcface` model shows whether
|
|
`min_face_px` should be one constant at all, or should scale with the embedder —
|
|
which matters because the model is a build-time choice (GR-004).
|
|
|
|
Usage
|
|
-----
|
|
python scripts/validation/min_face_size.py \
|
|
--images images \
|
|
--gallery gallery_lvface.h5 \
|
|
--arcface models/LVFace-B_Glint360K.onnx \
|
|
--actors 100 --seed 0 \
|
|
--out experiments/results/vr005_min_face_size
|
|
|
|
Writes <out>.csv, <out>.json and <out>.png (plus <out>.per_probe.csv with
|
|
--per-probe).
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import argparse
|
|
import csv
|
|
import json
|
|
import random
|
|
import re
|
|
import sys
|
|
import time
|
|
from pathlib import Path
|
|
|
|
import cv2
|
|
import numpy as np
|
|
|
|
REPO = Path(__file__).resolve().parent.parent.parent
|
|
sys.path.insert(0, str(REPO / "scripts"))
|
|
|
|
# ArcFace 5-point reference landmarks in the 112x112 aligned frame.
|
|
# Mirrors kArcFaceRef in src/types.hpp.
|
|
ARCFACE_REF = np.array([
|
|
[38.2946, 51.6963],
|
|
[73.5318, 51.5014],
|
|
[56.0252, 71.7366],
|
|
[41.5493, 92.3655],
|
|
[70.7299, 92.2041],
|
|
], dtype=np.float32)
|
|
|
|
IMAGE_EXTS = (".jpg", ".jpeg", ".png", ".webp")
|
|
JELLYFIN_ID_RE = re.compile(r"^[0-9a-f]{32}$")
|
|
|
|
# Interpolation used for the two halves of the degradation. Downscaling uses
|
|
# INTER_AREA (correct low-pass for shrinking, i.e. detail that a small detection
|
|
# genuinely never had); upscaling uses INTER_LINEAR, which is what warpAffine in
|
|
# align_face() uses when it blows a small detection up to 112x112.
|
|
INTERP = {
|
|
"area": cv2.INTER_AREA,
|
|
"linear": cv2.INTER_LINEAR,
|
|
"cubic": cv2.INTER_CUBIC,
|
|
"nearest": cv2.INTER_NEAREST,
|
|
"lanczos": cv2.INTER_LANCZOS4,
|
|
}
|
|
|
|
# House chart palette, shared with scripts/docs/experiment_charts.py so figures
|
|
# across the report read as one set.
|
|
INK, MUTED, GRID, SURFACE = "#0b0b0b", "#898781", "#e1e0d9", "#fcfcfb"
|
|
BLUE, GREEN, RED, AMBER = "#2a78d6", "#008300", "#e34948", "#eda100"
|
|
|
|
|
|
# ── SCRFD detector (port of SCRFDDecoder, src/backends/ort_backend.cpp) ────────
|
|
|
|
class SCRFDDetector:
|
|
"""SCRFD-with-keypoints decoder: letterbox to 640x640, decode strides 8/16/32
|
|
(/64 for a 12-output model) at 2 anchors each, then NMS."""
|
|
|
|
INPUT_W = 640
|
|
INPUT_H = 640
|
|
STRIDES = (8, 16, 32, 64)
|
|
ANCHORS = 2
|
|
|
|
def __init__(self, model_path: str, providers: list[str],
|
|
conf: float = 0.5, nms: float = 0.4):
|
|
import onnxruntime as ort
|
|
opts = ort.SessionOptions()
|
|
opts.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL
|
|
self.sess = ort.InferenceSession(model_path, opts, providers=providers)
|
|
self.input_name = self.sess.get_inputs()[0].name
|
|
self.out_names = [o.name for o in self.sess.get_outputs()]
|
|
n_out = len(self.out_names)
|
|
if n_out % 3 != 0 or not (9 <= n_out <= 12):
|
|
raise SystemExit(
|
|
f"[scrfd] expected 9 or 12 outputs (kps-variant model), got {n_out}: "
|
|
f"{model_path}")
|
|
self.fmc = n_out // 3
|
|
# Reject non-SCRFD models with the same output count (e.g. YuNet).
|
|
for gi, last in enumerate((1, 4, 10)):
|
|
for si in range(self.fmc):
|
|
shape = self.sess.get_outputs()[gi * self.fmc + si].shape
|
|
if not shape or shape[-1] != last:
|
|
raise SystemExit(
|
|
f"[scrfd] {model_path} does not look like InsightFace SCRFD: "
|
|
f"output '{self.out_names[gi * self.fmc + si]}' last-dim is "
|
|
f"{shape[-1] if shape else None}, expected {last}. "
|
|
f"Hint: pass scrfd_500m_bnkps.onnx, not yunet/*.onnx.")
|
|
self.conf = conf
|
|
self.nms = nms
|
|
|
|
def detect(self, img: np.ndarray) -> list[dict]:
|
|
h, w = img.shape[:2]
|
|
scale = min(self.INPUT_W / w, self.INPUT_H / h)
|
|
new_w, new_h = int(round(w * scale)), int(round(h * scale))
|
|
pad_x, pad_y = (self.INPUT_W - new_w) // 2, (self.INPUT_H - new_h) // 2
|
|
|
|
resized = cv2.resize(img, (new_w, new_h), interpolation=cv2.INTER_LINEAR)
|
|
letterboxed = np.full((self.INPUT_H, self.INPUT_W, 3), 114, dtype=img.dtype)
|
|
letterboxed[pad_y:pad_y + new_h, pad_x:pad_x + new_w] = resized
|
|
|
|
blob = cv2.dnn.blobFromImage(letterboxed, 1.0 / 128.0,
|
|
(self.INPUT_W, self.INPUT_H),
|
|
(127.5, 127.5, 127.5), swapRB=True, crop=False)
|
|
outs = self.sess.run(self.out_names, {self.input_name: blob})
|
|
|
|
boxes, scores, kpss = [], [], []
|
|
for si in range(self.fmc):
|
|
stride = self.STRIDES[si]
|
|
fw = self.INPUT_W // stride
|
|
s = np.asarray(outs[si]).reshape(-1)
|
|
b = np.asarray(outs[self.fmc + si]).reshape(-1, 4)
|
|
k = np.asarray(outs[self.fmc * 2 + si]).reshape(-1, 10)
|
|
|
|
keep = np.nonzero(s >= self.conf)[0]
|
|
if keep.size == 0:
|
|
continue
|
|
# idx = (r * fw + c) * ANCHORS + a -> anchor centres
|
|
cell = keep // self.ANCHORS
|
|
cx = (cell % fw).astype(np.float32) * stride
|
|
cy = (cell // fw).astype(np.float32) * stride
|
|
|
|
x1 = (cx - b[keep, 0] * stride - pad_x) / scale
|
|
y1 = (cy - b[keep, 1] * stride - pad_y) / scale
|
|
x2 = (cx + b[keep, 2] * stride - pad_x) / scale
|
|
y2 = (cy + b[keep, 3] * stride - pad_y) / scale
|
|
|
|
kp = k[keep].reshape(-1, 5, 2) * stride
|
|
kp[:, :, 0] = (kp[:, :, 0] + cx[:, None] - pad_x) / scale
|
|
kp[:, :, 1] = (kp[:, :, 1] + cy[:, None] - pad_y) / scale
|
|
|
|
boxes.append(np.stack([x1, y1, x2 - x1, y2 - y1], axis=1))
|
|
scores.append(s[keep])
|
|
kpss.append(kp)
|
|
|
|
if not boxes:
|
|
return []
|
|
boxes = np.concatenate(boxes).astype(np.float64)
|
|
scores = np.concatenate(scores).astype(np.float32)
|
|
kpss = np.concatenate(kpss).astype(np.float32)
|
|
|
|
keep = cv2.dnn.NMSBoxes(boxes.tolist(), scores.tolist(), self.conf, self.nms)
|
|
if keep is None or len(keep) == 0:
|
|
return []
|
|
keep = np.asarray(keep).reshape(-1)
|
|
|
|
faces = []
|
|
for i in keep:
|
|
x = max(0.0, float(boxes[i, 0]))
|
|
y = max(0.0, float(boxes[i, 1]))
|
|
faces.append({
|
|
"bbox": (x, y,
|
|
min(float(boxes[i, 2]), w - x),
|
|
min(float(boxes[i, 3]), h - y)),
|
|
"confidence": float(scores[i]),
|
|
"landmarks": kpss[i].copy(),
|
|
})
|
|
return faces
|
|
|
|
|
|
# ── Alignment (port of src/face_utils.hpp) ────────────────────────────────────
|
|
|
|
def align_face(img: np.ndarray, landmarks: np.ndarray) -> np.ndarray | None:
|
|
"""112x112 BGR crop via the ArcFace 5-point similarity transform."""
|
|
M, _ = cv2.estimateAffinePartial2D(landmarks.astype(np.float32), ARCFACE_REF,
|
|
method=cv2.RANSAC, ransacReprojThreshold=3.0)
|
|
if M is None:
|
|
return None
|
|
return cv2.warpAffine(img, M, (112, 112), flags=cv2.INTER_LINEAR,
|
|
borderMode=cv2.BORDER_CONSTANT, borderValue=(0, 0, 0))
|
|
|
|
|
|
def enhance_for_retry(img: np.ndarray) -> np.ndarray:
|
|
"""Border-replicate pad by 50% and CLAHE the luminance, so a detector that
|
|
found nothing gets a second try. Same as src/face_utils.hpp."""
|
|
pad_x, pad_y = img.shape[1] // 4, img.shape[0] // 4
|
|
padded = cv2.copyMakeBorder(img, pad_y, pad_y, pad_x, pad_x, cv2.BORDER_REPLICATE)
|
|
lab = cv2.cvtColor(padded, cv2.COLOR_BGR2Lab)
|
|
l, a, b = cv2.split(lab)
|
|
l = cv2.createCLAHE(clipLimit=2.0, tileGridSize=(8, 8)).apply(l)
|
|
return cv2.cvtColor(cv2.merge([l, a, b]), cv2.COLOR_Lab2BGR)
|
|
|
|
|
|
# ── Embedder (port of ArcFaceEmbedder, src/backends/ort_backend.cpp) ──────────
|
|
|
|
class Embedder:
|
|
"""112x112 BGR crops -> L2-normalised 512-d embeddings.
|
|
Input is BGR->RGB, scaled to [-1, 1] as (px - 127.5) / 128."""
|
|
|
|
def __init__(self, model_path: str, providers: list[str], batch: int = 16):
|
|
import onnxruntime as ort
|
|
opts = ort.SessionOptions()
|
|
opts.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL
|
|
self.sess = ort.InferenceSession(model_path, opts, providers=providers)
|
|
self.input_name = self.sess.get_inputs()[0].name
|
|
self.output_name = self.sess.get_outputs()[0].name
|
|
self.fp16 = "float16" in self.sess.get_inputs()[0].type
|
|
self.batch = max(1, self._probe_batch(batch))
|
|
|
|
def _probe_batch(self, batch: int) -> int:
|
|
"""Some exports pin the batch dimension. Try a small batch once and fall
|
|
back to 1 rather than failing halfway through the sweep."""
|
|
if batch <= 1:
|
|
return 1
|
|
n = min(batch, 4)
|
|
dummy = np.zeros((n, 3, 112, 112), np.float16 if self.fp16 else np.float32)
|
|
try:
|
|
out = self.sess.run([self.output_name], {self.input_name: dummy})[0]
|
|
except Exception as e: # noqa: BLE001 — any ORT shape/type rejection
|
|
print(f"[embed] batching unsupported ({e}); falling back to batch=1",
|
|
file=sys.stderr)
|
|
return 1
|
|
if out.shape[0] != n:
|
|
print(f"[embed] model returned {out.shape[0]} rows for a batch of {n}; "
|
|
f"falling back to batch=1", file=sys.stderr)
|
|
return 1
|
|
return batch
|
|
|
|
def embed(self, crops: list[np.ndarray]) -> np.ndarray:
|
|
if not crops:
|
|
return np.zeros((0, 512), np.float32)
|
|
out = np.empty((len(crops), 512), np.float32)
|
|
for i in range(0, len(crops), self.batch):
|
|
chunk = crops[i:i + self.batch]
|
|
rgbs = [cv2.cvtColor(c, cv2.COLOR_BGR2RGB) for c in chunk]
|
|
blob = cv2.dnn.blobFromImages(rgbs, 1.0 / 128.0, (112, 112),
|
|
(127.5, 127.5, 127.5),
|
|
swapRB=False, crop=False)
|
|
if self.fp16:
|
|
blob = blob.astype(np.float16)
|
|
raw = self.sess.run([self.output_name], {self.input_name: blob})[0]
|
|
out[i:i + len(chunk)] = np.asarray(raw, dtype=np.float32)
|
|
norms = np.linalg.norm(out, axis=1, keepdims=True)
|
|
return out / np.maximum(norms, 1e-6)
|
|
|
|
|
|
# ── Calibration (port of src/gallery/gallery_calibration.hpp) ─────────────────
|
|
|
|
MIN_EMB_FOR_POSITIVE = 5
|
|
DEDUP_SIM = 1.0 - 1e-7
|
|
HIST_BINS = 200
|
|
|
|
|
|
def _sigmoid(z: np.ndarray | float) -> np.ndarray | float:
|
|
return np.where(z >= 0, 1.0 / (1.0 + np.exp(-np.abs(z))),
|
|
np.exp(-np.abs(z)) / (1.0 + np.exp(-np.abs(z))))
|
|
|
|
|
|
def calibrate_gallery(emb: np.ndarray, actor: np.ndarray) -> dict:
|
|
"""Fit P(match) = sigma(a*sim + b) from intra-class (same actor, different
|
|
reference image) and inter-class pairs, exactly as calibrate_gallery() does:
|
|
per-actor dedup, actors with < 5 distinct embeddings contribute negatives
|
|
only, similarities bucketed into 200 bins, class-weighted gradient descent."""
|
|
n_actors = int(actor.max()) + 1 if actor.size else 0
|
|
|
|
keep_rows, eligible = [], np.zeros(n_actors, bool)
|
|
for ai in range(n_actors):
|
|
rows = np.nonzero(actor == ai)[0]
|
|
kept: list[int] = []
|
|
for r in rows:
|
|
if all(float(emb[r] @ emb[k]) <= DEDUP_SIM for k in kept):
|
|
kept.append(int(r))
|
|
eligible[ai] = len(kept) >= MIN_EMB_FOR_POSITIVE
|
|
keep_rows.extend(kept)
|
|
|
|
keep_rows = np.asarray(sorted(keep_rows), dtype=int)
|
|
e, a_idx = emb[keep_rows], actor[keep_rows]
|
|
n = len(keep_rows)
|
|
print(f"[calibration] dedup: {len(emb)} -> {n} embeddings "
|
|
f"({int(eligible.sum())}/{n_actors} actors have >= {MIN_EMB_FOR_POSITIVE} "
|
|
f"distinct embeddings, eligible for positive pairs)", file=sys.stderr)
|
|
if n < 2:
|
|
return {"a": 10.0, "b": -5.0, "valid": False}
|
|
|
|
S = e @ e.T
|
|
iu, ju = np.triu_indices(n, k=1)
|
|
sims = S[iu, ju]
|
|
same = a_idx[iu] == a_idx[ju]
|
|
pos_mask = same & eligible[a_idx[iu]]
|
|
neg_mask = ~same
|
|
|
|
bw = 2.0 / HIST_BINS
|
|
bin_idx = np.clip(((sims + 1.0) / bw).astype(int), 0, HIST_BINS - 1)
|
|
pos = np.bincount(bin_idx[pos_mask], minlength=HIST_BINS).astype(np.float64)
|
|
neg = np.bincount(bin_idx[neg_mask], minlength=HIST_BINS).astype(np.float64)
|
|
|
|
n_pos, n_neg = pos.sum(), neg.sum()
|
|
if n_pos < 2 or n_neg < 1:
|
|
print(f"[calibration] insufficient pairs (+{n_pos:.0f}/-{n_neg:.0f}) — "
|
|
f"calibration skipped", file=sys.stderr)
|
|
return {"a": 10.0, "b": -5.0, "valid": False}
|
|
|
|
total = n_pos + n_neg
|
|
w_pos, w_neg = total / (2.0 * n_pos), total / (2.0 * n_neg)
|
|
centers = -1.0 + (np.arange(HIST_BINS) + 0.5) * bw
|
|
|
|
a, b = 10.0, -5.0
|
|
lr, max_iter, tol = 0.05, 20000, 1e-7
|
|
for _ in range(max_iter):
|
|
sig = _sigmoid(a * centers + b)
|
|
err = (sig - 1.0) * w_pos * pos + sig * w_neg * neg
|
|
da = float((err * centers).sum()) / total
|
|
db = float(err.sum()) / total
|
|
a -= lr * da
|
|
b -= lr * db
|
|
if da * da + db * db < tol * tol:
|
|
break
|
|
|
|
sig = _sigmoid(a * centers + b)
|
|
correct = float(np.where(sig > 0.5, pos, neg).sum())
|
|
boundary = (0.0 - b) / a
|
|
print(f"[calibration] sigmoid fitted: a={a:.4f} b={b:.4f} "
|
|
f"boundary(P=0.5)=sim{boundary:.4f} pairs={int(total)} "
|
|
f"(+{int(n_pos)}/-{int(n_neg)}) bins={HIST_BINS} "
|
|
f"train_acc={100.0 * correct / total:.2f}%", file=sys.stderr)
|
|
return {"a": float(a), "b": float(b), "valid": True}
|
|
|
|
|
|
def probability(sim, a: float, b: float, log_prior_odds: float = 0.0):
|
|
return _sigmoid(a * np.asarray(sim, dtype=np.float64) + b + log_prior_odds)
|
|
|
|
|
|
# ── Runtime / actor discovery ─────────────────────────────────────────────────
|
|
|
|
def resolve_providers(requested: str) -> list[str]:
|
|
"""Keep only providers this onnxruntime build actually has — asking for an
|
|
absent one is a hard error in recent versions, and CUDA is routinely absent."""
|
|
import onnxruntime as ort
|
|
available = ort.get_available_providers()
|
|
keep = [p for p in (s.strip() for s in requested.split(",")) if p in available]
|
|
dropped = [p for p in (s.strip() for s in requested.split(",")) if p not in available]
|
|
if dropped:
|
|
print(f"[models] providers unavailable, skipping: {', '.join(dropped)} "
|
|
f"(have: {', '.join(available)})", file=sys.stderr)
|
|
return keep or ["CPUExecutionProvider"]
|
|
|
|
|
|
def normalise_name(name: str) -> str:
|
|
return re.sub(r"[^a-z0-9]+", "", name.lower())
|
|
|
|
|
|
def discover_actors(images_root: Path) -> list[dict]:
|
|
"""Enumerate the gallery-build image cache: <root>/<jellyfin_id>_<Name>/NN.jpg
|
|
(the layout make_jellyfin_gallery.py / reembed_gallery.py use)."""
|
|
actors = []
|
|
for d in sorted(p for p in images_root.iterdir() if p.is_dir()):
|
|
imgs = sorted(p for p in d.iterdir()
|
|
if p.is_file() and p.suffix.lower() in IMAGE_EXTS)
|
|
if not imgs:
|
|
continue
|
|
head, _, tail = d.name.partition("_")
|
|
if JELLYFIN_ID_RE.match(head) and tail:
|
|
jellyfin_id, name = head, tail.replace("_", " ")
|
|
else:
|
|
jellyfin_id, name = "", d.name.replace("_", " ")
|
|
actors.append({"dir": d, "jellyfin_id": jellyfin_id, "name": name,
|
|
"images": imgs})
|
|
return actors
|
|
|
|
|
|
def gallery_keys(gallery_path: Path) -> tuple[set[str], set[str]]:
|
|
"""(jellyfin ids, normalised names) of the actors an existing gallery holds."""
|
|
from sae_gallery import load_gallery_hdf5
|
|
g = load_gallery_hdf5(gallery_path)
|
|
ids = {a.get("jellyfin_id", "") for a in g["actors"] if a.get("jellyfin_id")}
|
|
names = {normalise_name(a.get("name", "")) for a in g["actors"] if a.get("name")}
|
|
return ids, names
|
|
|
|
|
|
# ── Degradation ───────────────────────────────────────────────────────────────
|
|
|
|
def degrade(crop: np.ndarray, size: int, down: int, up: int) -> np.ndarray:
|
|
"""Throw away everything a face detected at size x size never had, then warp
|
|
it back up to the 112x112 the embedder is fed."""
|
|
if size == 112:
|
|
return crop
|
|
small = cv2.resize(crop, (size, size), interpolation=down)
|
|
return cv2.resize(small, (112, 112), interpolation=up)
|
|
|
|
|
|
# ── Reporting ─────────────────────────────────────────────────────────────────
|
|
|
|
CAVEAT = (
|
|
"CAVEAT: FPI grows with gallery size. This ran against {n_actors} actors, so it "
|
|
"UNDERSTATES the false-positive rate of a production library of thousands. Read "
|
|
"FPI as relative across sizes, not as an absolute rate."
|
|
)
|
|
|
|
|
|
def write_plot(rows: list[dict], out_png: Path, meta: dict) -> None:
|
|
import matplotlib
|
|
matplotlib.use("Agg")
|
|
import matplotlib.pyplot as plt
|
|
|
|
plt.rcParams.update({
|
|
"figure.facecolor": SURFACE, "axes.facecolor": SURFACE,
|
|
"savefig.facecolor": SURFACE, "text.color": INK,
|
|
"axes.edgecolor": MUTED, "axes.labelcolor": INK,
|
|
"xtick.color": MUTED, "ytick.color": MUTED,
|
|
"axes.grid": True, "grid.color": GRID, "grid.linewidth": 0.8,
|
|
"axes.spines.top": False, "axes.spines.right": False,
|
|
})
|
|
|
|
sizes = [r["size_px"] for r in rows]
|
|
fig, ax = plt.subplots(figsize=(9, 5.6))
|
|
ax.plot(sizes, [100 * r["tpi_rate"] for r in rows], "-o", color=GREEN,
|
|
lw=2, label="TPI — identified, correct actor")
|
|
ax.plot(sizes, [100 * r["fpi_rate"] for r in rows], "-s", color=RED,
|
|
lw=2, label="FPI — identified, wrong actor")
|
|
ax.plot(sizes, [100 * r["unidentified_rate"] for r in rows], color=MUTED,
|
|
marker="^", lw=1.4, ls="--", label="unidentified (below P threshold)")
|
|
ax.plot(sizes, [100 * r["rank1_rate"] for r in rows], ":", color=BLUE,
|
|
lw=1.6, label="rank-1 correct (ignoring threshold)")
|
|
|
|
op = meta.get("operating_point")
|
|
if op:
|
|
ax.axvline(op, color=AMBER, lw=1.6, ls="-.", zorder=1)
|
|
ax.annotate(f"operating point {op} px", xy=(op, 50),
|
|
xytext=(4, 0), textcoords="offset points",
|
|
color=AMBER, fontsize=9, rotation=90, va="center")
|
|
|
|
ax.set_xlabel("probe face size before upscaling (px)")
|
|
ax.set_ylabel("% of probes")
|
|
ax.set_ylim(-2, 102)
|
|
ax.set_xticks(sizes)
|
|
ax.set_title(f"VR-005 — identification vs. probe face size\n"
|
|
f"{meta['model']}, {meta['n_actors']} actors, "
|
|
f"{meta['n_probes']} probes/size, gallery at native resolution",
|
|
fontsize=11, loc="left")
|
|
ax.legend(frameon=False, fontsize=9, loc="center left")
|
|
fig.text(0.01, 0.005, CAVEAT.format(n_actors=meta["n_actors"]),
|
|
fontsize=7.5, color=MUTED, wrap=True)
|
|
fig.tight_layout(rect=(0, 0.05, 1, 1))
|
|
out_png.parent.mkdir(parents=True, exist_ok=True)
|
|
fig.savefig(out_png, dpi=150)
|
|
plt.close(fig)
|
|
|
|
|
|
def pick_operating_point(rows: list[dict], retention: float, fpi_slack: float) -> int | None:
|
|
"""Smallest size that keeps `retention` of the undegraded (112 px control)
|
|
TPI rate and does not add more than `fpi_slack` absolute FPI over it.
|
|
A stated rule, not a magic number — change the rule, not the answer."""
|
|
control = next((r for r in rows if r["size_px"] == 112), None)
|
|
if control is None or control["n_probes"] == 0:
|
|
return None
|
|
tpi_floor = retention * control["tpi_rate"]
|
|
fpi_ceil = control["fpi_rate"] + fpi_slack
|
|
ok = [r["size_px"] for r in rows
|
|
if r["tpi_rate"] >= tpi_floor and r["fpi_rate"] <= fpi_ceil]
|
|
return min(ok) if ok else None
|
|
|
|
|
|
# ── Main ──────────────────────────────────────────────────────────────────────
|
|
|
|
def main() -> int:
|
|
p = argparse.ArgumentParser(
|
|
description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
|
p.add_argument("--images", required=True,
|
|
help="gallery image cache root (<jellyfin_id>_<Name>/NN.jpg)")
|
|
p.add_argument("--gallery", default=None,
|
|
help="gallery .h5 — restricts the actor pool to its members")
|
|
p.add_argument("--out", default=str(REPO / "experiments/results/vr005_min_face_size"),
|
|
help="output path prefix (.csv/.json/.png are appended)")
|
|
p.add_argument("--per-probe", action="store_true",
|
|
help="also write <out>.per_probe.csv, one row per probe per size")
|
|
|
|
p.add_argument("--actors", type=int, default=100, help="actors to sample (default 100)")
|
|
p.add_argument("--min-images", type=int, default=2,
|
|
help="minimum mugshots for an actor to be eligible (default 2)")
|
|
p.add_argument("--probes-per-actor", type=int, default=1,
|
|
help="images held out per actor; 1 is the VR-005 protocol")
|
|
p.add_argument("--seed", type=int, default=0, help="actor/probe selection seed")
|
|
p.add_argument("--keep-duplicates", action="store_true",
|
|
help="keep mugshots that are the same photograph twice; by "
|
|
"default they are dropped, since a probe identical to a "
|
|
"gallery reference is identified for free at every size")
|
|
p.add_argument("--sizes", default="12,16,20,24,32,40,48,56,64,72,80,96,112",
|
|
help="comma-separated probe sizes; 112 is the undegraded control")
|
|
|
|
p.add_argument("--models-dir", default=str(REPO / "models"))
|
|
p.add_argument("--arcface", default=None,
|
|
help="embedder ONNX (default <models-dir>/LVFace-B_Glint360K.onnx)")
|
|
p.add_argument("--detector", default=None,
|
|
help="SCRFD ONNX (default <models-dir>/scrfd_500m_bnkps.onnx)")
|
|
p.add_argument("--providers", default="CUDAExecutionProvider,CPUExecutionProvider",
|
|
help="onnxruntime execution providers, in preference order")
|
|
p.add_argument("--batch", type=int, default=16, help="embedder batch size")
|
|
p.add_argument("--conf", type=float, default=0.5, help="detector confidence")
|
|
p.add_argument("--nms", type=float, default=0.4, help="detector NMS IoU")
|
|
p.add_argument("--max-side", type=int, default=500,
|
|
help="downscale mugshots to this longest side before detection, "
|
|
"matching the gallery builders' embedder settings")
|
|
|
|
p.add_argument("--prob-threshold", type=float, default=0.754,
|
|
help="accept if P(match) exceeds this (Config::prob_threshold)")
|
|
p.add_argument("--match-prior", type=float, default=0.5,
|
|
help="base-rate prior (Config::match_prior)")
|
|
p.add_argument("--calib-a", type=float, default=None,
|
|
help="override the fitted sigmoid scale instead of fitting")
|
|
p.add_argument("--calib-b", type=float, default=None,
|
|
help="override the fitted sigmoid bias instead of fitting")
|
|
|
|
p.add_argument("--down-interp", default="area", choices=sorted(INTERP),
|
|
help="interpolation for the 112 -> S downscale (default area)")
|
|
p.add_argument("--up-interp", default="linear", choices=sorted(INTERP),
|
|
help="interpolation for the S -> 112 upscale (default linear, "
|
|
"as warpAffine uses in align_face)")
|
|
|
|
p.add_argument("--tpi-retention", type=float, default=0.95,
|
|
help="operating point keeps this fraction of the control TPI rate")
|
|
p.add_argument("--fpi-slack", type=float, default=0.01,
|
|
help="operating point may add at most this absolute FPI over control")
|
|
p.add_argument("--verify-against", default=None,
|
|
help="gallery .h5 built from --images with the same model: report "
|
|
"agreement and separation of recomputed vs stored embeddings "
|
|
"(a TensorRT-built gallery will not agree; see the docstring)")
|
|
args = p.parse_args()
|
|
|
|
if (args.calib_a is None) != (args.calib_b is None):
|
|
return err("--calib-a and --calib-b must be given together")
|
|
|
|
models_dir = Path(args.models_dir)
|
|
arcface = Path(args.arcface) if args.arcface else models_dir / "LVFace-B_Glint360K.onnx"
|
|
detector = Path(args.detector) if args.detector else models_dir / "scrfd_500m_bnkps.onnx"
|
|
for path, what in ((arcface, "embedder"), (detector, "detector")):
|
|
if not path.is_file():
|
|
return err(f"{what} model not found: {path}\n"
|
|
f"Run: bash scripts/download_models.sh")
|
|
|
|
images_root = Path(args.images)
|
|
if not images_root.is_dir():
|
|
return err(f"image cache not found: {images_root}")
|
|
|
|
sizes = sorted({int(s) for s in args.sizes.split(",") if s.strip()})
|
|
if not sizes:
|
|
return err("--sizes is empty")
|
|
if args.probes_per_actor < 1:
|
|
return err("--probes-per-actor must be >= 1")
|
|
|
|
cv2.setRNGSeed(args.seed) # estimateAffinePartial2D's RANSAC draws from this
|
|
|
|
# ── actor pool ────────────────────────────────────────────────────────────
|
|
pool = discover_actors(images_root)
|
|
print(f"[select] {len(pool)} actor dirs with images under {images_root}",
|
|
file=sys.stderr)
|
|
if args.gallery:
|
|
ids, names = gallery_keys(Path(args.gallery))
|
|
pool = [a for a in pool
|
|
if (a["jellyfin_id"] and a["jellyfin_id"] in ids)
|
|
or normalise_name(a["name"]) in names]
|
|
print(f"[select] {len(pool)} of them are in {args.gallery}", file=sys.stderr)
|
|
|
|
need = max(args.min_images, args.probes_per_actor + 1)
|
|
eligible = [a for a in pool if len(a["images"]) >= need]
|
|
print(f"[select] {len(eligible)} have >= {need} mugshots", file=sys.stderr)
|
|
if len(eligible) < 2:
|
|
return err(f"need at least 2 actors with >= {need} mugshots; found "
|
|
f"{len(eligible)}. Build the image cache first "
|
|
f"(scripts/make_jellyfin_gallery.py) or lower --min-images.")
|
|
|
|
rng = random.Random(args.seed)
|
|
selected = sorted(rng.sample(eligible, min(args.actors, len(eligible))),
|
|
key=lambda a: a["dir"].name)
|
|
if len(selected) < args.actors:
|
|
print(f"[select] WARNING: only {len(selected)} eligible actors, "
|
|
f"--actors {args.actors} requested. FPI is gallery-size dependent — "
|
|
f"see the caveat.", file=sys.stderr)
|
|
|
|
# ── detect + align every mugshot of the selected actors, once ─────────────
|
|
providers = resolve_providers(args.providers)
|
|
det = SCRFDDetector(str(detector), providers, args.conf, args.nms)
|
|
emb_model = Embedder(str(arcface), providers, args.batch)
|
|
print(f"[models] detector={detector.name} embedder={arcface.name} "
|
|
f"providers={emb_model.sess.get_providers()} batch={emb_model.batch}",
|
|
file=sys.stderr)
|
|
|
|
t0 = time.time()
|
|
crops: list[np.ndarray] = []
|
|
rows: list[dict] = [] # parallel to crops: {actor, actor_idx, image}
|
|
actors: list[dict] = []
|
|
n_nodetect = 0
|
|
for a in selected:
|
|
actor_crops, actor_paths = [], []
|
|
for img_path in a["images"]:
|
|
img = cv2.imread(str(img_path))
|
|
if img is None:
|
|
n_nodetect += 1
|
|
continue
|
|
if args.max_side > 0 and max(img.shape[:2]) > args.max_side:
|
|
s = args.max_side / max(img.shape[:2])
|
|
img = cv2.resize(img, None, fx=s, fy=s, interpolation=cv2.INTER_AREA)
|
|
faces = det.detect(img)
|
|
if not faces:
|
|
enhanced = enhance_for_retry(img)
|
|
faces = det.detect(enhanced)
|
|
if faces:
|
|
img = enhanced
|
|
if not faces:
|
|
n_nodetect += 1
|
|
continue
|
|
best = max(faces, key=lambda f: f["confidence"])
|
|
crop = align_face(img, best["landmarks"])
|
|
if crop is None:
|
|
n_nodetect += 1
|
|
continue
|
|
actor_crops.append(crop)
|
|
actor_paths.append(img_path)
|
|
if len(actor_crops) < args.probes_per_actor + 1:
|
|
continue
|
|
ai = len(actors)
|
|
actors.append({"name": a["name"], "jellyfin_id": a["jellyfin_id"],
|
|
"dir": a["dir"].name, "n_images": len(actor_crops)})
|
|
for crop, img_path in zip(actor_crops, actor_paths):
|
|
rows.append({"actor_idx": ai, "image": str(img_path)})
|
|
crops.append(crop)
|
|
if len(actors) % 20 == 0:
|
|
print(f" [align] {len(actors)}/{len(selected)} actors, "
|
|
f"{len(crops)} crops", file=sys.stderr)
|
|
|
|
if len(actors) < 2:
|
|
return err(f"only {len(actors)} actors survived detection/alignment — "
|
|
f"nothing to match against")
|
|
print(f"[align] {len(actors)} actors, {len(crops)} aligned crops, "
|
|
f"{n_nodetect} images skipped (no face / unreadable) in "
|
|
f"{time.time() - t0:.1f}s", file=sys.stderr)
|
|
|
|
actor_of = np.array([r["actor_idx"] for r in rows], dtype=int)
|
|
|
|
# ── embed everything at native resolution ────────────────────────────────
|
|
t0 = time.time()
|
|
native = emb_model.embed(crops)
|
|
print(f"[embed] {len(crops)} native crops in {time.time() - t0:.1f}s",
|
|
file=sys.stderr)
|
|
|
|
if args.verify_against:
|
|
verify_embeddings(Path(args.verify_against), rows, native, actor_of)
|
|
|
|
# ── drop duplicate mugshots ───────────────────────────────────────────────
|
|
# The cache holds the same photograph twice for some actors (two provider
|
|
# URLs, one picture). A probe that is identical to a gallery reference is
|
|
# identified for free at every size, which flatters the whole curve, so
|
|
# remove duplicates the same way calibrate_gallery does.
|
|
n_dup = 0
|
|
if not args.keep_duplicates:
|
|
keep = np.ones(len(rows), bool)
|
|
for ai in range(len(actors)):
|
|
kept: list[int] = []
|
|
for i in np.nonzero(actor_of == ai)[0]:
|
|
if any(float(native[i] @ native[k]) > DEDUP_SIM for k in kept):
|
|
keep[i] = False
|
|
else:
|
|
kept.append(int(i))
|
|
n_dup = int((~keep).sum())
|
|
|
|
# An actor left with too few distinct mugshots to hold one out drops out.
|
|
counts = np.bincount(actor_of[keep], minlength=len(actors))
|
|
drop_actor = counts < args.probes_per_actor + 1
|
|
keep &= ~drop_actor[actor_of]
|
|
|
|
remap = np.full(len(actors), -1, dtype=int)
|
|
remap[~drop_actor] = np.arange(int((~drop_actor).sum()))
|
|
actors = [a for a, d in zip(actors, drop_actor) if not d]
|
|
rows = [r for r, k in zip(rows, keep) if k]
|
|
crops = [c for c, k in zip(crops, keep) if k]
|
|
native = native[keep]
|
|
actor_of = remap[actor_of[keep]]
|
|
for r, ai in zip(rows, actor_of):
|
|
r["actor_idx"] = int(ai)
|
|
for ai, a in enumerate(actors):
|
|
a["n_images"] = int(np.sum(actor_of == ai))
|
|
print(f"[dedup] dropped {n_dup} duplicate mugshots and "
|
|
f"{int(drop_actor.sum())} actors left with too few; "
|
|
f"{len(actors)} actors, {len(rows)} images remain", file=sys.stderr)
|
|
if len(actors) < 2:
|
|
return err("fewer than 2 actors survive de-duplication — the image "
|
|
"cache holds too few distinct mugshots")
|
|
|
|
# ── hold out the probes ───────────────────────────────────────────────────
|
|
is_probe = np.zeros(len(rows), bool)
|
|
for ai in range(len(actors)):
|
|
idx = np.nonzero(actor_of == ai)[0]
|
|
# Seeded per actor so the choice does not depend on iteration order.
|
|
r = random.Random(f"{args.seed}:{actors[ai]['dir']}")
|
|
for pick in r.sample(list(idx), args.probes_per_actor):
|
|
is_probe[pick] = True
|
|
probe_rows = np.nonzero(is_probe)[0]
|
|
gal_rows = np.nonzero(~is_probe)[0]
|
|
print(f"[holdout] {len(probe_rows)} probes held out, "
|
|
f"{len(gal_rows)} gallery embeddings remain", file=sys.stderr)
|
|
|
|
gal_emb = native[gal_rows]
|
|
gal_actor = actor_of[gal_rows]
|
|
probe_actor = actor_of[probe_rows]
|
|
|
|
# Per-actor column masks for the best-of-N scan (identity_matcher_node).
|
|
actor_cols = [np.nonzero(gal_actor == ai)[0] for ai in range(len(actors))]
|
|
have_refs = np.array([len(c) > 0 for c in actor_cols])
|
|
if not have_refs.all():
|
|
return err("an actor ended up with no gallery references left; "
|
|
"raise --min-images")
|
|
|
|
# ── calibration ───────────────────────────────────────────────────────────
|
|
if args.calib_a is not None:
|
|
cal = {"a": args.calib_a, "b": args.calib_b, "valid": True}
|
|
print(f"[calibration] using supplied a={cal['a']} b={cal['b']}", file=sys.stderr)
|
|
else:
|
|
cal = calibrate_gallery(gal_emb, gal_actor)
|
|
if not cal["valid"]:
|
|
return err(
|
|
"calibration could not be fitted, and this study will not fall back to a "
|
|
"raw cosine threshold (CLAUDE.md invariant). Use more actors with >= "
|
|
f"{MIN_EMB_FOR_POSITIVE} mugshots, or pass --calib-a/--calib-b from a "
|
|
"production gallery.")
|
|
log_prior_odds = float(np.log(args.match_prior / (1.0 - args.match_prior)))
|
|
|
|
# ── sweep ─────────────────────────────────────────────────────────────────
|
|
down, up = INTERP[args.down_interp], INTERP[args.up_interp]
|
|
probe_crops = [crops[i] for i in probe_rows]
|
|
results, per_probe = [], []
|
|
for size in sizes:
|
|
t0 = time.time()
|
|
degraded = [degrade(c, size, down, up) for c in probe_crops]
|
|
q = emb_model.embed(degraded)
|
|
|
|
sims = q @ gal_emb.T # [n_probe, n_gal]
|
|
best_per_actor = np.stack([sims[:, cols].max(axis=1) for cols in actor_cols],
|
|
axis=1) # [n_probe, n_actor]
|
|
best_actor = best_per_actor.argmax(axis=1)
|
|
best_sim = best_per_actor.max(axis=1)
|
|
p_match = np.asarray(probability(best_sim, cal["a"], cal["b"], log_prior_odds))
|
|
|
|
accept = p_match > args.prob_threshold
|
|
correct = best_actor == probe_actor
|
|
tpi = int(np.sum(accept & correct))
|
|
fpi = int(np.sum(accept & ~correct))
|
|
unid = int(np.sum(~accept))
|
|
n = len(probe_rows)
|
|
|
|
results.append({
|
|
"size_px": size,
|
|
"n_probes": n,
|
|
"tpi": tpi, "fpi": fpi, "unidentified": unid,
|
|
"tpi_rate": tpi / n, "fpi_rate": fpi / n, "unidentified_rate": unid / n,
|
|
"rank1_rate": float(np.mean(correct)),
|
|
"mean_best_sim": float(np.mean(best_sim)),
|
|
"mean_p_match": float(np.mean(p_match)),
|
|
"mean_sim_true_actor": float(np.mean(
|
|
best_per_actor[np.arange(n), probe_actor])),
|
|
})
|
|
if args.per_probe:
|
|
for j in range(n):
|
|
per_probe.append({
|
|
"size_px": size,
|
|
"probe_image": rows[probe_rows[j]]["image"],
|
|
"true_actor": actors[probe_actor[j]]["name"],
|
|
"matched_actor": actors[best_actor[j]]["name"],
|
|
"best_sim": float(best_sim[j]),
|
|
"p_match": float(p_match[j]),
|
|
"outcome": ("TPI" if accept[j] and correct[j]
|
|
else "FPI" if accept[j] else "unidentified"),
|
|
})
|
|
print(f"[sweep] {size:3d}px TPI {tpi:4d} ({100 * tpi / n:5.1f}%) "
|
|
f"FPI {fpi:4d} ({100 * fpi / n:5.1f}%) "
|
|
f"unid {unid:4d} ({100 * unid / n:5.1f}%) "
|
|
f"rank1 {100 * np.mean(correct):5.1f}% "
|
|
f"[{time.time() - t0:.1f}s]", file=sys.stderr)
|
|
|
|
op = pick_operating_point(results, args.tpi_retention, args.fpi_slack)
|
|
|
|
# ── outputs ───────────────────────────────────────────────────────────────
|
|
out = Path(args.out)
|
|
out.parent.mkdir(parents=True, exist_ok=True)
|
|
# Append rather than with_suffix() so a prefix containing a dot keeps its name.
|
|
csv_path = out.with_name(out.name + ".csv")
|
|
json_path = out.with_name(out.name + ".json")
|
|
png_path = out.with_name(out.name + ".png")
|
|
fields = list(results[0].keys())
|
|
with open(csv_path, "w", newline="") as f:
|
|
w = csv.DictWriter(f, fieldnames=fields)
|
|
w.writeheader()
|
|
w.writerows(results)
|
|
|
|
meta = {
|
|
"requirement": "VR-005",
|
|
"caveat": CAVEAT.format(n_actors=len(actors)),
|
|
"model": arcface.stem,
|
|
"detector": detector.stem,
|
|
"backend": f"onnxruntime {'/'.join(emb_model.sess.get_providers())} (fp32 ONNX; "
|
|
f"a TensorRT fp16 build is a different embedding space)",
|
|
"n_actors": len(actors),
|
|
"n_probes": len(probe_rows),
|
|
"n_gallery_embeddings": len(gal_rows),
|
|
"probes_per_actor": args.probes_per_actor,
|
|
"seed": args.seed,
|
|
"sizes": sizes,
|
|
"prob_threshold": args.prob_threshold,
|
|
"match_prior": args.match_prior,
|
|
"calibration": cal,
|
|
"calibration_source": "supplied" if args.calib_a is not None else "fitted",
|
|
"sim_boundary_at_threshold": float(
|
|
(np.log(args.prob_threshold / (1 - args.prob_threshold))
|
|
- cal["b"] - log_prior_odds) / cal["a"]),
|
|
"down_interp": args.down_interp,
|
|
"up_interp": args.up_interp,
|
|
"max_side": args.max_side,
|
|
"duplicate_mugshots_dropped": n_dup,
|
|
"operating_point_rule": (
|
|
f"smallest size retaining >= {args.tpi_retention:.0%} of the 112 px "
|
|
f"control TPI rate with <= +{args.fpi_slack:.1%} absolute FPI"),
|
|
"operating_point": op,
|
|
"images_skipped_no_face": n_nodetect,
|
|
"curve": results,
|
|
"actors": actors,
|
|
}
|
|
json_path.write_text(json.dumps(meta, indent=2) + "\n")
|
|
|
|
if per_probe:
|
|
pp = out.with_name(out.name + ".per_probe.csv")
|
|
with open(pp, "w", newline="") as f:
|
|
w = csv.DictWriter(f, fieldnames=list(per_probe[0].keys()))
|
|
w.writeheader()
|
|
w.writerows(per_probe)
|
|
print(f"[out] {pp}", file=sys.stderr)
|
|
|
|
write_plot(results, png_path, meta)
|
|
|
|
# ── stdout report ─────────────────────────────────────────────────────────
|
|
print(f"\nVR-005 — minimum face size, {arcface.stem}")
|
|
print(f"{len(actors)} actors, {len(probe_rows)} probes/size, "
|
|
f"{len(gal_rows)} gallery embeddings at native resolution")
|
|
print(f"identify when P>{args.prob_threshold}, i.e. cosine above "
|
|
f"{meta['sim_boundary_at_threshold']:.4f} under the calibration fitted "
|
|
f"on this gallery\n")
|
|
print(f"{'size':>5} {'TPI':>8} {'FPI':>8} {'unid':>8} {'rank1':>8} {'mean sim':>9}")
|
|
for r in results:
|
|
print(f"{r['size_px']:>5} {100 * r['tpi_rate']:>7.1f}% "
|
|
f"{100 * r['fpi_rate']:>7.1f}% {100 * r['unidentified_rate']:>7.1f}% "
|
|
f"{100 * r['rank1_rate']:>7.1f}% {r['mean_best_sim']:>9.4f}")
|
|
print(f"\noperating point: {op if op else 'none of the swept sizes qualifies'}"
|
|
f" ({meta['operating_point_rule']})")
|
|
print(f"\n{meta['caveat']}")
|
|
print(f"\n[out] {csv_path}\n[out] {json_path}\n[out] {png_path}")
|
|
return 0
|
|
|
|
|
|
def _separation(emb: np.ndarray, actor: np.ndarray) -> tuple[float, float, float]:
|
|
"""(mean same-actor sim, mean different-actor sim, d') — the property that has
|
|
to survive for an embedding space to be usable, whatever its coordinates."""
|
|
iu, ju = np.triu_indices(len(actor), k=1)
|
|
sims = (emb @ emb.T)[iu, ju]
|
|
same = actor[iu] == actor[ju]
|
|
pos = sims[same & (sims < 0.9999)] # drop duplicate source images
|
|
neg = sims[~same]
|
|
if pos.size < 2 or neg.size < 2:
|
|
return float("nan"), float("nan"), float("nan")
|
|
d = (pos.mean() - neg.mean()) / np.sqrt(0.5 * (pos.var() + neg.var()))
|
|
return float(pos.mean()), float(neg.mean()), float(d)
|
|
|
|
|
|
def verify_embeddings(gallery_path: Path, rows: list[dict], native: np.ndarray,
|
|
actor_of: np.ndarray) -> None:
|
|
"""Cross-check this script's ONNX port against a gallery built by the C++
|
|
pipeline from the same mugshots.
|
|
|
|
Agreement is ~1.0 only if that gallery was built with the same backend. A
|
|
TensorRT fp16 build lands around 0.85 on LVFace-B while separating just as
|
|
well, so the separation figures — not the agreement — are what says whether
|
|
the port is sound."""
|
|
from sae_gallery import load_gallery_hdf5
|
|
g = load_gallery_hdf5(gallery_path)
|
|
stored: dict[tuple[str, str], np.ndarray] = {}
|
|
for a in g["actors"]:
|
|
key = a.get("jellyfin_id") or normalise_name(a.get("name", ""))
|
|
for e, src in zip(a.get("embeddings", []), a.get("source_images", [])):
|
|
if src:
|
|
stored[(key, src)] = np.asarray(e, np.float32)
|
|
|
|
sims, paired_mine, paired_ref, paired_actor = [], [], [], []
|
|
for i, r in enumerate(rows):
|
|
path = Path(r["image"])
|
|
head, _, _ = path.parent.name.partition("_")
|
|
key = head if JELLYFIN_ID_RE.match(head) else normalise_name(
|
|
path.parent.name.replace("_", " "))
|
|
ref = stored.get((key, path.name))
|
|
if ref is None or ref.shape != native[i].shape:
|
|
continue
|
|
ref = ref / max(float(np.linalg.norm(ref)), 1e-6)
|
|
sims.append(float(native[i] @ ref))
|
|
paired_mine.append(native[i])
|
|
paired_ref.append(ref)
|
|
paired_actor.append(actor_of[i])
|
|
if not sims:
|
|
print(f"[verify] no overlap with {gallery_path} — nothing checked",
|
|
file=sys.stderr)
|
|
return
|
|
|
|
sims_arr = np.asarray(sims)
|
|
print(f"[verify] {len(sims)} embeddings vs {gallery_path.name}: "
|
|
f"mean cos={sims_arr.mean():.4f} min={sims_arr.min():.4f}",
|
|
file=sys.stderr)
|
|
act = np.asarray(paired_actor)
|
|
for label, mat in (("this script", np.asarray(paired_mine)),
|
|
("stored gallery", np.asarray(paired_ref))):
|
|
pos, neg, d = _separation(mat, act)
|
|
print(f"[verify] {label:>14s}: same-actor {pos:.3f} "
|
|
f"different-actor {neg:.3f} d'={d:.2f}", file=sys.stderr)
|
|
if sims_arr.mean() < 0.99:
|
|
print("[verify] embeddings differ from the stored gallery. If d' is "
|
|
"comparable this is a backend difference (e.g. a TensorRT fp16 "
|
|
"build), not a broken port; the study is self-consistent either "
|
|
"way. If d' collapsed, the port is wrong.", file=sys.stderr)
|
|
|
|
|
|
def err(msg: str) -> int:
|
|
print(f"error: {msg}", file=sys.stderr)
|
|
return 2
|
|
|
|
|
|
if __name__ == "__main__":
|
|
sys.exit(main())
|