dtourolle
889018aa34
Sharpness is not a sufficient statistic for identity loss. Six cells at effectively identical measured sharpness span 15.3% to 91.0% TPI, ordered entirely by source size, because a scalar keyed on high-frequency energy cannot separate attenuated high frequencies from destroyed spatial sampling. AR-028's "kept separate, not collapsed into one scalar" now rests on a measurement rather than an argument -- and the reasoning it used to rest on, that the aligned crop is scale-normalised so a measure there cannot re-measure size, was wrong and is corrected in place. Variance of Laplacian -- the most widely used blur metric there is -- is anti-predictive at fixed degradation on all three blur families. The decile it calls sharpest is 2.6x less identifiable than the decile it calls blurriest, monotone across ten bins, because within a cell its residual variance is native contrast rather than detail, and hard shadows and JPEG ringing raise it while making a face harder to match. Gating on it would preferentially discard the more identifiable faces. Blur breaks confidence, not identity: rank-1 holds at 80.2% where TPI is 15.3%, and FPI never left 0.1% in any of the 108 cells. Degradation produces abstention, never a wrong name. That is also why sharpness fails as a compute gate -- even a visually destroyed face stays 46.9% identifiable, so a gate discards recoverable evidence at three times the cost of the free size filter. Discount, do not gate; the rule AR-028 already stated now has evidence for why it is right rather than merely cautious. Records the shape a discount must have (flat, then a cliff between sigma 2 and 3), that its cost scales with proximity to the decision boundary rather than with blur, and that the pose half of VR-012 has not been run. TRACES: VR-012, AR-028, AR-029, AR-030 | SR-002
3.8 KiB
scene-actor-extraction
A face-recognition pipeline that finds when each actor appears on screen in a film or TV episode, built on KPN++ (a C++20 Kahn Process Network library) for the detect, track, match, and scene pipeline, with a Jellyfin-integrated gallery and an X-Ray-validated optimizer.
This is a correctly scored second from a held-out film, one the optimizer never saw during tuning:
Every visible face is named at 100% confidence (Chris Noth, Hank Azaria, Bobby Cannavale), the background extra is correctly left unnamed, and the two credited cast members without a visible face are correctly reported present but not visible. This matches Amazon X-Ray's own record for this second exactly.
Results are not uniform across films. The hardest held-out film scores 46% F1. This report documents why: one tunable trade (extinction bridging at hard cuts), one structural limit (X-Ray credits people whose faces never appear on screen), and a small number of cases where the pipeline is correct and X-Ray's ground truth is not. Read how we score against X-Ray first. X-Ray's ground truth is scene-level; the pipeline's output is per-second. That difference shapes every finding below.
Findings
-
:material-trophy:{ .lg .middle } Which model is best?
Calibration curves first, independent of any threshold, then held-out F1 across three models. LVFace-B Glint360K wins both, and wins on every held-out film.
-
:material-filter:{ .lg .middle } Whole vs. cast-restricted gallery
Restricting the matcher to a film's credited cast improves F1, recall, and misID rate at once, but is not a shipped runtime feature yet.
-
:material-account-convert:{ .lg .middle } Does pose expansion help?
A training-set effect that did not reproduce on 5 held-out films once two methodology bugs in the comparison harness were found and fixed.
-
:material-blur:{ .lg .middle } What does blur cost?
Sharpness is not a sufficient statistic for identity loss, blur breaks confidence rather than ranking, and variance-of-Laplacian is anti-predictive at fixed resolution.
-
:material-magnify-expand:{ .lg .middle } Deep dive: LVFace-B Glint360K
The held-out generalization gap, the two mechanisms behind its errors, and every distinct case where it names someone outside the film's credited cast.
Full experiment log
- Full experiment log: the complete log behind the
four pages above, including how replaying against cached embeddings
inside the same KPN network makes a full model and configuration
comparison practical, the full results table, and every caveat. This is
where the shipped
src/config.hppdefaults come from. - Service conversion (proposal): design sketch for a native idle-GPU worker gated on screen lock, not yet built.
Reproducing the benchmarks
Gallery .h5 files, embedding dumps, the X-Ray corpus, montage frame
images, and DE trajectories are not committed to this repository. They are
pushed to the Gitea package registry and pulled on demand:
scripts/artifacts/pull_artifacts.sh galleries
scripts/artifacts/pull_artifacts.sh experiment-data
scripts/artifacts/pull_artifacts.sh montage-frames <film-slug>
See scripts/artifacts/push_artifacts.sh
for the upload side, which requires a GITEA_TOKEN with package write
scope.