dtourolle
889018aa34
Sharpness is not a sufficient statistic for identity loss. Six cells at effectively identical measured sharpness span 15.3% to 91.0% TPI, ordered entirely by source size, because a scalar keyed on high-frequency energy cannot separate attenuated high frequencies from destroyed spatial sampling. AR-028's "kept separate, not collapsed into one scalar" now rests on a measurement rather than an argument -- and the reasoning it used to rest on, that the aligned crop is scale-normalised so a measure there cannot re-measure size, was wrong and is corrected in place. Variance of Laplacian -- the most widely used blur metric there is -- is anti-predictive at fixed degradation on all three blur families. The decile it calls sharpest is 2.6x less identifiable than the decile it calls blurriest, monotone across ten bins, because within a cell its residual variance is native contrast rather than detail, and hard shadows and JPEG ringing raise it while making a face harder to match. Gating on it would preferentially discard the more identifiable faces. Blur breaks confidence, not identity: rank-1 holds at 80.2% where TPI is 15.3%, and FPI never left 0.1% in any of the 108 cells. Degradation produces abstention, never a wrong name. That is also why sharpness fails as a compute gate -- even a visually destroyed face stays 46.9% identifiable, so a gate discards recoverable evidence at three times the cost of the free size filter. Discount, do not gate; the rule AR-028 already stated now has evidence for why it is right rather than merely cautious. Records the shape a discount must have (flat, then a cliff between sigma 2 and 3), that its cost scales with proximity to the decision boundary rather than with blur, and that the pose half of VR-012 has not been run. TRACES: VR-012, AR-028, AR-029, AR-030 | SR-002
100 lines
3.8 KiB
Markdown
100 lines
3.8 KiB
Markdown
# scene-actor-extraction
|
|
|
|
A face-recognition pipeline that finds when each actor appears on screen in
|
|
a film or TV episode, built on [KPN++](https://gitea.tourolle.paris/dtourolle/KPN)
|
|
(a C++20 Kahn Process Network library) for the detect, track, match, and
|
|
scene pipeline, with a Jellyfin-integrated gallery and an X-Ray-validated
|
|
optimizer.
|
|
|
|
This is a correctly scored second from a held-out film, one the optimizer
|
|
never saw during tuning:
|
|
|
|
|
|
|
|
Every visible face is named at 100% confidence (Chris Noth, Hank Azaria,
|
|
Bobby Cannavale), the background extra is correctly left unnamed, and the
|
|
two credited cast members without a visible face are correctly reported
|
|
present but not visible. This matches Amazon X-Ray's own record for this
|
|
second exactly.
|
|
|
|
Results are not uniform across films. The hardest held-out film scores 46%
|
|
F1. This report documents why: one tunable trade (extinction bridging at
|
|
hard cuts), one structural limit (X-Ray credits people whose faces never
|
|
appear on screen), and a small number of cases where the pipeline is
|
|
correct and X-Ray's ground truth is not. Read
|
|
[how we score against X-Ray](methodology.md) first. X-Ray's ground truth is
|
|
scene-level; the pipeline's output is per-second. That difference shapes
|
|
every finding below.
|
|
|
|
## Findings
|
|
|
|
<div class="grid cards" markdown>
|
|
|
|
- :material-trophy:{ .lg .middle } **[Which model is best?](best-model.md)**
|
|
|
|
---
|
|
|
|
Calibration curves first, independent of any threshold, then held-out
|
|
F1 across three models. LVFace-B Glint360K wins both, and wins on every
|
|
held-out film.
|
|
|
|
- :material-filter:{ .lg .middle } **[Whole vs. cast-restricted gallery](gallery-scope.md)**
|
|
|
|
---
|
|
|
|
Restricting the matcher to a film's credited cast improves F1,
|
|
recall, and misID rate at once, but is not a shipped runtime feature
|
|
yet.
|
|
|
|
- :material-account-convert:{ .lg .middle } **[Does pose expansion help?](pose-expansion.md)**
|
|
|
|
---
|
|
|
|
A training-set effect that did not reproduce on 5 held-out films once
|
|
two methodology bugs in the comparison harness were found and fixed.
|
|
|
|
- :material-blur:{ .lg .middle } **[What does blur cost?](quality-knee.md)**
|
|
|
|
---
|
|
|
|
Sharpness is not a sufficient statistic for identity loss, blur breaks
|
|
confidence rather than ranking, and variance-of-Laplacian is
|
|
anti-predictive at fixed resolution.
|
|
|
|
- :material-magnify-expand:{ .lg .middle } **[Deep dive: LVFace-B Glint360K](lvface-deep-dive.md)**
|
|
|
|
---
|
|
|
|
The held-out generalization gap, the two mechanisms behind its errors,
|
|
and every distinct case where it names someone outside the film's
|
|
credited cast.
|
|
|
|
</div>
|
|
|
|
## Full experiment log
|
|
|
|
- **[Full experiment log](model-bakeoff.md)**: the complete log behind the
|
|
four pages above, including how replaying against cached embeddings
|
|
inside the same KPN network makes a full model and configuration
|
|
comparison practical, the full results table, and every caveat. This is
|
|
where the shipped [`src/config.hpp`](https://REPOLINK/src/config.hpp)
|
|
defaults come from.
|
|
- **[Service conversion (proposal)](service-conversion.md)**: design
|
|
sketch for a native idle-GPU worker gated on screen lock, not yet built.
|
|
|
|
## Reproducing the benchmarks
|
|
|
|
Gallery `.h5` files, embedding dumps, the X-Ray corpus, montage frame
|
|
images, and DE trajectories are not committed to this repository. They are
|
|
pushed to the Gitea package registry and pulled on demand:
|
|
|
|
```bash
|
|
scripts/artifacts/pull_artifacts.sh galleries
|
|
scripts/artifacts/pull_artifacts.sh experiment-data
|
|
scripts/artifacts/pull_artifacts.sh montage-frames <film-slug>
|
|
```
|
|
|
|
See [`scripts/artifacts/push_artifacts.sh`](https://REPOLINK/scripts/artifacts/push_artifacts.sh)
|
|
for the upload side, which requires a `GITEA_TOKEN` with package write
|
|
scope.
|