dtourolle
0bd2747069
Replaces narrative claims with verified numbers across all report pages: - Cross-model held-out validation (LVFace/mbf/r18, all 5 held-out films): LVFace wins every film outright, not just "consistent with" the training-set pick. r50 dropped from the detailed comparison (gallery has ~30% fewer reference images per actor than the other three models on identical source photos). - Per-film training breakdown: LVFace does not win every training film (mbf beats it on Lord of War); the 75.3% macro figure hides a 10.7pp spread. - Gallery coverage computed per film (20.3%-78.6%) instead of one flat 67%-missing average. - Found and fixed a real scoring bug in optimize.py: a candidate whose hardest film's replay timed out was averaged over survivors instead of penalized, silently rewarding partial coverage. Affected 3 of 16 training combos; corrected throughout, and optimize.py now scores an incomplete evaluation f1=0.0 instead of averaging over whichever films happened to finish. - Every FPI frame in the deep dive now comes from the proper montage renderer (Onscreen/Offscreen panel, ghosts never drawn as boxes), never the bare-box debug overlay used earlier. - Every distinct out-of-cast name across all 9 films gets its own frame at its first appearance (9 names, 4 films), not a single-example spot check: 2 ground-truth gaps, 1 photograph misread as a person, 6 genuine lookalike confusions. - New methodology.md: the scene-level-vs-per-second scoring mismatch that the rest of the report assumes, written out once. - Cut the deadlock/gdb debugging narrative from the experiment log; kept the one fact that matters (KPN's node/network split lets the expensive GPU stage run once and the cheap stage replay against cached embeddings). - Plain declarative style throughout, no em dashes, no blog voice.
92 lines
3.5 KiB
Markdown
92 lines
3.5 KiB
Markdown
# scene-actor-extraction
|
|
|
|
A face-recognition pipeline that finds when each actor appears on screen in
|
|
a film or TV episode, built on [KPN++](https://gitea.tourolle.paris/dtourolle/KPN)
|
|
(a C++20 Kahn Process Network library) for the detect, track, match, and
|
|
scene pipeline, with a Jellyfin-integrated gallery and an X-Ray-validated
|
|
optimizer.
|
|
|
|
This is a correctly scored second from a held-out film, one the optimizer
|
|
never saw during tuning:
|
|
|
|
|
|
|
|
Every visible face is named at 100% confidence (Chris Noth, Hank Azaria,
|
|
Bobby Cannavale), the background extra is correctly left unnamed, and the
|
|
two credited cast members without a visible face are correctly reported
|
|
present but not visible. This matches Amazon X-Ray's own record for this
|
|
second exactly.
|
|
|
|
Results are not uniform across films. The hardest held-out film scores 46%
|
|
F1. This report documents why: one tunable trade (extinction bridging at
|
|
hard cuts), one structural limit (X-Ray credits people whose faces never
|
|
appear on screen), and a small number of cases where the pipeline is
|
|
correct and X-Ray's ground truth is not. Read
|
|
[how we score against X-Ray](methodology.md) first. X-Ray's ground truth is
|
|
scene-level; the pipeline's output is per-second. That difference shapes
|
|
every finding below.
|
|
|
|
## Findings
|
|
|
|
<div class="grid cards" markdown>
|
|
|
|
- :material-trophy:{ .lg .middle } **[Which model is best?](best-model.md)**
|
|
|
|
---
|
|
|
|
Calibration curves first, independent of any threshold, then held-out
|
|
F1 across three models. LVFace-B Glint360K wins both, and wins on every
|
|
held-out film.
|
|
|
|
- :material-filter:{ .lg .middle } **[Whole vs. cast-restricted gallery](gallery-scope.md)**
|
|
|
|
---
|
|
|
|
Restricting the matcher to a film's credited cast improves F1,
|
|
recall, and misID rate at once, but is not a shipped runtime feature
|
|
yet.
|
|
|
|
- :material-account-convert:{ .lg .middle } **[Does pose expansion help?](pose-expansion.md)**
|
|
|
|
---
|
|
|
|
A training-set effect that did not reproduce on 5 held-out films once
|
|
two methodology bugs in the comparison harness were found and fixed.
|
|
|
|
- :material-magnify-expand:{ .lg .middle } **[Deep dive: LVFace-B Glint360K](lvface-deep-dive.md)**
|
|
|
|
---
|
|
|
|
The held-out generalization gap, the two mechanisms behind its errors,
|
|
and every distinct case where it names someone outside the film's
|
|
credited cast.
|
|
|
|
</div>
|
|
|
|
## Full experiment log
|
|
|
|
- **[Full experiment log](model-bakeoff.md)**: the complete log behind the
|
|
four pages above, including how replaying against cached embeddings
|
|
inside the same KPN network makes a full model and configuration
|
|
comparison practical, the full results table, and every caveat. This is
|
|
where the shipped [`src/config.hpp`](https://REPOLINK/src/config.hpp)
|
|
defaults come from.
|
|
- **[Service conversion (proposal)](service-conversion.md)**: design
|
|
sketch for a native idle-GPU worker gated on screen lock, not yet built.
|
|
|
|
## Reproducing the benchmarks
|
|
|
|
Gallery `.h5` files, embedding dumps, the X-Ray corpus, montage frame
|
|
images, and DE trajectories are not committed to this repository. They are
|
|
pushed to the Gitea package registry and pulled on demand:
|
|
|
|
```bash
|
|
scripts/artifacts/pull_artifacts.sh galleries
|
|
scripts/artifacts/pull_artifacts.sh experiment-data
|
|
scripts/artifacts/pull_artifacts.sh montage-frames <film-slug>
|
|
```
|
|
|
|
See [`scripts/artifacts/push_artifacts.sh`](https://REPOLINK/scripts/artifacts/push_artifacts.sh)
|
|
for the upload side, which requires a `GITEA_TOKEN` with package write
|
|
scope.
|