dtourolle
4b5557974b
- switch report frames to the scene best/worst montage renderer (Onscreen/Offscreen panels + TPI/FPI/FN legend): perfect-second hero, wedding couple, funeral 19-of-20, polygraph bridging, crew-scene FN ceiling, Robert Patrick ground-truth gap, rapid-cut double label, Herbie Hancock on an in-fiction screen - deep dive restructured: extinction bridging framed as designed behavior with a measurable cost (debug overlay draws the boxes; the shipped output is presence windows), plus the face-vs-presence ceiling and two X-Ray-is-wrong exhibits - Material polish: light/dark palette toggle, landing-page grid cards, figure/caption CSS, how-to-read admonition; site_url set so 404 links resolve under the Pages subpath - README: perfect-second and screen-call frames committed (gitignore exceptions), readme_example.jpg retired - build_site.sh: stage_frame helper downscales montage frames to 1920px and pulls any missing montage-frames packages
90 lines
3.8 KiB
Markdown
90 lines
3.8 KiB
Markdown
# scene-actor-extraction
|
|
|
|
A face-recognition pipeline that finds when each actor appears on screen in a
|
|
film or TV episode — built on [KPN++](https://gitea.tourolle.paris/dtourolle/KPN)
|
|
(a C++20 Kahn Process Network library) for the detect → track → match → scene
|
|
pipeline, with a Jellyfin-integrated gallery and an X-Ray-validated optimizer.
|
|
|
|
This is a perfect X-Ray second, on a film the optimizer never saw:
|
|
|
|
|
|
|
|
Every visible face named at 100% — Chris Noth, Hank Azaria, Bobby Cannavale —
|
|
the background extra honestly left unnamed, and the two credited cast without
|
|
a visible face correctly carried as present off-screen by the tracker's
|
|
presence windows. That's the pipeline exactly reproducing Amazon X-Ray's
|
|
record for this second.
|
|
|
|
It doesn't always go like that: the hardest held-out film scores 46% F1, and
|
|
the report is honest about *why* — one tunable trade (extinction bridging at
|
|
hard cuts), one structural ceiling (X-Ray credits people whose faces never
|
|
appear), and a few cases where the pipeline is right and X-Ray is wrong. The
|
|
evidence for all of it is in the pages below.
|
|
|
|
## Start here — four questions this bake-off answers
|
|
|
|
<div class="grid cards" markdown>
|
|
|
|
- :material-trophy:{ .lg .middle } **[Which model is best?](best-model.md)**
|
|
|
|
---
|
|
|
|
Calibration curves first (discriminative power, independent of any
|
|
threshold), then F1 on the actual benchmark. LVFace-B Glint360K wins
|
|
both.
|
|
|
|
- :material-filter:{ .lg .middle } **[Whole vs. cast-restricted gallery](gallery-scope.md)**
|
|
|
|
---
|
|
|
|
Restricting the matcher to a film's credited cast is a clean win on
|
|
every axis (+3.3pp F1, less than a third the misIDs) — but isn't a
|
|
shipped runtime feature yet.
|
|
|
|
- :material-account-convert:{ .lg .middle } **[Does pose expansion help?](pose-expansion.md)**
|
|
|
|
---
|
|
|
|
A convincing training-set effect that didn't reproduce on 5 held-out
|
|
films once two methodology bugs were caught and fixed. An honest null
|
|
result, not a forced narrative.
|
|
|
|
- :material-magnify-expand:{ .lg .middle } **[Deep dive: LVFace-B Glint360K](lvface-deep-dive.md)**
|
|
|
|
---
|
|
|
|
The held-out generalization gap, how the error budget decomposes
|
|
(extinction bridging at hard cuts, X-Ray's scene-membership vs.
|
|
on-screen-face ceiling), and the frames where the pipeline is right
|
|
and the ground truth is wrong.
|
|
|
|
</div>
|
|
|
|
## The full technical log
|
|
|
|
- **[Model bake-off + threshold re-tune](model-bakeoff.md)** —
|
|
the complete experiment log behind the four pages above: the ROCm teardown
|
|
deadlock root cause and fix, DE concurrency tuning, the full 16-combo
|
|
results table, and every caveat. This is where the shipped
|
|
[`src/config.hpp`](https://REPOLINK/src/config.hpp) defaults come from.
|
|
- **[Optimizer experiments (prior round)](optimizer-experiments.md)** — the
|
|
earlier scene-union-metric tuning pass, superseded by the per-second metric
|
|
used in the bake-off but kept for the ground-truth/architecture background.
|
|
- **[Service conversion (proposal)](service-conversion.md)** — design sketch
|
|
for a native idle-GPU worker gated on screen lock, not yet built.
|
|
|
|
## Reproducing the benchmarks
|
|
|
|
Gallery `.h5` files, embedding dumps, the X-Ray corpus, montage frame images,
|
|
and DE trajectories are not committed to this repository — they're pushed to
|
|
the Gitea package registry and pulled on demand:
|
|
|
|
```bash
|
|
scripts/artifacts/pull_artifacts.sh galleries
|
|
scripts/artifacts/pull_artifacts.sh experiment-data
|
|
scripts/artifacts/pull_artifacts.sh montage-frames <film-slug>
|
|
```
|
|
|
|
See [`scripts/artifacts/push_artifacts.sh`](https://REPOLINK/scripts/artifacts/push_artifacts.sh)
|
|
for the upload side (requires a `GITEA_TOKEN` with package write scope).
|