dtourolle
7556c836da
Add model-bakeoff.md for the opencv5 build: the ten-knob DE tuning and where each shipped config default comes from, the replay architecture, and the flood-fill-on-learned-boundaries step change (62.6% -> 74.9% presence F1, LOO). Rewrite index.md to lead with the learned scene-boundary result and point at the current pages, with the July four-model bake-off moved to an Archive section. Both pages build with no broken links.
100 lines
4.7 KiB
Markdown
100 lines
4.7 KiB
Markdown
# scene-actor-extraction
|
|
|
|
A face-recognition pipeline that finds when each actor appears on screen in
|
|
a film or TV episode, built on [KPN++](https://gitea.tourolle.paris/dtourolle/KPN)
|
|
(a C++20 Kahn Process Network library) for the detect, track, match, and
|
|
scene pipeline, with a Jellyfin-integrated gallery and an X-Ray-validated
|
|
optimizer.
|
|
|
|
This is a correctly scored second from a held-out film, one the optimizer
|
|
never saw during tuning:
|
|
|
|
|
|
|
|
Every visible face is named at 100% confidence (Chris Noth, Hank Azaria,
|
|
Bobby Cannavale), the background extra is correctly left unnamed, and the
|
|
two credited cast members without a visible face are correctly reported
|
|
present but not visible. This matches Amazon X-Ray's own record for this
|
|
second exactly.
|
|
|
|
## The headline: learned scene boundaries
|
|
|
|
The current opencv5 build's biggest gain is **flood-fill presence on a
|
|
learned scene-boundary detector**. An actor seen once inside a shot is
|
|
reported for the whole shot — but only if the shot boundaries are good. A
|
|
learned XGBoost boundary detector, scored **leave-one-out** so no film is
|
|
ever measured by a detector that trained on it, lifts per-second X-Ray
|
|
presence F1 across nine films and improves every one of them:
|
|
|
|
| boundary source for flood-fill | presence F1 |
|
|
| ------------------------------ | ----------: |
|
|
| track-extent (flood off) | 62.6% |
|
|
| flood + grayscale cuts | 64.0% |
|
|
| **flood + learned detector (LOO)** | **74.9%** |
|
|
|
|
|
|
|
|
The full story — why the old grayscale cut detector broke Scarface, what
|
|
features work, and the per-film breakdown — is on the
|
|
[learned scene-boundary detector](scene-boundary-detector.md) page.
|
|
|
|
## What the numbers mean, and their limits
|
|
|
|
Results are not uniform across films, and they should not be. X-Ray's ground
|
|
truth is scene-level and credits people whose faces never appear on screen;
|
|
the pipeline's output is per-second and can only name a face it can see.
|
|
That difference is a structural recall ceiling, not a bug. Read
|
|
[how we score against X-Ray](methodology.md) first — it defines F1,
|
|
precision, recall, and misID, and explains the two limits (off-screen cast
|
|
and gallery coverage) that shape every finding.
|
|
|
|
Precision on identified faces is near-perfect: where the pipeline names a
|
|
face, it is almost always a name X-Ray also credits to that scene. The
|
|
frames throughout this documentation make the tension visual — **green** =
|
|
true positive, **red** = false positive, **orange** = unknown, and a
|
|
**blue** panel lists credited cast present with no visible face.
|
|
|
|
## Full experiment log
|
|
|
|
- **[Full experiment log (opencv5)](model-bakeoff.md)**: the complete log
|
|
behind the current build — the ten-knob differential-evolution tuning, the
|
|
shipped [`src/config.hpp`](https://REPOLINK/src/config.hpp) defaults and
|
|
where each comes from, the replay architecture that makes a nine-film
|
|
search tractable, and the flood-fill step change.
|
|
- **[Learned scene-boundary detector](scene-boundary-detector.md)**: the
|
|
features, the model, leave-one-out results, and the two headline films.
|
|
- **[Benchmark — SuperHero](benchmark.md)**: the benchmark harness.
|
|
- **[Service conversion (proposal)](service-conversion.md)**: design sketch
|
|
for a native idle-GPU worker gated on screen lock, not yet built.
|
|
|
|
## Archive (July 2026)
|
|
|
|
The pre-opencv5 four-model ArcFace/LVFace bake-off is kept for provenance.
|
|
Its numbers are historical; the current build supersedes them.
|
|
|
|
- [Best model (July)](best-model-2026-07.md) — LVFace-B Glint360K wins on
|
|
calibration and on every held-out film.
|
|
- [Gallery scope (July)](gallery-scope-2026-07.md) — cast-restricted
|
|
gallery improves F1, recall, and misID at once.
|
|
- [Pose expansion (July)](pose-expansion-2026-07.md) — a training-set
|
|
effect that did not reproduce held-out.
|
|
- [LVFace deep dive (July)](lvface-deep-dive-2026-07.md) — the
|
|
generalization gap and every out-of-cast identification.
|
|
- [Full experiment log (July)](model-bakeoff-2026-07.md).
|
|
|
|
## Reproducing the benchmarks
|
|
|
|
Gallery `.h5` files, embedding dumps, the X-Ray corpus, montage frame
|
|
images, and DE trajectories are not committed to this repository. They are
|
|
pushed to the Gitea package registry and pulled on demand:
|
|
|
|
```bash
|
|
scripts/artifacts/pull_artifacts.sh galleries
|
|
scripts/artifacts/pull_artifacts.sh experiment-data
|
|
scripts/artifacts/pull_artifacts.sh montage-frames <film-slug>
|
|
```
|
|
|
|
See [`scripts/artifacts/push_artifacts.sh`](https://REPOLINK/scripts/artifacts/push_artifacts.sh)
|
|
for the upload side, which requires a `GITEA_TOKEN` with package write
|
|
scope.
|