scene-actor-extraction/docs/index.md

scene-actor-extraction

A face-recognition pipeline that finds when each actor appears on screen in a film or TV episode, built on KPN++ (a C++20 Kahn Process Network library) for the detect, track, match, and scene pipeline, with a Jellyfin-integrated gallery and an X-Ray-validated optimizer.

This is a correctly scored second from a held-out film, one the optimizer never saw during tuning:

Every visible face is named at 100% confidence (Chris Noth, Hank Azaria, Bobby Cannavale), the background extra is correctly left unnamed, and the two credited cast members without a visible face are correctly reported present but not visible. This matches Amazon X-Ray's own record for this second exactly.

The headline: learned scene boundaries

The current opencv5 build's biggest gain is flood-fill presence on a learned scene-boundary detector. An actor seen once inside a shot is reported for the whole shot — but only if the shot boundaries are good. A learned XGBoost boundary detector, scored leave-one-out so no film is ever measured by a detector that trained on it, lifts per-second X-Ray presence F1 across nine films and improves every one of them:

boundary source for flood-fill	presence F1
track-extent (flood off)	62.6%
flood + grayscale cuts	64.0%
flood + learned detector (LOO)	74.9%

The full story — why the old grayscale cut detector broke Scarface, what features work, and the per-film breakdown — is on the learned scene-boundary detector page.

What the numbers mean, and their limits

Results are not uniform across films, and they should not be. X-Ray's ground truth is scene-level and credits people whose faces never appear on screen; the pipeline's output is per-second and can only name a face it can see. That difference is a structural recall ceiling, not a bug. Read how we score against X-Ray first — it defines F1, precision, recall, and misID, and explains the two limits (off-screen cast and gallery coverage) that shape every finding.

Precision on identified faces is near-perfect: where the pipeline names a face, it is almost always a name X-Ray also credits to that scene. The frames throughout this documentation make the tension visual — green = true positive, red = false positive, orange = unknown, and a blue panel lists credited cast present with no visible face.

Full experiment log

• Full experiment log (opencv5): the complete log behind the current build — the ten-knob differential-evolution tuning, the shipped src/config.hpp defaults and where each comes from, the replay architecture that makes a nine-film search tractable, and the flood-fill step change.
• Learned scene-boundary detector: the features, the model, leave-one-out results, and the two headline films.
• Benchmark — SuperHero: the benchmark harness.
• Service conversion (proposal): design sketch for a native idle-GPU worker gated on screen lock, not yet built.

Archive (July 2026)

The pre-opencv5 four-model ArcFace/LVFace bake-off is kept for provenance. Its numbers are historical; the current build supersedes them.

• Best model (July) — LVFace-B Glint360K wins on calibration and on every held-out film.
• Gallery scope (July) — cast-restricted gallery improves F1, recall, and misID at once.
• Pose expansion (July) — a training-set effect that did not reproduce held-out.
• LVFace deep dive (July) — the generalization gap and every out-of-cast identification.
• Full experiment log (July).

Reproducing the benchmarks

Gallery .h5 files, embedding dumps, the X-Ray corpus, montage frame images, and DE trajectories are not committed to this repository. They are pushed to the Gitea package registry and pulled on demand:

scripts/artifacts/pull_artifacts.sh galleries
scripts/artifacts/pull_artifacts.sh experiment-data
scripts/artifacts/pull_artifacts.sh montage-frames <film-slug>

See scripts/artifacts/push_artifacts.sh for the upload side, which requires a GITEA_TOKEN with package write scope.