Skip to content

scene-actor-extraction

A face-recognition pipeline that finds when each actor appears on screen in a film or TV episode, built on KPN++ (a C++20 Kahn Process Network library) for the detect, track, match, and scene pipeline, with a Jellyfin-integrated gallery and an X-Ray-validated optimizer.

This is a correctly scored second from a held-out film, one the optimizer never saw during tuning:

A perfect X-Ray second: three faces named at 100%, two more correctly carried off-screen

Every visible face is named at 100% confidence (Chris Noth, Hank Azaria, Bobby Cannavale), the background extra is correctly left unnamed, and the two credited cast members without a visible face are correctly reported present but not visible. This matches Amazon X-Ray's own record for this second exactly.

Results are not uniform across films. The hardest held-out film scores 46% F1. This report documents why: one tunable trade (extinction bridging at hard cuts), one structural limit (X-Ray credits people whose faces never appear on screen), and a small number of cases where the pipeline is correct and X-Ray's ground truth is not. Read how we score against X-Ray first. X-Ray's ground truth is scene-level; the pipeline's output is per-second. That difference shapes every finding below.

Findings

  • Which model is best?

    Calibration curves first, independent of any threshold, then held-out F1 across three models. LVFace-B Glint360K wins both, and wins on every held-out film.

  • Whole vs. cast-restricted gallery

    Restricting the matcher to a film's credited cast improves F1, recall, and misID rate at once, but is not a shipped runtime feature yet.

  • Does pose expansion help?

    A training-set effect that did not reproduce on 5 held-out films once two methodology bugs in the comparison harness were found and fixed.

  • Deep dive: LVFace-B Glint360K

    The held-out generalization gap, the two mechanisms behind its errors, and every distinct case where it names someone outside the film's credited cast.

Full experiment log

  • Full experiment log: the complete log behind the four pages above, including how replaying against cached embeddings inside the same KPN network makes a full model and configuration comparison practical, the full results table, and every caveat. This is where the shipped src/config.hpp defaults come from.
  • Service conversion (proposal): design sketch for a native idle-GPU worker gated on screen lock, not yet built.

Reproducing the benchmarks

Gallery .h5 files, embedding dumps, the X-Ray corpus, montage frame images, and DE trajectories are not committed to this repository. They are pushed to the Gitea package registry and pulled on demand:

scripts/artifacts/pull_artifacts.sh galleries
scripts/artifacts/pull_artifacts.sh experiment-data
scripts/artifacts/pull_artifacts.sh montage-frames <film-slug>

See scripts/artifacts/push_artifacts.sh for the upload side, which requires a GITEA_TOKEN with package write scope.