Skip to content

scene-actor-extraction

A face-recognition pipeline that finds when each actor appears on screen in a film or TV episode — built on KPN++ (a C++20 Kahn Process Network library) for the detect → track → match → scene pipeline, with a Jellyfin-integrated gallery and an X-Ray-validated optimizer.

This is a perfect X-Ray second, on a film the optimizer never saw:

A perfect X-Ray second: three faces named at 100%, two more correctly carried off-screen

Every visible face named at 100% — Chris Noth, Hank Azaria, Bobby Cannavale — the background extra honestly left unnamed, and the two credited cast without a visible face correctly carried as present off-screen by the tracker's presence windows. That's the pipeline exactly reproducing Amazon X-Ray's record for this second.

It doesn't always go like that: the hardest held-out film scores 46% F1, and the report is honest about why — one tunable trade (extinction bridging at hard cuts), one structural ceiling (X-Ray credits people whose faces never appear), and a few cases where the pipeline is right and X-Ray is wrong. The evidence for all of it is in the pages below.

Start here — four questions this bake-off answers

  • Which model is best?

    Calibration curves first (discriminative power, independent of any threshold), then F1 on the actual benchmark. LVFace-B Glint360K wins both.

  • Whole vs. cast-restricted gallery

    Restricting the matcher to a film's credited cast is a clean win on every axis (+3.3pp F1, less than a third the misIDs) — but isn't a shipped runtime feature yet.

  • Does pose expansion help?

    A convincing training-set effect that didn't reproduce on 5 held-out films once two methodology bugs were caught and fixed. An honest null result, not a forced narrative.

  • Deep dive: LVFace-B Glint360K

    The held-out generalization gap, how the error budget decomposes (extinction bridging at hard cuts, X-Ray's scene-membership vs. on-screen-face ceiling), and the frames where the pipeline is right and the ground truth is wrong.

The full technical log

  • Model bake-off + threshold re-tune — the complete experiment log behind the four pages above: the ROCm teardown deadlock root cause and fix, DE concurrency tuning, the full 16-combo results table, and every caveat. This is where the shipped src/config.hpp defaults come from.
  • Optimizer experiments (prior round) — the earlier scene-union-metric tuning pass, superseded by the per-second metric used in the bake-off but kept for the ground-truth/architecture background.
  • Service conversion (proposal) — design sketch for a native idle-GPU worker gated on screen lock, not yet built.

Reproducing the benchmarks

Gallery .h5 files, embedding dumps, the X-Ray corpus, montage frame images, and DE trajectories are not committed to this repository — they're pushed to the Gitea package registry and pulled on demand:

scripts/artifacts/pull_artifacts.sh galleries
scripts/artifacts/pull_artifacts.sh experiment-data
scripts/artifacts/pull_artifacts.sh montage-frames <film-slug>

See scripts/artifacts/push_artifacts.sh for the upload side (requires a GITEA_TOKEN with package write scope).