Skip to content

scene-actor-extraction

A face-recognition pipeline that finds when each actor appears on screen in a film or TV episode — built on KPN++ (a C++20 Kahn Process Network library) for the detect → track → match → scene pipeline, with a Jellyfin-integrated gallery and an X-Ray-validated optimizer.

Start here — four questions this bake-off answers

  • Which model is best? — calibration curves first (discriminative power, independent of any threshold), then F1 on the actual benchmark. LVFace-B Glint360K wins both.
  • Whole gallery vs. limited (cast-restricted) gallery — restricting the matcher to a film's credited cast is a clean win on every axis (+3.3pp F1, less than a third the misIDs), but isn't a shipped runtime feature yet.
  • Does pose expansion help? — a real training-set effect that didn't reproduce on 5 held-out films once two methodology bugs were caught and fixed. An honest null result, not a forced narrative.
  • Deep dive: LVFace-B Glint360K — the winning model's held-out generalization gap, its two real failure modes (frozen-bbox "ghost tracks"), and one case where it correctly identified an actor that the X-Ray ground truth itself failed to credit.

The full technical log

  • Rep4 model bake-off + threshold re-tune — the complete experiment log behind the four pages above: the ROCm teardown deadlock root cause and fix, DE concurrency tuning, the full 16-combo results table, and every caveat. This is where the shipped src/config.hpp defaults come from.
  • Optimizer experiments (prior round) — the earlier scene-union-metric tuning pass, superseded by the per-second metric used in rep4 but kept for the ground-truth/architecture background.
  • Service conversion (proposal) — design sketch for an idle-GPU Docker worker, not yet built.

Reproducing the benchmarks

Gallery .h5 files, embedding dumps, the X-Ray corpus, montage frame images, and DE trajectories are not committed to this repository — they're pushed to the Gitea package registry and pulled on demand:

scripts/artifacts/pull_artifacts.sh galleries
scripts/artifacts/pull_artifacts.sh experiment-data
scripts/artifacts/pull_artifacts.sh montage-frames <film-slug>

See scripts/artifacts/push_artifacts.sh for the upload side (requires a GITEA_TOKEN with package write scope).