# scene-actor-extraction

A face-recognition pipeline that finds when each actor appears on screen in
a film or TV episode, built on [KPN++](https://gitea.tourolle.paris/dtourolle/KPN)
(a C++20 Kahn Process Network library) for the detect, track, match, and
scene pipeline, with a Jellyfin-integrated gallery and an X-Ray-validated
optimizer.

This is a correctly scored second from a held-out film, one the optimizer
never saw during tuning:

![A perfect X-Ray second: three faces named at 100%, two more correctly carried off-screen](assets/images/lovelace_perfect_second.jpg)

Every visible face is named at 100% confidence (Chris Noth, Hank Azaria,
Bobby Cannavale), the background extra is correctly left unnamed, and the
two credited cast members without a visible face are correctly reported
present but not visible. This matches Amazon X-Ray's own record for this
second exactly.

## The headline: learned scene boundaries

The current opencv5 build's biggest gain is **flood-fill presence on a
learned scene-boundary detector**. An actor seen once inside a shot is
reported for the whole shot — but only if the shot boundaries are good. A
learned XGBoost boundary detector, scored **leave-one-out** so no film is
ever measured by a detector that trained on it, lifts per-second X-Ray
presence F1 across nine films and improves every one of them:

| boundary source for flood-fill | presence F1 |
| ------------------------------ | ----------: |
| track-extent (flood off) | 62.6% |
| flood + grayscale cuts | 64.0% |
| **flood + learned detector (LOO)** | **74.9%** |

![Macro presence F1 by flood-fill boundary source](assets/images/scene_presence_macro.png)

The full story — why the old grayscale cut detector broke Scarface, what
features work, and the per-film breakdown — is on the
[learned scene-boundary detector](scene-boundary-detector.md) page.

## What the numbers mean, and their limits

Results are not uniform across films, and they should not be. X-Ray's ground
truth is scene-level and credits people whose faces never appear on screen;
the pipeline's output is per-second and can only name a face it can see.
That difference is a structural recall ceiling, not a bug. Read
[how we score against X-Ray](methodology.md) first — it defines F1,
precision, recall, and misID, and explains the two limits (off-screen cast
and gallery coverage) that shape every finding.

Precision on identified faces is near-perfect: where the pipeline names a
face, it is almost always a name X-Ray also credits to that scene. The
frames throughout this documentation make the tension visual — **green** =
true positive, **red** = false positive, **orange** = unknown, and a
**blue** panel lists credited cast present with no visible face.

## Full experiment log

- **[Full experiment log (opencv5)](model-bakeoff.md)**: the complete log
  behind the current build — the ten-knob differential-evolution tuning, the
  shipped [`src/config.hpp`](https://REPOLINK/src/config.hpp) defaults and
  where each comes from, the replay architecture that makes a nine-film
  search tractable, and the flood-fill step change.
- **[Learned scene-boundary detector](scene-boundary-detector.md)**: the
  features, the model, leave-one-out results, and the two headline films.
- **[Benchmark — SuperHero](benchmark.md)**: the benchmark harness.
- **[Service conversion (proposal)](service-conversion.md)**: design sketch
  for a native idle-GPU worker gated on screen lock, not yet built.

## Archive (July 2026)

The pre-opencv5 four-model ArcFace/LVFace bake-off is kept for provenance.
Its numbers are historical; the current build supersedes them.

- [Best model (July)](best-model-2026-07.md) — LVFace-B Glint360K wins on
  calibration and on every held-out film.
- [Gallery scope (July)](gallery-scope-2026-07.md) — cast-restricted
  gallery improves F1, recall, and misID at once.
- [Pose expansion (July)](pose-expansion-2026-07.md) — a training-set
  effect that did not reproduce held-out.
- [LVFace deep dive (July)](lvface-deep-dive-2026-07.md) — the
  generalization gap and every out-of-cast identification.
- [Full experiment log (July)](model-bakeoff-2026-07.md).

## Reproducing the benchmarks

Gallery `.h5` files, embedding dumps, the X-Ray corpus, montage frame
images, and DE trajectories are not committed to this repository. They are
pushed to the Gitea package registry and pulled on demand:

```bash
scripts/artifacts/pull_artifacts.sh galleries
scripts/artifacts/pull_artifacts.sh experiment-data
scripts/artifacts/pull_artifacts.sh montage-frames <film-slug>
```

See [`scripts/artifacts/push_artifacts.sh`](https://REPOLINK/scripts/artifacts/push_artifacts.sh)
for the upload side, which requires a `GITEA_TOKEN` with package write
scope.