# scene-actor-extraction

A face-recognition pipeline that finds when each actor appears on screen in
a film or TV episode, built on [KPN++](https://gitea.tourolle.paris/dtourolle/KPN)
(a C++20 Kahn Process Network library) for the detect, track, match, and
scene pipeline, with a Jellyfin-integrated gallery and an X-Ray-validated
optimizer.

This is a correctly scored second from a held-out film, one the optimizer
never saw during tuning:

![A perfect X-Ray second: three faces named at 100%, two more correctly carried off-screen](assets/images/lovelace_perfect_second.jpg)

Every visible face is named at 100% confidence (Chris Noth, Hank Azaria,
Bobby Cannavale), the background extra is correctly left unnamed, and the
two credited cast members without a visible face are correctly reported
present but not visible. This matches Amazon X-Ray's own record for this
second exactly.

Results are not uniform across films. The hardest held-out film scores 46%
F1. This report documents why: one tunable trade (extinction bridging at
hard cuts), one structural limit (X-Ray credits people whose faces never
appear on screen), and a small number of cases where the pipeline is
correct and X-Ray's ground truth is not. Read
[how we score against X-Ray](methodology.md) first. X-Ray's ground truth is
scene-level; the pipeline's output is per-second. That difference shapes
every finding below.

## Findings

<div class="grid cards" markdown>

-   :material-trophy:{ .lg .middle } **[Which model is best?](best-model.md)**

    ---

    Calibration curves first, independent of any threshold, then held-out
    F1 across three models. LVFace-B Glint360K wins both, and wins on every
    held-out film.

-   :material-filter:{ .lg .middle } **[Whole vs. cast-restricted gallery](gallery-scope.md)**

    ---

    Restricting the matcher to a film's credited cast improves F1,
    recall, and misID rate at once, but is not a shipped runtime feature
    yet.

-   :material-account-convert:{ .lg .middle } **[Does pose expansion help?](pose-expansion.md)**

    ---

    A training-set effect that did not reproduce on 5 held-out films once
    two methodology bugs in the comparison harness were found and fixed.

-   :material-magnify-expand:{ .lg .middle } **[Deep dive: LVFace-B Glint360K](lvface-deep-dive.md)**

    ---

    The held-out generalization gap, the two mechanisms behind its errors,
    and every distinct case where it names someone outside the film's
    credited cast.

</div>

## Full experiment log

- **[Full experiment log](model-bakeoff.md)**: the complete log behind the
  four pages above, including how replaying against cached embeddings
  inside the same KPN network makes a full model and configuration
  comparison practical, the full results table, and every caveat. This is
  where the shipped [`src/config.hpp`](https://REPOLINK/src/config.hpp)
  defaults come from.
- **[Service conversion (proposal)](service-conversion.md)**: design
  sketch for a native idle-GPU worker gated on screen lock, not yet built.

## Reproducing the benchmarks

Gallery `.h5` files, embedding dumps, the X-Ray corpus, montage frame
images, and DE trajectories are not committed to this repository. They are
pushed to the Gitea package registry and pulled on demand:

```bash
scripts/artifacts/pull_artifacts.sh galleries
scripts/artifacts/pull_artifacts.sh experiment-data
scripts/artifacts/pull_artifacts.sh montage-frames <film-slug>
```

See [`scripts/artifacts/push_artifacts.sh`](https://REPOLINK/scripts/artifacts/push_artifacts.sh)
for the upload side, which requires a `GITEA_TOKEN` with package write
scope.