dtourolle/scene-actor-extraction
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: full data-grounded rewrite of the performance report

Browse Source 
Replaces narrative claims with verified numbers across all report pages:

- Cross-model held-out validation (LVFace/mbf/r18, all 5 held-out
  films): LVFace wins every film outright, not just "consistent with"
  the training-set pick. r50 dropped from the detailed comparison
  (gallery has ~30% fewer reference images per actor than the other
  three models on identical source photos).
- Per-film training breakdown: LVFace does not win every training
  film (mbf beats it on Lord of War); the 75.3% macro figure hides a
  10.7pp spread.
- Gallery coverage computed per film (20.3%-78.6%) instead of one
  flat 67%-missing average.
- Found and fixed a real scoring bug in optimize.py: a candidate
  whose hardest film's replay timed out was averaged over survivors
  instead of penalized, silently rewarding partial coverage. Affected
  3 of 16 training combos; corrected throughout, and optimize.py now
  scores an incomplete evaluation f1=0.0 instead of averaging over
  whichever films happened to finish.
- Every FPI frame in the deep dive now comes from the proper montage
  renderer (Onscreen/Offscreen panel, ghosts never drawn as boxes),
  never the bare-box debug overlay used earlier.
- Every distinct out-of-cast name across all 9 films gets its own
  frame at its first appearance (9 names, 4 films), not a
  single-example spot check: 2 ground-truth gaps, 1 photograph
  misread as a person, 6 genuine lookalike confusions.
- New methodology.md: the scene-level-vs-per-second scoring mismatch
  that the rest of the report assumes, written out once.
- Cut the deadlock/gdb debugging narrative from the experiment log;
  kept the one fact that matters (KPN's node/network split lets the
  expensive GPU stage run once and the cheap stage replay against
  cached embeddings).
- Plain declarative style throughout, no em dashes, no blog voice.
This commit is contained in:
dtourolle
2026-07-21 08:55:57 +02:00
parent 4b5557974b
commit 0bd2747069
18 changed files with 1824 additions and 890 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/index.md
+46 -44
View File
@@ -1,27 +1,32 @@
# scene-actor-extraction
A face-recognition pipeline that finds when each actor appears on screen in a
film or TV episode — built on [KPN++](https://gitea.tourolle.paris/dtourolle/KPN)
(a C++20 Kahn Process Network library) for the detect → track → match → scene
pipeline, with a Jellyfin-integrated gallery and an X-Ray-validated optimizer.
A face-recognition pipeline that finds when each actor appears on screen in
a film or TV episode, built on [KPN++](https://gitea.tourolle.paris/dtourolle/KPN)
(a C++20 Kahn Process Network library) for the detect, track, match, and
scene pipeline, with a Jellyfin-integrated gallery and an X-Ray-validated
optimizer.
This is a perfect X-Ray second, on a film the optimizer never saw:
This is a correctly scored second from a held-out film, one the optimizer
never saw during tuning:
![A perfect X-Ray second: three faces named at 100%, two more correctly carried off-screen](assets/images/lovelace_perfect_second.jpg)
Every visible face named at 100% — Chris Noth, Hank Azaria, Bobby Cannavale —
the background extra honestly left unnamed, and the two credited cast without
a visible face correctly carried as present off-screen by the tracker's
presence windows. That's the pipeline exactly reproducing Amazon X-Ray's
record for this second.
Every visible face is named at 100% confidence (Chris Noth, Hank Azaria,
Bobby Cannavale), the background extra is correctly left unnamed, and the
two credited cast members without a visible face are correctly reported
present but not visible. This matches Amazon X-Ray's own record for this
second exactly.
It doesn't always go like that: the hardest held-out film scores 46% F1, and
the report is honest about *why* — one tunable trade (extinction bridging at
hard cuts), one structural ceiling (X-Ray credits people whose faces never
appear), and a few cases where the pipeline is right and X-Ray is wrong. The
evidence for all of it is in the pages below.
Results are not uniform across films. The hardest held-out film scores 46%
F1. This report documents why: one tunable trade (extinction bridging at
hard cuts), one structural limit (X-Ray credits people whose faces never
appear on screen), and a small number of cases where the pipeline is
correct and X-Ray's ground truth is not. Read
[how we score against X-Ray](methodology.md) first. X-Ray's ground truth is
scene-level; the pipeline's output is per-second. That difference shapes
every finding below.
## Start here — four questions this bake-off answers
## Findings
<div class="grid cards" markdown>
@@ -29,55 +34,51 @@ evidence for all of it is in the pages below.
---
Calibration curves first (discriminative power, independent of any
threshold), then F1 on the actual benchmark. LVFace-B Glint360K wins
both.
Calibration curves first, independent of any threshold, then held-out
F1 across three models. LVFace-B Glint360K wins both, and wins on every
held-out film.
- :material-filter:{ .lg .middle } **[Whole vs. cast-restricted gallery](gallery-scope.md)**
---
Restricting the matcher to a film's credited cast is a clean win on
every axis (+3.3pp F1, less than a third the misIDs) — but isn't a
shipped runtime feature yet.
Restricting the matcher to a film's credited cast improves F1,
recall, and misID rate at once, but is not a shipped runtime feature
yet.
- :material-account-convert:{ .lg .middle } **[Does pose expansion help?](pose-expansion.md)**
---
A convincing training-set effect that didn't reproduce on 5 held-out
films once two methodology bugs were caught and fixed. An honest null
result, not a forced narrative.
A training-set effect that did not reproduce on 5 held-out films once
two methodology bugs in the comparison harness were found and fixed.
- :material-magnify-expand:{ .lg .middle } **[Deep dive: LVFace-B Glint360K](lvface-deep-dive.md)**
---
The held-out generalization gap, how the error budget decomposes
(extinction bridging at hard cuts, X-Ray's scene-membership vs.
on-screen-face ceiling), and the frames where the pipeline is right
and the ground truth is wrong.
The held-out generalization gap, the two mechanisms behind its errors,
and every distinct case where it names someone outside the film's
credited cast.
</div>
## The full technical log
## Full experiment log
- **[Model bake-off + threshold re-tune](model-bakeoff.md)** —
the complete experiment log behind the four pages above: the ROCm teardown
deadlock root cause and fix, DE concurrency tuning, the full 16-combo
results table, and every caveat. This is where the shipped
[`src/config.hpp`](https://REPOLINK/src/config.hpp) defaults come from.
- **[Optimizer experiments (prior round)](optimizer-experiments.md)** — the
earlier scene-union-metric tuning pass, superseded by the per-second metric
used in the bake-off but kept for the ground-truth/architecture background.
- **[Service conversion (proposal)](service-conversion.md)** — design sketch
for a native idle-GPU worker gated on screen lock, not yet built.
- **[Full experiment log](model-bakeoff.md)**: the complete log behind the
four pages above, including how replaying against cached embeddings
inside the same KPN network makes a full model and configuration
comparison practical, the full results table, and every caveat. This is
where the shipped [`src/config.hpp`](https://REPOLINK/src/config.hpp)
defaults come from.
- **[Service conversion (proposal)](service-conversion.md)**: design
sketch for a native idle-GPU worker gated on screen lock, not yet built.
## Reproducing the benchmarks
Gallery `.h5` files, embedding dumps, the X-Ray corpus, montage frame images,
and DE trajectories are not committed to this repository — they're pushed to
the Gitea package registry and pulled on demand:
Gallery `.h5` files, embedding dumps, the X-Ray corpus, montage frame
images, and DE trajectories are not committed to this repository. They are
pushed to the Gitea package registry and pulled on demand:
```bash
scripts/artifacts/pull_artifacts.sh galleries
@@ -86,4 +87,5 @@ scripts/artifacts/pull_artifacts.sh montage-frames <film-slug>
```
See [`scripts/artifacts/push_artifacts.sh`](https://REPOLINK/scripts/artifacts/push_artifacts.sh)
for the upload side (requires a `GITEA_TOKEN` with package write scope).
for the upload side, which requires a `GITEA_TOKEN` with package write
scope.