dtourolle/scene-actor-extraction
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: opencv5 experiment log + rewritten Home

Browse Source 
Add model-bakeoff.md for the opencv5 build: the ten-knob DE tuning and where
each shipped config default comes from, the replay architecture, and the
flood-fill-on-learned-boundaries step change (62.6% -> 74.9% presence F1, LOO).
Rewrite index.md to lead with the learned scene-boundary result and point at
the current pages, with the July four-model bake-off moved to an Archive
section. Both pages build with no broken links.
This commit is contained in:
dtourolle
2026-08-10 08:41:54 +02:00
parent ef99951360
commit 7556c836da
2 changed files with 252 additions and 46 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/index.md
+54 -46
View File
@@ -17,62 +17,70 @@ two credited cast members without a visible face are correctly reported
present but not visible. This matches Amazon X-Ray's own record for this present but not visible. This matches Amazon X-Ray's own record for this
second exactly. second exactly.
Results are not uniform across films. The hardest held-out film scores 46% ## The headline: learned scene boundaries
F1. This report documents why: one tunable trade (extinction bridging at
hard cuts), one structural limit (X-Ray credits people whose faces never
appear on screen), and a small number of cases where the pipeline is
correct and X-Ray's ground truth is not. Read
[how we score against X-Ray](methodology.md) first. X-Ray's ground truth is
scene-level; the pipeline's output is per-second. That difference shapes
every finding below.
## Findings The current opencv5 build's biggest gain is **flood-fill presence on a
learned scene-boundary detector**. An actor seen once inside a shot is
reported for the whole shot — but only if the shot boundaries are good. A
learned XGBoost boundary detector, scored **leave-one-out** so no film is
ever measured by a detector that trained on it, lifts per-second X-Ray
presence F1 across nine films and improves every one of them:
<div class="grid cards" markdown> | boundary source for flood-fill | presence F1 |
| ------------------------------ | ----------: |
| track-extent (flood off) | 62.6% |
| flood + grayscale cuts | 64.0% |
| **flood + learned detector (LOO)** | **74.9%** |
- :material-trophy:{ .lg .middle } **[Which model is best?](best-model.md)** ![Macro presence F1 by flood-fill boundary source](assets/images/scene_presence_macro.png)
--- The full story — why the old grayscale cut detector broke Scarface, what
features work, and the per-film breakdown — is on the
[learned scene-boundary detector](scene-boundary-detector.md) page.
Calibration curves first, independent of any threshold, then held-out ## What the numbers mean, and their limits
F1 across three models. LVFace-B Glint360K wins both, and wins on every
held-out film.
- :material-filter:{ .lg .middle } **[Whole vs. cast-restricted gallery](gallery-scope.md)** Results are not uniform across films, and they should not be. X-Ray's ground
truth is scene-level and credits people whose faces never appear on screen;
the pipeline's output is per-second and can only name a face it can see.
That difference is a structural recall ceiling, not a bug. Read
[how we score against X-Ray](methodology.md) first — it defines F1,
precision, recall, and misID, and explains the two limits (off-screen cast
and gallery coverage) that shape every finding.
--- Precision on identified faces is near-perfect: where the pipeline names a
face, it is almost always a name X-Ray also credits to that scene. The
Restricting the matcher to a film's credited cast improves F1, frames throughout this documentation make the tension visual — **green** =
recall, and misID rate at once, but is not a shipped runtime feature true positive, **red** = false positive, **orange** = unknown, and a
yet. **blue** panel lists credited cast present with no visible face.
- :material-account-convert:{ .lg .middle } **[Does pose expansion help?](pose-expansion.md)**
---
A training-set effect that did not reproduce on 5 held-out films once
two methodology bugs in the comparison harness were found and fixed.
- :material-magnify-expand:{ .lg .middle } **[Deep dive: LVFace-B Glint360K](lvface-deep-dive.md)**
---
The held-out generalization gap, the two mechanisms behind its errors,
and every distinct case where it names someone outside the film's
credited cast.
</div>
## Full experiment log ## Full experiment log
- **[Full experiment log](model-bakeoff.md)**: the complete log behind the - **[Full experiment log (opencv5)](model-bakeoff.md)**: the complete log
four pages above, including how replaying against cached embeddings behind the current build — the ten-knob differential-evolution tuning, the
inside the same KPN network makes a full model and configuration shipped [`src/config.hpp`](https://REPOLINK/src/config.hpp) defaults and
comparison practical, the full results table, and every caveat. This is where each comes from, the replay architecture that makes a nine-film
where the shipped [`src/config.hpp`](https://REPOLINK/src/config.hpp) search tractable, and the flood-fill step change.
defaults come from. - **[Learned scene-boundary detector](scene-boundary-detector.md)**: the
- **[Service conversion (proposal)](service-conversion.md)**: design features, the model, leave-one-out results, and the two headline films.
sketch for a native idle-GPU worker gated on screen lock, not yet built. - **[Benchmark — SuperHero](benchmark.md)**: the benchmark harness.
- **[Service conversion (proposal)](service-conversion.md)**: design sketch
for a native idle-GPU worker gated on screen lock, not yet built.
## Archive (July 2026)
The pre-opencv5 four-model ArcFace/LVFace bake-off is kept for provenance.
Its numbers are historical; the current build supersedes them.
- [Best model (July)](best-model-2026-07.md) — LVFace-B Glint360K wins on
calibration and on every held-out film.
- [Gallery scope (July)](gallery-scope-2026-07.md) — cast-restricted
gallery improves F1, recall, and misID at once.
- [Pose expansion (July)](pose-expansion-2026-07.md) — a training-set
effect that did not reproduce held-out.
- [LVFace deep dive (July)](lvface-deep-dive-2026-07.md) — the
generalization gap and every out-of-cast identification.
- [Full experiment log (July)](model-bakeoff-2026-07.md).
## Reproducing the benchmarks ## Reproducing the benchmarks
docs/model-bakeoff.md
+198
View File
@@ -0,0 +1,198 @@
# Full experiment log (opencv5)
This is the complete log behind the current opencv5 build: how the pipeline is
tuned, what the shipped configuration is and where every number in it comes from,
and how the learned scene-boundary detector took per-second actor-presence F1 from
the low-60s to **74.9%** across the nine-film Amazon X-Ray benchmark — under honest
leave-one-out.
Read [How we score against X-Ray](methodology.md) first for what F1, precision,
recall, and misID mean here. Every number below uses the per-second metric
([`scripts/optimizer/second_score.py`](https://REPOLINK/scripts/optimizer/second_score.py)):
the film is sampled once per second, and at each second the set of names the
pipeline reports present is compared against Amazon X-Ray's scene cast for that
second. X-Ray's ground truth is scene-level; the pipeline's output is per-second.
That mismatch shapes every result.
## The benchmark
Nine films with public Amazon X-Ray scene data, all scored with the same
LVFace-B Glint360K gallery:
Benny & Joon · Café Society · Downton Abbey: A New Era · Lord of War · Lovelace ·
The Many Saints of Newark · Scarface · Sound of Metal · Valerian.
Two of these — Café Society and Scarface — are low-contrast, uniformly-graded
films that break naive cut detection. They are deliberately kept in the benchmark
because they are where the interesting failures live.
## Why replay makes this affordable
Decoding video and running face detection, alignment, and embedding is the
expensive part of the pipeline. Everything downstream — tracking, identity
matching, scene aggregation — is cheap. KPN++'s node/network structure keeps those
two halves as separate components joined by typed channels, so the expensive half
runs once per film and caches its output, and the cheap half can be re-run against
that cache as often as needed with different `Config` values.
`scene_analyze --dump-embeddings out.h5` runs the expensive half once and writes
per-frame detections, embeddings, and (for the scene detector) per-frame RGB
histograms to HDF5. [`scripts/optimizer/replay.py`](https://REPOLINK/scripts/optimizer/replay.py)
re-assembles the real C++ `face_tracker`, `identity_matcher`, and scene nodes into
a Python-driven KPN network and replays a film's cache through them, varying every
tuning knob freely. No GPU inference and no video decode happen during a replay, so
a full differential-evolution search over all nine films is tractable in one
session rather than requiring re-encodes.
Two concurrency limits are load-bearing and were paid for in wedged runs: replays
run at `DE_WORKERS=1` (concurrent DE candidates wedge the ROCm GPU), and each
candidate's per-film replays run at `REPLAY_WORKERS=8` with stderr discarded (the
replay sink's per-second prints otherwise flood the captured pipe and hang the
subprocess).
## The tuning knobs
The opencv5 refactor replaced the old three-knob search with a **ten-knob**
differential-evolution sweep. The knobs, and their shipped values:
| knob | shipped | what it controls |
| ---- | ------: | ---------------- |
| `prob_threshold` | 0.485 | posterior P(match) above which a track is named |
| `ownership_logodds` | 1.72 | log-odds a track needs before it produces presence |
| `track_extinction_sec` | 31.0 | how long an idle track is held for re-detection |
| `track_alpha` | 0.435 | tracker cost mix (0 = embedding only, 1 = spatial only) |
| `evidence_rho_max` | 0.204 | evidence weighting ceiling |
| `evidence_admit_below` | 0.784 | admit new evidence below this similarity |
| `match_prior` | 0.433 | base-rate prior on a match |
| `expand_band_lo` | 0.804 | low edge of the pose-expansion similarity band |
| `expand_band_hi` | 0.952 | high edge of the pose-expansion band |
| `presence_mode` | flood | track-extent vs scene flood-fill |
The DE run over the first nine knobs (flood off, track-extent presence) converged
at **64.0% macro F1** over 345 evaluations. Those values are the shipped
[`src/config.hpp`](https://REPOLINK/src/config.hpp) defaults.
![10-knob presence sweep (Differential Evolution)](assets/images/de_search_landscape.png)
The `track_extinction_sec` knob is worth calling out: at 31 s it holds an idle
track alive for re-detection long enough to bridge an actor turning away or leaving
frame briefly, without bridging across a genuine scene change. Getting this knob
and the tracker/registry to agree on **one clock** (the evidence watermark, not
wall-clock) was a correctness fix, not a tuning choice — before it, votes were
silently dropped at the reap horizon.
## The step change: flood-fill on learned boundaries
The 64.0% above is track-extent presence: an actor is reported only while an actual
track is alive. **Flood-fill** instead reports an actor for the whole shot once
they are seen in it — but that is only correct if the shot boundaries are good.
With the old grayscale cut detector as the boundary source, flood-fill barely beat
doing nothing (**64.0%**) and actively broke Scarface, where the detector fires
once in 10,204 frames and flood then smears every actor across the whole film
(precision collapses to 26%).
The [learned scene-boundary detector](scene-boundary-detector.md) — an XGBoost
regressor over histogram-delta and audio features, with a per-film knee threshold —
fixes this. Macro per-second presence F1, at the shipped presence config:
| boundary source for flood-fill | presence F1 |
| ------------------------------ | ----------: |
| track-extent (flood off) | 62.6% |
| flood + grayscale cuts | 64.0% |
| **flood + learned detector (LOO)** | **74.9%** |
![Macro presence F1 by flood-fill boundary source](assets/images/scene_presence_macro.png)
The learned column is **leave-one-out**: each film is scored by a detector trained
on the other eight, so no film's presence is ever measured with a detector that saw
it. That is the honest generalisation number, +12.3 points over track-extent, and
**it improves every one of the nine films**.
![Per-film presence F1 by boundary source](assets/images/scene_presence_by_source.png)
| film | track-extent | flood+grayscale | flood+learned (LOO) |
| ---- | -----------: | --------------: | ------------------: |
| Benny & Joon | 77.3 | 80.2 | 78.2 |
| Café Society | 59.1 | 62.2 | 69.8 |
| Downton Abbey | 41.0 | 51.8 | **78.6** |
| Lord of War | 74.8 | 77.1 | 77.8 |
| Lovelace | 70.3 | 74.0 | 78.2 |
| The Many Saints of Newark | 37.5 | 43.9 | 53.4 |
| Scarface | 62.6 | **40.9** | **74.9** |
| Sound of Metal | 75.0 | 78.1 | 86.8 |
| Valerian | 65.6 | 67.7 | 76.2 |
The two headline films — Scarface (grayscale flood *breaks* it, learned flood on a
film it never trained on takes it to 74.9%) and Downton Abbey (+37 points) — are
the strongest evidence the detector generalises. See the
[scene-boundary detector page](scene-boundary-detector.md) for the full story.
We re-ran the ten-knob DE on top of the good boundaries to check whether the
shipped config should change. It converged at 76.1% (+0.3 pp over the shipped
config on learned boundaries) — inside the noise, not worth re-shipping. The
boundaries, not the presence knobs, are where the win is.
## What the frames look like
`scripts/optimizer/dump_error_frames.py` pulls representative seconds and draws
each face box against X-Ray's scene cast: **green** = true positive, **red** =
false positive (a name X-Ray does not credit to this scene — the real error),
**orange** = an unknown detection. Cast X-Ray lists as present but for whom no face
was detected — the structural false-negatives a face pipeline can never box — are
listed as a **blue** panel.
![A correctly identified second: green true-positive boxes](assets/images/lovelace_perfect_second.jpg)
Every named frame in this documentation is regenerated against the current opencv5
pipeline by [`scripts/scene_detector/rematch_frames.py`](https://REPOLINK/scripts/scene_detector/rematch_frames.py),
which auto-matches each example by film, actor, and class (TP/FP) so the images
never drift from the shipped behaviour. Where the current pipeline no longer makes
a July-era error — the Zooey Deschanel misID in Many Saints is the clearest case —
the frame is dropped rather than staged, because the improvement is real.
## The structural recall ceiling
Precision against X-Ray is near-perfect on identified faces; recall is capped by
two things the pipeline cannot fix:
1. **X-Ray credits people whose faces never appear on screen** in a scene — voice,
back-of-head, or simply off-camera cast. No face pipeline can box a face that is
not there. These are the blue-panel names.
2. **Gallery coverage.** A large fraction of X-Ray cast has no reference image in
the gallery, so those actors can never be matched regardless of detection. This
is the dominant remaining recall limiter and is addressable by fetching more
reference photos, not by tuning.
Both are documented in [how we score against X-Ray](methodology.md).
## In the pipeline
The learned detector runs live inside `scene_analyze` as a post-EOF step (the
per-film knee needs every peak, so it can only run once the whole film is seen).
XGBoost inference is built into the binary via CMake (`SAE_SCENE_XGB`); the audio
log-PSD uses FFTW on the existing FFmpeg decode. The shipped model is trained on
the **C++-extracted** features so training and inference share one implementation.
Verified end to end through `scene_analyze` on a movie file and through the Jellyfin
work-queue worker.
```bash
scene_analyze --movie <file> --gallery <gallery.h5> \
--scene-xgb-model models/scene_boundary_xgb.json
```
## Reproducing the benchmarks
Gallery `.h5` files, embedding dumps, the X-Ray corpus, and DE trajectories are not
committed. They are pushed to the Gitea package registry and pulled on demand:
```bash
scripts/artifacts/pull_artifacts.sh galleries
scripts/artifacts/pull_artifacts.sh experiment-data
# per-second audio features, C++ feature matrices, train + downstream A/B
.venv-rocm/bin/python scripts/scene_detector/extract_audio_features.py \
--manifest experiments/manifests/films_LVFace_opencv5.json
.venv-rocm/bin/python scripts/scene_detector/train_xgb_cpp.py --train-all
scripts/scene_detector/downstream_presence.py
```