dtourolle/scene-actor-extraction
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: full data-grounded rewrite of the performance report

Browse Source 
Replaces narrative claims with verified numbers across all report pages:

- Cross-model held-out validation (LVFace/mbf/r18, all 5 held-out
  films): LVFace wins every film outright, not just "consistent with"
  the training-set pick. r50 dropped from the detailed comparison
  (gallery has ~30% fewer reference images per actor than the other
  three models on identical source photos).
- Per-film training breakdown: LVFace does not win every training
  film (mbf beats it on Lord of War); the 75.3% macro figure hides a
  10.7pp spread.
- Gallery coverage computed per film (20.3%-78.6%) instead of one
  flat 67%-missing average.
- Found and fixed a real scoring bug in optimize.py: a candidate
  whose hardest film's replay timed out was averaged over survivors
  instead of penalized, silently rewarding partial coverage. Affected
  3 of 16 training combos; corrected throughout, and optimize.py now
  scores an incomplete evaluation f1=0.0 instead of averaging over
  whichever films happened to finish.
- Every FPI frame in the deep dive now comes from the proper montage
  renderer (Onscreen/Offscreen panel, ghosts never drawn as boxes),
  never the bare-box debug overlay used earlier.
- Every distinct out-of-cast name across all 9 films gets its own
  frame at its first appearance (9 names, 4 films), not a
  single-example spot check: 2 ground-truth gaps, 1 photograph
  misread as a person, 6 genuine lookalike confusions.
- New methodology.md: the scene-level-vs-per-second scoring mismatch
  that the rest of the report assumes, written out once.
- Cut the deadlock/gdb debugging narrative from the experiment log;
  kept the one fact that matters (KPN's node/network split lets the
  expensive GPU stage run once and the cheap stage replay against
  cached embeddings).
- Plain declarative style throughout, no em dashes, no blog voice.
This commit is contained in:
dtourolle
2026-07-21 08:55:57 +02:00
parent 4b5557974b
commit 0bd2747069
18 changed files with 1824 additions and 890 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/pose-expansion.md
+83 -73
View File
@@ -1,42 +1,46 @@
# Pose expansion: does "learning" new poses mid-film help?
# Pose expansion: does promoting new poses mid-film help?
`expand_gallery` ([`src/gallery/track_gallery.hpp`](https://REPOLINK/src/gallery/track_gallery.hpp))
promotes a confidently-identified
track's novel-pose reference views into a per-film, in-memory gallery annex — the
idea being that once the pipeline is sure who someone is, a pose it hasn't seen
before (turned head, different lighting) becomes a free extra reference for
recognising that actor again later in the same film, without touching the baked
gallery.
`expand_gallery`
([`src/gallery/track_gallery.hpp`](https://REPOLINK/src/gallery/track_gallery.hpp))
promotes a confidently identified track's novel-pose reference views into a
per-film, in-memory gallery annex. The idea: once the pipeline is confident
about an identity, a pose it has not seen before (turned head, different
lighting) becomes an extra reference for recognizing that actor again later
in the same film, without touching the baked gallery.
## The training-set signal
## Training-set signal
Averaged across all 4 models, on the 4 films used for optimization:
Averaged across the 3 compared models (r50 excluded), on the 4 films used
for optimization. These are the corrected, full-coverage figures, see the
[dropped-film note](model-bakeoff.md#a-scoring-bug-worth-recording-dropped-film-evaluations)
in the experiment log for why an earlier version of this table overstated the
full-mode misID jump (209 → 864) that was itself partly a truncation artifact:
| scope | expansion | F1 | R | misID |
|---|---|---|---|---|
| full | off | 71.2% | 58.3% | 209 |
| full | **on** | 71.2% | 59.7% | **864** |
| restricted | off | 73.6% | 61.3% | 194 |
| restricted | **on** | **75.4%** | **64.5%** | 135 |
| full | off | 70.0% | 57.6% | 407 |
| full | on | 72.1% | 61.5% | 714 |
| restricted | off | 75.1% | 63.9% | 179 |
| restricted | on | 76.7% | 67.2% | 120 |
In `restricted` mode (matcher's candidate set capped to the film's own credited
cast) expansion looked like a clean win: +1.8pp F1, +3.2pp recall, misID actually
lower. In `full` mode it looked flat-to-costly: ~0 F1 change, recall +1.4pp, but
misID roughly quadrupled (209 → 864) — see the
[bake-off experiment log](model-bakeoff.md) for the per-model breakdown. That's the number that motivated this page: **does turning
expansion on actually change what gets recognised, frame by frame, or is the
aggregate F1 shift something else?**
In restricted mode, expansion looks like a clean win: +1.6pp F1, +3.3pp
recall, lower misID. In full mode it looks like a recall-for-misID trade:
+2.1pp F1, +3.9pp recall, but misID rises from 407 to 714. See
[the full experiment log](model-bakeoff.md) for the per-model breakdown.
This asymmetry motivated the question below: does turning expansion on
change what gets recognized frame by frame, or is the aggregate F1 shift
coming from something else.
## Held-out test: does it reproduce?
## Held-out test
Same model + same tuned config, `expand_gallery` toggled on vs. off, nothing else
changed — full gallery mode, per-second scoring against X-Ray. This isolates
expansion from every other variable (config, model, threshold) that differs
between the training-set `exp`/`noexp` rows above.
Same model, same tuned config, `expand_gallery` toggled on vs. off, nothing
else changed, full gallery mode, per-second scoring against X-Ray. This
isolates expansion from every other variable that differs between the
training-set rows above.
**LVFace-B Glint360K, all 5 held-out films** (films never seen by the optimizer):
LVFace-B Glint360K, all 5 held-out films:
| film | F1 (exp) | F1 (noexp) | TPI Δ | FN Δ |
| film | F1 (exp) | F1 (noexp) | TPI delta | FN delta |
|---|---|---|---|---|
| Benny & Joon | 83.0% | 83.0% | -2 | +2 |
| Downton Abbey: A New Era | 56.1% | 56.2% | -7 | +7 |
@@ -44,57 +48,63 @@ between the training-set `exp`/`noexp` rows above.
| The Many Saints of Newark | 46.3% | 46.3% | +2 | -2 |
| Valerian and the City of a Thousand Planets | 74.1% | 74.1% | +2 | -2 |
**ArcFace R18** (Benny & Joon, r18's own tuned config): F1 77.1% for both, TPI/FN
identical, FPI differs by 2 (noise).
ArcFace R18, Benny & Joon, r18's own tuned config: F1 77.1% for both, TPI
and FN identical, FPI differs by 2.
**Every film, both models tested: F1 within 0.1–0.2pp, TPI/FN swings in the tens
out of tens of thousands.** That's noise, not a signal — expansion made no
measurable difference to per-second onscreen identification anywhere it was
tested on unseen data.
Every film, both models tested: F1 differs by 0.1-0.2pp, TPI/FN swings are
in the tens out of tens of thousands. This is noise, not a signal.
Expansion made no measurable difference to per-second on-screen
identification on any held-out film tested.
## Two bugs this required catching (this section's own methodology)
## Two methodology bugs caught during this check
Getting to the clean table above took two wrong turns, both worth recording
since they're exactly the kind of error that produces a false positive "look,
expansion helped!" finding:
Getting to the table above required catching two wrong turns, both worth
recording because they are exactly the kind of error that produces a false
positive "expansion helped" finding.
1. **Timeout truncation.** The first Downton Abbey `exp` replay was cut off by a
60s subprocess timeout at ~76% through the film (5589 of 7368 expected
seconds) — a genuinely large, silent data loss that showed up as a large,
convincing-looking TPI gap (47938 vs 52032) purely because one run had a
quarter of the film missing. Caught by comparing `n_seconds` between runs
before trusting any score delta; fixed by re-running with a longer timeout.
2. **Bbox-matching bug.** An early per-second raw-annotation diff matched each
`exp` detection to the *first* `noexp` detection with IoU > 0.5, not the
*best*-overlapping one. With 3 faces close together in frame, this produced
spurious "disagreements" (e.g. "exp says Aidan Quinn, noexp says Johnny
Depp" at the same seconds) that vanished entirely once the match picked the
true best-IoU candidate — both configs had actually output the exact same
three names at the exact same three boxes.
1. **Timeout truncation.** The first Downton Abbey `exp` replay was cut off
by a 60-second subprocess timeout at about 76% through the film (5589 of
7368 expected seconds). This silent data loss produced a large,
convincing-looking TPI gap (47938 vs 52032) purely because one run was
missing a quarter of the film. Caught by comparing `n_seconds` between
runs before trusting any score delta; fixed by re-running with a longer
timeout.
2. **Bbox-matching bug.** An early per-second raw-annotation diff matched
each `exp` detection to the first `noexp` detection with IoU above 0.5,
not the best-overlapping one. With 3 faces close together in frame, this
produced spurious disagreements (for example "exp says Aidan Quinn,
noexp says Johnny Depp" at the same second) that vanished once the match
used the best-IoU candidate instead of the first one. Both configs had
actually output the same three names at the same three boxes.
Both bugs independently pointed toward "expansion is doing something," and both
were artifacts of the comparison harness, not the pipeline. Worth remembering
when a before/after diff looks dramatic: check that the two runs actually cover
the same seconds, and match entities by best overlap, not first-found.
Both bugs independently pointed toward "expansion is doing something," and
both were artifacts of the comparison harness, not the pipeline. Before
trusting a dramatic before/after diff, check that both runs cover the same
seconds and that entities are matched by best overlap, not first found.
## What this means
## Conclusion
The training-set aggregate effect (particularly the ~4x misID increase in full
mode) doesn't reproduce on held-out data — at minimum it's far smaller than the
training-set numbers suggested, and plausibly it's sampling variation from only
4 training films rather than a real, generalizable mechanism. This doesn't mean
`expand_gallery` never does anything (the mechanism is real — see
The training-set aggregate effect, particularly the full-mode misID
increase, does not reproduce on held-out data. At minimum it
is far smaller than the training-set numbers suggested; it may be sampling
variation from only 4 training films rather than a generalizable
mechanism. Note the same *class* of harness bug appears twice in this
investigation, the timeout truncation in bug #1 above, and the dropped-film
aggregation that inflated the raw training-set misID figures. Both make an
inert config look consequential; both are reasons to distrust a dramatic
training-set delta until it survives on held-out films, which this one did
not. This does not mean `expand_gallery` never does anything: the
mechanism is real, and
[`track_gallery.hpp`](https://REPOLINK/src/gallery/track_gallery.hpp)'s
promotion logging: tracks *do* get confirmed and views *do*
get promoted into the annex on every film tested), only that **whatever effect
it has on final per-second identification was too small to detect against 5
held-out films** with this scoring method. A cleaner test would need either many
more held-out films or a metric that can see the annex's direct contribution
(e.g. tagging which reference embedding won each match), neither of which this
pass had budget for.
promotion logging confirms tracks get confirmed and views get promoted
into the annex on every film tested. It means whatever effect expansion
has on final per-second identification was too small to detect against 5
held-out films with this scoring method. A cleaner test would need either
more held-out films or a metric that can see the annex's direct
contribution, such as tagging which reference embedding won each match;
neither was in scope for this pass.
**Practical takeaway**: don't treat the training-set `exp` vs `noexp` numbers in
the [bake-off experiment log](model-bakeoff.md) as proof that expansion
changes real-world behavior
in either direction — on the evidence gathered so far, it doesn't move the
needle enough to see.
Do not treat the training-set exp/noexp numbers in
[the full experiment log](model-bakeoff.md) as proof that expansion changes
real-world behavior in either direction. On the evidence gathered so far,
it does not move the needle enough to see.