Files

T

dtourolle 0bd2747069 docs: full data-grounded rewrite of the performance report

Replaces narrative claims with verified numbers across all report pages:

- Cross-model held-out validation (LVFace/mbf/r18, all 5 held-out
  films): LVFace wins every film outright, not just "consistent with"
  the training-set pick. r50 dropped from the detailed comparison
  (gallery has ~30% fewer reference images per actor than the other
  three models on identical source photos).
- Per-film training breakdown: LVFace does not win every training
  film (mbf beats it on Lord of War); the 75.3% macro figure hides a
  10.7pp spread.
- Gallery coverage computed per film (20.3%-78.6%) instead of one
  flat 67%-missing average.
- Found and fixed a real scoring bug in optimize.py: a candidate
  whose hardest film's replay timed out was averaged over survivors
  instead of penalized, silently rewarding partial coverage. Affected
  3 of 16 training combos; corrected throughout, and optimize.py now
  scores an incomplete evaluation f1=0.0 instead of averaging over
  whichever films happened to finish.
- Every FPI frame in the deep dive now comes from the proper montage
  renderer (Onscreen/Offscreen panel, ghosts never drawn as boxes),
  never the bare-box debug overlay used earlier.
- Every distinct out-of-cast name across all 9 films gets its own
  frame at its first appearance (9 names, 4 films), not a
  single-example spot check: 2 ground-truth gaps, 1 photograph
  misread as a person, 6 genuine lookalike confusions.
- New methodology.md: the scene-level-vs-per-second scoring mismatch
  that the rest of the report assumes, written out once.
- Cut the deadlock/gdb debugging narrative from the experiment log;
  kept the one fact that matters (KPN's node/network split lets the
  expensive GPU stage run once and the cheap stage replay against
  cached embeddings).
- Plain declarative style throughout, no em dashes, no blog voice.

2026-07-21 08:55:57 +02:00

5.5 KiB

Raw Permalink Blame History

Pose expansion: does promoting new poses mid-film help?

expand_gallery (src/gallery/track_gallery.hpp) promotes a confidently identified track's novel-pose reference views into a per-film, in-memory gallery annex. The idea: once the pipeline is confident about an identity, a pose it has not seen before (turned head, different lighting) becomes an extra reference for recognizing that actor again later in the same film, without touching the baked gallery.

Training-set signal

Averaged across the 3 compared models (r50 excluded), on the 4 films used for optimization. These are the corrected, full-coverage figures, see the dropped-film note in the experiment log for why an earlier version of this table overstated the full-mode misID jump (209 → 864) that was itself partly a truncation artifact:

scope	expansion	F1	R	misID
full	off	70.0%	57.6%	407
full	on	72.1%	61.5%	714
restricted	off	75.1%	63.9%	179
restricted	on	76.7%	67.2%	120

In restricted mode, expansion looks like a clean win: +1.6pp F1, +3.3pp recall, lower misID. In full mode it looks like a recall-for-misID trade: +2.1pp F1, +3.9pp recall, but misID rises from 407 to 714. See the full experiment log for the per-model breakdown. This asymmetry motivated the question below: does turning expansion on change what gets recognized frame by frame, or is the aggregate F1 shift coming from something else.

Held-out test

Same model, same tuned config, expand_gallery toggled on vs. off, nothing else changed, full gallery mode, per-second scoring against X-Ray. This isolates expansion from every other variable that differs between the training-set rows above.

LVFace-B Glint360K, all 5 held-out films:

film	F1 (exp)	F1 (noexp)	TPI delta	FN delta
Benny & Joon	83.0%	83.0%	-2	+2
Downton Abbey: A New Era	56.1%	56.2%	-7	+7
Lovelace	77.5%	77.4%	+33	-33
The Many Saints of Newark	46.3%	46.3%	+2	-2
Valerian and the City of a Thousand Planets	74.1%	74.1%	+2	-2

ArcFace R18, Benny & Joon, r18's own tuned config: F1 77.1% for both, TPI and FN identical, FPI differs by 2.

Every film, both models tested: F1 differs by 0.1-0.2pp, TPI/FN swings are in the tens out of tens of thousands. This is noise, not a signal. Expansion made no measurable difference to per-second on-screen identification on any held-out film tested.

Two methodology bugs caught during this check

Getting to the table above required catching two wrong turns, both worth recording because they are exactly the kind of error that produces a false positive "expansion helped" finding.

Timeout truncation. The first Downton Abbey exp replay was cut off by a 60-second subprocess timeout at about 76% through the film (5589 of 7368 expected seconds). This silent data loss produced a large, convincing-looking TPI gap (47938 vs 52032) purely because one run was missing a quarter of the film. Caught by comparing n_seconds between runs before trusting any score delta; fixed by re-running with a longer timeout.
Bbox-matching bug. An early per-second raw-annotation diff matched each exp detection to the first noexp detection with IoU above 0.5, not the best-overlapping one. With 3 faces close together in frame, this produced spurious disagreements (for example "exp says Aidan Quinn, noexp says Johnny Depp" at the same second) that vanished once the match used the best-IoU candidate instead of the first one. Both configs had actually output the same three names at the same three boxes.

Both bugs independently pointed toward "expansion is doing something," and both were artifacts of the comparison harness, not the pipeline. Before trusting a dramatic before/after diff, check that both runs cover the same seconds and that entities are matched by best overlap, not first found.

Conclusion

The training-set aggregate effect, particularly the full-mode misID increase, does not reproduce on held-out data. At minimum it is far smaller than the training-set numbers suggested; it may be sampling variation from only 4 training films rather than a generalizable mechanism. Note the same class of harness bug appears twice in this investigation, the timeout truncation in bug #1 above, and the dropped-film aggregation that inflated the raw training-set misID figures. Both make an inert config look consequential; both are reasons to distrust a dramatic training-set delta until it survives on held-out films, which this one did not. This does not mean expand_gallery never does anything: the mechanism is real, and track_gallery.hpp's promotion logging confirms tracks get confirmed and views get promoted into the annex on every film tested. It means whatever effect expansion has on final per-second identification was too small to detect against 5 held-out films with this scoring method. A cleaner test would need either more held-out films or a metric that can see the annex's direct contribution, such as tagging which reference embedding won each match; neither was in scope for this pass.

Do not treat the training-set exp/noexp numbers in the full experiment log as proof that expansion changes real-world behavior in either direction. On the evidence gathered so far, it does not move the needle enough to see.

5.5 KiB Raw Permalink Blame History