docs: four focused findings pages (best model, gallery scope, expansion, deep dive)

Splits the rep4 write-up's key findings into their own linkable pages:
- best-model.md: calibration curves first (discriminative power, independent
  of any threshold), then F1 on the benchmark — LVFace-B Glint360K wins both.
- gallery-scope.md: whole vs. cast-restricted gallery, isolated from model and
  expansion choice — restriction wins on every axis, but isn't a shipped
  runtime feature yet.
- pose-expansion.md: the training-set expand_gallery effect, and the held-out
  replication attempt that found it doesn't reproduce (5 films, 2 models,
  after catching and fixing a replay-timeout truncation bug and a bbox
  first-match-instead-of-best-match bug in the comparison harness itself). An
  honest null result, with the methodology errors documented since they're
  exactly the kind that manufacture a false "it works!" finding.
- lvface-deep-dive.md: the winning model's held-out generalization gap, its
  two failure modes (frozen-bbox ghost tracks), and a verified case (cross-
  checked against Jellyfin's independent cast metadata) where LVFace
  correctly identified an actor that X-Ray's ground truth failed to credit.

Adds a "report-highlights" artifact-registry package (scripts/artifacts/
push_artifacts.sh, pull_artifacts.sh) for hand-picked illustrative frames that
aren't reproducible via the automated best/worst montage selection, and wires
pulling it into scripts/docs/build_site.sh.
This commit is contained in:
2026-07-19 19:40:19 +02:00
parent d340da755a
commit 4925443e56
9 changed files with 384 additions and 13 deletions
+97
View File
@@ -0,0 +1,97 @@
# Pose expansion: does "learning" new poses mid-film help?
`expand_gallery` (`src/gallery/track_gallery.hpp`) promotes a confidently-identified
track's novel-pose reference views into a per-film, in-memory gallery annex — the
idea being that once the pipeline is sure who someone is, a pose it hasn't seen
before (turned head, different lighting) becomes a free extra reference for
recognising that actor again later in the same film, without touching the baked
gallery.
## The training-set signal
Averaged across all 4 models, on the 4 films used for optimization:
| scope | expansion | F1 | R | misID |
|---|---|---|---|---|
| full | off | 71.2% | 58.3% | 209 |
| full | **on** | 71.2% | 59.7% | **864** |
| restricted | off | 73.6% | 61.3% | 194 |
| restricted | **on** | **75.4%** | **64.5%** | 135 |
In `restricted` mode (matcher's candidate set capped to the film's own credited
cast) expansion looked like a clean win: +1.8pp F1, +3.2pp recall, misID actually
lower. In `full` mode it looked flat-to-costly: ~0 F1 change, recall +1.4pp, but
misID roughly quadrupled (209 → 864) — see `rep4-optimizer-results.md` for the
per-model breakdown. That's the number that motivated this page: **does turning
expansion on actually change what gets recognised, frame by frame, or is the
aggregate F1 shift something else?**
## Held-out test: does it reproduce?
Same model + same tuned config, `expand_gallery` toggled on vs. off, nothing else
changed — full gallery mode, per-second scoring against X-Ray. This isolates
expansion from every other variable (config, model, threshold) that differs
between the training-set `exp`/`noexp` rows above.
**LVFace-B Glint360K, all 5 held-out films** (films never seen by the optimizer):
| film | F1 (exp) | F1 (noexp) | TPI Δ | FN Δ |
|---|---|---|---|---|
| Benny & Joon | 83.0% | 83.0% | -2 | +2 |
| Downton Abbey: A New Era | 56.1% | 56.2% | -7 | +7 |
| Lovelace | 77.5% | 77.4% | +33 | -33 |
| The Many Saints of Newark | 46.3% | 46.3% | +2 | -2 |
| Valerian and the City of a Thousand Planets | 74.1% | 74.1% | +2 | -2 |
**ArcFace R18** (Benny & Joon, r18's own tuned config): F1 77.1% for both, TPI/FN
identical, FPI differs by 2 (noise).
**Every film, both models tested: F1 within 0.10.2pp, TPI/FN swings in the tens
out of tens of thousands.** That's noise, not a signal — expansion made no
measurable difference to per-second onscreen identification anywhere it was
tested on unseen data.
## Two bugs this required catching (this section's own methodology)
Getting to the clean table above took two wrong turns, both worth recording
since they're exactly the kind of error that produces a false positive "look,
expansion helped!" finding:
1. **Timeout truncation.** The first Downton Abbey `exp` replay was cut off by a
60s subprocess timeout at ~76% through the film (5589 of 7368 expected
seconds) — a genuinely large, silent data loss that showed up as a large,
convincing-looking TPI gap (47938 vs 52032) purely because one run had a
quarter of the film missing. Caught by comparing `n_seconds` between runs
before trusting any score delta; fixed by re-running with a longer timeout.
2. **Bbox-matching bug.** An early per-second raw-annotation diff matched each
`exp` detection to the *first* `noexp` detection with IoU > 0.5, not the
*best*-overlapping one. With 3 faces close together in frame, this produced
spurious "disagreements" (e.g. "exp says Aidan Quinn, noexp says Johnny
Depp" at the same seconds) that vanished entirely once the match picked the
true best-IoU candidate — both configs had actually output the exact same
three names at the exact same three boxes.
Both bugs independently pointed toward "expansion is doing something," and both
were artifacts of the comparison harness, not the pipeline. Worth remembering
when a before/after diff looks dramatic: check that the two runs actually cover
the same seconds, and match entities by best overlap, not first-found.
## What this means
The training-set aggregate effect (particularly the ~4x misID increase in full
mode) doesn't reproduce on held-out data — at minimum it's far smaller than the
training-set numbers suggested, and plausibly it's sampling variation from only
4 training films rather than a real, generalizable mechanism. This doesn't mean
`expand_gallery` never does anything (the mechanism is real — see
`track_gallery.hpp`'s promotion logging: tracks *do* get confirmed and views *do*
get promoted into the annex on every film tested), only that **whatever effect
it has on final per-second identification was too small to detect against 5
held-out films** with this scoring method. A cleaner test would need either many
more held-out films or a metric that can see the annex's direct contribution
(e.g. tagging which reference embedding won each match), neither of which this
pass had budget for.
**Practical takeaway**: don't treat the training-set `exp` vs `noexp` numbers in
`rep4-optimizer-results.md` as proof that expansion changes real-world behavior
in either direction — on the evidence gathered so far, it doesn't move the
needle enough to see.