docs: four focused findings pages (best model, gallery scope, expansion, deep dive)
Splits the rep4 write-up's key findings into their own linkable pages: - best-model.md: calibration curves first (discriminative power, independent of any threshold), then F1 on the benchmark — LVFace-B Glint360K wins both. - gallery-scope.md: whole vs. cast-restricted gallery, isolated from model and expansion choice — restriction wins on every axis, but isn't a shipped runtime feature yet. - pose-expansion.md: the training-set expand_gallery effect, and the held-out replication attempt that found it doesn't reproduce (5 films, 2 models, after catching and fixing a replay-timeout truncation bug and a bbox first-match-instead-of-best-match bug in the comparison harness itself). An honest null result, with the methodology errors documented since they're exactly the kind that manufacture a false "it works!" finding. - lvface-deep-dive.md: the winning model's held-out generalization gap, its two failure modes (frozen-bbox ghost tracks), and a verified case (cross- checked against Jellyfin's independent cast metadata) where LVFace correctly identified an actor that X-Ray's ground truth failed to credit. Adds a "report-highlights" artifact-registry package (scripts/artifacts/ push_artifacts.sh, pull_artifacts.sh) for hand-picked illustrative frames that aren't reproducible via the automated best/worst montage selection, and wires pulling it into scripts/docs/build_site.sh.
This commit is contained in:
@@ -0,0 +1,97 @@
|
||||
# Pose expansion: does "learning" new poses mid-film help?
|
||||
|
||||
`expand_gallery` (`src/gallery/track_gallery.hpp`) promotes a confidently-identified
|
||||
track's novel-pose reference views into a per-film, in-memory gallery annex — the
|
||||
idea being that once the pipeline is sure who someone is, a pose it hasn't seen
|
||||
before (turned head, different lighting) becomes a free extra reference for
|
||||
recognising that actor again later in the same film, without touching the baked
|
||||
gallery.
|
||||
|
||||
## The training-set signal
|
||||
|
||||
Averaged across all 4 models, on the 4 films used for optimization:
|
||||
|
||||
| scope | expansion | F1 | R | misID |
|
||||
|---|---|---|---|---|
|
||||
| full | off | 71.2% | 58.3% | 209 |
|
||||
| full | **on** | 71.2% | 59.7% | **864** |
|
||||
| restricted | off | 73.6% | 61.3% | 194 |
|
||||
| restricted | **on** | **75.4%** | **64.5%** | 135 |
|
||||
|
||||
In `restricted` mode (matcher's candidate set capped to the film's own credited
|
||||
cast) expansion looked like a clean win: +1.8pp F1, +3.2pp recall, misID actually
|
||||
lower. In `full` mode it looked flat-to-costly: ~0 F1 change, recall +1.4pp, but
|
||||
misID roughly quadrupled (209 → 864) — see `rep4-optimizer-results.md` for the
|
||||
per-model breakdown. That's the number that motivated this page: **does turning
|
||||
expansion on actually change what gets recognised, frame by frame, or is the
|
||||
aggregate F1 shift something else?**
|
||||
|
||||
## Held-out test: does it reproduce?
|
||||
|
||||
Same model + same tuned config, `expand_gallery` toggled on vs. off, nothing else
|
||||
changed — full gallery mode, per-second scoring against X-Ray. This isolates
|
||||
expansion from every other variable (config, model, threshold) that differs
|
||||
between the training-set `exp`/`noexp` rows above.
|
||||
|
||||
**LVFace-B Glint360K, all 5 held-out films** (films never seen by the optimizer):
|
||||
|
||||
| film | F1 (exp) | F1 (noexp) | TPI Δ | FN Δ |
|
||||
|---|---|---|---|---|
|
||||
| Benny & Joon | 83.0% | 83.0% | -2 | +2 |
|
||||
| Downton Abbey: A New Era | 56.1% | 56.2% | -7 | +7 |
|
||||
| Lovelace | 77.5% | 77.4% | +33 | -33 |
|
||||
| The Many Saints of Newark | 46.3% | 46.3% | +2 | -2 |
|
||||
| Valerian and the City of a Thousand Planets | 74.1% | 74.1% | +2 | -2 |
|
||||
|
||||
**ArcFace R18** (Benny & Joon, r18's own tuned config): F1 77.1% for both, TPI/FN
|
||||
identical, FPI differs by 2 (noise).
|
||||
|
||||
**Every film, both models tested: F1 within 0.1–0.2pp, TPI/FN swings in the tens
|
||||
out of tens of thousands.** That's noise, not a signal — expansion made no
|
||||
measurable difference to per-second onscreen identification anywhere it was
|
||||
tested on unseen data.
|
||||
|
||||
## Two bugs this required catching (this section's own methodology)
|
||||
|
||||
Getting to the clean table above took two wrong turns, both worth recording
|
||||
since they're exactly the kind of error that produces a false positive "look,
|
||||
expansion helped!" finding:
|
||||
|
||||
1. **Timeout truncation.** The first Downton Abbey `exp` replay was cut off by a
|
||||
60s subprocess timeout at ~76% through the film (5589 of 7368 expected
|
||||
seconds) — a genuinely large, silent data loss that showed up as a large,
|
||||
convincing-looking TPI gap (47938 vs 52032) purely because one run had a
|
||||
quarter of the film missing. Caught by comparing `n_seconds` between runs
|
||||
before trusting any score delta; fixed by re-running with a longer timeout.
|
||||
2. **Bbox-matching bug.** An early per-second raw-annotation diff matched each
|
||||
`exp` detection to the *first* `noexp` detection with IoU > 0.5, not the
|
||||
*best*-overlapping one. With 3 faces close together in frame, this produced
|
||||
spurious "disagreements" (e.g. "exp says Aidan Quinn, noexp says Johnny
|
||||
Depp" at the same seconds) that vanished entirely once the match picked the
|
||||
true best-IoU candidate — both configs had actually output the exact same
|
||||
three names at the exact same three boxes.
|
||||
|
||||
Both bugs independently pointed toward "expansion is doing something," and both
|
||||
were artifacts of the comparison harness, not the pipeline. Worth remembering
|
||||
when a before/after diff looks dramatic: check that the two runs actually cover
|
||||
the same seconds, and match entities by best overlap, not first-found.
|
||||
|
||||
## What this means
|
||||
|
||||
The training-set aggregate effect (particularly the ~4x misID increase in full
|
||||
mode) doesn't reproduce on held-out data — at minimum it's far smaller than the
|
||||
training-set numbers suggested, and plausibly it's sampling variation from only
|
||||
4 training films rather than a real, generalizable mechanism. This doesn't mean
|
||||
`expand_gallery` never does anything (the mechanism is real — see
|
||||
`track_gallery.hpp`'s promotion logging: tracks *do* get confirmed and views *do*
|
||||
get promoted into the annex on every film tested), only that **whatever effect
|
||||
it has on final per-second identification was too small to detect against 5
|
||||
held-out films** with this scoring method. A cleaner test would need either many
|
||||
more held-out films or a metric that can see the annex's direct contribution
|
||||
(e.g. tagging which reference embedding won each match), neither of which this
|
||||
pass had budget for.
|
||||
|
||||
**Practical takeaway**: don't treat the training-set `exp` vs `noexp` numbers in
|
||||
`rep4-optimizer-results.md` as proof that expansion changes real-world behavior
|
||||
in either direction — on the evidence gathered so far, it doesn't move the
|
||||
needle enough to see.
|
||||
Reference in New Issue
Block a user