# Pose expansion: does "learning" new poses mid-film help?

`expand_gallery` (`src/gallery/track_gallery.hpp`) promotes a confidently-identified
track's novel-pose reference views into a per-film, in-memory gallery annex — the
idea being that once the pipeline is sure who someone is, a pose it hasn't seen
before (turned head, different lighting) becomes a free extra reference for
recognising that actor again later in the same film, without touching the baked
gallery.

## The training-set signal

Averaged across all 4 models, on the 4 films used for optimization:

| scope | expansion | F1 | R | misID |
|---|---|---|---|---|
| full | off | 71.2% | 58.3% | 209 |
| full | **on** | 71.2% | 59.7% | **864** |
| restricted | off | 73.6% | 61.3% | 194 |
| restricted | **on** | **75.4%** | **64.5%** | 135 |

In `restricted` mode (matcher's candidate set capped to the film's own credited
cast) expansion looked like a clean win: +1.8pp F1, +3.2pp recall, misID actually
lower. In `full` mode it looked flat-to-costly: ~0 F1 change, recall +1.4pp, but
misID roughly quadrupled (209 → 864) — see `rep4-optimizer-results.md` for the
per-model breakdown. That's the number that motivated this page: **does turning
expansion on actually change what gets recognised, frame by frame, or is the
aggregate F1 shift something else?**

## Held-out test: does it reproduce?

Same model + same tuned config, `expand_gallery` toggled on vs. off, nothing else
changed — full gallery mode, per-second scoring against X-Ray. This isolates
expansion from every other variable (config, model, threshold) that differs
between the training-set `exp`/`noexp` rows above.

**LVFace-B Glint360K, all 5 held-out films** (films never seen by the optimizer):

| film | F1 (exp) | F1 (noexp) | TPI Δ | FN Δ |
|---|---|---|---|---|
| Benny & Joon | 83.0% | 83.0% | -2 | +2 |
| Downton Abbey: A New Era | 56.1% | 56.2% | -7 | +7 |
| Lovelace | 77.5% | 77.4% | +33 | -33 |
| The Many Saints of Newark | 46.3% | 46.3% | +2 | -2 |
| Valerian and the City of a Thousand Planets | 74.1% | 74.1% | +2 | -2 |

**ArcFace R18** (Benny & Joon, r18's own tuned config): F1 77.1% for both, TPI/FN
identical, FPI differs by 2 (noise).

**Every film, both models tested: F1 within 0.1–0.2pp, TPI/FN swings in the tens
out of tens of thousands.** That's noise, not a signal — expansion made no
measurable difference to per-second onscreen identification anywhere it was
tested on unseen data.

## Two bugs this required catching (this section's own methodology)

Getting to the clean table above took two wrong turns, both worth recording
since they're exactly the kind of error that produces a false positive "look,
expansion helped!" finding:

1. **Timeout truncation.** The first Downton Abbey `exp` replay was cut off by a
   60s subprocess timeout at ~76% through the film (5589 of 7368 expected
   seconds) — a genuinely large, silent data loss that showed up as a large,
   convincing-looking TPI gap (47938 vs 52032) purely because one run had a
   quarter of the film missing. Caught by comparing `n_seconds` between runs
   before trusting any score delta; fixed by re-running with a longer timeout.
2. **Bbox-matching bug.** An early per-second raw-annotation diff matched each
   `exp` detection to the *first* `noexp` detection with IoU > 0.5, not the
   *best*-overlapping one. With 3 faces close together in frame, this produced
   spurious "disagreements" (e.g. "exp says Aidan Quinn, noexp says Johnny
   Depp" at the same seconds) that vanished entirely once the match picked the
   true best-IoU candidate — both configs had actually output the exact same
   three names at the exact same three boxes.

Both bugs independently pointed toward "expansion is doing something," and both
were artifacts of the comparison harness, not the pipeline. Worth remembering
when a before/after diff looks dramatic: check that the two runs actually cover
the same seconds, and match entities by best overlap, not first-found.

## What this means

The training-set aggregate effect (particularly the ~4x misID increase in full
mode) doesn't reproduce on held-out data — at minimum it's far smaller than the
training-set numbers suggested, and plausibly it's sampling variation from only
4 training films rather than a real, generalizable mechanism. This doesn't mean
`expand_gallery` never does anything (the mechanism is real — see
`track_gallery.hpp`'s promotion logging: tracks *do* get confirmed and views *do*
get promoted into the annex on every film tested), only that **whatever effect
it has on final per-second identification was too small to detect against 5
held-out films** with this scoring method. A cleaner test would need either many
more held-out films or a metric that can see the annex's direct contribution
(e.g. tagging which reference embedding won each match), neither of which this
pass had budget for.

**Practical takeaway**: don't treat the training-set `exp` vs `noexp` numbers in
`rep4-optimizer-results.md` as proof that expansion changes real-world behavior
in either direction — on the evidence gathered so far, it doesn't move the
needle enough to see.