Pose expansion: does "learning" new poses mid-film help?

expand_gallery (src/gallery/track_gallery.hpp) promotes a confidently-identified track's novel-pose reference views into a per-film, in-memory gallery annex — the idea being that once the pipeline is sure who someone is, a pose it hasn't seen before (turned head, different lighting) becomes a free extra reference for recognising that actor again later in the same film, without touching the baked gallery.

The training-set signal

Averaged across all 4 models, on the 4 films used for optimization:

scope	expansion	F1	R	misID
full	off	71.2%	58.3%	209
full	on	71.2%	59.7%	864
restricted	off	73.6%	61.3%	194
restricted	on	75.4%	64.5%	135

In restricted mode (matcher's candidate set capped to the film's own credited cast) expansion looked like a clean win: +1.8pp F1, +3.2pp recall, misID actually lower. In full mode it looked flat-to-costly: ~0 F1 change, recall +1.4pp, but misID roughly quadrupled (209 → 864) — see the bake-off experiment log for the per-model breakdown. That's the number that motivated this page: does turning expansion on actually change what gets recognised, frame by frame, or is the aggregate F1 shift something else?

Held-out test: does it reproduce?

Same model + same tuned config, expand_gallery toggled on vs. off, nothing else changed — full gallery mode, per-second scoring against X-Ray. This isolates expansion from every other variable (config, model, threshold) that differs between the training-set exp/noexp rows above.

LVFace-B Glint360K, all 5 held-out films (films never seen by the optimizer):

film	F1 (exp)	F1 (noexp)	TPI Δ	FN Δ
Benny & Joon	83.0%	83.0%	-2	+2
Downton Abbey: A New Era	56.1%	56.2%	-7	+7
Lovelace	77.5%	77.4%	+33	-33
The Many Saints of Newark	46.3%	46.3%	+2	-2
Valerian and the City of a Thousand Planets	74.1%	74.1%	+2	-2

ArcFace R18 (Benny & Joon, r18's own tuned config): F1 77.1% for both, TPI/FN identical, FPI differs by 2 (noise).

Every film, both models tested: F1 within 0.1–0.2pp, TPI/FN swings in the tens out of tens of thousands. That's noise, not a signal — expansion made no measurable difference to per-second onscreen identification anywhere it was tested on unseen data.

Two bugs this required catching (this section's own methodology)

Getting to the clean table above took two wrong turns, both worth recording since they're exactly the kind of error that produces a false positive "look, expansion helped!" finding:

1. Timeout truncation. The first Downton Abbey exp replay was cut off by a 60s subprocess timeout at ~76% through the film (5589 of 7368 expected seconds) — a genuinely large, silent data loss that showed up as a large, convincing-looking TPI gap (47938 vs 52032) purely because one run had a quarter of the film missing. Caught by comparing n_seconds between runs before trusting any score delta; fixed by re-running with a longer timeout.
2. Bbox-matching bug. An early per-second raw-annotation diff matched each exp detection to the first noexp detection with IoU > 0.5, not the best-overlapping one. With 3 faces close together in frame, this produced spurious "disagreements" (e.g. "exp says Aidan Quinn, noexp says Johnny Depp" at the same seconds) that vanished entirely once the match picked the true best-IoU candidate — both configs had actually output the exact same three names at the exact same three boxes.

Both bugs independently pointed toward "expansion is doing something," and both were artifacts of the comparison harness, not the pipeline. Worth remembering when a before/after diff looks dramatic: check that the two runs actually cover the same seconds, and match entities by best overlap, not first-found.

What this means

The training-set aggregate effect (particularly the ~4x misID increase in full mode) doesn't reproduce on held-out data — at minimum it's far smaller than the training-set numbers suggested, and plausibly it's sampling variation from only 4 training films rather than a real, generalizable mechanism. This doesn't mean expand_gallery never does anything (the mechanism is real — see track_gallery.hpp's promotion logging: tracks do get confirmed and views do get promoted into the annex on every film tested), only that whatever effect it has on final per-second identification was too small to detect against 5 held-out films with this scoring method. A cleaner test would need either many more held-out films or a metric that can see the annex's direct contribution (e.g. tagging which reference embedding won each match), neither of which this pass had budget for.

Practical takeaway: don't treat the training-set exp vs noexp numbers in the bake-off experiment log as proof that expansion changes real-world behavior in either direction — on the evidence gathered so far, it doesn't move the needle enough to see.