Files

T

dtourolle 4925443e56 docs: four focused findings pages (best model, gallery scope, expansion, deep dive)

Splits the rep4 write-up's key findings into their own linkable pages:
- best-model.md: calibration curves first (discriminative power, independent
  of any threshold), then F1 on the benchmark — LVFace-B Glint360K wins both.
- gallery-scope.md: whole vs. cast-restricted gallery, isolated from model and
  expansion choice — restriction wins on every axis, but isn't a shipped
  runtime feature yet.
- pose-expansion.md: the training-set expand_gallery effect, and the held-out
  replication attempt that found it doesn't reproduce (5 films, 2 models,
  after catching and fixing a replay-timeout truncation bug and a bbox
  first-match-instead-of-best-match bug in the comparison harness itself). An
  honest null result, with the methodology errors documented since they're
  exactly the kind that manufacture a false "it works!" finding.
- lvface-deep-dive.md: the winning model's held-out generalization gap, its
  two failure modes (frozen-bbox ghost tracks), and a verified case (cross-
  checked against Jellyfin's independent cast metadata) where LVFace
  correctly identified an actor that X-Ray's ground truth failed to credit.

Adds a "report-highlights" artifact-registry package (scripts/artifacts/
push_artifacts.sh, pull_artifacts.sh) for hand-picked illustrative frames that
aren't reproducible via the automated best/worst montage selection, and wires
pulling it into scripts/docs/build_site.sh.

2026-07-19 19:40:19 +02:00

5.1 KiB

Raw Blame History

Pose expansion: does "learning" new poses mid-film help?

expand_gallery (src/gallery/track_gallery.hpp) promotes a confidently-identified track's novel-pose reference views into a per-film, in-memory gallery annex — the idea being that once the pipeline is sure who someone is, a pose it hasn't seen before (turned head, different lighting) becomes a free extra reference for recognising that actor again later in the same film, without touching the baked gallery.

The training-set signal

Averaged across all 4 models, on the 4 films used for optimization:

scope	expansion	F1	R	misID
full	off	71.2%	58.3%	209
full	on	71.2%	59.7%	864
restricted	off	73.6%	61.3%	194
restricted	on	75.4%	64.5%	135

In restricted mode (matcher's candidate set capped to the film's own credited cast) expansion looked like a clean win: +1.8pp F1, +3.2pp recall, misID actually lower. In full mode it looked flat-to-costly: ~0 F1 change, recall +1.4pp, but misID roughly quadrupled (209 → 864) — see rep4-optimizer-results.md for the per-model breakdown. That's the number that motivated this page: does turning expansion on actually change what gets recognised, frame by frame, or is the aggregate F1 shift something else?

Held-out test: does it reproduce?

Same model + same tuned config, expand_gallery toggled on vs. off, nothing else changed — full gallery mode, per-second scoring against X-Ray. This isolates expansion from every other variable (config, model, threshold) that differs between the training-set exp/noexp rows above.

LVFace-B Glint360K, all 5 held-out films (films never seen by the optimizer):

film	F1 (exp)	F1 (noexp)	TPI Δ	FN Δ
Benny & Joon	83.0%	83.0%	-2	+2
Downton Abbey: A New Era	56.1%	56.2%	-7	+7
Lovelace	77.5%	77.4%	+33	-33
The Many Saints of Newark	46.3%	46.3%	+2	-2
Valerian and the City of a Thousand Planets	74.1%	74.1%	+2	-2

ArcFace R18 (Benny & Joon, r18's own tuned config): F1 77.1% for both, TPI/FN identical, FPI differs by 2 (noise).

Every film, both models tested: F1 within 0.1–0.2pp, TPI/FN swings in the tens out of tens of thousands. That's noise, not a signal — expansion made no measurable difference to per-second onscreen identification anywhere it was tested on unseen data.

Two bugs this required catching (this section's own methodology)

Getting to the clean table above took two wrong turns, both worth recording since they're exactly the kind of error that produces a false positive "look, expansion helped!" finding:

Timeout truncation. The first Downton Abbey exp replay was cut off by a 60s subprocess timeout at ~76% through the film (5589 of 7368 expected seconds) — a genuinely large, silent data loss that showed up as a large, convincing-looking TPI gap (47938 vs 52032) purely because one run had a quarter of the film missing. Caught by comparing n_seconds between runs before trusting any score delta; fixed by re-running with a longer timeout.
Bbox-matching bug. An early per-second raw-annotation diff matched each exp detection to the first noexp detection with IoU > 0.5, not the best-overlapping one. With 3 faces close together in frame, this produced spurious "disagreements" (e.g. "exp says Aidan Quinn, noexp says Johnny Depp" at the same seconds) that vanished entirely once the match picked the true best-IoU candidate — both configs had actually output the exact same three names at the exact same three boxes.

Both bugs independently pointed toward "expansion is doing something," and both were artifacts of the comparison harness, not the pipeline. Worth remembering when a before/after diff looks dramatic: check that the two runs actually cover the same seconds, and match entities by best overlap, not first-found.

What this means

The training-set aggregate effect (particularly the ~4x misID increase in full mode) doesn't reproduce on held-out data — at minimum it's far smaller than the training-set numbers suggested, and plausibly it's sampling variation from only 4 training films rather than a real, generalizable mechanism. This doesn't mean expand_gallery never does anything (the mechanism is real — see track_gallery.hpp's promotion logging: tracks do get confirmed and views do get promoted into the annex on every film tested), only that whatever effect it has on final per-second identification was too small to detect against 5 held-out films with this scoring method. A cleaner test would need either many more held-out films or a metric that can see the annex's direct contribution (e.g. tagging which reference embedding won each match), neither of which this pass had budget for.

Practical takeaway: don't treat the training-set exp vs noexp numbers in rep4-optimizer-results.md as proof that expansion changes real-world behavior in either direction — on the evidence gathered so far, it doesn't move the needle enough to see.

5.1 KiB Raw Blame History Unescape Escape