docs: four focused findings pages (best model, gallery scope, expansion, deep dive)

Splits the rep4 write-up's key findings into their own linkable pages:
- best-model.md: calibration curves first (discriminative power, independent
  of any threshold), then F1 on the benchmark — LVFace-B Glint360K wins both.
- gallery-scope.md: whole vs. cast-restricted gallery, isolated from model and
  expansion choice — restriction wins on every axis, but isn't a shipped
  runtime feature yet.
- pose-expansion.md: the training-set expand_gallery effect, and the held-out
  replication attempt that found it doesn't reproduce (5 films, 2 models,
  after catching and fixing a replay-timeout truncation bug and a bbox
  first-match-instead-of-best-match bug in the comparison harness itself). An
  honest null result, with the methodology errors documented since they're
  exactly the kind that manufacture a false "it works!" finding.
- lvface-deep-dive.md: the winning model's held-out generalization gap, its
  two failure modes (frozen-bbox ghost tracks), and a verified case (cross-
  checked against Jellyfin's independent cast metadata) where LVFace
  correctly identified an actor that X-Ray's ground truth failed to credit.

Adds a "report-highlights" artifact-registry package (scripts/artifacts/
push_artifacts.sh, pull_artifacts.sh) for hand-picked illustrative frames that
aren't reproducible via the automated best/worst montage selection, and wires
pulling it into scripts/docs/build_site.sh.
This commit is contained in:
2026-07-19 19:40:19 +02:00
parent d340da755a
commit 4925443e56
9 changed files with 384 additions and 13 deletions
+61
View File
@@ -0,0 +1,61 @@
# Which embedding model is best?
Four candidates went into the bake-off: three ArcFace variants (w600k-R50,
R18, w600k-MBF) and LVFace-B (Glint360K), a Vision-Transformer embedder that's
a drop-in replacement for ArcFace's `[N,3,112,112]` input / 512-d output. The
open question: is LVFace (455MB) actually better, or just the biggest?
## First signal: calibration curves
Each gallery carries a fitted Platt sigmoid `P(match | cosine similarity) =
σ(a·sim + b)`, embedded directly in the gallery's HDF5 file
(`src/gallery/gallery_calibration.hpp`). This is a property of the embedding
space alone — computed from intra/inter-actor reference-image pairs, no
tracking or scene logic involved — so it's a clean first read on discriminative
power before running a single benchmark.
![Calibrated P(match|similarity) for all four models](assets/images/calibration_curves.png)
| model | `a` (steepness) | boundary at P=0.5 |
|---|---|---|
| **LVFace-B Glint360K** | **17.7** | **sim 0.228** |
| ArcFace w600k-MBF | 16.2 | sim 0.267 |
| ArcFace w600k-R50 | 15.4 | sim 0.301 |
| ArcFace R18 | 15.3 | sim 0.309 |
LVFace has both the steepest transition and the lowest decision boundary — it
separates same-actor from different-actor reference pairs more confidently, at
a *lower* similarity threshold, than any ArcFace variant. That's a genuine
head start before the tracking/scoring pipeline is even involved.
## Second signal: F1 on the actual benchmark
Best full-gallery (no cast-restriction) result per model, from the 16-combo
rep4 matrix (`rep4-optimizer-results.md`):
| model | F1 | P | R | misID |
|---|---|---|---|---|
| **LVFace-B Glint360K** | **75.3%** | 89.7% | **65.4%** | 232 |
| ArcFace w600k-MBF | 74.2% | 87.4% | 64.4% | 57 |
| ArcFace R18 | 69.1% | 87.6% | 57.7% | 242 |
| ArcFace w600k-R50 | 68.5% | 94.0% | 54.1% | 150 |
LVFace wins outright, with the highest recall of any full-mode combo. This
reverses an earlier conclusion from a prior (superseded) benchmarking pass
using a scene-union metric, which found the three models statistically
indistinguishable (~85% each) and concluded LVFace wasn't worth its size — that
metric hid out-of-cast false positives behind a gallery∩cast recall mask (see
`optimizer-experiments.md`); the per-second metric used here does not.
Held-out validation (5 films never seen by the optimizer) confirms LVFace's
lead holds up out of sample — see the deep-dive page for the full breakdown,
including where it fails.
## Caveat: model choice is an operational change
Switching the default embedder isn't just flipping a config value — the
gallery itself is model-specific (embeddings from different models aren't
comparable), so any existing gallery built against ArcFace w600k-R50 needs to
be rebuilt from source images against LVFace before the new default takes
effect. `scripts/optimizer/reembed_gallery.py` does this from a reference
gallery's cached source images without re-downloading anything.
+56
View File
@@ -0,0 +1,56 @@
# Whole gallery vs. limited (cast-restricted) gallery
Two ways to run the matcher: **full** scores every detected face against the
entire library gallery (2418 actors across the 9-film benchmark set); **restricted**
pre-filters each film's gallery down to just its Jellyfin-credited cast (typically
~15 top-billed actors) before the matcher ever runs.
## The result
Averaged across all 4 models and both expansion settings, on the 4 rep4 training
films:
| scope | F1 | P | R | total misID (8 evals) |
|---|---|---|---|---|
| full | 71.2% | 91.1% | 59.0% | 1073 |
| **restricted** | **74.5%** | 92.2% | **62.9%** | **329** |
This is not a precision/recall trade — restriction wins on every axis at once:
**+3.3pp F1, +3.9pp recall, and less than a third the total misIDs.** Fewer
candidates in the matcher's search space means fewer opportunities for a
look-alike false match (an actor who happens to share enough facial structure
with someone in the film, but isn't actually in it), and the recall gain shows
it isn't costing real detections to get there.
Per-model, every single model's best-scoring combo in the full 16-way matrix is
a `restricted` variant — see the full table in `rep4-optimizer-results.md`. Two
combos hit **zero** true out-of-cast misidentifications:
`arcface_w600k_mbf_restricted_exp` (F1 76.5%) and, in full mode,
`LVFace-B_Glint360K_full_noexp` (F1 72.4%) — restriction isn't the only way to
reach misid=0, but it's the more reliable one.
## Why this isn't the shipped default
Cast-restriction is implemented today only as an **offline optimizer technique**
(`scripts/optimizer/cast_restrict.py`): it pre-builds a filtered gallery file
per film, using Jellyfin's own cast list, before the benchmark ever calls the
matcher. There's no runtime "restrict matching to this title's credited cast"
switch in the shipped application — `scene_analyze` always matches against
whatever single gallery file it's given.
Building that as a real feature would need, at minimum:
- A live Jellyfin cast lookup at analysis time (the title is already known —
`run_from_jellyfin.py` already does this same lookup for its own
`filter_gallery`-based restriction path, just not wired into `scene_analyze`
itself as a first-class option).
- A decision on the *fallback*: what happens to a real, uncredited cameo
(see the Germar Terrell Gardner case in the LVFace deep-dive) if the gallery
never includes them at all?
- Regenerating the restricted-gallery cache whenever the title's Jellyfin cast
list changes.
This is why the shipped `src/config.hpp` defaults use the `full`-mode winner
(`LVFace-B_Glint360K_full_exp`, F1 75.3% training / 67.4% held-out macro) rather
than the higher-scoring `restricted_exp` (78.3%) — the 78.3% number describes a
capability the app doesn't have yet, not what actually ships.
+23 -5
View File
@@ -5,12 +5,30 @@ film or TV episode — built on [KPN++](https://gitea.tourolle.paris/dtourolle/K
(a C++20 Kahn Process Network library) for the detect → track → match → scene
pipeline, with a Jellyfin-integrated gallery and an X-Ray-validated optimizer.
## Start here
## Start here — four questions this bake-off answers
- **[Rep4 model bake-off + threshold re-tune](rep4-optimizer-results.md)** — the
current experiment log: model comparison, DE threshold tuning, held-out
validation, and the visual failure-mode evidence (frozen-bbox ghost tracks).
This is where the shipped `src/config.hpp` defaults come from.
- **[Which model is best?](best-model.md)** — calibration curves first
(discriminative power, independent of any threshold), then F1 on the actual
benchmark. LVFace-B Glint360K wins both.
- **[Whole gallery vs. limited (cast-restricted) gallery](gallery-scope.md)** —
restricting the matcher to a film's credited cast is a clean win on every
axis (+3.3pp F1, less than a third the misIDs), but isn't a shipped runtime
feature yet.
- **[Does pose expansion help?](pose-expansion.md)** — a real training-set
effect that didn't reproduce on 5 held-out films once two methodology bugs
were caught and fixed. An honest null result, not a forced narrative.
- **[Deep dive: LVFace-B Glint360K](lvface-deep-dive.md)** — the winning
model's held-out generalization gap, its two real failure modes (frozen-bbox
"ghost tracks"), and one case where it correctly identified an actor that
the X-Ray ground truth itself failed to credit.
## The full technical log
- **[Rep4 model bake-off + threshold re-tune](rep4-optimizer-results.md)** —
the complete experiment log behind the four pages above: the ROCm teardown
deadlock root cause and fix, DE concurrency tuning, the full 16-combo
results table, and every caveat. This is where the shipped `src/config.hpp`
defaults come from.
- **[Optimizer experiments (prior round)](optimizer-experiments.md)** — the
earlier scene-union-metric tuning pass, superseded by the per-second metric
used in rep4 but kept for the ground-truth/architecture background.
+98
View File
@@ -0,0 +1,98 @@
# Deep dive: LVFace-B Glint360K
LVFace won the model bake-off (see `best-model.md`) and is the shipped default
embedder. This page is the honest accounting of how it actually performs —
including where it's wrong, and one case where the ground truth itself is
wrong and LVFace is right.
## Training vs. held-out: the generalization gap
The shipped config (`prob_threshold=0.754, anneal_sec=35.54,
extinction_sec=57.43, expand_gallery=true`) was tuned against 4 films. Scored
against the 5 films the optimizer never saw:
| film | F1 | P | R | TPI | FPI | misid | FN |
|---|---|---|---|---|---|---|---|
| Benny & Joon | 83.0% | 89.1% | 77.7% | 15125 | 1846 | 0 | 4337 |
| Lovelace | 77.5% | 90.3% | 67.9% | 14990 | 1085 | 58 | 7085 |
| Valerian and the City of a Thousand Planets | 74.1% | 97.1% | 60.0% | 18663 | 548 | 0 | 12467 |
| Downton Abbey: A New Era | 56.2% | 97.8% | 39.4% | 52027 | 1173 | 0 | 80084 |
| **The Many Saints of Newark** | **46.3%** | **54.7%** | 40.1% | 15922 | 4394 | **974** | 23791 |
| **macro average** | **67.4%** | 85.8% | 57.0% | | | | |
**67.4% held-out vs. 75.3% on training** — an ~8pp drop, and a **37pp spread
between the best and worst held-out film**. The config does not generalize
uniformly; two films are outright failure cases, for two different reasons.
## Failure mode 1: frozen-bbox "ghost tracks"
Both Many Saints of Newark (974 misIDs) and Downton Abbey (FN=80084, the worst
recall of the five) trace to the same root cause, verified directly against
the raw per-frame stream and the HDF5 dump's own detection counts — not
inferred from the score alone.
![Frozen ghost boxes over background, The Many Saints of Newark](assets/images/many_saints_ghost_fpi.jpg)
At this second, three of the four labeled boxes ("Jon Bernthal", "Joey Diaz",
"Billy Magnussen") sit over empty background — a blurred wall, hanging
plates — with no face in them. The real face in frame carries a second,
colliding label from another frozen box.
![15 ghost boxes over a blank title card, Downton Abbey: A New Era](assets/images/downton_abbey_ghost_fpi.jpg)
This is the starkest case: **15 actors named, all wrong, over a completely
blank closing title card.** Confirmed against the dump directly: `face_count`
is 0 from this point onward (no detector output at all), yet the same 15
identities keep appearing with the *exact same bounding box, unchanged to the
pixel*, for 57+ consecutive seconds.
This is `SceneTrackerFunc::active_[actor_idx].last_bbox`
(`src/nodes/scene_tracker_node.hpp`) being re-emitted unchanged — the
extinction state machine working exactly as coded, not a bug. The film cuts
from a packed group shot straight into 40+ seconds of blank titles/credits,
and `extinction_sec=57.4` is comfortably long enough to bridge that entire gap
without expiring, so the tracker faithfully reports "last known position" for
a cast that is no longer on screen at all. `extinction_sec` was tuned toward
long windows specifically because they bridge real gaps (occlusion, a turned
face) in most training footage — this is the cost side of that trade,
surfacing only when a film has a long enough faceless stretch to expose it.
## Failure mode 2: a genuine misID (for contrast)
Not every held-out failure is a ghost. This is a real face, correctly
detected, confidently misidentified:
*(same many_saints_ghost_fpi.jpg frame above also shows Leslie Odom Jr.'s box
carrying a second, colliding "Michael Gandolfini" label — two real tracks'
frozen positions happening to overlap, not a detection error.)*
## Where LVFace beat X-Ray
Not every "misID" is actually wrong. `second_score.py` counts a name as a true
out-of-cast misID whenever the named actor isn't in X-Ray's credited cast list
for the film at all — but X-Ray's cast list is itself incomplete.
![LVFace correctly identifies Germar Terrell Gardner, uncredited by X-Ray](assets/images/germar_beats_xray.jpg)
Germar Terrell Gardner — a real, clean, high-confidence detection — is counted
as a misID here because he doesn't appear in X-Ray's `people.csv` for The Many
Saints of Newark at all. But Jellyfin's independent cast metadata *does* credit
him for this exact film (cross-checked via `experiments/manifests/
jellyfin_casts.json`, a completely separate data source from X-Ray). This
isn't a lookalike error or a gallery mixup — it's the pipeline correctly
recognising a real cast member that one ground-truth source happened to omit.
This doesn't mean every flagged misID is secretly correct — Many Saints'
974-count total is still overwhelmingly the frozen-bbox failure mode above,
not uncredited-but-real cameos. But it's a reminder that the X-Ray corpus is a
convenient, large-scale ground truth, not a perfect one, and the "misID" number
in any of these tables has some irreducible noise floor from ground-truth gaps
in the other direction too.
## Summary
LVFace is the right default: it wins the model comparison outright, and its
failures are traceable, understood, and mostly attributable to one tunable
knob (`extinction_sec`) rather than the embedder itself. The held-out
generalization gap (75.3% → 67.4%) is real and should be treated as the honest
expected performance, not the training-set number.
+97
View File
@@ -0,0 +1,97 @@
# Pose expansion: does "learning" new poses mid-film help?
`expand_gallery` (`src/gallery/track_gallery.hpp`) promotes a confidently-identified
track's novel-pose reference views into a per-film, in-memory gallery annex — the
idea being that once the pipeline is sure who someone is, a pose it hasn't seen
before (turned head, different lighting) becomes a free extra reference for
recognising that actor again later in the same film, without touching the baked
gallery.
## The training-set signal
Averaged across all 4 models, on the 4 films used for optimization:
| scope | expansion | F1 | R | misID |
|---|---|---|---|---|
| full | off | 71.2% | 58.3% | 209 |
| full | **on** | 71.2% | 59.7% | **864** |
| restricted | off | 73.6% | 61.3% | 194 |
| restricted | **on** | **75.4%** | **64.5%** | 135 |
In `restricted` mode (matcher's candidate set capped to the film's own credited
cast) expansion looked like a clean win: +1.8pp F1, +3.2pp recall, misID actually
lower. In `full` mode it looked flat-to-costly: ~0 F1 change, recall +1.4pp, but
misID roughly quadrupled (209 → 864) — see `rep4-optimizer-results.md` for the
per-model breakdown. That's the number that motivated this page: **does turning
expansion on actually change what gets recognised, frame by frame, or is the
aggregate F1 shift something else?**
## Held-out test: does it reproduce?
Same model + same tuned config, `expand_gallery` toggled on vs. off, nothing else
changed — full gallery mode, per-second scoring against X-Ray. This isolates
expansion from every other variable (config, model, threshold) that differs
between the training-set `exp`/`noexp` rows above.
**LVFace-B Glint360K, all 5 held-out films** (films never seen by the optimizer):
| film | F1 (exp) | F1 (noexp) | TPI Δ | FN Δ |
|---|---|---|---|---|
| Benny & Joon | 83.0% | 83.0% | -2 | +2 |
| Downton Abbey: A New Era | 56.1% | 56.2% | -7 | +7 |
| Lovelace | 77.5% | 77.4% | +33 | -33 |
| The Many Saints of Newark | 46.3% | 46.3% | +2 | -2 |
| Valerian and the City of a Thousand Planets | 74.1% | 74.1% | +2 | -2 |
**ArcFace R18** (Benny & Joon, r18's own tuned config): F1 77.1% for both, TPI/FN
identical, FPI differs by 2 (noise).
**Every film, both models tested: F1 within 0.10.2pp, TPI/FN swings in the tens
out of tens of thousands.** That's noise, not a signal — expansion made no
measurable difference to per-second onscreen identification anywhere it was
tested on unseen data.
## Two bugs this required catching (this section's own methodology)
Getting to the clean table above took two wrong turns, both worth recording
since they're exactly the kind of error that produces a false positive "look,
expansion helped!" finding:
1. **Timeout truncation.** The first Downton Abbey `exp` replay was cut off by a
60s subprocess timeout at ~76% through the film (5589 of 7368 expected
seconds) — a genuinely large, silent data loss that showed up as a large,
convincing-looking TPI gap (47938 vs 52032) purely because one run had a
quarter of the film missing. Caught by comparing `n_seconds` between runs
before trusting any score delta; fixed by re-running with a longer timeout.
2. **Bbox-matching bug.** An early per-second raw-annotation diff matched each
`exp` detection to the *first* `noexp` detection with IoU > 0.5, not the
*best*-overlapping one. With 3 faces close together in frame, this produced
spurious "disagreements" (e.g. "exp says Aidan Quinn, noexp says Johnny
Depp" at the same seconds) that vanished entirely once the match picked the
true best-IoU candidate — both configs had actually output the exact same
three names at the exact same three boxes.
Both bugs independently pointed toward "expansion is doing something," and both
were artifacts of the comparison harness, not the pipeline. Worth remembering
when a before/after diff looks dramatic: check that the two runs actually cover
the same seconds, and match entities by best overlap, not first-found.
## What this means
The training-set aggregate effect (particularly the ~4x misID increase in full
mode) doesn't reproduce on held-out data — at minimum it's far smaller than the
training-set numbers suggested, and plausibly it's sampling variation from only
4 training films rather than a real, generalizable mechanism. This doesn't mean
`expand_gallery` never does anything (the mechanism is real — see
`track_gallery.hpp`'s promotion logging: tracks *do* get confirmed and views *do*
get promoted into the annex on every film tested), only that **whatever effect
it has on final per-second identification was too small to detect against 5
held-out films** with this scoring method. A cleaner test would need either many
more held-out films or a metric that can see the annex's direct contribution
(e.g. tagging which reference embedding won each match), neither of which this
pass had budget for.
**Practical takeaway**: don't treat the training-set `exp` vs `noexp` numbers in
`rep4-optimizer-results.md` as proof that expansion changes real-world behavior
in either direction — on the evidence gathered so far, it doesn't move the
needle enough to see.
+6 -1
View File
@@ -18,7 +18,12 @@ theme:
nav:
- Home: index.md
- Rep4 Bake-off & Re-tune: rep4-optimizer-results.md
- Findings:
- Best Model: best-model.md
- Gallery Scope (Full vs. Limited): gallery-scope.md
- Pose Expansion: pose-expansion.md
- LVFace Deep Dive: lvface-deep-dive.md
- Rep4 Bake-off & Re-tune (full log): rep4-optimizer-results.md
- Optimizer Experiments (prior round): optimizer-experiments.md
- Service Conversion (proposal): service-conversion.md
+17 -1
View File
@@ -10,6 +10,7 @@
# scripts/artifacts/pull_artifacts.sh galleries [version]
# scripts/artifacts/pull_artifacts.sh montage-frames <film-slug> [version]
# scripts/artifacts/pull_artifacts.sh experiment-data [version]
# scripts/artifacts/pull_artifacts.sh report-highlights <name> [version]
# version defaults to "latest" (newest uploaded version, by created_at).
set -euo pipefail
@@ -74,10 +75,19 @@ pull_experiment_data() {
rm "$tmp"
}
pull_report_highlight() {
local version="$1" name="$2"
local dest="${REPO_ROOT}/docs/assets/images"
mkdir -p "$dest"
echo "=== report-highlights/${name} (version ${version}) ==="
curl -sf "${DL_BASE}/generic/report-highlights/${version}/${name}" -o "${dest}/${name}"
}
if [ $# -eq 0 ]; then
echo "usage: $0 galleries [version]" >&2
echo " $0 montage-frames <film-slug> [version]" >&2
echo " $0 experiment-data [version]" >&2
echo " $0 report-highlights <name> [version]" >&2
exit 1
fi
@@ -99,8 +109,14 @@ case "$TARGET" in
[ "$VERSION" = "latest" ] && VERSION="$(resolve_latest_version experiment-data)"
pull_experiment_data "$VERSION"
;;
report-highlights)
NAME="${2:?usage: $0 report-highlights <name> [version]}"
VERSION="${3:-latest}"
[ "$VERSION" = "latest" ] && VERSION="$(resolve_latest_version report-highlights)"
pull_report_highlight "$VERSION" "$NAME"
;;
*)
echo "unknown target: $TARGET (expected galleries, montage-frames, or experiment-data)" >&2
echo "unknown target: $TARGET (expected galleries, montage-frames, experiment-data, or report-highlights)" >&2
exit 1
;;
esac
+18 -3
View File
@@ -11,12 +11,16 @@
# scripts/artifacts/push_artifacts.sh galleries
# scripts/artifacts/push_artifacts.sh montage-frames
# scripts/artifacts/push_artifacts.sh experiment-data
# scripts/artifacts/push_artifacts.sh galleries montage-frames experiment-data
# scripts/artifacts/push_artifacts.sh report-highlights
# scripts/artifacts/push_artifacts.sh galleries montage-frames experiment-data report-highlights
#
# Package layout (owner=dtourolle, repo=scene-actor-extraction):
# generic/galleries/<version>/gallery_<model>.h5 (one file per model)
# generic/montage-frames/<version>/<film-slug>.zip (zipped per-film frames)
# generic/experiment-data/<version>/experiment-data.zip (manifests/trajectories/results)
# generic/report-highlights/<version>/<name>.jpg (individual, hand-picked
# illustrative frames referenced by docs/*.md that aren't reproducible via
# the automated best/worst montage selection — see scripts/docs/build_site.sh)
# version = current git short SHA, so artifacts are traceable to the code that
# produced them. Re-running with the same SHA overwrites that version's files.
set -euo pipefail
@@ -95,8 +99,18 @@ push_experiment_data() {
upload "experiment-data" "experiment-data.zip" "$zipfile"
}
push_report_highlights() {
echo "=== report-highlights (version ${VERSION}) ==="
local src="${REPO_ROOT}/experiments/results/holdout/montage/many_saints/out_of_cast_fpi/10_t003641.jpg"
if [ ! -f "$src" ]; then
echo " [warn] $src not found, skipping" >&2
return
fi
upload "report-highlights" "germar_beats_xray.jpg" "$src"
}
if [ $# -eq 0 ]; then
echo "usage: $0 <galleries|montage-frames|experiment-data> [...]" >&2
echo "usage: $0 <galleries|montage-frames|experiment-data|report-highlights> [...]" >&2
exit 1
fi
@@ -105,7 +119,8 @@ for target in "$@"; do
galleries) push_galleries ;;
montage-frames) push_montage_frames ;;
experiment-data) push_experiment_data ;;
*) echo "unknown target: $target (expected galleries, montage-frames, or experiment-data)" >&2; exit 1 ;;
report-highlights) push_report_highlights ;;
*) echo "unknown target: $target (expected galleries, montage-frames, experiment-data, or report-highlights)" >&2; exit 1 ;;
esac
done
+5
View File
@@ -26,6 +26,11 @@ cp -v "${FRAMES_ROOT}/many_saints/fpi/fpi_t03543.jpg" \
cp -v "${FRAMES_ROOT}/downton_abbey/fpi/fpi_t07242.jpg" \
"${ASSETS_DIR}/downton_abbey_ghost_fpi.jpg"
if [ ! -f "${ASSETS_DIR}/germar_beats_xray.jpg" ]; then
echo "==> pulling report-highlights/germar_beats_xray.jpg..."
scripts/artifacts/pull_artifacts.sh report-highlights germar_beats_xray.jpg
fi
if [ ! -d experiments/galleries ] || [ -z "$(ls -A experiments/galleries 2>/dev/null)" ]; then
echo "==> pulling galleries (not found locally)..."
scripts/artifacts/pull_artifacts.sh galleries