docs: full data-grounded rewrite of the performance report

Replaces narrative claims with verified numbers across all report pages:

- Cross-model held-out validation (LVFace/mbf/r18, all 5 held-out
  films): LVFace wins every film outright, not just "consistent with"
  the training-set pick. r50 dropped from the detailed comparison
  (gallery has ~30% fewer reference images per actor than the other
  three models on identical source photos).
- Per-film training breakdown: LVFace does not win every training
  film (mbf beats it on Lord of War); the 75.3% macro figure hides a
  10.7pp spread.
- Gallery coverage computed per film (20.3%-78.6%) instead of one
  flat 67%-missing average.
- Found and fixed a real scoring bug in optimize.py: a candidate
  whose hardest film's replay timed out was averaged over survivors
  instead of penalized, silently rewarding partial coverage. Affected
  3 of 16 training combos; corrected throughout, and optimize.py now
  scores an incomplete evaluation f1=0.0 instead of averaging over
  whichever films happened to finish.
- Every FPI frame in the deep dive now comes from the proper montage
  renderer (Onscreen/Offscreen panel, ghosts never drawn as boxes),
  never the bare-box debug overlay used earlier.
- Every distinct out-of-cast name across all 9 films gets its own
  frame at its first appearance (9 names, 4 films), not a
  single-example spot check: 2 ground-truth gaps, 1 photograph
  misread as a person, 6 genuine lookalike confusions.
- New methodology.md: the scene-level-vs-per-second scoring mismatch
  that the rest of the report assumes, written out once.
- Cut the deadlock/gdb debugging narrative from the experiment log;
  kept the one fact that matters (KPN's node/network split lets the
  expensive GPU stage run once and the cheap stage replay against
  cached embeddings).
- Plain declarative style throughout, no em dashes, no blog voice.
This commit is contained in:
2026-07-21 08:55:57 +02:00
parent 4b5557974b
commit 0bd2747069
18 changed files with 1824 additions and 890 deletions
+28 -4
View File
@@ -60,10 +60,34 @@ stage_frame "${MONTAGE_ROOT}/Valerian_and_the_City_of_a_Thousand_Plan/scene_4/4_
valerian_screen_call.jpg
stage_frame "${MONTAGE_ROOT}/The_Many_Saints_of_Newark/out_of_cast_fpi/4_worst_t000871.jpg" \
many_saints_outofcast_fpi.jpg
# debug-overlay example (extinction state drawn as frozen boxes) — from the
# dump_error_frames output, not the montage package
stage_frame "experiments/results/holdout/frames/many_saints/fpi/fpi_t03543.jpg" \
many_saints_ghost_fpi.jpg
# One frame per DISTINCT out-of-cast name across all 9 films, uniform rule
# (see scripts/docs/first_fpi_frames.py): the first second in the raw replay
# stream where the pipeline names someone not in the film's credited cast at
# all. Rendered with the proper montage renderer (Onscreen/Offscreen panel),
# never dump_error_frames.py's bare-box overlay. Regenerate with:
# python3 scripts/docs/first_fpi_frames.py
# 5 of 9 films have zero out-of-cast names in their whole runtime (Benny &
# Joon, Cafe Society, Downton Abbey, Sound of Metal, Valerian) and produce
# no frames here.
stage_frame "${MONTAGE_ROOT}/Lord_of_War/first_fpi_david_shumbris/first_fpi_t000418.jpg" \
lord_of_war_fpi_shumbris.jpg
stage_frame "${MONTAGE_ROOT}/Lord_of_War/first_fpi_ronald_reagan/first_fpi_t001003.jpg" \
lord_of_war_fpi_reagan_photo.jpg
stage_frame "${MONTAGE_ROOT}/Lord_of_War/first_fpi_lance_reddick/first_fpi_t006424.jpg" \
lord_of_war_fpi_reddick.jpg
stage_frame "${MONTAGE_ROOT}/Lovelace/first_fpi_chloë_sevigny/first_fpi_t002451.jpg" \
lovelace_fpi_sevigny.jpg
stage_frame "${MONTAGE_ROOT}/Scarface/first_fpi_kirstie_alley/first_fpi_t002451.jpg" \
scarface_fpi_alley.jpg
stage_frame "${MONTAGE_ROOT}/The_Many_Saints_of_Newark/first_fpi_germar_terrell_gardner/first_fpi_t000848.jpg" \
many_saints_fpi_gardner.jpg
stage_frame "${MONTAGE_ROOT}/The_Many_Saints_of_Newark/first_fpi_archie_yates/first_fpi_t002521.jpg" \
many_saints_fpi_yates.jpg
stage_frame "${MONTAGE_ROOT}/The_Many_Saints_of_Newark/first_fpi_zooey_deschanel/first_fpi_t002819.jpg" \
many_saints_fpi_deschanel.jpg
stage_frame "${MONTAGE_ROOT}/The_Many_Saints_of_Newark/first_fpi_talia_balsam/first_fpi_t004551.jpg" \
many_saints_fpi_balsam.jpg
if [ ! -f "${ASSETS_DIR}/germar_beats_xray.jpg" ]; then
echo "==> pulling report-highlights/germar_beats_xray.jpg..."