docs: full data-grounded rewrite of the performance report

Replaces narrative claims with verified numbers across all report pages:

- Cross-model held-out validation (LVFace/mbf/r18, all 5 held-out
  films): LVFace wins every film outright, not just "consistent with"
  the training-set pick. r50 dropped from the detailed comparison
  (gallery has ~30% fewer reference images per actor than the other
  three models on identical source photos).
- Per-film training breakdown: LVFace does not win every training
  film (mbf beats it on Lord of War); the 75.3% macro figure hides a
  10.7pp spread.
- Gallery coverage computed per film (20.3%-78.6%) instead of one
  flat 67%-missing average.
- Found and fixed a real scoring bug in optimize.py: a candidate
  whose hardest film's replay timed out was averaged over survivors
  instead of penalized, silently rewarding partial coverage. Affected
  3 of 16 training combos; corrected throughout, and optimize.py now
  scores an incomplete evaluation f1=0.0 instead of averaging over
  whichever films happened to finish.
- Every FPI frame in the deep dive now comes from the proper montage
  renderer (Onscreen/Offscreen panel, ghosts never drawn as boxes),
  never the bare-box debug overlay used earlier.
- Every distinct out-of-cast name across all 9 films gets its own
  frame at its first appearance (9 names, 4 films), not a
  single-example spot check: 2 ground-truth gaps, 1 photograph
  misread as a person, 6 genuine lookalike confusions.
- New methodology.md: the scene-level-vs-per-second scoring mismatch
  that the rest of the report assumes, written out once.
- Cut the deadlock/gdb debugging narrative from the experiment log;
  kept the one fact that matters (KPN's node/network split lets the
  expensive GPU stage run once and the cheap stage replay against
  cached embeddings).
- Plain declarative style throughout, no em dashes, no blog voice.
This commit is contained in:
2026-07-21 08:55:57 +02:00
parent 4b5557974b
commit 0bd2747069
18 changed files with 1824 additions and 890 deletions
+24 -6
View File
@@ -106,8 +106,12 @@ def evaluate(cfg, films, build_dir, step=None):
"""Objective = MACRO-mean over films of each film's duration-weighted per-scene F1.
Each film's replay runs in a subprocess (timeout-guarded) to survive the
intermittent ROCm teardown deadlock. A film whose replay times out is dropped
from the average rather than hanging the whole sweep.
intermittent ROCm teardown deadlock. If ANY film's replay times out, this
evaluation is scored f1=0.0 (see below) rather than averaging over the
survivors — a partial-coverage eval must never look better than a complete
one, or DE will converge onto configs that make the hardest film time out.
(An earlier version averaged over survivors, which silently rewarded
truncation; the rep4 `mbf_full_noexp` winner was one such corrupted eval.)
UNIFORM PER-SECOND scoring (second_score.py): every second of the film is sampled;
GT(t) = the cast of the X-Ray scene containing t, Pred(t) = actors whose presence
@@ -135,9 +139,21 @@ def evaluate(cfg, films, build_dir, step=None):
with ThreadPoolExecutor(max_workers=REPLAY_WORKERS) as ex:
per_film = [m for m in ex.map(_one, films) if m is not None]
n = len(per_film)
if not n:
return {"precision": 0.0, "recall": 0.0, "f1": 0.0, "agreement": 0.0,
"TPI": 0, "FPI": 0, "FPI_misid": 0, "FN": 0}
n_expected = len(films)
# Incomplete coverage (a replay timed out) is scored as a failure, not
# averaged over survivors: dropping the hardest film would otherwise inflate
# the score and let DE reward exactly the configs that cause timeouts. We
# still record the real survivor counts so a truncated eval is diagnosable
# in the trajectory (f1=0.0, films_scored < films_expected).
if n < n_expected:
agg = {"precision": 0.0, "recall": 0.0, "f1": 0.0, "agreement": 0.0,
"TPI": sum(m["TPI"] for m in per_film),
"FPI": sum(m["FPI"] for m in per_film),
"FPI_misid": sum(m["FPI_misid"] for m in per_film),
"FN": sum(m["FN"] for m in per_film)}
agg["films_scored"] = n
agg["films_expected"] = n_expected
return agg
return {"precision": sum(m["precision"] for m in per_film) / n,
"recall": sum(m["recall"] for m in per_film) / n,
"f1": sum(m["f1"] for m in per_film) / n,
@@ -145,7 +161,9 @@ def evaluate(cfg, films, build_dir, step=None):
"TPI": sum(m["TPI"] for m in per_film),
"FPI": sum(m["FPI"] for m in per_film),
"FPI_misid": sum(m["FPI_misid"] for m in per_film),
"FN": sum(m["FN"] for m in per_film)}
"FN": sum(m["FN"] for m in per_film),
"films_scored": n,
"films_expected": n_expected}
def main():