docs: full data-grounded rewrite of the performance report
Replaces narrative claims with verified numbers across all report pages: - Cross-model held-out validation (LVFace/mbf/r18, all 5 held-out films): LVFace wins every film outright, not just "consistent with" the training-set pick. r50 dropped from the detailed comparison (gallery has ~30% fewer reference images per actor than the other three models on identical source photos). - Per-film training breakdown: LVFace does not win every training film (mbf beats it on Lord of War); the 75.3% macro figure hides a 10.7pp spread. - Gallery coverage computed per film (20.3%-78.6%) instead of one flat 67%-missing average. - Found and fixed a real scoring bug in optimize.py: a candidate whose hardest film's replay timed out was averaged over survivors instead of penalized, silently rewarding partial coverage. Affected 3 of 16 training combos; corrected throughout, and optimize.py now scores an incomplete evaluation f1=0.0 instead of averaging over whichever films happened to finish. - Every FPI frame in the deep dive now comes from the proper montage renderer (Onscreen/Offscreen panel, ghosts never drawn as boxes), never the bare-box debug overlay used earlier. - Every distinct out-of-cast name across all 9 films gets its own frame at its first appearance (9 names, 4 films), not a single-example spot check: 2 ground-truth gaps, 1 photograph misread as a person, 6 genuine lookalike confusions. - New methodology.md: the scene-level-vs-per-second scoring mismatch that the rest of the report assumes, written out once. - Cut the deadlock/gdb debugging narrative from the experiment log; kept the one fact that matters (KPN's node/network split lets the expensive GPU stage run once and the cheap stage replay against cached embeddings). - Plain declarative style throughout, no em dashes, no blog voice.
This commit is contained in:
@@ -106,8 +106,12 @@ def evaluate(cfg, films, build_dir, step=None):
|
||||
"""Objective = MACRO-mean over films of each film's duration-weighted per-scene F1.
|
||||
|
||||
Each film's replay runs in a subprocess (timeout-guarded) to survive the
|
||||
intermittent ROCm teardown deadlock. A film whose replay times out is dropped
|
||||
from the average rather than hanging the whole sweep.
|
||||
intermittent ROCm teardown deadlock. If ANY film's replay times out, this
|
||||
evaluation is scored f1=0.0 (see below) rather than averaging over the
|
||||
survivors — a partial-coverage eval must never look better than a complete
|
||||
one, or DE will converge onto configs that make the hardest film time out.
|
||||
(An earlier version averaged over survivors, which silently rewarded
|
||||
truncation; the rep4 `mbf_full_noexp` winner was one such corrupted eval.)
|
||||
|
||||
UNIFORM PER-SECOND scoring (second_score.py): every second of the film is sampled;
|
||||
GT(t) = the cast of the X-Ray scene containing t, Pred(t) = actors whose presence
|
||||
@@ -135,9 +139,21 @@ def evaluate(cfg, films, build_dir, step=None):
|
||||
with ThreadPoolExecutor(max_workers=REPLAY_WORKERS) as ex:
|
||||
per_film = [m for m in ex.map(_one, films) if m is not None]
|
||||
n = len(per_film)
|
||||
if not n:
|
||||
return {"precision": 0.0, "recall": 0.0, "f1": 0.0, "agreement": 0.0,
|
||||
"TPI": 0, "FPI": 0, "FPI_misid": 0, "FN": 0}
|
||||
n_expected = len(films)
|
||||
# Incomplete coverage (a replay timed out) is scored as a failure, not
|
||||
# averaged over survivors: dropping the hardest film would otherwise inflate
|
||||
# the score and let DE reward exactly the configs that cause timeouts. We
|
||||
# still record the real survivor counts so a truncated eval is diagnosable
|
||||
# in the trajectory (f1=0.0, films_scored < films_expected).
|
||||
if n < n_expected:
|
||||
agg = {"precision": 0.0, "recall": 0.0, "f1": 0.0, "agreement": 0.0,
|
||||
"TPI": sum(m["TPI"] for m in per_film),
|
||||
"FPI": sum(m["FPI"] for m in per_film),
|
||||
"FPI_misid": sum(m["FPI_misid"] for m in per_film),
|
||||
"FN": sum(m["FN"] for m in per_film)}
|
||||
agg["films_scored"] = n
|
||||
agg["films_expected"] = n_expected
|
||||
return agg
|
||||
return {"precision": sum(m["precision"] for m in per_film) / n,
|
||||
"recall": sum(m["recall"] for m in per_film) / n,
|
||||
"f1": sum(m["f1"] for m in per_film) / n,
|
||||
@@ -145,7 +161,9 @@ def evaluate(cfg, films, build_dir, step=None):
|
||||
"TPI": sum(m["TPI"] for m in per_film),
|
||||
"FPI": sum(m["FPI"] for m in per_film),
|
||||
"FPI_misid": sum(m["FPI_misid"] for m in per_film),
|
||||
"FN": sum(m["FN"] for m in per_film)}
|
||||
"FN": sum(m["FN"] for m in per_film),
|
||||
"films_scored": n,
|
||||
"films_expected": n_expected}
|
||||
|
||||
|
||||
def main():
|
||||
|
||||
Reference in New Issue
Block a user