dtourolle/scene-actor-extraction
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: full data-grounded rewrite of the performance report

Browse Source 
Replaces narrative claims with verified numbers across all report pages:

- Cross-model held-out validation (LVFace/mbf/r18, all 5 held-out
  films): LVFace wins every film outright, not just "consistent with"
  the training-set pick. r50 dropped from the detailed comparison
  (gallery has ~30% fewer reference images per actor than the other
  three models on identical source photos).
- Per-film training breakdown: LVFace does not win every training
  film (mbf beats it on Lord of War); the 75.3% macro figure hides a
  10.7pp spread.
- Gallery coverage computed per film (20.3%-78.6%) instead of one
  flat 67%-missing average.
- Found and fixed a real scoring bug in optimize.py: a candidate
  whose hardest film's replay timed out was averaged over survivors
  instead of penalized, silently rewarding partial coverage. Affected
  3 of 16 training combos; corrected throughout, and optimize.py now
  scores an incomplete evaluation f1=0.0 instead of averaging over
  whichever films happened to finish.
- Every FPI frame in the deep dive now comes from the proper montage
  renderer (Onscreen/Offscreen panel, ghosts never drawn as boxes),
  never the bare-box debug overlay used earlier.
- Every distinct out-of-cast name across all 9 films gets its own
  frame at its first appearance (9 names, 4 films), not a
  single-example spot check: 2 ground-truth gaps, 1 photograph
  misread as a person, 6 genuine lookalike confusions.
- New methodology.md: the scene-level-vs-per-second scoring mismatch
  that the rest of the report assumes, written out once.
- Cut the deadlock/gdb debugging narrative from the experiment log;
  kept the one fact that matters (KPN's node/network split lets the
  expensive GPU stage run once and the cheap stage replay against
  cached embeddings).
- Plain declarative style throughout, no em dashes, no blog voice.
This commit is contained in:
dtourolle
2026-07-21 08:55:57 +02:00
parent 4b5557974b
commit 0bd2747069
18 changed files with 1824 additions and 890 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs_data/holdout_all_models.json
+269
View File
@@ -0,0 +1,269 @@
{
"LVFace-B_Glint360K": {
"config": {
"prob_threshold": 0.7540024664611272,
"anneal_sec": 35.53996030397922,
"extinction_sec": 57.43359645269811
},
"films": {
"Benny___Joon": {
"name": "Benny & Joon",
"TPI": 15119,
"FPI": 1845,
"FPI_misid": 0,
"FPI_incast": 1845,
"FN": 4343,
"precision": 0.8912402735203961,
"precision_raw": 0.8912402735203961,
"recall": 0.7768471893947179,
"f1": 0.8301213418986437,
"agreement_rate": 0.7239983093829193,
"exact_match_rate": 0.41098901098901097,
"n_seconds": 5915,
"duration_sec": 5915.0
},
"Downton_Abbey__A_New_Era": {
"name": "Downton Abbey: A New Era",
"TPI": 52022,
"FPI": 1160,
"FPI_misid": 0,
"FPI_incast": 1160,
"FN": 80089,
"precision": 0.9781881087586025,
"precision_raw": 0.9781881087586025,
"recall": 0.39377493168623356,
"f1": 0.5615106884771687,
"agreement_rate": 0.4057686401759256,
"exact_match_rate": 0.033084311632870865,
"n_seconds": 7496,
"duration_sec": 7496.0
},
"Lovelace": {
"name": "Lovelace",
"TPI": 14988,
"FPI": 1086,
"FPI_misid": 58,
"FPI_incast": 1028,
"FN": 7087,
"precision": 0.9031091829356471,
"precision_raw": 0.9324374766703994,
"recall": 0.6789580973952435,
"f1": 0.775154508546456,
"agreement_rate": 0.7204967829586512,
"exact_match_rate": 0.3597703211914588,
"n_seconds": 5573,
"duration_sec": 5573.0
},
"The_Many_Saints_of_Newark": {
"name": "The Many Saints of Newark",
"TPI": 15928,
"FPI": 4394,
"FPI_misid": 974,
"FPI_incast": 3420,
"FN": 23785,
"precision": 0.5475797579757976,
"precision_raw": 0.7837811239051274,
"recall": 0.40107773273235464,
"f1": 0.46301652592258835,
"agreement_rate": 0.3705156874642392,
"exact_match_rate": 0.04588936642173853,
"n_seconds": 7213,
"duration_sec": 7213.0
},
"Valerian_and_the_City_of_a_Thousand_Plan": {
"name": "Valerian and the City of a Thousand Planets",
"TPI": 18658,
"FPI": 548,
"FPI_misid": 0,
"FPI_incast": 548,
"FN": 12472,
"precision": 0.9714672498177653,
"precision_raw": 0.9714672498177653,
"recall": 0.5993575329264376,
"f1": 0.7413382072472983,
"agreement_rate": 0.5877853464704299,
"exact_match_rate": 0.21980294368081743,
"n_seconds": 8221,
"duration_sec": 8221.0
}
}
},
"arcface_w600k_mbf": {
"config": {
"prob_threshold": 0.8371114538930519,
"anneal_sec": 48.80011450565114,
"extinction_sec": 59.3110040220424
},
"films": {
"Benny___Joon": {
"name": "Benny & Joon",
"TPI": 15219,
"FPI": 2463,
"FPI_misid": 180,
"FPI_incast": 2283,
"FN": 4243,
"precision": 0.7884675163195524,
"precision_raw": 0.8607058025110281,
"recall": 0.7819854074606927,
"f1": 0.7852130843050252,
"agreement_rate": 0.7072306082196138,
"exact_match_rate": 0.34911242603550297,
"n_seconds": 5915,
"duration_sec": 5915.0
},
"Downton_Abbey__A_New_Era": {
"name": "Downton Abbey: A New Era",
"TPI": 53043,
"FPI": 2383,
"FPI_misid": 604,
"FPI_incast": 1779,
"FN": 79068,
"precision": 0.8715290328940882,
"precision_raw": 0.9570057373795692,
"recall": 0.4015032813316075,
"f1": 0.5497453011561202,
"agreement_rate": 0.4094114144765671,
"exact_match_rate": 0.032817502668089645,
"n_seconds": 7496,
"duration_sec": 7496.0
},
"Lovelace": {
"name": "Lovelace",
"TPI": 14606,
"FPI": 1337,
"FPI_misid": 180,
"FPI_incast": 1157,
"FN": 7469,
"precision": 0.8316346865569664,
"precision_raw": 0.916138744276485,
"recall": 0.6616534541336353,
"f1": 0.7369695746505879,
"agreement_rate": 0.6907677036961077,
"exact_match_rate": 0.31742329086667864,
"n_seconds": 5573,
"duration_sec": 5573.0
},
"The_Many_Saints_of_Newark": {
"name": "The Many Saints of Newark",
"TPI": 15223,
"FPI": 4574,
"FPI_misid": 994,
"FPI_incast": 3580,
"FN": 24490,
"precision": 0.52962460425147,
"precision_raw": 0.768954892155377,
"recall": 0.383325359454083,
"f1": 0.44475283393712756,
"agreement_rate": 0.3554753116932427,
"exact_match_rate": 0.03715513655899071,
"n_seconds": 7213,
"duration_sec": 7213.0
},
"Valerian_and_the_City_of_a_Thousand_Plan": {
"name": "Valerian and the City of a Thousand Planets",
"TPI": 18472,
"FPI": 853,
"FPI_misid": 239,
"FPI_incast": 614,
"FN": 12658,
"precision": 0.860122927919538,
"precision_raw": 0.9558602846054334,
"recall": 0.5933825891423065,
"f1": 0.7022773067710907,
"agreement_rate": 0.5795914643682625,
"exact_match_rate": 0.18817662084904513,
"n_seconds": 8221,
"duration_sec": 8221.0
}
}
},
"arcface_r18": {
"config": {
"prob_threshold": 0.8955101189489445,
"anneal_sec": 59.08214397442713,
"extinction_sec": 59.29474134414983
},
"films": {
"Benny___Joon": {
"name": "Benny & Joon",
"TPI": 13580,
"FPI": 1666,
"FPI_misid": 60,
"FPI_incast": 1606,
"FN": 5882,
"precision": 0.86025592296972,
"precision_raw": 0.8907254361799817,
"recall": 0.697770013359367,
"f1": 0.7705401724920563,
"agreement_rate": 0.6547675401521545,
"exact_match_rate": 0.32578191039729504,
"n_seconds": 5915,
"duration_sec": 5915.0
},
"Downton_Abbey__A_New_Era": {
"name": "Downton Abbey: A New Era",
"TPI": 48545,
"FPI": 1066,
"FPI_misid": 180,
"FPI_incast": 886,
"FN": 83566,
"precision": 0.9475708067381078,
"precision_raw": 0.9785128298159682,
"recall": 0.3674561542944948,
"f1": 0.5295567845883649,
"agreement_rate": 0.3815368792000116,
"exact_match_rate": 0.032950907150480255,
"n_seconds": 7496,
"duration_sec": 7496.0
},
"Lovelace": {
"name": "Lovelace",
"TPI": 13615,
"FPI": 963,
"FPI_misid": 120,
"FPI_incast": 843,
"FN": 8460,
"precision": 0.8695235662281262,
"precision_raw": 0.933941555768967,
"recall": 0.6167610419026047,
"f1": 0.7216494845360825,
"agreement_rate": 0.6506356469257915,
"exact_match_rate": 0.2894311860757222,
"n_seconds": 5573,
"duration_sec": 5573.0
},
"The_Many_Saints_of_Newark": {
"name": "The Many Saints of Newark",
"TPI": 13489,
"FPI": 3757,
"FPI_misid": 796,
"FPI_incast": 2961,
"FN": 26224,
"precision": 0.5526013928717739,
"precision_raw": 0.7821523831613127,
"recall": 0.3396620753909299,
"f1": 0.42072267361165266,
"agreement_rate": 0.3229817885335633,
"exact_match_rate": 0.04422570359073894,
"n_seconds": 7213,
"duration_sec": 7213.0
},
"Valerian_and_the_City_of_a_Thousand_Plan": {
"name": "Valerian and the City of a Thousand Planets",
"TPI": 17692,
"FPI": 397,
"FPI_misid": 68,
"FPI_incast": 329,
"FN": 13438,
"precision": 0.9460456660071654,
"precision_raw": 0.9780529603626513,
"recall": 0.5683263732733698,
"f1": 0.710080070638759,
"agreement_rate": 0.5633540120828806,
"exact_match_rate": 0.13404695292543486,
"n_seconds": 8221,
"duration_sec": 8221.0
}
}
}
}