dtourolle/scene-actor-extraction
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: full data-grounded rewrite of the performance report

Browse Source 
Replaces narrative claims with verified numbers across all report pages:

- Cross-model held-out validation (LVFace/mbf/r18, all 5 held-out
  films): LVFace wins every film outright, not just "consistent with"
  the training-set pick. r50 dropped from the detailed comparison
  (gallery has ~30% fewer reference images per actor than the other
  three models on identical source photos).
- Per-film training breakdown: LVFace does not win every training
  film (mbf beats it on Lord of War); the 75.3% macro figure hides a
  10.7pp spread.
- Gallery coverage computed per film (20.3%-78.6%) instead of one
  flat 67%-missing average.
- Found and fixed a real scoring bug in optimize.py: a candidate
  whose hardest film's replay timed out was averaged over survivors
  instead of penalized, silently rewarding partial coverage. Affected
  3 of 16 training combos; corrected throughout, and optimize.py now
  scores an incomplete evaluation f1=0.0 instead of averaging over
  whichever films happened to finish.
- Every FPI frame in the deep dive now comes from the proper montage
  renderer (Onscreen/Offscreen panel, ghosts never drawn as boxes),
  never the bare-box debug overlay used earlier.
- Every distinct out-of-cast name across all 9 films gets its own
  frame at its first appearance (9 names, 4 films), not a
  single-example spot check: 2 ground-truth gaps, 1 photograph
  misread as a person, 6 genuine lookalike confusions.
- New methodology.md: the scene-level-vs-per-second scoring mismatch
  that the rest of the report assumes, written out once.
- Cut the deadlock/gdb debugging narrative from the experiment log;
  kept the one fact that matters (KPN's node/network split lets the
  expensive GPU stage run once and the cheap stage replay against
  cached embeddings).
- Plain declarative style throughout, no em dashes, no blog voice.
This commit is contained in:
dtourolle
2026-07-21 08:55:57 +02:00
parent 4b5557974b
commit 0bd2747069
18 changed files with 1824 additions and 890 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs_data/holdout_all_models.json
+269
View File
@@ -0,0 +1,269 @@
{
"LVFace-B_Glint360K": {
"config": {
"prob_threshold": 0.7540024664611272,
"anneal_sec": 35.53996030397922,
"extinction_sec": 57.43359645269811
},
"films": {
"Benny___Joon": {
"name": "Benny & Joon",
"TPI": 15119,
"FPI": 1845,
"FPI_misid": 0,
"FPI_incast": 1845,
"FN": 4343,
"precision": 0.8912402735203961,
"precision_raw": 0.8912402735203961,
"recall": 0.7768471893947179,
"f1": 0.8301213418986437,
"agreement_rate": 0.7239983093829193,
"exact_match_rate": 0.41098901098901097,
"n_seconds": 5915,
"duration_sec": 5915.0
},
"Downton_Abbey__A_New_Era": {
"name": "Downton Abbey: A New Era",
"TPI": 52022,
"FPI": 1160,
"FPI_misid": 0,
"FPI_incast": 1160,
"FN": 80089,
"precision": 0.9781881087586025,
"precision_raw": 0.9781881087586025,
"recall": 0.39377493168623356,
"f1": 0.5615106884771687,
"agreement_rate": 0.4057686401759256,
"exact_match_rate": 0.033084311632870865,
"n_seconds": 7496,
"duration_sec": 7496.0
},
"Lovelace": {
"name": "Lovelace",
"TPI": 14988,
"FPI": 1086,
"FPI_misid": 58,
"FPI_incast": 1028,
"FN": 7087,
"precision": 0.9031091829356471,
"precision_raw": 0.9324374766703994,
"recall": 0.6789580973952435,
"f1": 0.775154508546456,
"agreement_rate": 0.7204967829586512,
"exact_match_rate": 0.3597703211914588,
"n_seconds": 5573,
"duration_sec": 5573.0
},
"The_Many_Saints_of_Newark": {
"name": "The Many Saints of Newark",
"TPI": 15928,
"FPI": 4394,
"FPI_misid": 974,
"FPI_incast": 3420,
"FN": 23785,
"precision": 0.5475797579757976,
"precision_raw": 0.7837811239051274,
"recall": 0.40107773273235464,
"f1": 0.46301652592258835,
"agreement_rate": 0.3705156874642392,
"exact_match_rate": 0.04588936642173853,
"n_seconds": 7213,
"duration_sec": 7213.0
},
"Valerian_and_the_City_of_a_Thousand_Plan": {
"name": "Valerian and the City of a Thousand Planets",
"TPI": 18658,
"FPI": 548,
"FPI_misid": 0,
"FPI_incast": 548,
"FN": 12472,
"precision": 0.9714672498177653,
"precision_raw": 0.9714672498177653,
"recall": 0.5993575329264376,
"f1": 0.7413382072472983,
"agreement_rate": 0.5877853464704299,
"exact_match_rate": 0.21980294368081743,
"n_seconds": 8221,
"duration_sec": 8221.0
}
}
},
"arcface_w600k_mbf": {
"config": {
"prob_threshold": 0.8371114538930519,
"anneal_sec": 48.80011450565114,
"extinction_sec": 59.3110040220424
},
"films": {
"Benny___Joon": {
"name": "Benny & Joon",
"TPI": 15219,
"FPI": 2463,
"FPI_misid": 180,
"FPI_incast": 2283,
"FN": 4243,
"precision": 0.7884675163195524,
"precision_raw": 0.8607058025110281,
"recall": 0.7819854074606927,
"f1": 0.7852130843050252,
"agreement_rate": 0.7072306082196138,
"exact_match_rate": 0.34911242603550297,
"n_seconds": 5915,
"duration_sec": 5915.0
},
"Downton_Abbey__A_New_Era": {
"name": "Downton Abbey: A New Era",
"TPI": 53043,
"FPI": 2383,
"FPI_misid": 604,
"FPI_incast": 1779,
"FN": 79068,
"precision": 0.8715290328940882,
"precision_raw": 0.9570057373795692,
"recall": 0.4015032813316075,
"f1": 0.5497453011561202,
"agreement_rate": 0.4094114144765671,
"exact_match_rate": 0.032817502668089645,
"n_seconds": 7496,
"duration_sec": 7496.0
},
"Lovelace": {
"name": "Lovelace",
"TPI": 14606,
"FPI": 1337,
"FPI_misid": 180,
"FPI_incast": 1157,
"FN": 7469,
"precision": 0.8316346865569664,
"precision_raw": 0.916138744276485,
"recall": 0.6616534541336353,
"f1": 0.7369695746505879,
"agreement_rate": 0.6907677036961077,
"exact_match_rate": 0.31742329086667864,
"n_seconds": 5573,
"duration_sec": 5573.0
},
"The_Many_Saints_of_Newark": {
"name": "The Many Saints of Newark",
"TPI": 15223,
"FPI": 4574,
"FPI_misid": 994,
"FPI_incast": 3580,
"FN": 24490,
"precision": 0.52962460425147,
"precision_raw": 0.768954892155377,
"recall": 0.383325359454083,
"f1": 0.44475283393712756,
"agreement_rate": 0.3554753116932427,
"exact_match_rate": 0.03715513655899071,
"n_seconds": 7213,
"duration_sec": 7213.0
},
"Valerian_and_the_City_of_a_Thousand_Plan": {
"name": "Valerian and the City of a Thousand Planets",
"TPI": 18472,
"FPI": 853,
"FPI_misid": 239,
"FPI_incast": 614,
"FN": 12658,
"precision": 0.860122927919538,
"precision_raw": 0.9558602846054334,
"recall": 0.5933825891423065,
"f1": 0.7022773067710907,
"agreement_rate": 0.5795914643682625,
"exact_match_rate": 0.18817662084904513,
"n_seconds": 8221,
"duration_sec": 8221.0
}
}
},
"arcface_r18": {
"config": {
"prob_threshold": 0.8955101189489445,
"anneal_sec": 59.08214397442713,
"extinction_sec": 59.29474134414983
},
"films": {
"Benny___Joon": {
"name": "Benny & Joon",
"TPI": 13580,
"FPI": 1666,
"FPI_misid": 60,
"FPI_incast": 1606,
"FN": 5882,
"precision": 0.86025592296972,
"precision_raw": 0.8907254361799817,
"recall": 0.697770013359367,
"f1": 0.7705401724920563,
"agreement_rate": 0.6547675401521545,
"exact_match_rate": 0.32578191039729504,
"n_seconds": 5915,
"duration_sec": 5915.0
},
"Downton_Abbey__A_New_Era": {
"name": "Downton Abbey: A New Era",
"TPI": 48545,
"FPI": 1066,
"FPI_misid": 180,
"FPI_incast": 886,
"FN": 83566,
"precision": 0.9475708067381078,
"precision_raw": 0.9785128298159682,
"recall": 0.3674561542944948,
"f1": 0.5295567845883649,
"agreement_rate": 0.3815368792000116,
"exact_match_rate": 0.032950907150480255,
"n_seconds": 7496,
"duration_sec": 7496.0
},
"Lovelace": {
"name": "Lovelace",
"TPI": 13615,
"FPI": 963,
"FPI_misid": 120,
"FPI_incast": 843,
"FN": 8460,
"precision": 0.8695235662281262,
"precision_raw": 0.933941555768967,
"recall": 0.6167610419026047,
"f1": 0.7216494845360825,
"agreement_rate": 0.6506356469257915,
"exact_match_rate": 0.2894311860757222,
"n_seconds": 5573,
"duration_sec": 5573.0
},
"The_Many_Saints_of_Newark": {
"name": "The Many Saints of Newark",
"TPI": 13489,
"FPI": 3757,
"FPI_misid": 796,
"FPI_incast": 2961,
"FN": 26224,
"precision": 0.5526013928717739,
"precision_raw": 0.7821523831613127,
"recall": 0.3396620753909299,
"f1": 0.42072267361165266,
"agreement_rate": 0.3229817885335633,
"exact_match_rate": 0.04422570359073894,
"n_seconds": 7213,
"duration_sec": 7213.0
},
"Valerian_and_the_City_of_a_Thousand_Plan": {
"name": "Valerian and the City of a Thousand Planets",
"TPI": 17692,
"FPI": 397,
"FPI_misid": 68,
"FPI_incast": 329,
"FN": 13438,
"precision": 0.9460456660071654,
"precision_raw": 0.9780529603626513,
"recall": 0.5683263732733698,
"f1": 0.710080070638759,
"agreement_rate": 0.5633540120828806,
"exact_match_rate": 0.13404695292543486,
"n_seconds": 8221,
"duration_sec": 8221.0
}
}
}
}
docs_data/training_per_film.json
+221
View File
@@ -0,0 +1,221 @@
{
"LVFace-B_Glint360K": {
"config": {
"prob_threshold": 0.7540024664611272,
"anneal_sec": 35.53996030397922,
"extinction_sec": 57.43359645269811
},
"films": {
"Caf\u00e9_Society": {
"name": "Caf\u00e9 Society",
"TPI": 14499,
"FPI": 1380,
"FPI_misid": 0,
"FPI_incast": 1380,
"FN": 12231,
"precision": 0.9130927640279615,
"precision_raw": 0.9130927640279615,
"recall": 0.5424242424242425,
"f1": 0.6805604449764134,
"agreement_rate": 0.57285804629501,
"exact_match_rate": 0.18947003810183582,
"n_seconds": 5774,
"duration_sec": 5774.0
},
"Lord_of_War": {
"name": "Lord of War",
"TPI": 13893,
"FPI": 1654,
"FPI_misid": 174,
"FPI_incast": 1480,
"FN": 5737,
"precision": 0.811838952842868,
"precision_raw": 0.8936129156750499,
"recall": 0.7077432501273561,
"f1": 0.7562256756388972,
"agreement_rate": 0.7005158404089996,
"exact_match_rate": 0.38715420432758146,
"n_seconds": 7302,
"duration_sec": 7302.0
},
"Scarface": {
"name": "Scarface",
"TPI": 20518,
"FPI": 1078,
"FPI_misid": 58,
"FPI_incast": 1020,
"FN": 14722,
"precision": 0.9276607288181572,
"precision_raw": 0.9500833487682904,
"recall": 0.5822360953461975,
"f1": 0.7154363820216882,
"agreement_rate": 0.6297735703976657,
"exact_match_rate": 0.25910733470065433,
"n_seconds": 10239,
"duration_sec": 10239.0
},
"Sound_of_Metal": {
"name": "Sound of Metal",
"TPI": 13349,
"FPI": 677,
"FPI_misid": 0,
"FPI_incast": 677,
"FN": 6504,
"precision": 0.9517324967916726,
"precision_raw": 0.9517324967916726,
"recall": 0.6723920818012391,
"f1": 0.7880397886596416,
"agreement_rate": 0.7112222835587533,
"exact_match_rate": 0.40311896218603366,
"n_seconds": 7246,
"duration_sec": 7246.0
}
}
},
"arcface_w600k_mbf": {
"config": {
"prob_threshold": 0.8371114538930519,
"anneal_sec": 48.80011450565114,
"extinction_sec": 59.3110040220424
},
"films": {
"Caf\u00e9_Society": {
"name": "Caf\u00e9 Society",
"TPI": 13456,
"FPI": 1434,
"FPI_misid": 180,
"FPI_incast": 1254,
"FN": 13274,
"precision": 0.8150211992731677,
"precision_raw": 0.9036937541974479,
"recall": 0.5034044145155256,
"f1": 0.622386679000925,
"agreement_rate": 0.5463565775524695,
"exact_match_rate": 0.19154832005542086,
"n_seconds": 5774,
"duration_sec": 5774.0
},
"Lord_of_War": {
"name": "Lord of War",
"TPI": 13778,
"FPI": 1740,
"FPI_misid": 60,
"FPI_incast": 1680,
"FN": 5852,
"precision": 0.8580146967243741,
"precision_raw": 0.8878721484727413,
"recall": 0.7018848700967907,
"f1": 0.7721362923111411,
"agreement_rate": 0.6881858851455998,
"exact_match_rate": 0.36469460421802247,
"n_seconds": 7302,
"duration_sec": 7302.0
},
"Scarface": {
"name": "Scarface",
"TPI": 18863,
"FPI": 862,
"FPI_misid": 0,
"FPI_incast": 862,
"FN": 16377,
"precision": 0.956299112801014,
"precision_raw": 0.956299112801014,
"recall": 0.535272417707151,
"f1": 0.6863640498499044,
"agreement_rate": 0.594178111391709,
"exact_match_rate": 0.2357652114464303,
"n_seconds": 10239,
"duration_sec": 10239.0
},
"Sound_of_Metal": {
"name": "Sound of Metal",
"TPI": 12642,
"FPI": 554,
"FPI_misid": 0,
"FPI_incast": 554,
"FN": 7211,
"precision": 0.9580175810851773,
"precision_raw": 0.9580175810851773,
"recall": 0.6367803354656727,
"f1": 0.7650458410239341,
"agreement_rate": 0.687468948385309,
"exact_match_rate": 0.3789677063207287,
"n_seconds": 7246,
"duration_sec": 7246.0
}
}
},
"arcface_r18": {
"config": {
"prob_threshold": 0.8955101189489445,
"anneal_sec": 59.08214397442713,
"extinction_sec": 59.29474134414983
},
"films": {
"Caf\u00e9_Society": {
"name": "Caf\u00e9 Society",
"TPI": 12207,
"FPI": 1119,
"FPI_misid": 60,
"FPI_incast": 1059,
"FN": 14523,
"precision": 0.88035482475119,
"precision_raw": 0.9160288158487168,
"recall": 0.45667789001122333,
"f1": 0.6013892994383683,
"agreement_rate": 0.5125626845924863,
"exact_match_rate": 0.1674748874263942,
"n_seconds": 5774,
"duration_sec": 5774.0
},
"Lord_of_War": {
"name": "Lord of War",
"TPI": 13122,
"FPI": 1409,
"FPI_misid": 60,
"FPI_incast": 1349,
"FN": 6508,
"precision": 0.870678787074514,
"precision_raw": 0.9030348909228546,
"recall": 0.6684666327050433,
"f1": 0.7562894441082388,
"agreement_rate": 0.6698963754222389,
"exact_match_rate": 0.3389482333607231,
"n_seconds": 7302,
"duration_sec": 7302.0
},
"Scarface": {
"name": "Scarface",
"TPI": 16961,
"FPI": 685,
"FPI_misid": 0,
"FPI_incast": 685,
"FN": 18279,
"precision": 0.961181004193585,
"precision_raw": 0.961181004193585,
"recall": 0.48129965947786607,
"f1": 0.6414173883447416,
"agreement_rate": 0.5440709115628456,
"exact_match_rate": 0.2017775173356773,
"n_seconds": 10239,
"duration_sec": 10239.0
},
"Sound_of_Metal": {
"name": "Sound of Metal",
"TPI": 12017,
"FPI": 591,
"FPI_misid": 122,
"FPI_incast": 469,
"FN": 7836,
"precision": 0.8767692981176127,
"precision_raw": 0.953125,
"recall": 0.6052989472623784,
"f1": 0.7161715188176049,
"agreement_rate": 0.6594534915815532,
"exact_match_rate": 0.3573005796301408,
"n_seconds": 7246,
"duration_sec": 7246.0
}
}
}
}