From ea922356f1e2b9c98e86616d0e258d5dbc59a258 Mon Sep 17 00:00:00 2001 From: Duncan Tourolle Date: Sun, 9 Aug 2026 10:48:45 +0200 Subject: [PATCH] docs: archive the July 2026 report; new methodology for the opencv5 run The July report (4-model ArcFace/LVFace bake-off, pre-opencv5 framework, 3-film training + held-out validation) is superseded by the opencv5 build: single-model LVFace-B, a 6-knob DE sweep over all 9 films, flood-fill presence, and the registry/decode fixes. Rather than overwrite it, archive it date-suffixed and start the current report fresh. - Rename the six July result pages to *-2026-07.md, rewrite their intra-archive cross-links, and add an "Archived (July 2026)" banner to each. - mkdocs nav: current report at top, the July set under an Archive section. - New docs/methodology.md for the opencv5 run: corrects the withdrawn anneal_sec/extinction_sec presence bridging (windows are now [first_seen, last_seen], AR-012/013), documents the two presence modes (track_extent / flood), and records that every eval scores all 9 films. The current experiment log (model-bakeoff.md) and Home rewrite land once the DE sweep converges and the final optimum is known. --- docs/{best-model.md => best-model-2026-07.md} | 8 +- ...lery-scope.md => gallery-scope-2026-07.md} | 8 +- ...ep-dive.md => lvface-deep-dive-2026-07.md} | 10 +- docs/methodology-2026-07.md | 136 ++++++++++++++++ docs/methodology.md | 148 +++++++++--------- ...el-bakeoff.md => model-bakeoff-2026-07.md} | 16 +- ...expansion.md => pose-expansion-2026-07.md} | 8 +- mkdocs.yml | 12 +- 8 files changed, 243 insertions(+), 103 deletions(-) rename docs/{best-model.md => best-model-2026-07.md} (89%) rename docs/{gallery-scope.md => gallery-scope-2026-07.md} (85%) rename docs/{lvface-deep-dive.md => lvface-deep-dive-2026-07.md} (96%) create mode 100644 docs/methodology-2026-07.md rename docs/{model-bakeoff.md => model-bakeoff-2026-07.md} (95%) rename docs/{pose-expansion.md => pose-expansion-2026-07.md} (91%) diff --git a/docs/best-model.md b/docs/best-model-2026-07.md similarity index 89% rename from docs/best-model.md rename to docs/best-model-2026-07.md index 5f433c2..6ffa3b4 100644 --- a/docs/best-model.md +++ b/docs/best-model-2026-07.md @@ -1,10 +1,12 @@ +> **Archived (July 2026).** This report covers the pre-opencv5 framework and the 4-model ArcFace/LVFace bake-off. It is superseded by the current [experiment log](model-bakeoff.md) for the opencv5 build. Kept for provenance; the numbers here are historical. + # Which embedding model is best? Three ArcFace variants (w600k-R50, R18, w600k-MBF) and LVFace-B (Glint360K, 455MB) were compared. r50 is excluded from the training/held-out comparison below; its gallery has roughly 30% fewer reference images per actor than the other three on the identical source photos, which confounds a direct score -comparison (see [the full experiment log](model-bakeoff.md) for detail). It +comparison (see [the full experiment log](model-bakeoff-2026-07.md) for detail). It remains in the calibration comparison, which does not depend on the gallery image count. @@ -63,7 +65,7 @@ than general performance. On training data, the ordering is not as clean: mbf beats LVFace on Lord of War (77.2% vs 75.6%), the only film in either table where LVFace does not score highest. LVFace's training-set macro -average (75.3%, see [the full experiment log](model-bakeoff.md)) is not a +average (75.3%, see [the full experiment log](model-bakeoff-2026-07.md)) is not a uniform win across every film it contributes to; the held-out result, where LVFace wins all 5 films outright, is the stronger claim. @@ -79,7 +81,7 @@ not. ![All 12 combos ranked by training-set F1](assets/images/rep4_matrix_f1.png) Best full-gallery combo per model (all three are `full_exp`), from the -training matrix in [the full experiment log](model-bakeoff.md): +training matrix in [the full experiment log](model-bakeoff-2026-07.md): | model | F1 | P | R | misID | |---|---|---|---|---| diff --git a/docs/gallery-scope.md b/docs/gallery-scope-2026-07.md similarity index 85% rename from docs/gallery-scope.md rename to docs/gallery-scope-2026-07.md index 62c50b7..1a04da4 100644 --- a/docs/gallery-scope.md +++ b/docs/gallery-scope-2026-07.md @@ -1,3 +1,5 @@ +> **Archived (July 2026).** This report covers the pre-opencv5 framework and the 4-model ArcFace/LVFace bake-off. It is superseded by the current [experiment log](model-bakeoff.md) for the opencv5 build. Kept for provenance; the numbers here are historical. + # Whole gallery vs. cast-restricted gallery Two ways to run the matcher. Full mode scores every detected face against @@ -8,7 +10,7 @@ top-billed actors) before the matcher runs. ## Result Averaged across the 3 compared models (r50 excluded, see -[the full experiment log](model-bakeoff.md)) and both expansion settings, on +[the full experiment log](model-bakeoff-2026-07.md)) and both expansion settings, on the 4 training films: | scope | F1 | P | R | total misID | @@ -27,7 +29,7 @@ restricted gallery: ![All combos ranked by training-set F1, filled dots are restricted](assets/images/rep4_matrix_f1.png) -See [the full experiment log](model-bakeoff.md) for the complete table. One +See [the full experiment log](model-bakeoff-2026-07.md) for the complete table. One combo reaches zero true out-of-cast misidentifications, `arcface_w600k_mbf_restricted_exp` (F1 76.2%), and it is a restricted one, consistent with restriction, not expansion, being what suppresses cross-film @@ -57,7 +59,7 @@ Building this as a real feature requires: option. - A decision on the fallback case: what happens to a real, uncredited cameo (see the Germar Terrell Gardner and Talia Balsam cases in the - [LVFace deep dive](lvface-deep-dive.md#where-lvface-beat-x-ray)) if the + [LVFace deep dive](lvface-deep-dive-2026-07.md#where-lvface-beat-x-ray)) if the restricted gallery never includes them at all. - Regenerating the restricted-gallery cache whenever a title's Jellyfin cast list changes. diff --git a/docs/lvface-deep-dive.md b/docs/lvface-deep-dive-2026-07.md similarity index 96% rename from docs/lvface-deep-dive.md rename to docs/lvface-deep-dive-2026-07.md index df62cb5..817bef4 100644 --- a/docs/lvface-deep-dive.md +++ b/docs/lvface-deep-dive-2026-07.md @@ -1,12 +1,14 @@ +> **Archived (July 2026).** This report covers the pre-opencv5 framework and the 4-model ArcFace/LVFace bake-off. It is superseded by the current [experiment log](model-bakeoff.md) for the opencv5 build. Kept for provenance; the numbers here are historical. + # Deep dive: LVFace-B Glint360K -LVFace won the model comparison (see [Which model is best?](best-model.md)) +LVFace won the model comparison (see [Which model is best?](best-model-2026-07.md)) and is the shipped default embedder. This page reports how it performs in detail: a baseline of correct output, the two mechanisms behind its errors, and every distinct case where it names someone who is not in the film's credited cast. -Read [How we score against X-Ray](methodology.md) first. X-Ray's ground truth +Read [How we score against X-Ray](methodology-2026-07.md) first. X-Ray's ground truth is scene-level, not per-frame. A name marked correct in the Offscreen column below is the pipeline correctly reporting scene membership, not a workaround. @@ -61,7 +63,7 @@ on the 5 films the optimizer never saw: | macro average | 67.4% | 85.8% | 57.0% | | | | | The `P` column is misID-weighted (each out-of-film name counts 10x in the -denominator; see [methodology](methodology.md#precision-recall-and-the-misid-weighting)). +denominator; see [methodology](methodology-2026-07.md#precision-recall-and-the-misid-weighting)). That weighting is why Many Saints reads 54.7% here despite naming mostly real, present faces: its raw (unweighted) precision is **78.4%**, and the gap is entirely its 974 misIDs paying the 10x penalty. The three zero-misID films @@ -70,7 +72,7 @@ Lovelace, with 58 misIDs, sits 3pp below its raw 93.3%. Held-out F1 is 67.4%, against 75.3% on training, an 8pp drop. The spread between the best and worst held-out film is 37pp. This is not unique to -LVFace: [the full experiment log](model-bakeoff.md#held-out-validation-all-3-models) +LVFace: [the full experiment log](model-bakeoff-2026-07.md#held-out-validation-all-3-models) shows mbf and r18 with the same shape of spread on the same films, at a uniformly lower level. Two mechanisms explain the spread. Both are shown below with frame-level evidence. diff --git a/docs/methodology-2026-07.md b/docs/methodology-2026-07.md new file mode 100644 index 0000000..2e553ec --- /dev/null +++ b/docs/methodology-2026-07.md @@ -0,0 +1,136 @@ +> **Archived (July 2026).** This report covers the pre-opencv5 framework and the 4-model ArcFace/LVFace bake-off. It is superseded by the current [experiment log](model-bakeoff.md) for the opencv5 build. Kept for provenance; the numbers here are historical. + +# How we score against X-Ray + +Every number in this report, every F1 and misID count, comes from one +comparison. The comparison has a mismatch at its core that shapes nearly +every finding in this report: the ground truth is scene-level, the +pipeline's output is per-second, and the two do not mean the same thing. +This page documents that comparison once, so the findings pages can rely on +it without re-explaining it. + +## What Amazon X-Ray records + +X-Ray ships three tables per film: `scenes.csv` (a list of `[start, end]` +timespans), `people_in_scenes.csv` (which actors are credited in each +scene), and `people.csv` (actor identities). There is no per-frame or +per-second annotation anywhere in X-Ray. A scene might run 45 seconds, and +X-Ray records one cast list for the entire span, not "on screen from +second 12 to second 30." + +To compare this against per-second predictions, `second_score.py` expands +every scene into per-second ground truth by copying the whole scene's cast +list onto every second inside it: + +```python +for sn, (t0, t1) in spans.items(): + cast = scene_cast.get(sn, []) + for t in range(int(t0), int(t1)): + timeline[t] = cast +``` + +That is the entire mechanism. If X-Ray credits five actors to a 30-second +scene, all five count as ground truth present for all 30 seconds, including +seconds where only one of them is on screen. This is not a simplification +introduced by the pipeline; it is the only reading of X-Ray's data that is +possible, because X-Ray itself does not record anything finer-grained. + +## Why an offscreen name can be scored correct + +A name listed under Offscreen with a correct (green) label is not the +pipeline guessing or padding its score. It is the pipeline correctly +answering the question X-Ray actually asks: is this actor part of this +scene. It answers that question using a presence window (`[start, end]`, +held open across cuts by `anneal_sec` and `extinction_sec`), which matches +X-Ray's scene-level semantics more closely than a raw per-frame detection +would. + +A system that only reported "this actor is visible in this exact frame" +would score worse against X-Ray's scene-level ground truth, producing a +false negative every time the camera cuts away from a character who is +still present in the scene. Not because it is wrong about the world, but +because it would be answering a stricter, different question than the one +X-Ray's data supports. The presence-window design exists specifically to +answer X-Ray's actual question. + +## What this resolves and what it does not + +This resolves the semantic mismatch between a scene and an instant. It does +not resolve two other limitations, both discussed in the +[LVFace deep dive](lvface-deep-dive-2026-07.md). + +**The face-vs-presence ceiling.** X-Ray credits scene membership regardless +of whether a face is ever visible: background crew, characters shot from +behind, voice-only presence. No amount of bridging recovers a face that +never appears on screen. This is a hard ceiling on recall, not a defect. + +**Extinction bridging can overshoot.** The same presence-window mechanism +that correctly answers "still in this scene" during a normal cut can also +bridge across a scene boundary it has no way to detect. A hard cut into a +different scene with no faces, such as closing credits, carries the +previous scene's identities forward until the window expires. This is the +mechanism behind Downton Abbey's recall collapse, documented in the deep +dive. + +## Precision, recall, and the misID weighting + +Per sampled second `t`: + +**TPI** (true positive instances): actors both X-Ray and the pipeline agree +are present. + +**FPI** (false positive instances): actors the pipeline reports that are +not in X-Ray's cast for this second. Split into two categories: + +- **FPI_incast**: the actor is in the film's cast, just not credited to + this particular scene. A timing or boundary slip. +- **FPI_misid**: the actor is not in the film's cast at all. A genuine + wrong-identity error, weighted 10x in the precision objective, because + naming someone who is not even in the film is a categorically worse + error than a few seconds of scene-boundary slop. + +!!! note "Every headline `P` and `F1` is misID-weighted" + + The precision reported throughout this report, and therefore the F1 + derived from it, puts each `FPI_misid` into the denominator **10 times** + (`precision = TPI / (TPI + FPI_incast + 10·FPI_misid)`, + [`second_score.py`](https://REPOLINK/scripts/optimizer/second_score.py)). + This is deliberate: the whole point is to punish naming an out-of-film + actor far harder than a scene-boundary slip. But it means the `P` column + is not raw precision, and a misID-heavy film's `P` is depressed + super-linearly. `second_score.py` also emits an unweighted `precision_raw` + (always ≥ the weighted `P`); where the gap matters, The Many Saints of + Newark, weighted `P` 54.7% vs. raw 78.4%, the [LVFace deep dive](lvface-deep-dive-2026-07.md) + reports both. When comparing `P` across films, remember you are comparing a + quantity that penalizes misIDs, not just a hit rate. + +**FN** (false negatives): actors X-Ray lists that the pipeline never +reports, counted only for actors who have a gallery reference embedding. +Across the 9-film benchmark, coverage of X-Ray's credited cast ranges from +20% to 79% by film (see +[the full experiment log](model-bakeoff-2026-07.md#gallery-coverage-per-film)); an +actor with no reference photo can never be recognized regardless of model +quality, and counting them as a miss would penalize gallery coverage, not +recognition accuracy. + +Two further numbers are reported alongside F1: + +**agreement_rate**: mean per-second Jaccard overlap +(`|Pred ∩ GT| / |Pred ∪ GT|`), partial credit. Naming 2 of 3 present actors +scores 2/3, not 0. + +**exact_match_rate**: the fraction of sampled seconds where the pipeline's +named set exactly equals X-Ray's, no partial credit. Far harsher, and +dominated by recall, since any single missed actor zeroes that second. + +## Reproduce + +```bash +python3 scripts/optimizer/second_score.py \ + --pred pred.json --xray experiments/xray/.../ \ + --gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5 +``` + +See also [the full experiment log](model-bakeoff-2026-07.md) for how `pred.json` is +produced, and the [LVFace deep dive](lvface-deep-dive-2026-07.md) for what these +mechanisms look like frame by frame. diff --git a/docs/methodology.md b/docs/methodology.md index e646693..fd5525a 100644 --- a/docs/methodology.md +++ b/docs/methodology.md @@ -1,11 +1,9 @@ # How we score against X-Ray -Every number in this report, every F1 and misID count, comes from one -comparison. The comparison has a mismatch at its core that shapes nearly -every finding in this report: the ground truth is scene-level, the -pipeline's output is per-second, and the two do not mean the same thing. -This page documents that comparison once, so the findings pages can rely on -it without re-explaining it. +Every number in this report comes from one comparison, and that comparison +has a mismatch at its core: the ground truth is scene-level, the pipeline's +output is per-second, and the two do not mean the same thing. This page +documents the comparison once so the findings can rely on it. ## What Amazon X-Ray records @@ -13,12 +11,12 @@ X-Ray ships three tables per film: `scenes.csv` (a list of `[start, end]` timespans), `people_in_scenes.csv` (which actors are credited in each scene), and `people.csv` (actor identities). There is no per-frame or per-second annotation anywhere in X-Ray. A scene might run 45 seconds, and -X-Ray records one cast list for the entire span, not "on screen from -second 12 to second 30." +X-Ray records one cast list for the entire span, not "on screen from second +12 to second 30." -To compare this against per-second predictions, `second_score.py` expands -every scene into per-second ground truth by copying the whole scene's cast -list onto every second inside it: +To compare against per-second predictions, `second_score.py` expands every +scene into per-second ground truth by copying the whole scene's cast list +onto every second inside it: ```python for sn, (t0, t1) in spans.items(): @@ -27,48 +25,46 @@ for sn, (t0, t1) in spans.items(): timeline[t] = cast ``` -That is the entire mechanism. If X-Ray credits five actors to a 30-second -scene, all five count as ground truth present for all 30 seconds, including -seconds where only one of them is on screen. This is not a simplification -introduced by the pipeline; it is the only reading of X-Ray's data that is -possible, because X-Ray itself does not record anything finer-grained. +If X-Ray credits five actors to a 30-second scene, all five count as ground +truth present for all 30 seconds, including seconds where only one is on +screen. This is not a simplification the pipeline introduces; it is the only +reading X-Ray's data supports, because X-Ray records nothing finer. -## Why an offscreen name can be scored correct +## How the pipeline reports presence -A name listed under Offscreen with a correct (green) label is not the -pipeline guessing or padding its score. It is the pipeline correctly -answering the question X-Ray actually asks: is this actor part of this -scene. It answers that question using a presence window (`[start, end]`, -held open across cuts by `anneal_sec` and `extinction_sec`), which matches -X-Ray's scene-level semantics more closely than a raw per-frame detection -would. +A presence claim is one actor owning one time window. How that window is +derived is a tunable choice — a knob the optimizer weighs — with two modes: -A system that only reported "this actor is visible in this exact frame" -would score worse against X-Ray's scene-level ground truth, producing a -false negative every time the camera cuts away from a character who is -still present in the scene. Not because it is wrong about the world, but -because it would be answering a stricter, different question than the one -X-Ray's data supports. The presence-window design exists specifically to -answer X-Ray's actual question. +- **`track_extent` (default).** A claim is exactly `[first_seen, last_seen]` + of a track the actor owned (AR-012), ending at the last sighting and never + after (AR-013). There is no keep-alive: the withdrawn `anneal_sec` and the + scene-tracker `extinction_sec` — which the July report's windows were held + open by — are **gone**. A track that survives its own gaps needs no bridge; + a gap after the final sighting is never claimed. +- **`flood`.** Each claim is snapped to the shot it sits in, so an actor seen + once anywhere in a shot is reported for the whole shot + `[prev_boundary, next_boundary]`. Boundaries come from TransNetV2 shot + detection when available, otherwise from the always-on histogram cut + detector (`is_cut`). This trades precision for recall against X-Ray's + scene-level granularity, and the optimizer decides per run whether it pays. -## What this resolves and what it does not +Do not confuse the surviving `track_extinction_sec` with the withdrawn +scene `extinction_sec`: the former bounds how long a lost track stays +available for **re-association** (a tracking question), and never extends a +presence claim. -This resolves the semantic mismatch between a scene and an instant. It does -not resolve two other limitations, both discussed in the -[LVFace deep dive](lvface-deep-dive.md). +## The two limits this does not resolve **The face-vs-presence ceiling.** X-Ray credits scene membership regardless of whether a face is ever visible: background crew, characters shot from -behind, voice-only presence. No amount of bridging recovers a face that -never appears on screen. This is a hard ceiling on recall, not a defect. +behind, voice-only presence. No face pipeline can recover a face that never +appears, so recall against X-Ray is a structural ceiling, not a defect. -**Extinction bridging can overshoot.** The same presence-window mechanism -that correctly answers "still in this scene" during a normal cut can also -bridge across a scene boundary it has no way to detect. A hard cut into a -different scene with no faces, such as closing credits, carries the -previous scene's identities forward until the window expires. This is the -mechanism behind Downton Abbey's recall collapse, documented in the deep -dive. +**Flood-fill can overshoot.** Snapping to a shot correctly answers "still in +this scene" through an intra-scene cut, but a shot boundary is not a scene +boundary: on a film with sparse cuts, flood-fill can carry an actor across a +long "shot" they only briefly appeared in. This is why flood-fill is a knob, +not a default — its value depends on the film's cut density. ## Precision, recall, and the misID weighting @@ -77,49 +73,46 @@ Per sampled second `t`: **TPI** (true positive instances): actors both X-Ray and the pipeline agree are present. -**FPI** (false positive instances): actors the pipeline reports that are -not in X-Ray's cast for this second. Split into two categories: +**FPI** (false positive instances): actors the pipeline reports that are not +in X-Ray's cast for this second, split into: -- **FPI_incast**: the actor is in the film's cast, just not credited to - this particular scene. A timing or boundary slip. +- **FPI_incast**: the actor is in the film's cast, just not credited to this + scene. A timing or boundary slip. - **FPI_misid**: the actor is not in the film's cast at all. A genuine - wrong-identity error, weighted 10x in the precision objective, because - naming someone who is not even in the film is a categorically worse - error than a few seconds of scene-boundary slop. + wrong-identity error, weighted **10×** in the precision objective, because + naming someone not even in the film is categorically worse than a few + seconds of scene-boundary slop. !!! note "Every headline `P` and `F1` is misID-weighted" - The precision reported throughout this report, and therefore the F1 - derived from it, puts each `FPI_misid` into the denominator **10 times** + Precision puts each `FPI_misid` into the denominator 10 times (`precision = TPI / (TPI + FPI_incast + 10·FPI_misid)`, [`second_score.py`](https://REPOLINK/scripts/optimizer/second_score.py)). - This is deliberate: the whole point is to punish naming an out-of-film - actor far harder than a scene-boundary slip. But it means the `P` column - is not raw precision, and a misID-heavy film's `P` is depressed - super-linearly. `second_score.py` also emits an unweighted `precision_raw` - (always ≥ the weighted `P`); where the gap matters, The Many Saints of - Newark, weighted `P` 54.7% vs. raw 78.4%, the [LVFace deep dive](lvface-deep-dive.md) - reports both. When comparing `P` across films, remember you are comparing a - quantity that penalizes misIDs, not just a hit rate. + This deliberately punishes naming an out-of-film actor far harder than a + boundary slip, so the `P` column is not raw precision and a misID-heavy + film's `P` is depressed super-linearly. -**FN** (false negatives): actors X-Ray lists that the pipeline never -reports, counted only for actors who have a gallery reference embedding. -Across the 9-film benchmark, coverage of X-Ray's credited cast ranges from -20% to 79% by film (see -[the full experiment log](model-bakeoff.md#gallery-coverage-per-film)); an -actor with no reference photo can never be recognized regardless of model -quality, and counting them as a miss would penalize gallery coverage, not -recognition accuracy. +**FN** (false negatives): actors X-Ray lists that the pipeline never reports, +counted **only** for actors who have a gallery reference embedding. An actor +with no reference photo can never be recognized, and counting them as a miss +would measure gallery coverage, not recognition accuracy. -Two further numbers are reported alongside F1: +Two further numbers accompany F1: **agreement_rate**: mean per-second Jaccard overlap -(`|Pred ∩ GT| / |Pred ∪ GT|`), partial credit. Naming 2 of 3 present actors -scores 2/3, not 0. +(`|Pred ∩ GT| / |Pred ∪ GT|`) — partial credit, so naming 2 of 3 present +actors scores 2/3, not 0. -**exact_match_rate**: the fraction of sampled seconds where the pipeline's -named set exactly equals X-Ray's, no partial credit. Far harsher, and -dominated by recall, since any single missed actor zeroes that second. +**exact_match_rate**: the fraction of seconds where the pipeline's named set +exactly equals X-Ray's — no partial credit, dominated by recall. + +## The benchmark set + +Unlike the July report — which trained on a 3-film subset and validated on +held-out films to keep evaluations fast — this run scores **all 9 films on +every evaluation**. The registry one-clock fix and uncapped dumps made +full-set replay affordable, so the reported optimum is tuned against the +complete set rather than a training subset. ## Reproduce @@ -129,6 +122,5 @@ python3 scripts/optimizer/second_score.py \ --gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5 ``` -See also [the full experiment log](model-bakeoff.md) for how `pred.json` is -produced, and the [LVFace deep dive](lvface-deep-dive.md) for what these -mechanisms look like frame by frame. +See the [full experiment log](model-bakeoff.md) for how `pred.json` is +produced and where the shipped `src/config.hpp` defaults come from. diff --git a/docs/model-bakeoff.md b/docs/model-bakeoff-2026-07.md similarity index 95% rename from docs/model-bakeoff.md rename to docs/model-bakeoff-2026-07.md index 5b38d27..e63da05 100644 --- a/docs/model-bakeoff.md +++ b/docs/model-bakeoff-2026-07.md @@ -1,3 +1,5 @@ +> **Archived (July 2026).** This report covers the pre-opencv5 framework and the 4-model ArcFace/LVFace bake-off. It is superseded by the current [experiment log](model-bakeoff.md) for the opencv5 build. Kept for provenance; the numbers here are historical. + # Full experiment log This page reports how the pipeline performs across three questions: which @@ -7,7 +9,7 @@ a per-film gallery annex helps. It also documents the replay architecture that made testing all three questions in one pass practical, and every caveat needed to trust the numbers. -Read [How we score against X-Ray](methodology.md) first for what F1, +Read [How we score against X-Ray](methodology-2026-07.md) first for what F1, precision, recall, and misID mean in this report. All numbers below use the per-second metric ([`scripts/optimizer/second_score.py`](https://REPOLINK/scripts/optimizer/second_score.py)). @@ -17,7 +19,7 @@ gallery was built with roughly 30% fewer reference images per actor than the other three models on the identical source photos (10808 vs 15055 total embeddings across the same 2418 actors), which confounds any direct comparison of its scores against the others. It remains in the -[calibration curve comparison](best-model.md#first-signal-calibration-curves), +[calibration curve comparison](best-model-2026-07.md#first-signal-calibration-curves), which does not depend on the training benchmark. ## Why replay makes this affordable @@ -104,7 +106,7 @@ missing references, not purely by model quality. Downton Abbey has 61% coverage, the second-best in the benchmark, yet the worst held-out recall of any film (39.4%, LVFace). Its recall problem is not primarily a coverage problem; it is the extinction-bridging failure documented in the -[LVFace deep dive](lvface-deep-dive.md#mechanism-1-extinction-bridging). +[LVFace deep dive](lvface-deep-dive-2026-07.md#mechanism-1-extinction-bridging). Reproduce with `scripts/docs/gallery_coverage_per_film.py`. ## Training results, 3 models × 2 gallery modes × 2 expansion settings @@ -141,7 +143,7 @@ restricted gallery, and LVFace leads within both gallery modes. `full_exp` (the shipped combination) is the best-scoring option that uses only features the running application currently supports; restriction is not wired into the application yet (see -[Whole vs. cast-restricted gallery](gallery-scope.md)). +[Whole vs. cast-restricted gallery](gallery-scope-2026-07.md)). ### A scoring bug worth recording: dropped-film evaluations @@ -211,7 +213,7 @@ never flips. Total misIDs across the 5 films: LVFace 1032, mbf 2197, r18 1224. LVFace has less than half mbf's misID count while also scoring higher on every film. This directly confirms the model choice out of sample; it is not inferred from the training numbers alone. See the -[LVFace deep dive](lvface-deep-dive.md) for frame-level detail on where and +[LVFace deep dive](lvface-deep-dive-2026-07.md) for frame-level detail on where and why LVFace still fails on the two worst films. Reproduce with `scripts/docs/run_holdout_all_models.py`. @@ -234,7 +236,7 @@ candidates in the matcher's search space means fewer opportunities for a lookalike false match, and the recall gain shows this does not cost real detections. Restriction is currently an offline optimizer technique, not a runtime feature of the application; see -[Whole vs. cast-restricted gallery](gallery-scope.md) for what building it +[Whole vs. cast-restricted gallery](gallery-scope-2026-07.md) for what building it into the application would require. **Pose expansion** (promoting a confidently identified track's novel-pose @@ -257,7 +259,7 @@ against the full 2418-actor gallery, where a confidently learned pose is more likely to match the wrong person. On the full gallery it is a recall-vs-misID trade, not a free gain. This training-set effect did not reproduce on held-out data; see -[Does pose expansion help?](pose-expansion.md) for the full held-out test +[Does pose expansion help?](pose-expansion-2026-07.md) for the full held-out test and the two methodology bugs caught while checking it. ## Calibration curves diff --git a/docs/pose-expansion.md b/docs/pose-expansion-2026-07.md similarity index 91% rename from docs/pose-expansion.md rename to docs/pose-expansion-2026-07.md index 81fd109..3be6c10 100644 --- a/docs/pose-expansion.md +++ b/docs/pose-expansion-2026-07.md @@ -1,3 +1,5 @@ +> **Archived (July 2026).** This report covers the pre-opencv5 framework and the 4-model ArcFace/LVFace bake-off. It is superseded by the current [experiment log](model-bakeoff.md) for the opencv5 build. Kept for provenance; the numbers here are historical. + # Pose expansion: does promoting new poses mid-film help? `expand_gallery` @@ -12,7 +14,7 @@ in the same film, without touching the baked gallery. Averaged across the 3 compared models (r50 excluded), on the 4 films used for optimization. These are the corrected, full-coverage figures, see the -[dropped-film note](model-bakeoff.md#a-scoring-bug-worth-recording-dropped-film-evaluations) +[dropped-film note](model-bakeoff-2026-07.md#a-scoring-bug-worth-recording-dropped-film-evaluations) in the experiment log for why an earlier version of this table overstated the full-mode misID jump (209 → 864) that was itself partly a truncation artifact: @@ -26,7 +28,7 @@ full-mode misID jump (209 → 864) that was itself partly a truncation artifact: In restricted mode, expansion looks like a clean win: +1.6pp F1, +3.3pp recall, lower misID. In full mode it looks like a recall-for-misID trade: +2.1pp F1, +3.9pp recall, but misID rises from 407 to 714. See -[the full experiment log](model-bakeoff.md) for the per-model breakdown. +[the full experiment log](model-bakeoff-2026-07.md) for the per-model breakdown. This asymmetry motivated the question below: does turning expansion on change what gets recognized frame by frame, or is the aggregate F1 shift coming from something else. @@ -105,6 +107,6 @@ contribution, such as tagging which reference embedding won each match; neither was in scope for this pass. Do not treat the training-set exp/noexp numbers in -[the full experiment log](model-bakeoff.md) as proof that expansion changes +[the full experiment log](model-bakeoff-2026-07.md) as proof that expansion changes real-world behavior in either direction. On the evidence gathered so far, it does not move the needle enough to see. diff --git a/mkdocs.yml b/mkdocs.yml index 0041ca7..a86e74a 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -36,13 +36,15 @@ nav: - Home: index.md - How We Score Against X-Ray: methodology.md - Benchmark — SuperHero: benchmark.md - - Findings: - - Best Model: best-model.md - - Gallery Scope (Full vs. Limited): gallery-scope.md - - Pose Expansion: pose-expansion.md - - LVFace Deep Dive: lvface-deep-dive.md - Full Experiment Log: model-bakeoff.md - Service Conversion (proposal): service-conversion.md + - Archive (July 2026): + - How We Scored (July): methodology-2026-07.md + - Best Model: best-model-2026-07.md + - Gallery Scope (Full vs. Limited): gallery-scope-2026-07.md + - Pose Expansion: pose-expansion-2026-07.md + - LVFace Deep Dive: lvface-deep-dive-2026-07.md + - Full Experiment Log (July): model-bakeoff-2026-07.md markdown_extensions: - admonition