dtourolle/scene-actor-extraction
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: archive the July 2026 report; new methodology for the opencv5 run

Browse Source 
The July report (4-model ArcFace/LVFace bake-off, pre-opencv5 framework,
3-film training + held-out validation) is superseded by the opencv5 build:
single-model LVFace-B, a 6-knob DE sweep over all 9 films, flood-fill
presence, and the registry/decode fixes. Rather than overwrite it, archive
it date-suffixed and start the current report fresh.

- Rename the six July result pages to *-2026-07.md, rewrite their
  intra-archive cross-links, and add an "Archived (July 2026)" banner to each.
- mkdocs nav: current report at top, the July set under an Archive section.
- New docs/methodology.md for the opencv5 run: corrects the withdrawn
  anneal_sec/extinction_sec presence bridging (windows are now
  [first_seen, last_seen], AR-012/013), documents the two presence modes
  (track_extent / flood), and records that every eval scores all 9 films.

The current experiment log (model-bakeoff.md) and Home rewrite land once
the DE sweep converges and the final optimum is known.
This commit is contained in:
dtourolle
2026-08-09 10:48:45 +02:00
parent e1423062e2
commit ea922356f1
8 changed files with 243 additions and 103 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/best-model.md → docs/best-model-2026-07.md
+5 -3
View File
@@ -1,10 +1,12 @@
> **Archived (July 2026).** This report covers the pre-opencv5 framework and the 4-model ArcFace/LVFace bake-off. It is superseded by the current [experiment log](model-bakeoff.md) for the opencv5 build. Kept for provenance; the numbers here are historical.
# Which embedding model is best?
Three ArcFace variants (w600k-R50, R18, w600k-MBF) and LVFace-B (Glint360K,
455MB) were compared. r50 is excluded from the training/held-out comparison
below; its gallery has roughly 30% fewer reference images per actor than the
other three on the identical source photos, which confounds a direct score
comparison (see [the full experiment log](model-bakeoff.md) for detail). It
comparison (see [the full experiment log](model-bakeoff-2026-07.md) for detail). It
remains in the calibration comparison, which does not depend on the gallery
image count.
@@ -63,7 +65,7 @@ than general performance. On training data, the ordering is not as clean:
mbf beats LVFace on Lord of War (77.2% vs 75.6%), the only film in either
table where LVFace does not score highest. LVFace's training-set macro
average (75.3%, see [the full experiment log](model-bakeoff.md)) is not a
average (75.3%, see [the full experiment log](model-bakeoff-2026-07.md)) is not a
uniform win across every film it contributes to; the held-out result, where
LVFace wins all 5 films outright, is the stronger claim.
@@ -79,7 +81,7 @@ not.
![All 12 combos ranked by training-set F1](assets/images/rep4_matrix_f1.png)
Best full-gallery combo per model (all three are `full_exp`), from the
training matrix in [the full experiment log](model-bakeoff.md):
training matrix in [the full experiment log](model-bakeoff-2026-07.md):
| model | F1 | P | R | misID |
|---|---|---|---|---|
docs/gallery-scope.md → docs/gallery-scope-2026-07.md
+5 -3
View File
@@ -1,3 +1,5 @@
> **Archived (July 2026).** This report covers the pre-opencv5 framework and the 4-model ArcFace/LVFace bake-off. It is superseded by the current [experiment log](model-bakeoff.md) for the opencv5 build. Kept for provenance; the numbers here are historical.
# Whole gallery vs. cast-restricted gallery
Two ways to run the matcher. Full mode scores every detected face against
@@ -8,7 +10,7 @@ top-billed actors) before the matcher runs.
## Result
Averaged across the 3 compared models (r50 excluded, see
[the full experiment log](model-bakeoff.md)) and both expansion settings, on
[the full experiment log](model-bakeoff-2026-07.md)) and both expansion settings, on
the 4 training films:
| scope | F1 | P | R | total misID |
@@ -27,7 +29,7 @@ restricted gallery:
![All combos ranked by training-set F1, filled dots are restricted](assets/images/rep4_matrix_f1.png)
See [the full experiment log](model-bakeoff.md) for the complete table. One
See [the full experiment log](model-bakeoff-2026-07.md) for the complete table. One
combo reaches zero true out-of-cast misidentifications,
`arcface_w600k_mbf_restricted_exp` (F1 76.2%), and it is a restricted one,
consistent with restriction, not expansion, being what suppresses cross-film
@@ -57,7 +59,7 @@ Building this as a real feature requires:
option.
- A decision on the fallback case: what happens to a real, uncredited
cameo (see the Germar Terrell Gardner and Talia Balsam cases in the
[LVFace deep dive](lvface-deep-dive.md#where-lvface-beat-x-ray)) if the
[LVFace deep dive](lvface-deep-dive-2026-07.md#where-lvface-beat-x-ray)) if the
restricted gallery never includes them at all.
- Regenerating the restricted-gallery cache whenever a title's Jellyfin
cast list changes.
docs/lvface-deep-dive.md → docs/lvface-deep-dive-2026-07.md
+6 -4
View File
@@ -1,12 +1,14 @@
> **Archived (July 2026).** This report covers the pre-opencv5 framework and the 4-model ArcFace/LVFace bake-off. It is superseded by the current [experiment log](model-bakeoff.md) for the opencv5 build. Kept for provenance; the numbers here are historical.
# Deep dive: LVFace-B Glint360K
LVFace won the model comparison (see [Which model is best?](best-model.md))
LVFace won the model comparison (see [Which model is best?](best-model-2026-07.md))
and is the shipped default embedder. This page reports how it performs in
detail: a baseline of correct output, the two mechanisms behind its errors,
and every distinct case where it names someone who is not in the film's
credited cast.
Read [How we score against X-Ray](methodology.md) first. X-Ray's ground truth
Read [How we score against X-Ray](methodology-2026-07.md) first. X-Ray's ground truth
is scene-level, not per-frame. A name marked correct in the Offscreen column
below is the pipeline correctly reporting scene membership, not a workaround.
@@ -61,7 +63,7 @@ on the 5 films the optimizer never saw:
| macro average | 67.4% | 85.8% | 57.0% | | | | |
The `P` column is misID-weighted (each out-of-film name counts 10x in the
denominator; see [methodology](methodology.md#precision-recall-and-the-misid-weighting)).
denominator; see [methodology](methodology-2026-07.md#precision-recall-and-the-misid-weighting)).
That weighting is why Many Saints reads 54.7% here despite naming mostly real,
present faces: its raw (unweighted) precision is **78.4%**, and the gap is
entirely its 974 misIDs paying the 10x penalty. The three zero-misID films
@@ -70,7 +72,7 @@ Lovelace, with 58 misIDs, sits 3pp below its raw 93.3%.
Held-out F1 is 67.4%, against 75.3% on training, an 8pp drop. The spread
between the best and worst held-out film is 37pp. This is not unique to
LVFace: [the full experiment log](model-bakeoff.md#held-out-validation-all-3-models)
LVFace: [the full experiment log](model-bakeoff-2026-07.md#held-out-validation-all-3-models)
shows mbf and r18 with the same shape of spread on the same films, at a
uniformly lower level. Two mechanisms explain the spread. Both are shown
below with frame-level evidence.
docs/methodology-2026-07.md
+136
View File
@@ -0,0 +1,136 @@
> **Archived (July 2026).** This report covers the pre-opencv5 framework and the 4-model ArcFace/LVFace bake-off. It is superseded by the current [experiment log](model-bakeoff.md) for the opencv5 build. Kept for provenance; the numbers here are historical.
# How we score against X-Ray
Every number in this report, every F1 and misID count, comes from one
comparison. The comparison has a mismatch at its core that shapes nearly
every finding in this report: the ground truth is scene-level, the
pipeline's output is per-second, and the two do not mean the same thing.
This page documents that comparison once, so the findings pages can rely on
it without re-explaining it.
## What Amazon X-Ray records
X-Ray ships three tables per film: `scenes.csv` (a list of `[start, end]`
timespans), `people_in_scenes.csv` (which actors are credited in each
scene), and `people.csv` (actor identities). There is no per-frame or
per-second annotation anywhere in X-Ray. A scene might run 45 seconds, and
X-Ray records one cast list for the entire span, not "on screen from
second 12 to second 30."
To compare this against per-second predictions, `second_score.py` expands
every scene into per-second ground truth by copying the whole scene's cast
list onto every second inside it:
```python
for sn, (t0, t1) in spans.items():
cast = scene_cast.get(sn, [])
for t in range(int(t0), int(t1)):
timeline[t] = cast
```
That is the entire mechanism. If X-Ray credits five actors to a 30-second
scene, all five count as ground truth present for all 30 seconds, including
seconds where only one of them is on screen. This is not a simplification
introduced by the pipeline; it is the only reading of X-Ray's data that is
possible, because X-Ray itself does not record anything finer-grained.
## Why an offscreen name can be scored correct
A name listed under Offscreen with a correct (green) label is not the
pipeline guessing or padding its score. It is the pipeline correctly
answering the question X-Ray actually asks: is this actor part of this
scene. It answers that question using a presence window (`[start, end]`,
held open across cuts by `anneal_sec` and `extinction_sec`), which matches
X-Ray's scene-level semantics more closely than a raw per-frame detection
would.
A system that only reported "this actor is visible in this exact frame"
would score worse against X-Ray's scene-level ground truth, producing a
false negative every time the camera cuts away from a character who is
still present in the scene. Not because it is wrong about the world, but
because it would be answering a stricter, different question than the one
X-Ray's data supports. The presence-window design exists specifically to
answer X-Ray's actual question.
## What this resolves and what it does not
This resolves the semantic mismatch between a scene and an instant. It does
not resolve two other limitations, both discussed in the
[LVFace deep dive](lvface-deep-dive-2026-07.md).
**The face-vs-presence ceiling.** X-Ray credits scene membership regardless
of whether a face is ever visible: background crew, characters shot from
behind, voice-only presence. No amount of bridging recovers a face that
never appears on screen. This is a hard ceiling on recall, not a defect.
**Extinction bridging can overshoot.** The same presence-window mechanism
that correctly answers "still in this scene" during a normal cut can also
bridge across a scene boundary it has no way to detect. A hard cut into a
different scene with no faces, such as closing credits, carries the
previous scene's identities forward until the window expires. This is the
mechanism behind Downton Abbey's recall collapse, documented in the deep
dive.
## Precision, recall, and the misID weighting
Per sampled second `t`:
**TPI** (true positive instances): actors both X-Ray and the pipeline agree
are present.
**FPI** (false positive instances): actors the pipeline reports that are
not in X-Ray's cast for this second. Split into two categories:
- **FPI_incast**: the actor is in the film's cast, just not credited to
this particular scene. A timing or boundary slip.
- **FPI_misid**: the actor is not in the film's cast at all. A genuine
wrong-identity error, weighted 10x in the precision objective, because
naming someone who is not even in the film is a categorically worse
error than a few seconds of scene-boundary slop.
!!! note "Every headline `P` and `F1` is misID-weighted"
The precision reported throughout this report, and therefore the F1
derived from it, puts each `FPI_misid` into the denominator **10 times**
(`precision = TPI / (TPI + FPI_incast + 10·FPI_misid)`,
[`second_score.py`](https://REPOLINK/scripts/optimizer/second_score.py)).
This is deliberate: the whole point is to punish naming an out-of-film
actor far harder than a scene-boundary slip. But it means the `P` column
is not raw precision, and a misID-heavy film's `P` is depressed
super-linearly. `second_score.py` also emits an unweighted `precision_raw`
(always ≥ the weighted `P`); where the gap matters, The Many Saints of
Newark, weighted `P` 54.7% vs. raw 78.4%, the [LVFace deep dive](lvface-deep-dive-2026-07.md)
reports both. When comparing `P` across films, remember you are comparing a
quantity that penalizes misIDs, not just a hit rate.
**FN** (false negatives): actors X-Ray lists that the pipeline never
reports, counted only for actors who have a gallery reference embedding.
Across the 9-film benchmark, coverage of X-Ray's credited cast ranges from
20% to 79% by film (see
[the full experiment log](model-bakeoff-2026-07.md#gallery-coverage-per-film)); an
actor with no reference photo can never be recognized regardless of model
quality, and counting them as a miss would penalize gallery coverage, not
recognition accuracy.
Two further numbers are reported alongside F1:
**agreement_rate**: mean per-second Jaccard overlap
(`|Pred ∩ GT| / |Pred ∪ GT|`), partial credit. Naming 2 of 3 present actors
scores 2/3, not 0.
**exact_match_rate**: the fraction of sampled seconds where the pipeline's
named set exactly equals X-Ray's, no partial credit. Far harsher, and
dominated by recall, since any single missed actor zeroes that second.
## Reproduce
```bash
python3 scripts/optimizer/second_score.py \
--pred pred.json --xray experiments/xray/.../<xray_dir> \
--gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5
```
See also [the full experiment log](model-bakeoff-2026-07.md) for how `pred.json` is
produced, and the [LVFace deep dive](lvface-deep-dive-2026-07.md) for what these
mechanisms look like frame by frame.
docs/methodology.md
+70 -78
View File
@@ -1,11 +1,9 @@
# How we score against X-Ray
Every number in this report, every F1 and misID count, comes from one
comparison. The comparison has a mismatch at its core that shapes nearly
every finding in this report: the ground truth is scene-level, the
pipeline's output is per-second, and the two do not mean the same thing.
This page documents that comparison once, so the findings pages can rely on
it without re-explaining it.
Every number in this report comes from one comparison, and that comparison
has a mismatch at its core: the ground truth is scene-level, the pipeline's
output is per-second, and the two do not mean the same thing. This page
documents the comparison once so the findings can rely on it.
## What Amazon X-Ray records
@@ -13,12 +11,12 @@ X-Ray ships three tables per film: `scenes.csv` (a list of `[start, end]`
timespans), `people_in_scenes.csv` (which actors are credited in each
scene), and `people.csv` (actor identities). There is no per-frame or
per-second annotation anywhere in X-Ray. A scene might run 45 seconds, and
X-Ray records one cast list for the entire span, not "on screen from
second 12 to second 30."
X-Ray records one cast list for the entire span, not "on screen from second
12 to second 30."
To compare this against per-second predictions, `second_score.py` expands
every scene into per-second ground truth by copying the whole scene's cast
list onto every second inside it:
To compare against per-second predictions, `second_score.py` expands every
scene into per-second ground truth by copying the whole scene's cast list
onto every second inside it:
```python
for sn, (t0, t1) in spans.items():
@@ -27,48 +25,46 @@ for sn, (t0, t1) in spans.items():
timeline[t] = cast
```
That is the entire mechanism. If X-Ray credits five actors to a 30-second
scene, all five count as ground truth present for all 30 seconds, including
seconds where only one of them is on screen. This is not a simplification
introduced by the pipeline; it is the only reading of X-Ray's data that is
possible, because X-Ray itself does not record anything finer-grained.
If X-Ray credits five actors to a 30-second scene, all five count as ground
truth present for all 30 seconds, including seconds where only one is on
screen. This is not a simplification the pipeline introduces; it is the only
reading X-Ray's data supports, because X-Ray records nothing finer.
## Why an offscreen name can be scored correct
## How the pipeline reports presence
A name listed under Offscreen with a correct (green) label is not the
pipeline guessing or padding its score. It is the pipeline correctly
answering the question X-Ray actually asks: is this actor part of this
scene. It answers that question using a presence window (`[start, end]`,
held open across cuts by `anneal_sec` and `extinction_sec`), which matches
X-Ray's scene-level semantics more closely than a raw per-frame detection
would.
A presence claim is one actor owning one time window. How that window is
derived is a tunable choice — a knob the optimizer weighs — with two modes:
A system that only reported "this actor is visible in this exact frame"
would score worse against X-Ray's scene-level ground truth, producing a
false negative every time the camera cuts away from a character who is
still present in the scene. Not because it is wrong about the world, but
because it would be answering a stricter, different question than the one
X-Ray's data supports. The presence-window design exists specifically to
answer X-Ray's actual question.
- **`track_extent` (default).** A claim is exactly `[first_seen, last_seen]`
of a track the actor owned (AR-012), ending at the last sighting and never
after (AR-013). There is no keep-alive: the withdrawn `anneal_sec` and the
scene-tracker `extinction_sec` — which the July report's windows were held
open by — are **gone**. A track that survives its own gaps needs no bridge;
a gap after the final sighting is never claimed.
- **`flood`.** Each claim is snapped to the shot it sits in, so an actor seen
once anywhere in a shot is reported for the whole shot
`[prev_boundary, next_boundary]`. Boundaries come from TransNetV2 shot
detection when available, otherwise from the always-on histogram cut
detector (`is_cut`). This trades precision for recall against X-Ray's
scene-level granularity, and the optimizer decides per run whether it pays.
## What this resolves and what it does not
Do not confuse the surviving `track_extinction_sec` with the withdrawn
scene `extinction_sec`: the former bounds how long a lost track stays
available for **re-association** (a tracking question), and never extends a
presence claim.
This resolves the semantic mismatch between a scene and an instant. It does
not resolve two other limitations, both discussed in the
[LVFace deep dive](lvface-deep-dive.md).
## The two limits this does not resolve
**The face-vs-presence ceiling.** X-Ray credits scene membership regardless
of whether a face is ever visible: background crew, characters shot from
behind, voice-only presence. No amount of bridging recovers a face that
never appears on screen. This is a hard ceiling on recall, not a defect.
behind, voice-only presence. No face pipeline can recover a face that never
appears, so recall against X-Ray is a structural ceiling, not a defect.
**Extinction bridging can overshoot.** The same presence-window mechanism
that correctly answers "still in this scene" during a normal cut can also
bridge across a scene boundary it has no way to detect. A hard cut into a
different scene with no faces, such as closing credits, carries the
previous scene's identities forward until the window expires. This is the
mechanism behind Downton Abbey's recall collapse, documented in the deep
dive.
**Flood-fill can overshoot.** Snapping to a shot correctly answers "still in
this scene" through an intra-scene cut, but a shot boundary is not a scene
boundary: on a film with sparse cuts, flood-fill can carry an actor across a
long "shot" they only briefly appeared in. This is why flood-fill is a knob,
not a default — its value depends on the film's cut density.
## Precision, recall, and the misID weighting
@@ -77,49 +73,46 @@ Per sampled second `t`:
**TPI** (true positive instances): actors both X-Ray and the pipeline agree
are present.
**FPI** (false positive instances): actors the pipeline reports that are
not in X-Ray's cast for this second. Split into two categories:
**FPI** (false positive instances): actors the pipeline reports that are not
in X-Ray's cast for this second, split into:
- **FPI_incast**: the actor is in the film's cast, just not credited to
this particular scene. A timing or boundary slip.
- **FPI_incast**: the actor is in the film's cast, just not credited to this
scene. A timing or boundary slip.
- **FPI_misid**: the actor is not in the film's cast at all. A genuine
wrong-identity error, weighted 10x in the precision objective, because
naming someone who is not even in the film is a categorically worse
error than a few seconds of scene-boundary slop.
wrong-identity error, weighted **10×** in the precision objective, because
naming someone not even in the film is categorically worse than a few
seconds of scene-boundary slop.
!!! note "Every headline `P` and `F1` is misID-weighted"
The precision reported throughout this report, and therefore the F1
derived from it, puts each `FPI_misid` into the denominator **10 times**
Precision puts each `FPI_misid` into the denominator 10 times
(`precision = TPI / (TPI + FPI_incast + 10·FPI_misid)`,
[`second_score.py`](https://REPOLINK/scripts/optimizer/second_score.py)).
This is deliberate: the whole point is to punish naming an out-of-film
actor far harder than a scene-boundary slip. But it means the `P` column
is not raw precision, and a misID-heavy film's `P` is depressed
super-linearly. `second_score.py` also emits an unweighted `precision_raw`
(always ≥ the weighted `P`); where the gap matters, The Many Saints of
Newark, weighted `P` 54.7% vs. raw 78.4%, the [LVFace deep dive](lvface-deep-dive.md)
reports both. When comparing `P` across films, remember you are comparing a
quantity that penalizes misIDs, not just a hit rate.
This deliberately punishes naming an out-of-film actor far harder than a
boundary slip, so the `P` column is not raw precision and a misID-heavy
film's `P` is depressed super-linearly.
**FN** (false negatives): actors X-Ray lists that the pipeline never
reports, counted only for actors who have a gallery reference embedding.
Across the 9-film benchmark, coverage of X-Ray's credited cast ranges from
20% to 79% by film (see
[the full experiment log](model-bakeoff.md#gallery-coverage-per-film)); an
actor with no reference photo can never be recognized regardless of model
quality, and counting them as a miss would penalize gallery coverage, not
recognition accuracy.
**FN** (false negatives): actors X-Ray lists that the pipeline never reports,
counted **only** for actors who have a gallery reference embedding. An actor
with no reference photo can never be recognized, and counting them as a miss
would measure gallery coverage, not recognition accuracy.
Two further numbers are reported alongside F1:
Two further numbers accompany F1:
**agreement_rate**: mean per-second Jaccard overlap
(`|Pred ∩ GT| / |Pred ∪ GT|`), partial credit. Naming 2 of 3 present actors
scores 2/3, not 0.
(`|Pred ∩ GT| / |Pred ∪ GT|`) — partial credit, so naming 2 of 3 present
actors scores 2/3, not 0.
**exact_match_rate**: the fraction of sampled seconds where the pipeline's
named set exactly equals X-Ray's, no partial credit. Far harsher, and
dominated by recall, since any single missed actor zeroes that second.
**exact_match_rate**: the fraction of seconds where the pipeline's named set
exactly equals X-Ray's — no partial credit, dominated by recall.
## The benchmark set
Unlike the July report — which trained on a 3-film subset and validated on
held-out films to keep evaluations fast — this run scores **all 9 films on
every evaluation**. The registry one-clock fix and uncapped dumps made
full-set replay affordable, so the reported optimum is tuned against the
complete set rather than a training subset.
## Reproduce
@@ -129,6 +122,5 @@ python3 scripts/optimizer/second_score.py \
--gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5
```
See also [the full experiment log](model-bakeoff.md) for how `pred.json` is
produced, and the [LVFace deep dive](lvface-deep-dive.md) for what these
mechanisms look like frame by frame.
See the [full experiment log](model-bakeoff.md) for how `pred.json` is
produced and where the shipped `src/config.hpp` defaults come from.
docs/model-bakeoff.md → docs/model-bakeoff-2026-07.md
+9 -7
View File
@@ -1,3 +1,5 @@
> **Archived (July 2026).** This report covers the pre-opencv5 framework and the 4-model ArcFace/LVFace bake-off. It is superseded by the current [experiment log](model-bakeoff.md) for the opencv5 build. Kept for provenance; the numbers here are historical.
# Full experiment log
This page reports how the pipeline performs across three questions: which
@@ -7,7 +9,7 @@ a per-film gallery annex helps. It also documents the replay architecture
that made testing all three questions in one pass practical, and every
caveat needed to trust the numbers.
Read [How we score against X-Ray](methodology.md) first for what F1,
Read [How we score against X-Ray](methodology-2026-07.md) first for what F1,
precision, recall, and misID mean in this report. All numbers below use the
per-second metric
([`scripts/optimizer/second_score.py`](https://REPOLINK/scripts/optimizer/second_score.py)).
@@ -17,7 +19,7 @@ gallery was built with roughly 30% fewer reference images per actor than the
other three models on the identical source photos (10808 vs 15055 total
embeddings across the same 2418 actors), which confounds any direct
comparison of its scores against the others. It remains in the
[calibration curve comparison](best-model.md#first-signal-calibration-curves),
[calibration curve comparison](best-model-2026-07.md#first-signal-calibration-curves),
which does not depend on the training benchmark.
## Why replay makes this affordable
@@ -104,7 +106,7 @@ missing references, not purely by model quality. Downton Abbey has 61%
coverage, the second-best in the benchmark, yet the worst held-out recall
of any film (39.4%, LVFace). Its recall problem is not primarily a coverage
problem; it is the extinction-bridging failure documented in the
[LVFace deep dive](lvface-deep-dive.md#mechanism-1-extinction-bridging).
[LVFace deep dive](lvface-deep-dive-2026-07.md#mechanism-1-extinction-bridging).
Reproduce with `scripts/docs/gallery_coverage_per_film.py`.
## Training results, 3 models × 2 gallery modes × 2 expansion settings
@@ -141,7 +143,7 @@ restricted gallery, and LVFace leads within both gallery modes. `full_exp`
(the shipped combination) is the best-scoring option that uses only
features the running application currently supports; restriction is not
wired into the application yet (see
[Whole vs. cast-restricted gallery](gallery-scope.md)).
[Whole vs. cast-restricted gallery](gallery-scope-2026-07.md)).
### A scoring bug worth recording: dropped-film evaluations
@@ -211,7 +213,7 @@ never flips. Total misIDs across the 5 films: LVFace 1032, mbf 2197, r18
1224. LVFace has less than half mbf's misID count while also scoring
higher on every film. This directly confirms the model choice out of
sample; it is not inferred from the training numbers alone. See the
[LVFace deep dive](lvface-deep-dive.md) for frame-level detail on where and
[LVFace deep dive](lvface-deep-dive-2026-07.md) for frame-level detail on where and
why LVFace still fails on the two worst films. Reproduce with
`scripts/docs/run_holdout_all_models.py`.
@@ -234,7 +236,7 @@ candidates in the matcher's search space means fewer opportunities for a
lookalike false match, and the recall gain shows this does not cost real
detections. Restriction is currently an offline optimizer technique, not a
runtime feature of the application; see
[Whole vs. cast-restricted gallery](gallery-scope.md) for what building it
[Whole vs. cast-restricted gallery](gallery-scope-2026-07.md) for what building it
into the application would require.
**Pose expansion** (promoting a confidently identified track's novel-pose
@@ -257,7 +259,7 @@ against the full 2418-actor gallery, where a confidently learned pose is more
likely to match the wrong person. On the full gallery it is a recall-vs-misID
trade, not a free gain. This training-set effect
did not reproduce on held-out data; see
[Does pose expansion help?](pose-expansion.md) for the full held-out test
[Does pose expansion help?](pose-expansion-2026-07.md) for the full held-out test
and the two methodology bugs caught while checking it.
## Calibration curves
docs/pose-expansion.md → docs/pose-expansion-2026-07.md
+5 -3
View File
@@ -1,3 +1,5 @@
> **Archived (July 2026).** This report covers the pre-opencv5 framework and the 4-model ArcFace/LVFace bake-off. It is superseded by the current [experiment log](model-bakeoff.md) for the opencv5 build. Kept for provenance; the numbers here are historical.
# Pose expansion: does promoting new poses mid-film help?
`expand_gallery`
@@ -12,7 +14,7 @@ in the same film, without touching the baked gallery.
Averaged across the 3 compared models (r50 excluded), on the 4 films used
for optimization. These are the corrected, full-coverage figures, see the
[dropped-film note](model-bakeoff.md#a-scoring-bug-worth-recording-dropped-film-evaluations)
[dropped-film note](model-bakeoff-2026-07.md#a-scoring-bug-worth-recording-dropped-film-evaluations)
in the experiment log for why an earlier version of this table overstated the
full-mode misID jump (209 → 864) that was itself partly a truncation artifact:
@@ -26,7 +28,7 @@ full-mode misID jump (209 → 864) that was itself partly a truncation artifact:
In restricted mode, expansion looks like a clean win: +1.6pp F1, +3.3pp
recall, lower misID. In full mode it looks like a recall-for-misID trade:
+2.1pp F1, +3.9pp recall, but misID rises from 407 to 714. See
[the full experiment log](model-bakeoff.md) for the per-model breakdown.
[the full experiment log](model-bakeoff-2026-07.md) for the per-model breakdown.
This asymmetry motivated the question below: does turning expansion on
change what gets recognized frame by frame, or is the aggregate F1 shift
coming from something else.
@@ -105,6 +107,6 @@ contribution, such as tagging which reference embedding won each match;
neither was in scope for this pass.
Do not treat the training-set exp/noexp numbers in
[the full experiment log](model-bakeoff.md) as proof that expansion changes
[the full experiment log](model-bakeoff-2026-07.md) as proof that expansion changes
real-world behavior in either direction. On the evidence gathered so far,
it does not move the needle enough to see.