docs: archive the July 2026 report; new methodology for the opencv5 run
The July report (4-model ArcFace/LVFace bake-off, pre-opencv5 framework, 3-film training + held-out validation) is superseded by the opencv5 build: single-model LVFace-B, a 6-knob DE sweep over all 9 films, flood-fill presence, and the registry/decode fixes. Rather than overwrite it, archive it date-suffixed and start the current report fresh. - Rename the six July result pages to *-2026-07.md, rewrite their intra-archive cross-links, and add an "Archived (July 2026)" banner to each. - mkdocs nav: current report at top, the July set under an Archive section. - New docs/methodology.md for the opencv5 run: corrects the withdrawn anneal_sec/extinction_sec presence bridging (windows are now [first_seen, last_seen], AR-012/013), documents the two presence modes (track_extent / flood), and records that every eval scores all 9 films. The current experiment log (model-bakeoff.md) and Home rewrite land once the DE sweep converges and the final optimum is known.
This commit is contained in:
@@ -1,10 +1,12 @@
|
|||||||
|
> **Archived (July 2026).** This report covers the pre-opencv5 framework and the 4-model ArcFace/LVFace bake-off. It is superseded by the current [experiment log](model-bakeoff.md) for the opencv5 build. Kept for provenance; the numbers here are historical.
|
||||||
|
|
||||||
# Which embedding model is best?
|
# Which embedding model is best?
|
||||||
|
|
||||||
Three ArcFace variants (w600k-R50, R18, w600k-MBF) and LVFace-B (Glint360K,
|
Three ArcFace variants (w600k-R50, R18, w600k-MBF) and LVFace-B (Glint360K,
|
||||||
455MB) were compared. r50 is excluded from the training/held-out comparison
|
455MB) were compared. r50 is excluded from the training/held-out comparison
|
||||||
below; its gallery has roughly 30% fewer reference images per actor than the
|
below; its gallery has roughly 30% fewer reference images per actor than the
|
||||||
other three on the identical source photos, which confounds a direct score
|
other three on the identical source photos, which confounds a direct score
|
||||||
comparison (see [the full experiment log](model-bakeoff.md) for detail). It
|
comparison (see [the full experiment log](model-bakeoff-2026-07.md) for detail). It
|
||||||
remains in the calibration comparison, which does not depend on the gallery
|
remains in the calibration comparison, which does not depend on the gallery
|
||||||
image count.
|
image count.
|
||||||
|
|
||||||
@@ -63,7 +65,7 @@ than general performance. On training data, the ordering is not as clean:
|
|||||||
|
|
||||||
mbf beats LVFace on Lord of War (77.2% vs 75.6%), the only film in either
|
mbf beats LVFace on Lord of War (77.2% vs 75.6%), the only film in either
|
||||||
table where LVFace does not score highest. LVFace's training-set macro
|
table where LVFace does not score highest. LVFace's training-set macro
|
||||||
average (75.3%, see [the full experiment log](model-bakeoff.md)) is not a
|
average (75.3%, see [the full experiment log](model-bakeoff-2026-07.md)) is not a
|
||||||
uniform win across every film it contributes to; the held-out result, where
|
uniform win across every film it contributes to; the held-out result, where
|
||||||
LVFace wins all 5 films outright, is the stronger claim.
|
LVFace wins all 5 films outright, is the stronger claim.
|
||||||
|
|
||||||
@@ -79,7 +81,7 @@ not.
|
|||||||

|

|
||||||
|
|
||||||
Best full-gallery combo per model (all three are `full_exp`), from the
|
Best full-gallery combo per model (all three are `full_exp`), from the
|
||||||
training matrix in [the full experiment log](model-bakeoff.md):
|
training matrix in [the full experiment log](model-bakeoff-2026-07.md):
|
||||||
|
|
||||||
| model | F1 | P | R | misID |
|
| model | F1 | P | R | misID |
|
||||||
|---|---|---|---|---|
|
|---|---|---|---|---|
|
||||||
@@ -1,3 +1,5 @@
|
|||||||
|
> **Archived (July 2026).** This report covers the pre-opencv5 framework and the 4-model ArcFace/LVFace bake-off. It is superseded by the current [experiment log](model-bakeoff.md) for the opencv5 build. Kept for provenance; the numbers here are historical.
|
||||||
|
|
||||||
# Whole gallery vs. cast-restricted gallery
|
# Whole gallery vs. cast-restricted gallery
|
||||||
|
|
||||||
Two ways to run the matcher. Full mode scores every detected face against
|
Two ways to run the matcher. Full mode scores every detected face against
|
||||||
@@ -8,7 +10,7 @@ top-billed actors) before the matcher runs.
|
|||||||
## Result
|
## Result
|
||||||
|
|
||||||
Averaged across the 3 compared models (r50 excluded, see
|
Averaged across the 3 compared models (r50 excluded, see
|
||||||
[the full experiment log](model-bakeoff.md)) and both expansion settings, on
|
[the full experiment log](model-bakeoff-2026-07.md)) and both expansion settings, on
|
||||||
the 4 training films:
|
the 4 training films:
|
||||||
|
|
||||||
| scope | F1 | P | R | total misID |
|
| scope | F1 | P | R | total misID |
|
||||||
@@ -27,7 +29,7 @@ restricted gallery:
|
|||||||
|
|
||||||

|

|
||||||
|
|
||||||
See [the full experiment log](model-bakeoff.md) for the complete table. One
|
See [the full experiment log](model-bakeoff-2026-07.md) for the complete table. One
|
||||||
combo reaches zero true out-of-cast misidentifications,
|
combo reaches zero true out-of-cast misidentifications,
|
||||||
`arcface_w600k_mbf_restricted_exp` (F1 76.2%), and it is a restricted one,
|
`arcface_w600k_mbf_restricted_exp` (F1 76.2%), and it is a restricted one,
|
||||||
consistent with restriction, not expansion, being what suppresses cross-film
|
consistent with restriction, not expansion, being what suppresses cross-film
|
||||||
@@ -57,7 +59,7 @@ Building this as a real feature requires:
|
|||||||
option.
|
option.
|
||||||
- A decision on the fallback case: what happens to a real, uncredited
|
- A decision on the fallback case: what happens to a real, uncredited
|
||||||
cameo (see the Germar Terrell Gardner and Talia Balsam cases in the
|
cameo (see the Germar Terrell Gardner and Talia Balsam cases in the
|
||||||
[LVFace deep dive](lvface-deep-dive.md#where-lvface-beat-x-ray)) if the
|
[LVFace deep dive](lvface-deep-dive-2026-07.md#where-lvface-beat-x-ray)) if the
|
||||||
restricted gallery never includes them at all.
|
restricted gallery never includes them at all.
|
||||||
- Regenerating the restricted-gallery cache whenever a title's Jellyfin
|
- Regenerating the restricted-gallery cache whenever a title's Jellyfin
|
||||||
cast list changes.
|
cast list changes.
|
||||||
@@ -1,12 +1,14 @@
|
|||||||
|
> **Archived (July 2026).** This report covers the pre-opencv5 framework and the 4-model ArcFace/LVFace bake-off. It is superseded by the current [experiment log](model-bakeoff.md) for the opencv5 build. Kept for provenance; the numbers here are historical.
|
||||||
|
|
||||||
# Deep dive: LVFace-B Glint360K
|
# Deep dive: LVFace-B Glint360K
|
||||||
|
|
||||||
LVFace won the model comparison (see [Which model is best?](best-model.md))
|
LVFace won the model comparison (see [Which model is best?](best-model-2026-07.md))
|
||||||
and is the shipped default embedder. This page reports how it performs in
|
and is the shipped default embedder. This page reports how it performs in
|
||||||
detail: a baseline of correct output, the two mechanisms behind its errors,
|
detail: a baseline of correct output, the two mechanisms behind its errors,
|
||||||
and every distinct case where it names someone who is not in the film's
|
and every distinct case where it names someone who is not in the film's
|
||||||
credited cast.
|
credited cast.
|
||||||
|
|
||||||
Read [How we score against X-Ray](methodology.md) first. X-Ray's ground truth
|
Read [How we score against X-Ray](methodology-2026-07.md) first. X-Ray's ground truth
|
||||||
is scene-level, not per-frame. A name marked correct in the Offscreen column
|
is scene-level, not per-frame. A name marked correct in the Offscreen column
|
||||||
below is the pipeline correctly reporting scene membership, not a workaround.
|
below is the pipeline correctly reporting scene membership, not a workaround.
|
||||||
|
|
||||||
@@ -61,7 +63,7 @@ on the 5 films the optimizer never saw:
|
|||||||
| macro average | 67.4% | 85.8% | 57.0% | | | | |
|
| macro average | 67.4% | 85.8% | 57.0% | | | | |
|
||||||
|
|
||||||
The `P` column is misID-weighted (each out-of-film name counts 10x in the
|
The `P` column is misID-weighted (each out-of-film name counts 10x in the
|
||||||
denominator; see [methodology](methodology.md#precision-recall-and-the-misid-weighting)).
|
denominator; see [methodology](methodology-2026-07.md#precision-recall-and-the-misid-weighting)).
|
||||||
That weighting is why Many Saints reads 54.7% here despite naming mostly real,
|
That weighting is why Many Saints reads 54.7% here despite naming mostly real,
|
||||||
present faces: its raw (unweighted) precision is **78.4%**, and the gap is
|
present faces: its raw (unweighted) precision is **78.4%**, and the gap is
|
||||||
entirely its 974 misIDs paying the 10x penalty. The three zero-misID films
|
entirely its 974 misIDs paying the 10x penalty. The three zero-misID films
|
||||||
@@ -70,7 +72,7 @@ Lovelace, with 58 misIDs, sits 3pp below its raw 93.3%.
|
|||||||
|
|
||||||
Held-out F1 is 67.4%, against 75.3% on training, an 8pp drop. The spread
|
Held-out F1 is 67.4%, against 75.3% on training, an 8pp drop. The spread
|
||||||
between the best and worst held-out film is 37pp. This is not unique to
|
between the best and worst held-out film is 37pp. This is not unique to
|
||||||
LVFace: [the full experiment log](model-bakeoff.md#held-out-validation-all-3-models)
|
LVFace: [the full experiment log](model-bakeoff-2026-07.md#held-out-validation-all-3-models)
|
||||||
shows mbf and r18 with the same shape of spread on the same films, at a
|
shows mbf and r18 with the same shape of spread on the same films, at a
|
||||||
uniformly lower level. Two mechanisms explain the spread. Both are shown
|
uniformly lower level. Two mechanisms explain the spread. Both are shown
|
||||||
below with frame-level evidence.
|
below with frame-level evidence.
|
||||||
@@ -0,0 +1,136 @@
|
|||||||
|
> **Archived (July 2026).** This report covers the pre-opencv5 framework and the 4-model ArcFace/LVFace bake-off. It is superseded by the current [experiment log](model-bakeoff.md) for the opencv5 build. Kept for provenance; the numbers here are historical.
|
||||||
|
|
||||||
|
# How we score against X-Ray
|
||||||
|
|
||||||
|
Every number in this report, every F1 and misID count, comes from one
|
||||||
|
comparison. The comparison has a mismatch at its core that shapes nearly
|
||||||
|
every finding in this report: the ground truth is scene-level, the
|
||||||
|
pipeline's output is per-second, and the two do not mean the same thing.
|
||||||
|
This page documents that comparison once, so the findings pages can rely on
|
||||||
|
it without re-explaining it.
|
||||||
|
|
||||||
|
## What Amazon X-Ray records
|
||||||
|
|
||||||
|
X-Ray ships three tables per film: `scenes.csv` (a list of `[start, end]`
|
||||||
|
timespans), `people_in_scenes.csv` (which actors are credited in each
|
||||||
|
scene), and `people.csv` (actor identities). There is no per-frame or
|
||||||
|
per-second annotation anywhere in X-Ray. A scene might run 45 seconds, and
|
||||||
|
X-Ray records one cast list for the entire span, not "on screen from
|
||||||
|
second 12 to second 30."
|
||||||
|
|
||||||
|
To compare this against per-second predictions, `second_score.py` expands
|
||||||
|
every scene into per-second ground truth by copying the whole scene's cast
|
||||||
|
list onto every second inside it:
|
||||||
|
|
||||||
|
```python
|
||||||
|
for sn, (t0, t1) in spans.items():
|
||||||
|
cast = scene_cast.get(sn, [])
|
||||||
|
for t in range(int(t0), int(t1)):
|
||||||
|
timeline[t] = cast
|
||||||
|
```
|
||||||
|
|
||||||
|
That is the entire mechanism. If X-Ray credits five actors to a 30-second
|
||||||
|
scene, all five count as ground truth present for all 30 seconds, including
|
||||||
|
seconds where only one of them is on screen. This is not a simplification
|
||||||
|
introduced by the pipeline; it is the only reading of X-Ray's data that is
|
||||||
|
possible, because X-Ray itself does not record anything finer-grained.
|
||||||
|
|
||||||
|
## Why an offscreen name can be scored correct
|
||||||
|
|
||||||
|
A name listed under Offscreen with a correct (green) label is not the
|
||||||
|
pipeline guessing or padding its score. It is the pipeline correctly
|
||||||
|
answering the question X-Ray actually asks: is this actor part of this
|
||||||
|
scene. It answers that question using a presence window (`[start, end]`,
|
||||||
|
held open across cuts by `anneal_sec` and `extinction_sec`), which matches
|
||||||
|
X-Ray's scene-level semantics more closely than a raw per-frame detection
|
||||||
|
would.
|
||||||
|
|
||||||
|
A system that only reported "this actor is visible in this exact frame"
|
||||||
|
would score worse against X-Ray's scene-level ground truth, producing a
|
||||||
|
false negative every time the camera cuts away from a character who is
|
||||||
|
still present in the scene. Not because it is wrong about the world, but
|
||||||
|
because it would be answering a stricter, different question than the one
|
||||||
|
X-Ray's data supports. The presence-window design exists specifically to
|
||||||
|
answer X-Ray's actual question.
|
||||||
|
|
||||||
|
## What this resolves and what it does not
|
||||||
|
|
||||||
|
This resolves the semantic mismatch between a scene and an instant. It does
|
||||||
|
not resolve two other limitations, both discussed in the
|
||||||
|
[LVFace deep dive](lvface-deep-dive-2026-07.md).
|
||||||
|
|
||||||
|
**The face-vs-presence ceiling.** X-Ray credits scene membership regardless
|
||||||
|
of whether a face is ever visible: background crew, characters shot from
|
||||||
|
behind, voice-only presence. No amount of bridging recovers a face that
|
||||||
|
never appears on screen. This is a hard ceiling on recall, not a defect.
|
||||||
|
|
||||||
|
**Extinction bridging can overshoot.** The same presence-window mechanism
|
||||||
|
that correctly answers "still in this scene" during a normal cut can also
|
||||||
|
bridge across a scene boundary it has no way to detect. A hard cut into a
|
||||||
|
different scene with no faces, such as closing credits, carries the
|
||||||
|
previous scene's identities forward until the window expires. This is the
|
||||||
|
mechanism behind Downton Abbey's recall collapse, documented in the deep
|
||||||
|
dive.
|
||||||
|
|
||||||
|
## Precision, recall, and the misID weighting
|
||||||
|
|
||||||
|
Per sampled second `t`:
|
||||||
|
|
||||||
|
**TPI** (true positive instances): actors both X-Ray and the pipeline agree
|
||||||
|
are present.
|
||||||
|
|
||||||
|
**FPI** (false positive instances): actors the pipeline reports that are
|
||||||
|
not in X-Ray's cast for this second. Split into two categories:
|
||||||
|
|
||||||
|
- **FPI_incast**: the actor is in the film's cast, just not credited to
|
||||||
|
this particular scene. A timing or boundary slip.
|
||||||
|
- **FPI_misid**: the actor is not in the film's cast at all. A genuine
|
||||||
|
wrong-identity error, weighted 10x in the precision objective, because
|
||||||
|
naming someone who is not even in the film is a categorically worse
|
||||||
|
error than a few seconds of scene-boundary slop.
|
||||||
|
|
||||||
|
!!! note "Every headline `P` and `F1` is misID-weighted"
|
||||||
|
|
||||||
|
The precision reported throughout this report, and therefore the F1
|
||||||
|
derived from it, puts each `FPI_misid` into the denominator **10 times**
|
||||||
|
(`precision = TPI / (TPI + FPI_incast + 10·FPI_misid)`,
|
||||||
|
[`second_score.py`](https://REPOLINK/scripts/optimizer/second_score.py)).
|
||||||
|
This is deliberate: the whole point is to punish naming an out-of-film
|
||||||
|
actor far harder than a scene-boundary slip. But it means the `P` column
|
||||||
|
is not raw precision, and a misID-heavy film's `P` is depressed
|
||||||
|
super-linearly. `second_score.py` also emits an unweighted `precision_raw`
|
||||||
|
(always ≥ the weighted `P`); where the gap matters, The Many Saints of
|
||||||
|
Newark, weighted `P` 54.7% vs. raw 78.4%, the [LVFace deep dive](lvface-deep-dive-2026-07.md)
|
||||||
|
reports both. When comparing `P` across films, remember you are comparing a
|
||||||
|
quantity that penalizes misIDs, not just a hit rate.
|
||||||
|
|
||||||
|
**FN** (false negatives): actors X-Ray lists that the pipeline never
|
||||||
|
reports, counted only for actors who have a gallery reference embedding.
|
||||||
|
Across the 9-film benchmark, coverage of X-Ray's credited cast ranges from
|
||||||
|
20% to 79% by film (see
|
||||||
|
[the full experiment log](model-bakeoff-2026-07.md#gallery-coverage-per-film)); an
|
||||||
|
actor with no reference photo can never be recognized regardless of model
|
||||||
|
quality, and counting them as a miss would penalize gallery coverage, not
|
||||||
|
recognition accuracy.
|
||||||
|
|
||||||
|
Two further numbers are reported alongside F1:
|
||||||
|
|
||||||
|
**agreement_rate**: mean per-second Jaccard overlap
|
||||||
|
(`|Pred ∩ GT| / |Pred ∪ GT|`), partial credit. Naming 2 of 3 present actors
|
||||||
|
scores 2/3, not 0.
|
||||||
|
|
||||||
|
**exact_match_rate**: the fraction of sampled seconds where the pipeline's
|
||||||
|
named set exactly equals X-Ray's, no partial credit. Far harsher, and
|
||||||
|
dominated by recall, since any single missed actor zeroes that second.
|
||||||
|
|
||||||
|
## Reproduce
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 scripts/optimizer/second_score.py \
|
||||||
|
--pred pred.json --xray experiments/xray/.../<xray_dir> \
|
||||||
|
--gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5
|
||||||
|
```
|
||||||
|
|
||||||
|
See also [the full experiment log](model-bakeoff-2026-07.md) for how `pred.json` is
|
||||||
|
produced, and the [LVFace deep dive](lvface-deep-dive-2026-07.md) for what these
|
||||||
|
mechanisms look like frame by frame.
|
||||||
+70
-78
@@ -1,11 +1,9 @@
|
|||||||
# How we score against X-Ray
|
# How we score against X-Ray
|
||||||
|
|
||||||
Every number in this report, every F1 and misID count, comes from one
|
Every number in this report comes from one comparison, and that comparison
|
||||||
comparison. The comparison has a mismatch at its core that shapes nearly
|
has a mismatch at its core: the ground truth is scene-level, the pipeline's
|
||||||
every finding in this report: the ground truth is scene-level, the
|
output is per-second, and the two do not mean the same thing. This page
|
||||||
pipeline's output is per-second, and the two do not mean the same thing.
|
documents the comparison once so the findings can rely on it.
|
||||||
This page documents that comparison once, so the findings pages can rely on
|
|
||||||
it without re-explaining it.
|
|
||||||
|
|
||||||
## What Amazon X-Ray records
|
## What Amazon X-Ray records
|
||||||
|
|
||||||
@@ -13,12 +11,12 @@ X-Ray ships three tables per film: `scenes.csv` (a list of `[start, end]`
|
|||||||
timespans), `people_in_scenes.csv` (which actors are credited in each
|
timespans), `people_in_scenes.csv` (which actors are credited in each
|
||||||
scene), and `people.csv` (actor identities). There is no per-frame or
|
scene), and `people.csv` (actor identities). There is no per-frame or
|
||||||
per-second annotation anywhere in X-Ray. A scene might run 45 seconds, and
|
per-second annotation anywhere in X-Ray. A scene might run 45 seconds, and
|
||||||
X-Ray records one cast list for the entire span, not "on screen from
|
X-Ray records one cast list for the entire span, not "on screen from second
|
||||||
second 12 to second 30."
|
12 to second 30."
|
||||||
|
|
||||||
To compare this against per-second predictions, `second_score.py` expands
|
To compare against per-second predictions, `second_score.py` expands every
|
||||||
every scene into per-second ground truth by copying the whole scene's cast
|
scene into per-second ground truth by copying the whole scene's cast list
|
||||||
list onto every second inside it:
|
onto every second inside it:
|
||||||
|
|
||||||
```python
|
```python
|
||||||
for sn, (t0, t1) in spans.items():
|
for sn, (t0, t1) in spans.items():
|
||||||
@@ -27,48 +25,46 @@ for sn, (t0, t1) in spans.items():
|
|||||||
timeline[t] = cast
|
timeline[t] = cast
|
||||||
```
|
```
|
||||||
|
|
||||||
That is the entire mechanism. If X-Ray credits five actors to a 30-second
|
If X-Ray credits five actors to a 30-second scene, all five count as ground
|
||||||
scene, all five count as ground truth present for all 30 seconds, including
|
truth present for all 30 seconds, including seconds where only one is on
|
||||||
seconds where only one of them is on screen. This is not a simplification
|
screen. This is not a simplification the pipeline introduces; it is the only
|
||||||
introduced by the pipeline; it is the only reading of X-Ray's data that is
|
reading X-Ray's data supports, because X-Ray records nothing finer.
|
||||||
possible, because X-Ray itself does not record anything finer-grained.
|
|
||||||
|
|
||||||
## Why an offscreen name can be scored correct
|
## How the pipeline reports presence
|
||||||
|
|
||||||
A name listed under Offscreen with a correct (green) label is not the
|
A presence claim is one actor owning one time window. How that window is
|
||||||
pipeline guessing or padding its score. It is the pipeline correctly
|
derived is a tunable choice — a knob the optimizer weighs — with two modes:
|
||||||
answering the question X-Ray actually asks: is this actor part of this
|
|
||||||
scene. It answers that question using a presence window (`[start, end]`,
|
|
||||||
held open across cuts by `anneal_sec` and `extinction_sec`), which matches
|
|
||||||
X-Ray's scene-level semantics more closely than a raw per-frame detection
|
|
||||||
would.
|
|
||||||
|
|
||||||
A system that only reported "this actor is visible in this exact frame"
|
- **`track_extent` (default).** A claim is exactly `[first_seen, last_seen]`
|
||||||
would score worse against X-Ray's scene-level ground truth, producing a
|
of a track the actor owned (AR-012), ending at the last sighting and never
|
||||||
false negative every time the camera cuts away from a character who is
|
after (AR-013). There is no keep-alive: the withdrawn `anneal_sec` and the
|
||||||
still present in the scene. Not because it is wrong about the world, but
|
scene-tracker `extinction_sec` — which the July report's windows were held
|
||||||
because it would be answering a stricter, different question than the one
|
open by — are **gone**. A track that survives its own gaps needs no bridge;
|
||||||
X-Ray's data supports. The presence-window design exists specifically to
|
a gap after the final sighting is never claimed.
|
||||||
answer X-Ray's actual question.
|
- **`flood`.** Each claim is snapped to the shot it sits in, so an actor seen
|
||||||
|
once anywhere in a shot is reported for the whole shot
|
||||||
|
`[prev_boundary, next_boundary]`. Boundaries come from TransNetV2 shot
|
||||||
|
detection when available, otherwise from the always-on histogram cut
|
||||||
|
detector (`is_cut`). This trades precision for recall against X-Ray's
|
||||||
|
scene-level granularity, and the optimizer decides per run whether it pays.
|
||||||
|
|
||||||
## What this resolves and what it does not
|
Do not confuse the surviving `track_extinction_sec` with the withdrawn
|
||||||
|
scene `extinction_sec`: the former bounds how long a lost track stays
|
||||||
|
available for **re-association** (a tracking question), and never extends a
|
||||||
|
presence claim.
|
||||||
|
|
||||||
This resolves the semantic mismatch between a scene and an instant. It does
|
## The two limits this does not resolve
|
||||||
not resolve two other limitations, both discussed in the
|
|
||||||
[LVFace deep dive](lvface-deep-dive.md).
|
|
||||||
|
|
||||||
**The face-vs-presence ceiling.** X-Ray credits scene membership regardless
|
**The face-vs-presence ceiling.** X-Ray credits scene membership regardless
|
||||||
of whether a face is ever visible: background crew, characters shot from
|
of whether a face is ever visible: background crew, characters shot from
|
||||||
behind, voice-only presence. No amount of bridging recovers a face that
|
behind, voice-only presence. No face pipeline can recover a face that never
|
||||||
never appears on screen. This is a hard ceiling on recall, not a defect.
|
appears, so recall against X-Ray is a structural ceiling, not a defect.
|
||||||
|
|
||||||
**Extinction bridging can overshoot.** The same presence-window mechanism
|
**Flood-fill can overshoot.** Snapping to a shot correctly answers "still in
|
||||||
that correctly answers "still in this scene" during a normal cut can also
|
this scene" through an intra-scene cut, but a shot boundary is not a scene
|
||||||
bridge across a scene boundary it has no way to detect. A hard cut into a
|
boundary: on a film with sparse cuts, flood-fill can carry an actor across a
|
||||||
different scene with no faces, such as closing credits, carries the
|
long "shot" they only briefly appeared in. This is why flood-fill is a knob,
|
||||||
previous scene's identities forward until the window expires. This is the
|
not a default — its value depends on the film's cut density.
|
||||||
mechanism behind Downton Abbey's recall collapse, documented in the deep
|
|
||||||
dive.
|
|
||||||
|
|
||||||
## Precision, recall, and the misID weighting
|
## Precision, recall, and the misID weighting
|
||||||
|
|
||||||
@@ -77,49 +73,46 @@ Per sampled second `t`:
|
|||||||
**TPI** (true positive instances): actors both X-Ray and the pipeline agree
|
**TPI** (true positive instances): actors both X-Ray and the pipeline agree
|
||||||
are present.
|
are present.
|
||||||
|
|
||||||
**FPI** (false positive instances): actors the pipeline reports that are
|
**FPI** (false positive instances): actors the pipeline reports that are not
|
||||||
not in X-Ray's cast for this second. Split into two categories:
|
in X-Ray's cast for this second, split into:
|
||||||
|
|
||||||
- **FPI_incast**: the actor is in the film's cast, just not credited to
|
- **FPI_incast**: the actor is in the film's cast, just not credited to this
|
||||||
this particular scene. A timing or boundary slip.
|
scene. A timing or boundary slip.
|
||||||
- **FPI_misid**: the actor is not in the film's cast at all. A genuine
|
- **FPI_misid**: the actor is not in the film's cast at all. A genuine
|
||||||
wrong-identity error, weighted 10x in the precision objective, because
|
wrong-identity error, weighted **10×** in the precision objective, because
|
||||||
naming someone who is not even in the film is a categorically worse
|
naming someone not even in the film is categorically worse than a few
|
||||||
error than a few seconds of scene-boundary slop.
|
seconds of scene-boundary slop.
|
||||||
|
|
||||||
!!! note "Every headline `P` and `F1` is misID-weighted"
|
!!! note "Every headline `P` and `F1` is misID-weighted"
|
||||||
|
|
||||||
The precision reported throughout this report, and therefore the F1
|
Precision puts each `FPI_misid` into the denominator 10 times
|
||||||
derived from it, puts each `FPI_misid` into the denominator **10 times**
|
|
||||||
(`precision = TPI / (TPI + FPI_incast + 10·FPI_misid)`,
|
(`precision = TPI / (TPI + FPI_incast + 10·FPI_misid)`,
|
||||||
[`second_score.py`](https://REPOLINK/scripts/optimizer/second_score.py)).
|
[`second_score.py`](https://REPOLINK/scripts/optimizer/second_score.py)).
|
||||||
This is deliberate: the whole point is to punish naming an out-of-film
|
This deliberately punishes naming an out-of-film actor far harder than a
|
||||||
actor far harder than a scene-boundary slip. But it means the `P` column
|
boundary slip, so the `P` column is not raw precision and a misID-heavy
|
||||||
is not raw precision, and a misID-heavy film's `P` is depressed
|
film's `P` is depressed super-linearly.
|
||||||
super-linearly. `second_score.py` also emits an unweighted `precision_raw`
|
|
||||||
(always ≥ the weighted `P`); where the gap matters, The Many Saints of
|
|
||||||
Newark, weighted `P` 54.7% vs. raw 78.4%, the [LVFace deep dive](lvface-deep-dive.md)
|
|
||||||
reports both. When comparing `P` across films, remember you are comparing a
|
|
||||||
quantity that penalizes misIDs, not just a hit rate.
|
|
||||||
|
|
||||||
**FN** (false negatives): actors X-Ray lists that the pipeline never
|
**FN** (false negatives): actors X-Ray lists that the pipeline never reports,
|
||||||
reports, counted only for actors who have a gallery reference embedding.
|
counted **only** for actors who have a gallery reference embedding. An actor
|
||||||
Across the 9-film benchmark, coverage of X-Ray's credited cast ranges from
|
with no reference photo can never be recognized, and counting them as a miss
|
||||||
20% to 79% by film (see
|
would measure gallery coverage, not recognition accuracy.
|
||||||
[the full experiment log](model-bakeoff.md#gallery-coverage-per-film)); an
|
|
||||||
actor with no reference photo can never be recognized regardless of model
|
|
||||||
quality, and counting them as a miss would penalize gallery coverage, not
|
|
||||||
recognition accuracy.
|
|
||||||
|
|
||||||
Two further numbers are reported alongside F1:
|
Two further numbers accompany F1:
|
||||||
|
|
||||||
**agreement_rate**: mean per-second Jaccard overlap
|
**agreement_rate**: mean per-second Jaccard overlap
|
||||||
(`|Pred ∩ GT| / |Pred ∪ GT|`), partial credit. Naming 2 of 3 present actors
|
(`|Pred ∩ GT| / |Pred ∪ GT|`) — partial credit, so naming 2 of 3 present
|
||||||
scores 2/3, not 0.
|
actors scores 2/3, not 0.
|
||||||
|
|
||||||
**exact_match_rate**: the fraction of sampled seconds where the pipeline's
|
**exact_match_rate**: the fraction of seconds where the pipeline's named set
|
||||||
named set exactly equals X-Ray's, no partial credit. Far harsher, and
|
exactly equals X-Ray's — no partial credit, dominated by recall.
|
||||||
dominated by recall, since any single missed actor zeroes that second.
|
|
||||||
|
## The benchmark set
|
||||||
|
|
||||||
|
Unlike the July report — which trained on a 3-film subset and validated on
|
||||||
|
held-out films to keep evaluations fast — this run scores **all 9 films on
|
||||||
|
every evaluation**. The registry one-clock fix and uncapped dumps made
|
||||||
|
full-set replay affordable, so the reported optimum is tuned against the
|
||||||
|
complete set rather than a training subset.
|
||||||
|
|
||||||
## Reproduce
|
## Reproduce
|
||||||
|
|
||||||
@@ -129,6 +122,5 @@ python3 scripts/optimizer/second_score.py \
|
|||||||
--gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5
|
--gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5
|
||||||
```
|
```
|
||||||
|
|
||||||
See also [the full experiment log](model-bakeoff.md) for how `pred.json` is
|
See the [full experiment log](model-bakeoff.md) for how `pred.json` is
|
||||||
produced, and the [LVFace deep dive](lvface-deep-dive.md) for what these
|
produced and where the shipped `src/config.hpp` defaults come from.
|
||||||
mechanisms look like frame by frame.
|
|
||||||
|
|||||||
@@ -1,3 +1,5 @@
|
|||||||
|
> **Archived (July 2026).** This report covers the pre-opencv5 framework and the 4-model ArcFace/LVFace bake-off. It is superseded by the current [experiment log](model-bakeoff.md) for the opencv5 build. Kept for provenance; the numbers here are historical.
|
||||||
|
|
||||||
# Full experiment log
|
# Full experiment log
|
||||||
|
|
||||||
This page reports how the pipeline performs across three questions: which
|
This page reports how the pipeline performs across three questions: which
|
||||||
@@ -7,7 +9,7 @@ a per-film gallery annex helps. It also documents the replay architecture
|
|||||||
that made testing all three questions in one pass practical, and every
|
that made testing all three questions in one pass practical, and every
|
||||||
caveat needed to trust the numbers.
|
caveat needed to trust the numbers.
|
||||||
|
|
||||||
Read [How we score against X-Ray](methodology.md) first for what F1,
|
Read [How we score against X-Ray](methodology-2026-07.md) first for what F1,
|
||||||
precision, recall, and misID mean in this report. All numbers below use the
|
precision, recall, and misID mean in this report. All numbers below use the
|
||||||
per-second metric
|
per-second metric
|
||||||
([`scripts/optimizer/second_score.py`](https://REPOLINK/scripts/optimizer/second_score.py)).
|
([`scripts/optimizer/second_score.py`](https://REPOLINK/scripts/optimizer/second_score.py)).
|
||||||
@@ -17,7 +19,7 @@ gallery was built with roughly 30% fewer reference images per actor than the
|
|||||||
other three models on the identical source photos (10808 vs 15055 total
|
other three models on the identical source photos (10808 vs 15055 total
|
||||||
embeddings across the same 2418 actors), which confounds any direct
|
embeddings across the same 2418 actors), which confounds any direct
|
||||||
comparison of its scores against the others. It remains in the
|
comparison of its scores against the others. It remains in the
|
||||||
[calibration curve comparison](best-model.md#first-signal-calibration-curves),
|
[calibration curve comparison](best-model-2026-07.md#first-signal-calibration-curves),
|
||||||
which does not depend on the training benchmark.
|
which does not depend on the training benchmark.
|
||||||
|
|
||||||
## Why replay makes this affordable
|
## Why replay makes this affordable
|
||||||
@@ -104,7 +106,7 @@ missing references, not purely by model quality. Downton Abbey has 61%
|
|||||||
coverage, the second-best in the benchmark, yet the worst held-out recall
|
coverage, the second-best in the benchmark, yet the worst held-out recall
|
||||||
of any film (39.4%, LVFace). Its recall problem is not primarily a coverage
|
of any film (39.4%, LVFace). Its recall problem is not primarily a coverage
|
||||||
problem; it is the extinction-bridging failure documented in the
|
problem; it is the extinction-bridging failure documented in the
|
||||||
[LVFace deep dive](lvface-deep-dive.md#mechanism-1-extinction-bridging).
|
[LVFace deep dive](lvface-deep-dive-2026-07.md#mechanism-1-extinction-bridging).
|
||||||
Reproduce with `scripts/docs/gallery_coverage_per_film.py`.
|
Reproduce with `scripts/docs/gallery_coverage_per_film.py`.
|
||||||
|
|
||||||
## Training results, 3 models × 2 gallery modes × 2 expansion settings
|
## Training results, 3 models × 2 gallery modes × 2 expansion settings
|
||||||
@@ -141,7 +143,7 @@ restricted gallery, and LVFace leads within both gallery modes. `full_exp`
|
|||||||
(the shipped combination) is the best-scoring option that uses only
|
(the shipped combination) is the best-scoring option that uses only
|
||||||
features the running application currently supports; restriction is not
|
features the running application currently supports; restriction is not
|
||||||
wired into the application yet (see
|
wired into the application yet (see
|
||||||
[Whole vs. cast-restricted gallery](gallery-scope.md)).
|
[Whole vs. cast-restricted gallery](gallery-scope-2026-07.md)).
|
||||||
|
|
||||||
### A scoring bug worth recording: dropped-film evaluations
|
### A scoring bug worth recording: dropped-film evaluations
|
||||||
|
|
||||||
@@ -211,7 +213,7 @@ never flips. Total misIDs across the 5 films: LVFace 1032, mbf 2197, r18
|
|||||||
1224. LVFace has less than half mbf's misID count while also scoring
|
1224. LVFace has less than half mbf's misID count while also scoring
|
||||||
higher on every film. This directly confirms the model choice out of
|
higher on every film. This directly confirms the model choice out of
|
||||||
sample; it is not inferred from the training numbers alone. See the
|
sample; it is not inferred from the training numbers alone. See the
|
||||||
[LVFace deep dive](lvface-deep-dive.md) for frame-level detail on where and
|
[LVFace deep dive](lvface-deep-dive-2026-07.md) for frame-level detail on where and
|
||||||
why LVFace still fails on the two worst films. Reproduce with
|
why LVFace still fails on the two worst films. Reproduce with
|
||||||
`scripts/docs/run_holdout_all_models.py`.
|
`scripts/docs/run_holdout_all_models.py`.
|
||||||
|
|
||||||
@@ -234,7 +236,7 @@ candidates in the matcher's search space means fewer opportunities for a
|
|||||||
lookalike false match, and the recall gain shows this does not cost real
|
lookalike false match, and the recall gain shows this does not cost real
|
||||||
detections. Restriction is currently an offline optimizer technique, not a
|
detections. Restriction is currently an offline optimizer technique, not a
|
||||||
runtime feature of the application; see
|
runtime feature of the application; see
|
||||||
[Whole vs. cast-restricted gallery](gallery-scope.md) for what building it
|
[Whole vs. cast-restricted gallery](gallery-scope-2026-07.md) for what building it
|
||||||
into the application would require.
|
into the application would require.
|
||||||
|
|
||||||
**Pose expansion** (promoting a confidently identified track's novel-pose
|
**Pose expansion** (promoting a confidently identified track's novel-pose
|
||||||
@@ -257,7 +259,7 @@ against the full 2418-actor gallery, where a confidently learned pose is more
|
|||||||
likely to match the wrong person. On the full gallery it is a recall-vs-misID
|
likely to match the wrong person. On the full gallery it is a recall-vs-misID
|
||||||
trade, not a free gain. This training-set effect
|
trade, not a free gain. This training-set effect
|
||||||
did not reproduce on held-out data; see
|
did not reproduce on held-out data; see
|
||||||
[Does pose expansion help?](pose-expansion.md) for the full held-out test
|
[Does pose expansion help?](pose-expansion-2026-07.md) for the full held-out test
|
||||||
and the two methodology bugs caught while checking it.
|
and the two methodology bugs caught while checking it.
|
||||||
|
|
||||||
## Calibration curves
|
## Calibration curves
|
||||||
@@ -1,3 +1,5 @@
|
|||||||
|
> **Archived (July 2026).** This report covers the pre-opencv5 framework and the 4-model ArcFace/LVFace bake-off. It is superseded by the current [experiment log](model-bakeoff.md) for the opencv5 build. Kept for provenance; the numbers here are historical.
|
||||||
|
|
||||||
# Pose expansion: does promoting new poses mid-film help?
|
# Pose expansion: does promoting new poses mid-film help?
|
||||||
|
|
||||||
`expand_gallery`
|
`expand_gallery`
|
||||||
@@ -12,7 +14,7 @@ in the same film, without touching the baked gallery.
|
|||||||
|
|
||||||
Averaged across the 3 compared models (r50 excluded), on the 4 films used
|
Averaged across the 3 compared models (r50 excluded), on the 4 films used
|
||||||
for optimization. These are the corrected, full-coverage figures, see the
|
for optimization. These are the corrected, full-coverage figures, see the
|
||||||
[dropped-film note](model-bakeoff.md#a-scoring-bug-worth-recording-dropped-film-evaluations)
|
[dropped-film note](model-bakeoff-2026-07.md#a-scoring-bug-worth-recording-dropped-film-evaluations)
|
||||||
in the experiment log for why an earlier version of this table overstated the
|
in the experiment log for why an earlier version of this table overstated the
|
||||||
full-mode misID jump (209 → 864) that was itself partly a truncation artifact:
|
full-mode misID jump (209 → 864) that was itself partly a truncation artifact:
|
||||||
|
|
||||||
@@ -26,7 +28,7 @@ full-mode misID jump (209 → 864) that was itself partly a truncation artifact:
|
|||||||
In restricted mode, expansion looks like a clean win: +1.6pp F1, +3.3pp
|
In restricted mode, expansion looks like a clean win: +1.6pp F1, +3.3pp
|
||||||
recall, lower misID. In full mode it looks like a recall-for-misID trade:
|
recall, lower misID. In full mode it looks like a recall-for-misID trade:
|
||||||
+2.1pp F1, +3.9pp recall, but misID rises from 407 to 714. See
|
+2.1pp F1, +3.9pp recall, but misID rises from 407 to 714. See
|
||||||
[the full experiment log](model-bakeoff.md) for the per-model breakdown.
|
[the full experiment log](model-bakeoff-2026-07.md) for the per-model breakdown.
|
||||||
This asymmetry motivated the question below: does turning expansion on
|
This asymmetry motivated the question below: does turning expansion on
|
||||||
change what gets recognized frame by frame, or is the aggregate F1 shift
|
change what gets recognized frame by frame, or is the aggregate F1 shift
|
||||||
coming from something else.
|
coming from something else.
|
||||||
@@ -105,6 +107,6 @@ contribution, such as tagging which reference embedding won each match;
|
|||||||
neither was in scope for this pass.
|
neither was in scope for this pass.
|
||||||
|
|
||||||
Do not treat the training-set exp/noexp numbers in
|
Do not treat the training-set exp/noexp numbers in
|
||||||
[the full experiment log](model-bakeoff.md) as proof that expansion changes
|
[the full experiment log](model-bakeoff-2026-07.md) as proof that expansion changes
|
||||||
real-world behavior in either direction. On the evidence gathered so far,
|
real-world behavior in either direction. On the evidence gathered so far,
|
||||||
it does not move the needle enough to see.
|
it does not move the needle enough to see.
|
||||||
+7
-5
@@ -36,13 +36,15 @@ nav:
|
|||||||
- Home: index.md
|
- Home: index.md
|
||||||
- How We Score Against X-Ray: methodology.md
|
- How We Score Against X-Ray: methodology.md
|
||||||
- Benchmark — SuperHero: benchmark.md
|
- Benchmark — SuperHero: benchmark.md
|
||||||
- Findings:
|
|
||||||
- Best Model: best-model.md
|
|
||||||
- Gallery Scope (Full vs. Limited): gallery-scope.md
|
|
||||||
- Pose Expansion: pose-expansion.md
|
|
||||||
- LVFace Deep Dive: lvface-deep-dive.md
|
|
||||||
- Full Experiment Log: model-bakeoff.md
|
- Full Experiment Log: model-bakeoff.md
|
||||||
- Service Conversion (proposal): service-conversion.md
|
- Service Conversion (proposal): service-conversion.md
|
||||||
|
- Archive (July 2026):
|
||||||
|
- How We Scored (July): methodology-2026-07.md
|
||||||
|
- Best Model: best-model-2026-07.md
|
||||||
|
- Gallery Scope (Full vs. Limited): gallery-scope-2026-07.md
|
||||||
|
- Pose Expansion: pose-expansion-2026-07.md
|
||||||
|
- LVFace Deep Dive: lvface-deep-dive-2026-07.md
|
||||||
|
- Full Experiment Log (July): model-bakeoff-2026-07.md
|
||||||
|
|
||||||
markdown_extensions:
|
markdown_extensions:
|
||||||
- admonition
|
- admonition
|
||||||
|
|||||||
Reference in New Issue
Block a user