dtourolle
4925443e56
Splits the rep4 write-up's key findings into their own linkable pages: - best-model.md: calibration curves first (discriminative power, independent of any threshold), then F1 on the benchmark — LVFace-B Glint360K wins both. - gallery-scope.md: whole vs. cast-restricted gallery, isolated from model and expansion choice — restriction wins on every axis, but isn't a shipped runtime feature yet. - pose-expansion.md: the training-set expand_gallery effect, and the held-out replication attempt that found it doesn't reproduce (5 films, 2 models, after catching and fixing a replay-timeout truncation bug and a bbox first-match-instead-of-best-match bug in the comparison harness itself). An honest null result, with the methodology errors documented since they're exactly the kind that manufacture a false "it works!" finding. - lvface-deep-dive.md: the winning model's held-out generalization gap, its two failure modes (frozen-bbox ghost tracks), and a verified case (cross- checked against Jellyfin's independent cast metadata) where LVFace correctly identified an actor that X-Ray's ground truth failed to credit. Adds a "report-highlights" artifact-registry package (scripts/artifacts/ push_artifacts.sh, pull_artifacts.sh) for hand-picked illustrative frames that aren't reproducible via the automated best/worst montage selection, and wires pulling it into scripts/docs/build_site.sh.
2.8 KiB
Whole gallery vs. limited (cast-restricted) gallery
Two ways to run the matcher: full scores every detected face against the entire library gallery (2418 actors across the 9-film benchmark set); restricted pre-filters each film's gallery down to just its Jellyfin-credited cast (typically ~15 top-billed actors) before the matcher ever runs.
The result
Averaged across all 4 models and both expansion settings, on the 4 rep4 training films:
| scope | F1 | P | R | total misID (8 evals) |
|---|---|---|---|---|
| full | 71.2% | 91.1% | 59.0% | 1073 |
| restricted | 74.5% | 92.2% | 62.9% | 329 |
This is not a precision/recall trade — restriction wins on every axis at once: +3.3pp F1, +3.9pp recall, and less than a third the total misIDs. Fewer candidates in the matcher's search space means fewer opportunities for a look-alike false match (an actor who happens to share enough facial structure with someone in the film, but isn't actually in it), and the recall gain shows it isn't costing real detections to get there.
Per-model, every single model's best-scoring combo in the full 16-way matrix is
a restricted variant — see the full table in rep4-optimizer-results.md. Two
combos hit zero true out-of-cast misidentifications:
arcface_w600k_mbf_restricted_exp (F1 76.5%) and, in full mode,
LVFace-B_Glint360K_full_noexp (F1 72.4%) — restriction isn't the only way to
reach misid=0, but it's the more reliable one.
Why this isn't the shipped default
Cast-restriction is implemented today only as an offline optimizer technique
(scripts/optimizer/cast_restrict.py): it pre-builds a filtered gallery file
per film, using Jellyfin's own cast list, before the benchmark ever calls the
matcher. There's no runtime "restrict matching to this title's credited cast"
switch in the shipped application — scene_analyze always matches against
whatever single gallery file it's given.
Building that as a real feature would need, at minimum:
- A live Jellyfin cast lookup at analysis time (the title is already known —
run_from_jellyfin.pyalready does this same lookup for its ownfilter_gallery-based restriction path, just not wired intoscene_analyzeitself as a first-class option). - A decision on the fallback: what happens to a real, uncredited cameo (see the Germar Terrell Gardner case in the LVFace deep-dive) if the gallery never includes them at all?
- Regenerating the restricted-gallery cache whenever the title's Jellyfin cast list changes.
This is why the shipped src/config.hpp defaults use the full-mode winner
(LVFace-B_Glint360K_full_exp, F1 75.3% training / 67.4% held-out macro) rather
than the higher-scoring restricted_exp (78.3%) — the 78.3% number describes a
capability the app doesn't have yet, not what actually ships.