dtourolle
4925443e56
Splits the rep4 write-up's key findings into their own linkable pages: - best-model.md: calibration curves first (discriminative power, independent of any threshold), then F1 on the benchmark — LVFace-B Glint360K wins both. - gallery-scope.md: whole vs. cast-restricted gallery, isolated from model and expansion choice — restriction wins on every axis, but isn't a shipped runtime feature yet. - pose-expansion.md: the training-set expand_gallery effect, and the held-out replication attempt that found it doesn't reproduce (5 films, 2 models, after catching and fixing a replay-timeout truncation bug and a bbox first-match-instead-of-best-match bug in the comparison harness itself). An honest null result, with the methodology errors documented since they're exactly the kind that manufacture a false "it works!" finding. - lvface-deep-dive.md: the winning model's held-out generalization gap, its two failure modes (frozen-bbox ghost tracks), and a verified case (cross- checked against Jellyfin's independent cast metadata) where LVFace correctly identified an actor that X-Ray's ground truth failed to credit. Adds a "report-highlights" artifact-registry package (scripts/artifacts/ push_artifacts.sh, pull_artifacts.sh) for hand-picked illustrative frames that aren't reproducible via the automated best/worst montage selection, and wires pulling it into scripts/docs/build_site.sh.
57 lines
2.8 KiB
Markdown
57 lines
2.8 KiB
Markdown
# Whole gallery vs. limited (cast-restricted) gallery
|
|
|
|
Two ways to run the matcher: **full** scores every detected face against the
|
|
entire library gallery (2418 actors across the 9-film benchmark set); **restricted**
|
|
pre-filters each film's gallery down to just its Jellyfin-credited cast (typically
|
|
~15 top-billed actors) before the matcher ever runs.
|
|
|
|
## The result
|
|
|
|
Averaged across all 4 models and both expansion settings, on the 4 rep4 training
|
|
films:
|
|
|
|
| scope | F1 | P | R | total misID (8 evals) |
|
|
|---|---|---|---|---|
|
|
| full | 71.2% | 91.1% | 59.0% | 1073 |
|
|
| **restricted** | **74.5%** | 92.2% | **62.9%** | **329** |
|
|
|
|
This is not a precision/recall trade — restriction wins on every axis at once:
|
|
**+3.3pp F1, +3.9pp recall, and less than a third the total misIDs.** Fewer
|
|
candidates in the matcher's search space means fewer opportunities for a
|
|
look-alike false match (an actor who happens to share enough facial structure
|
|
with someone in the film, but isn't actually in it), and the recall gain shows
|
|
it isn't costing real detections to get there.
|
|
|
|
Per-model, every single model's best-scoring combo in the full 16-way matrix is
|
|
a `restricted` variant — see the full table in `rep4-optimizer-results.md`. Two
|
|
combos hit **zero** true out-of-cast misidentifications:
|
|
`arcface_w600k_mbf_restricted_exp` (F1 76.5%) and, in full mode,
|
|
`LVFace-B_Glint360K_full_noexp` (F1 72.4%) — restriction isn't the only way to
|
|
reach misid=0, but it's the more reliable one.
|
|
|
|
## Why this isn't the shipped default
|
|
|
|
Cast-restriction is implemented today only as an **offline optimizer technique**
|
|
(`scripts/optimizer/cast_restrict.py`): it pre-builds a filtered gallery file
|
|
per film, using Jellyfin's own cast list, before the benchmark ever calls the
|
|
matcher. There's no runtime "restrict matching to this title's credited cast"
|
|
switch in the shipped application — `scene_analyze` always matches against
|
|
whatever single gallery file it's given.
|
|
|
|
Building that as a real feature would need, at minimum:
|
|
|
|
- A live Jellyfin cast lookup at analysis time (the title is already known —
|
|
`run_from_jellyfin.py` already does this same lookup for its own
|
|
`filter_gallery`-based restriction path, just not wired into `scene_analyze`
|
|
itself as a first-class option).
|
|
- A decision on the *fallback*: what happens to a real, uncredited cameo
|
|
(see the Germar Terrell Gardner case in the LVFace deep-dive) if the gallery
|
|
never includes them at all?
|
|
- Regenerating the restricted-gallery cache whenever the title's Jellyfin cast
|
|
list changes.
|
|
|
|
This is why the shipped `src/config.hpp` defaults use the `full`-mode winner
|
|
(`LVFace-B_Glint360K_full_exp`, F1 75.3% training / 67.4% held-out macro) rather
|
|
than the higher-scoring `restricted_exp` (78.3%) — the 78.3% number describes a
|
|
capability the app doesn't have yet, not what actually ships.
|