dtourolle
b1efefac6f
- experiment_charts.py generates 4 figures from experiments/ artifacts: held-out per-film F1, 16-combo ranking, DE search landscape, and the Downton detector-vs-tracker ghost timeline (replaces the blank title-card screenshot) - new frames: 19-correct wedding shot (success case), Many Saints ghost-vs-unknown frame (three error classes in one image) - rename rep4-optimizer-results.md -> model-bakeoff.md; rep4 kept only as the on-disk artifact prefix, explained once - repo file references are now links via https://REPOLINK/<path> placeholders; build_site.sh pins them to the HEAD commit's raw URLs and fails the build if a linked path doesn't exist at HEAD - drop references to removed scripts (scene_score.py, score_config.py) and to session-memory names; mark artifact-registry paths with their pull commands - commit readme_example.jpg + pipeline_topology.svg so README renders on the plain Gitea repo view - deploy_pages.sh: push built site/ to the gitea-pages branch
3.2 KiB
Whole gallery vs. limited (cast-restricted) gallery
Two ways to run the matcher: full scores every detected face against the entire library gallery (2418 actors across the 9-film benchmark set); restricted pre-filters each film's gallery down to just its Jellyfin-credited cast (typically ~15 top-billed actors) before the matcher ever runs.
The result
Averaged across all 4 models and both expansion settings, on the 4 bake-off training films:
| scope | F1 | P | R | total misID (8 evals) |
|---|---|---|---|---|
| full | 71.2% | 91.1% | 59.0% | 1073 |
| restricted | 74.5% | 92.2% | 62.9% | 329 |
This is not a precision/recall trade — restriction wins on every axis at once: +3.3pp F1, +3.9pp recall, and less than a third the total misIDs. Fewer candidates in the matcher's search space means fewer opportunities for a look-alike false match (an actor who happens to share enough facial structure with someone in the film, but isn't actually in it), and the recall gain shows it isn't costing real detections to get there.
Per-model, every single model's best-scoring combo in the full 16-way matrix is
a restricted variant — visible directly in the ranking below (filled dots =
restricted, open = full; the filled dots cluster at the top for every color):
See the full table in the
bake-off experiment log. Two
combos hit zero true out-of-cast misidentifications:
arcface_w600k_mbf_restricted_exp (F1 76.5%) and, in full mode,
LVFace-B_Glint360K_full_noexp (F1 72.4%) — restriction isn't the only way to
reach misid=0, but it's the more reliable one.
Why this isn't the shipped default
Cast-restriction is implemented today only as an offline optimizer technique
(scripts/optimizer/cast_restrict.py):
it pre-builds a filtered gallery file
per film, using Jellyfin's own cast list, before the benchmark ever calls the
matcher. There's no runtime "restrict matching to this title's credited cast"
switch in the shipped application — scene_analyze always matches against
whatever single gallery file it's given.
Building that as a real feature would need, at minimum:
- A live Jellyfin cast lookup at analysis time (the title is already known —
scripts/run_from_jellyfin.pyalready does this same lookup for its ownfilter_gallery-based restriction path, just not wired intoscene_analyzeitself as a first-class option). - A decision on the fallback: what happens to a real, uncredited cameo (see the Germar Terrell Gardner case in the LVFace deep-dive) if the gallery never includes them at all?
- Regenerating the restricted-gallery cache whenever the title's Jellyfin cast list changes.
This is why the shipped src/config.hpp
defaults use the full-mode winner
(LVFace-B_Glint360K_full_exp, F1 75.3% training / 67.4% held-out macro) rather
than the higher-scoring restricted_exp (78.3%) — the 78.3% number describes a
capability the app doesn't have yet, not what actually ships.