Files

T

dtourolle 4925443e56 docs: four focused findings pages (best model, gallery scope, expansion, deep dive)

Splits the rep4 write-up's key findings into their own linkable pages:
- best-model.md: calibration curves first (discriminative power, independent
  of any threshold), then F1 on the benchmark — LVFace-B Glint360K wins both.
- gallery-scope.md: whole vs. cast-restricted gallery, isolated from model and
  expansion choice — restriction wins on every axis, but isn't a shipped
  runtime feature yet.
- pose-expansion.md: the training-set expand_gallery effect, and the held-out
  replication attempt that found it doesn't reproduce (5 films, 2 models,
  after catching and fixing a replay-timeout truncation bug and a bbox
  first-match-instead-of-best-match bug in the comparison harness itself). An
  honest null result, with the methodology errors documented since they're
  exactly the kind that manufacture a false "it works!" finding.
- lvface-deep-dive.md: the winning model's held-out generalization gap, its
  two failure modes (frozen-bbox ghost tracks), and a verified case (cross-
  checked against Jellyfin's independent cast metadata) where LVFace
  correctly identified an actor that X-Ray's ground truth failed to credit.

Adds a "report-highlights" artifact-registry package (scripts/artifacts/
push_artifacts.sh, pull_artifacts.sh) for hand-picked illustrative frames that
aren't reproducible via the automated best/worst montage selection, and wires
pulling it into scripts/docs/build_site.sh.

2026-07-19 19:40:19 +02:00

5.1 KiB

Raw Blame History

Deep dive: LVFace-B Glint360K

LVFace won the model bake-off (see best-model.md) and is the shipped default embedder. This page is the honest accounting of how it actually performs — including where it's wrong, and one case where the ground truth itself is wrong and LVFace is right.

Training vs. held-out: the generalization gap

The shipped config (prob_threshold=0.754, anneal_sec=35.54, extinction_sec=57.43, expand_gallery=true) was tuned against 4 films. Scored against the 5 films the optimizer never saw:

film	F1	P	R	TPI	FPI	misid	FN
Benny & Joon	83.0%	89.1%	77.7%	15125	1846	0	4337
Lovelace	77.5%	90.3%	67.9%	14990	1085	58	7085
Valerian and the City of a Thousand Planets	74.1%	97.1%	60.0%	18663	548	0	12467
Downton Abbey: A New Era	56.2%	97.8%	39.4%	52027	1173	0	80084
The Many Saints of Newark	46.3%	54.7%	40.1%	15922	4394	974	23791
macro average	67.4%	85.8%	57.0%

67.4% held-out vs. 75.3% on training — an ~8pp drop, and a 37pp spread between the best and worst held-out film. The config does not generalize uniformly; two films are outright failure cases, for two different reasons.

Failure mode 1: frozen-bbox "ghost tracks"

Both Many Saints of Newark (974 misIDs) and Downton Abbey (FN=80084, the worst recall of the five) trace to the same root cause, verified directly against the raw per-frame stream and the HDF5 dump's own detection counts — not inferred from the score alone.

At this second, three of the four labeled boxes ("Jon Bernthal", "Joey Diaz", "Billy Magnussen") sit over empty background — a blurred wall, hanging plates — with no face in them. The real face in frame carries a second, colliding label from another frozen box.

This is the starkest case: 15 actors named, all wrong, over a completely blank closing title card. Confirmed against the dump directly: face_count is 0 from this point onward (no detector output at all), yet the same 15 identities keep appearing with the exact same bounding box, unchanged to the pixel, for 57+ consecutive seconds.

This is SceneTrackerFunc::active_[actor_idx].last_bbox (src/nodes/scene_tracker_node.hpp) being re-emitted unchanged — the extinction state machine working exactly as coded, not a bug. The film cuts from a packed group shot straight into 40+ seconds of blank titles/credits, and extinction_sec=57.4 is comfortably long enough to bridge that entire gap without expiring, so the tracker faithfully reports "last known position" for a cast that is no longer on screen at all. extinction_sec was tuned toward long windows specifically because they bridge real gaps (occlusion, a turned face) in most training footage — this is the cost side of that trade, surfacing only when a film has a long enough faceless stretch to expose it.

Failure mode 2: a genuine misID (for contrast)

Not every held-out failure is a ghost. This is a real face, correctly detected, confidently misidentified:

(same many_saints_ghost_fpi.jpg frame above also shows Leslie Odom Jr.'s box carrying a second, colliding "Michael Gandolfini" label — two real tracks' frozen positions happening to overlap, not a detection error.)

Where LVFace beat X-Ray

Not every "misID" is actually wrong. second_score.py counts a name as a true out-of-cast misID whenever the named actor isn't in X-Ray's credited cast list for the film at all — but X-Ray's cast list is itself incomplete.

Germar Terrell Gardner — a real, clean, high-confidence detection — is counted as a misID here because he doesn't appear in X-Ray's people.csv for The Many Saints of Newark at all. But Jellyfin's independent cast metadata does credit him for this exact film (cross-checked via experiments/manifests/ jellyfin_casts.json, a completely separate data source from X-Ray). This isn't a lookalike error or a gallery mixup — it's the pipeline correctly recognising a real cast member that one ground-truth source happened to omit.

This doesn't mean every flagged misID is secretly correct — Many Saints' 974-count total is still overwhelmingly the frozen-bbox failure mode above, not uncredited-but-real cameos. But it's a reminder that the X-Ray corpus is a convenient, large-scale ground truth, not a perfect one, and the "misID" number in any of these tables has some irreducible noise floor from ground-truth gaps in the other direction too.

Summary

LVFace is the right default: it wins the model comparison outright, and its failures are traceable, understood, and mostly attributable to one tunable knob (extinction_sec) rather than the embedder itself. The held-out generalization gap (75.3% → 67.4%) is real and should be treated as the honest expected performance, not the training-set number.

5.1 KiB Raw Blame History