dtourolle
d340da755a
docs/rep4-optimizer-results.md is the main deliverable: the model bake-off + threshold re-tune experiment log, including the ROCm teardown deadlock root cause and fix, DE concurrency tuning, the 16-combo results table, held-out validation against 5 films never seen by the optimizer (macro F1 67.4% vs. 75.3% training — a real generalization gap), the frozen-bbox "ghost track" failure mode found via annotated frame evidence, calibration curves per model, and an isolated-effects breakdown of gallery scope vs. pose expansion. MkDocs site (mkdocs.yml, docs/index.md) renders docs/*.md; scripts/docs/ pulls referenced images from the artifact registry and generates the calibration chart at build time (see the tooling commit) rather than committing images to the repo. experiments/ now keeps only scripts + README + SESSION_STATE.md in git — every data artifact (galleries, dumps, X-Ray corpus, montage frames, trajectories, manifests, results) moved to the Gitea package registry. film-lut.template.json is the committed placeholder for the gitignored file-lut.json (real local movie paths, never shared — some source filenames carry scene-release tags). Adds models/transnetv2.onnx (via Git LFS, matching the other ONNX models) for the new scene-detection path.
6.5 KiB
Threshold optimization against Amazon X-Ray — experiment log
Record of the July 2026 work that tuned the pipeline's recognition/tracking defaults against ground-truth per-scene actor presence, and the tooling built to do it.
TL;DR — what changed
| knob | old default | new default | why |
|---|---|---|---|
prob_threshold |
0.99 | 0.76 | 0.99 was far too strict — halved recall for a fraction of a precision point. DE optimum, tightly converged. |
extinction_sec |
5.0 | 1.5 | Long extinction smears presence into later scenes → FPs. DE converged tightly low. |
anneal_sec |
10.0 | 10.0 (unchanged) | DE found it insensitive (F1 flat ±0.3pp across 3–26s) — kept the round default. |
detector_conf |
0.5 | 0.5 (unchanged) | Sweep showed raising it only trades recall for precision at a net F1 loss — near-threshold detections are real faces, not phantoms. |
Net effect on the 9-film benchmark (strict per-scene, augmented gallery): recall 58% → ~72%, F1 70% → ~76%, precision ~85%, at no meaningful precision cost.
Ground truth
Public scene-level Amazon X-Ray dataset (Zenodo DOI 10.5281/zenodo.17659734,
CC-BY-4.0): per movie, people.csv (name_id/person/character), scenes.csv
(scene/start/end ms), people_in_scenes.csv. Films matched to the library by an
authoritative Jellyfin ID join (query /Items?IncludeItemTypes=Movie&Fields= ProviderIds,Path, join Imdb/Tmdb against X-Ray metadata) — NOT fuzzy title matching,
which collides badly (TV episodes vs same-named films). 9 genuine films with source
video on disk: Benny & Joon, Café Society, Downton Abbey: A New Era, Lord of War,
Lovelace, The Many Saints of Newark, Scarface, Sound of Metal, Valerian.
The scoring metric (evolved through review)
Comparison unit is the X-Ray scene, not sampled timepoints. For each scene
[start,end]: predicted set = union of actors detected anywhere in the span;
GT set = actors X-Ray lists for that scene. Per scene TP/FP/FN, then:
- Precision: STRICT. Any predicted actor not in the scene's X-Ray set is an FP,
including out-of-cast confusions (no gallery∩cast masking). An earlier
timepoint-sampled, cast-masked metric HID ~570 such FPs across 9 films and let the
optimizer drive
prob_thresholdto the 0.50 floor — a metric artifact. Counting them is essential. - Recall: FAIR. FN counts only X-Ray cast members who are in the gallery. 67%
of X-Ray cast (261/392) have no gallery reference embedding and can never be
recognised — counting them as misses penalises coverage, not the threshold. Both
recall(fair) andrecall_strict(all) are reported. - Aggregation: per-scene F1 → duration-weighted average within a movie (long scenes count more) → equal-weight mean across movies (macro; each film counts the same regardless of length). This is the DE objective.
Implemented in scripts/optimizer/scene_score.py.
The gallery coverage gap
Diagnosing low recall: only 131 of 392 X-Ray cast were in the gallery (33%). Every
in-gallery actor HAD embeddings (gallery well-formed) — the gap was pure coverage.
scripts/optimizer/fetch_missing_actors.py recovers missing actors:
nm-id → TMDB /find external_ids → /person/{id}/images → download → embed (sae_embed),
with a --wikidata fallback (P345→P18 Commons photo).
- TMDB recovered 143/261 (55%). 0 face-detection failures; the rest had no TMDB person (60) or no profile photo (58). Coverage 33% → 70%.
- Wikidata fallback: 0/118 of the TMDB failures — only 4 even had a Commons photo, none yielded a detectable face. → TheTVDB not worth pursuing: these remaining actors are obscure enough that no image source covers them, AND (see below) most are off-camera anyway.
Coverage vs detectability. Adding references lifted recall (58→68% at fixed config) but modestly. Per-film drill-down (Lord of War: 12 actors recovered, only 1 had a detectable on-camera face) showed most missing cast are a detectability gap — X-Ray credits them as cast-in-scene (incl. off-camera/background), but their face never appears clearly for the pipeline to detect. This is a fundamental ceiling of a face-recognition pipeline vs X-Ray's presence semantics, not a fixable gap.
Optimizer
scripts/optimizer/optimize.py — scipy differential_evolution over the knob space,
each candidate = full replay of all films through the real C++ nodes (see the
KPN replay architecture below) scored by the metric above. Global objective (one
config for all films, not per-film).
Convergence stability (augmented gallery, 233 evals):
| knob | top-20 range | verdict |
|---|---|---|
prob_threshold |
0.69–0.83 (σ 0.05) | TIGHT — trust 0.76 |
extinction_sec |
1.0–2.2 (σ 0.33) | TIGHT — trust 1.5 |
anneal_sec |
3.1–26.3 (σ 6.4) | LOOSE — insensitive, not hard-coded |
F1 varied only 0.3pp across the top-20 → objective is flat near the optimum, so only the tightly-converged knobs were adopted as defaults.
Replay architecture (how the sweep is cheap)
The optimizer never re-decodes video. scene_analyze --dump-embeddings out.h5 runs the
expensive half once (decode→detect→align→embed) and dumps per-frame face embeddings
- metadata to HDF5 (
scripts/optimizer/SCHEMA.md).scripts/optimizer/replay.pythen replays that dump through the real C++face_tracker → identity_matcher → scene_trackerassembled in a Python KPN network (sae_kpnnanobind module), varying Config knobs freely — no GPU embedding, no decode. Verified BYTE-EXACT againstscene_analyze's own output. The dumps are gallery-independent, so testing the augmented gallery needed no re-dump.detector_confis replayable UPWARD only (the dump floor is 0.5).
Reproduce
# 1. dump (once per film, needs video)
scene_analyze --movie <f> --gallery gallery.json --dump-embeddings dump.h5 --fps 1
# 2. build films manifest by Jellyfin ID join (see scripts/optimizer notes)
# 3. optimize
python scripts/optimizer/optimize.py --manifest films.json --gallery gallery.json \
--params prob_threshold:0.5:0.999 anneal_sec:1:30 extinction_sec:1:15 \
--popsize 8 --maxiter 20 --trajectory traj.jsonl --out opt.json
# 4. score a fixed config / validate on a held-out set
python scripts/optimizer/score_config.py --manifest heldout.json --gallery gallery.json \
--config '{"prob_threshold":0.76,"extinction_sec":1.5,"anneal_sec":10}'
See also memory: kpn-python-replay-optimizer, gallery-coverage-gap, xray-validation-*.