dtourolle
b1efefac6f
- experiment_charts.py generates 4 figures from experiments/ artifacts: held-out per-film F1, 16-combo ranking, DE search landscape, and the Downton detector-vs-tracker ghost timeline (replaces the blank title-card screenshot) - new frames: 19-correct wedding shot (success case), Many Saints ghost-vs-unknown frame (three error classes in one image) - rename rep4-optimizer-results.md -> model-bakeoff.md; rep4 kept only as the on-disk artifact prefix, explained once - repo file references are now links via https://REPOLINK/<path> placeholders; build_site.sh pins them to the HEAD commit's raw URLs and fails the build if a linked path doesn't exist at HEAD - drop references to removed scripts (scene_score.py, score_config.py) and to session-memory names; mark artifact-registry paths with their pull commands - commit readme_example.jpg + pipeline_topology.svg so README renders on the plain Gitea repo view - deploy_pages.sh: push built site/ to the gitea-pages branch
126 lines
7.0 KiB
Markdown
126 lines
7.0 KiB
Markdown
# Threshold optimization against Amazon X-Ray — experiment log
|
||
|
||
Record of the July 2026 work that tuned the pipeline's recognition/tracking defaults
|
||
against ground-truth per-scene actor presence, and the tooling built to do it.
|
||
|
||
## TL;DR — what changed
|
||
|
||
| knob | old default | new default | why |
|
||
| ---- | ----------- | ----------- | --- |
|
||
| `prob_threshold` | 0.99 | **0.76** | 0.99 was far too strict — halved recall for a fraction of a precision point. DE optimum, tightly converged. |
|
||
| `extinction_sec` | 5.0 | **1.5** | Long extinction smears presence into later scenes → FPs. DE converged tightly low. |
|
||
| `anneal_sec` | 10.0 | 10.0 (unchanged) | DE found it **insensitive** (F1 flat ±0.3pp across 3–26s) — kept the round default. |
|
||
| `detector_conf` | 0.5 | 0.5 (unchanged) | Sweep showed raising it only trades recall for precision at a net F1 loss — near-threshold detections are real faces, not phantoms. |
|
||
|
||
Net effect on the 9-film benchmark (strict per-scene, augmented gallery):
|
||
recall **58% → ~72%**, F1 **70% → ~76%**, precision ~85%, at no meaningful precision cost.
|
||
|
||
## Ground truth
|
||
|
||
Public scene-level **Amazon X-Ray** dataset (Zenodo DOI 10.5281/zenodo.17659734,
|
||
CC-BY-4.0): per movie, `people.csv` (name_id/person/character), `scenes.csv`
|
||
(scene/start/end ms), `people_in_scenes.csv`. Films matched to the library by an
|
||
**authoritative Jellyfin ID join** (query `/Items?IncludeItemTypes=Movie&Fields=
|
||
ProviderIds,Path`, join Imdb/Tmdb against X-Ray metadata) — NOT fuzzy title matching,
|
||
which collides badly (TV episodes vs same-named films). 9 genuine films with source
|
||
video on disk: Benny & Joon, Café Society, Downton Abbey: A New Era, Lord of War,
|
||
Lovelace, The Many Saints of Newark, Scarface, Sound of Metal, Valerian.
|
||
|
||
## The scoring metric (evolved through review)
|
||
|
||
Comparison unit is the **X-Ray scene**, not sampled timepoints. For each scene
|
||
`[start,end]`: predicted set = **union** of actors detected anywhere in the span;
|
||
GT set = actors X-Ray lists for that scene. Per scene TP/FP/FN, then:
|
||
|
||
- **Precision: STRICT.** Any predicted actor not in the scene's X-Ray set is an FP,
|
||
*including out-of-cast confusions* (no gallery∩cast masking). An earlier
|
||
timepoint-sampled, cast-masked metric HID ~570 such FPs across 9 films and let the
|
||
optimizer drive `prob_threshold` to the 0.50 floor — a metric artifact. Counting
|
||
them is essential.
|
||
- **Recall: FAIR.** FN counts only X-Ray cast members **who are in the gallery**. 67%
|
||
of X-Ray cast (261/392) have no gallery reference embedding and can never be
|
||
recognised — counting them as misses penalises coverage, not the threshold. Both
|
||
`recall` (fair) and `recall_strict` (all) are reported.
|
||
- **Aggregation:** per-scene F1 → **duration-weighted average within a movie** (long
|
||
scenes count more) → **equal-weight mean across movies** (macro; each film counts
|
||
the same regardless of length). This is the DE objective.
|
||
|
||
Implemented in `scripts/optimizer/scene_score.py` — since **removed** along
|
||
with this metric; its per-second successor is
|
||
[`scripts/optimizer/second_score.py`](https://REPOLINK/scripts/optimizer/second_score.py)
|
||
(see the [bake-off round](model-bakeoff.md)).
|
||
|
||
## The gallery coverage gap
|
||
|
||
Diagnosing low recall: only **131 of 392** X-Ray cast were in the gallery (33%). Every
|
||
in-gallery actor HAD embeddings (gallery well-formed) — the gap was pure coverage.
|
||
[`scripts/optimizer/fetch_missing_actors.py`](https://REPOLINK/scripts/optimizer/fetch_missing_actors.py)
|
||
recovers missing actors:
|
||
`nm-id → TMDB /find external_ids → /person/{id}/images → download → embed (sae_embed)`,
|
||
with a `--wikidata` fallback (P345→P18 Commons photo).
|
||
|
||
- **TMDB recovered 143/261** (55%). 0 face-detection failures; the rest had no TMDB
|
||
person (60) or no profile photo (58). Coverage 33% → **70%**.
|
||
- **Wikidata fallback: 0/118** of the TMDB failures — only 4 even had a Commons photo,
|
||
none yielded a detectable face. → **TheTVDB not worth pursuing**: these remaining
|
||
actors are obscure enough that no image source covers them, AND (see below) most are
|
||
off-camera anyway.
|
||
|
||
**Coverage vs detectability.** Adding references lifted recall (58→68% at fixed config)
|
||
but modestly. Per-film drill-down (Lord of War: 12 actors recovered, only 1 had a
|
||
detectable on-camera face) showed most missing cast are a **detectability gap** — X-Ray
|
||
credits them as cast-in-scene (incl. off-camera/background), but their face never
|
||
appears clearly for the pipeline to detect. This is a fundamental ceiling of a
|
||
face-recognition pipeline vs X-Ray's presence semantics, not a fixable gap.
|
||
|
||
## Optimizer
|
||
|
||
[`scripts/optimizer/optimize.py`](https://REPOLINK/scripts/optimizer/optimize.py)
|
||
— scipy `differential_evolution` over the knob space,
|
||
each candidate = full replay of all films through the **real** C++ nodes (see the
|
||
KPN replay architecture below) scored by the metric above. Global objective (one
|
||
config for all films, not per-film).
|
||
|
||
**Convergence stability (augmented gallery, 233 evals):**
|
||
|
||
| knob | top-20 range | verdict |
|
||
| ---- | ------------ | ------- |
|
||
| `prob_threshold` | 0.69–0.83 (σ 0.05) | TIGHT — trust 0.76 |
|
||
| `extinction_sec` | 1.0–2.2 (σ 0.33) | TIGHT — trust 1.5 |
|
||
| `anneal_sec` | 3.1–26.3 (σ 6.4) | LOOSE — insensitive, not hard-coded |
|
||
|
||
F1 varied only 0.3pp across the top-20 → objective is flat near the optimum, so only
|
||
the tightly-converged knobs were adopted as defaults.
|
||
|
||
## Replay architecture (how the sweep is cheap)
|
||
|
||
The optimizer never re-decodes video. `scene_analyze --dump-embeddings out.h5` runs the
|
||
expensive half once (decode→detect→align→embed) and dumps per-frame face embeddings
|
||
+ metadata to HDF5 ([`scripts/optimizer/SCHEMA.md`](https://REPOLINK/scripts/optimizer/SCHEMA.md)).
|
||
[`scripts/optimizer/replay.py`](https://REPOLINK/scripts/optimizer/replay.py) then
|
||
replays that dump through the **real** C++ `face_tracker → identity_matcher →
|
||
scene_tracker` assembled in a Python KPN network (`sae_kpn` nanobind module), varying
|
||
Config knobs freely — no GPU embedding, no decode. Verified BYTE-EXACT against
|
||
`scene_analyze`'s own output. The dumps are gallery-independent, so testing the
|
||
augmented gallery needed no re-dump. `detector_conf` is replayable UPWARD only (the
|
||
dump floor is 0.5).
|
||
|
||
## Reproduce
|
||
|
||
```bash
|
||
# 1. dump (once per film, needs video)
|
||
scene_analyze --movie <f> --gallery gallery.json --dump-embeddings dump.h5 --fps 1
|
||
# 2. build films manifest by Jellyfin ID join (see scripts/optimizer notes)
|
||
# 3. optimize
|
||
python scripts/optimizer/optimize.py --manifest films.json --gallery gallery.json \
|
||
--params prob_threshold:0.5:0.999 anneal_sec:1:30 extinction_sec:1:15 \
|
||
--popsize 8 --maxiter 20 --trajectory traj.jsonl --out opt.json
|
||
# 4. score a fixed config / validate on a held-out set
|
||
# (historical: score_config.py and scene_score.py were removed with the
|
||
# scene-union metric — use scripts/optimizer/second_score.py, per-second)
|
||
python scripts/optimizer/second_score.py --help
|
||
```
|
||
|
||
Superseded by the [model bake-off + re-tune](model-bakeoff.md), which
|
||
replaced this round's scene-union metric with per-second scoring.
|