dtourolle/scene-actor-extraction
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: rep4 bake-off write-up, MkDocs site, artifact-registry-backed experiments

Browse Source 
docs/rep4-optimizer-results.md is the main deliverable: the model bake-off +
threshold re-tune experiment log, including the ROCm teardown deadlock root
cause and fix, DE concurrency tuning, the 16-combo results table, held-out
validation against 5 films never seen by the optimizer (macro F1 67.4% vs.
75.3% training — a real generalization gap), the frozen-bbox "ghost track"
failure mode found via annotated frame evidence, calibration curves per model,
and an isolated-effects breakdown of gallery scope vs. pose expansion.

MkDocs site (mkdocs.yml, docs/index.md) renders docs/*.md; scripts/docs/
pulls referenced images from the artifact registry and generates the
calibration chart at build time (see the tooling commit) rather than
committing images to the repo.

experiments/ now keeps only scripts + README + SESSION_STATE.md in git — every
data artifact (galleries, dumps, X-Ray corpus, montage frames, trajectories,
manifests, results) moved to the Gitea package registry. film-lut.template.json
is the committed placeholder for the gitignored file-lut.json (real local
movie paths, never shared — some source filenames carry scene-release tags).

Adds models/transnetv2.onnx (via Git LFS, matching the other ONNX models) for
the new scene-detection path.
This commit is contained in:
dtourolle
2026-07-19 19:12:22 +02:00
parent 76df2f66aa
commit d340da755a
15 changed files with 1223 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitignore
+20 -1
View File
@@ -30,9 +30,12 @@ trt_cache/
# ORT pre-optimized model cache (generated on first run, provider-specific) # ORT pre-optimized model cache (generated on first run, provider-specific)
ort_cache/ ort_cache/
# Gallery JSON files (generated) # Gallery files (generated — HDF5 only, see src/gallery/gallery_store.cpp).
# Legacy JSON galleries from before that switch are also excluded.
gallery.json gallery.json
gallery_*.json gallery_*.json
gallery.h5
gallery_*.h5
# Calibration cache (generated alongside a gallery, per-embedder) # Calibration cache (generated alongside a gallery, per-embedder)
*.calib_cache.csv *.calib_cache.csv
@@ -61,6 +64,22 @@ eval/probe/
# Local reference repos kept for inspiration (each has its own .git) # Local reference repos kept for inspiration (each has its own .git)
inspiration/ inspiration/
# experiments/ has its own nested .gitignore for HDF5 galleries/dumps/X-Ray
# corpus (all pushed/pulled via scripts/artifacts/{push,pull}_artifacts.sh to
# the Gitea generic package registry instead of committed).
# Exception to the blanket *.json rule above: the committed placeholder for
# experiments/file-lut.json (see experiments/.gitignore).
!experiments/file-lut.template.json
# Site build output (mkdocs build). Rendered site is deployed to a
# gitea-pages branch, never committed to a working branch.
site/
docs_site/
# Images staged into docs/ from the artifact registry at build time
# (scripts/docs/build_site.sh) — not committed, pulled fresh on each build.
docs/assets/
# Python # Python
__pycache__/ __pycache__/
*.pyc *.pyc
README.md
+4 -4
View File
@@ -46,10 +46,10 @@ Optional flags:
## Models ## Models
The ONNX model weights live in `models/` (tracked via Git LFS): The ONNX model weights live in `models/` (tracked via Git LFS):
- `arcface_w600k_r50.onnx` — primary ArcFace embedder - `LVFace-B_Glint360K.onnx` — LVFace embedder (ViT backbone, ICCV 2025), the default
- `arcface_w600k_mbf.onnx`, `arcface_r18.onnx` — lighter alternatives (best F1 in the rep4 model bake-off, see `docs/rep4-optimizer-results.md`)
- `LVFace-B_Glint360K.onnx` — LVFace embedder (ViT backbone, ICCV 2025), drop-in - `arcface_w600k_r50.onnx` — ArcFace embedder, previous default
ArcFace-compatible: same `[N,3,112,112]` input and 512-d output (see below) - `arcface_w600k_mbf.onnx`, `arcface_r18.onnx` — lighter ArcFace alternatives
- `face_detection_yunet_2023mar.onnx` — YuNet face detector - `face_detection_yunet_2023mar.onnx` — YuNet face detector
- `scrfd_500m_bnkps.onnx` — SCRFD face detector - `scrfd_500m_bnkps.onnx` — SCRFD face detector
docs/index.md
+33
View File
@@ -0,0 +1,33 @@
# scene-actor-extraction
A face-recognition pipeline that finds when each actor appears on screen in a
film or TV episode — built on [KPN++](https://gitea.tourolle.paris/dtourolle/KPN)
(a C++20 Kahn Process Network library) for the detect → track → match → scene
pipeline, with a Jellyfin-integrated gallery and an X-Ray-validated optimizer.
## Start here
- **[Rep4 model bake-off + threshold re-tune](rep4-optimizer-results.md)** — the
current experiment log: model comparison, DE threshold tuning, held-out
validation, and the visual failure-mode evidence (frozen-bbox ghost tracks).
This is where the shipped `src/config.hpp` defaults come from.
- **[Optimizer experiments (prior round)](optimizer-experiments.md)** — the
earlier scene-union-metric tuning pass, superseded by the per-second metric
used in rep4 but kept for the ground-truth/architecture background.
- **[Service conversion (proposal)](service-conversion.md)** — design sketch
for an idle-GPU Docker worker, not yet built.
## Reproducing the benchmarks
Gallery `.h5` files, embedding dumps, the X-Ray corpus, montage frame images,
and DE trajectories are not committed to this repository — they're pushed to
the Gitea package registry and pulled on demand:
```bash
scripts/artifacts/pull_artifacts.sh galleries
scripts/artifacts/pull_artifacts.sh experiment-data
scripts/artifacts/pull_artifacts.sh montage-frames <film-slug>
```
See `scripts/artifacts/push_artifacts.sh` for the upload side (requires a
`GITEA_TOKEN` with package write scope).
docs/optimizer-experiments.md
+117
View File
@@ -0,0 +1,117 @@
# Threshold optimization against Amazon X-Ray — experiment log
Record of the July 2026 work that tuned the pipeline's recognition/tracking defaults
against ground-truth per-scene actor presence, and the tooling built to do it.
## TL;DR — what changed
| knob | old default | new default | why |
| ---- | ----------- | ----------- | --- |
| `prob_threshold` | 0.99 | **0.76** | 0.99 was far too strict — halved recall for a fraction of a precision point. DE optimum, tightly converged. |
| `extinction_sec` | 5.0 | **1.5** | Long extinction smears presence into later scenes → FPs. DE converged tightly low. |
| `anneal_sec` | 10.0 | 10.0 (unchanged) | DE found it **insensitive** (F1 flat ±0.3pp across 3–26s) — kept the round default. |
| `detector_conf` | 0.5 | 0.5 (unchanged) | Sweep showed raising it only trades recall for precision at a net F1 loss — near-threshold detections are real faces, not phantoms. |
Net effect on the 9-film benchmark (strict per-scene, augmented gallery):
recall **58% → ~72%**, F1 **70% → ~76%**, precision ~85%, at no meaningful precision cost.
## Ground truth
Public scene-level **Amazon X-Ray** dataset (Zenodo DOI 10.5281/zenodo.17659734,
CC-BY-4.0): per movie, `people.csv` (name_id/person/character), `scenes.csv`
(scene/start/end ms), `people_in_scenes.csv`. Films matched to the library by an
**authoritative Jellyfin ID join** (query `/Items?IncludeItemTypes=Movie&Fields=
ProviderIds,Path`, join Imdb/Tmdb against X-Ray metadata) — NOT fuzzy title matching,
which collides badly (TV episodes vs same-named films). 9 genuine films with source
video on disk: Benny & Joon, Café Society, Downton Abbey: A New Era, Lord of War,
Lovelace, The Many Saints of Newark, Scarface, Sound of Metal, Valerian.
## The scoring metric (evolved through review)
Comparison unit is the **X-Ray scene**, not sampled timepoints. For each scene
`[start,end]`: predicted set = **union** of actors detected anywhere in the span;
GT set = actors X-Ray lists for that scene. Per scene TP/FP/FN, then:
- **Precision: STRICT.** Any predicted actor not in the scene's X-Ray set is an FP,
*including out-of-cast confusions* (no gallery∩cast masking). An earlier
timepoint-sampled, cast-masked metric HID ~570 such FPs across 9 films and let the
optimizer drive `prob_threshold` to the 0.50 floor — a metric artifact. Counting
them is essential.
- **Recall: FAIR.** FN counts only X-Ray cast members **who are in the gallery**. 67%
of X-Ray cast (261/392) have no gallery reference embedding and can never be
recognised — counting them as misses penalises coverage, not the threshold. Both
`recall` (fair) and `recall_strict` (all) are reported.
- **Aggregation:** per-scene F1 → **duration-weighted average within a movie** (long
scenes count more) → **equal-weight mean across movies** (macro; each film counts
the same regardless of length). This is the DE objective.
Implemented in `scripts/optimizer/scene_score.py`.
## The gallery coverage gap
Diagnosing low recall: only **131 of 392** X-Ray cast were in the gallery (33%). Every
in-gallery actor HAD embeddings (gallery well-formed) — the gap was pure coverage.
`scripts/optimizer/fetch_missing_actors.py` recovers missing actors:
`nm-id → TMDB /find external_ids → /person/{id}/images → download → embed (sae_embed)`,
with a `--wikidata` fallback (P345→P18 Commons photo).
- **TMDB recovered 143/261** (55%). 0 face-detection failures; the rest had no TMDB
person (60) or no profile photo (58). Coverage 33% → **70%**.
- **Wikidata fallback: 0/118** of the TMDB failures — only 4 even had a Commons photo,
none yielded a detectable face. → **TheTVDB not worth pursuing**: these remaining
actors are obscure enough that no image source covers them, AND (see below) most are
off-camera anyway.
**Coverage vs detectability.** Adding references lifted recall (58→68% at fixed config)
but modestly. Per-film drill-down (Lord of War: 12 actors recovered, only 1 had a
detectable on-camera face) showed most missing cast are a **detectability gap** — X-Ray
credits them as cast-in-scene (incl. off-camera/background), but their face never
appears clearly for the pipeline to detect. This is a fundamental ceiling of a
face-recognition pipeline vs X-Ray's presence semantics, not a fixable gap.
## Optimizer
`scripts/optimizer/optimize.py` — scipy `differential_evolution` over the knob space,
each candidate = full replay of all films through the **real** C++ nodes (see the
KPN replay architecture below) scored by the metric above. Global objective (one
config for all films, not per-film).
**Convergence stability (augmented gallery, 233 evals):**
| knob | top-20 range | verdict |
| ---- | ------------ | ------- |
| `prob_threshold` | 0.69–0.83 (σ 0.05) | TIGHT — trust 0.76 |
| `extinction_sec` | 1.0–2.2 (σ 0.33) | TIGHT — trust 1.5 |
| `anneal_sec` | 3.1–26.3 (σ 6.4) | LOOSE — insensitive, not hard-coded |
F1 varied only 0.3pp across the top-20 → objective is flat near the optimum, so only
the tightly-converged knobs were adopted as defaults.
## Replay architecture (how the sweep is cheap)
The optimizer never re-decodes video. `scene_analyze --dump-embeddings out.h5` runs the
expensive half once (decode→detect→align→embed) and dumps per-frame face embeddings
+ metadata to HDF5 (`scripts/optimizer/SCHEMA.md`). `scripts/optimizer/replay.py` then
replays that dump through the **real** C++ `face_tracker → identity_matcher →
scene_tracker` assembled in a Python KPN network (`sae_kpn` nanobind module), varying
Config knobs freely — no GPU embedding, no decode. Verified BYTE-EXACT against
`scene_analyze`'s own output. The dumps are gallery-independent, so testing the
augmented gallery needed no re-dump. `detector_conf` is replayable UPWARD only (the
dump floor is 0.5).
## Reproduce
```bash
# 1. dump (once per film, needs video)
scene_analyze --movie <f> --gallery gallery.json --dump-embeddings dump.h5 --fps 1
# 2. build films manifest by Jellyfin ID join (see scripts/optimizer notes)
# 3. optimize
python scripts/optimizer/optimize.py --manifest films.json --gallery gallery.json \
--params prob_threshold:0.5:0.999 anneal_sec:1:30 extinction_sec:1:15 \
--popsize 8 --maxiter 20 --trajectory traj.jsonl --out opt.json
# 4. score a fixed config / validate on a held-out set
python scripts/optimizer/score_config.py --manifest heldout.json --gallery gallery.json \
--config '{"prob_threshold":0.76,"extinction_sec":1.5,"anneal_sec":10}'
```
See also memory: kpn-python-replay-optimizer, gallery-coverage-gap, xray-validation-*.
docs/rep4-optimizer-results.md
+408
View File
@@ -0,0 +1,408 @@
# Rep4 model bake-off + threshold re-tune — experiment log (2026-07-18/19)
Follow-on to `docs/optimizer-experiments.md`, which used an older, since-superseded
scene-union metric. This round uses the **per-second** metric
(`scripts/optimizer/second_score.py`) and answers three questions in one 16-run
matrix: which embedding model is best, does cast-restriction cut misIDs, and does
per-film gallery expansion help.
## Why this experiment, and what it actually delivered
Four goals going in, and an honest read on each after held-out validation (see
below):
1. **Find the best default parameters to ship.** Partially delivered. The DE optimum
generalizes *unevenly* — strong on 3 of 5 held-out films, badly broken on 2 (one
with a 974-count misID blowup). The tuned values are shipped anyway (see
Caveats) because they still beat the old defaults on average, but this is not a
settled, film-agnostic optimum.
2. **Find the best default model.** Delivered with more confidence. LVFace beat
r50/r18/mbf across all 4 training combos, and nothing in held-out validation
contradicts the model choice specifically — the held-out failures trace to
`extinction_sec`/threshold interactions and gallery coverage, not the embedder.
3. **Provide insight into how the application works.** The strongest, most durable
output. Found and fixed a real teardown deadlock bug (100% reproducible, not the
assumed rare GPU flake), established a real concurrency ceiling (8 parallel
replays, not more), and found a real parameter interaction (a strict
`prob_threshold` "earns" a longer extinction window before it starts hurting).
4. **Demonstrate limitations.** Delivered, and reinforced hard by held-out
validation — see the "Held-out validation" section below for concrete examples,
including a screenshot of the matcher naming 15 actors, none correctly, on a
completely blank title card.
## TL;DR — what changed in `src/config.hpp`
| knob | old default | new default | why |
| ---- | ----------- | ----------- | --- |
| `arcface_model` | `arcface_w600k_r50.onnx` | **`LVFace-B_Glint360K.onnx`** | Best F1 in the full-gallery bake-off (75.3% vs r50's 68.5%). LVFace was worth its size. |
| `prob_threshold` | 0.76 | **0.754** | Re-tuned for LVFace + per-second metric. |
| `extinction_sec` | 1.5 | **57.4** | Reverses the earlier "short is better" finding — see below. |
| `anneal_sec` | 10.0 | **35.5** | Same reversal; previously thought insensitive. |
| `expand_gallery` | false | **true** | Helped recall on the full (unrestricted) gallery for the winning model — opposite of the earlier assumption. |
These are the **`LVFace-B_Glint360K_full_exp`** winning values — the best result that
uses only features already live in the running app (full gallery, no cast
restriction; see below for why restricted mode isn't applied even though it scored
higher).
## Why re-run at all
`docs/optimizer-experiments.md`'s scene-union metric hid out-of-cast false positives
behind a gallery∩cast recall mask, and the earlier 9-film benchmark was scene-level
(union over a whole X-Ray scene), not a fair per-timepoint comparison. SESSION_STATE
flagged the old scene-metric bake-off numbers (R50≈LVFace≈MBF ~85%) as superseded.
This round uses `second_score.py`: uniform per-second sampling, GT = X-Ray scene's
cast at time *t*, pred = actors whose presence window covers *t*, FPI weighted 10×
when the named actor isn't in the film's cast at all (true misID) vs. an in-cast
timing slip. FN only counts gallery-known cast (fair recall — 67% of X-Ray cast have
no reference embedding, see `gallery-coverage-gap` memory).
## The deadlock that was blocking all of this
Every replay in this line of work goes through `scripts/optimizer/replay.py`, which
runs the real C++ tracker/matcher/scene_tracker nodes inside a Python-assembled KPN
network. Before this session, every subprocess replay **timed out at 45s, 100% of
the time** — not the documented ~20-30% ROCm rocBLAS-GEMM driver flake, but a plain
logic bug: `replay.py`'s CLI called `replay(net, ..., stop=False)` to *skip*
`net.stop()` (trying to dodge the GEMM deadlock), planning to `os._exit(0)`
immediately after. But:
- `PyNode::stop()` (`external/KPN/include/kpn/python/bindings.hpp`) is the *only*
code that sets `stop_flag_ = true` before joining the node's worker thread.
- The source node's `run_loop()` has `while (!stop_flag_)` as its only exit
condition (it has no input channels, so it never sees a channel-closed signal
either).
- Skipping `stop()` meant `stop_flag_` never became true. When `replay()` returned,
its local `net` went out of scope immediately, running `~PyNetwork` → `~PyNode` →
`thread_.join()` **synchronously inside `replay()`'s own call frame** — before
`main()` ever got control back to run `os._exit(0)`.
Root-caused via `gdb -p <pid> -batch -ex "thread apply all bt"` on a hung process:
the main thread was stuck in `~PyNode`'s `jthread::join()`; the worker thread was in
an ordinary `time.sleep()` inside the Python source callback, waiting for a stop
signal that was never sent. The two HSA `kfd_wait_on_events` threads visible in the
same trace are normal ROCm runtime housekeeping, not evidence of a wedged GPU kernel.
**Fix:** `replay.py` now calls `replay(..., stop=True)` (the removed `stop=False` +
`os._exit` workaround was actively harmful). Verified 3/3 clean runs at ~8s each
(down from a guaranteed 45s timeout), and a full DE sweep producing real, sensible
F1/precision/recall instead of flat 0.0%.
## Concurrency tuning
With the deadlock fixed, `optimize.py` was extended with DE-level parallelism —
`differential_evolution(..., workers=ThreadPoolExecutor.map)` — so multiple
population candidates evaluate concurrently, each spawning its own per-film replay
subprocesses (`REPLAY_WORKERS`). Total concurrent GPU replay processes ≈
`DE_WORKERS × REPLAY_WORKERS`.
| concurrent replays | result |
| --- | --- |
| 3 (`REPLAY_WORKERS=3`, no DE parallelism) | baseline, GPU underutilised |
| 6 (`DE_WORKERS=2 × REPLAY_WORKERS=3`) | clean, real scores, ~1 isolated timeout per run |
| 8 (`DE_WORKERS=2 × REPLAY_WORKERS=4`, 4-film manifest) | clean, real scores |
| 9 (`DE_WORKERS=3 × REPLAY_WORKERS=3`) | **broken** — every replay blew past the 45s timeout, all scores silently degraded to 0.0% |
9 concurrent replays looks like valid output (well-formed JSON, a real number) while
actually being garbage — a dangerous failure mode, not a crash. **8 concurrent is the
practical ceiling** on this GPU (gfx1100) for this workload. The matrix ran at
`REPLAY_WORKERS=4 DE_WORKERS=2`.
## Training films (rep4) and validation set
9 films total have dumped embeddings across all 4 models. 4 were used for
optimization (rep4), leaving 5 held out for validation:
- **Lord of War** (64-cast, "clean")
- **Scarface** (67-cast, "ensemble/lookalike")
- **Sound of Metal** (14-cast, "high gallery-coverage")
- **Café Society** (62-cast, added this round — similar ensemble size to Scarface but
different genre/lighting; picked to add diversity, not genre-overlap, over
Downton Abbey or The Many Saints of Newark)
Held out: Benny & Joon, Downton Abbey: A New Era, Lovelace, The Many Saints of
Newark, Valerian and the City of a Thousand Planets. The rep4 numbers below are
training-set fit — see "Held-out validation" further down for the real
generalization test.
## Search space and DE settings
`popsize=10, maxiter=15` (3 params → ≤480 evals/combo ceiling; DE's `tol` convergence
usually stops earlier). `anneal_sec`/`extinction_sec` bounds were **widened from
1–30/1–15 to 1–60/1–60 mid-run** (see below) — the 4 `arcface_w600k_r50` combos
finished before the widening and still use the old, narrower bounds, so they are
**not directly comparable** to the other 12 on those two params. Re-running r50 with
the wider bounds was deferred (diminishing-returns judgment call, not yet done).
## Results — all 16 combos (4 models × {full, restricted} × {expand, noexp})
Ranked by F1. `misid` = FPI_misid, count of true wrong-actor identifications (an
actor named who isn't in the film's cast at all) — distinct from `FPI`, which
includes in-cast timing slips.
| combo | F1 | P | R | TPI | FPI | misid | FN |
|---|---|---|---|---|---|---|---|
| LVFace-B_Glint360K_restricted_exp | **78.3%** | 91.0% | 68.9% | 42830 | 3782 | 60 | 19492 |
| LVFace-B_Glint360K_restricted_noexp | 76.7% | 91.5% | 66.2% | 41149 | 3400 | 59 | 21173 |
| arcface_w600k_mbf_restricted_exp | 76.5% | 90.7% | 66.3% | 41270 | 4431 | **0** | 21052 |
| arcface_r18_restricted_exp | 75.5% | 87.6% | 66.5% | 41399 | 5666 | 60 | 20923 |
| **LVFace-B_Glint360K_full_exp** | **75.3%** | 89.7% | 65.4% | 47757 | 3407 | 232 | 26966 |
| arcface_w600k_mbf_restricted_noexp | 75.0% | 91.1% | 63.9% | 39752 | 3465 | 60 | 22570 |
| arcface_w600k_mbf_full_noexp | 74.2% | 87.4% | 64.4% | 12645 | 1312 | 57 | 6985 |
| arcface_r18_restricted_noexp | 73.5% | 91.3% | 61.7% | 38299 | 3220 | 60 | 24023 |
| LVFace-B_Glint360K_full_noexp | 72.4% | 94.2% | 58.9% | 27077 | 1725 | **0** | 19506 |
| arcface_w600k_mbf_full_exp | 72.0% | 87.7% | 61.4% | 39875 | 3729 | 240 | 26338 |
| arcface_w600k_r50_full_noexp † | 71.6% | 96.7% | 56.9% | 22471 | 361 | 45 | 17012 |
| arcface_w600k_r50_restricted_exp † | 71.1% | 96.5% | 56.4% | 34954 | 1119 | 15 | 27368 |
| arcface_w600k_r50_restricted_noexp † | 69.2% | 97.9% | 53.6% | 21146 | 327 | 15 | 18337 |
| arcface_r18_full_exp | 69.1% | 87.6% | 57.7% | 37342 | 3119 | 242 | 28871 |
| arcface_w600k_r50_full_exp † | 68.5% | 94.0% | 54.1% | 34982 | 903 | 150 | 31231 |
| arcface_r18_full_noexp | 66.6% | 91.3% | 53.1% | 34314 | 2362 | 107 | 31899 |
† old, narrower anneal/extinction bounds (see above) — not directly comparable to
the other 12 on those two params.
## Calibration curves — discriminative power, independent of the threshold
Each model's gallery carries a fitted Platt sigmoid `P(match | sim) = σ(a·sim + b)`
(embedded directly in the gallery HDF5, see `src/gallery/gallery_calibration.hpp`).
Plotting all four side by side shows discriminative power directly, independent of
whatever `prob_threshold` a particular run happened to use:
![Calibrated P(match|similarity) for all four models](assets/images/calibration_curves.png)
LVFace-B has both the steepest curve (`a=17.7`, vs. 15.3–16.2 for the ArcFace
variants) and the lowest P=0.5 decision boundary (similarity 0.23 vs. 0.27–0.31) —
it separates same-actor from different-actor pairs more confidently at a lower
similarity, consistent with it winning the full-gallery F1 comparison below.
Generated by `scripts/docs/calibration_chart.py` (requires each gallery to have
been calibrated at least once — run any replay against it first).
## Two effects in isolation: gallery scope, and pose expansion
The matrix crosses two independent variables — averaging across all 4 models
isolates each one from model choice:
**Gallery scope (whole 2418-actor gallery vs. restricted to the film's credited
cast)** — averaged over both expansion settings and all 4 models:
| scope | F1 | P | R | total misID (16 evals→8 each) |
|---|---|---|---|---|
| full | 71.2% | 91.1% | 59.0% | 1073 |
| **restricted** | **74.5%** | 92.2% | **62.9%** | **329** |
Restriction wins outright on every axis — not a precision/recall trade, a clean
win: **+3.3pp F1, +3.9pp recall, and less than a third the total misIDs.** Fewer
candidates in the matcher's search space means fewer opportunities for a
look-alike false match, and (per the recall gain) doesn't cost real detections.
This is the single cleanest signal in the whole matrix — stronger than the model
choice itself — which is exactly why cast-restriction becoming a real runtime
feature (not just an optimizer trick) is the top item in Caveats below.
**Pose expansion (promoting a confidently-identified track's novel-pose views into
a per-film gallery annex — `track_gallery.hpp`)** is smaller and interacts with
scope rather than acting independently:
| scope | expansion | F1 | R | misID |
|---|---|---|---|---|
| full | off | 71.2% | 58.3% | 209 |
| full | **on** | 71.2% | 59.7% | **864** |
| restricted | off | 73.6% | 61.3% | 194 |
| restricted | **on** | **75.4%** | **64.5%** | 135 |
In **restricted** mode, expansion is a clean win (+1.8pp F1, +3.2pp recall, misID
actually *drops*) — the annex only ever competes against the film's own ~15-actor
cast, so a "confidently identified, new pose" view is unlikely to be mistaken for
someone else. In **full** mode, expansion buys essentially nothing on F1 (71.2% →
71.2%, recall +1.4pp) while **quadrupling misIDs** (209 → 864): a novel-pose view
promoted into the annex now competes against the whole 2418-actor gallery, so a
"confident" identity is confident against the wrong universe of candidates — the
expansion mechanism is "learning" a pose correctly, but the enlarged evidence pool
makes it easier for that learned pose to look like a plausible match for a
different actor. **Practical takeaway: gallery expansion should be paired with
cast restriction, not used on the full gallery** — the version currently shipped
as default (`full_exp`, see TL;DR) sits in the worse of these four cells for this
specific knob, even though it's the best available combo without cast-restriction
support in the app yet (see Caveats).
## What the data says
- **LVFace was worth its size.** It wins full-gallery mode outright (75.3% vs r50's
68.5%, r18's 69.1%, mbf's 72.0%) with the highest recall of any full-mode combo —
the earlier scene-union-metric conclusion ("not worth it") doesn't survive the
better metric.
- **Cast-restriction is a consistent, broad win.** Every model's best combo is
`restricted`. It isn't just precision-safe: `arcface_w600k_mbf_restricted_exp` and
`LVFace-B_Glint360K_full_noexp` both hit **misid=0** — zero true wrong-actor
identifications. But restriction is an **offline optimizer technique, not a live
app feature** — it pre-filters each film's gallery to its Jellyfin-credited cast
before the matcher ever runs; there's no runtime "restrict to this film's cast"
switch in the app today. Implementing it for real is future work, tracked
separately from this defaults update.
- **Gallery expansion (`expand_gallery`) is mode-dependent.** It helps on
`restricted` galleries (smaller, so novel-pose promotion adds real signal) and on
LVFace's full gallery, but **hurts** r50 and mbf in full mode (compare
`arcface_w600k_r50_full_exp` 68.5% vs `full_noexp` 71.6%). Don't assume it's a free
win — model- and mode-dependent.
- **arcface_r18 (smallest/cheapest) is last across all 4 modes** — model capacity
matters here, this isn't just parameter-count padding.
- **`anneal_sec`/`extinction_sec` kept pinning at the search ceiling.** With the
original 1–30/1–15 bounds, 3 of 4 r50 combos landed at ~93-98% of the upper bound.
Widened to 1–60/1–60 mid-run (after the r50 combos had already finished) — every
subsequent combo's best config landed at ~90%+ of the *new* ceiling too (e.g. the
LVFace winner: `ann=59.2, ext=59.2`, both ~99% of 60). The likely mechanism: a
strict `prob_threshold` "earns" a long extinction/anneal window — once false
matches are rare, a long window just bridges real presence gaps (occlusion, turned
face) instead of smearing false positives into later scenes, which is what made
short windows look better under the old, laxer thresholds. **Open question, not
resolved**: does this keep climbing past 60s, or does it actually plateau there?
Decided not to chase further this round (diminishing-returns judgment call) — flag
for a future sweep if it matters.
## Held-out validation — the number that actually matters
The 16-combo matrix above is training-set fit. This is the real test: the shipped
config (`LVFace-B_Glint360K_full_exp` — `prob_threshold=0.754, anneal_sec=35.5,
extinction_sec=57.4, expand_gallery=true`) replayed against the **5 films never seen
by the optimizer** (Benny & Joon, Downton Abbey: A New Era, Lovelace, The Many
Saints of Newark, Valerian and the City of a Thousand Planets), scored the same way.
| film | F1 | P | R | agree | TPI | FPI | misid | FN |
|---|---|---|---|---|---|---|---|---|
| Benny & Joon | 83.0% | 89.1% | 77.7% | 72.4% | 15125 | 1846 | 0 | 4337 |
| Lovelace | 77.5% | 90.3% | 67.9% | 72.1% | 14990 | 1085 | 58 | 7085 |
| Valerian and the City of a Thousand Planets | 74.1% | 97.1% | 60.0% | 58.8% | 18663 | 548 | 0 | 12467 |
| Downton Abbey: A New Era | 56.2% | 97.8% | 39.4% | 40.6% | 52027 | 1173 | 0 | 80084 |
| **The Many Saints of Newark** | **46.3%** | **54.7%** | 40.1% | 37.0% | 15922 | 4394 | **974** | 23791 |
| **macro average (5 films)** | **67.4%** | 85.8% | 57.0% | 56.2% | 116727 | 9046 | 1032 | 127764 |
**67.4% held out vs. 75.3% on training** — an ~8pp drop, and a much more informative
number than the training-set F1 alone: a **37pp spread between best and worst film**
(83.0% vs 46.3%). The config does not generalize uniformly.
Two films are outright failure cases, and rendering bounding boxes + names on the
extracted frames (`replay.py --raw-out` + `dump_error_frames.py --raw`, see
Reproduce) turned what looked like a same-scene misidentification into something
more precise and more damning:
- **The Many Saints of Newark** (mob-family drama, picked as a training-adjacent
genre test) has **974 true misIDs** — far more than any training combo saw at any
setting. The annotated frame below shows the same mechanism as Downton Abbey,
at smaller scale: **"Jon Bernthal 100%", "Joey Diaz 100%", and "Billy Magnussen
100%" are all frozen boxes over empty background — a blurred wall, hanging plates —
with no face in them at all.** Only one real face in frame has a box, and it
carries a *second*, colliding label ("Leslie Odom Jr." and "Michael Gandolfini"
both at high confidence on the same box) — likely two tracks whose frozen bboxes
happen to overlap.
![Frozen ghost boxes over background, The Many Saints of Newark](assets/images/many_saints_ghost_fpi.jpg)
*`experiments/results/holdout/frames/many_saints/fpi/fpi_t03543.jpg`*
- **Downton Abbey: A New Era** (large ensemble, 36-cast) has high precision (97.8%)
but recall collapses to 39.4% (FN=80084, by far the largest of the 5). The frame
below is the starkest evidence in this whole experiment: **the matcher named 15
actors — all of them wrong — over a completely blank closing title card with no
faces on screen at all.**
![15 ghost boxes over a blank title card, Downton Abbey: A New Era](assets/images/downton_abbey_ghost_fpi.jpg)
*`experiments/results/holdout/frames/downton_abbey/fpi/fpi_t07242.jpg`*
Both are the same mechanism, verified directly against the HDF5 dump and the raw
per-frame stream (not just inferred from the screenshot): at the Downton Abbey
title card (t=7242), the dump's own `face_count` is **0 from t≈7240 onward** — no
detector output at all, confirmed independently of the pipeline. Yet all 15 "wrong"
actors are still marked visible, each with the *exact same bbox, unchanged to the
pixel*, repeated every single frame back to t=7222 (verified for Hugh Bonneville:
`(1743.2, 0.0, 171.3, 317.8)` at every sampled second from 7222 through 7279+). That
is `SceneTrackerFunc`'s `active_[actor_idx].last_bbox` (`scene_tracker_node.hpp`)
being re-emitted unchanged — **this is the extinction state machine working exactly
as coded**, not a bug in the logic. The film cuts from a packed group shot straight
into ~40+ seconds of blank titles/credits with zero faces, and `extinction_sec=57.4`
is comfortably long enough to bridge that entire gap without expiring, so the
tracker faithfully keeps reporting "last known position" for a cast that is no
longer on screen at all.
This reframes the "long extinction window wins" DE-search pattern (see above): it
isn't unambiguously good. It buys recall by bridging real gaps (occlusion, turned
face) in some films, but on others — specifically, hard cuts into long faceless
footage — it manufactures a frozen-bbox ghost the tracker has no way to verify,
precisely the failure mode the *original* short-extinction-window default (`1.5s`)
was chosen to avoid. The training-set films apparently didn't have a long enough
faceless stretch after a confirmed identity to expose this; the held-out set did.
Frames for all three films (`benny_joon`, `many_saints`, `downton_abbey` — one strong
performer, two failure cases) are under `experiments/results/holdout/frames/`, each
with a `manifest.json` listing the bucket (`best`/`fpi`/`fn`), timestamp, and
predicted vs. ground-truth actors for every dumped frame. Frames are annotated with
bounding boxes + name/confidence (green = identified, orange = unknown), matching
`debug_renderer_node.hpp`'s colour convention. Generated by
`scripts/optimizer/dump_error_frames.py --raw <replay.py --raw-out output>` (see
Reproduce).
`dump_error_frames.py --interval-sec 600` also supports a per-N-second sweep
instead of the fixed best/fpi/fn buckets: one best (highest Jaccard) and one worst
(lowest Jaccard) frame per 10-minute window across the whole film, e.g.
`experiments/results/holdout/frames/many_saints_intervals/` (13 windows × 2 = 26
frames for the ~2h Many Saints runtime) — a way to sample "how are we doing" evenly
across a film's runtime rather than only at its most extreme seconds.
## Caveats / what this is not
- **r50's 4 combos used the old, narrower search bounds** and aren't fully
comparable to the other 12 on `anneal_sec`/`extinction_sec`.
- **The applied defaults use `full_exp`, not the higher-scoring `restricted_exp`**,
because cast-restriction isn't a real runtime feature yet (see above). The
78.3% F1 number is not what the shipped defaults will produce — 75.3% is.
- **`full_exp` is the best full-gallery combo, but not the safest.** Per the
isolated-effects analysis above, `expand_gallery=true` only cleanly pays off
when paired with cast-restriction; on the full gallery it's flat on F1 while
~4x-ing misIDs (209→864, averaged across models). `full_noexp` scores lower
(72.4% vs 75.3% for LVFace) but with **zero** true misIDs and higher precision
(94.2% vs 89.7%). Kept `full_exp` as shipped since it's the highest-F1 option
available without cast-restriction, but this is a real F1-vs-safety trade, not
a strictly-better choice — worth revisiting if misID rate matters more than
the last few points of F1 for a given deployment.
- **Switching the default model is an operational change, not just a config tweak**:
any existing gallery built from r50 embeddings is incompatible with LVFace
embeddings and needs rebuilding.
## Reproduce
```bash
# 4-film matrix, all 4 models × 2 modes × 2 expansion settings
bash experiments/run_rep4_subprocess.sh
# single combo
SAE_EXPAND=1 REPLAY_WORKERS=4 DE_WORKERS=2 python3 scripts/optimizer/optimize.py \
--manifest experiments/manifests/rep4_LVFace-B_Glint360K_full.json \
--gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5 \
--params prob_threshold:0.5:0.999 anneal_sec:1:60 extinction_sec:1:60 \
--popsize 10 --maxiter 15 --trajectory traj.jsonl --out best.json
# replay the shipped config against a held-out film — --raw-out is needed to draw
# bboxes later (the merged pred.json has no per-frame bbox, only actor windows)
python3 scripts/optimizer/replay.py \
--dump experiments/dumps/LVFace-B_Glint360K/dump_<slug>.h5 \
--gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5 \
--out pred.json --raw-out raw.jsonl --prob-threshold 0.754 --anneal-sec 35.54 \
--extinction-sec 57.43 --expand-gallery
# dump example frames (best-agreement / FPI / FN) for visual inspection, annotated
# with bounding boxes + names (--raw is optional; omit for unannotated frames)
python3 scripts/optimizer/dump_error_frames.py \
--pred pred.json --raw raw.jsonl --xray experiments/xray/.../<xray_dir> \
--movie "<path to source video>" \
--gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5 \
--out-dir experiments/results/holdout/frames/<name> --n-per-bucket 4
# or: one best + one worst frame per 10-minute window across the whole film
python3 scripts/optimizer/dump_error_frames.py \
--pred pred.json --raw raw.jsonl --xray experiments/xray/.../<xray_dir> \
--movie "<path to source video>" \
--gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5 \
--out-dir experiments/results/holdout/frames/<name>_intervals --interval-sec 600
```
See also: `docs/optimizer-experiments.md` (prior round, superseded metric),
`experiments/SESSION_STATE.md`, and memory: kpn-python-replay-optimizer,
gallery-coverage-gap, xray-validation-results, per-scene-presence-eval-design.
docs/service-conversion.md
+215
View File
@@ -0,0 +1,215 @@
# Conversion to service — a native idle-GPU worker
Status: **design / proposal**. Nothing here is built yet.
## The idea
Turn the CLI tools into a **turnkey batch worker that uses the machine's idle
GPU**: it analyses newly-added Jellyfin media when you're not using the computer
(screen locked), and stops the instant you come back. It's an overnight job on
your own Linux box.
**No Docker.** This runs on your own machine with your own drivers, so a container
buys little and costs a lot: GPU passthrough (nvidia-container-toolkit, or
`/dev/kfd`+`/dev/dri`+`video` group for ROCm) is the single most fragile part of a
containerised setup, and it exists *only* because of the container. Natively, the
GPU just works with the drivers you already have, and the media paths Jellyfin
reports are just real paths — no re-mounting. So we ship a **native installer**
instead of an image builder.
Two deliverables:
1. **An installer** — `scripts/build_install.py`. Detects your distro, ensures the
GPU/build dependencies are present (via `dnf`/`pacman`), compiles `scene_analyze`
for your GPU, and installs the binary + Python glue + two systemd **user**
units under `~/.local`.
2. **A screen-lock gate** — one of those systemd units watches logind lock/unlock
and starts/stops the worker. Lock → analyse. Unlock → stop.
## What already exists (reuse, don't rebuild)
The processing loop is already implemented — this is packaging, building, and
lock-gating, not new pipeline logic.
| Piece | Where | What it does |
|---|---|---|
| Analysis engine | `build/scene_analyze` | Video → face detect/align/embed → gallery match → result JSON |
| Backend selection | `CMakeLists.txt` (`SAE_INFERENCE_BACKEND`, `SAE_GEMM_BACKEND`) | ORT/TRT + ROCm/CUDA, chosen **at build time** |
| New-media queue | JRay plugin → `GET /Plugins/JRay/Tasks/Pending` | Backlog of items with no results yet |
| Worker loop | `run_from_jellyfin.py --worker` | Poll Pending → run `scene_analyze` → push results |
| Result push | `PUT /Plugins/JRay/Items/{id}/Truth` | Stores per-actor scene windows back in Jellyfin |
| Incremental gallery | `make_jellyfin_gallery.py --merge` | Embeds only cast not already in the gallery |
| Secrets loader | `.env` via `sae_env.py` | `JELLYFIN_URL`, `JELLYFIN_API_KEY`, `TMDB_API_KEY` |
## Installer config
One file. Build-time settings (fixed when we compile) vs. run-time settings (in the
worker's `.env`, editable without recompiling).
```yaml
# install.yaml — consumed by scripts/build_install.py
platform: nvidia # nvidia | amd | cpu → picks the cmake backend
model:
arcface: LVFace-B_Glint360K.onnx # embedder compiled against; gallery MUST match
schedule:
gallery_scan_interval: 24h # incremental --merge cadence; 0 disables the scanner
prefix: ~/.local # install root (bin, share, systemd user units)
# runtime (written to the worker .env, not compiled in):
runtime:
jellyfin_url: http://localhost:8096
# JELLYFIN_API_KEY / TMDB_API_KEY are filled into .env by hand after install
```
**Secrets never go in the repo or a build artifact** — the installer writes a
`.env` under the install prefix with blanks for the keys, and you fill them in
once. `sae_env.py` already loads it.
**Model ⇄ gallery coupling (guard, don't just document):** embeddings from
different recognition models aren't interchangeable. We compile against one
embedder; the gallery must be built with the same one. Stamp the embedder name
into `gallery.json`, and have the worker **refuse to start** if the gallery's
embedder ≠ the configured `model.arcface`, rather than silently mismatching.
## Dependencies via the system package manager
The heavy build/runtime deps (OpenCV, ffmpeg, the GPU stack) are best provided by
the distro, not vendored. The installer ships a per-distro dependency list and
either installs them or prints the exact command. Targets: **Fedora (dnf)** and
**Arch (pacman)** first.
| Dependency | Fedora (dnf) | Arch (pacman) |
|---|---|---|
| OpenCV | `opencv-devel` | `opencv` |
| ffmpeg | `ffmpeg-free`/`ffmpeg` (RPM Fusion) | `ffmpeg` |
| CMake / toolchain | `cmake gcc-c++` | `cmake gcc` |
| CUDA + TensorRT (nvidia) | NVIDIA CUDA repo + `libnvinfer-*` | `cuda`, `tensorrt` |
| ROCm (amd) | `rocm-hip-sdk` / `rocblas-devel` | `rocm-hip-sdk`, `rocblas` |
| ONNX Runtime | **not packaged** — installer fetches a pinned release tarball into the prefix | AUR `onnxruntime` (or same pinned-tarball fallback) |
So the flow is: **detect distro → check each package → install via the native
manager (or print `sudo dnf install …` / `sudo pacman -S …`)**, with ONNX Runtime
as the one known gap the installer fills itself (a pinned upstream release
extracted under the install prefix, so it doesn't depend on a system package that
may not exist). CUDA/ROCm being present is *assumed* — you already run a GPU
desktop; the installer verifies and points you at the vendor repo if not.
## What `build_install.py` does
```
build_install.py install.yaml
│
├─ detect distro (dnf vs pacman) and platform from config
├─ ensure deps: install via manager, or print the exact command; fetch ONNX Runtime if needed
├─ cmake + build scene_analyze with the platform's backend flags:
│ nvidia → -DSAE_INFERENCE_BACKEND=TRT -DSAE_GEMM_BACKEND=CUDA
│ amd → -DSAE_INFERENCE_BACKEND=ORT -DSAE_GEMM_BACKEND=ROCM
│ cpu → -DSAE_INFERENCE_BACKEND=ORT (CPU EP; slow, for smoke tests)
├─ install into <prefix>:
│ bin/sae-scene-analyze the compiled binary
│ share/sae-worker/ Python glue + a venv (requests, etc.), models/
│ share/sae-worker/.env runtime config (keys blank, url from config)
├─ install systemd --user units:
│ sae-worker.service runs the worker + gallery-scan supervisor
│ sae-lock-gate.service watches logind lock/unlock, start/stops the worker
└─ print next steps (edit .env, `systemctl --user enable --now sae-lock-gate`)
```
## The worker service (supervisor)
`sae-worker.service` runs a small Python supervisor as its main process:
- starts the **worker loop** (`run_from_jellyfin.py --worker`) — the hot path,
- starts a **gallery-scan timer** — sleeps `gallery_scan_interval`, runs
`make_jellyfin_gallery.py --merge`, repeats,
- exits cleanly on SIGTERM (see re-queue below).
## The lock gate
`sae-lock-gate.service` runs a tiny watcher that subscribes to logind
lock/unlock signals and drives the worker service:
```
screen locks → systemctl --user start sae-worker.service
screen unlocks → systemctl --user stop sae-worker.service (SIGTERM)
```
**Screen-lock is the only signal — deliberately.** We don't also gate on GPU/CPU
load, because our own worker *is* the load: a load threshold would form a feedback
loop (worker starts → GPU spikes → threshold trips → worker stops → load drops →
restart → …). Lock state is external to what the worker does, so it can't
oscillate.
Signal source is desktop-dependent: logind `Lock`/`Unlock` (GNOME/KDE via
`loginctl`/D-Bus) covers most setups; a `swayidle`/`xss-lock` hook is the fallback
for wlroots/X-only compositors. The installer picks based on what's present.
## On resume: hard stop + re-queue (it's free)
Stopping the worker mid-analysis costs nothing to reschedule, because of how the
JRay queue works: **an item only leaves `/Tasks/Pending` once its results are
pushed** (`push_truth`). A worker stopped mid-`scene_analyze` simply leaves that
item Pending — next lock picks it up again. No re-queue bookkeeping.
Two small correctness requirements (the only worker changes needed):
1. **Never push a partial result.** Already true — `push_truth` runs only after
`scene_analyze` returns; a killed run pushes nothing. ✓ (keep it that way).
2. **Clean up on signal.** `process_item` writes a temp filtered-gallery file and
unlinks it in a `finally`; a SIGKILL skips `finally`. Fix: write temps under a
dir the worker wipes on start, and/or a SIGTERM handler that unlinks before
exit. Minor.
Accepted trade-off: a partially-analysed title restarts from scratch next lock.
Fine for an overnight/idle workload; no mid-video checkpointing.
## The end-to-end UX
```bash
# once: build + install for your GPU + model
./scripts/build_install.py install.yaml
# detects Fedora/Arch, ensures deps, compiles, installs units under ~/.local
# once: set your keys, enable the gate
$EDITOR ~/.local/share/sae-worker/.env # JELLYFIN_API_KEY, TMDB_API_KEY
systemctl --user enable --now sae-lock-gate.service
# from then on: nothing. Lock your screen → it analyses. Unlock → it stops.
```
No Docker, no GPU passthrough config, no media re-mounting — the worker sees the
same filesystem and GPU as everything else on the box.
## Implementation plan (follow-up commits)
Ordered so each step stands alone:
1. **installer skeleton** — `scripts/build_install.py`: parse `install.yaml`,
distro detect, dependency check/print (start with cpu platform so it builds
without a GPU), cmake+build, copy into prefix.
2. **supervisor + cleanup** — `scripts/service.py` (worker loop + gallery-scan
timer + SIGTERM); temp-file cleanup fix in `run_from_jellyfin.py`.
3. **systemd units + lock gate** — generate/install `sae-worker.service`,
`sae-lock-gate.service`, and the logind lock watcher.
4. **gallery/model guard** — stamp embedder into `gallery.json`; startup mismatch
check.
5. **platform + distro matrix** — nvidia/amd backends; dnf/pacman dep lists; ONNX
Runtime fetch fallback.
6. **docs** — README "Run on your idle GPU" section.
## Settled decisions
- **ONNX Runtime build** — the installer fetches the **ROCm ORT** release. It
serves the `amd` platform, and its CPU execution provider covers the `cpu`
smoke-test fallback too, so one download handles both. (nvidia uses raw TRT and
doesn't need ORT.)
- **`dnf`/`pacman` invocation** — **auto-install.** The installer runs `sudo dnf
install …` / `sudo pacman -S …` itself (prompting for sudo), rather than only
printing the command. It still prints what it's about to install first.
- **Distro coverage** — **Fedora + Arch only** for now. Debian/Ubuntu (`apt`) is
out of scope.
## Open questions
*(none blocking — the spec above is buildable as-is.)*
experiments/.gitignore
+21
View File
@@ -0,0 +1,21 @@
# experiments/ in git keeps only scripts, README.md, SESSION_STATE.md, and this
# file. Every data artifact — galleries, embedding dumps, the X-Ray corpus,
# montage/frame images, DE trajectories, film manifests, and result summaries —
# is pushed/pulled via scripts/artifacts/{push,pull}_artifacts.sh to the Gitea
# generic package registry instead (see docs/rep4-optimizer-results.md).
xray/
dumps/
galleries/
*.h5
manifests/
trajectories/
results/
# Raw run logs and scratch scripts (regenerated by every run).
_scratch/
# Real local media paths (film slug -> path on this machine). Never committed —
# these paths embed the specific source file names, which can include
# scene-release tags. Only file-lut.template.json (placeholders) is tracked;
# copy it to file-lut.json and fill in your own paths.
file-lut.json
experiments/README.md
+59
View File
@@ -0,0 +1,59 @@
# experiments/ — X-Ray validation & optimizer artifacts
Durable home (in the repo tree, NOT `/tmp` scratch — a scratch wipe once cost an hour)
for the data behind the X-Ray threshold-optimization and embedding-model bake-off.
## Layout
- `xray/` — Amazon X-Ray Zenodo dataset (gitignored, ~140MB; DOI 10.5281/zenodo.17659734).
- `dumps/` — per-model embedding dumps, one HDF5 per (model, film). Gitignored (large).
Naming: `<model>/dump_<Film>.h5`. Regenerate with `scene_analyze --dump-embeddings`.
- `galleries/` — per-model galleries (gitignored JSON). `gallery_<model>.json` +
augmented variants. Regenerate with build_gallery / fetch_missing_actors.
- `manifests/` — film manifests (committed — small, and the Jellyfin ID join is the
authoritative record of which films/paths/X-Ray-dirs were used).
- `trajectories/` — DE trajectories, one JSONL per run (committed — the evidence).
- `results/` — final per-run metrics + the model comparison table (committed).
## Embedding-model bake-off (July 2026)
Question: is LVFace-B (455MB) actually the best vs X-Ray, or just the biggest?
Method: **optimize per model** — each model gets its own dumps + gallery + full DE run,
then compare each model at ITS OWN optimum (fairest — no model penalised by another's
threshold). Scored by the weighted per-scene metric (out-of-cast misID ×10; see
docs/optimizer-experiments.md).
Models:
| model | file | size | MovieNet rank-1 (prior) |
| ----- | ---- | ---- | ----------------------- |
| LVFace-B_Glint360K | models/LVFace-B_Glint360K.onnx | 455 MB | — |
| ArcFace w600k R50 | models/arcface_w600k_r50.onnx | 174 MB | 85.2% |
| ArcFace R18 | models/arcface_r18.onnx | 48 MB | 72.2% |
| ArcFace w600k MBF | models/arcface_w600k_mbf.onnx | 13 MB | 83.3% |
9 genuine X-Ray-overlap films (Jellyfin ID join): Benny & Joon, Café Society,
Downton Abbey: A New Era, Lord of War, Lovelace, The Many Saints of Newark, Scarface,
Sound of Metal, Valerian.
## Gallery-mode bake-off (full vs cast-restricted)
Second axis alongside the model comparison: does restricting the matcher's candidate
set to a title's credited cast reduce cross-film misIDs (e.g. naming Archie Yates in a
film he's not in) vs. matching against the whole 2418-actor gallery?
- **full** — match against the entire model gallery (2418 actors).
- **restricted** — per film, match only against its Jellyfin credited cast, filtered
from the gallery by jellyfin_id. This is what run_from_jellyfin.py does in production.
**LIMITATION — Jellyfin stores only ~15 actors per title.** Jellyfin's People list is
capped at the top-billed cast (~15 Actors), NOT the full IMDb/X-Ray cast (e.g. Scarface:
Jellyfin 15 vs X-Ray 67). This is a hard limit of the metadata Jellyfin imports — not a
query parameter (verified: /Items?Fields=People returns 15 regardless; the single-item
/Items/{id} endpoint 400s on this server). So the "restricted" arm restricts to the ~15
top-billed leads, which caps its achievable recall at whatever fraction of on-screen
actors are top-billed, but should drive out-of-cast misIDs toward zero. A production
deployment wanting fuller cast restriction would need a richer cast source than Jellyfin
(TMDB/IMDb full credits).
Matrix: 4 models × {full, restricted} = 8 DE runs, all reusing the 36 dumps + 4 baseline
galleries (no augmentation — avoids test-set leakage on either arm). Scored by the
duration-weighted per-scene metric with the misID split (report_rates.py).
experiments/SESSION_STATE.md
+105
View File
@@ -0,0 +1,105 @@
# Session state — X-Ray optimizer + model bake-off (as of 2026-07-18)
Handoff for a fresh session. Everything below is UNCOMMITTED — commit early next session.
## What we're doing
Optimizing the scene-actor pipeline's thresholds against Amazon X-Ray ground truth, and
running a **model bake-off** (4 embedding models × gallery-mode × expansion) to answer:
is LVFace (455MB) actually best, or just biggest? Does cast-restriction cut false IDs?
Does per-film gallery expansion help?
## The metric (final form — this is what to use)
`scripts/optimizer/second_score.py` — UNIFORM PER-SECOND sampling vs X-Ray:
- At each second t: GT = X-Ray scene's cast at t; Pred = actors whose window covers t.
- TPI / FPI / FN counted per second. **FPI weighted 10×** when the named actor isn't in
the film's cast at all (a true misID like naming Archie Yates in a film he's not in) vs
an in-cast timing slip.
- **FN is fair**: only counts gallery-known cast (67% of X-Ray cast have no reference
embedding, can't be recognised — see [[gallery-coverage-gap]]).
- **agreement_rate** = mean per-second Jaccard (partial credit: "% of on-screen actors we
agree with X-Ray about, over time"). NOT exact-set match.
- Objective = macro-mean per-second weighted F1.
## Two DIFFERENT hangs — do not conflate them (corrected 2026-07-18)
**(a) The self-inflicted 100% hang (FIXED).** replay.py's CLI briefly called
`replay(..., stop=False)` intending to `os._exit(0)` straight after, to "dodge" teardown.
That was wrong: `PyNode::stop()` is the ONLY thing that sets `stop_flag_=true`, which is the
ONLY exit condition for the source node's `run_loop()`. Skipping it meant the local `net`
destructor — which runs synchronously when `replay()` returns, BEFORE main() can reach
os._exit — joined a thread that could never stop. A **guaranteed** hang, not the driver
flake. Symptom: every solo replay timed out at 45s and DE reported flat F1=0.0%.
FIX: `replay(..., stop=True)` so `PyNode::stop()` signals the thread before the join;
removed the dead os._exit / unused os import. VERIFIED: 45s guaranteed timeout → clean ~8s
completion (3/3), and optimize.py's DE sweep returns correct non-zero metrics (F1 48-66%,
matching prior best-so-far). Only 1 isolated per-film timeout in 11 evals × 3 films.
**(b) The genuine ROCm flake (rare, tolerated).** net.stop()→jthread.join() CAN still hang
on a KPN worker stuck mid-rocBLAS-GEMM — a KNOWN ROCm bug
(github.com/RadeonOpenCompute/ROCT-Thunk-Interface#56), NOT our code. HSA_ENABLE_SDMA=0
makes it WORSE (breaks the matcher's DMA). It is much rarer than the ~20-30% figure quoted
earlier in this session — that number was inflated by (a). The existing subprocess + 45s
timeout absorbs it correctly.
## Both architectures are usable
- `scripts/optimizer/optimize.py` + `replay.py` — subprocess per film, simpler, tolerates the
rare true flake via its timeout. NOT broken; good for fallback / quick single-model runs.
- `scripts/optimizer/model_server.py` + `optimize_server.py` — ONE persistent net per
(model, gallery); replay each film by SWITCHING THE SOURCE (repoint frame list + reset
index), change thresholds via runtime SETTERS, os._exit(0) at the very end (after all work,
so no destructor-join problem). Higher throughput: skips gallery/build overhead per eval.
VERIFIED: "ready", replays, emits metrics, ~30-40s/eval (GPU-bound, films serial).
Still the preferred option for the long overnight matrix.
## Key C++ changes made (all in the KPN spec-and-tsan branch + our nodes)
1. Runtime setters: `IdentityMatcherFunc::set_prob_threshold`, `SceneTrackerFunc::set_extinction_sec`
(src/nodes/*). Exposed via sae_kpn: `set_prob_threshold(net,name,v)`, `set_extinction_sec(...)`.
Needed `ObjectVariantNodeWrapper::functor()` + `PyNetwork::node_ptr()` accessors.
2. `Channel::push_blocking()` (external/KPN/.../channel.hpp) — lossless backpressure push
(waits instead of dropping when full). Exposed on IVariantChannel/VariantChannel; PyNode's
run_loop now uses it. Reduced but did NOT fully fix a residual ~0.5% frame loss (25/5915)
— the loss is elsewhere (matcher output or reader EOF-race). DECISION: accept it, <0.5%
scattered doesn't change per-second F1 or rankings. Don't chase further.
3. `dump_embeddings` standalone exe + `--max-decode-fps` (fixes LVFace dump truncation under
parallel load). HDF5 gallery fast-load in gallery_store.cpp (18s JSON → 0.06s).
`scripts/optimizer/json_to_hdf5_gallery.py` converts; galleries are `.h5` now.
All of KPN, matcher, scene_tracker, bindings need a rebuild:
`cmake --build build --target sae_kpn sae_gallery dump_embeddings scene_analyze`
## Data on disk (durable, experiments/)
- `experiments/xray/` — X-Ray Zenodo dataset. `experiments/dumps/<model>/dump_<slug>.h5` —
all 9 films × 4 models, ALL FULL (LVFace re-dumped with --max-decode-fps 8). VERIFY counts
match R50 before trusting (LVFace truncated under parallel dumping earlier).
- `experiments/galleries/gallery_<model>.h5` (+ restricted/<model>/<slug>.h5, per-film cast-
filtered to Jellyfin's ~15 top-billed — Jellyfin's hard cap, see experiments/README.md).
- `experiments/manifests/rep3_<model>_<mode>.json` — 3 REPRESENTATIVE films (Lord of War /
Scarface / Sound of Metal = clean / ensemble-lookalike / high-coverage) to keep evals fast
(~28s vs ~90s for 9). Winner should be re-scored on all 9 after.
- `experiments/manifests/films_<model>_<mode>.json` — all 9 films.
## Salvaged partial results (per-second metric)
- R50 full +expand: **F1 66.4%** (208 evals, converged) — best so far
- R50 full noexp: 55-60% → **expansion helps ~+6-11 recall**
- MBF full noexp: 58.6%
- (older scene-metric runs, superseded: R50≈LVFace≈MBF ~85%, restricted>full, LVFace not
worth its size — but those used the OLD scene-union metric, redo with per-second.)
## TO DO next session
1. **COMMIT everything first** (logical chunks: KPN setters+push_blocking; sae_kpn+dump exe;
HDF5 gallery; optimizer scripts; per-second metric; experiments manifests/results/docs +
tuned config.hpp defaults prob_threshold 0.76 extinction 1.5).
2. Launch the full 16-run matrix via model_server on rep3 films (write trajectories to
experiments/, NOT /tmp — /tmp gets wiped mid-session and cost us hours). ~28s/eval ×
~84 evals × 16 = ~10hr. Runner pattern: experiments/run_overnight_rep3.sh but pointing
optimize_server.py at model_server.
3. assemble table: best model + expansion effect + misID, from experiments/results/*.json.
4. Consider upstreaming to KPN++: runtime node setters, push_blocking, node_ptr/functor().
## Gotchas that burned time (don't repeat)
- /tmp scratch gets WIPED mid-session → lost dumps + test files repeatedly. Use experiments/.
- Verify a launched runner script EXISTS and PRODUCES evals before walking away (a heredoc
once silently failed to write; a stale-code process ran the old metric for 12h).
- pgrep/ps "survivors" are often the grep's own shell wrapper — check via /proc cmdline or ps.
- Running many DE/replay processes in parallel on one GPU → deadlock/thrash. GPU peaks ~35%
(not saturated) but concurrency>2-3 wedges. Serial-ish is safer.
experiments/build_manifests.py
+66
View File
@@ -0,0 +1,66 @@
#!/usr/bin/env python3
"""
Build per-(model, mode) manifests for the model × gallery-mode bake-off.
Modes:
full — every film matches against the whole model gallery (2418 actors)
restricted — each film matches only its Jellyfin credited cast (~15 top-billed),
via a per-film gallery filtered from the model gallery by jellyfin_id.
Each manifest is a list of {name, xray, slug, dump, gallery} — no "movie" path (that's
resolved locally via experiments/file-lut.json, see run_montage_all.py, to avoid
embedding source filenames in a file that gets shared as an artifact). optimize.py
reads film["gallery"] per film, so restricted mode just points each film at its own
filtered gallery — no optimizer change needed.
Writes experiments/manifests/films_<model>_<mode>.json and the restricted galleries to
experiments/galleries/restricted/<model>/<slug>.h5.
"""
import json
import sys
from pathlib import Path
REPO = Path(__file__).resolve().parent.parent
sys.path.insert(0, str(REPO / "scripts" / "validation"))
sys.path.insert(0, str(REPO / "scripts"))
from identity import norm_name # noqa: E402
from sae_gallery import load_gallery_hdf5, save_gallery_hdf5 # noqa: E402
MODELS = ["arcface_w600k_r50", "arcface_r18", "arcface_w600k_mbf", "LVFace-B_Glint360K"]
films = json.loads((REPO / "experiments/manifests/films.json").read_text())
casts = json.loads((REPO / "experiments/manifests/jellyfin_casts.json").read_text())
def restrict_gallery(model_gallery: dict, cast_ids: set[str]) -> dict:
kept = [a for a in model_gallery["actors"] if a.get("jellyfin_id", "") in cast_ids]
return {"actors": kept}
for model in MODELS:
gpath = REPO / f"experiments/galleries/gallery_{model}.h5"
if not gpath.exists():
print(f"skip {model}: gallery not built yet ({gpath})")
continue
model_gal = load_gallery_hdf5(gpath)
# full mode
full = [{**f, "dump": f"experiments/dumps/{model}/dump_{f['slug']}.h5",
"gallery": f"experiments/galleries/gallery_{model}.h5"} for f in films]
(REPO / f"experiments/manifests/films_{model}_full.json").write_text(json.dumps(full, indent=2, ensure_ascii=False))
# restricted mode — per-film filtered gallery
rdir = REPO / f"experiments/galleries/restricted/{model}"
rdir.mkdir(parents=True, exist_ok=True)
restr = []
for f in films:
cast_ids = set(casts.get(f["name"], []))
rg = restrict_gallery(model_gal, cast_ids)
rgpath = rdir / f"{f['slug']}.h5"
save_gallery_hdf5(rg, rgpath)
restr.append({**f, "dump": f"experiments/dumps/{model}/dump_{f['slug']}.h5",
"gallery": str(rgpath.relative_to(REPO)),
"_cast_size": len(rg["actors"])})
(REPO / f"experiments/manifests/films_{model}_restricted.json").write_text(json.dumps(restr, indent=2, ensure_ascii=False))
avg = sum(r["_cast_size"] for r in restr) / len(restr)
print(f"{model}: full (2418) + restricted (avg {avg:.0f} actors/film) manifests written")
experiments/file-lut.template.json
+11
View File
@@ -0,0 +1,11 @@
{
"Benny___Joon": "/path/to/your/movies/<replace-with-your-file>.mp4",
"Café_Society": "/path/to/your/movies/<replace-with-your-file>.mp4",
"Downton_Abbey__A_New_Era": "/path/to/your/movies/<replace-with-your-file>.mp4",
"Lord_of_War": "/path/to/your/movies/<replace-with-your-file>.mp4",
"Lovelace": "/path/to/your/movies/<replace-with-your-file>.mp4",
"The_Many_Saints_of_Newark": "/path/to/your/movies/<replace-with-your-file>.mp4",
"Scarface": "/path/to/your/movies/<replace-with-your-file>.mp4",
"Sound_of_Metal": "/path/to/your/movies/<replace-with-your-file>.mp4",
"Valerian_and_the_City_of_a_Thousand_Plan": "/path/to/your/movies/<replace-with-your-file>.mp4"
}
experiments/run_montage_all.py
+93
View File
@@ -0,0 +1,93 @@
#!/usr/bin/env python3
"""
run_montage_all.py — replay + best/worst-per-scene montage for all films in a
manifest, using the shipped config.hpp defaults (LVFace, prob_threshold=0.754,
anneal_sec=35.54, extinction_sec=57.43, expand_gallery). Skips a film if its
montage manifest.json already exists (safe to re-run/resume).
Manifests no longer carry a "movie" path (removed to avoid embedding source
filenames — some carry scene-release tags — in a file that gets zipped and
pushed to the artifact registry). Movie paths are resolved locally via
experiments/file-lut.json (gitignored; copy file-lut.template.json and fill
in your own paths).
Usage:
python experiments/run_montage_all.py \
--manifest experiments/manifests/films_LVFace-B_Glint360K_full.json \
--gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5 \
--out-dir experiments/results/holdout/montage
"""
import argparse
import json
import subprocess
import sys
from pathlib import Path
REPO = Path(__file__).resolve().parent.parent
REPLAY = str(REPO / "scripts" / "optimizer" / "replay.py")
MONTAGE = str(REPO / "scripts" / "optimizer" / "dump_scene_montage.py")
FILE_LUT = REPO / "experiments" / "file-lut.json"
CFG = {"prob_threshold": "0.7540024664611272", "anneal_sec": "35.53996030397922",
"extinction_sec": "57.43359645269811"}
def main():
p = argparse.ArgumentParser()
p.add_argument("--manifest", required=True)
p.add_argument("--gallery", required=True)
p.add_argument("--out-dir", required=True)
args = p.parse_args()
if not FILE_LUT.exists():
sys.exit(f"{FILE_LUT} not found — copy file-lut.template.json to "
f"file-lut.json and fill in your local movie paths")
file_lut = json.loads(FILE_LUT.read_text())
films = json.loads(Path(args.manifest).read_text())
out_root = Path(args.out_dir)
out_root.mkdir(parents=True, exist_ok=True)
for f in films:
slug = f["slug"]
montage_dir = out_root / slug
manifest_path = montage_dir / "manifest.json"
if manifest_path.exists():
print(f"[run_montage_all] skip {slug} (already done)", file=sys.stderr)
continue
movie_path = file_lut.get(slug)
if not movie_path:
print(f"[run_montage_all] skip {slug}: no entry in {FILE_LUT}", file=sys.stderr)
continue
print(f"[run_montage_all] === {f['name']} ({slug}) ===", file=sys.stderr)
raw_path = out_root / f"raw_{slug}.jsonl"
pred_path = out_root / f"pred_{slug}.json"
replay_cmd = [sys.executable, REPLAY, "--dump", f["dump"], "--gallery", args.gallery,
"--out", str(pred_path), "--raw-out", str(raw_path),
"--prob-threshold", CFG["prob_threshold"],
"--anneal-sec", CFG["anneal_sec"],
"--extinction-sec", CFG["extinction_sec"], "--expand-gallery"]
r = subprocess.run(replay_cmd, capture_output=True, text=True, timeout=120)
if r.returncode != 0:
print(f"[run_montage_all] replay FAILED for {slug}: {r.stderr[-2000:]}",
file=sys.stderr)
continue
montage_cmd = [sys.executable, MONTAGE, "--raw", str(raw_path), "--dump", f["dump"],
"--xray", f["xray"], "--movie", movie_path, "--gallery", args.gallery,
"--out-dir", str(montage_dir)]
r = subprocess.run(montage_cmd, capture_output=True, text=True, timeout=1800)
if r.returncode != 0:
print(f"[run_montage_all] montage FAILED for {slug}: {r.stderr[-2000:]}",
file=sys.stderr)
continue
print(r.stderr.strip().splitlines()[-1] if r.stderr else "(no output)", file=sys.stderr)
print("[run_montage_all] ALL DONE", file=sys.stderr)
if __name__ == "__main__":
main()
experiments/run_rep4_subprocess.sh
+26
View File
@@ -0,0 +1,26 @@
#!/bin/bash
set -u
cd /home/dtourolle/Development/scene-actor-extraction
mkdir -p experiments/trajectories experiments/results
MODELS=(arcface_w600k_r50 arcface_r18 arcface_w600k_mbf LVFace-B_Glint360K)
MODES=(full restricted); EXPAND=(1 0)
for model in "${MODELS[@]}"; do
for mode in "${MODES[@]}"; do
for exp in "${EXPAND[@]}"; do
xtag=$([ "$exp" = 1 ] && echo exp || echo noexp); tag="${model}_${mode}_${xtag}"
best="experiments/results/rep4_best_${tag}.json"; traj="experiments/trajectories/rep4_${tag}.jsonl"
[ -f "$best" ] && { echo "skip $tag"; continue; }
: > "$traj"; echo "=== $tag START $(date '+%H:%M') ==="
SAE_EXPAND=$exp REPLAY_WORKERS=4 DE_WORKERS=2 python3 scripts/optimizer/optimize.py \
--manifest "experiments/manifests/rep4_${model}_${mode}.json" \
--gallery "experiments/galleries/gallery_${model}.h5" \
--params prob_threshold:0.5:0.999 anneal_sec:1:60 extinction_sec:1:60 \
--popsize 10 --maxiter 15 --seed 0 \
--trajectory "$traj" --out "experiments/results/opt_rep4_${tag}.json" \
> "experiments/results/rep4_de_${tag}.log" 2>&1
python3 -c "import json;r=[json.loads(l) for l in open('$traj')];b=max(r,key=lambda x:x['f1']);json.dump({'best':b,'n_evals':len(r)},open('$best','w'))" 2>/dev/null
echo "done $tag $(date '+%H:%M'): $(python3 -c "import json;d=json.load(open('$best'));print('F1=%.1f%% misID=%d'%(d['best']['f1']*100,d['best'].get('FPI_misid',-1)))" 2>/dev/null)"
done
done
done
echo "=== REP4 MATRIX COMPLETE $(date '+%F %H:%M') ==="
mkdocs.yml
+42
View File
@@ -0,0 +1,42 @@
site_name: scene-actor-extraction
site_description: Face-recognition pipeline for finding on-screen actor presence in film/TV, built on KPN++
repo_url: https://gitea.tourolle.paris/dtourolle/scene-actor-extraction
repo_name: dtourolle/scene-actor-extraction
theme:
name: material
palette:
- scheme: slate
primary: indigo
accent: indigo
features:
- navigation.tabs
- navigation.sections
- navigation.top
- content.code.copy
- content.code.annotate
nav:
- Home: index.md
- Rep4 Bake-off & Re-tune: rep4-optimizer-results.md
- Optimizer Experiments (prior round): optimizer-experiments.md
- Service Conversion (proposal): service-conversion.md
markdown_extensions:
- admonition
- toc:
permalink: true
- pymdownx.highlight:
anchor_linenums: true
line_spans: __span
pygments_lang_class: true
- pymdownx.inlinehilite
- pymdownx.superfences
- pymdownx.tabbed:
alternate_style: true
- pymdownx.snippets:
base_path: ['.']
check_paths: true
- pymdownx.details
- attr_list
- md_in_html
models/transnetv2.onnx
LFS
BIN
View File
Binary file not shown.