docs: rep4 bake-off write-up, MkDocs site, artifact-registry-backed experiments
docs/rep4-optimizer-results.md is the main deliverable: the model bake-off + threshold re-tune experiment log, including the ROCm teardown deadlock root cause and fix, DE concurrency tuning, the 16-combo results table, held-out validation against 5 films never seen by the optimizer (macro F1 67.4% vs. 75.3% training — a real generalization gap), the frozen-bbox "ghost track" failure mode found via annotated frame evidence, calibration curves per model, and an isolated-effects breakdown of gallery scope vs. pose expansion. MkDocs site (mkdocs.yml, docs/index.md) renders docs/*.md; scripts/docs/ pulls referenced images from the artifact registry and generates the calibration chart at build time (see the tooling commit) rather than committing images to the repo. experiments/ now keeps only scripts + README + SESSION_STATE.md in git — every data artifact (galleries, dumps, X-Ray corpus, montage frames, trajectories, manifests, results) moved to the Gitea package registry. film-lut.template.json is the committed placeholder for the gitignored file-lut.json (real local movie paths, never shared — some source filenames carry scene-release tags). Adds models/transnetv2.onnx (via Git LFS, matching the other ONNX models) for the new scene-detection path.
This commit is contained in:
+20
-1
@@ -30,9 +30,12 @@ trt_cache/
|
||||
# ORT pre-optimized model cache (generated on first run, provider-specific)
|
||||
ort_cache/
|
||||
|
||||
# Gallery JSON files (generated)
|
||||
# Gallery files (generated — HDF5 only, see src/gallery/gallery_store.cpp).
|
||||
# Legacy JSON galleries from before that switch are also excluded.
|
||||
gallery.json
|
||||
gallery_*.json
|
||||
gallery.h5
|
||||
gallery_*.h5
|
||||
|
||||
# Calibration cache (generated alongside a gallery, per-embedder)
|
||||
*.calib_cache.csv
|
||||
@@ -61,6 +64,22 @@ eval/probe/
|
||||
# Local reference repos kept for inspiration (each has its own .git)
|
||||
inspiration/
|
||||
|
||||
# experiments/ has its own nested .gitignore for HDF5 galleries/dumps/X-Ray
|
||||
# corpus (all pushed/pulled via scripts/artifacts/{push,pull}_artifacts.sh to
|
||||
# the Gitea generic package registry instead of committed).
|
||||
# Exception to the blanket *.json rule above: the committed placeholder for
|
||||
# experiments/file-lut.json (see experiments/.gitignore).
|
||||
!experiments/file-lut.template.json
|
||||
|
||||
# Site build output (mkdocs build). Rendered site is deployed to a
|
||||
# gitea-pages branch, never committed to a working branch.
|
||||
site/
|
||||
docs_site/
|
||||
|
||||
# Images staged into docs/ from the artifact registry at build time
|
||||
# (scripts/docs/build_site.sh) — not committed, pulled fresh on each build.
|
||||
docs/assets/
|
||||
|
||||
# Python
|
||||
__pycache__/
|
||||
*.pyc
|
||||
|
||||
@@ -46,10 +46,10 @@ Optional flags:
|
||||
## Models
|
||||
|
||||
The ONNX model weights live in `models/` (tracked via Git LFS):
|
||||
- `arcface_w600k_r50.onnx` — primary ArcFace embedder
|
||||
- `arcface_w600k_mbf.onnx`, `arcface_r18.onnx` — lighter alternatives
|
||||
- `LVFace-B_Glint360K.onnx` — LVFace embedder (ViT backbone, ICCV 2025), drop-in
|
||||
ArcFace-compatible: same `[N,3,112,112]` input and 512-d output (see below)
|
||||
- `LVFace-B_Glint360K.onnx` — LVFace embedder (ViT backbone, ICCV 2025), the default
|
||||
(best F1 in the rep4 model bake-off, see `docs/rep4-optimizer-results.md`)
|
||||
- `arcface_w600k_r50.onnx` — ArcFace embedder, previous default
|
||||
- `arcface_w600k_mbf.onnx`, `arcface_r18.onnx` — lighter ArcFace alternatives
|
||||
- `face_detection_yunet_2023mar.onnx` — YuNet face detector
|
||||
- `scrfd_500m_bnkps.onnx` — SCRFD face detector
|
||||
|
||||
|
||||
@@ -0,0 +1,33 @@
|
||||
# scene-actor-extraction
|
||||
|
||||
A face-recognition pipeline that finds when each actor appears on screen in a
|
||||
film or TV episode — built on [KPN++](https://gitea.tourolle.paris/dtourolle/KPN)
|
||||
(a C++20 Kahn Process Network library) for the detect → track → match → scene
|
||||
pipeline, with a Jellyfin-integrated gallery and an X-Ray-validated optimizer.
|
||||
|
||||
## Start here
|
||||
|
||||
- **[Rep4 model bake-off + threshold re-tune](rep4-optimizer-results.md)** — the
|
||||
current experiment log: model comparison, DE threshold tuning, held-out
|
||||
validation, and the visual failure-mode evidence (frozen-bbox ghost tracks).
|
||||
This is where the shipped `src/config.hpp` defaults come from.
|
||||
- **[Optimizer experiments (prior round)](optimizer-experiments.md)** — the
|
||||
earlier scene-union-metric tuning pass, superseded by the per-second metric
|
||||
used in rep4 but kept for the ground-truth/architecture background.
|
||||
- **[Service conversion (proposal)](service-conversion.md)** — design sketch
|
||||
for an idle-GPU Docker worker, not yet built.
|
||||
|
||||
## Reproducing the benchmarks
|
||||
|
||||
Gallery `.h5` files, embedding dumps, the X-Ray corpus, montage frame images,
|
||||
and DE trajectories are not committed to this repository — they're pushed to
|
||||
the Gitea package registry and pulled on demand:
|
||||
|
||||
```bash
|
||||
scripts/artifacts/pull_artifacts.sh galleries
|
||||
scripts/artifacts/pull_artifacts.sh experiment-data
|
||||
scripts/artifacts/pull_artifacts.sh montage-frames <film-slug>
|
||||
```
|
||||
|
||||
See `scripts/artifacts/push_artifacts.sh` for the upload side (requires a
|
||||
`GITEA_TOKEN` with package write scope).
|
||||
@@ -0,0 +1,117 @@
|
||||
# Threshold optimization against Amazon X-Ray — experiment log
|
||||
|
||||
Record of the July 2026 work that tuned the pipeline's recognition/tracking defaults
|
||||
against ground-truth per-scene actor presence, and the tooling built to do it.
|
||||
|
||||
## TL;DR — what changed
|
||||
|
||||
| knob | old default | new default | why |
|
||||
| ---- | ----------- | ----------- | --- |
|
||||
| `prob_threshold` | 0.99 | **0.76** | 0.99 was far too strict — halved recall for a fraction of a precision point. DE optimum, tightly converged. |
|
||||
| `extinction_sec` | 5.0 | **1.5** | Long extinction smears presence into later scenes → FPs. DE converged tightly low. |
|
||||
| `anneal_sec` | 10.0 | 10.0 (unchanged) | DE found it **insensitive** (F1 flat ±0.3pp across 3–26s) — kept the round default. |
|
||||
| `detector_conf` | 0.5 | 0.5 (unchanged) | Sweep showed raising it only trades recall for precision at a net F1 loss — near-threshold detections are real faces, not phantoms. |
|
||||
|
||||
Net effect on the 9-film benchmark (strict per-scene, augmented gallery):
|
||||
recall **58% → ~72%**, F1 **70% → ~76%**, precision ~85%, at no meaningful precision cost.
|
||||
|
||||
## Ground truth
|
||||
|
||||
Public scene-level **Amazon X-Ray** dataset (Zenodo DOI 10.5281/zenodo.17659734,
|
||||
CC-BY-4.0): per movie, `people.csv` (name_id/person/character), `scenes.csv`
|
||||
(scene/start/end ms), `people_in_scenes.csv`. Films matched to the library by an
|
||||
**authoritative Jellyfin ID join** (query `/Items?IncludeItemTypes=Movie&Fields=
|
||||
ProviderIds,Path`, join Imdb/Tmdb against X-Ray metadata) — NOT fuzzy title matching,
|
||||
which collides badly (TV episodes vs same-named films). 9 genuine films with source
|
||||
video on disk: Benny & Joon, Café Society, Downton Abbey: A New Era, Lord of War,
|
||||
Lovelace, The Many Saints of Newark, Scarface, Sound of Metal, Valerian.
|
||||
|
||||
## The scoring metric (evolved through review)
|
||||
|
||||
Comparison unit is the **X-Ray scene**, not sampled timepoints. For each scene
|
||||
`[start,end]`: predicted set = **union** of actors detected anywhere in the span;
|
||||
GT set = actors X-Ray lists for that scene. Per scene TP/FP/FN, then:
|
||||
|
||||
- **Precision: STRICT.** Any predicted actor not in the scene's X-Ray set is an FP,
|
||||
*including out-of-cast confusions* (no gallery∩cast masking). An earlier
|
||||
timepoint-sampled, cast-masked metric HID ~570 such FPs across 9 films and let the
|
||||
optimizer drive `prob_threshold` to the 0.50 floor — a metric artifact. Counting
|
||||
them is essential.
|
||||
- **Recall: FAIR.** FN counts only X-Ray cast members **who are in the gallery**. 67%
|
||||
of X-Ray cast (261/392) have no gallery reference embedding and can never be
|
||||
recognised — counting them as misses penalises coverage, not the threshold. Both
|
||||
`recall` (fair) and `recall_strict` (all) are reported.
|
||||
- **Aggregation:** per-scene F1 → **duration-weighted average within a movie** (long
|
||||
scenes count more) → **equal-weight mean across movies** (macro; each film counts
|
||||
the same regardless of length). This is the DE objective.
|
||||
|
||||
Implemented in `scripts/optimizer/scene_score.py`.
|
||||
|
||||
## The gallery coverage gap
|
||||
|
||||
Diagnosing low recall: only **131 of 392** X-Ray cast were in the gallery (33%). Every
|
||||
in-gallery actor HAD embeddings (gallery well-formed) — the gap was pure coverage.
|
||||
`scripts/optimizer/fetch_missing_actors.py` recovers missing actors:
|
||||
`nm-id → TMDB /find external_ids → /person/{id}/images → download → embed (sae_embed)`,
|
||||
with a `--wikidata` fallback (P345→P18 Commons photo).
|
||||
|
||||
- **TMDB recovered 143/261** (55%). 0 face-detection failures; the rest had no TMDB
|
||||
person (60) or no profile photo (58). Coverage 33% → **70%**.
|
||||
- **Wikidata fallback: 0/118** of the TMDB failures — only 4 even had a Commons photo,
|
||||
none yielded a detectable face. → **TheTVDB not worth pursuing**: these remaining
|
||||
actors are obscure enough that no image source covers them, AND (see below) most are
|
||||
off-camera anyway.
|
||||
|
||||
**Coverage vs detectability.** Adding references lifted recall (58→68% at fixed config)
|
||||
but modestly. Per-film drill-down (Lord of War: 12 actors recovered, only 1 had a
|
||||
detectable on-camera face) showed most missing cast are a **detectability gap** — X-Ray
|
||||
credits them as cast-in-scene (incl. off-camera/background), but their face never
|
||||
appears clearly for the pipeline to detect. This is a fundamental ceiling of a
|
||||
face-recognition pipeline vs X-Ray's presence semantics, not a fixable gap.
|
||||
|
||||
## Optimizer
|
||||
|
||||
`scripts/optimizer/optimize.py` — scipy `differential_evolution` over the knob space,
|
||||
each candidate = full replay of all films through the **real** C++ nodes (see the
|
||||
KPN replay architecture below) scored by the metric above. Global objective (one
|
||||
config for all films, not per-film).
|
||||
|
||||
**Convergence stability (augmented gallery, 233 evals):**
|
||||
|
||||
| knob | top-20 range | verdict |
|
||||
| ---- | ------------ | ------- |
|
||||
| `prob_threshold` | 0.69–0.83 (σ 0.05) | TIGHT — trust 0.76 |
|
||||
| `extinction_sec` | 1.0–2.2 (σ 0.33) | TIGHT — trust 1.5 |
|
||||
| `anneal_sec` | 3.1–26.3 (σ 6.4) | LOOSE — insensitive, not hard-coded |
|
||||
|
||||
F1 varied only 0.3pp across the top-20 → objective is flat near the optimum, so only
|
||||
the tightly-converged knobs were adopted as defaults.
|
||||
|
||||
## Replay architecture (how the sweep is cheap)
|
||||
|
||||
The optimizer never re-decodes video. `scene_analyze --dump-embeddings out.h5` runs the
|
||||
expensive half once (decode→detect→align→embed) and dumps per-frame face embeddings
|
||||
+ metadata to HDF5 (`scripts/optimizer/SCHEMA.md`). `scripts/optimizer/replay.py` then
|
||||
replays that dump through the **real** C++ `face_tracker → identity_matcher →
|
||||
scene_tracker` assembled in a Python KPN network (`sae_kpn` nanobind module), varying
|
||||
Config knobs freely — no GPU embedding, no decode. Verified BYTE-EXACT against
|
||||
`scene_analyze`'s own output. The dumps are gallery-independent, so testing the
|
||||
augmented gallery needed no re-dump. `detector_conf` is replayable UPWARD only (the
|
||||
dump floor is 0.5).
|
||||
|
||||
## Reproduce
|
||||
|
||||
```bash
|
||||
# 1. dump (once per film, needs video)
|
||||
scene_analyze --movie <f> --gallery gallery.json --dump-embeddings dump.h5 --fps 1
|
||||
# 2. build films manifest by Jellyfin ID join (see scripts/optimizer notes)
|
||||
# 3. optimize
|
||||
python scripts/optimizer/optimize.py --manifest films.json --gallery gallery.json \
|
||||
--params prob_threshold:0.5:0.999 anneal_sec:1:30 extinction_sec:1:15 \
|
||||
--popsize 8 --maxiter 20 --trajectory traj.jsonl --out opt.json
|
||||
# 4. score a fixed config / validate on a held-out set
|
||||
python scripts/optimizer/score_config.py --manifest heldout.json --gallery gallery.json \
|
||||
--config '{"prob_threshold":0.76,"extinction_sec":1.5,"anneal_sec":10}'
|
||||
```
|
||||
|
||||
See also memory: kpn-python-replay-optimizer, gallery-coverage-gap, xray-validation-*.
|
||||
@@ -0,0 +1,408 @@
|
||||
# Rep4 model bake-off + threshold re-tune — experiment log (2026-07-18/19)
|
||||
|
||||
Follow-on to `docs/optimizer-experiments.md`, which used an older, since-superseded
|
||||
scene-union metric. This round uses the **per-second** metric
|
||||
(`scripts/optimizer/second_score.py`) and answers three questions in one 16-run
|
||||
matrix: which embedding model is best, does cast-restriction cut misIDs, and does
|
||||
per-film gallery expansion help.
|
||||
|
||||
## Why this experiment, and what it actually delivered
|
||||
|
||||
Four goals going in, and an honest read on each after held-out validation (see
|
||||
below):
|
||||
|
||||
1. **Find the best default parameters to ship.** Partially delivered. The DE optimum
|
||||
generalizes *unevenly* — strong on 3 of 5 held-out films, badly broken on 2 (one
|
||||
with a 974-count misID blowup). The tuned values are shipped anyway (see
|
||||
Caveats) because they still beat the old defaults on average, but this is not a
|
||||
settled, film-agnostic optimum.
|
||||
2. **Find the best default model.** Delivered with more confidence. LVFace beat
|
||||
r50/r18/mbf across all 4 training combos, and nothing in held-out validation
|
||||
contradicts the model choice specifically — the held-out failures trace to
|
||||
`extinction_sec`/threshold interactions and gallery coverage, not the embedder.
|
||||
3. **Provide insight into how the application works.** The strongest, most durable
|
||||
output. Found and fixed a real teardown deadlock bug (100% reproducible, not the
|
||||
assumed rare GPU flake), established a real concurrency ceiling (8 parallel
|
||||
replays, not more), and found a real parameter interaction (a strict
|
||||
`prob_threshold` "earns" a longer extinction window before it starts hurting).
|
||||
4. **Demonstrate limitations.** Delivered, and reinforced hard by held-out
|
||||
validation — see the "Held-out validation" section below for concrete examples,
|
||||
including a screenshot of the matcher naming 15 actors, none correctly, on a
|
||||
completely blank title card.
|
||||
|
||||
## TL;DR — what changed in `src/config.hpp`
|
||||
|
||||
| knob | old default | new default | why |
|
||||
| ---- | ----------- | ----------- | --- |
|
||||
| `arcface_model` | `arcface_w600k_r50.onnx` | **`LVFace-B_Glint360K.onnx`** | Best F1 in the full-gallery bake-off (75.3% vs r50's 68.5%). LVFace was worth its size. |
|
||||
| `prob_threshold` | 0.76 | **0.754** | Re-tuned for LVFace + per-second metric. |
|
||||
| `extinction_sec` | 1.5 | **57.4** | Reverses the earlier "short is better" finding — see below. |
|
||||
| `anneal_sec` | 10.0 | **35.5** | Same reversal; previously thought insensitive. |
|
||||
| `expand_gallery` | false | **true** | Helped recall on the full (unrestricted) gallery for the winning model — opposite of the earlier assumption. |
|
||||
|
||||
These are the **`LVFace-B_Glint360K_full_exp`** winning values — the best result that
|
||||
uses only features already live in the running app (full gallery, no cast
|
||||
restriction; see below for why restricted mode isn't applied even though it scored
|
||||
higher).
|
||||
|
||||
## Why re-run at all
|
||||
|
||||
`docs/optimizer-experiments.md`'s scene-union metric hid out-of-cast false positives
|
||||
behind a gallery∩cast recall mask, and the earlier 9-film benchmark was scene-level
|
||||
(union over a whole X-Ray scene), not a fair per-timepoint comparison. SESSION_STATE
|
||||
flagged the old scene-metric bake-off numbers (R50≈LVFace≈MBF ~85%) as superseded.
|
||||
This round uses `second_score.py`: uniform per-second sampling, GT = X-Ray scene's
|
||||
cast at time *t*, pred = actors whose presence window covers *t*, FPI weighted 10×
|
||||
when the named actor isn't in the film's cast at all (true misID) vs. an in-cast
|
||||
timing slip. FN only counts gallery-known cast (fair recall — 67% of X-Ray cast have
|
||||
no reference embedding, see `gallery-coverage-gap` memory).
|
||||
|
||||
## The deadlock that was blocking all of this
|
||||
|
||||
Every replay in this line of work goes through `scripts/optimizer/replay.py`, which
|
||||
runs the real C++ tracker/matcher/scene_tracker nodes inside a Python-assembled KPN
|
||||
network. Before this session, every subprocess replay **timed out at 45s, 100% of
|
||||
the time** — not the documented ~20-30% ROCm rocBLAS-GEMM driver flake, but a plain
|
||||
logic bug: `replay.py`'s CLI called `replay(net, ..., stop=False)` to *skip*
|
||||
`net.stop()` (trying to dodge the GEMM deadlock), planning to `os._exit(0)`
|
||||
immediately after. But:
|
||||
|
||||
- `PyNode::stop()` (`external/KPN/include/kpn/python/bindings.hpp`) is the *only*
|
||||
code that sets `stop_flag_ = true` before joining the node's worker thread.
|
||||
- The source node's `run_loop()` has `while (!stop_flag_)` as its only exit
|
||||
condition (it has no input channels, so it never sees a channel-closed signal
|
||||
either).
|
||||
- Skipping `stop()` meant `stop_flag_` never became true. When `replay()` returned,
|
||||
its local `net` went out of scope immediately, running `~PyNetwork` → `~PyNode` →
|
||||
`thread_.join()` **synchronously inside `replay()`'s own call frame** — before
|
||||
`main()` ever got control back to run `os._exit(0)`.
|
||||
|
||||
Root-caused via `gdb -p <pid> -batch -ex "thread apply all bt"` on a hung process:
|
||||
the main thread was stuck in `~PyNode`'s `jthread::join()`; the worker thread was in
|
||||
an ordinary `time.sleep()` inside the Python source callback, waiting for a stop
|
||||
signal that was never sent. The two HSA `kfd_wait_on_events` threads visible in the
|
||||
same trace are normal ROCm runtime housekeeping, not evidence of a wedged GPU kernel.
|
||||
|
||||
**Fix:** `replay.py` now calls `replay(..., stop=True)` (the removed `stop=False` +
|
||||
`os._exit` workaround was actively harmful). Verified 3/3 clean runs at ~8s each
|
||||
(down from a guaranteed 45s timeout), and a full DE sweep producing real, sensible
|
||||
F1/precision/recall instead of flat 0.0%.
|
||||
|
||||
## Concurrency tuning
|
||||
|
||||
With the deadlock fixed, `optimize.py` was extended with DE-level parallelism —
|
||||
`differential_evolution(..., workers=ThreadPoolExecutor.map)` — so multiple
|
||||
population candidates evaluate concurrently, each spawning its own per-film replay
|
||||
subprocesses (`REPLAY_WORKERS`). Total concurrent GPU replay processes ≈
|
||||
`DE_WORKERS × REPLAY_WORKERS`.
|
||||
|
||||
| concurrent replays | result |
|
||||
| --- | --- |
|
||||
| 3 (`REPLAY_WORKERS=3`, no DE parallelism) | baseline, GPU underutilised |
|
||||
| 6 (`DE_WORKERS=2 × REPLAY_WORKERS=3`) | clean, real scores, ~1 isolated timeout per run |
|
||||
| 8 (`DE_WORKERS=2 × REPLAY_WORKERS=4`, 4-film manifest) | clean, real scores |
|
||||
| 9 (`DE_WORKERS=3 × REPLAY_WORKERS=3`) | **broken** — every replay blew past the 45s timeout, all scores silently degraded to 0.0% |
|
||||
|
||||
9 concurrent replays looks like valid output (well-formed JSON, a real number) while
|
||||
actually being garbage — a dangerous failure mode, not a crash. **8 concurrent is the
|
||||
practical ceiling** on this GPU (gfx1100) for this workload. The matrix ran at
|
||||
`REPLAY_WORKERS=4 DE_WORKERS=2`.
|
||||
|
||||
## Training films (rep4) and validation set
|
||||
|
||||
9 films total have dumped embeddings across all 4 models. 4 were used for
|
||||
optimization (rep4), leaving 5 held out for validation:
|
||||
|
||||
- **Lord of War** (64-cast, "clean")
|
||||
- **Scarface** (67-cast, "ensemble/lookalike")
|
||||
- **Sound of Metal** (14-cast, "high gallery-coverage")
|
||||
- **Café Society** (62-cast, added this round — similar ensemble size to Scarface but
|
||||
different genre/lighting; picked to add diversity, not genre-overlap, over
|
||||
Downton Abbey or The Many Saints of Newark)
|
||||
|
||||
Held out: Benny & Joon, Downton Abbey: A New Era, Lovelace, The Many Saints of
|
||||
Newark, Valerian and the City of a Thousand Planets. The rep4 numbers below are
|
||||
training-set fit — see "Held-out validation" further down for the real
|
||||
generalization test.
|
||||
|
||||
## Search space and DE settings
|
||||
|
||||
`popsize=10, maxiter=15` (3 params → ≤480 evals/combo ceiling; DE's `tol` convergence
|
||||
usually stops earlier). `anneal_sec`/`extinction_sec` bounds were **widened from
|
||||
1–30/1–15 to 1–60/1–60 mid-run** (see below) — the 4 `arcface_w600k_r50` combos
|
||||
finished before the widening and still use the old, narrower bounds, so they are
|
||||
**not directly comparable** to the other 12 on those two params. Re-running r50 with
|
||||
the wider bounds was deferred (diminishing-returns judgment call, not yet done).
|
||||
|
||||
## Results — all 16 combos (4 models × {full, restricted} × {expand, noexp})
|
||||
|
||||
Ranked by F1. `misid` = FPI_misid, count of true wrong-actor identifications (an
|
||||
actor named who isn't in the film's cast at all) — distinct from `FPI`, which
|
||||
includes in-cast timing slips.
|
||||
|
||||
| combo | F1 | P | R | TPI | FPI | misid | FN |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| LVFace-B_Glint360K_restricted_exp | **78.3%** | 91.0% | 68.9% | 42830 | 3782 | 60 | 19492 |
|
||||
| LVFace-B_Glint360K_restricted_noexp | 76.7% | 91.5% | 66.2% | 41149 | 3400 | 59 | 21173 |
|
||||
| arcface_w600k_mbf_restricted_exp | 76.5% | 90.7% | 66.3% | 41270 | 4431 | **0** | 21052 |
|
||||
| arcface_r18_restricted_exp | 75.5% | 87.6% | 66.5% | 41399 | 5666 | 60 | 20923 |
|
||||
| **LVFace-B_Glint360K_full_exp** | **75.3%** | 89.7% | 65.4% | 47757 | 3407 | 232 | 26966 |
|
||||
| arcface_w600k_mbf_restricted_noexp | 75.0% | 91.1% | 63.9% | 39752 | 3465 | 60 | 22570 |
|
||||
| arcface_w600k_mbf_full_noexp | 74.2% | 87.4% | 64.4% | 12645 | 1312 | 57 | 6985 |
|
||||
| arcface_r18_restricted_noexp | 73.5% | 91.3% | 61.7% | 38299 | 3220 | 60 | 24023 |
|
||||
| LVFace-B_Glint360K_full_noexp | 72.4% | 94.2% | 58.9% | 27077 | 1725 | **0** | 19506 |
|
||||
| arcface_w600k_mbf_full_exp | 72.0% | 87.7% | 61.4% | 39875 | 3729 | 240 | 26338 |
|
||||
| arcface_w600k_r50_full_noexp † | 71.6% | 96.7% | 56.9% | 22471 | 361 | 45 | 17012 |
|
||||
| arcface_w600k_r50_restricted_exp † | 71.1% | 96.5% | 56.4% | 34954 | 1119 | 15 | 27368 |
|
||||
| arcface_w600k_r50_restricted_noexp † | 69.2% | 97.9% | 53.6% | 21146 | 327 | 15 | 18337 |
|
||||
| arcface_r18_full_exp | 69.1% | 87.6% | 57.7% | 37342 | 3119 | 242 | 28871 |
|
||||
| arcface_w600k_r50_full_exp † | 68.5% | 94.0% | 54.1% | 34982 | 903 | 150 | 31231 |
|
||||
| arcface_r18_full_noexp | 66.6% | 91.3% | 53.1% | 34314 | 2362 | 107 | 31899 |
|
||||
|
||||
† old, narrower anneal/extinction bounds (see above) — not directly comparable to
|
||||
the other 12 on those two params.
|
||||
|
||||
## Calibration curves — discriminative power, independent of the threshold
|
||||
|
||||
Each model's gallery carries a fitted Platt sigmoid `P(match | sim) = σ(a·sim + b)`
|
||||
(embedded directly in the gallery HDF5, see `src/gallery/gallery_calibration.hpp`).
|
||||
Plotting all four side by side shows discriminative power directly, independent of
|
||||
whatever `prob_threshold` a particular run happened to use:
|
||||
|
||||
|
||||
|
||||
LVFace-B has both the steepest curve (`a=17.7`, vs. 15.3–16.2 for the ArcFace
|
||||
variants) and the lowest P=0.5 decision boundary (similarity 0.23 vs. 0.27–0.31) —
|
||||
it separates same-actor from different-actor pairs more confidently at a lower
|
||||
similarity, consistent with it winning the full-gallery F1 comparison below.
|
||||
Generated by `scripts/docs/calibration_chart.py` (requires each gallery to have
|
||||
been calibrated at least once — run any replay against it first).
|
||||
|
||||
## Two effects in isolation: gallery scope, and pose expansion
|
||||
|
||||
The matrix crosses two independent variables — averaging across all 4 models
|
||||
isolates each one from model choice:
|
||||
|
||||
**Gallery scope (whole 2418-actor gallery vs. restricted to the film's credited
|
||||
cast)** — averaged over both expansion settings and all 4 models:
|
||||
|
||||
| scope | F1 | P | R | total misID (16 evals→8 each) |
|
||||
|---|---|---|---|---|
|
||||
| full | 71.2% | 91.1% | 59.0% | 1073 |
|
||||
| **restricted** | **74.5%** | 92.2% | **62.9%** | **329** |
|
||||
|
||||
Restriction wins outright on every axis — not a precision/recall trade, a clean
|
||||
win: **+3.3pp F1, +3.9pp recall, and less than a third the total misIDs.** Fewer
|
||||
candidates in the matcher's search space means fewer opportunities for a
|
||||
look-alike false match, and (per the recall gain) doesn't cost real detections.
|
||||
This is the single cleanest signal in the whole matrix — stronger than the model
|
||||
choice itself — which is exactly why cast-restriction becoming a real runtime
|
||||
feature (not just an optimizer trick) is the top item in Caveats below.
|
||||
|
||||
**Pose expansion (promoting a confidently-identified track's novel-pose views into
|
||||
a per-film gallery annex — `track_gallery.hpp`)** is smaller and interacts with
|
||||
scope rather than acting independently:
|
||||
|
||||
| scope | expansion | F1 | R | misID |
|
||||
|---|---|---|---|---|
|
||||
| full | off | 71.2% | 58.3% | 209 |
|
||||
| full | **on** | 71.2% | 59.7% | **864** |
|
||||
| restricted | off | 73.6% | 61.3% | 194 |
|
||||
| restricted | **on** | **75.4%** | **64.5%** | 135 |
|
||||
|
||||
In **restricted** mode, expansion is a clean win (+1.8pp F1, +3.2pp recall, misID
|
||||
actually *drops*) — the annex only ever competes against the film's own ~15-actor
|
||||
cast, so a "confidently identified, new pose" view is unlikely to be mistaken for
|
||||
someone else. In **full** mode, expansion buys essentially nothing on F1 (71.2% →
|
||||
71.2%, recall +1.4pp) while **quadrupling misIDs** (209 → 864): a novel-pose view
|
||||
promoted into the annex now competes against the whole 2418-actor gallery, so a
|
||||
"confident" identity is confident against the wrong universe of candidates — the
|
||||
expansion mechanism is "learning" a pose correctly, but the enlarged evidence pool
|
||||
makes it easier for that learned pose to look like a plausible match for a
|
||||
different actor. **Practical takeaway: gallery expansion should be paired with
|
||||
cast restriction, not used on the full gallery** — the version currently shipped
|
||||
as default (`full_exp`, see TL;DR) sits in the worse of these four cells for this
|
||||
specific knob, even though it's the best available combo without cast-restriction
|
||||
support in the app yet (see Caveats).
|
||||
|
||||
## What the data says
|
||||
|
||||
- **LVFace was worth its size.** It wins full-gallery mode outright (75.3% vs r50's
|
||||
68.5%, r18's 69.1%, mbf's 72.0%) with the highest recall of any full-mode combo —
|
||||
the earlier scene-union-metric conclusion ("not worth it") doesn't survive the
|
||||
better metric.
|
||||
- **Cast-restriction is a consistent, broad win.** Every model's best combo is
|
||||
`restricted`. It isn't just precision-safe: `arcface_w600k_mbf_restricted_exp` and
|
||||
`LVFace-B_Glint360K_full_noexp` both hit **misid=0** — zero true wrong-actor
|
||||
identifications. But restriction is an **offline optimizer technique, not a live
|
||||
app feature** — it pre-filters each film's gallery to its Jellyfin-credited cast
|
||||
before the matcher ever runs; there's no runtime "restrict to this film's cast"
|
||||
switch in the app today. Implementing it for real is future work, tracked
|
||||
separately from this defaults update.
|
||||
- **Gallery expansion (`expand_gallery`) is mode-dependent.** It helps on
|
||||
`restricted` galleries (smaller, so novel-pose promotion adds real signal) and on
|
||||
LVFace's full gallery, but **hurts** r50 and mbf in full mode (compare
|
||||
`arcface_w600k_r50_full_exp` 68.5% vs `full_noexp` 71.6%). Don't assume it's a free
|
||||
win — model- and mode-dependent.
|
||||
- **arcface_r18 (smallest/cheapest) is last across all 4 modes** — model capacity
|
||||
matters here, this isn't just parameter-count padding.
|
||||
- **`anneal_sec`/`extinction_sec` kept pinning at the search ceiling.** With the
|
||||
original 1–30/1–15 bounds, 3 of 4 r50 combos landed at ~93-98% of the upper bound.
|
||||
Widened to 1–60/1–60 mid-run (after the r50 combos had already finished) — every
|
||||
subsequent combo's best config landed at ~90%+ of the *new* ceiling too (e.g. the
|
||||
LVFace winner: `ann=59.2, ext=59.2`, both ~99% of 60). The likely mechanism: a
|
||||
strict `prob_threshold` "earns" a long extinction/anneal window — once false
|
||||
matches are rare, a long window just bridges real presence gaps (occlusion, turned
|
||||
face) instead of smearing false positives into later scenes, which is what made
|
||||
short windows look better under the old, laxer thresholds. **Open question, not
|
||||
resolved**: does this keep climbing past 60s, or does it actually plateau there?
|
||||
Decided not to chase further this round (diminishing-returns judgment call) — flag
|
||||
for a future sweep if it matters.
|
||||
|
||||
## Held-out validation — the number that actually matters
|
||||
|
||||
The 16-combo matrix above is training-set fit. This is the real test: the shipped
|
||||
config (`LVFace-B_Glint360K_full_exp` — `prob_threshold=0.754, anneal_sec=35.5,
|
||||
extinction_sec=57.4, expand_gallery=true`) replayed against the **5 films never seen
|
||||
by the optimizer** (Benny & Joon, Downton Abbey: A New Era, Lovelace, The Many
|
||||
Saints of Newark, Valerian and the City of a Thousand Planets), scored the same way.
|
||||
|
||||
| film | F1 | P | R | agree | TPI | FPI | misid | FN |
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
| Benny & Joon | 83.0% | 89.1% | 77.7% | 72.4% | 15125 | 1846 | 0 | 4337 |
|
||||
| Lovelace | 77.5% | 90.3% | 67.9% | 72.1% | 14990 | 1085 | 58 | 7085 |
|
||||
| Valerian and the City of a Thousand Planets | 74.1% | 97.1% | 60.0% | 58.8% | 18663 | 548 | 0 | 12467 |
|
||||
| Downton Abbey: A New Era | 56.2% | 97.8% | 39.4% | 40.6% | 52027 | 1173 | 0 | 80084 |
|
||||
| **The Many Saints of Newark** | **46.3%** | **54.7%** | 40.1% | 37.0% | 15922 | 4394 | **974** | 23791 |
|
||||
| **macro average (5 films)** | **67.4%** | 85.8% | 57.0% | 56.2% | 116727 | 9046 | 1032 | 127764 |
|
||||
|
||||
**67.4% held out vs. 75.3% on training** — an ~8pp drop, and a much more informative
|
||||
number than the training-set F1 alone: a **37pp spread between best and worst film**
|
||||
(83.0% vs 46.3%). The config does not generalize uniformly.
|
||||
|
||||
Two films are outright failure cases, and rendering bounding boxes + names on the
|
||||
extracted frames (`replay.py --raw-out` + `dump_error_frames.py --raw`, see
|
||||
Reproduce) turned what looked like a same-scene misidentification into something
|
||||
more precise and more damning:
|
||||
|
||||
- **The Many Saints of Newark** (mob-family drama, picked as a training-adjacent
|
||||
genre test) has **974 true misIDs** — far more than any training combo saw at any
|
||||
setting. The annotated frame below shows the same mechanism as Downton Abbey,
|
||||
at smaller scale: **"Jon Bernthal 100%", "Joey Diaz 100%", and "Billy Magnussen
|
||||
100%" are all frozen boxes over empty background — a blurred wall, hanging plates —
|
||||
with no face in them at all.** Only one real face in frame has a box, and it
|
||||
carries a *second*, colliding label ("Leslie Odom Jr." and "Michael Gandolfini"
|
||||
both at high confidence on the same box) — likely two tracks whose frozen bboxes
|
||||
happen to overlap.
|
||||
|
||||
|
||||
*`experiments/results/holdout/frames/many_saints/fpi/fpi_t03543.jpg`*
|
||||
|
||||
- **Downton Abbey: A New Era** (large ensemble, 36-cast) has high precision (97.8%)
|
||||
but recall collapses to 39.4% (FN=80084, by far the largest of the 5). The frame
|
||||
below is the starkest evidence in this whole experiment: **the matcher named 15
|
||||
actors — all of them wrong — over a completely blank closing title card with no
|
||||
faces on screen at all.**
|
||||
|
||||
|
||||
*`experiments/results/holdout/frames/downton_abbey/fpi/fpi_t07242.jpg`*
|
||||
|
||||
Both are the same mechanism, verified directly against the HDF5 dump and the raw
|
||||
per-frame stream (not just inferred from the screenshot): at the Downton Abbey
|
||||
title card (t=7242), the dump's own `face_count` is **0 from t≈7240 onward** — no
|
||||
detector output at all, confirmed independently of the pipeline. Yet all 15 "wrong"
|
||||
actors are still marked visible, each with the *exact same bbox, unchanged to the
|
||||
pixel*, repeated every single frame back to t=7222 (verified for Hugh Bonneville:
|
||||
`(1743.2, 0.0, 171.3, 317.8)` at every sampled second from 7222 through 7279+). That
|
||||
is `SceneTrackerFunc`'s `active_[actor_idx].last_bbox` (`scene_tracker_node.hpp`)
|
||||
being re-emitted unchanged — **this is the extinction state machine working exactly
|
||||
as coded**, not a bug in the logic. The film cuts from a packed group shot straight
|
||||
into ~40+ seconds of blank titles/credits with zero faces, and `extinction_sec=57.4`
|
||||
is comfortably long enough to bridge that entire gap without expiring, so the
|
||||
tracker faithfully keeps reporting "last known position" for a cast that is no
|
||||
longer on screen at all.
|
||||
|
||||
This reframes the "long extinction window wins" DE-search pattern (see above): it
|
||||
isn't unambiguously good. It buys recall by bridging real gaps (occlusion, turned
|
||||
face) in some films, but on others — specifically, hard cuts into long faceless
|
||||
footage — it manufactures a frozen-bbox ghost the tracker has no way to verify,
|
||||
precisely the failure mode the *original* short-extinction-window default (`1.5s`)
|
||||
was chosen to avoid. The training-set films apparently didn't have a long enough
|
||||
faceless stretch after a confirmed identity to expose this; the held-out set did.
|
||||
|
||||
Frames for all three films (`benny_joon`, `many_saints`, `downton_abbey` — one strong
|
||||
performer, two failure cases) are under `experiments/results/holdout/frames/`, each
|
||||
with a `manifest.json` listing the bucket (`best`/`fpi`/`fn`), timestamp, and
|
||||
predicted vs. ground-truth actors for every dumped frame. Frames are annotated with
|
||||
bounding boxes + name/confidence (green = identified, orange = unknown), matching
|
||||
`debug_renderer_node.hpp`'s colour convention. Generated by
|
||||
`scripts/optimizer/dump_error_frames.py --raw <replay.py --raw-out output>` (see
|
||||
Reproduce).
|
||||
|
||||
`dump_error_frames.py --interval-sec 600` also supports a per-N-second sweep
|
||||
instead of the fixed best/fpi/fn buckets: one best (highest Jaccard) and one worst
|
||||
(lowest Jaccard) frame per 10-minute window across the whole film, e.g.
|
||||
`experiments/results/holdout/frames/many_saints_intervals/` (13 windows × 2 = 26
|
||||
frames for the ~2h Many Saints runtime) — a way to sample "how are we doing" evenly
|
||||
across a film's runtime rather than only at its most extreme seconds.
|
||||
|
||||
## Caveats / what this is not
|
||||
|
||||
- **r50's 4 combos used the old, narrower search bounds** and aren't fully
|
||||
comparable to the other 12 on `anneal_sec`/`extinction_sec`.
|
||||
- **The applied defaults use `full_exp`, not the higher-scoring `restricted_exp`**,
|
||||
because cast-restriction isn't a real runtime feature yet (see above). The
|
||||
78.3% F1 number is not what the shipped defaults will produce — 75.3% is.
|
||||
- **`full_exp` is the best full-gallery combo, but not the safest.** Per the
|
||||
isolated-effects analysis above, `expand_gallery=true` only cleanly pays off
|
||||
when paired with cast-restriction; on the full gallery it's flat on F1 while
|
||||
~4x-ing misIDs (209→864, averaged across models). `full_noexp` scores lower
|
||||
(72.4% vs 75.3% for LVFace) but with **zero** true misIDs and higher precision
|
||||
(94.2% vs 89.7%). Kept `full_exp` as shipped since it's the highest-F1 option
|
||||
available without cast-restriction, but this is a real F1-vs-safety trade, not
|
||||
a strictly-better choice — worth revisiting if misID rate matters more than
|
||||
the last few points of F1 for a given deployment.
|
||||
- **Switching the default model is an operational change, not just a config tweak**:
|
||||
any existing gallery built from r50 embeddings is incompatible with LVFace
|
||||
embeddings and needs rebuilding.
|
||||
|
||||
## Reproduce
|
||||
|
||||
```bash
|
||||
# 4-film matrix, all 4 models × 2 modes × 2 expansion settings
|
||||
bash experiments/run_rep4_subprocess.sh
|
||||
|
||||
# single combo
|
||||
SAE_EXPAND=1 REPLAY_WORKERS=4 DE_WORKERS=2 python3 scripts/optimizer/optimize.py \
|
||||
--manifest experiments/manifests/rep4_LVFace-B_Glint360K_full.json \
|
||||
--gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5 \
|
||||
--params prob_threshold:0.5:0.999 anneal_sec:1:60 extinction_sec:1:60 \
|
||||
--popsize 10 --maxiter 15 --trajectory traj.jsonl --out best.json
|
||||
|
||||
# replay the shipped config against a held-out film — --raw-out is needed to draw
|
||||
# bboxes later (the merged pred.json has no per-frame bbox, only actor windows)
|
||||
python3 scripts/optimizer/replay.py \
|
||||
--dump experiments/dumps/LVFace-B_Glint360K/dump_<slug>.h5 \
|
||||
--gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5 \
|
||||
--out pred.json --raw-out raw.jsonl --prob-threshold 0.754 --anneal-sec 35.54 \
|
||||
--extinction-sec 57.43 --expand-gallery
|
||||
|
||||
# dump example frames (best-agreement / FPI / FN) for visual inspection, annotated
|
||||
# with bounding boxes + names (--raw is optional; omit for unannotated frames)
|
||||
python3 scripts/optimizer/dump_error_frames.py \
|
||||
--pred pred.json --raw raw.jsonl --xray experiments/xray/.../<xray_dir> \
|
||||
--movie "<path to source video>" \
|
||||
--gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5 \
|
||||
--out-dir experiments/results/holdout/frames/<name> --n-per-bucket 4
|
||||
|
||||
# or: one best + one worst frame per 10-minute window across the whole film
|
||||
python3 scripts/optimizer/dump_error_frames.py \
|
||||
--pred pred.json --raw raw.jsonl --xray experiments/xray/.../<xray_dir> \
|
||||
--movie "<path to source video>" \
|
||||
--gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5 \
|
||||
--out-dir experiments/results/holdout/frames/<name>_intervals --interval-sec 600
|
||||
```
|
||||
|
||||
See also: `docs/optimizer-experiments.md` (prior round, superseded metric),
|
||||
`experiments/SESSION_STATE.md`, and memory: kpn-python-replay-optimizer,
|
||||
gallery-coverage-gap, xray-validation-results, per-scene-presence-eval-design.
|
||||
@@ -0,0 +1,215 @@
|
||||
# Conversion to service — a native idle-GPU worker
|
||||
|
||||
Status: **design / proposal**. Nothing here is built yet.
|
||||
|
||||
## The idea
|
||||
|
||||
Turn the CLI tools into a **turnkey batch worker that uses the machine's idle
|
||||
GPU**: it analyses newly-added Jellyfin media when you're not using the computer
|
||||
(screen locked), and stops the instant you come back. It's an overnight job on
|
||||
your own Linux box.
|
||||
|
||||
**No Docker.** This runs on your own machine with your own drivers, so a container
|
||||
buys little and costs a lot: GPU passthrough (nvidia-container-toolkit, or
|
||||
`/dev/kfd`+`/dev/dri`+`video` group for ROCm) is the single most fragile part of a
|
||||
containerised setup, and it exists *only* because of the container. Natively, the
|
||||
GPU just works with the drivers you already have, and the media paths Jellyfin
|
||||
reports are just real paths — no re-mounting. So we ship a **native installer**
|
||||
instead of an image builder.
|
||||
|
||||
Two deliverables:
|
||||
|
||||
1. **An installer** — `scripts/build_install.py`. Detects your distro, ensures the
|
||||
GPU/build dependencies are present (via `dnf`/`pacman`), compiles `scene_analyze`
|
||||
for your GPU, and installs the binary + Python glue + two systemd **user**
|
||||
units under `~/.local`.
|
||||
2. **A screen-lock gate** — one of those systemd units watches logind lock/unlock
|
||||
and starts/stops the worker. Lock → analyse. Unlock → stop.
|
||||
|
||||
## What already exists (reuse, don't rebuild)
|
||||
|
||||
The processing loop is already implemented — this is packaging, building, and
|
||||
lock-gating, not new pipeline logic.
|
||||
|
||||
| Piece | Where | What it does |
|
||||
|---|---|---|
|
||||
| Analysis engine | `build/scene_analyze` | Video → face detect/align/embed → gallery match → result JSON |
|
||||
| Backend selection | `CMakeLists.txt` (`SAE_INFERENCE_BACKEND`, `SAE_GEMM_BACKEND`) | ORT/TRT + ROCm/CUDA, chosen **at build time** |
|
||||
| New-media queue | JRay plugin → `GET /Plugins/JRay/Tasks/Pending` | Backlog of items with no results yet |
|
||||
| Worker loop | `run_from_jellyfin.py --worker` | Poll Pending → run `scene_analyze` → push results |
|
||||
| Result push | `PUT /Plugins/JRay/Items/{id}/Truth` | Stores per-actor scene windows back in Jellyfin |
|
||||
| Incremental gallery | `make_jellyfin_gallery.py --merge` | Embeds only cast not already in the gallery |
|
||||
| Secrets loader | `.env` via `sae_env.py` | `JELLYFIN_URL`, `JELLYFIN_API_KEY`, `TMDB_API_KEY` |
|
||||
|
||||
## Installer config
|
||||
|
||||
One file. Build-time settings (fixed when we compile) vs. run-time settings (in the
|
||||
worker's `.env`, editable without recompiling).
|
||||
|
||||
```yaml
|
||||
# install.yaml — consumed by scripts/build_install.py
|
||||
|
||||
platform: nvidia # nvidia | amd | cpu → picks the cmake backend
|
||||
model:
|
||||
arcface: LVFace-B_Glint360K.onnx # embedder compiled against; gallery MUST match
|
||||
schedule:
|
||||
gallery_scan_interval: 24h # incremental --merge cadence; 0 disables the scanner
|
||||
prefix: ~/.local # install root (bin, share, systemd user units)
|
||||
|
||||
# runtime (written to the worker .env, not compiled in):
|
||||
runtime:
|
||||
jellyfin_url: http://localhost:8096
|
||||
# JELLYFIN_API_KEY / TMDB_API_KEY are filled into .env by hand after install
|
||||
```
|
||||
|
||||
**Secrets never go in the repo or a build artifact** — the installer writes a
|
||||
`.env` under the install prefix with blanks for the keys, and you fill them in
|
||||
once. `sae_env.py` already loads it.
|
||||
|
||||
**Model ⇄ gallery coupling (guard, don't just document):** embeddings from
|
||||
different recognition models aren't interchangeable. We compile against one
|
||||
embedder; the gallery must be built with the same one. Stamp the embedder name
|
||||
into `gallery.json`, and have the worker **refuse to start** if the gallery's
|
||||
embedder ≠ the configured `model.arcface`, rather than silently mismatching.
|
||||
|
||||
## Dependencies via the system package manager
|
||||
|
||||
The heavy build/runtime deps (OpenCV, ffmpeg, the GPU stack) are best provided by
|
||||
the distro, not vendored. The installer ships a per-distro dependency list and
|
||||
either installs them or prints the exact command. Targets: **Fedora (dnf)** and
|
||||
**Arch (pacman)** first.
|
||||
|
||||
| Dependency | Fedora (dnf) | Arch (pacman) |
|
||||
|---|---|---|
|
||||
| OpenCV | `opencv-devel` | `opencv` |
|
||||
| ffmpeg | `ffmpeg-free`/`ffmpeg` (RPM Fusion) | `ffmpeg` |
|
||||
| CMake / toolchain | `cmake gcc-c++` | `cmake gcc` |
|
||||
| CUDA + TensorRT (nvidia) | NVIDIA CUDA repo + `libnvinfer-*` | `cuda`, `tensorrt` |
|
||||
| ROCm (amd) | `rocm-hip-sdk` / `rocblas-devel` | `rocm-hip-sdk`, `rocblas` |
|
||||
| ONNX Runtime | **not packaged** — installer fetches a pinned release tarball into the prefix | AUR `onnxruntime` (or same pinned-tarball fallback) |
|
||||
|
||||
So the flow is: **detect distro → check each package → install via the native
|
||||
manager (or print `sudo dnf install …` / `sudo pacman -S …`)**, with ONNX Runtime
|
||||
as the one known gap the installer fills itself (a pinned upstream release
|
||||
extracted under the install prefix, so it doesn't depend on a system package that
|
||||
may not exist). CUDA/ROCm being present is *assumed* — you already run a GPU
|
||||
desktop; the installer verifies and points you at the vendor repo if not.
|
||||
|
||||
## What `build_install.py` does
|
||||
|
||||
```
|
||||
build_install.py install.yaml
|
||||
│
|
||||
├─ detect distro (dnf vs pacman) and platform from config
|
||||
├─ ensure deps: install via manager, or print the exact command; fetch ONNX Runtime if needed
|
||||
├─ cmake + build scene_analyze with the platform's backend flags:
|
||||
│ nvidia → -DSAE_INFERENCE_BACKEND=TRT -DSAE_GEMM_BACKEND=CUDA
|
||||
│ amd → -DSAE_INFERENCE_BACKEND=ORT -DSAE_GEMM_BACKEND=ROCM
|
||||
│ cpu → -DSAE_INFERENCE_BACKEND=ORT (CPU EP; slow, for smoke tests)
|
||||
├─ install into <prefix>:
|
||||
│ bin/sae-scene-analyze the compiled binary
|
||||
│ share/sae-worker/ Python glue + a venv (requests, etc.), models/
|
||||
│ share/sae-worker/.env runtime config (keys blank, url from config)
|
||||
├─ install systemd --user units:
|
||||
│ sae-worker.service runs the worker + gallery-scan supervisor
|
||||
│ sae-lock-gate.service watches logind lock/unlock, start/stops the worker
|
||||
└─ print next steps (edit .env, `systemctl --user enable --now sae-lock-gate`)
|
||||
```
|
||||
|
||||
## The worker service (supervisor)
|
||||
|
||||
`sae-worker.service` runs a small Python supervisor as its main process:
|
||||
|
||||
- starts the **worker loop** (`run_from_jellyfin.py --worker`) — the hot path,
|
||||
- starts a **gallery-scan timer** — sleeps `gallery_scan_interval`, runs
|
||||
`make_jellyfin_gallery.py --merge`, repeats,
|
||||
- exits cleanly on SIGTERM (see re-queue below).
|
||||
|
||||
## The lock gate
|
||||
|
||||
`sae-lock-gate.service` runs a tiny watcher that subscribes to logind
|
||||
lock/unlock signals and drives the worker service:
|
||||
|
||||
```
|
||||
screen locks → systemctl --user start sae-worker.service
|
||||
screen unlocks → systemctl --user stop sae-worker.service (SIGTERM)
|
||||
```
|
||||
|
||||
**Screen-lock is the only signal — deliberately.** We don't also gate on GPU/CPU
|
||||
load, because our own worker *is* the load: a load threshold would form a feedback
|
||||
loop (worker starts → GPU spikes → threshold trips → worker stops → load drops →
|
||||
restart → …). Lock state is external to what the worker does, so it can't
|
||||
oscillate.
|
||||
|
||||
Signal source is desktop-dependent: logind `Lock`/`Unlock` (GNOME/KDE via
|
||||
`loginctl`/D-Bus) covers most setups; a `swayidle`/`xss-lock` hook is the fallback
|
||||
for wlroots/X-only compositors. The installer picks based on what's present.
|
||||
|
||||
## On resume: hard stop + re-queue (it's free)
|
||||
|
||||
Stopping the worker mid-analysis costs nothing to reschedule, because of how the
|
||||
JRay queue works: **an item only leaves `/Tasks/Pending` once its results are
|
||||
pushed** (`push_truth`). A worker stopped mid-`scene_analyze` simply leaves that
|
||||
item Pending — next lock picks it up again. No re-queue bookkeeping.
|
||||
|
||||
Two small correctness requirements (the only worker changes needed):
|
||||
|
||||
1. **Never push a partial result.** Already true — `push_truth` runs only after
|
||||
`scene_analyze` returns; a killed run pushes nothing. ✓ (keep it that way).
|
||||
2. **Clean up on signal.** `process_item` writes a temp filtered-gallery file and
|
||||
unlinks it in a `finally`; a SIGKILL skips `finally`. Fix: write temps under a
|
||||
dir the worker wipes on start, and/or a SIGTERM handler that unlinks before
|
||||
exit. Minor.
|
||||
|
||||
Accepted trade-off: a partially-analysed title restarts from scratch next lock.
|
||||
Fine for an overnight/idle workload; no mid-video checkpointing.
|
||||
|
||||
## The end-to-end UX
|
||||
|
||||
```bash
|
||||
# once: build + install for your GPU + model
|
||||
./scripts/build_install.py install.yaml
|
||||
# detects Fedora/Arch, ensures deps, compiles, installs units under ~/.local
|
||||
|
||||
# once: set your keys, enable the gate
|
||||
$EDITOR ~/.local/share/sae-worker/.env # JELLYFIN_API_KEY, TMDB_API_KEY
|
||||
systemctl --user enable --now sae-lock-gate.service
|
||||
|
||||
# from then on: nothing. Lock your screen → it analyses. Unlock → it stops.
|
||||
```
|
||||
|
||||
No Docker, no GPU passthrough config, no media re-mounting — the worker sees the
|
||||
same filesystem and GPU as everything else on the box.
|
||||
|
||||
## Implementation plan (follow-up commits)
|
||||
|
||||
Ordered so each step stands alone:
|
||||
|
||||
1. **installer skeleton** — `scripts/build_install.py`: parse `install.yaml`,
|
||||
distro detect, dependency check/print (start with cpu platform so it builds
|
||||
without a GPU), cmake+build, copy into prefix.
|
||||
2. **supervisor + cleanup** — `scripts/service.py` (worker loop + gallery-scan
|
||||
timer + SIGTERM); temp-file cleanup fix in `run_from_jellyfin.py`.
|
||||
3. **systemd units + lock gate** — generate/install `sae-worker.service`,
|
||||
`sae-lock-gate.service`, and the logind lock watcher.
|
||||
4. **gallery/model guard** — stamp embedder into `gallery.json`; startup mismatch
|
||||
check.
|
||||
5. **platform + distro matrix** — nvidia/amd backends; dnf/pacman dep lists; ONNX
|
||||
Runtime fetch fallback.
|
||||
6. **docs** — README "Run on your idle GPU" section.
|
||||
|
||||
## Settled decisions
|
||||
|
||||
- **ONNX Runtime build** — the installer fetches the **ROCm ORT** release. It
|
||||
serves the `amd` platform, and its CPU execution provider covers the `cpu`
|
||||
smoke-test fallback too, so one download handles both. (nvidia uses raw TRT and
|
||||
doesn't need ORT.)
|
||||
- **`dnf`/`pacman` invocation** — **auto-install.** The installer runs `sudo dnf
|
||||
install …` / `sudo pacman -S …` itself (prompting for sudo), rather than only
|
||||
printing the command. It still prints what it's about to install first.
|
||||
- **Distro coverage** — **Fedora + Arch only** for now. Debian/Ubuntu (`apt`) is
|
||||
out of scope.
|
||||
|
||||
## Open questions
|
||||
|
||||
*(none blocking — the spec above is buildable as-is.)*
|
||||
@@ -0,0 +1,21 @@
|
||||
# experiments/ in git keeps only scripts, README.md, SESSION_STATE.md, and this
|
||||
# file. Every data artifact — galleries, embedding dumps, the X-Ray corpus,
|
||||
# montage/frame images, DE trajectories, film manifests, and result summaries —
|
||||
# is pushed/pulled via scripts/artifacts/{push,pull}_artifacts.sh to the Gitea
|
||||
# generic package registry instead (see docs/rep4-optimizer-results.md).
|
||||
xray/
|
||||
dumps/
|
||||
galleries/
|
||||
*.h5
|
||||
manifests/
|
||||
trajectories/
|
||||
results/
|
||||
|
||||
# Raw run logs and scratch scripts (regenerated by every run).
|
||||
_scratch/
|
||||
|
||||
# Real local media paths (film slug -> path on this machine). Never committed —
|
||||
# these paths embed the specific source file names, which can include
|
||||
# scene-release tags. Only file-lut.template.json (placeholders) is tracked;
|
||||
# copy it to file-lut.json and fill in your own paths.
|
||||
file-lut.json
|
||||
@@ -0,0 +1,59 @@
|
||||
# experiments/ — X-Ray validation & optimizer artifacts
|
||||
|
||||
Durable home (in the repo tree, NOT `/tmp` scratch — a scratch wipe once cost an hour)
|
||||
for the data behind the X-Ray threshold-optimization and embedding-model bake-off.
|
||||
|
||||
## Layout
|
||||
- `xray/` — Amazon X-Ray Zenodo dataset (gitignored, ~140MB; DOI 10.5281/zenodo.17659734).
|
||||
- `dumps/` — per-model embedding dumps, one HDF5 per (model, film). Gitignored (large).
|
||||
Naming: `<model>/dump_<Film>.h5`. Regenerate with `scene_analyze --dump-embeddings`.
|
||||
- `galleries/` — per-model galleries (gitignored JSON). `gallery_<model>.json` +
|
||||
augmented variants. Regenerate with build_gallery / fetch_missing_actors.
|
||||
- `manifests/` — film manifests (committed — small, and the Jellyfin ID join is the
|
||||
authoritative record of which films/paths/X-Ray-dirs were used).
|
||||
- `trajectories/` — DE trajectories, one JSONL per run (committed — the evidence).
|
||||
- `results/` — final per-run metrics + the model comparison table (committed).
|
||||
|
||||
## Embedding-model bake-off (July 2026)
|
||||
|
||||
Question: is LVFace-B (455MB) actually the best vs X-Ray, or just the biggest?
|
||||
Method: **optimize per model** — each model gets its own dumps + gallery + full DE run,
|
||||
then compare each model at ITS OWN optimum (fairest — no model penalised by another's
|
||||
threshold). Scored by the weighted per-scene metric (out-of-cast misID ×10; see
|
||||
docs/optimizer-experiments.md).
|
||||
|
||||
Models:
|
||||
| model | file | size | MovieNet rank-1 (prior) |
|
||||
| ----- | ---- | ---- | ----------------------- |
|
||||
| LVFace-B_Glint360K | models/LVFace-B_Glint360K.onnx | 455 MB | — |
|
||||
| ArcFace w600k R50 | models/arcface_w600k_r50.onnx | 174 MB | 85.2% |
|
||||
| ArcFace R18 | models/arcface_r18.onnx | 48 MB | 72.2% |
|
||||
| ArcFace w600k MBF | models/arcface_w600k_mbf.onnx | 13 MB | 83.3% |
|
||||
|
||||
9 genuine X-Ray-overlap films (Jellyfin ID join): Benny & Joon, Café Society,
|
||||
Downton Abbey: A New Era, Lord of War, Lovelace, The Many Saints of Newark, Scarface,
|
||||
Sound of Metal, Valerian.
|
||||
|
||||
## Gallery-mode bake-off (full vs cast-restricted)
|
||||
|
||||
Second axis alongside the model comparison: does restricting the matcher's candidate
|
||||
set to a title's credited cast reduce cross-film misIDs (e.g. naming Archie Yates in a
|
||||
film he's not in) vs. matching against the whole 2418-actor gallery?
|
||||
|
||||
- **full** — match against the entire model gallery (2418 actors).
|
||||
- **restricted** — per film, match only against its Jellyfin credited cast, filtered
|
||||
from the gallery by jellyfin_id. This is what run_from_jellyfin.py does in production.
|
||||
|
||||
**LIMITATION — Jellyfin stores only ~15 actors per title.** Jellyfin's People list is
|
||||
capped at the top-billed cast (~15 Actors), NOT the full IMDb/X-Ray cast (e.g. Scarface:
|
||||
Jellyfin 15 vs X-Ray 67). This is a hard limit of the metadata Jellyfin imports — not a
|
||||
query parameter (verified: /Items?Fields=People returns 15 regardless; the single-item
|
||||
/Items/{id} endpoint 400s on this server). So the "restricted" arm restricts to the ~15
|
||||
top-billed leads, which caps its achievable recall at whatever fraction of on-screen
|
||||
actors are top-billed, but should drive out-of-cast misIDs toward zero. A production
|
||||
deployment wanting fuller cast restriction would need a richer cast source than Jellyfin
|
||||
(TMDB/IMDb full credits).
|
||||
|
||||
Matrix: 4 models × {full, restricted} = 8 DE runs, all reusing the 36 dumps + 4 baseline
|
||||
galleries (no augmentation — avoids test-set leakage on either arm). Scored by the
|
||||
duration-weighted per-scene metric with the misID split (report_rates.py).
|
||||
@@ -0,0 +1,105 @@
|
||||
# Session state — X-Ray optimizer + model bake-off (as of 2026-07-18)
|
||||
|
||||
Handoff for a fresh session. Everything below is UNCOMMITTED — commit early next session.
|
||||
|
||||
## What we're doing
|
||||
Optimizing the scene-actor pipeline's thresholds against Amazon X-Ray ground truth, and
|
||||
running a **model bake-off** (4 embedding models × gallery-mode × expansion) to answer:
|
||||
is LVFace (455MB) actually best, or just biggest? Does cast-restriction cut false IDs?
|
||||
Does per-film gallery expansion help?
|
||||
|
||||
## The metric (final form — this is what to use)
|
||||
`scripts/optimizer/second_score.py` — UNIFORM PER-SECOND sampling vs X-Ray:
|
||||
- At each second t: GT = X-Ray scene's cast at t; Pred = actors whose window covers t.
|
||||
- TPI / FPI / FN counted per second. **FPI weighted 10×** when the named actor isn't in
|
||||
the film's cast at all (a true misID like naming Archie Yates in a film he's not in) vs
|
||||
an in-cast timing slip.
|
||||
- **FN is fair**: only counts gallery-known cast (67% of X-Ray cast have no reference
|
||||
embedding, can't be recognised — see [[gallery-coverage-gap]]).
|
||||
- **agreement_rate** = mean per-second Jaccard (partial credit: "% of on-screen actors we
|
||||
agree with X-Ray about, over time"). NOT exact-set match.
|
||||
- Objective = macro-mean per-second weighted F1.
|
||||
|
||||
## Two DIFFERENT hangs — do not conflate them (corrected 2026-07-18)
|
||||
|
||||
**(a) The self-inflicted 100% hang (FIXED).** replay.py's CLI briefly called
|
||||
`replay(..., stop=False)` intending to `os._exit(0)` straight after, to "dodge" teardown.
|
||||
That was wrong: `PyNode::stop()` is the ONLY thing that sets `stop_flag_=true`, which is the
|
||||
ONLY exit condition for the source node's `run_loop()`. Skipping it meant the local `net`
|
||||
destructor — which runs synchronously when `replay()` returns, BEFORE main() can reach
|
||||
os._exit — joined a thread that could never stop. A **guaranteed** hang, not the driver
|
||||
flake. Symptom: every solo replay timed out at 45s and DE reported flat F1=0.0%.
|
||||
FIX: `replay(..., stop=True)` so `PyNode::stop()` signals the thread before the join;
|
||||
removed the dead os._exit / unused os import. VERIFIED: 45s guaranteed timeout → clean ~8s
|
||||
completion (3/3), and optimize.py's DE sweep returns correct non-zero metrics (F1 48-66%,
|
||||
matching prior best-so-far). Only 1 isolated per-film timeout in 11 evals × 3 films.
|
||||
|
||||
**(b) The genuine ROCm flake (rare, tolerated).** net.stop()→jthread.join() CAN still hang
|
||||
on a KPN worker stuck mid-rocBLAS-GEMM — a KNOWN ROCm bug
|
||||
(github.com/RadeonOpenCompute/ROCT-Thunk-Interface#56), NOT our code. HSA_ENABLE_SDMA=0
|
||||
makes it WORSE (breaks the matcher's DMA). It is much rarer than the ~20-30% figure quoted
|
||||
earlier in this session — that number was inflated by (a). The existing subprocess + 45s
|
||||
timeout absorbs it correctly.
|
||||
|
||||
## Both architectures are usable
|
||||
- `scripts/optimizer/optimize.py` + `replay.py` — subprocess per film, simpler, tolerates the
|
||||
rare true flake via its timeout. NOT broken; good for fallback / quick single-model runs.
|
||||
- `scripts/optimizer/model_server.py` + `optimize_server.py` — ONE persistent net per
|
||||
(model, gallery); replay each film by SWITCHING THE SOURCE (repoint frame list + reset
|
||||
index), change thresholds via runtime SETTERS, os._exit(0) at the very end (after all work,
|
||||
so no destructor-join problem). Higher throughput: skips gallery/build overhead per eval.
|
||||
VERIFIED: "ready", replays, emits metrics, ~30-40s/eval (GPU-bound, films serial).
|
||||
Still the preferred option for the long overnight matrix.
|
||||
|
||||
## Key C++ changes made (all in the KPN spec-and-tsan branch + our nodes)
|
||||
1. Runtime setters: `IdentityMatcherFunc::set_prob_threshold`, `SceneTrackerFunc::set_extinction_sec`
|
||||
(src/nodes/*). Exposed via sae_kpn: `set_prob_threshold(net,name,v)`, `set_extinction_sec(...)`.
|
||||
Needed `ObjectVariantNodeWrapper::functor()` + `PyNetwork::node_ptr()` accessors.
|
||||
2. `Channel::push_blocking()` (external/KPN/.../channel.hpp) — lossless backpressure push
|
||||
(waits instead of dropping when full). Exposed on IVariantChannel/VariantChannel; PyNode's
|
||||
run_loop now uses it. Reduced but did NOT fully fix a residual ~0.5% frame loss (25/5915)
|
||||
— the loss is elsewhere (matcher output or reader EOF-race). DECISION: accept it, <0.5%
|
||||
scattered doesn't change per-second F1 or rankings. Don't chase further.
|
||||
3. `dump_embeddings` standalone exe + `--max-decode-fps` (fixes LVFace dump truncation under
|
||||
parallel load). HDF5 gallery fast-load in gallery_store.cpp (18s JSON → 0.06s).
|
||||
`scripts/optimizer/json_to_hdf5_gallery.py` converts; galleries are `.h5` now.
|
||||
|
||||
All of KPN, matcher, scene_tracker, bindings need a rebuild:
|
||||
`cmake --build build --target sae_kpn sae_gallery dump_embeddings scene_analyze`
|
||||
|
||||
## Data on disk (durable, experiments/)
|
||||
- `experiments/xray/` — X-Ray Zenodo dataset. `experiments/dumps/<model>/dump_<slug>.h5` —
|
||||
all 9 films × 4 models, ALL FULL (LVFace re-dumped with --max-decode-fps 8). VERIFY counts
|
||||
match R50 before trusting (LVFace truncated under parallel dumping earlier).
|
||||
- `experiments/galleries/gallery_<model>.h5` (+ restricted/<model>/<slug>.h5, per-film cast-
|
||||
filtered to Jellyfin's ~15 top-billed — Jellyfin's hard cap, see experiments/README.md).
|
||||
- `experiments/manifests/rep3_<model>_<mode>.json` — 3 REPRESENTATIVE films (Lord of War /
|
||||
Scarface / Sound of Metal = clean / ensemble-lookalike / high-coverage) to keep evals fast
|
||||
(~28s vs ~90s for 9). Winner should be re-scored on all 9 after.
|
||||
- `experiments/manifests/films_<model>_<mode>.json` — all 9 films.
|
||||
|
||||
## Salvaged partial results (per-second metric)
|
||||
- R50 full +expand: **F1 66.4%** (208 evals, converged) — best so far
|
||||
- R50 full noexp: 55-60% → **expansion helps ~+6-11 recall**
|
||||
- MBF full noexp: 58.6%
|
||||
- (older scene-metric runs, superseded: R50≈LVFace≈MBF ~85%, restricted>full, LVFace not
|
||||
worth its size — but those used the OLD scene-union metric, redo with per-second.)
|
||||
|
||||
## TO DO next session
|
||||
1. **COMMIT everything first** (logical chunks: KPN setters+push_blocking; sae_kpn+dump exe;
|
||||
HDF5 gallery; optimizer scripts; per-second metric; experiments manifests/results/docs +
|
||||
tuned config.hpp defaults prob_threshold 0.76 extinction 1.5).
|
||||
2. Launch the full 16-run matrix via model_server on rep3 films (write trajectories to
|
||||
experiments/, NOT /tmp — /tmp gets wiped mid-session and cost us hours). ~28s/eval ×
|
||||
~84 evals × 16 = ~10hr. Runner pattern: experiments/run_overnight_rep3.sh but pointing
|
||||
optimize_server.py at model_server.
|
||||
3. assemble table: best model + expansion effect + misID, from experiments/results/*.json.
|
||||
4. Consider upstreaming to KPN++: runtime node setters, push_blocking, node_ptr/functor().
|
||||
|
||||
## Gotchas that burned time (don't repeat)
|
||||
- /tmp scratch gets WIPED mid-session → lost dumps + test files repeatedly. Use experiments/.
|
||||
- Verify a launched runner script EXISTS and PRODUCES evals before walking away (a heredoc
|
||||
once silently failed to write; a stale-code process ran the old metric for 12h).
|
||||
- pgrep/ps "survivors" are often the grep's own shell wrapper — check via /proc cmdline or ps.
|
||||
- Running many DE/replay processes in parallel on one GPU → deadlock/thrash. GPU peaks ~35%
|
||||
(not saturated) but concurrency>2-3 wedges. Serial-ish is safer.
|
||||
@@ -0,0 +1,66 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Build per-(model, mode) manifests for the model × gallery-mode bake-off.
|
||||
|
||||
Modes:
|
||||
full — every film matches against the whole model gallery (2418 actors)
|
||||
restricted — each film matches only its Jellyfin credited cast (~15 top-billed),
|
||||
via a per-film gallery filtered from the model gallery by jellyfin_id.
|
||||
|
||||
Each manifest is a list of {name, xray, slug, dump, gallery} — no "movie" path (that's
|
||||
resolved locally via experiments/file-lut.json, see run_montage_all.py, to avoid
|
||||
embedding source filenames in a file that gets shared as an artifact). optimize.py
|
||||
reads film["gallery"] per film, so restricted mode just points each film at its own
|
||||
filtered gallery — no optimizer change needed.
|
||||
|
||||
Writes experiments/manifests/films_<model>_<mode>.json and the restricted galleries to
|
||||
experiments/galleries/restricted/<model>/<slug>.h5.
|
||||
"""
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
REPO = Path(__file__).resolve().parent.parent
|
||||
sys.path.insert(0, str(REPO / "scripts" / "validation"))
|
||||
sys.path.insert(0, str(REPO / "scripts"))
|
||||
from identity import norm_name # noqa: E402
|
||||
from sae_gallery import load_gallery_hdf5, save_gallery_hdf5 # noqa: E402
|
||||
|
||||
MODELS = ["arcface_w600k_r50", "arcface_r18", "arcface_w600k_mbf", "LVFace-B_Glint360K"]
|
||||
|
||||
films = json.loads((REPO / "experiments/manifests/films.json").read_text())
|
||||
casts = json.loads((REPO / "experiments/manifests/jellyfin_casts.json").read_text())
|
||||
|
||||
|
||||
def restrict_gallery(model_gallery: dict, cast_ids: set[str]) -> dict:
|
||||
kept = [a for a in model_gallery["actors"] if a.get("jellyfin_id", "") in cast_ids]
|
||||
return {"actors": kept}
|
||||
|
||||
|
||||
for model in MODELS:
|
||||
gpath = REPO / f"experiments/galleries/gallery_{model}.h5"
|
||||
if not gpath.exists():
|
||||
print(f"skip {model}: gallery not built yet ({gpath})")
|
||||
continue
|
||||
model_gal = load_gallery_hdf5(gpath)
|
||||
|
||||
# full mode
|
||||
full = [{**f, "dump": f"experiments/dumps/{model}/dump_{f['slug']}.h5",
|
||||
"gallery": f"experiments/galleries/gallery_{model}.h5"} for f in films]
|
||||
(REPO / f"experiments/manifests/films_{model}_full.json").write_text(json.dumps(full, indent=2, ensure_ascii=False))
|
||||
|
||||
# restricted mode — per-film filtered gallery
|
||||
rdir = REPO / f"experiments/galleries/restricted/{model}"
|
||||
rdir.mkdir(parents=True, exist_ok=True)
|
||||
restr = []
|
||||
for f in films:
|
||||
cast_ids = set(casts.get(f["name"], []))
|
||||
rg = restrict_gallery(model_gal, cast_ids)
|
||||
rgpath = rdir / f"{f['slug']}.h5"
|
||||
save_gallery_hdf5(rg, rgpath)
|
||||
restr.append({**f, "dump": f"experiments/dumps/{model}/dump_{f['slug']}.h5",
|
||||
"gallery": str(rgpath.relative_to(REPO)),
|
||||
"_cast_size": len(rg["actors"])})
|
||||
(REPO / f"experiments/manifests/films_{model}_restricted.json").write_text(json.dumps(restr, indent=2, ensure_ascii=False))
|
||||
avg = sum(r["_cast_size"] for r in restr) / len(restr)
|
||||
print(f"{model}: full (2418) + restricted (avg {avg:.0f} actors/film) manifests written")
|
||||
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"Benny___Joon": "/path/to/your/movies/<replace-with-your-file>.mp4",
|
||||
"Café_Society": "/path/to/your/movies/<replace-with-your-file>.mp4",
|
||||
"Downton_Abbey__A_New_Era": "/path/to/your/movies/<replace-with-your-file>.mp4",
|
||||
"Lord_of_War": "/path/to/your/movies/<replace-with-your-file>.mp4",
|
||||
"Lovelace": "/path/to/your/movies/<replace-with-your-file>.mp4",
|
||||
"The_Many_Saints_of_Newark": "/path/to/your/movies/<replace-with-your-file>.mp4",
|
||||
"Scarface": "/path/to/your/movies/<replace-with-your-file>.mp4",
|
||||
"Sound_of_Metal": "/path/to/your/movies/<replace-with-your-file>.mp4",
|
||||
"Valerian_and_the_City_of_a_Thousand_Plan": "/path/to/your/movies/<replace-with-your-file>.mp4"
|
||||
}
|
||||
@@ -0,0 +1,93 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
run_montage_all.py — replay + best/worst-per-scene montage for all films in a
|
||||
manifest, using the shipped config.hpp defaults (LVFace, prob_threshold=0.754,
|
||||
anneal_sec=35.54, extinction_sec=57.43, expand_gallery). Skips a film if its
|
||||
montage manifest.json already exists (safe to re-run/resume).
|
||||
|
||||
Manifests no longer carry a "movie" path (removed to avoid embedding source
|
||||
filenames — some carry scene-release tags — in a file that gets zipped and
|
||||
pushed to the artifact registry). Movie paths are resolved locally via
|
||||
experiments/file-lut.json (gitignored; copy file-lut.template.json and fill
|
||||
in your own paths).
|
||||
|
||||
Usage:
|
||||
python experiments/run_montage_all.py \
|
||||
--manifest experiments/manifests/films_LVFace-B_Glint360K_full.json \
|
||||
--gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5 \
|
||||
--out-dir experiments/results/holdout/montage
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
REPO = Path(__file__).resolve().parent.parent
|
||||
REPLAY = str(REPO / "scripts" / "optimizer" / "replay.py")
|
||||
MONTAGE = str(REPO / "scripts" / "optimizer" / "dump_scene_montage.py")
|
||||
FILE_LUT = REPO / "experiments" / "file-lut.json"
|
||||
|
||||
CFG = {"prob_threshold": "0.7540024664611272", "anneal_sec": "35.53996030397922",
|
||||
"extinction_sec": "57.43359645269811"}
|
||||
|
||||
|
||||
def main():
|
||||
p = argparse.ArgumentParser()
|
||||
p.add_argument("--manifest", required=True)
|
||||
p.add_argument("--gallery", required=True)
|
||||
p.add_argument("--out-dir", required=True)
|
||||
args = p.parse_args()
|
||||
|
||||
if not FILE_LUT.exists():
|
||||
sys.exit(f"{FILE_LUT} not found — copy file-lut.template.json to "
|
||||
f"file-lut.json and fill in your local movie paths")
|
||||
file_lut = json.loads(FILE_LUT.read_text())
|
||||
|
||||
films = json.loads(Path(args.manifest).read_text())
|
||||
out_root = Path(args.out_dir)
|
||||
out_root.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
for f in films:
|
||||
slug = f["slug"]
|
||||
montage_dir = out_root / slug
|
||||
manifest_path = montage_dir / "manifest.json"
|
||||
if manifest_path.exists():
|
||||
print(f"[run_montage_all] skip {slug} (already done)", file=sys.stderr)
|
||||
continue
|
||||
|
||||
movie_path = file_lut.get(slug)
|
||||
if not movie_path:
|
||||
print(f"[run_montage_all] skip {slug}: no entry in {FILE_LUT}", file=sys.stderr)
|
||||
continue
|
||||
|
||||
print(f"[run_montage_all] === {f['name']} ({slug}) ===", file=sys.stderr)
|
||||
raw_path = out_root / f"raw_{slug}.jsonl"
|
||||
pred_path = out_root / f"pred_{slug}.json"
|
||||
|
||||
replay_cmd = [sys.executable, REPLAY, "--dump", f["dump"], "--gallery", args.gallery,
|
||||
"--out", str(pred_path), "--raw-out", str(raw_path),
|
||||
"--prob-threshold", CFG["prob_threshold"],
|
||||
"--anneal-sec", CFG["anneal_sec"],
|
||||
"--extinction-sec", CFG["extinction_sec"], "--expand-gallery"]
|
||||
r = subprocess.run(replay_cmd, capture_output=True, text=True, timeout=120)
|
||||
if r.returncode != 0:
|
||||
print(f"[run_montage_all] replay FAILED for {slug}: {r.stderr[-2000:]}",
|
||||
file=sys.stderr)
|
||||
continue
|
||||
|
||||
montage_cmd = [sys.executable, MONTAGE, "--raw", str(raw_path), "--dump", f["dump"],
|
||||
"--xray", f["xray"], "--movie", movie_path, "--gallery", args.gallery,
|
||||
"--out-dir", str(montage_dir)]
|
||||
r = subprocess.run(montage_cmd, capture_output=True, text=True, timeout=1800)
|
||||
if r.returncode != 0:
|
||||
print(f"[run_montage_all] montage FAILED for {slug}: {r.stderr[-2000:]}",
|
||||
file=sys.stderr)
|
||||
continue
|
||||
print(r.stderr.strip().splitlines()[-1] if r.stderr else "(no output)", file=sys.stderr)
|
||||
|
||||
print("[run_montage_all] ALL DONE", file=sys.stderr)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,26 @@
|
||||
#!/bin/bash
|
||||
set -u
|
||||
cd /home/dtourolle/Development/scene-actor-extraction
|
||||
mkdir -p experiments/trajectories experiments/results
|
||||
MODELS=(arcface_w600k_r50 arcface_r18 arcface_w600k_mbf LVFace-B_Glint360K)
|
||||
MODES=(full restricted); EXPAND=(1 0)
|
||||
for model in "${MODELS[@]}"; do
|
||||
for mode in "${MODES[@]}"; do
|
||||
for exp in "${EXPAND[@]}"; do
|
||||
xtag=$([ "$exp" = 1 ] && echo exp || echo noexp); tag="${model}_${mode}_${xtag}"
|
||||
best="experiments/results/rep4_best_${tag}.json"; traj="experiments/trajectories/rep4_${tag}.jsonl"
|
||||
[ -f "$best" ] && { echo "skip $tag"; continue; }
|
||||
: > "$traj"; echo "=== $tag START $(date '+%H:%M') ==="
|
||||
SAE_EXPAND=$exp REPLAY_WORKERS=4 DE_WORKERS=2 python3 scripts/optimizer/optimize.py \
|
||||
--manifest "experiments/manifests/rep4_${model}_${mode}.json" \
|
||||
--gallery "experiments/galleries/gallery_${model}.h5" \
|
||||
--params prob_threshold:0.5:0.999 anneal_sec:1:60 extinction_sec:1:60 \
|
||||
--popsize 10 --maxiter 15 --seed 0 \
|
||||
--trajectory "$traj" --out "experiments/results/opt_rep4_${tag}.json" \
|
||||
> "experiments/results/rep4_de_${tag}.log" 2>&1
|
||||
python3 -c "import json;r=[json.loads(l) for l in open('$traj')];b=max(r,key=lambda x:x['f1']);json.dump({'best':b,'n_evals':len(r)},open('$best','w'))" 2>/dev/null
|
||||
echo "done $tag $(date '+%H:%M'): $(python3 -c "import json;d=json.load(open('$best'));print('F1=%.1f%% misID=%d'%(d['best']['f1']*100,d['best'].get('FPI_misid',-1)))" 2>/dev/null)"
|
||||
done
|
||||
done
|
||||
done
|
||||
echo "=== REP4 MATRIX COMPLETE $(date '+%F %H:%M') ==="
|
||||
+42
@@ -0,0 +1,42 @@
|
||||
site_name: scene-actor-extraction
|
||||
site_description: Face-recognition pipeline for finding on-screen actor presence in film/TV, built on KPN++
|
||||
repo_url: https://gitea.tourolle.paris/dtourolle/scene-actor-extraction
|
||||
repo_name: dtourolle/scene-actor-extraction
|
||||
|
||||
theme:
|
||||
name: material
|
||||
palette:
|
||||
- scheme: slate
|
||||
primary: indigo
|
||||
accent: indigo
|
||||
features:
|
||||
- navigation.tabs
|
||||
- navigation.sections
|
||||
- navigation.top
|
||||
- content.code.copy
|
||||
- content.code.annotate
|
||||
|
||||
nav:
|
||||
- Home: index.md
|
||||
- Rep4 Bake-off & Re-tune: rep4-optimizer-results.md
|
||||
- Optimizer Experiments (prior round): optimizer-experiments.md
|
||||
- Service Conversion (proposal): service-conversion.md
|
||||
|
||||
markdown_extensions:
|
||||
- admonition
|
||||
- toc:
|
||||
permalink: true
|
||||
- pymdownx.highlight:
|
||||
anchor_linenums: true
|
||||
line_spans: __span
|
||||
pygments_lang_class: true
|
||||
- pymdownx.inlinehilite
|
||||
- pymdownx.superfences
|
||||
- pymdownx.tabbed:
|
||||
alternate_style: true
|
||||
- pymdownx.snippets:
|
||||
base_path: ['.']
|
||||
check_paths: true
|
||||
- pymdownx.details
|
||||
- attr_list
|
||||
- md_in_html
|
||||
BIN
Binary file not shown.
Reference in New Issue
Block a user