docs: rep4 bake-off write-up, MkDocs site, artifact-registry-backed experiments
docs/rep4-optimizer-results.md is the main deliverable: the model bake-off + threshold re-tune experiment log, including the ROCm teardown deadlock root cause and fix, DE concurrency tuning, the 16-combo results table, held-out validation against 5 films never seen by the optimizer (macro F1 67.4% vs. 75.3% training — a real generalization gap), the frozen-bbox "ghost track" failure mode found via annotated frame evidence, calibration curves per model, and an isolated-effects breakdown of gallery scope vs. pose expansion. MkDocs site (mkdocs.yml, docs/index.md) renders docs/*.md; scripts/docs/ pulls referenced images from the artifact registry and generates the calibration chart at build time (see the tooling commit) rather than committing images to the repo. experiments/ now keeps only scripts + README + SESSION_STATE.md in git — every data artifact (galleries, dumps, X-Ray corpus, montage frames, trajectories, manifests, results) moved to the Gitea package registry. film-lut.template.json is the committed placeholder for the gitignored file-lut.json (real local movie paths, never shared — some source filenames carry scene-release tags). Adds models/transnetv2.onnx (via Git LFS, matching the other ONNX models) for the new scene-detection path.
This commit is contained in:
@@ -0,0 +1,33 @@
|
||||
# scene-actor-extraction
|
||||
|
||||
A face-recognition pipeline that finds when each actor appears on screen in a
|
||||
film or TV episode — built on [KPN++](https://gitea.tourolle.paris/dtourolle/KPN)
|
||||
(a C++20 Kahn Process Network library) for the detect → track → match → scene
|
||||
pipeline, with a Jellyfin-integrated gallery and an X-Ray-validated optimizer.
|
||||
|
||||
## Start here
|
||||
|
||||
- **[Rep4 model bake-off + threshold re-tune](rep4-optimizer-results.md)** — the
|
||||
current experiment log: model comparison, DE threshold tuning, held-out
|
||||
validation, and the visual failure-mode evidence (frozen-bbox ghost tracks).
|
||||
This is where the shipped `src/config.hpp` defaults come from.
|
||||
- **[Optimizer experiments (prior round)](optimizer-experiments.md)** — the
|
||||
earlier scene-union-metric tuning pass, superseded by the per-second metric
|
||||
used in rep4 but kept for the ground-truth/architecture background.
|
||||
- **[Service conversion (proposal)](service-conversion.md)** — design sketch
|
||||
for an idle-GPU Docker worker, not yet built.
|
||||
|
||||
## Reproducing the benchmarks
|
||||
|
||||
Gallery `.h5` files, embedding dumps, the X-Ray corpus, montage frame images,
|
||||
and DE trajectories are not committed to this repository — they're pushed to
|
||||
the Gitea package registry and pulled on demand:
|
||||
|
||||
```bash
|
||||
scripts/artifacts/pull_artifacts.sh galleries
|
||||
scripts/artifacts/pull_artifacts.sh experiment-data
|
||||
scripts/artifacts/pull_artifacts.sh montage-frames <film-slug>
|
||||
```
|
||||
|
||||
See `scripts/artifacts/push_artifacts.sh` for the upload side (requires a
|
||||
`GITEA_TOKEN` with package write scope).
|
||||
@@ -0,0 +1,117 @@
|
||||
# Threshold optimization against Amazon X-Ray — experiment log
|
||||
|
||||
Record of the July 2026 work that tuned the pipeline's recognition/tracking defaults
|
||||
against ground-truth per-scene actor presence, and the tooling built to do it.
|
||||
|
||||
## TL;DR — what changed
|
||||
|
||||
| knob | old default | new default | why |
|
||||
| ---- | ----------- | ----------- | --- |
|
||||
| `prob_threshold` | 0.99 | **0.76** | 0.99 was far too strict — halved recall for a fraction of a precision point. DE optimum, tightly converged. |
|
||||
| `extinction_sec` | 5.0 | **1.5** | Long extinction smears presence into later scenes → FPs. DE converged tightly low. |
|
||||
| `anneal_sec` | 10.0 | 10.0 (unchanged) | DE found it **insensitive** (F1 flat ±0.3pp across 3–26s) — kept the round default. |
|
||||
| `detector_conf` | 0.5 | 0.5 (unchanged) | Sweep showed raising it only trades recall for precision at a net F1 loss — near-threshold detections are real faces, not phantoms. |
|
||||
|
||||
Net effect on the 9-film benchmark (strict per-scene, augmented gallery):
|
||||
recall **58% → ~72%**, F1 **70% → ~76%**, precision ~85%, at no meaningful precision cost.
|
||||
|
||||
## Ground truth
|
||||
|
||||
Public scene-level **Amazon X-Ray** dataset (Zenodo DOI 10.5281/zenodo.17659734,
|
||||
CC-BY-4.0): per movie, `people.csv` (name_id/person/character), `scenes.csv`
|
||||
(scene/start/end ms), `people_in_scenes.csv`. Films matched to the library by an
|
||||
**authoritative Jellyfin ID join** (query `/Items?IncludeItemTypes=Movie&Fields=
|
||||
ProviderIds,Path`, join Imdb/Tmdb against X-Ray metadata) — NOT fuzzy title matching,
|
||||
which collides badly (TV episodes vs same-named films). 9 genuine films with source
|
||||
video on disk: Benny & Joon, Café Society, Downton Abbey: A New Era, Lord of War,
|
||||
Lovelace, The Many Saints of Newark, Scarface, Sound of Metal, Valerian.
|
||||
|
||||
## The scoring metric (evolved through review)
|
||||
|
||||
Comparison unit is the **X-Ray scene**, not sampled timepoints. For each scene
|
||||
`[start,end]`: predicted set = **union** of actors detected anywhere in the span;
|
||||
GT set = actors X-Ray lists for that scene. Per scene TP/FP/FN, then:
|
||||
|
||||
- **Precision: STRICT.** Any predicted actor not in the scene's X-Ray set is an FP,
|
||||
*including out-of-cast confusions* (no gallery∩cast masking). An earlier
|
||||
timepoint-sampled, cast-masked metric HID ~570 such FPs across 9 films and let the
|
||||
optimizer drive `prob_threshold` to the 0.50 floor — a metric artifact. Counting
|
||||
them is essential.
|
||||
- **Recall: FAIR.** FN counts only X-Ray cast members **who are in the gallery**. 67%
|
||||
of X-Ray cast (261/392) have no gallery reference embedding and can never be
|
||||
recognised — counting them as misses penalises coverage, not the threshold. Both
|
||||
`recall` (fair) and `recall_strict` (all) are reported.
|
||||
- **Aggregation:** per-scene F1 → **duration-weighted average within a movie** (long
|
||||
scenes count more) → **equal-weight mean across movies** (macro; each film counts
|
||||
the same regardless of length). This is the DE objective.
|
||||
|
||||
Implemented in `scripts/optimizer/scene_score.py`.
|
||||
|
||||
## The gallery coverage gap
|
||||
|
||||
Diagnosing low recall: only **131 of 392** X-Ray cast were in the gallery (33%). Every
|
||||
in-gallery actor HAD embeddings (gallery well-formed) — the gap was pure coverage.
|
||||
`scripts/optimizer/fetch_missing_actors.py` recovers missing actors:
|
||||
`nm-id → TMDB /find external_ids → /person/{id}/images → download → embed (sae_embed)`,
|
||||
with a `--wikidata` fallback (P345→P18 Commons photo).
|
||||
|
||||
- **TMDB recovered 143/261** (55%). 0 face-detection failures; the rest had no TMDB
|
||||
person (60) or no profile photo (58). Coverage 33% → **70%**.
|
||||
- **Wikidata fallback: 0/118** of the TMDB failures — only 4 even had a Commons photo,
|
||||
none yielded a detectable face. → **TheTVDB not worth pursuing**: these remaining
|
||||
actors are obscure enough that no image source covers them, AND (see below) most are
|
||||
off-camera anyway.
|
||||
|
||||
**Coverage vs detectability.** Adding references lifted recall (58→68% at fixed config)
|
||||
but modestly. Per-film drill-down (Lord of War: 12 actors recovered, only 1 had a
|
||||
detectable on-camera face) showed most missing cast are a **detectability gap** — X-Ray
|
||||
credits them as cast-in-scene (incl. off-camera/background), but their face never
|
||||
appears clearly for the pipeline to detect. This is a fundamental ceiling of a
|
||||
face-recognition pipeline vs X-Ray's presence semantics, not a fixable gap.
|
||||
|
||||
## Optimizer
|
||||
|
||||
`scripts/optimizer/optimize.py` — scipy `differential_evolution` over the knob space,
|
||||
each candidate = full replay of all films through the **real** C++ nodes (see the
|
||||
KPN replay architecture below) scored by the metric above. Global objective (one
|
||||
config for all films, not per-film).
|
||||
|
||||
**Convergence stability (augmented gallery, 233 evals):**
|
||||
|
||||
| knob | top-20 range | verdict |
|
||||
| ---- | ------------ | ------- |
|
||||
| `prob_threshold` | 0.69–0.83 (σ 0.05) | TIGHT — trust 0.76 |
|
||||
| `extinction_sec` | 1.0–2.2 (σ 0.33) | TIGHT — trust 1.5 |
|
||||
| `anneal_sec` | 3.1–26.3 (σ 6.4) | LOOSE — insensitive, not hard-coded |
|
||||
|
||||
F1 varied only 0.3pp across the top-20 → objective is flat near the optimum, so only
|
||||
the tightly-converged knobs were adopted as defaults.
|
||||
|
||||
## Replay architecture (how the sweep is cheap)
|
||||
|
||||
The optimizer never re-decodes video. `scene_analyze --dump-embeddings out.h5` runs the
|
||||
expensive half once (decode→detect→align→embed) and dumps per-frame face embeddings
|
||||
+ metadata to HDF5 (`scripts/optimizer/SCHEMA.md`). `scripts/optimizer/replay.py` then
|
||||
replays that dump through the **real** C++ `face_tracker → identity_matcher →
|
||||
scene_tracker` assembled in a Python KPN network (`sae_kpn` nanobind module), varying
|
||||
Config knobs freely — no GPU embedding, no decode. Verified BYTE-EXACT against
|
||||
`scene_analyze`'s own output. The dumps are gallery-independent, so testing the
|
||||
augmented gallery needed no re-dump. `detector_conf` is replayable UPWARD only (the
|
||||
dump floor is 0.5).
|
||||
|
||||
## Reproduce
|
||||
|
||||
```bash
|
||||
# 1. dump (once per film, needs video)
|
||||
scene_analyze --movie <f> --gallery gallery.json --dump-embeddings dump.h5 --fps 1
|
||||
# 2. build films manifest by Jellyfin ID join (see scripts/optimizer notes)
|
||||
# 3. optimize
|
||||
python scripts/optimizer/optimize.py --manifest films.json --gallery gallery.json \
|
||||
--params prob_threshold:0.5:0.999 anneal_sec:1:30 extinction_sec:1:15 \
|
||||
--popsize 8 --maxiter 20 --trajectory traj.jsonl --out opt.json
|
||||
# 4. score a fixed config / validate on a held-out set
|
||||
python scripts/optimizer/score_config.py --manifest heldout.json --gallery gallery.json \
|
||||
--config '{"prob_threshold":0.76,"extinction_sec":1.5,"anneal_sec":10}'
|
||||
```
|
||||
|
||||
See also memory: kpn-python-replay-optimizer, gallery-coverage-gap, xray-validation-*.
|
||||
@@ -0,0 +1,408 @@
|
||||
# Rep4 model bake-off + threshold re-tune — experiment log (2026-07-18/19)
|
||||
|
||||
Follow-on to `docs/optimizer-experiments.md`, which used an older, since-superseded
|
||||
scene-union metric. This round uses the **per-second** metric
|
||||
(`scripts/optimizer/second_score.py`) and answers three questions in one 16-run
|
||||
matrix: which embedding model is best, does cast-restriction cut misIDs, and does
|
||||
per-film gallery expansion help.
|
||||
|
||||
## Why this experiment, and what it actually delivered
|
||||
|
||||
Four goals going in, and an honest read on each after held-out validation (see
|
||||
below):
|
||||
|
||||
1. **Find the best default parameters to ship.** Partially delivered. The DE optimum
|
||||
generalizes *unevenly* — strong on 3 of 5 held-out films, badly broken on 2 (one
|
||||
with a 974-count misID blowup). The tuned values are shipped anyway (see
|
||||
Caveats) because they still beat the old defaults on average, but this is not a
|
||||
settled, film-agnostic optimum.
|
||||
2. **Find the best default model.** Delivered with more confidence. LVFace beat
|
||||
r50/r18/mbf across all 4 training combos, and nothing in held-out validation
|
||||
contradicts the model choice specifically — the held-out failures trace to
|
||||
`extinction_sec`/threshold interactions and gallery coverage, not the embedder.
|
||||
3. **Provide insight into how the application works.** The strongest, most durable
|
||||
output. Found and fixed a real teardown deadlock bug (100% reproducible, not the
|
||||
assumed rare GPU flake), established a real concurrency ceiling (8 parallel
|
||||
replays, not more), and found a real parameter interaction (a strict
|
||||
`prob_threshold` "earns" a longer extinction window before it starts hurting).
|
||||
4. **Demonstrate limitations.** Delivered, and reinforced hard by held-out
|
||||
validation — see the "Held-out validation" section below for concrete examples,
|
||||
including a screenshot of the matcher naming 15 actors, none correctly, on a
|
||||
completely blank title card.
|
||||
|
||||
## TL;DR — what changed in `src/config.hpp`
|
||||
|
||||
| knob | old default | new default | why |
|
||||
| ---- | ----------- | ----------- | --- |
|
||||
| `arcface_model` | `arcface_w600k_r50.onnx` | **`LVFace-B_Glint360K.onnx`** | Best F1 in the full-gallery bake-off (75.3% vs r50's 68.5%). LVFace was worth its size. |
|
||||
| `prob_threshold` | 0.76 | **0.754** | Re-tuned for LVFace + per-second metric. |
|
||||
| `extinction_sec` | 1.5 | **57.4** | Reverses the earlier "short is better" finding — see below. |
|
||||
| `anneal_sec` | 10.0 | **35.5** | Same reversal; previously thought insensitive. |
|
||||
| `expand_gallery` | false | **true** | Helped recall on the full (unrestricted) gallery for the winning model — opposite of the earlier assumption. |
|
||||
|
||||
These are the **`LVFace-B_Glint360K_full_exp`** winning values — the best result that
|
||||
uses only features already live in the running app (full gallery, no cast
|
||||
restriction; see below for why restricted mode isn't applied even though it scored
|
||||
higher).
|
||||
|
||||
## Why re-run at all
|
||||
|
||||
`docs/optimizer-experiments.md`'s scene-union metric hid out-of-cast false positives
|
||||
behind a gallery∩cast recall mask, and the earlier 9-film benchmark was scene-level
|
||||
(union over a whole X-Ray scene), not a fair per-timepoint comparison. SESSION_STATE
|
||||
flagged the old scene-metric bake-off numbers (R50≈LVFace≈MBF ~85%) as superseded.
|
||||
This round uses `second_score.py`: uniform per-second sampling, GT = X-Ray scene's
|
||||
cast at time *t*, pred = actors whose presence window covers *t*, FPI weighted 10×
|
||||
when the named actor isn't in the film's cast at all (true misID) vs. an in-cast
|
||||
timing slip. FN only counts gallery-known cast (fair recall — 67% of X-Ray cast have
|
||||
no reference embedding, see `gallery-coverage-gap` memory).
|
||||
|
||||
## The deadlock that was blocking all of this
|
||||
|
||||
Every replay in this line of work goes through `scripts/optimizer/replay.py`, which
|
||||
runs the real C++ tracker/matcher/scene_tracker nodes inside a Python-assembled KPN
|
||||
network. Before this session, every subprocess replay **timed out at 45s, 100% of
|
||||
the time** — not the documented ~20-30% ROCm rocBLAS-GEMM driver flake, but a plain
|
||||
logic bug: `replay.py`'s CLI called `replay(net, ..., stop=False)` to *skip*
|
||||
`net.stop()` (trying to dodge the GEMM deadlock), planning to `os._exit(0)`
|
||||
immediately after. But:
|
||||
|
||||
- `PyNode::stop()` (`external/KPN/include/kpn/python/bindings.hpp`) is the *only*
|
||||
code that sets `stop_flag_ = true` before joining the node's worker thread.
|
||||
- The source node's `run_loop()` has `while (!stop_flag_)` as its only exit
|
||||
condition (it has no input channels, so it never sees a channel-closed signal
|
||||
either).
|
||||
- Skipping `stop()` meant `stop_flag_` never became true. When `replay()` returned,
|
||||
its local `net` went out of scope immediately, running `~PyNetwork` → `~PyNode` →
|
||||
`thread_.join()` **synchronously inside `replay()`'s own call frame** — before
|
||||
`main()` ever got control back to run `os._exit(0)`.
|
||||
|
||||
Root-caused via `gdb -p <pid> -batch -ex "thread apply all bt"` on a hung process:
|
||||
the main thread was stuck in `~PyNode`'s `jthread::join()`; the worker thread was in
|
||||
an ordinary `time.sleep()` inside the Python source callback, waiting for a stop
|
||||
signal that was never sent. The two HSA `kfd_wait_on_events` threads visible in the
|
||||
same trace are normal ROCm runtime housekeeping, not evidence of a wedged GPU kernel.
|
||||
|
||||
**Fix:** `replay.py` now calls `replay(..., stop=True)` (the removed `stop=False` +
|
||||
`os._exit` workaround was actively harmful). Verified 3/3 clean runs at ~8s each
|
||||
(down from a guaranteed 45s timeout), and a full DE sweep producing real, sensible
|
||||
F1/precision/recall instead of flat 0.0%.
|
||||
|
||||
## Concurrency tuning
|
||||
|
||||
With the deadlock fixed, `optimize.py` was extended with DE-level parallelism —
|
||||
`differential_evolution(..., workers=ThreadPoolExecutor.map)` — so multiple
|
||||
population candidates evaluate concurrently, each spawning its own per-film replay
|
||||
subprocesses (`REPLAY_WORKERS`). Total concurrent GPU replay processes ≈
|
||||
`DE_WORKERS × REPLAY_WORKERS`.
|
||||
|
||||
| concurrent replays | result |
|
||||
| --- | --- |
|
||||
| 3 (`REPLAY_WORKERS=3`, no DE parallelism) | baseline, GPU underutilised |
|
||||
| 6 (`DE_WORKERS=2 × REPLAY_WORKERS=3`) | clean, real scores, ~1 isolated timeout per run |
|
||||
| 8 (`DE_WORKERS=2 × REPLAY_WORKERS=4`, 4-film manifest) | clean, real scores |
|
||||
| 9 (`DE_WORKERS=3 × REPLAY_WORKERS=3`) | **broken** — every replay blew past the 45s timeout, all scores silently degraded to 0.0% |
|
||||
|
||||
9 concurrent replays looks like valid output (well-formed JSON, a real number) while
|
||||
actually being garbage — a dangerous failure mode, not a crash. **8 concurrent is the
|
||||
practical ceiling** on this GPU (gfx1100) for this workload. The matrix ran at
|
||||
`REPLAY_WORKERS=4 DE_WORKERS=2`.
|
||||
|
||||
## Training films (rep4) and validation set
|
||||
|
||||
9 films total have dumped embeddings across all 4 models. 4 were used for
|
||||
optimization (rep4), leaving 5 held out for validation:
|
||||
|
||||
- **Lord of War** (64-cast, "clean")
|
||||
- **Scarface** (67-cast, "ensemble/lookalike")
|
||||
- **Sound of Metal** (14-cast, "high gallery-coverage")
|
||||
- **Café Society** (62-cast, added this round — similar ensemble size to Scarface but
|
||||
different genre/lighting; picked to add diversity, not genre-overlap, over
|
||||
Downton Abbey or The Many Saints of Newark)
|
||||
|
||||
Held out: Benny & Joon, Downton Abbey: A New Era, Lovelace, The Many Saints of
|
||||
Newark, Valerian and the City of a Thousand Planets. The rep4 numbers below are
|
||||
training-set fit — see "Held-out validation" further down for the real
|
||||
generalization test.
|
||||
|
||||
## Search space and DE settings
|
||||
|
||||
`popsize=10, maxiter=15` (3 params → ≤480 evals/combo ceiling; DE's `tol` convergence
|
||||
usually stops earlier). `anneal_sec`/`extinction_sec` bounds were **widened from
|
||||
1–30/1–15 to 1–60/1–60 mid-run** (see below) — the 4 `arcface_w600k_r50` combos
|
||||
finished before the widening and still use the old, narrower bounds, so they are
|
||||
**not directly comparable** to the other 12 on those two params. Re-running r50 with
|
||||
the wider bounds was deferred (diminishing-returns judgment call, not yet done).
|
||||
|
||||
## Results — all 16 combos (4 models × {full, restricted} × {expand, noexp})
|
||||
|
||||
Ranked by F1. `misid` = FPI_misid, count of true wrong-actor identifications (an
|
||||
actor named who isn't in the film's cast at all) — distinct from `FPI`, which
|
||||
includes in-cast timing slips.
|
||||
|
||||
| combo | F1 | P | R | TPI | FPI | misid | FN |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| LVFace-B_Glint360K_restricted_exp | **78.3%** | 91.0% | 68.9% | 42830 | 3782 | 60 | 19492 |
|
||||
| LVFace-B_Glint360K_restricted_noexp | 76.7% | 91.5% | 66.2% | 41149 | 3400 | 59 | 21173 |
|
||||
| arcface_w600k_mbf_restricted_exp | 76.5% | 90.7% | 66.3% | 41270 | 4431 | **0** | 21052 |
|
||||
| arcface_r18_restricted_exp | 75.5% | 87.6% | 66.5% | 41399 | 5666 | 60 | 20923 |
|
||||
| **LVFace-B_Glint360K_full_exp** | **75.3%** | 89.7% | 65.4% | 47757 | 3407 | 232 | 26966 |
|
||||
| arcface_w600k_mbf_restricted_noexp | 75.0% | 91.1% | 63.9% | 39752 | 3465 | 60 | 22570 |
|
||||
| arcface_w600k_mbf_full_noexp | 74.2% | 87.4% | 64.4% | 12645 | 1312 | 57 | 6985 |
|
||||
| arcface_r18_restricted_noexp | 73.5% | 91.3% | 61.7% | 38299 | 3220 | 60 | 24023 |
|
||||
| LVFace-B_Glint360K_full_noexp | 72.4% | 94.2% | 58.9% | 27077 | 1725 | **0** | 19506 |
|
||||
| arcface_w600k_mbf_full_exp | 72.0% | 87.7% | 61.4% | 39875 | 3729 | 240 | 26338 |
|
||||
| arcface_w600k_r50_full_noexp † | 71.6% | 96.7% | 56.9% | 22471 | 361 | 45 | 17012 |
|
||||
| arcface_w600k_r50_restricted_exp † | 71.1% | 96.5% | 56.4% | 34954 | 1119 | 15 | 27368 |
|
||||
| arcface_w600k_r50_restricted_noexp † | 69.2% | 97.9% | 53.6% | 21146 | 327 | 15 | 18337 |
|
||||
| arcface_r18_full_exp | 69.1% | 87.6% | 57.7% | 37342 | 3119 | 242 | 28871 |
|
||||
| arcface_w600k_r50_full_exp † | 68.5% | 94.0% | 54.1% | 34982 | 903 | 150 | 31231 |
|
||||
| arcface_r18_full_noexp | 66.6% | 91.3% | 53.1% | 34314 | 2362 | 107 | 31899 |
|
||||
|
||||
† old, narrower anneal/extinction bounds (see above) — not directly comparable to
|
||||
the other 12 on those two params.
|
||||
|
||||
## Calibration curves — discriminative power, independent of the threshold
|
||||
|
||||
Each model's gallery carries a fitted Platt sigmoid `P(match | sim) = σ(a·sim + b)`
|
||||
(embedded directly in the gallery HDF5, see `src/gallery/gallery_calibration.hpp`).
|
||||
Plotting all four side by side shows discriminative power directly, independent of
|
||||
whatever `prob_threshold` a particular run happened to use:
|
||||
|
||||
|
||||
|
||||
LVFace-B has both the steepest curve (`a=17.7`, vs. 15.3–16.2 for the ArcFace
|
||||
variants) and the lowest P=0.5 decision boundary (similarity 0.23 vs. 0.27–0.31) —
|
||||
it separates same-actor from different-actor pairs more confidently at a lower
|
||||
similarity, consistent with it winning the full-gallery F1 comparison below.
|
||||
Generated by `scripts/docs/calibration_chart.py` (requires each gallery to have
|
||||
been calibrated at least once — run any replay against it first).
|
||||
|
||||
## Two effects in isolation: gallery scope, and pose expansion
|
||||
|
||||
The matrix crosses two independent variables — averaging across all 4 models
|
||||
isolates each one from model choice:
|
||||
|
||||
**Gallery scope (whole 2418-actor gallery vs. restricted to the film's credited
|
||||
cast)** — averaged over both expansion settings and all 4 models:
|
||||
|
||||
| scope | F1 | P | R | total misID (16 evals→8 each) |
|
||||
|---|---|---|---|---|
|
||||
| full | 71.2% | 91.1% | 59.0% | 1073 |
|
||||
| **restricted** | **74.5%** | 92.2% | **62.9%** | **329** |
|
||||
|
||||
Restriction wins outright on every axis — not a precision/recall trade, a clean
|
||||
win: **+3.3pp F1, +3.9pp recall, and less than a third the total misIDs.** Fewer
|
||||
candidates in the matcher's search space means fewer opportunities for a
|
||||
look-alike false match, and (per the recall gain) doesn't cost real detections.
|
||||
This is the single cleanest signal in the whole matrix — stronger than the model
|
||||
choice itself — which is exactly why cast-restriction becoming a real runtime
|
||||
feature (not just an optimizer trick) is the top item in Caveats below.
|
||||
|
||||
**Pose expansion (promoting a confidently-identified track's novel-pose views into
|
||||
a per-film gallery annex — `track_gallery.hpp`)** is smaller and interacts with
|
||||
scope rather than acting independently:
|
||||
|
||||
| scope | expansion | F1 | R | misID |
|
||||
|---|---|---|---|---|
|
||||
| full | off | 71.2% | 58.3% | 209 |
|
||||
| full | **on** | 71.2% | 59.7% | **864** |
|
||||
| restricted | off | 73.6% | 61.3% | 194 |
|
||||
| restricted | **on** | **75.4%** | **64.5%** | 135 |
|
||||
|
||||
In **restricted** mode, expansion is a clean win (+1.8pp F1, +3.2pp recall, misID
|
||||
actually *drops*) — the annex only ever competes against the film's own ~15-actor
|
||||
cast, so a "confidently identified, new pose" view is unlikely to be mistaken for
|
||||
someone else. In **full** mode, expansion buys essentially nothing on F1 (71.2% →
|
||||
71.2%, recall +1.4pp) while **quadrupling misIDs** (209 → 864): a novel-pose view
|
||||
promoted into the annex now competes against the whole 2418-actor gallery, so a
|
||||
"confident" identity is confident against the wrong universe of candidates — the
|
||||
expansion mechanism is "learning" a pose correctly, but the enlarged evidence pool
|
||||
makes it easier for that learned pose to look like a plausible match for a
|
||||
different actor. **Practical takeaway: gallery expansion should be paired with
|
||||
cast restriction, not used on the full gallery** — the version currently shipped
|
||||
as default (`full_exp`, see TL;DR) sits in the worse of these four cells for this
|
||||
specific knob, even though it's the best available combo without cast-restriction
|
||||
support in the app yet (see Caveats).
|
||||
|
||||
## What the data says
|
||||
|
||||
- **LVFace was worth its size.** It wins full-gallery mode outright (75.3% vs r50's
|
||||
68.5%, r18's 69.1%, mbf's 72.0%) with the highest recall of any full-mode combo —
|
||||
the earlier scene-union-metric conclusion ("not worth it") doesn't survive the
|
||||
better metric.
|
||||
- **Cast-restriction is a consistent, broad win.** Every model's best combo is
|
||||
`restricted`. It isn't just precision-safe: `arcface_w600k_mbf_restricted_exp` and
|
||||
`LVFace-B_Glint360K_full_noexp` both hit **misid=0** — zero true wrong-actor
|
||||
identifications. But restriction is an **offline optimizer technique, not a live
|
||||
app feature** — it pre-filters each film's gallery to its Jellyfin-credited cast
|
||||
before the matcher ever runs; there's no runtime "restrict to this film's cast"
|
||||
switch in the app today. Implementing it for real is future work, tracked
|
||||
separately from this defaults update.
|
||||
- **Gallery expansion (`expand_gallery`) is mode-dependent.** It helps on
|
||||
`restricted` galleries (smaller, so novel-pose promotion adds real signal) and on
|
||||
LVFace's full gallery, but **hurts** r50 and mbf in full mode (compare
|
||||
`arcface_w600k_r50_full_exp` 68.5% vs `full_noexp` 71.6%). Don't assume it's a free
|
||||
win — model- and mode-dependent.
|
||||
- **arcface_r18 (smallest/cheapest) is last across all 4 modes** — model capacity
|
||||
matters here, this isn't just parameter-count padding.
|
||||
- **`anneal_sec`/`extinction_sec` kept pinning at the search ceiling.** With the
|
||||
original 1–30/1–15 bounds, 3 of 4 r50 combos landed at ~93-98% of the upper bound.
|
||||
Widened to 1–60/1–60 mid-run (after the r50 combos had already finished) — every
|
||||
subsequent combo's best config landed at ~90%+ of the *new* ceiling too (e.g. the
|
||||
LVFace winner: `ann=59.2, ext=59.2`, both ~99% of 60). The likely mechanism: a
|
||||
strict `prob_threshold` "earns" a long extinction/anneal window — once false
|
||||
matches are rare, a long window just bridges real presence gaps (occlusion, turned
|
||||
face) instead of smearing false positives into later scenes, which is what made
|
||||
short windows look better under the old, laxer thresholds. **Open question, not
|
||||
resolved**: does this keep climbing past 60s, or does it actually plateau there?
|
||||
Decided not to chase further this round (diminishing-returns judgment call) — flag
|
||||
for a future sweep if it matters.
|
||||
|
||||
## Held-out validation — the number that actually matters
|
||||
|
||||
The 16-combo matrix above is training-set fit. This is the real test: the shipped
|
||||
config (`LVFace-B_Glint360K_full_exp` — `prob_threshold=0.754, anneal_sec=35.5,
|
||||
extinction_sec=57.4, expand_gallery=true`) replayed against the **5 films never seen
|
||||
by the optimizer** (Benny & Joon, Downton Abbey: A New Era, Lovelace, The Many
|
||||
Saints of Newark, Valerian and the City of a Thousand Planets), scored the same way.
|
||||
|
||||
| film | F1 | P | R | agree | TPI | FPI | misid | FN |
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
| Benny & Joon | 83.0% | 89.1% | 77.7% | 72.4% | 15125 | 1846 | 0 | 4337 |
|
||||
| Lovelace | 77.5% | 90.3% | 67.9% | 72.1% | 14990 | 1085 | 58 | 7085 |
|
||||
| Valerian and the City of a Thousand Planets | 74.1% | 97.1% | 60.0% | 58.8% | 18663 | 548 | 0 | 12467 |
|
||||
| Downton Abbey: A New Era | 56.2% | 97.8% | 39.4% | 40.6% | 52027 | 1173 | 0 | 80084 |
|
||||
| **The Many Saints of Newark** | **46.3%** | **54.7%** | 40.1% | 37.0% | 15922 | 4394 | **974** | 23791 |
|
||||
| **macro average (5 films)** | **67.4%** | 85.8% | 57.0% | 56.2% | 116727 | 9046 | 1032 | 127764 |
|
||||
|
||||
**67.4% held out vs. 75.3% on training** — an ~8pp drop, and a much more informative
|
||||
number than the training-set F1 alone: a **37pp spread between best and worst film**
|
||||
(83.0% vs 46.3%). The config does not generalize uniformly.
|
||||
|
||||
Two films are outright failure cases, and rendering bounding boxes + names on the
|
||||
extracted frames (`replay.py --raw-out` + `dump_error_frames.py --raw`, see
|
||||
Reproduce) turned what looked like a same-scene misidentification into something
|
||||
more precise and more damning:
|
||||
|
||||
- **The Many Saints of Newark** (mob-family drama, picked as a training-adjacent
|
||||
genre test) has **974 true misIDs** — far more than any training combo saw at any
|
||||
setting. The annotated frame below shows the same mechanism as Downton Abbey,
|
||||
at smaller scale: **"Jon Bernthal 100%", "Joey Diaz 100%", and "Billy Magnussen
|
||||
100%" are all frozen boxes over empty background — a blurred wall, hanging plates —
|
||||
with no face in them at all.** Only one real face in frame has a box, and it
|
||||
carries a *second*, colliding label ("Leslie Odom Jr." and "Michael Gandolfini"
|
||||
both at high confidence on the same box) — likely two tracks whose frozen bboxes
|
||||
happen to overlap.
|
||||
|
||||
|
||||
*`experiments/results/holdout/frames/many_saints/fpi/fpi_t03543.jpg`*
|
||||
|
||||
- **Downton Abbey: A New Era** (large ensemble, 36-cast) has high precision (97.8%)
|
||||
but recall collapses to 39.4% (FN=80084, by far the largest of the 5). The frame
|
||||
below is the starkest evidence in this whole experiment: **the matcher named 15
|
||||
actors — all of them wrong — over a completely blank closing title card with no
|
||||
faces on screen at all.**
|
||||
|
||||
|
||||
*`experiments/results/holdout/frames/downton_abbey/fpi/fpi_t07242.jpg`*
|
||||
|
||||
Both are the same mechanism, verified directly against the HDF5 dump and the raw
|
||||
per-frame stream (not just inferred from the screenshot): at the Downton Abbey
|
||||
title card (t=7242), the dump's own `face_count` is **0 from t≈7240 onward** — no
|
||||
detector output at all, confirmed independently of the pipeline. Yet all 15 "wrong"
|
||||
actors are still marked visible, each with the *exact same bbox, unchanged to the
|
||||
pixel*, repeated every single frame back to t=7222 (verified for Hugh Bonneville:
|
||||
`(1743.2, 0.0, 171.3, 317.8)` at every sampled second from 7222 through 7279+). That
|
||||
is `SceneTrackerFunc`'s `active_[actor_idx].last_bbox` (`scene_tracker_node.hpp`)
|
||||
being re-emitted unchanged — **this is the extinction state machine working exactly
|
||||
as coded**, not a bug in the logic. The film cuts from a packed group shot straight
|
||||
into ~40+ seconds of blank titles/credits with zero faces, and `extinction_sec=57.4`
|
||||
is comfortably long enough to bridge that entire gap without expiring, so the
|
||||
tracker faithfully keeps reporting "last known position" for a cast that is no
|
||||
longer on screen at all.
|
||||
|
||||
This reframes the "long extinction window wins" DE-search pattern (see above): it
|
||||
isn't unambiguously good. It buys recall by bridging real gaps (occlusion, turned
|
||||
face) in some films, but on others — specifically, hard cuts into long faceless
|
||||
footage — it manufactures a frozen-bbox ghost the tracker has no way to verify,
|
||||
precisely the failure mode the *original* short-extinction-window default (`1.5s`)
|
||||
was chosen to avoid. The training-set films apparently didn't have a long enough
|
||||
faceless stretch after a confirmed identity to expose this; the held-out set did.
|
||||
|
||||
Frames for all three films (`benny_joon`, `many_saints`, `downton_abbey` — one strong
|
||||
performer, two failure cases) are under `experiments/results/holdout/frames/`, each
|
||||
with a `manifest.json` listing the bucket (`best`/`fpi`/`fn`), timestamp, and
|
||||
predicted vs. ground-truth actors for every dumped frame. Frames are annotated with
|
||||
bounding boxes + name/confidence (green = identified, orange = unknown), matching
|
||||
`debug_renderer_node.hpp`'s colour convention. Generated by
|
||||
`scripts/optimizer/dump_error_frames.py --raw <replay.py --raw-out output>` (see
|
||||
Reproduce).
|
||||
|
||||
`dump_error_frames.py --interval-sec 600` also supports a per-N-second sweep
|
||||
instead of the fixed best/fpi/fn buckets: one best (highest Jaccard) and one worst
|
||||
(lowest Jaccard) frame per 10-minute window across the whole film, e.g.
|
||||
`experiments/results/holdout/frames/many_saints_intervals/` (13 windows × 2 = 26
|
||||
frames for the ~2h Many Saints runtime) — a way to sample "how are we doing" evenly
|
||||
across a film's runtime rather than only at its most extreme seconds.
|
||||
|
||||
## Caveats / what this is not
|
||||
|
||||
- **r50's 4 combos used the old, narrower search bounds** and aren't fully
|
||||
comparable to the other 12 on `anneal_sec`/`extinction_sec`.
|
||||
- **The applied defaults use `full_exp`, not the higher-scoring `restricted_exp`**,
|
||||
because cast-restriction isn't a real runtime feature yet (see above). The
|
||||
78.3% F1 number is not what the shipped defaults will produce — 75.3% is.
|
||||
- **`full_exp` is the best full-gallery combo, but not the safest.** Per the
|
||||
isolated-effects analysis above, `expand_gallery=true` only cleanly pays off
|
||||
when paired with cast-restriction; on the full gallery it's flat on F1 while
|
||||
~4x-ing misIDs (209→864, averaged across models). `full_noexp` scores lower
|
||||
(72.4% vs 75.3% for LVFace) but with **zero** true misIDs and higher precision
|
||||
(94.2% vs 89.7%). Kept `full_exp` as shipped since it's the highest-F1 option
|
||||
available without cast-restriction, but this is a real F1-vs-safety trade, not
|
||||
a strictly-better choice — worth revisiting if misID rate matters more than
|
||||
the last few points of F1 for a given deployment.
|
||||
- **Switching the default model is an operational change, not just a config tweak**:
|
||||
any existing gallery built from r50 embeddings is incompatible with LVFace
|
||||
embeddings and needs rebuilding.
|
||||
|
||||
## Reproduce
|
||||
|
||||
```bash
|
||||
# 4-film matrix, all 4 models × 2 modes × 2 expansion settings
|
||||
bash experiments/run_rep4_subprocess.sh
|
||||
|
||||
# single combo
|
||||
SAE_EXPAND=1 REPLAY_WORKERS=4 DE_WORKERS=2 python3 scripts/optimizer/optimize.py \
|
||||
--manifest experiments/manifests/rep4_LVFace-B_Glint360K_full.json \
|
||||
--gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5 \
|
||||
--params prob_threshold:0.5:0.999 anneal_sec:1:60 extinction_sec:1:60 \
|
||||
--popsize 10 --maxiter 15 --trajectory traj.jsonl --out best.json
|
||||
|
||||
# replay the shipped config against a held-out film — --raw-out is needed to draw
|
||||
# bboxes later (the merged pred.json has no per-frame bbox, only actor windows)
|
||||
python3 scripts/optimizer/replay.py \
|
||||
--dump experiments/dumps/LVFace-B_Glint360K/dump_<slug>.h5 \
|
||||
--gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5 \
|
||||
--out pred.json --raw-out raw.jsonl --prob-threshold 0.754 --anneal-sec 35.54 \
|
||||
--extinction-sec 57.43 --expand-gallery
|
||||
|
||||
# dump example frames (best-agreement / FPI / FN) for visual inspection, annotated
|
||||
# with bounding boxes + names (--raw is optional; omit for unannotated frames)
|
||||
python3 scripts/optimizer/dump_error_frames.py \
|
||||
--pred pred.json --raw raw.jsonl --xray experiments/xray/.../<xray_dir> \
|
||||
--movie "<path to source video>" \
|
||||
--gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5 \
|
||||
--out-dir experiments/results/holdout/frames/<name> --n-per-bucket 4
|
||||
|
||||
# or: one best + one worst frame per 10-minute window across the whole film
|
||||
python3 scripts/optimizer/dump_error_frames.py \
|
||||
--pred pred.json --raw raw.jsonl --xray experiments/xray/.../<xray_dir> \
|
||||
--movie "<path to source video>" \
|
||||
--gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5 \
|
||||
--out-dir experiments/results/holdout/frames/<name>_intervals --interval-sec 600
|
||||
```
|
||||
|
||||
See also: `docs/optimizer-experiments.md` (prior round, superseded metric),
|
||||
`experiments/SESSION_STATE.md`, and memory: kpn-python-replay-optimizer,
|
||||
gallery-coverage-gap, xray-validation-results, per-scene-presence-eval-design.
|
||||
@@ -0,0 +1,215 @@
|
||||
# Conversion to service — a native idle-GPU worker
|
||||
|
||||
Status: **design / proposal**. Nothing here is built yet.
|
||||
|
||||
## The idea
|
||||
|
||||
Turn the CLI tools into a **turnkey batch worker that uses the machine's idle
|
||||
GPU**: it analyses newly-added Jellyfin media when you're not using the computer
|
||||
(screen locked), and stops the instant you come back. It's an overnight job on
|
||||
your own Linux box.
|
||||
|
||||
**No Docker.** This runs on your own machine with your own drivers, so a container
|
||||
buys little and costs a lot: GPU passthrough (nvidia-container-toolkit, or
|
||||
`/dev/kfd`+`/dev/dri`+`video` group for ROCm) is the single most fragile part of a
|
||||
containerised setup, and it exists *only* because of the container. Natively, the
|
||||
GPU just works with the drivers you already have, and the media paths Jellyfin
|
||||
reports are just real paths — no re-mounting. So we ship a **native installer**
|
||||
instead of an image builder.
|
||||
|
||||
Two deliverables:
|
||||
|
||||
1. **An installer** — `scripts/build_install.py`. Detects your distro, ensures the
|
||||
GPU/build dependencies are present (via `dnf`/`pacman`), compiles `scene_analyze`
|
||||
for your GPU, and installs the binary + Python glue + two systemd **user**
|
||||
units under `~/.local`.
|
||||
2. **A screen-lock gate** — one of those systemd units watches logind lock/unlock
|
||||
and starts/stops the worker. Lock → analyse. Unlock → stop.
|
||||
|
||||
## What already exists (reuse, don't rebuild)
|
||||
|
||||
The processing loop is already implemented — this is packaging, building, and
|
||||
lock-gating, not new pipeline logic.
|
||||
|
||||
| Piece | Where | What it does |
|
||||
|---|---|---|
|
||||
| Analysis engine | `build/scene_analyze` | Video → face detect/align/embed → gallery match → result JSON |
|
||||
| Backend selection | `CMakeLists.txt` (`SAE_INFERENCE_BACKEND`, `SAE_GEMM_BACKEND`) | ORT/TRT + ROCm/CUDA, chosen **at build time** |
|
||||
| New-media queue | JRay plugin → `GET /Plugins/JRay/Tasks/Pending` | Backlog of items with no results yet |
|
||||
| Worker loop | `run_from_jellyfin.py --worker` | Poll Pending → run `scene_analyze` → push results |
|
||||
| Result push | `PUT /Plugins/JRay/Items/{id}/Truth` | Stores per-actor scene windows back in Jellyfin |
|
||||
| Incremental gallery | `make_jellyfin_gallery.py --merge` | Embeds only cast not already in the gallery |
|
||||
| Secrets loader | `.env` via `sae_env.py` | `JELLYFIN_URL`, `JELLYFIN_API_KEY`, `TMDB_API_KEY` |
|
||||
|
||||
## Installer config
|
||||
|
||||
One file. Build-time settings (fixed when we compile) vs. run-time settings (in the
|
||||
worker's `.env`, editable without recompiling).
|
||||
|
||||
```yaml
|
||||
# install.yaml — consumed by scripts/build_install.py
|
||||
|
||||
platform: nvidia # nvidia | amd | cpu → picks the cmake backend
|
||||
model:
|
||||
arcface: LVFace-B_Glint360K.onnx # embedder compiled against; gallery MUST match
|
||||
schedule:
|
||||
gallery_scan_interval: 24h # incremental --merge cadence; 0 disables the scanner
|
||||
prefix: ~/.local # install root (bin, share, systemd user units)
|
||||
|
||||
# runtime (written to the worker .env, not compiled in):
|
||||
runtime:
|
||||
jellyfin_url: http://localhost:8096
|
||||
# JELLYFIN_API_KEY / TMDB_API_KEY are filled into .env by hand after install
|
||||
```
|
||||
|
||||
**Secrets never go in the repo or a build artifact** — the installer writes a
|
||||
`.env` under the install prefix with blanks for the keys, and you fill them in
|
||||
once. `sae_env.py` already loads it.
|
||||
|
||||
**Model ⇄ gallery coupling (guard, don't just document):** embeddings from
|
||||
different recognition models aren't interchangeable. We compile against one
|
||||
embedder; the gallery must be built with the same one. Stamp the embedder name
|
||||
into `gallery.json`, and have the worker **refuse to start** if the gallery's
|
||||
embedder ≠ the configured `model.arcface`, rather than silently mismatching.
|
||||
|
||||
## Dependencies via the system package manager
|
||||
|
||||
The heavy build/runtime deps (OpenCV, ffmpeg, the GPU stack) are best provided by
|
||||
the distro, not vendored. The installer ships a per-distro dependency list and
|
||||
either installs them or prints the exact command. Targets: **Fedora (dnf)** and
|
||||
**Arch (pacman)** first.
|
||||
|
||||
| Dependency | Fedora (dnf) | Arch (pacman) |
|
||||
|---|---|---|
|
||||
| OpenCV | `opencv-devel` | `opencv` |
|
||||
| ffmpeg | `ffmpeg-free`/`ffmpeg` (RPM Fusion) | `ffmpeg` |
|
||||
| CMake / toolchain | `cmake gcc-c++` | `cmake gcc` |
|
||||
| CUDA + TensorRT (nvidia) | NVIDIA CUDA repo + `libnvinfer-*` | `cuda`, `tensorrt` |
|
||||
| ROCm (amd) | `rocm-hip-sdk` / `rocblas-devel` | `rocm-hip-sdk`, `rocblas` |
|
||||
| ONNX Runtime | **not packaged** — installer fetches a pinned release tarball into the prefix | AUR `onnxruntime` (or same pinned-tarball fallback) |
|
||||
|
||||
So the flow is: **detect distro → check each package → install via the native
|
||||
manager (or print `sudo dnf install …` / `sudo pacman -S …`)**, with ONNX Runtime
|
||||
as the one known gap the installer fills itself (a pinned upstream release
|
||||
extracted under the install prefix, so it doesn't depend on a system package that
|
||||
may not exist). CUDA/ROCm being present is *assumed* — you already run a GPU
|
||||
desktop; the installer verifies and points you at the vendor repo if not.
|
||||
|
||||
## What `build_install.py` does
|
||||
|
||||
```
|
||||
build_install.py install.yaml
|
||||
│
|
||||
├─ detect distro (dnf vs pacman) and platform from config
|
||||
├─ ensure deps: install via manager, or print the exact command; fetch ONNX Runtime if needed
|
||||
├─ cmake + build scene_analyze with the platform's backend flags:
|
||||
│ nvidia → -DSAE_INFERENCE_BACKEND=TRT -DSAE_GEMM_BACKEND=CUDA
|
||||
│ amd → -DSAE_INFERENCE_BACKEND=ORT -DSAE_GEMM_BACKEND=ROCM
|
||||
│ cpu → -DSAE_INFERENCE_BACKEND=ORT (CPU EP; slow, for smoke tests)
|
||||
├─ install into <prefix>:
|
||||
│ bin/sae-scene-analyze the compiled binary
|
||||
│ share/sae-worker/ Python glue + a venv (requests, etc.), models/
|
||||
│ share/sae-worker/.env runtime config (keys blank, url from config)
|
||||
├─ install systemd --user units:
|
||||
│ sae-worker.service runs the worker + gallery-scan supervisor
|
||||
│ sae-lock-gate.service watches logind lock/unlock, start/stops the worker
|
||||
└─ print next steps (edit .env, `systemctl --user enable --now sae-lock-gate`)
|
||||
```
|
||||
|
||||
## The worker service (supervisor)
|
||||
|
||||
`sae-worker.service` runs a small Python supervisor as its main process:
|
||||
|
||||
- starts the **worker loop** (`run_from_jellyfin.py --worker`) — the hot path,
|
||||
- starts a **gallery-scan timer** — sleeps `gallery_scan_interval`, runs
|
||||
`make_jellyfin_gallery.py --merge`, repeats,
|
||||
- exits cleanly on SIGTERM (see re-queue below).
|
||||
|
||||
## The lock gate
|
||||
|
||||
`sae-lock-gate.service` runs a tiny watcher that subscribes to logind
|
||||
lock/unlock signals and drives the worker service:
|
||||
|
||||
```
|
||||
screen locks → systemctl --user start sae-worker.service
|
||||
screen unlocks → systemctl --user stop sae-worker.service (SIGTERM)
|
||||
```
|
||||
|
||||
**Screen-lock is the only signal — deliberately.** We don't also gate on GPU/CPU
|
||||
load, because our own worker *is* the load: a load threshold would form a feedback
|
||||
loop (worker starts → GPU spikes → threshold trips → worker stops → load drops →
|
||||
restart → …). Lock state is external to what the worker does, so it can't
|
||||
oscillate.
|
||||
|
||||
Signal source is desktop-dependent: logind `Lock`/`Unlock` (GNOME/KDE via
|
||||
`loginctl`/D-Bus) covers most setups; a `swayidle`/`xss-lock` hook is the fallback
|
||||
for wlroots/X-only compositors. The installer picks based on what's present.
|
||||
|
||||
## On resume: hard stop + re-queue (it's free)
|
||||
|
||||
Stopping the worker mid-analysis costs nothing to reschedule, because of how the
|
||||
JRay queue works: **an item only leaves `/Tasks/Pending` once its results are
|
||||
pushed** (`push_truth`). A worker stopped mid-`scene_analyze` simply leaves that
|
||||
item Pending — next lock picks it up again. No re-queue bookkeeping.
|
||||
|
||||
Two small correctness requirements (the only worker changes needed):
|
||||
|
||||
1. **Never push a partial result.** Already true — `push_truth` runs only after
|
||||
`scene_analyze` returns; a killed run pushes nothing. ✓ (keep it that way).
|
||||
2. **Clean up on signal.** `process_item` writes a temp filtered-gallery file and
|
||||
unlinks it in a `finally`; a SIGKILL skips `finally`. Fix: write temps under a
|
||||
dir the worker wipes on start, and/or a SIGTERM handler that unlinks before
|
||||
exit. Minor.
|
||||
|
||||
Accepted trade-off: a partially-analysed title restarts from scratch next lock.
|
||||
Fine for an overnight/idle workload; no mid-video checkpointing.
|
||||
|
||||
## The end-to-end UX
|
||||
|
||||
```bash
|
||||
# once: build + install for your GPU + model
|
||||
./scripts/build_install.py install.yaml
|
||||
# detects Fedora/Arch, ensures deps, compiles, installs units under ~/.local
|
||||
|
||||
# once: set your keys, enable the gate
|
||||
$EDITOR ~/.local/share/sae-worker/.env # JELLYFIN_API_KEY, TMDB_API_KEY
|
||||
systemctl --user enable --now sae-lock-gate.service
|
||||
|
||||
# from then on: nothing. Lock your screen → it analyses. Unlock → it stops.
|
||||
```
|
||||
|
||||
No Docker, no GPU passthrough config, no media re-mounting — the worker sees the
|
||||
same filesystem and GPU as everything else on the box.
|
||||
|
||||
## Implementation plan (follow-up commits)
|
||||
|
||||
Ordered so each step stands alone:
|
||||
|
||||
1. **installer skeleton** — `scripts/build_install.py`: parse `install.yaml`,
|
||||
distro detect, dependency check/print (start with cpu platform so it builds
|
||||
without a GPU), cmake+build, copy into prefix.
|
||||
2. **supervisor + cleanup** — `scripts/service.py` (worker loop + gallery-scan
|
||||
timer + SIGTERM); temp-file cleanup fix in `run_from_jellyfin.py`.
|
||||
3. **systemd units + lock gate** — generate/install `sae-worker.service`,
|
||||
`sae-lock-gate.service`, and the logind lock watcher.
|
||||
4. **gallery/model guard** — stamp embedder into `gallery.json`; startup mismatch
|
||||
check.
|
||||
5. **platform + distro matrix** — nvidia/amd backends; dnf/pacman dep lists; ONNX
|
||||
Runtime fetch fallback.
|
||||
6. **docs** — README "Run on your idle GPU" section.
|
||||
|
||||
## Settled decisions
|
||||
|
||||
- **ONNX Runtime build** — the installer fetches the **ROCm ORT** release. It
|
||||
serves the `amd` platform, and its CPU execution provider covers the `cpu`
|
||||
smoke-test fallback too, so one download handles both. (nvidia uses raw TRT and
|
||||
doesn't need ORT.)
|
||||
- **`dnf`/`pacman` invocation** — **auto-install.** The installer runs `sudo dnf
|
||||
install …` / `sudo pacman -S …` itself (prompting for sudo), rather than only
|
||||
printing the command. It still prints what it's about to install first.
|
||||
- **Distro coverage** — **Fedora + Arch only** for now. Debian/Ubuntu (`apt`) is
|
||||
out of scope.
|
||||
|
||||
## Open questions
|
||||
|
||||
*(none blocking — the spec above is buildable as-is.)*
|
||||
Reference in New Issue
Block a user