dtourolle
d340da755a
docs/rep4-optimizer-results.md is the main deliverable: the model bake-off + threshold re-tune experiment log, including the ROCm teardown deadlock root cause and fix, DE concurrency tuning, the 16-combo results table, held-out validation against 5 films never seen by the optimizer (macro F1 67.4% vs. 75.3% training — a real generalization gap), the frozen-bbox "ghost track" failure mode found via annotated frame evidence, calibration curves per model, and an isolated-effects breakdown of gallery scope vs. pose expansion. MkDocs site (mkdocs.yml, docs/index.md) renders docs/*.md; scripts/docs/ pulls referenced images from the artifact registry and generates the calibration chart at build time (see the tooling commit) rather than committing images to the repo. experiments/ now keeps only scripts + README + SESSION_STATE.md in git — every data artifact (galleries, dumps, X-Ray corpus, montage frames, trajectories, manifests, results) moved to the Gitea package registry. film-lut.template.json is the committed placeholder for the gitignored file-lut.json (real local movie paths, never shared — some source filenames carry scene-release tags). Adds models/transnetv2.onnx (via Git LFS, matching the other ONNX models) for the new scene-detection path.
60 lines
3.4 KiB
Markdown
60 lines
3.4 KiB
Markdown
# experiments/ — X-Ray validation & optimizer artifacts
|
||
|
||
Durable home (in the repo tree, NOT `/tmp` scratch — a scratch wipe once cost an hour)
|
||
for the data behind the X-Ray threshold-optimization and embedding-model bake-off.
|
||
|
||
## Layout
|
||
- `xray/` — Amazon X-Ray Zenodo dataset (gitignored, ~140MB; DOI 10.5281/zenodo.17659734).
|
||
- `dumps/` — per-model embedding dumps, one HDF5 per (model, film). Gitignored (large).
|
||
Naming: `<model>/dump_<Film>.h5`. Regenerate with `scene_analyze --dump-embeddings`.
|
||
- `galleries/` — per-model galleries (gitignored JSON). `gallery_<model>.json` +
|
||
augmented variants. Regenerate with build_gallery / fetch_missing_actors.
|
||
- `manifests/` — film manifests (committed — small, and the Jellyfin ID join is the
|
||
authoritative record of which films/paths/X-Ray-dirs were used).
|
||
- `trajectories/` — DE trajectories, one JSONL per run (committed — the evidence).
|
||
- `results/` — final per-run metrics + the model comparison table (committed).
|
||
|
||
## Embedding-model bake-off (July 2026)
|
||
|
||
Question: is LVFace-B (455MB) actually the best vs X-Ray, or just the biggest?
|
||
Method: **optimize per model** — each model gets its own dumps + gallery + full DE run,
|
||
then compare each model at ITS OWN optimum (fairest — no model penalised by another's
|
||
threshold). Scored by the weighted per-scene metric (out-of-cast misID ×10; see
|
||
docs/optimizer-experiments.md).
|
||
|
||
Models:
|
||
| model | file | size | MovieNet rank-1 (prior) |
|
||
| ----- | ---- | ---- | ----------------------- |
|
||
| LVFace-B_Glint360K | models/LVFace-B_Glint360K.onnx | 455 MB | — |
|
||
| ArcFace w600k R50 | models/arcface_w600k_r50.onnx | 174 MB | 85.2% |
|
||
| ArcFace R18 | models/arcface_r18.onnx | 48 MB | 72.2% |
|
||
| ArcFace w600k MBF | models/arcface_w600k_mbf.onnx | 13 MB | 83.3% |
|
||
|
||
9 genuine X-Ray-overlap films (Jellyfin ID join): Benny & Joon, Café Society,
|
||
Downton Abbey: A New Era, Lord of War, Lovelace, The Many Saints of Newark, Scarface,
|
||
Sound of Metal, Valerian.
|
||
|
||
## Gallery-mode bake-off (full vs cast-restricted)
|
||
|
||
Second axis alongside the model comparison: does restricting the matcher's candidate
|
||
set to a title's credited cast reduce cross-film misIDs (e.g. naming Archie Yates in a
|
||
film he's not in) vs. matching against the whole 2418-actor gallery?
|
||
|
||
- **full** — match against the entire model gallery (2418 actors).
|
||
- **restricted** — per film, match only against its Jellyfin credited cast, filtered
|
||
from the gallery by jellyfin_id. This is what run_from_jellyfin.py does in production.
|
||
|
||
**LIMITATION — Jellyfin stores only ~15 actors per title.** Jellyfin's People list is
|
||
capped at the top-billed cast (~15 Actors), NOT the full IMDb/X-Ray cast (e.g. Scarface:
|
||
Jellyfin 15 vs X-Ray 67). This is a hard limit of the metadata Jellyfin imports — not a
|
||
query parameter (verified: /Items?Fields=People returns 15 regardless; the single-item
|
||
/Items/{id} endpoint 400s on this server). So the "restricted" arm restricts to the ~15
|
||
top-billed leads, which caps its achievable recall at whatever fraction of on-screen
|
||
actors are top-billed, but should drive out-of-cast misIDs toward zero. A production
|
||
deployment wanting fuller cast restriction would need a richer cast source than Jellyfin
|
||
(TMDB/IMDb full credits).
|
||
|
||
Matrix: 4 models × {full, restricted} = 8 DE runs, all reusing the 36 dumps + 4 baseline
|
||
galleries (no augmentation — avoids test-set leakage on either arm). Scored by the
|
||
duration-weighted per-scene metric with the misID split (report_rates.py).
|