dtourolle
d340da755a
docs/rep4-optimizer-results.md is the main deliverable: the model bake-off + threshold re-tune experiment log, including the ROCm teardown deadlock root cause and fix, DE concurrency tuning, the 16-combo results table, held-out validation against 5 films never seen by the optimizer (macro F1 67.4% vs. 75.3% training — a real generalization gap), the frozen-bbox "ghost track" failure mode found via annotated frame evidence, calibration curves per model, and an isolated-effects breakdown of gallery scope vs. pose expansion. MkDocs site (mkdocs.yml, docs/index.md) renders docs/*.md; scripts/docs/ pulls referenced images from the artifact registry and generates the calibration chart at build time (see the tooling commit) rather than committing images to the repo. experiments/ now keeps only scripts + README + SESSION_STATE.md in git — every data artifact (galleries, dumps, X-Ray corpus, montage frames, trajectories, manifests, results) moved to the Gitea package registry. film-lut.template.json is the committed placeholder for the gitignored file-lut.json (real local movie paths, never shared — some source filenames carry scene-release tags). Adds models/transnetv2.onnx (via Git LFS, matching the other ONNX models) for the new scene-detection path.
106 lines
7.1 KiB
Markdown
106 lines
7.1 KiB
Markdown
# Session state — X-Ray optimizer + model bake-off (as of 2026-07-18)
|
||
|
||
Handoff for a fresh session. Everything below is UNCOMMITTED — commit early next session.
|
||
|
||
## What we're doing
|
||
Optimizing the scene-actor pipeline's thresholds against Amazon X-Ray ground truth, and
|
||
running a **model bake-off** (4 embedding models × gallery-mode × expansion) to answer:
|
||
is LVFace (455MB) actually best, or just biggest? Does cast-restriction cut false IDs?
|
||
Does per-film gallery expansion help?
|
||
|
||
## The metric (final form — this is what to use)
|
||
`scripts/optimizer/second_score.py` — UNIFORM PER-SECOND sampling vs X-Ray:
|
||
- At each second t: GT = X-Ray scene's cast at t; Pred = actors whose window covers t.
|
||
- TPI / FPI / FN counted per second. **FPI weighted 10×** when the named actor isn't in
|
||
the film's cast at all (a true misID like naming Archie Yates in a film he's not in) vs
|
||
an in-cast timing slip.
|
||
- **FN is fair**: only counts gallery-known cast (67% of X-Ray cast have no reference
|
||
embedding, can't be recognised — see [[gallery-coverage-gap]]).
|
||
- **agreement_rate** = mean per-second Jaccard (partial credit: "% of on-screen actors we
|
||
agree with X-Ray about, over time"). NOT exact-set match.
|
||
- Objective = macro-mean per-second weighted F1.
|
||
|
||
## Two DIFFERENT hangs — do not conflate them (corrected 2026-07-18)
|
||
|
||
**(a) The self-inflicted 100% hang (FIXED).** replay.py's CLI briefly called
|
||
`replay(..., stop=False)` intending to `os._exit(0)` straight after, to "dodge" teardown.
|
||
That was wrong: `PyNode::stop()` is the ONLY thing that sets `stop_flag_=true`, which is the
|
||
ONLY exit condition for the source node's `run_loop()`. Skipping it meant the local `net`
|
||
destructor — which runs synchronously when `replay()` returns, BEFORE main() can reach
|
||
os._exit — joined a thread that could never stop. A **guaranteed** hang, not the driver
|
||
flake. Symptom: every solo replay timed out at 45s and DE reported flat F1=0.0%.
|
||
FIX: `replay(..., stop=True)` so `PyNode::stop()` signals the thread before the join;
|
||
removed the dead os._exit / unused os import. VERIFIED: 45s guaranteed timeout → clean ~8s
|
||
completion (3/3), and optimize.py's DE sweep returns correct non-zero metrics (F1 48-66%,
|
||
matching prior best-so-far). Only 1 isolated per-film timeout in 11 evals × 3 films.
|
||
|
||
**(b) The genuine ROCm flake (rare, tolerated).** net.stop()→jthread.join() CAN still hang
|
||
on a KPN worker stuck mid-rocBLAS-GEMM — a KNOWN ROCm bug
|
||
(github.com/RadeonOpenCompute/ROCT-Thunk-Interface#56), NOT our code. HSA_ENABLE_SDMA=0
|
||
makes it WORSE (breaks the matcher's DMA). It is much rarer than the ~20-30% figure quoted
|
||
earlier in this session — that number was inflated by (a). The existing subprocess + 45s
|
||
timeout absorbs it correctly.
|
||
|
||
## Both architectures are usable
|
||
- `scripts/optimizer/optimize.py` + `replay.py` — subprocess per film, simpler, tolerates the
|
||
rare true flake via its timeout. NOT broken; good for fallback / quick single-model runs.
|
||
- `scripts/optimizer/model_server.py` + `optimize_server.py` — ONE persistent net per
|
||
(model, gallery); replay each film by SWITCHING THE SOURCE (repoint frame list + reset
|
||
index), change thresholds via runtime SETTERS, os._exit(0) at the very end (after all work,
|
||
so no destructor-join problem). Higher throughput: skips gallery/build overhead per eval.
|
||
VERIFIED: "ready", replays, emits metrics, ~30-40s/eval (GPU-bound, films serial).
|
||
Still the preferred option for the long overnight matrix.
|
||
|
||
## Key C++ changes made (all in the KPN spec-and-tsan branch + our nodes)
|
||
1. Runtime setters: `IdentityMatcherFunc::set_prob_threshold`, `SceneTrackerFunc::set_extinction_sec`
|
||
(src/nodes/*). Exposed via sae_kpn: `set_prob_threshold(net,name,v)`, `set_extinction_sec(...)`.
|
||
Needed `ObjectVariantNodeWrapper::functor()` + `PyNetwork::node_ptr()` accessors.
|
||
2. `Channel::push_blocking()` (external/KPN/.../channel.hpp) — lossless backpressure push
|
||
(waits instead of dropping when full). Exposed on IVariantChannel/VariantChannel; PyNode's
|
||
run_loop now uses it. Reduced but did NOT fully fix a residual ~0.5% frame loss (25/5915)
|
||
— the loss is elsewhere (matcher output or reader EOF-race). DECISION: accept it, <0.5%
|
||
scattered doesn't change per-second F1 or rankings. Don't chase further.
|
||
3. `dump_embeddings` standalone exe + `--max-decode-fps` (fixes LVFace dump truncation under
|
||
parallel load). HDF5 gallery fast-load in gallery_store.cpp (18s JSON → 0.06s).
|
||
`scripts/optimizer/json_to_hdf5_gallery.py` converts; galleries are `.h5` now.
|
||
|
||
All of KPN, matcher, scene_tracker, bindings need a rebuild:
|
||
`cmake --build build --target sae_kpn sae_gallery dump_embeddings scene_analyze`
|
||
|
||
## Data on disk (durable, experiments/)
|
||
- `experiments/xray/` — X-Ray Zenodo dataset. `experiments/dumps/<model>/dump_<slug>.h5` —
|
||
all 9 films × 4 models, ALL FULL (LVFace re-dumped with --max-decode-fps 8). VERIFY counts
|
||
match R50 before trusting (LVFace truncated under parallel dumping earlier).
|
||
- `experiments/galleries/gallery_<model>.h5` (+ restricted/<model>/<slug>.h5, per-film cast-
|
||
filtered to Jellyfin's ~15 top-billed — Jellyfin's hard cap, see experiments/README.md).
|
||
- `experiments/manifests/rep3_<model>_<mode>.json` — 3 REPRESENTATIVE films (Lord of War /
|
||
Scarface / Sound of Metal = clean / ensemble-lookalike / high-coverage) to keep evals fast
|
||
(~28s vs ~90s for 9). Winner should be re-scored on all 9 after.
|
||
- `experiments/manifests/films_<model>_<mode>.json` — all 9 films.
|
||
|
||
## Salvaged partial results (per-second metric)
|
||
- R50 full +expand: **F1 66.4%** (208 evals, converged) — best so far
|
||
- R50 full noexp: 55-60% → **expansion helps ~+6-11 recall**
|
||
- MBF full noexp: 58.6%
|
||
- (older scene-metric runs, superseded: R50≈LVFace≈MBF ~85%, restricted>full, LVFace not
|
||
worth its size — but those used the OLD scene-union metric, redo with per-second.)
|
||
|
||
## TO DO next session
|
||
1. **COMMIT everything first** (logical chunks: KPN setters+push_blocking; sae_kpn+dump exe;
|
||
HDF5 gallery; optimizer scripts; per-second metric; experiments manifests/results/docs +
|
||
tuned config.hpp defaults prob_threshold 0.76 extinction 1.5).
|
||
2. Launch the full 16-run matrix via model_server on rep3 films (write trajectories to
|
||
experiments/, NOT /tmp — /tmp gets wiped mid-session and cost us hours). ~28s/eval ×
|
||
~84 evals × 16 = ~10hr. Runner pattern: experiments/run_overnight_rep3.sh but pointing
|
||
optimize_server.py at model_server.
|
||
3. assemble table: best model + expansion effect + misID, from experiments/results/*.json.
|
||
4. Consider upstreaming to KPN++: runtime node setters, push_blocking, node_ptr/functor().
|
||
|
||
## Gotchas that burned time (don't repeat)
|
||
- /tmp scratch gets WIPED mid-session → lost dumps + test files repeatedly. Use experiments/.
|
||
- Verify a launched runner script EXISTS and PRODUCES evals before walking away (a heredoc
|
||
once silently failed to write; a stale-code process ran the old metric for 12h).
|
||
- pgrep/ps "survivors" are often the grep's own shell wrapper — check via /proc cmdline or ps.
|
||
- Running many DE/replay processes in parallel on one GPU → deadlock/thrash. GPU peaks ~35%
|
||
(not saturated) but concurrency>2-3 wedges. Serial-ish is safer.
|