scene-actor-extraction/experiments/SESSION_STATE.md

Session state — X-Ray optimizer + model bake-off (as of 2026-07-18)

Handoff for a fresh session. Everything below is UNCOMMITTED — commit early next session.

What we're doing

Optimizing the scene-actor pipeline's thresholds against Amazon X-Ray ground truth, and running a model bake-off (4 embedding models × gallery-mode × expansion) to answer: is LVFace (455MB) actually best, or just biggest? Does cast-restriction cut false IDs? Does per-film gallery expansion help?

The metric (final form — this is what to use)

scripts/optimizer/second_score.py — UNIFORM PER-SECOND sampling vs X-Ray:

• At each second t: GT = X-Ray scene's cast at t; Pred = actors whose window covers t.
• TPI / FPI / FN counted per second. FPI weighted 10× when the named actor isn't in the film's cast at all (a true misID like naming Archie Yates in a film he's not in) vs an in-cast timing slip.
• FN is fair: only counts gallery-known cast (67% of X-Ray cast have no reference embedding, can't be recognised — see gallery-coverage-gap).
• agreement_rate = mean per-second Jaccard (partial credit: "% of on-screen actors we agree with X-Ray about, over time"). NOT exact-set match.
• Objective = macro-mean per-second weighted F1.

Two DIFFERENT hangs — do not conflate them (corrected 2026-07-18)

(a) The self-inflicted 100% hang (FIXED). replay.py's CLI briefly called replay(..., stop=False) intending to os._exit(0) straight after, to "dodge" teardown. That was wrong: PyNode::stop() is the ONLY thing that sets stop_flag_=true, which is the ONLY exit condition for the source node's run_loop(). Skipping it meant the local net destructor — which runs synchronously when replay() returns, BEFORE main() can reach os._exit — joined a thread that could never stop. A guaranteed hang, not the driver flake. Symptom: every solo replay timed out at 45s and DE reported flat F1=0.0%. FIX: replay(..., stop=True) so PyNode::stop() signals the thread before the join; removed the dead os._exit / unused os import. VERIFIED: 45s guaranteed timeout → clean ~8s completion (3/3), and optimize.py's DE sweep returns correct non-zero metrics (F1 48-66%, matching prior best-so-far). Only 1 isolated per-film timeout in 11 evals × 3 films.

(b) The genuine ROCm flake (rare, tolerated). net.stop()→jthread.join() CAN still hang on a KPN worker stuck mid-rocBLAS-GEMM — a KNOWN ROCm bug (github.com/RadeonOpenCompute/ROCT-Thunk-Interface#56), NOT our code. HSA_ENABLE_SDMA=0 makes it WORSE (breaks the matcher's DMA). It is much rarer than the ~20-30% figure quoted earlier in this session — that number was inflated by (a). The existing subprocess + 45s timeout absorbs it correctly.

Both architectures are usable

• scripts/optimizer/optimize.py + replay.py — subprocess per film, simpler, tolerates the rare true flake via its timeout. NOT broken; good for fallback / quick single-model runs.
• scripts/optimizer/model_server.py + optimize_server.py — ONE persistent net per (model, gallery); replay each film by SWITCHING THE SOURCE (repoint frame list + reset index), change thresholds via runtime SETTERS, os._exit(0) at the very end (after all work, so no destructor-join problem). Higher throughput: skips gallery/build overhead per eval. VERIFIED: "ready", replays, emits metrics, ~30-40s/eval (GPU-bound, films serial). Still the preferred option for the long overnight matrix.

Key C++ changes made (all in the KPN spec-and-tsan branch + our nodes)

1. Runtime setters: IdentityMatcherFunc::set_prob_threshold, SceneTrackerFunc::set_extinction_sec (src/nodes/*). Exposed via sae_kpn: set_prob_threshold(net,name,v), set_extinction_sec(...). Needed ObjectVariantNodeWrapper::functor() + PyNetwork::node_ptr() accessors.
2. Channel::push_blocking() (external/KPN/.../channel.hpp) — lossless backpressure push (waits instead of dropping when full). Exposed on IVariantChannel/VariantChannel; PyNode's run_loop now uses it. Reduced but did NOT fully fix a residual ~0.5% frame loss (25/5915) — the loss is elsewhere (matcher output or reader EOF-race). DECISION: accept it, <0.5% scattered doesn't change per-second F1 or rankings. Don't chase further.
3. dump_embeddings standalone exe + --max-decode-fps (fixes LVFace dump truncation under parallel load). HDF5 gallery fast-load in gallery_store.cpp (18s JSON → 0.06s). scripts/optimizer/json_to_hdf5_gallery.py converts; galleries are .h5 now.

All of KPN, matcher, scene_tracker, bindings need a rebuild: cmake --build build --target sae_kpn sae_gallery dump_embeddings scene_analyze

Data on disk (durable, experiments/)

• experiments/xray/ — X-Ray Zenodo dataset. experiments/dumps/<model>/dump_<slug>.h5 — all 9 films × 4 models, ALL FULL (LVFace re-dumped with --max-decode-fps 8). VERIFY counts match R50 before trusting (LVFace truncated under parallel dumping earlier).
• experiments/galleries/gallery_<model>.h5 (+ restricted//.h5, per-film cast- filtered to Jellyfin's ~15 top-billed — Jellyfin's hard cap, see experiments/README.md).
• experiments/manifests/rep3_<model>_<mode>.json — 3 REPRESENTATIVE films (Lord of War / Scarface / Sound of Metal = clean / ensemble-lookalike / high-coverage) to keep evals fast (~28s vs ~90s for 9). Winner should be re-scored on all 9 after.
• experiments/manifests/films_<model>_<mode>.json — all 9 films.

Salvaged partial results (per-second metric)

• R50 full +expand: F1 66.4% (208 evals, converged) — best so far
• R50 full noexp: 55-60% → expansion helps ~+6-11 recall
• MBF full noexp: 58.6%
• (older scene-metric runs, superseded: R50≈LVFace≈MBF ~85%, restricted>full, LVFace not worth its size — but those used the OLD scene-union metric, redo with per-second.)

TO DO next session

1. COMMIT everything first (logical chunks: KPN setters+push_blocking; sae_kpn+dump exe; HDF5 gallery; optimizer scripts; per-second metric; experiments manifests/results/docs + tuned config.hpp defaults prob_threshold 0.76 extinction 1.5).
2. Launch the full 16-run matrix via model_server on rep3 films (write trajectories to experiments/, NOT /tmp — /tmp gets wiped mid-session and cost us hours). ~28s/eval × ~84 evals × 16 = ~10hr. Runner pattern: experiments/run_overnight_rep3.sh but pointing optimize_server.py at model_server.
3. assemble table: best model + expansion effect + misID, from experiments/results/*.json.
4. Consider upstreaming to KPN++: runtime node setters, push_blocking, node_ptr/functor().

Gotchas that burned time (don't repeat)

• /tmp scratch gets WIPED mid-session → lost dumps + test files repeatedly. Use experiments/.
• Verify a launched runner script EXISTS and PRODUCES evals before walking away (a heredoc once silently failed to write; a stale-code process ran the old metric for 12h).
• pgrep/ps "survivors" are often the grep's own shell wrapper — check via /proc cmdline or ps.
• Running many DE/replay processes in parallel on one GPU → deadlock/thrash. GPU peaks ~35% (not saturated) but concurrency>2-3 wedges. Serial-ish is safer.