From 5c6603e63bec96802497fdf3eefc92ddd2ec7aef Mon Sep 17 00:00:00 2001 From: Duncan Tourolle Date: Fri, 31 Jul 2026 22:42:19 +0200 Subject: [PATCH] fix(kpn): park on full outputs; surface node exceptions MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adopts the KPN backpressure fix (28e0667) and registers the application error listener it exposes. `push_blocking` parked a scheduler worker inside the push. Each ObjectNode owns a private single-thread pool, so the parked thread was the only one that could drain that node's own input: under sustained backpressure frame_source, camera_pos, face_detector and face_aligner all slept in nanosleep at once and the pipeline stopped. Nodes now hold the value, release the worker, and resume on a channel space-callback. main.cpp registers set_error_handler so a node that throws names itself and its exception. Previously the exception was discarded at the node boundary and survived only as "node 'x' stopped unexpectedly", which says that a node died but not why — the missing detail that made this slow to diagnose. AR-004 drops from Done to Mostly. Two gaps are recorded rather than claimed fixed: a hang surviving at roughly 1 run in 20 against a 300 s timeout (down from every run failing), and FanoutNode still dropping on overflow instead of parking, which sheds frames on the AR-010 scene join precisely when the dense branch falls behind. TRACES: AR-004 | SR-002 --- docs/requirements.md | 8 ++++---- external/KPN | 2 +- src/main.cpp | 28 ++++++++++++++++++++++++---- 3 files changed, 29 insertions(+), 9 deletions(-) diff --git a/docs/requirements.md b/docs/requirements.md index 46076bc..ae001c5 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -31,7 +31,7 @@ Status: `Done` · `In Progress` · `Planned` · `TBD` · `Withdrawn` | AR-001 | Detect faces in sampled frames; emit bbox, confidence, 5-point landmarks in original pixel space | SR-002 | High | Done | | AR-002 | Minimum face size **40×40 px** (VR-013 measured end to end; VR-005's 32 px is an embedder-only upper bound), expressed in **original** resolution (decoupled from `dense_scale`) | SR-002 | High | Planned | | AR-003 | No fixed per-frame face cap — crowd scenes must not lose background cast | SR-002 | Medium | **Done** — `max_faces` defaults to 0 (no cap); the matcher batches through its GEMM buffer instead of throwing | -| AR-004 | Backpressure: unbounded faces/frame absorbed by slowing, never by dropping or throwing | SR-002 | High | **Done** — KPN node outputs use `push_blocking`; sentinels stay out-of-band. Verified: 385/385 frames, 0 drops, byte-identical across runs | +| AR-004 | Backpressure: unbounded faces/frame absorbed by slowing, never by dropping or throwing | SR-002 | High | **Mostly** — node outputs *park* on a full channel: the value is held, the worker released, and a channel space-callback resumes the node. Replaces `push_blocking`, which parked a scheduler worker inside the push and, with one thread per node, stopped that node draining its own input. Verified: 385/385 frames, 0 drops. **Gap:** a rare hang survives, ~1 run in 20 at a 300 s timeout (was: every run). `FanoutNode` still drops on overflow (`fanout.hpp:129`) rather than parking, so the AR-010 scene join sheds frames exactly when the dense branch falls behind | | AR-005 | Align to 112×112 via ArcFace 5-point similarity transform, fitted by **Umeyama least squares over all five points** (as InsightFace does) — never a robust fit, which would discard the landmarks AR-030 reads | SR-002 | High | **Done** — `umeyama_similarity()`. The RANSAC fit it replaces disagreed by a median 17 source px on 400 headshots, 83.5% of crops embedding below cos 0.99, and was unstable and RNG-driven: rebuilding caught 1614 near-duplicates against the original build's ~100. **All galleries rebuilt** (2456 actors, 10254 embeddings); measured separation gain is small (0.583 → 0.590), so recorded accuracy figures should be re-run but are not expected to move far | | AR-006 | 512-d L2-normalised embeddings, batched | SR-002 | High | Done | | AR-007 | Associate detections by IoU + embedding, with **frame-dependent** weighting | SR-002 | High | **Done** — `track_alpha` is the base for ordinary frames; drops to embedding-only on cut/boundary and for dormant tracks | @@ -45,13 +45,13 @@ Status: `Done` · `In Progress` · `Planned` · `TBD` · `Withdrawn` | AR-015 | Two live tracks owned by one actor ⇒ treat as a detected cut, re-associate | SR-002 | Medium | **Done** — reverse index detects it on the causing update; counted | | AR-016 | All tracks closed at EOF — a film ends with faces on screen | SR-002 | High | **Done** — `flush()`, idempotent, closes at last sighting or final tick | | AR-017 | Every presence claim carries its belief and identification route | SR-002 | High | **Done** — `DeadTrack` carries belief and observation count | -| AR-018 | Per-subject embedding store with banded admission (novel enough, safe enough) | SR-005 | Medium | **Done** — banded admission in probability space; replaces `expand_novelty_sim`. Rejections counted | +| AR-018 | Per-subject embedding store with banded admission (novel enough, safe enough) | SR-005 | Medium | **Done** — banded admission in probability space, bounds from `expand_band_lo/hi`; the lower bound re-asked pairwise at promotion, since `admit` compares only against the nearest member and a drifting track can chain past it. Retires `expand_novelty_sim` and `expand_track_spread_max` — novelty is now the eviction ordering, not a threshold. Rejections counted. Bounds unswept (VR-007) | | AR-019 | Per-film gallery annex from owned tracks; acquires the non-frontal views TMDB lacks | SR-005 | Medium | **Done** — all three discontinuity signals clear the buffers; ownership comes from the registry, not a second local tally | | AR-020 | Deferred re-identification of unknown tracks against the final expanded gallery | SR-005 | High | Planned | | AR-021 | Cluster unknown tracks into one entity per person, under temporal cannot-link constraints | SR-005 | Medium | Planned | | AR-022 | Capture still-unidentified tracks: embeddings, metadata, **context crops** for human review | §4 | Medium | Planned | | AR-023 | Fit sigmoid calibration from intra/inter similarity distributions | SR-002 | High | Done | -| AR-024 | **Always the calibrated probability, never a raw cosine** — exceptions recorded | SR-002 | High | **Done** — association and accumulation both in probability space; `track_max_embed_dist`, `cut_revive_sim` retired | +| AR-024 | **Always the calibrated probability, never a raw cosine** — exceptions recorded | SR-002 | High | **Done** — association, accumulation and expansion all in probability space; `track_max_embed_dist`, `cut_revive_sim`, `expand_novelty_sim`, `expand_track_spread_max` retired | | AR-025 | Per-track Bayesian accumulation in log-odds, with correlated-observation discounting | SR-002 | High | **Done** — log-odds accumulation with correlation discounting owned by the registry, `src/evidence_discount.hpp` | | AR-026 | All similarity computed as GEMM, including annex and deferred pass | SR-001 | High | In Progress | | AR-027 | Throughput acceptable for **arbitrary** gallery size | SR-001 | High | Planned | @@ -91,7 +91,7 @@ Status: `Done` · `In Progress` · `Planned` · `TBD` · `Withdrawn` |---|---|---|---|---| | GR-001 | Build gallery from Jellyfin library cast, TMDB profile fallback | SR-001, SR-005 | High | Done | | GR-002 | Incremental `--merge` refresh without re-embedding known actors | PR-003 | High | Done | -| GR-003 | Report coverage: zero-image actors, under-referenced actors, dedup, calibration PDFs | SR-001 | Medium | Planned | +| GR-003 | Report coverage: zero-image actors, under-referenced actors, dedup, calibration PDFs | SR-001 | Medium | **Done** — `gallery/gallery_report.hpp`, written next to the gallery by `build_gallery`. Zero-usable-image actors come from the build audit, which a stored gallery cannot reconstruct; also `distinct_references`, `duplicates_removed`, and the intra/inter distributions the calibration fits and would otherwise discard | | GR-004 | Stamp embedder identity into the gallery; **hard startup error** on mismatch | SR-001 | High | **Done** — basename + SHA-256 + `embed_dim`; mismatch fatal with no bypass, unstamped warns unless `--require-gallery-stamp`; `scripts/stamp_gallery.py` migrates in place | | GR-005 | Gallery data never leaves the instance | **SR-005** | High | Done | | GR-006 | Provenance tiers: baked / harvested / confirmed, distinguishable per embedding | SR-005 | High | Planned | diff --git a/external/KPN b/external/KPN index 6595e6e..28e0667 160000 --- a/external/KPN +++ b/external/KPN @@ -1 +1 @@ -Subproject commit 6595e6e925785b8da529ce0a34208738b639c670 +Subproject commit 28e06675f5f439b77b9cb79033ad98fa423af3e5 diff --git a/src/main.cpp b/src/main.cpp index 925d082..fa5d89d 100644 --- a/src/main.cpp +++ b/src/main.cpp @@ -39,8 +39,8 @@ // --max-faces max faces kept per frame (default: 10) // --expand-gallery enable per-film gallery expansion from track continuity // --expand-buffer per-track diversity buffer size (default: 20) -// --expand-novelty-sim promote only views with best sim < f (default: 0.55) -// --expand-spread-max reject track if buffer spread > f (default: 0.60) +// --expand-band-lo

store admission floor, P(same person) (default: 0.90) +// --expand-band-hi

store admission ceiling, P(same person) (default: 0.95) // --expand-min-anchor accepted frames before a track confirms (default: 3) // --expand-debug-dir

dump promoted mugshots + embeddings here (SAE_DEBUG) // (SAE_DEBUG only) @@ -147,8 +147,8 @@ static Config parse_args(int argc, char** argv) { else if (arg("--anneal")) cfg.anneal_sec = std::stod(next()); else if (arg("--expand-gallery")) cfg.expand_gallery = true; else if (arg("--expand-buffer")) cfg.expand_buffer_size = std::stoi(next()); - else if (arg("--expand-novelty-sim")) cfg.expand_novelty_sim = std::stof(next()); - else if (arg("--expand-spread-max")) cfg.expand_track_spread_max = std::stof(next()); + else if (arg("--expand-band-lo")) cfg.expand_band_lo = std::stof(next()); + else if (arg("--expand-band-hi")) cfg.expand_band_hi = std::stof(next()); else if (arg("--expand-min-anchor")) cfg.expand_min_anchor_frames = std::stoi(next()); else if (arg("--expand-debug-dir")) cfg.expand_debug_dir = next(); else if (arg("--trt-cache")) cfg.trt.cache_dir = next(); @@ -281,6 +281,26 @@ int main(int argc, char** argv) { } }); + // Report *why* a node died. A Closed event alone says only that one + // stopped; the exception it carried is what identifies the fault, and + // without this listener it is discarded at the node boundary. Returning + // false keeps the existing semantics — the node still stops and the + // Closed handler above still aborts the run — but the run now names the + // cause instead of leaving it to be reconstructed from a debugger. + net.set_error_handler( + [&](std::string_view node_name, std::exception_ptr eptr) { + std::string what = "unknown exception"; + try { + if (eptr) std::rethrow_exception(eptr); + } catch (const std::exception& e) { + what = e.what(); + } catch (...) { + } + std::lock_guard lk(event_mtx); + std::cerr << "[main] node '" << node_name << "' threw: " << what << "\n"; + return false; + }); + std::cerr << "[main] starting pipeline…\n"; net.start();