Twenty commits, of which the one that matters for this repo is the filter
and router losslessness. FilterNode and RouterNode were the last nodes on a
data path still using the throwing push() and swallowing the exception, so
`decimate` — which passes EOF by predicate — discarded that token whenever
its output was full. Which it reliably is: the embedder is the slowest node
in the chain. Nothing downstream ever received EOF, `done` was never set,
and the run had to be killed. That is the wedge.
The rest, in rough order of how much they affect a run here:
- A sentinel could be delivered ahead of a value still queued behind it,
losing that value to any consumer treating EOF as a hard stop.
- Two fire_once invocations for one node could overlap, which breaks the
one-slot park: a parked value can be overwritten with no drop recorded.
- A node's push/space callbacks were written while a running neighbour read
them — ten ThreadSanitizer races, and the root cause of the startup lost
wake that AR-004 records as closed by a level-triggered re-check.
- shutdown() polled every channel in the graph with no deadline, and could
fail to terminate outright on an index underflow in the fill calculation.
- Submitting to a stopped pool indexed a cleared vector: a segfault, which
reproduced 12 runs in 20 once sources stopped before their consumers.
- stop() returned while a firing was still touching the node's members.
- Idle pool workers burned ~6.5 cores while one task ran (1991 ms of CPU
against 0.4 ms). Latent here, since every node owns a private one-thread
pool, but not for anything using a shared one.
Two consequences worth holding onto. `decimate` is now a backpressure point
rather than a relief valve, so the source throttles to the face branch
instead of quietly thinning it — the intended behaviour, but it changes the
shape of a loaded run and has not been benchmarked. And a channel now
carries at most one undelivered sentinel; a second is refused and reported
rather than silently overwriting the first.
Verified: KPN 148/148 with ThreadSanitizer clean across six runs, this repo
138/138 with every target building warning-free against the new headers.
TRACES: AR-004 | SR-002
Bali was chosen because the TRECVID DVU set ships character mugshots, but
its reference crops are unusable at scale: median detected face 27 px
against a 69 px maximum, so every reference was upscaled 4x or more past
what the embedder was trained for (AR-011). A 66 px floor left 2 of 69
references; no threshold exists that both keeps the faces in distribution
and leaves enough of them to calibrate.
SuperHero is 69 px median and 241 px max. Its gallery builds at a 66 px
floor with 14 references over 5 characters, and calibrates on its own
(a=15.2867 b=-4.98633, 100% train accuracy) instead of borrowing constants.
Measured on the fused 17-minute film, one stream rather than per-scene
clips so presence windows cross real scene boundaries as SR-002 intends:
precision 1.00, recall 0.65, F1 0.79 — 13 true positives, 0 false
positives, 7 misses. Every out-of-gallery character was declined rather
than forced onto a nearest match. The misses are the short scenes (14 s,
38 s, 27 s), consistent with per-track accumulation needing sightings.
- build_gallery gains --min-face-px, filtering the *detected face* rather
than the crop. The DVU images are scene crops, not mugshots, so crop
dimensions say nothing about face scale. A poisoned reference is
permanent in a way a bad frame is not: it corrupts every future match
against that identity.
- scripts/fetch_dvu.sh fetches mugshots, scene graphs and segmentation for
any DVU film. NIST names the same film three different ways, so KG_DIR
and KG_FILE are overridable rather than derived. This exists as a script
because the first copy of this data was assembled ad hoc in /tmp and was
lost with it, taking the working gallery along.
- Replay fixtures move to the artifact registry: push/pull_artifacts.sh
gain a replay-fixtures target, and tests/fixtures/dumps/.gitignore keeps
them out of git. superhero.h5 is ~9 MB and regenerating it needs the
film, the models and a GPU — none of which CI has. The gallery ships
with the dumps, since a dump only replays against the gallery it was
produced with.
- AR-012 and AR-013 coverage is ported onto the new fixture rather than
dropped with the Bali cases: 12369 assertions, up from 7991, since the
film is an order of magnitude larger than the clips.
Suite: 15679 assertions, 101 test cases.
TRACES: AR-011, AR-012, AR-013 | VR-001, VR-005 | SR-002
Adopts the KPN backpressure fix (28e0667) and registers the application
error listener it exposes.
`push_blocking` parked a scheduler worker inside the push. Each ObjectNode
owns a private single-thread pool, so the parked thread was the only one
that could drain that node's own input: under sustained backpressure
frame_source, camera_pos, face_detector and face_aligner all slept in
nanosleep at once and the pipeline stopped. Nodes now hold the value,
release the worker, and resume on a channel space-callback.
main.cpp registers set_error_handler so a node that throws names itself
and its exception. Previously the exception was discarded at the node
boundary and survived only as "node 'x' stopped unexpectedly", which says
that a node died but not why — the missing detail that made this slow to
diagnose.
AR-004 drops from Done to Mostly. Two gaps are recorded rather than
claimed fixed: a hang surviving at roughly 1 run in 20 against a 300 s
timeout (down from every run failing), and FanoutNode still dropping on
overflow instead of parking, which sheds frames on the AR-010 scene join
precisely when the dense branch falls behind.
TRACES: AR-004 | SR-002
The recorded pointer was be6e922 — the backpressure fix as originally committed,
before it was rebased onto KPN master. That commit exists on no pushed branch,
so a fresh clone of this branch could not fetch the submodule at all.
Now 6595e6e, the same change on KPN master.
Worth noting for next time: rebasing a submodule commit after the superproject
has already recorded it silently invalidates the pointer. Nothing in the
superproject's status shows it, because the submodule working tree is clean and
at a valid commit — just not the one recorded.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
TRACES: AR-004 | SR-002
Picks up the KPN fix: node data outputs block on a full channel rather than
dropping. Sentinels stay out-of-band, so EOF can always overtake a stalled data
path and the hold-and-wait deadlock that comment warns about is not reachable.
Verified on a 77s clip at 5 fps, which should yield 385 sampled frames:
before 65 written, 320 dropped, 29s, two runs differ
after 385 written, 0 dropped, 17s, two runs byte-identical
The determinism is the part that matters. Golden fixtures were impossible while
what got dropped depended on timing; VR-001 fixture generation is unblocked by
this, and so is the CI replay strategy that depends on it.
Faster rather than slower, which is worth recording because the intuition runs
the other way: a dropped frame has already cost its decode, and the overflow
exception cost more still.
AR-004 is not fully closed. Channel capacity remains a count of items, while a
face carries a 112x112 crop and a 512-float embedding — so a crowded frame
occupies far more memory per slot than a sparse one. Bounding by bytes in
flight is the remaining half, and it matters once max_faces is removed (AR-003).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
TRACES: AR-004, VR-001 | SR-002
Gallery format switches from JSON to HDF5 exclusively (JSON read-only kept for
back-compat): save_gallery always writes HDF5, and the fitted Platt-sigmoid
calibration (a, b, valid, hash) is now embedded directly in the gallery file
instead of a sidecar .calib_cache.json — identity_matcher reads it from the
loaded gallery and writes back only when the embeddings actually changed
(hash mismatch), skipping the O(n^2) refit otherwise.
Also includes: TensorRT inference backend support (ort_backend.cpp,
trt_backend.cpp), gemm_backend improvements, TransNetV2-based scene-boundary
detection wired through frame_source/face_tracker/main, and CMake build
target updates for the new sources.
Bumps the KPN submodule to feature/persistent-pipeline-reuse (push_blocking
backpressure, node_ptr/node_stats introspection, ObjectVariantNodeWrapper for
stateful functors) — needed by the optimizer's sae_kpn Python bindings.
Register a KPN event handler in both scene_analyze and scene_preview:
- Overflow events accumulate per-node dropped-frame counts, printed on exit.
- A Closed event from any node other than result_sink at EOF means a stage
died; trip an atomic so the main loop bails out instead of hanging on
'done' forever, and exit non-zero.
Bumps external/KPN to the commit that exposes set_event_handler / NodeEvent.
dtourolle
