12 Commits

Include renames

Author	SHA1	Message	Date
dtourolle	a5c016833d	fix: install channel callbacks before any node runs ... ThreadSanitizer reported ten data races on a plain multi-node network, all the same one: Read in Channel::try_push -> push_callback_() (worker thread) Write in Channel::set_push_callback -> push_callback_ = ... (main thread) A node's push and space callbacks are std::function members living on channels it shares with its neighbours. register_callbacks() wrote them from inside start(), and a network starts its nodes one at a time — so by the time node N is being started, nodes 1..N-1 are already running and pushing into N's input channel, reading the very std::function that start() is assigning. Concurrent read and write of a std::function is a data race on its vtable pointer and buffer, not a benign one. This is the cause of the symptom a8cfe73 patched. That commit found nodes missing their startup wake because "enable_inputs() opens the channel several statements before register_callbacks() installs the push callback", and fixed it by re-asking the question with on_input_ready(). The gap it described is this race: the callback is not merely late, it is being written while another thread reads it. INode gains prepare(), which installs callbacks and starts nothing. Networks call it on every node before starting any of them, so every write happens while the pipeline is idle and the callbacks are read-only once it is live. start() calls prepare() itself when a node is used standalone, and prepare() is idempotent so both paths are safe. The flag is never cleared: the callbacks capture `this` and stay valid across a restart, so re-registering them would only add a pointless write to a live channel. a8cfe73's on_input_ready() stays, and is still needed — a network starts nodes one at a time, so an upstream node can still push into this one between its prepare() and its start(), where on_input_ready() returns early on stop_flag_ and the empty->non-empty edge is spent. It is now a level-triggered check against a benign ordering rather than cover for a race. Verified with -DKPN_SANITIZER=thread: ten races before, none of these after, across the unit suite and the contended channel stress suite. One unrelated race remains, on overlapping fire_once invocations; it is pre-existing and is fixed separately.	2026-08-05 13:39:26 +02:00
dtourolle	f53af260a2	fix: make the submit gate a single atomic ... 9c5ce5f established "a node never sleeps with a wake outstanding" and implemented it as two independent atomics: queued_ for "a firing is in flight", wake_pending_ for "a wake arrived during one". Two variables cannot express that invariant, because the release side has to read and write both and a wake can land in between: producer (try_submit) worker (release_and_recheck) ------------------------ ---------------------------- CAS reads queued_ == true, fails queued_.store(false) wake_pending_.exchange(false) -> false wake_pending_.store(true) queued_ false, wake_pending_ true, nothing running and nothing scheduled — exactly the state the invariant forbids. This is not a memory-ordering subtlety; the interleaving holds under seq_cst. SubmitGate replaces both with one atomic over three states, so "idle" and "wake outstanding" are the same variable and no interleaving can produce both. A release that finds a recorded wake keeps the claim and hands it to the next firing, so the node is never momentarily idle while a submission for it is in flight. What this does not do is fix a reproducible hang. Every current call site follows release_and_recheck() with a level re-check — on_input_ready(), or outputs_have_space() on the parked path — which rediscovers the state a lost wake would have signalled. The bug is masked, and I could not write a node-level test that fails before and passes after; claiming otherwise would be dishonest. The masking is a property of the call sites, not the mechanism: any future early return that forgets its re-check reintroduces a silent hang, and the pipeline has already been round that loop twice (28e0667, then 9c5ce5f, each of which moved the stall rather than removing it). So the tests are structural. The state machine is pinned by contract tests, and the defect it replaces is pinned by demonstration: LegacyGate in the test file is the old protocol with a seam between the failed CAS and the wake record, which makes the loss deterministic rather than something to wait for. It also keeps the defect on record now that the code implementing it is gone. Also ignores build-*/ so a sanitizer build tree cannot be committed by accident, which this commit did on its first attempt.	2026-08-05 13:22:03 +02:00
dtourolle	5628447ea8	fix: never self-move the parked output tuple ... push_outputs ended with else pending_ = std::move(result); and the retry path calls it as push_outputs(std::move(*pending_), …), so on that path `result` is the parked tuple itself. The assignment was a self-move-assignment. std::tuple's is elementwise, and libstdc++'s std::vector does not guard against self-move: _M_move_assign swaps its data into a temporary, which is then destroyed. The vector ends up empty. So the first park was clean — the argument there is a local temporary — and the second erased the payload. The value was still delivered, still in order, still counted, just empty. Downstream cannot distinguish that from a frame on which the node genuinely found nothing, which is why it would never surface as an error: in scene-actor-extraction it reads as "no faces in this frame" and the run completes with a quietly wrong answer. Scope, stated precisely because I first got it wrong: this needs a node with *two or more* outputs. With one output the only thing that resubmits a parked node is that output's own space callback, which by definition fires when there is room, so the retry always succeeds and never reassigns. With two, output A draining resubmits the node while output B is still full — the retry skips A (already delivered, tracked in pending_done_) and fails on B, and that is the reassignment that eats B's payload. Every node in the scene-actor-extraction pipeline currently has exactly one output, and the fanout is a separate class that does not use pending_, so this is latent there rather than active. It is reachable by any multi-output node under backpressure, which the library supports and documents. Verified in both directions: on 6a4f45f the parked payload arrives with size 0; here it arrives intact. The test drives raw channels rather than consumer nodes so each step is forced rather than raced, and both channels are capacity 1 — Channel fires the space callback only on the full->not-full edge, so a roomy channel A would never resubmit the node and the retry would never happen at all.	2026-08-05 13:10:41 +02:00
dtourolle	6a4f45f111	fix: a filter or router must not drop on a full output ... RouterNode and FilterNode were the last nodes on a data path still using the throwing push() and swallowing the result: try { out_ch_->push(val); } catch (const ChannelOverflowError&) {} 6595e6e made node outputs lossless, 28e0667 stopped them parking a worker, a8cfe73 did the same for FanoutNode. These two were in none of them. For ordinary values that is the familiar silent-loss problem: a dropped item does not degrade a downstream result, it silently changes one, and the consumer cannot tell it happened. For a sentinel it is a hang. EOF is what tells every downstream node to shut down and there is nothing after it to retry, so a filter that passes EOF by predicate but drops it by backpressure produces a pipeline that never terminates. scene-actor-extraction's decimator is exactly that shape — `if (f.eof) return true;` in the predicate, feeding a chain whose slowest node is an ONNX embedder, so the output is reliably full when EOF arrives. Everything downstream then waits forever for a token that was discarded, and the run has to be killed. Both now route sentinels out-of-band via push_sentinel, which consumes no ring capacity and cannot overflow, and retry ordinary values until taken. Like FanoutNode and unlike a pool node, these own a private thread, so waiting costs no scheduler worker and needs no space-callback park; stop_flag_ is rechecked every pass so teardown cannot hang on a full output. Time spent parked is charged to blocked rather than exec, so a held-up node does not report as busy. An out-of-range router selector still drops by design — that item was routed nowhere, which is not the same as lost. is_sentinel_value moves from pool_node.hpp to traits.hpp. Every node type that forwards a value needs it; these two not having it is the bug. Verified in both directions. On c73edff the new case delivers 6 of 40 values and never sets saw_eof; here it delivers 40 and terminates. EOF is emitted exactly once, as a real source does — a test source that re-offered it would mask the bug, since a later attempt could find the channel drained. Note for downstream: the decimator is now a backpressure point rather than a relief valve, so the source throttles to the face branch instead of quietly thinning it. That is the intended behaviour, but it changes the shape of a loaded run and is worth a benchmark comparison on a known clip.	2026-08-05 12:59:39 +02:00
dtourolle	c73edffe5c	chore: delete the unreachable duplicate of the parked-retry block ... PoolNode::fire_once carried the pending_ retry block twice, verbatim. The first copy returns on every path through it — parked, drained, or not pending at all — so the second was dead code from the moment it appeared. PoolObjectNode, which is otherwise a line-for-line twin of PoolNode, has it once. No behaviour change; the deleted 25 lines were unreachable. Worth doing before the fixes queued behind it, each of which has to be applied once per copy of this function. The duplication is a symptom: PoolNode and PoolObjectNode are ~400 lines of near-identical code maintained by parallel edit, and a block getting pasted twice into one of them is exactly the failure that arrangement invites. Factoring the shared body out is a larger change and wants its own review.	2026-08-05 12:51:21 +02:00
dtourolle	a8cfe7300a	fix: a lossless fanout, a node that starts awake, and the instrumentation that found them ... Three changes from one debugging session on the intermittent wedge, kept together because the instrumentation is what made the other two findable. **Fanout was never made lossless.** 6595e6e made node outputs lossless and 28e0667 stopped them parking a worker; FanoutNode was in neither and kept `catch (ChannelOverflowError&) {}` per output. Whichever branch fell behind lost items, silently, by an amount that depended on timing — so two runs of the same input could disagree. deliver() now retries each output independently until it is taken, rechecking stop_flag_ every pass so teardown cannot hang on a full output. A fanout owns a private thread, so waiting costs no scheduler worker. **A node could start with a wake already outstanding.** start() enables the input channel several statements before it installs the push callback, and StaticNetwork starts nodes sources-first, so an upstream node is already firing into the gap. A push landing there is accepted by the ring but wakes nobody: push_callback_ fires only on the empty→non-empty transition, and at that instant the callback is null. Every later push sees a non-empty ring and stays silent, so the node is never submitted. Asking on_input_ready() once at the end of start() converts the missed edge into a state check. The signature is distinctive — zero items delivered, not a stall partway. Under `ctest -j4` on a loaded machine it reproduced 7 times in 24 and never in 10 unloaded runs, which is almost certainly the "~1 run in 20" hang 28e0667 recorded as known-incomplete. **NodeSnapshot now carries scheduling state and a true exec total.** queued and wake_pending make the 9c5ce5f invariant observable at runtime; it could previously only be inspected in a debugger, and the bug does not reproduce under one. total_exec_us is a real sum — frames × ema_exec_us tracks the tail of a run, not the whole of it, and diverges badly on a workload whose per-frame cost varies. Both are exposed over the web debug JSON so a wedged pipeline can be interrogated without attaching to it.	2026-08-05 12:40:03 +02:00
dtourolle	9c5ce5f34a	fix: never drop a wake — a node must not sleep with one outstanding ... 28e0667 stopped nodes blocking a worker on a full output, but replaced an intermittent hang with a quieter one: the pipeline still wedged about 2 runs in 30, now with every worker idle in pthread_cond_wait rather than asleep in a push. Nothing was blocked; nothing had been woken. try_submit discarded any wake arriving while queued_ was up: if (queued_.compare_exchange_strong(expected, true, ...)) scheduler_->submit(...); // else: silently gone Wakes are edge-triggered — a channel fires its space callback once, on the transition — so a dropped one never returns. A node could park a value, release its worker, and sleep forever holding exactly the output its consumer was waiting for, while its producer parked on an input channel that would never drain. try_submit now records the drop in wake_pending_, and release_and_recheck() consumes it at every site that releases a node, giving one invariant: a node never sleeps with a wake outstanding. This subsumes the two ad-hoc re-checks added for the parked-retry and normal push paths, which only moved the stall (1211 items to 6472) because each new early return was a fresh chance to drop a wake. self_stop keeps a plain store — honouring a pending wake there would resubmit a dead node. Adds "a saturated chain never stalls", which reproduces this in about a second where the pipeline needed ~30 runs. It asserts *progress does not freeze* rather than a completion total: capacity-1 channels are slow, and slow must never be reported as wedged. Verified in both directions — it stalls after 1211 items on 28e0667 and passes here. The existing chain test could not catch it: 40 items drain before any strand occurs, and its producer emits forever, so fresh input keeps re-triggering on_input_ready() and flushing the stranded value. Tests: 122/122.	2026-08-02 18:33:17 +02:00
dtourolle	28e06675f5	fix: park nodes on a full output instead of blocking the worker ... push_blocking parked a scheduler worker inside the push. Nodes own a private single-thread pool, so the parked thread was the only one that could drain that node's own input — hold-and-wait, and under sustained backpressure four nodes of a five-node chain slept in nanosleep at once. channel.hpp already warned about this for sentinels; it applies just as much to data pushes. The scheduler was purely input-driven: on_input_ready() wakes a node when input arrives, with no counterpart for "my output has room". Lacking that signal, blocking the thread was the only way to handle a full output. This adds the missing half. - Channel::try_push + has_space + set_space_callback; the callback fires from both pop() and try_pop_now(). - PoolNode/PoolObjectNode keep a one-slot pending_ buffer with per-element done flags, so a retry cannot duplicate an already-accepted element. One slot suffices because queued_ admits at most one fire_once per node. - The re-check after clearing queued_ closes the lost-wakeup race where a space callback fires while the flag is still up and is swallowed. Two bugs surfaced once nodes actually parked, both fixed here: - pop_one reports an *empty* channel as ChannelClosedError, which is also the node's "upstream finished, self-stop" signal. A node woken by output space with empty inputs therefore killed itself. fire_once now releases the worker when its inputs are not ready rather than falling through. - The drained-park path resubmitted unconditionally instead of via on_input_ready(), firing nodes with nothing to read. compute_priority is now output-aware: mean output fill is deducted from mean input fill, mapped as 0.5·(1 + in - out). Input fill alone asks only "how much work is waiting for me"; a node whose outputs are already full cannot deliver, so running it just parks it again and wastes the slot while the node that would drain that channel waits behind it. The scheduler now favours whoever is furthest downstream of a bottleneck. Also adds a network-level error listener. A node's exception was discarded at the node boundary and survived only as a Closed event, which reports that a node stopped but not why — that missing detail is what made the above slow to diagnose. INode::set_network_error_callback plus StaticNetwork::set_error_handler forward it to the application. Tests: 121/121. test_backpressure_deadlock drives a five-node chain with capacity-2 channels against a slow sink and fails on the old code. The four test_pool_node overflow tests now assert parking rather than the removed drop-and-report behaviour. Known-incomplete: a rare hang remains, roughly 1 run in 20 against a 300s timeout, down from every run failing. Committed because the fix is a large strict improvement and the residual case needs its own reproduction.	2026-07-31 22:40:07 +02:00
dtourolleandClaude Opus 5	6595e6e925	fix: node outputs block instead of dropping on a full channel ... Every node output used the throwing push(), so a consumer falling behind cost values rather than time. push_blocking() already existed on Channel and OutputPort — "wait for the consumer to drain instead of dropping; the producer just runs slower" — but nothing called it. A dropped frame does not degrade a downstream result, it silently changes one, and the consumer has no way to tell it happened. For any pipeline whose output is a claim about its input, that is corruption rather than degradation. Safe because sentinels are already handled out-of-band, above this path: only data blocks, so the EOF token that unwinds the network can always overtake a stalled data path. That is exactly the hold-and-wait deadlock the push_sentinel comment warns about, and the reason it is not reachable here. Measured on a downstream consumer (face pipeline, 77s clip at 5 fps, expected 385 sampled frames): before 65 frames written, 320 dropped at one node, 29s after 385 frames written, 0 dropped, 17s Faster, not slower — a dropped frame has already cost its decode, and the overflow exception cost more. Two consecutive runs now produce byte-identical output, which they did not before: what got dropped depended on timing, so the same command could yield different results. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-07-31 11:31:53 +02:00
dtourolleandClaude Opus 4.8	19f5a2b0ae	Deliver EOF sentinels out-of-band to prevent teardown deadlock ... Channel::push() drops values on overflow (the intended backpressure policy for data), and PoolNode swallows the resulting ChannelOverflowError. For a control sentinel like EOF this is fatal: a single dropped EOF under backpressure wedges every downstream pop() forever, so the pipeline never tears down. Deliver sentinels out-of-band instead. Channel::push_sentinel() stores the token in a dedicated slot that does not consume ring capacity, so it can never overflow and — crucially — never blocks the caller. That non-blocking property is essential: each KPN node has a single worker thread, so a *blocking* push would park that thread and stop it draining its own input, cascading into a hold-and-wait deadlock under backpressure. The consumer's pop()/try_pop_now() drain the ring first, then deliver the sentinel, so it always arrives after every value pushed before it. approx_size() (which node readiness checks call) counts a pending sentinel as consumable work, so a channel carrying only a sentinel still schedules its consumer's next fire — without this the token would sit undelivered and the pipeline would still deadlock at teardown. PoolNode/PoolObjectNode route values carrying an eof flag (direct .eof or nested .source.eof) through push_sentinel via a SFINAE-safe is_sentinel_value trait; all other values keep the existing lossy throwing push. The trait compiles to false for types without an eof convention, so this is a no-op for pipelines that don't use one. Verified end-to-end: scene_analyze now reaches EOF, flushes its output, and exits cleanly instead of hanging. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-04 00:30:40 +02:00
dtourolle	6f384dc4b5	Added callbacks for node errors and fifo overflow ... Add new doc system which should/might deploy to pages.	2026-06-19 22:26:39 +02:00
dtourolle	f6bcaa15b0	Performance improvements, better readme and complete python bindings	2026-05-12 21:23:33 +02:00