Three changes from one debugging session on the intermittent wedge, kept
together because the instrumentation is what made the other two findable.
**Fanout was never made lossless.** 6595e6e made node outputs lossless and
28e0667 stopped them parking a worker; FanoutNode was in neither and kept
`catch (ChannelOverflowError&) {}` per output. Whichever branch fell behind
lost items, silently, by an amount that depended on timing — so two runs of
the same input could disagree. deliver() now retries each output
independently until it is taken, rechecking stop_flag_ every pass so
teardown cannot hang on a full output. A fanout owns a private thread, so
waiting costs no scheduler worker.
**A node could start with a wake already outstanding.** start() enables the
input channel several statements before it installs the push callback, and
StaticNetwork starts nodes sources-first, so an upstream node is already
firing into the gap. A push landing there is accepted by the ring but wakes
nobody: push_callback_ fires only on the empty→non-empty transition, and at
that instant the callback is null. Every later push sees a non-empty ring
and stays silent, so the node is never submitted. Asking on_input_ready()
once at the end of start() converts the missed edge into a state check.
The signature is distinctive — zero items delivered, not a stall partway.
Under `ctest -j4` on a loaded machine it reproduced 7 times in 24 and never
in 10 unloaded runs, which is almost certainly the "~1 run in 20" hang
28e0667 recorded as known-incomplete.
**NodeSnapshot now carries scheduling state and a true exec total.** queued
and wake_pending make the 9c5ce5f invariant observable at runtime; it could
previously only be inspected in a debugger, and the bug does not reproduce
under one. total_exec_us is a real sum — frames × ema_exec_us tracks the
tail of a run, not the whole of it, and diverges badly on a workload whose
per-frame cost varies. Both are exposed over the web debug JSON so a wedged
pipeline can be interrogated without attaching to it.
28e0667 stopped nodes blocking a worker on a full output, but replaced an
intermittent hang with a quieter one: the pipeline still wedged about 2
runs in 30, now with every worker idle in pthread_cond_wait rather than
asleep in a push. Nothing was blocked; nothing had been woken.
try_submit discarded any wake arriving while queued_ was up:
if (queued_.compare_exchange_strong(expected, true, ...))
scheduler_->submit(...);
// else: silently gone
Wakes are edge-triggered — a channel fires its space callback once, on the
transition — so a dropped one never returns. A node could park a value,
release its worker, and sleep forever holding exactly the output its
consumer was waiting for, while its producer parked on an input channel
that would never drain.
try_submit now records the drop in wake_pending_, and release_and_recheck()
consumes it at every site that releases a node, giving one invariant: a
node never sleeps with a wake outstanding. This subsumes the two ad-hoc
re-checks added for the parked-retry and normal push paths, which only
moved the stall (1211 items to 6472) because each new early return was a
fresh chance to drop a wake. self_stop keeps a plain store — honouring a
pending wake there would resubmit a dead node.
Adds "a saturated chain never stalls", which reproduces this in about a
second where the pipeline needed ~30 runs. It asserts *progress does not
freeze* rather than a completion total: capacity-1 channels are slow, and
slow must never be reported as wedged. Verified in both directions — it
stalls after 1211 items on 28e0667 and passes here.
The existing chain test could not catch it: 40 items drain before any
strand occurs, and its producer emits forever, so fresh input keeps
re-triggering on_input_ready() and flushing the stranded value.
Tests: 122/122.
push_blocking parked a scheduler worker inside the push. Nodes own a
private single-thread pool, so the parked thread was the only one that
could drain that node's own input — hold-and-wait, and under sustained
backpressure four nodes of a five-node chain slept in nanosleep at once.
channel.hpp already warned about this for sentinels; it applies just as
much to data pushes.
The scheduler was purely input-driven: on_input_ready() wakes a node when
input arrives, with no counterpart for "my output has room". Lacking that
signal, blocking the thread was the only way to handle a full output.
This adds the missing half.
- Channel::try_push + has_space + set_space_callback; the callback fires
from both pop() and try_pop_now().
- PoolNode/PoolObjectNode keep a one-slot pending_ buffer with per-element
done flags, so a retry cannot duplicate an already-accepted element. One
slot suffices because queued_ admits at most one fire_once per node.
- The re-check after clearing queued_ closes the lost-wakeup race where a
space callback fires while the flag is still up and is swallowed.
Two bugs surfaced once nodes actually parked, both fixed here:
- pop_one reports an *empty* channel as ChannelClosedError, which is also
the node's "upstream finished, self-stop" signal. A node woken by output
space with empty inputs therefore killed itself. fire_once now releases
the worker when its inputs are not ready rather than falling through.
- The drained-park path resubmitted unconditionally instead of via
on_input_ready(), firing nodes with nothing to read.
compute_priority is now output-aware: mean output fill is deducted from
mean input fill, mapped as 0.5·(1 + in - out). Input fill alone asks only
"how much work is waiting for me"; a node whose outputs are already full
cannot deliver, so running it just parks it again and wastes the slot
while the node that would drain that channel waits behind it. The
scheduler now favours whoever is furthest downstream of a bottleneck.
Also adds a network-level error listener. A node's exception was discarded
at the node boundary and survived only as a Closed event, which reports
that a node stopped but not why — that missing detail is what made the
above slow to diagnose. INode::set_network_error_callback plus
StaticNetwork::set_error_handler forward it to the application.
Tests: 121/121. test_backpressure_deadlock drives a five-node chain with
capacity-2 channels against a slow sink and fails on the old code. The
four test_pool_node overflow tests now assert parking rather than the
removed drop-and-report behaviour.
Known-incomplete: a rare hang remains, roughly 1 run in 20 against a 300s
timeout, down from every run failing. Committed because the fix is a large
strict improvement and the residual case needs its own reproduction.
Every node output used the throwing push(), so a consumer falling behind cost
values rather than time. push_blocking() already existed on Channel and
OutputPort — "wait for the consumer to drain instead of dropping; the producer
just runs slower" — but nothing called it.
A dropped frame does not degrade a downstream result, it silently changes one,
and the consumer has no way to tell it happened. For any pipeline whose output
is a claim about its input, that is corruption rather than degradation.
Safe because sentinels are already handled out-of-band, above this path: only
data blocks, so the EOF token that unwinds the network can always overtake a
stalled data path. That is exactly the hold-and-wait deadlock the push_sentinel
comment warns about, and the reason it is not reachable here.
Measured on a downstream consumer (face pipeline, 77s clip at 5 fps, expected
385 sampled frames):
before 65 frames written, 320 dropped at one node, 29s
after 385 frames written, 0 dropped, 17s
Faster, not slower — a dropped frame has already cost its decode, and the
overflow exception cost more. Two consecutive runs now produce byte-identical
output, which they did not before: what got dropped depended on timing, so the
same command could yield different results.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds three pieces needed to build one KPN network and reuse it across many
replays/configs instead of tearing down and rebuilding per run:
- Channel<T>::push_blocking (+ IVariantChannel/VariantChannel forwarding):
lossless backpressure push that waits for space instead of dropping when
the ring is full. PyNode's run_loop now uses it so a downstream consumer
lagging behind never silently drops a frame.
- PyNetwork::node_ptr / node_stats: raw node handle by name (for a binding
to dynamic_cast to a concrete wrapper and call functor-specific runtime
setters) and a per-node timing snapshot for profiling.
- ObjectVariantNodeWrapper: variant-node adapter for functors that need
runtime-constructed state (a Config, a loaded gallery), mirroring
VariantNodeWrapper's channel plumbing but backed by ObjectNode<Obj>.
Built and used downstream in scene-actor-extraction's sae_kpn Python replay
bindings for repeated threshold-sweep evaluation of the same pipeline.
The ThreadSanitizer job runs on Docker nested in an unprivileged LXC
container, whose kernel randomizes mmap addresses beyond the range TSan's
fixed shadow mapping expects. TSan aborted at init with "unexpected memory
mapping" before any test ran.
Disable ASLR per-process with `setarch -R`, which needs the personality(2)
syscall that Docker's default seccomp profile blocks; seccomp=unconfined on
the container permits it. Verified on the runner that both are required:
setarch -R alone gets EPERM, seccomp alone still aborts, both together run
clean. Scoped to the tsan job, which runs only our own test binaries.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Channel<T>::pop() surfaced the out-of-band sentinel from its empty branch
using the tail_ snapshot taken at the top of the loop. Under contention the
producer can push more values *and* the sentinel in the window between that
snapshot and take_sentinel(), so pop() could return the sentinel while real
values still sat in the ring — the sentinel jumping ahead of values pushed
before it. No value was lost (a consumer that keeps draining still receives
them, and approx_size() keeps counting them so a PoolNode reschedules), but a
consumer treating the sentinel as a hard "last message" barrier would act on
EOF early.
Re-confirm emptiness against a fresh tail_ load before taking the sentinel.
Costs one acquire-load on the empty-ring path only; never runs in steady
state. The spin and post-spin takes already reload tail_ on the line above
them; try_pop_now() already reads tail_ fresh in the same branch — both were
correct and are unchanged.
The two sentinel stress cases now assert the strict "sentinel is last, after
every value" ordering (previously relaxed to avoid the flake this fixes).
Verified TSan-clean (2606 assertions, no data races) over repeated runs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The spec had drifted far from the code. Key corrections:
- Execution model is reactive (PoolNode submits fire_once() to a
ThreadPool when inputs are ready), not one blocking thread per node
- Channel<T> is a lock-free SPSC ring buffer (atomic wait/notify +
spin-before-sleep), not a mutex+CV queue
- Remove latch<> ports (never implemented)
- NodeErrorHandler returns bool (skip vs stop); per-node
- Document new subsystems: scheduler, InterruptNode, Router/FilterNode,
MainThreadNode, SharedResource, DebugHub, diagnostics/stats layer
- Update StaticNetwork, Python auto_bind layer, examples 01-16
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a contended SPSC stress suite (tests/test_channel_stress.cpp) that
actually exercises the ring's memory-ordering pairing and spin/futex/
lost-wakeup logic, plus the CMake and CI plumbing to run it under TSan:
- KPN_SANITIZER cache var + kpn_sanitizer_flags() helper (no-op when unset)
- kpn_tests_stress executable, labelled "stress" for CTest
- reusable tsan.yaml workflow (gcc:14 builder image, already ships libtsan)
- ci.yaml gains a tsan job on the same code/dockerfile triggers as test
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Gitea's act_runner mangles boolean workflow_call/dispatch inputs passed
from an expression -- they arrive as false regardless of value. Declare
`push` as a string ("true"/"false") and compare with == 'true' so the
builder image is pushed on non-PR events again.
07_python_network and 08_python_subport now work and run as CI smoke
tests, so drop their "(pending)" markers and describe what they actually
demonstrate. Also surface the hosted documentation link at the top and
re-render README.md from README.md.in.
Neither Python example was registered as a test, so `ctest -L examples`
in CI skipped them entirely -- which is how 08's missing Python node
went unnoticed.
Add a kpn_python_example() helper (gated on KPN_BUILD_PYTHON) that runs
each script with PYTHONPATH pointed at the freshly-built module, so it
does not depend on cwd or a hard-coded build/python path, and register
07 and 08. Also del the network in 07 for deterministic teardown.
The example was named "python subport" but its graph was entirely C++
(ProduceNode -> DoubleItNode); Python only tapped the output, leaving a
dangling "#todo: return value to network".
Rewrite so the only node in the graph is a pure-Python py_triple, driven
from both ends via the subport taps: net.write() injects inputs and
net.read() pulls results back, closing the round trip. Also del the
network at the end so its callable cycle is reclaimed deterministically.
The Python Network holds each PyNode's callable, forming an
uncollectable instance -> callable -> globals() -> instance cycle that
tripped nanobind's leak check at interpreter shutdown.
Implement tp_traverse/tp_clear type slots on the Network binding so
Python's cyclic collector can see through the C++-held callables and
break the cycle. PyNode exposes its callable; PyNetwork visits and
clears them. Wired into both binding sites (auto_bind and the legacy
register_py_network).
Channel::push() drops values on overflow (the intended backpressure
policy for data), and PoolNode swallows the resulting ChannelOverflowError.
For a control sentinel like EOF this is fatal: a single dropped EOF under
backpressure wedges every downstream pop() forever, so the pipeline never
tears down.
Deliver sentinels out-of-band instead. Channel::push_sentinel() stores the
token in a dedicated slot that does not consume ring capacity, so it can
never overflow and — crucially — never blocks the caller. That non-blocking
property is essential: each KPN node has a single worker thread, so a
*blocking* push would park that thread and stop it draining its own input,
cascading into a hold-and-wait deadlock under backpressure. The consumer's
pop()/try_pop_now() drain the ring first, then deliver the sentinel, so it
always arrives after every value pushed before it.
approx_size() (which node readiness checks call) counts a pending sentinel
as consumable work, so a channel carrying only a sentinel still schedules
its consumer's next fire — without this the token would sit undelivered and
the pipeline would still deadlock at teardown.
PoolNode/PoolObjectNode route values carrying an eof flag (direct .eof or
nested .source.eof) through push_sentinel via a SFINAE-safe is_sentinel_value
trait; all other values keep the existing lossy throwing push. The trait
compiles to false for types without an eof convention, so this is a no-op
for pipelines that don't use one.
Verified end-to-end: scene_analyze now reaches EOF, flushes its output, and
exits cleanly instead of hanging.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Trivial comment changes to exercise the new CI orchestration: the
docker job should rebuild+push the builder image first, then test
and docs run against the fresh image.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
dtourolle
