SharedResource::acquire() blocks on a condition variable whose predicate only
becomes true when release() hands over ownership. No timeout, no stop
condition. A node parked there was not observing stop flags, so teardown had
no way to reach it: the worker never returned, the pool's join never
completed, and shutdown waited on a resource nobody was going to release —
which is precisely the situation when the holder is being stopped too.
close() wakes every waiter and refuses further acquisitions, and the waiters
leave through ResourceClosedError, which is an exception the node error path
already handles rather than a new mechanism. StaticNetwork calls it on
registered resources at the top of halt() and shutdown(), before stopping any
node, since a node stopped while parked cannot respond to being stopped.
The handover needed care in two places. A waiter woken by close() has not been
given ownership, so it takes no Guard and leaves held_ exactly as it found it;
and release() now skips handing over to waiters when closed, because handing
ownership to a thread that is on its way out would leave held_ true with
nobody holding it.
reopen() is there for reuse across runs, which the persistent-pipeline work
will want; teardown does not need it.
Verified in both directions: without close() the waiter thread never returns
and the test's join blocks; with it the waiter leaves through
ResourceClosedError while the holder still has the resource. 145/145.
a8cfe73 appended queued/wake_pending/total_exec_ms to the NodeSnapshot
aggregate in an order no call site used. Every node type fills the aggregate
positionally and all of them supply total_exec_ms as the element straight
after queue_wait_ms — but the struct declared the two bools there. So the
exec total landed in `queued`, `queued` landed in `wake_pending`, and
`wake_pending` landed in total_exec_ms.
GCC reported it as -Wnarrowing (bool to double), 88 times, once per node
instantiation across the test build. The build carried on.
This is worse than a cosmetic mix-up, because all three fields were added
specifically to diagnose a wedge. A wedged pipeline reported total_exec_ms
as 0.0 or 1.0, and `queued` as "has this node ever run" — true for every
node that had, including ones asleep with nothing to do. The web debug JSON
served the same values. Anyone reading them to find the stalled node would
have been pointed at the wrong one.
Moves total_exec_ms above the two bools to match every call site, and notes
in the struct why the order is load-bearing.
Verified in both directions: on a8cfe73 the new case reports frames=8
total=0 queued=true; here total >= ema and both flags are false.
Three changes from one debugging session on the intermittent wedge, kept
together because the instrumentation is what made the other two findable.
**Fanout was never made lossless.** 6595e6e made node outputs lossless and
28e0667 stopped them parking a worker; FanoutNode was in neither and kept
`catch (ChannelOverflowError&) {}` per output. Whichever branch fell behind
lost items, silently, by an amount that depended on timing — so two runs of
the same input could disagree. deliver() now retries each output
independently until it is taken, rechecking stop_flag_ every pass so
teardown cannot hang on a full output. A fanout owns a private thread, so
waiting costs no scheduler worker.
**A node could start with a wake already outstanding.** start() enables the
input channel several statements before it installs the push callback, and
StaticNetwork starts nodes sources-first, so an upstream node is already
firing into the gap. A push landing there is accepted by the ring but wakes
nobody: push_callback_ fires only on the empty→non-empty transition, and at
that instant the callback is null. Every later push sees a non-empty ring
and stays silent, so the node is never submitted. Asking on_input_ready()
once at the end of start() converts the missed edge into a state check.
The signature is distinctive — zero items delivered, not a stall partway.
Under `ctest -j4` on a loaded machine it reproduced 7 times in 24 and never
in 10 unloaded runs, which is almost certainly the "~1 run in 20" hang
28e0667 recorded as known-incomplete.
**NodeSnapshot now carries scheduling state and a true exec total.** queued
and wake_pending make the 9c5ce5f invariant observable at runtime; it could
previously only be inspected in a debugger, and the bug does not reproduce
under one. total_exec_us is a real sum — frames × ema_exec_us tracks the
tail of a run, not the whole of it, and diverges badly on a workload whose
per-frame cost varies. Both are exposed over the web debug JSON so a wedged
pipeline can be interrogated without attaching to it.
dtourolle