Commit Graph
6 Commits
Author SHA1 Message Date
dtourolle 091211cb19 fix: NodeSnapshot fields must line up with what nodes supply
a8cfe73 appended queued/wake_pending/total_exec_ms to the NodeSnapshot
aggregate in an order no call site used. Every node type fills the aggregate
positionally and all of them supply total_exec_ms as the element straight
after queue_wait_ms — but the struct declared the two bools there. So the
exec total landed in `queued`, `queued` landed in `wake_pending`, and
`wake_pending` landed in total_exec_ms.

GCC reported it as -Wnarrowing (bool to double), 88 times, once per node
instantiation across the test build. The build carried on.

This is worse than a cosmetic mix-up, because all three fields were added
specifically to diagnose a wedge. A wedged pipeline reported total_exec_ms
as 0.0 or 1.0, and `queued` as "has this node ever run" — true for every
node that had, including ones asleep with nothing to do. The web debug JSON
served the same values. Anyone reading them to find the stalled node would
have been pointed at the wrong one.

Moves total_exec_ms above the two bools to match every call site, and notes
in the struct why the order is load-bearing.

Verified in both directions: on a8cfe73 the new case reports frames=8
total=0 queued=true; here total >= ema and both flags are false.
2026-08-05 12:46:56 +02:00
dtourolle a8cfe7300a fix: a lossless fanout, a node that starts awake, and the instrumentation that found them
Three changes from one debugging session on the intermittent wedge, kept
together because the instrumentation is what made the other two findable.

**Fanout was never made lossless.** 6595e6e made node outputs lossless and
28e0667 stopped them parking a worker; FanoutNode was in neither and kept
`catch (ChannelOverflowError&) {}` per output. Whichever branch fell behind
lost items, silently, by an amount that depended on timing — so two runs of
the same input could disagree. deliver() now retries each output
independently until it is taken, rechecking stop_flag_ every pass so
teardown cannot hang on a full output. A fanout owns a private thread, so
waiting costs no scheduler worker.

**A node could start with a wake already outstanding.** start() enables the
input channel several statements before it installs the push callback, and
StaticNetwork starts nodes sources-first, so an upstream node is already
firing into the gap. A push landing there is accepted by the ring but wakes
nobody: push_callback_ fires only on the empty→non-empty transition, and at
that instant the callback is null. Every later push sees a non-empty ring
and stays silent, so the node is never submitted. Asking on_input_ready()
once at the end of start() converts the missed edge into a state check.

The signature is distinctive — zero items delivered, not a stall partway.
Under `ctest -j4` on a loaded machine it reproduced 7 times in 24 and never
in 10 unloaded runs, which is almost certainly the "~1 run in 20" hang
28e0667 recorded as known-incomplete.

**NodeSnapshot now carries scheduling state and a true exec total.** queued
and wake_pending make the 9c5ce5f invariant observable at runtime; it could
previously only be inspected in a debugger, and the bug does not reproduce
under one. total_exec_us is a real sum — frames × ema_exec_us tracks the
tail of a run, not the whole of it, and diverges badly on a workload whose
per-frame cost varies. Both are exposed over the web debug JSON so a wedged
pipeline can be interrogated without attaching to it.
2026-08-05 12:40:03 +02:00
dtourolle f6bcaa15b0 Performance improvements, better readme and complete python bindings
🧪 Test / test (push) Failing after 28m30s
2026-05-12 21:23:33 +02:00
dtourolle 1e9ba5ee66 Add a unified observability interface for applications with multiple networks
🧪 Test / test (push) Successful in 6m9s
2026-05-10 19:08:40 +02:00
dtourolle 278c122e8f Add shared reasource tag to allow coordination of usage
🧪 Test / test (push) Successful in 6m8s
2026-05-09 15:22:27 +02:00
dtourolle 5e77dc836b First attempt 2026-05-08 17:48:16 +02:00