fix: a lossless fanout, a node that starts awake, and the instrumentation that found them
Three changes from one debugging session on the intermittent wedge, kept together because the instrumentation is what made the other two findable. **Fanout was never made lossless.**6595e6emade node outputs lossless and28e0667stopped them parking a worker; FanoutNode was in neither and kept `catch (ChannelOverflowError&) {}` per output. Whichever branch fell behind lost items, silently, by an amount that depended on timing — so two runs of the same input could disagree. deliver() now retries each output independently until it is taken, rechecking stop_flag_ every pass so teardown cannot hang on a full output. A fanout owns a private thread, so waiting costs no scheduler worker. **A node could start with a wake already outstanding.** start() enables the input channel several statements before it installs the push callback, and StaticNetwork starts nodes sources-first, so an upstream node is already firing into the gap. A push landing there is accepted by the ring but wakes nobody: push_callback_ fires only on the empty→non-empty transition, and at that instant the callback is null. Every later push sees a non-empty ring and stays silent, so the node is never submitted. Asking on_input_ready() once at the end of start() converts the missed edge into a state check. The signature is distinctive — zero items delivered, not a stall partway. Under `ctest -j4` on a loaded machine it reproduced 7 times in 24 and never in 10 unloaded runs, which is almost certainly the "~1 run in 20" hang28e0667recorded as known-incomplete. **NodeSnapshot now carries scheduling state and a true exec total.** queued and wake_pending make the9c5ce5finvariant observable at runtime; it could previously only be inspected in a debugger, and the bug does not reproduce under one. total_exec_us is a real sum — frames × ema_exec_us tracks the tail of a run, not the whole of it, and diverges badly on a workload whose per-frame cost varies. Both are exposed over the web debug JSON so a wedged pipeline can be interrogated without attaching to it.
This commit is contained in:
@@ -65,6 +65,11 @@ static std::string to_json(const std::vector<NodeSnapshot>& nodes,
|
||||
<< ",\"fps\":" << n.throughput_fps
|
||||
<< ",\"total_cpu_ms\":" << n.total_cpu_ms
|
||||
<< ",\"cpu_util_pct\":" << n.cpu_util_pct
|
||||
// Scheduling state — lets a WEDGED pipeline be interrogated over HTTP
|
||||
// without a debugger, which matters because the lost-wake bug does not
|
||||
// reproduce under one. See NodeSnapshot for how to read the pair.
|
||||
<< ",\"queued\":" << (n.queued ? "true" : "false")
|
||||
<< ",\"wake_pending\":" << (n.wake_pending ? "true" : "false")
|
||||
<< "}";
|
||||
}
|
||||
o << "],\"edges\":[";
|
||||
|
||||
Reference in New Issue
Block a user