fix: a lossless fanout, a node that starts awake, and the instrumentation that found them
Three changes from one debugging session on the intermittent wedge, kept together because the instrumentation is what made the other two findable. **Fanout was never made lossless.**6595e6emade node outputs lossless and28e0667stopped them parking a worker; FanoutNode was in neither and kept `catch (ChannelOverflowError&) {}` per output. Whichever branch fell behind lost items, silently, by an amount that depended on timing — so two runs of the same input could disagree. deliver() now retries each output independently until it is taken, rechecking stop_flag_ every pass so teardown cannot hang on a full output. A fanout owns a private thread, so waiting costs no scheduler worker. **A node could start with a wake already outstanding.** start() enables the input channel several statements before it installs the push callback, and StaticNetwork starts nodes sources-first, so an upstream node is already firing into the gap. A push landing there is accepted by the ring but wakes nobody: push_callback_ fires only on the empty→non-empty transition, and at that instant the callback is null. Every later push sees a non-empty ring and stays silent, so the node is never submitted. Asking on_input_ready() once at the end of start() converts the missed edge into a state check. The signature is distinctive — zero items delivered, not a stall partway. Under `ctest -j4` on a loaded machine it reproduced 7 times in 24 and never in 10 unloaded runs, which is almost certainly the "~1 run in 20" hang28e0667recorded as known-incomplete. **NodeSnapshot now carries scheduling state and a true exec total.** queued and wake_pending make the9c5ce5finvariant observable at runtime; it could previously only be inspected in a debugger, and the bug does not reproduce under one. total_exec_us is a real sum — frames × ema_exec_us tracks the tail of a run, not the whole of it, and diverges badly on a workload whose per-frame cost varies. Both are exposed over the web debug JSON so a wedged pipeline can be interrogated without attaching to it.
This commit is contained in:
@@ -51,6 +51,14 @@ struct NodeStats {
|
||||
std::atomic<int64_t> max_exec_us{0};
|
||||
std::atomic<int64_t> total_blocked_us{0};
|
||||
|
||||
// Cumulative wall time inside fire_once, summed over every invocation.
|
||||
// The EMA above cannot be turned into a total: it is exponentially
|
||||
// weighted, so frames * ema_exec_us tracks the tail of the run rather than
|
||||
// the whole of it, and on a workload whose per-frame cost varies (a face
|
||||
// detector on a film: crowd scenes then empty landscapes) the two differ by
|
||||
// a lot. Answering "how much time went into this node" needs a real sum.
|
||||
std::atomic<int64_t> total_exec_us{0};
|
||||
|
||||
// Thread CPU time — actual CPU consumed by this node's thread,
|
||||
// measured via CLOCK_THREAD_CPUTIME_ID. Excludes time sleeping or
|
||||
// blocked on mutexes/channels. Sampled once per frame.
|
||||
@@ -89,6 +97,7 @@ struct NodeStats {
|
||||
frames_processed.fetch_add(1, std::memory_order_relaxed);
|
||||
|
||||
int64_t us = static_cast<int64_t>(exec_time.count() * 1000.0);
|
||||
total_exec_us.fetch_add(us, std::memory_order_relaxed);
|
||||
|
||||
uint64_t n = frames_processed.load(std::memory_order_relaxed);
|
||||
int64_t prev = ema_exec_us.load(std::memory_order_relaxed);
|
||||
@@ -147,6 +156,24 @@ struct NodeSnapshot {
|
||||
double total_cpu_ms; // cumulative CPU time consumed by this node's thread
|
||||
double cpu_util_pct; // exec_ms / (exec_ms + blocked_ms) * 100
|
||||
double queue_wait_ms{0}; // PoolNode: cumulative time spent in pool queue
|
||||
|
||||
// Live scheduling state, for observing the AR-004 invariant "a node never
|
||||
// sleeps with a wake outstanding". The invariant was previously asserted in
|
||||
// comments but invisible at runtime, so a lost wake could only be found in a
|
||||
// debugger — and this bug does not reproduce under one (it needs full speed).
|
||||
// Two atomic loads at snapshot time, nothing on the hot path.
|
||||
//
|
||||
// Read them together with the node's channel fill:
|
||||
// queued=0, wake=1 -> wake recorded and never consumed
|
||||
// queued=0, wake=0, input full -> wake never generated at all
|
||||
// queued=1 while nothing running -> submitted but never scheduled
|
||||
bool queued{false};
|
||||
bool wake_pending{false};
|
||||
// Cumulative wall time inside fire_once. Unlike ema_exec_ms this is a true
|
||||
// sum, so it is the field to use for "share of the run spent in this node".
|
||||
// Note it still includes time parked pushing into a full output channel;
|
||||
// total_cpu_ms is the part that backpressure cannot inflate.
|
||||
double total_exec_ms{0};
|
||||
};
|
||||
|
||||
// ── Pool statistics + snapshot ────────────────────────────────────────────────
|
||||
|
||||
Reference in New Issue
Block a user