dtourolle/KPN
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: a lossless fanout, a node that starts awake, and the instrumentation that found them

Browse Source 
Three changes from one debugging session on the intermittent wedge, kept
together because the instrumentation is what made the other two findable.

**Fanout was never made lossless.** 6595e6e made node outputs lossless and
28e0667 stopped them parking a worker; FanoutNode was in neither and kept
`catch (ChannelOverflowError&) {}` per output. Whichever branch fell behind
lost items, silently, by an amount that depended on timing — so two runs of
the same input could disagree. deliver() now retries each output
independently until it is taken, rechecking stop_flag_ every pass so
teardown cannot hang on a full output. A fanout owns a private thread, so
waiting costs no scheduler worker.

**A node could start with a wake already outstanding.** start() enables the
input channel several statements before it installs the push callback, and
StaticNetwork starts nodes sources-first, so an upstream node is already
firing into the gap. A push landing there is accepted by the ring but wakes
nobody: push_callback_ fires only on the empty→non-empty transition, and at
that instant the callback is null. Every later push sees a non-empty ring
and stays silent, so the node is never submitted. Asking on_input_ready()
once at the end of start() converts the missed edge into a state check.

The signature is distinctive — zero items delivered, not a stall partway.
Under `ctest -j4` on a loaded machine it reproduced 7 times in 24 and never
in 10 unloaded runs, which is almost certainly the "~1 run in 20" hang
28e0667 recorded as known-incomplete.

**NodeSnapshot now carries scheduling state and a true exec total.** queued
and wake_pending make the 9c5ce5f invariant observable at runtime; it could
previously only be inspected in a debugger, and the bug does not reproduce
under one. total_exec_us is a real sum — frames × ema_exec_us tracks the
tail of a run, not the whole of it, and diverges badly on a workload whose
per-frame cost varies. Both are exposed over the web debug JSON so a wedged
pipeline can be interrogated without attaching to it.
This commit is contained in:
dtourolle
2026-08-05 12:40:03 +02:00
parent 454f72c167
commit a8cfe7300a
7 changed files with 297 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files
include/kpn/fanout.hpp
+83 -8
View File
@@ -7,8 +7,10 @@
#include <array>
#include <atomic>
#include <chrono>
#include <iostream>
#include <memory>
#include <optional>
#include <thread>
#include <tuple>
#include <utility>
@@ -78,7 +80,9 @@ public:
blocked_ms,
elapsed_s > 0 ? frames / elapsed_s : 0.0,
stats_.total_cpu_us.load(std::memory_order_relaxed) / 1000.0,
total_ms > 0 ? 100.0 * exec_ms / total_ms : 0.0};
total_ms > 0 ? 100.0 * exec_ms / total_ms : 0.0,
0.0, // queue_wait_ms — fanout is not pool-scheduled
stats_.total_exec_us.load(std::memory_order_relaxed) / 1000.0};
}
// ── Port access ───────────────────────────────────────────────────────────
@@ -116,6 +120,76 @@ public:
}
private:
// Deliver `val` to every connected output, losslessly.
//
// Previously a full output cost the value: push() threw and the exception was
// swallowed per output. A dropped item does not degrade a downstream result,
// it silently changes one, and the consumer cannot tell it happened — so the
// fanout waits instead, and the producer upstream runs slower.
//
// Unlike a pool node, a fanout owns a private thread, so waiting here costs
// no scheduler worker and needs no space-callback park; a bounded retry is
// enough. `stop_flag_` is re-checked every pass so teardown cannot hang on a
// full output regardless of the order the network stops its nodes in.
//
// Outputs are retried independently, so a full output never delays delivery
// to one with room. Note what that does *not* buy: the next input is not
// popped until every output has accepted the current item, so one branch can
// never run ahead of another by more than the slower branch's buffering.
//
// **That bound is a precondition on any topology where the branches rejoin.**
// If a consumer on branch B blocks waiting for something branch A computes,
// B's buffering must exceed the lead A needs, or the two wedge — B waiting on
// A, A starved because the fanout is holding an item B will not take. Making
// the fanout lossless is what puts that precondition on the topology; while
// it dropped, the question could not arise.
//
// `parked` receives the time spent waiting on a full output, which the caller
// charges to blocked rather than exec.
//
// Returns false if stopped with the value undelivered.
bool deliver(const T& val, duration_t& parked) {
std::array<std::optional<T>, N> pending;
std::size_t outstanding = 0;
for (std::size_t i = 0; i < N; ++i)
if (out_channels_[i]) { pending[i].emplace(val); ++outstanding; }
bool first_pass = true;
auto park_from = clock_t::now();
for (;;) {
for (std::size_t i = 0; i < N; ++i) {
if (!pending[i]) continue;
if (out_channels_[i]->try_push(*pending[i])) {
pending[i].reset();
--outstanding;
}
}
if (first_pass) { park_from = clock_t::now(); first_pass = false; }
if (outstanding == 0) {
parked = duration_t(clock_t::now() - park_from);
return true;
}
if (stop_flag_.load(std::memory_order_relaxed)) {
// Teardown with work in hand. One last throwing push per
// outstanding output, purely so the channel's own stats record
// the loss (drop if it is disabled, overflow if it is merely
// full). The whole point of the lossless path is that a loss is
// never invisible, and a silent `return` here would reintroduce
// exactly the hole this function exists to close.
for (std::size_t i = 0; i < N; ++i) {
if (!pending[i]) continue;
try { out_channels_[i]->push(std::move(*pending[i])); }
catch (const ChannelOverflowError&) {}
}
parked = duration_t(clock_t::now() - park_from);
return false;
}
std::this_thread::sleep_for(std::chrono::microseconds(50));
}
}
void run_loop() {
while (!stop_flag_.load(std::memory_order_relaxed)) {
try {
@@ -124,16 +198,17 @@ private:
auto t1 = clock_t::now();
auto cpu0 = NodeStats::cpu_now();
for (std::size_t i = 0; i < N; ++i) {
if (out_channels_[i]) {
try { out_channels_[i]->push(val); }
catch (const ChannelOverflowError&) {} // drop for this output independently
}
}
duration_t parked{0};
const bool delivered = deliver(val, parked);
auto cpu1 = NodeStats::cpu_now();
auto t2 = clock_t::now();
stats_.record_exec(duration_t(t2 - t1), duration_t(t1 - t0), cpu0, cpu1);
// Time spent waiting on a full output is *blocked*, not exec: a
// parked fanout is idle, and charging it to exec would report the
// node as busy exactly when it is the one being held up.
stats_.record_exec(duration_t(t2 - t1) - parked,
duration_t(t1 - t0) + parked, cpu0, cpu1);
if (!delivered) break;
} catch (const ChannelClosedError&) {
break;
}