Every node output used the throwing push(), so a consumer falling behind cost
values rather than time. push_blocking() already existed on Channel and
OutputPort — "wait for the consumer to drain instead of dropping; the producer
just runs slower" — but nothing called it.
A dropped frame does not degrade a downstream result, it silently changes one,
and the consumer has no way to tell it happened. For any pipeline whose output
is a claim about its input, that is corruption rather than degradation.
Safe because sentinels are already handled out-of-band, above this path: only
data blocks, so the EOF token that unwinds the network can always overtake a
stalled data path. That is exactly the hold-and-wait deadlock the push_sentinel
comment warns about, and the reason it is not reachable here.
Measured on a downstream consumer (face pipeline, 77s clip at 5 fps, expected
385 sampled frames):
before 65 frames written, 320 dropped at one node, 29s
after 385 frames written, 0 dropped, 17s
Faster, not slower — a dropped frame has already cost its decode, and the
overflow exception cost more. Two consecutive runs now produce byte-identical
output, which they did not before: what got dropped depended on timing, so the
same command could yield different results.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Channel::push() drops values on overflow (the intended backpressure
policy for data), and PoolNode swallows the resulting ChannelOverflowError.
For a control sentinel like EOF this is fatal: a single dropped EOF under
backpressure wedges every downstream pop() forever, so the pipeline never
tears down.
Deliver sentinels out-of-band instead. Channel::push_sentinel() stores the
token in a dedicated slot that does not consume ring capacity, so it can
never overflow and — crucially — never blocks the caller. That non-blocking
property is essential: each KPN node has a single worker thread, so a
*blocking* push would park that thread and stop it draining its own input,
cascading into a hold-and-wait deadlock under backpressure. The consumer's
pop()/try_pop_now() drain the ring first, then deliver the sentinel, so it
always arrives after every value pushed before it.
approx_size() (which node readiness checks call) counts a pending sentinel
as consumable work, so a channel carrying only a sentinel still schedules
its consumer's next fire — without this the token would sit undelivered and
the pipeline would still deadlock at teardown.
PoolNode/PoolObjectNode route values carrying an eof flag (direct .eof or
nested .source.eof) through push_sentinel via a SFINAE-safe is_sentinel_value
trait; all other values keep the existing lossy throwing push. The trait
compiles to false for types without an eof convention, so this is a no-op
for pipelines that don't use one.
Verified end-to-end: scene_analyze now reaches EOF, flushes its output, and
exits cleanly instead of hanging.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
dtourolleandClaude Opus 5