dtourolle/KPN
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

merge into: dtourolle/KPN:gitea-pages
Branches Tags
dtourolle/KPN:master dtourolle/KPN:perf/phase0-harness dtourolle/KPN:fix/kpn-wedging-audit dtourolle/KPN:backpressure-fix dtourolle/KPN:feature/lossless-backpressure dtourolle/KPN:feature/persistent-pipeline-reuse dtourolle/KPN:spec-and-tsan dtourolle/KPN:gitea-pages dtourolle/KPN:better-examples
..
pull from: dtourolle/KPN:fix/kpn-wedging-audit
Branches Tags
dtourolle/KPN:master dtourolle/KPN:perf/phase0-harness dtourolle/KPN:fix/kpn-wedging-audit dtourolle/KPN:backpressure-fix dtourolle/KPN:feature/lossless-backpressure dtourolle/KPN:feature/persistent-pipeline-reuse dtourolle/KPN:spec-and-tsan dtourolle/KPN:gitea-pages dtourolle/KPN:better-examples
59 Commits
gitea-pages .. fix/kpn-wedging-audit
Author SHA1 Message Date
dtourolle 00245f5760 fix: Network::set_error_handler must actually deliver the handler
🚦 CI / changes (pull_request) Successful in 13s
Details
🚦 CI / docker (pull_request) Has been skipped
Details
🚦 CI / test (pull_request) Successful in 8m28s
Details
🚦 CI / tsan (pull_request) Successful in 3m30s
Details
🚦 CI / docs (pull_request) Has been skipped
Details 
The handler was stored in a member and never read. A node's exception was
discarded at the node boundary and the only surviving evidence was a Closed
event, which reports that a node stopped but not why — the difference between
a diagnosis and a guess. StaticNetwork has always wired this; Network accepted
the handler and silently dropped it, which is worse than not offering the
setter, because the caller believes they have a listener.

start() now delivers it to each node, exactly as StaticNetwork does.

The type changes with it. It was void(name, exception_ptr), which cannot
express the keep-running decision the node side needs — so it is now
NodeErrorHandler, the same alias StaticNetwork uses. That is a breaking change
in principle; in practice nothing in the tree called this setter, which is how
it stayed dead long enough to be worth finding.

Verified by the new case: the node throws, the handler receives the name and
the exception, returns true, and the node goes on to process the next value.
148/148.
 2026-08-05 18:05:24 +02:00
dtourolle 7a3e96cc99 fix: the watchdog must be interruptible, or stop() waits for its next tick 
start_watchdog looped on std::this_thread::sleep_for(watchdog_interval_), and
request_stop() cannot wake a sleeping thread. stop_watchdog()'s join therefore
blocked until the current sleep expired: three seconds on every teardown at
the default interval, and unbounded for anyone who set a long one to keep the
periodic report quiet.

Now a condition_variable_any waited on with the stop token, so request_stop()
ends the wait immediately.

Found while writing the next commit's test, which sets a one-hour interval to
silence the report and consequently hung for an hour in stop().

The test needs one non-obvious thing, and says so: a pause between start() and
stop(). Without it the test races the watchdog — stop_watchdog() runs before
the thread has entered its loop, the token is already set when it does, and it
exits without ever waiting. That passes against the bug as well as the fix,
which is exactly what the first version of this test did.

Verified in both directions: without the fix the case is killed at a 25 s
timeout; with it, stop() returns in 0 ms.
 2026-08-05 17:52:36 +02:00
dtourolle 80c2b1fb2f fix: try_push must distinguish delivered from discarded 
try_push returned bool, and returned *true* for a closed channel — so "the
value arrived" and "the value was thrown away because nobody is listening"
were the same answer.

Every caller was nonetheless correct, because both cases mean "stop trying,
do not park and retry". But nothing above the channel could tell the two
apart: a node counting successful pushes counted discards among them, and the
only record of the loss was the channel's own drop counter, visible solely to
whoever read the diagnostics table.

Now a three-way PushResult { Taken, Full, Closed }, matching the shape
SentinelResult already uses. Behaviour is unchanged at every call site —
each treats Closed the same as Taken, and only Full parks — but the
distinction is now available to anyone who needs it, and a scoped enum means
a future caller cannot silently reintroduce the conflation with `if (push)`.

deliver_one benefits immediately: it no longer reaches its teardown path for
a closed channel, only for one that is still full, so the last-ditch throwing
push it does there to record the loss now records an overflow rather than a
drop the channel had already counted.
 2026-08-05 16:14:50 +02:00
dtourolle 012b64dd3e fix: the sentinel must not be delivered ahead of a queued value 
pop() and try_pop_now() observe the ring empty and then call take_sentinel().
The producer can push a value *and* publish the sentinel in the window between
those two steps, so the sentinel was delivered with a real value still queued
behind it — breaking the "sentinel is strictly last" contract that downstream
teardown depends on, and losing that value to any consumer which, like the
stress cases here, treats the sentinel as EOF and stops draining.

a0c4bf5 closed the variant where the caller's emptiness check ran against a
stale tail_ snapshot. This is the one where the check is fresh and simply too
early.

take_sentinel now re-checks emptiness *after* observing has_eof_, which is
what makes it sound rather than merely narrower: the producer publishes the
sentinel with a release store after its ring pushes, so a consumer that has
observed has_eof_ has necessarily observed every tail_ advance before it. A
non-empty ring at that point means those values genuinely precede the
sentinel, and returning false hands them over first.

Rates, since this is a race and the numbers are the evidence. The existing
"sentinel is strictly last" stress cases fail about 1 run in 15 on the commit
before this one and 0 in 25 after; they did not fail in 25 runs of the
pre-series baseline, so something in this series widened the window rather
than opened it. I could not pin down which change, and it does not much
matter: the interleaving is reachable from the code as written, and the
narrower version was never correct.

No new test. The two existing stress cases already assert exactly this and
are what caught it; a deterministic reproduction would need a seam inside
pop() that the fix then makes unreachable.
 2026-08-05 16:11:20 +02:00
dtourolle 7b7f631e6d fix: a shared resource must be able to release its waiters 
SharedResource::acquire() blocks on a condition variable whose predicate only
becomes true when release() hands over ownership. No timeout, no stop
condition. A node parked there was not observing stop flags, so teardown had
no way to reach it: the worker never returned, the pool's join never
completed, and shutdown waited on a resource nobody was going to release —
which is precisely the situation when the holder is being stopped too.

close() wakes every waiter and refuses further acquisitions, and the waiters
leave through ResourceClosedError, which is an exception the node error path
already handles rather than a new mechanism. StaticNetwork calls it on
registered resources at the top of halt() and shutdown(), before stopping any
node, since a node stopped while parked cannot respond to being stopped.

The handover needed care in two places. A waiter woken by close() has not been
given ownership, so it takes no Guard and leaves held_ exactly as it found it;
and release() now skips handing over to waiters when closed, because handing
ownership to a thread that is on its way out would leave held_ true with
nobody holding it.

reopen() is there for reuse across runs, which the persistent-pipeline work
will want; teardown does not need it.

Verified in both directions: without close() the waiter thread never returns
and the test's join blocks; with it the waiter leaves through
ResourceClosedError while the holder still has the resource. 145/145.
 2026-08-05 15:51:26 +02:00
dtourolle 97670d8ba3 fix: stop() must not return while a firing is still running 
stop() set the flag, disabled the inputs and returned, leaving an executing
fire_once touching input_channels_, stats_ and pending_ while the caller went
on to destroy them. The comment was explicit about it: callers wanting the
guarantee should call scheduler_->drain() first. But ~PoolNode calls stop(),
and a destructor cannot ask its caller to have done that.

A node with a private pool survived by accident, because Node::stop() calls
pool->stop() and that joins the worker. A node sharing a pool — which
make_pool_node exists to create — had nothing joining it, so its own
destructor raced the firing.

stop() now waits on the submit gate, which is claimed for the whole of a
firing and released as its last act. A queued but unstarted firing also holds
it and will run, observe stop_flag_ and release, so the pool must still be
running when stop() is called; that is already the documented order and what
Node/ObjectNode do.

Two ways it declines to wait. It is bounded at five seconds, because a node
function that never returns must not convert teardown into a hang — it warns
and continues. And it returns immediately when called from the firing thread
itself, since an error handler that stops its own node would otherwise wait
for a firing that is waiting for it.

Verified in both directions: with the wait removed, stop() returns while the
node function is still sleeping and the flag it sets on the way out is still
false. 143/143.
 2026-08-05 15:46:27 +02:00
dtourolle b9698fae60 fix: idle workers must sleep while another worker is busy 
The wait predicate was `stopped_ || total_ > 0`, and total_ counts queued
*plus executing*. So while any one task ran, every other worker's predicate
was true: wait() returned immediately and the worker spun through try_pop,
try_steal and back to wait at full speed, try_lock-ing every peer queue on
each pass.

Measured on this tree, 8 workers and one 300 ms task: 1991 ms of CPU and
19205 voluntary context switches, against 0.4 ms and 10 with the fix. Call it
six and a half cores burned for the duration of one sleeping task.

Sleeping requires "no work is *waiting*", which total_ cannot express, so
queued_ is now tracked separately: incremented on submit, decremented when a
task leaves a queue, and adjusted for the tasks stop() discards. total_ stays
as it was for drain(), which genuinely does need to know about executing work.
The worker exit condition moves to queued_ for the same reason — waiting for
total_ to reach zero meant waiting for someone else's task to finish, which a
worker cannot help with and would spin through until it did. PoolSnapshot's
queue depth stops being an estimate as a side effect.

Latent for this pipeline, where each node owns a private single-thread pool
and there is no idle peer to spin. Any use of a shared pool, which
make_pool_node exists for, hits it immediately.

Reproducing it needs the right trigger, and the test says so, because my first
attempt got it wrong and passed against the bug: a worker that has never been
woken stays blocked in wait() and never re-evaluates the predicate. The spin
only appears once a worker *finishes* something and re-enters the loop while a
peer is still busy, so the case submits one long task plus a trivial one per
remaining worker. Submitting only the long task measures nothing.

Verified in both directions: 1969 ms of CPU before, 0.35 ms after, against a
300 ms threshold. Full suite 142/142.
 2026-08-05 15:34:33 +02:00
dtourolle 87c5f98d04 fix: start and stop in the topological order that was computed 
make_network computes Topo for the cycle check and then discarded it. The
node vector was filled in edge-declaration order — and then named
user_nodes_topo_ and relied upon as if it were sorted. halt() stops in its
reverse, and shutdown() walks it forwards stopping each node and draining its
outputs before the next, which is a graceful drain only if the order really
is sources-first.

It held for every network in this tree because edges happen to be declared in
pipeline order, so the two coincided. Declared any other way — which is legal
and which make_network otherwise accepts in silence — shutdown stops a
consumer before its producer and discards whatever was queued in front of it.

The order now comes from Topo::topo, which is already sources-first. Fanout
nodes appear there too and are skipped, since they are owned separately in
fanout_storage; they are still started after the user nodes and stopped
before them, so a fanout sitting between two user nodes is not staged
precisely during a drain. That is a smaller gap than the one being closed and
is left alone rather than restructured on the way past.

The test declares the sink edge first and the source edge last, and asserts
through shutdown() rather than by reading the order back — the order is
private, and what it buys is the point. A sources-first shutdown lets the
backlog queued in front of the slow relay reach the sink; stopping the relay
first discards all of it.

Worth recording how this went, because it is the more useful half: the new
test segfaulted, 12 runs in 20. Not a fault in the ordering change — it was
stopping sources first that finally put a live consumer behind a dead
producer, which is the condition the previous commit's crash needs. The
ordering fix did not introduce that bug, it made it reachable.

Verified in both directions: with declaration order the sink receives nothing
after shutdown begins; with topological order it receives the whole backlog.
 2026-08-05 15:14:35 +02:00
dtourolle abbb2d4770 fix: submitting to a stopped pool must be refused, not fatal 
ThreadPool::stop() ends with queues_.clear(), and submit() went straight to
queues_[target] with no check. A submission arriving after stop indexed an
empty vector and segfaulted.

This is not a contrived teardown ordering. A node's space callback fires from
whichever thread drained the channel, and that thread belongs to the
*consumer*; the callback it runs belongs to the *producer*. Stop the producer
first — which a sources-first shutdown does by design — and the consumer keeps
draining its backlog, firing the producer's space callback into a pool that
has already been torn down:

    ThreadPool::submit
      <- source node's space_callback
      <- Channel<int>::try_pop_now      (relay draining its input)
      <- relay fire_once

The static-network shutdown case in the next commit crashed about 12 runs in
20 on this. It survived until now because halt() stops in reverse topological
order — consumers first — so the producer whose callback might fire is always
still alive. shutdown() stops sources first and does not have that protection.

Reading stopped_ without a lock would not fix it: the window between the read
and the indexing is exactly where clear() runs. submit() takes a shared lock
and stop() an exclusive one, so submissions still proceed in parallel with
each other while being serialised against teardown. stop() sets the flag under
the lock, releases it to join — a worker's task may itself call submit, and
holding the lock across the join would deadlock against that — then retakes it
to destroy the queues.

Refusals are counted rather than silent. A teardown race is expected, but a
node repeatedly trying to run after its pool is gone is worth being able to
see. try_submit also checks stop_flag_ first, so a stopped node cannot claim
the submit gate and leave it held.

Not a smart-pointer problem, for anyone reading the crash: nothing here is
owned by a raw pointer. It is std::vector::operator[] on a vector that was
emptied by another thread.

Verified in both directions: with the guard removed the new scheduler cases
segfault; with it they pass.
 2026-08-05 15:14:06 +02:00
dtourolle 0f277c0f98 fix: the drain loops must terminate, and must drain the right channels 
shutdown()'s drain step was an unbounded

    while (anything, anywhere, is non-empty) poll every channel

Three defects in one loop.

It drained the wrong thing. Stopping a node should wait for that node's own
outputs before moving to the next layer; this waited for the entire graph to
fall idle each time. The dynamic Network's version made it explicit — it took
a node name and ignored it. Both now track which node feeds each probe and
wait only on those.

It had no deadline, so anything wedged downstream turned a graceful shutdown
into the hang it exists to avoid. Now bounded two ways, because a stalled
consumer and a slow one fail differently: a deadline for fill that never
changes, and a no-progress counter that keeps waiting as long as the queue is
shrinking, so a slow drain is not cut short merely for taking a while.

And it could fail to terminate with nothing wedged at all. current_fill came
from a snapshot that loaded tail_ before head_. A concurrent pop between the
two reads yields a head_ past the sampled tail_, and the unsigned difference
wraps to ~2^64 — so a poll for "is it empty yet" runs forever on a channel
that is in fact empty. Both indices only ever increase, so loading head_
first can at worst under-report a concurrent push, which this loop tolerates
and a wrap does not. size() and snapshot() are both corrected; size() feeds
approx_size(), which is what node readiness checks call.

Giving up is now reported rather than silent, because undrained data at that
point is about to be discarded by the stop that follows, and a graceful
shutdown quietly dropping values is the thing worth knowing about.

Verified in both directions: with the old loop the new case is killed at a
30 s timeout; with this it returns in under a second, having reported four
items its wedged consumer never took. Full suite 138/138.

Note the drain timeout is per node and defaults to 5 s, so a graph of N
stalled nodes can still take N x 5 s to shut down. That is a deliberate
trade against cutting off legitimate slow drains, and set_drain_timeout()
exists for callers who want it tighter.
 2026-08-05 14:25:00 +02:00
dtourolle 139bfbb794 fix: the sentinel slot holds one token and refuses a second 
push_sentinel wrote eof_value_ unconditionally. Offering a second token
before the first was taken did two wrong things at once.

It lost the first silently, and a lost EOF is not a lost frame — it is the
token every downstream node is waiting for in order to shut down, so losing
it wedges the pipeline.

And it wrote the storage while the consumer could be moving the previous
value out of it. I expected that to be a stale read; ThreadSanitizer shows
it is worse. On the shared_ptr storage that non-trivial types use, the
racing write tears the refcount, and the stress case added here reports
heap-use-after-free in extract() alongside the data race.

try_push_sentinel now refuses when the slot is occupied, which turns the
slot into a correct SPSC handshake: the producer is the only writer of
eof_value_ and the only one that sets has_eof_, the consumer is the only one
that clears it, so observing it false is what licenses the write. Refusal is
recorded as a drop, and PoolNode reports it through the overflow event
callback, because a refused control token going unnoticed is the failure
this commit exists to stop.

Refusing rather than queueing is deliberate. Two control tokens on one
channel means the stream ended twice, which is a caller protocol error and
not backpressure; parking and retrying would spin against a slot only the
consumer can free, and there is no sensible second value to deliver after
the end of a stream. The non-consuming try_push_sentinel exists so a refused
token is still the caller's to report — the consuming push_sentinel cannot
offer that, since the value has already been moved into its parameter.

Single-shot EOF is what every current caller does, so this is latent for
them today. It stops being latent the moment a pipeline is reused for a
second input, which is what the persistent-pipeline work in 4b6e498 sets up.

Verified in both directions under -DKPN_SANITIZER=thread: the new contended
case reports three data races and a heap-use-after-free against the old
overwrite, and is clean with the handshake. Full suite 137/137, TSan clean
across unit and stress suites.
 2026-08-05 14:06:26 +02:00
dtourolle 8d319eeb88 fix: an empty channel is not a closed one 
pop_one reported an empty channel the same way it reported a closed one, by
throwing ChannelClosedError, and fire_once treats that as "upstream is
finished" and calls self_stop(). self_stop disables the node's own inputs
*and* outputs, so a benign empty read does not merely skip a frame — it kills
the node and, through the disabled channels, whatever depended on it.

A node genuinely does get woken with empty inputs: a space callback fires
when its output drains, which has nothing to do with input arrival.
fire_once already guards against it by checking readiness before popping.
That guard is the live protection and it works; this commit makes the thing
it is guarding non-lethal.

So the pop_one path changed here is unreachable today, and I would rather say
that than imply a fixed hang. Its value is that the readiness check is now a
performance detail rather than the only thing standing between a routine wake
and a dead pipeline. Three separate comment blocks in fire_once exist to warn
about exactly this hazard; they were added because it had already been hit
during development, and the conflation they warn about is what this removes.

Verified in three directions. With the guard and the distinction: passes.
With the guard removed but the distinction present: still passes, which is
the point — the new ChannelEmptyError path catches what the guard used to.
With both removed, reproducing the original code: the node self-stops on the
firing that has nothing to read, the next value throws "channel closed" out
of its own output channel, and the relay handles one item instead of two.
 2026-08-05 13:55:57 +02:00
dtourolle 15e993f6ca fix: two firings of the same node must not overlap 
fire_once released the submit gate and then kept working:

    release_and_recheck();          // gate is now free
    if (stop_flag_) return;
    if (pending_) { ... }           // still reading node state
    on_input_ready();

The moment the gate is free another worker may enter fire_once for the same
node, so this invocation's reads of pending_ raced with the next one's writes
to pending_done_. ThreadSanitizer caught exactly that, between a firing
submitted by release_and_recheck and one submitted by try_submit.

The race is the visible half. The real damage is to the one-slot park, which
is sound only because "at most one fire_once runs per node at a time" — the
comment on pending_ says so explicitly. With two firings live, one can park a
value into the slot the other is about to overwrite, and the overwritten value
is gone with no drop recorded anywhere. That is silent data loss under
backpressure, from a node that reports itself healthy.

finish_firing() replaces release_and_recheck() at every exit: it evaluates the
follow-up decision — parked and waiting on output space, or drained and
waiting on input — while the claim is still held, and releases the gate as the
last thing the firing does. Nothing touches node state afterwards.

This also collapses three near-identical resubmit tails into one, which is
worth something on its own: the divergence between them is what 5628447 and
9c5ce5f were both picking at, and each fix had to be applied to every copy.

Pre-existing, not introduced by the gate rewrite: the old two-atomic version
cleared queued_ in the same place, with the same code after it.

Verified with -DKPN_SANITIZER=thread. The race is intermittent — roughly one
run in three before the fix — so five consecutive clean runs of the unit suite
plus the contended channel stress suite, all zero. Full suite 132/132.
 2026-08-05 13:48:13 +02:00
dtourolle a5c016833d fix: install channel callbacks before any node runs 
ThreadSanitizer reported ten data races on a plain multi-node network, all
the same one:

  Read  in Channel::try_push          -> push_callback_()      (worker thread)
  Write in Channel::set_push_callback -> push_callback_ = ...  (main thread)

A node's push and space callbacks are std::function members living on
channels it shares with its neighbours. register_callbacks() wrote them from
inside start(), and a network starts its nodes one at a time — so by the
time node N is being started, nodes 1..N-1 are already running and pushing
into N's input channel, reading the very std::function that start() is
assigning. Concurrent read and write of a std::function is a data race on
its vtable pointer and buffer, not a benign one.

This is the cause of the symptom a8cfe73 patched. That commit found nodes
missing their startup wake because "enable_inputs() opens the channel
several statements before register_callbacks() installs the push callback",
and fixed it by re-asking the question with on_input_ready(). The gap it
described is this race: the callback is not merely late, it is being written
while another thread reads it.

INode gains prepare(), which installs callbacks and starts nothing. Networks
call it on every node before starting any of them, so every write happens
while the pipeline is idle and the callbacks are read-only once it is live.
start() calls prepare() itself when a node is used standalone, and prepare()
is idempotent so both paths are safe. The flag is never cleared: the
callbacks capture `this` and stay valid across a restart, so re-registering
them would only add a pointless write to a live channel.

a8cfe73's on_input_ready() stays, and is still needed — a network starts
nodes one at a time, so an upstream node can still push into this one
between its prepare() and its start(), where on_input_ready() returns early
on stop_flag_ and the empty->non-empty edge is spent. It is now a
level-triggered check against a benign ordering rather than cover for a race.

Verified with -DKPN_SANITIZER=thread: ten races before, none of these after,
across the unit suite and the contended channel stress suite. One unrelated
race remains, on overlapping fire_once invocations; it is pre-existing and
is fixed separately.
 2026-08-05 13:39:26 +02:00
dtourolle f53af260a2 fix: make the submit gate a single atomic 
9c5ce5f established "a node never sleeps with a wake outstanding" and
implemented it as two independent atomics: queued_ for "a firing is in
flight", wake_pending_ for "a wake arrived during one". Two variables cannot
express that invariant, because the release side has to read and write both
and a wake can land in between:

  producer (try_submit)              worker (release_and_recheck)
  ------------------------           ----------------------------
  CAS reads queued_ == true, fails
                                     queued_.store(false)
                                     wake_pending_.exchange(false) -> false
  wake_pending_.store(true)

queued_ false, wake_pending_ true, nothing running and nothing scheduled —
exactly the state the invariant forbids. This is not a memory-ordering
subtlety; the interleaving holds under seq_cst.

SubmitGate replaces both with one atomic over three states, so "idle" and
"wake outstanding" are the same variable and no interleaving can produce
both. A release that finds a recorded wake keeps the claim and hands it to
the next firing, so the node is never momentarily idle while a submission
for it is in flight.

What this does not do is fix a reproducible hang. Every current call site
follows release_and_recheck() with a level re-check — on_input_ready(), or
outputs_have_space() on the parked path — which rediscovers the state a lost
wake would have signalled. The bug is masked, and I could not write a
node-level test that fails before and passes after; claiming otherwise would
be dishonest. The masking is a property of the call sites, not the
mechanism: any future early return that forgets its re-check reintroduces a
silent hang, and the pipeline has already been round that loop twice
(28e0667, then 9c5ce5f, each of which moved the stall rather than removing
it).

So the tests are structural. The state machine is pinned by contract tests,
and the defect it replaces is pinned by demonstration: LegacyGate in the
test file is the old protocol with a seam between the failed CAS and the
wake record, which makes the loss deterministic rather than something to
wait for. It also keeps the defect on record now that the code implementing
it is gone.

Also ignores build-*/ so a sanitizer build tree cannot be committed by
accident, which this commit did on its first attempt.
 2026-08-05 13:22:03 +02:00
dtourolle 5628447ea8 fix: never self-move the parked output tuple 
push_outputs ended with

    else pending_ = std::move(result);

and the retry path calls it as push_outputs(std::move(*pending_), …), so on
that path `result` is the parked tuple itself. The assignment was a
self-move-assignment. std::tuple's is elementwise, and libstdc++'s
std::vector does not guard against self-move: _M_move_assign swaps its data
into a temporary, which is then destroyed. The vector ends up empty.

So the first park was clean — the argument there is a local temporary — and
the second erased the payload. The value was still delivered, still in
order, still counted, just empty. Downstream cannot distinguish that from a
frame on which the node genuinely found nothing, which is why it would never
surface as an error: in scene-actor-extraction it reads as "no faces in this
frame" and the run completes with a quietly wrong answer.

Scope, stated precisely because I first got it wrong: this needs a node with
*two or more* outputs. With one output the only thing that resubmits a
parked node is that output's own space callback, which by definition fires
when there is room, so the retry always succeeds and never reassigns. With
two, output A draining resubmits the node while output B is still full — the
retry skips A (already delivered, tracked in pending_done_) and fails on B,
and that is the reassignment that eats B's payload.

Every node in the scene-actor-extraction pipeline currently has exactly one
output, and the fanout is a separate class that does not use pending_, so
this is latent there rather than active. It is reachable by any multi-output
node under backpressure, which the library supports and documents.

Verified in both directions: on 6a4f45f the parked payload arrives with size
0; here it arrives intact. The test drives raw channels rather than consumer
nodes so each step is forced rather than raced, and both channels are
capacity 1 — Channel fires the space callback only on the full->not-full
edge, so a roomy channel A would never resubmit the node and the retry would
never happen at all.
 2026-08-05 13:10:41 +02:00
dtourolle 6a4f45f111 fix: a filter or router must not drop on a full output 
RouterNode and FilterNode were the last nodes on a data path still using the
throwing push() and swallowing the result:

    try { out_ch_->push(val); } catch (const ChannelOverflowError&) {}

6595e6e made node outputs lossless, 28e0667 stopped them parking a worker,
a8cfe73 did the same for FanoutNode. These two were in none of them.

For ordinary values that is the familiar silent-loss problem: a dropped item
does not degrade a downstream result, it silently changes one, and the
consumer cannot tell it happened.

For a sentinel it is a hang. EOF is what tells every downstream node to shut
down and there is nothing after it to retry, so a filter that passes EOF by
predicate but drops it by backpressure produces a pipeline that never
terminates. scene-actor-extraction's decimator is exactly that shape —
`if (f.eof) return true;` in the predicate, feeding a chain whose slowest
node is an ONNX embedder, so the output is reliably full when EOF arrives.
Everything downstream then waits forever for a token that was discarded, and
the run has to be killed.

Both now route sentinels out-of-band via push_sentinel, which consumes no
ring capacity and cannot overflow, and retry ordinary values until taken.
Like FanoutNode and unlike a pool node, these own a private thread, so
waiting costs no scheduler worker and needs no space-callback park;
stop_flag_ is rechecked every pass so teardown cannot hang on a full output.
Time spent parked is charged to blocked rather than exec, so a held-up node
does not report as busy. An out-of-range router selector still drops by
design — that item was routed nowhere, which is not the same as lost.

is_sentinel_value moves from pool_node.hpp to traits.hpp. Every node type
that forwards a value needs it; these two not having it is the bug.

Verified in both directions. On c73edff the new case delivers 6 of 40 values
and never sets saw_eof; here it delivers 40 and terminates. EOF is emitted
exactly once, as a real source does — a test source that re-offered it would
mask the bug, since a later attempt could find the channel drained.

Note for downstream: the decimator is now a backpressure point rather than a
relief valve, so the source throttles to the face branch instead of quietly
thinning it. That is the intended behaviour, but it changes the shape of a
loaded run and is worth a benchmark comparison on a known clip.
 2026-08-05 12:59:39 +02:00
dtourolle c73edffe5c chore: delete the unreachable duplicate of the parked-retry block 
PoolNode::fire_once carried the pending_ retry block twice, verbatim. The
first copy returns on every path through it — parked, drained, or not
pending at all — so the second was dead code from the moment it appeared.
PoolObjectNode, which is otherwise a line-for-line twin of PoolNode, has it
once.

No behaviour change; the deleted 25 lines were unreachable. Worth doing
before the fixes queued behind it, each of which has to be applied once per
copy of this function.

The duplication is a symptom: PoolNode and PoolObjectNode are ~400 lines of
near-identical code maintained by parallel edit, and a block getting pasted
twice into one of them is exactly the failure that arrangement invites.
Factoring the shared body out is a larger change and wants its own review.
 2026-08-05 12:51:21 +02:00
dtourolle 091211cb19 fix: NodeSnapshot fields must line up with what nodes supply 
a8cfe73 appended queued/wake_pending/total_exec_ms to the NodeSnapshot
aggregate in an order no call site used. Every node type fills the aggregate
positionally and all of them supply total_exec_ms as the element straight
after queue_wait_ms — but the struct declared the two bools there. So the
exec total landed in `queued`, `queued` landed in `wake_pending`, and
`wake_pending` landed in total_exec_ms.

GCC reported it as -Wnarrowing (bool to double), 88 times, once per node
instantiation across the test build. The build carried on.

This is worse than a cosmetic mix-up, because all three fields were added
specifically to diagnose a wedge. A wedged pipeline reported total_exec_ms
as 0.0 or 1.0, and `queued` as "has this node ever run" — true for every
node that had, including ones asleep with nothing to do. The web debug JSON
served the same values. Anyone reading them to find the stalled node would
have been pointed at the wrong one.

Moves total_exec_ms above the two bools to match every call site, and notes
in the struct why the order is load-bearing.

Verified in both directions: on a8cfe73 the new case reports frames=8
total=0 queued=true; here total >= ema and both flags are false.
 2026-08-05 12:46:56 +02:00
dtourolle a8cfe7300a fix: a lossless fanout, a node that starts awake, and the instrumentation that found them 
Three changes from one debugging session on the intermittent wedge, kept
together because the instrumentation is what made the other two findable.

**Fanout was never made lossless.** 6595e6e made node outputs lossless and
28e0667 stopped them parking a worker; FanoutNode was in neither and kept
`catch (ChannelOverflowError&) {}` per output. Whichever branch fell behind
lost items, silently, by an amount that depended on timing — so two runs of
the same input could disagree. deliver() now retries each output
independently until it is taken, rechecking stop_flag_ every pass so
teardown cannot hang on a full output. A fanout owns a private thread, so
waiting costs no scheduler worker.

**A node could start with a wake already outstanding.** start() enables the
input channel several statements before it installs the push callback, and
StaticNetwork starts nodes sources-first, so an upstream node is already
firing into the gap. A push landing there is accepted by the ring but wakes
nobody: push_callback_ fires only on the empty→non-empty transition, and at
that instant the callback is null. Every later push sees a non-empty ring
and stays silent, so the node is never submitted. Asking on_input_ready()
once at the end of start() converts the missed edge into a state check.

The signature is distinctive — zero items delivered, not a stall partway.
Under `ctest -j4` on a loaded machine it reproduced 7 times in 24 and never
in 10 unloaded runs, which is almost certainly the "~1 run in 20" hang
28e0667 recorded as known-incomplete.

**NodeSnapshot now carries scheduling state and a true exec total.** queued
and wake_pending make the 9c5ce5f invariant observable at runtime; it could
previously only be inspected in a debugger, and the bug does not reproduce
under one. total_exec_us is a real sum — frames × ema_exec_us tracks the
tail of a run, not the whole of it, and diverges badly on a workload whose
per-frame cost varies. Both are exposed over the web debug JSON so a wedged
pipeline can be interrogated without attaching to it.
 2026-08-05 12:40:03 +02:00
dtourolle 454f72c167 chore: ignore generated ORT engine cache
🚦 CI / changes (push) Successful in 44s
Details
🚦 CI / docker (push) Has been skipped
Details
🚦 CI / test (push) Has been skipped
Details
🚦 CI / tsan (push) Has been skipped
Details
🚦 CI / docs (push) Has been skipped
Details 
ort_cache/ holds .ort engines built on first run from the .onnx models.
They are machine- and version-specific build products, not sources.
 2026-08-04 14:04:50 +02:00
dtourolle 9c5ce5f34a fix: never drop a wake — a node must not sleep with one outstanding
🚦 CI / changes (push) Successful in 14s
Details
🚦 CI / docker (push) Has been skipped
Details
🚦 CI / test (push) Failing after 4m21s
Details
🚦 CI / tsan (push) Failing after 3m20s
Details
🚦 CI / docs (push) Has been skipped
Details 
28e0667 stopped nodes blocking a worker on a full output, but replaced an
intermittent hang with a quieter one: the pipeline still wedged about 2
runs in 30, now with every worker idle in pthread_cond_wait rather than
asleep in a push. Nothing was blocked; nothing had been woken.

try_submit discarded any wake arriving while queued_ was up:

    if (queued_.compare_exchange_strong(expected, true, ...))
        scheduler_->submit(...);
    // else: silently gone

Wakes are edge-triggered — a channel fires its space callback once, on the
transition — so a dropped one never returns. A node could park a value,
release its worker, and sleep forever holding exactly the output its
consumer was waiting for, while its producer parked on an input channel
that would never drain.

try_submit now records the drop in wake_pending_, and release_and_recheck()
consumes it at every site that releases a node, giving one invariant: a
node never sleeps with a wake outstanding. This subsumes the two ad-hoc
re-checks added for the parked-retry and normal push paths, which only
moved the stall (1211 items to 6472) because each new early return was a
fresh chance to drop a wake. self_stop keeps a plain store — honouring a
pending wake there would resubmit a dead node.

Adds "a saturated chain never stalls", which reproduces this in about a
second where the pipeline needed ~30 runs. It asserts *progress does not
freeze* rather than a completion total: capacity-1 channels are slow, and
slow must never be reported as wedged. Verified in both directions — it
stalls after 1211 items on 28e0667 and passes here.

The existing chain test could not catch it: 40 items drain before any
strand occurs, and its producer emits forever, so fresh input keeps
re-triggering on_input_ready() and flushing the stranded value.

Tests: 122/122.
 2026-08-02 18:33:17 +02:00
dtourolle 28e06675f5 fix: park nodes on a full output instead of blocking the worker
🚦 CI / changes (push) Successful in 6s
Details
🚦 CI / docker (push) Has been skipped
Details
🚦 CI / test (push) Failing after 4m20s
Details
🚦 CI / tsan (push) Failing after 3m16s
Details
🚦 CI / docs (push) Has been skipped
Details 
push_blocking parked a scheduler worker inside the push. Nodes own a
private single-thread pool, so the parked thread was the only one that
could drain that node's own input — hold-and-wait, and under sustained
backpressure four nodes of a five-node chain slept in nanosleep at once.
channel.hpp already warned about this for sentinels; it applies just as
much to data pushes.

The scheduler was purely input-driven: on_input_ready() wakes a node when
input arrives, with no counterpart for "my output has room". Lacking that
signal, blocking the thread was the only way to handle a full output.
This adds the missing half.

- Channel::try_push + has_space + set_space_callback; the callback fires
  from both pop() and try_pop_now().
- PoolNode/PoolObjectNode keep a one-slot pending_ buffer with per-element
  done flags, so a retry cannot duplicate an already-accepted element. One
  slot suffices because queued_ admits at most one fire_once per node.
- The re-check after clearing queued_ closes the lost-wakeup race where a
  space callback fires while the flag is still up and is swallowed.

Two bugs surfaced once nodes actually parked, both fixed here:

- pop_one reports an *empty* channel as ChannelClosedError, which is also
  the node's "upstream finished, self-stop" signal. A node woken by output
  space with empty inputs therefore killed itself. fire_once now releases
  the worker when its inputs are not ready rather than falling through.
- The drained-park path resubmitted unconditionally instead of via
  on_input_ready(), firing nodes with nothing to read.

compute_priority is now output-aware: mean output fill is deducted from
mean input fill, mapped as 0.5·(1 + in - out). Input fill alone asks only
"how much work is waiting for me"; a node whose outputs are already full
cannot deliver, so running it just parks it again and wastes the slot
while the node that would drain that channel waits behind it. The
scheduler now favours whoever is furthest downstream of a bottleneck.

Also adds a network-level error listener. A node's exception was discarded
at the node boundary and survived only as a Closed event, which reports
that a node stopped but not why — that missing detail is what made the
above slow to diagnose. INode::set_network_error_callback plus
StaticNetwork::set_error_handler forward it to the application.

Tests: 121/121. test_backpressure_deadlock drives a five-node chain with
capacity-2 channels against a slow sink and fails on the old code. The
four test_pool_node overflow tests now assert parking rather than the
removed drop-and-report behaviour.

Known-incomplete: a rare hang remains, roughly 1 run in 20 against a 300s
timeout, down from every run failing. Committed because the fix is a large
strict improvement and the residual case needs its own reproduction.
 2026-07-31 22:40:07 +02:00
dtourolleandClaude Opus 5 6595e6e925 fix: node outputs block instead of dropping on a full channel
🚦 CI / changes (push) Successful in 30s
Details
🚦 CI / docker (push) Has been skipped
Details
🚦 CI / test (push) Failing after 1h44m2s
Details
🚦 CI / docs (push) Has been skipped
Details
🚦 CI / tsan (push) Failing after 3h14m57s
Details 
Every node output used the throwing push(), so a consumer falling behind cost
values rather than time. push_blocking() already existed on Channel and
OutputPort — "wait for the consumer to drain instead of dropping; the producer
just runs slower" — but nothing called it.

A dropped frame does not degrade a downstream result, it silently changes one,
and the consumer has no way to tell it happened. For any pipeline whose output
is a claim about its input, that is corruption rather than degradation.

Safe because sentinels are already handled out-of-band, above this path: only
data blocks, so the EOF token that unwinds the network can always overtake a
stalled data path. That is exactly the hold-and-wait deadlock the push_sentinel
comment warns about, and the reason it is not reachable here.

Measured on a downstream consumer (face pipeline, 77s clip at 5 fps, expected
385 sampled frames):

  before  65 frames written, 320 dropped at one node, 29s
  after   385 frames written, 0 dropped, 17s

Faster, not slower — a dropped frame has already cost its decode, and the
overflow exception cost more. Two consecutive runs now produce byte-identical
output, which they did not before: what got dropped depended on timing, so the
same command could yield different results.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-07-31 11:31:53 +02:00
dtourolle 75b34f31bb Merge pull request 'feat: persistent-pipeline reuse — push_blocking, node introspection, stateful wrapper' (#2) from feature/persistent-pipeline-reuse into master
🚦 CI / changes (push) Successful in 4s
Details
🚦 CI / docker (push) Has been skipped
Details
🚦 CI / test (push) Successful in 4m39s
Details
🚦 CI / tsan (push) Successful in 2m57s
Details
🚦 CI / docs (push) Has been skipped
Details 
Reviewed-on: #2
 2026-07-19 16:11:27 +00:00
dtourolle 4b6e498ba7 feat: persistent-pipeline reuse — push_blocking, node introspection, stateful wrapper
🚦 CI / changes (pull_request) Successful in 17s
Details
🚦 CI / docker (pull_request) Has been skipped
Details
🚦 CI / test (pull_request) Successful in 4m42s
Details
🚦 CI / tsan (pull_request) Successful in 2m56s
Details
🚦 CI / docs (pull_request) Has been skipped
Details 
Adds three pieces needed to build one KPN network and reuse it across many
replays/configs instead of tearing down and rebuilding per run:

- Channel<T>::push_blocking (+ IVariantChannel/VariantChannel forwarding):
  lossless backpressure push that waits for space instead of dropping when
  the ring is full. PyNode's run_loop now uses it so a downstream consumer
  lagging behind never silently drops a frame.
- PyNetwork::node_ptr / node_stats: raw node handle by name (for a binding
  to dynamic_cast to a concrete wrapper and call functor-specific runtime
  setters) and a per-node timing snapshot for profiling.
- ObjectVariantNodeWrapper: variant-node adapter for functors that need
  runtime-constructed state (a Config, a loaded gallery), mirroring
  VariantNodeWrapper's channel plumbing but backed by ObjectNode<Obj>.

Built and used downstream in scene-actor-extraction's sae_kpn Python replay
bindings for repeated threshold-sweep evaluation of the same pipeline.
 2026-07-19 16:55:36 +02:00
dtourolle 5ecf3cde4f Merge pull request 'spec-and-tsan' (#1) from spec-and-tsan into master
🚦 CI / changes (push) Successful in 4s
Details
🚦 CI / docker (push) Has been skipped
Details
🚦 CI / test (push) Successful in 4m36s
Details
🚦 CI / tsan (push) Successful in 2m54s
Details
🚦 CI / docs (push) Has been skipped
Details 
Reviewed-on: #1
 2026-07-17 18:11:51 +00:00
dtourolleandClaude Opus 4.8 ec19137ed9 ci: fix TSan aborting at init on the nested-LXC runner
🚦 CI / changes (pull_request) Successful in 6s
Details
🚦 CI / docker (pull_request) Has been skipped
Details
🚦 CI / test (pull_request) Successful in 4m32s
Details
🚦 CI / tsan (pull_request) Successful in 2m54s
Details
🚦 CI / docs (pull_request) Has been skipped
Details 
The ThreadSanitizer job runs on Docker nested in an unprivileged LXC
container, whose kernel randomizes mmap addresses beyond the range TSan's
fixed shadow mapping expects. TSan aborted at init with "unexpected memory
mapping" before any test ran.

Disable ASLR per-process with `setarch -R`, which needs the personality(2)
syscall that Docker's default seccomp profile blocks; seccomp=unconfined on
the container permits it. Verified on the runner that both are required:
setarch -R alone gets EPERM, seccomp alone still aborts, both together run
clean. Scoped to the tsan job, which runs only our own test binaries.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-07-17 20:01:57 +02:00
dtourolleandClaude Opus 4.8 a0c4bf580e fix: deliver EOF sentinel only when the ring is freshly empty
🚦 CI / changes (pull_request) Successful in 4s
Details
🚦 CI / docker (pull_request) Has been skipped
Details
🚦 CI / test (pull_request) Successful in 7m30s
Details
🚦 CI / tsan (pull_request) Failing after 2m21s
Details
🚦 CI / docs (pull_request) Has been skipped
Details 
Channel<T>::pop() surfaced the out-of-band sentinel from its empty branch
using the tail_ snapshot taken at the top of the loop. Under contention the
producer can push more values *and* the sentinel in the window between that
snapshot and take_sentinel(), so pop() could return the sentinel while real
values still sat in the ring — the sentinel jumping ahead of values pushed
before it. No value was lost (a consumer that keeps draining still receives
them, and approx_size() keeps counting them so a PoolNode reschedules), but a
consumer treating the sentinel as a hard "last message" barrier would act on
EOF early.

Re-confirm emptiness against a fresh tail_ load before taking the sentinel.
Costs one acquire-load on the empty-ring path only; never runs in steady
state. The spin and post-spin takes already reload tail_ on the line above
them; try_pop_now() already reads tail_ fresh in the same branch — both were
correct and are unchanged.

The two sentinel stress cases now assert the strict "sentinel is last, after
every value" ordering (previously relaxed to avoid the flake this fixes).
Verified TSan-clean (2606 assertions, no data races) over repeated runs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-07-14 23:14:45 +02:00
dtourolleandClaude Opus 4.8 3ac2242df1 docs: rewrite SPEC.md to match the implemented library 
The spec had drifted far from the code. Key corrections:

- Execution model is reactive (PoolNode submits fire_once() to a
  ThreadPool when inputs are ready), not one blocking thread per node
- Channel<T> is a lock-free SPSC ring buffer (atomic wait/notify +
  spin-before-sleep), not a mutex+CV queue
- Remove latch<> ports (never implemented)
- NodeErrorHandler returns bool (skip vs stop); per-node
- Document new subsystems: scheduler, InterruptNode, Router/FilterNode,
  MainThreadNode, SharedResource, DebugHub, diagnostics/stats layer
- Update StaticNetwork, Python auto_bind layer, examples 01-16

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-07-14 23:14:45 +02:00
dtourolleandClaude Opus 4.8 399ee4cf9b test: add ThreadSanitizer verification for lock-free Channel<T> 
Add a contended SPSC stress suite (tests/test_channel_stress.cpp) that
actually exercises the ring's memory-ordering pairing and spin/futex/
lost-wakeup logic, plus the CMake and CI plumbing to run it under TSan:

- KPN_SANITIZER cache var + kpn_sanitizer_flags() helper (no-op when unset)
- kpn_tests_stress executable, labelled "stress" for CTest
- reusable tsan.yaml workflow (gcc:14 builder image, already ships libtsan)
- ci.yaml gains a tsan job on the same code/dockerfile triggers as test

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-07-14 23:14:45 +02:00
dtourolle 66feb91821 ci: pass docker push input as a string, not a boolean
🚦 CI / changes (push) Successful in 6s
Details
🚦 CI / docker (push) Successful in 2m57s
Details
🚦 CI / test (push) Successful in 8m7s
Details
🚦 CI / docs (push) Successful in 6s
Details 
Gitea's act_runner mangles boolean workflow_call/dispatch inputs passed
from an expression -- they arrive as false regardless of value. Declare
`push` as a string ("true"/"false") and compare with == 'true' so the
builder image is pushed on non-PR events again.
 2026-07-04 15:07:27 +02:00
dtourolle 903dd4eea5 docs: update README examples table and add documentation link 
07_python_network and 08_python_subport now work and run as CI smoke
tests, so drop their "(pending)" markers and describe what they actually
demonstrate. Also surface the hosted documentation link at the top and
re-render README.md from README.md.in.
 2026-07-04 15:07:12 +02:00
dtourolle 298c9e770b examples: run Python examples 07 and 08 as CTest smoke tests 
Neither Python example was registered as a test, so `ctest -L examples`
in CI skipped them entirely -- which is how 08's missing Python node
went unnoticed.

Add a kpn_python_example() helper (gated on KPN_BUILD_PYTHON) that runs
each script with PYTHONPATH pointed at the freshly-built module, so it
does not depend on cwd or a hard-coded build/python path, and register
07 and 08. Also del the network in 07 for deterministic teardown.
 2026-07-04 14:44:30 +02:00
dtourolle 2b0873b61b examples: make 08_python_subport run a real Python node 
The example was named "python subport" but its graph was entirely C++
(ProduceNode -> DoubleItNode); Python only tapped the output, leaving a
dangling "#todo: return value to network".

Rewrite so the only node in the graph is a pure-Python py_triple, driven
from both ends via the subport taps: net.write() injects inputs and
net.read() pulls results back, closing the round trip. Also del the
network at the end so its callable cycle is reclaimed deterministically.
 2026-07-04 14:44:24 +02:00
dtourolle c4538f03ca python: fix nanobind Network reference leak via GC type slots 
The Python Network holds each PyNode's callable, forming an
uncollectable instance -> callable -> globals() -> instance cycle that
tripped nanobind's leak check at interpreter shutdown.

Implement tp_traverse/tp_clear type slots on the Network binding so
Python's cyclic collector can see through the C++-held callables and
break the cycle. PyNode exposes its callable; PyNetwork visits and
clears them. Wired into both binding sites (auto_bind and the legacy
register_py_network).
 2026-07-04 14:43:47 +02:00
dtourolleandClaude Opus 4.8 949c8134ef Ignore build_test/ (out-of-tree test build dir)
🚦 CI / changes (push) Successful in 3s
Details
🚦 CI / docker (push) Has been skipped
Details
🚦 CI / test (push) Has been skipped
Details
🚦 CI / docs (push) Has been skipped
Details 
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-07-04 11:02:54 +02:00
dtourolleandClaude Opus 4.8 19f5a2b0ae Deliver EOF sentinels out-of-band to prevent teardown deadlock
🚦 CI / changes (push) Successful in 18s
Details
🚦 CI / docker (push) Has been skipped
Details
🚦 CI / docs (push) Has been cancelled
Details
🚦 CI / test (push) Has been cancelled
Details 
Channel::push() drops values on overflow (the intended backpressure
policy for data), and PoolNode swallows the resulting ChannelOverflowError.
For a control sentinel like EOF this is fatal: a single dropped EOF under
backpressure wedges every downstream pop() forever, so the pipeline never
tears down.

Deliver sentinels out-of-band instead. Channel::push_sentinel() stores the
token in a dedicated slot that does not consume ring capacity, so it can
never overflow and — crucially — never blocks the caller. That non-blocking
property is essential: each KPN node has a single worker thread, so a
*blocking* push would park that thread and stop it draining its own input,
cascading into a hold-and-wait deadlock under backpressure. The consumer's
pop()/try_pop_now() drain the ring first, then deliver the sentinel, so it
always arrives after every value pushed before it.

approx_size() (which node readiness checks call) counts a pending sentinel
as consumable work, so a channel carrying only a sentinel still schedules
its consumer's next fire — without this the token would sit undelivered and
the pipeline would still deadlock at teardown.

PoolNode/PoolObjectNode route values carrying an eof flag (direct .eof or
nested .source.eof) through push_sentinel via a SFINAE-safe is_sentinel_value
trait; all other values keep the existing lossy throwing push. The trait
compiles to false for types without an eof convention, so this is a no-op
for pipelines that don't use one.

Verified end-to-end: scene_analyze now reaches EOF, flushes its output, and
exits cleanly instead of hanging.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-07-04 00:30:40 +02:00
dtourolle a4de64ea04 Add liscence and prepare for OSS release
🚦 CI / changes (push) Successful in 39s
Details
🚦 CI / docker (push) Has been skipped
Details
🚦 CI / test (push) Has been skipped
Details
🚦 CI / docs (push) Successful in 7s
Details
2026-06-28 12:05:56 +02:00
dtourolleandClaude Opus 4.8 7c6a8be2b7 ci: provoke pipeline (touch Dockerfile + code)
🚦 CI / changes (push) Successful in 32s
Details
🚦 CI / docker (push) Successful in 1m0s
Details
🚦 CI / test (push) Successful in 4m47s
Details
🚦 CI / docs (push) Successful in 13s
Details 
Trivial comment changes to exercise the new CI orchestration: the
docker job should rebuild+push the builder image first, then test
and docs run against the fresh image.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-20 09:46:35 +02:00
dtourolle 4c0f1f6923 fix CI
🚦 CI / changes (push) Successful in 33s
Details
🚦 CI / docker (push) Has been skipped
Details
🚦 CI / test (push) Has been skipped
Details
🚦 CI / docs (push) Has been skipped
Details
2026-06-20 09:38:26 +02:00
dtourolle 6b52526e44 Auto-build docker when docker file changes.
🚦 CI / changes (push) Failing after 5s
Details
🚦 CI / docker (push) Has been skipped
Details
🚦 CI / test (push) Has been skipped
Details
🚦 CI / docs (push) Has been skipped
Details
2026-06-20 09:16:27 +02:00
dtourolle 7cb92a4091 Build docs with a pre-configured docker
🧪 Test / test (push) Successful in 4m41s
Details
2026-06-20 08:55:48 +02:00
dtourolle 20668d6955 Fix docker image
🧪 Test / test (push) Successful in 7m31s
Details
2026-06-19 22:29:46 +02:00
dtourolle 6f384dc4b5 Added callbacks for node errors and fifo overflow
📚 Docs / deploy (push) Failing after 7s
Details
🧪 Test / test (push) Has been cancelled
Details 
Add new doc system which should/might deploy to pages.
 2026-06-19 22:26:39 +02:00
dtourolleandClaude Opus 4.8 79916f1da1 Set node names in make_network for user and fanout nodes
🧪 Test / test (push) Successful in 6m22s
Details 
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-12 15:20:50 +02:00
dtourolle f6bcaa15b0 Performance improvements, better readme and complete python bindings
🧪 Test / test (push) Failing after 28m30s
Details
2026-05-12 21:23:33 +02:00
dtourolle c39db82763 Add a per node error handler possibility
🧪 Test / test (push) Successful in 6m7s
Details
2026-05-10 19:51:23 +02:00
dtourolle 1e9ba5ee66 Add a unified observability interface for applications with multiple networks
🧪 Test / test (push) Successful in 6m9s
Details
2026-05-10 19:08:40 +02:00
dtourolle 278c122e8f Add shared reasource tag to allow coordination of usage
🧪 Test / test (push) Successful in 6m8s
Details
2026-05-09 15:22:27 +02:00
dtourolle 9acc42b2e9 Fix fanout naming for debug graph
🧪 Test / test (push) Failing after 2m38s
Details
2026-05-09 09:54:40 +02:00
dtourolle da8f4d9926 Add debug graph to static network 2026-05-09 08:59:19 +02:00
dtourolle 011b5eb35f fix web debuginterface 2026-05-09 08:40:06 +02:00
dtourolle 9ce581b5ce Fixed bug when generating identical fanouts 2026-05-09 08:36:51 +02:00
dtourolle 2bca2a7554 Add static network
🧪 Test / test (push) Successful in 6m4s
Details
2026-05-08 20:00:15 +02:00
dtourolle 3c683c821d Add more exmaples and fix CI
🧪 Test / test (push) Successful in 5m59s
Details
2026-05-08 18:28:12 +02:00
dtourolle 127ffb3849 Fix CI
🧪 Test / test (push) Successful in 6m47s
Details
🐳 Build Builder Image / build-and-push (push) Failing after 1s
Details
2026-05-08 18:09:13 +02:00
dtourolle 2a5c0a0b4d Add build infra
🧪 Test / test (push) Failing after 1s
Details
🐳 Build Builder Image / build-and-push (push) Failing after 1s
Details
2026-05-08 18:00:03 +02:00
dtourolle 5e77dc836b First attempt 2026-05-08 17:48:16 +02:00
148 changed files with 17257 additions and 18886 deletions
Expand all files Collapse all files
.gitea/workflows/ci.yaml
+87
View File
@@ -0,0 +1,87 @@
name: '🚦 CI'
# Single orchestrator. This is the only workflow that triggers on push/PR.
# It decides which reusable sub-workflows to run and in what order:
# changes ─┬─> docker (only if the Dockerfile/requirements changed) ─┬─> test
# │ └─> docs
# When the builder image is rebuilt it MUST finish (and push) before test/docs
# run, so they validate against the fresh image.
on:
push:
branches:
- master
- develop
pull_request:
branches:
- master
- develop
workflow_dispatch:
jobs:
# Detect which parts of the repo changed in this push/PR.
changes:
runs-on: linux/amd64
# Runs in the builder image because the host has no Node, which the
# JS-based checkout/paths-filter actions require.
container:
image: gitea.tourolle.paris/dtourolle/kpnpp-builder:latest
outputs:
dockerfile: ${{ steps.filter.outputs.dockerfile }}
code: ${{ steps.filter.outputs.code }}
docs: ${{ steps.filter.outputs.docs }}
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Detect changed paths
id: filter
uses: dorny/paths-filter@v3
with:
filters: |
dockerfile:
- 'Dockerfile.builder'
- 'docs/requirements.txt'
docs:
- 'docs/**'
- 'mkdocs.yml'
- 'examples/**/*.cpp'
code:
- 'src/**'
- 'include/**'
- 'tests/**'
- 'examples/**'
- 'python/**'
- 'CMakeLists.txt'
- '**/*.cpp'
- '**/*.hpp'
- '**/*.h'
# Rebuild the builder image first, but only when it actually changed.
# On pull requests we build to validate the Dockerfile but do not push.
docker:
needs: changes
if: ${{ needs.changes.outputs.dockerfile == 'true' }}
uses: ./.gitea/workflows/docker.yaml
with:
# Explicit string, not a boolean expression (act_runner mangles bools).
push: ${{ github.event_name == 'pull_request' && 'false' || 'true' }}
# Runs after docker (if docker ran). A skipped docker job is fine; a failed
# one blocks this via !failure(). Re-run tests when code OR the image changed.
test:
needs: [changes, docker]
if: ${{ !failure() && !cancelled() && (needs.changes.outputs.code == 'true' || needs.changes.outputs.dockerfile == 'true') }}
uses: ./.gitea/workflows/test.yaml
# ThreadSanitizer run for the lock-free Channel<T>. Same trigger conditions as
# test (code or image changed); runs in parallel with test.
tsan:
needs: [changes, docker]
if: ${{ !failure() && !cancelled() && (needs.changes.outputs.code == 'true' || needs.changes.outputs.dockerfile == 'true') }}
uses: ./.gitea/workflows/tsan.yaml
docs:
needs: [changes, docker]
if: ${{ !failure() && !cancelled() && github.ref == 'refs/heads/master' && (needs.changes.outputs.docs == 'true' || needs.changes.outputs.dockerfile == 'true') }}
uses: ./.gitea/workflows/docs.yaml
secrets: inherit
.gitea/workflows/docker.yaml
+57
View File
@@ -0,0 +1,57 @@
name: '🐳 Builder Image'
# Reusable workflow: builds (and optionally pushes) the kpnpp-builder image.
# It is called by ci.yaml only when Dockerfile.builder or docs/requirements.txt
# change. It runs on the host runner (NOT inside the builder container) because
# it needs the Docker CLI/daemon.
# Note: `push` is a STRING ("true"/"false"), not a boolean. Gitea's act_runner
# mangles boolean inputs passed from an expression (they arrive as false), so we
# pass an explicit string and compare with == 'true' below.
on:
workflow_call:
inputs:
push:
description: 'Push the built image to the registry ("true"/"false")'
type: string
default: 'true'
workflow_dispatch:
inputs:
push:
description: 'Push the built image to the registry ("true"/"false")'
type: string
default: 'true'
jobs:
build:
runs-on: linux/amd64
steps:
# This job runs on the host (not in a container) so it can reach the
# host Docker daemon and reuse the cached registry credentials. The host
# has no Node, so the JS-based actions/checkout can't run here; do a
# minimal shallow fetch of this commit with plain git instead.
- name: Checkout repository
run: |
git init -q .
git remote add origin "${{ github.server_url }}/${{ github.repository }}.git"
git -c http.extraheader="AUTHORIZATION: basic $(printf '%s' '${{ github.actor }}:${{ github.token }}' | base64 -w0)" \
fetch --depth 1 origin "${{ github.sha }}"
git checkout -q FETCH_HEAD
# No docker login step: the host runner was authenticated to
# gitea.tourolle.paris with `docker login` during setup, so its cached
# credentials in ~/.docker/config.json cover the push below.
- name: Build builder image
# Context is the repo root because Dockerfile.builder COPYs
# docs/requirements.txt during the build.
run: |
docker build \
-f Dockerfile.builder \
-t gitea.tourolle.paris/dtourolle/kpnpp-builder:latest \
-t gitea.tourolle.paris/dtourolle/kpnpp-builder:${{ github.sha }} \
.
- name: Push builder image
if: ${{ inputs.push == 'true' }}
run: |
docker push gitea.tourolle.paris/dtourolle/kpnpp-builder:latest
docker push gitea.tourolle.paris/dtourolle/kpnpp-builder:${{ github.sha }}
.gitea/workflows/docs.yaml
+34
View File
@@ -0,0 +1,34 @@
name: '📚 Docs'
# Triggering and path filtering are owned by ci.yaml (the orchestrator), which
# calls this as a reusable workflow. workflow_dispatch is kept for manual runs.
on:
workflow_call:
workflow_dispatch:
jobs:
deploy:
runs-on: linux/amd64
container:
image: gitea.tourolle.paris/dtourolle/kpnpp-builder:latest
steps:
- name: Checkout repository
uses: actions/checkout@v4
with:
fetch-depth: 0 # full history needed for mkdocs gh-deploy
- name: Configure git identity
run: |
git config user.name "Gitea Actions"
git config user.email "actions@gitea.tourolle.paris"
- name: Build and deploy to gitea-pages branch
env:
GH_TOKEN: ${{ github.token }}
run: |
mkdocs gh-deploy \
--force \
--remote-branch gitea-pages \
--remote-name origin \
--message "docs: deploy from ${{ github.sha }}"
.gitea/workflows/test.yaml
+67
View File
@@ -0,0 +1,67 @@
name: '🧪 Test'
# Triggering and path filtering are owned by ci.yaml (the orchestrator), which
# calls this as a reusable workflow. workflow_dispatch is kept for manual runs.
on:
workflow_call:
workflow_dispatch:
jobs:
test:
runs-on: linux/amd64
container:
image: gitea.tourolle.paris/dtourolle/kpnpp-builder:latest
steps:
- name: Checkout repository
uses: actions/checkout@v4
with:
path: test-${{ github.run_id }}
- name: Cache FetchContent dependencies
uses: actions/cache@v3
with:
path: ~/.cmake/fetchcontent
key: cmake-fetchcontent-${{ hashFiles('**/CMakeLists.txt') }}
restore-keys: cmake-fetchcontent-
- name: Configure
working-directory: test-${{ github.run_id }}
run: |
cmake -S . -B build \
-G Ninja \
-DCMAKE_BUILD_TYPE=Debug \
-DKPN_BUILD_TESTS=ON \
-DKPN_BUILD_EXAMPLES=ON \
-DKPN_BUILD_PYTHON=ON \
-DFETCHCONTENT_BASE_DIR=$HOME/.cmake/fetchcontent
- name: Build
working-directory: test-${{ github.run_id }}
run: cmake --build build --parallel
- name: Run unit tests
working-directory: test-${{ github.run_id }}
run: |
cd build
ctest --output-on-failure --output-junit test-results.xml --label-exclude examples
- name: Run example smoke tests
working-directory: test-${{ github.run_id }}
run: |
cd build
ctest --output-on-failure --output-junit example-results.xml -L examples
- name: Upload test results
if: always()
uses: actions/upload-artifact@v3
with:
name: test-results
path: |
test-${{ github.run_id }}/build/test-results.xml
test-${{ github.run_id }}/build/example-results.xml
retention-days: 7
- name: Cleanup
if: always()
run: rm -rf test-${{ github.run_id }}
.gitea/workflows/tsan.yaml
+77
View File
@@ -0,0 +1,77 @@
name: '🧵 ThreadSanitizer'
# Reusable workflow: builds the channel stress suite with ThreadSanitizer and
# runs it. This is the dynamic half of verifying the lock-free SPSC Channel<T>
# (the static half is the CDSChecker model-check harness in verify/).
#
# Triggering and path filtering are owned by ci.yaml (the orchestrator), which
# calls this only when code changed. workflow_dispatch is kept for manual runs.
#
# Runs in the prebuilt builder image (gcc:14), which already ships libtsan — no
# package installs at job time.
on:
workflow_call:
workflow_dispatch:
jobs:
tsan:
runs-on: linux/amd64
container:
image: gitea.tourolle.paris/dtourolle/kpnpp-builder:latest
# This runner is Docker nested in an unprivileged LXC container, whose
# kernel randomizes mmap addresses beyond the range TSan's fixed shadow
# mapping expects, so TSan aborts at init with "unexpected memory
# mapping". The fix is to disable ASLR per-process with `setarch -R`
# (below), which needs the personality(2) syscall that Docker's default
# seccomp profile blocks. seccomp=unconfined permits it. Verified on the
# runner: setarch -R alone gets EPERM, seccomp alone still aborts, both
# together run clean. Scoped to this job, which runs only our own tests.
options: --security-opt seccomp=unconfined
steps:
- name: Checkout repository
uses: actions/checkout@v4
with:
path: tsan-${{ github.run_id }}
- name: Cache FetchContent dependencies
uses: actions/cache@v3
with:
path: ~/.cmake/fetchcontent
key: cmake-fetchcontent-${{ hashFiles('**/CMakeLists.txt') }}
restore-keys: cmake-fetchcontent-
- name: Configure (TSan)
working-directory: tsan-${{ github.run_id }}
run: |
cmake -S . -B build \
-G Ninja \
-DCMAKE_BUILD_TYPE=Debug \
-DKPN_SANITIZER=thread \
-DKPN_BUILD_TESTS=ON \
-DKPN_BUILD_EXAMPLES=OFF \
-DKPN_BUILD_PYTHON=OFF \
-DFETCHCONTENT_BASE_DIR=$HOME/.cmake/fetchcontent
- name: Build (TSan)
working-directory: tsan-${{ github.run_id }}
run: cmake --build build --parallel --target kpn_tests kpn_tests_stress
- name: Run stress suite under TSan
working-directory: tsan-${{ github.run_id }}
# halt_on_error=1 makes the first detected race fail the job; the report
# (with both stacks) is printed to the log. second_deadlock_stack gives
# the full picture for lock-order issues.
env:
TSAN_OPTIONS: "halt_on_error=1 second_deadlock_stack=1"
# setarch -R disables ASLR for this process; see the container comment.
run: setarch -R ./build/tests/kpn_tests_stress
- name: Run unit tests under TSan
working-directory: tsan-${{ github.run_id }}
env:
TSAN_OPTIONS: "halt_on_error=1 second_deadlock_stack=1"
run: setarch -R ./build/tests/kpn_tests
- name: Cleanup
if: always()
run: rm -rf tsan-${{ github.run_id }}
.gitignore
+32
View File
@@ -0,0 +1,32 @@
# Build output
build/
build_test/
build_debug/
site/
# Python
__pycache__/
*.py[cod]
*.pyd
*.pyo
*.egg-info/
dist/
*.egg
.venv/
venv/
# Editors
.vscode/
.idea/
*.swp
*.swo
*~
# OS
.DS_Store
Thumbs.db
# Claude Code local settings
.claude/settings.local.json
include/kpn/ort_cache/
build-tsan/
build-*/
.nojekyll
View File
404.html
-830
View File
@@ -1,830 +0,0 @@
<!doctype html>
<html lang="en" class="no-js">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width,initial-scale=1">
<meta name="description" content="A C++20 Kahn Process Network library">
<link rel="icon" href="/assets/images/favicon.png">
<meta name="generator" content="mkdocs-1.6.1, mkdocs-material-9.7.6">
<title>KPN++</title>
<link rel="stylesheet" href="/assets/stylesheets/main.484c7ddc.min.css">
<link rel="stylesheet" href="/assets/stylesheets/palette.ab4e12ef.min.css">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Roboto:300,300i,400,400i,700,700i%7CRoboto+Mono:400,400i,700,700i&display=fallback">
<style>:root{--md-text-font:"Roboto";--md-code-font:"Roboto Mono"}</style>
<script>__md_scope=new URL("/",location),__md_hash=e=>[...e].reduce(((e,_)=>(e<<5)-e+_.charCodeAt(0)),0),__md_get=(e,_=localStorage,t=__md_scope)=>JSON.parse(_.getItem(t.pathname+"."+e)),__md_set=(e,_,t=localStorage,a=__md_scope)=>{try{t.setItem(a.pathname+"."+e,JSON.stringify(_))}catch(e){}}</script>
</head>
<body dir="ltr" data-md-color-scheme="slate" data-md-color-primary="indigo" data-md-color-accent="indigo">
<input class="md-toggle" data-md-toggle="drawer" type="checkbox" id="__drawer" autocomplete="off">
<input class="md-toggle" data-md-toggle="search" type="checkbox" id="__search" autocomplete="off">
<label class="md-overlay" for="__drawer"></label>
<div data-md-component="skip">
</div>
<div data-md-component="announce">
</div>
<header class="md-header" data-md-component="header">
<nav class="md-header__inner md-grid" aria-label="Header">
<a href="/." title="KPN++" class="md-header__button md-logo" aria-label="KPN++" data-md-component="logo">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 8a3 3 0 0 0 3-3 3 3 0 0 0-3-3 3 3 0 0 0-3 3 3 3 0 0 0 3 3m0 3.54C9.64 9.35 6.5 8 3 8v11c3.5 0 6.64 1.35 9 3.54 2.36-2.19 5.5-3.54 9-3.54V8c-3.5 0-6.64 1.35-9 3.54"/></svg>
</a>
<label class="md-header__button md-icon" for="__drawer">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M3 6h18v2H3zm0 5h18v2H3zm0 5h18v2H3z"/></svg>
</label>
<div class="md-header__title" data-md-component="header-title">
<div class="md-header__ellipsis">
<div class="md-header__topic">
<span class="md-ellipsis">
KPN++
</span>
</div>
<div class="md-header__topic" data-md-component="header-topic">
<span class="md-ellipsis">
</span>
</div>
</div>
</div>
<form class="md-header__option" data-md-component="palette">
<input class="md-option" data-md-color-media="" data-md-color-scheme="slate" data-md-color-primary="indigo" data-md-color-accent="indigo" aria-hidden="true" type="radio" name="__palette" id="__palette_0">
</form>
<script>var palette=__md_get("__palette");if(palette&&palette.color){if("(prefers-color-scheme)"===palette.color.media){var media=matchMedia("(prefers-color-scheme: light)"),input=document.querySelector(media.matches?"[data-md-color-media='(prefers-color-scheme: light)']":"[data-md-color-media='(prefers-color-scheme: dark)']");palette.color.media=input.getAttribute("data-md-color-media"),palette.color.scheme=input.getAttribute("data-md-color-scheme"),palette.color.primary=input.getAttribute("data-md-color-primary"),palette.color.accent=input.getAttribute("data-md-color-accent")}for(var[key,value]of Object.entries(palette.color))document.body.setAttribute("data-md-color-"+key,value)}</script>
<label class="md-header__button md-icon" for="__search">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M9.5 3A6.5 6.5 0 0 1 16 9.5c0 1.61-.59 3.09-1.56 4.23l.27.27h.79l5 5-1.5 1.5-5-5v-.79l-.27-.27A6.52 6.52 0 0 1 9.5 16 6.5 6.5 0 0 1 3 9.5 6.5 6.5 0 0 1 9.5 3m0 2C7 5 5 7 5 9.5S7 14 9.5 14 14 12 14 9.5 12 5 9.5 5"/></svg>
</label>
<div class="md-search" data-md-component="search" role="dialog">
<label class="md-search__overlay" for="__search"></label>
<div class="md-search__inner" role="search">
<form class="md-search__form" name="search">
<input type="text" class="md-search__input" name="query" aria-label="Search" placeholder="Search" autocapitalize="off" autocorrect="off" autocomplete="off" spellcheck="false" data-md-component="search-query" required>
<label class="md-search__icon md-icon" for="__search">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M9.5 3A6.5 6.5 0 0 1 16 9.5c0 1.61-.59 3.09-1.56 4.23l.27.27h.79l5 5-1.5 1.5-5-5v-.79l-.27-.27A6.52 6.52 0 0 1 9.5 16 6.5 6.5 0 0 1 3 9.5 6.5 6.5 0 0 1 9.5 3m0 2C7 5 5 7 5 9.5S7 14 9.5 14 14 12 14 9.5 12 5 9.5 5"/></svg>
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M20 11v2H8l5.5 5.5-1.42 1.42L4.16 12l7.92-7.92L13.5 5.5 8 11z"/></svg>
</label>
<nav class="md-search__options" aria-label="Search">
<button type="reset" class="md-search__icon md-icon" title="Clear" aria-label="Clear" tabindex="-1">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M19 6.41 17.59 5 12 10.59 6.41 5 5 6.41 10.59 12 5 17.59 6.41 19 12 13.41 17.59 19 19 17.59 13.41 12z"/></svg>
</button>
</nav>
</form>
<div class="md-search__output">
<div class="md-search__scrollwrap" tabindex="0" data-md-scrollfix>
<div class="md-search-result" data-md-component="search-result">
<div class="md-search-result__meta">
Initializing search
</div>
<ol class="md-search-result__list" role="presentation"></ol>
</div>
</div>
</div>
</div>
</div>
<div class="md-header__source">
<a href="https://gitea.tourolle.paris/dtourolle/KPN" title="Go to repository" class="md-source" data-md-component="source">
<div class="md-source__icon md-icon">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512"><!--! Font Awesome Free 7.1.0 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2025 Fonticons, Inc.--><path d="M439.6 236.1 244 40.5c-5.4-5.5-12.8-8.5-20.4-8.5s-15 3-20.4 8.4L162.5 81l51.5 51.5c27.1-9.1 52.7 16.8 43.4 43.7l49.7 49.7c34.2-11.8 61.2 31 35.5 56.7-26.5 26.5-70.2-2.9-56-37.3L240.3 199v121.9c25.3 12.5 22.3 41.8 9.1 55-6.4 6.4-15.2 10.1-24.3 10.1s-17.8-3.6-24.3-10.1c-17.6-17.6-11.1-46.9 11.2-56v-123c-20.8-8.5-24.6-30.7-18.6-45L142.6 101 8.5 235.1C3 240.6 0 247.9 0 255.5s3 15 8.5 20.4l195.6 195.7c5.4 5.4 12.7 8.4 20.4 8.4s15-3 20.4-8.4l194.7-194.7c5.4-5.4 8.4-12.8 8.4-20.4s-3-15-8.4-20.4"/></svg>
</div>
<div class="md-source__repository">
dtourolle/KPN
</div>
</a>
</div>
</nav>
</header>
<div class="md-container" data-md-component="container">
<nav class="md-tabs" aria-label="Tabs" data-md-component="tabs">
<div class="md-grid">
<ul class="md-tabs__list">
<li class="md-tabs__item">
<a href="/." class="md-tabs__link">
Home
</a>
</li>
<li class="md-tabs__item">
<a href="/getting-started/" class="md-tabs__link">
Getting Started
</a>
</li>
<li class="md-tabs__item">
<a href="/nodes/" class="md-tabs__link">
Concepts
</a>
</li>
<li class="md-tabs__item">
<a href="/error-handling/" class="md-tabs__link">
Error Handling & Events
</a>
</li>
<li class="md-tabs__item">
<a href="/static-network/" class="md-tabs__link">
Advanced
</a>
</li>
<li class="md-tabs__item">
<a href="/examples/" class="md-tabs__link">
Examples
</a>
</li>
</ul>
</div>
</nav>
<main class="md-main" data-md-component="main">
<div class="md-main__inner md-grid">
<div class="md-sidebar md-sidebar--primary" data-md-component="sidebar" data-md-type="navigation" >
<div class="md-sidebar__scrollwrap">
<div class="md-sidebar__inner">
<nav class="md-nav md-nav--primary md-nav--lifted" aria-label="Navigation" data-md-level="0">
<label class="md-nav__title" for="__drawer">
<a href="/." title="KPN++" class="md-nav__button md-logo" aria-label="KPN++" data-md-component="logo">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 8a3 3 0 0 0 3-3 3 3 0 0 0-3-3 3 3 0 0 0-3 3 3 3 0 0 0 3 3m0 3.54C9.64 9.35 6.5 8 3 8v11c3.5 0 6.64 1.35 9 3.54 2.36-2.19 5.5-3.54 9-3.54V8c-3.5 0-6.64 1.35-9 3.54"/></svg>
</a>
KPN++
</label>
<div class="md-nav__source">
<a href="https://gitea.tourolle.paris/dtourolle/KPN" title="Go to repository" class="md-source" data-md-component="source">
<div class="md-source__icon md-icon">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512"><!--! Font Awesome Free 7.1.0 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2025 Fonticons, Inc.--><path d="M439.6 236.1 244 40.5c-5.4-5.5-12.8-8.5-20.4-8.5s-15 3-20.4 8.4L162.5 81l51.5 51.5c27.1-9.1 52.7 16.8 43.4 43.7l49.7 49.7c34.2-11.8 61.2 31 35.5 56.7-26.5 26.5-70.2-2.9-56-37.3L240.3 199v121.9c25.3 12.5 22.3 41.8 9.1 55-6.4 6.4-15.2 10.1-24.3 10.1s-17.8-3.6-24.3-10.1c-17.6-17.6-11.1-46.9 11.2-56v-123c-20.8-8.5-24.6-30.7-18.6-45L142.6 101 8.5 235.1C3 240.6 0 247.9 0 255.5s3 15 8.5 20.4l195.6 195.7c5.4 5.4 12.7 8.4 20.4 8.4s15-3 20.4-8.4l194.7-194.7c5.4-5.4 8.4-12.8 8.4-20.4s-3-15-8.4-20.4"/></svg>
</div>
<div class="md-source__repository">
dtourolle/KPN
</div>
</a>
</div>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="/." class="md-nav__link">
<span class="md-ellipsis">
Home
</span>
</a>
</li>
<li class="md-nav__item">
<a href="/getting-started/" class="md-nav__link">
<span class="md-ellipsis">
Getting Started
</span>
</a>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3" >
<label class="md-nav__link" for="__nav_3" id="__nav_3_label" tabindex="0">
<span class="md-ellipsis">
Concepts
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_3_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3">
<span class="md-nav__icon md-icon"></span>
Concepts
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="/nodes/" class="md-nav__link">
<span class="md-ellipsis">
Nodes
</span>
</a>
</li>
<li class="md-nav__item">
<a href="/network/" class="md-nav__link">
<span class="md-ellipsis">
Networks
</span>
</a>
</li>
<li class="md-nav__item">
<a href="/channels/" class="md-nav__link">
<span class="md-ellipsis">
Channels
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item">
<a href="/error-handling/" class="md-nav__link">
<span class="md-ellipsis">
Error Handling & Events
</span>
</a>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_5" >
<label class="md-nav__link" for="__nav_5" id="__nav_5_label" tabindex="0">
<span class="md-ellipsis">
Advanced
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_5_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_5">
<span class="md-nav__icon md-icon"></span>
Advanced
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="/static-network/" class="md-nav__link">
<span class="md-ellipsis">
Static Networks
</span>
</a>
</li>
<li class="md-nav__item">
<a href="/shared-resource/" class="md-nav__link">
<span class="md-ellipsis">
Shared Resources
</span>
</a>
</li>
<li class="md-nav__item">
<a href="/fanout/" class="md-nav__link">
<span class="md-ellipsis">
Fan-out & Routing
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item">
<a href="/examples/" class="md-nav__link">
<span class="md-ellipsis">
Examples
</span>
</a>
</li>
</ul>
</nav>
</div>
</div>
</div>
<div class="md-sidebar md-sidebar--secondary" data-md-component="sidebar" data-md-type="toc" >
<div class="md-sidebar__scrollwrap">
<div class="md-sidebar__inner">
<nav class="md-nav md-nav--secondary" aria-label="Table of contents">
</nav>
</div>
</div>
</div>
<div class="md-content" data-md-component="content">
<article class="md-content__inner md-typeset">
<h1>404 - Not found</h1>
</article>
</div>
<script>var target=document.getElementById(location.hash.slice(1));target&&target.name&&(target.checked=target.name.startsWith("__tabbed_"))</script>
</div>
<button type="button" class="md-top md-icon" data-md-component="top" hidden>
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M13 20h-2V8l-5.5 5.5-1.42-1.42L12 4.16l7.92 7.92-1.42 1.42L13 8z"/></svg>
Back to top
</button>
</main>
<footer class="md-footer">
<div class="md-footer-meta md-typeset">
<div class="md-footer-meta__inner md-grid">
<div class="md-copyright">
Made with
<a href="https://squidfunk.github.io/mkdocs-material/" target="_blank" rel="noopener">
Material for MkDocs
</a>
</div>
</div>
</div>
</footer>
</div>
<div class="md-dialog" data-md-component="dialog">
<div class="md-dialog__inner md-typeset"></div>
</div>
<script id="__config" type="application/json">{"annotate": null, "base": "/", "features": ["navigation.tabs", "navigation.sections", "navigation.top", "content.code.copy", "content.code.annotate"], "search": "/assets/javascripts/workers/search.2c215733.min.js", "tags": null, "translations": {"clipboard.copied": "Copied to clipboard", "clipboard.copy": "Copy to clipboard", "search.result.more.one": "1 more on this page", "search.result.more.other": "# more on this page", "search.result.none": "No matching documents", "search.result.one": "1 matching document", "search.result.other": "# matching documents", "search.result.placeholder": "Type to start searching", "search.result.term.missing": "Missing", "select.version": "Select version"}, "version": null}</script>
<script src="/assets/javascripts/bundle.79ae519e.min.js"></script>
</body>
</html>
CMakeLists.txt
+112
View File
@@ -0,0 +1,112 @@
cmake_minimum_required(VERSION 3.21)
project(kpnpp VERSION 0.1.0 LANGUAGES CXX)
set(CMAKE_CXX_STANDARD 20)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
set(CMAKE_CXX_EXTENSIONS OFF)
option(KPN_BUILD_TESTS "Build tests" ON)
option(KPN_BUILD_PYTHON "Build Python bindings (requires nanobind)" ON)
option(KPN_BUILD_EXAMPLES "Build examples" ON)
option(KPN_WEB_DEBUG "Enable web debug UI (cpp-httplib)" OFF)
# Sanitizer build. Empty = off. Accepts "thread", "address", "undefined",
# or a combination like "address,undefined". Applied to all kpn targets via
# the kpn_sanitizer_flags() helper below.
#
# The lock-free SPSC Channel<T> (include/kpn/channel.hpp) has hand-reasoned
# acquire/release ordering; -DKPN_SANITIZER=thread + the channel stress test
# (tests/test_channel_stress.cpp) is the dynamic half of verifying it. The
# static half is the CDSChecker model-check harness (see verify/).
set(KPN_SANITIZER "" CACHE STRING
"Build with sanitizer: thread | address | undefined | <combo> (empty = off)")
# Translate KPN_SANITIZER into compile/link flags. No-op when empty.
function(kpn_sanitizer_flags out_var)
if(KPN_SANITIZER)
set(${out_var}
-fsanitize=${KPN_SANITIZER}
-fno-omit-frame-pointer
-g
PARENT_SCOPE)
else()
set(${out_var} "" PARENT_SCOPE)
endif()
endfunction()
# ── Core library (header-only) ────────────────────────────────────────────────
add_library(kpn INTERFACE)
target_include_directories(kpn INTERFACE
$<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/include>
$<INSTALL_INTERFACE:include>
)
target_compile_features(kpn INTERFACE cxx_std_20)
# Threads required by node/channel implementation
find_package(Threads REQUIRED)
target_link_libraries(kpn INTERFACE Threads::Threads)
# ── Web debug UI (optional) ───────────────────────────────────────────────────
if(KPN_WEB_DEBUG)
include(FetchContent)
FetchContent_Declare(
cpp-httplib
GIT_REPOSITORY https://github.com/yhirose/cpp-httplib.git
GIT_TAG v0.18.0
)
FetchContent_MakeAvailable(cpp-httplib)
# httplib made available but NOT forced onto kpn interface — targets opt in
# by defining KPN_WEB_DEBUG=1 and linking httplib::httplib themselves.
# This prevents tests and other examples from pulling in the HTTP server.
endif()
# Convenience function for targets that want web debug
function(kpn_target_enable_web_debug target)
target_compile_definitions(${target} PRIVATE KPN_WEB_DEBUG=1)
target_link_libraries(${target} PRIVATE httplib::httplib)
endfunction()
# ── Tests ─────────────────────────────────────────────────────────────────────
if(KPN_BUILD_TESTS)
enable_testing()
add_subdirectory(tests)
endif()
# ── Benchmarks ────────────────────────────────────────────────────────────────
option(KPN_BUILD_BENCHMARKS "Build benchmarks" OFF)
if(KPN_BUILD_BENCHMARKS)
add_subdirectory(benchmarks)
endif()
# ── Python bindings ───────────────────────────────────────────────────────────
if(KPN_BUILD_PYTHON)
find_package(Python 3.8 COMPONENTS Interpreter Development.Module REQUIRED)
find_package(nanobind CONFIG QUIET)
if(NOT nanobind_FOUND)
# Fall back to FetchContent if nanobind not installed system-wide
include(FetchContent)
FetchContent_Declare(
nanobind
GIT_REPOSITORY https://github.com/wjakob/nanobind.git
GIT_TAG v2.12.0
)
FetchContent_MakeAvailable(nanobind)
endif()
add_subdirectory(python)
endif()
# ── Examples ──────────────────────────────────────────────────────────────────
if(KPN_BUILD_EXAMPLES)
add_subdirectory(examples)
endif()
# ── Docs (README generation) ──────────────────────────────────────────────────
find_package(Python3 QUIET COMPONENTS Interpreter)
if(Python3_FOUND)
add_custom_target(docs
COMMAND ${Python3_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/scripts/render_readme.py
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
COMMENT "Rendering README.md from README.md.in"
VERBATIM
)
endif()
Dockerfile.builder
+27
View File
@@ -0,0 +1,27 @@
# KPN++ Builder Image (CI: pipeline trigger v2)
# Pre-built image with GCC, CMake, Ninja, and Python dev headers for building and testing KPN++
# Build: docker build -f Dockerfile.builder -t gitea.tourolle.paris/dtourolle/kpnpp-builder:latest .
# Push: docker push gitea.tourolle.paris/dtourolle/kpnpp-builder:latest
FROM gcc:14
RUN apt-get update && apt-get install -y --no-install-recommends \
cmake \
ninja-build \
python3 \
python3-dev \
python3-pip \
git \
ca-certificates \
nodejs \
&& rm -rf /var/lib/apt/lists/*
# Pre-install MkDocs dependencies so the docs workflow does not need to pip
# install at runtime. --break-system-packages is required because the Debian
# base marks the environment as externally managed (PEP 668); this is safe in
# a dedicated container image.
COPY docs/requirements.txt /tmp/docs-requirements.txt
RUN pip install --no-cache-dir --break-system-packages -r /tmp/docs-requirements.txt \
&& rm /tmp/docs-requirements.txt
WORKDIR /src
LICENSE
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2026 Duncan Tourolle
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
README.md
+612
View File
@@ -0,0 +1,612 @@
# KPN++
A C++20 Kahn Process Network (KPN) library. Each node wraps a function and runs in its own thread, communicating with downstream nodes via bounded FIFO channels. Includes Python bindings via nanobind.
📖 **[Documentation](https://pages.tourolle.paris/dtourolle/kpn/)**
---
## Requirements
| Dependency | Version | Notes |
|---|---|---|
| CMake | ≥ 3.21 | |
| C++ compiler | GCC ≥ 11, Clang ≥ 13, MSVC 19.29 | C++20 required |
| Threads | system | `find_package(Threads)` |
| nanobind | ≥ 2.1 | auto-fetched if not installed; Python ≥ 3.8 |
| Catch2 | v3 | auto-fetched for tests |
| Google Test | v1.14 | auto-fetched for tests |
| OpenCV | ≥ 4 | optional; only for example 09 |
---
## Build
```bash
cmake -B build -DKPN_BUILD_PYTHON=OFF # core + tests + C++ examples
cmake --build build --parallel
ctest --test-dir build
```
Enable Python bindings (requires nanobind and Python dev headers):
```bash
cmake -B build -DKPN_BUILD_PYTHON=ON
cmake --build build --parallel
```
Disable examples:
```bash
cmake -B build -DKPN_BUILD_EXAMPLES=OFF
```
---
## Core Concepts
### Nodes
A node wraps any callable. Its input types are taken from the function's parameter list; its output types from the return type. Multi-output nodes return `std::tuple<...>`.
```cpp
#include <kpn/kpn.hpp>
using namespace kpn;
```
Source, transform, and sink — from [`examples/01_hello_pipeline/main.cpp`](examples/01_hello_pipeline/main.cpp):
```cpp
static int produce() { return 42; }
static int double_it(int x) { return x * 2; }
static void print_it(int x) { std::cout << "result: " << x << '\n'; }
```
Multi-output node returning a tuple — from [`examples/03_multi_output/main.cpp`](examples/03_multi_output/main.cpp):
```cpp
// Multi-output: returns (key, value) as a tuple — KPN++ routes each element
// to its own output port automatically.
static std::tuple<std::string, std::string> parse(std::string kv) {
auto sep = kv.find('=');
if (sep == std::string::npos) return {kv, ""};
return {kv.substr(0, sep), kv.substr(sep + 1)};
}
```
### Creating Nodes
**Index-only ports** (from [`examples/01_hello_pipeline/main.cpp`](examples/01_hello_pipeline/main.cpp)):
```cpp
auto src = make_node<produce>(5);
auto dbl = make_node<double_it>(5);
auto sink = make_node<print_it>(5);
```
**Named ports** (from [`examples/02_named_ports/main.cpp`](examples/02_named_ports/main.cpp)):
```cpp
// tokenise: no inputs, one named output "words"
auto tok = make_node<tokenise>(out<"words">{}, 4);
// count_words: named input "words", named outputs "count" and "words"
auto cnt = make_node<count_words>(in<"words">{}, out<"count", "words">{}, 4);
// report: two named inputs
auto snk = make_node<report>(in<"count", "words">{}, 4);
```
**Multi-named output source** (from [`examples/09_opencv_cellshade/main.cpp`](examples/09_opencv_cellshade/main.cpp)):
```cpp
auto src = make_node<capture>(out<"colour","grey">{}, 8);
```
Port names are NTTP `fixed_string` values — resolved entirely at compile time, zero runtime cost.
### Building a Network
`Network` is **non-owning** — declare nodes first, then register them. Nodes must outlive the network.
From [`examples/02_named_ports/main.cpp`](examples/02_named_ports/main.cpp):
```cpp
Network net;
net.add("tok", tok)
.add("cnt", cnt)
.add("snk", snk)
.connect("tok", tok.template output<"words">(), "cnt", cnt.template input<"words">())
.connect("cnt", cnt.template output<"count">(), "snk", snk.template input<"count">())
.connect("cnt", cnt.template output<"words">(), "snk", snk.template input<"words">())
.build();
net.start();
std::this_thread::sleep_for(std::chrono::milliseconds(500));
net.stop();
```
`.build()` runs cycle detection — throws `NetworkCycleError` on cycles.
> **Named port syntax in template context:** when the node variable is `auto`-deduced, use `.template output<"name">()` and `.template input<"name">()` to help the parser.
### Channel Semantics
- **Bounded FIFO**: default capacity 5, configurable per-node at construction.
- **Blocking `pop()`**: consumer blocks until data is available (KPN semantics).
- **Throwing `push()`**: throws `ChannelOverflowError` if the channel is full and accepting.
- **Silent drop on disabled channel**: after `node.stop()`, its input channels are disabled — producers that push into them have the value silently dropped. No exception, no blocking.
- **Source throttling**: source nodes (no inputs) must sleep or yield to avoid overflowing downstream FIFOs. See example 09.
### Storage Policy
Large types (`sizeof > 8` or non-trivially-copyable) are stored as `std::shared_ptr<const T>` inside the channel — no copies, shared immutable ownership. Small trivially-copyable types are stored by value.
Override the policy for a specific type (from [`examples/04_storage_policy/main.cpp`](examples/04_storage_policy/main.cpp)):
```cpp
// Override: store Tag by value despite being a struct
// (it's trivially copyable and small — this just makes the policy explicit)
template<>
struct kpn::channel_storage_policy<Tag> {
static constexpr bool by_value = true;
};
```
### Diagnostics & Error Handling
Custom diagnostics handler — fires on the watchdog interval (from [`examples/05_error_handling/main.cpp`](examples/05_error_handling/main.cpp)):
```cpp
// Custom diagnostics handler — fires on the watchdog interval.
// Print a concise one-liner rather than the full table.
net.set_diagnostics_handler([](const std::vector<NodeSnapshot>& nodes,
const std::vector<ChannelSnapshot>& channels) {
std::cout << "[diag] ";
for (auto& n : nodes)
std::cout << n.name << "=" << n.throughput_fps << "fps ";
for (auto& c : channels)
std::cout << "channel fill=" << static_cast<int>(c.fill_pct()) << "% "
<< "overflows=" << c.overflows;
std::cout << '\n';
});
```
### Shutdown
`node.stop()` / `net.stop()`:
1. Sets `accepting_ = false` on all input channels (drops in-flight pushes silently).
2. Clears any queued items from those channels.
3. Unblocks any thread blocked on `pop()` (throws `ChannelClosedError` inside `run_loop`, which exits cleanly).
4. Joins the node thread.
---
## Named Ports — Design Notes
Port names use C++20 NTTP `fixed_string`. The deduction guide is required:
```cpp
template<std::size_t N>
fixed_string(const char (&)[N]) -> fixed_string<N>;
```
`fixed_string<4>` and `fixed_string<7>` are distinct types — `input<"img">()` and `input<"sigma">()` resolve to different template instantiations at compile time. Wrong names produce a `static_assert` at the call site with a readable message.
---
## Sub-Networks
`Network` implements `INode`, so it can be nested inside a larger `Network`:
```cpp
// Inner sub-network
Network pipe;
pipe.add("pre", pre_node)
.add("enh", enh_node)
.connect("pre", pre_node.output<0>(), "enh", enh_node.input<0>())
.expose_input("img", pre_node.input<0>())
.expose_output("result", enh_node.output<0>())
.build();
// Outer network
Network top;
top.add("pipe", pipe)
.add("sink", sink_node)
.connect("pipe", pipe.output<"result">(), "sink", sink_node.input<0>())
.build();
top.start();
```
---
## Display / GUI Nodes
**Do not wrap `imshow`/`waitKey` as a KPN node.** Qt and Wayland require these to run on the main thread (the thread that owns the event loop). Instead, derive from `MainThreadNode<>` — it owns the input channels, implements `INode`, and exposes a `step()` method to call on the main thread.
`DisplayNode` from [`examples/09_opencv_cellshade/main.cpp`](examples/09_opencv_cellshade/main.cpp):
```cpp
class DisplayNode : public kpn::MainThreadNode<DisplayNode,
kpn::in<"composite", "edges">,
cv::Mat, cv::Mat> {
public:
DisplayNode() : MainThreadNode(8) {
cv::namedWindow("Cell Shade", cv::WINDOW_NORMAL);
cv::namedWindow("Edge Mask", cv::WINDOW_NORMAL);
cv::resizeWindow("Cell Shade", 1280, 720);
cv::resizeWindow("Edge Mask", 640, 360);
}
~DisplayNode() { cv::destroyAllWindows(); }
bool operator()(cv::Mat composite, cv::Mat edges) {
cv::imshow("Cell Shade", composite);
cv::Mat edges_bgr;
cv::cvtColor(edges, edges_bgr, cv::COLOR_GRAY2BGR);
cv::imshow("Edge Mask", edges_bgr);
int key = cv::waitKey(1);
if (key == 'q' || key == 27) return false;
return window_open("Cell Shade") && window_open("Edge Mask");
}
private:
static bool window_open(const char* name) {
try { return cv::getWindowProperty(name, cv::WND_PROP_VISIBLE) >= 1; }
catch (const cv::Exception&) { return false; }
}
};
```
Wire it into the network and drive it from the main thread:
```cpp
net.start();
// Main thread drives display — imshow/waitKey stay on the GUI thread.
// step() returns false when operator() returns false (q pressed / window closed).
while (disp.step())
cv::waitKey(8); // yield event loop when no frame ready
net.stop();
```
---
## OpenCV Cell-Shading Example
Real-time cell-shading pipeline from [`examples/09_opencv_cellshade/main.cpp`](examples/09_opencv_cellshade/main.cpp).
**Source node** — returns two frames (colour + grey) as a tuple, routing them to separate downstream branches:
```cpp
static std::tuple<cv::Mat, cv::Mat> capture() {
constexpr int W = 640, H = 480;
static cv::VideoCapture cap;
static bool opened = false;
if (!opened) {
opened = true;
cap.open(0, cv::CAP_V4L2);
if (cap.isOpened()) {
cap.set(cv::CAP_PROP_FRAME_WIDTH, W);
cap.set(cv::CAP_PROP_FRAME_HEIGHT, H);
} else {
std::cerr << "[capture] no webcam — using synthetic animated pattern\n";
}
}
cv::Mat frame;
if (cap.isOpened()) {
auto t0 = std::chrono::steady_clock::now();
cap >> frame;
auto elapsed = std::chrono::steady_clock::now() - t0;
if (elapsed < std::chrono::milliseconds(20))
std::this_thread::sleep_for(std::chrono::milliseconds(33) - elapsed);
if (frame.empty()) frame = cv::Mat::zeros(H, W, CV_8UC3);
} else {
static int tick = 0;
static cv::Mat grad = make_gradient(W, H);
++tick;
frame = grad.clone();
int r = 150 + (tick % 80) * 4;
cv::circle(frame, {W/2, H/2}, r, {255, 200, 0}, -1);
cv::circle(frame, {W/2, H/2}, r / 2, { 0, 128, 255}, -1);
cv::circle(frame, {W*2/5, H*2/5}, r / 3, {200, 0, 200}, -1);
std::this_thread::sleep_for(std::chrono::milliseconds(33));
}
return {frame.clone(), frame.clone()};
}
```
**Full network wiring:**
```cpp
auto src = make_node<capture> (out<"colour","grey">{}, 8);
auto gray_node = make_node<to_gray> (in<"bgr">{}, out<"gray">{}, 8);
auto edge_node = make_node<edges_fn> (in<"gray">{}, out<"edges">{}, 8);
auto quant = make_node<quantise> (in<"bgr">{}, out<"quantised">{}, 8);
auto comp = make_node<composite>(in<"edges","colour">{}, out<"result","edges">{}, 8);
// DisplayNode: two windows opened in constructor, step() drives main thread.
DisplayNode disp;
Network net;
net.add("src", src)
.add("gray", gray_node)
.add("edges", edge_node)
.add("quant", quant)
.add("comp", comp)
.add("display", disp)
.connect("src", src.template output<"colour">(), "quant", quant.template input<"bgr">())
.connect("quant", quant.template output<"quantised">(), "comp", comp.template input<"colour">())
.connect("src", src.template output<"grey">(), "gray", gray_node.template input<"bgr">())
.connect("gray", gray_node.template output<"gray">(), "edges", edge_node.template input<"gray">())
.connect("edges", edge_node.template output<"edges">(), "comp", comp.template input<"edges">())
.connect("comp", comp.template output<"result">(), "display", disp.template input<"composite">())
.connect("comp", comp.template output<"edges">(), "display", disp.template input<"edges">())
.build();
```
---
## Fan-Out (Multi-Output)
From [`examples/03_multi_output/main.cpp`](examples/03_multi_output/main.cpp) — one node fans out to two independent sinks via a tuple return:
```cpp
auto gen = make_node<generate>(out<"kv">{}, 4);
auto par = make_node<parse> (in<"kv">{}, out<"key", "value">{}, 4);
auto keys = make_node<print_key> (in<"key">{}, 4);
auto vals = make_node<print_value>(in<"value">{}, 4);
Network net;
net.add("gen", gen)
.add("par", par)
.add("keys", keys)
.add("vals", vals)
.connect("gen", gen.template output<"kv">(), "par", par.template input<"kv">())
.connect("par", par.template output<"key">(), "keys", keys.template input<"key">())
.connect("par", par.template output<"value">(), "vals", vals.template input<"value">())
.build();
net.start();
std::this_thread::sleep_for(std::chrono::milliseconds(600));
net.stop();
```
---
## Python Bindings
> Python bindings are scaffolded but not yet fully implemented. See `python/kpn_python.cpp` and `include/kpn/python/bindings.hpp`.
A `PyNetwork` is constructed from a closed list of C++ node types. The variant of all port types is derived at compile time — no runtime type registration needed.
**GIL rules (non-negotiable):**
- Acquire the GIL only for the duration of a Python callable invocation.
- Release the GIL before any blocking channel operation (`pop()`, `push()`, `net.read()`, `net.write()`).
Violating the second rule deadlocks.
---
## Examples
| Example | What it shows |
|---|---|
| `01_hello_pipeline` | Linear pipeline, index-based port wiring |
| `02_named_ports` | `in<>`/`out<>` name tags, named port access |
| `03_multi_output` | Tuple-returning node, per-element sub-port routing |
| `04_storage_policy` | `channel_storage_policy` default and specialisation |
| `05_error_handling` | `ChannelOverflowError`, `ErrorHandler` |
| `06_watchdog` | Watchdog interval, stall detection |
| `07_python_network` | PyNetwork with a pure-Python node between a C++ source and sink |
| `08_python_subport` | Drive a Python node from Python via `net.write`/`net.read` sub-port taps |
| `09_opencv_cellshade` | Real-time cell-shading on webcam/pattern; requires OpenCV ≥ 4 |
Run the cell-shading example:
```bash
./build/examples/09_opencv_cellshade
# Press 'q' or close the window to stop.
# Falls back to an animated synthetic pattern if no webcam is found.
```
---
## Performance
Measured on Linux (x86-64, `-O3 -march=native`) with `benchmarks/bench_pipeline`.
Each topology pushes N items through the graph; `overhead_us/item` strips out the
per-node compute time to isolate framework cost.
Overhead formula: `(elapsed − (N + depth − 1) × work_us) / N` removes the expected
pipeline-fill cost so the number reflects pure framework latency.
### Baseline overhead (private pools, 100 µs/node)
| Topology | items/sec | overhead µs/item |
|---|---|---|
| chain depth-1 | 9 797 | ~2 |
| chain depth-4 | 9 448 | ~4 |
| chain depth-8 | 9 078 | ~7 |
| chain depth-16 | 7 004 | ~13 ← oversubscription |
| chain depth-32 | 4 179 | ~77 ← oversubscription |
| wide fanout-1 | 9 751 | ~3 |
| wide fanout-4 | 9 668 | ~3 |
| diamond (2×2) | 9 607 | ~4 |
Chain overhead is flat at **~2–7 µs/hop** for depths within the machine's core count,
then rises once threads compete for CPU. Wide and diamond topologies add no measurable
overhead as fanout increases — all branches run in parallel.
### Scheduling modes
`Node<>` gives each node a private `ThreadPool(1)`. `PoolNode<>` lets multiple
nodes share one pool. The right choice depends on the graph shape:
| Scenario | Recommended |
|---|---|
| Work per node < 100 µs, deep chain | Private pools — lower per-hop latency |
| Work per node ≥ 100 µs, wide/diamond | Shared pool, `threads = hardware_concurrency` |
| Any graph, bounded thread count required | Shared pool, `threads ≥ max parallel nodes` |
A shared single-thread pool (`threads=1`) fully serialises the graph — throughput
divides by depth for chains and by width for fanout topologies. A shared pool with
`threads ≥ max_concurrent_nodes` matches private-pool throughput while keeping the
OS thread count bounded.
### vs. TBB flow graph
Benchmarked against `tbb::flow::function_node<int,int>` (serial concurrency) with
`tbb::flow::broadcast_node<int>` for fanout. Run with `cmake -DKPN_BUILD_BENCHMARKS=ON`
— TBB benchmarks are included automatically when `find_package(TBB)` succeeds.
**Channel implementation:** lock-free SPSC ring buffer with `std::atomic::wait/notify_one`
(C++20 portable futex) plus a configurable spin-before-sleep window (default ~4 µs).
Large types are stored as `shared_ptr<const T>` — fanout copies reference counts,
not data.
Overhead µs/item at **work_us = 10** (framework overhead dominates):
| Topology | KPN++ | TBB |
|---|---|---|
| chain depth-1 | 1.7 | **1.4** |
| chain depth-4 | 2.5 | **2.2** |
| chain depth-8 | **3.0** | 3.6 |
| chain depth-16 | **9.3** | 13.0 |
| chain depth-32 | 23.2 | **14.2** |
| wide fanout-4 | 2.5 | **1.4** |
| diamond (2×2) | 3.4 | **1.9** |
Overhead µs/item at **work_us = 100** (moderate compute, KPN wins):
| Topology | KPN++ | TBB |
|---|---|---|
| chain depth-1 | **2.1** | 3.5 |
| chain depth-4 | **4.3** | 5.2 |
| chain depth-8 | **6.7** | 8.5 |
| chain depth-16 | **12.8** | 17.4 |
| chain depth-32 | **77** | 81 |
| wide fanout-4 | 3.4 | **1.9** |
| diamond (2×2) | **4.1** | 6.1 |
KPN++ pools beat TBB for every chain and diamond topology at 100 µs/node, and
match TBB within ~20% at 10 µs/node for shallow chains. TBB retains an edge on wide
fanout (serial dispatch loop vs. work-stealing pool) and at extreme oversubscription
depths (chain-32 at 10 µs). The remaining gap at light work is the cost of
`atomic::wait` vs. TBB's continuously-spinning worker threads.
### vs. TBB — API
The function signature is the node. KPN infers input and output types automatically;
there is no graph object to manage.
**Single-output node:**
```cpp
// KPN — 1 line
int scale(int x) { return x * 2; }
// TBB — must state types, concurrency policy, and carry a graph reference
tbb::flow::function_node<int,int> n(g, tbb::flow::serial, [](int x){ return x*2; });
```
**Multi-output node:**
```cpp
// KPN — return a tuple
std::tuple<cv::Mat,cv::Mat> split(cv::Mat f) { return {f, f}; }
// TBB — multifunction_node + explicit try_put per port
tbb::flow::multifunction_node<cv::Mat, std::tuple<cv::Mat,cv::Mat>> n(
g, tbb::flow::serial,
[](cv::Mat f, auto& ports) {
std::get<0>(ports).try_put(f);
std::get<1>(ports).try_put(f);
});
```
**Named ports** — compile-time checked, zero runtime cost, not available in TBB:
```cpp
auto node = make_node<split>(in<"frame">{}, out<"colour","grey">{}, 5);
net.connect("cam", cam.output<"frame">(), "split", node.input<"frame">());
// ^^^^^^^ typo → compile error
```
| | KPN | TBB |
|---|---|---|
| Node definition | plain function | `function_node<In,Out>` + explicit types |
| Multi-output | `return std::tuple<A,B>` | `multifunction_node` + `try_put` × N |
| Named ports | `in<"name">` / `out<"name">` compile-time | none |
| Graph lifetime | none | `graph g` must outlive all nodes |
| Shutdown | `net.stop()` | `g.wait_for_all()` + manual |
| Python bindings | designed-in | none |
Build the benchmarks with:
```bash
cmake -B build -DKPN_BUILD_BENCHMARKS=ON
cmake --build build --target bench_pipeline
./build/benchmarks/bench_pipeline | tee results.csv
```
---
## Project Structure
```
include/kpn/
fixed_string.hpp — NTTP string, in<>/out<> tags, index_of
traits.hpp — function_traits, normalised_return_t, output_count_v
channel.hpp — Channel<T>, channel_storage_policy, exceptions
port.hpp — InputPort<N,I>, OutputPort<N,I>
node.hpp — Node<Func,in<...>,out<...>>, make_node, INode
network.hpp — Network (builder, cycle detection, watchdog)
variant_node.hpp — VariantNode, PythonConverter<T>, unique_types (Python layer)
python/
bindings.hpp — nanobind helpers, GIL rule documentation
kpn.hpp — umbrella header
src/
network.cpp — non-template Network implementation
tests/
test_fixed_string.cpp
test_traits.cpp
test_channel.cpp
test_node.cpp
test_network.cpp
python/
kpn_python.cpp — nanobind module entry point
examples/
01_hello_pipeline/ … 09_opencv_cellshade/
scripts/
render_readme.py — regenerates README.md from README.md.in
```
---
## Contributing
Contributions are welcome. This project is hosted on a self-hosted Gitea
instance that accepts sign-in and registration with a GitHub account, so you
can log in with your existing GitHub identity to open issues and pull requests.
If you change any code that appears in a README snippet, edit `README.md.in`
(the template) rather than `README.md` directly, then regenerate:
```bash
cmake --build build --target readme # or: python scripts/render_readme.py
```
---
## Acknowledgments
AI tooling was used heavily throughout the development of this project,
including the design, implementation, tests, and documentation. All output
has been reviewed, but please keep this in mind when reading or building on the
code.
---
## License
Released under the [MIT License](LICENSE). Copyright (c) 2026 Duncan Tourolle.
README.md.in
+434
View File
@@ -0,0 +1,434 @@
# KPN++
A C++20 Kahn Process Network (KPN) library. Each node wraps a function and runs in its own thread, communicating with downstream nodes via bounded FIFO channels. Includes Python bindings via nanobind.
📖 **[Documentation](https://pages.tourolle.paris/dtourolle/kpn/)**
---
## Requirements
| Dependency | Version | Notes |
|---|---|---|
| CMake | ≥ 3.21 | |
| C++ compiler | GCC ≥ 11, Clang ≥ 13, MSVC 19.29 | C++20 required |
| Threads | system | `find_package(Threads)` |
| nanobind | ≥ 2.1 | auto-fetched if not installed; Python ≥ 3.8 |
| Catch2 | v3 | auto-fetched for tests |
| Google Test | v1.14 | auto-fetched for tests |
| OpenCV | ≥ 4 | optional; only for example 09 |
---
## Build
```bash
cmake -B build -DKPN_BUILD_PYTHON=OFF # core + tests + C++ examples
cmake --build build --parallel
ctest --test-dir build
```
Enable Python bindings (requires nanobind and Python dev headers):
```bash
cmake -B build -DKPN_BUILD_PYTHON=ON
cmake --build build --parallel
```
Disable examples:
```bash
cmake -B build -DKPN_BUILD_EXAMPLES=OFF
```
---
## Core Concepts
### Nodes
A node wraps any callable. Its input types are taken from the function's parameter list; its output types from the return type. Multi-output nodes return `std::tuple<...>`.
```cpp
#include <kpn/kpn.hpp>
using namespace kpn;
```
Source, transform, and sink — from [`examples/01_hello_pipeline/main.cpp`](examples/01_hello_pipeline/main.cpp):
<!-- @snippet examples/01_hello_pipeline/main.cpp basic_node_fns -->
Multi-output node returning a tuple — from [`examples/03_multi_output/main.cpp`](examples/03_multi_output/main.cpp):
<!-- @snippet examples/03_multi_output/main.cpp multi_output_fn -->
### Creating Nodes
**Index-only ports** (from [`examples/01_hello_pipeline/main.cpp`](examples/01_hello_pipeline/main.cpp)):
<!-- @snippet examples/01_hello_pipeline/main.cpp index_only_nodes -->
**Named ports** (from [`examples/02_named_ports/main.cpp`](examples/02_named_ports/main.cpp)):
<!-- @snippet examples/02_named_ports/main.cpp named_port_creation -->
**Multi-named output source** (from [`examples/09_opencv_cellshade/main.cpp`](examples/09_opencv_cellshade/main.cpp)):
```cpp
auto src = make_node<capture>(out<"colour","grey">{}, 8);
```
Port names are NTTP `fixed_string` values — resolved entirely at compile time, zero runtime cost.
### Building a Network
`Network` is **non-owning** — declare nodes first, then register them. Nodes must outlive the network.
From [`examples/02_named_ports/main.cpp`](examples/02_named_ports/main.cpp):
<!-- @snippet examples/02_named_ports/main.cpp named_port_network -->
`.build()` runs cycle detection — throws `NetworkCycleError` on cycles.
> **Named port syntax in template context:** when the node variable is `auto`-deduced, use `.template output<"name">()` and `.template input<"name">()` to help the parser.
### Channel Semantics
- **Bounded FIFO**: default capacity 5, configurable per-node at construction.
- **Blocking `pop()`**: consumer blocks until data is available (KPN semantics).
- **Throwing `push()`**: throws `ChannelOverflowError` if the channel is full and accepting.
- **Silent drop on disabled channel**: after `node.stop()`, its input channels are disabled — producers that push into them have the value silently dropped. No exception, no blocking.
- **Source throttling**: source nodes (no inputs) must sleep or yield to avoid overflowing downstream FIFOs. See example 09.
### Storage Policy
Large types (`sizeof > 8` or non-trivially-copyable) are stored as `std::shared_ptr<const T>` inside the channel — no copies, shared immutable ownership. Small trivially-copyable types are stored by value.
Override the policy for a specific type (from [`examples/04_storage_policy/main.cpp`](examples/04_storage_policy/main.cpp)):
<!-- @snippet examples/04_storage_policy/main.cpp storage_policy_spec -->
### Diagnostics & Error Handling
Custom diagnostics handler — fires on the watchdog interval (from [`examples/05_error_handling/main.cpp`](examples/05_error_handling/main.cpp)):
<!-- @snippet examples/05_error_handling/main.cpp diagnostics_handler -->
### Shutdown
`node.stop()` / `net.stop()`:
1. Sets `accepting_ = false` on all input channels (drops in-flight pushes silently).
2. Clears any queued items from those channels.
3. Unblocks any thread blocked on `pop()` (throws `ChannelClosedError` inside `run_loop`, which exits cleanly).
4. Joins the node thread.
---
## Named Ports — Design Notes
Port names use C++20 NTTP `fixed_string`. The deduction guide is required:
```cpp
template<std::size_t N>
fixed_string(const char (&)[N]) -> fixed_string<N>;
```
`fixed_string<4>` and `fixed_string<7>` are distinct types — `input<"img">()` and `input<"sigma">()` resolve to different template instantiations at compile time. Wrong names produce a `static_assert` at the call site with a readable message.
---
## Sub-Networks
`Network` implements `INode`, so it can be nested inside a larger `Network`:
```cpp
// Inner sub-network
Network pipe;
pipe.add("pre", pre_node)
.add("enh", enh_node)
.connect("pre", pre_node.output<0>(), "enh", enh_node.input<0>())
.expose_input("img", pre_node.input<0>())
.expose_output("result", enh_node.output<0>())
.build();
// Outer network
Network top;
top.add("pipe", pipe)
.add("sink", sink_node)
.connect("pipe", pipe.output<"result">(), "sink", sink_node.input<0>())
.build();
top.start();
```
---
## Display / GUI Nodes
**Do not wrap `imshow`/`waitKey` as a KPN node.** Qt and Wayland require these to run on the main thread (the thread that owns the event loop). Instead, derive from `MainThreadNode<>` — it owns the input channels, implements `INode`, and exposes a `step()` method to call on the main thread.
`DisplayNode` from [`examples/09_opencv_cellshade/main.cpp`](examples/09_opencv_cellshade/main.cpp):
<!-- @snippet examples/09_opencv_cellshade/main.cpp display_node -->
Wire it into the network and drive it from the main thread:
<!-- @snippet examples/09_opencv_cellshade/main.cpp main_thread_step -->
---
## OpenCV Cell-Shading Example
Real-time cell-shading pipeline from [`examples/09_opencv_cellshade/main.cpp`](examples/09_opencv_cellshade/main.cpp).
**Source node** — returns two frames (colour + grey) as a tuple, routing them to separate downstream branches:
<!-- @snippet examples/09_opencv_cellshade/main.cpp capture_fn -->
**Full network wiring:**
<!-- @snippet examples/09_opencv_cellshade/main.cpp opencv_network -->
---
## Fan-Out (Multi-Output)
From [`examples/03_multi_output/main.cpp`](examples/03_multi_output/main.cpp) — one node fans out to two independent sinks via a tuple return:
<!-- @snippet examples/03_multi_output/main.cpp fanout_network -->
---
## Python Bindings
> Python bindings are scaffolded but not yet fully implemented. See `python/kpn_python.cpp` and `include/kpn/python/bindings.hpp`.
A `PyNetwork` is constructed from a closed list of C++ node types. The variant of all port types is derived at compile time — no runtime type registration needed.
**GIL rules (non-negotiable):**
- Acquire the GIL only for the duration of a Python callable invocation.
- Release the GIL before any blocking channel operation (`pop()`, `push()`, `net.read()`, `net.write()`).
Violating the second rule deadlocks.
---
## Examples
| Example | What it shows |
|---|---|
| `01_hello_pipeline` | Linear pipeline, index-based port wiring |
| `02_named_ports` | `in<>`/`out<>` name tags, named port access |
| `03_multi_output` | Tuple-returning node, per-element sub-port routing |
| `04_storage_policy` | `channel_storage_policy` default and specialisation |
| `05_error_handling` | `ChannelOverflowError`, `ErrorHandler` |
| `06_watchdog` | Watchdog interval, stall detection |
| `07_python_network` | PyNetwork with a pure-Python node between a C++ source and sink |
| `08_python_subport` | Drive a Python node from Python via `net.write`/`net.read` sub-port taps |
| `09_opencv_cellshade` | Real-time cell-shading on webcam/pattern; requires OpenCV ≥ 4 |
Run the cell-shading example:
```bash
./build/examples/09_opencv_cellshade
# Press 'q' or close the window to stop.
# Falls back to an animated synthetic pattern if no webcam is found.
```
---
## Performance
Measured on Linux (x86-64, `-O3 -march=native`) with `benchmarks/bench_pipeline`.
Each topology pushes N items through the graph; `overhead_us/item` strips out the
per-node compute time to isolate framework cost.
Overhead formula: `(elapsed − (N + depth − 1) × work_us) / N` removes the expected
pipeline-fill cost so the number reflects pure framework latency.
### Baseline overhead (private pools, 100 µs/node)
| Topology | items/sec | overhead µs/item |
|---|---|---|
| chain depth-1 | 9 797 | ~2 |
| chain depth-4 | 9 448 | ~4 |
| chain depth-8 | 9 078 | ~7 |
| chain depth-16 | 7 004 | ~13 ← oversubscription |
| chain depth-32 | 4 179 | ~77 ← oversubscription |
| wide fanout-1 | 9 751 | ~3 |
| wide fanout-4 | 9 668 | ~3 |
| diamond (2×2) | 9 607 | ~4 |
Chain overhead is flat at **~2–7 µs/hop** for depths within the machine's core count,
then rises once threads compete for CPU. Wide and diamond topologies add no measurable
overhead as fanout increases — all branches run in parallel.
### Scheduling modes
`Node<>` gives each node a private `ThreadPool(1)`. `PoolNode<>` lets multiple
nodes share one pool. The right choice depends on the graph shape:
| Scenario | Recommended |
|---|---|
| Work per node < 100 µs, deep chain | Private pools — lower per-hop latency |
| Work per node ≥ 100 µs, wide/diamond | Shared pool, `threads = hardware_concurrency` |
| Any graph, bounded thread count required | Shared pool, `threads ≥ max parallel nodes` |
A shared single-thread pool (`threads=1`) fully serialises the graph — throughput
divides by depth for chains and by width for fanout topologies. A shared pool with
`threads ≥ max_concurrent_nodes` matches private-pool throughput while keeping the
OS thread count bounded.
### vs. TBB flow graph
Benchmarked against `tbb::flow::function_node<int,int>` (serial concurrency) with
`tbb::flow::broadcast_node<int>` for fanout. Run with `cmake -DKPN_BUILD_BENCHMARKS=ON`
— TBB benchmarks are included automatically when `find_package(TBB)` succeeds.
**Channel implementation:** lock-free SPSC ring buffer with `std::atomic::wait/notify_one`
(C++20 portable futex) plus a configurable spin-before-sleep window (default ~4 µs).
Large types are stored as `shared_ptr<const T>` — fanout copies reference counts,
not data.
Overhead µs/item at **work_us = 10** (framework overhead dominates):
| Topology | KPN++ | TBB |
|---|---|---|
| chain depth-1 | 1.7 | **1.4** |
| chain depth-4 | 2.5 | **2.2** |
| chain depth-8 | **3.0** | 3.6 |
| chain depth-16 | **9.3** | 13.0 |
| chain depth-32 | 23.2 | **14.2** |
| wide fanout-4 | 2.5 | **1.4** |
| diamond (2×2) | 3.4 | **1.9** |
Overhead µs/item at **work_us = 100** (moderate compute, KPN wins):
| Topology | KPN++ | TBB |
|---|---|---|
| chain depth-1 | **2.1** | 3.5 |
| chain depth-4 | **4.3** | 5.2 |
| chain depth-8 | **6.7** | 8.5 |
| chain depth-16 | **12.8** | 17.4 |
| chain depth-32 | **77** | 81 |
| wide fanout-4 | 3.4 | **1.9** |
| diamond (2×2) | **4.1** | 6.1 |
KPN++ pools beat TBB for every chain and diamond topology at 100 µs/node, and
match TBB within ~20% at 10 µs/node for shallow chains. TBB retains an edge on wide
fanout (serial dispatch loop vs. work-stealing pool) and at extreme oversubscription
depths (chain-32 at 10 µs). The remaining gap at light work is the cost of
`atomic::wait` vs. TBB's continuously-spinning worker threads.
### vs. TBB — API
The function signature is the node. KPN infers input and output types automatically;
there is no graph object to manage.
**Single-output node:**
```cpp
// KPN — 1 line
int scale(int x) { return x * 2; }
// TBB — must state types, concurrency policy, and carry a graph reference
tbb::flow::function_node<int,int> n(g, tbb::flow::serial, [](int x){ return x*2; });
```
**Multi-output node:**
```cpp
// KPN — return a tuple
std::tuple<cv::Mat,cv::Mat> split(cv::Mat f) { return {f, f}; }
// TBB — multifunction_node + explicit try_put per port
tbb::flow::multifunction_node<cv::Mat, std::tuple<cv::Mat,cv::Mat>> n(
g, tbb::flow::serial,
[](cv::Mat f, auto& ports) {
std::get<0>(ports).try_put(f);
std::get<1>(ports).try_put(f);
});
```
**Named ports** — compile-time checked, zero runtime cost, not available in TBB:
```cpp
auto node = make_node<split>(in<"frame">{}, out<"colour","grey">{}, 5);
net.connect("cam", cam.output<"frame">(), "split", node.input<"frame">());
// ^^^^^^^ typo → compile error
```
| | KPN | TBB |
|---|---|---|
| Node definition | plain function | `function_node<In,Out>` + explicit types |
| Multi-output | `return std::tuple<A,B>` | `multifunction_node` + `try_put` × N |
| Named ports | `in<"name">` / `out<"name">` compile-time | none |
| Graph lifetime | none | `graph g` must outlive all nodes |
| Shutdown | `net.stop()` | `g.wait_for_all()` + manual |
| Python bindings | designed-in | none |
Build the benchmarks with:
```bash
cmake -B build -DKPN_BUILD_BENCHMARKS=ON
cmake --build build --target bench_pipeline
./build/benchmarks/bench_pipeline | tee results.csv
```
---
## Project Structure
```
include/kpn/
fixed_string.hpp — NTTP string, in<>/out<> tags, index_of
traits.hpp — function_traits, normalised_return_t, output_count_v
channel.hpp — Channel<T>, channel_storage_policy, exceptions
port.hpp — InputPort<N,I>, OutputPort<N,I>
node.hpp — Node<Func,in<...>,out<...>>, make_node, INode
network.hpp — Network (builder, cycle detection, watchdog)
variant_node.hpp — VariantNode, PythonConverter<T>, unique_types (Python layer)
python/
bindings.hpp — nanobind helpers, GIL rule documentation
kpn.hpp — umbrella header
src/
network.cpp — non-template Network implementation
tests/
test_fixed_string.cpp
test_traits.cpp
test_channel.cpp
test_node.cpp
test_network.cpp
python/
kpn_python.cpp — nanobind module entry point
examples/
01_hello_pipeline/ … 09_opencv_cellshade/
scripts/
render_readme.py — regenerates README.md from README.md.in
```
---
## Contributing
Contributions are welcome. This project is hosted on a self-hosted Gitea
instance that accepts sign-in and registration with a GitHub account, so you
can log in with your existing GitHub identity to open issues and pull requests.
If you change any code that appears in a README snippet, edit `README.md.in`
(the template) rather than `README.md` directly, then regenerate:
```bash
cmake --build build --target readme # or: python scripts/render_readme.py
```
---
## Acknowledgments
AI tooling was used heavily throughout the development of this project,
including the design, implementation, tests, and documentation. All output
has been reviewed, but please keep this in mind when reading or building on the
code.
---
## License
Released under the [MIT License](LICENSE). Copyright (c) 2026 Duncan Tourolle.
SPEC.md
+789
View File
@@ -0,0 +1,789 @@
# KPN++ — Kahn Process Network Library Specification
## Overview
A header-only C++20 template-metaprogramming library for building Kahn Process Networks. Each
node wraps a function (or callable object); its input types are inferred from the parameter
list and its output types from the return type. Nodes communicate over bounded, lock-free
SPSC FIFO channels.
Unlike a naive "one blocking thread per node" model, KPN++ is **reactive**: a node is
scheduled onto a thread pool whenever all of its input channels have data. A node that wraps a
function with `Node<>` owns a private single-thread pool and behaves exactly like an
independent worker; multiple nodes can instead share one `ThreadPool` for bounded-thread
execution. Source nodes self-resubmit; event-driven sources (`InterruptNode`) fire on an
external trigger.
The library ships rich runtime diagnostics (per-node exec/CPU/throughput stats, per-channel
fill/bandwidth/overflow counters, pool and shared-resource utilisation), an optional in-process
web debug UI, and nanobind-based Python bindings (partially implemented).
> **Note on accuracy.** This document describes the code as it exists in `include/kpn/`. Where
> a behaviour is subtle the relevant header is named so the source remains the ground truth.
---
## Project Structure
```
kpn++/
├── CMakeLists.txt
├── include/kpn/
│ ├── fixed_string.hpp # NTTP string + in<>/out<> tags + index_of
│ ├── traits.hpp # function signature introspection, normalised_return_t, repeat_tuple
│ ├── diagnostics.hpp # NodeStats, ChannelStats, *Snapshot, IPoolProbe, IResourceProbe
│ ├── channel.hpp # lock-free SPSC ring-buffer Channel<T> + storage policy
│ ├── port.hpp # InputPort / OutputPort handles
│ ├── inode.hpp # INode interface, NodeErrorHandler, NodeEvent
│ ├── scheduler.hpp # IScheduler + work-stealing ThreadPool
│ ├── pool_node.hpp # PoolNode / PoolObjectNode (reactive, scheduler-driven)
│ ├── interrupt_node.hpp # InterruptNode (external-trigger source)
│ ├── node.hpp # Node / ObjectNode (PoolNode + private 1-thread pool) + make_node
│ ├── fanout.hpp # FanoutNode<T,N> + make_fanout
│ ├── branch.hpp # RouterNode<T,N> + FilterNode<T> + make_router / make_filter
│ ├── shared_resource.hpp # SharedResource<T> priority-arbitrated exclusive resource
│ ├── main_thread_node.hpp # MainThreadNode<> (GUI / main-thread-bound nodes)
│ ├── static_network.hpp # Edge<>, make_network(), StaticNetwork<>
│ ├── network.hpp # runtime Network builder + watchdog + diagnostics
│ ├── debug_hub.hpp # DebugHub multi-network web UI (KPN_WEB_DEBUG only)
│ ├── web_debug.hpp # single-network web debug server (KPN_WEB_DEBUG only)
│ ├── variant_node.hpp # runtime-typed nodes/channels for Python graphs
│ ├── tmp/
│ │ ├── fanout_groups.hpp # compile-time fan-out detection + edge expansion
│ │ ├── topo_sort.hpp # compile-time DFS cycle check + topological order
│ │ └── repeat_tuple.hpp # repeat_tuple_t<T,N>
│ ├── python/
│ │ ├── bindings.hpp # PyNetwork / PyNode nanobind helpers
│ │ └── auto_bind.hpp # NodeRegistry / Entry / bind_network / bind_debug
│ └── kpn.hpp # umbrella header
├── src/network.cpp
├── tests/ # Catch2 v3 + GoogleTest
├── examples/ # 01–16 (see Examples)
├── benchmarks/ # bench_pipeline (optional, KPN_BUILD_BENCHMARKS)
└── python/kpn_python.cpp # nanobind module definition
```
`kpn.hpp` is the umbrella header; including it pulls in the full C++ API (the Python layer is
included only by the binding TU).
---
## Component 0 — `fixed_string.hpp`: NTTP String + Port Tags
Named ports use C++20 non-type template parameters (NTTPs). `std::string_view` and
`const char*` are not valid NTTPs, so a `fixed_string` literal type provides `constexpr`
internal storage.
```cpp
template<std::size_t N>
struct fixed_string {
char data[N]{};
constexpr fixed_string(const char (&s)[N]) { std::copy_n(s, N, data); }
constexpr bool operator==(const fixed_string&) const = default;
constexpr std::string_view view() const { return {data, N - 1}; }
};
template<std::size_t N>
fixed_string(const char (&)[N]) -> fixed_string<N>; // deduction guide (required)
```
`fixed_string<4>` and `fixed_string<7>` are distinct types, so `input<"img">()` and
`input<"sigma">()` produce different instantiations — enabling zero-overhead compile-time
port dispatch.
Named-port lookup uses a `constexpr` `index_of` over the name pack; it returns the sentinel
`npos` on a miss so the `static_assert` fires at the `input<"img">()` **call site**, giving a
readable error at the point of use:
```cpp
inline constexpr std::size_t npos = std::size_t(-1);
template<fixed_string Name, fixed_string... Names>
constexpr std::size_t index_of(); // returns position or npos
```
### Port tags
`in<...>` and `out<...>` tag types disambiguate input vs. output name packs in the factory
API. Both are trivial empty structs; both are optional (omit to get index-only ports).
```cpp
template<fixed_string... Names> struct in {};
template<fixed_string... Names> struct out {};
```
> There is **no `latch<>` tag.** An earlier design sketched latched (most-recent-value)
> input ports; this was not implemented and the only input kind is the synchronous one.
---
## Component 1 — `traits.hpp`: Function Introspection
Extracts parameter and return types from any callable at compile time, for free functions,
function pointers, member function pointers (const and non-const), lambdas and `std::function`.
```cpp
// function_traits<F>::return_t, ::args (std::tuple<...>), ::arity
template<typename F> using return_t = ...; // return type
template<typename F> using args_t = ...; // std::tuple of parameters
template<typename F> inline constexpr std::size_t arity_v = ...;
```
The return type is normalised to a tuple so every node has a uniform output-tuple shape:
```cpp
// void → std::tuple<> (sink node, 0 outputs)
// T (non-tup) → std::tuple<T> (1 output)
// tuple<...> → tuple<...> (one output port per element)
template<typename T> using normalised_return_t = ...;
template<typename F> inline constexpr std::size_t output_count_v = ...;
```
`repeat_tuple_t<T, N>` (also surfaced via `tmp/repeat_tuple.hpp`) builds `std::tuple<T, …, T>`
with `N` repetitions — used by `FanoutNode` and `RouterNode` to describe their N identical
output ports.
---
## Component 2 — `diagnostics.hpp`: Statistics and Snapshots
Shared timing types: `clock_t = std::chrono::steady_clock`, `duration_t` is a
`double`-millisecond duration.
- **`NodeStats`** — atomic counters updated per fire: `frames_processed`, an EMA of wall-clock
exec time (`ema_exec_us`, warmup-mean for the first 5 frames then α=0.1), `max_exec_us`,
`total_blocked_us`, thread CPU time (`total_cpu_us` via `CLOCK_THREAD_CPUTIME_ID`),
`queue_wait_us` (pool queue latency), and `exec_start_us` (non-zero while executing; used by
the watchdog to detect hung nodes).
- **`ChannelStats`** — `pushes`, `bytes_pushed`, `drops`, `overflows`, `pops`, `peak_fill`.
- **Snapshots** — copyable plain structs taken by the watchdog / UI: `NodeSnapshot`,
`ChannelSnapshot` (with `fill_pct()`, `peak_pct()`, `bandwidth_mbs()`), `PoolSnapshot`,
`ResourceSnapshot`, and `NetworkSnapshot` (used by the `DebugHub`).
- **Probe interfaces** — `IPoolProbe` and `IResourceProbe` expose a `snapshot(name)` method so
pools and shared resources can be registered with a network for reporting.
### `ChannelDataSize<T>` trait
`bytes_pushed` is computed from a specialisable trait, defaulting to `sizeof(T)`. Specialise it
for heap-owning payloads to get accurate bandwidth:
```cpp
template<> struct kpn::ChannelDataSize<cv::Mat> {
static std::size_t bytes(const cv::Mat& m) { return m.total() * m.elemSize(); }
};
```
---
## Component 3 — `channel.hpp`: Lock-free Bounded FIFO + Storage Policy
### Storage policy
The type stored inside a channel is chosen by a specialisable trait. Small trivially-copyable
types are stored by value; everything else as `std::shared_ptr<const T>` so fan-out copies a
refcount, not data:
```cpp
template<typename T>
struct channel_storage_policy {
static constexpr bool by_value =
std::is_trivially_copyable_v<T> && sizeof(T) <= 8;
};
template<typename T>
using channel_storage_t = std::conditional_t<
channel_storage_policy<T>::by_value, T, std::shared_ptr<const T>>;
```
Override it to force value semantics for a custom small type. Push wraps a value in
`make_shared<const T>` when needed; pop dereferences it transparently, so a function taking
`const T&` works naturally and immutability is compiler-enforced.
### Channel — SPSC ring buffer
`Channel<T>` is a single-producer/single-consumer ring buffer (capacity rounded up to a power
of two). It uses C++20 `std::atomic::wait/notify_one` (portable futex) with a configurable
**spin-before-sleep** window so the common case never touches the kernel.
```cpp
template<typename T>
class Channel {
public:
using storage_type = channel_storage_t<T>;
explicit Channel(std::size_t capacity = 5, std::size_t spin_count = 200);
void push(T value); // drops if disabled; throws ChannelOverflowError if full
bool push_sentinel(T value); // out-of-band, non-blocking must-deliver token (EOF)
T pop(); // blocks (spin then futex); throws ChannelClosedError if disabled+empty
bool try_pop(T& out, std::chrono::milliseconds timeout); // polling (watchdog/display)
bool try_pop_now(T& out); // immediate, non-blocking
void enable(); // accept pushes
void disable(); // stop accepting + unblock any waiting pop()
void set_push_callback(std::function<void()>); // empty→non-empty notification
std::size_t size() const; // ring occupancy (excludes any pending sentinel)
std::size_t approx_size() const; // size() + 1 if a sentinel is pending (readiness checks)
std::size_t capacity() const;
bool is_accepting() const;
const ChannelStats& stats() const;
ChannelSnapshot snapshot(const std::string& name) const;
};
class ChannelOverflowError : public std::runtime_error { /* capacity + optional context */ };
class ChannelClosedError : public std::runtime_error {};
```
`head_` and `tail_`/`wake_` live on separate cache lines (`alignas(64)`) to avoid false
sharing between producer and consumer. `spin_hint()` issues a `pause`/`yield` instruction (or a
compiler fence on other ISAs).
### The `push_callback` — how reactivity works
`set_push_callback` registers a callback fired when a channel transitions empty→non-empty. A
consuming `PoolNode` installs this on each of its input channels; when an input becomes ready it
re-evaluates whether **all** inputs have data and, if so, submits itself to the scheduler. This
is the mechanism that replaces a dedicated blocking thread per node.
### The out-of-band EOF sentinel — `push_sentinel`
`push_sentinel(T value)` delivers a **must-deliver control token** (a graceful-EOF marker) that
cannot be dropped by backpressure. The value is stored in a dedicated slot **outside** the ring,
so it consumes no capacity, never throws `ChannelOverflowError`, and never blocks the producer.
This matters because a node's worker cannot afford to block on a downstream push: parking that
thread would stop it draining its own input, cascading into a hold-and-wait deadlock under
backpressure. `push_sentinel` sets a published flag (`has_eof_`) and returns immediately, keeping
the worker free to keep popping.
Ordering is preserved: the consumer's `pop()` / `try_pop_now()` drain the ring **first** and only
surface the sentinel once the ring is observed empty — so EOF always arrives after every value
pushed before it. `approx_size()` (used by node readiness checks) counts a pending sentinel as one
consumable item, so a channel carrying *only* a sentinel still schedules its consumer's next fire
and the token is never stranded. Same SPSC contract as `push()` (sole producer); returns `false`
if the channel is already disabled (teardown in progress → the token is moot).
### Backpressure and shutdown — `accepting_` flag
Each channel carries `std::atomic<bool> accepting_` (default `true`). It is the primary shutdown
mechanism; the only additional signal is the out-of-band EOF sentinel above, used for *graceful*
drain rather than an abrupt close.
- **`push()`** on a disabled channel silently drops the value (recorded as a `drop`). On a
full accepting channel it throws `ChannelOverflowError` (a sizing error).
- **`pop()`** blocks while empty and accepting; `disable()` wakes it and it throws
`ChannelClosedError`.
The **consumer node** owns its input channels and flips the flag: `start()` calls `enable()`,
`stop()` calls `disable()`. Producers never touch it.
### Ownership
Input channels are owned by their **consumer node** (held as `shared_ptr<Channel<T>>`). A
producer node holds a non-owning raw `Channel<T>*` to push into. `Network`/`StaticNetwork` are
otherwise non-owning of user nodes — see Components 8–9.
---
## Component 4 — `inode.hpp`: The Node Interface
Every node implements `INode`:
```cpp
struct INode {
virtual ~INode() = default;
virtual void start() = 0;
virtual void stop() = 0;
virtual bool running() const = 0;
virtual const NodeStats& stats() const = 0;
virtual NodeSnapshot node_snapshot(const std::string& name, double elapsed_s) const = 0;
virtual void set_name(std::string) = 0;
virtual void set_network_overflow_callback(NodeEventCallback) {} // network-injected
virtual void set_network_closed_callback(NodeEventCallback) {}
virtual void halt() { stop(); } // immediate, discard in-flight work
virtual void shutdown() { stop(); } // graceful topo-ordered drain (overridden by networks)
};
```
Supporting types:
```cpp
// Per-node error policy: return true to skip the failed fire and keep running,
// false to stop the node (and signal closed downstream).
using NodeErrorHandler = std::function<bool(std::string_view node_name, std::exception_ptr)>;
using NodeEventCallback = std::function<void(std::chrono::steady_clock::time_point)>;
enum class NodeEvent { Overflow, Closed };
```
---
## Component 5 — `scheduler.hpp`: Thread Pool
```cpp
struct IScheduler {
virtual void submit(std::function<void()> task, float priority = 0.5f) = 0;
virtual void start() = 0;
virtual void stop() = 0; // join workers, discard pending tasks
virtual void drain() = 0; // block until in-flight tasks complete (workers keep running)
};
```
`ThreadPool` is a **work-stealing** pool implementing both `IScheduler` and `IPoolProbe`. Each
worker owns a priority queue (max-heap by `priority`, FIFO within equal priority via a sequence
counter). `submit()` distributes round-robin; idle workers steal from the most-loaded peer
using `try_lock`, then sleep on a shared condition variable. The submit/notify path takes the CV
mutex around `notify` to close the lost-wakeup window; `drain()` waits on a separate counter of
in-flight tasks. `priority` lets a hot node (full input, empty output) be scheduled ahead of
others — see `PoolNode::compute_priority`.
---
## Component 6 — Node Types
All processing nodes share the same shape: typed input channels they own, raw output-channel
pointers set at wiring time, `args_tuple` / `return_tuple` aliases used by the connect-time
type check, and `static constexpr` `label()` / `unique_tag` / `input_count` / `output_count`.
### `PoolNode` / `PoolObjectNode` — reactive, scheduler-driven (`pool_node.hpp`)
The core node. Instead of a blocked thread, it submits a `fire_once()` to a shared
`IScheduler` whenever all inputs are ready; `queued_` ensures at most one `fire_once()` is
in flight. `fire_once()` pops every input (`try_pop_now`), runs the function, pushes each
normalised output, records stats, then resubmits if inputs remain ready. Source nodes
(`input_count == 0`) self-submit on `start()` and after each fire.
```cpp
template<auto Func,
typename InputTag = in<>,
typename OutputTag = out<>,
fixed_string Label = "",
std::size_t UniqueTag = 0>
class PoolNode : public INode { ... };
auto n = make_pool_node<func>(scheduler, fifo_capacity); // index ports
auto n = make_pool_node<func, "label", 0>(scheduler, in<"a">{}, out<"b">{}, cap);
```
`PoolObjectNode<Obj, …>` is the same for a stateful callable object (introspected via
`&Obj::operator()`); the object must outlive the node.
Per-node configuration: `set_error_handler(NodeErrorHandler)`, `set_overflow_callback`,
`set_closed_callback`, `set_max_exec_time`. Inside `fire_once()`:
`ChannelOverflowError` fires the overflow callbacks; `ChannelClosedError` (or an error handler
returning `false`) fires the closed callbacks and self-stops; any other exception consults the
error handler.
**Name-count contract** — a `static_assert` requires that the number of input names is `0` or
equals arity (same for outputs):
```cpp
static_assert(sizeof...(InNames) == 0 || sizeof...(InNames) == input_count,
"make_pool_node: number of input names must match function arity, or provide none");
```
### `Node` / `ObjectNode` — convenience wrappers (`node.hpp`)
`Node<>` privately owns a `ThreadPool(1)` and derives from `PoolNode<>` with the **same**
template signature, so each `Node` is a self-contained worker with no external scheduler. Its
`start()`/`stop()` start and stop the private pool around the base. This keeps the simple API —
`make_node<func>(5)` — while routing all execution through the one `fire_once()` code path.
```cpp
template<auto Func, typename InputTag = in<>, typename OutputTag = out<>,
fixed_string Label = "", std::size_t UniqueTag = 0>
class Node : public PoolNode<...> { ... };
auto src = make_node<produce>(5);
auto dbl = make_node<double_it, "dbl">(5);
auto cnt = make_node<count_words>(in<"words">{}, out<"count","words">{}, 4);
```
The `Label` NTTP gives a human-readable name for diagnostics; `UniqueTag` is a collision-breaker
required when the **same function** is used as two distinct vertices in a `StaticNetwork` (two
`make_node<blur>` would otherwise be the same type). Both default so existing code is unaffected.
To share one pool across many nodes for bounded-thread execution, use `make_pool_node` directly.
### `InterruptNode` — external-trigger source (`interrupt_node.hpp`)
A zero-input source driven by an external event (camera frame, timer, socket) instead of
self-resubmission. `get_trigger()` returns a thread-safe callable to hand to the event source;
each call increments a `pending_` counter and submits `fire_once()` on the 0→1 transition,
guaranteeing one execution per trigger even under bursts. It does not busy-loop.
```cpp
auto cam = make_interrupt_node<grab_frame>(scheduler, out<"frame">{});
camera_sdk.on_frame_ready(cam.get_trigger());
```
### `FanoutNode<T, N>` — explicit fan-out (`fanout.hpp`)
Reads one item and pushes a copy to each of N outputs (per-output overflow drops
independently). Runs on its own `std::jthread` blocking on `pop()`. Used directly in a runtime
`Network` via `make_fanout<T,N>`, and auto-inserted by `make_network()` for `StaticNetwork`.
### `RouterNode<T, N>` / `FilterNode<T>` — branching (`branch.hpp`)
Both run on a dedicated `jthread`. `RouterNode` pushes each item to exactly **one** of N
outputs chosen by a `selector(item) -> size_t` (out-of-range index drops). `FilterNode` forwards
an item only when `pred(item)` is true. Factories: `make_router<T,N>(sel)`, `make_filter<T>(pred)`.
### `MainThreadNode<Derived, in<…>, Args…>` — GUI / main-thread nodes (`main_thread_node.hpp`)
For work that *must* run on the thread owning a GUI event loop (OpenCV `imshow`/`waitKey` on
Wayland/Qt). It owns input channels and is registered as a normal `INode` (appears in
diagnostics) but spawns **no** thread. The application drives it by calling `step()` in a loop on
the main thread: `step()` does a zero-timeout `try_pop` on every input, and when all are ready
invokes the derived `operator()(Args…)` (returning `false` to stop). CRTP; the derived class
supplies the operator.
---
## Component 7 — `shared_resource.hpp`: Priority-arbitrated Exclusive Resource
`SharedResource<T>` wraps a singleton-like resource (an ONNX session, a CUDA stream) shared by
nodes across one or more networks, and arbitrates access with a **priority + aging** waiter
queue. Priority is re-evaluated at every release (so it reflects current queue state), and each
waiter's effective score grows with wait time (`kAgingPerSecond`) to prevent starvation.
```cpp
SharedResource<OrtSession> res(session_args...);
// inside a node functor:
auto guard = res.acquire_balanced(in_channel, out_channel); // RAII; releases on scope exit
guard->Run(...);
```
`acquire_balanced(in, out)` scores a waiter by `input_fill × output_headroom` — a node with a
full input queue and empty output is most urgent. `acquire(fn)` takes any `()->float` priority;
`acquire()` treats all waiters equally. Implements `IResourceProbe` so it shows up in
diagnostics and the debug hub. The factory is `make_shared_resource<T>(args…)`.
---
## Component 8 — `network.hpp`: Runtime Graph Builder + Watchdog
`Network` is **non-owning** (nodes outlive it; `add()` stores `INode*`). A builder collects the
full topology before `build()`, enabling cycle detection and topological ordering.
```cpp
class Network : public INode {
public:
template<typename NodeT> Network& add(std::string name, NodeT& node);
template<typename SrcNode, std::size_t SrcIdx, typename DstNode, std::size_t DstIdx>
Network& connect(const std::string& src, OutputPort<SrcNode, SrcIdx>,
const std::string& dst, InputPort<DstNode, DstIdx>);
Network& expose_input (std::string boundary_name, InputPort<NodeT, Idx>); // sub-network port
Network& expose_output(std::string boundary_name, OutputPort<NodeT, Idx>);
Network& build(); // DFS cycle check (throws NetworkCycleError) + topo sort
void start() override; // start nodes in topo order; launch watchdog (+ web UI)
void stop() override; // == halt()
void halt() override; // immediate: stop nodes in reverse topo order
void shutdown() override; // graceful: stop source layers, drain channels, descend
void set_watchdog_interval(std::chrono::milliseconds);
void set_error_handler(ErrorHandler); // void(node_name, exception_ptr)
void set_diagnostics_handler(DiagnosticsHandler); // fired each watchdog tick
void set_event_handler(EventHandler); // void(name, NodeEvent, timestamp)
void register_pool(const std::string&, IPoolProbe*);
void print_diagnostics(std::ostream& = std::cerr) const; // formatted table
};
```
- **`connect`** static-asserts that the source output type equals the destination input type
(via the nodes' `return_tuple` / `args_tuple`), sets the consumer's input channel as the
producer's output pointer, registers a `ChannelProbe` for diagnostics, and rejects a second
connection from the same output port (use `make_fanout`).
- **`build`** colours the graph DFS; a back-edge throws `NetworkCycleError`. It also wires each
node's network-level overflow/closed callbacks to the `EventHandler` if one is set.
- **`halt` vs `shutdown`** — `halt()` disables channels and stops nodes in reverse order
immediately; `shutdown()` walks source layers first, polling channel probes until they drain
before stopping the next layer.
- **Watchdog** — a `std::jthread` that wakes on `watchdog_interval_` (default 3 s), collects
snapshots, warns about nodes whose `exec_start_us` indicates an execution running > 5 s, and
either calls the diagnostics handler or prints the formatted report.
- **`expose_input`/`expose_output`** record boundary names (sub-network support is scaffolded;
`Network` is itself an `INode` and can be `add()`ed to an outer `Network`).
The formatted report includes node (frames, exec ms, max ms, blocked ms, fps, cpu ms, util%),
channel (fill%, peak%, pushes, drops, overflow, MB/s, item bytes), and pool tables, plus a
bottleneck hint (highest `ema_exec_ms`).
---
## Component 9 — `static_network.hpp`: Compile-time Graph Builder
For C++ graphs whose full topology is known at compile time. The complete edge list is a type
pack, so fan-out arity is known up front, cycle detection is a `static_assert`, and start/stop
are pointer-vector traversals rather than string-map + virtual dispatch.
```cpp
// edge() builds a typed Edge descriptor from two port handles.
template<typename SrcNode, std::size_t SrcIdx, typename DstNode, std::size_t DstIdx>
Edge<SrcNode, SrcIdx, DstNode, DstIdx>
edge(OutputPort<SrcNode, SrcIdx>, InputPort<DstNode, DstIdx>);
// make_network() takes all edges, expands fan-outs, wires channels, returns a StaticNetwork.
template<typename... Edges> auto make_network(Edges&&... edges);
```
Usage — no `add`/`connect`/`build`/string names; one source port feeding two destinations
auto-inserts a `FanoutNode`:
```cpp
auto src = make_node<produce, "src">(8);
auto blur = make_node<blur_func, "blur">(8);
auto detect = make_node<detect_func,"detect">(8);
auto sink = make_node<display, "sink">(8);
auto net = make_network(
edge(src.output<0>(), blur.input<0>()),
edge(src.output<0>(), detect.input<0>()), // same source port → FanoutNode<T,2> inserted
edge(blur.output<0>(), sink.input<0>()),
edge(detect.output<0>(), sink.input<1>()));
net.start(); /* … */ net.stop();
```
`make_network` performs, at compile time: fan-out detection and edge expansion
(`tmp/fanout_groups.hpp`), a duplicate-`(Func, UniqueTag)` check
(`static_assert` — "add a UniqueTag"), and a cycle check + topological order
(`tmp/topo_sort.hpp`, `static_assert` — "graph contains a directed cycle"). At run time it
heap-allocates owned `FanoutNode` storage, collects user-node pointers in edge order, sets each
node's display name (`Label`, else `node[UniqueTag]`; fan-outs become `"<src>_fanout"`), wires
every expanded edge, and builds channel probes.
`StaticNetwork<FanoutStorage, TopoNodeList>` implements `INode` (so it can be embedded in a
runtime `Network`). It owns the fan-out nodes, holds user nodes by pointer, and provides
`start`/`halt`/`shutdown`, an `EventHandler`, `register_resource` / `register_pool`,
`print_diagnostics`, and `network_snapshot()` (consumed by the `DebugHub`). Compile-time labels
are read from each `NodeType::label()`.
> `make_fanout<T,N>` remains for explicit fan-out in a runtime `Network`; `make_network` users
> never call it.
---
## Component 10 — Web Debugging (optional, `KPN_WEB_DEBUG`)
Zero cost when disabled — guarded headers, no symbols, no dependency. Depends on **cpp-httplib**
(single-header, fetched by CMake when the option is on) and loads **D3.js v7** from CDN. Enable
per-target:
```cpp
#define KPN_WEB_DEBUG 1
#include <kpn/kpn.hpp>
```
### Single-network server (`web_debug.hpp`)
When enabled, `Network` / `StaticNetwork` gain `set_web_debug_port(uint16_t)` (default 9090) and
auto-start an in-process HTTP server in `start()`. It serves an inline single-page app at `/` and
a JSON snapshot at `/api/snapshot` (nodes, channels/edges, pools, resources, elapsed). The page
renders a force-directed graph: node colour encodes `ema_exec_ms`, edge colour encodes fill%,
with hover tooltips for the full stat set; it polls every 500 ms.
### `DebugHub` — multi-network UI (`debug_hub.hpp`)
A standalone server aggregating several networks under one endpoint:
```cpp
DebugHub hub(9090);
hub.register_network("detect", detect_net); // disables that net's own server
hub.register_network("classify", classify_net);
hub.register_resource("gpu", &gpu_resource); // shows utilisation cards
hub.start();
```
The hub UI has one tab per registered network plus an "All Networks" tab with shared-resource
cards and a cross-network node table. `register_network` calls `net.disable_web_server()` so the
hub is the single debug endpoint; call it before `net.start()`.
---
## Component 11 — Python Bindings (partial)
> Status: scaffolded and partially implemented. The variant machinery, `PyNetwork`/`PyNode`, and
> the auto-binding layer exist; the demo module wires a hello-pipeline. Full sub-port read/write
> and mixed C++/Python graphs are still in progress.
Python graphs cannot resolve types at compile time, so a `PyNetwork` is parameterised by a
`std::variant` derived (at compile time, via `unique_types`) from the port types of a **closed
list of registered C++ node types**. The variant only appears at the C++/Python boundary; each
node's internal `Channel<T>` still stores raw `T` (`variant_node.hpp`: `IVariantChannel`,
`VariantChannel<T,Variant>`, `IVariantNode`, `VariantNodeWrapper`).
### Auto-binding (`python/auto_bind.hpp`)
The node list is declared once with a `NodeRegistry` of `Entry<func, "name">`. `bind_network`
registers the `PyNetwork` class, a `make_<name>(capacity)` factory and a `<Name>Node` class per
entry, and auto-registers `PythonConverter` for each port type. `bind_debug` additionally exposes
each raw C++ function as a free Python callable for testing without a network. Recompiling the
extension is the registration step — there is no CMake code-gen.
```cpp
using DemoNodes = kpn::python::NodeRegistry<
kpn::python::Entry<produce, "produce">,
kpn::python::Entry<double_it, "double_it">,
kpn::python::Entry<print_it, "print_it">>; // variant auto-deduced as std::variant<int>
NB_MODULE(kpn_python, m) {
bind_network<DemoNodes>(m);
bind_debug<DemoNodes>(m);
}
```
Custom types are supported by specialising `kpn::PythonConverter<T>` (`to_python` / `from_python`,
optional `type_name`) before `bind_network`.
### GIL rules (non-negotiable)
1. **Acquire for callback** — hold the GIL only for the duration of a Python callable
invocation (`nb::gil_scoped_acquire` around the call site).
2. **Release while blocking** — release the GIL before any blocking channel op
(`nb::gil_scoped_release`), then re-acquire. Violating this deadlocks: a PyNode thread
waiting for the GIL cannot proceed while another thread holds it and blocks on a channel
waiting for that PyNode.
---
## Error Handling Contract
| Situation | Behaviour |
|---|---|
| FIFO overflow (full, accepting) | `ChannelOverflowError` thrown in producer; node overflow callbacks fire |
| Push to a disabled channel | Value silently dropped (counted as a `drop`) |
| Node function throws | Routed to the node's `NodeErrorHandler` → `true` skips & continues, `false` stops the node |
| Node stopped / channel closed | `ChannelClosedError` → node fires closed callbacks and self-stops |
| Type mismatch (C++) | `static_assert` at `connect()` / `make_network()` |
| Cycle in graph (runtime) | `NetworkCycleError` thrown at `build()` |
| Cycle in graph (static) | `static_assert` at `make_network()` |
| Duplicate `(Func, UniqueTag)` (static) | `static_assert` at `make_network()` — add a `UniqueTag` |
| Hung node | Watchdog warning after threshold |
`Network` additionally exposes an aggregate `EventHandler(name, NodeEvent, timestamp)` for
overflow/closed events across all nodes.
---
## Thread Model
KPN++ is **reactive**, not one-thread-per-node:
- A `PoolNode` owns no thread. It registers a push-callback on each input channel; when all
inputs are ready it submits `fire_once()` to a shared `IScheduler` (a `ThreadPool`).
- `Node<>` wraps a `PoolNode` plus a **private `ThreadPool(1)`**, recovering "independent
worker" semantics with the simple `make_node` API. Many nodes can instead share one pool
(`make_pool_node`) for a bounded OS thread count.
- `FanoutNode`, `RouterNode`, and `FilterNode` do run a dedicated `std::jthread` blocking on
`pop()` (they are simple, latency-sensitive routers).
- `InterruptNode` fires on an external trigger; `MainThreadNode` runs on the caller's main
thread via `step()`.
`std::jthread` (C++20) and its `stop_token` are used where a thread is owned, simplifying
cooperative shutdown. Benchmarks (`benchmarks/bench_pipeline`) show ~2–7 µs/hop framework
overhead for chains within the core count, rising under oversubscription.
---
## Platform and Compiler Requirements
C++20 is required.
| Feature | Min compiler |
|---|---|
| NTTP structural types (`fixed_string`) | GCC 11, Clang 13, MSVC 19.29 |
| `std::atomic::wait/notify` (channel futex) | GCC 11, Clang 13, MSVC 19.29 |
| `std::jthread` + `stop_token` | GCC 11, Clang 14, MSVC 19.29 |
| `auto` NTTPs, fold expressions, `if constexpr`, concepts | C++20 / C++17 baseline |
`CLOCK_THREAD_CPUTIME_ID` (per-thread CPU stats in `diagnostics.hpp`) is POSIX. nanobind
requires Python 3.8+ (auto-fetched when `KPN_BUILD_PYTHON=ON`).
---
## Testing Strategy
**Catch2 v3** for behaviour/integration tests and **GoogleTest** for unit and death tests; both
are auto-fetched. Existing suites: `test_fixed_string`, `test_traits`, `test_channel`,
`test_node`, `test_network`, `test_static_network`, `test_scheduler`, `test_pool_node`,
`test_shared_resource`.
Cases covered explicitly include: channel blocking/unblocking and overflow; shutdown races
(`stop()` while blocked on `pop()`); `try_pop_now`; fan-out delivery; tuple unpacking to
sub-channels; runtime cycle detection and static cycle/duplicate-tag `static_assert`s; named
port lookup and wrong-name-count `static_assert`s; storage-policy by-value vs `shared_ptr`;
scheduler submit/steal/drain; `PoolNode` reactive scheduling; and `SharedResource` priority +
aging.
---
## Examples
Self-contained programs under `examples/`, built by default (`-DKPN_BUILD_EXAMPLES=OFF` to
skip). They double as documentation and smoke tests.
| Example | What it shows |
|---|---|
| `01_hello_pipeline` | Linear pipeline, index-based wiring, `Network` builder |
| `02_named_ports` | `in<>`/`out<>` tags, named port access, wrong-name `static_assert` |
| `03_multi_output` | Tuple-returning node, per-element sub-port routing |
| `04_storage_policy` | `channel_storage_policy` default + specialisation |
| `05_error_handling` | `ChannelOverflowError`, diagnostics handler |
| `06_watchdog` | Watchdog interval, stall detection |
| `07_python_network` | `PyNetwork` with a pure-Python node *(pending)* |
| `08_python_subport` | `net.read` / `net.write`, sub-port tap *(pending)* |
| `09_opencv_cellshade` | Real-time cell-shading on webcam; named ports, fan-out, `MainThreadNode` display (requires OpenCV) |
| `10_static_hello_pipeline` | `make_network()` version of 01 — compile-time topology |
| `11_static_fanout` | Auto-inserted `FanoutNode` from a duplicated source port |
| `12_static_cellshade` | Static cell-shading with auto fan-out and `Label` NTTPs |
| `13_debug_cellshade` | One-op-per-node pipeline + variadic `DebugCanvas<N>` tiling node |
| `14_debug_hub` | Two networks sharing a `SharedResource` via `DebugHub` |
| `15_node_error_handler` | Per-node `set_error_handler` (skip-and-continue vs stop) |
| `16_event_callbacks` | `set_overflow_callback` + network `set_event_handler` |
---
## Future Extension Points (Heterogeneous Execution)
Not implemented, but the design keeps these doors open:
- **`IChannel` abstract interface** — `Channel<T>` and a future `RemoteChannel<T>` (socket /
shared-memory) sharing one `push`/`pop` surface so nodes are agnostic to channel location.
- **`Serializer<T>` trait** — parallel to `channel_storage_policy` / `PythonConverter`, for
cross-device serialisation (MessagePack for embedded, pinned memory for GPU zero-copy).
- **`NodeKind` tag** — e.g. `{ Local, Gpu, Remote }` on `INode`, letting the watchdog apply
per-device health-check and timeout strategies.
The `IScheduler` abstraction already decouples node execution from any specific thread model,
making a cooperative or device-specific executor a drop-in.
---
## Resolved Design Decisions
| Question | Decision |
|---|---|
| Execution model | Reactive: nodes submit `fire_once()` to an `IScheduler` when inputs are ready, not one blocking thread per node |
| `Node<>` vs `PoolNode<>` | `Node<>` owns a private `ThreadPool(1)`; `PoolNode<>` shares a pool for bounded threads |
| Channel | Lock-free SPSC ring buffer, `atomic::wait/notify` + spin-before-sleep |
| Shutdown | Per-channel `accepting_` flag; `disable()` unblocks `pop()` (→ `ChannelClosedError`) |
| Overflow | `ChannelOverflowError` on full accepting channel; silent drop on disabled channel |
| Node error policy | Per-node `NodeErrorHandler` returning bool (skip vs stop) |
| Network ownership | Non-owning; user declares nodes, network stores `INode*` |
| Fan-out | Explicit `FanoutNode<T,N>` for runtime `Network`; auto-inserted by `make_network()` |
| Branching | `RouterNode<T,N>` (select one of N) and `FilterNode<T>` (predicate gate) |
| Static vs runtime graph | Both; `StaticNetwork` for compile-time C++ topology, `Network` for dynamic/Python; `StaticNetwork` is an `INode` so it embeds in `Network` |
| Node identity (static graphs) | `Label` NTTP (name) + `UniqueTag` NTTP (collision-breaker); both default |
| Shared device resource | `SharedResource<T>` with priority + aging arbitration |
| Main-thread / GUI work | `MainThreadNode<>` driven by `step()` on the main thread |
| External-event sources | `InterruptNode` with a thread-safe `get_trigger()` |
| Web debugging | Per-network server + multi-network `DebugHub`, behind `KPN_WEB_DEBUG` |
| Mixed-rate latched inputs | **Not implemented** — no `latch<>` ports |
assets/images/favicon.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 1.8 KiB

assets/javascripts/bundle.79ae519e.min.js
Vendored
-16
View File
File diff suppressed because one or more lines are too long
assets/javascripts/bundle.79ae519e.min.js.map
-7
View File
File diff suppressed because one or more lines are too long
assets/javascripts/lunr/min/lunr.ar.min.js
Vendored
-1
View File
File diff suppressed because one or more lines are too long
assets/javascripts/lunr/min/lunr.da.min.js
Vendored
-18
View File
@@ -1,18 +0,0 @@
/*!
* Lunr languages, `Danish` language
* https://github.com/MihaiValentin/lunr-languages
*
* Copyright 2014, Mihai Valentin
* http://www.mozilla.org/MPL/
*/
/*!
* based on
* Snowball JavaScript Library v0.3
* http://code.google.com/p/urim/
* http://snowball.tartarus.org/
*
* Copyright 2010, Oleg Mazko
* http://www.mozilla.org/MPL/
*/
!function(e,r){"function"==typeof define&&define.amd?define(r):"object"==typeof exports?module.exports=r():r()(e.lunr)}(this,function(){return function(e){if(void 0===e)throw new Error("Lunr is not present. Please include / require Lunr before this script.");if(void 0===e.stemmerSupport)throw new Error("Lunr stemmer support is not present. Please include / require Lunr stemmer support before this script.");e.da=function(){this.pipeline.reset(),this.pipeline.add(e.da.trimmer,e.da.stopWordFilter,e.da.stemmer),this.searchPipeline&&(this.searchPipeline.reset(),this.searchPipeline.add(e.da.stemmer))},e.da.wordCharacters="A-Za-zªºÀ-ÖØ-öø-ʸˠ-ˤᴀ-ᴥᴬ-ᵜᵢ-ᵥᵫ-ᵷᵹ-ᶾḀ-ỿⁱⁿₐ-ₜKÅℲⅎⅠ-ↈⱠ-ⱿꜢ-ꞇꞋ-ꞭꞰ-ꞷꟷ-ꟿꬰ-ꭚꭜ-ꭤff-stＡ-Ｚａ-ｚ",e.da.trimmer=e.trimmerSupport.generateTrimmer(e.da.wordCharacters),e.Pipeline.registerFunction(e.da.trimmer,"trimmer-da"),e.da.stemmer=function(){var r=e.stemmerSupport.Among,i=e.stemmerSupport.SnowballProgram,n=new function(){function e(){var e,r=f.cursor+3;if(d=f.limit,0<=r&&r<=f.limit){for(a=r;;){if(e=f.cursor,f.in_grouping(w,97,248)){f.cursor=e;break}if(f.cursor=e,e>=f.limit)return;f.cursor++}for(;!f.out_grouping(w,97,248);){if(f.cursor>=f.limit)return;f.cursor++}d=f.cursor,d<a&&(d=a)}}function n(){var e,r;if(f.cursor>=d&&(r=f.limit_backward,f.limit_backward=d,f.ket=f.cursor,e=f.find_among_b(c,32),f.limit_backward=r,e))switch(f.bra=f.cursor,e){case 1:f.slice_del();break;case 2:f.in_grouping_b(p,97,229)&&f.slice_del()}}function t(){var e,r=f.limit-f.cursor;f.cursor>=d&&(e=f.limit_backward,f.limit_backward=d,f.ket=f.cursor,f.find_among_b(l,4)?(f.bra=f.cursor,f.limit_backward=e,f.cursor=f.limit-r,f.cursor>f.limit_backward&&(f.cursor--,f.bra=f.cursor,f.slice_del())):f.limit_backward=e)}function s(){var e,r,i,n=f.limit-f.cursor;if(f.ket=f.cursor,f.eq_s_b(2,"st")&&(f.bra=f.cursor,f.eq_s_b(2,"ig")&&f.slice_del()),f.cursor=f.limit-n,f.cursor>=d&&(r=f.limit_backward,f.limit_backward=d,f.ket=f.cursor,e=f.find_among_b(m,5),f.limit_backward=r,e))switch(f.bra=f.cursor,e){case 1:f.slice_del(),i=f.limit-f.cursor,t(),f.cursor=f.limit-i;break;case 2:f.slice_from("løs")}}function o(){var e;f.cursor>=d&&(e=f.limit_backward,f.limit_backward=d,f.ket=f.cursor,f.out_grouping_b(w,97,248)?(f.bra=f.cursor,u=f.slice_to(u),f.limit_backward=e,f.eq_v_b(u)&&f.slice_del()):f.limit_backward=e)}var a,d,u,c=[new r("hed",-1,1),new r("ethed",0,1),new r("ered",-1,1),new r("e",-1,1),new r("erede",3,1),new r("ende",3,1),new r("erende",5,1),new r("ene",3,1),new r("erne",3,1),new r("ere",3,1),new r("en",-1,1),new r("heden",10,1),new r("eren",10,1),new r("er",-1,1),new r("heder",13,1),new r("erer",13,1),new r("s",-1,2),new r("heds",16,1),new r("es",16,1),new r("endes",18,1),new r("erendes",19,1),new r("enes",18,1),new r("ernes",18,1),new r("eres",18,1),new r("ens",16,1),new r("hedens",24,1),new r("erens",24,1),new r("ers",16,1),new r("ets",16,1),new r("erets",28,1),new r("et",-1,1),new r("eret",30,1)],l=[new r("gd",-1,-1),new r("dt",-1,-1),new r("gt",-1,-1),new r("kt",-1,-1)],m=[new r("ig",-1,1),new r("lig",0,1),new r("elig",1,1),new r("els",-1,1),new r("løst",-1,2)],w=[17,65,16,1,0,0,0,0,0,0,0,0,0,0,0,0,48,0,128],p=[239,254,42,3,0,0,0,0,0,0,0,0,0,0,0,0,16],f=new i;this.setCurrent=function(e){f.setCurrent(e)},this.getCurrent=function(){return f.getCurrent()},this.stem=function(){var r=f.cursor;return e(),f.limit_backward=r,f.cursor=f.limit,n(),f.cursor=f.limit,t(),f.cursor=f.limit,s(),f.cursor=f.limit,o(),!0}};return function(e){return"function"==typeof e.update?e.update(function(e){return n.setCurrent(e),n.stem(),n.getCurrent()}):(n.setCurrent(e),n.stem(),n.getCurrent())}}(),e.Pipeline.registerFunction(e.da.stemmer,"stemmer-da"),e.da.stopWordFilter=e.generateStopWordFilter("ad af alle alt anden at blev blive bliver da de dem den denne der deres det dette dig din disse dog du efter eller en end er et for fra ham han hans har havde have hende hendes her hos hun hvad hvis hvor i ikke ind jeg jer jo kunne man mange med meget men mig min mine mit mod ned noget nogle nu når og også om op os over på selv sig sin sine sit skal skulle som sådan thi til ud under var vi vil ville vor være været".split(" ")),e.Pipeline.registerFunction(e.da.stopWordFilter,"stopWordFilter-da")}});
assets/javascripts/lunr/min/lunr.de.min.js
Vendored
-18
View File
File diff suppressed because one or more lines are too long
assets/javascripts/lunr/min/lunr.du.min.js
Vendored
-18
View File
File diff suppressed because one or more lines are too long
assets/javascripts/lunr/min/lunr.el.min.js
Vendored
-1
View File
File diff suppressed because one or more lines are too long
assets/javascripts/lunr/min/lunr.es.min.js
Vendored
-18
View File
File diff suppressed because one or more lines are too long
assets/javascripts/lunr/min/lunr.fi.min.js
Vendored
-18
View File
File diff suppressed because one or more lines are too long
assets/javascripts/lunr/min/lunr.fr.min.js
Vendored
-18
View File
File diff suppressed because one or more lines are too long
assets/javascripts/lunr/min/lunr.he.min.js
Vendored
-1
View File
File diff suppressed because one or more lines are too long
assets/javascripts/lunr/min/lunr.hi.min.js
Vendored
-1
View File
@@ -1 +0,0 @@
!function(e,r){"function"==typeof define&&define.amd?define(r):"object"==typeof exports?module.exports=r():r()(e.lunr)}(this,function(){return function(e){if(void 0===e)throw new Error("Lunr is not present. Please include / require Lunr before this script.");if(void 0===e.stemmerSupport)throw new Error("Lunr stemmer support is not present. Please include / require Lunr stemmer support before this script.");e.hi=function(){this.pipeline.reset(),this.pipeline.add(e.hi.trimmer,e.hi.stopWordFilter,e.hi.stemmer),this.searchPipeline&&(this.searchPipeline.reset(),this.searchPipeline.add(e.hi.stemmer))},e.hi.wordCharacters="ऀ-ःऄ-एऐ-टठ-यर-िी-ॏॐ-य़ॠ-९॰-ॿa-zA-Zａ-ｚＡ-Ｚ0-9０-９",e.hi.trimmer=e.trimmerSupport.generateTrimmer(e.hi.wordCharacters),e.Pipeline.registerFunction(e.hi.trimmer,"trimmer-hi"),e.hi.stopWordFilter=e.generateStopWordFilter("अत अपना अपनी अपने अभी अंदर आदि आप इत्यादि इन इनका इन्हीं इन्हें इन्हों इस इसका इसकी इसके इसमें इसी इसे उन उनका उनकी उनके उनको उन्हीं उन्हें उन्हों उस उसके उसी उसे एक एवं एस ऐसे और कई कर करता करते करना करने करें कहते कहा का काफ़ी कि कितना किन्हें किन्हों किया किर किस किसी किसे की कुछ कुल के को कोई कौन कौनसा गया घर जब जहाँ जा जितना जिन जिन्हें जिन्हों जिस जिसे जीधर जैसा जैसे जो तक तब तरह तिन तिन्हें तिन्हों तिस तिसे तो था थी थे दबारा दिया दुसरा दूसरे दो द्वारा न नके नहीं ना निहायत नीचे ने पर पहले पूरा पे फिर बनी बही बहुत बाद बाला बिलकुल भी भीतर मगर मानो मे में यदि यह यहाँ यही या यिह ये रखें रहा रहे ऱ्वासा लिए लिये लेकिन व वग़ैरह वर्ग वह वहाँ वहीं वाले वुह वे वो सकता सकते सबसे सभी साथ साबुत साभ सारा से सो संग ही हुआ हुई हुए है हैं हो होता होती होते होना होने".split(" ")),e.hi.stemmer=function(){return function(e){return"function"==typeof e.update?e.update(function(e){return e}):e}}();var r=e.wordcut;r.init(),e.hi.tokenizer=function(i){if(!arguments.length||null==i||void 0==i)return[];if(Array.isArray(i))return i.map(function(r){return isLunr2?new e.Token(r.toLowerCase()):r.toLowerCase()});var t=i.toString().toLowerCase().replace(/^\s+/,"");return r.cut(t).split("|")},e.Pipeline.registerFunction(e.hi.stemmer,"stemmer-hi"),e.Pipeline.registerFunction(e.hi.stopWordFilter,"stopWordFilter-hi")}});
assets/javascripts/lunr/min/lunr.hu.min.js
Vendored
-18
View File
File diff suppressed because one or more lines are too long
assets/javascripts/lunr/min/lunr.hy.min.js
Vendored
-1
View File
@@ -1 +0,0 @@
!function(e,r){"function"==typeof define&&define.amd?define(r):"object"==typeof exports?module.exports=r():r()(e.lunr)}(this,function(){return function(e){if(void 0===e)throw new Error("Lunr is not present. Please include / require Lunr before this script.");if(void 0===e.stemmerSupport)throw new Error("Lunr stemmer support is not present. Please include / require Lunr stemmer support before this script.");e.hy=function(){this.pipeline.reset(),this.pipeline.add(e.hy.trimmer,e.hy.stopWordFilter)},e.hy.wordCharacters="[A-Za-z԰-֏ff-ﭏ]",e.hy.trimmer=e.trimmerSupport.generateTrimmer(e.hy.wordCharacters),e.Pipeline.registerFunction(e.hy.trimmer,"trimmer-hy"),e.hy.stopWordFilter=e.generateStopWordFilter("դու և եք էիր էիք հետո նաև նրանք որը վրա է որ պիտի են այս մեջ ն իր ու ի այդ որոնք այն կամ էր մի ես համար այլ իսկ էին ենք հետ ին թ էինք մենք նրա նա դուք եմ էի ըստ որպես ում".split(" ")),e.Pipeline.registerFunction(e.hy.stopWordFilter,"stopWordFilter-hy"),e.hy.stemmer=function(){return function(e){return"function"==typeof e.update?e.update(function(e){return e}):e}}(),e.Pipeline.registerFunction(e.hy.stemmer,"stemmer-hy")}});
assets/javascripts/lunr/min/lunr.it.min.js
Vendored
-18
View File
File diff suppressed because one or more lines are too long
assets/javascripts/lunr/min/lunr.ja.min.js
Vendored
-1
View File
@@ -1 +0,0 @@
!function(e,r){"function"==typeof define&&define.amd?define(r):"object"==typeof exports?module.exports=r():r()(e.lunr)}(this,function(){return function(e){if(void 0===e)throw new Error("Lunr is not present. Please include / require Lunr before this script.");if(void 0===e.stemmerSupport)throw new Error("Lunr stemmer support is not present. Please include / require Lunr stemmer support before this script.");var r="2"==e.version[0];e.ja=function(){this.pipeline.reset(),this.pipeline.add(e.ja.trimmer,e.ja.stopWordFilter,e.ja.stemmer),r?this.tokenizer=e.ja.tokenizer:(e.tokenizer&&(e.tokenizer=e.ja.tokenizer),this.tokenizerFn&&(this.tokenizerFn=e.ja.tokenizer))};var t=new e.TinySegmenter;e.ja.tokenizer=function(i){var n,o,s,p,a,u,m,l,c,f;if(!arguments.length||null==i||void 0==i)return[];if(Array.isArray(i))return i.map(function(t){return r?new e.Token(t.toLowerCase()):t.toLowerCase()});for(o=i.toString().toLowerCase().replace(/^\s+/,""),n=o.length-1;n>=0;n--)if(/\S/.test(o.charAt(n))){o=o.substring(0,n+1);break}for(a=[],s=o.length,c=0,l=0;c<=s;c++)if(u=o.charAt(c),m=c-l,u.match(/\s/)||c==s){if(m>0)for(p=t.segment(o.slice(l,c)).filter(function(e){return!!e}),f=l,n=0;n<p.length;n++)r?a.push(new e.Token(p[n],{position:[f,p[n].length],index:a.length})):a.push(p[n]),f+=p[n].length;l=c+1}return a},e.ja.stemmer=function(){return function(e){return e}}(),e.Pipeline.registerFunction(e.ja.stemmer,"stemmer-ja"),e.ja.wordCharacters="一二三四五六七八九十百千万億兆一-龠々〆ヵヶぁ-んァ-ヴーｱ-ﾝﾞa-zA-Zａ-ｚＡ-Ｚ0-9０-９",e.ja.trimmer=e.trimmerSupport.generateTrimmer(e.ja.wordCharacters),e.Pipeline.registerFunction(e.ja.trimmer,"trimmer-ja"),e.ja.stopWordFilter=e.generateStopWordFilter("これ それ あれ この その あの ここ そこ あそこ こちら どこ だれ なに なん 何 私 貴方 貴方方 我々 私達 あの人 あのかた 彼女 彼 です あります おります います は が の に を で え から まで より も どの と し それで しかし".split(" ")),e.Pipeline.registerFunction(e.ja.stopWordFilter,"stopWordFilter-ja"),e.jp=e.ja,e.Pipeline.registerFunction(e.jp.stemmer,"stemmer-jp"),e.Pipeline.registerFunction(e.jp.trimmer,"trimmer-jp"),e.Pipeline.registerFunction(e.jp.stopWordFilter,"stopWordFilter-jp")}});
assets/javascripts/lunr/min/lunr.jp.min.js
Vendored
-1
View File
@@ -1 +0,0 @@
module.exports=require("./lunr.ja");
assets/javascripts/lunr/min/lunr.kn.min.js
Vendored
-1
View File
@@ -1 +0,0 @@
!function(e,r){"function"==typeof define&&define.amd?define(r):"object"==typeof exports?module.exports=r():r()(e.lunr)}(this,function(){return function(e){if(void 0===e)throw new Error("Lunr is not present. Please include / require Lunr before this script.");if(void 0===e.stemmerSupport)throw new Error("Lunr stemmer support is not present. Please include / require Lunr stemmer support before this script.");e.kn=function(){this.pipeline.reset(),this.pipeline.add(e.kn.trimmer,e.kn.stopWordFilter,e.kn.stemmer),this.searchPipeline&&(this.searchPipeline.reset(),this.searchPipeline.add(e.kn.stemmer))},e.kn.wordCharacters="ಀ-಄ಅ-ಔಕ-ಹಾ-ೌ಼-ಽೕ-ೖೝ-ೞೠ-ೡೢ-ೣ೤೥೦-೯ೱ-ೳ",e.kn.trimmer=e.trimmerSupport.generateTrimmer(e.kn.wordCharacters),e.Pipeline.registerFunction(e.kn.trimmer,"trimmer-kn"),e.kn.stopWordFilter=e.generateStopWordFilter("ಮತ್ತು ಈ ಒಂದು ರಲ್ಲಿ ಹಾಗೂ ಎಂದು ಅಥವಾ ಇದು ರ ಅವರು ಎಂಬ ಮೇಲೆ ಅವರ ತನ್ನ ಆದರೆ ತಮ್ಮ ನಂತರ ಮೂಲಕ ಹೆಚ್ಚು ನ ಆ ಕೆಲವು ಅನೇಕ ಎರಡು ಹಾಗು ಪ್ರಮುಖ ಇದನ್ನು ಇದರ ಸುಮಾರು ಅದರ ಅದು ಮೊದಲ ಬಗ್ಗೆ ನಲ್ಲಿ ರಂದು ಇತರ ಅತ್ಯಂತ ಹೆಚ್ಚಿನ ಸಹ ಸಾಮಾನ್ಯವಾಗಿ ನೇ ಹಲವಾರು ಹೊಸ ದಿ ಕಡಿಮೆ ಯಾವುದೇ ಹೊಂದಿದೆ ದೊಡ್ಡ ಅನ್ನು ಇವರು ಪ್ರಕಾರ ಇದೆ ಮಾತ್ರ ಕೂಡ ಇಲ್ಲಿ ಎಲ್ಲಾ ವಿವಿಧ ಅದನ್ನು ಹಲವು ರಿಂದ ಕೇವಲ ದ ದಕ್ಷಿಣ ಗೆ ಅವನ ಅತಿ ನೆಯ ಬಹಳ ಕೆಲಸ ಎಲ್ಲ ಪ್ರತಿ ಇತ್ಯಾದಿ ಇವು ಬೇರೆ ಹೀಗೆ ನಡುವೆ ಇದಕ್ಕೆ ಎಸ್ ಇವರ ಮೊದಲು ಶ್ರೀ ಮಾಡುವ ಇದರಲ್ಲಿ ರೀತಿಯ ಮಾಡಿದ ಕಾಲ ಅಲ್ಲಿ ಮಾಡಲು ಅದೇ ಈಗ ಅವು ಗಳು ಎ ಎಂಬುದು ಅವನು ಅಂದರೆ ಅವರಿಗೆ ಇರುವ ವಿಶೇಷ ಮುಂದೆ ಅವುಗಳ ಮುಂತಾದ ಮೂಲ ಬಿ ಮೀ ಒಂದೇ ಇನ್ನೂ ಹೆಚ್ಚಾಗಿ ಮಾಡಿ ಅವರನ್ನು ಇದೇ ಯ ರೀತಿಯಲ್ಲಿ ಜೊತೆ ಅದರಲ್ಲಿ ಮಾಡಿದರು ನಡೆದ ಆಗ ಮತ್ತೆ ಪೂರ್ವ ಆತ ಬಂದ ಯಾವ ಒಟ್ಟು ಇತರೆ ಹಿಂದೆ ಪ್ರಮಾಣದ ಗಳನ್ನು ಕುರಿತು ಯು ಆದ್ದರಿಂದ ಅಲ್ಲದೆ ನಗರದ ಮೇಲಿನ ಏಕೆಂದರೆ ರಷ್ಟು ಎಂಬುದನ್ನು ಬಾರಿ ಎಂದರೆ ಹಿಂದಿನ ಆದರೂ ಆದ ಸಂಬಂಧಿಸಿದ ಮತ್ತೊಂದು ಸಿ ಆತನ ".split(" ")),e.kn.stemmer=function(){return function(e){return"function"==typeof e.update?e.update(function(e){return e}):e}}();var r=e.wordcut;r.init(),e.kn.tokenizer=function(t){if(!arguments.length||null==t||void 0==t)return[];if(Array.isArray(t))return t.map(function(r){return isLunr2?new e.Token(r.toLowerCase()):r.toLowerCase()});var n=t.toString().toLowerCase().replace(/^\s+/,"");return r.cut(n).split("|")},e.Pipeline.registerFunction(e.kn.stemmer,"stemmer-kn"),e.Pipeline.registerFunction(e.kn.stopWordFilter,"stopWordFilter-kn")}});
assets/javascripts/lunr/min/lunr.ko.min.js
Vendored
-1
View File
File diff suppressed because one or more lines are too long
assets/javascripts/lunr/min/lunr.multi.min.js
Vendored
-1
View File
@@ -1 +0,0 @@
!function(e,t){"function"==typeof define&&define.amd?define(t):"object"==typeof exports?module.exports=t():t()(e.lunr)}(this,function(){return function(e){e.multiLanguage=function(){for(var t=Array.prototype.slice.call(arguments),i=t.join("-"),r="",n=[],s=[],p=0;p<t.length;++p)"en"==t[p]?(r+="\\w",n.unshift(e.stopWordFilter),n.push(e.stemmer),s.push(e.stemmer)):(r+=e[t[p]].wordCharacters,e[t[p]].stopWordFilter&&n.unshift(e[t[p]].stopWordFilter),e[t[p]].stemmer&&(n.push(e[t[p]].stemmer),s.push(e[t[p]].stemmer)));var o=e.trimmerSupport.generateTrimmer(r);return e.Pipeline.registerFunction(o,"lunr-multi-trimmer-"+i),n.unshift(o),function(){this.pipeline.reset(),this.pipeline.add.apply(this.pipeline,n),this.searchPipeline&&(this.searchPipeline.reset(),this.searchPipeline.add.apply(this.searchPipeline,s))}}}});
assets/javascripts/lunr/min/lunr.nl.min.js
Vendored
-18
View File
File diff suppressed because one or more lines are too long
assets/javascripts/lunr/min/lunr.no.min.js
Vendored
-18
View File
@@ -1,18 +0,0 @@
/*!
* Lunr languages, `Norwegian` language
* https://github.com/MihaiValentin/lunr-languages
*
* Copyright 2014, Mihai Valentin
* http://www.mozilla.org/MPL/
*/
/*!
* based on
* Snowball JavaScript Library v0.3
* http://code.google.com/p/urim/
* http://snowball.tartarus.org/
*
* Copyright 2010, Oleg Mazko
* http://www.mozilla.org/MPL/
*/
!function(e,r){"function"==typeof define&&define.amd?define(r):"object"==typeof exports?module.exports=r():r()(e.lunr)}(this,function(){return function(e){if(void 0===e)throw new Error("Lunr is not present. Please include / require Lunr before this script.");if(void 0===e.stemmerSupport)throw new Error("Lunr stemmer support is not present. Please include / require Lunr stemmer support before this script.");e.no=function(){this.pipeline.reset(),this.pipeline.add(e.no.trimmer,e.no.stopWordFilter,e.no.stemmer),this.searchPipeline&&(this.searchPipeline.reset(),this.searchPipeline.add(e.no.stemmer))},e.no.wordCharacters="A-Za-zªºÀ-ÖØ-öø-ʸˠ-ˤᴀ-ᴥᴬ-ᵜᵢ-ᵥᵫ-ᵷᵹ-ᶾḀ-ỿⁱⁿₐ-ₜKÅℲⅎⅠ-ↈⱠ-ⱿꜢ-ꞇꞋ-ꞭꞰ-ꞷꟷ-ꟿꬰ-ꭚꭜ-ꭤff-stＡ-Ｚａ-ｚ",e.no.trimmer=e.trimmerSupport.generateTrimmer(e.no.wordCharacters),e.Pipeline.registerFunction(e.no.trimmer,"trimmer-no"),e.no.stemmer=function(){var r=e.stemmerSupport.Among,n=e.stemmerSupport.SnowballProgram,i=new function(){function e(){var e,r=w.cursor+3;if(a=w.limit,0<=r||r<=w.limit){for(s=r;;){if(e=w.cursor,w.in_grouping(d,97,248)){w.cursor=e;break}if(e>=w.limit)return;w.cursor=e+1}for(;!w.out_grouping(d,97,248);){if(w.cursor>=w.limit)return;w.cursor++}a=w.cursor,a<s&&(a=s)}}function i(){var e,r,n;if(w.cursor>=a&&(r=w.limit_backward,w.limit_backward=a,w.ket=w.cursor,e=w.find_among_b(m,29),w.limit_backward=r,e))switch(w.bra=w.cursor,e){case 1:w.slice_del();break;case 2:n=w.limit-w.cursor,w.in_grouping_b(c,98,122)?w.slice_del():(w.cursor=w.limit-n,w.eq_s_b(1,"k")&&w.out_grouping_b(d,97,248)&&w.slice_del());break;case 3:w.slice_from("er")}}function t(){var e,r=w.limit-w.cursor;w.cursor>=a&&(e=w.limit_backward,w.limit_backward=a,w.ket=w.cursor,w.find_among_b(u,2)?(w.bra=w.cursor,w.limit_backward=e,w.cursor=w.limit-r,w.cursor>w.limit_backward&&(w.cursor--,w.bra=w.cursor,w.slice_del())):w.limit_backward=e)}function o(){var e,r;w.cursor>=a&&(r=w.limit_backward,w.limit_backward=a,w.ket=w.cursor,e=w.find_among_b(l,11),e?(w.bra=w.cursor,w.limit_backward=r,1==e&&w.slice_del()):w.limit_backward=r)}var s,a,m=[new r("a",-1,1),new r("e",-1,1),new r("ede",1,1),new r("ande",1,1),new r("ende",1,1),new r("ane",1,1),new r("ene",1,1),new r("hetene",6,1),new r("erte",1,3),new r("en",-1,1),new r("heten",9,1),new r("ar",-1,1),new r("er",-1,1),new r("heter",12,1),new r("s",-1,2),new r("as",14,1),new r("es",14,1),new r("edes",16,1),new r("endes",16,1),new r("enes",16,1),new r("hetenes",19,1),new r("ens",14,1),new r("hetens",21,1),new r("ers",14,1),new r("ets",14,1),new r("et",-1,1),new r("het",25,1),new r("ert",-1,3),new r("ast",-1,1)],u=[new r("dt",-1,-1),new r("vt",-1,-1)],l=[new r("leg",-1,1),new r("eleg",0,1),new r("ig",-1,1),new r("eig",2,1),new r("lig",2,1),new r("elig",4,1),new r("els",-1,1),new r("lov",-1,1),new r("elov",7,1),new r("slov",7,1),new r("hetslov",9,1)],d=[17,65,16,1,0,0,0,0,0,0,0,0,0,0,0,0,48,0,128],c=[119,125,149,1],w=new n;this.setCurrent=function(e){w.setCurrent(e)},this.getCurrent=function(){return w.getCurrent()},this.stem=function(){var r=w.cursor;return e(),w.limit_backward=r,w.cursor=w.limit,i(),w.cursor=w.limit,t(),w.cursor=w.limit,o(),!0}};return function(e){return"function"==typeof e.update?e.update(function(e){return i.setCurrent(e),i.stem(),i.getCurrent()}):(i.setCurrent(e),i.stem(),i.getCurrent())}}(),e.Pipeline.registerFunction(e.no.stemmer,"stemmer-no"),e.no.stopWordFilter=e.generateStopWordFilter("alle at av bare begge ble blei bli blir blitt både båe da de deg dei deim deira deires dem den denne der dere deres det dette di din disse ditt du dykk dykkar då eg ein eit eitt eller elles en enn er et ett etter for fordi fra før ha hadde han hans har hennar henne hennes her hjå ho hoe honom hoss hossen hun hva hvem hver hvilke hvilken hvis hvor hvordan hvorfor i ikke ikkje ikkje ingen ingi inkje inn inni ja jeg kan kom korleis korso kun kunne kva kvar kvarhelst kven kvi kvifor man mange me med medan meg meget mellom men mi min mine mitt mot mykje ned no noe noen noka noko nokon nokor nokre nå når og også om opp oss over på samme seg selv si si sia sidan siden sin sine sitt sjøl skal skulle slik so som som somme somt så sånn til um upp ut uten var vart varte ved vere verte vi vil ville vore vors vort vår være være vært å".split(" ")),e.Pipeline.registerFunction(e.no.stopWordFilter,"stopWordFilter-no")}});
assets/javascripts/lunr/min/lunr.pt.min.js
Vendored
-18
View File
File diff suppressed because one or more lines are too long
assets/javascripts/lunr/min/lunr.ro.min.js
Vendored
-18
View File
File diff suppressed because one or more lines are too long
assets/javascripts/lunr/min/lunr.ru.min.js
Vendored
-18
View File
File diff suppressed because one or more lines are too long
assets/javascripts/lunr/min/lunr.sa.min.js
Vendored
-1
View File
@@ -1 +0,0 @@
!function(e,r){"function"==typeof define&&define.amd?define(r):"object"==typeof exports?module.exports=r():r()(e.lunr)}(this,function(){return function(e){if(void 0===e)throw new Error("Lunr is not present. Please include / require Lunr before this script.");if(void 0===e.stemmerSupport)throw new Error("Lunr stemmer support is not present. Please include / require Lunr stemmer support before this script.");e.sa=function(){this.pipeline.reset(),this.pipeline.add(e.sa.trimmer,e.sa.stopWordFilter,e.sa.stemmer),this.searchPipeline&&(this.searchPipeline.reset(),this.searchPipeline.add(e.sa.stemmer))},e.sa.wordCharacters="ऀ-ःऄ-एऐ-टठ-यर-िी-ॏॐ-य़ॠ-९॰-ॿ꣠-꣱ꣲ-ꣷ꣸-ꣻ꣼-ꣽꣾ-ꣿᆰ0-ᆰ9",e.sa.trimmer=e.trimmerSupport.generateTrimmer(e.sa.wordCharacters),e.Pipeline.registerFunction(e.sa.trimmer,"trimmer-sa"),e.sa.stopWordFilter=e.generateStopWordFilter('तथा अयम्‌ एकम्‌ इत्यस्मिन्‌ तथा तत्‌ वा अयम्‌ इत्यस्य ते आहूत उपरि तेषाम्‌ किन्तु तेषाम्‌ तदा इत्यनेन अधिकः इत्यस्य तत्‌ केचन बहवः द्वि तथा महत्वपूर्णः अयम्‌ अस्य विषये अयं अस्ति तत्‌ प्रथमः विषये इत्युपरि इत्युपरि इतर अधिकतमः अधिकः अपि सामान्यतया ठ इतरेतर नूतनम्‌ द न्यूनम्‌ कश्चित्‌ वा विशालः द सः अस्ति तदनुसारम् तत्र अस्ति केवलम्‌ अपि अत्र सर्वे विविधाः तत्‌ बहवः यतः इदानीम्‌ द दक्षिण इत्यस्मै तस्य उपरि नथ अतीव कार्यम्‌ सर्वे एकैकम्‌ इत्यादि। एते सन्ति उत इत्थम्‌ मध्ये एतदर्थं . स कस्य प्रथमः श्री. करोति अस्मिन् प्रकारः निर्मिता कालः तत्र कर्तुं समान अधुना ते सन्ति स एकः अस्ति सः अर्थात् तेषां कृते . स्थितम् विशेषः अग्रिम तेषाम्‌ समान स्रोतः ख म समान इदानीमपि अधिकतया करोतु ते समान इत्यस्य वीथी सह यस्मिन् कृतवान्‌ धृतः तदा पुनः पूर्वं सः आगतः किम्‌ कुल इतर पुरा मात्रा स विषये उ अतएव अपि नगरस्य उपरि यतः प्रतिशतं कतरः कालः साधनानि भूत तथापि जात सम्बन्धि अन्यत्‌ ग अतः अस्माकं स्वकीयाः अस्माकं इदानीं अन्तः इत्यादयः भवन्तः इत्यादयः एते एताः तस्य अस्य इदम् एते तेषां तेषां तेषां तान् तेषां तेषां तेषां समानः सः एकः च तादृशाः बहवः अन्ये च वदन्ति यत् कियत् कस्मै कस्मै यस्मै यस्मै यस्मै यस्मै न अतिनीचः किन्तु प्रथमं सम्पूर्णतया ततः चिरकालानन्तरं पुस्तकं सम्पूर्णतया अन्तः किन्तु अत्र वा इह इव श्रद्धाय अवशिष्यते परन्तु अन्ये वर्गाः सन्ति ते सन्ति शक्नुवन्ति सर्वे मिलित्वा सर्वे एकत्र"'.split(" ")),e.sa.stemmer=function(){return function(e){return"function"==typeof e.update?e.update(function(e){return e}):e}}();var r=e.wordcut;r.init(),e.sa.tokenizer=function(t){if(!arguments.length||null==t||void 0==t)return[];if(Array.isArray(t))return t.map(function(r){return isLunr2?new e.Token(r.toLowerCase()):r.toLowerCase()});var i=t.toString().toLowerCase().replace(/^\s+/,"");return r.cut(i).split("|")},e.Pipeline.registerFunction(e.sa.stemmer,"stemmer-sa"),e.Pipeline.registerFunction(e.sa.stopWordFilter,"stopWordFilter-sa")}});
assets/javascripts/lunr/min/lunr.stemmer.support.min.js
Vendored
-1
View File
@@ -1 +0,0 @@
!function(r,t){"function"==typeof define&&define.amd?define(t):"object"==typeof exports?module.exports=t():t()(r.lunr)}(this,function(){return function(r){r.stemmerSupport={Among:function(r,t,i,s){if(this.toCharArray=function(r){for(var t=r.length,i=new Array(t),s=0;s<t;s++)i[s]=r.charCodeAt(s);return i},!r&&""!=r||!t&&0!=t||!i)throw"Bad Among initialisation: s:"+r+", substring_i: "+t+", result: "+i;this.s_size=r.length,this.s=this.toCharArray(r),this.substring_i=t,this.result=i,this.method=s},SnowballProgram:function(){var r;return{bra:0,ket:0,limit:0,cursor:0,limit_backward:0,setCurrent:function(t){r=t,this.cursor=0,this.limit=t.length,this.limit_backward=0,this.bra=this.cursor,this.ket=this.limit},getCurrent:function(){var t=r;return r=null,t},in_grouping:function(t,i,s){if(this.cursor<this.limit){var e=r.charCodeAt(this.cursor);if(e<=s&&e>=i&&(e-=i,t[e>>3]&1<<(7&e)))return this.cursor++,!0}return!1},in_grouping_b:function(t,i,s){if(this.cursor>this.limit_backward){var e=r.charCodeAt(this.cursor-1);if(e<=s&&e>=i&&(e-=i,t[e>>3]&1<<(7&e)))return this.cursor--,!0}return!1},out_grouping:function(t,i,s){if(this.cursor<this.limit){var e=r.charCodeAt(this.cursor);if(e>s||e<i)return this.cursor++,!0;if(e-=i,!(t[e>>3]&1<<(7&e)))return this.cursor++,!0}return!1},out_grouping_b:function(t,i,s){if(this.cursor>this.limit_backward){var e=r.charCodeAt(this.cursor-1);if(e>s||e<i)return this.cursor--,!0;if(e-=i,!(t[e>>3]&1<<(7&e)))return this.cursor--,!0}return!1},eq_s:function(t,i){if(this.limit-this.cursor<t)return!1;for(var s=0;s<t;s++)if(r.charCodeAt(this.cursor+s)!=i.charCodeAt(s))return!1;return this.cursor+=t,!0},eq_s_b:function(t,i){if(this.cursor-this.limit_backward<t)return!1;for(var s=0;s<t;s++)if(r.charCodeAt(this.cursor-t+s)!=i.charCodeAt(s))return!1;return this.cursor-=t,!0},find_among:function(t,i){for(var s=0,e=i,n=this.cursor,u=this.limit,o=0,h=0,c=!1;;){for(var a=s+(e-s>>1),f=0,l=o<h?o:h,_=t[a],m=l;m<_.s_size;m++){if(n+l==u){f=-1;break}if(f=r.charCodeAt(n+l)-_.s[m])break;l++}if(f<0?(e=a,h=l):(s=a,o=l),e-s<=1){if(s>0||e==s||c)break;c=!0}}for(;;){var _=t[s];if(o>=_.s_size){if(this.cursor=n+_.s_size,!_.method)return _.result;var b=_.method();if(this.cursor=n+_.s_size,b)return _.result}if((s=_.substring_i)<0)return 0}},find_among_b:function(t,i){for(var s=0,e=i,n=this.cursor,u=this.limit_backward,o=0,h=0,c=!1;;){for(var a=s+(e-s>>1),f=0,l=o<h?o:h,_=t[a],m=_.s_size-1-l;m>=0;m--){if(n-l==u){f=-1;break}if(f=r.charCodeAt(n-1-l)-_.s[m])break;l++}if(f<0?(e=a,h=l):(s=a,o=l),e-s<=1){if(s>0||e==s||c)break;c=!0}}for(;;){var _=t[s];if(o>=_.s_size){if(this.cursor=n-_.s_size,!_.method)return _.result;var b=_.method();if(this.cursor=n-_.s_size,b)return _.result}if((s=_.substring_i)<0)return 0}},replace_s:function(t,i,s){var e=s.length-(i-t),n=r.substring(0,t),u=r.substring(i);return r=n+s+u,this.limit+=e,this.cursor>=i?this.cursor+=e:this.cursor>t&&(this.cursor=t),e},slice_check:function(){if(this.bra<0||this.bra>this.ket||this.ket>this.limit||this.limit>r.length)throw"faulty slice operation"},slice_from:function(r){this.slice_check(),this.replace_s(this.bra,this.ket,r)},slice_del:function(){this.slice_from("")},insert:function(r,t,i){var s=this.replace_s(r,t,i);r<=this.bra&&(this.bra+=s),r<=this.ket&&(this.ket+=s)},slice_to:function(){return this.slice_check(),r.substring(this.bra,this.ket)},eq_v_b:function(r){return this.eq_s_b(r.length,r)}}}},r.trimmerSupport={generateTrimmer:function(r){var t=new RegExp("^[^"+r+"]+"),i=new RegExp("[^"+r+"]+$");return function(r){return"function"==typeof r.update?r.update(function(r){return r.replace(t,"").replace(i,"")}):r.replace(t,"").replace(i,"")}}}}});
assets/javascripts/lunr/min/lunr.sv.min.js
Vendored
-18
View File
@@ -1,18 +0,0 @@
/*!
* Lunr languages, `Swedish` language
* https://github.com/MihaiValentin/lunr-languages
*
* Copyright 2014, Mihai Valentin
* http://www.mozilla.org/MPL/
*/
/*!
* based on
* Snowball JavaScript Library v0.3
* http://code.google.com/p/urim/
* http://snowball.tartarus.org/
*
* Copyright 2010, Oleg Mazko
* http://www.mozilla.org/MPL/
*/
!function(e,r){"function"==typeof define&&define.amd?define(r):"object"==typeof exports?module.exports=r():r()(e.lunr)}(this,function(){return function(e){if(void 0===e)throw new Error("Lunr is not present. Please include / require Lunr before this script.");if(void 0===e.stemmerSupport)throw new Error("Lunr stemmer support is not present. Please include / require Lunr stemmer support before this script.");e.sv=function(){this.pipeline.reset(),this.pipeline.add(e.sv.trimmer,e.sv.stopWordFilter,e.sv.stemmer),this.searchPipeline&&(this.searchPipeline.reset(),this.searchPipeline.add(e.sv.stemmer))},e.sv.wordCharacters="A-Za-zªºÀ-ÖØ-öø-ʸˠ-ˤᴀ-ᴥᴬ-ᵜᵢ-ᵥᵫ-ᵷᵹ-ᶾḀ-ỿⁱⁿₐ-ₜKÅℲⅎⅠ-ↈⱠ-ⱿꜢ-ꞇꞋ-ꞭꞰ-ꞷꟷ-ꟿꬰ-ꭚꭜ-ꭤff-stＡ-Ｚａ-ｚ",e.sv.trimmer=e.trimmerSupport.generateTrimmer(e.sv.wordCharacters),e.Pipeline.registerFunction(e.sv.trimmer,"trimmer-sv"),e.sv.stemmer=function(){var r=e.stemmerSupport.Among,n=e.stemmerSupport.SnowballProgram,t=new function(){function e(){var e,r=w.cursor+3;if(o=w.limit,0<=r||r<=w.limit){for(a=r;;){if(e=w.cursor,w.in_grouping(l,97,246)){w.cursor=e;break}if(w.cursor=e,w.cursor>=w.limit)return;w.cursor++}for(;!w.out_grouping(l,97,246);){if(w.cursor>=w.limit)return;w.cursor++}o=w.cursor,o<a&&(o=a)}}function t(){var e,r=w.limit_backward;if(w.cursor>=o&&(w.limit_backward=o,w.cursor=w.limit,w.ket=w.cursor,e=w.find_among_b(u,37),w.limit_backward=r,e))switch(w.bra=w.cursor,e){case 1:w.slice_del();break;case 2:w.in_grouping_b(d,98,121)&&w.slice_del()}}function i(){var e=w.limit_backward;w.cursor>=o&&(w.limit_backward=o,w.cursor=w.limit,w.find_among_b(c,7)&&(w.cursor=w.limit,w.ket=w.cursor,w.cursor>w.limit_backward&&(w.bra=--w.cursor,w.slice_del())),w.limit_backward=e)}function s(){var e,r;if(w.cursor>=o){if(r=w.limit_backward,w.limit_backward=o,w.cursor=w.limit,w.ket=w.cursor,e=w.find_among_b(m,5))switch(w.bra=w.cursor,e){case 1:w.slice_del();break;case 2:w.slice_from("lös");break;case 3:w.slice_from("full")}w.limit_backward=r}}var a,o,u=[new r("a",-1,1),new r("arna",0,1),new r("erna",0,1),new r("heterna",2,1),new r("orna",0,1),new r("ad",-1,1),new r("e",-1,1),new r("ade",6,1),new r("ande",6,1),new r("arne",6,1),new r("are",6,1),new r("aste",6,1),new r("en",-1,1),new r("anden",12,1),new r("aren",12,1),new r("heten",12,1),new r("ern",-1,1),new r("ar",-1,1),new r("er",-1,1),new r("heter",18,1),new r("or",-1,1),new r("s",-1,2),new r("as",21,1),new r("arnas",22,1),new r("ernas",22,1),new r("ornas",22,1),new r("es",21,1),new r("ades",26,1),new r("andes",26,1),new r("ens",21,1),new r("arens",29,1),new r("hetens",29,1),new r("erns",21,1),new r("at",-1,1),new r("andet",-1,1),new r("het",-1,1),new r("ast",-1,1)],c=[new r("dd",-1,-1),new r("gd",-1,-1),new r("nn",-1,-1),new r("dt",-1,-1),new r("gt",-1,-1),new r("kt",-1,-1),new r("tt",-1,-1)],m=[new r("ig",-1,1),new r("lig",0,1),new r("els",-1,1),new r("fullt",-1,3),new r("löst",-1,2)],l=[17,65,16,1,0,0,0,0,0,0,0,0,0,0,0,0,24,0,32],d=[119,127,149],w=new n;this.setCurrent=function(e){w.setCurrent(e)},this.getCurrent=function(){return w.getCurrent()},this.stem=function(){var r=w.cursor;return e(),w.limit_backward=r,w.cursor=w.limit,t(),w.cursor=w.limit,i(),w.cursor=w.limit,s(),!0}};return function(e){return"function"==typeof e.update?e.update(function(e){return t.setCurrent(e),t.stem(),t.getCurrent()}):(t.setCurrent(e),t.stem(),t.getCurrent())}}(),e.Pipeline.registerFunction(e.sv.stemmer,"stemmer-sv"),e.sv.stopWordFilter=e.generateStopWordFilter("alla allt att av blev bli blir blivit de dem den denna deras dess dessa det detta dig din dina ditt du där då efter ej eller en er era ert ett från för ha hade han hans har henne hennes hon honom hur här i icke ingen inom inte jag ju kan kunde man med mellan men mig min mina mitt mot mycket ni nu när någon något några och om oss på samma sedan sig sin sina sitta själv skulle som så sådan sådana sådant till under upp ut utan vad var vara varför varit varje vars vart vem vi vid vilka vilkas vilken vilket vår våra vårt än är åt över".split(" ")),e.Pipeline.registerFunction(e.sv.stopWordFilter,"stopWordFilter-sv")}});
assets/javascripts/lunr/min/lunr.ta.min.js
Vendored
-1
View File
@@ -1 +0,0 @@
!function(e,t){"function"==typeof define&&define.amd?define(t):"object"==typeof exports?module.exports=t():t()(e.lunr)}(this,function(){return function(e){if(void 0===e)throw new Error("Lunr is not present. Please include / require Lunr before this script.");if(void 0===e.stemmerSupport)throw new Error("Lunr stemmer support is not present. Please include / require Lunr stemmer support before this script.");e.ta=function(){this.pipeline.reset(),this.pipeline.add(e.ta.trimmer,e.ta.stopWordFilter,e.ta.stemmer),this.searchPipeline&&(this.searchPipeline.reset(),this.searchPipeline.add(e.ta.stemmer))},e.ta.wordCharacters="஀-உஊ-ஏஐ-ஙச-ட஠-னப-யர-ஹ஺-ிீ-௉ொ-௏ௐ-௙௚-௟௠-௩௪-௯௰-௹௺-௿a-zA-Zａ-ｚＡ-Ｚ0-9０-９",e.ta.trimmer=e.trimmerSupport.generateTrimmer(e.ta.wordCharacters),e.Pipeline.registerFunction(e.ta.trimmer,"trimmer-ta"),e.ta.stopWordFilter=e.generateStopWordFilter("அங்கு அங்கே அது அதை அந்த அவர் அவர்கள் அவள் அவன் அவை ஆக ஆகவே ஆகையால் ஆதலால் ஆதலினால் ஆனாலும் ஆனால் இங்கு இங்கே இது இதை இந்த இப்படி இவர் இவர்கள் இவள் இவன் இவை இவ்வளவு உனக்கு உனது உன் உன்னால் எங்கு எங்கே எது எதை எந்த எப்படி எவர் எவர்கள் எவள் எவன் எவை எவ்வளவு எனக்கு எனது எனவே என் என்ன என்னால் ஏது ஏன் தனது தன்னால் தானே தான் நாங்கள் நாம் நான் நீ நீங்கள்".split(" ")),e.ta.stemmer=function(){return function(e){return"function"==typeof e.update?e.update(function(e){return e}):e}}();var t=e.wordcut;t.init(),e.ta.tokenizer=function(r){if(!arguments.length||null==r||void 0==r)return[];if(Array.isArray(r))return r.map(function(t){return isLunr2?new e.Token(t.toLowerCase()):t.toLowerCase()});var i=r.toString().toLowerCase().replace(/^\s+/,"");return t.cut(i).split("|")},e.Pipeline.registerFunction(e.ta.stemmer,"stemmer-ta"),e.Pipeline.registerFunction(e.ta.stopWordFilter,"stopWordFilter-ta")}});
assets/javascripts/lunr/min/lunr.te.min.js
Vendored
-1
View File
@@ -1 +0,0 @@
!function(e,t){"function"==typeof define&&define.amd?define(t):"object"==typeof exports?module.exports=t():t()(e.lunr)}(this,function(){return function(e){if(void 0===e)throw new Error("Lunr is not present. Please include / require Lunr before this script.");if(void 0===e.stemmerSupport)throw new Error("Lunr stemmer support is not present. Please include / require Lunr stemmer support before this script.");e.te=function(){this.pipeline.reset(),this.pipeline.add(e.te.trimmer,e.te.stopWordFilter,e.te.stemmer),this.searchPipeline&&(this.searchPipeline.reset(),this.searchPipeline.add(e.te.stemmer))},e.te.wordCharacters="ఀ-ఄఅ-ఔక-హా-ౌౕ-ౖౘ-ౚౠ-ౡౢ-ౣ౦-౯౸-౿఼ఽ్ౝ౷౤౥",e.te.trimmer=e.trimmerSupport.generateTrimmer(e.te.wordCharacters),e.Pipeline.registerFunction(e.te.trimmer,"trimmer-te"),e.te.stopWordFilter=e.generateStopWordFilter("అందరూ అందుబాటులో అడగండి అడగడం అడ్డంగా అనుగుణంగా అనుమతించు అనుమతిస్తుంది అయితే ఇప్పటికే ఉన్నారు ఎక్కడైనా ఎప్పుడు ఎవరైనా ఎవరో ఏ ఏదైనా ఏమైనప్పటికి ఒక ఒకరు కనిపిస్తాయి కాదు కూడా గా గురించి చుట్టూ చేయగలిగింది తగిన తర్వాత దాదాపు దూరంగా నిజంగా పై ప్రకారం ప్రక్కన మధ్య మరియు మరొక మళ్ళీ మాత్రమే మెచ్చుకో వద్ద వెంట వేరుగా వ్యతిరేకంగా సంబంధం".split(" ")),e.te.stemmer=function(){return function(e){return"function"==typeof e.update?e.update(function(e){return e}):e}}();var t=e.wordcut;t.init(),e.te.tokenizer=function(r){if(!arguments.length||null==r||void 0==r)return[];if(Array.isArray(r))return r.map(function(t){return isLunr2?new e.Token(t.toLowerCase()):t.toLowerCase()});var i=r.toString().toLowerCase().replace(/^\s+/,"");return t.cut(i).split("|")},e.Pipeline.registerFunction(e.te.stemmer,"stemmer-te"),e.Pipeline.registerFunction(e.te.stopWordFilter,"stopWordFilter-te")}});
assets/javascripts/lunr/min/lunr.th.min.js
Vendored
-1
View File
@@ -1 +0,0 @@
!function(e,r){"function"==typeof define&&define.amd?define(r):"object"==typeof exports?module.exports=r():r()(e.lunr)}(this,function(){return function(e){if(void 0===e)throw new Error("Lunr is not present. Please include / require Lunr before this script.");if(void 0===e.stemmerSupport)throw new Error("Lunr stemmer support is not present. Please include / require Lunr stemmer support before this script.");var r="2"==e.version[0];e.th=function(){this.pipeline.reset(),this.pipeline.add(e.th.trimmer),r?this.tokenizer=e.th.tokenizer:(e.tokenizer&&(e.tokenizer=e.th.tokenizer),this.tokenizerFn&&(this.tokenizerFn=e.th.tokenizer))},e.th.wordCharacters="[฀-๿]",e.th.trimmer=e.trimmerSupport.generateTrimmer(e.th.wordCharacters),e.Pipeline.registerFunction(e.th.trimmer,"trimmer-th");var t=e.wordcut;t.init(),e.th.tokenizer=function(i){if(!arguments.length||null==i||void 0==i)return[];if(Array.isArray(i))return i.map(function(t){return r?new e.Token(t):t});var n=i.toString().replace(/^\s+/,"");return t.cut(n).split("|")}}});
assets/javascripts/lunr/min/lunr.tr.min.js
Vendored
-18
View File
File diff suppressed because one or more lines are too long
assets/javascripts/lunr/min/lunr.vi.min.js
Vendored
-1
View File
@@ -1 +0,0 @@
!function(e,r){"function"==typeof define&&define.amd?define(r):"object"==typeof exports?module.exports=r():r()(e.lunr)}(this,function(){return function(e){if(void 0===e)throw new Error("Lunr is not present. Please include / require Lunr before this script.");if(void 0===e.stemmerSupport)throw new Error("Lunr stemmer support is not present. Please include / require Lunr stemmer support before this script.");e.vi=function(){this.pipeline.reset(),this.pipeline.add(e.vi.stopWordFilter,e.vi.trimmer)},e.vi.wordCharacters="[A-Za-ẓ̀͐́͑̉̃̓ÂâÊêÔôĂ-ăĐ-đƠ-ơƯ-ư]",e.vi.trimmer=e.trimmerSupport.generateTrimmer(e.vi.wordCharacters),e.Pipeline.registerFunction(e.vi.trimmer,"trimmer-vi"),e.vi.stopWordFilter=e.generateStopWordFilter("là cái nhưng mà".split(" "))}});
assets/javascripts/lunr/min/lunr.zh.min.js
Vendored
-1
View File
@@ -1 +0,0 @@
!function(e,r){"function"==typeof define&&define.amd?define(r):"object"==typeof exports?module.exports=r(require("@node-rs/jieba")):r()(e.lunr)}(this,function(e){return function(r,t){if(void 0===r)throw new Error("Lunr is not present. Please include / require Lunr before this script.");if(void 0===r.stemmerSupport)throw new Error("Lunr stemmer support is not present. Please include / require Lunr stemmer support before this script.");var i="2"==r.version[0];r.zh=function(){this.pipeline.reset(),this.pipeline.add(r.zh.trimmer,r.zh.stopWordFilter,r.zh.stemmer),i?this.tokenizer=r.zh.tokenizer:(r.tokenizer&&(r.tokenizer=r.zh.tokenizer),this.tokenizerFn&&(this.tokenizerFn=r.zh.tokenizer))},r.zh.tokenizer=function(n){if(!arguments.length||null==n||void 0==n)return[];if(Array.isArray(n))return n.map(function(e){return i?new r.Token(e.toLowerCase()):e.toLowerCase()});t&&e.load(t);var o=n.toString().trim().toLowerCase(),s=[];e.cut(o,!0).forEach(function(e){s=s.concat(e.split(" "))}),s=s.filter(function(e){return!!e});var u=0;return s.map(function(e,t){if(i){var n=o.indexOf(e,u),s={};return s.position=[n,e.length],s.index=t,u=n,new r.Token(e,s)}return e})},r.zh.wordCharacters="\\w一-龥",r.zh.trimmer=r.trimmerSupport.generateTrimmer(r.zh.wordCharacters),r.Pipeline.registerFunction(r.zh.trimmer,"trimmer-zh"),r.zh.stemmer=function(){return function(e){return e}}(),r.Pipeline.registerFunction(r.zh.stemmer,"stemmer-zh"),r.zh.stopWordFilter=r.generateStopWordFilter("的 一 不 在 人 有 是 为 為 以 于 於 上 他 而 后 後 之 来 來 及 了 因 下 可 到 由 这 這 与 與 也 此 但 并 並 个 個 其 已 无 無 小 我 们 們 起 最 再 今 去 好 只 又 或 很 亦 某 把 那 你 乃 它 吧 被 比 别 趁 当 當 从 從 得 打 凡 儿 兒 尔 爾 该 該 各 给 給 跟 和 何 还 還 即 几 幾 既 看 据 據 距 靠 啦 另 么 麽 每 嘛 拿 哪 您 凭 憑 且 却 卻 让 讓 仍 啥 如 若 使 谁 誰 虽 雖 随 隨 同 所 她 哇 嗡 往 些 向 沿 哟 喲 用 咱 则 則 怎 曾 至 致 着 著 诸 諸 自".split(" ")),r.Pipeline.registerFunction(r.zh.stopWordFilter,"stopWordFilter-zh")}});
assets/javascripts/lunr/tinyseg.js
-206
View File
@@ -1,206 +0,0 @@
/**
* export the module via AMD, CommonJS or as a browser global
* Export code from https://github.com/umdjs/umd/blob/master/returnExports.js
*/
;(function (root, factory) {
if (typeof define === 'function' && define.amd) {
// AMD. Register as an anonymous module.
define(factory)
} else if (typeof exports === 'object') {
/**
* Node. Does not work with strict CommonJS, but
* only CommonJS-like environments that support module.exports,
* like Node.
*/
module.exports = factory()
} else {
// Browser globals (root is window)
factory()(root.lunr);
}
}(this, function () {
/**
* Just return a value to define the module export.
* This example returns an object, but the module
* can return a function as the exported value.
*/
return function(lunr) {
// TinySegmenter 0.1 -- Super compact Japanese tokenizer in Javascript
// (c) 2008 Taku Kudo <taku@chasen.org>
// TinySegmenter is freely distributable under the terms of a new BSD licence.
// For details, see http://chasen.org/~taku/software/TinySegmenter/LICENCE.txt
function TinySegmenter() {
var patterns = {
"[一二三四五六七八九十百千万億兆]":"M",
"[一-龠々〆ヵヶ]":"H",
"[ぁ-ん]":"I",
"[ァ-ヴーｱ-ﾝﾞｰ]":"K",
"[a-zA-Zａ-ｚＡ-Ｚ]":"A",
"[0-9０-９]":"N"
}
this.chartype_ = [];
for (var i in patterns) {
var regexp = new RegExp(i);
this.chartype_.push([regexp, patterns[i]]);
}
this.BIAS__ = -332
this.BC1__ = {"HH":6,"II":2461,"KH":406,"OH":-1378};
this.BC2__ = {"AA":-3267,"AI":2744,"AN":-878,"HH":-4070,"HM":-1711,"HN":4012,"HO":3761,"IA":1327,"IH":-1184,"II":-1332,"IK":1721,"IO":5492,"KI":3831,"KK":-8741,"MH":-3132,"MK":3334,"OO":-2920};
this.BC3__ = {"HH":996,"HI":626,"HK":-721,"HN":-1307,"HO":-836,"IH":-301,"KK":2762,"MK":1079,"MM":4034,"OA":-1652,"OH":266};
this.BP1__ = {"BB":295,"OB":304,"OO":-125,"UB":352};
this.BP2__ = {"BO":60,"OO":-1762};
this.BQ1__ = {"BHH":1150,"BHM":1521,"BII":-1158,"BIM":886,"BMH":1208,"BNH":449,"BOH":-91,"BOO":-2597,"OHI":451,"OIH":-296,"OKA":1851,"OKH":-1020,"OKK":904,"OOO":2965};
this.BQ2__ = {"BHH":118,"BHI":-1159,"BHM":466,"BIH":-919,"BKK":-1720,"BKO":864,"OHH":-1139,"OHM":-181,"OIH":153,"UHI":-1146};
this.BQ3__ = {"BHH":-792,"BHI":2664,"BII":-299,"BKI":419,"BMH":937,"BMM":8335,"BNN":998,"BOH":775,"OHH":2174,"OHM":439,"OII":280,"OKH":1798,"OKI":-793,"OKO":-2242,"OMH":-2402,"OOO":11699};
this.BQ4__ = {"BHH":-3895,"BIH":3761,"BII":-4654,"BIK":1348,"BKK":-1806,"BMI":-3385,"BOO":-12396,"OAH":926,"OHH":266,"OHK":-2036,"ONN":-973};
this.BW1__ = {",と":660,",同":727,"B1あ":1404,"B1同":542,"、と":660,"、同":727,"」と":1682,"あっ":1505,"いう":1743,"いっ":-2055,"いる":672,"うし":-4817,"うん":665,"から":3472,"がら":600,"こう":-790,"こと":2083,"こん":-1262,"さら":-4143,"さん":4573,"した":2641,"して":1104,"すで":-3399,"そこ":1977,"それ":-871,"たち":1122,"ため":601,"った":3463,"つい":-802,"てい":805,"てき":1249,"でき":1127,"です":3445,"では":844,"とい":-4915,"とみ":1922,"どこ":3887,"ない":5713,"なっ":3015,"など":7379,"なん":-1113,"にし":2468,"には":1498,"にも":1671,"に対":-912,"の一":-501,"の中":741,"ませ":2448,"まで":1711,"まま":2600,"まる":-2155,"やむ":-1947,"よっ":-2565,"れた":2369,"れで":-913,"をし":1860,"を見":731,"亡く":-1886,"京都":2558,"取り":-2784,"大き":-2604,"大阪":1497,"平方":-2314,"引き":-1336,"日本":-195,"本当":-2423,"毎日":-2113,"目指":-724,"Ｂ１あ":1404,"Ｂ１同":542,"｣と":1682};
this.BW2__ = {"..":-11822,"11":-669,"――":-5730,"−−":-13175,"いう":-1609,"うか":2490,"かし":-1350,"かも":-602,"から":-7194,"かれ":4612,"がい":853,"がら":-3198,"きた":1941,"くな":-1597,"こと":-8392,"この":-4193,"させ":4533,"され":13168,"さん":-3977,"しい":-1819,"しか":-545,"した":5078,"して":972,"しな":939,"その":-3744,"たい":-1253,"たた":-662,"ただ":-3857,"たち":-786,"たと":1224,"たは":-939,"った":4589,"って":1647,"っと":-2094,"てい":6144,"てき":3640,"てく":2551,"ては":-3110,"ても":-3065,"でい":2666,"でき":-1528,"でし":-3828,"です":-4761,"でも":-4203,"とい":1890,"とこ":-1746,"とと":-2279,"との":720,"とみ":5168,"とも":-3941,"ない":-2488,"なが":-1313,"など":-6509,"なの":2614,"なん":3099,"にお":-1615,"にし":2748,"にな":2454,"によ":-7236,"に対":-14943,"に従":-4688,"に関":-11388,"のか":2093,"ので":-7059,"のに":-6041,"のの":-6125,"はい":1073,"はが":-1033,"はず":-2532,"ばれ":1813,"まし":-1316,"まで":-6621,"まれ":5409,"めて":-3153,"もい":2230,"もの":-10713,"らか":-944,"らし":-1611,"らに":-1897,"りし":651,"りま":1620,"れた":4270,"れて":849,"れば":4114,"ろう":6067,"われ":7901,"を通":-11877,"んだ":728,"んな":-4115,"一人":602,"一方":-1375,"一日":970,"一部":-1051,"上が":-4479,"会社":-1116,"出て":2163,"分の":-7758,"同党":970,"同日":-913,"大阪":-2471,"委員":-1250,"少な":-1050,"年度":-8669,"年間":-1626,"府県":-2363,"手権":-1982,"新聞":-4066,"日新":-722,"日本":-7068,"日米":3372,"曜日":-601,"朝鮮":-2355,"本人":-2697,"東京":-1543,"然と":-1384,"社会":-1276,"立て":-990,"第に":-1612,"米国":-4268,"１１":-669};
this.BW3__ = {"あた":-2194,"あり":719,"ある":3846,"い.":-1185,"い。":-1185,"いい":5308,"いえ":2079,"いく":3029,"いた":2056,"いっ":1883,"いる":5600,"いわ":1527,"うち":1117,"うと":4798,"えと":1454,"か.":2857,"か。":2857,"かけ":-743,"かっ":-4098,"かに":-669,"から":6520,"かり":-2670,"が,":1816,"が、":1816,"がき":-4855,"がけ":-1127,"がっ":-913,"がら":-4977,"がり":-2064,"きた":1645,"けど":1374,"こと":7397,"この":1542,"ころ":-2757,"さい":-714,"さを":976,"し,":1557,"し、":1557,"しい":-3714,"した":3562,"して":1449,"しな":2608,"しま":1200,"す.":-1310,"す。":-1310,"する":6521,"ず,":3426,"ず、":3426,"ずに":841,"そう":428,"た.":8875,"た。":8875,"たい":-594,"たの":812,"たり":-1183,"たる":-853,"だ.":4098,"だ。":4098,"だっ":1004,"った":-4748,"って":300,"てい":6240,"てお":855,"ても":302,"です":1437,"でに":-1482,"では":2295,"とう":-1387,"とし":2266,"との":541,"とも":-3543,"どう":4664,"ない":1796,"なく":-903,"など":2135,"に,":-1021,"に、":-1021,"にし":1771,"にな":1906,"には":2644,"の,":-724,"の、":-724,"の子":-1000,"は,":1337,"は、":1337,"べき":2181,"まし":1113,"ます":6943,"まっ":-1549,"まで":6154,"まれ":-793,"らし":1479,"られ":6820,"るる":3818,"れ,":854,"れ、":854,"れた":1850,"れて":1375,"れば":-3246,"れる":1091,"われ":-605,"んだ":606,"んで":798,"カ月":990,"会議":860,"入り":1232,"大会":2217,"始め":1681,"市":965,"新聞":-5055,"日,":974,"日、":974,"社会":2024,"ｶ月":990};
this.TC1__ = {"AAA":1093,"HHH":1029,"HHM":580,"HII":998,"HOH":-390,"HOM":-331,"IHI":1169,"IOH":-142,"IOI":-1015,"IOM":467,"MMH":187,"OOI":-1832};
this.TC2__ = {"HHO":2088,"HII":-1023,"HMM":-1154,"IHI":-1965,"KKH":703,"OII":-2649};
this.TC3__ = {"AAA":-294,"HHH":346,"HHI":-341,"HII":-1088,"HIK":731,"HOH":-1486,"IHH":128,"IHI":-3041,"IHO":-1935,"IIH":-825,"IIM":-1035,"IOI":-542,"KHH":-1216,"KKA":491,"KKH":-1217,"KOK":-1009,"MHH":-2694,"MHM":-457,"MHO":123,"MMH":-471,"NNH":-1689,"NNO":662,"OHO":-3393};
this.TC4__ = {"HHH":-203,"HHI":1344,"HHK":365,"HHM":-122,"HHN":182,"HHO":669,"HIH":804,"HII":679,"HOH":446,"IHH":695,"IHO":-2324,"IIH":321,"III":1497,"IIO":656,"IOO":54,"KAK":4845,"KKA":3386,"KKK":3065,"MHH":-405,"MHI":201,"MMH":-241,"MMM":661,"MOM":841};
this.TQ1__ = {"BHHH":-227,"BHHI":316,"BHIH":-132,"BIHH":60,"BIII":1595,"BNHH":-744,"BOHH":225,"BOOO":-908,"OAKK":482,"OHHH":281,"OHIH":249,"OIHI":200,"OIIH":-68};
this.TQ2__ = {"BIHH":-1401,"BIII":-1033,"BKAK":-543,"BOOO":-5591};
this.TQ3__ = {"BHHH":478,"BHHM":-1073,"BHIH":222,"BHII":-504,"BIIH":-116,"BIII":-105,"BMHI":-863,"BMHM":-464,"BOMH":620,"OHHH":346,"OHHI":1729,"OHII":997,"OHMH":481,"OIHH":623,"OIIH":1344,"OKAK":2792,"OKHH":587,"OKKA":679,"OOHH":110,"OOII":-685};
this.TQ4__ = {"BHHH":-721,"BHHM":-3604,"BHII":-966,"BIIH":-607,"BIII":-2181,"OAAA":-2763,"OAKK":180,"OHHH":-294,"OHHI":2446,"OHHO":480,"OHIH":-1573,"OIHH":1935,"OIHI":-493,"OIIH":626,"OIII":-4007,"OKAK":-8156};
this.TW1__ = {"につい":-4681,"東京都":2026};
this.TW2__ = {"ある程":-2049,"いった":-1256,"ころが":-2434,"しょう":3873,"その後":-4430,"だって":-1049,"ていた":1833,"として":-4657,"ともに":-4517,"もので":1882,"一気に":-792,"初めて":-1512,"同時に":-8097,"大きな":-1255,"対して":-2721,"社会党":-3216};
this.TW3__ = {"いただ":-1734,"してい":1314,"として":-4314,"につい":-5483,"にとっ":-5989,"に当た":-6247,"ので,":-727,"ので、":-727,"のもの":-600,"れから":-3752,"十二月":-2287};
this.TW4__ = {"いう.":8576,"いう。":8576,"からな":-2348,"してい":2958,"たが,":1516,"たが、":1516,"ている":1538,"という":1349,"ました":5543,"ません":1097,"ようと":-4258,"よると":5865};
this.UC1__ = {"A":484,"K":93,"M":645,"O":-505};
this.UC2__ = {"A":819,"H":1059,"I":409,"M":3987,"N":5775,"O":646};
this.UC3__ = {"A":-1370,"I":2311};
this.UC4__ = {"A":-2643,"H":1809,"I":-1032,"K":-3450,"M":3565,"N":3876,"O":6646};
this.UC5__ = {"H":313,"I":-1238,"K":-799,"M":539,"O":-831};
this.UC6__ = {"H":-506,"I":-253,"K":87,"M":247,"O":-387};
this.UP1__ = {"O":-214};
this.UP2__ = {"B":69,"O":935};
this.UP3__ = {"B":189};
this.UQ1__ = {"BH":21,"BI":-12,"BK":-99,"BN":142,"BO":-56,"OH":-95,"OI":477,"OK":410,"OO":-2422};
this.UQ2__ = {"BH":216,"BI":113,"OK":1759};
this.UQ3__ = {"BA":-479,"BH":42,"BI":1913,"BK":-7198,"BM":3160,"BN":6427,"BO":14761,"OI":-827,"ON":-3212};
this.UW1__ = {",":156,"、":156,"「":-463,"あ":-941,"う":-127,"が":-553,"き":121,"こ":505,"で":-201,"と":-547,"ど":-123,"に":-789,"の":-185,"は":-847,"も":-466,"や":-470,"よ":182,"ら":-292,"り":208,"れ":169,"を":-446,"ん":-137,"・":-135,"主":-402,"京":-268,"区":-912,"午":871,"国":-460,"大":561,"委":729,"市":-411,"日":-141,"理":361,"生":-408,"県":-386,"都":-718,"｢":-463,"･":-135};
this.UW2__ = {",":-829,"、":-829,"〇":892,"「":-645,"」":3145,"あ":-538,"い":505,"う":134,"お":-502,"か":1454,"が":-856,"く":-412,"こ":1141,"さ":878,"ざ":540,"し":1529,"す":-675,"せ":300,"そ":-1011,"た":188,"だ":1837,"つ":-949,"て":-291,"で":-268,"と":-981,"ど":1273,"な":1063,"に":-1764,"の":130,"は":-409,"ひ":-1273,"べ":1261,"ま":600,"も":-1263,"や":-402,"よ":1639,"り":-579,"る":-694,"れ":571,"を":-2516,"ん":2095,"ア":-587,"カ":306,"キ":568,"ッ":831,"三":-758,"不":-2150,"世":-302,"中":-968,"主":-861,"事":492,"人":-123,"会":978,"保":362,"入":548,"初":-3025,"副":-1566,"北":-3414,"区":-422,"大":-1769,"天":-865,"太":-483,"子":-1519,"学":760,"実":1023,"小":-2009,"市":-813,"年":-1060,"強":1067,"手":-1519,"揺":-1033,"政":1522,"文":-1355,"新":-1682,"日":-1815,"明":-1462,"最":-630,"朝":-1843,"本":-1650,"東":-931,"果":-665,"次":-2378,"民":-180,"気":-1740,"理":752,"発":529,"目":-1584,"相":-242,"県":-1165,"立":-763,"第":810,"米":509,"自":-1353,"行":838,"西":-744,"見":-3874,"調":1010,"議":1198,"込":3041,"開":1758,"間":-1257,"｢":-645,"｣":3145,"ｯ":831,"ｱ":-587,"ｶ":306,"ｷ":568};
this.UW3__ = {",":4889,"1":-800,"−":-1723,"、":4889,"々":-2311,"〇":5827,"」":2670,"〓":-3573,"あ":-2696,"い":1006,"う":2342,"え":1983,"お":-4864,"か":-1163,"が":3271,"く":1004,"け":388,"げ":401,"こ":-3552,"ご":-3116,"さ":-1058,"し":-395,"す":584,"せ":3685,"そ":-5228,"た":842,"ち":-521,"っ":-1444,"つ":-1081,"て":6167,"で":2318,"と":1691,"ど":-899,"な":-2788,"に":2745,"の":4056,"は":4555,"ひ":-2171,"ふ":-1798,"へ":1199,"ほ":-5516,"ま":-4384,"み":-120,"め":1205,"も":2323,"や":-788,"よ":-202,"ら":727,"り":649,"る":5905,"れ":2773,"わ":-1207,"を":6620,"ん":-518,"ア":551,"グ":1319,"ス":874,"ッ":-1350,"ト":521,"ム":1109,"ル":1591,"ロ":2201,"ン":278,"・":-3794,"一":-1619,"下":-1759,"世":-2087,"両":3815,"中":653,"主":-758,"予":-1193,"二":974,"人":2742,"今":792,"他":1889,"以":-1368,"低":811,"何":4265,"作":-361,"保":-2439,"元":4858,"党":3593,"全":1574,"公":-3030,"六":755,"共":-1880,"円":5807,"再":3095,"分":457,"初":2475,"別":1129,"前":2286,"副":4437,"力":365,"動":-949,"務":-1872,"化":1327,"北":-1038,"区":4646,"千":-2309,"午":-783,"協":-1006,"口":483,"右":1233,"各":3588,"合":-241,"同":3906,"和":-837,"員":4513,"国":642,"型":1389,"場":1219,"外":-241,"妻":2016,"学":-1356,"安":-423,"実":-1008,"家":1078,"小":-513,"少":-3102,"州":1155,"市":3197,"平":-1804,"年":2416,"広":-1030,"府":1605,"度":1452,"建":-2352,"当":-3885,"得":1905,"思":-1291,"性":1822,"戸":-488,"指":-3973,"政":-2013,"教":-1479,"数":3222,"文":-1489,"新":1764,"日":2099,"旧":5792,"昨":-661,"時":-1248,"曜":-951,"最":-937,"月":4125,"期":360,"李":3094,"村":364,"東":-805,"核":5156,"森":2438,"業":484,"氏":2613,"民":-1694,"決":-1073,"法":1868,"海":-495,"無":979,"物":461,"特":-3850,"生":-273,"用":914,"町":1215,"的":7313,"直":-1835,"省":792,"県":6293,"知":-1528,"私":4231,"税":401,"立":-960,"第":1201,"米":7767,"系":3066,"約":3663,"級":1384,"統":-4229,"総":1163,"線":1255,"者":6457,"能":725,"自":-2869,"英":785,"見":1044,"調":-562,"財":-733,"費":1777,"車":1835,"軍":1375,"込":-1504,"通":-1136,"選":-681,"郎":1026,"郡":4404,"部":1200,"金":2163,"長":421,"開":-1432,"間":1302,"関":-1282,"雨":2009,"電":-1045,"非":2066,"駅":1620,"１":-800,"｣":2670,"･":-3794,"ｯ":-1350,"ｱ":551,"ｸﾞ":1319,"ｽ":874,"ﾄ":521,"ﾑ":1109,"ﾙ":1591,"ﾛ":2201,"ﾝ":278};
this.UW4__ = {",":3930,".":3508,"―":-4841,"、":3930,"。":3508,"〇":4999,"「":1895,"」":3798,"〓":-5156,"あ":4752,"い":-3435,"う":-640,"え":-2514,"お":2405,"か":530,"が":6006,"き":-4482,"ぎ":-3821,"く":-3788,"け":-4376,"げ":-4734,"こ":2255,"ご":1979,"さ":2864,"し":-843,"じ":-2506,"す":-731,"ず":1251,"せ":181,"そ":4091,"た":5034,"だ":5408,"ち":-3654,"っ":-5882,"つ":-1659,"て":3994,"で":7410,"と":4547,"な":5433,"に":6499,"ぬ":1853,"ね":1413,"の":7396,"は":8578,"ば":1940,"ひ":4249,"び":-4134,"ふ":1345,"へ":6665,"べ":-744,"ほ":1464,"ま":1051,"み":-2082,"む":-882,"め":-5046,"も":4169,"ゃ":-2666,"や":2795,"ょ":-1544,"よ":3351,"ら":-2922,"り":-9726,"る":-14896,"れ":-2613,"ろ":-4570,"わ":-1783,"を":13150,"ん":-2352,"カ":2145,"コ":1789,"セ":1287,"ッ":-724,"ト":-403,"メ":-1635,"ラ":-881,"リ":-541,"ル":-856,"ン":-3637,"・":-4371,"ー":-11870,"一":-2069,"中":2210,"予":782,"事":-190,"井":-1768,"人":1036,"以":544,"会":950,"体":-1286,"作":530,"側":4292,"先":601,"党":-2006,"共":-1212,"内":584,"円":788,"初":1347,"前":1623,"副":3879,"力":-302,"動":-740,"務":-2715,"化":776,"区":4517,"協":1013,"参":1555,"合":-1834,"和":-681,"員":-910,"器":-851,"回":1500,"国":-619,"園":-1200,"地":866,"場":-1410,"塁":-2094,"士":-1413,"多":1067,"大":571,"子":-4802,"学":-1397,"定":-1057,"寺":-809,"小":1910,"屋":-1328,"山":-1500,"島":-2056,"川":-2667,"市":2771,"年":374,"庁":-4556,"後":456,"性":553,"感":916,"所":-1566,"支":856,"改":787,"政":2182,"教":704,"文":522,"方":-856,"日":1798,"時":1829,"最":845,"月":-9066,"木":-485,"来":-442,"校":-360,"業":-1043,"氏":5388,"民":-2716,"気":-910,"沢":-939,"済":-543,"物":-735,"率":672,"球":-1267,"生":-1286,"産":-1101,"田":-2900,"町":1826,"的":2586,"目":922,"省":-3485,"県":2997,"空":-867,"立":-2112,"第":788,"米":2937,"系":786,"約":2171,"経":1146,"統":-1169,"総":940,"線":-994,"署":749,"者":2145,"能":-730,"般":-852,"行":-792,"規":792,"警":-1184,"議":-244,"谷":-1000,"賞":730,"車":-1481,"軍":1158,"輪":-1433,"込":-3370,"近":929,"道":-1291,"選":2596,"郎":-4866,"都":1192,"野":-1100,"銀":-2213,"長":357,"間":-2344,"院":-2297,"際":-2604,"電":-878,"領":-1659,"題":-792,"館":-1984,"首":1749,"高":2120,"｢":1895,"｣":3798,"･":-4371,"ｯ":-724,"ｰ":-11870,"ｶ":2145,"ｺ":1789,"ｾ":1287,"ﾄ":-403,"ﾒ":-1635,"ﾗ":-881,"ﾘ":-541,"ﾙ":-856,"ﾝ":-3637};
this.UW5__ = {",":465,".":-299,"1":-514,"E2":-32768,"]":-2762,"、":465,"。":-299,"「":363,"あ":1655,"い":331,"う":-503,"え":1199,"お":527,"か":647,"が":-421,"き":1624,"ぎ":1971,"く":312,"げ":-983,"さ":-1537,"し":-1371,"す":-852,"だ":-1186,"ち":1093,"っ":52,"つ":921,"て":-18,"で":-850,"と":-127,"ど":1682,"な":-787,"に":-1224,"の":-635,"は":-578,"べ":1001,"み":502,"め":865,"ゃ":3350,"ょ":854,"り":-208,"る":429,"れ":504,"わ":419,"を":-1264,"ん":327,"イ":241,"ル":451,"ン":-343,"中":-871,"京":722,"会":-1153,"党":-654,"務":3519,"区":-901,"告":848,"員":2104,"大":-1296,"学":-548,"定":1785,"嵐":-1304,"市":-2991,"席":921,"年":1763,"思":872,"所":-814,"挙":1618,"新":-1682,"日":218,"月":-4353,"査":932,"格":1356,"機":-1508,"氏":-1347,"田":240,"町":-3912,"的":-3149,"相":1319,"省":-1052,"県":-4003,"研":-997,"社":-278,"空":-813,"統":1955,"者":-2233,"表":663,"語":-1073,"議":1219,"選":-1018,"郎":-368,"長":786,"間":1191,"題":2368,"館":-689,"１":-514,"Ｅ２":-32768,"｢":363,"ｲ":241,"ﾙ":451,"ﾝ":-343};
this.UW6__ = {",":227,".":808,"1":-270,"E1":306,"、":227,"。":808,"あ":-307,"う":189,"か":241,"が":-73,"く":-121,"こ":-200,"じ":1782,"す":383,"た":-428,"っ":573,"て":-1014,"で":101,"と":-105,"な":-253,"に":-149,"の":-417,"は":-236,"も":-206,"り":187,"る":-135,"を":195,"ル":-673,"ン":-496,"一":-277,"中":201,"件":-800,"会":624,"前":302,"区":1792,"員":-1212,"委":798,"学":-960,"市":887,"広":-695,"後":535,"業":-697,"相":753,"社":-507,"福":974,"空":-822,"者":1811,"連":463,"郎":1082,"１":-270,"Ｅ１":306,"ﾙ":-673,"ﾝ":-496};
return this;
}
TinySegmenter.prototype.ctype_ = function(str) {
for (var i in this.chartype_) {
if (str.match(this.chartype_[i][0])) {
return this.chartype_[i][1];
}
}
return "O";
}
TinySegmenter.prototype.ts_ = function(v) {
if (v) { return v; }
return 0;
}
TinySegmenter.prototype.segment = function(input) {
if (input == null || input == undefined || input == "") {
return [];
}
var result = [];
var seg = ["B3","B2","B1"];
var ctype = ["O","O","O"];
var o = input.split("");
for (i = 0; i < o.length; ++i) {
seg.push(o[i]);
ctype.push(this.ctype_(o[i]))
}
seg.push("E1");
seg.push("E2");
seg.push("E3");
ctype.push("O");
ctype.push("O");
ctype.push("O");
var word = seg[3];
var p1 = "U";
var p2 = "U";
var p3 = "U";
for (var i = 4; i < seg.length - 3; ++i) {
var score = this.BIAS__;
var w1 = seg[i-3];
var w2 = seg[i-2];
var w3 = seg[i-1];
var w4 = seg[i];
var w5 = seg[i+1];
var w6 = seg[i+2];
var c1 = ctype[i-3];
var c2 = ctype[i-2];
var c3 = ctype[i-1];
var c4 = ctype[i];
var c5 = ctype[i+1];
var c6 = ctype[i+2];
score += this.ts_(this.UP1__[p1]);
score += this.ts_(this.UP2__[p2]);
score += this.ts_(this.UP3__[p3]);
score += this.ts_(this.BP1__[p1 + p2]);
score += this.ts_(this.BP2__[p2 + p3]);
score += this.ts_(this.UW1__[w1]);
score += this.ts_(this.UW2__[w2]);
score += this.ts_(this.UW3__[w3]);
score += this.ts_(this.UW4__[w4]);
score += this.ts_(this.UW5__[w5]);
score += this.ts_(this.UW6__[w6]);
score += this.ts_(this.BW1__[w2 + w3]);
score += this.ts_(this.BW2__[w3 + w4]);
score += this.ts_(this.BW3__[w4 + w5]);
score += this.ts_(this.TW1__[w1 + w2 + w3]);
score += this.ts_(this.TW2__[w2 + w3 + w4]);
score += this.ts_(this.TW3__[w3 + w4 + w5]);
score += this.ts_(this.TW4__[w4 + w5 + w6]);
score += this.ts_(this.UC1__[c1]);
score += this.ts_(this.UC2__[c2]);
score += this.ts_(this.UC3__[c3]);
score += this.ts_(this.UC4__[c4]);
score += this.ts_(this.UC5__[c5]);
score += this.ts_(this.UC6__[c6]);
score += this.ts_(this.BC1__[c2 + c3]);
score += this.ts_(this.BC2__[c3 + c4]);
score += this.ts_(this.BC3__[c4 + c5]);
score += this.ts_(this.TC1__[c1 + c2 + c3]);
score += this.ts_(this.TC2__[c2 + c3 + c4]);
score += this.ts_(this.TC3__[c3 + c4 + c5]);
score += this.ts_(this.TC4__[c4 + c5 + c6]);
// score += this.ts_(this.TC5__[c4 + c5 + c6]);
score += this.ts_(this.UQ1__[p1 + c1]);
score += this.ts_(this.UQ2__[p2 + c2]);
score += this.ts_(this.UQ3__[p3 + c3]);
score += this.ts_(this.BQ1__[p2 + c2 + c3]);
score += this.ts_(this.BQ2__[p2 + c3 + c4]);
score += this.ts_(this.BQ3__[p3 + c2 + c3]);
score += this.ts_(this.BQ4__[p3 + c3 + c4]);
score += this.ts_(this.TQ1__[p2 + c1 + c2 + c3]);
score += this.ts_(this.TQ2__[p2 + c2 + c3 + c4]);
score += this.ts_(this.TQ3__[p3 + c1 + c2 + c3]);
score += this.ts_(this.TQ4__[p3 + c2 + c3 + c4]);
var p = "O";
if (score > 0) {
result.push(word);
word = "";
p = "B";
}
p1 = p2;
p2 = p3;
p3 = p;
word += seg[i];
}
result.push(word);
return result;
}
lunr.TinySegmenter = TinySegmenter;
};
}));
assets/javascripts/lunr/wordcut.js
-6708
View File
File diff suppressed because one or more lines are too long
assets/javascripts/workers/search.2c215733.min.js
Vendored
-42
View File
File diff suppressed because one or more lines are too long
assets/javascripts/workers/search.2c215733.min.js.map
-7
View File
File diff suppressed because one or more lines are too long
assets/stylesheets/main.484c7ddc.min.css
Vendored
-1
View File
File diff suppressed because one or more lines are too long
assets/stylesheets/main.484c7ddc.min.css.map
-1
View File
File diff suppressed because one or more lines are too long
assets/stylesheets/palette.ab4e12ef.min.css
Vendored
-1
View File
File diff suppressed because one or more lines are too long
assets/stylesheets/palette.ab4e12ef.min.css.map
-1
View File
@@ -1 +0,0 @@
{"version":3,"sources":["src/templates/assets/stylesheets/palette/_scheme.scss","../../../../src/templates/assets/stylesheets/palette.scss","src/templates/assets/stylesheets/palette/_accent.scss","src/templates/assets/stylesheets/palette/_primary.scss","src/templates/assets/stylesheets/utilities/_break.scss"],"names":[],"mappings":"AA2BA,cAGE,6BAME,sDAAA,CACA,6DAAA,CACA,+DAAA,CACA,gEAAA,CACA,mDAAA,CACA,6DAAA,CACA,+DAAA,CACA,gEAAA,CAGA,mDAAA,CACA,gDAAA,CACA,yDAAA,CACA,4DAAA,CAGA,0BAAA,CACA,mCAAA,CAGA,iCAAA,CACA,kCAAA,CACA,mCAAA,CACA,mCAAA,CACA,kCAAA,CACA,iCAAA,CACA,+CAAA,CACA,6DAAA,CACA,gEAAA,CACA,4DAAA,CACA,4DAAA,CACA,6DAAA,CAGA,6CAAA,CAGA,+CAAA,CAGA,uDAAA,CACA,6DAAA,CACA,2DAAA,CAGA,iCAAA,CAGA,yDAAA,CACA,iEAAA,CAGA,mDAAA,CACA,mDAAA,CAGA,qDAAA,CACA,uDAAA,CAGA,8DAAA,CAKA,8DAAA,CAKA,0DAAA,CAzEA,iBCiBF,CD6DE,kHAEE,YC3DJ,CDkFE,yDACE,4BChFJ,CD+EE,2DACE,4BC7EJ,CD4EE,gEACE,4BC1EJ,CDyEE,2DACE,4BCvEJ,CDsEE,yDACE,4BCpEJ,CDmEE,0DACE,4BCjEJ,CDgEE,gEACE,4BC9DJ,CD6DE,0DACE,4BC3DJ,CD0DE,2OACE,4BC/CJ,CDsDA,+FAGE,iCCpDF,CACF,CCjDE,2BACE,4BAAA,CACA,2CAAA,CAOE,yBAAA,CACA,qCD6CN,CCvDE,4BACE,4BAAA,CACA,2CAAA,CAOE,yBAAA,CACA,qCDoDN,CC9DE,8BACE,4BAAA,CACA,2CAAA,CAOE,yBAAA,CACA,qCD2DN,CCrEE,mCACE,4BAAA,CACA,2CAAA,CAOE,yBAAA,CACA,qCDkEN,CC5EE,8BACE,4BAAA,CACA,2CAAA,CAOE,yBAAA,CACA,qCDyEN,CCnFE,4BACE,4BAAA,CACA,2CAAA,CAOE,yBAAA,CACA,qCDgFN,CC1FE,kCACE,4BAAA,CACA,2CAAA,CAOE,yBAAA,CACA,qCDuFN,CCjGE,4BACE,4BAAA,CACA,2CAAA,CAOE,yBAAA,CACA,qCD8FN,CCxGE,4BACE,4BAAA,CACA,2CAAA,CAOE,yBAAA,CACA,qCDqGN,CC/GE,6BACE,4BAAA,CACA,2CAAA,CAOE,yBAAA,CACA,qCD4GN,CCtHE,mCACE,4BAAA,CACA,2CAAA,CAOE,yBAAA,CACA,qCDmHN,CC7HE,4BACE,4BAAA,CACA,2CAAA,CAIE,8BAAA,CACA,qCD6HN,CCpIE,8BACE,4BAAA,CACA,2CAAA,CAIE,8BAAA,CACA,qCDoIN,CC3IE,6BACE,yBAAA,CACA,2CAAA,CAIE,8BAAA,CACA,qCD2IN,CClJE,8BACE,4BAAA,CACA,2CAAA,CAIE,8BAAA,CACA,qCDkJN,CCzJE,mCACE,4BAAA,CACA,2CAAA,CAOE,yBAAA,CACA,qCDsJN,CE3JE,4BACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCFwJN,CEnKE,6BACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCFgKN,CE3KE,+BACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCFwKN,CEnLE,oCACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCFgLN,CE3LE,+BACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCFwLN,CEnME,6BACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCFgMN,CE3ME,mCACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCFwMN,CEnNE,6BACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCFgNN,CE3NE,6BACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCFwNN,CEnOE,8BACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCFgON,CE3OE,oCACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCFwON,CEnPE,6BACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAIE,+BAAA,CACA,sCFmPN,CE3PE,+BACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAIE,+BAAA,CACA,sCF2PN,CEnQE,8BACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAIE,+BAAA,CACA,sCFmQN,CE3QE,+BACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAIE,+BAAA,CACA,sCF2QN,CEnRE,oCACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCFgRN,CE3RE,8BACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCFwRN,CEnSE,6BACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCAAA,CAKA,4BF4RN,CE5SE,kCACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCAAA,CAKA,4BFqSN,CEtRE,sEACE,4BFyRJ,CE1RE,+DACE,4BF6RJ,CE9RE,iEACE,4BFiSJ,CElSE,gEACE,4BFqSJ,CEtSE,iEACE,4BFySJ,CEhSA,8BACE,mDAAA,CACA,4DAAA,CACA,0DAAA,CACA,oDAAA,CACA,2DAAA,CAGA,4BFiSF,CE9RE,yCACE,+BFgSJ,CE7RI,kDAEE,0CAAA,CACA,sCAAA,CAFA,mCFiSN,CG7MI,mCD1EA,+CACE,8CF0RJ,CEvRI,qDACE,8CFyRN,CEpRE,iEACE,mCFsRJ,CACF,CGxNI,sCDvDA,uCACE,oCFkRJ,CACF,CEzQA,8BACE,kDAAA,CACA,4DAAA,CACA,wDAAA,CACA,oDAAA,CACA,6DAAA,CAGA,4BF0QF,CEvQE,yCACE,+BFyQJ,CEtQI,kDAEE,0CAAA,CACA,sCAAA,CAFA,mCF0QN,CEnQE,yCACE,6CFqQJ,CG9NI,0CDhCA,8CACE,gDFiQJ,CACF,CGnOI,0CDvBA,iFACE,6CF6PJ,CACF,CG3PI,sCDKA,uCACE,6CFyPJ,CACF","file":"palette.css"}
benchmarks/CMakeLists.txt
+14
View File
@@ -0,0 +1,14 @@
cmake_minimum_required(VERSION 3.21)
add_executable(bench_pipeline bench_pipeline.cpp)
target_link_libraries(bench_pipeline PRIVATE kpn)
target_compile_options(bench_pipeline PRIVATE -O3 -march=native)
find_package(TBB QUIET)
if(TBB_FOUND)
target_link_libraries(bench_pipeline PRIVATE TBB::tbb)
target_compile_definitions(bench_pipeline PRIVATE KPN_BENCH_TBB=1)
message(STATUS "TBB found — enabling TBB benchmarks")
else()
message(STATUS "TBB not found — TBB benchmarks disabled")
endif()
benchmarks/bench_pipeline.cpp
+526
View File
@@ -0,0 +1,526 @@
// Throughput benchmark: items/second vs. graph topology and size.
//
// Topologies:
// chain — linear depth D: push → n[0..D-1] → pop
// wide — fanout<W>: push → fanout → W parallel nodes → W pops
// diamond — push → fanout<2> → 2×2 nodes → 2 pops
//
// Two scheduling modes for each topology:
// private — each node owns a private ThreadPool(1) [Node<>]
// pool — all nodes share one ThreadPool(T) [PoolNode<> + shared pool]
//
// Usage: ./bench_pipeline | tee results.csv
#include <kpn/kpn.hpp>
#ifdef KPN_BENCH_TBB
#include <oneapi/tbb/flow_graph.h>
namespace tbb_flow = oneapi::tbb::flow;
#endif
#include <array>
#include <atomic>
#include <chrono>
#include <cstdio>
#include <memory>
#include <string>
#include <thread>
#include <vector>
using namespace kpn;
using namespace std::chrono_literals;
using sclock = std::chrono::steady_clock;
// ── configurable work ─────────────────────────────────────────────────────────
static std::atomic<int> g_work_us{0};
static int chain_fn(int x) {
int us = g_work_us.load(std::memory_order_relaxed);
if (us > 0) {
auto end = sclock::now() + std::chrono::microseconds(us);
while (sclock::now() < end);
}
return x;
}
using ChainNode = Node<chain_fn, in<>, out<>>;
using PoolChainNode = PoolNode<chain_fn, in<>, out<>>;
// ── push helper: yield-spin on overflow (no artificial sleep latency) ─────────
static void push_retry(Channel<int>& ch, int val) {
while (true) {
try { ch.push(val); return; }
catch (const ChannelOverflowError&) { std::this_thread::yield(); }
catch (const ChannelClosedError&) { return; }
}
}
// ── result ────────────────────────────────────────────────────────────────────
struct Result {
const char* topology;
int size;
int work_us;
int threads; // 0 = private (1 thread per node), N = shared pool size
double items_per_sec;
double overhead_us;
};
// ── chain ─────────────────────────────────────────────────────────────────────
static int items_for(int work_us, int depth = 1) {
int effective = std::max(1, work_us) * std::max(1, depth);
if (effective <= 1) return 5000;
if (effective <= 10) return 3000;
if (effective <= 100) return 1000;
if (effective <= 1000) return 200;
return 50;
}
static Result bench_chain(int depth, int work_us) {
const int N = items_for(work_us, depth);
const int CAP = N;
std::vector<std::shared_ptr<Channel<int>>> chs;
for (int i = 0; i <= depth; ++i)
chs.push_back(std::make_shared<Channel<int>>(CAP));
std::vector<std::unique_ptr<ChainNode>> nodes;
for (int i = 0; i < depth; ++i) {
nodes.push_back(std::make_unique<ChainNode>(CAP));
nodes.back()->set_input_channel<0>(chs[i]);
nodes.back()->set_output_channel<0>(chs[i + 1].get());
}
for (auto& n : nodes) n->start();
std::atomic<sclock::time_point> t1;
std::thread reader([&] {
for (int i = 0; i < N; ++i) chs.back()->pop();
t1.store(sclock::now(), std::memory_order_release);
});
auto t0 = sclock::now();
std::thread pusher([&] {
for (int i = 0; i < N; ++i) push_retry(*chs[0], i);
});
pusher.join();
reader.join();
for (auto& n : nodes) n->stop();
double elapsed = std::chrono::duration<double>(
t1.load(std::memory_order_acquire) - t0).count();
// Subtract theoretical pipeline fill cost (depth-1)*W so that overhead
// reflects only framework latency, not the expected pipeline startup time.
double pipeline_us = static_cast<double>(work_us) * (N + depth - 1);
double wus = (elapsed * 1e6 - pipeline_us) / N;
return {"chain", depth, work_us, 0, N / elapsed, wus};
}
static Result bench_chain_pool(int depth, int work_us, int pool_threads) {
const int N = items_for(work_us, depth);
const int CAP = N;
auto pool = std::make_shared<ThreadPool>(pool_threads);
std::vector<std::shared_ptr<Channel<int>>> chs;
for (int i = 0; i <= depth; ++i)
chs.push_back(std::make_shared<Channel<int>>(CAP));
std::vector<std::unique_ptr<PoolChainNode>> nodes;
for (int i = 0; i < depth; ++i) {
nodes.push_back(std::make_unique<PoolChainNode>(pool, CAP));
nodes.back()->set_input_channel<0>(chs[i]);
nodes.back()->set_output_channel<0>(chs[i + 1].get());
}
pool->start();
for (auto& n : nodes) n->start();
std::atomic<sclock::time_point> t1;
std::thread reader([&] {
for (int i = 0; i < N; ++i) chs.back()->pop();
t1.store(sclock::now(), std::memory_order_release);
});
auto t0 = sclock::now();
std::thread pusher([&] {
for (int i = 0; i < N; ++i) push_retry(*chs[0], i);
});
pusher.join();
reader.join();
for (auto& n : nodes) n->stop();
pool->stop();
double elapsed = std::chrono::duration<double>(
t1.load(std::memory_order_acquire) - t0).count();
double pipeline_us = static_cast<double>(work_us) * (N + depth - 1);
double wus = (elapsed * 1e6 - pipeline_us) / N;
return {"chain", depth, work_us, pool_threads, N / elapsed, wus};
}
// ── wide (fanout<W>) ──────────────────────────────────────────────────────────
template<std::size_t W>
static Result bench_wide(int work_us) {
const int N = items_for(work_us);
const int CAP = N;
auto src_ch = std::make_shared<Channel<int>>(CAP);
auto fan = std::make_unique<FanoutNode<int, W>>(CAP);
fan->template set_input_channel<0>(src_ch);
std::array<std::unique_ptr<ChainNode>, W> nodes;
std::array<std::shared_ptr<Channel<int>>, W> sink_chs;
for (std::size_t i = 0; i < W; ++i) {
nodes[i] = std::make_unique<ChainNode>(CAP);
sink_chs[i] = std::make_shared<Channel<int>>(CAP);
nodes[i]->template set_output_channel<0>(sink_chs[i].get());
}
[&]<std::size_t... Is>(std::index_sequence<Is...>) {
(fan->template set_output_channel<Is>(
&nodes[Is]->template input_channel<0>()), ...);
}(std::make_index_sequence<W>{});
fan->start();
for (auto& n : nodes) n->start();
std::array<std::thread, W> readers;
std::atomic<sclock::time_point> t1;
std::atomic<int> readers_done{0};
for (std::size_t w = 0; w < W; ++w) {
readers[w] = std::thread([&, w] {
for (int i = 0; i < N; ++i) sink_chs[w]->pop();
if (readers_done.fetch_add(1, std::memory_order_acq_rel) + 1
== static_cast<int>(W))
t1.store(sclock::now(), std::memory_order_release);
});
}
auto t0 = sclock::now();
std::thread pusher([&] {
for (int i = 0; i < N; ++i) push_retry(*src_ch, i);
});
pusher.join();
for (auto& r : readers) r.join();
fan->stop();
for (auto& n : nodes) n->stop();
double elapsed = std::chrono::duration<double>(
t1.load(std::memory_order_acquire) - t0).count();
double wus = (elapsed * 1e6) / N - static_cast<double>(work_us);
return {"wide", static_cast<int>(W), work_us, 0, N / elapsed, wus};
}
template<std::size_t W>
static Result bench_wide_pool(int work_us, int pool_threads) {
const int N = items_for(work_us);
const int CAP = N;
auto pool = std::make_shared<ThreadPool>(pool_threads);
auto src_ch = std::make_shared<Channel<int>>(CAP);
auto fan = std::make_unique<FanoutNode<int, W>>(CAP);
fan->template set_input_channel<0>(src_ch);
std::array<std::unique_ptr<PoolChainNode>, W> nodes;
std::array<std::shared_ptr<Channel<int>>, W> sink_chs;
for (std::size_t i = 0; i < W; ++i) {
nodes[i] = std::make_unique<PoolChainNode>(pool, CAP);
sink_chs[i] = std::make_shared<Channel<int>>(CAP);
nodes[i]->template set_output_channel<0>(sink_chs[i].get());
}
[&]<std::size_t... Is>(std::index_sequence<Is...>) {
(fan->template set_output_channel<Is>(
&nodes[Is]->template input_channel<0>()), ...);
}(std::make_index_sequence<W>{});
fan->start();
pool->start();
for (auto& n : nodes) n->start();
std::array<std::thread, W> readers;
std::atomic<sclock::time_point> t1;
std::atomic<int> readers_done{0};
for (std::size_t w = 0; w < W; ++w) {
readers[w] = std::thread([&, w] {
for (int i = 0; i < N; ++i) sink_chs[w]->pop();
if (readers_done.fetch_add(1, std::memory_order_acq_rel) + 1
== static_cast<int>(W))
t1.store(sclock::now(), std::memory_order_release);
});
}
auto t0 = sclock::now();
std::thread pusher([&] {
for (int i = 0; i < N; ++i) push_retry(*src_ch, i);
});
pusher.join();
for (auto& r : readers) r.join();
fan->stop();
for (auto& n : nodes) n->stop();
pool->stop();
double elapsed = std::chrono::duration<double>(
t1.load(std::memory_order_acquire) - t0).count();
double wus = (elapsed * 1e6) / N - static_cast<double>(work_us);
return {"wide", static_cast<int>(W), work_us, pool_threads, N / elapsed, wus};
}
// ── diamond ───────────────────────────────────────────────────────────────────
static Result bench_diamond(int work_us) {
const int N = items_for(work_us, 2);
const int CAP = N;
auto src_ch = std::make_shared<Channel<int>>(CAP);
auto fan = std::make_unique<FanoutNode<int, 2>>(CAP);
fan->template set_input_channel<0>(src_ch);
auto nL = std::make_unique<ChainNode>(CAP);
auto nR = std::make_unique<ChainNode>(CAP);
auto nL2 = std::make_unique<ChainNode>(CAP);
auto nR2 = std::make_unique<ChainNode>(CAP);
auto chL = std::make_shared<Channel<int>>(CAP);
auto chR = std::make_shared<Channel<int>>(CAP);
auto snkL = std::make_shared<Channel<int>>(CAP);
auto snkR = std::make_shared<Channel<int>>(CAP);
fan->template set_output_channel<0>(&nL->template input_channel<0>());
fan->template set_output_channel<1>(&nR->template input_channel<0>());
nL->set_output_channel<0>(chL.get());
nR->set_output_channel<0>(chR.get());
nL2->set_input_channel<0>(chL);
nR2->set_input_channel<0>(chR);
nL2->set_output_channel<0>(snkL.get());
nR2->set_output_channel<0>(snkR.get());
fan->start(); nL->start(); nR->start(); nL2->start(); nR2->start();
std::atomic<sclock::time_point> t1;
std::atomic<int> done{0};
auto make_reader = [&](Channel<int>& ch) {
return std::thread([&] {
for (int i = 0; i < N; ++i) ch.pop();
if (done.fetch_add(1, std::memory_order_acq_rel) + 1 == 2)
t1.store(sclock::now(), std::memory_order_release);
});
};
auto rL = make_reader(*snkL);
auto rR = make_reader(*snkR);
auto t0 = sclock::now();
std::thread pusher([&] {
for (int i = 0; i < N; ++i) push_retry(*src_ch, i);
});
pusher.join(); rL.join(); rR.join();
fan->stop(); nL->stop(); nR->stop(); nL2->stop(); nR2->stop();
double elapsed = std::chrono::duration<double>(
t1.load(std::memory_order_acquire) - t0).count();
double wus = (elapsed * 1e6) / N - static_cast<double>(work_us);
return {"diamond", 4, work_us, 0, N / elapsed, wus};
}
static Result bench_diamond_pool(int work_us, int pool_threads) {
const int N = items_for(work_us, 2);
const int CAP = N;
auto pool = std::make_shared<ThreadPool>(pool_threads);
auto src_ch = std::make_shared<Channel<int>>(CAP);
auto fan = std::make_unique<FanoutNode<int, 2>>(CAP);
fan->template set_input_channel<0>(src_ch);
auto nL = std::make_unique<PoolChainNode>(pool, CAP);
auto nR = std::make_unique<PoolChainNode>(pool, CAP);
auto nL2 = std::make_unique<PoolChainNode>(pool, CAP);
auto nR2 = std::make_unique<PoolChainNode>(pool, CAP);
auto chL = std::make_shared<Channel<int>>(CAP);
auto chR = std::make_shared<Channel<int>>(CAP);
auto snkL = std::make_shared<Channel<int>>(CAP);
auto snkR = std::make_shared<Channel<int>>(CAP);
fan->template set_output_channel<0>(&nL->template input_channel<0>());
fan->template set_output_channel<1>(&nR->template input_channel<0>());
nL->set_output_channel<0>(chL.get());
nR->set_output_channel<0>(chR.get());
nL2->set_input_channel<0>(chL);
nR2->set_input_channel<0>(chR);
nL2->set_output_channel<0>(snkL.get());
nR2->set_output_channel<0>(snkR.get());
fan->start();
pool->start();
nL->start(); nR->start(); nL2->start(); nR2->start();
std::atomic<sclock::time_point> t1;
std::atomic<int> done{0};
auto make_reader = [&](Channel<int>& ch) {
return std::thread([&] {
for (int i = 0; i < N; ++i) ch.pop();
if (done.fetch_add(1, std::memory_order_acq_rel) + 1 == 2)
t1.store(sclock::now(), std::memory_order_release);
});
};
auto rL = make_reader(*snkL);
auto rR = make_reader(*snkR);
auto t0 = sclock::now();
std::thread pusher([&] {
for (int i = 0; i < N; ++i) push_retry(*src_ch, i);
});
pusher.join(); rL.join(); rR.join();
fan->stop();
nL->stop(); nR->stop(); nL2->stop(); nR2->stop();
pool->stop();
double elapsed = std::chrono::duration<double>(
t1.load(std::memory_order_acquire) - t0).count();
double wus = (elapsed * 1e6) / N - static_cast<double>(work_us);
return {"diamond", 4, work_us, pool_threads, N / elapsed, wus};
}
// ── TBB flow graph ────────────────────────────────────────────────────────────
#ifdef KPN_BENCH_TBB
static Result bench_chain_tbb(int depth, int work_us) {
const int N = items_for(work_us, depth);
tbb_flow::graph g;
using FN = tbb_flow::function_node<int, int>;
std::vector<std::unique_ptr<FN>> nodes;
nodes.reserve(depth);
for (int i = 0; i < depth; ++i)
nodes.push_back(std::make_unique<FN>(g, tbb_flow::serial,
[](int x) -> int { return chain_fn(x); }));
for (int i = 0; i + 1 < depth; ++i)
tbb_flow::make_edge(*nodes[i], *nodes[i + 1]);
auto t0 = sclock::now();
for (int i = 0; i < N; ++i) nodes[0]->try_put(i);
g.wait_for_all();
auto t1 = sclock::now();
double elapsed = std::chrono::duration<double>(t1 - t0).count();
double pipeline_us = static_cast<double>(work_us) * (N + depth - 1);
double wus = (elapsed * 1e6 - pipeline_us) / N;
return {"chain_tbb", depth, work_us, -1, N / elapsed, wus};
}
template<std::size_t W>
static Result bench_wide_tbb(int work_us) {
const int N = items_for(work_us);
tbb_flow::graph g;
tbb_flow::broadcast_node<int> fan(g);
using FN = tbb_flow::function_node<int, int>;
std::array<std::unique_ptr<FN>, W> nodes;
for (auto& n : nodes) {
n = std::make_unique<FN>(g, tbb_flow::serial,
[](int x) -> int { return chain_fn(x); });
tbb_flow::make_edge(fan, *n);
}
auto t0 = sclock::now();
for (int i = 0; i < N; ++i) fan.try_put(i);
g.wait_for_all();
auto t1 = sclock::now();
double elapsed = std::chrono::duration<double>(t1 - t0).count();
double wus = (elapsed * 1e6) / N - static_cast<double>(work_us);
return {"wide_tbb", static_cast<int>(W), work_us, -1, N / elapsed, wus};
}
static Result bench_diamond_tbb(int work_us) {
const int N = items_for(work_us, 2);
tbb_flow::graph g;
tbb_flow::broadcast_node<int> fan(g);
using FN = tbb_flow::function_node<int, int>;
auto fn = [](int x) -> int { return chain_fn(x); };
FN nL(g, tbb_flow::serial, fn), nR(g, tbb_flow::serial, fn);
FN nL2(g, tbb_flow::serial, fn), nR2(g, tbb_flow::serial, fn);
tbb_flow::make_edge(fan, nL); tbb_flow::make_edge(fan, nR);
tbb_flow::make_edge(nL, nL2); tbb_flow::make_edge(nR, nR2);
auto t0 = sclock::now();
for (int i = 0; i < N; ++i) fan.try_put(i);
g.wait_for_all();
auto t1 = sclock::now();
double elapsed = std::chrono::duration<double>(t1 - t0).count();
double wus = (elapsed * 1e6) / N - static_cast<double>(work_us);
return {"diamond_tbb", 4, work_us, -1, N / elapsed, wus};
}
#endif // KPN_BENCH_TBB
// ── main ──────────────────────────────────────────────────────────────────────
int main() {
const int work_amts[] = {10, 100, 1000};
const int pool_sizes[] = {1, 2, 4};
std::fprintf(stderr, "%-12s %-8s %-10s %-8s %-18s %-20s\n",
"topology", "size", "work_us", "threads", "items/sec", "overhead_us/item");
std::fprintf(stderr, "%s\n", std::string(78, '-').c_str());
std::printf("topology,size,work_us,threads,items_per_sec,overhead_us_per_item\n");
auto emit = [](const Result& r) {
std::string sched = r.threads < 0 ? "tbb"
: r.threads == 0 ? "priv"
: std::to_string(r.threads);
std::fprintf(stderr, "%-12s %-8d %-10d %-8s %-18.0f %-20.1f\n",
r.topology, r.size, r.work_us, sched.c_str(),
r.items_per_sec, r.overhead_us);
std::printf("%s,%d,%d,%s,%.0f,%.2f\n",
r.topology, r.size, r.work_us, sched.c_str(),
r.items_per_sec, r.overhead_us);
std::fflush(stdout);
};
for (int w : work_amts) {
g_work_us.store(w, std::memory_order_relaxed);
std::fprintf(stderr, "\n── work_us=%-4d private pools ───────────────────────────────────────\n", w);
for (int d : {1, 2, 4, 8, 16, 32}) emit(bench_chain(d, w));
emit(bench_wide<1>(w));
emit(bench_wide<2>(w));
emit(bench_wide<3>(w));
emit(bench_wide<4>(w));
emit(bench_diamond(w));
for (int pt : pool_sizes) {
std::fprintf(stderr, "\n── work_us=%-4d shared pool (%d thread%s) ─────────────────────────────\n",
w, pt, pt == 1 ? "" : "s");
for (int d : {1, 2, 4, 8, 16, 32}) emit(bench_chain_pool(d, w, pt));
emit(bench_wide_pool<1>(w, pt));
emit(bench_wide_pool<2>(w, pt));
emit(bench_wide_pool<3>(w, pt));
emit(bench_wide_pool<4>(w, pt));
emit(bench_diamond_pool(w, pt));
}
#ifdef KPN_BENCH_TBB
std::fprintf(stderr, "\n── work_us=%-4d TBB flow graph ──────────────────────────────────────\n", w);
for (int d : {1, 2, 4, 8, 16, 32}) emit(bench_chain_tbb(d, w));
emit(bench_wide_tbb<1>(w));
emit(bench_wide_tbb<2>(w));
emit(bench_wide_tbb<3>(w));
emit(bench_wide_tbb<4>(w));
emit(bench_diamond_tbb(w));
#endif
}
}
channels/index.html
-1073
View File
File diff suppressed because it is too large Load Diff
docs/channels.md
+55
View File
@@ -0,0 +1,55 @@
# Channels
A `Channel<T>` is a lock-free SPSC (single-producer, single-consumer) ring buffer with atomic wait/notify.
## Semantics
- **Bounded**: fixed capacity set at construction. Default is 5 items.
- **Backpressure**: when full, `push()` throws `ChannelOverflowError` immediately — no blocking, no spin.
- **Blocking consumer**: `pop()` blocks until an item is available or the channel is disabled.
- **Disable**: `channel.disable()` stops accepting pushes and unblocks any waiting `pop()` with `ChannelClosedError`.
## Storage policy
Small trivially-copyable types (≤ 8 bytes) are stored by value. Larger types are heap-allocated and passed via `shared_ptr<const T>` — one allocation per push, zero-copy fan-out:
```cpp
--8<-- "examples/04_storage_policy/main.cpp:storage_policy_spec"
```
Specialize `kpn::ChannelDataSize<T>` for accurate bandwidth reporting on heap-owning types:
```cpp
template<>
struct kpn::ChannelDataSize<cv::Mat> {
static std::size_t bytes(const cv::Mat& m) { return m.total() * m.elemSize(); }
};
```
## Named ports
`in<"name">` and `out<"name">` tag nodes for readable wiring:
```cpp
--8<-- "examples/02_named_ports/main.cpp:named_port_creation"
```
Named ports are checked at compile time — a typo in a port name is a compile error.
## Capacity tuning
Set capacity per node at construction:
```cpp
auto node = make_node<my_func>(/*capacity=*/20);
```
Capacity is rounded up internally to the next power of two. Monitor fill levels via diagnostics to tune for your workload — a too-small capacity causes overflows; a too-large one wastes memory and hides producer/consumer speed mismatches.
## Spin count
`Channel` spins for up to ~4 µs (200 `pause` hints at ~20 ns each on x86) before sleeping on a futex. Set to 0 for power-constrained or predominantly-idle pipelines:
```cpp
Channel<int> ch(/*capacity=*/5, /*spin_count=*/0);
```
docs/error-handling.md
+84
View File
@@ -0,0 +1,84 @@
# Error Handling & Events
KPN++ provides three complementary layers for observing and reacting to failures.
---
## 1. Per-node error handler
Called when a node's function throws an unhandled exception. Return `true` to skip the failed invocation and keep running; `false` to stop the node.
```cpp
--8<-- "examples/15_node_error_handler/main.cpp:error_handler"
```
When a node stops (either from `false` return or no handler installed), it:
1. Disables its **input** channels — upstream stops pushing into dead queues.
2. Disables its **output** channels — downstream nodes receive `ChannelClosedError` on their next pop, propagating the shutdown naturally through the graph.
---
## 2. Per-node overflow callback
Fired with a timestamp each time an output push is dropped because the channel is full. The node name is known at registration so it is not included — keeping the callback zero-overhead when unused.
```cpp
--8<-- "examples/16_event_callbacks/main.cpp:per_node_callback"
```
!!! note
The callback is purely informational — the node always continues after an overflow. To stop the node on overflow, call `node.stop()` from inside the callback.
A matching `set_closed_callback()` fires (also with just a timestamp) when the node stops due to a closed upstream channel:
```cpp
node.set_closed_callback([](std::chrono::steady_clock::time_point ts) {
std::cerr << "node stopped at t=" << ts.time_since_epoch().count() << '\n';
});
```
Each node holds two callback slots per event type — one user-set (registered above) and one injected by the network (see below). Both fire independently.
---
## 3. Network-level event handler
One callback for the whole network. Receives the node name (captured in a closure by the network at `build()` / `start()`), a `NodeEvent`, and a timestamp:
```cpp
--8<-- "examples/16_event_callbacks/main.cpp:network_event_handler"
```
`NodeEvent` values:
| Value | Meaning |
|---|---|
| `NodeEvent::Overflow` | An output push was dropped (channel full) |
| `NodeEvent::Closed` | The node stopped (crash or upstream close cascade) |
The network handler and any per-node callbacks are **independent** — both fire when set.
---
## Complete example
`examples/16_event_callbacks/main.cpp` shows a fast producer overflowing a slow consumer, with both a per-node overflow callback and a network-level event handler active simultaneously.
Node functions:
```cpp
--8<-- "examples/16_event_callbacks/main.cpp:node_fns"
```
Per-node overflow callback:
```cpp
--8<-- "examples/16_event_callbacks/main.cpp:per_node_callback"
```
Network-level event handler:
```cpp
--8<-- "examples/16_event_callbacks/main.cpp:network_event_handler"
```
docs/examples.md
+40
View File
@@ -0,0 +1,40 @@
# Examples
All C++ examples are built by default and registered as CTest smoke tests. Run them all with:
```bash
ctest --test-dir build -L examples
```
## Index
| Example | What it shows |
|---|---|
| `01_hello_pipeline` | Linear pipeline, index-based port wiring |
| `02_named_ports` | `in<>`/`out<>` name tags, named port access |
| `03_multi_output` | Tuple-returning node, per-element routing |
| `04_storage_policy` | `channel_storage_policy` specialisation |
| `05_error_handling` | Diagnostics handler, overflow channel stats |
| `06_watchdog` | Watchdog interval, stall detection |
| `10_static_hello_pipeline` | `StaticNetwork` + `make_network()` |
| `11_static_fanout` | `StaticNetwork` with `FanoutNode` |
| `15_node_error_handler` | `set_error_handler()` — skip or stop on exception |
| `16_event_callbacks` | `set_overflow_callback()`, `set_event_handler()` |
## OpenCV examples (optional)
Built only when OpenCV ≥ 4 is found:
| Example | What it shows |
|---|---|
| `09_opencv_cellshade` | Real-time cell-shading on webcam; `MainThreadNode` for display |
| `12_static_cellshade` | Same pipeline as a `StaticNetwork` |
| `13_debug_cellshade` | Web debug UI overlay on the cell-shading pipeline |
Run the cell-shading example:
```bash
./build/examples/09_opencv_cellshade
# Press 'q' or close the window to stop.
# Falls back to an animated synthetic pattern if no webcam is found.
```
docs/fanout.md
+44
View File
@@ -0,0 +1,44 @@
# Fan-out & Routing
## FanoutNode
Reads one item and pushes a copy to each of N output channels. All downstream nodes receive every item.
```cpp
auto fan = make_fanout<Image, 2>(/*capacity=*/8);
net.connect("src", src.output<0>(), "fan", fan.input<0>())
.connect("fan", fan.output<0>(), "nodeA", nodeA.input<0>())
.connect("fan", fan.output<1>(), "nodeB", nodeB.input<0>());
```
If one downstream channel overflows, that output drops the item independently — the other outputs are unaffected.
See `examples/11_static_fanout`.
## RouterNode
Reads one item and pushes it to exactly one of N outputs, chosen by a selector function:
```cpp
auto router = make_router<Frame, 3>(
[](const Frame& f) -> std::size_t { return f.stream_id % 3; });
net.connect("src", src.output<0>(), "router", router.input<0>())
.connect("router", router.output<0>(), "nodeA", nodeA.input<0>())
.connect("router", router.output<1>(), "nodeB", nodeB.input<0>())
.connect("router", router.output<2>(), "nodeC", nodeC.input<0>());
```
If the selector returns `>= N` the item is silently dropped.
## FilterNode
Reads one item and passes it downstream only when a predicate returns `true`:
```cpp
auto filt = make_filter<Frame>([](const Frame& f) { return f.valid; });
net.connect("src", src.output<0>(), "filt", filt.input<0>())
.connect("filt", filt.output<0>(), "dst", dst.input<0>());
```
docs/getting-started.md
+76
View File
@@ -0,0 +1,76 @@
# Getting Started
## Requirements
| Dependency | Version | Notes |
|---|---|---|
| CMake | ≥ 3.21 | |
| C++ compiler | GCC ≥ 11, Clang ≥ 13 | C++20 required |
| nanobind | ≥ 2.1 | auto-fetched; Python ≥ 3.8 |
| Catch2 | v3 | auto-fetched for tests |
| OpenCV | ≥ 4 | optional; only for examples 09/12/13 |
## Build
```bash
cmake -B build # core + tests + C++ examples
cmake --build build --parallel
ctest --test-dir build # run all tests including example smoke tests
```
Enable Python bindings:
```bash
cmake -B build -DKPN_BUILD_PYTHON=ON
cmake --build build --parallel
```
Skip examples:
```bash
cmake -B build -DKPN_BUILD_EXAMPLES=OFF
```
## Your first pipeline
Three functions — source, transform, sink — wired into a `Network`:
```cpp
--8<-- "examples/01_hello_pipeline/main.cpp:basic_node_fns"
```
Create nodes, connect them, build and run:
```cpp
--8<-- "examples/01_hello_pipeline/main.cpp:network_build"
```
That's it. Types are inferred from function signatures. The channel between `src` and `dbl` carries `int`; the channel between `dbl` and `prn` also carries `int`. A type mismatch is a compile error.
## Named ports
For nodes with multiple inputs or outputs, name the ports for clarity:
```cpp
--8<-- "examples/02_named_ports/main.cpp:named_port_creation"
```
Wire by name instead of index:
```cpp
--8<-- "examples/02_named_ports/main.cpp:named_port_network"
```
## Multi-output nodes
Return a `std::tuple` to fan out to multiple downstream nodes:
```cpp
--8<-- "examples/03_multi_output/main.cpp:multi_output_fn"
```
Wire each tuple element to its own downstream node:
```cpp
--8<-- "examples/03_multi_output/main.cpp:fanout_network"
```
docs/index.md
+39
View File
@@ -0,0 +1,39 @@
# KPN++
A C++20 [Kahn Process Network](https://en.wikipedia.org/wiki/Kahn_process_networks) library. Each node wraps a plain function and runs concurrently, communicating with downstream nodes via bounded FIFO channels. Includes Python bindings via nanobind.
---
## Why KPN++?
- **Zero boilerplate** — wrap any callable as a node; types flow automatically from the function signature
- **Bounded channels** — backpressure is structural, not bolted on
- **Observable** — per-node and network-level callbacks for overflow and stop events; diagnostics snapshots; optional web UI
- **Composable** — `Network` for runtime wiring, `StaticNetwork` for compile-time topology with zero overhead
---
## Quick example
```cpp
#include <kpn/kpn.hpp>
using namespace kpn;
--8<-- "examples/01_hello_pipeline/main.cpp:basic_node_fns"
int main() {
--8<-- "examples/01_hello_pipeline/main.cpp:network_build"
}
```
---
## Install & build
```bash
cmake -B build
cmake --build build --parallel
ctest --test-dir build # unit tests + example smoke tests
```
See [Getting Started](getting-started.md) for full build options.
docs/network.md
+67
View File
@@ -0,0 +1,67 @@
# Networks
A `Network` wires nodes together at runtime using a builder chain.
## Building a network
```cpp
--8<-- "examples/01_hello_pipeline/main.cpp:network_build"
```
The builder chain:
| Method | Purpose |
|---|---|
| `.add(name, node)` | Register a node; assigns its name |
| `.connect(src, port, dst, port)` | Wire one output port to one input port |
| `.build()` | Compute topological order; inject network callbacks |
| `.start()` | Start nodes in topological order |
| `.stop()` | Stop all nodes immediately |
| `.shutdown()` | Graceful drain: stop sources first, wait for channels to empty, then stop downstream |
## Port access
Ports are accessed by index or by name:
```cpp
// By index
net.connect("src", src.output<0>(), "dst", dst.input<0>());
// By name (requires named ports)
--8<-- "examples/02_named_ports/main.cpp:named_port_network"
```
## Diagnostics
Install a diagnostics handler to receive periodic snapshots of every node and channel:
```cpp
--8<-- "examples/05_error_handling/main.cpp:diagnostics_handler"
```
Or print a full report at any time:
```cpp
net.print_diagnostics(); // writes to stderr by default
net.print_diagnostics(std::cout);
```
## Network-level event handler
Observe overflow and node-stop events across the entire network in one place:
```cpp
--8<-- "examples/16_event_callbacks/main.cpp:network_event_handler"
```
`NodeEvent` is either `NodeEvent::Overflow` (item dropped on full channel) or `NodeEvent::Closed` (node stopped due to crash or closed upstream channel). See [Error Handling & Events](error-handling.md).
## Shutdown
`net.stop()` halts immediately — all nodes stop in reverse topological order.
`net.shutdown()` drains gracefully: source nodes stop first; their output channels are polled until empty; then the next layer stops, and so on. This ensures no items are lost if downstream nodes are still consuming.
## StaticNetwork
For zero-overhead compile-time topology, see [Static Networks](static-network.md).
docs/nodes.md
+81
View File
@@ -0,0 +1,81 @@
# Nodes
A node wraps any callable. Its input types are inferred from the function's parameter list; its output types from the return type.
## Node types
| Type | Thread model | Use case |
|---|---|---|
| `Node<Func>` | Dedicated thread per node | Default — simplest, most isolated |
| `PoolNode<Func>` | Shared `ThreadPool` | Many nodes, resource-bounded execution |
| `InterruptNode<Func>` | Event-driven, no thread | Camera frame ready, timer tick, socket |
| `FanoutNode<T, N>` | Dedicated thread | Broadcast one item to N outputs |
| `RouterNode<T, N>` | Dedicated thread | Route one item to one of N outputs |
| `FilterNode<T>` | Dedicated thread | Pass items matching a predicate |
## Creating nodes
All node types are created via factory functions that infer types from the callable:
```cpp
// Free function — simplest case
auto node = make_node<my_func>();
// Stateful functor (operator() is the function)
MyProcessor proc;
auto node = make_node(proc);
// Pool node — shares a ThreadPool with other nodes
auto pool = std::make_shared<ThreadPool>(4);
auto node = make_pool_node<my_func>(pool);
// Interrupt node — triggered externally
auto sched = std::make_shared<ThreadPool>(2);
auto node = make_interrupt_node<produce_frame>(sched, out<"frame">{});
camera_sdk.on_frame_ready(node.get_trigger());
```
## Channel capacity
Each node's input FIFO has a configurable capacity (default 5):
```cpp
auto node = make_node<my_func>(/*capacity=*/20);
auto node = make_pool_node<my_func>(pool, /*capacity=*/20);
```
When an upstream push would exceed capacity, `ChannelOverflowError` is thrown and the item is dropped. See [Error Handling & Events](error-handling.md) to observe and react to this.
## Source nodes
A node with no inputs is a source. It self-submits immediately on `start()` and re-submits after each execution:
```cpp
static int produce() {
std::this_thread::sleep_for(std::chrono::milliseconds(10));
return ++counter;
}
auto src = make_node<produce>();
```
!!! tip
Source nodes must sleep or yield to avoid overflowing their output channel. The channel capacity provides the only bound.
## Sink nodes
A node with a `void` return is a sink — it consumes items without producing output:
```cpp
static void print_it(int x) { std::cout << x << '\n'; }
auto snk = make_node<print_it>();
```
## Error handler
When a node's function throws an unhandled exception, the default behaviour is to stop the node (disabling its channels so the shutdown cascades downstream). Install a handler to override:
```cpp
--8<-- "examples/15_node_error_handler/main.cpp:error_handler"
```
See [Error Handling & Events](error-handling.md) for the full picture.
requirements.txt → docs/requirements.txt
View File
docs/shared-resource.md
+42
View File
@@ -0,0 +1,42 @@
# Shared Resources
`SharedResource<T>` arbitrates exclusive access to a resource (ONNX session, CUDA stream, serial port) across multiple nodes using a priority-based waiter queue with starvation prevention.
## Usage
```cpp
#include <kpn/shared_resource.hpp>
using namespace kpn;
SharedResource<OnnxSession> model(session_args...);
static cv::Mat run_inference(cv::Mat frame) {
// Acquires the model; releases automatically on scope exit.
auto guard = model.acquire_balanced(in_channel, out_channel);
return guard->Run(frame);
}
```
## Acquire modes
| Method | Priority |
|---|---|
| `acquire()` | Equal (fair FIFO) |
| `acquire(fn)` | Custom — `fn()` returns `float` in `[0, 1]` |
| `acquire_balanced(in_ch, out_ch)` | `input_fill × output_headroom` — highest urgency wins |
`acquire_balanced` favours nodes with full input queues and empty output queues — the node that has the most work to do and nowhere to stall wins the resource next.
## Starvation prevention
Each waiter's effective score grows with elapsed wait time (`0.05` per second by default), ensuring a low-priority node eventually gets served regardless of how frequently higher-priority nodes compete.
## Diagnostics
Register with the network for snapshot reporting:
```cpp
net.register_resource("model", &model);
```
The diagnostics table then shows acquisition count, mean wait time, and current waiter count.
docs/static-network.md
+46
View File
@@ -0,0 +1,46 @@
# Static Networks
`StaticNetwork` encodes the entire topology at compile time using a `make_network()` builder. Nodes and channel types are verified statically with zero runtime overhead.
## Usage
```cpp
#include <kpn/kpn.hpp>
using namespace kpn;
static int produce() { return 42; }
static int double_it(int x) { return x * 2; }
static void print_it(int x) { std::cout << x << '\n'; }
int main() {
auto src = make_node<produce> ();
auto dbl = make_node<double_it>();
auto prn = make_node<print_it> ();
auto net = make_network(
edge(src, src.output<0>(), dbl, dbl.input<0>()),
edge(dbl, dbl.output<0>(), prn, prn.input<0>())
);
net.set_event_handler([](std::string_view name, NodeEvent ev, auto ts) {
// same API as Network
});
net.start();
std::this_thread::sleep_for(std::chrono::milliseconds(100));
net.stop();
}
```
See `examples/10_static_hello_pipeline` and `examples/11_static_fanout`.
## When to use
| | `Network` | `StaticNetwork` |
|---|---|---|
| Topology known at | Runtime | Compile time |
| Type checking | Runtime (`dynamic_cast`) | Compile time |
| Overhead | Minimal | Zero |
| Flexibility | Add nodes dynamically | Fixed at compile time |
For most applications `Network` is sufficient. Use `StaticNetwork` when you need the absolute minimum overhead or want compile-time topology verification.
error-handling/index.html
-1107
View File
File diff suppressed because it is too large Load Diff
examples/01_hello_pipeline/main.cpp
+44
View File
@@ -0,0 +1,44 @@
#include <kpn/kpn.hpp>
#include <iostream>
#include <chrono>
#include <thread>
// A minimal linear pipeline: producer → double → print
//
// [produce] --int--> [double_it] --int--> [print_it]
// --8<-- [start:basic_node_fns]
static int produce() { return 42; }
static int double_it(int x) { return x * 2; }
static void print_it(int x) { std::cout << "result: " << x << '\n'; }
// --8<-- [end:basic_node_fns]
int main() {
using namespace kpn;
// --8<-- [start:index_only_nodes]
auto src = make_node<produce>(5);
auto dbl = make_node<double_it>(5);
auto sink = make_node<print_it>(5);
// --8<-- [end:index_only_nodes]
// Wire channels
auto& dbl_in = dbl.input_channel<0>();
auto& sink_in = sink.input_channel<0>();
src.set_output_channel<0>(&dbl_in);
dbl.set_output_channel<0>(&sink_in);
// --8<-- [start:network_build]
Network net;
net.add("src", src)
.add("dbl", dbl)
.add("sink", sink)
.connect("src", src.output<0>(), "dbl", dbl.input<0>())
.connect("dbl", dbl.output<0>(), "sink", sink.input<0>())
.build();
net.start();
std::this_thread::sleep_for(std::chrono::milliseconds(100));
net.stop();
// --8<-- [end:network_build]
}
examples/02_named_ports/main.cpp
+82
View File
@@ -0,0 +1,82 @@
// Example 02 — Named Ports
//
// A three-stage text pipeline where port names carry semantic meaning:
//
// [tokenise] --"words"--> [count_words] --"count"--> [report]
// --"words"--> [report]
//
// Named ports let the connect() call read like documentation:
// connect("tok", tok.output<"words">(), "cnt", cnt.input<"words">())
// A typo in the name is a compile-time error, not a runtime surprise.
#include <kpn/kpn.hpp>
#include <chrono>
#include <iostream>
#include <sstream>
#include <string>
#include <thread>
#include <tuple>
#include <vector>
// ── Node functions ────────────────────────────────────────────────────────────
static int sentence_index = 0;
static std::vector<std::string> tokenise() {
static const char* sentences[] = {
"the quick brown fox jumps over the lazy dog",
"kahn process networks are a model of concurrent computation",
"each node runs in its own thread communicating via channels",
"named ports catch wiring mistakes at compile time",
};
std::istringstream ss(sentences[sentence_index++ % 4]);
std::vector<std::string> words;
std::string w;
while (ss >> w) words.push_back(w);
std::this_thread::sleep_for(std::chrono::milliseconds(50));
return words;
}
// Returns (word_count, original_words) — two outputs via tuple
static std::tuple<int, std::vector<std::string>>
count_words(std::vector<std::string> words) {
return {static_cast<int>(words.size()), std::move(words)};
}
static void report(int count, std::vector<std::string> words) {
std::cout << "[" << count << " words] ";
for (auto& w : words) std::cout << w << ' ';
std::cout << '\n';
}
// ── main ──────────────────────────────────────────────────────────────────────
int main() {
using namespace kpn;
// --8<-- [start:named_port_creation]
// tokenise: no inputs, one named output "words"
auto tok = make_node<tokenise>(out<"words">{}, 4);
// count_words: named input "words", named outputs "count" and "words"
auto cnt = make_node<count_words>(in<"words">{}, out<"count", "words">{}, 4);
// report: two named inputs
auto snk = make_node<report>(in<"count", "words">{}, 4);
// --8<-- [end:named_port_creation]
// --8<-- [start:named_port_network]
Network net;
net.add("tok", tok)
.add("cnt", cnt)
.add("snk", snk)
.connect("tok", tok.template output<"words">(), "cnt", cnt.template input<"words">())
.connect("cnt", cnt.template output<"count">(), "snk", snk.template input<"count">())
.connect("cnt", cnt.template output<"words">(), "snk", snk.template input<"words">())
.build();
net.start();
std::this_thread::sleep_for(std::chrono::milliseconds(500));
net.stop();
// --8<-- [end:named_port_network]
}
examples/03_multi_output/main.cpp
+79
View File
@@ -0,0 +1,79 @@
// Example 03 — Multi-Output (Fan-Out)
//
// A single "parse" node reads "KEY=VALUE" strings and fans out to two
// independent downstream sinks — one for keys, one for values.
//
// +--> [print_key]
// [generate] --string--> [parse]
// +--> [print_value]
//
// The parser returns std::tuple<std::string, std::string>.
// Network::connect() routes each element of the tuple to a different node.
#include <kpn/kpn.hpp>
#include <chrono>
#include <iostream>
#include <string>
#include <thread>
#include <tuple>
// ── Node functions ────────────────────────────────────────────────────────────
static int gen_index = 0;
static std::string generate() {
static const char* pairs[] = {
"host=localhost",
"port=8080",
"timeout=30s",
"retries=3",
"protocol=http2",
};
std::this_thread::sleep_for(std::chrono::milliseconds(60));
return pairs[gen_index++ % 5];
}
// --8<-- [start:multi_output_fn]
// Multi-output: returns (key, value) as a tuple — KPN++ routes each element
// to its own output port automatically.
static std::tuple<std::string, std::string> parse(std::string kv) {
auto sep = kv.find('=');
if (sep == std::string::npos) return {kv, ""};
return {kv.substr(0, sep), kv.substr(sep + 1)};
}
// --8<-- [end:multi_output_fn]
static void print_key(std::string key) {
std::cout << "KEY → " << key << '\n';
}
static void print_value(std::string value) {
std::cout << "VALUE → " << value << '\n';
}
// ── main ──────────────────────────────────────────────────────────────────────
int main() {
using namespace kpn;
// --8<-- [start:fanout_network]
auto gen = make_node<generate>(out<"kv">{}, 4);
auto par = make_node<parse> (in<"kv">{}, out<"key", "value">{}, 4);
auto keys = make_node<print_key> (in<"key">{}, 4);
auto vals = make_node<print_value>(in<"value">{}, 4);
Network net;
net.add("gen", gen)
.add("par", par)
.add("keys", keys)
.add("vals", vals)
.connect("gen", gen.template output<"kv">(), "par", par.template input<"kv">())
.connect("par", par.template output<"key">(), "keys", keys.template input<"key">())
.connect("par", par.template output<"value">(), "vals", vals.template input<"value">())
.build();
net.start();
std::this_thread::sleep_for(std::chrono::milliseconds(600));
net.stop();
// --8<-- [end:fanout_network]
}
examples/04_storage_policy/main.cpp
+120
View File
@@ -0,0 +1,120 @@
// Example 04 — Storage Policy
//
// Demonstrates how KPN++ chooses between by-value and shared_ptr storage
// inside channels depending on the type.
//
// Small trivially-copyable types (int, double, etc.) are stored by value.
// Large or non-trivially-copyable types are stored as shared_ptr<const T>
// — zero copies even across multiple downstream consumers.
//
// This example also shows how to override the policy for a specific type
// with a template specialisation.
//
// [produce_frame] --Frame--> [process_frame] --Frame--> [consume_frame]
// [produce_small] --int----> [double_it] --int----> [print_int]
#include <kpn/kpn.hpp>
#include <array>
#include <chrono>
#include <iostream>
#include <string>
#include <thread>
#include <type_traits>
// ── Types ─────────────────────────────────────────────────────────────────────
// Large frame type — will be stored as shared_ptr<const Frame> by default
struct Frame {
std::array<uint8_t, 4096> pixels{};
int id = 0;
};
// A small struct we force to be stored by value via policy specialisation
struct Tag {
int value = 0;
};
// --8<-- [start:storage_policy_spec]
// Override: store Tag by value despite being a struct
// (it's trivially copyable and small — this just makes the policy explicit)
template<>
struct kpn::channel_storage_policy<Tag> {
static constexpr bool by_value = true;
};
// --8<-- [end:storage_policy_spec]
// ── Node functions ────────────────────────────────────────────────────────────
static int frame_id = 0;
static int tag_id = 0;
static Frame produce_frame() {
std::this_thread::sleep_for(std::chrono::milliseconds(40));
Frame f;
f.id = frame_id++;
f.pixels.fill(static_cast<uint8_t>(f.id & 0xFF));
return f;
}
static Frame process_frame(Frame f) {
// Simulate some work
for (auto& p : f.pixels) p = static_cast<uint8_t>(255 - p);
return f;
}
static void consume_frame(Frame f) {
std::cout << "[frame] id=" << f.id
<< " first_px=" << static_cast<int>(f.pixels[0])
<< " storage=shared_ptr (sizeof Frame = " << sizeof(Frame) << " B)\n";
}
static Tag produce_tag() {
std::this_thread::sleep_for(std::chrono::milliseconds(40));
return {tag_id++};
}
static Tag double_tag(Tag t) { return {t.value * 2}; }
static void print_tag(Tag t) {
std::cout << "[tag] value=" << t.value
<< " storage=by_value (sizeof Tag = " << sizeof(Tag) << " B)\n";
}
// ── main ──────────────────────────────────────────────────────────────────────
int main() {
using namespace kpn;
// Verify storage decisions at compile time
static_assert(!channel_storage_policy<Frame>::by_value,
"Frame should use shared_ptr storage");
static_assert(channel_storage_policy<Tag>::by_value,
"Tag should use by_value storage (via specialisation)");
static_assert(channel_storage_policy<int>::by_value,
"int should use by_value storage");
auto prod_f = make_node<produce_frame> (out<"frame">{}, 4);
auto proc_f = make_node<process_frame> (in<"frame">{}, out<"frame">{}, 4);
auto cons_f = make_node<consume_frame> (in<"frame">{}, 4);
auto prod_t = make_node<produce_tag> (out<"tag">{}, 4);
auto dbl_t = make_node<double_tag> (in<"tag">{}, out<"tag">{}, 4);
auto print_t = make_node<print_tag> (in<"tag">{}, 4);
Network net;
net.add("prod_f", prod_f)
.add("proc_f", proc_f)
.add("cons_f", cons_f)
.add("prod_t", prod_t)
.add("dbl_t", dbl_t)
.add("print_t", print_t)
.connect("prod_f", prod_f.template output<"frame">(), "proc_f", proc_f.template input<"frame">())
.connect("proc_f", proc_f.template output<"frame">(), "cons_f", cons_f.template input<"frame">())
.connect("prod_t", prod_t.template output<"tag">(), "dbl_t", dbl_t.template input<"tag">())
.connect("dbl_t", dbl_t.template output<"tag">(), "print_t", print_t.template input<"tag">())
.build();
net.start();
std::this_thread::sleep_for(std::chrono::milliseconds(400));
net.stop();
}
examples/05_error_handling/main.cpp
+80
View File
@@ -0,0 +1,80 @@
// Example 05 — Error Handling & Diagnostics
//
// Demonstrates observable failure modes and the diagnostics system:
//
// 1. ChannelOverflowError — a fast producer saturates a slow consumer.
// When the channel is full, push() throws ChannelOverflowError.
// The node's run_loop catches it and prints to stderr.
// Channel statistics (overflows, peak fill) accumulate for the report.
//
// 2. Custom diagnostics handler — instead of the default periodic table,
// install a handler that surfaces only the metrics you care about.
//
// 3. net.print_diagnostics() — print a full report at any time.
//
// Pipeline: [producer] --int--> [slow_consumer]
#include <kpn/kpn.hpp>
#include <chrono>
#include <iostream>
#include <thread>
// ── Node functions ────────────────────────────────────────────────────────────
// Produces at ~100/s — faster than the consumer can keep up (50 ms each)
static int producer() {
std::this_thread::sleep_for(std::chrono::milliseconds(10));
static int n = 0;
return ++n;
}
// Slow consumer: 50 ms per item — will cause channel to fill and overflow
static void slow_consumer(int x) {
std::this_thread::sleep_for(std::chrono::milliseconds(50));
std::cout << "[consumed] " << x << '\n';
}
// ── main ──────────────────────────────────────────────────────────────────────
int main() {
using namespace kpn;
// Capacity=4: fills up quickly when producer outpaces consumer 5:1
auto prod = make_node<producer> (out<"v">{}, /*capacity=*/4);
auto cons = make_node<slow_consumer> (in<"v">{}, /*capacity=*/4);
Network net;
// --8<-- [start:diagnostics_handler]
// Custom diagnostics handler — fires on the watchdog interval.
// Print a concise one-liner rather than the full table.
net.set_diagnostics_handler([](const std::vector<NodeSnapshot>& nodes,
const std::vector<ChannelSnapshot>& channels) {
std::cout << "[diag] ";
for (auto& n : nodes)
std::cout << n.name << "=" << n.throughput_fps << "fps ";
for (auto& c : channels)
std::cout << "channel fill=" << static_cast<int>(c.fill_pct()) << "% "
<< "overflows=" << c.overflows;
std::cout << '\n';
});
// --8<-- [end:diagnostics_handler]
net.set_watchdog_interval(std::chrono::milliseconds(200));
net.add("prod", prod)
.add("cons", cons)
.connect("prod", prod.template output<"v">(), "cons", cons.template input<"v">())
.build();
std::cout << "producer: 10ms/item, consumer: 50ms/item, capacity=4\n"
<< "Overflow messages appear on stderr; diagnostics on stdout.\n\n";
net.start();
std::this_thread::sleep_for(std::chrono::milliseconds(800));
net.stop();
// Full report after shutdown — overflow and drop counts are preserved
std::cout << '\n';
net.print_diagnostics();
}
examples/06_watchdog/main.cpp
+90
View File
@@ -0,0 +1,90 @@
// Example 06 — Watchdog & Diagnostics (+ optional web debug UI)
//
// A four-stage pipeline with deliberately uneven node speeds:
//
// [source] --int--> [fast_filter] --int--> [slow_transform] --int--> [sink]
//
// The watchdog fires every second and prints a full diagnostics report:
// - frames processed, throughput fps, exec time, blocked time per node
// - channel fill %, peak fill %, pushes, overflows, bandwidth
// - bottleneck hint (node with highest avg exec time)
//
// "slow_transform" sleeps 30 ms per item, making it the obvious bottleneck.
// Watch the channel upstream of it saturate and the fps converge to ~33.
//
// Build with -DKPN_WEB_DEBUG=ON to also get a live D3 graph at localhost:9090.
#include <kpn/kpn.hpp>
#include <chrono>
#include <cmath>
#include <iostream>
#include <thread>
// ── Node functions ────────────────────────────────────────────────────────────
static int source() {
// Produces at ~50 fps — faster than slow_transform (33 fps) so the
// channel between filter and slow gradually fills, but not catastrophically
std::this_thread::sleep_for(std::chrono::milliseconds(20));
static int n = 0;
return ++n;
}
static int fast_filter(int x) {
// Trivial work: ~0.1 ms
return (x % 2 == 0) ? x : x + 1;
}
static int slow_transform(int x) {
// Simulates expensive processing (e.g. a neural net inference step)
std::this_thread::sleep_for(std::chrono::milliseconds(30));
return x * x;
}
static void sink(int x) {
// Print every 10th result to avoid flooding the terminal
if (x % 100 < 4)
std::cout << "[sink] " << x << '\n';
}
// ── main ──────────────────────────────────────────────────────────────────────
int main() {
using namespace kpn;
// Larger capacity buffers so the fast nodes don't immediately overflow
auto src = make_node<source> (out<"v">{}, 16);
auto filt = make_node<fast_filter> (in<"v">{}, out<"v">{}, 16);
auto slow = make_node<slow_transform> (in<"v">{}, out<"v">{}, 8);
auto snk = make_node<sink> (in<"v">{}, 8);
Network net;
// Watchdog fires every 1 second — prints the built-in diagnostics table
// including the "Bottleneck hint" line
net.set_watchdog_interval(std::chrono::milliseconds(1000));
net.add("source", src)
.add("filter", filt)
.add("slow", slow)
.add("sink", snk)
.connect("source", src.template output<"v">(), "filter", filt.template input<"v">())
.connect("filter", filt.template output<"v">(), "slow", slow.template input<"v">())
.connect("slow", slow.template output<"v">(), "sink", snk.template input<"v">())
.build();
std::cout << "Running for 30 seconds — watch 'slow' become the bottleneck.\n"
<< "Watchdog diagnostics print every 1 second.\n";
#ifdef KPN_WEB_DEBUG
net.set_web_debug_port(9090);
std::cout << "Web debug UI: http://localhost:9090 (live graph, auto-updates every 500 ms)\n";
#endif
std::cout << '\n';
net.start();
std::this_thread::sleep_for(std::chrono::seconds(30));
net.stop();
std::cout << "\n=== Final diagnostics ===\n";
net.print_diagnostics();
}
examples/07_python_network/example.py
+37
View File
@@ -0,0 +1,37 @@
"""
07_python_network — hello pipeline with a Python node in the middle.
Graph:
[ProduceNode] --int--> [py_double] --int--> [PrintItNode]
ProduceNode and PrintItNode are C++ nodes wrapped in VariantNodeWrapper.
py_double is a pure Python callable — doubles its input using Python arithmetic.
"""
import sys
import time
sys.path.insert(0, "build/python")
import kpn_python as kpn
def py_double(x: int) -> int:
return x * 2
net = kpn.Network()
net.add("src", kpn.make_produce())
net.add_node("dbl", py_double, inputs=["int"], outputs=["int"])
net.add("sink", kpn.make_print_it())
net.connect("src", 0, "dbl", 0)
net.connect("dbl", 0, "sink", 0)
net.build()
net.start()
time.sleep(0.1)
net.stop()
# Drop the network deterministically: it holds the Python callable, which forms
# a reference cycle via globals(). Deleting the global breaks it so the network
# is reclaimed now rather than lingering to interpreter shutdown.
del net
examples/08_python_subport/example.py
+55
View File
@@ -0,0 +1,55 @@
"""
08_python_subport — drive a *Python* node from Python via write()/read() taps.
Graph:
(fed by net.write()) --int--> [py_triple] --int--> (tapped by net.read())
Unlike 07, there is no C++ source or sink here: the only node in the network is
a pure-Python function, py_triple. Python plays *both* the producer and the
consumer by using the subport taps:
* net.write("py", 0, v) injects v into py_triple's input (Python -> network)
* net.read("py", 0) pulls py_triple's output back out (network -> Python)
This closes the loop the old version left as a "#todo": a value flows from
Python, through a Python node running inside the network, and back to Python.
"""
import sys
sys.path.insert(0, "build/python") # for `python examples/.../example.py` from repo root
import kpn_python as kpn
def py_triple(x: int) -> int:
return x * 3
net = kpn.Network()
# The whole network is a single Python node with a tapped input and output.
net.add_node("py", py_triple, inputs=["int"], outputs=["int"])
net.build()
net.start()
# Push values in from Python and read the Python node's results back out.
inputs = [1, 2, 7, 10, 100]
results = []
for v in inputs:
net.write("py", 0, v) # Python -> py_triple input
results.append(net.read("py", 0)) # py_triple output -> Python
net.stop()
print("inputs written from Python: ", inputs)
print("outputs read from py_triple:", results)
expected = [v * 3 for v in inputs]
assert results == expected, f"expected {expected}, got {results}"
print("all correct (x * 3 computed by a Python node inside the network)")
# Drop the network deterministically. The network holds the Python callable,
# which (via globals) forms a reference cycle; deleting the global breaks it so
# the network is reclaimed promptly instead of lingering to interpreter exit.
del net
examples/09_opencv_cellshade/bench_capture.cpp
+53
View File
@@ -0,0 +1,53 @@
#include <opencv2/videoio.hpp>
#include <chrono>
#include <algorithm>
#include <numeric>
#include <vector>
#include <cstdio>
static void bench(const char* name, int device, int backend, int W, int H, int frames) {
cv::VideoCapture cap(device, backend);
if (!cap.isOpened()) {
std::printf("%-12s failed to open /dev/video%d\n", name, device);
return;
}
cap.set(cv::CAP_PROP_FRAME_WIDTH, W);
cap.set(cv::CAP_PROP_FRAME_HEIGHT, H);
// Warm up
for (int i = 0; i < 5; ++i) cap.grab();
std::vector<double> times;
times.reserve(frames);
for (int i = 0; i < frames; ++i) {
auto t0 = std::chrono::steady_clock::now();
bool ok = cap.grab();
double ms = std::chrono::duration<double, std::milli>(
std::chrono::steady_clock::now() - t0).count();
if (ok) times.push_back(ms);
}
cap.release();
if (times.empty()) {
std::printf("%-12s no frames captured\n", name);
return;
}
std::sort(times.begin(), times.end());
double avg = std::accumulate(times.begin(), times.end(), 0.0) / times.size();
double mn = times.front();
double mx = times.back();
double p95 = times[times.size() * 95 / 100];
std::printf("%-12s avg=%6.1fms min=%5.1fms p95=%6.1fms max=%6.1fms (%zu frames)\n",
name, avg, mn, p95, mx, times.size());
}
int main(int argc, char** argv) {
int device = argc > 1 ? std::atoi(argv[1]) : 0;
int frames = argc > 2 ? std::atoi(argv[2]) : 60;
int W = 1920, H = 1080;
std::printf("Benchmarking /dev/video%d at %dx%d, %d frames each\n\n", device, W, H, frames);
bench("V4L2", device, cv::CAP_V4L2, W, H, frames);
bench("GStreamer", device, cv::CAP_GSTREAMER, W, H, frames);
return 0;
}
examples/09_opencv_cellshade/example_hybrid.py
+101
View File
@@ -0,0 +1,101 @@
"""
09_opencv_cellshade/example_hybrid.py
──────────────────────────────────────
Hybrid cell-shading pipeline: C++ nodes handle capture, grayscale conversion,
and edge detection; a Python/numpy function replaces the C++ quantise node;
Python drives the display loop using cv2.
Pipeline:
┌─[py_quantise]──────────────┐
[CaptureNode] ─────┤ ├──[CompositeNode]──result──▶ cv2.imshow
out0=colour └─[ToGrayNode]─[EdgesNode]───┘ edges───▶ cv2.imshow
out1=grey
For a pure-C++ version see main.cpp; for the C++ static-network version see
12_static_cellshade/main.cpp.
Press 'q' or Esc to stop.
"""
import sys
import os
# Adjust path to wherever CMake placed the .so
BUILD_DIR = os.environ.get("KPN_BUILD_DIR",
os.path.join(os.path.dirname(__file__),
"../../build/examples"))
sys.path.insert(0, BUILD_DIR)
import numpy as np
import cv2
import kpn_opencv as kpn
# ── Python node: replace the C++ quantise with numpy ─────────────────────────
# Receives and returns a BGR numpy array (H×W×3 uint8).
def py_quantise(bgr: np.ndarray) -> np.ndarray:
levels = 4
step = 256 // levels
q = (bgr.astype(np.int32) // step) * step + (step // 2)
return q.clip(0, 255).astype(np.uint8)
# ── Build network ─────────────────────────────────────────────────────────────
net = kpn.Network()
net.add("src", kpn.make_capture()) # out0=colour, out1=grey
net.add_node("quant", py_quantise, # Python node — numpy in/out
inputs=["mat"], outputs=["mat"])
net.add("gray", kpn.make_to_gray()) # in0=bgr → out0=gray
net.add("edges", kpn.make_edges()) # in0=gray → out0=edge_mask
net.add("comp", kpn.make_composite()) # in0=edge_mask, in1=colour
# out0=result, out1=edge_mask
# src.colour → py_quantise
net.connect("src", 0, "quant", 0)
# src.grey → to_gray
net.connect("src", 1, "gray", 0)
# gray → edges
net.connect("gray", 0, "edges", 0)
# quantised colour → composite.colour (input slot 1)
net.connect("quant", 0, "comp", 1)
# edge mask → composite.edges (input slot 0)
net.connect("edges", 0, "comp", 0)
net.build()
net.start()
# ── Display loop (drives GUI on this thread) ──────────────────────────────────
cv2.namedWindow("Cell Shade (Python quant)", cv2.WINDOW_NORMAL)
cv2.namedWindow("Edge Mask", cv2.WINDOW_NORMAL)
cv2.resizeWindow("Cell Shade (Python quant)", 1280, 720)
cv2.resizeWindow("Edge Mask", 640, 360)
try:
while True:
# Blocking reads — GIL released while waiting so C++ threads can run
result = net.read("comp", 0) # composite frame (BGR numpy array)
edges = net.read("comp", 1) # edge mask (grayscale numpy array)
cv2.imshow("Cell Shade (Python quant)", result)
cv2.imshow("Edge Mask", cv2.cvtColor(edges, cv2.COLOR_GRAY2BGR))
key = cv2.waitKey(1)
if key in (ord('q'), 27):
break
# Check windows still open
try:
if cv2.getWindowProperty("Cell Shade (Python quant)",
cv2.WND_PROP_VISIBLE) < 1:
break
except cv2.error:
break
finally:
net.stop()
cv2.destroyAllWindows()
del net # let C++ destructor run before nanobind tears down
examples/09_opencv_cellshade/kpn_opencv.cpp
+177
View File
@@ -0,0 +1,177 @@
#define KPN_BUILD_PYTHON
#include <kpn/python/auto_bind.hpp>
#include <nanobind/ndarray.h>
#include <opencv2/core.hpp>
#include <opencv2/imgproc.hpp>
#include <opencv2/videoio.hpp>
#include <chrono>
#include <cmath>
#include <iostream>
#include <thread>
#include <tuple>
namespace nb = nanobind;
using namespace kpn;
using namespace kpn::python;
// ── PythonConverter<cv::Mat> ──────────────────────────────────────────────────
// Converts cv::Mat ↔ numpy array (uint8, HxW or HxWxC shape).
//
// to_python: clones the mat onto the heap; the numpy array owns it via a
// capsule deleter — no shared cv::Mat refcount dangling after the Variant dies.
// from_python: calls numpy.ascontiguousarray, then clones into an owned cv::Mat.
namespace kpn {
template<> struct PythonConverter<cv::Mat> {
static constexpr const char* type_name = "mat";
static nb::object to_python(const cv::Mat& m) {
// Must be called with the GIL held (always true: called from read() or
// from within the gil_scoped_acquire block in PyNode::run_loop).
auto np = nb::module_::import_("numpy");
cv::Mat c = m.clone(); // ensure contiguous, independently owned
nb::bytes raw(reinterpret_cast<const char*>(c.data),
c.total() * c.elemSize());
nb::object arr = np.attr("frombuffer")(raw, "uint8");
int H = c.rows, W = c.cols, C = c.channels();
arr = arr.attr("reshape")(
C > 1 ? nb::make_tuple(H, W, C) : nb::make_tuple(H, W));
return arr.attr("copy")(); // writable, lifetime-independent copy
}
static cv::Mat from_python(nb::object o) {
auto np = nb::module_::import_("numpy");
// Ensure contiguous uint8 layout (in-place if already compatible)
nb::object arr = np.attr("ascontiguousarray")(o, "uint8");
auto shape = nb::cast<std::vector<int>>(arr.attr("shape"));
if (shape.size() < 2 || shape.size() > 3)
throw std::runtime_error(
"cv::Mat from_python: expected 2D (H×W) or 3D (H×W×C) uint8 array");
int H = shape[0], W = shape[1];
int C = (shape.size() == 3) ? shape[2] : 1;
int type = C > 1 ? CV_8UC(C) : CV_8UC1;
// Cast to ndarray to get the raw data pointer
auto binfo = nb::cast<nb::ndarray<nb::numpy, uint8_t>>(arr);
cv::Mat wrap(H, W, type, binfo.data());
return wrap.clone(); // own the pixel data
}
};
} // namespace kpn
// ── Pipeline functions ────────────────────────────────────────────────────────
static cv::Mat make_gradient(int W, int H) {
cv::Mat xr(H, W, CV_8UC1), yg(H, W, CV_8UC1), b(H, W, CV_8UC1, cv::Scalar(128));
for (int x = 0; x < W; ++x) xr.col(x).setTo(x * 255 / W);
for (int y = 0; y < H; ++y) yg.row(y).setTo(y * 255 / H);
cv::Mat channels[3] = {b, yg, xr};
cv::Mat grad;
cv::merge(channels, 3, grad);
return grad;
}
static std::tuple<cv::Mat, cv::Mat> capture() {
constexpr int W = 640, H = 480;
static cv::VideoCapture cap;
static bool opened = false;
if (!opened) {
opened = true;
cap.open(0, cv::CAP_V4L2);
if (cap.isOpened()) {
cap.set(cv::CAP_PROP_FRAME_WIDTH, W);
cap.set(cv::CAP_PROP_FRAME_HEIGHT, H);
} else {
std::cerr << "[capture] no webcam — using synthetic animated pattern\n";
}
}
cv::Mat frame;
if (cap.isOpened()) {
auto t0 = std::chrono::steady_clock::now();
cap >> frame;
auto elapsed = std::chrono::steady_clock::now() - t0;
if (elapsed < std::chrono::milliseconds(20))
std::this_thread::sleep_for(std::chrono::milliseconds(33) - elapsed);
if (frame.empty()) frame = cv::Mat::zeros(H, W, CV_8UC3);
} else {
static int tick = 0;
static cv::Mat grad = make_gradient(W, H);
++tick;
frame = grad.clone();
int r = 150 + (tick % 80) * 4;
cv::circle(frame, {W/2, H/2}, r, {255, 200, 0}, -1);
cv::circle(frame, {W/2, H/2}, r / 2, { 0, 128, 255}, -1);
cv::circle(frame, {W*2/5, H*2/5}, r / 3, {200, 0, 200}, -1);
std::this_thread::sleep_for(std::chrono::milliseconds(33));
}
return {frame.clone(), frame.clone()};
}
static cv::Mat to_gray(cv::Mat bgr) {
cv::Mat gray;
cv::cvtColor(bgr, gray, cv::COLOR_BGR2GRAY);
return gray;
}
static cv::Mat edges_fn(cv::Mat gray) {
cv::Mat blurred, mask;
cv::GaussianBlur(gray, blurred, {5, 5}, 0);
cv::Canny(blurred, mask, 50, 150);
return mask;
}
static cv::Mat quantise(cv::Mat bgr) {
constexpr int levels = 4;
constexpr double step = 256.0 / levels;
static const cv::Mat lut = []() {
cv::Mat l(1, 256, CV_8UC1);
for (int i = 0; i < 256; ++i)
l.at<uchar>(i) = cv::saturate_cast<uchar>(
std::floor(i / step) * step + step / 2.0);
return l;
}();
cv::Mat out;
cv::LUT(bgr, lut, out);
return out;
}
// Returns composite frame AND edge mask so the display node can show both
// without needing a fan-out on the edges channel.
static std::tuple<cv::Mat, cv::Mat> composite(cv::Mat edge_mask, cv::Mat colour) {
cv::Mat result = colour.clone();
result.setTo(cv::Scalar(0, 0, 0), edge_mask);
return {result, edge_mask};
}
// ── Registry ──────────────────────────────────────────────────────────────────
// Variant deduced as std::variant<cv::Mat> — every node uses only cv::Mat.
using CvNodes = NodeRegistry<
Entry<capture, "capture">,
Entry<to_gray, "to_gray">,
Entry<edges_fn, "edges">,
Entry<quantise, "quantise">,
Entry<composite, "composite">
>;
// ── Module ────────────────────────────────────────────────────────────────────
NB_MODULE(kpn_opencv, m) {
m.doc() = "KPN++ OpenCV bindings for the cell-shading pipeline";
// Registers: Network, INode, CaptureNode, ToGrayNode, EdgesNode,
// QuantiseNode, CompositeNode, and make_<name>() factories.
// Network.add_node(name, callable, inputs=["mat"], outputs=["mat"])
// accepts Python callables that receive/return numpy uint8 arrays.
bind_network<CvNodes>(m);
// Note: bind_debug is omitted here — cv::Mat functions cannot be called
// directly from Python without the variant/network machinery. Use
// net.write() + net.read() to inject/inspect individual nodes instead.
}
examples/09_opencv_cellshade/main.cpp
+206
View File
@@ -0,0 +1,206 @@
#include <kpn/kpn.hpp>
#include <opencv2/core.hpp>
#include <opencv2/imgproc.hpp>
#include <opencv2/highgui.hpp>
#include <opencv2/videoio.hpp>
#include <iostream>
#include <tuple>
#include <thread>
#include <chrono>
// Teach KPN how many bytes a cv::Mat actually carries (header + pixel data).
template<>
struct kpn::ChannelDataSize<cv::Mat> {
static std::size_t bytes(const cv::Mat& m) { return m.total() * m.elemSize(); }
};
// ── Cell-shading pipeline ─────────────────────────────────────────────────────
//
// [capture] --"colour"--> [quantise] ──────────────────────────┐
// ├──> [composite] ──"result"──┐
// [capture] --"grey"---> [to_gray] --> [edges] ──"edges"───────┘ │
// └──────────────────────────────"edges"──────────┴──> [display]
//
// DisplayNode derives from MainThreadNode<> — two inputs (composite + raw edges),
// two windows opened in the constructor. step() is called on the main thread.
// ── Gradient base for synthetic pattern ──────────────────────────────────────
static cv::Mat make_gradient(int W, int H) {
cv::Mat xr(H, W, CV_8UC1), yg(H, W, CV_8UC1), b(H, W, CV_8UC1, cv::Scalar(128));
for (int x = 0; x < W; ++x) xr.col(x).setTo(x * 255 / W);
for (int y = 0; y < H; ++y) yg.row(y).setTo(y * 255 / H);
cv::Mat channels[3] = {b, yg, xr};
cv::Mat grad;
cv::merge(channels, 3, grad);
return grad;
}
// ── Pipeline functions ────────────────────────────────────────────────────────
// --8<-- [start:capture_fn]
static std::tuple<cv::Mat, cv::Mat> capture() {
constexpr int W = 640, H = 480;
static cv::VideoCapture cap;
static bool opened = false;
if (!opened) {
opened = true;
cap.open(0, cv::CAP_V4L2);
if (cap.isOpened()) {
cap.set(cv::CAP_PROP_FRAME_WIDTH, W);
cap.set(cv::CAP_PROP_FRAME_HEIGHT, H);
} else {
std::cerr << "[capture] no webcam — using synthetic animated pattern\n";
}
}
cv::Mat frame;
if (cap.isOpened()) {
auto t0 = std::chrono::steady_clock::now();
cap >> frame;
auto elapsed = std::chrono::steady_clock::now() - t0;
if (elapsed < std::chrono::milliseconds(20))
std::this_thread::sleep_for(std::chrono::milliseconds(33) - elapsed);
if (frame.empty()) frame = cv::Mat::zeros(H, W, CV_8UC3);
} else {
static int tick = 0;
static cv::Mat grad = make_gradient(W, H);
++tick;
frame = grad.clone();
int r = 150 + (tick % 80) * 4;
cv::circle(frame, {W/2, H/2}, r, {255, 200, 0}, -1);
cv::circle(frame, {W/2, H/2}, r / 2, { 0, 128, 255}, -1);
cv::circle(frame, {W*2/5, H*2/5}, r / 3, {200, 0, 200}, -1);
std::this_thread::sleep_for(std::chrono::milliseconds(33));
}
return {frame.clone(), frame.clone()};
}
// --8<-- [end:capture_fn]
static cv::Mat to_gray(cv::Mat bgr) {
cv::Mat gray;
cv::cvtColor(bgr, gray, cv::COLOR_BGR2GRAY);
return gray;
}
static cv::Mat edges_fn(cv::Mat gray) {
cv::Mat blurred, mask;
cv::GaussianBlur(gray, blurred, {5, 5}, 0);
cv::Canny(blurred, mask, 50, 150);
return mask;
}
static cv::Mat quantise(cv::Mat bgr) {
constexpr int levels = 4;
constexpr double step = 256.0 / levels;
static const cv::Mat lut = []() {
cv::Mat l(1, 256, CV_8UC1);
for (int i = 0; i < 256; ++i)
l.at<uchar>(i) = cv::saturate_cast<uchar>(
std::floor(i / step) * step + step / 2.0);
return l;
}();
cv::Mat out;
cv::LUT(bgr, lut, out);
return out;
}
// Returns both the composite frame and the original edge mask so downstream
// nodes (display) can receive both without fan-out on the edges channel.
static std::tuple<cv::Mat, cv::Mat> composite(cv::Mat edge_mask, cv::Mat colour) {
cv::Mat result = colour.clone();
result.setTo(cv::Scalar(0, 0, 0), edge_mask);
return {result, edge_mask};
}
// ── DisplayNode ───────────────────────────────────────────────────────────────
//
// Two inputs: the cell-shaded composite frame and the raw edge mask.
// MainThreadNode<> provides channel ownership, INode boilerplate, and stats.
// The constructor opens both windows on the main thread (Wayland requirement).
// operator() is called by step() whenever both channels have a frame ready.
// --8<-- [start:display_node]
class DisplayNode : public kpn::MainThreadNode<DisplayNode,
kpn::in<"composite", "edges">,
cv::Mat, cv::Mat> {
public:
DisplayNode() : MainThreadNode(8) {
cv::namedWindow("Cell Shade", cv::WINDOW_NORMAL);
cv::namedWindow("Edge Mask", cv::WINDOW_NORMAL);
cv::resizeWindow("Cell Shade", 1280, 720);
cv::resizeWindow("Edge Mask", 640, 360);
}
~DisplayNode() { cv::destroyAllWindows(); }
bool operator()(cv::Mat composite, cv::Mat edges) {
cv::imshow("Cell Shade", composite);
cv::Mat edges_bgr;
cv::cvtColor(edges, edges_bgr, cv::COLOR_GRAY2BGR);
cv::imshow("Edge Mask", edges_bgr);
int key = cv::waitKey(1);
if (key == 'q' || key == 27) return false;
return window_open("Cell Shade") && window_open("Edge Mask");
}
private:
static bool window_open(const char* name) {
try { return cv::getWindowProperty(name, cv::WND_PROP_VISIBLE) >= 1; }
catch (const cv::Exception&) { return false; }
}
};
// --8<-- [end:display_node]
// ─────────────────────────────────────────────────────────────────────────────
int main() {
using namespace kpn;
// --8<-- [start:opencv_network]
auto src = make_node<capture> (out<"colour","grey">{}, 8);
auto gray_node = make_node<to_gray> (in<"bgr">{}, out<"gray">{}, 8);
auto edge_node = make_node<edges_fn> (in<"gray">{}, out<"edges">{}, 8);
auto quant = make_node<quantise> (in<"bgr">{}, out<"quantised">{}, 8);
auto comp = make_node<composite>(in<"edges","colour">{}, out<"result","edges">{}, 8);
// DisplayNode: two windows opened in constructor, step() drives main thread.
DisplayNode disp;
Network net;
net.add("src", src)
.add("gray", gray_node)
.add("edges", edge_node)
.add("quant", quant)
.add("comp", comp)
.add("display", disp)
.connect("src", src.template output<"colour">(), "quant", quant.template input<"bgr">())
.connect("quant", quant.template output<"quantised">(), "comp", comp.template input<"colour">())
.connect("src", src.template output<"grey">(), "gray", gray_node.template input<"bgr">())
.connect("gray", gray_node.template output<"gray">(), "edges", edge_node.template input<"gray">())
.connect("edges", edge_node.template output<"edges">(), "comp", comp.template input<"edges">())
.connect("comp", comp.template output<"result">(), "display", disp.template input<"composite">())
.connect("comp", comp.template output<"edges">(), "display", disp.template input<"edges">())
.build();
// --8<-- [end:opencv_network]
net.set_watchdog_interval(std::chrono::milliseconds(5000));
#ifdef KPN_WEB_DEBUG
net.set_web_debug_port(9090);
#endif
std::cout << "Cell-shading pipeline running. Press 'q' to stop.\n";
std::cout << "Web debug UI: http://localhost:9090\n";
// --8<-- [start:main_thread_step]
net.start();
// Main thread drives display — imshow/waitKey stay on the GUI thread.
// step() returns false when operator() returns false (q pressed / window closed).
while (disp.step())
cv::waitKey(8); // yield event loop when no frame ready
net.stop();
// --8<-- [end:main_thread_step]
return 0;
}
examples/10_static_hello_pipeline/main.cpp
+34
View File
@@ -0,0 +1,34 @@
// Example 10 — Static Hello Pipeline
//
// The same linear pipeline as example 01, built with make_network() instead
// of the runtime Network builder. The topology is fully known at compile time:
// cycle detection is a static_assert, no build() step is needed, and start/stop
// require no virtual dispatch through a string-keyed node map.
//
// [produce] --int--> [double_it] --int--> [print_it]
#include <kpn/kpn.hpp>
#include <chrono>
#include <iostream>
#include <thread>
static int produce() { return 42; }
static int double_it(int x) { return x * 2; }
static void print_it(int x) { std::cout << "result: " << x << '\n'; }
int main() {
using namespace kpn;
auto src = make_node<produce, "src" >(5);
auto dbl = make_node<double_it, "dbl" >(5);
auto sink = make_node<print_it, "sink">(5);
auto net = make_network(
edge(src.output<0>(), dbl.input<0>()),
edge(dbl.output<0>(), sink.input<0>())
);
net.start();
std::this_thread::sleep_for(std::chrono::milliseconds(100));
net.stop();
}
examples/11_static_fanout/main.cpp
+55
View File
@@ -0,0 +1,55 @@
// Example 11 — Static Fan-Out (automatic FanoutNode insertion)
//
// Two consumers read from the same output port of [generate]. With the runtime
// Network builder this would require an explicit make_fanout<>. With make_network()
// the duplicate source port is detected at compile time and a FanoutNode<string,2>
// is inserted automatically — the user just writes two edges from the same port.
//
// +--> [print_key]
// [generate] --string--> [fan]
// +--> [print_upper]
//
// The FanoutNode<string,2> is owned by the StaticNetwork and invisible to the user.
#include <kpn/kpn.hpp>
#include <algorithm>
#include <chrono>
#include <iostream>
#include <string>
#include <thread>
static int gen_index = 0;
static std::string generate() {
static const char* words[] = {"hello", "kpn", "fanout", "static", "network"};
std::this_thread::sleep_for(std::chrono::milliseconds(80));
return words[gen_index++ % 5];
}
static void print_lower(std::string s) {
std::cout << "lower: " << s << '\n';
}
static void print_upper(std::string s) {
std::transform(s.begin(), s.end(), s.begin(), ::toupper);
std::cout << "upper: " << s << '\n';
}
int main() {
using namespace kpn;
auto gen = make_node<generate, "gen" >(5);
auto lower = make_node<print_lower, "lower">(5);
auto upper = make_node<print_upper, "upper">(5);
// Two edges from gen.output<0>() — make_network() detects the fan-out
// and inserts FanoutNode<std::string, 2> automatically.
auto net = make_network(
edge(gen.output<0>(), lower.input<0>()),
edge(gen.output<0>(), upper.input<0>())
);
net.start();
std::this_thread::sleep_for(std::chrono::milliseconds(500));
net.stop();
}
examples/12_static_cellshade/main.cpp
+217
View File
@@ -0,0 +1,217 @@
// Example 12 — Static Cell-Shading Pipeline with Auto Fan-Out
//
// The same cell-shading effect as example 09, rebuilt with make_network().
//
// Key differences from example 09:
//
// 1. Fan-out is automatic. The edge detector output feeds both the compositing
// node and the debug display window. In example 09 this required composite()
// to re-output the edge mask as a second return value (a workaround). Here
// make_network() detects the duplicate source port and inserts
// FanoutNode<cv::Mat, 2> automatically — composite() is a clean single-output
// node.
//
// 2. No add()/connect()/build() ceremony. The full topology is expressed once
// in the make_network() call. Cycle detection and duplicate-tag checking are
// compile-time static_asserts.
//
// 3. Every node has a Label NTTP so the web debug UI shows real names.
//
// Topology:
//
// [capture] --colour--> [quant] ──────────────────────────────> [comp] --> [display_composite]
// [capture] --grey----> [to_gray] --> [edges] --edges--(fan)--> [comp]
// --edges----------> [display_edges] ← auto-fanout
//
// Note: capture returns std::tuple<cv::Mat, cv::Mat> (colour, grey).
// The two outputs are separate ports routed independently.
#include <kpn/kpn.hpp>
#include <opencv2/core.hpp>
#include <opencv2/imgproc.hpp>
#include <opencv2/highgui.hpp>
#include <opencv2/videoio.hpp>
#include <iostream>
#include <tuple>
#include <thread>
#include <chrono>
// ── Gradient base for synthetic pattern ───────────────────────────────────────
static cv::Mat make_gradient(int W, int H) {
cv::Mat xr(H, W, CV_8UC1), yg(H, W, CV_8UC1), b(H, W, CV_8UC1, cv::Scalar(128));
for (int x = 0; x < W; ++x) xr.col(x).setTo(x * 255 / W);
for (int y = 0; y < H; ++y) yg.row(y).setTo(y * 255 / H);
cv::Mat channels[3] = {b, yg, xr};
cv::Mat grad;
cv::merge(channels, 3, grad);
return grad;
}
// ── Pipeline functions ────────────────────────────────────────────────────────
static std::tuple<cv::Mat, cv::Mat> capture() {
constexpr int W = 640, H = 480;
static cv::VideoCapture cap;
static bool opened = false;
if (!opened) {
opened = true;
cap.open(0, cv::CAP_V4L2);
if (cap.isOpened()) {
cap.set(cv::CAP_PROP_FRAME_WIDTH, W);
cap.set(cv::CAP_PROP_FRAME_HEIGHT, H);
} else {
std::cerr << "[capture] no webcam — using synthetic animated pattern\n";
}
}
cv::Mat frame;
if (cap.isOpened()) {
auto t0 = std::chrono::steady_clock::now();
cap >> frame;
auto elapsed = std::chrono::steady_clock::now() - t0;
if (elapsed < std::chrono::milliseconds(20))
std::this_thread::sleep_for(std::chrono::milliseconds(33) - elapsed);
if (frame.empty()) frame = cv::Mat::zeros(H, W, CV_8UC3);
} else {
static int tick = 0;
static cv::Mat grad = make_gradient(W, H);
++tick;
frame = grad.clone();
int r = 150 + (tick % 80) * 4;
cv::circle(frame, {W/2, H/2}, r, {255, 200, 0}, -1);
cv::circle(frame, {W/2, H/2}, r / 2, { 0, 128, 255}, -1);
cv::circle(frame, {W*2/5, H*2/5}, r / 3, {200, 0, 200}, -1);
std::this_thread::sleep_for(std::chrono::milliseconds(33));
}
return {frame.clone(), frame.clone()};
}
static cv::Mat to_gray(cv::Mat bgr) {
cv::Mat gray;
cv::cvtColor(bgr, gray, cv::COLOR_BGR2GRAY);
return gray;
}
static cv::Mat edges_fn(cv::Mat gray) {
cv::Mat blurred, mask;
cv::GaussianBlur(gray, blurred, {5, 5}, 0);
cv::Canny(blurred, mask, 50, 150);
return mask;
}
static cv::Mat quantise(cv::Mat bgr) {
constexpr int levels = 4;
constexpr double step = 256.0 / levels;
static const cv::Mat lut = []() {
cv::Mat l(1, 256, CV_8UC1);
for (int i = 0; i < 256; ++i)
l.at<uchar>(i) = cv::saturate_cast<uchar>(
std::floor(i / step) * step + step / 2.0);
return l;
}();
cv::Mat out;
cv::LUT(bgr, lut, out);
return out;
}
// Clean single-output composite — no longer needs to pass edges through.
static cv::Mat composite(cv::Mat edge_mask, cv::Mat colour) {
cv::Mat result = colour.clone();
result.setTo(cv::Scalar(0, 0, 0), edge_mask);
return result;
}
// ── Display nodes ─────────────────────────────────────────────────────────────
//
// Two separate MainThreadNode subclasses — one for the composited result,
// one for the raw edge mask. Each runs on the main thread via step().
// The fan-out from [edges] to both consumers is inserted automatically by
// make_network().
class DisplayComposite : public kpn::MainThreadNode<DisplayComposite,
kpn::in<"composite">,
cv::Mat> {
public:
// Label and unique_tag for StaticNetwork identity
static constexpr std::string_view label() { return "display_composite"; }
static constexpr std::size_t unique_tag = 0;
DisplayComposite() : MainThreadNode(8) {
cv::namedWindow("Cell Shade", cv::WINDOW_NORMAL);
cv::resizeWindow("Cell Shade", 1280, 720);
}
~DisplayComposite() { cv::destroyWindow("Cell Shade"); }
bool operator()(cv::Mat frame) {
cv::imshow("Cell Shade", frame);
int key = cv::waitKey(1);
if (key == 'q' || key == 27) return false;
try { return cv::getWindowProperty("Cell Shade", cv::WND_PROP_VISIBLE) >= 1; }
catch (const cv::Exception&) { return false; }
}
};
class DisplayEdges : public kpn::MainThreadNode<DisplayEdges,
kpn::in<"edges">,
cv::Mat> {
public:
static constexpr std::string_view label() { return "display_edges"; }
static constexpr std::size_t unique_tag = 1;
DisplayEdges() : MainThreadNode(8) {
cv::namedWindow("Edge Mask", cv::WINDOW_NORMAL);
cv::resizeWindow("Edge Mask", 640, 360);
}
~DisplayEdges() { cv::destroyWindow("Edge Mask"); }
bool operator()(cv::Mat mask) {
cv::Mat bgr;
cv::cvtColor(mask, bgr, cv::COLOR_GRAY2BGR);
cv::imshow("Edge Mask", bgr);
cv::waitKey(1);
try { return cv::getWindowProperty("Edge Mask", cv::WND_PROP_VISIBLE) >= 1; }
catch (const cv::Exception&) { return false; }
}
};
// ─────────────────────────────────────────────────────────────────────────────
int main() {
using namespace kpn;
// Nodes — all labelled for web debug UI
auto src = make_node<capture, "capture">(8);
auto gray_node = make_node<to_gray, "to_gray">(8);
auto edge_node = make_node<edges_fn, "edges" >(8);
auto quant = make_node<quantise, "quant" >(8);
auto comp = make_node<composite, "comp" >(8);
// DisplayNodes live on the main thread — registered as sinks
DisplayComposite disp_comp;
DisplayEdges disp_edges;
// make_network() detects that edge_node.output<0>() feeds two consumers
// (comp and disp_edges) and inserts FanoutNode<cv::Mat, 2> automatically.
auto net = make_network(
edge(src.output<0>(), quant.input<0>()), // colour → quant
edge(src.output<1>(), gray_node.input<0>()), // grey → to_gray
edge(gray_node.output<0>(), edge_node.input<0>()), // gray → edges
edge(edge_node.output<0>(), comp.input<0>()), // edges → comp (fan-out src)
edge(edge_node.output<0>(), disp_edges.input<0>()), // edges → display_edges (auto fanout)
edge(quant.output<0>(), comp.input<1>()), // quantised → comp
edge(comp.output<0>(), disp_comp.input<0>()) // result → display_composite
);
std::cout << "Cell-shading pipeline (static) running. Press 'q' to stop.\n";
net.start();
// Main thread drives both display nodes — step() returns false when
// operator() returns false (q pressed or window closed).
while (disp_comp.step() && disp_edges.step())
cv::waitKey(8);
net.stop();
return 0;
}
examples/13_debug_cellshade/main.cpp
+288
View File
@@ -0,0 +1,288 @@
// Example 13 — Debug Cell-Shading Pipeline with Tiled Debug Canvas
//
// An improved cell-shading pipeline where every processing node performs
// exactly one OpenCV operation. A variadic DebugCanvas<N> node tiles N
// cv::Mat inputs into a single debug window, making each pipeline stage
// visible side-by-side at runtime.
//
// Improvements over example 12:
// - Bilateral filter before quantisation (edge-preserving smoothing):
// flattens colour regions without softening object edges
// - Dilated edge mask for bolder black outlines
// - 6-level quantisation for richer tonal detail
//
// Topology (auto-fanouts inserted by make_network):
//
// [capture]──┬──> [bilateral]──> [quant]──┬──> [comp]──> [debug:0 result]
// │ └──> [debug:1 quantised]
// ├──> [to_gray]──> [blur]──> [canny]──┬──> [dilate]──> [comp]
// │ └──> [debug:2 edges]
// └──> [debug:3 original]
//
// make_network detects the 3-way fan from [capture], the 2-way fan from
// [quant], and the 2-way fan from [canny], inserting FanoutNode instances
// automatically.
#include <kpn/kpn.hpp>
#include <opencv2/core.hpp>
#include <opencv2/imgproc.hpp>
#include <opencv2/highgui.hpp>
#include <opencv2/videoio.hpp>
#include <algorithm>
#include <chrono>
#include <cmath>
#include <iostream>
#include <string>
#include <thread>
#include <type_traits>
#include <vector>
// ── Synthetic source for environments without a webcam ───────────────────────
static cv::Mat make_gradient(int W, int H) {
cv::Mat xr(H, W, CV_8UC1), yg(H, W, CV_8UC1), b(H, W, CV_8UC1, cv::Scalar(128));
for (int x = 0; x < W; ++x) xr.col(x).setTo(x * 255 / W);
for (int y = 0; y < H; ++y) yg.row(y).setTo(y * 255 / H);
cv::Mat channels[3] = {b, yg, xr};
cv::Mat grad;
cv::merge(channels, 3, grad);
return grad;
}
// ── Pipeline nodes — one cv:: call per function ───────────────────────────────
static cv::Mat capture() {
constexpr int W = 640, H = 480;
static cv::VideoCapture cap;
static bool opened = false;
if (!opened) {
opened = true;
cap.open(0, cv::CAP_V4L2);
if (cap.isOpened()) {
cap.set(cv::CAP_PROP_FRAME_WIDTH, W);
cap.set(cv::CAP_PROP_FRAME_HEIGHT, H);
} else {
std::cerr << "[capture] no webcam — using synthetic animated pattern\n";
}
}
cv::Mat frame;
if (cap.isOpened()) {
auto t0 = std::chrono::steady_clock::now();
cap >> frame;
auto elapsed = std::chrono::steady_clock::now() - t0;
if (elapsed < std::chrono::milliseconds(20))
std::this_thread::sleep_for(std::chrono::milliseconds(33) - elapsed);
if (frame.empty()) frame = cv::Mat::zeros(H, W, CV_8UC3);
} else {
static int tick = 0;
static cv::Mat grad = make_gradient(W, H);
++tick;
frame = grad.clone();
int r = 150 + (tick % 80) * 4;
cv::circle(frame, {W/2, H/2}, r, {255, 200, 0}, -1);
cv::circle(frame, {W/2, H/2}, r / 2, { 0, 128, 255}, -1);
cv::circle(frame, {W*2/5, H*2/5}, r / 3, {200, 0, 200}, -1);
std::this_thread::sleep_for(std::chrono::milliseconds(33));
}
return frame.clone();
}
// Edge-preserving smooth: flattens colour within regions while keeping sharp
// boundaries — much better than Gaussian blur as a pre-quantisation step.
static cv::Mat bilateral_filter(cv::Mat bgr) {
cv::Mat out;
cv::bilateralFilter(bgr, out, 5, 75, 75);
return out;
}
// Snap each channel to N discrete tonal levels.
static cv::Mat quantise(cv::Mat bgr) {
constexpr int levels = 6;
constexpr double step = 256.0 / levels;
static const cv::Mat lut = []() {
cv::Mat l(1, 256, CV_8UC1);
for (int i = 0; i < 256; ++i)
l.at<uchar>(i) = cv::saturate_cast<uchar>(
std::floor(i / step) * step + step / 2.0);
return l;
}();
cv::Mat out;
cv::LUT(bgr, lut, out);
return out;
}
// BGR → greyscale; the edge path works on the original (not bilateral-filtered)
// frame so that fine edge detail is preserved.
static cv::Mat to_gray(cv::Mat bgr) {
cv::Mat gray;
cv::cvtColor(bgr, gray, cv::COLOR_BGR2GRAY);
return gray;
}
// Suppress high-frequency noise before the Canny detector.
static cv::Mat gaussian_blur(cv::Mat gray) {
cv::Mat out;
cv::GaussianBlur(gray, out, {5, 5}, 0);
return out;
}
// Detect strong edges; returns a binary mask (CV_8UC1).
static cv::Mat canny_edges(cv::Mat blurred) {
cv::Mat out;
cv::Canny(blurred, out, 50, 150);
return out;
}
// Widen the edge mask for bolder cartoon outlines.
static cv::Mat dilate_edges(cv::Mat edges) {
static const cv::Mat kernel = cv::getStructuringElement(cv::MORPH_RECT, {3, 3});
cv::Mat out;
cv::dilate(edges, out, kernel);
return out;
}
// Burn black outlines into the quantised colour image.
static cv::Mat composite(cv::Mat quantised, cv::Mat thick_edges) {
cv::Mat result = quantised.clone();
result.setTo(cv::Scalar(0, 0, 0), thick_edges);
return result;
}
// ── DebugCanvas<N> ────────────────────────────────────────────────────────────
//
// Variadic MainThreadNode that accepts N cv::Mat inputs (any mix of BGR and
// greyscale) and tiles them into one debug window arranged as a
// ceil(sqrt(N)) × ceil(N/cols) grid.
//
// Template trick: MatArg<I> aliases cv::Mat for all I, so the pack expansion
// MatArg<0>, MatArg<1>, ..., MatArg<N-1>
// produces exactly N cv::Mat arguments — enough to drive the MainThreadNode
// base without manually spelling out the type N times.
template<std::size_t>
using MatArg = cv::Mat;
template<std::size_t N>
class DebugCanvas;
template<std::size_t N, typename Seq = std::make_index_sequence<N>>
struct DebugCanvasBase;
template<std::size_t N, std::size_t... Is>
struct DebugCanvasBase<N, std::index_sequence<Is...>> {
using type = kpn::MainThreadNode<DebugCanvas<N>, kpn::in<>, MatArg<Is>...>;
};
template<std::size_t N>
class DebugCanvas : public DebugCanvasBase<N>::type {
using Base = typename DebugCanvasBase<N>::type;
public:
static constexpr std::string_view label() { return "debug_canvas"; }
static constexpr std::size_t unique_tag = 0;
explicit DebugCanvas(std::vector<std::string> slot_labels = {},
std::size_t fifo_cap = 4)
: Base(fifo_cap), labels_(std::move(slot_labels))
{
cv::namedWindow("Debug Canvas", cv::WINDOW_NORMAL);
cv::resizeWindow("Debug Canvas", cols() * CW, rows() * CH);
}
~DebugCanvas() { cv::destroyWindow("Debug Canvas"); }
// Called by MainThreadNode::step() with exactly N cv::Mat arguments.
template<typename... Ms>
bool operator()(Ms&&... mats) {
static_assert(sizeof...(Ms) == N, "DebugCanvas: wrong number of inputs");
std::vector<cv::Mat> imgs;
imgs.reserve(N);
(imgs.push_back(to_bgr(std::forward<Ms>(mats))), ...);
cv::imshow("Debug Canvas", tile(imgs));
int key = cv::waitKey(1);
if (key == 'q' || key == 27) return false;
try { return cv::getWindowProperty("Debug Canvas", cv::WND_PROP_VISIBLE) >= 1; }
catch (const cv::Exception&) { return false; }
}
private:
static constexpr int CW = 640, CH = 480;
static int cols() { return std::max(1, (int)std::ceil(std::sqrt((double)N))); }
static int rows() { return ((int)N + cols() - 1) / cols(); }
std::vector<std::string> labels_;
static cv::Mat to_bgr(const cv::Mat& m) {
if (m.channels() == 1) {
cv::Mat bgr;
cv::cvtColor(m, bgr, cv::COLOR_GRAY2BGR);
return bgr;
}
return m;
}
cv::Mat tile(const std::vector<cv::Mat>& imgs) const {
const int c = cols(), r = rows();
cv::Mat canvas(r * CH, c * CW, CV_8UC3, cv::Scalar(30, 30, 30));
for (int i = 0; i < (int)imgs.size(); ++i) {
if (imgs[i].empty()) continue;
cv::Mat cell = canvas(cv::Rect((i % c) * CW, (i / c) * CH, CW, CH));
cv::Mat resized;
cv::resize(imgs[i], resized, {CW, CH});
resized.copyTo(cell);
if (i < (int)labels_.size() && !labels_[i].empty())
cv::putText(cell, labels_[i], {8, 36},
cv::FONT_HERSHEY_SIMPLEX, 1.0, {0, 255, 255}, 2,
cv::LINE_AA);
}
return canvas;
}
};
// ── main ──────────────────────────────────────────────────────────────────────
int main() {
using namespace kpn;
auto src = make_node<capture, "capture" >(4);
auto bilateral= make_node<bilateral_filter, "bilateral" >(4);
auto quant = make_node<quantise, "quant" >(4);
auto gray = make_node<to_gray, "to_gray" >(4);
auto blur = make_node<gaussian_blur, "blur" >(4);
auto canny = make_node<canny_edges, "canny" >(4);
auto dilate = make_node<dilate_edges, "dilate" >(4);
auto comp = make_node<composite, "comp" >(4);
DebugCanvas<4> debug({"result", "quantised", "edges", "original"});
// make_network auto-inserts FanoutNode instances wherever a source port
// feeds more than one consumer:
// capture → 3-way fanout (bilateral, to_gray, debug[3])
// quant → 2-way fanout (comp, debug[1])
// canny → 2-way fanout (dilate, debug[2])
auto net = make_network(
edge(src.output<0>(), bilateral.input<0>()), // frame → bilateral
edge(src.output<0>(), gray.input<0>()), // frame → to_gray
edge(src.output<0>(), debug.input<3>()), // frame → debug[3] original
edge(bilateral.output<0>(), quant.input<0>()), // smooth → quant
edge(quant.output<0>(), comp.input<0>()), // quant → comp
edge(quant.output<0>(), debug.input<1>()), // quant → debug[1] quantised
edge(gray.output<0>(), blur.input<0>()), // gray → blur
edge(blur.output<0>(), canny.input<0>()), // blurred→ canny
edge(canny.output<0>(), dilate.input<0>()), // edges → dilate
edge(canny.output<0>(), debug.input<2>()), // edges → debug[2] edges
edge(dilate.output<0>(), comp.input<1>()), // thick → comp
edge(comp.output<0>(), debug.input<0>()) // result → debug[0] result
);
std::cout << "Debug cell-shading pipeline running — press 'q' to stop.\n";
std::cout << "Canvas: [0] result [1] quantised [2] edges [3] original\n";
net.start();
while (debug.step())
cv::waitKey(8);
net.stop();
return 0;
}
examples/14_debug_hub/main.cpp
+159
View File
@@ -0,0 +1,159 @@
// Example 14 — DebugHub with Shared Resource Token
//
// Two independent KPN networks compete for one shared inference resource
// (simulating a small GPU or single-session ONNX runtime). The DebugHub
// serves a single web UI at http://localhost:9090 with:
//
// [All Networks] — resource utilisation cards + cross-network node table
// [detect] — force-directed graph for the detection pipeline
// [classify] — force-directed graph for the classification pipeline
//
// Topology:
//
// detect pipeline:
// [source_detect] ──> [run_detect] ──> [sink_detect]
//
// classify pipeline:
// [source_classify] ──> [run_classify] ──> [sink_classify]
//
// Both [run_detect] and [run_classify] call gpu.acquire() before touching the
// simulated device. The priority-based token awards the next slot to the
// waiter that is more likely to make useful progress (higher priority score).
//
// Build: cmake -DKPN_WEB_DEBUG=ON .. && cmake --build .
// Run: ./14_debug_hub
// UI: http://localhost:9090
#ifdef KPN_WEB_DEBUG
#include <kpn/kpn.hpp>
#include <atomic>
#include <chrono>
#include <iostream>
#include <thread>
using namespace kpn;
using namespace std::chrono_literals;
// ── Simulated inference device ────────────────────────────────────────────────
//
// Represents any exclusive, serialised accelerator: GPU session, ONNX runtime,
// hardware encoder, etc. Only one caller can hold it at a time.
struct GPU {
// Detection model: fast, 8 ms per frame.
int detect(int frame_id) {
std::this_thread::sleep_for(8ms);
return frame_id * 2; // synthetic "score"
}
// Classification model: heavier, 14 ms per frame.
int classify(int frame_id) {
std::this_thread::sleep_for(14ms);
return frame_id % 10; // synthetic "label"
}
};
// Global pointer so free-function nodes can reach the resource.
// In production code, capture by reference inside an ObjectNode functor instead.
static SharedResource<GPU>* g_gpu = nullptr;
// ── Detection pipeline ────────────────────────────────────────────────────────
static int source_detect() {
static std::atomic<int> id{0};
std::this_thread::sleep_for(25ms); // ~40 fps source rate
return id.fetch_add(1, std::memory_order_relaxed);
}
static int run_detect(int frame_id) {
// Higher priority: detection is latency-critical.
auto guard = g_gpu->acquire([] { return 0.7f; });
return guard->detect(frame_id);
}
static std::atomic<uint64_t> detect_out{0};
static void sink_detect(int) {
detect_out.fetch_add(1, std::memory_order_relaxed);
}
// ── Classification pipeline ───────────────────────────────────────────────────
static int source_classify() {
static std::atomic<int> id{0};
std::this_thread::sleep_for(40ms); // ~25 fps source rate
return id.fetch_add(1, std::memory_order_relaxed);
}
static int run_classify(int frame_id) {
// Lower priority: classification is best-effort.
auto guard = g_gpu->acquire([] { return 0.3f; });
return guard->classify(frame_id);
}
static std::atomic<uint64_t> classify_out{0};
static void sink_classify(int) {
classify_out.fetch_add(1, std::memory_order_relaxed);
}
// ── main ──────────────────────────────────────────────────────────────────────
int main() {
SharedResource<GPU> gpu;
g_gpu = &gpu;
// ── Detection network ─────────────────────────────────────────────────────
auto src_det = make_node<source_detect, "source_detect">(4);
auto inf_det = make_node<run_detect, "run_detect" >(4);
auto snk_det = make_node<sink_detect, "sink_detect" >(4);
auto net_detect = make_network(
edge(src_det.output<0>(), inf_det.input<0>()),
edge(inf_det.output<0>(), snk_det.input<0>())
);
// ── Classification network ────────────────────────────────────────────────
auto src_cls = make_node<source_classify, "source_classify">(4);
auto inf_cls = make_node<run_classify, "run_classify" >(4);
auto snk_cls = make_node<sink_classify, "sink_classify" >(4);
auto net_classify = make_network(
edge(src_cls.output<0>(), inf_cls.input<0>()),
edge(inf_cls.output<0>(), snk_cls.input<0>())
);
// ── Hub — one debug server for both networks + the shared resource ─────────
DebugHub hub(9090);
hub.register_network("detect", net_detect);
hub.register_network("classify", net_classify);
hub.register_resource("gpu", &gpu);
net_detect.start();
net_classify.start();
hub.start();
std::cout << "Running — open http://localhost:9090\n"
<< "Tabs: [All Networks] [detect] [classify]\n"
<< "Press Enter to stop.\n";
std::cin.get();
net_detect.stop();
net_classify.stop();
std::cout << "\nResults:\n"
<< " detect: " << detect_out.load() << " frames\n"
<< " classify: " << classify_out.load() << " frames\n";
return 0;
}
#else // no KPN_WEB_DEBUG
#include <iostream>
int main() {
std::cerr << "This example requires KPN_WEB_DEBUG.\n"
<< "Rebuild with: cmake -DKPN_WEB_DEBUG=ON ..\n";
return 1;
}
#endif
examples/15_node_error_handler/main.cpp
+75
View File
@@ -0,0 +1,75 @@
// Example 15 — Per-node Error Handler
//
// Demonstrates set_error_handler() for deciding whether a network can
// continue when a node throws an exception.
//
// The "validator" node rejects even numbers by throwing std::runtime_error.
// Its error handler logs the failure and returns true (skip & continue),
// so odd numbers still flow through to the sink.
//
// Compare: a second handler (commented below) returns false instead,
// which stops the node and gracefully shuts the downstream side down.
//
// Pipeline: [source] --int--> [validator] --int--> [sink]
#include <kpn/kpn.hpp>
#include <chrono>
#include <iostream>
#include <thread>
static int counter = 0;
static int source() {
std::this_thread::sleep_for(std::chrono::milliseconds(20));
return ++counter;
}
static int validate(int x) {
if (x % 2 == 0)
throw std::runtime_error("even number rejected: " + std::to_string(x));
return x;
}
static int received = 0;
static void sink(int x) {
std::cout << " processed: " << x << '\n';
++received;
}
int main() {
using namespace kpn;
auto src = make_node<source> ();
auto proc = make_node<validate>();
auto snk = make_node<sink> ();
// --8<-- [start:error_handler]
// Return true → skip this invocation, keep the node running.
// Return false → stop the node (downstream drains then also stops).
proc.set_error_handler([](std::string_view name, std::exception_ptr ep) {
try { std::rethrow_exception(ep); }
catch (const std::exception& e) {
std::cerr << "[" << name << "] skipping item — " << e.what() << '\n';
}
return true;
});
// --8<-- [end:error_handler]
Network net;
net.add("source", src)
.add("validator", proc)
.add("sink", snk)
.connect("source", src.output<0>(), "validator", proc.input<0>())
.connect("validator", proc.output<0>(), "sink", snk.input<0>())
.build();
std::cout << "source emits 1..N; validator rejects even numbers.\n"
<< "Error messages on stderr, accepted items on stdout.\n\n";
net.start();
std::this_thread::sleep_for(std::chrono::milliseconds(300));
net.stop();
std::cout << "\nItems accepted by sink: " << received << '\n';
}
examples/16_event_callbacks/main.cpp
+83
View File
@@ -0,0 +1,83 @@
// Example 16 — Event Callbacks: overflow and node-stopped signals
//
// Two complementary observation mechanisms:
//
// 1. Per-node overflow callback set_overflow_callback()
// Fired (with a timestamp) when a node's output channel is full and an
// item is dropped. Useful for targeted monitoring of a specific node.
//
// 2. Network-level event handler net.set_event_handler()
// Aggregate callback covering every node: receives the node name, a
// NodeEvent (Overflow or Closed), and a timestamp. Register once and
// observe the whole network.
//
// Pipeline: [fast_source] --int--> [slow_sink]
//
// fast_source produces at ~500 items/s; slow_sink consumes at ~20 items/s.
// The channel capacity is 3, so overflows appear within milliseconds.
#include <kpn/kpn.hpp>
#include <atomic>
#include <chrono>
#include <iostream>
#include <thread>
using namespace kpn;
using namespace std::chrono;
// ── Node functions ────────────────────────────────────────────────────────────
// --8<-- [start:node_fns]
static std::atomic<int> g_seq{0};
static int fast_source() {
std::this_thread::sleep_for(milliseconds(2)); // ~500/s
return g_seq.fetch_add(1);
}
static void slow_sink(int x) {
std::this_thread::sleep_for(milliseconds(50)); // ~20/s
std::cout << " consumed: " << x << '\n';
}
// --8<-- [end:node_fns]
// ── main ──────────────────────────────────────────────────────────────────────
int main() {
auto src = make_node<fast_source>(/*capacity=*/3);
auto snk = make_node<slow_sink> (/*capacity=*/3);
// --8<-- [start:per_node_callback]
// Per-node overflow callback — no node name needed, known at registration.
std::atomic<int> overflow_count{0};
src.set_overflow_callback([&](steady_clock::time_point ts) {
auto ms = duration_cast<milliseconds>(ts.time_since_epoch()).count();
std::cerr << "[overflow] fast_source at t=" << ms << "ms\n";
overflow_count.fetch_add(1);
});
// --8<-- [end:per_node_callback]
Network net;
// --8<-- [start:network_event_handler]
// Network-level aggregate handler — covers every node, includes node name.
net.set_event_handler([](std::string_view name, NodeEvent ev,
steady_clock::time_point ts) {
auto ms = duration_cast<milliseconds>(ts.time_since_epoch()).count();
std::string_view kind = (ev == NodeEvent::Overflow) ? "overflow" : "closed";
std::cerr << "[net:" << kind << "] node=" << name << " t=" << ms << "ms\n";
});
// --8<-- [end:network_event_handler]
net.add("source", src)
.add("sink", snk)
.connect("source", src.output<0>(), "sink", snk.input<0>())
.build()
.start();
std::this_thread::sleep_for(milliseconds(300));
net.stop();
std::cout << "\nTotal overflows observed by per-node callback: "
<< overflow_count.load() << '\n';
}
examples/CMakeLists.txt
+83
View File
@@ -0,0 +1,83 @@
cmake_minimum_required(VERSION 3.21)
# Build an example and register it as a CTest smoke test.
# Examples that are self-terminating (fixed sleep → net.stop()) pass when
# they exit 0 within TIMEOUT seconds. OpenCV/UI examples are excluded.
function(kpn_example name)
add_executable(${name} ${name}/main.cpp)
target_link_libraries(${name} PRIVATE kpn)
add_test(NAME example_${name} COMMAND ${name})
set_tests_properties(example_${name} PROPERTIES
TIMEOUT 15
LABELS examples
)
endfunction()
# Register a Python example script as a CTest smoke test. Runs the script with
# PYTHONPATH pointing at the freshly-built kpn_python module, so it does not
# depend on the caller's working directory or a hard-coded "build/python" path.
function(kpn_python_example name)
if(NOT KPN_BUILD_PYTHON)
return()
endif()
add_test(
NAME example_${name}
COMMAND ${CMAKE_COMMAND} -E env
"PYTHONPATH=$<TARGET_FILE_DIR:kpn_python>"
${Python_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/${name}/example.py
)
set_tests_properties(example_${name} PROPERTIES
TIMEOUT 15
LABELS examples
)
endfunction()
kpn_example(01_hello_pipeline)
kpn_example(02_named_ports)
kpn_example(03_multi_output)
kpn_example(04_storage_policy)
kpn_example(05_error_handling)
kpn_example(06_watchdog)
set_tests_properties(example_06_watchdog PROPERTIES TIMEOUT 40)
kpn_example(10_static_hello_pipeline)
kpn_example(11_static_fanout)
kpn_example(15_node_error_handler)
kpn_example(16_event_callbacks)
if(KPN_WEB_DEBUG)
kpn_target_enable_web_debug(06_watchdog)
add_executable(14_debug_hub 14_debug_hub/main.cpp)
target_link_libraries(14_debug_hub PRIVATE kpn)
kpn_target_enable_web_debug(14_debug_hub)
endif()
# 07 and 08 are Python scripts — no compiled target, but run as smoke tests.
kpn_python_example(07_python_network)
kpn_python_example(08_python_subport)
# 09 requires OpenCV — only build if found
find_package(OpenCV QUIET COMPONENTS core imgproc highgui videoio)
if(OpenCV_FOUND)
# Hybrid Python example: kpn_opencv module (requires both OpenCV and nanobind)
if(KPN_BUILD_PYTHON)
nanobind_add_module(kpn_opencv 09_opencv_cellshade/kpn_opencv.cpp)
target_link_libraries(kpn_opencv PRIVATE kpn ${OpenCV_LIBS})
target_compile_definitions(kpn_opencv PRIVATE KPN_BUILD_PYTHON)
message(STATUS "KPN++ kpn_opencv Python module: building (OpenCV ${OpenCV_VERSION})")
endif()
add_executable(09_opencv_cellshade 09_opencv_cellshade/main.cpp)
target_link_libraries(09_opencv_cellshade PRIVATE kpn ${OpenCV_LIBS})
add_executable(12_static_cellshade 12_static_cellshade/main.cpp)
target_link_libraries(12_static_cellshade PRIVATE kpn ${OpenCV_LIBS})
add_executable(13_debug_cellshade 13_debug_cellshade/main.cpp)
target_link_libraries(13_debug_cellshade PRIVATE kpn ${OpenCV_LIBS})
if(KPN_WEB_DEBUG)
kpn_target_enable_web_debug(09_opencv_cellshade)
kpn_target_enable_web_debug(13_debug_cellshade)
endif()
message(STATUS "KPN++ example 09_opencv_cellshade: OpenCV ${OpenCV_VERSION} found — building")
else()
message(STATUS "KPN++ example 09_opencv_cellshade: OpenCV not found — skipping")
endif()
examples/index.html
-1039
View File
File diff suppressed because it is too large Load Diff
fanout/index.html
-1013
View File
File diff suppressed because it is too large Load Diff
getting-started/index.html
-1148
View File
File diff suppressed because it is too large Load Diff
include/kpn/branch.hpp
+379
View File
@@ -0,0 +1,379 @@
#pragma once
#include "channel.hpp"
#include "diagnostics.hpp"
#include "inode.hpp"
#include "port.hpp"
#include "traits.hpp"
#include <array>
#include <atomic>
#include <chrono>
#include <functional>
#include <memory>
#include <thread>
namespace kpn {
// ── Lossless single-output delivery ───────────────────────────────────────────
//
// Shared by RouterNode and FilterNode, which each deliver a value to exactly one
// channel. Both previously did
//
// try { ch->push(val); } catch (const ChannelOverflowError&) {}
//
// which discards the value whenever the consumer is behind. 6595e6e made node
// outputs lossless, 28e0667 stopped them parking a worker, and a8cfe73 did the
// same for FanoutNode — these two were in none of them, and were the last
// remaining users of the throwing push() on a data path.
//
// A dropped item does not degrade a downstream result, it silently changes one.
// Worse, a dropped *sentinel* wedges the pipeline outright: EOF is what tells
// every downstream node to shut down, and there is nothing after it to retry.
// A filter that passes EOF by predicate but drops it by backpressure is a
// pipeline that never terminates.
//
// So sentinels go out-of-band via push_sentinel (a dedicated slot that consumes
// no ring capacity and cannot overflow), and everything else is retried until
// taken. Like FanoutNode and unlike a pool node, these own a private thread, so
// waiting here costs no scheduler worker and needs no space-callback park.
// stop_flag_ is rechecked every pass so teardown cannot hang on a full output.
//
// `parked` receives the time spent waiting, which the caller charges to blocked
// rather than exec — a parked node is idle, and charging it to exec reports the
// node as busy exactly when it is the one being held up.
//
// Returns false if stopped with the value undelivered.
template<typename T>
bool deliver_one(Channel<T>* ch, T& val, const std::atomic<bool>& stop_flag,
duration_t& parked) {
if (is_sentinel_value(val)) {
ch->push_sentinel(std::move(val));
return true;
}
const auto park_from = clock_t::now();
for (;;) {
switch (ch->try_push(val)) {
case Channel<T>::PushResult::Taken:
parked = duration_t(clock_t::now() - park_from);
return true;
case Channel<T>::PushResult::Closed:
// Nobody is listening any more; the channel has recorded the
// drop. Retrying would spin until teardown noticed.
parked = duration_t(clock_t::now() - park_from);
return false;
case Channel<T>::PushResult::Full:
break; // fall through to the retry logic
}
if (stop_flag.load(std::memory_order_relaxed)) {
// Teardown with work in hand and the output still full. One last
// throwing push, purely so the channel's own stats record the
// overflow — the point of the lossless path is that a loss is never
// invisible, and a silent return here would reintroduce exactly the
// hole this function exists to close.
try { ch->push(std::move(val)); }
catch (const ChannelOverflowError&) {}
parked = duration_t(clock_t::now() - park_from);
return false;
}
std::this_thread::sleep_for(std::chrono::microseconds(50));
}
}
// ── RouterNode ────────────────────────────────────────────────────────────────
//
// Reads one item and pushes it to exactly one of N output channels, chosen by
// selector(item). If selector returns >= N the item is silently dropped.
//
// Usage:
// auto router = make_router<Image, 3>(
// [](const Image& img) -> std::size_t { return img.stream_id % 3; });
// net.connect("src", src.output<0>(), "router", router.input<0>())
// .connect("router", router.output<0>(), "nodeA", nodeA.input<0>())
// .connect("router", router.output<1>(), "nodeB", nodeB.input<0>())
// .connect("router", router.output<2>(), "nodeC", nodeC.input<0>());
template<typename T, std::size_t N, std::size_t Id = 0>
class RouterNode : public INode {
public:
using Selector = std::function<std::size_t(const T&)>;
using args_tuple = std::tuple<T>;
using return_tuple = repeat_tuple_t<T, N>;
using return_raw = return_tuple;
static constexpr std::size_t input_count = 1;
static constexpr std::size_t output_count = N;
static constexpr std::size_t unique_tag = Id;
static constexpr bool is_router_node = true;
explicit RouterNode(Selector sel, std::size_t fifo_capacity = 5)
: selector_(std::move(sel))
, fifo_capacity_(fifo_capacity)
{
input_ch_ = std::make_shared<Channel<T>>(fifo_capacity);
}
~RouterNode() override { stop(); }
// ── INode ─────────────────────────────────────────────────────────────────
void start() override {
input_ch_->enable();
stop_flag_.store(false, std::memory_order_relaxed);
thread_ = std::jthread([this](std::stop_token) { run_loop(); });
}
void stop() override {
stop_flag_.store(true, std::memory_order_relaxed);
input_ch_->disable();
if (thread_.joinable()) thread_.request_stop(), thread_.join();
}
bool running() const override {
return thread_.joinable() && !stop_flag_.load(std::memory_order_relaxed);
}
void set_name(std::string name) override { name_ = std::move(name); }
const NodeStats& stats() const override { return stats_; }
NodeSnapshot node_snapshot(const std::string& name, double elapsed_s) const override {
uint64_t frames = stats_.frames_processed.load(std::memory_order_relaxed);
double exec_ms = stats_.ema_exec_us.load(std::memory_order_relaxed) / 1000.0;
double blocked_ms = stats_.total_blocked_us.load(std::memory_order_relaxed) / 1000.0;
double total_ms = exec_ms + blocked_ms;
return {name, frames, exec_ms,
stats_.max_exec_us.load(std::memory_order_relaxed) / 1000.0,
blocked_ms,
elapsed_s > 0 ? frames / elapsed_s : 0.0,
stats_.total_cpu_us.load(std::memory_order_relaxed) / 1000.0,
total_ms > 0 ? 100.0 * exec_ms / total_ms : 0.0};
}
// ── Port access ───────────────────────────────────────────────────────────
template<std::size_t I = 0>
InputPort<RouterNode, I> input() {
static_assert(I == 0, "RouterNode has exactly one input");
return {*this};
}
template<std::size_t I>
OutputPort<RouterNode, I> output() {
static_assert(I < N, "RouterNode output index out of range");
return {*this};
}
// ── Internal channel accessors (called by Network::connect) ───────────────
template<std::size_t I>
Channel<T>& input_channel() {
static_assert(I == 0);
return *input_ch_;
}
template<std::size_t I>
void set_input_channel(std::shared_ptr<Channel<T>> ch) {
static_assert(I == 0);
input_ch_ = std::move(ch);
}
template<std::size_t I>
void set_output_channel(Channel<T>* ch) {
static_assert(I < N);
out_channels_[I] = ch;
}
private:
void run_loop() {
while (!stop_flag_.load(std::memory_order_relaxed)) {
try {
auto t0 = clock_t::now();
T val = input_ch_->pop();
auto t1 = clock_t::now();
auto cpu0 = NodeStats::cpu_now();
// An out-of-range selector still drops by design (documented on
// the class): the item was routed nowhere, not lost to a full
// channel. Only the latter is what deliver_one exists to stop.
std::size_t idx = selector_(val);
duration_t parked{0};
bool delivered = true;
if (idx < N && out_channels_[idx])
delivered = deliver_one(out_channels_[idx], val, stop_flag_, parked);
auto cpu1 = NodeStats::cpu_now();
auto t2 = clock_t::now();
stats_.record_exec(duration_t(t2 - t1) - parked,
duration_t(t1 - t0) + parked, cpu0, cpu1);
if (!delivered) break;
} catch (const ChannelClosedError&) {
break;
}
}
}
std::string name_;
std::size_t fifo_capacity_;
Selector selector_;
std::shared_ptr<Channel<T>> input_ch_;
std::array<Channel<T>*, N> out_channels_{};
std::atomic<bool> stop_flag_{false};
std::jthread thread_;
NodeStats stats_;
};
// ── FilterNode ────────────────────────────────────────────────────────────────
//
// Reads one item and pushes it downstream only when pred(item) is true.
// Dropped items are not counted as processed frames.
//
// Usage:
// auto filt = make_filter<Frame>([](const Frame& f) { return f.valid; });
// net.connect("src", src.output<0>(), "filt", filt.input<0>())
// .connect("filt", filt.output<0>(), "dst", dst.input<0>());
template<typename T, std::size_t Id = 0>
class FilterNode : public INode {
public:
using Predicate = std::function<bool(const T&)>;
using args_tuple = std::tuple<T>;
using return_tuple = std::tuple<T>;
using return_raw = return_tuple;
static constexpr std::size_t input_count = 1;
static constexpr std::size_t output_count = 1;
static constexpr std::size_t unique_tag = Id;
static constexpr bool is_filter_node = true;
explicit FilterNode(Predicate pred, std::size_t fifo_capacity = 5)
: pred_(std::move(pred))
, fifo_capacity_(fifo_capacity)
{
input_ch_ = std::make_shared<Channel<T>>(fifo_capacity);
}
~FilterNode() override { stop(); }
// ── INode ─────────────────────────────────────────────────────────────────
void start() override {
input_ch_->enable();
stop_flag_.store(false, std::memory_order_relaxed);
thread_ = std::jthread([this](std::stop_token) { run_loop(); });
}
void stop() override {
stop_flag_.store(true, std::memory_order_relaxed);
input_ch_->disable();
if (thread_.joinable()) thread_.request_stop(), thread_.join();
}
bool running() const override {
return thread_.joinable() && !stop_flag_.load(std::memory_order_relaxed);
}
void set_name(std::string name) override { name_ = std::move(name); }
const NodeStats& stats() const override { return stats_; }
NodeSnapshot node_snapshot(const std::string& name, double elapsed_s) const override {
uint64_t frames = stats_.frames_processed.load(std::memory_order_relaxed);
double exec_ms = stats_.ema_exec_us.load(std::memory_order_relaxed) / 1000.0;
double blocked_ms = stats_.total_blocked_us.load(std::memory_order_relaxed) / 1000.0;
double total_ms = exec_ms + blocked_ms;
return {name, frames, exec_ms,
stats_.max_exec_us.load(std::memory_order_relaxed) / 1000.0,
blocked_ms,
elapsed_s > 0 ? frames / elapsed_s : 0.0,
stats_.total_cpu_us.load(std::memory_order_relaxed) / 1000.0,
total_ms > 0 ? 100.0 * exec_ms / total_ms : 0.0};
}
// ── Port access ───────────────────────────────────────────────────────────
template<std::size_t I = 0>
InputPort<FilterNode, I> input() {
static_assert(I == 0, "FilterNode has exactly one input");
return {*this};
}
template<std::size_t I = 0>
OutputPort<FilterNode, I> output() {
static_assert(I == 0, "FilterNode has exactly one output");
return {*this};
}
// ── Internal channel accessors (called by Network::connect) ───────────────
template<std::size_t I>
Channel<T>& input_channel() {
static_assert(I == 0);
return *input_ch_;
}
template<std::size_t I>
void set_input_channel(std::shared_ptr<Channel<T>> ch) {
static_assert(I == 0);
input_ch_ = std::move(ch);
}
template<std::size_t I>
void set_output_channel(Channel<T>* ch) {
static_assert(I == 0);
out_ch_ = ch;
}
private:
void run_loop() {
while (!stop_flag_.load(std::memory_order_relaxed)) {
try {
auto t0 = clock_t::now();
T val = input_ch_->pop();
auto t1 = clock_t::now();
auto cpu0 = NodeStats::cpu_now();
// A value the predicate rejects is dropped by design and is not
// counted as a processed frame. One it accepts is now delivered
// losslessly — including a sentinel, which a filter typically
// passes unconditionally so downstream can shut down, and which
// the old throwing push discarded whenever the output was full.
if (pred_(val) && out_ch_) {
duration_t parked{0};
const bool delivered = deliver_one(out_ch_, val, stop_flag_, parked);
auto cpu1 = NodeStats::cpu_now();
auto t2 = clock_t::now();
stats_.record_exec(duration_t(t2 - t1) - parked,
duration_t(t1 - t0) + parked, cpu0, cpu1);
if (!delivered) break;
}
} catch (const ChannelClosedError&) {
break;
}
}
}
std::string name_;
std::size_t fifo_capacity_;
Predicate pred_;
std::shared_ptr<Channel<T>> input_ch_;
Channel<T>* out_ch_{nullptr};
std::atomic<bool> stop_flag_{false};
std::jthread thread_;
NodeStats stats_;
};
// ── Factories ─────────────────────────────────────────────────────────────────
template<typename T, std::size_t N>
RouterNode<T, N> make_router(std::function<std::size_t(const T&)> sel,
std::size_t capacity = 5) {
return RouterNode<T, N, 0>(std::move(sel), capacity);
}
template<typename T>
FilterNode<T> make_filter(std::function<bool(const T&)> pred,
std::size_t capacity = 5) {
return FilterNode<T, 0>(std::move(pred), capacity);
}
} // namespace kpn
include/kpn/channel.hpp
+524
View File
@@ -0,0 +1,524 @@
#pragma once
#include "diagnostics.hpp"
#include <atomic>
#include <chrono>
#include <cstdint>
#include <functional>
#include <memory>
#include <stdexcept>
#include <string>
#include <thread>
#include <type_traits>
namespace kpn {
// ── Data size trait ───────────────────────────────────────────────────────────
// Returns the number of bytes of logical payload carried by a value.
// Defaults to sizeof(T), which is correct for PODs and fixed-size types.
// Specialize for heap-owning types (e.g. cv::Mat) to get accurate bandwidth:
//
// template<> struct kpn::ChannelDataSize<cv::Mat> {
// static std::size_t bytes(const cv::Mat& m) { return m.total() * m.elemSize(); }
// };
template<typename T>
struct ChannelDataSize {
static std::size_t bytes(const T&) { return sizeof(T); }
};
// ── Storage policy ────────────────────────────────────────────────────────────
template<typename T>
struct channel_storage_policy {
static constexpr bool by_value =
std::is_trivially_copyable_v<T> && sizeof(T) <= 8;
};
template<typename T>
using channel_storage_t = std::conditional_t<
channel_storage_policy<T>::by_value,
T,
std::shared_ptr<const T>
>;
// ── Exceptions ────────────────────────────────────────────────────────────────
class ChannelOverflowError : public std::runtime_error {
public:
explicit ChannelOverflowError(std::size_t capacity)
: std::runtime_error("channel overflow: capacity " + std::to_string(capacity) +
" exceeded") {}
ChannelOverflowError(std::size_t capacity, std::string context)
: std::runtime_error(std::move(context) + ": capacity " + std::to_string(capacity) +
" exceeded") {}
};
class ChannelClosedError : public std::runtime_error {
public:
ChannelClosedError() : std::runtime_error("channel closed") {}
};
// Nothing available *right now* on a channel that is still open. Distinct from
// ChannelClosedError, which means upstream is finished and never coming back.
//
// Conflating the two is expensive in one direction only: a consumer that reads
// "empty" as "closed" stops a live node permanently, and because a stopping
// node disables its own inputs and outputs, one benign empty read takes the
// rest of the pipeline with it. The reverse costs nothing.
class ChannelEmptyError : public std::runtime_error {
public:
ChannelEmptyError() : std::runtime_error("channel empty") {}
};
// ── CPU pause hint ────────────────────────────────────────────────────────────
// Signals the CPU that this is a spin-wait loop, improving HT sibling throughput
// and preventing branch-predictor thrash on x86. Falls back to a compiler barrier.
[[maybe_unused]] static void spin_hint() noexcept {
#if defined(__x86_64__) || defined(__i386__)
__asm__ volatile("pause" ::: "memory");
#elif defined(__aarch64__) || defined(__arm__)
__asm__ volatile("yield" ::: "memory");
#else
std::atomic_signal_fence(std::memory_order_seq_cst);
#endif
}
// ── Channel ───────────────────────────────────────────────────────────────────
// SPSC ring buffer with atomic wait/notify and configurable spin-before-sleep.
//
// `spin_count` (constructor arg, default 200): number of pause-hint iterations
// before falling back to atomic::wait (futex). At ~20 ns/pause on x86 this is
// ~4 µs. Set to 0 to disable spinning (useful for power-constrained or
// predominantly-idle pipelines).
//
// Memory ordering contract (SPSC):
// push(): tail_.store(release) pairs with pop()'s tail_.load(acquire)
// head_.load(acquire) pairs with pop()'s head_.store(release)
// pop(): head_.store(release) pairs with push()'s head_.load(acquire)
// tail_.load(acquire) pairs with push()'s tail_.store(release)
template<typename T>
class Channel {
public:
using storage_type = channel_storage_t<T>;
explicit Channel(std::size_t capacity = 5, std::size_t spin_count = 200)
: capacity_(capacity), spin_count_(spin_count)
{
std::size_t rs = 1;
while (rs <= capacity) rs <<= 1; // smallest power-of-2 > capacity
ring_mask_ = rs - 1;
buf_ = std::make_unique<storage_type[]>(rs);
}
Channel(const Channel&) = delete;
Channel& operator=(const Channel&) = delete;
// Push a value.
// - If channel is disabled (accepting_ == false): silently drop.
// - If channel is full (fill >= capacity_): throw ChannelOverflowError.
void push(T value) {
if (!accepting_.load(std::memory_order_relaxed)) {
stats_.record_drop();
return;
}
const std::size_t data_bytes = ChannelDataSize<T>::bytes(value);
const std::size_t t = tail_.load(std::memory_order_relaxed);
const std::size_t h = head_.load(std::memory_order_acquire);
if (!accepting_.load(std::memory_order_acquire)) {
stats_.record_drop();
return;
}
if (t - h >= capacity_) {
stats_.record_overflow();
throw ChannelOverflowError(capacity_);
}
const bool was_empty = (t == h);
buf_[t & ring_mask_] = make_storage(std::move(value));
tail_.store(t + 1, std::memory_order_release);
stats_.record_push(t - h + 1, data_bytes);
wake_.fetch_add(1, std::memory_order_release);
wake_.notify_one();
if (was_empty && push_callback_)
push_callback_();
}
/// Called when a pop frees a slot in a previously-full ring.
///
/// The mirror of `set_push_callback`, and it exists for the same reason:
/// a producer must be able to *park* rather than spin. Without it the only
/// lossless option is `push_blocking`, which sleeps inside the caller's
/// thread — and when that thread is a scheduler worker, parking it starves
/// every node pinned to it (see the hold-and-wait note on push_sentinel).
void set_space_callback(std::function<void()> cb) { space_callback_ = std::move(cb); }
/// True when a push would currently succeed. Used to close the lost-wakeup
/// race: a producer that parks must re-check after clearing its queued flag,
/// because a space_callback fired in between would otherwise be swallowed.
bool has_space() const {
return tail_.load(std::memory_order_relaxed) -
head_.load(std::memory_order_acquire) < capacity_;
}
/// Outcome of a non-blocking push.
///
/// try_push used to return bool, and returned *true* for a closed channel —
/// so "delivered" and "discarded because nobody is listening" were the same
/// answer. Both mean "stop trying", which is why the callers were correct,
/// but neither they nor the producer's own accounting could tell a value
/// that arrived from one that was thrown away. Only the channel's drop
/// counter knew.
enum class PushResult { Taken, Full, Closed };
/// Non-blocking, lossless push. Returns Full when the ring is full, having
/// changed nothing — the caller keeps the value and retries when woken.
PushResult try_push(T& value) {
if (!accepting_.load(std::memory_order_acquire)) {
stats_.record_drop();
return PushResult::Closed;
}
const std::size_t t = tail_.load(std::memory_order_relaxed);
const std::size_t h = head_.load(std::memory_order_acquire);
if (t - h >= capacity_) return PushResult::Full;
const std::size_t data_bytes = ChannelDataSize<T>::bytes(value);
const bool was_empty = (t == h);
buf_[t & ring_mask_] = make_storage(std::move(value));
tail_.store(t + 1, std::memory_order_release);
stats_.record_push(t - h + 1, data_bytes);
wake_.fetch_add(1, std::memory_order_release);
wake_.notify_one();
if (was_empty && push_callback_) push_callback_();
return PushResult::Taken;
}
// Lossless push with BACKPRESSURE: if the ring is full, wait for the consumer to
// drain instead of dropping (the throwing push()) — the producer just runs slower.
// Use when every value must be delivered (e.g. replaying a dump for scoring, where
// a dropped frame silently corrupts the result). SPSC: only the sole producer may
// call it. Returns false if the channel was disabled while waiting.
bool push_blocking(T value) {
for (;;) {
if (!accepting_.load(std::memory_order_acquire)) {
stats_.record_drop();
return false;
}
const std::size_t t = tail_.load(std::memory_order_relaxed);
const std::size_t h = head_.load(std::memory_order_acquire);
if (t - h < capacity_) { // space available → normal push
const std::size_t data_bytes = ChannelDataSize<T>::bytes(value);
const bool was_empty = (t == h);
buf_[t & ring_mask_] = make_storage(std::move(value));
tail_.store(t + 1, std::memory_order_release);
stats_.record_push(t - h + 1, data_bytes);
wake_.fetch_add(1, std::memory_order_release);
wake_.notify_one();
if (was_empty && push_callback_) push_callback_();
return true;
}
// full: yield briefly and retry (consumer will drain)
std::this_thread::sleep_for(std::chrono::microseconds(50));
}
}
// Lossless, non-blocking delivery for a must-deliver control token (EOF).
//
// A sentinel is stored out-of-band — in a dedicated slot that does NOT
// consume ring capacity — so this can never overflow and never blocks the
// caller. That distinction is essential: each KPN node has a single worker
// thread, so a *blocking* push would park that thread and stop it draining
// its own input, cascading into a hold-and-wait deadlock under backpressure.
// Setting a flag and returning keeps the worker free to keep popping.
//
// The consumer's pop() drains the ring first, then delivers this sentinel,
// preserving ordering (EOF arrives after all data pushed before it).
//
// Only the sole producer may call it (SPSC contract, same as push()).
//
// The slot holds exactly one undelivered token. A second offered before the
// first is taken is refused, not queued and not overwritten: two control
// tokens on one channel means the stream ended twice, which is a caller
// protocol error rather than backpressure, and silently coalescing them
// would hide it.
/// Outcome of offering a sentinel. SlotBusy is a protocol error, not
/// backpressure: it means a second control token was offered while the
/// first was still undelivered, and a channel carries at most one.
enum class SentinelResult { Taken, Closed, SlotBusy };
/// Non-consuming form. `value` is left untouched unless the result is
/// Taken, so a refused token is still the caller's to report.
SentinelResult try_push_sentinel(T& value) {
if (!accepting_.load(std::memory_order_acquire)) {
stats_.record_drop();
return SentinelResult::Closed;
}
// Refuse rather than overwrite. Overwriting lost the first token
// silently, and worse, wrote eof_value_ while the consumer could be
// moving the previous one out of it — a data race on the storage, which
// for a shared_ptr payload is a torn refcount rather than a stale read.
//
// Checking here is what makes the slot a correct SPSC handshake: the
// producer is the only writer of eof_value_ and the only one that sets
// has_eof_, the consumer is the only one that clears it, so observing
// false here means the consumer has finished with the storage and will
// not touch it again until this store publishes the next token.
if (has_eof_.load(std::memory_order_acquire)) {
stats_.record_drop();
return SentinelResult::SlotBusy;
}
eof_value_ = make_storage(std::move(value));
has_eof_.store(true, std::memory_order_release);
// Wake a consumer blocked in pop(): the sentinel is now deliverable even
// though the ring may be empty.
wake_.fetch_add(1, std::memory_order_release);
wake_.notify_one();
if (push_callback_) push_callback_();
return SentinelResult::Taken;
}
/// Consuming convenience form. Returns false when the token was not stored,
/// whether because the channel is closed or because one is already pending.
bool push_sentinel(T value) {
return try_push_sentinel(value) == SentinelResult::Taken;
}
// Blocking pop. Returns when an item is available.
// Throws ChannelClosedError if the channel is disabled (regardless of fill).
T pop() {
for (;;) {
// Snapshot wake_ BEFORE reading tail_ to prevent lost wakeups.
const uint32_t w = wake_.load(std::memory_order_relaxed);
const std::size_t h = head_.load(std::memory_order_relaxed);
std::size_t t = tail_.load(std::memory_order_acquire);
// If empty, spin before sleeping: avoids the futex when the next item
// arrives within the spin window (~4 µs at default spin_count=200 on x86).
if (h == t) {
// Ring drained — deliver any pending out-of-band sentinel (EOF)
// now, so it always arrives after the data pushed before it.
//
// Re-confirm emptiness against a fresh tail_ first: the snapshot
// at the top of the loop may be stale (the producer can push more
// values *and* the sentinel in the window since), and the sentinel
// must never jump ahead of ring values pushed before it. The spin
// and post-spin takes below already reload tail_ on the line above
// them; this is the one take that used the loop-top snapshot.
if (h == tail_.load(std::memory_order_acquire)) {
T s; if (take_sentinel(s)) return s;
}
if (!accepting_.load(std::memory_order_acquire))
throw ChannelClosedError{};
for (std::size_t si = 0; si < spin_count_; ++si) {
spin_hint();
t = tail_.load(std::memory_order_acquire);
if (t != h) break;
{ T s; if (take_sentinel(s)) return s; }
if (!accepting_.load(std::memory_order_relaxed))
throw ChannelClosedError{};
}
if (h == t) {
// Still empty after spin — sleep until push()/push_sentinel()
// or disable() fires. Re-check tail and the sentinel after
// loading w to guard against a lost wakeup.
if (tail_.load(std::memory_order_acquire) != h) continue;
if (has_eof_.load(std::memory_order_acquire)) continue;
wake_.wait(w, std::memory_order_relaxed);
continue;
}
}
// Item available (found immediately or during spin).
if (!accepting_.load(std::memory_order_acquire))
throw ChannelClosedError{};
T value = extract(std::move(buf_[h & ring_mask_]));
head_.store(h + 1, std::memory_order_release);
// A slot just freed: wake any producer parked on this channel.
if (t - h >= capacity_ && space_callback_) space_callback_();
stats_.record_pop();
return value;
}
}
// Non-blocking pop with timeout. For watchdog/display use only.
bool try_pop(T& out, std::chrono::milliseconds timeout) {
const auto deadline = std::chrono::steady_clock::now() + timeout;
for (;;) {
if (try_pop_now(out)) return true;
if (!accepting_.load(std::memory_order_relaxed)) return false;
if (std::chrono::steady_clock::now() >= deadline) return false;
std::this_thread::sleep_for(std::chrono::microseconds(50));
}
}
// Immediate non-blocking pop. Returns false if the ring is empty.
// Once the ring is drained, delivers any pending out-of-band sentinel (EOF)
// so pool nodes — which pop only via this path — still receive the token.
bool try_pop_now(T& out) {
const std::size_t h = head_.load(std::memory_order_relaxed);
const std::size_t t = tail_.load(std::memory_order_acquire);
if (h == t)
return take_sentinel(out);
out = extract(std::move(buf_[h & ring_mask_]));
head_.store(h + 1, std::memory_order_release);
stats_.record_pop();
// Pool nodes pop only through here, so this is where a parked producer
// gets woken: the ring was full, and it no longer is.
if (t - h >= capacity_ && space_callback_) space_callback_();
return true;
}
// Enable the channel (called by consumer node on start()).
void enable() {
accepting_.store(true, std::memory_order_relaxed);
}
// Disable the channel: stop accepting new pushes, unblock any waiting pop().
// Items already in the ring are abandoned and freed when the Channel is destroyed.
void disable() {
accepting_.store(false, std::memory_order_release);
wake_.fetch_add(1, std::memory_order_release);
wake_.notify_all();
}
// Register a callback fired when the queue transitions empty→non-empty.
void set_push_callback(std::function<void()> cb) {
push_callback_ = std::move(cb);
}
// Ring occupancy, derived lazily from indices — no separate counter on the
// hot path. Excludes any out-of-band sentinel (that lives outside the ring).
// head_ is loaded first, deliberately. Both indices only ever increase, so
// reading head_ before tail_ can at worst under-report a concurrent push;
// the other order can read a head_ that has advanced past the tail_ already
// sampled, and the unsigned difference then wraps to ~2^64. A caller
// polling "is this channel empty yet" against that value never terminates.
std::size_t size() const {
const std::size_t h = head_.load(std::memory_order_relaxed);
const std::size_t t = tail_.load(std::memory_order_acquire);
return t - h;
}
// A pending out-of-band sentinel (EOF) counts as consumable work here even
// though it holds no ring slot. This is what node readiness checks call, so
// a channel carrying only a sentinel still schedules its consumer's next
// fire — without this the sentinel would never be popped and the pipeline
// would deadlock at teardown.
std::size_t approx_size() const {
return size() + (has_eof_.load(std::memory_order_acquire) ? 1u : 0u);
}
std::size_t capacity() const { return capacity_; }
bool is_accepting() const { return accepting_.load(std::memory_order_relaxed); }
const ChannelStats& stats() const { return stats_; }
ChannelSnapshot snapshot(const std::string& name) const {
// head_ before tail_, for the reason given on size().
const std::size_t h = head_.load(std::memory_order_relaxed);
const std::size_t t = tail_.load(std::memory_order_acquire);
return {
name,
capacity_,
t - h,
stats_.peak_fill.load(std::memory_order_relaxed),
stats_.pushes.load(std::memory_order_relaxed),
stats_.bytes_pushed.load(std::memory_order_relaxed),
stats_.drops.load(std::memory_order_relaxed),
stats_.overflows.load(std::memory_order_relaxed),
stats_.pops.load(std::memory_order_relaxed),
sizeof(T),
};
}
private:
static storage_type make_storage(T&& v) {
if constexpr (channel_storage_policy<T>::by_value)
return std::move(v);
else
return std::make_shared<const T>(std::move(v));
}
static T extract(storage_type&& s) {
if constexpr (channel_storage_policy<T>::by_value)
return std::move(s);
else
return *s;
}
// Consume the out-of-band sentinel if one is pending. Consumer-only.
// Called only when the ring is observed empty, so the sentinel is always
// delivered after every value pushed before it.
bool take_sentinel(T& out) {
if (!has_eof_.load(std::memory_order_acquire)) return false;
// Re-check emptiness *after* observing has_eof_, not before.
//
// Callers check the ring is empty and then call this, but the producer
// can push a value and publish the sentinel in the window between those
// two steps — so the sentinel would be delivered with a real value still
// queued behind it, breaking the "sentinel is strictly last" contract
// that downstream teardown depends on. a0c4bf5 closed the variant where
// the caller's emptiness check used a stale tail_ snapshot; this is the
// one where the check is fresh but simply too early.
//
// Checking here is what makes it sound: the producer publishes the
// sentinel with a release store *after* its ring pushes, so a consumer
// that has observed has_eof_ has also observed every tail_ advance
// before it. If the ring is non-empty now, those values genuinely
// precede the sentinel and must be delivered first.
if (head_.load(std::memory_order_relaxed)
!= tail_.load(std::memory_order_acquire))
return false;
out = extract(std::move(eof_value_));
has_eof_.store(false, std::memory_order_release);
stats_.record_pop();
return true;
}
const std::size_t capacity_;
const std::size_t spin_count_;
std::size_t ring_mask_;
std::unique_ptr<storage_type[]> buf_;
std::function<void()> push_callback_;
std::function<void()> space_callback_;
ChannelStats stats_;
// Out-of-band sentinel (EOF): stored outside the ring so its delivery never
// depends on ring capacity and never blocks the producer. Written by the
// producer (push_sentinel), read+cleared by the consumer (take_sentinel);
// has_eof_ is the publish/consume handshake.
storage_type eof_value_{};
std::atomic<bool> has_eof_{false};
// Separate cache lines: head_ is written only by the consumer;
// tail_ and wake_ are written only by the producer.
// wake_ wakes a blocked pop() on enqueue or on a pending sentinel.
alignas(64) std::atomic<std::size_t> head_{0};
alignas(64) std::atomic<std::size_t> tail_{0};
std::atomic<uint32_t> wake_{0};
std::atomic<bool> accepting_{true};
};
// ── Channel probe — type-erased snapshot accessor ─────────────────────────────
// Used by both Network and StaticNetwork for diagnostics.
struct IChannelProbe {
virtual ~IChannelProbe() = default;
virtual ChannelSnapshot snapshot() const = 0;
};
template<typename T>
struct ChannelProbe : IChannelProbe {
const Channel<T>& ch;
std::string name;
ChannelProbe(const Channel<T>& c, std::string n) : ch(c), name(std::move(n)) {}
ChannelSnapshot snapshot() const override { return ch.snapshot(name); }
};
} // namespace kpn
include/kpn/debug_hub.hpp
+456
View File
@@ -0,0 +1,456 @@
#pragma once
// Only active when KPN_WEB_DEBUG is defined.
#ifdef KPN_WEB_DEBUG
#include "diagnostics.hpp"
#include "web_debug.hpp"
#include <functional>
#include <memory>
#include <sstream>
#include <string>
#include <utility>
#include <vector>
namespace kpn {
// ── Hub HTML ──────────────────────────────────────────────────────────────────
// Multi-tab UI: one tab per registered network (force-directed graph) +
// an "All Networks" tab showing shared resource cards and a cross-network
// node table.
static const char* HUB_HTML = R"html(<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>KPN++ Debug Hub</title>
<style>
*{box-sizing:border-box}
body{margin:0;background:#1a1a2e;color:#eee;font-family:monospace}
#hdr{display:flex;align-items:center;padding:0 16px;background:#16213e;
border-bottom:1px solid #0f3460;height:44px;gap:8px;overflow-x:auto}
#hdr h1{margin:0;font-size:16px;color:#e94560;white-space:nowrap;margin-right:12px}
#tab-bar{display:flex;gap:2px;flex:1}
.tab{padding:4px 14px;border:none;background:#0f3460;color:#aaa;
cursor:pointer;font-family:monospace;font-size:11px;border-radius:2px;white-space:nowrap}
.tab.active{background:#e94560;color:#fff}
.tab:hover:not(.active){background:#1e3a6e;color:#eee}
#status{font-size:10px;color:#555;white-space:nowrap}
.panel{display:none}
.panel.active{display:block}
/* ── All Networks tab ─────────────────────────────────────── */
#panel-all{height:calc(100vh - 44px);overflow-y:auto;padding:16px;
display:none;gap:16px;grid-template-columns:300px 1fr}
#panel-all.active{display:grid;align-content:start}
#panel-all h2{font-size:11px;color:#e94560;margin:0 0 8px;
letter-spacing:1px;text-transform:uppercase}
#res-col{grid-column:1}
.res-card{background:#16213e;border:1px solid #0f3460;border-radius:4px;
padding:10px 12px;margin-bottom:8px}
.res-head{display:flex;justify-content:space-between;align-items:center}
.res-name{font-size:12px}
.badge{font-size:9px;padding:2px 6px;border-radius:2px}
.held{background:#e94560}.free{background:#4CAF50;color:#111}
.res-meta{font-size:9px;color:#666;margin-top:5px;display:flex;gap:12px;flex-wrap:wrap}
.bar-wrap{height:3px;background:#0f3460;border-radius:2px;margin-top:7px}
.bar{height:3px;border-radius:2px;transition:width 0.4s}
#nodes-col{grid-column:2;overflow-y:auto;max-height:calc(100vh - 76px)}
table{width:100%;border-collapse:collapse;font-size:10px}
th{padding:4px 8px;color:#555;border-bottom:1px solid #0f3460;text-align:left;
position:sticky;top:0;background:#1a1a2e;z-index:1}
td{padding:2px 8px;border-bottom:1px solid #16213e}
tr:hover td{background:#16213e}
.ntag{font-size:9px;background:#0f3460;padding:1px 4px;border-radius:2px;color:#4CAF50}
/* ── Per-network graph panels ─────────────────────────────── */
.graph-panel{width:100vw;height:calc(100vh - 44px)}
svg.net{width:100%;height:100%}
.node circle{stroke:#fff;stroke-width:1.5px}
.node text{font-size:11px;fill:#eee;pointer-events:none;text-anchor:middle}
.node .st{font-size:9px;fill:#aaa}
.link{fill:none;stroke-width:2px}
.lbl{font-size:9px;fill:#ccc}
#tip{position:absolute;background:#0f3460;border:1px solid #e94560;border-radius:4px;
padding:8px 12px;font-size:11px;pointer-events:none;display:none;
white-space:pre;line-height:1.6}
</style>
</head>
<body>
<div id="hdr">
<h1>KPN++ Debug Hub</h1>
<div id="tab-bar"></div>
<span id="status">connecting…</span>
</div>
<div id="panels">
<div id="panel-all" class="panel"></div>
</div>
<div id="tip"></div>
<script src="https://d3js.org/d3.v7.min.js"></script>
<script>
const R = 28;
const tip = d3.select('#tip');
const nc = ema => ema>100?'#e94560':ema>50?'#e07040':ema>10?'#f0c040':'#4CAF50';
const ec = pct => pct>=80?'#e94560':pct>=50?'#f0c040':'#4CAF50';
const ea = pct => pct>=80?'url(#a2)':pct>=50?'url(#a1)':'url(#a0)';
// ── Tab management ────────────────────────────────────────────────────────────
let activeTab = null;
function ensureTab(id, label) {
if (document.getElementById('tab-' + id)) return;
const b = document.createElement('button');
b.className = 'tab'; b.id = 'tab-' + id; b.textContent = label;
b.onclick = () => showTab(id);
document.getElementById('tab-bar').appendChild(b);
}
function showTab(id) {
activeTab = id;
document.querySelectorAll('.tab').forEach(b =>
b.classList.toggle('active', b.id === 'tab-' + id));
document.querySelectorAll('.panel').forEach(p =>
p.classList.toggle('active', p.id === 'panel-' + id));
}
// ── All Networks tab ──────────────────────────────────────────────────────────
function renderAll(data) {
const panel = document.getElementById('panel-all');
// Resources column
let rhtml = '<div id="res-col"><h2>Shared Resources</h2>';
if (!data.resources || !data.resources.length)
rhtml += '<div style="color:#444;font-size:11px">None registered</div>';
for (const r of (data.resources || [])) {
const wpct = Math.min(100, r.avg_wait_ms).toFixed(1);
const bc = r.current_waiters > 0 ? '#e94560' : '#4CAF50';
rhtml += `<div class="res-card">
<div class="res-head">
<span class="res-name">${r.name}</span>
<span class="badge ${r.held ? 'held' : 'free'}">${r.held ? 'HELD' : 'free'}</span>
</div>
<div class="res-meta">
<span>avg wait ${r.avg_wait_ms.toFixed(1)} ms</span>
<span>waiters ${r.current_waiters} / peak ${r.peak_waiters}</span>
<span>${r.acquisitions} acq</span>
</div>
<div class="bar-wrap">
<div class="bar" style="width:${wpct}%;background:${bc}"></div>
</div>
</div>`;
}
rhtml += '</div>';
// Nodes column — all networks in one table
let rows = '';
for (const net of data.networks) {
for (const n of net.nodes) {
rows += `<tr>
<td><span class="ntag">${net.name}</span></td>
<td>${n.id}</td>
<td>${n.fps.toFixed(1)}</td>
<td>${n.ema_exec_ms.toFixed(2)}</td>
<td>${n.max_exec_ms.toFixed(2)}</td>
<td>${n.blocked_ms.toFixed(2)}</td>
<td>${n.cpu_util_pct.toFixed(1)}</td>
</tr>`;
}
}
const thtml = `<div id="nodes-col"><h2>All Nodes</h2>
<table>
<tr><th>Network</th><th>Node</th><th>fps</th>
<th>exec ema (ms)</th><th>exec max (ms)</th>
<th>blocked (ms)</th><th>cpu %</th></tr>
${rows}
</table></div>`;
panel.innerHTML = rhtml + thtml;
}
// ── Per-network graph ─────────────────────────────────────────────────────────
const nets = {};
function initNet(netData) {
const name = netData.name;
const div = document.createElement('div');
div.id = 'panel-' + name;
div.className = 'panel graph-panel';
document.getElementById('panels').appendChild(div);
const svg = d3.select(div).append('svg').attr('class', 'net');
const W = () => div.clientWidth || window.innerWidth;
const H = () => div.clientHeight || (window.innerHeight - 44);
const defs = svg.append('defs');
['#4CAF50','#f0c040','#e94560'].forEach((col, i) =>
defs.append('marker').attr('id','a'+i)
.attr('viewBox','0 -5 10 10').attr('refX',10).attr('refY',0)
.attr('markerWidth',6).attr('markerHeight',6).attr('orient','auto')
.append('path').attr('d','M0,-5L10,0L0,5').attr('fill',col));
const g = svg.append('g');
svg.call(d3.zoom().on('zoom', e => g.attr('transform', e.transform)));
const nodes = netData.nodes.map(n => ({...n, x: W()/2, y: H()/2}));
const byId = Object.fromEntries(nodes.map(n => [n.id, n]));
const links = netData.edges
.map(e => ({...e, source: byId[e.source], target: byId[e.target]}))
.filter(e => e.source && e.target);
const sim = d3.forceSimulation(nodes)
.force('link', d3.forceLink(links).distance(150).strength(0.5))
.force('charge', d3.forceManyBody().strength(-350))
.force('center', d3.forceCenter(W()/2, H()/2))
.force('collide', d3.forceCollide(R + 18))
.on('tick', tick);
const lsel = g.append('g').selectAll('line').data(links).join('line')
.attr('class','link')
.attr('stroke', d => ec(d.fill_pct))
.attr('marker-end', d => ea(d.fill_pct));
const llbl = g.append('g').selectAll('text').data(links).join('text')
.attr('class','lbl').text(d => `${d.fill_pct.toFixed(0)}%`);
const ng = g.append('g').selectAll('g').data(nodes).join('g').attr('class','node')
.call(d3.drag()
.on('start',(e,d)=>{ if(!e.active) sim.alphaTarget(0.3).restart(); d.fx=d.x; d.fy=d.y; })
.on('drag', (e,d)=>{ d.fx=e.x; d.fy=e.y; })
.on('end', (e,d)=>{ if(!e.active) sim.alphaTarget(0); d.fx=null; d.fy=null; }));
ng.append('circle').attr('r', R).attr('fill', d => nc(d.ema_exec_ms));
ng.append('text').attr('dy', 4).text(d => d.id);
ng.append('text').attr('class','st').attr('dy', 18)
.text(d => `${d.ema_exec_ms.toFixed(1)}ms ${d.fps.toFixed(1)}fps`);
ng.on('mousemove', (e,d) =>
tip.style('display','block')
.style('left',(e.pageX+12)+'px').style('top',(e.pageY+12)+'px')
.text(`${d.id}\nframes: ${d.frames} fps: ${d.fps.toFixed(2)}\n` +
`exec ema: ${d.ema_exec_ms.toFixed(2)}ms max: ${d.max_exec_ms.toFixed(2)}ms\n` +
`blocked: ${d.blocked_ms.toFixed(2)}ms cpu: ${d.cpu_util_pct.toFixed(1)}%`))
.on('mouseleave', () => tip.style('display','none'));
g.selectAll('.link')
.on('mousemove', (e,d) =>
tip.style('display','block')
.style('left',(e.pageX+12)+'px').style('top',(e.pageY+12)+'px')
.text(`${d.name}\nfill: ${d.fill_pct.toFixed(1)}% peak: ${d.peak_pct.toFixed(1)}%\n` +
`cap: ${d.capacity} pushes: ${d.pushes} drops: ${d.drops}\n` +
`bandwidth: ${(d.bw_mbs||0).toFixed(2)} MB/s`))
.on('mouseleave', () => tip.style('display','none'));
function tick() {
const w = W(), h = H();
nodes.forEach(d => {
d.x = Math.max(R, Math.min(w - R, d.x));
d.y = Math.max(R, Math.min(h - R, d.y));
});
lsel
.attr('x1', d => d.source.x).attr('y1', d => d.source.y)
.attr('x2', d => { const dx=d.target.x-d.source.x, dy=d.target.y-d.source.y,
dist=Math.sqrt(dx*dx+dy*dy)||1;
return d.target.x-(dx/dist)*(R+8); })
.attr('y2', d => { const dx=d.target.x-d.source.x, dy=d.target.y-d.source.y,
dist=Math.sqrt(dx*dx+dy*dy)||1;
return d.target.y-(dy/dist)*(R+8); });
llbl.attr('x', d => (d.source.x+d.target.x)/2)
.attr('y', d => (d.source.y+d.target.y)/2 - 6);
ng.attr('transform', d => `translate(${d.x},${d.y})`);
}
nets[name] = {nodes, links, sim, ng, lsel, llbl};
}
function updateNet(netData) {
const st = nets[netData.name];
if (!st) return;
const byId = Object.fromEntries(netData.nodes.map(n => [n.id, n]));
st.nodes.forEach(n => {
const f = byId[n.id];
if (f) Object.assign(n, {frames:f.frames, ema_exec_ms:f.ema_exec_ms,
max_exec_ms:f.max_exec_ms, blocked_ms:f.blocked_ms, fps:f.fps,
total_cpu_ms:f.total_cpu_ms, cpu_util_pct:f.cpu_util_pct});
});
netData.edges.forEach((e,i) => {
if (st.links[i]) Object.assign(st.links[i], {fill_pct:e.fill_pct,
peak_pct:e.peak_pct, pushes:e.pushes, drops:e.drops,
overflows:e.overflows, current:e.current, bw_mbs:e.bw_mbs});
});
st.ng.select('circle').attr('fill', d => nc(d.ema_exec_ms));
st.ng.select('.st').text(d => `${d.ema_exec_ms.toFixed(1)}ms ${d.fps.toFixed(1)}fps`);
st.lsel.attr('stroke', d => ec(d.fill_pct)).attr('marker-end', d => ea(d.fill_pct));
st.llbl.text(d => `${d.fill_pct.toFixed(0)}%`);
}
// ── Poll loop ─────────────────────────────────────────────────────────────────
let init = false;
async function poll() {
try {
const r = await fetch('/api/snapshot');
if (!r.ok) throw new Error(r.status);
const data = await r.json();
if (!init) {
ensureTab('all', 'All Networks');
data.networks.forEach(net => ensureTab(net.name, net.name));
showTab('all');
data.networks.forEach(initNet);
init = true;
}
renderAll(data);
data.networks.forEach(updateNet);
document.getElementById('status').textContent =
`${new Date().toLocaleTimeString()} · ${data.networks.length} nets · ${(data.resources||[]).length} resources`;
} catch(e) {
document.getElementById('status').textContent = 'error: ' + e;
}
}
poll();
setInterval(poll, 500);
window.addEventListener('resize', () =>
Object.values(nets).forEach(st =>
st.sim.force('center', d3.forceCenter(
(document.getElementById('panel-' + Object.keys(nets).find(k => nets[k] === st))?.clientWidth || window.innerWidth) / 2,
(document.getElementById('panel-' + Object.keys(nets).find(k => nets[k] === st))?.clientHeight || window.innerHeight - 44) / 2
)).alpha(0.1).restart()));
</script>
</body>
</html>
)html";
// ── DebugHub ──────────────────────────────────────────────────────────────────
class DebugHub {
public:
explicit DebugHub(uint16_t port = 9090) : port_(port) {}
~DebugHub() { stop(); }
DebugHub(const DebugHub&) = delete;
DebugHub& operator=(const DebugHub&) = delete;
// Register a network. Disables that network's own web server so the hub
// becomes the single debug endpoint. Call before network.start().
template<typename Net>
void register_network(const std::string& name, Net& net) {
net.disable_web_server();
networks_.push_back({name, [&net, name] {
auto s = net.network_snapshot();
s.name = name;
return s;
}});
}
// Register a shared resource — appears in the "All Networks" resource panel.
void register_resource(const std::string& name, IResourceProbe* probe) {
resources_.emplace_back(name, probe);
}
void start() {
server_ = std::make_unique<web_debug::WebDebugServer>(
port_,
[this] { return build_json(); },
HUB_HTML);
server_->start();
std::cerr << "[kpn] hub debug UI: http://localhost:" << port_ << "\n";
}
void stop() { if (server_) server_->stop(); }
private:
// Serialise nodes array for one network snapshot
static void write_nodes(std::ostream& o, const NetworkSnapshot& s) {
o << "[";
for (std::size_t i = 0; i < s.nodes.size(); ++i) {
const auto& n = s.nodes[i];
if (i) o << ',';
o << "{\"id\":\"" << web_debug::escape_json(n.name) << "\""
<< ",\"frames\":" << n.frames_processed
<< ",\"ema_exec_ms\":" << n.ema_exec_ms
<< ",\"max_exec_ms\":" << n.max_exec_ms
<< ",\"blocked_ms\":" << n.total_blocked_ms
<< ",\"fps\":" << n.throughput_fps
<< ",\"total_cpu_ms\":" << n.total_cpu_ms
<< ",\"cpu_util_pct\":" << n.cpu_util_pct
<< "}";
}
o << "]";
}
// Serialise edges array for one network snapshot
static void write_edges(std::ostream& o, const NetworkSnapshot& s) {
o << "[";
for (std::size_t i = 0; i < s.channels.size(); ++i) {
const auto& c = s.channels[i];
if (i) o << ',';
auto [src, dst] = web_debug::parse_edge_name(c.name);
o << "{\"name\":\"" << web_debug::escape_json(c.name) << "\""
<< ",\"source\":\"" << web_debug::escape_json(src) << "\""
<< ",\"target\":\"" << web_debug::escape_json(dst) << "\""
<< ",\"capacity\":" << c.capacity
<< ",\"current\":" << c.current_fill
<< ",\"fill_pct\":" << c.fill_pct()
<< ",\"peak_pct\":" << c.peak_pct()
<< ",\"pushes\":" << c.pushes
<< ",\"drops\":" << c.drops
<< ",\"overflows\":" << c.overflows
<< ",\"item_bytes\":" << c.item_bytes
<< ",\"bw_mbs\":" << c.bandwidth_mbs(s.elapsed_s)
<< "}";
}
o << "]";
}
std::string build_json() const {
std::ostringstream o;
o << std::fixed;
o.precision(2);
o << "{\"networks\":[";
for (std::size_t i = 0; i < networks_.size(); ++i) {
if (i) o << ',';
auto s = networks_[i].fn();
o << "{\"name\":\"" << web_debug::escape_json(networks_[i].name) << "\""
<< ",\"nodes\":"; write_nodes(o, s);
o << ",\"edges\":"; write_edges(o, s);
o << "}";
}
o << "],\"resources\":[";
for (std::size_t i = 0; i < resources_.size(); ++i) {
if (i) o << ',';
const auto r = resources_[i].second->snapshot(resources_[i].first);
o << "{\"name\":\"" << web_debug::escape_json(r.name) << "\""
<< ",\"acquisitions\":" << r.acquisitions
<< ",\"avg_wait_ms\":" << r.avg_wait_ms
<< ",\"peak_waiters\":" << r.peak_waiters
<< ",\"current_waiters\":" << r.current_waiters
<< ",\"held\":" << (r.held ? "true" : "false")
<< "}";
}
o << "]}";
return o.str();
}
struct Entry {
std::string name;
std::function<NetworkSnapshot()> fn;
};
uint16_t port_;
std::vector<Entry> networks_;
std::vector<std::pair<std::string,IResourceProbe*>> resources_;
std::unique_ptr<web_debug::WebDebugServer> server_;
};
} // namespace kpn
#endif // KPN_WEB_DEBUG
include/kpn/diagnostics.hpp
+230
View File
@@ -0,0 +1,230 @@
#pragma once
#include <atomic>
#include <chrono>
#include <cstddef>
#include <cstdint>
#include <string>
#include <time.h> // clock_gettime, CLOCK_THREAD_CPUTIME_ID
namespace kpn {
using clock_t = std::chrono::steady_clock;
using duration_t = std::chrono::duration<double, std::milli>; // milliseconds
// ── Per-channel statistics ────────────────────────────────────────────────────
struct ChannelStats {
std::atomic<uint64_t> pushes{0};
std::atomic<uint64_t> bytes_pushed{0};
std::atomic<uint64_t> drops{0};
std::atomic<uint64_t> overflows{0};
std::atomic<uint64_t> pops{0};
std::atomic<std::size_t> peak_fill{0};
ChannelStats() = default;
ChannelStats(const ChannelStats&) = delete;
ChannelStats& operator=(const ChannelStats&) = delete;
void record_push(std::size_t current_fill, std::size_t data_bytes) {
pushes.fetch_add(1, std::memory_order_relaxed);
bytes_pushed.fetch_add(data_bytes, std::memory_order_relaxed);
std::size_t prev = peak_fill.load(std::memory_order_relaxed);
while (current_fill > prev &&
!peak_fill.compare_exchange_weak(prev, current_fill,
std::memory_order_relaxed, std::memory_order_relaxed))
;
}
void record_drop() { drops.fetch_add(1, std::memory_order_relaxed); }
void record_overflow() { overflows.fetch_add(1, std::memory_order_relaxed); }
void record_pop() { pops.fetch_add(1, std::memory_order_relaxed); }
};
// ── Per-node statistics ───────────────────────────────────────────────────────
struct NodeStats {
std::atomic<uint64_t> frames_processed{0};
// Wall-clock execution time EMA — warmup mean for first WARMUP_FRAMES,
// then EMA alpha=0.1. Stored as integer microseconds for atomic updates.
static constexpr int WARMUP_FRAMES = 5;
std::atomic<int64_t> ema_exec_us{0};
std::atomic<int64_t> max_exec_us{0};
std::atomic<int64_t> total_blocked_us{0};
// Cumulative wall time inside fire_once, summed over every invocation.
// The EMA above cannot be turned into a total: it is exponentially
// weighted, so frames * ema_exec_us tracks the tail of the run rather than
// the whole of it, and on a workload whose per-frame cost varies (a face
// detector on a film: crowd scenes then empty landscapes) the two differ by
// a lot. Answering "how much time went into this node" needs a real sum.
std::atomic<int64_t> total_exec_us{0};
// Thread CPU time — actual CPU consumed by this node's thread,
// measured via CLOCK_THREAD_CPUTIME_ID. Excludes time sleeping or
// blocked on mutexes/channels. Sampled once per frame.
std::atomic<int64_t> total_cpu_us{0}; // cumulative CPU µs consumed
// Pool scheduling stats — only meaningful for PoolNode / InterruptNode.
// exec_start_us: wall-clock µs when fire_once began; 0 when idle.
// Used by the watchdog to detect hung nodes (elapsed > max_exec_time).
std::atomic<int64_t> queue_wait_us{0}; // cumulative µs spent in pool queue
std::atomic<int64_t> exec_start_us{0}; // non-zero while fire_once is running
NodeStats() = default;
NodeStats(const NodeStats&) = delete;
NodeStats& operator=(const NodeStats&) = delete;
// Call at the start of run_loop to capture thread CPU baseline.
// Returns the raw timespec for use in record_exec.
static struct timespec cpu_now() {
struct timespec ts{};
clock_gettime(CLOCK_THREAD_CPUTIME_ID, &ts);
return ts;
}
static int64_t timespec_us(const struct timespec& ts) {
return static_cast<int64_t>(ts.tv_sec) * 1'000'000
+ static_cast<int64_t>(ts.tv_nsec) / 1'000;
}
void record_queue_wait(duration_t wait) {
int64_t us = static_cast<int64_t>(wait.count() * 1000.0);
if (us > 0) queue_wait_us.fetch_add(us, std::memory_order_relaxed);
}
void record_exec(duration_t exec_time, duration_t blocked_time,
const struct timespec& cpu_before, const struct timespec& cpu_after) {
frames_processed.fetch_add(1, std::memory_order_relaxed);
int64_t us = static_cast<int64_t>(exec_time.count() * 1000.0);
total_exec_us.fetch_add(us, std::memory_order_relaxed);
uint64_t n = frames_processed.load(std::memory_order_relaxed);
int64_t prev = ema_exec_us.load(std::memory_order_relaxed);
int64_t next = (n <= static_cast<uint64_t>(WARMUP_FRAMES))
? prev + (us - prev) / static_cast<int64_t>(n)
: prev + (us - prev) / 10;
ema_exec_us.store(next, std::memory_order_relaxed);
int64_t cur_max = max_exec_us.load(std::memory_order_relaxed);
if (us > cur_max)
max_exec_us.store(us, std::memory_order_relaxed);
int64_t blocked_us = static_cast<int64_t>(blocked_time.count() * 1000.0);
total_blocked_us.fetch_add(blocked_us, std::memory_order_relaxed);
int64_t cpu_delta = timespec_us(cpu_after) - timespec_us(cpu_before);
if (cpu_delta > 0)
total_cpu_us.fetch_add(cpu_delta, std::memory_order_relaxed);
}
};
// ── Snapshot for reporting (copyable, taken by watchdog) ─────────────────────
struct ChannelSnapshot {
std::string name;
std::size_t capacity;
std::size_t current_fill;
std::size_t peak_fill;
uint64_t pushes;
uint64_t bytes_pushed; // actual bytes accumulated via channel_data_size<T>
uint64_t drops;
uint64_t overflows;
uint64_t pops;
std::size_t item_bytes; // sizeof(T) — nominal struct size, not necessarily data size
double fill_pct() const {
return capacity ? 100.0 * current_fill / capacity : 0.0;
}
double peak_pct() const {
return capacity ? 100.0 * peak_fill / capacity : 0.0;
}
// Bandwidth in MB/s: actual bytes transferred / elapsed seconds
double bandwidth_mbs(double elapsed_s) const {
if (elapsed_s <= 0.0) return 0.0;
return static_cast<double>(bytes_pushed) / elapsed_s / 1e6;
}
};
struct NodeSnapshot {
std::string name;
uint64_t frames_processed;
double ema_exec_ms;
double max_exec_ms;
double total_blocked_ms; // ThreadPerNode: time blocked in channel pop
double throughput_fps;
double total_cpu_ms; // cumulative CPU time consumed by this node's thread
double cpu_util_pct; // exec_ms / (exec_ms + blocked_ms) * 100
double queue_wait_ms{0}; // PoolNode: cumulative time spent in pool queue
// Cumulative wall time inside fire_once. Unlike ema_exec_ms this is a true
// sum, so it is the field to use for "share of the run spent in this node".
// Note it still includes time parked pushing into a full output channel;
// total_cpu_ms is the part that backpressure cannot inflate.
//
// Declared before the two bools below because every node type initialises
// this aggregate positionally, and all of them supply total_exec_ms as the
// element after queue_wait_ms.
double total_exec_ms{0};
// Live scheduling state, for observing the AR-004 invariant "a node never
// sleeps with a wake outstanding". The invariant was previously asserted in
// comments but invisible at runtime, so a lost wake could only be found in a
// debugger — and this bug does not reproduce under one (it needs full speed).
// Two atomic loads at snapshot time, nothing on the hot path.
//
// Read them together with the node's channel fill:
// queued=0, wake=1 -> wake recorded and never consumed
// queued=0, wake=0, input full -> wake never generated at all
// queued=1 while nothing running -> submitted but never scheduled
bool queued{false};
bool wake_pending{false};
};
// ── Pool statistics + snapshot ────────────────────────────────────────────────
struct PoolSnapshot {
std::string name;
std::size_t thread_count;
std::size_t queue_depth; // tasks waiting in the priority queue
std::size_t active_count; // tasks currently executing
uint64_t tasks_submitted;
uint64_t tasks_completed;
};
struct IPoolProbe {
virtual ~IPoolProbe() = default;
virtual PoolSnapshot snapshot(const std::string& name) const = 0;
};
// ── Cross-network snapshot (used by DebugHub) ─────────────────────────────────
struct NetworkSnapshot {
std::string name;
std::vector<NodeSnapshot> nodes;
std::vector<ChannelSnapshot> channels;
double elapsed_s;
};
// ── Resource statistics + snapshot ───────────────────────────────────────────
struct ResourceSnapshot {
std::string name;
uint64_t acquisitions;
double avg_wait_ms;
uint64_t peak_waiters;
uint64_t current_waiters;
bool held;
};
struct IResourceProbe {
virtual ~IResourceProbe() = default;
virtual ResourceSnapshot snapshot(const std::string& name) const = 0;
/// Release every thread waiting for the resource, so teardown is not held
/// up by one. A network calls this on the resources registered with it when
/// it halts; default no-op for probes with nothing to wake.
virtual void close() {}
};
} // namespace kpn
include/kpn/fanout.hpp
+237
View File
@@ -0,0 +1,237 @@
#pragma once
#include "channel.hpp"
#include "diagnostics.hpp"
#include "inode.hpp"
#include "port.hpp"
#include "traits.hpp"
#include <array>
#include <atomic>
#include <chrono>
#include <iostream>
#include <memory>
#include <optional>
#include <thread>
#include <tuple>
#include <utility>
namespace kpn {
// ── FanoutNode ────────────────────────────────────────────────────────────────
//
// Reads one item from its single input channel and pushes a copy to each of
// N output channels. All N downstream nodes receive every item.
//
// Usage:
// auto fan = make_fanout<Image, 2>(/*capacity=*/8);
// net.connect("src", src.output<0>(), "fan", fan.input<0>())
// .connect("fan", fan.output<0>(), "nodeA", nodeA.input<0>())
// .connect("fan", fan.output<1>(), "nodeB", nodeB.input<0>())
template<typename T, std::size_t N, std::size_t Id = 0>
class FanoutNode : public INode {
public:
using args_tuple = std::tuple<T>;
using return_tuple = repeat_tuple_t<T, N>;
using return_raw = return_tuple;
static constexpr std::size_t input_count = 1;
static constexpr std::size_t output_count = N;
static constexpr std::size_t unique_tag = Id;
static constexpr bool is_fanout_node = true;
explicit FanoutNode(std::size_t fifo_capacity = 5)
: fifo_capacity_(fifo_capacity)
{
input_ch_ = std::make_shared<Channel<T>>(fifo_capacity);
}
~FanoutNode() override { stop(); }
// ── INode ─────────────────────────────────────────────────────────────────
void start() override {
input_ch_->enable();
stop_flag_.store(false, std::memory_order_relaxed);
thread_ = std::jthread([this](std::stop_token) { run_loop(); });
}
void stop() override {
stop_flag_.store(true, std::memory_order_relaxed);
input_ch_->disable();
if (thread_.joinable()) thread_.request_stop(), thread_.join();
}
bool running() const override {
return thread_.joinable() && !stop_flag_.load(std::memory_order_relaxed);
}
void set_name(std::string name) override { name_ = std::move(name); }
const NodeStats& stats() const override { return stats_; }
NodeSnapshot node_snapshot(const std::string& name, double elapsed_s) const override {
uint64_t frames = stats_.frames_processed.load(std::memory_order_relaxed);
double exec_ms = stats_.ema_exec_us.load(std::memory_order_relaxed) / 1000.0;
double blocked_ms = stats_.total_blocked_us.load(std::memory_order_relaxed) / 1000.0;
double total_ms = exec_ms + blocked_ms;
return {name, frames, exec_ms,
stats_.max_exec_us.load(std::memory_order_relaxed) / 1000.0,
blocked_ms,
elapsed_s > 0 ? frames / elapsed_s : 0.0,
stats_.total_cpu_us.load(std::memory_order_relaxed) / 1000.0,
total_ms > 0 ? 100.0 * exec_ms / total_ms : 0.0,
0.0, // queue_wait_ms — fanout is not pool-scheduled
stats_.total_exec_us.load(std::memory_order_relaxed) / 1000.0};
}
// ── Port access ───────────────────────────────────────────────────────────
template<std::size_t I = 0>
InputPort<FanoutNode, I> input() {
static_assert(I == 0, "FanoutNode has exactly one input");
return {*this};
}
template<std::size_t I>
OutputPort<FanoutNode, I> output() {
static_assert(I < N, "FanoutNode output index out of range");
return {*this};
}
// ── Internal channel accessors (called by Network::connect) ───────────────
template<std::size_t I>
Channel<T>& input_channel() {
static_assert(I == 0);
return *input_ch_;
}
template<std::size_t I>
void set_input_channel(std::shared_ptr<Channel<T>> ch) {
static_assert(I == 0);
input_ch_ = std::move(ch);
}
template<std::size_t I>
void set_output_channel(Channel<T>* ch) {
static_assert(I < N);
out_channels_[I] = ch;
}
private:
// Deliver `val` to every connected output, losslessly.
//
// Previously a full output cost the value: push() threw and the exception was
// swallowed per output. A dropped item does not degrade a downstream result,
// it silently changes one, and the consumer cannot tell it happened — so the
// fanout waits instead, and the producer upstream runs slower.
//
// Unlike a pool node, a fanout owns a private thread, so waiting here costs
// no scheduler worker and needs no space-callback park; a bounded retry is
// enough. `stop_flag_` is re-checked every pass so teardown cannot hang on a
// full output regardless of the order the network stops its nodes in.
//
// Outputs are retried independently, so a full output never delays delivery
// to one with room. Note what that does *not* buy: the next input is not
// popped until every output has accepted the current item, so one branch can
// never run ahead of another by more than the slower branch's buffering.
//
// **That bound is a precondition on any topology where the branches rejoin.**
// If a consumer on branch B blocks waiting for something branch A computes,
// B's buffering must exceed the lead A needs, or the two wedge — B waiting on
// A, A starved because the fanout is holding an item B will not take. Making
// the fanout lossless is what puts that precondition on the topology; while
// it dropped, the question could not arise.
//
// `parked` receives the time spent waiting on a full output, which the caller
// charges to blocked rather than exec.
//
// Returns false if stopped with the value undelivered.
bool deliver(const T& val, duration_t& parked) {
std::array<std::optional<T>, N> pending;
std::size_t outstanding = 0;
for (std::size_t i = 0; i < N; ++i)
if (out_channels_[i]) { pending[i].emplace(val); ++outstanding; }
bool first_pass = true;
auto park_from = clock_t::now();
for (;;) {
for (std::size_t i = 0; i < N; ++i) {
if (!pending[i]) continue;
// Taken or Closed both mean "stop trying" — delivered, or gone
// with the drop recorded. Only Full is worth another pass.
if (out_channels_[i]->try_push(*pending[i])
!= Channel<T>::PushResult::Full) {
pending[i].reset();
--outstanding;
}
}
if (first_pass) { park_from = clock_t::now(); first_pass = false; }
if (outstanding == 0) {
parked = duration_t(clock_t::now() - park_from);
return true;
}
if (stop_flag_.load(std::memory_order_relaxed)) {
// Teardown with work in hand. One last throwing push per
// outstanding output, purely so the channel's own stats record
// the loss (drop if it is disabled, overflow if it is merely
// full). The whole point of the lossless path is that a loss is
// never invisible, and a silent `return` here would reintroduce
// exactly the hole this function exists to close.
for (std::size_t i = 0; i < N; ++i) {
if (!pending[i]) continue;
try { out_channels_[i]->push(std::move(*pending[i])); }
catch (const ChannelOverflowError&) {}
}
parked = duration_t(clock_t::now() - park_from);
return false;
}
std::this_thread::sleep_for(std::chrono::microseconds(50));
}
}
void run_loop() {
while (!stop_flag_.load(std::memory_order_relaxed)) {
try {
auto t0 = clock_t::now();
T val = input_ch_->pop();
auto t1 = clock_t::now();
auto cpu0 = NodeStats::cpu_now();
duration_t parked{0};
const bool delivered = deliver(val, parked);
auto cpu1 = NodeStats::cpu_now();
auto t2 = clock_t::now();
// Time spent waiting on a full output is *blocked*, not exec: a
// parked fanout is idle, and charging it to exec would report the
// node as busy exactly when it is the one being held up.
stats_.record_exec(duration_t(t2 - t1) - parked,
duration_t(t1 - t0) + parked, cpu0, cpu1);
if (!delivered) break;
} catch (const ChannelClosedError&) {
break;
}
}
}
std::string name_;
std::size_t fifo_capacity_;
std::shared_ptr<Channel<T>> input_ch_;
std::array<Channel<T>*, N> out_channels_{};
std::atomic<bool> stop_flag_{false};
std::jthread thread_;
NodeStats stats_;
};
// ── Factory ───────────────────────────────────────────────────────────────────
template<typename T, std::size_t N>
FanoutNode<T, N> make_fanout(std::size_t fifo_capacity = 5) {
return FanoutNode<T, N, 0>(fifo_capacity);
}
} // namespace kpn

Some files were not shown because too many files have changed in this diff Show More