nobody is waiting for
B9 from PERF_PLAN, plus the notify gate. Two changes to submit(), both
aimed at the same cost: on any pool of two or more threads, every single
dispatch paid a futex wake.
submit() round-robins, so a worker resubmitting -- which is what
fire_once does on every token -- handed the task to a *different*
worker, and that worker was asleep. bench_dispatch measured it as 182
ns/dispatch on ThreadPool(1) against 3197 ns on 20 threads, with
voluntary context switches per task rising 0.00 -> 1.19 in step: the
1-thread pool is fast precisely because it resubmits into its own queue
and finds the work already there. A submission originating on one of the
pool's own workers now goes to that worker's queue, extending that
property to any pool size. try_steal still corrects the imbalance.
The identity check is against `this`, not merely "am I a pool worker":
a worker of pool A submitting into pool B must not use A's index, which
may exceed B's thread_count_. Nested networks do exactly this. tls_pool
is a non-owning identity tag, only ever compared, never dereferenced --
its lifetime is strictly nested inside the pool's, since stop() joins
every worker before clearing queues_.
The notify gate skips the cv_mx_ round-trip and notify_one() when
waiters_ is zero. waiters_ is maintained under cv_mx_ and incremented
before the predicate is evaluated, so reading zero in submit() means no
worker can be in wait() -- as opposed to reading zero because we raced
one, which the mutex prevents. stop()'s notify_all() is deliberately
left ungated.
Shared pools, work_us=10, items/sec: chain-1 59512 -> 66662 (+12.0%),
wide-4 53698 -> 61705 (+14.9%), chain-8 +6.8%, chain-32 +3.7%. Private
pools -- the Node<> default -- are unchanged at -1.3% to +3.6%, inside
the gate's tolerance.
Two things tried and removed, recorded in comments so they are not
retried:
Raising the steal threshold to >1, to stop a thief winning the race for
a self-submitted task, DEADLOCKS. An external submit() round-robins a
single task onto an idle worker's queue; if that worker is parked, no
peer will take it, because a queue of one is no longer stealable.
latency mode hangs at 12 and 20 threads. It was also 2x slower in steady
state, 2229 -> 4546 ns.
B5, bounded spin before parking, does not pay: swept at 50/200/1000
rounds, 2123 / 2230 / 2574 ns against 2229 ns without it, with
vcsw/task flat at ~0.97. The spin cannot catch what it targets, because
a peer is woken the moment queued_ becomes non-zero -- before this
worker reaches the spin at all.
Guardrails: 150/150 ctest including soak and examples, and
ThreadSanitizer clean over the full suite (128 cases, 302 assertions).
That also pins the reference count PERF_PLAN section 6 flagged as
uncertain: it is 150, not 146 or 136.
Caveat: the throughput figures above were taken on a build that also
carried the since-removed spin experiment. The scheduler logic is
identical to this tree and correctness was re-verified on it, but the
numbers are one build stale and predate the 7-pass gate, which has not
been run on this change.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
abbb2d4 guarded submit() against stop() with a shared/exclusive lock, because
stop() ends with queues_.clear() and submit() indexes queues_. It missed the
other writer: start() rebuilds the same vector and took no lock at all.
A submission can genuinely land while a pool is inside start(). A network
starts its nodes one at a time, and a node already started fires into the next
one's channel, whose push callback submits. ThreadSanitizer reports it as the
read at scheduler.hpp:114 against the write at :59, and the consequence is
worse than a torn read: push_back can reallocate the vector under a reader
that has already indexed it.
Surfaced by 6802328. Firing push_callback_ unconditionally is correct — the
argument in that commit holds — and it makes callbacks frequent enough during
startup to hit this window. It went from unobserved to 4 races in one run of
the unit suite.
Holding the lock across the thread spawn is safe: all queues are constructed
before any worker starts, and worker_loop never takes the lifecycle lock, so
there is nothing for it to deadlock against.
Verified with -DKPN_SANITIZER=thread: 4 races in one run of five before, 0
across five unit runs and two stress runs after. 148/148.
The wait predicate was `stopped_ || total_ > 0`, and total_ counts queued
*plus executing*. So while any one task ran, every other worker's predicate
was true: wait() returned immediately and the worker spun through try_pop,
try_steal and back to wait at full speed, try_lock-ing every peer queue on
each pass.
Measured on this tree, 8 workers and one 300 ms task: 1991 ms of CPU and
19205 voluntary context switches, against 0.4 ms and 10 with the fix. Call it
six and a half cores burned for the duration of one sleeping task.
Sleeping requires "no work is *waiting*", which total_ cannot express, so
queued_ is now tracked separately: incremented on submit, decremented when a
task leaves a queue, and adjusted for the tasks stop() discards. total_ stays
as it was for drain(), which genuinely does need to know about executing work.
The worker exit condition moves to queued_ for the same reason — waiting for
total_ to reach zero meant waiting for someone else's task to finish, which a
worker cannot help with and would spin through until it did. PoolSnapshot's
queue depth stops being an estimate as a side effect.
Latent for this pipeline, where each node owns a private single-thread pool
and there is no idle peer to spin. Any use of a shared pool, which
make_pool_node exists for, hits it immediately.
Reproducing it needs the right trigger, and the test says so, because my first
attempt got it wrong and passed against the bug: a worker that has never been
woken stays blocked in wait() and never re-evaluates the predicate. The spin
only appears once a worker *finishes* something and re-enters the loop while a
peer is still busy, so the case submits one long task plus a trivial one per
remaining worker. Submitting only the long task measures nothing.
Verified in both directions: 1969 ms of CPU before, 0.35 ms after, against a
300 ms threshold. Full suite 142/142.
ThreadPool::stop() ends with queues_.clear(), and submit() went straight to
queues_[target] with no check. A submission arriving after stop indexed an
empty vector and segfaulted.
This is not a contrived teardown ordering. A node's space callback fires from
whichever thread drained the channel, and that thread belongs to the
*consumer*; the callback it runs belongs to the *producer*. Stop the producer
first — which a sources-first shutdown does by design — and the consumer keeps
draining its backlog, firing the producer's space callback into a pool that
has already been torn down:
ThreadPool::submit
<- source node's space_callback
<- Channel<int>::try_pop_now (relay draining its input)
<- relay fire_once
The static-network shutdown case in the next commit crashed about 12 runs in
20 on this. It survived until now because halt() stops in reverse topological
order — consumers first — so the producer whose callback might fire is always
still alive. shutdown() stops sources first and does not have that protection.
Reading stopped_ without a lock would not fix it: the window between the read
and the indexing is exactly where clear() runs. submit() takes a shared lock
and stop() an exclusive one, so submissions still proceed in parallel with
each other while being serialised against teardown. stop() sets the flag under
the lock, releases it to join — a worker's task may itself call submit, and
holding the lock across the join would deadlock against that — then retakes it
to destroy the queues.
Refusals are counted rather than silent. A teardown race is expected, but a
node repeatedly trying to run after its pool is gone is worth being able to
see. try_submit also checks stop_flag_ first, so a stopped node cannot claim
the submit gate and leave it held.
Not a smart-pointer problem, for anyone reading the crash: nothing here is
owned by a raw pointer. It is std::vector::operator[] on a vector that was
emptied by another thread.
Verified in both directions: with the guard removed the new scheduler cases
segfault; with it they pass.
dtourolleandClaude Opus 5