nobody is waiting for
B9 from PERF_PLAN, plus the notify gate. Two changes to submit(), both
aimed at the same cost: on any pool of two or more threads, every single
dispatch paid a futex wake.
submit() round-robins, so a worker resubmitting -- which is what
fire_once does on every token -- handed the task to a *different*
worker, and that worker was asleep. bench_dispatch measured it as 182
ns/dispatch on ThreadPool(1) against 3197 ns on 20 threads, with
voluntary context switches per task rising 0.00 -> 1.19 in step: the
1-thread pool is fast precisely because it resubmits into its own queue
and finds the work already there. A submission originating on one of the
pool's own workers now goes to that worker's queue, extending that
property to any pool size. try_steal still corrects the imbalance.
The identity check is against `this`, not merely "am I a pool worker":
a worker of pool A submitting into pool B must not use A's index, which
may exceed B's thread_count_. Nested networks do exactly this. tls_pool
is a non-owning identity tag, only ever compared, never dereferenced --
its lifetime is strictly nested inside the pool's, since stop() joins
every worker before clearing queues_.
The notify gate skips the cv_mx_ round-trip and notify_one() when
waiters_ is zero. waiters_ is maintained under cv_mx_ and incremented
before the predicate is evaluated, so reading zero in submit() means no
worker can be in wait() -- as opposed to reading zero because we raced
one, which the mutex prevents. stop()'s notify_all() is deliberately
left ungated.
Shared pools, work_us=10, items/sec: chain-1 59512 -> 66662 (+12.0%),
wide-4 53698 -> 61705 (+14.9%), chain-8 +6.8%, chain-32 +3.7%. Private
pools -- the Node<> default -- are unchanged at -1.3% to +3.6%, inside
the gate's tolerance.
Two things tried and removed, recorded in comments so they are not
retried:
Raising the steal threshold to >1, to stop a thief winning the race for
a self-submitted task, DEADLOCKS. An external submit() round-robins a
single task onto an idle worker's queue; if that worker is parked, no
peer will take it, because a queue of one is no longer stealable.
latency mode hangs at 12 and 20 threads. It was also 2x slower in steady
state, 2229 -> 4546 ns.
B5, bounded spin before parking, does not pay: swept at 50/200/1000
rounds, 2123 / 2230 / 2574 ns against 2229 ns without it, with
vcsw/task flat at ~0.97. The spin cannot catch what it targets, because
a peer is woken the moment queued_ becomes non-zero -- before this
worker reaches the spin at all.
Guardrails: 150/150 ctest including soak and examples, and
ThreadSanitizer clean over the full suite (128 cases, 302 assertions).
That also pins the reference count PERF_PLAN section 6 flagged as
uncertain: it is 150, not 146 or 136.
Caveat: the throughput figures above were taken on a build that also
carried the since-removed spin experiment. The scheduler logic is
identical to this tree and correctness was re-verified on it, but the
numbers are one build stale and predate the 7-pass gate, which has not
been run on this change.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
dtourolleandClaude Opus 5
b500570c47