dtourolle/KPN
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

perf: make the benchmark able to answer the question, then ask it

Browse Source 
PERF_PLAN phase 0, plus B1/B2 which turned out to cost seconds rather
than the minutes budgeted for them. No library code is touched.

The harness could not support the conclusions drawn from it. items_for()
shrank the sample as work per item grew, so exactly the rows under
investigation -- chain-16 and chain-32 -- ran 50 to 200 items and swung
4-8x between passes. Sample size now derives from a time budget with a
floor, using work_us * stages / units as the per-item cost. The old
ladder's error was treating depth as a throughput cost: past the core
count it is, below it depth costs only latency.

Rows now report median of N repetitions after a discarded warm-up, with
IQR and range, so an unreliable row says so instead of being averaged
into a table. The CSV header records nproc, governor and AC state, which
immediately caught this laptop running on battery under powersave.

A3 needed no experiment in the end: ru_nivcsw and ru_nvcsw are captured
around every timed region and reported per item, so involuntary switches
against depth is a column rather than a run.

bench_dispatch answers B1 and B2 without instrumenting the scheduler.
Sleeping is inferred from ru_nvcsw, since a thread blocking on a
condition variable books a voluntary context switch. B1: a ThreadPool(1)
dispatch is 291 ns null, 466 ns with a payload, against the ~290 ns the
plan estimated -- so the abandon criterion is not met and workstream B
stays alive.

B2's answer is not the one the question expected. It is not whether
workers sleep but which pool: on a private ThreadPool(1) the worker never
sleeps, because it resubmits into its own queue and finds the work
already there; on any pool of two or more it sleeps exactly once per
task, because submit() round-robins to a different worker, which is
asleep. That is the whole 466 ns to 1.7 us difference, and it inverts
half the plan. B5 (bounded spin) buys nothing in the default
configuration, and A5 must not make a shared pool the default until the
wake cost is fixed, or every graph that already fits its cores gets 3-4x
worse per dispatch.

G1 lands as tests/soak_wedge.cpp, superseding benchmarks/repro_wedge.cpp,
which was never wired into any build. Always compiled so it cannot rot;
its CTest cases register only under -DKPN_ENABLE_SOAK_TESTS=ON, so the
default test count is unchanged. A wedge is a hang, and a hang under
CTest is an unattributable timeout, so it carries a watchdog that aborts
naming the iteration and phase.

Phase 0's gate is not yet cleared: the acceptance run belongs on the
reference machine, not here. A 3-pass check lands every row within 0.7%
against the 4-8x swings described above, which is encouraging and is not
the same thing.

Provisional, recorded so it can be checked: chain-16 came out 6% behind
TBB rather than 28.5%. If that survives a proper run, the deep-chain
deficit is substantially an artefact of the N=200 rows.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
dtourolle
2026-08-08 12:13:11 +02:00
co-authored by Claude Opus 5
parent 771b9f8593
commit a3f61fcb3c
9 changed files with 1587 additions and 124 deletions
Download Patch File Download Diff File Expand all files Collapse all files
benchmarks/CMakeLists.txt
+7
View File
@@ -4,6 +4,13 @@ add_executable(bench_pipeline bench_pipeline.cpp)
target_link_libraries(bench_pipeline PRIVATE kpn)
target_compile_options(bench_pipeline PRIVATE -O3 -march=native)
# Dispatch microbenchmark (PERF_PLAN B1/B2): ns per ThreadPool dispatch, and
# whether a worker actually sleeps per task. No TBB comparison — it measures
# KPN's own scheduler, not a competitor.
add_executable(bench_dispatch bench_dispatch.cpp)
target_link_libraries(bench_dispatch PRIVATE kpn)
target_compile_options(bench_dispatch PRIVATE -O3 -march=native)
find_package(TBB QUIET)
if(TBB_FOUND)
target_link_libraries(bench_pipeline PRIVATE TBB::tbb)
benchmarks/bench_dispatch.cpp
+307
View File
@@ -0,0 +1,307 @@
// Dispatch microbenchmark — PERF_PLAN B1 and B2.
//
// B1 asks what a single ThreadPool dispatch costs. B2 asks whether a worker
// actually sleeps per item, because the whole spin-window hypothesis (B5)
// depends on the answer: if workers are not sleeping, a spin window buys
// nothing and drops off the list.
//
// Both are answered here without touching the library. Sleeping is inferred
// from ru_nvcsw — a thread blocking on a condition variable books a voluntary
// context switch — so `vcsw/task` near 1.0 means a sleep per dispatch and near
// 0 means the worker never went to sleep at all.
//
// Three modes, because "the cost of a dispatch" is three different numbers:
//
// latency — one task in flight, pool idle in between. The worker is asleep
// at every submission, so this is dispatch cost *including* a
// wake. Worst case, and the case a spin window would attack.
//
// batch — submit K no-op tasks flat out, then drain. The worker is never
// idle, so this is the amortised floor: queue and heap operations
// with no wake at all. Reports the producer-side submit() cost
// separately from end-to-end throughput.
//
// steady — the task resubmits its successor, one in flight, each doing
// --work-us of work. This is what a KPN node actually does:
// fire_once processes a token and resubmits. On ThreadPool(1) the
// worker resubmits to its own queue; on ThreadPool(4) round-robin
// hands the task to a *different* worker, which may be asleep.
// That difference is the fanout cost wide-4 pays ~5x per item.
//
// Usage: ./bench_dispatch [--threads=1,2,4] [--mode=latency,batch,steady]
// [--tasks=200000] [--work-us=0] [--reps=5] [--warmup=1]
#include <kpn/kpn.hpp>
#include "bench_env.hpp"
#include <atomic>
#include <chrono>
#include <condition_variable>
#include <cstdio>
#include <cstdlib>
#include <mutex>
#include <string>
#include <vector>
using namespace kpn;
using sclock = std::chrono::steady_clock;
struct Opts {
std::vector<int> threads {1, 2, 4};
std::vector<std::string> modes {"latency", "batch", "steady"};
long tasks = 200000;
int work_us = 0;
int reps = 5;
int warmup = 1;
};
static Opts g_opts;
static void busy_us(int us) {
if (us <= 0) return;
auto end = sclock::now() + std::chrono::microseconds(us);
while (sclock::now() < end);
}
struct Sample {
double ns_per_dispatch = 0; // end-to-end, minus the work payload
double submit_ns = 0; // producer side only (batch mode)
double vcsw_per_task = 0; // B2: sleeps per dispatch
double ivcsw_per_task = 0;
};
// ── latency: one task at a time, worker asleep between submissions ────────────
static Sample run_latency(int threads, long tasks) {
ThreadPool pool(threads);
pool.start();
std::mutex mx;
std::condition_variable cv;
bool done = false;
bench::RusageDelta ru; ru.start();
auto t0 = sclock::now();
for (long i = 0; i < tasks; ++i) {
{ std::lock_guard lk(mx); done = false; }
pool.submit([&] {
busy_us(g_opts.work_us);
{ std::lock_guard lk(mx); done = true; }
cv.notify_one();
});
std::unique_lock lk(mx);
cv.wait(lk, [&] { return done; });
}
auto t1 = sclock::now();
Sample s;
long iv = 0, vc = 0;
ru.finish(iv, vc);
pool.stop();
double elapsed_ns = std::chrono::duration<double, std::nano>(t1 - t0).count();
s.ns_per_dispatch = elapsed_ns / tasks - g_opts.work_us * 1000.0;
// The requesting thread blocks once per task too, so it books a voluntary
// switch of its own; halve to attribute per side rather than per process.
s.vcsw_per_task = static_cast<double>(vc) / tasks / 2.0;
s.ivcsw_per_task = static_cast<double>(iv) / tasks;
return s;
}
// ── batch: submit flat out, drain once. No wake in the steady state ───────────
static Sample run_batch(int threads, long tasks) {
ThreadPool pool(threads);
pool.start();
std::atomic<long> ran{0};
bench::RusageDelta ru; ru.start();
auto t0 = sclock::now();
for (long i = 0; i < tasks; ++i)
pool.submit([&] {
busy_us(g_opts.work_us);
ran.fetch_add(1, std::memory_order_relaxed);
});
auto t_submitted = sclock::now();
pool.drain();
auto t1 = sclock::now();
Sample s;
long iv = 0, vc = 0;
ru.finish(iv, vc);
pool.stop();
if (ran.load() != tasks)
std::fprintf(stderr, "WARNING: batch ran %ld of %ld tasks\n",
ran.load(), tasks);
double elapsed_ns = std::chrono::duration<double, std::nano>(t1 - t0).count();
s.ns_per_dispatch = elapsed_ns / tasks - g_opts.work_us * 1000.0;
s.submit_ns = std::chrono::duration<double, std::nano>(t_submitted - t0).count() / tasks;
s.vcsw_per_task = static_cast<double>(vc) / tasks;
s.ivcsw_per_task = static_cast<double>(iv) / tasks;
return s;
}
// ── steady: the task resubmits its successor, as fire_once does ──────────────
static Sample run_steady(int threads, long tasks) {
ThreadPool pool(threads);
pool.start();
std::mutex mx;
std::condition_variable cv;
std::atomic<long> count{0};
bool finished = false;
// Recursive submission: hold the chain in a std::function so the task can
// resubmit itself. Captured by reference; it outlives the drain below.
//
// The counter is atomic rather than mutex-guarded so that this loop
// measures the pool's dispatch path and not a lock of the benchmark's own.
std::function<void()> step = [&] {
busy_us(g_opts.work_us);
long n = count.fetch_add(1, std::memory_order_relaxed) + 1;
if (n < tasks) {
pool.submit(step);
} else {
{ std::lock_guard lk(mx); finished = true; }
cv.notify_one();
}
};
bench::RusageDelta ru; ru.start();
auto t0 = sclock::now();
pool.submit(step);
{
std::unique_lock lk(mx);
cv.wait(lk, [&] { return finished; });
}
auto t1 = sclock::now();
Sample s;
long iv = 0, vc = 0;
ru.finish(iv, vc);
pool.stop();
double elapsed_ns = std::chrono::duration<double, std::nano>(t1 - t0).count();
s.ns_per_dispatch = elapsed_ns / tasks - g_opts.work_us * 1000.0;
s.vcsw_per_task = static_cast<double>(vc) / tasks;
s.ivcsw_per_task = static_cast<double>(iv) / tasks;
return s;
}
// ── driver ────────────────────────────────────────────────────────────────────
static void run_row(const std::string& mode, int threads, long tasks) {
auto once = [&] {
if (mode == "latency") return run_latency(threads, tasks);
if (mode == "batch") return run_batch(threads, tasks);
return run_steady(threads, tasks);
};
for (int i = 0; i < g_opts.warmup; ++i) (void)once();
std::vector<double> ns, sub, vcsw, ivcsw;
for (int i = 0; i < g_opts.reps; ++i) {
Sample s = once();
ns.push_back(s.ns_per_dispatch);
sub.push_back(s.submit_ns);
vcsw.push_back(s.vcsw_per_task);
ivcsw.push_back(s.ivcsw_per_task);
}
const double med = bench::percentile(ns, 0.5);
const double q1 = bench::percentile(ns, 0.25);
const double q3 = bench::percentile(ns, 0.75);
const double iqr = med > 0 ? 100.0 * (q3 - q1) / med : 0.0;
const double sleeps = bench::percentile(vcsw, 0.5);
std::fprintf(stderr, "%-9s %-8d %-8d %-10ld %-12.0f %-7.1f %-11.0f %-10.2f %-10.2f\n",
mode.c_str(), threads, g_opts.work_us, tasks, med, iqr,
bench::percentile(sub, 0.5), sleeps,
bench::percentile(ivcsw, 0.5));
// Column names deliberately match bench_pipeline's key columns so that
// scripts/bench_repro_check.py can gate this benchmark too.
std::printf("%s,%d,%d,%d,%ld,%d,%.1f,%.2f,%.1f,%.3f,%.3f\n",
mode.c_str(), threads, g_opts.work_us, threads, tasks,
g_opts.reps, med, iqr, bench::percentile(sub, 0.5),
sleeps, bench::percentile(ivcsw, 0.5));
std::fflush(stdout);
}
static std::vector<int> parse_int_list(const char* s) {
std::vector<int> out;
const char* p = s;
while (*p) {
char* end = nullptr;
long v = std::strtol(p, &end, 10);
if (end == p) break;
out.push_back(static_cast<int>(v));
p = end;
while (*p == ',' || *p == ' ') ++p;
}
return out;
}
static std::vector<std::string> parse_word_list(const std::string& s) {
std::vector<std::string> out;
std::size_t pos = 0;
while (pos <= s.size()) {
std::size_t c = s.find(',', pos);
if (c == std::string::npos) c = s.size();
if (c > pos) out.push_back(s.substr(pos, c - pos));
pos = c + 1;
}
return out;
}
static void usage() {
std::fprintf(stderr,
"usage: bench_dispatch [options]\n"
" --threads=1,2,4 pool sizes\n"
" --mode=latency,batch,steady which measurements to run\n"
" --tasks=200000 dispatches per repetition\n"
" --work-us=0 payload per task\n"
" --reps=5 --warmup=1\n");
}
int main(int argc, char** argv) {
for (int i = 1; i < argc; ++i) {
std::string a = argv[i];
auto eq = a.find('=');
std::string key = a.substr(0, eq);
std::string val = eq == std::string::npos ? "" : a.substr(eq + 1);
if (key == "--help" || key == "-h") { usage(); return 0; }
else if (key == "--threads") g_opts.threads = parse_int_list(val.c_str());
else if (key == "--mode") g_opts.modes = parse_word_list(val);
else if (key == "--tasks") g_opts.tasks = std::atol(val.c_str());
else if (key == "--work-us") g_opts.work_us = std::atoi(val.c_str());
else if (key == "--reps") g_opts.reps = std::atoi(val.c_str());
else if (key == "--warmup") g_opts.warmup = std::atoi(val.c_str());
else { std::fprintf(stderr, "unknown option: %s\n", a.c_str()); usage(); return 2; }
}
if (g_opts.reps < 1) g_opts.reps = 1;
if (g_opts.warmup < 0) g_opts.warmup = 0;
char cfg[160];
std::snprintf(cfg, sizeof cfg, "tasks=%ld work_us=%d reps=%d warmup=%d",
g_opts.tasks, g_opts.work_us, g_opts.reps, g_opts.warmup);
bench::print_environment(cfg);
std::fprintf(stderr, "\n%-9s %-8s %-8s %-10s %-12s %-7s %-11s %-10s %-10s\n",
"mode", "threads", "work_us", "tasks", "ns/dispatch", "iqr%",
"submit_ns", "vcsw/task", "ivcsw/task");
std::fprintf(stderr, "%s\n", std::string(96, '-').c_str());
std::printf("topology,size,work_us,threads,items,reps,ns_per_dispatch,"
"iqr_pct,submit_ns,vcsw_per_task,ivcsw_per_task\n");
// latency is a round trip per task, so it is far slower per dispatch than
// the other modes; scale it down rather than run for minutes.
for (const auto& mode : g_opts.modes)
for (int t : g_opts.threads) {
long tasks = mode == "latency"
? std::max(2000L, g_opts.tasks / 20)
: g_opts.tasks;
run_row(mode, t, tasks);
}
}
benchmarks/bench_env.hpp
+108
View File
@@ -0,0 +1,108 @@
// Shared benchmark plumbing: machine attribution (PERF_PLAN M6), repetition
// statistics (M3), and context-switch capture.
//
// The attribution is not decoration. A result taken under the powersave
// governor or on battery is not comparable with one taken on AC under
// performance, and a stored CSV that does not say which it was cannot be
// argued about later.
#pragma once
#include <algorithm>
#include <cmath>
#include <cstdio>
#include <string>
#include <thread>
#include <vector>
#include <sys/resource.h>
namespace bench {
inline int hw_units() {
unsigned n = std::thread::hardware_concurrency();
return n ? static_cast<int>(n) : 1;
}
inline std::string read_line_of(const char* path) {
std::FILE* f = std::fopen(path, "r");
if (!f) return "unknown";
char buf[128] = {0};
if (!std::fgets(buf, sizeof buf, f)) { std::fclose(f); return "unknown"; }
std::fclose(f);
std::string s(buf);
while (!s.empty() && (s.back() == '\n' || s.back() == ' ')) s.pop_back();
return s.empty() ? "unknown" : s;
}
inline std::string ac_state() {
for (const char* p : {"/sys/class/power_supply/AC/online",
"/sys/class/power_supply/AC0/online",
"/sys/class/power_supply/ACAD/online",
"/sys/class/power_supply/ADP1/online"}) {
std::string v = read_line_of(p);
if (v != "unknown") return v == "1" ? "ac" : "battery";
}
return "unknown";
}
// M6 — emitted to both streams: the CSV so a stored result can be attributed,
// the terminal so a run under the wrong governor is noticed while it happens.
inline void print_environment(const std::string& config_line) {
const std::string gov = read_line_of(
"/sys/devices/system/cpu/cpu0/cpufreq/scaling_governor");
const std::string ac = ac_state();
for (std::FILE* out : {stdout, stderr}) {
std::fprintf(out, "# nproc=%d governor=%s power=%s\n",
hw_units(), gov.c_str(), ac.c_str());
if (!config_line.empty())
std::fprintf(out, "# %s\n", config_line.c_str());
#if defined(__GNUC__) && !defined(__clang__)
std::fprintf(out, "# compiler=gcc-%d.%d.%d\n",
__GNUC__, __GNUC_MINOR__, __GNUC_PATCHLEVEL__);
#elif defined(__clang__)
std::fprintf(out, "# compiler=clang-%d.%d.%d\n",
__clang_major__, __clang_minor__, __clang_patchlevel__);
#endif
}
if (gov != "performance" || ac == "battery")
std::fprintf(stderr,
"# WARNING: governor=%s power=%s — results are not comparable with\n"
"# a run on AC power under the performance governor.\n",
gov.c_str(), ac.c_str());
}
inline double percentile(std::vector<double> v, double p) {
if (v.empty()) return 0;
std::sort(v.begin(), v.end());
double idx = p * (v.size() - 1);
auto lo = static_cast<std::size_t>(std::floor(idx));
auto hi = static_cast<std::size_t>(std::ceil(idx));
return v[lo] + (v[hi] - v[lo]) * (idx - lo);
}
// Process-wide context-switch counters, sampled around a timed region.
//
// ru_nvcsw (voluntary) is the cheap answer to PERF_PLAN B2: a thread that
// blocks on a condition variable books a voluntary switch, so voluntary
// switches per dispatch is, near enough, sleeps per dispatch. ru_nivcsw
// (involuntary) is preemption, which is what oversubscription looks like (A3).
struct RusageDelta {
long ivcsw0 = 0, vcsw0 = 0;
void start() {
rusage ru{};
getrusage(RUSAGE_SELF, &ru);
ivcsw0 = ru.ru_nivcsw;
vcsw0 = ru.ru_nvcsw;
}
void finish(long& nivcsw, long& nvcsw) const {
rusage ru{};
getrusage(RUSAGE_SELF, &ru);
nivcsw = ru.ru_nivcsw - ivcsw0;
nvcsw = ru.ru_nvcsw - vcsw0;
}
};
} // namespace bench
benchmarks/bench_pipeline.cpp
+360 -122
View File
@@ -9,24 +9,38 @@
// private — each node owns a private ThreadPool(1) [Node<>]
// pool — all nodes share one ThreadPool(T) [PoolNode<> + shared pool]
//
// Usage: ./bench_pipeline | tee results.csv
// Each row is run --reps times (plus discarded warm-up runs); the reported
// figure is the median items/sec, with the inter-quartile spread as a
// reliability indicator. A row whose iqr_pct is above a few percent is not
// measuring what it claims to measure.
//
// Usage: ./bench_pipeline [options] | tee results.csv
// ./bench_pipeline --help
#include <kpn/kpn.hpp>
#include "bench_env.hpp"
#ifdef KPN_BENCH_TBB
#include <oneapi/tbb/flow_graph.h>
namespace tbb_flow = oneapi::tbb::flow;
#endif
#include <algorithm>
#include <array>
#include <atomic>
#include <chrono>
#include <cmath>
#include <cstdio>
#include <cstdlib>
#include <cstring>
#include <memory>
#include <string>
#include <thread>
#include <vector>
#include <sys/resource.h>
using namespace kpn;
using namespace std::chrono_literals;
using sclock = std::chrono::steady_clock;
@@ -57,31 +71,63 @@ static void push_retry(Channel<int>& ch, int val) {
}
}
// ── result ────────────────────────────────────────────────────────────────────
// ── configuration (M1, M3, M4, M5) ────────────────────────────────────────────
struct Result {
const char* topology;
int size;
int work_us;
int threads; // 0 = private (1 thread per node), N = shared pool size
double items_per_sec;
double overhead_us;
struct Config {
std::vector<int> work_amts {10, 100, 1000};
std::vector<int> pool_sizes{1, 2, 4, 8, 16, 20}; // M5
std::vector<int> depths {1, 2, 4, 8, 16, 32};
std::vector<int> widths {1, 2, 3, 4};
int reps = 5; // M3: measured repetitions per row
int warmup = 1; // M4: discarded repetitions per row
double target_sec = 0.30; // aimed-for duration of one repetition
long min_items = 2000; // M1: floor, independent of work_us and depth
double max_sec = 3.0; // ceiling; only bites where min_items cannot fit
bool do_chain = true, do_wide = true, do_diamond = true;
bool do_priv = true, do_pool = true, do_tbb = true;
};
static Config g_cfg;
// M1 — sample size from a time budget with a hard floor, rather than a
// hand-tuned ladder that collapsed to 50–200 items on exactly the rows under
// investigation.
//
// `stages` is the number of node firings per item; `units` the number of
// threads able to run them concurrently. Steady-state throughput of the
// pipeline is bounded by work_us * stages / units, so that is the per-item
// cost the sample size is derived from. Depth beyond `units` costs throughput;
// depth below it costs only latency, which does not scale the run.
static long pick_items(int work_us, int stages, int units) {
units = std::max(1, std::min(units, bench::hw_units()));
const double per_item_us =
std::max(1.0, static_cast<double>(work_us)) *
std::max(1.0, static_cast<double>(stages) / units);
long want = static_cast<long>(g_cfg.target_sec * 1e6 / per_item_us);
long cap = static_cast<long>(g_cfg.max_sec * 1e6 / per_item_us);
want = std::max(want, g_cfg.min_items);
// The floor wins unless honouring it would blow the time ceiling by more
// than the ceiling allows; such rows are reported with their true N so the
// reader can see they are short.
if (want > cap) want = std::max(cap, 200L);
return want;
}
// ── one measured repetition ───────────────────────────────────────────────────
struct Sample {
double items_per_sec = 0;
double overhead_us = 0;
long nivcsw = 0; // involuntary context switches during the run
long nvcsw = 0; // voluntary context switches during the run
};
// ── chain ─────────────────────────────────────────────────────────────────────
static int items_for(int work_us, int depth = 1) {
int effective = std::max(1, work_us) * std::max(1, depth);
if (effective <= 1) return 5000;
if (effective <= 10) return 3000;
if (effective <= 100) return 1000;
if (effective <= 1000) return 200;
return 50;
}
static Result bench_chain(int depth, int work_us) {
const int N = items_for(work_us, depth);
const int CAP = N;
static Sample bench_chain(int depth, int work_us, long N) {
const std::size_t CAP = static_cast<std::size_t>(N);
std::vector<std::shared_ptr<Channel<int>>> chs;
for (int i = 0; i <= depth; ++i)
@@ -98,17 +144,20 @@ static Result bench_chain(int depth, int work_us) {
std::atomic<sclock::time_point> t1;
std::thread reader([&] {
for (int i = 0; i < N; ++i) chs.back()->pop();
for (long i = 0; i < N; ++i) chs.back()->pop();
t1.store(sclock::now(), std::memory_order_release);
});
bench::RusageDelta ru; ru.start();
auto t0 = sclock::now();
std::thread pusher([&] {
for (int i = 0; i < N; ++i) push_retry(*chs[0], i);
for (long i = 0; i < N; ++i) push_retry(*chs[0], static_cast<int>(i));
});
pusher.join();
reader.join();
Sample s;
ru.finish(s.nivcsw, s.nvcsw);
for (auto& n : nodes) n->stop();
double elapsed = std::chrono::duration<double>(
@@ -116,13 +165,13 @@ static Result bench_chain(int depth, int work_us) {
// Subtract theoretical pipeline fill cost (depth-1)*W so that overhead
// reflects only framework latency, not the expected pipeline startup time.
double pipeline_us = static_cast<double>(work_us) * (N + depth - 1);
double wus = (elapsed * 1e6 - pipeline_us) / N;
return {"chain", depth, work_us, 0, N / elapsed, wus};
s.overhead_us = (elapsed * 1e6 - pipeline_us) / N;
s.items_per_sec = N / elapsed;
return s;
}
static Result bench_chain_pool(int depth, int work_us, int pool_threads) {
const int N = items_for(work_us, depth);
const int CAP = N;
static Sample bench_chain_pool(int depth, int work_us, int pool_threads, long N) {
const std::size_t CAP = static_cast<std::size_t>(N);
auto pool = std::make_shared<ThreadPool>(pool_threads);
@@ -142,33 +191,36 @@ static Result bench_chain_pool(int depth, int work_us, int pool_threads) {
std::atomic<sclock::time_point> t1;
std::thread reader([&] {
for (int i = 0; i < N; ++i) chs.back()->pop();
for (long i = 0; i < N; ++i) chs.back()->pop();
t1.store(sclock::now(), std::memory_order_release);
});
bench::RusageDelta ru; ru.start();
auto t0 = sclock::now();
std::thread pusher([&] {
for (int i = 0; i < N; ++i) push_retry(*chs[0], i);
for (long i = 0; i < N; ++i) push_retry(*chs[0], static_cast<int>(i));
});
pusher.join();
reader.join();
Sample s;
ru.finish(s.nivcsw, s.nvcsw);
for (auto& n : nodes) n->stop();
pool->stop();
double elapsed = std::chrono::duration<double>(
t1.load(std::memory_order_acquire) - t0).count();
double pipeline_us = static_cast<double>(work_us) * (N + depth - 1);
double wus = (elapsed * 1e6 - pipeline_us) / N;
return {"chain", depth, work_us, pool_threads, N / elapsed, wus};
s.overhead_us = (elapsed * 1e6 - pipeline_us) / N;
s.items_per_sec = N / elapsed;
return s;
}
// ── wide (fanout<W>) ──────────────────────────────────────────────────────────
template<std::size_t W>
static Result bench_wide(int work_us) {
const int N = items_for(work_us);
const int CAP = N;
static Sample bench_wide(int work_us, long N) {
const std::size_t CAP = static_cast<std::size_t>(N);
auto src_ch = std::make_shared<Channel<int>>(CAP);
auto fan = std::make_unique<FanoutNode<int, W>>(CAP);
@@ -197,33 +249,36 @@ static Result bench_wide(int work_us) {
for (std::size_t w = 0; w < W; ++w) {
readers[w] = std::thread([&, w] {
for (int i = 0; i < N; ++i) sink_chs[w]->pop();
for (long i = 0; i < N; ++i) sink_chs[w]->pop();
if (readers_done.fetch_add(1, std::memory_order_acq_rel) + 1
== static_cast<int>(W))
t1.store(sclock::now(), std::memory_order_release);
});
}
bench::RusageDelta ru; ru.start();
auto t0 = sclock::now();
std::thread pusher([&] {
for (int i = 0; i < N; ++i) push_retry(*src_ch, i);
for (long i = 0; i < N; ++i) push_retry(*src_ch, static_cast<int>(i));
});
pusher.join();
for (auto& r : readers) r.join();
Sample s;
ru.finish(s.nivcsw, s.nvcsw);
fan->stop();
for (auto& n : nodes) n->stop();
double elapsed = std::chrono::duration<double>(
t1.load(std::memory_order_acquire) - t0).count();
double wus = (elapsed * 1e6) / N - static_cast<double>(work_us);
return {"wide", static_cast<int>(W), work_us, 0, N / elapsed, wus};
s.overhead_us = (elapsed * 1e6) / N - static_cast<double>(work_us);
s.items_per_sec = N / elapsed;
return s;
}
template<std::size_t W>
static Result bench_wide_pool(int work_us, int pool_threads) {
const int N = items_for(work_us);
const int CAP = N;
static Sample bench_wide_pool(int work_us, int pool_threads, long N) {
const std::size_t CAP = static_cast<std::size_t>(N);
auto pool = std::make_shared<ThreadPool>(pool_threads);
auto src_ch = std::make_shared<Channel<int>>(CAP);
@@ -254,35 +309,38 @@ static Result bench_wide_pool(int work_us, int pool_threads) {
for (std::size_t w = 0; w < W; ++w) {
readers[w] = std::thread([&, w] {
for (int i = 0; i < N; ++i) sink_chs[w]->pop();
for (long i = 0; i < N; ++i) sink_chs[w]->pop();
if (readers_done.fetch_add(1, std::memory_order_acq_rel) + 1
== static_cast<int>(W))
t1.store(sclock::now(), std::memory_order_release);
});
}
bench::RusageDelta ru; ru.start();
auto t0 = sclock::now();
std::thread pusher([&] {
for (int i = 0; i < N; ++i) push_retry(*src_ch, i);
for (long i = 0; i < N; ++i) push_retry(*src_ch, static_cast<int>(i));
});
pusher.join();
for (auto& r : readers) r.join();
Sample s;
ru.finish(s.nivcsw, s.nvcsw);
fan->stop();
for (auto& n : nodes) n->stop();
pool->stop();
double elapsed = std::chrono::duration<double>(
t1.load(std::memory_order_acquire) - t0).count();
double wus = (elapsed * 1e6) / N - static_cast<double>(work_us);
return {"wide", static_cast<int>(W), work_us, pool_threads, N / elapsed, wus};
s.overhead_us = (elapsed * 1e6) / N - static_cast<double>(work_us);
s.items_per_sec = N / elapsed;
return s;
}
// ── diamond ───────────────────────────────────────────────────────────────────
static Result bench_diamond(int work_us) {
const int N = items_for(work_us, 2);
const int CAP = N;
static Sample bench_diamond(int work_us, long N) {
const std::size_t CAP = static_cast<std::size_t>(N);
auto src_ch = std::make_shared<Channel<int>>(CAP);
auto fan = std::make_unique<FanoutNode<int, 2>>(CAP);
@@ -312,7 +370,7 @@ static Result bench_diamond(int work_us) {
std::atomic<int> done{0};
auto make_reader = [&](Channel<int>& ch) {
return std::thread([&] {
for (int i = 0; i < N; ++i) ch.pop();
for (long i = 0; i < N; ++i) ch.pop();
if (done.fetch_add(1, std::memory_order_acq_rel) + 1 == 2)
t1.store(sclock::now(), std::memory_order_release);
});
@@ -320,23 +378,26 @@ static Result bench_diamond(int work_us) {
auto rL = make_reader(*snkL);
auto rR = make_reader(*snkR);
bench::RusageDelta ru; ru.start();
auto t0 = sclock::now();
std::thread pusher([&] {
for (int i = 0; i < N; ++i) push_retry(*src_ch, i);
for (long i = 0; i < N; ++i) push_retry(*src_ch, static_cast<int>(i));
});
pusher.join(); rL.join(); rR.join();
Sample s;
ru.finish(s.nivcsw, s.nvcsw);
fan->stop(); nL->stop(); nR->stop(); nL2->stop(); nR2->stop();
double elapsed = std::chrono::duration<double>(
t1.load(std::memory_order_acquire) - t0).count();
double wus = (elapsed * 1e6) / N - static_cast<double>(work_us);
return {"diamond", 4, work_us, 0, N / elapsed, wus};
s.overhead_us = (elapsed * 1e6) / N - static_cast<double>(work_us);
s.items_per_sec = N / elapsed;
return s;
}
static Result bench_diamond_pool(int work_us, int pool_threads) {
const int N = items_for(work_us, 2);
const int CAP = N;
static Sample bench_diamond_pool(int work_us, int pool_threads, long N) {
const std::size_t CAP = static_cast<std::size_t>(N);
auto pool = std::make_shared<ThreadPool>(pool_threads);
auto src_ch = std::make_shared<Channel<int>>(CAP);
@@ -369,7 +430,7 @@ static Result bench_diamond_pool(int work_us, int pool_threads) {
std::atomic<int> done{0};
auto make_reader = [&](Channel<int>& ch) {
return std::thread([&] {
for (int i = 0; i < N; ++i) ch.pop();
for (long i = 0; i < N; ++i) ch.pop();
if (done.fetch_add(1, std::memory_order_acq_rel) + 1 == 2)
t1.store(sclock::now(), std::memory_order_release);
});
@@ -377,28 +438,30 @@ static Result bench_diamond_pool(int work_us, int pool_threads) {
auto rL = make_reader(*snkL);
auto rR = make_reader(*snkR);
bench::RusageDelta ru; ru.start();
auto t0 = sclock::now();
std::thread pusher([&] {
for (int i = 0; i < N; ++i) push_retry(*src_ch, i);
for (long i = 0; i < N; ++i) push_retry(*src_ch, static_cast<int>(i));
});
pusher.join(); rL.join(); rR.join();
Sample s;
ru.finish(s.nivcsw, s.nvcsw);
fan->stop();
nL->stop(); nR->stop(); nL2->stop(); nR2->stop();
pool->stop();
double elapsed = std::chrono::duration<double>(
t1.load(std::memory_order_acquire) - t0).count();
double wus = (elapsed * 1e6) / N - static_cast<double>(work_us);
return {"diamond", 4, work_us, pool_threads, N / elapsed, wus};
s.overhead_us = (elapsed * 1e6) / N - static_cast<double>(work_us);
s.items_per_sec = N / elapsed;
return s;
}
// ── TBB flow graph ────────────────────────────────────────────────────────────
#ifdef KPN_BENCH_TBB
static Result bench_chain_tbb(int depth, int work_us) {
const int N = items_for(work_us, depth);
static Sample bench_chain_tbb(int depth, int work_us, long N) {
tbb_flow::graph g;
using FN = tbb_flow::function_node<int, int>;
std::vector<std::unique_ptr<FN>> nodes;
@@ -409,21 +472,23 @@ static Result bench_chain_tbb(int depth, int work_us) {
for (int i = 0; i + 1 < depth; ++i)
tbb_flow::make_edge(*nodes[i], *nodes[i + 1]);
bench::RusageDelta ru; ru.start();
auto t0 = sclock::now();
for (int i = 0; i < N; ++i) nodes[0]->try_put(i);
for (long i = 0; i < N; ++i) nodes[0]->try_put(static_cast<int>(i));
g.wait_for_all();
auto t1 = sclock::now();
Sample s;
ru.finish(s.nivcsw, s.nvcsw);
double elapsed = std::chrono::duration<double>(t1 - t0).count();
double elapsed = std::chrono::duration<double>(t1 - t0).count();
double pipeline_us = static_cast<double>(work_us) * (N + depth - 1);
double wus = (elapsed * 1e6 - pipeline_us) / N;
return {"chain_tbb", depth, work_us, -1, N / elapsed, wus};
s.overhead_us = (elapsed * 1e6 - pipeline_us) / N;
s.items_per_sec = N / elapsed;
return s;
}
template<std::size_t W>
static Result bench_wide_tbb(int work_us) {
const int N = items_for(work_us);
static Sample bench_wide_tbb(int work_us, long N) {
tbb_flow::graph g;
tbb_flow::broadcast_node<int> fan(g);
using FN = tbb_flow::function_node<int, int>;
@@ -434,19 +499,21 @@ static Result bench_wide_tbb(int work_us) {
tbb_flow::make_edge(fan, *n);
}
bench::RusageDelta ru; ru.start();
auto t0 = sclock::now();
for (int i = 0; i < N; ++i) fan.try_put(i);
for (long i = 0; i < N; ++i) fan.try_put(static_cast<int>(i));
g.wait_for_all();
auto t1 = sclock::now();
Sample s;
ru.finish(s.nivcsw, s.nvcsw);
double elapsed = std::chrono::duration<double>(t1 - t0).count();
double wus = (elapsed * 1e6) / N - static_cast<double>(work_us);
return {"wide_tbb", static_cast<int>(W), work_us, -1, N / elapsed, wus};
s.overhead_us = (elapsed * 1e6) / N - static_cast<double>(work_us);
s.items_per_sec = N / elapsed;
return s;
}
static Result bench_diamond_tbb(int work_us) {
const int N = items_for(work_us, 2);
static Sample bench_diamond_tbb(int work_us, long N) {
tbb_flow::graph g;
tbb_flow::broadcast_node<int> fan(g);
using FN = tbb_flow::function_node<int, int>;
@@ -456,71 +523,242 @@ static Result bench_diamond_tbb(int work_us) {
tbb_flow::make_edge(fan, nL); tbb_flow::make_edge(fan, nR);
tbb_flow::make_edge(nL, nL2); tbb_flow::make_edge(nR, nR2);
bench::RusageDelta ru; ru.start();
auto t0 = sclock::now();
for (int i = 0; i < N; ++i) fan.try_put(i);
for (long i = 0; i < N; ++i) fan.try_put(static_cast<int>(i));
g.wait_for_all();
auto t1 = sclock::now();
Sample s;
ru.finish(s.nivcsw, s.nvcsw);
double elapsed = std::chrono::duration<double>(t1 - t0).count();
double wus = (elapsed * 1e6) / N - static_cast<double>(work_us);
return {"diamond_tbb", 4, work_us, -1, N / elapsed, wus};
s.overhead_us = (elapsed * 1e6) / N - static_cast<double>(work_us);
s.items_per_sec = N / elapsed;
return s;
}
#endif // KPN_BENCH_TBB
// ── repetition driver (M2, M3, M4) ────────────────────────────────────────────
using bench::percentile;
// A row: median of `reps` repetitions, after `warmup` discarded ones.
// M2 — items/sec is the primary figure; derived overhead is secondary,
// because it is a difference of large numbers and magnifies noise ~10×.
template<class Fn>
static void run_row(const char* topology, int size, int work_us, int sched,
long N, Fn&& one_rep) {
for (int i = 0; i < g_cfg.warmup; ++i) (void)one_rep(); // M4
std::vector<double> ips, ovh;
long ivcsw = 0, vcsw = 0;
for (int i = 0; i < g_cfg.reps; ++i) {
Sample s = one_rep();
ips.push_back(s.items_per_sec);
ovh.push_back(s.overhead_us);
ivcsw += s.nivcsw;
vcsw += s.nvcsw;
}
const double med = percentile(ips, 0.5);
const double q1 = percentile(ips, 0.25);
const double q3 = percentile(ips, 0.75);
const double iqr = med > 0 ? 100.0 * (q3 - q1) / med : 0.0;
const double lo = *std::min_element(ips.begin(), ips.end());
const double hi = *std::max_element(ips.begin(), ips.end());
const double spread = med > 0 ? 100.0 * (hi - lo) / med : 0.0;
const double ivcsw_per_item = static_cast<double>(ivcsw) / (double(N) * g_cfg.reps);
const double vcsw_per_item = static_cast<double>(vcsw) / (double(N) * g_cfg.reps);
const std::string s = sched < 0 ? "tbb"
: sched == 0 ? "priv"
: std::to_string(sched);
std::fprintf(stderr, "%-10s %-5d %-8d %-6s %-8ld %-12.0f %-7.1f %-7.1f %-9.1f %-8.2f %-8.2f\n",
topology, size, work_us, s.c_str(), N,
med, iqr, spread, percentile(ovh, 0.5), ivcsw_per_item, vcsw_per_item);
std::printf("%s,%d,%d,%s,%ld,%d,%.0f,%.0f,%.0f,%.2f,%.2f,%.2f,%.3f,%.3f\n",
topology, size, work_us, s.c_str(), N, g_cfg.reps,
med, lo, hi, iqr, spread, percentile(ovh, 0.5),
ivcsw_per_item, vcsw_per_item);
std::fflush(stdout);
}
// ── argument parsing ──────────────────────────────────────────────────────────
static std::vector<int> parse_int_list(const char* s) {
std::vector<int> out;
const char* p = s;
while (*p) {
char* end = nullptr;
long v = std::strtol(p, &end, 10);
if (end == p) break;
out.push_back(static_cast<int>(v));
p = end;
while (*p == ',' || *p == ' ') ++p;
}
return out;
}
static bool has_word(const std::string& csv, const char* word) {
return csv.find(word) != std::string::npos;
}
static void usage() {
std::fprintf(stderr,
"usage: bench_pipeline [options]\n"
" --work=10,100,1000 per-node busy-work, microseconds\n"
" --depths=1,2,4,8,16,32 chain depths\n"
" --widths=1,2,3,4 fanout widths\n"
" --pools=1,2,4,8,16,20 shared-pool thread counts\n"
" --topos=chain,wide,diamond\n"
" --modes=priv,pool,tbb\n"
" --reps=5 measured repetitions per row\n"
" --warmup=1 discarded repetitions per row\n"
" --target-sec=0.30 aimed-for duration of one repetition\n"
" --min-items=2000 sample-size floor\n"
" --max-sec=3.0 per-repetition ceiling (overrides the floor)\n");
}
static bool parse_args(int argc, char** argv) {
for (int i = 1; i < argc; ++i) {
std::string a = argv[i];
auto eq = a.find('=');
std::string key = a.substr(0, eq);
std::string val = eq == std::string::npos ? "" : a.substr(eq + 1);
if (key == "--help" || key == "-h") { usage(); std::exit(0); }
else if (key == "--work") g_cfg.work_amts = parse_int_list(val.c_str());
else if (key == "--depths") g_cfg.depths = parse_int_list(val.c_str());
else if (key == "--widths") g_cfg.widths = parse_int_list(val.c_str());
else if (key == "--pools") g_cfg.pool_sizes = parse_int_list(val.c_str());
else if (key == "--reps") g_cfg.reps = std::atoi(val.c_str());
else if (key == "--warmup") g_cfg.warmup = std::atoi(val.c_str());
else if (key == "--target-sec") g_cfg.target_sec = std::atof(val.c_str());
else if (key == "--min-items") g_cfg.min_items = std::atol(val.c_str());
else if (key == "--max-sec") g_cfg.max_sec = std::atof(val.c_str());
else if (key == "--topos") {
g_cfg.do_chain = has_word(val, "chain");
g_cfg.do_wide = has_word(val, "wide");
g_cfg.do_diamond = has_word(val, "diamond");
}
else if (key == "--modes") {
g_cfg.do_priv = has_word(val, "priv");
g_cfg.do_pool = has_word(val, "pool");
g_cfg.do_tbb = has_word(val, "tbb");
}
else { std::fprintf(stderr, "unknown option: %s\n", a.c_str()); usage(); return false; }
}
if (g_cfg.reps < 1) g_cfg.reps = 1;
if (g_cfg.warmup < 0) g_cfg.warmup = 0;
return true;
}
// `wide` is templated on W, so dispatch the runtime width through a switch.
template<class F>
static void with_width(int w, F&& f) {
switch (w) {
case 1: f(std::integral_constant<std::size_t, 1>{}); break;
case 2: f(std::integral_constant<std::size_t, 2>{}); break;
case 3: f(std::integral_constant<std::size_t, 3>{}); break;
case 4: f(std::integral_constant<std::size_t, 4>{}); break;
default:
std::fprintf(stderr, "width %d not instantiated (1..4 only)\n", w);
}
}
// ── main ──────────────────────────────────────────────────────────────────────
int main() {
const int work_amts[] = {10, 100, 1000};
const int pool_sizes[] = {1, 2, 4};
int main(int argc, char** argv) {
// A rejected option must fail loudly: a harness driver that silently got
// no CSV back is worse than one that stops.
if (!parse_args(argc, argv)) return 2;
std::fprintf(stderr, "%-12s %-8s %-10s %-8s %-18s %-20s\n",
"topology", "size", "work_us", "threads", "items/sec", "overhead_us/item");
std::fprintf(stderr, "%s\n", std::string(78, '-').c_str());
std::printf("topology,size,work_us,threads,items_per_sec,overhead_us_per_item\n");
char cfg[192];
std::snprintf(cfg, sizeof cfg,
"reps=%d warmup=%d target_sec=%.2f min_items=%ld max_sec=%.1f",
g_cfg.reps, g_cfg.warmup, g_cfg.target_sec,
g_cfg.min_items, g_cfg.max_sec);
bench::print_environment(cfg);
auto emit = [](const Result& r) {
std::string sched = r.threads < 0 ? "tbb"
: r.threads == 0 ? "priv"
: std::to_string(r.threads);
std::fprintf(stderr, "%-12s %-8d %-10d %-8s %-18.0f %-20.1f\n",
r.topology, r.size, r.work_us, sched.c_str(),
r.items_per_sec, r.overhead_us);
std::printf("%s,%d,%d,%s,%.0f,%.2f\n",
r.topology, r.size, r.work_us, sched.c_str(),
r.items_per_sec, r.overhead_us);
std::fflush(stdout);
};
std::fprintf(stderr, "\n%-10s %-5s %-8s %-6s %-8s %-12s %-7s %-7s %-9s %-8s %-8s\n",
"topology", "size", "work_us", "sched", "items", "items/sec",
"iqr%", "range%", "ovh_us", "ivcsw/it", "vcsw/it");
std::fprintf(stderr, "%s\n", std::string(104, '-').c_str());
std::printf("topology,size,work_us,threads,items,reps,items_per_sec,"
"items_per_sec_min,items_per_sec_max,iqr_pct,range_pct,"
"overhead_us_per_item,ivcsw_per_item,vcsw_per_item\n");
for (int w : work_amts) {
for (int w : g_cfg.work_amts) {
g_work_us.store(w, std::memory_order_relaxed);
std::fprintf(stderr, "\n── work_us=%-4d private pools ───────────────────────────────────────\n", w);
for (int d : {1, 2, 4, 8, 16, 32}) emit(bench_chain(d, w));
emit(bench_wide<1>(w));
emit(bench_wide<2>(w));
emit(bench_wide<3>(w));
emit(bench_wide<4>(w));
emit(bench_diamond(w));
if (g_cfg.do_priv) {
std::fprintf(stderr, "\n── work_us=%-4d private pools ──────────────────────\n", w);
if (g_cfg.do_chain)
for (int d : g_cfg.depths) {
long N = pick_items(w, d, d);
run_row("chain", d, w, 0, N, [&] { return bench_chain(d, w, N); });
}
if (g_cfg.do_wide)
for (int wd : g_cfg.widths)
with_width(wd, [&](auto W) {
long N = pick_items(w, W.value, W.value);
run_row("wide", static_cast<int>(W.value), w, 0, N,
[&] { return bench_wide<W.value>(w, N); });
});
if (g_cfg.do_diamond) {
long N = pick_items(w, 4, 4);
run_row("diamond", 4, w, 0, N, [&] { return bench_diamond(w, N); });
}
}
for (int pt : pool_sizes) {
std::fprintf(stderr, "\n── work_us=%-4d shared pool (%d thread%s) ─────────────────────────────\n",
w, pt, pt == 1 ? "" : "s");
for (int d : {1, 2, 4, 8, 16, 32}) emit(bench_chain_pool(d, w, pt));
emit(bench_wide_pool<1>(w, pt));
emit(bench_wide_pool<2>(w, pt));
emit(bench_wide_pool<3>(w, pt));
emit(bench_wide_pool<4>(w, pt));
emit(bench_diamond_pool(w, pt));
if (g_cfg.do_pool) {
for (int pt : g_cfg.pool_sizes) {
std::fprintf(stderr, "\n── work_us=%-4d shared pool (%d thread%s) ───────────\n",
w, pt, pt == 1 ? "" : "s");
if (g_cfg.do_chain)
for (int d : g_cfg.depths) {
long N = pick_items(w, d, pt);
run_row("chain", d, w, pt, N,
[&] { return bench_chain_pool(d, w, pt, N); });
}
if (g_cfg.do_wide)
for (int wd : g_cfg.widths)
with_width(wd, [&](auto W) {
long N = pick_items(w, W.value, pt);
run_row("wide", static_cast<int>(W.value), w, pt, N,
[&] { return bench_wide_pool<W.value>(w, pt, N); });
});
if (g_cfg.do_diamond) {
long N = pick_items(w, 4, pt);
run_row("diamond", 4, w, pt, N,
[&] { return bench_diamond_pool(w, pt, N); });
}
}
}
#ifdef KPN_BENCH_TBB
std::fprintf(stderr, "\n── work_us=%-4d TBB flow graph ──────────────────────────────────────\n", w);
for (int d : {1, 2, 4, 8, 16, 32}) emit(bench_chain_tbb(d, w));
emit(bench_wide_tbb<1>(w));
emit(bench_wide_tbb<2>(w));
emit(bench_wide_tbb<3>(w));
emit(bench_wide_tbb<4>(w));
emit(bench_diamond_tbb(w));
if (g_cfg.do_tbb) {
std::fprintf(stderr, "\n── work_us=%-4d TBB flow graph ─────────────────────\n", w);
if (g_cfg.do_chain)
for (int d : g_cfg.depths) {
long N = pick_items(w, d, d);
run_row("chain_tbb", d, w, -1, N,
[&] { return bench_chain_tbb(d, w, N); });
}
if (g_cfg.do_wide)
for (int wd : g_cfg.widths)
with_width(wd, [&](auto W) {
long N = pick_items(w, W.value, W.value);
run_row("wide_tbb", static_cast<int>(W.value), w, -1, N,
[&] { return bench_wide_tbb<W.value>(w, N); });
});
if (g_cfg.do_diamond) {
long N = pick_items(w, 4, 4);
run_row("diamond_tbb", 4, w, -1, N,
[&] { return bench_diamond_tbb(w, N); });
}
}
#endif
}
}