perf(bench): make a multi-hour acceptance run observable and restartable
The Phase-0 gate is 7 passes over the full row set, which is long enough that capture_output=True was the wrong default: no rows existed anywhere until a pass ended, so a slow run and a wedged one looked identical, and killing either threw away everything measured. Rows now stream to --out-dir/pass-NN.csv with a flush per line, so an interrupted run keeps what it had. --resume reuses complete pass files and reruns incomplete ones -- checked against the row count rather than merely existing, because a partial file silently averaged in as a whole pass would corrupt the verdict rather than fail it. Progress is a tqdm bar over rows, not passes; a pass counter would sit at 1/7 for twenty minutes and report nothing useful. The row total comes from probing the binary with --reps=0 (0.8s) rather than reimplementing the sweep in Python, which would drift from the C++ defaults. There is a plain-stderr fallback when tqdm is absent -- refusing to start a benchmark over a missing progress dependency is the wrong trade. bench_runs/ is gitignored: the new default output path would otherwise land in git status. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -25,6 +25,9 @@ venv/
|
||||
.DS_Store
|
||||
Thumbs.db
|
||||
|
||||
# Benchmark output (scripts/bench_repro_check.py --out-dir)
|
||||
bench_runs/
|
||||
|
||||
# Claude Code local settings
|
||||
.claude/settings.local.json
|
||||
include/kpn/ort_cache/
|
||||
|
||||
Reference in New Issue
Block a user