Measure the performance targets §8 has been promising, and fail on a regression

docs/requirements.md §8 has said since it was written that performance is
verified by "an automated benchmark suite against a synthetic 50k catalog, run
per-commit … A regression beyond stated tolerance fails the build." There was
none. No benches/, no [[bench]], no criterion, no synthetic catalog, and three
CI workflows that between them measured nothing. Ten performance requirements
could therefore be neither passed nor failed, and five of them carried a
TRACES: tag regardless.

tools/bench is the half of that promise that can be kept honestly on a runner
with no GPU and no display.

# The fixture

Rows are cheap and pixels are not, so it builds fifty thousand catalog rows
over a pool of a dozen real files, each referenced by several thousand of them.
Everything the catalog half touches is rows and is exact at full scale;
everything the pixel half touches is one file at a time and does not care how
many rows point at it. Fourteen megabytes on disk instead of two terabytes, and
neither half is flattered by the trade. It is reproducible from a seed, and a
stamp beside it — seed, row count, source size, dr-catalog's schema version —
rebuilds it rather than letting a run be compared against a baseline that
describes a different library.

# What it can now pass or fail

NFR-P1, and R2's second sentence with it: Catalog::open plus the count, first
window and timeline the grid cannot paint without. The interesting part turned
out to be the open itself — schema::backfill runs three passes over the images
table on every open, which is O(library) work on a path whose budget is stated
in absolute seconds. Tagged TRACES: NFR-P1, on a gate that fails if it breaks.

NFR-P3: thumbnail throughput on the embedded preview path, through the same
per-image work spawn_thumbnail_sweep does and in the same shape — chunks of 96,
lanes owning disjoint slices, the single thread that owns the store writing the
finished chunk. Mirrored rather than called, because that function takes a
RemoteBackend and would measure somebody's network. Tagged TRACES: NFR-P3.

# What it deliberately does not claim

NFR-P7 is the whole chain, and only the encode half of it runs without an
adapter. So the export row is a one-sided gate — over two seconds in the encode
alone violates the requirement; under it proves nothing — and there is no
TRACES: NFR-P7 anywhere. NFR-P8 is about the application at idle, and the probe
is a process holding the catalog and nothing else, so it records the catalog
layer's share and carries no budget until somebody decides what that share
should be. No tag there either. CONTRIBUTING.md asks that a requirement be
closed by a test that would fail if the behaviour were removed, and two more
plumbing tags is what this repository already has too many of.

NFR-P8 also gets the answer §4.1 demands: RSS is exclusive of device-local GPU
allocations and cannot be made otherwise, because such an allocation never
enters the process's address space. The requirement should be restated as two
figures, and docs/benchmarks.md says so.

# Two gates, and why one of them steps aside off the reference desktop

The budget is the requirement's own number and never moves. The baseline is
what the reference desktop last measured, and drifting 15% past it fails the
build even while still inside the budget — which is how performance rot
actually arrives, never over the line, always a little worse.

A budget written for twenty-four threads cannot be asserted on a two-core
container. §8 names the reference desktop, not CI, so each metric declares
whether its budget is machine-sensitive; those are asserted under --reference
and reported everywhere else. Catalog open is not one of them: two seconds
against an expected figure two orders of magnitude smaller is a threshold any
machine can be held to. This is the trap core/dr-gpu/tests/frame_budget.rs
already refuses — a red gate everybody learns to ignore.

# The baseline ships with no numbers in it

Every recorded field is null, because nobody has run it yet. Writing
plausible-looking figures would make every later comparison a comparison
against a guess, and the first real regression would be invisible. Run
`dr-bench record --reference` on the reference desktop and commit the diff;
until then the budget gate works and the report says the other one cannot.

# CI

.gitea/workflows/benchmark.yml, and its own workflow rather than a step in
build-and-test.yml: a red "Build and test" says the code is wrong, a red
"Benchmarks" says it got slower, and the second must not be reachable by
retrying a flaky compile. The cpu job runs on every push and builds -p dr-bench
alone — which is why that crate depends on no GPU and no UI crate. The gpu job
is the frame budget that already exists and already skips without an adapter,
on workflow_dispatch, because building wgpu on every commit to rediscover that
the runner has no device is not a use of anybody's minutes.

docs/outstanding.md

@@ -285,31 +285,43 @@ load-bearing for interoperating with the editors FR-CAT-14 imports from.
 
 ---
 
-## 8. The performance targets are unverified, not unmet
+## 8. The performance targets are half-verified, and the half that is left is the hard one
 
-Eleven of the fifteen §4.1 targets carry no tag: NFR-P2, -P3, -P4, -P6, -P7, -P8, -P10, -P11, -P12,
--P14, -P15. That is the uninteresting part of this section.
+§8 and §4.1 both require the same thing in the same words: an automated benchmark suite against a
+synthetic 50k catalog, run per commit, where **"a regression beyond a stated tolerance is a build
+failure, not a notification."** For most of this project's life it did not exist — no `benches/`, no
+criterion, no synthetic catalog, and three CI workflows that between them measured nothing.
 
-The interesting part is that §8 and §4.1 both require the same thing, in the same words, and it does
-not exist: an automated benchmark suite against a synthetic 50k catalog, run per commit, where **"a
-regression beyond a stated tolerance is a build failure, not a notification."** There is no
-`benches/` directory in the workspace, no criterion dependency, and no synthetic catalog. The three
-CI workflows run `cargo fmt --check`, clippy, `cargo test --workspace`, a release build, an Android
-cross-build and a layering check. None of them measures anything, so there is no baseline to
-regress against and no tolerance to exceed.
+**It exists now, for everything that does not need a frame.** [`tools/bench`](../tools/bench) builds
+a deterministic 50,000-row catalog over a pool of a dozen real files, measures against it, and fails
+the build on a violated budget or a drift past tolerance;
+[`.gitea/workflows/benchmark.yml`](../.gitea/workflows/benchmark.yml) runs it on every push, and
+[benchmarks.md](benchmarks.md) is the account of what it does and does not cover. **NFR-P1** and
+**NFR-P3** are now genuinely gated, and R2's "catalog opens in under 2s" clause with them.
 
-What does exist is narrower and genuinely good: `dr-gpu/examples/frame_budget` is a real instrument,
-its results are committed in [frame-budget.md](frame-budget.md) with the machine and profile named,
-and TD-4's before-and-after was measured with it. But it is run by hand — frame-budget.md's own
-instruction is "rerun and diff this file" — and the guard version that does live in CI skips itself
-where there is no GPU adapter, which the workflow notes is the normal case on a runner, while
-asserting its CPU half only when `debug_assertions` is off, which a dev-profile `cargo test` is not.
-In CI it therefore asserts approximately nothing.
+Three qualifications, all of them stated in the harness itself rather than only here:
 
-**The claim to take from this is precise.** Nothing here says the performance targets are missed.
-Several are plausibly met. It says that if one were broken tomorrow, nobody would find out — which
-is the failure mode §8 was written to prevent, and the reason it belongs in this document rather
-than in a backlog.
+- **The numbers have not been recorded yet.** Every `recorded` field in
+  [bench-baseline.json](bench-baseline.json) is `null`, deliberately: a fabricated baseline is worse
+  than none. Until `dr-bench record --reference` is run on the reference desktop and committed, the
+  budget gate works and the regression gate does not.
+- **NFR-P7 and NFR-P8 are half-measured and are not tagged.** The export row covers the encode half
+  of the chain and no GPU render, so it can fail the requirement and cannot pass it. The memory row
+  covers a process holding the catalog and nothing else — no toolkit, no adapter — so it is the
+  catalog layer's share of the 500 MB rather than the figure NFR-P8 is about. Neither carries a
+  `TRACES:` tag, which is the point.
+- **NFR-P8 needs a decision, not more code.** How much of its 500 MB belongs below the UI is
+  unstated, and until somebody says, the metric can record but not judge. [benchmarks.md](benchmarks.md)
+  also answers the question §4.1 raises about GPU memory — RSS cannot see device-local allocations
+  at all — and recommends restating the requirement as two figures.
+
+**What is left is the frame-timing half, and it is the hard one.** NFR-P2, -P4, -P5, -P6, -P9, -P10,
+-P11, -P12, -P13, -P14 and -P15 all need a probe inside a running Slint application, a GPU adapter,
+or both. `dr-gpu/examples/frame_budget` is a real instrument for the GPU part and its results are
+committed in [frame-budget.md](frame-budget.md) with the machine and profile named — but it is run
+by hand, and the guard version in CI skips itself where there is no adapter, which is the normal
+case on a runner. So the claim to take from this section is now narrower than it was, and still
+true: **a scroll that dropped to 30 fps tomorrow would reach a user before it reached CI.**
 
 ---