Actual coverage is 90% (`bun run traces:coverage`), so the gate had ~8 points
of slack — a requirement could stop being traced and CI would not notice.
Per the ratchet policy in the workflow, move it up to sit just under the real
figure.
MIN_COVERAGE_PERCENT in scripts/extract-traces.ts moves in lockstep: the
workflow comment says to keep the two in sync and extract-traces.test.ts
asserts it, so changing only the YAML turns the frontend suite red.
(The stale "fails below 50%" comment in scripts/test-all.sh, wrong since the
threshold moved to 82, was corrected in the preceding commit along with the
rest of that file.)
Four gates that were documented but unenforced, plus the flaky test that
made a full-suite run untrustworthy.
Rust lint/format: CLAUDE.md has required `cargo fmt` and `cargo clippy`
before every commit for as long as the rule existed, yet neither ran
anywhere in CI — the requirement rested on memory alone. Both now run in
build-and-test.yml and build-release.yml. rustfmt and clippy are already
baked into the builder image, so nothing is installed at job time.
`cargo fmt --all -- --check` is strict immediately (the tree is clean).
Clippy is advisory for now: ~51 pre-existing warnings mean `-D warnings`
would fail on unrelated work, so the step carries a TODO to flip the flag
once the backlog clears. A compile error still fails it, so it is not a
no-op.
Traceability threshold: MIN_THRESHOLD sat at 50 while real coverage was
86%, so nearly half the matrix could rot before the gate objected.
Ratcheted to 82 with the policy written down — it only ever goes up, and
is never lowered to make a red build pass. The same figure lives in
MIN_COVERAGE_PERCENT so `traces:coverage` gates locally on the same bar,
and a test fails if the two drift.
Dangling IDs: a TRACES comment could name any well-formed ID and the
extractor accepted it silently, so typos and renames that missed a call
site passed unnoticed. `bun run traces:validate` cross-checks every
traced ID against the table rows in requirements.md and fails with the
referencing files listed. It spans UT/IT as well, which the coverage
orphan list ignores by design. This currently reports DR-189 and UT-188,
which are being defined separately.
Flaky offlineCatalog test: the first dynamic import of the service paid
~1s to transform its dependency graph, charged to a test body against
vitest's 5s default. Alone it passed; under suite-wide contention it
timed out. The import is now warmed at collection time, so no test is
timing the compiler — the timeout is deliberately unchanged. The store
shim also drops subscribers from module instances discarded by
resetModules, which previously leaked across tests.
The coverage gate divided traced counts by hardcoded literals (UR/39,
IR/24, DR/48, JA/3, TOTAL_REQS=114) that had fallen out of date as
requirements grew to 211. It reported 158% coverage — JA alone printed
800% — so the 50% threshold was mathematically unreachable and the job
could not fail. Coverage could have collapsed to 30% and CI would still
have printed a green tick.
Real coverage is 86%. The number was fine; the gate was dead.
extract-traces.ts now owns both sides of the fraction:
- countDefinedRequirements() counts an ID only where it leads a markdown
table row, ignoring the "Traces To" column and prose. IDs are
deduplicated because requirements.md lists every UR twice (§1
definition + §3 matrix), which would otherwise report UR as 121/61.
- computeCoverage() uses the intersection of traced and defined IDs, so
a TRACES comment naming a deleted or typo'd requirement is reported as
`orphaned` rather than inflating the ratio past 100%. UT/IT test
identifiers are excluded as a separate taxonomy.
- CI reads .coverage.percent and fails on <50% or >100%; a >100% reading
is now a hard error rather than the condition that hid this bug.
- New `bun run traces:coverage` runs the same computation locally.
- scripts/ added to the scan roots — the coverage tool was invisible to
the matrix it generates.
Tests written first (15, over fixtures so they don't drift as
requirements are added). vitest include widened to scripts/** so build
tooling is covered by the normal suite.
Verified empirically rather than by inspection: forcing the threshold to
99% fails; adding a requirement lowers coverage 86%→85%; a TRACES: DR-999
lands in `orphaned` without changing `covered`.
traceability-ci.md documented the same stale numbers and would have let
the broken arithmetic be reconstructed — replaced with a pointer to the
live command.
dtourolle
