ci: enforce the checks the contributor rules already required

Four gates that were documented but unenforced, plus the flaky test that
made a full-suite run untrustworthy.

Rust lint/format: CLAUDE.md has required `cargo fmt` and `cargo clippy`
before every commit for as long as the rule existed, yet neither ran
anywhere in CI — the requirement rested on memory alone. Both now run in
build-and-test.yml and build-release.yml. rustfmt and clippy are already
baked into the builder image, so nothing is installed at job time.
`cargo fmt --all -- --check` is strict immediately (the tree is clean).
Clippy is advisory for now: ~51 pre-existing warnings mean `-D warnings`
would fail on unrelated work, so the step carries a TODO to flip the flag
once the backlog clears. A compile error still fails it, so it is not a
no-op.

Traceability threshold: MIN_THRESHOLD sat at 50 while real coverage was
86%, so nearly half the matrix could rot before the gate objected.
Ratcheted to 82 with the policy written down — it only ever goes up, and
is never lowered to make a red build pass. The same figure lives in
MIN_COVERAGE_PERCENT so `traces:coverage` gates locally on the same bar,
and a test fails if the two drift.

Dangling IDs: a TRACES comment could name any well-formed ID and the
extractor accepted it silently, so typos and renames that missed a call
site passed unnoticed. `bun run traces:validate` cross-checks every
traced ID against the table rows in requirements.md and fails with the
referencing files listed. It spans UT/IT as well, which the coverage
orphan list ignores by design. This currently reports DR-189 and UT-188,
which are being defined separately.

Flaky offlineCatalog test: the first dynamic import of the service paid
~1s to transform its dependency graph, charged to a test body against
vitest's 5s default. Alone it passed; under suite-wide contention it
timed out. The import is now warmed at collection time, so no test is
timing the compiler — the timeout is deliberately unchanged. The store
shim also drops subscribers from module instances discarded by
resetModules, which previously leaked across tests.
This commit is contained in:
2026-08-16 22:51:44 +02:00
parent 73641e192c
commit b9dab56379
11 changed files with 355 additions and 31 deletions
+42 -10
View File
@@ -15,7 +15,7 @@ The CI/CD pipeline automatically validates that code changes are properly traced
Traceability validation lives in `.gitea/workflows/traceability-check.yml`:
- ✅ Automatic trace extraction
- ✅ Coverage validation against minimum threshold (50%)
- ✅ Coverage validation against minimum threshold (82%, ratcheted)
- ✅ Modified file checking
- ✅ Artifact preservation
- ✅ Summary reports
@@ -43,7 +43,7 @@ Extracts all TRACES comments from:
### 2. Coverage Thresholds
The workflow checks:
- **Minimum overall coverage:** 50%
- **Minimum overall coverage:** 82% (`MIN_THRESHOLD`)
Denominators are **derived from `docs/requirements.md` at run time** — they are
never hardcoded here or in the workflow. Run `bun run traces:coverage` for the
@@ -61,8 +61,39 @@ a `TRACES:` comment but is not defined in `requirements.md` is reported as
**orphaned** and does not count toward coverage. UT/IT test identifiers are a
separate taxonomy and are excluded entirely.
The workflow **fails** and blocks merge if coverage drops below 50% — or if it
computes above 100%, which can only mean the gate is miscounting.
The workflow **fails** and blocks merge if coverage drops below the threshold —
or if it computes above 100%, which can only mean the gate is miscounting.
#### Ratchet policy
`MIN_THRESHOLD` **only ever goes up.** It is deliberately set a few points below
the coverage actually achieved (82 against a real 86%), so a genuine regression
trips it. It previously sat at 50 while true coverage was 86%: nearly half the
matrix could have rotted before CI objected.
When coverage rises durably, raise the threshold to just under the new figure.
**Never lower it to make a red build pass** — add the missing TRACES comments
instead. The same number lives in `MIN_COVERAGE_PERCENT` in
`scripts/extract-traces.ts` (so `bun run traces:coverage` gates locally on the
same bar); `scripts/extract-traces.test.ts` fails if the two drift apart.
### 2b. Dangling requirement IDs
```bash
bun run traces:validate
```
Every ID named by a `TRACES:` comment must be defined as a table row in
`docs/requirements.md`. The extractor used to accept any well-formed ID
silently, so a typo or a rename that missed a call site passed unnoticed —
`DR-189` and `UT-188` were referenced from three source files, defined nowhere,
for months.
This check spans **all six** ID types (UR/IR/DR/JA/UT/IT), unlike the coverage
`orphaned` list above, which considers only the four requirement types so that
UT/IT noise cannot bury a real typo in the ratio's reporting. The workflow step
**fails the build** on any dangling ID and prints each offender with the files
that reference it.
### 3. Modified File Checking
On pull requests, the workflow:
@@ -120,13 +151,13 @@ TRACES: [UR-###, ...] | [IR-###, ...] | [DR-###, ...] | [JA-###, ...]
### On Push to Main Branch
1. ✅ Extracts all traces from code
2. ✅ Validates coverage is >= 50%
2. ✅ Validates coverage is >= 82%
3. ✅ Generates full traceability report
4. ✅ Saves report as artifact
### On Pull Request
1. ✅ Extracts all traces
2. ✅ Validates coverage >= 50%
2. ✅ Validates coverage >= 82%
3. ✅ Checks modified files for TRACES
4. ✅ Warns if new code lacks TRACES
5. ✅ Suggests proper format
@@ -134,7 +165,8 @@ TRACES: [UR-###, ...] | [IR-###, ...] | [DR-###, ...] | [JA-###, ...]
### Failure Scenarios
The workflow **fails** (blocks merge) if:
- Coverage drops below 50%
- Coverage drops below 82%
- A `TRACES:` comment names an ID `docs/requirements.md` does not define
- JSON extraction fails
- Invalid trace format
@@ -174,7 +206,7 @@ made the broken CI arithmetic look plausible for so long.
As of July 2026 overall coverage is ~86% (182/212).
### Targets
- **Short term** (Sprint): Maintain ≥50% overall
- **Short term** (Sprint): Maintain ≥82% overall (the current ratchet)
- **Medium term** (Month): Reach 70% overall coverage
- **Long term** (Release): Reach 90% coverage with focus on:
- IR requirements (API clients)
@@ -209,14 +241,14 @@ When submitting a pull request:
- [ ] All new code has TRACES comments linking to requirements
- [ ] TRACES format is correct: `// TRACES: UR-001 | DR-002`
- [ ] Workflow passes (coverage ≥ 50%)
- [ ] Workflow passes (coverage ≥ 82%)
- [ ] No coverage regressions
- [ ] Artifact traceability report was generated
## Troubleshooting
### "Coverage below minimum threshold"
**Problem:** Workflow fails with coverage < 50%
**Problem:** Workflow fails with coverage < 82%
**Solution:**
1. Run `bun run traces:json` locally