fix(ci): derive traceability denominators from requirements.md (DR-093)

The coverage gate divided traced counts by hardcoded literals (UR/39,
IR/24, DR/48, JA/3, TOTAL_REQS=114) that had fallen out of date as
requirements grew to 211. It reported 158% coverage — JA alone printed
800% — so the 50% threshold was mathematically unreachable and the job
could not fail. Coverage could have collapsed to 30% and CI would still
have printed a green tick.

Real coverage is 86%. The number was fine; the gate was dead.

extract-traces.ts now owns both sides of the fraction:

- countDefinedRequirements() counts an ID only where it leads a markdown
  table row, ignoring the "Traces To" column and prose. IDs are
  deduplicated because requirements.md lists every UR twice (§1
  definition + §3 matrix), which would otherwise report UR as 121/61.
- computeCoverage() uses the intersection of traced and defined IDs, so
  a TRACES comment naming a deleted or typo'd requirement is reported as
  `orphaned` rather than inflating the ratio past 100%. UT/IT test
  identifiers are excluded as a separate taxonomy.
- CI reads .coverage.percent and fails on <50% or >100%; a >100% reading
  is now a hard error rather than the condition that hid this bug.
- New `bun run traces:coverage` runs the same computation locally.
- scripts/ added to the scan roots — the coverage tool was invisible to
  the matrix it generates.

Tests written first (15, over fixtures so they don't drift as
requirements are added). vitest include widened to scripts/** so build
tooling is covered by the normal suite.

Verified empirically rather than by inspection: forcing the threshold to
99% fails; adding a requirement lowers coverage 86%→85%; a TRACES: DR-999
lands in `orphaned` without changing `covered`.

traceability-ci.md documented the same stale numbers and would have let
the broken arithmetic be reconstructed — replaced with a pointer to the
live command.
This commit is contained in:
2026-07-30 10:30:08 +02:00
parent 75bae2556c
commit 0da0a9f16c
6 changed files with 435 additions and 44 deletions
+26 -12
View File
@@ -43,14 +43,26 @@ Extracts all TRACES comments from:
### 2. Coverage Thresholds
The workflow checks:
- **Minimum overall coverage:** 50% (57+ requirements traced)
- **Requirements by type:**
- UR (User): 23+ of 39
- IR (Integration): 5+ of 24
- DR (Development): 28+ of 48
- JA (Jellyfin API): 0+ of 3
- **Minimum overall coverage:** 50%
If coverage drops below threshold, the workflow **fails** and blocks merge.
Denominators are **derived from `docs/requirements.md` at run time** — they are
never hardcoded here or in the workflow. Run `bun run traces:coverage` for the
current per-type breakdown; any number written into this document is a snapshot
that will drift.
> **Why this matters.** The workflow used to divide by frozen literals
> (UR/39, IR/24, DR/48, JA/3, total 114) while `requirements.md` had grown past
> 200. It reported **158%** coverage, so the 50% threshold was unreachable and
> the job could not fail regardless of how far coverage dropped. See
> [specs/traceability-gate-repair.md](specs/traceability-gate-repair.md).
Coverage is the *intersection* of traced and defined IDs: an ID that appears in
a `TRACES:` comment but is not defined in `requirements.md` is reported as
**orphaned** and does not count toward coverage. UT/IT test identifiers are a
separate taxonomy and are excluded entirely.
The workflow **fails** and blocks merge if coverage drops below 50% — or if it
computes above 100%, which can only mean the gate is miscounting.
### 3. Modified File Checking
On pull requests, the workflow:
@@ -153,11 +165,13 @@ cat docs/traceability.md
## Coverage Goals
### Current Status
- Overall: 51% (56/114)
- UR: 59% (23/39)
- IR: 21% (5/24)
- DR: 58% (28/48)
- JA: 0% (0/3)
Run `bun run traces:coverage` — it prints the live figure and exits non-zero
below threshold. Numbers are deliberately not pinned here; the previous snapshot
in this section (51%, 56/114) was stale by roughly 100 requirements and was what
made the broken CI arithmetic look plausible for so long.
As of July 2026 overall coverage is ~86% (182/212).
### Targets
- **Short term** (Sprint): Maintain ≥50% overall