fix(ci): derive traceability denominators from requirements.md (DR-093)
The coverage gate divided traced counts by hardcoded literals (UR/39, IR/24, DR/48, JA/3, TOTAL_REQS=114) that had fallen out of date as requirements grew to 211. It reported 158% coverage — JA alone printed 800% — so the 50% threshold was mathematically unreachable and the job could not fail. Coverage could have collapsed to 30% and CI would still have printed a green tick. Real coverage is 86%. The number was fine; the gate was dead. extract-traces.ts now owns both sides of the fraction: - countDefinedRequirements() counts an ID only where it leads a markdown table row, ignoring the "Traces To" column and prose. IDs are deduplicated because requirements.md lists every UR twice (§1 definition + §3 matrix), which would otherwise report UR as 121/61. - computeCoverage() uses the intersection of traced and defined IDs, so a TRACES comment naming a deleted or typo'd requirement is reported as `orphaned` rather than inflating the ratio past 100%. UT/IT test identifiers are excluded as a separate taxonomy. - CI reads .coverage.percent and fails on <50% or >100%; a >100% reading is now a hard error rather than the condition that hid this bug. - New `bun run traces:coverage` runs the same computation locally. - scripts/ added to the scan roots — the coverage tool was invisible to the matrix it generates. Tests written first (15, over fixtures so they don't drift as requirements are added). vitest include widened to scripts/** so build tooling is covered by the normal suite. Verified empirically rather than by inspection: forcing the threshold to 99% fails; adding a requirement lowers coverage 86%→85%; a TRACES: DR-999 lands in `orphaned` without changing `covered`. traceability-ci.md documented the same stale numbers and would have let the broken arithmetic be reconstructed — replaced with a pointer to the live command.
This commit is contained in:
+26
-12
@@ -43,14 +43,26 @@ Extracts all TRACES comments from:
|
||||
|
||||
### 2. Coverage Thresholds
|
||||
The workflow checks:
|
||||
- **Minimum overall coverage:** 50% (57+ requirements traced)
|
||||
- **Requirements by type:**
|
||||
- UR (User): 23+ of 39
|
||||
- IR (Integration): 5+ of 24
|
||||
- DR (Development): 28+ of 48
|
||||
- JA (Jellyfin API): 0+ of 3
|
||||
- **Minimum overall coverage:** 50%
|
||||
|
||||
If coverage drops below threshold, the workflow **fails** and blocks merge.
|
||||
Denominators are **derived from `docs/requirements.md` at run time** — they are
|
||||
never hardcoded here or in the workflow. Run `bun run traces:coverage` for the
|
||||
current per-type breakdown; any number written into this document is a snapshot
|
||||
that will drift.
|
||||
|
||||
> **Why this matters.** The workflow used to divide by frozen literals
|
||||
> (UR/39, IR/24, DR/48, JA/3, total 114) while `requirements.md` had grown past
|
||||
> 200. It reported **158%** coverage, so the 50% threshold was unreachable and
|
||||
> the job could not fail regardless of how far coverage dropped. See
|
||||
> [specs/traceability-gate-repair.md](specs/traceability-gate-repair.md).
|
||||
|
||||
Coverage is the *intersection* of traced and defined IDs: an ID that appears in
|
||||
a `TRACES:` comment but is not defined in `requirements.md` is reported as
|
||||
**orphaned** and does not count toward coverage. UT/IT test identifiers are a
|
||||
separate taxonomy and are excluded entirely.
|
||||
|
||||
The workflow **fails** and blocks merge if coverage drops below 50% — or if it
|
||||
computes above 100%, which can only mean the gate is miscounting.
|
||||
|
||||
### 3. Modified File Checking
|
||||
On pull requests, the workflow:
|
||||
@@ -153,11 +165,13 @@ cat docs/traceability.md
|
||||
## Coverage Goals
|
||||
|
||||
### Current Status
|
||||
- Overall: 51% (56/114)
|
||||
- UR: 59% (23/39)
|
||||
- IR: 21% (5/24)
|
||||
- DR: 58% (28/48)
|
||||
- JA: 0% (0/3)
|
||||
|
||||
Run `bun run traces:coverage` — it prints the live figure and exits non-zero
|
||||
below threshold. Numbers are deliberately not pinned here; the previous snapshot
|
||||
in this section (51%, 56/114) was stale by roughly 100 requirements and was what
|
||||
made the broken CI arithmetic look plausible for so long.
|
||||
|
||||
As of July 2026 overall coverage is ~86% (182/212).
|
||||
|
||||
### Targets
|
||||
- **Short term** (Sprint): Maintain ≥50% overall
|
||||
|
||||
Reference in New Issue
Block a user