fix(ci): derive traceability denominators from requirements.md (DR-093)

The coverage gate divided traced counts by hardcoded literals (UR/39,
IR/24, DR/48, JA/3, TOTAL_REQS=114) that had fallen out of date as
requirements grew to 211. It reported 158% coverage — JA alone printed
800% — so the 50% threshold was mathematically unreachable and the job
could not fail. Coverage could have collapsed to 30% and CI would still
have printed a green tick.

Real coverage is 86%. The number was fine; the gate was dead.

extract-traces.ts now owns both sides of the fraction:

- countDefinedRequirements() counts an ID only where it leads a markdown
  table row, ignoring the "Traces To" column and prose. IDs are
  deduplicated because requirements.md lists every UR twice (§1
  definition + §3 matrix), which would otherwise report UR as 121/61.
- computeCoverage() uses the intersection of traced and defined IDs, so
  a TRACES comment naming a deleted or typo'd requirement is reported as
  `orphaned` rather than inflating the ratio past 100%. UT/IT test
  identifiers are excluded as a separate taxonomy.
- CI reads .coverage.percent and fails on <50% or >100%; a >100% reading
  is now a hard error rather than the condition that hid this bug.
- New `bun run traces:coverage` runs the same computation locally.
- scripts/ added to the scan roots — the coverage tool was invisible to
  the matrix it generates.

Tests written first (15, over fixtures so they don't drift as
requirements are added). vitest include widened to scripts/** so build
tooling is covered by the normal suite.

Verified empirically rather than by inspection: forcing the threshold to
99% fails; adding a requirement lowers coverage 86%→85%; a TRACES: DR-999
lands in `orphaned` without changing `covered`.

traceability-ci.md documented the same stale numbers and would have let
the broken arithmetic be reconstructed — replaced with a pointer to the
live command.
This commit is contained in:
2026-07-30 10:30:08 +02:00
parent 75bae2556c
commit 0da0a9f16c
6 changed files with 435 additions and 44 deletions
+30 -15
View File
@@ -42,30 +42,45 @@ jobs:
echo "📊 Validating requirement traceability..."
echo ""
# Parse JSON
# Denominators come from docs/requirements.md at run time — NEVER
# hardcode them here. This step previously divided by frozen literals
# (UR/39, IR/24, DR/48, JA/3, total 114) while the file had grown to
# 211 requirements, so it reported 158% coverage and the threshold
# below could never trip. See docs/specs/traceability-gate-repair.md.
TOTAL_TRACES=$(jq '.totalTraces' traces-report.json)
UR=$(jq '.byType.UR | length' traces-report.json)
IR=$(jq '.byType.IR | length' traces-report.json)
DR=$(jq '.byType.DR | length' traces-report.json)
JA=$(jq '.byType.JA | length' traces-report.json)
COVERED=$(jq '.coverage.covered' traces-report.json)
TOTAL_REQS=$(jq '.coverage.total' traces-report.json)
COVERAGE=$(jq '.coverage.percent' traces-report.json)
# Print coverage report
echo "✅ TRACES Found: $TOTAL_TRACES"
echo ""
echo "📋 Coverage Summary:"
echo " User Requirements (UR): $UR / 39 ($(( UR * 100 / 39 ))%)"
echo " Integration Requirements (IR): $IR / 24 ($(( IR * 100 / 24 ))%)"
echo " Development Requirements (DR): $DR / 48 ($(( DR * 100 / 48 ))%)"
echo " Jellyfin API Requirements (JA): $JA / 3 ($(( JA * 100 / 3 ))%)"
echo "📋 Coverage Summary (traced / defined):"
for T in UR IR DR JA; do
TRACED=$(jq --arg t "$T" '[.byType[$t][] | select(. != null)] | length' traces-report.json)
DEFINED=$(jq --arg t "$T" '.defined[$t]' traces-report.json)
echo " $T: $TRACED / $DEFINED"
done
echo ""
COVERED=$((UR + IR + DR + JA))
TOTAL_REQS=114
COVERAGE=$((COVERED * 100 / TOTAL_REQS))
echo "📈 Overall Coverage: $COVERED / $TOTAL_REQS ($COVERAGE%)"
echo ""
# Traced IDs that requirements.md does not define (typo, or a deleted
# requirement). These do not count toward coverage.
ORPHANED=$(jq -c '.coverage.orphaned' traces-report.json)
if [ "$ORPHANED" != "[]" ]; then
echo "⚠️ Traced but not defined in requirements.md: $ORPHANED"
echo ""
fi
# A ratio above 100% means the computation is broken — the exact
# condition that hid the stale-denominator bug. Fail loudly.
if [ "$COVERAGE" -gt 100 ]; then
echo "❌ ERROR: Coverage ($COVERAGE%) exceeds 100% — the gate is miscomputing."
echo " Orphaned IDs: $ORPHANED"
exit 1
fi
# Check minimum threshold
MIN_THRESHOLD=50
if [ "$COVERAGE" -lt "$MIN_THRESHOLD" ]; then