fix(ci): derive traceability denominators from requirements.md (DR-093)
The coverage gate divided traced counts by hardcoded literals (UR/39, IR/24, DR/48, JA/3, TOTAL_REQS=114) that had fallen out of date as requirements grew to 211. It reported 158% coverage — JA alone printed 800% — so the 50% threshold was mathematically unreachable and the job could not fail. Coverage could have collapsed to 30% and CI would still have printed a green tick. Real coverage is 86%. The number was fine; the gate was dead. extract-traces.ts now owns both sides of the fraction: - countDefinedRequirements() counts an ID only where it leads a markdown table row, ignoring the "Traces To" column and prose. IDs are deduplicated because requirements.md lists every UR twice (§1 definition + §3 matrix), which would otherwise report UR as 121/61. - computeCoverage() uses the intersection of traced and defined IDs, so a TRACES comment naming a deleted or typo'd requirement is reported as `orphaned` rather than inflating the ratio past 100%. UT/IT test identifiers are excluded as a separate taxonomy. - CI reads .coverage.percent and fails on <50% or >100%; a >100% reading is now a hard error rather than the condition that hid this bug. - New `bun run traces:coverage` runs the same computation locally. - scripts/ added to the scan roots — the coverage tool was invisible to the matrix it generates. Tests written first (15, over fixtures so they don't drift as requirements are added). vitest include widened to scripts/** so build tooling is covered by the normal suite. Verified empirically rather than by inspection: forcing the threshold to 99% fails; adding a requirement lowers coverage 86%→85%; a TRACES: DR-999 lands in `orphaned` without changing `covered`. traceability-ci.md documented the same stale numbers and would have let the broken arithmetic be reconstructed — replaced with a pointer to the live command.
This commit is contained in:
@@ -42,30 +42,45 @@ jobs:
|
||||
echo "📊 Validating requirement traceability..."
|
||||
echo ""
|
||||
|
||||
# Parse JSON
|
||||
# Denominators come from docs/requirements.md at run time — NEVER
|
||||
# hardcode them here. This step previously divided by frozen literals
|
||||
# (UR/39, IR/24, DR/48, JA/3, total 114) while the file had grown to
|
||||
# 211 requirements, so it reported 158% coverage and the threshold
|
||||
# below could never trip. See docs/specs/traceability-gate-repair.md.
|
||||
TOTAL_TRACES=$(jq '.totalTraces' traces-report.json)
|
||||
UR=$(jq '.byType.UR | length' traces-report.json)
|
||||
IR=$(jq '.byType.IR | length' traces-report.json)
|
||||
DR=$(jq '.byType.DR | length' traces-report.json)
|
||||
JA=$(jq '.byType.JA | length' traces-report.json)
|
||||
COVERED=$(jq '.coverage.covered' traces-report.json)
|
||||
TOTAL_REQS=$(jq '.coverage.total' traces-report.json)
|
||||
COVERAGE=$(jq '.coverage.percent' traces-report.json)
|
||||
|
||||
# Print coverage report
|
||||
echo "✅ TRACES Found: $TOTAL_TRACES"
|
||||
echo ""
|
||||
echo "📋 Coverage Summary:"
|
||||
echo " User Requirements (UR): $UR / 39 ($(( UR * 100 / 39 ))%)"
|
||||
echo " Integration Requirements (IR): $IR / 24 ($(( IR * 100 / 24 ))%)"
|
||||
echo " Development Requirements (DR): $DR / 48 ($(( DR * 100 / 48 ))%)"
|
||||
echo " Jellyfin API Requirements (JA): $JA / 3 ($(( JA * 100 / 3 ))%)"
|
||||
echo "📋 Coverage Summary (traced / defined):"
|
||||
for T in UR IR DR JA; do
|
||||
TRACED=$(jq --arg t "$T" '[.byType[$t][] | select(. != null)] | length' traces-report.json)
|
||||
DEFINED=$(jq --arg t "$T" '.defined[$t]' traces-report.json)
|
||||
echo " $T: $TRACED / $DEFINED"
|
||||
done
|
||||
echo ""
|
||||
|
||||
COVERED=$((UR + IR + DR + JA))
|
||||
TOTAL_REQS=114
|
||||
COVERAGE=$((COVERED * 100 / TOTAL_REQS))
|
||||
|
||||
echo "📈 Overall Coverage: $COVERED / $TOTAL_REQS ($COVERAGE%)"
|
||||
echo ""
|
||||
|
||||
# Traced IDs that requirements.md does not define (typo, or a deleted
|
||||
# requirement). These do not count toward coverage.
|
||||
ORPHANED=$(jq -c '.coverage.orphaned' traces-report.json)
|
||||
if [ "$ORPHANED" != "[]" ]; then
|
||||
echo "⚠️ Traced but not defined in requirements.md: $ORPHANED"
|
||||
echo ""
|
||||
fi
|
||||
|
||||
# A ratio above 100% means the computation is broken — the exact
|
||||
# condition that hid the stale-denominator bug. Fail loudly.
|
||||
if [ "$COVERAGE" -gt 100 ]; then
|
||||
echo "❌ ERROR: Coverage ($COVERAGE%) exceeds 100% — the gate is miscomputing."
|
||||
echo " Orphaned IDs: $ORPHANED"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check minimum threshold
|
||||
MIN_THRESHOLD=50
|
||||
if [ "$COVERAGE" -lt "$MIN_THRESHOLD" ]; then
|
||||
|
||||
Reference in New Issue
Block a user