The coverage gate divided traced counts by hardcoded literals (UR/39, IR/24, DR/48, JA/3, TOTAL_REQS=114) that had fallen out of date as requirements grew to 211. It reported 158% coverage — JA alone printed 800% — so the 50% threshold was mathematically unreachable and the job could not fail. Coverage could have collapsed to 30% and CI would still have printed a green tick. Real coverage is 86%. The number was fine; the gate was dead. extract-traces.ts now owns both sides of the fraction: - countDefinedRequirements() counts an ID only where it leads a markdown table row, ignoring the "Traces To" column and prose. IDs are deduplicated because requirements.md lists every UR twice (§1 definition + §3 matrix), which would otherwise report UR as 121/61. - computeCoverage() uses the intersection of traced and defined IDs, so a TRACES comment naming a deleted or typo'd requirement is reported as `orphaned` rather than inflating the ratio past 100%. UT/IT test identifiers are excluded as a separate taxonomy. - CI reads .coverage.percent and fails on <50% or >100%; a >100% reading is now a hard error rather than the condition that hid this bug. - New `bun run traces:coverage` runs the same computation locally. - scripts/ added to the scan roots — the coverage tool was invisible to the matrix it generates. Tests written first (15, over fixtures so they don't drift as requirements are added). vitest include widened to scripts/** so build tooling is covered by the normal suite. Verified empirically rather than by inspection: forcing the threshold to 99% fails; adding a requirement lowers coverage 86%→85%; a TRACES: DR-999 lands in `orphaned` without changing `covered`. traceability-ci.md documented the same stale numbers and would have let the broken arithmetic be reconstructed — replaced with a pointer to the live command.
167 lines
5.5 KiB
YAML
167 lines
5.5 KiB
YAML
name: Traceability Validation
|
|
|
|
on:
|
|
push:
|
|
branches:
|
|
- master
|
|
- main
|
|
- develop
|
|
pull_request:
|
|
branches:
|
|
- master
|
|
- main
|
|
- develop
|
|
|
|
jobs:
|
|
validate-traces:
|
|
runs-on: linux/amd64
|
|
name: Check Requirement Traces
|
|
container:
|
|
image: gitea.tourolle.paris/dtourolle/jellytau-builder:latest
|
|
|
|
steps:
|
|
- name: Checkout repository
|
|
uses: actions/checkout@v4
|
|
with:
|
|
fetch-depth: 0
|
|
|
|
# bun is baked into jellytau-builder (see Dockerfile.builder); no setup-bun
|
|
# action needed — fetching it stalls on this Gitea runner.
|
|
- name: Install dependencies
|
|
run: bun install
|
|
|
|
- name: Extract traces
|
|
run: |
|
|
echo "🔍 Extracting requirement traces..."
|
|
bun run traces:json > traces-report.json
|
|
|
|
- name: Validate traces
|
|
run: |
|
|
set -e
|
|
|
|
echo "📊 Validating requirement traceability..."
|
|
echo ""
|
|
|
|
# Denominators come from docs/requirements.md at run time — NEVER
|
|
# hardcode them here. This step previously divided by frozen literals
|
|
# (UR/39, IR/24, DR/48, JA/3, total 114) while the file had grown to
|
|
# 211 requirements, so it reported 158% coverage and the threshold
|
|
# below could never trip. See docs/specs/traceability-gate-repair.md.
|
|
TOTAL_TRACES=$(jq '.totalTraces' traces-report.json)
|
|
COVERED=$(jq '.coverage.covered' traces-report.json)
|
|
TOTAL_REQS=$(jq '.coverage.total' traces-report.json)
|
|
COVERAGE=$(jq '.coverage.percent' traces-report.json)
|
|
|
|
echo "✅ TRACES Found: $TOTAL_TRACES"
|
|
echo ""
|
|
echo "📋 Coverage Summary (traced / defined):"
|
|
for T in UR IR DR JA; do
|
|
TRACED=$(jq --arg t "$T" '[.byType[$t][] | select(. != null)] | length' traces-report.json)
|
|
DEFINED=$(jq --arg t "$T" '.defined[$t]' traces-report.json)
|
|
echo " $T: $TRACED / $DEFINED"
|
|
done
|
|
echo ""
|
|
|
|
echo "📈 Overall Coverage: $COVERED / $TOTAL_REQS ($COVERAGE%)"
|
|
echo ""
|
|
|
|
# Traced IDs that requirements.md does not define (typo, or a deleted
|
|
# requirement). These do not count toward coverage.
|
|
ORPHANED=$(jq -c '.coverage.orphaned' traces-report.json)
|
|
if [ "$ORPHANED" != "[]" ]; then
|
|
echo "⚠️ Traced but not defined in requirements.md: $ORPHANED"
|
|
echo ""
|
|
fi
|
|
|
|
# A ratio above 100% means the computation is broken — the exact
|
|
# condition that hid the stale-denominator bug. Fail loudly.
|
|
if [ "$COVERAGE" -gt 100 ]; then
|
|
echo "❌ ERROR: Coverage ($COVERAGE%) exceeds 100% — the gate is miscomputing."
|
|
echo " Orphaned IDs: $ORPHANED"
|
|
exit 1
|
|
fi
|
|
|
|
# Check minimum threshold
|
|
MIN_THRESHOLD=50
|
|
if [ "$COVERAGE" -lt "$MIN_THRESHOLD" ]; then
|
|
echo "❌ ERROR: Coverage ($COVERAGE%) is below minimum threshold ($MIN_THRESHOLD%)"
|
|
exit 1
|
|
fi
|
|
|
|
echo "✅ Coverage is acceptable ($COVERAGE% >= $MIN_THRESHOLD%)"
|
|
|
|
- name: Check modified files
|
|
if: github.event_name == 'pull_request'
|
|
run: |
|
|
echo "🔍 Checking modified files for traces..."
|
|
echo ""
|
|
|
|
# Get changed files
|
|
CHANGED=$(git diff --name-only origin/${{ github.base_ref }}...HEAD | grep -E '\.(ts|tsx|svelte|rs)$' || echo "")
|
|
|
|
if [ -z "$CHANGED" ]; then
|
|
echo "✅ No TypeScript/Rust files changed"
|
|
exit 0
|
|
fi
|
|
|
|
echo "📝 Changed files:"
|
|
echo "$CHANGED" | sed 's/^/ /'
|
|
echo ""
|
|
|
|
# Check each file
|
|
# Pipe into the loop instead of a here-string (<<<) so this step works
|
|
# under POSIX sh/dash, not just bash. Use `case` instead of `[[ == ]]`
|
|
# for the same reason. The loop runs in a subshell (so a counter var
|
|
# wouldn't survive), so we record warnings in a temp file and count it
|
|
# afterwards.
|
|
MISSING_FILE=$(mktemp)
|
|
echo "$CHANGED" | while IFS= read -r file; do
|
|
# Skip test files
|
|
case "$file" in
|
|
*.test.*) continue ;;
|
|
esac
|
|
|
|
if [ -f "$file" ]; then
|
|
if ! grep -q "TRACES:" "$file"; then
|
|
echo "⚠️ Missing TRACES: $file"
|
|
echo "$file" >> "$MISSING_FILE"
|
|
fi
|
|
fi
|
|
done
|
|
|
|
MISSING_TRACES=$(wc -l < "$MISSING_FILE" | tr -d ' ')
|
|
rm -f "$MISSING_FILE"
|
|
|
|
if [ "$MISSING_TRACES" -gt 0 ]; then
|
|
echo ""
|
|
echo "📝 Recommendation: Add TRACES comments to new/modified code"
|
|
echo " Format: // TRACES: UR-001, UR-002 | DR-003"
|
|
echo ""
|
|
echo "💡 For more info, see: scripts/README.md"
|
|
fi
|
|
|
|
- name: Generate full report
|
|
if: always()
|
|
run: |
|
|
echo "📄 Generating full traceability report..."
|
|
bun run traces:markdown
|
|
|
|
- name: Display report summary
|
|
if: always()
|
|
run: |
|
|
echo ""
|
|
echo "📊 Full Report Generated"
|
|
echo "📁 Location: docs/traceability.md"
|
|
echo ""
|
|
head -50 docs/traceability.md || true
|
|
|
|
- name: Save artifacts
|
|
if: always()
|
|
uses: actions/upload-artifact@v3
|
|
with:
|
|
name: traceability-reports
|
|
path: |
|
|
traces-report.json
|
|
docs/traceability.md
|
|
retention-days: 30
|