Files
jellytau/.gitea/workflows/traceability-check.yml
T
dtourolle 0da0a9f16c fix(ci): derive traceability denominators from requirements.md (DR-093)
The coverage gate divided traced counts by hardcoded literals (UR/39,
IR/24, DR/48, JA/3, TOTAL_REQS=114) that had fallen out of date as
requirements grew to 211. It reported 158% coverage — JA alone printed
800% — so the 50% threshold was mathematically unreachable and the job
could not fail. Coverage could have collapsed to 30% and CI would still
have printed a green tick.

Real coverage is 86%. The number was fine; the gate was dead.

extract-traces.ts now owns both sides of the fraction:

- countDefinedRequirements() counts an ID only where it leads a markdown
  table row, ignoring the "Traces To" column and prose. IDs are
  deduplicated because requirements.md lists every UR twice (§1
  definition + §3 matrix), which would otherwise report UR as 121/61.
- computeCoverage() uses the intersection of traced and defined IDs, so
  a TRACES comment naming a deleted or typo'd requirement is reported as
  `orphaned` rather than inflating the ratio past 100%. UT/IT test
  identifiers are excluded as a separate taxonomy.
- CI reads .coverage.percent and fails on <50% or >100%; a >100% reading
  is now a hard error rather than the condition that hid this bug.
- New `bun run traces:coverage` runs the same computation locally.
- scripts/ added to the scan roots — the coverage tool was invisible to
  the matrix it generates.

Tests written first (15, over fixtures so they don't drift as
requirements are added). vitest include widened to scripts/** so build
tooling is covered by the normal suite.

Verified empirically rather than by inspection: forcing the threshold to
99% fails; adding a requirement lowers coverage 86%→85%; a TRACES: DR-999
lands in `orphaned` without changing `covered`.

traceability-ci.md documented the same stale numbers and would have let
the broken arithmetic be reconstructed — replaced with a pointer to the
live command.
2026-07-30 10:30:08 +02:00

167 lines
5.5 KiB
YAML

name: Traceability Validation
on:
push:
branches:
- master
- main
- develop
pull_request:
branches:
- master
- main
- develop
jobs:
validate-traces:
runs-on: linux/amd64
name: Check Requirement Traces
container:
image: gitea.tourolle.paris/dtourolle/jellytau-builder:latest
steps:
- name: Checkout repository
uses: actions/checkout@v4
with:
fetch-depth: 0
# bun is baked into jellytau-builder (see Dockerfile.builder); no setup-bun
# action needed — fetching it stalls on this Gitea runner.
- name: Install dependencies
run: bun install
- name: Extract traces
run: |
echo "🔍 Extracting requirement traces..."
bun run traces:json > traces-report.json
- name: Validate traces
run: |
set -e
echo "📊 Validating requirement traceability..."
echo ""
# Denominators come from docs/requirements.md at run time — NEVER
# hardcode them here. This step previously divided by frozen literals
# (UR/39, IR/24, DR/48, JA/3, total 114) while the file had grown to
# 211 requirements, so it reported 158% coverage and the threshold
# below could never trip. See docs/specs/traceability-gate-repair.md.
TOTAL_TRACES=$(jq '.totalTraces' traces-report.json)
COVERED=$(jq '.coverage.covered' traces-report.json)
TOTAL_REQS=$(jq '.coverage.total' traces-report.json)
COVERAGE=$(jq '.coverage.percent' traces-report.json)
echo "✅ TRACES Found: $TOTAL_TRACES"
echo ""
echo "📋 Coverage Summary (traced / defined):"
for T in UR IR DR JA; do
TRACED=$(jq --arg t "$T" '[.byType[$t][] | select(. != null)] | length' traces-report.json)
DEFINED=$(jq --arg t "$T" '.defined[$t]' traces-report.json)
echo " $T: $TRACED / $DEFINED"
done
echo ""
echo "📈 Overall Coverage: $COVERED / $TOTAL_REQS ($COVERAGE%)"
echo ""
# Traced IDs that requirements.md does not define (typo, or a deleted
# requirement). These do not count toward coverage.
ORPHANED=$(jq -c '.coverage.orphaned' traces-report.json)
if [ "$ORPHANED" != "[]" ]; then
echo "⚠️ Traced but not defined in requirements.md: $ORPHANED"
echo ""
fi
# A ratio above 100% means the computation is broken — the exact
# condition that hid the stale-denominator bug. Fail loudly.
if [ "$COVERAGE" -gt 100 ]; then
echo "❌ ERROR: Coverage ($COVERAGE%) exceeds 100% — the gate is miscomputing."
echo " Orphaned IDs: $ORPHANED"
exit 1
fi
# Check minimum threshold
MIN_THRESHOLD=50
if [ "$COVERAGE" -lt "$MIN_THRESHOLD" ]; then
echo "❌ ERROR: Coverage ($COVERAGE%) is below minimum threshold ($MIN_THRESHOLD%)"
exit 1
fi
echo "✅ Coverage is acceptable ($COVERAGE% >= $MIN_THRESHOLD%)"
- name: Check modified files
if: github.event_name == 'pull_request'
run: |
echo "🔍 Checking modified files for traces..."
echo ""
# Get changed files
CHANGED=$(git diff --name-only origin/${{ github.base_ref }}...HEAD | grep -E '\.(ts|tsx|svelte|rs)$' || echo "")
if [ -z "$CHANGED" ]; then
echo "✅ No TypeScript/Rust files changed"
exit 0
fi
echo "📝 Changed files:"
echo "$CHANGED" | sed 's/^/ /'
echo ""
# Check each file
# Pipe into the loop instead of a here-string (<<<) so this step works
# under POSIX sh/dash, not just bash. Use `case` instead of `[[ == ]]`
# for the same reason. The loop runs in a subshell (so a counter var
# wouldn't survive), so we record warnings in a temp file and count it
# afterwards.
MISSING_FILE=$(mktemp)
echo "$CHANGED" | while IFS= read -r file; do
# Skip test files
case "$file" in
*.test.*) continue ;;
esac
if [ -f "$file" ]; then
if ! grep -q "TRACES:" "$file"; then
echo "⚠️ Missing TRACES: $file"
echo "$file" >> "$MISSING_FILE"
fi
fi
done
MISSING_TRACES=$(wc -l < "$MISSING_FILE" | tr -d ' ')
rm -f "$MISSING_FILE"
if [ "$MISSING_TRACES" -gt 0 ]; then
echo ""
echo "📝 Recommendation: Add TRACES comments to new/modified code"
echo " Format: // TRACES: UR-001, UR-002 | DR-003"
echo ""
echo "💡 For more info, see: scripts/README.md"
fi
- name: Generate full report
if: always()
run: |
echo "📄 Generating full traceability report..."
bun run traces:markdown
- name: Display report summary
if: always()
run: |
echo ""
echo "📊 Full Report Generated"
echo "📁 Location: docs/traceability.md"
echo ""
head -50 docs/traceability.md || true
- name: Save artifacts
if: always()
uses: actions/upload-artifact@v3
with:
name: traceability-reports
path: |
traces-report.json
docs/traceability.md
retention-days: 30