jellytau/docs/specs/traceability-gate-repair.md

Spec: Repair the traceability coverage gate

Status: Implemented Requirements: DR-093 → supports the traceability practice described in CLAUDE.md UX spec: n/a — developer tooling, no user-facing surface. Supersedes / revises: n/a

Summary

The CI traceability gate has been passing unconditionally for an unknown length of time because it divides traced-requirement counts by hardcoded denominators that no longer match requirements.md. It currently reports 158% overall coverage (and JA 24 / 3 = 800%), so the 50% threshold is mathematically unreachable and the job cannot fail. This spec makes the gate derive its denominators from requirements.md at run time, so it reports the real number (85% today) and can actually fail again.

Motivation

.gitea/workflows/traceability-check.yml hardcodes UR/39, IR/24, DR/48, JA/3 and TOTAL_REQS=114. The real counts are UR 61, IR 29, DR 89, JA 32 — 211 total. Requirements were added over time; the divisors were never updated.

The consequence is not a cosmetic reporting bug. The gate is the only automated defence for the traceability practice, and it is dead:

CI today:  181 / 114 = 158%  → threshold 50% can never trip
Reality:   181 / 211 =  85%  → healthy, but unguarded

Coverage could collapse to 30% and CI would still print a green "✅ Coverage is acceptable". An audit of the design principles found that every principle with a working automated check is in good shape, and the ones that drifted are exactly the ones whose checks were broken or too narrow — this is the clearest instance.

A second, related defect is handled in a sibling spec: scripts/check-req-coverage.sh is separately broken and orphaned (see req-coverage-script-removal.md).

Layer assignment

This spec touches only CI/build tooling — no application logic crosses the Rust/Svelte boundary. The table is filled in for completeness.

Logic / responsibility	Layer	Why it belongs there
Counting requirement IDs defined in requirements.md	Build tooling (scripts/)	Neither runtime layer; it is repo metadata analysis. Belongs beside extract-traces.ts, not in the workflow YAML, so it is runnable and testable locally.
Counting traced requirement IDs	Build tooling — existing extract-traces.ts	Already implemented and correct; this spec consumes it rather than duplicating it.
Threshold policy (the 50% number)	CI workflow	Deployment policy, not analysis. Keeping it in YAML lets it be tuned without touching the script.

No frontend or Rust logic is added, so no taxonomy leak is possible.

Design

1. Denominators come from requirements.md, not literals

requirements.md defines requirements in markdown tables with a stable leading cell, e.g.:

| DR-001 | Player state machine (idle, loading, …) | Player | UR-005 | Done |
| UR-002 | Access media when online or offline | High | Done |

Extend scripts/extract-traces.ts to also emit the defined counts, so one tool owns both sides of the fraction and CI does no arithmetic on stale literals. Add a defined key to the JSON report:

{
  "byType":   { "UR": [...], "IR": [...], "DR": [...], "JA": [...] },  // traced (existing)
  "defined":  { "UR": 61, "IR": 29, "DR": 89, "JA": 32 },              // NEW
  "coverage": { "covered": 181, "total": 211, "percent": 85 },         // NEW
  "requirements": { ... },                                             // existing
  "totalTraces": 318, "totalFiles": …, "timestamp": "…"                // existing
}

Parsing rule for a defined requirement: a line in docs/requirements.md matching ^\|\s*(UR|IR|DR|JA)-\d{3}\s*\| — the ID must be the table's first cell. This deliberately does not count IDs mentioned in the Traces To column or in prose, which is why a naive grep -o over the whole file overcounts.

defined counts IDs that exist in the spec; byType counts IDs that appear in a TRACES: comment somewhere in the source. Coverage is |byType ∩ defined| / |defined|.

Intersection, not raw length. A TRACES: comment naming an ID that requirements.md does not define (a typo, or a requirement later deleted) must not inflate the numerator — that is how a ratio exceeds 100% in the first place. Such IDs are reported separately as orphaned so they get fixed rather than silently counted or silently dropped.

"orphaned": ["DR-097"]   // traced in code but not defined in requirements.md

2. The workflow consumes the computed number

Replace the arithmetic in .gitea/workflows/traceability-check.yml (lines 46–76) with reads of the precomputed fields:

COVERAGE=$(jq '.coverage.percent' traces-report.json)
COVERED=$(jq '.coverage.covered' traces-report.json)
TOTAL_REQS=$(jq '.coverage.total'   traces-report.json)

for T in UR IR DR JA; do
  TRACED=$(jq --arg t "$T" '.byType[$t] | length' traces-report.json)
  DEFINED=$(jq --arg t "$T" '.defined[$t]'        traces-report.json)
  echo "   $T: $TRACED / $DEFINED"
done

MIN_THRESHOLD=50
[ "$COVERAGE" -lt "$MIN_THRESHOLD" ] && { echo "❌ …"; exit 1; }

No hardcoded denominator survives anywhere in the workflow.

3. A self-check so this cannot silently rot again

The root cause was a number that drifted with nothing watching it. Add a guard that fails the job on an arithmetically impossible result:

if [ "$COVERAGE" -gt 100 ]; then
  echo "❌ Coverage > 100% — the gate is miscomputing; orphaned IDs: $(jq -c '.orphaned' traces-report.json)"
  exit 1
fi

A >100% reading is now a hard failure rather than a green tick.

4. Local parity

Add a script so the gate is runnable outside CI:

"traces:coverage": "bun run scripts/extract-traces.ts --format coverage"

Prints the same table CI prints and exits non-zero below threshold.

Threshold

Keep MIN_THRESHOLD=50 in this spec. Real coverage is 85%, so raising the bar is tempting, but doing it in the same change that repairs the gate conflates "restore the safety net" with "tighten the policy" — if the build then fails, it is ambiguous which change caused it. Ratcheting is deliberately deferred to follow-up work once the honest number has been observed on master for a few builds.

Out of scope

• Raising MIN_THRESHOLD above 50 (see above).
• Fixing/removing scripts/check-req-coverage.sh — req-coverage-script-removal.md.
• Adding TRACES comments to raise the actual coverage number.
• Changing the TRACES: comment format or the extractor's parsing of it.
• The PR "modified files missing TRACES" step (lines 78–126), which is advisory by design and stays advisory.

Acceptance criteria

• bun run traces:json emits defined, coverage, and orphaned keys.
• coverage.total equals the count of requirement IDs defined in requirements.md (211 at time of writing), not a literal.
• coverage.percent reports 85 (±1 for rounding) on the current tree — i.e. the honest number, not 158.
• No hardcoded requirement denominator (39, 24, 48, 3, 114) remains in .gitea/workflows/traceability-check.yml. Verify: grep -nE '/ *(39|24|48|3|114)\b' .gitea/workflows/traceability-check.yml returns nothing.
• Adding a new requirement row to requirements.md lowers reported coverage until it is traced (proves the denominator is live).
• A TRACES: comment naming an undefined ID appears in orphaned and does not raise coverage.percent.
• The job fails if coverage is forced below 50% (test by temporarily raising MIN_THRESHOLD to 99 locally) — proving the gate can fail again.
• The job fails if coverage computes >100%.
• bun run check and bun run test pass.
• bun run check:boundary passes.
• New requirement-implementing code carries // TRACES: comments.
• No Rust types changed, so no bindings.ts regeneration needed.

Testing

extract-traces.ts currently has no test coverage. Add scripts/extract-traces.test.ts (vitest) over fixture strings rather than the live requirements.md, so the tests do not change meaning as requirements are added:

• UT: counts a well-formed table row as a defined requirement.
• UT: does not count an ID appearing only in the Traces To column or in prose — the specific overcounting bug this parse rule avoids.
• UT: coverage is the intersection — a traced-but-undefined ID lands in orphaned and does not inflate the numerator.
• UT: coverage of an empty trace set is 0%, not a divide-by-zero.
• UT: all-traced fixture reports exactly 100%, never above.

CI behaviour is verified by the acceptance criteria above (the forced-failure check is the important one — a gate nobody has watched fail is not known to work).

TRACES

Allocate in requirements.md:

• DR-093 — "Traceability coverage gate derives requirement denominators from requirements.md at run time (not hardcoded literals), computes coverage as the intersection of traced and defined IDs, reports IDs traced but undefined as orphaned, and fails on an impossible >100% result." Category: Tooling. Status: Done on merge.

Tag:

// scripts/extract-traces.ts
// TRACES: | DR-093

Tests carry @req-test: UT-089 … onward (next free UT is UT-089).

Notes for the implementer

• A parallel Claude session may be active in this repo — run git diff before "repairing" unexpected changes (CLAUDE.md §Gotchas).
• Do not add tooling to the CI image for this. jq and bun are already in jellytau-builder; this spec needs nothing else. Installing a system package in a workflow step violates the hard CI rule in CLAUDE.md.
• Keep traces:json's existing keys intact — release-notes.ts and traces:markdown consume the same report, and the CI workflow uploads it as an artifact. This is an additive change.
• The head -50 docs/traceability.md and artifact-upload steps are unaffected.
• Expect the first green build after this change to print a lower number than before (85% vs 158%). That is the fix working, not a regression.