jellytau/docs/specs/traceability-gate-repair.md

# Spec: Repair the traceability coverage gate

**Status:** Implemented
**Requirements:** DR-093 → supports the traceability practice described in CLAUDE.md
**UX spec:** n/a — developer tooling, no user-facing surface.
**Supersedes / revises:** n/a

## Summary

The CI traceability gate has been passing unconditionally for an unknown length
of time because it divides traced-requirement counts by **hardcoded denominators
that no longer match [requirements.md](../requirements.md)**. It currently
reports **158% overall coverage** (and `JA 24 / 3 = 800%`), so the 50% threshold
is mathematically unreachable and the job cannot fail. This spec makes the gate
derive its denominators from `requirements.md` at run time, so it reports the
real number (**85%** today) and can actually fail again.

## Motivation

`.gitea/workflows/traceability-check.yml` hardcodes `UR/39, IR/24, DR/48, JA/3`
and `TOTAL_REQS=114`. The real counts are **UR 61, IR 29, DR 89, JA 32 — 211
total**. Requirements were added over time; the divisors were never updated.

The consequence is not a cosmetic reporting bug. The gate is the *only*
automated defence for the traceability practice, and it is dead:

```
CI today:  181 / 114 = 158%  → threshold 50% can never trip
Reality:   181 / 211 =  85%  → healthy, but unguarded
```

Coverage could collapse to 30% and CI would still print a green
"✅ Coverage is acceptable". An audit of the design principles found that every
principle with a *working* automated check is in good shape, and the ones that
drifted are exactly the ones whose checks were broken or too narrow — this is
the clearest instance.

A second, related defect is handled in a sibling spec: `scripts/check-req-coverage.sh`
is separately broken and orphaned (see
[req-coverage-script-removal.md](req-coverage-script-removal.md)).

## Layer assignment

This spec touches only CI/build tooling — no application logic crosses the
Rust/Svelte boundary. The table is filled in for completeness.

| Logic / responsibility | Layer | Why it belongs there |
|------------------------|-------|----------------------|
| Counting requirement IDs defined in `requirements.md` | Build tooling (`scripts/`) | Neither runtime layer; it is repo metadata analysis. Belongs beside `extract-traces.ts`, not in the workflow YAML, so it is runnable and testable locally. |
| Counting *traced* requirement IDs | Build tooling — existing `extract-traces.ts` | Already implemented and correct; this spec consumes it rather than duplicating it. |
| Threshold policy (the 50% number) | CI workflow | Deployment policy, not analysis. Keeping it in YAML lets it be tuned without touching the script. |

No frontend or Rust logic is added, so no taxonomy leak is possible.

## Design

### 1. Denominators come from `requirements.md`, not literals

`requirements.md` defines requirements in markdown tables with a stable leading
cell, e.g.:

```
| DR-001 | Player state machine (idle, loading, …) | Player | UR-005 | Done |
| UR-002 | Access media when online or offline | High | Done |
```

Extend [scripts/extract-traces.ts](../../scripts/extract-traces.ts) to also emit
the *defined* counts, so one tool owns both sides of the fraction and CI does no
arithmetic on stale literals. Add a `defined` key to the JSON report:

```jsonc
{
  "byType":   { "UR": [...], "IR": [...], "DR": [...], "JA": [...] },  // traced (existing)
  "defined":  { "UR": 61, "IR": 29, "DR": 89, "JA": 32 },              // NEW
  "coverage": { "covered": 181, "total": 211, "percent": 85 },         // NEW
  "requirements": { ... },                                             // existing
  "totalTraces": 318, "totalFiles": …, "timestamp": "…"                // existing
}
```

Parsing rule for a *defined* requirement: a line in `docs/requirements.md`
matching `^\|\s*(UR|IR|DR|JA)-\d{3}\s*\|` — the ID must be the table's first
cell. This deliberately does **not** count IDs mentioned in the `Traces To`
column or in prose, which is why a naive `grep -o` over the whole file
overcounts.

`defined` counts IDs that exist in the spec; `byType` counts IDs that appear in
a `TRACES:` comment somewhere in the source. Coverage is
`|byType ∩ defined| / |defined|`.

> **Intersection, not raw length.** A `TRACES:` comment naming an ID that
> `requirements.md` does not define (a typo, or a requirement later deleted)
> must **not** inflate the numerator — that is how a ratio exceeds 100% in the
> first place. Such IDs are reported separately as `orphaned` so they get fixed
> rather than silently counted or silently dropped.

```jsonc
"orphaned": ["DR-097"]   // traced in code but not defined in requirements.md
```

### 2. The workflow consumes the computed number

Replace the arithmetic in `.gitea/workflows/traceability-check.yml` (lines
46–76) with reads of the precomputed fields:

```sh
COVERAGE=$(jq '.coverage.percent' traces-report.json)
COVERED=$(jq '.coverage.covered' traces-report.json)
TOTAL_REQS=$(jq '.coverage.total'   traces-report.json)

for T in UR IR DR JA; do
  TRACED=$(jq --arg t "$T" '.byType[$t] | length' traces-report.json)
  DEFINED=$(jq --arg t "$T" '.defined[$t]'        traces-report.json)
  echo "   $T: $TRACED / $DEFINED"
done

MIN_THRESHOLD=50
[ "$COVERAGE" -lt "$MIN_THRESHOLD" ] && { echo "❌ …"; exit 1; }
```

No hardcoded denominator survives anywhere in the workflow.

### 3. A self-check so this cannot silently rot again

The root cause was a number that drifted with nothing watching it. Add a
guard that fails the job on an arithmetically impossible result:

```sh
if [ "$COVERAGE" -gt 100 ]; then
  echo "❌ Coverage > 100% — the gate is miscomputing; orphaned IDs: $(jq -c '.orphaned' traces-report.json)"
  exit 1
fi
```

A >100% reading is now a hard failure rather than a green tick.

### 4. Local parity

Add a script so the gate is runnable outside CI:

```jsonc
"traces:coverage": "bun run scripts/extract-traces.ts --format coverage"
```

Prints the same table CI prints and exits non-zero below threshold.

### Threshold

Keep `MIN_THRESHOLD=50` in this spec. Real coverage is 85%, so raising the bar
is tempting, but doing it in the same change that repairs the gate conflates
"restore the safety net" with "tighten the policy" — if the build then fails, it
is ambiguous which change caused it. Ratcheting is deliberately deferred to
follow-up work once the honest number has been observed on `master` for a few
builds.

## Out of scope

- Raising `MIN_THRESHOLD` above 50 (see above).
- Fixing/removing `scripts/check-req-coverage.sh` — [req-coverage-script-removal.md](req-coverage-script-removal.md).
- Adding TRACES comments to raise the actual coverage number.
- Changing the `TRACES:` comment format or the extractor's parsing of it.
- The PR "modified files missing TRACES" step (lines 78–126), which is advisory
  by design and stays advisory.

## Acceptance criteria

- [ ] `bun run traces:json` emits `defined`, `coverage`, and `orphaned` keys.
- [ ] `coverage.total` equals the count of requirement IDs defined in
      `requirements.md` (**211** at time of writing), not a literal.
- [ ] `coverage.percent` reports **85** (±1 for rounding) on the current tree —
      i.e. the honest number, not 158.
- [ ] No hardcoded requirement denominator (`39`, `24`, `48`, `3`, `114`) remains
      in `.gitea/workflows/traceability-check.yml`. Verify:
      `grep -nE '/ *(39|24|48|3|114)\b' .gitea/workflows/traceability-check.yml`
      returns nothing.
- [ ] Adding a new requirement row to `requirements.md` **lowers** reported
      coverage until it is traced (proves the denominator is live).
- [ ] A `TRACES:` comment naming an undefined ID appears in `orphaned` and does
      **not** raise `coverage.percent`.
- [ ] The job fails if coverage is forced below 50% (test by temporarily raising
      `MIN_THRESHOLD` to 99 locally) — proving the gate can fail again.
- [ ] The job fails if coverage computes >100%.
- [ ] `bun run check` and `bun run test` pass.
- [ ] `bun run check:boundary` passes.
- [ ] New requirement-implementing code carries `// TRACES:` comments.
- [ ] No Rust types changed, so no `bindings.ts` regeneration needed.

## Testing

`extract-traces.ts` currently has no test coverage. Add
`scripts/extract-traces.test.ts` (vitest) over fixture strings rather than the
live `requirements.md`, so the tests do not change meaning as requirements are
added:

- **UT:** counts a well-formed table row as a defined requirement.
- **UT:** does **not** count an ID appearing only in the `Traces To` column or
  in prose — the specific overcounting bug this parse rule avoids.
- **UT:** coverage is the intersection — a traced-but-undefined ID lands in
  `orphaned` and does not inflate the numerator.
- **UT:** coverage of an empty trace set is 0%, not a divide-by-zero.
- **UT:** all-traced fixture reports exactly 100%, never above.

CI behaviour is verified by the acceptance criteria above (the forced-failure
check is the important one — a gate nobody has watched fail is not known to
work).

## TRACES

Allocate in `requirements.md`:

- **DR-093** — "Traceability coverage gate derives requirement denominators from
  `requirements.md` at run time (not hardcoded literals), computes coverage as
  the intersection of traced and defined IDs, reports IDs traced but undefined
  as orphaned, and fails on an impossible >100% result." Category: Tooling.
  Status: Done on merge.

Tag:

```typescript
// scripts/extract-traces.ts
// TRACES: | DR-093
```

Tests carry `@req-test: UT-089 …` onward (next free UT is **UT-089**).

## Notes for the implementer

- A parallel Claude session may be active in this repo — run `git diff` before
  "repairing" unexpected changes (CLAUDE.md §Gotchas).
- **Do not add tooling to the CI image for this.** `jq` and `bun` are already in
  `jellytau-builder`; this spec needs nothing else. Installing a system package
  in a workflow step violates the hard CI rule in CLAUDE.md.
- Keep `traces:json`'s existing keys intact — `release-notes.ts` and
  `traces:markdown` consume the same report, and the CI workflow uploads it as
  an artifact. This is an additive change.
- The `head -50 docs/traceability.md` and artifact-upload steps are unaffected.
- Expect the first green build after this change to print a *lower* number than
  before (85% vs 158%). That is the fix working, not a regression.