docs(specs): design-principles audit — five remediation specs
Audit of the principles in CLAUDE.md and docs/architecture/ against the actual code. Principles with a working automated check (poison-tolerant locking, Android source sync, one-directional playback state, graceful backend init, reachability-from-traffic) all held up. The two that drifted are exactly the two whose checks were broken or too narrow: - traceability-gate-repair: CI divided by hardcoded denominators (UR/39, IR/24, DR/48, JA/3, total 114) while requirements.md had grown to 211, reporting 158% coverage — the 50% threshold was unreachable and the job could not fail. - req-coverage-script-removal: check-req-coverage.sh reports "1 requirement" and prints "all requirements have implementations". - scoped-search-boundary-implementation: the founding boundary incident was specced but never built; the leak is still live. - boundary-tripwire-hardening: check:boundary passes on that same leak — the pattern is anchored to the query site, so a named const evades it. - player-facade-enforcement: 52 direct commands.player* call sites outside the facade, and no automated check at all. Each spec follows SPEC-TEMPLATE.md with a filled-in Layer assignment table and is checked against SPEC-REVIEW-CHECKLIST.md.
This commit is contained in:
@@ -0,0 +1,238 @@
|
||||
# Spec: Repair the traceability coverage gate
|
||||
|
||||
**Status:** Implemented
|
||||
**Requirements:** DR-093 → supports the traceability practice described in CLAUDE.md
|
||||
**UX spec:** n/a — developer tooling, no user-facing surface.
|
||||
**Supersedes / revises:** n/a
|
||||
|
||||
## Summary
|
||||
|
||||
The CI traceability gate has been passing unconditionally for an unknown length
|
||||
of time because it divides traced-requirement counts by **hardcoded denominators
|
||||
that no longer match [requirements.md](../requirements.md)**. It currently
|
||||
reports **158% overall coverage** (and `JA 24 / 3 = 800%`), so the 50% threshold
|
||||
is mathematically unreachable and the job cannot fail. This spec makes the gate
|
||||
derive its denominators from `requirements.md` at run time, so it reports the
|
||||
real number (**85%** today) and can actually fail again.
|
||||
|
||||
## Motivation
|
||||
|
||||
`.gitea/workflows/traceability-check.yml` hardcodes `UR/39, IR/24, DR/48, JA/3`
|
||||
and `TOTAL_REQS=114`. The real counts are **UR 61, IR 29, DR 89, JA 32 — 211
|
||||
total**. Requirements were added over time; the divisors were never updated.
|
||||
|
||||
The consequence is not a cosmetic reporting bug. The gate is the *only*
|
||||
automated defence for the traceability practice, and it is dead:
|
||||
|
||||
```
|
||||
CI today: 181 / 114 = 158% → threshold 50% can never trip
|
||||
Reality: 181 / 211 = 85% → healthy, but unguarded
|
||||
```
|
||||
|
||||
Coverage could collapse to 30% and CI would still print a green
|
||||
"✅ Coverage is acceptable". An audit of the design principles found that every
|
||||
principle with a *working* automated check is in good shape, and the ones that
|
||||
drifted are exactly the ones whose checks were broken or too narrow — this is
|
||||
the clearest instance.
|
||||
|
||||
A second, related defect is handled in a sibling spec: `scripts/check-req-coverage.sh`
|
||||
is separately broken and orphaned (see
|
||||
[req-coverage-script-removal.md](req-coverage-script-removal.md)).
|
||||
|
||||
## Layer assignment
|
||||
|
||||
This spec touches only CI/build tooling — no application logic crosses the
|
||||
Rust/Svelte boundary. The table is filled in for completeness.
|
||||
|
||||
| Logic / responsibility | Layer | Why it belongs there |
|
||||
|------------------------|-------|----------------------|
|
||||
| Counting requirement IDs defined in `requirements.md` | Build tooling (`scripts/`) | Neither runtime layer; it is repo metadata analysis. Belongs beside `extract-traces.ts`, not in the workflow YAML, so it is runnable and testable locally. |
|
||||
| Counting *traced* requirement IDs | Build tooling — existing `extract-traces.ts` | Already implemented and correct; this spec consumes it rather than duplicating it. |
|
||||
| Threshold policy (the 50% number) | CI workflow | Deployment policy, not analysis. Keeping it in YAML lets it be tuned without touching the script. |
|
||||
|
||||
No frontend or Rust logic is added, so no taxonomy leak is possible.
|
||||
|
||||
## Design
|
||||
|
||||
### 1. Denominators come from `requirements.md`, not literals
|
||||
|
||||
`requirements.md` defines requirements in markdown tables with a stable leading
|
||||
cell, e.g.:
|
||||
|
||||
```
|
||||
| DR-001 | Player state machine (idle, loading, …) | Player | UR-005 | Done |
|
||||
| UR-002 | Access media when online or offline | High | Done |
|
||||
```
|
||||
|
||||
Extend [scripts/extract-traces.ts](../../scripts/extract-traces.ts) to also emit
|
||||
the *defined* counts, so one tool owns both sides of the fraction and CI does no
|
||||
arithmetic on stale literals. Add a `defined` key to the JSON report:
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"byType": { "UR": [...], "IR": [...], "DR": [...], "JA": [...] }, // traced (existing)
|
||||
"defined": { "UR": 61, "IR": 29, "DR": 89, "JA": 32 }, // NEW
|
||||
"coverage": { "covered": 181, "total": 211, "percent": 85 }, // NEW
|
||||
"requirements": { ... }, // existing
|
||||
"totalTraces": 318, "totalFiles": …, "timestamp": "…" // existing
|
||||
}
|
||||
```
|
||||
|
||||
Parsing rule for a *defined* requirement: a line in `docs/requirements.md`
|
||||
matching `^\|\s*(UR|IR|DR|JA)-\d{3}\s*\|` — the ID must be the table's first
|
||||
cell. This deliberately does **not** count IDs mentioned in the `Traces To`
|
||||
column or in prose, which is why a naive `grep -o` over the whole file
|
||||
overcounts.
|
||||
|
||||
`defined` counts IDs that exist in the spec; `byType` counts IDs that appear in
|
||||
a `TRACES:` comment somewhere in the source. Coverage is
|
||||
`|byType ∩ defined| / |defined|`.
|
||||
|
||||
> **Intersection, not raw length.** A `TRACES:` comment naming an ID that
|
||||
> `requirements.md` does not define (a typo, or a requirement later deleted)
|
||||
> must **not** inflate the numerator — that is how a ratio exceeds 100% in the
|
||||
> first place. Such IDs are reported separately as `orphaned` so they get fixed
|
||||
> rather than silently counted or silently dropped.
|
||||
|
||||
```jsonc
|
||||
"orphaned": ["DR-097"] // traced in code but not defined in requirements.md
|
||||
```
|
||||
|
||||
### 2. The workflow consumes the computed number
|
||||
|
||||
Replace the arithmetic in `.gitea/workflows/traceability-check.yml` (lines
|
||||
46–76) with reads of the precomputed fields:
|
||||
|
||||
```sh
|
||||
COVERAGE=$(jq '.coverage.percent' traces-report.json)
|
||||
COVERED=$(jq '.coverage.covered' traces-report.json)
|
||||
TOTAL_REQS=$(jq '.coverage.total' traces-report.json)
|
||||
|
||||
for T in UR IR DR JA; do
|
||||
TRACED=$(jq --arg t "$T" '.byType[$t] | length' traces-report.json)
|
||||
DEFINED=$(jq --arg t "$T" '.defined[$t]' traces-report.json)
|
||||
echo " $T: $TRACED / $DEFINED"
|
||||
done
|
||||
|
||||
MIN_THRESHOLD=50
|
||||
[ "$COVERAGE" -lt "$MIN_THRESHOLD" ] && { echo "❌ …"; exit 1; }
|
||||
```
|
||||
|
||||
No hardcoded denominator survives anywhere in the workflow.
|
||||
|
||||
### 3. A self-check so this cannot silently rot again
|
||||
|
||||
The root cause was a number that drifted with nothing watching it. Add a
|
||||
guard that fails the job on an arithmetically impossible result:
|
||||
|
||||
```sh
|
||||
if [ "$COVERAGE" -gt 100 ]; then
|
||||
echo "❌ Coverage > 100% — the gate is miscomputing; orphaned IDs: $(jq -c '.orphaned' traces-report.json)"
|
||||
exit 1
|
||||
fi
|
||||
```
|
||||
|
||||
A >100% reading is now a hard failure rather than a green tick.
|
||||
|
||||
### 4. Local parity
|
||||
|
||||
Add a script so the gate is runnable outside CI:
|
||||
|
||||
```jsonc
|
||||
"traces:coverage": "bun run scripts/extract-traces.ts --format coverage"
|
||||
```
|
||||
|
||||
Prints the same table CI prints and exits non-zero below threshold.
|
||||
|
||||
### Threshold
|
||||
|
||||
Keep `MIN_THRESHOLD=50` in this spec. Real coverage is 85%, so raising the bar
|
||||
is tempting, but doing it in the same change that repairs the gate conflates
|
||||
"restore the safety net" with "tighten the policy" — if the build then fails, it
|
||||
is ambiguous which change caused it. Ratcheting is deliberately deferred to
|
||||
follow-up work once the honest number has been observed on `master` for a few
|
||||
builds.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- Raising `MIN_THRESHOLD` above 50 (see above).
|
||||
- Fixing/removing `scripts/check-req-coverage.sh` — [req-coverage-script-removal.md](req-coverage-script-removal.md).
|
||||
- Adding TRACES comments to raise the actual coverage number.
|
||||
- Changing the `TRACES:` comment format or the extractor's parsing of it.
|
||||
- The PR "modified files missing TRACES" step (lines 78–126), which is advisory
|
||||
by design and stays advisory.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- [ ] `bun run traces:json` emits `defined`, `coverage`, and `orphaned` keys.
|
||||
- [ ] `coverage.total` equals the count of requirement IDs defined in
|
||||
`requirements.md` (**211** at time of writing), not a literal.
|
||||
- [ ] `coverage.percent` reports **85** (±1 for rounding) on the current tree —
|
||||
i.e. the honest number, not 158.
|
||||
- [ ] No hardcoded requirement denominator (`39`, `24`, `48`, `3`, `114`) remains
|
||||
in `.gitea/workflows/traceability-check.yml`. Verify:
|
||||
`grep -nE '/ *(39|24|48|3|114)\b' .gitea/workflows/traceability-check.yml`
|
||||
returns nothing.
|
||||
- [ ] Adding a new requirement row to `requirements.md` **lowers** reported
|
||||
coverage until it is traced (proves the denominator is live).
|
||||
- [ ] A `TRACES:` comment naming an undefined ID appears in `orphaned` and does
|
||||
**not** raise `coverage.percent`.
|
||||
- [ ] The job fails if coverage is forced below 50% (test by temporarily raising
|
||||
`MIN_THRESHOLD` to 99 locally) — proving the gate can fail again.
|
||||
- [ ] The job fails if coverage computes >100%.
|
||||
- [ ] `bun run check` and `bun run test` pass.
|
||||
- [ ] `bun run check:boundary` passes.
|
||||
- [ ] New requirement-implementing code carries `// TRACES:` comments.
|
||||
- [ ] No Rust types changed, so no `bindings.ts` regeneration needed.
|
||||
|
||||
## Testing
|
||||
|
||||
`extract-traces.ts` currently has no test coverage. Add
|
||||
`scripts/extract-traces.test.ts` (vitest) over fixture strings rather than the
|
||||
live `requirements.md`, so the tests do not change meaning as requirements are
|
||||
added:
|
||||
|
||||
- **UT:** counts a well-formed table row as a defined requirement.
|
||||
- **UT:** does **not** count an ID appearing only in the `Traces To` column or
|
||||
in prose — the specific overcounting bug this parse rule avoids.
|
||||
- **UT:** coverage is the intersection — a traced-but-undefined ID lands in
|
||||
`orphaned` and does not inflate the numerator.
|
||||
- **UT:** coverage of an empty trace set is 0%, not a divide-by-zero.
|
||||
- **UT:** all-traced fixture reports exactly 100%, never above.
|
||||
|
||||
CI behaviour is verified by the acceptance criteria above (the forced-failure
|
||||
check is the important one — a gate nobody has watched fail is not known to
|
||||
work).
|
||||
|
||||
## TRACES
|
||||
|
||||
Allocate in `requirements.md`:
|
||||
|
||||
- **DR-093** — "Traceability coverage gate derives requirement denominators from
|
||||
`requirements.md` at run time (not hardcoded literals), computes coverage as
|
||||
the intersection of traced and defined IDs, reports IDs traced but undefined
|
||||
as orphaned, and fails on an impossible >100% result." Category: Tooling.
|
||||
Status: Done on merge.
|
||||
|
||||
Tag:
|
||||
|
||||
```typescript
|
||||
// scripts/extract-traces.ts
|
||||
// TRACES: | DR-093
|
||||
```
|
||||
|
||||
Tests carry `@req-test: UT-089 …` onward (next free UT is **UT-089**).
|
||||
|
||||
## Notes for the implementer
|
||||
|
||||
- A parallel Claude session may be active in this repo — run `git diff` before
|
||||
"repairing" unexpected changes (CLAUDE.md §Gotchas).
|
||||
- **Do not add tooling to the CI image for this.** `jq` and `bun` are already in
|
||||
`jellytau-builder`; this spec needs nothing else. Installing a system package
|
||||
in a workflow step violates the hard CI rule in CLAUDE.md.
|
||||
- Keep `traces:json`'s existing keys intact — `release-notes.ts` and
|
||||
`traces:markdown` consume the same report, and the CI workflow uploads it as
|
||||
an artifact. This is an additive change.
|
||||
- The `head -50 docs/traceability.md` and artifact-upload steps are unaffected.
|
||||
- Expect the first green build after this change to print a *lower* number than
|
||||
before (85% vs 158%). That is the fix working, not a regression.
|
||||
Reference in New Issue
Block a user