From 4567c637970b49672d589e39b161600c75a7c445 Mon Sep 17 00:00:00 2001 From: Duncan Tourolle Date: Thu, 20 Aug 2026 19:30:48 +0200 Subject: [PATCH] docs: raise the documented traceability gate to 88% The spec review checklist still asked for >= 50%, the figure the gate sat at before it was found to be unreachable; traceability-ci.md carried 82% throughout. Both now read 88%, matching the ratchet, and the checklist points at `bun run traces:coverage` rather than inviting anyone to trust a number written in a document. Also refreshes the two stale coverage snapshots in traceability-ci.md (~86% from July 2026, and targets of 70% and 90% that the current 90% already passes) and records the 50 -> 82 -> 88 ratchet history. --- docs/specs/SPEC-REVIEW-CHECKLIST.md | 3 ++- docs/traceability-ci.md | 27 ++++++++++++++------------- 2 files changed, 16 insertions(+), 14 deletions(-) diff --git a/docs/specs/SPEC-REVIEW-CHECKLIST.md b/docs/specs/SPEC-REVIEW-CHECKLIST.md index e31015c8..f6114b35 100644 --- a/docs/specs/SPEC-REVIEW-CHECKLIST.md +++ b/docs/specs/SPEC-REVIEW-CHECKLIST.md @@ -50,7 +50,8 @@ Copy the boxes into the review comment (or the PR) and tick them. - [ ] Linked to existing URs, or new URs/DRs are allocated in [requirements.md](../requirements.md). - [ ] Requirement-implementing code will carry `// TRACES:` comments (CLAUDE.md). -- [ ] Traceability coverage stays ≥ 50% (the CI gate). +- [ ] Traceability coverage stays ≥ 88% (the CI gate — a ratchet, so check + `bun run traces:coverage` rather than trusting this number). ## Conflicts & hygiene diff --git a/docs/traceability-ci.md b/docs/traceability-ci.md index 33853173..33f3854a 100644 --- a/docs/traceability-ci.md +++ b/docs/traceability-ci.md @@ -15,7 +15,7 @@ The CI/CD pipeline automatically validates that code changes are properly traced Traceability validation lives in `.gitea/workflows/traceability-check.yml`: - ✅ Automatic trace extraction -- ✅ Coverage validation against minimum threshold (82%, ratcheted) +- ✅ Coverage validation against minimum threshold (88%, ratcheted) - ✅ Modified file checking - ✅ Artifact preservation - ✅ Summary reports @@ -43,7 +43,7 @@ Extracts all TRACES comments from: ### 2. Coverage Thresholds The workflow checks: -- **Minimum overall coverage:** 82% (`MIN_THRESHOLD`) +- **Minimum overall coverage:** 88% (`MIN_THRESHOLD`) Denominators are **derived from `docs/requirements.md` at run time** — they are never hardcoded here or in the workflow. Run `bun run traces:coverage` for the @@ -67,9 +67,10 @@ or if it computes above 100%, which can only mean the gate is miscounting. #### Ratchet policy `MIN_THRESHOLD` **only ever goes up.** It is deliberately set a few points below -the coverage actually achieved (82 against a real 86%), so a genuine regression +the coverage actually achieved (88 against a real ~90%), so a genuine regression trips it. It previously sat at 50 while true coverage was 86%: nearly half the -matrix could have rotted before CI objected. +matrix could have rotted before CI objected. It was ratcheted 50 → 82 when that +was found, and 82 → 88 once coverage had held above 88% for several releases. When coverage rises durably, raise the threshold to just under the new figure. **Never lower it to make a red build pass** — add the missing TRACES comments @@ -151,13 +152,13 @@ TRACES: [UR-###, ...] | [IR-###, ...] | [DR-###, ...] | [JA-###, ...] ### On Push to Main Branch 1. ✅ Extracts all traces from code -2. ✅ Validates coverage is >= 82% +2. ✅ Validates coverage is >= 88% 3. ✅ Generates full traceability report 4. ✅ Saves report as artifact ### On Pull Request 1. ✅ Extracts all traces -2. ✅ Validates coverage >= 82% +2. ✅ Validates coverage >= 88% 3. ✅ Checks modified files for TRACES 4. ✅ Warns if new code lacks TRACES 5. ✅ Suggests proper format @@ -165,7 +166,7 @@ TRACES: [UR-###, ...] | [IR-###, ...] | [DR-###, ...] | [JA-###, ...] ### Failure Scenarios The workflow **fails** (blocks merge) if: -- Coverage drops below 82% +- Coverage drops below 88% - A `TRACES:` comment names an ID `docs/requirements.md` does not define - JSON extraction fails - Invalid trace format @@ -203,12 +204,12 @@ below threshold. Numbers are deliberately not pinned here; the previous snapshot in this section (51%, 56/114) was stale by roughly 100 requirements and was what made the broken CI arithmetic look plausible for so long. -As of July 2026 overall coverage is ~86% (182/212). +As of August 2026 overall coverage is ~90%. ### Targets -- **Short term** (Sprint): Maintain ≥82% overall (the current ratchet) -- **Medium term** (Month): Reach 70% overall coverage -- **Long term** (Release): Reach 90% coverage with focus on: +- **Short term** (Sprint): Maintain ≥88% overall (the current ratchet) +- **Medium term** (Month): Hold above 90% and ratchet the gate to match +- **Long term** (Release): Reach 95% coverage with focus on: - IR requirements (API clients) - JA requirements (Jellyfin API endpoints) - Remaining UR/DR requirements @@ -241,14 +242,14 @@ When submitting a pull request: - [ ] All new code has TRACES comments linking to requirements - [ ] TRACES format is correct: `// TRACES: UR-001 | DR-002` -- [ ] Workflow passes (coverage ≥ 82%) +- [ ] Workflow passes (coverage ≥ 88%) - [ ] No coverage regressions - [ ] Artifact traceability report was generated ## Troubleshooting ### "Coverage below minimum threshold" -**Problem:** Workflow fails with coverage < 82% +**Problem:** Workflow fails with coverage < 88% **Solution:** 1. Run `bun run traces:json` locally