Ports JellyTau's traceability tooling, rewritten in stdlib Python because
this repo is C++/Python and adding a bun/node toolchain to check source
comments would be a worse trade than writing the scanner.
scripts/traceability/extract_traces.py scans .cpp/.hpp/.py under src, tests,
scripts, experiments and eval for the house tag format
/// TRACES: AR-012, AR-013 | SR-002
and reports EXCEPTION tags separately. An exception is a recorded decision to
depart from an invariant, so folding it into coverage would invert its
meaning; it is listed with its reason, and a missing reason is flagged.
Two rules carried over from JellyTau's gate repair:
* Denominators are parsed out of docs/requirements.md at run time. A
requirement is defined only by a row in a table whose header is
`| ID | Requirement | ... |`, so references in the Traces to column, in
prose, and in the verification-plan table do not inflate the count.
Adding a register row lowers coverage until it is traced - the property
that dies the moment a denominator is frozen.
* Coverage above 100% is a hard failure. It cannot happen through the
intersection, which is the point: if it ever does, the arithmetic is
broken and the run must not be reported as a pass.
One rule specific to this repo: CI is an Intel N100 with no discrete GPU. The
extractor reads each requirement's verification tier from requirements.md and
reports T4/GPU-only requirements as tagged but unexecuted, never as covered.
Counting a test that can never run is the same failure mode as the 158% bug.
MIN_COVERAGE starts at 0 because almost nothing is tagged yet - tags are added
as the pipeline is built. That is not a gate that cannot fail: orphan tags, a
>100% ratio, a register that parses to nothing, and an empty source scan are
all hard failures from day one. The threshold lives in traceability-gate.sh
alone, never duplicated into the workflow YAML.
53 tests over fixture strings, so their meaning does not drift as requirements
are added.
126 lines
4.4 KiB
YAML
126 lines
4.4 KiB
YAML
name: Traceability Validation
|
|
|
|
# Mirrors JellyTau's .gitea/workflows/traceability-check.yml, adapted for a
|
|
# C++/Python repo: the extractor is Python and needs nothing but python3, so
|
|
# there is no toolchain install step and no jq.
|
|
#
|
|
# NOTE: the runner here is an Intel N100 with no discrete GPU. This job is only
|
|
# ever static analysis of source comments plus markdown parsing, so it is cheap;
|
|
# the requirements it reports as "tagged but unexecuted" are the ones that need
|
|
# a GPU host, and they are deliberately never counted as covered.
|
|
|
|
on:
|
|
push:
|
|
branches:
|
|
- main
|
|
- master
|
|
- develop
|
|
pull_request:
|
|
branches:
|
|
- main
|
|
- master
|
|
- develop
|
|
|
|
jobs:
|
|
validate-traces:
|
|
runs-on: linux/amd64
|
|
name: Check requirement traces
|
|
|
|
steps:
|
|
- name: Checkout repository
|
|
uses: actions/checkout@v4
|
|
with:
|
|
fetch-depth: 0
|
|
|
|
- name: Check Python is available
|
|
run: |
|
|
set -e
|
|
command -v python3 >/dev/null 2>&1 || {
|
|
echo "python3 is missing from the runner image."
|
|
echo "The traceability tooling is stdlib-only Python 3.9+;"
|
|
echo "no other dependency is needed."
|
|
exit 1
|
|
}
|
|
python3 --version
|
|
|
|
# The gate's own arithmetic is the thing being trusted, so its tests run
|
|
# before it does. JellyTau's gate was believed for months while it was
|
|
# dividing by frozen literals; untested gate logic is how that happens.
|
|
- name: Test the extractor
|
|
run: python3 scripts/traceability/test_extract_traces.py
|
|
|
|
# Threshold policy lives in traceability-gate.sh, not here, so local runs
|
|
# and CI runs cannot disagree about what "passing" means. Denominators
|
|
# come from docs/requirements.md at run time and are never hardcoded --
|
|
# in this file or anywhere else.
|
|
- name: Traceability gate
|
|
run: sh scripts/traceability/traceability-gate.sh
|
|
|
|
- name: Check modified files for traces
|
|
if: github.event_name == 'pull_request'
|
|
run: |
|
|
set -e
|
|
echo "Checking modified sources for TRACES tags..."
|
|
|
|
CHANGED=$(git diff --name-only "origin/${{ github.base_ref }}...HEAD" \
|
|
| grep -E '\.(cpp|cc|cxx|hpp|hxx|h|cu|cuh|py)$' || true)
|
|
|
|
if [ -z "$CHANGED" ]; then
|
|
echo "No C++/Python files changed."
|
|
exit 0
|
|
fi
|
|
|
|
echo "Changed files:"
|
|
echo "$CHANGED" | sed 's/^/ /'
|
|
echo ""
|
|
|
|
# Advisory by design: not every file implements a requirement, and a
|
|
# tag on every function is noise that rots faster than it helps
|
|
# (CLAUDE.md: tag the unit that decides). This step exists to prompt,
|
|
# not to block. The blocking checks are in the gate step above.
|
|
#
|
|
# Piped into the loop rather than a here-string, and `case` rather
|
|
# than `[[ == ]]`, so this works under dash as well as bash. The loop
|
|
# body runs in a subshell, so misses are recorded in a file.
|
|
MISSING=$(mktemp)
|
|
echo "$CHANGED" | while IFS= read -r file; do
|
|
case "$file" in
|
|
*/test_*.py|*_test.py|tests/*|*/tests/*) continue ;;
|
|
esac
|
|
[ -f "$file" ] || continue
|
|
if ! grep -q 'TRACES:' "$file"; then
|
|
echo " no TRACES tag: $file"
|
|
echo "$file" >> "$MISSING"
|
|
fi
|
|
done
|
|
|
|
COUNT=$(wc -l < "$MISSING" | tr -d ' ')
|
|
rm -f "$MISSING"
|
|
|
|
if [ "$COUNT" -gt 0 ]; then
|
|
echo ""
|
|
echo "$COUNT changed file(s) carry no requirement tag."
|
|
echo "Format: // TRACES: AR-012, AR-013 | SR-002"
|
|
echo " (pipe separates requirement types, comma separates IDs)"
|
|
echo "A deliberate invariant exception is tagged separately:"
|
|
echo " // EXCEPTION: AR-024 <reason>"
|
|
echo "See CLAUDE.md and SPEC.md section 6."
|
|
fi
|
|
|
|
- name: Report summary
|
|
if: always()
|
|
run: |
|
|
echo "Traceability matrix: docs/traceability.md"
|
|
echo ""
|
|
head -40 docs/traceability.md || true
|
|
|
|
- name: Save reports
|
|
if: always()
|
|
uses: actions/upload-artifact@v3
|
|
with:
|
|
name: traceability-reports
|
|
path: |
|
|
traces-report.json
|
|
docs/traceability.md
|
|
retention-days: 30
|