build: requirement traceability extractor, gate, and CI workflow
Ports JellyTau's traceability tooling, rewritten in stdlib Python because
this repo is C++/Python and adding a bun/node toolchain to check source
comments would be a worse trade than writing the scanner.
scripts/traceability/extract_traces.py scans .cpp/.hpp/.py under src, tests,
scripts, experiments and eval for the house tag format
/// TRACES: AR-012, AR-013 | SR-002
and reports EXCEPTION tags separately. An exception is a recorded decision to
depart from an invariant, so folding it into coverage would invert its
meaning; it is listed with its reason, and a missing reason is flagged.
Two rules carried over from JellyTau's gate repair:
* Denominators are parsed out of docs/requirements.md at run time. A
requirement is defined only by a row in a table whose header is
`| ID | Requirement | ... |`, so references in the Traces to column, in
prose, and in the verification-plan table do not inflate the count.
Adding a register row lowers coverage until it is traced - the property
that dies the moment a denominator is frozen.
* Coverage above 100% is a hard failure. It cannot happen through the
intersection, which is the point: if it ever does, the arithmetic is
broken and the run must not be reported as a pass.
One rule specific to this repo: CI is an Intel N100 with no discrete GPU. The
extractor reads each requirement's verification tier from requirements.md and
reports T4/GPU-only requirements as tagged but unexecuted, never as covered.
Counting a test that can never run is the same failure mode as the 158% bug.
MIN_COVERAGE starts at 0 because almost nothing is tagged yet - tags are added
as the pipeline is built. That is not a gate that cannot fail: orphan tags, a
>100% ratio, a register that parses to nothing, and an empty source scan are
all hard failures from day one. The threshold lives in traceability-gate.sh
alone, never duplicated into the workflow YAML.
53 tests over fixture strings, so their meaning does not drift as requirements
are added.
This commit is contained in:
Executable
+64
@@ -0,0 +1,64 @@
|
||||
#!/bin/sh
|
||||
#
|
||||
# Requirement traceability gate. Run locally exactly as CI runs it:
|
||||
#
|
||||
# scripts/traceability/traceability-gate.sh
|
||||
#
|
||||
# Writes traces-report.json and docs/traceability.md, prints the coverage
|
||||
# report, and exits non-zero when the gate fails.
|
||||
#
|
||||
# Environment:
|
||||
# MIN_COVERAGE minimum overall coverage percent (default 0 - see below)
|
||||
# ALLOW_ORPHANS set to 1 to report orphan tags without failing
|
||||
# TRACES_JSON JSON report path (default traces-report.json)
|
||||
# TRACES_MD markdown matrix path (default docs/traceability.md)
|
||||
# SYSTEM_SPEC optional path to the umbrella SPEC.md, which defines the
|
||||
# PR/SR IDs; when given, PR/SR orphans are reported too. That
|
||||
# file lives in the parent project, not in this repo, so CI
|
||||
# normally leaves it unset.
|
||||
#
|
||||
# Threshold policy lives here and nowhere else. It is deliberately NOT
|
||||
# duplicated into the workflow YAML: a threshold written in two places is a
|
||||
# threshold that will disagree with itself.
|
||||
#
|
||||
# MIN_COVERAGE defaults to 0 because almost nothing is tagged yet - tags are
|
||||
# added as the pipeline is built, so a low number today is accurate rather than
|
||||
# alarming. A zero threshold does NOT mean the gate cannot fail: orphan tags,
|
||||
# a >100% ratio, a register that parses to nothing, and an empty source scan
|
||||
# are all hard failures from day one. Raise MIN_COVERAGE as tags land; treat
|
||||
# every raise as a ratchet, never a reset.
|
||||
#
|
||||
# POSIX sh, no bashisms, no jq - the extractor does its own arithmetic and
|
||||
# printing so CI needs nothing beyond python3.
|
||||
|
||||
set -eu
|
||||
|
||||
SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
|
||||
REPO_ROOT=$(CDPATH= cd -- "$SCRIPT_DIR/../.." && pwd)
|
||||
|
||||
MIN_COVERAGE="${MIN_COVERAGE:-0}"
|
||||
TRACES_JSON="${TRACES_JSON:-$REPO_ROOT/traces-report.json}"
|
||||
TRACES_MD="${TRACES_MD:-$REPO_ROOT/docs/traceability.md}"
|
||||
|
||||
PYTHON="${PYTHON:-python3}"
|
||||
command -v "$PYTHON" >/dev/null 2>&1 || {
|
||||
echo "FAILED: $PYTHON not found. The traceability gate needs Python 3.9+" >&2
|
||||
exit 2
|
||||
}
|
||||
|
||||
set -- \
|
||||
--root "$REPO_ROOT" \
|
||||
--format coverage \
|
||||
--json-out "$TRACES_JSON" \
|
||||
--markdown-out "$TRACES_MD" \
|
||||
--min-coverage "$MIN_COVERAGE"
|
||||
|
||||
if [ "${ALLOW_ORPHANS:-0}" = "1" ]; then
|
||||
set -- "$@" --allow-orphans
|
||||
fi
|
||||
|
||||
if [ -n "${SYSTEM_SPEC:-}" ]; then
|
||||
set -- "$@" --system-spec "$SYSTEM_SPEC"
|
||||
fi
|
||||
|
||||
exec "$PYTHON" "$SCRIPT_DIR/extract_traces.py" "$@"
|
||||
Reference in New Issue
Block a user