dtourolle
054c4c8b8f
Ports JellyTau's traceability tooling, rewritten in stdlib Python because
this repo is C++/Python and adding a bun/node toolchain to check source
comments would be a worse trade than writing the scanner.
scripts/traceability/extract_traces.py scans .cpp/.hpp/.py under src, tests,
scripts, experiments and eval for the house tag format
/// TRACES: AR-012, AR-013 | SR-002
and reports EXCEPTION tags separately. An exception is a recorded decision to
depart from an invariant, so folding it into coverage would invert its
meaning; it is listed with its reason, and a missing reason is flagged.
Two rules carried over from JellyTau's gate repair:
* Denominators are parsed out of docs/requirements.md at run time. A
requirement is defined only by a row in a table whose header is
`| ID | Requirement | ... |`, so references in the Traces to column, in
prose, and in the verification-plan table do not inflate the count.
Adding a register row lowers coverage until it is traced - the property
that dies the moment a denominator is frozen.
* Coverage above 100% is a hard failure. It cannot happen through the
intersection, which is the point: if it ever does, the arithmetic is
broken and the run must not be reported as a pass.
One rule specific to this repo: CI is an Intel N100 with no discrete GPU. The
extractor reads each requirement's verification tier from requirements.md and
reports T4/GPU-only requirements as tagged but unexecuted, never as covered.
Counting a test that can never run is the same failure mode as the 158% bug.
MIN_COVERAGE starts at 0 because almost nothing is tagged yet - tags are added
as the pipeline is built. That is not a gate that cannot fail: orphan tags, a
>100% ratio, a register that parses to nothing, and an empty source scan are
all hard failures from day one. The threshold lives in traceability-gate.sh
alone, never duplicated into the workflow YAML.
53 tests over fixture strings, so their meaning does not drift as requirements
are added.
65 lines
2.3 KiB
Bash
Executable File
65 lines
2.3 KiB
Bash
Executable File
#!/bin/sh
|
|
#
|
|
# Requirement traceability gate. Run locally exactly as CI runs it:
|
|
#
|
|
# scripts/traceability/traceability-gate.sh
|
|
#
|
|
# Writes traces-report.json and docs/traceability.md, prints the coverage
|
|
# report, and exits non-zero when the gate fails.
|
|
#
|
|
# Environment:
|
|
# MIN_COVERAGE minimum overall coverage percent (default 0 - see below)
|
|
# ALLOW_ORPHANS set to 1 to report orphan tags without failing
|
|
# TRACES_JSON JSON report path (default traces-report.json)
|
|
# TRACES_MD markdown matrix path (default docs/traceability.md)
|
|
# SYSTEM_SPEC optional path to the umbrella SPEC.md, which defines the
|
|
# PR/SR IDs; when given, PR/SR orphans are reported too. That
|
|
# file lives in the parent project, not in this repo, so CI
|
|
# normally leaves it unset.
|
|
#
|
|
# Threshold policy lives here and nowhere else. It is deliberately NOT
|
|
# duplicated into the workflow YAML: a threshold written in two places is a
|
|
# threshold that will disagree with itself.
|
|
#
|
|
# MIN_COVERAGE defaults to 0 because almost nothing is tagged yet - tags are
|
|
# added as the pipeline is built, so a low number today is accurate rather than
|
|
# alarming. A zero threshold does NOT mean the gate cannot fail: orphan tags,
|
|
# a >100% ratio, a register that parses to nothing, and an empty source scan
|
|
# are all hard failures from day one. Raise MIN_COVERAGE as tags land; treat
|
|
# every raise as a ratchet, never a reset.
|
|
#
|
|
# POSIX sh, no bashisms, no jq - the extractor does its own arithmetic and
|
|
# printing so CI needs nothing beyond python3.
|
|
|
|
set -eu
|
|
|
|
SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
|
|
REPO_ROOT=$(CDPATH= cd -- "$SCRIPT_DIR/../.." && pwd)
|
|
|
|
MIN_COVERAGE="${MIN_COVERAGE:-0}"
|
|
TRACES_JSON="${TRACES_JSON:-$REPO_ROOT/traces-report.json}"
|
|
TRACES_MD="${TRACES_MD:-$REPO_ROOT/docs/traceability.md}"
|
|
|
|
PYTHON="${PYTHON:-python3}"
|
|
command -v "$PYTHON" >/dev/null 2>&1 || {
|
|
echo "FAILED: $PYTHON not found. The traceability gate needs Python 3.9+" >&2
|
|
exit 2
|
|
}
|
|
|
|
set -- \
|
|
--root "$REPO_ROOT" \
|
|
--format coverage \
|
|
--json-out "$TRACES_JSON" \
|
|
--markdown-out "$TRACES_MD" \
|
|
--min-coverage "$MIN_COVERAGE"
|
|
|
|
if [ "${ALLOW_ORPHANS:-0}" = "1" ]; then
|
|
set -- "$@" --allow-orphans
|
|
fi
|
|
|
|
if [ -n "${SYSTEM_SPEC:-}" ]; then
|
|
set -- "$@" --system-spec "$SYSTEM_SPEC"
|
|
fi
|
|
|
|
exec "$PYTHON" "$SCRIPT_DIR/extract_traces.py" "$@"
|