AR-024's register row gives its verification tier as "Static check -- no bare cosine outside a tagged EXCEPTION". No such check existed, so the invariant was enforced by reading, and reading had missed a live violation. scripts/ci/check_raw_cosine.py is that check, wired into the traceability workflow as a blocking step. It is honest about its reach: it catches direct cosine_similarity() uses not routed through a calibration, and it cannot follow a cosine through a variable across statements. That limit is documented in the script rather than left for someone to discover after trusting a pass. What it caught, and what this commit removes with it: The identity matcher's no-calibration fallback thresholded raw cosine distance (match_threshold) plus a ratio test (match_ratio, match_ratio_ceil). Worse than the invariant breach: it fed max(0, cosine) into TrackRegistry::observe, whose contract reads "posterior is a calibrated probability, never a raw cosine (AR-024) ... so the accumulation cannot be fed an uncalibrated number by a careless caller". It could, and did. And it disagreed with the rest of the pipeline about what "the fit failed" means -- same_person_probability answers that with the untuned default sigmoid and a loud warning, so association stayed in probability space while matching alone left it. One run, two policies, no announcement. Now one rule: cal_.probability() always, with a warning when the fit is not real. A worse answer than a fitted calibration, a better one than a number whose units nothing else shares. TrackGallery::set_calibration is mandatory for the same reason. Its default was max(0, cosine), which made expand_band_lo = 0.90 mean "cosine > 0.9" in a test and "P(same person) > 0.9" in production. FaceTrackerFunc already threw without one; the expansion store now matches. One exception is recorded, in the calibration's own dedup. It is not a close call: at 1 - 1e-7 it asks whether two vectors are the same vector, and it runs on the fit's input, so a calibrated comparison there would have to be calibrated by the fit it is feeding. Also drops seven dead keys from the optimizer's CFG_KEYS. Config keys are read with a contains() check, so each one had been silently inert since the field behind it was deleted -- a sweep varying one of them measured nothing and reported an ordinary-looking F1. TRACES: AR-024, AR-023 | SR-002
155 lines
6.0 KiB
YAML
155 lines
6.0 KiB
YAML
name: Traceability Validation
|
|
|
|
# Mirrors JellyTau's .gitea/workflows/traceability-check.yml. The extractor is
|
|
# stdlib Python, so there is no toolchain install step and no jq.
|
|
#
|
|
# This workflow is component-agnostic: every repo-specific setting - which ID
|
|
# prefixes count, which file suffixes are source, which directories to scan,
|
|
# the threshold - lives in traceability.toml at the repo root, and the same
|
|
# extractor is shared by all three JRay components. Copying this file into
|
|
# another component needs no edits.
|
|
#
|
|
# NOTE: the runner here is an Intel N100 with no discrete GPU. This job is only
|
|
# ever static analysis of source comments plus markdown parsing, so it is cheap;
|
|
# the requirements it reports as "tagged but unexecuted" are the ones that need
|
|
# a GPU host, and they are deliberately never counted as covered.
|
|
|
|
on:
|
|
push:
|
|
branches:
|
|
- main
|
|
- master
|
|
- develop
|
|
pull_request:
|
|
branches:
|
|
- main
|
|
- master
|
|
- develop
|
|
|
|
jobs:
|
|
validate-traces:
|
|
runs-on: linux/amd64
|
|
name: Check requirement traces
|
|
|
|
steps:
|
|
- name: Checkout repository
|
|
uses: actions/checkout@v4
|
|
with:
|
|
fetch-depth: 0
|
|
submodules: recursive
|
|
|
|
- name: Check Python is available
|
|
run: |
|
|
set -e
|
|
command -v python3 >/dev/null 2>&1 || {
|
|
echo "python3 is missing from the runner image."
|
|
echo "The traceability tooling is stdlib-only Python;"
|
|
echo "3.9+ with CLI flags, 3.11+ to read traceability.toml."
|
|
exit 1
|
|
}
|
|
python3 --version
|
|
|
|
# The gate's own arithmetic is the thing being trusted, so its tests run
|
|
# before it does. JellyTau's gate was believed for months while it was
|
|
# dividing by frozen literals; untested gate logic is how that happens.
|
|
- name: Test the extractor
|
|
run: python3 scripts/vendor/jray-project/scripts/traceability/test_extract_traces.py
|
|
|
|
# Threshold policy and every other repo-specific setting live in
|
|
# traceability.toml, not here, so local runs and CI runs cannot disagree
|
|
# about what "passing" means. Denominators come from docs/requirements.md
|
|
# at run time and are never hardcoded -- in this file or anywhere else.
|
|
#
|
|
# A misconfigured run (zero requirements parsed, zero files scanned) is a
|
|
# hard failure rather than a plausible-looking 0%.
|
|
- name: Traceability gate
|
|
run: sh scripts/vendor/jray-project/scripts/traceability/traceability-gate.sh
|
|
|
|
# AR-024's register row names its verification tier as "Static check --
|
|
# no bare cosine outside a tagged EXCEPTION". This is that check, and it
|
|
# belongs here rather than in unit-tests.yml because it is static
|
|
# analysis of source text, like everything else in this job, and needs
|
|
# no toolchain. It blocks: an untagged bare cosine is a defect by the
|
|
# invariant's own wording, not a warning.
|
|
- name: AR-024 — no bare cosine outside a recorded exception
|
|
run: python3 scripts/ci/check_raw_cosine.py
|
|
|
|
- name: Check modified files for traces
|
|
if: github.event_name == 'pull_request'
|
|
run: |
|
|
set -e
|
|
echo "Checking modified sources for TRACES tags..."
|
|
|
|
# The extensions come from the report the gate just wrote, which got
|
|
# them from traceability.toml. Restating them here would be a second
|
|
# place for the source-file definition to live, and the two would
|
|
# drift the first time a language is added.
|
|
PATTERN=$(python3 -c "
|
|
import json, re, sys
|
|
suffixes = json.load(open('traces-report.json'))['config']['sourceSuffixes']
|
|
print('(' + '|'.join(re.escape(s) + '\$' for s in suffixes) + ')')
|
|
")
|
|
echo "Source suffixes from traceability.toml: $PATTERN"
|
|
|
|
CHANGED=$(git diff --name-only "origin/${{ github.base_ref }}...HEAD" \
|
|
| grep -E "$PATTERN" || true)
|
|
|
|
if [ -z "$CHANGED" ]; then
|
|
echo "No source files changed."
|
|
exit 0
|
|
fi
|
|
|
|
echo "Changed files:"
|
|
echo "$CHANGED" | sed 's/^/ /'
|
|
echo ""
|
|
|
|
# Advisory by design: not every file implements a requirement, and a
|
|
# tag on every function is noise that rots faster than it helps
|
|
# (CLAUDE.md: tag the unit that decides). This step exists to prompt,
|
|
# not to block. The blocking checks are in the gate step above.
|
|
#
|
|
# Piped into the loop rather than a here-string, and `case` rather
|
|
# than `[[ == ]]`, so this works under dash as well as bash. The loop
|
|
# body runs in a subshell, so misses are recorded in a file.
|
|
MISSING=$(mktemp)
|
|
echo "$CHANGED" | while IFS= read -r file; do
|
|
case "$file" in
|
|
*/test_*.py|*_test.py|*Tests.cs|tests/*|*/tests/*) continue ;;
|
|
esac
|
|
[ -f "$file" ] || continue
|
|
if ! grep -q 'TRACES:' "$file"; then
|
|
echo " no TRACES tag: $file"
|
|
echo "$file" >> "$MISSING"
|
|
fi
|
|
done
|
|
|
|
COUNT=$(wc -l < "$MISSING" | tr -d ' ')
|
|
rm -f "$MISSING"
|
|
|
|
if [ "$COUNT" -gt 0 ]; then
|
|
echo ""
|
|
echo "$COUNT changed file(s) carry no requirement tag."
|
|
echo "Format: // TRACES: AR-012, AR-013 | SR-002"
|
|
echo " (pipe separates requirement types, comma separates IDs)"
|
|
echo "A deliberate invariant exception is tagged separately:"
|
|
echo " // EXCEPTION: AR-024 <reason>"
|
|
echo "See CLAUDE.md and SPEC.md section 6."
|
|
fi
|
|
|
|
- name: Report summary
|
|
if: always()
|
|
run: |
|
|
echo "Traceability matrix: docs/traceability.md"
|
|
echo ""
|
|
head -40 docs/traceability.md || true
|
|
|
|
- name: Save reports
|
|
if: always()
|
|
uses: actions/upload-artifact@v3
|
|
with:
|
|
name: traceability-reports
|
|
path: |
|
|
traces-report.json
|
|
docs/traceability.md
|
|
retention-days: 30
|