feat(ar-024): enforce the invariant statically, and delete the fallback it caught
AR-024's register row gives its verification tier as "Static check -- no bare cosine outside a tagged EXCEPTION". No such check existed, so the invariant was enforced by reading, and reading had missed a live violation. scripts/ci/check_raw_cosine.py is that check, wired into the traceability workflow as a blocking step. It is honest about its reach: it catches direct cosine_similarity() uses not routed through a calibration, and it cannot follow a cosine through a variable across statements. That limit is documented in the script rather than left for someone to discover after trusting a pass. What it caught, and what this commit removes with it: The identity matcher's no-calibration fallback thresholded raw cosine distance (match_threshold) plus a ratio test (match_ratio, match_ratio_ceil). Worse than the invariant breach: it fed max(0, cosine) into TrackRegistry::observe, whose contract reads "posterior is a calibrated probability, never a raw cosine (AR-024) ... so the accumulation cannot be fed an uncalibrated number by a careless caller". It could, and did. And it disagreed with the rest of the pipeline about what "the fit failed" means -- same_person_probability answers that with the untuned default sigmoid and a loud warning, so association stayed in probability space while matching alone left it. One run, two policies, no announcement. Now one rule: cal_.probability() always, with a warning when the fit is not real. A worse answer than a fitted calibration, a better one than a number whose units nothing else shares. TrackGallery::set_calibration is mandatory for the same reason. Its default was max(0, cosine), which made expand_band_lo = 0.90 mean "cosine > 0.9" in a test and "P(same person) > 0.9" in production. FaceTrackerFunc already threw without one; the expansion store now matches. One exception is recorded, in the calibration's own dedup. It is not a close call: at 1 - 1e-7 it asks whether two vectors are the same vector, and it runs on the fit's input, so a calibrated comparison there would have to be calibrated by the fit it is feeding. Also drops seven dead keys from the optimizer's CFG_KEYS. Config keys are read with a contains() check, so each one had been silently inert since the field behind it was deleted -- a sweep varying one of them measured nothing and reported an ordinary-looking F1. TRACES: AR-024, AR-023 | SR-002
This commit is contained in:
Executable
+220
@@ -0,0 +1,220 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Enforce the AR-024 invariant: never a raw cosine, always the calibration.
|
||||
|
||||
TRACES: AR-024 | SR-002
|
||||
|
||||
docs/requirements.md gives AR-024's verification tier as "Static check -- no
|
||||
bare cosine outside a tagged EXCEPTION | Grep-based; this is the invariant's
|
||||
enforcement". This is that check. Until it existed the invariant was enforced
|
||||
by reading, and reading missed a live violation: the identity matcher's
|
||||
no-calibration fallback thresholded raw cosine distance and fed `max(0, cosine)`
|
||||
into the Bayesian accumulation as though it were a posterior.
|
||||
|
||||
WHAT IT CHECKS, precisely, because a static check that overclaims its reach is
|
||||
worse than one with a stated scope:
|
||||
|
||||
Every call to `cosine_similarity(...)` in C++ source must either
|
||||
|
||||
(a) have its result consumed immediately by a calibration -- the call is
|
||||
textually wrapped in `cal_(...)`, `calibrate_(...)`, `.probability(...)`
|
||||
or similar; or
|
||||
(b) sit under an `EXCEPTION: AR-024 <reason>` comment within
|
||||
EXCEPTION_SCOPE_LINES above it.
|
||||
|
||||
Anything else is a defect, per CLAUDE.md: "treat any bare cosine comparison in
|
||||
the code as a defect to be fixed".
|
||||
|
||||
WHAT IT DOES NOT CHECK, and why you should not read a pass as more than it is:
|
||||
|
||||
- It cannot follow a cosine through a variable across statements. A file that
|
||||
stores `float s = cosine_similarity(a, b);` and compares `s` three lines
|
||||
later is not caught. The codebase does not currently do this, and this check
|
||||
exists partly to keep it that way, but it is a convention backed by review,
|
||||
not by the tool.
|
||||
- It says nothing about GEMM output. The similarity engine returns a whole
|
||||
matrix of cosines and the matcher reads them directly; that path is correct
|
||||
by inspection (every value goes through `cal_.probability`) and is not
|
||||
verified here.
|
||||
- A retired constant reintroduced under a new name is invisible to it.
|
||||
|
||||
Exit status is 0 when clean, 1 when a violation is found, 2 on a usage error.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import pathlib
|
||||
import re
|
||||
import sys
|
||||
|
||||
# How far above a use an `EXCEPTION: AR-024` tag may sit and still cover it.
|
||||
# Generous, because the house style puts a paragraph of reasoning between the
|
||||
# tag and the code -- but bounded, so a tag cannot silently cover a whole file.
|
||||
EXCEPTION_SCOPE_LINES = 25
|
||||
|
||||
CPP_SUFFIXES = {".h", ".hpp", ".hxx", ".cc", ".cpp", ".cxx", ".cu", ".cuh"}
|
||||
|
||||
# src only, deliberately. The invariant governs what the PIPELINE decides --
|
||||
# CLAUDE.md's rule is "tag the unit that decides" -- whereas a test legitimately
|
||||
# asserts properties of the metric space itself (that a vector's cosine with
|
||||
# itself is 1, that the annex ended up holding the spoke it should have). Those
|
||||
# are measurements of the code under test, not decisions shipped to a user, and
|
||||
# sweeping them in would produce a wall of blanket EXCEPTION tags that would
|
||||
# devalue the tag everywhere else. Pass --source-root tests to scan them anyway.
|
||||
DEFAULT_ROOTS = ["src"]
|
||||
|
||||
# Directories that are never this repo's code.
|
||||
EXCLUDE_DIRS = {
|
||||
"build", "build-ort", "external", "vendor", "__pycache__",
|
||||
".git", "node_modules", "models",
|
||||
}
|
||||
|
||||
COSINE_CALL = re.compile(r"\bcosine_similarity\s*\(")
|
||||
|
||||
# The result is immediately handed to a calibration. Matches the house shapes:
|
||||
# cal_(cosine_similarity(a, b))
|
||||
# calibrate_(cosine_similarity(a, b))
|
||||
# same_person(cosine_similarity(a, b))
|
||||
# cal_.probability(cosine_similarity(a, b))
|
||||
CALIBRATED = re.compile(
|
||||
r"(?:\b(?:cal_|cal|calibrate_|calibrate|same_person|same_person_probability)"
|
||||
r"\s*(?:\.\s*probability\s*)?\(\s*|\.\s*probability\s*\(\s*)"
|
||||
r"cosine_similarity\s*\("
|
||||
)
|
||||
|
||||
EXCEPTION_TAG = re.compile(r"EXCEPTION:\s*AR-024\b(.*)")
|
||||
|
||||
# The function's own definition is not a use of it.
|
||||
DEFINITION = re.compile(r"^\s*(?:inline\s+|static\s+|constexpr\s+)*float\s+"
|
||||
r"cosine_similarity\s*\(")
|
||||
|
||||
# The house style wraps long calls across lines:
|
||||
# const float p = calibrate_(
|
||||
# cosine_similarity(a, b));
|
||||
# so the calibration and the call it guards are not always on one line. Joining
|
||||
# a small window before testing is what makes this check usable on real code
|
||||
# rather than a generator of false positives that trains people to ignore it.
|
||||
JOIN_LOOKBEHIND = 2
|
||||
|
||||
|
||||
def iter_sources(root: pathlib.Path, roots):
|
||||
for rel in roots:
|
||||
base = root / rel
|
||||
if not base.exists():
|
||||
continue
|
||||
for p in sorted(base.rglob("*")):
|
||||
if p.suffix.lower() not in CPP_SUFFIXES:
|
||||
continue
|
||||
if any(part in EXCLUDE_DIRS for part in p.relative_to(root).parts):
|
||||
continue
|
||||
yield p
|
||||
|
||||
|
||||
def covering_exception(lines, idx):
|
||||
"""Return the reason text of an EXCEPTION: AR-024 tag covering line `idx`."""
|
||||
lo = max(0, idx - EXCEPTION_SCOPE_LINES)
|
||||
for j in range(idx, lo - 1, -1):
|
||||
m = EXCEPTION_TAG.search(lines[j])
|
||||
if m:
|
||||
return m.group(1).strip(" -—*/") or "(no reason given)"
|
||||
return None
|
||||
|
||||
|
||||
def check_file(path: pathlib.Path, root: pathlib.Path):
|
||||
violations, exceptions = [], []
|
||||
try:
|
||||
lines = path.read_text(encoding="utf-8", errors="replace").splitlines()
|
||||
except OSError as e:
|
||||
print(f"error: cannot read {path}: {e}", file=sys.stderr)
|
||||
return violations, exceptions
|
||||
|
||||
rel = path.relative_to(root)
|
||||
for i, line in enumerate(lines):
|
||||
if not COSINE_CALL.search(line):
|
||||
continue
|
||||
# A comment mentioning the function is prose, not a use.
|
||||
stripped = line.lstrip()
|
||||
if stripped.startswith(("//", "///", "*", "/*")):
|
||||
continue
|
||||
if DEFINITION.match(line):
|
||||
continue
|
||||
# Join a small window so a call wrapped across lines is still seen as
|
||||
# calibrated. Whitespace is collapsed so the join reads as one statement.
|
||||
window = " ".join(
|
||||
lines[max(0, i - JOIN_LOOKBEHIND):i + 1]
|
||||
)
|
||||
window = re.sub(r"\s+", " ", window)
|
||||
if CALIBRATED.search(window):
|
||||
continue
|
||||
reason = covering_exception(lines, i)
|
||||
if reason:
|
||||
exceptions.append((rel, i + 1, line.strip(), reason))
|
||||
else:
|
||||
violations.append((rel, i + 1, line.strip()))
|
||||
return violations, exceptions
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__,
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument("--root", default=None,
|
||||
help="repository root (default: the script's ../..)")
|
||||
ap.add_argument("--source-root", action="append", default=None,
|
||||
help="directory to scan; repeatable (default: src, tests)")
|
||||
args = ap.parse_args()
|
||||
|
||||
root = pathlib.Path(args.root) if args.root \
|
||||
else pathlib.Path(__file__).resolve().parents[2]
|
||||
roots = args.source_root or DEFAULT_ROOTS
|
||||
|
||||
if not root.is_dir():
|
||||
print(f"error: root {root} is not a directory", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
all_violations, all_exceptions, n_files = [], [], 0
|
||||
for p in iter_sources(root, roots):
|
||||
n_files += 1
|
||||
v, e = check_file(p, root)
|
||||
all_violations += v
|
||||
all_exceptions += e
|
||||
|
||||
if n_files == 0:
|
||||
# A scan that found nothing to read is a misconfiguration reporting a
|
||||
# pass, which is the failure mode the traceability gate also guards.
|
||||
print(f"error: scanned 0 source files under {root} ({', '.join(roots)})",
|
||||
file=sys.stderr)
|
||||
return 2
|
||||
|
||||
print("AR-024 — always the calibrated probability, never a raw cosine")
|
||||
print("=" * 72)
|
||||
print(f"Repo root : {root}")
|
||||
print(f"Files scanned : {n_files} ({', '.join(roots)})")
|
||||
print(f"Recorded excs. : {len(all_exceptions)}")
|
||||
print(f"Violations : {len(all_violations)}")
|
||||
|
||||
if all_exceptions:
|
||||
print("\nRecorded exceptions (allowed, and each one is a claim to re-read):")
|
||||
for rel, ln, src, reason in all_exceptions:
|
||||
print(f" {rel}:{ln} {reason}")
|
||||
print(f" {src}")
|
||||
|
||||
if all_violations:
|
||||
print("\nVIOLATIONS — a bare cosine with no recorded exception:")
|
||||
for rel, ln, src in all_violations:
|
||||
print(f" {rel}:{ln}")
|
||||
print(f" {src}")
|
||||
print("\nEvery similarity is converted through the sigmoid calibration")
|
||||
print("before it is used, compared, or thresholded. A raw cosine means")
|
||||
print("something different for every model, gallery and face size, and")
|
||||
print("it cannot be combined with anything else.")
|
||||
print("\nEither route it through the calibration, or, if the use is")
|
||||
print("genuinely about the metric space rather than about a decision,")
|
||||
print("record it:")
|
||||
print(" // EXCEPTION: AR-024 <why this one is not a decision>")
|
||||
print("and add a row to CLAUDE.md's agreed-exceptions table.")
|
||||
return 1
|
||||
|
||||
print("\nOK: no bare cosine outside a recorded exception.")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Reference in New Issue
Block a user