ci(tests): actually run the tier the verification strategy is built on
docs/requirements.md describes four verification tiers and argues that T1 (functor unit) and T2 (replay) are "the only tiers that can exist in CI at all". The traceability gate then reports a CI-scope coverage fraction over exactly those tiers. Nothing ran them: the only workflow was the gate itself, which reads source comments, and SAE_BUILD_TESTS defaults to OFF. "Covered" meant a TRACES tag existed in a file. That is the same failure the gate's own config warns about one level up -- counting a test that cannot run -- and the gate cannot see it, because a tag is all a static reader has. Runs in the pinned DP-007 CPU builder image, which the image script already expected this workflow to exist (it names unit-tests.yml and asserts its tag). Nothing here calls a model: T1 constructs node functors directly and T2 replays a precomputed dump, so the GPU-free N100 runner is sufficient by construction rather than by concession. Two deliberate hard failures. A missing replay fixture fails the job instead of skipping, because pull_artifacts.sh warns-and-continues and a T2 test whose input never arrived must not look like a pass. And the image reporting a tag other than the pinned one fails rather than building against an unknown toolchain. It found a real bug on its first run: see the preceding commit. ctest runs each case in its own process, which turned a 1-in-4 heap corruption from noise in the aggregate binary into a reproducible failure. TRACES: DP-007 | PR-004
This commit is contained in:
@@ -0,0 +1,140 @@
|
||||
name: Unit tests
|
||||
|
||||
# TRACES: DP-007 | PR-004
|
||||
#
|
||||
# The tier the verification strategy is built on, finally executing.
|
||||
#
|
||||
# docs/requirements.md describes a four-tier plan in which T1 (functor unit)
|
||||
# and T2 (replay) are "the only tiers that can exist in CI at all", and the
|
||||
# traceability gate reports a CI-scope coverage fraction over exactly those
|
||||
# tiers. Until this workflow existed, nothing ran them: "covered" meant a
|
||||
# TRACES tag was present in a file, not that any test had been executed. That
|
||||
# is the same failure mode as counting a test that cannot run, one level up,
|
||||
# and the gate cannot detect it because a tag is all it can see.
|
||||
#
|
||||
# The runner is an Intel N100 with no discrete GPU. Nothing here calls a model:
|
||||
# T1 constructs node functors directly, and T2 replays a precomputed HDF5 dump.
|
||||
# T3 (ORT CPU smoke) and T4 (GPU) are deliberately absent -- the embedder is
|
||||
# ~930 ms/frame on this hardware, so a 77 s clip at 5 fps would be six minutes
|
||||
# of inference alone.
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- main
|
||||
- master
|
||||
- develop
|
||||
pull_request:
|
||||
branches:
|
||||
- main
|
||||
- master
|
||||
- develop
|
||||
|
||||
jobs:
|
||||
unit-tests:
|
||||
runs-on: linux/amd64
|
||||
name: Build and run the GPU-free suite
|
||||
|
||||
# Pinned by tag, never `latest`, so rebuilding the image cannot silently
|
||||
# change what a previous green build meant. Bumping the dependency set means
|
||||
# bumping the tag in scripts/ci/build_builder_image.sh AND here, in one
|
||||
# commit -- see that script's header.
|
||||
container:
|
||||
image: gitea.tourolle.paris/dtourolle/sae-builder-cpu:v1
|
||||
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
fetch-depth: 0
|
||||
# KPN is a submodule and the pipeline does not build without it.
|
||||
#
|
||||
# NOTE: this checks out the commit this repo PINS, which is the whole
|
||||
# point and is also the first thing this job will disagree with a
|
||||
# developer about. A local KPN working copy that is ahead of
|
||||
# origin/master builds and passes here while CI builds something else
|
||||
# entirely; the AR-004 evidence in docs/requirements.md was gathered
|
||||
# that way. If this job fails on tests that pass locally, check
|
||||
# `git -C external/KPN log origin/master..HEAD` before suspecting the
|
||||
# tests.
|
||||
# LFS is deliberately NOT fetched: SAE_MODELS_DIR is baked into the
|
||||
# binary as a path string and nothing in T1/T2 opens a model file, so
|
||||
# pulling ~hundreds of MB of ONNX would cost the job everything and
|
||||
# buy it nothing.
|
||||
submodules: recursive
|
||||
lfs: false
|
||||
|
||||
- name: Assert the builder image is the pinned one
|
||||
run: |
|
||||
set -e
|
||||
echo "builder=$SAE_BUILDER version=$SAE_BUILDER_VERSION"
|
||||
echo "ort=$SAE_ORT_VERSION opencv=$SAE_OPENCV_VERSION"
|
||||
# The image reports its own tag. A mismatch means the `container:`
|
||||
# line above and the image that actually landed disagree, which is
|
||||
# exactly the drift the pinning exists to prevent -- so it fails the
|
||||
# job rather than building against an unknown toolchain.
|
||||
[ "$SAE_BUILDER_VERSION" = "v1" ] || {
|
||||
echo "image reports version '$SAE_BUILDER_VERSION', workflow pins v1" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
- name: Fetch replay fixtures
|
||||
env:
|
||||
GITEA_TOKEN: ${{ secrets.GITEA_TOKEN }}
|
||||
# bash, not sh: the script declares #!/bin/bash and uses `set -o
|
||||
# pipefail` and arrays, which dash does not have.
|
||||
run: bash scripts/artifacts/pull_artifacts.sh replay-fixtures latest
|
||||
|
||||
# pull_artifacts.sh warns and continues when a package version is missing,
|
||||
# which is right for a developer pulling one artifact of several and wrong
|
||||
# here. A T2 test whose fixture never arrived must not look like a pass:
|
||||
# the dumps are the entire input to the replay tier, and VR-002's claim is
|
||||
# that replay drives the real nodes over real data.
|
||||
- name: Verify the fixtures actually arrived
|
||||
run: |
|
||||
set -e
|
||||
missing=0
|
||||
for f in tests/fixtures/dumps/superhero.h5; do
|
||||
if [ -s "$f" ]; then
|
||||
echo " ok: $f ($(wc -c < "$f") bytes)"
|
||||
else
|
||||
echo " MISSING: $f" >&2
|
||||
missing=1
|
||||
fi
|
||||
done
|
||||
if [ "$missing" -ne 0 ]; then
|
||||
echo "" >&2
|
||||
echo "Replay fixtures are absent, so the T2 tier cannot run." >&2
|
||||
echo "They are not in git (tests/fixtures/dumps/.gitignore) -- they" >&2
|
||||
echo "live in the Gitea generic package registry and are pulled by" >&2
|
||||
echo "the step above, which needs GITEA_TOKEN to resolve 'latest'." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Configure
|
||||
run: |
|
||||
set -e
|
||||
# SAE_GEMM_BACKEND defaults to ROCM and the auto-detect prefers a GPU
|
||||
# backend where it finds one; CPU is stated explicitly so this job
|
||||
# cannot start depending on what happens to be installed on the runner.
|
||||
# The CPU kernel is OpenBLAS in this image (tests/CMakeLists.txt fails
|
||||
# the configure if it is not), so the suite exercises the kernel the
|
||||
# CPU release actually ships.
|
||||
cmake -S . -B build \
|
||||
-DCMAKE_BUILD_TYPE=Release \
|
||||
-DSAE_BUILD_TESTS=ON \
|
||||
-DSAE_GEMM_BACKEND=CPU
|
||||
|
||||
- name: Build the test suite
|
||||
run: cmake --build build --target sae_tests --parallel
|
||||
|
||||
- name: Run the tests
|
||||
run: ctest --test-dir build --output-on-failure
|
||||
|
||||
- name: Save test output
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v3
|
||||
with:
|
||||
name: unit-test-results
|
||||
path: build/Testing/
|
||||
retention-days: 30
|
||||
Reference in New Issue
Block a user