dtourolle/scene-actor-extraction
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: dtourolle/scene-actor-extraction:e1423062e260d6a393affc0c2ffe258dec7ce612
Branches Tags
dtourolle/scene-actor-extraction:master dtourolle/scene-actor-extraction:feature/opencv5 dtourolle/scene-actor-extraction:feature/quality-knee dtourolle/scene-actor-extraction:feature/traceability-tooling dtourolle/scene-actor-extraction:gitea-pages
dtourolle/scene-actor-extraction:v1.0.0
..
compare: dtourolle/scene-actor-extraction:feature/quality-knee
Branches Tags
dtourolle/scene-actor-extraction:master dtourolle/scene-actor-extraction:feature/opencv5 dtourolle/scene-actor-extraction:feature/quality-knee dtourolle/scene-actor-extraction:feature/traceability-tooling dtourolle/scene-actor-extraction:gitea-pages
dtourolle/scene-actor-extraction:v1.0.0
4 Commits
e1423062e2 .. feature/quality-knee
Author SHA1 Message Date
dtourolle 889018aa34 docs(VR-012): the quality knee, and what it retires 
Sharpness is not a sufficient statistic for identity loss. Six cells at
effectively identical measured sharpness span 15.3% to 91.0% TPI,
ordered entirely by source size, because a scalar keyed on
high-frequency energy cannot separate attenuated high frequencies from
destroyed spatial sampling. AR-028's "kept separate, not collapsed into
one scalar" now rests on a measurement rather than an argument -- and
the reasoning it used to rest on, that the aligned crop is
scale-normalised so a measure there cannot re-measure size, was wrong
and is corrected in place.

Variance of Laplacian -- the most widely used blur metric there is -- is
anti-predictive at fixed degradation on all three blur families. The
decile it calls sharpest is 2.6x less identifiable than the decile it
calls blurriest, monotone across ten bins, because within a cell its
residual variance is native contrast rather than detail, and hard
shadows and JPEG ringing raise it while making a face harder to match.
Gating on it would preferentially discard the more identifiable faces.

Blur breaks confidence, not identity: rank-1 holds at 80.2% where TPI is
15.3%, and FPI never left 0.1% in any of the 108 cells. Degradation
produces abstention, never a wrong name. That is also why sharpness
fails as a compute gate -- even a visually destroyed face stays 46.9%
identifiable, so a gate discards recoverable evidence at three times the
cost of the free size filter. Discount, do not gate; the rule AR-028
already stated now has evidence for why it is right rather than merely
cautious.

Records the shape a discount must have (flat, then a cliff between sigma
2 and 3), that its cost scales with proximity to the decision boundary
rather than with blur, and that the pose half of VR-012 has not been
run.

TRACES: VR-012, AR-028, AR-029, AR-030 | SR-002
 2026-07-31 22:29:47 +02:00
dtourolle 26de01b2e3 study(VR-012): quality knee over a joint size x blur grid, three blur families 
Extends the VR-005 protocol -- hold out one mugshot per actor, degrade
only the probe, match against a gallery held at native resolution,
decide through the Platt calibration -- from one axis to two, over 1670
actors rather than 100.

Joint rather than separable, because the interaction is the question: a
16 px face upscaled to 112 has already lost its high frequencies, so
further blur costs it almost nothing, while the same blur at full
resolution is expensive. Sweeping the axes independently would measure
each with the other implicitly at its best and miss that entirely.

Three blur families, compared at matched per-axis PSF spread rather than
at equal raw parameter. Optical defocus is a uniform disc whose transfer
function is a jinc with exact zeros, not a Gaussian that merely rolls
off, and it is also how a face ends up large and useless -- the case a
size filter cannot catch. Sweeping Gaussian alone, as the first version
did, understates real lens blur by a factor of five in error rate.

Every candidate measure is scored on every degraded crop and the
candidates are ranked by how well each predicts the pipeline's actual
decision, not by how smooth its synthetic ladder looks. Both a pooled
and a within-cell AUC are reported: they answer different questions and
the candidates rank differently under each.

Runs through sae_embed throughout. Stages gains optional engine paths so
the same study can drive a TRT build, which is what makes the full grid
five minutes rather than four and a half hours.

TRACES: VR-012, AR-028, AR-029 | SR-002
 2026-07-31 22:28:06 +02:00
dtourolle 41d30395da fix(ort): CUDA detection was gated behind the TensorRT-EP build flag 
detect_ort_provider() only tested for CUDAExecutionProvider inside
#ifdef SAE_ORT_WITH_TRT_EP, so any build that did not also opt into the
TensorRT execution provider could never select CUDA and fell straight
through to the CPU. The two are independent: the TRT EP needs the
headers and profile plumbing and is rightly an opt-in, CUDA is a plain
ORT provider and is not.

The failure is silent rather than loud, which is why it survived --
inference runs on the CPU and every answer is still correct, just far
slower. Measured on the VR-012 study: 0.32 s/crop against 0.0021 s/crop
once a GPU backend is actually used, with the card sitting at 212 MiB
and 0% utilisation throughout.

Only the TensorrtExecutionProvider line stays inside the guard.
 2026-07-31 22:25:00 +02:00
dtourolle 1ae88376e1 feat(quality): five AR-029 sharpness candidates on the aligned crop 
assess_sharpness() scores a 112x112 crop on variance-of-Laplacian, a
contrast-normalised variant, Tenengrad, a spectral high-frequency ratio
and dir_min_tenengrad, over a fixed 64x64 window on the face interior.
The window excludes the corners because studio headshots are routinely
shot at a wide aperture, and background bokeh measured over the whole
crop would drag the score down on the sharpest images in the set.

Five rather than one because AR-029's threshold has to be located, not
chosen: VR-012 ranks them by how well each predicts real identity loss.

The T1 ladders drove two corrections during development. The spectral
ratio applied its Hann window before removing the mean, so the DC term
smeared into the low-frequency bins and the "ratio" tracked absolute
brightness (a 20/255 brightening moved it 23%). And no measure taken
from the literature survived directional blur: normalising by total
energy divides out the loss being measured, so both ratio measures are
U-shaped in motion-blur length and score a 21 px smear about as sharp as
a 3 px one. dir_min_tenengrad exists to fix that -- a low-frequency
contrast denominator that blur leaves alone, and the worse of the two
Sobel axes rather than their sum.

The tests pin the disqualifying behaviours as well as the desirable
ones, so a change that makes var_laplacian contrast-free is a deliberate
act rather than an accident. They also record that every candidate falls
under downscale-upscale as well as under blur: the aligned crop is
scale-normalised geometrically, not informationally.

Exposed through sae_embed alongside the AR-030 alignment residual, so a
study scores through shipped code rather than a numpy copy -- the same
argument that already applies to the calibration.

TRACES: AR-028, AR-029 | SR-002
 2026-07-31 22:24:39 +02:00
94 changed files with 2796 additions and 7414 deletions
Expand all files Collapse all files
.gitea/workflows/traceability-check.yml
-9
View File
@@ -65,15 +65,6 @@ jobs:
- name: Traceability gate
run: sh scripts/vendor/jray-project/scripts/traceability/traceability-gate.sh
# AR-024's register row names its verification tier as "Static check --
# no bare cosine outside a tagged EXCEPTION". This is that check, and it
# belongs here rather than in unit-tests.yml because it is static
# analysis of source text, like everything else in this job, and needs
# no toolchain. It blocks: an untagged bare cosine is a defect by the
# invariant's own wording, not a warning.
- name: AR-024 — no bare cosine outside a recorded exception
run: python3 scripts/ci/check_raw_cosine.py
- name: Check modified files for traces
if: github.event_name == 'pull_request'
run: |
.gitea/workflows/unit-tests.yml
-140
View File
@@ -1,140 +0,0 @@
name: Unit tests
# TRACES: DP-007 | PR-004
#
# The tier the verification strategy is built on, finally executing.
#
# docs/requirements.md describes a four-tier plan in which T1 (functor unit)
# and T2 (replay) are "the only tiers that can exist in CI at all", and the
# traceability gate reports a CI-scope coverage fraction over exactly those
# tiers. Until this workflow existed, nothing ran them: "covered" meant a
# TRACES tag was present in a file, not that any test had been executed. That
# is the same failure mode as counting a test that cannot run, one level up,
# and the gate cannot detect it because a tag is all it can see.
#
# The runner is an Intel N100 with no discrete GPU. Nothing here calls a model:
# T1 constructs node functors directly, and T2 replays a precomputed HDF5 dump.
# T3 (ORT CPU smoke) and T4 (GPU) are deliberately absent -- the embedder is
# ~930 ms/frame on this hardware, so a 77 s clip at 5 fps would be six minutes
# of inference alone.
on:
push:
branches:
- main
- master
- develop
pull_request:
branches:
- main
- master
- develop
jobs:
unit-tests:
runs-on: linux/amd64
name: Build and run the GPU-free suite
# Pinned by tag, never `latest`, so rebuilding the image cannot silently
# change what a previous green build meant. Bumping the dependency set means
# bumping the tag in scripts/ci/build_builder_image.sh AND here, in one
# commit -- see that script's header.
container:
image: gitea.tourolle.paris/dtourolle/sae-builder-cpu:v1
steps:
- name: Checkout repository
uses: actions/checkout@v4
with:
fetch-depth: 0
# KPN is a submodule and the pipeline does not build without it.
#
# NOTE: this checks out the commit this repo PINS, which is the whole
# point and is also the first thing this job will disagree with a
# developer about. A local KPN working copy that is ahead of
# origin/master builds and passes here while CI builds something else
# entirely; the AR-004 evidence in docs/requirements.md was gathered
# that way. If this job fails on tests that pass locally, check
# `git -C external/KPN log origin/master..HEAD` before suspecting the
# tests.
# LFS is deliberately NOT fetched: SAE_MODELS_DIR is baked into the
# binary as a path string and nothing in T1/T2 opens a model file, so
# pulling ~hundreds of MB of ONNX would cost the job everything and
# buy it nothing.
submodules: recursive
lfs: false
- name: Assert the builder image is the pinned one
run: |
set -e
echo "builder=$SAE_BUILDER version=$SAE_BUILDER_VERSION"
echo "ort=$SAE_ORT_VERSION opencv=$SAE_OPENCV_VERSION"
# The image reports its own tag. A mismatch means the `container:`
# line above and the image that actually landed disagree, which is
# exactly the drift the pinning exists to prevent -- so it fails the
# job rather than building against an unknown toolchain.
[ "$SAE_BUILDER_VERSION" = "v1" ] || {
echo "image reports version '$SAE_BUILDER_VERSION', workflow pins v1" >&2
exit 1
}
- name: Fetch replay fixtures
env:
GITEA_TOKEN: ${{ secrets.GITEA_TOKEN }}
# bash, not sh: the script declares #!/bin/bash and uses `set -o
# pipefail` and arrays, which dash does not have.
run: bash scripts/artifacts/pull_artifacts.sh replay-fixtures latest
# pull_artifacts.sh warns and continues when a package version is missing,
# which is right for a developer pulling one artifact of several and wrong
# here. A T2 test whose fixture never arrived must not look like a pass:
# the dumps are the entire input to the replay tier, and VR-002's claim is
# that replay drives the real nodes over real data.
- name: Verify the fixtures actually arrived
run: |
set -e
missing=0
for f in tests/fixtures/dumps/superhero.h5; do
if [ -s "$f" ]; then
echo " ok: $f ($(wc -c < "$f") bytes)"
else
echo " MISSING: $f" >&2
missing=1
fi
done
if [ "$missing" -ne 0 ]; then
echo "" >&2
echo "Replay fixtures are absent, so the T2 tier cannot run." >&2
echo "They are not in git (tests/fixtures/dumps/.gitignore) -- they" >&2
echo "live in the Gitea generic package registry and are pulled by" >&2
echo "the step above, which needs GITEA_TOKEN to resolve 'latest'." >&2
exit 1
fi
- name: Configure
run: |
set -e
# SAE_GEMM_BACKEND defaults to ROCM and the auto-detect prefers a GPU
# backend where it finds one; CPU is stated explicitly so this job
# cannot start depending on what happens to be installed on the runner.
# The CPU kernel is OpenBLAS in this image (tests/CMakeLists.txt fails
# the configure if it is not), so the suite exercises the kernel the
# CPU release actually ships.
cmake -S . -B build \
-DCMAKE_BUILD_TYPE=Release \
-DSAE_BUILD_TESTS=ON \
-DSAE_GEMM_BACKEND=CPU
- name: Build the test suite
run: cmake --build build --target sae_tests --parallel
- name: Run the tests
run: ctest --test-dir build --output-on-failure
- name: Save test output
if: always()
uses: actions/upload-artifact@v3
with:
name: unit-test-results
path: build/Testing/
retention-days: 30
.gitignore
-1
View File
@@ -1,6 +1,5 @@
# Build
build/
build-*/
cmake-build-*/
CMakeCache.txt
CMakeFiles/
CMakeLists.txt
+9 -40
View File
@@ -55,13 +55,6 @@ set_property(CACHE SAE_GEMM_BACKEND PROPERTY STRINGS ROCM CUDA CPU)
# default so ROCm/CPU builds don't reference unavailable EPs.
option(SAE_ORT_TRT_EP "ORT backend: enable TensorRT/CUDA execution providers" OFF)
# AR-026/AR-027: the CPU GEMM path is backed by OpenBLAS, and its absence is a
# configure error rather than a silent downgrade to the scalar loop. Declared at
# top level because the unit-test target compiles the CPU kernel regardless of
# which backend the main build selected, and both must make the same choice.
option(SAE_ALLOW_SCALAR_GEMM
"Permit the scalar-loop GEMM fallback when OpenBLAS is absent" OFF)
# Back-compat: a legacy -DSAE_WITH_TRT=ON/OFF seeds the new vars (ON⇒TRT+CUDA,
# OFF⇒ORT+ROCM) unless the user set them explicitly.
if(DEFINED SAE_WITH_TRT)
@@ -160,16 +153,10 @@ if(SAE_GEMM_BACKEND STREQUAL "CPU")
target_include_directories(gemm_backend PRIVATE src)
target_compile_definitions(gemm_backend PRIVATE SAE_GEMM_CPU)
# AR-026/AR-027: the CPU path is backed by OpenBLAS, and that is REQUIRED
# rather than opportunistic. The CPU backend is what CI (no GPU) and the cpu
# builder image actually run, so a silent fall back to the scalar loop means
# AR-027 is measured — or worse, believed — on a path no release uses. A
# missing dependency should stop the build and name itself, not degrade into
# a slower answer nobody notices.
#
# The scalar loop survives as the correctness oracle the two backends are
# diffed against; -DSAE_ALLOW_SCALAR_GEMM=ON is how you ask for it, which
# keeps that an explicit, visible choice.
# AR-026/AR-027: back the CPU path with OpenBLAS when present. Optional, so
# the build gains no hard dependency — but without it the fallback is a
# scalar loop, which does not hold up against a library-scale gallery, and
# the CPU path is exactly what CI (no GPU) and the cpu builder image use.
find_package(PkgConfig QUIET)
if(PkgConfig_FOUND)
pkg_check_modules(OPENBLAS QUIET openblas)
@@ -179,16 +166,9 @@ if(SAE_GEMM_BACKEND STREQUAL "CPU")
target_compile_definitions(gemm_backend PRIVATE SAE_GEMM_CBLAS)
target_include_directories(gemm_backend PRIVATE ${OPENBLAS_INCLUDE_DIRS})
target_link_libraries(gemm_backend PRIVATE ${OPENBLAS_LINK_LIBRARIES})
elseif(SAE_ALLOW_SCALAR_GEMM)
message(WARNING "GEMM backend: CPU scalar fallback (SAE_ALLOW_SCALAR_GEMM=ON) — "
"correct, but slow on a large gallery. Do not measure AR-027 here.")
else()
message(FATAL_ERROR
"OpenBLAS not found, and the CPU GEMM backend requires it (AR-026/AR-027).\n"
" Install it: Fedora dnf install openblas-devel\n"
" Arch pacman -S openblas\n"
" Debian apt install libopenblas-dev\n"
" Or build the scalar fallback deliberately: -DSAE_ALLOW_SCALAR_GEMM=ON")
message(WARNING "GEMM backend: CPU scalar fallback — OpenBLAS not found. "
"Correct, but slow on a large gallery (AR-027).")
endif()
elseif(SAE_GEMM_BACKEND STREQUAL "CUDA")
find_library(CUBLAS_LIB cublas
@@ -322,22 +302,11 @@ nanobind_add_module(sae_embed src/python_bindings.cpp)
target_link_libraries(sae_embed PRIVATE sae_gallery)
# ── sae_kpn — Python module: run the real downstream nodes over dumped embeddings ─
# Assembles face_tracker/identity_matcher/frame_annotation in a Python-driven KPN
# Assembles face_tracker/identity_matcher/scene_tracker in a Python-driven KPN
# network (KPN_BUILD_PYTHON is enabled per-TU inside the .cpp). Powers the
# threshold-sweep optimizer in scripts/optimizer/.
#
# TRACES: VR-011 | PR-002
# ON again. It was OFF for one commit because it had not compiled since the
# AR-007/AR-008 tracker redesign -- the binding built FaceTrackerFunc from a
# Config alone, and the tracker had required a registry and a calibration since.
# VR-011 replaced the three per-node factories with one `add_pipeline` that
# builds the chain in main.cpp's order, which is the only order that satisfies
# those dependencies, so the failure mode cannot recur from Python.
option(SAE_BUILD_KPN_BINDINGS "Build the sae_kpn Python module" ON)
if(SAE_BUILD_KPN_BINDINGS)
nanobind_add_module(sae_kpn src/kpn_bindings.cpp)
target_link_libraries(sae_kpn PRIVATE sae_gallery)
endif()
nanobind_add_module(sae_kpn src/kpn_bindings.cpp)
target_link_libraries(sae_kpn PRIVATE sae_gallery)
# ── sae_audio — Python module: the v1 audio signature (IR-004) ────────────────
# Compiles audio_signature.cpp directly and links only FFmpeg, rather than
Dockerfile.builder-cpu
-342
View File
@@ -1,342 +0,0 @@
# sae-builder-cpu — the CI build image
#
# TRACES: DP-007 | PR-004
#
# Build/push: scripts/ci/build_builder_image.sh --push
# Consumed by: .gitea/workflows/unit-tests.yml (pinned by tag, never :latest)
# Docs: docs/ci-image.md
#
# This is the CPU corner of the DP-008 builder matrix and the DP-007 CI image at
# the same time — one artifact, two uses. The CUDA and ROCm siblings differ only
# in the accelerator stack layered on top of this dependency set.
#
# CI runs on an Intel N100 with no discrete GPU. Everything here is chosen so
# that `-DSAE_INFERENCE_BACKEND=ORT -DSAE_GEMM_BACKEND=CPU -DSAE_BUILD_TESTS=ON`
# configures, builds and runs without a GPU, without a model, and without
# reaching GitHub.
# ─── Base image ──────────────────────────────────────────────────────────────
#
# Chosen for the OLDEST glibc to be supported, not for recency. A binary built
# in a container runs against the *host's* glibc; glibc is backward compatible
# but not forward, so the build base sets the floor for every machine DP-008's
# binaries can ever run on. Building on a newer base than the oldest supported
# host produces the classic `GLIBC_2.xx not found` failure at load time.
#
# Debian 12 "bookworm" = glibc 2.36 (Aug 2022). What that floor covers:
#
# Distro glibc Covered?
# Arch / CachyOS (rolling) 2.41+ yes
# Fedora 37 and later 2.36+ yes ← DP-005's targets are Fedora+Arch
# Debian 12 / 13 2.36+ yes
# Ubuntu 24.04 LTS 2.39 yes
# Ubuntu 22.04 LTS 2.35 NO
# RHEL / Rocky / Alma 9 2.34 NO
# Debian 11 2.31 NO
#
# The three misses are accepted deliberately: DP-005 puts Debian/Ubuntu out of
# installer scope and names Fedora + Arch as the supported distros, and every
# supported Fedora is 2.36 or newer. Going lower costs the toolchain rather than
# buying reach — Debian 11 ships GCC 10 (incomplete C++20) and Python 3.9, which
# has no `tomllib` and therefore cannot read the traceability gate's
# traceability.toml.
#
# Escape hatch, recorded now so it is not rediscovered under pressure: if the
# floor must drop to glibc 2.28 (RHEL 8 / manylinux_2_28 — the same baseline the
# ONNX Runtime and PyTorch wheels target), the move is a Rocky 8 base plus
# gcc-toolset-13, and OpenCV/FFmpeg/HDF5 all leave apt for source or
# EPEL/RPM Fusion. That is a different image, not a flag on this one.
#
# Not a glibc problem but worth stating: the binaries this image produces also
# link OpenCV, FFmpeg and HDF5 shared objects by soname. Making a *portable*
# release binary (DP-008) is a separate question from the glibc floor, and is
# answered by static linking or bundling, not by the base image.
FROM debian:12-slim
# Pins. Every version this image installs from source is an ARG so a rebuild is
# a one-line diff and `docker history` records what a given tag actually holds.
#
# ORT 1.28.0 and OpenCV 5.0.0 match the developer machine, so CI and local
# builds exercise the same libraries rather than merely similar ones.
# Catch2 / nlohmann_json / nanobind match the FetchContent pins in
# CMakeLists.txt:248 and tests/CMakeLists.txt:12 exactly — a vendored copy at a
# different version would be a silent divergence, not a convenience.
ARG ORT_VERSION=1.28.0
ARG OPENCV_VERSION=5.0.0
ARG CATCH2_VERSION=v3.5.3
ARG NLOHMANN_JSON_VERSION=v3.11.3
ARG NANOBIND_VERSION=v2.4.0
# Stamped so a build can prove which image it ran in, and so a green tick can be
# traced back to a specific dependency set. See the "Confirm the builder image"
# step in .gitea/workflows/unit-tests.yml.
ARG IMAGE_TAG=dev
ENV SAE_BUILDER=cpu \
SAE_BUILDER_VERSION=${IMAGE_TAG} \
SAE_ORT_VERSION=${ORT_VERSION} \
SAE_OPENCV_VERSION=${OPENCV_VERSION} \
DEBIAN_FRONTEND=noninteractive
# ─── System dependencies ─────────────────────────────────────────────────────
#
# One layer, ordered by why it is here rather than alphabetically.
RUN apt-get update && apt-get install -y --no-install-recommends \
# Toolchain. bookworm's default gcc is 12.2 — enough for the C++20 the
# project sets unconditionally (CMakeLists.txt:4). cmake is 3.25, above the
# 3.21 minimum. Ninja because the N100 has four cores and every second of
# build scheduling shows.
build-essential \
cmake \
ninja-build \
pkg-config \
git \
ca-certificates \
curl \
# Gitea's act_runner executes JS actions (actions/checkout, upload-artifact)
# with the `node` found *inside* the container. Without this the job cannot
# even check the repository out. Same reason as the kpnpp-builder image.
nodejs \
# HDF5 with the C++ API: galleries are HDF5-native and it is also the VR-001
# dump format. find_package(HDF5 COMPONENTS CXX) at CMakeLists.txt:273.
libhdf5-dev \
# FFmpeg decode. swresample is on this list deliberately: the audio
# signature (IR-004) downmixes to mono and resamples to 11025 Hz, and
# tests/test_audio_signature.cpp decodes the golden FLAC fixture, so the
# test build needs it as much as the main build does.
libavformat-dev \
libavcodec-dev \
libavutil-dev \
libswscale-dev \
libswresample-dev \
# OpenBLAS — required here, not optional. CI has no GPU, so SAE_GEMM_BACKEND
# =CPU is the only path it ever exercises, and without OpenBLAS the CPU GEMM
# falls back to a scalar loop that does not scale against a library-sized
# gallery (AR-027). The build only *warns* when it is missing so a developer
# without it still gets a working tree; the image must never be that case.
# Both the main build (CMakeLists.txt:162) and the test target
# (tests/CMakeLists.txt:42) discover it through pkg-config `openblas`.
libopenblas-dev \
# Python: the build itself needs the interpreter and headers
# (find_package(Python COMPONENTS Interpreter Development.Module) at
# CMakeLists.txt:254, for the nanobind modules). numpy/h5py/scipy are for
# the Python-side tooling — fixture generation, replay, validation scripts.
# From apt rather than pip: bookworm marks the environment externally
# managed (PEP 668), and apt's h5py is already linked against the same
# libhdf5 installed above. bookworm's python3 is 3.11, which has tomllib —
# the traceability gate needs it to read traceability.toml.
python3 \
python3-dev \
python3-numpy \
python3-h5py \
python3-scipy \
# Image codecs for the OpenCV build below. Without these OpenCV silently
# builds an imgcodecs that cannot read a JPEG, which fails at run time in a
# gallery build rather than at compile time here.
libjpeg62-turbo-dev \
libpng-dev \
libtiff-dev \
libwebp-dev \
libopenjp2-7-dev \
zlib1g-dev \
&& rm -rf /var/lib/apt/lists/*
# Fail the image build, not the CI run, if OpenBLAS or swresample are not
# discoverable the way CMakeLists.txt discovers them. An image that ships
# libopenblas but no openblas.pc would compile the scalar fallback in silence.
RUN set -eux; \
pkg-config --exists openblas; \
echo "openblas $(pkg-config --modversion openblas)"; \
pkg-config --exists libswresample; \
echo "swresample $(pkg-config --modversion libswresample)"
# ─── ONNX Runtime, CPU provider only ─────────────────────────────────────────
#
# The official prebuilt linux-x64 tarball is the CPU build: no CUDA, no
# TensorRT, no ROCm execution providers. That is the whole requirement here —
# excluding the GPU providers is not a size optimisation, it is the point.
#
# Verified against the 1.28.0 tarball: the shared object's highest versioned
# symbol requirement is GLIBC_2.27 / GLIBCXX_3.4.21, well under this base's
# 2.36, so ORT does not raise the floor set above.
#
# Installed to /usr/local/{lib,include/onnxruntime} because CMakeLists.txt
# includes <onnxruntime/onnxruntime_cxx_api.h> and needs the *parent* of that
# directory on the include path (CMakeLists.txt:108-113).
#
# CI never calls a model — the embedder measures ~930 ms/frame on this CPU
# provider — so ORT is present to satisfy the link, not to run inference.
RUN set -eux; \
curl -fsSL -o /tmp/ort.tgz \
"https://github.com/microsoft/onnxruntime/releases/download/v${ORT_VERSION}/onnxruntime-linux-x64-${ORT_VERSION}.tgz"; \
mkdir -p /tmp/ort; \
tar -xzf /tmp/ort.tgz -C /tmp/ort --strip-components=1; \
cp -a /tmp/ort/lib/libonnxruntime.so* /usr/local/lib/; \
mkdir -p /usr/local/include/onnxruntime; \
cp -a /tmp/ort/include/. /usr/local/include/onnxruntime/; \
ldconfig; \
rm -rf /tmp/ort /tmp/ort.tgz; \
test -f /usr/local/include/onnxruntime/onnxruntime_cxx_api.h
# ─── OpenCV 5, from source ───────────────────────────────────────────────────
#
# This is the reason the image is prebuilt at all. CMakeLists.txt:25 probes for
# OpenCV 5 first and falls back to 4; the branch targets 5, which no Debian
# release ships (bookworm has 4.6), and building it inside every CI run would
# dominate the run on an N100.
#
# BUILD_LIST is exactly the seven components find_package asks for
# (CMakeLists.txt:25-29) — OpenCV resolves their internal dependencies itself.
# Everything else is off: tests, samples, Java/Python bindings, and the apps.
#
# No GUI backend. highgui still builds (find_package REQUIREs the component) but
# with a stub — CI never calls imshow, and pulling GTK/Qt into a headless build
# image buys nothing. scene_preview is a developer tool, not a CI target.
#
# CUDA/cuDNN explicitly off: DP-007 excludes the GPU stack outright.
#
# The source tree and build tree are removed in the same layer, so the ~3 GB of
# intermediates cost nothing in the published image.
RUN set -eux; \
curl -fsSL -o /tmp/opencv.tar.gz \
"https://github.com/opencv/opencv/archive/refs/tags/${OPENCV_VERSION}.tar.gz"; \
mkdir -p /tmp/opencv-src; \
tar -xzf /tmp/opencv.tar.gz -C /tmp/opencv-src --strip-components=1; \
cmake -S /tmp/opencv-src -B /tmp/opencv-build -G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_INSTALL_PREFIX=/usr/local \
-DBUILD_LIST=core,imgproc,imgcodecs,videoio,dnn,objdetect,highgui \
-DBUILD_SHARED_LIBS=ON \
-DBUILD_TESTS=OFF \
-DBUILD_PERF_TESTS=OFF \
-DBUILD_EXAMPLES=OFF \
-DBUILD_DOCS=OFF \
-DBUILD_opencv_apps=OFF \
-DBUILD_JAVA=OFF \
-DBUILD_opencv_python3=OFF \
-DWITH_FFMPEG=ON \
-DWITH_GTK=OFF \
-DWITH_QT=OFF \
-DWITH_OPENGL=OFF \
-DWITH_CUDA=OFF \
-DWITH_CUDNN=OFF \
-DOPENCV_GENERATE_PKGCONFIG=ON \
-DCMAKE_INSTALL_RPATH=/usr/local/lib; \
cmake --build /tmp/opencv-build --parallel; \
cmake --install /tmp/opencv-build; \
ldconfig; \
rm -rf /tmp/opencv-src /tmp/opencv-build /tmp/opencv.tar.gz
# ─── Vendored dependencies: Catch2, nlohmann/json, nanobind ──────────────────
#
# All three are FetchContent'ed by the build today, which makes every CI run
# depend on GitHub being reachable — a network outage would present as a code
# failure. Baking them in removes that dependency entirely.
#
# Catch2 is *installed*, so tests/CMakeLists.txt:6 `find_package(Catch2 3 QUIET)`
# succeeds and the FetchContent fallback is never reached. Its source is kept as
# well so the override below can cover the case where find_package somehow does
# not fire.
#
# nanobind must be cloned with submodules: its `ext/robin_map` is a git
# submodule, and a GitHub source tarball does not contain it. This is the one
# dependency where "download the tarball" produces a tree that configures and
# then fails to compile.
RUN set -eux; \
mkdir -p /opt/vendor; \
git clone --depth 1 --branch "${NLOHMANN_JSON_VERSION}" \
https://github.com/nlohmann/json.git /opt/vendor/nlohmann_json; \
git clone --depth 1 --branch "${NANOBIND_VERSION}" --recurse-submodules \
https://github.com/wjakob/nanobind.git /opt/vendor/nanobind; \
git clone --depth 1 --branch "${CATCH2_VERSION}" \
https://github.com/catchorg/Catch2.git /opt/vendor/Catch2; \
cmake -S /opt/vendor/Catch2 -B /tmp/catch2-build -G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_INSTALL_PREFIX=/usr/local \
-DBUILD_TESTING=OFF; \
cmake --build /tmp/catch2-build --parallel; \
cmake --install /tmp/catch2-build; \
rm -rf /tmp/catch2-build; \
find /opt/vendor -maxdepth 2 -name .git -exec rm -rf {} +; \
ldconfig
# The initial-cache script the build is configured with. It lives in the image,
# not in the workflow, so the vendor paths have exactly one owner: move a
# directory here and no consumer needs editing.
#
# FETCHCONTENT_FULLY_DISCONNECTED=ON is the load-bearing line. With it, any
# FetchContent dependency that is *not* covered by an override above is a hard
# configure error instead of a silent download — so "this build does not touch
# GitHub" is enforced by the build system rather than asserted in a comment.
RUN set -eux; \
printf '%s\n' \
'# Baked into sae-builder-cpu. Use with: cmake -C /opt/vendor/vendored-deps.cmake ...' \
'# TRACES: DP-007' \
'set(FETCHCONTENT_SOURCE_DIR_NLOHMANN_JSON "/opt/vendor/nlohmann_json" CACHE PATH "vendored in the CI image")' \
'set(FETCHCONTENT_SOURCE_DIR_NANOBIND "/opt/vendor/nanobind" CACHE PATH "vendored in the CI image")' \
'set(FETCHCONTENT_SOURCE_DIR_CATCH2 "/opt/vendor/Catch2" CACHE PATH "vendored in the CI image")' \
'set(FETCHCONTENT_FULLY_DISCONNECTED ON CACHE BOOL "no CI build may fetch from the network")' \
> /opt/vendor/vendored-deps.cmake; \
cat /opt/vendor/vendored-deps.cmake
# ─── Self-check ──────────────────────────────────────────────────────────────
#
# Run the project's own dependency discovery — the same find_package and
# pkg_check_modules calls CMakeLists.txt makes — against this image, at image
# build time. An image that cannot satisfy them should fail here, loudly, once,
# rather than in every CI run that pulls it.
#
# Deliberately not a build of the project: the image must be buildable without
# the repository, and the repository's own configure step is what CI is for.
RUN set -eux; \
mkdir -p /tmp/selfcheck; \
printf '%s\n' \
'cmake_minimum_required(VERSION 3.21)' \
'project(sae_image_selfcheck LANGUAGES CXX)' \
'set(CMAKE_CXX_STANDARD 20)' \
'set(CMAKE_CXX_STANDARD_REQUIRED ON)' \
'find_package(OpenCV 5 REQUIRED COMPONENTS core imgproc imgcodecs videoio dnn objdetect highgui)' \
'message(STATUS "OpenCV ${OpenCV_VERSION}")' \
'find_package(HDF5 REQUIRED COMPONENTS CXX)' \
'message(STATUS "HDF5 ${HDF5_VERSION}")' \
'find_package(Catch2 3 REQUIRED)' \
'message(STATUS "Catch2 ${Catch2_VERSION}")' \
'find_package(Python 3.8 REQUIRED COMPONENTS Interpreter Development.Module)' \
'find_package(PkgConfig REQUIRED)' \
'pkg_check_modules(AVFORMAT REQUIRED libavformat)' \
'pkg_check_modules(AVCODEC REQUIRED libavcodec)' \
'pkg_check_modules(AVUTIL REQUIRED libavutil)' \
'pkg_check_modules(SWSCALE REQUIRED libswscale)' \
'pkg_check_modules(SWRESAMPLE REQUIRED libswresample)' \
'pkg_check_modules(OPENBLAS REQUIRED openblas)' \
'find_library(ORT_LIB onnxruntime REQUIRED HINTS /usr/lib /usr/local/lib)' \
'find_path(ORT_INCLUDE onnxruntime_cxx_api.h PATH_SUFFIXES onnxruntime' \
' HINTS /usr/include/onnxruntime /usr/local/include/onnxruntime /usr/local/include REQUIRED)' \
'message(STATUS "ORT ${ORT_LIB} / ${ORT_INCLUDE}")' \
> /tmp/selfcheck/CMakeLists.txt; \
cmake -S /tmp/selfcheck -B /tmp/selfcheck/build -G Ninja; \
rm -rf /tmp/selfcheck
# Python-side tooling the fixture and validation scripts import. Checked here so
# a missing wheel is an image failure rather than a mid-run traceback.
RUN python3 -c "import numpy, h5py, scipy; print('numpy', numpy.__version__, 'h5py', h5py.__version__, 'scipy', scipy.__version__)"
# ─── What is deliberately NOT here ───────────────────────────────────────────
#
# CUDA, TensorRT, ROCm, and the ORT GPU execution providers
# No GPU to use them. They belong to the sae-builder-cuda and
# sae-builder-rocm siblings (DP-008).
#
# The ONNX models
# Seven files, ~725 MB, in Git LFS. T1/T2 tests are model-free by design
# (tests/CMakeLists.txt:1-4), so the CI image needs none of them, and
# baking them in would inflate the image roughly tenfold to serve the T3
# smoke tests alone. Those pull the model they need via LFS in a separate
# job. The CI workflow checks out with LFS off for the same reason.
#
# The repository
# Nothing from the source tree is COPYed in. The image is a toolchain, and
# a toolchain that embeds the code it builds has to be rebuilt whenever the
# code changes — which is exactly the per-run cost this image exists to
# avoid.
WORKDIR /src
docs/SPEC.md
+113 -240
View File
@@ -128,43 +128,10 @@ Two consequences worth stating:
depends on timing. The same command run twice can produce different dumps, and
a golden fixture cannot be built on that.
**Current:** fixed in KPN. Node data outputs *park* on a full channel — the
value is held in a one-slot buffer, the worker is released, and the channel's
space callback resubmits the node once the consumer drains. That replaced
`push_blocking`, which slept inside the push and, with one thread per node,
stopped that node draining its own input. Sentinels remain out-of-band so EOF
can always overtake a stalled data path. Verified on the same clip: 385 of 385
sampled frames written, zero drops, and two consecutive runs byte-identical
where previously they were not.
A later audit found the losslessness was still incomplete in three places, all
now closed and each pinned by a regression case in the KPN suite:
- **`FilterNode` and `RouterNode`** were the last data paths still using the
throwing `push()` with the exception swallowed. A full output discarded the
value, and that included the **EOF sentinel**. The decimator passes EOF by
predicate but its output is reliably full — the embedder is the slowest node
in the chain — so the token was discarded, nothing downstream shut down, and
the run had to be killed. This was the wedge.
- **The sentinel could arrive ahead of a value still queued behind it.** `pop()`
observed the ring empty and then took the sentinel; a producer can push a
value *and* publish the sentinel inside that window, so a consumer treating
EOF as a hard stop loses the tail.
- **Two firings of one node could overlap**, because the submit gate was
released before the firing had finished with the node's state. That breaks the
one-slot park itself: a parked value can be overwritten by the other firing,
with no drop recorded anywhere.
**New constraint:** a channel carries at most one undelivered sentinel. A second
offered before the first is taken is refused and reported, never queued and
never overwritten — two control tokens on one channel means the stream ended
twice. Single-shot EOF is what everything does today; this becomes live the
moment a pipeline is reused for a second input.
**Consequence:** a lossless decimator is a backpressure point, not a relief
valve. The source now throttles to the face branch rather than quietly thinning
it. That is what this requirement asks for, but it changes the shape of a loaded
run and has not yet been benchmarked.
**Current:** fixed in KPN — node data outputs use `push_blocking`, sentinels
remain out-of-band so EOF can always overtake a stalled data path. Verified on
the same clip: 385 of 385 sampled frames written, zero drops, and two
consecutive runs byte-identical where previously they were not.
It also ran *faster* (29 s → 17 s). A dropped frame has already cost its decode,
and the overflow exception cost more — so the lossy path was paying for work it
@@ -281,31 +248,29 @@ the same response.
- **Size** — already AR-002, floor at 40×40 px in original resolution, measured
end to end by VR-013. It is the precedent for the other two: the
threshold was *located*, not chosen.
- **Sharpness** — motion blur and soft focus destroy the high-frequency detail
the embedder keys on, and unlike size they leave the bounding box looking
perfectly healthy. Measured on the **112×112 aligned crop**, not the raw box:
the crop is already scale-normalised, so a measure taken there cannot silently
re-measure face size and double-count it against AR-002.
- **Sharpness** — motion blur and optical defocus destroy the high-frequency
detail the embedder keys on, and unlike size they leave the bounding box
looking perfectly healthy. Measured on the **112×112 aligned crop**, not the
raw box.
The measure is the **variance of the Laplacian divided by the variance of the
crop** — `crop_sharpness()`, dimensionless. The division is the part that
earns its place: a raw Laplacian variance, the textbook measure, scales with
the square of image contrast, so a dim scene reads as soft and a graded-up one
as sharp, and VR-012 would locate a different knee in every film. That is
AR-024's objection to the raw cosine in another metric. Normalised, the axis
means the same thing everywhere, which is the precondition for a single knee
existing at all.
An earlier version of this clause argued the crop is scale-normalised and so a
measure taken there "cannot re-measure face size and double-count it against
AR-002". **That reasoning is wrong and VR-012 measured it wrong.** The
normalisation is geometric, not informational: a 40 px face upscaled into the
canonical frame genuinely carries less high-frequency content than a 400 px
one downscaled into it, so every candidate measure *does* respond to source
size. What the crop yields is **effective resolution in canonical space** —
the union of "was small" and "was blurred", not blur alone.
Read spectrally it is `E[|ω|⁴]` under the crop's own energy distribution, so
the blur ladder is monotone by construction rather than by fitting: Gaussian
blur multiplies that distribution by `e^{-σ²|ω|²}`, which can only move mass
downward. Two consequences follow from the same identity and are recorded on
the function: it needs the low-frequency mass real images have (on a
flat-spectrum synthetic an anisotropic smear makes it *rise*, because the
surviving perpendicular detail really is as fine as before), and it conflates
focus with intrinsic texture, so a bearded face outscores a smooth one at equal
focus. Both are true of every no-reference sharpness measure, and both are
reasons AR-028 carries the number rather than thresholding on it.
The conclusion survives, for a better reason. VR-012 sorted its grid by
measured sharpness and found the six cells at effectively identical sharpness
(0.0003–0.0005) spanning **15.3% to 91.0% TPI**, ordered entirely by source
size. Sharpness is therefore not a sufficient statistic for identity loss: a
scalar keyed on high-frequency energy cannot separate *attenuated* high
frequencies from *destroyed* spatial sampling, because blur preserves
mid-frequency facial geometry exactly while downsampling destroys it. The two
axes are not redundant and neither substitutes for the other — which is what
"not collapsed into one scalar" above now rests on.
- **Visibility** — extreme pose or occlusion means the face presents fewer of the
features the embedding assumes are present. The measure is the **residual of
the AR-005 alignment fit**: the RMS landmark error, in canonical 112×112
@@ -386,43 +351,63 @@ hand-chosen cutoff on an uncalibrated measure is the same unfalsifiable magic
number AR-024 retired for similarity, and it would fail the same way: meaning
something different for every detector, every embedder and every film.
**Current:** all three axes are measured and carried, and the vector reaches the
dump. `FaceAlignerFunc` is where it is filled in, because both measured axes fall
out of work the warp already does: visibility is the residual from
`estimate_alignment()`, and sharpness is `crop_sharpness()` on the 112×112 crop
the node has just produced. Size stays `bbox` — deliberately not copied into a
field of its own, since that would hold the same quantity in two coordinate
spaces and the copy is the one that drifts. No face is admitted unscored, so a
negative value downstream is a bug rather than a poor-quality face. The
degenerate-fit case is still dropped — it has no crop and no fit to score — but
is now **counted** and reported once at EOF instead of vanishing.
**A discount curve on sharpness must be flat, then steep.** VR-012 measured the
response as a cliff rather than a gradient: Gaussian sigma up to 1.5 costs under
1.5 points of TPI in every cell — at 16 px it is very slightly *positive*,
smoothing upscale artifacts — sigma 2 costs 1–3, and the 2→3 step costs 7–19. A
linear or sigmoid discount over the measure would penalise the whole flat region
where blur demonstrably costs nothing.
`sharpness` and `alignment_residual` are written to the VR-001 dump as per-face
columns parallel to `confidence`, taking the dump to `schema_version` 2. The bump
is not for readers — both sides check by name, and a v1 dump still replays — but
so that a consumer of the vector can tell *never scored* from *scored zero*,
which is a real reading on this axis. Nothing yet *consumes* any of it.
**Which blur is modelled is a first-order decision, not a detail.** VR-012 swept
three families at matched per-axis PSF spread, and at σ=3 px on a 112 px face
they cost 9%, 18% and **53%** error for Gaussian, motion and optical defocus
respectively. Defocus is the destructive one because its disc PSF has a jinc
transfer function with **exact zeros** — bands annihilated rather than
attenuated — where a Gaussian merely rolls off. It is also the case AR-002
cannot catch, since a defocused face is large and confidently detected. Any
future study that sweeps blur states its family and its justification; a
Gaussian-only sweep understated the effect by a factor of five and would have
retired this axis as not worth its cost.
**Gap:** three, in the order they block each other.
**The cost of blur is proportional to proximity to the decision boundary, not to
blur itself.** Sigma 3 costs −22.5 points at 24 px, but only −7.9 at 112 px
(margin to spare) and −8.3 at 16 px (already below threshold). This is why the
axes must combine multiplicatively in `EvidenceDiscounter` rather than each
gating independently.
1. **The fixtures do not carry the vector.** They are v1, and re-dumping needs a
GPU host (`scripts/make_fixtures.sh`), so until that runs VR-012 has recorded
data available in principle and none in hand.
2. **AR-030's discount does not exist.** The measure must reach
`EvidenceDiscounter` as the reliability term, multiplying the novelty weight
rather than replacing it.
3. **Two properties of the sharpness measure are recorded but unquantified on
real faces**, and both distort the low end of the axis, which is where a knee
would go. It is exactly contrast-invariant in the algebra, but the 8-bit
quantisation floor lands in the numerator, so a crop that is *dim and soft*
reads sharper than it is — on the synthetic ladder a half-contrast copy reads
0.9% high when sharp and 148% high at σ 2.5. Separately, `align_face` warps
with `BORDER_CONSTANT`, so a face crossing the frame edge brings a hard black
step into the crop, and a step edge is high-frequency; the normalisation
blunts this but does not remove it. Neither is corrected here. The candidate
fixes are a validity mask or a different border mode, and the second changes
what the embedder is fed (AR-011) — so VR-012 measures the size of each effect
on the dumped distribution first, and no correction is chosen before that.
**Sharpness discounts; it must never gate.** VR-012 tried the gate directly, as
a compute saving: skipping the embed below a sharpness threshold costs 15.1% of
true identifications to save 20% of the work, against the size filter's 4.7% at
16.7% — three times the damage, from a measure that needs the warped crop plus a
DFT where size is a bbox dimension available for free. The reason is a ceiling
no measure can beat: **at 112 px with defocus radius 6 — visually destroyed —
46.9% of faces still identify correctly, and rank-1 is still 94.8%.** Apparent
blur does not determine the outcome. The size filter wins only because smallness
destroys identity more completely than blur does (16 px succeeds 23.5% of the
time), and that asymmetry is the measured justification for the rule above:
**failing sharpness discounts the observation, failing size may drop it.**
**Current:** visibility is measured and carried — `estimate_alignment()` in
`src/face_utils.hpp` returns the residual alongside the transform, and
`FaceAlignerFunc` writes it to `DetectedFace::alignment_residual`. Sharpness is
measured: `assess_sharpness()` in `src/quality.hpp` returns five AR-029
candidates over a fixed 64×64 window on the face interior, and VR-012 has ranked
them — `var_laplacian` and `tenengrad` are disqualified as discounts (see
AR-029), leaving `hf_energy_ratio` as the only correctly-signed survivor. Size
is `min_face_px` (40, decoded-frame space — AR-002 still open). All three are
exposed to studies through `sae_embed`. Nothing yet *consumes* any of it: no
discount is applied, and `align_face()` still drops the degenerate-fit case
without counting it.
**Gap:** the discount itself, on every axis. Neither sharpness nor the residual
reaches `EvidenceDiscounter`, whose weight remains pure novelty — so a profile
or defocused view still moves a track's belief hardest when it deserves the
least trust. Neither reaches the VR-001 dump either, so VR-012 must still re-run
video rather than replay fixtures. VR-012's **pose half is not started**: the
AR-030 residual has no arm in the grid, so whether the 5-point proxy suffices or
a dedicated landmark model is needed remains open. And the sharpness result is
weak enough (best within-cell AUC 0.530) that whether AR-029 earns a discount at
all is still a judgement, not a measurement.
## AR-007, AR-008 — Tracking
@@ -516,7 +501,7 @@ Both feed AR-007 as **association hints**: they tell the tracker that spatial
continuity is broken and that association should weight embedding over IoU.
Neither ends a presence window (AR-012).
In dense mode the source decodes at `scene_decode_fps` (default 0 = native) and a
In dense mode the source decodes at `scene_decode_fps` (default 12) and a
decimator splits the stream: full-resolution sampled frames to the face pipeline,
downscaled dense frames to the scene detector
(`frame_source_node.hpp:63`). `sample_fps` is independent of this — the face
@@ -537,44 +522,31 @@ degrading what a single inference sees. A model run off-distribution produces
confident, plausible, wrong output, and the error is invisible without a study
that should not have been necessary.
Two places this was violated, both now closed:
Two places this is currently violated:
1. **`scene_decode_fps = 12` starved TransNetV2.** `kWindow` is 100 frames. At
native 25 fps that window spans ~4 s; at 12 fps it spanned ~8.3 s, so the
model saw roughly half-speed motion over twice the temporal context it was
trained on. **Requirement: feed TransNetV2 at the source's native frame
rate**, so a 100-frame window covers the duration the model expects. The
"tolerates ~12fps" note in `config.hpp` described a compromise, and the
recorded margin was consistent with it — a non-boundary baseline at ~0.50 with
1. **`scene_decode_fps = 12` starves TransNetV2.** `kWindow` is 100 frames. At
native 25 fps that window spans ~4 s; at 12 fps it spans ~8.3 s, so the model
sees roughly half-speed motion over twice the temporal context it was trained
on. **Requirement: feed TransNetV2 at the source's native frame rate**, so a
100-frame window covers the duration the model expects. The
"tolerates ~12fps" note in `config.hpp` describes a compromise, and the
recorded margin is consistent with it — a non-boundary baseline at ~0.50 with
real boundaries reaching only ~0.7+ is a compressed separation, not a healthy
one. **Done:** `scene_decode_fps` defaults to 0.
one.
2. **Hardcoded 25 fps in boundary dedup.** The node merged boundaries closer than
`0.04 s` — "~1 frame @25fps". **Requirement: derive this from the source's
actual frame rate. Done:** `SceneDetectorFunc::dedup_window_sec()` takes the
median of the frame intervals the detector was actually fed and halves it.
Half a frame rather than a whole one, because the only thing being merged is
one frame scored by two overlapping windows; two distinct frames are a full
interval apart and both have to survive.
The two are one change, not two. A native-rate stream is where the old constant
did the most damage — at 30 fps, 0.04 s is wider than a frame, so two cuts on
consecutive frames merged into one and the loss showed up nowhere: the file
simply had fewer boundaries.
2. **Hardcoded 25 fps in boundary dedup.** `scene_detector_node.hpp:138` merges
boundaries closer than `0.04 s` — "~1 frame @25fps". **Requirement: derive
this from the source's actual frame rate.**
Dense decode is the pipeline's cost driver, so (1) is not free. The cost is
accepted: the alternative is a boundary signal that steers association (AR-007) while
being quietly unreliable. `dense_scale` remains available as a spatial reduction,
since downscaling is a documented, understood degradation rather than a temporal
one the model has no defence against — and TransNetV2 downsamples to 48×27
regardless.
one the model has no defence against.
**Current:** histogram cut in the decoder; `scene_detector_node.hpp` for
TransNetV2, fed at native rate with a framerate-derived dedup window.
**Gap:** `scene_threshold` (0.60) is still the value picked against 12 fps input
and is now certainly wrong — VR-006 re-fits it, and until it does, boundary
recall at native rate is untuned rather than better. `--scene-detect` is
default-off despite now feeding association.
TransNetV2. **Gap:** native-rate dense decode; framerate-derived dedup;
`--scene-detect` is default-off despite now feeding association.
## AR-012 … AR-017 — Track-level identity propagation — **CHANGED BEHAVIOUR**
@@ -757,30 +729,12 @@ An embedding is admitted only if its similarity to one already in the store fall
admitting it risks poisoning the store.
A starting band of roughly **0.90–0.95** is the working estimate, to be tuned
(VR-007). Note this is deliberately conservative compared to the retired
`expand_novelty_sim` (0.55), which promoted embeddings *far* from the gallery —
(VR-007). Note this is deliberately conservative compared to the current
`expand_novelty_sim` (0.55), which promotes embeddings *far* from the gallery —
much more aggressive, and much more exposed to admitting the wrong person.
Both bounds must be expressed as calibrated probabilities, not raw cosines (AR-024).
The lower bound is asked twice. `admit` compares a newcomer against its
*closest* existing member, which a gradually drifting track can chain past: every
step inside the band while the endpoints are strangers — the shape a track-ID
collision takes over a slow pan. So the same bound is re-applied across **every
pair** in the store before promotion. One bound, two enforcement points; not a
second constant.
Novelty is deliberately **not** a threshold. The store's eviction policy orders
its members by similarity to the actor's existing references and drops the
best-recognised one, so novelty-seeking is a ranking with nothing to tune, and
the band's upper bound already refuses the redundant views at the door.
**Current:** implemented in `gallery/track_gallery.hpp` — `admit` at the door,
`store_coherence` at promotion, both bounds from `Config::expand_band_lo/hi`.
Refusals are counted (`band_rejected`).
**Gap:** the bounds themselves are unswept working values (VR-007).
### AR-019 — Expansion of known actors
When a track is owned (AR-012), its store is promoted into a **per-film, in-memory
@@ -881,14 +835,14 @@ natural unit for anonymous presence, should that be adopted (AR-012, TBD).
open (VR-007).
**Current:** `src/gallery/track_gallery.hpp` implements a per-track diversity
buffer with eviction biased to gallery-far poses, admission and promotion both
gated on the AR-018 band in probability space, cleared on `is_cut`. Wired at
`identity_matcher_node.hpp:227`, cleared at `:126`; the calibration is handed
over at `:114`.
buffer with eviction biased to gallery-far poses, promotion gated on
`expand_novelty_sim` / `expand_track_spread_max`, cleared on `is_cut`. Wired at
`identity_matcher_node.hpp:227`, cleared at `:126`.
**Gap:** all three quiet-signal conditions rather than only `is_cut`; and the
whole of AR-020 — the TBI queue, the deferred pass, and deferring output until it
completes.
**Gap:** the band rule of AR-018 replacing the current novelty/spread gates; all
three quiet-signal conditions rather than only `is_cut`; probability space
throughout (AR-024); and the whole of AR-020 — the TBI queue, the deferred pass, and
deferring output until it completes.
## AR-022 — Unidentified-track capture
@@ -1036,45 +990,21 @@ is the only viable formulation — a per-pair loop is orders of magnitude off.
**All similarity computation goes through the GEMM path**, with no exception
justified by "this set is small". Three call sites:
1. **Baked gallery** — GEMM (`sim_engine_->compute()`, backend from
`SAE_GEMM_BACKEND`). ✓
2. **Per-film annex** — was a **CPU loop**, justified in-comment by "tens of
embeddings". AR-018…AR-021 invalidated that assumption: every owned track
contributes, so the annex grows with cast size and film length. Now appended
to the gallery matrix rather than scored separately — promotions are pushed
into the engine's resident matrix (`ISimilarityEngine::append_rows`,
capacity doubling, device-to-device on the GPU backends) and `flat_actor_`
grows in lockstep, so one multiply covers baked and promoted references and
best-of-N is a single pass over one similarity column. ✓
1. **Baked gallery** — already GEMM (`sim_engine_->compute()`,
`identity_matcher_node.hpp:143`, backend from `SAE_GEMM_BACKEND`). ✓
2. **Per-film annex** — currently a **CPU loop**
(`identity_matcher_node.hpp:159-162`), justified in-comment by "tens of
embeddings". AR-018…AR-021 invalidates that assumption: every owned track now
contributes, so the annex grows with cast size and film length. It must move
into the GEMM path — appended to the gallery matrix, or a second multiply.
3. **Deferred TBI pass (AR-020)** — the most GEMM-friendly operation in the
pipeline: all TBI embeddings against the full gallery-plus-annex, offline,
operands resident, no streaming. One large multiply, not a loop over entries.
Not yet built; AR-020 owns it.
**Current:** 1 and 2 done. `TrackGallery` holds the annex as a contiguous
row-major matrix plus a parallel actor index, and hands newly promoted rows to
the matcher once per frame (`drain_promotions`), which is what call site 3 will
score against.
**Gap:** call site 3, gated on AR-020 existing at all.
This constrains AR-018…AR-021's implementation: the annex must be a **contiguous matrix**
with promotions appended, plus a parallel actor-index mapping — exactly the
`flat_emb_`/`flat_actor_` arrangement the baked gallery already uses.
**Ordering note.** Absorbing promotions is a once-per-frame step that runs after
every face in the frame has been scored, not mid-frame. Appending mid-frame would
invalidate the similarity pointer the matcher is still reading, and it also
removes an accidental dependence on face order within a frame: a promotion helps
subsequent frames, never the one that produced it, which is the semantics the
expansion store already documents.
**The CPU GEMM path requires OpenBLAS.** It is what CI and the cpu builder image
run, so a silent fall back to the scalar loop would mean AR-027 is measured — or
believed — on a path no release uses. Absence is a configure error; the scalar
loop survives as the correctness oracle, reachable only via
`-DSAE_ALLOW_SCALAR_GEMM=ON`.
### Scaling characteristics that must be known, not assumed
- **Throughput versus gallery size must be measured** (VR-008) and published. The
@@ -1653,10 +1583,7 @@ Persist pipeline state at the point where the expensive work ends.
per-frame index table (`face_offset`, `face_count`) pointing into them. Avoids
variable-length HDF5 types and reads straight into numpy.
- Stores per frame: `timestamp_sec`, `frame_idx`, `is_cut`, `is_scene_boundary`.
Per face: `embedding` [N,512], `bbox` [N,4], `landmarks` [N,10], `confidence`,
and from v2 the AR-028 quality vector — `sharpness` [N] and
`alignment_residual` [N]. Size, its third axis, is `bbox` and is not
duplicated.
Per face: `embedding` [N,512], `bbox` [N,4], `landmarks` [N,10], `confidence`.
- Invariants: embeddings unit-norm; `face_offset` contiguous; bboxes and
landmarks in **decoded-frame** pixels with `bbox_upscale` recorded alongside
(the dump is a faithful tap, so it does not transform what the tracker saw —
@@ -1667,16 +1594,10 @@ Persist pipeline state at the point where the expensive work ends.
Schema owned by [`scripts/optimizer/SCHEMA.md`](../scripts/optimizer/SCHEMA.md).
**Current:** C++ dump sink (`embedding_dump_node.hpp`, `dump_embeddings.cpp`),
read by `replay.py`. At `schema_version` 2, which AR-028 took it to by adding the
quality columns; readers on both sides check the datasets by name, so a v1 dump
still replays and reports the vector as unknown rather than as zero.
**Gap:** **AR-012 breaks the replay contract.** Track extents
read by `replay.py`. **Gap:** **AR-012 breaks the replay contract.** Track extents
are decided in the tracker, which is *downstream* of the dump — so a replay can
reproduce them, but only if the dump preserves everything the tracker needs.
Verify `landmarks`/`bbox`/`is_cut` suffice, and bump `schema_version` if not.
The committed fixtures are still v1, so they carry no quality vector until
`scripts/make_fixtures.sh` is re-run on a GPU host.
## VR-002 — Replay and sweep
@@ -1799,54 +1720,6 @@ round 1 seeding references that corrupt round 2.
"expansion helps live matching" from "expansion helps the second pass", which the
current all-or-nothing `expand_gallery` flag cannot distinguish.
## VR-015 — Per-node cost and bottleneck attribution
**Requirement: a run must be able to report where its time went, per node, and
which node is setting the pace.** Without it, optimisation is guesswork, and
worse than guesswork — the obvious number is wrong in a specific, repeatable
direction, so acting on it makes the pipeline slower.
**Why the obvious number is wrong.** KPN times a node across `fire_once`, which
wraps the functor *and* `push_outputs`. Under AR-004 a push parks on a full
downstream channel, so a node that is merely waiting bills that wait to itself.
On the SuperHero reference run (`docs/benchmark.md`) `frame_source` reported
`ema=141.899ms` per frame while its own decoder logged 12-18 ms: it was
backpressured, and the report named the *fastest* node in the graph as the most
expensive one. A second trap sits behind the first — `ema_exec_ms` is an
exponentially weighted average, so `frames × ema` is not a total; on a film whose
per-frame cost swings between crowd scenes and landscapes the two differ
substantially.
**Method.** Three measurements per node, none of which is sufficient alone:
| Measure | What it is | What it cannot tell you |
|---|---|---|
| `cpu_ms` | thread CPU time (`CLOCK_THREAD_CPUTIME_ID`) | GPU wait — a device-bound node looks idle |
| `exec_ms` | cumulative wall time inside the node | work from waiting — backpressure inflates it |
| `pressure` | mean input fill − mean output fill | how expensive the node is, only that it paces |
Queue occupancy has to be **sampled during the run**. `current_fill` is
instantaneous and every channel has drained by shutdown, so a single read at the
end describes an idle pipeline however congested it was.
**The number that matters** is `pressure`, because work piles up in front of the
bottleneck and starves everything after it, and that ordering holds whether the
node is waiting on a core, a GPU or a disk. `cpu_share` then selects the repair:
a pacing node with a saturated thread is CPU-bound and the work must get cheaper,
while a pacing node with an idle thread is device-bound, where batch size and
engine precision are the knobs and the C++ is not.
**Current:** `--benchmark <path>` writes the JSON report and prints a table at
shutdown; `src/benchmark.hpp`. Attribution is a pure function over KPN snapshots,
so it is verified on CI's GPU-free N100 (UT-120…UT-124) rather than only by
running the pipeline. The node graph is recovered from KPN's channel names, so a
re-wired topology needs no change here. Required `NodeStats::total_exec_us` in
the KPN submodule — the EMA could not be turned into a total.
**Gap:** GPU utilisation and memory are not sampled, so a device-bound verdict
says *that* a node waits on the GPU, not whether the GPU is saturated or merely
badly fed. That distinction needs NVML, and it is what VR-008 will want anyway.
## VR-008 — Gallery scaling benchmark
Establish the throughput-versus-gallery-size curve required by A10.
docs/benchmark.md
-344
View File
@@ -1,344 +0,0 @@
# Benchmark — SuperHero
The reference film for end-to-end accuracy. Replaces Road to Bali, which was
withdrawn for the reason in [Why not Road to Bali](#why-not-road-to-bali).
TRACES: AR-011, AR-012, AR-013 | VR-001, VR-005 | SR-002
---
## The film
SuperHero, from the [NIST TRECVID Deep Video Understanding development
set](https://www-nlpir.nist.gov/projects/trecvid/dvu/dvu.development.dataset/).
14 films are asserted Creative Commons and need no data agreement; only the 5
KinoLorber test films are gated.
| | |
|---|---|
| Runtime | 1025.5 s (17.1 min), 10 scenes |
| Resolution | 640×360 |
| Ground truth | Per-scene presence, from the scene knowledge graphs |
| Gallery | 5 characters, 14 references |
The DVU set is what makes this workable: it ships **character** face crops cut
from the film itself, so ground truth and gallery are both in character space
and scoring needs no actor→character mapping.
**Licence caveat.** NIST links licence evidence for only 4 of the 14 films, and
SuperHero is not one of them — its end credits carry no copyright or CC notice,
list a "Temporary Musical Score" and a SAG cast, and it has no traceable online
release. Fine for internal benchmarking; do not redistribute frames from it.
Valkaama is the one film with an independently documented licence (CC BY-SA 3.0)
if provenance ever has to be defended.
---
## Reproducing it
```sh
# 1. Annotations, character mugshots, scene segmentation.
# NIST names the same film three different ways, hence the overrides.
KG_DIR=superHero KG_FILE=superhero scripts/fetch_dvu.sh SuperHero ../dvu-hero
# 2. Scene clips (movie.shots), then fuse them into one stream.
# Fusing matters — see "Run it as one film" below.
# SuperHero-1.webm … SuperHero-10.webm from
# <dataset>/movie.shots/, then:
ffmpeg -f concat -safe 0 -i concat.txt -c copy SuperHero_full.webm
# 3. Gallery, with the face-size floor that keeps references in distribution.
./build/build_gallery --root ../dvu-hero/root \
--output ../dvu-hero/hero66.h5 --min-face-px 66
# 4. Run, on the GPU path (see "Check you are on the GPU").
./build/scene_analyze --movie hero/SuperHero_full.webm \
--gallery ../dvu-hero/hero66.h5 \
--detector-engine trt_cache/scrfd.scrfd_500m_bnkps.640.fp16.engine \
--arcface-engine trt_cache/arcface.LVFace-B_Glint360K.b4.fp16.engine \
--fps 5 --min-face-px 32 --expand-gallery \
--output pred.json
```
Nothing here is in git: the clips are ~130 MB and the annotations are
regenerable. Replay fixtures derived from the run ship through the artifact
registry instead:
```sh
scripts/artifacts/push_artifacts.sh replay-fixtures
scripts/artifacts/pull_artifacts.sh replay-fixtures [version]
```
The gallery travels in the same archive as the dumps deliberately — a dump only
replays meaningfully against the gallery it was produced with, and pairing one
with a different gallery silently changes every identity decision in it.
---
## Results
Measured on the fused film, gallery expansion on.
| Metric | Value |
|---|---|
| Precision | **1.00** |
| Recall | 0.65 |
| F1 | 0.79 |
| True positives | 13 |
| False positives | **0** |
| False negatives | 7 |
Six of ten scenes scored exactly right, including the three-character scenes 4
and 5.
**Zero false positives is the result worth keeping.** Every out-of-gallery
character — Beast, Mighty Celestial, Ms. Johnson, Doctor, two Masked Persons —
was declined rather than forced onto a nearest match. That is the calibrated
probability (AR-024) doing its job, and it is the right failure direction for an
X-Ray overlay: a miss is a gap, an invention is a lie.
**The misses have a shape.** Scenes 1, 2, 3 and 8 were missed, and 1–3 are the
three shortest scenes in the film (14 s, 38 s, 27 s). That is consistent with
per-track Bayesian accumulation (AR-025) needing enough sightings before belief
crosses threshold. Scene 8 is 65 s and does not fit that story — it is the one
to look at first when improving recall.
Running the same scenes as isolated clips did *not* do better, so cross-scene
gallery expansion is not currently compensating for short scenes.
### Run it as one film, not as clips
Per-scene clips defeat per-film gallery expansion (AR-019), which grows a
temporary gallery from track continuity across the whole film and re-assesses
unknown tracks at the end. Ten isolated clips give it nothing to work with, and
pay model and gallery load ten times over.
Fusing also makes presence windows cross real scene boundaries, which is how
SR-002's scene-scoped question is asked in production. Note the joins are
artificial cuts — consecutive scenes were never contiguous footage — so presence
bleeding across a boundary may be the join rather than a tracking fault.
---
## Throughput
| Path | Realtime factor | Sampled fps | 17-min film |
|---|---|---|---|
| `build/` (TensorRT) | **8.25×** | 41.3 | **2.1 min** |
| `build-ort/` (ORT) | 0.54× | 2.7 | ~32 min |
TensorRT figure re-measured 2026-08-04 over the whole film at `--fps 5
--min-face-px 32 --expand-gallery`: 5129 frames, 1025.4 s of film in 124.2 s
wall. Two runs agreed to 0.4% (124.2 s clean, 124.7 s under gdb). It supersedes
an earlier 2.0×; that figure predates the current tree and was not re-derived
here, so treat the gain as measured rather than explained.
Throughput varies strongly with face density, and **a short window is not a
sample of the film**. The opening 60 s benchmarks at 23.9× — decode there costs
4-6 ms/frame against a 12.35 ms whole-film mean (n=510), because seeking forward
in VP8/WebM gets dearer the deeper you go, and there are few faces. Always quote
the whole-film average.
### Where the time goes (VR-015)
Measured over the whole film, 2026-08-04:
| node | cpu_s | % of pipeline CPU | cpu/f | exec/f | stall/f | in% | out% |
|---|---|---|---|---|---|---|---|
| **embedder** | **91.0** | **60%** | 17.74 | 21.61 | 3.87 | 12 | 0 |
| **face_detector** ▶ | 41.4 | 27% | 8.07 | 24.20 | **16.14** | **99** | **0** |
| frame_source | 12.1 | 8% | 2.36 | 11.26 | 8.90 | — | 97 |
| camera_pos | 3.2 | 2% | 0.63 | 0.64 | 0.01 | 97 | 99 |
| face_aligner | 1.7 | 1% | 0.33 | 0.34 | 0.01 | 0 | 12 |
| identity_matcher | 1.3 | 1% | 0.26 | 0.34 | 0.09 | 0 | 0 |
| tracker / sink | 0.5 | <1% | — | — | — | 0 | 0 |
**`face_detector` paces the run**: its input channel is 97.8% full while its
output is 99.4% empty — everything upstream jammed, everything downstream
starved. It occupies 5129 × 24.20 ms ≈ 124.1 s of a 124.2 s run, essentially
100% wall occupancy, yet only 33% of that is CPU. The other 16.14 ms/frame is
device wait.
**The embedder is the larger cost but not the constraint**: 60% of all pipeline
CPU, 73% of wall as thread-busy. Whether that is real work or a spinning
`cudaStreamSynchronize` is unresolved — see the sync caveat below, which is a
one-line experiment.
**`frame_source` is the trap this table exists to defuse.** It reports
`exec/f = 11.26 ms` against `cpu/f = 2.36 ms`, and its output channel is 97%
full: it is backpressured, not expensive. The old KPN `ema` reading made it look
like the most costly node in the pipeline at 141.899 ms/frame.
`--benchmark <path>` writes a per-node timing report and prints a table at
shutdown. `hero/run_bench.sh` is `run_trt.sh` with it switched on:
```bash
./build/scene_analyze … --benchmark $H/bench_trt.json --output $H/pred_bench.json
```
**Do not read the `ema` column of the old KPN diagnostics block as a cost.** KPN
times a node across `fire_once`, which wraps the functor *and* the push to the
next channel, and a push parks when that channel is full (AR-004). A
backpressured node therefore bills its waiting to itself. On this film that
produced a genuinely inverted answer:
```
│ frame_source frames=5132 ema=141.899ms ← reported cost
[frame_source] decode avg=16.6127ms fps=60.19 ← actual decode
```
The source is not expensive; it is idle, holding a frame nobody has taken yet.
Optimising against that number means optimising the fastest node in the graph.
The benchmark report separates the two:
| Column | Meaning | Blind spot |
|---|---|---|
| `cpu_s`, `cpu%tot` | thread CPU time, and this node's share of all of it | a GPU wait looks like idleness |
| `cpu/f` | CPU ms per frame — backpressure cannot inflate it | as above |
| `exec/f` | wall ms per frame in the node, **including parked pushes** | overstates a blocked node |
| `stall/f` | `exec/f − cpu/f`: parked, or waiting on a device | does not say which |
| `in%`, `out%` | mean fill of the node's input and output channels | — |
| `press` | `in% − out%`; **the node marked ▶ is pacing the run** | not a cost, an ordering |
Read `press` first: work queues up in front of the bottleneck and starves
everything after it, so the pacing node is the one with a full input and an empty
output. Then read `cpu%run` to decide the repair — a saturated thread means the
work itself must get cheaper, while an idle thread under pressure means the node
is waiting on the GPU or the disk, where batch size and engine precision are the
knobs and the C++ is not.
Channel fills are sampled every 100 ms (`--benchmark-interval-ms`) because
`current_fill` is instantaneous: by shutdown every channel has drained, so a
single read at the end reports an idle pipeline no matter how congested it was.
#### Check the GPU is not throttled before comparing anything
**On this hardware, thermal state moves the result more than any code change
we are likely to make.** The same binary measured **8.25× cool and 3.12× once
heat-soaked** — a 2.6× swing — because the laptop RTX 3050 hits `SW Thermal
Slowdown` and pins the SM clock to **210 MHz out of 2100**:
```
$ nvidia-smi -q -d PERFORMANCE | grep -E "SW Power Cap|SW Thermal"
SW Power Cap : Active
SW Thermal Slowdown : Active
```
A number recorded without its clock state is not comparable to any other
number, and back-to-back full-film runs guarantee the later ones are throttled.
`run_bench.sh` now records `nvidia-smi` either side of the run into
`bench_gpu.txt`; check it before believing a regression. Let the GPU idle back
to full clock between measurements, and never A/B two runs across a heat-soak.
This one cost real time here: a 2.7× "regression" was attributed to a code
change and reverted on that basis, when the change was innocent and the GPU had
simply warmed up between the two measurements.
#### `cpu_s` on a GPU node is mostly spin — measured
CUDA's default sync policy (`cudaDeviceScheduleAuto`) spin-waits before it
yields, so `cudaStreamSynchronize` charges the *calling thread's* CPU while the
GPU works. A GPU-bound node therefore reports a large `cpu_s` and reads as
CPU-bound.
`SAE_CUDA_BLOCKING_SYNC=1` switches to a blocking wait. Measured over 300 s of
film, four cases, identical otherwise:
| case | realtime | total CPU | embedder CPU |
|---|---|---|---|
| baseline | 3.29× | 103 s | 66 s |
| **`SAE_CUDA_BLOCKING_SYNC=1`** | 3.29× | **23 s** | **4 s** |
| `SAE_CV_THREADS=1` | 3.30× | 101 s | 66 s |
| both | 3.29× | 25 s | 5 s |
**94% of the embedder's CPU was spin, not work**, and 78% of the pipeline's.
Throughput is unchanged, so this is free CPU — which matters for a service
sharing a box (DP-003) and makes `cpu_s` mean what it says. Prefer it for any
run where the CPU numbers are being read.
`SAE_CV_THREADS=1` does nothing measurable: the only OpenCV-heavy node is
`face_aligner` at 1-2% of the pipeline, so the TBB arena is not worth removing
and `warpAffine` is not worth replacing.
**Caveat: measured with the GPU clamped at 210 MHz** (see below). A device at
full clock spends less time in the sync, so the absolute spin figure will fall;
the ranking should not.
#### `cpu_s` counts one thread — mind the TBB arena
OpenCV 5 here is built against TBB, and every OpenCV module links it, so
`cv::parallel_for_` dispatches onto a TBB arena of `nproc − 1` workers (19 on the
20-core dev box; visible as `libtbb.so.12` frames in a thread dump). Since
`CLOCK_THREAD_CPUTIME_ID` is per-thread, work a node fans out that way is billed
to the TBB workers, **not** to the node.
So a node using `warpAffine`, a histogram compare or a colour conversion reads
cheaper in `cpu_s` than it really is, and the missing time appears in `stall/f`,
where it looks identical to a GPU wait. `exec/f` does capture it — the functor
does not return until the parallel region joins — so the tell is a node whose
`exec/f` far exceeds its `cpu/f` **while its output channel is empty**: that is
fan-out, not blocking.
Worth knowing for its own sake, too: 9 KPN node threads plus 19 TBB workers plus
the CUDA and NVDEC threads is heavy oversubscription on 20 cores.
The JSON carries the same data plus the run's configuration, so two runs can be
diffed directly — which is the point, when sweeping `--embed-batch`, `--fps` or
an engine precision.
### Check you are on the GPU
ORT's CUDA execution provider fails to load on this machine and **silently falls
back to CPU**:
```
Failed to load library libonnxruntime_providers_cuda.so:
undefined symbol: cudnnGetConvolutionBackwardDataAlgorithm_v7
```
That symbol was removed in cuDNN 9; the packaged ORT is built against cuDNN 8.
ORT logs this once at startup and then runs happily on CPU, so a `build-ort`
timing is a CPU number wearing a GPU label — a 15× error with no symptom other
than a figure you have no baseline for. Grep the log for `Failed to load
library` before trusting any throughput measurement.
The TensorRT path (`build/`) needs prebuilt engines from
`scripts/build_trt_engines.sh` and reports what it loaded:
```
[TrtScrfd] loaded: … [TrtArcFace] loaded: … max_batch=4
[similarity] cuBLAS/CUDA engine: gallery resident on GPU
```
---
## Why not Road to Bali
Bali was chosen because DVU ships character mugshots for it. It was withdrawn on
**face scale**, measured on its own reference crops:
| | Bali | SuperHero |
|---|---|---|
| Median detected face | 27 px | **69 px** |
| Maximum detected face | 69 px | **241 px** |
| References ≥66 px | 2 of 69 | 14 of 27 |
The DVU images are scene crops, not mugshots, so the crop dimensions say nothing
about face scale — the face has to be detected and measured. Bali's median
reference was being upscaled roughly 4× to reach ArcFace's 112×112, and the
worst 7×, which violates AR-011: every model gets the input it was trained for.
A model run off-distribution returns confident, plausible, wrong output.
In a gallery that error is permanent. A bad frame costs one frame; a poisoned
reference corrupts every future match against that identity.
No threshold rescued it. At 66 px only 2 of 69 references survived — the largest
face in the entire set is 69 px — so there was no cut that both kept references
in distribution and left enough of them to calibrate. SuperHero's gallery builds
at a 66 px floor and calibrates on its own (`a=15.2867 b=-4.98633`, 100 % train
accuracy) rather than borrowing constants.
Any accuracy figure recorded against Bali predates this and should be treated as
measuring upscaling artifacts as much as the pipeline.
docs/index.md
+8
View File
@@ -53,6 +53,14 @@ every finding below.
A training-set effect that did not reproduce on 5 held-out films once
two methodology bugs in the comparison harness were found and fixed.
- :material-blur:{ .lg .middle } **[What does blur cost?](quality-knee.md)**
---
Sharpness is not a sufficient statistic for identity loss, blur breaks
confidence rather than ranking, and variance-of-Laplacian is
anti-predictive at fixed resolution.
- :material-magnify-expand:{ .lg .middle } **[Deep dive: LVFace-B Glint360K](lvface-deep-dive.md)**
---
docs/plan.md
+4 -7
View File
@@ -255,11 +255,8 @@ Context crops opt-in behind `--dump-unidentified-crops`.
## AR-026, AR-027 — GEMM and scale
**Depends on:** nothing to start. The annex CPU loop has moved into the GEMM
path: the annex is a contiguous matrix, promotions are appended to the engine's
resident gallery, and the CPU backend now requires OpenBLAS. What is left of
AR-026 is call site 3, the deferred pass — so the rest of AR-026 lands *with*
AR-020 rather than before it.
**Depends on:** nothing to start. The annex CPU loop
(`identity_matcher_node.hpp:159-162`) moves into the GEMM path.
---
@@ -366,8 +363,8 @@ one identity out of the gallery would fix that.
thing measured is the thing that ships.
`scripts/validation/test_audio_offset.py` over
`tests/fixtures/audio/superhero_offset_200s.flac`: 200 s of public-domain film audio
(the same SuperHero clips the replay fixtures use), long enough for a 120 s
`tests/fixtures/audio/bali_offset_200s.flac`: 200 s of public-domain film audio
(the same Road to Bali clips the replay fixtures use), long enough for a 120 s
window to slide past the ±600-frame search cap. The slide itself is numpy here
on purpose — matching belongs to the consumer, so writing it out keeps this a
test of the signature rather than of somebody's matcher.
docs/quality-knee.md
+308
View File
@@ -0,0 +1,308 @@
# Quality knee: what does a blurred or small face cost, and can a measure predict it?
VR-012. Companion to the minimum-face-size studies VR-005 and VR-013 (see the
[requirement register](requirements.md)), which located the size floor at 40 px;
this asks the same question for **sharpness**, and asks whether any cheap
measure taken on the aligned crop can be acted on at inference.
Run by
[`scripts/validation/quality_knee.py`](https://REPOLINK/scripts/validation/quality_knee.py)
through the `sae_embed` bindings — detection, the ArcFace warp, the embedder,
the five candidate measures and the Platt calibration are all the shipped C++.
## Protocol
1670 gallery actors with 3 or more mugshots (of 2456 total), one image held out
per actor as a probe, the remaining 10326 embeddings staying in the gallery at
native resolution. Only the probe degrades — reference mugshots are clean and
the face coming out of the video is not.
Each probe passes through a **joint grid**: downscale to *S*×*S* and back to
112 (the sampling loss), then blur at level *L* in canonical pixels. Three blur
families, 36 cells each, 60120 probe-cell records per family:
| family | models | parameter |
|---|---|---|
| Gaussian | soft focus, a generic stand-in | sigma 0 … 3 |
| **Disc** | **real optical defocus** — the circle of confusion | radius 0 … 6 |
| Motion | camera pan or moving subject | length 0 … 21 px |
The three are not interchangeable, and sweeping only the first was the original
design error — one that would have produced a wrong answer, not merely an
incomplete one (Result 3). A defocused lens spreads a point into a **uniform
disc**, whose transfer function is a jinc — `2·J1(x)/x` — that crosses zero and
goes negative, annihilating whole frequency bands and returning the ones beyond
each zero phase-reversed. A Gaussian MTF is strictly positive and monotone and
does neither. More practically: defocus and motion are how a face ends up
**large and useless**, while Gaussian blur as swept here mostly co-occurs with
small faces. That difference decides whether sharpness carries anything the size
filter does not.
Families are compared at matched **per-axis PSF standard deviation** (σ for a
Gaussian, R/2 for a disc, L/√12 for a linear smear), never at equal raw
parameter, which would compare different amounts of damage.
Identification is the pipeline's own decision: per-actor best-of-N cosine →
Platt sigmoid → accept above `prob_threshold` 0.754. Never a raw cosine
(AR-024).
## Result 1 — sharpness is not a sufficient statistic
Sorting the 36 Gaussian cells by `hf_energy_ratio`, the six sigma-3 cells land
at effectively identical measured sharpness:
| size | sigma | hf_energy_ratio | TPI |
|---|---|---|---|
| 16 | 3 | 0.0003 | **15.3%** |
| 24 | 3 | 0.0003 | 63.2% |
| 32 | 3 | 0.0003 | 79.4% |
| 48 | 3 | 0.0003 | 86.6% |
| 64 | 3 | 0.0004 | 88.4% |
| 112 | 3 | 0.0005 | **91.0%** |
Same measured sharpness, a **76-point spread in identification**. It inverts
too: 16 px unblurred measures 0.0033 and scores 23.5%, while 48 px at sigma 2
measures *lower* at 0.0021 and scores 96.6%.
A canonical-frame sharpness scalar cannot separate *attenuated* high
frequencies from *destroyed* spatial sampling. Blur suppresses the high band
while preserving mid-frequency facial geometry exactly; downsampling to 16 px
destroys that geometry outright. Both look alike to any measure keyed on
high-frequency energy.
This is the measured basis for AR-028's rule that the axes are **kept separate
and not collapsed into one scalar**, and it settles the double-counting
question: size and sharpness are not redundant, and neither substitutes for the
other.
## Result 2 — blur is a cliff, and it breaks confidence, not identity
TPI % by size (rows) against Gaussian sigma (columns):
| size | 0 | 0.5 | 1 | 1.5 | 2 | 3 |
|---|---|---|---|---|---|---|
| 16 | 23.5 | 24.0 | 25.0 | 24.6 | 23.9 | 15.3 |
| 24 | 85.7 | 85.1 | 85.6 | 85.9 | 82.6 | 63.2 |
| 32 | 95.9 | 95.9 | 96.0 | 95.5 | 93.7 | 79.4 |
| 48 | 98.7 | 98.7 | 98.4 | 98.1 | 96.6 | 86.6 |
| 64 | 98.6 | 98.6 | 98.8 | 98.4 | 97.5 | 88.4 |
| 112 | 98.9 | 98.9 | 98.8 | 98.6 | 98.0 | 91.0 |
Three regimes: **sigma ≤ 1.5 is free** (every cell moves under 1.5 points, sign
flipping at random — at 16 px it slightly *improves*, smoothing upscale
artifacts); sigma 2 costs 1–3 points; the 2→3 step costs 7–19. A smooth
discount curve is therefore the wrong shape — the response is flat, then falls
off a cliff.
**The cost peaks at the size knee, not at full resolution.** Sigma 3 costs
−22.5 points at 24 px but only −7.9 at 112 px and −8.3 at 16 px. Blur has no
intrinsic cost; it costs in proportion to how close the observation already sits
to the decision boundary. At 112 px there is margin to spare, at 16 px the probe
is already below threshold, and at 24 px it sits exactly on the knee.
**What blur destroys is confidence, not ranking.** Rank-1 barely moves: 99.3% →
99.2% at 112 px across the whole sigma range. The extreme case is 16 px at sigma
3, where rank-1 is **80.2%** while TPI is **15.3%** — 65 points of probes have
the correct actor ranked first and are rejected anyway for falling under the
probability threshold.
That is why **FPI never left 0.1% in any of the 108 cells across all three
families**. Degradation produces TBI, never a wrong name. The calibration
degrades gracefully, which is what SR-002 needs.
## Result 3 — the blur *family* matters more than the blur *amount*
Comparing families by their raw parameter is meaningless — sigma, radius and
length are different units. They are matched here by the **per-axis standard
deviation of the PSF**, which puts them on one scale:
| family | per-axis σ | level giving σ = 3 px |
|---|---|---|
| Gaussian σ | σ | 3 |
| Disc radius R | R/2 | 6 |
| Motion length L | L/√12 | 10.4 |
For reference the ArcFace template places the eyes 35.2 canonical px apart, so
σ = 3 px is 9% of the inter-ocular distance.
TPI at matched severity, interpolated within each family:
| size | σ=3 Gaussian | σ=3 Motion | σ=3 **Defocus** | defocus penalty |
|---|---|---|---|---|
| 16 | 15.3 | 15.1 | 11.0 | +4.3 |
| 24 | 63.2 | 61.1 | 41.4 | +21.9 |
| 32 | 79.4 | 76.1 | 50.4 | +29.0 |
| 48 | 86.6 | 80.9 | 52.6 | +34.0 |
| 64 | 88.4 | 81.7 | 51.0 | +37.4 |
| 112 | 91.0 | 82.1 | **46.9** | **+44.0** |
**Optical defocus is up to 44 points more destructive than a Gaussian of
identical spread**, and the ordering is defocus ≫ motion > Gaussian throughout.
At σ=1 the three families are indistinguishable, and at σ=2 they differ by under
5 points; the divergence appears only when both the blur is severe *and* the face
is large.
That pattern is physically consistent. At 16 px the resampling has already
removed the high frequencies, so the PSF's shape has nothing left to act on and
all three agree. At 112 px the full spectrum is present and shape decides: a
Gaussian MTF rolls off gently and always leaves *some* energy at every
frequency, so the embedder receives a merely attenuated signal, while a disc MTF
is a jinc that **hits exact zeros** — whole frequency bands annihilated rather
than attenuated, with the bands beyond each zero returning phase-reversed.
Motion sits between them because it ruins one axis and leaves the perpendicular
one untouched.
**The methodological consequence is the important one.** This study originally
swept Gaussian blur alone and concluded blur was a minor effect. On the family
that actually occurs in film, the same nominal severity costs **53% error
instead of 9%** at full resolution. A threshold set from the Gaussian arm would
have been wrong by a factor of five in error rate, and the axis would probably
have been dropped as not worth its cost.
**Defocus is also the case a size gate cannot catch.** Every one of those 112 px
faces is large and confidently detected, and sails through AR-002 untouched.
That, not the Gaussian result, is what justifies a sharpness axis existing at
all.
## Result 4 — variance of Laplacian is anti-predictive at fixed degradation
Pooled across all cells, every candidate scores AUC 0.76–0.80 for predicting
correct identification, with textbook `var_laplacian` top. That number is close
to worthless: it rewards a measure for detecting *how degraded the crop is*,
which all five do. The question a per-observation discount needs is whether, at
a **fixed** degradation, the measure predicts which faces fail:
| measure | Gaussian | Defocus | Motion |
|---|---|---|---|
| `hf_energy_ratio` | **0.530** | **0.521** | **0.557** |
| `norm_var_laplacian` | 0.520 | 0.507 | 0.539 |
| `dir_min_tenengrad` | 0.524 | 0.512 | 0.506 |
| `tenengrad` | 0.433 | 0.437 | 0.457 |
| `var_laplacian` | 0.423 | 0.422 | 0.473 |
Best is 0.557 — barely above chance, and `hf_energy_ratio` wins on all three
families. `var_laplacian` is anti-predictive on all three too, so that finding
does not depend on the blur model.
**The two metrics measure different jobs, and the candidates split along that
line.** On the motion arm `dir_min_tenengrad` has the best *pooled* AUC by a
wide margin — **0.854** against 0.792 for the next — exactly as its synthetic
directional-blur ladder predicted, yet its within-cell AUC there is 0.506. It is
an excellent detector of *how badly smeared a crop is* and no guide at all to
*which face will be recognised*. Pooled AUC is the right metric for a
gross-degradation flag; within-cell AUC is the right one for a per-observation
discount; a measure can be strong at one and useless at the other.
Deciles within the 16 px Gaussian cell, where 1277 failures give the test real
power:
| `var_laplacian` decile | TPI |
|---|---|
| 0.00071–0.00192 (blurriest) | **37.1%** |
| 0.00242–0.00278 | 22.8% |
| 0.00397–0.00447 | 25.7% |
| 0.00625–0.01445 (sharpest) | **14.4%** |
The faces the measure calls sharpest are **2.6x less identifiable** than those
it calls blurriest, monotone across ten bins of 167. Within a cell every crop
received identical degradation, so the residual variance is *native contrast*,
not native detail — and hard shadows, high-contrast lighting, sharpening halos
and JPEG ringing all raise Laplacian variance while making a face harder to
match. The measure reads photographic style and encoding artifacts and calls
them sharpness.
`hf_energy_ratio` is the only candidate with a correctly-signed within-cell
trend (16.2% → 35.3% across the same deciles), being a pure ratio in which the
contrast factor cancels.
**Consequence:** a per-face quality *discount* keyed on variance of Laplacian —
the most widely used blur metric in production vision pipelines — would
systematically down-weight the *more* identifiable faces. It is worse than no
discount.
## Result 5 — as a compute gate, sharpness loses to the size filter
Skipping the embed for crops below a threshold, measured as compute saved
against true identifications lost:
| gate | skipped | true IDs lost | of skipped, doomed anyway |
|---|---|---|---|
| `hf_energy_ratio` < 0.00023 | 10.0% | 7.6% | 37.9% |
| `hf_energy_ratio` < 0.00051 | 20.0% | 15.1% | 38.7% |
| **source size < 24 px** | **16.7%** | **4.7%** | **77.3%** |
At a comparable skip rate the size filter loses **4.7% against sharpness's
15.1%** — three times less damage — and it is free, being a bbox dimension
available before alignment or embedding, where sharpness needs the warped crop
plus a colour convert, three convolutions and a 64×64 DFT.
Restricting to large faces (≥64 px) on the **defocus** arm, where the size
filter is blind, improves the gate's precision 3.5x (37% of skipped crops doomed
versus 10.7% on the Gaussian arm) but not its trade: skip 10%, lose 7.0%.
A hard ceiling explains why. **At 112 px with defocus radius 6 — visually
destroyed — 46.9% of faces still identify correctly and rank-1 is still 94.8%.**
Blur does not determine the outcome, so any gate keyed on apparent blur is
predicting a coin flip. The size filter wins not because size is better
measured, but because *smallness destroys identity more completely than blur
does*: 16 px faces succeed only 23.5% of the time, so discarding them is cheap.
## What this means for the requirements
**Do not gate on sharpness; discount on it.** Heavily defocused faces remain
~47% identifiable, so a gate destroys recoverable evidence. This is the first
hard evidence that AR-028's "**discounts the observation, never deletes the
detection**" is right on the merits rather than merely cautious. Since ranking
survives where confidence does not, the per-track accumulation (AR-025) should
recover much of what a single-frame threshold rejects — which is also the
argument for the discount living in `EvidenceDiscounter` rather than in a filter.
**`var_laplacian` and `tenengrad` are disqualified as discounts** by Result 4,
on all three blur families. They remain usable as coarse *gross-degradation*
detectors, the role in which their pooled AUC is real — the same role the size
filter plays — but they must never weight a per-observation belief.
**`hf_energy_ratio` is the only surviving discount candidate**, best on all
three families, and its within-cell signal (0.52–0.56) is weak enough that
shipping a discount on it needs justification beyond this study.
**`dir_min_tenengrad` earns a different job.** Its pooled 0.854 on the motion arm
makes it the best available detector of gross directional smear — useful as a
per-frame "this shot is unusable" flag, which is a decision about a *frame*, not
a weighting of an *observation*. If AR-029 ships two measures for two roles, this
is the second one, and it must not be confused with the first.
**Model the blur family, not just its amount.** Result 3 makes the choice of
degradation model a first-order design decision rather than a detail: the same
matched severity costs 9% or 53% error depending on the PSF. Any future study
that sweeps blur must state which family it used and why.
**Any discount curve must be flat then steep**, not linear or sigmoid over the
measure. Blur costs nothing until it costs a great deal.
## Limitations
- **Cooperative population.** Gallery mugshots are frontal and well-lit;
within-cell failures are likely dominated by cross-view mismatch, which no
sharpness measure can predict. Read the ~chance within-cell AUCs as "sharpness
does not predict the dominant failure mode *here*", not as "sharpness is
meaningless".
- **Uniform grid, not a natural distribution.** Sizes and blur levels are
sampled evenly, so "skip 16.7%" is exactly the 16 px row. The gate comparisons
are like-for-like on identical records, but the absolute savings are not what
a film would show.
- **TensorRT fp16.** A different realisation of the embedder from the fp32 ONNX
reference — VR-005 measured ~0.85 cosine agreement with separation intact.
Gallery and probes share one session so the study is internally consistent,
but the absolute knee belongs to the fp16 space.
- **Blur is applied in the canonical frame**, after resampling, so its width is
independent of the cell's size. Real optics blur before sampling.
- **The top motion rung is an anchor, not an operating point.** Length 21 is a
per-axis σ of 6.1 — 17% of the inter-ocular distance, a streak rather than a
face — and it is swept to bound the curve, not because a frame like that is
worth reasoning about. Its 3.4% TPI at 112 px should not be quoted as a
headline. The same caution applies less severely to defocus radius 6 (σ = 3).
- **Per-axis σ equates spread, not perceptual damage.** It is the fairest single
scalar for comparing PSFs, but Result 3 is precisely the finding that equal
spread does *not* mean equal harm, so the matched-severity tables compare
like-for-like inputs, not like-for-like severity as a face would experience it.
docs/requirements.md
+32 -58
View File
@@ -29,47 +29,47 @@ Status: `Done` · `In Progress` · `Planned` · `TBD` · `Withdrawn`
| ID | Requirement | Traces to | Priority | Status |
|---|---|---|---|---|
| AR-001 | Detect faces in sampled frames; emit bbox, confidence, 5-point landmarks in original pixel space | SR-002 | High | Done |
| AR-002 | Minimum face size **40×40 px** (VR-013 measured end to end; VR-005's 32 px is an embedder-only upper bound), expressed in **original** resolution (decoupled from `dense_scale`) | SR-002 | High | **Done** — `FaceDetectorFunc::drop_undersized()`. The threshold is divided by `bbox_upscale` rather than every box multiplied, which keeps the comparison on the detector's own numbers and means turning `dense_scale` on cannot silently raise the minimum face the pipeline accepts. Verified at the threshold and at `dense_scale` 0.5 (UT-002), and end to end on the fixture (IT-001) — the superhero dump's smallest side is *exactly* its recorded 32 px, so the filter is binding there rather than vacuously satisfied |
| AR-002 | Minimum face size **40×40 px** (VR-013 measured end to end; VR-005's 32 px is an embedder-only upper bound), expressed in **original** resolution (decoupled from `dense_scale`) | SR-002 | High | Planned |
| AR-003 | No fixed per-frame face cap — crowd scenes must not lose background cast | SR-002 | Medium | **Done** — `max_faces` defaults to 0 (no cap); the matcher batches through its GEMM buffer instead of throwing |
| AR-004 | Backpressure: unbounded faces/frame absorbed by slowing, never by dropping or throwing | SR-002 | High | **Mostly** — node outputs *park* on a full channel: the value is held, the worker released, and a channel space-callback resumes the node. Replaces `push_blocking`, which parked a scheduler worker inside the push and, with one thread per node, stopped that node draining its own input. Verified: 385/385 frames, 0 drops. Holes closed since, in the order they surfaced: **(a)** `FanoutNode` dropped on overflow rather than waiting, so the AR-010 scene join shed frames exactly when the dense branch fell behind — measured at **9 of 2192 items delivered** to the slower of two branches, now lossless with the fast branch throttled to within its buffering; **(b)** the residual hang, recorded as ~1 run in 20 at a 300 s timeout, was a **startup** lost wake, not a mid-stream one — `start()` enables a node's inputs several statements before it installs the push callback, and a producer firing into that gap is accepted by the ring while waking nobody, since `Channel::push` signals only the empty→non-empty edge. Signature is zero items delivered, never a partial stall. Reproduced 7 times in 24 under CPU contention and 0 in 10 without; `start()` now closes with the level-triggered `on_input_ready()`, giving 0 in 24 on the same harness — though the *cause* was narrower than recorded there and is fixed properly in **(e)**; **(c)** `FilterNode` and `RouterNode` were the last data paths still using the throwing `push()` with the exception swallowed, so a full output discarded the value — including the **EOF sentinel**. The decimator passes EOF by predicate (`if (f.eof) return true;`) but its output is reliably full, the embedder being the slowest node in the chain, so the token was discarded, nothing downstream ever shut down, and the run had to be killed. **This is the wedge.** Both now route sentinels out-of-band and retry data until taken; the regression case delivers 6 of 40 values and never sets `saw_eof` before, 40 and terminating after; **(d)** the sentinel could be delivered *ahead of* a value still queued behind it — `pop()` observed the ring empty and then took the sentinel, and a producer can push a value *and* publish the sentinel inside that window, so any consumer treating EOF as a hard stop loses the tail. `take_sentinel` now re-checks emptiness *after* observing `has_eof_`, which is sound because the sentinel is published with a release store after the ring pushes. ~1 run in 15 before, 0 in 25 after; **(e)** two `fire_once` invocations for one node could overlap, because the submit gate was released before the firing had finished touching node state. That breaks the one-slot park the whole scheme rests on — a parked value can be overwritten by the other firing, with no drop recorded anywhere. ThreadSanitizer caught it as a race on `pending_done_`; the release is now the last act of a firing. The same sweep found the callbacks themselves being written while a running neighbour read them (ten TSan races), which is the *actual* cause of the startup lost wake in **(b)** — callbacks are now installed in a `prepare()` pass before any node starts. **New constraint:** a channel carries at most **one undelivered sentinel**; a second offered before the first is taken is refused and reported, never queued and never overwritten, since two control tokens on one channel means the stream ended twice. Single-shot EOF today, live the moment a pipeline is reused for a second input. **Consequence to hold onto:** a lossless fanout makes join depth a correctness precondition — one branch can now run ahead of another only by the slower branch's buffering, so `kSceneJoinDepth` must exceed the TransNetV2 window. Making the decimator lossless also makes it a backpressure point rather than a relief valve: the source now throttles to the face branch instead of quietly thinning it. Correct under this requirement, but it changes the shape of a loaded run and is **not yet benchmarked**. **Gap:** capacity is still counted in *items*, not bytes, so a crowd frame carrying 60 crops occupies one slot exactly as an empty one does — the memory ceiling the plan asks for is unenforced |
| AR-004 | Backpressure: unbounded faces/frame absorbed by slowing, never by dropping or throwing | SR-002 | High | **Done** — KPN node outputs use `push_blocking`; sentinels stay out-of-band. Verified: 385/385 frames, 0 drops, byte-identical across runs |
| AR-005 | Align to 112×112 via ArcFace 5-point similarity transform, fitted by **Umeyama least squares over all five points** (as InsightFace does) — never a robust fit, which would discard the landmarks AR-030 reads | SR-002 | High | **Done** — `umeyama_similarity()`. The RANSAC fit it replaces disagreed by a median 17 source px on 400 headshots, 83.5% of crops embedding below cos 0.99, and was unstable and RNG-driven: rebuilding caught 1614 near-duplicates against the original build's ~100. **All galleries rebuilt** (2456 actors, 10254 embeddings); measured separation gain is small (0.583 → 0.590), so recorded accuracy figures should be re-run but are not expected to move far |
| AR-006 | 512-d L2-normalised embeddings, batched | SR-002 | High | Done |
| AR-007 | Associate detections by IoU + embedding, with **frame-dependent** weighting | SR-002 | High | **Done** — `track_alpha` is the base for ordinary frames; drops to embedding-only on cut/boundary and for dormant tracks |
| AR-008 | One track pool keyed on `last_seen`; no separate revival path | SR-002 | High | **Done** — one pool keyed on `last_seen`; park/revive branch deleted |
| AR-009 | Camera-cut detection (histogram) as an association hint | SR-002 | High | Done |
| AR-010 | Scene-boundary detection (TransNetV2) as an association hint | SR-002 | Medium | **Done** — decode butterfly joined via `SceneBoundaries`; the sampled branch waits for the detector's watermark. Frames past its last scored window are counted as unverified, never assumed boundary-free |
| AR-011 | **Every model is fed the input it was trained for** — cost reduced by running less often, never by degrading one inference | SR-002 | High | **Done** — both violations SPEC.md named are closed. (1) `scene_decode_fps` defaults to 0 (native): at 12 fps a 100-frame `kWindow` spanned ~8.3 s instead of the ~4 s TransNetV2 was trained on, half-speed motion over twice its temporal context. (2) The boundary dedup window is derived from the cadence the detector was actually fed (`SceneDetectorFunc::dedup_window_sec()`, median observed interval, halved) rather than the literal 0.04 s — one frame at 25 fps, and at 30 fps wider than a frame, so two cuts on consecutive frames merged into one and the loss was invisible: the file simply had fewer boundaries. Derivation checked at 24/25/30 fps and under a seek (UT-003). **Consequence, not a gap:** `scene_threshold` 0.60 was fitted against the 12 fps input and is now certainly wrong — VR-006 re-fits it, and until then boundary recall at native rate is untuned rather than better. Dense decode is the cost driver, so this is not free; `dense_scale` and `scene_stride` remain the reductions that do not run the model off-distribution. **Half-applied until now:** the derived window reached `scenes.json` and nothing else. `SceneBoundaries` — the path that actually feeds `is_scene_boundary` to the tracker — kept the literal 0.04 s under a comment claiming the two views agreed. They did not. The detector now supplies the window it derived to both |
| AR-011 | **Every model is fed the input it was trained for** — cost reduced by running less often, never by degrading one inference | SR-002 | High | Planned |
| AR-012 | Presence follows **track extent**, not per-frame recognition | **SR-002** | High | **Done** — `src/track_registry.hpp`; window is `[first_seen, last_seen]` of an owned track |
| AR-013 | `last_seen` optional state machine; window ends at last sighting, never after | SR-002 | High | **Done** — `last_seen` optional is the whole state machine; interior gaps absorbed, trailing cool-down never claimed |
| AR-014 | Belief swap A→B terminates the track and starts a new one | SR-002 | Medium | **Done** — swap closes at `last_seen` and opens a successor at the swap frame; counted |
| AR-015 | Two live tracks owned by one actor ⇒ treat as a detected cut, re-associate | SR-002 | Medium | **Done** — reverse index detects it on the causing update; counted |
| AR-016 | All tracks closed at EOF — a film ends with faces on screen | SR-002 | High | **Done** — `flush()`, idempotent, closes at last sighting or final tick |
| AR-017 | Every presence claim carries its belief and identification route | SR-002 | High | **Done** — `DeadTrack` carries belief, observation count, and now a `route` enum. The route was previously the literal string `"live"` written at serialisation time, so the published field could not distinguish anything and AR-017's own edge case ("deferred and pooled routes distinguishable") was unmeetable. Only `live` occurs until AR-020 lands; `deferred` exists so that pass has somewhere to write instead of a schema change to make |
| AR-018 | Per-subject embedding store with banded admission (novel enough, safe enough) | SR-005 | Medium | **Done** — banded admission in probability space, bounds from `expand_band_lo/hi`; the lower bound re-asked pairwise at promotion, since `admit` compares only against the nearest member and a drifting track can chain past it. Retires `expand_novelty_sim` and `expand_track_spread_max` — novelty is now the eviction ordering, not a threshold. Rejections counted. Bounds unswept (VR-007) |
| AR-019 | Per-film gallery annex from owned tracks; acquires the non-frontal views TMDB lacks | SR-005 | Medium | **Done** — all three discontinuity signals clear the buffers; ownership comes from the registry, not a second local tally. **Correction:** the local tally was still there and still deciding. Promotion fired on a local accepted-frame count and fell back to a local per-actor plurality whenever the registry had not yet claimed the track — which is the common case, since three accepted frames arrive well before a posterior crosses `ownership_logodds`. So in practice the plurality usually decided, and it could not see the AR-025 discounting it was supposed to defer to. Promotion now requires the registry's verdict; the accepted-frame count is an explicit evidence floor. `forget()`, which had no callers under a comment claiming the matcher called it, is replaced by `prune_dead` against the registry's own liveness |
| AR-017 | Every presence claim carries its belief and identification route | SR-002 | High | **Done** — `DeadTrack` carries belief and observation count |
| AR-018 | Per-subject embedding store with banded admission (novel enough, safe enough) | SR-005 | Medium | **Done** — banded admission in probability space; replaces `expand_novelty_sim`. Rejections counted |
| AR-019 | Per-film gallery annex from owned tracks; acquires the non-frontal views TMDB lacks | SR-005 | Medium | **Done** — all three discontinuity signals clear the buffers; ownership comes from the registry, not a second local tally |
| AR-020 | Deferred re-identification of unknown tracks against the final expanded gallery | SR-005 | High | Planned |
| AR-021 | Cluster unknown tracks into one entity per person, under temporal cannot-link constraints | SR-005 | Medium | Planned |
| AR-022 | Capture still-unidentified tracks: embeddings, metadata, **context crops** for human review | §4 | Medium | Planned |
| AR-023 | Fit sigmoid calibration from intra/inter similarity distributions | SR-002 | High | **Done** — and the meaning of "the fit failed" is now uniform. `valid=false` used to send the matcher to a raw-cosine accept rule while `same_person_probability` sent every other stage to the untuned default sigmoid: one run, two policies, no announcement. Both now take the default sigmoid and warn loudly that the probabilities are not meaningful |
| AR-024 | **Always the calibrated probability, never a raw cosine** — exceptions recorded | SR-002 | High | **Done** — association, accumulation and expansion all in probability space; `track_max_embed_dist`, `cut_revive_sim`, `expand_novelty_sim`, `expand_track_spread_max` retired. Enforcement now exists rather than being asserted: `scripts/ci/check_raw_cosine.py` blocks in CI. It immediately caught a live violation — the matcher's no-calibration fallback thresholded raw cosine distance **and fed `max(0, cosine)` into `TrackRegistry::observe`**, whose contract says in terms that it cannot be handed an uncalibrated number by a careless caller. `match_threshold`, `match_ratio` and `match_ratio_ceil` are retired with it, and `TrackGallery`'s `max(0, cosine)` default calibration is now a hard error. One exception recorded, in the calibration's own dedup |
| AR-025 | Per-track Bayesian accumulation in log-odds, with correlated-observation discounting | SR-002 | High | **Done** — log-odds accumulation with correlation discounting owned by the registry, `src/evidence_discount.hpp`. The four constants governing this — `ownership_logodds`, `rho_max`, `admit_below`, `max_views` — were unreachable in-class defaults until now; see VR-007 |
| AR-026 | All similarity computed as GEMM, including annex and deferred pass | SR-001 | High | **In Progress** — two of the three call sites done. Baked gallery was already GEMM; the annex now is too — it is a contiguous row-major matrix (`track_gallery.hpp`) whose promoted rows are appended to the engine's resident matrix (`ISimilarityEngine::append_rows`), so one multiply covers baked and promoted references and the host-side cosine loop is gone. CPU path requires OpenBLAS (scalar fallback now opt-in behind `SAE_ALLOW_SCALAR_GEMM`). Remaining: the deferred pass, which does not exist until AR-020 |
| AR-023 | Fit sigmoid calibration from intra/inter similarity distributions | SR-002 | High | Done |
| AR-024 | **Always the calibrated probability, never a raw cosine** — exceptions recorded | SR-002 | High | **Done** — association and accumulation both in probability space; `track_max_embed_dist`, `cut_revive_sim` retired |
| AR-025 | Per-track Bayesian accumulation in log-odds, with correlated-observation discounting | SR-002 | High | **Done** — log-odds accumulation with correlation discounting owned by the registry, `src/evidence_discount.hpp` |
| AR-026 | All similarity computed as GEMM, including annex and deferred pass | SR-001 | High | In Progress |
| AR-027 | Throughput acceptable for **arbitrary** gallery size | SR-001 | High | Planned |
| AR-028 | **Embedding input quality assessed and carried** — every face scored on size, sharpness and visibility before its embedding is used as identity evidence; the vector travels with the face and reaches the VR-001 dump | SR-002 | High | **Done** — filled in by `FaceAlignerFunc`, where both measured axes come free from the warp; carried on `DetectedFace` and written to the dump as `faces/sharpness` + `faces/alignment_residual`, taking it to `schema_version` 2. Size is `bbox`, not duplicated into a field that would drift. No face is admitted unscored (-1 sentinel), and the degenerate-fit case is now counted and reported rather than silently dropped. **Carried, not consumed** — no discount and no threshold, which is AR-030 and VR-012. Verified UT-137, UT-138 (aligner) and UT-139…UT-141 (dump round-trip, version, sentinel). The committed fixtures are still v1, so they carry no vector until `make_fixtures.sh` is re-run on a GPU host |
| AR-029 | Sharpness measure on the **aligned crop** (scale-normalised, so it cannot re-measure size) | SR-002 | Medium | **Done** — `crop_sharpness()`: variance of the Laplacian over variance of the crop, so contrast cannot leak in the way it does for the raw textbook measure. Both blur ladders monotone, Gaussian and motion. Three properties recorded on the function for VR-012 rather than corrected here: the contrast invariance is exact in the algebra but bends at the 8-bit quantisation floor (a dim *and* soft crop reads sharper than it is — 148% high at σ 2.5), `BORDER_CONSTANT` fill from a frame-edge face adds a step edge, and the measure conflates focus with intrinsic texture. Verified UT-130…UT-136 |
| AR-028 | **Embedding input quality assessed and carried** — every face scored on size, sharpness and visibility before its embedding is used as identity evidence; the vector travels with the face and reaches the VR-001 dump | SR-002 | High | Planned |
| AR-029 | Sharpness measure on the **aligned crop**, consumed as a discount and **never as a gate** | SR-002 | Medium | **In Progress** — five candidates implemented (`src/quality.hpp`) and ranked by VR-012 over three blur families. `var_laplacian` and `tenengrad` are **disqualified as discounts**: within a fixed degradation they are anti-predictive on *all three* families (AUC 0.42–0.47; the decile the measure calls sharpest is 2.6× *less* identifiable), since their residual variance is native contrast, not detail. `hf_energy_ratio` is the only correctly-signed survivor, best on all three, and weak (0.52–0.56). `dir_min_tenengrad` is the best *gross-smear detector* (pooled AUC 0.854 on motion) but ~chance within-cell, so it serves a per-frame flag, not a per-observation weight. The parenthetical this row used to carry — "scale-normalised, so it cannot re-measure size" — was wrong: every candidate responds to source size, and the axes are separable for a different reason (see AR-028) |
| AR-030 | Visibility measure from the AR-001 5-point landmarks — extreme pose or occlusion **discounts the observation, never deletes the detection** | SR-002 | Medium | **In Progress** — measure is the AR-005 alignment residual (`estimate_alignment()`), carried on `DetectedFace`; roll/scale invariance and monotonicity under foreshortening asserted. Nothing consumes it as a discount yet |
## Deployment (DP)
| ID | Requirement | Traces to | Priority | Status |
|---|---|---|---|---|
| DP-001 | One analysis core; modes are front-ends and must not fork pipeline logic | PR-004 | High | **Done, after a repair.** `scene_preview` had forked the construction sequence and then rotted: it built `FaceTrackerFunc{cfg}` against a signature that stopped existing with the AR-007/AR-008 redesign, so **it had not compiled since**, and it never wired registry claims into its sink. It now mirrors `main.cpp` exactly — matcher, then registry, then tracker. The lesson is that "must not fork" needs the build to notice; a front-end nothing compiles is a fork that rots in silence |
| DP-001 | One analysis core; modes are front-ends and must not fork pipeline logic | PR-004 | High | Done |
| DP-002 | Batch CLI over one title | PR-004 | High | Done |
| DP-003 | On-demand resident service with bounded, observable queue | PR-004 | Medium | Planned |
| DP-004 | Opportunistic/idle mode: external trigger, hard stop, implicit re-queue | PR-004 | Medium | Planned |
| DP-005 | Native installer, no Docker; Fedora + Arch | PR-004 | Medium | Planned |
| DP-006 | Background incremental gallery refresh on a timer | PR-003 | Medium | Planned |
| DP-007 | CI builder image, CPU-only, pinned by tag in the Gitea container registry | PR-004 | High | **Mostly** — image and publish script exist (`Dockerfile.builder-cpu`, `scripts/ci/build_builder_image.sh`) and `.gitea/workflows/unit-tests.yml` now consumes it, pinned to `v1` and asserting at run time that the image reports that tag. **Gap:** the image is built and pushed by hand from an authenticated host; nothing rebuilds it on a change to the Dockerfile |
| DP-007 | CI builder image, CPU-only, pinned by tag in the Gitea container registry | PR-004 | High | Planned |
| DP-008 | Builder images + release jobs per backend (cpu / cuda / rocm); ship binaries, not engines | PR-004 | Medium | Planned |
## Integration (IR)
@@ -79,7 +79,7 @@ Status: `Done` · `In Progress` · `Planned` · `TBD` · `Withdrawn`
| IR-001 | Emit the JRay truth format as sibling `.jray.json` | SR-003 | High | Done |
| IR-002 | Windows carry belief + route; `extraction.*` carries `extinction_sec`, `gallery_scope` | SR-003 | High | **Done** — `schema_version: 2`; windows are objects with `belief` + `route`; `extraction.*` carries `extinction_sec` and `gallery_scope`; `anneal_sec` removed |
| IR-003 | Output written **after** the deferred pass, not at EOF | SR-003 | High | **In Progress** — sink builds windows from registry claims and flushes at EOF; the deferred pass (AR-020) does not exist yet, so output is still final at EOF |
| IR-004 | Compute the audio signature exactly per server spec §3 | SR-003 | Medium | **Done** — `src/audio_signature.*`; not yet emitted into the truth file (IR-002). One real defect found and fixed since: the resampler's `AVChannelLayout`s were not zero-initialised, and `av_channel_layout_copy` uninitialises its destination first, so `av_freep` was handed stack garbage. It aborted about 1 run in 4 of UT-103 — invisible in the aggregate test binary, where the case usually passes, and absent under a sanitizer build because it is stack-dependent. `ctest`, one process per case, is what turned it into a reproducible failure |
| IR-004 | Compute the audio signature exactly per server spec §3 | SR-003 | Medium | **Done** — `src/audio_signature.*`; not yet emitted into the truth file (IR-002) |
| IR-005 | Golden-vector fixture shared with the plugin repo to prove bit-exactness | SR-003 | High | **Done** — `tests/fixtures/audio/`; v1 parameters now normative in server spec §3 |
| IR-007 | Media < 120 s: emit no signature, apply no sync offset — identical rule in both producers | SR-003 | Low | **Done** |
| IR-008 | Emit and honour the signature's own `v1:` version prefix | SR-003 | Low | **Done** |
@@ -91,7 +91,7 @@ Status: `Done` · `In Progress` · `Planned` · `TBD` · `Withdrawn`
|---|---|---|---|---|
| GR-001 | Build gallery from Jellyfin library cast, TMDB profile fallback | SR-001, SR-005 | High | Done |
| GR-002 | Incremental `--merge` refresh without re-embedding known actors | PR-003 | High | Done |
| GR-003 | Report coverage: zero-image actors, under-referenced actors, dedup, calibration PDFs | SR-001 | Medium | **Done** — `gallery/gallery_report.hpp`, written next to the gallery by `build_gallery`. Zero-usable-image actors come from the build audit, which a stored gallery cannot reconstruct; also `distinct_references`, `duplicates_removed`, and the intra/inter distributions the calibration fits and would otherwise discard |
| GR-003 | Report coverage: zero-image actors, under-referenced actors, dedup, calibration PDFs | SR-001 | Medium | Planned |
| GR-004 | Stamp embedder identity into the gallery; **hard startup error** on mismatch | SR-001 | High | **Done** — basename + SHA-256 + `embed_dim`; mismatch fatal with no bypass, unstamped warns unless `--require-gallery-stamp`; `scripts/stamp_gallery.py` migrates in place |
| GR-005 | Gallery data never leaves the instance | **SR-005** | High | Done |
| GR-006 | Provenance tiers: baked / harvested / confirmed, distinguishable per embedding | SR-005 | High | Planned |
@@ -104,22 +104,19 @@ Status: `Done` · `In Progress` · `Planned` · `TBD` · `Withdrawn`
| ID | Requirement | Traces to | Priority | Status |
|---|---|---|---|---|
| VR-001 | HDF5 post-inference dump at the embedded-frame boundary | PR-002 | High | Done |
| VR-002 | Replay drives the **real** KPN nodes, not a reimplementation | PR-002 | High | **Done** — including the sink, as of VR-011. Worth recording what the reimplementation was hiding: `build_minimal` rebuilt windows in Python from per-frame annotations, which never consult the registry, so it kept producing plausible output while registry-based presence in replay was returning **nothing at all**. The first run of the real chain emitted 0 actors on a film where 1647 frames carried an identified face. A reimplementation does not merely risk disagreeing with the pipeline; it can conceal the pipeline being broken |
| VR-002 | Replay drives the **real** KPN nodes, not a reimplementation | PR-002 | High | **Done** — replay driven from committed fixtures in `tests/test_replay_fixtures.cpp`; determinism asserted |
| VR-003 | Scoring: micro-F1 against X-Ray, precision/recall logged at every evaluation | PR-002 | High | Done |
| VR-004 | Reproducible validation corpus with ground truth | PR-002 | High | Done |
| VR-005 | Minimum face size study — TPI/FPI vs probe size, gallery held at native res | PR-002 | Medium | **Done** — knee at 24–32 px; 32 px gives 98.1% TPI, 0.0 FPI at every size. Degrades an already-aligned 112×112 crop, so it isolates the embedder and is an **upper bound**; VR-013 measures the same question end to end and AR-002 takes its number, not this one |
| VR-006 | Re-tune `scene_threshold` once native-rate decode lands | PR-002 | **Medium** | **Planned, now unblocked** — native-rate decode landed with AR-011, so the prerequisite is met and the current 0.60 is a value fitted against input the pipeline no longer produces. Raised from Low for that reason: it is no longer a refinement, it is a stale constant |
| VR-007 | Expansion band, clustering threshold, deferred-pass ablation, **and the AR-025 accumulation knobs** | PR-002 | Medium | **Planned — scope corrected.** `rho_max`'s own comment already deferred to this row, and four constants it names were unreachable: `ownership_logodds` on `TrackRegistry::Config`, and `max_views`/`admit_below`/`rho_max` on `EvidenceDiscounter::Config`, which `main` built through the one-argument constructor. No sweep could vary them. They are in `Config` with CLI flags now, so this row can be run. `ownership_logodds` is the one to start with: below it a track makes **no presence claim at all**, so it decides whether an actor is reported rather than how confidently |
| VR-006 | Re-tune `scene_threshold` once native-rate decode lands | PR-002 | Low | Planned |
| VR-007 | Expansion band, clustering threshold, and deferred-pass ablation | PR-002 | Medium | Planned |
| VR-008 | Gallery scaling benchmark — throughput vs gallery size | PR-002 | Medium | Planned |
| VR-009 | Verify accumulated posteriors are calibrated against held-out tracks | PR-002 | High | Planned |
| VR-010 | Dump provenance attributes — embedder model, detector settings, `dense_scale`, `scene_detect`, sample rate | PR-002 | **High** | **Done** — `DumpProvenance` in `embedding_dump_node.hpp`, written as root attributes and read back tolerantly. Every field is optional so a pre-VR-010 dump reads as *unknown* rather than as a default; a silently-defaulted `detector_conf` is exactly the fabricated provenance this exists to prevent. This row said `Planned` while five VR-010 tags sat in the code — stale in the opposite direction to the rest of this register |
| VR-011 | Rewrite the replay harness for the post-AR-012 output contract | PR-002 | High | **Done** — `sae_kpn` compiles again and the replay drives the whole chain including `ResultSinkFunc`, so presence comes from `TrackRegistry` claims rather than being rebuilt in Python. The three per-node factories are replaced by one `add_pipeline` that mirrors `main.cpp`'s construction order — the ordering constraint (matcher fits the calibration, registry needs a discounter from it, tracker needs both, sink needs the claims) is what a factory-per-node API could not express, and is why the tracker factory kept building `FaceTrackerFunc{cfg}` against a signature that had stopped existing. `build_minimal` and `anneal_sec` are gone. Verified end to end on the SuperHero fixture: 5 actors, 32 windows, 0 dropped votes |
| VR-012 | Quality-knee study — TPI/FPI vs sharpness and vs pose, as VR-005 did for size; also settles whether the 5-point pose proxy needs a dedicated landmark model | PR-002 | Medium | Planned |
| VR-010 | Dump provenance attributes — embedder model, detector settings, `dense_scale`, `scene_detect`, sample rate | PR-002 | **High** | Planned |
| VR-011 | Rewrite the replay harness for the post-AR-012 output contract | PR-002 | High | Planned |
| VR-012 | Quality-knee study — TPI/FPI vs sharpness and vs pose, as VR-005 did for size; also settles whether the 5-point pose proxy needs a dedicated landmark model | PR-002 | Medium | **In Progress** — sharpness half done ([`docs/quality-knee.md`](quality-knee.md)): 1670 actors, joint size×blur grid over three blur families (Gaussian, disc defocus, linear motion), 60120 probe-cell records each. Sharpness is **not a sufficient statistic** (equal measured sharpness spans 15.3–91.0% TPI, ordered by source size); **the blur family matters more than its amount** — at matched per-axis σ=3 on a 112 px face, Gaussian/motion/defocus cost 9/18/**53**% error, so a Gaussian-only sweep understates real lens blur fivefold; blur breaks **confidence, not ranking** (rank-1 80.2% where TPI is 15.3%), so FPI never left 0.1% in any of the 108 cells; a sharpness **gate** loses 3× more true presence than the free size filter at equal saving, because even destroyed faces stay 46.9% identifiable. **Pose half not started** — the AR-030 residual is exposed via `sae_embed.alignment_residual` but no pose arm has been run, so the dedicated-landmark-model question is still open |
| VR-014 | Audio-signature **offset recovery on real content** — a known trim recovered from film audio, not from the synthetic golden tone | PR-002 | Medium | **Done** — 40 random in-cap offsets, every one recovered to the nearest frame: **worst error 46 ms against a 500 ms budget**, and 46 ms is the floor rather than a result, since the offset is quantised to whole 92.88 ms frames. The `runtime/2` anchor confirmed through real head-trimmed files (a `delta` trim moves the window by `delta/2`). The one soft spot is **tier labelling, not accuracy**: the score falls with sub-frame misalignment (0.94–0.99 near a frame boundary, 0.69–0.73 at half a frame), so 27/40 correct alignments were demoted to `loose`. ±1 frame of slack in the *score* fixes it — measured, all 40 back to `audio` (min 0.906), false matches unmoved at 0.12–0.16, costing 81 ms of the budget |
| VR-015 | Per-node cost and bottleneck attribution for a run — where the time actually goes | PR-004 | High | **Done** — `--benchmark <path>` on `scene_analyze`; `src/benchmark.hpp`. Reports cumulative CPU and wall time per node, and locates the pacing node from sampled channel occupancy rather than from time-in-node, which backpressure inflates. Verified UT-120…UT-124 |
| VR-013 | Cross-source identification probe — gallery from one recording, probes from another, swept over input resolution end to end | PR-002 | Medium | **In Progress** — holding 90% of the plateau needs ~50 px end to end against VR-005's ~22 px, the gap being detection and landmark error; **`min_face_px` 40, since 32 admits faces in the falling region** (AR-002). FPI 0.0% at every scale. Ceiling is cross-view, not resolution |
| VR-016 | **Cut-detection cadence study** — is `cut_threshold` 0.70 meaningful at the rate `camera_pos` is actually fed? | PR-002 | Medium | **Planned.** The histogram cut detector is the one always-on signal with no recorded provenance, and its input rate is not the rate it was fitted at. With `--scene-detect` off, `camera_pos` sits downstream of a source already decimated to `sample_fps`, so at the 1.0 default it compares frames **one second apart** — inside a single shot those differ enormously, and 0.70 correlation is a low bar to clear. With `--scene-detect` on it sees native-rate frames instead, so the same constant means two different things depending on an unrelated flag. This is AR-011's argument ("every model gets the input it was trained for") applied to a non-neural detector, and it matters because `is_cut` drives `track_alpha` to 0 and clears every expansion buffer. Cheap first measurement: run `camera_pos` over a `hero/` clip at 1/2/5 fps and compare cut counts against `tests/fixtures/dumps/scene_bounds.json`. The committed 5 fps dump shows 2.6% of frames flagged; nobody has measured 1 fps |
| VR-017 | **Vote-lag study** — how often does the matcher fall more than `track_extinction_sec` behind the tracker on real content? | PR-002 | **High** | **Planned.** Channel depth is a correctness parameter between `face_tracker` and `identity_matcher`, and the constraint runs opposite to the scene join's: there `kSceneJoinDepth` must EXCEED the TransNetV2 window, here the depth must be UNDER `track_extinction_sec × sample_fps`. Backpressure is what makes it bite — it is working, and a lossless channel converts depth into lag by design. Both nodes are 16 deep in `main.cpp`, which at the default `sample_fps` 1.0 is ~16 s of lag against a 5 s window, so `scene_analyze` can drop identity votes and until now said nothing. It now reports `dropped_votes` at shutdown; this row is the measurement that decides whether that should be fatal, and whether the right fix is bounding the depth or removing the coupling (reap on the matcher's clock rather than the tracker's, so a vote cannot be late by construction) |
---
@@ -216,7 +213,6 @@ as such rather than counted as covered.
| GR-001 … GR-005 | T1 + T3 | Gallery assembly is I/O and bookkeeping; embedding is T3 smoke |
| GR-006 … GR-008 | T1 | Tiering and outlier detection operate on stored embeddings |
| VR-* | Out of CI | Studies are run deliberately and their results committed as documents |
| VR-014 | **T2** | The exception, and the reason the blanket row above is not the whole story: its fixture is committed and its signature is CPU-only DSP, so the study *is* a test a CI host can run — not a measurement someone has to remember to repeat |
**One consequence worth stating:** AR-027 (arbitrary gallery scale) is
structurally unverifiable on the CI host. It needs a GPU host and a synthetic
@@ -234,9 +230,9 @@ time**, and CI consumes the result as data.
This is what makes the T1/T2 split load-bearing rather than a preference: T1 and
T2 are the only tiers that can exist in CI at all.
### Fixture corpus — `hero/`
### Fixture corpus — `bali/`
Five clips of **SuperHero (1952)**, ~77 s each, 480×360, 30 fps, 42 MB total.
Five clips of **Road to Bali (1952)**, ~77 s each, 480×360, 30 fps, 42 MB total.
Public domain, and that is the reason to use it rather than a convenience:
**derived fixtures — dumps, crops, golden outputs — can be committed without the
@@ -315,7 +311,7 @@ because it will be trusted.
| AR-001 | T3 | Detector returns plausible boxes on a known frame | — smoke only |
| AR-002 | T2 | Faces below 40 px (original res) are dropped | Exactly at threshold; with `dense_scale` 0.5 — the interaction that motivated the requirement |
| AR-003 | T2 | No cap applied; a 40-face frame yields 40 | Crowd frame |
| AR-004 | T1 | Saturated input blocks rather than drops or throws | Bounded queue at capacity; **byte-based** limit with large crops; SIGTERM mid-block. Cases the KPN suite now pins, each of which failed before being written: a fanout feeding an unequal pair loses nothing *and* throttles the fast branch (either assertion alone passes on a broken implementation); a filter delivers EOF into a saturated output; a sentinel is never delivered ahead of a queued value; a twice-parked value keeps its payload; and a node started with data already in its input still fires — the startup lost wake, which needs no contention to reproduce once the state is constructed directly |
| AR-004 | T1 | Saturated input blocks rather than drops or throws | Bounded queue at capacity; **byte-based** limit with large crops; SIGTERM mid-block |
| AR-005 | T1 | Known landmarks → expected 112×112 warp; the fit never mirrors | Landmarks near frame edge; degenerate/collinear points; a mirrored set — SVD returns a reflection unless the determinant guard rejects it |
| AR-006 | T3 | Embeddings are unit-norm | Batch smaller than, equal to, larger than `embed_batch_size` |
| AR-007 | T2 | Association picks the right track | Two faces crossing paths; one leaving frame as another enters |
@@ -327,23 +323,22 @@ because it will be trusted.
| AR-014 | T2 | Belief swap closes one window, opens another | No blended window; no overlap at the swap frame |
| AR-015 | T2 | Two live tracks on one actor trigger re-association | Counter increments |
| AR-016 | **T2** | Every track closed at EOF | Film ending mid-shot — window ends at final frame, not dropped |
| AR-017 | T1 | Claim carries posterior and route | Deferred and pooled routes distinguishable — now possible: `route` is an enum on `DeadTrack` rather than the literal `"live"` the sink used to write. Only `live` occurs until AR-020 exists, so the test that matters today is that the field survives serialisation |
| AR-017 | T1 | Claim carries posterior and route | Deferred and pooled routes distinguishable |
| AR-018 | T1 | Band admits only within bounds | At each bound exactly; store never admits below lower bound |
| AR-019 | T2 | Promotion only when all three signals quiet | Cut mid-track blocks promotion |
| AR-020 | **T2** | Unknown resolved after expansion | Track failing at minute 12, resolved at EOF — the ordering-independence claim |
| AR-021 | T2 | Clustering merges same person, respects cannot-link | **Temporally overlapping tracks never merge**; measure how many merges the constraint rejects |
| AR-022 | T1 | Context crops retained, bounded per track | Track running for minutes |
| AR-023 | T1 | Sigmoid fit on synthetic separable data | Too few positive pairs → `valid=false`, and the fallback that engages is the **default sigmoid**, not the retired cosine rule. Assert the warning fires: an unfitted sigmoid returns plausible-looking probabilities, so nothing downstream can tell |
| AR-024 | **Static check** | No bare cosine outside a tagged `EXCEPTION` | `scripts/ci/check_raw_cosine.py`, blocking in the traceability workflow. Honest about its reach: it catches direct `cosine_similarity()` uses not routed through a calibration and **cannot follow a cosine through a variable across statements**, which is a convention backed by review rather than by the tool. Scans `src` only — a test legitimately asserts properties of the metric space, and sweeping those in would produce blanket exceptions that devalue the tag |
| AR-023 | T1 | Sigmoid fit on synthetic separable data | Too few positive pairs → `valid=false`, fallback engages |
| AR-024 | **Static check** | No bare cosine outside a tagged `EXCEPTION` | Grep-based; this is the invariant's enforcement |
| AR-025 | T1 | Log-odds accumulate; correlated frames discounted | 30 identical frames must **not** reach the certainty of 30 diverse ones |
| AR-026 | T1 + T4 | GEMM path produces same result as reference loop | Equivalence on small input in CI; throughput on GPU host |
| AR-027 | **T4** | Throughput at 10²…10⁵ actors | Scheduled, not on-demand |
| AR-028 | **T2** | No embedding reaches the matcher unscored; the vector survives into the dump | Face failing exactly one axis; all three healthy; a face whose landmarks are degenerate — dropped for want of a crop to score, but **counted** rather than silently vanished (UT-138) |
| AR-029 | T1 | Synthetic blur ladder → monotonically falling sharpness | Gaussian vs motion blur; **small sharp face vs large soft one** — size must not leak into this axis. The blur ladder must be measured on a **1/f texture**: on a flat-spectrum one the motion ladder *rises*, since an anisotropic smear takes energy out of numerator and denominator together (UT-131). Contrast must not leak either — exact in the algebra, and the 8-bit floor that bends it is pinned by UT-133 |
| AR-028 | **T2** | No embedding reaches the matcher unscored; the vector survives into the dump | Face failing exactly one axis; all three healthy; a face whose landmarks are degenerate — scored, not silently vanished |
| AR-029 | T1 | Synthetic blur ladder → monotonically falling sharpness | Gaussian vs motion blur; **small sharp face vs large soft one** — size must not leak into this axis |
| AR-030 | T1 | Alignment residual rises monotonically with foreshortening | **In-plane roll, scale and translation must leave it at zero** — the property that makes it a pose measure rather than a pose-and-everything-else measure; face size must not shift it; degenerate landmarks report not-ok rather than a number |
| VR-012 | **T4** | Knee located per axis on held-out films | Report each candidate threshold's cost in **lost true presence**, not only its gain in precision — a gate that improves misID by discarding half the cast has not helped |
| VR-013 | **T4** | Identification holds across two recordings of the same people, and degrades to TBI rather than to a wrong name as input resolution falls | Gallery and probes must come from *different* recordings — a hold-one-out over one recording measures a much easier problem and will not surface the cross-view failure. Ground truth is hand-sorted; labels propagated by embedding similarity would keep only the faces the embedder already gets right |
| VR-016 | **T2** | Cut rate as a function of the cadence `camera_pos` is fed | Same clip at 1/2/5 fps, `--scene-detect` on and off. The dump already records `cut_threshold` and `sample_fps` (VR-010), so a replay can score this without re-decoding. A finding of "0.70 is fine at every rate" is a real result and should be recorded as one |
| IR-001/002 | T1 | Serialised output matches golden file | Zero-length window; actor with many windows |
| IR-003 | T1 | Output written after deferred pass | Not at EOF |
| IR-004/005 | **T1** | Signature matches golden vector bit-for-bit | Identical result in both producer repos |
@@ -372,29 +367,8 @@ accumulation from being decoration.
| — | `anneal_sec` window merging | Superseded by AR-012/AR-013: a track survives its own gaps, so there is nothing to anneal |
| — | `extinction_sec` actor keep-alive | Superseded by AR-013: windows end at last sighting, which is what this over-claimed |
Both are now deleted rather than retained at zero — a field naming a mechanism
the pipeline no longer has is actively misleading (see `SPEC.md` A6.6).
**This paragraph was false for some time, and the failure is worth keeping.** It
was written in the present perfect as though the removal had happened. It had
not: `Config::extinction_sec` (57.4) and `Config::anneal_sec` (35.5) were still
there, `--extinction` and `--anneal` still parsed, and `SceneTrackerFunc` still
ran its keep-alive in both shipped pipelines, printing its timeout at every
startup. `SPEC.md`'s removal list ends "grep for both names and expect no
survivors"; there were about forty.
Nothing in the tooling could have caught it. The traceability gate reads tags,
not behaviour, and a withdrawn requirement has no tag to be orphaned — the
register simply asserted a state of the code, and no test asked. The general
form is worth stating: **a status column is a claim, and the only claims this
project can check automatically are the ones a test or a static check makes.**
The same pattern produced three other rows corrected in this pass (AR-011,
AR-017, AR-019), each recorded as done and done in one place out of two.
`SceneTrackerFunc` is replaced by the stateless `FrameAnnotationFunc`. One
visible consequence: `--verbosity standard`'s `frames[].identified` used to
include every actor inside the keep-alive window, and now lists what was matched
in that frame. Minimal and xray output never consulted the node.
Both were deleted rather than retained at zero — a field naming a mechanism the
pipeline no longer has is actively misleading (see `SPEC.md` A6.6).
---
docs/traceability.md
+190 -643
View File
File diff suppressed because it is too large Load Diff
experiments/dump_lvface_opencv5.sh
-54
View File
@@ -1,54 +0,0 @@
#!/usr/bin/env bash
# Fresh LVFace-B embedding dumps (HDF5) for all 9 X-Ray films with the current
# feature/opencv5 build, for the flood-fill GA optimisation. Plain front-half
# (decode -> campos -> detect -> align -> embed); no scene detection (histogram
# cuts is_cut are baked in for flood-fill). Hardware VAAPI decode, no MIGraphX,
# no crash. Serial -- ROCm GPU wedges at concurrency>2-3.
set -uo pipefail
REPO="/home/dtourolle/Development/scene-actor-extraction"
cd "$REPO"
ARC="models/LVFace-B_Glint360K.onnx"
BIN="build/dump_embeddings"
LUT="experiments/file-lut.json"
FILMS="experiments/manifests/films.json"
OUT="experiments/dumps/LVFace-B_Glint360K_opencv5"
mkdir -p "$OUT"
# Persist MIOpen tuning so SCRFD/ArcFace kernel search is paid once, not per film.
export MIOPEN_USER_DB_PATH="$HOME/.cache/miopen-sae"
export MIOPEN_FIND_MODE=NORMAL
mkdir -p "$MIOPEN_USER_DB_PATH"
mapfile -t SLUGS < <(python3 -c 'import json;[print(f["slug"]) for f in json.load(open("'"$FILMS"'"))]')
echo "=== LVFace-B dumps (feature/opencv5) — $(date) ===" | tee "$OUT/dump.log"
for slug in "${SLUGS[@]}"; do
movie="$(python3 -c 'import json;print(json.load(open("'"$LUT"'"))["'"$slug"'"])')"
out="$OUT/dump_${slug}.h5"
echo "" | tee -a "$OUT/dump.log"
echo ">>> $slug" | tee -a "$OUT/dump.log"
if [ -f "$out" ]; then echo " exists, skip" | tee -a "$OUT/dump.log"; continue; fi
if [ ! -f "$movie" ]; then echo " SKIP missing: $movie" | tee -a "$OUT/dump.log"; continue; fi
# No --max-decode-fps cap: that cap existed only to stop LVFace dump truncation
# under PARALLEL load (3 concurrent dumps). This runner is serial, so the cap
# just halved throughput for nothing — measured 54s vs 27s per 300s of film,
# identical face counts. Uncapped ~9 min/film vs ~18 min capped.
"$BIN" --movie "$movie" --arcface "$ARC" --out "$out" --fps 1 \
>"$OUT/${slug}.log" 2>&1
rc=$?
if [ $rc -ne 0 ] || [ ! -f "$out" ]; then
echo " DUMP FAILED (rc=$rc) — see ${slug}.log" | tee -a "$OUT/dump.log"
else
stats=$(python3 -c 'import h5py,sys
f=h5py.File(sys.argv[1])
n=f["frames/timestamp_sec"].shape[0]
faces=f["faces/embedding"].shape[0]
cuts=int(f["frames/is_cut"][:].sum())
print(f"frames={n} faces={faces} cuts={cuts}")' "$out" 2>/dev/null)
echo " ok ($(du -h "$out" | cut -f1), $stats)" | tee -a "$OUT/dump.log"
fi
done
echo "" | tee -a "$OUT/dump.log"
echo "=== DONE — $(date) ===" | tee -a "$OUT/dump.log"
experiments/run_xray_lvface_opencv5.sh
-60
View File
@@ -1,60 +0,0 @@
#!/usr/bin/env bash
# Re-benchmark the feature/opencv5 pipeline against Amazon X-Ray, all 9 films, LVFace-B.
# Full end-to-end scene_analyze (decode→detect→scene→embed→match→presence) — NOT a replay,
# because the framework changed enough that old embedding dumps no longer represent the front half.
# Outputs land in experiments/results/xray_opencv5_lvface/ (durable; /tmp gets wiped).
set -uo pipefail
REPO="/home/dtourolle/Development/scene-actor-extraction"
cd "$REPO"
ARC="models/LVFace-B_Glint360K.onnx"
GAL="experiments/galleries/gallery_LVFace-B_Glint360K.h5"
OUT="experiments/results/xray_opencv5_lvface"
mkdir -p "$OUT"
BIN="build/scene_analyze"
LUT="experiments/file-lut.json"
FILMS="experiments/manifests/films.json"
# film slugs and their xray dirs, from films.json
mapfile -t ROWS < <(python3 -c '
import json
for f in json.load(open("'"$FILMS"'")):
print(f["slug"] + "\t" + f["xray"])
')
echo "=== X-Ray re-benchmark (feature/opencv5, LVFace-B) — $(date) ===" | tee "$OUT/run.log"
for row in "${ROWS[@]}"; do
slug="${row%%$'\t'*}"
xray="${row#*$'\t'}"
movie="$(python3 -c 'import json,sys; print(json.load(open("'"$LUT"'"))["'"$slug"'"])')"
pred="$OUT/${slug}.json"
echo "" | tee -a "$OUT/run.log"
echo ">>> $slug" | tee -a "$OUT/run.log"
if [ ! -f "$movie" ]; then
echo " SKIP: movie missing: $movie" | tee -a "$OUT/run.log"
continue
fi
# Run the full pipeline (serial — ROCm GPU wedges at concurrency>2-3).
"$BIN" --movie "$movie" --arcface "$ARC" --gallery "$GAL" \
--output "$pred" >"$OUT/${slug}.pipeline.log" 2>&1
rc=$?
if [ $rc -ne 0 ] || [ ! -f "$pred" ]; then
echo " PIPELINE FAILED (rc=$rc) — see ${slug}.pipeline.log" | tee -a "$OUT/run.log"
continue
fi
echo " pipeline ok" | tee -a "$OUT/run.log"
# Score against X-Ray, masked to gallery∩GT, 1s grid.
python scripts/validation/sample_eval.py \
--pred "$pred" --xray "$xray" --gallery "$GAL" --step 1.0 \
>"$OUT/${slug}.eval.txt" 2>&1
tail -8 "$OUT/${slug}.eval.txt" | tee -a "$OUT/run.log"
done
echo "" | tee -a "$OUT/run.log"
echo "=== DONE — $(date) ===" | tee -a "$OUT/run.log"
experiments/xsource/landmark_voting.py
+7 -65
View File
@@ -30,8 +30,6 @@ import sae_embed # before cv2 — see alignment_compare.py
import numpy as np
import cv2
import argparse
ROOT = "/home/dtourolle/Development/Jray-project/scene-actor-extraction/"
M = ROOT + "models/"
CLIPS = ["5157344", "5157339"]
@@ -39,31 +37,16 @@ PROB_THRESHOLD = 0.754
GROUP_IOU = 0.55 # detections overlapping this much are the same face
MATCH_IOU = 0.35 # tie a detection to the hand-labelled face
_ap = argparse.ArgumentParser()
_ap.add_argument("--detector", default="scrfd_500m_bnkps.onnx",
help="detector under models/. SCRFD sizes 500m / 2.5g / 10g come "
"from InsightFace's buffalo_sc / buffalo_m / buffalo_l packs")
_ap.add_argument("--vote-conf", type=float, default=None,
help="confidence floor for the voting pass. Omit to auto-tune "
"it to --target-votes")
_ap.add_argument("--target-votes", type=int, default=3,
help="votes per face to tune --vote-conf towards, so detectors "
"are compared at equal redundancy rather than equal settings")
_args = _ap.parse_args()
base_eng = sae_embed.FaceEmbedder(detector_model=M + _args.detector,
base_eng = sae_embed.FaceEmbedder(detector_model=M + "scrfd_500m_bnkps.onnx",
arcface_model=M + "LVFace-B_Glint360K.onnx",
conf=0.5, nms=0.4, max_side=0)
def _make_vote_engine(conf):
# Same models, looser suppression: keep the duplicates NMS would have removed.
return sae_embed.FaceEmbedder(detector_model=M + _args.detector,
# Same models, looser suppression: keep the duplicates NMS would have removed.
vote_eng = sae_embed.FaceEmbedder(detector_model=M + "scrfd_500m_bnkps.onnx",
arcface_model=M + "LVFace-B_Glint360K.onnx",
conf=conf, nms=0.9, max_side=0)
conf=0.3, nms=0.9, max_side=0)
cal = sae_embed.gallery_calibration(ROOT + "gallery_lvface.h5")
def iou(a, b):
ax, ay, aw, ah = a; bx, by, bw, bh = b
x0, y0 = max(ax, bx), max(ay, by)
@@ -96,46 +79,6 @@ def vote(dets):
return out
def tune_vote_conf(target, sample=6):
"""Pick the confidence floor giving ~target detections per face to average.
A larger SCRFD is more confident and suppresses harder, so at a fixed floor
it emits fewer overlapping anchors — median 2 against 500m's 3. Comparing
detectors at equal SETTINGS therefore also compares them at unequal
redundancy, and the voting arm is handicapped for the bigger models. Tuning
each to the same votes-per-face isolates landmark quality from how much
there was to average.
"""
frames = sorted(glob.glob(f"frames/d{CLIPS[0]}_*.png"))[:sample]
imgs = [cv2.imread(f) for f in frames]
best = (None, None, 1e9)
for conf in (0.30, 0.20, 0.12, 0.07, 0.04, 0.02, 0.01):
eng = _make_vote_engine(conf)
sizes = [n for img in imgs for _, _, _, n in vote(eng.detect(img))]
if not sizes:
continue
med = float(np.median(sizes))
if abs(med - target) < best[2]:
best = (conf, eng, abs(med - target))
print(f"[tune] conf={conf:.2f} -> median {med:.0f} votes/face", file=sys.stderr)
if med >= target:
break
if best[1] is None:
print(f"[tune] no confidence floor reached {target} votes/face; "
f"falling back to 0.30", file=sys.stderr)
return 0.30, _make_vote_engine(0.30)
print(f"[tune] chose conf={best[0]:.2f} for ~{target} votes/face", file=sys.stderr)
return best[0], best[1]
if _args.vote_conf is not None:
VOTE_CONF, vote_eng = _args.vote_conf, _make_vote_engine(_args.vote_conf)
else:
VOTE_CONF, vote_eng = tune_vote_conf(_args.target_votes)
cal = sae_embed.gallery_calibration(ROOT + "gallery_lvface.h5")
def collect(clip):
lab = {os.path.basename(p): os.path.basename(os.path.dirname(p))
for p in glob.glob(f"labelling/{clip}/*/*.jpg")
@@ -181,8 +124,7 @@ print(f"[voting] group size: median {np.median(allv):.0f}, "
file=sys.stderr)
GAL, PRB = "5157344", "5157339"
print(f"\ndetector={_args.detector} vote_conf={VOTE_CONF:.2f} "
f"gallery {GAL} -> probe {PRB}, P>{PROB_THRESHOLD}\n")
print(f"\ngallery {GAL} -> probe {PRB}, P>{PROB_THRESHOLD}\n")
print(f"{'align':>8}{'person':>8}{'n_gal':>7}{'n_prb':>7}"
f"{'within-clip':>13}{'cross-clip':>12}{'hit rate':>10}")
summary = {}
experiments/xsource/resolution_sweep.py
-2
View File
@@ -1,8 +1,6 @@
#!/usr/bin/env python3
"""Impact of input resolution on cross-source identification.
TRACES: VR-013 | PR-002
Gallery is built from one clip at NATIVE resolution. Probes come from the other
clip with the WHOLE FRAME downscaled before it reaches the detector, so
detection and landmark regression degrade together with the pixels. That is the
experiments/xsource/verify_labels.py
-8
View File
@@ -1,14 +1,6 @@
#!/usr/bin/env python3
"""Integrity check on the labelled set, before it is used as ground truth.
TRACES: VR-013 | PR-002
VR-013's ground truth is hand-sorted rather than propagated by embedding
similarity, because propagation would keep only the faces the embedder already
gets right and silently drop the ones the sweep exists to find. This script is
what makes that claim checkable, so it is part of the requirement rather than a
helper of it.
Checks, loudest failure first:
1. INDEX INTEGRITY. Each crop's embedding is taken by re-detecting its source
external/KPN
Vendored
+1 -1
Submodule external/KPN updated: 771b9f8593...6595e6e925
mkdocs.yml
+1 -1
View File
@@ -35,11 +35,11 @@ extra_css:
nav:
- Home: index.md
- How We Score Against X-Ray: methodology.md
- Benchmark — SuperHero: benchmark.md
- Findings:
- Best Model: best-model.md
- Gallery Scope (Full vs. Limited): gallery-scope.md
- Pose Expansion: pose-expansion.md
- Quality Knee (Blur and Size): quality-knee.md
- LVFace Deep Dive: lvface-deep-dive.md
- Full Experiment Log: model-bakeoff.md
- Service Conversion (proposal): service-conversion.md
models/2d106det.onnx
LFS
BIN
View File
Binary file not shown.
models/scrfd_10g_bnkps.onnx
LFS
BIN
View File
Binary file not shown.
models/scrfd_2.5g_bnkps.onnx
LFS
BIN
View File
Binary file not shown.
scripts/artifacts/pull_artifacts.sh
+1 -23
View File
@@ -10,7 +10,6 @@
# scripts/artifacts/pull_artifacts.sh galleries [version]
# scripts/artifacts/pull_artifacts.sh montage-frames <film-slug> [version]
# scripts/artifacts/pull_artifacts.sh experiment-data [version]
# scripts/artifacts/pull_artifacts.sh replay-fixtures [version]
# scripts/artifacts/pull_artifacts.sh report-highlights <name> [version]
# scripts/artifacts/pull_artifacts.sh xsource [version]
# version defaults to "latest" (newest uploaded version, by created_at).
@@ -43,22 +42,6 @@ print(matches[-1]['version'])
"
}
pull_replay_fixtures() {
local version="$1"
local dest="${REPO_ROOT}/tests/fixtures/dumps"
mkdir -p "$dest"
echo "=== replay-fixtures (version ${version}) ==="
local tmp; tmp="$(mktemp -d)"
if curl -sf "${DL_BASE}/generic/replay-fixtures/${version}/replay-fixtures.zip" \
-o "${tmp}/f.zip"; then
unzip -qo "${tmp}/f.zip" -d "$dest"
echo " restored: $(ls "$dest" | wc -l) files into tests/fixtures/dumps/"
else
echo " [warn] replay-fixtures.zip not found at version ${version}" >&2
fi
rm -rf "$tmp"
}
pull_galleries() {
local version="$1"
local dest="${REPO_ROOT}/experiments/galleries"
@@ -198,13 +181,8 @@ case "$TARGET" in
[ "$VERSION" = "latest" ] && VERSION="$(resolve_latest_version xsource)"
pull_xsource "$VERSION"
;;
replay-fixtures)
VERSION="${2:-latest}"
[ "$VERSION" = "latest" ] && VERSION="$(resolve_latest_version replay-fixtures)"
pull_replay_fixtures "$VERSION"
;;
*)
echo "unknown target: $TARGET (expected galleries, montage-frames, experiment-data, report-highlights, xsource, or replay-fixtures)" >&2
echo "unknown target: $TARGET (expected galleries, montage-frames, experiment-data, report-highlights, or xsource)" >&2
exit 1
;;
esac
scripts/artifacts/push_artifacts.sh
+1 -27
View File
@@ -13,12 +13,10 @@
# scripts/artifacts/push_artifacts.sh experiment-data
# scripts/artifacts/push_artifacts.sh report-highlights
# scripts/artifacts/push_artifacts.sh xsource
# scripts/artifacts/push_artifacts.sh replay-fixtures
# scripts/artifacts/push_artifacts.sh galleries montage-frames experiment-data report-highlights
#
# Package layout (owner=dtourolle, repo=scene-actor-extraction):
# generic/galleries/<version>/gallery_<model>.h5 (one file per model)
# generic/replay-fixtures/<version>/replay-fixtures.zip (T2 dumps + their gallery)
# generic/montage-frames/<version>/<film-slug>.zip (zipped per-film frames)
# generic/experiment-data/<version>/experiment-data.zip (manifests/trajectories/results)
# generic/report-highlights/<version>/<name>.jpg (individual, hand-picked
@@ -52,29 +50,6 @@ upload() {
-o /dev/null -w " HTTP %{http_code}\n"
}
push_replay_fixtures() {
echo "=== replay-fixtures (version ${VERSION}) ==="
# T2 replay fixtures: per-frame detections, landmarks and embeddings dumped
# from a real run, so the tracker and identity stages can be replayed on CPU
# with no GPU, no models and no film. Too large for git (superhero.h5 alone
# is ~9 MB) and regenerating them needs the film plus a GPU, which CI has
# neither of — so they ship as artifacts and CI pulls them.
#
# The gallery travels with them: a dump replays against the gallery it was
# produced with, and pairing a dump with a different gallery silently
# changes every identity decision in it.
local dir="${REPO_ROOT}/tests/fixtures/dumps"
if [ ! -d "$dir" ]; then
echo " no tests/fixtures/dumps dir, skipping" >&2
return
fi
local tmp
tmp="$(mktemp -d)"
( cd "$dir" && zip -qr "$tmp/replay-fixtures.zip" . )
upload "replay-fixtures" "replay-fixtures.zip" "$tmp/replay-fixtures.zip"
rm -rf "$tmp"
}
push_galleries() {
echo "=== galleries (version ${VERSION}) ==="
local dir="${REPO_ROOT}/experiments/galleries"
@@ -174,8 +149,7 @@ for target in "$@"; do
experiment-data) push_experiment_data ;;
report-highlights) push_report_highlights ;;
xsource) push_xsource ;;
replay-fixtures) push_replay_fixtures ;;
*) echo "unknown target: $target (expected galleries, montage-frames, experiment-data, report-highlights, xsource, or replay-fixtures)" >&2; exit 1 ;;
*) echo "unknown target: $target (expected galleries, montage-frames, experiment-data, report-highlights, or xsource)" >&2; exit 1 ;;
esac
done
scripts/ci/build_builder_image.sh
-118
View File
@@ -1,118 +0,0 @@
#!/bin/bash
# build_builder_image.sh — build and publish the DP-007 CI builder image to the
# Gitea container registry.
#
# TRACES: DP-007 | PR-004
#
# Usage:
# scripts/ci/build_builder_image.sh # build only, tag v1
# scripts/ci/build_builder_image.sh --push # build and push
# scripts/ci/build_builder_image.sh --tag v2 --push # bump the pinned tag
# scripts/ci/build_builder_image.sh --no-cache # force a clean rebuild
#
# The tag is the contract with CI. .gitea/workflows/unit-tests.yml names an
# explicit tag in its `container:` block and never `latest`, so that rebuilding
# the image cannot silently change what a previous green build meant. Bumping
# the dependency set means bumping the tag AND editing the workflow — the two
# edits landing in the same commit is the point, not an inconvenience.
#
# Registry auth: this script does not log in. Do it once, out of band:
# docker login gitea.tourolle.paris
# The CI host is already authenticated this way (its cached credentials in
# ~/.docker/config.json are what the kpnpp-builder push relies on), so a
# workflow that calls this script needs no secret plumbing.
set -euo pipefail
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
REGISTRY="gitea.tourolle.paris"
OWNER="dtourolle"
IMAGE="sae-builder-cpu"
DOCKERFILE="Dockerfile.builder-cpu"
# The tag CI pins to today. Keep this in step with the `container.image` line in
# .gitea/workflows/unit-tests.yml; the workflow asserts at run time that the
# image it landed in reports this same version, so a drift shows up as a failed
# job rather than as a build against the wrong toolchain.
TAG="v1"
PUSH=0
EXTRA_ARGS=()
while [ $# -gt 0 ]; do
case "$1" in
--push) PUSH=1 ;;
--tag) TAG="${2:?--tag needs a value}"; shift ;;
--no-cache) EXTRA_ARGS+=(--no-cache) ;;
-h|--help) sed -n '2,30p' "${BASH_SOURCE[0]}"; exit 0 ;;
*) echo "error: unknown argument '$1'" >&2; exit 2 ;;
esac
shift
done
if [ "$TAG" = "latest" ]; then
echo "error: refusing to build the tag 'latest'." >&2
echo "DP-007 requires CI to pin an immutable tag. A moving 'latest' means a" >&2
echo "rebuild retroactively changes what every earlier green build proved." >&2
exit 2
fi
REF="${REGISTRY}/${OWNER}/${IMAGE}:${TAG}"
# A second tag carrying the commit that produced the image. The workflow pins
# the human-readable tag; this one is the audit trail — given any image you can
# recover the Dockerfile that built it.
SHA="$(git -C "$REPO_ROOT" rev-parse --short HEAD)"
REF_SHA="${REGISTRY}/${OWNER}/${IMAGE}:${TAG}-${SHA}"
# The Dockerfile COPYs nothing from the repository on purpose (see its closing
# comment), so the build context is an empty directory rather than the repo
# root. Sending ~1 GB of models, fixtures and experiment data to the daemon for
# a build that reads none of it is pure latency.
CONTEXT="$(mktemp -d)"
trap 'rm -rf "$CONTEXT"' EXIT
echo "=== building ${REF}"
echo " dockerfile: ${REPO_ROOT}/${DOCKERFILE}"
echo " context: (empty — the image embeds no repository content)"
echo
echo " Expect this to take a while: OpenCV 5 is compiled from source because"
echo " no Debian release ships it. That cost is paid once per image, which is"
echo " the entire reason DP-007 asks for a prebuilt image instead of"
echo " installing dependencies inside each CI run."
echo
docker build \
"${EXTRA_ARGS[@]}" \
--build-arg "IMAGE_TAG=${TAG}" \
-f "${REPO_ROOT}/${DOCKERFILE}" \
-t "${REF}" \
-t "${REF_SHA}" \
"${CONTEXT}"
echo
echo "=== built"
docker image inspect "${REF}" --format ' {{.RepoTags}} {{.Size}} bytes'
docker run --rm "${REF}" sh -c 'echo " SAE_BUILDER=$SAE_BUILDER version=$SAE_BUILDER_VERSION ort=$SAE_ORT_VERSION opencv=$SAE_OPENCV_VERSION"'
if [ "$PUSH" -eq 0 ]; then
echo
echo "Not pushed. Re-run with --push, or push by hand:"
echo " docker push ${REF}"
echo " docker push ${REF_SHA}"
exit 0
fi
echo
echo "=== pushing"
# No `latest` tag is pushed, by design. Publishing one invites a workflow to use
# it, and DP-007 exists to prevent exactly that.
docker push "${REF}"
docker push "${REF_SHA}"
echo
echo "=== published ${REF}"
echo "If this was a dependency-set change, bump the tag in"
echo " .gitea/workflows/unit-tests.yml (container.image)"
echo " scripts/ci/build_builder_image.sh (TAG, above)"
echo "in the same commit, so no run can build against an image the repository"
echo "does not describe."
scripts/ci/check_raw_cosine.py
-225
View File
@@ -1,225 +0,0 @@
#!/usr/bin/env python3
"""Enforce the AR-024 invariant: never a raw cosine, always the calibration.
TRACES: AR-024 | SR-002
docs/requirements.md gives AR-024's verification tier as "Static check -- no
bare cosine outside a tagged EXCEPTION | Grep-based; this is the invariant's
enforcement". This is that check. Until it existed the invariant was enforced
by reading, and reading missed a live violation: the identity matcher's
no-calibration fallback thresholded raw cosine distance and fed `max(0, cosine)`
into the Bayesian accumulation as though it were a posterior.
WHAT IT CHECKS, precisely, because a static check that overclaims its reach is
worse than one with a stated scope:
Every call to `cosine_similarity(...)` in C++ source must either
(a) have its result consumed immediately by a calibration -- the call is
textually wrapped in `cal_(...)`, `calibrate_(...)`, `.probability(...)`
or similar; or
(b) sit under an exception comment -- the token is `EXCEPTION:` followed by
`AR-024` and a reason -- within EXCEPTION_SCOPE_LINES above it.
Note that this file deliberately never spells that token out. The traceability
extractor scans scripts/ as source, so prose here describing the tag would be
counted as recorded exceptions; four of them were, until this was noticed. The
same trap the shared config warns about for the vendored parser tests.
Anything else is a defect, per CLAUDE.md: "treat any bare cosine comparison in
the code as a defect to be fixed".
WHAT IT DOES NOT CHECK, and why you should not read a pass as more than it is:
- It cannot follow a cosine through a variable across statements. A file that
stores `float s = cosine_similarity(a, b);` and compares `s` three lines
later is not caught. The codebase does not currently do this, and this check
exists partly to keep it that way, but it is a convention backed by review,
not by the tool.
- It says nothing about GEMM output. The similarity engine returns a whole
matrix of cosines and the matcher reads them directly; that path is correct
by inspection (every value goes through `cal_.probability`) and is not
verified here.
- A retired constant reintroduced under a new name is invisible to it.
Exit status is 0 when clean, 1 when a violation is found, 2 on a usage error.
"""
import argparse
import pathlib
import re
import sys
# How far above a use an exception tag may sit and still cover it.
# Generous, because the house style puts a paragraph of reasoning between the
# tag and the code -- but bounded, so a tag cannot silently cover a whole file.
EXCEPTION_SCOPE_LINES = 25
CPP_SUFFIXES = {".h", ".hpp", ".hxx", ".cc", ".cpp", ".cxx", ".cu", ".cuh"}
# src only, deliberately. The invariant governs what the PIPELINE decides --
# CLAUDE.md's rule is "tag the unit that decides" -- whereas a test legitimately
# asserts properties of the metric space itself (that a vector's cosine with
# itself is 1, that the annex ended up holding the spoke it should have). Those
# are measurements of the code under test, not decisions shipped to a user, and
# sweeping them in would produce a wall of blanket EXCEPTION tags that would
# devalue the tag everywhere else. Pass --source-root tests to scan them anyway.
DEFAULT_ROOTS = ["src"]
# Directories that are never this repo's code.
EXCLUDE_DIRS = {
"build", "build-ort", "external", "vendor", "__pycache__",
".git", "node_modules", "models",
}
COSINE_CALL = re.compile(r"\bcosine_similarity\s*\(")
# The result is immediately handed to a calibration. Matches the house shapes:
# cal_(cosine_similarity(a, b))
# calibrate_(cosine_similarity(a, b))
# same_person(cosine_similarity(a, b))
# cal_.probability(cosine_similarity(a, b))
CALIBRATED = re.compile(
r"(?:\b(?:cal_|cal|calibrate_|calibrate|same_person|same_person_probability)"
r"\s*(?:\.\s*probability\s*)?\(\s*|\.\s*probability\s*\(\s*)"
r"cosine_similarity\s*\("
)
EXCEPTION_TAG = re.compile(r"EXCEPT" + r"ION:\s*AR-" + r"024\b(.*)")
# The function's own definition is not a use of it.
DEFINITION = re.compile(r"^\s*(?:inline\s+|static\s+|constexpr\s+)*float\s+"
r"cosine_similarity\s*\(")
# The house style wraps long calls across lines:
# const float p = calibrate_(
# cosine_similarity(a, b));
# so the calibration and the call it guards are not always on one line. Joining
# a small window before testing is what makes this check usable on real code
# rather than a generator of false positives that trains people to ignore it.
JOIN_LOOKBEHIND = 2
def iter_sources(root: pathlib.Path, roots):
for rel in roots:
base = root / rel
if not base.exists():
continue
for p in sorted(base.rglob("*")):
if p.suffix.lower() not in CPP_SUFFIXES:
continue
if any(part in EXCLUDE_DIRS for part in p.relative_to(root).parts):
continue
yield p
def covering_exception(lines, idx):
"""Return the reason text of an exception tag covering line `idx`."""
lo = max(0, idx - EXCEPTION_SCOPE_LINES)
for j in range(idx, lo - 1, -1):
m = EXCEPTION_TAG.search(lines[j])
if m:
return m.group(1).strip(" -—*/") or "(no reason given)"
return None
def check_file(path: pathlib.Path, root: pathlib.Path):
violations, exceptions = [], []
try:
lines = path.read_text(encoding="utf-8", errors="replace").splitlines()
except OSError as e:
print(f"error: cannot read {path}: {e}", file=sys.stderr)
return violations, exceptions
rel = path.relative_to(root)
for i, line in enumerate(lines):
if not COSINE_CALL.search(line):
continue
# A comment mentioning the function is prose, not a use.
stripped = line.lstrip()
if stripped.startswith(("//", "///", "*", "/*")):
continue
if DEFINITION.match(line):
continue
# Join a small window so a call wrapped across lines is still seen as
# calibrated. Whitespace is collapsed so the join reads as one statement.
window = " ".join(
lines[max(0, i - JOIN_LOOKBEHIND):i + 1]
)
window = re.sub(r"\s+", " ", window)
if CALIBRATED.search(window):
continue
reason = covering_exception(lines, i)
if reason:
exceptions.append((rel, i + 1, line.strip(), reason))
else:
violations.append((rel, i + 1, line.strip()))
return violations, exceptions
def main():
ap = argparse.ArgumentParser(description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("--root", default=None,
help="repository root (default: the script's ../..)")
ap.add_argument("--source-root", action="append", default=None,
help="directory to scan; repeatable (default: src, tests)")
args = ap.parse_args()
root = pathlib.Path(args.root) if args.root \
else pathlib.Path(__file__).resolve().parents[2]
roots = args.source_root or DEFAULT_ROOTS
if not root.is_dir():
print(f"error: root {root} is not a directory", file=sys.stderr)
return 2
all_violations, all_exceptions, n_files = [], [], 0
for p in iter_sources(root, roots):
n_files += 1
v, e = check_file(p, root)
all_violations += v
all_exceptions += e
if n_files == 0:
# A scan that found nothing to read is a misconfiguration reporting a
# pass, which is the failure mode the traceability gate also guards.
print(f"error: scanned 0 source files under {root} ({', '.join(roots)})",
file=sys.stderr)
return 2
print("AR-024 — always the calibrated probability, never a raw cosine")
print("=" * 72)
print(f"Repo root : {root}")
print(f"Files scanned : {n_files} ({', '.join(roots)})")
print(f"Recorded excs. : {len(all_exceptions)}")
print(f"Violations : {len(all_violations)}")
if all_exceptions:
print("\nRecorded exceptions (allowed, and each one is a claim to re-read):")
for rel, ln, src, reason in all_exceptions:
print(f" {rel}:{ln} {reason}")
print(f" {src}")
if all_violations:
print("\nVIOLATIONS — a bare cosine with no recorded exception:")
for rel, ln, src in all_violations:
print(f" {rel}:{ln}")
print(f" {src}")
print("\nEvery similarity is converted through the sigmoid calibration")
print("before it is used, compared, or thresholded. A raw cosine means")
print("something different for every model, gallery and face size, and")
print("it cannot be combined with anything else.")
print("\nEither route it through the calibration, or, if the use is")
print("genuinely about the metric space rather than about a decision,")
print("record it:")
print(" // " + "EXCEPT" + "ION: AR-" + "024 <why this one is not a decision>")
print("and add a row to CLAUDE.md's agreed-exceptions table.")
return 1
print("\nOK: no bare cosine outside a recorded exception.")
return 0
if __name__ == "__main__":
sys.exit(main())
scripts/docs/run_holdout_all_models.py
+2 -3
View File
@@ -79,9 +79,8 @@ def main():
"--dump", str(dump), "--gallery", str(gallery),
"--out", str(pred_path),
"--prob-threshold", str(cfg["prob_threshold"]),
# anneal_sec and extinction_sec are both gone: presence is
# the registry's, built from track extents (AR-012/AR-013), and
# replay.py no longer windows anything itself (VR-011).
"--anneal-sec", str(cfg["anneal_sec"]),
"--extinction-sec", str(cfg["extinction_sec"]),
"--expand-gallery",
]
print(f"RUN {model}/{film['slug']}...", file=sys.stderr)
scripts/fetch_dvu.sh
-98
View File
@@ -1,98 +0,0 @@
#!/bin/bash
# fetch_dvu.sh — pull one film's character mugshots and presence annotations from
# the NIST TRECVID Deep Video Understanding development set.
#
# The DVU dev set is the reason Road to Bali is our benchmark film: it ships
# 5-7 face crops per *character*, cut from the film itself, alongside
# scene-scoped presence annotations. That matches SR-002 directly — presence is
# per scene, not per frame — and it keeps ground truth in character space, so
# scoring needs no actor->character mapping.
#
# This exists as a script, rather than as ad hoc commands, because the first
# copy of this data lived in a temp directory and was lost to a /tmp wipe,
# taking the working gallery with it.
#
# 14 films are asserted Creative Commons and need no data agreement (only the
# 5 KinoLorber test films are gated).
#
# Usage:
# scripts/fetch_dvu.sh [film] [dest]
# film default Road_To_Bali
# dest default ./dvu
set -euo pipefail
BASE="https://www-nlpir.nist.gov/projects/trecvid/dvu/dvu.development.dataset"
FILM="${1:-Road_To_Bali}"
DEST="${2:-dvu}"
mkdir -p "$DEST/images" "$DEST/scenes"
echo "[dvu] $FILM -> $DEST"
# Scene segmentation: start/end as HH:MM:SS. Note valkaama.csv line 38 carries a
# shift-key typo (01:!4:00) — parse defensively if you extend this to that film.
echo "[dvu] scene segmentation"
curl -fsSL "$BASE/scene.segmentation.reference/${FILM}.csv" \
-o "$DEST/${FILM}.csv" || echo " (missing: ${FILM}.csv)"
# Entity types: which entities are Person vs Location/Concept. Only Person rows
# become gallery identities — the images/ directory also holds Location and
# Concept crops (bedroom, boat, ...), which must not enter a face gallery.
#
# Directory and file naming are inconsistent with the film slug used elsewhere:
# the folder is Road_to_Bali (lowercase "to") while the entity file is
# RoadToBali.entity.types.txt. Both are derived here rather than assumed.
# NIST is inconsistent across all three axes, and not by a rule worth deriving:
# Road to Bali is Road_To_Bali.csv / Road_to_Bali/ / RoadToBali.entity.types.txt,
# while SuperHero is SuperHero.csv / superHero/ / superhero.entity.types.txt.
# Defaults cover the Bali shape; override per film rather than guessing.
# KG_DIR=superHero KG_FILE=superhero scripts/fetch_dvu.sh SuperHero dvu-hero
KG_DIR="${KG_DIR:-${FILM//_To_/_to_}}"
KG_FILE="${KG_FILE:-$(echo "$FILM" | sed -E 's/_([a-z])/\U\1/g; s/_//g')}"
echo "[dvu] entity types ($KG_DIR/$KG_FILE)"
curl -fsSL "$BASE/movie_knowledge_graph/${KG_DIR}/${KG_FILE}.entity.types.txt" \
-o "$DEST/${FILM}.entity.types.txt" || echo " (missing: entity types)"
# Character face crops. Names are discovered from the directory listing rather
# than probed as <Character>_N, since the crop count varies per character and
# the listing is authoritative.
echo "[dvu] character mugshots"
PERSONS="$DEST/persons.txt"
if [ -f "$DEST/${FILM}.entity.types.txt" ]; then
grep -iE "person" "$DEST/${FILM}.entity.types.txt" \
| sed -E 's/[[:space:]]*[:,].*$//' | tr -d '\r' \
| awk '{print tolower($1)}' | sort -u > "$PERSONS"
fi
curl -fsSL "$BASE/movie_knowledge_graph/${KG_DIR}/images/" 2>/dev/null \
| grep -oE 'href="[^"?/][^"]*\.png"' | sed -E 's/href="//; s/"//' | sort -u \
> "$DEST/all_images.txt"
while read -r img; do
[ -z "$img" ] && continue
# Strip the trailing _N to recover the entity name.
who="$(echo "$img" | sed -E 's/_[0-9]+\.png$//' | awk '{print tolower($0)}')"
if [ -s "$PERSONS" ] && ! grep -qx "$who" "$PERSONS"; then
continue # Location/Concept crop, not a face
fi
curl -fsSL "$BASE/movie_knowledge_graph/${KG_DIR}/images/${img}" \
-o "$DEST/images/${img}" 2>/dev/null || rm -f "$DEST/images/${img}"
done < "$DEST/all_images.txt"
# Per-scene knowledge graphs. A Person->Location edge means that person was
# present for the whole scene. Some of these contain a stray ", ," that breaks
# strict JSON parsers.
echo "[dvu] scene graphs"
for n in $(seq 1 60); do
curl -fsSL "$BASE/scenes_knowledge_graphs/${FILM//_/ }-${n}.json" \
-o "$DEST/scenes/${FILM}-${n}.json" 2>/dev/null \
|| curl -fsSL "$BASE/scenes_knowledge_graphs/${FILM}-${n}.json" \
-o "$DEST/scenes/${FILM}-${n}.json" 2>/dev/null \
|| rm -f "$DEST/scenes/${FILM}-${n}.json"
done
echo "[dvu] done:"
echo " mugshots: $(ls "$DEST/images" 2>/dev/null | wc -l)"
echo " scenes: $(ls "$DEST/scenes" 2>/dev/null | wc -l)"
echo " csv: $([ -f "$DEST/${FILM}.csv" ] && echo yes || echo no)"
scripts/filter_gallery.py
+1 -2
View File
@@ -77,8 +77,7 @@ def main():
if missing > 0:
print(f"[warn] {missing} cast member(s) not present in gallery (not yet embedded)", file=sys.stderr)
# TRACES: GR-004 | SR-001
# a filtered gallery holds the SAME vectors as its
# TRACES: GR-004 | SR-001 — a filtered gallery holds the SAME vectors as its
# source, so it inherits the source's binding. Dropping the stamp here would
# silently launder a stamped gallery into an unstamped one.
save_gallery_hdf5({"actors": actors}, Path(args.output),
scripts/make_fixtures.sh
+7 -14
View File
@@ -19,16 +19,13 @@
# on a full channel (AR-004). Before that fix the same command produced
# different dumps run to run, since what got dropped depended on timing.
#
# Source: hero/ — SuperHero, from the TRECVID DVU development set. Chosen over
# SuperHero on face scale: Bali reference crops had a median detected face of
# 27 px against a 69 px maximum, so every reference was upscaled far past what
# the embedder was trained for. SuperHero is 69 px median, 241 px max. That matters: derived
# Source: bali/ — Road to Bali (1952), public domain. That matters: derived
# fixtures can be committed, where anything cut from a copyrighted title could
# not live in the repository at all.
set -euo pipefail
REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
CLIPS="${CLIPS:-$REPO/../hero}"
CLIPS="${CLIPS:-$REPO/../bali}"
GALLERY="${GALLERY:-$REPO/gallery_lvface.h5}"
BIN="${BIN:-$REPO/build/scene_analyze}"
OUT="$REPO/tests/fixtures/dumps"
@@ -36,12 +33,8 @@ OUT="$REPO/tests/fixtures/dumps"
# Pinned. Changing either invalidates every committed fixture.
# fps 5 — 1 fps over a 77 s clip is 77 frames, too thin to exercise an
# extinction window measured in tens of seconds.
# min-face — 32 px. This is a *fixture* setting, deliberately below AR-002's
# production floor of 40 px (VR-013, measured end to end): the
# corpus is 480x360, where faces run 40-80 px, so pinning at 40
# would thin the dumps for reasons unrelated to what they test.
# 32 px is where VR-005 still shows 98.1% TPI, so the faces kept
# are identifiable; it is not the threshold the pipeline ships.
# min-face — 32 px, the VR-005 measured floor (98.1% TPI). The corpus is
# 480x360, so a stricter value would reject most faces present.
FPS=5
MIN_FACE_PX=32
@@ -51,12 +44,12 @@ MIN_FACE_PX=32
mkdir -p "$OUT"
for clip in "$CLIPS"/SuperHero-*.webm; do
for clip in "$CLIPS"/Road_To_Bali-*.webm; do
n="$(basename "$clip" .webm)"; n="${n##*-}"
echo "── superhero_$n"
echo "── bali_$n"
"$BIN" --movie "$clip" --gallery "$GALLERY" \
--fps "$FPS" --min-face-px "$MIN_FACE_PX" \
--dump-embeddings "$OUT/superhero_$n.h5" \
--dump-embeddings "$OUT/bali_$n.h5" \
--output /dev/null 2>&1 | grep -E "wrote|dropped" || true
done
scripts/make_gallery.py
+1 -2
View File
@@ -178,8 +178,7 @@ def main():
output = Path(args.output)
image_root = Path(args.image_dir) if args.image_dir else output.parent / "images"
# TRACES: GR-004 | SR-001
# stamp with the model actually loaded, resolved
# TRACES: GR-004 | SR-001 — stamp with the model actually loaded, resolved
# through the same helper load_embedder uses so the two cannot diverge.
arcface_path = resolve_arcface(args.models_dir, args.arcface)
embedder = load_embedder(args.build_dir, args.models_dir, args.arcface)
scripts/make_jellyfin_gallery.py
+1 -2
View File
@@ -453,8 +453,7 @@ def main():
existing_actors = {}
if args.merge and output.is_file():
existing = load_gallery_hdf5(output)
# TRACES: GR-004 | SR-001
# --merge keeps the existing actors' vectors and
# TRACES: GR-004 | SR-001 — --merge keeps the existing actors' vectors and
# embeds the new ones with THIS model. If they disagree, the result is one
# gallery holding two incompatible embedding spaces, which is worse than a
# mismatched gallery: no later check can separate them again.
scripts/movienet_eval.py
+1 -2
View File
@@ -62,8 +62,7 @@ def main():
args = p.parse_args()
embedder = load_embedder(args.build_dir, args.models_dir, args.arcface)
# TRACES: GR-004 | SR-001
# match() below is a bare dot product against the
# TRACES: GR-004 | SR-001 — match() below is a bare dot product against the
# gallery's vectors; if the gallery came from another model those numbers are
# noise wearing a similarity's clothes.
verify_gallery_stamp(args.gallery,
scripts/optimizer/SCHEMA.md
+4 -49
View File
@@ -1,4 +1,4 @@
# Embedding-dump HDF5 schema (v2)
# Embedding-dump HDF5 schema (v1)
One file per analysed title. Captures the pipeline state at the `EmbeddedSceneFrame`
channel — i.e. after decode → detect → align → embed, but **before** tracking and
@@ -18,7 +18,7 @@ variable-length HDF5 types and reads straight into numpy.
```
/ (root)
attrs:
schema_version : int = 2
schema_version : int = 1
embed_dim : int = 512
# ── what produced the vectors (GR-004) ──────────────────────────────────
@@ -60,12 +60,6 @@ variable-length HDF5 types and reads straight into numpy.
landmarks : float32 [N, 10] 5 (x,y) pairs, SCRFD/ArcFace order,
same space as bbox
confidence : float32 [N] detector confidence
# ── embedding input quality (AR-028), v2 onward ─────────────────────────
sharpness : float32 [N] normalised Laplacian variance on the
112x112 aligned crop (AR-029)
alignment_residual : float32 [N] RMS landmark misfit in canonical px,
after the AR-005 similarity fit (AR-030)
```
`F` = number of sampled frames, `N` = total faces (= sum of face_count).
@@ -93,9 +87,8 @@ exactly the fact the committed fixtures needed to state.)
Reading is by name with a default or an existence check on **both** sides —
`replay.py` (`f.attrs.get(...)`) and `read_dump_provenance()` in
`src/nodes/embedding_dump_node.hpp` (`attrExists`). So the attributes are
additive and did not themselves move `schema_version` off 1: a pre-VR-010 dump
still loads, and a post-VR-010 dump still reads on old code. (AR-028 later took
it to 2 by adding *datasets* — see below.)
additive and `schema_version` stays 1: a pre-VR-010 dump still loads, and a
post-VR-010 dump still reads on old code.
A missing attribute means **unknown**, never a default value. Substituting
`detector_conf = 0.5` for a dump that does not say so manufactures the provenance
@@ -104,40 +97,6 @@ provenance is unknown is worse than no fixture, because it will be trusted."*
The committed `tests/fixtures/dumps/*.h5` predate VR-010 and carry none of these
attributes; re-dump to bind them, as with GR-004.
## Embedding input quality (AR-028) — and why this one bumps the version
`sharpness` and `alignment_residual` are two of the three AR-028 quality axes,
written beside the embedding they describe. **The third axis, size, is already
here**: it is `bbox`, scaled by `bbox_upscale` to reach the original resolution
AR-002 thresholds in. It is not duplicated into a third column, because that
would put the same quantity in two coordinate spaces inside one file — the trap
the `bbox_upscale` note below records — and the copy is the one that drifts.
The vector is **carried, not consumed**. Nothing in the pipeline thresholds or
discounts on it yet; VR-012 locates the knees from these columns, which is only
possible if they were recorded at inference. A study cannot recover how sharp a
face was from an embedding, any more than it can recover which model produced it.
**This is the change that bumps `schema_version` to 2**, where VR-010's
attributes did not. The rule is unchanged — a bump is for the *datasets* — and
so is the reason behind it. Readers are fine either way: `replay.py` and
`test_replay_fixtures.cpp` take these datasets by name with an existence check,
so a v1 dump still replays and loses only what it never had. The version exists
for a *consumer of the quality vector*, which otherwise cannot tell **"this
film's faces were never scored"** from **"this film's faces scored zero"** —
sharpness 0 is a real reading, meaning a featureless crop. That is the same
distinction `scene_detect` exists to make, and it is equally unrecoverable from
the arrays.
A v1 dump reports the vector as **unknown, never as a default** — `load_frames`
omits the keys rather than filling zeros, and the C++ side leaves the
`DetectedFace` fields at their -1 "unscored" sentinel. Re-dump to acquire it;
there is no migration, for the same reason GR-004 has none.
> The committed `tests/fixtures/dumps/*.h5` are v1 and carry no quality vector.
> Re-dumping needs a GPU host (`scripts/make_fixtures.sh`), so until that runs,
> anything driven from the fixtures sees the sentinel.
## Model binding (GR-004)
`embedder_model` / `embedder_sha256` record which embedder produced every vector
@@ -185,7 +144,3 @@ one never received. Two further reasons:
original resolution (see above).
- A frame with no faces has `face_count == 0` (still gets a row, so timestamps stay dense).
- EOF sentinel frames are NOT written.
- v2 onward: `sharpness` and `alignment_residual` are `[N]`, parallel to
`confidence`, so face *i*'s quality indexes with the same slice as its
embedding. Both are `>= 0` for any face the aligner admitted; a negative value
means unscored and must never be read as a quality.
scripts/optimizer/fetch_missing_actors.py
+2 -4
View File
@@ -106,8 +106,7 @@ def fetch(missing_path, out_path, token, build_dir, models_dir, arcface,
f"(wiki={n_via_wikidata}) no_tmdb={n_no_tmdb} no_img={n_no_img} "
f"no_face={n_no_face}", file=sys.stderr)
# TRACES: GR-004 | SR-001
# the legacy JSON gallery carries the same stamp as
# TRACES: GR-004 | SR-001 — the legacy JSON gallery carries the same stamp as
# the HDF5 one; src/gallery/gallery_store.cpp reads it from either.
Path(out_path).write_text(json.dumps({"embedder": stamp, "actors": actors}, indent=2))
n_emb = sum(len(a["embeddings"]) for a in actors)
@@ -121,8 +120,7 @@ def fetch(missing_path, out_path, token, build_dir, models_dir, arcface,
def merge(base_path, add_path, out_path):
base = json.loads(Path(base_path).read_text())
add = json.loads(Path(add_path).read_text())
# TRACES: GR-004 | SR-001
# merging two galleries from different models makes
# TRACES: GR-004 | SR-001 — merging two galleries from different models makes
# ONE file containing two incompatible embedding spaces. Nothing downstream can
# ever untangle that, so this is the one place the check must run before, not
# after, the write.
scripts/optimizer/optimize.py
+3 -18
View File
@@ -17,7 +17,7 @@ point from the trajectory (--trajectory).
Usage:
python scripts/optimizer/optimize.py --manifest films.json \
--gallery gallery_arcface_w600k_r50.json \
--params prob_threshold:0.5:0.999 ownership_logodds:0.5:4 track_alpha:0:1 \
--params prob_threshold:0.5:0.999 anneal_sec:1:30 extinction_sec:1:15 \
--popsize 20 --maxiter 25 --trajectory traj.json
"""
from __future__ import annotations
@@ -199,8 +199,7 @@ def main():
if not Path(f["dump"]).exists():
sys.exit(f"[opt] missing dump for {f['name']}: {f['dump']}")
# TRACES: GR-004 | SR-001
# every (dump, gallery) pair is checked ONCE here,
# TRACES: GR-004 | SR-001 — every (dump, gallery) pair is checked ONCE here,
# before the first evaluation. A DE sweep is thousands of replays; discovering
# a cross-model pair at the end (or never) means every number it produced was
# noise. Each replay subprocess re-checks its own pair anyway.
@@ -229,20 +228,6 @@ def main():
cfg = {}
for k, v in zip(names, x):
cfg[k] = int(round(v)) if k in int_knobs else float(v)
# The expansion band is [lo, hi]; independent DE bounds can invert it,
# and an inverted band admits nothing (track_gallery.hpp). Order them so
# every candidate is a valid band rather than wasting evals on empties.
if "expand_band_lo" in cfg and "expand_band_hi" in cfg:
lo, hi = sorted((cfg["expand_band_lo"], cfg["expand_band_hi"]))
cfg["expand_band_lo"], cfg["expand_band_hi"] = lo, max(hi, lo + 1e-3)
# presence_flood is a continuous DE knob (bounds 0:1) standing in for a
# boolean: >=0.5 selects flood-fill presence. It maps to presence_mode,
# which is what replay/the bindings read; track_extent is the default so
# the knob is simply omitted below the threshold.
if "presence_flood" in cfg:
flood = cfg.pop("presence_flood") >= 0.5
if flood:
cfg["presence_mode"] = "flood"
return cfg
def objective(x):
@@ -253,7 +238,7 @@ def main():
rec = {"eval": evals[0], "config": cfg, **m, "t": round(time.time() - t0, 1)}
traj.append(rec)
print(f"[opt] eval {evals[0]:3d} thr={cfg['prob_threshold']:.2f} "
f"own={cfg.get('ownership_logodds', float('nan')):.2f} → "
f"ann={cfg['anneal_sec']:.0f} ext={cfg['extinction_sec']:.1f} → "
f"F1={m['f1']*100:.1f}% P={m['precision']*100:.1f}% R={m['recall']*100:.1f}% "
f"agree={m.get('agreement', 0)*100:.1f}% misID={m.get('FPI_misid', 0)}",
file=sys.stderr)
scripts/optimizer/reembed_gallery.py
+1 -2
View File
@@ -59,8 +59,7 @@ def main():
ref = load_gallery_hdf5(Path(args.ref))
images_root = Path(args.images)
embedder = load_embedder(args.build_dir, args.models_dir, args.arcface)
# TRACES: GR-004 | SR-001
# this script exists to produce a gallery in a
# TRACES: GR-004 | SR-001 — this script exists to produce a gallery in a
# DIFFERENT model's space from the reference. The output must therefore never
# inherit the reference's stamp; it carries the stamp of --arcface, which is
# the whole point of the bake-off being safe to run.
scripts/optimizer/replay.py
+102 -221
View File
@@ -2,22 +2,18 @@
"""
replay.py — replay a dumped embedding HDF5 through the real KPN downstream nodes.
TRACES: VR-002, VR-011 | PR-002
TRACES: VR-002 | PR-002
Reads an embedding dump (scripts/optimizer/SCHEMA.md), feeds each frame as an
EmbeddedSceneFrame into a Python-assembled KPN network wiring the *real* C++
face_tracker → identity_matcher → frame_annotation → result_sink, and reads back
the truth file that sink wrote. No decode, no GPU embedding — only the cheap
downstream tail runs, so a sweep can vary Config knobs freely.
The sink is part of the network, not a Python reimplementation of it. That is
VR-011: presence comes from TrackRegistry claims, so a replayed window and a
scene_analyze window are produced by the same code rather than by two functions
that agreed once. See [[kpn-python-replay-optimizer]].
face_tracker → identity_matcher → scene_tracker, and returns the same presence-window
JSON that scene_analyze's result_sink produces (minimal schema). No decode, no GPU
embedding — only the cheap downstream tail runs, so a sweep can vary Config knobs
freely. See [[kpn-python-replay-optimizer]].
CLI:
python scripts/optimizer/replay.py --dump film.h5 --gallery gallery.json \
--out replayed.json [--prob-threshold 0.99] [--track-extinction-sec 5] ...
--out replayed.json [--prob-threshold 0.99] [--anneal 10] ...
"""
from __future__ import annotations
@@ -61,27 +57,12 @@ def load_frames(dump_path: str, min_conf: float = 0.0):
ts = f["frames/timestamp_sec"][:]
fidx = f["frames/frame_idx"][:]
cut = f["frames/is_cut"][:]
# is_scene_boundary is present only in scene-detect dumps; a dump made
# without --scene-detect has no such dataset. Read as all-false rather
# than a default, so flood-fill on such a dump is a clean no-op.
if "frames/is_scene_boundary" in f:
scb = f["frames/is_scene_boundary"][:]
else:
scb = np.zeros(len(ts), dtype=np.uint8)
off = f["frames/face_offset"][:]
cnt = f["frames/face_count"][:]
emb = f["faces/embedding"][:]
bbox = f["faces/bbox"][:]
lmk = f["faces/landmarks"][:]
conf = f["faces/confidence"][:]
# TRACES: AR-028 | SR-002
# The quality vector, present from schema v2. A v1 dump predates AR-028
# and simply has no such dataset — read as absent, never as a default,
# so a face from an old dump stays at the C++ -1 "unscored" sentinel
# rather than acquiring a fabricated sharpness of 0 (which is a real
# value on this axis, meaning a featureless crop).
qual = {k: f[f"faces/{k}"][:] for k in ("sharpness", "alignment_residual")
if f"faces/{k}" in f}
movie = f.attrs.get("movie", "")
fps = float(f.attrs.get("sample_fps", 1.0))
@@ -95,61 +76,41 @@ def load_frames(dump_path: str, min_conf: float = 0.0):
sel = np.where(m)[0]
frames.append({
"timestamp_sec": float(ts[i]), "frame_idx": int(fidx[i]),
"is_cut": bool(cut[i]), "is_scene_boundary": bool(scb[i]), "eof": False,
"is_cut": bool(cut[i]), "eof": False,
"bbox": np.ascontiguousarray(bbox[keep][sel], dtype=np.float32),
"landmarks": np.ascontiguousarray(lmk[keep][sel], dtype=np.float32),
"confidence": np.ascontiguousarray(c[sel], dtype=np.float32),
"embeddings": np.ascontiguousarray(emb[keep][sel], dtype=np.float32),
**{k: np.ascontiguousarray(v[keep][sel], dtype=np.float32)
for k, v in qual.items()},
})
else:
frames.append({
"timestamp_sec": float(ts[i]), "frame_idx": int(fidx[i]),
"is_cut": bool(cut[i]), "is_scene_boundary": bool(scb[i]), "eof": False,
"is_cut": bool(cut[i]), "eof": False,
"bbox": np.ascontiguousarray(bbox[keep], dtype=np.float32),
"landmarks": np.ascontiguousarray(lmk[keep], dtype=np.float32),
"confidence": c,
"embeddings": np.ascontiguousarray(emb[keep], dtype=np.float32),
**{k: np.ascontiguousarray(v[keep], dtype=np.float32)
for k, v in qual.items()},
})
last_ts = float(ts[-1]) if len(ts) else 0.0
frames.append({"timestamp_sec": last_ts, "eof": True})
return frames, str(movie), fps
def replay(dump_path: str, gallery: str, cfg: dict, build_dir: str,
out_path: str, stop: bool = True, raw_out: str | None = None,
eof_timeout: float = 300.0) -> dict:
"""Run the dump through the real KPN chain and return the truth file it wrote.
def replay(dump_path: str, gallery: str, cfg: dict, build_dir: str, stop: bool = True,
raw_out: str | None = None) -> dict:
"""Run the dump through the real KPN chain; return minimal-schema presence JSON.
TRACES: VR-011, VR-002 | PR-002
cfg may include "detector_conf" to prune dumped detections below that confidence
(upward-only from the 0.5 dump floor) before matching.
`out_path` is where the C++ sink writes. That is the change VR-011 makes:
the presence windows in that file are built by ResultSinkFunc from
TrackRegistry claims -- the extent of a track an actor owned (AR-012),
ending at the last sighting (AR-013) -- and are byte-for-byte the same
construction scene_analyze ships. This function used to build them itself,
in Python, by annealing gaps between per-frame detections, which is what the
pipeline did BEFORE AR-012. A sweep tuned against that was tuning a contract
the shipped code had stopped honouring.
cfg may include "detector_conf" to prune dumped detections below that
confidence (upward-only from the 0.5 dump floor) before matching.
raw_out: if set, also write per-frame annotations as JSON lines for the
montage renderers. Derived from the truth file's own `frames` array rather
than tapped separately out of the network -- see write_raw_frames.
eof_timeout: how long to wait for the sink to write. A replay that never
reaches EOF is a wedged pipeline, and returning an empty result would look
like a film with no cast rather than like a failure."""
raw_out: if set, also write the raw per-frame annotations (timestamp, actor_idx,
name, bbox, similarity — one entry per input frame, before merging into windows)
as JSON lines to this path. Needed to draw bounding boxes on extracted frames;
the merged window schema returned by this function has no per-frame bbox."""
sys.path.insert(0, build_dir)
import sae_kpn
# TRACES: GR-004 | SR-001
# checked here, before any network is built, so a
# TRACES: GR-004 | SR-001 — checked here, before any network is built, so a
# cross-model replay dies with one readable error instead of producing a
# plausible-looking score. add_identity_matcher re-checks it C++-side below;
# that is the backstop for any other caller of the binding.
@@ -178,171 +139,101 @@ def replay(dump_path: str, gallery: str, cfg: dict, build_dir: str,
time.sleep(0.05)
return eof
# TRACES: VR-011 | AR-004 | PR-002
# Purely a throughput and memory choice, and that is the point: the answer
# must not depend on it. It used to be `len(frames) * 2 + 64` -- the whole
# film -- to dodge a PyNode overflow drop that AR-004 has since replaced
# with parking.
#
# Removing backpressure that way was catastrophic and silent. The registry
# reaped on the TRACKER's clock while evidence arrived later from the
# matcher, so a deep channel closed tracks before their votes landed: on the
# SuperHero fixture, capacity 32 gave 5 actors and capacity 10322 gave 0,
# from identical input.
#
# The fix was NOT to bound this against track_extinction_sec. That would put
# an algorithm constant in charge of a throughput knob and leave presence a
# function of scheduling. The registry now reaps on the matcher's evidence
# watermark (TrackRegistry::advance_evidence), so a vote cannot be late by
# construction and this number is free again.
cap = 64
# Channel capacity must exceed the frame count so the fast source can't overflow
# a downstream FIFO before the serial reader drains it — PyNode DROPS on overflow,
# which would silently truncate the replay. Size to the whole film + slack.
# Every channel gets capacity ≥ the whole film so NOTHING can ever overflow-drop:
# the source can push all frames before any downstream node has drained, and a
# dropped frame silently corrupts the score. Memory is cheap (a few k pointers);
# correctness is not. Generous slack on top.
cap = len(frames) * 2 + 64
sae_kpn.add_node_python(net, "replay", source, [], ["EmbeddedSceneFrame"], cap)
# TRACES: VR-011, VR-002 | DP-001 | PR-002
# One call builds tracker -> matcher -> annotation -> sink in the only order
# that works (the matcher fits the calibration the tracker needs, and the
# sink needs the registry's claims). This used to be three factory calls
# assembled here, which is how the seam broke: the ordering constraint could
# not be expressed, so the tracker was built from a Config alone long after
# it had started requiring a registry and a calibration.
cfg = dict(cfg)
cfg["output_path"] = out_path
cfg["movie_path"] = movie
cfg["sample_fps"] = fps
# Verbosity 1 (standard) adds the per-frame array; only pay for it when the
# caller wants raw frames, since it retains every annotation in memory.
cfg["verbosity"] = 1 if raw_out else 0
sae_kpn.add_pipeline(net, gallery, cfg, cap,
stamp["model_name"], stamp["model_sha256"])
sae_kpn.add_face_tracker(net, "tracker", cfg, cap)
sae_kpn.add_identity_matcher(net, "matcher", gallery, cfg, cap,
stamp["model_name"], stamp["model_sha256"])
sae_kpn.add_scene_tracker(net, "scene", cfg, cap)
net.connect("replay", 0, "tracker", 0)
net.connect("tracker", 0, "matcher", 0)
net.connect("matcher", 0, "annotation", 0)
net.connect("annotation", 0, "sink", 0)
net.connect("matcher", 0, "scene", 0)
net.build()
net.start()
# The sink writes on the EOF annotation. Wait for it rather than reading
# anything back through the seam: presence is the registry's answer, and the
# registry lives entirely on the C++ side.
#
# This replaces a read loop that pulled one SceneAnnotation per input frame
# and rebuilt windows in Python. That loop needed a heuristic -- "keep
# reading past eof until we've collected all n_frames annotations, or hit a
# run of 8 consecutive eofs" -- to work around a tail it was losing. None of
# that exists now: nothing is read per frame, so nothing can be lost per
# frame.
deadline = time.time() + eof_timeout
while not sae_kpn.pipeline_done(net):
if time.time() > deadline:
sae_kpn.release_pipeline(net)
raise TimeoutError(
f"replay did not finish within {eof_timeout}s "
f"({len(frames) - 1} frames); the sink never saw EOF")
time.sleep(0.02)
diag = sae_kpn.pipeline_diagnostics(net)
if stop:
net.stop()
sae_kpn.release_pipeline(net)
# TRACES: VR-011 | PR-002
# A dropped vote means the matcher lagged the tracker by more than
# track_extinction_sec of film, so evidence arrived for a track that had
# already been reaped. The result is not a slightly worse score -- it is a
# silently emptier one, and this is exactly how the whole-film capacity bug
# presented. Refuse the number rather than report it.
# A dropped vote means a vote landed on a track already reaped. The
# tracker/registry one-clock fix (candidates() and reap share the evidence
# watermark + track_extinction_sec horizon) removed the systematic case, but a
# small residual persists on some films from EOF-flush / same-tick ordering.
# The catastrophic capacity bug this guard was built for dropped THOUSANDS,
# emptying the output; a scattered fraction of a percent does not move the
# per-second F1 or the sweep rankings (measured; SESSION_STATE). So abort only
# when the drop ratio is large enough to distort the score, not on any drop.
dropped = int(diag.get("dropped_votes", 0))
total_faces = sum(len(f.get("embeddings", [])) for f in frames if not f.get("eof"))
drop_ratio = dropped / total_faces if total_faces else 0.0
kMaxDropRatio = 0.02 # 2%: well above the ~0.5% residual, far below a real bug
if dropped and drop_ratio > kMaxDropRatio:
raise RuntimeError(
f"replay dropped {dropped} identity votes ({drop_ratio:.1%} of "
f"{total_faces} faces): the matcher fell more than track_extinction_sec "
f"behind the tracker, so presence is under-reported. Lower the channel "
f"capacity (currently {cap}) or raise track_extinction_sec.")
if dropped:
print(f"[replay] tolerated {dropped} dropped votes "
f"({drop_ratio:.2%} of {total_faces} faces)", file=sys.stderr)
with open(out_path) as f:
result = json.load(f)
# Read exactly one annotation per input frame. The source emits EOF as an ordinary
# value AFTER the last frame, but the concurrent pipeline lets that EOF OVERTAKE
# the last few real frames still flowing tracker→matcher→scene. Breaking on the
# first eof therefore dropped a random tail (~0.5–1%, race-dependent). Instead we
# keep reading past eof until we've collected all n_frames annotations (or hit a
# run of consecutive eofs meaning the pipeline is genuinely drained).
n_expected = len(frames) - 1 # excludes the trailing eof frame
annotations = []
eof_streak = 0
max_reads = n_expected * 2 + 32
for _ in range(max_reads):
sa = net.read("scene", 0)
if sa.get("eof"):
eof_streak += 1
# stragglers can still arrive after an eof; only stop once we've either
# got everything or seen several eofs in a row (truly drained).
if len(annotations) >= n_expected or eof_streak >= 8:
break
continue
eof_streak = 0
annotations.append(sa)
if len(annotations) >= n_expected:
break
if raw_out:
write_raw_frames(result, raw_out)
with open(raw_out, "w") as f:
for sa in annotations:
f.write(json.dumps(sa) + "\n")
result = build_minimal(annotations, movie, fps, cfg)
if stop:
net.stop()
return result
def write_raw_frames(truth: dict, raw_out: str) -> None:
"""Per-frame annotations as JSONL, for the montage/error-frame renderers.
def build_minimal(annotations, movie, fps, cfg) -> dict:
"""Reproduce result_sink's minimal schema: per-actor annealed [start,end] windows.
TRACES: VR-011 | PR-002
Derived from the truth file's own `frames` array (verbosity 1) rather than
from a second stream tapped out of the network. One producer, one set of
numbers: a bbox drawn on a montage is now provably the bbox the sink
recorded, which it was not when Python read annotations separately.
The shape is the legacy one -- {timestamp_sec, visible_actors:[...]} with
actor_idx/bbox/name/similarity -- because dump_scene_montage.py and
dump_error_frames.py read exactly those fields, and rewriting them is not
what this requirement is about.
Mirrors ResultSinkFunc::build_actor_windows — merge each actor's detection
timestamps into windows, bridging gaps shorter than anneal_sec.
"""
with open(raw_out, "w") as f:
for fr in truth.get("frames", []):
visible = []
for a in fr.get("identified", []):
visible.append({
"actor_idx": 0, # >= 0 means "known"; the renderers
# test the sign, never the value
"name": a.get("name", ""),
"imdb_id": a.get("imdb_id", ""),
"tmdb_id": a.get("tmdb_id", ""),
"jellyfin_id": a.get("jellyfin_id", ""),
"similarity": a.get("similarity", 0.0),
"track_id": a.get("track_id", -1),
"bbox": a.get("bbox", [0, 0, 0, 0]),
})
for u in fr.get("unknowns", []):
visible.append({
"actor_idx": -1,
"name": "",
"similarity": u.get("confidence", 0.0),
"track_id": u.get("track_id", -1),
"bbox": u.get("bbox", [0, 0, 0, 0]),
})
f.write(json.dumps({"timestamp_sec": fr.get("t", 0.0),
"visible_actors": visible}) + "\n")
anneal = float(cfg.get("anneal_sec", 10.0))
info = {} # actor_idx -> identity fields
times = {} # actor_idx -> [timestamps]
for sa in annotations:
for a in sa["visible_actors"]:
if a["actor_idx"] < 0:
continue
info[a["actor_idx"]] = a
times.setdefault(a["actor_idx"], []).append(sa["timestamp_sec"])
actors = []
for idx, ts in times.items():
ts.sort()
scenes = []
ws = we = ts[0]
for t in ts[1:]:
if t - we > anneal:
scenes.append([ws, we])
ws = t
we = t
scenes.append([ws, we])
a = info[idx]
actors.append({
"name": a["name"], "imdb_id": a["imdb_id"], "tmdb_id": a["tmdb_id"],
"jellyfin_id": a["jellyfin_id"], "scenes": scenes,
})
return {"schema_version": 1, "movie": movie, "sample_fps": fps,
"anneal_sec": anneal, "actors": actors}
CFG_KEYS = ["detector_conf", "prob_threshold", "match_prior",
"track_alpha", "track_min_iou", "track_assoc_min_prob",
"track_extinction_sec",
# AR-025 ownership and evidence accumulation. Newly reachable:
# these were in-class defaults no sweep could vary, which is why
# VR-007 never covered them despite rho_max deferring to it.
"ownership_logodds", "evidence_rho_max", "evidence_admit_below",
"evidence_max_views",
# AR-018 expansion bands (probability space). Only active with
# --expand-gallery; the config comment asks for both to be swept.
"expand_band_lo", "expand_band_hi"]
# TRACES: VR-011 | PR-002
# REPLAY_LOCAL_KEYS is gone with build_minimal. It held anneal_sec, the last
# parameter this harness applied itself -- and the only reason it needed a
# separate list was that the harness was still doing windowing the pipeline had
# stopped doing. Every key is a Config key now, because every decision is the
# pipeline's.
CFG_KEYS = ["detector_conf", "prob_threshold", "match_prior", "match_threshold", "match_ratio",
"match_ratio_ceil", "track_alpha", "track_min_iou", "track_max_embed_dist",
"track_max_frames_missing", "cut_revive_sim", "cut_inactive_max_frames",
"extinction_sec", "anneal_sec"]
def main():
@@ -358,11 +249,7 @@ def main():
# per-film gallery expansion: promotes pose-varied views of confidently-identified
# actors into an in-memory annex, recovering ~+4 recall at no precision cost.
p.add_argument("--expand-gallery", action="store_true")
# Presence derivation. flood snaps each claim to its shot; needs a
# scene-detect dump (is_scene_boundary), else it no-ops back to track-extent.
p.add_argument("--presence-mode", choices=["track_extent", "flood"], default=None)
# TRACES: GR-004 | SR-001
# promote an unprovable gallery/dump binding from a
# TRACES: GR-004 | SR-001 — promote an unprovable gallery/dump binding from a
# loud warning to a hard error. Measurement sweeps should set this (or
# SAE_REQUIRE_GALLERY_STAMP=1) so no number comes from an unbound pair.
p.add_argument("--require-gallery-stamp", action="store_true")
@@ -371,21 +258,15 @@ def main():
cfg = {k: getattr(args, k) for k in CFG_KEYS if getattr(args, k) is not None}
if args.expand_gallery:
cfg["expand_gallery"] = True
if args.presence_mode:
cfg["presence_mode"] = args.presence_mode
if args.require_gallery_stamp:
cfg["require_gallery_stamp"] = True
# stop=True: PyNode::stop() sets stop_flag_ before joining, so the source
# thread's run_loop actually exits. stop=False skips that, leaving stop_flag_
# false forever — the PyNode destructor's jthread.join() then blocks forever
# (verified via gdb: stuck in the source node's run_loop, not the GEMM path).
result = replay(args.dump, args.gallery, cfg, args.build_dir,
out_path=args.out, stop=True, raw_out=args.raw_out)
# NOT rewritten here: the sink already wrote args.out, and that file is the
# artifact. Dumping `result` back over it would make this script the last
# writer of a file it did not produce -- and any formatting difference would
# be a diff between the replayed truth file and a scene_analyze one that is
# this script's doing rather than the pipeline's.
result = replay(args.dump, args.gallery, cfg, args.build_dir, stop=True,
raw_out=args.raw_out)
Path(args.out).write_text(json.dumps(result, indent=2))
print(f"[replay] {len(result['actors'])} actors → {args.out}", file=sys.stderr)
scripts/optimizer/second_score.py
+1 -9
View File
@@ -95,15 +95,7 @@ def load_pred_intervals(pred_json: dict):
for a in pred_json.get("actors", []):
keys = frozenset(keys_for(imdb_id=a.get("imdb_id"), tmdb_id=a.get("tmdb_id"),
jellyfin_id=a.get("jellyfin_id"), name=a.get("name")))
# schema_version 1: scenes is [[t0, t1], ...]; schema_version 2:
# scenes is [{"start":…, "end":…, "belief":…, "route":…}, …].
windows = []
for s in a.get("scenes", []):
if isinstance(s, dict):
windows.append((float(s["start"]), float(s["end"])))
else:
windows.append((float(s[0]), float(s[1])))
out.append((keys, windows))
out.append((keys, [(float(t0), float(t1)) for t0, t1 in a.get("scenes", [])]))
return out
scripts/optimizer/test_sae_kpn.py
+31 -74
View File
@@ -1,29 +1,16 @@
#!/usr/bin/env python3
"""
Smoke test for the sae_kpn module: assemble the real downstream pipeline
(tracker → matcher → annotation → sink) in a Python-driven KPN network, fed by a
no-input Python source node, and verify the sink writes a truth file.
TRACES: VR-011 | PR-002
Smoke test for the sae_kpn module: assemble the real downstream pipeline nodes
(face_tracker → identity_matcher → scene_tracker) in a Python-driven KPN network,
fed by a no-input Python source node, and verify SceneAnnotations flow out.
Proves the KPN-native replay path works without any numpy port of node logic.
Rewritten for `add_pipeline`. It previously called three node factories and read
SceneAnnotations back through the seam, asserting on what came out per frame.
Neither half of that survives VR-011: the factories are gone because the chain
has a construction order Python could not express, and presence is now the C++
sink's answer, derived from TrackRegistry claims. Nothing is read per frame, so
the assertions are on the file the sink writes.
Run: python scripts/optimizer/test_sae_kpn.py [gallery.json] [build_dir]
"""
import json
import sys
import tempfile
import time
from pathlib import Path
import queue
import numpy as np
from pathlib import Path
REPO = Path(__file__).resolve().parent.parent.parent
GAL = sys.argv[1] if len(sys.argv) > 1 else str(REPO / "gallery_arcface_w600k_r50.json")
@@ -44,6 +31,7 @@ def make_frame(t, n):
def main():
net = sae_kpn.Network()
sae_kpn._register_types(net)
cfg = {"prob_threshold": 0.99, "anneal_sec": 10.0, "extinction_sec": 5.0}
frames = [make_frame(float(t), 1) for t in range(3)]
frames.append({"timestamp_sec": 3.0, "eof": True})
@@ -51,67 +39,36 @@ def main():
eof_frame = {"timestamp_sec": 3.0, "eof": True}
def source():
# Emit each frame once, then keep returning EOF so the node thread stays
# responsive to stop(). The sleep matters: a no-input source is called in
# a tight loop, and hot-spinning EOFs pegs a core and floods the channel.
# Emit each frame once, then keep returning EOF (never block) so the node
# thread stays responsive to stop() after the sink has seen EOF.
i = idx[0]
idx[0] += 1
if i < len(frames):
return frames[i]
time.sleep(0.05)
return eof_frame
return frames[i] if i < len(frames) else eof_frame
with tempfile.TemporaryDirectory() as tmp:
out_path = str(Path(tmp) / "truth.json")
cfg = {
"prob_threshold": 0.99,
"track_extinction_sec": 5.0,
"output_path": out_path,
"movie_path": "sae_kpn smoke test",
"sample_fps": 1.0,
# Standard verbosity emits the per-frame array this test asserts on.
# At 0 the file carries only the actor epochs, and three random
# embeddings against a real gallery need not produce any.
"verbosity": 1,
}
sae_kpn.add_node_python(net, "replay", source, [], ["EmbeddedSceneFrame"], 8)
sae_kpn.add_face_tracker(net, "tracker", cfg, 16)
sae_kpn.add_identity_matcher(net, "matcher", GAL, cfg, 16)
sae_kpn.add_scene_tracker(net, "scene", cfg, 16)
net.connect("replay", 0, "tracker", 0)
net.connect("tracker", 0, "matcher", 0)
net.connect("matcher", 0, "scene", 0)
net.build()
net.start()
sae_kpn.add_node_python(net, "replay", source, [], ["EmbeddedSceneFrame"], 16)
# No embedder stamp: these embeddings are random, not the output of any
# model, so there is nothing truthful to claim. That warns rather than
# failing, and would be fatal under SAE_REQUIRE_GALLERY_STAMP — which is
# correct, since an unverifiable binding is exactly what it guards.
sae_kpn.add_pipeline(net, GAL, cfg, 16)
got = []
for _ in range(4):
sa = net.read("scene", 0)
got.append(sa)
if sa.get("eof"):
break
net.stop()
net.connect("replay", 0, "tracker", 0)
net.connect("tracker", 0, "matcher", 0)
net.connect("matcher", 0, "annotation", 0)
net.connect("annotation", 0, "sink", 0)
net.build()
net.start()
# The sink writes on the EOF annotation. Wait for that rather than
# reading anything back: presence lives entirely on the C++ side.
deadline = time.time() + 30.0
while not sae_kpn.pipeline_done(net):
if time.time() > deadline:
sae_kpn.release_pipeline(net)
raise TimeoutError("sink never saw EOF within 30s")
time.sleep(0.02)
net.stop()
sae_kpn.release_pipeline(net)
with open(out_path) as f:
truth = json.load(f)
per_frame = truth.get("frames", [])
assert "actors" in truth, "truth file has no actors array"
assert len(per_frame) == 3, f"expected 3 frames, got {len(per_frame)}"
# EOF is a control token, not an observation: the sink flushes on it and does
# not record it, so three inputs give three frames and never four.
assert [f["t"] for f in per_frame] == [0.0, 1.0, 2.0], "timestamps wrong"
assert all("identified" in f for f in per_frame), "missing identified"
print(f"OK: {len(per_frame)} frames through the real KPN chain, sink wrote its truth file")
non_eof = [g for g in got if not g.get("eof")]
assert len(non_eof) == 3, f"expected 3 annotations, got {len(non_eof)}"
assert got[-1].get("eof"), "expected trailing EOF"
assert [g["timestamp_sec"] for g in non_eof] == [0.0, 1.0, 2.0], "timestamps wrong"
assert all("visible_actors" in g for g in non_eof), "missing visible_actors"
print(f"OK: {len(non_eof)} annotations through the real KPN chain, EOF received")
if __name__ == "__main__":
scripts/sae_embed_loader.py
+5 -16
View File
@@ -11,7 +11,6 @@ argument, which is often None.
"""
import sys
import os
from pathlib import Path
DEFAULT_ARCFACE = "arcface_w600k_r50.onnx"
@@ -20,10 +19,8 @@ DEFAULT_ARCFACE = "arcface_w600k_r50.onnx"
def resolve_arcface(models_dir: str, arcface: str | None = None) -> str:
"""The ArcFace/LVFace ONNX path load_embedder would use for these arguments.
TRACES: GR-004 | SR-001
Single source of truth for "which model is this", so the stamp written into
a gallery can never drift from the model loaded."""
TRACES: GR-004 | SR-001 — single source of truth for "which model is this",
so the stamp written into a gallery can never drift from the model loaded."""
return arcface if arcface else str(Path(models_dir) / DEFAULT_ARCFACE)
@@ -52,21 +49,13 @@ def load_embedder(build_dir: str, models_dir: str, arcface: str | None = None,
sys.exit(f"{name} model not found: {model}\nRun: bash scripts/download_models.sh")
# A TRT-backend build cannot load .onnx; it needs pre-built engines from
# scripts/build_trt_engines.sh.
#
# These are passed only on request. The old comment here claimed they were
# "ignored by ORT" — they are not. The ORT backend treats an engine path as
# an instruction and raises, which is the right behaviour (silently ignoring
# a requested engine would be worse), but it meant that merely HAVING a
# populated trt_cache/ broke every ORT gallery build in the repo, with an
# error naming a flag the caller never set.
use_engines = os.environ.get("SAE_USE_TRT_ENGINES", "") not in ("", "0", "false")
# scripts/build_trt_engines.sh. Pass them when present (ignored by ORT).
trt = Path(models_path).parent / "trt_cache"
det_engine = trt / "scrfd.scrfd_500m_bnkps.640.fp16.engine"
arc_engine = trt / f"arcface.{Path(arcface_path).stem}.b4.fp16.engine"
return sae_embed.FaceEmbedder(
detector_path, arcface_path, conf, nms, max_side,
str(det_engine) if (use_engines and det_engine.is_file()) else "",
str(arc_engine) if (use_engines and arc_engine.is_file()) else "",
str(det_engine) if det_engine.is_file() else "",
str(arc_engine) if arc_engine.is_file() else "",
)
scripts/sae_gallery.py
+2 -4
View File
@@ -168,8 +168,7 @@ def save_gallery_hdf5(gallery: dict, output: Path, embedder: dict | None = None)
f.create_dataset("jellyfin_id", data=np.asarray(jf, dtype=object), dtype=str_t)
f.create_dataset("name", data=np.asarray(name, dtype=object), dtype=str_t)
f.create_dataset("source_images", data=np.asarray(src_images, dtype=object), dtype=str_t)
# TRACES: GR-004 | SR-001
# omitted entirely when unknown, so "unstamped"
# TRACES: GR-004 | SR-001 — omitted entirely when unknown, so "unstamped"
# round-trips as unstamped rather than as a stamp naming no model.
if not _stamp_empty(embedder):
g = f.create_group("embedder")
@@ -197,8 +196,7 @@ def load_gallery_hdf5(path: Path) -> dict:
if "source_images" in f:
src_images = [s.decode() if isinstance(s, bytes) else s
for s in f["source_images"][:]]
# TRACES: GR-004 | SR-001
# carried through so a derived gallery (filter,
# TRACES: GR-004 | SR-001 — carried through so a derived gallery (filter,
# merge, cast-restrict) keeps the binding of the gallery it came from.
stamp = None
if "embedder" in f:
scripts/validation/ground_truth.py
-5
View File
@@ -17,11 +17,6 @@ X-Ray/MovieNet key on IMDb nm-ids — see [[per-scene-presence-eval-design]].
Two sources implemented:
* XRayGroundTruth — Zenodo scene-level Amazon X-Ray CSVs (cast-in-scene).
* MovieNetGroundTruth — MovieNet-PS per-shot face annotations (on-screen faces).
Both are published corpora addressed by title, so a scoring run is reproducible
from the identifiers alone — no annotation of ours travels with the code.
TRACES: VR-004 | PR-002
"""
from __future__ import annotations
scripts/validation/min_face_size.py
+8 -2
View File
@@ -183,10 +183,16 @@ DEDUP_SIM = 1.0 - 1e-7
class Stages:
"""Thin holder so the rest of the script has one object to call."""
def __init__(self, detector: str, arcface: str, conf: float, nms: float):
def __init__(self, detector: str, arcface: str, conf: float, nms: float,
detector_engine: str = "", arcface_engine: str = ""):
# The engine paths are only consulted by a TRT-backend build, where they
# are mandatory — that backend loads a pre-built .engine and will not
# fall back to reading the .onnx. An ORT build ignores them, so passing
# them unconditionally is safe and keeps one constructor for both.
self.engine = sae_embed.FaceEmbedder(
detector_model=detector, arcface_model=arcface,
conf=conf, nms=nms, max_side=0)
conf=conf, nms=nms, max_side=0,
detector_engine=detector_engine, arcface_engine=arcface_engine)
def detect(self, img):
return self.engine.detect(img)
scripts/validation/quality_knee.py
+667
View File
@@ -0,0 +1,667 @@
#!/usr/bin/env python3
"""
quality_knee.py — VR-012: what does a blurred or small face cost in identification,
and which sharpness measure predicts it?
TRACES: VR-012, AR-028, AR-029
VR-005 located the size floor by degrading held-out gallery mugshots and watching
TPI/FPI fall. This does the same over a **joint size x blur grid**, and adds the
part that makes the result usable at inference.
Why a joint grid and not two sweeps
-----------------------------------
A 16 px face upscaled to 112 has already lost its high frequencies, so additional
blur costs it far less than it costs a 112 px one. Sweeping the axes separately
measures each in the presence of an implicit "other axis at its best" and misses
that interaction entirely — and the interaction is the whole question, because
AR-002 already gates on size and AR-029 proposes to discount on sharpness. If
identity loss turns out to be a function of the sharpness measure alone, then one
axis carries the information and discounting on both double-counts. If a
small-but-sharp and a large-but-blurred probe at equal measure lose different
amounts, the axes are genuinely separate and both belong.
Why sigma is not the answer
---------------------------
Sigma is a lab variable. At inference nothing knows how blurred a face is, so a
knee expressed in sigma cannot be acted on. What AR-028/AR-030 can consume is
measure value -> expected identity reliability
so the controlled degradation exists to *select and calibrate the measure*, and
the measure is what ships. Every candidate is therefore scored on every degraded
crop, and the candidates are ranked by how well each predicts the identification
outcome (AUC over probe-cell records), not by how smooth its ladder looks.
Protocol (VR-005's, extended)
-----------------------------
1. Every gallery actor with at least `--min-images` mugshots. At the default 3,
holding one out still leaves two references per actor.
2. Hold out ONE image per actor as the probe; the rest stay in the gallery at
native resolution. Only the probe degrades — reference mugshots are clean and
the face coming out of the video is not, which is the production case.
3. For each (size, sigma) cell: downscale the probe crop to size x size and back
to 112 (the sampling loss), then Gaussian blur at sigma canonical px (the
optical/motion loss). Resolution first, then blur, so sigma always means the
same thing in the frame AR-029 measures in, whatever the cell's size.
4. Score all five AR-029 candidates on the degraded crop, through the C++
binding.
5. Embed, match against the whole gallery, record TPI/FPI/unidentified.
Decision rule is the pipeline's: per-actor best-of-N cosine -> Platt sigmoid ->
accept if P > prob_threshold. Never a raw cosine (CLAUDE.md invariant, AR-024).
Everything runs through `sae_embed` — detection, the ArcFace warp, the embedder,
the sharpness measures and the calibration are all the shipped C++. Nothing here
re-implements a pipeline stage in numpy; the analysis on top of the recorded
numbers (AUC, knee location) is analysis and is numpy's job.
CAVEAT — FPI IS RELATIVE, NOT ABSOLUTE
--------------------------------------
False positives grow with the number of actors competing. Read FPI as a curve
across cells, not as a production rate. This runs the whole eligible gallery
rather than VR-005's 100-actor sample, so the understatement is much smaller,
but a production library is larger still.
Usage
-----
python scripts/validation/quality_knee.py \
--images images --gallery gallery_lvface.h5 \
--arcface models/LVFace-B_Glint360K.onnx \
--min-images 3 --out experiments/results/vr012_quality_knee
"""
from __future__ import annotations
import argparse
import csv
import json
import random
import sys
import time
from pathlib import Path
REPO = Path(__file__).resolve().parent.parent.parent
sys.path.insert(0, str(REPO / "scripts"))
sys.path.insert(0, str(Path(__file__).resolve().parent))
# min_face_size owns the shared scaffolding — actor discovery, the sae_embed
# locator, the Stages wrapper, the calibration-through-the-binding and the house
# plot palette. Importing it keeps one copy of each; a second copy of the
# calibration path in particular is what AR-024 exists to prevent.
import min_face_size as vr005 # noqa: E402
from min_face_size import ( # noqa: E402
DEDUP_SIM, INTERP, Stages, calibrate_gallery, discover_actors,
gallery_keys, normalise_name, probability, err,
INK, MUTED, GRID, SURFACE, BLUE, GREEN, RED, AMBER,
)
import cv2 # noqa: E402
import numpy as np # noqa: E402
import sae_embed # noqa: E402
# The five AR-029 candidates, in quality.hpp's order. Names match the binding's
# attributes so the CSV columns and the C++ fields cannot drift apart.
MEASURES = ["var_laplacian", "norm_var_laplacian", "tenengrad",
"hf_energy_ratio", "dir_min_tenengrad"]
# ── Degradation ───────────────────────────────────────────────────────────────
def disc_kernel(radius: float) -> np.ndarray:
"""The circle-of-confusion PSF of a defocused lens.
Optical defocus is **not** Gaussian, and the difference is not cosmetic. A
lens out of focus spreads a point into a uniform disc, whose transfer
function is a jinc — `2·J1(x)/x` — which crosses zero and goes negative.
Defocus therefore reverses contrast at particular spatial frequencies and
can leave *more* energy in some high bands than a Gaussian of the same
nominal width. A Gaussian MTF is strictly positive and monotonically
decreasing and does neither.
That matters here beyond realism: defocus is how a face ends up **large and
useless**. A focus pull, a shallow depth of field, an actor stepping off the
focal plane — all leave a big, confidently-detected face carrying no usable
detail, and all sail straight through a size gate. Gaussian blur was the one
family that mostly co-occurs with small faces, which is precisely why
sharpness looked redundant against AR-002 on the first grid.
The disc is supersampled 8x before downsampling so its edge is
anti-aliased; a hard-edged binary disc at small radii is a poor circle and
its spectrum carries the staircase, not the optics.
"""
ss = 8
n = int(np.ceil(radius)) * 2 + 1
hi = np.zeros((n * ss, n * ss), np.float32)
c = (n * ss - 1) / 2.0
y, x = np.ogrid[:n * ss, :n * ss]
hi[((x - c) ** 2 + (y - c) ** 2) <= (radius * ss) ** 2] = 1.0
k = hi.reshape(n, ss, n, ss).mean(axis=(1, 3))
s = k.sum()
return (k / s) if s > 0 else np.ones((1, 1), np.float32)
def motion_kernel(length: int, angle_deg: float) -> np.ndarray:
"""Linear motion blur — a camera pan or a moving subject.
Directional by construction: it destroys detail along one axis and leaves
the perpendicular axis untouched. That is the property that separates the
AR-029 candidates, since a measure normalising by total energy divides out
the loss and reads a heavy smear as mild (see tests/test_quality.cpp).
"""
k = np.zeros((length, length), np.float32)
k[length // 2, :] = 1.0
m = cv2.getRotationMatrix2D(((length - 1) / 2.0, (length - 1) / 2.0),
angle_deg, 1.0)
k = cv2.warpAffine(k, m, (length, length))
s = k.sum()
return (k / s) if s > 0 else np.ones((1, 1), np.float32)
def degrade(crop: np.ndarray, size: int, level: float, kind: str,
down: int, up: int, angle: float = 0.0) -> np.ndarray:
"""Resolution loss, then blur of the requested family.
Order matters and this one is deliberate. Sampling happens in the source
frame, so the downscale/upscale pair models a face that was `size` px when
detected. The blur is then applied in the canonical frame, so `level` means
the same number of canonical pixels in every cell of the grid — which is what
lets the two axes be read independently. Blurring first would make the
effective width depend on the cell's size, and the grid would no longer be
factorial.
`level` is the family's natural parameter: Gaussian sigma, disc radius, or
motion length in canonical px. They are NOT equivalent at equal numbers —
matching families by parameter would compare different amounts of damage, so
the analysis matches them on measured effect instead.
"""
out = crop
if size != 112:
small = cv2.resize(out, (size, size), interpolation=down)
out = cv2.resize(small, (112, 112), interpolation=up)
if level > 0:
if kind == "gaussian":
out = cv2.GaussianBlur(out, (0, 0), level, level)
elif kind == "disc":
out = cv2.filter2D(out, -1, disc_kernel(level))
elif kind == "motion":
out = cv2.filter2D(out, -1, motion_kernel(int(round(level)), angle))
else:
raise ValueError(f"unknown blur kind: {kind}")
return out
def score_sharpness(crop: np.ndarray) -> dict:
"""All five candidates, from the shipped C++ (quality.hpp)."""
s = sae_embed.assess_sharpness(np.ascontiguousarray(crop))
d = {m: float(getattr(s, m)) for m in MEASURES}
d["ok"] = bool(s.ok)
return d
# ── Analysis ──────────────────────────────────────────────────────────────────
def auc(scores: np.ndarray, positive: np.ndarray) -> float:
"""Area under the ROC for `scores` predicting `positive`, by the rank
(Mann-Whitney U) identity. 0.5 is chance; 1.0 is a measure that orders every
correctly-identified probe above every failure.
This is the ranking criterion for AR-029. A measure earns the job by
predicting *the decision the pipeline makes*, not by having a tidy response
to synthetic blur — a candidate can be beautifully monotone in sigma and
still be a poor guide to whether this particular face will be recognised.
"""
pos = scores[positive]
neg = scores[~positive]
if pos.size == 0 or neg.size == 0:
return float("nan")
order = np.argsort(np.concatenate([pos, neg]), kind="mergesort")
ranks = np.empty(order.size, dtype=np.float64)
ranks[order] = np.arange(1, order.size + 1)
# Average ranks over ties, or a measure with many equal values is scored
# arbitrarily by input order.
vals = np.concatenate([pos, neg])
sv = vals[order]
i = 0
while i < sv.size:
j = i
while j + 1 < sv.size and sv[j + 1] == sv[i]:
j += 1
if j > i:
ranks[order[i:j + 1]] = ranks[order[i:j + 1]].mean()
i = j + 1
r_pos = ranks[:pos.size].sum()
return float((r_pos - pos.size * (pos.size + 1) / 2) / (pos.size * neg.size))
def knee_from_measure(records: list[dict], measure: str, retention: float,
n_bins: int = 20) -> dict:
"""Where on `measure`'s own scale does identification start to fall apart?
Bins the probe-cell records by measure value and reports the TPI rate in
each. The threshold is the lowest bin edge whose bin and every bin above it
retain `retention` of the undegraded control's TPI rate — a stated rule, so
changing the answer means changing the rule rather than picking a number.
"""
vals = np.array([r[measure] for r in records], dtype=np.float64)
tpi = np.array([r["outcome"] == "TPI" for r in records])
control = np.array([r["size_px"] == 112 and r["sigma"] == 0.0 for r in records])
if control.sum() == 0:
return {}
floor = retention * float(tpi[control].mean())
# Quantile edges: the measures have wildly different scales and heavy tails,
# so equal-width bins would put almost everything in one bucket.
edges = np.unique(np.quantile(vals, np.linspace(0, 1, n_bins + 1)))
if edges.size < 3:
return {}
idx = np.clip(np.digitize(vals, edges[1:-1]), 0, edges.size - 2)
bins = []
for b in range(edges.size - 1):
m = idx == b
if m.sum() == 0:
continue
bins.append({"lo": float(edges[b]), "hi": float(edges[b + 1]),
"n": int(m.sum()), "tpi_rate": float(tpi[m].mean()),
"fpi_rate": float(np.mean([r["outcome"] == "FPI"
for r, k in zip(records, m) if k]))})
# Walk down from the top; the threshold is where retention first breaks.
thr = None
for b in reversed(bins):
if b["tpi_rate"] < floor:
thr = b["hi"]
break
return {"measure": measure, "control_tpi": float(tpi[control].mean()),
"tpi_floor": floor, "threshold": thr, "bins": bins}
# ── Plot ──────────────────────────────────────────────────────────────────────
def write_plots(cells: list[dict], records: list[dict], ranking: list[dict],
out_png: Path, meta: dict) -> None:
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
plt.rcParams.update({
"figure.facecolor": SURFACE, "axes.facecolor": SURFACE,
"savefig.facecolor": SURFACE, "text.color": INK,
"axes.edgecolor": MUTED, "axes.labelcolor": INK,
"xtick.color": MUTED, "ytick.color": MUTED,
"axes.grid": True, "grid.color": GRID, "grid.linewidth": 0.8,
"axes.spines.top": False, "axes.spines.right": False,
})
sizes = sorted({c["size_px"] for c in cells})
sigmas = sorted({c["sigma"] for c in cells})
fig, axes = plt.subplots(1, 3, figsize=(17, 5.4))
# (a) the joint grid as TPI heat map
grid = np.full((len(sigmas), len(sizes)), np.nan)
for c in cells:
grid[sigmas.index(c["sigma"]), sizes.index(c["size_px"])] = 100 * c["tpi_rate"]
im = axes[0].imshow(grid, origin="lower", aspect="auto", cmap="viridis",
vmin=0, vmax=100)
axes[0].set_xticks(range(len(sizes)), [str(s) for s in sizes])
axes[0].set_yticks(range(len(sigmas)), [f"{s:g}" for s in sigmas])
axes[0].set_xlabel("probe size before upscaling (px)")
axes[0].set_ylabel("Gaussian sigma (canonical px)")
axes[0].set_title("TPI % over the joint grid", fontsize=11, loc="left")
axes[0].grid(False)
fig.colorbar(im, ax=axes[0], fraction=0.046)
# (b) TPI against the winning measure — the curve a discount is built from
best = ranking[0]["measure"]
vals = np.array([r[best] for r in records])
tpi = np.array([r["outcome"] == "TPI" for r in records])
edges = np.unique(np.quantile(vals, np.linspace(0, 1, 21)))
centres, rates = [], []
for i in range(edges.size - 1):
m = (vals >= edges[i]) & (vals <= edges[i + 1])
if m.sum() > 20:
centres.append(0.5 * (edges[i] + edges[i + 1]))
rates.append(100 * tpi[m].mean())
axes[1].plot(centres, rates, "-o", color=GREEN, lw=2)
axes[1].set_xscale("log")
axes[1].set_xlabel(f"{best} (log scale)")
axes[1].set_ylabel("TPI %")
axes[1].set_title(f"identification vs the measure\nbest predictor: {best} "
f"(AUC {ranking[0]['auc']:.3f})", fontsize=11, loc="left")
# (c) how well each candidate predicts the decision
names = [r["measure"] for r in ranking]
aucs = [r["auc"] for r in ranking]
axes[2].barh(range(len(names)), aucs, color=BLUE)
axes[2].axvline(0.5, color=RED, lw=1.4, ls="--")
axes[2].set_yticks(range(len(names)), names, fontsize=9)
axes[2].set_xlim(0.4, 1.0)
axes[2].set_xlabel("AUC — predicts correct identification")
axes[2].set_title("AR-029 candidate ranking", fontsize=11, loc="left")
axes[2].invert_yaxis()
fig.suptitle(f"VR-012 — quality knee, {meta['model']}, {meta['n_actors']} actors, "
f"{meta['n_probes']} probes x {len(cells)} cells",
fontsize=12, x=0.01, ha="left")
fig.tight_layout(rect=(0, 0.02, 1, 0.97))
out_png.parent.mkdir(parents=True, exist_ok=True)
fig.savefig(out_png, dpi=150)
plt.close(fig)
# ── Main ──────────────────────────────────────────────────────────────────────
def main() -> int:
p = argparse.ArgumentParser(
description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
p.add_argument("--images", default=str(REPO / "images"))
p.add_argument("--gallery", default=str(REPO / "gallery_lvface.h5"))
p.add_argument("--out", default=str(REPO / "experiments/results/vr012_quality_knee"))
p.add_argument("--actors", type=int, default=0,
help="cap the actor pool (0 = every eligible actor, the default: "
"FPI is gallery-size dependent and the whole gallery is the "
"least understated estimate available)")
p.add_argument("--min-images", type=int, default=3,
help="minimum mugshots to be eligible (default 3, so holding one "
"out still leaves two references)")
p.add_argument("--seed", type=int, default=0)
p.add_argument("--sizes", default="16,24,32,48,64,112",
help="probe sizes before upscaling; 112 is undegraded")
p.add_argument("--sigmas", default="0,0.5,1,1.5,2,3",
help="blur level in canonical px; 0 is unblurred. Meaning "
"depends on --blur-kind: Gaussian sigma, disc radius, "
"or motion length")
p.add_argument("--blur-kind", default="gaussian",
choices=["gaussian", "disc", "motion"],
help="blur family. gaussian is a soft-focus stand-in; disc "
"is the circle-of-confusion PSF of real optical "
"defocus (non-Gaussian, jinc MTF with zero crossings); "
"motion is a linear smear. The last two are how a face "
"ends up large and useless, which a size gate cannot "
"catch")
p.add_argument("--motion-angle", type=float, default=0.0,
help="motion blur direction in degrees (--blur-kind motion)")
p.add_argument("--keep-duplicates", action="store_true")
p.add_argument("--models-dir", default=str(REPO / "models"))
p.add_argument("--arcface", default=None)
p.add_argument("--detector", default=None)
p.add_argument("--conf", type=float, default=0.5)
p.add_argument("--nms", type=float, default=0.4)
p.add_argument("--max-side", type=int, default=500)
# Required by a TRT-backend build, ignored by an ORT one. A TensorRT fp16
# run is a different realisation of the embedder — VR-005 measured ~0.85
# cosine agreement with the fp32 ONNX path on LVFace-B, with separation
# essentially intact — so a knee located here belongs to the fp16 space.
# The study stays internally consistent because gallery and probes are both
# embedded in this one session.
p.add_argument("--detector-engine", default="",
help="pre-built SCRFD .engine (TRT builds only)")
p.add_argument("--arcface-engine", default="",
help="pre-built ArcFace .engine (TRT builds only)")
p.add_argument("--prob-threshold", type=float, default=0.754)
p.add_argument("--match-prior", type=float, default=0.5)
p.add_argument("--tpi-retention", type=float, default=0.95)
p.add_argument("--down-interp", default="area", choices=sorted(INTERP))
p.add_argument("--up-interp", default="linear", choices=sorted(INTERP))
args = p.parse_args()
models_dir = Path(args.models_dir)
arcface = Path(args.arcface) if args.arcface else models_dir / "LVFace-B_Glint360K.onnx"
detector = Path(args.detector) if args.detector else models_dir / "scrfd_500m_bnkps.onnx"
for path, what in ((arcface, "embedder"), (detector, "detector")):
if not path.is_file():
return err(f"{what} model not found: {path}")
images_root = Path(args.images)
if not images_root.is_dir():
return err(f"image cache not found: {images_root}")
sizes = sorted({int(s) for s in args.sizes.split(",") if s.strip()})
sigmas = sorted({float(s) for s in args.sigmas.split(",") if s.strip()})
cv2.setRNGSeed(args.seed)
# ── actor pool ────────────────────────────────────────────────────────────
pool = discover_actors(images_root)
print(f"[select] {len(pool)} actor dirs under {images_root}", file=sys.stderr)
if args.gallery and Path(args.gallery).is_file():
ids, names = gallery_keys(Path(args.gallery))
pool = [a for a in pool
if (a["jellyfin_id"] and a["jellyfin_id"] in ids)
or normalise_name(a["name"]) in names]
print(f"[select] {len(pool)} are in {args.gallery}", file=sys.stderr)
eligible = [a for a in pool if len(a["images"]) >= args.min_images]
print(f"[select] {len(eligible)} have >= {args.min_images} mugshots",
file=sys.stderr)
if len(eligible) < 2:
return err(f"need at least 2 eligible actors; found {len(eligible)}")
rng = random.Random(args.seed)
selected = (sorted(rng.sample(eligible, min(args.actors, len(eligible))),
key=lambda a: a["dir"].name)
if args.actors else eligible)
# ── detect + align every mugshot once ─────────────────────────────────────
stages = Stages(str(detector), str(arcface), args.conf, args.nms,
args.detector_engine, args.arcface_engine)
print(f"[models] detector={detector.name} embedder={arcface.name} "
f"batch={stages.engine.max_batch}", file=sys.stderr)
t0 = time.time()
crops, rows, actors = [], [], []
n_nodetect = 0
for a in selected:
actor_crops, actor_paths = [], []
for img_path in a["images"]:
img = cv2.imread(str(img_path))
if img is None:
n_nodetect += 1
continue
if args.max_side > 0 and max(img.shape[:2]) > args.max_side:
s = args.max_side / max(img.shape[:2])
img = cv2.resize(img, None, fx=s, fy=s, interpolation=cv2.INTER_AREA)
faces = stages.detect(img)
if not faces:
enhanced = stages.enhance(img)
faces = stages.detect(enhanced)
if faces:
img = enhanced
if not faces:
n_nodetect += 1
continue
best = max(faces, key=lambda f: f.confidence)
crop = stages.align(img, best.landmarks)
if crop is None:
n_nodetect += 1
continue
actor_crops.append(crop)
actor_paths.append(img_path)
if len(actor_crops) < 2:
continue
ai = len(actors)
actors.append({"name": a["name"], "jellyfin_id": a["jellyfin_id"],
"dir": a["dir"].name, "n_images": len(actor_crops)})
for crop, img_path in zip(actor_crops, actor_paths):
rows.append({"actor_idx": ai, "image": str(img_path)})
crops.append(crop)
if len(actors) % 200 == 0:
print(f" [align] {len(actors)}/{len(selected)} actors, "
f"{len(crops)} crops", file=sys.stderr)
if len(actors) < 2:
return err(f"only {len(actors)} actors survived detection/alignment")
print(f"[align] {len(actors)} actors, {len(crops)} crops, {n_nodetect} skipped "
f"in {time.time() - t0:.1f}s", file=sys.stderr)
actor_of = np.array([r["actor_idx"] for r in rows], dtype=int)
t0 = time.time()
native = stages.embed(crops)
print(f"[embed] {len(crops)} native crops in {time.time() - t0:.1f}s",
file=sys.stderr)
# ── drop duplicate mugshots ───────────────────────────────────────────────
if not args.keep_duplicates:
keep = np.ones(len(rows), bool)
for ai in range(len(actors)):
kept: list[int] = []
for i in np.nonzero(actor_of == ai)[0]:
if any(float(native[i] @ native[k]) > DEDUP_SIM for k in kept):
keep[i] = False
else:
kept.append(int(i))
n_dup = int((~keep).sum())
counts = np.bincount(actor_of[keep], minlength=len(actors))
drop_actor = counts < 2
keep &= ~drop_actor[actor_of]
remap = np.full(len(actors), -1, dtype=int)
remap[~drop_actor] = np.arange(int((~drop_actor).sum()))
actors = [a for a, d in zip(actors, drop_actor) if not d]
rows = [r for r, k in zip(rows, keep) if k]
crops = [c for c, k in zip(crops, keep) if k]
native = native[keep]
actor_of = remap[actor_of[keep]]
print(f"[dedup] dropped {n_dup} duplicates and {int(drop_actor.sum())} "
f"actors; {len(actors)} actors, {len(rows)} images remain",
file=sys.stderr)
# ── hold out one probe per actor ──────────────────────────────────────────
is_probe = np.zeros(len(rows), bool)
for ai in range(len(actors)):
idx = np.nonzero(actor_of == ai)[0]
r = random.Random(f"{args.seed}:{actors[ai]['dir']}")
is_probe[r.choice(list(idx))] = True
probe_rows = np.nonzero(is_probe)[0]
gal_rows = np.nonzero(~is_probe)[0]
print(f"[holdout] {len(probe_rows)} probes, {len(gal_rows)} gallery embeddings",
file=sys.stderr)
gal_emb = native[gal_rows]
gal_actor = actor_of[gal_rows]
probe_actor = actor_of[probe_rows]
actor_cols = [np.nonzero(gal_actor == ai)[0] for ai in range(len(actors))]
if not all(len(c) for c in actor_cols):
return err("an actor has no gallery references left; raise --min-images")
cal = calibrate_gallery(gal_emb, gal_actor)
if not cal["valid"]:
return err("calibration could not be fitted; this study will not fall back "
"to a raw cosine threshold (CLAUDE.md invariant)")
log_prior_odds = float(np.log(args.match_prior / (1.0 - args.match_prior)))
# ── the grid ──────────────────────────────────────────────────────────────
down, up = INTERP[args.down_interp], INTERP[args.up_interp]
probe_crops = [crops[i] for i in probe_rows]
cells, records = [], []
n = len(probe_rows)
for size in sizes:
for sigma in sigmas:
t0 = time.time()
degraded = [degrade(c, size, sigma, args.blur_kind, down, up,
args.motion_angle) for c in probe_crops]
sharp = [score_sharpness(d) for d in degraded]
q = stages.embed(degraded)
sims = q @ gal_emb.T
best_per_actor = np.stack([sims[:, c].max(axis=1) for c in actor_cols],
axis=1)
best_actor = best_per_actor.argmax(axis=1)
best_sim = best_per_actor.max(axis=1)
p_match = np.asarray(probability(best_sim, cal["a"], cal["b"],
log_prior_odds))
accept = p_match > args.prob_threshold
correct = best_actor == probe_actor
tpi = int(np.sum(accept & correct))
fpi = int(np.sum(accept & ~correct))
unid = int(np.sum(~accept))
cell = {"size_px": size, "sigma": sigma, "blur_kind": args.blur_kind,
"n_probes": n,
"tpi": tpi, "fpi": fpi, "unidentified": unid,
"tpi_rate": tpi / n, "fpi_rate": fpi / n,
"unidentified_rate": unid / n,
"rank1_rate": float(np.mean(correct)),
"mean_p_match": float(np.mean(p_match))}
for m in MEASURES:
cell[f"mean_{m}"] = float(np.mean([s[m] for s in sharp]))
cells.append(cell)
for j in range(n):
rec = {"size_px": size, "sigma": sigma,
"blur_kind": args.blur_kind,
"probe_image": rows[probe_rows[j]]["image"],
"p_match": float(p_match[j]),
"outcome": ("TPI" if accept[j] and correct[j]
else "FPI" if accept[j] else "unidentified")}
rec.update({m: sharp[j][m] for m in MEASURES})
records.append(rec)
print(f"[grid] {size:3d}px {args.blur_kind[:4]} {sigma:<4g} TPI {100*tpi/n:5.1f}% "
f"FPI {100*fpi/n:5.1f}% unid {100*unid/n:5.1f}% "
f"rank1 {100*np.mean(correct):5.1f}% [{time.time()-t0:.1f}s]",
file=sys.stderr)
# ── rank the candidates, then locate the knee on the winner ───────────────
is_tpi = np.array([r["outcome"] == "TPI" for r in records])
ranking = sorted(
({"measure": m,
"auc": auc(np.array([r[m] for r in records], dtype=np.float64), is_tpi)}
for m in MEASURES),
key=lambda d: -d["auc"])
knees = [knee_from_measure(records, r["measure"], args.tpi_retention)
for r in ranking]
# ── outputs ───────────────────────────────────────────────────────────────
out = Path(args.out)
out.parent.mkdir(parents=True, exist_ok=True)
csv_path = out.with_name(out.name + ".csv")
with open(csv_path, "w", newline="") as f:
w = csv.DictWriter(f, fieldnames=list(cells[0].keys()))
w.writeheader()
w.writerows(cells)
rec_path = out.with_name(out.name + ".records.csv")
with open(rec_path, "w", newline="") as f:
w = csv.DictWriter(f, fieldnames=list(records[0].keys()))
w.writeheader()
w.writerows(records)
meta = {
"requirement": "VR-012",
"model": arcface.stem, "detector": detector.stem,
"n_actors": len(actors), "n_probes": len(probe_rows),
"n_gallery_embeddings": len(gal_rows),
"min_images": args.min_images, "seed": args.seed,
"sizes": sizes, "sigmas": sigmas, "blur_kind": args.blur_kind,
"motion_angle": args.motion_angle,
"prob_threshold": args.prob_threshold, "match_prior": args.match_prior,
"calibration": cal,
"measure_ranking": ranking,
"knees": knees,
"sharpness_window": list(sae_embed.sharpness_window()),
"caveat": (f"FPI grows with gallery size; this ran against {len(actors)} "
f"actors and still understates a production library."),
"grid": cells,
}
json_path = out.with_name(out.name + ".json")
json_path.write_text(json.dumps(meta, indent=2) + "\n")
png_path = out.with_name(out.name + ".png")
write_plots(cells, records, ranking, png_path, meta)
# ── stdout report ─────────────────────────────────────────────────────────
print(f"\nVR-012 — quality knee, {arcface.stem}")
print(f"{len(actors)} actors, {len(probe_rows)} probes x {len(cells)} cells\n")
print(f"{'size':>5} {'sigma':>6} {'TPI':>8} {'FPI':>8} {'unid':>8} {'rank1':>8}")
for c in cells:
print(f"{c['size_px']:>5} {c['sigma']:>6g} {100*c['tpi_rate']:>7.1f}% "
f"{100*c['fpi_rate']:>7.1f}% {100*c['unidentified_rate']:>7.1f}% "
f"{100*c['rank1_rate']:>7.1f}%")
print("\nAR-029 candidate ranking — AUC for predicting correct identification:")
for r in ranking:
print(f" {r['measure']:>20} {r['auc']:.4f}")
print(f"\n[out] {csv_path}\n[out] {rec_path}\n[out] {json_path}\n[out] {png_path}")
print(f"\n{meta['caveat']}")
return 0
if __name__ == "__main__":
sys.exit(main())
scripts/validation/sample_eval.py
+1 -9
View File
@@ -61,15 +61,7 @@ class Prediction:
keys = keys_for(imdb_id=a.get("imdb_id"), tmdb_id=a.get("tmdb_id"),
jellyfin_id=a.get("jellyfin_id"), name=a.get("name"),
crosswalk=crosswalk)
# schema_version 1: scenes is [[t0, t1], ...] (list of pairs)
# schema_version 2: scenes is [{"start":…, "end":…, "belief":…, …}, …]
windows = []
for s in a.get("scenes", []):
if isinstance(s, dict):
windows.append((float(s["start"]), float(s["end"])))
else:
t0, t1 = s[0], s[1]
windows.append((float(t0), float(t1)))
windows = [(float(t0), float(t1)) for t0, t1 in a.get("scenes", [])]
for _, t1 in windows:
self._max_t = max(self._max_t, t1)
self.actors.append({"keys": keys, "windows": windows})
scripts/validation/test_audio_offset.py
+1 -1
View File
@@ -51,7 +51,7 @@ sys.path.insert(0, str(BUILD))
import sae_audio # noqa: E402
FIXTURE = REPO / "tests" / "fixtures" / "audio" / "superhero_offset_200s.flac"
FIXTURE = REPO / "tests" / "fixtures" / "audio" / "bali_offset_200s.flac"
TONE = REPO / "tests" / "fixtures" / "audio" / "jray_audio_v1_tone.flac"
# Server SPEC.md section 3, "Matching and offset recovery". The cap is the
src/audio_signature.cpp
+2 -17
View File
@@ -155,24 +155,9 @@ struct DecodeCtx {
bool open_resampler(DecodeCtx& c, const AVFrame* f) {
#if LIBAVUTIL_VERSION_INT >= AV_VERSION_INT(57, 24, 100)
// Both MUST be zero-initialised. av_channel_layout_copy documents that it
// "will always uninitialize the destination before copy", and
// av_channel_layout_uninit() calls av_freep() on u.map — so a declaration
// without {} hands free() whatever pointer-shaped garbage the stack frame
// happened to hold. That is a real crash ("free(): invalid pointer"), not a
// theoretical one: it reproduced in roughly 1 run in 4 of UT-103, the only
// test that exercises this branch, because it is the only one whose input
// is stereo and so the only one that reaches the downmix path at all.
//
// It hid for two reasons worth remembering. It is stack-dependent, so it
// vanishes under a sanitizer build and looks like a flake in the aggregate
// test binary; and the golden-vector tests (UT-101) pass a mono 11025 Hz
// fixture, which is chosen precisely so the vector does not depend on the
// resampler — so bit-exactness against the golden vector proves nothing
// about this function.
AVChannelLayout out_layout{};
AVChannelLayout out_layout;
av_channel_layout_default(&out_layout, 1); // mono
AVChannelLayout in_layout{};
AVChannelLayout in_layout;
if (av_channel_layout_copy(&in_layout, &f->ch_layout) < 0) return false;
if (in_layout.nb_channels <= 0) {
av_channel_layout_uninit(&in_layout);
src/backends/gemm_backend.cpp
+14 -79
View File
@@ -42,20 +42,17 @@ constexpr int kDim = 512;
// Used for CI and as the correctness oracle for the GPU backends.
//
// TRACES: AR-026, AR-027 | SR-001
// Backed by CBLAS (OpenBLAS), which CMake now REQUIRES for this backend. The
// scalar loop below is portable but scales badly: scoring one face against a
// Backed by CBLAS (OpenBLAS) when available, falling back to a scalar loop when
// not. The fallback is portable but scales badly: scoring one face against a
// 5000-embedding gallery is 2.6 MFLOP, and a crowded frame multiplies that by
// the face count. Since AR-003 removed the per-frame face cap and CI has no GPU,
// the CPU path is the one that has to hold up under a library-scale gallery
// (AR-027) rather than merely be correct — so falling back to it silently would
// mean measuring AR-027 on a path no release runs.
// the CPU path is now the one that has to hold up under a library-scale gallery
// (AR-027) rather than merely be correct.
//
// The fallback is kept as the correctness oracle the two BLAS backends are
// diffed against when a similarity looks wrong, and is reachable only via
// -DSAE_ALLOW_SCALAR_GEMM=ON. The gallery is L2-normalised (as are the queries),
// so each similarity is a plain dot product. S is stored column-major to match
// the GPU backends: the gallery similarities for face fi start at
// result + fi*n_gallery().
// The fallback is kept rather than made mandatory so the build has no hard new
// dependency, and so the two can be diffed when a similarity looks wrong. The gallery is L2-normalised (as are the queries), so each
// similarity is a plain dot product. S is stored column-major to match the GPU
// backends: the gallery similarities for face fi start at result + fi*n_gallery.
class SimilarityEngine final : public ISimilarityEngine {
public:
SimilarityEngine(const float* gallery_row_major, int n_gallery, int max_faces)
@@ -75,19 +72,6 @@ public:
}
int max_faces() const override { return max_faces_; }
int n_gallery() const override { return n_gallery_; }
/// TRACES: AR-026 | SR-001
/// Promotions join the resident matrix, so the annex is scored by the same
/// SGEMM as the baked references. std::vector already grows geometrically,
/// so this is amortised O(1) per row.
void append_rows(const float* rows_row_major, int n_rows) override {
if (n_rows <= 0) return;
gallery_.insert(gallery_.end(), rows_row_major,
rows_row_major + static_cast<size_t>(n_rows) * kDim);
n_gallery_ += n_rows;
host_sims_.resize(static_cast<size_t>(max_faces_) * n_gallery_);
}
const float* compute(const float* query_row_major, int n_faces) override {
if (n_faces <= 0) return host_sims_.data();
@@ -159,7 +143,6 @@ inline void gpu_free(void* p) { cudaFree(p)
inline void gpu_memcpy_h2d(void* dst, const void* src, size_t n, stream_t s) { check_gpu(cudaMemcpyAsync(dst, src, n, cudaMemcpyHostToDevice, s), "H2D"); }
inline void gpu_memcpy_d2h(void* dst, const void* src, size_t n, stream_t s) { check_gpu(cudaMemcpyAsync(dst, src, n, cudaMemcpyDeviceToHost, s), "D2H"); }
inline void gpu_memcpy_h2d_sync(void* dst, const void* src, size_t n) { check_gpu(cudaMemcpy(dst, src, n, cudaMemcpyHostToDevice), "H2D_sync"); }
inline void gpu_memcpy_d2d_sync(void* dst, const void* src, size_t n) { check_gpu(cudaMemcpy(dst, src, n, cudaMemcpyDeviceToDevice), "D2D_sync"); }
inline void stream_create(stream_t* s) { check_gpu(cudaStreamCreate(s), "cudaStreamCreate"); }
inline void stream_destroy(stream_t s) { cudaStreamDestroy(s); }
inline void stream_sync(stream_t s) { check_gpu(cudaStreamSynchronize(s), "cudaStreamSync"); }
@@ -194,7 +177,6 @@ inline void gpu_free(void* p) { (void)hipFr
inline void gpu_memcpy_h2d(void* dst, const void* src, size_t n, stream_t s) { check_gpu(hipMemcpyAsync(dst, src, n, hipMemcpyHostToDevice, s), "H2D"); }
inline void gpu_memcpy_d2h(void* dst, const void* src, size_t n, stream_t s) { check_gpu(hipMemcpyAsync(dst, src, n, hipMemcpyDeviceToHost, s), "D2H"); }
inline void gpu_memcpy_h2d_sync(void* dst, const void* src, size_t n) { check_gpu(hipMemcpy(dst, src, n, hipMemcpyHostToDevice), "H2D_sync"); }
inline void gpu_memcpy_d2d_sync(void* dst, const void* src, size_t n) { check_gpu(hipMemcpy(dst, src, n, hipMemcpyDeviceToDevice), "D2D_sync"); }
inline void stream_create(stream_t* s) { check_gpu(hipStreamCreate(s), "hipStreamCreate"); }
inline void stream_destroy(stream_t s) { (void)hipStreamDestroy(s); }
inline void stream_sync(stream_t s) { check_gpu(hipStreamSynchronize(s), "hipStreamSync"); }
@@ -219,15 +201,14 @@ public:
SimilarityEngine(const float* gallery_row_major, int n_gallery, int max_faces)
: n_gallery_(n_gallery), max_faces_(max_faces)
{
const size_t gallery_floats = static_cast<size_t>(n_gallery_) * kDim;
gpu_malloc(reinterpret_cast<void**>(&d_gallery_), gallery_floats * sizeof(float));
gpu_memcpy_h2d_sync(d_gallery_, gallery_row_major, gallery_floats * sizeof(float));
gpu_malloc(reinterpret_cast<void**>(&d_query_),
static_cast<size_t>(max_faces_) * kDim * sizeof(float));
// Allocates d_gallery_/d_sims_ at the initial row count; append_rows()
// grows them geometrically from here.
reserve_rows(std::max(n_gallery_, 1));
const size_t gallery_floats = static_cast<size_t>(n_gallery_) * kDim;
if (gallery_floats)
gpu_memcpy_h2d_sync(d_gallery_, gallery_row_major, gallery_floats * sizeof(float));
gpu_malloc(reinterpret_cast<void**>(&d_sims_),
static_cast<size_t>(max_faces_) * n_gallery_ * sizeof(float));
stream_create(&stream_);
blas_create(&handle_);
@@ -251,24 +232,6 @@ public:
SimilarityEngine& operator=(const SimilarityEngine&) = delete;
int max_faces() const override { return max_faces_; }
int n_gallery() const override { return n_gallery_; }
/// TRACES: AR-026 | SR-001
/// Promotions join the GPU-resident matrix, so the annex is scored by the
/// same SGEMM as the baked references rather than by a host-side loop.
/// Capacity doubles on overflow, so the gallery is re-uploaded O(log n)
/// times over a film rather than once per promotion.
void append_rows(const float* rows_row_major, int n_rows) override {
if (n_rows <= 0) return;
const int want = n_gallery_ + n_rows;
if (want > capacity_) reserve_rows(std::max(want, capacity_ * 2));
gpu_memcpy_h2d_sync(d_gallery_ + static_cast<size_t>(n_gallery_) * kDim,
rows_row_major,
static_cast<size_t>(n_rows) * kDim * sizeof(float));
n_gallery_ = want;
host_sims_.resize(static_cast<size_t>(max_faces_) * n_gallery_);
}
const float* compute(const float* query_row_major, int n_faces) override {
if (n_faces <= 0) return host_sims_.data();
@@ -288,35 +251,7 @@ public:
}
private:
// Grow the resident gallery (and the similarity output sized against it) to
// `rows` capacity, preserving the n_gallery_ rows already there. The copy is
// device-to-device, so a promotion never re-uploads the baked gallery across
// the bus.
void reserve_rows(int rows) {
if (rows <= capacity_) return;
float* d_new_gallery = nullptr;
gpu_malloc(reinterpret_cast<void**>(&d_new_gallery),
static_cast<size_t>(rows) * kDim * sizeof(float));
if (d_gallery_ && n_gallery_ > 0)
gpu_memcpy_d2d_sync(d_new_gallery, d_gallery_,
static_cast<size_t>(n_gallery_) * kDim * sizeof(float));
if (d_gallery_) gpu_free(d_gallery_);
d_gallery_ = d_new_gallery;
// S is (capacity × n_faces); its contents are rewritten by every
// compute(), so this one is a plain reallocation with nothing to keep.
float* d_new_sims = nullptr;
gpu_malloc(reinterpret_cast<void**>(&d_new_sims),
static_cast<size_t>(max_faces_) * rows * sizeof(float));
if (d_sims_) gpu_free(d_sims_);
d_sims_ = d_new_sims;
capacity_ = rows;
}
int n_gallery_{0};
int capacity_{0};
int max_faces_{0};
float* d_gallery_{nullptr};
float* d_query_{nullptr};
src/backends/ort_provider.hpp
+13 -15
View File
@@ -23,12 +23,23 @@
enum class OrtProvider { CPU, CUDA, ROCm, TensorRT };
inline OrtProvider detect_ort_provider() {
// ORT returns these in its own preference order (TensorRT, CUDA, ..., CPU
// last), so the first recognised entry is the best available and the loop
// returns on it.
auto available = Ort::GetAvailableProviders();
for (const auto& p : available) {
// Only the TensorRT *EP* is a build-time opt-in — it needs the headers
// and the profile plumbing below. CUDA is not: it is a plain ORT
// provider, and gating its detection on the TRT flag (as this did) made
// the CUDA branch unreachable in every build that did not also ask for
// TensorRT. The symptom is silent rather than loud — inference simply
// runs on the CPU and everything still returns correct answers — which
// is why it survived: a 300-actor VR-012 grid cell took 76 s on the CPU
// with the GPU idle at 212 MiB.
#ifdef SAE_ORT_WITH_TRT_EP
if (p == "TensorrtExecutionProvider") return OrtProvider::TensorRT;
if (p == "CUDAExecutionProvider") return OrtProvider::CUDA;
#endif
if (p == "CUDAExecutionProvider") return OrtProvider::CUDA;
if (p == "ROCMExecutionProvider") return OrtProvider::ROCm;
}
return OrtProvider::CPU;
@@ -124,21 +135,8 @@ inline OrtProvider apply_ort_provider(Ort::SessionOptions& opts,
try {
OrtROCMProviderOptions rocm{};
rocm.device_id = 0;
// Without these MIOpen runs convolutions on the no-workspace GEMM
// fallback (the "GemmFwdRest, provided ptr: 0 size: 0" warnings), which
// is the slow path — most visible on the conv-heavy TransNetV2 scene
// detector. Exhaustive search lets MIOpen pick the fast conv kernel,
// and TunableOp autotunes the GEMMs; both cache to the MIOpen user DB
// (MIOPEN_USER_DB_PATH), so the tuning cost is paid once per shape.
// Opt-out via SAE_ROCM_NOTUNE=1 for a quick no-warmup run.
const bool tune = std::getenv("SAE_ROCM_NOTUNE") == nullptr;
rocm.miopen_conv_exhaustive_search = tune ? 1 : 0;
rocm.tunable_op_enable = tune;
rocm.tunable_op_tuning_enable = tune;
opts.AppendExecutionProvider_ROCM(rocm);
std::cerr << "[" << label << "] ROCm provider"
<< (tune ? " (MIOpen exhaustive + TunableOp)" : " (untuned)")
<< "\n";
std::cerr << "[" << label << "] ROCm provider\n";
return OrtProvider::ROCm;
} catch (const Ort::Exception& e) {
std::cerr << "[" << label << "] ROCm unavailable ("
src/backends/trt_backend.cpp
-38
View File
@@ -19,7 +19,6 @@
#include <NvInfer.h>
#include <cuda_runtime_api.h>
#include <cstdlib>
#include <opencv2/dnn.hpp>
#include <opencv2/imgproc.hpp>
@@ -46,40 +45,6 @@ inline void check_cuda(cudaError_t e, const char* what) {
throw CudaError(std::string(what) + ": " + cudaGetErrorString(e));
}
/// TRACES: VR-015 | PR-004
/// Select how a thread waits for the GPU. Must run before the CUDA context is
/// created, so every engine constructor calls it and the first one wins.
///
/// The default (`cudaDeviceScheduleAuto`) spin-waits: `cudaStreamSynchronize`
/// burns the calling thread's CPU for the whole of the device's work. Measured
/// here, the embedder thread sat at 99.7% *user* time with 0.5 s of system time
/// across 183 s — i.e. no blocking syscalls at all — while the GPU ran flat out.
///
/// On this laptop that is not merely wasted CPU. `nvidia-powerd` arbitrates one
/// power budget across CPU and GPU, and the GPU's ceiling was observed dropping
/// from 20 W idle to 15 W under our load, with the SM clock *falling* from
/// 1005 MHz to 210 MHz once work started. Spinning may therefore be buying
/// watts away from the device the pipeline is actually waiting on.
///
/// SAE_CUDA_BLOCKING_SYNC=1 switches to a blocking wait so the A/B needs no
/// rebuild. Default is unchanged until the measurement says otherwise.
inline void configure_cuda_sync_once() {
static const bool done = [] {
const char* env = std::getenv("SAE_CUDA_BLOCKING_SYNC");
if (env && env[0] == '1') {
cudaError_t e = cudaSetDeviceFlags(cudaDeviceScheduleBlockingSync);
std::cerr << "[cuda] sync policy: BlockingSync"
<< (e == cudaSuccess ? "" : " (FAILED — context already created)")
<< "\n";
} else {
std::cerr << "[cuda] sync policy: default (spin) — "
"set SAE_CUDA_BLOCKING_SYNC=1 to compare\n";
}
return true;
}();
(void)done;
}
class TrtLogger : public nvinfer1::ILogger {
public:
void log(Severity sev, const char* msg) noexcept override {
@@ -144,7 +109,6 @@ public:
(output_is_fp16_ ? 2 : 4);
check_cuda(cudaMalloc(&d_input_, in_bytes), "cudaMalloc input");
check_cuda(cudaMalloc(&d_output_, out_bytes), "cudaMalloc output");
configure_cuda_sync_once();
check_cuda(cudaStreamCreate(&stream_), "cudaStreamCreate");
context_->setTensorAddress(input_name_.c_str(), d_input_);
@@ -329,7 +293,6 @@ public:
out_elem_counts_[oi] = count;
}
configure_cuda_sync_once();
check_cuda(cudaStreamCreate(&stream_), "cudaStreamCreate");
std::cerr << "[TrtScrfd] loaded: " << engine_path
@@ -499,7 +462,6 @@ public:
const std::size_t out_count = static_cast<std::size_t>(kWindow);
check_cuda(cudaMalloc(&d_input_, in_count * 4), "cudaMalloc input");
check_cuda(cudaMalloc(&d_output_, out_count * 4), "cudaMalloc output");
configure_cuda_sync_once();
check_cuda(cudaStreamCreate(&stream_), "cudaStreamCreate");
context_->setTensorAddress(input_name_.c_str(), d_input_);
context_->setTensorAddress(output_name_.c_str(), d_output_);
src/benchmark.hpp
-590
View File
@@ -1,590 +0,0 @@
#pragma once
/// TRACES: VR-015 | PR-004
///
/// Pipeline throughput benchmark — how much of a run is spent in each node.
///
/// The KPN network already counts most of what an optimiser needs, and
/// `print_diagnostics()` throws nearly all of it away: it prints frames and
/// `ema` per node, and passes `elapsed_s = 0`, which zeroes throughput. Two
/// things had to change before "time per node" could be answered honestly.
///
/// **`ema` is not a total.** It is an exponentially weighted average, so
/// `frames * ema` tracks the end of the run rather than the whole of it. On a
/// film that is a real difference — a detector costs one thing in a crowd scene
/// and another over a landscape. `NodeStats::total_exec_us` (added alongside
/// this) is the true sum.
///
/// **Wall time inside a node is not all work.** `PoolObjectNode::fire_once`
/// times the functor *and* `push_outputs`, and `push_outputs` parks on a full
/// downstream channel (AR-004). A node that is merely backpressured therefore
/// bills the time it spent waiting to whoever is ahead of it: SuperHero's
/// `frame_source` reported 141.9 ms/frame against a decoder logging 12-18 ms.
/// Optimising against that number means optimising the fastest node in the
/// graph.
///
/// So each node is reported three ways, and the three together are what
/// identify a cost:
///
/// - `exec_ms` — cumulative wall time in the node, work *and* parked pushes
/// - `cpu_ms` — thread CPU time (CLOCK_THREAD_CPUTIME_ID). Backpressure
/// cannot inflate it, because a parked node holds no thread.
/// - `pressure` — mean fill of its input channels minus that of its outputs
///
/// **`cpu_ms` cannot tell real work from a spinning GPU wait.** CUDA's default
/// sync policy (`cudaDeviceScheduleAuto`) spin-waits before yielding, so
/// `cudaStreamSynchronize` burns the calling thread's CPU while the GPU works.
/// A node that is purely GPU-bound can therefore report a high `cpu_ms` and read
/// as CPU-bound. `cudaSetDeviceFlags(cudaDeviceScheduleBlockingSync)` settles it
/// in one line: if a node's `cpu_ms` collapses under blocking sync, that CPU was
/// spin, not work.
///
/// **`cpu_ms` counts one thread only.** `CLOCK_THREAD_CPUTIME_ID` is per-thread,
/// and OpenCV here is built against TBB, so any node whose functor goes through
/// `cv::parallel_for_` (histogram compare, `warpAffine`, colour conversion) has
/// that work executed on TBB's arena — 19 workers on a 20-core box — and billed
/// to those threads rather than to the node. Such a node reads *cheaper* than it
/// is, and the difference shows up in `stall/f` instead, indistinguishable from a
/// GPU wait. `exec_ms` does capture it, since the functor does not return until
/// the parallel region joins: a node where `exec/f` greatly exceeds `cpu/f`
/// while its output channel is empty is fanning out, not waiting.
///
/// Work piles up *in front of* a bottleneck and starves everything *after* it,
/// so `pressure` is maximal at the node setting the pace. `cpu_ms` then says
/// which repair applies: high pressure with a saturated thread is CPU-bound and
/// the work must get cheaper, while high pressure with an idle thread is
/// waiting on a device, where batch size and engine precision are the knobs.
///
/// Occupancy has to be sampled during the run. `current_fill` is instantaneous
/// and every channel has drained by the time the network stops, so a single
/// read at the end reports an idle pipeline however congested it was.
///
/// Nothing here is specific to this pipeline's topology: the node graph is
/// recovered from KPN's channel names, so it keeps working when the graph
/// changes.
#include <kpn/diagnostics.hpp>
#include <nlohmann/json.hpp>
#include <algorithm>
#include <atomic>
#include <chrono>
#include <cstdint>
#include <functional>
#include <iomanip>
#include <map>
#include <ostream>
#include <sstream>
#include <string>
#include <thread>
#include <vector>
namespace sae::bench {
// ── Edge naming ──────────────────────────────────────────────────────────────
/// TRACES: VR-015 | PR-004
/// KPN names a channel "<src>:<idx> → <dst>:<idx>" (static_network.hpp).
/// Recovering the two node names from it is what keeps attribution
/// topology-agnostic: the graph is read back out of the channel names rather
/// than hard-coded here, so a new node or a re-wired branch needs no change.
/// Leaves both outputs untouched if the name does not carry an arrow.
inline void split_edge_name(const std::string& name,
std::string& producer, std::string& consumer) {
static const std::string kArrow = " \xe2\x86\x92 "; // " → "
const auto arrow = name.find(kArrow);
if (arrow == std::string::npos) return;
auto strip_port = [](std::string s) {
const auto colon = s.rfind(':');
return colon == std::string::npos ? s : s.substr(0, colon);
};
producer = strip_port(name.substr(0, arrow));
consumer = strip_port(name.substr(arrow + kArrow.size()));
}
// ── Channel occupancy, time-averaged ─────────────────────────────────────────
/// TRACES: VR-015 | PR-004
/// One channel's fill level integrated over the run. `peak_fill` is already
/// cumulative in `ChannelStats`, but a peak cannot distinguish "full once" from
/// "full throughout", and those are opposite diagnoses. A mean can.
struct ChannelOccupancy {
std::string name; // "src:0 → dst:0", as KPN names it
std::string producer; // node name left of the arrow
std::string consumer; // node name right of the arrow
std::size_t capacity{0};
std::uint64_t samples{0};
double fill_sum{0.0};
std::uint64_t samples_full{0};
std::uint64_t samples_empty{0};
// Final-snapshot totals (monotonic counters, so the last read is the total).
std::size_t peak_fill{0};
std::uint64_t pushes{0};
std::uint64_t pops{0};
std::uint64_t drops{0};
std::uint64_t overflows{0};
std::uint64_t bytes_pushed{0};
double mean_fill() const { return samples ? fill_sum / static_cast<double>(samples) : 0.0; }
double mean_fill_pct() const { return capacity ? 100.0 * mean_fill() / static_cast<double>(capacity) : 0.0; }
double peak_pct() const { return capacity ? 100.0 * static_cast<double>(peak_fill) / static_cast<double>(capacity) : 0.0; }
double full_pct() const { return samples ? 100.0 * static_cast<double>(samples_full) / static_cast<double>(samples) : 0.0; }
double empty_pct() const { return samples ? 100.0 * static_cast<double>(samples_empty) / static_cast<double>(samples) : 0.0; }
double bandwidth_mbs(double wall_s) const {
return wall_s > 0.0 ? static_cast<double>(bytes_pushed) / wall_s / 1e6 : 0.0;
}
};
// ── Per-node attributed cost ─────────────────────────────────────────────────
/// TRACES: VR-015 | PR-004
struct NodeCost {
std::string name;
std::uint64_t frames{0};
// Cumulative wall time in the node — the answer to "where did the run go",
// but only for a node that is not backpressured; it includes parked pushes.
double exec_ms{0.0};
double exec_ms_per_frame{0.0}; // true mean, not the EMA
double exec_share{0.0}; // exec_ms / wall_ms, 0..1
double ema_exec_ms{0.0}; // KPN's EMA, kept for continuity with the old report
double max_exec_ms{0.0};
// Thread CPU time: excludes sleeping, parking and waiting on a device, so it
// is the one number backpressure cannot inflate.
double cpu_ms{0.0};
double cpu_ms_per_frame{0.0};
double cpu_share{0.0}; // cpu_ms / wall_ms — thread saturation, 0..1
double cpu_pct_of_pipeline{0.0}; // this node's share of all nodes' CPU time
// Per frame, time inside the node not spent on its own CPU: parked on a
// full output channel, or waiting on the GPU. `pressure` separates those —
// a backpressured node has a full output, a device-bound one does not.
double stall_ms_per_frame{0.0};
// Queue occupancy either side of the node, in percent of capacity.
double in_fill_pct{0.0};
double out_fill_pct{0.0};
double pressure{0.0}; // in − out; maximal at the pacing node
bool has_input{false};
bool has_output{false};
double queue_wait_ms{0.0};
bool is_bottleneck{false};
/// TRACES: VR-015 | AR-004 | PR-004
/// Live scheduling state, so a wedged run says *why* it is wedged rather
/// than only that it is. With `queued=0, wake=1` a wake was recorded and
/// never consumed; with `queued=0, wake=0` and a full input, no wake was
/// ever generated. Those are different bugs in different files.
bool queued{false};
bool wake_pending{false};
};
/// TRACES: VR-015 | PR-004
/// Attribute cost to nodes from a KPN node snapshot plus sampled channel
/// occupancy. Pure — no clocks, no threads, no network — so the ranking is
/// unit-testable on CI hardware that can never run the pipeline itself.
///
/// A node with several inputs takes the **minimum** input fill: it can only run
/// once every input has data, so the emptiest one gates it, and a full sibling
/// channel means that channel's producer is blocked rather than this node being
/// slow. A node with several outputs takes the **maximum** output fill, since
/// parking on any one branch stops the node.
///
/// Terminals are the infinite-reservoir limit of the same rule: a source has
/// unlimited work available (input treated as 100% full) and a sink unlimited
/// drain (output treated as empty), so both stay rankable against the interior
/// nodes instead of dropping out of the comparison.
inline std::vector<NodeCost> attribute_cost(
const std::vector<kpn::NodeSnapshot>& nodes,
const std::vector<ChannelOccupancy>& channels,
double wall_sec)
{
const double wall_ms = wall_sec * 1000.0;
double cpu_total = 0.0;
for (const auto& n : nodes) cpu_total += n.total_cpu_ms;
std::vector<NodeCost> out;
out.reserve(nodes.size());
for (const auto& n : nodes) {
NodeCost c;
c.name = n.name;
c.frames = n.frames_processed;
c.exec_ms = n.total_exec_ms;
c.ema_exec_ms = n.ema_exec_ms;
c.max_exec_ms = n.max_exec_ms;
c.cpu_ms = n.total_cpu_ms;
c.queue_wait_ms = n.queue_wait_ms;
c.queued = n.queued;
c.wake_pending = n.wake_pending;
c.exec_ms_per_frame = c.frames ? c.exec_ms / static_cast<double>(c.frames) : 0.0;
c.cpu_ms_per_frame = c.frames ? c.cpu_ms / static_cast<double>(c.frames) : 0.0;
c.exec_share = wall_ms > 0.0 ? c.exec_ms / wall_ms : 0.0;
c.cpu_share = wall_ms > 0.0 ? c.cpu_ms / wall_ms : 0.0;
c.cpu_pct_of_pipeline = cpu_total > 0.0 ? 100.0 * c.cpu_ms / cpu_total : 0.0;
c.stall_ms_per_frame = c.exec_ms_per_frame - c.cpu_ms_per_frame;
if (c.stall_ms_per_frame < 0.0) c.stall_ms_per_frame = 0.0;
double in_min = 0.0; bool have_in = false;
double out_max = 0.0; bool have_out = false;
for (const auto& ch : channels) {
if (ch.consumer == n.name) {
const double f = ch.mean_fill_pct();
if (!have_in || f < in_min) in_min = f;
have_in = true;
}
if (ch.producer == n.name) {
const double f = ch.mean_fill_pct();
if (!have_out || f > out_max) out_max = f;
have_out = true;
}
}
c.has_input = have_in;
c.has_output = have_out;
c.in_fill_pct = have_in ? in_min : 100.0; // source: always has work
c.out_fill_pct = have_out ? out_max : 0.0; // sink: never blocks
c.pressure = c.in_fill_pct - c.out_fill_pct;
out.push_back(std::move(c));
}
// Rank, but only among nodes that actually ran: a node with zero frames has
// no cost to attribute and its neighbouring channels never moved.
auto best = out.end();
for (auto it = out.begin(); it != out.end(); ++it) {
if (it->frames == 0) continue;
if (best == out.end() || it->pressure > best->pressure) best = it;
}
if (best != out.end()) best->is_bottleneck = true;
return out;
}
/// TRACES: VR-015 | PR-004
/// One line of plain English about the winning node, since the point of the
/// report is to say what to change next. A saturated thread means the node's
/// own work is the limit; an idle thread under pressure means it is waiting on
/// a device, and those are different repairs.
inline std::string verdict(const std::vector<NodeCost>& costs) {
for (const auto& c : costs) {
if (!c.is_bottleneck) continue;
std::ostringstream os;
os << std::fixed << c.name << " sets the pace: ";
// A source's 100% input is the infinite-reservoir convention, not a
// measured queue — saying "work is backed up in front of it" would be
// asserting something no counter observed.
if (!c.has_input)
os << "nothing downstream is waiting on it (output "
<< std::setprecision(1) << c.out_fill_pct
<< "% full), so the pipeline is running as fast as this node can feed it. ";
else if (!c.has_output)
os << std::setprecision(1) << c.in_fill_pct
<< "% full input and nothing to block on, so it is the drain. ";
else
os << std::setprecision(1) << c.in_fill_pct << "% full input, "
<< c.out_fill_pct << "% full output. ";
os << std::setprecision(2) << c.cpu_ms_per_frame << " ms/frame on CPU. ";
if (c.cpu_share >= 0.85)
os << "CPU-bound — its thread is busy " << std::setprecision(0)
<< (100.0 * c.cpu_share) << "% of the run, so the work itself has to get"
" cheaper or be split across more threads.";
else if (c.cpu_share <= 0.35 && c.stall_ms_per_frame > c.cpu_ms_per_frame)
os << "Device-bound — its thread is busy only " << std::setprecision(0)
<< (100.0 * c.cpu_share) << "% of the run and it spends "
<< std::setprecision(2) << c.stall_ms_per_frame
<< " ms/frame off-CPU, so it is waiting on the GPU or the disk: batch size,"
" engine precision and the decode path are the knobs, not the C++.";
else
os << "Mixed — thread busy " << std::setprecision(0) << (100.0 * c.cpu_share)
<< "% of the run, " << std::setprecision(2) << c.stall_ms_per_frame
<< " ms/frame off-CPU.";
return os.str();
}
return "no node processed a frame — nothing to attribute";
}
// ── Recorder ─────────────────────────────────────────────────────────────────
/// TRACES: VR-015 | PR-004
/// Samples the live network on a timer and emits the report at the end.
///
/// The sampler only reads relaxed atomics, so it does not perturb what it
/// measures — which matters, since this exists to be trusted as a timing
/// measurement.
class BenchmarkRecorder {
public:
using Sampler = std::function<kpn::NetworkSnapshot()>;
explicit BenchmarkRecorder(int sample_interval_ms = 100)
: interval_(std::chrono::milliseconds(sample_interval_ms)) {}
~BenchmarkRecorder() { stop(); }
void start(Sampler sampler) {
sampler_ = std::move(sampler);
running_.store(true, std::memory_order_release);
thread_ = std::thread([this] {
while (running_.load(std::memory_order_acquire)) {
accumulate(sampler_());
std::this_thread::sleep_for(interval_);
}
});
}
/// Stops sampling and latches the final counter values. Call while the
/// network object is still alive: the monotonic counters stay valid after
/// `net.stop()`, but they die with the object.
void stop() {
if (!running_.exchange(false, std::memory_order_acq_rel)) return;
if (thread_.joinable()) thread_.join();
if (sampler_) {
final_ = sampler_();
// Occupancy is deliberately NOT accumulated from this last read:
// the pipeline has drained by now, and folding an idle sample into
// the mean biases every channel toward "never congested".
for (const auto& ch : final_.channels) {
auto& occ = occupancy_[ch.name];
if (occ.name.empty()) { // a channel that never moved
occ.name = ch.name;
occ.capacity = ch.capacity;
split_edge_name(ch.name, occ.producer, occ.consumer);
}
occ.peak_fill = ch.peak_fill;
occ.pushes = ch.pushes;
occ.pops = ch.pops;
occ.drops = ch.drops;
occ.overflows = ch.overflows;
occ.bytes_pushed = ch.bytes_pushed;
}
}
stopped_ = true;
}
bool has_data() const { return stopped_ && !final_.nodes.empty(); }
double wall_sec() const { return final_.elapsed_s; }
std::vector<ChannelOccupancy> channels() const {
std::vector<ChannelOccupancy> v;
v.reserve(occupancy_.size());
for (const auto& [_, occ] : occupancy_) v.push_back(occ);
return v;
}
std::vector<NodeCost> costs() const {
return attribute_cost(final_.nodes, channels(), final_.elapsed_s);
}
/// TRACES: VR-015 | PR-004
/// Machine-readable report, for sweeping configurations and diffing runs.
/// `film_sec` is the last timestamp the pipeline reached, so
/// `realtime_factor` answers what the optimiser is really asking: seconds
/// of film per second of wall clock. It is 0 for a topology with no result
/// sink (the dump-only path), and the field is then omitted rather than
/// reported as zero throughput.
nlohmann::json to_json(const nlohmann::json& run_config, double film_sec) const {
using nlohmann::json;
const double wall = final_.elapsed_s;
const auto chans = channels();
const auto cost = attribute_cost(final_.nodes, chans, wall);
json j;
j["schema_version"] = 1;
j["config"] = run_config;
json summary;
summary["wall_sec"] = wall;
summary["sample_count"] = sample_count_;
summary["sample_interval_ms"] = interval_.count();
if (film_sec > 0.0) {
summary["film_sec"] = film_sec;
summary["realtime_factor"] = wall > 0.0 ? film_sec / wall : 0.0;
}
for (const auto& c : cost)
if (c.is_bottleneck) { summary["bottleneck"] = c.name; break; }
summary["verdict"] = verdict(cost);
j["summary"] = summary;
json jnodes = json::array();
for (const auto& c : cost) {
jnodes.push_back({
{"name", c.name},
{"frames", c.frames},
{"fps", wall > 0.0 ? c.frames / wall : 0.0},
{"exec_ms", c.exec_ms},
{"exec_ms_per_frame", c.exec_ms_per_frame},
{"exec_share", c.exec_share},
{"ema_exec_ms", c.ema_exec_ms},
{"max_exec_ms", c.max_exec_ms},
{"cpu_ms", c.cpu_ms},
{"cpu_ms_per_frame", c.cpu_ms_per_frame},
{"cpu_share", c.cpu_share},
{"cpu_pct_of_pipeline", c.cpu_pct_of_pipeline},
{"stall_ms_per_frame", c.stall_ms_per_frame},
{"queue_wait_ms", c.queue_wait_ms},
{"in_fill_pct", c.in_fill_pct},
{"out_fill_pct", c.out_fill_pct},
{"pressure", c.pressure},
{"is_bottleneck", c.is_bottleneck},
{"queued", c.queued},
{"wake_pending", c.wake_pending},
});
}
j["nodes"] = std::move(jnodes);
json jch = json::array();
for (const auto& ch : chans) {
jch.push_back({
{"name", ch.name},
{"producer", ch.producer},
{"consumer", ch.consumer},
{"capacity", ch.capacity},
{"mean_fill", ch.mean_fill()},
{"mean_fill_pct", ch.mean_fill_pct()},
{"peak_fill", ch.peak_fill},
{"peak_pct", ch.peak_pct()},
{"full_pct", ch.full_pct()},
{"empty_pct", ch.empty_pct()},
{"pushes", ch.pushes},
{"pops", ch.pops},
{"drops", ch.drops},
{"overflows", ch.overflows},
{"mb_per_sec", ch.bandwidth_mbs(wall)},
});
}
j["channels"] = std::move(jch);
return j;
}
/// TRACES: VR-015 | PR-004
/// Human-readable form of the same data, so a run is legible without
/// opening the JSON.
void print(std::ostream& os, double film_sec) const {
print_impl(os, final_, film_sec);
}
/// TRACES: VR-015 | AR-004 | PR-004
/// Dump the report from a LIVE snapshot, mid-run, without stopping anything.
///
/// A report that only exists at shutdown is no use against the failure this
/// pipeline actually has: a wedged run never reaches shutdown, so the one
/// moment the numbers matter most is the one moment they were unavailable.
/// Channel occupancy names the stalled node directly — it is the one whose
/// input is full and whose output is empty — which is otherwise a debug-build
/// and a gdb session away.
///
/// Safe to call from the wait loop while the pipeline is running or hung: it
/// takes the same lock-free snapshot the sampler does.
void dump_live(std::ostream& os, double film_sec) const {
if (!sampler_) { os << "[benchmark] no sampler — run with --benchmark\n"; return; }
print_impl(os, sampler_(), film_sec);
}
private:
void print_impl(std::ostream& os, const kpn::NetworkSnapshot& snap,
double film_sec) const {
const double wall = snap.elapsed_s;
const auto chans = channels();
const auto cost = attribute_cost(snap.nodes, chans, wall);
os << "\n┌─ Pipeline benchmark (VR-015) ──────────────────────────────────────────────\n";
os << "│ wall " << std::fixed << std::setprecision(1) << wall << "s";
if (film_sec > 0.0)
os << " film " << film_sec << "s realtime x" << std::setprecision(2)
<< (wall > 0.0 ? film_sec / wall : 0.0);
os << " samples " << sample_count_ << "\n│\n";
os << "│ node frames cpu_s cpu%run cpu%tot cpu/f"
" exec/f stall/f in% out% press q/w\n";
for (const auto& c : cost) {
os << "│ " << (c.is_bottleneck ? "▶ " : " ") << std::left << std::setw(16)
<< c.name << std::right
<< std::setw(7) << c.frames
<< std::setw(10) << std::setprecision(1) << (c.cpu_ms / 1000.0)
<< std::setw(9) << std::setprecision(0) << (100.0 * c.cpu_share)
<< std::setw(9) << std::setprecision(0) << c.cpu_pct_of_pipeline
<< std::setw(8) << std::setprecision(2) << c.cpu_ms_per_frame
<< std::setw(8) << std::setprecision(2) << c.exec_ms_per_frame
<< std::setw(9) << std::setprecision(2) << c.stall_ms_per_frame
<< std::setw(7) << std::setprecision(0) << c.in_fill_pct
<< std::setw(7) << std::setprecision(0) << c.out_fill_pct
<< std::setw(8) << std::setprecision(1) << c.pressure
<< " " << int(c.queued) << "/" << int(c.wake_pending)
<< "\n";
}
/// TRACES: VR-015 | AR-004 | PR-004
// Fires only when the scheduling state is actually wrong, so a healthy
// run stays quiet and a wedged one names the fault — instead of leaving
// it to be reconstructed under a debugger that suppresses the bug.
for (const auto& c : cost) {
if (c.queued || !c.has_input) continue;
if (c.wake_pending)
os << "│ !! " << c.name << " idle with a wake outstanding"
" (queued=0 wake=1): the wake was recorded and never"
" consumed — submit/release handshake.\n";
else if (c.in_fill_pct > 50.0)
os << "│ !! " << c.name << " idle with a "
<< std::setprecision(0) << c.in_fill_pct
<< "% full input and no wake pending: the wake was never"
" generated — channel edge detection.\n";
}
os << "│\n│ channel cap mean% peak% full%"
" empty% MB/s\n";
for (const auto& ch : chans) {
os << "│ " << std::left << std::setw(36) << ch.name << std::right
<< std::setw(5) << ch.capacity
<< std::setw(7) << std::setprecision(1) << ch.mean_fill_pct()
<< std::setw(7) << ch.peak_pct()
<< std::setw(7) << ch.full_pct()
<< std::setw(7) << ch.empty_pct()
<< std::setw(9) << std::setprecision(1) << ch.bandwidth_mbs(wall)
<< "\n";
}
os << "│\n│ " << verdict(cost) << "\n";
os << "└────────────────────────────────────────────────────────────────────────────\n";
os << " cpu_s / cpu%tot is where the run's compute actually went. exec/f is wall\n"
" time in the node INCLUDING time parked on a full output channel, so it\n"
" overstates a backpressured node — compare it against cpu/f, which cannot\n"
" be inflated that way. press = input fill − output fill, and locates the\n"
" node that work is queueing up in front of.\n"
" cpu_s counts THIS node's thread only: work OpenCV fans out via TBB is\n"
" billed to the TBB arena, so a node using cv::parallel_for_ reads cheaper\n"
" than it is and the difference surfaces in stall/f.\n";
}
private:
void accumulate(const kpn::NetworkSnapshot& snap) {
++sample_count_;
for (const auto& ch : snap.channels) {
auto& occ = occupancy_[ch.name];
if (occ.name.empty()) {
occ.name = ch.name;
occ.capacity = ch.capacity;
split_edge_name(ch.name, occ.producer, occ.consumer);
}
occ.fill_sum += static_cast<double>(ch.current_fill);
++occ.samples;
if (ch.capacity && ch.current_fill >= ch.capacity) ++occ.samples_full;
if (ch.current_fill == 0) ++occ.samples_empty;
}
}
std::chrono::milliseconds interval_;
Sampler sampler_;
std::thread thread_;
std::atomic<bool> running_{false};
bool stopped_{false};
std::uint64_t sample_count_{0};
std::map<std::string, ChannelOccupancy> occupancy_;
kpn::NetworkSnapshot final_{};
};
} // namespace sae::bench
src/build_gallery.cpp
+1 -5
View File
@@ -34,7 +34,6 @@ int main(int argc, char** argv) {
std::string arcface_model = kDefaultArcfaceModel;
float conf = 0.5f, nms_thr = 0.4f;
int max_side = 500;
float min_face_px = 0.f;
for (int i = 1; i < argc; ++i) {
auto arg = [&](const char* f) { return std::strcmp(argv[i], f) == 0; };
@@ -51,7 +50,6 @@ int main(int argc, char** argv) {
else if (arg("--conf")) conf = std::stof(next());
else if (arg("--nms")) nms_thr = std::stof(next());
else if (arg("--max-side")) max_side = std::stoi(next());
else if (arg("--min-face-px")) min_face_px = std::stof(next());
else { std::cerr << "[warn] unknown flag: " << argv[i] << "\n"; }
} catch (const std::exception& e) {
std::cerr << "Error: " << e.what() << "\n";
@@ -61,8 +59,7 @@ int main(int argc, char** argv) {
if (root_path.empty() || output_path.empty()) {
std::cerr << "Usage: build_gallery --root <dir> --output <gallery.json> "
"[--detector <path>] [--arcface <path>] [--max-side <N>]\n"
" [--min-face-px <px>]\n";
"[--detector <path>] [--arcface <path>] [--max-side <N>]\n";
return 1;
}
@@ -73,7 +70,6 @@ int main(int argc, char** argv) {
cfg.detector_conf = conf;
cfg.detector_nms = nms_thr;
cfg.max_side = max_side;
cfg.min_face_px = min_face_px;
try {
ActorGallery gallery = build_gallery(cfg);
src/config.hpp
+36 -145
View File
@@ -11,20 +11,6 @@ enum class Verbosity {
standard, // per-frame detail: bbox, similarity, unknowns logged
xray, // Jellyfin-Xray format: {"second": ["Actor", ...], ...}
};
// How a track's accepted frames become a reported presence window.
enum class PresenceMode {
// A claim IS its track's [first_seen, last_seen] (AR-012/AR-013). The
// default and the only mode whose semantics the register validated.
track_extent,
// Flood-fill: snap each claim to the shot it sits in, so an actor seen once
// anywhere in a scene is reported for the whole scene [prev_boundary,
// next_boundary]. Trades precision for recall against X-Ray's per-scene cast
// granularity. Snaps to TransNetV2 shot boundaries (is_scene_boundary) when a
// scene detector populated them, else to the always-on histogram cuts
// (is_cut). With no boundaries at all it degrades to track_extent per claim.
flood,
};
// debug verbosity = compile with -DSAE_DEBUG → scene_analyze_debug binary
struct Config {
@@ -46,14 +32,6 @@ struct Config {
// scripts/optimizer/SCHEMA.md) for offline threshold-sweep replay via sae_kpn.
std::string dump_embeddings_path;
/// TRACES: VR-015 | PR-004
// When set, write a per-node timing and bottleneck report here (src/
// benchmark.hpp) and print it at shutdown. Costs one background thread
// reading relaxed atomics on a timer, so it is safe to leave on, but a
// measurement run should still be isolated (nothing else on the GPU).
std::string benchmark_path;
int benchmark_interval_ms{100}; // channel-occupancy sampling period
// ── Sampling ─────────────────────────────────────────────────────────────
float sample_fps{1.0f}; // frames to analyse per second of movie
float max_decode_fps{0.f}; // wall-clock cap on source decode rate (0 = uncapped)
@@ -86,52 +64,15 @@ struct Config {
std::string arcface_engine; // optional path to a pre-built TRT engine; bypasses ORT
int embed_batch_size{4}; // max faces per ORT Run() call — bounds per-call latency
float match_prior{0.5f}; // base-rate prior; 0.5 = use calibrated sigmoid directly
// Tuned by Differential Evolution against Amazon X-Ray per-second presence
// over the 4-film rep4 matrix. Best model+mode: LVFace-B_Glint360K, full
// gallery, expansion on. Supersedes an earlier 9-film scene-union tuning
// (0.76); that metric hid out-of-cast false positives.
//
// **Read the provenance before trusting the value.** Two things about it:
//
// 1. The document it came from no longer exists under that name. It was
// docs/rep4-optimizer-results.md, renamed to docs/model-bakeoff.md and
// then rewritten (0bd2747). This comment pointed at the dead path for
// long enough that the number looked unsourced. The original is still
// readable at `git show d340da7:docs/rep4-optimizer-results.md`, where
// the shipped triple appears as
// `prob_threshold=0.754, anneal_sec=35.5`.
//
// 2. **0.754 predates a scoring bug fix and was never re-derived.** That
// same rewrite reports finding "a real scoring bug in optimize.py: a
// candidate whose hardest film's replay timed out was averaged over
// survivors instead of penalized, silently rewarding partial coverage.
// Affected 3 of 16 training combos". The corrected sweep converged
// somewhere else — the surviving document records anneal_sec=59.2,
// extinction_sec=59.2 against the 35.5/57.4 shipped alongside this
// threshold — and no corrected prob_threshold is recorded anywhere.
// (The other two constants are now withdrawn outright, which is why
// only this one still matters.)
//
// The doc is also candid that the optimum "generalizes unevenly — strong on
// 3 of 5 held-out films, badly broken on 2 (one with a 974-count misID
// blowup)", and that it is shipped anyway because it still beats the old
// defaults on average. That is a defensible call and not a settled,
// film-agnostic optimum; it should be visible here rather than only in a
// document this comment used to point at incorrectly.
// prob_threshold tuned by Differential Evolution against Amazon X-Ray per-scene
// presence over 4 films, per-second metric (see docs/rep4-optimizer-results.md).
// Best model+mode: LVFace-B_Glint360K, full gallery, expansion on. Supersedes the
// earlier 9-film scene-union-metric tuning (0.76) — that metric is now known to
// have hidden out-of-cast false positives (see docs/optimizer-experiments.md).
float prob_threshold{0.754f}; // posterior P(match | sim, prior) threshold
// TRACES: AR-024 | SR-002
// match_threshold (0.45), match_ratio (0.80) and match_ratio_ceil (0.65) are
// RETIRED, joining track_max_embed_dist, cut_revive_sim, expand_novelty_sim
// and expand_track_spread_max. All were raw cosine distances, and they were
// the accept rule whenever the calibration fit failed — so the one situation
// in which the pipeline knew its probabilities were untrustworthy was the
// one in which it stopped using them. An unfitted sigmoid is now the
// fallback everywhere, which is at least the same wrong number in every
// stage. See identity_matcher_node.hpp.
// ── Presence derivation ──────────────────────────────────────────────────
// How accepted frames become a reported window. flood requires scene_detect.
PresenceMode presence_mode{PresenceMode::track_extent};
float match_threshold{0.45f}; // cosine distance hard ceiling fallback (no calibration)
float match_ratio{0.80f}; // ratio test fallback: accept if best/second < ratio
float match_ratio_ceil{0.65f}; // ratio test only fires below this absolute distance
// ── Cut detection ────────────────────────────────────────────────────────
float cut_threshold{0.70f}; // grayscale histogram correlation below this → hard cut
@@ -150,28 +91,17 @@ struct Config {
// at ~0.50; real boundaries spike to ~0.7+)
int scene_stride{50}; // frames advanced between windows (≤ kWindow)
// Dense-decode knobs (only active with scene_detect). Dense decode of every
// native-rate frame is the pipeline's cost driver, which is what made the
// temporal shortcut below tempting.
/// TRACES: AR-011 | SR-002
// scene_decode_fps: rate the source decodes at in dense mode.
// **0 = native, and native is the only correct setting.** kWindow is 100
// frames: at native 25 fps that window spans ~4 s, which is what
// TransNetV2 was trained on; at the 12 fps this used to default to it
// spans ~8.3 s, so the model saw half-speed motion over twice its
// temporal context. Boundary *timestamps* stay right either way — which
// is exactly why the degradation was invisible, and why the compressed
// separation it produced (~0.50 baseline against ~0.7+ peaks) was read
// as a property of the export rather than of the input. Lowering this
// buys decode time by running the model off-distribution; reach for
// dense_scale or scene_stride instead, which do not.
// Dense-decode throughput knobs (only active with scene_detect). Dense decode
// of every native-rate frame is the pipeline's cost driver; these trade a
// little boundary precision for a large speedup.
// scene_decode_fps: rate the source decodes at in dense mode. Lower =
// fewer frames decoded. TransNetV2 tolerates ~12fps; boundary timestamps
// stay correct (keyed off each frame's real timestamp). 0 = native fps.
// dense_scale: downscale factor applied to decoded frames in dense mode
// (0<f≤1; e.g. 0.5 = half size). Cheaper sws_scale + smaller frames
// through the fanout. A spatial reduction, and TransNetV2 downsamples to
// 48×27 regardless, so unlike the above it is a documented, understood
// degradation. NOTE: also shrinks what the face detector sees — keep
// ≥0.5 on 1080p sources so SCRFD still resolves small faces. 1 = off.
float scene_decode_fps{0.f}; // dense decode rate (0 = native)
// through the fanout. NOTE: also shrinks what the face detector sees —
// keep ≥0.5 on 1080p sources so SCRFD still resolves small faces. 1 = off.
float scene_decode_fps{12.0f}; // dense decode rate (0 = native)
float dense_scale{1.0f}; // dense-mode frame downscale (1 = off)
// ── Face tracking (frame-to-frame) ───────────────────────────────────────
@@ -195,55 +125,17 @@ struct Config {
// ordinary miss, once for a cut). Seconds mean one thing at any sample rate.
double track_extinction_sec{5.0};
// ── Ownership and evidence accumulation (AR-025) ──────────────────────────
// TRACES: AR-025, AR-017 | SR-002
// These four decided how presence is claimed and were unreachable: they
// lived as in-class initialisers on TrackRegistry::Config and
// EvidenceDiscounter::Config, and main constructed the discounter with the
// one-argument constructor, so nothing short of a recompile could move
// them. rho_max's own comment defers to "the sweep (VR-007)" for where it
// belongs — a sweep that could not reach it.
//
// ownership_logodds is arguably the most consequential constant in the
// pipeline after prob_threshold: below it a track produces no presence
// claim at all, so it decides whether an actor is reported rather than how
// confidently. 2.0 is a posterior of ~0.88. Unswept.
float ownership_logodds{2.0f};
// How much a single observation may move a track's belief. n_eff =
// n / (1 + (n-1)·rho), so rho_max caps what a repeated view can ever be
// worth: 0.5 caps it at two independent observations however long the shot
// runs. It is deliberately below 1 — a held pose still yields a fresh
// detection, alignment and noise realisation, so a little independent
// evidence survives. Setting it to 1 freezes belief after the first frame,
// which is the bug this replaced.
float evidence_rho_max{0.5f};
// P(same view) below this and the observation counts as a genuinely new
// look, so it joins the per-track view set.
float evidence_admit_below{0.6f};
// Distinct views remembered per track, which bounds the novelty comparison.
int evidence_max_views{8};
// ── Scene tracking ────────────────────────────────────────────────────────
// TRACES: AR-012, AR-013 | SR-002
// extinction_sec (57.4) and anneal_sec (35.5) are GONE, along with
// SceneTrackerFunc, which is what read the first of them. docs/SPEC.md
// specified this removal and ended it "grep for both names and expect no
// survivors"; there were about forty, and the register meanwhile recorded
// both as Withdrawn and "deleted rather than retained at zero" on the
// grounds that a field naming a mechanism the pipeline no longer has is
// actively misleading.
//
// Both existed to bridge gaps between isolated accepted frames. A track
// that survives its own gaps leaves them nothing to do: AR-012 makes a
// window the extent of a track an actor owns, and AR-013 ends it at the
// last sighting. The keep-alive answered the same question again and
// answered it worse, by re-opening exactly the trailing cool-down AR-013
// refuses.
//
// track_extinction_sec above is NOT the same knob under a new name. It
// bounds how long a lost track stays available for re-association, which is
// a tracking question; it never extends a presence claim.
// extinction_sec re-tuned by DE against X-Ray per-second presence, 4-film rep4
// matrix (docs/rep4-optimizer-results.md). Reverses the earlier "short is better"
// finding: with a stricter prob_threshold, a long extinction window bridges real
// presence gaps (occlusion, turned face) instead of just smearing FPs — every
// model's best config pushed to ~90%+ of the search ceiling (tried up to 60s).
// The ceiling kept getting hit, so treat 60 as "good enough", not a proven optimum.
double extinction_sec{57.4}; // keep actor active this many seconds after last detection
// anneal_sec: previously found INSENSITIVE at a 1–30s range; the wider rep4 sweep
// (1–60s) also pushed this to the ceiling alongside extinction_sec (see above).
double anneal_sec{35.5}; // merge actor windows separated by less than this into one epoch
// ── Per-film gallery expansion ────────────────────────────────────────────
// Within one uncut track every face is the same physical person — a free
@@ -252,26 +144,25 @@ struct Config {
// new reference views; they are promoted into a per-film, in-memory annex so
// later frames/tracks of that actor at similar poses recognise. See
// gallery/track_gallery.hpp.
// Default ON: the rep4 matrix (docs/model-bakeoff.md, "Two effects in
// isolation") found expansion helps recall on the full (unrestricted)
// gallery for the winning model/mode — the opposite of the earlier
// assumption that it only helps restricted galleries. The same section is
// explicit that on the full gallery it buys +2.1pp F1 and +3.9pp recall
// "at a real cost" in misIDs, where in restricted mode it is a clean win.
// Default ON: rep4 matrix (docs/rep4-optimizer-results.md) found expansion helps
// recall on the full (unrestricted) gallery for the winning model/mode — the
// opposite of the earlier assumption that it only helps restricted galleries.
bool expand_gallery{true}; // master switch
int expand_buffer_size{20}; // per-track diversity buffer capacity
// TRACES: AR-018, AR-024 | SR-005
// Banded admission for the per-subject store, in PROBABILITY space. An
// embedding joins only if P(same person) against something already stored
// lands inside [lo, hi]: above hi it is redundant, below lo it is evidence
// the track is not one person. The same lo is re-applied to the whole store
// at promotion time — see track_gallery.hpp. This is the only threshold the
// expansion path has: it replaces the raw-cosine expand_novelty_sim (0.55)
// and expand_track_spread_max (0.60), which are retired (AR-024).
// the track is not one person. Replaces expand_novelty_sim, a raw cosine.
// Working values pending VR-007; sweep both bounds, they fail in opposite
// directions.
float expand_band_lo{0.90f};
float expand_band_hi{0.95f};
float expand_novelty_sim{0.55f}; // promote only embeddings whose best sim to the
// actor's refs is below this (gallery-far / novel)
float expand_track_spread_max{0.60f}; // reject promotion if the retained buffer's
// internal spread (1 - min pairwise sim) exceeds
// this — guards track-ID collisions / two people
int expand_min_anchor_frames{3}; // require ≥N accepted frames naming the actor before
// the track is confirmed and its buffer promoted
std::string expand_debug_dir; // if set, dump promoted mugshots + embeddings here
src/dump_embeddings.cpp
-6
View File
@@ -8,12 +8,6 @@
// gallery file needed. Purpose-built for the optimizer's replay corpus and the
// embedding-model bake-off (dump each --arcface model over the film set).
//
// The camera-position (histogram) cut detector runs, so frames/is_cut is recorded
// in the dump. Flood-fill presence snaps to those cuts. TransNetV2 scene detection
// is NOT run here: on the ROCm build it needs MIGraphX, whose statically-linked
// LLVM collides with the VAAPI decoder's system libLLVM and crashes at compile, so
// the two cannot share a process. is_scene_boundary therefore stays 0 here.
//
// Usage:
// dump_embeddings --movie <path> --out <dump.h5> [--arcface <model.onnx>]
// [--detector <model.onnx>] [--fps 1] [--start S] [--end S]
src/face_utils.hpp
+1 -91
View File
@@ -1,5 +1,5 @@
#pragma once
/// TRACES: AR-005, AR-029, AR-030 | SR-002
/// TRACES: AR-005, AR-030 | SR-002
#include "types.hpp"
#include <opencv2/core.hpp>
@@ -143,96 +143,6 @@ inline cv::Mat align_face(const cv::Mat& img,
return crop;
}
// ── crop_sharpness ────────────────────────────────────────────────────────────
/// TRACES: AR-029 | SR-002
//
// Normalised variance of the Laplacian over the aligned 112×112 crop: the AR-029
// sharpness axis. Returns -1 for an empty crop (unscored), matching the
// DetectedFace sentinel.
//
// sharpness = Var(∇²I) / Var(I)
//
// Two normalisations, each removing a quantity that would otherwise be read as
// blur:
//
// - **Divided by the image variance, so contrast cannot leak in.** Scaling
// intensity by α scales the Laplacian by α too, so both variances scale by α²
// and the ratio is unchanged. A raw Var(∇²I) — the textbook measure — instead
// falls with exposure, so a dim scene reads as soft and a graded-up one as
// sharp. VR-012 has to locate one knee across films whose grading differs by
// more than their focus does; an uncalibrated measure would put the knee in a
// different place per film, which is the AR-024 failure in another metric.
// - **Measured on the aligned crop, so size cannot leak in.** The destination
// frame is fixed at 112×112 (AR-002 owns size, and double-counting it here
// would make every small face read as blurred). What the ratio reports is the
// detail actually present in the embedder's input — so a small sharp face can
// and does outscore a large soft one. That is the claim; it is *not* a claim
// of invariance to source resolution, because a 40 px face warped up to 112
// genuinely carries less detail, and hiding that would defeat the point.
//
// Frequency-domain reading of why the blur ladder is monotone: with
// Var(∇²I) = ∫|ω|⁴|F(ω)|² and Var(I) = ∫|F(ω)|², the ratio is E[|ω|⁴] under the
// image's own spectral measure. Gaussian blur multiplies that measure by
// e^{-σ²|ω|²}, concentrating it at low |ω|, so the expectation falls strictly
// with σ. It is a property of the construction, not a fitted behaviour.
//
// **Three known hazards, for VR-012 to check rather than for a threshold to
// absorb.** All are recorded here because they are properties of the measure,
// visible in the dumped distribution, and neither should be papered over by a
// correction chosen before that distribution has been looked at.
//
// 1. **Border fill.** `align_face` warps with BORDER_CONSTANT, so a face
// crossing the frame edge brings a hard black step into the crop, and a
// step edge is high-frequency. The normalisation blunts it — the fill
// inflates Var(I) as well as Var(∇²I) — but does not remove it, so
// heavily-cropped faces may read sharper than they are. The fix is either a
// validity mask or a different border mode, and the second changes what the
// embedder is fed (AR-011).
//
// 2. **The contrast invariance is exact in the algebra and approximate in
// 8 bits.** Scaling I by α cancels exactly; what does not cancel is the
// quantisation floor of a stored crop, which is broadband and so lands in
// the numerator. It matters only where there is little signal left to
// compete with it: on the AR-029 test texture a half-contrast copy reads
// 0.9% high when sharp, 24% high at sigma 1.2 and 148% high at sigma 2.5.
// A crop that is both **dim and soft therefore reads sharper than it is** —
// the low corner of the axis, and the corner VR-012 must put a knee in.
//
// 3. **It reports where the energy sits, not how much there is.** A crop whose
// energy is *already* concentrated at high frequency — dense film grain,
// a face against foliage — loses numerator and denominator together under
// blur, so the ratio moves less than the damage does. Measured on a
// flat-spectrum synthetic, an anisotropic (motion) smear even makes it rise,
// because the surviving perpendicular detail really is as fine as before.
// Natural crops have the low-frequency mass that keeps the denominator
// steady, and on those both ladders fall (see the AR-029 tests, which use a
// 1/f texture for exactly this reason). The same property means the axis
// conflates focus with intrinsic texture — a bearded face outscores a smooth
// one at equal focus — which is true of every no-reference sharpness measure
// and is why AR-028 carries the number instead of thresholding on it.
inline float crop_sharpness(const cv::Mat& crop) {
if (crop.empty()) return -1.f;
cv::Mat gray;
if (crop.channels() == 3) cv::cvtColor(crop, gray, cv::COLOR_BGR2GRAY);
else gray = crop;
cv::Mat lap;
cv::Laplacian(gray, lap, CV_32F, 3);
cv::Scalar mean_i, sd_i, mean_l, sd_l;
cv::meanStdDev(gray, mean_i, sd_i);
cv::meanStdDev(lap, mean_l, sd_l);
const double var_i = sd_i[0] * sd_i[0];
// A flat crop has no detail to be sharp or soft about, and the ratio is 0/0.
// Zero is the honest answer and keeps the axis finite; -1 would claim the
// face was never scored, which is a different fact.
if (var_i < 1e-6) return 0.f;
return static_cast<float>((sd_l[0] * sd_l[0]) / var_i);
}
// ── enhance_for_retry ────────────────────────────────────────────────────────
// Used when initial face detection finds nothing. Pads the image by 50%
// (border-replicated, so the detector doesn't see a hard edge) and applies
src/gallery/gallery_builder.cpp
-21
View File
@@ -96,27 +96,6 @@ ActorGallery build_gallery(const BuildConfig& cfg) {
return a.confidence < b.confidence;
});
// Reject faces too small to embed honestly.
//
// The reference images are crops cut from the film, not mugshots, so
// the detected face can be a small fraction of the image. Upscaling a
// 30 px face to ArcFace's 112x112 feeds the model an input it was
// never trained for, and it answers with a confident, plausible,
// wrong embedding.
//
// At inference that costs one frame. Here it is permanent: a poisoned
// reference sits in the gallery and corrupts every future match
// against that character, which is exactly the kind of error that is
// invisible without a study that should not have been needed.
if (cfg.min_face_px > 0.f) {
const float side = std::min(best.bbox.width, best.bbox.height);
if (side < cfg.min_face_px) {
std::cerr << " [skip] face " << side << "px < " << cfg.min_face_px
<< "px: " << img_file.path().filename() << "\n";
continue;
}
}
cv::Mat crop = align_face(img, best.landmarks);
if (crop.empty()) {
std::cerr << " [skip] alignment failed: " << img_file.path().filename() << "\n";
src/gallery/gallery_builder.hpp
-10
View File
@@ -29,16 +29,6 @@ struct BuildConfig {
float detector_conf{0.5f};
float detector_nms{0.4f};
int max_side{500}; // downscale source images to this max dimension
/// Minimum detected-face side, in pixels of the (possibly downscaled)
/// source image. 0 disables the check.
///
/// References below this are dropped rather than upscaled: a face smaller
/// than the embedder's input is off-distribution, and a bad reference
/// poisons every match against that identity for the life of the gallery.
/// Mirrors the inference-side --min-face-px so the gallery is built from
/// the same face scales it will be matched against.
float min_face_px{0.f};
// before detection — TMDB portraits are ~2k px,
// SCRFD trains on smaller faces and detection
// confidence drops on huge inputs. 0 = disabled.
src/gallery/gallery_calibration.hpp
-13
View File
@@ -162,19 +162,6 @@ inline GalleryCalibration calibrate_gallery(
for (const auto& e : by_actor[ai]) {
bool dup = false;
for (const auto& k : kept) {
// EXCEPTION: AR-024 this asks whether two vectors are THE SAME
// VECTOR, not whether two faces are the same person.
//
// Two independent reasons, either sufficient. First, at
// 1 - 1e-7 the threshold is a floating-point identity test: it
// catches one source image embedded twice, and no genuine pair
// of distinct photographs lands there. Nothing about it is a
// decision, so there is nothing for a probability to mean.
//
// Second, and structurally: this IS the calibration fit. The
// dedup runs on its input, before (a, b) exist. A calibrated
// comparison here would have to be calibrated by the fit it is
// feeding, which is not a thing that can be arranged.
if (cosine_similarity(e, k) > kDedupSimThreshold) { dup = true; break; }
}
if (!dup) kept.push_back(e);
src/gallery/track_gallery.hpp
+102 -177
View File
@@ -9,7 +9,6 @@
#include <iostream>
#include <limits>
#include <map>
#include <stdexcept>
#include <string>
#include <vector>
@@ -37,41 +36,39 @@
// 2. Promotion (on confirmation). A track is "owned" by actor A once ≥N frames
// have been accepted (by the matcher's calibrated posterior) as A. On
// confirmation the retained buffer — the hard, gallery-far poses — is
// promoted into A's per-film annex, subject to one safety gate: the band's
// lower bound, re-applied across the whole store (see `store_coherence`).
// promoted into A's per-film annex, after two safety gates:
// • novelty: only embeddings whose best sim to A's refs is below
// expand_novelty_sim are added (skip poses already covered);
// • spread: if the retained buffer's internal spread (1 − min pairwise
// cosine sim) exceeds expand_track_spread_max the whole track is
// rejected — such spread signals a track-ID collision merging two
// people, whose embeddings must never enter A's annex.
//
// There is exactly one threshold here, the AR-018 band, and it is a calibrated
// probability. Novelty is no longer a threshold at all — the eviction policy
// above *orders* by gallery similarity rather than cutting at a constant, and
// the band's upper bound refuses the redundant views at the door. The raw
// cosines this replaces, expand_novelty_sim and expand_track_spread_max, are
// retired under AR-024.
//
// TRACES: AR-026 | SR-001
// The annex is in-memory and discarded when the process exits, but it is NOT
// small: every owned track contributes, so it grows with cast size and film
// length. It is therefore held as a contiguous row-major matrix with a parallel
// actor index — the same flat_emb_/flat_actor_ shape the baked gallery uses —
// and the matcher hands promoted rows to the similarity engine rather than
// scanning them with a host-side loop. The deferred pass (AR-020) needs the same
// contiguous operand to score the TBI queue against in one multiply.
//
// Promoted embeddings only help SUBSEQUENT frames and later tracks of A — the
// pipeline stays streaming, no emitted output is buffered or relabelled.
// The annex is CPU-side and in-memory: it is small (tens of embeddings) so the
// matcher scans it with a scalar loop, and it is discarded when the process
// exits. Promoted embeddings only help SUBSEQUENT frames and later tracks of A —
// the pipeline stays streaming, no emitted output is buffered or relabelled.
struct TrackGallery {
// One promoted reference view held in the per-actor annex.
struct AnnexEntry {
Embedding emb;
int actor_idx{-1};
};
explicit TrackGallery(const Config& cfg)
: enabled_(cfg.expand_gallery)
, buffer_size_(std::max(1, cfg.expand_buffer_size))
, band_lo_(cfg.expand_band_lo)
, band_hi_(cfg.expand_band_hi)
, novelty_sim_(cfg.expand_novelty_sim)
, spread_max_(cfg.expand_track_spread_max)
, min_anchor_frames_(std::max(1, cfg.expand_min_anchor_frames))
, debug_dir_(cfg.expand_debug_dir)
{
if (!enabled_) return;
std::cerr << "[track_gallery] per-film expansion ON"
<< " buffer=" << buffer_size_
<< " band=[" << band_lo_ << ", " << band_hi_ << "]"
<< " novelty_sim<" << novelty_sim_
<< " spread_max=" << spread_max_
<< " min_anchor_frames=" << min_anchor_frames_;
if (!debug_dir_.empty()) {
std::filesystem::create_directories(debug_dir_);
@@ -82,47 +79,16 @@ struct TrackGallery {
bool enabled() const { return enabled_; }
/// TRACES: AR-026 | SR-001
/// The annex as a contiguous row-major matrix (annex_size() × 512) plus the
/// parallel actor index. Only ever grows, never reordered, so a row index is
/// stable for the life of the film — which is what lets the similarity
/// engine hold the same rows and the actor mapping stay a plain vector.
int annex_size() const { return static_cast<int>(annex_actor_.size()); }
const float* annex_data() const { return annex_emb_.data(); }
const std::vector<int>& annex_actors() const { return annex_actor_; }
/// One annex row (512 floats). The deferred pass (AR-020) scores the whole
/// matrix at once via annex_data(); this is for inspecting a single view.
const float* annex_row(int i) const {
return annex_emb_.data() + static_cast<size_t>(i) * kEmbDim;
}
/// TRACES: AR-026 | SR-001
/// Hand the caller every row promoted since the previous call, appending to
/// its buffers, and return how many. The matcher pushes these into the
/// similarity engine so the next frame's single GEMM covers the annex —
/// draining rather than re-reading the whole matrix keeps that O(promoted),
/// not O(annex), per frame.
int drain_promotions(std::vector<float>& emb_out, std::vector<int>& actor_out) {
const int pending = annex_size() - drained_;
if (pending <= 0) return 0;
emb_out.insert(emb_out.end(),
annex_emb_.begin() + static_cast<size_t>(drained_) * kEmbDim,
annex_emb_.end());
actor_out.insert(actor_out.end(),
annex_actor_.begin() + drained_, annex_actor_.end());
drained_ = annex_size();
return pending;
}
// Current annex contents (empty when disabled). The matcher scans these
// alongside the baked gallery so a promoted view can win best-of-N for its
// actor. Returned by const-ref; only grows, never reordered.
const std::vector<AnnexEntry>& annex() const { return annex_; }
// Offer one observed face to its track's diversity buffer.
// track_id : face_tracker track (−1 = untracked, ignored)
// emb : this frame's raw embedding
// best_actor : actor with the highest gallery similarity for this face
// best_gal_sim : that similarity (best sim to best_actor's baked+annex
// refs) — a raw cosine, the last one in this class: it is
// calibrated on entry and only the probability is stored
// best_gal_sim : that similarity (best sim to best_actor's baked+annex refs)
// accepted : true if the matcher accepted this face as best_actor
// crop : aligned crop, retained only when debug dumping is on
void observe(int track_id, const Embedding& emb,
@@ -133,52 +99,25 @@ struct TrackGallery {
TrackState& ts = tracks_[track_id];
/// TRACES: AR-019 | SR-005
// accepted_frames is an EVIDENCE FLOOR, not an identity decision: it
// asks "has this track been recognised often enough to be worth
// promoting", never "who is it". Who it is comes from the registry.
//
// There used to be a per-actor tally here too, and promote() fell back
// to its plurality winner. That made two answers to "who is this track"
// able to coexist, and the local one ignored the Bayesian accumulation
// entirely -- weighting thirty near-identical looks the same as thirty
// distinct ones, which is exactly what AR-025's discounting exists to
// stop. Since promotion only fired on the local count, the fallback was
// reachable in the live pipeline and not merely in tests: three
// accepted frames arrive well before a posterior crosses ownership.
if (accepted && best_actor >= 0) ts.accepted_frames++;
// Vote toward ownership: only accepted frames name an actor, and a track
// that flip-flops between actors is ambiguous, so we tally per actor and
// pick the plurality winner at confirmation time.
if (accepted && best_actor >= 0) {
ts.actor_votes[best_actor]++;
ts.accepted_frames++;
}
insert_into_buffer(ts, emb, best_gal_sim, crop);
// Confirm and promote once BOTH hold: the registry owns this track, and
// enough frames have been accepted to be worth the slots. Ownership is
// the necessary one -- without it there is no actor to promote into.
if (!ts.promoted && ts.registry_owner >= 0 &&
ts.accepted_frames >= min_anchor_frames_)
// Confirm and promote as soon as the anchor threshold is met, once.
if (!ts.promoted && ts.accepted_frames >= min_anchor_frames_)
promote(track_id, ts);
}
/// TRACES: AR-019 | SR-005
/// Drop the buffers of tracks the registry no longer has.
///
/// `alive` is the registry's own liveness test, so this annotates the track
/// pool rather than duplicating it — the same shape as FaceTrackerFunc's
/// prune_boxes, and for the same reason: a second opinion about which
/// tracks exist is a second thing that can be wrong.
///
/// This replaces a `forget(int)` that had NO callers, under a comment
/// asserting "called by the matcher when it observes a cut or track
/// disappearance". The cut half was true by another route (clear_tracks);
/// the disappearance half was not, so a track that died quietly kept its
/// buffer until the next cut cleared everything.
template <typename AlivePredicate>
void prune_dead(const AlivePredicate& alive) {
if (!enabled_) return;
for (auto it = tracks_.begin(); it != tracks_.end(); ) {
if (alive(it->first)) ++it;
else it = tracks_.erase(it);
}
}
// Drop a track's buffer when the face_tracker expires it or on a scene cut,
// so stale/cross-cut embeddings can never be promoted later. Called by the
// matcher when it observes a cut or track disappearance.
void forget(int track_id) { tracks_.erase(track_id); }
/// TRACES: AR-019 | SR-005
/// The registry's verdict on who this track is. Authoritative: it comes from
@@ -191,20 +130,11 @@ struct TrackGallery {
}
/// TRACES: AR-024 | SR-005
/// Supply the calibration belonging to the active embedder.
///
/// Required, not optional. The default used to be `max(0, cosine)` — a raw
/// cosine worn as a probability, which made `expand_band_lo = 0.90` mean
/// "cosine above 0.9" in a test and "P(same person) above 0.9" in
/// production. Those are wildly different gates, and nothing announced the
/// switch. `FaceTrackerFunc` already refuses to construct without a
/// calibration for the same reason; this now matches it.
void set_calibration(std::function<float(float)> c) {
if (!c) throw std::invalid_argument(
"track_gallery: a calibration is required — the admission band is "
"expressed in probability space (AR-024)");
calibrate_ = std::move(c);
}
/// Supply the calibration belonging to the active embedder. Without it the
/// band falls back to treating cosine as probability, which is wrong but
/// bounded — and the default is loud in the header rather than silent.
void set_calibration(std::function<float(float)> c) { calibrate_ = std::move(c); }
void set_band(float lo, float hi) { band_lo_ = lo; band_hi_ = hi; }
/// Embeddings the band refused. A store that admits nothing is as wrong as
/// one that admits everything, and neither is visible without this.
@@ -216,15 +146,13 @@ struct TrackGallery {
private:
struct BufEntry {
Embedding emb;
/// P(same person) against the owning actor's refs when observed —
/// calibrated at the door (AR-024), so the eviction ordering below is a
/// comparison of probabilities and the struct holds no bare cosine.
float gal_p{0.f};
float gal_sim{0.f}; // best sim to owning actor's refs when observed
cv::Mat crop; // populated only when debug_dir_ set
};
struct TrackState {
std::vector<BufEntry> buf;
std::map<int, int> actor_votes; // actor_idx → accepted-frame count
int accepted_frames{0};
bool promoted{false};
int registry_owner{-1}; ///< AR-019: authoritative
@@ -265,8 +193,8 @@ private:
if (!admit(ts, emb)) { ++rejected_; return; }
BufEntry e;
e.emb = emb;
e.gal_p = calibrate_(gal_sim);
e.emb = emb;
e.gal_sim = gal_sim;
if (!debug_dir_.empty() && !crop.empty()) e.crop = crop.clone();
if (static_cast<int>(ts.buf.size()) < buffer_size_) {
@@ -275,17 +203,13 @@ private:
}
// Buffer full: evict the member the gallery recognises best (highest
// gal_p) — least informative — but only if the newcomer is at least as
// gal_sim) — least informative — but only if the newcomer is at least as
// novel. Keeping the most gallery-far views is the whole point.
//
// This is an *ordering*, not a threshold: there is no constant to tune,
// and novelty-seeking lives here rather than in a cutoff. It ranks
// probabilities, so it says the same thing across models (AR-024).
int worst_i = -1;
float worst_p = e.gal_p; // newcomer's probability is the bar to beat
int worst_i = -1;
float worst_sim = e.gal_sim; // newcomer's sim is the bar to beat
for (int i = 0; i < static_cast<int>(ts.buf.size()); ++i) {
if (ts.buf[i].gal_p > worst_p) {
worst_p = ts.buf[i].gal_p;
if (ts.buf[i].gal_sim > worst_sim) {
worst_sim = ts.buf[i].gal_sim;
worst_i = i;
}
}
@@ -297,25 +221,29 @@ private:
void promote(int track_id, TrackState& ts) {
ts.promoted = true; // idempotent: never promote a track twice
const int actor = ts.registry_owner;
if (actor < 0) return; // unreachable: observe() gates on this
int actor = owning_actor(ts);
if (actor < 0) return;
// ── Safety gate: the band's lower bound, across the whole store ──────
float worst = store_coherence(ts.buf);
if (worst < band_lo_) {
// ── Safety gate: internal spread ─────────────────────────────────────
// A legitimate single-person track varies in pose but stays reasonably
// self-similar. Large spread signals two people merged under one track
// ID — reject the whole track rather than poison the actor's annex.
float spread = buffer_spread(ts.buf);
if (spread > spread_max_) {
std::cerr << "[track_gallery] track " << track_id
<< " → actor " << actor
<< " REJECTED (worst pairwise P=" << worst
<< " < " << band_lo_ << ", likely ID collision)\n";
<< " REJECTED (spread " << spread
<< " > " << spread_max_ << ", likely ID collision)\n";
return;
}
int added = 0;
for (const auto& be : ts.buf) {
// Row-major append: the matrix stays contiguous so the matcher can
// hand whole blocks of new rows to the GEMM path (AR-026).
annex_emb_.insert(annex_emb_.end(), be.emb.begin(), be.emb.end());
annex_actor_.push_back(actor);
// ── Safety gate: novelty ─────────────────────────────────────────
// Skip poses the gallery already covers; only gallery-far views are
// worth the annex slot (and the extra per-frame scan cost).
if (be.gal_sim >= novelty_sim_) continue;
annex_.push_back({be.emb, actor});
if (!debug_dir_.empty() && !be.crop.empty())
dump_mugshot(track_id, actor, added, be);
++added;
@@ -323,39 +251,42 @@ private:
std::cerr << "[track_gallery] track " << track_id
<< " confirmed actor " << actor
<< " (" << ts.accepted_frames << " accepted frames, worst "
<< "pairwise P=" << worst << ") — promoted " << added
<< " views; annex now " << annex_size() << "\n";
<< " (" << ts.accepted_frames << " accepted frames, spread "
<< spread << ") — promoted " << added << "/"
<< ts.buf.size() << " views; annex now "
<< annex_.size() << "\n";
}
/// TRACES: AR-018, AR-024 | SR-005
/// The store's weakest pairwise P(same person) — the band's lower bound
/// asked of every pair, not just of the best match at the door.
///
/// `admit` compares a newcomer against its *closest* existing member, so a
/// track that drifts gradually can chain A→B→C with every step inside the
/// band while A and C are strangers. That is precisely the shape a track-ID
/// collision takes when two people are merged over a slow pan, so the bound
/// is re-asked here across all pairs before anything reaches an actor's
/// annex. Same bound, same probability space — not a second constant.
///
/// A store of one has no pair to disagree; it is coherent by construction,
/// hence 1.
float store_coherence(const std::vector<BufEntry>& buf) const {
float worst = std::numeric_limits<float>::max();
/// Prefer the registry's verdict; fall back to the local tally only when no
/// registry is attached (unit tests, replay harness).
static int owning_actor(const TrackState& ts) {
if (ts.registry_owner >= 0) return ts.registry_owner;
return plurality_actor(ts);
}
static int plurality_actor(const TrackState& ts) {
int best = -1, best_votes = 0;
for (const auto& [ai, v] : ts.actor_votes) {
if (v > best_votes) { best_votes = v; best = ai; }
}
return best;
}
// Spread = 1 − min pairwise cosine similarity over the buffer (0 when <2).
static float buffer_spread(const std::vector<BufEntry>& buf) {
float min_sim = std::numeric_limits<float>::max();
for (size_t i = 0; i < buf.size(); ++i)
for (size_t j = i + 1; j < buf.size(); ++j)
worst = std::min(worst,
calibrate_(cosine_similarity(buf[i].emb, buf[j].emb)));
if (worst == std::numeric_limits<float>::max()) return 1.f;
return worst;
min_sim = std::min(min_sim, cosine_similarity(buf[i].emb, buf[j].emb));
if (min_sim == std::numeric_limits<float>::max()) return 0.f;
return 1.f - min_sim;
}
void dump_mugshot(int track_id, int actor, int idx, const BufEntry& be) {
#ifdef SAE_DEBUG
char name[64];
std::snprintf(name, sizeof(name), "trk%d_actor%d_%d_p%.3f.jpg",
track_id, actor, idx, be.gal_p);
std::snprintf(name, sizeof(name), "trk%d_actor%d_%d_sim%.3f.jpg",
track_id, actor, idx, be.gal_sim);
cv::imwrite((std::filesystem::path(debug_dir_) / name).string(), be.crop);
#else
(void)track_id; (void)actor; (void)idx; (void)be;
@@ -363,25 +294,19 @@ private:
}
/// cosine → P(same person). The one probability space the pipeline reasons
/// in; see gallery_calibration.hpp's same_person_probability. Never default
/// constructed to an identity-ish stand-in — see set_calibration.
std::function<float(float)> calibrate_;
/// in; see gallery_calibration.hpp's same_person_probability.
std::function<float(float)> calibrate_{[](float c) { return std::max(0.f, c); }};
float band_lo_{0.90f};
float band_hi_{0.95f};
std::size_t rejected_{0}; ///< admissions refused by the band
bool enabled_;
int buffer_size_;
float band_lo_; ///< AR-018, from cfg.expand_band_lo
float band_hi_; ///< AR-018, from cfg.expand_band_hi
float novelty_sim_;
float spread_max_;
int min_anchor_frames_;
std::string debug_dir_;
std::map<int, TrackState> tracks_;
/// TRACES: AR-026 | SR-001
/// Contiguous annex matrix and its parallel actor index. `drained_` marks
/// how much of it the similarity engine already holds.
static constexpr int kEmbDim = 512;
std::vector<float> annex_emb_; ///< annex_size() × 512, row-major
std::vector<int> annex_actor_; ///< actor index per annex row
int drained_{0};
std::vector<AnnexEntry> annex_;
};
src/inference/similarity.hpp
+2 -23
View File
@@ -15,15 +15,6 @@
// by CMake (SAE_GEMM_BACKEND); backends/gemm_backend.cpp provides
// make_similarity_engine(). The core matcher node sees only this interface and
// holds no CUDA/HIP/BLAS headers.
//
// TRACES: AR-026 | SR-001
// The resident matrix GROWS. Per-film expansion (AR-018/AR-019) promotes new
// reference views mid-film, and those have to be scored by the same multiply as
// the baked references rather than by a side loop — "this set is small" is not
// an exception, because the annex grows with cast size and film length. Rows are
// therefore appended to the resident matrix and the next compute() covers baked
// and promoted references alike, in one GEMM. The deferred pass (AR-020) then
// inherits a single contiguous operand to score the TBI queue against.
struct ISimilarityEngine {
virtual ~ISimilarityEngine() = default;
@@ -31,23 +22,11 @@ struct ISimilarityEngine {
// Largest n_faces accepted by compute() per call (bounds GPU buffer sizes).
virtual int max_faces() const = 0;
// Rows currently resident: the baked gallery plus every appended promotion.
// This is compute()'s column stride, and it changes as rows are appended —
// read it per call rather than caching it across frames.
virtual int n_gallery() const = 0;
/// TRACES: AR-026 | SR-001
/// Append n_rows unit-norm embeddings (row-major, 512 floats each) to the
/// resident matrix. Amortised O(1) per row: capacity grows geometrically, so
/// a promotion does not re-upload the gallery. Invalidates any pointer
/// previously returned by compute().
virtual void append_rows(const float* rows_row_major, int n_rows) = 0;
// Compute similarities for n_faces query embeddings.
// query_row_major: n_faces × 512, row fi at query + fi*512.
// Returns a pointer to host memory holding S column-major: the gallery
// similarities for face fi start at result + fi*n_gallery(). The pointer is
// owned by the engine and valid until the next compute() or append_rows().
// similarities for face fi start at result + fi*n_gallery. The pointer is
// owned by the engine and valid until the next compute() call.
virtual const float* compute(const float* query_row_major, int n_faces) = 0;
};
src/kpn_bindings.cpp
+62 -273
View File
@@ -1,38 +1,11 @@
// sae_kpn — run the real downstream pipeline inside a Python-assembled KPN
// network, fed by a Python HDF5 replay source. Lets a parameter sweep re-run the
// exact C++ tracking/matching/presence logic over dumped embeddings — no video
// decode, no GPU — with different Config knobs each run.
//
/// TRACES: VR-011, VR-002 | PR-002
//
// **The whole chain is C++, including the sink.** That is the VR-011 change and
// it is the point of the requirement: replay must drive the real nodes, not a
// reimplementation. Two things were wrong before.
//
// 1. It did not compile. `add_face_tracker` built `FaceTrackerFunc` from a
// Config alone, and the tracker has required a TrackRegistry and a
// calibration since AR-007/AR-008 moved association into probability
// space. Any .so in a stale build/ predates that.
//
// 2. Presence was rebuilt in Python. `replay.py::build_minimal` merged
// per-frame detections into windows by annealing gaps — which is what the
// pipeline did before AR-012. The sink now builds a window from a
// TrackRegistry claim: the extent of a track an actor owned, starting when
// they appeared rather than when recognition first succeeded. Those answer
// different questions, so every sweep was tuning against a contract the
// shipped code had stopped honouring.
//
// Both had the same root cause, which is why this is one binding and not three.
// The chain has a construction ORDER — the matcher fits the calibration, the
// registry needs a discounter built from it, the tracker needs both, and the
// sink needs the registry's claims — and a factory-per-node API cannot express
// it. `add_pipeline` mirrors main.cpp exactly and is the only way to build the
// chain, so the ordering cannot be got wrong again from Python.
// sae_kpn — run the real downstream pipeline nodes (face_tracker, identity_matcher,
// scene_tracker) inside a Python-assembled KPN network, fed by a Python HDF5 replay
// source. Lets a parameter sweep re-run the exact C++ matching/tracking logic over
// dumped embeddings — no video decode, no GPU — with different Config knobs each run.
//
// Boundary types (cross the Python seam):
// EmbeddedSceneFrame IN (built by the Python replay source from HDF5 arrays)
// SceneAnnotation OUT (optional tee for per-frame debug rendering only —
// the presence output is written by the C++ sink)
// SceneAnnotation OUT (read by the Python sink → presence JSON)
// Intermediate types (TrackedSceneFrame, MatchedSceneFrame) flow C++→C++ only, but
// still need channel factories + converters registered so PyNetwork can wire them.
@@ -46,10 +19,7 @@
#include "gallery/gallery_store.hpp"
#include "nodes/face_tracker_node.hpp"
#include "nodes/identity_matcher_node.hpp"
#include "nodes/frame_annotation_node.hpp"
#include "nodes/result_sink_node.hpp"
#include "track_registry.hpp"
#include "evidence_discount.hpp"
#include "nodes/scene_tracker_node.hpp"
#include <nanobind/nanobind.h>
#include <nanobind/ndarray.h>
@@ -57,61 +27,16 @@
#include <nanobind/stl/vector.h>
#include <nanobind/stl/map.h>
#include <atomic>
#include <map>
#include <memory>
#include <optional>
#include <variant>
namespace nb = nanobind;
using namespace nb::literals;
// ── ReplaySession ─────────────────────────────────────────────────────────────
/// TRACES: VR-011 | PR-002
/// State the network's nodes reference but do not own.
///
/// ResultSinkFunc holds `std::atomic<bool>&`, exactly as it does under main(),
/// where it is a stack local in a function that outlives the pipeline. There is
/// no such frame here -- the network is built and torn down from Python -- so
/// the flag lives in a session held for the network's lifetime and released
/// explicitly. The registry is here for the same reason: the sink's claim
/// callback captures it.
struct ReplaySession {
/// Owns the Config, and must. ResultSinkFunc holds `const Config&` -- under
/// main() that is a stack local in a frame which outlives the pipeline, so
/// the reference is fine there. There is no such frame here: the network is
/// built inside a binding call and torn down from Python, so a Config local
/// to add_pipeline dies the moment it returns and the sink is left reading
/// freed memory. It presented as an empty output_path -- the sink announced
/// `[result_sink] writing ` and wrote nothing.
Config cfg;
std::atomic<bool> done{false};
std::shared_ptr<TrackRegistry> registry;
};
// Function-local static so ordering against other translation units cannot bite.
inline std::map<void*, std::shared_ptr<ReplaySession>>& sessions() {
static std::map<void*, std::shared_ptr<ReplaySession>> s;
return s;
}
// The variant spanning every type that flows on a channel in the replay chain.
using SaeVariant = std::variant<EmbeddedSceneFrame, TrackedSceneFrame,
MatchedSceneFrame, SceneAnnotation>;
// ── Node wrapper aliases ──────────────────────────────────────────────────────
// Named once so add_pipeline and the runtime setters cannot disagree about a
// node's port names: a mismatch there is a dynamic_cast that returns null, i.e.
// a runtime setter that silently does nothing.
using MatcherWrap = kpn::ObjectVariantNodeWrapper<
IdentityMatcherFunc, SaeVariant, kpn::in<"tracked">, kpn::out<"matched">>;
using TrackerWrap = kpn::ObjectVariantNodeWrapper<
FaceTrackerFunc, SaeVariant, kpn::in<"embedded">, kpn::out<"tracked">>;
using AnnotWrap = kpn::ObjectVariantNodeWrapper<
FrameAnnotationFunc, SaeVariant, kpn::in<"matched">, kpn::out<"annotation">>;
using SinkWrap = kpn::ObjectVariantNodeWrapper<
ResultSinkFunc, SaeVariant, kpn::in<"annotation">, kpn::out<>>;
// ── Converters ─────────────────────────────────────────────────────────────────
// Only EmbeddedSceneFrame (in) and SceneAnnotation (out) actually cross the seam;
// the two intermediates get identity-ish stubs (never converted in practice) so the
@@ -136,8 +61,6 @@ template<> struct PythonConverter<EmbeddedSceneFrame> {
ef.source.frame_idx = d.contains("frame_idx") ? nb::cast<int64_t>(d["frame_idx"]) : -1;
ef.source.eof = d.contains("eof") ? nb::cast<bool>(d["eof"]) : false;
ef.source.is_cut = d.contains("is_cut") ? nb::cast<bool>(d["is_cut"]) : false;
ef.source.is_scene_boundary = d.contains("is_scene_boundary")
? nb::cast<bool>(d["is_scene_boundary"]) : false;
if (ef.source.eof) return ef;
// faces: (N,4) bbox, (N,10) landmarks, (N,) confidence, (N,512) embeddings
@@ -146,22 +69,6 @@ template<> struct PythonConverter<EmbeddedSceneFrame> {
auto conf = nb::cast<nb::ndarray<float, nb::shape<-1>, nb::c_contig>>(d["confidence"]);
auto emb = nb::cast<nb::ndarray<float, nb::shape<-1, 512>, nb::c_contig>>(d["embeddings"]);
// AR-028 quality vector. Optional because a v1 dump predates it — absent
// leaves the DetectedFace sentinels at -1, which reads as *unscored*, not
// as a bad face. There is no live aligner on this path to recompute it:
// the replay starts at the embedded-frame channel, so what the dump does
// not carry is genuinely gone.
//
// Held in named locals, like the four above, because the ndarray owns the
// reference that keeps the buffer alive — reading .data() off a temporary
// would leave the pointer dangling at the end of the statement.
using FloatCol = nb::ndarray<float, nb::shape<-1>, nb::c_contig>;
std::optional<FloatCol> sharp_col, resid_col;
if (d.contains("sharpness")) sharp_col = nb::cast<FloatCol>(d["sharpness"]);
if (d.contains("alignment_residual")) resid_col = nb::cast<FloatCol>(d["alignment_residual"]);
const float* sp = sharp_col ? sharp_col->data() : nullptr;
const float* rp = resid_col ? resid_col->data() : nullptr;
const size_t n = bbox.shape(0);
ef.faces.reserve(n);
ef.embeddings.reserve(n);
@@ -175,8 +82,6 @@ template<> struct PythonConverter<EmbeddedSceneFrame> {
for (int k = 0; k < 5; ++k)
f.landmarks[k] = cv::Point2f(lp[i*10 + k*2], lp[i*10 + k*2 + 1]);
f.confidence = cp[i];
if (sp) f.sharpness = sp[i];
if (rp) f.alignment_residual = rp[i];
ef.faces.push_back(f);
Embedding e;
@@ -244,67 +149,31 @@ static Config config_from_dict(nb::dict d) {
// identity matcher
getf("match_prior", cfg.match_prior);
getf("prob_threshold", cfg.prob_threshold);
getf("match_threshold", cfg.match_threshold);
getf("match_ratio", cfg.match_ratio);
getf("match_ratio_ceil", cfg.match_ratio_ceil);
// face tracker
getf("track_alpha", cfg.track_alpha);
getf("track_min_iou", cfg.track_min_iou);
getf("track_assoc_min_prob", cfg.track_assoc_min_prob);
getd("track_extinction_sec", cfg.track_extinction_sec);
// AR-025: swept knobs, previously unreachable from any config.
getf("ownership_logodds", cfg.ownership_logodds);
getf("evidence_rho_max", cfg.evidence_rho_max);
getf("evidence_admit_below", cfg.evidence_admit_below);
geti("evidence_max_views", cfg.evidence_max_views);
getf("track_max_embed_dist", cfg.track_max_embed_dist);
geti("track_max_frames_missing", cfg.track_max_frames_missing);
getf("cut_revive_sim", cfg.cut_revive_sim);
geti("cut_inactive_max_frames", cfg.cut_inactive_max_frames);
// scene tracker
getd("extinction_sec", cfg.extinction_sec);
getd("anneal_sec", cfg.anneal_sec);
// gallery expansion (usually off for sweeps; expose so it can be toggled)
if (d.contains("expand_gallery")) cfg.expand_gallery = nb::cast<bool>(d["expand_gallery"]);
// AR-018: banded admission bounds for the per-film annex, in probability
// space. Reachable from a sweep — the config comment asks for both to be
// swept, and they are ignored unless expand_gallery is on. See track_gallery.hpp.
getf("expand_band_lo", cfg.expand_band_lo);
getf("expand_band_hi", cfg.expand_band_hi);
// Presence derivation. Accepts a string ("flood"/"track_extent") or a
// number (DE only produces floats: >=0.5 → flood) so the sweep can toggle
// it as a sixth knob. flood snaps to boundaries in the replayed frames
// (is_scene_boundary if present, else is_cut).
if (d.contains("presence_mode")) {
const auto& pm = d["presence_mode"];
bool flood = false;
if (nb::isinstance<nb::str>(pm)) flood = (nb::cast<std::string>(pm) == "flood");
else flood = (nb::cast<double>(pm) >= 0.5);
cfg.presence_mode = flood ? PresenceMode::flood : PresenceMode::track_extent;
}
/// TRACES: GR-004 | SR-001
if (d.contains("require_gallery_stamp"))
cfg.require_gallery_stamp = nb::cast<bool>(d["require_gallery_stamp"]);
/// TRACES: VR-011 | IR-001 | PR-002 | SR-003
// The sink is a real node in this network now, so it needs the two things
// that decide what it writes and where. Both used to be irrelevant here
// because the replay never had a sink -- Python rebuilt presence instead,
// which is the reimplementation VR-002 forbids and VR-011 removes.
if (d.contains("output_path"))
cfg.output_path = nb::cast<std::string>(d["output_path"]);
if (d.contains("verbosity")) {
const int v = nb::cast<int>(d["verbosity"]);
cfg.verbosity = v == 2 ? Verbosity::xray
: v == 1 ? Verbosity::standard
: Verbosity::minimal;
}
// Reported verbatim in the truth file's extraction block, so a replayed
// manifest says which gallery scope produced it (IR-002).
if (d.contains("gallery_scope"))
cfg.gallery_scope = nb::cast<std::string>(d["gallery_scope"]);
if (d.contains("sample_fps"))
cfg.sample_fps = nb::cast<float>(d["sample_fps"]);
if (d.contains("movie_path"))
cfg.movie_path = nb::cast<std::string>(d["movie_path"]);
return cfg;
}
using Net = kpn::python::PyNetwork<SaeVariant>;
NB_MODULE(sae_kpn, m) {
m.doc() = "Real KPN downstream nodes (tracker/matcher/frame_annotation) for Python replay sweeps";
m.doc() = "Real KPN downstream nodes (tracker/matcher/scene_tracker) for Python replay sweeps";
kpn::python::register_py_network<SaeVariant>(m, "Network");
@@ -337,51 +206,38 @@ NB_MODULE(sae_kpn, m) {
std::move(outs), cap);
}, "net"_a, "name"_a, "callable"_a, "inputs"_a, "outputs"_a, "capacity"_a = 5);
// ── The pipeline ────────────────────────────────────────────────────────────
/// TRACES: VR-011, VR-002 | DP-001 | PR-002, PR-004
///
/// One call builds the whole downstream chain, in the one order that works:
///
/// matcher (fits the calibration)
/// -> registry (needs a discounter built from it)
/// -> tracker (needs both)
/// -> frame_annotation
/// -> result_sink (needs the registry's claims)
///
/// This replaces add_face_tracker / add_identity_matcher / add_frame_annotation.
/// They were separate because the network is assembled node by node from
/// Python -- and that is exactly how the seam broke: the tracker's dependency
/// on a calibration that only exists once the matcher is built cannot be
/// expressed as three independent factories, so the tracker factory kept
/// constructing FaceTrackerFunc{cfg} against a signature that no longer
/// existed. A binding that cannot represent the order will eventually be
/// called in the wrong one.
///
/// DP-001 -- "modes are front-ends and must not fork pipeline logic" -- is
/// the requirement this serves. The replay harness is a front-end. Its job is
/// to supply frames and read the result, not to re-derive presence.
m.def("add_pipeline", [](Net& net, std::string gallery_path, nb::dict cfg_dict,
std::size_t cap, std::string embedder_model,
std::string embedder_sha256) {
// ── Real node factories ─────────────────────────────────────────────────────
m.def("add_face_tracker", [](Net& net, std::string name, nb::dict cfg_dict, std::size_t cap) {
Config cfg = config_from_dict(cfg_dict);
cfg.gallery_path = gallery_path; // so a refreshed calibration persists back
auto node = std::make_shared<kpn::ObjectVariantNodeWrapper<
FaceTrackerFunc, SaeVariant, kpn::in<"embedded">, kpn::out<"tracked">>>(cap, cfg);
net.add(std::move(name), std::move(node));
}, "net"_a, "name"_a, "config"_a, "capacity"_a = 16);
/// TRACES: GR-004 | SR-001
// embedder_model / embedder_sha256 identify whatever produced the embeddings
// that will be fed in. In a replay those come from the dump's own stamp (see
// scripts/optimizer/SCHEMA.md), because there is no live embedder in the
// network — the dump *is* the embedder as far as this gallery is concerned.
// Passing neither leaves the binding unverifiable, which warns loudly and is
// fatal under SAE_REQUIRE_GALLERY_STAMP.
m.def("add_identity_matcher", [](Net& net, std::string name, std::string gallery_path,
nb::dict cfg_dict, std::size_t cap,
std::string embedder_model,
std::string embedder_sha256) {
Config cfg = config_from_dict(cfg_dict);
cfg.gallery_path = gallery_path; // needed to persist refreshed calibration back
// Cache loaded galleries by path so a threshold sweep (many networks, same
// gallery) pays the parse once. The matcher holds a const ref; the cache
// keeps the gallery alive for the process lifetime.
// gallery) pays the ~24s JSON parse only once. The matcher holds a const
// ref; the cache keeps the gallery alive for the process lifetime.
static std::map<std::string, std::shared_ptr<ActorGallery>> cache;
auto it = cache.find(gallery_path);
if (it == cache.end())
it = cache.emplace(gallery_path,
std::make_shared<ActorGallery>(load_gallery(gallery_path))).first;
/// TRACES: GR-004 | SR-001
// embedder_model / embedder_sha256 identify whatever produced the
// embeddings that will be fed in. In a replay those come from the dump's
// own stamp: there is no live embedder here, so the dump *is* the
// embedder as far as this gallery is concerned. Checked on every
// construction, not only on a cache miss -- one process may replay
// several dumps against one cached gallery.
// Checked on every construction, not only on the cache miss: the same
// process may replay several dumps against one cached gallery.
EmbedderStamp feeding;
feeding.model_name = std::move(embedder_model);
feeding.model_sha256 = std::move(embedder_sha256);
@@ -391,104 +247,37 @@ NB_MODULE(sae_kpn, m) {
: feeding.model_name,
cfg.require_gallery_stamp);
// 1. Matcher first: its constructor fits (or loads) the calibration.
auto matcher = std::make_shared<MatcherWrap>(cap, *it->second, cfg);
// 2. The calibration every other stage must decide in (AR-024).
auto same_person = same_person_probability(matcher->functor().calibration());
// 3. Registry + discounter, from Config (AR-025).
TrackRegistry::Config reg_cfg;
reg_cfg.track_extinction_sec = cfg.track_extinction_sec;
reg_cfg.ownership_logodds = cfg.ownership_logodds;
EvidenceDiscounter::Config disc_cfg;
disc_cfg.max_views = cfg.evidence_max_views;
disc_cfg.admit_below = cfg.evidence_admit_below;
disc_cfg.rho_max = cfg.evidence_rho_max;
auto registry = std::make_shared<TrackRegistry>(
reg_cfg, EvidenceDiscounter(same_person, disc_cfg));
matcher->functor().set_registry(registry);
// 4. Tracker, which needs both.
auto tracker = std::make_shared<TrackerWrap>(cap, cfg, registry, same_person);
// 5. Projection, stateless.
auto annot = std::make_shared<AnnotWrap>(cap);
// 6. The real sink. `done` outlives the network via the session below;
// ResultSinkFunc holds it by reference, as it does in main.cpp.
auto session = std::make_shared<ReplaySession>();
session->cfg = cfg; // the sink holds this by reference
session->registry = registry;
auto sink = std::make_shared<SinkWrap>(cap, session->cfg, session->done);
/// TRACES: AR-012, AR-016 | IR-003 | SR-002
// The claim path, identical to main.cpp's. Without the flush hook every
// track still live at EOF is silently dropped -- which in a replay is
// most of the closing scene, and reads as a recognition miss rather than
// as a missing wire.
ResultSinkFunc& sink_fn = sink->functor();
registry->on_track_dead([&sink_fn](const DeadTrack& d) { sink_fn.add_claim(d); });
sink_fn.set_pre_write_hook([registry](double last_ts) { registry->flush(last_ts); });
net.add("tracker", tracker);
net.add("matcher", matcher);
net.add("annotation", annot);
net.add("sink", sink);
// Keyed by network so release_pipeline can free it. Not a leak-by-design:
// a sweep builds one network per replay, and the sink accumulates every
// annotation, so holding these forever would grow with films x configs.
sessions()[&net] = session;
}, "net"_a, "gallery"_a, "config"_a, "capacity"_a = 16,
auto node = std::make_shared<kpn::ObjectVariantNodeWrapper<
IdentityMatcherFunc, SaeVariant, kpn::in<"tracked">, kpn::out<"matched">>>(
cap, *it->second, cfg);
net.add(std::move(name), std::move(node));
}, "net"_a, "name"_a, "gallery"_a, "config"_a, "capacity"_a = 16,
"embedder_model"_a = "", "embedder_sha256"_a = "");
/// Drop the session for a network. Idempotent. Call after net.stop(); not
/// calling it holds one registry and one sink's accumulated frames per
/// replay, which a long sweep will notice.
m.def("release_pipeline", [](Net& net) { sessions().erase(&net); }, "net"_a);
/// TRACES: VR-011 | AR-025 | PR-002
/// The registry's own count of how often it was wrong, exposed so a replay
/// can fail on it instead of returning a plausible-looking empty answer.
///
/// `dropped_votes` is the one that matters here and it earned its keep
/// immediately. A vote lands on a track the registry has already reaped when
/// the matcher lags the tracker by more than track_extinction_sec of film.
/// In scene_analyze that cannot happen -- channels are 16-64 deep, so
/// backpressure pins the two nodes within a few frames of each other. This
/// harness sized every channel to the whole film to avoid a PyNode overflow
/// drop, which removed the backpressure entirely: the tracker ran the film
/// to the end while the matcher was still in its first minute, every vote
/// arrived after its track was gone, no track was ever owned, and the run
/// produced zero presence windows while cheerfully reporting 1647 frames
/// with an identified face.
m.def("pipeline_diagnostics", [](Net& net) {
nb::dict d;
auto it = sessions().find(&net);
if (it == sessions().end() || !it->second->registry) return d;
const auto& r = *it->second->registry;
d["dropped_votes"] = r.dropped_votes();
d["belief_swaps"] = r.belief_swaps();
d["actor_conflicts"] = r.actor_conflicts();
d["live_tracks"] = static_cast<int>(r.live());
return d;
}, "net"_a);
/// True once the sink has written its output. The sink flushes on the EOF
/// annotation, so a caller that reads the file before this is racing it.
m.def("pipeline_done", [](Net& net) {
auto it = sessions().find(&net);
return it != sessions().end()
&& it->second->done.load(std::memory_order_acquire);
}, "net"_a);
m.def("add_scene_tracker", [](Net& net, std::string name, nb::dict cfg_dict, std::size_t cap) {
Config cfg = config_from_dict(cfg_dict);
auto node = std::make_shared<kpn::ObjectVariantNodeWrapper<
SceneTrackerFunc, SaeVariant, kpn::in<"matched">, kpn::out<"annotation">>>(cap, cfg);
net.add(std::move(name), std::move(node));
}, "net"_a, "name"_a, "config"_a, "capacity"_a = 16);
// ── Runtime setters (persistent-pipeline reuse across a threshold sweep) ─────
// Build the network once, then change thresholds between replays — no rebuild,
// no teardown (which is where the ROCm deadlock lives), no gallery reload.
using MatcherWrap = kpn::ObjectVariantNodeWrapper<
IdentityMatcherFunc, SaeVariant, kpn::in<"tracked">, kpn::out<"matched">>;
using SceneWrap = kpn::ObjectVariantNodeWrapper<
SceneTrackerFunc, SaeVariant, kpn::in<"matched">, kpn::out<"annotation">>;
m.def("set_prob_threshold", [](Net& net, std::string name, float t) {
auto* w = dynamic_cast<MatcherWrap*>(net.node_ptr(name));
if (!w) throw std::runtime_error("set_prob_threshold: '" + name + "' is not an identity_matcher");
w->functor().set_prob_threshold(t);
}, "net"_a, "name"_a, "value"_a);
m.def("set_extinction_sec", [](Net& net, std::string name, double s) {
auto* w = dynamic_cast<SceneWrap*>(net.node_ptr(name));
if (!w) throw std::runtime_error("set_extinction_sec: '" + name + "' is not a scene_tracker");
w->functor().set_extinction_sec(s);
}, "net"_a, "name"_a, "value"_a);
}
src/main.cpp
+32 -267
View File
@@ -8,11 +8,11 @@
//
// [frame_source] ──Frame──► [face_detector] ──SceneFrame──► [face_aligner]
// ──AlignedSceneFrame──► [embedder] ──EmbeddedSceneFrame──►
// [identity_matcher] ──MatchedSceneFrame──► [frame_annotation]
// [identity_matcher] ──MatchedSceneFrame──► [scene_tracker]
// ──SceneAnnotation──► [result_sink]
//
// Debug build (SAE_DEBUG=1):
// [identity_matcher] output fans out to both [frame_annotation] AND [debug_renderer].
// [identity_matcher] output fans out to both [scene_tracker] AND [debug_renderer].
// FanoutNode<MatchedSceneFrame, 2> is auto-inserted by make_network().
//
// Usage:
@@ -22,7 +22,8 @@
// --output <path> output JSON (default: annotations.json)
// --fps <N> sample rate in frames/sec (default: 1.0)
// --verbosity <0|1|2> 0=minimal, 1=standard, 2=jellyfin-xray (default: 0)
// --prob-threshold <f> posterior P(match) to accept (default: 0.754)
// --match-threshold <f> cosine dist threshold (default: 0.45)
// --extinction <f> actor extinction window in seconds (default: 5.0)
// --detector <path> override SCRFD detector model path
// --arcface <path> override ArcFace model path
// --scene-detect enable TransNetV2 shot-boundary detection (dense decode;
@@ -31,34 +32,21 @@
// --scene-detector-engine <path> pre-built TransNetV2 TRT engine (TRT backend)
// --scene-threshold <f> boundary sigmoid prob above this → cut (default: 0.60)
// --scene-stride <N> frames between TransNetV2 windows (default: 50, ≤100)
// --scene-decode-fps <f> dense decode rate in scene-detect mode (default: 0 =
// native, the only rate TransNetV2 is calibrated for;
// AR-011). Lowering it runs the model off-distribution.
// --scene-decode-fps <f> dense decode rate in scene-detect mode (default: 12;
// 0 = native fps). Lower = faster, coarser boundaries.
// --dense-scale <f> downscale decoded frames in scene-detect mode (0<f≤1,
// default 1=off). Speeds decode; keep ≥0.5 on 1080p.
// --max-faces <N> max faces kept per frame (default: 0 = uncapped)
// --ownership-logodds <f> belief needed to own a track (default: 2.0 ≈ P 0.88).
// Below it a track makes no presence claim at all.
// --evidence-rho-max <f> ceiling on correlation between two observations of
// one track (default: 0.5 = a repeated view is worth
// at most two independent ones). AR-025.
// --evidence-admit-below <p> P(same view) under this counts as a new look
// --evidence-max-views <N> distinct views remembered per track
// --max-faces <N> max faces kept per frame (default: 10)
// --expand-gallery enable per-film gallery expansion from track continuity
// --expand-buffer <N> per-track diversity buffer size (default: 20)
// --expand-band-lo <p> store admission floor, P(same person) (default: 0.90)
// --expand-band-hi <p> store admission ceiling, P(same person) (default: 0.95)
// --expand-novelty-sim <f> promote only views with best sim < f (default: 0.55)
// --expand-spread-max <f> reject track if buffer spread > f (default: 0.60)
// --expand-min-anchor <N> accepted frames before a track confirms (default: 3)
// --expand-debug-dir <p> dump promoted mugshots + embeddings here (SAE_DEBUG)
// --benchmark <path> write a per-node timing + bottleneck report (JSON) and
// print it at shutdown. Says where the run's time went
// and which node is pacing it. See src/benchmark.hpp.
// --benchmark-interval-ms <N> channel-occupancy sampling period (default: 100)
// (SAE_DEBUG only)
// --debug-dir <path> debug frames output dir (default: debug_frames)
// --crop-context <f> bbox expansion factor for context crops (default: 1.5)
#include "benchmark.hpp"
#include "config.hpp"
#include "types.hpp"
#include "gallery/embedder_stamp.hpp"
@@ -70,8 +58,7 @@
#include "nodes/embedder_node.hpp"
#include "nodes/face_tracker_node.hpp"
#include "nodes/identity_matcher_node.hpp"
#include "nodes/frame_annotation_node.hpp"
#include "inference/scene_detector.hpp" // ISceneDetector::kWindow, for the join-depth derivation
#include "nodes/scene_tracker_node.hpp"
#include "nodes/scene_detector_node.hpp"
#include "scene_boundaries.hpp"
#include "nodes/scene_boundary_annotator_node.hpp"
@@ -83,16 +70,9 @@
#include <kpn/kpn.hpp>
#include <opencv2/core/utility.hpp> // cv::setNumThreads (SAE_CV_THREADS)
#include <algorithm>
#include <atomic>
#include <chrono>
#include <cmath>
#include <csignal>
#include <cstdlib>
#include <cstring>
#include <fstream>
#include <iostream>
#include <map>
#include <mutex>
@@ -104,85 +84,17 @@
// ── CLI parsing ───────────────────────────────────────────────────────────────
/// TRACES: AR-010, AR-004 | SR-002
/// Depth of the dense branch's own input queue. Part of how far behind the
/// fanout head TransNetV2 can be, and therefore an input to the join depth.
static constexpr std::size_t kSceneInputDepth = 128;
/// TRACES: AR-010, AR-004 | SR-002
/// How far the sampled branch must trail the dense one, in seconds of film.
///
/// TransNetV2 needs kWindow (100) dense frames before it can score any of
/// them, and its input queue can hold kSceneInputDepth more, so in the worst
/// case it has scored only up to (kSceneInputDepth + kWindow) frames behind
/// whatever the fanout has just delivered. The face branch must be at least
/// that far behind, or `scene_annotate` asks about frames nobody has looked at
/// yet. Backpressure turns depth into lag: the fanout blocks on the slower
/// branch rather than dropping, so the dense branch simply runs ahead.
///
/// Divided by a *lower bound* on native frame rate, because a slower source
/// makes the same frame count span more film — 24 fps is the floor for the
/// material this runs on, so it is the conservative choice.
static constexpr double kMinNativeFps = 24.0;
static constexpr double kSceneJoinLagSec =
(kSceneInputDepth + ISceneDetector::kWindow) / kMinNativeFps; // ~9.5 s
/// Margin over that minimum, for jitter in TransNetV2's inference time.
static constexpr double kSceneJoinSafety = 2.0;
/// TRACES: AR-004 | SR-002
/// Slots the sampled branch needs to hold `kSceneJoinLagSec` of film.
///
/// This used to be a constant 256, which is the whole bug: the requirement is a
/// span of *film*, and the slots needed to hold it depend on `sample_fps`.
/// Pinned at 256 it was ~256 s of lag at 1 fps — 27x what the join needs — and
/// nothing recomputed it if `sample_fps` changed, so the one number the join's
/// correctness rests on drifted silently with an unrelated knob.
///
/// It is also the largest single memory item in the pipeline. Every message
/// embeds `Frame source`, so a slot on this branch holds a full decoded image:
/// 256 of them is ~1.5 GB at 1080p, against ~110 MB for the derived depth at
/// 1 fps. See AR-004 — capacity is counted in items, and only the byte figure
/// (now correct, see types.hpp) shows what a slot really costs.
static std::size_t scene_join_depth(float sample_fps) {
const double slots = kSceneJoinSafety * kSceneJoinLagSec * sample_fps;
// Floor of 16: below that the queue stops absorbing ordinary jitter and
// starts throttling the fanout, which would slow the dense branch it
// exists to let run ahead.
return std::max<std::size_t>(16, static_cast<std::size_t>(std::ceil(slots)));
}
/// TRACES: AR-004 | SR-002
/// The decimator's input, on the *full-rate* stream.
///
/// This was also kSceneJoinDepth, which put a 256-slot buffer of full-rate
/// frames in front of the decimator — and at 1 fps against 24 fps native, 23 of
/// every 24 of those frames exist only to be discarded a moment later. Holding
/// ~1.5 GB of decoded images for frames the very next node throws away is the
/// worst available use of the memory budget.
///
/// A filter is a pass-through, not a reservoir: the lag belongs *after*
/// decimation, where a slot buys `1/sample_fps` seconds of film instead of
/// `1/native_fps`. Sized only to keep the decimator fed.
static constexpr std::size_t kDecimatorInputDepth = 16;
/// How deeply the sampled branch is buffered behind the dense one. TransNetV2
/// needs kWindow (100) dense frames before it can score any of them, so the face
/// branch must lag by at least that much or it asks about frames nobody has
/// looked at yet. Backpressure turns depth into lag: the fanout blocks on the
/// slower branch rather than dropping, so the detector simply runs ahead.
static constexpr std::size_t kSceneJoinDepth = 256;
/// Set when the scene branch is built, so shutdown can report whether the join
/// actually worked.
static std::shared_ptr<SceneBoundaries> scene_stats;
/// TRACES: VR-015 | AR-004 | PR-004
/// Set by SIGUSR1, serviced by the wait loop. `kill -USR1 <pid>` on a running
/// or WEDGED run prints the benchmark table immediately — channel occupancy
/// names the stalled node (full input, empty output) without a debug build or a
/// debugger, which is the difference between diagnosing the AR-004 hang in
/// seconds and reproducing it under gdb.
///
/// The handler only stores a flag; all printing happens on the main thread,
/// since nothing in the report is async-signal-safe.
static std::atomic<bool> g_dump_request{false};
extern "C" void sae_on_dump_signal(int) {
g_dump_request.store(true, std::memory_order_relaxed);
}
static Config parse_args(int argc, char** argv) {
Config cfg;
cfg.detector_model = kDefaultDetectorModel;
@@ -201,14 +113,11 @@ static Config parse_args(int argc, char** argv) {
else if (arg("--gallery")) cfg.gallery_path = next();
else if (arg("--output")) cfg.output_path = next();
else if (arg("--dump-embeddings")) cfg.dump_embeddings_path = next();
else if (arg("--benchmark")) cfg.benchmark_path = next();
else if (arg("--benchmark-interval-ms")) cfg.benchmark_interval_ms = std::stoi(next());
else if (arg("--fps")) cfg.sample_fps = std::stof(next());
else if (arg("--max-decode-fps")) cfg.max_decode_fps = std::stof(next());
else if (arg("--start")) cfg.start_sec = std::stod(next());
else if (arg("--end")) cfg.end_sec = std::stod(next());
else if (arg("--cut-threshold")) cfg.cut_threshold = std::stof(next());
else if (arg("--presence-mode")) { std::string m = next(); cfg.presence_mode = (m == "flood") ? PresenceMode::flood : PresenceMode::track_extent; }
else if (arg("--scene-detect")) cfg.scene_detect = true;
else if (arg("--scene-detector")) cfg.scene_model = next();
else if (arg("--scene-detector-engine")) cfg.scene_engine = next();
@@ -219,6 +128,8 @@ static Config parse_args(int argc, char** argv) {
else if (arg("--verbosity")) { int v = std::stoi(next()); cfg.verbosity = v == 2 ? Verbosity::xray : v == 1 ? Verbosity::standard : Verbosity::minimal; }
else if (arg("--prior")) cfg.match_prior = std::stof(next());
else if (arg("--prob-threshold")) cfg.prob_threshold = std::stof(next());
else if (arg("--match-threshold")) cfg.match_threshold = std::stof(next());
else if (arg("--extinction")) cfg.extinction_sec = std::stod(next());
else if (arg("--detector")) cfg.detector_model = next();
else if (arg("--detector-engine")) cfg.detector_engine = next();
else if (arg("--arcface")) cfg.arcface_model = next();
@@ -227,18 +138,17 @@ static Config parse_args(int argc, char** argv) {
else if (arg("--conf")) cfg.detector_conf = std::stof(next());
else if (arg("--max-faces")) cfg.max_faces = std::stoi(next());
else if (arg("--min-face-px")) cfg.min_face_px = std::stof(next());
else if (arg("--ratio")) cfg.match_ratio = std::stof(next());
else if (arg("--ratio-ceil")) cfg.match_ratio_ceil = std::stof(next());
else if (arg("--track-alpha")) cfg.track_alpha = std::stof(next());
else if (arg("--track-min-iou")) cfg.track_min_iou = std::stof(next());
else if (arg("--track-min-prob")) cfg.track_assoc_min_prob = std::stof(next());
else if (arg("--track-extinction")) cfg.track_extinction_sec = std::stod(next());
else if (arg("--ownership-logodds")) cfg.ownership_logodds = std::stof(next());
else if (arg("--evidence-rho-max")) cfg.evidence_rho_max = std::stof(next());
else if (arg("--evidence-admit-below")) cfg.evidence_admit_below = std::stof(next());
else if (arg("--evidence-max-views")) cfg.evidence_max_views = std::stoi(next());
else if (arg("--anneal")) cfg.anneal_sec = std::stod(next());
else if (arg("--expand-gallery")) cfg.expand_gallery = true;
else if (arg("--expand-buffer")) cfg.expand_buffer_size = std::stoi(next());
else if (arg("--expand-band-lo")) cfg.expand_band_lo = std::stof(next());
else if (arg("--expand-band-hi")) cfg.expand_band_hi = std::stof(next());
else if (arg("--expand-novelty-sim")) cfg.expand_novelty_sim = std::stof(next());
else if (arg("--expand-spread-max")) cfg.expand_track_spread_max = std::stof(next());
else if (arg("--expand-min-anchor")) cfg.expand_min_anchor_frames = std::stoi(next());
else if (arg("--expand-debug-dir")) cfg.expand_debug_dir = next();
else if (arg("--trt-cache")) cfg.trt.cache_dir = next();
@@ -263,21 +173,6 @@ static Config parse_args(int argc, char** argv) {
// ── Main ──────────────────────────────────────────────────────────────────────
int main(int argc, char** argv) {
/// TRACES: VR-015 | PR-004
// OpenCV here is built against TBB, so cv::parallel_for_ opens an arena of
// nproc-1 workers (19 on a 20-core box) *on top of* KPN's one thread per
// node. Two schedulers, neither aware of the other, on the same cores.
//
// SAE_CV_THREADS=1 hands concurrency entirely to KPN, which is where this
// pipeline's parallelism is supposed to come from. Worth measuring rather
// than assuming: TBB fan-out inside warpAffine is free speed when the
// pipeline is otherwise idle, so this can cut either way. Unset = default.
if (const char* t = std::getenv("SAE_CV_THREADS")) {
const int n = std::atoi(t);
cv::setNumThreads(n);
std::cerr << "[opencv] cv::setNumThreads(" << n << ")\n";
}
Config cfg;
try {
cfg = parse_args(argc, argv);
@@ -321,24 +216,15 @@ int main(int argc, char** argv) {
// and its final answer is only known when a track dies.
auto same_person = same_person_probability(matcher_fn.calibration());
TrackRegistry::Config reg_cfg;
reg_cfg.track_extinction_sec = cfg.track_extinction_sec;
reg_cfg.ownership_logodds = cfg.ownership_logodds;
/// TRACES: AR-025 | SR-002
// The discounter's parameters come from Config now. They used to be
// in-class defaults reached through the one-argument constructor, so the
// VR-007 sweep that rho_max's own comment defers to could not vary it.
EvidenceDiscounter::Config disc_cfg;
disc_cfg.max_views = cfg.evidence_max_views;
disc_cfg.admit_below = cfg.evidence_admit_below;
disc_cfg.rho_max = cfg.evidence_rho_max;
reg_cfg.extinction_sec = cfg.track_extinction_sec;
auto registry = std::make_shared<TrackRegistry>(
reg_cfg, EvidenceDiscounter(same_person, disc_cfg));
reg_cfg, EvidenceDiscounter(same_person));
matcher_fn.set_registry(registry);
FaceTrackerFunc ftracker_fn{cfg, registry, same_person};
FrameAnnotationFunc tracker_fn {};
SceneTrackerFunc tracker_fn {cfg};
ResultSinkFunc sink_fn {cfg, done};
/// TRACES: AR-012, AR-016 | IR-002, IR-003 | SR-002
@@ -349,15 +235,7 @@ int main(int argc, char** argv) {
// AR-016: a film ends with faces on screen and those tracks have not timed
// out. Without this flush the closing scene's cast is silently never
// emitted — a loss that reads as a recognition miss, not a bookkeeping bug.
/// TRACES: VR-015 | PR-004
// Last timestamp the pipeline reached, latched on the way out. It is what
// turns wall-clock seconds into the number that matters — seconds of film
// per second of run — and the sink is the only node that knows it.
std::atomic<double> film_sec{0.0};
sink_fn.set_pre_write_hook([registry, &film_sec](double last_ts) {
film_sec.store(last_ts, std::memory_order_release);
registry->flush(last_ts);
});
sink_fn.set_pre_write_hook([registry](double last_ts) { registry->flush(last_ts); });
#ifdef SAE_DEBUG
DebugRendererFunc debug_fn {cfg};
#endif
@@ -374,7 +252,7 @@ int main(int argc, char** argv) {
kpn::ObjectNode<EmbedderFunc, kpn::in<"aligned">, kpn::out<"embedded">, "embedder", 0> embedder (embedder_fn, 32);
kpn::ObjectNode<FaceTrackerFunc, kpn::in<"embedded">, kpn::out<"tracked">, "face_tracker", 0> ftracker (ftracker_fn, 16);
kpn::ObjectNode<IdentityMatcherFunc, kpn::in<"tracked">, kpn::out<"matched">, "identity_matcher", 0> matcher (matcher_fn, 16);
kpn::ObjectNode<FrameAnnotationFunc, kpn::in<"matched">, kpn::out<"annotation">, "frame_annotation", 0> tracker (tracker_fn, 16);
kpn::ObjectNode<SceneTrackerFunc, kpn::in<"matched">, kpn::out<"annotation">, "scene_tracker", 0> tracker (tracker_fn, 16);
kpn::ObjectNode<ResultSinkFunc, kpn::in<"annotation">,kpn::out<>, "result_sink", 0> sink (sink_fn, 16);
// ── Pipeline observability + run loop (topology-agnostic) ──────────────────
@@ -403,43 +281,8 @@ int main(int argc, char** argv) {
}
});
// Report *why* a node died. A Closed event alone says only that one
// stopped; the exception it carried is what identifies the fault, and
// without this listener it is discarded at the node boundary. Returning
// false keeps the existing semantics — the node still stops and the
// Closed handler above still aborts the run — but the run now names the
// cause instead of leaving it to be reconstructed from a debugger.
net.set_error_handler(
[&](std::string_view node_name, std::exception_ptr eptr) {
std::string what = "unknown exception";
try {
if (eptr) std::rethrow_exception(eptr);
} catch (const std::exception& e) {
what = e.what();
} catch (...) {
}
std::lock_guard<std::mutex> lk(event_mtx);
std::cerr << "[main] node '" << node_name << "' threw: " << what << "\n";
return false;
});
/// TRACES: VR-015 | PR-004
// Sampling must start with the network and stop before it is destroyed:
// channel fill is instantaneous, and by the time a run ends everything
// has drained, so a single read at shutdown reports an idle pipeline no
// matter how congested it was.
sae::bench::BenchmarkRecorder bench{cfg.benchmark_interval_ms};
const bool benchmarking = !cfg.benchmark_path.empty();
std::cerr << "[main] starting pipeline…\n";
net.start();
if (benchmarking) {
bench.start([&net] { return net.network_snapshot(); });
std::signal(SIGUSR1, sae_on_dump_signal);
std::cerr << "[benchmark] sampling every " << cfg.benchmark_interval_ms
<< "ms — `kill -USR1 " << getpid()
<< "` to dump the table now (works while hung)\n";
}
// Wait until BOTH terminal branches finish: result_sink (face pipeline)
// and, when enabled, scene_detector (the dense TransNetV2 branch, which
@@ -447,51 +290,12 @@ int main(int argc, char** argv) {
// pre-set true when scene detection is disabled.
while ((!done.load(std::memory_order_acquire) ||
!scene_done.load(std::memory_order_acquire)) &&
!node_crashed.load(std::memory_order_acquire)) {
!node_crashed.load(std::memory_order_acquire))
std::this_thread::sleep_for(std::chrono::milliseconds(100));
/// TRACES: VR-015 | AR-004 | PR-004
if (g_dump_request.exchange(false, std::memory_order_relaxed))
bench.dump_live(std::cerr, film_sec.load(std::memory_order_acquire));
}
// Latch the counters before stop(): they stay readable afterwards, but
// only while the network object is alive, and this keeps the numbers
// describing the run rather than the teardown.
if (benchmarking) bench.stop();
net.stop();
net.print_diagnostics();
/// TRACES: VR-015 | PR-004
if (benchmarking && bench.has_data()) {
const double film = film_sec.load(std::memory_order_acquire);
bench.print(std::cerr, film);
nlohmann::json run_cfg{
{"movie", cfg.movie_path},
{"gallery", cfg.gallery_path},
{"gallery_actors", gallery.actors.size()},
{"sample_fps", cfg.sample_fps},
{"min_face_px", cfg.min_face_px},
{"max_faces", cfg.max_faces},
{"embed_batch", cfg.embed_batch_size},
{"expand_gallery", cfg.expand_gallery},
{"scene_detect", cfg.scene_detect},
{"detector_engine", cfg.detector_engine},
{"arcface_engine", cfg.arcface_engine},
{"detector_model", cfg.detector_model},
{"arcface_model", cfg.arcface_model},
};
std::ofstream bf(cfg.benchmark_path);
if (bf) {
bf << bench.to_json(run_cfg, film).dump(2) << "\n";
std::cerr << "[benchmark] wrote " << cfg.benchmark_path << "\n";
} else {
std::cerr << "[benchmark] ERROR: could not write "
<< cfg.benchmark_path << "\n";
}
}
/// TRACES: AR-004 | SR-002
// A dropped frame does not degrade a result, it silently changes one —
// the output is a claim about footage that was never analysed, and
@@ -517,45 +321,6 @@ int main(int argc, char** argv) {
std::cerr << "\n";
}
/// TRACES: AR-025, AR-012 | SR-002
// How often the registry was asked about a track it had already reaped.
//
// A vote is dropped when the matcher lags the tracker by more than
// track_extinction_sec of FILM time. The two are adjacent nodes with a
// 16-deep channel between them, and the matcher is much the slower of
// the pair (a GEMM over the whole gallery against a Hungarian solve over
// a handful of boxes), so that channel runs full and the lag is close to
// its depth. In frames:
//
// lag_sec ~= channel_depth / sample_fps
//
// At the default sample_fps of 1.0 that is ~16 s against a 5 s window,
// so votes CAN be dropped here, and each one is identity evidence that
// never reached the track it belonged to -- presence under-reported, in
// a way that reads as a recognition miss.
//
// Reported rather than fatal, deliberately, and the distinction from the
// dropped-frame case below is real: a dropped frame means the output
// describes footage nobody analysed, which is always wrong. A dropped
// vote means one observation of a track went missing, which degrades a
// claim without falsifying it. There is also no measurement yet of how
// often it happens on real content -- so this prints the number that
// would justify a harder line rather than presuming it. See VR-017.
if (registry) {
const int dv = registry->dropped_votes();
if (dv > 0) {
std::cerr << "[registry] WARNING: " << dv << " identity vote(s) "
"arrived for already-reaped tracks. The matcher is "
"lagging the tracker by more than track_extinction_sec ("
<< cfg.track_extinction_sec << "s) of film; presence is "
"under-reported. Raise --track-extinction or reduce the "
"face_tracker/identity_matcher channel depth.\n";
}
std::cerr << "[registry] belief_swaps=" << registry->belief_swaps()
<< " actor_conflicts=" << registry->actor_conflicts()
<< " dropped_votes=" << dv << "\n";
}
bool dropped = false;
{
std::lock_guard<std::mutex> lk(event_mtx);
@@ -626,7 +391,7 @@ int main(int argc, char** argv) {
auto boundaries = std::make_shared<SceneBoundaries>();
scene_fn.set_boundaries(boundaries);
kpn::ObjectNode<SceneDetectorFunc, kpn::in<"dense">, kpn::out<>, "scene_detector", 0>
scene_node(scene_fn, kSceneInputDepth);
scene_node(scene_fn, 128);
// Decimator: keep frames on the sample_fps cadence, drop the rest.
// eof always passes so downstream shuts down cleanly. Stateful — one
@@ -641,7 +406,7 @@ int main(int argc, char** argv) {
return true;
}
return false;
}, kDecimatorInputDepth);
}, kSceneJoinDepth);
/// TRACES: AR-010 | SR-002
// Stamp is_scene_boundary from the detector's published verdict. tol is
@@ -656,7 +421,7 @@ int main(int argc, char** argv) {
// otherwise look exactly like "no boundary here".
SceneBoundaryAnnotatorFunc annotate_fn{boundaries, 0.5 / cfg.sample_fps};
kpn::ObjectNode<SceneBoundaryAnnotatorFunc, kpn::in<"frame">, kpn::out<"frame">,
"scene_annotate", 0> annotate(annotate_fn, scene_join_depth(cfg.sample_fps));
"scene_annotate", 0> annotate(annotate_fn, kSceneJoinDepth);
// Reported at shutdown: without this the join is unverifiable, and an
// annotator that never fired looks identical to footage with no
src/nodes/embedding_dump_node.hpp
+3 -26
View File
@@ -1,5 +1,5 @@
#pragma once
/// TRACES: AR-028 | VR-001, VR-010 | PR-002
/// TRACES: VR-001, VR-010 | PR-002
#include "types.hpp"
#include "config.hpp"
#include "gallery/embedder_stamp.hpp"
@@ -178,16 +178,6 @@ struct EmbeddingDumpFunc {
lmk_.push_back(f.landmarks[k].y);
}
conf_.push_back(f.confidence);
/// TRACES: AR-028 | SR-002
// The quality vector, carried rather than consumed: written beside
// the embedding it describes so VR-012 can locate its knees against
// recorded data instead of by re-running video. Size is the third
// axis and is already here as bbox + the bbox_upscale attribute.
// Both are -1 only if a face reached the dump unscored, which the
// aligner does not allow — the sentinel is preserved rather than
// clamped so that a future path which did would be visible.
sharp_.push_back(f.sharpness);
resid_.push_back(f.alignment_residual);
const auto& e = ef.embeddings[i];
emb_.insert(emb_.end(), e.begin(), e.end());
}
@@ -204,20 +194,11 @@ struct EmbeddingDumpFunc {
}
private:
// Root attributes are additive: schema_version stayed 1 across VR-010, because
// Root attributes are additive: schema_version stays 1 across VR-010, because
// every reader takes attributes by name with a default (replay.py) or an
// existence check (read_dump_provenance), so an old dump loses nothing and a
// new dump breaks nothing. A bump is for a change to the *datasets*.
//
// v2 is that change: AR-028 adds faces/sharpness and faces/alignment_residual.
// The bump is not about readers — those check for the datasets by name, and a
// v1 dump still replays. It is so a *consumer of the quality vector* can tell
// "this film's faces were never scored" from "this film's faces scored zero",
// which is the same distinction scene_detect exists to make and is likewise
// not recoverable from the arrays. A v1 dump reports the vector as unknown;
// re-dump to acquire it, since nobody can assert after the fact how sharp a
// face was.
static constexpr int kSchemaVersion = 2;
static constexpr int kSchemaVersion = 1;
static constexpr int kEmbedDim = 512;
static std::string basename_of(const std::string& path) {
@@ -293,9 +274,6 @@ private:
write_vec(faces, "bbox", bbox_, H5::PredType::NATIVE_FLOAT, 4);
write_vec(faces, "landmarks", lmk_, H5::PredType::NATIVE_FLOAT, 10);
write_vec(faces, "confidence", conf_, H5::PredType::NATIVE_FLOAT);
/// TRACES: AR-028 | SR-002
write_vec(faces, "sharpness", sharp_, H5::PredType::NATIVE_FLOAT);
write_vec(faces, "alignment_residual", resid_, H5::PredType::NATIVE_FLOAT);
std::cerr << "[embedding_dump] wrote " << ts_.size() << " frames, "
<< conf_.size() << " faces → " << path_ << "\n";
@@ -314,5 +292,4 @@ private:
std::vector<int64_t> face_off_;
std::vector<int32_t> face_cnt_;
std::vector<float> emb_, bbox_, lmk_, conf_;
std::vector<float> sharp_, resid_; // AR-028 quality vector, parallel to conf_
};
src/nodes/face_aligner_node.hpp
+5 -62
View File
@@ -1,55 +1,21 @@
#pragma once
#include "face_utils.hpp"
#include <cstdint>
#include <iostream>
// ── FaceAlignerFunc ───────────────────────────────────────────────────────────
/// TRACES: AR-005, AR-028, AR-029, AR-030 | SR-002
///
// KPN node: applies a 5-point similarity transform to each detected face,
// producing a 112×112 BGR crop suitable for ArcFace inference.
//
// Alignment is an Umeyama least-squares fit over all five landmarks (AR-005),
// not a robust one: a RANSAC fit discards the very landmarks AR-030 reads.
//
// This is also where the AR-028 quality vector is filled in, because this is
// where the inputs to it already exist:
//
// - **Visibility** (AR-030) is the fit's residual, and is genuinely free — the
// transform is computed for the warp regardless, and the residual is what
// that fit could not explain.
// - **Sharpness** (AR-029) is measured on the crop this node just produced,
// which is the only place it *can* be measured: the aligned canvas is what
// makes the number scale-normalised, and downstream of the embedder the crop
// is only forwarded for debug rendering. It is not free — 33 us per face
// single-threaded (cvtColor, one Laplacian, two meanStdDev over 112x112) —
// but it is two orders below the embedder inference it qualifies, and it
// runs per face rather than per frame, so a landscape shot costs nothing.
//
// Size, the third axis, is `bbox` and needs no work here.
//
// No face is admitted unscored: every face in the output carries both numbers,
// so a negative value downstream is a bug rather than a poor-quality face.
// Nothing is dropped or discounted on quality — that is AR-030's discount and
// VR-012's knee, both still open.
//
// Degenerate detections (where the fit fails) cannot be scored, since there is
// no crop and no residual to score, and are therefore dropped — but they are
// **counted**, not silently discarded. A nonzero tally means the detector is
// emitting landmark sets the aligner cannot use, which is a fact about the
// detector; losing it leaves a hole in the dump that looks like footage with
// no faces in it.
// Alignment uses cv::estimateAffinePartial2D (RANSAC) to fit the detected
// landmarks to ArcFace canonical positions. Degenerate detections (where the
// affine fit fails) are silently dropped from the output vectors.
struct FaceAlignerFunc {
static constexpr std::string_view label() { return "face_aligner"; }
AlignedSceneFrame operator()(SceneFrame sf) {
if (sf.source.eof) {
report();
return {std::move(sf.source), {}, {}};
}
if (sf.faces.empty())
if (sf.source.eof || sf.faces.empty())
return {std::move(sf.source), {}, {}};
std::vector<DetectedFace> good_faces;
@@ -63,38 +29,15 @@ struct FaceAlignerFunc {
float residual = -1.f;
cv::Mat crop = align_face(sf.source.image, face.landmarks, &residual);
if (crop.empty()) {
++degenerate_;
std::cerr << "[face_aligner] degenerate detection skipped\n";
continue;
}
face.alignment_residual = residual;
face.sharpness = crop_sharpness(crop);
good_faces.push_back(face);
crops.push_back(std::move(crop));
++scored_;
}
return {std::move(sf.source), std::move(good_faces), std::move(crops)};
}
/// Faces that carry a full quality vector, and faces the fit could not use.
uint64_t scored() const { return scored_; }
uint64_t degenerate() const { return degenerate_; }
private:
// Reported once at EOF rather than per occurrence: a run with a systematic
// landmark problem would otherwise emit one line per face for the length of
// a film, which is how the count came to be ignored.
void report() {
if (reported_) return;
reported_ = true;
if (degenerate_)
std::cerr << "[face_aligner] " << degenerate_ << " of "
<< (degenerate_ + scored_)
<< " detections had a degenerate landmark fit and were dropped"
" (no crop, so no embedding and no quality vector)\n";
}
uint64_t scored_{0};
uint64_t degenerate_{0};
bool reported_{false};
};
src/nodes/face_detector_node.hpp
+11 -28
View File
@@ -6,7 +6,6 @@
#include <algorithm>
#include <memory>
#include <string>
#include <vector>
// ── FaceDetectorFunc ──────────────────────────────────────────────────────────
// KPN node: runs SCRFD-500MF to detect ALL faces in a frame.
@@ -23,38 +22,22 @@ struct FaceDetectorFunc {
, min_face_px_(cfg.min_face_px)
{}
/// TRACES: AR-002 | SR-002
// Drop faces below the minimum size — too small for reliable ArcFace
// alignment, and below the resolution where identification still holds
// (VR-013 measured the knee end to end).
//
// The minimum is expressed in ORIGINAL video resolution, which is what makes
// it a property of the footage rather than of a throughput knob. When
// dense_scale downscaled the frame the detector's boxes are in downscaled
// space, and `bbox_upscale` is what maps them back; dividing the threshold by
// it rather than multiplying every box keeps the comparison on the detector's
// own numbers and the physical cutoff constant across scales.
//
// Strictly less-than: a face exactly at the minimum is admissible, which is
// what a "minimum of 40x40" means.
static void drop_undersized(std::vector<DetectedFace>& faces,
float min_face_px,
float bbox_upscale) {
const float min_px = (bbox_upscale > 0.f) ? min_face_px / bbox_upscale
: min_face_px;
faces.erase(
std::remove_if(faces.begin(), faces.end(), [&](const DetectedFace& d) {
return d.bbox.width < min_px || d.bbox.height < min_px;
}),
faces.end());
}
SceneFrame operator()(Frame f) {
if (f.eof) return {std::move(f), {}};
auto faces = detector_->detect(f.image);
drop_undersized(faces, min_face_px_, f.bbox_upscale);
// Drop faces below minimum pixel size (too small for reliable ArcFace
// alignment). Note: when dense_scale downscaled the frame, both the
// detection coords and min_face_px are in downscaled space — so scale
// the threshold down to match, keeping the physical size cutoff constant.
const float min_px = (f.bbox_upscale != 1.f)
? min_face_px_ / f.bbox_upscale : min_face_px_;
faces.erase(
std::remove_if(faces.begin(), faces.end(), [&](const DetectedFace& d) {
return d.bbox.width < min_px || d.bbox.height < min_px;
}),
faces.end());
// Sort largest-first so max_faces_ keeps the most informative detections
std::sort(faces.begin(), faces.end(),
src/nodes/frame_annotation_node.hpp
-48
View File
@@ -1,48 +0,0 @@
#pragma once
/// TRACES: AR-012, AR-013 | SR-002
///
/// FrameAnnotationFunc — project a matched frame into a per-frame annotation.
///
/// Stateless, and that is the entire point of it.
///
/// It replaces `SceneTrackerFunc`, which kept an extinction timer per actor and
/// reported an actor as visible for `extinction_sec` (57.4 s) after their last
/// detection. docs/SPEC.md specified that node's deletion -- "anneal_sec and
/// extinction_sec are deleted, not re-tuned ... SceneTrackerFunc goes with
/// them", with a removal list ending "grep for both names and expect no
/// survivors" -- and docs/requirements.md recorded both constants as Withdrawn,
/// deleted "rather than retained at zero", on the grounds that a field naming a
/// mechanism the pipeline no longer has is actively misleading. None of that
/// removal had happened. The node was still wired into both shipped pipelines
/// and still printed its timeout at every startup.
///
/// **Presence is not this node's business.** AR-012 moved it to TrackRegistry,
/// where a window is `[first_seen, last_seen]` of a track an actor owns, and
/// AR-013 ends that window at the last sighting rather than after it. A
/// keep-alive here answered the same question a second time and answered it
/// worse: it re-opened the trailing cool-down the registry exists to refuse.
///
/// What a consumer sees change: `--verbosity standard`'s `frames[].identified`
/// used to list every actor still inside the keep-alive, including ones absent
/// from the frame. It now lists what was actually matched in that frame. The
/// minimal and xray outputs are unaffected -- they were already built from
/// registry claims and never consulted this node.
#include "types.hpp"
#include <string_view>
#include <utility>
struct FrameAnnotationFunc {
static constexpr std::string_view label() { return "frame_annotation"; }
SceneAnnotation operator()(MatchedSceneFrame mf) {
if (mf.source.eof) return {0.0, {}, /*eof=*/true};
SceneAnnotation sa;
sa.timestamp_sec = mf.source.timestamp_sec;
sa.visible_actors = std::move(mf.actors);
sa.is_cut = mf.source.is_cut;
sa.is_scene_boundary = mf.source.is_scene_boundary;
return sa;
}
};
src/nodes/identity_matcher_node.hpp
+78 -148
View File
@@ -19,36 +19,19 @@
// KPN node: compares each embedding against every reference embedding in the
// actor gallery using cosine similarity.
//
// Matching strategy — one mode, always.
// Matching strategy — two modes selected at construction time:
//
// Gallery calibration fits a sigmoid P(match) = σ(a·similarity + b) from
// intra/inter-class pairs. A face is accepted if P(match | best_actor) >
// prob_threshold. Per-actor best similarity is the closest reference
// embedding (best-of-N).
// Calibrated (preferred): gallery calibration fits a sigmoid
// P(match) = σ(a·similarity + b) from intra/inter-class pairs.
// A face is accepted if P(match | best_actor) > prob_threshold.
//
/// TRACES: AR-024 | SR-002
// **There is no raw-cosine fallback.** There used to be: when the fit was
// invalid this node switched to a cosine-distance ceiling plus a ratio test
// (`match_threshold`, `match_ratio`, `match_ratio_ceil`). Three things were
// wrong with it, and the third is the one that mattered.
// Fallback (no calibration): dual-criterion accept —
// (a) best cosine distance < match_threshold, OR
// (b) ratio test: best_dist/second_best_dist < match_ratio
// AND best_dist < match_ratio_ceil.
//
// 1. It violated AR-024 outright, untagged — a bare cosine threshold means
// something different for every model, gallery and face size.
// 2. It disagreed with the rest of the pipeline about what "calibration
// failed" means. `same_person_probability` answers that question by
// falling back to the untuned default sigmoid and saying so loudly, so
// tracking and evidence weighting stayed in probability space while
// matching alone left it. One run, two policies.
// 3. Its accepted faces were still fed to `TrackRegistry::observe`, whose
// contract reads "posterior is a calibrated probability, never a raw
// cosine (AR-024) ... so the accumulation cannot be fed an uncalibrated
// number by a careless caller". It could. `max(0, cosine)` went straight
// into the log-odds accumulation as though it were a probability.
//
// An invalid fit now behaves exactly as everywhere else: the default sigmoid,
// with a warning that says the probabilities are not meaningful. That is a
// worse answer than a fitted calibration and a better one than a number whose
// units nothing else in the pipeline shares.
// In both modes, per-actor best similarity is determined by scanning
// reference embeddings and taking the closest (best-of-N).
//
// Gallery scan: the full reference set (tens of thousands of 512-dim
// embeddings) is uploaded to the GPU once at construction time and stays
@@ -56,13 +39,6 @@
// uploaded and a single SGEMM computes the full similarity matrix in well under
// a millisecond. The GPU math backend (cuBLAS or rocBLAS) lives behind
// ISimilarityEngine (backends/gemm_backend.cpp) and is selected at compile time.
//
// TRACES: AR-026 | SR-001
// That resident matrix grows during a film: per-film expansion (AR-019) promotes
// pose-varied views, and they are APPENDED to it rather than scored separately,
// so one multiply covers baked and promoted references alike and best-of-N is a
// single pass over one similarity column. There is no second similarity path in
// this node to fall out of step with the first.
struct IdentityMatcherFunc {
static constexpr std::string_view label() { return "identity_matcher"; }
@@ -74,6 +50,9 @@ struct IdentityMatcherFunc {
: gallery_(gallery)
, prob_threshold_(cfg.prob_threshold)
, log_prior_odds_(std::log(cfg.match_prior / (1.f - cfg.match_prior)))
, threshold_(cfg.match_threshold)
, ratio_(cfg.match_ratio)
, ratio_ceil_(cfg.match_ratio_ceil)
, track_gallery_(cfg)
{
std::cerr << "[identity_matcher] flattening gallery embeddings...\n";
@@ -106,21 +85,16 @@ struct IdentityMatcherFunc {
<< cfg.gallery_path << "\n";
}
/// TRACES: AR-024 | SR-002
// Same sentence either way, because it is the same decision rule; only
// the provenance of (a, b) differs. An unfitted sigmoid still returns
// plausible-looking probabilities, so the warning has to be the thing
// that distinguishes them — nothing downstream can.
std::cerr << "[identity_matcher] calibrated Bayesian matching"
<< " prior=" << cfg.match_prior
<< " P_threshold=" << prob_threshold_
<< " effective_sim_boundary="
<< cal_.boundary_at(prob_threshold_, log_prior_odds_) << "\n";
if (!cal_.valid) {
std::cerr << "[identity_matcher] WARNING: the calibration is NOT fitted "
"(a=" << cal_.a << ", b=" << cal_.b << ") — matching runs "
"on the untuned default sigmoid, so prob_threshold is not "
"comparable to a tuned run's.\n";
if (cal_.valid) {
std::cerr << "[identity_matcher] calibrated Bayesian matching"
<< " prior=" << cfg.match_prior
<< " P_threshold=" << prob_threshold_
<< " effective_sim_boundary="
<< cal_.boundary_at(prob_threshold_, log_prior_odds_) << "\n";
} else {
std::cerr << "[identity_matcher] threshold matching (calibration skipped)"
<< " threshold=" << threshold_
<< " ratio=" << ratio_ << " ratio_ceil=" << ratio_ceil_ << "\n";
}
std::cerr << "[identity_matcher] gallery: "
<< gallery_.actors.size() << " actors, "
@@ -152,12 +126,7 @@ struct IdentityMatcherFunc {
/// Where per-frame identity evidence reaches the registry. Optional: with no
/// registry attached the matcher behaves exactly as before, which keeps the
/// replay harness and the unit tests working unchanged.
void set_registry(std::shared_ptr<TrackRegistry> r) {
registry_ = std::move(r);
// This node is the evidence source, so the registry must not close a
// track until this node's watermark has passed it (AR-013).
if (registry_) registry_->expect_evidence();
}
void set_registry(std::shared_ptr<TrackRegistry> r) { registry_ = std::move(r); }
// Runtime setter — lets a persistent pipeline be reused across a threshold sweep
// without rebuilding the (expensive, gallery-resident) matcher. The gallery,
@@ -170,22 +139,6 @@ struct IdentityMatcherFunc {
return {std::move(tf.source), {}};
}
/// TRACES: AR-012, AR-013 | SR-002
// Publish the evidence watermark BEFORE voting on this frame: every
// observation strictly before it has now been folded in, so the registry
// may reap against it. Unconditional -- a frame with no faces still
// advances the watermark, or a long faceless stretch would stall reaping
// and hold every dormant track open to the end of the film.
//
// This is what makes presence independent of node speed. The registry
// used to reap on the TRACKER's clock, and backpressure (working as
// AR-004 intends) means the tracker can be a whole channel's depth ahead
// of this node -- so tracks were closed before their votes arrived, the
// votes were dropped, and the run silently under-reported. Measured on
// the SuperHero fixture before this change: channel depth 32 gave 5
// actors, depth 10322 gave 0, from identical input.
if (registry_) registry_->advance_evidence(tf.source.timestamp_sec);
// A hard cut changes the camera viewpoint. The face_tracker may revive a
// track_id across the cut (identity continuity), but promotion must never
// mix embeddings from two viewpoints under one buffer, so we still drop
@@ -230,51 +183,62 @@ struct IdentityMatcherFunc {
tf.embeddings[base + k].data(), 512 * sizeof(float));
}
/// TRACES: AR-026 | SR-001
// One GEMM now covers baked references AND the per-film annex: promoted
// rows were appended to the engine's resident matrix, so they are just
// more gallery rows with an entry in flat_actor_. The annex used to be
// folded in afterwards by a host-side cosine loop, justified by "tens of
// embeddings" — an assumption AR-018/AR-019 retired, since every owned
// track promotes and the annex grows with cast size and film length.
//
// n_gallery() is read per frame, not cached: it grows as promotions land.
const int n_gal = sim_engine_->n_gallery();
// S (N_gallery × chunk) col-major: face k's gallery sims at sims + k*n_gallery.
const float* host_sims = sim_engine_->compute(host_query.data(), chunk);
for (int ci = 0; ci < chunk; ++ci) {
const int fi = base + ci;
const float* sims = host_sims + static_cast<size_t>(ci) * n_gal;
const float* sims = host_sims + static_cast<size_t>(ci) * n_gallery_;
std::vector<float> best_sim(gallery_.actors.size(),
-std::numeric_limits<float>::max());
for (int ei = 0; ei < n_gal; ++ei) {
for (int ei = 0; ei < n_gallery_; ++ei) {
float sim = sims[ei];
int ai = flat_actor_[ei];
if (sim > best_sim[ai]) best_sim[ai] = sim;
}
// Only the best matters now. The runner-up was tracked solely for
// the retired ratio test, which asked whether the best cosine stood
// out from the second — a question the calibrated posterior does
// not need, since it already says how likely the best match is to
// be right rather than how much it beat its neighbour by.
int best_actor = -1;
float best_s = -std::numeric_limits<float>::max();
for (int ai = 0; ai < static_cast<int>(best_sim.size()); ++ai) {
if (best_sim[ai] > best_s) {
best_s = best_sim[ai];
best_actor = ai;
}
// Fold in the per-film annex (CPU-side, tens of embeddings). Promoted
// pose-varied views compete for best-of-N exactly like baked refs, so
// a face at a pose the gallery lacked can now win its true actor.
for (const auto& ae : track_gallery_.annex()) {
float sim = cosine_similarity(tf.embeddings[fi], ae.emb);
if (sim > best_sim[ae.actor_idx]) best_sim[ae.actor_idx] = sim;
}
/// TRACES: AR-024 | SR-002
// One rule, whatever the fit's provenance. The cosine reaches a
// comparison only through cal_.probability().
const float best_p = best_actor >= 0
? cal_.probability(best_s, log_prior_odds_)
: 0.f;
const bool accept = best_actor >= 0 && best_p > prob_threshold_;
int best_actor = -1;
int second_actor = -1;
float best_s = -std::numeric_limits<float>::max();
float second_s = -std::numeric_limits<float>::max();
for (int ai = 0; ai < static_cast<int>(best_sim.size()); ++ai) {
if (best_sim[ai] > best_s) {
second_s = best_s;
second_actor = best_actor;
best_s = best_sim[ai];
best_actor = ai;
} else if (best_sim[ai] > second_s) {
second_s = best_sim[ai];
second_actor = ai;
}
}
(void)second_actor;
bool accept = false;
if (best_actor >= 0) {
if (cal_.valid) {
accept = cal_.probability(best_s, log_prior_odds_) > prob_threshold_;
} else {
float best_d = 1.f - best_s;
float second_d = (second_s > -std::numeric_limits<float>::max())
? 1.f - second_s
: std::numeric_limits<float>::max();
bool absolute = best_d < threshold_;
bool ratio = (best_d < ratio_ceil_) &&
(second_d == std::numeric_limits<float>::max() ||
best_d / second_d < ratio_);
accept = absolute || ratio;
}
}
IdentifiedActor ia;
// Map bbox back to original video resolution when dense_scale
@@ -295,7 +259,9 @@ struct IdentityMatcherFunc {
ia.imdb_id = gallery_.actors[best_actor].imdb_id;
ia.tmdb_id = gallery_.actors[best_actor].tmdb_id;
ia.jellyfin_id = gallery_.actors[best_actor].jellyfin_id;
ia.similarity = best_p;
ia.similarity = cal_.valid
? cal_.probability(best_s, log_prior_odds_)
: best_s;
}
// Feed this face into per-film gallery expansion. best_actor/best_s
@@ -309,9 +275,12 @@ struct IdentityMatcherFunc {
// would make ownership depend on a per-frame threshold the redesign
// exists to stop relying on. The registry discounts for correlation
// and decides ownership from the accumulated posterior (AR-025).
if (registry_ && best_actor >= 0 && tf.track_ids[fi] >= 0)
registry_->observe(tf.track_ids[fi], best_actor, best_p,
tf.embeddings[fi]);
if (registry_ && best_actor >= 0 && tf.track_ids[fi] >= 0) {
const float p = cal_.valid
? cal_.probability(best_s, log_prior_odds_)
: std::max(0.f, best_s);
registry_->observe(tf.track_ids[fi], best_actor, p, tf.embeddings[fi]);
}
// TRACES: AR-019 | SR-005
// Ownership is the registry's, computed once. TrackGallery used to
@@ -330,60 +299,21 @@ struct IdentityMatcherFunc {
}
} // chunk loop
absorb_promotions();
/// TRACES: AR-019 | SR-005
// Drop buffers for tracks the registry has reaped. Without this a track
// that simply went off screen kept its diversity buffer until the next
// cut, so the store grew with the film rather than with what is on
// screen — and a buffer that outlives its track is evidence about a
// person nobody is looking at any more.
if (registry_)
track_gallery_.prune_dead(
[this](int id) { return registry_->is_live(id); });
return {std::move(tf.source), std::move(actors)};
}
private:
/// TRACES: AR-026 | SR-001
/// Move rows promoted during this frame into the resident gallery matrix,
/// extending the actor mapping in lockstep so row i keeps naming the actor
/// at flat_actor_[i]. Runs once per frame, after every face has been scored:
/// appending mid-frame would invalidate the similarity pointer the chunk
/// loop is still reading, and it is also the semantics the expansion store
/// documents — a promotion helps SUBSEQUENT frames, never the one that
/// produced it, so identification cannot depend on face order within a frame.
void absorb_promotions() {
if (!track_gallery_.enabled()) return;
pending_emb_.clear();
pending_actor_.clear();
const int n = track_gallery_.drain_promotions(pending_emb_, pending_actor_);
if (n == 0) return;
sim_engine_->append_rows(pending_emb_.data(), n);
flat_actor_.insert(flat_actor_.end(),
pending_actor_.begin(), pending_actor_.end());
n_gallery_ = sim_engine_->n_gallery();
}
ActorGallery gallery_;
GalleryCalibration cal_;
float prob_threshold_;
float log_prior_odds_;
/// flat_emb_ is the BAKED reference set only — it is the calibration fit's
/// input (AR-023) and is not touched again after construction. flat_actor_,
/// by contrast, is the actor mapping parallel to the *engine's* rows, so it
/// grows with every promotion absorbed (AR-026) and is the longer of the two.
float threshold_;
float ratio_;
float ratio_ceil_;
std::vector<Embedding> flat_emb_;
std::vector<int> flat_actor_;
int n_gallery_{0};
// Reused across frames so absorbing a promotion allocates nothing.
std::vector<float> pending_emb_;
std::vector<int> pending_actor_;
std::unique_ptr<ISimilarityEngine> sim_engine_;
TrackGallery track_gallery_;
std::shared_ptr<TrackRegistry> registry_;
src/nodes/result_sink_node.hpp
+5 -61
View File
@@ -23,8 +23,7 @@ using json = nlohmann::json;
//
// Verbosity::minimal — merges per-frame presence into contiguous time windows.
// Output: {
// "schema_version": 2, "movie": "...",
// "extraction": { "sample_fps": ..., "extinction_sec": ..., "gallery_scope": ... },
// "schema_version": 1, "movie": "...", "sample_fps": ..., "anneal_sec": ...,
// "actors": [{ "name", "imdb_id", "tmdb_id", "jellyfin_id", "scenes": [[t0,t1], ...] }]
// }
// An optional top-level "jellyfin_item_id" (the analysed title's Jellyfin item
@@ -124,9 +123,8 @@ private:
/// schema_version 2, per jRay/SPEC.md JR-002. anneal_sec is REMOVED
/// rather than zeroed: a field naming a mechanism the pipeline no
/// longer has is actively misleading, and would outlive everyone who
/// remembers why it reads 0. The extraction block reports
/// track_extinction_sec, which bounds re-association -- not the
/// withdrawn actor keep-alive that shared its name.
/// remembers why it reads 0. extinction_sec succeeds it as the
/// parameter that actually shapes window extent.
root["schema_version"] = kSchemaVersion;
root["movie"] = cfg_.movie_path;
root["extraction"] = {
@@ -152,7 +150,6 @@ private:
double start{0.0};
double end{0.0};
float belief{0.f}; ///< the posterior that justified the claim (AR-017)
Route route{Route::live}; ///< how it was identified (AR-017)
};
struct ActorWindow {
std::string name, imdb_id, tmdb_id, jellyfin_id;
@@ -181,21 +178,7 @@ private:
aw.jellyfin_id = it->second.jellyfin_id;
}
}
aw.scenes.push_back({c.first_seen, c.last_seen, c.belief, c.route});
}
// Flood-fill: snap each claim to the shot it sits in, so an actor seen
// once in a scene is reported across the whole scene. Bounded by real
// TransNetV2 boundaries — a window never crosses one — and a no-op when
// scene detection found no boundaries (nothing to snap to).
if (cfg_.presence_mode == PresenceMode::flood) {
const std::vector<double> bounds = scene_boundaries();
if (!bounds.empty())
for (auto& [idx, aw] : by_actor)
for (auto& w : aw.scenes) {
w.start = boundary_at_or_before(bounds, w.start);
w.end = boundary_after(bounds, w.end);
}
aw.scenes.push_back({c.first_seen, c.last_seen, c.belief});
}
std::vector<ActorWindow> result;
@@ -207,45 +190,6 @@ private:
return result;
}
// Sorted, de-duplicated boundary timestamps seen this run, framed by the
// film's own extent so the first and last shots are closed intervals. Derived
// from frames_ rather than a separate accumulator: the frames are already
// retained and this runs once.
//
// Prefers TransNetV2 shot boundaries (is_scene_boundary) when a scene detector
// populated them; otherwise falls back to the always-on histogram cuts
// (is_cut, camera_position_change_detector). On this ROCm box the scene
// detector cannot run in-process (see the dumper note), so is_cut is what
// flood-fill actually snaps to — coarser than true shot boundaries (cuts also
// fire on in-shot angle changes) but present with no extra pass.
std::vector<double> scene_boundaries() const {
bool have_scene = false;
for (const auto& sa : frames_)
if (sa.is_scene_boundary) { have_scene = true; break; }
std::vector<double> b;
b.push_back(0.0);
for (const auto& sa : frames_) {
const bool boundary = have_scene ? sa.is_scene_boundary : sa.is_cut;
if (boundary) b.push_back(sa.timestamp_sec);
}
b.push_back(last_ts_ + 1.0); // a right edge past the final sample
std::sort(b.begin(), b.end());
b.erase(std::unique(b.begin(), b.end()), b.end());
return b;
}
// The boundary opening the shot that contains t (largest boundary ≤ t).
static double boundary_at_or_before(const std::vector<double>& b, double t) {
auto it = std::upper_bound(b.begin(), b.end(), t);
return (it == b.begin()) ? b.front() : *(it - 1);
}
// The boundary closing the shot that contains t (smallest boundary > t).
static double boundary_after(const std::vector<double>& b, double t) {
auto it = std::upper_bound(b.begin(), b.end(), t);
return (it == b.end()) ? b.back() : *it;
}
json build_epochs() {
json actors = json::array();
for (const auto& aw : build_actor_windows()) {
@@ -258,7 +202,7 @@ private:
windows.push_back({{"start", w.start},
{"end", w.end},
{"belief", w.belief},
{"route", route_name(w.route)}});
{"route", "live"}});
json ja;
ja["name"] = aw.name;
ja["imdb_id"] = aw.imdb_id;
src/nodes/scene_boundary_annotator_node.hpp
+3 -11
View File
@@ -47,17 +47,9 @@ struct SceneBoundaryAnnotatorFunc {
// consumer is slower, and this branch is orders of magnitude faster per
// frame than TransNetV2. Blocking here is what makes the join real.
//
// What makes that safe is **join depth**, not branch independence. Now
// that the fanout is lossless (AR-004) it stops popping once this branch
// stops taking, so stalling here does eventually starve the detector —
// the two would wedge if this node could ask about a frame the detector
// has not been given the frames to score. It cannot, by a wide margin:
// the fanout can run the dense branch ahead by the whole of this
// branch's buffering, which is kSceneJoinDepth (256) *sampled* frames,
// and at sample_fps 5 against a ~25 fps source that is on the order of
// 1200 dense frames against TransNetV2's 100-frame window.
//
// Cutting kSceneJoinDepth below the window would reintroduce the wedge.
// Safe under backpressure because the branches are independent: this
// node stalling does not stop the detector consuming dense frames, and
// the fanout keeps feeding it.
if (!bounds_->wait_until_scored(f.timestamp_sec)) {
// The detector finished without covering this frame — the tail after
// its last full window. Unknown, not negative; counted so it cannot
src/nodes/scene_detector_node.hpp
+6 -109
View File
@@ -6,8 +6,6 @@
#include <memory>
#include "inference/scene_detector.hpp"
#include <opencv2/imgproc.hpp> // cv::resize, for to_model_input
#include <nlohmann/json.hpp>
#include <algorithm>
#include <atomic>
@@ -69,16 +67,7 @@ struct SceneDetectorFunc {
return;
}
/// TRACES: AR-011 | SR-002
// Learn the cadence of the stream from the stream itself, rather than
// assuming one. See dedup_window_sec().
if (prev_ts_ >= 0.0 && intervals_.size() < kCadenceSamples) {
const double dt = f.timestamp_sec - prev_ts_;
if (dt > 0.0) intervals_.push_back(dt);
}
prev_ts_ = f.timestamp_sec;
images_.push_back(to_model_input(f.image));
images_.push_back(f.image);
times_.push_back(f.timestamp_sec);
// Once we have a full window, score it and slide forward by `stride`.
@@ -92,76 +81,6 @@ struct SceneDetectorFunc {
}
}
/// TRACES: AR-004, AR-010 | SR-002
/// Reduce a decoded frame to exactly what TransNetV2 consumes, once.
///
/// The window used to hold the frames as decoded — full resolution — and
/// leave the downscale to the backend. But the model's input is 48x27
/// (`ISceneDetector::kFrameW/H`; the config note for `dense_scale` says so
/// outright: "TransNetV2 downsamples to 48x27 regardless"), so the buffer
/// held ~590 MB at 1080p to feed something that needs ~380 KB. That is not
/// a channel capacity, so no amount of tuning channel depths would ever
/// have found it.
///
/// It is also redundant work. Windows overlap by `kWindow - stride`, so a
/// frame appears in several of them and was re-downscaled once per window;
/// now it is downscaled once, when it arrives.
///
/// **This must reproduce the backends' preprocessing exactly**, because the
/// project invariant is that every model gets the input it was trained for
/// — a model run off-distribution returns confident, plausible, wrong
/// output, and here that means fabricated shot boundaries. Both
/// ort_backend.cpp and trt_backend.cpp guard mis-sized input with, in this
/// order, `convertTo(CV_8UC3)` then
/// `cv::resize(..., {kFrameW, kFrameH}, 0, 0, cv::INTER_AREA)`. The same
/// two operations are done here, so the tensor the model receives is
/// unchanged; the backend guard then sees a correctly-sized frame and does
/// nothing. The interface has always specified this shape as the caller's
/// job ("Each frame must already be kFrameW x kFrameH, BGR, CV_8UC3"), so
/// this makes the node meet a contract it was already given.
static cv::Mat to_model_input(const cv::Mat& src) {
cv::Mat typed;
if (src.type() != CV_8UC3) src.convertTo(typed, CV_8UC3);
else typed = src;
if (typed.cols == ISceneDetector::kFrameW &&
typed.rows == ISceneDetector::kFrameH)
return typed;
cv::Mat small;
cv::resize(typed, small, {ISceneDetector::kFrameW, ISceneDetector::kFrameH},
0, 0, cv::INTER_AREA);
return small;
}
/// TRACES: AR-011 | SR-002
// How close two boundaries have to be before they are the same boundary,
// derived from the cadence the detector was actually fed.
//
// What this replaces is a literal 0.04 s — one frame at 25 fps, and silently
// wrong at any other rate. On a 30 fps source it spans more than a frame, so
// two cuts on consecutive frames merge into one and a real boundary is lost;
// the output does not show this, it simply contains fewer cuts. Assuming a
// frame rate is the same class of mistake as feeding a model the wrong rate,
// which is why this belongs to AR-011 and not to a tidy-up.
//
// Half a frame, not a whole one, because the only thing being deduplicated is
// one frame scored by two overlapping windows — a gap of zero. Two distinct
// frames are a full interval apart and must both survive. Half an interval
// separates those two cases without putting the decision on the knife-edge
// where floating-point error settles it.
//
// Median, not mean: a seek, or a gap where the decoder dropped a frame,
// contributes one long interval that would drag a mean and cannot move a
// median.
static double dedup_window_sec(std::vector<double> intervals) {
if (intervals.empty()) return 0.0; // <2 frames: nothing to deduplicate
const std::size_t mid = intervals.size() / 2;
std::nth_element(intervals.begin(), intervals.begin() + mid,
intervals.end());
return intervals[mid] * 0.5;
}
private:
// Run TransNetV2 on the leading kWindow frames of the buffer and record any
// boundaries found within the trusted centre region.
@@ -193,15 +112,8 @@ private:
// final verdict. The face branch consults this for frames it has not
// reached yet, and the watermark is what lets it tell "no boundary
// here" from "not scored yet".
/// TRACES: AR-011 | SR-002
// Hand the join the same dedup window scenes.json uses, derived from the
// observed cadence rather than assumed. Set on every window because the
// median refines as intervals accumulate; it converges within the first
// window and costs a double assignment thereafter.
if (shared_) {
shared_->set_merge_window(dedup_window_sec(intervals_));
if (hi > lo) shared_->publish(fresh, times_[hi - 1]);
}
if (shared_ && hi > lo)
shared_->publish(fresh, times_[hi - 1]);
}
// At EOF the tail (< kWindow frames) never formed a full window. Pad it out
@@ -234,10 +146,7 @@ private:
// the last full window — reach the join with no verdict and are treated
// as boundary-free without evidence, which is precisely the ambiguity
// the watermark exists to prevent.
if (shared_ && n > 0) {
shared_->set_merge_window(dedup_window_sec(intervals_));
shared_->publish(fresh, times_[n - 1]);
}
if (shared_ && n > 0) shared_->publish(fresh, times_[n - 1]);
}
void write_output() {
@@ -245,8 +154,6 @@ private:
written_ = true;
// Merge boundaries closer than one frame apart (dedup across window seams).
const double dedup_sec = dedup_window_sec(intervals_);
std::sort(boundaries_.begin(), boundaries_.end(),
[](const Boundary& a, const Boundary& b) {
return a.t < b.t;
@@ -260,7 +167,7 @@ private:
nlohmann::json cuts = nlohmann::json::array();
double last_t = -1e9;
for (const auto& b : boundaries_) {
if (b.t - last_t < dedup_sec) continue;
if (b.t - last_t < 0.04) continue; // ~1 frame @25fps dedup
cuts.push_back({{"t", b.t}, {"probability", b.prob}});
last_t = b.t;
}
@@ -273,12 +180,8 @@ private:
return;
}
f << root.dump(2) << "\n";
// Report the derived cadence: VR-006 re-tunes scene_threshold against it,
// and a rate that is not the source's is the first thing to suspect.
std::cerr << "\n[scene_detector] wrote " << root["cuts"].size()
<< " boundaries → " << output_path_
<< " (dedup=" << dedup_sec << "s from "
<< (dedup_sec > 0.0 ? 0.5 / dedup_sec : 0.0) << " fps)\n";
<< " boundaries → " << output_path_ << "\n";
}
static int kLast_() { return ISceneDetector::kWindow - 1; }
@@ -292,10 +195,6 @@ private:
struct Boundary { double t; float prob; };
// Enough to establish a rate; bounded so a feature-length film does not
// accumulate one double per frame for a number that stops moving early.
static constexpr std::size_t kCadenceSamples = 512;
std::unique_ptr<ISceneDetector> detector_;
float threshold_;
int stride_;
@@ -308,8 +207,6 @@ private:
std::deque<double> times_;
int64_t window_base_{0}; // frame index of images_.front()
std::vector<Boundary> boundaries_;
double prev_ts_{-1.0}; // AR-011: cadence, learned not assumed
std::vector<double> intervals_;
bool written_{false};
std::shared_ptr<SceneBoundaries> shared_; ///< AR-010 join point
};
src/nodes/scene_tracker_node.hpp
+102
View File
@@ -0,0 +1,102 @@
#pragma once
#include "types.hpp"
#include "config.hpp"
#include <map>
#include <iostream>
// ── SceneTrackerFunc ──────────────────────────────────────────────────────────
// KPN node: maintains an extinction-timer state machine per identified actor.
//
// On each MatchedSceneFrame:
// 1. Update last_seen for every matched known actor.
// 2. Expire actors whose last_seen is older than extinction_sec.
// 3. Emit SceneAnnotation with all currently active (non-expired) actors,
// including their most recently seen bbox and best similarity score.
//
// Unknown faces (actor_idx == -1) are passed through per-frame but are NOT
// tracked across frames — each frame reports its own unknowns independently.
struct SceneTrackerFunc {
static constexpr std::string_view label() { return "scene_tracker"; }
explicit SceneTrackerFunc(const Config& cfg)
: extinction_sec_(cfg.extinction_sec)
{
std::cerr << "[scene_tracker] extinction_sec=" << extinction_sec_ << "\n";
}
// Runtime setter for pipeline reuse across a sweep. Also clears the active-actor
// state so a re-run starts clean (no carry-over from the previous config's film).
void set_extinction_sec(double s) { extinction_sec_ = s; active_.clear(); }
SceneAnnotation operator()(MatchedSceneFrame mf) {
if (mf.source.eof) return {0.0, {}, /*eof=*/true};
double now = mf.source.timestamp_sec;
// Update known actors
for (const auto& ia : mf.actors) {
if (ia.actor_idx < 0) continue; // skip unknowns
auto& slot = active_[ia.actor_idx];
slot.last_seen = now;
slot.last_bbox = ia.bbox;
slot.last_crop = ia.crop;
slot.name = ia.name;
slot.imdb_id = ia.imdb_id;
slot.tmdb_id = ia.tmdb_id;
slot.jellyfin_id = ia.jellyfin_id;
// Keep the best (highest) similarity seen in this window
if (ia.similarity > slot.best_similarity)
slot.best_similarity = ia.similarity;
}
// Expire stale actors
for (auto it = active_.begin(); it != active_.end(); ) {
if ((now - it->second.last_seen) > extinction_sec_)
it = active_.erase(it);
else
++it;
}
// Build annotation: active known actors
std::vector<IdentifiedActor> visible;
visible.reserve(active_.size() + mf.actors.size());
for (const auto& [actor_idx, slot] : active_) {
IdentifiedActor ia;
ia.actor_idx = actor_idx;
ia.name = slot.name;
ia.imdb_id = slot.imdb_id;
ia.tmdb_id = slot.tmdb_id;
ia.jellyfin_id = slot.jellyfin_id;
ia.similarity = slot.best_similarity;
ia.bbox = slot.last_bbox;
ia.crop = slot.last_crop;
visible.push_back(ia);
}
// Append per-frame unknowns (actor_idx == -1) directly
for (const auto& ia : mf.actors) {
if (ia.actor_idx < 0) visible.push_back(ia);
}
return {now, std::move(visible)};
}
private:
struct Slot {
double last_seen{0.0};
float best_similarity{0.f};
cv::Rect2f last_bbox;
cv::Mat last_crop;
std::string name;
std::string imdb_id;
std::string tmdb_id;
std::string jellyfin_id;
};
double extinction_sec_;
std::map<int, Slot> active_; // actor_idx → state
};
src/python_bindings.cpp
+50
View File
@@ -16,6 +16,7 @@
#include "face_embedder_engine.hpp"
#include "gallery/gallery_calibration.hpp"
#include "gallery/gallery_store.hpp"
#include "quality.hpp"
#include <nanobind/nanobind.h>
#include <nanobind/ndarray.h>
@@ -176,6 +177,55 @@ NB_MODULE(sae_embed, m) {
}, "image"_a,
"Border-replicate pad by 50% and CLAHE, for a detector second try.");
// ── Quality (AR-028 … AR-030) ────────────────────────────────────────────
// Exposed for the same reason the calibration is: VR-012 has to select
// among the AR-029 candidates, and the measure it selects must be the one
// that ships. A numpy copy scored during the study would leave the shipped
// measure unmeasured, which is precisely the failure the whole quality axis
// exists to prevent.
nb::class_<SharpnessScores>(m, "SharpnessScores")
.def_ro("var_laplacian", &SharpnessScores::var_laplacian)
.def_ro("norm_var_laplacian", &SharpnessScores::norm_var_laplacian)
.def_ro("tenengrad", &SharpnessScores::tenengrad)
.def_ro("hf_energy_ratio", &SharpnessScores::hf_energy_ratio)
.def_ro("dir_min_tenengrad", &SharpnessScores::dir_min_tenengrad)
.def_ro("ok", &SharpnessScores::ok)
.def("__repr__", [](const SharpnessScores& s) {
return "<SharpnessScores varlap=" + std::to_string(s.var_laplacian) +
" normvarlap=" + std::to_string(s.norm_var_laplacian) +
" tenengrad=" + std::to_string(s.tenengrad) +
" hf=" + std::to_string(s.hf_energy_ratio) +
(s.ok ? ">" : " NOT-OK>");
});
m.def("assess_sharpness", [](ImageArray crop) {
return ::assess_sharpness(as_mat(crop));
}, "crop"_a,
"All four AR-029 sharpness candidates for a 112x112 aligned crop "
"(quality.hpp). Higher is sharper for every measure; scales are not "
"comparable between measures. Scored over a fixed 64x64 window on the "
"face interior, so background bokeh and hairstyle do not enter.");
m.def("sharpness_window", [] {
const cv::Rect w = ::sharpness_window();
return std::vector<int>{w.x, w.y, w.width, w.height};
},
"The (x, y, w, h) canonical-pixel window every sharpness measure is "
"taken over, so a study can show the pixels a score came from.");
m.def("alignment_residual", [](nb::ndarray<const float, nb::shape<5, 2>,
nb::c_contig, nb::device::cpu> landmarks)
-> std::optional<float> {
const Alignment a = ::estimate_alignment(as_landmarks(landmarks));
if (!a.ok) return std::nullopt; // degenerate landmarks
return a.residual;
}, "landmarks"_a,
"The AR-030 visibility measure: RMS landmark error in canonical "
"112x112 px left over after the best similarity fit onto the ArcFace "
"template (face_utils.hpp). None when the landmarks are degenerate. "
"Exposed so VR-012 can check whether blur leaks into the pose axis — "
"if it does, discounting on both would double-count one cause.");
// ── Calibration ──────────────────────────────────────────────────────────
// AR-024: the pipeline reasons in one probability space. Exposed so Python
// scores through the same sigmoid the C++ matcher uses, rather than a numpy
src/quality.hpp
+256
View File
@@ -0,0 +1,256 @@
#pragma once
/// TRACES: AR-028, AR-029 | SR-002
///
/// Sharpness of the aligned crop — candidate measures for AR-029.
///
/// Motion blur and soft focus destroy the high-frequency detail the embedder
/// keys on, and unlike face size they leave the bounding box looking perfectly
/// healthy. An embedder handed such a face does not fail: it returns a
/// confident, plausible, wrong vector that then competes on equal terms with
/// every good one in the gallery.
///
/// **Why four measures and not one.** AR-029's threshold has to be *located*,
/// the way VR-005 located the size floor, not chosen. Locating it means letting
/// a study rank candidates by how well each predicts real identity loss, so all
/// four ship and VR-012 picks the winner. Until that study reports, none of
/// these is "the" sharpness measure.
///
/// **They are computed on the 112×112 aligned crop**, never the raw box. The
/// crop is geometrically scale-normalised, so a measure taken there cannot
/// re-express face size the way a raw-pixel one would.
///
/// That normalisation is geometric, not informational, and the distinction
/// matters: a 40 px face upscaled into the canonical frame genuinely carries
/// less high-frequency detail than a 400 px one downscaled into it, so every
/// measure here *does* respond to source face size. It reads **effective
/// resolution in canonical space**, which is the union of "was small" and "was
/// blurred", not blur alone. Whether that makes a sharpness discount a
/// double-count against AR-002's size gate is VR-012's joint size×sigma grid to
/// settle: if identity loss is a function of the measure alone, one axis
/// suffices; if a small-but-sharp and a large-but-blurred face at equal measure
/// lose different amounts, the axes are genuinely separate. The unit test
/// `sharpness falls under downscale-upscale as well as under blur` pins this as
/// known behaviour rather than leaving it to be discovered as a surprise.
#include <opencv2/core.hpp>
#include <opencv2/imgproc.hpp>
#include <cmath>
// ── The measurement window ────────────────────────────────────────────────────
// All four measures see the same pixels, so a comparison between them is about
// the operator and not about the window each happened to pick.
//
// A 64×64 region centred on the face interior, not the whole crop. Under the
// ArcFace template the landmarks span x ∈ [38.3, 73.5], y ∈ [51.5, 92.4]; this
// window covers that plus the surrounding cheeks, brow and chin while excluding
// the corners.
//
// The corners are excluded because they are where the background lives, and
// studio headshots — the gallery's entire population — are very often shot at a
// wide aperture with a deliberately blurred background. Measured over the full
// crop, that bokeh drags the score down on exactly the sharpest, most
// cooperative images in the set, which would put the measure's response
// backwards on the population used to calibrate it. Hair is excluded for the
// weaker version of the same reason: its high-frequency content varies with
// hairstyle rather than with capture quality.
//
// 64 is also a power of two, so the DFT below gets its natural size.
inline constexpr int kSharpWindow = 64;
inline constexpr int kSharpWindowX = 24; // (24,36) … (88,100) in canonical px
inline constexpr int kSharpWindowY = 36;
/// Every candidate, computed in one pass over the window.
///
/// Higher is sharper for all four, so a discount curve has the same orientation
/// whichever one VR-012 selects. Scales are *not* comparable between measures —
/// only within one.
struct SharpnessScores {
/// Variance of the Laplacian. The textbook measure, included as the
/// baseline every other candidate has to beat. Second derivatives amplify
/// sensor noise, and the value scales with image contrast, so a
/// low-contrast sharp face reads as blurred. Expected to lose; it should
/// lose on the record rather than by assertion.
float var_laplacian{0.f};
/// Variance of the Laplacian over the variance of the intensity. Divides
/// out the first-order contrast dependence that var_laplacian carries,
/// which is the single confound most likely to matter on a gallery drawn
/// from thousands of different cameras, lighting setups and JPEG pipelines.
float norm_var_laplacian{0.f};
/// Tenengrad: mean squared Sobel gradient magnitude. A first derivative, so
/// markedly less noise-amplifying than the Laplacian, at the cost of
/// responding to coarser structure. Still contrast-dependent.
float tenengrad{0.f};
/// Fraction of spectral energy above a quarter of Nyquist, DC excluded.
/// A ratio, so contrast divides out by construction rather than by an
/// explicit correction, and it is the most direct statement of "how much
/// fine detail is actually present". Bounded in [0,1], which makes it the
/// easiest of the four to turn into a discount.
float hf_energy_ratio{0.f};
/// The worse of the two Sobel axes, normalised by a low-frequency contrast
/// estimate. The only candidate here that satisfies both requirements at
/// once, and it exists because the other four do not.
///
/// Two independent fixes, each answering a measured failure of the four
/// above (numbers from the T1 ladders in tests/test_quality.cpp):
///
/// - **Normalise by low frequencies, not by total energy.** Dividing by
/// the whole intensity variance puts the detail being measured into the
/// denominator as well as the numerator, so a blur shrinks both and the
/// quotient barely moves. A Gaussian at sigma 4 canonical px keeps
/// illumination and coarse facial structure and discards detail, giving
/// a contrast estimate that blur leaves alone.
/// - **Take the minimum over direction, not the sum.** Motion blur is
/// directional: a horizontal smear destroys horizontal detail and
/// leaves vertical detail untouched. Summing the two axes (as
/// Tenengrad does) lets the surviving axis mask the destroyed one — the
/// reason both ratio measures are U-shaped in blur length, scoring a
/// 21 px smear about as sharp as a 3 px one. The minimum tracks the
/// axis that was ruined, which is the one the embedder suffers from.
///
/// Falls 510 → 12 monotonically across that same motion-blur ladder, stays
/// monotone under Gaussian blur and resampling, and moves 0.4% when
/// contrast is halved.
float dir_min_tenengrad{0.f};
/// False when the crop was the wrong size or degenerate (flat). Scored,
/// never silently dropped: a face whose sharpness cannot be computed is a
/// fact the dump should record, not an absence.
bool ok{false};
};
/// The window every measure is taken over. Exposed so a study can show the
/// pixels a score was computed from rather than trusting the constants.
inline cv::Rect sharpness_window() {
return {kSharpWindowX, kSharpWindowY, kSharpWindow, kSharpWindow};
}
namespace detail {
/// Fraction of spectral energy above `cutoff` × Nyquist, DC bin excluded.
///
/// A Hann window is applied first. Without it the DFT sees the region's edges
/// as a step discontinuity, and that step is broadband: it deposits energy at
/// every frequency including the high band being measured, so a uniformly
/// blurry crop still scores a substantial high-frequency fraction and the
/// measure's dynamic range collapses.
///
/// **The mean is removed before the window, not after.** Windowing a signal
/// that still carries its DC offset multiplies that constant by the Hann taper,
/// and the taper's own spectrum is not a single bin — the offset smears across
/// the low-frequency neighbourhood, where dropping bin (0,0) no longer removes
/// it. The leaked energy lands in the denominator without scaling with image
/// contrast, so the "ratio" silently becomes a function of absolute brightness:
/// on the synthetic crop, a 20/255 brightening moved it 23% and halving the
/// contrast moved it by a factor of 3.6. Subtracting the mean first restores
/// the invariance the ratio form is supposed to provide for free.
inline float hf_ratio(const cv::Mat& gray32, float cutoff = 0.25f) {
static const cv::Mat hann = [] {
cv::Mat w(kSharpWindow, kSharpWindow, CV_32F);
for (int y = 0; y < kSharpWindow; ++y) {
const float wy = 0.5f * (1.f - std::cos(2.f * float(CV_PI) * y / (kSharpWindow - 1)));
for (int x = 0; x < kSharpWindow; ++x) {
const float wx = 0.5f * (1.f - std::cos(2.f * float(CV_PI) * x / (kSharpWindow - 1)));
w.at<float>(y, x) = wx * wy;
}
}
return w;
}();
cv::Mat centred;
cv::subtract(gray32, cv::mean(gray32), centred);
cv::Mat windowed;
cv::multiply(centred, hann, windowed);
cv::Mat spectrum;
cv::dft(windowed, spectrum, cv::DFT_COMPLEX_OUTPUT);
// Quadrants are wrapped: frequency index n maps to the signed frequency
// n - N for n > N/2, so the radius has to be computed on the wrapped index.
const int N = kSharpWindow;
const float nyquist = N / 2.f;
const float r_cut = cutoff * nyquist;
double total = 0.0, high = 0.0;
for (int y = 0; y < N; ++y) {
const float fy = (y <= N / 2) ? float(y) : float(y - N);
for (int x = 0; x < N; ++x) {
if (x == 0 && y == 0) continue; // DC carries no detail
const float fx = (x <= N / 2) ? float(x) : float(x - N);
const auto& c = spectrum.at<cv::Vec2f>(y, x);
const double e = double(c[0]) * c[0] + double(c[1]) * c[1];
total += e;
if (std::sqrt(fx * fx + fy * fy) > r_cut) high += e;
}
}
if (total < 1e-12) return 0.f; // flat region
return static_cast<float>(high / total);
}
} // namespace detail
/// Score a 112×112 aligned BGR (or single-channel) crop on all four candidates.
///
/// Costs one colour conversion and three small convolutions over a 64×64 window
/// — negligible beside the embedder inference it guards.
inline SharpnessScores assess_sharpness(const cv::Mat& crop) {
SharpnessScores s;
const cv::Rect win = sharpness_window();
if (crop.empty() ||
win.x + win.width > crop.cols || win.y + win.height > crop.rows)
return s;
cv::Mat gray;
if (crop.channels() == 3) cv::cvtColor(crop(win), gray, cv::COLOR_BGR2GRAY);
else gray = crop(win).clone();
// Scale into [0,1] so a score does not depend on the 8-bit convention, and
// so the two contrast-normalised measures are comparable across builds.
cv::Mat g32;
gray.convertTo(g32, CV_32F, 1.0 / 255.0);
cv::Scalar mu, sigma;
cv::meanStdDev(g32, mu, sigma);
const double var_img = sigma[0] * sigma[0];
cv::Mat lap;
cv::Laplacian(g32, lap, CV_32F, 3);
cv::Scalar lmu, lsigma;
cv::meanStdDev(lap, lmu, lsigma);
const double var_lap = lsigma[0] * lsigma[0];
cv::Mat gx, gy;
cv::Sobel(g32, gx, CV_32F, 1, 0, 3);
cv::Sobel(g32, gy, CV_32F, 0, 1, 3);
cv::Mat gx2, gy2;
cv::multiply(gx, gx, gx2);
cv::multiply(gy, gy, gy2);
cv::Mat mag2 = gx2 + gy2;
// Contrast from low frequencies only — see dir_min_tenengrad. Blur leaves
// this denominator alone, which is exactly what the other two normalised
// measures lack.
cv::Mat lf;
cv::GaussianBlur(g32, lf, cv::Size(0, 0), 4.0);
cv::Scalar lfmu, lfsigma;
cv::meanStdDev(lf, lfmu, lfsigma);
const double var_lf = lfsigma[0] * lfsigma[0];
s.var_laplacian = static_cast<float>(var_lap);
// A flat window has no contrast to normalise by. Reporting 0 (rather than a
// huge quotient) keeps "less sharp" pointing the same way for a degenerate
// input as for a blurred one.
s.norm_var_laplacian = var_img > 1e-9 ? static_cast<float>(var_lap / var_img) : 0.f;
s.tenengrad = static_cast<float>(cv::mean(mag2)[0]);
s.hf_energy_ratio = detail::hf_ratio(g32);
s.dir_min_tenengrad = var_lf > 1e-9
? static_cast<float>(std::min(cv::mean(gx2)[0], cv::mean(gy2)[0]) / var_lf)
: 0.f;
s.ok = var_img > 1e-9;
return s;
}
src/scene_boundaries.hpp
+4 -25
View File
@@ -27,35 +27,15 @@
class SceneBoundaries {
public:
/// TRACES: AR-011 | SR-002
/// Peaks closer than this are one boundary.
///
/// Supplied by the detector, derived from the cadence it was actually fed
/// (SceneDetectorFunc::dedup_window_sec), NOT assumed. It used to be a hard
/// 0.04 here, and AR-011 is recorded as having replaced that literal --
/// which it did, but only for scenes.json. This path, the one that feeds
/// is_scene_boundary into the tracker, kept the constant while the comment
/// above it claimed "matches the dedup scenes.json applies, so the two
/// views agree". They did not agree. 0.04 s is one frame at 25 fps and
/// wider than a frame at 30, so two cuts on consecutive frames merged into
/// one and the loss was invisible: the pipeline simply saw fewer
/// boundaries.
///
/// Zero until the detector sets it, which makes the pre-cadence state a
/// no-op dedup rather than a wrong one -- adjacent peaks stay separate
/// until there is evidence about how far apart frames are, and is_boundary
/// absorbs duplicates in its tolerance anyway.
void set_merge_window(double sec) {
std::lock_guard<std::mutex> g(mu_);
merge_sec_ = sec;
}
/// Peaks closer than this are one boundary. Matches the dedup scenes.json
/// applies, so the two views agree.
static constexpr double kMergeSec = 0.04;
/// Called by the scene detector as each window is scored. `through` is the
/// timestamp up to which its verdict is now final.
void publish(const std::vector<double>& ts, double through) {
{
std::lock_guard<std::mutex> g(mu_);
const double merge = merge_sec_;
// Dedup on insert, matching what scenes.json does at write time. A run
// of adjacent high-scoring frames is one boundary, not several, and
// leaving them raw made this view report 357 where the file said 13 —
@@ -65,7 +45,7 @@ public:
bounds_.insert(bounds_.end(), ts.begin(), ts.end());
std::sort(bounds_.begin(), bounds_.end());
bounds_.erase(std::unique(bounds_.begin(), bounds_.end(),
[merge](double a, double b) { return b - a < merge; }),
[](double a, double b) { return b - a < kMergeSec; }),
bounds_.end());
scored_through_ = std::max(scored_through_, through);
}
@@ -140,7 +120,6 @@ private:
mutable std::mutex mu_;
mutable std::condition_variable cv_;
bool finished_{false};
double merge_sec_{0.0}; ///< set by the detector; see set_merge_window
std::vector<double> bounds_;
double scored_through_{-1.0};
mutable std::size_t outran_{0};
src/scene_preview.cpp
+15 -38
View File
@@ -8,7 +8,7 @@
//
// camera_pos (histogram cut detector) stamps Frame::cut_score / is_cut, which
// ride through to the preview HUD's cut-score meter.
// ├──► [frame_annotation] ──► [result_sink] (background thread)
// ├──► [scene_tracker] ──► [result_sink] (background thread)
// └──► [preview_node] (main thread)
//
// The main thread drives preview_node via preview.step(). When the movie ends
@@ -30,9 +30,7 @@
#include "nodes/embedder_node.hpp"
#include "nodes/face_tracker_node.hpp"
#include "nodes/identity_matcher_node.hpp"
#include "track_registry.hpp"
#include "evidence_discount.hpp"
#include "nodes/frame_annotation_node.hpp"
#include "nodes/scene_tracker_node.hpp"
#include "nodes/result_sink_node.hpp"
#include "nodes/preview_node.hpp"
@@ -73,6 +71,8 @@ static Config parse_args(int argc, char** argv) {
else if (arg("--verbosity")) { int v = std::stoi(next()); cfg.verbosity = v == 2 ? Verbosity::xray : v == 1 ? Verbosity::standard : Verbosity::minimal; }
else if (arg("--prior")) cfg.match_prior = std::stof(next());
else if (arg("--prob-threshold")) cfg.prob_threshold = std::stof(next());
else if (arg("--match-threshold")) cfg.match_threshold = std::stof(next());
else if (arg("--extinction")) cfg.extinction_sec = std::stod(next());
else if (arg("--detector")) cfg.detector_model = next();
else if (arg("--detector-engine")) cfg.detector_engine = next();
else if (arg("--arcface")) cfg.arcface_model = next();
@@ -81,10 +81,13 @@ static Config parse_args(int argc, char** argv) {
else if (arg("--conf")) cfg.detector_conf = std::stof(next());
else if (arg("--max-faces")) cfg.max_faces = std::stoi(next());
else if (arg("--min-face-px")) cfg.min_face_px = std::stof(next());
else if (arg("--ratio")) cfg.match_ratio = std::stof(next());
else if (arg("--ratio-ceil")) cfg.match_ratio_ceil = std::stof(next());
else if (arg("--track-alpha")) cfg.track_alpha = std::stof(next());
else if (arg("--track-min-iou")) cfg.track_min_iou = std::stof(next());
else if (arg("--track-min-prob")) cfg.track_assoc_min_prob = std::stof(next());
else if (arg("--track-extinction")) cfg.track_extinction_sec = std::stod(next());
else if (arg("--track-max-embed")) cfg.track_max_embed_dist = std::stof(next());
else if (arg("--track-max-missing")) cfg.track_max_frames_missing = std::stoi(next());
else if (arg("--anneal")) cfg.anneal_sec = std::stod(next());
else if (arg("--trt-cache")) cfg.trt.cache_dir = next();
else if (arg("--trt-fp16")) cfg.trt.fp16 = true;
else if (arg("--no-trt-fp16")) cfg.trt.fp16 = false;
@@ -93,8 +96,8 @@ static Config parse_args(int argc, char** argv) {
// Per-film gallery expansion — preview supports it (same cfg fields).
else if (arg("--expand-gallery")) cfg.expand_gallery = true;
else if (arg("--expand-buffer")) cfg.expand_buffer_size = std::stoi(next());
else if (arg("--expand-band-lo")) cfg.expand_band_lo = std::stof(next());
else if (arg("--expand-band-hi")) cfg.expand_band_hi = std::stof(next());
else if (arg("--expand-novelty-sim")) cfg.expand_novelty_sim = std::stof(next());
else if (arg("--expand-spread-max")) cfg.expand_track_spread_max = std::stof(next());
else if (arg("--expand-min-anchor")) cfg.expand_min_anchor_frames = std::stoi(next());
// Scene detection is scene_analyze-only (needs the dense TransNetV2 branch).
// Accept the flags so a shared command line runs, but note they're inert
@@ -144,37 +147,11 @@ int main(int argc, char** argv) {
FaceDetectorFunc detector_fn{cfg};
FaceAlignerFunc aligner_fn;
EmbedderFunc embedder_fn{cfg};
/// TRACES: AR-007, AR-012, AR-024 | DP-001 | SR-002 | PR-004
// Construction order matters and is the same as main.cpp's, deliberately:
// the matcher fits (or loads) the calibration, the registry needs a
// discounter built from it, and the tracker needs both. DP-001 says modes
// are front-ends that must not fork pipeline logic -- this file had forked
// it and then rotted, constructing FaceTrackerFunc{cfg} against a signature
// that stopped existing with the AR-007/AR-008 redesign, so scene_preview
// has not compiled since. Keeping the order identical is what stops that
// recurring.
FaceTrackerFunc ftracker_fn{cfg};
IdentityMatcherFunc matcher_fn {gallery, cfg};
auto same_person = same_person_probability(matcher_fn.calibration());
TrackRegistry::Config reg_cfg;
reg_cfg.track_extinction_sec = cfg.track_extinction_sec;
reg_cfg.ownership_logodds = cfg.ownership_logodds;
EvidenceDiscounter::Config disc_cfg;
disc_cfg.max_views = cfg.evidence_max_views;
disc_cfg.admit_below = cfg.evidence_admit_below;
disc_cfg.rho_max = cfg.evidence_rho_max;
auto registry = std::make_shared<TrackRegistry>(
reg_cfg, EvidenceDiscounter(same_person, disc_cfg));
matcher_fn.set_registry(registry);
FaceTrackerFunc ftracker_fn{cfg, registry, same_person};
FrameAnnotationFunc tracker_fn {};
SceneTrackerFunc tracker_fn {cfg};
ResultSinkFunc sink_fn {cfg, done};
// AR-012/AR-016: windows come from registry claims, and tracks still live
// at EOF must be flushed or the closing scene's cast is never emitted.
registry->on_track_dead([&sink_fn](const DeadTrack& d) { sink_fn.add_claim(d); });
sink_fn.set_pre_write_hook([registry](double last_ts) { registry->flush(last_ts); });
// ── KPN ObjectNodes ───────────────────────────────────────────────────────
kpn::ObjectNode<FrameSourceFunc, kpn::in<>, kpn::out<"raw">, "frame_source", 0> source (source_fn, 32);
kpn::ObjectNode<CameraPositionChangeDetectorFunc, kpn::in<"raw">, kpn::out<"frame">, "camera_pos", 0> campos (campos_fn, 32);
@@ -183,13 +160,13 @@ int main(int argc, char** argv) {
kpn::ObjectNode<EmbedderFunc, kpn::in<"aligned">, kpn::out<"embedded">, "embedder", 0> embedder (embedder_fn, 32);
kpn::ObjectNode<FaceTrackerFunc, kpn::in<"embedded">, kpn::out<"tracked">, "face_tracker", 0> ftracker (ftracker_fn, 16);
kpn::ObjectNode<IdentityMatcherFunc, kpn::in<"tracked">, kpn::out<"matched">, "identity_matcher", 0> matcher (matcher_fn, 16);
kpn::ObjectNode<FrameAnnotationFunc, kpn::in<"matched">, kpn::out<"annotation">, "frame_annotation", 0> tracker (tracker_fn, 16);
kpn::ObjectNode<SceneTrackerFunc, kpn::in<"matched">, kpn::out<"annotation">, "scene_tracker", 0> tracker (tracker_fn, 16);
kpn::ObjectNode<ResultSinkFunc, kpn::in<"annotation">,kpn::out<>, "result_sink", 0> sink (sink_fn, 16);
// MainThreadNode — no thread spawned; driven by preview.step() below
PreviewNode preview{cfg, 16};
// matcher → FanoutNode<MatchedSceneFrame,2> → [frame_annotation, preview] (auto-inserted)
// matcher → FanoutNode<MatchedSceneFrame,2> → [scene_tracker, preview] (auto-inserted)
auto net = kpn::make_network(
kpn::edge(source.output<"raw">(), campos.input<"raw">()),
kpn::edge(campos.output<"frame">(), detector.input<"frame">()),
src/track_registry.hpp
+20 -196
View File
@@ -44,39 +44,12 @@
// A finished presence claim, emitted exactly once when a track is reaped or
// flushed. Immutable by construction: it carries everything needed to justify
// itself (AR-017), with no back-reference into registry state.
/// TRACES: AR-017 | IR-002 | SR-002, SR-003
/// How an actor came to be attached to a track.
///
/// AR-017 requires every presence claim to carry its identification route, and
/// IR-002 publishes it per window. Until now the sink wrote the string "live"
/// unconditionally, so the field existed but could not distinguish anything --
/// and AR-017's own verification asks for "deferred and pooled routes
/// distinguishable".
///
/// Only `live` occurs today. `deferred` is what AR-020's pass will set when it
/// resolves a track that failed during streaming and was identified against the
/// final expanded gallery; the value exists now so that pass has somewhere to
/// write rather than a serialisation change to make.
enum class Route {
live, ///< identified while streaming, from accumulated per-frame evidence
deferred, ///< resolved after EOF against the expanded gallery (AR-020)
};
inline const char* route_name(Route r) {
switch (r) {
case Route::deferred: return "deferred";
case Route::live: break;
}
return "live";
}
struct DeadTrack {
int track_id{-1};
double first_seen{0.0};
double last_seen{0.0}; ///< always the last sighting, never the death time
int actor_idx{-1}; ///< -1 when the track was never owned
float belief{0.0f}; ///< accumulated posterior for actor_idx
Route route{Route::live}; ///< how the actor was attached (AR-017)
int observations{0}; ///< evidence updates that landed on this track
float effective_obs{0.f}; ///< sum of discounted weights — the evidence that counted
};
@@ -97,12 +70,7 @@ struct Track {
Embedding mean{}; ///< running directional mean
std::vector<Embedding> views; ///< distinct looks, for AR-025 discounting
float discounted_weight{0.f}; ///< sum of applied weights
int n_obs{0}; ///< every scored face on this track
/// Observations that were actually evidence, and so spent the correlation
/// budget. Indexing the effective-sample correction by this rather than by
/// n_obs is what stops non-matches exhausting it — see
/// Config::evidence_floor_p.
int n_evidence{0};
int n_obs{0};
bool on_screen() const { return !last_seen.has_value(); }
};
@@ -113,50 +81,8 @@ public:
using DeadTrackFn = std::function<void(const DeadTrack&)>;
struct Config {
/// How long a lost track stays available for re-association.
///
/// Named to match Config::track_extinction_sec, which feeds it, and
/// deliberately NOT `extinction_sec`: that name belonged to the
/// withdrawn actor keep-alive, and SPEC.md's removal list ends "grep
/// for both names and expect no survivors". A survivor here would be
/// the one false positive in that grep, on a field that means
/// something else entirely -- this one bounds re-association and never
/// extends a presence claim.
double track_extinction_sec{5.0};
double extinction_sec{5.0}; ///< how long a lost track stays revivable
float ownership_logodds{2.0f}; ///< belief needed to own a track (~0.88 posterior)
/// TRACES: AR-025 | SR-002
/// Posterior below which an observation is not evidence *for* an actor,
/// and so does not spend that actor's correlation budget.
///
/// The budget is an effective-sample correction: with observations
/// correlated at rho, the weight of the n-th is
/// `n_eff(n+1) - n_eff(n) = 2/((n+1)(n+2))` at rho=0.5, so it decays
/// quadratically and the total converges to 1/rho = 2. That is the
/// intended behaviour — a long static shot must not out-argue varied
/// evidence purely by lasting longer.
///
/// What was not intended is *who spends it*. Every scored face was
/// folded in, so an observation at p=0.02 — which contributes
/// log(0.98) = -0.02 of belief, nothing — consumed the same increment
/// as one at p=0.95. On SuperHero-2 track 3 that exhausted the budget
/// on the frames that recognised nobody: 103 observations, effective
/// weight 2.026, belief 0.455 against a 0.881 threshold, with the 51
/// frames that did identify the actor arriving when each was worth
/// 0.0002. The identification was lost.
///
/// It also made the answer depend on frame rate, which is the defect
/// AR-013 already had to fix once: deliver more frames, dilute the
/// budget with more non-matches, and a track that was owned stops
/// being owned. Measured — the same clip identified the actor before a
/// KPN throughput fix and not after, from identical input.
///
/// 0.5 is the point where the posterior stops favouring the hypothesis
/// at all, not a tuned threshold. Near-misses still count, which is the
/// design: an observation at 0.6 is evidence and is folded in. Below
/// 0.5 the observation argues *against*, which noisy-OR cannot
/// represent, so nothing is lost by declining to spend a budget on it.
float evidence_floor_p{0.5f};
};
/// The discounter is a constructor argument rather than an option: there is
@@ -176,53 +102,13 @@ public:
FrameScope(TrackRegistry& reg, double now)
: reg_(reg), lock_(reg.mu_) { reg_.tick_locked(now); }
/// All ASSOCIABLE tracks — **one pool**. `last_seen` tells the caller
/// whether IoU is meaningful; a dormant track is matched on embedding
/// alone. There is no separate revival path (AR-008).
///
/// TRACES: AR-008, AR-013 | SR-002
/// Association and reaping share ONE clock — the evidence watermark when a
/// matcher is attached, the tracker clock otherwise (they coincide when
/// there is only one). `candidates()` and `reap_locked()` apply the SAME
/// `track_extinction_sec` horizon against that clock, so the offered pool
/// and the live pool are the same set:
///
/// offered ⟺ (clock - last_seen) ≤ track_extinction_sec
/// reaped/erased ⟺ (clock - last_seen) > track_extinction_sec
///
/// This closes two symmetric failures. (1) Offering on the tracker's clock
/// (ahead of the watermark) let a face associate onto a track the registry
/// had ALREADY reaped on the watermark; the vote then landed on a dead id
/// and was dropped (record_vote → dropped_votes_). Rare live (small lag),
/// but replay runs the tracker far ahead of the matcher and lost ~0.3% of
/// votes. (2) Historically, offering on a LOOSER horizon than the reap left
/// retired tracks in the pool while the matcher lagged, so a new face
/// re-associated onto a long-dead track and two people merged into one
/// window (measured: 5 actors/16 windows at depth 32 vs 3/5 at depth 10322).
/// A single clock and a single threshold make both impossible: nothing is
/// offered past its reap horizon, nothing is reaped while still offerable.
/// All live tracks — **one pool**. `last_seen` tells the caller whether
/// IoU is meaningful; a dormant track is matched on embedding alone.
/// There is no separate revival path (AR-008).
std::vector<Track*> candidates() {
std::vector<Track*> out;
out.reserve(reg_.tracks_.size());
// Filter association on the SAME clock reaping uses (the evidence
// watermark when a matcher is attached, else the tracker clock). The
// two used to differ deliberately — the tracker offered on now_ while
// the registry reaped on evidence_through_ — but that let the tracker
// associate a face onto a track the registry had already reaped on the
// watermark, whose vote then landed on a dead id and was dropped
// (record_vote → dropped_votes_). In the live pipeline the lag is tiny
// so it rarely bit; in replay the Python source runs the tracker far
// ahead of the matcher and ~0.3% of votes were lost. One clock for both
// "may this associate?" and "is this reaped?" closes the race: a track
// past the horizon is neither offered nor reaped-out-from-under a vote.
const double clock =
reg_.awaits_evidence_ ? reg_.evidence_through_ : reg_.now_;
for (auto& [id, t] : reg_.tracks_) {
if (t.last_seen &&
(clock - *t.last_seen) > reg_.cfg_.track_extinction_sec)
continue; // retired from association; still awaiting evidence
out.push_back(&t);
}
for (auto& [id, t] : reg_.tracks_) out.push_back(&t);
return out;
}
@@ -242,50 +128,6 @@ public:
/// happens to appear, and a film ending mid-track never closes.
void tick(double now) { std::lock_guard g(mu_); tick_locked(now); }
/// TRACES: AR-012, AR-013, AR-025 | SR-002
/// The evidence watermark: every observation up to `t` has been folded in.
///
/// Reaping is driven by THIS, not by the tracker's clock, and the difference
/// is what stops a correct answer from depending on how fast two nodes run.
///
/// The tracker and the matcher are separate KPN nodes with a channel between
/// them, and the matcher is much the slower of the pair. Backpressure —
/// working exactly as AR-004 intends — turns that channel's depth into lag,
/// so the tracker's timestamp can be far ahead of the last frame anybody has
/// actually voted on. Reaping on the tracker's clock therefore closed tracks
/// before their evidence arrived: the votes landed on ids that no longer
/// existed, were counted as dropped, and the track was emitted unowned or
/// not at all. Deeper channel, fewer identifications, from identical input.
///
/// The fix is not to bound the channel against `track_extinction_sec`. That
/// makes an algorithm constant police a throughput knob, and leaves the
/// answer a function of scheduling. It is to reap on the watermark, which is
/// the same device `SceneBoundaries::scored_through()` uses for the AR-010
/// join: a consumer past that point is asking about frames nobody has looked
/// at yet, and the honest response is to wait rather than to guess.
///
/// Monotonic, and only ever *delays* a reap, so no window can be extended by
/// it — AR-013's "a window ends at the last sighting, never after" is a
/// property of `emit_locked`, which takes `last_seen` and never `now`.
void advance_evidence(double t) {
std::lock_guard g(mu_);
if (t > evidence_through_) evidence_through_ = t;
reap_locked();
}
/// TRACES: AR-013, AR-025 | SR-002
/// Declare that some stage will publish an evidence watermark, so reaping
/// must wait for it.
///
/// Explicit rather than inferred from "has anyone voted yet". Inferring it
/// re-opens the bug exactly at startup: before the matcher's first frame no
/// vote has been seen, so the registry would fall back to the tracker's
/// clock during precisely the window in which the tracker is furthest
/// ahead. `IdentityMatcherFunc::set_registry` calls this, so any pipeline
/// with a matcher waits, and a test that drives the tracker alone keeps the
/// simple behaviour instead of hanging on a watermark nobody will publish.
void expect_evidence() { std::lock_guard g(mu_); awaits_evidence_ = true; }
// ── Evidence ─────────────────────────────────────────────────────────────
/// Fold one observation into a track's belief (AR-025).
///
@@ -310,13 +152,7 @@ public:
if (it == tracks_.end()) { ++dropped_votes_; return; }
Track& t = it->second;
++t.n_obs; // every scored face is seen, whether or not it is evidence
// Not evidence *for* this actor: contributes ~nothing to the belief and
// must not spend the correlation budget. See Config::evidence_floor_p.
if (posterior < cfg_.evidence_floor_p) return;
const float w = discounter_.weight(t.views, t.n_evidence, e);
const float w = discounter_.weight(t.views, t.n_obs, e);
// Weighted lazy-OR: P_new = 1 − (1 − P_old)·(1 − p)^w, which in log
// space is a plain sum. w is the discounted evidence (AR-025), so a
@@ -325,7 +161,7 @@ public:
const float p = std::min(1.f - 1e-6f, std::max(1e-6f, posterior));
t.belief[actor_idx] += w * std::log(1.f - p);
t.discounted_weight += w;
++t.n_evidence;
++t.n_obs;
const int best = argmax_belief(t);
const float best_p = 1.f - std::exp(t.belief[best]);
@@ -370,15 +206,6 @@ public:
// ── Diagnostics ──────────────────────────────────────────────────────────
// These measure how often tracking is silently wrong, which nothing in the
// pipeline currently reveals.
/// Whether the registry still holds this track. The authority on which
/// tracks exist, so annotating structures elsewhere (spatial boxes in the
/// tracker, diversity buffers in the expansion store) can prune against it
/// rather than keeping a second opinion.
bool is_live(int track_id) const {
std::lock_guard g(mu_);
return tracks_.count(track_id) != 0;
}
int dropped_votes() const { std::lock_guard g(mu_); return dropped_votes_; }
int belief_swaps() const { std::lock_guard g(mu_); return belief_swaps_; }
int actor_conflicts() const { std::lock_guard g(mu_); return actor_conflicts_; }
@@ -387,20 +214,9 @@ public:
private:
// ── Locked internals ─────────────────────────────────────────────────────
void tick_locked(double now) {
// The tracker's clock still bounds association (a dormant track is only
// a candidate while it is alive), but it no longer decides death.
now_ = now;
reap_locked();
}
/// Reap against the evidence watermark when a producer of one is attached
/// (see expect_evidence); otherwise against the tracker's clock, which is
/// the same thing when there is only one clock.
void reap_locked() {
const double clock = awaits_evidence_ ? evidence_through_ : now_;
for (auto it = tracks_.begin(); it != tracks_.end(); ) {
const auto& ls = it->second.last_seen;
if (ls && (clock - *ls) > cfg_.track_extinction_sec) {
if (ls && (now - *ls) > cfg_.extinction_sec) {
emit_locked(it->second, *ls);
it = tracks_.erase(it);
} else {
@@ -515,6 +331,17 @@ private:
for (int i = 0; i < 512; ++i) t.mean[i] = static_cast<float>(t.mean[i] / norm);
}
static float logistic(float z) {
return z >= 0 ? 1.f / (1.f + std::exp(-z))
: std::exp(z) / (1.f + std::exp(z));
}
static float logit(float p) {
const float eps = 1e-6f;
p = std::min(1.f - eps, std::max(eps, p));
return std::log(p / (1.f - p));
}
Config cfg_;
EvidenceDiscounter discounter_;
mutable std::mutex mu_;
@@ -522,9 +349,6 @@ private:
std::map<int, int> owner_index_; ///< actor_idx → live track_id (AR-015)
DeadTrackFn on_dead_;
int next_id_{0};
double now_{0.0}; ///< tracker's clock (association)
double evidence_through_{0.0}; ///< matcher's watermark (reaping)
bool awaits_evidence_{false};
int dropped_votes_{0};
int belief_swaps_{0};
int actor_conflicts_{0};
src/types.hpp
-140
View File
@@ -59,36 +59,11 @@ inline constexpr float kArcFaceRef[5][2] = {
// Landmark order matches ArcFace convention (same as SCRFD output order):
// [0] right-eye-centre [1] left-eye-centre [2] nose
// [3] right-mouth [4] left-mouth
/// TRACES: AR-028 | SR-002
struct DetectedFace {
cv::Rect2f bbox;
std::array<cv::Point2f, 5> landmarks;
float confidence{0.f};
// ── AR-028 quality vector ────────────────────────────────────────────────
// Three axes, kept separate and never collapsed into one scalar: they fail
// for different reasons, have different remedies, and do not earn the same
// response. Carried, not consumed — the vector travels with the face into
// the VR-001 dump so a threshold can be re-litigated against recorded data
// rather than by re-running video.
//
// **Size is the third axis and is deliberately not a field here.** It is
// `bbox`, which every consumer already has, scaled by the frame's
// `bbox_upscale` to reach the original resolution AR-002 thresholds in.
// Copying it into a second field would put the same quantity in two
// coordinate spaces inside one struct — the trap SCHEMA.md records for
// `bbox_upscale` — and the copy would be the one that drifts.
//
// Both fields below are -1 until the aligner runs, so *unscored* is
// distinguishable from *scored badly*. Nothing downstream may read a
// negative value as a quality.
// AR-029 sharpness: normalised Laplacian variance over the aligned crop,
// dimensionless. Falls with motion blur and soft focus; invariant to
// contrast, and taken on the fixed 112×112 canvas so it cannot re-measure
// face size. See crop_sharpness() for the construction and its one hazard.
float sharpness{-1.f};
// AR-030 visibility: RMS landmark misfit, in canonical 112×112 pixels, left
// over after the best similarity fit to the ArcFace template. Rises with
// out-of-plane pose and with occlusion; blind to in-plane roll and to face
@@ -155,13 +130,6 @@ struct SceneAnnotation {
double timestamp_sec{0.0};
std::vector<IdentifiedActor> visible_actors;
bool eof{false};
// Carried through from Frame so the sink can collect boundaries for flood-fill
// presence (PresenceMode::flood). is_cut is the always-on histogram cut
// (camera_position_change_detector) — the boundary flood-fill uses by default.
// is_scene_boundary is the opt-in TransNetV2 shot boundary (0 unless scene
// detection ran); kept for a future out-of-process scene detector.
bool is_cut{false};
bool is_scene_boundary{false};
};
// ── Actor gallery ─────────────────────────────────────────────────────────────
@@ -191,111 +159,3 @@ struct ActorGallery {
bool calib_valid{false};
uint64_t calib_hash{0};
};
// ── Channel byte accounting ───────────────────────────────────────────────────
/// TRACES: AR-004 | SR-002
///
/// KPN measures a channel's occupancy in *items* and its bandwidth in bytes,
/// and gets the byte figure from `kpn::ChannelDataSize<T>`. That primary
/// template returns `sizeof(T)` — right for a POD, badly wrong for every type
/// below, each of which is a handful of vectors and a `cv::Mat` header owning
/// megabytes on the heap.
///
/// Unspecialised, the diagnostics reported roughly 200 bytes for a message
/// carrying a full decoded frame — off by four orders of magnitude at 1080p.
/// That is not merely a cosmetic stat: it is the one instrument for choosing
/// channel capacities against a memory ceiling, which is the open half of
/// AR-004, and it was reading fiction.
///
/// **What the number means.** `cv::Mat` is reference-counted, so one decoded
/// frame referenced from several messages is counted once per reference. The
/// sum is therefore an upper bound on distinct bytes, and the right bound for
/// the question being asked: how much would this channel keep alive if nothing
/// else held it.
///
/// Declared against a forward declaration rather than including
/// `<kpn/channel.hpp>` here, so the message definitions keep no dependency on
/// the framework that carries them — and so any translation unit that can see
/// these types also sees their sizes, which is what stops one channel being
/// instantiated with the default and another with the specialisation.
namespace kpn { template<typename T> struct ChannelDataSize; }
namespace sae::bytes {
inline std::size_t of(const cv::Mat& m) {
return m.empty() ? 0u : m.total() * m.elemSize();
}
inline std::size_t of(const std::vector<cv::Mat>& v) {
std::size_t n = 0;
for (const auto& m : v) n += of(m);
return n;
}
inline std::size_t of(const Frame& f) { return sizeof(Frame) + of(f.image); }
inline std::size_t of(const std::vector<IdentifiedActor>& v) {
std::size_t n = v.size() * sizeof(IdentifiedActor);
for (const auto& a : v) {
n += of(a.crop);
// The id strings are short but there is one set per actor per frame,
// and a crowd frame carries dozens.
n += a.name.capacity() + a.imdb_id.capacity()
+ a.tmdb_id.capacity() + a.jellyfin_id.capacity();
}
return n;
}
} // namespace sae::bytes
template<> struct kpn::ChannelDataSize<Frame> {
static std::size_t bytes(const Frame& f) { return sae::bytes::of(f); }
};
template<> struct kpn::ChannelDataSize<SceneFrame> {
static std::size_t bytes(const SceneFrame& v) {
return sizeof(SceneFrame) + sae::bytes::of(v.source)
+ v.faces.size() * sizeof(DetectedFace);
}
};
template<> struct kpn::ChannelDataSize<AlignedSceneFrame> {
static std::size_t bytes(const AlignedSceneFrame& v) {
return sizeof(AlignedSceneFrame) + sae::bytes::of(v.source)
+ v.faces.size() * sizeof(DetectedFace)
+ sae::bytes::of(v.crops);
}
};
template<> struct kpn::ChannelDataSize<EmbeddedSceneFrame> {
static std::size_t bytes(const EmbeddedSceneFrame& v) {
return sizeof(EmbeddedSceneFrame) + sae::bytes::of(v.source)
+ v.faces.size() * sizeof(DetectedFace)
+ sae::bytes::of(v.crops)
+ v.embeddings.size() * sizeof(Embedding);
}
};
template<> struct kpn::ChannelDataSize<TrackedSceneFrame> {
static std::size_t bytes(const TrackedSceneFrame& v) {
return sizeof(TrackedSceneFrame) + sae::bytes::of(v.source)
+ v.faces.size() * sizeof(DetectedFace)
+ sae::bytes::of(v.crops)
+ v.track_ids.size() * sizeof(int)
+ v.embeddings.size() * sizeof(Embedding);
}
};
template<> struct kpn::ChannelDataSize<MatchedSceneFrame> {
static std::size_t bytes(const MatchedSceneFrame& v) {
return sizeof(MatchedSceneFrame) + sae::bytes::of(v.source)
+ sae::bytes::of(v.actors);
}
};
template<> struct kpn::ChannelDataSize<SceneAnnotation> {
static std::size_t bytes(const SceneAnnotation& v) {
return sizeof(SceneAnnotation) + sae::bytes::of(v.visible_actors);
}
};
// CutEvent owns nothing on the heap, so the default sizeof(T) is already right.
tests/CMakeLists.txt
+1 -17
View File
@@ -19,16 +19,12 @@ add_executable(sae_tests
test_calibration.cpp
test_gallery_store.cpp
test_face_utils.cpp
test_quality.cpp
test_track_gallery.cpp
test_face_tracker.cpp
test_track_registry.cpp
test_face_detector_node.cpp
test_scene_detector_node.cpp
test_replay_fixtures.cpp
test_embedding_dump.cpp
test_audio_signature.cpp
test_benchmark.cpp
test_channel_bytes.cpp
${CMAKE_SOURCE_DIR}/src/backends/gemm_backend.cpp
${CMAKE_SOURCE_DIR}/src/gallery/gallery_store.cpp
${CMAKE_SOURCE_DIR}/src/audio_signature.cpp
@@ -49,14 +45,6 @@ endif()
if(OPENBLAS_T_FOUND)
target_include_directories(sae_tests PRIVATE ${OPENBLAS_T_INCLUDE_DIRS})
target_link_libraries(sae_tests PRIVATE ${OPENBLAS_T_LINK_LIBRARIES})
elseif(NOT SAE_ALLOW_SCALAR_GEMM)
# Same rule as the CPU backend itself: testing the scalar loop while the
# shipped CPU path is OpenBLAS means the suite is not evidence about the
# kernel that runs.
message(FATAL_ERROR
"OpenBLAS not found, and the unit tests compile the CPU GEMM kernel "
"(AR-026). Install openblas-devel, or pass -DSAE_ALLOW_SCALAR_GEMM=ON "
"to test the scalar fallback deliberately.")
endif()
target_compile_definitions(sae_tests PRIVATE
@@ -72,10 +60,6 @@ target_compile_definitions(sae_tests PRIVATE
target_link_libraries(sae_tests PRIVATE
Catch2::Catch2WithMain
nlohmann_json::nlohmann_json
# VR-015: test_benchmark.cpp includes src/benchmark.hpp, which reads KPN's
# diagnostics structs. Header-only — no KPN network is constructed here, so
# the cost attribution stays testable on CI's GPU-free N100.
kpn
ffmpeg_libs
${OpenCV_LIBS}
${HDF5_CXX_LIBRARIES})
tests/fixtures/audio/make_offset_fixture.sh
Vendored
+6 -6
View File
@@ -1,6 +1,6 @@
#!/bin/sh
#
# Regenerate superhero_offset_200s.flac — the real-audio fixture behind VR-014, the
# Regenerate bali_offset_200s.flac — the real-audio fixture behind VR-014, the
# audio-signature offset-recovery validation.
#
# sh make_offset_fixture.sh /path/to/clips
@@ -12,8 +12,8 @@
# one signature against another finds the true alignment and only the true
# alignment. Tones are pathologically easy for that; dialogue and score are not.
#
# Source: scene clips from SuperHero (TRECVID DVU development set), the corpus
# this repo already uses for the replay fixtures — tests/fixtures/dumps/superhero.h5
# Source: five scene clips from "Road to Bali" (1952), the public-domain corpus
# this repo already uses for the replay fixtures — tests/fixtures/dumps/bali_*.h5
# are dumps of these same clips. Each is under the 120 s window on its own
# (29-77 s), so they are concatenated in scene order to make a source long
# enough that a 120 s window can slide inside it.
@@ -41,13 +41,13 @@
set -eu
CLIPS="${1:-../../../../hero}"
OUT="$(dirname "$0")/superhero_offset_200s.flac"
CLIPS="${1:-../../../../bali}"
OUT="$(dirname "$0")/bali_offset_200s.flac"
LIST="$(mktemp)"
trap 'rm -f "$LIST"' EXIT
for scene in 13 27 28 31 46; do
clip="$CLIPS/SuperHero-$scene.webm"
clip="$CLIPS/Road_To_Bali-$scene.webm"
[ -f "$clip" ] || { echo "missing clip: $clip" >&2; exit 1; }
echo "file '$(cd "$(dirname "$clip")" && pwd)/$(basename "$clip")'" >> "$LIST"
done
tests/fixtures/dumps/.gitignore
-17
View File
@@ -1,17 +0,0 @@
# Replay fixtures are distributed as artifacts, not through git.
#
# They are large (superhero.h5 is ~9 MB) and regenerating one needs the film,
# the models and a GPU — none of which CI has. So they live in the Gitea
# generic package registry and are fetched on demand:
#
# scripts/artifacts/pull_artifacts.sh replay-fixtures [version]
# scripts/artifacts/push_artifacts.sh replay-fixtures
#
# The gallery ships alongside the dumps deliberately: a dump only replays
# meaningfully against the gallery it was produced with.
#
# bali_*.h5 predate this and remain tracked; do not add more to git.
superhero.h5
hero66.h5
gt.json
scene_bounds.json
tests/test_benchmark.cpp
-221
View File
@@ -1,221 +0,0 @@
// Cost attribution for the pipeline benchmark.
//
// TRACES: VR-015 | UT-120, UT-121, UT-122, UT-123, UT-124 | PR-004
//
// `attribute_cost` is pure — no clock, no thread, no network — precisely so the
// ranking can be tested on CI hardware that can never run the pipeline. These
// are T1 tests: they build the snapshots the KPN network would have produced
// and assert which node gets blamed.
//
// The case that matters is UT-121. On SuperHero the real run reported
// `frame_source ema=141.899ms` against a decoder logging 12-18 ms, because
// `fire_once` bills time parked pushing into a full downstream channel to the
// node doing the pushing. Any metric that ranks nodes by wall time inside the
// node picks the source — the fastest node in the graph — as the thing to
// optimise. That is the mistake this file exists to prevent regressing.
#include "benchmark.hpp"
#include <catch2/catch_test_macros.hpp>
#include <catch2/matchers/catch_matchers_floating_point.hpp>
#include <stdexcept>
#include <string_view>
using Catch::Matchers::WithinAbs;
using Catch::Matchers::WithinRel;
using namespace sae::bench;
namespace {
/// A KPN node snapshot with only the fields attribution reads.
kpn::NodeSnapshot node(std::string name, std::uint64_t frames,
double ema_ms, double cpu_ms, double exec_ms) {
kpn::NodeSnapshot s{};
s.name = std::move(name);
s.frames_processed = frames;
s.ema_exec_ms = ema_ms;
s.total_cpu_ms = cpu_ms;
s.total_exec_ms = exec_ms;
return s;
}
/// A channel whose mean fill is `fill_pct` of `capacity`.
ChannelOccupancy chan(const std::string& producer, const std::string& consumer,
std::size_t capacity, double fill_pct) {
ChannelOccupancy c;
c.name = producer + ":0 \xe2\x86\x92 " + consumer + ":0";
c.producer = producer;
c.consumer = consumer;
c.capacity = capacity;
c.samples = 1000;
c.fill_sum = static_cast<double>(c.samples) * static_cast<double>(capacity) * fill_pct / 100.0;
return c;
}
// Returned by value: a reference into `v` bound to a name built from a string
// literal trips -Wdangling-reference, and the struct is small enough not to care.
NodeCost by_name(const std::vector<NodeCost>& v, std::string_view name) {
for (const auto& c : v) if (c.name == name) return c;
throw std::runtime_error("no such node: " + std::string(name));
}
NodeCost bottleneck(const std::vector<NodeCost>& v) {
for (const auto& c : v) if (c.is_bottleneck) return c;
throw std::runtime_error("no bottleneck flagged");
}
} // namespace
// UT-120 — the node work queues up in front of is the one blamed.
TEST_CASE("attribute_cost blames the node with a full input and an empty output",
"[benchmark][VR-015]") {
// source → mid → sink. mid is slow: its input backs up, its output drains.
const auto nodes = std::vector<kpn::NodeSnapshot>{
node("source", 1000, 10.0, 2'000.0, 10'000.0),
node("mid", 1000, 50.0, 45'000.0, 50'000.0),
node("sink", 1000, 0.5, 400.0, 500.0),
};
const auto channels = std::vector<ChannelOccupancy>{
chan("source", "mid", 32, 95.0), // full: work piling up in front of mid
chan("mid", "sink", 16, 2.0), // empty: mid starves everything after
};
const auto costs = attribute_cost(nodes, channels, /*wall_sec=*/50.0);
CHECK(bottleneck(costs).name == "mid");
CHECK(by_name(costs, "mid").pressure > by_name(costs, "source").pressure);
CHECK(by_name(costs, "mid").pressure > by_name(costs, "sink").pressure);
}
// UT-121 — the SuperHero regression: a backpressured source reports a huge
// wall time per frame and must NOT be mistaken for the bottleneck.
TEST_CASE("a backpressured source is not blamed for the time it spent parked",
"[benchmark][VR-015]") {
// Numbers taken from the real SuperHero TRT run: the source reports
// 141.9 ms/frame inside fire_once while its decoder logs ~15 ms, because
// the remaining ~127 ms is spent parked on a full output channel.
const auto nodes = std::vector<kpn::NodeSnapshot>{
node("frame_source", 5132, 141.899, 77'000.0, 728'000.0),
node("face_detector", 5129, 6.830, 480'000.0, 35'000.0),
node("result_sink", 5129, 0.080, 410.0, 410.0),
};
const auto channels = std::vector<ChannelOccupancy>{
chan("frame_source", "face_detector", 32, 99.0), // source blocked on this
chan("face_detector", "result_sink", 16, 1.0),
};
const auto costs = attribute_cost(nodes, channels, /*wall_sec=*/500.0);
const auto& src = by_name(costs, "frame_source");
const auto& det = by_name(costs, "face_detector");
// The trap: by wall time inside the node, the source looks 20x costlier.
REQUIRE(src.exec_ms_per_frame > det.exec_ms_per_frame * 10.0);
// The fix: it is not blamed, because its own output channel is the thing
// that is full — it is waiting, not working.
CHECK_FALSE(src.is_bottleneck);
CHECK(bottleneck(costs).name == "face_detector");
// And CPU time, which parking cannot inflate, agrees: the detector burns
// 480 s of thread time against the source's 77 s.
CHECK(det.cpu_ms > src.cpu_ms);
CHECK(det.cpu_pct_of_pipeline > src.cpu_pct_of_pipeline);
}
// UT-122 — terminals stay rankable via the infinite-reservoir convention.
TEST_CASE("a source with an empty output is blamed; a sink with a full input is too",
"[benchmark][VR-015]") {
SECTION("starved pipeline: the source cannot keep up") {
const auto nodes = std::vector<kpn::NodeSnapshot>{
node("source", 100, 90.0, 9'000.0, 9'000.0),
node("mid", 100, 1.0, 100.0, 100.0),
};
// Nothing ever accumulates: the source is the constraint.
const auto costs = attribute_cost(nodes, {chan("source", "mid", 32, 1.0)}, 10.0);
CHECK(bottleneck(costs).name == "source");
// Source has no input channel, so it is treated as always having work.
CHECK_THAT(by_name(costs, "source").in_fill_pct, WithinAbs(100.0, 1e-9));
}
SECTION("congested pipeline: the sink cannot drain") {
const auto nodes = std::vector<kpn::NodeSnapshot>{
node("mid", 100, 1.0, 100.0, 100.0),
node("sink", 100, 90.0, 9'000.0, 9'000.0),
};
const auto costs = attribute_cost(nodes, {chan("mid", "sink", 16, 98.0)}, 10.0);
CHECK(bottleneck(costs).name == "sink");
// Sink has no output channel, so it is treated as never blocking.
CHECK_THAT(by_name(costs, "sink").out_fill_pct, WithinAbs(0.0, 1e-9));
}
}
// UT-123 — the per-node time figures are the ones an optimiser would act on.
TEST_CASE("cost shares are computed against wall clock and pipeline total",
"[benchmark][VR-015]") {
const auto nodes = std::vector<kpn::NodeSnapshot>{
node("a", 100, 1.0, 30'000.0, 40'000.0), // 30 s CPU
node("b", 100, 1.0, 10'000.0, 12'000.0), // 10 s CPU
};
const auto costs = attribute_cost(nodes, {chan("a", "b", 8, 50.0)}, /*wall_sec=*/50.0);
const auto& a = by_name(costs, "a");
CHECK_THAT(a.cpu_ms_per_frame, WithinRel(300.0, 1e-9)); // 30 s / 100 frames
CHECK_THAT(a.exec_ms_per_frame, WithinRel(400.0, 1e-9));
CHECK_THAT(a.cpu_share, WithinRel(0.6, 1e-9)); // 30 s of a 50 s run
CHECK_THAT(a.exec_share, WithinRel(0.8, 1e-9));
CHECK_THAT(a.cpu_pct_of_pipeline, WithinRel(75.0, 1e-9)); // 30 of 40 s total
// Time inside the node that was not spent on its own CPU: parked, or on GPU.
CHECK_THAT(a.stall_ms_per_frame, WithinRel(100.0, 1e-9));
// A node that never ran cannot be the bottleneck, and contributes no cost.
const auto idle = std::vector<kpn::NodeSnapshot>{
node("ran", 10, 1.0, 100.0, 100.0),
node("idle", 0, 0.0, 0.0, 0.0),
};
const auto idle_costs = attribute_cost(idle, {}, 10.0);
CHECK(bottleneck(idle_costs).name == "ran");
CHECK_FALSE(by_name(idle_costs, "idle").is_bottleneck);
}
// UT-124 — the node graph is recovered from KPN's channel names, which is what
// keeps attribution working when the topology changes.
TEST_CASE("channel names split back into producer and consumer",
"[benchmark][VR-015]") {
std::string p, c;
split_edge_name("frame_source:0 \xe2\x86\x92 camera_pos:0", p, c);
CHECK(p == "frame_source");
CHECK(c == "camera_pos");
// Multi-port nodes: the port index is stripped, the node name is not.
split_edge_name("detector:2 \xe2\x86\x92 aligner:1", p, c);
CHECK(p == "detector");
CHECK(c == "aligner");
// A name with no arrow leaves both untouched rather than inventing an edge.
std::string q = "unset", r = "unset";
split_edge_name("not an edge", q, r);
CHECK(q == "unset");
CHECK(r == "unset");
}
// A node with several inputs is gated by its emptiest one, and blocked by its
// fullest output — the multi-branch case the scene-detect topology creates.
TEST_CASE("multi-port nodes take min input fill and max output fill",
"[benchmark][VR-015]") {
const auto nodes = std::vector<kpn::NodeSnapshot>{
node("join", 100, 1.0, 1'000.0, 1'000.0),
};
const auto channels = std::vector<ChannelOccupancy>{
chan("up_a", "join", 32, 99.0), // full, but...
chan("up_b", "join", 32, 4.0), // ...this one gates the node
chan("join", "down_a", 16, 10.0),
chan("join", "down_b", 16, 80.0), // parking on this stops the node
};
const auto costs = attribute_cost(nodes, channels, 10.0);
const auto& j = by_name(costs, "join");
CHECK_THAT(j.in_fill_pct, WithinAbs( 4.0, 1e-9));
CHECK_THAT(j.out_fill_pct, WithinAbs(80.0, 1e-9));
CHECK(j.pressure < 0.0); // starved, not congested
}
tests/test_calibration.cpp
-9
View File
@@ -1,12 +1,6 @@
// TRACES: AR-023 | SR-002
//
// Unit tests for gallery calibration: the sigmoid math, the pairwise fit on
// separable data, and the in-memory hash-keyed cache (hit / stale / cold).
// All pure, GPU-free, model-free.
//
// The three `[report]` cases at the bottom carry their own GR-003 tags: they
// verify the build report, which is fitted from the same distributions but is a
// separate requirement.
#include <catch2/catch_test_macros.hpp>
#include <catch2/matchers/catch_matchers_floating_point.hpp>
@@ -189,7 +183,6 @@ Embedding unit_axis(int slot) {
}
} // namespace
// TRACES: GR-003 | SR-001
TEST_CASE("report surfaces actors that can never be recognised", "[report][GR-003]") {
// An actor with no usable image is a silent recall ceiling: the pipeline
// will never name them, and nothing in the gallery says why. This is the
@@ -219,7 +212,6 @@ TEST_CASE("report surfaces actors that can never be recognised", "[report][GR-00
CHECK(r.actors[1].references == 0);
}
// TRACES: GR-003 | SR-001
TEST_CASE("report surfaces actors too thin to calibrate on", "[report][GR-003]") {
// Below the positive-pair threshold an actor contributes nothing to the
// intra-class side of the fit. They are not broken, so nothing complains —
@@ -245,7 +237,6 @@ TEST_CASE("report surfaces actors too thin to calibrate on", "[report][GR-003]")
CHECK(r.actors_below_positive_threshold >= 1);
}
// TRACES: GR-003 | SR-001
TEST_CASE("report round-trips", "[report][GR-003]") {
ActorGallery g;
ActorGallery::Actor act;
tests/test_channel_bytes.cpp
-109
View File
@@ -1,109 +0,0 @@
// Channel byte accounting for the pipeline message types.
//
// TRACES: AR-004 | SR-002
//
// kpn::ChannelDataSize<T> is what a channel reports as bytes pushed, and its
// primary template returns sizeof(T). Every message type here is a handful of
// vectors and a cv::Mat header owning megabytes on the heap, so unspecialised
// the diagnostics reported ~200 bytes for a message carrying a full decoded
// frame — off by four orders of magnitude at 1080p.
//
// That is the instrument for choosing channel capacities against a memory
// ceiling, which is the open half of AR-004. These cases assert it measures the
// payload rather than the header, because a stat that is quietly wrong is worse
// than no stat: it was read as evidence.
#include <catch2/catch_test_macros.hpp>
#include <kpn/channel.hpp>
#include "types.hpp"
namespace {
Frame frame_with_image(int w, int h) {
Frame f;
f.image = cv::Mat(h, w, CV_8UC3, cv::Scalar(0, 0, 0));
f.timestamp_sec = 1.0;
return f;
}
} // namespace
TEST_CASE("frame bytes count the decoded image, not the header", "[channel_bytes]") {
const Frame f = frame_with_image(1920, 1080);
const std::size_t got = kpn::ChannelDataSize<Frame>::bytes(f);
// 1920 * 1080 * 3 = 6,220,800 payload bytes.
REQUIRE(got >= 1920u * 1080u * 3u);
// The header is a rounding error next to it; this is the assertion that
// fails on the unspecialised default.
CHECK(got > 100u * sizeof(Frame));
}
TEST_CASE("an empty frame costs only its header", "[channel_bytes]") {
// The eof sentinel carries no image, and must not be charged for one.
Frame eof;
eof.eof = true;
CHECK(kpn::ChannelDataSize<Frame>::bytes(eof) == sizeof(Frame));
}
TEST_CASE("crops and embeddings are counted on top of the frame", "[channel_bytes]") {
// The case AR-003 created: a crowd frame occupies one slot exactly as an
// empty one does, and only the byte figure distinguishes them.
EmbeddedSceneFrame v;
v.source = frame_with_image(640, 360);
const std::size_t bare = kpn::ChannelDataSize<EmbeddedSceneFrame>::bytes(v);
constexpr int kFaces = 60;
for (int i = 0; i < kFaces; ++i) {
v.faces.push_back({});
v.crops.emplace_back(112, 112, CV_8UC3, cv::Scalar(0, 0, 0));
v.embeddings.emplace_back();
}
const std::size_t crowded = kpn::ChannelDataSize<EmbeddedSceneFrame>::bytes(v);
// 60 crops at 112*112*3 = 2,257,920 bytes, plus 60 * 2 KiB of embeddings.
CHECK(crowded - bare >= kFaces * (112u * 112u * 3u + sizeof(Embedding)));
// And the crowd frame really is the multiple of the empty one that the
// item-count capacity cannot see: 640x360x3 is ~691 KB, the crops ~2.26 MB.
CHECK(crowded > 3 * bare);
}
TEST_CASE("every message type on a channel measures its payload", "[channel_bytes]") {
// A specialisation missing for any one of these silently reverts that
// channel to sizeof(T), which is exactly how this went unnoticed.
const Frame f = frame_with_image(320, 240);
const std::size_t img = 320u * 240u * 3u;
SceneFrame sf; sf.source = f;
AlignedSceneFrame af; af.source = f;
EmbeddedSceneFrame ef; ef.source = f;
TrackedSceneFrame tf; tf.source = f;
MatchedSceneFrame mf; mf.source = f;
CHECK(kpn::ChannelDataSize<SceneFrame>::bytes(sf) >= img);
CHECK(kpn::ChannelDataSize<AlignedSceneFrame>::bytes(af) >= img);
CHECK(kpn::ChannelDataSize<EmbeddedSceneFrame>::bytes(ef) >= img);
CHECK(kpn::ChannelDataSize<TrackedSceneFrame>::bytes(tf) >= img);
CHECK(kpn::ChannelDataSize<MatchedSceneFrame>::bytes(mf) >= img);
// SceneAnnotation carries no source frame — only the actors it identified,
// each with its own crop.
SceneAnnotation sa;
sa.visible_actors.push_back({});
sa.visible_actors.back().crop = cv::Mat(112, 112, CV_8UC3, cv::Scalar(0, 0, 0));
CHECK(kpn::ChannelDataSize<SceneAnnotation>::bytes(sa) >= 112u * 112u * 3u);
}
TEST_CASE("a shared image is charged to each message holding it", "[channel_bytes]") {
// cv::Mat is reference-counted, so a frame referenced from several messages
// is counted once per reference. The sum is an upper bound on distinct
// bytes, and the right bound for "what would this channel keep alive if
// nothing else held it" — which is the question a capacity answers.
const Frame f = frame_with_image(320, 240);
SceneFrame a; a.source = f;
SceneFrame b; b.source = f; // shares the same pixel buffer
CHECK(kpn::ChannelDataSize<SceneFrame>::bytes(a)
== kpn::ChannelDataSize<SceneFrame>::bytes(b));
}
tests/test_embedding_dump.cpp
-169
View File
@@ -1,169 +0,0 @@
// TRACES: AR-028 | VR-001 | UT-139, UT-140, UT-141 | SR-002
//
// The other half of AR-028: the quality vector has to *survive into the dump*.
// Measuring it at inference and then leaving it in a struct that dies at the
// EmbeddedSceneFrame channel would satisfy the letter of "assessed" and none of
// the point — VR-012 sets its knees from recorded data, and what the dump does
// not carry cannot be re-litigated without re-running video on a GPU.
//
// Tier T2, but cheap: EmbeddingDumpFunc is a sink, so it can be driven directly
// with hand-built frames. No model, no video, no gallery — the embedder stamp
// tolerates an unset model path (GR-004 records it as unverifiable).
#include <catch2/catch_test_macros.hpp>
#include <catch2/matchers/catch_matchers_floating_point.hpp>
#include "config.hpp"
#include "nodes/embedding_dump_node.hpp"
#include "types.hpp"
#include <H5Cpp.h>
#include <atomic>
#include <cstdio>
#include <filesystem>
#include <string>
#include <vector>
using Catch::Matchers::WithinAbs;
namespace {
namespace fs = std::filesystem;
// Removes the file on scope exit so a failing assertion cannot leave the next
// run reading a stale dump.
struct TempDump {
fs::path path;
explicit TempDump(const char* stem)
: path(fs::temp_directory_path() / (std::string("sae_") + stem + ".h5")) {
std::remove(path.c_str());
}
~TempDump() { std::error_code ec; fs::remove(path, ec); }
};
EmbeddedSceneFrame frame_with(double ts, const std::vector<std::pair<float, float>>& quality) {
EmbeddedSceneFrame ef;
ef.source.timestamp_sec = ts;
ef.source.frame_idx = static_cast<int64_t>(ts * 5.0);
for (const auto& [sharpness, residual] : quality) {
DetectedFace f;
f.bbox = cv::Rect2f(10.f, 20.f, 60.f, 60.f);
f.confidence = 0.8f;
f.sharpness = sharpness;
f.alignment_residual = residual;
ef.faces.push_back(f);
Embedding e{};
e[0] = 1.f;
ef.embeddings.push_back(e);
}
return ef;
}
EmbeddedSceneFrame eof_frame() {
EmbeddedSceneFrame ef;
ef.source.eof = true;
return ef;
}
std::vector<float> read_face_col(const H5::H5File& f, const char* name) {
H5::DataSet ds = f.openDataSet(std::string("faces/") + name);
hsize_t n = 0;
ds.getSpace().getSimpleExtentDims(&n, nullptr);
std::vector<float> out(n);
if (n) ds.read(out.data(), H5::PredType::NATIVE_FLOAT);
return out;
}
int read_schema_version(const H5::H5File& f) {
int v = 0;
f.openAttribute("schema_version").read(H5::PredType::NATIVE_INT, &v);
return v;
}
} // namespace
TEST_CASE("the quality vector survives into the dump", "[dump][AR-028][UT-139]") {
TempDump tmp("quality_roundtrip");
Config cfg;
cfg.dump_embeddings_path = tmp.path.string();
cfg.movie_path = "synthetic";
cfg.sample_fps = 5.f;
std::atomic<bool> done{false};
{
EmbeddingDumpFunc dump(cfg, done);
dump(frame_with(0.0, {{3.25f, 0.75f}, {0.5f, 4.5f}}));
dump(frame_with(0.2, {})); // a frame with no faces
dump(frame_with(0.4, {{12.0f, 0.0f}}));
dump(eof_frame());
}
REQUIRE(done.load());
REQUIRE(fs::exists(tmp.path));
H5::H5File f(tmp.path.string(), H5F_ACC_RDONLY);
const std::vector<float> sharp = read_face_col(f, "sharpness");
const std::vector<float> resid = read_face_col(f, "alignment_residual");
const std::vector<float> conf = read_face_col(f, "confidence");
// Parallel to every other per-face array, so a consumer can index the
// quality of face i with the same slice it uses for the embedding.
REQUIRE(sharp.size() == conf.size());
REQUIRE(resid.size() == conf.size());
REQUIRE(sharp.size() == 3);
CHECK_THAT(sharp[0], WithinAbs(3.25f, 1e-6f));
CHECK_THAT(sharp[1], WithinAbs(0.50f, 1e-6f));
CHECK_THAT(sharp[2], WithinAbs(12.0f, 1e-6f));
CHECK_THAT(resid[0], WithinAbs(0.75f, 1e-6f));
CHECK_THAT(resid[1], WithinAbs(4.50f, 1e-6f));
CHECK_THAT(resid[2], WithinAbs(0.00f, 1e-6f));
}
TEST_CASE("a dump carrying the quality vector announces itself as v2", "[dump][AR-028][UT-140]") {
// The bump is not for readers — they check for the datasets by name, and a
// v1 dump still replays. It is so a consumer of the vector can tell "these
// faces were never scored" from "these faces scored zero", which is not
// recoverable from the arrays. Same reason scene_detect is an attribute.
TempDump tmp("quality_version");
Config cfg;
cfg.dump_embeddings_path = tmp.path.string();
cfg.movie_path = "synthetic";
std::atomic<bool> done{false};
{
EmbeddingDumpFunc dump(cfg, done);
dump(frame_with(0.0, {{1.f, 1.f}}));
dump(eof_frame());
}
H5::H5File f(tmp.path.string(), H5F_ACC_RDONLY);
CHECK(read_schema_version(f) == 2);
}
TEST_CASE("an unscored face keeps its sentinel through the dump", "[dump][AR-028][UT-141]") {
// The aligner admits no unscored face, so this state should be unreachable.
// The dump still must not clamp it: -1 is how a future path that skipped
// scoring would be caught, and rewriting it to 0 would hide that path behind
// a legitimate-looking "featureless crop" reading.
TempDump tmp("quality_sentinel");
Config cfg;
cfg.dump_embeddings_path = tmp.path.string();
cfg.movie_path = "synthetic";
std::atomic<bool> done{false};
{
EmbeddingDumpFunc dump(cfg, done);
dump(frame_with(0.0, {{-1.f, -1.f}}));
dump(eof_frame());
}
H5::H5File f(tmp.path.string(), H5F_ACC_RDONLY);
CHECK(read_face_col(f, "sharpness")[0] < 0.f);
CHECK(read_face_col(f, "alignment_residual")[0] < 0.f);
}
tests/test_face_detector_node.cpp
-114
View File
@@ -1,114 +0,0 @@
// AR-002 — minimum face size, in original video resolution.
//
// TRACES: AR-002 | SR-002 | UT-002
//
// Tier T1 here, T2 in test_replay_fixtures.cpp. The requirement is arithmetic on
// bounding boxes, so the two edge cases that matter — a face sitting exactly on
// the threshold, and the same face seen through a downscaled decode — are
// reachable without a detector, a model or a GPU. What the fixture check adds is
// that the rule was actually applied on the way to a dump; what this adds is that
// it is applied *correctly*, which no real dump can demonstrate because real
// footage does not contain a 39.999 px face on demand.
//
// FaceDetectorFunc is never constructed: its constructor loads SCRFD. Only the
// static rule is called, so make_face_detector() is never odr-used and nothing
// here needs a backend.
#include <catch2/catch_test_macros.hpp>
#include "nodes/face_detector_node.hpp"
#include "types.hpp"
#include <vector>
namespace {
DetectedFace box(float w, float h) {
DetectedFace f;
f.bbox = cv::Rect2f(10.f, 10.f, w, h);
f.confidence = 0.9f;
return f;
}
// Sizes the rule kept, in the order given.
std::vector<float> surviving_widths(std::vector<DetectedFace> faces,
float min_face_px, float bbox_upscale) {
FaceDetectorFunc::drop_undersized(faces, min_face_px, bbox_upscale);
std::vector<float> out;
for (const auto& f : faces) out.push_back(f.bbox.width);
return out;
}
constexpr float kMin = 40.f; // Config::min_face_px default, and AR-002's number
} // namespace
// ── Exactly at the threshold ─────────────────────────────────────────────────
// The boundary case is the whole content of a minimum: "40x40" has to mean 40 is
// admissible, or the requirement says 41.
TEST_CASE("a face exactly at the minimum is kept", "[detector][AR-002]") {
CHECK(surviving_widths({box(kMin, kMin)}, kMin, 1.f).size() == 1);
}
TEST_CASE("a face one tenth of a pixel under the minimum is dropped",
"[detector][AR-002]") {
CHECK(surviving_widths({box(39.9f, 100.f)}, kMin, 1.f).empty());
CHECK(surviving_widths({box(100.f, 39.9f)}, kMin, 1.f).empty());
}
TEST_CASE("both sides must clear the minimum, not the larger one",
"[detector][AR-002]") {
// A wide, short box has enough pixels and is still unusable: ArcFace
// alignment needs both dimensions. Area would admit this; the rule must not.
CHECK(surviving_widths({box(400.f, 20.f)}, kMin, 1.f).empty());
}
TEST_CASE("the filter is a filter, not a reordering", "[detector][AR-002]") {
auto kept = surviving_widths(
{box(80.f, 80.f), box(10.f, 10.f), box(60.f, 60.f), box(39.f, 39.f)},
kMin, 1.f);
REQUIRE(kept.size() == 2);
// Order is load-bearing downstream (AR-003's largest-first sort tie-breaks on
// it, and the Hungarian solver tie-breaks on index) — erase-remove must not
// shuffle the survivors.
CHECK(kept[0] == 80.f);
CHECK(kept[1] == 60.f);
}
// ── The dense_scale interaction the requirement exists for ───────────────────
// dense_scale 0.5 halves the decoded frame, so the detector reports a 40 px face
// as 20 px. If the threshold were applied to those numbers, turning on a
// throughput knob would silently double the minimum face size the pipeline
// accepts — a recall change with no line in the config to explain it. AR-002
// pins the minimum to the ORIGINAL resolution instead.
TEST_CASE("at dense_scale 0.5 the cutoff stays 40 px of original footage",
"[detector][AR-002]") {
constexpr float kUpscale = 2.f; // frame_source_node: 1 / dense_scale
// 20 px in downscaled space is exactly 40 px of original footage: kept.
CHECK(surviving_widths({box(20.f, 20.f)}, kMin, kUpscale).size() == 1);
// 19.9 px downscaled is 39.8 px original: dropped.
CHECK(surviving_widths({box(19.9f, 19.9f)}, kMin, kUpscale).empty());
// And the interaction stated as one claim: a face of a given original size is
// admitted or refused identically whether or not the decode was downscaled.
for (float original : {30.f, 39.f, 40.f, 41.f, 80.f}) {
INFO("original size " << original);
const bool full = !surviving_widths({box(original, original)},
kMin, 1.f).empty();
const bool dense = !surviving_widths({box(original / kUpscale,
original / kUpscale)},
kMin, kUpscale).empty();
CHECK(full == dense);
CHECK(full == (original >= kMin));
}
}
TEST_CASE("an absent or degenerate upscale falls back to the raw threshold",
"[detector][AR-002]") {
// bbox_upscale is 1 on every non-dense frame; 0 would mean the frame source
// never set it. Dividing by that would reject every face in the film, which
// is a failure worth not having.
CHECK(surviving_widths({box(kMin, kMin)}, kMin, 0.f).size() == 1);
CHECK(surviving_widths({box(39.f, 39.f)}, kMin, 0.f).empty());
}
tests/test_face_tracker.cpp
+6 -10
View File
@@ -1,17 +1,13 @@
// TRACES: AR-007, AR-008 | SR-002
//
// Unit tests for FaceTrackerFunc (nodes/face_tracker_node.hpp): frame-to-frame
// track linking and, crucially, cross-cut re-association. Pure, GPU-free,
// model-free — drives the node's operator() with hand-built EmbeddedSceneFrames
// and inspects the emitted track_ids.
//
// The behaviour under test: there is one track pool keyed on `last_seen`
// (AR-008), so a face lost across a camera-angle change (Frame::is_cut) is an
// ordinary association candidate rather than a parked track needing a revival
// path — the raw-cosine `cut_revive_sim` that guarded that path is retired
// (AR-024). On a cut the association weight drops to embedding-only (AR-007),
// and IoU is deliberately driven to 0 across the cut (boxes moved) so only the
// embedding path can re-link — exactly the scenario a cut creates.
// The behaviour under test: on a camera-angle change (Frame::is_cut) the tracker
// parks its tracks instead of destroying them, and revives a parked track_id
// when a post-cut detection's raw last-frame-embedding cosine similarity clears
// cut_revive_sim. IoU is deliberately driven to 0 across the cut (boxes moved) so
// only the embedding path can re-link — exactly the scenario a cut creates.
#include <catch2/catch_test_macros.hpp>
#include "config.hpp"
@@ -73,7 +69,7 @@ struct Rig {
: reg(std::make_shared<TrackRegistry>(
[extinction] {
TrackRegistry::Config c;
c.track_extinction_sec = extinction;
c.extinction_sec = extinction;
return c;
}(),
EvidenceDiscounter([](float cos) { return std::max(0.f, cos); })))
tests/test_face_utils.cpp
+3 -275
View File
@@ -1,26 +1,18 @@
// TRACES: AR-005, AR-028, AR-029, AR-030 | UT-130, UT-131, UT-132, UT-133, UT-134, UT-135, UT-136, UT-137, UT-138 | SR-002
// TRACES: AR-005, AR-030 | SR-002
//
// Unit tests for the geometric/numeric helpers in types.hpp and face_utils.hpp:
// cosine_similarity, the ArcFace 5-point alignment transform, and the two
// measured axes of the AR-028 quality vector — the alignment residual AR-030
// reads as visibility, and the normalised Laplacian variance AR-029 reads as
// sharpness. The aligner node is exercised here too, since it is the unit that
// fills the vector in. GPU-free, model-free.
// cosine_similarity, the ArcFace 5-point alignment transform, and the alignment
// residual that AR-030 reads as its visibility measure. GPU-free, model-free.
#include <catch2/catch_test_macros.hpp>
#include <catch2/matchers/catch_matchers_floating_point.hpp>
#include "face_utils.hpp"
#include "nodes/face_aligner_node.hpp"
#include "types.hpp"
#include <algorithm>
#include <array>
#include <cmath>
#include <cstdint>
#include <limits>
using Catch::Matchers::WithinAbs;
using Catch::Matchers::WithinRel;
TEST_CASE("cosine_similarity of a unit vector with itself is 1", "[types]") {
std::array<float, 512> raw{};
@@ -187,267 +179,3 @@ TEST_CASE("degenerate landmarks report not-ok rather than a residual", "[face_ut
CHECK_FALSE(a.ok);
CHECK(a.M.empty());
}
// ── AR-029: normalised Laplacian variance as a sharpness measure ──────────────
// As with AR-030 above, these assert the *properties* the measure is relied on
// for rather than magic values: no threshold is set here or anywhere else, so a
// number that drifted with the OpenCV version would still be usable — a number
// that stopped falling with blur, or started tracking exposure, would not.
namespace {
// Pink noise: broadband, but with a 1/f spectrum, so most of the energy sits at
// low frequency the way it does in a photograph. An LCG rather than cv::randu so
// the ladder is identical on every machine and every OpenCV build.
//
// **The 1/f part is load-bearing, not decoration.** On a flat-spectrum texture
// (raw white noise) the Gaussian ladder still falls, but the motion-blur ladder
// *rises* — 80.1 → 90.0 across the same kernel lengths used below. That is not a
// bug in the measure, it is what a normalised measure must do on such an input:
// a horizontal smear takes energy out of the numerator and the denominator
// together, and what survives is vertical detail that really is just as fine.
// Real crops have the low-frequency mass that keeps the denominator steady while
// the numerator falls. See the second hazard note on crop_sharpness().
cv::Mat pink(int size, uint32_t seed = 12345u) {
cv::Mat white(size, size, CV_32F);
uint32_t s = seed;
for (int y = 0; y < size; ++y)
for (int x = 0; x < size; ++x) {
s = s * 1664525u + 1013904223u;
white.at<float>(y, x) = float((s >> 16) & 0xFFFF) / 65535.f - 0.5f;
}
// Octaves weighted 1/f. The sigma=0 band keeps genuine per-pixel detail in,
// so the top of the blur ladder is a sharp image rather than an already-soft
// one.
cv::Mat acc = cv::Mat::zeros(size, size, CV_32F);
const double sigma[] = {0.0, 1.0, 2.0, 4.0, 8.0};
const double weight[] = {1.0, 2.0, 4.0, 8.0, 16.0};
for (int k = 0; k < 5; ++k) {
cv::Mat band;
if (sigma[k] <= 0.0) band = white.clone();
else cv::GaussianBlur(white, band, {0, 0}, sigma[k], sigma[k], cv::BORDER_REPLICATE);
acc += band * weight[k];
}
// Into [40, 215]: 8-bit like a real crop, with headroom at both ends so the
// contrast test can halve it without clipping.
double lo = 0, hi = 0;
cv::minMaxLoc(acc, &lo, &hi);
const double scale = 175.0 / (hi - lo);
cv::Mat out;
acc.convertTo(out, CV_8U, scale, 40.0 - lo * scale);
return out;
}
// One master pattern, resampled. So "the same face at 40 px and at 400 px" is
// literally the same image at two resolutions, and a test about source
// resolution is not accidentally a test about two different textures.
// INTER_AREA because area-averaging is what a sensor does when it images the
// same subject onto fewer pixels.
cv::Mat texture(int size) {
static const cv::Mat master = pink(448);
if (size == master.cols) return master;
cv::Mat out;
cv::resize(master, out, {size, size}, 0, 0, cv::INTER_AREA);
return out;
}
cv::Mat gaussian(const cv::Mat& in, double sigma) {
if (sigma <= 0.0) return in.clone();
cv::Mat out;
cv::GaussianBlur(in, out, {0, 0}, sigma, sigma, cv::BORDER_REPLICATE);
return out;
}
// Horizontal box smear — motion blur, which is anisotropic and so attenuates
// only one axis of the spectrum. A measure tuned to the isotropic case can
// miss it.
cv::Mat motion(const cv::Mat& in, int len) {
if (len <= 1) return in.clone();
const cv::Mat k(1, len, CV_32F, cv::Scalar(1.0 / len));
cv::Mat out;
cv::filter2D(in, out, -1, k, {-1, -1}, 0, cv::BORDER_REPLICATE);
return out;
}
// The pipeline reaches 112×112 through warpAffine's INTER_LINEAR; resize with
// the same interpolation so a test about source resolution is not really a test
// about which resampler was used.
cv::Mat to_crop(const cv::Mat& in) {
cv::Mat out;
cv::resize(in, out, {112, 112}, 0, 0, cv::INTER_LINEAR);
return out;
}
} // namespace
TEST_CASE("sharpness falls monotonically along a Gaussian blur ladder", "[face_utils][AR-029][UT-130]") {
const cv::Mat src = texture(112);
float prev = std::numeric_limits<float>::infinity();
for (double sigma : {0.0, 0.6, 1.0, 1.6, 2.5, 4.0}) {
const float s = crop_sharpness(gaussian(src, sigma));
CHECK(s < prev);
CHECK(s > 0.f);
prev = s;
}
}
TEST_CASE("sharpness falls monotonically under motion blur too", "[face_utils][AR-029][UT-131]") {
// Motion blur is the failure mode that leaves the bounding box looking
// perfectly healthy, so it is the one the measure exists for.
const cv::Mat src = texture(112);
float prev = std::numeric_limits<float>::infinity();
for (int len : {1, 3, 5, 9, 15}) {
const float s = crop_sharpness(motion(src, len));
CHECK(s < prev);
CHECK(s > 0.f);
prev = s;
}
}
TEST_CASE("contrast does not leak into sharpness", "[face_utils][AR-029][UT-132]") {
// The normalisation that makes the axis mean the same thing in a dim scene
// and a bright one. Without it VR-012 would locate a different knee per
// film — a magic number wearing a measurement's clothes (AR-024).
const cv::Mat src = texture(112);
cv::Mat dim;
src.convertTo(dim, CV_8U, 0.5, 64.0); // half contrast, re-centred, no clipping
const float a = crop_sharpness(src);
const float b = crop_sharpness(dim);
REQUIRE(a > 0.f);
CHECK_THAT(b, WithinRel(a, 0.03f));
}
TEST_CASE("the contrast invariance is exact, and 8-bit sampling is what bends it",
"[face_utils][AR-029]") {
// Worth separating because the two have different consequences. The
// algebra is exact — scaling I by α scales the Laplacian by α, so both
// variances scale by α² and cancel — which is why halving a float crop
// changes nothing at all.
//
// What deviates is the 8-bit *round trip*: halving the contrast of a stored
// crop throws away a bit of dynamic range, and the quantisation floor it
// leaves behind is broadband, so it lands almost entirely in the numerator.
// The effect scales with how little signal is left to compete with it —
// measured on this texture, a half-contrast copy reads 0.9% high when sharp,
// 24% high at sigma 1.2 and 148% high at sigma 2.5.
//
// So: a dim *and* soft crop reads sharper than it is, and that is the corner
// of the axis VR-012 has to put a knee in. Asserted here rather than left as
// a comment, because "the measure is contrast-invariant" is the kind of claim
// that gets repeated without its precondition.
cv::Mat src;
gaussian(texture(112), 1.2).convertTo(src, CV_32F);
const cv::Mat half = src * 0.5 + 64.0;
const float a = crop_sharpness(src);
const float b = crop_sharpness(half);
REQUIRE(a > 0.f);
CHECK_THAT(b, WithinRel(a, 1e-5f));
}
TEST_CASE("a small sharp face outscores a large soft one", "[face_utils][AR-029][UT-134]") {
// The register's named edge case: "size must not leak into this axis". What
// that means operationally is that the measure is not a monotone function of
// source face size — it reports the detail present in the embedder's input,
// so the ordering can and must invert when the large face is the blurred one.
//
// Small sharp: 40 px of real detail, upsampled 2.8x → finest scale ~2.8 crop px.
// Large soft: 400 px blurred at sigma 20, downsampled 3.57x → ~5.6 crop px.
const float small_sharp = crop_sharpness(to_crop(texture(40)));
const float large_soft = crop_sharpness(to_crop(gaussian(texture(400), 20.0)));
CHECK(small_sharp > large_soft);
}
TEST_CASE("a flat crop scores zero rather than dividing by zero", "[face_utils][AR-029][UT-135]") {
const cv::Mat flat(112, 112, CV_8UC3, cv::Scalar(90, 90, 90));
const float s = crop_sharpness(flat);
CHECK(std::isfinite(s));
CHECK_THAT(s, WithinAbs(0.0f, 1e-6f));
}
TEST_CASE("an empty crop is unscored, not zero", "[face_utils][AR-029][UT-136]") {
// -1 says "nothing measured this"; 0 says "measured, and there was no
// detail". Collapsing them would put unscored faces at the bottom of the
// quality axis, where VR-012 would read them as the blurriest in the film.
CHECK(crop_sharpness(cv::Mat()) < 0.f);
}
// ── AR-028: the aligner fills the vector, and loses nothing quietly ───────────
namespace {
// A face at `centre` in an image with enough texture for sharpness to be a real
// number rather than the flat-crop zero.
std::array<cv::Point2f, 5> face_at(cv::Point2f centre, float scale) {
std::array<cv::Point2f, 5> lm;
for (int i = 0; i < 5; ++i)
lm[i] = {centre.x + (kArcFaceRef[i][0] - 56.f) * scale,
centre.y + (kArcFaceRef[i][1] - 56.f) * scale};
return lm;
}
cv::Mat textured_frame(int w, int h) {
cv::Mat gray = texture(std::max(w, h));
cv::Mat bgr;
cv::cvtColor(gray(cv::Rect(0, 0, w, h)), bgr, cv::COLOR_GRAY2BGR);
return bgr;
}
} // namespace
TEST_CASE("every face the aligner admits carries a full quality vector", "[face_utils][AR-028][UT-137]") {
SceneFrame sf;
sf.source.image = textured_frame(400, 300);
for (auto c : {cv::Point2f{120.f, 100.f}, cv::Point2f{280.f, 190.f}}) {
DetectedFace f;
f.landmarks = face_at(c, 1.2f);
f.bbox = cv::Rect2f(c.x - 60.f, c.y - 60.f, 120.f, 120.f);
f.confidence = 0.9f;
sf.faces.push_back(f);
}
FaceAlignerFunc aligner;
const AlignedSceneFrame out = aligner(std::move(sf));
REQUIRE(out.faces.size() == 2);
for (const auto& f : out.faces) {
// Not "is it good quality" — that is VR-012's to decide. Only that the
// sentinel is gone, so no embedding reaches the matcher unscored.
CHECK(f.sharpness >= 0.f);
CHECK(f.alignment_residual >= 0.f);
}
CHECK(aligner.scored() == 2);
CHECK(aligner.degenerate() == 0);
}
TEST_CASE("a degenerate detection is counted, not silently vanished", "[face_utils][AR-028][UT-138]") {
// It cannot be scored — there is no crop and no fit to score — so it is
// dropped. The requirement is that the drop leaves a trace: without the
// tally, a detector emitting unusable landmark sets produces a dump that
// looks exactly like footage with fewer faces in it.
SceneFrame sf;
sf.source.image = textured_frame(400, 300);
DetectedFace good;
good.landmarks = face_at({150.f, 140.f}, 1.2f);
good.confidence = 0.9f;
sf.faces.push_back(good);
DetectedFace degenerate;
for (auto& p : degenerate.landmarks) p = {200.f, 200.f};
degenerate.confidence = 0.9f;
sf.faces.push_back(degenerate);
FaceAlignerFunc aligner;
const AlignedSceneFrame out = aligner(std::move(sf));
CHECK(out.faces.size() == 1);
CHECK(out.crops.size() == 1);
CHECK(aligner.scored() == 1);
CHECK(aligner.degenerate() == 1);
}
tests/test_quality.cpp
+302
View File
@@ -0,0 +1,302 @@
// TRACES: AR-029 | SR-002
//
// T1 for the AR-029 sharpness candidates: the properties that have to hold
// before a study is allowed to pick between them. GPU-free, model-free.
//
// The register's acceptance criterion is "synthetic blur ladder ->
// monotonically falling sharpness; Gaussian vs motion blur; small sharp face vs
// large soft one — size must not leak into this axis". The last clause needs
// care, and the tests below split it in two:
//
// - What must NOT leak is *geometric* scale. The measure is taken in the
// canonical frame, so changing how big the face was in the source while
// preserving its detail must not move the score. That is structural: the
// window is fixed at 64x64 canonical px.
// - What DOES legitimately move the score is lost *detail*. A face that was
// 40 px before being warped up to 112 really does carry less
// high-frequency content than one that was 400 px, and a measure blind to
// that would be blind to the thing it exists to catch.
//
// So "size must not leak" cannot mean "invariant to the source face size", and
// the ladder test below asserts the opposite on purpose. What it buys is that
// the overlap with AR-002 is a recorded property with a test naming it, rather
// than a surprise VR-012 discovers when the two axes turn out to be correlated.
#include <catch2/catch_test_macros.hpp>
#include <catch2/matchers/catch_matchers_floating_point.hpp>
#include "quality.hpp"
#include "types.hpp" // kArcFaceRef, for the window-placement test
#include <algorithm>
#include <cmath>
#include <utility>
#include <vector>
using Catch::Matchers::WithinAbs;
using Catch::Matchers::WithinRel;
namespace {
// A deterministic 112x112 stand-in for a face crop.
//
// **Broadband, not a sum of a few sinusoids.** An earlier version of this
// fixture used three discrete spatial frequencies, and the resampling ladder
// below was non-monotone for hf_energy_ratio because of it: a period-7
// component downsampled to 32 px lands exactly at Nyquist and aliases, so the
// ratio rose at one rung instead of falling. That is a property of a
// three-tone test pattern meeting a resampler, not of the measure or of any
// face — a real crop has energy spread across the band, where such a
// resonance averages out. Deterministic value noise, smoothed to give the
// roughly 1/f falloff of a photograph, exercises the whole band at once.
//
// Mid-grey base with bounded amplitude, so scaling the contrast in the tests
// below does not clip.
cv::Mat synthetic_crop() {
// Fixed LCG rather than cv::randu: the suite must not depend on OpenCV's
// RNG state, which other tests share.
uint32_t seed = 0x5eed1234u;
auto next = [&seed] {
seed = seed * 1664525u + 1013904223u;
return (seed >> 16) & 0xffffu;
};
cv::Mat noise(112, 112, CV_32F);
for (int y = 0; y < 112; ++y)
for (int x = 0; x < 112; ++x)
noise.at<float>(y, x) = float(next()) / 65535.f - 0.5f;
// Mild smoothing: white noise is flat to Nyquist, which no lens produces
// and which would make the sharpest rung of every ladder unrealistic.
cv::Mat smooth;
cv::GaussianBlur(noise, smooth, cv::Size(0, 0), 0.8);
cv::normalize(smooth, smooth, -1.0, 1.0, cv::NORM_MINMAX);
cv::Mat img(112, 112, CV_8UC3);
for (int y = 0; y < 112; ++y) {
for (int x = 0; x < 112; ++x) {
double v = 128.0 + 70.0 * smooth.at<float>(y, x);
const auto b = static_cast<uchar>(std::clamp(v, 0.0, 255.0));
img.at<cv::Vec3b>(y, x) = {b, b, b};
}
}
return img;
}
cv::Mat gaussian(const cv::Mat& src, double sigma) {
cv::Mat out;
cv::GaussianBlur(src, out, cv::Size(0, 0), sigma, sigma);
return out;
}
// Horizontal box blur — the camera-pan case, and the one an isotropic measure
// could in principle miss.
cv::Mat motion(const cv::Mat& src, int len) {
cv::Mat kernel = cv::Mat::zeros(1, len, CV_32F);
kernel.setTo(1.0f / len);
cv::Mat out;
cv::filter2D(src, out, -1, kernel);
return out;
}
// Throw away detail a face detected at size x size never had, then warp back up
// to the 112x112 the embedder is fed — the VR-005 degradation.
cv::Mat rescale(const cv::Mat& src, int size) {
if (size == 112) return src.clone();
cv::Mat small, out;
cv::resize(src, small, {size, size}, 0, 0, cv::INTER_AREA);
cv::resize(small, out, {112, 112}, 0, 0, cv::INTER_LINEAR);
return out;
}
std::vector<float> field(const std::vector<SharpnessScores>& s,
float SharpnessScores::* m) {
std::vector<float> v;
v.reserve(s.size());
for (const auto& x : s) v.push_back(x.*m);
return v;
}
void check_strictly_falling(const std::vector<float>& v, const char* what) {
INFO(what);
for (size_t i = 1; i < v.size(); ++i) {
INFO("step " << i << ": " << v[i - 1] << " -> " << v[i]);
CHECK(v[i] < v[i - 1]);
}
}
const std::vector<std::pair<const char*, float SharpnessScores::*>> kMeasures{
{"var_laplacian", &SharpnessScores::var_laplacian},
{"norm_var_laplacian", &SharpnessScores::norm_var_laplacian},
{"tenengrad", &SharpnessScores::tenengrad},
{"hf_energy_ratio", &SharpnessScores::hf_energy_ratio},
{"dir_min_tenengrad", &SharpnessScores::dir_min_tenengrad},
};
} // namespace
TEST_CASE("every candidate falls monotonically along a Gaussian blur ladder",
"[quality][AR-029]") {
const cv::Mat base = synthetic_crop();
std::vector<SharpnessScores> ladder;
for (double sigma : {0.0, 0.5, 1.0, 1.5, 2.0, 3.0})
ladder.push_back(assess_sharpness(sigma == 0.0 ? base : gaussian(base, sigma)));
for (const auto& [name, m] : kMeasures) {
REQUIRE(ladder.front().ok);
check_strictly_falling(field(ladder, m), name);
}
}
TEST_CASE("only the absolute and directional measures survive motion blur",
"[quality][AR-029]") {
// Motion blur is the commonest way a film frame is unusable, and it is
// where the candidates separate. A horizontal smear destroys horizontal
// detail and leaves vertical detail untouched, so what a measure does here
// depends on whether it can be fooled by the surviving axis.
const cv::Mat base = synthetic_crop();
std::vector<SharpnessScores> ladder{assess_sharpness(base)};
for (int len : {3, 5, 9, 15, 21})
ladder.push_back(assess_sharpness(motion(base, len)));
// Total gradient/Laplacian energy keeps falling: nothing replaces what the
// smear removed.
check_strictly_falling(field(ladder, &SharpnessScores::var_laplacian),
"var_laplacian");
check_strictly_falling(field(ladder, &SharpnessScores::tenengrad),
"tenengrad");
// The fix for the two below: low-frequency denominator, and the worse of
// the two axes rather than their sum.
check_strictly_falling(field(ladder, &SharpnessScores::dir_min_tenengrad),
"dir_min_tenengrad");
// The disqualifying behaviour, pinned rather than hidden. Both measures
// normalise by a quantity that contains the detail they are measuring, so
// once the horizontal band is gone the quotient climbs back toward its
// unblurred value: each is U-shaped in blur length, and a single score
// maps to two very different amounts of blur. A 21 px smear scores about
// as sharp as a 3 px one.
for (const auto& [name, m] : {
std::pair{"norm_var_laplacian", &SharpnessScores::norm_var_laplacian},
std::pair{"hf_energy_ratio", &SharpnessScores::hf_energy_ratio}}) {
const std::vector<float> v = field(ladder, m);
INFO(name);
const auto trough = std::min_element(v.begin(), v.end());
CHECK(trough != v.begin()); // it does fall at first …
CHECK(trough != v.end() - 1); // … then turns back up
CHECK(v.back() > 0.8f * v[1]); // recovering most of one rung
}
}
TEST_CASE("sharpness falls under downscale-upscale as well as under blur",
"[quality][AR-029]") {
// The overlap with AR-002, asserted rather than assumed. Losing resolution
// and losing focus are the same loss of high-frequency content, so every
// candidate reads a small upscaled face as less sharp. VR-012's joint
// size x sigma grid decides whether that makes a sharpness discount a
// double-count against the size gate, or whether the two axes carry
// separable information.
const cv::Mat base = synthetic_crop();
std::vector<SharpnessScores> ladder;
for (int size : {112, 64, 48, 32, 24, 16})
ladder.push_back(assess_sharpness(rescale(base, size)));
for (const auto& [name, m] : kMeasures)
check_strictly_falling(field(ladder, m), name);
}
TEST_CASE("the ratio measures are contrast-free and the raw ones are not",
"[quality][AR-029]") {
// The confound that decides the bake-off. A gallery drawn from thousands of
// cameras, lighting setups and JPEG pipelines varies enormously in
// contrast, and a measure that reads a low-contrast sharp face as blurred
// would discount it for the photographer's choices rather than for anything
// the embedder cares about.
const cv::Mat base = synthetic_crop();
// Halve the contrast about mid-grey, leaving spatial structure untouched.
cv::Mat low;
base.convertTo(low, CV_8UC3, 0.5, 64.0);
const auto s_hi = assess_sharpness(base);
const auto s_lo = assess_sharpness(low);
REQUIRE(s_hi.ok);
REQUIRE(s_lo.ok);
// Invariant by construction: both are ratios in which the contrast factor
// cancels.
CHECK_THAT(s_lo.norm_var_laplacian,
WithinRel(s_hi.norm_var_laplacian, 0.02f));
CHECK_THAT(s_lo.hf_energy_ratio, WithinRel(s_hi.hf_energy_ratio, 0.02f));
// Not invariant: both scale with the square of the contrast factor, so
// halving the contrast quarters them. This is the disqualifying behaviour,
// pinned so that a change making them contrast-free is a deliberate one.
CHECK_THAT(s_lo.var_laplacian, WithinRel(0.25f * s_hi.var_laplacian, 0.05f));
CHECK_THAT(s_lo.tenengrad, WithinRel(0.25f * s_hi.tenengrad, 0.05f));
}
TEST_CASE("brightness alone moves nothing", "[quality][AR-029]") {
const cv::Mat base = synthetic_crop();
cv::Mat bright;
base.convertTo(bright, CV_8UC3, 1.0, 20.0);
const auto a = assess_sharpness(base);
const auto b = assess_sharpness(bright);
for (const auto& [name, m] : kMeasures) {
INFO(name);
CHECK_THAT(b.*m, WithinRel(a.*m, 0.02f));
}
}
TEST_CASE("a flat crop is scored not-ok rather than given a number",
"[quality][AR-029]") {
// A face whose sharpness cannot be computed is a fact to record, not an
// absence — the same rule AR-030 follows for degenerate landmarks.
const cv::Mat flat(112, 112, CV_8UC3, cv::Scalar(128, 128, 128));
const auto s = assess_sharpness(flat);
CHECK_FALSE(s.ok);
for (const auto& [name, m] : kMeasures) {
INFO(name);
CHECK_THAT(s.*m, WithinAbs(0.0f, 1e-6f));
CHECK_FALSE(std::isnan(s.*m));
}
}
TEST_CASE("a crop smaller than the measurement window is scored not-ok",
"[quality][AR-029]") {
const cv::Mat small(64, 64, CV_8UC3, cv::Scalar(40, 90, 160));
CHECK_FALSE(assess_sharpness(small).ok);
CHECK_FALSE(assess_sharpness(cv::Mat()).ok);
}
TEST_CASE("the measurement window covers the face interior of the crop",
"[quality][AR-029]") {
// The landmarks the ArcFace template pins must all fall inside the window,
// or the measure is scoring background and hair rather than the face.
const cv::Rect w = sharpness_window();
CHECK(w.x >= 0);
CHECK(w.y >= 0);
CHECK(w.x + w.width <= 112);
CHECK(w.y + w.height <= 112);
for (int i = 0; i < 5; ++i) {
INFO("landmark " << i);
CHECK(w.contains(cv::Point(static_cast<int>(kArcFaceRef[i][0]),
static_cast<int>(kArcFaceRef[i][1]))));
}
}
TEST_CASE("a single-channel crop scores the same as its BGR equivalent",
"[quality][AR-029]") {
// The dump replays crops; nothing should depend on whether they arrived as
// three identical channels or one.
const cv::Mat base = synthetic_crop();
cv::Mat gray;
cv::cvtColor(base, gray, cv::COLOR_BGR2GRAY);
const auto a = assess_sharpness(base);
const auto b = assess_sharpness(gray);
for (const auto& [name, m] : kMeasures) {
INFO(name);
CHECK_THAT(b.*m, WithinRel(a.*m, 1e-3f));
}
}
tests/test_replay_fixtures.cpp
+67 -60
View File
@@ -1,6 +1,6 @@
// Replay tests — the real tracker and registry driven from committed fixtures.
//
// TRACES: AR-002, AR-004, AR-012, AR-013 | VR-001, VR-002 | IT-001
// TRACES: AR-004, AR-012, AR-013 | VR-001, VR-002 | IT-001
//
// Tier T2: composition, not units. The registry tests construct awkward states
// directly; these check that the pieces behave when wired together and fed real
@@ -27,7 +27,6 @@
#include <algorithm>
#include <cmath>
#include <limits>
#include <memory>
#include <string>
#include <vector>
@@ -45,8 +44,6 @@ struct Dump {
std::vector<Embedding> emb;
std::vector<float> bbox; // 4 per face
std::string embedder;
float min_face_px{0.f}; // AR-002, as the run was configured
float bbox_upscale{1.f}; // bbox × this = original resolution
std::size_t frames() const { return ts.size(); }
std::size_t faces() const { return emb.size(); }
@@ -100,17 +97,6 @@ Dump load(const std::string& path) {
H5::StrType vlen(H5::PredType::C_S1, H5T_VARIABLE);
f.openAttribute("embedder_model").read(vlen, d.embedder);
}
// AR-002: the threshold the run was configured with, and the scale its boxes
// are in. Read from the dump rather than assumed, so the check is against
// what this fixture was actually generated with — the hero clips predate the
// move to 40 px and were dumped at 32.
if (f.attrExists("min_face_px"))
f.openAttribute("min_face_px").read(H5::PredType::NATIVE_FLOAT,
&d.min_face_px);
if (f.attrExists("bbox_upscale"))
f.openAttribute("bbox_upscale").read(H5::PredType::NATIVE_FLOAT,
&d.bbox_upscale);
return d;
}
@@ -128,7 +114,7 @@ struct Replay {
Replay run(const Dump& d, double extinction = 10.0) {
Replay r;
TrackRegistry::Config rc;
rc.track_extinction_sec = extinction;
rc.extinction_sec = extinction;
auto cal = [](float cos) { return std::max(0.f, cos); };
auto reg = std::make_shared<TrackRegistry>(rc, EvidenceDiscounter(cal));
@@ -167,61 +153,60 @@ Replay run(const Dump& d, double extinction = 10.0) {
} // namespace
// ── AR-004 / VR-001 — the fixtures are intact and self-describing ────────────
TEST_CASE("superhero fixture is complete", "[replay][VR-001]") {
Dump d = load(fixture("superhero.h5"));
CHECK(d.frames() == 5128);
CHECK(d.faces() == 4307);
CHECK(d.embedder == "LVFace-B_Glint360K.onnx");
TEST_CASE("fixtures are complete and carry their embedder identity",
"[replay][AR-004][VR-001]") {
// Frame counts are exact rather than approximate. Before node outputs
// blocked on a full channel, generation lost most of a clip and what it
// lost depended on timing — these numbers could not have been asserted.
struct Expect { const char* file; std::size_t frames, faces; };
const Expect all[] = {
{"bali_13.h5", 385, 693},
{"bali_27.h5", 335, 335},
{"bali_28.h5", 345, 368},
{"bali_31.h5", 145, 203},
{"bali_46.h5", 385, 140},
};
int64_t running = 0;
for (std::size_t i = 0; i < d.frames(); ++i) {
REQUIRE(d.face_offset[i] == running);
running += d.face_count[i];
for (const auto& x : all) {
INFO(x.file);
Dump d = load(fixture(x.file));
CHECK(d.frames() == x.frames);
CHECK(d.faces() == x.faces);
CHECK(d.embedder == "LVFace-B_Glint360K.onnx");
// face_offset must be contiguous: a gap means faces went missing
// between frames, which no consumer could detect.
int64_t running = 0;
for (std::size_t i = 0; i < d.frames(); ++i) {
REQUIRE(d.face_offset[i] == running);
running += d.face_count[i];
}
CHECK(static_cast<std::size_t>(running) == d.faces());
}
CHECK(static_cast<std::size_t>(running) == d.faces());
}
// ── AR-002 — the size filter held, all the way to the dump ───────────────────
// The T1 arithmetic is in test_face_detector_node.cpp. This is the other half:
// that the rule was applied on real footage and nothing downstream of it let an
// undersized face back in.
TEST_CASE("no dumped face is below the configured minimum size",
"[replay][AR-002]") {
Dump d = load(fixture("superhero.h5"));
// A dump that did not record its threshold cannot be checked against one.
REQUIRE(d.min_face_px > 0.f);
REQUIRE(d.bbox_upscale > 0.f);
float smallest_side = std::numeric_limits<float>::max();
for (std::size_t i = 0; i < d.faces(); ++i) {
const float w = d.bbox[i * 4 + 2] * d.bbox_upscale; // original resolution
const float h = d.bbox[i * 4 + 3] * d.bbox_upscale;
REQUIRE(w >= d.min_face_px);
REQUIRE(h >= d.min_face_px);
smallest_side = std::min({smallest_side, w, h});
}
// And the filter was binding, not vacuously satisfied. 480x360 footage puts
// faces right on the cutoff, which is what makes this fixture worth checking:
// if the threshold stopped being applied the assertions above would still
// pass on a corpus of close-ups.
CHECK(smallest_side < d.min_face_px * 1.05f);
}
TEST_CASE("replaying the superhero fixture twice gives identical tracks",
"[replay][VR-002]") {
Dump d = load(fixture("superhero.h5"));
// ── VR-002 — replay is deterministic ─────────────────────────────────────────
TEST_CASE("replaying a fixture twice gives identical tracks", "[replay][VR-002]") {
// The property the whole fixture strategy rests on. If this fails, every
// golden output derived from a fixture is unreliable and the CI replay
// tier is worthless.
Dump d = load(fixture("bali_28.h5"));
Replay a = run(d);
Replay b = run(d);
REQUIRE(a.track_ids.size() == b.track_ids.size());
CHECK(a.track_ids == b.track_ids);
REQUIRE(a.claims.size() == b.claims.size());
for (std::size_t i = 0; i < a.claims.size(); ++i) {
CHECK(a.claims[i].first_seen == b.claims[i].first_seen);
CHECK(a.claims[i].last_seen == b.claims[i].last_seen);
}
}
// ── AR-012 / AR-013 — window invariants on real footage ──────────────────────
TEST_CASE("every face is assigned a track and every track closes",
"[replay][AR-012]") {
Dump d = load(fixture("superhero.h5"));
Dump d = load(fixture("bali_13.h5"));
Replay r = run(d);
CHECK(r.track_ids.size() == r.faces_seen);
@@ -232,9 +217,9 @@ TEST_CASE("every face is assigned a track and every track closes",
CHECK(r.claims.size() > 0);
}
TEST_CASE("windows are well-formed and inside the film", "[replay][AR-013]") {
for (const char* f : {"superhero.h5", "superhero.h5", "superhero.h5",
"superhero.h5", "superhero.h5"}) {
TEST_CASE("windows are well-formed and inside the clip", "[replay][AR-013]") {
for (const char* f : {"bali_13.h5", "bali_27.h5", "bali_28.h5",
"bali_31.h5", "bali_46.h5"}) {
INFO(f);
Dump d = load(fixture(f));
Replay r = run(d);
@@ -249,3 +234,25 @@ TEST_CASE("windows are well-formed and inside the film", "[replay][AR-013]") {
}
}
}
TEST_CASE("a longer extinction window yields fewer, longer tracks",
"[replay][AR-013]") {
// The timeout decides whether a gap is absorbed into one window or splits
// it in two, so lengthening it must merge tracks rather than multiply them.
// On sparse footage this is the difference the constant actually makes.
Dump d = load(fixture("bali_46.h5")); // 140 faces over 385 frames
Replay tight = run(d, /*extinction=*/1.0);
Replay loose = run(d, /*extinction=*/30.0);
CHECK(loose.claims.size() <= tight.claims.size());
}
// ── AR-007 — cuts are exercised by the corpus, not just by construction ──────
TEST_CASE("the cut-heavy fixture actually contains cuts", "[replay][AR-007]") {
// Guards the corpus rather than the code: if a regeneration produced a
// fixture with no cuts, the association tests above would still pass while
// silently testing nothing about viewpoint changes.
Dump d = load(fixture("bali_28.h5"));
const int cuts = std::count(d.is_cut.begin(), d.is_cut.end(), uint8_t{1});
CHECK(cuts >= 5);
}
tests/test_scene_detector_node.cpp
-191
View File
@@ -1,191 +0,0 @@
// AR-011 — the boundary dedup window is derived from the stream's cadence, not
// assumed.
//
// TRACES: AR-011 | SR-002 | UT-003
//
// Tier T1: the derivation is arithmetic on frame timestamps, so it is checked
// against synthetic cadences at 24, 25 and 30 fps rather than against a decode.
// The number this replaced was 0.04 s — one frame at 25 fps, correct for exactly
// one of those three and quietly wrong for the other two.
//
// SceneDetectorFunc is never constructed: its constructor loads TransNetV2. Only
// the static rule is called, so make_scene_detector() is never odr-used.
#include <catch2/catch_test_macros.hpp>
#include "nodes/scene_detector_node.hpp"
#include <opencv2/imgproc.hpp>
#include <utility>
#include <vector>
namespace {
// The intervals the node accumulates from a steady stream at `fps`.
std::vector<double> cadence(double fps, int n = 200) {
return std::vector<double>(static_cast<std::size_t>(n), 1.0 / fps);
}
double window(double fps) {
return SceneDetectorFunc::dedup_window_sec(cadence(fps));
}
} // namespace
// ── The property that has to hold at every rate ──────────────────────────────
// The window has exactly one job: tell "one frame scored twice by two
// overlapping windows" (a gap of zero) from "two adjacent frames, both of them
// real cuts" (a gap of one frame interval). It has to sit strictly between.
TEST_CASE("the dedup window separates a duplicate from an adjacent frame",
"[scene][AR-011]") {
for (double fps : {24.0, 25.0, 30.0, 23.976, 29.97, 50.0, 60.0}) {
INFO("source at " << fps << " fps");
const double frame = 1.0 / fps;
const double w = window(fps);
CHECK(w > 0.0); // a duplicate (gap 0) is still merged
CHECK(w < frame); // two consecutive frames both survive
}
}
// The concrete failure the hardcoded constant caused: at 30 fps a frame is
// 0.0333 s, so a 0.04 s window swallowed a cut on the very next frame. Nothing in
// the output showed it — the file just had fewer boundaries.
TEST_CASE("cuts on consecutive frames survive at 30 fps", "[scene][AR-011]") {
const double frame = 1.0 / 30.0;
CHECK(window(30.0) < frame);
CHECK(0.04 > frame); // the constant that was there, for the record
}
TEST_CASE("the window tracks the rate rather than a constant",
"[scene][AR-011]") {
// If it were still assumed, these would be equal.
CHECK(window(24.0) > window(30.0));
CHECK(window(30.0) > window(60.0));
CHECK(window(25.0) == 0.5 / 25.0);
}
// ── Robustness of the estimate ───────────────────────────────────────────────
TEST_CASE("a seek or a dropped frame does not move the derived cadence",
"[scene][AR-011]") {
auto intervals = cadence(25.0);
intervals[0] = 3.5; // a seek at the start
intervals[97] = 0.4; // a gap where the decoder lost frames
// Median, not mean: two long intervals out of 200 cannot shift it at all.
CHECK(SceneDetectorFunc::dedup_window_sec(intervals) == 0.5 / 25.0);
}
TEST_CASE("too few frames to have a cadence yields an inert window",
"[scene][AR-011]") {
// Under two frames there is no interval to measure — and also no second
// boundary to merge with, so a window of 0 changes nothing. Guessing a rate
// here would be the mistake this requirement is about.
CHECK(SceneDetectorFunc::dedup_window_sec({}) == 0.0);
}
TEST_CASE("a single observed interval is enough", "[scene][AR-011]") {
CHECK(SceneDetectorFunc::dedup_window_sec({1.0 / 24.0}) == 0.5 / 24.0);
}
// ── AR-004 — the window stores the model's input, not the decoded frame ───────
//
// TRACES: AR-004, AR-010 | SR-002 | UT-003
//
// The rolling window held frames as decoded, at full resolution, and left the
// downscale to the backend — ~590 MB at 1080p to feed a model whose input is
// 48x27, about 380 KB. Not a channel capacity, so no amount of tuning channel
// depths would have found it.
//
// The risk in fixing it is the project invariant: every model gets the input it
// was trained for. A model run off-distribution returns confident, plausible,
// wrong output, and here that means fabricated shot boundaries — which would be
// indistinguishable from a real cut in the output.
//
// So these cases do not check that the frames got smaller. They check that the
// pixels are *identical* to what the backend would have produced from the full
// frame, by performing the backend's own two operations independently and
// comparing byte for byte. Both ort_backend.cpp and trt_backend.cpp guard
// mis-sized input with convertTo(CV_8UC3) then
// cv::resize(..., {kFrameW, kFrameH}, 0, 0, cv::INTER_AREA), in that order.
namespace {
cv::Mat gradient(int w, int h) {
// Structured content, not a flat fill: INTER_AREA averages, so a constant
// image would compare equal under almost any resize and prove nothing.
cv::Mat m(h, w, CV_8UC3);
for (int y = 0; y < h; ++y)
for (int x = 0; x < w; ++x)
m.at<cv::Vec3b>(y, x) = cv::Vec3b(
static_cast<uchar>((x * 7 + y * 3) % 256),
static_cast<uchar>((x * 13 + y * 5) % 256),
static_cast<uchar>((x * 3 + y * 11) % 256));
return m;
}
bool identical(const cv::Mat& a, const cv::Mat& b) {
if (a.size() != b.size() || a.type() != b.type()) return false;
cv::Mat diff;
cv::absdiff(a, b, diff);
return cv::countNonZero(diff.reshape(1)) == 0;
}
} // namespace
TEST_CASE("the window frame is what the backend would have produced",
"[scene][AR-004]") {
for (auto [w, h] : {std::pair{1920, 1080}, std::pair{640, 360}, std::pair{720, 480}}) {
INFO("source " << w << "x" << h);
const cv::Mat full = gradient(w, h);
// The backend's own guard, performed here independently.
cv::Mat expected;
cv::resize(full, expected, {ISceneDetector::kFrameW, ISceneDetector::kFrameH},
0, 0, cv::INTER_AREA);
const cv::Mat got = SceneDetectorFunc::to_model_input(full);
REQUIRE(got.cols == ISceneDetector::kFrameW);
REQUIRE(got.rows == ISceneDetector::kFrameH);
REQUIRE(got.type() == CV_8UC3);
CHECK(identical(got, expected));
}
}
TEST_CASE("a frame already at model size is passed through untouched",
"[scene][AR-004]") {
// The backend skips its guard for a correctly-sized frame, so this path must
// not resize either — resampling an already-48x27 image would change it.
const cv::Mat exact = gradient(ISceneDetector::kFrameW, ISceneDetector::kFrameH);
CHECK(identical(SceneDetectorFunc::to_model_input(exact), exact));
}
TEST_CASE("conversion happens before the resize, as the backend does it",
"[scene][AR-004]") {
// Order matters: converting a 4-channel frame after downscaling averages
// alpha into the colour channels and gives different pixels. The backends
// convert first, so this must too.
cv::Mat four(360, 640, CV_8UC4, cv::Scalar(10, 20, 30, 255));
cv::Mat typed;
four.convertTo(typed, CV_8UC3);
cv::Mat expected;
cv::resize(typed, expected, {ISceneDetector::kFrameW, ISceneDetector::kFrameH},
0, 0, cv::INTER_AREA);
CHECK(identical(SceneDetectorFunc::to_model_input(four), expected));
}
TEST_CASE("the window's memory is bounded by the model input, not the source",
"[scene][AR-004]") {
// The point of the change, stated as a number: a full window of 1080p
// frames is ~590 MB as decoded and ~380 KB as model input.
const cv::Mat full = gradient(1920, 1080);
const cv::Mat small = SceneDetectorFunc::to_model_input(full);
const std::size_t decoded = full.total() * full.elemSize();
const std::size_t stored = small.total() * small.elemSize();
INFO("decoded " << decoded << " B, stored " << stored << " B");
CHECK(stored * 1000 < decoded); // three orders of magnitude
CHECK(stored == ISceneDetector::kFrameW * ISceneDetector::kFrameH * 3u);
}
tests/test_similarity.cpp
-107
View File
@@ -1,11 +1,5 @@
// TRACES: UT-004 | AR-026 | SR-001
//
// Unit tests for the CPU reference similarity engine (backends/gemm_backend.cpp,
// SAE_GEMM_CPU) and the l2_normalise helper. All pure, GPU-free, model-free.
//
// This is the CI half of AR-026: equivalence between the GEMM path and
// hand-computed dot products on small input. Throughput at scale (AR-027) is T4
// and cannot run here.
#include <catch2/catch_test_macros.hpp>
#include <catch2/matchers/catch_matchers_floating_point.hpp>
@@ -82,107 +76,6 @@ TEST_CASE("CPU similarity engine matches hand-computed dot products", "[similari
CHECK_THAT(S[2 + 1 * n_gallery], WithinAbs(s, 1e-6f));
}
// TRACES: UT-004 | AR-026 | SR-001
// The annex half of AR-026: promoted rows are appended to the resident matrix
// and scored by the same GEMM as the baked references. What used to be a
// host-side cosine loop over TrackGallery::annex() is now these extra columns,
// so the equivalence that matters is "an appended row scores exactly what the
// reference dot product says", and "appending changes nothing about the rows
// already there".
TEST_CASE("appended rows are scored by the same GEMM as the baked gallery",
"[similarity][AR-026]") {
std::vector<float> gallery;
for (int slot : {0, 1}) {
auto e = one_hot(slot);
gallery.insert(gallery.end(), e.begin(), e.end());
}
auto engine = make_similarity_engine(gallery.data(), /*n_gallery=*/2, /*max_faces=*/2);
REQUIRE(engine->n_gallery() == 2);
// One query face at 45° between slots 1 and 2. Slot 2 is not in the baked
// gallery yet, so the face is currently "unrecognised at that pose".
const float s = std::sqrt(0.5f);
std::vector<float> query(static_cast<size_t>(2) * 512, 0.0f);
query[1] = s;
query[2] = s;
const float* before = engine->compute(query.data(), 1);
CHECK_THAT(before[0], WithinAbs(0.0f, 1e-6f)); // vs slot 0
CHECK_THAT(before[1], WithinAbs(s, 1e-6f)); // vs slot 1
// Promote the missing view — the annex row an owned track would contribute.
auto promoted = one_hot(2);
engine->append_rows(promoted.data(), 1);
REQUIRE(engine->n_gallery() == 3);
const float* after = engine->compute(query.data(), 1);
CHECK_THAT(after[0], WithinAbs(0.0f, 1e-6f)); // baked rows unchanged
CHECK_THAT(after[1], WithinAbs(s, 1e-6f));
CHECK_THAT(after[2], WithinAbs(s, 1e-6f)); // appended row, same multiply
}
TEST_CASE("appending many rows keeps every similarity exact", "[similarity][AR-026]") {
// Start from a one-row gallery and append past the initial capacity several
// times over — the growth path has to preserve what is already resident, and
// a film promotes far more rows than the gallery starts with.
auto seed = one_hot(0);
auto engine = make_similarity_engine(seed.data(), /*n_gallery=*/1, /*max_faces=*/1);
constexpr int kAppended = 40;
for (int i = 1; i <= kAppended; ++i) {
auto e = one_hot(i);
engine->append_rows(e.data(), 1);
}
REQUIRE(engine->n_gallery() == kAppended + 1);
// Query one-hot slot k: similarity is 1 against row k and 0 against all others.
for (int k : {0, 1, 17, kAppended}) {
auto q = one_hot(k);
const float* S = engine->compute(q.data(), 1);
for (int g = 0; g <= kAppended; ++g)
CHECK_THAT(S[g], WithinAbs(g == k ? 1.0f : 0.0f, 1e-6f));
}
}
TEST_CASE("appending a block of rows matches appending them one at a time",
"[similarity][AR-026]") {
// A promotion hands over a whole diversity buffer at once; that must be
// indistinguishable from the same rows arriving singly.
auto seed = one_hot(0);
std::vector<float> block;
for (int slot : {1, 2, 3}) {
auto e = one_hot(slot);
block.insert(block.end(), e.begin(), e.end());
}
auto bulk = make_similarity_engine(seed.data(), 1, 1);
bulk->append_rows(block.data(), 3);
auto singly = make_similarity_engine(seed.data(), 1, 1);
for (int i = 0; i < 3; ++i) singly->append_rows(block.data() + i * 512, 1);
REQUIRE(bulk->n_gallery() == singly->n_gallery());
const float t = std::sqrt(1.0f / 3.0f);
std::array<float, 512> q{};
q[1] = t; q[2] = t; q[3] = t;
const float* a = bulk->compute(q.data(), 1);
std::vector<float> a_copy(a, a + bulk->n_gallery());
const float* b = singly->compute(q.data(), 1);
for (int g = 0; g < bulk->n_gallery(); ++g)
CHECK_THAT(a_copy[g], WithinAbs(b[g], 1e-6f));
}
TEST_CASE("appending zero rows is a no-op", "[similarity][AR-026]") {
auto e = one_hot(0);
auto engine = make_similarity_engine(e.data(), 1, 1);
CHECK_NOTHROW(engine->append_rows(nullptr, 0));
CHECK(engine->n_gallery() == 1);
}
TEST_CASE("CPU similarity engine rejects too many faces", "[similarity]") {
auto e = one_hot(0);
auto engine = make_similarity_engine(e.data(), /*n_gallery=*/1, /*max_faces=*/1);
tests/test_track_gallery.cpp
+71 -345
View File
@@ -1,21 +1,7 @@
// TRACES: UT-005 | AR-018, AR-019, AR-024, AR-026 | SR-005, SR-001
//
// Unit tests for TrackGallery (gallery/track_gallery.hpp): per-film gallery
// expansion driven by track continuity. Pure, GPU-free, model-free — exercises
// the AR-018 banded admission at both bounds, the promotion-time coherence
// gate, the diversity-buffer eviction policy, plurality ownership, and
// idempotent promotion, all through the public interface.
//
// The band is defined in PROBABILITY space (AR-024), so every case below states
// its own cosine → probability map. It has to: set_calibration is now mandatory
// and there is no header default to inherit.
//
// There used to be one — `max(0, cosine)` — and it was the reason this comment
// was originally needed. Under it the two spaces coincided, so a test that
// forgot to name the mapping still passed, and a gate that silently reverted to
// raw cosine passed with it. The default is gone rather than merely discouraged,
// which is why identity_cal below is now an explicit choice a case makes and not
// a restatement of what would have happened anyway.
// the diversity-buffer eviction policy, the novelty/spread safety gates,
// plurality ownership, and idempotent promotion via the public interface.
#include <catch2/catch_test_macros.hpp>
#include <catch2/matchers/catch_matchers_floating_point.hpp>
@@ -23,79 +9,35 @@
#include "gallery/track_gallery.hpp"
#include "types.hpp"
#include <algorithm>
#include <stdexcept>
#include <array>
#include <cmath>
#include <vector>
namespace {
constexpr float kPi = 3.14159265358979323846f;
// The band as the tests drive it. Kept in one place so a change to the shipped
// defaults does not silently invalidate the arithmetic in each case.
constexpr float kBandLo = 0.90f;
constexpr float kBandHi = 0.95f;
// Identity map: probability == cosine, so a case can place an embedding at an
// exact probability. cosine_similarity is a bare dot product over unit vectors
// (types.hpp), so the placements below are bit-exact, not approximate.
float identity_cal(float c) { return c; }
// Unit-norm embedding pointing along one axis. Cosine sim to another one-hot is
// 0, to itself 1.
// Unit-norm embedding pointing along one axis (cosine sim to another one-hot is
// 0, to itself 1) — lets tests dial gallery similarity precisely.
Embedding one_hot(int slot) {
Embedding e{};
e[slot] = 1.0f;
return e;
}
// TRACES: AR-026 | SR-001
// The annex is a contiguous row-major matrix, not a vector of structs, so that
// the matcher can hand whole blocks of new rows to the GEMM path. Tests that
// want to compare one promoted view read it back through this.
Embedding annex_view(const TrackGallery& tg, int row) {
const float* p = tg.annex_row(row);
Embedding e{};
std::copy(p, p + 512, e.begin());
return e;
}
// Unit-norm embedding in the plane of axes i,j at cosine `cos_t` from axis i.
// Cosine sim to one_hot(i) is exactly cos_t.
// Unit-norm embedding in the plane of axes i,j at angle t from i. Cosine sim to
// one_hot(i) is cos(t) — used to place a view at a chosen gallery similarity.
Embedding at_sim(int i, int j, float cos_t) {
Embedding e{};
float s = std::sqrt(std::max(0.f, 1.f - cos_t * cos_t));
e[i] = cos_t;
e[j] = std::sqrt(std::max(0.f, 1.f - cos_t * cos_t));
return e;
}
// A spoke: shares axis 0 with every other spoke, and is otherwise unique. Any
// two DISTINCT spokes have cosine similarity exactly cos_t², so one constant
// places a whole mutually-in-band store. A spoke against itself is 1.0 — above
// the band's ceiling, i.e. redundant, which is the intended reading.
Embedding spoke(int k, float cos_t) { return at_sim(0, k, cos_t); }
// cos_t chosen so pairwise similarity between distinct spokes is 0.9197 —
// comfortably inside [0.90, 0.95], clear of both bounds.
constexpr float kSpokeCos = 0.959f;
// Two embeddings `deg` apart in the plane of axes 0,1. Cosine is cos(deg), so a
// chain of these can step through the band while its endpoints fall outside it.
Embedding on_circle(float deg) {
Embedding e{};
e[0] = std::cos(deg * kPi / 180.f);
e[1] = std::sin(deg * kPi / 180.f);
e[j] = s;
return e;
}
Config expand_cfg() {
Config cfg;
cfg.expand_gallery = true;
cfg.expand_buffer_size = 3;
cfg.expand_band_lo = kBandLo;
cfg.expand_band_hi = kBandHi;
cfg.expand_gallery = true;
cfg.expand_buffer_size = 3;
cfg.expand_novelty_sim = 0.55f;
cfg.expand_track_spread_max = 0.60f;
cfg.expand_min_anchor_frames = 3;
return cfg;
}
@@ -114,311 +56,95 @@ TEST_CASE("disabled: no annex growth when expand_gallery is off", "[track_galler
REQUIRE_FALSE(tg.enabled());
for (int f = 0; f < 10; ++f)
tg.observe(1, one_hot(1), /*actor*/ 0, /*sim*/ 0.2f, /*accept*/ true, kNoCrop);
CHECK(tg.annex_size() == 0);
CHECK(tg.annex().empty());
}
// ── AR-018: the band ─────────────────────────────────────────────────────────
TEST_CASE("band bounds come from config, not a hardcoded default", "[track_gallery][AR-018]") {
// The bounds were declared in Config and read nowhere, so the gate ran at
// whatever the header happened to initialise. Drive them somewhere the
// defaults are not and require the gate to follow.
Config cfg = expand_cfg();
cfg.expand_band_lo = 0.40f;
cfg.expand_band_hi = 0.60f;
TrackGallery tg(cfg);
tg.set_calibration(identity_cal);
tg.observe(1, one_hot(0), 0, 0.30f, true, kNoCrop);
// P = 0.50: inside the configured band, far below the shipped default lo.
tg.observe(1, at_sim(0, 1, 0.50f), 0, 0.30f, true, kNoCrop);
CHECK(tg.band_rejected() == 0);
// P = 0.92: inside the shipped default band, above the configured ceiling.
tg.observe(1, at_sim(0, 2, 0.92f), 0, 0.30f, true, kNoCrop);
CHECK(tg.band_rejected() == 1);
}
TEST_CASE("band admits at each bound exactly", "[track_gallery][AR-018]") {
// The verification plan asks for the bounds themselves, not a point safely
// inside them: an off-by-one in the comparison is invisible anywhere else.
// Both bounds are inclusive.
SECTION("lower bound exactly") {
TrackGallery tg(expand_cfg());
tg.set_calibration(identity_cal);
tg.observe(1, one_hot(0), 0, 0.30f, true, kNoCrop);
tg.observe(1, at_sim(0, 1, kBandLo), 0, 0.30f, true, kNoCrop);
CHECK(tg.band_rejected() == 0);
}
SECTION("upper bound exactly") {
TrackGallery tg(expand_cfg());
tg.set_calibration(identity_cal);
tg.observe(1, one_hot(0), 0, 0.30f, true, kNoCrop);
tg.observe(1, at_sim(0, 1, kBandHi), 0, 0.30f, true, kNoCrop);
CHECK(tg.band_rejected() == 0);
}
}
TEST_CASE("store never admits below the lower bound", "[track_gallery][AR-018]") {
// The lower bound is the poisoning guard: an embedding unlike everything
// already on the track is evidence the track is not one person.
TEST_CASE("confirmed track promotes gallery-far views", "[track_gallery]") {
TrackGallery tg(expand_cfg());
tg.set_calibration(identity_cal);
tg.observe(1, one_hot(0), 0, 0.30f, true, kNoCrop);
REQUIRE(tg.enabled());
tg.observe(1, at_sim(0, 1, kBandLo - 0.01f), 0, 0.30f, true, kNoCrop);
CHECK(tg.band_rejected() == 1);
// A track owned by actor 0. Every frame is accepted as actor 0, but each
// view is gallery-far (sim 0.30 < novelty 0.55) yet mutually self-similar
// enough to pass the spread gate.
for (int f = 0; f < 3; ++f)
tg.observe(7, at_sim(0, 1, 0.30f + 0.001f * f), 0, 0.30f + 0.001f * f, true, kNoCrop);
tg.observe(1, one_hot(400), 0, 0.30f, true, kNoCrop); // orthogonal: P = 0
CHECK(tg.band_rejected() == 2);
// 3 accepted frames == min_anchor_frames → confirmed and promoted.
CHECK_FALSE(tg.annex().empty());
for (const auto& ae : tg.annex()) CHECK(ae.actor_idx == 0);
}
TEST_CASE("store never admits above the upper bound", "[track_gallery][AR-018]") {
// The upper bound is the redundancy guard: another look at a pose the store
// already covers teaches the annex nothing and costs a slot.
TEST_CASE("novelty gate skips views the gallery already covers", "[track_gallery]") {
TrackGallery tg(expand_cfg());
tg.set_calibration(identity_cal);
tg.observe(1, one_hot(0), 0, 0.30f, true, kNoCrop);
tg.observe(1, at_sim(0, 1, kBandHi + 0.01f), 0, 0.30f, true, kNoCrop);
CHECK(tg.band_rejected() == 1);
tg.observe(1, one_hot(0), 0, 0.30f, true, kNoCrop); // identical: P = 1
CHECK(tg.band_rejected() == 2);
}
TEST_CASE("band thresholds probability, not cosine", "[track_gallery][AR-018][AR-024]") {
// The invariant's actual claim, and the one a raw-cosine gate passes by
// accident under an identity calibration. With a calibration that shifts by
// +0.10, two embeddings get the OPPOSITE verdict from the one their bare
// cosines would earn — so admission here can only come from the calibrated
// value having been used.
TrackGallery tg(expand_cfg());
tg.set_calibration([](float c) { return c + 0.10f; });
tg.observe(1, one_hot(0), 0, 0.30f, true, kNoCrop);
// cosine 0.84 (below lo, would be refused raw) → P = 0.94, inside the band.
tg.observe(1, at_sim(0, 1, 0.84f), 0, 0.30f, true, kNoCrop);
CHECK(tg.band_rejected() == 0);
// cosine 0.92 (inside the band, would be admitted raw) → P = 1.02, above it.
tg.observe(1, at_sim(0, 2, 0.92f), 0, 0.30f, true, kNoCrop);
CHECK(tg.band_rejected() == 1);
// All views are recognised well (sim 0.90 ≥ novelty 0.55): nothing worth
// promoting even though the track is confirmed.
for (int f = 0; f < 3; ++f)
tg.observe(2, one_hot(0), 0, 0.90f, true, kNoCrop);
CHECK(tg.annex().empty());
}
TEST_CASE("a two-person track never poisons the annex", "[track_gallery][AR-018]") {
TrackGallery tg(expand_cfg());
tg.set_calibration(identity_cal);
// AR-019: promotion needs the registry's verdict; the local
// accepted-frame plurality that used to supply it is gone.
tg.set_owner(3, 0);
// Two orthogonal identities under one track ID — a track-ID collision.
// The band refuses the outsider at the door, so the store never becomes
// two-person in the first place.
tg.observe(3, spoke(1, kSpokeCos), 0, 0.30f, true, kNoCrop);
tg.observe(3, spoke(2, kSpokeCos), 0, 0.30f, true, kNoCrop);
//
// The banded admission (AR-018) now catches this EARLIER than the spread
// gate did: an embedding unlike everything already on the track falls below
// the band's lower bound and is refused entry, so the buffer never becomes
// two-person in the first place. The spread gate remains as a second line
// for a track that drifts gradually rather than jumping.
//
// The assertion is on the outcome, not the mechanism: whichever gate fires,
// the outsider must not reach the actor's annex.
tg.observe(3, at_sim(0, 1, 0.30f), 0, 0.30f, true, kNoCrop);
tg.observe(3, at_sim(0, 1, 0.30f), 0, 0.30f, true, kNoCrop);
tg.observe(3, one_hot(400), 0, 0.30f, true, kNoCrop); // orthogonal outlier
CHECK(tg.band_rejected() == 1); // refused at the door
REQUIRE(tg.annex_size() > 0); // the legitimate views still promote
for (int i = 0; i < tg.annex_size(); ++i)
CHECK(cosine_similarity(annex_view(tg, i), one_hot(400)) < 0.5f);
CHECK(tg.band_rejected() > 0); // refused at the door
for (const auto& e : tg.annex())
CHECK(cosine_similarity(e.emb, one_hot(400)) < 0.5f);
}
TEST_CASE("a track that drifts through the band is refused at promotion",
"[track_gallery][AR-018]") {
// `admit` compares a newcomer against its CLOSEST existing member, so a
// gradual drift chains past it: each step is in-band while the endpoints are
// strangers. This is the shape a collision takes over a slow pan, and the
// reason the lower bound is re-asked across every pair before promotion.
TEST_CASE("unconfirmed track (too few accepts) does not promote", "[track_gallery]") {
TrackGallery tg(expand_cfg());
tg.set_calibration(identity_cal);
tg.observe(5, on_circle(0.f), 0, 0.30f, true, kNoCrop);
tg.observe(5, on_circle(25.f), 0, 0.30f, true, kNoCrop); // P=0.906 vs 0° → in band
tg.observe(5, on_circle(50.f), 0, 0.30f, true, kNoCrop); // P=0.906 vs 25° → in band
CHECK(tg.band_rejected() == 0); // every step passed the door...
// ...but 0° and 50° are P=0.643 apart, below the floor: the whole track goes.
CHECK(tg.annex_size() == 0);
// Only 2 accepted frames < min_anchor_frames 3; extra non-accepted frames
// fill the buffer but don't count toward ownership.
tg.observe(4, at_sim(0, 1, 0.30f), 0, 0.30f, true, kNoCrop);
tg.observe(4, at_sim(0, 1, 0.31f), 0, 0.31f, true, kNoCrop);
tg.observe(4, at_sim(0, 1, 0.32f), 0, 0.32f, false, kNoCrop);
CHECK(tg.annex().empty());
}
// ── AR-019: ownership and promotion ──────────────────────────────────────────
TEST_CASE("confirmed track promotes its store", "[track_gallery][AR-019]") {
TrackGallery tg(expand_cfg());
tg.set_calibration(identity_cal);
// AR-019: promotion needs the registry's verdict; the local
// accepted-frame plurality that used to supply it is gone.
tg.set_owner(7, 0);
REQUIRE(tg.enabled());
// A track owned by actor 0: every frame accepted, every view mutually
// in-band (P = 0.9197 between distinct spokes) and gallery-far (0.30).
for (int k = 1; k <= 3; ++k)
tg.observe(7, spoke(k, kSpokeCos), 0, 0.30f, true, kNoCrop);
// 3 accepted frames == min_anchor_frames → confirmed and promoted.
CHECK(tg.annex_size() == 3);
for (int actor : tg.annex_actors()) CHECK(actor == 0);
TEST_CASE("plurality actor wins a mixed-vote track", "[track_gallery]") {
Config cfg = expand_cfg();
cfg.expand_min_anchor_frames = 3;
TrackGallery tg(cfg);
// Actor 5 accepted twice, actor 6 once → plurality is 5. All views novel.
tg.observe(8, at_sim(0, 1, 0.30f), 5, 0.30f, true, kNoCrop);
tg.observe(8, at_sim(0, 1, 0.31f), 5, 0.31f, true, kNoCrop);
tg.observe(8, at_sim(0, 1, 0.32f), 6, 0.32f, true, kNoCrop);
REQUIRE_FALSE(tg.annex().empty());
for (const auto& ae : tg.annex()) CHECK(ae.actor_idx == 5);
}
TEST_CASE("registry ownership overrides the local tally", "[track_gallery][AR-019]") {
TEST_CASE("promotion is idempotent across a long track", "[track_gallery]") {
TrackGallery tg(expand_cfg());
tg.set_calibration(identity_cal);
// Local accepted-frame plurality says actor 5; the registry's accumulated
// posterior says actor 9. The registry is authoritative.
tg.set_owner(11, 9);
for (int k = 1; k <= 3; ++k)
tg.observe(11, spoke(k, kSpokeCos), 5, 0.30f, true, kNoCrop);
REQUIRE(tg.annex_size() > 0);
for (int actor : tg.annex_actors()) CHECK(actor == 9);
}
TEST_CASE("unconfirmed track (too few accepts) does not promote", "[track_gallery][AR-019]") {
TrackGallery tg(expand_cfg());
tg.set_calibration(identity_cal);
// Only 2 accepted frames < min_anchor_frames 3; the third fills the buffer
// but doesn't count toward ownership.
tg.observe(4, spoke(1, kSpokeCos), 0, 0.30f, true, kNoCrop);
tg.observe(4, spoke(2, kSpokeCos), 0, 0.30f, true, kNoCrop);
tg.observe(4, spoke(3, kSpokeCos), 0, 0.30f, false, kNoCrop);
CHECK(tg.annex_size() == 0);
}
TEST_CASE("an unowned track never promotes, however many frames it accepts",
"[track_gallery][AR-019]") {
TrackGallery tg(expand_cfg());
tg.set_calibration(identity_cal);
// This case used to assert the opposite: it drove a mixed-vote track with
// no registry owner and expected the local plurality winner (actor 5) to
// take the promotion. That fallback is gone. AR-019 says ownership comes
// from the registry and not from a second local tally, and the tally could
// not see the AR-025 discounting -- so it weighted thirty near-identical
// looks like thirty distinct ones.
//
// Ten accepted frames, no set_owner, nothing promoted.
for (int k = 1; k <= 10; ++k)
tg.observe(8, spoke(k, kSpokeCos), 5, 0.30f, true, kNoCrop);
CHECK(tg.annex_size() == 0);
}
TEST_CASE("promotion is idempotent across a long track", "[track_gallery][AR-019]") {
TrackGallery tg(expand_cfg());
tg.set_calibration(identity_cal);
// AR-019: promotion needs the registry's verdict; the local
// accepted-frame plurality that used to supply it is gone.
tg.set_owner(9, 0);
for (int k = 1; k <= 3; ++k)
tg.observe(9, spoke(k, kSpokeCos), 0, 0.30f, true, kNoCrop);
const int after_confirm = tg.annex_size();
for (int f = 0; f < 3; ++f)
tg.observe(9, at_sim(0, 1, 0.30f + 0.001f * f), 0, 0.30f + 0.001f * f, true, kNoCrop);
size_t after_confirm = tg.annex().size();
REQUIRE(after_confirm > 0);
// Keep feeding the confirmed track: annex must not grow again.
for (int f = 0; f < 10; ++f)
tg.observe(9, spoke(4 + f, kSpokeCos), 0, 0.30f, true, kNoCrop);
CHECK(tg.annex_size() == after_confirm);
tg.observe(9, at_sim(0, 1, 0.30f), 0, 0.30f, true, kNoCrop);
CHECK(tg.annex().size() == after_confirm);
}
TEST_CASE("clear_tracks drops buffers before confirmation", "[track_gallery][AR-019]") {
TEST_CASE("clear_tracks drops buffers before confirmation", "[track_gallery]") {
TrackGallery tg(expand_cfg());
tg.set_calibration(identity_cal);
// Two accepts, then a cut clears buffers; the third accept starts fresh and
// can't reach the anchor threshold on its own.
tg.observe(1, spoke(1, kSpokeCos), 0, 0.30f, true, kNoCrop);
tg.observe(1, spoke(2, kSpokeCos), 0, 0.30f, true, kNoCrop);
tg.observe(1, at_sim(0, 1, 0.30f), 0, 0.30f, true, kNoCrop);
tg.observe(1, at_sim(0, 1, 0.31f), 0, 0.31f, true, kNoCrop);
tg.clear_tracks();
tg.observe(1, spoke(3, kSpokeCos), 0, 0.30f, true, kNoCrop);
CHECK(tg.annex_size() == 0);
}
TEST_CASE("eviction keeps the gallery-far views", "[track_gallery][AR-018]") {
// Novelty is no longer a threshold — it is this ordering. With the buffer
// full, a more gallery-far newcomer must displace the best-recognised
// member, and a less novel one must be dropped rather than displace a
// better sample.
Config cfg = expand_cfg();
cfg.expand_buffer_size = 2;
cfg.expand_min_anchor_frames = 4;
TrackGallery tg(cfg);
tg.set_calibration(identity_cal);
// AR-019: promotion needs the registry's verdict; the local
// accepted-frame plurality that used to supply it is gone.
tg.set_owner(6, 0);
tg.observe(6, spoke(1, kSpokeCos), 0, 0.80f, true, kNoCrop); // well recognised
tg.observe(6, spoke(2, kSpokeCos), 0, 0.40f, true, kNoCrop);
tg.observe(6, spoke(3, kSpokeCos), 0, 0.20f, true, kNoCrop); // novel: evicts the 0.80
tg.observe(6, spoke(4, kSpokeCos), 0, 0.90f, true, kNoCrop); // least novel: dropped
REQUIRE(tg.annex_size() == 2);
// Survivors are the two most gallery-far views: spokes 2 and 3.
for (int i = 0; i < tg.annex_size(); ++i) {
const Embedding view = annex_view(tg, i);
const bool is_2 = cosine_similarity(view, spoke(2, kSpokeCos)) > 0.99f;
const bool is_3 = cosine_similarity(view, spoke(3, kSpokeCos)) > 0.99f;
CHECK((is_2 || is_3));
}
}
// TRACES: UT-005 | AR-026 | SR-001
// The annex reaches the GEMM path by being drained, not re-read: the matcher
// pushes newly promoted rows into the similarity engine once per frame. Draining
// must therefore be exactly-once — a row handed over twice becomes a duplicate
// gallery entry that quietly doubles an actor's best-of-N chances, and a row
// never handed over is a promotion that silently does nothing.
TEST_CASE("promotions drain exactly once, in matrix order",
"[track_gallery][AR-026]") {
TrackGallery tg(expand_cfg());
tg.set_calibration(identity_cal);
// AR-019: promotion needs the registry's verdict; the local
// accepted-frame plurality that used to supply it is gone.
tg.set_owner(7, 0);
tg.set_owner(8, 4);
std::vector<float> emb;
std::vector<int> actor;
CHECK(tg.drain_promotions(emb, actor) == 0); // nothing promoted yet
for (int k = 1; k <= 3; ++k)
tg.observe(7, spoke(k, kSpokeCos), 0, 0.30f, true, kNoCrop);
REQUIRE(tg.annex_size() == 3);
const int drained = tg.drain_promotions(emb, actor);
CHECK(drained == 3);
CHECK(actor.size() == 3);
CHECK(emb.size() == 3 * 512);
for (int a : actor) CHECK(a == 0);
// Drained rows are the annex rows, in the same order — the engine's row i
// and flat_actor_[i] have to keep naming the same view.
for (int i = 0; i < drained; ++i)
for (int d = 0; d < 512; ++d)
CHECK(emb[static_cast<size_t>(i) * 512 + d] == tg.annex_row(i)[d]);
// Draining again yields nothing: the engine already holds these.
CHECK(tg.drain_promotions(emb, actor) == 0);
CHECK(actor.size() == 3);
// A second track promotes, and only its rows are handed over.
for (int k = 1; k <= 3; ++k)
tg.observe(8, spoke(k, kSpokeCos), 4, 0.30f, true, kNoCrop);
CHECK(tg.drain_promotions(emb, actor) == 3);
CHECK(actor.size() == 6);
CHECK(actor[5] == 4);
}
// ── AR-024: the calibration is not optional ─────────────────────────────────
/// TRACES: UT-005 | AR-024 | SR-005
TEST_CASE("a null calibration is refused, not silently replaced", "[track_gallery][AR-024]") {
// The class used to default calibrate_ to max(0, cosine). That made
// expand_band_lo = 0.90 mean "cosine above 0.9" here and "P(same person)
// above 0.9" in production — two very different gates, with nothing
// announcing which one was in force. FaceTrackerFunc already refused to
// construct without a calibration for exactly this reason; the expansion
// store now matches it.
TrackGallery tg(expand_cfg());
CHECK_THROWS_AS(tg.set_calibration(nullptr), std::invalid_argument);
tg.observe(1, at_sim(0, 1, 0.32f), 0, 0.32f, true, kNoCrop);
CHECK(tg.annex().empty());
}
tests/test_track_registry.cpp
+1 -148
View File
@@ -45,7 +45,7 @@ EvidenceDiscounter disc() {
TrackRegistry::Config cfg(double extinction = 5.0, float own = 2.0f) {
TrackRegistry::Config c;
c.track_extinction_sec = extinction;
c.extinction_sec = extinction;
c.ownership_logodds = own;
return c;
}
@@ -370,150 +370,3 @@ TEST_CASE("confidence grows across frames of the same face", "[registry][AR-025]
// ...but it must still be worth far less than 50 independent looks would be.
CHECK(sink.claims[0].effective_obs < 25.0f);
}
// ── The evidence watermark: presence must not depend on node speed ───────────
/// TRACES: UT-001 | AR-012, AR-013, AR-025 | SR-002
TEST_CASE("a track is not reaped until the evidence clock passes it",
"[registry][AR-013]") {
// The tracker and the matcher are separate KPN nodes, and backpressure --
// working exactly as AR-004 intends -- lets the tracker run a whole
// channel's depth ahead. Reaping on the tracker's clock therefore closed
// tracks before their votes arrived: the votes landed on ids that no longer
// existed and the run silently under-reported. On the SuperHero fixture that
// was 5 actors at channel depth 32 against 0 actors at depth 10322, from
// identical input.
TrackRegistry reg(cfg(/*extinction=*/5.0), disc());
Sink sink; sink.attach(reg);
reg.expect_evidence(); // as IdentityMatcherFunc::set_registry does
int id;
{
auto s = reg.begin_frame(0.0);
id = s.create(0.0, axis(1));
s.mark_lost(id, 0.0);
}
// The tracker races 100 s ahead. Nothing has voted yet, so nothing may die:
// an unvoted track is not a finished track, it is an unanswered question.
{ auto s = reg.begin_frame(100.0); (void)s; }
CHECK(sink.claims.empty());
// A vote arriving very late still lands, because the track is still there.
reg.observe(id, /*actor*/ 3, /*posterior*/ 0.99f, axis(1));
CHECK(reg.dropped_votes() == 0);
// Only once the evidence clock passes last_seen + extinction does it close.
reg.advance_evidence(4.0);
CHECK(sink.claims.empty());
reg.advance_evidence(6.0);
REQUIRE(sink.claims.size() == 1);
CHECK(sink.claims[0].actor_idx == 3);
// AR-013 still holds: the window ends at the last sighting, never at the
// moment of death, and never at the watermark that authorised it.
CHECK(sink.claims[0].last_seen == 0.0);
}
/// TRACES: UT-001 | AR-008, AR-013 | SR-002
TEST_CASE("a track retired from association is still open to evidence",
"[registry][AR-008]") {
// The two clocks answer different questions and must not share an answer.
// Association asks "may this detection link to that track?" on the tracker's
// clock; reaping asks "is that track finished?" and cannot answer until the
// votes are in. Deferring both to the evidence clock was the second half of
// this bug: retired tracks lingered in the candidate pool for as long as the
// matcher lagged, so a new face re-associated onto a long-dead track and two
// people merged into one window.
TrackRegistry reg(cfg(/*extinction=*/5.0), disc());
Sink sink; sink.attach(reg);
reg.expect_evidence();
int id;
{
auto s = reg.begin_frame(0.0);
id = s.create(0.0, axis(1));
s.mark_lost(id, 0.0);
}
{
auto s = reg.begin_frame(3.0); // inside the window
CHECK(s.candidates().size() == 1); // still associable
}
{
auto s = reg.begin_frame(50.0); // far outside it
CHECK(s.candidates().empty()); // retired from association...
}
// ...but not gone, and still able to receive the votes in flight for it.
reg.observe(id, 7, 0.99f, axis(1));
CHECK(reg.dropped_votes() == 0);
reg.advance_evidence(50.0);
REQUIRE(sink.claims.size() == 1);
CHECK(sink.claims[0].actor_idx == 7);
}
// ── AR-025 — non-matches must not spend an actor's evidence budget ───────────
TEST_CASE("frames that recognise nobody do not exhaust the budget",
"[registry][AR-025]") {
// The correlation discount is an effective-sample correction: with
// observations correlated at rho, the n-th is worth
// n_eff(n+1) - n_eff(n) = 2/((n+1)(n+2)) at rho=0.5, so it decays
// quadratically and the total converges to 1/rho = 2. That saturation is
// deliberate — a long static shot must not out-argue varied evidence by
// lasting longer.
//
// What was not deliberate is that every scored face spent it, including
// ones that matched nobody. An observation at p=0.02 contributes
// log(0.98) = -0.02 of belief — nothing — while consuming the same
// increment as one at p=0.95. Measured on SuperHero-2: 103 observations on
// one track, effective weight 2.026, belief 0.455 against a 0.881
// threshold, with the 51 frames that *did* identify the actor arriving
// when each was worth 0.0002. The identification was lost.
//
// It also made the answer depend on frame rate — deliver more frames,
// dilute the budget with more non-matches, and a track that was owned stops
// being owned — which is the defect AR-013 already had to fix once.
TrackRegistry reg(cfg(), disc());
Sink sink; sink.attach(reg);
int id;
{ auto f = reg.begin_frame(0.0); id = f.create(0.0, axis(0)); }
// A long run of frames that match nobody: the detector saw a face, the
// matcher could not place it. These are not evidence for actor 1.
for (int i = 0; i < 40; ++i) reg.observe(id, 1, 0.02f, axis(0));
// Then the actor is clearly recognised. Before this fix the budget was
// already spent and these could not move the belief.
for (int i = 0; i < 6; ++i) reg.observe(id, 1, 0.9f, axis(0));
reg.flush(1.0);
REQUIRE(sink.claims.size() == 1);
INFO("belief " << sink.claims[0].belief
<< " effective_obs " << sink.claims[0].effective_obs);
CHECK(sink.claims[0].actor_idx == 1);
CHECK(sink.claims[0].belief > 0.88f);
}
TEST_CASE("a near-miss is still evidence", "[registry][AR-025]") {
// The floor is at 0.5 — where the posterior stops favouring the hypothesis
// — not at the matcher's acceptance threshold. A run of near-misses for one
// actor is informative and must still accumulate, which is the property the
// identity matcher's comment relies on when it feeds every scored face
// rather than only the accepted ones.
TrackRegistry reg(cfg(), disc());
Sink sink; sink.attach(reg);
int id;
{ auto f = reg.begin_frame(0.0); id = f.create(0.0, axis(0)); }
// 0.7 is below the matcher's acceptance threshold (0.754 on the SuperHero
// gallery) and above the 0.5 floor: a frame that would not be reported as
// an identification, but is still evidence. Twelve of them accumulate to
// ~0.89, past the 0.881 ownership threshold.
for (int i = 0; i < 12; ++i) reg.observe(id, 3, 0.7f, axis(0));
reg.flush(1.0);
REQUIRE(sink.claims.size() == 1);
INFO("belief " << sink.claims[0].belief);
CHECK(sink.claims[0].actor_idx == 3); // owned on near-misses alone
}