Compare commits
4
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 dtourolle
|
889018aa34 | ||
|
dtourolle
|
26de01b2e3 | ||
|
dtourolle
|
41d30395da | ||
|
dtourolle
|
1ae88376e1 |
@@ -65,15 +65,6 @@ jobs:
|
||||
- name: Traceability gate
|
||||
run: sh scripts/vendor/jray-project/scripts/traceability/traceability-gate.sh
|
||||
|
||||
# AR-024's register row names its verification tier as "Static check --
|
||||
# no bare cosine outside a tagged EXCEPTION". This is that check, and it
|
||||
# belongs here rather than in unit-tests.yml because it is static
|
||||
# analysis of source text, like everything else in this job, and needs
|
||||
# no toolchain. It blocks: an untagged bare cosine is a defect by the
|
||||
# invariant's own wording, not a warning.
|
||||
- name: AR-024 — no bare cosine outside a recorded exception
|
||||
run: python3 scripts/ci/check_raw_cosine.py
|
||||
|
||||
- name: Check modified files for traces
|
||||
if: github.event_name == 'pull_request'
|
||||
run: |
|
||||
|
||||
@@ -1,140 +0,0 @@
|
||||
name: Unit tests
|
||||
|
||||
# TRACES: DP-007 | PR-004
|
||||
#
|
||||
# The tier the verification strategy is built on, finally executing.
|
||||
#
|
||||
# docs/requirements.md describes a four-tier plan in which T1 (functor unit)
|
||||
# and T2 (replay) are "the only tiers that can exist in CI at all", and the
|
||||
# traceability gate reports a CI-scope coverage fraction over exactly those
|
||||
# tiers. Until this workflow existed, nothing ran them: "covered" meant a
|
||||
# TRACES tag was present in a file, not that any test had been executed. That
|
||||
# is the same failure mode as counting a test that cannot run, one level up,
|
||||
# and the gate cannot detect it because a tag is all it can see.
|
||||
#
|
||||
# The runner is an Intel N100 with no discrete GPU. Nothing here calls a model:
|
||||
# T1 constructs node functors directly, and T2 replays a precomputed HDF5 dump.
|
||||
# T3 (ORT CPU smoke) and T4 (GPU) are deliberately absent -- the embedder is
|
||||
# ~930 ms/frame on this hardware, so a 77 s clip at 5 fps would be six minutes
|
||||
# of inference alone.
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- main
|
||||
- master
|
||||
- develop
|
||||
pull_request:
|
||||
branches:
|
||||
- main
|
||||
- master
|
||||
- develop
|
||||
|
||||
jobs:
|
||||
unit-tests:
|
||||
runs-on: linux/amd64
|
||||
name: Build and run the GPU-free suite
|
||||
|
||||
# Pinned by tag, never `latest`, so rebuilding the image cannot silently
|
||||
# change what a previous green build meant. Bumping the dependency set means
|
||||
# bumping the tag in scripts/ci/build_builder_image.sh AND here, in one
|
||||
# commit -- see that script's header.
|
||||
container:
|
||||
image: gitea.tourolle.paris/dtourolle/sae-builder-cpu:v1
|
||||
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
fetch-depth: 0
|
||||
# KPN is a submodule and the pipeline does not build without it.
|
||||
#
|
||||
# NOTE: this checks out the commit this repo PINS, which is the whole
|
||||
# point and is also the first thing this job will disagree with a
|
||||
# developer about. A local KPN working copy that is ahead of
|
||||
# origin/master builds and passes here while CI builds something else
|
||||
# entirely; the AR-004 evidence in docs/requirements.md was gathered
|
||||
# that way. If this job fails on tests that pass locally, check
|
||||
# `git -C external/KPN log origin/master..HEAD` before suspecting the
|
||||
# tests.
|
||||
# LFS is deliberately NOT fetched: SAE_MODELS_DIR is baked into the
|
||||
# binary as a path string and nothing in T1/T2 opens a model file, so
|
||||
# pulling ~hundreds of MB of ONNX would cost the job everything and
|
||||
# buy it nothing.
|
||||
submodules: recursive
|
||||
lfs: false
|
||||
|
||||
- name: Assert the builder image is the pinned one
|
||||
run: |
|
||||
set -e
|
||||
echo "builder=$SAE_BUILDER version=$SAE_BUILDER_VERSION"
|
||||
echo "ort=$SAE_ORT_VERSION opencv=$SAE_OPENCV_VERSION"
|
||||
# The image reports its own tag. A mismatch means the `container:`
|
||||
# line above and the image that actually landed disagree, which is
|
||||
# exactly the drift the pinning exists to prevent -- so it fails the
|
||||
# job rather than building against an unknown toolchain.
|
||||
[ "$SAE_BUILDER_VERSION" = "v1" ] || {
|
||||
echo "image reports version '$SAE_BUILDER_VERSION', workflow pins v1" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
- name: Fetch replay fixtures
|
||||
env:
|
||||
GITEA_TOKEN: ${{ secrets.GITEA_TOKEN }}
|
||||
# bash, not sh: the script declares #!/bin/bash and uses `set -o
|
||||
# pipefail` and arrays, which dash does not have.
|
||||
run: bash scripts/artifacts/pull_artifacts.sh replay-fixtures latest
|
||||
|
||||
# pull_artifacts.sh warns and continues when a package version is missing,
|
||||
# which is right for a developer pulling one artifact of several and wrong
|
||||
# here. A T2 test whose fixture never arrived must not look like a pass:
|
||||
# the dumps are the entire input to the replay tier, and VR-002's claim is
|
||||
# that replay drives the real nodes over real data.
|
||||
- name: Verify the fixtures actually arrived
|
||||
run: |
|
||||
set -e
|
||||
missing=0
|
||||
for f in tests/fixtures/dumps/superhero.h5; do
|
||||
if [ -s "$f" ]; then
|
||||
echo " ok: $f ($(wc -c < "$f") bytes)"
|
||||
else
|
||||
echo " MISSING: $f" >&2
|
||||
missing=1
|
||||
fi
|
||||
done
|
||||
if [ "$missing" -ne 0 ]; then
|
||||
echo "" >&2
|
||||
echo "Replay fixtures are absent, so the T2 tier cannot run." >&2
|
||||
echo "They are not in git (tests/fixtures/dumps/.gitignore) -- they" >&2
|
||||
echo "live in the Gitea generic package registry and are pulled by" >&2
|
||||
echo "the step above, which needs GITEA_TOKEN to resolve 'latest'." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Configure
|
||||
run: |
|
||||
set -e
|
||||
# SAE_GEMM_BACKEND defaults to ROCM and the auto-detect prefers a GPU
|
||||
# backend where it finds one; CPU is stated explicitly so this job
|
||||
# cannot start depending on what happens to be installed on the runner.
|
||||
# The CPU kernel is OpenBLAS in this image (tests/CMakeLists.txt fails
|
||||
# the configure if it is not), so the suite exercises the kernel the
|
||||
# CPU release actually ships.
|
||||
cmake -S . -B build \
|
||||
-DCMAKE_BUILD_TYPE=Release \
|
||||
-DSAE_BUILD_TESTS=ON \
|
||||
-DSAE_GEMM_BACKEND=CPU
|
||||
|
||||
- name: Build the test suite
|
||||
run: cmake --build build --target sae_tests --parallel
|
||||
|
||||
- name: Run the tests
|
||||
run: ctest --test-dir build --output-on-failure
|
||||
|
||||
- name: Save test output
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v3
|
||||
with:
|
||||
name: unit-test-results
|
||||
path: build/Testing/
|
||||
retention-days: 30
|
||||
@@ -1,6 +1,5 @@
|
||||
# Build
|
||||
build/
|
||||
build-*/
|
||||
cmake-build-*/
|
||||
CMakeCache.txt
|
||||
CMakeFiles/
|
||||
@@ -118,6 +117,3 @@ venv/
|
||||
*.swo
|
||||
.DS_Store
|
||||
Thumbs.db
|
||||
.venv-rocm/
|
||||
!models/scene_boundary_xgb.json
|
||||
experiments/dump_review/
|
||||
|
||||
+12
-88
@@ -55,13 +55,6 @@ set_property(CACHE SAE_GEMM_BACKEND PROPERTY STRINGS ROCM CUDA CPU)
|
||||
# default so ROCm/CPU builds don't reference unavailable EPs.
|
||||
option(SAE_ORT_TRT_EP "ORT backend: enable TensorRT/CUDA execution providers" OFF)
|
||||
|
||||
# AR-026/AR-027: the CPU GEMM path is backed by OpenBLAS, and its absence is a
|
||||
# configure error rather than a silent downgrade to the scalar loop. Declared at
|
||||
# top level because the unit-test target compiles the CPU kernel regardless of
|
||||
# which backend the main build selected, and both must make the same choice.
|
||||
option(SAE_ALLOW_SCALAR_GEMM
|
||||
"Permit the scalar-loop GEMM fallback when OpenBLAS is absent" OFF)
|
||||
|
||||
# Back-compat: a legacy -DSAE_WITH_TRT=ON/OFF seeds the new vars (ON⇒TRT+CUDA,
|
||||
# OFF⇒ORT+ROCM) unless the user set them explicitly.
|
||||
if(DEFINED SAE_WITH_TRT)
|
||||
@@ -160,16 +153,10 @@ if(SAE_GEMM_BACKEND STREQUAL "CPU")
|
||||
target_include_directories(gemm_backend PRIVATE src)
|
||||
target_compile_definitions(gemm_backend PRIVATE SAE_GEMM_CPU)
|
||||
|
||||
# AR-026/AR-027: the CPU path is backed by OpenBLAS, and that is REQUIRED
|
||||
# rather than opportunistic. The CPU backend is what CI (no GPU) and the cpu
|
||||
# builder image actually run, so a silent fall back to the scalar loop means
|
||||
# AR-027 is measured — or worse, believed — on a path no release uses. A
|
||||
# missing dependency should stop the build and name itself, not degrade into
|
||||
# a slower answer nobody notices.
|
||||
#
|
||||
# The scalar loop survives as the correctness oracle the two backends are
|
||||
# diffed against; -DSAE_ALLOW_SCALAR_GEMM=ON is how you ask for it, which
|
||||
# keeps that an explicit, visible choice.
|
||||
# AR-026/AR-027: back the CPU path with OpenBLAS when present. Optional, so
|
||||
# the build gains no hard dependency — but without it the fallback is a
|
||||
# scalar loop, which does not hold up against a library-scale gallery, and
|
||||
# the CPU path is exactly what CI (no GPU) and the cpu builder image use.
|
||||
find_package(PkgConfig QUIET)
|
||||
if(PkgConfig_FOUND)
|
||||
pkg_check_modules(OPENBLAS QUIET openblas)
|
||||
@@ -179,16 +166,9 @@ if(SAE_GEMM_BACKEND STREQUAL "CPU")
|
||||
target_compile_definitions(gemm_backend PRIVATE SAE_GEMM_CBLAS)
|
||||
target_include_directories(gemm_backend PRIVATE ${OPENBLAS_INCLUDE_DIRS})
|
||||
target_link_libraries(gemm_backend PRIVATE ${OPENBLAS_LINK_LIBRARIES})
|
||||
elseif(SAE_ALLOW_SCALAR_GEMM)
|
||||
message(WARNING "GEMM backend: CPU scalar fallback (SAE_ALLOW_SCALAR_GEMM=ON) — "
|
||||
"correct, but slow on a large gallery. Do not measure AR-027 here.")
|
||||
else()
|
||||
message(FATAL_ERROR
|
||||
"OpenBLAS not found, and the CPU GEMM backend requires it (AR-026/AR-027).\n"
|
||||
" Install it: Fedora dnf install openblas-devel\n"
|
||||
" Arch pacman -S openblas\n"
|
||||
" Debian apt install libopenblas-dev\n"
|
||||
" Or build the scalar fallback deliberately: -DSAE_ALLOW_SCALAR_GEMM=ON")
|
||||
message(WARNING "GEMM backend: CPU scalar fallback — OpenBLAS not found. "
|
||||
"Correct, but slow on a large gallery (AR-027).")
|
||||
endif()
|
||||
elseif(SAE_GEMM_BACKEND STREQUAL "CUDA")
|
||||
find_library(CUBLAS_LIB cublas
|
||||
@@ -280,24 +260,6 @@ FetchContent_Declare(
|
||||
)
|
||||
FetchContent_MakeAvailable(nanobind)
|
||||
|
||||
# XGBoost (learned scene-boundary detector for flood-fill presence). Fetched and
|
||||
# built from source so we get both the C API header and a matching libxgboost,
|
||||
# reproducibly — the pip wheel ships the .so but no header. Heavy first build, so
|
||||
# it is opt-in; the scene-boundary node is compiled only when SAE_SCENE_XGB is on.
|
||||
option(SAE_SCENE_XGB "Build the XGBoost scene-boundary detector node" ON)
|
||||
if(SAE_SCENE_XGB)
|
||||
set(BUILD_STATIC_LIB ON CACHE BOOL "" FORCE) # link xgboost statically
|
||||
set(USE_OPENMP ON CACHE BOOL "" FORCE)
|
||||
FetchContent_Declare(
|
||||
xgboost
|
||||
GIT_REPOSITORY https://github.com/dmlc/xgboost.git
|
||||
GIT_TAG v2.1.1
|
||||
GIT_SHALLOW TRUE
|
||||
GIT_SUBMODULES_RECURSE TRUE
|
||||
)
|
||||
FetchContent_MakeAvailable(xgboost)
|
||||
endif()
|
||||
|
||||
# ── Model paths ───────────────────────────────────────────────────────────────
|
||||
set(SAE_MODELS_DIR "${CMAKE_SOURCE_DIR}/models"
|
||||
CACHE PATH "Directory containing ONNX model files")
|
||||
@@ -340,22 +302,11 @@ nanobind_add_module(sae_embed src/python_bindings.cpp)
|
||||
target_link_libraries(sae_embed PRIVATE sae_gallery)
|
||||
|
||||
# ── sae_kpn — Python module: run the real downstream nodes over dumped embeddings ─
|
||||
# Assembles face_tracker/identity_matcher/frame_annotation in a Python-driven KPN
|
||||
# Assembles face_tracker/identity_matcher/scene_tracker in a Python-driven KPN
|
||||
# network (KPN_BUILD_PYTHON is enabled per-TU inside the .cpp). Powers the
|
||||
# threshold-sweep optimizer in scripts/optimizer/.
|
||||
#
|
||||
# TRACES: VR-011 | PR-002
|
||||
# ON again. It was OFF for one commit because it had not compiled since the
|
||||
# AR-007/AR-008 tracker redesign -- the binding built FaceTrackerFunc from a
|
||||
# Config alone, and the tracker had required a registry and a calibration since.
|
||||
# VR-011 replaced the three per-node factories with one `add_pipeline` that
|
||||
# builds the chain in main.cpp's order, which is the only order that satisfies
|
||||
# those dependencies, so the failure mode cannot recur from Python.
|
||||
option(SAE_BUILD_KPN_BINDINGS "Build the sae_kpn Python module" ON)
|
||||
if(SAE_BUILD_KPN_BINDINGS)
|
||||
nanobind_add_module(sae_kpn src/kpn_bindings.cpp)
|
||||
target_link_libraries(sae_kpn PRIVATE sae_gallery)
|
||||
endif()
|
||||
nanobind_add_module(sae_kpn src/kpn_bindings.cpp)
|
||||
target_link_libraries(sae_kpn PRIVATE sae_gallery)
|
||||
|
||||
# ── sae_audio — Python module: the v1 audio signature (IR-004) ────────────────
|
||||
# Compiles audio_signature.cpp directly and links only FFmpeg, rather than
|
||||
@@ -370,43 +321,16 @@ target_link_libraries(sae_audio PRIVATE ffmpeg_libs)
|
||||
# HDF5 already found above (before sae_gallery); vars HDF5_CXX_LIBRARIES / _INCLUDE_DIRS
|
||||
# are reused by scene_analyze / dump_embeddings below.
|
||||
|
||||
# The learned scene-boundary detector is compiled into the sink (result_sink →
|
||||
# xgb_scene_boundary + audio_logpsd) when SAE_SCENE_XGB is on, so the analysis
|
||||
# binaries need xgboost + FFTW + ffmpeg and the define. Found once here.
|
||||
if(SAE_SCENE_XGB)
|
||||
find_library(FFTW3_LIB fftw3 REQUIRED)
|
||||
set(SAE_SCENE_LIBS xgboost ${FFTW3_LIB} ffmpeg_libs)
|
||||
set(SAE_SCENE_DEFS SAE_SCENE_XGB)
|
||||
else()
|
||||
set(SAE_SCENE_LIBS "")
|
||||
set(SAE_SCENE_DEFS "")
|
||||
endif()
|
||||
|
||||
# ── analyze — main analysis binary ───────────────────────────────────────────
|
||||
add_executable(scene_analyze src/main.cpp)
|
||||
target_link_libraries(scene_analyze PRIVATE sae_gallery ${HDF5_CXX_LIBRARIES} ${SAE_SCENE_LIBS})
|
||||
target_link_libraries(scene_analyze PRIVATE sae_gallery ${HDF5_CXX_LIBRARIES})
|
||||
target_include_directories(scene_analyze PRIVATE ${HDF5_INCLUDE_DIRS})
|
||||
target_compile_definitions(scene_analyze PRIVATE ${SAE_SCENE_DEFS})
|
||||
|
||||
# ── xgb_boundary_parity — prove C++ scene-boundary inference matches Python ───
|
||||
if(SAE_SCENE_XGB)
|
||||
add_executable(xgb_boundary_parity src/tools/xgb_boundary_parity.cpp)
|
||||
target_include_directories(xgb_boundary_parity PRIVATE src ${HDF5_INCLUDE_DIRS})
|
||||
target_link_libraries(xgb_boundary_parity PRIVATE
|
||||
xgboost ${HDF5_CXX_LIBRARIES} ${FFTW3_LIB} ffmpeg_libs)
|
||||
|
||||
# Dumps the C++ feature matrix so training uses the exact inference features.
|
||||
add_executable(scene_features_dump src/tools/scene_features_dump.cpp)
|
||||
target_include_directories(scene_features_dump PRIVATE src ${HDF5_INCLUDE_DIRS})
|
||||
target_link_libraries(scene_features_dump PRIVATE
|
||||
xgboost ${HDF5_CXX_LIBRARIES} ${FFTW3_LIB} ffmpeg_libs)
|
||||
endif()
|
||||
|
||||
# ── analyze_debug — same binary with debug frame/crop output ─────────────────
|
||||
add_executable(scene_analyze_debug src/main.cpp)
|
||||
target_link_libraries(scene_analyze_debug PRIVATE sae_gallery ${HDF5_CXX_LIBRARIES} ${SAE_SCENE_LIBS})
|
||||
target_link_libraries(scene_analyze_debug PRIVATE sae_gallery ${HDF5_CXX_LIBRARIES})
|
||||
target_include_directories(scene_analyze_debug PRIVATE ${HDF5_INCLUDE_DIRS})
|
||||
target_compile_definitions(scene_analyze_debug PRIVATE SAE_DEBUG=1 ${SAE_SCENE_DEFS})
|
||||
target_compile_definitions(scene_analyze_debug PRIVATE SAE_DEBUG=1)
|
||||
|
||||
# ── dump_embeddings — standalone embedding dumper, NO gallery/matcher ─────────
|
||||
# Front-half only (decode→detect→align→embed→HDF5) for the optimizer replay corpus
|
||||
|
||||
@@ -1,342 +0,0 @@
|
||||
# sae-builder-cpu — the CI build image
|
||||
#
|
||||
# TRACES: DP-007 | PR-004
|
||||
#
|
||||
# Build/push: scripts/ci/build_builder_image.sh --push
|
||||
# Consumed by: .gitea/workflows/unit-tests.yml (pinned by tag, never :latest)
|
||||
# Docs: docs/ci-image.md
|
||||
#
|
||||
# This is the CPU corner of the DP-008 builder matrix and the DP-007 CI image at
|
||||
# the same time — one artifact, two uses. The CUDA and ROCm siblings differ only
|
||||
# in the accelerator stack layered on top of this dependency set.
|
||||
#
|
||||
# CI runs on an Intel N100 with no discrete GPU. Everything here is chosen so
|
||||
# that `-DSAE_INFERENCE_BACKEND=ORT -DSAE_GEMM_BACKEND=CPU -DSAE_BUILD_TESTS=ON`
|
||||
# configures, builds and runs without a GPU, without a model, and without
|
||||
# reaching GitHub.
|
||||
|
||||
# ─── Base image ──────────────────────────────────────────────────────────────
|
||||
#
|
||||
# Chosen for the OLDEST glibc to be supported, not for recency. A binary built
|
||||
# in a container runs against the *host's* glibc; glibc is backward compatible
|
||||
# but not forward, so the build base sets the floor for every machine DP-008's
|
||||
# binaries can ever run on. Building on a newer base than the oldest supported
|
||||
# host produces the classic `GLIBC_2.xx not found` failure at load time.
|
||||
#
|
||||
# Debian 12 "bookworm" = glibc 2.36 (Aug 2022). What that floor covers:
|
||||
#
|
||||
# Distro glibc Covered?
|
||||
# Arch / CachyOS (rolling) 2.41+ yes
|
||||
# Fedora 37 and later 2.36+ yes ← DP-005's targets are Fedora+Arch
|
||||
# Debian 12 / 13 2.36+ yes
|
||||
# Ubuntu 24.04 LTS 2.39 yes
|
||||
# Ubuntu 22.04 LTS 2.35 NO
|
||||
# RHEL / Rocky / Alma 9 2.34 NO
|
||||
# Debian 11 2.31 NO
|
||||
#
|
||||
# The three misses are accepted deliberately: DP-005 puts Debian/Ubuntu out of
|
||||
# installer scope and names Fedora + Arch as the supported distros, and every
|
||||
# supported Fedora is 2.36 or newer. Going lower costs the toolchain rather than
|
||||
# buying reach — Debian 11 ships GCC 10 (incomplete C++20) and Python 3.9, which
|
||||
# has no `tomllib` and therefore cannot read the traceability gate's
|
||||
# traceability.toml.
|
||||
#
|
||||
# Escape hatch, recorded now so it is not rediscovered under pressure: if the
|
||||
# floor must drop to glibc 2.28 (RHEL 8 / manylinux_2_28 — the same baseline the
|
||||
# ONNX Runtime and PyTorch wheels target), the move is a Rocky 8 base plus
|
||||
# gcc-toolset-13, and OpenCV/FFmpeg/HDF5 all leave apt for source or
|
||||
# EPEL/RPM Fusion. That is a different image, not a flag on this one.
|
||||
#
|
||||
# Not a glibc problem but worth stating: the binaries this image produces also
|
||||
# link OpenCV, FFmpeg and HDF5 shared objects by soname. Making a *portable*
|
||||
# release binary (DP-008) is a separate question from the glibc floor, and is
|
||||
# answered by static linking or bundling, not by the base image.
|
||||
FROM debian:12-slim
|
||||
|
||||
# Pins. Every version this image installs from source is an ARG so a rebuild is
|
||||
# a one-line diff and `docker history` records what a given tag actually holds.
|
||||
#
|
||||
# ORT 1.28.0 and OpenCV 5.0.0 match the developer machine, so CI and local
|
||||
# builds exercise the same libraries rather than merely similar ones.
|
||||
# Catch2 / nlohmann_json / nanobind match the FetchContent pins in
|
||||
# CMakeLists.txt:248 and tests/CMakeLists.txt:12 exactly — a vendored copy at a
|
||||
# different version would be a silent divergence, not a convenience.
|
||||
ARG ORT_VERSION=1.28.0
|
||||
ARG OPENCV_VERSION=5.0.0
|
||||
ARG CATCH2_VERSION=v3.5.3
|
||||
ARG NLOHMANN_JSON_VERSION=v3.11.3
|
||||
ARG NANOBIND_VERSION=v2.4.0
|
||||
|
||||
# Stamped so a build can prove which image it ran in, and so a green tick can be
|
||||
# traced back to a specific dependency set. See the "Confirm the builder image"
|
||||
# step in .gitea/workflows/unit-tests.yml.
|
||||
ARG IMAGE_TAG=dev
|
||||
ENV SAE_BUILDER=cpu \
|
||||
SAE_BUILDER_VERSION=${IMAGE_TAG} \
|
||||
SAE_ORT_VERSION=${ORT_VERSION} \
|
||||
SAE_OPENCV_VERSION=${OPENCV_VERSION} \
|
||||
DEBIAN_FRONTEND=noninteractive
|
||||
|
||||
# ─── System dependencies ─────────────────────────────────────────────────────
|
||||
#
|
||||
# One layer, ordered by why it is here rather than alphabetically.
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
# Toolchain. bookworm's default gcc is 12.2 — enough for the C++20 the
|
||||
# project sets unconditionally (CMakeLists.txt:4). cmake is 3.25, above the
|
||||
# 3.21 minimum. Ninja because the N100 has four cores and every second of
|
||||
# build scheduling shows.
|
||||
build-essential \
|
||||
cmake \
|
||||
ninja-build \
|
||||
pkg-config \
|
||||
git \
|
||||
ca-certificates \
|
||||
curl \
|
||||
# Gitea's act_runner executes JS actions (actions/checkout, upload-artifact)
|
||||
# with the `node` found *inside* the container. Without this the job cannot
|
||||
# even check the repository out. Same reason as the kpnpp-builder image.
|
||||
nodejs \
|
||||
# HDF5 with the C++ API: galleries are HDF5-native and it is also the VR-001
|
||||
# dump format. find_package(HDF5 COMPONENTS CXX) at CMakeLists.txt:273.
|
||||
libhdf5-dev \
|
||||
# FFmpeg decode. swresample is on this list deliberately: the audio
|
||||
# signature (IR-004) downmixes to mono and resamples to 11025 Hz, and
|
||||
# tests/test_audio_signature.cpp decodes the golden FLAC fixture, so the
|
||||
# test build needs it as much as the main build does.
|
||||
libavformat-dev \
|
||||
libavcodec-dev \
|
||||
libavutil-dev \
|
||||
libswscale-dev \
|
||||
libswresample-dev \
|
||||
# OpenBLAS — required here, not optional. CI has no GPU, so SAE_GEMM_BACKEND
|
||||
# =CPU is the only path it ever exercises, and without OpenBLAS the CPU GEMM
|
||||
# falls back to a scalar loop that does not scale against a library-sized
|
||||
# gallery (AR-027). The build only *warns* when it is missing so a developer
|
||||
# without it still gets a working tree; the image must never be that case.
|
||||
# Both the main build (CMakeLists.txt:162) and the test target
|
||||
# (tests/CMakeLists.txt:42) discover it through pkg-config `openblas`.
|
||||
libopenblas-dev \
|
||||
# Python: the build itself needs the interpreter and headers
|
||||
# (find_package(Python COMPONENTS Interpreter Development.Module) at
|
||||
# CMakeLists.txt:254, for the nanobind modules). numpy/h5py/scipy are for
|
||||
# the Python-side tooling — fixture generation, replay, validation scripts.
|
||||
# From apt rather than pip: bookworm marks the environment externally
|
||||
# managed (PEP 668), and apt's h5py is already linked against the same
|
||||
# libhdf5 installed above. bookworm's python3 is 3.11, which has tomllib —
|
||||
# the traceability gate needs it to read traceability.toml.
|
||||
python3 \
|
||||
python3-dev \
|
||||
python3-numpy \
|
||||
python3-h5py \
|
||||
python3-scipy \
|
||||
# Image codecs for the OpenCV build below. Without these OpenCV silently
|
||||
# builds an imgcodecs that cannot read a JPEG, which fails at run time in a
|
||||
# gallery build rather than at compile time here.
|
||||
libjpeg62-turbo-dev \
|
||||
libpng-dev \
|
||||
libtiff-dev \
|
||||
libwebp-dev \
|
||||
libopenjp2-7-dev \
|
||||
zlib1g-dev \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Fail the image build, not the CI run, if OpenBLAS or swresample are not
|
||||
# discoverable the way CMakeLists.txt discovers them. An image that ships
|
||||
# libopenblas but no openblas.pc would compile the scalar fallback in silence.
|
||||
RUN set -eux; \
|
||||
pkg-config --exists openblas; \
|
||||
echo "openblas $(pkg-config --modversion openblas)"; \
|
||||
pkg-config --exists libswresample; \
|
||||
echo "swresample $(pkg-config --modversion libswresample)"
|
||||
|
||||
# ─── ONNX Runtime, CPU provider only ─────────────────────────────────────────
|
||||
#
|
||||
# The official prebuilt linux-x64 tarball is the CPU build: no CUDA, no
|
||||
# TensorRT, no ROCm execution providers. That is the whole requirement here —
|
||||
# excluding the GPU providers is not a size optimisation, it is the point.
|
||||
#
|
||||
# Verified against the 1.28.0 tarball: the shared object's highest versioned
|
||||
# symbol requirement is GLIBC_2.27 / GLIBCXX_3.4.21, well under this base's
|
||||
# 2.36, so ORT does not raise the floor set above.
|
||||
#
|
||||
# Installed to /usr/local/{lib,include/onnxruntime} because CMakeLists.txt
|
||||
# includes <onnxruntime/onnxruntime_cxx_api.h> and needs the *parent* of that
|
||||
# directory on the include path (CMakeLists.txt:108-113).
|
||||
#
|
||||
# CI never calls a model — the embedder measures ~930 ms/frame on this CPU
|
||||
# provider — so ORT is present to satisfy the link, not to run inference.
|
||||
RUN set -eux; \
|
||||
curl -fsSL -o /tmp/ort.tgz \
|
||||
"https://github.com/microsoft/onnxruntime/releases/download/v${ORT_VERSION}/onnxruntime-linux-x64-${ORT_VERSION}.tgz"; \
|
||||
mkdir -p /tmp/ort; \
|
||||
tar -xzf /tmp/ort.tgz -C /tmp/ort --strip-components=1; \
|
||||
cp -a /tmp/ort/lib/libonnxruntime.so* /usr/local/lib/; \
|
||||
mkdir -p /usr/local/include/onnxruntime; \
|
||||
cp -a /tmp/ort/include/. /usr/local/include/onnxruntime/; \
|
||||
ldconfig; \
|
||||
rm -rf /tmp/ort /tmp/ort.tgz; \
|
||||
test -f /usr/local/include/onnxruntime/onnxruntime_cxx_api.h
|
||||
|
||||
# ─── OpenCV 5, from source ───────────────────────────────────────────────────
|
||||
#
|
||||
# This is the reason the image is prebuilt at all. CMakeLists.txt:25 probes for
|
||||
# OpenCV 5 first and falls back to 4; the branch targets 5, which no Debian
|
||||
# release ships (bookworm has 4.6), and building it inside every CI run would
|
||||
# dominate the run on an N100.
|
||||
#
|
||||
# BUILD_LIST is exactly the seven components find_package asks for
|
||||
# (CMakeLists.txt:25-29) — OpenCV resolves their internal dependencies itself.
|
||||
# Everything else is off: tests, samples, Java/Python bindings, and the apps.
|
||||
#
|
||||
# No GUI backend. highgui still builds (find_package REQUIREs the component) but
|
||||
# with a stub — CI never calls imshow, and pulling GTK/Qt into a headless build
|
||||
# image buys nothing. scene_preview is a developer tool, not a CI target.
|
||||
#
|
||||
# CUDA/cuDNN explicitly off: DP-007 excludes the GPU stack outright.
|
||||
#
|
||||
# The source tree and build tree are removed in the same layer, so the ~3 GB of
|
||||
# intermediates cost nothing in the published image.
|
||||
RUN set -eux; \
|
||||
curl -fsSL -o /tmp/opencv.tar.gz \
|
||||
"https://github.com/opencv/opencv/archive/refs/tags/${OPENCV_VERSION}.tar.gz"; \
|
||||
mkdir -p /tmp/opencv-src; \
|
||||
tar -xzf /tmp/opencv.tar.gz -C /tmp/opencv-src --strip-components=1; \
|
||||
cmake -S /tmp/opencv-src -B /tmp/opencv-build -G Ninja \
|
||||
-DCMAKE_BUILD_TYPE=Release \
|
||||
-DCMAKE_INSTALL_PREFIX=/usr/local \
|
||||
-DBUILD_LIST=core,imgproc,imgcodecs,videoio,dnn,objdetect,highgui \
|
||||
-DBUILD_SHARED_LIBS=ON \
|
||||
-DBUILD_TESTS=OFF \
|
||||
-DBUILD_PERF_TESTS=OFF \
|
||||
-DBUILD_EXAMPLES=OFF \
|
||||
-DBUILD_DOCS=OFF \
|
||||
-DBUILD_opencv_apps=OFF \
|
||||
-DBUILD_JAVA=OFF \
|
||||
-DBUILD_opencv_python3=OFF \
|
||||
-DWITH_FFMPEG=ON \
|
||||
-DWITH_GTK=OFF \
|
||||
-DWITH_QT=OFF \
|
||||
-DWITH_OPENGL=OFF \
|
||||
-DWITH_CUDA=OFF \
|
||||
-DWITH_CUDNN=OFF \
|
||||
-DOPENCV_GENERATE_PKGCONFIG=ON \
|
||||
-DCMAKE_INSTALL_RPATH=/usr/local/lib; \
|
||||
cmake --build /tmp/opencv-build --parallel; \
|
||||
cmake --install /tmp/opencv-build; \
|
||||
ldconfig; \
|
||||
rm -rf /tmp/opencv-src /tmp/opencv-build /tmp/opencv.tar.gz
|
||||
|
||||
# ─── Vendored dependencies: Catch2, nlohmann/json, nanobind ──────────────────
|
||||
#
|
||||
# All three are FetchContent'ed by the build today, which makes every CI run
|
||||
# depend on GitHub being reachable — a network outage would present as a code
|
||||
# failure. Baking them in removes that dependency entirely.
|
||||
#
|
||||
# Catch2 is *installed*, so tests/CMakeLists.txt:6 `find_package(Catch2 3 QUIET)`
|
||||
# succeeds and the FetchContent fallback is never reached. Its source is kept as
|
||||
# well so the override below can cover the case where find_package somehow does
|
||||
# not fire.
|
||||
#
|
||||
# nanobind must be cloned with submodules: its `ext/robin_map` is a git
|
||||
# submodule, and a GitHub source tarball does not contain it. This is the one
|
||||
# dependency where "download the tarball" produces a tree that configures and
|
||||
# then fails to compile.
|
||||
RUN set -eux; \
|
||||
mkdir -p /opt/vendor; \
|
||||
git clone --depth 1 --branch "${NLOHMANN_JSON_VERSION}" \
|
||||
https://github.com/nlohmann/json.git /opt/vendor/nlohmann_json; \
|
||||
git clone --depth 1 --branch "${NANOBIND_VERSION}" --recurse-submodules \
|
||||
https://github.com/wjakob/nanobind.git /opt/vendor/nanobind; \
|
||||
git clone --depth 1 --branch "${CATCH2_VERSION}" \
|
||||
https://github.com/catchorg/Catch2.git /opt/vendor/Catch2; \
|
||||
cmake -S /opt/vendor/Catch2 -B /tmp/catch2-build -G Ninja \
|
||||
-DCMAKE_BUILD_TYPE=Release \
|
||||
-DCMAKE_INSTALL_PREFIX=/usr/local \
|
||||
-DBUILD_TESTING=OFF; \
|
||||
cmake --build /tmp/catch2-build --parallel; \
|
||||
cmake --install /tmp/catch2-build; \
|
||||
rm -rf /tmp/catch2-build; \
|
||||
find /opt/vendor -maxdepth 2 -name .git -exec rm -rf {} +; \
|
||||
ldconfig
|
||||
|
||||
# The initial-cache script the build is configured with. It lives in the image,
|
||||
# not in the workflow, so the vendor paths have exactly one owner: move a
|
||||
# directory here and no consumer needs editing.
|
||||
#
|
||||
# FETCHCONTENT_FULLY_DISCONNECTED=ON is the load-bearing line. With it, any
|
||||
# FetchContent dependency that is *not* covered by an override above is a hard
|
||||
# configure error instead of a silent download — so "this build does not touch
|
||||
# GitHub" is enforced by the build system rather than asserted in a comment.
|
||||
RUN set -eux; \
|
||||
printf '%s\n' \
|
||||
'# Baked into sae-builder-cpu. Use with: cmake -C /opt/vendor/vendored-deps.cmake ...' \
|
||||
'# TRACES: DP-007' \
|
||||
'set(FETCHCONTENT_SOURCE_DIR_NLOHMANN_JSON "/opt/vendor/nlohmann_json" CACHE PATH "vendored in the CI image")' \
|
||||
'set(FETCHCONTENT_SOURCE_DIR_NANOBIND "/opt/vendor/nanobind" CACHE PATH "vendored in the CI image")' \
|
||||
'set(FETCHCONTENT_SOURCE_DIR_CATCH2 "/opt/vendor/Catch2" CACHE PATH "vendored in the CI image")' \
|
||||
'set(FETCHCONTENT_FULLY_DISCONNECTED ON CACHE BOOL "no CI build may fetch from the network")' \
|
||||
> /opt/vendor/vendored-deps.cmake; \
|
||||
cat /opt/vendor/vendored-deps.cmake
|
||||
|
||||
# ─── Self-check ──────────────────────────────────────────────────────────────
|
||||
#
|
||||
# Run the project's own dependency discovery — the same find_package and
|
||||
# pkg_check_modules calls CMakeLists.txt makes — against this image, at image
|
||||
# build time. An image that cannot satisfy them should fail here, loudly, once,
|
||||
# rather than in every CI run that pulls it.
|
||||
#
|
||||
# Deliberately not a build of the project: the image must be buildable without
|
||||
# the repository, and the repository's own configure step is what CI is for.
|
||||
RUN set -eux; \
|
||||
mkdir -p /tmp/selfcheck; \
|
||||
printf '%s\n' \
|
||||
'cmake_minimum_required(VERSION 3.21)' \
|
||||
'project(sae_image_selfcheck LANGUAGES CXX)' \
|
||||
'set(CMAKE_CXX_STANDARD 20)' \
|
||||
'set(CMAKE_CXX_STANDARD_REQUIRED ON)' \
|
||||
'find_package(OpenCV 5 REQUIRED COMPONENTS core imgproc imgcodecs videoio dnn objdetect highgui)' \
|
||||
'message(STATUS "OpenCV ${OpenCV_VERSION}")' \
|
||||
'find_package(HDF5 REQUIRED COMPONENTS CXX)' \
|
||||
'message(STATUS "HDF5 ${HDF5_VERSION}")' \
|
||||
'find_package(Catch2 3 REQUIRED)' \
|
||||
'message(STATUS "Catch2 ${Catch2_VERSION}")' \
|
||||
'find_package(Python 3.8 REQUIRED COMPONENTS Interpreter Development.Module)' \
|
||||
'find_package(PkgConfig REQUIRED)' \
|
||||
'pkg_check_modules(AVFORMAT REQUIRED libavformat)' \
|
||||
'pkg_check_modules(AVCODEC REQUIRED libavcodec)' \
|
||||
'pkg_check_modules(AVUTIL REQUIRED libavutil)' \
|
||||
'pkg_check_modules(SWSCALE REQUIRED libswscale)' \
|
||||
'pkg_check_modules(SWRESAMPLE REQUIRED libswresample)' \
|
||||
'pkg_check_modules(OPENBLAS REQUIRED openblas)' \
|
||||
'find_library(ORT_LIB onnxruntime REQUIRED HINTS /usr/lib /usr/local/lib)' \
|
||||
'find_path(ORT_INCLUDE onnxruntime_cxx_api.h PATH_SUFFIXES onnxruntime' \
|
||||
' HINTS /usr/include/onnxruntime /usr/local/include/onnxruntime /usr/local/include REQUIRED)' \
|
||||
'message(STATUS "ORT ${ORT_LIB} / ${ORT_INCLUDE}")' \
|
||||
> /tmp/selfcheck/CMakeLists.txt; \
|
||||
cmake -S /tmp/selfcheck -B /tmp/selfcheck/build -G Ninja; \
|
||||
rm -rf /tmp/selfcheck
|
||||
|
||||
# Python-side tooling the fixture and validation scripts import. Checked here so
|
||||
# a missing wheel is an image failure rather than a mid-run traceback.
|
||||
RUN python3 -c "import numpy, h5py, scipy; print('numpy', numpy.__version__, 'h5py', h5py.__version__, 'scipy', scipy.__version__)"
|
||||
|
||||
# ─── What is deliberately NOT here ───────────────────────────────────────────
|
||||
#
|
||||
# CUDA, TensorRT, ROCm, and the ORT GPU execution providers
|
||||
# No GPU to use them. They belong to the sae-builder-cuda and
|
||||
# sae-builder-rocm siblings (DP-008).
|
||||
#
|
||||
# The ONNX models
|
||||
# Seven files, ~725 MB, in Git LFS. T1/T2 tests are model-free by design
|
||||
# (tests/CMakeLists.txt:1-4), so the CI image needs none of them, and
|
||||
# baking them in would inflate the image roughly tenfold to serve the T3
|
||||
# smoke tests alone. Those pull the model they need via LFS in a separate
|
||||
# job. The CI workflow checks out with LFS off for the same reason.
|
||||
#
|
||||
# The repository
|
||||
# Nothing from the source tree is COPYed in. The image is a toolchain, and
|
||||
# a toolchain that embeds the code it builds has to be rebuilt whenever the
|
||||
# code changes — which is exactly the per-run cost this image exists to
|
||||
# avoid.
|
||||
|
||||
WORKDIR /src
|
||||
+113
-240
@@ -128,43 +128,10 @@ Two consequences worth stating:
|
||||
depends on timing. The same command run twice can produce different dumps, and
|
||||
a golden fixture cannot be built on that.
|
||||
|
||||
**Current:** fixed in KPN. Node data outputs *park* on a full channel — the
|
||||
value is held in a one-slot buffer, the worker is released, and the channel's
|
||||
space callback resubmits the node once the consumer drains. That replaced
|
||||
`push_blocking`, which slept inside the push and, with one thread per node,
|
||||
stopped that node draining its own input. Sentinels remain out-of-band so EOF
|
||||
can always overtake a stalled data path. Verified on the same clip: 385 of 385
|
||||
sampled frames written, zero drops, and two consecutive runs byte-identical
|
||||
where previously they were not.
|
||||
|
||||
A later audit found the losslessness was still incomplete in three places, all
|
||||
now closed and each pinned by a regression case in the KPN suite:
|
||||
|
||||
- **`FilterNode` and `RouterNode`** were the last data paths still using the
|
||||
throwing `push()` with the exception swallowed. A full output discarded the
|
||||
value, and that included the **EOF sentinel**. The decimator passes EOF by
|
||||
predicate but its output is reliably full — the embedder is the slowest node
|
||||
in the chain — so the token was discarded, nothing downstream shut down, and
|
||||
the run had to be killed. This was the wedge.
|
||||
- **The sentinel could arrive ahead of a value still queued behind it.** `pop()`
|
||||
observed the ring empty and then took the sentinel; a producer can push a
|
||||
value *and* publish the sentinel inside that window, so a consumer treating
|
||||
EOF as a hard stop loses the tail.
|
||||
- **Two firings of one node could overlap**, because the submit gate was
|
||||
released before the firing had finished with the node's state. That breaks the
|
||||
one-slot park itself: a parked value can be overwritten by the other firing,
|
||||
with no drop recorded anywhere.
|
||||
|
||||
**New constraint:** a channel carries at most one undelivered sentinel. A second
|
||||
offered before the first is taken is refused and reported, never queued and
|
||||
never overwritten — two control tokens on one channel means the stream ended
|
||||
twice. Single-shot EOF is what everything does today; this becomes live the
|
||||
moment a pipeline is reused for a second input.
|
||||
|
||||
**Consequence:** a lossless decimator is a backpressure point, not a relief
|
||||
valve. The source now throttles to the face branch rather than quietly thinning
|
||||
it. That is what this requirement asks for, but it changes the shape of a loaded
|
||||
run and has not yet been benchmarked.
|
||||
**Current:** fixed in KPN — node data outputs use `push_blocking`, sentinels
|
||||
remain out-of-band so EOF can always overtake a stalled data path. Verified on
|
||||
the same clip: 385 of 385 sampled frames written, zero drops, and two
|
||||
consecutive runs byte-identical where previously they were not.
|
||||
|
||||
It also ran *faster* (29 s → 17 s). A dropped frame has already cost its decode,
|
||||
and the overflow exception cost more — so the lossy path was paying for work it
|
||||
@@ -281,31 +248,29 @@ the same response.
|
||||
- **Size** — already AR-002, floor at 40×40 px in original resolution, measured
|
||||
end to end by VR-013. It is the precedent for the other two: the
|
||||
threshold was *located*, not chosen.
|
||||
- **Sharpness** — motion blur and soft focus destroy the high-frequency detail
|
||||
the embedder keys on, and unlike size they leave the bounding box looking
|
||||
perfectly healthy. Measured on the **112×112 aligned crop**, not the raw box:
|
||||
the crop is already scale-normalised, so a measure taken there cannot silently
|
||||
re-measure face size and double-count it against AR-002.
|
||||
- **Sharpness** — motion blur and optical defocus destroy the high-frequency
|
||||
detail the embedder keys on, and unlike size they leave the bounding box
|
||||
looking perfectly healthy. Measured on the **112×112 aligned crop**, not the
|
||||
raw box.
|
||||
|
||||
The measure is the **variance of the Laplacian divided by the variance of the
|
||||
crop** — `crop_sharpness()`, dimensionless. The division is the part that
|
||||
earns its place: a raw Laplacian variance, the textbook measure, scales with
|
||||
the square of image contrast, so a dim scene reads as soft and a graded-up one
|
||||
as sharp, and VR-012 would locate a different knee in every film. That is
|
||||
AR-024's objection to the raw cosine in another metric. Normalised, the axis
|
||||
means the same thing everywhere, which is the precondition for a single knee
|
||||
existing at all.
|
||||
An earlier version of this clause argued the crop is scale-normalised and so a
|
||||
measure taken there "cannot re-measure face size and double-count it against
|
||||
AR-002". **That reasoning is wrong and VR-012 measured it wrong.** The
|
||||
normalisation is geometric, not informational: a 40 px face upscaled into the
|
||||
canonical frame genuinely carries less high-frequency content than a 400 px
|
||||
one downscaled into it, so every candidate measure *does* respond to source
|
||||
size. What the crop yields is **effective resolution in canonical space** —
|
||||
the union of "was small" and "was blurred", not blur alone.
|
||||
|
||||
Read spectrally it is `E[|ω|⁴]` under the crop's own energy distribution, so
|
||||
the blur ladder is monotone by construction rather than by fitting: Gaussian
|
||||
blur multiplies that distribution by `e^{-σ²|ω|²}`, which can only move mass
|
||||
downward. Two consequences follow from the same identity and are recorded on
|
||||
the function: it needs the low-frequency mass real images have (on a
|
||||
flat-spectrum synthetic an anisotropic smear makes it *rise*, because the
|
||||
surviving perpendicular detail really is as fine as before), and it conflates
|
||||
focus with intrinsic texture, so a bearded face outscores a smooth one at equal
|
||||
focus. Both are true of every no-reference sharpness measure, and both are
|
||||
reasons AR-028 carries the number rather than thresholding on it.
|
||||
The conclusion survives, for a better reason. VR-012 sorted its grid by
|
||||
measured sharpness and found the six cells at effectively identical sharpness
|
||||
(0.0003–0.0005) spanning **15.3% to 91.0% TPI**, ordered entirely by source
|
||||
size. Sharpness is therefore not a sufficient statistic for identity loss: a
|
||||
scalar keyed on high-frequency energy cannot separate *attenuated* high
|
||||
frequencies from *destroyed* spatial sampling, because blur preserves
|
||||
mid-frequency facial geometry exactly while downsampling destroys it. The two
|
||||
axes are not redundant and neither substitutes for the other — which is what
|
||||
"not collapsed into one scalar" above now rests on.
|
||||
- **Visibility** — extreme pose or occlusion means the face presents fewer of the
|
||||
features the embedding assumes are present. The measure is the **residual of
|
||||
the AR-005 alignment fit**: the RMS landmark error, in canonical 112×112
|
||||
@@ -386,43 +351,63 @@ hand-chosen cutoff on an uncalibrated measure is the same unfalsifiable magic
|
||||
number AR-024 retired for similarity, and it would fail the same way: meaning
|
||||
something different for every detector, every embedder and every film.
|
||||
|
||||
**Current:** all three axes are measured and carried, and the vector reaches the
|
||||
dump. `FaceAlignerFunc` is where it is filled in, because both measured axes fall
|
||||
out of work the warp already does: visibility is the residual from
|
||||
`estimate_alignment()`, and sharpness is `crop_sharpness()` on the 112×112 crop
|
||||
the node has just produced. Size stays `bbox` — deliberately not copied into a
|
||||
field of its own, since that would hold the same quantity in two coordinate
|
||||
spaces and the copy is the one that drifts. No face is admitted unscored, so a
|
||||
negative value downstream is a bug rather than a poor-quality face. The
|
||||
degenerate-fit case is still dropped — it has no crop and no fit to score — but
|
||||
is now **counted** and reported once at EOF instead of vanishing.
|
||||
**A discount curve on sharpness must be flat, then steep.** VR-012 measured the
|
||||
response as a cliff rather than a gradient: Gaussian sigma up to 1.5 costs under
|
||||
1.5 points of TPI in every cell — at 16 px it is very slightly *positive*,
|
||||
smoothing upscale artifacts — sigma 2 costs 1–3, and the 2→3 step costs 7–19. A
|
||||
linear or sigmoid discount over the measure would penalise the whole flat region
|
||||
where blur demonstrably costs nothing.
|
||||
|
||||
`sharpness` and `alignment_residual` are written to the VR-001 dump as per-face
|
||||
columns parallel to `confidence`, taking the dump to `schema_version` 2. The bump
|
||||
is not for readers — both sides check by name, and a v1 dump still replays — but
|
||||
so that a consumer of the vector can tell *never scored* from *scored zero*,
|
||||
which is a real reading on this axis. Nothing yet *consumes* any of it.
|
||||
**Which blur is modelled is a first-order decision, not a detail.** VR-012 swept
|
||||
three families at matched per-axis PSF spread, and at σ=3 px on a 112 px face
|
||||
they cost 9%, 18% and **53%** error for Gaussian, motion and optical defocus
|
||||
respectively. Defocus is the destructive one because its disc PSF has a jinc
|
||||
transfer function with **exact zeros** — bands annihilated rather than
|
||||
attenuated — where a Gaussian merely rolls off. It is also the case AR-002
|
||||
cannot catch, since a defocused face is large and confidently detected. Any
|
||||
future study that sweeps blur states its family and its justification; a
|
||||
Gaussian-only sweep understated the effect by a factor of five and would have
|
||||
retired this axis as not worth its cost.
|
||||
|
||||
**Gap:** three, in the order they block each other.
|
||||
**The cost of blur is proportional to proximity to the decision boundary, not to
|
||||
blur itself.** Sigma 3 costs −22.5 points at 24 px, but only −7.9 at 112 px
|
||||
(margin to spare) and −8.3 at 16 px (already below threshold). This is why the
|
||||
axes must combine multiplicatively in `EvidenceDiscounter` rather than each
|
||||
gating independently.
|
||||
|
||||
1. **The fixtures do not carry the vector.** They are v1, and re-dumping needs a
|
||||
GPU host (`scripts/make_fixtures.sh`), so until that runs VR-012 has recorded
|
||||
data available in principle and none in hand.
|
||||
2. **AR-030's discount does not exist.** The measure must reach
|
||||
`EvidenceDiscounter` as the reliability term, multiplying the novelty weight
|
||||
rather than replacing it.
|
||||
3. **Two properties of the sharpness measure are recorded but unquantified on
|
||||
real faces**, and both distort the low end of the axis, which is where a knee
|
||||
would go. It is exactly contrast-invariant in the algebra, but the 8-bit
|
||||
quantisation floor lands in the numerator, so a crop that is *dim and soft*
|
||||
reads sharper than it is — on the synthetic ladder a half-contrast copy reads
|
||||
0.9% high when sharp and 148% high at σ 2.5. Separately, `align_face` warps
|
||||
with `BORDER_CONSTANT`, so a face crossing the frame edge brings a hard black
|
||||
step into the crop, and a step edge is high-frequency; the normalisation
|
||||
blunts this but does not remove it. Neither is corrected here. The candidate
|
||||
fixes are a validity mask or a different border mode, and the second changes
|
||||
what the embedder is fed (AR-011) — so VR-012 measures the size of each effect
|
||||
on the dumped distribution first, and no correction is chosen before that.
|
||||
**Sharpness discounts; it must never gate.** VR-012 tried the gate directly, as
|
||||
a compute saving: skipping the embed below a sharpness threshold costs 15.1% of
|
||||
true identifications to save 20% of the work, against the size filter's 4.7% at
|
||||
16.7% — three times the damage, from a measure that needs the warped crop plus a
|
||||
DFT where size is a bbox dimension available for free. The reason is a ceiling
|
||||
no measure can beat: **at 112 px with defocus radius 6 — visually destroyed —
|
||||
46.9% of faces still identify correctly, and rank-1 is still 94.8%.** Apparent
|
||||
blur does not determine the outcome. The size filter wins only because smallness
|
||||
destroys identity more completely than blur does (16 px succeeds 23.5% of the
|
||||
time), and that asymmetry is the measured justification for the rule above:
|
||||
**failing sharpness discounts the observation, failing size may drop it.**
|
||||
|
||||
**Current:** visibility is measured and carried — `estimate_alignment()` in
|
||||
`src/face_utils.hpp` returns the residual alongside the transform, and
|
||||
`FaceAlignerFunc` writes it to `DetectedFace::alignment_residual`. Sharpness is
|
||||
measured: `assess_sharpness()` in `src/quality.hpp` returns five AR-029
|
||||
candidates over a fixed 64×64 window on the face interior, and VR-012 has ranked
|
||||
them — `var_laplacian` and `tenengrad` are disqualified as discounts (see
|
||||
AR-029), leaving `hf_energy_ratio` as the only correctly-signed survivor. Size
|
||||
is `min_face_px` (40, decoded-frame space — AR-002 still open). All three are
|
||||
exposed to studies through `sae_embed`. Nothing yet *consumes* any of it: no
|
||||
discount is applied, and `align_face()` still drops the degenerate-fit case
|
||||
without counting it.
|
||||
|
||||
**Gap:** the discount itself, on every axis. Neither sharpness nor the residual
|
||||
reaches `EvidenceDiscounter`, whose weight remains pure novelty — so a profile
|
||||
or defocused view still moves a track's belief hardest when it deserves the
|
||||
least trust. Neither reaches the VR-001 dump either, so VR-012 must still re-run
|
||||
video rather than replay fixtures. VR-012's **pose half is not started**: the
|
||||
AR-030 residual has no arm in the grid, so whether the 5-point proxy suffices or
|
||||
a dedicated landmark model is needed remains open. And the sharpness result is
|
||||
weak enough (best within-cell AUC 0.530) that whether AR-029 earns a discount at
|
||||
all is still a judgement, not a measurement.
|
||||
|
||||
## AR-007, AR-008 — Tracking
|
||||
|
||||
@@ -516,7 +501,7 @@ Both feed AR-007 as **association hints**: they tell the tracker that spatial
|
||||
continuity is broken and that association should weight embedding over IoU.
|
||||
Neither ends a presence window (AR-012).
|
||||
|
||||
In dense mode the source decodes at `scene_decode_fps` (default 0 = native) and a
|
||||
In dense mode the source decodes at `scene_decode_fps` (default 12) and a
|
||||
decimator splits the stream: full-resolution sampled frames to the face pipeline,
|
||||
downscaled dense frames to the scene detector
|
||||
(`frame_source_node.hpp:63`). `sample_fps` is independent of this — the face
|
||||
@@ -537,44 +522,31 @@ degrading what a single inference sees. A model run off-distribution produces
|
||||
confident, plausible, wrong output, and the error is invisible without a study
|
||||
that should not have been necessary.
|
||||
|
||||
Two places this was violated, both now closed:
|
||||
Two places this is currently violated:
|
||||
|
||||
1. **`scene_decode_fps = 12` starved TransNetV2.** `kWindow` is 100 frames. At
|
||||
native 25 fps that window spans ~4 s; at 12 fps it spanned ~8.3 s, so the
|
||||
model saw roughly half-speed motion over twice the temporal context it was
|
||||
trained on. **Requirement: feed TransNetV2 at the source's native frame
|
||||
rate**, so a 100-frame window covers the duration the model expects. The
|
||||
"tolerates ~12fps" note in `config.hpp` described a compromise, and the
|
||||
recorded margin was consistent with it — a non-boundary baseline at ~0.50 with
|
||||
1. **`scene_decode_fps = 12` starves TransNetV2.** `kWindow` is 100 frames. At
|
||||
native 25 fps that window spans ~4 s; at 12 fps it spans ~8.3 s, so the model
|
||||
sees roughly half-speed motion over twice the temporal context it was trained
|
||||
on. **Requirement: feed TransNetV2 at the source's native frame rate**, so a
|
||||
100-frame window covers the duration the model expects. The
|
||||
"tolerates ~12fps" note in `config.hpp` describes a compromise, and the
|
||||
recorded margin is consistent with it — a non-boundary baseline at ~0.50 with
|
||||
real boundaries reaching only ~0.7+ is a compressed separation, not a healthy
|
||||
one. **Done:** `scene_decode_fps` defaults to 0.
|
||||
one.
|
||||
|
||||
2. **Hardcoded 25 fps in boundary dedup.** The node merged boundaries closer than
|
||||
`0.04 s` — "~1 frame @25fps". **Requirement: derive this from the source's
|
||||
actual frame rate. Done:** `SceneDetectorFunc::dedup_window_sec()` takes the
|
||||
median of the frame intervals the detector was actually fed and halves it.
|
||||
Half a frame rather than a whole one, because the only thing being merged is
|
||||
one frame scored by two overlapping windows; two distinct frames are a full
|
||||
interval apart and both have to survive.
|
||||
|
||||
The two are one change, not two. A native-rate stream is where the old constant
|
||||
did the most damage — at 30 fps, 0.04 s is wider than a frame, so two cuts on
|
||||
consecutive frames merged into one and the loss showed up nowhere: the file
|
||||
simply had fewer boundaries.
|
||||
2. **Hardcoded 25 fps in boundary dedup.** `scene_detector_node.hpp:138` merges
|
||||
boundaries closer than `0.04 s` — "~1 frame @25fps". **Requirement: derive
|
||||
this from the source's actual frame rate.**
|
||||
|
||||
Dense decode is the pipeline's cost driver, so (1) is not free. The cost is
|
||||
accepted: the alternative is a boundary signal that steers association (AR-007) while
|
||||
being quietly unreliable. `dense_scale` remains available as a spatial reduction,
|
||||
since downscaling is a documented, understood degradation rather than a temporal
|
||||
one the model has no defence against — and TransNetV2 downsamples to 48×27
|
||||
regardless.
|
||||
one the model has no defence against.
|
||||
|
||||
**Current:** histogram cut in the decoder; `scene_detector_node.hpp` for
|
||||
TransNetV2, fed at native rate with a framerate-derived dedup window.
|
||||
**Gap:** `scene_threshold` (0.60) is still the value picked against 12 fps input
|
||||
and is now certainly wrong — VR-006 re-fits it, and until it does, boundary
|
||||
recall at native rate is untuned rather than better. `--scene-detect` is
|
||||
default-off despite now feeding association.
|
||||
TransNetV2. **Gap:** native-rate dense decode; framerate-derived dedup;
|
||||
`--scene-detect` is default-off despite now feeding association.
|
||||
|
||||
## AR-012 … AR-017 — Track-level identity propagation — **CHANGED BEHAVIOUR**
|
||||
|
||||
@@ -757,30 +729,12 @@ An embedding is admitted only if its similarity to one already in the store fall
|
||||
admitting it risks poisoning the store.
|
||||
|
||||
A starting band of roughly **0.90–0.95** is the working estimate, to be tuned
|
||||
(VR-007). Note this is deliberately conservative compared to the retired
|
||||
`expand_novelty_sim` (0.55), which promoted embeddings *far* from the gallery —
|
||||
(VR-007). Note this is deliberately conservative compared to the current
|
||||
`expand_novelty_sim` (0.55), which promotes embeddings *far* from the gallery —
|
||||
much more aggressive, and much more exposed to admitting the wrong person.
|
||||
|
||||
Both bounds must be expressed as calibrated probabilities, not raw cosines (AR-024).
|
||||
|
||||
The lower bound is asked twice. `admit` compares a newcomer against its
|
||||
*closest* existing member, which a gradually drifting track can chain past: every
|
||||
step inside the band while the endpoints are strangers — the shape a track-ID
|
||||
collision takes over a slow pan. So the same bound is re-applied across **every
|
||||
pair** in the store before promotion. One bound, two enforcement points; not a
|
||||
second constant.
|
||||
|
||||
Novelty is deliberately **not** a threshold. The store's eviction policy orders
|
||||
its members by similarity to the actor's existing references and drops the
|
||||
best-recognised one, so novelty-seeking is a ranking with nothing to tune, and
|
||||
the band's upper bound already refuses the redundant views at the door.
|
||||
|
||||
**Current:** implemented in `gallery/track_gallery.hpp` — `admit` at the door,
|
||||
`store_coherence` at promotion, both bounds from `Config::expand_band_lo/hi`.
|
||||
Refusals are counted (`band_rejected`).
|
||||
|
||||
**Gap:** the bounds themselves are unswept working values (VR-007).
|
||||
|
||||
### AR-019 — Expansion of known actors
|
||||
|
||||
When a track is owned (AR-012), its store is promoted into a **per-film, in-memory
|
||||
@@ -881,14 +835,14 @@ natural unit for anonymous presence, should that be adopted (AR-012, TBD).
|
||||
open (VR-007).
|
||||
|
||||
**Current:** `src/gallery/track_gallery.hpp` implements a per-track diversity
|
||||
buffer with eviction biased to gallery-far poses, admission and promotion both
|
||||
gated on the AR-018 band in probability space, cleared on `is_cut`. Wired at
|
||||
`identity_matcher_node.hpp:227`, cleared at `:126`; the calibration is handed
|
||||
over at `:114`.
|
||||
buffer with eviction biased to gallery-far poses, promotion gated on
|
||||
`expand_novelty_sim` / `expand_track_spread_max`, cleared on `is_cut`. Wired at
|
||||
`identity_matcher_node.hpp:227`, cleared at `:126`.
|
||||
|
||||
**Gap:** all three quiet-signal conditions rather than only `is_cut`; and the
|
||||
whole of AR-020 — the TBI queue, the deferred pass, and deferring output until it
|
||||
completes.
|
||||
**Gap:** the band rule of AR-018 replacing the current novelty/spread gates; all
|
||||
three quiet-signal conditions rather than only `is_cut`; probability space
|
||||
throughout (AR-024); and the whole of AR-020 — the TBI queue, the deferred pass, and
|
||||
deferring output until it completes.
|
||||
|
||||
## AR-022 — Unidentified-track capture
|
||||
|
||||
@@ -1036,45 +990,21 @@ is the only viable formulation — a per-pair loop is orders of magnitude off.
|
||||
**All similarity computation goes through the GEMM path**, with no exception
|
||||
justified by "this set is small". Three call sites:
|
||||
|
||||
1. **Baked gallery** — GEMM (`sim_engine_->compute()`, backend from
|
||||
`SAE_GEMM_BACKEND`). ✓
|
||||
2. **Per-film annex** — was a **CPU loop**, justified in-comment by "tens of
|
||||
embeddings". AR-018…AR-021 invalidated that assumption: every owned track
|
||||
contributes, so the annex grows with cast size and film length. Now appended
|
||||
to the gallery matrix rather than scored separately — promotions are pushed
|
||||
into the engine's resident matrix (`ISimilarityEngine::append_rows`,
|
||||
capacity doubling, device-to-device on the GPU backends) and `flat_actor_`
|
||||
grows in lockstep, so one multiply covers baked and promoted references and
|
||||
best-of-N is a single pass over one similarity column. ✓
|
||||
1. **Baked gallery** — already GEMM (`sim_engine_->compute()`,
|
||||
`identity_matcher_node.hpp:143`, backend from `SAE_GEMM_BACKEND`). ✓
|
||||
2. **Per-film annex** — currently a **CPU loop**
|
||||
(`identity_matcher_node.hpp:159-162`), justified in-comment by "tens of
|
||||
embeddings". AR-018…AR-021 invalidates that assumption: every owned track now
|
||||
contributes, so the annex grows with cast size and film length. It must move
|
||||
into the GEMM path — appended to the gallery matrix, or a second multiply.
|
||||
3. **Deferred TBI pass (AR-020)** — the most GEMM-friendly operation in the
|
||||
pipeline: all TBI embeddings against the full gallery-plus-annex, offline,
|
||||
operands resident, no streaming. One large multiply, not a loop over entries.
|
||||
Not yet built; AR-020 owns it.
|
||||
|
||||
**Current:** 1 and 2 done. `TrackGallery` holds the annex as a contiguous
|
||||
row-major matrix plus a parallel actor index, and hands newly promoted rows to
|
||||
the matcher once per frame (`drain_promotions`), which is what call site 3 will
|
||||
score against.
|
||||
|
||||
**Gap:** call site 3, gated on AR-020 existing at all.
|
||||
|
||||
This constrains AR-018…AR-021's implementation: the annex must be a **contiguous matrix**
|
||||
with promotions appended, plus a parallel actor-index mapping — exactly the
|
||||
`flat_emb_`/`flat_actor_` arrangement the baked gallery already uses.
|
||||
|
||||
**Ordering note.** Absorbing promotions is a once-per-frame step that runs after
|
||||
every face in the frame has been scored, not mid-frame. Appending mid-frame would
|
||||
invalidate the similarity pointer the matcher is still reading, and it also
|
||||
removes an accidental dependence on face order within a frame: a promotion helps
|
||||
subsequent frames, never the one that produced it, which is the semantics the
|
||||
expansion store already documents.
|
||||
|
||||
**The CPU GEMM path requires OpenBLAS.** It is what CI and the cpu builder image
|
||||
run, so a silent fall back to the scalar loop would mean AR-027 is measured — or
|
||||
believed — on a path no release uses. Absence is a configure error; the scalar
|
||||
loop survives as the correctness oracle, reachable only via
|
||||
`-DSAE_ALLOW_SCALAR_GEMM=ON`.
|
||||
|
||||
### Scaling characteristics that must be known, not assumed
|
||||
|
||||
- **Throughput versus gallery size must be measured** (VR-008) and published. The
|
||||
@@ -1653,10 +1583,7 @@ Persist pipeline state at the point where the expensive work ends.
|
||||
per-frame index table (`face_offset`, `face_count`) pointing into them. Avoids
|
||||
variable-length HDF5 types and reads straight into numpy.
|
||||
- Stores per frame: `timestamp_sec`, `frame_idx`, `is_cut`, `is_scene_boundary`.
|
||||
Per face: `embedding` [N,512], `bbox` [N,4], `landmarks` [N,10], `confidence`,
|
||||
and from v2 the AR-028 quality vector — `sharpness` [N] and
|
||||
`alignment_residual` [N]. Size, its third axis, is `bbox` and is not
|
||||
duplicated.
|
||||
Per face: `embedding` [N,512], `bbox` [N,4], `landmarks` [N,10], `confidence`.
|
||||
- Invariants: embeddings unit-norm; `face_offset` contiguous; bboxes and
|
||||
landmarks in **decoded-frame** pixels with `bbox_upscale` recorded alongside
|
||||
(the dump is a faithful tap, so it does not transform what the tracker saw —
|
||||
@@ -1667,16 +1594,10 @@ Persist pipeline state at the point where the expensive work ends.
|
||||
Schema owned by [`scripts/optimizer/SCHEMA.md`](../scripts/optimizer/SCHEMA.md).
|
||||
|
||||
**Current:** C++ dump sink (`embedding_dump_node.hpp`, `dump_embeddings.cpp`),
|
||||
read by `replay.py`. At `schema_version` 2, which AR-028 took it to by adding the
|
||||
quality columns; readers on both sides check the datasets by name, so a v1 dump
|
||||
still replays and reports the vector as unknown rather than as zero.
|
||||
|
||||
**Gap:** **AR-012 breaks the replay contract.** Track extents
|
||||
read by `replay.py`. **Gap:** **AR-012 breaks the replay contract.** Track extents
|
||||
are decided in the tracker, which is *downstream* of the dump — so a replay can
|
||||
reproduce them, but only if the dump preserves everything the tracker needs.
|
||||
Verify `landmarks`/`bbox`/`is_cut` suffice, and bump `schema_version` if not.
|
||||
The committed fixtures are still v1, so they carry no quality vector until
|
||||
`scripts/make_fixtures.sh` is re-run on a GPU host.
|
||||
|
||||
## VR-002 — Replay and sweep
|
||||
|
||||
@@ -1799,54 +1720,6 @@ round 1 seeding references that corrupt round 2.
|
||||
"expansion helps live matching" from "expansion helps the second pass", which the
|
||||
current all-or-nothing `expand_gallery` flag cannot distinguish.
|
||||
|
||||
## VR-015 — Per-node cost and bottleneck attribution
|
||||
|
||||
**Requirement: a run must be able to report where its time went, per node, and
|
||||
which node is setting the pace.** Without it, optimisation is guesswork, and
|
||||
worse than guesswork — the obvious number is wrong in a specific, repeatable
|
||||
direction, so acting on it makes the pipeline slower.
|
||||
|
||||
**Why the obvious number is wrong.** KPN times a node across `fire_once`, which
|
||||
wraps the functor *and* `push_outputs`. Under AR-004 a push parks on a full
|
||||
downstream channel, so a node that is merely waiting bills that wait to itself.
|
||||
On the SuperHero reference run (`docs/benchmark.md`) `frame_source` reported
|
||||
`ema=141.899ms` per frame while its own decoder logged 12-18 ms: it was
|
||||
backpressured, and the report named the *fastest* node in the graph as the most
|
||||
expensive one. A second trap sits behind the first — `ema_exec_ms` is an
|
||||
exponentially weighted average, so `frames × ema` is not a total; on a film whose
|
||||
per-frame cost swings between crowd scenes and landscapes the two differ
|
||||
substantially.
|
||||
|
||||
**Method.** Three measurements per node, none of which is sufficient alone:
|
||||
|
||||
| Measure | What it is | What it cannot tell you |
|
||||
|---|---|---|
|
||||
| `cpu_ms` | thread CPU time (`CLOCK_THREAD_CPUTIME_ID`) | GPU wait — a device-bound node looks idle |
|
||||
| `exec_ms` | cumulative wall time inside the node | work from waiting — backpressure inflates it |
|
||||
| `pressure` | mean input fill − mean output fill | how expensive the node is, only that it paces |
|
||||
|
||||
Queue occupancy has to be **sampled during the run**. `current_fill` is
|
||||
instantaneous and every channel has drained by shutdown, so a single read at the
|
||||
end describes an idle pipeline however congested it was.
|
||||
|
||||
**The number that matters** is `pressure`, because work piles up in front of the
|
||||
bottleneck and starves everything after it, and that ordering holds whether the
|
||||
node is waiting on a core, a GPU or a disk. `cpu_share` then selects the repair:
|
||||
a pacing node with a saturated thread is CPU-bound and the work must get cheaper,
|
||||
while a pacing node with an idle thread is device-bound, where batch size and
|
||||
engine precision are the knobs and the C++ is not.
|
||||
|
||||
**Current:** `--benchmark <path>` writes the JSON report and prints a table at
|
||||
shutdown; `src/benchmark.hpp`. Attribution is a pure function over KPN snapshots,
|
||||
so it is verified on CI's GPU-free N100 (UT-120…UT-124) rather than only by
|
||||
running the pipeline. The node graph is recovered from KPN's channel names, so a
|
||||
re-wired topology needs no change here. Required `NodeStats::total_exec_us` in
|
||||
the KPN submodule — the EMA could not be turned into a total.
|
||||
|
||||
**Gap:** GPU utilisation and memory are not sampled, so a device-bound verdict
|
||||
says *that* a node waits on the GPU, not whether the GPU is saturated or merely
|
||||
badly fed. That distinction needs NVML, and it is what VR-008 will want anyway.
|
||||
|
||||
## VR-008 — Gallery scaling benchmark
|
||||
|
||||
Establish the throughput-versus-gallery-size curve required by A10.
|
||||
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 404 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 367 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 367 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 367 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 218 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 252 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 221 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 180 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 268 KiB After Width: | Height: | Size: 239 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 268 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 226 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 156 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 180 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 113 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 47 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 53 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 47 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 80 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 447 KiB After Width: | Height: | Size: 165 KiB |
@@ -1,344 +0,0 @@
|
||||
# Benchmark — SuperHero
|
||||
|
||||
The reference film for end-to-end accuracy. Replaces Road to Bali, which was
|
||||
withdrawn for the reason in [Why not Road to Bali](#why-not-road-to-bali).
|
||||
|
||||
TRACES: AR-011, AR-012, AR-013 | VR-001, VR-005 | SR-002
|
||||
|
||||
---
|
||||
|
||||
## The film
|
||||
|
||||
SuperHero, from the [NIST TRECVID Deep Video Understanding development
|
||||
set](https://www-nlpir.nist.gov/projects/trecvid/dvu/dvu.development.dataset/).
|
||||
14 films are asserted Creative Commons and need no data agreement; only the 5
|
||||
KinoLorber test films are gated.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Runtime | 1025.5 s (17.1 min), 10 scenes |
|
||||
| Resolution | 640×360 |
|
||||
| Ground truth | Per-scene presence, from the scene knowledge graphs |
|
||||
| Gallery | 5 characters, 14 references |
|
||||
|
||||
The DVU set is what makes this workable: it ships **character** face crops cut
|
||||
from the film itself, so ground truth and gallery are both in character space
|
||||
and scoring needs no actor→character mapping.
|
||||
|
||||
**Licence caveat.** NIST links licence evidence for only 4 of the 14 films, and
|
||||
SuperHero is not one of them — its end credits carry no copyright or CC notice,
|
||||
list a "Temporary Musical Score" and a SAG cast, and it has no traceable online
|
||||
release. Fine for internal benchmarking; do not redistribute frames from it.
|
||||
Valkaama is the one film with an independently documented licence (CC BY-SA 3.0)
|
||||
if provenance ever has to be defended.
|
||||
|
||||
---
|
||||
|
||||
## Reproducing it
|
||||
|
||||
```sh
|
||||
# 1. Annotations, character mugshots, scene segmentation.
|
||||
# NIST names the same film three different ways, hence the overrides.
|
||||
KG_DIR=superHero KG_FILE=superhero scripts/fetch_dvu.sh SuperHero ../dvu-hero
|
||||
|
||||
# 2. Scene clips (movie.shots), then fuse them into one stream.
|
||||
# Fusing matters — see "Run it as one film" below.
|
||||
# SuperHero-1.webm … SuperHero-10.webm from
|
||||
# <dataset>/movie.shots/, then:
|
||||
ffmpeg -f concat -safe 0 -i concat.txt -c copy SuperHero_full.webm
|
||||
|
||||
# 3. Gallery, with the face-size floor that keeps references in distribution.
|
||||
./build/build_gallery --root ../dvu-hero/root \
|
||||
--output ../dvu-hero/hero66.h5 --min-face-px 66
|
||||
|
||||
# 4. Run, on the GPU path (see "Check you are on the GPU").
|
||||
./build/scene_analyze --movie hero/SuperHero_full.webm \
|
||||
--gallery ../dvu-hero/hero66.h5 \
|
||||
--detector-engine trt_cache/scrfd.scrfd_500m_bnkps.640.fp16.engine \
|
||||
--arcface-engine trt_cache/arcface.LVFace-B_Glint360K.b4.fp16.engine \
|
||||
--fps 5 --min-face-px 32 --expand-gallery \
|
||||
--output pred.json
|
||||
```
|
||||
|
||||
Nothing here is in git: the clips are ~130 MB and the annotations are
|
||||
regenerable. Replay fixtures derived from the run ship through the artifact
|
||||
registry instead:
|
||||
|
||||
```sh
|
||||
scripts/artifacts/push_artifacts.sh replay-fixtures
|
||||
scripts/artifacts/pull_artifacts.sh replay-fixtures [version]
|
||||
```
|
||||
|
||||
The gallery travels in the same archive as the dumps deliberately — a dump only
|
||||
replays meaningfully against the gallery it was produced with, and pairing one
|
||||
with a different gallery silently changes every identity decision in it.
|
||||
|
||||
---
|
||||
|
||||
## Results
|
||||
|
||||
Measured on the fused film, gallery expansion on.
|
||||
|
||||
| Metric | Value |
|
||||
|---|---|
|
||||
| Precision | **1.00** |
|
||||
| Recall | 0.65 |
|
||||
| F1 | 0.79 |
|
||||
| True positives | 13 |
|
||||
| False positives | **0** |
|
||||
| False negatives | 7 |
|
||||
|
||||
Six of ten scenes scored exactly right, including the three-character scenes 4
|
||||
and 5.
|
||||
|
||||
**Zero false positives is the result worth keeping.** Every out-of-gallery
|
||||
character — Beast, Mighty Celestial, Ms. Johnson, Doctor, two Masked Persons —
|
||||
was declined rather than forced onto a nearest match. That is the calibrated
|
||||
probability (AR-024) doing its job, and it is the right failure direction for an
|
||||
X-Ray overlay: a miss is a gap, an invention is a lie.
|
||||
|
||||
**The misses have a shape.** Scenes 1, 2, 3 and 8 were missed, and 1–3 are the
|
||||
three shortest scenes in the film (14 s, 38 s, 27 s). That is consistent with
|
||||
per-track Bayesian accumulation (AR-025) needing enough sightings before belief
|
||||
crosses threshold. Scene 8 is 65 s and does not fit that story — it is the one
|
||||
to look at first when improving recall.
|
||||
|
||||
Running the same scenes as isolated clips did *not* do better, so cross-scene
|
||||
gallery expansion is not currently compensating for short scenes.
|
||||
|
||||
### Run it as one film, not as clips
|
||||
|
||||
Per-scene clips defeat per-film gallery expansion (AR-019), which grows a
|
||||
temporary gallery from track continuity across the whole film and re-assesses
|
||||
unknown tracks at the end. Ten isolated clips give it nothing to work with, and
|
||||
pay model and gallery load ten times over.
|
||||
|
||||
Fusing also makes presence windows cross real scene boundaries, which is how
|
||||
SR-002's scene-scoped question is asked in production. Note the joins are
|
||||
artificial cuts — consecutive scenes were never contiguous footage — so presence
|
||||
bleeding across a boundary may be the join rather than a tracking fault.
|
||||
|
||||
---
|
||||
|
||||
## Throughput
|
||||
|
||||
| Path | Realtime factor | Sampled fps | 17-min film |
|
||||
|---|---|---|---|
|
||||
| `build/` (TensorRT) | **8.25×** | 41.3 | **2.1 min** |
|
||||
| `build-ort/` (ORT) | 0.54× | 2.7 | ~32 min |
|
||||
|
||||
TensorRT figure re-measured 2026-08-04 over the whole film at `--fps 5
|
||||
--min-face-px 32 --expand-gallery`: 5129 frames, 1025.4 s of film in 124.2 s
|
||||
wall. Two runs agreed to 0.4% (124.2 s clean, 124.7 s under gdb). It supersedes
|
||||
an earlier 2.0×; that figure predates the current tree and was not re-derived
|
||||
here, so treat the gain as measured rather than explained.
|
||||
|
||||
Throughput varies strongly with face density, and **a short window is not a
|
||||
sample of the film**. The opening 60 s benchmarks at 23.9× — decode there costs
|
||||
4-6 ms/frame against a 12.35 ms whole-film mean (n=510), because seeking forward
|
||||
in VP8/WebM gets dearer the deeper you go, and there are few faces. Always quote
|
||||
the whole-film average.
|
||||
|
||||
### Where the time goes (VR-015)
|
||||
|
||||
Measured over the whole film, 2026-08-04:
|
||||
|
||||
| node | cpu_s | % of pipeline CPU | cpu/f | exec/f | stall/f | in% | out% |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| **embedder** | **91.0** | **60%** | 17.74 | 21.61 | 3.87 | 12 | 0 |
|
||||
| **face_detector** ▶ | 41.4 | 27% | 8.07 | 24.20 | **16.14** | **99** | **0** |
|
||||
| frame_source | 12.1 | 8% | 2.36 | 11.26 | 8.90 | — | 97 |
|
||||
| camera_pos | 3.2 | 2% | 0.63 | 0.64 | 0.01 | 97 | 99 |
|
||||
| face_aligner | 1.7 | 1% | 0.33 | 0.34 | 0.01 | 0 | 12 |
|
||||
| identity_matcher | 1.3 | 1% | 0.26 | 0.34 | 0.09 | 0 | 0 |
|
||||
| tracker / sink | 0.5 | <1% | — | — | — | 0 | 0 |
|
||||
|
||||
**`face_detector` paces the run**: its input channel is 97.8% full while its
|
||||
output is 99.4% empty — everything upstream jammed, everything downstream
|
||||
starved. It occupies 5129 × 24.20 ms ≈ 124.1 s of a 124.2 s run, essentially
|
||||
100% wall occupancy, yet only 33% of that is CPU. The other 16.14 ms/frame is
|
||||
device wait.
|
||||
|
||||
**The embedder is the larger cost but not the constraint**: 60% of all pipeline
|
||||
CPU, 73% of wall as thread-busy. Whether that is real work or a spinning
|
||||
`cudaStreamSynchronize` is unresolved — see the sync caveat below, which is a
|
||||
one-line experiment.
|
||||
|
||||
**`frame_source` is the trap this table exists to defuse.** It reports
|
||||
`exec/f = 11.26 ms` against `cpu/f = 2.36 ms`, and its output channel is 97%
|
||||
full: it is backpressured, not expensive. The old KPN `ema` reading made it look
|
||||
like the most costly node in the pipeline at 141.899 ms/frame.
|
||||
|
||||
|
||||
`--benchmark <path>` writes a per-node timing report and prints a table at
|
||||
shutdown. `hero/run_bench.sh` is `run_trt.sh` with it switched on:
|
||||
|
||||
```bash
|
||||
./build/scene_analyze … --benchmark $H/bench_trt.json --output $H/pred_bench.json
|
||||
```
|
||||
|
||||
**Do not read the `ema` column of the old KPN diagnostics block as a cost.** KPN
|
||||
times a node across `fire_once`, which wraps the functor *and* the push to the
|
||||
next channel, and a push parks when that channel is full (AR-004). A
|
||||
backpressured node therefore bills its waiting to itself. On this film that
|
||||
produced a genuinely inverted answer:
|
||||
|
||||
```
|
||||
│ frame_source frames=5132 ema=141.899ms ← reported cost
|
||||
[frame_source] decode avg=16.6127ms fps=60.19 ← actual decode
|
||||
```
|
||||
|
||||
The source is not expensive; it is idle, holding a frame nobody has taken yet.
|
||||
Optimising against that number means optimising the fastest node in the graph.
|
||||
|
||||
The benchmark report separates the two:
|
||||
|
||||
| Column | Meaning | Blind spot |
|
||||
|---|---|---|
|
||||
| `cpu_s`, `cpu%tot` | thread CPU time, and this node's share of all of it | a GPU wait looks like idleness |
|
||||
| `cpu/f` | CPU ms per frame — backpressure cannot inflate it | as above |
|
||||
| `exec/f` | wall ms per frame in the node, **including parked pushes** | overstates a blocked node |
|
||||
| `stall/f` | `exec/f − cpu/f`: parked, or waiting on a device | does not say which |
|
||||
| `in%`, `out%` | mean fill of the node's input and output channels | — |
|
||||
| `press` | `in% − out%`; **the node marked ▶ is pacing the run** | not a cost, an ordering |
|
||||
|
||||
Read `press` first: work queues up in front of the bottleneck and starves
|
||||
everything after it, so the pacing node is the one with a full input and an empty
|
||||
output. Then read `cpu%run` to decide the repair — a saturated thread means the
|
||||
work itself must get cheaper, while an idle thread under pressure means the node
|
||||
is waiting on the GPU or the disk, where batch size and engine precision are the
|
||||
knobs and the C++ is not.
|
||||
|
||||
Channel fills are sampled every 100 ms (`--benchmark-interval-ms`) because
|
||||
`current_fill` is instantaneous: by shutdown every channel has drained, so a
|
||||
single read at the end reports an idle pipeline no matter how congested it was.
|
||||
|
||||
#### Check the GPU is not throttled before comparing anything
|
||||
|
||||
**On this hardware, thermal state moves the result more than any code change
|
||||
we are likely to make.** The same binary measured **8.25× cool and 3.12× once
|
||||
heat-soaked** — a 2.6× swing — because the laptop RTX 3050 hits `SW Thermal
|
||||
Slowdown` and pins the SM clock to **210 MHz out of 2100**:
|
||||
|
||||
```
|
||||
$ nvidia-smi -q -d PERFORMANCE | grep -E "SW Power Cap|SW Thermal"
|
||||
SW Power Cap : Active
|
||||
SW Thermal Slowdown : Active
|
||||
```
|
||||
|
||||
A number recorded without its clock state is not comparable to any other
|
||||
number, and back-to-back full-film runs guarantee the later ones are throttled.
|
||||
`run_bench.sh` now records `nvidia-smi` either side of the run into
|
||||
`bench_gpu.txt`; check it before believing a regression. Let the GPU idle back
|
||||
to full clock between measurements, and never A/B two runs across a heat-soak.
|
||||
|
||||
This one cost real time here: a 2.7× "regression" was attributed to a code
|
||||
change and reverted on that basis, when the change was innocent and the GPU had
|
||||
simply warmed up between the two measurements.
|
||||
|
||||
#### `cpu_s` on a GPU node is mostly spin — measured
|
||||
|
||||
CUDA's default sync policy (`cudaDeviceScheduleAuto`) spin-waits before it
|
||||
yields, so `cudaStreamSynchronize` charges the *calling thread's* CPU while the
|
||||
GPU works. A GPU-bound node therefore reports a large `cpu_s` and reads as
|
||||
CPU-bound.
|
||||
|
||||
`SAE_CUDA_BLOCKING_SYNC=1` switches to a blocking wait. Measured over 300 s of
|
||||
film, four cases, identical otherwise:
|
||||
|
||||
| case | realtime | total CPU | embedder CPU |
|
||||
|---|---|---|---|
|
||||
| baseline | 3.29× | 103 s | 66 s |
|
||||
| **`SAE_CUDA_BLOCKING_SYNC=1`** | 3.29× | **23 s** | **4 s** |
|
||||
| `SAE_CV_THREADS=1` | 3.30× | 101 s | 66 s |
|
||||
| both | 3.29× | 25 s | 5 s |
|
||||
|
||||
**94% of the embedder's CPU was spin, not work**, and 78% of the pipeline's.
|
||||
Throughput is unchanged, so this is free CPU — which matters for a service
|
||||
sharing a box (DP-003) and makes `cpu_s` mean what it says. Prefer it for any
|
||||
run where the CPU numbers are being read.
|
||||
|
||||
`SAE_CV_THREADS=1` does nothing measurable: the only OpenCV-heavy node is
|
||||
`face_aligner` at 1-2% of the pipeline, so the TBB arena is not worth removing
|
||||
and `warpAffine` is not worth replacing.
|
||||
|
||||
**Caveat: measured with the GPU clamped at 210 MHz** (see below). A device at
|
||||
full clock spends less time in the sync, so the absolute spin figure will fall;
|
||||
the ranking should not.
|
||||
|
||||
#### `cpu_s` counts one thread — mind the TBB arena
|
||||
|
||||
OpenCV 5 here is built against TBB, and every OpenCV module links it, so
|
||||
`cv::parallel_for_` dispatches onto a TBB arena of `nproc − 1` workers (19 on the
|
||||
20-core dev box; visible as `libtbb.so.12` frames in a thread dump). Since
|
||||
`CLOCK_THREAD_CPUTIME_ID` is per-thread, work a node fans out that way is billed
|
||||
to the TBB workers, **not** to the node.
|
||||
|
||||
So a node using `warpAffine`, a histogram compare or a colour conversion reads
|
||||
cheaper in `cpu_s` than it really is, and the missing time appears in `stall/f`,
|
||||
where it looks identical to a GPU wait. `exec/f` does capture it — the functor
|
||||
does not return until the parallel region joins — so the tell is a node whose
|
||||
`exec/f` far exceeds its `cpu/f` **while its output channel is empty**: that is
|
||||
fan-out, not blocking.
|
||||
|
||||
Worth knowing for its own sake, too: 9 KPN node threads plus 19 TBB workers plus
|
||||
the CUDA and NVDEC threads is heavy oversubscription on 20 cores.
|
||||
|
||||
The JSON carries the same data plus the run's configuration, so two runs can be
|
||||
diffed directly — which is the point, when sweeping `--embed-batch`, `--fps` or
|
||||
an engine precision.
|
||||
|
||||
### Check you are on the GPU
|
||||
|
||||
ORT's CUDA execution provider fails to load on this machine and **silently falls
|
||||
back to CPU**:
|
||||
|
||||
```
|
||||
Failed to load library libonnxruntime_providers_cuda.so:
|
||||
undefined symbol: cudnnGetConvolutionBackwardDataAlgorithm_v7
|
||||
```
|
||||
|
||||
That symbol was removed in cuDNN 9; the packaged ORT is built against cuDNN 8.
|
||||
ORT logs this once at startup and then runs happily on CPU, so a `build-ort`
|
||||
timing is a CPU number wearing a GPU label — a 15× error with no symptom other
|
||||
than a figure you have no baseline for. Grep the log for `Failed to load
|
||||
library` before trusting any throughput measurement.
|
||||
|
||||
The TensorRT path (`build/`) needs prebuilt engines from
|
||||
`scripts/build_trt_engines.sh` and reports what it loaded:
|
||||
|
||||
```
|
||||
[TrtScrfd] loaded: … [TrtArcFace] loaded: … max_batch=4
|
||||
[similarity] cuBLAS/CUDA engine: gallery resident on GPU
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Why not Road to Bali
|
||||
|
||||
Bali was chosen because DVU ships character mugshots for it. It was withdrawn on
|
||||
**face scale**, measured on its own reference crops:
|
||||
|
||||
| | Bali | SuperHero |
|
||||
|---|---|---|
|
||||
| Median detected face | 27 px | **69 px** |
|
||||
| Maximum detected face | 69 px | **241 px** |
|
||||
| References ≥66 px | 2 of 69 | 14 of 27 |
|
||||
|
||||
The DVU images are scene crops, not mugshots, so the crop dimensions say nothing
|
||||
about face scale — the face has to be detected and measured. Bali's median
|
||||
reference was being upscaled roughly 4× to reach ArcFace's 112×112, and the
|
||||
worst 7×, which violates AR-011: every model gets the input it was trained for.
|
||||
A model run off-distribution returns confident, plausible, wrong output.
|
||||
|
||||
In a gallery that error is permanent. A bad frame costs one frame; a poisoned
|
||||
reference corrupts every future match against that identity.
|
||||
|
||||
No threshold rescued it. At 66 px only 2 of 69 references survived — the largest
|
||||
face in the entire set is 69 px — so there was no cut that both kept references
|
||||
in distribution and left enough of them to calibrate. SuperHero's gallery builds
|
||||
at a 66 px floor and calibrates on its own (`a=15.2867 b=-4.98633`, 100 % train
|
||||
accuracy) rather than borrowing constants.
|
||||
|
||||
Any accuracy figure recorded against Bali predates this and should be treated as
|
||||
measuring upscaling artifacts as much as the pipeline.
|
||||
@@ -1,12 +1,10 @@
|
||||
> **Archived (July 2026).** This report covers the pre-opencv5 framework and the 4-model ArcFace/LVFace bake-off. It is superseded by the current [experiment log](model-bakeoff.md) for the opencv5 build. Kept for provenance; the numbers here are historical.
|
||||
|
||||
# Which embedding model is best?
|
||||
|
||||
Three ArcFace variants (w600k-R50, R18, w600k-MBF) and LVFace-B (Glint360K,
|
||||
455MB) were compared. r50 is excluded from the training/held-out comparison
|
||||
below; its gallery has roughly 30% fewer reference images per actor than the
|
||||
other three on the identical source photos, which confounds a direct score
|
||||
comparison (see [the full experiment log](model-bakeoff-2026-07.md) for detail). It
|
||||
comparison (see [the full experiment log](model-bakeoff.md) for detail). It
|
||||
remains in the calibration comparison, which does not depend on the gallery
|
||||
image count.
|
||||
|
||||
@@ -65,7 +63,7 @@ than general performance. On training data, the ordering is not as clean:
|
||||
|
||||
mbf beats LVFace on Lord of War (77.2% vs 75.6%), the only film in either
|
||||
table where LVFace does not score highest. LVFace's training-set macro
|
||||
average (75.3%, see [the full experiment log](model-bakeoff-2026-07.md)) is not a
|
||||
average (75.3%, see [the full experiment log](model-bakeoff.md)) is not a
|
||||
uniform win across every film it contributes to; the held-out result, where
|
||||
LVFace wins all 5 films outright, is the stronger claim.
|
||||
|
||||
@@ -81,7 +79,7 @@ not.
|
||||
|
||||
|
||||
Best full-gallery combo per model (all three are `full_exp`), from the
|
||||
training matrix in [the full experiment log](model-bakeoff-2026-07.md):
|
||||
training matrix in [the full experiment log](model-bakeoff.md):
|
||||
|
||||
| model | F1 | P | R | misID |
|
||||
|---|---|---|---|---|
|
||||
@@ -1,5 +1,3 @@
|
||||
> **Archived (July 2026).** This report covers the pre-opencv5 framework and the 4-model ArcFace/LVFace bake-off. It is superseded by the current [experiment log](model-bakeoff.md) for the opencv5 build. Kept for provenance; the numbers here are historical.
|
||||
|
||||
# Whole gallery vs. cast-restricted gallery
|
||||
|
||||
Two ways to run the matcher. Full mode scores every detected face against
|
||||
@@ -10,7 +8,7 @@ top-billed actors) before the matcher runs.
|
||||
## Result
|
||||
|
||||
Averaged across the 3 compared models (r50 excluded, see
|
||||
[the full experiment log](model-bakeoff-2026-07.md)) and both expansion settings, on
|
||||
[the full experiment log](model-bakeoff.md)) and both expansion settings, on
|
||||
the 4 training films:
|
||||
|
||||
| scope | F1 | P | R | total misID |
|
||||
@@ -29,7 +27,7 @@ restricted gallery:
|
||||
|
||||
|
||||
|
||||
See [the full experiment log](model-bakeoff-2026-07.md) for the complete table. One
|
||||
See [the full experiment log](model-bakeoff.md) for the complete table. One
|
||||
combo reaches zero true out-of-cast misidentifications,
|
||||
`arcface_w600k_mbf_restricted_exp` (F1 76.2%), and it is a restricted one,
|
||||
consistent with restriction, not expansion, being what suppresses cross-film
|
||||
@@ -59,7 +57,7 @@ Building this as a real feature requires:
|
||||
option.
|
||||
- A decision on the fallback case: what happens to a real, uncredited
|
||||
cameo (see the Germar Terrell Gardner and Talia Balsam cases in the
|
||||
[LVFace deep dive](lvface-deep-dive-2026-07.md#where-lvface-beat-x-ray)) if the
|
||||
[LVFace deep dive](lvface-deep-dive.md#where-lvface-beat-x-ray)) if the
|
||||
restricted gallery never includes them at all.
|
||||
- Regenerating the restricted-gallery cache whenever a title's Jellyfin
|
||||
cast list changes.
|
||||
+54
-54
@@ -17,70 +17,70 @@ two credited cast members without a visible face are correctly reported
|
||||
present but not visible. This matches Amazon X-Ray's own record for this
|
||||
second exactly.
|
||||
|
||||
## The headline: learned scene boundaries
|
||||
Results are not uniform across films. The hardest held-out film scores 46%
|
||||
F1. This report documents why: one tunable trade (extinction bridging at
|
||||
hard cuts), one structural limit (X-Ray credits people whose faces never
|
||||
appear on screen), and a small number of cases where the pipeline is
|
||||
correct and X-Ray's ground truth is not. Read
|
||||
[how we score against X-Ray](methodology.md) first. X-Ray's ground truth is
|
||||
scene-level; the pipeline's output is per-second. That difference shapes
|
||||
every finding below.
|
||||
|
||||
The current opencv5 build's biggest gain is **flood-fill presence on a
|
||||
learned scene-boundary detector**. An actor seen once inside a shot is
|
||||
reported for the whole shot — but only if the shot boundaries are good. A
|
||||
learned XGBoost boundary detector, scored **leave-one-out** so no film is
|
||||
ever measured by a detector that trained on it, lifts per-second X-Ray
|
||||
presence F1 across nine films and improves every one of them:
|
||||
## Findings
|
||||
|
||||
| boundary source for flood-fill | presence F1 |
|
||||
| ------------------------------ | ----------: |
|
||||
| track-extent (flood off) | 62.6% |
|
||||
| flood + grayscale cuts | 64.0% |
|
||||
| **flood + learned detector (LOO)** | **74.9%** |
|
||||
<div class="grid cards" markdown>
|
||||
|
||||
|
||||
- :material-trophy:{ .lg .middle } **[Which model is best?](best-model.md)**
|
||||
|
||||
The full story — why the old grayscale cut detector broke Scarface, what
|
||||
features work, and the per-film breakdown — is on the
|
||||
[learned scene-boundary detector](scene-boundary-detector.md) page.
|
||||
---
|
||||
|
||||
## What the numbers mean, and their limits
|
||||
Calibration curves first, independent of any threshold, then held-out
|
||||
F1 across three models. LVFace-B Glint360K wins both, and wins on every
|
||||
held-out film.
|
||||
|
||||
Results are not uniform across films, and they should not be. X-Ray's ground
|
||||
truth is scene-level and credits people whose faces never appear on screen;
|
||||
the pipeline's output is per-second and can only name a face it can see.
|
||||
That difference is a structural recall ceiling, not a bug. Read
|
||||
[how we score against X-Ray](methodology.md) first — it defines F1,
|
||||
precision, recall, and misID, and explains the two limits (off-screen cast
|
||||
and gallery coverage) that shape every finding.
|
||||
- :material-filter:{ .lg .middle } **[Whole vs. cast-restricted gallery](gallery-scope.md)**
|
||||
|
||||
Precision on identified faces is near-perfect: where the pipeline names a
|
||||
face, it is almost always a name X-Ray also credits to that scene. The
|
||||
frames throughout this documentation make the tension visual — **green** =
|
||||
true positive, **red** = false positive, **orange** = unknown, and a
|
||||
**blue** panel lists credited cast present with no visible face.
|
||||
---
|
||||
|
||||
Restricting the matcher to a film's credited cast improves F1,
|
||||
recall, and misID rate at once, but is not a shipped runtime feature
|
||||
yet.
|
||||
|
||||
- :material-account-convert:{ .lg .middle } **[Does pose expansion help?](pose-expansion.md)**
|
||||
|
||||
---
|
||||
|
||||
A training-set effect that did not reproduce on 5 held-out films once
|
||||
two methodology bugs in the comparison harness were found and fixed.
|
||||
|
||||
- :material-blur:{ .lg .middle } **[What does blur cost?](quality-knee.md)**
|
||||
|
||||
---
|
||||
|
||||
Sharpness is not a sufficient statistic for identity loss, blur breaks
|
||||
confidence rather than ranking, and variance-of-Laplacian is
|
||||
anti-predictive at fixed resolution.
|
||||
|
||||
- :material-magnify-expand:{ .lg .middle } **[Deep dive: LVFace-B Glint360K](lvface-deep-dive.md)**
|
||||
|
||||
---
|
||||
|
||||
The held-out generalization gap, the two mechanisms behind its errors,
|
||||
and every distinct case where it names someone outside the film's
|
||||
credited cast.
|
||||
|
||||
</div>
|
||||
|
||||
## Full experiment log
|
||||
|
||||
- **[Full experiment log (opencv5)](model-bakeoff.md)**: the complete log
|
||||
behind the current build — the ten-knob differential-evolution tuning, the
|
||||
shipped [`src/config.hpp`](https://REPOLINK/src/config.hpp) defaults and
|
||||
where each comes from, the replay architecture that makes a nine-film
|
||||
search tractable, and the flood-fill step change.
|
||||
- **[Learned scene-boundary detector](scene-boundary-detector.md)**: the
|
||||
features, the model, leave-one-out results, and the two headline films.
|
||||
- **[Benchmark — SuperHero](benchmark.md)**: the benchmark harness.
|
||||
- **[Service conversion (proposal)](service-conversion.md)**: design sketch
|
||||
for a native idle-GPU worker gated on screen lock, not yet built.
|
||||
|
||||
## Archive (July 2026)
|
||||
|
||||
The pre-opencv5 four-model ArcFace/LVFace bake-off is kept for provenance.
|
||||
Its numbers are historical; the current build supersedes them.
|
||||
|
||||
- [Best model (July)](best-model-2026-07.md) — LVFace-B Glint360K wins on
|
||||
calibration and on every held-out film.
|
||||
- [Gallery scope (July)](gallery-scope-2026-07.md) — cast-restricted
|
||||
gallery improves F1, recall, and misID at once.
|
||||
- [Pose expansion (July)](pose-expansion-2026-07.md) — a training-set
|
||||
effect that did not reproduce held-out.
|
||||
- [LVFace deep dive (July)](lvface-deep-dive-2026-07.md) — the
|
||||
generalization gap and every out-of-cast identification.
|
||||
- [Full experiment log (July)](model-bakeoff-2026-07.md).
|
||||
- **[Full experiment log](model-bakeoff.md)**: the complete log behind the
|
||||
four pages above, including how replaying against cached embeddings
|
||||
inside the same KPN network makes a full model and configuration
|
||||
comparison practical, the full results table, and every caveat. This is
|
||||
where the shipped [`src/config.hpp`](https://REPOLINK/src/config.hpp)
|
||||
defaults come from.
|
||||
- **[Service conversion (proposal)](service-conversion.md)**: design
|
||||
sketch for a native idle-GPU worker gated on screen lock, not yet built.
|
||||
|
||||
## Reproducing the benchmarks
|
||||
|
||||
|
||||
@@ -1,14 +1,12 @@
|
||||
> **Archived (July 2026).** This report covers the pre-opencv5 framework and the 4-model ArcFace/LVFace bake-off. It is superseded by the current [experiment log](model-bakeoff.md) for the opencv5 build. Kept for provenance; the numbers here are historical.
|
||||
|
||||
# Deep dive: LVFace-B Glint360K
|
||||
|
||||
LVFace won the model comparison (see [Which model is best?](best-model-2026-07.md))
|
||||
LVFace won the model comparison (see [Which model is best?](best-model.md))
|
||||
and is the shipped default embedder. This page reports how it performs in
|
||||
detail: a baseline of correct output, the two mechanisms behind its errors,
|
||||
and every distinct case where it names someone who is not in the film's
|
||||
credited cast.
|
||||
|
||||
Read [How we score against X-Ray](methodology-2026-07.md) first. X-Ray's ground truth
|
||||
Read [How we score against X-Ray](methodology.md) first. X-Ray's ground truth
|
||||
is scene-level, not per-frame. A name marked correct in the Offscreen column
|
||||
below is the pipeline correctly reporting scene membership, not a workaround.
|
||||
|
||||
@@ -63,7 +61,7 @@ on the 5 films the optimizer never saw:
|
||||
| macro average | 67.4% | 85.8% | 57.0% | | | | |
|
||||
|
||||
The `P` column is misID-weighted (each out-of-film name counts 10x in the
|
||||
denominator; see [methodology](methodology-2026-07.md#precision-recall-and-the-misid-weighting)).
|
||||
denominator; see [methodology](methodology.md#precision-recall-and-the-misid-weighting)).
|
||||
That weighting is why Many Saints reads 54.7% here despite naming mostly real,
|
||||
present faces: its raw (unweighted) precision is **78.4%**, and the gap is
|
||||
entirely its 974 misIDs paying the 10x penalty. The three zero-misID films
|
||||
@@ -72,7 +70,7 @@ Lovelace, with 58 misIDs, sits 3pp below its raw 93.3%.
|
||||
|
||||
Held-out F1 is 67.4%, against 75.3% on training, an 8pp drop. The spread
|
||||
between the best and worst held-out film is 37pp. This is not unique to
|
||||
LVFace: [the full experiment log](model-bakeoff-2026-07.md#held-out-validation-all-3-models)
|
||||
LVFace: [the full experiment log](model-bakeoff.md#held-out-validation-all-3-models)
|
||||
shows mbf and r18 with the same shape of spread on the same films, at a
|
||||
uniformly lower level. Two mechanisms explain the spread. Both are shown
|
||||
below with frame-level evidence.
|
||||
@@ -176,10 +174,10 @@ ground-truth gap, not a model error.
|
||||
Archie Yates, t=2521s, 78% confidence. A real detected face, a genuine
|
||||
lookalike confusion.
|
||||
|
||||
Zooey Deschanel, t=2819s, 99% confidence — a high-confidence lookalike
|
||||
confusion in the July pipeline. **The current opencv5 pipeline no longer makes
|
||||
this identification**; the tighter tracker/registry and re-tuned matching removed
|
||||
it, so there is no annotated frame for it here.
|
||||
|
||||
|
||||
Zooey Deschanel, t=2819s, 99% confidence. A real detected face at a dinner
|
||||
table, high-confidence lookalike confusion.
|
||||
|
||||
|
||||
|
||||
@@ -1,136 +0,0 @@
|
||||
> **Archived (July 2026).** This report covers the pre-opencv5 framework and the 4-model ArcFace/LVFace bake-off. It is superseded by the current [experiment log](model-bakeoff.md) for the opencv5 build. Kept for provenance; the numbers here are historical.
|
||||
|
||||
# How we score against X-Ray
|
||||
|
||||
Every number in this report, every F1 and misID count, comes from one
|
||||
comparison. The comparison has a mismatch at its core that shapes nearly
|
||||
every finding in this report: the ground truth is scene-level, the
|
||||
pipeline's output is per-second, and the two do not mean the same thing.
|
||||
This page documents that comparison once, so the findings pages can rely on
|
||||
it without re-explaining it.
|
||||
|
||||
## What Amazon X-Ray records
|
||||
|
||||
X-Ray ships three tables per film: `scenes.csv` (a list of `[start, end]`
|
||||
timespans), `people_in_scenes.csv` (which actors are credited in each
|
||||
scene), and `people.csv` (actor identities). There is no per-frame or
|
||||
per-second annotation anywhere in X-Ray. A scene might run 45 seconds, and
|
||||
X-Ray records one cast list for the entire span, not "on screen from
|
||||
second 12 to second 30."
|
||||
|
||||
To compare this against per-second predictions, `second_score.py` expands
|
||||
every scene into per-second ground truth by copying the whole scene's cast
|
||||
list onto every second inside it:
|
||||
|
||||
```python
|
||||
for sn, (t0, t1) in spans.items():
|
||||
cast = scene_cast.get(sn, [])
|
||||
for t in range(int(t0), int(t1)):
|
||||
timeline[t] = cast
|
||||
```
|
||||
|
||||
That is the entire mechanism. If X-Ray credits five actors to a 30-second
|
||||
scene, all five count as ground truth present for all 30 seconds, including
|
||||
seconds where only one of them is on screen. This is not a simplification
|
||||
introduced by the pipeline; it is the only reading of X-Ray's data that is
|
||||
possible, because X-Ray itself does not record anything finer-grained.
|
||||
|
||||
## Why an offscreen name can be scored correct
|
||||
|
||||
A name listed under Offscreen with a correct (green) label is not the
|
||||
pipeline guessing or padding its score. It is the pipeline correctly
|
||||
answering the question X-Ray actually asks: is this actor part of this
|
||||
scene. It answers that question using a presence window (`[start, end]`,
|
||||
held open across cuts by `anneal_sec` and `extinction_sec`), which matches
|
||||
X-Ray's scene-level semantics more closely than a raw per-frame detection
|
||||
would.
|
||||
|
||||
A system that only reported "this actor is visible in this exact frame"
|
||||
would score worse against X-Ray's scene-level ground truth, producing a
|
||||
false negative every time the camera cuts away from a character who is
|
||||
still present in the scene. Not because it is wrong about the world, but
|
||||
because it would be answering a stricter, different question than the one
|
||||
X-Ray's data supports. The presence-window design exists specifically to
|
||||
answer X-Ray's actual question.
|
||||
|
||||
## What this resolves and what it does not
|
||||
|
||||
This resolves the semantic mismatch between a scene and an instant. It does
|
||||
not resolve two other limitations, both discussed in the
|
||||
[LVFace deep dive](lvface-deep-dive-2026-07.md).
|
||||
|
||||
**The face-vs-presence ceiling.** X-Ray credits scene membership regardless
|
||||
of whether a face is ever visible: background crew, characters shot from
|
||||
behind, voice-only presence. No amount of bridging recovers a face that
|
||||
never appears on screen. This is a hard ceiling on recall, not a defect.
|
||||
|
||||
**Extinction bridging can overshoot.** The same presence-window mechanism
|
||||
that correctly answers "still in this scene" during a normal cut can also
|
||||
bridge across a scene boundary it has no way to detect. A hard cut into a
|
||||
different scene with no faces, such as closing credits, carries the
|
||||
previous scene's identities forward until the window expires. This is the
|
||||
mechanism behind Downton Abbey's recall collapse, documented in the deep
|
||||
dive.
|
||||
|
||||
## Precision, recall, and the misID weighting
|
||||
|
||||
Per sampled second `t`:
|
||||
|
||||
**TPI** (true positive instances): actors both X-Ray and the pipeline agree
|
||||
are present.
|
||||
|
||||
**FPI** (false positive instances): actors the pipeline reports that are
|
||||
not in X-Ray's cast for this second. Split into two categories:
|
||||
|
||||
- **FPI_incast**: the actor is in the film's cast, just not credited to
|
||||
this particular scene. A timing or boundary slip.
|
||||
- **FPI_misid**: the actor is not in the film's cast at all. A genuine
|
||||
wrong-identity error, weighted 10x in the precision objective, because
|
||||
naming someone who is not even in the film is a categorically worse
|
||||
error than a few seconds of scene-boundary slop.
|
||||
|
||||
!!! note "Every headline `P` and `F1` is misID-weighted"
|
||||
|
||||
The precision reported throughout this report, and therefore the F1
|
||||
derived from it, puts each `FPI_misid` into the denominator **10 times**
|
||||
(`precision = TPI / (TPI + FPI_incast + 10·FPI_misid)`,
|
||||
[`second_score.py`](https://REPOLINK/scripts/optimizer/second_score.py)).
|
||||
This is deliberate: the whole point is to punish naming an out-of-film
|
||||
actor far harder than a scene-boundary slip. But it means the `P` column
|
||||
is not raw precision, and a misID-heavy film's `P` is depressed
|
||||
super-linearly. `second_score.py` also emits an unweighted `precision_raw`
|
||||
(always ≥ the weighted `P`); where the gap matters, The Many Saints of
|
||||
Newark, weighted `P` 54.7% vs. raw 78.4%, the [LVFace deep dive](lvface-deep-dive-2026-07.md)
|
||||
reports both. When comparing `P` across films, remember you are comparing a
|
||||
quantity that penalizes misIDs, not just a hit rate.
|
||||
|
||||
**FN** (false negatives): actors X-Ray lists that the pipeline never
|
||||
reports, counted only for actors who have a gallery reference embedding.
|
||||
Across the 9-film benchmark, coverage of X-Ray's credited cast ranges from
|
||||
20% to 79% by film (see
|
||||
[the full experiment log](model-bakeoff-2026-07.md#gallery-coverage-per-film)); an
|
||||
actor with no reference photo can never be recognized regardless of model
|
||||
quality, and counting them as a miss would penalize gallery coverage, not
|
||||
recognition accuracy.
|
||||
|
||||
Two further numbers are reported alongside F1:
|
||||
|
||||
**agreement_rate**: mean per-second Jaccard overlap
|
||||
(`|Pred ∩ GT| / |Pred ∪ GT|`), partial credit. Naming 2 of 3 present actors
|
||||
scores 2/3, not 0.
|
||||
|
||||
**exact_match_rate**: the fraction of sampled seconds where the pipeline's
|
||||
named set exactly equals X-Ray's, no partial credit. Far harsher, and
|
||||
dominated by recall, since any single missed actor zeroes that second.
|
||||
|
||||
## Reproduce
|
||||
|
||||
```bash
|
||||
python3 scripts/optimizer/second_score.py \
|
||||
--pred pred.json --xray experiments/xray/.../<xray_dir> \
|
||||
--gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5
|
||||
```
|
||||
|
||||
See also [the full experiment log](model-bakeoff-2026-07.md) for how `pred.json` is
|
||||
produced, and the [LVFace deep dive](lvface-deep-dive-2026-07.md) for what these
|
||||
mechanisms look like frame by frame.
|
||||
+78
-70
@@ -1,9 +1,11 @@
|
||||
# How we score against X-Ray
|
||||
|
||||
Every number in this report comes from one comparison, and that comparison
|
||||
has a mismatch at its core: the ground truth is scene-level, the pipeline's
|
||||
output is per-second, and the two do not mean the same thing. This page
|
||||
documents the comparison once so the findings can rely on it.
|
||||
Every number in this report, every F1 and misID count, comes from one
|
||||
comparison. The comparison has a mismatch at its core that shapes nearly
|
||||
every finding in this report: the ground truth is scene-level, the
|
||||
pipeline's output is per-second, and the two do not mean the same thing.
|
||||
This page documents that comparison once, so the findings pages can rely on
|
||||
it without re-explaining it.
|
||||
|
||||
## What Amazon X-Ray records
|
||||
|
||||
@@ -11,12 +13,12 @@ X-Ray ships three tables per film: `scenes.csv` (a list of `[start, end]`
|
||||
timespans), `people_in_scenes.csv` (which actors are credited in each
|
||||
scene), and `people.csv` (actor identities). There is no per-frame or
|
||||
per-second annotation anywhere in X-Ray. A scene might run 45 seconds, and
|
||||
X-Ray records one cast list for the entire span, not "on screen from second
|
||||
12 to second 30."
|
||||
X-Ray records one cast list for the entire span, not "on screen from
|
||||
second 12 to second 30."
|
||||
|
||||
To compare against per-second predictions, `second_score.py` expands every
|
||||
scene into per-second ground truth by copying the whole scene's cast list
|
||||
onto every second inside it:
|
||||
To compare this against per-second predictions, `second_score.py` expands
|
||||
every scene into per-second ground truth by copying the whole scene's cast
|
||||
list onto every second inside it:
|
||||
|
||||
```python
|
||||
for sn, (t0, t1) in spans.items():
|
||||
@@ -25,46 +27,48 @@ for sn, (t0, t1) in spans.items():
|
||||
timeline[t] = cast
|
||||
```
|
||||
|
||||
If X-Ray credits five actors to a 30-second scene, all five count as ground
|
||||
truth present for all 30 seconds, including seconds where only one is on
|
||||
screen. This is not a simplification the pipeline introduces; it is the only
|
||||
reading X-Ray's data supports, because X-Ray records nothing finer.
|
||||
That is the entire mechanism. If X-Ray credits five actors to a 30-second
|
||||
scene, all five count as ground truth present for all 30 seconds, including
|
||||
seconds where only one of them is on screen. This is not a simplification
|
||||
introduced by the pipeline; it is the only reading of X-Ray's data that is
|
||||
possible, because X-Ray itself does not record anything finer-grained.
|
||||
|
||||
## How the pipeline reports presence
|
||||
## Why an offscreen name can be scored correct
|
||||
|
||||
A presence claim is one actor owning one time window. How that window is
|
||||
derived is a tunable choice — a knob the optimizer weighs — with two modes:
|
||||
A name listed under Offscreen with a correct (green) label is not the
|
||||
pipeline guessing or padding its score. It is the pipeline correctly
|
||||
answering the question X-Ray actually asks: is this actor part of this
|
||||
scene. It answers that question using a presence window (`[start, end]`,
|
||||
held open across cuts by `anneal_sec` and `extinction_sec`), which matches
|
||||
X-Ray's scene-level semantics more closely than a raw per-frame detection
|
||||
would.
|
||||
|
||||
- **`track_extent` (default).** A claim is exactly `[first_seen, last_seen]`
|
||||
of a track the actor owned (AR-012), ending at the last sighting and never
|
||||
after (AR-013). There is no keep-alive: the withdrawn `anneal_sec` and the
|
||||
scene-tracker `extinction_sec` — which the July report's windows were held
|
||||
open by — are **gone**. A track that survives its own gaps needs no bridge;
|
||||
a gap after the final sighting is never claimed.
|
||||
- **`flood`.** Each claim is snapped to the shot it sits in, so an actor seen
|
||||
once anywhere in a shot is reported for the whole shot
|
||||
`[prev_boundary, next_boundary]`. Boundaries come from TransNetV2 shot
|
||||
detection when available, otherwise from the always-on histogram cut
|
||||
detector (`is_cut`). This trades precision for recall against X-Ray's
|
||||
scene-level granularity, and the optimizer decides per run whether it pays.
|
||||
A system that only reported "this actor is visible in this exact frame"
|
||||
would score worse against X-Ray's scene-level ground truth, producing a
|
||||
false negative every time the camera cuts away from a character who is
|
||||
still present in the scene. Not because it is wrong about the world, but
|
||||
because it would be answering a stricter, different question than the one
|
||||
X-Ray's data supports. The presence-window design exists specifically to
|
||||
answer X-Ray's actual question.
|
||||
|
||||
Do not confuse the surviving `track_extinction_sec` with the withdrawn
|
||||
scene `extinction_sec`: the former bounds how long a lost track stays
|
||||
available for **re-association** (a tracking question), and never extends a
|
||||
presence claim.
|
||||
## What this resolves and what it does not
|
||||
|
||||
## The two limits this does not resolve
|
||||
This resolves the semantic mismatch between a scene and an instant. It does
|
||||
not resolve two other limitations, both discussed in the
|
||||
[LVFace deep dive](lvface-deep-dive.md).
|
||||
|
||||
**The face-vs-presence ceiling.** X-Ray credits scene membership regardless
|
||||
of whether a face is ever visible: background crew, characters shot from
|
||||
behind, voice-only presence. No face pipeline can recover a face that never
|
||||
appears, so recall against X-Ray is a structural ceiling, not a defect.
|
||||
behind, voice-only presence. No amount of bridging recovers a face that
|
||||
never appears on screen. This is a hard ceiling on recall, not a defect.
|
||||
|
||||
**Flood-fill can overshoot.** Snapping to a shot correctly answers "still in
|
||||
this scene" through an intra-scene cut, but a shot boundary is not a scene
|
||||
boundary: on a film with sparse cuts, flood-fill can carry an actor across a
|
||||
long "shot" they only briefly appeared in. This is why flood-fill is a knob,
|
||||
not a default — its value depends on the film's cut density.
|
||||
**Extinction bridging can overshoot.** The same presence-window mechanism
|
||||
that correctly answers "still in this scene" during a normal cut can also
|
||||
bridge across a scene boundary it has no way to detect. A hard cut into a
|
||||
different scene with no faces, such as closing credits, carries the
|
||||
previous scene's identities forward until the window expires. This is the
|
||||
mechanism behind Downton Abbey's recall collapse, documented in the deep
|
||||
dive.
|
||||
|
||||
## Precision, recall, and the misID weighting
|
||||
|
||||
@@ -73,46 +77,49 @@ Per sampled second `t`:
|
||||
**TPI** (true positive instances): actors both X-Ray and the pipeline agree
|
||||
are present.
|
||||
|
||||
**FPI** (false positive instances): actors the pipeline reports that are not
|
||||
in X-Ray's cast for this second, split into:
|
||||
**FPI** (false positive instances): actors the pipeline reports that are
|
||||
not in X-Ray's cast for this second. Split into two categories:
|
||||
|
||||
- **FPI_incast**: the actor is in the film's cast, just not credited to this
|
||||
scene. A timing or boundary slip.
|
||||
- **FPI_incast**: the actor is in the film's cast, just not credited to
|
||||
this particular scene. A timing or boundary slip.
|
||||
- **FPI_misid**: the actor is not in the film's cast at all. A genuine
|
||||
wrong-identity error, weighted **10×** in the precision objective, because
|
||||
naming someone not even in the film is categorically worse than a few
|
||||
seconds of scene-boundary slop.
|
||||
wrong-identity error, weighted 10x in the precision objective, because
|
||||
naming someone who is not even in the film is a categorically worse
|
||||
error than a few seconds of scene-boundary slop.
|
||||
|
||||
!!! note "Every headline `P` and `F1` is misID-weighted"
|
||||
|
||||
Precision puts each `FPI_misid` into the denominator 10 times
|
||||
The precision reported throughout this report, and therefore the F1
|
||||
derived from it, puts each `FPI_misid` into the denominator **10 times**
|
||||
(`precision = TPI / (TPI + FPI_incast + 10·FPI_misid)`,
|
||||
[`second_score.py`](https://REPOLINK/scripts/optimizer/second_score.py)).
|
||||
This deliberately punishes naming an out-of-film actor far harder than a
|
||||
boundary slip, so the `P` column is not raw precision and a misID-heavy
|
||||
film's `P` is depressed super-linearly.
|
||||
This is deliberate: the whole point is to punish naming an out-of-film
|
||||
actor far harder than a scene-boundary slip. But it means the `P` column
|
||||
is not raw precision, and a misID-heavy film's `P` is depressed
|
||||
super-linearly. `second_score.py` also emits an unweighted `precision_raw`
|
||||
(always ≥ the weighted `P`); where the gap matters, The Many Saints of
|
||||
Newark, weighted `P` 54.7% vs. raw 78.4%, the [LVFace deep dive](lvface-deep-dive.md)
|
||||
reports both. When comparing `P` across films, remember you are comparing a
|
||||
quantity that penalizes misIDs, not just a hit rate.
|
||||
|
||||
**FN** (false negatives): actors X-Ray lists that the pipeline never reports,
|
||||
counted **only** for actors who have a gallery reference embedding. An actor
|
||||
with no reference photo can never be recognized, and counting them as a miss
|
||||
would measure gallery coverage, not recognition accuracy.
|
||||
**FN** (false negatives): actors X-Ray lists that the pipeline never
|
||||
reports, counted only for actors who have a gallery reference embedding.
|
||||
Across the 9-film benchmark, coverage of X-Ray's credited cast ranges from
|
||||
20% to 79% by film (see
|
||||
[the full experiment log](model-bakeoff.md#gallery-coverage-per-film)); an
|
||||
actor with no reference photo can never be recognized regardless of model
|
||||
quality, and counting them as a miss would penalize gallery coverage, not
|
||||
recognition accuracy.
|
||||
|
||||
Two further numbers accompany F1:
|
||||
Two further numbers are reported alongside F1:
|
||||
|
||||
**agreement_rate**: mean per-second Jaccard overlap
|
||||
(`|Pred ∩ GT| / |Pred ∪ GT|`) — partial credit, so naming 2 of 3 present
|
||||
actors scores 2/3, not 0.
|
||||
(`|Pred ∩ GT| / |Pred ∪ GT|`), partial credit. Naming 2 of 3 present actors
|
||||
scores 2/3, not 0.
|
||||
|
||||
**exact_match_rate**: the fraction of seconds where the pipeline's named set
|
||||
exactly equals X-Ray's — no partial credit, dominated by recall.
|
||||
|
||||
## The benchmark set
|
||||
|
||||
Unlike the July report — which trained on a 3-film subset and validated on
|
||||
held-out films to keep evaluations fast — this run scores **all 9 films on
|
||||
every evaluation**. The registry one-clock fix and uncapped dumps made
|
||||
full-set replay affordable, so the reported optimum is tuned against the
|
||||
complete set rather than a training subset.
|
||||
**exact_match_rate**: the fraction of sampled seconds where the pipeline's
|
||||
named set exactly equals X-Ray's, no partial credit. Far harsher, and
|
||||
dominated by recall, since any single missed actor zeroes that second.
|
||||
|
||||
## Reproduce
|
||||
|
||||
@@ -122,5 +129,6 @@ python3 scripts/optimizer/second_score.py \
|
||||
--gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5
|
||||
```
|
||||
|
||||
See the [full experiment log](model-bakeoff.md) for how `pred.json` is
|
||||
produced and where the shipped `src/config.hpp` defaults come from.
|
||||
See also [the full experiment log](model-bakeoff.md) for how `pred.json` is
|
||||
produced, and the [LVFace deep dive](lvface-deep-dive.md) for what these
|
||||
mechanisms look like frame by frame.
|
||||
|
||||
@@ -1,348 +0,0 @@
|
||||
> **Archived (July 2026).** This report covers the pre-opencv5 framework and the 4-model ArcFace/LVFace bake-off. It is superseded by the current [experiment log](model-bakeoff.md) for the opencv5 build. Kept for provenance; the numbers here are historical.
|
||||
|
||||
# Full experiment log
|
||||
|
||||
This page reports how the pipeline performs across three questions: which
|
||||
embedding model is best, whether restricting the gallery to a film's
|
||||
credited cast helps, and whether promoting confidently identified poses into
|
||||
a per-film gallery annex helps. It also documents the replay architecture
|
||||
that made testing all three questions in one pass practical, and every
|
||||
caveat needed to trust the numbers.
|
||||
|
||||
Read [How we score against X-Ray](methodology-2026-07.md) first for what F1,
|
||||
precision, recall, and misID mean in this report. All numbers below use the
|
||||
per-second metric
|
||||
([`scripts/optimizer/second_score.py`](https://REPOLINK/scripts/optimizer/second_score.py)).
|
||||
|
||||
r50 (ArcFace w600k-R50) is excluded from the detailed comparison below. Its
|
||||
gallery was built with roughly 30% fewer reference images per actor than the
|
||||
other three models on the identical source photos (10808 vs 15055 total
|
||||
embeddings across the same 2418 actors), which confounds any direct
|
||||
comparison of its scores against the others. It remains in the
|
||||
[calibration curve comparison](best-model-2026-07.md#first-signal-calibration-curves),
|
||||
which does not depend on the training benchmark.
|
||||
|
||||
## Why replay makes this affordable
|
||||
|
||||
Decoding video and running face detection, alignment, and embedding is the
|
||||
expensive part of this pipeline. Everything downstream of that (tracking,
|
||||
identity matching, scene aggregation) is cheap. KPN++'s node/network
|
||||
structure means those two stages are separate components connected by
|
||||
typed channels, so the expensive stage can run once per film, cache its
|
||||
output, and the cheap stage can be re-run against that cache as many times
|
||||
as needed with different Config values.
|
||||
|
||||
`scene_analyze --dump-embeddings out.h5` runs the expensive half once per
|
||||
film and writes per-frame face detections and embeddings to HDF5
|
||||
([`scripts/optimizer/SCHEMA.md`](https://REPOLINK/scripts/optimizer/SCHEMA.md)).
|
||||
[`scripts/optimizer/replay.py`](https://REPOLINK/scripts/optimizer/replay.py)
|
||||
then re-assembles the real C++ `face_tracker`, `identity_matcher`, and
|
||||
`scene_tracker` nodes into a Python-driven KPN network and replays a
|
||||
film's cached embeddings through them, varying `prob_threshold`,
|
||||
`anneal_sec`, `extinction_sec`, and `expand_gallery` freely. No GPU
|
||||
inference and no video decode happen during a replay; each one completes
|
||||
in seconds. This is what makes a 512-evaluation differential-evolution
|
||||
search per model, per gallery mode, per expansion setting, tractable, and
|
||||
what made the full held-out validation across three models in this report
|
||||
possible in one session rather than requiring three full re-encodes of the
|
||||
benchmark set.
|
||||
|
||||
`optimize.py` runs `differential_evolution` over this replay function as its
|
||||
objective, with DE-level parallelism (multiple candidate configs evaluated
|
||||
concurrently, each spawning its own replay subprocesses) on top of it. The
|
||||
practical ceiling on this machine's GPU was 8 concurrent replay processes;
|
||||
9 silently degraded every score to 0.0% (well-formed output, wrong numbers,
|
||||
not a crash), so `optimize.py` was run at `REPLAY_WORKERS=4 DE_WORKERS=2`.
|
||||
|
||||
## Search space
|
||||
|
||||
`popsize=10, maxiter=15` per combo (3 parameters, up to 512 evaluations,
|
||||
usually stopping earlier on DE's convergence tolerance).
|
||||
`anneal_sec`/`extinction_sec` bounds were widened from 1-30/1-15 to 1-60/1-60
|
||||
partway through the sweep. r50's 4 combos finished before the widening and
|
||||
used the old, narrower bounds; this is one more reason r50 is excluded from
|
||||
direct comparison here.
|
||||
|
||||
## Training films and held-out films
|
||||
|
||||
9 films have dumped embeddings across all 4 models. 4 were used for
|
||||
optimization:
|
||||
|
||||
- Café Society (62-cast)
|
||||
- Lord of War (64-cast)
|
||||
- Scarface (67-cast)
|
||||
- Sound of Metal (14-cast)
|
||||
|
||||
5 were held out, never seen by any optimizer run:
|
||||
|
||||
- Benny & Joon
|
||||
- Downton Abbey: A New Era
|
||||
- Lovelace
|
||||
- The Many Saints of Newark
|
||||
- Valerian and the City of a Thousand Planets
|
||||
|
||||
## Gallery coverage per film
|
||||
|
||||
The gallery has reference embeddings for 2418 actors, but coverage of any
|
||||
given film's credited cast varies widely. This was previously reported as
|
||||
one flat number (67% of X-Ray cast lacking a reference embedding, averaged
|
||||
across the whole benchmark); the per-film breakdown is:
|
||||
|
||||
| film | cast credited | in gallery | coverage |
|
||||
|---|---|---|---|
|
||||
| Lord of War | 64 | 13 | 20.3% |
|
||||
| Scarface | 67 | 15 | 22.4% |
|
||||
| The Many Saints of Newark | 48 | 13 | 27.1% |
|
||||
| Café Society | 62 | 17 | 27.4% |
|
||||
| Lovelace | 42 | 15 | 35.7% |
|
||||
| Valerian and the City of a Thousand Planets | 36 | 13 | 36.1% |
|
||||
| Benny & Joon | 23 | 12 | 52.2% |
|
||||
| Downton Abbey: A New Era | 36 | 22 | 61.1% |
|
||||
| Sound of Metal | 14 | 11 | 78.6% |
|
||||
|
||||
Two training films (Lord of War, Scarface) have the worst coverage in the
|
||||
set, 20-22%. Their training-set F1 numbers below are partly capped by
|
||||
missing references, not purely by model quality. Downton Abbey has 61%
|
||||
coverage, the second-best in the benchmark, yet the worst held-out recall
|
||||
of any film (39.4%, LVFace). Its recall problem is not primarily a coverage
|
||||
problem; it is the extinction-bridging failure documented in the
|
||||
[LVFace deep dive](lvface-deep-dive-2026-07.md#mechanism-1-extinction-bridging).
|
||||
Reproduce with `scripts/docs/gallery_coverage_per_film.py`.
|
||||
|
||||
## Training results, 3 models × 2 gallery modes × 2 expansion settings
|
||||
|
||||
Ranked by F1. misid = FPI_misid, the count of true wrong-actor
|
||||
identifications (naming someone not in the film's cast at all), distinct
|
||||
from FPI, which also includes in-cast timing slips.
|
||||
|
||||
Each combo's row is its best **full-coverage** evaluation: the highest-F1 DE
|
||||
evaluation in which all 4 training films replayed without a timeout (see
|
||||
[Dropped-film scoring](#a-scoring-bug-worth-recording-dropped-film-evaluations)
|
||||
below for why this qualifier is load-bearing and not the same as `argmax F1`
|
||||
over the raw sweep).
|
||||
|
||||
| combo | F1 | P | R | TPI | FPI | misid | FN |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| LVFace-B_Glint360K_restricted_exp | 78.3% | 91.0% | 68.9% | 42830 | 3782 | 60 | 19492 |
|
||||
| LVFace-B_Glint360K_restricted_noexp | 76.7% | 91.5% | 66.2% | 41149 | 3400 | 59 | 21173 |
|
||||
| arcface_w600k_mbf_restricted_exp | 76.2% | 90.0% | 66.2% | 64328 | 7480 | 0 | 33234 |
|
||||
| arcface_r18_restricted_exp | 75.5% | 87.6% | 66.5% | 41399 | 5666 | 60 | 20923 |
|
||||
| LVFace-B_Glint360K_full_exp | 75.3% | 89.7% | 65.4% | 47757 | 3407 | 232 | 26966 |
|
||||
| arcface_w600k_mbf_restricted_noexp | 75.0% | 91.1% | 63.9% | 39752 | 3465 | 60 | 22570 |
|
||||
| arcface_r18_restricted_noexp | 73.5% | 91.3% | 61.7% | 38299 | 3220 | 60 | 24023 |
|
||||
| LVFace-B_Glint360K_full_noexp | 72.3% | 88.3% | 61.8% | 40363 | 3503 | 244 | 25850 |
|
||||
| arcface_w600k_mbf_full_exp | 72.0% | 87.7% | 61.4% | 39875 | 3729 | 240 | 26338 |
|
||||
| arcface_w600k_mbf_full_noexp | 71.0% | 93.2% | 57.9% | 41699 | 2472 | 56 | 33024 |
|
||||
| arcface_r18_full_exp | 69.1% | 87.6% | 57.7% | 37342 | 3119 | 242 | 28871 |
|
||||
| arcface_r18_full_noexp | 66.6% | 91.3% | 53.1% | 34314 | 2362 | 107 | 31899 |
|
||||
|
||||
|
||||
|
||||
The two clearest patterns: every model's best-scoring combo uses the
|
||||
restricted gallery, and LVFace leads within both gallery modes. `full_exp`
|
||||
(the shipped combination) is the best-scoring option that uses only
|
||||
features the running application currently supports; restriction is not
|
||||
wired into the application yet (see
|
||||
[Whole vs. cast-restricted gallery](gallery-scope-2026-07.md)).
|
||||
|
||||
### A scoring bug worth recording: dropped-film evaluations
|
||||
|
||||
The numbers above are corrected ones. The raw `rep4_best_*.json` files, and an
|
||||
earlier version of this table, reported a different `arcface_w600k_mbf_full_noexp`
|
||||
row: **74.2% F1 at TPI 12645**, a third the TPI of every sibling combo. That was
|
||||
not a better config; it was an artifact of how the optimizer aggregates.
|
||||
|
||||
`optimize.py` builds each candidate's score from only the films whose replay
|
||||
subprocess returned (`per_film = [m for m in ex.map(_one, films) if m is not
|
||||
None]`), then **averages** F1/precision/recall and **sums** TPI/FPI/misID over
|
||||
just those survivors. When a film's replay times out (the sweep ran near the
|
||||
8-process concurrency ceiling, so this happened intermittently), that film
|
||||
silently drops from both. A candidate whose hardest film timed out is therefore
|
||||
scored on an easier subset, and differential evolution, maximizing that score,
|
||||
will happily converge onto exactly such a candidate. For `mbf_full_noexp` the
|
||||
reported winner was one of 7 evaluations (out of 512) whose TPI had collapsed to
|
||||
a partial-film subset; its median-coverage evaluations sit around 51686 TPI.
|
||||
|
||||
The fix here was to re-derive each combo's best row from its DE trajectory
|
||||
(`experiments/trajectories/rep4_*.jsonl`), keeping only evaluations within 30% of
|
||||
that combo's median TPI (full 4-film coverage) before taking the best F1. This
|
||||
needs no re-running, the honest best configuration was already in the sweep,
|
||||
just not the one `argmax F1` selected. Three combos moved: `mbf_full_noexp`
|
||||
74.2% → **71.0%**, `LVFace_full_noexp` 72.4% → **72.3%** (and its misID, 0 → 244,
|
||||
was itself a dropped-film artifact), `mbf_restricted_exp` 76.5% → **76.2%**. The
|
||||
shipped LVFace `full_exp` winner was unaffected, its reported evaluation already
|
||||
had full coverage (TPI 47757 ≈ median). `experiment_charts.py` applies the same
|
||||
`clean_best` filter, so every figure on this page matches the corrected table.
|
||||
The underlying `optimize.py` aggregation is also being fixed so a dropped-film
|
||||
evaluation can never be selected as a winner again.
|
||||
|
||||
### Per-film training breakdown
|
||||
|
||||
The 75.3% LVFace training figure is a macro average across 4 films, not a
|
||||
uniform result:
|
||||
|
||||
| film | LVFace F1 | mbf F1 | r18 F1 | best model |
|
||||
|---|---|---|---|---|
|
||||
| Café Society | 68.1% | 62.2% | 60.1% | LVFace |
|
||||
| Lord of War | 75.6% | 77.2% | 75.6% | mbf |
|
||||
| Scarface | 71.5% | 68.6% | 64.1% | LVFace |
|
||||
| Sound of Metal | 78.8% | 76.5% | 71.6% | LVFace |
|
||||
|
||||
LVFace does not win every training film. mbf scores higher on Lord of War
|
||||
(77.2% vs 75.6%). LVFace's own training-film range is 68.1% to 78.8%, a
|
||||
10.7pp spread, smaller than the 37pp spread seen on held-out films but real.
|
||||
Reproduce with `scripts/docs/run_holdout_all_models.py --films training`.
|
||||
|
||||
## Held-out validation, all 3 models
|
||||
|
||||
The training matrix above is training-set fit. Each model's own tuned
|
||||
`full_exp` config was replayed against the 5 held-out films, scored the
|
||||
same way:
|
||||
|
||||
| film | LVFace F1 | mbf F1 | r18 F1 |
|
||||
|---|---|---|---|
|
||||
| Benny & Joon | 83.0% | 78.5% | 77.1% |
|
||||
| Lovelace | 77.5% | 73.7% | 72.2% |
|
||||
| Valerian and the City of a Thousand Planets | 74.1% | 70.2% | 71.0% |
|
||||
| Downton Abbey: A New Era | 56.2% | 55.0% | 53.0% |
|
||||
| The Many Saints of Newark | 46.3% | 44.5% | 42.1% |
|
||||
| **macro average** | **67.4%** | **64.4%** | **63.1%** |
|
||||
|
||||
LVFace scores highest on every one of the 5 held-out films; the ranking
|
||||
never flips. Total misIDs across the 5 films: LVFace 1032, mbf 2197, r18
|
||||
1224. LVFace has less than half mbf's misID count while also scoring
|
||||
higher on every film. This directly confirms the model choice out of
|
||||
sample; it is not inferred from the training numbers alone. See the
|
||||
[LVFace deep dive](lvface-deep-dive-2026-07.md) for frame-level detail on where and
|
||||
why LVFace still fails on the two worst films. Reproduce with
|
||||
`scripts/docs/run_holdout_all_models.py`.
|
||||
|
||||
## Two effects in isolation: gallery scope and pose expansion
|
||||
|
||||
Averaging across the 3 compared models (r50 excluded) isolates each variable
|
||||
from model choice.
|
||||
|
||||
**Gallery scope**, averaged over both expansion settings and all 3 models
|
||||
(6 evaluations per row):
|
||||
|
||||
| scope | F1 | P | R | total misID |
|
||||
|---|---|---|---|---|
|
||||
| full | 71.1% | 89.6% | 59.6% | 1121 |
|
||||
| restricted | 75.9% | 90.4% | 65.6% | 299 |
|
||||
|
||||
Restriction improves every metric at once. This is not a precision/recall
|
||||
trade: +4.8pp F1, +6.0pp recall, and roughly a quarter the misIDs. Fewer
|
||||
candidates in the matcher's search space means fewer opportunities for a
|
||||
lookalike false match, and the recall gain shows this does not cost real
|
||||
detections. Restriction is currently an offline optimizer technique, not a
|
||||
runtime feature of the application; see
|
||||
[Whole vs. cast-restricted gallery](gallery-scope-2026-07.md) for what building it
|
||||
into the application would require.
|
||||
|
||||
**Pose expansion** (promoting a confidently identified track's novel-pose
|
||||
views into a per-film gallery annex,
|
||||
[`src/gallery/track_gallery.hpp`](https://REPOLINK/src/gallery/track_gallery.hpp)):
|
||||
|
||||
| scope | expansion | F1 | R | misID |
|
||||
|---|---|---|---|---|
|
||||
| full | off | 70.0% | 57.6% | 407 |
|
||||
| full | on | 72.1% | 61.5% | 714 |
|
||||
| restricted | off | 75.1% | 63.9% | 179 |
|
||||
| restricted | on | 76.7% | 67.2% | 120 |
|
||||
|
||||
In restricted mode, expansion is a clean win: +1.6pp F1, +3.3pp recall,
|
||||
misID drops. The annex only competes against the film's own roughly 15-actor
|
||||
cast, so a new pose of a known actor is unlikely to be confused with someone
|
||||
else. In full mode, expansion buys +2.1pp F1 and +3.9pp recall but at a real
|
||||
cost: misID rises from 407 to 714 as the same new-pose view now competes
|
||||
against the full 2418-actor gallery, where a confidently learned pose is more
|
||||
likely to match the wrong person. On the full gallery it is a recall-vs-misID
|
||||
trade, not a free gain. This training-set effect
|
||||
did not reproduce on held-out data; see
|
||||
[Does pose expansion help?](pose-expansion-2026-07.md) for the full held-out test
|
||||
and the two methodology bugs caught while checking it.
|
||||
|
||||
## Calibration curves
|
||||
|
||||
Each gallery carries a fitted Platt sigmoid `P(match | sim) = σ(a·sim + b)`,
|
||||
stored directly in the gallery HDF5
|
||||
([`src/gallery/gallery_calibration.hpp`](https://REPOLINK/src/gallery/gallery_calibration.hpp)).
|
||||
This measures discriminative power independent of whatever
|
||||
`prob_threshold` a given run used:
|
||||
|
||||
|
||||
|
||||
LVFace has the steepest curve (`a=17.7` vs 15.3-16.2 for the ArcFace
|
||||
variants) and the lowest P=0.5 decision boundary (similarity 0.23 vs
|
||||
0.27-0.31), separating same-actor from different-actor pairs more
|
||||
confidently at a lower similarity than any ArcFace variant tested,
|
||||
including r50. Generated by
|
||||
[`scripts/docs/calibration_chart.py`](https://REPOLINK/scripts/docs/calibration_chart.py).
|
||||
|
||||
## Extinction and anneal window search
|
||||
|
||||
Every one of the 512 DE evaluations for the winning LVFace `full_exp`
|
||||
combo, plotted over the `prob_threshold` × `extinction_sec` plane:
|
||||
|
||||
|
||||
|
||||
Nearly everything scoring well sits at `extinction_sec` above 50, across a
|
||||
wide range of thresholds. Short extinction windows are uniformly weaker:
|
||||
under a strict threshold, there is no good configuration in that region of
|
||||
the search space. The optimizer converged with `anneal_sec=59.2,
|
||||
extinction_sec=59.2`, about 99% of the widened 60s bound, which raises an
|
||||
open question not resolved in this round: does performance keep improving
|
||||
past 60s, or does it plateau there. Not chased further this pass.
|
||||
|
||||
## Caveats
|
||||
|
||||
- r50's 4 combos used the older, narrower search bounds (1-30/1-15 instead
|
||||
of 1-60/1-60) and are further confounded by its thinner gallery. Excluded
|
||||
from all comparisons above except calibration.
|
||||
- The shipped defaults use `full_exp` (75.3% training F1), not the
|
||||
higher-scoring `restricted_exp` (78.3%), because cast restriction is not
|
||||
a runtime feature of the application yet.
|
||||
- `expand_gallery` is mode-dependent, not a free win. Averaged across models
|
||||
on the full gallery it trades misIDs for recall (see the pose-expansion
|
||||
table). For LVFace specifically, though, `full_exp` beats `full_noexp` on
|
||||
every axis at once (F1 75.3 vs 72.3, precision 89.7 vs 88.3, recall 65.4 vs
|
||||
61.8, misID 232 vs 244), so the shipped `full_exp` is a clean choice for
|
||||
this model, not an F1-vs-safety trade. (An earlier version of this page
|
||||
reported `full_noexp` at 72.4% with zero misIDs and higher precision, which
|
||||
made it look like the safer option; that was the dropped-film artifact
|
||||
described above, not a real property of the config.)
|
||||
- Switching the default model is an operational change: any gallery built
|
||||
from a different model's embeddings must be rebuilt before the new
|
||||
default takes effect.
|
||||
|
||||
## Reproduce
|
||||
|
||||
```bash
|
||||
# 4-film training matrix, all 4 models × 2 gallery modes × 2 expansion settings
|
||||
bash experiments/run_rep4_subprocess.sh
|
||||
|
||||
# single combo
|
||||
SAE_EXPAND=1 REPLAY_WORKERS=4 DE_WORKERS=2 python3 scripts/optimizer/optimize.py \
|
||||
--manifest experiments/manifests/rep4_LVFace-B_Glint360K_full.json \
|
||||
--gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5 \
|
||||
--params prob_threshold:0.5:0.999 anneal_sec:1:60 extinction_sec:1:60 \
|
||||
--popsize 10 --maxiter 15 --trajectory traj.jsonl --out best.json
|
||||
|
||||
# held-out validation, all 3 models, 5 films
|
||||
python3 scripts/docs/run_holdout_all_models.py --out docs_data/holdout_all_models.json
|
||||
|
||||
# per-film training breakdown, all 3 models, 4 films
|
||||
python3 scripts/docs/run_holdout_all_models.py --films training --out docs_data/training_per_film.json
|
||||
|
||||
# gallery coverage per film
|
||||
python3 scripts/docs/gallery_coverage_per_film.py --out docs_data/gallery_coverage_per_film.json
|
||||
|
||||
# regenerate this page's charts from experiments/ artifacts
|
||||
python3 scripts/docs/experiment_charts.py --out-dir docs/assets/images
|
||||
|
||||
# one frame per distinct out-of-cast name across all 9 films (used in the deep dive)
|
||||
python3 scripts/docs/first_fpi_frames.py
|
||||
```
|
||||
|
||||
See also the session log
|
||||
[`experiments/SESSION_STATE.md`](https://REPOLINK/experiments/SESSION_STATE.md).
|
||||
+306
-158
@@ -1,198 +1,346 @@
|
||||
# Full experiment log (opencv5)
|
||||
# Full experiment log
|
||||
|
||||
This is the complete log behind the current opencv5 build: how the pipeline is
|
||||
tuned, what the shipped configuration is and where every number in it comes from,
|
||||
and how the learned scene-boundary detector took per-second actor-presence F1 from
|
||||
the low-60s to **74.9%** across the nine-film Amazon X-Ray benchmark — under honest
|
||||
leave-one-out.
|
||||
This page reports how the pipeline performs across three questions: which
|
||||
embedding model is best, whether restricting the gallery to a film's
|
||||
credited cast helps, and whether promoting confidently identified poses into
|
||||
a per-film gallery annex helps. It also documents the replay architecture
|
||||
that made testing all three questions in one pass practical, and every
|
||||
caveat needed to trust the numbers.
|
||||
|
||||
Read [How we score against X-Ray](methodology.md) first for what F1, precision,
|
||||
recall, and misID mean here. Every number below uses the per-second metric
|
||||
([`scripts/optimizer/second_score.py`](https://REPOLINK/scripts/optimizer/second_score.py)):
|
||||
the film is sampled once per second, and at each second the set of names the
|
||||
pipeline reports present is compared against Amazon X-Ray's scene cast for that
|
||||
second. X-Ray's ground truth is scene-level; the pipeline's output is per-second.
|
||||
That mismatch shapes every result.
|
||||
Read [How we score against X-Ray](methodology.md) first for what F1,
|
||||
precision, recall, and misID mean in this report. All numbers below use the
|
||||
per-second metric
|
||||
([`scripts/optimizer/second_score.py`](https://REPOLINK/scripts/optimizer/second_score.py)).
|
||||
|
||||
## The benchmark
|
||||
|
||||
Nine films with public Amazon X-Ray scene data, all scored with the same
|
||||
LVFace-B Glint360K gallery:
|
||||
|
||||
Benny & Joon · Café Society · Downton Abbey: A New Era · Lord of War · Lovelace ·
|
||||
The Many Saints of Newark · Scarface · Sound of Metal · Valerian.
|
||||
|
||||
Two of these — Café Society and Scarface — are low-contrast, uniformly-graded
|
||||
films that break naive cut detection. They are deliberately kept in the benchmark
|
||||
because they are where the interesting failures live.
|
||||
r50 (ArcFace w600k-R50) is excluded from the detailed comparison below. Its
|
||||
gallery was built with roughly 30% fewer reference images per actor than the
|
||||
other three models on the identical source photos (10808 vs 15055 total
|
||||
embeddings across the same 2418 actors), which confounds any direct
|
||||
comparison of its scores against the others. It remains in the
|
||||
[calibration curve comparison](best-model.md#first-signal-calibration-curves),
|
||||
which does not depend on the training benchmark.
|
||||
|
||||
## Why replay makes this affordable
|
||||
|
||||
Decoding video and running face detection, alignment, and embedding is the
|
||||
expensive part of the pipeline. Everything downstream — tracking, identity
|
||||
matching, scene aggregation — is cheap. KPN++'s node/network structure keeps those
|
||||
two halves as separate components joined by typed channels, so the expensive half
|
||||
runs once per film and caches its output, and the cheap half can be re-run against
|
||||
that cache as often as needed with different `Config` values.
|
||||
expensive part of this pipeline. Everything downstream of that (tracking,
|
||||
identity matching, scene aggregation) is cheap. KPN++'s node/network
|
||||
structure means those two stages are separate components connected by
|
||||
typed channels, so the expensive stage can run once per film, cache its
|
||||
output, and the cheap stage can be re-run against that cache as many times
|
||||
as needed with different Config values.
|
||||
|
||||
`scene_analyze --dump-embeddings out.h5` runs the expensive half once and writes
|
||||
per-frame detections, embeddings, and (for the scene detector) per-frame RGB
|
||||
histograms to HDF5. [`scripts/optimizer/replay.py`](https://REPOLINK/scripts/optimizer/replay.py)
|
||||
re-assembles the real C++ `face_tracker`, `identity_matcher`, and scene nodes into
|
||||
a Python-driven KPN network and replays a film's cache through them, varying every
|
||||
tuning knob freely. No GPU inference and no video decode happen during a replay, so
|
||||
a full differential-evolution search over all nine films is tractable in one
|
||||
session rather than requiring re-encodes.
|
||||
`scene_analyze --dump-embeddings out.h5` runs the expensive half once per
|
||||
film and writes per-frame face detections and embeddings to HDF5
|
||||
([`scripts/optimizer/SCHEMA.md`](https://REPOLINK/scripts/optimizer/SCHEMA.md)).
|
||||
[`scripts/optimizer/replay.py`](https://REPOLINK/scripts/optimizer/replay.py)
|
||||
then re-assembles the real C++ `face_tracker`, `identity_matcher`, and
|
||||
`scene_tracker` nodes into a Python-driven KPN network and replays a
|
||||
film's cached embeddings through them, varying `prob_threshold`,
|
||||
`anneal_sec`, `extinction_sec`, and `expand_gallery` freely. No GPU
|
||||
inference and no video decode happen during a replay; each one completes
|
||||
in seconds. This is what makes a 512-evaluation differential-evolution
|
||||
search per model, per gallery mode, per expansion setting, tractable, and
|
||||
what made the full held-out validation across three models in this report
|
||||
possible in one session rather than requiring three full re-encodes of the
|
||||
benchmark set.
|
||||
|
||||
Two concurrency limits are load-bearing and were paid for in wedged runs: replays
|
||||
run at `DE_WORKERS=1` (concurrent DE candidates wedge the ROCm GPU), and each
|
||||
candidate's per-film replays run at `REPLAY_WORKERS=8` with stderr discarded (the
|
||||
replay sink's per-second prints otherwise flood the captured pipe and hang the
|
||||
subprocess).
|
||||
`optimize.py` runs `differential_evolution` over this replay function as its
|
||||
objective, with DE-level parallelism (multiple candidate configs evaluated
|
||||
concurrently, each spawning its own replay subprocesses) on top of it. The
|
||||
practical ceiling on this machine's GPU was 8 concurrent replay processes;
|
||||
9 silently degraded every score to 0.0% (well-formed output, wrong numbers,
|
||||
not a crash), so `optimize.py` was run at `REPLAY_WORKERS=4 DE_WORKERS=2`.
|
||||
|
||||
## The tuning knobs
|
||||
## Search space
|
||||
|
||||
The opencv5 refactor replaced the old three-knob search with a **ten-knob**
|
||||
differential-evolution sweep. The knobs, and their shipped values:
|
||||
`popsize=10, maxiter=15` per combo (3 parameters, up to 512 evaluations,
|
||||
usually stopping earlier on DE's convergence tolerance).
|
||||
`anneal_sec`/`extinction_sec` bounds were widened from 1-30/1-15 to 1-60/1-60
|
||||
partway through the sweep. r50's 4 combos finished before the widening and
|
||||
used the old, narrower bounds; this is one more reason r50 is excluded from
|
||||
direct comparison here.
|
||||
|
||||
| knob | shipped | what it controls |
|
||||
| ---- | ------: | ---------------- |
|
||||
| `prob_threshold` | 0.485 | posterior P(match) above which a track is named |
|
||||
| `ownership_logodds` | 1.72 | log-odds a track needs before it produces presence |
|
||||
| `track_extinction_sec` | 31.0 | how long an idle track is held for re-detection |
|
||||
| `track_alpha` | 0.435 | tracker cost mix (0 = embedding only, 1 = spatial only) |
|
||||
| `evidence_rho_max` | 0.204 | evidence weighting ceiling |
|
||||
| `evidence_admit_below` | 0.784 | admit new evidence below this similarity |
|
||||
| `match_prior` | 0.433 | base-rate prior on a match |
|
||||
| `expand_band_lo` | 0.804 | low edge of the pose-expansion similarity band |
|
||||
| `expand_band_hi` | 0.952 | high edge of the pose-expansion band |
|
||||
| `presence_mode` | flood | track-extent vs scene flood-fill |
|
||||
## Training films and held-out films
|
||||
|
||||
The DE run over the first nine knobs (flood off, track-extent presence) converged
|
||||
at **64.0% macro F1** over 345 evaluations. Those values are the shipped
|
||||
[`src/config.hpp`](https://REPOLINK/src/config.hpp) defaults.
|
||||
9 films have dumped embeddings across all 4 models. 4 were used for
|
||||
optimization:
|
||||
|
||||
|
||||
- Café Society (62-cast)
|
||||
- Lord of War (64-cast)
|
||||
- Scarface (67-cast)
|
||||
- Sound of Metal (14-cast)
|
||||
|
||||
The `track_extinction_sec` knob is worth calling out: at 31 s it holds an idle
|
||||
track alive for re-detection long enough to bridge an actor turning away or leaving
|
||||
frame briefly, without bridging across a genuine scene change. Getting this knob
|
||||
and the tracker/registry to agree on **one clock** (the evidence watermark, not
|
||||
wall-clock) was a correctness fix, not a tuning choice — before it, votes were
|
||||
silently dropped at the reap horizon.
|
||||
5 were held out, never seen by any optimizer run:
|
||||
|
||||
## The step change: flood-fill on learned boundaries
|
||||
- Benny & Joon
|
||||
- Downton Abbey: A New Era
|
||||
- Lovelace
|
||||
- The Many Saints of Newark
|
||||
- Valerian and the City of a Thousand Planets
|
||||
|
||||
The 64.0% above is track-extent presence: an actor is reported only while an actual
|
||||
track is alive. **Flood-fill** instead reports an actor for the whole shot once
|
||||
they are seen in it — but that is only correct if the shot boundaries are good.
|
||||
## Gallery coverage per film
|
||||
|
||||
With the old grayscale cut detector as the boundary source, flood-fill barely beat
|
||||
doing nothing (**64.0%**) and actively broke Scarface, where the detector fires
|
||||
once in 10,204 frames and flood then smears every actor across the whole film
|
||||
(precision collapses to 26%).
|
||||
The gallery has reference embeddings for 2418 actors, but coverage of any
|
||||
given film's credited cast varies widely. This was previously reported as
|
||||
one flat number (67% of X-Ray cast lacking a reference embedding, averaged
|
||||
across the whole benchmark); the per-film breakdown is:
|
||||
|
||||
The [learned scene-boundary detector](scene-boundary-detector.md) — an XGBoost
|
||||
regressor over histogram-delta and audio features, with a per-film knee threshold —
|
||||
fixes this. Macro per-second presence F1, at the shipped presence config:
|
||||
| film | cast credited | in gallery | coverage |
|
||||
|---|---|---|---|
|
||||
| Lord of War | 64 | 13 | 20.3% |
|
||||
| Scarface | 67 | 15 | 22.4% |
|
||||
| The Many Saints of Newark | 48 | 13 | 27.1% |
|
||||
| Café Society | 62 | 17 | 27.4% |
|
||||
| Lovelace | 42 | 15 | 35.7% |
|
||||
| Valerian and the City of a Thousand Planets | 36 | 13 | 36.1% |
|
||||
| Benny & Joon | 23 | 12 | 52.2% |
|
||||
| Downton Abbey: A New Era | 36 | 22 | 61.1% |
|
||||
| Sound of Metal | 14 | 11 | 78.6% |
|
||||
|
||||
| boundary source for flood-fill | presence F1 |
|
||||
| ------------------------------ | ----------: |
|
||||
| track-extent (flood off) | 62.6% |
|
||||
| flood + grayscale cuts | 64.0% |
|
||||
| **flood + learned detector (LOO)** | **74.9%** |
|
||||
Two training films (Lord of War, Scarface) have the worst coverage in the
|
||||
set, 20-22%. Their training-set F1 numbers below are partly capped by
|
||||
missing references, not purely by model quality. Downton Abbey has 61%
|
||||
coverage, the second-best in the benchmark, yet the worst held-out recall
|
||||
of any film (39.4%, LVFace). Its recall problem is not primarily a coverage
|
||||
problem; it is the extinction-bridging failure documented in the
|
||||
[LVFace deep dive](lvface-deep-dive.md#mechanism-1-extinction-bridging).
|
||||
Reproduce with `scripts/docs/gallery_coverage_per_film.py`.
|
||||
|
||||
|
||||
## Training results, 3 models × 2 gallery modes × 2 expansion settings
|
||||
|
||||
The learned column is **leave-one-out**: each film is scored by a detector trained
|
||||
on the other eight, so no film's presence is ever measured with a detector that saw
|
||||
it. That is the honest generalisation number, +12.3 points over track-extent, and
|
||||
**it improves every one of the nine films**.
|
||||
Ranked by F1. misid = FPI_misid, the count of true wrong-actor
|
||||
identifications (naming someone not in the film's cast at all), distinct
|
||||
from FPI, which also includes in-cast timing slips.
|
||||
|
||||
|
||||
Each combo's row is its best **full-coverage** evaluation: the highest-F1 DE
|
||||
evaluation in which all 4 training films replayed without a timeout (see
|
||||
[Dropped-film scoring](#a-scoring-bug-worth-recording-dropped-film-evaluations)
|
||||
below for why this qualifier is load-bearing and not the same as `argmax F1`
|
||||
over the raw sweep).
|
||||
|
||||
| film | track-extent | flood+grayscale | flood+learned (LOO) |
|
||||
| ---- | -----------: | --------------: | ------------------: |
|
||||
| Benny & Joon | 77.3 | 80.2 | 78.2 |
|
||||
| Café Society | 59.1 | 62.2 | 69.8 |
|
||||
| Downton Abbey | 41.0 | 51.8 | **78.6** |
|
||||
| Lord of War | 74.8 | 77.1 | 77.8 |
|
||||
| Lovelace | 70.3 | 74.0 | 78.2 |
|
||||
| The Many Saints of Newark | 37.5 | 43.9 | 53.4 |
|
||||
| Scarface | 62.6 | **40.9** | **74.9** |
|
||||
| Sound of Metal | 75.0 | 78.1 | 86.8 |
|
||||
| Valerian | 65.6 | 67.7 | 76.2 |
|
||||
| combo | F1 | P | R | TPI | FPI | misid | FN |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| LVFace-B_Glint360K_restricted_exp | 78.3% | 91.0% | 68.9% | 42830 | 3782 | 60 | 19492 |
|
||||
| LVFace-B_Glint360K_restricted_noexp | 76.7% | 91.5% | 66.2% | 41149 | 3400 | 59 | 21173 |
|
||||
| arcface_w600k_mbf_restricted_exp | 76.2% | 90.0% | 66.2% | 64328 | 7480 | 0 | 33234 |
|
||||
| arcface_r18_restricted_exp | 75.5% | 87.6% | 66.5% | 41399 | 5666 | 60 | 20923 |
|
||||
| LVFace-B_Glint360K_full_exp | 75.3% | 89.7% | 65.4% | 47757 | 3407 | 232 | 26966 |
|
||||
| arcface_w600k_mbf_restricted_noexp | 75.0% | 91.1% | 63.9% | 39752 | 3465 | 60 | 22570 |
|
||||
| arcface_r18_restricted_noexp | 73.5% | 91.3% | 61.7% | 38299 | 3220 | 60 | 24023 |
|
||||
| LVFace-B_Glint360K_full_noexp | 72.3% | 88.3% | 61.8% | 40363 | 3503 | 244 | 25850 |
|
||||
| arcface_w600k_mbf_full_exp | 72.0% | 87.7% | 61.4% | 39875 | 3729 | 240 | 26338 |
|
||||
| arcface_w600k_mbf_full_noexp | 71.0% | 93.2% | 57.9% | 41699 | 2472 | 56 | 33024 |
|
||||
| arcface_r18_full_exp | 69.1% | 87.6% | 57.7% | 37342 | 3119 | 242 | 28871 |
|
||||
| arcface_r18_full_noexp | 66.6% | 91.3% | 53.1% | 34314 | 2362 | 107 | 31899 |
|
||||
|
||||
The two headline films — Scarface (grayscale flood *breaks* it, learned flood on a
|
||||
film it never trained on takes it to 74.9%) and Downton Abbey (+37 points) — are
|
||||
the strongest evidence the detector generalises. See the
|
||||
[scene-boundary detector page](scene-boundary-detector.md) for the full story.
|
||||
|
||||
|
||||
We re-ran the ten-knob DE on top of the good boundaries to check whether the
|
||||
shipped config should change. It converged at 76.1% (+0.3 pp over the shipped
|
||||
config on learned boundaries) — inside the noise, not worth re-shipping. The
|
||||
boundaries, not the presence knobs, are where the win is.
|
||||
The two clearest patterns: every model's best-scoring combo uses the
|
||||
restricted gallery, and LVFace leads within both gallery modes. `full_exp`
|
||||
(the shipped combination) is the best-scoring option that uses only
|
||||
features the running application currently supports; restriction is not
|
||||
wired into the application yet (see
|
||||
[Whole vs. cast-restricted gallery](gallery-scope.md)).
|
||||
|
||||
## What the frames look like
|
||||
### A scoring bug worth recording: dropped-film evaluations
|
||||
|
||||
`scripts/optimizer/dump_error_frames.py` pulls representative seconds and draws
|
||||
each face box against X-Ray's scene cast: **green** = true positive, **red** =
|
||||
false positive (a name X-Ray does not credit to this scene — the real error),
|
||||
**orange** = an unknown detection. Cast X-Ray lists as present but for whom no face
|
||||
was detected — the structural false-negatives a face pipeline can never box — are
|
||||
listed as a **blue** panel.
|
||||
The numbers above are corrected ones. The raw `rep4_best_*.json` files, and an
|
||||
earlier version of this table, reported a different `arcface_w600k_mbf_full_noexp`
|
||||
row: **74.2% F1 at TPI 12645**, a third the TPI of every sibling combo. That was
|
||||
not a better config; it was an artifact of how the optimizer aggregates.
|
||||
|
||||
|
||||
`optimize.py` builds each candidate's score from only the films whose replay
|
||||
subprocess returned (`per_film = [m for m in ex.map(_one, films) if m is not
|
||||
None]`), then **averages** F1/precision/recall and **sums** TPI/FPI/misID over
|
||||
just those survivors. When a film's replay times out (the sweep ran near the
|
||||
8-process concurrency ceiling, so this happened intermittently), that film
|
||||
silently drops from both. A candidate whose hardest film timed out is therefore
|
||||
scored on an easier subset, and differential evolution, maximizing that score,
|
||||
will happily converge onto exactly such a candidate. For `mbf_full_noexp` the
|
||||
reported winner was one of 7 evaluations (out of 512) whose TPI had collapsed to
|
||||
a partial-film subset; its median-coverage evaluations sit around 51686 TPI.
|
||||
|
||||
Every named frame in this documentation is regenerated against the current opencv5
|
||||
pipeline by [`scripts/scene_detector/rematch_frames.py`](https://REPOLINK/scripts/scene_detector/rematch_frames.py),
|
||||
which auto-matches each example by film, actor, and class (TP/FP) so the images
|
||||
never drift from the shipped behaviour. Where the current pipeline no longer makes
|
||||
a July-era error — the Zooey Deschanel misID in Many Saints is the clearest case —
|
||||
the frame is dropped rather than staged, because the improvement is real.
|
||||
The fix here was to re-derive each combo's best row from its DE trajectory
|
||||
(`experiments/trajectories/rep4_*.jsonl`), keeping only evaluations within 30% of
|
||||
that combo's median TPI (full 4-film coverage) before taking the best F1. This
|
||||
needs no re-running, the honest best configuration was already in the sweep,
|
||||
just not the one `argmax F1` selected. Three combos moved: `mbf_full_noexp`
|
||||
74.2% → **71.0%**, `LVFace_full_noexp` 72.4% → **72.3%** (and its misID, 0 → 244,
|
||||
was itself a dropped-film artifact), `mbf_restricted_exp` 76.5% → **76.2%**. The
|
||||
shipped LVFace `full_exp` winner was unaffected, its reported evaluation already
|
||||
had full coverage (TPI 47757 ≈ median). `experiment_charts.py` applies the same
|
||||
`clean_best` filter, so every figure on this page matches the corrected table.
|
||||
The underlying `optimize.py` aggregation is also being fixed so a dropped-film
|
||||
evaluation can never be selected as a winner again.
|
||||
|
||||
## The structural recall ceiling
|
||||
### Per-film training breakdown
|
||||
|
||||
Precision against X-Ray is near-perfect on identified faces; recall is capped by
|
||||
two things the pipeline cannot fix:
|
||||
The 75.3% LVFace training figure is a macro average across 4 films, not a
|
||||
uniform result:
|
||||
|
||||
1. **X-Ray credits people whose faces never appear on screen** in a scene — voice,
|
||||
back-of-head, or simply off-camera cast. No face pipeline can box a face that is
|
||||
not there. These are the blue-panel names.
|
||||
2. **Gallery coverage.** A large fraction of X-Ray cast has no reference image in
|
||||
the gallery, so those actors can never be matched regardless of detection. This
|
||||
is the dominant remaining recall limiter and is addressable by fetching more
|
||||
reference photos, not by tuning.
|
||||
| film | LVFace F1 | mbf F1 | r18 F1 | best model |
|
||||
|---|---|---|---|---|
|
||||
| Café Society | 68.1% | 62.2% | 60.1% | LVFace |
|
||||
| Lord of War | 75.6% | 77.2% | 75.6% | mbf |
|
||||
| Scarface | 71.5% | 68.6% | 64.1% | LVFace |
|
||||
| Sound of Metal | 78.8% | 76.5% | 71.6% | LVFace |
|
||||
|
||||
Both are documented in [how we score against X-Ray](methodology.md).
|
||||
LVFace does not win every training film. mbf scores higher on Lord of War
|
||||
(77.2% vs 75.6%). LVFace's own training-film range is 68.1% to 78.8%, a
|
||||
10.7pp spread, smaller than the 37pp spread seen on held-out films but real.
|
||||
Reproduce with `scripts/docs/run_holdout_all_models.py --films training`.
|
||||
|
||||
## In the pipeline
|
||||
## Held-out validation, all 3 models
|
||||
|
||||
The learned detector runs live inside `scene_analyze` as a post-EOF step (the
|
||||
per-film knee needs every peak, so it can only run once the whole film is seen).
|
||||
XGBoost inference is built into the binary via CMake (`SAE_SCENE_XGB`); the audio
|
||||
log-PSD uses FFTW on the existing FFmpeg decode. The shipped model is trained on
|
||||
the **C++-extracted** features so training and inference share one implementation.
|
||||
Verified end to end through `scene_analyze` on a movie file and through the Jellyfin
|
||||
work-queue worker.
|
||||
The training matrix above is training-set fit. Each model's own tuned
|
||||
`full_exp` config was replayed against the 5 held-out films, scored the
|
||||
same way:
|
||||
|
||||
| film | LVFace F1 | mbf F1 | r18 F1 |
|
||||
|---|---|---|---|
|
||||
| Benny & Joon | 83.0% | 78.5% | 77.1% |
|
||||
| Lovelace | 77.5% | 73.7% | 72.2% |
|
||||
| Valerian and the City of a Thousand Planets | 74.1% | 70.2% | 71.0% |
|
||||
| Downton Abbey: A New Era | 56.2% | 55.0% | 53.0% |
|
||||
| The Many Saints of Newark | 46.3% | 44.5% | 42.1% |
|
||||
| **macro average** | **67.4%** | **64.4%** | **63.1%** |
|
||||
|
||||
LVFace scores highest on every one of the 5 held-out films; the ranking
|
||||
never flips. Total misIDs across the 5 films: LVFace 1032, mbf 2197, r18
|
||||
1224. LVFace has less than half mbf's misID count while also scoring
|
||||
higher on every film. This directly confirms the model choice out of
|
||||
sample; it is not inferred from the training numbers alone. See the
|
||||
[LVFace deep dive](lvface-deep-dive.md) for frame-level detail on where and
|
||||
why LVFace still fails on the two worst films. Reproduce with
|
||||
`scripts/docs/run_holdout_all_models.py`.
|
||||
|
||||
## Two effects in isolation: gallery scope and pose expansion
|
||||
|
||||
Averaging across the 3 compared models (r50 excluded) isolates each variable
|
||||
from model choice.
|
||||
|
||||
**Gallery scope**, averaged over both expansion settings and all 3 models
|
||||
(6 evaluations per row):
|
||||
|
||||
| scope | F1 | P | R | total misID |
|
||||
|---|---|---|---|---|
|
||||
| full | 71.1% | 89.6% | 59.6% | 1121 |
|
||||
| restricted | 75.9% | 90.4% | 65.6% | 299 |
|
||||
|
||||
Restriction improves every metric at once. This is not a precision/recall
|
||||
trade: +4.8pp F1, +6.0pp recall, and roughly a quarter the misIDs. Fewer
|
||||
candidates in the matcher's search space means fewer opportunities for a
|
||||
lookalike false match, and the recall gain shows this does not cost real
|
||||
detections. Restriction is currently an offline optimizer technique, not a
|
||||
runtime feature of the application; see
|
||||
[Whole vs. cast-restricted gallery](gallery-scope.md) for what building it
|
||||
into the application would require.
|
||||
|
||||
**Pose expansion** (promoting a confidently identified track's novel-pose
|
||||
views into a per-film gallery annex,
|
||||
[`src/gallery/track_gallery.hpp`](https://REPOLINK/src/gallery/track_gallery.hpp)):
|
||||
|
||||
| scope | expansion | F1 | R | misID |
|
||||
|---|---|---|---|---|
|
||||
| full | off | 70.0% | 57.6% | 407 |
|
||||
| full | on | 72.1% | 61.5% | 714 |
|
||||
| restricted | off | 75.1% | 63.9% | 179 |
|
||||
| restricted | on | 76.7% | 67.2% | 120 |
|
||||
|
||||
In restricted mode, expansion is a clean win: +1.6pp F1, +3.3pp recall,
|
||||
misID drops. The annex only competes against the film's own roughly 15-actor
|
||||
cast, so a new pose of a known actor is unlikely to be confused with someone
|
||||
else. In full mode, expansion buys +2.1pp F1 and +3.9pp recall but at a real
|
||||
cost: misID rises from 407 to 714 as the same new-pose view now competes
|
||||
against the full 2418-actor gallery, where a confidently learned pose is more
|
||||
likely to match the wrong person. On the full gallery it is a recall-vs-misID
|
||||
trade, not a free gain. This training-set effect
|
||||
did not reproduce on held-out data; see
|
||||
[Does pose expansion help?](pose-expansion.md) for the full held-out test
|
||||
and the two methodology bugs caught while checking it.
|
||||
|
||||
## Calibration curves
|
||||
|
||||
Each gallery carries a fitted Platt sigmoid `P(match | sim) = σ(a·sim + b)`,
|
||||
stored directly in the gallery HDF5
|
||||
([`src/gallery/gallery_calibration.hpp`](https://REPOLINK/src/gallery/gallery_calibration.hpp)).
|
||||
This measures discriminative power independent of whatever
|
||||
`prob_threshold` a given run used:
|
||||
|
||||
|
||||
|
||||
LVFace has the steepest curve (`a=17.7` vs 15.3-16.2 for the ArcFace
|
||||
variants) and the lowest P=0.5 decision boundary (similarity 0.23 vs
|
||||
0.27-0.31), separating same-actor from different-actor pairs more
|
||||
confidently at a lower similarity than any ArcFace variant tested,
|
||||
including r50. Generated by
|
||||
[`scripts/docs/calibration_chart.py`](https://REPOLINK/scripts/docs/calibration_chart.py).
|
||||
|
||||
## Extinction and anneal window search
|
||||
|
||||
Every one of the 512 DE evaluations for the winning LVFace `full_exp`
|
||||
combo, plotted over the `prob_threshold` × `extinction_sec` plane:
|
||||
|
||||
|
||||
|
||||
Nearly everything scoring well sits at `extinction_sec` above 50, across a
|
||||
wide range of thresholds. Short extinction windows are uniformly weaker:
|
||||
under a strict threshold, there is no good configuration in that region of
|
||||
the search space. The optimizer converged with `anneal_sec=59.2,
|
||||
extinction_sec=59.2`, about 99% of the widened 60s bound, which raises an
|
||||
open question not resolved in this round: does performance keep improving
|
||||
past 60s, or does it plateau there. Not chased further this pass.
|
||||
|
||||
## Caveats
|
||||
|
||||
- r50's 4 combos used the older, narrower search bounds (1-30/1-15 instead
|
||||
of 1-60/1-60) and are further confounded by its thinner gallery. Excluded
|
||||
from all comparisons above except calibration.
|
||||
- The shipped defaults use `full_exp` (75.3% training F1), not the
|
||||
higher-scoring `restricted_exp` (78.3%), because cast restriction is not
|
||||
a runtime feature of the application yet.
|
||||
- `expand_gallery` is mode-dependent, not a free win. Averaged across models
|
||||
on the full gallery it trades misIDs for recall (see the pose-expansion
|
||||
table). For LVFace specifically, though, `full_exp` beats `full_noexp` on
|
||||
every axis at once (F1 75.3 vs 72.3, precision 89.7 vs 88.3, recall 65.4 vs
|
||||
61.8, misID 232 vs 244), so the shipped `full_exp` is a clean choice for
|
||||
this model, not an F1-vs-safety trade. (An earlier version of this page
|
||||
reported `full_noexp` at 72.4% with zero misIDs and higher precision, which
|
||||
made it look like the safer option; that was the dropped-film artifact
|
||||
described above, not a real property of the config.)
|
||||
- Switching the default model is an operational change: any gallery built
|
||||
from a different model's embeddings must be rebuilt before the new
|
||||
default takes effect.
|
||||
|
||||
## Reproduce
|
||||
|
||||
```bash
|
||||
scene_analyze --movie <file> --gallery <gallery.h5> \
|
||||
--scene-xgb-model models/scene_boundary_xgb.json
|
||||
# 4-film training matrix, all 4 models × 2 gallery modes × 2 expansion settings
|
||||
bash experiments/run_rep4_subprocess.sh
|
||||
|
||||
# single combo
|
||||
SAE_EXPAND=1 REPLAY_WORKERS=4 DE_WORKERS=2 python3 scripts/optimizer/optimize.py \
|
||||
--manifest experiments/manifests/rep4_LVFace-B_Glint360K_full.json \
|
||||
--gallery experiments/galleries/gallery_LVFace-B_Glint360K.h5 \
|
||||
--params prob_threshold:0.5:0.999 anneal_sec:1:60 extinction_sec:1:60 \
|
||||
--popsize 10 --maxiter 15 --trajectory traj.jsonl --out best.json
|
||||
|
||||
# held-out validation, all 3 models, 5 films
|
||||
python3 scripts/docs/run_holdout_all_models.py --out docs_data/holdout_all_models.json
|
||||
|
||||
# per-film training breakdown, all 3 models, 4 films
|
||||
python3 scripts/docs/run_holdout_all_models.py --films training --out docs_data/training_per_film.json
|
||||
|
||||
# gallery coverage per film
|
||||
python3 scripts/docs/gallery_coverage_per_film.py --out docs_data/gallery_coverage_per_film.json
|
||||
|
||||
# regenerate this page's charts from experiments/ artifacts
|
||||
python3 scripts/docs/experiment_charts.py --out-dir docs/assets/images
|
||||
|
||||
# one frame per distinct out-of-cast name across all 9 films (used in the deep dive)
|
||||
python3 scripts/docs/first_fpi_frames.py
|
||||
```
|
||||
|
||||
## Reproducing the benchmarks
|
||||
|
||||
Gallery `.h5` files, embedding dumps, the X-Ray corpus, and DE trajectories are not
|
||||
committed. They are pushed to the Gitea package registry and pulled on demand:
|
||||
|
||||
```bash
|
||||
scripts/artifacts/pull_artifacts.sh galleries
|
||||
scripts/artifacts/pull_artifacts.sh experiment-data
|
||||
|
||||
# per-second audio features, C++ feature matrices, train + downstream A/B
|
||||
.venv-rocm/bin/python scripts/scene_detector/extract_audio_features.py \
|
||||
--manifest experiments/manifests/films_LVFace_opencv5.json
|
||||
.venv-rocm/bin/python scripts/scene_detector/train_xgb_cpp.py --train-all
|
||||
scripts/scene_detector/downstream_presence.py
|
||||
```
|
||||
See also the session log
|
||||
[`experiments/SESSION_STATE.md`](https://REPOLINK/experiments/SESSION_STATE.md).
|
||||
|
||||
+4
-7
@@ -255,11 +255,8 @@ Context crops opt-in behind `--dump-unidentified-crops`.
|
||||
|
||||
## AR-026, AR-027 — GEMM and scale
|
||||
|
||||
**Depends on:** nothing to start. The annex CPU loop has moved into the GEMM
|
||||
path: the annex is a contiguous matrix, promotions are appended to the engine's
|
||||
resident gallery, and the CPU backend now requires OpenBLAS. What is left of
|
||||
AR-026 is call site 3, the deferred pass — so the rest of AR-026 lands *with*
|
||||
AR-020 rather than before it.
|
||||
**Depends on:** nothing to start. The annex CPU loop
|
||||
(`identity_matcher_node.hpp:159-162`) moves into the GEMM path.
|
||||
|
||||
---
|
||||
|
||||
@@ -366,8 +363,8 @@ one identity out of the gallery would fix that.
|
||||
thing measured is the thing that ships.
|
||||
|
||||
`scripts/validation/test_audio_offset.py` over
|
||||
`tests/fixtures/audio/superhero_offset_200s.flac`: 200 s of public-domain film audio
|
||||
(the same SuperHero clips the replay fixtures use), long enough for a 120 s
|
||||
`tests/fixtures/audio/bali_offset_200s.flac`: 200 s of public-domain film audio
|
||||
(the same Road to Bali clips the replay fixtures use), long enough for a 120 s
|
||||
window to slide past the ±600-frame search cap. The slide itself is numpy here
|
||||
on purpose — matching belongs to the consumer, so writing it out keeps this a
|
||||
test of the signature rather than of somebody's matcher.
|
||||
|
||||
@@ -1,5 +1,3 @@
|
||||
> **Archived (July 2026).** This report covers the pre-opencv5 framework and the 4-model ArcFace/LVFace bake-off. It is superseded by the current [experiment log](model-bakeoff.md) for the opencv5 build. Kept for provenance; the numbers here are historical.
|
||||
|
||||
# Pose expansion: does promoting new poses mid-film help?
|
||||
|
||||
`expand_gallery`
|
||||
@@ -14,7 +12,7 @@ in the same film, without touching the baked gallery.
|
||||
|
||||
Averaged across the 3 compared models (r50 excluded), on the 4 films used
|
||||
for optimization. These are the corrected, full-coverage figures, see the
|
||||
[dropped-film note](model-bakeoff-2026-07.md#a-scoring-bug-worth-recording-dropped-film-evaluations)
|
||||
[dropped-film note](model-bakeoff.md#a-scoring-bug-worth-recording-dropped-film-evaluations)
|
||||
in the experiment log for why an earlier version of this table overstated the
|
||||
full-mode misID jump (209 → 864) that was itself partly a truncation artifact:
|
||||
|
||||
@@ -28,7 +26,7 @@ full-mode misID jump (209 → 864) that was itself partly a truncation artifact:
|
||||
In restricted mode, expansion looks like a clean win: +1.6pp F1, +3.3pp
|
||||
recall, lower misID. In full mode it looks like a recall-for-misID trade:
|
||||
+2.1pp F1, +3.9pp recall, but misID rises from 407 to 714. See
|
||||
[the full experiment log](model-bakeoff-2026-07.md) for the per-model breakdown.
|
||||
[the full experiment log](model-bakeoff.md) for the per-model breakdown.
|
||||
This asymmetry motivated the question below: does turning expansion on
|
||||
change what gets recognized frame by frame, or is the aggregate F1 shift
|
||||
coming from something else.
|
||||
@@ -107,6 +105,6 @@ contribution, such as tagging which reference embedding won each match;
|
||||
neither was in scope for this pass.
|
||||
|
||||
Do not treat the training-set exp/noexp numbers in
|
||||
[the full experiment log](model-bakeoff-2026-07.md) as proof that expansion changes
|
||||
[the full experiment log](model-bakeoff.md) as proof that expansion changes
|
||||
real-world behavior in either direction. On the evidence gathered so far,
|
||||
it does not move the needle enough to see.
|
||||
@@ -0,0 +1,308 @@
|
||||
# Quality knee: what does a blurred or small face cost, and can a measure predict it?
|
||||
|
||||
VR-012. Companion to the minimum-face-size studies VR-005 and VR-013 (see the
|
||||
[requirement register](requirements.md)), which located the size floor at 40 px;
|
||||
this asks the same question for **sharpness**, and asks whether any cheap
|
||||
measure taken on the aligned crop can be acted on at inference.
|
||||
|
||||
Run by
|
||||
[`scripts/validation/quality_knee.py`](https://REPOLINK/scripts/validation/quality_knee.py)
|
||||
through the `sae_embed` bindings — detection, the ArcFace warp, the embedder,
|
||||
the five candidate measures and the Platt calibration are all the shipped C++.
|
||||
|
||||
## Protocol
|
||||
|
||||
1670 gallery actors with 3 or more mugshots (of 2456 total), one image held out
|
||||
per actor as a probe, the remaining 10326 embeddings staying in the gallery at
|
||||
native resolution. Only the probe degrades — reference mugshots are clean and
|
||||
the face coming out of the video is not.
|
||||
|
||||
Each probe passes through a **joint grid**: downscale to *S*×*S* and back to
|
||||
112 (the sampling loss), then blur at level *L* in canonical pixels. Three blur
|
||||
families, 36 cells each, 60120 probe-cell records per family:
|
||||
|
||||
| family | models | parameter |
|
||||
|---|---|---|
|
||||
| Gaussian | soft focus, a generic stand-in | sigma 0 … 3 |
|
||||
| **Disc** | **real optical defocus** — the circle of confusion | radius 0 … 6 |
|
||||
| Motion | camera pan or moving subject | length 0 … 21 px |
|
||||
|
||||
The three are not interchangeable, and sweeping only the first was the original
|
||||
design error — one that would have produced a wrong answer, not merely an
|
||||
incomplete one (Result 3). A defocused lens spreads a point into a **uniform
|
||||
disc**, whose transfer function is a jinc — `2·J1(x)/x` — that crosses zero and
|
||||
goes negative, annihilating whole frequency bands and returning the ones beyond
|
||||
each zero phase-reversed. A Gaussian MTF is strictly positive and monotone and
|
||||
does neither. More practically: defocus and motion are how a face ends up
|
||||
**large and useless**, while Gaussian blur as swept here mostly co-occurs with
|
||||
small faces. That difference decides whether sharpness carries anything the size
|
||||
filter does not.
|
||||
|
||||
Families are compared at matched **per-axis PSF standard deviation** (σ for a
|
||||
Gaussian, R/2 for a disc, L/√12 for a linear smear), never at equal raw
|
||||
parameter, which would compare different amounts of damage.
|
||||
|
||||
Identification is the pipeline's own decision: per-actor best-of-N cosine →
|
||||
Platt sigmoid → accept above `prob_threshold` 0.754. Never a raw cosine
|
||||
(AR-024).
|
||||
|
||||
## Result 1 — sharpness is not a sufficient statistic
|
||||
|
||||
Sorting the 36 Gaussian cells by `hf_energy_ratio`, the six sigma-3 cells land
|
||||
at effectively identical measured sharpness:
|
||||
|
||||
| size | sigma | hf_energy_ratio | TPI |
|
||||
|---|---|---|---|
|
||||
| 16 | 3 | 0.0003 | **15.3%** |
|
||||
| 24 | 3 | 0.0003 | 63.2% |
|
||||
| 32 | 3 | 0.0003 | 79.4% |
|
||||
| 48 | 3 | 0.0003 | 86.6% |
|
||||
| 64 | 3 | 0.0004 | 88.4% |
|
||||
| 112 | 3 | 0.0005 | **91.0%** |
|
||||
|
||||
Same measured sharpness, a **76-point spread in identification**. It inverts
|
||||
too: 16 px unblurred measures 0.0033 and scores 23.5%, while 48 px at sigma 2
|
||||
measures *lower* at 0.0021 and scores 96.6%.
|
||||
|
||||
A canonical-frame sharpness scalar cannot separate *attenuated* high
|
||||
frequencies from *destroyed* spatial sampling. Blur suppresses the high band
|
||||
while preserving mid-frequency facial geometry exactly; downsampling to 16 px
|
||||
destroys that geometry outright. Both look alike to any measure keyed on
|
||||
high-frequency energy.
|
||||
|
||||
This is the measured basis for AR-028's rule that the axes are **kept separate
|
||||
and not collapsed into one scalar**, and it settles the double-counting
|
||||
question: size and sharpness are not redundant, and neither substitutes for the
|
||||
other.
|
||||
|
||||
## Result 2 — blur is a cliff, and it breaks confidence, not identity
|
||||
|
||||
TPI % by size (rows) against Gaussian sigma (columns):
|
||||
|
||||
| size | 0 | 0.5 | 1 | 1.5 | 2 | 3 |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 16 | 23.5 | 24.0 | 25.0 | 24.6 | 23.9 | 15.3 |
|
||||
| 24 | 85.7 | 85.1 | 85.6 | 85.9 | 82.6 | 63.2 |
|
||||
| 32 | 95.9 | 95.9 | 96.0 | 95.5 | 93.7 | 79.4 |
|
||||
| 48 | 98.7 | 98.7 | 98.4 | 98.1 | 96.6 | 86.6 |
|
||||
| 64 | 98.6 | 98.6 | 98.8 | 98.4 | 97.5 | 88.4 |
|
||||
| 112 | 98.9 | 98.9 | 98.8 | 98.6 | 98.0 | 91.0 |
|
||||
|
||||
Three regimes: **sigma ≤ 1.5 is free** (every cell moves under 1.5 points, sign
|
||||
flipping at random — at 16 px it slightly *improves*, smoothing upscale
|
||||
artifacts); sigma 2 costs 1–3 points; the 2→3 step costs 7–19. A smooth
|
||||
discount curve is therefore the wrong shape — the response is flat, then falls
|
||||
off a cliff.
|
||||
|
||||
**The cost peaks at the size knee, not at full resolution.** Sigma 3 costs
|
||||
−22.5 points at 24 px but only −7.9 at 112 px and −8.3 at 16 px. Blur has no
|
||||
intrinsic cost; it costs in proportion to how close the observation already sits
|
||||
to the decision boundary. At 112 px there is margin to spare, at 16 px the probe
|
||||
is already below threshold, and at 24 px it sits exactly on the knee.
|
||||
|
||||
**What blur destroys is confidence, not ranking.** Rank-1 barely moves: 99.3% →
|
||||
99.2% at 112 px across the whole sigma range. The extreme case is 16 px at sigma
|
||||
3, where rank-1 is **80.2%** while TPI is **15.3%** — 65 points of probes have
|
||||
the correct actor ranked first and are rejected anyway for falling under the
|
||||
probability threshold.
|
||||
|
||||
That is why **FPI never left 0.1% in any of the 108 cells across all three
|
||||
families**. Degradation produces TBI, never a wrong name. The calibration
|
||||
degrades gracefully, which is what SR-002 needs.
|
||||
|
||||
## Result 3 — the blur *family* matters more than the blur *amount*
|
||||
|
||||
Comparing families by their raw parameter is meaningless — sigma, radius and
|
||||
length are different units. They are matched here by the **per-axis standard
|
||||
deviation of the PSF**, which puts them on one scale:
|
||||
|
||||
| family | per-axis σ | level giving σ = 3 px |
|
||||
|---|---|---|
|
||||
| Gaussian σ | σ | 3 |
|
||||
| Disc radius R | R/2 | 6 |
|
||||
| Motion length L | L/√12 | 10.4 |
|
||||
|
||||
For reference the ArcFace template places the eyes 35.2 canonical px apart, so
|
||||
σ = 3 px is 9% of the inter-ocular distance.
|
||||
|
||||
TPI at matched severity, interpolated within each family:
|
||||
|
||||
| size | σ=3 Gaussian | σ=3 Motion | σ=3 **Defocus** | defocus penalty |
|
||||
|---|---|---|---|---|
|
||||
| 16 | 15.3 | 15.1 | 11.0 | +4.3 |
|
||||
| 24 | 63.2 | 61.1 | 41.4 | +21.9 |
|
||||
| 32 | 79.4 | 76.1 | 50.4 | +29.0 |
|
||||
| 48 | 86.6 | 80.9 | 52.6 | +34.0 |
|
||||
| 64 | 88.4 | 81.7 | 51.0 | +37.4 |
|
||||
| 112 | 91.0 | 82.1 | **46.9** | **+44.0** |
|
||||
|
||||
**Optical defocus is up to 44 points more destructive than a Gaussian of
|
||||
identical spread**, and the ordering is defocus ≫ motion > Gaussian throughout.
|
||||
At σ=1 the three families are indistinguishable, and at σ=2 they differ by under
|
||||
5 points; the divergence appears only when both the blur is severe *and* the face
|
||||
is large.
|
||||
|
||||
That pattern is physically consistent. At 16 px the resampling has already
|
||||
removed the high frequencies, so the PSF's shape has nothing left to act on and
|
||||
all three agree. At 112 px the full spectrum is present and shape decides: a
|
||||
Gaussian MTF rolls off gently and always leaves *some* energy at every
|
||||
frequency, so the embedder receives a merely attenuated signal, while a disc MTF
|
||||
is a jinc that **hits exact zeros** — whole frequency bands annihilated rather
|
||||
than attenuated, with the bands beyond each zero returning phase-reversed.
|
||||
Motion sits between them because it ruins one axis and leaves the perpendicular
|
||||
one untouched.
|
||||
|
||||
**The methodological consequence is the important one.** This study originally
|
||||
swept Gaussian blur alone and concluded blur was a minor effect. On the family
|
||||
that actually occurs in film, the same nominal severity costs **53% error
|
||||
instead of 9%** at full resolution. A threshold set from the Gaussian arm would
|
||||
have been wrong by a factor of five in error rate, and the axis would probably
|
||||
have been dropped as not worth its cost.
|
||||
|
||||
**Defocus is also the case a size gate cannot catch.** Every one of those 112 px
|
||||
faces is large and confidently detected, and sails through AR-002 untouched.
|
||||
That, not the Gaussian result, is what justifies a sharpness axis existing at
|
||||
all.
|
||||
|
||||
## Result 4 — variance of Laplacian is anti-predictive at fixed degradation
|
||||
|
||||
Pooled across all cells, every candidate scores AUC 0.76–0.80 for predicting
|
||||
correct identification, with textbook `var_laplacian` top. That number is close
|
||||
to worthless: it rewards a measure for detecting *how degraded the crop is*,
|
||||
which all five do. The question a per-observation discount needs is whether, at
|
||||
a **fixed** degradation, the measure predicts which faces fail:
|
||||
|
||||
| measure | Gaussian | Defocus | Motion |
|
||||
|---|---|---|---|
|
||||
| `hf_energy_ratio` | **0.530** | **0.521** | **0.557** |
|
||||
| `norm_var_laplacian` | 0.520 | 0.507 | 0.539 |
|
||||
| `dir_min_tenengrad` | 0.524 | 0.512 | 0.506 |
|
||||
| `tenengrad` | 0.433 | 0.437 | 0.457 |
|
||||
| `var_laplacian` | 0.423 | 0.422 | 0.473 |
|
||||
|
||||
Best is 0.557 — barely above chance, and `hf_energy_ratio` wins on all three
|
||||
families. `var_laplacian` is anti-predictive on all three too, so that finding
|
||||
does not depend on the blur model.
|
||||
|
||||
**The two metrics measure different jobs, and the candidates split along that
|
||||
line.** On the motion arm `dir_min_tenengrad` has the best *pooled* AUC by a
|
||||
wide margin — **0.854** against 0.792 for the next — exactly as its synthetic
|
||||
directional-blur ladder predicted, yet its within-cell AUC there is 0.506. It is
|
||||
an excellent detector of *how badly smeared a crop is* and no guide at all to
|
||||
*which face will be recognised*. Pooled AUC is the right metric for a
|
||||
gross-degradation flag; within-cell AUC is the right one for a per-observation
|
||||
discount; a measure can be strong at one and useless at the other.
|
||||
|
||||
Deciles within the 16 px Gaussian cell, where 1277 failures give the test real
|
||||
power:
|
||||
|
||||
| `var_laplacian` decile | TPI |
|
||||
|---|---|
|
||||
| 0.00071–0.00192 (blurriest) | **37.1%** |
|
||||
| 0.00242–0.00278 | 22.8% |
|
||||
| 0.00397–0.00447 | 25.7% |
|
||||
| 0.00625–0.01445 (sharpest) | **14.4%** |
|
||||
|
||||
The faces the measure calls sharpest are **2.6x less identifiable** than those
|
||||
it calls blurriest, monotone across ten bins of 167. Within a cell every crop
|
||||
received identical degradation, so the residual variance is *native contrast*,
|
||||
not native detail — and hard shadows, high-contrast lighting, sharpening halos
|
||||
and JPEG ringing all raise Laplacian variance while making a face harder to
|
||||
match. The measure reads photographic style and encoding artifacts and calls
|
||||
them sharpness.
|
||||
|
||||
`hf_energy_ratio` is the only candidate with a correctly-signed within-cell
|
||||
trend (16.2% → 35.3% across the same deciles), being a pure ratio in which the
|
||||
contrast factor cancels.
|
||||
|
||||
**Consequence:** a per-face quality *discount* keyed on variance of Laplacian —
|
||||
the most widely used blur metric in production vision pipelines — would
|
||||
systematically down-weight the *more* identifiable faces. It is worse than no
|
||||
discount.
|
||||
|
||||
## Result 5 — as a compute gate, sharpness loses to the size filter
|
||||
|
||||
Skipping the embed for crops below a threshold, measured as compute saved
|
||||
against true identifications lost:
|
||||
|
||||
| gate | skipped | true IDs lost | of skipped, doomed anyway |
|
||||
|---|---|---|---|
|
||||
| `hf_energy_ratio` < 0.00023 | 10.0% | 7.6% | 37.9% |
|
||||
| `hf_energy_ratio` < 0.00051 | 20.0% | 15.1% | 38.7% |
|
||||
| **source size < 24 px** | **16.7%** | **4.7%** | **77.3%** |
|
||||
|
||||
At a comparable skip rate the size filter loses **4.7% against sharpness's
|
||||
15.1%** — three times less damage — and it is free, being a bbox dimension
|
||||
available before alignment or embedding, where sharpness needs the warped crop
|
||||
plus a colour convert, three convolutions and a 64×64 DFT.
|
||||
|
||||
Restricting to large faces (≥64 px) on the **defocus** arm, where the size
|
||||
filter is blind, improves the gate's precision 3.5x (37% of skipped crops doomed
|
||||
versus 10.7% on the Gaussian arm) but not its trade: skip 10%, lose 7.0%.
|
||||
|
||||
A hard ceiling explains why. **At 112 px with defocus radius 6 — visually
|
||||
destroyed — 46.9% of faces still identify correctly and rank-1 is still 94.8%.**
|
||||
Blur does not determine the outcome, so any gate keyed on apparent blur is
|
||||
predicting a coin flip. The size filter wins not because size is better
|
||||
measured, but because *smallness destroys identity more completely than blur
|
||||
does*: 16 px faces succeed only 23.5% of the time, so discarding them is cheap.
|
||||
|
||||
## What this means for the requirements
|
||||
|
||||
**Do not gate on sharpness; discount on it.** Heavily defocused faces remain
|
||||
~47% identifiable, so a gate destroys recoverable evidence. This is the first
|
||||
hard evidence that AR-028's "**discounts the observation, never deletes the
|
||||
detection**" is right on the merits rather than merely cautious. Since ranking
|
||||
survives where confidence does not, the per-track accumulation (AR-025) should
|
||||
recover much of what a single-frame threshold rejects — which is also the
|
||||
argument for the discount living in `EvidenceDiscounter` rather than in a filter.
|
||||
|
||||
**`var_laplacian` and `tenengrad` are disqualified as discounts** by Result 4,
|
||||
on all three blur families. They remain usable as coarse *gross-degradation*
|
||||
detectors, the role in which their pooled AUC is real — the same role the size
|
||||
filter plays — but they must never weight a per-observation belief.
|
||||
|
||||
**`hf_energy_ratio` is the only surviving discount candidate**, best on all
|
||||
three families, and its within-cell signal (0.52–0.56) is weak enough that
|
||||
shipping a discount on it needs justification beyond this study.
|
||||
|
||||
**`dir_min_tenengrad` earns a different job.** Its pooled 0.854 on the motion arm
|
||||
makes it the best available detector of gross directional smear — useful as a
|
||||
per-frame "this shot is unusable" flag, which is a decision about a *frame*, not
|
||||
a weighting of an *observation*. If AR-029 ships two measures for two roles, this
|
||||
is the second one, and it must not be confused with the first.
|
||||
|
||||
**Model the blur family, not just its amount.** Result 3 makes the choice of
|
||||
degradation model a first-order design decision rather than a detail: the same
|
||||
matched severity costs 9% or 53% error depending on the PSF. Any future study
|
||||
that sweeps blur must state which family it used and why.
|
||||
|
||||
**Any discount curve must be flat then steep**, not linear or sigmoid over the
|
||||
measure. Blur costs nothing until it costs a great deal.
|
||||
|
||||
## Limitations
|
||||
|
||||
- **Cooperative population.** Gallery mugshots are frontal and well-lit;
|
||||
within-cell failures are likely dominated by cross-view mismatch, which no
|
||||
sharpness measure can predict. Read the ~chance within-cell AUCs as "sharpness
|
||||
does not predict the dominant failure mode *here*", not as "sharpness is
|
||||
meaningless".
|
||||
- **Uniform grid, not a natural distribution.** Sizes and blur levels are
|
||||
sampled evenly, so "skip 16.7%" is exactly the 16 px row. The gate comparisons
|
||||
are like-for-like on identical records, but the absolute savings are not what
|
||||
a film would show.
|
||||
- **TensorRT fp16.** A different realisation of the embedder from the fp32 ONNX
|
||||
reference — VR-005 measured ~0.85 cosine agreement with separation intact.
|
||||
Gallery and probes share one session so the study is internally consistent,
|
||||
but the absolute knee belongs to the fp16 space.
|
||||
- **Blur is applied in the canonical frame**, after resampling, so its width is
|
||||
independent of the cell's size. Real optics blur before sampling.
|
||||
- **The top motion rung is an anchor, not an operating point.** Length 21 is a
|
||||
per-axis σ of 6.1 — 17% of the inter-ocular distance, a streak rather than a
|
||||
face — and it is swept to bound the curve, not because a frame like that is
|
||||
worth reasoning about. Its 3.4% TPI at 112 px should not be quoted as a
|
||||
headline. The same caution applies less severely to defocus radius 6 (σ = 3).
|
||||
- **Per-axis σ equates spread, not perceptual damage.** It is the fairest single
|
||||
scalar for comparing PSFs, but Result 3 is precisely the finding that equal
|
||||
spread does *not* mean equal harm, so the matched-severity tables compare
|
||||
like-for-like inputs, not like-for-like severity as a face would experience it.
|
||||
+32
-58
@@ -29,47 +29,47 @@ Status: `Done` · `In Progress` · `Planned` · `TBD` · `Withdrawn`
|
||||
| ID | Requirement | Traces to | Priority | Status |
|
||||
|---|---|---|---|---|
|
||||
| AR-001 | Detect faces in sampled frames; emit bbox, confidence, 5-point landmarks in original pixel space | SR-002 | High | Done |
|
||||
| AR-002 | Minimum face size **40×40 px** (VR-013 measured end to end; VR-005's 32 px is an embedder-only upper bound), expressed in **original** resolution (decoupled from `dense_scale`) | SR-002 | High | **Done** — `FaceDetectorFunc::drop_undersized()`. The threshold is divided by `bbox_upscale` rather than every box multiplied, which keeps the comparison on the detector's own numbers and means turning `dense_scale` on cannot silently raise the minimum face the pipeline accepts. Verified at the threshold and at `dense_scale` 0.5 (UT-002), and end to end on the fixture (IT-001) — the superhero dump's smallest side is *exactly* its recorded 32 px, so the filter is binding there rather than vacuously satisfied |
|
||||
| AR-002 | Minimum face size **40×40 px** (VR-013 measured end to end; VR-005's 32 px is an embedder-only upper bound), expressed in **original** resolution (decoupled from `dense_scale`) | SR-002 | High | Planned |
|
||||
| AR-003 | No fixed per-frame face cap — crowd scenes must not lose background cast | SR-002 | Medium | **Done** — `max_faces` defaults to 0 (no cap); the matcher batches through its GEMM buffer instead of throwing |
|
||||
| AR-004 | Backpressure: unbounded faces/frame absorbed by slowing, never by dropping or throwing | SR-002 | High | **Mostly** — node outputs *park* on a full channel: the value is held, the worker released, and a channel space-callback resumes the node. Replaces `push_blocking`, which parked a scheduler worker inside the push and, with one thread per node, stopped that node draining its own input. Verified: 385/385 frames, 0 drops. Holes closed since, in the order they surfaced: **(a)** `FanoutNode` dropped on overflow rather than waiting, so the AR-010 scene join shed frames exactly when the dense branch fell behind — measured at **9 of 2192 items delivered** to the slower of two branches, now lossless with the fast branch throttled to within its buffering; **(b)** the residual hang, recorded as ~1 run in 20 at a 300 s timeout, was a **startup** lost wake, not a mid-stream one — `start()` enables a node's inputs several statements before it installs the push callback, and a producer firing into that gap is accepted by the ring while waking nobody, since `Channel::push` signals only the empty→non-empty edge. Signature is zero items delivered, never a partial stall. Reproduced 7 times in 24 under CPU contention and 0 in 10 without; `start()` now closes with the level-triggered `on_input_ready()`, giving 0 in 24 on the same harness — though the *cause* was narrower than recorded there and is fixed properly in **(e)**; **(c)** `FilterNode` and `RouterNode` were the last data paths still using the throwing `push()` with the exception swallowed, so a full output discarded the value — including the **EOF sentinel**. The decimator passes EOF by predicate (`if (f.eof) return true;`) but its output is reliably full, the embedder being the slowest node in the chain, so the token was discarded, nothing downstream ever shut down, and the run had to be killed. **This is the wedge.** Both now route sentinels out-of-band and retry data until taken; the regression case delivers 6 of 40 values and never sets `saw_eof` before, 40 and terminating after; **(d)** the sentinel could be delivered *ahead of* a value still queued behind it — `pop()` observed the ring empty and then took the sentinel, and a producer can push a value *and* publish the sentinel inside that window, so any consumer treating EOF as a hard stop loses the tail. `take_sentinel` now re-checks emptiness *after* observing `has_eof_`, which is sound because the sentinel is published with a release store after the ring pushes. ~1 run in 15 before, 0 in 25 after; **(e)** two `fire_once` invocations for one node could overlap, because the submit gate was released before the firing had finished touching node state. That breaks the one-slot park the whole scheme rests on — a parked value can be overwritten by the other firing, with no drop recorded anywhere. ThreadSanitizer caught it as a race on `pending_done_`; the release is now the last act of a firing. The same sweep found the callbacks themselves being written while a running neighbour read them (ten TSan races), which is the *actual* cause of the startup lost wake in **(b)** — callbacks are now installed in a `prepare()` pass before any node starts. **New constraint:** a channel carries at most **one undelivered sentinel**; a second offered before the first is taken is refused and reported, never queued and never overwritten, since two control tokens on one channel means the stream ended twice. Single-shot EOF today, live the moment a pipeline is reused for a second input. **Consequence to hold onto:** a lossless fanout makes join depth a correctness precondition — one branch can now run ahead of another only by the slower branch's buffering, so `kSceneJoinDepth` must exceed the TransNetV2 window. Making the decimator lossless also makes it a backpressure point rather than a relief valve: the source now throttles to the face branch instead of quietly thinning it. Correct under this requirement, but it changes the shape of a loaded run and is **not yet benchmarked**. **Gap:** capacity is still counted in *items*, not bytes, so a crowd frame carrying 60 crops occupies one slot exactly as an empty one does — the memory ceiling the plan asks for is unenforced |
|
||||
| AR-004 | Backpressure: unbounded faces/frame absorbed by slowing, never by dropping or throwing | SR-002 | High | **Done** — KPN node outputs use `push_blocking`; sentinels stay out-of-band. Verified: 385/385 frames, 0 drops, byte-identical across runs |
|
||||
| AR-005 | Align to 112×112 via ArcFace 5-point similarity transform, fitted by **Umeyama least squares over all five points** (as InsightFace does) — never a robust fit, which would discard the landmarks AR-030 reads | SR-002 | High | **Done** — `umeyama_similarity()`. The RANSAC fit it replaces disagreed by a median 17 source px on 400 headshots, 83.5% of crops embedding below cos 0.99, and was unstable and RNG-driven: rebuilding caught 1614 near-duplicates against the original build's ~100. **All galleries rebuilt** (2456 actors, 10254 embeddings); measured separation gain is small (0.583 → 0.590), so recorded accuracy figures should be re-run but are not expected to move far |
|
||||
| AR-006 | 512-d L2-normalised embeddings, batched | SR-002 | High | Done |
|
||||
| AR-007 | Associate detections by IoU + embedding, with **frame-dependent** weighting | SR-002 | High | **Done** — `track_alpha` is the base for ordinary frames; drops to embedding-only on cut/boundary and for dormant tracks |
|
||||
| AR-008 | One track pool keyed on `last_seen`; no separate revival path | SR-002 | High | **Done** — one pool keyed on `last_seen`; park/revive branch deleted |
|
||||
| AR-009 | Camera-cut detection (histogram) as an association hint | SR-002 | High | Done |
|
||||
| AR-010 | Scene-boundary detection (TransNetV2) as an association hint | SR-002 | Medium | **Done** — decode butterfly joined via `SceneBoundaries`; the sampled branch waits for the detector's watermark. Frames past its last scored window are counted as unverified, never assumed boundary-free |
|
||||
| AR-011 | **Every model is fed the input it was trained for** — cost reduced by running less often, never by degrading one inference | SR-002 | High | **Done** — both violations SPEC.md named are closed. (1) `scene_decode_fps` defaults to 0 (native): at 12 fps a 100-frame `kWindow` spanned ~8.3 s instead of the ~4 s TransNetV2 was trained on, half-speed motion over twice its temporal context. (2) The boundary dedup window is derived from the cadence the detector was actually fed (`SceneDetectorFunc::dedup_window_sec()`, median observed interval, halved) rather than the literal 0.04 s — one frame at 25 fps, and at 30 fps wider than a frame, so two cuts on consecutive frames merged into one and the loss was invisible: the file simply had fewer boundaries. Derivation checked at 24/25/30 fps and under a seek (UT-003). **Consequence, not a gap:** `scene_threshold` 0.60 was fitted against the 12 fps input and is now certainly wrong — VR-006 re-fits it, and until then boundary recall at native rate is untuned rather than better. Dense decode is the cost driver, so this is not free; `dense_scale` and `scene_stride` remain the reductions that do not run the model off-distribution. **Half-applied until now:** the derived window reached `scenes.json` and nothing else. `SceneBoundaries` — the path that actually feeds `is_scene_boundary` to the tracker — kept the literal 0.04 s under a comment claiming the two views agreed. They did not. The detector now supplies the window it derived to both |
|
||||
| AR-011 | **Every model is fed the input it was trained for** — cost reduced by running less often, never by degrading one inference | SR-002 | High | Planned |
|
||||
| AR-012 | Presence follows **track extent**, not per-frame recognition | **SR-002** | High | **Done** — `src/track_registry.hpp`; window is `[first_seen, last_seen]` of an owned track |
|
||||
| AR-013 | `last_seen` optional state machine; window ends at last sighting, never after | SR-002 | High | **Done** — `last_seen` optional is the whole state machine; interior gaps absorbed, trailing cool-down never claimed |
|
||||
| AR-014 | Belief swap A→B terminates the track and starts a new one | SR-002 | Medium | **Done** — swap closes at `last_seen` and opens a successor at the swap frame; counted |
|
||||
| AR-015 | Two live tracks owned by one actor ⇒ treat as a detected cut, re-associate | SR-002 | Medium | **Done** — reverse index detects it on the causing update; counted |
|
||||
| AR-016 | All tracks closed at EOF — a film ends with faces on screen | SR-002 | High | **Done** — `flush()`, idempotent, closes at last sighting or final tick |
|
||||
| AR-017 | Every presence claim carries its belief and identification route | SR-002 | High | **Done** — `DeadTrack` carries belief, observation count, and now a `route` enum. The route was previously the literal string `"live"` written at serialisation time, so the published field could not distinguish anything and AR-017's own edge case ("deferred and pooled routes distinguishable") was unmeetable. Only `live` occurs until AR-020 lands; `deferred` exists so that pass has somewhere to write instead of a schema change to make |
|
||||
| AR-018 | Per-subject embedding store with banded admission (novel enough, safe enough) | SR-005 | Medium | **Done** — banded admission in probability space, bounds from `expand_band_lo/hi`; the lower bound re-asked pairwise at promotion, since `admit` compares only against the nearest member and a drifting track can chain past it. Retires `expand_novelty_sim` and `expand_track_spread_max` — novelty is now the eviction ordering, not a threshold. Rejections counted. Bounds unswept (VR-007) |
|
||||
| AR-019 | Per-film gallery annex from owned tracks; acquires the non-frontal views TMDB lacks | SR-005 | Medium | **Done** — all three discontinuity signals clear the buffers; ownership comes from the registry, not a second local tally. **Correction:** the local tally was still there and still deciding. Promotion fired on a local accepted-frame count and fell back to a local per-actor plurality whenever the registry had not yet claimed the track — which is the common case, since three accepted frames arrive well before a posterior crosses `ownership_logodds`. So in practice the plurality usually decided, and it could not see the AR-025 discounting it was supposed to defer to. Promotion now requires the registry's verdict; the accepted-frame count is an explicit evidence floor. `forget()`, which had no callers under a comment claiming the matcher called it, is replaced by `prune_dead` against the registry's own liveness |
|
||||
| AR-017 | Every presence claim carries its belief and identification route | SR-002 | High | **Done** — `DeadTrack` carries belief and observation count |
|
||||
| AR-018 | Per-subject embedding store with banded admission (novel enough, safe enough) | SR-005 | Medium | **Done** — banded admission in probability space; replaces `expand_novelty_sim`. Rejections counted |
|
||||
| AR-019 | Per-film gallery annex from owned tracks; acquires the non-frontal views TMDB lacks | SR-005 | Medium | **Done** — all three discontinuity signals clear the buffers; ownership comes from the registry, not a second local tally |
|
||||
| AR-020 | Deferred re-identification of unknown tracks against the final expanded gallery | SR-005 | High | Planned |
|
||||
| AR-021 | Cluster unknown tracks into one entity per person, under temporal cannot-link constraints | SR-005 | Medium | Planned |
|
||||
| AR-022 | Capture still-unidentified tracks: embeddings, metadata, **context crops** for human review | §4 | Medium | Planned |
|
||||
| AR-023 | Fit sigmoid calibration from intra/inter similarity distributions | SR-002 | High | **Done** — and the meaning of "the fit failed" is now uniform. `valid=false` used to send the matcher to a raw-cosine accept rule while `same_person_probability` sent every other stage to the untuned default sigmoid: one run, two policies, no announcement. Both now take the default sigmoid and warn loudly that the probabilities are not meaningful |
|
||||
| AR-024 | **Always the calibrated probability, never a raw cosine** — exceptions recorded | SR-002 | High | **Done** — association, accumulation and expansion all in probability space; `track_max_embed_dist`, `cut_revive_sim`, `expand_novelty_sim`, `expand_track_spread_max` retired. Enforcement now exists rather than being asserted: `scripts/ci/check_raw_cosine.py` blocks in CI. It immediately caught a live violation — the matcher's no-calibration fallback thresholded raw cosine distance **and fed `max(0, cosine)` into `TrackRegistry::observe`**, whose contract says in terms that it cannot be handed an uncalibrated number by a careless caller. `match_threshold`, `match_ratio` and `match_ratio_ceil` are retired with it, and `TrackGallery`'s `max(0, cosine)` default calibration is now a hard error. One exception recorded, in the calibration's own dedup |
|
||||
| AR-025 | Per-track Bayesian accumulation in log-odds, with correlated-observation discounting | SR-002 | High | **Done** — log-odds accumulation with correlation discounting owned by the registry, `src/evidence_discount.hpp`. The four constants governing this — `ownership_logodds`, `rho_max`, `admit_below`, `max_views` — were unreachable in-class defaults until now; see VR-007 |
|
||||
| AR-026 | All similarity computed as GEMM, including annex and deferred pass | SR-001 | High | **In Progress** — two of the three call sites done. Baked gallery was already GEMM; the annex now is too — it is a contiguous row-major matrix (`track_gallery.hpp`) whose promoted rows are appended to the engine's resident matrix (`ISimilarityEngine::append_rows`), so one multiply covers baked and promoted references and the host-side cosine loop is gone. CPU path requires OpenBLAS (scalar fallback now opt-in behind `SAE_ALLOW_SCALAR_GEMM`). Remaining: the deferred pass, which does not exist until AR-020 |
|
||||
| AR-023 | Fit sigmoid calibration from intra/inter similarity distributions | SR-002 | High | Done |
|
||||
| AR-024 | **Always the calibrated probability, never a raw cosine** — exceptions recorded | SR-002 | High | **Done** — association and accumulation both in probability space; `track_max_embed_dist`, `cut_revive_sim` retired |
|
||||
| AR-025 | Per-track Bayesian accumulation in log-odds, with correlated-observation discounting | SR-002 | High | **Done** — log-odds accumulation with correlation discounting owned by the registry, `src/evidence_discount.hpp` |
|
||||
| AR-026 | All similarity computed as GEMM, including annex and deferred pass | SR-001 | High | In Progress |
|
||||
| AR-027 | Throughput acceptable for **arbitrary** gallery size | SR-001 | High | Planned |
|
||||
| AR-028 | **Embedding input quality assessed and carried** — every face scored on size, sharpness and visibility before its embedding is used as identity evidence; the vector travels with the face and reaches the VR-001 dump | SR-002 | High | **Done** — filled in by `FaceAlignerFunc`, where both measured axes come free from the warp; carried on `DetectedFace` and written to the dump as `faces/sharpness` + `faces/alignment_residual`, taking it to `schema_version` 2. Size is `bbox`, not duplicated into a field that would drift. No face is admitted unscored (-1 sentinel), and the degenerate-fit case is now counted and reported rather than silently dropped. **Carried, not consumed** — no discount and no threshold, which is AR-030 and VR-012. Verified UT-137, UT-138 (aligner) and UT-139…UT-141 (dump round-trip, version, sentinel). The committed fixtures are still v1, so they carry no vector until `make_fixtures.sh` is re-run on a GPU host |
|
||||
| AR-029 | Sharpness measure on the **aligned crop** (scale-normalised, so it cannot re-measure size) | SR-002 | Medium | **Done** — `crop_sharpness()`: variance of the Laplacian over variance of the crop, so contrast cannot leak in the way it does for the raw textbook measure. Both blur ladders monotone, Gaussian and motion. Three properties recorded on the function for VR-012 rather than corrected here: the contrast invariance is exact in the algebra but bends at the 8-bit quantisation floor (a dim *and* soft crop reads sharper than it is — 148% high at σ 2.5), `BORDER_CONSTANT` fill from a frame-edge face adds a step edge, and the measure conflates focus with intrinsic texture. Verified UT-130…UT-136 |
|
||||
| AR-028 | **Embedding input quality assessed and carried** — every face scored on size, sharpness and visibility before its embedding is used as identity evidence; the vector travels with the face and reaches the VR-001 dump | SR-002 | High | Planned |
|
||||
| AR-029 | Sharpness measure on the **aligned crop**, consumed as a discount and **never as a gate** | SR-002 | Medium | **In Progress** — five candidates implemented (`src/quality.hpp`) and ranked by VR-012 over three blur families. `var_laplacian` and `tenengrad` are **disqualified as discounts**: within a fixed degradation they are anti-predictive on *all three* families (AUC 0.42–0.47; the decile the measure calls sharpest is 2.6× *less* identifiable), since their residual variance is native contrast, not detail. `hf_energy_ratio` is the only correctly-signed survivor, best on all three, and weak (0.52–0.56). `dir_min_tenengrad` is the best *gross-smear detector* (pooled AUC 0.854 on motion) but ~chance within-cell, so it serves a per-frame flag, not a per-observation weight. The parenthetical this row used to carry — "scale-normalised, so it cannot re-measure size" — was wrong: every candidate responds to source size, and the axes are separable for a different reason (see AR-028) |
|
||||
| AR-030 | Visibility measure from the AR-001 5-point landmarks — extreme pose or occlusion **discounts the observation, never deletes the detection** | SR-002 | Medium | **In Progress** — measure is the AR-005 alignment residual (`estimate_alignment()`), carried on `DetectedFace`; roll/scale invariance and monotonicity under foreshortening asserted. Nothing consumes it as a discount yet |
|
||||
|
||||
## Deployment (DP)
|
||||
|
||||
| ID | Requirement | Traces to | Priority | Status |
|
||||
|---|---|---|---|---|
|
||||
| DP-001 | One analysis core; modes are front-ends and must not fork pipeline logic | PR-004 | High | **Done, after a repair.** `scene_preview` had forked the construction sequence and then rotted: it built `FaceTrackerFunc{cfg}` against a signature that stopped existing with the AR-007/AR-008 redesign, so **it had not compiled since**, and it never wired registry claims into its sink. It now mirrors `main.cpp` exactly — matcher, then registry, then tracker. The lesson is that "must not fork" needs the build to notice; a front-end nothing compiles is a fork that rots in silence |
|
||||
| DP-001 | One analysis core; modes are front-ends and must not fork pipeline logic | PR-004 | High | Done |
|
||||
| DP-002 | Batch CLI over one title | PR-004 | High | Done |
|
||||
| DP-003 | On-demand resident service with bounded, observable queue | PR-004 | Medium | Planned |
|
||||
| DP-004 | Opportunistic/idle mode: external trigger, hard stop, implicit re-queue | PR-004 | Medium | Planned |
|
||||
| DP-005 | Native installer, no Docker; Fedora + Arch | PR-004 | Medium | Planned |
|
||||
| DP-006 | Background incremental gallery refresh on a timer | PR-003 | Medium | Planned |
|
||||
| DP-007 | CI builder image, CPU-only, pinned by tag in the Gitea container registry | PR-004 | High | **Mostly** — image and publish script exist (`Dockerfile.builder-cpu`, `scripts/ci/build_builder_image.sh`) and `.gitea/workflows/unit-tests.yml` now consumes it, pinned to `v1` and asserting at run time that the image reports that tag. **Gap:** the image is built and pushed by hand from an authenticated host; nothing rebuilds it on a change to the Dockerfile |
|
||||
| DP-007 | CI builder image, CPU-only, pinned by tag in the Gitea container registry | PR-004 | High | Planned |
|
||||
| DP-008 | Builder images + release jobs per backend (cpu / cuda / rocm); ship binaries, not engines | PR-004 | Medium | Planned |
|
||||
|
||||
## Integration (IR)
|
||||
@@ -79,7 +79,7 @@ Status: `Done` · `In Progress` · `Planned` · `TBD` · `Withdrawn`
|
||||
| IR-001 | Emit the JRay truth format as sibling `.jray.json` | SR-003 | High | Done |
|
||||
| IR-002 | Windows carry belief + route; `extraction.*` carries `extinction_sec`, `gallery_scope` | SR-003 | High | **Done** — `schema_version: 2`; windows are objects with `belief` + `route`; `extraction.*` carries `extinction_sec` and `gallery_scope`; `anneal_sec` removed |
|
||||
| IR-003 | Output written **after** the deferred pass, not at EOF | SR-003 | High | **In Progress** — sink builds windows from registry claims and flushes at EOF; the deferred pass (AR-020) does not exist yet, so output is still final at EOF |
|
||||
| IR-004 | Compute the audio signature exactly per server spec §3 | SR-003 | Medium | **Done** — `src/audio_signature.*`; not yet emitted into the truth file (IR-002). One real defect found and fixed since: the resampler's `AVChannelLayout`s were not zero-initialised, and `av_channel_layout_copy` uninitialises its destination first, so `av_freep` was handed stack garbage. It aborted about 1 run in 4 of UT-103 — invisible in the aggregate test binary, where the case usually passes, and absent under a sanitizer build because it is stack-dependent. `ctest`, one process per case, is what turned it into a reproducible failure |
|
||||
| IR-004 | Compute the audio signature exactly per server spec §3 | SR-003 | Medium | **Done** — `src/audio_signature.*`; not yet emitted into the truth file (IR-002) |
|
||||
| IR-005 | Golden-vector fixture shared with the plugin repo to prove bit-exactness | SR-003 | High | **Done** — `tests/fixtures/audio/`; v1 parameters now normative in server spec §3 |
|
||||
| IR-007 | Media < 120 s: emit no signature, apply no sync offset — identical rule in both producers | SR-003 | Low | **Done** |
|
||||
| IR-008 | Emit and honour the signature's own `v1:` version prefix | SR-003 | Low | **Done** |
|
||||
@@ -91,7 +91,7 @@ Status: `Done` · `In Progress` · `Planned` · `TBD` · `Withdrawn`
|
||||
|---|---|---|---|---|
|
||||
| GR-001 | Build gallery from Jellyfin library cast, TMDB profile fallback | SR-001, SR-005 | High | Done |
|
||||
| GR-002 | Incremental `--merge` refresh without re-embedding known actors | PR-003 | High | Done |
|
||||
| GR-003 | Report coverage: zero-image actors, under-referenced actors, dedup, calibration PDFs | SR-001 | Medium | **Done** — `gallery/gallery_report.hpp`, written next to the gallery by `build_gallery`. Zero-usable-image actors come from the build audit, which a stored gallery cannot reconstruct; also `distinct_references`, `duplicates_removed`, and the intra/inter distributions the calibration fits and would otherwise discard |
|
||||
| GR-003 | Report coverage: zero-image actors, under-referenced actors, dedup, calibration PDFs | SR-001 | Medium | Planned |
|
||||
| GR-004 | Stamp embedder identity into the gallery; **hard startup error** on mismatch | SR-001 | High | **Done** — basename + SHA-256 + `embed_dim`; mismatch fatal with no bypass, unstamped warns unless `--require-gallery-stamp`; `scripts/stamp_gallery.py` migrates in place |
|
||||
| GR-005 | Gallery data never leaves the instance | **SR-005** | High | Done |
|
||||
| GR-006 | Provenance tiers: baked / harvested / confirmed, distinguishable per embedding | SR-005 | High | Planned |
|
||||
@@ -104,22 +104,19 @@ Status: `Done` · `In Progress` · `Planned` · `TBD` · `Withdrawn`
|
||||
| ID | Requirement | Traces to | Priority | Status |
|
||||
|---|---|---|---|---|
|
||||
| VR-001 | HDF5 post-inference dump at the embedded-frame boundary | PR-002 | High | Done |
|
||||
| VR-002 | Replay drives the **real** KPN nodes, not a reimplementation | PR-002 | High | **Done** — including the sink, as of VR-011. Worth recording what the reimplementation was hiding: `build_minimal` rebuilt windows in Python from per-frame annotations, which never consult the registry, so it kept producing plausible output while registry-based presence in replay was returning **nothing at all**. The first run of the real chain emitted 0 actors on a film where 1647 frames carried an identified face. A reimplementation does not merely risk disagreeing with the pipeline; it can conceal the pipeline being broken |
|
||||
| VR-002 | Replay drives the **real** KPN nodes, not a reimplementation | PR-002 | High | **Done** — replay driven from committed fixtures in `tests/test_replay_fixtures.cpp`; determinism asserted |
|
||||
| VR-003 | Scoring: micro-F1 against X-Ray, precision/recall logged at every evaluation | PR-002 | High | Done |
|
||||
| VR-004 | Reproducible validation corpus with ground truth | PR-002 | High | Done |
|
||||
| VR-005 | Minimum face size study — TPI/FPI vs probe size, gallery held at native res | PR-002 | Medium | **Done** — knee at 24–32 px; 32 px gives 98.1% TPI, 0.0 FPI at every size. Degrades an already-aligned 112×112 crop, so it isolates the embedder and is an **upper bound**; VR-013 measures the same question end to end and AR-002 takes its number, not this one |
|
||||
| VR-006 | Re-tune `scene_threshold` once native-rate decode lands | PR-002 | **Medium** | **Planned, now unblocked** — native-rate decode landed with AR-011, so the prerequisite is met and the current 0.60 is a value fitted against input the pipeline no longer produces. Raised from Low for that reason: it is no longer a refinement, it is a stale constant |
|
||||
| VR-007 | Expansion band, clustering threshold, deferred-pass ablation, **and the AR-025 accumulation knobs** | PR-002 | Medium | **Planned — scope corrected.** `rho_max`'s own comment already deferred to this row, and four constants it names were unreachable: `ownership_logodds` on `TrackRegistry::Config`, and `max_views`/`admit_below`/`rho_max` on `EvidenceDiscounter::Config`, which `main` built through the one-argument constructor. No sweep could vary them. They are in `Config` with CLI flags now, so this row can be run. `ownership_logodds` is the one to start with: below it a track makes **no presence claim at all**, so it decides whether an actor is reported rather than how confidently |
|
||||
| VR-006 | Re-tune `scene_threshold` once native-rate decode lands | PR-002 | Low | Planned |
|
||||
| VR-007 | Expansion band, clustering threshold, and deferred-pass ablation | PR-002 | Medium | Planned |
|
||||
| VR-008 | Gallery scaling benchmark — throughput vs gallery size | PR-002 | Medium | Planned |
|
||||
| VR-009 | Verify accumulated posteriors are calibrated against held-out tracks | PR-002 | High | Planned |
|
||||
| VR-010 | Dump provenance attributes — embedder model, detector settings, `dense_scale`, `scene_detect`, sample rate | PR-002 | **High** | **Done** — `DumpProvenance` in `embedding_dump_node.hpp`, written as root attributes and read back tolerantly. Every field is optional so a pre-VR-010 dump reads as *unknown* rather than as a default; a silently-defaulted `detector_conf` is exactly the fabricated provenance this exists to prevent. This row said `Planned` while five VR-010 tags sat in the code — stale in the opposite direction to the rest of this register |
|
||||
| VR-011 | Rewrite the replay harness for the post-AR-012 output contract | PR-002 | High | **Done** — `sae_kpn` compiles again and the replay drives the whole chain including `ResultSinkFunc`, so presence comes from `TrackRegistry` claims rather than being rebuilt in Python. The three per-node factories are replaced by one `add_pipeline` that mirrors `main.cpp`'s construction order — the ordering constraint (matcher fits the calibration, registry needs a discounter from it, tracker needs both, sink needs the claims) is what a factory-per-node API could not express, and is why the tracker factory kept building `FaceTrackerFunc{cfg}` against a signature that had stopped existing. `build_minimal` and `anneal_sec` are gone. Verified end to end on the SuperHero fixture: 5 actors, 32 windows, 0 dropped votes |
|
||||
| VR-012 | Quality-knee study — TPI/FPI vs sharpness and vs pose, as VR-005 did for size; also settles whether the 5-point pose proxy needs a dedicated landmark model | PR-002 | Medium | Planned |
|
||||
| VR-010 | Dump provenance attributes — embedder model, detector settings, `dense_scale`, `scene_detect`, sample rate | PR-002 | **High** | Planned |
|
||||
| VR-011 | Rewrite the replay harness for the post-AR-012 output contract | PR-002 | High | Planned |
|
||||
| VR-012 | Quality-knee study — TPI/FPI vs sharpness and vs pose, as VR-005 did for size; also settles whether the 5-point pose proxy needs a dedicated landmark model | PR-002 | Medium | **In Progress** — sharpness half done ([`docs/quality-knee.md`](quality-knee.md)): 1670 actors, joint size×blur grid over three blur families (Gaussian, disc defocus, linear motion), 60120 probe-cell records each. Sharpness is **not a sufficient statistic** (equal measured sharpness spans 15.3–91.0% TPI, ordered by source size); **the blur family matters more than its amount** — at matched per-axis σ=3 on a 112 px face, Gaussian/motion/defocus cost 9/18/**53**% error, so a Gaussian-only sweep understates real lens blur fivefold; blur breaks **confidence, not ranking** (rank-1 80.2% where TPI is 15.3%), so FPI never left 0.1% in any of the 108 cells; a sharpness **gate** loses 3× more true presence than the free size filter at equal saving, because even destroyed faces stay 46.9% identifiable. **Pose half not started** — the AR-030 residual is exposed via `sae_embed.alignment_residual` but no pose arm has been run, so the dedicated-landmark-model question is still open |
|
||||
| VR-014 | Audio-signature **offset recovery on real content** — a known trim recovered from film audio, not from the synthetic golden tone | PR-002 | Medium | **Done** — 40 random in-cap offsets, every one recovered to the nearest frame: **worst error 46 ms against a 500 ms budget**, and 46 ms is the floor rather than a result, since the offset is quantised to whole 92.88 ms frames. The `runtime/2` anchor confirmed through real head-trimmed files (a `delta` trim moves the window by `delta/2`). The one soft spot is **tier labelling, not accuracy**: the score falls with sub-frame misalignment (0.94–0.99 near a frame boundary, 0.69–0.73 at half a frame), so 27/40 correct alignments were demoted to `loose`. ±1 frame of slack in the *score* fixes it — measured, all 40 back to `audio` (min 0.906), false matches unmoved at 0.12–0.16, costing 81 ms of the budget |
|
||||
| VR-015 | Per-node cost and bottleneck attribution for a run — where the time actually goes | PR-004 | High | **Done** — `--benchmark <path>` on `scene_analyze`; `src/benchmark.hpp`. Reports cumulative CPU and wall time per node, and locates the pacing node from sampled channel occupancy rather than from time-in-node, which backpressure inflates. Verified UT-120…UT-124 |
|
||||
| VR-013 | Cross-source identification probe — gallery from one recording, probes from another, swept over input resolution end to end | PR-002 | Medium | **In Progress** — holding 90% of the plateau needs ~50 px end to end against VR-005's ~22 px, the gap being detection and landmark error; **`min_face_px` 40, since 32 admits faces in the falling region** (AR-002). FPI 0.0% at every scale. Ceiling is cross-view, not resolution |
|
||||
| VR-016 | **Cut-detection cadence study** — is `cut_threshold` 0.70 meaningful at the rate `camera_pos` is actually fed? | PR-002 | Medium | **Planned.** The histogram cut detector is the one always-on signal with no recorded provenance, and its input rate is not the rate it was fitted at. With `--scene-detect` off, `camera_pos` sits downstream of a source already decimated to `sample_fps`, so at the 1.0 default it compares frames **one second apart** — inside a single shot those differ enormously, and 0.70 correlation is a low bar to clear. With `--scene-detect` on it sees native-rate frames instead, so the same constant means two different things depending on an unrelated flag. This is AR-011's argument ("every model gets the input it was trained for") applied to a non-neural detector, and it matters because `is_cut` drives `track_alpha` to 0 and clears every expansion buffer. Cheap first measurement: run `camera_pos` over a `hero/` clip at 1/2/5 fps and compare cut counts against `tests/fixtures/dumps/scene_bounds.json`. The committed 5 fps dump shows 2.6% of frames flagged; nobody has measured 1 fps |
|
||||
| VR-017 | **Vote-lag study** — how often does the matcher fall more than `track_extinction_sec` behind the tracker on real content? | PR-002 | **High** | **Planned.** Channel depth is a correctness parameter between `face_tracker` and `identity_matcher`, and the constraint runs opposite to the scene join's: there `kSceneJoinDepth` must EXCEED the TransNetV2 window, here the depth must be UNDER `track_extinction_sec × sample_fps`. Backpressure is what makes it bite — it is working, and a lossless channel converts depth into lag by design. Both nodes are 16 deep in `main.cpp`, which at the default `sample_fps` 1.0 is ~16 s of lag against a 5 s window, so `scene_analyze` can drop identity votes and until now said nothing. It now reports `dropped_votes` at shutdown; this row is the measurement that decides whether that should be fatal, and whether the right fix is bounding the depth or removing the coupling (reap on the matcher's clock rather than the tracker's, so a vote cannot be late by construction) |
|
||||
|
||||
---
|
||||
|
||||
@@ -216,7 +213,6 @@ as such rather than counted as covered.
|
||||
| GR-001 … GR-005 | T1 + T3 | Gallery assembly is I/O and bookkeeping; embedding is T3 smoke |
|
||||
| GR-006 … GR-008 | T1 | Tiering and outlier detection operate on stored embeddings |
|
||||
| VR-* | Out of CI | Studies are run deliberately and their results committed as documents |
|
||||
| VR-014 | **T2** | The exception, and the reason the blanket row above is not the whole story: its fixture is committed and its signature is CPU-only DSP, so the study *is* a test a CI host can run — not a measurement someone has to remember to repeat |
|
||||
|
||||
**One consequence worth stating:** AR-027 (arbitrary gallery scale) is
|
||||
structurally unverifiable on the CI host. It needs a GPU host and a synthetic
|
||||
@@ -234,9 +230,9 @@ time**, and CI consumes the result as data.
|
||||
This is what makes the T1/T2 split load-bearing rather than a preference: T1 and
|
||||
T2 are the only tiers that can exist in CI at all.
|
||||
|
||||
### Fixture corpus — `hero/`
|
||||
### Fixture corpus — `bali/`
|
||||
|
||||
Five clips of **SuperHero (1952)**, ~77 s each, 480×360, 30 fps, 42 MB total.
|
||||
Five clips of **Road to Bali (1952)**, ~77 s each, 480×360, 30 fps, 42 MB total.
|
||||
|
||||
Public domain, and that is the reason to use it rather than a convenience:
|
||||
**derived fixtures — dumps, crops, golden outputs — can be committed without the
|
||||
@@ -315,7 +311,7 @@ because it will be trusted.
|
||||
| AR-001 | T3 | Detector returns plausible boxes on a known frame | — smoke only |
|
||||
| AR-002 | T2 | Faces below 40 px (original res) are dropped | Exactly at threshold; with `dense_scale` 0.5 — the interaction that motivated the requirement |
|
||||
| AR-003 | T2 | No cap applied; a 40-face frame yields 40 | Crowd frame |
|
||||
| AR-004 | T1 | Saturated input blocks rather than drops or throws | Bounded queue at capacity; **byte-based** limit with large crops; SIGTERM mid-block. Cases the KPN suite now pins, each of which failed before being written: a fanout feeding an unequal pair loses nothing *and* throttles the fast branch (either assertion alone passes on a broken implementation); a filter delivers EOF into a saturated output; a sentinel is never delivered ahead of a queued value; a twice-parked value keeps its payload; and a node started with data already in its input still fires — the startup lost wake, which needs no contention to reproduce once the state is constructed directly |
|
||||
| AR-004 | T1 | Saturated input blocks rather than drops or throws | Bounded queue at capacity; **byte-based** limit with large crops; SIGTERM mid-block |
|
||||
| AR-005 | T1 | Known landmarks → expected 112×112 warp; the fit never mirrors | Landmarks near frame edge; degenerate/collinear points; a mirrored set — SVD returns a reflection unless the determinant guard rejects it |
|
||||
| AR-006 | T3 | Embeddings are unit-norm | Batch smaller than, equal to, larger than `embed_batch_size` |
|
||||
| AR-007 | T2 | Association picks the right track | Two faces crossing paths; one leaving frame as another enters |
|
||||
@@ -327,23 +323,22 @@ because it will be trusted.
|
||||
| AR-014 | T2 | Belief swap closes one window, opens another | No blended window; no overlap at the swap frame |
|
||||
| AR-015 | T2 | Two live tracks on one actor trigger re-association | Counter increments |
|
||||
| AR-016 | **T2** | Every track closed at EOF | Film ending mid-shot — window ends at final frame, not dropped |
|
||||
| AR-017 | T1 | Claim carries posterior and route | Deferred and pooled routes distinguishable — now possible: `route` is an enum on `DeadTrack` rather than the literal `"live"` the sink used to write. Only `live` occurs until AR-020 exists, so the test that matters today is that the field survives serialisation |
|
||||
| AR-017 | T1 | Claim carries posterior and route | Deferred and pooled routes distinguishable |
|
||||
| AR-018 | T1 | Band admits only within bounds | At each bound exactly; store never admits below lower bound |
|
||||
| AR-019 | T2 | Promotion only when all three signals quiet | Cut mid-track blocks promotion |
|
||||
| AR-020 | **T2** | Unknown resolved after expansion | Track failing at minute 12, resolved at EOF — the ordering-independence claim |
|
||||
| AR-021 | T2 | Clustering merges same person, respects cannot-link | **Temporally overlapping tracks never merge**; measure how many merges the constraint rejects |
|
||||
| AR-022 | T1 | Context crops retained, bounded per track | Track running for minutes |
|
||||
| AR-023 | T1 | Sigmoid fit on synthetic separable data | Too few positive pairs → `valid=false`, and the fallback that engages is the **default sigmoid**, not the retired cosine rule. Assert the warning fires: an unfitted sigmoid returns plausible-looking probabilities, so nothing downstream can tell |
|
||||
| AR-024 | **Static check** | No bare cosine outside a tagged `EXCEPTION` | `scripts/ci/check_raw_cosine.py`, blocking in the traceability workflow. Honest about its reach: it catches direct `cosine_similarity()` uses not routed through a calibration and **cannot follow a cosine through a variable across statements**, which is a convention backed by review rather than by the tool. Scans `src` only — a test legitimately asserts properties of the metric space, and sweeping those in would produce blanket exceptions that devalue the tag |
|
||||
| AR-023 | T1 | Sigmoid fit on synthetic separable data | Too few positive pairs → `valid=false`, fallback engages |
|
||||
| AR-024 | **Static check** | No bare cosine outside a tagged `EXCEPTION` | Grep-based; this is the invariant's enforcement |
|
||||
| AR-025 | T1 | Log-odds accumulate; correlated frames discounted | 30 identical frames must **not** reach the certainty of 30 diverse ones |
|
||||
| AR-026 | T1 + T4 | GEMM path produces same result as reference loop | Equivalence on small input in CI; throughput on GPU host |
|
||||
| AR-027 | **T4** | Throughput at 10²…10⁵ actors | Scheduled, not on-demand |
|
||||
| AR-028 | **T2** | No embedding reaches the matcher unscored; the vector survives into the dump | Face failing exactly one axis; all three healthy; a face whose landmarks are degenerate — dropped for want of a crop to score, but **counted** rather than silently vanished (UT-138) |
|
||||
| AR-029 | T1 | Synthetic blur ladder → monotonically falling sharpness | Gaussian vs motion blur; **small sharp face vs large soft one** — size must not leak into this axis. The blur ladder must be measured on a **1/f texture**: on a flat-spectrum one the motion ladder *rises*, since an anisotropic smear takes energy out of numerator and denominator together (UT-131). Contrast must not leak either — exact in the algebra, and the 8-bit floor that bends it is pinned by UT-133 |
|
||||
| AR-028 | **T2** | No embedding reaches the matcher unscored; the vector survives into the dump | Face failing exactly one axis; all three healthy; a face whose landmarks are degenerate — scored, not silently vanished |
|
||||
| AR-029 | T1 | Synthetic blur ladder → monotonically falling sharpness | Gaussian vs motion blur; **small sharp face vs large soft one** — size must not leak into this axis |
|
||||
| AR-030 | T1 | Alignment residual rises monotonically with foreshortening | **In-plane roll, scale and translation must leave it at zero** — the property that makes it a pose measure rather than a pose-and-everything-else measure; face size must not shift it; degenerate landmarks report not-ok rather than a number |
|
||||
| VR-012 | **T4** | Knee located per axis on held-out films | Report each candidate threshold's cost in **lost true presence**, not only its gain in precision — a gate that improves misID by discarding half the cast has not helped |
|
||||
| VR-013 | **T4** | Identification holds across two recordings of the same people, and degrades to TBI rather than to a wrong name as input resolution falls | Gallery and probes must come from *different* recordings — a hold-one-out over one recording measures a much easier problem and will not surface the cross-view failure. Ground truth is hand-sorted; labels propagated by embedding similarity would keep only the faces the embedder already gets right |
|
||||
| VR-016 | **T2** | Cut rate as a function of the cadence `camera_pos` is fed | Same clip at 1/2/5 fps, `--scene-detect` on and off. The dump already records `cut_threshold` and `sample_fps` (VR-010), so a replay can score this without re-decoding. A finding of "0.70 is fine at every rate" is a real result and should be recorded as one |
|
||||
| IR-001/002 | T1 | Serialised output matches golden file | Zero-length window; actor with many windows |
|
||||
| IR-003 | T1 | Output written after deferred pass | Not at EOF |
|
||||
| IR-004/005 | **T1** | Signature matches golden vector bit-for-bit | Identical result in both producer repos |
|
||||
@@ -372,29 +367,8 @@ accumulation from being decoration.
|
||||
| — | `anneal_sec` window merging | Superseded by AR-012/AR-013: a track survives its own gaps, so there is nothing to anneal |
|
||||
| — | `extinction_sec` actor keep-alive | Superseded by AR-013: windows end at last sighting, which is what this over-claimed |
|
||||
|
||||
Both are now deleted rather than retained at zero — a field naming a mechanism
|
||||
the pipeline no longer has is actively misleading (see `SPEC.md` A6.6).
|
||||
|
||||
**This paragraph was false for some time, and the failure is worth keeping.** It
|
||||
was written in the present perfect as though the removal had happened. It had
|
||||
not: `Config::extinction_sec` (57.4) and `Config::anneal_sec` (35.5) were still
|
||||
there, `--extinction` and `--anneal` still parsed, and `SceneTrackerFunc` still
|
||||
ran its keep-alive in both shipped pipelines, printing its timeout at every
|
||||
startup. `SPEC.md`'s removal list ends "grep for both names and expect no
|
||||
survivors"; there were about forty.
|
||||
|
||||
Nothing in the tooling could have caught it. The traceability gate reads tags,
|
||||
not behaviour, and a withdrawn requirement has no tag to be orphaned — the
|
||||
register simply asserted a state of the code, and no test asked. The general
|
||||
form is worth stating: **a status column is a claim, and the only claims this
|
||||
project can check automatically are the ones a test or a static check makes.**
|
||||
The same pattern produced three other rows corrected in this pass (AR-011,
|
||||
AR-017, AR-019), each recorded as done and done in one place out of two.
|
||||
|
||||
`SceneTrackerFunc` is replaced by the stateless `FrameAnnotationFunc`. One
|
||||
visible consequence: `--verbosity standard`'s `frames[].identified` used to
|
||||
include every actor inside the keep-alive window, and now lists what was matched
|
||||
in that frame. Minimal and xray output never consulted the node.
|
||||
Both were deleted rather than retained at zero — a field naming a mechanism the
|
||||
pipeline no longer has is actively misleading (see `SPEC.md` A6.6).
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -1,205 +0,0 @@
|
||||
# The learned scene-boundary detector
|
||||
|
||||
Presence uses **flood-fill**: an actor seen once inside a shot is reported for the
|
||||
whole shot (`[prev_boundary, next_boundary]`). That only works if the boundaries
|
||||
are good. This page is the story of getting them good — a learned scene-boundary
|
||||
detector that lifts per-second actor-presence F1 from **62.6% to 74.9%** across
|
||||
the nine-film X-Ray benchmark, and fixes the film where naive flood-fill was
|
||||
actively harmful.
|
||||
|
||||
That 74.9% is the **leave-one-out** figure: each film is scored by a detector
|
||||
trained on the *other eight*, so no film's presence is measured with a detector
|
||||
that ever saw it. It is the honest generalisation number, and it is only ~1 point
|
||||
below the all-nine-trained model (75.8%) — the detector barely overfits.
|
||||
|
||||
## Why the old cut detector wasn't enough
|
||||
|
||||
The always-on boundary source was the grayscale histogram-correlation cut detector
|
||||
(`camera_position_change_detector`): mark a cut when the frame-to-frame grayscale
|
||||
histogram correlation drops below 0.70. It is cheap and it fires on obvious hard
|
||||
cuts, but on a low-contrast, uniformly-graded film it is nearly blind. On
|
||||
**Scarface** it fired **once in 10,204 frames**. Flood-fill then snapped every
|
||||
actor across essentially the whole film:
|
||||
|
||||
| Scarface | precision | recall |
|
||||
| -------- | --------- | ------ |
|
||||
| flood + grayscale cuts | **26%** | 95% |
|
||||
| track-extent (no flood) | 92% | 45% |
|
||||
|
||||
That single failure is what motivated everything below: flood-fill needs a
|
||||
boundary source that works regardless of grade.
|
||||
|
||||
## What we are detecting, and why it is hard
|
||||
|
||||
The training target is **Amazon X-Ray scene boundaries** (`scenes.csv`). These are
|
||||
*narrative* scenes — a new location or beat in the story — not shot cuts. There
|
||||
are only ~20–60 of them per film (median scene ~170 s), and many transition
|
||||
*within* continuous visual style and continuous audio. So the signal is sparse and
|
||||
often genuinely faint: a boundary detector working from audio-visual features can
|
||||
never recall a narrative cut that has no audio-visual signature.
|
||||
|
||||
This shapes every result: absolute boundary-F1 is modest by construction. What
|
||||
matters is the **downstream** number — does snapping flood-fill to these
|
||||
boundaries name the right actors — and there the gain is large.
|
||||
|
||||
## The features (what worked, measured)
|
||||
|
||||
Everything is per second, aligned to the 1-fps presence grid.
|
||||
|
||||
- **Delta histograms, not raw histograms.** The raw RGB histogram encodes what a
|
||||
frame *looks like*, not that it *changed* — measured boundary separability ~1.4×.
|
||||
The **symmetric histogram delta** `|hist(t+k) − hist(t−k)|` separates boundaries
|
||||
**4–5×**. Leading with deltas (k = 1,2,4,8 s) and dropping the raw histogram was
|
||||
the single biggest feature win (LSTM F1 7.5% → 10.8%).
|
||||
- **A multi-scale "ramp" bank.** Antisymmetric matched filters at half-widths
|
||||
H = 2,4,6,8,10 s; the model weights the scales. Different films' boundaries peak
|
||||
at different widths.
|
||||
- **A time-since-last-boundary "debounce" clock**, scaled by the corpus mean scene
|
||||
length (~205 s), encoding that scenes don't restart moments apart.
|
||||
- **Audio log-PSD** (per-second, 4 s window, ~57 log-frequency bins). Measured
|
||||
weak on its own — a standalone audio cutter scored only 3–6% held-out F1, because
|
||||
narrative boundaries usually have continuous audio — but it is complementary on
|
||||
the films where video is weak (Downton, Sound of Metal), so it is included and
|
||||
the model uses it where it helps.
|
||||
|
||||
|
||||
|
||||
The left panel is the *feature* development, scored at a strict ±2 s tolerance so
|
||||
each change is visible — this is where "delta beats raw histogram" was measured, not
|
||||
the shipped tolerance. The right panel is the shipped detector at the ±20 s
|
||||
tolerance the pipeline actually uses (see below). The two panels are on different
|
||||
tolerances by design and must not be read as one curve.
|
||||
|
||||
Dead ends, all measured and discarded: audio-only detection; raw
|
||||
histograms/PSDs as input; a two-tower BiLSTM (no better than the tree, far slower);
|
||||
larger FFT windows / more frequency bins (worse — boundaries are short events);
|
||||
and TransNetV2 (a Conv3D net that will not co-reside with the ROCm/VAAPI stack).
|
||||
|
||||
## The model
|
||||
|
||||
- **XGBoost regressor** over a ±3 s window of the features above, predicting a
|
||||
**soft Gaussian proximity-to-boundary target** (`exp(-(d/σ)²)`, σ = 10 s).
|
||||
Regression to a soft target — rather than a hard 0/1 label — stops a near-miss
|
||||
from being trained as a hard negative, and yields a smooth score whose **peaks**
|
||||
are the boundaries.
|
||||
- **Per-film knee threshold.** The predicted peak heights form a
|
||||
convex-decreasing curve; the knee (max drop below the endpoints' chord) is where
|
||||
real boundaries give way to noise. Selecting at the knee **self-calibrates the
|
||||
boundary count** to roughly the true scene count, per film, with no global
|
||||
threshold that would be wrong for every grade.
|
||||
- **Trained on all nine films** for the shipped model. Keeping the low-contrast
|
||||
grades (Café Society, Scarface) in training matters most: on its own training
|
||||
films the shipped model reaches **72.9% macro boundary-F1** (per-film 51–86%),
|
||||
versus **29.8%** for the grayscale baseline on the same films.
|
||||
|
||||
Boundary detection, held out (leave-one-out, ±20 s tolerance — appropriate given
|
||||
~170 s scenes): **44.1% macro F1, versus 29.8% for the grayscale baseline** — the
|
||||
honest generalisation number, each film scored by a detector trained on the other
|
||||
eight. Even the low-contrast grades generalise (Scarface held out 32%, Café Society
|
||||
51%), where the grayscale detector scores 0% and 31%. The absolute number is capped
|
||||
by the narrative-vs-audiovisual mismatch above — many boundaries have no
|
||||
audio-visual signature at all — so the point is the downstream effect, below.
|
||||
|
||||
| boundary-F1 @±20 s | grayscale | learned (LOO) | learned (train-all) |
|
||||
| ------------------ | --------: | ------------: | ------------------: |
|
||||
| macro over 9 films | 29.8% | **44.1%** | 72.9% |
|
||||
|
||||
## The result that matters: actor presence
|
||||
|
||||
Per-second X-Ray presence F1, macro over the nine films, at the shipped presence
|
||||
config. The learned column is **leave-one-out** — each film scored by a detector
|
||||
trained on the other eight:
|
||||
|
||||
| boundary source for flood-fill | presence F1 |
|
||||
| ------------------------------ | ----------- |
|
||||
| track-extent (flood off) | 62.6% |
|
||||
| flood + grayscale cuts | 64.0% |
|
||||
| **flood + learned detector (LOO)** | **74.9%** |
|
||||
|
||||
|
||||
|
||||
**+12.3 points over track-extent, +10.9 over the grayscale-cut flood, and it
|
||||
improves every one of the nine films — under honest leave-one-out.** Per film:
|
||||
|
||||
|
||||
|
||||
| film | track-extent | flood+grayscale | flood+learned (LOO) |
|
||||
| ---- | -----------: | --------------: | ------------------: |
|
||||
| Benny & Joon | 77.3 | 80.2 | 78.2 |
|
||||
| Café Society | 59.1 | 62.2 | 69.8 |
|
||||
| Downton Abbey | 41.0 | 51.8 | **78.6** |
|
||||
| Lord of War | 74.8 | 77.1 | 77.8 |
|
||||
| Lovelace | 70.3 | 74.0 | 78.2 |
|
||||
| The Many Saints of Newark | 37.5 | 43.9 | 53.4 |
|
||||
| Scarface | 62.6 | **40.9** | **74.9** |
|
||||
| Sound of Metal | 75.0 | 78.1 | 86.8 |
|
||||
| Valerian | 65.6 | 67.7 | 76.2 |
|
||||
|
||||
The two headline cases:
|
||||
|
||||
- **Scarface**: the grayscale-cut flood *breaks* it (62.6 → 40.9), because it
|
||||
detects one cut in the whole film. The learned detector — **on a film it never
|
||||
trained on** — takes it to **74.9%**. This is the strongest evidence the
|
||||
detector generalises: it fixes the exact failure that motivated it, held out.
|
||||
- **Downton Abbey**: 41.0 (track-extent) → 51.8 (grayscale) → **78.6** — a
|
||||
+37-point swing on the hardest film.
|
||||
|
||||
Naive flood-fill barely beat doing nothing (64% vs 62%) and broke a film. With a
|
||||
real boundary detector, flood-fill is decisively the right mode.
|
||||
|
||||
### What the frames look like
|
||||
|
||||
`scripts/optimizer/dump_error_frames.py` pulls representative seconds and draws
|
||||
each face box coloured against X-Ray's scene cast: **green** = true positive (a
|
||||
name X-Ray also credits to this scene), **red** = false positive (a name X-Ray
|
||||
does *not* credit here — the real error), **orange** = an unknown detection. Cast
|
||||
X-Ray lists as present but for whom no face was detected — the structural
|
||||
false-negatives a face pipeline can never box — are listed as a **blue** panel.
|
||||
|
||||
|
||||
|
||||
Above: three faces named correctly (green). Below: the face-vs-scene-cast tension
|
||||
made visual — the one visible face is confidently named (here it is a red
|
||||
false-positive, a lead X-Ray did not credit to this exact scene), while six
|
||||
credited cast members are off-camera with no face to detect (blue). This is why
|
||||
recall against X-Ray has a structural ceiling, not a fixable bug.
|
||||
|
||||
|
||||
|
||||
## In the pipeline
|
||||
|
||||
Boundary detection is a **post-EOF step**, like flood-fill itself: the per-film
|
||||
knee needs every peak, so it can only run once the whole film is seen. The
|
||||
`camera_position_change_detector` stamps a per-frame RGB histogram onto each frame;
|
||||
it rides through to the result sink; at end-of-stream the sink runs the detector
|
||||
over the collected histograms plus the movie's audio log-PSD and snaps the
|
||||
presence windows to the result. Enable it with:
|
||||
|
||||
```bash
|
||||
scene_analyze --movie <file> --gallery <gallery.h5> \
|
||||
--scene-xgb-model models/scene_boundary_xgb.json
|
||||
```
|
||||
|
||||
Inference is real XGBoost, built into the binary via CMake (`SAE_SCENE_XGB`); the
|
||||
audio log-PSD uses FFTW + the existing FFmpeg decode. To keep training and
|
||||
inference on one feature implementation, the shipped model is **trained on the
|
||||
C++-extracted features** (`scene_features_dump` → `train_xgb_cpp.py`) rather than a
|
||||
re-implementation in Python — parity by construction. Verified end to end through
|
||||
`scene_analyze` on a movie file and through the Jellyfin work-queue worker.
|
||||
|
||||
## Reproduce
|
||||
|
||||
```bash
|
||||
# per-second audio log-PSD for each film
|
||||
.venv-rocm/bin/python scripts/scene_detector/extract_audio_features.py \
|
||||
--manifest experiments/manifests/films_LVFace_opencv5.json
|
||||
|
||||
# C++ feature matrices (same features training and inference share)
|
||||
build/scene_features_dump <dump.h5> <movie> <features.h5>
|
||||
|
||||
# train the shipped model on all nine films
|
||||
.venv-rocm/bin/python scripts/scene_detector/train_xgb_cpp.py --train-all
|
||||
|
||||
# downstream A/B (track-extent vs flood+grayscale vs flood+learned)
|
||||
scripts/scene_detector/downstream_presence.py
|
||||
```
|
||||
+190
-643
File diff suppressed because it is too large
Load Diff
@@ -1,54 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Fresh LVFace-B embedding dumps (HDF5) for all 9 X-Ray films with the current
|
||||
# feature/opencv5 build, for the flood-fill GA optimisation. Plain front-half
|
||||
# (decode -> campos -> detect -> align -> embed); no scene detection (histogram
|
||||
# cuts is_cut are baked in for flood-fill). Hardware VAAPI decode, no MIGraphX,
|
||||
# no crash. Serial -- ROCm GPU wedges at concurrency>2-3.
|
||||
set -uo pipefail
|
||||
|
||||
REPO="/home/dtourolle/Development/scene-actor-extraction"
|
||||
cd "$REPO"
|
||||
|
||||
ARC="models/LVFace-B_Glint360K.onnx"
|
||||
BIN="build/dump_embeddings"
|
||||
LUT="experiments/file-lut.json"
|
||||
FILMS="experiments/manifests/films.json"
|
||||
OUT="experiments/dumps/LVFace-B_Glint360K_opencv5"
|
||||
mkdir -p "$OUT"
|
||||
|
||||
# Persist MIOpen tuning so SCRFD/ArcFace kernel search is paid once, not per film.
|
||||
export MIOPEN_USER_DB_PATH="$HOME/.cache/miopen-sae"
|
||||
export MIOPEN_FIND_MODE=NORMAL
|
||||
mkdir -p "$MIOPEN_USER_DB_PATH"
|
||||
|
||||
mapfile -t SLUGS < <(python3 -c 'import json;[print(f["slug"]) for f in json.load(open("'"$FILMS"'"))]')
|
||||
|
||||
echo "=== LVFace-B dumps (feature/opencv5) — $(date) ===" | tee "$OUT/dump.log"
|
||||
for slug in "${SLUGS[@]}"; do
|
||||
movie="$(python3 -c 'import json;print(json.load(open("'"$LUT"'"))["'"$slug"'"])')"
|
||||
out="$OUT/dump_${slug}.h5"
|
||||
echo "" | tee -a "$OUT/dump.log"
|
||||
echo ">>> $slug" | tee -a "$OUT/dump.log"
|
||||
if [ -f "$out" ]; then echo " exists, skip" | tee -a "$OUT/dump.log"; continue; fi
|
||||
if [ ! -f "$movie" ]; then echo " SKIP missing: $movie" | tee -a "$OUT/dump.log"; continue; fi
|
||||
# No --max-decode-fps cap: that cap existed only to stop LVFace dump truncation
|
||||
# under PARALLEL load (3 concurrent dumps). This runner is serial, so the cap
|
||||
# just halved throughput for nothing — measured 54s vs 27s per 300s of film,
|
||||
# identical face counts. Uncapped ~9 min/film vs ~18 min capped.
|
||||
"$BIN" --movie "$movie" --arcface "$ARC" --out "$out" --fps 1 \
|
||||
>"$OUT/${slug}.log" 2>&1
|
||||
rc=$?
|
||||
if [ $rc -ne 0 ] || [ ! -f "$out" ]; then
|
||||
echo " DUMP FAILED (rc=$rc) — see ${slug}.log" | tee -a "$OUT/dump.log"
|
||||
else
|
||||
stats=$(python3 -c 'import h5py,sys
|
||||
f=h5py.File(sys.argv[1])
|
||||
n=f["frames/timestamp_sec"].shape[0]
|
||||
faces=f["faces/embedding"].shape[0]
|
||||
cuts=int(f["frames/is_cut"][:].sum())
|
||||
print(f"frames={n} faces={faces} cuts={cuts}")' "$out" 2>/dev/null)
|
||||
echo " ok ($(du -h "$out" | cut -f1), $stats)" | tee -a "$OUT/dump.log"
|
||||
fi
|
||||
done
|
||||
echo "" | tee -a "$OUT/dump.log"
|
||||
echo "=== DONE — $(date) ===" | tee -a "$OUT/dump.log"
|
||||
@@ -1,44 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Regenerate annotated TP/FP/FN frame examples for ALL 9 films against the current
|
||||
# opencv5 pipeline (learned-boundary flood, shipped config). Replays each film with
|
||||
# --raw-out for bboxes, then dump_error_frames.py draws GT-aware boxes
|
||||
# (green TP / red FP / orange unknown / blue FN panel). Frames land in
|
||||
# experiments/dump_review/<slug>/ (regenerable; gitignored). Hand-pick the ones a
|
||||
# doc needs from there.
|
||||
set -uo pipefail
|
||||
REPO="/home/dtourolle/Development/scene-actor-extraction"; cd "$REPO"
|
||||
export MIOPEN_USER_DB_PATH="$HOME/.cache/miopen-sae"
|
||||
GAL=experiments/galleries/gallery_LVFace-B_Glint360K.h5
|
||||
LUT=experiments/file-lut.json
|
||||
CFG=(--prob-threshold 0.485 --ownership-logodds 1.72 --track-extinction-sec 31
|
||||
--track-alpha 0.435 --evidence-rho-max 0.204 --evidence-admit-below 0.784
|
||||
--match-prior 0.433 --expand-band-lo 0.804 --expand-band-hi 0.952
|
||||
--expand-gallery --presence-mode flood)
|
||||
mapfile -t ROWS < <(python3 -c '
|
||||
import json
|
||||
for f in json.load(open("experiments/manifests/films_LVFace_opencv5.json")):
|
||||
print(f["slug"]+"\t"+f["xray"])')
|
||||
SP=/tmp/claude-1000/-home-dtourolle-Development-scene-actor-extraction/c579f8cf-2974-4cbd-be88-afec68dbbf58/scratchpad
|
||||
for row in "${ROWS[@]}"; do
|
||||
slug="${row%%$'\t'*}"; xray="${row#*$'\t'}"
|
||||
movie="$(python3 -c "import json;print(json.load(open('$LUT'))['$slug'])")"
|
||||
echo "=== $slug ==="
|
||||
[ -f "experiments/dump_review/$slug/manifest.json" ] && { echo " exists, skip"; continue; }
|
||||
# replay the learned-boundary (LOO) dump so frames reflect true generalization
|
||||
dump="experiments/dumps/injected_loo/${slug}.h5"
|
||||
[ -f "$dump" ] || dump="experiments/dumps/LVFace-B_Glint360K_opencv5/dump_${slug}.h5"
|
||||
for try in 1 2 3; do
|
||||
timeout 280 python scripts/optimizer/replay.py --dump "$dump" --gallery "$GAL" \
|
||||
--out "$SP/${slug}_pred.json" --raw-out "$SP/${slug}_raw.jsonl" "${CFG[@]}" \
|
||||
>"$SP/${slug}_replay.log" 2>&1 && break
|
||||
echo " replay try $try failed, retrying"
|
||||
done
|
||||
[ -s "$SP/${slug}_raw.jsonl" ] || { echo " no raw output, skip"; continue; }
|
||||
python3 scripts/optimizer/dump_error_frames.py \
|
||||
--pred "$SP/${slug}_pred.json" --raw "$SP/${slug}_raw.jsonl" \
|
||||
--xray "$xray" --movie "$movie" --gallery "$GAL" \
|
||||
--out-dir "experiments/dump_review/$slug" --n-per-bucket 4 \
|
||||
>"$SP/${slug}_frames.log" 2>&1
|
||||
echo " $(grep -oE 'wrote [0-9]+ frames' "$SP/${slug}_frames.log" | tail -1)"
|
||||
done
|
||||
echo "=== DONE ==="
|
||||
@@ -1,60 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Re-benchmark the feature/opencv5 pipeline against Amazon X-Ray, all 9 films, LVFace-B.
|
||||
# Full end-to-end scene_analyze (decode→detect→scene→embed→match→presence) — NOT a replay,
|
||||
# because the framework changed enough that old embedding dumps no longer represent the front half.
|
||||
# Outputs land in experiments/results/xray_opencv5_lvface/ (durable; /tmp gets wiped).
|
||||
set -uo pipefail
|
||||
|
||||
REPO="/home/dtourolle/Development/scene-actor-extraction"
|
||||
cd "$REPO"
|
||||
|
||||
ARC="models/LVFace-B_Glint360K.onnx"
|
||||
GAL="experiments/galleries/gallery_LVFace-B_Glint360K.h5"
|
||||
OUT="experiments/results/xray_opencv5_lvface"
|
||||
mkdir -p "$OUT"
|
||||
|
||||
BIN="build/scene_analyze"
|
||||
LUT="experiments/file-lut.json"
|
||||
FILMS="experiments/manifests/films.json"
|
||||
|
||||
# film slugs and their xray dirs, from films.json
|
||||
mapfile -t ROWS < <(python3 -c '
|
||||
import json
|
||||
for f in json.load(open("'"$FILMS"'")):
|
||||
print(f["slug"] + "\t" + f["xray"])
|
||||
')
|
||||
|
||||
echo "=== X-Ray re-benchmark (feature/opencv5, LVFace-B) — $(date) ===" | tee "$OUT/run.log"
|
||||
|
||||
for row in "${ROWS[@]}"; do
|
||||
slug="${row%%$'\t'*}"
|
||||
xray="${row#*$'\t'}"
|
||||
movie="$(python3 -c 'import json,sys; print(json.load(open("'"$LUT"'"))["'"$slug"'"])')"
|
||||
pred="$OUT/${slug}.json"
|
||||
|
||||
echo "" | tee -a "$OUT/run.log"
|
||||
echo ">>> $slug" | tee -a "$OUT/run.log"
|
||||
if [ ! -f "$movie" ]; then
|
||||
echo " SKIP: movie missing: $movie" | tee -a "$OUT/run.log"
|
||||
continue
|
||||
fi
|
||||
|
||||
# Run the full pipeline (serial — ROCm GPU wedges at concurrency>2-3).
|
||||
"$BIN" --movie "$movie" --arcface "$ARC" --gallery "$GAL" \
|
||||
--output "$pred" >"$OUT/${slug}.pipeline.log" 2>&1
|
||||
rc=$?
|
||||
if [ $rc -ne 0 ] || [ ! -f "$pred" ]; then
|
||||
echo " PIPELINE FAILED (rc=$rc) — see ${slug}.pipeline.log" | tee -a "$OUT/run.log"
|
||||
continue
|
||||
fi
|
||||
echo " pipeline ok" | tee -a "$OUT/run.log"
|
||||
|
||||
# Score against X-Ray, masked to gallery∩GT, 1s grid.
|
||||
python scripts/validation/sample_eval.py \
|
||||
--pred "$pred" --xray "$xray" --gallery "$GAL" --step 1.0 \
|
||||
>"$OUT/${slug}.eval.txt" 2>&1
|
||||
tail -8 "$OUT/${slug}.eval.txt" | tee -a "$OUT/run.log"
|
||||
done
|
||||
|
||||
echo "" | tee -a "$OUT/run.log"
|
||||
echo "=== DONE — $(date) ===" | tee -a "$OUT/run.log"
|
||||
@@ -30,8 +30,6 @@ import sae_embed # before cv2 — see alignment_compare.py
|
||||
import numpy as np
|
||||
import cv2
|
||||
|
||||
import argparse
|
||||
|
||||
ROOT = "/home/dtourolle/Development/Jray-project/scene-actor-extraction/"
|
||||
M = ROOT + "models/"
|
||||
CLIPS = ["5157344", "5157339"]
|
||||
@@ -39,31 +37,16 @@ PROB_THRESHOLD = 0.754
|
||||
GROUP_IOU = 0.55 # detections overlapping this much are the same face
|
||||
MATCH_IOU = 0.35 # tie a detection to the hand-labelled face
|
||||
|
||||
_ap = argparse.ArgumentParser()
|
||||
_ap.add_argument("--detector", default="scrfd_500m_bnkps.onnx",
|
||||
help="detector under models/. SCRFD sizes 500m / 2.5g / 10g come "
|
||||
"from InsightFace's buffalo_sc / buffalo_m / buffalo_l packs")
|
||||
_ap.add_argument("--vote-conf", type=float, default=None,
|
||||
help="confidence floor for the voting pass. Omit to auto-tune "
|
||||
"it to --target-votes")
|
||||
_ap.add_argument("--target-votes", type=int, default=3,
|
||||
help="votes per face to tune --vote-conf towards, so detectors "
|
||||
"are compared at equal redundancy rather than equal settings")
|
||||
_args = _ap.parse_args()
|
||||
|
||||
base_eng = sae_embed.FaceEmbedder(detector_model=M + _args.detector,
|
||||
base_eng = sae_embed.FaceEmbedder(detector_model=M + "scrfd_500m_bnkps.onnx",
|
||||
arcface_model=M + "LVFace-B_Glint360K.onnx",
|
||||
conf=0.5, nms=0.4, max_side=0)
|
||||
|
||||
|
||||
def _make_vote_engine(conf):
|
||||
# Same models, looser suppression: keep the duplicates NMS would have removed.
|
||||
return sae_embed.FaceEmbedder(detector_model=M + _args.detector,
|
||||
# Same models, looser suppression: keep the duplicates NMS would have removed.
|
||||
vote_eng = sae_embed.FaceEmbedder(detector_model=M + "scrfd_500m_bnkps.onnx",
|
||||
arcface_model=M + "LVFace-B_Glint360K.onnx",
|
||||
conf=conf, nms=0.9, max_side=0)
|
||||
|
||||
|
||||
conf=0.3, nms=0.9, max_side=0)
|
||||
cal = sae_embed.gallery_calibration(ROOT + "gallery_lvface.h5")
|
||||
|
||||
|
||||
def iou(a, b):
|
||||
ax, ay, aw, ah = a; bx, by, bw, bh = b
|
||||
x0, y0 = max(ax, bx), max(ay, by)
|
||||
@@ -96,46 +79,6 @@ def vote(dets):
|
||||
return out
|
||||
|
||||
|
||||
def tune_vote_conf(target, sample=6):
|
||||
"""Pick the confidence floor giving ~target detections per face to average.
|
||||
|
||||
A larger SCRFD is more confident and suppresses harder, so at a fixed floor
|
||||
it emits fewer overlapping anchors — median 2 against 500m's 3. Comparing
|
||||
detectors at equal SETTINGS therefore also compares them at unequal
|
||||
redundancy, and the voting arm is handicapped for the bigger models. Tuning
|
||||
each to the same votes-per-face isolates landmark quality from how much
|
||||
there was to average.
|
||||
"""
|
||||
frames = sorted(glob.glob(f"frames/d{CLIPS[0]}_*.png"))[:sample]
|
||||
imgs = [cv2.imread(f) for f in frames]
|
||||
best = (None, None, 1e9)
|
||||
for conf in (0.30, 0.20, 0.12, 0.07, 0.04, 0.02, 0.01):
|
||||
eng = _make_vote_engine(conf)
|
||||
sizes = [n for img in imgs for _, _, _, n in vote(eng.detect(img))]
|
||||
if not sizes:
|
||||
continue
|
||||
med = float(np.median(sizes))
|
||||
if abs(med - target) < best[2]:
|
||||
best = (conf, eng, abs(med - target))
|
||||
print(f"[tune] conf={conf:.2f} -> median {med:.0f} votes/face", file=sys.stderr)
|
||||
if med >= target:
|
||||
break
|
||||
if best[1] is None:
|
||||
print(f"[tune] no confidence floor reached {target} votes/face; "
|
||||
f"falling back to 0.30", file=sys.stderr)
|
||||
return 0.30, _make_vote_engine(0.30)
|
||||
print(f"[tune] chose conf={best[0]:.2f} for ~{target} votes/face", file=sys.stderr)
|
||||
return best[0], best[1]
|
||||
|
||||
|
||||
if _args.vote_conf is not None:
|
||||
VOTE_CONF, vote_eng = _args.vote_conf, _make_vote_engine(_args.vote_conf)
|
||||
else:
|
||||
VOTE_CONF, vote_eng = tune_vote_conf(_args.target_votes)
|
||||
|
||||
cal = sae_embed.gallery_calibration(ROOT + "gallery_lvface.h5")
|
||||
|
||||
|
||||
def collect(clip):
|
||||
lab = {os.path.basename(p): os.path.basename(os.path.dirname(p))
|
||||
for p in glob.glob(f"labelling/{clip}/*/*.jpg")
|
||||
@@ -181,8 +124,7 @@ print(f"[voting] group size: median {np.median(allv):.0f}, "
|
||||
file=sys.stderr)
|
||||
|
||||
GAL, PRB = "5157344", "5157339"
|
||||
print(f"\ndetector={_args.detector} vote_conf={VOTE_CONF:.2f} "
|
||||
f"gallery {GAL} -> probe {PRB}, P>{PROB_THRESHOLD}\n")
|
||||
print(f"\ngallery {GAL} -> probe {PRB}, P>{PROB_THRESHOLD}\n")
|
||||
print(f"{'align':>8}{'person':>8}{'n_gal':>7}{'n_prb':>7}"
|
||||
f"{'within-clip':>13}{'cross-clip':>12}{'hit rate':>10}")
|
||||
summary = {}
|
||||
|
||||
@@ -1,8 +1,6 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Impact of input resolution on cross-source identification.
|
||||
|
||||
TRACES: VR-013 | PR-002
|
||||
|
||||
Gallery is built from one clip at NATIVE resolution. Probes come from the other
|
||||
clip with the WHOLE FRAME downscaled before it reaches the detector, so
|
||||
detection and landmark regression degrade together with the pixels. That is the
|
||||
|
||||
@@ -1,14 +1,6 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Integrity check on the labelled set, before it is used as ground truth.
|
||||
|
||||
TRACES: VR-013 | PR-002
|
||||
|
||||
VR-013's ground truth is hand-sorted rather than propagated by embedding
|
||||
similarity, because propagation would keep only the faces the embedder already
|
||||
gets right and silently drop the ones the sweep exists to find. This script is
|
||||
what makes that claim checkable, so it is part of the requirement rather than a
|
||||
helper of it.
|
||||
|
||||
Checks, loudest failure first:
|
||||
|
||||
1. INDEX INTEGRITY. Each crop's embedding is taken by re-detecting its source
|
||||
|
||||
Vendored
+1
-1
Submodule external/KPN updated: 771b9f8593...6595e6e925
+6
-9
@@ -35,17 +35,14 @@ extra_css:
|
||||
nav:
|
||||
- Home: index.md
|
||||
- How We Score Against X-Ray: methodology.md
|
||||
- Learned Scene-Boundary Detector: scene-boundary-detector.md
|
||||
- Benchmark — SuperHero: benchmark.md
|
||||
- Findings:
|
||||
- Best Model: best-model.md
|
||||
- Gallery Scope (Full vs. Limited): gallery-scope.md
|
||||
- Pose Expansion: pose-expansion.md
|
||||
- Quality Knee (Blur and Size): quality-knee.md
|
||||
- LVFace Deep Dive: lvface-deep-dive.md
|
||||
- Full Experiment Log: model-bakeoff.md
|
||||
- Service Conversion (proposal): service-conversion.md
|
||||
- Archive (July 2026):
|
||||
- How We Scored (July): methodology-2026-07.md
|
||||
- Best Model: best-model-2026-07.md
|
||||
- Gallery Scope (Full vs. Limited): gallery-scope-2026-07.md
|
||||
- Pose Expansion: pose-expansion-2026-07.md
|
||||
- LVFace Deep Dive: lvface-deep-dive-2026-07.md
|
||||
- Full Experiment Log (July): model-bakeoff-2026-07.md
|
||||
|
||||
markdown_extensions:
|
||||
- admonition
|
||||
|
||||
BIN
Binary file not shown.
File diff suppressed because one or more lines are too long
BIN
Binary file not shown.
Binary file not shown.
@@ -10,7 +10,6 @@
|
||||
# scripts/artifacts/pull_artifacts.sh galleries [version]
|
||||
# scripts/artifacts/pull_artifacts.sh montage-frames <film-slug> [version]
|
||||
# scripts/artifacts/pull_artifacts.sh experiment-data [version]
|
||||
# scripts/artifacts/pull_artifacts.sh replay-fixtures [version]
|
||||
# scripts/artifacts/pull_artifacts.sh report-highlights <name> [version]
|
||||
# scripts/artifacts/pull_artifacts.sh xsource [version]
|
||||
# version defaults to "latest" (newest uploaded version, by created_at).
|
||||
@@ -43,22 +42,6 @@ print(matches[-1]['version'])
|
||||
"
|
||||
}
|
||||
|
||||
pull_replay_fixtures() {
|
||||
local version="$1"
|
||||
local dest="${REPO_ROOT}/tests/fixtures/dumps"
|
||||
mkdir -p "$dest"
|
||||
echo "=== replay-fixtures (version ${version}) ==="
|
||||
local tmp; tmp="$(mktemp -d)"
|
||||
if curl -sf "${DL_BASE}/generic/replay-fixtures/${version}/replay-fixtures.zip" \
|
||||
-o "${tmp}/f.zip"; then
|
||||
unzip -qo "${tmp}/f.zip" -d "$dest"
|
||||
echo " restored: $(ls "$dest" | wc -l) files into tests/fixtures/dumps/"
|
||||
else
|
||||
echo " [warn] replay-fixtures.zip not found at version ${version}" >&2
|
||||
fi
|
||||
rm -rf "$tmp"
|
||||
}
|
||||
|
||||
pull_galleries() {
|
||||
local version="$1"
|
||||
local dest="${REPO_ROOT}/experiments/galleries"
|
||||
@@ -198,13 +181,8 @@ case "$TARGET" in
|
||||
[ "$VERSION" = "latest" ] && VERSION="$(resolve_latest_version xsource)"
|
||||
pull_xsource "$VERSION"
|
||||
;;
|
||||
replay-fixtures)
|
||||
VERSION="${2:-latest}"
|
||||
[ "$VERSION" = "latest" ] && VERSION="$(resolve_latest_version replay-fixtures)"
|
||||
pull_replay_fixtures "$VERSION"
|
||||
;;
|
||||
*)
|
||||
echo "unknown target: $TARGET (expected galleries, montage-frames, experiment-data, report-highlights, xsource, or replay-fixtures)" >&2
|
||||
echo "unknown target: $TARGET (expected galleries, montage-frames, experiment-data, report-highlights, or xsource)" >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
@@ -13,12 +13,10 @@
|
||||
# scripts/artifacts/push_artifacts.sh experiment-data
|
||||
# scripts/artifacts/push_artifacts.sh report-highlights
|
||||
# scripts/artifacts/push_artifacts.sh xsource
|
||||
# scripts/artifacts/push_artifacts.sh replay-fixtures
|
||||
# scripts/artifacts/push_artifacts.sh galleries montage-frames experiment-data report-highlights
|
||||
#
|
||||
# Package layout (owner=dtourolle, repo=scene-actor-extraction):
|
||||
# generic/galleries/<version>/gallery_<model>.h5 (one file per model)
|
||||
# generic/replay-fixtures/<version>/replay-fixtures.zip (T2 dumps + their gallery)
|
||||
# generic/montage-frames/<version>/<film-slug>.zip (zipped per-film frames)
|
||||
# generic/experiment-data/<version>/experiment-data.zip (manifests/trajectories/results)
|
||||
# generic/report-highlights/<version>/<name>.jpg (individual, hand-picked
|
||||
@@ -52,29 +50,6 @@ upload() {
|
||||
-o /dev/null -w " HTTP %{http_code}\n"
|
||||
}
|
||||
|
||||
push_replay_fixtures() {
|
||||
echo "=== replay-fixtures (version ${VERSION}) ==="
|
||||
# T2 replay fixtures: per-frame detections, landmarks and embeddings dumped
|
||||
# from a real run, so the tracker and identity stages can be replayed on CPU
|
||||
# with no GPU, no models and no film. Too large for git (superhero.h5 alone
|
||||
# is ~9 MB) and regenerating them needs the film plus a GPU, which CI has
|
||||
# neither of — so they ship as artifacts and CI pulls them.
|
||||
#
|
||||
# The gallery travels with them: a dump replays against the gallery it was
|
||||
# produced with, and pairing a dump with a different gallery silently
|
||||
# changes every identity decision in it.
|
||||
local dir="${REPO_ROOT}/tests/fixtures/dumps"
|
||||
if [ ! -d "$dir" ]; then
|
||||
echo " no tests/fixtures/dumps dir, skipping" >&2
|
||||
return
|
||||
fi
|
||||
local tmp
|
||||
tmp="$(mktemp -d)"
|
||||
( cd "$dir" && zip -qr "$tmp/replay-fixtures.zip" . )
|
||||
upload "replay-fixtures" "replay-fixtures.zip" "$tmp/replay-fixtures.zip"
|
||||
rm -rf "$tmp"
|
||||
}
|
||||
|
||||
push_galleries() {
|
||||
echo "=== galleries (version ${VERSION}) ==="
|
||||
local dir="${REPO_ROOT}/experiments/galleries"
|
||||
@@ -174,8 +149,7 @@ for target in "$@"; do
|
||||
experiment-data) push_experiment_data ;;
|
||||
report-highlights) push_report_highlights ;;
|
||||
xsource) push_xsource ;;
|
||||
replay-fixtures) push_replay_fixtures ;;
|
||||
*) echo "unknown target: $target (expected galleries, montage-frames, experiment-data, report-highlights, xsource, or replay-fixtures)" >&2; exit 1 ;;
|
||||
*) echo "unknown target: $target (expected galleries, montage-frames, experiment-data, report-highlights, or xsource)" >&2; exit 1 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
|
||||
@@ -1,118 +0,0 @@
|
||||
#!/bin/bash
|
||||
# build_builder_image.sh — build and publish the DP-007 CI builder image to the
|
||||
# Gitea container registry.
|
||||
#
|
||||
# TRACES: DP-007 | PR-004
|
||||
#
|
||||
# Usage:
|
||||
# scripts/ci/build_builder_image.sh # build only, tag v1
|
||||
# scripts/ci/build_builder_image.sh --push # build and push
|
||||
# scripts/ci/build_builder_image.sh --tag v2 --push # bump the pinned tag
|
||||
# scripts/ci/build_builder_image.sh --no-cache # force a clean rebuild
|
||||
#
|
||||
# The tag is the contract with CI. .gitea/workflows/unit-tests.yml names an
|
||||
# explicit tag in its `container:` block and never `latest`, so that rebuilding
|
||||
# the image cannot silently change what a previous green build meant. Bumping
|
||||
# the dependency set means bumping the tag AND editing the workflow — the two
|
||||
# edits landing in the same commit is the point, not an inconvenience.
|
||||
#
|
||||
# Registry auth: this script does not log in. Do it once, out of band:
|
||||
# docker login gitea.tourolle.paris
|
||||
# The CI host is already authenticated this way (its cached credentials in
|
||||
# ~/.docker/config.json are what the kpnpp-builder push relies on), so a
|
||||
# workflow that calls this script needs no secret plumbing.
|
||||
set -euo pipefail
|
||||
|
||||
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
|
||||
|
||||
REGISTRY="gitea.tourolle.paris"
|
||||
OWNER="dtourolle"
|
||||
IMAGE="sae-builder-cpu"
|
||||
DOCKERFILE="Dockerfile.builder-cpu"
|
||||
|
||||
# The tag CI pins to today. Keep this in step with the `container.image` line in
|
||||
# .gitea/workflows/unit-tests.yml; the workflow asserts at run time that the
|
||||
# image it landed in reports this same version, so a drift shows up as a failed
|
||||
# job rather than as a build against the wrong toolchain.
|
||||
TAG="v1"
|
||||
|
||||
PUSH=0
|
||||
EXTRA_ARGS=()
|
||||
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--push) PUSH=1 ;;
|
||||
--tag) TAG="${2:?--tag needs a value}"; shift ;;
|
||||
--no-cache) EXTRA_ARGS+=(--no-cache) ;;
|
||||
-h|--help) sed -n '2,30p' "${BASH_SOURCE[0]}"; exit 0 ;;
|
||||
*) echo "error: unknown argument '$1'" >&2; exit 2 ;;
|
||||
esac
|
||||
shift
|
||||
done
|
||||
|
||||
if [ "$TAG" = "latest" ]; then
|
||||
echo "error: refusing to build the tag 'latest'." >&2
|
||||
echo "DP-007 requires CI to pin an immutable tag. A moving 'latest' means a" >&2
|
||||
echo "rebuild retroactively changes what every earlier green build proved." >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
REF="${REGISTRY}/${OWNER}/${IMAGE}:${TAG}"
|
||||
# A second tag carrying the commit that produced the image. The workflow pins
|
||||
# the human-readable tag; this one is the audit trail — given any image you can
|
||||
# recover the Dockerfile that built it.
|
||||
SHA="$(git -C "$REPO_ROOT" rev-parse --short HEAD)"
|
||||
REF_SHA="${REGISTRY}/${OWNER}/${IMAGE}:${TAG}-${SHA}"
|
||||
|
||||
# The Dockerfile COPYs nothing from the repository on purpose (see its closing
|
||||
# comment), so the build context is an empty directory rather than the repo
|
||||
# root. Sending ~1 GB of models, fixtures and experiment data to the daemon for
|
||||
# a build that reads none of it is pure latency.
|
||||
CONTEXT="$(mktemp -d)"
|
||||
trap 'rm -rf "$CONTEXT"' EXIT
|
||||
|
||||
echo "=== building ${REF}"
|
||||
echo " dockerfile: ${REPO_ROOT}/${DOCKERFILE}"
|
||||
echo " context: (empty — the image embeds no repository content)"
|
||||
echo
|
||||
echo " Expect this to take a while: OpenCV 5 is compiled from source because"
|
||||
echo " no Debian release ships it. That cost is paid once per image, which is"
|
||||
echo " the entire reason DP-007 asks for a prebuilt image instead of"
|
||||
echo " installing dependencies inside each CI run."
|
||||
echo
|
||||
|
||||
docker build \
|
||||
"${EXTRA_ARGS[@]}" \
|
||||
--build-arg "IMAGE_TAG=${TAG}" \
|
||||
-f "${REPO_ROOT}/${DOCKERFILE}" \
|
||||
-t "${REF}" \
|
||||
-t "${REF_SHA}" \
|
||||
"${CONTEXT}"
|
||||
|
||||
echo
|
||||
echo "=== built"
|
||||
docker image inspect "${REF}" --format ' {{.RepoTags}} {{.Size}} bytes'
|
||||
docker run --rm "${REF}" sh -c 'echo " SAE_BUILDER=$SAE_BUILDER version=$SAE_BUILDER_VERSION ort=$SAE_ORT_VERSION opencv=$SAE_OPENCV_VERSION"'
|
||||
|
||||
if [ "$PUSH" -eq 0 ]; then
|
||||
echo
|
||||
echo "Not pushed. Re-run with --push, or push by hand:"
|
||||
echo " docker push ${REF}"
|
||||
echo " docker push ${REF_SHA}"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo
|
||||
echo "=== pushing"
|
||||
# No `latest` tag is pushed, by design. Publishing one invites a workflow to use
|
||||
# it, and DP-007 exists to prevent exactly that.
|
||||
docker push "${REF}"
|
||||
docker push "${REF_SHA}"
|
||||
|
||||
echo
|
||||
echo "=== published ${REF}"
|
||||
echo "If this was a dependency-set change, bump the tag in"
|
||||
echo " .gitea/workflows/unit-tests.yml (container.image)"
|
||||
echo " scripts/ci/build_builder_image.sh (TAG, above)"
|
||||
echo "in the same commit, so no run can build against an image the repository"
|
||||
echo "does not describe."
|
||||
@@ -1,225 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Enforce the AR-024 invariant: never a raw cosine, always the calibration.
|
||||
|
||||
TRACES: AR-024 | SR-002
|
||||
|
||||
docs/requirements.md gives AR-024's verification tier as "Static check -- no
|
||||
bare cosine outside a tagged EXCEPTION | Grep-based; this is the invariant's
|
||||
enforcement". This is that check. Until it existed the invariant was enforced
|
||||
by reading, and reading missed a live violation: the identity matcher's
|
||||
no-calibration fallback thresholded raw cosine distance and fed `max(0, cosine)`
|
||||
into the Bayesian accumulation as though it were a posterior.
|
||||
|
||||
WHAT IT CHECKS, precisely, because a static check that overclaims its reach is
|
||||
worse than one with a stated scope:
|
||||
|
||||
Every call to `cosine_similarity(...)` in C++ source must either
|
||||
|
||||
(a) have its result consumed immediately by a calibration -- the call is
|
||||
textually wrapped in `cal_(...)`, `calibrate_(...)`, `.probability(...)`
|
||||
or similar; or
|
||||
(b) sit under an exception comment -- the token is `EXCEPTION:` followed by
|
||||
`AR-024` and a reason -- within EXCEPTION_SCOPE_LINES above it.
|
||||
|
||||
Note that this file deliberately never spells that token out. The traceability
|
||||
extractor scans scripts/ as source, so prose here describing the tag would be
|
||||
counted as recorded exceptions; four of them were, until this was noticed. The
|
||||
same trap the shared config warns about for the vendored parser tests.
|
||||
|
||||
Anything else is a defect, per CLAUDE.md: "treat any bare cosine comparison in
|
||||
the code as a defect to be fixed".
|
||||
|
||||
WHAT IT DOES NOT CHECK, and why you should not read a pass as more than it is:
|
||||
|
||||
- It cannot follow a cosine through a variable across statements. A file that
|
||||
stores `float s = cosine_similarity(a, b);` and compares `s` three lines
|
||||
later is not caught. The codebase does not currently do this, and this check
|
||||
exists partly to keep it that way, but it is a convention backed by review,
|
||||
not by the tool.
|
||||
- It says nothing about GEMM output. The similarity engine returns a whole
|
||||
matrix of cosines and the matcher reads them directly; that path is correct
|
||||
by inspection (every value goes through `cal_.probability`) and is not
|
||||
verified here.
|
||||
- A retired constant reintroduced under a new name is invisible to it.
|
||||
|
||||
Exit status is 0 when clean, 1 when a violation is found, 2 on a usage error.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import pathlib
|
||||
import re
|
||||
import sys
|
||||
|
||||
# How far above a use an exception tag may sit and still cover it.
|
||||
# Generous, because the house style puts a paragraph of reasoning between the
|
||||
# tag and the code -- but bounded, so a tag cannot silently cover a whole file.
|
||||
EXCEPTION_SCOPE_LINES = 25
|
||||
|
||||
CPP_SUFFIXES = {".h", ".hpp", ".hxx", ".cc", ".cpp", ".cxx", ".cu", ".cuh"}
|
||||
|
||||
# src only, deliberately. The invariant governs what the PIPELINE decides --
|
||||
# CLAUDE.md's rule is "tag the unit that decides" -- whereas a test legitimately
|
||||
# asserts properties of the metric space itself (that a vector's cosine with
|
||||
# itself is 1, that the annex ended up holding the spoke it should have). Those
|
||||
# are measurements of the code under test, not decisions shipped to a user, and
|
||||
# sweeping them in would produce a wall of blanket EXCEPTION tags that would
|
||||
# devalue the tag everywhere else. Pass --source-root tests to scan them anyway.
|
||||
DEFAULT_ROOTS = ["src"]
|
||||
|
||||
# Directories that are never this repo's code.
|
||||
EXCLUDE_DIRS = {
|
||||
"build", "build-ort", "external", "vendor", "__pycache__",
|
||||
".git", "node_modules", "models",
|
||||
}
|
||||
|
||||
COSINE_CALL = re.compile(r"\bcosine_similarity\s*\(")
|
||||
|
||||
# The result is immediately handed to a calibration. Matches the house shapes:
|
||||
# cal_(cosine_similarity(a, b))
|
||||
# calibrate_(cosine_similarity(a, b))
|
||||
# same_person(cosine_similarity(a, b))
|
||||
# cal_.probability(cosine_similarity(a, b))
|
||||
CALIBRATED = re.compile(
|
||||
r"(?:\b(?:cal_|cal|calibrate_|calibrate|same_person|same_person_probability)"
|
||||
r"\s*(?:\.\s*probability\s*)?\(\s*|\.\s*probability\s*\(\s*)"
|
||||
r"cosine_similarity\s*\("
|
||||
)
|
||||
|
||||
EXCEPTION_TAG = re.compile(r"EXCEPT" + r"ION:\s*AR-" + r"024\b(.*)")
|
||||
|
||||
# The function's own definition is not a use of it.
|
||||
DEFINITION = re.compile(r"^\s*(?:inline\s+|static\s+|constexpr\s+)*float\s+"
|
||||
r"cosine_similarity\s*\(")
|
||||
|
||||
# The house style wraps long calls across lines:
|
||||
# const float p = calibrate_(
|
||||
# cosine_similarity(a, b));
|
||||
# so the calibration and the call it guards are not always on one line. Joining
|
||||
# a small window before testing is what makes this check usable on real code
|
||||
# rather than a generator of false positives that trains people to ignore it.
|
||||
JOIN_LOOKBEHIND = 2
|
||||
|
||||
|
||||
def iter_sources(root: pathlib.Path, roots):
|
||||
for rel in roots:
|
||||
base = root / rel
|
||||
if not base.exists():
|
||||
continue
|
||||
for p in sorted(base.rglob("*")):
|
||||
if p.suffix.lower() not in CPP_SUFFIXES:
|
||||
continue
|
||||
if any(part in EXCLUDE_DIRS for part in p.relative_to(root).parts):
|
||||
continue
|
||||
yield p
|
||||
|
||||
|
||||
def covering_exception(lines, idx):
|
||||
"""Return the reason text of an exception tag covering line `idx`."""
|
||||
lo = max(0, idx - EXCEPTION_SCOPE_LINES)
|
||||
for j in range(idx, lo - 1, -1):
|
||||
m = EXCEPTION_TAG.search(lines[j])
|
||||
if m:
|
||||
return m.group(1).strip(" -—*/") or "(no reason given)"
|
||||
return None
|
||||
|
||||
|
||||
def check_file(path: pathlib.Path, root: pathlib.Path):
|
||||
violations, exceptions = [], []
|
||||
try:
|
||||
lines = path.read_text(encoding="utf-8", errors="replace").splitlines()
|
||||
except OSError as e:
|
||||
print(f"error: cannot read {path}: {e}", file=sys.stderr)
|
||||
return violations, exceptions
|
||||
|
||||
rel = path.relative_to(root)
|
||||
for i, line in enumerate(lines):
|
||||
if not COSINE_CALL.search(line):
|
||||
continue
|
||||
# A comment mentioning the function is prose, not a use.
|
||||
stripped = line.lstrip()
|
||||
if stripped.startswith(("//", "///", "*", "/*")):
|
||||
continue
|
||||
if DEFINITION.match(line):
|
||||
continue
|
||||
# Join a small window so a call wrapped across lines is still seen as
|
||||
# calibrated. Whitespace is collapsed so the join reads as one statement.
|
||||
window = " ".join(
|
||||
lines[max(0, i - JOIN_LOOKBEHIND):i + 1]
|
||||
)
|
||||
window = re.sub(r"\s+", " ", window)
|
||||
if CALIBRATED.search(window):
|
||||
continue
|
||||
reason = covering_exception(lines, i)
|
||||
if reason:
|
||||
exceptions.append((rel, i + 1, line.strip(), reason))
|
||||
else:
|
||||
violations.append((rel, i + 1, line.strip()))
|
||||
return violations, exceptions
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__,
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument("--root", default=None,
|
||||
help="repository root (default: the script's ../..)")
|
||||
ap.add_argument("--source-root", action="append", default=None,
|
||||
help="directory to scan; repeatable (default: src, tests)")
|
||||
args = ap.parse_args()
|
||||
|
||||
root = pathlib.Path(args.root) if args.root \
|
||||
else pathlib.Path(__file__).resolve().parents[2]
|
||||
roots = args.source_root or DEFAULT_ROOTS
|
||||
|
||||
if not root.is_dir():
|
||||
print(f"error: root {root} is not a directory", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
all_violations, all_exceptions, n_files = [], [], 0
|
||||
for p in iter_sources(root, roots):
|
||||
n_files += 1
|
||||
v, e = check_file(p, root)
|
||||
all_violations += v
|
||||
all_exceptions += e
|
||||
|
||||
if n_files == 0:
|
||||
# A scan that found nothing to read is a misconfiguration reporting a
|
||||
# pass, which is the failure mode the traceability gate also guards.
|
||||
print(f"error: scanned 0 source files under {root} ({', '.join(roots)})",
|
||||
file=sys.stderr)
|
||||
return 2
|
||||
|
||||
print("AR-024 — always the calibrated probability, never a raw cosine")
|
||||
print("=" * 72)
|
||||
print(f"Repo root : {root}")
|
||||
print(f"Files scanned : {n_files} ({', '.join(roots)})")
|
||||
print(f"Recorded excs. : {len(all_exceptions)}")
|
||||
print(f"Violations : {len(all_violations)}")
|
||||
|
||||
if all_exceptions:
|
||||
print("\nRecorded exceptions (allowed, and each one is a claim to re-read):")
|
||||
for rel, ln, src, reason in all_exceptions:
|
||||
print(f" {rel}:{ln} {reason}")
|
||||
print(f" {src}")
|
||||
|
||||
if all_violations:
|
||||
print("\nVIOLATIONS — a bare cosine with no recorded exception:")
|
||||
for rel, ln, src in all_violations:
|
||||
print(f" {rel}:{ln}")
|
||||
print(f" {src}")
|
||||
print("\nEvery similarity is converted through the sigmoid calibration")
|
||||
print("before it is used, compared, or thresholded. A raw cosine means")
|
||||
print("something different for every model, gallery and face size, and")
|
||||
print("it cannot be combined with anything else.")
|
||||
print("\nEither route it through the calibration, or, if the use is")
|
||||
print("genuinely about the metric space rather than about a decision,")
|
||||
print("record it:")
|
||||
print(" // " + "EXCEPT" + "ION: AR-" + "024 <why this one is not a decision>")
|
||||
print("and add a row to CLAUDE.md's agreed-exceptions table.")
|
||||
return 1
|
||||
|
||||
print("\nOK: no bare cosine outside a recorded exception.")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -79,9 +79,8 @@ def main():
|
||||
"--dump", str(dump), "--gallery", str(gallery),
|
||||
"--out", str(pred_path),
|
||||
"--prob-threshold", str(cfg["prob_threshold"]),
|
||||
# anneal_sec and extinction_sec are both gone: presence is
|
||||
# the registry's, built from track extents (AR-012/AR-013), and
|
||||
# replay.py no longer windows anything itself (VR-011).
|
||||
"--anneal-sec", str(cfg["anneal_sec"]),
|
||||
"--extinction-sec", str(cfg["extinction_sec"]),
|
||||
"--expand-gallery",
|
||||
]
|
||||
print(f"RUN {model}/{film['slug']}...", file=sys.stderr)
|
||||
|
||||
@@ -1,98 +0,0 @@
|
||||
#!/bin/bash
|
||||
# fetch_dvu.sh — pull one film's character mugshots and presence annotations from
|
||||
# the NIST TRECVID Deep Video Understanding development set.
|
||||
#
|
||||
# The DVU dev set is the reason Road to Bali is our benchmark film: it ships
|
||||
# 5-7 face crops per *character*, cut from the film itself, alongside
|
||||
# scene-scoped presence annotations. That matches SR-002 directly — presence is
|
||||
# per scene, not per frame — and it keeps ground truth in character space, so
|
||||
# scoring needs no actor->character mapping.
|
||||
#
|
||||
# This exists as a script, rather than as ad hoc commands, because the first
|
||||
# copy of this data lived in a temp directory and was lost to a /tmp wipe,
|
||||
# taking the working gallery with it.
|
||||
#
|
||||
# 14 films are asserted Creative Commons and need no data agreement (only the
|
||||
# 5 KinoLorber test films are gated).
|
||||
#
|
||||
# Usage:
|
||||
# scripts/fetch_dvu.sh [film] [dest]
|
||||
# film default Road_To_Bali
|
||||
# dest default ./dvu
|
||||
set -euo pipefail
|
||||
|
||||
BASE="https://www-nlpir.nist.gov/projects/trecvid/dvu/dvu.development.dataset"
|
||||
FILM="${1:-Road_To_Bali}"
|
||||
DEST="${2:-dvu}"
|
||||
|
||||
mkdir -p "$DEST/images" "$DEST/scenes"
|
||||
|
||||
echo "[dvu] $FILM -> $DEST"
|
||||
|
||||
# Scene segmentation: start/end as HH:MM:SS. Note valkaama.csv line 38 carries a
|
||||
# shift-key typo (01:!4:00) — parse defensively if you extend this to that film.
|
||||
echo "[dvu] scene segmentation"
|
||||
curl -fsSL "$BASE/scene.segmentation.reference/${FILM}.csv" \
|
||||
-o "$DEST/${FILM}.csv" || echo " (missing: ${FILM}.csv)"
|
||||
|
||||
# Entity types: which entities are Person vs Location/Concept. Only Person rows
|
||||
# become gallery identities — the images/ directory also holds Location and
|
||||
# Concept crops (bedroom, boat, ...), which must not enter a face gallery.
|
||||
#
|
||||
# Directory and file naming are inconsistent with the film slug used elsewhere:
|
||||
# the folder is Road_to_Bali (lowercase "to") while the entity file is
|
||||
# RoadToBali.entity.types.txt. Both are derived here rather than assumed.
|
||||
# NIST is inconsistent across all three axes, and not by a rule worth deriving:
|
||||
# Road to Bali is Road_To_Bali.csv / Road_to_Bali/ / RoadToBali.entity.types.txt,
|
||||
# while SuperHero is SuperHero.csv / superHero/ / superhero.entity.types.txt.
|
||||
# Defaults cover the Bali shape; override per film rather than guessing.
|
||||
# KG_DIR=superHero KG_FILE=superhero scripts/fetch_dvu.sh SuperHero dvu-hero
|
||||
KG_DIR="${KG_DIR:-${FILM//_To_/_to_}}"
|
||||
KG_FILE="${KG_FILE:-$(echo "$FILM" | sed -E 's/_([a-z])/\U\1/g; s/_//g')}"
|
||||
|
||||
echo "[dvu] entity types ($KG_DIR/$KG_FILE)"
|
||||
curl -fsSL "$BASE/movie_knowledge_graph/${KG_DIR}/${KG_FILE}.entity.types.txt" \
|
||||
-o "$DEST/${FILM}.entity.types.txt" || echo " (missing: entity types)"
|
||||
|
||||
# Character face crops. Names are discovered from the directory listing rather
|
||||
# than probed as <Character>_N, since the crop count varies per character and
|
||||
# the listing is authoritative.
|
||||
echo "[dvu] character mugshots"
|
||||
PERSONS="$DEST/persons.txt"
|
||||
if [ -f "$DEST/${FILM}.entity.types.txt" ]; then
|
||||
grep -iE "person" "$DEST/${FILM}.entity.types.txt" \
|
||||
| sed -E 's/[[:space:]]*[:,].*$//' | tr -d '\r' \
|
||||
| awk '{print tolower($1)}' | sort -u > "$PERSONS"
|
||||
fi
|
||||
|
||||
curl -fsSL "$BASE/movie_knowledge_graph/${KG_DIR}/images/" 2>/dev/null \
|
||||
| grep -oE 'href="[^"?/][^"]*\.png"' | sed -E 's/href="//; s/"//' | sort -u \
|
||||
> "$DEST/all_images.txt"
|
||||
|
||||
while read -r img; do
|
||||
[ -z "$img" ] && continue
|
||||
# Strip the trailing _N to recover the entity name.
|
||||
who="$(echo "$img" | sed -E 's/_[0-9]+\.png$//' | awk '{print tolower($0)}')"
|
||||
if [ -s "$PERSONS" ] && ! grep -qx "$who" "$PERSONS"; then
|
||||
continue # Location/Concept crop, not a face
|
||||
fi
|
||||
curl -fsSL "$BASE/movie_knowledge_graph/${KG_DIR}/images/${img}" \
|
||||
-o "$DEST/images/${img}" 2>/dev/null || rm -f "$DEST/images/${img}"
|
||||
done < "$DEST/all_images.txt"
|
||||
|
||||
# Per-scene knowledge graphs. A Person->Location edge means that person was
|
||||
# present for the whole scene. Some of these contain a stray ", ," that breaks
|
||||
# strict JSON parsers.
|
||||
echo "[dvu] scene graphs"
|
||||
for n in $(seq 1 60); do
|
||||
curl -fsSL "$BASE/scenes_knowledge_graphs/${FILM//_/ }-${n}.json" \
|
||||
-o "$DEST/scenes/${FILM}-${n}.json" 2>/dev/null \
|
||||
|| curl -fsSL "$BASE/scenes_knowledge_graphs/${FILM}-${n}.json" \
|
||||
-o "$DEST/scenes/${FILM}-${n}.json" 2>/dev/null \
|
||||
|| rm -f "$DEST/scenes/${FILM}-${n}.json"
|
||||
done
|
||||
|
||||
echo "[dvu] done:"
|
||||
echo " mugshots: $(ls "$DEST/images" 2>/dev/null | wc -l)"
|
||||
echo " scenes: $(ls "$DEST/scenes" 2>/dev/null | wc -l)"
|
||||
echo " csv: $([ -f "$DEST/${FILM}.csv" ] && echo yes || echo no)"
|
||||
@@ -77,8 +77,7 @@ def main():
|
||||
if missing > 0:
|
||||
print(f"[warn] {missing} cast member(s) not present in gallery (not yet embedded)", file=sys.stderr)
|
||||
|
||||
# TRACES: GR-004 | SR-001
|
||||
# a filtered gallery holds the SAME vectors as its
|
||||
# TRACES: GR-004 | SR-001 — a filtered gallery holds the SAME vectors as its
|
||||
# source, so it inherits the source's binding. Dropping the stamp here would
|
||||
# silently launder a stamped gallery into an unstamped one.
|
||||
save_gallery_hdf5({"actors": actors}, Path(args.output),
|
||||
|
||||
@@ -19,16 +19,13 @@
|
||||
# on a full channel (AR-004). Before that fix the same command produced
|
||||
# different dumps run to run, since what got dropped depended on timing.
|
||||
#
|
||||
# Source: hero/ — SuperHero, from the TRECVID DVU development set. Chosen over
|
||||
# SuperHero on face scale: Bali reference crops had a median detected face of
|
||||
# 27 px against a 69 px maximum, so every reference was upscaled far past what
|
||||
# the embedder was trained for. SuperHero is 69 px median, 241 px max. That matters: derived
|
||||
# Source: bali/ — Road to Bali (1952), public domain. That matters: derived
|
||||
# fixtures can be committed, where anything cut from a copyrighted title could
|
||||
# not live in the repository at all.
|
||||
set -euo pipefail
|
||||
|
||||
REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
CLIPS="${CLIPS:-$REPO/../hero}"
|
||||
CLIPS="${CLIPS:-$REPO/../bali}"
|
||||
GALLERY="${GALLERY:-$REPO/gallery_lvface.h5}"
|
||||
BIN="${BIN:-$REPO/build/scene_analyze}"
|
||||
OUT="$REPO/tests/fixtures/dumps"
|
||||
@@ -36,12 +33,8 @@ OUT="$REPO/tests/fixtures/dumps"
|
||||
# Pinned. Changing either invalidates every committed fixture.
|
||||
# fps 5 — 1 fps over a 77 s clip is 77 frames, too thin to exercise an
|
||||
# extinction window measured in tens of seconds.
|
||||
# min-face — 32 px. This is a *fixture* setting, deliberately below AR-002's
|
||||
# production floor of 40 px (VR-013, measured end to end): the
|
||||
# corpus is 480x360, where faces run 40-80 px, so pinning at 40
|
||||
# would thin the dumps for reasons unrelated to what they test.
|
||||
# 32 px is where VR-005 still shows 98.1% TPI, so the faces kept
|
||||
# are identifiable; it is not the threshold the pipeline ships.
|
||||
# min-face — 32 px, the VR-005 measured floor (98.1% TPI). The corpus is
|
||||
# 480x360, so a stricter value would reject most faces present.
|
||||
FPS=5
|
||||
MIN_FACE_PX=32
|
||||
|
||||
@@ -51,12 +44,12 @@ MIN_FACE_PX=32
|
||||
|
||||
mkdir -p "$OUT"
|
||||
|
||||
for clip in "$CLIPS"/SuperHero-*.webm; do
|
||||
for clip in "$CLIPS"/Road_To_Bali-*.webm; do
|
||||
n="$(basename "$clip" .webm)"; n="${n##*-}"
|
||||
echo "── superhero_$n"
|
||||
echo "── bali_$n"
|
||||
"$BIN" --movie "$clip" --gallery "$GALLERY" \
|
||||
--fps "$FPS" --min-face-px "$MIN_FACE_PX" \
|
||||
--dump-embeddings "$OUT/superhero_$n.h5" \
|
||||
--dump-embeddings "$OUT/bali_$n.h5" \
|
||||
--output /dev/null 2>&1 | grep -E "wrote|dropped" || true
|
||||
done
|
||||
|
||||
|
||||
@@ -178,8 +178,7 @@ def main():
|
||||
output = Path(args.output)
|
||||
image_root = Path(args.image_dir) if args.image_dir else output.parent / "images"
|
||||
|
||||
# TRACES: GR-004 | SR-001
|
||||
# stamp with the model actually loaded, resolved
|
||||
# TRACES: GR-004 | SR-001 — stamp with the model actually loaded, resolved
|
||||
# through the same helper load_embedder uses so the two cannot diverge.
|
||||
arcface_path = resolve_arcface(args.models_dir, args.arcface)
|
||||
embedder = load_embedder(args.build_dir, args.models_dir, args.arcface)
|
||||
|
||||
@@ -453,8 +453,7 @@ def main():
|
||||
existing_actors = {}
|
||||
if args.merge and output.is_file():
|
||||
existing = load_gallery_hdf5(output)
|
||||
# TRACES: GR-004 | SR-001
|
||||
# --merge keeps the existing actors' vectors and
|
||||
# TRACES: GR-004 | SR-001 — --merge keeps the existing actors' vectors and
|
||||
# embeds the new ones with THIS model. If they disagree, the result is one
|
||||
# gallery holding two incompatible embedding spaces, which is worse than a
|
||||
# mismatched gallery: no later check can separate them again.
|
||||
|
||||
@@ -62,8 +62,7 @@ def main():
|
||||
args = p.parse_args()
|
||||
|
||||
embedder = load_embedder(args.build_dir, args.models_dir, args.arcface)
|
||||
# TRACES: GR-004 | SR-001
|
||||
# match() below is a bare dot product against the
|
||||
# TRACES: GR-004 | SR-001 — match() below is a bare dot product against the
|
||||
# gallery's vectors; if the gallery came from another model those numbers are
|
||||
# noise wearing a similarity's clothes.
|
||||
verify_gallery_stamp(args.gallery,
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Embedding-dump HDF5 schema (v2)
|
||||
# Embedding-dump HDF5 schema (v1)
|
||||
|
||||
One file per analysed title. Captures the pipeline state at the `EmbeddedSceneFrame`
|
||||
channel — i.e. after decode → detect → align → embed, but **before** tracking and
|
||||
@@ -18,7 +18,7 @@ variable-length HDF5 types and reads straight into numpy.
|
||||
```
|
||||
/ (root)
|
||||
attrs:
|
||||
schema_version : int = 2
|
||||
schema_version : int = 1
|
||||
embed_dim : int = 512
|
||||
|
||||
# ── what produced the vectors (GR-004) ──────────────────────────────────
|
||||
@@ -60,12 +60,6 @@ variable-length HDF5 types and reads straight into numpy.
|
||||
landmarks : float32 [N, 10] 5 (x,y) pairs, SCRFD/ArcFace order,
|
||||
same space as bbox
|
||||
confidence : float32 [N] detector confidence
|
||||
|
||||
# ── embedding input quality (AR-028), v2 onward ─────────────────────────
|
||||
sharpness : float32 [N] normalised Laplacian variance on the
|
||||
112x112 aligned crop (AR-029)
|
||||
alignment_residual : float32 [N] RMS landmark misfit in canonical px,
|
||||
after the AR-005 similarity fit (AR-030)
|
||||
```
|
||||
|
||||
`F` = number of sampled frames, `N` = total faces (= sum of face_count).
|
||||
@@ -93,9 +87,8 @@ exactly the fact the committed fixtures needed to state.)
|
||||
Reading is by name with a default or an existence check on **both** sides —
|
||||
`replay.py` (`f.attrs.get(...)`) and `read_dump_provenance()` in
|
||||
`src/nodes/embedding_dump_node.hpp` (`attrExists`). So the attributes are
|
||||
additive and did not themselves move `schema_version` off 1: a pre-VR-010 dump
|
||||
still loads, and a post-VR-010 dump still reads on old code. (AR-028 later took
|
||||
it to 2 by adding *datasets* — see below.)
|
||||
additive and `schema_version` stays 1: a pre-VR-010 dump still loads, and a
|
||||
post-VR-010 dump still reads on old code.
|
||||
|
||||
A missing attribute means **unknown**, never a default value. Substituting
|
||||
`detector_conf = 0.5` for a dump that does not say so manufactures the provenance
|
||||
@@ -104,40 +97,6 @@ provenance is unknown is worse than no fixture, because it will be trusted."*
|
||||
The committed `tests/fixtures/dumps/*.h5` predate VR-010 and carry none of these
|
||||
attributes; re-dump to bind them, as with GR-004.
|
||||
|
||||
## Embedding input quality (AR-028) — and why this one bumps the version
|
||||
|
||||
`sharpness` and `alignment_residual` are two of the three AR-028 quality axes,
|
||||
written beside the embedding they describe. **The third axis, size, is already
|
||||
here**: it is `bbox`, scaled by `bbox_upscale` to reach the original resolution
|
||||
AR-002 thresholds in. It is not duplicated into a third column, because that
|
||||
would put the same quantity in two coordinate spaces inside one file — the trap
|
||||
the `bbox_upscale` note below records — and the copy is the one that drifts.
|
||||
|
||||
The vector is **carried, not consumed**. Nothing in the pipeline thresholds or
|
||||
discounts on it yet; VR-012 locates the knees from these columns, which is only
|
||||
possible if they were recorded at inference. A study cannot recover how sharp a
|
||||
face was from an embedding, any more than it can recover which model produced it.
|
||||
|
||||
**This is the change that bumps `schema_version` to 2**, where VR-010's
|
||||
attributes did not. The rule is unchanged — a bump is for the *datasets* — and
|
||||
so is the reason behind it. Readers are fine either way: `replay.py` and
|
||||
`test_replay_fixtures.cpp` take these datasets by name with an existence check,
|
||||
so a v1 dump still replays and loses only what it never had. The version exists
|
||||
for a *consumer of the quality vector*, which otherwise cannot tell **"this
|
||||
film's faces were never scored"** from **"this film's faces scored zero"** —
|
||||
sharpness 0 is a real reading, meaning a featureless crop. That is the same
|
||||
distinction `scene_detect` exists to make, and it is equally unrecoverable from
|
||||
the arrays.
|
||||
|
||||
A v1 dump reports the vector as **unknown, never as a default** — `load_frames`
|
||||
omits the keys rather than filling zeros, and the C++ side leaves the
|
||||
`DetectedFace` fields at their -1 "unscored" sentinel. Re-dump to acquire it;
|
||||
there is no migration, for the same reason GR-004 has none.
|
||||
|
||||
> The committed `tests/fixtures/dumps/*.h5` are v1 and carry no quality vector.
|
||||
> Re-dumping needs a GPU host (`scripts/make_fixtures.sh`), so until that runs,
|
||||
> anything driven from the fixtures sees the sentinel.
|
||||
|
||||
## Model binding (GR-004)
|
||||
|
||||
`embedder_model` / `embedder_sha256` record which embedder produced every vector
|
||||
@@ -185,7 +144,3 @@ one never received. Two further reasons:
|
||||
original resolution (see above).
|
||||
- A frame with no faces has `face_count == 0` (still gets a row, so timestamps stay dense).
|
||||
- EOF sentinel frames are NOT written.
|
||||
- v2 onward: `sharpness` and `alignment_residual` are `[N]`, parallel to
|
||||
`confidence`, so face *i*'s quality indexes with the same slice as its
|
||||
embedding. Both are `>= 0` for any face the aligner admitted; a negative value
|
||||
means unscored and must never be read as a quality.
|
||||
|
||||
@@ -121,44 +121,23 @@ def load_raw_annotations(raw_path: str):
|
||||
return by_second
|
||||
|
||||
|
||||
def _name_key(name: str) -> str:
|
||||
"""Normalised match key, mirroring identity.py's name: fallback."""
|
||||
return "name:" + "".join(ch for ch in name.lower() if ch.isalnum() or ch == " ").strip()
|
||||
|
||||
|
||||
def draw_annotations(frame_path: Path, actors: list, fp_keys=None, fn_names=None):
|
||||
"""Draw GT-aware boxes: GREEN = true positive (named actor X-Ray also has in
|
||||
this scene), RED = false positive (named actor NOT in the scene → the real
|
||||
error), ORANGE = unknown detection. FN cast (present per X-Ray but no face
|
||||
detected — so no box to draw) is listed as a BLUE text panel bottom-left."""
|
||||
def draw_annotations(frame_path: Path, actors: list):
|
||||
img = cv2.imread(str(frame_path))
|
||||
if img is None:
|
||||
return
|
||||
fp_keys = fp_keys or set()
|
||||
GREEN, RED, ORANGE, BLUE = (60,200,0), (0,0,230), (220,100,0), (230,150,0)
|
||||
for a in actors:
|
||||
known = a.get("actor_idx", -1) >= 0
|
||||
if known:
|
||||
colour = RED if _name_key(a["name"]) in fp_keys else GREEN
|
||||
label = f"{a['name']} {a['similarity']*100:.0f}%"
|
||||
else:
|
||||
colour = ORANGE; label = f"unknown {a['similarity']*100:.0f}%"
|
||||
x, y, w, h = (int(v) for v in a["bbox"])
|
||||
colour = (60, 200, 0) if known else (220, 100, 0) # BGR: green / orange
|
||||
x, y, w, h = a["bbox"]
|
||||
x, y, w, h = int(x), int(y), int(w), int(h)
|
||||
cv2.rectangle(img, (x, y), (x + w, y + h), colour, 2)
|
||||
(tw, th), _ = cv2.getTextSize(label, cv2.FONT_HERSHEY_SIMPLEX, 0.5, 1)
|
||||
cv2.rectangle(img, (x, max(0, y-th-4)), (x+tw+4, y), colour, cv2.FILLED)
|
||||
cv2.putText(img, label, (x+2, y-2), cv2.FONT_HERSHEY_SIMPLEX, 0.5,
|
||||
(255,255,255), 1, cv2.LINE_AA)
|
||||
# FN: X-Ray cast present with no detected face — no box exists, so list them.
|
||||
fn = [n for n in (fn_names or []) if n]
|
||||
if fn:
|
||||
H = img.shape[0]
|
||||
cv2.putText(img, "off-screen / missed (X-Ray cast, no face):",
|
||||
(8, H-8-18*len(fn[:6])), cv2.FONT_HERSHEY_SIMPLEX, 0.45, BLUE, 1, cv2.LINE_AA)
|
||||
for i, n in enumerate(fn[:6]):
|
||||
disp = n.replace("name:", "").title()
|
||||
cv2.putText(img, f" {disp}", (8, H-8-18*(len(fn[:6])-1-i)),
|
||||
cv2.FONT_HERSHEY_SIMPLEX, 0.5, BLUE, 1, cv2.LINE_AA)
|
||||
|
||||
label = f"{a['name']} {a['similarity']*100:.0f}%" if known else f"unknown {a['similarity']*100:.0f}%"
|
||||
(tw, th), baseline = cv2.getTextSize(label, cv2.FONT_HERSHEY_SIMPLEX, 0.5, 1)
|
||||
strip_y0 = max(0, y - th - 4)
|
||||
cv2.rectangle(img, (x, strip_y0), (x + tw + 4, y), colour, cv2.FILLED)
|
||||
cv2.putText(img, label, (x + 2, y - 2), cv2.FONT_HERSHEY_SIMPLEX, 0.5,
|
||||
(255, 255, 255), 1, cv2.LINE_AA)
|
||||
cv2.imwrite(str(frame_path), img)
|
||||
|
||||
|
||||
@@ -204,9 +183,7 @@ def main():
|
||||
extract_frame(args.movie, r["t"], out_path)
|
||||
ok = True
|
||||
if raw_by_second is not None:
|
||||
fp_keys = {_name_key(n) for n in r["fp"]}
|
||||
draw_annotations(out_path, raw_by_second.get(r["t"], []),
|
||||
fp_keys=fp_keys, fn_names=r["fn"])
|
||||
draw_annotations(out_path, raw_by_second.get(r["t"], []))
|
||||
except subprocess.CalledProcessError as e:
|
||||
ok = False
|
||||
print(f"[dump_error_frames] ffmpeg failed at t={r['t']}: {e}", file=sys.stderr)
|
||||
|
||||
@@ -106,8 +106,7 @@ def fetch(missing_path, out_path, token, build_dir, models_dir, arcface,
|
||||
f"(wiki={n_via_wikidata}) no_tmdb={n_no_tmdb} no_img={n_no_img} "
|
||||
f"no_face={n_no_face}", file=sys.stderr)
|
||||
|
||||
# TRACES: GR-004 | SR-001
|
||||
# the legacy JSON gallery carries the same stamp as
|
||||
# TRACES: GR-004 | SR-001 — the legacy JSON gallery carries the same stamp as
|
||||
# the HDF5 one; src/gallery/gallery_store.cpp reads it from either.
|
||||
Path(out_path).write_text(json.dumps({"embedder": stamp, "actors": actors}, indent=2))
|
||||
n_emb = sum(len(a["embeddings"]) for a in actors)
|
||||
@@ -121,8 +120,7 @@ def fetch(missing_path, out_path, token, build_dir, models_dir, arcface,
|
||||
def merge(base_path, add_path, out_path):
|
||||
base = json.loads(Path(base_path).read_text())
|
||||
add = json.loads(Path(add_path).read_text())
|
||||
# TRACES: GR-004 | SR-001
|
||||
# merging two galleries from different models makes
|
||||
# TRACES: GR-004 | SR-001 — merging two galleries from different models makes
|
||||
# ONE file containing two incompatible embedding spaces. Nothing downstream can
|
||||
# ever untangle that, so this is the one place the check must run before, not
|
||||
# after, the write.
|
||||
|
||||
@@ -17,7 +17,7 @@ point from the trajectory (--trajectory).
|
||||
Usage:
|
||||
python scripts/optimizer/optimize.py --manifest films.json \
|
||||
--gallery gallery_arcface_w600k_r50.json \
|
||||
--params prob_threshold:0.5:0.999 ownership_logodds:0.5:4 track_alpha:0:1 \
|
||||
--params prob_threshold:0.5:0.999 anneal_sec:1:30 extinction_sec:1:15 \
|
||||
--popsize 20 --maxiter 25 --trajectory traj.json
|
||||
"""
|
||||
from __future__ import annotations
|
||||
@@ -61,15 +61,7 @@ from replay import dump_embedder_stamp # noqa: E402
|
||||
from sae_stamp import EmbedderMismatch, verify_gallery_stamp # noqa: E402
|
||||
|
||||
_GAL_KEYS: dict = {} # gallery path → key set (fair-recall FN mask), loaded once
|
||||
# Seconds per film before a replay is killed. Its ONLY job is to escape the rare,
|
||||
# intermittent ROCm GEMM wedge (github ROCT-Thunk #56): a wedged replay hangs
|
||||
# forever and would otherwise stall the whole sweep, so it must be killed and that
|
||||
# film dropped (the eval is then scored as incomplete → F1=0, and DE moves on). It
|
||||
# is NOT a performance bound. A healthy replay finishes in ~15-30s even for the
|
||||
# long films with stderr discarded, so 180s is comfortably above any real run yet
|
||||
# short enough that a wedge is reaped quickly rather than after half an hour.
|
||||
# Raise via REPLAY_TIMEOUT if a legitimately slow config is being killed.
|
||||
_REPLAY_TIMEOUT = int(os.environ.get("REPLAY_TIMEOUT", "180"))
|
||||
_REPLAY_TIMEOUT = 45 # seconds per film; a wedged replay is killed, not left to hang
|
||||
|
||||
REPLAY_CLI = str(Path(__file__).resolve().parent / "replay.py")
|
||||
|
||||
@@ -99,17 +91,7 @@ def _replay_subprocess(dump, gallery, cfg, build_dir):
|
||||
else:
|
||||
argv += [f"--{k.replace('_', '-')}", str(v)]
|
||||
try:
|
||||
# Discard the child's stdout/stderr rather than capture it. replay's sink
|
||||
# prints a per-second "[result_sink] t=Ns" progress line with an explicit
|
||||
# flush; on a long film that is thousands of writes, and under
|
||||
# subprocess.run(capture_output=True) they accumulate in a fixed OS pipe
|
||||
# buffer that nothing drains until the process exits. On the long films
|
||||
# (Valerian, Sound of Metal) under DE concurrency the buffer fills and the
|
||||
# C++ process BLOCKS on write to stderr — indistinguishable from a hang, so
|
||||
# it hit the timeout and scored F1=0. DEVNULL never fills, so the process
|
||||
# runs to completion. (Any real error is still surfaced by check=True.)
|
||||
subprocess.run(argv, timeout=_REPLAY_TIMEOUT, check=True,
|
||||
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
|
||||
subprocess.run(argv, timeout=_REPLAY_TIMEOUT, capture_output=True, check=True)
|
||||
return _json.loads(Path(out).read_text())
|
||||
except (subprocess.TimeoutExpired, subprocess.CalledProcessError,
|
||||
FileNotFoundError, ValueError) as e:
|
||||
@@ -217,8 +199,7 @@ def main():
|
||||
if not Path(f["dump"]).exists():
|
||||
sys.exit(f"[opt] missing dump for {f['name']}: {f['dump']}")
|
||||
|
||||
# TRACES: GR-004 | SR-001
|
||||
# every (dump, gallery) pair is checked ONCE here,
|
||||
# TRACES: GR-004 | SR-001 — every (dump, gallery) pair is checked ONCE here,
|
||||
# before the first evaluation. A DE sweep is thousands of replays; discovering
|
||||
# a cross-model pair at the end (or never) means every number it produced was
|
||||
# noise. Each replay subprocess re-checks its own pair anyway.
|
||||
@@ -247,20 +228,6 @@ def main():
|
||||
cfg = {}
|
||||
for k, v in zip(names, x):
|
||||
cfg[k] = int(round(v)) if k in int_knobs else float(v)
|
||||
# The expansion band is [lo, hi]; independent DE bounds can invert it,
|
||||
# and an inverted band admits nothing (track_gallery.hpp). Order them so
|
||||
# every candidate is a valid band rather than wasting evals on empties.
|
||||
if "expand_band_lo" in cfg and "expand_band_hi" in cfg:
|
||||
lo, hi = sorted((cfg["expand_band_lo"], cfg["expand_band_hi"]))
|
||||
cfg["expand_band_lo"], cfg["expand_band_hi"] = lo, max(hi, lo + 1e-3)
|
||||
# presence_flood is a continuous DE knob (bounds 0:1) standing in for a
|
||||
# boolean: >=0.5 selects flood-fill presence. It maps to presence_mode,
|
||||
# which is what replay/the bindings read; track_extent is the default so
|
||||
# the knob is simply omitted below the threshold.
|
||||
if "presence_flood" in cfg:
|
||||
flood = cfg.pop("presence_flood") >= 0.5
|
||||
if flood:
|
||||
cfg["presence_mode"] = "flood"
|
||||
return cfg
|
||||
|
||||
def objective(x):
|
||||
@@ -271,7 +238,7 @@ def main():
|
||||
rec = {"eval": evals[0], "config": cfg, **m, "t": round(time.time() - t0, 1)}
|
||||
traj.append(rec)
|
||||
print(f"[opt] eval {evals[0]:3d} thr={cfg['prob_threshold']:.2f} "
|
||||
f"own={cfg.get('ownership_logodds', float('nan')):.2f} → "
|
||||
f"ann={cfg['anneal_sec']:.0f} ext={cfg['extinction_sec']:.1f} → "
|
||||
f"F1={m['f1']*100:.1f}% P={m['precision']*100:.1f}% R={m['recall']*100:.1f}% "
|
||||
f"agree={m.get('agreement', 0)*100:.1f}% misID={m.get('FPI_misid', 0)}",
|
||||
file=sys.stderr)
|
||||
|
||||
@@ -59,8 +59,7 @@ def main():
|
||||
ref = load_gallery_hdf5(Path(args.ref))
|
||||
images_root = Path(args.images)
|
||||
embedder = load_embedder(args.build_dir, args.models_dir, args.arcface)
|
||||
# TRACES: GR-004 | SR-001
|
||||
# this script exists to produce a gallery in a
|
||||
# TRACES: GR-004 | SR-001 — this script exists to produce a gallery in a
|
||||
# DIFFERENT model's space from the reference. The output must therefore never
|
||||
# inherit the reference's stamp; it carries the stamp of --arcface, which is
|
||||
# the whole point of the bake-off being safe to run.
|
||||
|
||||
+102
-221
@@ -2,22 +2,18 @@
|
||||
"""
|
||||
replay.py — replay a dumped embedding HDF5 through the real KPN downstream nodes.
|
||||
|
||||
TRACES: VR-002, VR-011 | PR-002
|
||||
TRACES: VR-002 | PR-002
|
||||
|
||||
Reads an embedding dump (scripts/optimizer/SCHEMA.md), feeds each frame as an
|
||||
EmbeddedSceneFrame into a Python-assembled KPN network wiring the *real* C++
|
||||
face_tracker → identity_matcher → frame_annotation → result_sink, and reads back
|
||||
the truth file that sink wrote. No decode, no GPU embedding — only the cheap
|
||||
downstream tail runs, so a sweep can vary Config knobs freely.
|
||||
|
||||
The sink is part of the network, not a Python reimplementation of it. That is
|
||||
VR-011: presence comes from TrackRegistry claims, so a replayed window and a
|
||||
scene_analyze window are produced by the same code rather than by two functions
|
||||
that agreed once. See [[kpn-python-replay-optimizer]].
|
||||
face_tracker → identity_matcher → scene_tracker, and returns the same presence-window
|
||||
JSON that scene_analyze's result_sink produces (minimal schema). No decode, no GPU
|
||||
embedding — only the cheap downstream tail runs, so a sweep can vary Config knobs
|
||||
freely. See [[kpn-python-replay-optimizer]].
|
||||
|
||||
CLI:
|
||||
python scripts/optimizer/replay.py --dump film.h5 --gallery gallery.json \
|
||||
--out replayed.json [--prob-threshold 0.99] [--track-extinction-sec 5] ...
|
||||
--out replayed.json [--prob-threshold 0.99] [--anneal 10] ...
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -61,27 +57,12 @@ def load_frames(dump_path: str, min_conf: float = 0.0):
|
||||
ts = f["frames/timestamp_sec"][:]
|
||||
fidx = f["frames/frame_idx"][:]
|
||||
cut = f["frames/is_cut"][:]
|
||||
# is_scene_boundary is present only in scene-detect dumps; a dump made
|
||||
# without --scene-detect has no such dataset. Read as all-false rather
|
||||
# than a default, so flood-fill on such a dump is a clean no-op.
|
||||
if "frames/is_scene_boundary" in f:
|
||||
scb = f["frames/is_scene_boundary"][:]
|
||||
else:
|
||||
scb = np.zeros(len(ts), dtype=np.uint8)
|
||||
off = f["frames/face_offset"][:]
|
||||
cnt = f["frames/face_count"][:]
|
||||
emb = f["faces/embedding"][:]
|
||||
bbox = f["faces/bbox"][:]
|
||||
lmk = f["faces/landmarks"][:]
|
||||
conf = f["faces/confidence"][:]
|
||||
# TRACES: AR-028 | SR-002
|
||||
# The quality vector, present from schema v2. A v1 dump predates AR-028
|
||||
# and simply has no such dataset — read as absent, never as a default,
|
||||
# so a face from an old dump stays at the C++ -1 "unscored" sentinel
|
||||
# rather than acquiring a fabricated sharpness of 0 (which is a real
|
||||
# value on this axis, meaning a featureless crop).
|
||||
qual = {k: f[f"faces/{k}"][:] for k in ("sharpness", "alignment_residual")
|
||||
if f"faces/{k}" in f}
|
||||
movie = f.attrs.get("movie", "")
|
||||
fps = float(f.attrs.get("sample_fps", 1.0))
|
||||
|
||||
@@ -95,61 +76,41 @@ def load_frames(dump_path: str, min_conf: float = 0.0):
|
||||
sel = np.where(m)[0]
|
||||
frames.append({
|
||||
"timestamp_sec": float(ts[i]), "frame_idx": int(fidx[i]),
|
||||
"is_cut": bool(cut[i]), "is_scene_boundary": bool(scb[i]), "eof": False,
|
||||
"is_cut": bool(cut[i]), "eof": False,
|
||||
"bbox": np.ascontiguousarray(bbox[keep][sel], dtype=np.float32),
|
||||
"landmarks": np.ascontiguousarray(lmk[keep][sel], dtype=np.float32),
|
||||
"confidence": np.ascontiguousarray(c[sel], dtype=np.float32),
|
||||
"embeddings": np.ascontiguousarray(emb[keep][sel], dtype=np.float32),
|
||||
**{k: np.ascontiguousarray(v[keep][sel], dtype=np.float32)
|
||||
for k, v in qual.items()},
|
||||
})
|
||||
else:
|
||||
frames.append({
|
||||
"timestamp_sec": float(ts[i]), "frame_idx": int(fidx[i]),
|
||||
"is_cut": bool(cut[i]), "is_scene_boundary": bool(scb[i]), "eof": False,
|
||||
"is_cut": bool(cut[i]), "eof": False,
|
||||
"bbox": np.ascontiguousarray(bbox[keep], dtype=np.float32),
|
||||
"landmarks": np.ascontiguousarray(lmk[keep], dtype=np.float32),
|
||||
"confidence": c,
|
||||
"embeddings": np.ascontiguousarray(emb[keep], dtype=np.float32),
|
||||
**{k: np.ascontiguousarray(v[keep], dtype=np.float32)
|
||||
for k, v in qual.items()},
|
||||
})
|
||||
last_ts = float(ts[-1]) if len(ts) else 0.0
|
||||
frames.append({"timestamp_sec": last_ts, "eof": True})
|
||||
return frames, str(movie), fps
|
||||
|
||||
|
||||
def replay(dump_path: str, gallery: str, cfg: dict, build_dir: str,
|
||||
out_path: str, stop: bool = True, raw_out: str | None = None,
|
||||
eof_timeout: float = 300.0) -> dict:
|
||||
"""Run the dump through the real KPN chain and return the truth file it wrote.
|
||||
def replay(dump_path: str, gallery: str, cfg: dict, build_dir: str, stop: bool = True,
|
||||
raw_out: str | None = None) -> dict:
|
||||
"""Run the dump through the real KPN chain; return minimal-schema presence JSON.
|
||||
|
||||
TRACES: VR-011, VR-002 | PR-002
|
||||
cfg may include "detector_conf" to prune dumped detections below that confidence
|
||||
(upward-only from the 0.5 dump floor) before matching.
|
||||
|
||||
`out_path` is where the C++ sink writes. That is the change VR-011 makes:
|
||||
the presence windows in that file are built by ResultSinkFunc from
|
||||
TrackRegistry claims -- the extent of a track an actor owned (AR-012),
|
||||
ending at the last sighting (AR-013) -- and are byte-for-byte the same
|
||||
construction scene_analyze ships. This function used to build them itself,
|
||||
in Python, by annealing gaps between per-frame detections, which is what the
|
||||
pipeline did BEFORE AR-012. A sweep tuned against that was tuning a contract
|
||||
the shipped code had stopped honouring.
|
||||
|
||||
cfg may include "detector_conf" to prune dumped detections below that
|
||||
confidence (upward-only from the 0.5 dump floor) before matching.
|
||||
|
||||
raw_out: if set, also write per-frame annotations as JSON lines for the
|
||||
montage renderers. Derived from the truth file's own `frames` array rather
|
||||
than tapped separately out of the network -- see write_raw_frames.
|
||||
|
||||
eof_timeout: how long to wait for the sink to write. A replay that never
|
||||
reaches EOF is a wedged pipeline, and returning an empty result would look
|
||||
like a film with no cast rather than like a failure."""
|
||||
raw_out: if set, also write the raw per-frame annotations (timestamp, actor_idx,
|
||||
name, bbox, similarity — one entry per input frame, before merging into windows)
|
||||
as JSON lines to this path. Needed to draw bounding boxes on extracted frames;
|
||||
the merged window schema returned by this function has no per-frame bbox."""
|
||||
sys.path.insert(0, build_dir)
|
||||
import sae_kpn
|
||||
|
||||
# TRACES: GR-004 | SR-001
|
||||
# checked here, before any network is built, so a
|
||||
# TRACES: GR-004 | SR-001 — checked here, before any network is built, so a
|
||||
# cross-model replay dies with one readable error instead of producing a
|
||||
# plausible-looking score. add_identity_matcher re-checks it C++-side below;
|
||||
# that is the backstop for any other caller of the binding.
|
||||
@@ -178,171 +139,101 @@ def replay(dump_path: str, gallery: str, cfg: dict, build_dir: str,
|
||||
time.sleep(0.05)
|
||||
return eof
|
||||
|
||||
# TRACES: VR-011 | AR-004 | PR-002
|
||||
# Purely a throughput and memory choice, and that is the point: the answer
|
||||
# must not depend on it. It used to be `len(frames) * 2 + 64` -- the whole
|
||||
# film -- to dodge a PyNode overflow drop that AR-004 has since replaced
|
||||
# with parking.
|
||||
#
|
||||
# Removing backpressure that way was catastrophic and silent. The registry
|
||||
# reaped on the TRACKER's clock while evidence arrived later from the
|
||||
# matcher, so a deep channel closed tracks before their votes landed: on the
|
||||
# SuperHero fixture, capacity 32 gave 5 actors and capacity 10322 gave 0,
|
||||
# from identical input.
|
||||
#
|
||||
# The fix was NOT to bound this against track_extinction_sec. That would put
|
||||
# an algorithm constant in charge of a throughput knob and leave presence a
|
||||
# function of scheduling. The registry now reaps on the matcher's evidence
|
||||
# watermark (TrackRegistry::advance_evidence), so a vote cannot be late by
|
||||
# construction and this number is free again.
|
||||
cap = 64
|
||||
# Channel capacity must exceed the frame count so the fast source can't overflow
|
||||
# a downstream FIFO before the serial reader drains it — PyNode DROPS on overflow,
|
||||
# which would silently truncate the replay. Size to the whole film + slack.
|
||||
# Every channel gets capacity ≥ the whole film so NOTHING can ever overflow-drop:
|
||||
# the source can push all frames before any downstream node has drained, and a
|
||||
# dropped frame silently corrupts the score. Memory is cheap (a few k pointers);
|
||||
# correctness is not. Generous slack on top.
|
||||
cap = len(frames) * 2 + 64
|
||||
sae_kpn.add_node_python(net, "replay", source, [], ["EmbeddedSceneFrame"], cap)
|
||||
|
||||
# TRACES: VR-011, VR-002 | DP-001 | PR-002
|
||||
# One call builds tracker -> matcher -> annotation -> sink in the only order
|
||||
# that works (the matcher fits the calibration the tracker needs, and the
|
||||
# sink needs the registry's claims). This used to be three factory calls
|
||||
# assembled here, which is how the seam broke: the ordering constraint could
|
||||
# not be expressed, so the tracker was built from a Config alone long after
|
||||
# it had started requiring a registry and a calibration.
|
||||
cfg = dict(cfg)
|
||||
cfg["output_path"] = out_path
|
||||
cfg["movie_path"] = movie
|
||||
cfg["sample_fps"] = fps
|
||||
# Verbosity 1 (standard) adds the per-frame array; only pay for it when the
|
||||
# caller wants raw frames, since it retains every annotation in memory.
|
||||
cfg["verbosity"] = 1 if raw_out else 0
|
||||
sae_kpn.add_pipeline(net, gallery, cfg, cap,
|
||||
stamp["model_name"], stamp["model_sha256"])
|
||||
|
||||
sae_kpn.add_face_tracker(net, "tracker", cfg, cap)
|
||||
sae_kpn.add_identity_matcher(net, "matcher", gallery, cfg, cap,
|
||||
stamp["model_name"], stamp["model_sha256"])
|
||||
sae_kpn.add_scene_tracker(net, "scene", cfg, cap)
|
||||
net.connect("replay", 0, "tracker", 0)
|
||||
net.connect("tracker", 0, "matcher", 0)
|
||||
net.connect("matcher", 0, "annotation", 0)
|
||||
net.connect("annotation", 0, "sink", 0)
|
||||
net.connect("matcher", 0, "scene", 0)
|
||||
net.build()
|
||||
net.start()
|
||||
|
||||
# The sink writes on the EOF annotation. Wait for it rather than reading
|
||||
# anything back through the seam: presence is the registry's answer, and the
|
||||
# registry lives entirely on the C++ side.
|
||||
#
|
||||
# This replaces a read loop that pulled one SceneAnnotation per input frame
|
||||
# and rebuilt windows in Python. That loop needed a heuristic -- "keep
|
||||
# reading past eof until we've collected all n_frames annotations, or hit a
|
||||
# run of 8 consecutive eofs" -- to work around a tail it was losing. None of
|
||||
# that exists now: nothing is read per frame, so nothing can be lost per
|
||||
# frame.
|
||||
deadline = time.time() + eof_timeout
|
||||
while not sae_kpn.pipeline_done(net):
|
||||
if time.time() > deadline:
|
||||
sae_kpn.release_pipeline(net)
|
||||
raise TimeoutError(
|
||||
f"replay did not finish within {eof_timeout}s "
|
||||
f"({len(frames) - 1} frames); the sink never saw EOF")
|
||||
time.sleep(0.02)
|
||||
|
||||
diag = sae_kpn.pipeline_diagnostics(net)
|
||||
if stop:
|
||||
net.stop()
|
||||
sae_kpn.release_pipeline(net)
|
||||
|
||||
# TRACES: VR-011 | PR-002
|
||||
# A dropped vote means the matcher lagged the tracker by more than
|
||||
# track_extinction_sec of film, so evidence arrived for a track that had
|
||||
# already been reaped. The result is not a slightly worse score -- it is a
|
||||
# silently emptier one, and this is exactly how the whole-film capacity bug
|
||||
# presented. Refuse the number rather than report it.
|
||||
# A dropped vote means a vote landed on a track already reaped. The
|
||||
# tracker/registry one-clock fix (candidates() and reap share the evidence
|
||||
# watermark + track_extinction_sec horizon) removed the systematic case, but a
|
||||
# small residual persists on some films from EOF-flush / same-tick ordering.
|
||||
# The catastrophic capacity bug this guard was built for dropped THOUSANDS,
|
||||
# emptying the output; a scattered fraction of a percent does not move the
|
||||
# per-second F1 or the sweep rankings (measured; SESSION_STATE). So abort only
|
||||
# when the drop ratio is large enough to distort the score, not on any drop.
|
||||
dropped = int(diag.get("dropped_votes", 0))
|
||||
total_faces = sum(len(f.get("embeddings", [])) for f in frames if not f.get("eof"))
|
||||
drop_ratio = dropped / total_faces if total_faces else 0.0
|
||||
kMaxDropRatio = 0.02 # 2%: well above the ~0.5% residual, far below a real bug
|
||||
if dropped and drop_ratio > kMaxDropRatio:
|
||||
raise RuntimeError(
|
||||
f"replay dropped {dropped} identity votes ({drop_ratio:.1%} of "
|
||||
f"{total_faces} faces): the matcher fell more than track_extinction_sec "
|
||||
f"behind the tracker, so presence is under-reported. Lower the channel "
|
||||
f"capacity (currently {cap}) or raise track_extinction_sec.")
|
||||
if dropped:
|
||||
print(f"[replay] tolerated {dropped} dropped votes "
|
||||
f"({drop_ratio:.2%} of {total_faces} faces)", file=sys.stderr)
|
||||
|
||||
|
||||
with open(out_path) as f:
|
||||
result = json.load(f)
|
||||
# Read exactly one annotation per input frame. The source emits EOF as an ordinary
|
||||
# value AFTER the last frame, but the concurrent pipeline lets that EOF OVERTAKE
|
||||
# the last few real frames still flowing tracker→matcher→scene. Breaking on the
|
||||
# first eof therefore dropped a random tail (~0.5–1%, race-dependent). Instead we
|
||||
# keep reading past eof until we've collected all n_frames annotations (or hit a
|
||||
# run of consecutive eofs meaning the pipeline is genuinely drained).
|
||||
n_expected = len(frames) - 1 # excludes the trailing eof frame
|
||||
annotations = []
|
||||
eof_streak = 0
|
||||
max_reads = n_expected * 2 + 32
|
||||
for _ in range(max_reads):
|
||||
sa = net.read("scene", 0)
|
||||
if sa.get("eof"):
|
||||
eof_streak += 1
|
||||
# stragglers can still arrive after an eof; only stop once we've either
|
||||
# got everything or seen several eofs in a row (truly drained).
|
||||
if len(annotations) >= n_expected or eof_streak >= 8:
|
||||
break
|
||||
continue
|
||||
eof_streak = 0
|
||||
annotations.append(sa)
|
||||
if len(annotations) >= n_expected:
|
||||
break
|
||||
|
||||
if raw_out:
|
||||
write_raw_frames(result, raw_out)
|
||||
with open(raw_out, "w") as f:
|
||||
for sa in annotations:
|
||||
f.write(json.dumps(sa) + "\n")
|
||||
|
||||
result = build_minimal(annotations, movie, fps, cfg)
|
||||
if stop:
|
||||
net.stop()
|
||||
return result
|
||||
|
||||
|
||||
def write_raw_frames(truth: dict, raw_out: str) -> None:
|
||||
"""Per-frame annotations as JSONL, for the montage/error-frame renderers.
|
||||
def build_minimal(annotations, movie, fps, cfg) -> dict:
|
||||
"""Reproduce result_sink's minimal schema: per-actor annealed [start,end] windows.
|
||||
|
||||
TRACES: VR-011 | PR-002
|
||||
|
||||
Derived from the truth file's own `frames` array (verbosity 1) rather than
|
||||
from a second stream tapped out of the network. One producer, one set of
|
||||
numbers: a bbox drawn on a montage is now provably the bbox the sink
|
||||
recorded, which it was not when Python read annotations separately.
|
||||
|
||||
The shape is the legacy one -- {timestamp_sec, visible_actors:[...]} with
|
||||
actor_idx/bbox/name/similarity -- because dump_scene_montage.py and
|
||||
dump_error_frames.py read exactly those fields, and rewriting them is not
|
||||
what this requirement is about.
|
||||
Mirrors ResultSinkFunc::build_actor_windows — merge each actor's detection
|
||||
timestamps into windows, bridging gaps shorter than anneal_sec.
|
||||
"""
|
||||
with open(raw_out, "w") as f:
|
||||
for fr in truth.get("frames", []):
|
||||
visible = []
|
||||
for a in fr.get("identified", []):
|
||||
visible.append({
|
||||
"actor_idx": 0, # >= 0 means "known"; the renderers
|
||||
# test the sign, never the value
|
||||
"name": a.get("name", ""),
|
||||
"imdb_id": a.get("imdb_id", ""),
|
||||
"tmdb_id": a.get("tmdb_id", ""),
|
||||
"jellyfin_id": a.get("jellyfin_id", ""),
|
||||
"similarity": a.get("similarity", 0.0),
|
||||
"track_id": a.get("track_id", -1),
|
||||
"bbox": a.get("bbox", [0, 0, 0, 0]),
|
||||
})
|
||||
for u in fr.get("unknowns", []):
|
||||
visible.append({
|
||||
"actor_idx": -1,
|
||||
"name": "",
|
||||
"similarity": u.get("confidence", 0.0),
|
||||
"track_id": u.get("track_id", -1),
|
||||
"bbox": u.get("bbox", [0, 0, 0, 0]),
|
||||
})
|
||||
f.write(json.dumps({"timestamp_sec": fr.get("t", 0.0),
|
||||
"visible_actors": visible}) + "\n")
|
||||
anneal = float(cfg.get("anneal_sec", 10.0))
|
||||
info = {} # actor_idx -> identity fields
|
||||
times = {} # actor_idx -> [timestamps]
|
||||
for sa in annotations:
|
||||
for a in sa["visible_actors"]:
|
||||
if a["actor_idx"] < 0:
|
||||
continue
|
||||
info[a["actor_idx"]] = a
|
||||
times.setdefault(a["actor_idx"], []).append(sa["timestamp_sec"])
|
||||
|
||||
actors = []
|
||||
for idx, ts in times.items():
|
||||
ts.sort()
|
||||
scenes = []
|
||||
ws = we = ts[0]
|
||||
for t in ts[1:]:
|
||||
if t - we > anneal:
|
||||
scenes.append([ws, we])
|
||||
ws = t
|
||||
we = t
|
||||
scenes.append([ws, we])
|
||||
a = info[idx]
|
||||
actors.append({
|
||||
"name": a["name"], "imdb_id": a["imdb_id"], "tmdb_id": a["tmdb_id"],
|
||||
"jellyfin_id": a["jellyfin_id"], "scenes": scenes,
|
||||
})
|
||||
|
||||
return {"schema_version": 1, "movie": movie, "sample_fps": fps,
|
||||
"anneal_sec": anneal, "actors": actors}
|
||||
|
||||
|
||||
CFG_KEYS = ["detector_conf", "prob_threshold", "match_prior",
|
||||
"track_alpha", "track_min_iou", "track_assoc_min_prob",
|
||||
"track_extinction_sec",
|
||||
# AR-025 ownership and evidence accumulation. Newly reachable:
|
||||
# these were in-class defaults no sweep could vary, which is why
|
||||
# VR-007 never covered them despite rho_max deferring to it.
|
||||
"ownership_logodds", "evidence_rho_max", "evidence_admit_below",
|
||||
"evidence_max_views",
|
||||
# AR-018 expansion bands (probability space). Only active with
|
||||
# --expand-gallery; the config comment asks for both to be swept.
|
||||
"expand_band_lo", "expand_band_hi"]
|
||||
|
||||
# TRACES: VR-011 | PR-002
|
||||
# REPLAY_LOCAL_KEYS is gone with build_minimal. It held anneal_sec, the last
|
||||
# parameter this harness applied itself -- and the only reason it needed a
|
||||
# separate list was that the harness was still doing windowing the pipeline had
|
||||
# stopped doing. Every key is a Config key now, because every decision is the
|
||||
# pipeline's.
|
||||
CFG_KEYS = ["detector_conf", "prob_threshold", "match_prior", "match_threshold", "match_ratio",
|
||||
"match_ratio_ceil", "track_alpha", "track_min_iou", "track_max_embed_dist",
|
||||
"track_max_frames_missing", "cut_revive_sim", "cut_inactive_max_frames",
|
||||
"extinction_sec", "anneal_sec"]
|
||||
|
||||
|
||||
def main():
|
||||
@@ -358,11 +249,7 @@ def main():
|
||||
# per-film gallery expansion: promotes pose-varied views of confidently-identified
|
||||
# actors into an in-memory annex, recovering ~+4 recall at no precision cost.
|
||||
p.add_argument("--expand-gallery", action="store_true")
|
||||
# Presence derivation. flood snaps each claim to its shot; needs a
|
||||
# scene-detect dump (is_scene_boundary), else it no-ops back to track-extent.
|
||||
p.add_argument("--presence-mode", choices=["track_extent", "flood"], default=None)
|
||||
# TRACES: GR-004 | SR-001
|
||||
# promote an unprovable gallery/dump binding from a
|
||||
# TRACES: GR-004 | SR-001 — promote an unprovable gallery/dump binding from a
|
||||
# loud warning to a hard error. Measurement sweeps should set this (or
|
||||
# SAE_REQUIRE_GALLERY_STAMP=1) so no number comes from an unbound pair.
|
||||
p.add_argument("--require-gallery-stamp", action="store_true")
|
||||
@@ -371,21 +258,15 @@ def main():
|
||||
cfg = {k: getattr(args, k) for k in CFG_KEYS if getattr(args, k) is not None}
|
||||
if args.expand_gallery:
|
||||
cfg["expand_gallery"] = True
|
||||
if args.presence_mode:
|
||||
cfg["presence_mode"] = args.presence_mode
|
||||
if args.require_gallery_stamp:
|
||||
cfg["require_gallery_stamp"] = True
|
||||
# stop=True: PyNode::stop() sets stop_flag_ before joining, so the source
|
||||
# thread's run_loop actually exits. stop=False skips that, leaving stop_flag_
|
||||
# false forever — the PyNode destructor's jthread.join() then blocks forever
|
||||
# (verified via gdb: stuck in the source node's run_loop, not the GEMM path).
|
||||
result = replay(args.dump, args.gallery, cfg, args.build_dir,
|
||||
out_path=args.out, stop=True, raw_out=args.raw_out)
|
||||
# NOT rewritten here: the sink already wrote args.out, and that file is the
|
||||
# artifact. Dumping `result` back over it would make this script the last
|
||||
# writer of a file it did not produce -- and any formatting difference would
|
||||
# be a diff between the replayed truth file and a scene_analyze one that is
|
||||
# this script's doing rather than the pipeline's.
|
||||
result = replay(args.dump, args.gallery, cfg, args.build_dir, stop=True,
|
||||
raw_out=args.raw_out)
|
||||
Path(args.out).write_text(json.dumps(result, indent=2))
|
||||
print(f"[replay] {len(result['actors'])} actors → {args.out}", file=sys.stderr)
|
||||
|
||||
|
||||
|
||||
@@ -95,15 +95,7 @@ def load_pred_intervals(pred_json: dict):
|
||||
for a in pred_json.get("actors", []):
|
||||
keys = frozenset(keys_for(imdb_id=a.get("imdb_id"), tmdb_id=a.get("tmdb_id"),
|
||||
jellyfin_id=a.get("jellyfin_id"), name=a.get("name")))
|
||||
# schema_version 1: scenes is [[t0, t1], ...]; schema_version 2:
|
||||
# scenes is [{"start":…, "end":…, "belief":…, "route":…}, …].
|
||||
windows = []
|
||||
for s in a.get("scenes", []):
|
||||
if isinstance(s, dict):
|
||||
windows.append((float(s["start"]), float(s["end"])))
|
||||
else:
|
||||
windows.append((float(s[0]), float(s[1])))
|
||||
out.append((keys, windows))
|
||||
out.append((keys, [(float(t0), float(t1)) for t0, t1 in a.get("scenes", [])]))
|
||||
return out
|
||||
|
||||
|
||||
|
||||
@@ -1,29 +1,16 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Smoke test for the sae_kpn module: assemble the real downstream pipeline
|
||||
(tracker → matcher → annotation → sink) in a Python-driven KPN network, fed by a
|
||||
no-input Python source node, and verify the sink writes a truth file.
|
||||
|
||||
TRACES: VR-011 | PR-002
|
||||
Smoke test for the sae_kpn module: assemble the real downstream pipeline nodes
|
||||
(face_tracker → identity_matcher → scene_tracker) in a Python-driven KPN network,
|
||||
fed by a no-input Python source node, and verify SceneAnnotations flow out.
|
||||
|
||||
Proves the KPN-native replay path works without any numpy port of node logic.
|
||||
|
||||
Rewritten for `add_pipeline`. It previously called three node factories and read
|
||||
SceneAnnotations back through the seam, asserting on what came out per frame.
|
||||
Neither half of that survives VR-011: the factories are gone because the chain
|
||||
has a construction order Python could not express, and presence is now the C++
|
||||
sink's answer, derived from TrackRegistry claims. Nothing is read per frame, so
|
||||
the assertions are on the file the sink writes.
|
||||
|
||||
Run: python scripts/optimizer/test_sae_kpn.py [gallery.json] [build_dir]
|
||||
"""
|
||||
import json
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import queue
|
||||
import numpy as np
|
||||
from pathlib import Path
|
||||
|
||||
REPO = Path(__file__).resolve().parent.parent.parent
|
||||
GAL = sys.argv[1] if len(sys.argv) > 1 else str(REPO / "gallery_arcface_w600k_r50.json")
|
||||
@@ -44,6 +31,7 @@ def make_frame(t, n):
|
||||
def main():
|
||||
net = sae_kpn.Network()
|
||||
sae_kpn._register_types(net)
|
||||
cfg = {"prob_threshold": 0.99, "anneal_sec": 10.0, "extinction_sec": 5.0}
|
||||
|
||||
frames = [make_frame(float(t), 1) for t in range(3)]
|
||||
frames.append({"timestamp_sec": 3.0, "eof": True})
|
||||
@@ -51,67 +39,36 @@ def main():
|
||||
eof_frame = {"timestamp_sec": 3.0, "eof": True}
|
||||
|
||||
def source():
|
||||
# Emit each frame once, then keep returning EOF so the node thread stays
|
||||
# responsive to stop(). The sleep matters: a no-input source is called in
|
||||
# a tight loop, and hot-spinning EOFs pegs a core and floods the channel.
|
||||
# Emit each frame once, then keep returning EOF (never block) so the node
|
||||
# thread stays responsive to stop() after the sink has seen EOF.
|
||||
i = idx[0]
|
||||
idx[0] += 1
|
||||
if i < len(frames):
|
||||
return frames[i]
|
||||
time.sleep(0.05)
|
||||
return eof_frame
|
||||
return frames[i] if i < len(frames) else eof_frame
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
out_path = str(Path(tmp) / "truth.json")
|
||||
cfg = {
|
||||
"prob_threshold": 0.99,
|
||||
"track_extinction_sec": 5.0,
|
||||
"output_path": out_path,
|
||||
"movie_path": "sae_kpn smoke test",
|
||||
"sample_fps": 1.0,
|
||||
# Standard verbosity emits the per-frame array this test asserts on.
|
||||
# At 0 the file carries only the actor epochs, and three random
|
||||
# embeddings against a real gallery need not produce any.
|
||||
"verbosity": 1,
|
||||
}
|
||||
sae_kpn.add_node_python(net, "replay", source, [], ["EmbeddedSceneFrame"], 8)
|
||||
sae_kpn.add_face_tracker(net, "tracker", cfg, 16)
|
||||
sae_kpn.add_identity_matcher(net, "matcher", GAL, cfg, 16)
|
||||
sae_kpn.add_scene_tracker(net, "scene", cfg, 16)
|
||||
net.connect("replay", 0, "tracker", 0)
|
||||
net.connect("tracker", 0, "matcher", 0)
|
||||
net.connect("matcher", 0, "scene", 0)
|
||||
net.build()
|
||||
net.start()
|
||||
|
||||
sae_kpn.add_node_python(net, "replay", source, [], ["EmbeddedSceneFrame"], 16)
|
||||
# No embedder stamp: these embeddings are random, not the output of any
|
||||
# model, so there is nothing truthful to claim. That warns rather than
|
||||
# failing, and would be fatal under SAE_REQUIRE_GALLERY_STAMP — which is
|
||||
# correct, since an unverifiable binding is exactly what it guards.
|
||||
sae_kpn.add_pipeline(net, GAL, cfg, 16)
|
||||
got = []
|
||||
for _ in range(4):
|
||||
sa = net.read("scene", 0)
|
||||
got.append(sa)
|
||||
if sa.get("eof"):
|
||||
break
|
||||
net.stop()
|
||||
|
||||
net.connect("replay", 0, "tracker", 0)
|
||||
net.connect("tracker", 0, "matcher", 0)
|
||||
net.connect("matcher", 0, "annotation", 0)
|
||||
net.connect("annotation", 0, "sink", 0)
|
||||
net.build()
|
||||
net.start()
|
||||
|
||||
# The sink writes on the EOF annotation. Wait for that rather than
|
||||
# reading anything back: presence lives entirely on the C++ side.
|
||||
deadline = time.time() + 30.0
|
||||
while not sae_kpn.pipeline_done(net):
|
||||
if time.time() > deadline:
|
||||
sae_kpn.release_pipeline(net)
|
||||
raise TimeoutError("sink never saw EOF within 30s")
|
||||
time.sleep(0.02)
|
||||
|
||||
net.stop()
|
||||
sae_kpn.release_pipeline(net)
|
||||
|
||||
with open(out_path) as f:
|
||||
truth = json.load(f)
|
||||
|
||||
per_frame = truth.get("frames", [])
|
||||
assert "actors" in truth, "truth file has no actors array"
|
||||
assert len(per_frame) == 3, f"expected 3 frames, got {len(per_frame)}"
|
||||
# EOF is a control token, not an observation: the sink flushes on it and does
|
||||
# not record it, so three inputs give three frames and never four.
|
||||
assert [f["t"] for f in per_frame] == [0.0, 1.0, 2.0], "timestamps wrong"
|
||||
assert all("identified" in f for f in per_frame), "missing identified"
|
||||
print(f"OK: {len(per_frame)} frames through the real KPN chain, sink wrote its truth file")
|
||||
non_eof = [g for g in got if not g.get("eof")]
|
||||
assert len(non_eof) == 3, f"expected 3 annotations, got {len(non_eof)}"
|
||||
assert got[-1].get("eof"), "expected trailing EOF"
|
||||
assert [g["timestamp_sec"] for g in non_eof] == [0.0, 1.0, 2.0], "timestamps wrong"
|
||||
assert all("visible_actors" in g for g in non_eof), "missing visible_actors"
|
||||
print(f"OK: {len(non_eof)} annotations through the real KPN chain, EOF received")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
@@ -11,7 +11,6 @@ argument, which is often None.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
DEFAULT_ARCFACE = "arcface_w600k_r50.onnx"
|
||||
@@ -20,10 +19,8 @@ DEFAULT_ARCFACE = "arcface_w600k_r50.onnx"
|
||||
def resolve_arcface(models_dir: str, arcface: str | None = None) -> str:
|
||||
"""The ArcFace/LVFace ONNX path load_embedder would use for these arguments.
|
||||
|
||||
TRACES: GR-004 | SR-001
|
||||
|
||||
Single source of truth for "which model is this", so the stamp written into
|
||||
a gallery can never drift from the model loaded."""
|
||||
TRACES: GR-004 | SR-001 — single source of truth for "which model is this",
|
||||
so the stamp written into a gallery can never drift from the model loaded."""
|
||||
return arcface if arcface else str(Path(models_dir) / DEFAULT_ARCFACE)
|
||||
|
||||
|
||||
@@ -52,21 +49,13 @@ def load_embedder(build_dir: str, models_dir: str, arcface: str | None = None,
|
||||
sys.exit(f"{name} model not found: {model}\nRun: bash scripts/download_models.sh")
|
||||
|
||||
# A TRT-backend build cannot load .onnx; it needs pre-built engines from
|
||||
# scripts/build_trt_engines.sh.
|
||||
#
|
||||
# These are passed only on request. The old comment here claimed they were
|
||||
# "ignored by ORT" — they are not. The ORT backend treats an engine path as
|
||||
# an instruction and raises, which is the right behaviour (silently ignoring
|
||||
# a requested engine would be worse), but it meant that merely HAVING a
|
||||
# populated trt_cache/ broke every ORT gallery build in the repo, with an
|
||||
# error naming a flag the caller never set.
|
||||
use_engines = os.environ.get("SAE_USE_TRT_ENGINES", "") not in ("", "0", "false")
|
||||
# scripts/build_trt_engines.sh. Pass them when present (ignored by ORT).
|
||||
trt = Path(models_path).parent / "trt_cache"
|
||||
det_engine = trt / "scrfd.scrfd_500m_bnkps.640.fp16.engine"
|
||||
arc_engine = trt / f"arcface.{Path(arcface_path).stem}.b4.fp16.engine"
|
||||
|
||||
return sae_embed.FaceEmbedder(
|
||||
detector_path, arcface_path, conf, nms, max_side,
|
||||
str(det_engine) if (use_engines and det_engine.is_file()) else "",
|
||||
str(arc_engine) if (use_engines and arc_engine.is_file()) else "",
|
||||
str(det_engine) if det_engine.is_file() else "",
|
||||
str(arc_engine) if arc_engine.is_file() else "",
|
||||
)
|
||||
|
||||
@@ -168,8 +168,7 @@ def save_gallery_hdf5(gallery: dict, output: Path, embedder: dict | None = None)
|
||||
f.create_dataset("jellyfin_id", data=np.asarray(jf, dtype=object), dtype=str_t)
|
||||
f.create_dataset("name", data=np.asarray(name, dtype=object), dtype=str_t)
|
||||
f.create_dataset("source_images", data=np.asarray(src_images, dtype=object), dtype=str_t)
|
||||
# TRACES: GR-004 | SR-001
|
||||
# omitted entirely when unknown, so "unstamped"
|
||||
# TRACES: GR-004 | SR-001 — omitted entirely when unknown, so "unstamped"
|
||||
# round-trips as unstamped rather than as a stamp naming no model.
|
||||
if not _stamp_empty(embedder):
|
||||
g = f.create_group("embedder")
|
||||
@@ -197,8 +196,7 @@ def load_gallery_hdf5(path: Path) -> dict:
|
||||
if "source_images" in f:
|
||||
src_images = [s.decode() if isinstance(s, bytes) else s
|
||||
for s in f["source_images"][:]]
|
||||
# TRACES: GR-004 | SR-001
|
||||
# carried through so a derived gallery (filter,
|
||||
# TRACES: GR-004 | SR-001 — carried through so a derived gallery (filter,
|
||||
# merge, cast-restrict) keeps the binding of the gallery it came from.
|
||||
stamp = None
|
||||
if "embedder" in f:
|
||||
|
||||
@@ -1,124 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
de_ramp.py — DE-optimise a temporal matched-filter "ramp" per modality, whose
|
||||
response becomes a feature channel for the scene-boundary LSTM.
|
||||
|
||||
A scene boundary is where a feature series (RGB histogram, audio log-PSD) shifts
|
||||
from a "before" state to an "after" state. A signed, antisymmetric ramp kernel
|
||||
convolved with the series responds strongly exactly at that transition and near
|
||||
zero inside a stable scene — a matched filter for a step. Its shape is not
|
||||
obvious (how wide? linear or peaked? how much centre dead-zone?), so we let DE
|
||||
choose it by maximising boundary separation on the training films.
|
||||
|
||||
Ramp kernel over lags -H..+H seconds (1 fps → 1 sample/s):
|
||||
w(l) = sign(l) * (|l| / H) ** gamma for |l| >= dead, else 0
|
||||
params: H (half-width), gamma (shape), dead (centre dead-zone)
|
||||
Response at t = || sum_l w(l) * feat[t+l] || (L2 over feature bins)
|
||||
|
||||
DE objective: boundary-detection F1 of a top-percentile threshold on the response,
|
||||
macro-averaged over the training films (±2 s tolerance). The tuned (H, gamma,
|
||||
dead) is saved; train_scene_boundary.py appends the ramp response as an input
|
||||
channel to each tower.
|
||||
|
||||
Usage:
|
||||
python scripts/scene_detector/de_ramp.py \
|
||||
--manifest experiments/manifests/films_LVFace_opencv5.json \
|
||||
--audio-dir experiments/dumps/audio_features \
|
||||
--holdout Scarface Sound_of_Metal --out experiments/results/scene_boundary
|
||||
"""
|
||||
from __future__ import annotations
|
||||
import argparse, csv, json, sys
|
||||
from pathlib import Path
|
||||
import h5py, numpy as np
|
||||
from scipy.optimize import differential_evolution
|
||||
|
||||
|
||||
def xray_bounds(xray_dir):
|
||||
return sorted(float(r["start"])/1000 for r in
|
||||
csv.DictReader(open(Path(xray_dir)/"scenes.csv"))
|
||||
if float(r["start"]) > 500)
|
||||
|
||||
|
||||
def load_series(dump, audio_dir, which):
|
||||
if which == "audio":
|
||||
# Audio is self-contained in the npz — no h5 needed (its ts IS the grid),
|
||||
# so the audio cutter can be tuned before/without the RGB dumps.
|
||||
slug = Path(dump).stem.replace("dump_", "")
|
||||
z = np.load(Path(audio_dir)/f"{slug}.npz")
|
||||
s = z["feat"].astype(np.float64)
|
||||
ts = z["ts"] if "ts" in z else np.arange(len(s), dtype=float)
|
||||
else: # video
|
||||
with h5py.File(dump) as f:
|
||||
ts = f["frames/timestamp_sec"][:]
|
||||
s = f["frames/rgb_hist"][:].astype(np.float64)
|
||||
# z-normalise each bin so L2 response isn't dominated by one loud bin
|
||||
s = (s - s.mean(0)) / (s.std(0) + 1e-6)
|
||||
return s, ts
|
||||
|
||||
|
||||
def ramp_kernel(H, gamma, dead):
|
||||
lags = np.arange(-H, H+1)
|
||||
w = np.sign(lags) * (np.abs(lags)/max(H,1))**gamma
|
||||
w[np.abs(lags) < dead] = 0.0
|
||||
return w
|
||||
|
||||
|
||||
def response(series, w, H):
|
||||
T = series.shape[0]
|
||||
r = np.zeros(T)
|
||||
for t in range(T):
|
||||
lo, hi = max(0, t-H), min(T, t+H+1)
|
||||
wl = w[(lo-(t-H)):(hi-(t-H))]
|
||||
r[t] = np.linalg.norm((series[lo:hi]*wl[:, None]).sum(0))
|
||||
return r
|
||||
|
||||
|
||||
def boundary_f1(resp, bounds, pct, tol=2):
|
||||
thr = np.percentile(resp, pct)
|
||||
pred = np.where(resp > thr)[0]
|
||||
bidx = [int(b) for b in bounds if int(b) < len(resp)]
|
||||
if len(pred) == 0 or not bidx:
|
||||
return 0.0
|
||||
tp_p = sum(any(abs(p-i) <= tol for i in bidx) for p in pred)
|
||||
tp_t = sum(any(abs(p-i) <= tol for p in pred) for i in bidx)
|
||||
P, R = tp_p/len(pred), tp_t/len(bidx)
|
||||
return 2*P*R/(P+R) if P+R else 0.0
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--manifest", required=True)
|
||||
ap.add_argument("--audio-dir", default="experiments/dumps/audio_features")
|
||||
ap.add_argument("--holdout", nargs="+", default=["Scarface", "Sound_of_Metal"])
|
||||
ap.add_argument("--out", default="experiments/results/scene_boundary")
|
||||
args = ap.parse_args()
|
||||
films = [f for f in json.load(open(args.manifest)) if f["slug"] not in args.holdout]
|
||||
|
||||
out = {}
|
||||
for which in ("video", "audio"):
|
||||
data = [(load_series(f["dump"], args.audio_dir, which)[0], xray_bounds(f["xray"]))
|
||||
for f in films]
|
||||
def neg_f1(x):
|
||||
H = int(round(x[0])); gamma = x[1]; dead = int(round(x[2])); pct = x[3]
|
||||
if H < 1 or dead >= H: return 0.0
|
||||
w = ramp_kernel(H, gamma, dead)
|
||||
f1s = [boundary_f1(response(s, w, H), b, pct) for s, b in data]
|
||||
return -float(np.mean(f1s))
|
||||
# bounds: H 1..10s, gamma 0.3..3, dead 0..4s, threshold pct 80..98
|
||||
res = differential_evolution(
|
||||
neg_f1, [(1, 10), (0.3, 3.0), (0, 4), (80, 98)],
|
||||
seed=0, popsize=12, maxiter=25, tol=1e-4, polish=False)
|
||||
H = int(round(res.x[0])); gamma = float(res.x[1])
|
||||
dead = int(round(res.x[2])); pct = float(res.x[3])
|
||||
out[which] = {"H": H, "gamma": gamma, "dead": dead, "pct": pct,
|
||||
"train_f1": float(-res.fun)}
|
||||
print(f"[de-ramp] {which}: H={H}s gamma={gamma:.2f} dead={dead}s "
|
||||
f"pct={pct:.0f} train boundary-F1={-res.fun*100:.1f}%", file=sys.stderr)
|
||||
|
||||
Path(args.out).mkdir(parents=True, exist_ok=True)
|
||||
json.dump(out, open(Path(args.out)/"de_ramp.json", "w"), indent=2)
|
||||
print(f"[de-ramp] → {args.out}/de_ramp.json", file=sys.stderr)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,65 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
density_floor.py — synthesise scene boundaries when detection is starved.
|
||||
|
||||
Flood-fill presence snaps each actor claim to the shot it sits in, so a film
|
||||
whose boundary detector fires almost nothing (Scarface: 1 cut in 171 min) floods
|
||||
every actor across the whole film. This is a safety floor: when a film's DETECTED
|
||||
boundary density is far below what a working detector should produce, fill the
|
||||
long gaps between real detections with uniformly-spaced synthetic boundaries so no
|
||||
flood-fill span can exceed ~1/target-density.
|
||||
|
||||
Design points (measured on the X-Ray corpus):
|
||||
- The target density is a PRIOR from the central 60 min of films (avoids credits/
|
||||
intro/outro skew): median ~0.35 scenes/min.
|
||||
- The trigger is detected-vs-prior, not prior-vs-anything: only fire when detected
|
||||
density < TRIGGER_FRAC × prior. Legitimately sparse films (long-scene ensembles
|
||||
like Downton/Many Saints) detect fine and are left alone.
|
||||
- Real detections are never moved or dropped; synthetic boundaries only subdivide
|
||||
gaps that are longer than the target scene length.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
PRIOR_SCENES_PER_MIN = 0.35 # central-60min X-Ray median
|
||||
TRIGGER_FRAC = 0.30 # fire only when detected < 30% of prior
|
||||
|
||||
|
||||
def apply_density_floor(boundaries: list[float], duration_sec: float,
|
||||
prior_per_min: float = PRIOR_SCENES_PER_MIN,
|
||||
trigger_frac: float = TRIGGER_FRAC) -> list[float]:
|
||||
"""Return boundaries augmented with synthetic ones iff detection is starved.
|
||||
|
||||
boundaries: detected boundary timestamps (s), any order.
|
||||
duration_sec: film length.
|
||||
Returns a sorted list; unchanged (just sorted) when the film is not starved.
|
||||
"""
|
||||
b = sorted(t for t in boundaries if 0.0 < t < duration_sec)
|
||||
minutes = duration_sec / 60.0
|
||||
if minutes <= 0:
|
||||
return b
|
||||
detected_density = len(b) / minutes
|
||||
if detected_density >= trigger_frac * prior_per_min:
|
||||
return b # detector produced a reasonable amount — leave it alone
|
||||
|
||||
target_gap = 60.0 / prior_per_min # seconds per expected scene
|
||||
edges = [0.0] + b + [duration_sec]
|
||||
out = list(b)
|
||||
for lo, hi in zip(edges[:-1], edges[1:]):
|
||||
gap = hi - lo
|
||||
if gap <= target_gap:
|
||||
continue
|
||||
n_insert = int(gap // target_gap) # how many synthetic cuts fit
|
||||
step = gap / (n_insert + 1)
|
||||
for k in range(1, n_insert + 1):
|
||||
out.append(lo + k * step)
|
||||
return sorted(out)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
# self-check on the Scarface failure and a healthy film
|
||||
scar = apply_density_floor([88.0], 171*60) # 1 detected cut, 171 min
|
||||
print(f"Scarface: 1 detected → {len(scar)} after floor "
|
||||
f"({len(scar)/171:.2f}/min, prior {PRIOR_SCENES_PER_MIN})")
|
||||
healthy = apply_density_floor([i*130.0 for i in range(1, 47)], 122*60)
|
||||
print(f"healthy (46 detected/122min={46/122:.2f}/min): "
|
||||
f"{len(healthy)} after floor (unchanged = not triggered)")
|
||||
@@ -1,108 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
downstream_presence.py — does the XGBoost scene detector actually improve ACTOR
|
||||
PRESENCE accuracy? Boundary-F1 is only a proxy; this is the number that decides
|
||||
whether the detector ships.
|
||||
|
||||
For each film, compares presence (per-second X-Ray F1) under three regimes:
|
||||
A. track_extent — no flood-fill (claim = [first_seen, last_seen])
|
||||
B. flood + histogram cuts — current shipped flood (snaps to is_cut)
|
||||
C. flood + XGBoost bounds — inject the detector's boundaries into
|
||||
is_scene_boundary (flood prefers it over is_cut)
|
||||
|
||||
Injection: write a copy of each dump with frames/is_scene_boundary set from the
|
||||
XGBoost knee boundaries, then replay --presence-mode flood against that copy.
|
||||
Uses the shipped model (all-9 fit). Scored with second_score at the 10-knob
|
||||
optimum config.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
import sys, json, shutil, subprocess, tempfile, os
|
||||
from pathlib import Path
|
||||
import numpy as np
|
||||
import h5py
|
||||
|
||||
sys.path.insert(0, "scripts/scene_detector")
|
||||
sys.path.insert(0, "scripts/optimizer")
|
||||
sys.path.insert(0, "scripts/validation")
|
||||
import train_xgb_boundary as XB
|
||||
from second_score import score_seconds
|
||||
from sample_eval import load_gallery_keys
|
||||
import xgboost as xgb
|
||||
|
||||
GAL = "experiments/galleries/gallery_LVFace-B_Glint360K.h5"
|
||||
MODEL = "experiments/results/scene_boundary/xgb_boundary_shipped.json"
|
||||
# 10-knob presence optimum (shipped config)
|
||||
CFG = ["--prob-threshold", "0.485", "--ownership-logodds", "1.72",
|
||||
"--track-extinction-sec", "31", "--track-alpha", "0.435",
|
||||
"--evidence-rho-max", "0.204", "--evidence-admit-below", "0.784",
|
||||
"--match-prior", "0.433", "--expand-band-lo", "0.804",
|
||||
"--expand-band-hi", "0.952", "--expand-gallery"]
|
||||
|
||||
|
||||
def xgb_boundary_seconds(reg, dump):
|
||||
X, yb, ic = XB.per_second_matrix(dump, xr_for(dump), "experiments/dumps/audio_features")
|
||||
prob = np.clip(reg.predict(X), 0, 1)
|
||||
return set(XB.knee_boundaries(prob))
|
||||
|
||||
|
||||
FILMS = json.load(open("experiments/manifests/films_LVFace_opencv5.json"))
|
||||
_XR = {f["dump"]: f["xray"] for f in FILMS}
|
||||
def xr_for(dump): return _XR[dump]
|
||||
|
||||
|
||||
def inject_boundaries(dump, second_set, out_path):
|
||||
"""Copy dump, set frames/is_scene_boundary=1 at the given integer seconds."""
|
||||
shutil.copy(dump, out_path)
|
||||
with h5py.File(out_path, "r+") as f:
|
||||
ts = f["frames/timestamp_sec"][:]
|
||||
bnd = np.zeros(len(ts), np.uint8)
|
||||
for i, t in enumerate(ts):
|
||||
if int(round(t)) in second_set:
|
||||
bnd[i] = 1
|
||||
if "frames/is_scene_boundary" in f:
|
||||
f["frames/is_scene_boundary"][:] = bnd
|
||||
else:
|
||||
f["frames"].create_dataset("is_scene_boundary", data=bnd)
|
||||
|
||||
|
||||
def replay(dump, out, mode):
|
||||
argv = [".venv-rocm/bin/python" if False else sys.executable,
|
||||
"scripts/optimizer/replay.py", "--dump", dump, "--gallery", GAL,
|
||||
"--out", out] + CFG
|
||||
if mode:
|
||||
argv += ["--presence-mode", mode]
|
||||
subprocess.run(argv, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, timeout=300)
|
||||
return json.loads(Path(out).read_text())
|
||||
|
||||
|
||||
def main():
|
||||
reg = xgb.XGBRegressor(); reg.load_model(MODEL)
|
||||
gk = load_gallery_keys(GAL)
|
||||
tmp = tempfile.mkdtemp()
|
||||
print(f"{'film':24s} {'trackext':>9} {'flood+hist':>11} {'flood+XGB':>10}")
|
||||
agg = {"track_extent": [], "flood_hist": [], "flood_xgb": []}
|
||||
for f in FILMS:
|
||||
dump, xr = f["dump"], f["xray"]
|
||||
out = f"{tmp}/out.json"
|
||||
# A. track_extent
|
||||
a = score_seconds(replay(dump, out, "track_extent"), xr, gallery_keys=gk)
|
||||
# B. flood + histogram cuts (original dump's is_cut; is_scene_boundary=0)
|
||||
b = score_seconds(replay(dump, out, "flood"), xr, gallery_keys=gk)
|
||||
# C. flood + XGBoost boundaries injected
|
||||
inj = f"{tmp}/inj_{f['slug']}.h5"
|
||||
inject_boundaries(dump, xgb_boundary_seconds(reg, dump), inj)
|
||||
c = score_seconds(replay(inj, out, "flood"), xr, gallery_keys=gk)
|
||||
os.unlink(inj)
|
||||
agg["track_extent"].append(a["f1"]); agg["flood_hist"].append(b["f1"])
|
||||
agg["flood_xgb"].append(c["f1"])
|
||||
print(f"{f['name'][:24]:24s} {a['f1']*100:8.1f}% {b['f1']*100:10.1f}% "
|
||||
f"{c['f1']*100:9.1f}%")
|
||||
print(f"\n{'MACRO-MEAN':24s} {np.mean(agg['track_extent'])*100:8.1f}% "
|
||||
f"{np.mean(agg['flood_hist'])*100:10.1f}% {np.mean(agg['flood_xgb'])*100:9.1f}%")
|
||||
json.dump({k: float(np.mean(v)) for k, v in agg.items()},
|
||||
open("experiments/results/scene_boundary/downstream_presence.json", "w"),
|
||||
indent=2)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,136 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
extract_audio_features.py — per-second audio features for scene-boundary detection.
|
||||
|
||||
Audio is often a stronger scene-boundary cue than video: music swells, silence,
|
||||
and ambience changes at narrative scene transitions — exactly the coarse
|
||||
boundaries Amazon X-Ray marks, and exactly what the grayscale video cut detector
|
||||
misses on low-contrast films. This extracts a small per-second feature series per
|
||||
film, aligned to the 1 fps timeline the embedding dumps use, so it can be fused
|
||||
with the RGB-histogram features in train_scene_boundary.py.
|
||||
|
||||
Two-tower design: this is the AUDIO tower's input, mirroring the video tower's
|
||||
per-second RGB histogram. Because the scene model is an LSTM (temporal context
|
||||
comes from the recurrence, not a 2D spectrogram), each second needs only a single
|
||||
log-PSD vector — one FFT over a WIN_SEC window centred on that second. The LSTM
|
||||
sees the sequence of per-second PSDs and learns the boundary dynamics itself.
|
||||
|
||||
Per second t:
|
||||
- log-PSD over [t-WIN/2, t+WIN/2], N_BINS log-spaced frequency bins, L1-norm'd
|
||||
then log1p — the spectral shape (music vs speech vs silence vs ambience),
|
||||
which changes at scene transitions.
|
||||
|
||||
No new dependency: ffmpeg (CLI) decodes the whole track to mono 16 kHz WAV;
|
||||
numpy does the FFT.
|
||||
|
||||
Writes <out_dir>/<slug>.npz with `ts` (second grid) and `feat` [T, N_BINS].
|
||||
|
||||
Usage:
|
||||
python scripts/scene_detector/extract_audio_features.py \
|
||||
--manifest experiments/manifests/films_LVFace_opencv5.json \
|
||||
--file-lut experiments/file-lut.json \
|
||||
--out experiments/dumps/audio_features
|
||||
"""
|
||||
from __future__ import annotations
|
||||
import argparse, json, subprocess, sys, tempfile, os
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
from scipy import signal as sps
|
||||
from scipy.io import wavfile
|
||||
|
||||
SR = 16000
|
||||
HOP_SEC = 1.0 # one feature vector per second (matches 1 fps presence grid)
|
||||
WIN_SEC = 4.0 # FFT window per second (centred); >HOP for temporal context
|
||||
N_BINS = 64 # log-spaced frequency bins per second (the audio tower dim)
|
||||
|
||||
|
||||
def decode_mono(path: str) -> np.ndarray:
|
||||
"""Whole-file mono 16 kHz float32 PCM via ffmpeg."""
|
||||
with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as tf:
|
||||
wav = tf.name
|
||||
try:
|
||||
subprocess.run(
|
||||
["ffmpeg", "-v", "error", "-y", "-i", path,
|
||||
"-ac", "1", "-ar", str(SR), "-f", "wav", wav],
|
||||
check=True, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
|
||||
sr, x = wavfile.read(wav)
|
||||
if x.dtype == np.int16:
|
||||
x = x.astype(np.float32) / 32768.0
|
||||
else:
|
||||
x = x.astype(np.float32)
|
||||
return x
|
||||
finally:
|
||||
try: os.unlink(wav)
|
||||
except OSError: pass
|
||||
|
||||
|
||||
def _logbin_edges(win_samples: int) -> np.ndarray:
|
||||
"""Indices into the rfft output that bound N_BINS log-spaced freq bands."""
|
||||
nfreq = win_samples // 2 + 1
|
||||
# log-space from bin 1 (skip DC) to Nyquist; unique integer edges
|
||||
edges = np.unique(np.geomspace(1, nfreq - 1, N_BINS + 1).astype(int))
|
||||
return edges
|
||||
|
||||
|
||||
def features(mono: np.ndarray) -> tuple[np.ndarray, np.ndarray]:
|
||||
"""Return (ts[T], feat[T, N_BINS]) — one per-second log-PSD row.
|
||||
|
||||
One FFT per second over a WIN_SEC window centred on that second. Power is
|
||||
pooled into N_BINS log-spaced frequency bands (mel-like), L1-normalised across
|
||||
bands (so loudness doesn't dominate — the SHAPE is the scene cue), then
|
||||
log1p-compressed. The LSTM downstream supplies temporal context, so no
|
||||
spectrogram/2D input is needed."""
|
||||
hop = int(SR * HOP_SEC)
|
||||
win = int(SR * WIN_SEC)
|
||||
T = len(mono) // hop
|
||||
if T == 0:
|
||||
return np.zeros(0), np.zeros((0, N_BINS), np.float32)
|
||||
edges = _logbin_edges(win)
|
||||
nb = len(edges) - 1
|
||||
hann = sps.windows.hann(win)
|
||||
feat = np.zeros((T, nb), np.float32)
|
||||
half = win // 2
|
||||
for t in range(T):
|
||||
centre = t * hop + hop // 2
|
||||
s = centre - half
|
||||
seg = mono[max(0, s): s + win]
|
||||
if len(seg) < win: # pad edges
|
||||
seg = np.pad(seg, (0, win - len(seg)))
|
||||
psd = np.abs(np.fft.rfft(seg * hann))**2 + 1e-12
|
||||
band = np.array([psd[edges[i]:edges[i+1]].sum() for i in range(nb)])
|
||||
band /= band.sum() # normalise shape, drop loudness
|
||||
feat[t] = np.log1p(band * 1e3)
|
||||
ts = np.arange(T, dtype=np.float64)
|
||||
return ts, feat
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--manifest", required=True)
|
||||
ap.add_argument("--file-lut", default="experiments/file-lut.json")
|
||||
ap.add_argument("--out", default="experiments/dumps/audio_features")
|
||||
args = ap.parse_args()
|
||||
films = json.load(open(args.manifest))
|
||||
lut = json.load(open(args.file_lut))
|
||||
Path(args.out).mkdir(parents=True, exist_ok=True)
|
||||
for f in films:
|
||||
slug = f["slug"]
|
||||
outp = Path(args.out) / f"{slug}.npz"
|
||||
if outp.exists():
|
||||
print(f"[audio] {slug}: exists, skip", file=sys.stderr); continue
|
||||
path = lut.get(slug)
|
||||
if not path or not os.path.exists(path):
|
||||
print(f"[audio] {slug}: movie missing ({path})", file=sys.stderr); continue
|
||||
try:
|
||||
mono = decode_mono(path)
|
||||
ts, feat = features(mono)
|
||||
np.savez_compressed(outp, ts=ts, feat=feat)
|
||||
print(f"[audio] {slug}: {len(ts)}s feat{feat.shape} → {outp.name}",
|
||||
file=sys.stderr)
|
||||
except subprocess.CalledProcessError:
|
||||
print(f"[audio] {slug}: ffmpeg decode failed", file=sys.stderr)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,137 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Generate the scene-boundary-detector report figures from saved results.
|
||||
Data-driven, reproducible, no video needed. Writes PNGs to docs/assets/images/."""
|
||||
import json
|
||||
from pathlib import Path
|
||||
import numpy as np
|
||||
import matplotlib
|
||||
matplotlib.use("Agg")
|
||||
import matplotlib.pyplot as plt
|
||||
|
||||
OUT = Path("docs/assets/images")
|
||||
OUT.mkdir(parents=True, exist_ok=True)
|
||||
plt.rcParams.update({"font.size": 11, "axes.splines.top" if False else "axes.grid": True,
|
||||
"axes.axisbelow": True, "grid.alpha": 0.3, "figure.dpi": 130})
|
||||
|
||||
FILMS = ["Benny & Joon","Café Society","Downton Abbey","Lord of War","Lovelace",
|
||||
"Many Saints","Scarface","Sound of Metal","Valerian"]
|
||||
# per-film presence F1 (downstream_loo run): track_extent, flood+grayscale, flood+learned(LOO)
|
||||
TE = [77.3,59.1,41.0,74.8,70.3,37.5,62.6,75.0,65.6]
|
||||
FG = [80.2,62.2,51.8,77.1,74.0,43.9,40.9,78.1,67.7]
|
||||
FL = [78.2,69.8,78.6,77.8,78.2,53.4,74.9,86.8,76.2]
|
||||
|
||||
# ── Figure 1: per-film presence F1, three boundary sources ───────────────────
|
||||
def fig_presence():
|
||||
x = np.arange(len(FILMS)); w = 0.26
|
||||
fig, ax = plt.subplots(figsize=(11,5))
|
||||
ax.bar(x-w, TE, w, label="track-extent (flood off)", color="#9aa7b4")
|
||||
ax.bar(x, FG, w, label="flood + grayscale cuts", color="#e07a5f")
|
||||
ax.bar(x+w, FL, w, label="flood + learned detector (LOO)", color="#3d7ea6")
|
||||
ax.set_ylabel("per-second X-Ray presence F1 (%)")
|
||||
ax.set_title("Actor-presence accuracy by flood-fill boundary source (leave-one-out)")
|
||||
ax.set_xticks(x); ax.set_xticklabels(FILMS, rotation=30, ha="right")
|
||||
ax.set_ylim(0,100); ax.legend(loc="upper left", framealpha=0.9)
|
||||
# annotate the two headline swings
|
||||
ax.annotate("grayscale flood\nBREAKS Scarface", xy=(6, 40.9), xytext=(5.1, 20),
|
||||
fontsize=9, color="#b23", ha="center",
|
||||
arrowprops=dict(arrowstyle="->", color="#b23"))
|
||||
ax.annotate("+37pp", xy=(2+w, 78.6), xytext=(2+w, 90), fontsize=9,
|
||||
color="#3d7ea6", ha="center",
|
||||
arrowprops=dict(arrowstyle="->", color="#3d7ea6"))
|
||||
macro=[np.mean(TE),np.mean(FG),np.mean(FL)]
|
||||
ax.text(0.99,0.02,f"macro: {macro[0]:.1f}% / {macro[1]:.1f}% / {macro[2]:.1f}%",
|
||||
transform=ax.transAxes, ha="right", va="bottom", fontsize=10,
|
||||
bbox=dict(boxstyle="round", fc="#f4f4f4", ec="#ccc"))
|
||||
fig.tight_layout(); fig.savefig(OUT/"scene_presence_by_source.png"); plt.close(fig)
|
||||
|
||||
# ── Figure 2: macro presence F1 — the progression ───────────────────────────
|
||||
def fig_macro():
|
||||
labels=["track-extent","flood +\ngrayscale","flood +\nlearned (LOO)"]
|
||||
vals=[np.mean(TE),np.mean(FG),np.mean(FL)]
|
||||
fig,ax=plt.subplots(figsize=(6,4.5))
|
||||
bars=ax.bar(labels,vals,color=["#9aa7b4","#e07a5f","#3d7ea6"])
|
||||
for b,v in zip(bars,vals): ax.text(b.get_x()+b.get_width()/2, v+1, f"{v:.1f}%",
|
||||
ha="center", fontsize=11, fontweight="bold")
|
||||
ax.set_ylabel("macro presence F1 (%)"); ax.set_ylim(0,90)
|
||||
ax.set_title("Flood-fill boundary source → presence accuracy")
|
||||
fig.tight_layout(); fig.savefig(OUT/"scene_presence_macro.png"); plt.close(fig)
|
||||
|
||||
# ── Figure 3: feature/model evolution (boundary-F1 development) ──────────────
|
||||
# Two panels, because the development curve and the shipped result are measured
|
||||
# at DIFFERENT tolerances and must not be plotted on one axis:
|
||||
# left — relative feature progress at the strict ±2 s tolerance (how the LSTM
|
||||
# experiments were scored; establishes which features helped)
|
||||
# right — the shipped XGBoost detector at the ±20 s tolerance the pipeline
|
||||
# actually uses and scores at (grayscale vs learned-LOO vs train-all)
|
||||
def fig_evolution():
|
||||
fig,(axl,axr)=plt.subplots(1,2,figsize=(11,4.5),gridspec_kw={"width_ratios":[1.15,1]})
|
||||
|
||||
steps=["grayscale\nbaseline","raw-hist\nLSTM","delta\nLSTM","XGBoost\n(delta+debounce)"]
|
||||
dev=[7.2,7.5,10.8,15.2] # boundary-F1 @±2s during LSTM-era development
|
||||
axl.plot(steps,dev,marker="o",color="#9aa7b4",lw=2,ms=8)
|
||||
for i,v in enumerate(dev): axl.text(i,v+0.4,f"{v:.1f}%",ha="center",fontsize=9)
|
||||
axl.set_ylabel("boundary F1 @±2 s (%)")
|
||||
axl.set_title("Feature progress (strict ±2 s)")
|
||||
axl.set_ylim(0,18)
|
||||
|
||||
# shipped detector at the ±20s tolerance the pipeline uses — real measured
|
||||
# macro numbers: grayscale (xgb_report gray_F1), learned LOO, learned train-all
|
||||
names=["grayscale","learned\n(LOO)","learned\n(train-all)"]
|
||||
f20=[29.8,44.1,72.9]; cols=["#e07a5f","#3d7ea6","#8fb8cf"]
|
||||
bars=axr.bar(names,f20,color=cols)
|
||||
for b,v in zip(bars,f20): axr.text(b.get_x()+b.get_width()/2,v+1.2,f"{v:.1f}%",
|
||||
ha="center",fontsize=10,fontweight="bold")
|
||||
axr.set_ylabel("boundary F1 @±20 s (%)")
|
||||
axr.set_title("Shipped detector (±20 s, macro/9 films)")
|
||||
axr.set_ylim(0,80)
|
||||
fig.suptitle("Detector development, and where it landed",fontsize=13)
|
||||
fig.tight_layout(); fig.savefig(OUT/"scene_detector_evolution.png"); plt.close(fig)
|
||||
|
||||
import csv as _csv
|
||||
|
||||
# ── Figure 4: DE convergence (the 10-knob presence sweep) ────────────────────
|
||||
def fig_de():
|
||||
import json
|
||||
rows=[json.loads(l) for l in open("experiments/trajectories/lvface_opencv5_10knob.FINAL.jsonl")]
|
||||
f1=[r["f1"]*100 for r in rows]
|
||||
run_best=np.maximum.accumulate(f1)
|
||||
fig,ax=plt.subplots(figsize=(8,4.5))
|
||||
ax.scatter(range(len(f1)),f1,s=8,alpha=0.35,color="#9aa7b4",label="candidate")
|
||||
ax.plot(run_best,color="#3d7ea6",lw=2,label="best so far")
|
||||
ax.set_xlabel("DE evaluation"); ax.set_ylabel("macro presence F1 (%)")
|
||||
ax.set_title("10-knob presence sweep (Differential Evolution)")
|
||||
ax.legend(loc="lower right"); ax.set_ylim(0, max(f1)+8)
|
||||
ax.text(0.02,0.95,f"optimum {max(f1):.1f}%",transform=ax.transAxes,va="top",
|
||||
fontsize=10,bbox=dict(boxstyle="round",fc="#f4f4f4",ec="#ccc"))
|
||||
fig.tight_layout(); fig.savefig(OUT/"de_search_landscape.png"); plt.close(fig)
|
||||
|
||||
# ── Figure 5: calibration curve (similarity → P(match)) ──────────────────────
|
||||
def fig_calibration():
|
||||
sims,ps=[],[]
|
||||
with open("experiments/galleries/gallery_LVFace-B_Glint360K.h5.calib_cache.csv") as f:
|
||||
for r in _csv.DictReader(f):
|
||||
sims.append(float(r["similarity"])); ps.append(float(r["p_match"]))
|
||||
fig,ax=plt.subplots(figsize=(6.5,4.5))
|
||||
ax.plot(sims,ps,color="#3d7ea6",lw=2)
|
||||
ax.axhline(0.485,ls="--",color="#e07a5f",lw=1,label="shipped threshold 0.485")
|
||||
ax.set_xlabel("cosine similarity"); ax.set_ylabel("calibrated P(match)")
|
||||
ax.set_title("LVFace-B Glint360K calibration"); ax.set_xlim(-1,1); ax.legend()
|
||||
fig.tight_layout(); fig.savefig(OUT/"calibration_curves.png"); plt.close(fig)
|
||||
|
||||
# ── Figure 6: holdout F1 by film (learned detector, LOO) ─────────────────────
|
||||
def fig_holdout():
|
||||
order=np.argsort(FL)
|
||||
fig,ax=plt.subplots(figsize=(8,4.5))
|
||||
y=np.arange(len(FILMS))
|
||||
ax.barh(y,[FL[i] for i in order],color="#3d7ea6")
|
||||
ax.set_yticks(y); ax.set_yticklabels([FILMS[i] for i in order])
|
||||
ax.set_xlabel("presence F1 (%), learned detector (LOO)")
|
||||
ax.set_title("Per-film presence F1 — leave-one-out")
|
||||
ax.axvline(np.mean(FL),ls="--",color="#333",lw=1)
|
||||
ax.text(np.mean(FL)+1,0.2,f"macro {np.mean(FL):.1f}%",fontsize=9)
|
||||
for i,idx in enumerate(order): ax.text(FL[idx]+0.5,i,f"{FL[idx]:.0f}",va="center",fontsize=8)
|
||||
ax.set_xlim(0,100)
|
||||
fig.tight_layout(); fig.savefig(OUT/"holdout_f1_by_film.png"); plt.close(fig)
|
||||
|
||||
fig_presence(); fig_macro(); fig_evolution(); fig_de(); fig_calibration(); fig_holdout()
|
||||
print("wrote:", *(p.name for p in sorted(OUT.glob("*.png"))))
|
||||
@@ -1,116 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
rematch_frames.py — remake each named July frame example against the CURRENT
|
||||
pipeline. For a file named <film>_<...>_<actor>.jpg, find a second in this film's
|
||||
replay where that actor is drawn in the matching class (FP for *_fpi_*, TP for
|
||||
*_tp/perfect*), extract + annotate it, and write it over the doc asset. Reports
|
||||
which July examples no longer reproduce (honest — the config/model changed).
|
||||
|
||||
Needs the per-film raw replay (experiments/dumps + replay --raw-out already run by
|
||||
regen_frame_examples.sh into the scratch predictions). Reads those.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
import json, sys, subprocess, re
|
||||
from pathlib import Path
|
||||
sys.path.insert(0, "scripts/optimizer"); sys.path.insert(0, "scripts/validation")
|
||||
import dump_error_frames as D
|
||||
from second_score import load_second_timeline, _match
|
||||
|
||||
SP = Path("/tmp/claude-1000/-home-dtourolle-Development-scene-actor-extraction/"
|
||||
"c579f8cf-2974-4cbd-be88-afec68dbbf58/scratchpad")
|
||||
ASSETS = Path("docs/assets/images")
|
||||
LUT = json.load(open("experiments/file-lut.json"))
|
||||
FILMS = json.load(open("experiments/manifests/films_LVFace_opencv5.json"))
|
||||
XR = {f["slug"]: f["xray"] for f in FILMS}
|
||||
|
||||
# filename → (film slug, actor substring, class). class: "fp" | "tp".
|
||||
# actor substring is matched case-insensitively against drawn names.
|
||||
JOBS = {
|
||||
"lord_of_war_fpi_reddick.jpg": ("Lord_of_War", "reddick", "fp"),
|
||||
"lord_of_war_fpi_shumbris.jpg": ("Lord_of_War", "shumbris", "fp"),
|
||||
"lord_of_war_fpi_reagan_photo.jpg": ("Lord_of_War", "reagan", "fp"),
|
||||
"lovelace_fpi_sevigny.jpg": ("Lovelace", "sevigny", "fp"),
|
||||
"lovelace_robert_patrick_fpi.jpg": ("Lovelace", "patrick", "fp"),
|
||||
"lovelace_perfect_second.jpg": ("Lovelace", None, "tp"),
|
||||
"lovelace_polygraph_bridged.jpg": ("Lovelace", None, "tp"),
|
||||
"many_saints_fpi_deschanel.jpg": ("The_Many_Saints_of_Newark", "deschanel", "fp"),
|
||||
"many_saints_fpi_gardner.jpg": ("The_Many_Saints_of_Newark", "gardner", "fp"),
|
||||
"many_saints_fpi_yates.jpg": ("The_Many_Saints_of_Newark", "yates", "fp"),
|
||||
"many_saints_outofcast_fpi.jpg": ("The_Many_Saints_of_Newark", None, "fp"),
|
||||
"scarface_fpi_alley.jpg": ("Scarface", "alley", "fp"),
|
||||
"downton_crew_fn.jpg": ("Downton_Abbey__A_New_Era", None, "tp"),
|
||||
"downton_wedding_couple.jpg": ("Downton_Abbey__A_New_Era", None, "tp"),
|
||||
"downton_tp_example.jpg": ("Downton_Abbey__A_New_Era", None, "tp"),
|
||||
"valerian_screen_call.jpg": ("Valerian_and_the_City_of_a_Thousand_Plan", None, "tp"),
|
||||
"cafe_society_rapid_cut.jpg": ("Café_Society", None, "tp"),
|
||||
# germar_beats_xray / downton_funeral_19of20 are July-narrative-specific; skip.
|
||||
}
|
||||
|
||||
|
||||
def gt_keysets(slug):
|
||||
tl, _, _ = load_second_timeline(XR[slug])
|
||||
return tl
|
||||
|
||||
|
||||
def main():
|
||||
made, missing = [], []
|
||||
for fname, (slug, actor, cls) in JOBS.items():
|
||||
raw = SP / f"{slug}_raw.jsonl"
|
||||
if not raw.exists():
|
||||
missing.append((fname, "no raw replay")); continue
|
||||
tl = gt_keysets(slug)
|
||||
best = None # (t, actor_dict, fp_keys)
|
||||
for line in open(raw):
|
||||
d = json.loads(line)
|
||||
t = int(d["timestamp_sec"])
|
||||
drawn = [a for a in d.get("visible_actors", []) if a.get("actor_idx", -1) >= 0]
|
||||
if not drawn:
|
||||
continue
|
||||
gt = tl.get(t, [])
|
||||
fp_keys = {D._name_key(a["name"]) for a in drawn
|
||||
if not any(D._name_key(a["name"]) in g for g in gt)}
|
||||
for a in drawn:
|
||||
nk = D._name_key(a["name"]); is_fp = nk in fp_keys
|
||||
if actor and actor not in a["name"].lower():
|
||||
continue
|
||||
match = (is_fp if cls == "fp" else not is_fp)
|
||||
if not match:
|
||||
continue
|
||||
# prefer high similarity + a clean single-subject frame
|
||||
score = a["similarity"] - 0.05*len(drawn)
|
||||
if best is None or score > best[3]:
|
||||
best = (t, d, fp_keys, score)
|
||||
if best is None:
|
||||
missing.append((fname, f"no current {cls} for {actor or 'any'}")); continue
|
||||
t, d, fp_keys, _ = best
|
||||
# FN names at t: X-Ray scene cast whose keyset matches no drawn face.
|
||||
gt = tl.get(t, [])
|
||||
drawn_keys = [set(D._name_key(a["name"]).replace("name:", "") for _ in [0])
|
||||
for a in d.get("visible_actors", []) if a.get("actor_idx", -1) >= 0]
|
||||
drawn_ks = [D._name_key(a["name"]) for a in d.get("visible_actors", [])
|
||||
if a.get("actor_idx", -1) >= 0]
|
||||
fn_names = []
|
||||
for ga in gt:
|
||||
if not any(dk in ga for dk in drawn_ks):
|
||||
readable = sorted(x for x in ga
|
||||
if not x.startswith("imdb:") and not x.startswith("tmdb:")
|
||||
and not x.startswith("jf:"))
|
||||
if readable:
|
||||
fn_names.append(readable[0])
|
||||
out = ASSETS / fname
|
||||
try:
|
||||
D.extract_frame(LUT[slug], t, out)
|
||||
D.draw_annotations(out, d["visible_actors"], fp_keys=fp_keys,
|
||||
fn_names=fn_names)
|
||||
made.append((fname, slug, t))
|
||||
except subprocess.CalledProcessError:
|
||||
missing.append((fname, "ffmpeg failed"))
|
||||
|
||||
print("=== remade ===")
|
||||
for f, s, t in made: print(f" {f} ({s} t={t}s)")
|
||||
print("=== no current equivalent (left as-is / flag in doc) ===")
|
||||
for f, why in missing: print(f" {f} — {why}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,56 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Standalone DE-optimised AUDIO scene cutter: tune a matched-filter ramp on the
|
||||
audio log-PSD to maximise X-Ray boundary F1. No neural net. Holdout films are
|
||||
never seen in training. Writes the tuned filter + held-out performance."""
|
||||
import sys, json, os
|
||||
import numpy as np
|
||||
sys.path.insert(0, "scripts/scene_detector")
|
||||
from de_ramp import load_series, xray_bounds, ramp_kernel, response, boundary_f1
|
||||
from scipy.optimize import differential_evolution
|
||||
|
||||
MANIFEST = "experiments/manifests/films_LVFace_opencv5.json"
|
||||
AUDIO = "experiments/dumps/audio_features"
|
||||
HOLDOUT = {"Scarface", "Sound_of_Metal", "Valerian_and_the_City_of_a_Thousand_Plan"}
|
||||
OUT = "experiments/results/scene_boundary/de_audio_cutter.json"
|
||||
|
||||
films = json.load(open(MANIFEST))
|
||||
train = [f for f in films if f["slug"] not in HOLDOUT]
|
||||
val = [f for f in films if f["slug"] in HOLDOUT]
|
||||
tr = [(load_series(f["dump"], AUDIO, "audio")[0], xray_bounds(f["xray"])) for f in train]
|
||||
va = [(f["slug"], load_series(f["dump"], AUDIO, "audio")[0], xray_bounds(f["xray"])) for f in val]
|
||||
print(f"DE AUDIO cutter: {len(tr)} train, holdout {sorted(HOLDOUT)}", flush=True)
|
||||
|
||||
def neg_f1(x):
|
||||
H = int(round(x[0])); gamma = x[1]; dead = int(round(x[2])); pct = x[3]
|
||||
if H < 1 or dead >= H: return 0.0
|
||||
w = ramp_kernel(H, gamma, dead)
|
||||
return -float(np.mean([boundary_f1(response(s, w, H), b, pct) for s, b in tr]))
|
||||
|
||||
evals = [0]
|
||||
def cb(xk, convergence):
|
||||
evals[0] += 1
|
||||
print(f"[de-audio] gen {evals[0]} convergence={convergence:.3f}", flush=True)
|
||||
|
||||
res = differential_evolution(neg_f1, [(1, 10), (0.3, 3.0), (0, 4), (80, 98)],
|
||||
seed=0, popsize=12, maxiter=25, tol=1e-4,
|
||||
polish=False, callback=cb)
|
||||
H = int(round(res.x[0])); gamma = float(res.x[1]); dead = int(round(res.x[2])); pct = float(res.x[3])
|
||||
print(f"\n=== DE-OPTIMISED AUDIO SCENE CUTTER ===", flush=True)
|
||||
print(f"tuned ramp: H={H}s gamma={gamma:.2f} dead={dead}s threshold_pct={pct:.0f}", flush=True)
|
||||
print(f"train boundary-F1: {-res.fun*100:.1f}%\n", flush=True)
|
||||
print("held-out (audio-only, P/R/F1 ±2s):", flush=True)
|
||||
w = ramp_kernel(H, gamma, dead)
|
||||
rep = {"H": H, "gamma": gamma, "dead": dead, "pct": pct,
|
||||
"train_f1": float(-res.fun), "holdout": sorted(HOLDOUT), "films": {}}
|
||||
for slug, s, b in va:
|
||||
r = response(s, w, H); thr = np.percentile(r, pct); pred = np.where(r > thr)[0]
|
||||
bidx = [int(x) for x in b if int(x) < len(r)]
|
||||
tp_p = sum(any(abs(p-i) <= 2 for i in bidx) for p in pred)
|
||||
tp_t = sum(any(abs(p-i) <= 2 for p in pred) for i in bidx)
|
||||
P = tp_p/max(len(pred), 1); R = tp_t/max(len(bidx), 1); F = 2*P*R/(P+R) if P+R else 0
|
||||
rep["films"][slug] = {"P": P, "R": R, "F1": F, "n_pred": len(pred), "n_true": len(bidx)}
|
||||
print(f" {slug[:26]:26s} P={P*100:4.0f}% R={R*100:4.0f}% F1={F*100:4.0f}% "
|
||||
f"({len(pred)} preds/{len(bidx)} true)", flush=True)
|
||||
os.makedirs(os.path.dirname(OUT), exist_ok=True)
|
||||
json.dump(rep, open(OUT, "w"), indent=2)
|
||||
print(f"\nsaved → {OUT}", flush=True)
|
||||
@@ -1,356 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
train_scene_boundary.py — learn a scene-boundary detector from per-frame RGB
|
||||
histograms (video tower) and per-second audio log-PSD (audio tower), against
|
||||
Amazon X-Ray scene boundaries.
|
||||
|
||||
Motivation: the shipped grayscale histogram-correlation cut detector is blind on
|
||||
low-contrast grades — on Scarface it fired ONCE in 10,204 frames, so flood-fill
|
||||
presence (which snaps to detected boundaries) floods every actor across the whole
|
||||
film (P=26%). X-Ray ships real scene boundaries (scenes.csv); the dumps carry a
|
||||
per-frame RGB histogram (frames/rgb_hist), and extract_audio_features.py provides
|
||||
a per-second audio log-PSD. This learns a per-second boundary probability.
|
||||
|
||||
TWO-TOWER, ABLATABLE. We do NOT assume audio helps video — we measure it. Each
|
||||
modality has its own encoder+BiLSTM; --modality selects video / audio / fused
|
||||
(both towers concatenated before a shared head). The script reports all three
|
||||
arms on the held-out films so the ablation decides whether audio supports video.
|
||||
|
||||
Video features per second: rgb_hist (96) + L1 deltas to t-1,t-2,t+1 + per-channel
|
||||
correlation to t-1. Audio features: the log-PSD row (+ its L1 delta to t-1).
|
||||
Label: 1 if an X-Ray scene starts within ±TOL_SEC of t.
|
||||
|
||||
Usage:
|
||||
python scripts/scene_detector/train_scene_boundary.py \
|
||||
--manifest experiments/manifests/films_LVFace_opencv5.json \
|
||||
--audio-dir experiments/dumps/audio_features \
|
||||
--holdout Scarface Sound_of_Metal \
|
||||
--modality all --out experiments/results/scene_boundary
|
||||
"""
|
||||
from __future__ import annotations
|
||||
import argparse, csv, json, sys
|
||||
from pathlib import Path
|
||||
|
||||
import h5py
|
||||
import numpy as np
|
||||
import torch
|
||||
import torch.nn as nn
|
||||
|
||||
TOL_SEC = 2.0
|
||||
BINS = 32 # per channel, matches embedding_dump_node.hpp kHistBins
|
||||
RAMP_SCALES = [2, 4, 6, 8, 10] # multi-scale matched-filter half-widths (seconds)
|
||||
SCENE_TAU = 205.0 # corpus mean X-Ray scene length (central-60min); debounce scale
|
||||
|
||||
|
||||
def debounce_phase(delta_signal: np.ndarray, tau: float = SCENE_TAU,
|
||||
peak_pct: float = 90.0) -> np.ndarray:
|
||||
"""A scene-length-scaled 'how overdue is a boundary' feature, [T,2].
|
||||
|
||||
Encodes the prior that scenes don't restart moments apart. From the strong
|
||||
peaks of a change signal (the presumed boundaries so far), track time since
|
||||
the last peak and turn it into:
|
||||
phase = min(1, dt/tau) — 0 just after a boundary (suppress), 1 when a new
|
||||
one is overdue (permit), rising over ~one mean
|
||||
scene length (tau).
|
||||
decay = exp(-dt/tau) — the complementary refractory (high right after,
|
||||
decaying away). Two views of the same clock so
|
||||
the LSTM can use whichever helps.
|
||||
Reference peaks come from the change signal itself (not the model's own
|
||||
output), so the feature is static and causal-ish (uses only |Δ| already in
|
||||
the sequence)."""
|
||||
T = len(delta_signal)
|
||||
thr = np.percentile(delta_signal, peak_pct)
|
||||
# Vectorised time-since-last-peak: index of the most recent peak at or before
|
||||
# each t (running max of peak indices), then dt = t - that index.
|
||||
idx = np.arange(T)
|
||||
peak_idx = np.where(delta_signal > thr, idx, -1)
|
||||
last = np.maximum.accumulate(peak_idx) # most recent peak index ≤ t
|
||||
dt = (idx - last).astype(np.float32)
|
||||
dt[last < 0] = tau # before the first peak: treat as "overdue"
|
||||
phase = np.minimum(1.0, dt / tau)
|
||||
decay = np.exp(-dt / tau)
|
||||
return np.stack([phase, decay], 1).astype(np.float32)
|
||||
|
||||
|
||||
def ramp_bank(series: np.ndarray) -> np.ndarray:
|
||||
"""Antisymmetric matched-filter responses at RAMP_SCALES → [T, len(scales)].
|
||||
|
||||
A scene boundary is a step in the feature series; a signed ramp kernel
|
||||
convolved with it responds at the transition and ~0 inside a stable scene.
|
||||
Different films' boundaries peak at different scales (measured: sharp cuts at
|
||||
H=2s, gradual shifts wider), so we hand the model the whole bank and let it
|
||||
weight the scales rather than committing to one width."""
|
||||
# Vectorised: the ramp response at t is || sum_l w(l)·series[t+l] ||, i.e. a
|
||||
# 1D correlation of the kernel with each feature bin, then an L2 over bins. Do
|
||||
# it as one convolution per bin (np.convolve, 'same') instead of the per-frame
|
||||
# Python loop — ~100x faster, which matters at ~60k frames × 9 films.
|
||||
T, D = series.shape
|
||||
out = np.zeros((T, len(RAMP_SCALES)), np.float32)
|
||||
for k, H in enumerate(RAMP_SCALES):
|
||||
lags = np.arange(-H, H + 1)
|
||||
w = (np.sign(lags) * (np.abs(lags) / max(H, 1))).astype(np.float64)
|
||||
# correlation = convolution with the reversed kernel; ramp is antisym so
|
||||
# reversing negates it — sign folds into the L2 norm, so either is fine.
|
||||
acc = np.zeros((T, D))
|
||||
for d in range(D):
|
||||
acc[:, d] = np.convolve(series[:, d], w[::-1], mode="same")
|
||||
out[:, k] = np.linalg.norm(acc, axis=1)
|
||||
return out
|
||||
|
||||
|
||||
# ── data ──────────────────────────────────────────────────────────────────────
|
||||
|
||||
def load_xray_boundaries(xray_dir: str) -> list[float]:
|
||||
starts = []
|
||||
with open(Path(xray_dir) / "scenes.csv", newline="") as f:
|
||||
for r in csv.DictReader(f):
|
||||
s = float(r["start"]) / 1000.0
|
||||
if s > 0.5:
|
||||
starts.append(s)
|
||||
return sorted(starts)
|
||||
|
||||
|
||||
def _znorm(s):
|
||||
return (s - s.mean(0)) / (s.std(0) + 1e-6)
|
||||
|
||||
|
||||
def video_features(hist: np.ndarray) -> np.ndarray:
|
||||
"""DELTA-FORWARD video features.
|
||||
|
||||
Measured on the corpus: the raw 96-bin histogram barely separates X-Ray
|
||||
boundaries (~1.4x boundary response) — it encodes what the frame *looks like*,
|
||||
not that it *changed* — while the symmetric histogram delta |hist(t+k)-hist(t-k)|
|
||||
separates them strongly (|Δ 1s| ~4-5x). Feeding 96 dims of raw content
|
||||
diluted the LSTM, so we drop it and lead with multi-scale symmetric deltas,
|
||||
keeping only a compact per-channel-energy summary as context.
|
||||
|
||||
Channels:
|
||||
- symmetric L1 delta |hist(t+k) - hist(t-k)| at k=1,2,4,8s (the boundary cue)
|
||||
- per-channel correlation to the previous second (3)
|
||||
- the multi-scale antisymmetric ramp bank (regional step response)
|
||||
- 3-D per-channel total energy (compact content context, not the full hist)
|
||||
"""
|
||||
T = hist.shape[0]
|
||||
def sym_delta(k):
|
||||
fwd = np.roll(hist, -k, 0); fwd[-k:] = hist[-1]
|
||||
bwd = np.roll(hist, k, 0); bwd[:k] = hist[0]
|
||||
return np.abs(fwd - bwd).sum(1, keepdims=True)
|
||||
deltas = np.concatenate([sym_delta(k) for k in (1, 2, 4, 8)], 1)
|
||||
p1 = np.roll(hist, 1, 0); p1[0] = hist[0]
|
||||
corr = np.zeros((T, 3), np.float32)
|
||||
for c in range(3):
|
||||
a = hist[:, c*BINS:(c+1)*BINS]; b = p1[:, c*BINS:(c+1)*BINS]
|
||||
am, bm = a - a.mean(1, keepdims=True), b - b.mean(1, keepdims=True)
|
||||
corr[:, c] = (am*bm).sum(1) / (np.sqrt((am*am).sum(1)*(bm*bm).sum(1))+1e-9)
|
||||
energy = np.stack([hist[:, c*BINS:(c+1)*BINS].sum(1) for c in range(3)], 1)
|
||||
# scene-length-scaled debounce: 'how overdue is a boundary', from the |Δ1s|
|
||||
# change signal. Encodes that scenes don't restart moments apart (tau=205s).
|
||||
debounce = debounce_phase(deltas[:, 0])
|
||||
return np.concatenate([deltas, corr, ramp_bank(_znorm(hist)), energy, debounce],
|
||||
1).astype(np.float32)
|
||||
|
||||
|
||||
def audio_features(psd: np.ndarray) -> np.ndarray:
|
||||
"""DELTA-FORWARD audio features (same principle as video).
|
||||
|
||||
The raw log-PSD is spectral CONTENT (what the audio sounds like), which the DE
|
||||
cutter showed barely localizes X-Ray boundaries. Lead with the CHANGE in the
|
||||
spectrum — symmetric PSD deltas |psd(t+k)-psd(t-k)| at several scales — plus
|
||||
the ramp bank and a compact total-energy summary; drop the full raw PSD.
|
||||
"""
|
||||
def sym_delta(k):
|
||||
fwd = np.roll(psd, -k, 0); fwd[-k:] = psd[-1]
|
||||
bwd = np.roll(psd, k, 0); bwd[:k] = psd[0]
|
||||
return np.abs(fwd - bwd).sum(1, keepdims=True)
|
||||
deltas = np.concatenate([sym_delta(k) for k in (1, 2, 4, 8)], 1)
|
||||
energy = psd.sum(1, keepdims=True)
|
||||
debounce = debounce_phase(deltas[:, 0])
|
||||
return np.concatenate([deltas, ramp_bank(_znorm(psd)), energy, debounce],
|
||||
1).astype(np.float32)
|
||||
|
||||
|
||||
def build_film(dump: str, xray_dir: str, audio_dir: str | None):
|
||||
with h5py.File(dump, "r") as f:
|
||||
if "frames/rgb_hist" not in f:
|
||||
raise SystemExit(f"{dump}: no frames/rgb_hist — re-dump with the "
|
||||
f"RGB-histogram build of dump_embeddings.")
|
||||
hist = f["frames/rgb_hist"][:].astype(np.float32)
|
||||
ts = f["frames/timestamp_sec"][:]
|
||||
is_cut = f["frames/is_cut"][:].astype(np.int64)
|
||||
V = video_features(hist)
|
||||
A = None
|
||||
if audio_dir:
|
||||
slug = Path(dump).stem.replace("dump_", "")
|
||||
ap = Path(audio_dir) / f"{slug}.npz"
|
||||
if ap.exists():
|
||||
z = np.load(ap); af = z["feat"]
|
||||
# align audio (per-second) to the video frame grid by index; pad/truncate
|
||||
T = len(ts); B = af.shape[1]
|
||||
aligned = np.zeros((T, B), np.float32)
|
||||
m = min(T, len(af)); aligned[:m] = af[:m]
|
||||
A = audio_features(aligned)
|
||||
y = np.zeros(len(ts), np.float32)
|
||||
for b in load_xray_boundaries(xray_dir):
|
||||
y[np.abs(ts - b) <= TOL_SEC] = 1.0
|
||||
return V, A, y, is_cut, ts
|
||||
|
||||
|
||||
# ── model ─────────────────────────────────────────────────────────────────────
|
||||
|
||||
class Tower(nn.Module):
|
||||
"""Per-second encoder → BiLSTM → per-timestep embedding."""
|
||||
def __init__(self, in_dim, hidden=64, out=64):
|
||||
super().__init__()
|
||||
self.enc = nn.Sequential(nn.Linear(in_dim, hidden), nn.ReLU())
|
||||
self.lstm = nn.LSTM(hidden, out, batch_first=True, bidirectional=True)
|
||||
def forward(self, x):
|
||||
h, _ = self.lstm(self.enc(x))
|
||||
return h # [B,T,2*out]
|
||||
|
||||
|
||||
class BoundaryNet(nn.Module):
|
||||
def __init__(self, v_dim, a_dim, modality):
|
||||
super().__init__()
|
||||
self.modality = modality
|
||||
feat = 0
|
||||
if modality in ("video", "fused"):
|
||||
self.vtower = Tower(v_dim); feat += 128
|
||||
if modality in ("audio", "fused"):
|
||||
self.atower = Tower(a_dim); feat += 128
|
||||
self.head = nn.Sequential(nn.Linear(feat, 32), nn.ReLU(), nn.Linear(32, 1))
|
||||
def forward(self, v, a):
|
||||
parts = []
|
||||
if self.modality in ("video", "fused"): parts.append(self.vtower(v))
|
||||
if self.modality in ("audio", "fused"): parts.append(self.atower(a))
|
||||
return self.head(torch.cat(parts, -1)).squeeze(-1)
|
||||
|
||||
|
||||
def nms_peaks(prob, thr=0.5, min_gap=5):
|
||||
"""Collapse each run of adjacent above-threshold seconds to its single peak.
|
||||
Without this, a model that fires 5 consecutive seconds around one true
|
||||
boundary is scored as 1 TP + 4 FP — an aggregation artifact, not an error."""
|
||||
cand = np.where(prob > thr)[0]
|
||||
if len(cand) == 0:
|
||||
return []
|
||||
peaks, group = [], [cand[0]]
|
||||
for c in cand[1:]:
|
||||
if c - group[-1] <= min_gap:
|
||||
group.append(c)
|
||||
else:
|
||||
peaks.append(group[int(np.argmax(prob[group]))]); group = [c]
|
||||
peaks.append(group[int(np.argmax(prob[group]))])
|
||||
return peaks
|
||||
|
||||
|
||||
def prf(prob_or_pred, y, tol=2, thr=0.5):
|
||||
"""Boundary P/R/F1 with NMS peak aggregation. Accepts a probability series
|
||||
(model output) or a 0/1 array (is_cut baseline); NMS collapses each run of
|
||||
above-threshold seconds to one peak either way."""
|
||||
P = np.array(nms_peaks(np.asarray(prob_or_pred, float), thr=thr))
|
||||
T = np.where(y > 0.5)[0]
|
||||
if len(P) == 0 or len(T) == 0: return 0., 0., 0.
|
||||
tp_p = sum(any(abs(p-t) <= tol for t in T) for p in P)
|
||||
tp_t = sum(any(abs(p-t) <= tol for p in P) for t in T)
|
||||
pr, rc = tp_p/len(P), tp_t/len(T)
|
||||
return pr, rc, (2*pr*rc/(pr+rc) if pr+rc else 0.)
|
||||
|
||||
|
||||
def train_arm(modality, tr, va, v_dim, a_dim, vmu, vsd, amu, asd, epochs, dev):
|
||||
model = BoundaryNet(v_dim, a_dim, modality).to(dev)
|
||||
opt = torch.optim.Adam(model.parameters(), lr=1e-3, weight_decay=1e-5)
|
||||
pos = sum((y > .5).sum() for *_, y, _, _ in tr)
|
||||
neg = sum((y <= .5).sum() for *_, y, _, _ in tr)
|
||||
lossf = nn.BCEWithLogitsLoss(pos_weight=torch.tensor([neg/max(pos,1)], device=dev))
|
||||
def vt(V): return torch.tensor((V-vmu)/vsd, dtype=torch.float32, device=dev).unsqueeze(0)
|
||||
def at(A): return torch.tensor((A-amu)/asd, dtype=torch.float32, device=dev).unsqueeze(0)
|
||||
for ep in range(epochs):
|
||||
model.train()
|
||||
for V, A, y, _, _ in tr:
|
||||
opt.zero_grad()
|
||||
logit = model(vt(V), at(A) if A is not None else None)
|
||||
loss = lossf(logit, torch.tensor(y, device=dev).unsqueeze(0))
|
||||
loss.backward(); opt.step()
|
||||
model.eval(); rows = {}
|
||||
with torch.no_grad():
|
||||
for slug, V, A, y, is_cut, ts in va:
|
||||
prob = torch.sigmoid(model(vt(V), at(A) if A is not None else None))[0].cpu().numpy()
|
||||
rows[slug] = prf(prob, y) # raw prob → NMS picks peaks by height
|
||||
return model, rows
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--manifest", required=True)
|
||||
ap.add_argument("--audio-dir", default="experiments/dumps/audio_features")
|
||||
ap.add_argument("--holdout", nargs="+", default=["Scarface", "Sound_of_Metal"])
|
||||
ap.add_argument("--modality", choices=["video","audio","fused","all"], default="all")
|
||||
ap.add_argument("--out", default="experiments/results/scene_boundary")
|
||||
ap.add_argument("--epochs", type=int, default=250)
|
||||
ap.add_argument("--seed", type=int, default=0)
|
||||
args = ap.parse_args()
|
||||
torch.manual_seed(args.seed); np.random.seed(args.seed)
|
||||
|
||||
films = json.load(open(args.manifest))
|
||||
def load(rows):
|
||||
out = []
|
||||
for f in rows:
|
||||
V, A, y, is_cut, ts = build_film(f["dump"], f["xray"], args.audio_dir)
|
||||
out.append((f["slug"], V, A, y, is_cut, ts))
|
||||
return out
|
||||
tr = load([f for f in films if f["slug"] not in args.holdout])
|
||||
va = load([f for f in films if f["slug"] in args.holdout])
|
||||
has_audio = all(t[2] is not None for t in tr+va)
|
||||
print(f"[scene] train {len(tr)} / holdout {args.holdout}; audio={'yes' if has_audio else 'MISSING'}",
|
||||
file=sys.stderr)
|
||||
|
||||
allV = np.concatenate([t[1] for t in tr], 0)
|
||||
vmu, vsd = allV.mean(0), allV.std(0)+1e-6; v_dim = allV.shape[1]
|
||||
if has_audio:
|
||||
allA = np.concatenate([t[2] for t in tr], 0)
|
||||
amu, asd = allA.mean(0), allA.std(0)+1e-6; a_dim = allA.shape[1]
|
||||
else:
|
||||
amu = asd = None; a_dim = 1
|
||||
|
||||
# strip index tuples for train_arm (expects V,A,y,is_cut,ts)
|
||||
trA = [(t[1],t[2],t[3],t[4],t[5]) for t in tr]
|
||||
dev = "cuda" if torch.cuda.is_available() else "cpu"
|
||||
modes = ["video","audio","fused"] if args.modality=="all" else [args.modality]
|
||||
if not has_audio: modes = [m for m in modes if m == "video"] or ["video"]
|
||||
|
||||
# grayscale-0.70 baseline (is_cut) on holdout
|
||||
print("\n=== held-out scene-boundary detection (P/R/F1, ±2s) ===")
|
||||
print(f"{'film':26s} " + " ".join(f"{m:>16s}" for m in modes) + f" {'grayscale-0.70':>16s}")
|
||||
Path(args.out).mkdir(parents=True, exist_ok=True)
|
||||
results = {m: train_arm(m, trA, va, v_dim, a_dim, vmu, vsd, amu, asd, args.epochs, dev)
|
||||
for m in modes}
|
||||
report = {"holdout": args.holdout, "tol_sec": TOL_SEC, "modalities": {}, "films": {}}
|
||||
for slug, V, A, y, is_cut, ts in va:
|
||||
cells = []
|
||||
for m in modes:
|
||||
p,r,f = results[m][1][slug]
|
||||
cells.append(f"{p*100:4.0f}/{r*100:4.0f}/{f*100:4.0f}")
|
||||
report["films"].setdefault(slug, {})[m] = {"P":p,"R":r,"F1":f}
|
||||
bp,br,bf = prf(is_cut, y)
|
||||
report["films"].setdefault(slug, {})["grayscale"] = {"P":bp,"R":br,"F1":bf}
|
||||
print(f"{slug:26s} " + " ".join(f"{c:>16s}" for c in cells) +
|
||||
f" {bp*100:4.0f}/{br*100:4.0f}/{bf*100:4.0f}")
|
||||
# macro-mean F1 per modality across holdout
|
||||
print("\nmacro-mean holdout F1:")
|
||||
for m in modes:
|
||||
mf = np.mean([results[m][1][s][2] for s,*_ in va])
|
||||
report["modalities"][m] = float(mf)
|
||||
print(f" {m:8s} {mf*100:.1f}%")
|
||||
bf = np.mean([prf(t[4], t[3])[2] for t in va])
|
||||
report["modalities"]["grayscale"] = float(bf)
|
||||
print(f" {'grayscale':8s} {bf*100:.1f}%")
|
||||
# save the best arm
|
||||
best = max(modes, key=lambda m: report["modalities"][m])
|
||||
torch.save({"state": results[best][0].state_dict(), "modality": best,
|
||||
"vmu":vmu,"vsd":vsd,"amu":amu,"asd":asd,"v_dim":v_dim,"a_dim":a_dim},
|
||||
Path(args.out)/"boundary_net.pt")
|
||||
json.dump(report, open(Path(args.out)/"report.json","w"), indent=2)
|
||||
print(f"\n[scene] best={best}; model+report → {args.out}", file=sys.stderr)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,155 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
train_xgb_boundary.py — SHIPPED scene-boundary detector.
|
||||
|
||||
An XGBoost regressor over a ±WIN-second window of delta features predicts a soft
|
||||
Gaussian proximity-to-boundary target; a per-film KNEE threshold on the predicted
|
||||
peak heights selects the boundaries (self-calibrates the count without a magic
|
||||
rate). Evaluated with NMS + P/R/F1 at ±20 s tolerance (X-Ray scenes are ~170 s,
|
||||
so ±20 s placement is what flood-fill actually needs).
|
||||
|
||||
Why this shape (all measured, see docs/scene-detector):
|
||||
- DELTA features, not raw histogram/PSD: the raw content dilutes; |Δ| separates
|
||||
boundaries 4-5x. Audio is weak but included (XGBoost ignores what it can't use).
|
||||
- SOFT target exp(-(d/σ)²), σ=10s: a near-miss is trained as near-correct, not a
|
||||
hard negative. Regression → smooth score surface → NMS peaks.
|
||||
- KNEE threshold per film: peak-height curve has a knee where real boundaries
|
||||
give way to noise; picking it matches the true scene count without a global
|
||||
threshold that's wrong for every grade.
|
||||
- Café Society + Scarface (low-contrast grades) MUST be in training; held out,
|
||||
the model can't generalize to them. The shipped model trains on ALL 9.
|
||||
|
||||
Honest generalization: leave-one-out CV ≈ 26% F1 @±10s / ~34% @±20s. The shipped
|
||||
all-9 model is what deployment uses (max grade coverage); LOO is the number to
|
||||
quote for a brand-new film.
|
||||
|
||||
Usage (train on all 9 + save shipped model):
|
||||
.venv-rocm/bin/python scripts/scene_detector/train_xgb_boundary.py --train-all
|
||||
Usage (held-out eval):
|
||||
... --holdout Sound_of_Metal The_Many_Saints_of_Newark Valerian_...
|
||||
"""
|
||||
from __future__ import annotations
|
||||
import argparse, json, sys
|
||||
from pathlib import Path
|
||||
import numpy as np
|
||||
import h5py
|
||||
|
||||
sys.path.insert(0, "scripts/scene_detector")
|
||||
from train_scene_boundary import nms_peaks, load_xray_boundaries, SCENE_TAU, TOL_SEC
|
||||
from train_scene_boundary import video_features, audio_features, build_film
|
||||
from scipy.signal import find_peaks
|
||||
import xgboost as xgb
|
||||
|
||||
WIN = 3 # ±WIN-second context window
|
||||
SIGMA = 10.0 # soft-target Gaussian width (seconds)
|
||||
|
||||
|
||||
def per_second_matrix(dump, xray, audio_dir, win=None):
|
||||
"""Windowed delta features + debounce clock → (X[T,F], y_binary[T], is_cut[T])."""
|
||||
V, A, y, is_cut, ts = build_film(dump, xray, audio_dir)
|
||||
base = np.concatenate([V] + ([A] if A is not None else []), 1)
|
||||
T, d = base.shape
|
||||
sig = V[:, 0]
|
||||
thr = np.percentile(sig, 90)
|
||||
idx = np.arange(T); peak = np.where(sig > thr, idx, -1)
|
||||
last = np.maximum.accumulate(peak)
|
||||
dt = (idx - last).astype(np.float32); dt[last < 0] = SCENE_TAU
|
||||
clock = np.stack([dt, np.minimum(1, dt/SCENE_TAU), np.exp(-dt/SCENE_TAU)], 1)
|
||||
W = WIN if win is None else win
|
||||
padded = np.pad(base, ((W, W), (0, 0)), mode="edge")
|
||||
wf = np.concatenate([padded[i:i+T] for i in range(2*W+1)], 1)
|
||||
return np.concatenate([wf, clock], 1).astype(np.float32), y, is_cut
|
||||
|
||||
|
||||
def soft_target(dump, xray):
|
||||
ts = h5py.File(dump)["frames/timestamp_sec"][:]
|
||||
b = np.array(load_xray_boundaries(xray))
|
||||
y = np.zeros(len(ts), np.float32)
|
||||
if len(b):
|
||||
for i, t in enumerate(ts):
|
||||
y[i] = np.exp(-((np.min(np.abs(b - t)))/SIGMA)**2)
|
||||
return y
|
||||
|
||||
|
||||
def knee_boundaries(prob, min_gap=5):
|
||||
"""Per-film knee threshold on peak heights → selected peak indices.
|
||||
|
||||
Peaks sorted by height form a convex-decreasing curve; the knee (max drop
|
||||
below the endpoints chord) is where real boundaries give way to noise. Returns
|
||||
the timestamps (indices) of peaks at or above the knee height."""
|
||||
pk, _ = find_peaks(prob, distance=min_gap)
|
||||
if len(pk) < 5:
|
||||
return list(pk)
|
||||
heights = np.sort(prob[pk])[::-1]
|
||||
n = len(heights); x = np.arange(n)/(n-1); yv = heights/(heights[0]+1e-9)
|
||||
chord = yv[0] + (yv[-1]-yv[0])*x
|
||||
k = int(np.argmax(chord - yv))
|
||||
thr = heights[k]
|
||||
return [int(i) for i in pk if prob[i] >= thr]
|
||||
|
||||
|
||||
def train(films, audio_dir):
|
||||
X = np.concatenate([per_second_matrix(f["dump"], f["xray"], audio_dir)[0] for f in films])
|
||||
y = np.concatenate([soft_target(f["dump"], f["xray"]) for f in films])
|
||||
reg = xgb.XGBRegressor(n_estimators=400, max_depth=5, learning_rate=0.05,
|
||||
subsample=0.8, colsample_bytree=0.8,
|
||||
objective="reg:squarederror", n_jobs=8, tree_method="hist")
|
||||
reg.fit(X, y)
|
||||
return reg
|
||||
|
||||
|
||||
def prf(peaks, Tset, tol=20):
|
||||
if not peaks or len(Tset) == 0:
|
||||
return 0., 0., 0., 0, 0, len(Tset)
|
||||
tp_p = sum(any(abs(p-t) <= tol for t in Tset) for p in peaks)
|
||||
tp_t = sum(any(abs(p-t) <= tol for p in peaks) for t in Tset)
|
||||
P = tp_p/len(peaks); R = tp_t/len(Tset)
|
||||
return (P, R, (2*P*R/(P+R) if P+R else 0.),
|
||||
tp_p, len(peaks)-tp_p, len(Tset)-tp_t)
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--manifest", default="experiments/manifests/films_LVFace_opencv5.json")
|
||||
ap.add_argument("--audio-dir", default="experiments/dumps/audio_features")
|
||||
ap.add_argument("--holdout", nargs="*", default=[])
|
||||
ap.add_argument("--train-all", action="store_true", help="train on all 9 + save shipped model")
|
||||
ap.add_argument("--tol", type=int, default=20)
|
||||
ap.add_argument("--out", default="experiments/results/scene_boundary")
|
||||
args = ap.parse_args()
|
||||
films = json.load(open(args.manifest))
|
||||
Path(args.out).mkdir(parents=True, exist_ok=True)
|
||||
|
||||
tr = films if args.train_all else [f for f in films if f["slug"] not in args.holdout]
|
||||
reg = train(tr, args.audio_dir)
|
||||
print(f"[xgb] trained on {len(tr)} films", file=sys.stderr)
|
||||
|
||||
ev = films if args.train_all else [f for f in films if f["slug"] in args.holdout]
|
||||
tag = "TRAIN-FIT (all 9)" if args.train_all else "HELD-OUT"
|
||||
print(f"\n=== {tag} boundary detection (knee, NMS, ±{args.tol}s) ===")
|
||||
print(f"{'film':26s} {'TP':>4}{'FP':>5}{'FN':>5} {'P':>5}{'R':>5}{'F1':>5} {'gray F1':>7}")
|
||||
rep = {"win": WIN, "sigma": SIGMA, "tol": args.tol, "train_all": args.train_all,
|
||||
"holdout": args.holdout, "films": {}}
|
||||
f1s, gf1s = [], []
|
||||
for f in ev:
|
||||
X, yb, ic = per_second_matrix(f["dump"], f["xray"], args.audio_dir)
|
||||
prob = np.clip(reg.predict(X), 0, 1)
|
||||
peaks = knee_boundaries(prob)
|
||||
Tset = np.where(yb > 0.5)[0]
|
||||
P, R, F, tp, fp, fn = prf(peaks, Tset, args.tol)
|
||||
gpk = nms_peaks(ic.astype(float)); _, _, gF, *_ = prf(gpk, Tset, args.tol)
|
||||
f1s.append(F); gf1s.append(gF)
|
||||
rep["films"][f["slug"]] = {"TP": tp, "FP": fp, "FN": fn, "P": P, "R": R, "F1": F,
|
||||
"n_pred": len(peaks), "n_true": len(Tset), "gray_F1": gF}
|
||||
print(f"{f['slug'][:26]:26s} {tp:>4}{fp:>5}{fn:>5} {P*100:4.0f}%{R*100:4.0f}%"
|
||||
f"{F*100:4.0f}% {gF*100:5.0f}%")
|
||||
print(f"\nmacro-F1: detector {np.mean(f1s)*100:.1f}% grayscale {np.mean(gf1s)*100:.1f}%")
|
||||
rep["macro_f1"] = {"detector": float(np.mean(f1s)), "grayscale": float(np.mean(gf1s))}
|
||||
if args.train_all:
|
||||
reg.save_model(str(Path(args.out) / "xgb_boundary_shipped.json"))
|
||||
print(f"[xgb] shipped model → {args.out}/xgb_boundary_shipped.json", file=sys.stderr)
|
||||
json.dump(rep, open(Path(args.out) / "xgb_report.json", "w"), indent=2)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,81 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
train_xgb_cpp.py — train the scene-boundary XGBoost on the C++-EXTRACTED feature
|
||||
matrices (experiments/dumps/cpp_features/<slug>.h5, written by scene_features_dump).
|
||||
|
||||
This is the parity-by-construction path: the model is fit on exactly the features
|
||||
the C++ XGBSceneBoundary produces at inference, so C++ boundaries match by
|
||||
construction — no numpy-vs-C++ feature drift to chase. Same soft Gaussian target,
|
||||
knee threshold, and ±20s eval as train_xgb_boundary.py.
|
||||
|
||||
Usage (train all 9 + save shipped model):
|
||||
.venv-rocm/bin/python scripts/scene_detector/train_xgb_cpp.py --train-all
|
||||
"""
|
||||
from __future__ import annotations
|
||||
import argparse, json, sys
|
||||
from pathlib import Path
|
||||
import numpy as np, h5py
|
||||
sys.path.insert(0, "scripts/scene_detector")
|
||||
from train_scene_boundary import load_xray_boundaries, nms_peaks
|
||||
from train_xgb_boundary import knee_boundaries, prf, SIGMA
|
||||
import xgboost as xgb
|
||||
|
||||
CPP_DIR = "experiments/dumps/cpp_features"
|
||||
|
||||
|
||||
def load(slug, xray):
|
||||
with h5py.File(f"{CPP_DIR}/{slug}.h5") as f:
|
||||
X = f["features"][:].astype(np.float32)
|
||||
ts = f["timestamp_sec"][:]
|
||||
b = np.array(load_xray_boundaries(xray))
|
||||
y = np.zeros(len(ts), np.float32)
|
||||
if len(b):
|
||||
for i, t in enumerate(ts):
|
||||
y[i] = np.exp(-((np.min(np.abs(b - t)))/SIGMA)**2)
|
||||
yb = np.zeros(len(ts), np.float32)
|
||||
for bb in b:
|
||||
yb[np.abs(ts - bb) <= 2.0] = 1.0
|
||||
return X, y, yb, ts
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--manifest", default="experiments/manifests/films_LVFace_opencv5.json")
|
||||
ap.add_argument("--holdout", nargs="*", default=[])
|
||||
ap.add_argument("--train-all", action="store_true")
|
||||
ap.add_argument("--tol", type=int, default=20)
|
||||
ap.add_argument("--out", default="experiments/results/scene_boundary")
|
||||
args = ap.parse_args()
|
||||
films = json.load(open(args.manifest))
|
||||
Path(args.out).mkdir(parents=True, exist_ok=True)
|
||||
|
||||
tr = films if args.train_all else [f for f in films if f["slug"] not in args.holdout]
|
||||
Xtr = np.concatenate([load(f["slug"], f["xray"])[0] for f in tr])
|
||||
ytr = np.concatenate([load(f["slug"], f["xray"])[1] for f in tr])
|
||||
reg = xgb.XGBRegressor(n_estimators=400, max_depth=5, learning_rate=0.05,
|
||||
subsample=0.8, colsample_bytree=0.8,
|
||||
objective="reg:squarederror", n_jobs=8, tree_method="hist")
|
||||
reg.fit(Xtr, ytr)
|
||||
print(f"[xgb-cpp] trained on {len(tr)} films", file=sys.stderr)
|
||||
|
||||
ev = films if args.train_all else [f for f in films if f["slug"] in args.holdout]
|
||||
tag = "TRAIN-FIT (all 9)" if args.train_all else "HELD-OUT"
|
||||
print(f"\n=== {tag} (C++ features, knee, ±{args.tol}s) ===")
|
||||
print(f"{'film':26s} {'TP':>4}{'FP':>5}{'FN':>5} {'P':>5}{'R':>5}{'F1':>5}")
|
||||
f1s = []
|
||||
for f in ev:
|
||||
X, y, yb, ts = load(f["slug"], f["xray"])
|
||||
prob = np.clip(reg.predict(X), 0, 1)
|
||||
peaks = knee_boundaries(prob)
|
||||
Tset = np.where(yb > 0.5)[0]
|
||||
P, R, F, tp, fp, fn = prf(peaks, Tset, args.tol)
|
||||
f1s.append(F)
|
||||
print(f"{f['slug'][:26]:26s} {tp:>4}{fp:>5}{fn:>5} {P*100:4.0f}%{R*100:4.0f}%{F*100:4.0f}%")
|
||||
print(f"\nmacro-F1: {np.mean(f1s)*100:.1f}%")
|
||||
if args.train_all:
|
||||
reg.save_model(str(Path(args.out) / "xgb_boundary_cpp.json"))
|
||||
print(f"[xgb-cpp] shipped model → {args.out}/xgb_boundary_cpp.json", file=sys.stderr)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -17,11 +17,6 @@ X-Ray/MovieNet key on IMDb nm-ids — see [[per-scene-presence-eval-design]].
|
||||
Two sources implemented:
|
||||
* XRayGroundTruth — Zenodo scene-level Amazon X-Ray CSVs (cast-in-scene).
|
||||
* MovieNetGroundTruth — MovieNet-PS per-shot face annotations (on-screen faces).
|
||||
|
||||
Both are published corpora addressed by title, so a scoring run is reproducible
|
||||
from the identifiers alone — no annotation of ours travels with the code.
|
||||
|
||||
TRACES: VR-004 | PR-002
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -183,10 +183,16 @@ DEDUP_SIM = 1.0 - 1e-7
|
||||
class Stages:
|
||||
"""Thin holder so the rest of the script has one object to call."""
|
||||
|
||||
def __init__(self, detector: str, arcface: str, conf: float, nms: float):
|
||||
def __init__(self, detector: str, arcface: str, conf: float, nms: float,
|
||||
detector_engine: str = "", arcface_engine: str = ""):
|
||||
# The engine paths are only consulted by a TRT-backend build, where they
|
||||
# are mandatory — that backend loads a pre-built .engine and will not
|
||||
# fall back to reading the .onnx. An ORT build ignores them, so passing
|
||||
# them unconditionally is safe and keeps one constructor for both.
|
||||
self.engine = sae_embed.FaceEmbedder(
|
||||
detector_model=detector, arcface_model=arcface,
|
||||
conf=conf, nms=nms, max_side=0)
|
||||
conf=conf, nms=nms, max_side=0,
|
||||
detector_engine=detector_engine, arcface_engine=arcface_engine)
|
||||
|
||||
def detect(self, img):
|
||||
return self.engine.detect(img)
|
||||
|
||||
@@ -0,0 +1,667 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
quality_knee.py — VR-012: what does a blurred or small face cost in identification,
|
||||
and which sharpness measure predicts it?
|
||||
|
||||
TRACES: VR-012, AR-028, AR-029
|
||||
|
||||
VR-005 located the size floor by degrading held-out gallery mugshots and watching
|
||||
TPI/FPI fall. This does the same over a **joint size x blur grid**, and adds the
|
||||
part that makes the result usable at inference.
|
||||
|
||||
Why a joint grid and not two sweeps
|
||||
-----------------------------------
|
||||
A 16 px face upscaled to 112 has already lost its high frequencies, so additional
|
||||
blur costs it far less than it costs a 112 px one. Sweeping the axes separately
|
||||
measures each in the presence of an implicit "other axis at its best" and misses
|
||||
that interaction entirely — and the interaction is the whole question, because
|
||||
AR-002 already gates on size and AR-029 proposes to discount on sharpness. If
|
||||
identity loss turns out to be a function of the sharpness measure alone, then one
|
||||
axis carries the information and discounting on both double-counts. If a
|
||||
small-but-sharp and a large-but-blurred probe at equal measure lose different
|
||||
amounts, the axes are genuinely separate and both belong.
|
||||
|
||||
Why sigma is not the answer
|
||||
---------------------------
|
||||
Sigma is a lab variable. At inference nothing knows how blurred a face is, so a
|
||||
knee expressed in sigma cannot be acted on. What AR-028/AR-030 can consume is
|
||||
measure value -> expected identity reliability
|
||||
so the controlled degradation exists to *select and calibrate the measure*, and
|
||||
the measure is what ships. Every candidate is therefore scored on every degraded
|
||||
crop, and the candidates are ranked by how well each predicts the identification
|
||||
outcome (AUC over probe-cell records), not by how smooth its ladder looks.
|
||||
|
||||
Protocol (VR-005's, extended)
|
||||
-----------------------------
|
||||
1. Every gallery actor with at least `--min-images` mugshots. At the default 3,
|
||||
holding one out still leaves two references per actor.
|
||||
2. Hold out ONE image per actor as the probe; the rest stay in the gallery at
|
||||
native resolution. Only the probe degrades — reference mugshots are clean and
|
||||
the face coming out of the video is not, which is the production case.
|
||||
3. For each (size, sigma) cell: downscale the probe crop to size x size and back
|
||||
to 112 (the sampling loss), then Gaussian blur at sigma canonical px (the
|
||||
optical/motion loss). Resolution first, then blur, so sigma always means the
|
||||
same thing in the frame AR-029 measures in, whatever the cell's size.
|
||||
4. Score all five AR-029 candidates on the degraded crop, through the C++
|
||||
binding.
|
||||
5. Embed, match against the whole gallery, record TPI/FPI/unidentified.
|
||||
|
||||
Decision rule is the pipeline's: per-actor best-of-N cosine -> Platt sigmoid ->
|
||||
accept if P > prob_threshold. Never a raw cosine (CLAUDE.md invariant, AR-024).
|
||||
|
||||
Everything runs through `sae_embed` — detection, the ArcFace warp, the embedder,
|
||||
the sharpness measures and the calibration are all the shipped C++. Nothing here
|
||||
re-implements a pipeline stage in numpy; the analysis on top of the recorded
|
||||
numbers (AUC, knee location) is analysis and is numpy's job.
|
||||
|
||||
CAVEAT — FPI IS RELATIVE, NOT ABSOLUTE
|
||||
--------------------------------------
|
||||
False positives grow with the number of actors competing. Read FPI as a curve
|
||||
across cells, not as a production rate. This runs the whole eligible gallery
|
||||
rather than VR-005's 100-actor sample, so the understatement is much smaller,
|
||||
but a production library is larger still.
|
||||
|
||||
Usage
|
||||
-----
|
||||
python scripts/validation/quality_knee.py \
|
||||
--images images --gallery gallery_lvface.h5 \
|
||||
--arcface models/LVFace-B_Glint360K.onnx \
|
||||
--min-images 3 --out experiments/results/vr012_quality_knee
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import csv
|
||||
import json
|
||||
import random
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
REPO = Path(__file__).resolve().parent.parent.parent
|
||||
sys.path.insert(0, str(REPO / "scripts"))
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
|
||||
# min_face_size owns the shared scaffolding — actor discovery, the sae_embed
|
||||
# locator, the Stages wrapper, the calibration-through-the-binding and the house
|
||||
# plot palette. Importing it keeps one copy of each; a second copy of the
|
||||
# calibration path in particular is what AR-024 exists to prevent.
|
||||
import min_face_size as vr005 # noqa: E402
|
||||
from min_face_size import ( # noqa: E402
|
||||
DEDUP_SIM, INTERP, Stages, calibrate_gallery, discover_actors,
|
||||
gallery_keys, normalise_name, probability, err,
|
||||
INK, MUTED, GRID, SURFACE, BLUE, GREEN, RED, AMBER,
|
||||
)
|
||||
|
||||
import cv2 # noqa: E402
|
||||
import numpy as np # noqa: E402
|
||||
import sae_embed # noqa: E402
|
||||
|
||||
# The five AR-029 candidates, in quality.hpp's order. Names match the binding's
|
||||
# attributes so the CSV columns and the C++ fields cannot drift apart.
|
||||
MEASURES = ["var_laplacian", "norm_var_laplacian", "tenengrad",
|
||||
"hf_energy_ratio", "dir_min_tenengrad"]
|
||||
|
||||
|
||||
# ── Degradation ───────────────────────────────────────────────────────────────
|
||||
|
||||
def disc_kernel(radius: float) -> np.ndarray:
|
||||
"""The circle-of-confusion PSF of a defocused lens.
|
||||
|
||||
Optical defocus is **not** Gaussian, and the difference is not cosmetic. A
|
||||
lens out of focus spreads a point into a uniform disc, whose transfer
|
||||
function is a jinc — `2·J1(x)/x` — which crosses zero and goes negative.
|
||||
Defocus therefore reverses contrast at particular spatial frequencies and
|
||||
can leave *more* energy in some high bands than a Gaussian of the same
|
||||
nominal width. A Gaussian MTF is strictly positive and monotonically
|
||||
decreasing and does neither.
|
||||
|
||||
That matters here beyond realism: defocus is how a face ends up **large and
|
||||
useless**. A focus pull, a shallow depth of field, an actor stepping off the
|
||||
focal plane — all leave a big, confidently-detected face carrying no usable
|
||||
detail, and all sail straight through a size gate. Gaussian blur was the one
|
||||
family that mostly co-occurs with small faces, which is precisely why
|
||||
sharpness looked redundant against AR-002 on the first grid.
|
||||
|
||||
The disc is supersampled 8x before downsampling so its edge is
|
||||
anti-aliased; a hard-edged binary disc at small radii is a poor circle and
|
||||
its spectrum carries the staircase, not the optics.
|
||||
"""
|
||||
ss = 8
|
||||
n = int(np.ceil(radius)) * 2 + 1
|
||||
hi = np.zeros((n * ss, n * ss), np.float32)
|
||||
c = (n * ss - 1) / 2.0
|
||||
y, x = np.ogrid[:n * ss, :n * ss]
|
||||
hi[((x - c) ** 2 + (y - c) ** 2) <= (radius * ss) ** 2] = 1.0
|
||||
k = hi.reshape(n, ss, n, ss).mean(axis=(1, 3))
|
||||
s = k.sum()
|
||||
return (k / s) if s > 0 else np.ones((1, 1), np.float32)
|
||||
|
||||
|
||||
def motion_kernel(length: int, angle_deg: float) -> np.ndarray:
|
||||
"""Linear motion blur — a camera pan or a moving subject.
|
||||
|
||||
Directional by construction: it destroys detail along one axis and leaves
|
||||
the perpendicular axis untouched. That is the property that separates the
|
||||
AR-029 candidates, since a measure normalising by total energy divides out
|
||||
the loss and reads a heavy smear as mild (see tests/test_quality.cpp).
|
||||
"""
|
||||
k = np.zeros((length, length), np.float32)
|
||||
k[length // 2, :] = 1.0
|
||||
m = cv2.getRotationMatrix2D(((length - 1) / 2.0, (length - 1) / 2.0),
|
||||
angle_deg, 1.0)
|
||||
k = cv2.warpAffine(k, m, (length, length))
|
||||
s = k.sum()
|
||||
return (k / s) if s > 0 else np.ones((1, 1), np.float32)
|
||||
|
||||
|
||||
def degrade(crop: np.ndarray, size: int, level: float, kind: str,
|
||||
down: int, up: int, angle: float = 0.0) -> np.ndarray:
|
||||
"""Resolution loss, then blur of the requested family.
|
||||
|
||||
Order matters and this one is deliberate. Sampling happens in the source
|
||||
frame, so the downscale/upscale pair models a face that was `size` px when
|
||||
detected. The blur is then applied in the canonical frame, so `level` means
|
||||
the same number of canonical pixels in every cell of the grid — which is what
|
||||
lets the two axes be read independently. Blurring first would make the
|
||||
effective width depend on the cell's size, and the grid would no longer be
|
||||
factorial.
|
||||
|
||||
`level` is the family's natural parameter: Gaussian sigma, disc radius, or
|
||||
motion length in canonical px. They are NOT equivalent at equal numbers —
|
||||
matching families by parameter would compare different amounts of damage, so
|
||||
the analysis matches them on measured effect instead.
|
||||
"""
|
||||
out = crop
|
||||
if size != 112:
|
||||
small = cv2.resize(out, (size, size), interpolation=down)
|
||||
out = cv2.resize(small, (112, 112), interpolation=up)
|
||||
if level > 0:
|
||||
if kind == "gaussian":
|
||||
out = cv2.GaussianBlur(out, (0, 0), level, level)
|
||||
elif kind == "disc":
|
||||
out = cv2.filter2D(out, -1, disc_kernel(level))
|
||||
elif kind == "motion":
|
||||
out = cv2.filter2D(out, -1, motion_kernel(int(round(level)), angle))
|
||||
else:
|
||||
raise ValueError(f"unknown blur kind: {kind}")
|
||||
return out
|
||||
|
||||
|
||||
def score_sharpness(crop: np.ndarray) -> dict:
|
||||
"""All five candidates, from the shipped C++ (quality.hpp)."""
|
||||
s = sae_embed.assess_sharpness(np.ascontiguousarray(crop))
|
||||
d = {m: float(getattr(s, m)) for m in MEASURES}
|
||||
d["ok"] = bool(s.ok)
|
||||
return d
|
||||
|
||||
|
||||
# ── Analysis ──────────────────────────────────────────────────────────────────
|
||||
|
||||
def auc(scores: np.ndarray, positive: np.ndarray) -> float:
|
||||
"""Area under the ROC for `scores` predicting `positive`, by the rank
|
||||
(Mann-Whitney U) identity. 0.5 is chance; 1.0 is a measure that orders every
|
||||
correctly-identified probe above every failure.
|
||||
|
||||
This is the ranking criterion for AR-029. A measure earns the job by
|
||||
predicting *the decision the pipeline makes*, not by having a tidy response
|
||||
to synthetic blur — a candidate can be beautifully monotone in sigma and
|
||||
still be a poor guide to whether this particular face will be recognised.
|
||||
"""
|
||||
pos = scores[positive]
|
||||
neg = scores[~positive]
|
||||
if pos.size == 0 or neg.size == 0:
|
||||
return float("nan")
|
||||
order = np.argsort(np.concatenate([pos, neg]), kind="mergesort")
|
||||
ranks = np.empty(order.size, dtype=np.float64)
|
||||
ranks[order] = np.arange(1, order.size + 1)
|
||||
# Average ranks over ties, or a measure with many equal values is scored
|
||||
# arbitrarily by input order.
|
||||
vals = np.concatenate([pos, neg])
|
||||
sv = vals[order]
|
||||
i = 0
|
||||
while i < sv.size:
|
||||
j = i
|
||||
while j + 1 < sv.size and sv[j + 1] == sv[i]:
|
||||
j += 1
|
||||
if j > i:
|
||||
ranks[order[i:j + 1]] = ranks[order[i:j + 1]].mean()
|
||||
i = j + 1
|
||||
r_pos = ranks[:pos.size].sum()
|
||||
return float((r_pos - pos.size * (pos.size + 1) / 2) / (pos.size * neg.size))
|
||||
|
||||
|
||||
def knee_from_measure(records: list[dict], measure: str, retention: float,
|
||||
n_bins: int = 20) -> dict:
|
||||
"""Where on `measure`'s own scale does identification start to fall apart?
|
||||
|
||||
Bins the probe-cell records by measure value and reports the TPI rate in
|
||||
each. The threshold is the lowest bin edge whose bin and every bin above it
|
||||
retain `retention` of the undegraded control's TPI rate — a stated rule, so
|
||||
changing the answer means changing the rule rather than picking a number.
|
||||
"""
|
||||
vals = np.array([r[measure] for r in records], dtype=np.float64)
|
||||
tpi = np.array([r["outcome"] == "TPI" for r in records])
|
||||
control = np.array([r["size_px"] == 112 and r["sigma"] == 0.0 for r in records])
|
||||
if control.sum() == 0:
|
||||
return {}
|
||||
floor = retention * float(tpi[control].mean())
|
||||
|
||||
# Quantile edges: the measures have wildly different scales and heavy tails,
|
||||
# so equal-width bins would put almost everything in one bucket.
|
||||
edges = np.unique(np.quantile(vals, np.linspace(0, 1, n_bins + 1)))
|
||||
if edges.size < 3:
|
||||
return {}
|
||||
idx = np.clip(np.digitize(vals, edges[1:-1]), 0, edges.size - 2)
|
||||
|
||||
bins = []
|
||||
for b in range(edges.size - 1):
|
||||
m = idx == b
|
||||
if m.sum() == 0:
|
||||
continue
|
||||
bins.append({"lo": float(edges[b]), "hi": float(edges[b + 1]),
|
||||
"n": int(m.sum()), "tpi_rate": float(tpi[m].mean()),
|
||||
"fpi_rate": float(np.mean([r["outcome"] == "FPI"
|
||||
for r, k in zip(records, m) if k]))})
|
||||
# Walk down from the top; the threshold is where retention first breaks.
|
||||
thr = None
|
||||
for b in reversed(bins):
|
||||
if b["tpi_rate"] < floor:
|
||||
thr = b["hi"]
|
||||
break
|
||||
return {"measure": measure, "control_tpi": float(tpi[control].mean()),
|
||||
"tpi_floor": floor, "threshold": thr, "bins": bins}
|
||||
|
||||
|
||||
# ── Plot ──────────────────────────────────────────────────────────────────────
|
||||
|
||||
def write_plots(cells: list[dict], records: list[dict], ranking: list[dict],
|
||||
out_png: Path, meta: dict) -> None:
|
||||
import matplotlib
|
||||
matplotlib.use("Agg")
|
||||
import matplotlib.pyplot as plt
|
||||
|
||||
plt.rcParams.update({
|
||||
"figure.facecolor": SURFACE, "axes.facecolor": SURFACE,
|
||||
"savefig.facecolor": SURFACE, "text.color": INK,
|
||||
"axes.edgecolor": MUTED, "axes.labelcolor": INK,
|
||||
"xtick.color": MUTED, "ytick.color": MUTED,
|
||||
"axes.grid": True, "grid.color": GRID, "grid.linewidth": 0.8,
|
||||
"axes.spines.top": False, "axes.spines.right": False,
|
||||
})
|
||||
|
||||
sizes = sorted({c["size_px"] for c in cells})
|
||||
sigmas = sorted({c["sigma"] for c in cells})
|
||||
fig, axes = plt.subplots(1, 3, figsize=(17, 5.4))
|
||||
|
||||
# (a) the joint grid as TPI heat map
|
||||
grid = np.full((len(sigmas), len(sizes)), np.nan)
|
||||
for c in cells:
|
||||
grid[sigmas.index(c["sigma"]), sizes.index(c["size_px"])] = 100 * c["tpi_rate"]
|
||||
im = axes[0].imshow(grid, origin="lower", aspect="auto", cmap="viridis",
|
||||
vmin=0, vmax=100)
|
||||
axes[0].set_xticks(range(len(sizes)), [str(s) for s in sizes])
|
||||
axes[0].set_yticks(range(len(sigmas)), [f"{s:g}" for s in sigmas])
|
||||
axes[0].set_xlabel("probe size before upscaling (px)")
|
||||
axes[0].set_ylabel("Gaussian sigma (canonical px)")
|
||||
axes[0].set_title("TPI % over the joint grid", fontsize=11, loc="left")
|
||||
axes[0].grid(False)
|
||||
fig.colorbar(im, ax=axes[0], fraction=0.046)
|
||||
|
||||
# (b) TPI against the winning measure — the curve a discount is built from
|
||||
best = ranking[0]["measure"]
|
||||
vals = np.array([r[best] for r in records])
|
||||
tpi = np.array([r["outcome"] == "TPI" for r in records])
|
||||
edges = np.unique(np.quantile(vals, np.linspace(0, 1, 21)))
|
||||
centres, rates = [], []
|
||||
for i in range(edges.size - 1):
|
||||
m = (vals >= edges[i]) & (vals <= edges[i + 1])
|
||||
if m.sum() > 20:
|
||||
centres.append(0.5 * (edges[i] + edges[i + 1]))
|
||||
rates.append(100 * tpi[m].mean())
|
||||
axes[1].plot(centres, rates, "-o", color=GREEN, lw=2)
|
||||
axes[1].set_xscale("log")
|
||||
axes[1].set_xlabel(f"{best} (log scale)")
|
||||
axes[1].set_ylabel("TPI %")
|
||||
axes[1].set_title(f"identification vs the measure\nbest predictor: {best} "
|
||||
f"(AUC {ranking[0]['auc']:.3f})", fontsize=11, loc="left")
|
||||
|
||||
# (c) how well each candidate predicts the decision
|
||||
names = [r["measure"] for r in ranking]
|
||||
aucs = [r["auc"] for r in ranking]
|
||||
axes[2].barh(range(len(names)), aucs, color=BLUE)
|
||||
axes[2].axvline(0.5, color=RED, lw=1.4, ls="--")
|
||||
axes[2].set_yticks(range(len(names)), names, fontsize=9)
|
||||
axes[2].set_xlim(0.4, 1.0)
|
||||
axes[2].set_xlabel("AUC — predicts correct identification")
|
||||
axes[2].set_title("AR-029 candidate ranking", fontsize=11, loc="left")
|
||||
axes[2].invert_yaxis()
|
||||
|
||||
fig.suptitle(f"VR-012 — quality knee, {meta['model']}, {meta['n_actors']} actors, "
|
||||
f"{meta['n_probes']} probes x {len(cells)} cells",
|
||||
fontsize=12, x=0.01, ha="left")
|
||||
fig.tight_layout(rect=(0, 0.02, 1, 0.97))
|
||||
out_png.parent.mkdir(parents=True, exist_ok=True)
|
||||
fig.savefig(out_png, dpi=150)
|
||||
plt.close(fig)
|
||||
|
||||
|
||||
# ── Main ──────────────────────────────────────────────────────────────────────
|
||||
|
||||
def main() -> int:
|
||||
p = argparse.ArgumentParser(
|
||||
description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
p.add_argument("--images", default=str(REPO / "images"))
|
||||
p.add_argument("--gallery", default=str(REPO / "gallery_lvface.h5"))
|
||||
p.add_argument("--out", default=str(REPO / "experiments/results/vr012_quality_knee"))
|
||||
p.add_argument("--actors", type=int, default=0,
|
||||
help="cap the actor pool (0 = every eligible actor, the default: "
|
||||
"FPI is gallery-size dependent and the whole gallery is the "
|
||||
"least understated estimate available)")
|
||||
p.add_argument("--min-images", type=int, default=3,
|
||||
help="minimum mugshots to be eligible (default 3, so holding one "
|
||||
"out still leaves two references)")
|
||||
p.add_argument("--seed", type=int, default=0)
|
||||
p.add_argument("--sizes", default="16,24,32,48,64,112",
|
||||
help="probe sizes before upscaling; 112 is undegraded")
|
||||
p.add_argument("--sigmas", default="0,0.5,1,1.5,2,3",
|
||||
help="blur level in canonical px; 0 is unblurred. Meaning "
|
||||
"depends on --blur-kind: Gaussian sigma, disc radius, "
|
||||
"or motion length")
|
||||
p.add_argument("--blur-kind", default="gaussian",
|
||||
choices=["gaussian", "disc", "motion"],
|
||||
help="blur family. gaussian is a soft-focus stand-in; disc "
|
||||
"is the circle-of-confusion PSF of real optical "
|
||||
"defocus (non-Gaussian, jinc MTF with zero crossings); "
|
||||
"motion is a linear smear. The last two are how a face "
|
||||
"ends up large and useless, which a size gate cannot "
|
||||
"catch")
|
||||
p.add_argument("--motion-angle", type=float, default=0.0,
|
||||
help="motion blur direction in degrees (--blur-kind motion)")
|
||||
p.add_argument("--keep-duplicates", action="store_true")
|
||||
|
||||
p.add_argument("--models-dir", default=str(REPO / "models"))
|
||||
p.add_argument("--arcface", default=None)
|
||||
p.add_argument("--detector", default=None)
|
||||
p.add_argument("--conf", type=float, default=0.5)
|
||||
p.add_argument("--nms", type=float, default=0.4)
|
||||
p.add_argument("--max-side", type=int, default=500)
|
||||
# Required by a TRT-backend build, ignored by an ORT one. A TensorRT fp16
|
||||
# run is a different realisation of the embedder — VR-005 measured ~0.85
|
||||
# cosine agreement with the fp32 ONNX path on LVFace-B, with separation
|
||||
# essentially intact — so a knee located here belongs to the fp16 space.
|
||||
# The study stays internally consistent because gallery and probes are both
|
||||
# embedded in this one session.
|
||||
p.add_argument("--detector-engine", default="",
|
||||
help="pre-built SCRFD .engine (TRT builds only)")
|
||||
p.add_argument("--arcface-engine", default="",
|
||||
help="pre-built ArcFace .engine (TRT builds only)")
|
||||
|
||||
p.add_argument("--prob-threshold", type=float, default=0.754)
|
||||
p.add_argument("--match-prior", type=float, default=0.5)
|
||||
p.add_argument("--tpi-retention", type=float, default=0.95)
|
||||
p.add_argument("--down-interp", default="area", choices=sorted(INTERP))
|
||||
p.add_argument("--up-interp", default="linear", choices=sorted(INTERP))
|
||||
args = p.parse_args()
|
||||
|
||||
models_dir = Path(args.models_dir)
|
||||
arcface = Path(args.arcface) if args.arcface else models_dir / "LVFace-B_Glint360K.onnx"
|
||||
detector = Path(args.detector) if args.detector else models_dir / "scrfd_500m_bnkps.onnx"
|
||||
for path, what in ((arcface, "embedder"), (detector, "detector")):
|
||||
if not path.is_file():
|
||||
return err(f"{what} model not found: {path}")
|
||||
|
||||
images_root = Path(args.images)
|
||||
if not images_root.is_dir():
|
||||
return err(f"image cache not found: {images_root}")
|
||||
|
||||
sizes = sorted({int(s) for s in args.sizes.split(",") if s.strip()})
|
||||
sigmas = sorted({float(s) for s in args.sigmas.split(",") if s.strip()})
|
||||
cv2.setRNGSeed(args.seed)
|
||||
|
||||
# ── actor pool ────────────────────────────────────────────────────────────
|
||||
pool = discover_actors(images_root)
|
||||
print(f"[select] {len(pool)} actor dirs under {images_root}", file=sys.stderr)
|
||||
if args.gallery and Path(args.gallery).is_file():
|
||||
ids, names = gallery_keys(Path(args.gallery))
|
||||
pool = [a for a in pool
|
||||
if (a["jellyfin_id"] and a["jellyfin_id"] in ids)
|
||||
or normalise_name(a["name"]) in names]
|
||||
print(f"[select] {len(pool)} are in {args.gallery}", file=sys.stderr)
|
||||
|
||||
eligible = [a for a in pool if len(a["images"]) >= args.min_images]
|
||||
print(f"[select] {len(eligible)} have >= {args.min_images} mugshots",
|
||||
file=sys.stderr)
|
||||
if len(eligible) < 2:
|
||||
return err(f"need at least 2 eligible actors; found {len(eligible)}")
|
||||
|
||||
rng = random.Random(args.seed)
|
||||
selected = (sorted(rng.sample(eligible, min(args.actors, len(eligible))),
|
||||
key=lambda a: a["dir"].name)
|
||||
if args.actors else eligible)
|
||||
|
||||
# ── detect + align every mugshot once ─────────────────────────────────────
|
||||
stages = Stages(str(detector), str(arcface), args.conf, args.nms,
|
||||
args.detector_engine, args.arcface_engine)
|
||||
print(f"[models] detector={detector.name} embedder={arcface.name} "
|
||||
f"batch={stages.engine.max_batch}", file=sys.stderr)
|
||||
|
||||
t0 = time.time()
|
||||
crops, rows, actors = [], [], []
|
||||
n_nodetect = 0
|
||||
for a in selected:
|
||||
actor_crops, actor_paths = [], []
|
||||
for img_path in a["images"]:
|
||||
img = cv2.imread(str(img_path))
|
||||
if img is None:
|
||||
n_nodetect += 1
|
||||
continue
|
||||
if args.max_side > 0 and max(img.shape[:2]) > args.max_side:
|
||||
s = args.max_side / max(img.shape[:2])
|
||||
img = cv2.resize(img, None, fx=s, fy=s, interpolation=cv2.INTER_AREA)
|
||||
faces = stages.detect(img)
|
||||
if not faces:
|
||||
enhanced = stages.enhance(img)
|
||||
faces = stages.detect(enhanced)
|
||||
if faces:
|
||||
img = enhanced
|
||||
if not faces:
|
||||
n_nodetect += 1
|
||||
continue
|
||||
best = max(faces, key=lambda f: f.confidence)
|
||||
crop = stages.align(img, best.landmarks)
|
||||
if crop is None:
|
||||
n_nodetect += 1
|
||||
continue
|
||||
actor_crops.append(crop)
|
||||
actor_paths.append(img_path)
|
||||
if len(actor_crops) < 2:
|
||||
continue
|
||||
ai = len(actors)
|
||||
actors.append({"name": a["name"], "jellyfin_id": a["jellyfin_id"],
|
||||
"dir": a["dir"].name, "n_images": len(actor_crops)})
|
||||
for crop, img_path in zip(actor_crops, actor_paths):
|
||||
rows.append({"actor_idx": ai, "image": str(img_path)})
|
||||
crops.append(crop)
|
||||
if len(actors) % 200 == 0:
|
||||
print(f" [align] {len(actors)}/{len(selected)} actors, "
|
||||
f"{len(crops)} crops", file=sys.stderr)
|
||||
|
||||
if len(actors) < 2:
|
||||
return err(f"only {len(actors)} actors survived detection/alignment")
|
||||
print(f"[align] {len(actors)} actors, {len(crops)} crops, {n_nodetect} skipped "
|
||||
f"in {time.time() - t0:.1f}s", file=sys.stderr)
|
||||
|
||||
actor_of = np.array([r["actor_idx"] for r in rows], dtype=int)
|
||||
|
||||
t0 = time.time()
|
||||
native = stages.embed(crops)
|
||||
print(f"[embed] {len(crops)} native crops in {time.time() - t0:.1f}s",
|
||||
file=sys.stderr)
|
||||
|
||||
# ── drop duplicate mugshots ───────────────────────────────────────────────
|
||||
if not args.keep_duplicates:
|
||||
keep = np.ones(len(rows), bool)
|
||||
for ai in range(len(actors)):
|
||||
kept: list[int] = []
|
||||
for i in np.nonzero(actor_of == ai)[0]:
|
||||
if any(float(native[i] @ native[k]) > DEDUP_SIM for k in kept):
|
||||
keep[i] = False
|
||||
else:
|
||||
kept.append(int(i))
|
||||
n_dup = int((~keep).sum())
|
||||
counts = np.bincount(actor_of[keep], minlength=len(actors))
|
||||
drop_actor = counts < 2
|
||||
keep &= ~drop_actor[actor_of]
|
||||
remap = np.full(len(actors), -1, dtype=int)
|
||||
remap[~drop_actor] = np.arange(int((~drop_actor).sum()))
|
||||
actors = [a for a, d in zip(actors, drop_actor) if not d]
|
||||
rows = [r for r, k in zip(rows, keep) if k]
|
||||
crops = [c for c, k in zip(crops, keep) if k]
|
||||
native = native[keep]
|
||||
actor_of = remap[actor_of[keep]]
|
||||
print(f"[dedup] dropped {n_dup} duplicates and {int(drop_actor.sum())} "
|
||||
f"actors; {len(actors)} actors, {len(rows)} images remain",
|
||||
file=sys.stderr)
|
||||
|
||||
# ── hold out one probe per actor ──────────────────────────────────────────
|
||||
is_probe = np.zeros(len(rows), bool)
|
||||
for ai in range(len(actors)):
|
||||
idx = np.nonzero(actor_of == ai)[0]
|
||||
r = random.Random(f"{args.seed}:{actors[ai]['dir']}")
|
||||
is_probe[r.choice(list(idx))] = True
|
||||
probe_rows = np.nonzero(is_probe)[0]
|
||||
gal_rows = np.nonzero(~is_probe)[0]
|
||||
print(f"[holdout] {len(probe_rows)} probes, {len(gal_rows)} gallery embeddings",
|
||||
file=sys.stderr)
|
||||
|
||||
gal_emb = native[gal_rows]
|
||||
gal_actor = actor_of[gal_rows]
|
||||
probe_actor = actor_of[probe_rows]
|
||||
actor_cols = [np.nonzero(gal_actor == ai)[0] for ai in range(len(actors))]
|
||||
if not all(len(c) for c in actor_cols):
|
||||
return err("an actor has no gallery references left; raise --min-images")
|
||||
|
||||
cal = calibrate_gallery(gal_emb, gal_actor)
|
||||
if not cal["valid"]:
|
||||
return err("calibration could not be fitted; this study will not fall back "
|
||||
"to a raw cosine threshold (CLAUDE.md invariant)")
|
||||
log_prior_odds = float(np.log(args.match_prior / (1.0 - args.match_prior)))
|
||||
|
||||
# ── the grid ──────────────────────────────────────────────────────────────
|
||||
down, up = INTERP[args.down_interp], INTERP[args.up_interp]
|
||||
probe_crops = [crops[i] for i in probe_rows]
|
||||
cells, records = [], []
|
||||
n = len(probe_rows)
|
||||
|
||||
for size in sizes:
|
||||
for sigma in sigmas:
|
||||
t0 = time.time()
|
||||
degraded = [degrade(c, size, sigma, args.blur_kind, down, up,
|
||||
args.motion_angle) for c in probe_crops]
|
||||
sharp = [score_sharpness(d) for d in degraded]
|
||||
q = stages.embed(degraded)
|
||||
|
||||
sims = q @ gal_emb.T
|
||||
best_per_actor = np.stack([sims[:, c].max(axis=1) for c in actor_cols],
|
||||
axis=1)
|
||||
best_actor = best_per_actor.argmax(axis=1)
|
||||
best_sim = best_per_actor.max(axis=1)
|
||||
p_match = np.asarray(probability(best_sim, cal["a"], cal["b"],
|
||||
log_prior_odds))
|
||||
accept = p_match > args.prob_threshold
|
||||
correct = best_actor == probe_actor
|
||||
tpi = int(np.sum(accept & correct))
|
||||
fpi = int(np.sum(accept & ~correct))
|
||||
unid = int(np.sum(~accept))
|
||||
|
||||
cell = {"size_px": size, "sigma": sigma, "blur_kind": args.blur_kind,
|
||||
"n_probes": n,
|
||||
"tpi": tpi, "fpi": fpi, "unidentified": unid,
|
||||
"tpi_rate": tpi / n, "fpi_rate": fpi / n,
|
||||
"unidentified_rate": unid / n,
|
||||
"rank1_rate": float(np.mean(correct)),
|
||||
"mean_p_match": float(np.mean(p_match))}
|
||||
for m in MEASURES:
|
||||
cell[f"mean_{m}"] = float(np.mean([s[m] for s in sharp]))
|
||||
cells.append(cell)
|
||||
|
||||
for j in range(n):
|
||||
rec = {"size_px": size, "sigma": sigma,
|
||||
"blur_kind": args.blur_kind,
|
||||
"probe_image": rows[probe_rows[j]]["image"],
|
||||
"p_match": float(p_match[j]),
|
||||
"outcome": ("TPI" if accept[j] and correct[j]
|
||||
else "FPI" if accept[j] else "unidentified")}
|
||||
rec.update({m: sharp[j][m] for m in MEASURES})
|
||||
records.append(rec)
|
||||
|
||||
print(f"[grid] {size:3d}px {args.blur_kind[:4]} {sigma:<4g} TPI {100*tpi/n:5.1f}% "
|
||||
f"FPI {100*fpi/n:5.1f}% unid {100*unid/n:5.1f}% "
|
||||
f"rank1 {100*np.mean(correct):5.1f}% [{time.time()-t0:.1f}s]",
|
||||
file=sys.stderr)
|
||||
|
||||
# ── rank the candidates, then locate the knee on the winner ───────────────
|
||||
is_tpi = np.array([r["outcome"] == "TPI" for r in records])
|
||||
ranking = sorted(
|
||||
({"measure": m,
|
||||
"auc": auc(np.array([r[m] for r in records], dtype=np.float64), is_tpi)}
|
||||
for m in MEASURES),
|
||||
key=lambda d: -d["auc"])
|
||||
knees = [knee_from_measure(records, r["measure"], args.tpi_retention)
|
||||
for r in ranking]
|
||||
|
||||
# ── outputs ───────────────────────────────────────────────────────────────
|
||||
out = Path(args.out)
|
||||
out.parent.mkdir(parents=True, exist_ok=True)
|
||||
csv_path = out.with_name(out.name + ".csv")
|
||||
with open(csv_path, "w", newline="") as f:
|
||||
w = csv.DictWriter(f, fieldnames=list(cells[0].keys()))
|
||||
w.writeheader()
|
||||
w.writerows(cells)
|
||||
rec_path = out.with_name(out.name + ".records.csv")
|
||||
with open(rec_path, "w", newline="") as f:
|
||||
w = csv.DictWriter(f, fieldnames=list(records[0].keys()))
|
||||
w.writeheader()
|
||||
w.writerows(records)
|
||||
|
||||
meta = {
|
||||
"requirement": "VR-012",
|
||||
"model": arcface.stem, "detector": detector.stem,
|
||||
"n_actors": len(actors), "n_probes": len(probe_rows),
|
||||
"n_gallery_embeddings": len(gal_rows),
|
||||
"min_images": args.min_images, "seed": args.seed,
|
||||
"sizes": sizes, "sigmas": sigmas, "blur_kind": args.blur_kind,
|
||||
"motion_angle": args.motion_angle,
|
||||
"prob_threshold": args.prob_threshold, "match_prior": args.match_prior,
|
||||
"calibration": cal,
|
||||
"measure_ranking": ranking,
|
||||
"knees": knees,
|
||||
"sharpness_window": list(sae_embed.sharpness_window()),
|
||||
"caveat": (f"FPI grows with gallery size; this ran against {len(actors)} "
|
||||
f"actors and still understates a production library."),
|
||||
"grid": cells,
|
||||
}
|
||||
json_path = out.with_name(out.name + ".json")
|
||||
json_path.write_text(json.dumps(meta, indent=2) + "\n")
|
||||
png_path = out.with_name(out.name + ".png")
|
||||
write_plots(cells, records, ranking, png_path, meta)
|
||||
|
||||
# ── stdout report ─────────────────────────────────────────────────────────
|
||||
print(f"\nVR-012 — quality knee, {arcface.stem}")
|
||||
print(f"{len(actors)} actors, {len(probe_rows)} probes x {len(cells)} cells\n")
|
||||
print(f"{'size':>5} {'sigma':>6} {'TPI':>8} {'FPI':>8} {'unid':>8} {'rank1':>8}")
|
||||
for c in cells:
|
||||
print(f"{c['size_px']:>5} {c['sigma']:>6g} {100*c['tpi_rate']:>7.1f}% "
|
||||
f"{100*c['fpi_rate']:>7.1f}% {100*c['unidentified_rate']:>7.1f}% "
|
||||
f"{100*c['rank1_rate']:>7.1f}%")
|
||||
print("\nAR-029 candidate ranking — AUC for predicting correct identification:")
|
||||
for r in ranking:
|
||||
print(f" {r['measure']:>20} {r['auc']:.4f}")
|
||||
print(f"\n[out] {csv_path}\n[out] {rec_path}\n[out] {json_path}\n[out] {png_path}")
|
||||
print(f"\n{meta['caveat']}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -61,15 +61,7 @@ class Prediction:
|
||||
keys = keys_for(imdb_id=a.get("imdb_id"), tmdb_id=a.get("tmdb_id"),
|
||||
jellyfin_id=a.get("jellyfin_id"), name=a.get("name"),
|
||||
crosswalk=crosswalk)
|
||||
# schema_version 1: scenes is [[t0, t1], ...] (list of pairs)
|
||||
# schema_version 2: scenes is [{"start":…, "end":…, "belief":…, …}, …]
|
||||
windows = []
|
||||
for s in a.get("scenes", []):
|
||||
if isinstance(s, dict):
|
||||
windows.append((float(s["start"]), float(s["end"])))
|
||||
else:
|
||||
t0, t1 = s[0], s[1]
|
||||
windows.append((float(t0), float(t1)))
|
||||
windows = [(float(t0), float(t1)) for t0, t1 in a.get("scenes", [])]
|
||||
for _, t1 in windows:
|
||||
self._max_t = max(self._max_t, t1)
|
||||
self.actors.append({"keys": keys, "windows": windows})
|
||||
|
||||
@@ -51,7 +51,7 @@ sys.path.insert(0, str(BUILD))
|
||||
|
||||
import sae_audio # noqa: E402
|
||||
|
||||
FIXTURE = REPO / "tests" / "fixtures" / "audio" / "superhero_offset_200s.flac"
|
||||
FIXTURE = REPO / "tests" / "fixtures" / "audio" / "bali_offset_200s.flac"
|
||||
TONE = REPO / "tests" / "fixtures" / "audio" / "jray_audio_v1_tone.flac"
|
||||
|
||||
# Server SPEC.md section 3, "Matching and offset recovery". The cap is the
|
||||
|
||||
+2
-17
@@ -155,24 +155,9 @@ struct DecodeCtx {
|
||||
|
||||
bool open_resampler(DecodeCtx& c, const AVFrame* f) {
|
||||
#if LIBAVUTIL_VERSION_INT >= AV_VERSION_INT(57, 24, 100)
|
||||
// Both MUST be zero-initialised. av_channel_layout_copy documents that it
|
||||
// "will always uninitialize the destination before copy", and
|
||||
// av_channel_layout_uninit() calls av_freep() on u.map — so a declaration
|
||||
// without {} hands free() whatever pointer-shaped garbage the stack frame
|
||||
// happened to hold. That is a real crash ("free(): invalid pointer"), not a
|
||||
// theoretical one: it reproduced in roughly 1 run in 4 of UT-103, the only
|
||||
// test that exercises this branch, because it is the only one whose input
|
||||
// is stereo and so the only one that reaches the downmix path at all.
|
||||
//
|
||||
// It hid for two reasons worth remembering. It is stack-dependent, so it
|
||||
// vanishes under a sanitizer build and looks like a flake in the aggregate
|
||||
// test binary; and the golden-vector tests (UT-101) pass a mono 11025 Hz
|
||||
// fixture, which is chosen precisely so the vector does not depend on the
|
||||
// resampler — so bit-exactness against the golden vector proves nothing
|
||||
// about this function.
|
||||
AVChannelLayout out_layout{};
|
||||
AVChannelLayout out_layout;
|
||||
av_channel_layout_default(&out_layout, 1); // mono
|
||||
AVChannelLayout in_layout{};
|
||||
AVChannelLayout in_layout;
|
||||
if (av_channel_layout_copy(&in_layout, &f->ch_layout) < 0) return false;
|
||||
if (in_layout.nb_channels <= 0) {
|
||||
av_channel_layout_uninit(&in_layout);
|
||||
|
||||
@@ -42,20 +42,17 @@ constexpr int kDim = 512;
|
||||
// Used for CI and as the correctness oracle for the GPU backends.
|
||||
//
|
||||
// TRACES: AR-026, AR-027 | SR-001
|
||||
// Backed by CBLAS (OpenBLAS), which CMake now REQUIRES for this backend. The
|
||||
// scalar loop below is portable but scales badly: scoring one face against a
|
||||
// Backed by CBLAS (OpenBLAS) when available, falling back to a scalar loop when
|
||||
// not. The fallback is portable but scales badly: scoring one face against a
|
||||
// 5000-embedding gallery is 2.6 MFLOP, and a crowded frame multiplies that by
|
||||
// the face count. Since AR-003 removed the per-frame face cap and CI has no GPU,
|
||||
// the CPU path is the one that has to hold up under a library-scale gallery
|
||||
// (AR-027) rather than merely be correct — so falling back to it silently would
|
||||
// mean measuring AR-027 on a path no release runs.
|
||||
// the CPU path is now the one that has to hold up under a library-scale gallery
|
||||
// (AR-027) rather than merely be correct.
|
||||
//
|
||||
// The fallback is kept as the correctness oracle the two BLAS backends are
|
||||
// diffed against when a similarity looks wrong, and is reachable only via
|
||||
// -DSAE_ALLOW_SCALAR_GEMM=ON. The gallery is L2-normalised (as are the queries),
|
||||
// so each similarity is a plain dot product. S is stored column-major to match
|
||||
// the GPU backends: the gallery similarities for face fi start at
|
||||
// result + fi*n_gallery().
|
||||
// The fallback is kept rather than made mandatory so the build has no hard new
|
||||
// dependency, and so the two can be diffed when a similarity looks wrong. The gallery is L2-normalised (as are the queries), so each
|
||||
// similarity is a plain dot product. S is stored column-major to match the GPU
|
||||
// backends: the gallery similarities for face fi start at result + fi*n_gallery.
|
||||
class SimilarityEngine final : public ISimilarityEngine {
|
||||
public:
|
||||
SimilarityEngine(const float* gallery_row_major, int n_gallery, int max_faces)
|
||||
@@ -75,19 +72,6 @@ public:
|
||||
}
|
||||
|
||||
int max_faces() const override { return max_faces_; }
|
||||
int n_gallery() const override { return n_gallery_; }
|
||||
|
||||
/// TRACES: AR-026 | SR-001
|
||||
/// Promotions join the resident matrix, so the annex is scored by the same
|
||||
/// SGEMM as the baked references. std::vector already grows geometrically,
|
||||
/// so this is amortised O(1) per row.
|
||||
void append_rows(const float* rows_row_major, int n_rows) override {
|
||||
if (n_rows <= 0) return;
|
||||
gallery_.insert(gallery_.end(), rows_row_major,
|
||||
rows_row_major + static_cast<size_t>(n_rows) * kDim);
|
||||
n_gallery_ += n_rows;
|
||||
host_sims_.resize(static_cast<size_t>(max_faces_) * n_gallery_);
|
||||
}
|
||||
|
||||
const float* compute(const float* query_row_major, int n_faces) override {
|
||||
if (n_faces <= 0) return host_sims_.data();
|
||||
@@ -159,7 +143,6 @@ inline void gpu_free(void* p) { cudaFree(p)
|
||||
inline void gpu_memcpy_h2d(void* dst, const void* src, size_t n, stream_t s) { check_gpu(cudaMemcpyAsync(dst, src, n, cudaMemcpyHostToDevice, s), "H2D"); }
|
||||
inline void gpu_memcpy_d2h(void* dst, const void* src, size_t n, stream_t s) { check_gpu(cudaMemcpyAsync(dst, src, n, cudaMemcpyDeviceToHost, s), "D2H"); }
|
||||
inline void gpu_memcpy_h2d_sync(void* dst, const void* src, size_t n) { check_gpu(cudaMemcpy(dst, src, n, cudaMemcpyHostToDevice), "H2D_sync"); }
|
||||
inline void gpu_memcpy_d2d_sync(void* dst, const void* src, size_t n) { check_gpu(cudaMemcpy(dst, src, n, cudaMemcpyDeviceToDevice), "D2D_sync"); }
|
||||
inline void stream_create(stream_t* s) { check_gpu(cudaStreamCreate(s), "cudaStreamCreate"); }
|
||||
inline void stream_destroy(stream_t s) { cudaStreamDestroy(s); }
|
||||
inline void stream_sync(stream_t s) { check_gpu(cudaStreamSynchronize(s), "cudaStreamSync"); }
|
||||
@@ -194,7 +177,6 @@ inline void gpu_free(void* p) { (void)hipFr
|
||||
inline void gpu_memcpy_h2d(void* dst, const void* src, size_t n, stream_t s) { check_gpu(hipMemcpyAsync(dst, src, n, hipMemcpyHostToDevice, s), "H2D"); }
|
||||
inline void gpu_memcpy_d2h(void* dst, const void* src, size_t n, stream_t s) { check_gpu(hipMemcpyAsync(dst, src, n, hipMemcpyDeviceToHost, s), "D2H"); }
|
||||
inline void gpu_memcpy_h2d_sync(void* dst, const void* src, size_t n) { check_gpu(hipMemcpy(dst, src, n, hipMemcpyHostToDevice), "H2D_sync"); }
|
||||
inline void gpu_memcpy_d2d_sync(void* dst, const void* src, size_t n) { check_gpu(hipMemcpy(dst, src, n, hipMemcpyDeviceToDevice), "D2D_sync"); }
|
||||
inline void stream_create(stream_t* s) { check_gpu(hipStreamCreate(s), "hipStreamCreate"); }
|
||||
inline void stream_destroy(stream_t s) { (void)hipStreamDestroy(s); }
|
||||
inline void stream_sync(stream_t s) { check_gpu(hipStreamSynchronize(s), "hipStreamSync"); }
|
||||
@@ -219,15 +201,14 @@ public:
|
||||
SimilarityEngine(const float* gallery_row_major, int n_gallery, int max_faces)
|
||||
: n_gallery_(n_gallery), max_faces_(max_faces)
|
||||
{
|
||||
const size_t gallery_floats = static_cast<size_t>(n_gallery_) * kDim;
|
||||
gpu_malloc(reinterpret_cast<void**>(&d_gallery_), gallery_floats * sizeof(float));
|
||||
gpu_memcpy_h2d_sync(d_gallery_, gallery_row_major, gallery_floats * sizeof(float));
|
||||
|
||||
gpu_malloc(reinterpret_cast<void**>(&d_query_),
|
||||
static_cast<size_t>(max_faces_) * kDim * sizeof(float));
|
||||
|
||||
// Allocates d_gallery_/d_sims_ at the initial row count; append_rows()
|
||||
// grows them geometrically from here.
|
||||
reserve_rows(std::max(n_gallery_, 1));
|
||||
const size_t gallery_floats = static_cast<size_t>(n_gallery_) * kDim;
|
||||
if (gallery_floats)
|
||||
gpu_memcpy_h2d_sync(d_gallery_, gallery_row_major, gallery_floats * sizeof(float));
|
||||
gpu_malloc(reinterpret_cast<void**>(&d_sims_),
|
||||
static_cast<size_t>(max_faces_) * n_gallery_ * sizeof(float));
|
||||
|
||||
stream_create(&stream_);
|
||||
blas_create(&handle_);
|
||||
@@ -251,24 +232,6 @@ public:
|
||||
SimilarityEngine& operator=(const SimilarityEngine&) = delete;
|
||||
|
||||
int max_faces() const override { return max_faces_; }
|
||||
int n_gallery() const override { return n_gallery_; }
|
||||
|
||||
/// TRACES: AR-026 | SR-001
|
||||
/// Promotions join the GPU-resident matrix, so the annex is scored by the
|
||||
/// same SGEMM as the baked references rather than by a host-side loop.
|
||||
/// Capacity doubles on overflow, so the gallery is re-uploaded O(log n)
|
||||
/// times over a film rather than once per promotion.
|
||||
void append_rows(const float* rows_row_major, int n_rows) override {
|
||||
if (n_rows <= 0) return;
|
||||
const int want = n_gallery_ + n_rows;
|
||||
if (want > capacity_) reserve_rows(std::max(want, capacity_ * 2));
|
||||
|
||||
gpu_memcpy_h2d_sync(d_gallery_ + static_cast<size_t>(n_gallery_) * kDim,
|
||||
rows_row_major,
|
||||
static_cast<size_t>(n_rows) * kDim * sizeof(float));
|
||||
n_gallery_ = want;
|
||||
host_sims_.resize(static_cast<size_t>(max_faces_) * n_gallery_);
|
||||
}
|
||||
|
||||
const float* compute(const float* query_row_major, int n_faces) override {
|
||||
if (n_faces <= 0) return host_sims_.data();
|
||||
@@ -288,35 +251,7 @@ public:
|
||||
}
|
||||
|
||||
private:
|
||||
// Grow the resident gallery (and the similarity output sized against it) to
|
||||
// `rows` capacity, preserving the n_gallery_ rows already there. The copy is
|
||||
// device-to-device, so a promotion never re-uploads the baked gallery across
|
||||
// the bus.
|
||||
void reserve_rows(int rows) {
|
||||
if (rows <= capacity_) return;
|
||||
|
||||
float* d_new_gallery = nullptr;
|
||||
gpu_malloc(reinterpret_cast<void**>(&d_new_gallery),
|
||||
static_cast<size_t>(rows) * kDim * sizeof(float));
|
||||
if (d_gallery_ && n_gallery_ > 0)
|
||||
gpu_memcpy_d2d_sync(d_new_gallery, d_gallery_,
|
||||
static_cast<size_t>(n_gallery_) * kDim * sizeof(float));
|
||||
if (d_gallery_) gpu_free(d_gallery_);
|
||||
d_gallery_ = d_new_gallery;
|
||||
|
||||
// S is (capacity × n_faces); its contents are rewritten by every
|
||||
// compute(), so this one is a plain reallocation with nothing to keep.
|
||||
float* d_new_sims = nullptr;
|
||||
gpu_malloc(reinterpret_cast<void**>(&d_new_sims),
|
||||
static_cast<size_t>(max_faces_) * rows * sizeof(float));
|
||||
if (d_sims_) gpu_free(d_sims_);
|
||||
d_sims_ = d_new_sims;
|
||||
|
||||
capacity_ = rows;
|
||||
}
|
||||
|
||||
int n_gallery_{0};
|
||||
int capacity_{0};
|
||||
int max_faces_{0};
|
||||
float* d_gallery_{nullptr};
|
||||
float* d_query_{nullptr};
|
||||
|
||||
@@ -23,12 +23,23 @@
|
||||
enum class OrtProvider { CPU, CUDA, ROCm, TensorRT };
|
||||
|
||||
inline OrtProvider detect_ort_provider() {
|
||||
// ORT returns these in its own preference order (TensorRT, CUDA, ..., CPU
|
||||
// last), so the first recognised entry is the best available and the loop
|
||||
// returns on it.
|
||||
auto available = Ort::GetAvailableProviders();
|
||||
for (const auto& p : available) {
|
||||
// Only the TensorRT *EP* is a build-time opt-in — it needs the headers
|
||||
// and the profile plumbing below. CUDA is not: it is a plain ORT
|
||||
// provider, and gating its detection on the TRT flag (as this did) made
|
||||
// the CUDA branch unreachable in every build that did not also ask for
|
||||
// TensorRT. The symptom is silent rather than loud — inference simply
|
||||
// runs on the CPU and everything still returns correct answers — which
|
||||
// is why it survived: a 300-actor VR-012 grid cell took 76 s on the CPU
|
||||
// with the GPU idle at 212 MiB.
|
||||
#ifdef SAE_ORT_WITH_TRT_EP
|
||||
if (p == "TensorrtExecutionProvider") return OrtProvider::TensorRT;
|
||||
if (p == "CUDAExecutionProvider") return OrtProvider::CUDA;
|
||||
#endif
|
||||
if (p == "CUDAExecutionProvider") return OrtProvider::CUDA;
|
||||
if (p == "ROCMExecutionProvider") return OrtProvider::ROCm;
|
||||
}
|
||||
return OrtProvider::CPU;
|
||||
@@ -124,21 +135,8 @@ inline OrtProvider apply_ort_provider(Ort::SessionOptions& opts,
|
||||
try {
|
||||
OrtROCMProviderOptions rocm{};
|
||||
rocm.device_id = 0;
|
||||
// Without these MIOpen runs convolutions on the no-workspace GEMM
|
||||
// fallback (the "GemmFwdRest, provided ptr: 0 size: 0" warnings), which
|
||||
// is the slow path — most visible on the conv-heavy TransNetV2 scene
|
||||
// detector. Exhaustive search lets MIOpen pick the fast conv kernel,
|
||||
// and TunableOp autotunes the GEMMs; both cache to the MIOpen user DB
|
||||
// (MIOPEN_USER_DB_PATH), so the tuning cost is paid once per shape.
|
||||
// Opt-out via SAE_ROCM_NOTUNE=1 for a quick no-warmup run.
|
||||
const bool tune = std::getenv("SAE_ROCM_NOTUNE") == nullptr;
|
||||
rocm.miopen_conv_exhaustive_search = tune ? 1 : 0;
|
||||
rocm.tunable_op_enable = tune;
|
||||
rocm.tunable_op_tuning_enable = tune;
|
||||
opts.AppendExecutionProvider_ROCM(rocm);
|
||||
std::cerr << "[" << label << "] ROCm provider"
|
||||
<< (tune ? " (MIOpen exhaustive + TunableOp)" : " (untuned)")
|
||||
<< "\n";
|
||||
std::cerr << "[" << label << "] ROCm provider\n";
|
||||
return OrtProvider::ROCm;
|
||||
} catch (const Ort::Exception& e) {
|
||||
std::cerr << "[" << label << "] ROCm unavailable ("
|
||||
|
||||
@@ -19,7 +19,6 @@
|
||||
|
||||
#include <NvInfer.h>
|
||||
#include <cuda_runtime_api.h>
|
||||
#include <cstdlib>
|
||||
#include <opencv2/dnn.hpp>
|
||||
#include <opencv2/imgproc.hpp>
|
||||
|
||||
@@ -46,40 +45,6 @@ inline void check_cuda(cudaError_t e, const char* what) {
|
||||
throw CudaError(std::string(what) + ": " + cudaGetErrorString(e));
|
||||
}
|
||||
|
||||
/// TRACES: VR-015 | PR-004
|
||||
/// Select how a thread waits for the GPU. Must run before the CUDA context is
|
||||
/// created, so every engine constructor calls it and the first one wins.
|
||||
///
|
||||
/// The default (`cudaDeviceScheduleAuto`) spin-waits: `cudaStreamSynchronize`
|
||||
/// burns the calling thread's CPU for the whole of the device's work. Measured
|
||||
/// here, the embedder thread sat at 99.7% *user* time with 0.5 s of system time
|
||||
/// across 183 s — i.e. no blocking syscalls at all — while the GPU ran flat out.
|
||||
///
|
||||
/// On this laptop that is not merely wasted CPU. `nvidia-powerd` arbitrates one
|
||||
/// power budget across CPU and GPU, and the GPU's ceiling was observed dropping
|
||||
/// from 20 W idle to 15 W under our load, with the SM clock *falling* from
|
||||
/// 1005 MHz to 210 MHz once work started. Spinning may therefore be buying
|
||||
/// watts away from the device the pipeline is actually waiting on.
|
||||
///
|
||||
/// SAE_CUDA_BLOCKING_SYNC=1 switches to a blocking wait so the A/B needs no
|
||||
/// rebuild. Default is unchanged until the measurement says otherwise.
|
||||
inline void configure_cuda_sync_once() {
|
||||
static const bool done = [] {
|
||||
const char* env = std::getenv("SAE_CUDA_BLOCKING_SYNC");
|
||||
if (env && env[0] == '1') {
|
||||
cudaError_t e = cudaSetDeviceFlags(cudaDeviceScheduleBlockingSync);
|
||||
std::cerr << "[cuda] sync policy: BlockingSync"
|
||||
<< (e == cudaSuccess ? "" : " (FAILED — context already created)")
|
||||
<< "\n";
|
||||
} else {
|
||||
std::cerr << "[cuda] sync policy: default (spin) — "
|
||||
"set SAE_CUDA_BLOCKING_SYNC=1 to compare\n";
|
||||
}
|
||||
return true;
|
||||
}();
|
||||
(void)done;
|
||||
}
|
||||
|
||||
class TrtLogger : public nvinfer1::ILogger {
|
||||
public:
|
||||
void log(Severity sev, const char* msg) noexcept override {
|
||||
@@ -144,7 +109,6 @@ public:
|
||||
(output_is_fp16_ ? 2 : 4);
|
||||
check_cuda(cudaMalloc(&d_input_, in_bytes), "cudaMalloc input");
|
||||
check_cuda(cudaMalloc(&d_output_, out_bytes), "cudaMalloc output");
|
||||
configure_cuda_sync_once();
|
||||
check_cuda(cudaStreamCreate(&stream_), "cudaStreamCreate");
|
||||
|
||||
context_->setTensorAddress(input_name_.c_str(), d_input_);
|
||||
@@ -329,7 +293,6 @@ public:
|
||||
out_elem_counts_[oi] = count;
|
||||
}
|
||||
|
||||
configure_cuda_sync_once();
|
||||
check_cuda(cudaStreamCreate(&stream_), "cudaStreamCreate");
|
||||
|
||||
std::cerr << "[TrtScrfd] loaded: " << engine_path
|
||||
@@ -499,7 +462,6 @@ public:
|
||||
const std::size_t out_count = static_cast<std::size_t>(kWindow);
|
||||
check_cuda(cudaMalloc(&d_input_, in_count * 4), "cudaMalloc input");
|
||||
check_cuda(cudaMalloc(&d_output_, out_count * 4), "cudaMalloc output");
|
||||
configure_cuda_sync_once();
|
||||
check_cuda(cudaStreamCreate(&stream_), "cudaStreamCreate");
|
||||
context_->setTensorAddress(input_name_.c_str(), d_input_);
|
||||
context_->setTensorAddress(output_name_.c_str(), d_output_);
|
||||
|
||||
@@ -1,590 +0,0 @@
|
||||
#pragma once
|
||||
/// TRACES: VR-015 | PR-004
|
||||
///
|
||||
/// Pipeline throughput benchmark — how much of a run is spent in each node.
|
||||
///
|
||||
/// The KPN network already counts most of what an optimiser needs, and
|
||||
/// `print_diagnostics()` throws nearly all of it away: it prints frames and
|
||||
/// `ema` per node, and passes `elapsed_s = 0`, which zeroes throughput. Two
|
||||
/// things had to change before "time per node" could be answered honestly.
|
||||
///
|
||||
/// **`ema` is not a total.** It is an exponentially weighted average, so
|
||||
/// `frames * ema` tracks the end of the run rather than the whole of it. On a
|
||||
/// film that is a real difference — a detector costs one thing in a crowd scene
|
||||
/// and another over a landscape. `NodeStats::total_exec_us` (added alongside
|
||||
/// this) is the true sum.
|
||||
///
|
||||
/// **Wall time inside a node is not all work.** `PoolObjectNode::fire_once`
|
||||
/// times the functor *and* `push_outputs`, and `push_outputs` parks on a full
|
||||
/// downstream channel (AR-004). A node that is merely backpressured therefore
|
||||
/// bills the time it spent waiting to whoever is ahead of it: SuperHero's
|
||||
/// `frame_source` reported 141.9 ms/frame against a decoder logging 12-18 ms.
|
||||
/// Optimising against that number means optimising the fastest node in the
|
||||
/// graph.
|
||||
///
|
||||
/// So each node is reported three ways, and the three together are what
|
||||
/// identify a cost:
|
||||
///
|
||||
/// - `exec_ms` — cumulative wall time in the node, work *and* parked pushes
|
||||
/// - `cpu_ms` — thread CPU time (CLOCK_THREAD_CPUTIME_ID). Backpressure
|
||||
/// cannot inflate it, because a parked node holds no thread.
|
||||
/// - `pressure` — mean fill of its input channels minus that of its outputs
|
||||
///
|
||||
/// **`cpu_ms` cannot tell real work from a spinning GPU wait.** CUDA's default
|
||||
/// sync policy (`cudaDeviceScheduleAuto`) spin-waits before yielding, so
|
||||
/// `cudaStreamSynchronize` burns the calling thread's CPU while the GPU works.
|
||||
/// A node that is purely GPU-bound can therefore report a high `cpu_ms` and read
|
||||
/// as CPU-bound. `cudaSetDeviceFlags(cudaDeviceScheduleBlockingSync)` settles it
|
||||
/// in one line: if a node's `cpu_ms` collapses under blocking sync, that CPU was
|
||||
/// spin, not work.
|
||||
///
|
||||
/// **`cpu_ms` counts one thread only.** `CLOCK_THREAD_CPUTIME_ID` is per-thread,
|
||||
/// and OpenCV here is built against TBB, so any node whose functor goes through
|
||||
/// `cv::parallel_for_` (histogram compare, `warpAffine`, colour conversion) has
|
||||
/// that work executed on TBB's arena — 19 workers on a 20-core box — and billed
|
||||
/// to those threads rather than to the node. Such a node reads *cheaper* than it
|
||||
/// is, and the difference shows up in `stall/f` instead, indistinguishable from a
|
||||
/// GPU wait. `exec_ms` does capture it, since the functor does not return until
|
||||
/// the parallel region joins: a node where `exec/f` greatly exceeds `cpu/f`
|
||||
/// while its output channel is empty is fanning out, not waiting.
|
||||
///
|
||||
/// Work piles up *in front of* a bottleneck and starves everything *after* it,
|
||||
/// so `pressure` is maximal at the node setting the pace. `cpu_ms` then says
|
||||
/// which repair applies: high pressure with a saturated thread is CPU-bound and
|
||||
/// the work must get cheaper, while high pressure with an idle thread is
|
||||
/// waiting on a device, where batch size and engine precision are the knobs.
|
||||
///
|
||||
/// Occupancy has to be sampled during the run. `current_fill` is instantaneous
|
||||
/// and every channel has drained by the time the network stops, so a single
|
||||
/// read at the end reports an idle pipeline however congested it was.
|
||||
///
|
||||
/// Nothing here is specific to this pipeline's topology: the node graph is
|
||||
/// recovered from KPN's channel names, so it keeps working when the graph
|
||||
/// changes.
|
||||
|
||||
#include <kpn/diagnostics.hpp>
|
||||
|
||||
#include <nlohmann/json.hpp>
|
||||
|
||||
#include <algorithm>
|
||||
#include <atomic>
|
||||
#include <chrono>
|
||||
#include <cstdint>
|
||||
#include <functional>
|
||||
#include <iomanip>
|
||||
#include <map>
|
||||
#include <ostream>
|
||||
#include <sstream>
|
||||
#include <string>
|
||||
#include <thread>
|
||||
#include <vector>
|
||||
|
||||
namespace sae::bench {
|
||||
|
||||
// ── Edge naming ──────────────────────────────────────────────────────────────
|
||||
|
||||
/// TRACES: VR-015 | PR-004
|
||||
/// KPN names a channel "<src>:<idx> → <dst>:<idx>" (static_network.hpp).
|
||||
/// Recovering the two node names from it is what keeps attribution
|
||||
/// topology-agnostic: the graph is read back out of the channel names rather
|
||||
/// than hard-coded here, so a new node or a re-wired branch needs no change.
|
||||
/// Leaves both outputs untouched if the name does not carry an arrow.
|
||||
inline void split_edge_name(const std::string& name,
|
||||
std::string& producer, std::string& consumer) {
|
||||
static const std::string kArrow = " \xe2\x86\x92 "; // " → "
|
||||
const auto arrow = name.find(kArrow);
|
||||
if (arrow == std::string::npos) return;
|
||||
auto strip_port = [](std::string s) {
|
||||
const auto colon = s.rfind(':');
|
||||
return colon == std::string::npos ? s : s.substr(0, colon);
|
||||
};
|
||||
producer = strip_port(name.substr(0, arrow));
|
||||
consumer = strip_port(name.substr(arrow + kArrow.size()));
|
||||
}
|
||||
|
||||
// ── Channel occupancy, time-averaged ─────────────────────────────────────────
|
||||
|
||||
/// TRACES: VR-015 | PR-004
|
||||
/// One channel's fill level integrated over the run. `peak_fill` is already
|
||||
/// cumulative in `ChannelStats`, but a peak cannot distinguish "full once" from
|
||||
/// "full throughout", and those are opposite diagnoses. A mean can.
|
||||
struct ChannelOccupancy {
|
||||
std::string name; // "src:0 → dst:0", as KPN names it
|
||||
std::string producer; // node name left of the arrow
|
||||
std::string consumer; // node name right of the arrow
|
||||
std::size_t capacity{0};
|
||||
std::uint64_t samples{0};
|
||||
double fill_sum{0.0};
|
||||
std::uint64_t samples_full{0};
|
||||
std::uint64_t samples_empty{0};
|
||||
|
||||
// Final-snapshot totals (monotonic counters, so the last read is the total).
|
||||
std::size_t peak_fill{0};
|
||||
std::uint64_t pushes{0};
|
||||
std::uint64_t pops{0};
|
||||
std::uint64_t drops{0};
|
||||
std::uint64_t overflows{0};
|
||||
std::uint64_t bytes_pushed{0};
|
||||
|
||||
double mean_fill() const { return samples ? fill_sum / static_cast<double>(samples) : 0.0; }
|
||||
double mean_fill_pct() const { return capacity ? 100.0 * mean_fill() / static_cast<double>(capacity) : 0.0; }
|
||||
double peak_pct() const { return capacity ? 100.0 * static_cast<double>(peak_fill) / static_cast<double>(capacity) : 0.0; }
|
||||
double full_pct() const { return samples ? 100.0 * static_cast<double>(samples_full) / static_cast<double>(samples) : 0.0; }
|
||||
double empty_pct() const { return samples ? 100.0 * static_cast<double>(samples_empty) / static_cast<double>(samples) : 0.0; }
|
||||
double bandwidth_mbs(double wall_s) const {
|
||||
return wall_s > 0.0 ? static_cast<double>(bytes_pushed) / wall_s / 1e6 : 0.0;
|
||||
}
|
||||
};
|
||||
|
||||
// ── Per-node attributed cost ─────────────────────────────────────────────────
|
||||
|
||||
/// TRACES: VR-015 | PR-004
|
||||
struct NodeCost {
|
||||
std::string name;
|
||||
std::uint64_t frames{0};
|
||||
|
||||
// Cumulative wall time in the node — the answer to "where did the run go",
|
||||
// but only for a node that is not backpressured; it includes parked pushes.
|
||||
double exec_ms{0.0};
|
||||
double exec_ms_per_frame{0.0}; // true mean, not the EMA
|
||||
double exec_share{0.0}; // exec_ms / wall_ms, 0..1
|
||||
double ema_exec_ms{0.0}; // KPN's EMA, kept for continuity with the old report
|
||||
double max_exec_ms{0.0};
|
||||
|
||||
// Thread CPU time: excludes sleeping, parking and waiting on a device, so it
|
||||
// is the one number backpressure cannot inflate.
|
||||
double cpu_ms{0.0};
|
||||
double cpu_ms_per_frame{0.0};
|
||||
double cpu_share{0.0}; // cpu_ms / wall_ms — thread saturation, 0..1
|
||||
double cpu_pct_of_pipeline{0.0}; // this node's share of all nodes' CPU time
|
||||
|
||||
// Per frame, time inside the node not spent on its own CPU: parked on a
|
||||
// full output channel, or waiting on the GPU. `pressure` separates those —
|
||||
// a backpressured node has a full output, a device-bound one does not.
|
||||
double stall_ms_per_frame{0.0};
|
||||
|
||||
// Queue occupancy either side of the node, in percent of capacity.
|
||||
double in_fill_pct{0.0};
|
||||
double out_fill_pct{0.0};
|
||||
double pressure{0.0}; // in − out; maximal at the pacing node
|
||||
bool has_input{false};
|
||||
bool has_output{false};
|
||||
|
||||
double queue_wait_ms{0.0};
|
||||
bool is_bottleneck{false};
|
||||
|
||||
/// TRACES: VR-015 | AR-004 | PR-004
|
||||
/// Live scheduling state, so a wedged run says *why* it is wedged rather
|
||||
/// than only that it is. With `queued=0, wake=1` a wake was recorded and
|
||||
/// never consumed; with `queued=0, wake=0` and a full input, no wake was
|
||||
/// ever generated. Those are different bugs in different files.
|
||||
bool queued{false};
|
||||
bool wake_pending{false};
|
||||
};
|
||||
|
||||
/// TRACES: VR-015 | PR-004
|
||||
/// Attribute cost to nodes from a KPN node snapshot plus sampled channel
|
||||
/// occupancy. Pure — no clocks, no threads, no network — so the ranking is
|
||||
/// unit-testable on CI hardware that can never run the pipeline itself.
|
||||
///
|
||||
/// A node with several inputs takes the **minimum** input fill: it can only run
|
||||
/// once every input has data, so the emptiest one gates it, and a full sibling
|
||||
/// channel means that channel's producer is blocked rather than this node being
|
||||
/// slow. A node with several outputs takes the **maximum** output fill, since
|
||||
/// parking on any one branch stops the node.
|
||||
///
|
||||
/// Terminals are the infinite-reservoir limit of the same rule: a source has
|
||||
/// unlimited work available (input treated as 100% full) and a sink unlimited
|
||||
/// drain (output treated as empty), so both stay rankable against the interior
|
||||
/// nodes instead of dropping out of the comparison.
|
||||
inline std::vector<NodeCost> attribute_cost(
|
||||
const std::vector<kpn::NodeSnapshot>& nodes,
|
||||
const std::vector<ChannelOccupancy>& channels,
|
||||
double wall_sec)
|
||||
{
|
||||
const double wall_ms = wall_sec * 1000.0;
|
||||
|
||||
double cpu_total = 0.0;
|
||||
for (const auto& n : nodes) cpu_total += n.total_cpu_ms;
|
||||
|
||||
std::vector<NodeCost> out;
|
||||
out.reserve(nodes.size());
|
||||
|
||||
for (const auto& n : nodes) {
|
||||
NodeCost c;
|
||||
c.name = n.name;
|
||||
c.frames = n.frames_processed;
|
||||
c.exec_ms = n.total_exec_ms;
|
||||
c.ema_exec_ms = n.ema_exec_ms;
|
||||
c.max_exec_ms = n.max_exec_ms;
|
||||
c.cpu_ms = n.total_cpu_ms;
|
||||
c.queue_wait_ms = n.queue_wait_ms;
|
||||
c.queued = n.queued;
|
||||
c.wake_pending = n.wake_pending;
|
||||
|
||||
c.exec_ms_per_frame = c.frames ? c.exec_ms / static_cast<double>(c.frames) : 0.0;
|
||||
c.cpu_ms_per_frame = c.frames ? c.cpu_ms / static_cast<double>(c.frames) : 0.0;
|
||||
c.exec_share = wall_ms > 0.0 ? c.exec_ms / wall_ms : 0.0;
|
||||
c.cpu_share = wall_ms > 0.0 ? c.cpu_ms / wall_ms : 0.0;
|
||||
c.cpu_pct_of_pipeline = cpu_total > 0.0 ? 100.0 * c.cpu_ms / cpu_total : 0.0;
|
||||
c.stall_ms_per_frame = c.exec_ms_per_frame - c.cpu_ms_per_frame;
|
||||
if (c.stall_ms_per_frame < 0.0) c.stall_ms_per_frame = 0.0;
|
||||
|
||||
double in_min = 0.0; bool have_in = false;
|
||||
double out_max = 0.0; bool have_out = false;
|
||||
for (const auto& ch : channels) {
|
||||
if (ch.consumer == n.name) {
|
||||
const double f = ch.mean_fill_pct();
|
||||
if (!have_in || f < in_min) in_min = f;
|
||||
have_in = true;
|
||||
}
|
||||
if (ch.producer == n.name) {
|
||||
const double f = ch.mean_fill_pct();
|
||||
if (!have_out || f > out_max) out_max = f;
|
||||
have_out = true;
|
||||
}
|
||||
}
|
||||
c.has_input = have_in;
|
||||
c.has_output = have_out;
|
||||
c.in_fill_pct = have_in ? in_min : 100.0; // source: always has work
|
||||
c.out_fill_pct = have_out ? out_max : 0.0; // sink: never blocks
|
||||
c.pressure = c.in_fill_pct - c.out_fill_pct;
|
||||
out.push_back(std::move(c));
|
||||
}
|
||||
|
||||
// Rank, but only among nodes that actually ran: a node with zero frames has
|
||||
// no cost to attribute and its neighbouring channels never moved.
|
||||
auto best = out.end();
|
||||
for (auto it = out.begin(); it != out.end(); ++it) {
|
||||
if (it->frames == 0) continue;
|
||||
if (best == out.end() || it->pressure > best->pressure) best = it;
|
||||
}
|
||||
if (best != out.end()) best->is_bottleneck = true;
|
||||
|
||||
return out;
|
||||
}
|
||||
|
||||
/// TRACES: VR-015 | PR-004
|
||||
/// One line of plain English about the winning node, since the point of the
|
||||
/// report is to say what to change next. A saturated thread means the node's
|
||||
/// own work is the limit; an idle thread under pressure means it is waiting on
|
||||
/// a device, and those are different repairs.
|
||||
inline std::string verdict(const std::vector<NodeCost>& costs) {
|
||||
for (const auto& c : costs) {
|
||||
if (!c.is_bottleneck) continue;
|
||||
std::ostringstream os;
|
||||
os << std::fixed << c.name << " sets the pace: ";
|
||||
|
||||
// A source's 100% input is the infinite-reservoir convention, not a
|
||||
// measured queue — saying "work is backed up in front of it" would be
|
||||
// asserting something no counter observed.
|
||||
if (!c.has_input)
|
||||
os << "nothing downstream is waiting on it (output "
|
||||
<< std::setprecision(1) << c.out_fill_pct
|
||||
<< "% full), so the pipeline is running as fast as this node can feed it. ";
|
||||
else if (!c.has_output)
|
||||
os << std::setprecision(1) << c.in_fill_pct
|
||||
<< "% full input and nothing to block on, so it is the drain. ";
|
||||
else
|
||||
os << std::setprecision(1) << c.in_fill_pct << "% full input, "
|
||||
<< c.out_fill_pct << "% full output. ";
|
||||
|
||||
os << std::setprecision(2) << c.cpu_ms_per_frame << " ms/frame on CPU. ";
|
||||
|
||||
if (c.cpu_share >= 0.85)
|
||||
os << "CPU-bound — its thread is busy " << std::setprecision(0)
|
||||
<< (100.0 * c.cpu_share) << "% of the run, so the work itself has to get"
|
||||
" cheaper or be split across more threads.";
|
||||
else if (c.cpu_share <= 0.35 && c.stall_ms_per_frame > c.cpu_ms_per_frame)
|
||||
os << "Device-bound — its thread is busy only " << std::setprecision(0)
|
||||
<< (100.0 * c.cpu_share) << "% of the run and it spends "
|
||||
<< std::setprecision(2) << c.stall_ms_per_frame
|
||||
<< " ms/frame off-CPU, so it is waiting on the GPU or the disk: batch size,"
|
||||
" engine precision and the decode path are the knobs, not the C++.";
|
||||
else
|
||||
os << "Mixed — thread busy " << std::setprecision(0) << (100.0 * c.cpu_share)
|
||||
<< "% of the run, " << std::setprecision(2) << c.stall_ms_per_frame
|
||||
<< " ms/frame off-CPU.";
|
||||
return os.str();
|
||||
}
|
||||
return "no node processed a frame — nothing to attribute";
|
||||
}
|
||||
|
||||
// ── Recorder ─────────────────────────────────────────────────────────────────
|
||||
|
||||
/// TRACES: VR-015 | PR-004
|
||||
/// Samples the live network on a timer and emits the report at the end.
|
||||
///
|
||||
/// The sampler only reads relaxed atomics, so it does not perturb what it
|
||||
/// measures — which matters, since this exists to be trusted as a timing
|
||||
/// measurement.
|
||||
class BenchmarkRecorder {
|
||||
public:
|
||||
using Sampler = std::function<kpn::NetworkSnapshot()>;
|
||||
|
||||
explicit BenchmarkRecorder(int sample_interval_ms = 100)
|
||||
: interval_(std::chrono::milliseconds(sample_interval_ms)) {}
|
||||
|
||||
~BenchmarkRecorder() { stop(); }
|
||||
|
||||
void start(Sampler sampler) {
|
||||
sampler_ = std::move(sampler);
|
||||
running_.store(true, std::memory_order_release);
|
||||
thread_ = std::thread([this] {
|
||||
while (running_.load(std::memory_order_acquire)) {
|
||||
accumulate(sampler_());
|
||||
std::this_thread::sleep_for(interval_);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
/// Stops sampling and latches the final counter values. Call while the
|
||||
/// network object is still alive: the monotonic counters stay valid after
|
||||
/// `net.stop()`, but they die with the object.
|
||||
void stop() {
|
||||
if (!running_.exchange(false, std::memory_order_acq_rel)) return;
|
||||
if (thread_.joinable()) thread_.join();
|
||||
if (sampler_) {
|
||||
final_ = sampler_();
|
||||
// Occupancy is deliberately NOT accumulated from this last read:
|
||||
// the pipeline has drained by now, and folding an idle sample into
|
||||
// the mean biases every channel toward "never congested".
|
||||
for (const auto& ch : final_.channels) {
|
||||
auto& occ = occupancy_[ch.name];
|
||||
if (occ.name.empty()) { // a channel that never moved
|
||||
occ.name = ch.name;
|
||||
occ.capacity = ch.capacity;
|
||||
split_edge_name(ch.name, occ.producer, occ.consumer);
|
||||
}
|
||||
occ.peak_fill = ch.peak_fill;
|
||||
occ.pushes = ch.pushes;
|
||||
occ.pops = ch.pops;
|
||||
occ.drops = ch.drops;
|
||||
occ.overflows = ch.overflows;
|
||||
occ.bytes_pushed = ch.bytes_pushed;
|
||||
}
|
||||
}
|
||||
stopped_ = true;
|
||||
}
|
||||
|
||||
bool has_data() const { return stopped_ && !final_.nodes.empty(); }
|
||||
double wall_sec() const { return final_.elapsed_s; }
|
||||
|
||||
std::vector<ChannelOccupancy> channels() const {
|
||||
std::vector<ChannelOccupancy> v;
|
||||
v.reserve(occupancy_.size());
|
||||
for (const auto& [_, occ] : occupancy_) v.push_back(occ);
|
||||
return v;
|
||||
}
|
||||
|
||||
std::vector<NodeCost> costs() const {
|
||||
return attribute_cost(final_.nodes, channels(), final_.elapsed_s);
|
||||
}
|
||||
|
||||
/// TRACES: VR-015 | PR-004
|
||||
/// Machine-readable report, for sweeping configurations and diffing runs.
|
||||
/// `film_sec` is the last timestamp the pipeline reached, so
|
||||
/// `realtime_factor` answers what the optimiser is really asking: seconds
|
||||
/// of film per second of wall clock. It is 0 for a topology with no result
|
||||
/// sink (the dump-only path), and the field is then omitted rather than
|
||||
/// reported as zero throughput.
|
||||
nlohmann::json to_json(const nlohmann::json& run_config, double film_sec) const {
|
||||
using nlohmann::json;
|
||||
const double wall = final_.elapsed_s;
|
||||
const auto chans = channels();
|
||||
const auto cost = attribute_cost(final_.nodes, chans, wall);
|
||||
|
||||
json j;
|
||||
j["schema_version"] = 1;
|
||||
j["config"] = run_config;
|
||||
|
||||
json summary;
|
||||
summary["wall_sec"] = wall;
|
||||
summary["sample_count"] = sample_count_;
|
||||
summary["sample_interval_ms"] = interval_.count();
|
||||
if (film_sec > 0.0) {
|
||||
summary["film_sec"] = film_sec;
|
||||
summary["realtime_factor"] = wall > 0.0 ? film_sec / wall : 0.0;
|
||||
}
|
||||
for (const auto& c : cost)
|
||||
if (c.is_bottleneck) { summary["bottleneck"] = c.name; break; }
|
||||
summary["verdict"] = verdict(cost);
|
||||
j["summary"] = summary;
|
||||
|
||||
json jnodes = json::array();
|
||||
for (const auto& c : cost) {
|
||||
jnodes.push_back({
|
||||
{"name", c.name},
|
||||
{"frames", c.frames},
|
||||
{"fps", wall > 0.0 ? c.frames / wall : 0.0},
|
||||
{"exec_ms", c.exec_ms},
|
||||
{"exec_ms_per_frame", c.exec_ms_per_frame},
|
||||
{"exec_share", c.exec_share},
|
||||
{"ema_exec_ms", c.ema_exec_ms},
|
||||
{"max_exec_ms", c.max_exec_ms},
|
||||
{"cpu_ms", c.cpu_ms},
|
||||
{"cpu_ms_per_frame", c.cpu_ms_per_frame},
|
||||
{"cpu_share", c.cpu_share},
|
||||
{"cpu_pct_of_pipeline", c.cpu_pct_of_pipeline},
|
||||
{"stall_ms_per_frame", c.stall_ms_per_frame},
|
||||
{"queue_wait_ms", c.queue_wait_ms},
|
||||
{"in_fill_pct", c.in_fill_pct},
|
||||
{"out_fill_pct", c.out_fill_pct},
|
||||
{"pressure", c.pressure},
|
||||
{"is_bottleneck", c.is_bottleneck},
|
||||
{"queued", c.queued},
|
||||
{"wake_pending", c.wake_pending},
|
||||
});
|
||||
}
|
||||
j["nodes"] = std::move(jnodes);
|
||||
|
||||
json jch = json::array();
|
||||
for (const auto& ch : chans) {
|
||||
jch.push_back({
|
||||
{"name", ch.name},
|
||||
{"producer", ch.producer},
|
||||
{"consumer", ch.consumer},
|
||||
{"capacity", ch.capacity},
|
||||
{"mean_fill", ch.mean_fill()},
|
||||
{"mean_fill_pct", ch.mean_fill_pct()},
|
||||
{"peak_fill", ch.peak_fill},
|
||||
{"peak_pct", ch.peak_pct()},
|
||||
{"full_pct", ch.full_pct()},
|
||||
{"empty_pct", ch.empty_pct()},
|
||||
{"pushes", ch.pushes},
|
||||
{"pops", ch.pops},
|
||||
{"drops", ch.drops},
|
||||
{"overflows", ch.overflows},
|
||||
{"mb_per_sec", ch.bandwidth_mbs(wall)},
|
||||
});
|
||||
}
|
||||
j["channels"] = std::move(jch);
|
||||
return j;
|
||||
}
|
||||
|
||||
/// TRACES: VR-015 | PR-004
|
||||
/// Human-readable form of the same data, so a run is legible without
|
||||
/// opening the JSON.
|
||||
void print(std::ostream& os, double film_sec) const {
|
||||
print_impl(os, final_, film_sec);
|
||||
}
|
||||
|
||||
/// TRACES: VR-015 | AR-004 | PR-004
|
||||
/// Dump the report from a LIVE snapshot, mid-run, without stopping anything.
|
||||
///
|
||||
/// A report that only exists at shutdown is no use against the failure this
|
||||
/// pipeline actually has: a wedged run never reaches shutdown, so the one
|
||||
/// moment the numbers matter most is the one moment they were unavailable.
|
||||
/// Channel occupancy names the stalled node directly — it is the one whose
|
||||
/// input is full and whose output is empty — which is otherwise a debug-build
|
||||
/// and a gdb session away.
|
||||
///
|
||||
/// Safe to call from the wait loop while the pipeline is running or hung: it
|
||||
/// takes the same lock-free snapshot the sampler does.
|
||||
void dump_live(std::ostream& os, double film_sec) const {
|
||||
if (!sampler_) { os << "[benchmark] no sampler — run with --benchmark\n"; return; }
|
||||
print_impl(os, sampler_(), film_sec);
|
||||
}
|
||||
|
||||
private:
|
||||
void print_impl(std::ostream& os, const kpn::NetworkSnapshot& snap,
|
||||
double film_sec) const {
|
||||
const double wall = snap.elapsed_s;
|
||||
const auto chans = channels();
|
||||
const auto cost = attribute_cost(snap.nodes, chans, wall);
|
||||
|
||||
os << "\n┌─ Pipeline benchmark (VR-015) ──────────────────────────────────────────────\n";
|
||||
os << "│ wall " << std::fixed << std::setprecision(1) << wall << "s";
|
||||
if (film_sec > 0.0)
|
||||
os << " film " << film_sec << "s realtime x" << std::setprecision(2)
|
||||
<< (wall > 0.0 ? film_sec / wall : 0.0);
|
||||
os << " samples " << sample_count_ << "\n│\n";
|
||||
|
||||
os << "│ node frames cpu_s cpu%run cpu%tot cpu/f"
|
||||
" exec/f stall/f in% out% press q/w\n";
|
||||
for (const auto& c : cost) {
|
||||
os << "│ " << (c.is_bottleneck ? "▶ " : " ") << std::left << std::setw(16)
|
||||
<< c.name << std::right
|
||||
<< std::setw(7) << c.frames
|
||||
<< std::setw(10) << std::setprecision(1) << (c.cpu_ms / 1000.0)
|
||||
<< std::setw(9) << std::setprecision(0) << (100.0 * c.cpu_share)
|
||||
<< std::setw(9) << std::setprecision(0) << c.cpu_pct_of_pipeline
|
||||
<< std::setw(8) << std::setprecision(2) << c.cpu_ms_per_frame
|
||||
<< std::setw(8) << std::setprecision(2) << c.exec_ms_per_frame
|
||||
<< std::setw(9) << std::setprecision(2) << c.stall_ms_per_frame
|
||||
<< std::setw(7) << std::setprecision(0) << c.in_fill_pct
|
||||
<< std::setw(7) << std::setprecision(0) << c.out_fill_pct
|
||||
<< std::setw(8) << std::setprecision(1) << c.pressure
|
||||
<< " " << int(c.queued) << "/" << int(c.wake_pending)
|
||||
<< "\n";
|
||||
}
|
||||
|
||||
/// TRACES: VR-015 | AR-004 | PR-004
|
||||
// Fires only when the scheduling state is actually wrong, so a healthy
|
||||
// run stays quiet and a wedged one names the fault — instead of leaving
|
||||
// it to be reconstructed under a debugger that suppresses the bug.
|
||||
for (const auto& c : cost) {
|
||||
if (c.queued || !c.has_input) continue;
|
||||
if (c.wake_pending)
|
||||
os << "│ !! " << c.name << " idle with a wake outstanding"
|
||||
" (queued=0 wake=1): the wake was recorded and never"
|
||||
" consumed — submit/release handshake.\n";
|
||||
else if (c.in_fill_pct > 50.0)
|
||||
os << "│ !! " << c.name << " idle with a "
|
||||
<< std::setprecision(0) << c.in_fill_pct
|
||||
<< "% full input and no wake pending: the wake was never"
|
||||
" generated — channel edge detection.\n";
|
||||
}
|
||||
|
||||
os << "│\n│ channel cap mean% peak% full%"
|
||||
" empty% MB/s\n";
|
||||
for (const auto& ch : chans) {
|
||||
os << "│ " << std::left << std::setw(36) << ch.name << std::right
|
||||
<< std::setw(5) << ch.capacity
|
||||
<< std::setw(7) << std::setprecision(1) << ch.mean_fill_pct()
|
||||
<< std::setw(7) << ch.peak_pct()
|
||||
<< std::setw(7) << ch.full_pct()
|
||||
<< std::setw(7) << ch.empty_pct()
|
||||
<< std::setw(9) << std::setprecision(1) << ch.bandwidth_mbs(wall)
|
||||
<< "\n";
|
||||
}
|
||||
os << "│\n│ " << verdict(cost) << "\n";
|
||||
os << "└────────────────────────────────────────────────────────────────────────────\n";
|
||||
os << " cpu_s / cpu%tot is where the run's compute actually went. exec/f is wall\n"
|
||||
" time in the node INCLUDING time parked on a full output channel, so it\n"
|
||||
" overstates a backpressured node — compare it against cpu/f, which cannot\n"
|
||||
" be inflated that way. press = input fill − output fill, and locates the\n"
|
||||
" node that work is queueing up in front of.\n"
|
||||
" cpu_s counts THIS node's thread only: work OpenCV fans out via TBB is\n"
|
||||
" billed to the TBB arena, so a node using cv::parallel_for_ reads cheaper\n"
|
||||
" than it is and the difference surfaces in stall/f.\n";
|
||||
}
|
||||
|
||||
private:
|
||||
void accumulate(const kpn::NetworkSnapshot& snap) {
|
||||
++sample_count_;
|
||||
for (const auto& ch : snap.channels) {
|
||||
auto& occ = occupancy_[ch.name];
|
||||
if (occ.name.empty()) {
|
||||
occ.name = ch.name;
|
||||
occ.capacity = ch.capacity;
|
||||
split_edge_name(ch.name, occ.producer, occ.consumer);
|
||||
}
|
||||
occ.fill_sum += static_cast<double>(ch.current_fill);
|
||||
++occ.samples;
|
||||
if (ch.capacity && ch.current_fill >= ch.capacity) ++occ.samples_full;
|
||||
if (ch.current_fill == 0) ++occ.samples_empty;
|
||||
}
|
||||
}
|
||||
|
||||
std::chrono::milliseconds interval_;
|
||||
Sampler sampler_;
|
||||
std::thread thread_;
|
||||
std::atomic<bool> running_{false};
|
||||
bool stopped_{false};
|
||||
std::uint64_t sample_count_{0};
|
||||
std::map<std::string, ChannelOccupancy> occupancy_;
|
||||
kpn::NetworkSnapshot final_{};
|
||||
};
|
||||
|
||||
} // namespace sae::bench
|
||||
@@ -34,7 +34,6 @@ int main(int argc, char** argv) {
|
||||
std::string arcface_model = kDefaultArcfaceModel;
|
||||
float conf = 0.5f, nms_thr = 0.4f;
|
||||
int max_side = 500;
|
||||
float min_face_px = 0.f;
|
||||
|
||||
for (int i = 1; i < argc; ++i) {
|
||||
auto arg = [&](const char* f) { return std::strcmp(argv[i], f) == 0; };
|
||||
@@ -51,7 +50,6 @@ int main(int argc, char** argv) {
|
||||
else if (arg("--conf")) conf = std::stof(next());
|
||||
else if (arg("--nms")) nms_thr = std::stof(next());
|
||||
else if (arg("--max-side")) max_side = std::stoi(next());
|
||||
else if (arg("--min-face-px")) min_face_px = std::stof(next());
|
||||
else { std::cerr << "[warn] unknown flag: " << argv[i] << "\n"; }
|
||||
} catch (const std::exception& e) {
|
||||
std::cerr << "Error: " << e.what() << "\n";
|
||||
@@ -61,8 +59,7 @@ int main(int argc, char** argv) {
|
||||
|
||||
if (root_path.empty() || output_path.empty()) {
|
||||
std::cerr << "Usage: build_gallery --root <dir> --output <gallery.json> "
|
||||
"[--detector <path>] [--arcface <path>] [--max-side <N>]\n"
|
||||
" [--min-face-px <px>]\n";
|
||||
"[--detector <path>] [--arcface <path>] [--max-side <N>]\n";
|
||||
return 1;
|
||||
}
|
||||
|
||||
@@ -73,7 +70,6 @@ int main(int argc, char** argv) {
|
||||
cfg.detector_conf = conf;
|
||||
cfg.detector_nms = nms_thr;
|
||||
cfg.max_side = max_side;
|
||||
cfg.min_face_px = min_face_px;
|
||||
|
||||
try {
|
||||
ActorGallery gallery = build_gallery(cfg);
|
||||
|
||||
+44
-154
@@ -11,20 +11,6 @@ enum class Verbosity {
|
||||
standard, // per-frame detail: bbox, similarity, unknowns logged
|
||||
xray, // Jellyfin-Xray format: {"second": ["Actor", ...], ...}
|
||||
};
|
||||
|
||||
// How a track's accepted frames become a reported presence window.
|
||||
enum class PresenceMode {
|
||||
// A claim IS its track's [first_seen, last_seen] (AR-012/AR-013). The
|
||||
// default and the only mode whose semantics the register validated.
|
||||
track_extent,
|
||||
// Flood-fill: snap each claim to the shot it sits in, so an actor seen once
|
||||
// anywhere in a scene is reported for the whole scene [prev_boundary,
|
||||
// next_boundary]. Trades precision for recall against X-Ray's per-scene cast
|
||||
// granularity. Snaps to TransNetV2 shot boundaries (is_scene_boundary) when a
|
||||
// scene detector populated them, else to the always-on histogram cuts
|
||||
// (is_cut). With no boundaries at all it degrades to track_extent per claim.
|
||||
flood,
|
||||
};
|
||||
// debug verbosity = compile with -DSAE_DEBUG → scene_analyze_debug binary
|
||||
|
||||
struct Config {
|
||||
@@ -46,14 +32,6 @@ struct Config {
|
||||
// scripts/optimizer/SCHEMA.md) for offline threshold-sweep replay via sae_kpn.
|
||||
std::string dump_embeddings_path;
|
||||
|
||||
/// TRACES: VR-015 | PR-004
|
||||
// When set, write a per-node timing and bottleneck report here (src/
|
||||
// benchmark.hpp) and print it at shutdown. Costs one background thread
|
||||
// reading relaxed atomics on a timer, so it is safe to leave on, but a
|
||||
// measurement run should still be isolated (nothing else on the GPU).
|
||||
std::string benchmark_path;
|
||||
int benchmark_interval_ms{100}; // channel-occupancy sampling period
|
||||
|
||||
// ── Sampling ─────────────────────────────────────────────────────────────
|
||||
float sample_fps{1.0f}; // frames to analyse per second of movie
|
||||
float max_decode_fps{0.f}; // wall-clock cap on source decode rate (0 = uncapped)
|
||||
@@ -85,51 +63,16 @@ struct Config {
|
||||
std::string arcface_model;
|
||||
std::string arcface_engine; // optional path to a pre-built TRT engine; bypasses ORT
|
||||
int embed_batch_size{4}; // max faces per ORT Run() call — bounds per-call latency
|
||||
float match_prior{0.433f}; // base-rate prior; 10-knob DE optimum (was 0.5)
|
||||
// Tuned by Differential Evolution against Amazon X-Ray per-second presence
|
||||
// over ALL 9 films (opencv5 build, LVFace-B_Glint360K, full gallery,
|
||||
// expansion on), a 10-parameter sweep — see docs/model-bakeoff.md. The
|
||||
// per-second misID-weighted macro-F1 optimum is 64.0% (P 79.0%, R 61.1%).
|
||||
//
|
||||
// This is a permissive operating point: the sweep discovered that with
|
||||
// flood-fill presence recovering recall, a LOW threshold pays off. It
|
||||
// supersedes the earlier 0.754, which came from a 4-film subset under the
|
||||
// now-withdrawn anneal/extinction windows and was never re-derived after a
|
||||
// scoring-bug fix. The full-9-film sweep at 0.485 beats it.
|
||||
//
|
||||
// Caveat, still true: the optimum generalises unevenly. It is strong on 7 of
|
||||
// 9 films (F1 62–80%) and weak on two — The Many Saints of Newark (an
|
||||
// ensemble of look-alikes; nearly all the run's misIDs land here) and
|
||||
// Scarface (sparse cuts, so flood-fill over-extends: R 95% / P 26%). Both
|
||||
// were the low outliers in every prior run too. Shipped because it wins on
|
||||
// average and on the misID-weighted objective; not a settled, film-agnostic
|
||||
// constant.
|
||||
float prob_threshold{0.485f}; // posterior P(match | sim, prior) threshold
|
||||
// TRACES: AR-024 | SR-002
|
||||
// match_threshold (0.45), match_ratio (0.80) and match_ratio_ceil (0.65) are
|
||||
// RETIRED, joining track_max_embed_dist, cut_revive_sim, expand_novelty_sim
|
||||
// and expand_track_spread_max. All were raw cosine distances, and they were
|
||||
// the accept rule whenever the calibration fit failed — so the one situation
|
||||
// in which the pipeline knew its probabilities were untrustworthy was the
|
||||
// one in which it stopped using them. An unfitted sigmoid is now the
|
||||
// fallback everywhere, which is at least the same wrong number in every
|
||||
// stage. See identity_matcher_node.hpp.
|
||||
|
||||
// ── Presence derivation ──────────────────────────────────────────────────
|
||||
// How accepted frames become a reported window. flood requires scene_detect.
|
||||
// Default flood: the 10-knob DE optimum uses it — snapping presence to the
|
||||
// shot recovers enough recall against X-Ray's scene-level cast to win the
|
||||
// misID-weighted F1, at a precision cost that is a net gain on 7 of 9 films.
|
||||
// Falls back to track_extent per claim when no boundaries exist. See
|
||||
// docs/model-bakeoff.md and PresenceMode above.
|
||||
PresenceMode presence_mode{PresenceMode::flood};
|
||||
|
||||
// Path to the learned XGBoost scene-boundary model. When set (build has
|
||||
// SAE_SCENE_XGB), the camera-position node stamps a per-frame RGB histogram
|
||||
// and the sink runs the detector post-EOF to supply flood-fill boundaries —
|
||||
// the measured best flood boundary source (presence F1 ~76% vs ~64% for the
|
||||
// always-on histogram cut). Empty → flood falls back to is_cut.
|
||||
std::string scene_xgb_model;
|
||||
float match_prior{0.5f}; // base-rate prior; 0.5 = use calibrated sigmoid directly
|
||||
// prob_threshold tuned by Differential Evolution against Amazon X-Ray per-scene
|
||||
// presence over 4 films, per-second metric (see docs/rep4-optimizer-results.md).
|
||||
// Best model+mode: LVFace-B_Glint360K, full gallery, expansion on. Supersedes the
|
||||
// earlier 9-film scene-union-metric tuning (0.76) — that metric is now known to
|
||||
// have hidden out-of-cast false positives (see docs/optimizer-experiments.md).
|
||||
float prob_threshold{0.754f}; // posterior P(match | sim, prior) threshold
|
||||
float match_threshold{0.45f}; // cosine distance hard ceiling fallback (no calibration)
|
||||
float match_ratio{0.80f}; // ratio test fallback: accept if best/second < ratio
|
||||
float match_ratio_ceil{0.65f}; // ratio test only fires below this absolute distance
|
||||
|
||||
// ── Cut detection ────────────────────────────────────────────────────────
|
||||
float cut_threshold{0.70f}; // grayscale histogram correlation below this → hard cut
|
||||
@@ -148,28 +91,17 @@ struct Config {
|
||||
// at ~0.50; real boundaries spike to ~0.7+)
|
||||
int scene_stride{50}; // frames advanced between windows (≤ kWindow)
|
||||
|
||||
// Dense-decode knobs (only active with scene_detect). Dense decode of every
|
||||
// native-rate frame is the pipeline's cost driver, which is what made the
|
||||
// temporal shortcut below tempting.
|
||||
/// TRACES: AR-011 | SR-002
|
||||
// scene_decode_fps: rate the source decodes at in dense mode.
|
||||
// **0 = native, and native is the only correct setting.** kWindow is 100
|
||||
// frames: at native 25 fps that window spans ~4 s, which is what
|
||||
// TransNetV2 was trained on; at the 12 fps this used to default to it
|
||||
// spans ~8.3 s, so the model saw half-speed motion over twice its
|
||||
// temporal context. Boundary *timestamps* stay right either way — which
|
||||
// is exactly why the degradation was invisible, and why the compressed
|
||||
// separation it produced (~0.50 baseline against ~0.7+ peaks) was read
|
||||
// as a property of the export rather than of the input. Lowering this
|
||||
// buys decode time by running the model off-distribution; reach for
|
||||
// dense_scale or scene_stride instead, which do not.
|
||||
// Dense-decode throughput knobs (only active with scene_detect). Dense decode
|
||||
// of every native-rate frame is the pipeline's cost driver; these trade a
|
||||
// little boundary precision for a large speedup.
|
||||
// scene_decode_fps: rate the source decodes at in dense mode. Lower =
|
||||
// fewer frames decoded. TransNetV2 tolerates ~12fps; boundary timestamps
|
||||
// stay correct (keyed off each frame's real timestamp). 0 = native fps.
|
||||
// dense_scale: downscale factor applied to decoded frames in dense mode
|
||||
// (0<f≤1; e.g. 0.5 = half size). Cheaper sws_scale + smaller frames
|
||||
// through the fanout. A spatial reduction, and TransNetV2 downsamples to
|
||||
// 48×27 regardless, so unlike the above it is a documented, understood
|
||||
// degradation. NOTE: also shrinks what the face detector sees — keep
|
||||
// ≥0.5 on 1080p sources so SCRFD still resolves small faces. 1 = off.
|
||||
float scene_decode_fps{0.f}; // dense decode rate (0 = native)
|
||||
// through the fanout. NOTE: also shrinks what the face detector sees —
|
||||
// keep ≥0.5 on 1080p sources so SCRFD still resolves small faces. 1 = off.
|
||||
float scene_decode_fps{12.0f}; // dense decode rate (0 = native)
|
||||
float dense_scale{1.0f}; // dense-mode frame downscale (1 = off)
|
||||
|
||||
// ── Face tracking (frame-to-frame) ───────────────────────────────────────
|
||||
@@ -178,7 +110,7 @@ struct Config {
|
||||
// frame-dependent (AR-007): on is_cut / is_scene_boundary, and for any track
|
||||
// that is no longer on screen, it drops to 0 (embedding only), because
|
||||
// position carries no information across a viewpoint change or a gap.
|
||||
float track_alpha{0.435f}; // base cost weight: 0=embedding only, 1=spatial only (10-knob DE optimum)
|
||||
float track_alpha{0.4f}; // base cost weight: 0=embedding only, 1=spatial only
|
||||
float track_min_iou{0.1f}; // IoU below which spatial link alone is rejected
|
||||
// Minimum P(same person) for an association to be admissible on appearance
|
||||
// alone. This replaces track_max_embed_dist (a raw cosine distance, AR-024).
|
||||
@@ -191,60 +123,19 @@ struct Config {
|
||||
// Replaces track_max_frames_missing: a frame count silently changed meaning
|
||||
// with sample_fps, and the same number had to be guessed twice (once for an
|
||||
// ordinary miss, once for a cut). Seconds mean one thing at any sample rate.
|
||||
double track_extinction_sec{31.0}; // 10-knob DE optimum (was 5.0)
|
||||
|
||||
// ── Ownership and evidence accumulation (AR-025) ──────────────────────────
|
||||
// TRACES: AR-025, AR-017 | SR-002
|
||||
// These four decided how presence is claimed and were unreachable: they
|
||||
// lived as in-class initialisers on TrackRegistry::Config and
|
||||
// EvidenceDiscounter::Config, and main constructed the discounter with the
|
||||
// one-argument constructor, so nothing short of a recompile could move
|
||||
// them. rho_max's own comment defers to "the sweep (VR-007)" for where it
|
||||
// belongs — a sweep that could not reach it.
|
||||
//
|
||||
// ownership_logodds is arguably the most consequential constant in the
|
||||
// pipeline after prob_threshold: below it a track produces no presence
|
||||
// claim at all, so it decides whether an actor is reported rather than how
|
||||
// confidently. 1.72 is a posterior of ~0.85 — the 10-knob DE optimum (was
|
||||
// an unswept 2.0 ≈ 0.88); slightly more permissive, consistent with the
|
||||
// low-threshold operating point the sweep converged on.
|
||||
float ownership_logodds{1.72f};
|
||||
|
||||
// How much a single observation may move a track's belief. n_eff =
|
||||
// n / (1 + (n-1)·rho), so rho_max caps what a repeated view can ever be
|
||||
// worth: 0.5 caps it at two independent observations however long the shot
|
||||
// runs. It is deliberately below 1 — a held pose still yields a fresh
|
||||
// detection, alignment and noise realisation, so a little independent
|
||||
// evidence survives. Setting it to 1 freezes belief after the first frame,
|
||||
// which is the bug this replaced.
|
||||
float evidence_rho_max{0.204f}; // 10-knob DE optimum (was 0.5): weights a
|
||||
// held pose closer to a single observation
|
||||
// P(same view) below this and the observation counts as a genuinely new
|
||||
// look, so it joins the per-track view set.
|
||||
float evidence_admit_below{0.784f}; // 10-knob DE optimum (was 0.6)
|
||||
// Distinct views remembered per track, which bounds the novelty comparison.
|
||||
int evidence_max_views{8};
|
||||
double track_extinction_sec{5.0};
|
||||
|
||||
// ── Scene tracking ────────────────────────────────────────────────────────
|
||||
// TRACES: AR-012, AR-013 | SR-002
|
||||
// extinction_sec (57.4) and anneal_sec (35.5) are GONE, along with
|
||||
// SceneTrackerFunc, which is what read the first of them. docs/SPEC.md
|
||||
// specified this removal and ended it "grep for both names and expect no
|
||||
// survivors"; there were about forty, and the register meanwhile recorded
|
||||
// both as Withdrawn and "deleted rather than retained at zero" on the
|
||||
// grounds that a field naming a mechanism the pipeline no longer has is
|
||||
// actively misleading.
|
||||
//
|
||||
// Both existed to bridge gaps between isolated accepted frames. A track
|
||||
// that survives its own gaps leaves them nothing to do: AR-012 makes a
|
||||
// window the extent of a track an actor owns, and AR-013 ends it at the
|
||||
// last sighting. The keep-alive answered the same question again and
|
||||
// answered it worse, by re-opening exactly the trailing cool-down AR-013
|
||||
// refuses.
|
||||
//
|
||||
// track_extinction_sec above is NOT the same knob under a new name. It
|
||||
// bounds how long a lost track stays available for re-association, which is
|
||||
// a tracking question; it never extends a presence claim.
|
||||
// extinction_sec re-tuned by DE against X-Ray per-second presence, 4-film rep4
|
||||
// matrix (docs/rep4-optimizer-results.md). Reverses the earlier "short is better"
|
||||
// finding: with a stricter prob_threshold, a long extinction window bridges real
|
||||
// presence gaps (occlusion, turned face) instead of just smearing FPs — every
|
||||
// model's best config pushed to ~90%+ of the search ceiling (tried up to 60s).
|
||||
// The ceiling kept getting hit, so treat 60 as "good enough", not a proven optimum.
|
||||
double extinction_sec{57.4}; // keep actor active this many seconds after last detection
|
||||
// anneal_sec: previously found INSENSITIVE at a 1–30s range; the wider rep4 sweep
|
||||
// (1–60s) also pushed this to the ceiling alongside extinction_sec (see above).
|
||||
double anneal_sec{35.5}; // merge actor windows separated by less than this into one epoch
|
||||
|
||||
// ── Per-film gallery expansion ────────────────────────────────────────────
|
||||
// Within one uncut track every face is the same physical person — a free
|
||||
@@ -253,26 +144,25 @@ struct Config {
|
||||
// new reference views; they are promoted into a per-film, in-memory annex so
|
||||
// later frames/tracks of that actor at similar poses recognise. See
|
||||
// gallery/track_gallery.hpp.
|
||||
// Default ON: the rep4 matrix (docs/model-bakeoff.md, "Two effects in
|
||||
// isolation") found expansion helps recall on the full (unrestricted)
|
||||
// gallery for the winning model/mode — the opposite of the earlier
|
||||
// assumption that it only helps restricted galleries. The same section is
|
||||
// explicit that on the full gallery it buys +2.1pp F1 and +3.9pp recall
|
||||
// "at a real cost" in misIDs, where in restricted mode it is a clean win.
|
||||
// Default ON: rep4 matrix (docs/rep4-optimizer-results.md) found expansion helps
|
||||
// recall on the full (unrestricted) gallery for the winning model/mode — the
|
||||
// opposite of the earlier assumption that it only helps restricted galleries.
|
||||
bool expand_gallery{true}; // master switch
|
||||
int expand_buffer_size{20}; // per-track diversity buffer capacity
|
||||
// TRACES: AR-018, AR-024 | SR-005
|
||||
// Banded admission for the per-subject store, in PROBABILITY space. An
|
||||
// embedding joins only if P(same person) against something already stored
|
||||
// lands inside [lo, hi]: above hi it is redundant, below lo it is evidence
|
||||
// the track is not one person. The same lo is re-applied to the whole store
|
||||
// at promotion time — see track_gallery.hpp. This is the only threshold the
|
||||
// expansion path has: it replaces the raw-cosine expand_novelty_sim (0.55)
|
||||
// and expand_track_spread_max (0.60), which are retired (AR-024).
|
||||
// 10-knob DE optimum (was 0.90/0.95). The sweep widened the band — a lower lo
|
||||
// admits more pose-varied views into the annex — which the optimum preferred.
|
||||
float expand_band_lo{0.804f};
|
||||
float expand_band_hi{0.952f};
|
||||
// the track is not one person. Replaces expand_novelty_sim, a raw cosine.
|
||||
// Working values pending VR-007; sweep both bounds, they fail in opposite
|
||||
// directions.
|
||||
float expand_band_lo{0.90f};
|
||||
float expand_band_hi{0.95f};
|
||||
float expand_novelty_sim{0.55f}; // promote only embeddings whose best sim to the
|
||||
// actor's refs is below this (gallery-far / novel)
|
||||
float expand_track_spread_max{0.60f}; // reject promotion if the retained buffer's
|
||||
// internal spread (1 - min pairwise sim) exceeds
|
||||
// this — guards track-ID collisions / two people
|
||||
int expand_min_anchor_frames{3}; // require ≥N accepted frames naming the actor before
|
||||
// the track is confirmed and its buffer promoted
|
||||
std::string expand_debug_dir; // if set, dump promoted mugshots + embeddings here
|
||||
|
||||
@@ -8,12 +8,6 @@
|
||||
// gallery file needed. Purpose-built for the optimizer's replay corpus and the
|
||||
// embedding-model bake-off (dump each --arcface model over the film set).
|
||||
//
|
||||
// The camera-position (histogram) cut detector runs, so frames/is_cut is recorded
|
||||
// in the dump. Flood-fill presence snaps to those cuts. TransNetV2 scene detection
|
||||
// is NOT run here: on the ROCm build it needs MIGraphX, whose statically-linked
|
||||
// LLVM collides with the VAAPI decoder's system libLLVM and crashes at compile, so
|
||||
// the two cannot share a process. is_scene_boundary therefore stays 0 here.
|
||||
//
|
||||
// Usage:
|
||||
// dump_embeddings --movie <path> --out <dump.h5> [--arcface <model.onnx>]
|
||||
// [--detector <model.onnx>] [--fps 1] [--start S] [--end S]
|
||||
|
||||
+1
-91
@@ -1,5 +1,5 @@
|
||||
#pragma once
|
||||
/// TRACES: AR-005, AR-029, AR-030 | SR-002
|
||||
/// TRACES: AR-005, AR-030 | SR-002
|
||||
#include "types.hpp"
|
||||
|
||||
#include <opencv2/core.hpp>
|
||||
@@ -143,96 +143,6 @@ inline cv::Mat align_face(const cv::Mat& img,
|
||||
return crop;
|
||||
}
|
||||
|
||||
// ── crop_sharpness ────────────────────────────────────────────────────────────
|
||||
/// TRACES: AR-029 | SR-002
|
||||
//
|
||||
// Normalised variance of the Laplacian over the aligned 112×112 crop: the AR-029
|
||||
// sharpness axis. Returns -1 for an empty crop (unscored), matching the
|
||||
// DetectedFace sentinel.
|
||||
//
|
||||
// sharpness = Var(∇²I) / Var(I)
|
||||
//
|
||||
// Two normalisations, each removing a quantity that would otherwise be read as
|
||||
// blur:
|
||||
//
|
||||
// - **Divided by the image variance, so contrast cannot leak in.** Scaling
|
||||
// intensity by α scales the Laplacian by α too, so both variances scale by α²
|
||||
// and the ratio is unchanged. A raw Var(∇²I) — the textbook measure — instead
|
||||
// falls with exposure, so a dim scene reads as soft and a graded-up one as
|
||||
// sharp. VR-012 has to locate one knee across films whose grading differs by
|
||||
// more than their focus does; an uncalibrated measure would put the knee in a
|
||||
// different place per film, which is the AR-024 failure in another metric.
|
||||
// - **Measured on the aligned crop, so size cannot leak in.** The destination
|
||||
// frame is fixed at 112×112 (AR-002 owns size, and double-counting it here
|
||||
// would make every small face read as blurred). What the ratio reports is the
|
||||
// detail actually present in the embedder's input — so a small sharp face can
|
||||
// and does outscore a large soft one. That is the claim; it is *not* a claim
|
||||
// of invariance to source resolution, because a 40 px face warped up to 112
|
||||
// genuinely carries less detail, and hiding that would defeat the point.
|
||||
//
|
||||
// Frequency-domain reading of why the blur ladder is monotone: with
|
||||
// Var(∇²I) = ∫|ω|⁴|F(ω)|² and Var(I) = ∫|F(ω)|², the ratio is E[|ω|⁴] under the
|
||||
// image's own spectral measure. Gaussian blur multiplies that measure by
|
||||
// e^{-σ²|ω|²}, concentrating it at low |ω|, so the expectation falls strictly
|
||||
// with σ. It is a property of the construction, not a fitted behaviour.
|
||||
//
|
||||
// **Three known hazards, for VR-012 to check rather than for a threshold to
|
||||
// absorb.** All are recorded here because they are properties of the measure,
|
||||
// visible in the dumped distribution, and neither should be papered over by a
|
||||
// correction chosen before that distribution has been looked at.
|
||||
//
|
||||
// 1. **Border fill.** `align_face` warps with BORDER_CONSTANT, so a face
|
||||
// crossing the frame edge brings a hard black step into the crop, and a
|
||||
// step edge is high-frequency. The normalisation blunts it — the fill
|
||||
// inflates Var(I) as well as Var(∇²I) — but does not remove it, so
|
||||
// heavily-cropped faces may read sharper than they are. The fix is either a
|
||||
// validity mask or a different border mode, and the second changes what the
|
||||
// embedder is fed (AR-011).
|
||||
//
|
||||
// 2. **The contrast invariance is exact in the algebra and approximate in
|
||||
// 8 bits.** Scaling I by α cancels exactly; what does not cancel is the
|
||||
// quantisation floor of a stored crop, which is broadband and so lands in
|
||||
// the numerator. It matters only where there is little signal left to
|
||||
// compete with it: on the AR-029 test texture a half-contrast copy reads
|
||||
// 0.9% high when sharp, 24% high at sigma 1.2 and 148% high at sigma 2.5.
|
||||
// A crop that is both **dim and soft therefore reads sharper than it is** —
|
||||
// the low corner of the axis, and the corner VR-012 must put a knee in.
|
||||
//
|
||||
// 3. **It reports where the energy sits, not how much there is.** A crop whose
|
||||
// energy is *already* concentrated at high frequency — dense film grain,
|
||||
// a face against foliage — loses numerator and denominator together under
|
||||
// blur, so the ratio moves less than the damage does. Measured on a
|
||||
// flat-spectrum synthetic, an anisotropic (motion) smear even makes it rise,
|
||||
// because the surviving perpendicular detail really is as fine as before.
|
||||
// Natural crops have the low-frequency mass that keeps the denominator
|
||||
// steady, and on those both ladders fall (see the AR-029 tests, which use a
|
||||
// 1/f texture for exactly this reason). The same property means the axis
|
||||
// conflates focus with intrinsic texture — a bearded face outscores a smooth
|
||||
// one at equal focus — which is true of every no-reference sharpness measure
|
||||
// and is why AR-028 carries the number instead of thresholding on it.
|
||||
inline float crop_sharpness(const cv::Mat& crop) {
|
||||
if (crop.empty()) return -1.f;
|
||||
|
||||
cv::Mat gray;
|
||||
if (crop.channels() == 3) cv::cvtColor(crop, gray, cv::COLOR_BGR2GRAY);
|
||||
else gray = crop;
|
||||
|
||||
cv::Mat lap;
|
||||
cv::Laplacian(gray, lap, CV_32F, 3);
|
||||
|
||||
cv::Scalar mean_i, sd_i, mean_l, sd_l;
|
||||
cv::meanStdDev(gray, mean_i, sd_i);
|
||||
cv::meanStdDev(lap, mean_l, sd_l);
|
||||
|
||||
const double var_i = sd_i[0] * sd_i[0];
|
||||
// A flat crop has no detail to be sharp or soft about, and the ratio is 0/0.
|
||||
// Zero is the honest answer and keeps the axis finite; -1 would claim the
|
||||
// face was never scored, which is a different fact.
|
||||
if (var_i < 1e-6) return 0.f;
|
||||
|
||||
return static_cast<float>((sd_l[0] * sd_l[0]) / var_i);
|
||||
}
|
||||
|
||||
// ── enhance_for_retry ────────────────────────────────────────────────────────
|
||||
// Used when initial face detection finds nothing. Pads the image by 50%
|
||||
// (border-replicated, so the detector doesn't see a hard edge) and applies
|
||||
|
||||
@@ -96,27 +96,6 @@ ActorGallery build_gallery(const BuildConfig& cfg) {
|
||||
return a.confidence < b.confidence;
|
||||
});
|
||||
|
||||
// Reject faces too small to embed honestly.
|
||||
//
|
||||
// The reference images are crops cut from the film, not mugshots, so
|
||||
// the detected face can be a small fraction of the image. Upscaling a
|
||||
// 30 px face to ArcFace's 112x112 feeds the model an input it was
|
||||
// never trained for, and it answers with a confident, plausible,
|
||||
// wrong embedding.
|
||||
//
|
||||
// At inference that costs one frame. Here it is permanent: a poisoned
|
||||
// reference sits in the gallery and corrupts every future match
|
||||
// against that character, which is exactly the kind of error that is
|
||||
// invisible without a study that should not have been needed.
|
||||
if (cfg.min_face_px > 0.f) {
|
||||
const float side = std::min(best.bbox.width, best.bbox.height);
|
||||
if (side < cfg.min_face_px) {
|
||||
std::cerr << " [skip] face " << side << "px < " << cfg.min_face_px
|
||||
<< "px: " << img_file.path().filename() << "\n";
|
||||
continue;
|
||||
}
|
||||
}
|
||||
|
||||
cv::Mat crop = align_face(img, best.landmarks);
|
||||
if (crop.empty()) {
|
||||
std::cerr << " [skip] alignment failed: " << img_file.path().filename() << "\n";
|
||||
|
||||
@@ -29,16 +29,6 @@ struct BuildConfig {
|
||||
float detector_conf{0.5f};
|
||||
float detector_nms{0.4f};
|
||||
int max_side{500}; // downscale source images to this max dimension
|
||||
|
||||
/// Minimum detected-face side, in pixels of the (possibly downscaled)
|
||||
/// source image. 0 disables the check.
|
||||
///
|
||||
/// References below this are dropped rather than upscaled: a face smaller
|
||||
/// than the embedder's input is off-distribution, and a bad reference
|
||||
/// poisons every match against that identity for the life of the gallery.
|
||||
/// Mirrors the inference-side --min-face-px so the gallery is built from
|
||||
/// the same face scales it will be matched against.
|
||||
float min_face_px{0.f};
|
||||
// before detection — TMDB portraits are ~2k px,
|
||||
// SCRFD trains on smaller faces and detection
|
||||
// confidence drops on huge inputs. 0 = disabled.
|
||||
|
||||
@@ -162,19 +162,6 @@ inline GalleryCalibration calibrate_gallery(
|
||||
for (const auto& e : by_actor[ai]) {
|
||||
bool dup = false;
|
||||
for (const auto& k : kept) {
|
||||
// EXCEPTION: AR-024 this asks whether two vectors are THE SAME
|
||||
// VECTOR, not whether two faces are the same person.
|
||||
//
|
||||
// Two independent reasons, either sufficient. First, at
|
||||
// 1 - 1e-7 the threshold is a floating-point identity test: it
|
||||
// catches one source image embedded twice, and no genuine pair
|
||||
// of distinct photographs lands there. Nothing about it is a
|
||||
// decision, so there is nothing for a probability to mean.
|
||||
//
|
||||
// Second, and structurally: this IS the calibration fit. The
|
||||
// dedup runs on its input, before (a, b) exist. A calibrated
|
||||
// comparison here would have to be calibrated by the fit it is
|
||||
// feeding, which is not a thing that can be arranged.
|
||||
if (cosine_similarity(e, k) > kDedupSimThreshold) { dup = true; break; }
|
||||
}
|
||||
if (!dup) kept.push_back(e);
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user