dtourolle/scene-actor-extraction
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: dtourolle/scene-actor-extraction:cd62d6452d48964d0ffb33d5306f984164e7e0b8
Branches Tags
dtourolle/scene-actor-extraction:master dtourolle/scene-actor-extraction:ci/green-the-runner dtourolle/scene-actor-extraction:feature/opencv5 dtourolle/scene-actor-extraction:feature/quality-knee dtourolle/scene-actor-extraction:feature/traceability-tooling dtourolle/scene-actor-extraction:gitea-pages
dtourolle/scene-actor-extraction:v1.0.0
..
compare: dtourolle/scene-actor-extraction:feature/traceability-tooling
Branches Tags
dtourolle/scene-actor-extraction:master dtourolle/scene-actor-extraction:ci/green-the-runner dtourolle/scene-actor-extraction:feature/opencv5 dtourolle/scene-actor-extraction:feature/quality-knee dtourolle/scene-actor-extraction:feature/traceability-tooling dtourolle/scene-actor-extraction:gitea-pages
dtourolle/scene-actor-extraction:v1.0.0
2 Commits
cd62d6452d .. feature/traceability-tooling
Author SHA1 Message Date
dtourolle faaa71fa09 refactor(traceability): parameterise the extractor for all three components 
The tool is moving into the jray-project submodule to be shared by
scene-actor-extraction (C++/Python), jRay (C#) and JRay-public-server
(Rust). Two constants blocked that: LOCAL_TYPES and SOURCE_SUFFIXES were
hardcoded to this repo, so either sibling parsed zero requirements and
scanned zero files. Both, plus the register path, scan roots, system-spec
path, exclude list and CI-executable tier set, are now configuration.

One implementation, parameterised. A second copy for "the other language"
is how two implementations start drifting apart, so there is exactly one -
the same code path now produces:

  scene-actor-extraction  AR/DP/IR/GR/VR   59 defined   0 tagged    0.0%
  jRay                    JR               46 defined  24 covered  52.2%
  JRay-public-server      UR/DR            32 defined  23 covered  71.9%

Configuration is traceability.toml at the component repo root, CLI flags,
or both (flags win). Its directory defines the repo root, so the gate works
from any subdirectory. `--print-example-config` emits the annotated schema.
The JSON report echoes the settings it ran with, since a shared tool's
output is otherwise ambiguous about which repo it describes.

The refusal behaviour is kept and sharpened, because parameterising is
exactly what makes it easy to point a repo at the wrong prefixes or the
wrong suffixes. Zero requirements parsed or zero files scanned is still a
hard failure, and the message now names the setting that is wrong rather
than printing a plausible 0%. Config errors exit 2, not 1: a broken config
is not a coverage failure, and conflating them makes CI logs lie about why
the job went red.

Also fixed while adapting to the sibling registers, which are read but not
modified here:

  * escaped `\|` inside a markdown cell no longer shifts every later column
    (the server's register contains `small-\|M\|`);
  * a tag above an attribute-decorated declaration attributes to the
    declaration, not to `[HttpGet(...)]` or `#[derive(...)]` - the gap
    jRay's register calls out;
  * Rust and C# declaration patterns for context extraction;
  * the missing-tier warning is suppressed for a register that assigns no
    tiers at all, rather than listing every requirement in it.

The workflow is now component-agnostic too: the changed-file check reads
its extension list out of the report the gate just wrote, so the definition
of "source file" lives in one place.

79 tests, still fixture-based, now including the cross-repo cases: the same
parser over JR and UR/DR registers, the same scanner over Rust and C#, and
both misconfigurations failing loudly.
 2026-07-30 18:35:56 +02:00
dtourolle 054c4c8b8f build: requirement traceability extractor, gate, and CI workflow 
Ports JellyTau's traceability tooling, rewritten in stdlib Python because
this repo is C++/Python and adding a bun/node toolchain to check source
comments would be a worse trade than writing the scanner.

scripts/traceability/extract_traces.py scans .cpp/.hpp/.py under src, tests,
scripts, experiments and eval for the house tag format

    /// TRACES: AR-012, AR-013 | SR-002

and reports EXCEPTION tags separately. An exception is a recorded decision to
depart from an invariant, so folding it into coverage would invert its
meaning; it is listed with its reason, and a missing reason is flagged.

Two rules carried over from JellyTau's gate repair:

  * Denominators are parsed out of docs/requirements.md at run time. A
    requirement is defined only by a row in a table whose header is
    `| ID | Requirement | ... |`, so references in the Traces to column, in
    prose, and in the verification-plan table do not inflate the count.
    Adding a register row lowers coverage until it is traced - the property
    that dies the moment a denominator is frozen.
  * Coverage above 100% is a hard failure. It cannot happen through the
    intersection, which is the point: if it ever does, the arithmetic is
    broken and the run must not be reported as a pass.

One rule specific to this repo: CI is an Intel N100 with no discrete GPU. The
extractor reads each requirement's verification tier from requirements.md and
reports T4/GPU-only requirements as tagged but unexecuted, never as covered.
Counting a test that can never run is the same failure mode as the 158% bug.

MIN_COVERAGE starts at 0 because almost nothing is tagged yet - tags are added
as the pipeline is built. That is not a gate that cannot fail: orphan tags, a
>100% ratio, a register that parses to nothing, and an empty source scan are
all hard failures from day one. The threshold lives in traceability-gate.sh
alone, never duplicated into the workflow YAML.

53 tests over fixture strings, so their meaning does not drift as requirements
are added.
 2026-07-30 18:16:58 +02:00
65 changed files with 3177 additions and 5124 deletions
Expand all files Collapse all files
.gitea/workflows/traceability-check.yml
+2 -2
View File
@@ -53,7 +53,7 @@ jobs:
# before it does. JellyTau's gate was believed for months while it was
# dividing by frozen literals; untested gate logic is how that happens.
- name: Test the extractor
run: python3 scripts/vendor/jray-project/scripts/traceability/test_extract_traces.py
run: python3 scripts/traceability/test_extract_traces.py
# Threshold policy and every other repo-specific setting live in
# traceability.toml, not here, so local runs and CI runs cannot disagree
@@ -63,7 +63,7 @@ jobs:
# A misconfigured run (zero requirements parsed, zero files scanned) is a
# hard failure rather than a plausible-looking 0%.
- name: Traceability gate
run: sh scripts/vendor/jray-project/scripts/traceability/traceability-gate.sh
run: sh scripts/traceability/traceability-gate.sh
- name: Check modified files for traces
if: github.event_name == 'pull_request'
.gitignore
-4
View File
@@ -19,10 +19,6 @@ compile_commands.json
# coverage). Regenerate with scripts/docs/run_holdout_all_models.py and
# scripts/docs/gallery_coverage_per_film.py.
!docs_data/*.json
# Exception: test fixtures are inputs, not build output. The audio golden
# vector (IR-005) is shared verbatim with the jRay plugin repo, so it has to be
# tracked. Regenerate the media with tests/fixtures/audio/make_fixture.py.
!tests/fixtures/**
# Video files
*.mp4
*.mkv
.gitmodules
-3
View File
@@ -2,6 +2,3 @@
path = external/KPN
url = https://gitea.tourolle.paris/dtourolle/KPN.git
branch = master
[submodule "jray-project"]
path = scripts/vendor/jray-project
url = git@gitea.tourolle.paris:dtourolle/jray-project.git
CMakeLists.txt
+8 -18
View File
@@ -196,31 +196,23 @@ endif()
# FFmpeg (hwaccel video decode: CUDA/VAAPI, runtime-detected + swscale colour
# conversion). Hwaccel support is built into libavcodec/libavutil; no extra
# libraries are needed here.
# libswresample is the audio side of the same dependency — downmix + resample
# for the audio signature (IR-004, src/audio_signature.cpp). Not a new project
# dependency: it ships with the libav* set already required above.
find_package(PkgConfig REQUIRED)
pkg_check_modules(AVFORMAT REQUIRED libavformat)
pkg_check_modules(AVCODEC REQUIRED libavcodec)
pkg_check_modules(AVUTIL REQUIRED libavutil)
pkg_check_modules(SWSCALE REQUIRED libswscale)
pkg_check_modules(SWRESAMPLE REQUIRED libswresample)
pkg_check_modules(AVFORMAT REQUIRED libavformat)
pkg_check_modules(AVCODEC REQUIRED libavcodec)
pkg_check_modules(AVUTIL REQUIRED libavutil)
pkg_check_modules(SWSCALE REQUIRED libswscale)
add_library(ffmpeg_libs INTERFACE)
target_compile_options(ffmpeg_libs INTERFACE
${AVFORMAT_CFLAGS_OTHER} ${AVCODEC_CFLAGS_OTHER}
${AVUTIL_CFLAGS_OTHER} ${SWSCALE_CFLAGS_OTHER}
${SWRESAMPLE_CFLAGS_OTHER})
${AVUTIL_CFLAGS_OTHER} ${SWSCALE_CFLAGS_OTHER})
target_include_directories(ffmpeg_libs INTERFACE
${AVFORMAT_INCLUDE_DIRS} ${AVCODEC_INCLUDE_DIRS}
${AVUTIL_INCLUDE_DIRS} ${SWSCALE_INCLUDE_DIRS}
${SWRESAMPLE_INCLUDE_DIRS})
${AVUTIL_INCLUDE_DIRS} ${SWSCALE_INCLUDE_DIRS})
target_link_libraries(ffmpeg_libs INTERFACE
${AVFORMAT_LIBRARIES} ${AVCODEC_LIBRARIES}
${AVUTIL_LIBRARIES} ${SWSCALE_LIBRARIES}
${SWRESAMPLE_LIBRARIES})
message(STATUS "FFmpeg: avformat=${AVFORMAT_VERSION} avcodec=${AVCODEC_VERSION} "
"swresample=${SWRESAMPLE_VERSION}")
${AVUTIL_LIBRARIES} ${SWSCALE_LIBRARIES})
message(STATUS "FFmpeg: avformat=${AVFORMAT_VERSION} avcodec=${AVCODEC_VERSION}")
# nlohmann/json (gallery + output serialisation)
include(FetchContent)
@@ -257,8 +249,6 @@ find_package(HDF5 REQUIRED COMPONENTS CXX)
add_library(sae_gallery STATIC
src/gallery/gallery_store.cpp
src/gallery/gallery_builder.cpp
src/audio_signature.cpp # IR-004 — content-derived audio signature
src/gallery/embedder_stamp.cpp # GR-004 — gallery/embedder binding
)
set_target_properties(sae_gallery PROPERTIES POSITION_INDEPENDENT_CODE ON)
target_include_directories(sae_gallery PUBLIC src ${HDF5_INCLUDE_DIRS})
docs/SPEC.md
+9 -239
View File
@@ -80,49 +80,8 @@ by dropping work or growing without limit.
- Memory is the real limit: faces carry 112×112 crops plus 512-float embeddings.
Backpressure must engage on bytes in flight, not just item counts.
### The fix is not in this repo
**Every node output in KPN uses the dropping `push()`** (`pool_node.hpp:404`,
`:710`; also `branch.hpp`, `fanout.hpp`, `interrupt_node.hpp`). A lossless
`push_blocking()` — "wait for the consumer to drain instead of dropping; the
producer just runs slower" — already exists on both `Channel`
(`channel.hpp:144`) and `OutputPort` (`variant_node.hpp:81`), **and nothing
calls it.**
So AR-004 is a change to the KPN repository, not to this one. It needs either a
per-channel lossless policy or a network-wide default, and this pipeline should
select lossless: a dropped frame here does not degrade a result, it silently
changes one.
**Measured, not inferred.** One 77 s clip at 5 fps should yield ~385 sampled
frames. On CPU it produced 49, ending at 51 s, with 285 frames dropped at
`camera_pos` and 51 at `face_aligner`. Rebuilt with CUDA the same clip ran in
29 s and reached EOF correctly — and still dropped **320** frames at
`camera_pos`, yielding 65. Faster hardware moves where the queue backs up; it
does not change what happens when it does.
Two consequences worth stating:
- **Raising channel capacity is a stopgap, not a fix.** It lowers the
probability of overflow without changing the behaviour on overflow, and the
failure it hides is silent corruption of the output.
- **Fixture generation is blocked on this** (VR-001), because what gets dropped
depends on timing. The same command run twice can produce different dumps, and
a golden fixture cannot be built on that.
**Current:** fixed in KPN — node data outputs use `push_blocking`, sentinels
remain out-of-band so EOF can always overtake a stalled data path. Verified on
the same clip: 385 of 385 sampled frames written, zero drops, and two
consecutive runs byte-identical where previously they were not.
It also ran *faster* (29 s → 17 s). A dropped frame has already cost its decode,
and the overflow exception cost more — so the lossy path was paying for work it
then discarded.
**Gap:** the remaining half — bounding by **bytes in flight** rather than item
count. Channel capacity is still a count of items, and a face carries a 112×112
crop plus a 512-float embedding, so a crowded frame occupies far more memory per
slot than a sparse one. That matters once `max_faces` is removed (AR-003).
**Gap:** entire requirement. This is a prerequisite for removing `max_faces`, not
a follow-up to it.
## AR-005 — Face alignment and crop
@@ -195,26 +154,6 @@ Two distinct signals, deliberately kept separate:
- **`is_scene_boundary`** — opt-in (`--scene-detect`). TransNetV2 over a densely
decoded, downscaled stream flags a *true shot/scene boundary*.
> **`is_scene_boundary` currently has no producer.** `grep -rn is_scene_boundary
> src/` finds no assignment anywhere: `SceneDetectorFunc` is a *terminal sink*
> (`main.cpp:298-300`, `kpn::out<>`) that writes `scenes.json` and never
> annotates the `Frame` flowing to the face pipeline. The field is therefore
> always `false`, and the dump column (`embedding_dump_node.hpp:38`) is a
> constant 0. Compounding it, `main.cpp:280` returns from the
> `--dump-embeddings` branch *before* the `scene_detect` branch at `:296`, so no
> dump-producing path even instantiates the detector.
>
> This makes AR-010 **not implemented**, not "in progress" — and it means a T2
> test of the frame-dependent `track_alpha` (AR-007) would **pass vacuously**,
> which is the worst possible failure for a verification gate. The fix is in the
> producer, not the schema: make `SceneDetectorFunc` a pass-through (or add a
> boundary annotator before the decimator) and add the scene branch to
> `dump_embeddings.cpp`. **No `schema_version` bump** — the column exists and
> merely stops being constant.
>
> Fixtures generated before the fix must be marked in provenance, since `0` is
> presently indistinguishable from "no boundary here".
Both feed AR-007 as **association hints**: they tell the tracker that spatial
continuity is broken and that association should weight embedding over IoU.
Neither ends a presence window (AR-012).
@@ -818,16 +757,9 @@ prerequisite.
## DP-005 — Installation and provisioning
- Native install, **no Docker at runtime** — GPU passthrough is the most fragile
part of a containerised setup and exists only because of the container.
Natively the GPU works with the host drivers and media paths need no
re-mounting. This constrains how the software *runs*, not how it is *built*:
DP-008 uses containers as build environments precisely because that side has
none of these problems.
- The installer may **fetch a prebuilt binary** (DP-008) instead of compiling.
Compiling stays supported, but should not be the only path — it is the slowest
and most fragile step of a first install. TRT engines are still built locally
either way (DP-008).
- Native install, **no Docker** — GPU passthrough is the most fragile part of a
containerised setup and exists only because of the container. Natively the GPU
works with the host drivers and media paths need no re-mounting.
- An installer (`scripts/build_install.py`) consuming one `install.yaml`:
platform (nvidia/amd/cpu), embedder model, gallery scan cadence, install
prefix; runtime secrets written to a `.env`, editable without recompiling.
@@ -838,117 +770,6 @@ prerequisite.
**Gap:** installer unbuilt.
## DP-007 — CI build image
CI runs on an Intel N100 with no discrete GPU, so the test build must configure
**CPU-only** and must not require CUDA, TensorRT or ROCm:
```
-DSAE_INFERENCE_BACKEND=ORT -DSAE_GEMM_BACKEND=CPU
```
A prebuilt container image supplies the toolchain, published to the **Gitea
container registry** and pinned by tag — matching the `jellytau-builder`
precedent. Building dependencies per CI run is untenable on an N100, and OpenCV 5
from source would dominate every run.
The same registry stores corpus dump fixtures as generic packages (see the
fixtures table in `requirements.md`). Rebuild the image when its dependency set
changes, not per run, and pin CI to a tag rather than `latest` so a rebuild
cannot silently change what a green build meant.
**Required in the image:**
| Dependency | Why |
|---|---|
| CMake, C++ toolchain, pkg-config | Build |
| **OpenCV 5** | `CMakeLists.txt:25` prefers 5, falls back to 4. The branch targets 5, so the image should carry it — it is not yet in most distro repos and building it per-run is prohibitive |
| HDF5 (C++) | Galleries are HDF5-native; also the dump format |
| FFmpeg dev libs — `libavformat`, `libavcodec`, `libavutil`, `libswscale`, **`libswresample`** | Decode. See the note below on swresample |
| Python 3 + numpy, h5py, scipy | Python-side tests, replay, traceability tooling |
| Catch2, nlohmann/json | **Vendored into the image, not fetched.** Both are `FetchContent`-ed today (`CMakeLists.txt:220`, `tests/CMakeLists.txt:8`), which makes every CI run depend on GitHub reachability |
**Deliberately excluded:** CUDA, TensorRT, ROCm — no GPU to use them. Also the
ONNX Runtime *GPU* providers; only the CPU provider is relevant, and only for T3
smoke tests.
**Models are not baked into the image.** The seven ONNX files total ~725 MB and
live in Git LFS. T1/T2 tests are model-free by design
(`tests/CMakeLists.txt:1-4`), so the default image needs none. T3 smoke tests
require a model and should pull it via LFS in a separate job rather than
inflating the image tenfold for a minority of tests.
**`libswresample` is a real gap, not a formality.** The current
`pkg_check_modules` list (`CMakeLists.txt:200-203`) covers avformat, avcodec,
avutil and swscale but **not** swresample — which IR-004 needs to downmix to mono
and resample to 11025 Hz. It must be added alongside the audio-signature work.
**Gap:** entire requirement. The image does not exist, and no CI config is
present in this repo.
## DP-008 — Builder images and release binaries
Produce prebuilt binaries per backend so deployment does not require every user
to compile the project.
**This does not contradict DP-005.** That requirement rejects Docker as a
*runtime* — GPU passthrough is the most fragile part of a containerised setup and
exists only because of the container. Using Docker as a *build* environment is
the opposite case: hermetic, reproducible, and it lets one machine produce
binaries for backends it cannot itself run. Build in a container; run natively.
### Image matrix
The build has two independent axes (`CMakeLists.txt:48-49`), so the useful
combinations are:
| Image | `SAE_INFERENCE_BACKEND` | `SAE_GEMM_BACKEND` | Target |
|---|---|---|---|
| `sae-builder-cpu` | ORT | CPU | CI (DP-007), and the smoke-test fallback |
| `sae-builder-cuda` | TRT | CUDA | NVIDIA |
| `sae-builder-rocm` | ORT | ROCM | AMD |
All three carry the DP-007 dependency set (OpenCV 5, HDF5, FFmpeg incl.
swresample, vendored Catch2/nlohmann) and differ only in the accelerator stack.
The CPU image is the CI image — one artifact, two uses.
Published to the Gitea container registry, pinned by tag, rebuilt when the
dependency set changes rather than per run.
### What ships, and what cannot
**Ships:** the `scene_analyze` binary and its companions, per backend.
**Cannot ship: TensorRT engines.** `.engine` files are specific to the GPU
architecture and TRT version they were built on — `scripts/build_trt_engines.sh`
must still run on the target machine. A prebuilt binary shortens the install; it
does not remove the local engine-build step, and the installer must not imply
otherwise.
**Cannot ship: models.** ~725 MB in LFS, and orthogonal to the binary.
### The constraint that decides the base image
**A binary built in a container runs against the host's glibc.** Build on a
newer base than the oldest supported host and it fails at load with
`GLIBC_2.xx not found` — the classic and entirely avoidable trap when shipping
binaries out of containers.
So the base is chosen for the *oldest* glibc to be supported, not for
convenience or recency. Accelerator libraries have the same shape of problem:
the binary links against a driver-provided runtime, so each image must document
the CUDA/ROCm version range its output is compatible with, and the installer
must check it rather than discovering a mismatch at first inference.
### Jobs
A release job per backend, producing a tagged artifact in the registry. These are
**not** the CI gate — the gate runs the CPU image on every push (DP-007);
release builds run on tag. Their outputs are what DP-005's installer fetches
when the user does not want to compile.
**Gap:** entire requirement. No images, no release jobs.
## DP-006 — Gallery maintenance as a background concern
- Incremental gallery refresh runs on a timer (`gallery_scan_interval`, default
@@ -1038,10 +859,8 @@ rewritten.
**Media shorter than 120 s.** The window `runtime/2 ± 60 s` underflows, so no
signature is emitted and **no sync offset is applied**. Such items fall back to
the runtime tier, which is adequate: a 90-second extra or trailer is not the
content whose cut alignment matters. (There is no `exact` tier: the file-hash
tier was withdrawn on legal grounds — it fingerprinted an individual copy rather
than the cut the timings describe. See the server spec §3.) Both producers must apply the identical
the runtime/exact tiers, which is adequate: a 90-second extra or trailer is not
the content whose cut alignment matters. Both producers must apply the identical
rule, or they diverge on exactly the short items most likely to be
mis-identified.
@@ -1126,57 +945,8 @@ surfaced as a build report.
different models are meaningless but *look* plausible — this fails silently and
expensively otherwise.
### The stamp
Two fields, written together: the model file's **basename** and the **SHA-256 of
its bytes** (plus `embed_dim` as a cheap extra guard). Stored as the `/embedder`
group in the gallery HDF5, and as an optional top-level `"embedder"` object in
the legacy JSON format.
The hash *decides*; the name is what a human *reads*. Neither alone is enough. A
name is a promise rather than a fact — models get re-exported, re-quantised and
overwritten in place under an unchanged filename, which is exactly the case where
the weights differ and nothing else does, so a name-only stamp is blind to the
failure it exists to catch. A hash alone is correct but unactionable: *"expected
3f2a…, got 9c1b…"* tells an operator nothing about what to do next. SHA-256 over
the file is derived from the artefact rather than asserted about it, needs no
registry kept up to date, and costs ~0.1 s for a 250 MB ONNX once per process.
### Verdicts
| Verdict | When | Default | Under strict mode |
|---|---|---|---|
| `match` | hashes agree | proceed | proceed |
| `weak_match` | names agree, one side unhashable | **warn** | **error** |
| `unstamped` | gallery predates GR-004 | **warn** | **error** |
| `unknown_embedder` | gallery stamped, embedder unidentifiable | **warn** | **error** |
| `mismatch` | proven different models | **error** | **error** |
**A mismatch is fatal in every mode, with no bypass**, and the message names both
sides — what the gallery was built with and what is loaded.
The three "cannot prove it" verdicts warn loudly instead, because they describe an
*unknown* state rather than a *known-bad* one, and because every gallery built
before this requirement is unstamped. Hard-failing all of them would make the
check something people route around rather than trust. Strict mode
(`--require-gallery-stamp`, or `SAE_REQUIRE_GALLERY_STAMP=1`, which propagates to
subprocesses) promotes them to errors — that is the mode measurement work runs in.
`scripts/stamp_gallery.py` re-binds an existing gallery without re-embedding, so
migration costs one command; that is what makes "warn" a temporary state rather
than a permanent one.
### Scope of the check
Embedding **dumps** carry the same stamp (`embedder_model` / `embedder_sha256`
root attributes, `scripts/optimizer/SCHEMA.md`): a replay has no live embedder, so
the dump *is* the embedder as far as the gallery is concerned. Derived galleries
(filter, cast-restrict) inherit their source's stamp; `--merge` and the JSON
gallery merge check *before* writing, since a merged file holding two embedding
spaces cannot be untangled afterwards by any later check.
**Gap:** none. Stamped in `gallery_builder.cpp` and the Python builders; verified
in `scene_analyze`, `scene_preview`, the `sae_kpn` matcher binding, `replay.py`,
`optimize.py`, `movienet_eval.py` and the merge paths.
**Gap:** named as step 4 of the `service-conversion.md` implementation plan;
unbuilt. This is the highest-value small fix in the document.
## GR-006 … GR-009 — Provenance tiers and poisoning guard
docs/plan.md
+5 -15
View File
@@ -260,23 +260,13 @@ Context crops opt-in behind `--dump-unidentified-crops`.
# Gallery
## GR-004 — Model binding — **DONE**
## GR-004 — Model binding
**Depended on:** nothing. Landed before any measurement work, as intended.
**Depends on:** nothing. **Startable immediately, highest value per line.**
Stamp = model basename + SHA-256 of the ONNX, written as the `/embedder` group at
build time (`gallery_builder.cpp`, `sae_gallery.save_gallery_hdf5`) and verified
at load in `scene_analyze`, `scene_preview`, the `sae_kpn` matcher binding,
`replay.py`, `optimize.py` and `movienet_eval.py`. Mismatch is a hard error naming
both sides, with no bypass. Embedding dumps carry the same stamp, since a replay
has no live embedder to check against.
Unstamped legacy galleries **warn loudly and proceed** rather than failing:
unknown is not known-bad, and hard-failing every pre-existing gallery would turn
the check into something people disable. `--require-gallery-stamp` /
`SAE_REQUIRE_GALLERY_STAMP=1` promotes that to a hard error — measurement runs
should set it. `scripts/stamp_gallery.py` re-binds an existing gallery without
re-embedding, so the warning state is cheap to leave.
Stamp embedder identity into the gallery at build; verify at load in
`scene_analyze`, `replay.py` and the optimizer. Mismatch is a hard error naming
both sides.
Cross-model similarities are meaningless but *look* plausible — this fails
silently and expensively, and it would corrupt every measurement taken during the
docs/requirements.md
+43 -145
View File
@@ -29,30 +29,30 @@ Status: `Done` · `In Progress` · `Planned` · `TBD` · `Withdrawn`
| ID | Requirement | Traces to | Priority | Status |
|---|---|---|---|---|
| AR-001 | Detect faces in sampled frames; emit bbox, confidence, 5-point landmarks in original pixel space | SR-002 | High | Done |
| AR-002 | Minimum face size **32×32 px** (VR-005 measured), expressed in **original** resolution (decoupled from `dense_scale`) | SR-002 | High | Planned |
| AR-002 | Minimum face size 66×66 px, expressed in **original** resolution (decoupled from `dense_scale`) | SR-002 | High | Planned |
| AR-003 | No fixed per-frame face cap — crowd scenes must not lose background cast | SR-002 | Medium | Planned |
| AR-004 | Backpressure: unbounded faces/frame absorbed by slowing, never by dropping or throwing | SR-002 | High | **Done** — KPN node outputs use `push_blocking`; sentinels stay out-of-band. Verified: 385/385 frames, 0 drops, byte-identical across runs |
| AR-004 | Backpressure: unbounded faces/frame absorbed by slowing, never by dropping or throwing | SR-002 | High | Planned |
| AR-005 | Align to 112×112 via ArcFace 5-point similarity transform | SR-002 | High | Done |
| AR-006 | 512-d L2-normalised embeddings, batched | SR-002 | High | Done |
| AR-007 | Associate detections by IoU + embedding, with **frame-dependent** weighting | SR-002 | High | **Done** — `track_alpha` is the base for ordinary frames; drops to embedding-only on cut/boundary and for dormant tracks |
| AR-008 | One track pool keyed on `last_seen`; no separate revival path | SR-002 | High | **Done** — one pool keyed on `last_seen`; park/revive branch deleted |
| AR-007 | Associate detections by IoU + embedding, with **frame-dependent** weighting | SR-002 | High | In Progress |
| AR-008 | One track pool keyed on `last_seen`; no separate revival path | SR-002 | High | Planned |
| AR-009 | Camera-cut detection (histogram) as an association hint | SR-002 | High | Done |
| AR-010 | Scene-boundary detection (TransNetV2) as an association hint | SR-002 | Medium | **Not started** — `is_scene_boundary` has no producer; `SceneDetectorFunc` is a terminal sink and never annotates the frame |
| AR-010 | Scene-boundary detection (TransNetV2) as an association hint | SR-002 | Medium | In Progress |
| AR-011 | **Every model is fed the input it was trained for** — cost reduced by running less often, never by degrading one inference | SR-002 | High | Planned |
| AR-012 | Presence follows **track extent**, not per-frame recognition | **SR-002** | High | **Done** — `src/track_registry.hpp`; window is `[first_seen, last_seen]` of an owned track |
| AR-013 | `last_seen` optional state machine; window ends at last sighting, never after | SR-002 | High | **Done** — `last_seen` optional is the whole state machine; interior gaps absorbed, trailing cool-down never claimed |
| AR-014 | Belief swap A→B terminates the track and starts a new one | SR-002 | Medium | **Done** — swap closes at `last_seen` and opens a successor at the swap frame; counted |
| AR-015 | Two live tracks owned by one actor ⇒ treat as a detected cut, re-associate | SR-002 | Medium | **Done** — reverse index detects it on the causing update; counted |
| AR-016 | All tracks closed at EOF — a film ends with faces on screen | SR-002 | High | **Done** — `flush()`, idempotent, closes at last sighting or final tick |
| AR-017 | Every presence claim carries its belief and identification route | SR-002 | High | **Done** — `DeadTrack` carries belief and observation count |
| AR-012 | Presence follows **track extent**, not per-frame recognition | **SR-002** | High | Planned |
| AR-013 | `last_seen` optional state machine; window ends at last sighting, never after | SR-002 | High | Planned |
| AR-014 | Belief swap A→B terminates the track and starts a new one | SR-002 | Medium | Planned |
| AR-015 | Two live tracks owned by one actor ⇒ treat as a detected cut, re-associate | SR-002 | Medium | Planned |
| AR-016 | All tracks closed at EOF — a film ends with faces on screen | SR-002 | High | Planned |
| AR-017 | Every presence claim carries its belief and identification route | SR-002 | High | Planned |
| AR-018 | Per-subject embedding store with banded admission (novel enough, safe enough) | SR-005 | Medium | Planned |
| AR-019 | Per-film gallery annex from owned tracks; acquires the non-frontal views TMDB lacks | SR-005 | Medium | In Progress |
| AR-020 | Deferred re-identification of unknown tracks against the final expanded gallery | SR-005 | High | Planned |
| AR-021 | Cluster unknown tracks into one entity per person, under temporal cannot-link constraints | SR-005 | Medium | Planned |
| AR-022 | Capture still-unidentified tracks: embeddings, metadata, **context crops** for human review | §4 | Medium | Planned |
| AR-023 | Fit sigmoid calibration from intra/inter similarity distributions | SR-002 | High | Done |
| AR-024 | **Always the calibrated probability, never a raw cosine** — exceptions recorded | SR-002 | High | **Done** — association and accumulation both in probability space; `track_max_embed_dist`, `cut_revive_sim` retired |
| AR-025 | Per-track Bayesian accumulation in log-odds, with correlated-observation discounting | SR-002 | High | **Done** — log-odds accumulation with correlation discounting owned by the registry, `src/evidence_discount.hpp` |
| AR-024 | **Always the calibrated probability, never a raw cosine** — exceptions recorded | SR-002 | High | Planned |
| AR-025 | Per-track Bayesian accumulation in log-odds, with correlated-observation discounting | SR-002 | High | Planned |
| AR-026 | All similarity computed as GEMM, including annex and deferred pass | SR-001 | High | In Progress |
| AR-027 | Throughput acceptable for **arbitrary** gallery size | SR-001 | High | Planned |
@@ -66,20 +66,18 @@ Status: `Done` · `In Progress` · `Planned` · `TBD` · `Withdrawn`
| DP-004 | Opportunistic/idle mode: external trigger, hard stop, implicit re-queue | PR-004 | Medium | Planned |
| DP-005 | Native installer, no Docker; Fedora + Arch | PR-004 | Medium | Planned |
| DP-006 | Background incremental gallery refresh on a timer | PR-003 | Medium | Planned |
| DP-007 | CI builder image, CPU-only, pinned by tag in the Gitea container registry | PR-004 | High | Planned |
| DP-008 | Builder images + release jobs per backend (cpu / cuda / rocm); ship binaries, not engines | PR-004 | Medium | Planned |
## Integration (IR)
| ID | Requirement | Traces to | Priority | Status |
|---|---|---|---|---|
| IR-001 | Emit the JRay truth format as sibling `.jray.json` | SR-003 | High | Done |
| IR-002 | Windows carry belief + route; `extraction.*` carries `extinction_sec`, `gallery_scope` | SR-003 | High | **Done** — `schema_version: 2`; windows are objects with `belief` + `route`; `extraction.*` carries `extinction_sec` and `gallery_scope`; `anneal_sec` removed |
| IR-003 | Output written **after** the deferred pass, not at EOF | SR-003 | High | **In Progress** — sink builds windows from registry claims and flushes at EOF; the deferred pass (AR-020) does not exist yet, so output is still final at EOF |
| IR-004 | Compute the audio signature exactly per server spec §3 | SR-003 | Medium | **Done** — `src/audio_signature.*`; not yet emitted into the truth file (IR-002) |
| IR-005 | Golden-vector fixture shared with the plugin repo to prove bit-exactness | SR-003 | High | **Done** — `tests/fixtures/audio/`; v1 parameters now normative in server spec §3 |
| IR-007 | Media < 120 s: emit no signature, apply no sync offset — identical rule in both producers | SR-003 | Low | **Done** |
| IR-008 | Emit and honour the signature's own `v1:` version prefix | SR-003 | Low | **Done** |
| IR-002 | Windows carry belief + route; `extraction.*` carries `extinction_sec`, `gallery_scope` | SR-003 | High | Planned |
| IR-003 | Output written **after** the deferred pass, not at EOF | SR-003 | High | Planned |
| IR-004 | Compute the audio signature exactly per server spec §3 | SR-003 | Medium | Planned |
| IR-005 | Golden-vector fixture shared with the plugin repo to prove bit-exactness | SR-003 | High | Planned |
| IR-007 | Media < 120 s: emit no signature, apply no sync offset — identical rule in both producers | SR-003 | Low | Planned |
| IR-008 | Emit and honour the signature's own `v1:` version prefix | SR-003 | Low | Planned |
| IR-006 | Jellyfin round-trip: pull pending queue, push complete results only | SR-001 | High | Done |
## Gallery (GR)
@@ -89,7 +87,7 @@ Status: `Done` · `In Progress` · `Planned` · `TBD` · `Withdrawn`
| GR-001 | Build gallery from Jellyfin library cast, TMDB profile fallback | SR-001, SR-005 | High | Done |
| GR-002 | Incremental `--merge` refresh without re-embedding known actors | PR-003 | High | Done |
| GR-003 | Report coverage: zero-image actors, under-referenced actors, dedup, calibration PDFs | SR-001 | Medium | Planned |
| GR-004 | Stamp embedder identity into the gallery; **hard startup error** on mismatch | SR-001 | High | **Done** — basename + SHA-256 + `embed_dim`; mismatch fatal with no bypass, unstamped warns unless `--require-gallery-stamp`; `scripts/stamp_gallery.py` migrates in place |
| GR-004 | Stamp embedder identity into the gallery; **hard startup error** on mismatch | SR-001 | High | Planned |
| GR-005 | Gallery data never leaves the instance | **SR-005** | High | Done |
| GR-006 | Provenance tiers: baked / harvested / confirmed, distinguishable per embedding | SR-005 | High | Planned |
| GR-007 | Persist harvested embeddings **flagged and reviewable**, never silently equal to baked | SR-005 | Medium | Planned |
@@ -101,16 +99,14 @@ Status: `Done` · `In Progress` · `Planned` · `TBD` · `Withdrawn`
| ID | Requirement | Traces to | Priority | Status |
|---|---|---|---|---|
| VR-001 | HDF5 post-inference dump at the embedded-frame boundary | PR-002 | High | Done |
| VR-002 | Replay drives the **real** KPN nodes, not a reimplementation | PR-002 | High | **Done** — replay driven from committed fixtures in `tests/test_replay_fixtures.cpp`; determinism asserted |
| VR-002 | Replay drives the **real** KPN nodes, not a reimplementation | PR-002 | High | Done |
| VR-003 | Scoring: micro-F1 against X-Ray, precision/recall logged at every evaluation | PR-002 | High | Done |
| VR-004 | Reproducible validation corpus with ground truth | PR-002 | High | Done |
| VR-005 | Minimum face size study — TPI/FPI vs probe size, gallery held at native res | PR-002 | Medium | **Done** — knee at 24–32 px; 32 px gives 98.1% TPI, 0.0 FPI at every size |
| VR-005 | Minimum face size study — TPI/FPI vs probe size, gallery held at native res | PR-002 | Medium | Planned |
| VR-006 | Re-tune `scene_threshold` once native-rate decode lands | PR-002 | Low | Planned |
| VR-007 | Expansion band, clustering threshold, and deferred-pass ablation | PR-002 | Medium | Planned |
| VR-008 | Gallery scaling benchmark — throughput vs gallery size | PR-002 | Medium | Planned |
| VR-009 | Verify accumulated posteriors are calibrated against held-out tracks | PR-002 | High | Planned |
| VR-010 | Dump provenance attributes — embedder model, detector settings, `dense_scale`, `scene_detect`, sample rate | PR-002 | **High** | Planned |
| VR-011 | Rewrite the replay harness for the post-AR-012 output contract | PR-002 | High | Planned |
---
@@ -124,62 +120,21 @@ Four tiers, in decreasing order of preference:
| Tier | Runs in CI | What it covers |
|---|---|---|
| **T1 — Functor unit** | Yes | A KPN node's `operator()` driven directly with hand-built inputs |
| **T2 — Replay** | Yes | The composed pipeline driven from an HDF5 fixture — no GPU, no video |
| **T1 — CPU unit** | Yes | Pure logic: registry state machine, belief accumulation, clustering, band admission, calibration maths |
| **T2 — Replay** | Yes | Real pipeline nodes driven from an HDF5 fixture — no GPU, no video |
| **T3 — CPU inference** | Yes, slowly | ORT CPU provider over a handful of frames; smoke tests only |
| **T4 — GPU** | **No** | Throughput, TRT engines, large-gallery GEMM |
### T1 is the primary tier, and KPN is why
**T2 is the reason this is workable.** The HDF5 dump (VR-001) captures state
after decode → detect → align → embed and before tracking and matching, so
everything downstream — which is where nearly all of the new design lives — is
cheap CPU maths replayable from a fixture. Tracking, presence windows, belief
accumulation, expansion, deferred re-identification and clustering are all
verifiable on an N100 at full fidelity, not in miniature.
**Node functors are plain callable structs, constructed independently of the
network that wraps them** (`main.cpp:186-207` builds them as stack objects;
`ObjectNode` merely adapts them). So a node is testable by constructing it and
calling `operator()` — no channels, no threads, no network, no fixture.
This is already the established pattern, not a proposal:
`tests/test_face_tracker.cpp` "drives the node's `operator()` with hand-built
`EmbeddedSceneFrame`s and inspects the emitted `track_ids`", and does so
"pure, GPU-free, model-free".
The consequence is that most of the redesign is verifiable **without any
fixture at all**: construct exactly the awkward state — a belief swap, two live
tracks converging on one actor, a film ending mid-track, a gap one frame under
the timeout — rather than hunting for a clip that happens to exhibit it.
Four hazards this removes outright:
- **No fixture-provenance risk** for these tests — the inputs are synthetic and
explicit.
- **No "fixture must be replayed from frame 0"** concern — state is constructed
directly.
- **No cross-test state leakage** (e.g. a tracker's `next_id_` persisting) — each
test constructs a fresh functor.
- **No replay-harness nondeterminism** — no channels, so no EOF-tail heuristics
or silent drops.
It also means **a dead upstream producer does not block testing a downstream
consumer.** `is_scene_boundary` currently has no producer (see AR-010), which
would make a *replay* test of the frame-dependent `track_alpha` pass vacuously —
but a T1 test simply constructs a frame with `is_scene_boundary = true` and
asserts the weighting changes. The producer gap is a pipeline defect to fix, not
a verification blocker.
### T2 covers what T1 cannot
Replay remains necessary for **composition** — that the nodes wired together
behave as the sum of their parts — and for realistic data at scale, which
synthetic inputs cannot honestly imitate. It is the tier that would catch a
wiring error, a channel-capacity problem, or an ordering assumption that only
appears under concurrency.
The HDF5 dump (VR-001) captures state after decode → detect → align → embed, so
replay needs no GPU and no video. That was built for the optimizer; it doubles as
CI, which is a strong argument for keeping the schema honest and for replay
driving the *real* nodes rather than a reimplementation (VR-002).
**Fixtures and studies are generated locally**, on the development machine where
the models, galleries and media already exist. CI consumes them; it never
produces them.
That was already true for the optimizer. It now doubles as the CI strategy, which
is a strong argument for keeping the dump schema honest (VR-001) and for the
replay driving the *real* nodes rather than a reimplementation (VR-002).
**Small committed fixtures are required.** A few HDF5 dumps covering the awkward
cases — a cut, a belief swap, two live tracks converging, a film ending
@@ -195,7 +150,7 @@ as such rather than counted as covered.
| Requirement | Tier | Note |
|---|---|---|
| AR-001, AR-005, AR-006 | T3 | Smoke only — correctness of detection/embedding is a model property, not ours |
| AR-002 | Minimum face size **32×32 px** (VR-005 measured), expressed in **original** resolution (decoupled from `dense_scale`) | Size filtering is arithmetic on dumped bboxes |
| AR-002 | T2 | Size filtering is arithmetic on dumped bboxes |
| AR-003, AR-004 | T1 + T4 | Backpressure logic is unit-testable; saturation behaviour needs real load |
| AR-007 … AR-017 | **T2** | The core of the redesign — fully replayable |
| AR-018 … AR-022 | **T2** | Expansion, deferred pass, clustering: all post-embedding |
@@ -213,46 +168,6 @@ structurally unverifiable on the CI host. It needs a GPU host and a synthetic
large gallery, so it is the requirement most likely to silently regress. Its
benchmark (VR-008) should run on a schedule rather than on demand.
### CI never calls a model
**Not "should not" — cannot.** The N100 has no GPU, and even the ONNX Runtime CPU
provider is impractical: a measured run of the embedder on this hardware sits at
~930 ms per frame, so a 77 s clip at 5 fps would take roughly six minutes of
inference alone. Every model invocation therefore happens **locally, ahead of
time**, and CI consumes the result as data.
This is what makes the T1/T2 split load-bearing rather than a preference: T1 and
T2 are the only tiers that can exist in CI at all.
### Fixture corpus — `bali/`
Five clips of **Road to Bali (1952)**, ~77 s each, 480×360, 30 fps, 42 MB total.
Public domain, and that is the reason to use it rather than a convenience:
**derived fixtures — dumps, crops, golden outputs — can be committed without the
rights question that rules out sharing gallery data (SR-005).** A fixture cut
from a copyrighted title could not live in the repository at all.
Two properties to design around rather than discover:
- **480×360 means small faces.** At this resolution a face is often 40–80 px, so
the AR-002 minimum of 66 px (original resolution) rejects much of what is
there. Fixture generation must set `--min-face-px` explicitly and record it,
or the dumps will be sparse for reasons unrelated to what is being tested.
- **77 s is short.** At 1 fps that is 77 frames — too thin to exercise an
extinction window measured in tens of seconds. Generate at 5 fps (≈385 frames,
~1 MB) and record the rate in provenance, since the behaviour under test
changes with it.
> **AR-004 blocks reproducible fixture generation.** A trial run of one clip
> produced 49 frames of an expected ~385, ending at 51 s of 77 s, with the
> diagnostics reporting 285 frames dropped at `camera_pos` and 51 at
> `face_aligner`. Channels overflow and **drop** rather than blocking, and what
> gets dropped depends on timing — so the same command run twice can produce
> different dumps. Golden fixtures cannot be built on that. AR-004 is therefore
> a prerequisite for VR-001 fixtures, not merely a throughput concern for crowd
> scenes.
### Fixtures — precomputed inference, pulled by CI
The N100 cannot run inference at any useful rate, so **inference output is
@@ -261,29 +176,16 @@ what looks like GPU work into pure CPU replay.
| Fixture | Contents | Size | Storage |
|---|---|---|---|
| **Edge-case dumps** | ~6 short clips (30–60 s), one per awkward behaviour | ~0.1–1 MB each | **Committed in-repo** |
| **Corpus dumps** | Full-length titles from the validation corpus | ~21–38 MB each | **Gitea package registry**, pinned by version + checksum |
| **Edge-case dumps** | ~6 short clips (30–60 s), one per awkward behaviour | ~1 MB each | **Committed in-repo** |
| **Corpus dumps** | Full-length titles from the validation corpus | ~30 MB each | Pinned artifact, fetched by checksum |
| **Synthetic gallery** | Random unit-norm embeddings, fixed seed | small | Generated at test time |
| **Golden truth files** | Expected output for each edge-case dump | KB | Committed |
| **Audio golden vectors** | FLAC + expected signature + parameter contract | ~600 KB | Committed, **shared with the plugin repo** |
| **Audio golden vectors** | Short WAV + expected signature | KB | Committed, **shared with the plugin repo** |
Edge-case dumps are small enough to commit, and being in-repo means they version
with the code that reads them.
**Corpus dumps go to the Gitea package registry, not Git LFS.** Both are
available — the models already use LFS — but their fetch semantics differ in a
way that matters here. LFS objects are pulled on clone unless a developer
explicitly skips them, so ~38 MB per title behind LFS taxes everyone who clones,
forever, for data that only CI and the optimizer ever read. Registry artifacts
are fetched on demand by the job that needs them.
Rule of thumb: **LFS for what the build needs; the package registry for what a
particular job needs.** Models are the former; corpus dumps and the CI image
(DP-007) are the latter.
Pin by version and verify by checksum on fetch. A fixture that changes silently
under CI is worse than a missing one, because the failure presents as a code
regression.
with the code that reads them. Corpus dumps are pulled by pinned checksum from
the artifact store rather than committed, since they are large and change only
when the dump schema does.
**Generation must be reproducible and versioned.** A script, run on a GPU host,
regenerates every fixture from source clips; it is re-run when the VR-001 schema
@@ -302,7 +204,7 @@ because it will be trusted.
| ID | Tier | Test asserts | Edge cases to cover |
|---|---|---|---|
| AR-001 | T3 | Detector returns plausible boxes on a known frame | — smoke only |
| AR-002 | Minimum face size **32×32 px** (VR-005 measured), expressed in **original** resolution (decoupled from `dense_scale`) | Faces below 66 px (original res) are dropped | Exactly at threshold; with `dense_scale` 0.5 — the interaction that motivated the requirement |
| AR-002 | T2 | Faces below 66 px (original res) are dropped | Exactly at threshold; with `dense_scale` 0.5 — the interaction that motivated the requirement |
| AR-003 | T2 | No cap applied; a 40-face frame yields 40 | Crowd frame |
| AR-004 | T1 | Saturated input blocks rather than drops or throws | Bounded queue at capacity; **byte-based** limit with large crops; SIGTERM mid-block |
| AR-005 | T1 | Known landmarks → expected 112×112 warp | Landmarks near frame edge; degenerate/collinear points |
@@ -329,12 +231,8 @@ because it will be trusted.
| AR-027 | **T4** | Throughput at 10²…10⁵ actors | Scheduled, not on-demand |
| IR-001/002 | T1 | Serialised output matches golden file | Zero-length window; actor with many windows |
| IR-003 | T1 | Output written after deferred pass | Not at EOF |
| IR-004/005 | **T1** | Signature matches golden vector bit-for-bit | Identical result in both producer repos |
| IR-006 | T1 + manual | Queue pull and result push against a stubbed Jellyfin API | Partial result never pushed; push only after the deferred pass |
| IR-007 | **T1** | Media < 120 s emits no signature at all | Exactly 120 s; just under; zero-length audio. Must match the plugin's cutoff exactly — a caller-varying window length is what SR-004 forbids |
| IR-008 | T1 | `v1:` prefix emitted and honoured on read | Unknown prefix rejected, not guessed |
| GR-009 | T1 | Human-confirmed associations persist and are tier-tagged | Survives a gallery rebuild; distinguishable from baked and harvested |
| GR-004 | T1 | Mismatched embedder → hard startup error | Error names both sides; **unstamped warns, and errors under `SAE_REQUIRE_GALLERY_STAMP`**; same filename + different SHA-256 must still be a mismatch |
| IR-004/005 | **T1** | Signature matches golden vector bit-for-bit | **Media < 120 s → no signature**; identical result in both repos |
| GR-004 | T1 | Mismatched embedder → hard startup error | Error names both sides |
| GR-008 | T1 | Outlier flagged among an actor's references | Injected poisoned embedding detected |
| VR-009 | T1 | Posterior calibration holds | A 0.99 posterior is wrong ~1% of the time on held-out tracks |
docs/traceability.md
+51 -511
View File
@@ -3,7 +3,7 @@
<!-- GENERATED FILE - do not edit by hand. -->
<!-- Regenerate: scripts/traceability/traceability-gate.sh -->
**Generated:** 2026-07-31T08:35:29+00:00
**Generated:** 2026-07-30T16:35:36+00:00
Denominators are read from [`requirements.md`](requirements.md) at run time, never hardcoded. Coverage counts a requirement only when it is tagged in source **and** has a verification tier this repo's CI host can execute (`T1, T2, T3, static`).
@@ -11,29 +11,26 @@ Denominators are read from [`requirements.md`](requirements.md) at run time, nev
| Metric | Value |
|---|---|
| Source files scanned | 95 |
| TRACES tags found | 90 |
| Source files scanned | 86 |
| TRACES tags found | 0 |
| EXCEPTION tags found | 0 |
| Requirements defined | 63 |
| Requirements covered | 26 |
| **Coverage** | **41.3%** (26/63) |
| Coverage of CI-executable scope | 50.0% (26/52) |
| Tagged but unexecuted in CI | 3 |
| Requirements defined | 59 |
| Requirements covered | 0 |
| **Coverage** | **0.0%** (0/59) |
| Coverage of CI-executable scope | 0.0% (0/50) |
| Tagged but unexecuted in CI | 0 |
| Orphan tags | 0 |
### By type
| Type | Covered | Tagged but unexecuted | Defined |
|---|---|---|---|
| AR | 13 | 0 | 27 |
| DP | 2 | 0 | 8 |
| IR | 8 | 0 | 8 |
| GR | 3 | 0 | 9 |
| VR | 0 | 3 | 11 |
| AR | 0 | 0 | 27 |
| DP | 0 | 0 | 6 |
| IR | 0 | 0 | 8 |
| GR | 0 | 0 | 9 |
| VR | 0 | 0 | 9 |
- **UT** tags present (separate taxonomy, not counted in coverage): UT-001, UT-101, UT-102, UT-103, UT-104
- **PR** tags present (separate taxonomy, not counted in coverage): PR-002, PR-004
- **SR** tags present (separate taxonomy, not counted in coverage): SR-001, SR-002, SR-003, SR-005
## Not executable in CI
@@ -42,18 +39,14 @@ These requirements have no verification tier this repo's CI host can run, so a t
| ID | Tiers | Tagged in source | Requirement |
|---|---|---|---|
| AR-027 | T4 | no | Throughput acceptable for **arbitrary** gallery size |
| VR-001 | out-of-ci | yes | HDF5 post-inference dump at the embedded-frame boundary |
| VR-002 | out-of-ci | yes | Replay drives the **real** KPN nodes, not a reimplementation |
| VR-003 | out-of-ci | yes | Scoring: micro-F1 against X-Ray, precision/recall logged at every eva… |
| VR-001 | out-of-ci | no | HDF5 post-inference dump at the embedded-frame boundary |
| VR-002 | out-of-ci | no | Replay drives the **real** KPN nodes, not a reimplementation |
| VR-003 | out-of-ci | no | Scoring: micro-F1 against X-Ray, precision/recall logged at every eva… |
| VR-004 | out-of-ci | no | Reproducible validation corpus with ground truth |
| VR-005 | out-of-ci | no | Minimum face size study — TPI/FPI vs probe size, gallery held at nati… |
| VR-006 | out-of-ci | no | Re-tune `scene_threshold` once native-rate decode lands |
| VR-007 | out-of-ci | no | Expansion band, clustering threshold, and deferred-pass ablation |
| VR-008 | out-of-ci | no | Gallery scaling benchmark — throughput vs gallery size |
| VR-010 | out-of-ci | no | Dump provenance attributes — embedder model, detector settings, `dens… |
| VR-011 | out-of-ci | no | Rewrite the replay harness for the post-AR-012 output contract |
**Tagged but unexecuted:** VR-001, VR-002, VR-003 — a test exists and is tagged, but this CI host cannot run it. Report those runs separately.
## Orphan tags
@@ -77,520 +70,67 @@ _None._
| ID | Status | Tier | Traces to | Trace state | Tagged in | Requirement |
|---|---|---|---|---|---|---|
| AR-001 | Done | T3 | SR-002 | covered | `src/nodes/face_detector_node.hpp` | Detect faces in sampled frames; emit bbox, confidence, 5-point landma… |
| AR-002 | Planned | unset | SR-002 | untagged | - | Minimum face size **32×32 px** (VR-005 measured), expressed in **orig… |
| AR-001 | Done | T3 | SR-002 | untagged | - | Detect faces in sampled frames; emit bbox, confidence, 5-point landma… |
| AR-002 | Planned | T2 | SR-002 | untagged | - | Minimum face size 66×66 px, expressed in **original** resolution (dec… |
| AR-003 | Planned | T1, T2, T4 | SR-002 | untagged | - | No fixed per-frame face cap — crowd scenes must not lose background c… |
| AR-004 | **Done** — KPN node… | T1, T4 | SR-002 | untagged | - | Backpressure: unbounded faces/frame absorbed by slowing, never by dro… |
| AR-005 | Done | T1, T3 | SR-002 | covered | `src/face_utils.hpp` | Align to 112×112 via ArcFace 5-point similarity transform |
| AR-004 | Planned | T1, T4 | SR-002 | untagged | - | Backpressure: unbounded faces/frame absorbed by slowing, never by dro… |
| AR-005 | Done | T1, T3 | SR-002 | untagged | - | Align to 112×112 via ArcFace 5-point similarity transform |
| AR-006 | Done | T3 | SR-002 | untagged | - | 512-d L2-normalised embeddings, batched |
| AR-007 | **Done** — `track_a… | T2 | SR-002 | covered | `src/config.hpp`, `src/main.cpp`, `src/nodes/face_tracker_node.hpp` | Associate detections by IoU + embedding, with **frame-dependent** wei… |
| AR-008 | **Done** — one pool… | T2 | SR-002 | covered | `src/config.hpp`, `src/main.cpp`, `src/nodes/face_tracker_node.hpp` | One track pool keyed on `last_seen`; no separate revival path |
| AR-007 | In Progress | T2 | SR-002 | untagged | - | Associate detections by IoU + embedding, with **frame-dependent** wei… |
| AR-008 | Planned | T2 | SR-002 | untagged | - | One track pool keyed on `last_seen`; no separate revival path |
| AR-009 | Done | T2 | SR-002 | untagged | - | Camera-cut detection (histogram) as an association hint |
| AR-010 | **Not started** — `… | T2 | SR-002 | untagged | - | Scene-boundary detection (TransNetV2) as an association hint |
| AR-010 | In Progress | T2 | SR-002 | untagged | - | Scene-boundary detection (TransNetV2) as an association hint |
| AR-011 | Planned | T1, T2 | SR-002 | untagged | - | **Every model is fed the input it was trained for** — cost reduced by… |
| AR-012 | **Done** — `src/tra… | T2 | **SR-002** | covered | `src/main.cpp`, `src/nodes/identity_matcher_node.hpp`, `src/nodes/result_sink_node.hpp`, `src/track_registry.hpp`, `tests/test_track_registry.cpp` | Presence follows **track extent**, not per-frame recognition |
| AR-013 | **Done** — `last_se… | T2 | SR-002 | covered | `src/track_registry.hpp`, `tests/test_track_registry.cpp` | `last_seen` optional state machine; window ends at last sighting, nev… |
| AR-014 | **Done** — swap clo… | T2 | SR-002 | covered | `src/track_registry.hpp`, `tests/test_track_registry.cpp` | Belief swap A→B terminates the track and starts a new one |
| AR-015 | **Done** — reverse … | T2 | SR-002 | covered | `src/track_registry.hpp`, `tests/test_track_registry.cpp` | Two live tracks owned by one actor ⇒ treat as a detected cut, re-asso… |
| AR-016 | **Done** — `flush()… | T2 | SR-002 | covered | `src/main.cpp`, `src/nodes/result_sink_node.hpp`, `src/track_registry.hpp`, `tests/test_track_registry.cpp` | All tracks closed at EOF — a film ends with faces on screen |
| AR-017 | **Done** — `DeadTra… | T1, T2 | SR-002 | covered | `src/nodes/result_sink_node.hpp`, `src/track_registry.hpp`, `tests/test_track_registry.cpp` | Every presence claim carries its belief and identification route |
| AR-012 | Planned | T2 | **SR-002** | untagged | - | Presence follows **track extent**, not per-frame recognition |
| AR-013 | Planned | T2 | SR-002 | untagged | - | `last_seen` optional state machine; window ends at last sighting, nev… |
| AR-014 | Planned | T2 | SR-002 | untagged | - | Belief swap A→B terminates the track and starts a new one |
| AR-015 | Planned | T2 | SR-002 | untagged | - | Two live tracks owned by one actor ⇒ treat as a detected cut, re-asso… |
| AR-016 | Planned | T2 | SR-002 | untagged | - | All tracks closed at EOF — a film ends with faces on screen |
| AR-017 | Planned | T1, T2 | SR-002 | untagged | - | Every presence claim carries its belief and identification route |
| AR-018 | Planned | T1, T2 | SR-005 | untagged | - | Per-subject embedding store with banded admission (novel enough, safe… |
| AR-019 | In Progress | T2 | SR-005 | untagged | - | Per-film gallery annex from owned tracks; acquires the non-frontal vi… |
| AR-020 | Planned | T2 | SR-005 | untagged | - | Deferred re-identification of unknown tracks against the final expand… |
| AR-021 | Planned | T2 | SR-005 | untagged | - | Cluster unknown tracks into one entity per person, under temporal can… |
| AR-022 | Planned | T1, T2 | §4 | untagged | - | Capture still-unidentified tracks: embeddings, metadata, **context cr… |
| AR-023 | Done | T1 | SR-002 | covered | `src/gallery/gallery_calibration.hpp`, `src/nodes/identity_matcher_node.hpp` | Fit sigmoid calibration from intra/inter similarity distributions |
| AR-024 | **Done** — associat… | T1, static | SR-002 | covered | `src/config.hpp`, `src/evidence_discount.hpp`, `src/gallery/gallery_calibration.hpp`, `src/main.cpp`, `src/nodes/face_tracker_node.hpp`, `src/nodes/identity_matcher_node.hpp` | **Always the calibrated probability, never a raw cosine** — exception… |
| AR-025 | **Done** — log-odds… | T1 | SR-002 | covered | `src/evidence_discount.hpp`, `src/nodes/identity_matcher_node.hpp` | Per-track Bayesian accumulation in log-odds, with correlated-observat… |
| AR-023 | Done | T1 | SR-002 | untagged | - | Fit sigmoid calibration from intra/inter similarity distributions |
| AR-024 | Planned | T1, static | SR-002 | untagged | - | **Always the calibrated probability, never a raw cosine** — exception… |
| AR-025 | Planned | T1 | SR-002 | untagged | - | Per-track Bayesian accumulation in log-odds, with correlated-observat… |
| AR-026 | In Progress | T1, T4 | SR-001 | untagged | - | All similarity computed as GEMM, including annex and deferred pass |
| AR-027 | Planned | T4 | SR-001 | untagged | - | Throughput acceptable for **arbitrary** gallery size |
| DP-001 | Done | T1, manual | PR-004 | covered | `src/main.cpp` | One analysis core; modes are front-ends and must not fork pipeline lo… |
| DP-002 | Done | T1, manual | PR-004 | covered | `src/main.cpp` | Batch CLI over one title |
| DP-001 | Done | T1, manual | PR-004 | untagged | - | One analysis core; modes are front-ends and must not fork pipeline lo… |
| DP-002 | Done | T1, manual | PR-004 | untagged | - | Batch CLI over one title |
| DP-003 | Planned | T1, manual | PR-004 | untagged | - | On-demand resident service with bounded, observable queue |
| DP-004 | Planned | T1, manual | PR-004 | untagged | - | Opportunistic/idle mode: external trigger, hard stop, implicit re-que… |
| DP-005 | Planned | T1, manual | PR-004 | untagged | - | Native installer, no Docker; Fedora + Arch |
| DP-006 | Planned | T1, manual | PR-003 | untagged | - | Background incremental gallery refresh on a timer |
| DP-007 | Planned | T1, manual | PR-004 | untagged | - | CI builder image, CPU-only, pinned by tag in the Gitea container regi… |
| DP-008 | Planned | T1, manual | PR-004 | untagged | - | Builder images + release jobs per backend (cpu / cuda / rocm); ship b… |
| IR-001 | Done | T1 | SR-003 | covered | `src/nodes/result_sink_node.hpp` | Emit the JRay truth format as sibling `.jray.json` |
| IR-002 | **Done** — `schema_… | T1 | SR-003 | covered | `src/config.hpp`, `src/main.cpp`, `src/nodes/result_sink_node.hpp` | Windows carry belief + route; `extraction.*` carries `extinction_sec`… |
| IR-003 | **In Progress** — s… | T1 | SR-003 | covered | `src/main.cpp` | Output written **after** the deferred pass, not at EOF |
| IR-004 | **Done** — `src/aud… | T1 | SR-003 | covered | `src/audio_signature.cpp`, `src/audio_signature.hpp`, `tests/test_audio_signature.cpp` | Compute the audio signature exactly per server spec §3 |
| IR-005 | **Done** — `tests/f… | T1 | SR-003 | covered | `src/audio_signature.cpp`, `src/audio_signature.hpp`, `tests/test_audio_signature.cpp` | Golden-vector fixture shared with the plugin repo to prove bit-exactn… |
| IR-006 | Done | T1, manual | SR-001 | covered | `scripts/run_from_jellyfin.py` | Jellyfin round-trip: pull pending queue, push complete results only |
| IR-007 | **Done** | T1 | SR-003 | covered | `src/audio_signature.cpp`, `src/audio_signature.hpp`, `tests/test_audio_signature.cpp` | Media < 120 s: emit no signature, apply no sync offset — identical ru… |
| IR-008 | **Done** | T1 | SR-003 | covered | `src/audio_signature.cpp`, `src/audio_signature.hpp`, `tests/test_audio_signature.cpp` | Emit and honour the signature's own `v1:` version prefix |
| GR-001 | Done | T1, T3 | SR-001, SR-005 | covered | `scripts/make_jellyfin_gallery.py` | Build gallery from Jellyfin library cast, TMDB profile fallback |
| GR-002 | Done | T1, T3 | PR-003 | covered | `scripts/make_jellyfin_gallery.py` | Incremental `--merge` refresh without re-embedding known actors |
| IR-001 | Done | T1 | SR-003 | untagged | - | Emit the JRay truth format as sibling `.jray.json` |
| IR-002 | Planned | T1 | SR-003 | untagged | - | Windows carry belief + route; `extraction.*` carries `extinction_sec`… |
| IR-003 | Planned | T1 | SR-003 | untagged | - | Output written **after** the deferred pass, not at EOF |
| IR-004 | Planned | T1 | SR-003 | untagged | - | Compute the audio signature exactly per server spec §3 |
| IR-005 | Planned | T1 | SR-003 | untagged | - | Golden-vector fixture shared with the plugin repo to prove bit-exactn… |
| IR-006 | Done | unset | SR-001 | untagged | - | Jellyfin round-trip: pull pending queue, push complete results only |
| IR-007 | Planned | unset | SR-003 | untagged | - | Media < 120 s: emit no signature, apply no sync offset — identical ru… |
| IR-008 | Planned | unset | SR-003 | untagged | - | Emit and honour the signature's own `v1:` version prefix |
| GR-001 | Done | T1, T3 | SR-001, SR-005 | untagged | - | Build gallery from Jellyfin library cast, TMDB profile fallback |
| GR-002 | Done | T1, T3 | PR-003 | untagged | - | Incremental `--merge` refresh without re-embedding known actors |
| GR-003 | Planned | T1, T3 | SR-001 | untagged | - | Report coverage: zero-image actors, under-referenced actors, dedup, c… |
| GR-004 | **Done** — basename… | T1, T3 | SR-001 | covered | `scripts/filter_gallery.py`, `scripts/make_gallery.py`, `scripts/make_jellyfin_gallery.py`, `scripts/movienet_eval.py`, `scripts/optimizer/fetch_missing_actors.py`, `scripts/optimizer/optimize.py`, `scripts/optimizer/reembed_gallery.py`, `scripts/optimizer/replay.py`, `scripts/sae_embed_loader.py`, `scripts/sae_gallery.py`, `scripts/sae_stamp.py`, `scripts/stamp_gallery.py`, `src/config.hpp`, `src/gallery/embedder_stamp.cpp`, `src/gallery/embedder_stamp.hpp`, `src/gallery/gallery_builder.cpp`, `src/gallery/gallery_store.cpp`, `src/kpn_bindings.cpp`, `src/main.cpp`, `src/nodes/embedding_dump_node.hpp`, `src/scene_preview.cpp`, `src/types.hpp`, `tests/test_gallery_store.cpp` | Stamp embedder identity into the gallery; **hard startup error** on m… |
| GR-004 | Planned | T1, T3 | SR-001 | untagged | - | Stamp embedder identity into the gallery; **hard startup error** on m… |
| GR-005 | Done | T1, T3 | **SR-005** | untagged | - | Gallery data never leaves the instance |
| GR-006 | Planned | T1 | SR-005 | untagged | - | Provenance tiers: baked / harvested / confirmed, distinguishable per … |
| GR-007 | Planned | T1 | SR-005 | untagged | - | Persist harvested embeddings **flagged and reviewable**, never silent… |
| GR-008 | Planned | T1 | SR-005 | untagged | - | Flag distributional outliers among an actor's references (poisoning g… |
| GR-009 | TBD | T1 | §4 | untagged | - | Human-confirmed associations persist and improve future extractions |
| VR-001 | Done | out-of-ci | PR-002 | tagged, unexecuted | `src/nodes/embedding_dump_node.hpp` | HDF5 post-inference dump at the embedded-frame boundary |
| VR-002 | Done | out-of-ci | PR-002 | tagged, unexecuted | `scripts/optimizer/replay.py` | Replay drives the **real** KPN nodes, not a reimplementation |
| VR-003 | Done | out-of-ci | PR-002 | tagged, unexecuted | `scripts/optimizer/second_score.py` | Scoring: micro-F1 against X-Ray, precision/recall logged at every eva… |
| GR-009 | TBD | unset | §4 | untagged | - | Human-confirmed associations persist and improve future extractions |
| VR-001 | Done | out-of-ci | PR-002 | untagged | - | HDF5 post-inference dump at the embedded-frame boundary |
| VR-002 | Done | out-of-ci | PR-002 | untagged | - | Replay drives the **real** KPN nodes, not a reimplementation |
| VR-003 | Done | out-of-ci | PR-002 | untagged | - | Scoring: micro-F1 against X-Ray, precision/recall logged at every eva… |
| VR-004 | Done | out-of-ci | PR-002 | untagged | - | Reproducible validation corpus with ground truth |
| VR-005 | **Done** — knee at … | out-of-ci | PR-002 | untagged | - | Minimum face size study — TPI/FPI vs probe size, gallery held at nati… |
| VR-005 | Planned | out-of-ci | PR-002 | untagged | - | Minimum face size study — TPI/FPI vs probe size, gallery held at nati… |
| VR-006 | Planned | out-of-ci | PR-002 | untagged | - | Re-tune `scene_threshold` once native-rate decode lands |
| VR-007 | Planned | out-of-ci | PR-002 | untagged | - | Expansion band, clustering threshold, and deferred-pass ablation |
| VR-008 | Planned | out-of-ci | PR-002 | untagged | - | Gallery scaling benchmark — throughput vs gallery size |
| VR-009 | Planned | T1, out-of-ci | PR-002 | untagged | - | Verify accumulated posteriors are calibrated against held-out tracks |
| VR-010 | Planned | out-of-ci | PR-002 | untagged | - | Dump provenance attributes — embedder model, detector settings, `dens… |
| VR-011 | Planned | out-of-ci | PR-002 | untagged | - | Rewrite the replay harness for the post-AR-012 output contract |
## Detailed mapping
### AR-001
**Locations:** 1
- [`src/nodes/face_detector_node.hpp:2`](../src/nodes/face_detector_node.hpp#L2) — `Unknown`
### AR-005
**Locations:** 1
- [`src/face_utils.hpp:2`](../src/face_utils.hpp#L2) — `inline cv::Mat align_face(const cv::Mat& img,`
### AR-007
**Locations:** 3
- [`src/config.hpp:103`](../src/config.hpp#L103) — `Unknown`
- [`src/main.cpp:199`](../src/main.cpp#L199) — `reg_cfg, EvidenceDiscounter(same_person));`
- [`src/nodes/face_tracker_node.hpp:2`](../src/nodes/face_tracker_node.hpp#L2) — `Unknown`
### AR-008
**Locations:** 3
- [`src/config.hpp:103`](../src/config.hpp#L103) — `Unknown`
- [`src/main.cpp:199`](../src/main.cpp#L199) — `reg_cfg, EvidenceDiscounter(same_person));`
- [`src/nodes/face_tracker_node.hpp:2`](../src/nodes/face_tracker_node.hpp#L2) — `Unknown`
### AR-012
**Locations:** 8
- [`src/main.cpp:199`](../src/main.cpp#L199) — `reg_cfg, EvidenceDiscounter(same_person));`
- [`src/main.cpp:216`](../src/main.cpp#L216) — `reg_cfg, EvidenceDiscounter(same_person));`
- [`src/nodes/identity_matcher_node.hpp:119`](../src/nodes/identity_matcher_node.hpp#L119) — `const GalleryCalibration& calibration() const { return cal_; }`
- [`src/nodes/identity_matcher_node.hpp:242`](../src/nodes/identity_matcher_node.hpp#L242) — `Unknown`
- [`src/nodes/result_sink_node.hpp:49`](../src/nodes/result_sink_node.hpp#L49) — `static constexpr std::string_view label() { return "result_sink"; }`
- [`src/nodes/result_sink_node.hpp:161`](../src/nodes/result_sink_node.hpp#L161) — `struct ActorMeta { std::string name, imdb_id, tmdb_id, jellyfin_id; };`
- [`src/track_registry.hpp:2`](../src/track_registry.hpp#L2) — `Unknown`
- [`tests/test_track_registry.cpp:3`](../tests/test_track_registry.cpp#L3) — `Unknown`
### AR-013
**Locations:** 2
- [`src/track_registry.hpp:2`](../src/track_registry.hpp#L2) — `Unknown`
- [`tests/test_track_registry.cpp:3`](../tests/test_track_registry.cpp#L3) — `Unknown`
### AR-014
**Locations:** 2
- [`src/track_registry.hpp:2`](../src/track_registry.hpp#L2) — `Unknown`
- [`tests/test_track_registry.cpp:3`](../tests/test_track_registry.cpp#L3) — `Unknown`
### AR-015
**Locations:** 2
- [`src/track_registry.hpp:2`](../src/track_registry.hpp#L2) — `Unknown`
- [`tests/test_track_registry.cpp:3`](../tests/test_track_registry.cpp#L3) — `Unknown`
### AR-016
**Locations:** 4
- [`src/main.cpp:216`](../src/main.cpp#L216) — `reg_cfg, EvidenceDiscounter(same_person));`
- [`src/nodes/result_sink_node.hpp:63`](../src/nodes/result_sink_node.hpp#L63) — `void set_pre_write_hook(std::function<void(double)> fn) { pre_write_ = std::move(fn); }`
- [`src/track_registry.hpp:2`](../src/track_registry.hpp#L2) — `Unknown`
- [`tests/test_track_registry.cpp:3`](../tests/test_track_registry.cpp#L3) — `Unknown`
### AR-017
**Locations:** 3
- [`src/nodes/result_sink_node.hpp:49`](../src/nodes/result_sink_node.hpp#L49) — `static constexpr std::string_view label() { return "result_sink"; }`
- [`src/track_registry.hpp:2`](../src/track_registry.hpp#L2) — `Unknown`
- [`tests/test_track_registry.cpp:3`](../tests/test_track_registry.cpp#L3) — `Unknown`
### AR-023
**Locations:** 3
- [`src/gallery/gallery_calibration.hpp:2`](../src/gallery/gallery_calibration.hpp#L2) — `Unknown`
- [`src/gallery/gallery_calibration.hpp:53`](../src/gallery/gallery_calibration.hpp#L53) — `float boundary_at(float p = 0.5f, float log_prior_odds = 0.f) const`
- [`src/nodes/identity_matcher_node.hpp:111`](../src/nodes/identity_matcher_node.hpp#L111) — `const GalleryCalibration& calibration() const { return cal_; }`
### AR-024
**Locations:** 6
- [`src/config.hpp:103`](../src/config.hpp#L103) — `Unknown`
- [`src/evidence_discount.hpp:2`](../src/evidence_discount.hpp#L2) — `Unknown`
- [`src/gallery/gallery_calibration.hpp:53`](../src/gallery/gallery_calibration.hpp#L53) — `float boundary_at(float p = 0.5f, float log_prior_odds = 0.f) const`
- [`src/main.cpp:199`](../src/main.cpp#L199) — `reg_cfg, EvidenceDiscounter(same_person));`
- [`src/nodes/face_tracker_node.hpp:2`](../src/nodes/face_tracker_node.hpp#L2) — `Unknown`
- [`src/nodes/identity_matcher_node.hpp:111`](../src/nodes/identity_matcher_node.hpp#L111) — `const GalleryCalibration& calibration() const { return cal_; }`
### AR-025
**Locations:** 3
- [`src/evidence_discount.hpp:2`](../src/evidence_discount.hpp#L2) — `Unknown`
- [`src/nodes/identity_matcher_node.hpp:119`](../src/nodes/identity_matcher_node.hpp#L119) — `const GalleryCalibration& calibration() const { return cal_; }`
- [`src/nodes/identity_matcher_node.hpp:242`](../src/nodes/identity_matcher_node.hpp#L242) — `Unknown`
### DP-001
**Locations:** 1
- [`src/main.cpp:3`](../src/main.cpp#L3) — `Unknown`
### DP-002
**Locations:** 1
- [`src/main.cpp:3`](../src/main.cpp#L3) — `Unknown`
### GR-001
**Locations:** 1
- [`scripts/make_jellyfin_gallery.py:4`](../scripts/make_jellyfin_gallery.py#L4) — `Unknown`
### GR-002
**Locations:** 1
- [`scripts/make_jellyfin_gallery.py:4`](../scripts/make_jellyfin_gallery.py#L4) — `Unknown`
### GR-004
**Locations:** 44
- [`src/config.hpp:49`](../src/config.hpp#L49) — `Unknown`
- [`src/gallery/embedder_stamp.cpp:1`](../src/gallery/embedder_stamp.cpp#L1) — `Unknown`
- [`src/gallery/embedder_stamp.hpp:2`](../src/gallery/embedder_stamp.hpp#L2) — `Unknown`
- [`src/gallery/gallery_builder.cpp:45`](../src/gallery/gallery_builder.cpp#L45) — `ActorGallery build_gallery(const BuildConfig& cfg)`
- [`src/gallery/gallery_store.cpp:82`](../src/gallery/gallery_store.cpp#L82) — `H5::StrType str(H5::PredType::C_S1, H5T_VARIABLE);`
- [`src/gallery/gallery_store.cpp:167`](../src/gallery/gallery_store.cpp#L167) — `H5::DataSpace scalar(H5S_SCALAR);`
- [`src/gallery/gallery_store.cpp:219`](../src/gallery/gallery_store.cpp#L219) — `Unknown`
- [`src/kpn_bindings.cpp:167`](../src/kpn_bindings.cpp#L167) — `Unknown`
- [`src/kpn_bindings.cpp:217`](../src/kpn_bindings.cpp#L217) — `Unknown`
- [`src/main.cpp:174`](../src/main.cpp#L174) — `Unknown`
- [`src/nodes/embedding_dump_node.hpp:30`](../src/nodes/embedding_dump_node.hpp#L30) — `static constexpr std::string_view label() { return "embedding_dump"; }`
- [`src/nodes/embedding_dump_node.hpp:102`](../src/nodes/embedding_dump_node.hpp#L102) — `H5::StrType str(H5::PredType::C_S1, H5T_VARIABLE);`
- [`src/scene_preview.cpp:133`](../src/scene_preview.cpp#L133) — `int main(int argc, char** argv)`
- [`src/types.hpp:141`](../src/types.hpp#L141) — `struct Actor`
- [`tests/test_gallery_store.cpp:182`](../tests/test_gallery_store.cpp#L182) — `TempFile tf("gallery_stamped.h5");`
- [`tests/test_gallery_store.cpp:201`](../tests/test_gallery_store.cpp#L201) — `TempFile tf("gallery_stamped.h5");`
- [`tests/test_gallery_store.cpp:219`](../tests/test_gallery_store.cpp#L219) — `TempFile tf("gallery_unstamped.h5");`
- [`tests/test_gallery_store.cpp:240`](../tests/test_gallery_store.cpp#L240) — `TempFile tf("gallery_json_stamp.json");`
- [`tests/test_gallery_store.cpp:252`](../tests/test_gallery_store.cpp#L252) — `Unknown`
- [`tests/test_gallery_store.cpp:268`](../tests/test_gallery_store.cpp#L268) — `Unknown`
- [`tests/test_gallery_store.cpp:300`](../tests/test_gallery_store.cpp#L300) — `Unknown`
- [`tests/test_gallery_store.cpp:309`](../tests/test_gallery_store.cpp#L309) — `Unknown`
- [`tests/test_gallery_store.cpp:333`](../tests/test_gallery_store.cpp#L333) — `Unknown`
- [`tests/test_gallery_store.cpp:348`](../tests/test_gallery_store.cpp#L348) — `Unknown`
- [`tests/test_gallery_store.cpp:367`](../tests/test_gallery_store.cpp#L367) — `Unknown`
- [`tests/test_gallery_store.cpp:384`](../tests/test_gallery_store.cpp#L384) — `TempFile tf("fake_model.onnx");`
- [`scripts/filter_gallery.py:80`](../scripts/filter_gallery.py#L80) — `if actor_jellyfin_id(a) in cast_ids]`
- [`scripts/make_gallery.py:181`](../scripts/make_gallery.py#L181) — `Unknown`
- [`scripts/make_jellyfin_gallery.py:448`](../scripts/make_jellyfin_gallery.py#L448) — `Unknown`
- [`scripts/make_jellyfin_gallery.py:456`](../scripts/make_jellyfin_gallery.py#L456) — `Unknown`
- [`scripts/movienet_eval.py:65`](../scripts/movienet_eval.py#L65) — `with open(args.gt) as f:`
- [`scripts/optimizer/fetch_missing_actors.py:62`](../scripts/optimizer/fetch_missing_actors.py#L62) — `def fetch(missing_path, out_path, token, build_dir, models_dir, arcface,`
- [`scripts/optimizer/fetch_missing_actors.py:109`](../scripts/optimizer/fetch_missing_actors.py#L109) — `def merge(base_path, add_path, out_path):`
- [`scripts/optimizer/fetch_missing_actors.py:123`](../scripts/optimizer/fetch_missing_actors.py#L123) — `def merge(base_path, add_path, out_path):`
- [`scripts/optimizer/optimize.py:186`](../scripts/optimizer/optimize.py#L186) — `Unknown`
- [`scripts/optimizer/optimize.py:202`](../scripts/optimizer/optimize.py#L202) — `if not Path(f["dump"]).exists():`
- [`scripts/optimizer/reembed_gallery.py:62`](../scripts/optimizer/reembed_gallery.py#L62) — `for i, a in enumerate(ref["actors"], 1):`
- [`scripts/optimizer/replay.py:113`](../scripts/optimizer/replay.py#L113) — `Unknown`
- [`scripts/optimizer/replay.py:252`](../scripts/optimizer/replay.py#L252) — `Unknown`
- [`scripts/sae_embed_loader.py:22`](../scripts/sae_embed_loader.py#L22) — `return arcface if arcface else str(Path(models_dir) / DEFAULT_ARCFACE)`
- [`scripts/sae_gallery.py:171`](../scripts/sae_gallery.py#L171) — `if not _stamp_empty(embedder):`
- [`scripts/sae_gallery.py:199`](../scripts/sae_gallery.py#L199) — `for a in range(len(offset)):`
- [`scripts/sae_stamp.py:3`](../scripts/sae_stamp.py#L3) — `Unknown`
- [`scripts/stamp_gallery.py:4`](../scripts/stamp_gallery.py#L4) — `Unknown`
### IR-001
**Locations:** 1
- [`src/nodes/result_sink_node.hpp:2`](../src/nodes/result_sink_node.hpp#L2) — `Unknown`
### IR-002
**Locations:** 5
- [`src/config.hpp:20`](../src/config.hpp#L20) — `struct Config`
- [`src/main.cpp:216`](../src/main.cpp#L216) — `reg_cfg, EvidenceDiscounter(same_person));`
- [`src/nodes/result_sink_node.hpp:49`](../src/nodes/result_sink_node.hpp#L49) — `static constexpr std::string_view label() { return "result_sink"; }`
- [`src/nodes/result_sink_node.hpp:122`](../src/nodes/result_sink_node.hpp#L122) — `void write_output()`
- [`src/nodes/result_sink_node.hpp:161`](../src/nodes/result_sink_node.hpp#L161) — `struct ActorMeta { std::string name, imdb_id, tmdb_id, jellyfin_id; };`
### IR-003
**Locations:** 1
- [`src/main.cpp:216`](../src/main.cpp#L216) — `reg_cfg, EvidenceDiscounter(same_person));`
### IR-004
**Locations:** 16
- [`src/audio_signature.cpp:3`](../src/audio_signature.cpp#L3) — `Unknown`
- [`src/audio_signature.cpp:265`](../src/audio_signature.cpp#L265) — `std::vector<std::uint8_t> pack_frames(const std::vector<float>& mono)`
- [`src/audio_signature.cpp:320`](../src/audio_signature.cpp#L320) — `std::optional<std::string> signature_from_mono(const std::vector<float>& mono)`
- [`src/audio_signature.cpp:327`](../src/audio_signature.cpp#L327) — `std::optional<std::vector<float>> decode_centre_window(const std::string& path)`
- [`src/audio_signature.cpp:421`](../src/audio_signature.cpp#L421) — `std::optional<std::string> compute_signature(const std::string& path)`
- [`src/audio_signature.hpp:4`](../src/audio_signature.hpp#L4) — `Unknown`
- [`tests/test_audio_signature.cpp:3`](../tests/test_audio_signature.cpp#L3) — `Unknown`
- [`tests/test_audio_signature.cpp:129`](../tests/test_audio_signature.cpp#L129) — `Unknown`
- [`tests/test_audio_signature.cpp:151`](../tests/test_audio_signature.cpp#L151) — `Unknown`
- [`tests/test_audio_signature.cpp:167`](../tests/test_audio_signature.cpp#L167) — `Unknown`
- [`tests/test_audio_signature.cpp:236`](../tests/test_audio_signature.cpp#L236) — `Unknown`
- [`tests/test_audio_signature.cpp:263`](../tests/test_audio_signature.cpp#L263) — `TempWav w("centred300");`
- [`tests/test_audio_signature.cpp:301`](../tests/test_audio_signature.cpp#L301) — `Unknown`
- [`tests/test_audio_signature.cpp:314`](../tests/test_audio_signature.cpp#L314) — `kNumBands * std::log10(1000.0 / kBandLoHz) / std::log10(kBandHiHz / kBandLoHz)));`
- [`tests/test_audio_signature.cpp:328`](../tests/test_audio_signature.cpp#L328) — `std::vector<float> a(kWindowSamples / 50);`
- [`tests/test_audio_signature.cpp:344`](../tests/test_audio_signature.cpp#L344) — `return base64_encode(reinterpret_cast<const std::uint8_t*>(s.data()), s.size());`
### IR-005
**Locations:** 5
- [`src/audio_signature.cpp:421`](../src/audio_signature.cpp#L421) — `std::optional<std::string> compute_signature(const std::string& path)`
- [`src/audio_signature.hpp:4`](../src/audio_signature.hpp#L4) — `Unknown`
- [`tests/test_audio_signature.cpp:3`](../tests/test_audio_signature.cpp#L3) — `Unknown`
- [`tests/test_audio_signature.cpp:129`](../tests/test_audio_signature.cpp#L129) — `Unknown`
- [`tests/test_audio_signature.cpp:138`](../tests/test_audio_signature.cpp#L138) — `Unknown`
### IR-006
**Locations:** 1
- [`scripts/run_from_jellyfin.py:4`](../scripts/run_from_jellyfin.py#L4) — `Unknown`
### IR-007
**Locations:** 8
- [`src/audio_signature.cpp:3`](../src/audio_signature.cpp#L3) — `Unknown`
- [`src/audio_signature.cpp:327`](../src/audio_signature.cpp#L327) — `std::optional<std::vector<float>> decode_centre_window(const std::string& path)`
- [`src/audio_signature.cpp:421`](../src/audio_signature.cpp#L421) — `std::optional<std::string> compute_signature(const std::string& path)`
- [`src/audio_signature.hpp:4`](../src/audio_signature.hpp#L4) — `Unknown`
- [`tests/test_audio_signature.cpp:3`](../tests/test_audio_signature.cpp#L3) — `Unknown`
- [`tests/test_audio_signature.cpp:201`](../tests/test_audio_signature.cpp#L201) — `TempWav w("short30");`
- [`tests/test_audio_signature.cpp:219`](../tests/test_audio_signature.cpp#L219) — `TempWav w("exact120");`
- [`tests/test_audio_signature.cpp:229`](../tests/test_audio_signature.cpp#L229) — `TempWav w("exact120");`
### IR-008
**Locations:** 7
- [`src/audio_signature.cpp:3`](../src/audio_signature.cpp#L3) — `Unknown`
- [`src/audio_signature.cpp:320`](../src/audio_signature.cpp#L320) — `std::optional<std::string> signature_from_mono(const std::vector<float>& mono)`
- [`src/audio_signature.cpp:421`](../src/audio_signature.cpp#L421) — `std::optional<std::string> compute_signature(const std::string& path)`
- [`src/audio_signature.hpp:4`](../src/audio_signature.hpp#L4) — `Unknown`
- [`tests/test_audio_signature.cpp:3`](../tests/test_audio_signature.cpp#L3) — `Unknown`
- [`tests/test_audio_signature.cpp:129`](../tests/test_audio_signature.cpp#L129) — `Unknown`
- [`tests/test_audio_signature.cpp:167`](../tests/test_audio_signature.cpp#L167) — `Unknown`
### PR-002
**Locations:** 3
- [`src/nodes/embedding_dump_node.hpp:2`](../src/nodes/embedding_dump_node.hpp#L2) — `Unknown`
- [`scripts/optimizer/replay.py:5`](../scripts/optimizer/replay.py#L5) — `Reads an embedding dump (scripts/optimizer/SCHEMA.md), feeds each frame as an`
- [`scripts/optimizer/second_score.py:5`](../scripts/optimizer/second_score.py#L5) — `Unknown`
### PR-004
**Locations:** 1
- [`src/main.cpp:3`](../src/main.cpp#L3) — `Unknown`
### SR-001
**Locations:** 46
- [`src/config.hpp:49`](../src/config.hpp#L49) — `Unknown`
- [`src/gallery/embedder_stamp.cpp:1`](../src/gallery/embedder_stamp.cpp#L1) — `Unknown`
- [`src/gallery/embedder_stamp.hpp:2`](../src/gallery/embedder_stamp.hpp#L2) — `Unknown`
- [`src/gallery/gallery_builder.cpp:45`](../src/gallery/gallery_builder.cpp#L45) — `ActorGallery build_gallery(const BuildConfig& cfg)`
- [`src/gallery/gallery_store.cpp:82`](../src/gallery/gallery_store.cpp#L82) — `H5::StrType str(H5::PredType::C_S1, H5T_VARIABLE);`
- [`src/gallery/gallery_store.cpp:167`](../src/gallery/gallery_store.cpp#L167) — `H5::DataSpace scalar(H5S_SCALAR);`
- [`src/gallery/gallery_store.cpp:219`](../src/gallery/gallery_store.cpp#L219) — `Unknown`
- [`src/kpn_bindings.cpp:167`](../src/kpn_bindings.cpp#L167) — `Unknown`
- [`src/kpn_bindings.cpp:217`](../src/kpn_bindings.cpp#L217) — `Unknown`
- [`src/main.cpp:174`](../src/main.cpp#L174) — `Unknown`
- [`src/nodes/embedding_dump_node.hpp:30`](../src/nodes/embedding_dump_node.hpp#L30) — `static constexpr std::string_view label() { return "embedding_dump"; }`
- [`src/nodes/embedding_dump_node.hpp:102`](../src/nodes/embedding_dump_node.hpp#L102) — `H5::StrType str(H5::PredType::C_S1, H5T_VARIABLE);`
- [`src/scene_preview.cpp:133`](../src/scene_preview.cpp#L133) — `int main(int argc, char** argv)`
- [`src/types.hpp:141`](../src/types.hpp#L141) — `struct Actor`
- [`tests/test_gallery_store.cpp:182`](../tests/test_gallery_store.cpp#L182) — `TempFile tf("gallery_stamped.h5");`
- [`tests/test_gallery_store.cpp:201`](../tests/test_gallery_store.cpp#L201) — `TempFile tf("gallery_stamped.h5");`
- [`tests/test_gallery_store.cpp:219`](../tests/test_gallery_store.cpp#L219) — `TempFile tf("gallery_unstamped.h5");`
- [`tests/test_gallery_store.cpp:240`](../tests/test_gallery_store.cpp#L240) — `TempFile tf("gallery_json_stamp.json");`
- [`tests/test_gallery_store.cpp:252`](../tests/test_gallery_store.cpp#L252) — `Unknown`
- [`tests/test_gallery_store.cpp:268`](../tests/test_gallery_store.cpp#L268) — `Unknown`
- [`tests/test_gallery_store.cpp:300`](../tests/test_gallery_store.cpp#L300) — `Unknown`
- [`tests/test_gallery_store.cpp:309`](../tests/test_gallery_store.cpp#L309) — `Unknown`
- [`tests/test_gallery_store.cpp:333`](../tests/test_gallery_store.cpp#L333) — `Unknown`
- [`tests/test_gallery_store.cpp:348`](../tests/test_gallery_store.cpp#L348) — `Unknown`
- [`tests/test_gallery_store.cpp:367`](../tests/test_gallery_store.cpp#L367) — `Unknown`
- [`tests/test_gallery_store.cpp:384`](../tests/test_gallery_store.cpp#L384) — `TempFile tf("fake_model.onnx");`
- [`scripts/filter_gallery.py:80`](../scripts/filter_gallery.py#L80) — `if actor_jellyfin_id(a) in cast_ids]`
- [`scripts/make_gallery.py:181`](../scripts/make_gallery.py#L181) — `Unknown`
- [`scripts/make_jellyfin_gallery.py:4`](../scripts/make_jellyfin_gallery.py#L4) — `Unknown`
- [`scripts/make_jellyfin_gallery.py:448`](../scripts/make_jellyfin_gallery.py#L448) — `Unknown`
- [`scripts/make_jellyfin_gallery.py:456`](../scripts/make_jellyfin_gallery.py#L456) — `Unknown`
- [`scripts/movienet_eval.py:65`](../scripts/movienet_eval.py#L65) — `with open(args.gt) as f:`
- [`scripts/optimizer/fetch_missing_actors.py:62`](../scripts/optimizer/fetch_missing_actors.py#L62) — `def fetch(missing_path, out_path, token, build_dir, models_dir, arcface,`
- [`scripts/optimizer/fetch_missing_actors.py:109`](../scripts/optimizer/fetch_missing_actors.py#L109) — `def merge(base_path, add_path, out_path):`
- [`scripts/optimizer/fetch_missing_actors.py:123`](../scripts/optimizer/fetch_missing_actors.py#L123) — `def merge(base_path, add_path, out_path):`
- [`scripts/optimizer/optimize.py:186`](../scripts/optimizer/optimize.py#L186) — `Unknown`
- [`scripts/optimizer/optimize.py:202`](../scripts/optimizer/optimize.py#L202) — `if not Path(f["dump"]).exists():`
- [`scripts/optimizer/reembed_gallery.py:62`](../scripts/optimizer/reembed_gallery.py#L62) — `for i, a in enumerate(ref["actors"], 1):`
- [`scripts/optimizer/replay.py:113`](../scripts/optimizer/replay.py#L113) — `Unknown`
- [`scripts/optimizer/replay.py:252`](../scripts/optimizer/replay.py#L252) — `Unknown`
- [`scripts/run_from_jellyfin.py:4`](../scripts/run_from_jellyfin.py#L4) — `Unknown`
- [`scripts/sae_embed_loader.py:22`](../scripts/sae_embed_loader.py#L22) — `return arcface if arcface else str(Path(models_dir) / DEFAULT_ARCFACE)`
- [`scripts/sae_gallery.py:171`](../scripts/sae_gallery.py#L171) — `if not _stamp_empty(embedder):`
- [`scripts/sae_gallery.py:199`](../scripts/sae_gallery.py#L199) — `for a in range(len(offset)):`
- [`scripts/sae_stamp.py:3`](../scripts/sae_stamp.py#L3) — `Unknown`
- [`scripts/stamp_gallery.py:4`](../scripts/stamp_gallery.py#L4) — `Unknown`
### SR-002
**Locations:** 16
- [`src/config.hpp:103`](../src/config.hpp#L103) — `Unknown`
- [`src/evidence_discount.hpp:2`](../src/evidence_discount.hpp#L2) — `Unknown`
- [`src/face_utils.hpp:2`](../src/face_utils.hpp#L2) — `inline cv::Mat align_face(const cv::Mat& img,`
- [`src/gallery/gallery_calibration.hpp:2`](../src/gallery/gallery_calibration.hpp#L2) — `Unknown`
- [`src/gallery/gallery_calibration.hpp:53`](../src/gallery/gallery_calibration.hpp#L53) — `float boundary_at(float p = 0.5f, float log_prior_odds = 0.f) const`
- [`src/main.cpp:199`](../src/main.cpp#L199) — `reg_cfg, EvidenceDiscounter(same_person));`
- [`src/main.cpp:216`](../src/main.cpp#L216) — `reg_cfg, EvidenceDiscounter(same_person));`
- [`src/nodes/face_detector_node.hpp:2`](../src/nodes/face_detector_node.hpp#L2) — `Unknown`
- [`src/nodes/face_tracker_node.hpp:2`](../src/nodes/face_tracker_node.hpp#L2) — `Unknown`
- [`src/nodes/identity_matcher_node.hpp:111`](../src/nodes/identity_matcher_node.hpp#L111) — `const GalleryCalibration& calibration() const { return cal_; }`
- [`src/nodes/identity_matcher_node.hpp:119`](../src/nodes/identity_matcher_node.hpp#L119) — `const GalleryCalibration& calibration() const { return cal_; }`
- [`src/nodes/identity_matcher_node.hpp:242`](../src/nodes/identity_matcher_node.hpp#L242) — `Unknown`
- [`src/nodes/result_sink_node.hpp:49`](../src/nodes/result_sink_node.hpp#L49) — `static constexpr std::string_view label() { return "result_sink"; }`
- [`src/nodes/result_sink_node.hpp:63`](../src/nodes/result_sink_node.hpp#L63) — `void set_pre_write_hook(std::function<void(double)> fn) { pre_write_ = std::move(fn); }`
- [`src/nodes/result_sink_node.hpp:161`](../src/nodes/result_sink_node.hpp#L161) — `struct ActorMeta { std::string name, imdb_id, tmdb_id, jellyfin_id; };`
- [`src/track_registry.hpp:2`](../src/track_registry.hpp#L2) — `Unknown`
### SR-003
**Locations:** 6
- [`src/audio_signature.cpp:3`](../src/audio_signature.cpp#L3) — `Unknown`
- [`src/audio_signature.hpp:4`](../src/audio_signature.hpp#L4) — `Unknown`
- [`src/config.hpp:20`](../src/config.hpp#L20) — `struct Config`
- [`src/nodes/result_sink_node.hpp:2`](../src/nodes/result_sink_node.hpp#L2) — `Unknown`
- [`src/nodes/result_sink_node.hpp:49`](../src/nodes/result_sink_node.hpp#L49) — `static constexpr std::string_view label() { return "result_sink"; }`
- [`src/nodes/result_sink_node.hpp:122`](../src/nodes/result_sink_node.hpp#L122) — `void write_output()`
### SR-005
**Locations:** 1
- [`scripts/make_jellyfin_gallery.py:4`](../scripts/make_jellyfin_gallery.py#L4) — `Unknown`
### UT-001
**Locations:** 1
- [`tests/test_track_registry.cpp:3`](../tests/test_track_registry.cpp#L3) — `Unknown`
### UT-101
**Locations:** 5
- [`tests/test_audio_signature.cpp:3`](../tests/test_audio_signature.cpp#L3) — `Unknown`
- [`tests/test_audio_signature.cpp:129`](../tests/test_audio_signature.cpp#L129) — `Unknown`
- [`tests/test_audio_signature.cpp:138`](../tests/test_audio_signature.cpp#L138) — `Unknown`
- [`tests/test_audio_signature.cpp:151`](../tests/test_audio_signature.cpp#L151) — `Unknown`
- [`tests/test_audio_signature.cpp:167`](../tests/test_audio_signature.cpp#L167) — `Unknown`
### UT-102
**Locations:** 5
- [`tests/test_audio_signature.cpp:3`](../tests/test_audio_signature.cpp#L3) — `Unknown`
- [`tests/test_audio_signature.cpp:201`](../tests/test_audio_signature.cpp#L201) — `TempWav w("short30");`
- [`tests/test_audio_signature.cpp:219`](../tests/test_audio_signature.cpp#L219) — `TempWav w("exact120");`
- [`tests/test_audio_signature.cpp:229`](../tests/test_audio_signature.cpp#L229) — `TempWav w("exact120");`
- [`tests/test_audio_signature.cpp:236`](../tests/test_audio_signature.cpp#L236) — `Unknown`
### UT-103
**Locations:** 2
- [`tests/test_audio_signature.cpp:3`](../tests/test_audio_signature.cpp#L3) — `Unknown`
- [`tests/test_audio_signature.cpp:263`](../tests/test_audio_signature.cpp#L263) — `TempWav w("centred300");`
### UT-104
**Locations:** 5
- [`tests/test_audio_signature.cpp:3`](../tests/test_audio_signature.cpp#L3) — `Unknown`
- [`tests/test_audio_signature.cpp:301`](../tests/test_audio_signature.cpp#L301) — `Unknown`
- [`tests/test_audio_signature.cpp:314`](../tests/test_audio_signature.cpp#L314) — `kNumBands * std::log10(1000.0 / kBandLoHz) / std::log10(kBandHiHz / kBandLoHz)));`
- [`tests/test_audio_signature.cpp:328`](../tests/test_audio_signature.cpp#L328) — `std::vector<float> a(kWindowSamples / 50);`
- [`tests/test_audio_signature.cpp:344`](../tests/test_audio_signature.cpp#L344) — `return base64_encode(reinterpret_cast<const std::uint8_t*>(s.data()), s.size());`
### VR-001
**Locations:** 1
- [`src/nodes/embedding_dump_node.hpp:2`](../src/nodes/embedding_dump_node.hpp#L2) — `Unknown`
### VR-002
**Locations:** 1
- [`scripts/optimizer/replay.py:5`](../scripts/optimizer/replay.py#L5) — `Reads an embedding dump (scripts/optimizer/SCHEMA.md), feeds each frame as an`
### VR-003
**Locations:** 1
- [`scripts/optimizer/second_score.py:5`](../scripts/optimizer/second_score.py#L5) — `Unknown`
## Tag diagnostics
**Malformed tags:**
- `scripts/filter_gallery.py:80` — {'ignored': ['— a filtered gallery holds the SAME vectors as its']}
- `scripts/make_gallery.py:181` — {'ignored': ['— stamp with the model actually loaded', 'resolved']}
- `scripts/make_jellyfin_gallery.py:456` — {'ignored': ["— --merge keeps the existing actors' vectors and"]}
- `scripts/movienet_eval.py:65` — {'ignored': ['— match() below is a bare dot product against the']}
- `scripts/optimizer/fetch_missing_actors.py:109` — {'ignored': ['— the legacy JSON gallery carries the same stamp as']}
- `scripts/optimizer/fetch_missing_actors.py:123` — {'ignored': ['— merging two galleries from different models makes']}
- `scripts/optimizer/optimize.py:202` — {'ignored': ['— every (dump', 'gallery) pair is checked ONCE here']}
- `scripts/optimizer/reembed_gallery.py:62` — {'ignored': ['— this script exists to produce a gallery in a']}
- `scripts/optimizer/replay.py:113` — {'ignored': ['— checked here', 'before any network is built', 'so a']}
- `scripts/optimizer/replay.py:252` — {'ignored': ['— promote an unprovable gallery/dump binding from a']}
- `scripts/sae_embed_loader.py:22` — {'ignored': ['— single source of truth for "which model is this"']}
- `scripts/sae_gallery.py:171` — {'ignored': ['— omitted entirely when unknown', 'so "unstamped"']}
- `scripts/sae_gallery.py:199` — {'ignored': ['— carried through so a derived gallery (filter']}
**Groups mixing requirement types (pipe separates types):**
- `src/main.cpp:216` — {'group': ['AR-012', 'AR-016', 'IR-002', 'IR-003']}
- `src/nodes/result_sink_node.hpp:49` — {'group': ['AR-012', 'AR-017', 'IR-002']}
- `src/nodes/result_sink_node.hpp:161` — {'group': ['AR-012', 'IR-002']}
_No TRACES tags found yet. Tags are added as code is written; an empty matrix on a new tree is the correct reading, not a failure._
external/KPN
Vendored
+1 -1
Submodule external/KPN updated: be6e92268c...4b6e498ba7
scripts/filter_gallery.py
+1 -5
View File
@@ -77,11 +77,7 @@ def main():
if missing > 0:
print(f"[warn] {missing} cast member(s) not present in gallery (not yet embedded)", file=sys.stderr)
# TRACES: GR-004 | SR-001 — a filtered gallery holds the SAME vectors as its
# source, so it inherits the source's binding. Dropping the stamp here would
# silently launder a stamped gallery into an unstamped one.
save_gallery_hdf5({"actors": actors}, Path(args.output),
gallery.get("embedder"))
save_gallery_hdf5({"actors": actors}, Path(args.output))
print(f"Saved {len(actors)} actor(s) to {args.output}", file=sys.stderr)
scripts/make_fixtures.sh
-60
View File
@@ -1,60 +0,0 @@
#!/usr/bin/env bash
# make_fixtures.sh — regenerate the committed replay fixtures.
#
# TRACES: VR-001 | PR-002
#
# CI never calls a model (see docs/requirements.md, "CI never calls a model"):
# the embedder is impractical on the N100 CI host, so inference happens HERE, on
# a machine with a GPU, and CI consumes the HDF5 dumps as data. Everything
# downstream of embedding — tracking, presence windows, belief accumulation,
# expansion — is cheap CPU maths and replays from these files.
#
# Reproducibility is a requirement, not a nicety. A fixture whose provenance is
# unknown is worse than no fixture, because it will be trusted. Every parameter
# that affects the output is pinned below rather than left to a default, and the
# dumps carry the embedder identity and SHA-256 (GR-004) so a replay cannot be
# silently scored against the wrong gallery.
#
# These are byte-reproducible only because node outputs block rather than drop
# on a full channel (AR-004). Before that fix the same command produced
# different dumps run to run, since what got dropped depended on timing.
#
# Source: bali/ — Road to Bali (1952), public domain. That matters: derived
# fixtures can be committed, where anything cut from a copyrighted title could
# not live in the repository at all.
set -euo pipefail
REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
CLIPS="${CLIPS:-$REPO/../bali}"
GALLERY="${GALLERY:-$REPO/gallery_lvface.h5}"
BIN="${BIN:-$REPO/build/scene_analyze}"
OUT="$REPO/tests/fixtures/dumps"
# Pinned. Changing either invalidates every committed fixture.
# fps 5 — 1 fps over a 77 s clip is 77 frames, too thin to exercise an
# extinction window measured in tens of seconds.
# min-face — 32 px, the VR-005 measured floor (98.1% TPI). The corpus is
# 480x360, so a stricter value would reject most faces present.
FPS=5
MIN_FACE_PX=32
[[ -x "$BIN" ]] || { echo "no scene_analyze at $BIN (set BIN=)" >&2; exit 1; }
[[ -f "$GALLERY" ]] || { echo "no gallery at $GALLERY (set GALLERY=)" >&2; exit 1; }
[[ -d "$CLIPS" ]] || { echo "no clips at $CLIPS (set CLIPS=)" >&2; exit 1; }
mkdir -p "$OUT"
for clip in "$CLIPS"/Road_To_Bali-*.webm; do
n="$(basename "$clip" .webm)"; n="${n##*-}"
echo "── bali_$n"
"$BIN" --movie "$clip" --gallery "$GALLERY" \
--fps "$FPS" --min-face-px "$MIN_FACE_PX" \
--dump-embeddings "$OUT/bali_$n.h5" \
--output /dev/null 2>&1 | grep -E "wrote|dropped" || true
done
echo
echo "Regenerated in $OUT — verify the diff is empty if nothing upstream changed."
echo "A non-empty diff means detection, alignment or embedding moved. That is"
echo "either a regression or a deliberate change, and either way the golden"
echo "outputs derived from these fixtures need reviewing."
scripts/make_gallery.py
+3 -8
View File
@@ -36,9 +36,8 @@ import time
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent))
from sae_embed_loader import load_embedder, resolve_arcface
from sae_gallery import (download_images, embedder_stamp, save_gallery,
wikidata_image_urls)
from sae_embed_loader import load_embedder
from sae_gallery import download_images, save_gallery, wikidata_image_urls
from sae_tmdb import TMDB_IMG, tmdb_get, tmdb_id_from_imdb
@@ -178,11 +177,7 @@ def main():
output = Path(args.output)
image_root = Path(args.image_dir) if args.image_dir else output.parent / "images"
# TRACES: GR-004 | SR-001 — stamp with the model actually loaded, resolved
# through the same helper load_embedder uses so the two cannot diverge.
arcface_path = resolve_arcface(args.models_dir, args.arcface)
embedder = load_embedder(args.build_dir, args.models_dir, args.arcface)
stamp = embedder_stamp(arcface_path)
# Resolve movie ID
movie_id = args.movie_id
@@ -208,7 +203,7 @@ def main():
if n_actors == 0:
sys.exit("No actors could be processed — check models and images.")
save_gallery(gallery, missing, output, embedder=stamp)
save_gallery(gallery, missing, output)
if __name__ == "__main__":
scripts/make_jellyfin_gallery.py
+3 -15
View File
@@ -1,8 +1,6 @@
#!/usr/bin/env python3
"""make_jellyfin_gallery.py — build a gallery.h5 spanning an entire Jellyfin library.
TRACES: GR-001, GR-002 | SR-001, SR-005
Queries the Jellyfin API for every Movie/Series, collects the unique cast
across the whole library, downloads each actor's headshot directly from
Jellyfin (no TMDB key needed), embeds them with the sae_embed module (SCRFD +
@@ -53,9 +51,8 @@ import requests
sys.path.insert(0, str(Path(__file__).resolve().parent))
import sae_env # noqa: F401 — loads .env into os.environ on import
from sae_embed_loader import load_embedder, resolve_arcface
from sae_gallery import (download_image, download_images, embedder_stamp,
enforce_embedder_stamp, load_gallery_hdf5,
from sae_embed_loader import load_embedder
from sae_gallery import (download_image, download_images, load_gallery_hdf5,
save_gallery, wikidata_image_urls)
from sae_jellyfin import actor_jellyfin_id, jf_get, normalize_jellyfin_url
from sae_tmdb import (
@@ -445,20 +442,11 @@ def main():
image_root = Path(args.image_dir) if args.image_dir else output.parent / "images"
item_types = [t.strip() for t in args.item_types.split(",") if t.strip()]
# TRACES: GR-004 | SR-001
arcface_path = resolve_arcface(args.models_dir, args.arcface)
embedder = load_embedder(args.build_dir, args.models_dir, args.arcface)
stamp = embedder_stamp(arcface_path)
existing_actors = {}
if args.merge and output.is_file():
existing = load_gallery_hdf5(output)
# TRACES: GR-004 | SR-001 — --merge keeps the existing actors' vectors and
# embeds the new ones with THIS model. If they disagree, the result is one
# gallery holding two incompatible embedding spaces, which is worse than a
# mismatched gallery: no later check can separate them again.
enforce_embedder_stamp(existing.get("embedder"), stamp, str(output),
arcface_path)
for actor in existing.get("actors", []):
pid = actor_jellyfin_id(actor)
if pid:
@@ -487,7 +475,7 @@ def main():
if n_actors == 0:
sys.exit("No actors could be processed — check Jellyfin URL/API key and models.")
save_gallery(gallery, missing, output, embedder=stamp)
save_gallery(gallery, missing, output)
if __name__ == "__main__":
scripts/movienet_eval.py
+2 -7
View File
@@ -22,8 +22,8 @@ from pathlib import Path
import numpy as np
sys.path.insert(0, str(Path(__file__).resolve().parent))
from sae_embed_loader import load_embedder, resolve_arcface
from sae_gallery import load_gallery_hdf5, verify_gallery_stamp
from sae_embed_loader import load_embedder
from sae_gallery import load_gallery_hdf5
def load_gallery(path: str) -> dict[str, dict]:
@@ -62,11 +62,6 @@ def main():
args = p.parse_args()
embedder = load_embedder(args.build_dir, args.models_dir, args.arcface)
# TRACES: GR-004 | SR-001 — match() below is a bare dot product against the
# gallery's vectors; if the gallery came from another model those numbers are
# noise wearing a similarity's clothes.
verify_gallery_stamp(args.gallery,
resolve_arcface(args.models_dir, args.arcface))
gallery = load_gallery(args.gallery)
print(f"[eval] gallery: {len(gallery)} actors", file=sys.stderr)
scripts/optimizer/SCHEMA.md
-16
View File
@@ -22,8 +22,6 @@ variable-length HDF5 types and reads straight into numpy.
movie : str (source video path)
sample_fps : float
embed_dim : int = 512
embedder_model : str basename of the embedding model (GR-004)
embedder_sha256: str SHA-256 of that model file (GR-004)
frames/ group — one row per sampled frame
timestamp_sec : float64 [F]
@@ -43,20 +41,6 @@ variable-length HDF5 types and reads straight into numpy.
`F` = number of sampled frames, `N` = total faces (= sum of face_count).
Frame *i*'s faces are `faces/*[ face_offset[i] : face_offset[i]+face_count[i] ]`.
## Model binding (GR-004)
`embedder_model` / `embedder_sha256` record which embedder produced every vector
in `faces/embedding`. A replay has no live embedder, so **the dump is the embedder
as far as the gallery is concerned**: `replay.py` checks these two attributes
against the gallery's own `/embedder` stamp and refuses to run on a mismatch,
naming both sides. Cross-model cosines are meaningless but look plausible.
The attributes are additive, not a format break — `schema_version` stays 1. Dumps
written before GR-004 simply lack them, which reports as *unverifiable* (a loud
warning, or a hard error under `SAE_REQUIRE_GALLERY_STAMP=1`) rather than as a
pass. Re-dump to bind an old dump; there is no in-place migration, because unlike
a gallery nobody can assert after the fact which model produced a vector.
## Invariants
- `embedding` rows are unit-norm (cosine == dot product against the gallery).
- `face_offset[0] == 0`; `face_offset[i+1] == face_offset[i] + face_count[i]`.
scripts/optimizer/fetch_missing_actors.py
+2 -13
View File
@@ -35,9 +35,7 @@ REPO = Path(__file__).resolve().parent.parent.parent
sys.path.insert(0, str(REPO / "scripts"))
import sae_env # noqa: E402 loads .env
from sae_tmdb import tmdb_get, tmdb_person_for_imdb, TMDB_IMG # noqa: E402
from sae_embed_loader import resolve_arcface # noqa: E402
from sae_gallery import (download_images, embedder_stamp, # noqa: E402
enforce_embedder_stamp, wikidata_image_urls)
from sae_gallery import download_images, wikidata_image_urls # noqa: E402
from sae_embed_loader import load_embedder # noqa: E402
@@ -59,7 +57,6 @@ def fetch(missing_path, out_path, token, build_dir, models_dir, arcface,
src = "TMDB + Wikidata fallback" if use_wikidata else "TMDB"
print(f"[fetch] {len(missing)} missing actors to resolve via {src}", file=sys.stderr)
embedder = load_embedder(build_dir, models_dir, arcface)
stamp = embedder_stamp(resolve_arcface(models_dir, arcface)) # TRACES: GR-004 | SR-001
img_root = Path(tempfile.mkdtemp(prefix="missing_gallery_"))
actors = []
@@ -106,9 +103,7 @@ def fetch(missing_path, out_path, token, build_dir, models_dir, arcface,
f"(wiki={n_via_wikidata}) no_tmdb={n_no_tmdb} no_img={n_no_img} "
f"no_face={n_no_face}", file=sys.stderr)
# TRACES: GR-004 | SR-001 — the legacy JSON gallery carries the same stamp as
# the HDF5 one; src/gallery/gallery_store.cpp reads it from either.
Path(out_path).write_text(json.dumps({"embedder": stamp, "actors": actors}, indent=2))
Path(out_path).write_text(json.dumps({"actors": actors}, indent=2))
n_emb = sum(len(a["embeddings"]) for a in actors)
print(f"\n[fetch] recovered {n_resolved}/{len(missing)} actors "
f"({n_via_wikidata} via Wikidata), {n_emb} embeddings → {out_path}",
@@ -120,12 +115,6 @@ def fetch(missing_path, out_path, token, build_dir, models_dir, arcface,
def merge(base_path, add_path, out_path):
base = json.loads(Path(base_path).read_text())
add = json.loads(Path(add_path).read_text())
# TRACES: GR-004 | SR-001 — merging two galleries from different models makes
# ONE file containing two incompatible embedding spaces. Nothing downstream can
# ever untangle that, so this is the one place the check must run before, not
# after, the write.
enforce_embedder_stamp(base.get("embedder"), add.get("embedder"),
str(base_path), str(add_path))
have = {a.get("imdb_id") for a in base["actors"] if a.get("imdb_id")}
added = [a for a in add["actors"] if a.get("imdb_id") not in have]
base["actors"].extend(added)
scripts/optimizer/optimize.py
-23
View File
@@ -34,7 +34,6 @@ from scipy.optimize import differential_evolution
REPO = Path(__file__).resolve().parent.parent.parent
sys.path.insert(0, str(REPO / "scripts" / "optimizer"))
sys.path.insert(0, str(REPO / "scripts" / "validation"))
sys.path.insert(0, str(REPO / "scripts"))
import json as _json
import os
@@ -57,8 +56,6 @@ DE_WORKERS = int(os.environ.get("DE_WORKERS", "1"))
from second_score import score_seconds # noqa: E402 uniform per-second TPI/FPI scoring
from sample_eval import load_gallery_keys # noqa: E402
from replay import dump_embedder_stamp # noqa: E402
from sae_stamp import EmbedderMismatch, verify_gallery_stamp # noqa: E402
_GAL_KEYS: dict = {} # gallery path → key set (fair-recall FN mask), loaded once
_REPLAY_TIMEOUT = 45 # seconds per film; a wedged replay is killed, not left to hang
@@ -183,15 +180,7 @@ def main():
p.add_argument("--seed", type=int, default=0)
p.add_argument("--trajectory", help="write every evaluation here (JSON lines)")
p.add_argument("--out", help="write best config + metrics")
# TRACES: GR-004 | SR-001
p.add_argument("--require-gallery-stamp", action="store_true",
help="unprovable gallery/dump model binding is a hard error, "
"not a warning (also via SAE_REQUIRE_GALLERY_STAMP=1)")
args = p.parse_args()
if args.require_gallery_stamp:
# Set the env var rather than threading a flag through cfg: replays run as
# subprocesses and inherit it, so strictness cannot be lost in the handoff.
os.environ["SAE_REQUIRE_GALLERY_STAMP"] = "1"
films = json.loads(Path(args.manifest).read_text())
for f in films:
@@ -199,18 +188,6 @@ def main():
if not Path(f["dump"]).exists():
sys.exit(f"[opt] missing dump for {f['name']}: {f['dump']}")
# TRACES: GR-004 | SR-001 — every (dump, gallery) pair is checked ONCE here,
# before the first evaluation. A DE sweep is thousands of replays; discovering
# a cross-model pair at the end (or never) means every number it produced was
# noise. Each replay subprocess re-checks its own pair anyway.
for f in films:
try:
verify_gallery_stamp(f["gallery"], stamp=dump_embedder_stamp(f["dump"]),
embedder_desc=f"embedding dump {Path(f['dump']).name}",
require_stamp=args.require_gallery_stamp)
except EmbedderMismatch as e:
sys.exit(f"[opt] {f['name']}: {e}")
names, bounds = [], []
int_knobs = {"track_max_frames_missing", "cut_inactive_max_frames"}
for spec in args.params:
scripts/optimizer/reembed_gallery.py
+3 -9
View File
@@ -27,9 +27,8 @@ from pathlib import Path
REPO = Path(__file__).resolve().parent.parent.parent
sys.path.insert(0, str(REPO / "scripts"))
from sae_embed_loader import load_embedder, resolve_arcface # noqa: E402
from sae_gallery import (embedder_stamp, load_gallery_hdf5, # noqa: E402
save_gallery_hdf5)
from sae_embed_loader import load_embedder # noqa: E402
from sae_gallery import load_gallery_hdf5, save_gallery_hdf5 # noqa: E402
def find_dir(images_root: Path, jellyfin_id: str, name: str) -> Path | None:
@@ -59,11 +58,6 @@ def main():
ref = load_gallery_hdf5(Path(args.ref))
images_root = Path(args.images)
embedder = load_embedder(args.build_dir, args.models_dir, args.arcface)
# TRACES: GR-004 | SR-001 — this script exists to produce a gallery in a
# DIFFERENT model's space from the reference. The output must therefore never
# inherit the reference's stamp; it carries the stamp of --arcface, which is
# the whole point of the bake-off being safe to run.
stamp = embedder_stamp(resolve_arcface(args.models_dir, args.arcface))
out_actors = []
n_ok = n_nodir = n_noemb = 0
@@ -90,7 +84,7 @@ def main():
print(f" [{i}/{total}] ok={n_ok} no_dir={n_nodir} no_emb={n_noemb}",
file=sys.stderr)
save_gallery_hdf5({"actors": out_actors}, Path(args.out), stamp)
save_gallery_hdf5({"actors": out_actors}, Path(args.out))
n_emb = sum(len(a["embeddings"]) for a in out_actors)
print(f"[reembed] {Path(args.arcface).stem}: {n_ok}/{total} actors, {n_emb} embeddings "
f"→ {args.out}", file=sys.stderr)
scripts/optimizer/replay.py
+1 -35
View File
@@ -2,8 +2,6 @@
"""
replay.py — replay a dumped embedding HDF5 through the real KPN downstream nodes.
TRACES: VR-002 | PR-002
Reads an embedding dump (scripts/optimizer/SCHEMA.md), feeds each frame as an
EmbeddedSceneFrame into a Python-assembled KPN network wiring the *real* C++
face_tracker → identity_matcher → scene_tracker, and returns the same presence-window
@@ -27,22 +25,6 @@ import h5py
import numpy as np
REPO = Path(__file__).resolve().parent.parent.parent
sys.path.insert(0, str(REPO / "scripts"))
from sae_stamp import verify_gallery_stamp # noqa: E402
def dump_embedder_stamp(dump_path: str) -> dict:
"""The GR-004 embedder stamp recorded in an embedding dump.
A replay has no live embedder — the dump IS the embedder as far as the gallery
is concerned, so the dump's stamp is what the gallery must be checked against.
Dumps written before GR-004 have no attributes and yield an empty stamp, which
the check reports as unverifiable rather than silently accepting."""
with h5py.File(dump_path, "r") as f:
name = f.attrs.get("embedder_model", "")
sha = f.attrs.get("embedder_sha256", "")
dec = lambda v: v.decode() if isinstance(v, bytes) else ("" if v is None else str(v))
return {"model_name": dec(name), "model_sha256": dec(sha), "embed_dim": 512}
def load_frames(dump_path: str, min_conf: float = 0.0):
@@ -110,15 +92,6 @@ def replay(dump_path: str, gallery: str, cfg: dict, build_dir: str, stop: bool =
sys.path.insert(0, build_dir)
import sae_kpn
# TRACES: GR-004 | SR-001 — checked here, before any network is built, so a
# cross-model replay dies with one readable error instead of producing a
# plausible-looking score. add_identity_matcher re-checks it C++-side below;
# that is the backstop for any other caller of the binding.
stamp = dump_embedder_stamp(dump_path)
verify_gallery_stamp(gallery, stamp=stamp,
embedder_desc=f"embedding dump {Path(dump_path).name}",
require_stamp=bool(cfg.get("require_gallery_stamp", False)))
frames, movie, fps = load_frames(dump_path, min_conf=float(cfg.get("detector_conf", 0.0)))
net = sae_kpn.Network()
@@ -149,8 +122,7 @@ def replay(dump_path: str, gallery: str, cfg: dict, build_dir: str, stop: bool =
cap = len(frames) * 2 + 64
sae_kpn.add_node_python(net, "replay", source, [], ["EmbeddedSceneFrame"], cap)
sae_kpn.add_face_tracker(net, "tracker", cfg, cap)
sae_kpn.add_identity_matcher(net, "matcher", gallery, cfg, cap,
stamp["model_name"], stamp["model_sha256"])
sae_kpn.add_identity_matcher(net, "matcher", gallery, cfg, cap)
sae_kpn.add_scene_tracker(net, "scene", cfg, cap)
net.connect("replay", 0, "tracker", 0)
net.connect("tracker", 0, "matcher", 0)
@@ -249,17 +221,11 @@ def main():
# per-film gallery expansion: promotes pose-varied views of confidently-identified
# actors into an in-memory annex, recovering ~+4 recall at no precision cost.
p.add_argument("--expand-gallery", action="store_true")
# TRACES: GR-004 | SR-001 — promote an unprovable gallery/dump binding from a
# loud warning to a hard error. Measurement sweeps should set this (or
# SAE_REQUIRE_GALLERY_STAMP=1) so no number comes from an unbound pair.
p.add_argument("--require-gallery-stamp", action="store_true")
args = p.parse_args()
cfg = {k: getattr(args, k) for k in CFG_KEYS if getattr(args, k) is not None}
if args.expand_gallery:
cfg["expand_gallery"] = True
if args.require_gallery_stamp:
cfg["require_gallery_stamp"] = True
# stop=True: PyNode::stop() sets stop_flag_ before joining, so the source
# thread's run_loop actually exits. stop=False skips that, leaving stop_flag_
# false forever — the PyNode destructor's jthread.join() then blocks forever
scripts/optimizer/second_score.py
-2
View File
@@ -2,8 +2,6 @@
"""
second_score.py — uniform per-second agreement with X-Ray.
TRACES: VR-003 | PR-002
Unlike scene_score.py (which unions our detections over a whole X-Ray scene), this
samples EVERY SECOND of the film and asks: at second t, do we name the same actors
X-Ray says are on screen?
scripts/run_from_jellyfin.py
-2
View File
@@ -1,8 +1,6 @@
#!/usr/bin/env python3
"""run_from_jellyfin.py — resolve a Jellyfin title to its media file and run scene_analyze.
TRACES: IR-006 | SR-001
Looks up a Movie/Episode in Jellyfin, reads its on-disk Path (Jellyfin and this
tool must share the same media mount), filters the gallery down to that
title's credited cast (via filter_gallery's logic, fewer look-alike
scripts/sae_embed_loader.py
+1 -16
View File
@@ -3,26 +3,11 @@
sae_embed.FaceEmbedder loads both ONNX sessions once and exposes an
embed(path) -> FaceResult method, avoiding the per-process model reload cost
of spawning the embed_faces CLI binary for every image.
resolve_arcface() exposes the same default-resolution logic load_embedder uses,
so a caller can stamp the gallery it is about to write with the model that
actually produced its embeddings (GR-004) — the resolved path, not the CLI
argument, which is often None.
"""
import sys
from pathlib import Path
DEFAULT_ARCFACE = "arcface_w600k_r50.onnx"
def resolve_arcface(models_dir: str, arcface: str | None = None) -> str:
"""The ArcFace/LVFace ONNX path load_embedder would use for these arguments.
TRACES: GR-004 | SR-001 — single source of truth for "which model is this",
so the stamp written into a gallery can never drift from the model loaded."""
return arcface if arcface else str(Path(models_dir) / DEFAULT_ARCFACE)
def load_embedder(build_dir: str, models_dir: str, arcface: str | None = None,
conf: float = 0.5, nms: float = 0.4, max_side: int = 500):
@@ -43,7 +28,7 @@ def load_embedder(build_dir: str, models_dir: str, arcface: str | None = None,
models_path = Path(models_dir)
detector_path = str(models_path / "scrfd_500m_bnkps.onnx")
arcface_path = resolve_arcface(models_dir, arcface)
arcface_path = arcface if arcface else str(models_path / "arcface_w600k_r50.onnx")
for model, name in [(detector_path, "SCRFD"), (arcface_path, "ArcFace")]:
if not Path(model).is_file():
sys.exit(f"{name} model not found: {model}\nRun: bash scripts/download_models.sh")
scripts/sae_gallery.py
+10 -60
View File
@@ -6,13 +6,9 @@ make_jellyfin_gallery.download_urls + download_person_images) and the duplicated
Galleries are written directly as HDF5 — never JSON. Same layout the C++ side
reads/writes (src/gallery/gallery_store.cpp): flat [N,512] embeddings + per-actor
offset/count, parallel imdb_id/tmdb_id/jellyfin_id/name string arrays, a
per-embedding-row source_images array, and an /embedder group carrying the
GR-004 model binding. calibration is left absent (calib_hash=0); the C++
identity_matcher fits and writes it back into the file on first use.
The GR-004 embedder stamp written into that /embedder group lives in sae_stamp
and is re-exported below, so existing callers keep importing it from here.
offset/count, parallel imdb_id/tmdb_id/jellyfin_id/name string arrays, and a
per-embedding-row source_images array. calibration is left absent (calib_hash=0);
the C++ identity_matcher fits and writes it back into the file on first use.
"""
import io
@@ -105,36 +101,11 @@ def download_images(urls: list[str], dest_dir: Path, n: int,
return paths
# ── GR-004: gallery ↔ embedder binding ───────────────────────────────────────
# Implemented in sae_stamp (kept dependency-light so the optimizer's replay
# subprocesses can import it without pulling requests/Pillow); re-exported here
# because the gallery writers and every existing caller reach for it via this
# module. See src/gallery/embedder_stamp.hpp for the C++ twin and the rationale.
from sae_stamp import ( # noqa: F401
EmbedderMismatch,
check_embedder_stamp,
describe_stamp,
embedder_stamp,
enforce_embedder_stamp,
read_gallery_stamp,
require_gallery_stamp_from_env,
sha256_file,
verify_gallery_stamp,
_as_str,
_stamp_empty,
)
def save_gallery_hdf5(gallery: dict, output: Path, embedder: dict | None = None) -> None:
def save_gallery_hdf5(gallery: dict, output: Path) -> None:
"""Write a gallery dict ({"actors": [...]}) directly as HDF5 — same schema
src/gallery/gallery_store.cpp reads/writes. No calibration group; the
C++ identity_matcher computes and writes it back into this file on first
use against an unseen set of embeddings.
`embedder` is the GR-004 stamp (see embedder_stamp()); it may also be carried
on the gallery dict under "embedder", which is how a filtered/derived gallery
keeps its binding without the caller having to re-hash anything."""
embedder = embedder if embedder is not None else gallery.get("embedder")
use against an unseen set of embeddings."""
actors = gallery["actors"]
embs, offsets, counts = [], [], []
imdb, tmdb, jf, name, src_images = [], [], [], [], []
@@ -168,17 +139,8 @@ def save_gallery_hdf5(gallery: dict, output: Path, embedder: dict | None = None)
f.create_dataset("jellyfin_id", data=np.asarray(jf, dtype=object), dtype=str_t)
f.create_dataset("name", data=np.asarray(name, dtype=object), dtype=str_t)
f.create_dataset("source_images", data=np.asarray(src_images, dtype=object), dtype=str_t)
# TRACES: GR-004 | SR-001 — omitted entirely when unknown, so "unstamped"
# round-trips as unstamped rather than as a stamp naming no model.
if not _stamp_empty(embedder):
g = f.create_group("embedder")
g.attrs["model_name"] = embedder.get("model_name", "")
g.attrs["model_sha256"] = embedder.get("model_sha256", "")
g.attrs["embed_dim"] = np.int32(embedder.get("embed_dim", 512))
stamp_note = (f", embedder {embedder['model_name']}" if not _stamp_empty(embedder)
else ", NO EMBEDDER STAMP (GR-004)")
print(f"Saved: {output} ({len(actors)} actors, {emb_arr.shape[0]} embeddings"
f"{stamp_note})", file=sys.stderr)
print(f"Saved: {output} ({len(actors)} actors, {emb_arr.shape[0]} embeddings)",
file=sys.stderr)
def load_gallery_hdf5(path: Path) -> dict:
@@ -196,14 +158,6 @@ def load_gallery_hdf5(path: Path) -> dict:
if "source_images" in f:
src_images = [s.decode() if isinstance(s, bytes) else s
for s in f["source_images"][:]]
# TRACES: GR-004 | SR-001 — carried through so a derived gallery (filter,
# merge, cast-restrict) keeps the binding of the gallery it came from.
stamp = None
if "embedder" in f:
a = f["embedder"].attrs
stamp = {"model_name": _as_str(a.get("model_name", "")),
"model_sha256": _as_str(a.get("model_sha256", "")),
"embed_dim": int(a.get("embed_dim", 512))}
actors = []
for a in range(len(offset)):
@@ -213,19 +167,15 @@ def load_gallery_hdf5(path: Path) -> dict:
if src_images is not None:
actor["source_images"] = [src_images[s + i] for i in range(n)]
actors.append(actor)
out = {"actors": actors}
if stamp is not None:
out["embedder"] = stamp
return out
return {"actors": actors}
def save_gallery(gallery: dict, missing: list[dict], output: Path,
embedder: dict | None = None) -> None:
def save_gallery(gallery: dict, missing: list[dict], output: Path) -> None:
"""Write the gallery as HDF5 (forcing a .h5 extension) and, if any actors
lack images, a .missing_images.json sidecar."""
if output.suffix not in (".h5", ".hdf5"):
output = output.with_suffix(".h5")
save_gallery_hdf5(gallery, output, embedder)
save_gallery_hdf5(gallery, output)
if missing:
missing_path = output.with_name(output.stem + ".missing_images.json")
scripts/sae_stamp.py
-217
View File
@@ -1,217 +0,0 @@
"""Gallery ↔ embedder model binding (GR-004).
TRACES: GR-004 | SR-001
Python twin of src/gallery/embedder_stamp.{hpp,cpp}; the two implement the same
comparison rules and must stay in agreement. Kept as its own module — rather than
folded into sae_gallery — because scripts/optimizer/replay.py imports it once per
replay subprocess, thousands of times in a DE sweep, and must not pay for
sae_gallery's requests/Pillow imports to ask "were these made by the same model?".
Dependencies here are hashlib, json and h5py, all of which a replay already loads.
A gallery is only valid for the embedder that built it: cosine similarities across
models are meaningless but look plausible, so the mistake is silent and every
measurement taken afterwards is suspect. Identity = model filename + SHA-256 of
the model file. The hash decides (a model re-exported in place keeps its name but
not its bytes); the name is what makes the error readable. See
src/gallery/embedder_stamp.hpp for the full rationale.
"""
import hashlib
import json
import os
import sys
from pathlib import Path
import h5py
def _as_str(v) -> str:
return v.decode() if isinstance(v, bytes) else ("" if v is None else str(v))
_STAMP_CACHE: dict = {}
class EmbedderMismatch(RuntimeError):
"""Gallery was built with a different embedder than the one about to be used."""
def sha256_file(path) -> str:
"""Lowercase hex SHA-256 of a file's bytes; "" if it cannot be read."""
path = Path(path)
try:
st = path.stat()
except OSError:
return ""
key = (str(path), st.st_mtime_ns, st.st_size)
if key in _STAMP_CACHE:
return _STAMP_CACHE[key]
h = hashlib.sha256()
try:
with open(path, "rb") as f:
for chunk in iter(lambda: f.read(1 << 20), b""):
h.update(chunk)
except OSError:
return ""
_STAMP_CACHE[key] = h.hexdigest()
return _STAMP_CACHE[key]
def embedder_stamp(model_path, embed_dim: int = 512) -> dict:
"""Identify an embedder model file → {"model_name", "model_sha256", "embed_dim"}.
A model file that is absent (e.g. a TRT deployment running from a prebuilt
.engine) yields a name-only stamp: still comparable, just not provable."""
if not model_path:
return {"model_name": "", "model_sha256": "", "embed_dim": embed_dim}
sha = sha256_file(model_path)
if not sha:
print(f"[gallery] cannot hash embedder model {model_path} — model binding "
f"falls back to filename only (GR-004)", file=sys.stderr)
return {"model_name": Path(model_path).name, "model_sha256": sha,
"embed_dim": embed_dim}
def _stamp_empty(s) -> bool:
return not s or (not s.get("model_name") and not s.get("model_sha256"))
def describe_stamp(s) -> str:
if _stamp_empty(s):
return "UNKNOWN"
name = s.get("model_name") or "<unnamed model>"
sha = s.get("model_sha256") or ""
return f"{name} (sha256 {sha[:12]}…)" if sha else f"{name} (sha256 unavailable)"
def require_gallery_stamp_from_env() -> bool:
"""SAE_REQUIRE_GALLERY_STAMP=1 → an unprovable binding is fatal, not a warning."""
return os.environ.get("SAE_REQUIRE_GALLERY_STAMP", "0") not in ("", "0")
def check_embedder_stamp(built_with: dict | None, loading_with: dict | None,
gallery_desc: str = "gallery",
embedder_desc: str = "embedder") -> tuple[str, str]:
"""Pure comparison. Returns (verdict, message); verdict is one of
match / weak_match / unstamped / unknown_embedder / mismatch.
Same rules as compare_embedder_stamps() in src/gallery/embedder_stamp.cpp."""
if _stamp_empty(built_with):
return "unstamped", (
f"gallery '{gallery_desc}' carries no embedder stamp (GR-004).\n"
f" gallery was built with : UNKNOWN — this file predates model binding\n"
f" embedder now loaded : {describe_stamp(loading_with)} [{embedder_desc}]\n"
f" If these are not the same model every similarity from this run is\n"
f" meaningless but will look plausible. Rebuild or re-stamp the gallery\n"
f" (scripts/stamp_gallery.py), or run with SAE_REQUIRE_GALLERY_STAMP=1 to\n"
f" make this a hard error.")
if _stamp_empty(loading_with):
return "unknown_embedder", (
f"cannot identify the embedder being used against gallery "
f"'{gallery_desc}' (GR-004).\n"
f" gallery was built with : {describe_stamp(built_with)}\n"
f" embedder now loaded : UNKNOWN [{embedder_desc}]\n"
f" The binding cannot be checked, so it is not being checked.")
mismatch_tail = (
" Cosine similarities between embeddings from different models are\n"
" meaningless but look plausible. Rebuild the gallery with the loaded\n"
" model, or point the embedder at the model the gallery was built with.")
if int(built_with.get("embed_dim", 512)) != int(loading_with.get("embed_dim", 512)):
return "mismatch", (
f"gallery/embedder MODEL MISMATCH — refusing to run (GR-004).\n"
f" gallery was built with : {describe_stamp(built_with)}, "
f"dim={built_with.get('embed_dim')} [{gallery_desc}]\n"
f" embedder now loaded : {describe_stamp(loading_with)}, "
f"dim={loading_with.get('embed_dim')} [{embedder_desc}]\n"
f" Embedding dimensions differ; these are not the same space.")
a, b = built_with.get("model_sha256", ""), loading_with.get("model_sha256", "")
if a and b:
if a == b:
note = ""
if built_with.get("model_name") != loading_with.get("model_name"):
note = (f" (gallery recorded it as '{built_with.get('model_name')}', "
f"loaded from '{loading_with.get('model_name')}' — "
f"same bytes, renamed file)")
return "match", f"embedder binding verified: {describe_stamp(built_with)}{note}"
return "mismatch", (
f"gallery/embedder MODEL MISMATCH — refusing to run (GR-004).\n"
f" gallery was built with : {built_with.get('model_name')} sha256={a}\n"
f" [{gallery_desc}]\n"
f" embedder now loaded : {loading_with.get('model_name')} sha256={b}\n"
f" [{embedder_desc}]\n" + mismatch_tail)
if built_with.get("model_name") and \
built_with.get("model_name") == loading_with.get("model_name"):
return "weak_match", (
f"embedder binding UNPROVEN for gallery '{gallery_desc}' (GR-004).\n"
f" gallery was built with : {describe_stamp(built_with)}\n"
f" embedder now loaded : {describe_stamp(loading_with)} [{embedder_desc}]\n"
f" Filenames agree but at least one SHA-256 is unavailable, so an\n"
f" in-place re-export under the same name would not be detected.")
return "mismatch", (
f"gallery/embedder MODEL MISMATCH — refusing to run (GR-004).\n"
f" gallery was built with : {describe_stamp(built_with)} [{gallery_desc}]\n"
f" embedder now loaded : {describe_stamp(loading_with)} [{embedder_desc}]\n"
+ mismatch_tail)
def enforce_embedder_stamp(built_with, loading_with, gallery_desc, embedder_desc,
require_stamp: bool = False) -> str:
"""Apply check_embedder_stamp: raise EmbedderMismatch when fatal, else warn.
A mismatch is fatal unconditionally — there is no bypass, because a mismatch is
a known-wrong state, not an unknown one. The three "cannot prove it" verdicts
warn loudly and become fatal under require_stamp / SAE_REQUIRE_GALLERY_STAMP."""
strict = require_stamp or require_gallery_stamp_from_env()
verdict, msg = check_embedder_stamp(built_with, loading_with,
gallery_desc, embedder_desc)
if verdict == "mismatch":
raise EmbedderMismatch(msg)
if strict and verdict != "match":
raise EmbedderMismatch(
msg + "\n (fatal because SAE_REQUIRE_GALLERY_STAMP is set)")
if verdict == "match":
print(f"[gallery] {msg}", file=sys.stderr)
else:
print(f"\n[gallery] ***** WARNING (GR-004) *****\n{msg}\n"
f"[gallery] ****************************\n", file=sys.stderr)
return verdict
def read_gallery_stamp(path) -> dict | None:
"""The embedder stamp recorded in a gallery file, or None if unstamped.
Handles both the HDF5 /embedder group and the legacy JSON "embedder" object."""
path = Path(path)
if path.suffix in (".h5", ".hdf5"):
with h5py.File(path, "r") as f:
if "embedder" not in f:
return None
a = f["embedder"].attrs
return {"model_name": _as_str(a.get("model_name", "")),
"model_sha256": _as_str(a.get("model_sha256", "")),
"embed_dim": int(a.get("embed_dim", 512))}
data = json.loads(path.read_text())
return data.get("embedder") or None
def verify_gallery_stamp(gallery_path, model_path=None, *, stamp=None,
embedder_desc: str | None = None,
require_stamp: bool = False) -> str:
"""Load a gallery's stamp and check it against a model file (or an explicit
stamp, e.g. one read off an embedding dump). Raises EmbedderMismatch."""
loading = stamp if stamp is not None else embedder_stamp(model_path)
return enforce_embedder_stamp(read_gallery_stamp(gallery_path), loading,
str(gallery_path),
embedder_desc or str(model_path or "unknown"),
require_stamp)
def _as_str(v) -> str:
return v.decode() if isinstance(v, bytes) else ("" if v is None else str(v))
scripts/stamp_gallery.py
-86
View File
@@ -1,86 +0,0 @@
#!/usr/bin/env python3
"""stamp_gallery.py — bind an existing gallery to the embedder that built it.
TRACES: GR-004 | SR-001
Galleries built before model binding carry no embedder stamp. They still load,
but every consumer warns that it cannot tell whether the gallery and the embedder
belong together — and under SAE_REQUIRE_GALLERY_STAMP=1 they refuse to run.
This is the migration path, and the reason the unstamped case is a warning rather
than a hard failure: re-binding an existing gallery costs one command and no
re-embedding, so nobody has to choose between a bricked setup and a check they
route around.
python scripts/stamp_gallery.py --gallery gallery.h5 \\
--arcface models/LVFace-B_Glint360K.onnx
The stamp is an ASSERTION: you are stating which model produced these vectors.
Nothing can verify it from the vectors themselves, which is exactly why the stamp
has to be written at build time going forward. Stamping the wrong model is worse
than leaving it unstamped, because it converts a loud warning into a false
all-clear — so --show it first if you are not certain.
python scripts/stamp_gallery.py --gallery gallery.h5 --show
"""
from __future__ import annotations
import argparse
import sys
from pathlib import Path
import h5py
sys.path.insert(0, str(Path(__file__).resolve().parent))
from sae_gallery import (describe_stamp, embedder_stamp, # noqa: E402
read_gallery_stamp)
def main() -> int:
p = argparse.ArgumentParser(description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
p.add_argument("--gallery", required=True, help="gallery .h5 to stamp in place")
p.add_argument("--arcface", help="the ONNX that built it (hashed into the stamp)")
p.add_argument("--show", action="store_true", help="print the current stamp and exit")
p.add_argument("--force", action="store_true",
help="overwrite an existing stamp (refused otherwise)")
args = p.parse_args()
path = Path(args.gallery)
if path.suffix not in (".h5", ".hdf5"):
return err(f"{path}: only HDF5 galleries can be stamped in place")
current = read_gallery_stamp(path)
print(f"{path}: current stamp = {describe_stamp(current)}", file=sys.stderr)
if args.show:
return 0
if not args.arcface:
return err("--arcface is required (or use --show)")
if current and not args.force:
return err("gallery is already stamped — pass --force to overwrite, but be "
"sure: a wrong stamp turns a warning into a false all-clear")
stamp = embedder_stamp(args.arcface)
if not stamp["model_sha256"]:
return err(f"cannot hash {args.arcface} — refusing to write a name-only "
"stamp, which would claim more certainty than it has")
with h5py.File(path, "r+") as f:
if "embedder" in f:
del f["embedder"]
g = f.create_group("embedder")
g.attrs["model_name"] = stamp["model_name"]
g.attrs["model_sha256"] = stamp["model_sha256"]
g.attrs["embed_dim"] = stamp["embed_dim"]
print(f"{path}: stamped with {describe_stamp(stamp)}", file=sys.stderr)
return 0
def err(msg: str) -> int:
print(f"[stamp_gallery] {msg}", file=sys.stderr)
return 1
if __name__ == "__main__":
raise SystemExit(main())
scripts/traceability/extract_traces.py
Executable
+1643
View File
File diff suppressed because it is too large Load Diff
scripts/traceability/test_extract_traces.py
Executable
+1047
View File
File diff suppressed because it is too large Load Diff
scripts/traceability/traceability-gate.sh
Executable
+62
View File
@@ -0,0 +1,62 @@
#!/bin/sh
#
# Requirement traceability gate. Run locally exactly as CI runs it, from the
# component repo root:
#
# scripts/traceability/traceability-gate.sh
#
# Writes the JSON report and the markdown matrix, prints the coverage report,
# and exits non-zero when the gate fails.
#
# This script is shared by every JRay component, so it knows nothing about any
# one repo. All repo-specific settings - requirement ID prefixes, source
# suffixes, scan roots, register path, thresholds - live in `traceability.toml`
# at the component repo root. Run
#
# scripts/traceability/extract_traces.py --print-example-config
#
# for the annotated schema. A repo whose config is wrong parses zero
# requirements or scans zero files, and the gate refuses to report rather than
# printing a misleading 0%.
#
# Environment (all optional; each overrides the config file):
# TRACES_CONFIG path to traceability.toml
# TRACES_ROOT repo root (default: nearest dir containing traceability.toml)
# MIN_COVERAGE minimum overall coverage percent
# ALLOW_ORPHANS 1 to report orphan tags without failing
# TRACES_JSON JSON report path
# TRACES_MD markdown matrix path
# SYSTEM_SPEC SPEC.md defining PR/SR; enables PR/SR orphan checking
# PYTHON interpreter (default: python3)
#
# Threshold policy belongs in traceability.toml, not here and not in the
# workflow YAML: a threshold written in two places is a threshold that will
# disagree with itself.
#
# POSIX sh, no bashisms, no jq - the extractor does its own arithmetic and
# printing, so CI needs nothing beyond python3.
set -eu
SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
PYTHON="${PYTHON:-python3}"
command -v "$PYTHON" >/dev/null 2>&1 || {
echo "FAILED: $PYTHON not found. The traceability gate needs Python 3.9+," >&2
echo " or 3.11+ to read traceability.toml." >&2
exit 2
}
set -- --format coverage
# Explicit `if` rather than `[ ... ] && ...`, because a trailing false test in
# an && list exits under `set -e` in some POSIX shells.
if [ -n "${TRACES_CONFIG:-}" ]; then set -- "$@" --config "$TRACES_CONFIG"; fi
if [ -n "${TRACES_ROOT:-}" ]; then set -- "$@" --root "$TRACES_ROOT"; fi
if [ -n "${MIN_COVERAGE:-}" ]; then set -- "$@" --min-coverage "$MIN_COVERAGE"; fi
if [ -n "${TRACES_JSON:-}" ]; then set -- "$@" --json-out "$TRACES_JSON"; fi
if [ -n "${TRACES_MD:-}" ]; then set -- "$@" --markdown-out "$TRACES_MD"; fi
if [ -n "${SYSTEM_SPEC:-}" ]; then set -- "$@" --system-spec "$SYSTEM_SPEC"; fi
if [ "${ALLOW_ORPHANS:-0}" = "1" ]; then set -- "$@" --allow-orphans; fi
exec "$PYTHON" "$SCRIPT_DIR/extract_traces.py" "$@"
scripts/vendor/jray-project
Vendored
-1
Submodule scripts/vendor/jray-project deleted from 17106f3370
src/audio_signature.cpp
-428
View File
@@ -1,428 +0,0 @@
// ── JRay audio signature, v1 — implementation ────────────────────────────────
//
/// TRACES: IR-004, IR-007, IR-008 | SR-003
//
// The contract this implements is documented in full in audio_signature.hpp;
// read that before changing anything here. Every constant is load-bearing: the
// JRay Jellyfin plugin computes the same bytes in C#, and a signature that
// differs in any parameter simply does not match.
//
// Audio decode is a *second stream from an existing dependency* — the pipeline
// already links libavformat/libavcodec/libavutil for video (ffmpeg_decoder.hpp);
// this adds libswresample for the downmix+resample, no new project dependency.
// The FFT is written out here rather than pulled from a library for the same
// reason the plugin vendors one: it is a fixed, fully specified transform, and
// a dependency whose version could change the numerics is a liability when the
// output has to be bit-identical across two languages.
#include "audio_signature.hpp"
extern "C" {
#include <libavcodec/avcodec.h>
#include <libavformat/avformat.h>
#include <libavutil/avutil.h>
#include <libavutil/channel_layout.h>
#include <libavutil/opt.h>
#include <libavutil/samplefmt.h>
#include <libswresample/swresample.h>
}
#include <algorithm>
#include <cmath>
#include <cstring>
#include <vector>
namespace sae::audio {
namespace {
constexpr double kPi = 3.14159265358979323846;
// ── Band table ───────────────────────────────────────────────────────────────
// edge[b] = 300 * 10^(b/32); band b owns FFT bins [k_lo[b], k_lo[b+1]).
// ceil() of the edge in bins, so membership is decided once by integers rather
// than by a float comparison per bin per frame. The bands tile [112, 1115)
// contiguously with no gap and no overlap, which is what lets the frame energy
// below be accumulated from the per-band sums.
std::array<std::pair<int, int>, kNumBands> build_band_table() {
const double hz_per_bin = static_cast<double>(kSampleRate) / kFrameSize;
std::array<int, kNumBands + 1> k{};
for (int b = 0; b <= kNumBands; ++b) {
const double edge = kBandLoHz * std::pow(kBandHiHz / kBandLoHz,
static_cast<double>(b) / kNumBands);
k[b] = static_cast<int>(std::ceil(edge / hz_per_bin));
}
std::array<std::pair<int, int>, kNumBands> tbl{};
for (int b = 0; b < kNumBands; ++b) tbl[b] = {k[b], k[b + 1]};
return tbl;
}
// Hann, periodic: w[n] = 0.5 * (1 - cos(2*pi*n/N)). Not the symmetric (N-1)
// variant — the two differ, and the difference is observable.
const std::vector<double>& hann_window() {
static const std::vector<double> w = [] {
std::vector<double> v(kFrameSize);
for (int n = 0; n < kFrameSize; ++n)
v[n] = 0.5 * (1.0 - std::cos(2.0 * kPi * n / kFrameSize));
return v;
}();
return w;
}
// ── Radix-2 decimation-in-time complex FFT, in place, no normalisation ──────
// Twiddles are precomputed per stage from cos/sin of -2*pi*j/len so the angle
// is an exactly reproducible double in any language and only the libm rounding
// of cos/sin (≤1 ulp) can differ — orders of magnitude below the decision
// margins in the golden fixture.
struct FftTables {
std::vector<int> rev; // bit-reversal permutation
std::vector<std::vector<double>> wr, wi; // per stage
};
const FftTables& fft_tables() {
static const FftTables t = [] {
FftTables f;
f.rev.resize(kFrameSize);
int bits = 0;
while ((1 << bits) < kFrameSize) ++bits;
for (int i = 0; i < kFrameSize; ++i) {
int r = 0;
for (int b = 0; b < bits; ++b)
if (i & (1 << b)) r |= 1 << (bits - 1 - b);
f.rev[i] = r;
}
for (int len = 2; len <= kFrameSize; len <<= 1) {
const int half = len / 2;
std::vector<double> cr(half), ci(half);
for (int j = 0; j < half; ++j) {
const double ang = -2.0 * kPi * j / len;
cr[j] = std::cos(ang);
ci[j] = std::sin(ang);
}
f.wr.push_back(std::move(cr));
f.wi.push_back(std::move(ci));
}
return f;
}();
return t;
}
void fft_4096(std::vector<double>& re, std::vector<double>& im) {
const FftTables& t = fft_tables();
for (int i = 0; i < kFrameSize; ++i) {
const int j = t.rev[i];
if (i < j) { std::swap(re[i], re[j]); std::swap(im[i], im[j]); }
}
int stage = 0;
for (int len = 2; len <= kFrameSize; len <<= 1, ++stage) {
const int half = len / 2;
const std::vector<double>& wr = t.wr[stage];
const std::vector<double>& wi = t.wi[stage];
for (int base = 0; base < kFrameSize; base += len) {
for (int j = 0; j < half; ++j) {
const int a = base + j;
const int b = a + half;
const double tr = re[b] * wr[j] - im[b] * wi[j];
const double ti = re[b] * wi[j] + im[b] * wr[j];
re[b] = re[a] - tr; im[b] = im[a] - ti;
re[a] = re[a] + tr; im[a] = im[a] + ti;
}
}
}
}
int energy_class(double r) {
if (r < kEnergyClassEdges[0]) return 0;
if (r < kEnergyClassEdges[1]) return 1;
if (r < kEnergyClassEdges[2]) return 2;
return 3;
}
// ── FFmpeg RAII ─────────────────────────────────────────────────────────────
struct DecodeCtx {
AVFormatContext* fmt = nullptr;
AVCodecContext* dec = nullptr;
SwrContext* swr = nullptr;
AVFrame* frm = nullptr;
AVPacket* pkt = nullptr;
~DecodeCtx() {
if (swr) swr_free(&swr);
if (frm) av_frame_free(&frm);
if (pkt) av_packet_free(&pkt);
if (dec) avcodec_free_context(&dec);
if (fmt) avformat_close_input(&fmt);
}
};
bool open_resampler(DecodeCtx& c, const AVFrame* f) {
#if LIBAVUTIL_VERSION_INT >= AV_VERSION_INT(57, 24, 100)
AVChannelLayout out_layout;
av_channel_layout_default(&out_layout, 1); // mono
AVChannelLayout in_layout;
if (av_channel_layout_copy(&in_layout, &f->ch_layout) < 0) return false;
if (in_layout.nb_channels <= 0) {
av_channel_layout_uninit(&in_layout);
av_channel_layout_default(&in_layout, 1);
}
const int rc = swr_alloc_set_opts2(
&c.swr,
&out_layout, AV_SAMPLE_FMT_FLT, kSampleRate,
&in_layout, static_cast<AVSampleFormat>(f->format),
f->sample_rate ? f->sample_rate : kSampleRate,
0, nullptr);
av_channel_layout_uninit(&in_layout);
av_channel_layout_uninit(&out_layout);
if (rc < 0 || !c.swr) return false;
#else
const int64_t in_layout = f->channel_layout
? static_cast<int64_t>(f->channel_layout)
: av_get_default_channel_layout(f->channels ? f->channels : 1);
c.swr = swr_alloc_set_opts(
nullptr,
AV_CH_LAYOUT_MONO, AV_SAMPLE_FMT_FLT, kSampleRate,
in_layout, static_cast<AVSampleFormat>(f->format),
f->sample_rate ? f->sample_rate : kSampleRate,
0, nullptr);
if (!c.swr) return false;
#endif
return swr_init(c.swr) >= 0;
}
// Push one decoded frame (or a flush) through the resampler, dropping the
// leading `to_skip` output samples, and append to `out`.
void drain(SwrContext* swr, const AVFrame* f, int in_rate,
std::size_t& to_skip, std::vector<float>& out) {
const int64_t delay = swr_get_delay(swr, in_rate ? in_rate : kSampleRate);
const int in_n = f ? f->nb_samples : 0;
const int max_out = static_cast<int>(av_rescale_rnd(
delay + in_n, kSampleRate, in_rate ? in_rate : kSampleRate, AV_ROUND_UP)) + 32;
if (max_out <= 0) return;
std::vector<float> buf(static_cast<std::size_t>(max_out));
uint8_t* dst = reinterpret_cast<uint8_t*>(buf.data());
const int n = swr_convert(swr, &dst, max_out,
f ? const_cast<const uint8_t**>(f->extended_data) : nullptr,
in_n);
if (n <= 0) return;
std::size_t produced = static_cast<std::size_t>(n);
std::size_t off = 0;
if (to_skip) {
const std::size_t drop = std::min(to_skip, produced);
to_skip -= drop;
off = drop;
produced -= drop;
}
if (produced)
out.insert(out.end(), buf.begin() + off, buf.begin() + off + produced);
}
} // namespace
// ── Public surface ──────────────────────────────────────────────────────────
const std::array<std::pair<int, int>, kNumBands>& band_fft_bins() {
static const std::array<std::pair<int, int>, kNumBands> tbl = build_band_table();
return tbl;
}
std::string base64_encode(const std::uint8_t* data, std::size_t n) {
static constexpr char kAlphabet[] =
"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
std::string out;
out.reserve(((n + 2) / 3) * 4);
std::size_t i = 0;
for (; i + 3 <= n; i += 3) {
const std::uint32_t v = (std::uint32_t(data[i]) << 16) |
(std::uint32_t(data[i + 1]) << 8) |
std::uint32_t(data[i + 2]);
out += kAlphabet[(v >> 18) & 0x3F];
out += kAlphabet[(v >> 12) & 0x3F];
out += kAlphabet[(v >> 6) & 0x3F];
out += kAlphabet[v & 0x3F];
}
if (i < n) {
const bool two = (n - i) == 2;
const std::uint32_t v = (std::uint32_t(data[i]) << 16) |
(two ? (std::uint32_t(data[i + 1]) << 8) : 0u);
out += kAlphabet[(v >> 18) & 0x3F];
out += kAlphabet[(v >> 12) & 0x3F];
out += two ? kAlphabet[(v >> 6) & 0x3F] : '=';
out += '=';
}
return out;
}
std::uint64_t fnv1a64(const void* data, std::size_t n) {
const auto* p = static_cast<const std::uint8_t*>(data);
std::uint64_t h = 0xcbf29ce484222325ULL;
for (std::size_t i = 0; i < n; ++i) {
h ^= p[i];
h *= 0x100000001b3ULL;
}
return h;
}
/// TRACES: IR-004
std::vector<std::uint8_t> pack_frames(const std::vector<float>& mono) {
if (mono.size() < static_cast<std::size_t>(kFrameSize)) return {};
const std::size_t nframes = 1 + (mono.size() - kFrameSize) / kHopSize;
const auto& bands = band_fft_bins();
const auto& win = hann_window();
const int k_lo = bands.front().first;
const int k_hi = bands.back().second; // exclusive
const double bin_count = static_cast<double>(k_hi - k_lo);
std::vector<double> re(kFrameSize), im(kFrameSize);
std::vector<std::uint8_t> peak(nframes);
std::vector<double> energy(nframes);
for (std::size_t f = 0; f < nframes; ++f) {
const float* src = mono.data() + f * kHopSize;
for (int n = 0; n < kFrameSize; ++n) {
re[n] = static_cast<double>(src[n]) * win[n];
im[n] = 0.0;
}
fft_4096(re, im);
// Per-band mean magnitude; the bands tile the 300–3000 Hz range with no
// gaps, so the frame's band-limited energy is the sum of the band sums.
double best = -1.0, total = 0.0;
int best_b = 0;
for (int b = 0; b < kNumBands; ++b) {
double sum = 0.0;
for (int k = bands[b].first; k < bands[b].second; ++k)
sum += std::sqrt(re[k] * re[k] + im[k] * im[k]);
total += sum;
const double mean = sum / (bands[b].second - bands[b].first);
if (mean > best) { best = mean; best_b = b; } // ties → lowest index
}
peak[f] = static_cast<std::uint8_t>(best_b);
energy[f] = total / bin_count;
}
// Reference is the upper median of the frame energies: an actually observed
// value (no averaging of the two middle samples), so it is bit-reproducible,
// gain-invariant and barely moves when the window is trimmed.
std::vector<double> sorted = energy;
std::sort(sorted.begin(), sorted.end());
const double ref = sorted[sorted.size() / 2];
std::vector<std::uint8_t> out(nframes);
for (std::size_t f = 0; f < nframes; ++f) {
const double r = std::log10((energy[f] + kEnergyEps) / (ref + kEnergyEps));
out[f] = static_cast<std::uint8_t>(((peak[f] & 0x1F) << 2) |
(energy_class(r) & 0x03));
}
return out;
}
/// TRACES: IR-004, IR-008
std::optional<std::string> signature_from_mono(const std::vector<float>& mono) {
const std::vector<std::uint8_t> packed = pack_frames(mono);
if (packed.empty()) return std::nullopt;
return std::string(kVersionPrefix) + base64_encode(packed.data(), packed.size());
}
/// TRACES: IR-004, IR-007
std::optional<std::vector<float>> decode_centre_window(const std::string& path) {
av_log_set_level(AV_LOG_ERROR);
DecodeCtx c;
if (avformat_open_input(&c.fmt, path.c_str(), nullptr, nullptr) < 0)
return std::nullopt;
if (avformat_find_stream_info(c.fmt, nullptr) < 0) return std::nullopt;
if (c.fmt->duration == AV_NOPTS_VALUE) return std::nullopt;
const double duration = static_cast<double>(c.fmt->duration) / AV_TIME_BASE;
// IR-007 — the window underflows, so there is no signature and no sync
// offset downstream. The plugin applies the identical rule.
if (duration < kWindowSec) return std::nullopt;
const int idx = av_find_best_stream(c.fmt, AVMEDIA_TYPE_AUDIO, -1, -1, nullptr, 0);
if (idx < 0) return std::nullopt; // no audio → no signature
AVStream* st = c.fmt->streams[idx];
const AVCodec* codec = avcodec_find_decoder(st->codecpar->codec_id);
if (!codec) return std::nullopt;
c.dec = avcodec_alloc_context3(codec);
if (!c.dec) return std::nullopt;
if (avcodec_parameters_to_context(c.dec, st->codecpar) < 0) return std::nullopt;
c.dec->thread_count = 0;
if (avcodec_open2(c.dec, codec, nullptr) < 0) return std::nullopt;
const double start_sec = duration / 2.0 - kWindowSec / 2.0;
// Seek to a packet at or before the window start; the exact start is then
// reached by discarding the leading output samples, which is what
// `ffmpeg -ss <t> -i <file>` does and therefore what the plugin sees.
if (start_sec > 0.0) {
const int64_t tgt = av_rescale_q(
static_cast<int64_t>(start_sec * AV_TIME_BASE), AV_TIME_BASE_Q, st->time_base);
if (av_seek_frame(c.fmt, idx, tgt, AVSEEK_FLAG_BACKWARD) >= 0)
avcodec_flush_buffers(c.dec);
}
c.frm = av_frame_alloc();
c.pkt = av_packet_alloc();
if (!c.frm || !c.pkt) return std::nullopt;
std::vector<float> mono;
mono.reserve(kWindowSamples + kSampleRate);
std::size_t to_skip = 0;
bool have_swr = false;
int in_rate = kSampleRate;
bool eof = false;
while (mono.size() < kWindowSamples && !eof) {
const int rr = av_read_frame(c.fmt, c.pkt);
if (rr < 0) {
eof = true;
avcodec_send_packet(c.dec, nullptr); // flush the decoder
} else if (c.pkt->stream_index != idx) {
av_packet_unref(c.pkt);
continue;
} else {
avcodec_send_packet(c.dec, c.pkt);
av_packet_unref(c.pkt);
}
while (avcodec_receive_frame(c.dec, c.frm) == 0) {
if (!have_swr) {
if (!open_resampler(c, c.frm)) return std::nullopt;
have_swr = true;
in_rate = c.frm->sample_rate ? c.frm->sample_rate : kSampleRate;
int64_t pts = c.frm->best_effort_timestamp;
if (pts == AV_NOPTS_VALUE) pts = c.frm->pts;
const double t0 = (pts == AV_NOPTS_VALUE)
? start_sec : av_q2d(st->time_base) * static_cast<double>(pts);
const double lead = start_sec - t0;
to_skip = lead > 0.0
? static_cast<std::size_t>(std::llround(lead * kSampleRate)) : 0;
}
drain(c.swr, c.frm, in_rate, to_skip, mono);
av_frame_unref(c.frm);
if (mono.size() >= kWindowSamples) break;
}
}
if (have_swr && mono.size() < kWindowSamples)
drain(c.swr, nullptr, in_rate, to_skip, mono); // flush the resampler
if (mono.empty()) return std::nullopt;
// Truncate to exactly 120.000 s so the frame count is 1288 for every input
// and does not wobble with seek granularity or the resampler tail.
if (mono.size() > kWindowSamples) mono.resize(kWindowSamples);
return mono;
}
/// TRACES: IR-004, IR-005, IR-007, IR-008
std::optional<std::string> compute_signature(const std::string& path) {
const std::optional<std::vector<float>> mono = decode_centre_window(path);
if (!mono) return std::nullopt;
return signature_from_mono(*mono);
}
} // namespace sae::audio
src/audio_signature.hpp
-158
View File
@@ -1,158 +0,0 @@
#pragma once
// ── JRay audio signature, v1 ─────────────────────────────────────────────────
//
/// TRACES: IR-004, IR-005, IR-007, IR-008 | SR-003
//
// A content-derived spectral-peak signature taken from the *centre* of the
// media, so a truth file is self-identifying: a consumer can tell whether a
// local file is the same cut as the one a manifest describes, and recover the
// frame offset when it is the same cut trimmed differently.
//
// The construction is owned by `JRay-public-server/SPEC.md` §3 and is
// reproduced by the JRay Jellyfin plugin in C#. **The two implementations must
// agree byte for byte** — a signature that differs in any parameter simply does
// not match, which defeats the entire point. Every deviation is therefore a
// breaking change and must go through the `v1:` prefix (see kVersionPrefix).
//
// Server spec §3, restated:
//
// 1. Decode a 120 s window centred on the midpoint (runtime/2 ± 60 s).
// 2. Downmix to mono, resample to 11025 Hz.
// 3. STFT: 4096-sample frame, 1024-sample hop, Hann window (~1290 frames).
// 4. Per frame, log-magnitude spectrum over 300–3000 Hz.
// 5. 32 logarithmically spaced bins; peak bin index + coarse 2-bit energy
// class.
// 6. Pack one byte per frame; base64-encode.
// 7. Prefix `v1:`.
//
// ── Details the server spec leaves open, pinned here for v1 ──────────────────
//
// The prose above is not sufficient to reproduce a byte stream, so the choices
// below are the contract. They are mirrored in
// `tests/fixtures/audio/jray_audio_v1_golden.json`, which is the artefact
// shared with the plugin repo (IR-005).
//
// Arithmetic All DSP in IEEE-754 **double**. float32 is not sufficient:
// the golden fixture has frames whose two strongest bands are
// within 1.3% of each other, which double resolves identically
// everywhere and float32 does not.
// Sample scale FFmpeg's native s16→flt conversion, x * (1/32768), then
// widened to double. Values in [-1, 1).
// Framing Only whole frames: n_frames = 1 + (n_samples - 4096) / 1024,
// integer division, 0 when n_samples < 4096. A 120.000 s
// window is 1 323 000 samples → **1288 frames**.
// ("~1290" in the spec; the server accepts a tolerance.)
// Window Hann, **periodic**: w[n] = 0.5 * (1 - cos(2*pi*n/4096)).
// Not the symmetric (N-1) variant.
// Transform Plain radix-2 decimation-in-time complex FFT over 4096 real
// samples (imag = 0), no normalisation. Magnitude is
// sqrt(re² + im²). Twiddles from cos/sin of
// -2*pi*k/len computed in double.
// Band edges edge[b] = 300 * (3000/300)^(b/32), b = 0..32. Band b spans
// FFT bins [k_lo[b], k_lo[b+1]) with
// k_lo[b] = ceil(edge[b] * 4096 / 11025) — i.e. bins 112..1114
// inclusive, 8 bins in the narrowest band. Precomputed as an
// integer table so no float comparison decides membership.
// Band value **Mean** of the linear magnitudes in the band. Mean, not
// sum, so a wide high band is not favoured over a narrow low
// one; magnitude, not power, because it is an energy proxy and
// more codec-robust than a single bin's peak.
// Peak bin argmax over the 32 band values; ties resolve to the **lowest
// index**. The log of step 4 is a monotone squash and so
// cannot change an argmax — it is applied only where it is
// observable, in the energy class below.
// Energy class The spec says "coarse 2-bit energy class" and no more. v1
// defines it as the frame's band-limited energy relative to
// the window, which is invariant to gain (loudness
// normalisation must not change a signature) and robust to
// trimming (the median barely moves):
// E_f = mean magnitude over *all* FFT bins 112..1114
// Eref = median over frames of E_f, taken as the upper
// median sorted[n/2] — no averaging of the two middle
// values, so the reference is always an actual
// observed value and is bit-reproducible
// r = log10((E_f + 1e-12) / (Eref + 1e-12))
// class = 0 if r < -0.6, 1 if r < -0.2, 2 if r < 0.2, else 3
// The thresholds deliberately straddle r = 0 rather than sit
// on it, so the median frame itself is not on a boundary.
// Byte layout bit 7 = 0 (reserved), bits 6..2 = 5-bit band index,
// bits 1..0 = 2-bit energy class:
// byte = (band << 2) | class → always 0..127
// This is the structural constraint the server validates on
// upload (§3 "Validation and abuse").
// Base64 Standard alphabet A–Za–z0–9+/ with '=' padding.
//
// ── Short media (IR-007) ─────────────────────────────────────────────────────
//
// `runtime/2 ± 60 s` underflows below 120 s, so **no signature is emitted** and
// no sync offset is applied downstream. Both producers apply the identical
// rule; diverging here would break exactly the short items most likely to be
// misidentified. `compute_signature` returns `std::nullopt`.
//
// The same nullopt is returned for a file with no audio stream, an unopenable
// file, or an unknown duration. UR-9 is an enhancement and must never be able
// to break a fetch — degradation, not failure.
#include <array>
#include <cstddef>
#include <cstdint>
#include <optional>
#include <string>
#include <utility>
#include <vector>
namespace sae::audio {
// ── Contract constants — changing any of these is a `v1:` bump ───────────────
inline constexpr int kSampleRate = 11025;
inline constexpr int kFrameSize = 4096;
inline constexpr int kHopSize = 1024;
inline constexpr int kNumBands = 32;
inline constexpr double kBandLoHz = 300.0;
inline constexpr double kBandHiHz = 3000.0;
inline constexpr double kWindowSec = 120.0;
inline constexpr double kEnergyEps = 1e-12;
// Class thresholds on log10(E_frame / E_median); see the header comment.
inline constexpr double kEnergyClassEdges[3] = {-0.6, -0.2, 0.2};
// 120.000 s at 11025 Hz. The decoded window is truncated to exactly this so the
// frame count does not wobble with seek granularity or resampler tail.
inline constexpr std::size_t kWindowSamples =
static_cast<std::size_t>(kWindowSec * kSampleRate); // 1 323 000
inline constexpr std::size_t kExpectedFrames =
1 + (kWindowSamples - kFrameSize) / kHopSize; // 1288
static_assert(kWindowSamples == 1323000, "120 s at 11025 Hz");
static_assert(kExpectedFrames == 1288, "server spec's ~1290 frames");
/// The version prefix is the signature's own, separate from `schema_version`:
/// a future change to the DSP chain must be *detectable* rather than silently
/// producing non-matching signatures (IR-008).
inline constexpr const char* kVersionPrefix = "v1:";
/// FFT bin range [first, last) for each of the 32 log-spaced bands.
/// Computed once from the constants above; exposed so the golden fixture can
/// assert the table itself, not merely the signature it produces.
const std::array<std::pair<int, int>, kNumBands>& band_fft_bins();
/// Decode the centre window of `path` as mono float PCM at 11025 Hz.
/// nullopt when the media is shorter than 120 s (IR-007), has no audio stream,
/// or cannot be opened. Never throws.
std::optional<std::vector<float>> decode_centre_window(const std::string& path);
/// One packed byte per whole STFT frame. Empty when `mono` is shorter than one
/// frame. This is the payload that gets base64-encoded.
std::vector<std::uint8_t> pack_frames(const std::vector<float>& mono);
/// `v1:` + base64(pack_frames(mono)). nullopt when no whole frame fits.
std::optional<std::string> signature_from_mono(const std::vector<float>& mono);
/// Decode + sign. The one call the pipeline makes. nullopt per IR-007 and on
/// any decode failure — degradation, not failure.
std::optional<std::string> compute_signature(const std::string& path);
// ── Small utilities, exposed for the golden-fixture test ────────────────────
std::string base64_encode(const std::uint8_t* data, std::size_t n);
/// FNV-1a 64. Used only to pin the *decoded PCM* in the golden fixture, so a
/// codec-level difference is distinguishable from a DSP-level one.
std::uint64_t fnv1a64(const void* data, std::size_t n);
} // namespace sae::audio
src/config.hpp
+15 -33
View File
@@ -16,13 +16,7 @@ enum class Verbosity {
struct Config {
// ── Input ─────────────────────────────────────────────────────────────────
std::string movie_path;
std::string gallery_path;
// TRACES: IR-002 | SR-003
// "global" (matched against the whole library) or "limited" (this title's
// credited cast only). The strongest single quality signal when two
// manifests compete for the same cut: identical gallery_size can mean very
// different recall depending on which was used.
std::string gallery_scope{"global"}; // gallery.json produced by build_gallery
std::string gallery_path; // gallery.json produced by build_gallery
// ── Output ───────────────────────────────────────────────────────────────
std::string output_path; // annotations.json
@@ -46,14 +40,6 @@ struct Config {
float detector_conf{0.5f};
float detector_nms{0.4f};
/// TRACES: GR-004 | SR-001
// Gallery ↔ embedder binding. A gallery built with a different model than the
// one loaded here is a hard error, always. This flag additionally promotes
// "cannot prove they match" (unstamped legacy gallery, or a name-only match
// because the ONNX could not be hashed) from a loud warning to a hard error.
// Also settable via SAE_REQUIRE_GALLERY_STAMP=1. Measurement runs want it on.
bool require_gallery_stamp{false}; // --require-gallery-stamp
// ── Recognition (ArcFace ONNX) ────────────────────────────────────────────
std::string arcface_model;
std::string arcface_engine; // optional path to a pre-built TRT engine; bypasses ORT
@@ -100,25 +86,21 @@ struct Config {
float dense_scale{1.0f}; // dense-mode frame downscale (1 = off)
// ── Face tracking (frame-to-frame) ───────────────────────────────────────
/// TRACES: AR-007, AR-008, AR-024 | SR-002
// track_alpha is the *base* weight, used on ordinary frames. It is
// frame-dependent (AR-007): on is_cut / is_scene_boundary, and for any track
// that is no longer on screen, it drops to 0 (embedding only), because
// position carries no information across a viewpoint change or a gap.
float track_alpha{0.4f}; // base cost weight: 0=embedding only, 1=spatial only
float track_alpha{0.4f}; // cost weight: 0=embedding only, 1=spatial only
float track_min_iou{0.1f}; // IoU below which spatial link alone is rejected
// Minimum P(same person) for an association to be admissible on appearance
// alone. This replaces track_max_embed_dist (a raw cosine distance, AR-024).
// 0.5 is not a tuned constant: it is the decision boundary. Below it the pair
// is more likely two people than one, and no amount of IoU makes that a link
// worth asserting on identity grounds.
float track_assoc_min_prob{0.5f};
// How long a track that has gone off screen stays available for association
// before the registry reaps it and emits its presence claim (AR-013).
// Replaces track_max_frames_missing: a frame count silently changed meaning
// with sample_fps, and the same number had to be guessed twice (once for an
// ordinary miss, once for a cut). Seconds mean one thing at any sample rate.
double track_extinction_sec{5.0};
float track_max_embed_dist{0.7f}; // cosine dist above which embedding link alone is rejected
int track_max_frames_missing{5}; // expire track after N consecutive missed frames
// ── Cross-cut track re-association ────────────────────────────────────────
// A camera-angle change (Frame::is_cut) breaks spatial (IoU) continuity but
// not identity: the same people are usually still on screen from a new angle.
// Instead of destroying tracks on a cut, the tracker parks them in an
// inactive pool. A post-cut detection whose raw cosine similarity to a parked
// track's last-frame embedding is ≥ cut_revive_sim revives that track_id
// (identity continuity survives the cut); otherwise it starts a fresh track.
// Parked tracks that go unrevived for cut_inactive_max_frames are dropped.
float cut_revive_sim{0.50f}; // min raw cosine sim (last-frame emb) to revive across a cut
int cut_inactive_max_frames{5}; // drop a parked track after N frames without revival
// ── Scene tracking ────────────────────────────────────────────────────────
// extinction_sec re-tuned by DE against X-Ray per-second presence, 4-film rep4
src/evidence_discount.hpp
-85
View File
@@ -1,85 +0,0 @@
#pragma once
/// TRACES: AR-024, AR-025 | SR-002
///
/// EvidenceDiscounter — how much a single observation is allowed to move a
/// track's belief.
///
/// **The independence problem.** Per-frame identity evidence is accumulated as
/// log-odds along a track (AR-025), which is only valid for *independent*
/// observations. Consecutive frames of one track are nothing of the kind: near
/// identical pose, lighting and expression. Treating them as independent drives
/// the posterior to certainty on what is effectively one measurement — thirty
/// frames of the same face at the same angle is not thirty pieces of evidence.
///
/// The mitigation is to weight each observation by how much it *adds*: a view
/// the track has already contributed is discounted toward zero, a genuinely new
/// pose counts in full. This reuses the same judgement the diversity buffer
/// makes for gallery expansion (AR-019) — which embeddings on a track are
/// mutually distinct — rather than inventing a second notion of novelty.
///
/// Owned by TrackRegistry rather than left to callers. A caller that forgot to
/// discount, or applied it twice, would silently produce confident wrong
/// answers, and the registry is the one place where all evidence converges.
///
/// **Similarity enters as a calibrated probability, never a raw cosine**
/// (AR-024): "is this the same view" is a decision, and a bare cosine threshold
/// means something different for every model and every face size.
#include "types.hpp"
#include <algorithm>
#include <cmath>
#include <functional>
#include <vector>
class EvidenceDiscounter {
public:
/// cosine similarity → P(same view). Supplied by the caller so the
/// calibration fitted for the active embedder is used (AR-023/AR-024).
using Calibrate = std::function<float(float)>;
struct Config {
int max_views{8}; ///< distinct views remembered per track
float admit_below{0.6f}; ///< P(same view) under this ⇒ a new view
float floor{0.0f}; ///< minimum weight for a redundant observation
};
// Two constructors rather than a defaulted argument: `Config{}` as a default
// argument would reference Config's own member initializers before the
// enclosing class is complete, which is ill-formed.
explicit EvidenceDiscounter(Calibrate cal)
: cal_(std::move(cal)), cfg_() {}
EvidenceDiscounter(Calibrate cal, Config cfg)
: cal_(std::move(cal)), cfg_(cfg) {}
/// Weight in [0,1] for one observation, updating `views` when the
/// observation is novel enough to count as a distinct look at the subject.
///
/// The first observation on a track always counts in full: there is nothing
/// for it to be redundant with.
float weight(std::vector<Embedding>& views, const Embedding& e) const {
if (views.empty()) {
views.push_back(e);
return 1.0f;
}
float p_same = 0.0f;
for (const auto& v : views)
p_same = std::max(p_same, cal_(cosine_similarity(v, e)));
// Weight is the probability this is *not* a repeat of something already
// counted. A near-duplicate contributes ~0; an unseen pose ~1.
const float w = std::max(cfg_.floor, 1.0f - p_same);
if (p_same < cfg_.admit_below &&
static_cast<int>(views.size()) < cfg_.max_views) {
views.push_back(e);
}
return w;
}
private:
Calibrate cal_;
Config cfg_;
};
src/face_utils.hpp
-1
View File
@@ -1,5 +1,4 @@
#pragma once
/// TRACES: AR-005 | SR-002
#include "types.hpp"
#include <opencv2/calib3d.hpp>
src/gallery/embedder_stamp.cpp
-326
View File
@@ -1,326 +0,0 @@
/// TRACES: GR-004 | SR-001
#include "embedder_stamp.hpp"
#include "types.hpp"
#include <algorithm>
#include <cstdlib>
#include <cstring>
#include <filesystem>
#include <fstream>
#include <iostream>
#include <map>
#include <mutex>
#include <sstream>
#include <stdexcept>
#include <vector>
namespace fs = std::filesystem;
// ── SHA-256 (FIPS 180-4) ──────────────────────────────────────────────────────
// Self-contained rather than pulled from OpenSSL: the gallery library already
// links OpenCV, HDF5, FFmpeg and a GPU backend, and the unit tests deliberately
// link none of those crypto stacks. ~80 lines of table-driven code is cheaper
// than another find_package that CI has to satisfy on an Intel N100.
namespace {
struct Sha256 {
uint32_t h[8] = {0x6a09e667u, 0xbb67ae85u, 0x3c6ef372u, 0xa54ff53au,
0x510e527fu, 0x9b05688cu, 0x1f83d9abu, 0x5be0cd19u};
uint64_t len = 0;
uint8_t buf[64]{};
size_t buf_n = 0;
static uint32_t ror(uint32_t x, int n) { return (x >> n) | (x << (32 - n)); }
void block(const uint8_t* p) {
static const uint32_t k[64] = {
0x428a2f98u,0x71374491u,0xb5c0fbcfu,0xe9b5dba5u,0x3956c25bu,0x59f111f1u,
0x923f82a4u,0xab1c5ed5u,0xd807aa98u,0x12835b01u,0x243185beu,0x550c7dc3u,
0x72be5d74u,0x80deb1feu,0x9bdc06a7u,0xc19bf174u,0xe49b69c1u,0xefbe4786u,
0x0fc19dc6u,0x240ca1ccu,0x2de92c6fu,0x4a7484aau,0x5cb0a9dcu,0x76f988dau,
0x983e5152u,0xa831c66du,0xb00327c8u,0xbf597fc7u,0xc6e00bf3u,0xd5a79147u,
0x06ca6351u,0x14292967u,0x27b70a85u,0x2e1b2138u,0x4d2c6dfcu,0x53380d13u,
0x650a7354u,0x766a0abbu,0x81c2c92eu,0x92722c85u,0xa2bfe8a1u,0xa81a664bu,
0xc24b8b70u,0xc76c51a3u,0xd192e819u,0xd6990624u,0xf40e3585u,0x106aa070u,
0x19a4c116u,0x1e376c08u,0x2748774cu,0x34b0bcb5u,0x391c0cb3u,0x4ed8aa4au,
0x5b9cca4fu,0x682e6ff3u,0x748f82eeu,0x78a5636fu,0x84c87814u,0x8cc70208u,
0x90befffau,0xa4506cebu,0xbef9a3f7u,0xc67178f2u};
uint32_t w[64];
for (int i = 0; i < 16; ++i)
w[i] = (uint32_t(p[i * 4]) << 24) | (uint32_t(p[i * 4 + 1]) << 16) |
(uint32_t(p[i * 4 + 2]) << 8) | uint32_t(p[i * 4 + 3]);
for (int i = 16; i < 64; ++i) {
uint32_t s0 = ror(w[i - 15], 7) ^ ror(w[i - 15], 18) ^ (w[i - 15] >> 3);
uint32_t s1 = ror(w[i - 2], 17) ^ ror(w[i - 2], 19) ^ (w[i - 2] >> 10);
w[i] = w[i - 16] + s0 + w[i - 7] + s1;
}
uint32_t a = h[0], b = h[1], c = h[2], d = h[3];
uint32_t e = h[4], f = h[5], g = h[6], hh = h[7];
for (int i = 0; i < 64; ++i) {
uint32_t S1 = ror(e, 6) ^ ror(e, 11) ^ ror(e, 25);
uint32_t ch = (e & f) ^ (~e & g);
uint32_t t1 = hh + S1 + ch + k[i] + w[i];
uint32_t S0 = ror(a, 2) ^ ror(a, 13) ^ ror(a, 22);
uint32_t mj = (a & b) ^ (a & c) ^ (b & c);
uint32_t t2 = S0 + mj;
hh = g; g = f; f = e; e = d + t1;
d = c; c = b; b = a; a = t1 + t2;
}
h[0] += a; h[1] += b; h[2] += c; h[3] += d;
h[4] += e; h[5] += f; h[6] += g; h[7] += hh;
}
void update(const uint8_t* p, size_t n) {
len += n;
while (n) {
size_t take = std::min(n, size_t(64) - buf_n);
std::memcpy(buf + buf_n, p, take);
buf_n += take; p += take; n -= take;
if (buf_n == 64) { block(buf); buf_n = 0; }
}
}
std::string hex() {
uint64_t bits = len * 8;
uint8_t pad = 0x80;
update(&pad, 1);
uint8_t zero = 0;
while (buf_n != 56) update(&zero, 1);
uint8_t tail[8];
for (int i = 0; i < 8; ++i) tail[i] = uint8_t(bits >> (56 - i * 8));
// update() would re-count these into len, but len is already frozen in bits.
std::memcpy(buf + buf_n, tail, 8);
block(buf);
buf_n = 0;
static const char* d = "0123456789abcdef";
std::string out;
out.reserve(64);
for (int i = 0; i < 8; ++i)
for (int s = 28; s >= 0; s -= 4)
out += d[(h[i] >> s) & 0xF];
return out;
}
};
// (path, mtime, size) → digest. Hashing a 250 MB ONNX is cheap but not free, and
// the optimizer constructs many networks in one process against the same model.
std::mutex g_hash_mu;
std::map<std::string, std::string> g_hash_cache;
std::string short_hash(const std::string& hex) {
return hex.size() > 12 ? hex.substr(0, 12) + "…" : hex;
}
} // namespace
std::string sha256_hex(const std::string& bytes) {
Sha256 s;
s.update(reinterpret_cast<const uint8_t*>(bytes.data()), bytes.size());
return s.hex();
}
std::string sha256_file_hex(const std::string& path) {
if (path.empty()) return "";
std::error_code ec;
auto size = fs::file_size(path, ec);
if (ec) return "";
auto mtime = fs::last_write_time(path, ec);
if (ec) return "";
std::ostringstream key;
key << path << '|' << size << '|'
<< mtime.time_since_epoch().count();
{
std::lock_guard<std::mutex> lk(g_hash_mu);
auto it = g_hash_cache.find(key.str());
if (it != g_hash_cache.end()) return it->second;
}
std::ifstream f(path, std::ios::binary);
if (!f) return "";
Sha256 s;
std::vector<char> chunk(1 << 20);
while (f) {
f.read(chunk.data(), static_cast<std::streamsize>(chunk.size()));
std::streamsize got = f.gcount();
if (got > 0) s.update(reinterpret_cast<const uint8_t*>(chunk.data()),
static_cast<size_t>(got));
}
std::string hex = s.hex();
std::lock_guard<std::mutex> lk(g_hash_mu);
g_hash_cache[key.str()] = hex;
return hex;
}
// ── EmbedderStamp ─────────────────────────────────────────────────────────────
std::string EmbedderStamp::describe() const {
std::string name = model_name.empty() ? "<unnamed model>" : model_name;
if (model_sha256.empty())
return name + " (sha256 unavailable)";
return name + " (sha256 " + short_hash(model_sha256) + ")";
}
EmbedderStamp make_embedder_stamp(const std::string& model_path) {
EmbedderStamp s;
if (model_path.empty()) return s;
s.model_name = fs::path(model_path).filename().string();
s.model_sha256 = sha256_file_hex(model_path);
if (s.model_sha256.empty())
std::cerr << "[gallery] cannot hash embedder model " << model_path
<< " — model binding falls back to filename only (GR-004)\n";
return s;
}
bool require_gallery_stamp_from_env() {
const char* v = std::getenv("SAE_REQUIRE_GALLERY_STAMP");
return v && *v && std::strcmp(v, "0") != 0;
}
// ── Comparison ────────────────────────────────────────────────────────────────
StampCheck compare_embedder_stamps(const EmbedderStamp& built_with,
const EmbedderStamp& loading_with,
const std::string& gallery_desc,
const std::string& embedder_desc) {
StampCheck out;
std::ostringstream m;
// The gallery predates GR-004 (or was written by a tool that does not stamp).
if (built_with.empty()) {
out.verdict = StampVerdict::unstamped;
m << "gallery '" << gallery_desc << "' carries no embedder stamp (GR-004).\n"
<< " gallery was built with : UNKNOWN — this file predates model binding\n"
<< " embedder now loaded : " << loading_with.describe()
<< " [" << embedder_desc << "]\n"
<< " If these are not the same model every similarity from this run is\n"
<< " meaningless but will look plausible. Rebuild or re-stamp the gallery\n"
<< " (scripts/stamp_gallery.py), or run with SAE_REQUIRE_GALLERY_STAMP=1 to\n"
<< " make this a hard error.";
out.message = m.str();
return out;
}
// Gallery is stamped but we cannot say what is about to embed.
if (loading_with.empty()) {
out.verdict = StampVerdict::unknown_embedder;
m << "cannot identify the embedder being used against gallery '"
<< gallery_desc << "' (GR-004).\n"
<< " gallery was built with : " << built_with.describe() << "\n"
<< " embedder now loaded : UNKNOWN [" << embedder_desc << "]\n"
<< " The binding cannot be checked, so it is not being checked.";
out.message = m.str();
return out;
}
const bool have_both_hashes =
!built_with.model_sha256.empty() && !loading_with.model_sha256.empty();
// Embedding width disagreeing is a mismatch on its own terms — different
// spaces entirely, and it will not even be caught by a cosine that "looks fine".
if (built_with.embed_dim != loading_with.embed_dim) {
out.verdict = StampVerdict::mismatch;
m << "gallery/embedder MODEL MISMATCH — refusing to run (GR-004).\n"
<< " gallery was built with : " << built_with.describe()
<< ", dim=" << built_with.embed_dim << " [" << gallery_desc << "]\n"
<< " embedder now loaded : " << loading_with.describe()
<< ", dim=" << loading_with.embed_dim << " [" << embedder_desc << "]\n"
<< " Embedding dimensions differ; these are not the same space.";
out.message = m.str();
return out;
}
if (have_both_hashes) {
if (built_with.model_sha256 == loading_with.model_sha256) {
out.verdict = StampVerdict::match;
m << "embedder binding verified: " << built_with.describe();
if (built_with.model_name != loading_with.model_name)
m << " (gallery recorded it as '" << built_with.model_name
<< "', loaded from '" << loading_with.model_name
<< "' — same bytes, renamed file)";
out.message = m.str();
return out;
}
out.verdict = StampVerdict::mismatch;
m << "gallery/embedder MODEL MISMATCH — refusing to run (GR-004).\n"
<< " gallery was built with : " << built_with.model_name
<< " sha256=" << built_with.model_sha256 << "\n"
<< " [" << gallery_desc << "]\n"
<< " embedder now loaded : " << loading_with.model_name
<< " sha256=" << loading_with.model_sha256 << "\n"
<< " [" << embedder_desc << "]\n"
<< " Cosine similarities between embeddings from different models are\n"
<< " meaningless but look plausible. Rebuild the gallery with the loaded\n"
<< " model, or point the embedder at the model the gallery was built with.";
out.message = m.str();
return out;
}
// One side has no hash (e.g. a TRT deployment with the .onnx absent). Names
// are all we have; agreeing on them is evidence, not proof.
if (!built_with.model_name.empty() &&
built_with.model_name == loading_with.model_name) {
out.verdict = StampVerdict::weak_match;
m << "embedder binding UNPROVEN for gallery '" << gallery_desc << "' (GR-004).\n"
<< " gallery was built with : " << built_with.describe() << "\n"
<< " embedder now loaded : " << loading_with.describe()
<< " [" << embedder_desc << "]\n"
<< " Filenames agree but at least one SHA-256 is unavailable, so an\n"
<< " in-place re-export under the same name would not be detected.";
out.message = m.str();
return out;
}
out.verdict = StampVerdict::mismatch;
m << "gallery/embedder MODEL MISMATCH — refusing to run (GR-004).\n"
<< " gallery was built with : " << built_with.describe()
<< " [" << gallery_desc << "]\n"
<< " embedder now loaded : " << loading_with.describe()
<< " [" << embedder_desc << "]\n"
<< " Cosine similarities between embeddings from different models are\n"
<< " meaningless but look plausible. Rebuild the gallery with the loaded\n"
<< " model, or point the embedder at the model the gallery was built with.";
out.message = m.str();
return out;
}
void enforce_embedder_stamp(const EmbedderStamp& built_with,
const EmbedderStamp& loading_with,
const std::string& gallery_desc,
const std::string& embedder_desc,
bool require_stamp) {
const bool strict = require_stamp || require_gallery_stamp_from_env();
StampCheck chk = compare_embedder_stamps(built_with, loading_with,
gallery_desc, embedder_desc);
if (chk.fatal(strict)) {
if (chk.verdict != StampVerdict::mismatch)
throw std::runtime_error(chk.message +
"\n (fatal because SAE_REQUIRE_GALLERY_STAMP / --require-gallery-stamp is set)");
throw std::runtime_error(chk.message);
}
if (chk.verdict == StampVerdict::match) {
std::cerr << "[gallery] " << chk.message << "\n";
} else {
std::cerr << "\n[gallery] ***** WARNING (GR-004) *****\n"
<< chk.message << "\n"
<< "[gallery] ****************************\n\n";
}
}
void verify_gallery_embedder(const ActorGallery& gallery,
const std::string& gallery_path,
const std::string& arcface_model_path,
bool require_stamp) {
enforce_embedder_stamp(gallery.embedder,
make_embedder_stamp(arcface_model_path),
gallery_path,
arcface_model_path.empty() ? "no --arcface given"
: arcface_model_path,
require_stamp);
}
src/gallery/embedder_stamp.hpp
-115
View File
@@ -1,115 +0,0 @@
#pragma once
/// TRACES: GR-004 | SR-001
//
// Gallery ↔ embedder binding.
//
// A gallery is only valid for the embedder that built it. Cosine similarities
// between embeddings from two different models are meaningless but *look*
// plausible — nothing crashes, nothing is obviously wrong, and every number
// measured downstream is quietly garbage. So the embedder's identity is stamped
// into the gallery at build time and checked by every consumer at load time.
//
// ── What identifies an embedder ───────────────────────────────────────────────
// Two fields, carried together:
//
// model_name basename of the model file, e.g. "LVFace-B_Glint360K.onnx"
// model_sha256 hex SHA-256 of that file's bytes
//
// The hash is what *decides*; the name is what a human *reads*. Neither alone is
// enough:
//
// • A name alone is a promise, not a fact. Models get re-exported, re-quantised
// and overwritten in place under an unchanged filename — which is precisely
// the case where the weights differ and nothing else does. A name-only stamp
// is blind to exactly the failure it exists to catch.
// • A hash alone is correct but unreadable: "expected 3f2a… got 9c1b…" tells an
// operator nothing about what to do next.
//
// SHA-256 over the file bytes is derived from the artefact rather than asserted
// about it, is stable across machines and filesystems, and needs no registry to
// be kept up to date. Cost is ~0.1 s for a 250 MB ONNX, paid once per process
// (results are memoised on path+mtime+size), which is noise next to model load.
//
// ── Degraded and legacy cases ─────────────────────────────────────────────────
// A TRT-backend deployment may run from a prebuilt .engine with the source .onnx
// absent, so the hash cannot be computed. Then the name is compared alone and the
// result is reported as a *weak* match — believed, not proven.
//
// Galleries built before GR-004 carry no stamp at all. They warn loudly rather
// than fail, because the state is unknown rather than known-bad, and because
// hard-failing every pre-existing gallery would make the check something people
// route around rather than trust. Set require_stamp (or SAE_REQUIRE_GALLERY_STAMP=1)
// to promote "unknown" to a hard error — that is the mode measurement work runs in.
//
// A *mismatch* is always fatal, in every mode, with no bypass.
#include <cstdint>
#include <string>
struct EmbedderStamp {
std::string model_name; // basename of the model file
std::string model_sha256; // lowercase hex SHA-256 of the file's bytes ("" = unavailable)
int32_t embed_dim{512};
bool empty() const { return model_name.empty() && model_sha256.empty(); }
// "LVFace-B_Glint360K.onnx (sha256 3f2a1c4d…)" — for error messages.
std::string describe() const;
};
// Identify the model at `model_path`. Missing/unreadable file → name filled from
// the path, hash left empty (the weak-match path). Empty path → empty stamp.
EmbedderStamp make_embedder_stamp(const std::string& model_path);
enum class StampVerdict {
match, // hashes agree — binding proven
weak_match, // names agree, no hash on one side — believed, unproven
unstamped, // gallery predates GR-004 / was written without a stamp
unknown_embedder, // gallery is stamped but the loaded embedder can't be identified
mismatch, // proven different models — always fatal
};
struct StampCheck {
StampVerdict verdict{StampVerdict::match};
std::string message; // human-readable, names BOTH sides
// A mismatch is fatal unconditionally. The three "cannot prove it" verdicts
// are fatal only in strict mode.
bool fatal(bool require_stamp) const {
return verdict == StampVerdict::mismatch ||
(require_stamp && verdict != StampVerdict::match);
}
};
// Pure comparison — no file I/O, no model loading. This is the unit under test.
// `gallery_desc`/`embedder_desc` are only used to make the message locatable
// (a gallery path, a dump path, "the embedder being loaded", …).
StampCheck compare_embedder_stamps(const EmbedderStamp& built_with,
const EmbedderStamp& loading_with,
const std::string& gallery_desc = "gallery",
const std::string& embedder_desc = "embedder");
// Apply the comparison: throw std::runtime_error on a fatal verdict, otherwise
// log to stderr. `require_stamp` is OR-ed with SAE_REQUIRE_GALLERY_STAMP.
void enforce_embedder_stamp(const EmbedderStamp& built_with,
const EmbedderStamp& loading_with,
const std::string& gallery_desc,
const std::string& embedder_desc,
bool require_stamp);
// Convenience for the common consumer shape: "I loaded this gallery and I am
// about to embed with this model file." Hashes the model, then enforces.
struct ActorGallery;
void verify_gallery_embedder(const ActorGallery& gallery,
const std::string& gallery_path,
const std::string& arcface_model_path,
bool require_stamp);
// SAE_REQUIRE_GALLERY_STAMP=1 → treat an unprovable binding as fatal.
bool require_gallery_stamp_from_env();
// Lowercase hex SHA-256. Exposed so a test can pin the digest against the
// published vectors, which is what guarantees the C++ and Python (hashlib)
// stamps of the same file agree.
std::string sha256_hex(const std::string& bytes);
std::string sha256_file_hex(const std::string& path); // "" if unreadable
src/gallery/gallery_builder.cpp
-7
View File
@@ -1,6 +1,5 @@
#include "gallery_builder.hpp"
#include "config.hpp"
#include "embedder_stamp.hpp"
#include "face_utils.hpp"
#include "inference/face_detector.hpp"
#include "inference/face_embedder.hpp"
@@ -42,12 +41,6 @@ ActorGallery build_gallery(const BuildConfig& cfg) {
ActorGallery gallery;
/// TRACES: GR-004 | SR-001
// Stamp before the first embedding exists, so there is no window in which a
// gallery holds vectors without recording what produced them.
gallery.embedder = make_embedder_stamp(cfg.arcface_model);
std::cerr << "[build_gallery] embedder: " << gallery.embedder.describe() << "\n";
for (const auto& actor_dir : fs::directory_iterator(cfg.gallery_root)) {
if (!actor_dir.is_directory()) continue;
src/gallery/gallery_calibration.hpp
-29
View File
@@ -1,5 +1,4 @@
#pragma once
/// TRACES: AR-023 | SR-002
#include "types.hpp"
#include <algorithm>
@@ -9,7 +8,6 @@
#include <cstdlib>
#include <cstring>
#include <fstream>
#include <functional>
#include <iostream>
#include <stdexcept>
#include <string>
@@ -50,33 +48,6 @@ struct GalleryCalibration {
}
};
/// TRACES: AR-023, AR-024 | SR-002
///
/// cosine → P(same person). The one probability space the pipeline reasons in.
///
/// Handed to every stage that has to decide whether two embeddings are the same
/// person — track association (AR-007), evidence discounting (AR-025), identity
/// matching — so a threshold of 0.5 means the same thing in all of them. A stage
/// that thresholded a raw cosine instead would be using a number that means
/// something different for every model, gallery and face size (AR-024).
///
/// **No prior term.** `log_prior_odds` adjusts for the gallery's base rate, which
/// is a question about *which of N actors*; association asks whether two faces
/// are one person, where the balanced fit is the right answer. Passing the
/// matcher's prior here would silently bias tracking by the size of the cast.
inline std::function<float(float)> same_person_probability(const GalleryCalibration& cal) {
if (!cal.valid) {
// Loud, because the failure mode is invisible: an untuned sigmoid still
// returns plausible probabilities, and every threshold downstream of it
// is then a guess wearing a calibrated number's clothes.
std::cerr << "[calibration] WARNING: no fitted calibration — association and "
"evidence weighting fall back to the untuned default sigmoid "
"(a=" << cal.a << ", b=" << cal.b << "). Probabilities are "
"not meaningful for this embedder.\n";
}
return [cal](float similarity) { return cal.probability(similarity); };
}
// Fit a logistic sigmoid to gallery pair similarities.
// Positive pairs: same actor, different reference images.
// Negative pairs: different actors (all cross-actor embedding pairs).
src/gallery/gallery_store.cpp
-40
View File
@@ -79,21 +79,6 @@ static ActorGallery load_gallery_hdf5(const std::string& path) {
gallery.actors.push_back(std::move(actor));
}
/// TRACES: GR-004 | SR-001
// Absent /embedder group == a gallery written before model binding existed.
// It stays readable; verify_gallery_embedder() decides what that means.
if (file.nameExists("embedder")) {
H5::Group eg = file.openGroup("embedder");
H5::StrType str(H5::PredType::C_S1, H5T_VARIABLE);
if (eg.attrExists("model_name"))
eg.openAttribute("model_name").read(str, gallery.embedder.model_name);
if (eg.attrExists("model_sha256"))
eg.openAttribute("model_sha256").read(str, gallery.embedder.model_sha256);
if (eg.attrExists("embed_dim"))
eg.openAttribute("embed_dim").read(H5::PredType::NATIVE_INT32,
&gallery.embedder.embed_dim);
}
if (file.nameExists("calibration")) {
H5::Group cal = file.openGroup("calibration");
cal.openAttribute("a").read(H5::PredType::NATIVE_FLOAT, &gallery.calib_a);
@@ -164,20 +149,6 @@ static void save_gallery_hdf5(const std::string& path, const ActorGallery& galle
write_str_dataset(file, "name", name);
write_str_dataset(file, "source_images", src_images);
/// TRACES: GR-004 | SR-001
// Bind the file to the embedder that produced its vectors. Written only when
// known — an empty stamp must round-trip as "unstamped", not as a stamp
// claiming an unnamed model.
if (!gallery.embedder.empty()) {
H5::Group eg = file.createGroup("embedder");
H5::DataSpace scalar(H5S_SCALAR);
H5::StrType str(H5::PredType::C_S1, H5T_VARIABLE);
eg.createAttribute("model_name", str, scalar).write(str, gallery.embedder.model_name);
eg.createAttribute("model_sha256", str, scalar).write(str, gallery.embedder.model_sha256);
eg.createAttribute("embed_dim", H5::PredType::NATIVE_INT32, scalar)
.write(H5::PredType::NATIVE_INT32, &gallery.embedder.embed_dim);
}
if (gallery.calib_hash != 0) {
H5::Group cal = file.createGroup("calibration");
H5::DataSpace scalar(H5S_SCALAR);
@@ -215,17 +186,6 @@ ActorGallery load_gallery(const std::string& path) {
<< std::chrono::duration<double>(t1 - t0).count() << "s\n";
ActorGallery gallery;
/// TRACES: GR-004 | SR-001
// Optional top-level "embedder" object, matching the HDF5 /embedder group.
// Written by the JSON-era helper scripts; absent in anything older.
if (j.contains("embedder") && j.at("embedder").is_object()) {
const auto& je = j.at("embedder");
gallery.embedder.model_name = je.value("model_name", "");
gallery.embedder.model_sha256 = je.value("model_sha256", "");
gallery.embedder.embed_dim = je.value("embed_dim", 512);
}
for (const auto& ja : j.at("actors")) {
ActorGallery::Actor actor;
actor.imdb_id = ja.value("imdb_id", "");
src/gallery/gallery_store.hpp
-6
View File
@@ -12,11 +12,6 @@
// /count int32 [A] number of refs for actor a
// /imdb_id /tmdb_id /jellyfin_id /name : variable-length string [A]
// /source_images : variable-length string [N], parallel to /embeddings rows
// /embedder/model_name : scalar var-len string attr — embedder file basename
// /embedder/model_sha256 : scalar var-len string attr — SHA-256 of that file
// /embedder/embed_dim : scalar int32 attr
// The GR-004 model binding. Absent group == unstamped
// (pre-GR-004 file); see gallery/embedder_stamp.hpp.
// /calibration/a, /b : scalar float32 attrs — Platt-sigmoid P(match|sim) fit
// /calibration/valid : scalar int8 attr (0/1)
// /calibration/hash : scalar uint64 attr — hash of the embeddings the fit
@@ -24,7 +19,6 @@
//
// Legacy JSON format (read-only):
// {
// "embedder": {"model_name": "...", "model_sha256": "...", "embed_dim": 512},
// "actors": [
// {
// "imdb_id": "nm0000093", // optional, "" if unknown
src/kpn_bindings.cpp
+2 -28
View File
@@ -15,7 +15,6 @@
#include "types.hpp"
#include "config.hpp"
#include "gallery/embedder_stamp.hpp"
#include "gallery/gallery_store.hpp"
#include "nodes/face_tracker_node.hpp"
#include "nodes/identity_matcher_node.hpp"
@@ -164,9 +163,6 @@ static Config config_from_dict(nb::dict d) {
getd("anneal_sec", cfg.anneal_sec);
// gallery expansion (usually off for sweeps; expose so it can be toggled)
if (d.contains("expand_gallery")) cfg.expand_gallery = nb::cast<bool>(d["expand_gallery"]);
/// TRACES: GR-004 | SR-001
if (d.contains("require_gallery_stamp"))
cfg.require_gallery_stamp = nb::cast<bool>(d["require_gallery_stamp"]);
return cfg;
}
@@ -214,17 +210,8 @@ NB_MODULE(sae_kpn, m) {
net.add(std::move(name), std::move(node));
}, "net"_a, "name"_a, "config"_a, "capacity"_a = 16);
/// TRACES: GR-004 | SR-001
// embedder_model / embedder_sha256 identify whatever produced the embeddings
// that will be fed in. In a replay those come from the dump's own stamp (see
// scripts/optimizer/SCHEMA.md), because there is no live embedder in the
// network — the dump *is* the embedder as far as this gallery is concerned.
// Passing neither leaves the binding unverifiable, which warns loudly and is
// fatal under SAE_REQUIRE_GALLERY_STAMP.
m.def("add_identity_matcher", [](Net& net, std::string name, std::string gallery_path,
nb::dict cfg_dict, std::size_t cap,
std::string embedder_model,
std::string embedder_sha256) {
nb::dict cfg_dict, std::size_t cap) {
Config cfg = config_from_dict(cfg_dict);
cfg.gallery_path = gallery_path; // needed to persist refreshed calibration back
// Cache loaded galleries by path so a threshold sweep (many networks, same
@@ -235,24 +222,11 @@ NB_MODULE(sae_kpn, m) {
if (it == cache.end())
it = cache.emplace(gallery_path,
std::make_shared<ActorGallery>(load_gallery(gallery_path))).first;
// Checked on every construction, not only on the cache miss: the same
// process may replay several dumps against one cached gallery.
EmbedderStamp feeding;
feeding.model_name = std::move(embedder_model);
feeding.model_sha256 = std::move(embedder_sha256);
enforce_embedder_stamp(it->second->embedder, feeding, gallery_path,
feeding.model_name.empty()
? "embeddings fed into this network"
: feeding.model_name,
cfg.require_gallery_stamp);
auto node = std::make_shared<kpn::ObjectVariantNodeWrapper<
IdentityMatcherFunc, SaeVariant, kpn::in<"tracked">, kpn::out<"matched">>>(
cap, *it->second, cfg);
net.add(std::move(name), std::move(node));
}, "net"_a, "name"_a, "gallery"_a, "config"_a, "capacity"_a = 16,
"embedder_model"_a = "", "embedder_sha256"_a = "");
}, "net"_a, "name"_a, "gallery"_a, "config"_a, "capacity"_a = 16);
m.def("add_scene_tracker", [](Net& net, std::string name, nb::dict cfg_dict, std::size_t cap) {
Config cfg = config_from_dict(cfg_dict);
src/main.cpp
+7 -57
View File
@@ -1,9 +1,5 @@
// scene_analyze — identify actors in a movie using a KPN pipeline
//
// TRACES: DP-001, DP-002 | PR-004
// One analysis core; the CLI is a front-end over it and must not fork pipeline
// logic. Other deployment modes (DP-003, DP-004) wrap this same core.
//
// KPN topology (release build):
//
// [frame_source] ──Frame──► [face_detector] ──SceneFrame──► [face_aligner]
@@ -49,7 +45,6 @@
#include "config.hpp"
#include "types.hpp"
#include "gallery/embedder_stamp.hpp"
#include "gallery/gallery_store.hpp"
#include "nodes/frame_source_node.hpp"
#include "nodes/camera_position_change_detector_node.hpp"
@@ -119,7 +114,6 @@ static Config parse_args(int argc, char** argv) {
else if (arg("--detector")) cfg.detector_model = next();
else if (arg("--detector-engine")) cfg.detector_engine = next();
else if (arg("--arcface")) cfg.arcface_model = next();
else if (arg("--require-gallery-stamp")) cfg.require_gallery_stamp = true;
else if (arg("--arcface-engine")) cfg.arcface_engine = next();
else if (arg("--conf")) cfg.detector_conf = std::stof(next());
else if (arg("--max-faces")) cfg.max_faces = std::stoi(next());
@@ -128,8 +122,10 @@ static Config parse_args(int argc, char** argv) {
else if (arg("--ratio-ceil")) cfg.match_ratio_ceil = std::stof(next());
else if (arg("--track-alpha")) cfg.track_alpha = std::stof(next());
else if (arg("--track-min-iou")) cfg.track_min_iou = std::stof(next());
else if (arg("--track-min-prob")) cfg.track_assoc_min_prob = std::stof(next());
else if (arg("--track-extinction")) cfg.track_extinction_sec = std::stod(next());
else if (arg("--track-max-embed")) cfg.track_max_embed_dist = std::stof(next());
else if (arg("--track-max-missing")) cfg.track_max_frames_missing = std::stoi(next());
else if (arg("--cut-revive-sim")) cfg.cut_revive_sim = std::stof(next());
else if (arg("--cut-inactive-max")) cfg.cut_inactive_max_frames = std::stoi(next());
else if (arg("--anneal")) cfg.anneal_sec = std::stod(next());
else if (arg("--expand-gallery")) cfg.expand_gallery = true;
else if (arg("--expand-buffer")) cfg.expand_buffer_size = std::stoi(next());
@@ -171,11 +167,6 @@ int main(int argc, char** argv) {
ActorGallery gallery;
try {
gallery = load_gallery(cfg.gallery_path);
/// TRACES: GR-004 | SR-001
// Hard startup error before a single frame is decoded: a gallery built
// with another embedder yields plausible-looking, meaningless matches.
verify_gallery_embedder(gallery, cfg.gallery_path, cfg.arcface_model,
cfg.require_gallery_stamp);
} catch (const std::exception& e) {
std::cerr << "Gallery error: " << e.what() << "\n";
return 1;
@@ -192,36 +183,10 @@ int main(int argc, char** argv) {
FaceDetectorFunc detector_fn{cfg};
FaceAlignerFunc aligner_fn;
EmbedderFunc embedder_fn{cfg};
// Constructed before the tracker: it fits (or loads) the calibration, and
// the tracker must decide in that same probability space (AR-024).
FaceTrackerFunc ftracker_fn{cfg};
IdentityMatcherFunc matcher_fn {gallery, cfg};
/// TRACES: AR-007, AR-008, AR-012, AR-024 | SR-002
// The registry is created here and shared, not owned by a node: track state
// is not a stage in the stream, it is state several stages read and write,
// and its final answer is only known when a track dies.
auto same_person = same_person_probability(matcher_fn.calibration());
TrackRegistry::Config reg_cfg;
reg_cfg.extinction_sec = cfg.track_extinction_sec;
auto registry = std::make_shared<TrackRegistry>(
reg_cfg, EvidenceDiscounter(same_person));
matcher_fn.set_registry(registry);
FaceTrackerFunc ftracker_fn{cfg, registry, same_person};
SceneTrackerFunc tracker_fn {cfg};
ResultSinkFunc sink_fn {cfg, done};
/// TRACES: AR-012, AR-016, IR-002, IR-003 | SR-002
// A reaped track goes straight to the aggregator, so the registry holds only
// live tracks and its size is bounded by concurrent on-screen faces rather
// than growing with the film.
registry->on_track_dead([&sink_fn](const DeadTrack& d) { sink_fn.add_claim(d); });
// AR-016: a film ends with faces on screen and those tracks have not timed
// out. Without this flush the closing scene's cast is silently never
// emitted — a loss that reads as a recognition miss, not a bookkeeping bug.
sink_fn.set_pre_write_hook([registry](double last_ts) { registry->flush(last_ts); });
#ifdef SAE_DEBUG
DebugRendererFunc debug_fn {cfg};
#endif
@@ -282,30 +247,15 @@ int main(int argc, char** argv) {
net.stop();
net.print_diagnostics();
/// TRACES: AR-004 | SR-002
// A dropped frame does not degrade a result, it silently changes one —
// the output is a claim about footage that was never analysed, and
// nothing in the file says so. Since AR-004 made data pushes block, a
// drop can no longer happen on the data path, so any drop here means
// either that fix regressed (it lives in the KPN submodule, one line,
// easy to lose in an update) or a channel was disabled mid-run.
//
// Reporting it in a footer and exiting 0 made both invisible: the run
// "succeeded" and the truth file looked complete. Fail instead.
bool dropped = false;
{
std::lock_guard<std::mutex> lk(event_mtx);
if (!overflow_counts.empty()) {
dropped = true;
std::cerr << "[main] ERROR: frames were dropped (channel overflow):\n";
std::cerr << "[main] dropped frames (channel overflow):\n";
for (const auto& [name, count] : overflow_counts)
std::cerr << " " << name << ": " << count << "\n";
std::cerr << "[main] The output would describe footage that was never "
"analysed. Refusing to report success.\n";
}
}
if (node_crashed.load(std::memory_order_acquire)) return 1;
return dropped ? 2 : 0;
return node_crashed.load(std::memory_order_acquire) ? 1 : 0;
};
// ── Build static network and run ──────────────────────────────────────────
src/nodes/embedding_dump_node.hpp
+2 -14
View File
@@ -1,8 +1,6 @@
#pragma once
/// TRACES: VR-001 | PR-002
#include "types.hpp"
#include "config.hpp"
#include "gallery/embedder_stamp.hpp"
#include <H5Cpp.h>
@@ -27,13 +25,7 @@ struct EmbeddingDumpFunc {
: path_(cfg.dump_embeddings_path), movie_(cfg.movie_path),
sample_fps_(cfg.sample_fps), done_(done)
{
/// TRACES: GR-004 | SR-001
// A dump is a bag of embeddings with no model attached, replayed against a
// gallery hours or weeks later — the same silent cross-model hazard as the
// gallery itself, so it carries the same stamp.
stamp_ = make_embedder_stamp(cfg.arcface_model);
std::cerr << "[embedding_dump] writing " << path_
<< " embedder: " << stamp_.describe() << "\n";
std::cerr << "[embedding_dump] writing " << path_ << "\n";
}
void operator()(EmbeddedSceneFrame ef) {
@@ -99,9 +91,6 @@ private:
H5::StrType str(H5::PredType::C_S1, H5T_VARIABLE);
auto mv = file.createAttribute("movie", str, scalar);
mv.write(str, movie_);
/// TRACES: GR-004 | SR-001
file.createAttribute("embedder_model", str, scalar).write(str, stamp_.model_name);
file.createAttribute("embedder_sha256", str, scalar).write(str, stamp_.model_sha256);
H5::Group frames = file.createGroup("frames");
write_vec(frames, "timestamp_sec", ts_, H5::PredType::NATIVE_DOUBLE);
@@ -121,8 +110,7 @@ private:
<< conf_.size() << " faces → " << path_ << "\n";
}
std::string path_, movie_;
EmbedderStamp stamp_;
std::string path_, movie_;
float sample_fps_;
std::atomic<bool>& done_;
std::atomic<bool> written_{false};
src/nodes/face_detector_node.hpp
-1
View File
@@ -1,5 +1,4 @@
#pragma once
/// TRACES: AR-001 | SR-002
#include "config.hpp"
#include "inference/face_detector.hpp"
src/nodes/face_tracker_node.hpp
+164 -171
View File
@@ -1,124 +1,102 @@
#pragma once
/// TRACES: AR-007, AR-008, AR-024 | SR-002
///
/// FaceTrackerFunc — KPN node that links face detections into tracks.
///
/// **The registry is the tracker's state.** The node owns no track map of its
/// own: it drives `TrackRegistry` through a `FrameScope` and reads the same
/// `Track` objects everything else reads. Two parallel copies could disagree,
/// and every divergence would surface as a wrong presence window rather than as
/// a crash — silently, and only in the output.
///
/// **One candidate pool** (AR-008). `last_seen` alone distinguishes a track that
/// is on screen from one that is dormant, and it only affects whether IoU means
/// anything. There is no parked pool and no revival branch: re-associating a
/// track whose face was lost — across a cut or not — is ordinary inter-frame
/// association, and it falls out of the embedding comparison already being done.
///
/// Assignment cost (track i, detection j):
///
/// p = P(same person | cosine(track mean, detection)) ← calibrated
/// alpha = base weight, or 0 when position carries no information
/// cost = alpha·(1 − IoU) + (1 − alpha)·(1 − p)
///
/// gated to INF unless the pair is admissible on position *or* on identity.
///
/// **alpha is frame- and track-dependent** (AR-007). It falls to 0 —
/// embedding only — when either:
/// - the frame is flagged `is_cut` / `is_scene_boundary`: the viewpoint
/// changed, so the same person is at a new position; or
/// - the track is dormant (`last_seen` set): time has passed since its box was
/// last observed, so that box is stale regardless of cuts.
/// Both are the same statement — spatial continuity is broken — arrived at from
/// two directions, which is why they collapse into one rule rather than two
/// branches.
///
/// **Everything is thresholded in probability space** (AR-024). The cosine goes
/// through the calibration before it is compared to anything; the raw-cosine
/// constants `track_max_embed_dist` and `cut_revive_sim` are retired.
#include "types.hpp"
#include "config.hpp"
#include "track_registry.hpp"
#include <algorithm>
#include <cmath>
#include <functional>
#include <iostream>
#include <limits>
#include <map>
#include <memory>
#include <stdexcept>
#include <string_view>
#include <vector>
// ── FaceTrackerFunc ───────────────────────────────────────────────────────────
// KPN node: links face detections across consecutive frames using the Hungarian
// algorithm on a combined spatial (IoU) + embedding (cosine distance) cost.
//
// Each track accumulates a running directional mean of its ArcFace embeddings
// (averaged then re-normalised to the unit sphere), used as the embedding side
// of the assignment cost below for more stable track continuity.
//
// Assignment cost (track i, detection j):
// cost = alpha * (1 - IoU) + (1-alpha) * min(cosine_dist/2, 1)
// Gated to INF when IoU < min_iou AND cosine_dist > max_embed_dist.
//
// Unmatched tracks have their frames_missing counter incremented; they are
// expired once frames_missing > max_frames_missing.
//
// Cross-cut re-association. A camera-angle change (Frame::is_cut, set by
// camera_position_change_detector) destroys spatial (IoU) continuity — the same
// person reappears at a new position — but not identity. On a cut the tracker
// does NOT discard its tracks; it parks them in an inactive pool keyed by their
// last-frame raw embedding. A post-cut detection whose raw cosine similarity to
// a parked track's last-frame embedding is ≥ cut_revive_sim revives that track:
// the original track_id, mean embedding and n_frames are restored (only the bbox
// jumps to the new detection), so identity continuity survives the cut. Parked
// tracks left unrevived for cut_inactive_max_frames are finally dropped.
struct FaceTrackerFunc {
static constexpr std::string_view label() { return "face_tracker"; }
/// cosine similarity → P(same person). Supplied by the caller so the fit
/// belonging to the active embedder is used (AR-023/AR-024) — the same
/// pattern, and normally the same function object, as
/// `EvidenceDiscounter::Calibrate`.
using Calibrate = std::function<float(float)>;
struct TrackState {
cv::Rect2f bbox;
Embedding mean_emb{};
Embedding last_emb{}; // raw embedding of the most recent matched frame
int n_frames{0};
int frames_missing{0};
};
/// The registry is a constructor argument, not an option: a tracker without
/// one would have to keep its own tracks, which is the defect this replaces.
FaceTrackerFunc(const Config& cfg,
std::shared_ptr<TrackRegistry> registry,
Calibrate calibrate)
: registry_(std::move(registry))
, calibrate_(std::move(calibrate))
, alpha_base_(cfg.track_alpha)
explicit FaceTrackerFunc(const Config& cfg)
: alpha_(cfg.track_alpha)
, min_iou_(cfg.track_min_iou)
, min_assoc_prob_(cfg.track_assoc_min_prob)
, max_embed_dist_(cfg.track_max_embed_dist)
, max_missing_(cfg.track_max_frames_missing)
, revive_sim_(cfg.cut_revive_sim)
, inactive_max_(cfg.cut_inactive_max_frames)
{
if (!registry_)
throw std::invalid_argument("face_tracker: registry must not be null");
if (!calibrate_)
throw std::invalid_argument("face_tracker: a calibration is required — "
"association is decided in probability space");
std::cerr << "[face_tracker] alpha_base=" << alpha_base_
std::cerr << "[face_tracker] alpha=" << alpha_
<< " min_iou=" << min_iou_
<< " min_assoc_prob=" << min_assoc_prob_ << "\n";
<< " max_embed_dist=" << max_embed_dist_
<< " max_missing=" << max_missing_
<< " cut_revive_sim=" << revive_sim_
<< " cut_inactive_max=" << inactive_max_ << "\n";
}
TrackedSceneFrame operator()(EmbeddedSceneFrame ef) {
if (ef.source.eof) {
// Deliberately does *not* flush the registry. The identity matcher
// runs downstream and its votes for the final frames are still in
// flight; reaping here would drop them (they would land on ids that
// no longer exist and show up as dropped_votes). AR-016's flush
// belongs at the pipeline's termination point, after the last vote.
boxes_.clear();
tracks_.clear();
inactive_.clear();
TrackedSceneFrame out;
out.source = std::move(ef.source);
return out;
}
const double t = ef.source.timestamp_sec;
const int n_det = static_cast<int>(ef.embeddings.size());
const int n_det = static_cast<int>(ef.embeddings.size());
// Unconditional: the clock must advance on frames with no detections
// too, or a track only dies when some unrelated face happens to appear
// and a film that ends mid-track never closes it (AR-013).
auto scope = registry_->begin_frame(t);
// Camera-angle change: park active tracks instead of destroying them so
// they can be revived by identity (raw last-frame embedding cosine) once
// the same people reappear from the new angle.
if (ef.source.is_cut && !tracks_.empty()) {
std::cerr << "[face_tracker] cut — parking " << tracks_.size()
<< " track(s) into inactive pool\n";
for (auto& [tid, ts] : tracks_) {
ts.frames_missing = 0; // repurpose as time-since-parked counter
inactive_[tid] = std::move(ts);
}
tracks_.clear();
}
// One pool (AR-008) — on-screen and dormant tracks compete together.
std::vector<Track*> cands = scope.candidates();
const int n_trk = static_cast<int>(cands.size());
// Age the inactive pool every frame and drop tracks parked too long.
for (auto it = inactive_.begin(); it != inactive_.end(); ) {
it->second.frames_missing++;
it = (it->second.frames_missing > inactive_max_)
? inactive_.erase(it) : std::next(it);
}
prune_boxes(cands);
std::vector<Spatial*> sp(n_trk);
for (int ti = 0; ti < n_trk; ++ti)
sp[ti] = &boxes_.try_emplace(cands[ti]->id, Spatial{{}, t, false})
.first->second;
// AR-007 — the frame half of the frame-dependent weighting. Both flags
// say the same thing to the tracker: whatever was at that position is
// not there any more.
const bool viewpoint_change =
ef.source.is_cut || ef.source.is_scene_boundary;
// Snapshot active track IDs so the map can be modified safely below
std::vector<int> tids;
tids.reserve(tracks_.size());
for (auto& [tid, _] : tracks_) tids.push_back(tid);
const int n_trk = static_cast<int>(tids.size());
// ── Cost matrix [n_trk × n_det] ──────────────────────────────────────
constexpr float INF_COST = 1e6f;
@@ -126,108 +104,109 @@ struct FaceTrackerFunc {
std::vector<float>(n_det, INF_COST));
for (int ti = 0; ti < n_trk; ++ti) {
// Spatial continuity holds only for a track that was on screen, whose
// box we have actually observed, on a frame that did not change the
// viewpoint. Otherwise the box is stale and IoU is noise.
const bool spatial_meaningful =
sp[ti]->observed && cands[ti]->on_screen() && !viewpoint_change;
const float alpha = spatial_meaningful ? alpha_base_ : 0.f;
const TrackState& ts = tracks_[tids[ti]];
for (int di = 0; di < n_det; ++di) {
// AR-024 — the cosine is converted before it is used for
// anything, including the gate below.
const float p = calibrate_(
cosine_similarity(cands[ti]->mean, ef.embeddings[di]));
const float iou_v = spatial_meaningful
? iou(sp[ti]->bbox, ef.faces[di].bbox) : 0.f;
// Either signal on its own can admit a link: a face that moved a
// little but whose embedding degraded (blur, profile turn) is
// still linkable on position, and a face that jumped across the
// frame is still linkable on identity. Neither ⇒ no link.
const bool spatial_ok = spatial_meaningful && iou_v >= min_iou_;
const bool identity_ok = p >= min_assoc_prob_;
if (!spatial_ok && !identity_ok) continue;
cost[ti][di] = alpha * (1.f - iou_v) + (1.f - alpha) * (1.f - p);
float iou_v = iou(ts.bbox, ef.faces[di].bbox);
float emb_d = (ts.n_frames > 0)
? 1.f - cosine_similarity(ts.mean_emb, ef.embeddings[di])
: 1.f;
if (iou_v < min_iou_ && emb_d > max_embed_dist_) continue;
float s = 1.f - iou_v;
float e = std::min(emb_d * 0.5f, 1.f);
cost[ti][di] = alpha_ * s + (1.f - alpha_) * e;
}
}
// ── Hungarian assignment ─────────────────────────────────────────────
// ── Hungarian assignment ──────────────────────────────────────────────
std::vector<int> assign(n_trk, -1);
if (n_trk > 0 && n_det > 0)
assign = hungarian(cost, n_trk, n_det);
// ── Build output frame ───────────────────────────────────────────────
// ── Build output frame ────────────────────────────────────────────────
TrackedSceneFrame out;
out.source = ef.source;
out.faces = ef.faces;
out.crops = ef.crops;
out.embeddings = ef.embeddings;
out.source = ef.source;
out.faces = ef.faces;
out.crops = ef.crops;
out.embeddings = ef.embeddings;
out.track_ids.assign(n_det, -1);
std::vector<bool> det_matched(n_det, false);
// Update matched tracks
for (int ti = 0; ti < n_trk; ++ti) {
const int di = assign[ti];
const int id = cands[ti]->id;
const bool valid =
(di >= 0 && di < n_det && cost[ti][di] < INF_COST * 0.5f);
int di = assign[ti];
bool valid = (di >= 0 && di < n_det && cost[ti][di] < INF_COST * 0.5f);
TrackState& ts = tracks_[tids[ti]];
if (!valid) {
// Only a track that *was* on screen can become lost, and it
// becomes lost as of its last sighting, never as of now — the
// gap after the final sighting is never claimed (AR-013). A
// track already dormant is left alone so its extinction clock
// keeps running from the right instant.
if (cands[ti]->on_screen()) scope.mark_lost(id, sp[ti]->last_ts);
ts.frames_missing++;
continue;
}
update_mean(ts.mean_emb, ts.n_frames, ef.embeddings[di]);
ts.last_emb = ef.embeddings[di];
ts.bbox = ef.faces[di].bbox;
ts.n_frames++;
ts.frames_missing = 0;
det_matched[di] = true;
out.track_ids[di] = tids[ti];
}
// Handle unmatched detections: first try to revive a parked track by
// identity (raw last-frame embedding cosine), else start a fresh track.
for (int di = 0; di < n_det; ++di) {
if (det_matched[di]) continue;
int tid = revive_from_inactive(ef.embeddings[di]);
if (tid >= 0) {
// Restore the parked track: keep its identity statistics
// (mean_emb, n_frames), jump the bbox to the new detection.
TrackState ts = std::move(inactive_[tid]);
inactive_.erase(tid);
update_mean(ts.mean_emb, ts.n_frames, ef.embeddings[di]);
ts.last_emb = ef.embeddings[di];
ts.bbox = ef.faces[di].bbox;
ts.n_frames++;
ts.frames_missing = 0;
tracks_[tid] = std::move(ts);
out.track_ids[di] = tid;
std::cerr << "[face_tracker] revived track " << tid
<< " across cut\n";
continue;
}
scope.mark_seen(id, t, ef.embeddings[di]);
sp[ti]->bbox = ef.faces[di].bbox;
sp[ti]->last_ts = t;
sp[ti]->observed = true;
det_matched[di] = true;
out.track_ids[di] = id;
tid = next_id_++;
TrackState ts;
ts.bbox = ef.faces[di].bbox;
ts.mean_emb = ef.embeddings[di];
ts.last_emb = ef.embeddings[di];
ts.n_frames = 1;
tracks_[tid] = ts;
out.track_ids[di] = tid;
}
for (int di = 0; di < n_det; ++di) {
if (det_matched[di]) continue;
const int id = scope.create(t, ef.embeddings[di]);
boxes_[id] = Spatial{ef.faces[di].bbox, t, true};
out.track_ids[di] = id;
// Expire stale tracks
for (auto it = tracks_.begin(); it != tracks_.end(); ) {
it = (it->second.frames_missing > max_missing_)
? tracks_.erase(it) : std::next(it);
}
// No reaping here: begin_frame's tick owns the extinction sweep, so
// there is exactly one place a track can die.
return out;
}
private:
// ── Spatial annotation ───────────────────────────────────────────────────
// The one piece of per-track state the registry does not hold, because it is
// not about presence: where the face was, and when it was last seen there.
// Keyed by registry track id and pruned against `candidates()` every frame,
// so it cannot outlive or contradict the registry — it annotates the pool
// rather than duplicating it.
struct Spatial {
cv::Rect2f bbox{};
double last_ts{0.0}; ///< timestamp of the last frame this track matched
bool observed{false}; ///< false until a detection has been assigned
};
// Drop boxes for ids the registry no longer has. `candidates()` is the
// authority on what exists; anything else is a leak (and, for a reused id,
// would be a stale box attached to a different person).
void prune_boxes(const std::vector<Track*>& cands) {
if (boxes_.size() == cands.size()) return; // common case: nothing died
std::map<int, Spatial> kept;
for (const Track* t : cands) {
auto it = boxes_.find(t->id);
if (it != boxes_.end()) kept.emplace(t->id, it->second);
// Pick the parked track whose last-frame embedding is most similar to emb,
// returning its id if that raw cosine similarity clears revive_sim_, else -1.
// The caller removes the returned track from the pool, so a later detection in
// the same frame cannot claim it again.
int revive_from_inactive(const Embedding& emb) const {
int best_tid = -1;
float best_sim = revive_sim_; // threshold is the bar to beat (inclusive)
for (const auto& [tid, ts] : inactive_) {
float sim = cosine_similarity(ts.last_emb, emb);
if (sim >= best_sim) { best_sim = sim; best_tid = tid; }
// subsequent ties keep the later id; harmless, all clear the threshold
}
boxes_.swap(kept);
return best_tid;
}
// IoU of two axis-aligned bounding boxes
@@ -241,6 +220,17 @@ private:
return inter / (a.width * a.height + b.width * b.height - inter);
}
// Online directional mean: average then re-normalise to unit sphere
static void update_mean(Embedding& mean, int n_prev, const Embedding& emb) {
float norm_sq = 0.f;
for (int k = 0; k < 512; ++k) {
mean[k] = (mean[k] * n_prev + emb[k]) / (n_prev + 1);
norm_sq += mean[k] * mean[k];
}
float inv = 1.f / std::sqrt(norm_sq);
for (int k = 0; k < 512; ++k) mean[k] *= inv;
}
// O(n³) potential-based Hungarian algorithm (Jonker-Volgenant / Kuhn-Munkres).
// Returns assign[row] = col (0-indexed), or -1 when row is matched to a
// padded virtual column (i.e., unmatched). Rectangular matrices are padded
@@ -302,10 +292,13 @@ private:
return ans;
}
std::shared_ptr<TrackRegistry> registry_;
Calibrate calibrate_;
std::map<int, Spatial> boxes_; ///< track id → where it was, when
float alpha_base_;
std::map<int, TrackState> tracks_;
std::map<int, TrackState> inactive_; // parked across a cut, keyed by track id
int next_id_{0};
float alpha_;
float min_iou_;
float min_assoc_prob_;
float max_embed_dist_;
int max_missing_;
float revive_sim_;
int inactive_max_;
};
src/nodes/identity_matcher_node.hpp
-29
View File
@@ -5,7 +5,6 @@
#include "gallery/gallery_store.hpp"
#include "gallery/gallery_calibration.hpp"
#include "gallery/track_gallery.hpp"
#include "track_registry.hpp"
#include <cstdint>
#include <cstring>
@@ -108,20 +107,6 @@ struct IdentityMatcherFunc {
sim_engine_ = make_similarity_engine(host_gallery.data(), n_gallery_, kMaxFaces);
}
/// TRACES: AR-023, AR-024 | SR-002
/// The fitted sigmoid. Exposed because the matcher is where it gets fitted
/// (and cached back to the gallery), but it is not the matcher's private
/// property: track association and evidence weighting must threshold in the
/// *same* probability space, or a "0.5" in one stage and a "0.5" in another
/// mean different things. See `same_person_probability`.
const GalleryCalibration& calibration() const { return cal_; }
/// TRACES: AR-012, AR-025 | SR-002
/// Where per-frame identity evidence reaches the registry. Optional: with no
/// registry attached the matcher behaves exactly as before, which keeps the
/// replay harness and the unit tests working unchanged.
void set_registry(std::shared_ptr<TrackRegistry> r) { registry_ = std::move(r); }
// Runtime setter — lets a persistent pipeline be reused across a threshold sweep
// without rebuilding the (expensive, gallery-resident) matcher. The gallery,
// calibration and GPU sim-engine stay put; only the accept threshold changes.
@@ -239,19 +224,6 @@ struct IdentityMatcherFunc {
// face (annex already folded in above); the track's diversity buffer
// keeps the gallery-far views and promotes them once the track is
// confirmed. No-op unless --expand-gallery is set.
// TRACES: AR-012, AR-025 | SR-002
// Every scored face is evidence, not only the accepted ones: a run of
// near-misses for one actor is itself informative, and discarding it
// would make ownership depend on a per-frame threshold the redesign
// exists to stop relying on. The registry discounts for correlation
// and decides ownership from the accumulated posterior (AR-025).
if (registry_ && best_actor >= 0 && tf.track_ids[fi] >= 0) {
const float p = cal_.valid
? cal_.probability(best_s, log_prior_odds_)
: std::max(0.f, best_s);
registry_->observe(tf.track_ids[fi], best_actor, p, tf.embeddings[fi]);
}
track_gallery_.observe(tf.track_ids[fi], tf.embeddings[fi],
best_actor, best_s, accept, tf.crops[fi]);
@@ -275,5 +247,4 @@ private:
std::unique_ptr<ISimilarityEngine> sim_engine_;
TrackGallery track_gallery_;
std::shared_ptr<TrackRegistry> registry_;
};
src/nodes/result_sink_node.hpp
+34 -86
View File
@@ -1,8 +1,6 @@
#pragma once
/// TRACES: IR-001 | SR-003
#include "types.hpp"
#include "config.hpp"
#include "track_registry.hpp"
#include <nlohmann/json.hpp>
#include <algorithm>
@@ -11,8 +9,6 @@
#include <fstream>
#include <iostream>
#include <map>
#include <mutex>
#include <functional>
#include <string>
#include <vector>
@@ -46,26 +42,10 @@ using json = nlohmann::json;
struct ResultSinkFunc {
static constexpr std::string_view label() { return "result_sink"; }
/// TRACES: AR-012, AR-017, IR-002 | SR-002, SR-003
/// A finished presence claim from the registry. Called from inside the
/// registry's reap while it holds its own lock, so this must stay a cheap
/// push and must never re-enter the registry.
void add_claim(const DeadTrack& d) {
if (d.actor_idx < 0) return; // never owned: nothing to claim
std::lock_guard<std::mutex> g(claims_mu_);
claims_.push_back(d);
}
ResultSinkFunc(const Config& cfg, std::atomic<bool>& done)
: cfg_(cfg), done_(done)
{}
/// TRACES: AR-016 | SR-002
/// Runs immediately before the output is written, with the last timestamp
/// seen. Used to flush tracks still live at EOF, which have not timed out
/// and would otherwise never be emitted.
void set_pre_write_hook(std::function<void(double)> fn) { pre_write_ = std::move(fn); }
void operator()(SceneAnnotation sa) {
if (sa.eof) {
flush();
@@ -78,20 +58,12 @@ struct ResultSinkFunc {
<< " unknowns=" << count_unknown(sa.visible_actors)
<< std::flush;
for (const auto& ia : sa.visible_actors) {
if (ia.actor_idx < 0) continue;
auto& m = actor_meta_[ia.actor_idx];
if (m.name.empty())
m = {ia.name, ia.imdb_id, ia.tmdb_id, ia.jellyfin_id};
}
last_ts_ = sa.timestamp_sec;
frames_.push_back(std::move(sa));
}
// Write accumulated results and signal done. Safe to call more than once.
void flush() {
if (written_.exchange(true)) return;
if (pre_write_) pre_write_(last_ts_);
write_output();
done_.store(true, std::memory_order_release);
}
@@ -99,7 +71,7 @@ struct ResultSinkFunc {
private:
// Bump when the minimal/standard output JSON structure changes in a way
// the Jellyfin plugin needs to detect.
static constexpr int kSchemaVersion = 2; // SR-003 coordinated bump
static constexpr int kSchemaVersion = 1;
static int count_known(const std::vector<IdentifiedActor>& v) {
int n = 0;
@@ -119,19 +91,10 @@ private:
if (cfg_.verbosity == Verbosity::xray) {
root = build_xray();
} else {
/// TRACES: IR-002 | SR-003
/// schema_version 2, per jRay/SPEC.md JR-002. anneal_sec is REMOVED
/// rather than zeroed: a field naming a mechanism the pipeline no
/// longer has is actively misleading, and would outlive everyone who
/// remembers why it reads 0. extinction_sec succeeds it as the
/// parameter that actually shapes window extent.
root["schema_version"] = kSchemaVersion;
root["movie"] = cfg_.movie_path;
root["extraction"] = {
{"sample_fps", cfg_.sample_fps},
{"extinction_sec", cfg_.track_extinction_sec},
{"gallery_scope", cfg_.gallery_scope},
};
root["movie"] = cfg_.movie_path;
root["sample_fps"] = cfg_.sample_fps;
root["anneal_sec"] = cfg_.anneal_sec;
root["actors"] = build_epochs();
if (cfg_.verbosity == Verbosity::standard)
root["frames"] = build_standard();
@@ -146,45 +109,42 @@ private:
std::cerr << "[result_sink] done.\n";
}
struct Window {
double start{0.0};
double end{0.0};
float belief{0.f}; ///< the posterior that justified the claim (AR-017)
};
struct ActorWindow {
std::string name, imdb_id, tmdb_id, jellyfin_id;
std::vector<Window> scenes;
std::vector<std::pair<double, double>> scenes; // [start_sec, end_sec]
};
struct ActorMeta { std::string name, imdb_id, tmdb_id, jellyfin_id; };
// Core logic: merge per-frame detections into annealed [start, end] windows.
/// TRACES: AR-012, IR-002 | SR-002
/// A claim already IS a window — `[first_seen, last_seen]` of a track the
/// actor owned. There is no annealing pass: `anneal_sec` existed to bridge
/// gaps between isolated accepted frames, and a track that survives its own
/// gaps leaves it nothing to do (see the AR-012 withdrawal note).
std::vector<ActorWindow> build_actor_windows() {
std::lock_guard<std::mutex> g(claims_mu_);
struct Info { std::string name, imdb_id, tmdb_id, jellyfin_id; };
std::map<int, Info> actor_info;
std::map<int, std::vector<double>> timestamps;
std::map<int, ActorWindow> by_actor;
for (const auto& c : claims_) {
auto& aw = by_actor[c.actor_idx];
if (aw.name.empty()) {
auto it = actor_meta_.find(c.actor_idx);
if (it != actor_meta_.end()) {
aw.name = it->second.name;
aw.imdb_id = it->second.imdb_id;
aw.tmdb_id = it->second.tmdb_id;
aw.jellyfin_id = it->second.jellyfin_id;
}
for (const auto& frame : frames_) {
for (const auto& ia : frame.visible_actors) {
if (ia.actor_idx < 0) continue;
actor_info[ia.actor_idx] = {ia.name, ia.imdb_id, ia.tmdb_id, ia.jellyfin_id};
timestamps[ia.actor_idx].push_back(frame.timestamp_sec);
}
aw.scenes.push_back({c.first_seen, c.last_seen, c.belief});
}
std::vector<ActorWindow> result;
for (auto& [idx, aw] : by_actor) {
std::sort(aw.scenes.begin(), aw.scenes.end(),
[](const Window& a, const Window& b) { return a.start < b.start; });
for (auto& [idx, ts_vec] : timestamps) {
ActorWindow aw;
aw.name = actor_info[idx].name;
aw.imdb_id = actor_info[idx].imdb_id;
aw.tmdb_id = actor_info[idx].tmdb_id;
aw.jellyfin_id = actor_info[idx].jellyfin_id;
double win_start = ts_vec[0], win_end = ts_vec[0];
for (size_t i = 1; i < ts_vec.size(); ++i) {
if (ts_vec[i] - win_end > cfg_.anneal_sec) {
aw.scenes.push_back({win_start, win_end});
win_start = ts_vec[i];
}
win_end = ts_vec[i];
}
aw.scenes.push_back({win_start, win_end});
result.push_back(std::move(aw));
}
return result;
@@ -193,16 +153,9 @@ private:
json build_epochs() {
json actors = json::array();
for (const auto& aw : build_actor_windows()) {
// Objects, not float pairs: a window carries the belief that
// justified it and the route by which it was identified (AR-017),
// so a consumer can caveat or filter rather than treating every
// window as equally certain.
json windows = json::array();
for (const auto& w : aw.scenes)
windows.push_back({{"start", w.start},
{"end", w.end},
{"belief", w.belief},
{"route", "live"}});
for (const auto& [s, e] : aw.scenes)
windows.push_back({s, e});
json ja;
ja["name"] = aw.name;
ja["imdb_id"] = aw.imdb_id;
@@ -220,9 +173,9 @@ private:
json build_xray() {
std::map<int, std::vector<std::string>> xray;
for (const auto& aw : build_actor_windows()) {
for (const auto& w : aw.scenes) {
int t0 = static_cast<int>(std::floor(w.start));
int t1 = static_cast<int>(std::ceil(w.end));
for (const auto& [start, end] : aw.scenes) {
int t0 = static_cast<int>(std::floor(start));
int t1 = static_cast<int>(std::ceil(end));
for (int t = t0; t <= t1; ++t)
xray[t].push_back(aw.name);
}
@@ -273,9 +226,4 @@ private:
std::atomic<bool>& done_;
std::atomic<bool> written_{false};
std::vector<SceneAnnotation> frames_;
std::function<void(double)> pre_write_;
double last_ts_{0.0};
std::mutex claims_mu_;
std::vector<DeadTrack> claims_;
std::map<int, ActorMeta> actor_meta_; ///< actor_idx → identity keys
};
src/scene_preview.cpp
+1 -8
View File
@@ -21,7 +21,6 @@
#include "config.hpp"
#include "types.hpp"
#include "gallery/embedder_stamp.hpp"
#include "gallery/gallery_store.hpp"
#include "nodes/frame_source_node.hpp"
#include "nodes/camera_position_change_detector_node.hpp"
@@ -77,7 +76,6 @@ static Config parse_args(int argc, char** argv) {
else if (arg("--detector-engine")) cfg.detector_engine = next();
else if (arg("--arcface")) cfg.arcface_model = next();
else if (arg("--arcface-engine")) cfg.arcface_engine = next();
else if (arg("--require-gallery-stamp")) cfg.require_gallery_stamp = true;
else if (arg("--conf")) cfg.detector_conf = std::stof(next());
else if (arg("--max-faces")) cfg.max_faces = std::stoi(next());
else if (arg("--min-face-px")) cfg.min_face_px = std::stof(next());
@@ -128,12 +126,7 @@ int main(int argc, char** argv) {
}
ActorGallery gallery;
try {
gallery = load_gallery(cfg.gallery_path);
/// TRACES: GR-004 | SR-001
verify_gallery_embedder(gallery, cfg.gallery_path, cfg.arcface_model,
cfg.require_gallery_stamp);
}
try { gallery = load_gallery(cfg.gallery_path); }
catch (const std::exception& e) {
std::cerr << "Gallery error: " << e.what() << "\n";
return 1;
src/track_registry.hpp
-335
View File
@@ -1,335 +0,0 @@
#pragma once
/// TRACES: AR-012, AR-013, AR-014, AR-015, AR-016, AR-017 | SR-002
///
/// TrackRegistry — the single owner of track state and of presence.
///
/// Presence follows **track extent**, not per-frame recognition (AR-012): a
/// window is `[first_seen, last_seen]` of a track an actor owns, so it starts
/// when the actor appeared rather than when the recogniser first succeeded.
///
/// `last_seen` carries the entire liveness state (AR-013):
///
/// unset → on screen now
/// set → went off screen at that timestamp, still revivable
/// reaped → emitted to the aggregator and erased
///
/// There is no missing-frame counter and no expired flag; the optional *is* the
/// state machine, and it subsumes what was previously a two-pool split in the
/// tracker (active vs. parked-across-a-cut).
///
/// **Interior gaps are claimed, the trailing cool-down is not.** A face lost at
/// t1 and re-associated at t2 within the timeout never closed its track, so the
/// actor is present across [t1, t2] — correct, since someone briefly occluded or
/// off-camera has not left the scene. But a track that dies ends its window at
/// `last_seen`, never at the moment of death. That asymmetry is what removes the
/// over-claim the retired `extinction_sec` keep-alive produced.
///
/// The registry is created in `main` and shared by `shared_ptr`; it is *not* a
/// KPN node. Ownership is not a stage in the stream — it is state several stages
/// read and write, whose final answer is only known when a track dies.
#include "types.hpp"
#include "evidence_discount.hpp"
#include <cmath>
#include <cstdint>
#include <functional>
#include <map>
#include <mutex>
#include <optional>
#include <string>
#include <vector>
// ── DeadTrack ────────────────────────────────────────────────────────────────
// A finished presence claim, emitted exactly once when a track is reaped or
// flushed. Immutable by construction: it carries everything needed to justify
// itself (AR-017), with no back-reference into registry state.
struct DeadTrack {
int track_id{-1};
double first_seen{0.0};
double last_seen{0.0}; ///< always the last sighting, never the death time
int actor_idx{-1}; ///< -1 when the track was never owned
float belief{0.0f}; ///< accumulated posterior for actor_idx
int observations{0}; ///< evidence updates that landed on this track
float effective_obs{0.f}; ///< sum of discounted weights — the evidence that counted
};
// ── Track ────────────────────────────────────────────────────────────────────
struct Track {
int id{-1};
double first_seen{0.0};
std::optional<double> last_seen; ///< unset ⇒ on screen
std::optional<int> actor; ///< set once a posterior crosses
std::map<int, float> belief; ///< actor_idx → accumulated log-odds
Embedding mean{}; ///< running directional mean
std::vector<Embedding> views; ///< distinct looks, for AR-025 discounting
float discounted_weight{0.f}; ///< sum of applied weights
int n_obs{0};
bool on_screen() const { return !last_seen.has_value(); }
};
// ── TrackRegistry ────────────────────────────────────────────────────────────
class TrackRegistry {
public:
using DeadTrackFn = std::function<void(const DeadTrack&)>;
struct Config {
double extinction_sec{5.0}; ///< how long a lost track stays revivable
float ownership_logodds{2.0f}; ///< belief needed to own a track (~0.88 posterior)
};
/// The discounter is a constructor argument rather than an option: there is
/// no correct way to accumulate per-frame evidence without it.
TrackRegistry(Config cfg, EvidenceDiscounter discounter)
: cfg_(cfg), discounter_(std::move(discounter)) {}
void on_track_dead(DeadTrackFn fn) { on_dead_ = std::move(fn); }
// ── Frame scope ──────────────────────────────────────────────────────────
// The tracker mutates registry state across a whole association pass, so
// that pass must be atomic as a unit — per-call locking would let another
// thread observe a half-updated frame. FrameScope holds the lock for its
// lifetime and exposes the mutating operations without re-locking.
class FrameScope {
public:
FrameScope(TrackRegistry& reg, double now)
: reg_(reg), lock_(reg.mu_) { reg_.tick_locked(now); }
/// All live tracks — **one pool**. `last_seen` tells the caller whether
/// IoU is meaningful; a dormant track is matched on embedding alone.
/// There is no separate revival path (AR-008).
std::vector<Track*> candidates() {
std::vector<Track*> out;
out.reserve(reg_.tracks_.size());
for (auto& [id, t] : reg_.tracks_) out.push_back(&t);
return out;
}
int create(double t, const Embedding& e) { return reg_.create_locked(t, e); }
void mark_seen(int id, double t, const Embedding& e){ reg_.mark_seen_locked(id, t, e); }
void mark_lost(int id, double last_on_screen) { reg_.mark_lost_locked(id, last_on_screen); }
private:
TrackRegistry& reg_;
std::unique_lock<std::mutex> lock_;
};
FrameScope begin_frame(double now) { return FrameScope(*this, now); }
/// Advance the clock and reap. Called every sampled frame **whether or not
/// it had detections** — without it a track only dies when some other face
/// happens to appear, and a film ending mid-track never closes.
void tick(double now) { std::lock_guard g(mu_); tick_locked(now); }
// ── Evidence ─────────────────────────────────────────────────────────────
/// Fold one observation into a track's belief (AR-025).
///
/// `posterior` is a **calibrated probability**, never a raw cosine
/// (AR-024) — the registry converts it to log-odds itself, so the
/// accumulation cannot be fed an uncalibrated number by a careless caller.
///
/// Correlation discounting is applied **here**, not by the caller.
/// Consecutive frames of one track are near-identical, and accumulating
/// them as independent evidence drives the posterior to certainty on what
/// is effectively a single measurement. Leaving that to callers would mean
/// a forgotten or doubly-applied discount produces confident wrong answers
/// silently; the registry is the one place all evidence converges, so it is
/// the one place the correction belongs.
///
/// A vote for a track that has already been reaped is dropped and counted:
/// a nonzero `dropped_votes()` means the timeout is shorter than the
/// matcher's lag, which is a real misconfiguration and must not be silent.
void observe(int track_id, int actor_idx, float posterior, const Embedding& e) {
std::lock_guard g(mu_);
auto it = tracks_.find(track_id);
if (it == tracks_.end()) { ++dropped_votes_; return; }
Track& t = it->second;
const float w = discounter_.weight(t.views, e);
t.belief[actor_idx] += w * logit(posterior);
t.discounted_weight += w;
++t.n_obs;
const int best = argmax_belief(t);
const float best_lo = t.belief[best];
if (best_lo < cfg_.ownership_logodds) return;
if (!t.actor.has_value()) {
claim_locked(t, best);
return;
}
if (*t.actor != best) {
// AR-014 — belief swapped A→B. Not a correction: a track_id almost
// certainly carried across a viewpoint change onto a different
// person. Two non-twins both clearing the threshold on one face is
// not realistic; a track spanning two people is. Continuing would
// emit one window blending both, so close here and start afresh.
split_locked(t, best);
}
}
/// Snapshot read: tally and verdict under one lock. Reading them separately
/// would let a track be both unowned and owned within a single promotion
/// decision, since the matcher may be voting concurrently.
std::optional<int> owner(int track_id) const {
std::lock_guard g(mu_);
auto it = tracks_.find(track_id);
return it == tracks_.end() ? std::nullopt : it->second.actor;
}
// ── Termination ──────────────────────────────────────────────────────────
/// Emit every still-live track and empty the registry (AR-016). A film ends
/// with faces on screen and those tracks have not timed out, so without this
/// the closing scene's cast is silently never emitted — a loss that presents
/// as a recognition miss rather than a bookkeeping bug.
///
/// Idempotent: calling it twice emits nothing the second time.
void flush(double final_ts) {
std::lock_guard g(mu_);
for (auto& [id, t] : tracks_) emit_locked(t, t.last_seen.value_or(final_ts));
tracks_.clear();
}
// ── Diagnostics ──────────────────────────────────────────────────────────
// These measure how often tracking is silently wrong, which nothing in the
// pipeline currently reveals.
int dropped_votes() const { std::lock_guard g(mu_); return dropped_votes_; }
int belief_swaps() const { std::lock_guard g(mu_); return belief_swaps_; }
int actor_conflicts() const { std::lock_guard g(mu_); return actor_conflicts_; }
std::size_t live() const { std::lock_guard g(mu_); return tracks_.size(); }
private:
// ── Locked internals ─────────────────────────────────────────────────────
void tick_locked(double now) {
for (auto it = tracks_.begin(); it != tracks_.end(); ) {
const auto& ls = it->second.last_seen;
if (ls && (now - *ls) > cfg_.extinction_sec) {
emit_locked(it->second, *ls);
it = tracks_.erase(it);
} else {
++it;
}
}
}
int create_locked(double t, const Embedding& e) {
const int id = next_id_++;
Track tr;
tr.id = id;
tr.first_seen = t;
tr.mean = e;
tr.n_obs = 0;
tracks_.emplace(id, std::move(tr));
return id;
}
void mark_seen_locked(int id, double t, const Embedding& e) {
auto it = tracks_.find(id);
if (it == tracks_.end()) return;
Track& tr = it->second;
tr.last_seen.reset(); // back on screen; the gap is absorbed
update_mean(tr, e);
(void)t;
}
void mark_lost_locked(int id, double last_on_screen) {
auto it = tracks_.find(id);
if (it == tracks_.end()) return;
it->second.last_seen = last_on_screen;
}
void claim_locked(Track& t, int actor) {
// AR-015 — if another live track already owns this actor, at least one
// is wrong: a person cannot be in two places at once. The cause is the
// same as a belief swap — a missed camera or scene change. Detected on
// the update that causes it via the reverse index, not by scanning.
auto seen = owner_index_.find(actor);
if (seen != owner_index_.end() && seen->second != t.id
&& tracks_.count(seen->second)) {
++actor_conflicts_;
}
t.actor = actor;
owner_index_[actor] = t.id;
}
void split_locked(Track& t, int new_actor) {
++belief_swaps_;
const double boundary = t.last_seen.value_or(t.first_seen);
emit_locked(t, boundary);
// The successor inherits the embedding and the belief that caused the
// swap, and starts at the swap frame — so the two windows abut without
// overlapping and neither blends the two people.
Track next;
next.id = next_id_++;
next.first_seen = boundary;
next.mean = t.mean;
next.belief[new_actor] = t.belief[new_actor];
next.n_obs = 1;
const int old_id = t.id;
Track stash = std::move(next);
tracks_.erase(old_id);
const int nid = stash.id;
tracks_.emplace(nid, std::move(stash));
claim_locked(tracks_.at(nid), new_actor);
}
void emit_locked(Track& t, double end_ts) {
if (!on_dead_) return;
DeadTrack d;
d.track_id = t.id;
d.first_seen = t.first_seen;
d.last_seen = end_ts;
d.observations = t.n_obs;
d.effective_obs = t.discounted_weight;
if (t.actor) {
d.actor_idx = *t.actor;
d.belief = logistic(t.belief[*t.actor]);
auto oi = owner_index_.find(*t.actor);
if (oi != owner_index_.end() && oi->second == t.id) owner_index_.erase(oi);
}
on_dead_(d);
}
static int argmax_belief(const Track& t) {
int best = -1;
float hi = -1e30f;
for (const auto& [a, lo] : t.belief) if (lo > hi) { hi = lo; best = a; }
return best;
}
static void update_mean(Track& t, const Embedding& e) {
// Directional mean: accumulate then re-normalise to the unit sphere, so
// cosine against it stays a plain dot product.
double norm = 0.0;
for (int i = 0; i < 512; ++i) {
t.mean[i] = t.mean[i] * static_cast<float>(t.n_obs ? t.n_obs : 1) + e[i];
norm += static_cast<double>(t.mean[i]) * t.mean[i];
}
norm = norm > 0 ? std::sqrt(norm) : 1.0;
for (int i = 0; i < 512; ++i) t.mean[i] = static_cast<float>(t.mean[i] / norm);
}
static float logistic(float z) {
return z >= 0 ? 1.f / (1.f + std::exp(-z))
: std::exp(z) / (1.f + std::exp(z));
}
static float logit(float p) {
const float eps = 1e-6f;
p = std::min(1.f - eps, std::max(eps, p));
return std::log(p / (1.f - p));
}
Config cfg_;
EvidenceDiscounter discounter_;
mutable std::mutex mu_;
std::map<int, Track> tracks_;
std::map<int, int> owner_index_; ///< actor_idx → live track_id (AR-015)
DeadTrackFn on_dead_;
int next_id_{0};
int dropped_votes_{0};
int belief_swaps_{0};
int actor_conflicts_{0};
};
src/types.hpp
-7
View File
@@ -6,8 +6,6 @@
#include <opencv2/core.hpp>
#include "gallery/embedder_stamp.hpp"
// ── Embedding ─────────────────────────────────────────────────────────────────
// 512-dim L2-normalised ArcFace embedding
using Embedding = std::array<float, 512>;
@@ -138,11 +136,6 @@ struct ActorGallery {
};
std::vector<Actor> actors;
/// TRACES: GR-004 | SR-001
// Which embedder produced every embedding above. Empty == the file predates
// model binding; see gallery/embedder_stamp.hpp for what is checked and why.
EmbedderStamp embedder;
// Cached Platt-sigmoid calibration (see gallery/gallery_calibration.hpp),
// stored alongside the gallery in HDF5 so it never needs recomputing
// unless the reference embeddings actually change. calib_valid=false and
tests/CMakeLists.txt
+1 -12
View File
@@ -21,32 +21,21 @@ add_executable(sae_tests
test_face_utils.cpp
test_track_gallery.cpp
test_face_tracker.cpp
test_track_registry.cpp
test_replay_fixtures.cpp
test_audio_signature.cpp
${CMAKE_SOURCE_DIR}/src/backends/gemm_backend.cpp
${CMAKE_SOURCE_DIR}/src/gallery/gallery_store.cpp
${CMAKE_SOURCE_DIR}/src/audio_signature.cpp
${CMAKE_SOURCE_DIR}/src/gallery/embedder_stamp.cpp
)
target_include_directories(sae_tests PRIVATE ${CMAKE_SOURCE_DIR}/src)
# SAE_GEMM_CPU: build the CPU reference GEMM regardless of the main backend.
# SAE_MODELS_DIR: config.hpp (pulled in by track_gallery.hpp) bakes model paths.
# SAE_TEST_FIXTURES_DIR: the audio golden vector is read from the source tree,
# not copied, so the file the plugin repo shares is the file under test.
target_compile_definitions(sae_tests PRIVATE
SAE_GEMM_CPU
SAE_MODELS_DIR="${SAE_MODELS_DIR}"
SAE_TEST_FIXTURES_DIR="${CMAKE_CURRENT_SOURCE_DIR}/fixtures")
SAE_MODELS_DIR="${SAE_MODELS_DIR}")
# gallery_store.cpp + gallery_calibration.hpp use nlohmann/json and HDF5
# (galleries are HDF5-native, see src/gallery/gallery_store.cpp); face_utils.hpp
# and the calibration GEMM pull in OpenCV (calib3d/imgproc/core) via types.hpp.
# ffmpeg_libs: audio_signature.cpp decodes the golden fixture (avformat/avcodec/
# avutil/swresample). Still GPU-free — the audio path is pure CPU.
target_link_libraries(sae_tests PRIVATE
Catch2::Catch2WithMain
nlohmann_json::nlohmann_json
ffmpeg_libs
${OpenCV_LIBS}
${HDF5_CXX_LIBRARIES})
target_include_directories(sae_tests PRIVATE ${HDF5_INCLUDE_DIRS})
tests/fixtures/audio/jray_audio_v1_golden.json
Vendored
-194
View File
@@ -1,194 +0,0 @@
{
"_": "Golden vector for the JRay v1 audio signature (JRay-public-server SPEC.md \u00a73). Shared verbatim between scene-actor-extraction (C++) and the jRay Jellyfin plugin (C#) so the two implementations can be proven bit-identical. IR-004, IR-005, IR-007, IR-008.",
"version": "v1",
"signature": "v1:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAeHx8eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh8fHzk5OTk5OTk5OTk5OTk5OTk5OTk5OTk5OTk5OTk5V1dXV1dXV1dXV1dXV1dXV1dXV1dXV1dXV1dXV1dXV1dXV1dycnJycnJycnJycnJycnJycnJycnJycnJycnJycnMPDgwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKytFRUVFRUVFRUVFRUVFRUVFRUVFRUVFRUVFRUVFRWNjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2Njfn5+fn5+fn5+fn5+fn5+fn5+fn5+fn5+fn5+fn5/GxoZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGRkZGTc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3Nzc3UlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSU1JsbGxsbGxsbGxsbGxsbGxsbGxsbGxsbGxsbGxsbAoLCwoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCwsLJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSVDQ0NDQ0NDQ0NDQ0NDQ0NDQ0NDQ0NDQ0NDQ0NDQ0NDQ0NDQ15eXl5eXl5eXl5eXl5eXl5eXl5eXl5eXl5eXl5eX19eeXl5eXl5eXl5eXl5eXl5eXl5eXl5eXl5eXl5eXkXFxcXFxcXFxcXFxcXFxcXFxcXFxcXFxcXFxcXFxcXFxcXFzExMTExMTExMTExMTExMTExMTExMTExMTExMTExT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09qampqampqampqampqampqampqampqampqampqamsHBwUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyM+Pj4+Pj4+Pj4+Pj4+Pj4+Pj4+Pj4+Pj4+Pj4+Pj4/PlhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYd3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3ExMRERERERERERERERERERERERERERERERERERERES8vLy8vLy8vLy8vLy8vLy8vLy8vLy8vLy8vLy8vLy8vLy8vSkpKSkpKSkpKSkpKSkpKSkpKSkpKSkpKSkpKSkpLS0plZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZQMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDHR0dHR0dHR0dHR0dHR0dHR0dHR0dHR0dHR0dHR0eHh44ODg4ODg4ODg4ODg4ODg4ODg4ODg4ODg4ODg4OFdXV1ZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWVlZWV1dXcXFxcXFxcXFxcXFxcXFxcXFxcXFxcXFxcXFxcXEPDw8PDw8PDw8PDw8PDw8PDw8PDw8PDw8PDw8PDw8PDw8PDyoqKioqKioqKioqKioqKioqKioqKioqKioqKioqKysqRERERERERERERERERERERERERERERERERERERERjY2NjY2NjY2NjYw==",
"frame_count": 1288,
"media": {
"file": "jray_audio_v1_tone.flac",
"generator": "make_fixture.py",
"container": "FLAC (lossless \u2014 decodes to exactly the PCM make_fixture.py emits)",
"duration_sec": 120.0,
"sample_rate": 11025,
"channels": 1,
"sample_format": "s16",
"sha256": "912ecd426cd426dccb37753e0249694227619c701cb9f533502b37da0fbe8096",
"bytes": 585142
},
"decoded_window": {
"_": "Checksums of the 120 s centre window after downmix to mono and resample to 11025 Hz, i.e. exactly the stream `ffmpeg -ss <mid-60> -t 120 -i <file> -vn -ac 1 -ar 11025 -f f32le -` produces. Check these first: a mismatch here is a decode problem, not a DSP one.",
"samples": 1323000,
"f32le_fnv1a64": "0x1ef7899cd4d12662",
"s16le_fnv1a64": "0xf824fa56f125c0dc"
},
"params": {
"window_sec": 120.0,
"window_centre": "runtime/2, i.e. samples from runtime/2 - 60 s; truncated to exactly 1323000 samples",
"min_duration_sec": 120.0,
"min_duration_rule": "IR-007 \u2014 below this emit NO signature and apply no sync offset",
"sample_rate": 11025,
"channels": 1,
"arithmetic": "IEEE-754 double throughout; float32 is not sufficient",
"sample_scale": "s16 * (1/32768), FFmpeg's native s16->flt",
"frame_size": 4096,
"hop_size": 1024,
"frame_count_rule": "1 + (n_samples - 4096) / 1024, integer division; whole frames only",
"window_fn": "Hann, PERIODIC: w[n] = 0.5 * (1 - cos(2*pi*n/4096))",
"transform": "radix-2 DIT complex FFT over the 4096 real samples (imag=0), no normalisation",
"magnitude": "sqrt(re^2 + im^2), linear",
"band_lo_hz": 300.0,
"band_hi_hz": 3000.0,
"num_bands": 32,
"band_edges": "edge[b] = 300 * (3000/300)^(b/32), b = 0..32",
"band_bins": "band b owns FFT bins [k_lo[b], k_lo[b+1]) with k_lo[b] = ceil(edge[b] * 4096 / 11025); see band_fft_bins",
"band_value": "MEAN of the linear magnitudes in the band (not sum, not max)",
"peak_bin": "argmax over the 32 band values; ties resolve to the LOWEST index",
"energy_metric": "E = mean magnitude over all FFT bins 112..1114, i.e. the whole 300-3000 Hz band",
"energy_reference": "upper median of E over all frames: sorted[n/2], no averaging of the two middle values",
"energy_ratio": "r = log10((E + 1e-12) / (E_ref + 1e-12))",
"energy_class_edges": [
-0.6,
-0.2,
0.2
],
"energy_class": "0 if r < -0.6, 1 if r < -0.2, 2 if r < 0.2, else 3",
"byte_layout": "bit7 = 0 (reserved), bits6..2 = 5-bit band index, bits1..0 = 2-bit energy class; byte = (band << 2) | class",
"base64": "standard alphabet A-Za-z0-9+/ with '=' padding",
"prefix": "v1:"
},
"band_fft_bins": [
[
112,
120
],
[
120,
129
],
[
129,
139
],
[
139,
149
],
[
149,
160
],
[
160,
172
],
[
172,
185
],
[
185,
199
],
[
199,
213
],
[
213,
229
],
[
229,
246
],
[
246,
265
],
[
265,
285
],
[
285,
306
],
[
306,
328
],
[
328,
353
],
[
353,
379
],
[
379,
408
],
[
408,
438
],
[
438,
471
],
[
471,
506
],
[
506,
543
],
[
543,
584
],
[
584,
627
],
[
627,
674
],
[
674,
724
],
[
724,
778
],
[
778,
836
],
[
836,
899
],
[
899,
966
],
[
966,
1038
],
[
1038,
1115
]
],
"notes": [
"The server spec fixes the window, rate, STFT geometry, band and the 5+2 bit packing. Everything under params beyond that (Hann periodicity, band aggregation, the energy-class definition, tie-breaking, base64 alphabet) is pinned HERE for v1 \u2014 the spec does not constrain it, and two implementations that guess differently produce non-matching signatures.",
"Decision margins on this fixture: the two strongest bands are within 1.3% on the closest frame, and the closest frame to an energy-class edge is 3.6e-3 away in log10. Both are many orders of magnitude above double-precision FFT differences, so any two correct double- precision implementations agree; a float32 implementation is not guaranteed to.",
"Coverage: all 32 bands and all 4 energy classes appear in the golden signature.",
"Robustness observed on this fixture: identical peak-bin sequence after a stereo/44100 Hz round trip and after AAC 128 kbit/s re-encoding."
]
}
tests/fixtures/audio/jray_audio_v1_tone.flac
Vendored
BIN
View File
Binary file not shown.
tests/fixtures/audio/make_fixture.py
Vendored
-114
View File
@@ -1,114 +0,0 @@
#!/usr/bin/env python3
"""Regenerate the JRay audio-signature golden fixture.
python3 make_fixture.py # writes jray_audio_v1_tone.flac here
This is the *source of truth* for the fixture media: `jray_audio_v1_tone.flac`
is a lossless FLAC encoding of exactly the PCM this script emits, so any repo
that wants to check its own audio-signature implementation against the golden
vector in `jray_audio_v1_golden.json` can regenerate the input from scratch and
confirm it is byte-identical (the golden file records `pcm_fnv1a64`, a hash of
the decoded 16-bit samples).
Deliberately dependency-free (no numpy) and written in plain arithmetic so it
ports to any language in ~20 lines.
Signal — 120.000 s, mono, 11025 Hz, 16-bit signed PCM:
* split into segments of 32768 samples (~2.97 s), 40.4 segments in total;
* segment `s` carries one sine at the geometric centre of log-band
`(s * 7) mod 32` of the 300-3000 Hz band, so all 32 bands are exercised;
* its amplitude walks a golden-ratio low-discrepancy sequence over
[10^-1.55, 10^-0.02] so frame energies spread continuously across ~1.5
decades and all four energy classes are exercised, without a dense cluster
of frames sitting on a class boundary;
* phase is carried across segment boundaries (no clicks);
* a constant, far quieter 777 Hz tone sits underneath so no frame is
degenerate;
* samples are quantised with floor(x * 32767 + 0.5).
Why FLAC and not WAV: 120 s of 11025 Hz 16-bit PCM is 2.6 MB and does not
compress in git. FLAC is lossless — FFmpeg decodes it to exactly the PCM
written here — and is ~3.5x smaller. `--wav` writes the uncompressed original
if you want to diff it.
"""
import math
import struct
import subprocess
import sys
import os
SAMPLE_RATE = 11025
DURATION_SEC = 120.0
SEGMENT = 32768 # samples per tone segment
BAND_STRIDE = 7 # coprime with 32 -> visits every band
BAND_LO_HZ = 300.0
BAND_HI_HZ = 3000.0
NUM_BANDS = 32
AMP_LOG_MIN = -1.55 # 10^-1.55 ~= 0.028
AMP_LOG_SPAN = 1.53 # up to 10^-0.02 ~= 0.955
PHI_FRAC = 0.6180339887498949
BG_HZ = 777.0
BG_AMP = 0.004
OUT_FLAC = "jray_audio_v1_tone.flac"
OUT_WAV = "jray_audio_v1_tone.wav"
def generate():
"""Return the 120 s signal as a list of int16 sample values."""
n = int(round(SAMPLE_RATE * DURATION_SEC))
out = [0] * n
phase = 0.0
two_pi = 2.0 * math.pi
for start in range(0, n, SEGMENT):
s = start // SEGMENT
end = min(n, start + SEGMENT)
band = (s * BAND_STRIDE) % NUM_BANDS
# geometric centre of log-band `band`
freq = BAND_LO_HZ * (BAND_HI_HZ / BAND_LO_HZ) ** ((band + 0.5) / NUM_BANDS)
amp = 10.0 ** (AMP_LOG_MIN + AMP_LOG_SPAN * ((s * PHI_FRAC) % 1.0))
step = two_pi * freq / SAMPLE_RATE
for k in range(end - start):
i = start + k
x = amp * math.sin(phase + step * k)
x += BG_AMP * math.sin(two_pi * BG_HZ * i / SAMPLE_RATE)
if x > 1.0:
x = 1.0
elif x < -1.0:
x = -1.0
out[i] = int(math.floor(x * 32767.0 + 0.5))
phase = (phase + step * (end - start)) % two_pi
return out
def write_wav(path, samples):
data = struct.pack("<%dh" % len(samples), *samples)
hdr = b"RIFF" + struct.pack("<I", 36 + len(data)) + b"WAVE"
hdr += b"fmt " + struct.pack("<IHHIIHH", 16, 1, 1, SAMPLE_RATE,
SAMPLE_RATE * 2, 2, 16)
hdr += b"data" + struct.pack("<I", len(data))
with open(path, "wb") as fh:
fh.write(hdr + data)
def main():
here = os.path.dirname(os.path.abspath(__file__))
samples = generate()
wav = os.path.join(here, OUT_WAV)
write_wav(wav, samples)
if "--wav" in sys.argv:
print("wrote", wav)
return
flac = os.path.join(here, OUT_FLAC)
# -compression_level 12 is deterministic for a given libFLAC/ffmpeg build;
# only the container bytes vary, never the decoded PCM.
subprocess.run(["ffmpeg", "-nostdin", "-v", "error", "-y", "-i", wav,
"-c:a", "flac", "-compression_level", "12", flac],
check=True)
os.remove(wav)
print("wrote", flac)
if __name__ == "__main__":
main()
tests/fixtures/dumps/bali_13.h5
Vendored
BIN
View File
Binary file not shown.
tests/fixtures/dumps/bali_27.h5
Vendored
BIN
View File
Binary file not shown.
tests/fixtures/dumps/bali_28.h5
Vendored
BIN
View File
Binary file not shown.
tests/fixtures/dumps/bali_31.h5
Vendored
BIN
View File
Binary file not shown.
tests/fixtures/dumps/bali_46.h5
Vendored
BIN
View File
Binary file not shown.
tests/test_audio_signature.cpp
-359
View File
@@ -1,359 +0,0 @@
// Unit tests for the JRay v1 audio signature (src/audio_signature.*).
//
/// TRACES: UT-101, UT-102, UT-103, UT-104 | IR-004, IR-005, IR-007, IR-008
//
// The headline test is the golden vector: a deterministic tone fixture checked
// into tests/fixtures/audio/ together with the signature it must produce. That
// fixture is the artefact shared with the jRay plugin repo, and it is what
// makes "both producers agree bit-for-bit" a checked claim rather than an
// assertion (IR-005).
//
// GPU-free, model-free, no network. Pure CPU DSP plus an FFmpeg decode of a
// 585 KB file — which is precisely why this is the right cross-repo check: it
// runs anywhere, including the N100 CI host.
#include <catch2/catch_test_macros.hpp>
#include <catch2/matchers/catch_matchers_floating_point.hpp>
#include "audio_signature.hpp"
#include <nlohmann/json.hpp>
#include <cmath>
#include <cstdint>
#include <filesystem>
#include <fstream>
#include <stdexcept>
#include <string>
#include <vector>
using namespace sae::audio;
namespace fs = std::filesystem;
namespace {
const std::string kFixtureDir = SAE_TEST_FIXTURES_DIR "/audio";
const std::string kGoldenPath = kFixtureDir + "/jray_audio_v1_golden.json";
const std::string kMediaPath = kFixtureDir + "/jray_audio_v1_tone.flac";
const nlohmann::json& golden() {
static const nlohmann::json j = [] {
std::ifstream in(kGoldenPath);
if (!in.good())
throw std::runtime_error("golden fixture not found: " + kGoldenPath);
nlohmann::json parsed;
in >> parsed;
return parsed;
}();
return j;
}
std::uint64_t hex64(const std::string& s) {
return std::stoull(s, nullptr, 16);
}
// The fixture is 120 s of audio: decoding and signing it is the expensive part
// of this file, so both results are computed once and shared. Every test below
// still asserts against the on-disk golden values, not against each other.
const std::optional<std::vector<float>>& fixture_window() {
static const std::optional<std::vector<float>> w = decode_centre_window(kMediaPath);
return w;
}
const std::optional<std::string>& fixture_signature() {
static const std::optional<std::string> s = compute_signature(kMediaPath);
return s;
}
// ── Minimal WAV writer, so the short-media and resample cases need no fixture ─
// 16-bit PCM, interleaved.
struct TempWav {
fs::path path;
explicit TempWav(const std::string& name)
: path(fs::temp_directory_path() / ("sae_audio_test_" + name + ".wav")) {}
~TempWav() { std::error_code ec; fs::remove(path, ec); }
void write(const std::vector<std::int16_t>& samples, int rate, int channels) const {
const std::uint32_t bytes = static_cast<std::uint32_t>(samples.size() * 2);
const std::uint32_t byte_rate = static_cast<std::uint32_t>(rate * channels * 2);
std::ofstream out(path, std::ios::binary);
auto u32 = [&](std::uint32_t v) { out.write(reinterpret_cast<const char*>(&v), 4); };
auto u16 = [&](std::uint16_t v) { out.write(reinterpret_cast<const char*>(&v), 2); };
out.write("RIFF", 4); u32(36 + bytes); out.write("WAVE", 4);
out.write("fmt ", 4); u32(16); u16(1); u16(static_cast<std::uint16_t>(channels));
u32(static_cast<std::uint32_t>(rate)); u32(byte_rate);
u16(static_cast<std::uint16_t>(channels * 2)); u16(16);
out.write("data", 4); u32(bytes);
out.write(reinterpret_cast<const char*>(samples.data()), bytes);
}
};
// A plain 1 kHz tone, mono, at the signature's own rate.
std::vector<std::int16_t> tone(double seconds, int rate = kSampleRate) {
const std::size_t n = static_cast<std::size_t>(std::llround(seconds * rate));
std::vector<std::int16_t> s(n);
for (std::size_t i = 0; i < n; ++i)
s[i] = static_cast<std::int16_t>(std::llround(
20000.0 * std::sin(2.0 * 3.14159265358979323846 * 1000.0 * double(i) / rate)));
return s;
}
std::vector<std::uint8_t> base64_decode(const std::string& in) {
auto val = [](char c) -> int {
if (c >= 'A' && c <= 'Z') return c - 'A';
if (c >= 'a' && c <= 'z') return c - 'a' + 26;
if (c >= '0' && c <= '9') return c - '0' + 52;
if (c == '+') return 62;
if (c == '/') return 63;
return -1;
};
std::vector<std::uint8_t> out;
std::uint32_t acc = 0;
int bits = 0;
for (char c : in) {
const int v = val(c);
if (v < 0) continue; // '=' padding
acc = (acc << 6) | static_cast<std::uint32_t>(v);
bits += 6;
if (bits >= 8) {
bits -= 8;
out.push_back(static_cast<std::uint8_t>((acc >> bits) & 0xFF));
}
}
return out;
}
} // namespace
// ── UT-101 — the golden vector ──────────────────────────────────────────────
/// TRACES: UT-101 | IR-004, IR-005, IR-008
TEST_CASE("signature of the golden fixture matches the recorded value exactly",
"[audio_signature][golden]") {
REQUIRE(fs::exists(kMediaPath));
const std::optional<std::string>& sig = fixture_signature();
REQUIRE(sig.has_value());
CHECK(*sig == golden()["signature"].get<std::string>());
}
/// TRACES: UT-101 | IR-005
TEST_CASE("decoded centre window matches the recorded PCM checksum",
"[audio_signature][golden]") {
// Checked separately from the signature so a codec-level difference is
// distinguishable from a DSP-level one: if this passes and the signature
// test fails, the DSP diverged; if this fails, the decode did.
const std::optional<std::vector<float>>& mono = fixture_window();
REQUIRE(mono.has_value());
CHECK(mono->size() == golden()["decoded_window"]["samples"].get<std::size_t>());
CHECK(fnv1a64(mono->data(), mono->size() * sizeof(float)) ==
hex64(golden()["decoded_window"]["f32le_fnv1a64"].get<std::string>()));
}
/// TRACES: UT-101 | IR-004
TEST_CASE("log-spaced band table matches the recorded one", "[audio_signature][golden]") {
// The band->FFT-bin table is the part of the construction most likely to
// drift between two implementations, so it is pinned independently of the
// signature it produces.
const auto& tbl = band_fft_bins();
const auto& want = golden()["band_fft_bins"];
REQUIRE(want.size() == tbl.size());
for (std::size_t b = 0; b < tbl.size(); ++b) {
CHECK(tbl[b].first == want[b][0].get<int>());
CHECK(tbl[b].second == want[b][1].get<int>());
CHECK(tbl[b].second > tbl[b].first); // no empty band
if (b) CHECK(tbl[b].first == tbl[b - 1].second); // contiguous, no overlap
}
}
/// TRACES: UT-101 | IR-004, IR-008
TEST_CASE("signature is well-formed: v1 prefix, 1288 frames, structural bytes",
"[audio_signature][golden]") {
const std::optional<std::string>& sig = fixture_signature();
REQUIRE(sig.has_value());
// IR-008 — the signature carries its own version, separate from
// schema_version, so a future DSP change is detectable rather than silently
// producing non-matching signatures.
REQUIRE(sig->rfind(kVersionPrefix, 0) == 0);
const std::vector<std::uint8_t> bytes = base64_decode(sig->substr(3));
CHECK(bytes.size() == kExpectedFrames);
CHECK(bytes.size() == golden()["frame_count"].get<std::size_t>());
// The server validates this structure on upload (server SPEC §3): each byte
// is a 5-bit band index plus a 2-bit energy class, so bit 7 is always clear
// and arbitrary bytes are invalid. That is what keeps the field from being
// a payload channel.
bool bands_seen[kNumBands] = {};
bool classes_seen[4] = {};
for (std::uint8_t b : bytes) {
REQUIRE((b & 0x80) == 0);
bands_seen[(b >> 2) & 0x1F] = true;
classes_seen[b & 0x03] = true;
}
// The fixture is built to exercise the whole output alphabet — if it ever
// stops doing so, the golden vector has become a weaker check than it looks.
for (bool seen : bands_seen) CHECK(seen);
for (bool seen : classes_seen) CHECK(seen);
}
// ── UT-102 — IR-007, media shorter than the window ──────────────────────────
/// TRACES: UT-102 | IR-007
TEST_CASE("media shorter than 120 s emits no signature", "[audio_signature][short]") {
// The window runtime/2 ± 60 s underflows, so there is no signature and no
// sync offset downstream. Both producers must apply the identical rule or
// they diverge on exactly the short items most likely to be misidentified.
SECTION("30 s") {
TempWav w("short30");
w.write(tone(30.0), kSampleRate, 1);
CHECK_FALSE(compute_signature(w.path.string()).has_value());
CHECK_FALSE(decode_centre_window(w.path.string()).has_value());
}
SECTION("just under the boundary") {
TempWav w("short11999");
w.write(tone(119.99), kSampleRate, 1);
CHECK_FALSE(compute_signature(w.path.string()).has_value());
}
}
/// TRACES: UT-102 | IR-007
TEST_CASE("media of exactly 120 s emits a full-length signature",
"[audio_signature][short]") {
TempWav w("exact120");
w.write(tone(120.0), kSampleRate, 1);
const std::optional<std::string> sig = compute_signature(w.path.string());
REQUIRE(sig.has_value());
CHECK(base64_decode(sig->substr(3)).size() == kExpectedFrames);
}
/// TRACES: UT-102 | IR-007
TEST_CASE("unreadable media degrades to no signature rather than failing",
"[audio_signature][short]") {
// UR-9 is an enhancement and must never be able to break a fetch.
CHECK_FALSE(compute_signature("/nonexistent/definitely-not-here.mkv").has_value());
}
/// TRACES: UT-102 | IR-004
TEST_CASE("the window is taken from the centre, not the head",
"[audio_signature][centre]") {
// Sampling from the centre is the whole reason the construction avoids the
// head and tail (logos, cold opens, credits), so it needs its own check:
// wrap the fixture's own 120 s in 90 s of silence either side and the
// signature of the 300 s file must be the golden value, byte for byte.
// Nothing else pins the seek offset — a head-anchored window would pass
// every other test in this file.
const std::optional<std::vector<float>>& mono = fixture_window();
REQUIRE(mono.has_value());
const std::size_t pad = 90 * kSampleRate;
std::vector<std::int16_t> padded(pad * 2 + mono->size(), 0);
for (std::size_t i = 0; i < mono->size(); ++i)
padded[pad + i] = static_cast<std::int16_t>(std::llround(double((*mono)[i]) * 32768.0));
TempWav w("centred300");
w.write(padded, kSampleRate, 1);
const std::optional<std::string> sig = compute_signature(w.path.string());
REQUIRE(sig.has_value());
CHECK(*sig == golden()["signature"].get<std::string>());
}
// ── UT-103 — downmix and resample ───────────────────────────────────────────
/// TRACES: UT-103 | IR-004
TEST_CASE("stereo, non-native sample rate yields the same peak-bin sequence",
"[audio_signature][resample]") {
// The golden fixture is already mono at 11025 Hz so the golden vector does
// not depend on the resampler's version. This case exercises the path that
// real media takes — downmix plus resample — by rebuilding the fixture's own
// audio as 22050 Hz stereo and checking the peak bins survive it.
const std::optional<std::vector<float>>& mono = fixture_window();
REQUIRE(mono.has_value());
std::vector<std::int16_t> stereo;
stereo.reserve(mono->size() * 4);
for (float f : *mono) {
const auto s = static_cast<std::int16_t>(std::llround(double(f) * 32768.0));
stereo.push_back(s); stereo.push_back(s); // sample 1, L/R
stereo.push_back(s); stereo.push_back(s); // sample 2 (zero-order hold)
}
TempWav w("stereo22050");
w.write(stereo, 2 * kSampleRate, 2);
const std::optional<std::string> sig = compute_signature(w.path.string());
REQUIRE(sig.has_value());
const std::vector<std::uint8_t> got = base64_decode(sig->substr(3));
const std::vector<std::uint8_t> want =
base64_decode(golden()["signature"].get<std::string>().substr(3));
REQUIRE(got.size() == want.size());
std::size_t agree = 0;
for (std::size_t i = 0; i < got.size(); ++i)
agree += ((got[i] >> 2) == (want[i] >> 2)) ? 1 : 0;
// The server treats ≥ 0.85 as the `audio` match tier; this path scores 1.0
// in practice, and the margin is left for libswresample version drift.
CHECK(double(agree) / double(got.size()) >= 0.85);
}
// ── UT-104 — the pure DSP surface ───────────────────────────────────────────
/// TRACES: UT-104 | IR-004
TEST_CASE("pack_frames uses whole frames only", "[audio_signature][dsp]") {
CHECK(pack_frames(std::vector<float>(kFrameSize - 1, 0.f)).empty());
CHECK(pack_frames(std::vector<float>(kFrameSize, 0.f)).size() == 1);
CHECK(pack_frames(std::vector<float>(kFrameSize + kHopSize - 1, 0.f)).size() == 1);
CHECK(pack_frames(std::vector<float>(kFrameSize + kHopSize, 0.f)).size() == 2);
// The full 120 s window is 1288 frames — asserted as a constant rather than
// by running the DSP over 1.3 M zeros, which is the same claim for free.
CHECK(kWindowSamples == 1323000u);
CHECK(kExpectedFrames == 1288u);
CHECK_FALSE(signature_from_mono(std::vector<float>(kFrameSize - 1, 0.f)).has_value());
}
/// TRACES: UT-104 | IR-004
TEST_CASE("a pure tone lands in the band that contains it", "[audio_signature][dsp]") {
// 1000 Hz sits in log-band floor(32 * log10(1000/300)) = 16.
const int expect = static_cast<int>(std::floor(
kNumBands * std::log10(1000.0 / kBandLoHz) / std::log10(kBandHiHz / kBandLoHz)));
std::vector<float> mono(kWindowSamples / 100);
for (std::size_t i = 0; i < mono.size(); ++i)
mono[i] = static_cast<float>(0.5 * std::sin(
2.0 * 3.14159265358979323846 * 1000.0 * double(i) / kSampleRate));
const std::vector<std::uint8_t> packed = pack_frames(mono);
REQUIRE_FALSE(packed.empty());
for (std::uint8_t b : packed) CHECK(((b >> 2) & 0x1F) == expect);
}
/// TRACES: UT-104 | IR-004
TEST_CASE("signature is invariant to overall gain", "[audio_signature][dsp]") {
// Loudness normalisation between two releases of the same cut must not
// change the signature — that is why the energy class is relative.
std::vector<float> a(kWindowSamples / 50);
for (std::size_t i = 0; i < a.size(); ++i) {
const double t = double(i) / kSampleRate;
a[i] = static_cast<float>(0.4 * std::sin(2.0 * 3.14159265358979323846 * 640.0 * t) +
0.2 * std::sin(2.0 * 3.14159265358979323846 * 1900.0 * t) *
std::sin(2.0 * 3.14159265358979323846 * 0.7 * t));
}
std::vector<float> b(a.size());
for (std::size_t i = 0; i < a.size(); ++i) b[i] = a[i] * 0.25f;
CHECK(pack_frames(a) == pack_frames(b));
}
/// TRACES: UT-104 | IR-004
TEST_CASE("base64 encoder matches the standard alphabet and padding",
"[audio_signature][dsp]") {
auto enc = [](const std::string& s) {
return base64_encode(reinterpret_cast<const std::uint8_t*>(s.data()), s.size());
};
CHECK(enc("") == "");
CHECK(enc("f") == "Zg==");
CHECK(enc("fo") == "Zm8=");
CHECK(enc("foo") == "Zm9v");
CHECK(enc("foob") == "Zm9vYg==");
CHECK(enc("fooba") == "Zm9vYmE=");
CHECK(enc("foobar") == "Zm9vYmFy");
const std::uint8_t all[] = {0xFB, 0xFF, 0xBF}; // exercises '+' and '/'
CHECK(base64_encode(all, 3) == "+/+/");
}
tests/test_face_tracker.cpp
+52 -116
View File
@@ -13,12 +13,8 @@
#include "config.hpp"
#include "nodes/face_tracker_node.hpp"
#include "types.hpp"
#include "track_registry.hpp"
#include "evidence_discount.hpp"
#include <algorithm>
#include <cmath>
#include <memory>
namespace {
@@ -59,145 +55,85 @@ EmbeddedSceneFrame frame(double t, float x, float y, const Embedding& emb,
return ef;
}
// Build a tracker over a fresh registry. The registry IS the tracker's state
// now (AR-008), so a test constructs both together and can inspect either.
struct Rig {
std::shared_ptr<TrackRegistry> reg;
FaceTrackerFunc ft;
explicit Rig(double extinction = 30.0, float assoc_min_prob = 0.5f)
: reg(std::make_shared<TrackRegistry>(
[extinction] {
TrackRegistry::Config c;
c.extinction_sec = extinction;
return c;
}(),
EvidenceDiscounter([](float cos) { return std::max(0.f, cos); })))
, ft([&] {
Config c;
c.track_assoc_min_prob = assoc_min_prob;
return c;
}(),
reg,
// Trivial calibration: cosine passed through as P(same). Real runs use
// the fit belonging to the active embedder (AR-023/AR-024).
[](float cos) { return std::max(0.f, cos); })
{}
int track_of(EmbeddedSceneFrame f) { return ft(std::move(f)).track_ids[0]; }
};
Config tracker_cfg() {
Config cfg;
cfg.cut_revive_sim = 0.50f;
cfg.cut_inactive_max_frames = 5;
return cfg;
}
} // namespace
// ── AR-008 — one pool, ordinary association ──────────────────────────────────
TEST_CASE("track id is stable across ordinary frames", "[face_tracker][AR-008]") {
Rig r;
TEST_CASE("track id is stable across ordinary frames", "[face_tracker]") {
FaceTrackerFunc ft(tracker_cfg());
Embedding e = axis(0);
int id0 = r.track_of(frame(0.0, 10, 10, e));
int id1 = r.track_of(frame(1.0, 11, 10, e)); // overlaps → same track
int id0 = ft(frame(0.0, 10, 10, e)).track_ids[0];
int id1 = ft(frame(1.0, 11, 10, e)).track_ids[0]; // overlaps → same track
CHECK(id0 >= 0);
CHECK(id1 == id0);
}
TEST_CASE("a face lost across a cut and re-associated is the SAME track",
"[face_tracker][AR-008]") {
// Previously this was a distinct "revival" path guarded by a raw-cosine
// constant. There is no such path now: a dormant track is an ordinary
// association candidate, and continuity falls out of the embedding match.
Rig r;
TEST_CASE("cut revives the same track id for a matching identity", "[face_tracker]") {
FaceTrackerFunc ft(tracker_cfg());
// Pre-cut: establish a track for a person whose embedding is near-identical
// across the cut (sim well above cut_revive_sim), but whose box jumps so IoU
// is 0 — the ordinary spatial path cannot re-link it.
Embedding pre = at_sim(0, 1, 0.99f);
int id_pre = r.track_of(frame(0.0, 10, 10, pre));
int id_pre = ft(frame(0.0, 10, 10, pre)).track_ids[0];
REQUIRE(id_pre >= 0);
// Box jumps so IoU is zero — only the embedding can link it.
Embedding post = at_sim(0, 1, 0.98f);
CHECK(r.track_of(frame(1.0, 300, 300, post, /*is_cut=*/true)) == id_pre);
Embedding post = at_sim(0, 1, 0.98f); // cos(diff) ≈ 0.9997 > 0.50
auto out = ft(frame(1.0, 300, 300, post, /*is_cut=*/true));
CHECK(out.track_ids[0] == id_pre); // revived, not a fresh id
}
TEST_CASE("a cut starts a fresh track when identity does not match",
"[face_tracker][AR-008]") {
Rig r;
int id_pre = r.track_of(frame(0.0, 10, 10, axis(0)));
TEST_CASE("cut starts a fresh track when identity does not match", "[face_tracker]") {
FaceTrackerFunc ft(tracker_cfg());
int id_pre = ft(frame(0.0, 10, 10, axis(0))).track_ids[0];
REQUIRE(id_pre >= 0);
// Orthogonal embedding and disjoint box: nothing links them.
int id_post = r.track_of(frame(1.0, 300, 300, axis(5), /*is_cut=*/true));
CHECK(id_post != id_pre);
CHECK(id_post >= 0);
// Post-cut face is orthogonal (sim 0 < cut_revive_sim) and spatially disjoint
// → no revival, brand-new id.
auto out = ft(frame(1.0, 300, 300, axis(5), /*is_cut=*/true));
CHECK(out.track_ids[0] != id_pre);
CHECK(out.track_ids[0] >= 0);
}
// ── AR-007 — a cut makes association ignore position ─────────────────────────
TEST_CASE("on a cut, identity follows the embedding rather than the box",
"[face_tracker][AR-007]") {
// Two people swap screen positions across a cut while keeping their faces.
// If IoU still carried weight the ids would follow the boxes and swap; with
// alpha driven to embedding-only on a cut, they must follow the faces.
Rig r;
Embedding a = at_sim(0, 1, 0.99f);
Embedding b = at_sim(2, 3, 0.99f);
EmbeddedSceneFrame f0;
f0.source.timestamp_sec = 0.0;
f0.faces = {face_at(10, 10), face_at(300, 300)};
f0.crops = {cv::Mat(), cv::Mat()};
f0.embeddings = {a, b};
auto out0 = r.ft(std::move(f0));
const int id_a = out0.track_ids[0];
const int id_b = out0.track_ids[1];
REQUIRE(id_a >= 0);
REQUIRE(id_b >= 0);
REQUIRE(id_a != id_b);
// Same two people, positions exchanged, on a cut frame.
EmbeddedSceneFrame f1;
f1.source.timestamp_sec = 1.0;
f1.source.is_cut = true;
f1.faces = {face_at(300, 300), face_at(10, 10)};
f1.crops = {cv::Mat(), cv::Mat()};
f1.embeddings = {a, b};
auto out1 = r.ft(std::move(f1));
CHECK(out1.track_ids[0] == id_a); // A kept its id despite moving to B's box
CHECK(out1.track_ids[1] == id_b);
}
// ── AR-013 — extinction replaces the parked-pool frame counter ───────────────
TEST_CASE("a track past the extinction window is gone, not revived",
"[face_tracker][AR-013]") {
// The old design aged a parked pool in frames, which silently changed
// meaning with sample_fps. Extinction is in seconds and lives in the
// registry, so the tracker no longer counts anything.
Rig r(/*extinction=*/2.0);
TEST_CASE("parked track expires after cut_inactive_max_frames", "[face_tracker]") {
Config cfg = tracker_cfg();
cfg.cut_inactive_max_frames = 2;
FaceTrackerFunc ft(cfg);
Embedding person = at_sim(0, 1, 0.99f);
int id_pre = r.track_of(frame(0.0, 10, 10, person));
int id_pre = ft(frame(0.0, 10, 10, person)).track_ids[0];
REQUIRE(id_pre >= 0);
// Unrelated faces elsewhere while the clock runs well past extinction.
r.track_of(frame(1.0, 300, 300, axis(7), /*is_cut=*/true));
r.track_of(frame(10.0, 300, 300, axis(7)));
// Cut with an unrelated face parks id_pre; then let the pool age past its
// limit with more unrelated, spatially-disjoint faces (each ages the pool by
// one). By the time the person returns, id_pre must be gone.
ft(frame(1.0, 300, 300, axis(7), /*is_cut=*/true)); // park (age 1)
ft(frame(2.0, 300, 300, axis(7))); // age 2
ft(frame(3.0, 300, 300, axis(7))); // age 3 → id_pre dropped
CHECK(r.track_of(frame(11.0, 10, 10, person)) != id_pre);
auto out = ft(frame(4.0, 10, 10, person)); // same identity returns
CHECK(out.track_ids[0] != id_pre); // too late — fresh id
}
TEST_CASE("a track within the extinction window is still a candidate",
"[face_tracker][AR-013]") {
Rig r(/*extinction=*/30.0);
TEST_CASE("eof clears active and parked tracks", "[face_tracker]") {
FaceTrackerFunc ft(tracker_cfg());
Embedding person = at_sim(0, 1, 0.99f);
int id_pre = r.track_of(frame(0.0, 10, 10, person));
r.track_of(frame(1.0, 300, 300, axis(7), /*is_cut=*/true));
int id_pre = ft(frame(0.0, 10, 10, person)).track_ids[0];
ft(frame(1.0, 300, 300, axis(7), /*is_cut=*/true)); // park id_pre
// Back inside the window: the same person continues the same track, so the
// gap is absorbed into one window rather than splitting it.
CHECK(r.track_of(frame(3.0, 10, 10, person)) == id_pre);
}
TEST_CASE("eof is forwarded", "[face_tracker]") {
Rig r;
EmbeddedSceneFrame eof;
eof.source.eof = true;
CHECK(r.ft(std::move(eof)).source.eof);
auto out = ft(std::move(eof));
CHECK(out.source.eof);
// After eof the pools are empty: the returning identity must get a fresh id,
// not the parked one.
auto out2 = ft(frame(2.0, 10, 10, person));
CHECK(out2.track_ids[0] != id_pre);
}
tests/test_gallery_store.cpp
+1 -251
View File
@@ -1,17 +1,13 @@
// Unit tests for gallery (de)serialisation: HDF5 round-trip fidelity (the only
// format save_gallery writes), legacy JSON read back-compat (optional field
// defaults, the legacy "jellyfin_person_id" fallback), and the GR-004 embedder
// stamp. GPU-free, model-free — the stamp tests exercise the comparison logic
// with synthetic stamps and never load an ONNX, so they run on CI's Intel N100.
// defaults, the legacy "jellyfin_person_id" fallback). GPU-free, model-free.
#include <catch2/catch_test_macros.hpp>
#include "gallery/embedder_stamp.hpp"
#include "gallery/gallery_store.hpp"
#include "types.hpp"
#include <cstdio>
#include <fstream>
#include <stdexcept>
#include <string>
#include <nlohmann/json.hpp>
@@ -31,22 +27,6 @@ Embedding make_embedding(float base) {
return e;
}
// A stamp built by hand — no ONNX is read, so these tests never need a model.
EmbedderStamp stamp(const std::string& name, const std::string& sha, int32_t dim = 512) {
EmbedderStamp s;
s.model_name = name;
s.model_sha256 = sha;
s.embed_dim = dim;
return s;
}
const std::string kShaA(64, 'a');
const std::string kShaB(64, 'b');
bool mentions(const std::string& haystack, const std::string& needle) {
return haystack.find(needle) != std::string::npos;
}
} // namespace
TEST_CASE("gallery save/load round-trips actors and embeddings", "[gallery]") {
@@ -173,233 +153,3 @@ TEST_CASE("load_gallery reads the legacy jellyfin_person_id key", "[gallery]") {
TEST_CASE("load_gallery throws on a missing file", "[gallery]") {
CHECK_THROWS(load_gallery("/nonexistent/path/gallery.json"));
}
// ── GR-004: gallery ↔ embedder binding ───────────────────────────────────────
// Verification plan row GR-004/T1: "Mismatched embedder → hard startup error;
// error names both sides." The comparison is a pure function over two stamps, so
// none of this needs a GPU, an ONNX, or even a file.
/// TRACES: GR-004 | SR-001
TEST_CASE("gallery save/load round-trips the embedder stamp", "[gallery][GR-004]") {
ActorGallery g;
ActorGallery::Actor a;
a.name = "Stamped Actor";
a.embeddings = {make_embedding(0.3f)};
g.actors.push_back(a);
g.embedder = stamp("LVFace-B_Glint360K.onnx", kShaA);
TempFile tf("gallery_stamped.h5");
save_gallery(tf.path, g);
ActorGallery loaded = load_gallery(tf.path);
CHECK(loaded.embedder.model_name == "LVFace-B_Glint360K.onnx");
CHECK(loaded.embedder.model_sha256 == kShaA);
CHECK(loaded.embedder.embed_dim == 512);
CHECK_FALSE(loaded.embedder.empty());
}
/// TRACES: GR-004 | SR-001
TEST_CASE("a gallery written without a stamp loads as unstamped", "[gallery][GR-004]") {
// The back-compat case: pre-GR-004 files have no /embedder group at all. The
// absence must survive the round trip as an absence — a stamp naming no model
// would read as "checked and fine" to every consumer.
ActorGallery g;
ActorGallery::Actor a;
a.name = "Legacy Actor";
a.embeddings = {make_embedding(0.f)};
g.actors.push_back(a);
TempFile tf("gallery_unstamped.h5");
save_gallery(tf.path, g);
ActorGallery loaded = load_gallery(tf.path);
CHECK(loaded.embedder.empty());
}
/// TRACES: GR-004 | SR-001
TEST_CASE("legacy JSON galleries carry an optional embedder stamp", "[gallery][GR-004]") {
nlohmann::json j;
j["embedder"] = {{"model_name", "arcface_w600k_r50.onnx"},
{"model_sha256", kShaB},
{"embed_dim", 512}};
j["actors"] = nlohmann::json::array();
nlohmann::json ja;
ja["name"] = "JSON Actor";
ja["embeddings"] = nlohmann::json::array();
ja["embeddings"].push_back(std::vector<float>(512, 0.1f));
j["actors"].push_back(ja);
TempFile tf("gallery_json_stamp.json");
{ std::ofstream out(tf.path); out << j.dump(); }
ActorGallery g = load_gallery(tf.path);
CHECK(g.embedder.model_name == "arcface_w600k_r50.onnx");
CHECK(g.embedder.model_sha256 == kShaB);
}
/// TRACES: GR-004 | SR-001
TEST_CASE("matching embedder stamps pass", "[gallery][GR-004]") {
auto chk = compare_embedder_stamps(stamp("model.onnx", kShaA),
stamp("model.onnx", kShaA));
CHECK(chk.verdict == StampVerdict::match);
CHECK_FALSE(chk.fatal(false));
CHECK_FALSE(chk.fatal(true)); // a proven match is never fatal, even in strict mode
CHECK_NOTHROW(enforce_embedder_stamp(stamp("model.onnx", kShaA),
stamp("model.onnx", kShaA),
"g.h5", "model.onnx", true));
}
/// TRACES: GR-004 | SR-001
TEST_CASE("the hash decides, not the filename", "[gallery][GR-004]") {
// Same bytes under a different filename is the SAME model — a renamed or
// relocated file must not be treated as a different one.
auto same = compare_embedder_stamps(stamp("lvface.onnx", kShaA),
stamp("LVFace-B_Glint360K.onnx", kShaA));
CHECK(same.verdict == StampVerdict::match);
// Different bytes under the SAME filename is a DIFFERENT model — this is the
// in-place re-export a name-only stamp would miss entirely, and the reason the
// stamp carries a hash at all.
auto differ = compare_embedder_stamps(stamp("model.onnx", kShaA),
stamp("model.onnx", kShaB));
CHECK(differ.verdict == StampVerdict::mismatch);
}
/// TRACES: GR-004 | SR-001
TEST_CASE("mismatched embedder is fatal and names both sides", "[gallery][GR-004]") {
const auto built = stamp("LVFace-B_Glint360K.onnx", kShaA);
const auto loaded = stamp("arcface_w600k_r50.onnx", kShaB);
auto chk = compare_embedder_stamps(built, loaded, "cast.h5", "models/r50.onnx");
REQUIRE(chk.verdict == StampVerdict::mismatch);
CHECK(chk.fatal(false)); // no bypass: a mismatch is fatal in every mode
CHECK(chk.fatal(true));
// Both sides must be identifiable from the message alone.
CHECK(mentions(chk.message, "LVFace-B_Glint360K.onnx"));
CHECK(mentions(chk.message, "arcface_w600k_r50.onnx"));
CHECK(mentions(chk.message, kShaA));
CHECK(mentions(chk.message, kShaB));
CHECK(mentions(chk.message, "cast.h5"));
CHECK(mentions(chk.message, "models/r50.onnx"));
// ...and it must reach the caller as an error, not a log line.
CHECK_THROWS_AS(enforce_embedder_stamp(built, loaded, "cast.h5",
"models/r50.onnx", false),
std::runtime_error);
try {
enforce_embedder_stamp(built, loaded, "cast.h5", "models/r50.onnx", false);
FAIL("mismatch must throw");
} catch (const std::runtime_error& e) {
const std::string what = e.what();
CHECK(mentions(what, "LVFace-B_Glint360K.onnx"));
CHECK(mentions(what, "arcface_w600k_r50.onnx"));
}
}
/// TRACES: GR-004 | SR-001
TEST_CASE("differing embedding width is a mismatch", "[gallery][GR-004]") {
auto chk = compare_embedder_stamps(stamp("a.onnx", kShaA, 512),
stamp("a.onnx", kShaA, 256));
CHECK(chk.verdict == StampVerdict::mismatch);
CHECK(mentions(chk.message, "512"));
CHECK(mentions(chk.message, "256"));
}
/// TRACES: GR-004 | SR-001
TEST_CASE("an unstamped gallery warns by default and fails under strict",
"[gallery][GR-004]") {
// Decision recorded in src/gallery/embedder_stamp.hpp: unstamped is UNKNOWN,
// not known-bad, and every pre-GR-004 gallery is unstamped. Hard-failing them
// all would make the check something people disable rather than trust; so it
// warns loudly, names the risk, and is promotable to fatal for measurement runs.
EmbedderStamp none;
auto chk = compare_embedder_stamps(none, stamp("model.onnx", kShaA), "old.h5");
REQUIRE(chk.verdict == StampVerdict::unstamped);
CHECK_FALSE(chk.fatal(false));
CHECK(chk.fatal(true));
CHECK(mentions(chk.message, "old.h5"));
CHECK(mentions(chk.message, "model.onnx")); // the loaded side is still named
CHECK(mentions(chk.message, "UNKNOWN")); // ...and the gallery side is honest
CHECK_NOTHROW(enforce_embedder_stamp(none, stamp("model.onnx", kShaA),
"old.h5", "model.onnx", false));
CHECK_THROWS_AS(enforce_embedder_stamp(none, stamp("model.onnx", kShaA),
"old.h5", "model.onnx", true),
std::runtime_error);
}
/// TRACES: GR-004 | SR-001
TEST_CASE("an unidentifiable embedder against a stamped gallery is not silent",
"[gallery][GR-004]") {
// e.g. a replay whose dump predates GR-004: we know what built the gallery but
// not what produced the vectors being fed in. Unverifiable, so it must not
// report success.
auto chk = compare_embedder_stamps(stamp("model.onnx", kShaA), EmbedderStamp{},
"g.h5", "old dump.h5");
CHECK(chk.verdict == StampVerdict::unknown_embedder);
CHECK_FALSE(chk.fatal(false));
CHECK(chk.fatal(true));
CHECK(mentions(chk.message, "model.onnx"));
CHECK(mentions(chk.message, "old dump.h5"));
}
/// TRACES: GR-004 | SR-001
TEST_CASE("name-only agreement is a weak match, not a clean pass", "[gallery][GR-004]") {
// A TRT deployment can run from a prebuilt .engine with the .onnx absent, so
// no hash is computable. Names agreeing is evidence, not proof.
auto weak = compare_embedder_stamps(stamp("model.onnx", kShaA),
stamp("model.onnx", ""));
CHECK(weak.verdict == StampVerdict::weak_match);
CHECK_FALSE(weak.fatal(false));
CHECK(weak.fatal(true));
// Names disagreeing with no hash available is still a mismatch — the weaker
// evidence is enough to convict, just not to acquit.
auto bad = compare_embedder_stamps(stamp("lvface.onnx", ""),
stamp("arcface.onnx", ""));
CHECK(bad.verdict == StampVerdict::mismatch);
CHECK(mentions(bad.message, "lvface.onnx"));
CHECK(mentions(bad.message, "arcface.onnx"));
}
/// TRACES: GR-004 | SR-001
TEST_CASE("sha256 matches the published vectors", "[gallery][GR-004]") {
// Pins the in-tree FIPS 180-4 implementation against the standard vectors.
// This is what guarantees the C++ stamp and the Python (hashlib) stamp in
// scripts/sae_gallery.py agree on the same model file — without it the two
// halves of GR-004 could silently diverge and every check would be a mismatch.
CHECK(sha256_hex("") ==
"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855");
CHECK(sha256_hex("abc") ==
"ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad");
CHECK(sha256_hex("abcdbcdecdefdefgefghfghighijhijkijkljklmklmnlmnomnopnopq") ==
"248d6a61d20638b8e5c026930c3e6039a33ce45964ff2167f6ecedd419db06c1");
// Multi-block input, exercising the length-padding path past 64 bytes.
CHECK(sha256_hex(std::string(1000, 'a')) ==
"41edece42d63e8d9bf515a9ba6932e1c20cbc9f5a5d134645adb5db1b9737ea3");
}
/// TRACES: GR-004 | SR-001
TEST_CASE("make_embedder_stamp hashes a real file and degrades gracefully",
"[gallery][GR-004]") {
// Stands in for an ONNX: the stamp does not care what the bytes mean.
TempFile tf("fake_model.onnx");
{ std::ofstream out(tf.path, std::ios::binary); out << "abc"; }
EmbedderStamp s = make_embedder_stamp(tf.path);
CHECK(s.model_sha256 ==
"ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad");
CHECK_FALSE(s.model_name.empty());
CHECK(s.model_name.find('/') == std::string::npos); // basename, not full path
// A model path that does not exist still yields a comparable name-only stamp
// rather than an empty one, which is what keeps engine-only deployments usable.
EmbedderStamp missing = make_embedder_stamp("/nonexistent/models/foo.onnx");
CHECK(missing.model_name == "foo.onnx");
CHECK(missing.model_sha256.empty());
CHECK_FALSE(missing.empty());
CHECK(make_embedder_stamp("").empty());
}
tests/test_replay_fixtures.cpp
-258
View File
@@ -1,258 +0,0 @@
// Replay tests — the real tracker and registry driven from committed fixtures.
//
// TRACES: AR-012, AR-013, AR-004, VR-001, VR-002 | IT-001
//
// Tier T2: composition, not units. The registry tests construct awkward states
// directly; these check that the pieces behave when wired together and fed real
// footage — 480x360 public-domain clips at 5 fps, with the cuts, gaps and
// crowded frames that actual film produces and synthetic input does not.
//
// No GPU and no model: the fixtures are HDF5 dumps taken after embedding, so
// everything here is CPU maths. That is what lets this run on the CI host at
// all (see docs/requirements.md, "CI never calls a model").
//
// Driving the node functors directly rather than through a KPN network is
// deliberate: functors are plain objects, so there are no threads, no channels
// and no scheduling — the same input gives the same output every time, which is
// exactly what a fixture-based test needs.
#include <catch2/catch_test_macros.hpp>
#include "config.hpp"
#include "evidence_discount.hpp"
#include "nodes/face_tracker_node.hpp"
#include "track_registry.hpp"
#include "types.hpp"
#include <H5Cpp.h>
#include <algorithm>
#include <cmath>
#include <memory>
#include <string>
#include <vector>
namespace {
// ── Fixture reader ───────────────────────────────────────────────────────────
// The flat/ragged layout of scripts/optimizer/SCHEMA.md: per-face arrays
// concatenated, with a per-frame index table pointing into them.
struct Dump {
std::vector<double> ts;
std::vector<uint8_t> is_cut;
std::vector<int64_t> face_offset;
std::vector<int32_t> face_count;
std::vector<Embedding> emb;
std::vector<float> bbox; // 4 per face
std::string embedder;
std::size_t frames() const { return ts.size(); }
std::size_t faces() const { return emb.size(); }
};
template <typename T>
std::vector<T> read1d(H5::Group& g, const char* name, const H5::DataType& dt) {
H5::DataSet ds = g.openDataSet(name);
hsize_t n = 0;
ds.getSpace().getSimpleExtentDims(&n, nullptr);
std::vector<T> out(n);
if (n) ds.read(out.data(), dt);
return out;
}
Dump load(const std::string& path) {
H5::H5File f(path, H5F_ACC_RDONLY);
H5::Group frames = f.openGroup("frames");
H5::Group faces = f.openGroup("faces");
Dump d;
d.ts = read1d<double>(frames, "timestamp_sec", H5::PredType::NATIVE_DOUBLE);
d.is_cut = read1d<uint8_t>(frames, "is_cut", H5::PredType::NATIVE_UINT8);
d.face_offset = read1d<int64_t>(frames, "face_offset", H5::PredType::NATIVE_INT64);
d.face_count = read1d<int32_t>(frames, "face_count", H5::PredType::NATIVE_INT32);
H5::DataSet e = faces.openDataSet("embedding");
hsize_t dims[2]{0, 0};
e.getSpace().getSimpleExtentDims(dims, nullptr);
std::vector<float> flat(dims[0] * dims[1]);
if (!flat.empty()) e.read(flat.data(), H5::PredType::NATIVE_FLOAT);
d.emb.resize(dims[0]);
for (hsize_t i = 0; i < dims[0]; ++i)
std::copy_n(flat.begin() + i * dims[1], 512, d.emb[i].begin());
// bbox is 2-D [N,4]; reading it with the 1-D helper would size the buffer
// from the first extent only and then read four times that many floats.
{
H5::DataSet bs = faces.openDataSet("bbox");
hsize_t bd[2]{0, 0};
bs.getSpace().getSimpleExtentDims(bd, nullptr);
d.bbox.resize(bd[0] * bd[1]);
if (!d.bbox.empty()) bs.read(d.bbox.data(), H5::PredType::NATIVE_FLOAT);
}
// GR-004: the dump records which embedder produced it, so a replay cannot
// be silently scored against a gallery from a different model.
if (f.attrExists("embedder_model")) {
// Written as a variable-length string (embedding_dump_node.hpp:99), so
// the read must name the same type explicitly.
H5::StrType vlen(H5::PredType::C_S1, H5T_VARIABLE);
f.openAttribute("embedder_model").read(vlen, d.embedder);
}
return d;
}
std::string fixture(const char* name) {
return std::string(SAE_TEST_FIXTURES_DIR) + "/dumps/" + name;
}
// ── Harness ──────────────────────────────────────────────────────────────────
struct Replay {
std::vector<DeadTrack> claims;
std::vector<int> track_ids; // per face, in fixture order
std::size_t faces_seen{0};
};
Replay run(const Dump& d, double extinction = 10.0) {
Replay r;
TrackRegistry::Config rc;
rc.extinction_sec = extinction;
auto cal = [](float cos) { return std::max(0.f, cos); };
auto reg = std::make_shared<TrackRegistry>(rc, EvidenceDiscounter(cal));
reg->on_track_dead([&r](const DeadTrack& t) { r.claims.push_back(t); });
Config cfg;
cfg.track_assoc_min_prob = 0.5f;
FaceTrackerFunc ft(cfg, reg, cal);
for (std::size_t i = 0; i < d.frames(); ++i) {
EmbeddedSceneFrame ef;
ef.source.timestamp_sec = d.ts[i];
ef.source.is_cut = d.is_cut[i] != 0;
const int64_t off = d.face_offset[i];
const int32_t n = d.face_count[i];
for (int32_t k = 0; k < n; ++k) {
DetectedFace face;
const float* b = &d.bbox[(off + k) * 4];
face.bbox = cv::Rect2f(b[0], b[1], b[2], b[3]);
face.confidence = 1.0f;
ef.faces.push_back(face);
ef.crops.push_back(cv::Mat());
ef.embeddings.push_back(d.emb[off + k]);
}
r.faces_seen += static_cast<std::size_t>(n);
auto out = ft(std::move(ef));
for (int id : out.track_ids) r.track_ids.push_back(id);
}
reg->flush(d.ts.empty() ? 0.0 : d.ts.back());
return r;
}
} // namespace
// ── AR-004 / VR-001 — the fixtures are intact and self-describing ────────────
TEST_CASE("fixtures are complete and carry their embedder identity",
"[replay][AR-004][VR-001]") {
// Frame counts are exact rather than approximate. Before node outputs
// blocked on a full channel, generation lost most of a clip and what it
// lost depended on timing — these numbers could not have been asserted.
struct Expect { const char* file; std::size_t frames, faces; };
const Expect all[] = {
{"bali_13.h5", 385, 693},
{"bali_27.h5", 335, 335},
{"bali_28.h5", 345, 368},
{"bali_31.h5", 145, 203},
{"bali_46.h5", 385, 140},
};
for (const auto& x : all) {
INFO(x.file);
Dump d = load(fixture(x.file));
CHECK(d.frames() == x.frames);
CHECK(d.faces() == x.faces);
CHECK(d.embedder == "LVFace-B_Glint360K.onnx");
// face_offset must be contiguous: a gap means faces went missing
// between frames, which no consumer could detect.
int64_t running = 0;
for (std::size_t i = 0; i < d.frames(); ++i) {
REQUIRE(d.face_offset[i] == running);
running += d.face_count[i];
}
CHECK(static_cast<std::size_t>(running) == d.faces());
}
}
// ── VR-002 — replay is deterministic ─────────────────────────────────────────
TEST_CASE("replaying a fixture twice gives identical tracks", "[replay][VR-002]") {
// The property the whole fixture strategy rests on. If this fails, every
// golden output derived from a fixture is unreliable and the CI replay
// tier is worthless.
Dump d = load(fixture("bali_28.h5"));
Replay a = run(d);
Replay b = run(d);
REQUIRE(a.track_ids.size() == b.track_ids.size());
CHECK(a.track_ids == b.track_ids);
REQUIRE(a.claims.size() == b.claims.size());
for (std::size_t i = 0; i < a.claims.size(); ++i) {
CHECK(a.claims[i].first_seen == b.claims[i].first_seen);
CHECK(a.claims[i].last_seen == b.claims[i].last_seen);
}
}
// ── AR-012 / AR-013 — window invariants on real footage ──────────────────────
TEST_CASE("every face is assigned a track and every track closes",
"[replay][AR-012]") {
Dump d = load(fixture("bali_13.h5"));
Replay r = run(d);
CHECK(r.track_ids.size() == r.faces_seen);
for (int id : r.track_ids) CHECK(id >= 0); // nothing silently unassigned
// flush() must leave nothing behind: a track still open at EOF would be a
// window that never reaches the output.
CHECK(r.claims.size() > 0);
}
TEST_CASE("windows are well-formed and inside the clip", "[replay][AR-013]") {
for (const char* f : {"bali_13.h5", "bali_27.h5", "bali_28.h5",
"bali_31.h5", "bali_46.h5"}) {
INFO(f);
Dump d = load(fixture(f));
Replay r = run(d);
const double t0 = d.ts.front(), t1 = d.ts.back();
for (const auto& c : r.claims) {
// A window ends at the last sighting, never after it — so it can
// never extend past the footage that produced it.
CHECK(c.first_seen <= c.last_seen);
CHECK(c.first_seen >= t0);
CHECK(c.last_seen <= t1);
}
}
}
TEST_CASE("a longer extinction window yields fewer, longer tracks",
"[replay][AR-013]") {
// The timeout decides whether a gap is absorbed into one window or splits
// it in two, so lengthening it must merge tracks rather than multiply them.
// On sparse footage this is the difference the constant actually makes.
Dump d = load(fixture("bali_46.h5")); // 140 faces over 385 frames
Replay tight = run(d, /*extinction=*/1.0);
Replay loose = run(d, /*extinction=*/30.0);
CHECK(loose.claims.size() <= tight.claims.size());
}
// ── AR-007 — cuts are exercised by the corpus, not just by construction ──────
TEST_CASE("the cut-heavy fixture actually contains cuts", "[replay][AR-007]") {
// Guards the corpus rather than the code: if a regeneration produced a
// fixture with no cuts, the association tests above would still pass while
// silently testing nothing about viewpoint changes.
Dump d = load(fixture("bali_28.h5"));
const int cuts = std::count(d.is_cut.begin(), d.is_cut.end(), uint8_t{1});
CHECK(cuts >= 5);
}
tests/test_track_registry.cpp
-332
View File
@@ -1,332 +0,0 @@
// Unit tests for TrackRegistry (track_registry.hpp): presence as track extent.
//
// TRACES: AR-012, AR-013, AR-014, AR-015, AR-016, AR-017 | UT-001
//
// Pure, GPU-free, model-free — drives the registry directly with synthetic
// timestamps and evidence. Node functors and this registry are plain objects
// constructed outside the KPN network, so the awkward cases can be built
// exactly rather than hunted for in a clip: a gap one frame under the timeout,
// a belief swap, two live tracks converging on one actor, a film ending
// mid-track.
#include <catch2/catch_test_macros.hpp>
#include "track_registry.hpp"
#include "evidence_discount.hpp"
#include <algorithm>
#include <vector>
namespace {
Embedding axis(int slot) {
Embedding e{};
e[slot] = 1.0f;
return e;
}
// Collects the claims a registry emits, which is the whole observable output.
struct Sink {
std::vector<DeadTrack> claims;
void attach(TrackRegistry& r) {
r.on_track_dead([this](const DeadTrack& d) { claims.push_back(d); });
}
const DeadTrack* forActor(int a) const {
for (const auto& c : claims) if (c.actor_idx == a) return &c;
return nullptr;
}
};
// A discounter whose calibration is deliberately trivial, so the tests exercise
// registry behaviour rather than a fitted sigmoid.
EvidenceDiscounter disc() {
return EvidenceDiscounter([](float cos) { return std::max(0.f, cos); });
}
TrackRegistry::Config cfg(double extinction = 5.0, float own = 2.0f) {
TrackRegistry::Config c;
c.extinction_sec = extinction;
c.ownership_logodds = own;
return c;
}
} // namespace
// ── AR-012 — the change this whole redesign exists for ───────────────────────
TEST_CASE("window starts at first sighting, not at first recognition", "[registry][AR-012]") {
TrackRegistry reg(cfg(), disc());
Sink sink; sink.attach(reg);
int id;
{ auto f = reg.begin_frame(10.0); id = f.create(10.0, axis(0)); }
// Seen for 20s but only recognised at the very end — the pose was wrong
// until then. This is the case the old per-frame design got wrong: it would
// have reported presence starting at 30, not 10.
for (double t = 11.0; t <= 30.0; t += 1.0) {
auto f = reg.begin_frame(t);
f.mark_seen(id, t, axis(0));
}
reg.observe(id, 7, 0.99f, axis(7));
{ auto f = reg.begin_frame(31.0); f.mark_lost(id, 30.0); }
reg.tick(40.0);
REQUIRE(sink.claims.size() == 1);
CHECK(sink.claims[0].actor_idx == 7);
CHECK(sink.claims[0].first_seen == 10.0); // ← not 30.0
CHECK(sink.claims[0].last_seen == 30.0);
}
// ── AR-013 — the asymmetry that removes the old over-claim ───────────────────
TEST_CASE("interior gaps are absorbed; the trailing cool-down is not",
"[registry][AR-013]") {
TrackRegistry reg(cfg(/*extinction=*/5.0), disc());
Sink sink; sink.attach(reg);
int id;
{ auto f = reg.begin_frame(0.0); id = f.create(0.0, axis(0)); }
reg.observe(id, 3, 0.99f, axis(3));
// Off screen at 10, back at 13 — inside the timeout, so the same track
// continues and the actor is claimed present *through* the gap.
{ auto f = reg.begin_frame(10.0); f.mark_lost(id, 10.0); }
{ auto f = reg.begin_frame(13.0); f.mark_seen(id, 13.0, axis(0)); }
CHECK(sink.claims.empty()); // nothing closed
CHECK(reg.live() == 1);
// Lost for good at 20. The window must end there, not at the death time.
{ auto f = reg.begin_frame(20.0); f.mark_lost(id, 20.0); }
reg.tick(20.0 + 5.0 + 0.001);
REQUIRE(sink.claims.size() == 1);
CHECK(sink.claims[0].first_seen == 0.0);
CHECK(sink.claims[0].last_seen == 20.0); // ← not 25.001
}
TEST_CASE("a gap past the timeout yields two tracks, not one", "[registry][AR-013]") {
TrackRegistry reg(cfg(/*extinction=*/5.0), disc());
Sink sink; sink.attach(reg);
int a;
{ auto f = reg.begin_frame(0.0); a = f.create(0.0, axis(0)); }
reg.observe(a, 1, 0.99f, axis(1));
{ auto f = reg.begin_frame(10.0); f.mark_lost(a, 10.0); }
reg.tick(30.0); // well past extinction
REQUIRE(sink.claims.size() == 1);
CHECK(sink.claims[0].last_seen == 10.0);
// A face reappearing after the timeout is genuinely a new track: past the
// re-acquisition window there are no grounds to assert continuity.
int b;
{ auto f = reg.begin_frame(31.0); b = f.create(31.0, axis(0)); }
CHECK(b != a);
}
// ── AR-016 — the silent-loss guard ───────────────────────────────────────────
TEST_CASE("EOF flush closes tracks still on screen", "[registry][AR-016]") {
TrackRegistry reg(cfg(), disc());
Sink sink; sink.attach(reg);
int id;
{ auto f = reg.begin_frame(100.0); id = f.create(100.0, axis(0)); }
reg.observe(id, 5, 0.99f, axis(5));
// A film almost always ends with faces on screen; these have not timed out.
reg.flush(/*final_ts=*/120.0);
REQUIRE(sink.claims.size() == 1);
CHECK(sink.claims[0].actor_idx == 5);
CHECK(sink.claims[0].last_seen == 120.0);
sink.claims.clear();
reg.flush(130.0);
CHECK(sink.claims.empty()); // idempotent
CHECK(reg.live() == 0);
}
TEST_CASE("flush closes a lost-but-unreaped track at its last sighting",
"[registry][AR-016]") {
TrackRegistry reg(cfg(/*extinction=*/60.0), disc());
Sink sink; sink.attach(reg);
int id;
{ auto f = reg.begin_frame(0.0); id = f.create(0.0, axis(0)); }
reg.observe(id, 2, 0.99f, axis(2));
{ auto f = reg.begin_frame(10.0); f.mark_lost(id, 10.0); }
reg.flush(/*final_ts=*/50.0);
REQUIRE(sink.claims.size() == 1);
CHECK(sink.claims[0].last_seen == 10.0); // last sighting, not EOF
}
// ── AR-014 — belief swap is a track boundary, not a correction ───────────────
TEST_CASE("belief swap closes one window and opens another", "[registry][AR-014]") {
TrackRegistry reg(cfg(), disc());
Sink sink; sink.attach(reg);
int id;
{ auto f = reg.begin_frame(0.0); id = f.create(0.0, axis(0)); }
reg.observe(id, 1, 0.99f, axis(1)); // owned by actor 1
{ auto f = reg.begin_frame(5.0); f.mark_lost(id, 5.0); }
// The swap must out-accumulate the incumbent, not merely tie it: one
// contrary observation is noise, and a tie leaves ownership where it is.
reg.observe(id, 2, 0.99f, axis(2));
reg.observe(id, 2, 0.99f, axis(3));
CHECK(reg.belief_swaps() == 1);
REQUIRE(sink.claims.size() == 1);
CHECK(sink.claims[0].actor_idx == 1);
CHECK(sink.claims[0].last_seen == 5.0); // closed at its last sighting
// The successor is a distinct track, so nothing blends the two people.
reg.flush(9.0);
const DeadTrack* second = sink.forActor(2);
REQUIRE(second != nullptr);
CHECK(second->track_id != id);
CHECK(second->first_seen == 5.0); // abuts, does not overlap
}
// ── AR-015 — identity contradiction as a cut detector ────────────────────────
TEST_CASE("two live tracks owned by one actor is counted", "[registry][AR-015]") {
TrackRegistry reg(cfg(), disc());
Sink sink; sink.attach(reg);
int a, b;
{ auto f = reg.begin_frame(0.0); a = f.create(0.0, axis(0)); b = f.create(0.0, axis(1)); }
reg.observe(a, 9, 0.99f, axis(9));
CHECK(reg.actor_conflicts() == 0);
// One person cannot be in two places at once, so this is a missed camera or
// scene change that split them — detected on the update that causes it.
reg.observe(b, 9, 0.99f, axis(9));
CHECK(reg.actor_conflicts() == 1);
}
// ── AR-017 / diagnostics ─────────────────────────────────────────────────────
TEST_CASE("an unowned track emits no claim", "[registry][AR-012]") {
TrackRegistry reg(cfg(), disc());
Sink sink; sink.attach(reg);
int id;
{ auto f = reg.begin_frame(0.0); id = f.create(0.0, axis(0)); }
reg.observe(id, 4, 0.62f, axis(4)); // never clears the ownership threshold
{ auto f = reg.begin_frame(1.0); f.mark_lost(id, 1.0); }
reg.tick(100.0);
// Someone was there, but nothing can be claimed about who.
CHECK(sink.claims.size() == 1);
CHECK(sink.claims[0].actor_idx == -1);
}
TEST_CASE("claims carry the belief that justified them", "[registry][AR-017]") {
TrackRegistry reg(cfg(), disc());
Sink sink; sink.attach(reg);
int id;
{ auto f = reg.begin_frame(0.0); id = f.create(0.0, axis(0)); }
reg.observe(id, 6, 0.99f, axis(6));
reg.flush(1.0);
REQUIRE(sink.claims.size() == 1);
CHECK(sink.claims[0].belief > 0.9f); // logistic(4.0) ≈ 0.982
CHECK(sink.claims[0].observations == 1);
}
TEST_CASE("a vote for a reaped track is dropped and counted", "[registry][AR-013]") {
TrackRegistry reg(cfg(/*extinction=*/1.0), disc());
Sink sink; sink.attach(reg);
int id;
{ auto f = reg.begin_frame(0.0); id = f.create(0.0, axis(0)); }
{ auto f = reg.begin_frame(1.0); f.mark_lost(id, 1.0); }
reg.tick(10.0); // reaped
// The matcher runs downstream of the tracker, so a late vote is expected.
// Silently ignoring it would hide a timeout shorter than the matcher's lag.
reg.observe(id, 3, 0.99f, axis(3));
CHECK(reg.dropped_votes() == 1);
}
TEST_CASE("a single-frame track yields a zero-length window", "[registry][AR-012]") {
TrackRegistry reg(cfg(), disc());
Sink sink; sink.attach(reg);
int id;
{ auto f = reg.begin_frame(42.0); id = f.create(42.0, axis(0)); }
reg.observe(id, 8, 0.99f, axis(8));
{ auto f = reg.begin_frame(43.0); f.mark_lost(id, 42.0); }
reg.tick(100.0);
REQUIRE(sink.claims.size() == 1);
CHECK(sink.claims[0].first_seen == 42.0);
CHECK(sink.claims[0].last_seen == 42.0);
}
// ── AR-025 — correlated observations must not accumulate as independent ──────
TEST_CASE("repeated identical views do not reach the certainty of distinct ones",
"[registry][AR-025]") {
// Thirty frames of the same face at the same angle is not thirty pieces of
// evidence. Without discounting, log-odds accumulate linearly and the
// posterior saturates on what is effectively a single measurement.
TrackRegistry same(cfg(), disc());
TrackRegistry varied(cfg(), disc());
Sink s_same, s_varied;
s_same.attach(same);
s_varied.attach(varied);
int a, b;
{ auto f = same.begin_frame(0.0); a = f.create(0.0, axis(0)); }
{ auto f = varied.begin_frame(0.0); b = f.create(0.0, axis(0)); }
for (int i = 0; i < 8; ++i) {
same.observe(a, 1, 0.9f, axis(0)); // the identical view, every time
varied.observe(b, 1, 0.9f, axis(i + 1)); // a genuinely new look each time
}
same.flush(1.0);
varied.flush(1.0);
REQUIRE(s_same.claims.size() == 1);
REQUIRE(s_varied.claims.size() == 1);
// Same raw observation count, but only the varied track earned the evidence.
CHECK(s_same.claims[0].observations == s_varied.claims[0].observations);
CHECK(s_same.claims[0].effective_obs < s_varied.claims[0].effective_obs);
CHECK(s_same.claims[0].effective_obs < 2.0f); // ~one view's worth
}
TEST_CASE("the first observation on a track always counts in full",
"[registry][AR-025]") {
// There is nothing for it to be redundant with.
TrackRegistry reg(cfg(), disc());
Sink sink; sink.attach(reg);
int id;
{ auto f = reg.begin_frame(0.0); id = f.create(0.0, axis(0)); }
reg.observe(id, 1, 0.9f, axis(0));
reg.flush(1.0);
REQUIRE(sink.claims.size() == 1);
CHECK(sink.claims[0].effective_obs == 1.0f);
}
TEST_CASE("the registry takes a probability, not a cosine", "[registry][AR-024]") {
// A posterior at the decision boundary must not move belief at all: 0.5
// carries no information either way, and its log-odds are zero. Feeding a
// raw cosine here would be silently wrong rather than obviously so, which
// is why the conversion lives inside the registry.
TrackRegistry reg(cfg(), disc());
Sink sink; sink.attach(reg);
int id;
{ auto f = reg.begin_frame(0.0); id = f.create(0.0, axis(0)); }
reg.observe(id, 1, 0.5f, axis(0));
reg.flush(1.0);
REQUIRE(sink.claims.size() == 1);
CHECK(sink.claims[0].actor_idx == -1); // never owned
}
traceability.toml
+1 -11
View File
@@ -17,16 +17,6 @@ languages = ["cpp", "python"]
source_roots = ["src", "tests", "scripts", "experiments", "eval"]
# exclude_dirs is deliberately NOT set. The tool's defaults already exclude
# `vendor` (among __pycache__, external, build, node_modules and friends), which
# covers the submodule at scripts/vendor/jray-project — that is not this repo's
# code, and its parser tests carry literal TRACES: strings that would otherwise
# be credited here as coverage.
#
# Note the key REPLACES the defaults rather than adding to them, and matching is
# on path components, not prefixes: setting it to ["scripts/vendor"] both fails
# to match anything and silently drops every default exclusion.
# CI is an Intel N100 with no discrete GPU. T4 is deliberately absent: a
# requirement verifiable only on GPU hardware is reported as tagged but
# unexecuted and never counted as covered, because counting a test that cannot
@@ -41,4 +31,4 @@ min_coverage = 0.0
# The system spec owning PR/SR is vendored per-component as a submodule. Point
# at it once that lands to turn on PR/SR orphan checking:
system_spec = "scripts/vendor/jray-project/SPEC.md"
# system_spec = "scripts/vendor/jray-project/SPEC.md"