From 2919ed68d17c7586a960e32c328b62df8cc8227a Mon Sep 17 00:00:00 2001 From: Duncan Tourolle Date: Thu, 30 Jul 2026 18:33:32 +0200 Subject: [PATCH] docs: audit findings, verification tiers, CI image, artifact storage MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Corrections from the dump audit and the completed agent work: - AR-010 is not started, not in progress: is_scene_boundary has no producer anywhere. SceneDetectorFunc is a terminal sink writing scenes.json and never annotates the frame, so the field is permanently false and the dump column a constant 0. A replay test of the frame-dependent track_alpha would pass vacuously — the worst failure mode for a verification gate. - T1 (functor-level) becomes the primary verification tier, not T2. KPN node functors are plain callables constructed outside the network, so a node is tested by calling operator() with hand-built inputs. That removes four hazards at once: fixture provenance, replay-from-frame-0, cross-test state leakage, and replay-harness nondeterminism. It also means a dead upstream producer no longer blocks testing its consumer. - VR-010 (dump provenance) and VR-011 (replay harness rewrite) added. A dump made with LVFace is currently byte-indistinguishable from one made with w600k-R50 — the GR-004 problem again, in the dump. - Four requirements had no verification tier at all; the traceability gate found them. - DP-007: CI builder image, CPU-only, pinned by tag in the Gitea container registry. Corpus fixtures go to the package registry rather than LFS: LFS is pulled on clone and would tax every developer for data only CI reads. - IR-004/005/007/008 marked done; the v1 DSP parameters they had to pin are now normative in the server spec. Co-Authored-By: Claude Opus 5 TRACES: AR-010, DP-007, IR-004, IR-005, IR-007, IR-008, VR-001, VR-010, VR-011 | SR-002, SR-003 --- docs/SPEC.md | 68 +++++++++++++++++++++++++++ docs/requirements.md | 106 +++++++++++++++++++++++++++++++++---------- 2 files changed, 151 insertions(+), 23 deletions(-) diff --git a/docs/SPEC.md b/docs/SPEC.md index 65742a2..b54b5d0 100644 --- a/docs/SPEC.md +++ b/docs/SPEC.md @@ -154,6 +154,26 @@ Two distinct signals, deliberately kept separate: - **`is_scene_boundary`** — opt-in (`--scene-detect`). TransNetV2 over a densely decoded, downscaled stream flags a *true shot/scene boundary*. +> **`is_scene_boundary` currently has no producer.** `grep -rn is_scene_boundary +> src/` finds no assignment anywhere: `SceneDetectorFunc` is a *terminal sink* +> (`main.cpp:298-300`, `kpn::out<>`) that writes `scenes.json` and never +> annotates the `Frame` flowing to the face pipeline. The field is therefore +> always `false`, and the dump column (`embedding_dump_node.hpp:38`) is a +> constant 0. Compounding it, `main.cpp:280` returns from the +> `--dump-embeddings` branch *before* the `scene_detect` branch at `:296`, so no +> dump-producing path even instantiates the detector. +> +> This makes AR-010 **not implemented**, not "in progress" — and it means a T2 +> test of the frame-dependent `track_alpha` (AR-007) would **pass vacuously**, +> which is the worst possible failure for a verification gate. The fix is in the +> producer, not the schema: make `SceneDetectorFunc` a pass-through (or add a +> boundary annotator before the decimator) and add the scene branch to +> `dump_embeddings.cpp`. **No `schema_version` bump** — the column exists and +> merely stops being constant. +> +> Fixtures generated before the fix must be marked in provenance, since `0` is +> presently indistinguishable from "no boundary here". + Both feed AR-007 as **association hints**: they tell the tracker that spatial continuity is broken and that association should weight embedding over IoU. Neither ends a presence window (AR-012). @@ -770,6 +790,54 @@ prerequisite. **Gap:** installer unbuilt. +## DP-007 — CI build image + +CI runs on an Intel N100 with no discrete GPU, so the test build must configure +**CPU-only** and must not require CUDA, TensorRT or ROCm: + +``` +-DSAE_INFERENCE_BACKEND=ORT -DSAE_GEMM_BACKEND=CPU +``` + +A prebuilt container image supplies the toolchain, published to the **Gitea +container registry** and pinned by tag — matching the `jellytau-builder` +precedent. Building dependencies per CI run is untenable on an N100, and OpenCV 5 +from source would dominate every run. + +The same registry stores corpus dump fixtures as generic packages (see the +fixtures table in `requirements.md`). Rebuild the image when its dependency set +changes, not per run, and pin CI to a tag rather than `latest` so a rebuild +cannot silently change what a green build meant. + +**Required in the image:** + +| Dependency | Why | +|---|---| +| CMake, C++ toolchain, pkg-config | Build | +| **OpenCV 5** | `CMakeLists.txt:25` prefers 5, falls back to 4. The branch targets 5, so the image should carry it — it is not yet in most distro repos and building it per-run is prohibitive | +| HDF5 (C++) | Galleries are HDF5-native; also the dump format | +| FFmpeg dev libs — `libavformat`, `libavcodec`, `libavutil`, `libswscale`, **`libswresample`** | Decode. See the note below on swresample | +| Python 3 + numpy, h5py, scipy | Python-side tests, replay, traceability tooling | +| Catch2, nlohmann/json | **Vendored into the image, not fetched.** Both are `FetchContent`-ed today (`CMakeLists.txt:220`, `tests/CMakeLists.txt:8`), which makes every CI run depend on GitHub reachability | + +**Deliberately excluded:** CUDA, TensorRT, ROCm — no GPU to use them. Also the +ONNX Runtime *GPU* providers; only the CPU provider is relevant, and only for T3 +smoke tests. + +**Models are not baked into the image.** The seven ONNX files total ~725 MB and +live in Git LFS. T1/T2 tests are model-free by design +(`tests/CMakeLists.txt:1-4`), so the default image needs none. T3 smoke tests +require a model and should pull it via LFS in a separate job rather than +inflating the image tenfold for a minority of tests. + +**`libswresample` is a real gap, not a formality.** The current +`pkg_check_modules` list (`CMakeLists.txt:200-203`) covers avformat, avcodec, +avutil and swscale but **not** swresample — which IR-004 needs to downmix to mono +and resample to 11025 Hz. It must be added alongside the audio-signature work. + +**Gap:** entire requirement. The image does not exist, and no CI config is +present in this repo. + ## DP-006 — Gallery maintenance as a background concern - Incremental gallery refresh runs on a timer (`gallery_scan_interval`, default diff --git a/docs/requirements.md b/docs/requirements.md index 6d2f909..98c562d 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -37,7 +37,7 @@ Status: `Done` · `In Progress` · `Planned` · `TBD` · `Withdrawn` | AR-007 | Associate detections by IoU + embedding, with **frame-dependent** weighting | SR-002 | High | In Progress | | AR-008 | One track pool keyed on `last_seen`; no separate revival path | SR-002 | High | Planned | | AR-009 | Camera-cut detection (histogram) as an association hint | SR-002 | High | Done | -| AR-010 | Scene-boundary detection (TransNetV2) as an association hint | SR-002 | Medium | In Progress | +| AR-010 | Scene-boundary detection (TransNetV2) as an association hint | SR-002 | Medium | **Not started** — `is_scene_boundary` has no producer; `SceneDetectorFunc` is a terminal sink and never annotates the frame | | AR-011 | **Every model is fed the input it was trained for** — cost reduced by running less often, never by degrading one inference | SR-002 | High | Planned | | AR-012 | Presence follows **track extent**, not per-frame recognition | **SR-002** | High | Planned | | AR-013 | `last_seen` optional state machine; window ends at last sighting, never after | SR-002 | High | Planned | @@ -74,10 +74,10 @@ Status: `Done` · `In Progress` · `Planned` · `TBD` · `Withdrawn` | IR-001 | Emit the JRay truth format as sibling `.jray.json` | SR-003 | High | Done | | IR-002 | Windows carry belief + route; `extraction.*` carries `extinction_sec`, `gallery_scope` | SR-003 | High | Planned | | IR-003 | Output written **after** the deferred pass, not at EOF | SR-003 | High | Planned | -| IR-004 | Compute the audio signature exactly per server spec §3 | SR-003 | Medium | Planned | -| IR-005 | Golden-vector fixture shared with the plugin repo to prove bit-exactness | SR-003 | High | Planned | -| IR-007 | Media < 120 s: emit no signature, apply no sync offset — identical rule in both producers | SR-003 | Low | Planned | -| IR-008 | Emit and honour the signature's own `v1:` version prefix | SR-003 | Low | Planned | +| IR-004 | Compute the audio signature exactly per server spec §3 | SR-003 | Medium | **Done** — `src/audio_signature.*`; not yet emitted into the truth file (IR-002) | +| IR-005 | Golden-vector fixture shared with the plugin repo to prove bit-exactness | SR-003 | High | **Done** — `tests/fixtures/audio/`; v1 parameters now normative in server spec §3 | +| IR-007 | Media < 120 s: emit no signature, apply no sync offset — identical rule in both producers | SR-003 | Low | **Done** | +| IR-008 | Emit and honour the signature's own `v1:` version prefix | SR-003 | Low | **Done** | | IR-006 | Jellyfin round-trip: pull pending queue, push complete results only | SR-001 | High | Done | ## Gallery (GR) @@ -107,6 +107,8 @@ Status: `Done` · `In Progress` · `Planned` · `TBD` · `Withdrawn` | VR-007 | Expansion band, clustering threshold, and deferred-pass ablation | PR-002 | Medium | Planned | | VR-008 | Gallery scaling benchmark — throughput vs gallery size | PR-002 | Medium | Planned | | VR-009 | Verify accumulated posteriors are calibrated against held-out tracks | PR-002 | High | Planned | +| VR-010 | Dump provenance attributes — embedder model, detector settings, `dense_scale`, `scene_detect`, sample rate | PR-002 | **High** | Planned | +| VR-011 | Rewrite the replay harness for the post-AR-012 output contract | PR-002 | High | Planned | --- @@ -120,21 +122,62 @@ Four tiers, in decreasing order of preference: | Tier | Runs in CI | What it covers | |---|---|---| -| **T1 — CPU unit** | Yes | Pure logic: registry state machine, belief accumulation, clustering, band admission, calibration maths | -| **T2 — Replay** | Yes | Real pipeline nodes driven from an HDF5 fixture — no GPU, no video | +| **T1 — Functor unit** | Yes | A KPN node's `operator()` driven directly with hand-built inputs | +| **T2 — Replay** | Yes | The composed pipeline driven from an HDF5 fixture — no GPU, no video | | **T3 — CPU inference** | Yes, slowly | ORT CPU provider over a handful of frames; smoke tests only | | **T4 — GPU** | **No** | Throughput, TRT engines, large-gallery GEMM | -**T2 is the reason this is workable.** The HDF5 dump (VR-001) captures state -after decode → detect → align → embed and before tracking and matching, so -everything downstream — which is where nearly all of the new design lives — is -cheap CPU maths replayable from a fixture. Tracking, presence windows, belief -accumulation, expansion, deferred re-identification and clustering are all -verifiable on an N100 at full fidelity, not in miniature. +### T1 is the primary tier, and KPN is why -That was already true for the optimizer. It now doubles as the CI strategy, which -is a strong argument for keeping the dump schema honest (VR-001) and for the -replay driving the *real* nodes rather than a reimplementation (VR-002). +**Node functors are plain callable structs, constructed independently of the +network that wraps them** (`main.cpp:186-207` builds them as stack objects; +`ObjectNode` merely adapts them). So a node is testable by constructing it and +calling `operator()` — no channels, no threads, no network, no fixture. + +This is already the established pattern, not a proposal: +`tests/test_face_tracker.cpp` "drives the node's `operator()` with hand-built +`EmbeddedSceneFrame`s and inspects the emitted `track_ids`", and does so +"pure, GPU-free, model-free". + +The consequence is that most of the redesign is verifiable **without any +fixture at all**: construct exactly the awkward state — a belief swap, two live +tracks converging on one actor, a film ending mid-track, a gap one frame under +the timeout — rather than hunting for a clip that happens to exhibit it. + +Four hazards this removes outright: + +- **No fixture-provenance risk** for these tests — the inputs are synthetic and + explicit. +- **No "fixture must be replayed from frame 0"** concern — state is constructed + directly. +- **No cross-test state leakage** (e.g. a tracker's `next_id_` persisting) — each + test constructs a fresh functor. +- **No replay-harness nondeterminism** — no channels, so no EOF-tail heuristics + or silent drops. + +It also means **a dead upstream producer does not block testing a downstream +consumer.** `is_scene_boundary` currently has no producer (see AR-010), which +would make a *replay* test of the frame-dependent `track_alpha` pass vacuously — +but a T1 test simply constructs a frame with `is_scene_boundary = true` and +asserts the weighting changes. The producer gap is a pipeline defect to fix, not +a verification blocker. + +### T2 covers what T1 cannot + +Replay remains necessary for **composition** — that the nodes wired together +behave as the sum of their parts — and for realistic data at scale, which +synthetic inputs cannot honestly imitate. It is the tier that would catch a +wiring error, a channel-capacity problem, or an ordering assumption that only +appears under concurrency. + +The HDF5 dump (VR-001) captures state after decode → detect → align → embed, so +replay needs no GPU and no video. That was built for the optimizer; it doubles as +CI, which is a strong argument for keeping the schema honest and for replay +driving the *real* nodes rather than a reimplementation (VR-002). + +**Fixtures and studies are generated locally**, on the development machine where +the models, galleries and media already exist. CI consumes them; it never +produces them. **Small committed fixtures are required.** A few HDF5 dumps covering the awkward cases — a cut, a belief swap, two live tracks converging, a film ending @@ -176,16 +219,29 @@ what looks like GPU work into pure CPU replay. | Fixture | Contents | Size | Storage | |---|---|---|---| -| **Edge-case dumps** | ~6 short clips (30–60 s), one per awkward behaviour | ~1 MB each | **Committed in-repo** | -| **Corpus dumps** | Full-length titles from the validation corpus | ~30 MB each | Pinned artifact, fetched by checksum | +| **Edge-case dumps** | ~6 short clips (30–60 s), one per awkward behaviour | ~0.1–1 MB each | **Committed in-repo** | +| **Corpus dumps** | Full-length titles from the validation corpus | ~21–38 MB each | **Gitea package registry**, pinned by version + checksum | | **Synthetic gallery** | Random unit-norm embeddings, fixed seed | small | Generated at test time | | **Golden truth files** | Expected output for each edge-case dump | KB | Committed | -| **Audio golden vectors** | Short WAV + expected signature | KB | Committed, **shared with the plugin repo** | +| **Audio golden vectors** | FLAC + expected signature + parameter contract | ~600 KB | Committed, **shared with the plugin repo** | Edge-case dumps are small enough to commit, and being in-repo means they version -with the code that reads them. Corpus dumps are pulled by pinned checksum from -the artifact store rather than committed, since they are large and change only -when the dump schema does. +with the code that reads them. + +**Corpus dumps go to the Gitea package registry, not Git LFS.** Both are +available — the models already use LFS — but their fetch semantics differ in a +way that matters here. LFS objects are pulled on clone unless a developer +explicitly skips them, so ~38 MB per title behind LFS taxes everyone who clones, +forever, for data that only CI and the optimizer ever read. Registry artifacts +are fetched on demand by the job that needs them. + +Rule of thumb: **LFS for what the build needs; the package registry for what a +particular job needs.** Models are the former; corpus dumps and the CI image +(DP-007) are the latter. + +Pin by version and verify by checksum on fetch. A fixture that changes silently +under CI is worse than a missing one, because the failure presents as a code +regression. **Generation must be reproducible and versioned.** A script, run on a GPU host, regenerates every fixture from source clips; it is re-run when the VR-001 schema @@ -231,7 +287,11 @@ because it will be trusted. | AR-027 | **T4** | Throughput at 10²…10⁵ actors | Scheduled, not on-demand | | IR-001/002 | T1 | Serialised output matches golden file | Zero-length window; actor with many windows | | IR-003 | T1 | Output written after deferred pass | Not at EOF | -| IR-004/005 | **T1** | Signature matches golden vector bit-for-bit | **Media < 120 s → no signature**; identical result in both repos | +| IR-004/005 | **T1** | Signature matches golden vector bit-for-bit | Identical result in both producer repos | +| IR-006 | T1 + manual | Queue pull and result push against a stubbed Jellyfin API | Partial result never pushed; push only after the deferred pass | +| IR-007 | **T1** | Media < 120 s emits no signature at all | Exactly 120 s; just under; zero-length audio. Must match the plugin's cutoff exactly — a caller-varying window length is what SR-004 forbids | +| IR-008 | T1 | `v1:` prefix emitted and honoured on read | Unknown prefix rejected, not guessed | +| GR-009 | T1 | Human-confirmed associations persist and are tier-tagged | Survives a gallery rebuild; distinguishable from baked and harvested | | GR-004 | T1 | Mismatched embedder → hard startup error | Error names both sides | | GR-008 | T1 | Outlier flagged among an actor's references | Injected poisoned embedding detected | | VR-009 | T1 | Posterior calibration holds | A 0.99 posterior is wrong ~1% of the time on held-out tracks |