docs: audit findings, verification tiers, CI image, artifact storage
Corrections from the dump audit and the completed agent work: - AR-010 is not started, not in progress: is_scene_boundary has no producer anywhere. SceneDetectorFunc is a terminal sink writing scenes.json and never annotates the frame, so the field is permanently false and the dump column a constant 0. A replay test of the frame-dependent track_alpha would pass vacuously — the worst failure mode for a verification gate. - T1 (functor-level) becomes the primary verification tier, not T2. KPN node functors are plain callables constructed outside the network, so a node is tested by calling operator() with hand-built inputs. That removes four hazards at once: fixture provenance, replay-from-frame-0, cross-test state leakage, and replay-harness nondeterminism. It also means a dead upstream producer no longer blocks testing its consumer. - VR-010 (dump provenance) and VR-011 (replay harness rewrite) added. A dump made with LVFace is currently byte-indistinguishable from one made with w600k-R50 — the GR-004 problem again, in the dump. - Four requirements had no verification tier at all; the traceability gate found them. - DP-007: CI builder image, CPU-only, pinned by tag in the Gitea container registry. Corpus fixtures go to the package registry rather than LFS: LFS is pulled on clone and would tax every developer for data only CI reads. - IR-004/005/007/008 marked done; the v1 DSP parameters they had to pin are now normative in the server spec. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> TRACES: AR-010, DP-007, IR-004, IR-005, IR-007, IR-008, VR-001, VR-010, VR-011 | SR-002, SR-003
This commit is contained in:
@@ -154,6 +154,26 @@ Two distinct signals, deliberately kept separate:
|
|||||||
- **`is_scene_boundary`** — opt-in (`--scene-detect`). TransNetV2 over a densely
|
- **`is_scene_boundary`** — opt-in (`--scene-detect`). TransNetV2 over a densely
|
||||||
decoded, downscaled stream flags a *true shot/scene boundary*.
|
decoded, downscaled stream flags a *true shot/scene boundary*.
|
||||||
|
|
||||||
|
> **`is_scene_boundary` currently has no producer.** `grep -rn is_scene_boundary
|
||||||
|
> src/` finds no assignment anywhere: `SceneDetectorFunc` is a *terminal sink*
|
||||||
|
> (`main.cpp:298-300`, `kpn::out<>`) that writes `scenes.json` and never
|
||||||
|
> annotates the `Frame` flowing to the face pipeline. The field is therefore
|
||||||
|
> always `false`, and the dump column (`embedding_dump_node.hpp:38`) is a
|
||||||
|
> constant 0. Compounding it, `main.cpp:280` returns from the
|
||||||
|
> `--dump-embeddings` branch *before* the `scene_detect` branch at `:296`, so no
|
||||||
|
> dump-producing path even instantiates the detector.
|
||||||
|
>
|
||||||
|
> This makes AR-010 **not implemented**, not "in progress" — and it means a T2
|
||||||
|
> test of the frame-dependent `track_alpha` (AR-007) would **pass vacuously**,
|
||||||
|
> which is the worst possible failure for a verification gate. The fix is in the
|
||||||
|
> producer, not the schema: make `SceneDetectorFunc` a pass-through (or add a
|
||||||
|
> boundary annotator before the decimator) and add the scene branch to
|
||||||
|
> `dump_embeddings.cpp`. **No `schema_version` bump** — the column exists and
|
||||||
|
> merely stops being constant.
|
||||||
|
>
|
||||||
|
> Fixtures generated before the fix must be marked in provenance, since `0` is
|
||||||
|
> presently indistinguishable from "no boundary here".
|
||||||
|
|
||||||
Both feed AR-007 as **association hints**: they tell the tracker that spatial
|
Both feed AR-007 as **association hints**: they tell the tracker that spatial
|
||||||
continuity is broken and that association should weight embedding over IoU.
|
continuity is broken and that association should weight embedding over IoU.
|
||||||
Neither ends a presence window (AR-012).
|
Neither ends a presence window (AR-012).
|
||||||
@@ -770,6 +790,54 @@ prerequisite.
|
|||||||
|
|
||||||
**Gap:** installer unbuilt.
|
**Gap:** installer unbuilt.
|
||||||
|
|
||||||
|
## DP-007 — CI build image
|
||||||
|
|
||||||
|
CI runs on an Intel N100 with no discrete GPU, so the test build must configure
|
||||||
|
**CPU-only** and must not require CUDA, TensorRT or ROCm:
|
||||||
|
|
||||||
|
```
|
||||||
|
-DSAE_INFERENCE_BACKEND=ORT -DSAE_GEMM_BACKEND=CPU
|
||||||
|
```
|
||||||
|
|
||||||
|
A prebuilt container image supplies the toolchain, published to the **Gitea
|
||||||
|
container registry** and pinned by tag — matching the `jellytau-builder`
|
||||||
|
precedent. Building dependencies per CI run is untenable on an N100, and OpenCV 5
|
||||||
|
from source would dominate every run.
|
||||||
|
|
||||||
|
The same registry stores corpus dump fixtures as generic packages (see the
|
||||||
|
fixtures table in `requirements.md`). Rebuild the image when its dependency set
|
||||||
|
changes, not per run, and pin CI to a tag rather than `latest` so a rebuild
|
||||||
|
cannot silently change what a green build meant.
|
||||||
|
|
||||||
|
**Required in the image:**
|
||||||
|
|
||||||
|
| Dependency | Why |
|
||||||
|
|---|---|
|
||||||
|
| CMake, C++ toolchain, pkg-config | Build |
|
||||||
|
| **OpenCV 5** | `CMakeLists.txt:25` prefers 5, falls back to 4. The branch targets 5, so the image should carry it — it is not yet in most distro repos and building it per-run is prohibitive |
|
||||||
|
| HDF5 (C++) | Galleries are HDF5-native; also the dump format |
|
||||||
|
| FFmpeg dev libs — `libavformat`, `libavcodec`, `libavutil`, `libswscale`, **`libswresample`** | Decode. See the note below on swresample |
|
||||||
|
| Python 3 + numpy, h5py, scipy | Python-side tests, replay, traceability tooling |
|
||||||
|
| Catch2, nlohmann/json | **Vendored into the image, not fetched.** Both are `FetchContent`-ed today (`CMakeLists.txt:220`, `tests/CMakeLists.txt:8`), which makes every CI run depend on GitHub reachability |
|
||||||
|
|
||||||
|
**Deliberately excluded:** CUDA, TensorRT, ROCm — no GPU to use them. Also the
|
||||||
|
ONNX Runtime *GPU* providers; only the CPU provider is relevant, and only for T3
|
||||||
|
smoke tests.
|
||||||
|
|
||||||
|
**Models are not baked into the image.** The seven ONNX files total ~725 MB and
|
||||||
|
live in Git LFS. T1/T2 tests are model-free by design
|
||||||
|
(`tests/CMakeLists.txt:1-4`), so the default image needs none. T3 smoke tests
|
||||||
|
require a model and should pull it via LFS in a separate job rather than
|
||||||
|
inflating the image tenfold for a minority of tests.
|
||||||
|
|
||||||
|
**`libswresample` is a real gap, not a formality.** The current
|
||||||
|
`pkg_check_modules` list (`CMakeLists.txt:200-203`) covers avformat, avcodec,
|
||||||
|
avutil and swscale but **not** swresample — which IR-004 needs to downmix to mono
|
||||||
|
and resample to 11025 Hz. It must be added alongside the audio-signature work.
|
||||||
|
|
||||||
|
**Gap:** entire requirement. The image does not exist, and no CI config is
|
||||||
|
present in this repo.
|
||||||
|
|
||||||
## DP-006 — Gallery maintenance as a background concern
|
## DP-006 — Gallery maintenance as a background concern
|
||||||
|
|
||||||
- Incremental gallery refresh runs on a timer (`gallery_scan_interval`, default
|
- Incremental gallery refresh runs on a timer (`gallery_scan_interval`, default
|
||||||
|
|||||||
+83
-23
@@ -37,7 +37,7 @@ Status: `Done` · `In Progress` · `Planned` · `TBD` · `Withdrawn`
|
|||||||
| AR-007 | Associate detections by IoU + embedding, with **frame-dependent** weighting | SR-002 | High | In Progress |
|
| AR-007 | Associate detections by IoU + embedding, with **frame-dependent** weighting | SR-002 | High | In Progress |
|
||||||
| AR-008 | One track pool keyed on `last_seen`; no separate revival path | SR-002 | High | Planned |
|
| AR-008 | One track pool keyed on `last_seen`; no separate revival path | SR-002 | High | Planned |
|
||||||
| AR-009 | Camera-cut detection (histogram) as an association hint | SR-002 | High | Done |
|
| AR-009 | Camera-cut detection (histogram) as an association hint | SR-002 | High | Done |
|
||||||
| AR-010 | Scene-boundary detection (TransNetV2) as an association hint | SR-002 | Medium | In Progress |
|
| AR-010 | Scene-boundary detection (TransNetV2) as an association hint | SR-002 | Medium | **Not started** — `is_scene_boundary` has no producer; `SceneDetectorFunc` is a terminal sink and never annotates the frame |
|
||||||
| AR-011 | **Every model is fed the input it was trained for** — cost reduced by running less often, never by degrading one inference | SR-002 | High | Planned |
|
| AR-011 | **Every model is fed the input it was trained for** — cost reduced by running less often, never by degrading one inference | SR-002 | High | Planned |
|
||||||
| AR-012 | Presence follows **track extent**, not per-frame recognition | **SR-002** | High | Planned |
|
| AR-012 | Presence follows **track extent**, not per-frame recognition | **SR-002** | High | Planned |
|
||||||
| AR-013 | `last_seen` optional state machine; window ends at last sighting, never after | SR-002 | High | Planned |
|
| AR-013 | `last_seen` optional state machine; window ends at last sighting, never after | SR-002 | High | Planned |
|
||||||
@@ -74,10 +74,10 @@ Status: `Done` · `In Progress` · `Planned` · `TBD` · `Withdrawn`
|
|||||||
| IR-001 | Emit the JRay truth format as sibling `.jray.json` | SR-003 | High | Done |
|
| IR-001 | Emit the JRay truth format as sibling `.jray.json` | SR-003 | High | Done |
|
||||||
| IR-002 | Windows carry belief + route; `extraction.*` carries `extinction_sec`, `gallery_scope` | SR-003 | High | Planned |
|
| IR-002 | Windows carry belief + route; `extraction.*` carries `extinction_sec`, `gallery_scope` | SR-003 | High | Planned |
|
||||||
| IR-003 | Output written **after** the deferred pass, not at EOF | SR-003 | High | Planned |
|
| IR-003 | Output written **after** the deferred pass, not at EOF | SR-003 | High | Planned |
|
||||||
| IR-004 | Compute the audio signature exactly per server spec §3 | SR-003 | Medium | Planned |
|
| IR-004 | Compute the audio signature exactly per server spec §3 | SR-003 | Medium | **Done** — `src/audio_signature.*`; not yet emitted into the truth file (IR-002) |
|
||||||
| IR-005 | Golden-vector fixture shared with the plugin repo to prove bit-exactness | SR-003 | High | Planned |
|
| IR-005 | Golden-vector fixture shared with the plugin repo to prove bit-exactness | SR-003 | High | **Done** — `tests/fixtures/audio/`; v1 parameters now normative in server spec §3 |
|
||||||
| IR-007 | Media < 120 s: emit no signature, apply no sync offset — identical rule in both producers | SR-003 | Low | Planned |
|
| IR-007 | Media < 120 s: emit no signature, apply no sync offset — identical rule in both producers | SR-003 | Low | **Done** |
|
||||||
| IR-008 | Emit and honour the signature's own `v1:` version prefix | SR-003 | Low | Planned |
|
| IR-008 | Emit and honour the signature's own `v1:` version prefix | SR-003 | Low | **Done** |
|
||||||
| IR-006 | Jellyfin round-trip: pull pending queue, push complete results only | SR-001 | High | Done |
|
| IR-006 | Jellyfin round-trip: pull pending queue, push complete results only | SR-001 | High | Done |
|
||||||
|
|
||||||
## Gallery (GR)
|
## Gallery (GR)
|
||||||
@@ -107,6 +107,8 @@ Status: `Done` · `In Progress` · `Planned` · `TBD` · `Withdrawn`
|
|||||||
| VR-007 | Expansion band, clustering threshold, and deferred-pass ablation | PR-002 | Medium | Planned |
|
| VR-007 | Expansion band, clustering threshold, and deferred-pass ablation | PR-002 | Medium | Planned |
|
||||||
| VR-008 | Gallery scaling benchmark — throughput vs gallery size | PR-002 | Medium | Planned |
|
| VR-008 | Gallery scaling benchmark — throughput vs gallery size | PR-002 | Medium | Planned |
|
||||||
| VR-009 | Verify accumulated posteriors are calibrated against held-out tracks | PR-002 | High | Planned |
|
| VR-009 | Verify accumulated posteriors are calibrated against held-out tracks | PR-002 | High | Planned |
|
||||||
|
| VR-010 | Dump provenance attributes — embedder model, detector settings, `dense_scale`, `scene_detect`, sample rate | PR-002 | **High** | Planned |
|
||||||
|
| VR-011 | Rewrite the replay harness for the post-AR-012 output contract | PR-002 | High | Planned |
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -120,21 +122,62 @@ Four tiers, in decreasing order of preference:
|
|||||||
|
|
||||||
| Tier | Runs in CI | What it covers |
|
| Tier | Runs in CI | What it covers |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| **T1 — CPU unit** | Yes | Pure logic: registry state machine, belief accumulation, clustering, band admission, calibration maths |
|
| **T1 — Functor unit** | Yes | A KPN node's `operator()` driven directly with hand-built inputs |
|
||||||
| **T2 — Replay** | Yes | Real pipeline nodes driven from an HDF5 fixture — no GPU, no video |
|
| **T2 — Replay** | Yes | The composed pipeline driven from an HDF5 fixture — no GPU, no video |
|
||||||
| **T3 — CPU inference** | Yes, slowly | ORT CPU provider over a handful of frames; smoke tests only |
|
| **T3 — CPU inference** | Yes, slowly | ORT CPU provider over a handful of frames; smoke tests only |
|
||||||
| **T4 — GPU** | **No** | Throughput, TRT engines, large-gallery GEMM |
|
| **T4 — GPU** | **No** | Throughput, TRT engines, large-gallery GEMM |
|
||||||
|
|
||||||
**T2 is the reason this is workable.** The HDF5 dump (VR-001) captures state
|
### T1 is the primary tier, and KPN is why
|
||||||
after decode → detect → align → embed and before tracking and matching, so
|
|
||||||
everything downstream — which is where nearly all of the new design lives — is
|
|
||||||
cheap CPU maths replayable from a fixture. Tracking, presence windows, belief
|
|
||||||
accumulation, expansion, deferred re-identification and clustering are all
|
|
||||||
verifiable on an N100 at full fidelity, not in miniature.
|
|
||||||
|
|
||||||
That was already true for the optimizer. It now doubles as the CI strategy, which
|
**Node functors are plain callable structs, constructed independently of the
|
||||||
is a strong argument for keeping the dump schema honest (VR-001) and for the
|
network that wraps them** (`main.cpp:186-207` builds them as stack objects;
|
||||||
replay driving the *real* nodes rather than a reimplementation (VR-002).
|
`ObjectNode` merely adapts them). So a node is testable by constructing it and
|
||||||
|
calling `operator()` — no channels, no threads, no network, no fixture.
|
||||||
|
|
||||||
|
This is already the established pattern, not a proposal:
|
||||||
|
`tests/test_face_tracker.cpp` "drives the node's `operator()` with hand-built
|
||||||
|
`EmbeddedSceneFrame`s and inspects the emitted `track_ids`", and does so
|
||||||
|
"pure, GPU-free, model-free".
|
||||||
|
|
||||||
|
The consequence is that most of the redesign is verifiable **without any
|
||||||
|
fixture at all**: construct exactly the awkward state — a belief swap, two live
|
||||||
|
tracks converging on one actor, a film ending mid-track, a gap one frame under
|
||||||
|
the timeout — rather than hunting for a clip that happens to exhibit it.
|
||||||
|
|
||||||
|
Four hazards this removes outright:
|
||||||
|
|
||||||
|
- **No fixture-provenance risk** for these tests — the inputs are synthetic and
|
||||||
|
explicit.
|
||||||
|
- **No "fixture must be replayed from frame 0"** concern — state is constructed
|
||||||
|
directly.
|
||||||
|
- **No cross-test state leakage** (e.g. a tracker's `next_id_` persisting) — each
|
||||||
|
test constructs a fresh functor.
|
||||||
|
- **No replay-harness nondeterminism** — no channels, so no EOF-tail heuristics
|
||||||
|
or silent drops.
|
||||||
|
|
||||||
|
It also means **a dead upstream producer does not block testing a downstream
|
||||||
|
consumer.** `is_scene_boundary` currently has no producer (see AR-010), which
|
||||||
|
would make a *replay* test of the frame-dependent `track_alpha` pass vacuously —
|
||||||
|
but a T1 test simply constructs a frame with `is_scene_boundary = true` and
|
||||||
|
asserts the weighting changes. The producer gap is a pipeline defect to fix, not
|
||||||
|
a verification blocker.
|
||||||
|
|
||||||
|
### T2 covers what T1 cannot
|
||||||
|
|
||||||
|
Replay remains necessary for **composition** — that the nodes wired together
|
||||||
|
behave as the sum of their parts — and for realistic data at scale, which
|
||||||
|
synthetic inputs cannot honestly imitate. It is the tier that would catch a
|
||||||
|
wiring error, a channel-capacity problem, or an ordering assumption that only
|
||||||
|
appears under concurrency.
|
||||||
|
|
||||||
|
The HDF5 dump (VR-001) captures state after decode → detect → align → embed, so
|
||||||
|
replay needs no GPU and no video. That was built for the optimizer; it doubles as
|
||||||
|
CI, which is a strong argument for keeping the schema honest and for replay
|
||||||
|
driving the *real* nodes rather than a reimplementation (VR-002).
|
||||||
|
|
||||||
|
**Fixtures and studies are generated locally**, on the development machine where
|
||||||
|
the models, galleries and media already exist. CI consumes them; it never
|
||||||
|
produces them.
|
||||||
|
|
||||||
**Small committed fixtures are required.** A few HDF5 dumps covering the awkward
|
**Small committed fixtures are required.** A few HDF5 dumps covering the awkward
|
||||||
cases — a cut, a belief swap, two live tracks converging, a film ending
|
cases — a cut, a belief swap, two live tracks converging, a film ending
|
||||||
@@ -176,16 +219,29 @@ what looks like GPU work into pure CPU replay.
|
|||||||
|
|
||||||
| Fixture | Contents | Size | Storage |
|
| Fixture | Contents | Size | Storage |
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| **Edge-case dumps** | ~6 short clips (30–60 s), one per awkward behaviour | ~1 MB each | **Committed in-repo** |
|
| **Edge-case dumps** | ~6 short clips (30–60 s), one per awkward behaviour | ~0.1–1 MB each | **Committed in-repo** |
|
||||||
| **Corpus dumps** | Full-length titles from the validation corpus | ~30 MB each | Pinned artifact, fetched by checksum |
|
| **Corpus dumps** | Full-length titles from the validation corpus | ~21–38 MB each | **Gitea package registry**, pinned by version + checksum |
|
||||||
| **Synthetic gallery** | Random unit-norm embeddings, fixed seed | small | Generated at test time |
|
| **Synthetic gallery** | Random unit-norm embeddings, fixed seed | small | Generated at test time |
|
||||||
| **Golden truth files** | Expected output for each edge-case dump | KB | Committed |
|
| **Golden truth files** | Expected output for each edge-case dump | KB | Committed |
|
||||||
| **Audio golden vectors** | Short WAV + expected signature | KB | Committed, **shared with the plugin repo** |
|
| **Audio golden vectors** | FLAC + expected signature + parameter contract | ~600 KB | Committed, **shared with the plugin repo** |
|
||||||
|
|
||||||
Edge-case dumps are small enough to commit, and being in-repo means they version
|
Edge-case dumps are small enough to commit, and being in-repo means they version
|
||||||
with the code that reads them. Corpus dumps are pulled by pinned checksum from
|
with the code that reads them.
|
||||||
the artifact store rather than committed, since they are large and change only
|
|
||||||
when the dump schema does.
|
**Corpus dumps go to the Gitea package registry, not Git LFS.** Both are
|
||||||
|
available — the models already use LFS — but their fetch semantics differ in a
|
||||||
|
way that matters here. LFS objects are pulled on clone unless a developer
|
||||||
|
explicitly skips them, so ~38 MB per title behind LFS taxes everyone who clones,
|
||||||
|
forever, for data that only CI and the optimizer ever read. Registry artifacts
|
||||||
|
are fetched on demand by the job that needs them.
|
||||||
|
|
||||||
|
Rule of thumb: **LFS for what the build needs; the package registry for what a
|
||||||
|
particular job needs.** Models are the former; corpus dumps and the CI image
|
||||||
|
(DP-007) are the latter.
|
||||||
|
|
||||||
|
Pin by version and verify by checksum on fetch. A fixture that changes silently
|
||||||
|
under CI is worse than a missing one, because the failure presents as a code
|
||||||
|
regression.
|
||||||
|
|
||||||
**Generation must be reproducible and versioned.** A script, run on a GPU host,
|
**Generation must be reproducible and versioned.** A script, run on a GPU host,
|
||||||
regenerates every fixture from source clips; it is re-run when the VR-001 schema
|
regenerates every fixture from source clips; it is re-run when the VR-001 schema
|
||||||
@@ -231,7 +287,11 @@ because it will be trusted.
|
|||||||
| AR-027 | **T4** | Throughput at 10²…10⁵ actors | Scheduled, not on-demand |
|
| AR-027 | **T4** | Throughput at 10²…10⁵ actors | Scheduled, not on-demand |
|
||||||
| IR-001/002 | T1 | Serialised output matches golden file | Zero-length window; actor with many windows |
|
| IR-001/002 | T1 | Serialised output matches golden file | Zero-length window; actor with many windows |
|
||||||
| IR-003 | T1 | Output written after deferred pass | Not at EOF |
|
| IR-003 | T1 | Output written after deferred pass | Not at EOF |
|
||||||
| IR-004/005 | **T1** | Signature matches golden vector bit-for-bit | **Media < 120 s → no signature**; identical result in both repos |
|
| IR-004/005 | **T1** | Signature matches golden vector bit-for-bit | Identical result in both producer repos |
|
||||||
|
| IR-006 | T1 + manual | Queue pull and result push against a stubbed Jellyfin API | Partial result never pushed; push only after the deferred pass |
|
||||||
|
| IR-007 | **T1** | Media < 120 s emits no signature at all | Exactly 120 s; just under; zero-length audio. Must match the plugin's cutoff exactly — a caller-varying window length is what SR-004 forbids |
|
||||||
|
| IR-008 | T1 | `v1:` prefix emitted and honoured on read | Unknown prefix rejected, not guessed |
|
||||||
|
| GR-009 | T1 | Human-confirmed associations persist and are tier-tagged | Survives a gallery rebuild; distinguishable from baked and harvested |
|
||||||
| GR-004 | T1 | Mismatched embedder → hard startup error | Error names both sides |
|
| GR-004 | T1 | Mismatched embedder → hard startup error | Error names both sides |
|
||||||
| GR-008 | T1 | Outlier flagged among an actor's references | Injected poisoned embedding detected |
|
| GR-008 | T1 | Outlier flagged among an actor's references | Injected poisoned embedding detected |
|
||||||
| VR-009 | T1 | Posterior calibration holds | A 0.99 posterior is wrong ~1% of the time on held-out tracks |
|
| VR-009 | T1 | Posterior calibration holds | A 0.99 posterior is wrong ~1% of the time on held-out tracks |
|
||||||
|
|||||||
Reference in New Issue
Block a user