scene-actor-extraction/docs/requirements.md

scene-actor-extraction — requirements register

Stable IDs for every requirement in SPEC.md, which holds the prose. This file is the authoritative list; the CI gate reads its denominators from here (see ../../SPEC.md §6).

IDs are permanent. A withdrawn requirement is marked Withdrawn and its number is never reused — renumbering is what produces orphan TRACES tags. This register replaces the earlier thematic A1…E8 scheme, which had already produced an A1a and an out-of-order E6.

Tag code with // TRACES: AR-012 | SR-002.

Type	Scope
AR	Algorithm — the extraction pipeline itself
DP	Deployment — how it runs
IR	Integration — contracts with other components
GR	Gallery — building and maintaining actor references
VR	Validation — parameter studies and benchmarks
UT / IT	Unit / integration tests

Status: Done · In Progress · Planned · TBD · Withdrawn

---

Algorithm (AR)

ID	Requirement	Traces to	Priority	Status
AR-001	Detect faces in sampled frames; emit bbox, confidence, 5-point landmarks in original pixel space	SR-002	High	Done
AR-002	Minimum face size 32×32 px (VR-005 measured), expressed in original resolution (decoupled from dense_scale)	SR-002	High	Planned
AR-003	No fixed per-frame face cap — crowd scenes must not lose background cast	SR-002	Medium	Done — max_faces defaults to 0 (no cap); the matcher batches through its GEMM buffer instead of throwing
AR-004	Backpressure: unbounded faces/frame absorbed by slowing, never by dropping or throwing	SR-002	High	Done — KPN node outputs use push_blocking; sentinels stay out-of-band. Verified: 385/385 frames, 0 drops, byte-identical across runs
AR-005	Align to 112×112 via ArcFace 5-point similarity transform	SR-002	High	Done
AR-006	512-d L2-normalised embeddings, batched	SR-002	High	Done
AR-007	Associate detections by IoU + embedding, with frame-dependent weighting	SR-002	High	Done — track_alpha is the base for ordinary frames; drops to embedding-only on cut/boundary and for dormant tracks
AR-008	One track pool keyed on last_seen; no separate revival path	SR-002	High	Done — one pool keyed on last_seen; park/revive branch deleted
AR-009	Camera-cut detection (histogram) as an association hint	SR-002	High	Done
AR-010	Scene-boundary detection (TransNetV2) as an association hint	SR-002	Medium	Not started — is_scene_boundary has no producer; SceneDetectorFunc is a terminal sink and never annotates the frame
AR-011	Every model is fed the input it was trained for — cost reduced by running less often, never by degrading one inference	SR-002	High	Planned
AR-012	Presence follows track extent, not per-frame recognition	SR-002	High	Done — src/track_registry.hpp; window is [first_seen, last_seen] of an owned track
AR-013	last_seen optional state machine; window ends at last sighting, never after	SR-002	High	Done — last_seen optional is the whole state machine; interior gaps absorbed, trailing cool-down never claimed
AR-014	Belief swap A→B terminates the track and starts a new one	SR-002	Medium	Done — swap closes at last_seen and opens a successor at the swap frame; counted
AR-015	Two live tracks owned by one actor ⇒ treat as a detected cut, re-associate	SR-002	Medium	Done — reverse index detects it on the causing update; counted
AR-016	All tracks closed at EOF — a film ends with faces on screen	SR-002	High	Done — flush(), idempotent, closes at last sighting or final tick
AR-017	Every presence claim carries its belief and identification route	SR-002	High	Done — DeadTrack carries belief and observation count
AR-018	Per-subject embedding store with banded admission (novel enough, safe enough)	SR-005	Medium	Planned
AR-019	Per-film gallery annex from owned tracks; acquires the non-frontal views TMDB lacks	SR-005	Medium	In Progress
AR-020	Deferred re-identification of unknown tracks against the final expanded gallery	SR-005	High	Planned
AR-021	Cluster unknown tracks into one entity per person, under temporal cannot-link constraints	SR-005	Medium	Planned
AR-022	Capture still-unidentified tracks: embeddings, metadata, context crops for human review	§4	Medium	Planned
AR-023	Fit sigmoid calibration from intra/inter similarity distributions	SR-002	High	Done
AR-024	Always the calibrated probability, never a raw cosine — exceptions recorded	SR-002	High	Done — association and accumulation both in probability space; track_max_embed_dist, cut_revive_sim retired
AR-025	Per-track Bayesian accumulation in log-odds, with correlated-observation discounting	SR-002	High	Done — log-odds accumulation with correlation discounting owned by the registry, src/evidence_discount.hpp
AR-026	All similarity computed as GEMM, including annex and deferred pass	SR-001	High	In Progress
AR-027	Throughput acceptable for arbitrary gallery size	SR-001	High	Planned

Deployment (DP)

ID	Requirement	Traces to	Priority	Status
DP-001	One analysis core; modes are front-ends and must not fork pipeline logic	PR-004	High	Done
DP-002	Batch CLI over one title	PR-004	High	Done
DP-003	On-demand resident service with bounded, observable queue	PR-004	Medium	Planned
DP-004	Opportunistic/idle mode: external trigger, hard stop, implicit re-queue	PR-004	Medium	Planned
DP-005	Native installer, no Docker; Fedora + Arch	PR-004	Medium	Planned
DP-006	Background incremental gallery refresh on a timer	PR-003	Medium	Planned
DP-007	CI builder image, CPU-only, pinned by tag in the Gitea container registry	PR-004	High	Planned
DP-008	Builder images + release jobs per backend (cpu / cuda / rocm); ship binaries, not engines	PR-004	Medium	Planned

Integration (IR)

ID	Requirement	Traces to	Priority	Status
IR-001	Emit the JRay truth format as sibling .jray.json	SR-003	High	Done
IR-002	Windows carry belief + route; extraction.* carries extinction_sec, gallery_scope	SR-003	High	Done — schema_version: 2; windows are objects with belief + route; extraction.* carries extinction_sec and gallery_scope; anneal_sec removed
IR-003	Output written after the deferred pass, not at EOF	SR-003	High	In Progress — sink builds windows from registry claims and flushes at EOF; the deferred pass (AR-020) does not exist yet, so output is still final at EOF
IR-004	Compute the audio signature exactly per server spec §3	SR-003	Medium	Done — src/audio_signature.*; not yet emitted into the truth file (IR-002)
IR-005	Golden-vector fixture shared with the plugin repo to prove bit-exactness	SR-003	High	Done — tests/fixtures/audio/; v1 parameters now normative in server spec §3
IR-007	Media < 120 s: emit no signature, apply no sync offset — identical rule in both producers	SR-003	Low	Done
IR-008	Emit and honour the signature's own v1: version prefix	SR-003	Low	Done
IR-006	Jellyfin round-trip: pull pending queue, push complete results only	SR-001	High	Done

Gallery (GR)

ID	Requirement	Traces to	Priority	Status
GR-001	Build gallery from Jellyfin library cast, TMDB profile fallback	SR-001, SR-005	High	Done
GR-002	Incremental --merge refresh without re-embedding known actors	PR-003	High	Done
GR-003	Report coverage: zero-image actors, under-referenced actors, dedup, calibration PDFs	SR-001	Medium	Planned
GR-004	Stamp embedder identity into the gallery; hard startup error on mismatch	SR-001	High	Done — basename + SHA-256 + embed_dim; mismatch fatal with no bypass, unstamped warns unless --require-gallery-stamp; scripts/stamp_gallery.py migrates in place
GR-005	Gallery data never leaves the instance	SR-005	High	Done
GR-006	Provenance tiers: baked / harvested / confirmed, distinguishable per embedding	SR-005	High	Planned
GR-007	Persist harvested embeddings flagged and reviewable, never silently equal to baked	SR-005	Medium	Planned
GR-008	Flag distributional outliers among an actor's references (poisoning guard) — EXCEPTION: AR-024	SR-005	Medium	Planned
GR-009	Human-confirmed associations persist and improve future extractions	§4	Medium	TBD

Validation (VR)

ID	Requirement	Traces to	Priority	Status
VR-001	HDF5 post-inference dump at the embedded-frame boundary	PR-002	High	Done
VR-002	Replay drives the real KPN nodes, not a reimplementation	PR-002	High	Done — replay driven from committed fixtures in tests/test_replay_fixtures.cpp; determinism asserted
VR-003	Scoring: micro-F1 against X-Ray, precision/recall logged at every evaluation	PR-002	High	Done
VR-004	Reproducible validation corpus with ground truth	PR-002	High	Done
VR-005	Minimum face size study — TPI/FPI vs probe size, gallery held at native res	PR-002	Medium	Done — knee at 24–32 px; 32 px gives 98.1% TPI, 0.0 FPI at every size
VR-006	Re-tune scene_threshold once native-rate decode lands	PR-002	Low	Planned
VR-007	Expansion band, clustering threshold, and deferred-pass ablation	PR-002	Medium	Planned
VR-008	Gallery scaling benchmark — throughput vs gallery size	PR-002	Medium	Planned
VR-009	Verify accumulated posteriors are calibrated against held-out tracks	PR-002	High	Planned
VR-010	Dump provenance attributes — embedder model, detector settings, dense_scale, scene_detect, sample rate	PR-002	High	Planned
VR-011	Rewrite the replay harness for the post-AR-012 output contract	PR-002	High	Planned

---

Verification strategy

CI runs on an Intel N100 with no discrete GPU. That is a hard constraint on how each requirement can be verified, and it shapes the test design rather than merely limiting it.

Four tiers, in decreasing order of preference:

Tier	Runs in CI	What it covers
T1 — Functor unit	Yes	A KPN node's operator() driven directly with hand-built inputs
T2 — Replay	Yes	The composed pipeline driven from an HDF5 fixture — no GPU, no video
T3 — CPU inference	Yes, slowly	ORT CPU provider over a handful of frames; smoke tests only
T4 — GPU	No	Throughput, TRT engines, large-gallery GEMM

T1 is the primary tier, and KPN is why

Node functors are plain callable structs, constructed independently of the network that wraps them (main.cpp:186-207 builds them as stack objects; ObjectNode merely adapts them). So a node is testable by constructing it and calling operator() — no channels, no threads, no network, no fixture.

This is already the established pattern, not a proposal: tests/test_face_tracker.cpp "drives the node's operator() with hand-built EmbeddedSceneFrames and inspects the emitted track_ids", and does so "pure, GPU-free, model-free".

The consequence is that most of the redesign is verifiable without any fixture at all: construct exactly the awkward state — a belief swap, two live tracks converging on one actor, a film ending mid-track, a gap one frame under the timeout — rather than hunting for a clip that happens to exhibit it.

Four hazards this removes outright:

• No fixture-provenance risk for these tests — the inputs are synthetic and explicit.
• No "fixture must be replayed from frame 0" concern — state is constructed directly.
• No cross-test state leakage (e.g. a tracker's next_id_ persisting) — each test constructs a fresh functor.
• No replay-harness nondeterminism — no channels, so no EOF-tail heuristics or silent drops.

It also means a dead upstream producer does not block testing a downstream consumer. is_scene_boundary currently has no producer (see AR-010), which would make a replay test of the frame-dependent track_alpha pass vacuously — but a T1 test simply constructs a frame with is_scene_boundary = true and asserts the weighting changes. The producer gap is a pipeline defect to fix, not a verification blocker.

T2 covers what T1 cannot

Replay remains necessary for composition — that the nodes wired together behave as the sum of their parts — and for realistic data at scale, which synthetic inputs cannot honestly imitate. It is the tier that would catch a wiring error, a channel-capacity problem, or an ordering assumption that only appears under concurrency.

The HDF5 dump (VR-001) captures state after decode → detect → align → embed, so replay needs no GPU and no video. That was built for the optimizer; it doubles as CI, which is a strong argument for keeping the schema honest and for replay driving the real nodes rather than a reimplementation (VR-002).

Fixtures and studies are generated locally, on the development machine where the models, galleries and media already exist. CI consumes them; it never produces them.

Small committed fixtures are required. A few HDF5 dumps covering the awkward cases — a cut, a belief swap, two live tracks converging, a film ending mid-track, an unknown track that only resolves after expansion — are worth more than a large corpus, and they are small enough to commit.

T4 requirements cannot pass in CI, and the gate must not pretend otherwise. For these, CI verifies that a test exists and is tagged, not that it passes; the run happens on a GPU host, nightly or manually, and reports separately. A requirement whose only evidence is a test that never executes should be visible as such rather than counted as covered.

Requirement	Tier	Note
AR-001, AR-005, AR-006	T3	Smoke only — correctness of detection/embedding is a model property, not ours
AR-002	Minimum face size 32×32 px (VR-005 measured), expressed in original resolution (decoupled from dense_scale)	Size filtering is arithmetic on dumped bboxes
AR-003, AR-004	T1 + T4	Backpressure logic is unit-testable; saturation behaviour needs real load
AR-007 … AR-017	T2	The core of the redesign — fully replayable
AR-018 … AR-022	T2	Expansion, deferred pass, clustering: all post-embedding
AR-023 … AR-025	T1	Calibration fit and log-odds accumulation are pure maths
AR-026, AR-027	T4	GEMM throughput and scaling — GPU host only
DP-*	T1 + manual	Lifecycle logic unit-tested; install paths are manual
IR-001 … IR-003	T1	Serialisation against a golden truth file
IR-004, IR-005	T1	Audio signature is CPU DSP — the golden-vector fixture runs anywhere, which is precisely why it is the right cross-repo check
GR-001 … GR-005	T1 + T3	Gallery assembly is I/O and bookkeeping; embedding is T3 smoke
GR-006 … GR-008	T1	Tiering and outlier detection operate on stored embeddings
VR-*	Out of CI	Studies are run deliberately and their results committed as documents

One consequence worth stating: AR-027 (arbitrary gallery scale) is structurally unverifiable on the CI host. It needs a GPU host and a synthetic large gallery, so it is the requirement most likely to silently regress. Its benchmark (VR-008) should run on a schedule rather than on demand.

CI never calls a model

Not "should not" — cannot. The N100 has no GPU, and even the ONNX Runtime CPU provider is impractical: a measured run of the embedder on this hardware sits at ~930 ms per frame, so a 77 s clip at 5 fps would take roughly six minutes of inference alone. Every model invocation therefore happens locally, ahead of time, and CI consumes the result as data.

This is what makes the T1/T2 split load-bearing rather than a preference: T1 and T2 are the only tiers that can exist in CI at all.

Fixture corpus — bali/

Five clips of Road to Bali (1952), ~77 s each, 480×360, 30 fps, 42 MB total.

Public domain, and that is the reason to use it rather than a convenience: derived fixtures — dumps, crops, golden outputs — can be committed without the rights question that rules out sharing gallery data (SR-005). A fixture cut from a copyrighted title could not live in the repository at all.

Two properties to design around rather than discover:

• 480×360 means small faces. At this resolution a face is often 40–80 px, so the AR-002 minimum of 66 px (original resolution) rejects much of what is there. Fixture generation must set --min-face-px explicitly and record it, or the dumps will be sparse for reasons unrelated to what is being tested.
• 77 s is short. At 1 fps that is 77 frames — too thin to exercise an extinction window measured in tens of seconds. Generate at 5 fps (≈385 frames, ~1 MB) and record the rate in provenance, since the behaviour under test changes with it.

AR-004 blocks reproducible fixture generation. A trial run of one clip produced 49 frames of an expected ~385, ending at 51 s of 77 s, with the diagnostics reporting 285 frames dropped at camera_pos and 51 at face_aligner. Channels overflow and drop rather than blocking, and what gets dropped depends on timing — so the same command run twice can produce different dumps. Golden fixtures cannot be built on that. AR-004 is therefore a prerequisite for VR-001 fixtures, not merely a throughput concern for crowd scenes.

Fixtures — precomputed inference, pulled by CI

The N100 cannot run inference at any useful rate, so inference output is precomputed on a GPU host and consumed by CI as data. This converts most of what looks like GPU work into pure CPU replay.

Fixture	Contents	Size	Storage
Edge-case dumps	~6 short clips (30–60 s), one per awkward behaviour	~0.1–1 MB each	Committed in-repo
Corpus dumps	Full-length titles from the validation corpus	~21–38 MB each	Gitea package registry, pinned by version + checksum
Synthetic gallery	Random unit-norm embeddings, fixed seed	small	Generated at test time
Golden truth files	Expected output for each edge-case dump	KB	Committed
Audio golden vectors	FLAC + expected signature + parameter contract	~600 KB	Committed, shared with the plugin repo

Edge-case dumps are small enough to commit, and being in-repo means they version with the code that reads them.

Corpus dumps go to the Gitea package registry, not Git LFS. Both are available — the models already use LFS — but their fetch semantics differ in a way that matters here. LFS objects are pulled on clone unless a developer explicitly skips them, so ~38 MB per title behind LFS taxes everyone who clones, forever, for data that only CI and the optimizer ever read. Registry artifacts are fetched on demand by the job that needs them.

Rule of thumb: LFS for what the build needs; the package registry for what a particular job needs. Models are the former; corpus dumps and the CI image (DP-007) are the latter.

Pin by version and verify by checksum on fetch. A fixture that changes silently under CI is worse than a missing one, because the failure presents as a code regression.

Generation must be reproducible and versioned. A script, run on a GPU host, regenerates every fixture from source clips; it is re-run when the VR-001 schema version bumps. A fixture whose provenance is unknown is worse than no fixture, because it will be trusted.

The limitation that must stay visible: replay fixtures freeze upstream behaviour. A test driven from a dump verifies AR-007 onward given those embeddings — it cannot detect a regression in detection, alignment or embedding, because those produced the fixture. Nothing in CI can. That gap is covered only by the T3 smoke test and the scheduled GPU run, and it should not be papered over by a high replay-coverage number.

Per-requirement verification plan

ID	Tier	Test asserts	Edge cases to cover
AR-001	T3	Detector returns plausible boxes on a known frame	— smoke only
AR-002	Minimum face size 32×32 px (VR-005 measured), expressed in original resolution (decoupled from dense_scale)	Faces below 66 px (original res) are dropped	Exactly at threshold; with dense_scale 0.5 — the interaction that motivated the requirement
AR-003	T2	No cap applied; a 40-face frame yields 40	Crowd frame
AR-004	T1	Saturated input blocks rather than drops or throws	Bounded queue at capacity; byte-based limit with large crops; SIGTERM mid-block
AR-005	T1	Known landmarks → expected 112×112 warp	Landmarks near frame edge; degenerate/collinear points
AR-006	T3	Embeddings are unit-norm	Batch smaller than, equal to, larger than embed_batch_size
AR-007	T2	Association picks the right track	Two faces crossing paths; one leaving frame as another enters
AR-008	T2	One pool; dormant tracks match on embedding, not IoU	Dormant track whose old bbox overlaps a different new face — must not match on position
AR-009/010	T2	Cut/boundary shifts weighting toward embedding	Cut with same people; cut with all-new people
AR-011	T1	TransNetV2 receives native-rate frames	Source at 24/25/30 fps — dedup window derived, not assumed
AR-012	T2	Window spans full track extent, not first recognition	Actor recognised only at track end — window must still start at first_seen
AR-013	T2	last_seen set/unset; window ends at last sighting	Gap just under vs just over timeout; reappearance after timeout → two windows
AR-014	T2	Belief swap closes one window, opens another	No blended window; no overlap at the swap frame
AR-015	T2	Two live tracks on one actor trigger re-association	Counter increments
AR-016	T2	Every track closed at EOF	Film ending mid-shot — window ends at final frame, not dropped
AR-017	T1	Claim carries posterior and route	Deferred and pooled routes distinguishable
AR-018	T1	Band admits only within bounds	At each bound exactly; store never admits below lower bound
AR-019	T2	Promotion only when all three signals quiet	Cut mid-track blocks promotion
AR-020	T2	Unknown resolved after expansion	Track failing at minute 12, resolved at EOF — the ordering-independence claim
AR-021	T2	Clustering merges same person, respects cannot-link	Temporally overlapping tracks never merge; measure how many merges the constraint rejects
AR-022	T1	Context crops retained, bounded per track	Track running for minutes
AR-023	T1	Sigmoid fit on synthetic separable data	Too few positive pairs → valid=false, fallback engages
AR-024	Static check	No bare cosine outside a tagged EXCEPTION	Grep-based; this is the invariant's enforcement
AR-025	T1	Log-odds accumulate; correlated frames discounted	30 identical frames must not reach the certainty of 30 diverse ones
AR-026	T1 + T4	GEMM path produces same result as reference loop	Equivalence on small input in CI; throughput on GPU host
AR-027	T4	Throughput at 10²…10⁵ actors	Scheduled, not on-demand
IR-001/002	T1	Serialised output matches golden file	Zero-length window; actor with many windows
IR-003	T1	Output written after deferred pass	Not at EOF
IR-004/005	T1	Signature matches golden vector bit-for-bit	Identical result in both producer repos
IR-006	T1 + manual	Queue pull and result push against a stubbed Jellyfin API	Partial result never pushed; push only after the deferred pass
IR-007	T1	Media < 120 s emits no signature at all	Exactly 120 s; just under; zero-length audio. Must match the plugin's cutoff exactly — a caller-varying window length is what SR-004 forbids
IR-008	T1	v1: prefix emitted and honoured on read	Unknown prefix rejected, not guessed
GR-009	T1	Human-confirmed associations persist and are tier-tagged	Survives a gallery rebuild; distinguishable from baked and harvested
GR-004	T1	Mismatched embedder → hard startup error	Error names both sides; unstamped warns, and errors under SAE_REQUIRE_GALLERY_STAMP; same filename + different SHA-256 must still be a mismatch
GR-008	T1	Outlier flagged among an actor's references	Injected poisoned embedding detected
VR-009	T1	Posterior calibration holds	A 0.99 posterior is wrong ~1% of the time on held-out tracks

Three of these are worth singling out because they verify claims that would otherwise be assertions: AR-012 (window starts at first_seen even when recognition comes late) is the entire point of the redesign; AR-020 (a track failing mid-film resolves at EOF) is the claim that ordering stops mattering; and AR-025 (30 identical frames ≠ 30 diverse ones) is what stops the Bayesian accumulation from being decoration.

---

Withdrawn

ID	Requirement	Reason
—	anneal_sec window merging	Superseded by AR-012/AR-013: a track survives its own gaps, so there is nothing to anneal
—	extinction_sec actor keep-alive	Superseded by AR-013: windows end at last sighting, which is what this over-claimed

Both were deleted rather than retained at zero — a field naming a mechanism the pipeline no longer has is actively misleading (see SPEC.md A6.6).

---

Notes on coverage

• VR-* traces to PR-002 (scene-granularity answers) rather than to a system requirement: parameter studies are single-repo work serving accuracy, and this is correct rather than a gap.
• PR-005 (leak nothing) has no AR/DP row. It is satisfied structurally by SR-004 and GR-005 — the server holds no binary, the gallery never leaves the instance — not by any component doing something. It cannot be verified by pointing at code, and it dies the moment either prohibition is relaxed.