diff --git a/docs/IMPLEMENTATION-PLAN.md b/docs/IMPLEMENTATION-PLAN.md new file mode 100644 index 0000000..1c01580 --- /dev/null +++ b/docs/IMPLEMENTATION-PLAN.md @@ -0,0 +1,627 @@ +# Implementation plan + +Companion to [`SPEC.md`](SPEC.md). Sequences the gaps identified there into +phases, with dependencies and acceptance criteria. + +Ordering principle: **land the things that make measurement trustworthy before +the things that require measurement.** The central change (A6) invalidates every +tuned constant in `config.hpp`, so anything that could corrupt a retune must be +fixed first, and the retune itself gates the release. + +| Phase | Contents | Gated on | Parallelisable | +|---|---|---|---| +| 0 | D3 model binding, E1 dump audit | — | with 4 | +| 1 | TrackRegistry + A6 track-extent presence | 0 | — | +| 2 | Retune (A6 constants), E4 | 1 | — | +| 3 | A8 unidentified capture, D2 gallery report | 0 | with 1, 2 | +| 4 | C2 audio signature | — | with everything | + +Phases 3 and 4 touch disjoint code from 1 and 2 and can proceed alongside them. + +--- + +## Phase 0 — De-risk measurement + +Small, self-contained, and prerequisite to trusting any number produced later. + +### 0.1 Gallery/model binding (D3) + +Stamp the embedder identity into the gallery at build time; verify at load. + +- `gallery_builder` writes the embedder model identity (filename + hash of the + ONNX, or an explicit version string) into the gallery file. +- Every consumer — `scene_analyze`, `replay.py`, the optimizer — checks it at + startup against the embedder it is about to use. +- Mismatch is a **hard error naming both sides**, never a warning. + +*Why first:* cross-model cosine similarities are meaningless but look entirely +plausible. A retune against a mismatched gallery would produce numbers that are +wrong and undetectably so. This is the cheapest insurance in the document. + +**Acceptance:** a deliberately mismatched gallery/model pair fails at startup +with a message naming both; a matched pair is unaffected. Covered by a test in +`tests/test_gallery_store.cpp`. + +### 0.2 Dump audit for track-aware replay (E1) + +The dump captures state at `EmbeddedSceneFrame` — *upstream* of tracking. After +A6, presence depends on tracker output, so replay must be able to reconstruct +tracks exactly as the live pipeline would. + +- Confirm `bbox`, `landmarks`, `confidence`, `is_cut`, `is_scene_boundary` and + frame timestamps are sufficient to re-run `FaceTrackerFunc` deterministically. +- Specifically verify the **cross-cut park/revive path** (A4) is reproducible: it + depends on `is_cut` and on last-frame embeddings, both of which should be + present — confirm rather than assume. +- If anything is missing, add it and bump `schema_version` in + [`scripts/optimizer/SCHEMA.md`](../scripts/optimizer/SCHEMA.md). + +**Acceptance:** for one film, tracks reconstructed from the dump are identical +(same `track_id` partitioning of the same faces) to those from a live run. +This equivalence test is the foundation of Phase 2 and should be kept as a +regression test. + +--- + +## Phase 1 — TrackRegistry and track-extent presence (A6) + +The central change. Presence moves from "frames where the actor was recognised" +to "extent of tracks the actor owns". + +### 1.1 Design: TrackRegistry as a shared resource + +**Decision:** track ownership lives in a `TrackRegistry` object that is *external +to the dataflow network*, not in a node and not inside `TrackGallery`. + +This is idiomatic for this codebase rather than a workaround: + +- Node functors are already constructed outside the network and passed in by + reference — `main.cpp:186-207` builds `ftracker_fn`, `tracker_fn`, `sink_fn` as + stack objects and `ObjectNode` wraps them ("the object must outlive the node", + KPN SPEC §371). +- KPN provides `SharedResource` (`external/KPN/shared_resource.hpp`) precisely + for state shared across nodes, and shared resources can be registered with a + network for reporting (KPN SPEC §163, §445). + +Why not the two alternatives: + +- **Not a node.** Ownership is not a stage in the stream — it is state that + several stages read and write, at different points, with the final answer only + known at EOF. Modelling it as a node would force ownership to be decided at a + single point in the flow, which is exactly what it cannot be. +- **Not inside `TrackGallery`.** Ownership is already computed there for + expansion, so putting presence there too would avoid duplication — but it + couples presence semantics to `expand_gallery`, a switchable feature. Turning + expansion off would silently revert A6. + +**Ownership and lifetime:** the registry is created in `main` and handed to every +node that needs it as a `std::shared_ptr`. Nodes hold their own +`shared_ptr`, so lifetime is guaranteed by refcount rather than by the +"object must outlive the node" convention — no ordering assumption between +network teardown and the registry's destruction. + +**Per-track state:** + +``` +Track + first_seen : double set once, at track creation + last_seen : optional UNSET while on screen; set to the last + on-screen timestamp when the face is lost + actor : optional set when a posterior crosses the threshold + belief : {actor_idx -> accumulated_logodds} Bayesian, not a tally (A9) + embedding : Embedding running directional mean, for association +``` + +`last_seen` carries the entire liveness state. Unset means *on screen now*; set +means *went off screen at T*. There is no separate "missing frames" counter and +no expired flag in the registry — the optional is the state machine. + +**Lifecycle:** + +``` +face detected, no match to an existing track + → new track, first_seen = t, last_seen = unset + → begin embedding + +actor identified + → record vote; set actor once the ownership rule fires + +face lost + → last_seen = t_last_on_screen (track stays alive, revivable) + +face seen again, embedding matches a track with last_seen set + → last_seen = unset (same track continues) + +tick(t) where last_seen is set and t - last_seen > timeout + → track is DEAD: emit it to the result aggregator, delete the entry +``` + +**A dead track is passed to the result aggregator.** Reaping is a handoff, not a +deletion into a holding pen: the reaped track — `first_seen`, `last_seen`, owning +actor, vote tally — is pushed downstream the moment it dies, and the registry +drops it. The registry therefore contains **only live tracks**, and its size is +bounded by concurrent on-screen faces rather than growing with the film. + +That gives a clean division: the registry answers "who is on screen now and which +tracks are still revivable"; the aggregator accumulates finished presence. Neither +needs the other's state. + +A face appearing after the timeout starts a genuinely new track with a new +`track_id` — correct, since past the re-acquisition window there are no grounds +to assert continuity. An actor who leaves for half an hour and returns gets two +windows rather than one spanning their absence. + +The registry thus has exactly three states, all implied by `last_seen`: on screen +(unset), revivable (set, within timeout), and gone (emitted and removed). There is +no "expired but retained" state to reason about. + +**Closing a track *is* the presence assertion.** This is the property the design +turns on. There is no later stage where presence gets decided, reconciled or +merged — the act of closing a track emits exactly one complete, immutable claim: +*this actor was on screen from a to b*. It is produced once, never revised, and +carries everything needed to justify it (the vote tally that named the actor). + +Several things follow, which is why it is worth naming explicitly: + +- **Presence is append-only.** The output is a stream of finished claims, so a + partial result is a *prefix* of the full result, not a corrupted version of it. +- **Each claim is independently checkable.** A wrong window can be traced to one + track and its votes, rather than to an emergent interaction between three + timeout constants. +- **Nothing downstream needs to be clever.** The aggregator groups claims by + actor and writes them out. It holds no state machine of its own, which is + precisely why `SceneTrackerFunc` disappears. + +Compare the current design, where presence is inferred at the end from a pile of +per-frame detections via two gap-bridging constants, and no single moment is "the +decision". That indirection is the source of both the late-start bug and the +untunability. + +An actor's presence window is simply `[first_seen, last_seen]` of each track they +own. Nothing else. + +**This removes annealing entirely.** `anneal_sec` (35.5 s) and `extinction_sec` +(57.4 s) exist only because presence is currently assembled from *isolated +accepted frames*, which are full of holes — both constants are gap-bridging +patches over that. Under this model a track survives its own gaps by +construction: a face lost and re-acquired by embedding match is the **same +track**, so there is no second window to merge and nothing to anneal. Presence +continuity is inherited from track continuity rather than reconstructed from it. + +This is also what makes camera cuts fall out for free. A shot/reverse-shot sets +`last_seen` on the cut and unsets it on the next matching detection; the extent +never breaks. The existing cross-cut park/revive machinery +(`face_tracker_node.hpp:154-172`, `cut_revive_sim`) is exactly this mechanism +already, applied only to cuts — the model generalises it to every disappearance, +which is why cuts stop needing a special case. + +**One timeout, not three.** The only surviving tunable is how long a track stays +revivable after the face is lost. It does two jobs at once, and both are wanted: + +- **Gap absorption.** A face lost at `t₁` and re-acquired at `t₂` within the + timeout never closed its track, so the window runs straight through — the actor + **is claimed present across `[t₁, t₂]`**. That is correct: someone who turns + away, is briefly occluded, or is off-camera while the shot cuts to whoever they + are talking to has not left the scene. +- **Identity continuity.** The re-acquisition is only accepted on an embedding + match, so the gap is bridged on evidence that it is the *same person*. + +That second point is the substantive improvement over `anneal_sec`, which merged +windows purely on elapsed time and could therefore stitch together two different +people. Same smoothing, now evidence-gated. + +**The asymmetry that keeps this honest:** interior gaps are claimed, the trailing +cool-down is not. A track that dies ends its window at `last_seen` — the last +frame the face was actually seen — not at the moment of death. So the timeout +buys gap-smoothing without over-claiming the tail. + +**The documented credits overshoot is fixed by this asymmetry, not by scene +detection.** `extinction_sec` kept an actor *active* for 57 s after their last +detection, actively emitting presence into the closing credits (the Downton Abbey +recall collapse, `lvface-deep-dive.md`). A window ending at `last_seen` never +enters the credits at all, because nothing was seen there. + +**Cuts and scene boundaries are association hints, not presence events.** Both +signals say the same thing to the tracker: *spatial continuity is broken — stop +trusting IoU, associate on embedding similarity instead.* Neither closes a window. + +| Signal | Meaning | Effect | +|---|---|---| +| `is_cut` (histogram) | Camera-angle change within a scene | Weight association toward embedding | +| `is_scene_boundary` (TransNetV2) | Different scene | Same, more strongly | + +This is what the tracker's park/revive path already does for cuts +(`face_tracker_node.hpp:154-172`): a post-cut detection is matched on raw cosine +similarity to a parked track's last-frame embedding, with IoU out of the picture. +Generalising it makes `track_alpha` (the spatial/embedding cost weight) +frame-dependent rather than constant — normal frames use the tuned blend, flagged +frames drop toward embedding-only. + +An actor genuinely continuing across a boundary is therefore *kept*, which is +correct; one who does not reappear simply times out and closes at `last_seen`. + +This makes `--scene-detect` **load-bearing for presence correctness**, not the +opt-in extra it is today (`config.hpp`, default off). Two consequences to decide +in Phase 2: whether it becomes default-on despite its dense-decode cost, and what +the degraded behaviour is when it is off — presumably the timeout alone, which is +the current situation and carries the known overshoot. + +It follows that the timeout **is** a presence knob and **cannot** simply be made +generous: it is precisely "how long an absence do we tolerate before calling it a +departure". Too short fragments one continuous appearance into several windows; +too long absorbs a genuine exit-and-return into a single window claiming presence +the actor did not have. It replaces both `anneal_sec` and `extinction_sec` and +inherits their tuning burden — this is the constant Phase 2 must fit. + +`track_max_frames_missing` and `cut_inactive_max_frames` already serve this role +in the tracker and should be reconciled into it. Note the units differ: the +tracker counts *frames* missing while windows are in *seconds*; whichever side +owns the timeout should own the conversion, so `sample_fps` changes cannot +desynchronise them. + +**Ownership rule** (per SPEC A9): each track carries accumulated **log-odds per +candidate actor**, updated as frames arrive. Ownership is "posterior exceeds +threshold", not "≥ N accepted frames" — so `votes` in the state sketch above is a +`{actor_idx → accumulated_logodds}` map, and `on_vote()` is an *update*, not an +increment. + +Consequences for the registry: + +- Every similarity reaching it is already a probability (A9); the registry never + sees a raw cosine. +- Correlated-frame discounting (A9) applies at update time. The registry should + take the already-discounted evidence rather than deciding the discount itself — + that judgement belongs with the diversity buffer that identifies novel poses. +- Ownership is established at **first crossing**, not deferred to track death. + +**Two contradiction rules the registry must enforce** (SPEC A6). Both exist +because the same underlying fault — a missed camera or scene change — shows up in +identity space, and both are detectable *online*: + +| Condition | Meaning | Action | +|---|---|---| +| Belief on one track swaps A → B | `track_id` carried across a viewpoint change onto a different person | Close the track at `last_seen`, open a new one for B at the swap frame | +| Two **live** tracks owned by the same actor | One person split into two tracks, or an identity attached to the wrong one | Treat as a detected cut: reset the affected tracking state and re-associate on embedding | + +The second is the more interesting: it makes identity a **third cut detector**, +independent of the histogram and TransNetV2, firing exactly where those failed. A +cut subtle enough to evade pixel-based detection is not necessarily subtle in +identity space. + +Both must be **counted and reported** (SPEC D2) — the rates are a direct measure +of how often tracking is silently wrong, which nothing currently reveals. + +Implementation note: "two live tracks owned by the same actor" is a cheap check +because the registry already holds every live track and its belief. Maintain a +reverse index `actor_idx → live track_ids` and the condition is detected on the +update that causes it, not by scanning. + +**The registry *is* the tracker's state.** `FaceTrackerFunc` does not keep its own +`tracks_`/`inactive_` maps and mirror them into a registry — it is constructed +with `std::shared_ptr` and operates on it directly. One copy of the +track set, one owner of liveness. + +This matters beyond tidiness. Had the tracker kept private maps and *reported* +into a parallel registry, the two could disagree — the tracker expiring a track +the registry still thinks alive, or vice versa — and every such divergence would +surface as wrong presence windows, silently. Merging them makes that class of bug +unrepresentable rather than merely tested-against. + +It also subsumes the tracker's existing two-pool split: `tracks_` becomes the +tracks with `last_seen` unset, `inactive_` becomes those with it set. Same +structure, one map, and the cross-cut park/revive path becomes the general +re-acquisition path rather than a special case. + +**Interface:** + +``` +TrackRegistry + // tracker-facing: state it owns and mutates + tick(timestamp) ← FaceTrackerFunc, every frame + candidates() -> span → all live tracks; last_seen tells + the caller whether IoU applies + create(timestamp, embedding) -> track_id + mark_seen(track_id, timestamp, embedding) → updates mean, clears last_seen + mark_lost(track_id, last_on_screen_timestamp) + + // matcher-facing + on_vote(track_id, actor_idx, posterior) ← IdentityMatcherFunc (accepted frames only) + + // reader-facing + owner(track_id) -> optional → TrackGallery (expansion confirmation) + + // output: dead tracks are pushed out as they are reaped + on_track_dead : callback(DeadTrack) → ResultSinkFunc + flush() ← at EOF: emit all live tracks, then clear +``` + +`DeadTrack` carries `first_seen`, `last_seen`, the owning `actor_idx` (or none), +and the vote tally — everything the aggregator needs, with no back-reference into +registry state. + +**Every track must be closed at EOF.** `flush()` emits every still-live track +through the same callback, closing each at `last_seen` if set and at the final +tick timestamp otherwise. This is not a tidy-up detail: a film almost always ends +with faces on screen, and those tracks have not timed out, so without an explicit +flush they are simply never emitted — the closing scene's actors disappear from +the output. That failure is silent and looks like a recognition miss rather than +a bookkeeping bug. + +Requirements: + +- `flush()` is idempotent and leaves the registry empty; calling it twice emits + nothing the second time. The sink's existing `written_.exchange(true)` guard + (`result_sink_node.hpp:66`) shows the shape. +- It must run on **every** termination path that produces output, not just clean + EOF — the `eof` sentinel, and the early-exit paths (`--end-sec`, decode error, + user interrupt) if those still write results. +- Finalisation goes through the same code path as a natural death, so a track + closed by EOF is indistinguishable in form from one closed by timeout. +- Deliberately *not* covered: SIGTERM during opportunistic runs (SPEC B4). Those + push no partial result at all, so there is nothing to flush — the item stays + pending and restarts. Flush is for runs that produce output. + +**Acceptance:** a clip ending mid-shot yields a window for the on-screen actor +whose `end` equals the final frame timestamp. This is a specific test, not an +incidental one. + +The tracker's association step reads `candidates()` — **one pool, not two**. A +track with `last_seen` unset was seen last frame, so IoU is meaningful; one with +`last_seen` set is dormant and matched on embedding alone. There is no separate +revival path: matching a dormant track is ordinary inter-frame association, and +the property falls out of the embedding comparison rather than being a mechanism. +The step then calls `create`/`mark_seen`/`mark_lost`. Reaping happens in `tick()`. + +Since the tracker mutates registry state across a frame's association pass, that +pass needs to be atomic as a unit — a `frame_scope()` handle holding the lock for +the duration is cleaner than making each accessor independently locked and hoping +the composite is safe. This is the one place where per-call atomicity is *not* +sufficient. + +`tick()` advances the clock so dead tracks are reaped independently of detection +activity — without it the registry only learns about time when something is +detected, and tracks would only die when some *other* face happened to appear. It +is called once per sampled frame whether or not that frame had detections. + +**Locking.** Nodes run concurrently and do not coordinate, so the registry is +responsible for its own consistency. Two granularities apply: the tracker's +per-frame association pass holds the lock for its whole duration (`frame_scope()` +above), while every other caller's operations must be individually atomic. + +Two cases constrain the API shape: + +- `owner()` is a **read-modify-read** in disguise: `TrackGallery` calls it to + decide promotion while `IdentityMatcher` may be concurrently voting on the same + track. Tally and verdict must be read under one lock as a snapshot — not "read + tally, release, decide" — or a track can be both unowned and owned within a + single promotion decision. +- `on_vote()` for a frame arrives from `IdentityMatcher`, downstream of the + tracker's `tick()`/`mark_seen()` for that same frame. A vote may therefore land + after the clock has moved on. Rule: a vote for a known track always lands on + that track's tally, regardless of clock position. Only reaping is clock-driven. + A vote for a track already reaped is dropped and **counted** — a nonzero count + means the timeout is shorter than the matcher's lag, which is a real + misconfiguration and should not fail silently. +- The `on_track_dead` callback fires from inside `tick()`, which the tracker calls + while holding the frame lock. The callback must therefore not re-enter the + registry, or it self-deadlocks. Keep it to a push onto the aggregator's own + storage; anything heavier belongs downstream of that. + +A single `std::mutex` over the whole registry is the right starting point: +contention is a handful of small updates per frame against per-frame work +measured in GPU milliseconds. Anything finer needs a profile, not an assumption. + +### 1.2 Wiring + +- `FaceTrackerFunc` is constructed with the `shared_ptr` and uses + it as its state — its own `tracks_`/`inactive_` maps go away. Per frame: + `tick()`, then association over `candidates()`, then + `create`/`mark_seen`/`mark_lost`. Its `tracks_`/`inactive_` split and the + cross-cut revival branch both collapse into one pool keyed on `last_seen`. +- `IdentityMatcherFunc` reports accepted-frame votes. It keeps its existing + per-frame acceptance logic unchanged — A6 changes what is *done* with + acceptances, not how they are decided. +- `TrackGallery` replaces its internal confirmation counter with + `registry.owner()`, so ownership is computed **once**. +- `ResultSinkFunc` becomes the result aggregator: it receives dead tracks via the + callback and groups them by actor. The per-frame timestamp collection and + gap-merging at `result_sink_node.hpp:123-147` is deleted. **No annealing pass** + — a dead track already *is* a window. +- `SceneTrackerFunc` (`scene_tracker_node.hpp`) is the extinction-timer state + machine keyed on `actor_idx`. Under this model it has nothing left to do: its + entire job was keeping actors alive across detection gaps. Expect to delete it + from the network rather than adapt it. + +### 1.3 Comparing against current behaviour + +The old path is not worth preserving behind a flag. It is not a variant of the +new one — it is a different pipeline shape (`SceneTrackerFunc` present, annealing +in the sink, two extra constants), so keeping both runnable means maintaining two +sink implementations and a node that otherwise gets deleted. + +Compare against **recorded output** instead: keep the current binary's results for +the validation corpus as reference JSON, and diff the new pipeline against them. +That gives the same A/B for Phase 2 without carrying dead code through it. + +**Acceptance:** + +- `tests/test_face_tracker.cpp` extended: a lost-then-re-acquired face continues + the **same** track and yields one unbroken window; a cut does the same; a face + re-appearing *after* the timeout yields two separate tracks and two windows; + a two-actor conflict resolves to the majority and increments the conflict + counter; a single-frame track yields a zero-length window. +- A registry test: reaping emits exactly once per track; `flush()` at EOF emits + every live track and nothing twice; a vote landing on a reaped track is dropped + and counted. +- Contradiction tests: a belief swap A→B closes one window at `last_seen` and + opens a second starting at the swap frame, with no overlap and no blended + window; two live tracks converging on one actor trigger a re-association and + increment the counter. +- A concurrency test hammering `tick`/`mark_seen`/`mark_lost`/`on_vote` from + multiple threads against `owner()`, under TSan. The vote-tally-plus-verdict read + is the case to target — correct only if atomic as a unit. +- On a known film, every actor's first window starts no later than in the recorded + reference output, and strictly earlier for at least one — the late-start bug + this change exists to fix. + +--- + +## Phase 2 — Delete two constants, tune the rest + +`anneal_sec` (35.5) and `extinction_sec` (57.4) are **replaced, not discredited**. +They exist to answer X-Ray's scene-level question — "is this actor in this scene" +— by holding windows open across cuts, and they answer it with elapsed time +because that was the only signal available at the sink. Phase 1 answers the same +question with better evidence: an embedding-matched re-acquisition, plus a true +scene boundary to stop at. So they are deleted along with `SceneTrackerFunc`, and +their *job* transfers to the re-acquisition timeout rather than disappearing. + +What actually needs tuning: + +- **The ownership posterior threshold** — replaces both `prob_threshold` (0.754) + as a presence decision and `expand_min_anchor_frames` (3) as a vote count. A + track is owned when its accumulated posterior for an actor crosses this (A9). + A single false accept can no longer create a window on its own, so the + operating point should sit lower than the old per-frame threshold. +- **The correlated-frame discount** (A9) — whatever form it takes, it is a fitted + quantity and belongs in the sweep. It directly controls how fast belief + accumulates along a track, so it trades against the ownership threshold and + cannot be tuned separately. +- **The re-acquisition timeout** — reconciled from `track_max_frames_missing` (5) + and `cut_inactive_max_frames` (5). **This is the successor to both `anneal_sec` + and `extinction_sec` and carries their tuning burden.** It decides how long an + absence is absorbed into a presence window versus treated as a departure, so it + trades recall (bridging real gaps) against precision (claiming presence during + a genuine exit) directly. Current values are 5 *frames*, inherited from a + tracker-continuity role; as a presence constant it is likely to want a much + larger value, and should be swept over seconds rather than nudged. + +- **`scene_detect`** — now load-bearing for presence (§1.1), so the sweep must + cover *with* and *without*, and `scene_threshold` (0.60) becomes a presence + constant rather than a diagnostic one. A missed boundary reintroduces the + overshoot; a spurious one truncates a scene's cast early. + +Method per SPEC E3: DE over the validation corpus, objective micro-F1, but +precision and recall logged at every evaluation and printed at the optimum. X-Ray +recall is a face-vs-cast-in-scene ceiling, so unconstrained F1 pushes thresholds +down chasing unreachable recall and trades away real precision. Pick the operating +point deliberately from the trajectory. + +**This resolves SPEC Open Question 1 by construction rather than by measurement.** +The question was whether `extinction_sec` survives alongside track extents; the +answer is that the mechanism it patched no longer exists. + +One thing to watch: the search space is now 3 knobs instead of 3 gap-constants, but +they are *not* independent — a longer re-acquisition timeout means longer tracks, +which means more frames to clear `expand_min_anchor_frames`. Sweep jointly. + +**Acceptance:** `anneal_sec` and `extinction_sec` removed from `config.hpp` and +`Config`; new constants committed with the trajectory and the precision/recall +trade-off documented in the manner of `rep4-optimizer-results.md`; results +compared against the recorded reference output from 1.3. + +--- + +## Phase 3 — Diagnostics + +Independent of Phases 1–2; can run in parallel. + +### 3.1 Unidentified-track capture (A8) + +Today only *promoted* mugshots are dumped (`expand_debug_dir`) — the successes. +This captures the failures. + +- Flag `--dump-unidentified `. +- For every track never identified: all embeddings, track metadata (`track_id`, + first/last timestamp, frame count, per-frame bbox and confidence), and **the + best similarity achieved and which actor it was against**. +- That last field is the point of the feature: it separates "actor missing from + the gallery" from "actor present but scored below threshold" — a gallery + coverage problem versus a threshold problem. +- Crops are **opt-in** (`--dump-unidentified-crops`); embeddings + metadata are + the default. Crops for every unidentified track across a library is a lot of + disk. +- When crops are enabled, store both the 112×112 aligned crop *and* a wider + **context crop** for a bounded number of representative frames per track. The + aligned crop serves diagnostics; the context crop serves the human-in-the-loop + association capability ([`../../SPEC.md`](../../SPEC.md) §4), where someone has + to actually recognise the person — which a tightly-cropped, geometrically + normalised face often makes impossible. +- Naturally expressed against `TrackRegistry`: unidentified = tracks with no + owner at EOF. + +**Acceptance:** on a film with a known out-of-gallery face, that track appears in +the dump with its near-miss actor and similarity. + +### 3.2 Gallery build report (D2) + +Surface what the calibration already computes internally but discards: + +- actors with zero usable images (a silent recall ceiling); +- actors below the 5-embedding threshold for positive pairs; +- near-duplicate references removed; +- the fitted calibration **and the intra/inter similarity distributions behind + it** (`gallery_calibration.hpp` histograms these at `kHistBins = 200` and + throws them away — persist them). + +**Acceptance:** a build report written alongside the gallery; the intra/inter +PDFs are recoverable for inspection. + +### 3.3 Resolve the prior (SPEC Open Question 4) + +SPEC A9 asks for a prior of `intra/(intra+inter)`; the shipped default is +`match_prior = 0.5` (use the calibrated sigmoid directly). These disagree. With +3.2 landed the real value is known, so: either adopt it, or document 0.5 as a +deliberate override with the reason. Cheap once the distributions are persisted. + +--- + +## Phase 4 — Audio signature (C2) + +Largest self-contained chunk; no dependency on any other phase. + +- Implement [`JRay-public-server/SPEC.md` §3](../../JRay-public-server/SPEC.md) + **exactly**: 120 s centred on the midpoint, mono 11025 Hz, 4096/1024 Hann STFT, + 300–3000 Hz, 32 log bins, peak bin + 2-bit energy class, one byte per frame, + base64. +- Audio decode is a second stream from the FFmpeg dependency already linked for + video (`ffmpeg_decoder.hpp`) — not a new dependency. +- Emit in the truth file → **`schema_version` bump**, coordinated with + `jRay/SPEC.md` and the plugin. Neither can be changed unilaterally. +- **Golden-vector cross-check is a hard requirement**, not a nicety: two + independent implementations of the same DSP chain will drift. Fixture: a short + audio file with its expected signature, checked into both repos and asserted in + both test suites (SPEC Open Question 3, resolved this way — a shared fixture + rather than a shared implementation, since the coupling cost of the latter + exceeds the benefit). +- This pipeline **produces only**. Matching and offset recovery stay consumer-side. + +**Acceptance:** pipeline and plugin produce identical signatures for the same +file; the golden-vector test passes in both repos. + +--- + +## Deferred + +Deployment (SPEC Part B) is deliberately excluded from this plan. B3 (on-demand +service) and B4 (opportunistic worker) are packaging concerns over a stable core, +and the core is about to change under A6. `service-conversion.md` remains the +design of record; it should be executed once Phase 2 fixes the constants, so the +installer is not shipping values that are about to be replaced. + +One exception worth pulling forward if convenient: the **temp-file cleanup fix** +in `run_from_jellyfin.py` (SPEC B4) is small, independent, and a named +prerequisite for the worker. + +--- + +## Risks + +| Risk | Mitigation | +|---|---| +| A6 widens presence and precision drops more than recall gains | Phase 2 leaves the operating point explicit; both modes runnable via flag for direct comparison | +| Retune produces a worse optimum than the current constants | Trajectory is logged; the old operating point stays available. A6 is behaviourally correct even if the metric disagrees — decide deliberately, do not let the metric silently veto it | +| Track-ID collisions merge two people into one extent | Conflict counter (1.1) makes the rate visible; `expand_track_spread_max` already guards the expansion side | +| Registry races produce non-deterministic presence between identical runs | Every operation atomic as a unit (1.1); TSan test in Phase 1 acceptance. A race here is especially costly — it would surface as irreproducible optimizer scores in Phase 2, where it would look like metric noise rather than a bug | +| X-Ray metric blindness | `methodology.md` documents the known failure (scene-union hid out-of-cast FPs). Any metric change gets checked for the same class of blindness | +| Two audio implementations drift | Golden vectors in both repos (Phase 4) | diff --git a/docs/SPEC.md b/docs/SPEC.md new file mode 100644 index 0000000..4488f84 --- /dev/null +++ b/docs/SPEC.md @@ -0,0 +1,1210 @@ +# scene-actor-extraction — software specification + +Status: **draft**. Consolidates the requirements for the extraction pipeline, its +deployment modes, and its integration contracts. + +**This is a software spec implementing the system spec at +[`../../SPEC.md`](../../SPEC.md).** Requirements that span more than one repo — +presence semantics, schema-version coordination, gallery locality, identity keys +— are owned there. Where this document restates one, the system spec governs. + +This is a *requirements* document, not a design doc. Most of what follows is +already built; each requirement therefore carries a **Current** / **Gap** note so +the document doubles as a work list. Where a requirement is fully met, the gap +reads "none". + +Related documents, which this spec references rather than restates: + +| Document | Owns | +|---|---| +| [`../../jRay/SPEC.md`](../../jRay/SPEC.md) | Truth-file format and the Jellyfin plugin's read API | +| [`../../JRay-public-server/SPEC.md`](../../JRay-public-server/SPEC.md) | Jmanifest exchange format, cut matching, audio signature (§3) | +| [`scripts/optimizer/SCHEMA.md`](../scripts/optimizer/SCHEMA.md) | Embedding-dump HDF5 layout | +| [`service-conversion.md`](service-conversion.md) | Idle-GPU worker design (one deployment mode of §B) | +| [`methodology.md`](methodology.md) | X-Ray scoring methodology used by the optimizer | + +--- + +# Part A — Algorithm requirements + +The processing chain is a KPN dataflow network (`src/nodes/`). Requirements +below are ordered along that chain. + +## AR-001 … AR-003 — Face detection + +Detect faces in sampled video frames. + +- Detector runs on frames sampled at `sample_fps`, not every decoded frame. + The default (1.0) originates from I-frame decode speed, not from an accuracy + requirement — it is a **cost knob and may be adjusted**. Under scene-scoped + presence (SR-002) a lower rate still answers the question, but it lengthens the + interval between samples and so weakens IoU-based association; sweep the two + together (VR-002). +- **Minimum face size is 66×66 px**, expressed in **original video resolution**, + not decoded-frame pixels. Stating it in original space decouples it from + `dense_scale`: otherwise a 0.5 downscale silently doubles the effective + threshold, and dense mode is exactly what scene detection uses. + 66 is a working estimate of where ArcFace embeddings stop being reliable, not a + measured value — it should be replaced by the result of VR-005. +- Emits bounding box, detector confidence, and 5-point landmarks. +- Bounding boxes must be reported in **original video pixel space**. When + `dense_scale < 1` downscales the decoded frame, coordinates are rescaled by + `bbox_upscale` before leaving the pipeline. +- **No fixed cap on faces per frame.** `max_faces` (10, largest-first) is removed + so crowded scenes do not systematically lose their background cast — which + X-Ray credits as scene members (SR-002). See the backpressure requirement below; + the cap is currently the only thing bounding per-frame cost, so it cannot be + removed on its own. + +**Current:** SCRFD-500MF via `face_detector_node.hpp`, thresholds in `config.hpp` +(`detector_conf` 0.5, `detector_nms` 0.4), `min_face_px` 40, `max_faces` 10. + +**Gap:** `min_face_px` → 66 and re-expressed in original resolution; `max_faces` +removed, gated on backpressure (AR-004). + +## AR-004 — Backpressure + +Removing the per-frame face cap makes the number of faces entering the pipeline +unbounded and content-dependent — a crowd scene can produce an order of magnitude +more than a dialogue scene. The network must absorb that by **slowing down**, not +by dropping work or growing without limit. + +- The embedder is the bottleneck and must exert backpressure upstream: when its + input is saturated, the detector and decoder block rather than queue. +- Channel capacities are currently fixed at 16 (`main.cpp:204-207`) and were + chosen against a bounded ≤10 faces/frame. They must be re-derived, and + overflow must block rather than throw. +- `kMaxFaces` in `identity_matcher_node.hpp:133` **throws** when exceeded. With no + cap upstream that becomes a crash on crowd scenes; it has to go or become a + batching bound rather than an error. +- Memory is the real limit: faces carry 112×112 crops plus 512-float embeddings. + Backpressure must engage on bytes in flight, not just item counts. + +**Gap:** entire requirement. This is a prerequisite for removing `max_faces`, not +a follow-up to it. + +## AR-005 — Face alignment and crop + +Produce the exact input ArcFace expects. + +- 112×112 BGR crop via the standard ArcFace 5-point similarity transform. +- Landmark order must match the SCRFD/ArcFace convention (left eye, right eye, + nose, left mouth, right mouth). +- Alignment is the *only* geometric normalisation; no additional augmentation at + inference. + +**Current:** `align_face()` in `src/face_utils.hpp:9-22`, `cv::warpAffine` to +`{112, 112}`. **Gap:** none. + +## AR-006 — Embedding + +Generate a 512-d embedding per aligned crop. + +- Output embeddings are **L2-normalised**, so cosine similarity is a plain dot + product. Every downstream threshold assumes unit norm. +- Faces are embedded in batches of at most `embed_batch_size` to bound + per-call latency. +- The embedding model is a build-time choice; the gallery must have been built + with the *same* model (see GR-004). + +**Current:** `embedder_node.hpp` + `face_embedder_engine.hpp`; default +LVFace-B_Glint360K. **Gap:** none. + +## AR-007, AR-008 — Tracking + +Link detections across frames into tracks representing one physical person. + +Association cost combines three signals: + +- **Spatial** — IoU between the candidate detection and the track's last box. + Meaningful only for tracks seen in the immediately preceding frame. +- **Appearance** — similarity to the track's running directional mean embedding + (averaged, then re-normalised to the unit sphere), **expressed as a + probability** (AR-024), never as a raw cosine. +- **Weighting** — `track_alpha` interpolates the two, and is **frame-dependent**: + on `is_cut` / `is_scene_boundary` (AR-009, AR-010) it drops toward embedding-only, because + position carries no information across a viewpoint change. + +**There is no separate re-acquisition mechanism.** A track whose face is lost sets +`last_seen` and stays a candidate for association until extinction; matching it to +a later detection is ordinary inter-frame tracking, not a distinct revival path. +The property falls out of the embedding comparison the tracker already does. + +Consequences: + +- **One candidate pool**, not an active set plus a parked set. `last_seen` alone + distinguishes them, and it only affects whether IoU means anything — dormant + tracks are matched on embedding, since time has passed and position is stale. +- The current cross-cut park/revive path (`cut_revive_sim`, + `cut_inactive_max_frames`) is this same mechanism special-cased to cuts. It + collapses into the general path. +- A track dies only by extinction: `last_seen` set for longer than the timeout. + +**Current:** `face_tracker_node.hpp` maintains separate `tracks_`/`inactive_` +maps with distinct thresholds and a revival branch. **Gap:** unify into one pool +keyed on `last_seen`; make `track_alpha` frame-dependent; move association into +probability space (AR-024). + +## AR-009 … AR-011 — Cut and scene-boundary detection + +Two distinct signals, deliberately kept separate: + +- **`is_cut`** — always on. Grayscale histogram correlation below + `cut_threshold` flags an *intra-scene camera-angle change* (shot/reverse-shot). +- **`is_scene_boundary`** — opt-in (`--scene-detect`). TransNetV2 over a densely + decoded, downscaled stream flags a *true shot/scene boundary*. + +Both feed AR-007 as **association hints**: they tell the tracker that spatial +continuity is broken and that association should weight embedding over IoU. +Neither ends a presence window (AR-012). + +In dense mode the source decodes at `scene_decode_fps` (default 12) and a +decimator splits the stream: full-resolution sampled frames to the face pipeline, +downscaled dense frames to the scene detector +(`frame_source_node.hpp:63`). `sample_fps` is independent of this — the face +pipeline's rate is not derived from the scene detector's. + +Boundary timestamps are keyed off each frame's real timestamp, so they stay +correct despite the reduced rate. + +`dense_scale` must stay ≥ 0.5 on 1080p sources — it also shrinks what the face +detector sees. + +### Every model gets the input it was trained for + +A general rule, stated once here because it applies throughout: **models are fed +their expected input, not a cheaper approximation.** Where cost must come down, +it comes down by running the model less often or on fewer regions — never by +degrading what a single inference sees. A model run off-distribution produces +confident, plausible, wrong output, and the error is invisible without a study +that should not have been necessary. + +Two places this is currently violated: + +1. **`scene_decode_fps = 12` starves TransNetV2.** `kWindow` is 100 frames. At + native 25 fps that window spans ~4 s; at 12 fps it spans ~8.3 s, so the model + sees roughly half-speed motion over twice the temporal context it was trained + on. **Requirement: feed TransNetV2 at the source's native frame rate**, so a + 100-frame window covers the duration the model expects. The + "tolerates ~12fps" note in `config.hpp` describes a compromise, and the + recorded margin is consistent with it — a non-boundary baseline at ~0.50 with + real boundaries reaching only ~0.7+ is a compressed separation, not a healthy + one. + +2. **Hardcoded 25 fps in boundary dedup.** `scene_detector_node.hpp:138` merges + boundaries closer than `0.04 s` — "~1 frame @25fps". **Requirement: derive + this from the source's actual frame rate.** + +Dense decode is the pipeline's cost driver, so (1) is not free. The cost is +accepted: the alternative is a boundary signal that steers association (AR-007) while +being quietly unreliable. `dense_scale` remains available as a spatial reduction, +since downscaling is a documented, understood degradation rather than a temporal +one the model has no defence against. + +**Current:** histogram cut in the decoder; `scene_detector_node.hpp` for +TransNetV2. **Gap:** native-rate dense decode; framerate-derived dedup; +`--scene-detect` is default-off despite now feeding association. + +## AR-012 … AR-017 — Track-level identity propagation — **CHANGED BEHAVIOUR** + +> **Requirement:** when a face is identified, it is considered identified for the +> **entire track**. + +A track is one physical person by construction (AR-007), so an identification +anywhere on the track is evidence about all of it. Presence windows must +therefore be derived from **track extents**, not from the subset of frames that +happened to match. + +Required semantics: + +- A track is **owned** by actor A once the accumulated posterior for A exceeds + the ownership threshold (AR-025). This is a Bayesian accumulation over the track's + frames, not a count of accepted frames — so a single decisive observation and a + long run of marginal ones are distinguished rather than conflated. +- Once owned, the actor is present for the track's **full extent** — + `[first_seen, last_seen]`, including frames where the face was turned, + occluded, or simply scored below threshold. +- **A track with no owner emits nothing.** No posterior crossed, so there is no + claim to make. Such tracks are captured for diagnosis (AR-022) but do not appear in + output. See "Unidentified presence" below. +- **Every presence claim carries its belief.** The accumulated posterior that + justified the claim travels with it — through the aggregator, into the truth + file (IR-001), and onward. Confidence is not a serialisation concern bolted on at + the boundary; the pipeline computes belief for every decision it makes (AR-025), so + discarding it and emitting a bare interval throws away information that was + free. + + A claim also carries **how it was identified** — live, deferred (AR-020), or + pooled from a cluster (AR-021) — because those routes rely on different evidence. + A deferred identification leans on harvested references rather than baked ones; + a pooled one aggregates across tracks. The posterior already reflects the + strength of each, but the provenance tells a consumer *why*, and tells us which + route is producing errors when one is. + +### Belief swap terminates the track + +If a track is owned by A and the accumulated belief subsequently swaps to a +different actor B, **the track is terminated and a new one started for B**. The +old track closes normally at its `last_seen`; the new track begins at the frame +where the swap occurred. + +The rationale is that a swap is not a correction — it is almost certainly a +**missed camera or scene change**, where the tracker carried one `track_id` +across a viewpoint change and began following a different person. Two genuinely +different individuals both accumulating a threshold-crossing posterior on the +same face is not realistic short of identical twins; a `track_id` spanning two +people is. + +Treating it as a swap-and-continue would emit one window blending two people. +Treating it as a track boundary yields two windows that are each correct. + +Ownership is therefore **established at first crossing, not deferred to track +death** — the first crossing is a real claim about the frames seen so far, and a +later contradiction ends that claim rather than revising it. + +### Simultaneous ownership + +If two actors cross the threshold on one track at effectively the same time, the +**highest posterior wins**. This should be rare — the swap rule above catches the +common form of the problem — and it must be counted and reported (GR-003) as a +track-ID collision indicator. + +### Identity contradiction is a cut detector + +If **two live tracks are both owned by the same actor**, at least one is wrong: a +person cannot be in two places at once. The cause is the same as the belief swap +— a missed camera or scene change that split one person into two tracks, or +attached an identity to the wrong one. + +**Requirement: treat this as a detected cut.** Reset the affected tracking state +and re-associate on embedding, exactly as an `is_cut` signal would (AR-007, AR-009). + +This gives a third cut-detection signal, derived from *identity* rather than +pixels, and it fires precisely where the pixel-based detectors failed — a cut +subtle enough for the histogram and TransNetV2 to miss is not necessarily subtle +in identity space. It is also self-correcting rather than diagnostic: the +contradiction is detectable **online**, the moment both tracks hold the belief, +not at output time. + +A consequence worth noting: with this in place, overlapping windows for one actor +should be rare rather than routine, because the condition that produces them is +now caught and repaired while tracking. Any that survive to output indicate the +repair failed and should be counted. + +### Unidentified presence — TBD + +A track that is never owned is still *someone* on screen. Emitting it as +anonymous presence would let a consumer show "unidentified person", and would +give the human-in-the-loop association tool ([`../../SPEC.md`](../../SPEC.md) §4) +its work queue directly. Combined with the context crops of AR-022, it may also be +the path to recognising extras and background cast the gallery has no entry for. + +**This is deliberately undecided.** It changes the truth format and invites +consumers to display something that may not be useful. Recorded here so the +option is not lost; not specified until the AR-022 debug output shows whether these +tracks are worth surfacing. + +**Current:** presence is built from *per-frame accepted detections only*. +`identity_matcher_node.hpp:145-231` decides acceptance independently per face per +frame; `result_sink_node.hpp:123-129` collects the timestamps of accepted frames; +`result_sink_node.hpp:139-147` sets `win_start = ts_vec[0]`, the first accepted +timestamp. `track_id` is carried on `IdentifiedActor` but is used only for debug +output — it never gates or backfills a window. A window therefore starts when the +actor was first *recognised*, not when their track began. + +**Gap — this is the main behavioural change in this spec.** + +Design is settled in [`IMPLEMENTATION-PLAN.md`](IMPLEMENTATION-PLAN.md) §1.1: a +`TrackRegistry`, held by `shared_ptr` and used by `FaceTrackerFunc` as its state, +where each track carries `first_seen` plus an **optional `last_seen`** — unset +while on screen, set to the last on-screen timestamp when the face is lost, unset +again when a later detection associates to it. A track whose `last_seen` exceeds +the extinction timeout is reaped and **pushed to the result aggregator** as one +finished claim: *this actor was on screen from a to b*. + +Consequences: + +1. The tracker's `tracks_`/`inactive_` maps become one track set, distinguished + only by whether `last_seen` is set; there is no separate revival path (AR-007). +2. Closing a track *is* the presence assertion — emitted once, complete, never + revised. No later reconciliation stage exists. +3. **`anneal_sec` and `extinction_sec` are deleted, not re-tuned.** Both exist + only to bridge gaps between isolated accepted frames; a track that survives + its own gaps leaves them nothing to do. `SceneTrackerFunc` goes with them. + What remains to tune is the **ownership posterior threshold**, the + **correlated-frame discount** (AR-025), and the **extinction timeout**. +4. Every track must be closed at EOF. A film ends with faces on screen and those + tracks have not timed out, so without an explicit flush the closing scene's + actors are never emitted — a silent presence loss that looks like a + recognition miss. +5. Zero-length windows (`start == end`, a single-frame track) remain possible; + the truth format permits them, and `build_xray()` floors/ceils into integer + seconds. +6. **Pull it out by the roots — including the published field.** `anneal_sec` is + not just a constant: it appears in the truth schema + ([`jRay/SPEC.md`](../../jRay/SPEC.md)), in the Jmanifest format and in the + server's storage columns + ([`JRay-public-server/SPEC.md`](../../JRay-public-server/SPEC.md)). Retaining + it as a vestigial `0` would be worse than removing it: a field that names a + mechanism no longer in the pipeline is actively misleading to anyone reading a + manifest, and it would outlive everyone who remembers why it is zero. It goes + from all three repos under one coordinated `schema_version` bump — the same + bump IR-004 already requires, so there is exactly one breaking change, not two. + + Removal list: `Config::anneal_sec`, `Config::extinction_sec`, + `SceneTrackerFunc` and its node wiring, the sink's annealing pass + (`result_sink_node.hpp:139-147`), the truth-file field, the Jmanifest field, + the server column, and the optimizer's parameter entries. Grep for both names + and expect no survivors. + +## AR-018 … AR-021 — Per-film gallery expansion and deferred identification + +Three requirements that only make sense together, so they are specified together. + +### The problem being solved + +**TMDB headshots are overwhelmingly frontal.** Films are not. An actor is +recognised easily in the shots that resemble a publicity still and missed in +profile, three-quarter, low-angle, poorly lit, or partially occluded views — even +though those are most of their screen time. + +So the pipeline's failures are dominated by **pose**, not by identity: the same +person the gallery knows, at an angle it does not. Both mechanisms below exploit +that. + +### AR-018 — Per-subject embedding store + +Every track maintains a running store of its own embeddings, **identified or +not** — the structure is the same for both. + +An embedding is admitted only if its similarity to one already in the store falls +**inside a band**: + +- **Upper bound** — too similar and it is redundant, teaching nothing the store + already covers. +- **Lower bound** — too dissimilar and it may not be the same person at all; + admitting it risks poisoning the store. + +A starting band of roughly **0.90–0.95** is the working estimate, to be tuned +(VR-007). Note this is deliberately conservative compared to the current +`expand_novelty_sim` (0.55), which promotes embeddings *far* from the gallery — +much more aggressive, and much more exposed to admitting the wrong person. + +Both bounds must be expressed as calibrated probabilities, not raw cosines (AR-024). + +### AR-019 — Expansion of known actors + +When a track is owned (AR-012), its store is promoted into a **per-film, in-memory +annex** for that actor, folded into best-of-N scoring alongside the baked +references. The annex does not persist. + +Promotion requires **certainty that the span is one person**: + +- the track is owned, with the belief stable (no swap, AR-012); +- no camera cut, scene boundary, or identity contradiction occurred within the + span (AR-009, AR-010, AR-015) — all three signals must be quiet, not just the histogram cut as + today; +- the band of AR-018 is satisfied. + +The purpose is precisely to acquire the **non-frontal views TMDB lacks**, at a +confidence the gallery alone cannot supply. + +### AR-020 — Deferred re-identification of unknown tracks + +**This is what the expansion is for.** + +- An unowned track, on extinction, is **not discarded**. Its embedding store, + metadata and context crops (AR-022) move to a **to-be-identified (TBI) queue**. +- At **end of playback**, TBI entries are pooled (AR-021) and re-assessed against + the *final* expanded gallery — which by then holds the pose-varied views + harvested from the whole film. +- Entries that now cross the ownership threshold emit presence windows exactly as + a normally-owned track would. Entries that still do not are the output of AR-022, + and the work queue for human association + ([`../../SPEC.md`](../../SPEC.md) §4). + +The asymmetry this exploits: an actor confirmed frontally early in a film +contributes profile views to the annex, and a profile-shot track that failed at +minute 12 matches once the film is over. Ordering ceases to matter. + +**This is cheap.** Embeddings are already computed; re-assessment is matching +against an in-memory annex — no decode, no detection, no embedding. The cost is +retaining unowned track stores until EOF. + +### AR-021 — Pool unknown tracks before matching + +Re-assessing each unknown track alone wastes the strongest evidence available. +A single track is short and pose-poor; the *same unknown person* usually appears +across many tracks throughout the film. + +**Requirement: cluster the unknown tracks, treat each cluster as one identity, +and match the pooled cluster against the full gallery.** + +Each unknown track's store is effectively a small gallery of one unnamed person. +Clustering merges the ones that are the same person, and the pooled result is a +far richer representation — many poses, lightings and expressions — which stands a +much better chance against the gallery than any constituent track. + +**The evidence is also better-conditioned.** AR-025 warns that consecutive frames +within a track are highly correlated and must be discounted. Embeddings from +*different tracks* are far more independent: different scenes, angles, lighting. +Pooled cross-track evidence is therefore worth more per observation than +within-track evidence, and the discount should reflect that. + +**Temporal exclusion is a free constraint.** Two tracks that **overlap in time +cannot be the same person** — the same fact AR-012 uses to detect missed cuts. This +is a cannot-link constraint on the clustering and it costs nothing to apply, +since track extents are already known. Use it: it prevents exactly the merge that +would otherwise pool two people who share the screen. + +**Clustering must be conservative.** A wrong merge pools two people and then +mis-identifies *both*, across every track in the cluster — strictly worse than +leaving them separate. Prefer many small correct clusters to few large ambiguous +ones; the merge threshold is a calibrated probability (AR-024), swept in VR-007. + +**Order of operations at end of playback:** + +1. Cluster TBI tracks under temporal cannot-link constraints. +2. Pool each cluster's embeddings into one composite identity. +3. Match each cluster against gallery + annex (one GEMM, A10). +4. Cluster crosses threshold → **every member track** emits presence for that + actor. +5. Cluster does not → it becomes one **unknown person** entity, not N orphan + tracks. + +That last point matters beyond recognition: it is the difference between asking a +human to label twelve disconnected faces and asking them to name one person who +appears in twelve places ([`../../SPEC.md`](../../SPEC.md) §4). It is also the +natural unit for anonymous presence, should that be adopted (AR-012, TBD). + +### Consequences + +- **The result aggregator cannot finalise at EOF-flush.** Owned tracks emit on + death as before, but the TBI pass runs after, and may add windows. Output is + written after re-identification completes, not when the last frame is read. +- **Expansion errors are now more costly.** A wrongly promoted embedding no + longer affects only later frames — it is applied to every unknown in the film + during the second pass. The AR-018 band is the guard, and its lower bound is the + part doing that work. +- **Iteration is possible but unspecified.** A TBI entry that becomes identified + could itself expand the gallery and enable further identifications — a fixpoint + loop. Whether to iterate to convergence, run one pass, or bound the rounds is + open (VR-007). + +**Current:** `src/gallery/track_gallery.hpp` implements a per-track diversity +buffer with eviction biased to gallery-far poses, promotion gated on +`expand_novelty_sim` / `expand_track_spread_max`, cleared on `is_cut`. Wired at +`identity_matcher_node.hpp:227`, cleared at `:126`. + +**Gap:** the band rule of AR-018 replacing the current novelty/spread gates; all +three quiet-signal conditions rather than only `is_cut`; probability space +throughout (AR-024); and the whole of AR-020 — the TBI queue, the deferred pass, and +deferring output until it completes. + +## AR-022 — Unidentified-track capture + +Persist everything needed to diagnose a miss — and everything a *human* would +need to resolve one. + +Scope note: the TBI queue of AR-020 is **not** debug-only, because deferred +re-identification depends on it. What is behind a flag is the *persisted output* +for tracks that survive re-assessment still unidentified; retaining stores until +the deferred pass runs is unconditional. + +For every track **still unidentified after the deferred pass (AR-020)**, store: + +- all embeddings on the track, +- the aligned 112×112 crops, +- **context crops** — a wider region around the face than the 112×112 aligned + crop, for a handful of representative frames per track; +- track metadata: `track_id`, first/last timestamp, frame count, per-frame + detector confidence and bbox, +- the best similarity achieved and which actor it was against, so near-misses are + distinguishable from faces with no gallery counterpart at all. + +**Why the context crop is a separate artifact.** The 112×112 crop is optimised +for ArcFace: tightly cropped, geometrically normalised, and frequently +unrecognisable to a person out of context. A human deciding *who this is* needs +the surrounding shot — hair, costume, who they are standing next to. This +requirement exists to serve the human-in-the-loop association capability +([`../../SPEC.md`](../../SPEC.md) §4), where the user names the face the pipeline +could not. + +Retain a bounded number of representative frames per track (largest/sharpest +detections), not every frame — an unidentified track can run for minutes. + +This is the raw material for deciding whether a miss is a gallery coverage +problem (actor absent or under-represented) or a threshold problem (actor present +but scored below acceptance). + +**Current:** only *promoted* mugshots are dumped, via `expand_debug_dir` +(`track_gallery.hpp`, guarded by `SAE_DEBUG`) — i.e. the successes, not the +failures. **Gap:** the whole requirement. Needs a flag (`--dump-unidentified +`), a per-track accumulator that survives until the track ends, and a +decision on retention: crops for a feature-length film are large, so default to +embeddings + metadata with crops opt-in. + +## AR-023 … AR-025 — Calibration, probability space, and per-track Bayesian accumulation + +**Every similarity in the pipeline is converted to a probability through the +sigmoid calibration before it is used or thresholded. No component compares raw +cosine values against a hand-set constant.** + +This is a system-wide rule, not a detail of identity matching. Raw cosine +thresholds are unfalsifiable magic numbers that mean different things for +different models, different galleries, and different face sizes; a calibrated +probability means the same thing everywhere. It replaces `track_max_embed_dist` +(0.7), `cut_revive_sim` (0.50), `expand_novelty_sim` (0.55) and +`expand_track_spread_max` (0.60) with quantities that can be reasoned about +jointly. + +### Per-frame evidence is a Bayesian update on the track + +Once similarities are probabilities, a track's identity is not a vote count — it +is a **posterior accumulated across the frames of that track**. Each frame +contributes a likelihood that this face is actor A; the track's belief is updated +frame by frame, and ownership (AR-012) is "posterior exceeds threshold" rather than +"≥ N accepted frames". + +Working in **log-odds** makes this an addition per frame: + +``` +logit(A | frames 1..n) = logit_prior(A) + Σ_i [ logit(A | sim_i) − logit_prior(A) ] +``` + +This is strictly better than counting accepted frames: a long run of marginal +observations and a single decisive one are no longer conflated, and a track that +accumulates weak contrary evidence can lose ownership rather than keeping it on a +stale count. + +**The independence problem, which must be handled explicitly.** Consecutive +frames of one track are highly correlated — near-identical pose, lighting and +expression. Treating them as independent observations overcounts evidence +dramatically: thirty frames of the same face at the same angle is nowhere near +thirty independent measurements, and naive accumulation will drive the posterior +to certainty on what is effectively one observation. + +Mitigations, in preference order: + +1. **Update only on sufficiently novel observations.** The diversity buffer + (AR-018…AR-021) already identifies which embeddings on a track are gallery-far and + mutually distinct — precisely the more-independent ones. Reuse that judgement + rather than inventing a second one. +2. **Discount correlated updates** by a per-frame weight below 1, fitted so the + accumulated posterior is calibrated against held-out tracks. +3. **Cap total evidence per track**, the crude fallback. + +Whichever is chosen, the accumulated posterior must be **validated against +ground truth** — a posterior of 0.99 should be wrong about 1% of the time. If it +is not, the independence handling is inadequate and the number is decoration. + +### Fitting the calibration + +Build a PDF of **intra-subject** similarity and a PDF of **inter-subject** +similarity across the gallery; the prior is `intra / (intra + inter)`. + +- **Positive pairs** — same actor, different reference images. +- **Negative pairs** — all cross-actor pairs. +- Near-duplicate references are de-duplicated per actor first (similarity above + `1 − 1e-7`), so the same image embedded twice cannot inflate the positive side. +- Actors with fewer than 5 distinct embeddings contribute negatives only — they + cannot supply a meaningful positive pair. +- A Platt-style sigmoid `P(match) = σ(a·sim + b)` is fitted to the two + distributions with class weights balancing the (heavily skewed) pos/neg ratio. +- The base rate enters as log-prior-odds: `P = σ(a·sim + b + log(p₀/(1−p₀)))`, + with `p₀ = match_prior`. +- Acceptance is `P(match | sim, prior) > prob_threshold` (default 0.754, DE-tuned). +- **Fallback** when calibration is invalid (too few positive pairs): a hard cosine + distance ceiling `match_threshold`, OR a ratio test — accept if + `best/second < match_ratio` and `best_distance < match_ratio_ceil`. + +**Current:** `src/gallery/gallery_calibration.hpp` implements the sigmoid fit, +dedup, eligibility filter, and prior adjustment; the fallback lives at +`identity_matcher_node.hpp:181-196`. **Gap:** the fit currently histograms the two +similarity distributions internally (`kHistBins = 200`) but does not *emit* them. +For this requirement to be inspectable, the intra/inter PDFs and the derived +prior should be written alongside the gallery, so calibration quality is +auditable rather than implicit. Also note the shipped `match_prior` default is +0.5 (use the calibrated sigmoid directly) rather than the gallery-derived +`intra/(intra+inter)` — reconcile: either compute and store it at gallery-build +time, or document 0.5 as a deliberate override. + +## AR-026, AR-027 — Scale — performance must hold for arbitrary gallery size + +**Requirement: analysis throughput must remain acceptable as the gallery grows +arbitrarily large.** Gallery size is set by the user's library, not by us: a +whole-library gallery spans every credited actor across every title +(GR-001), which is thousands today and grows monotonically as titles are added. A +design that is fast at 500 actors and unusable at 50,000 has a defect, not a +limitation. + +### Similarity is a matrix multiply + +Every embedding is unit-norm (AR-006), so cosine similarity is a dot product and +scoring a batch of faces against the gallery is one GEMM. At library scale that +is the only viable formulation — a per-pair loop is orders of magnitude off. + +**All similarity computation goes through the GEMM path**, with no exception +justified by "this set is small". Three call sites: + +1. **Baked gallery** — already GEMM (`sim_engine_->compute()`, + `identity_matcher_node.hpp:143`, backend from `SAE_GEMM_BACKEND`). ✓ +2. **Per-film annex** — currently a **CPU loop** + (`identity_matcher_node.hpp:159-162`), justified in-comment by "tens of + embeddings". AR-018…AR-021 invalidates that assumption: every owned track now + contributes, so the annex grows with cast size and film length. It must move + into the GEMM path — appended to the gallery matrix, or a second multiply. +3. **Deferred TBI pass (AR-020)** — the most GEMM-friendly operation in the + pipeline: all TBI embeddings against the full gallery-plus-annex, offline, + operands resident, no streaming. One large multiply, not a loop over entries. + +This constrains AR-018…AR-021's implementation: the annex must be a **contiguous matrix** +with promotions appended, plus a parallel actor-index mapping — exactly the +`flat_emb_`/`flat_actor_` arrangement the baked gallery already uses. + +### Scaling characteristics that must be known, not assumed + +- **Throughput versus gallery size must be measured** (VR-008) and published. The + useful output is the curve and the point where gallery scoring starts to + dominate total runtime, not a single number. +- **Memory is a real ceiling.** 512 floats × 4 bytes = 2 KB per reference. At + five references per actor that is ~10 MB per 1000 actors — comfortable at + 10,000 actors, worth planning for beyond. +- **Calibration cost grows quadratically.** AR-023 fits on cross-actor pairs, which + is O(N²) in references. This is a gallery-build cost, not a per-title one, but + it will bite first — sampling negatives rather than enumerating them is the + obvious mitigation and should be specified before it becomes urgent. + +### If GEMM stops being enough + +Approximate nearest-neighbour indexing (IVF/HNSW) is the standard next step, and +it trades **exactness** for speed. That trade interacts badly with AR-023: an +approximate search returns approximate similarities, and a calibrated posterior +built on them is no longer calibrated. Not ruled out, but it requires +re-validating calibration against the approximation — so it is a later decision +with a real cost, not a drop-in. + +**Gap:** annex GEMM path; the scaling benchmark (VR-008); negative-pair sampling in +calibration. + +--- + +# Part B — Deployment requirements + +The pipeline must support **multiple deployment modes over one core**. Modes +differ in *what triggers work* and *what constrains it*, not in what the analysis +does. + +## DP-001 — Common core + +- One analysis implementation, exercised identically by every mode. Modes are + front-ends; none may fork the pipeline logic. +- Backend selection (ORT/TRT, CUDA/ROCm/CPU) is a **build-time** choice + (`SAE_INFERENCE_BACKEND`, `SAE_GEMM_BACKEND`). Prebuilt TRT engines may be + supplied at runtime via `detector_engine` / `arcface_engine`, bypassing ORT. +- Models load once per process. Any mode processing more than one title must + amortise model and gallery load across titles. +- No mid-video checkpointing. A run either completes and emits a result, or emits + nothing (see DP-004). + +## DP-002 — Mode: batch CLI + +One-shot invocation over one title. The reference mode and the substrate for the +others. + +- `scene_analyze -o out.json`, exit non-zero on failure. +- Must be safe to invoke concurrently by an external scheduler, subject to GPU + memory. + +**Current:** `src/main.cpp`. **Gap:** none. + +## DP-003 — Mode: on-demand service + +A resident process on a server, analysing on request. + +- Models and gallery stay resident; requests carry a media path plus optional + overrides. +- Requests are queued with a bounded depth and processed serially per GPU; + the queue must be observable (depth, in-flight title, ETA). +- Health endpoint reporting model, gallery fingerprint, backend, and GPU state. +- Graceful drain on shutdown: stop accepting, finish or abandon in flight per DP-004. + +**Gap:** not built. `scripts/run_from_jellyfin.py --worker` is a polling loop, not +a request-driven service; it is the closest existing shape. + +## DP-004 — Mode: opportunistic / idle-triggered + +Analyse when the machine is otherwise unused; yield the instant it is not. + +- **Trigger is external and non-oscillating.** Screen-lock (logind + `Lock`/`Unlock`) is the reference signal. GPU/CPU load must *not* be used: the + worker is itself the load, so a load threshold forms a feedback loop. +- **Stop is a hard stop.** On the resume signal, SIGTERM the worker mid-analysis. +- **Re-queue is free and implicit.** An item leaves the pending queue only when + its result is pushed, so a killed run simply stays pending. This requires: + 1. never push a partial result — push only after the analysis returns cleanly; + 2. clean up temp files on signal — write temps under a directory wiped on + start, and unlink on SIGTERM (a SIGKILL skips `finally`). +- Accepted trade-off: a partially analysed title restarts from scratch. Fine for + an overnight workload. +- Other triggers (idle timer, AC power, scheduled window) must fit the same + contract: external signal, hard stop, implicit re-queue. + +**Current:** designed in detail in [`service-conversion.md`](service-conversion.md) +as `sae-worker.service` + `sae-lock-gate.service` under systemd **user** units. +**Gap:** unbuilt; the temp-cleanup fix in `run_from_jellyfin.py` is a named +prerequisite. + +## DP-005 — Installation and provisioning + +- Native install, **no Docker** — GPU passthrough is the most fragile part of a + containerised setup and exists only because of the container. Natively the GPU + works with the host drivers and media paths need no re-mounting. +- An installer (`scripts/build_install.py`) consuming one `install.yaml`: + platform (nvidia/amd/cpu), embedder model, gallery scan cadence, install + prefix; runtime secrets written to a `.env`, editable without recompiling. +- Distro coverage: Fedora + Arch (`dnf`/`pacman`), auto-installing dependencies + after printing them. Debian/Ubuntu out of scope. +- Model acquisition (`scripts/download_models.sh`) and TRT engine build + (`scripts/build_trt_engines.sh`) are provisioning steps, not runtime steps. + +**Gap:** installer unbuilt. + +## DP-006 — Gallery maintenance as a background concern + +- Incremental gallery refresh runs on a timer (`gallery_scan_interval`, default + 24 h) independently of analysis, so newly added titles' cast is embedded before + their media is analysed. +- A gallery/model mismatch must be detected **at startup**, not silently produce + garbage similarities. See GR-004. + +--- + +# Part C — Integration requirements + +## IR-001 … IR-003 — Truth-file output + +Emit the JRay truth format, `schema_version: 1`, `Verbosity::minimal`. + +- Sibling file `Movie.jray.json` next to the media (suffix configurable + plugin-side). +- Per actor: `name`, `imdb_id`, `tmdb_id`, `jellyfin_id` (each `""` if + unresolved), and `scenes` windows. +- **Each window carries its belief** and its identification route (AR-012) — the + posterior is computed for every claim anyway, so it is serialised rather than + discarded. This lets a consumer caveat or filter low-confidence presence + instead of treating every window as equally certain. +- `jellyfin_item_id` is stamped in *after* analysis by `run_from_jellyfin.py` — + `scene_analyze` does not know it. +- Additional verbosities: `standard` (per-frame bboxes, similarity, unknowns) and + `xray` (Jellyfin-Xray `{"second": ["Actor", …]}`, dense integer seconds). + +### `extraction.*` provenance + +Consumers — and the public server, which ranks competing manifests — need to know +what produced a result: + +- `sample_fps`, `pipeline_version`, `gallery_size` — as today. +- **`extinction_sec`** — replaces `anneal_sec`, which is dropped entirely (AR-012 withdrawal note). + It is the parameter that shapes window extent, so it is what a consumer needs + to interpret them. +- **`gallery_scope`** — `"global"` or `"limited"`. **The single most useful + quality signal**: two galleries of identical size differ enormously depending + on whether matching ran against the whole library or only the title's credited + cast. A limited gallery cannot find an uncredited or mis-credited appearance at + all; a global one competes against every actor in the library. Default is + **global**. +- Optionally, the `tmdb_id`s of the actors the gallery was built from — useful + for reproducibility locally. Not proposed for the Jmanifest, where a + thousand-entry id list is bulk for little gain over `gallery_size` + scope. + +Format is owned by [`../../jRay/SPEC.md`](../../jRay/SPEC.md); this pipeline is +the producer. Any change is a coordinated schema-version bump. + +**Current:** `result_sink_node.hpp`. **Gap:** the schema changes in several ways +at once — `anneal_sec` out, `extinction_sec` and `gallery_scope` in, per-window +belief added, audio signature added (IR-004). All breaking, so they ship as **one** +`schema_version` bump coordinated across all three repos. The `scenes` values also +change under AR-012. + +**Output timing:** the file is written after the deferred pass (AR-020/AR-021) +completes, not at EOF — deferred and pooled identifications add windows after the +last frame is read. + +## IR-004, IR-005, IR-007, IR-008 — Audio signature + +Emit the content-derived audio signature in the truth file, so a truth file is +self-identifying without a plugin round-trip. + +Construction is specified in +[`../../JRay-public-server/SPEC.md` §3](../../JRay-public-server/SPEC.md) and must +be implemented **exactly** — a signature that differs in any parameter will not +match one computed by the plugin: + +1. Decode a 120 s window centred on the midpoint (`runtime/2 ± 60 s`) — avoids + logos/cold opens at the head and credits at the tail. +2. Downmix to mono, resample to 11025 Hz. +3. STFT: 4096-sample frame, 1024-sample hop (~93 ms, ~1290 frames), Hann window. +4. Log-magnitude spectrum over 300–3000 Hz. +5. 32 logarithmically spaced bins; record peak-bin index + 2-bit energy class. +6. One byte per frame → ~1290-byte array, base64-encoded. + +Peak-bin rather than full spectrum: peaks survive lossy re-encoding, loudness +normalisation and channel-layout changes; absolute magnitudes do not. + +Matching (sliding ±600 frames ≈ ±56 s, scoring the fraction of overlapping frames +whose peak bin matches) is a **consumer** concern — this pipeline produces the +signature, it does not match. Offsets are applied client-side; manifests are never +rewritten. + +**Media shorter than 120 s.** The window `runtime/2 ± 60 s` underflows, so no +signature is emitted and **no sync offset is applied**. Such items fall back to +the runtime/exact tiers, which is adequate: a 90-second extra or trailer is not +the content whose cut alignment matters. Both producers must apply the identical +rule, or they diverge on exactly the short items most likely to be +mis-identified. + +**Signature versioning.** The signature carries its own `v1:` prefix, separate +from `schema_version` (server spec §3 example: `"v1:v7fA3k…"`). Emit and honour +it, so a future change to the DSP chain is *detectable* rather than silently +producing non-matching signatures. + +**Decision (this spec):** the pipeline computes and emits it *in addition to* the +plugin. Consequences to carry through: + +- The truth schema gains a field → **`schema_version` bump**, coordinated with + `jRay/SPEC.md` and the plugin. +- The pipeline needs an audio decode path. It already links FFmpeg + (`ffmpeg_decoder.hpp`) for video, so this is a second stream from an existing + dependency, not a new one. +- Both producers must agree bit-for-bit. A cross-check test — plugin signature vs. + pipeline signature over the same file — is a hard requirement, not a nicety. +- Files never processed by this pipeline still get a signature from the plugin; + the two paths coexist deliberately. + +**Gap:** entire requirement — no audio path exists in the pipeline today. + +## IR-006 — Jellyfin round-trip + +- Pull the work queue: `GET /Plugins/JRay/Tasks/Pending?limit=N` — items with no + results yet. +- Push results: `PUT /Plugins/JRay/Items/{itemId}/Truth` (admin API key). Managed + truth takes precedence over a sidecar file for the same item. +- Push only complete results (DP-004). + +**Current:** `scripts/run_from_jellyfin.py`. **Gap:** none. + +--- + +# Part D — Gallery construction requirements + +## GR-001, GR-002, GR-005 — Sources + +Build a gallery of actor reference embeddings from Jellyfin and TMDB. + +- **Jellyfin-wide** (`make_jellyfin_gallery.py`): enumerate every Movie/Series, + collect the unique cast across the whole library, download each actor's + headshot from Jellyfin directly (no TMDB key required), embed, write one global + gallery. +- **TMDB fallback**: for actors with no usable Jellyfin image, fall back to TMDB + profile images (`--tmdb-key`). +- **Incremental merge** (`--merge`): re-runs pick up newly added titles without + re-embedding actors already present. This is what makes DP-006 cheap enough to run + daily. +- **Cast restriction** (`filter_gallery.py`): derive a per-title gallery limited + to credited cast. Faster and fewer look-alike mismatches, but note + [`gallery-scope.md`](gallery-scope.md) — the rep4 matrix found the *full* + gallery won for the shipped model, so restriction is a tool, not the default. + +Each actor carries `name`, `imdb_id`, `tmdb_id`, `jellyfin_id` (whichever +resolve), one embedding per reference image, and the source image paths. + +**Current:** `make_jellyfin_gallery.py`, `make_gallery.py`, `filter_gallery.py`, +`sae_jellyfin.py`, `sae_tmdb.py`, `src/gallery/gallery_builder.*`. **Gap:** none. + +## GR-003 — Quality and coverage reporting + +Gallery build must report, not just produce: + +- actors with zero usable images (they can never be recognised — a silent recall + ceiling); +- actors below the 5-embedding threshold for positive pairs (AR-023), which + degrades calibration; +- the fitted calibration and the intra/inter distributions behind it (AR-023 gap); +- duplicate/near-duplicate references removed. + +**Gap:** partial. Dedup and eligibility are computed inside calibration but not +surfaced as a build report. + +## GR-004 — Model binding + +- A gallery is only valid for the embedder that built it. The embedder identity + must be **stamped into the gallery file**, and checked at startup by any + consumer. +- Mismatch is a hard startup error. Cosine similarities between embeddings from + different models are meaningless but *look* plausible — this fails silently and + expensively otherwise. + +**Gap:** named as step 4 of the `service-conversion.md` implementation plan; +unbuilt. This is the highest-value small fix in the document. + +## GR-006 … GR-009 — Provenance tiers and poisoning guard + +Reference embeddings now come from three sources with different trust, and they +must be **distinguishable in the gallery**, not merged into an undifferentiated +pile: + +| Tier | Source | Persists | Trust | +|---|---|---|---| +| **Baked** | Jellyfin / TMDB headshots | Yes | High — curated, externally sourced | +| **Harvested** | Per-film annex (AR-019), promoted from owned tracks | **Yes, flagged** | Unverified — machine-derived | +| **Confirmed** | Human association (`../../SPEC.md` §4) | Yes | Highest — a person said so | + +**Harvested embeddings are retained rather than discarded at exit**, because they +are exactly the non-frontal views the baked gallery lacks and their value +compounds across a library. But they carry the risk the ephemeral annex avoided: +**a promotion error becomes permanent instead of dying with the process.** + +They are therefore **flagged as harvested and reviewable**, never silently equal +to a baked reference. The tier must be recorded per embedding so that a suspected +poisoning can be traced, audited, and reverted without rebuilding the gallery. + +### Bell-curve outlier detection + +An actor's own embeddings should form a **roughly normal distribution in cosine +space** around their centroid. A harvested embedding that falls outside that +distribution is unlikely to be the same person — which is precisely what a +poisoned entry looks like. + +**Requirement: flag harvested embeddings that are distributional outliers among +that actor's references**, for review or automatic exclusion. + +> **`EXCEPTION: AR-024` — raw cosine is used here deliberately.** +> +> AR-024 requires calibrated probabilities everywhere, and this is an agreed +> exception. The reason: the calibration is a monotonic squash mapping similarity +> onto `P(same person)`. That is exactly right for making a *decision*, and wrong +> for characterising a *distribution* — the sigmoid compresses the tails, which +> is where outliers live, and would flatten the very structure being tested. +> Distribution shape and outlier distance are properties of the metric space, so +> they are measured in it. +> +> Scope of the exception: distributional analysis of an actor's own reference set +> only. Any match, association, or admission decision still goes through the +> calibration. + +**Gap:** entire requirement — tiering, persistence of harvested embeddings, the +flag, and the outlier check. + +**Open question:** whether human-confirmed associations should be a distinct +audited tier (individually revocable if someone mislabels) or simply more +embeddings for that `tmdb_id`. Deferred. + +--- + +# Part E — Parameter-study requirements + +The tuned constants in `config.hpp` are empirical. Retuning must stay cheap, or +it will not happen — and AR-012 makes a retune mandatory. + +## VR-001 — Post-inference dump + +Persist pipeline state at the point where the expensive work ends. + +- Dump at the `EmbeddedSceneFrame` channel — after decode → detect → align → + embed, **before** tracking and identity matching. Everything downstream is + cheap CPU maths, so a replay re-runs the whole tail with no GPU and no video. +- HDF5, one file per title, flat/ragged: per-face arrays concatenated, with a + per-frame index table (`face_offset`, `face_count`) pointing into them. Avoids + variable-length HDF5 types and reads straight into numpy. +- Stores per frame: `timestamp_sec`, `frame_idx`, `is_cut`, `is_scene_boundary`. + Per face: `embedding` [N,512], `bbox` [N,4], `landmarks` [N,10], `confidence`. +- Invariants: embeddings unit-norm; `face_offset` contiguous; bboxes already in + original resolution; frames with no faces still get a row so timestamps stay + dense; EOF sentinels not written. +- Enabled by `--dump-embeddings out.h5`; teeing must not perturb the live result. + +Schema owned by [`scripts/optimizer/SCHEMA.md`](../scripts/optimizer/SCHEMA.md). + +**Current:** C++ dump sink (`embedding_dump_node.hpp`, `dump_embeddings.cpp`), +read by `replay.py`. **Gap:** **AR-012 breaks the replay contract.** Track extents +are decided in the tracker, which is *downstream* of the dump — so a replay can +reproduce them, but only if the dump preserves everything the tracker needs. +Verify `landmarks`/`bbox`/`is_cut` suffice, and bump `schema_version` if not. + +## VR-002 — Replay and sweep + +- Replay drives the **real KPN nodes** over dumped embeddings, not a + reimplementation — a sweep that optimises a divergent copy is worthless. +- The gallery loads once per process and is cached by path, so one evaluation is + N cheap replays. +- Differential Evolution over the continuous knob space + (`prob_threshold`, `expand_min_anchor_frames`, the re-acquisition timeout, …), + scored against reference presence data. Note these are *not* independent — a + longer timeout yields longer tracks, hence more frames to clear the anchor + count — so they must be swept jointly. + +**Current:** `scripts/optimizer/optimize.py`, `replay.py`, `second_score.py`. +**Gap:** none, pending E1. + +## VR-003 — Scoring methodology + +- Micro-averaged per-second presence against Amazon X-Ray, per + [`methodology.md`](methodology.md). +- **Objective is F1, but precision and recall are logged at every evaluation and + printed at the optimum.** X-Ray recall is a face-vs-cast-in-scene ceiling, so + unconstrained F1 pushes `prob_threshold` *down* chasing unreachable recall, + trading real precision away. The trade-off must stay visible so another + operating point can be chosen from the trajectory (`--trajectory`). +- Known metric hazard: the earlier 9-film scene-union metric hid out-of-cast + false positives; the 4-film rep4 per-second metric supersedes it. Any new + metric must be checked for the same class of blindness. + +**Current:** implemented; documented in `methodology.md`, +`rep4-optimizer-results.md`, `optimizer-experiments.md`. **Gap:** none. + +## VR-004 — Validation corpus + +- A manifest-driven film set with ground truth (`scripts/validation/`), scored + reproducibly. Benchmarking practice is already established — see + [`methodology.md`](methodology.md), [`model-bakeoff.md`](model-bakeoff.md), + [`best-model.md`](best-model.md) and `rep4-optimizer-results.md`; AR-012 changes + what is measured, not how. +- After AR-012, `prob_threshold` (0.754) must be re-derived — it was fitted against + per-frame presence semantics and now governs *voting*, not presence. + `anneal_sec` (35.5) and `extinction_sec` (57.4) are not re-derived; they are + deleted (AR-012 withdrawal note). + +## VR-005 — Minimum face size study + +Quantify where ArcFace degrades, replacing the 66×66 estimate in A1 with a +measurement. + +**Method.** + +1. Select ~100 gallery actors having more than one mugshot. +2. Per actor, hold out **one** image as the probe; its remaining images stay in + the gallery at native resolution. +3. For each target size *S*, downscale the probe to *S*×*S* and upscale back to + 112×112, then embed. +4. Match each degraded probe against the full 100-actor gallery and record + **TPI/FPI** — identified as the correct actor, or as someone else. +5. Repeat across sizes to get the curve. + +The asymmetry is the point: **the gallery stays high-res and only the probe +degrades**, which is exactly the production case — reference mugshots are clean, +the face from the video is small. It also measures the decision the pipeline +actually makes (probe against gallery) rather than embedding drift, which can be +large without harming separability and small in a direction that destroys it. + +**Caveat on gallery size.** FPI grows with the number of actors competing, so a +100-actor gallery understates the false-positive rate against a full library of +thousands. Treat the FPI numbers as *relative* across sizes rather than as an +absolute rate, or re-run at production scale before setting a threshold from +them. + +**Cheap to run** — no video needed, gallery images are already on disk, and the +embedding/matching machinery exists (`scripts/sae_embed_loader.py`, +`gallery_calibration.hpp`). Expect a knee rather than a cliff; the output is a +size-versus-TPI/FPI curve plus a chosen operating point, not a single number. + +**Secondary output:** the same curve shows whether `min_face_px` should be a +constant at all or should scale per embedder — relevant since the model is a +build-time choice (GR-004). + +## VR-007 — Expansion band and deferred-pass study + +Tune the AR-018 admission band and establish whether AR-020 pays. + +**Band.** Sweep the lower and upper bounds around the 0.90–0.95 working estimate. +The two bounds fail in opposite directions and must be read separately: too low a +lower bound admits the wrong person (precision collapse, amplified by the +deferred pass); too low an upper bound admits only redundant views (no recall +gain, wasted annex). Report both, not a single F1. + +**Deferred pass.** Measure recall recovered by AR-020 — how many TBI entries are +identified on re-assessment — and precision of those late identifications +specifically. They should be scrutinised separately from first-pass +identifications, because they are the ones relying on harvested rather than baked +references. + +**Clustering (AR-021).** Sweep the merge threshold and report cluster purity +against known-cast ground truth, not just downstream F1. The two error modes are +asymmetric and must be reported separately: an over-merge mis-identifies every +track in the cluster at once, while an under-merge only forfeits the pooling +benefit. Verify the temporal cannot-link constraint is actually binding — measure +how many candidate merges it rejects, since if the answer is zero the constraint +is not doing the work claimed for it. + +**Iteration.** Test one pass versus iterating to convergence (AR-020). Report +whether round 2+ recovers enough to justify the complexity, and whether precision +degrades with each round — the failure mode being a wrong identification in +round 1 seeding references that corrupt round 2. + +**Ablation worth having:** expansion on with deferred pass off. It separates +"expansion helps live matching" from "expansion helps the second pass", which the +current all-or-nothing `expand_gallery` flag cannot distinguish. + +## VR-008 — Gallery scaling benchmark + +Establish the throughput-versus-gallery-size curve required by A10. + +**Method.** Synthesise galleries at 10², 10³, 10⁴, 10⁵ actors (random unit-norm +embeddings suffice — this measures compute, not accuracy) and record per-frame +matching time, end-to-end throughput, and GPU memory. + +**The number that matters** is where gallery scoring stops being negligible and +starts dominating runtime relative to decode, detection and embedding. Below that +point gallery growth is free; above it, it sets the pace. + +**Also measure the deferred pass (AR-020) separately.** It has a different shape — +one large offline multiply rather than many small streaming ones — so it may +scale quite differently and could well become the dominant cost on a film with +many unknowns. + +**Report calibration build time too.** AR-023's cross-actor pair enumeration is +O(N²); this benchmark is where that becomes visible, and it will likely be the +first thing to break at scale. + +## VR-006 — Re-tune `scene_threshold` at native rate + +Not a study of whether to feed TransNetV2 correctly — AR-011 settles that it must be. +This is the consequence: `scene_threshold` (0.60) was picked against 12 fps input, +where the model's separation was compressed. At native rate the boundary/ +non-boundary margin should widen, so the operating point moves. + +Small and mechanical: score boundaries across a threshold range on a few titles +once native-rate decode lands, and pick the new point. Expect a cleaner +separation than the ~0.50 baseline / ~0.7+ peak recorded at 12 fps; if it does +*not* improve, that is evidence worth having about the export itself. + +--- + +# Open questions + +Resolved during planning, recorded here so the reasoning is not lost: + +- ~~**AR-012 vs. extinction.**~~ Wholly subsumed. The `last_seen` model ends windows + at the last sighting, which removes the over-claim `extinction_sec` caused; + both it and `anneal_sec` are deleted rather than re-fitted (AR-012 withdrawal note). +- ~~**AR-022 retention.**~~ Embeddings + metadata by default, crops opt-in behind + `--dump-unidentified-crops`. +- ~~**IR-004 bit-exactness.**~~ Golden-vector fixture checked into both repos, not a + shared implementation — the coupling cost of the latter exceeds the benefit. +- ~~**AR-023 prior.**~~ Decide once GR-003 persists the intra/inter distributions, so the + real value is known rather than argued about. + +Still open (pipeline-local): + +1. **Re-acquisition timeout scope.** `track_max_frames_missing` (5) and + `cut_inactive_max_frames` (5) currently distinguish an ordinary miss from a + cross-cut park. Under AR-012 both become the same thing. Do they collapse to one + constant, or does a cut still warrant a different window? Cheap to test both + in the Phase 2 sweep. +2. **Context-crop budget.** How many representative frames per unidentified + track, and chosen how (largest? sharpest? most frontal?). Bounded by review-UI + usefulness rather than by diagnostics. + +Escalated to the system spec ([`../../SPEC.md`](../../SPEC.md) §5), since they +bind more than one repo: `schema_version` coordination for the pending bump, and +whether unidentified presence should be published in the truth format. diff --git a/docs/requirements.md b/docs/requirements.md new file mode 100644 index 0000000..6d2f909 --- /dev/null +++ b/docs/requirements.md @@ -0,0 +1,268 @@ +# scene-actor-extraction — requirements register + +Stable IDs for every requirement in [`SPEC.md`](SPEC.md), which holds the prose. +This file is the **authoritative list**; the CI gate reads its denominators from +here (see [`../../SPEC.md`](../../SPEC.md) §6). + +**IDs are permanent.** A withdrawn requirement is marked `Withdrawn` and its +number is never reused — renumbering is what produces orphan TRACES tags. This +register replaces the earlier thematic `A1…E8` scheme, which had already produced +an `A1a` and an out-of-order `E6`. + +Tag code with `// TRACES: AR-012 | SR-002`. + +| Type | Scope | +|---|---| +| `AR` | Algorithm — the extraction pipeline itself | +| `DP` | Deployment — how it runs | +| `IR` | Integration — contracts with other components | +| `GR` | Gallery — building and maintaining actor references | +| `VR` | Validation — parameter studies and benchmarks | +| `UT` / `IT` | Unit / integration tests | + +Status: `Done` · `In Progress` · `Planned` · `TBD` · `Withdrawn` + +--- + +## Algorithm (AR) + +| ID | Requirement | Traces to | Priority | Status | +|---|---|---|---|---| +| AR-001 | Detect faces in sampled frames; emit bbox, confidence, 5-point landmarks in original pixel space | SR-002 | High | Done | +| AR-002 | Minimum face size 66×66 px, expressed in **original** resolution (decoupled from `dense_scale`) | SR-002 | High | Planned | +| AR-003 | No fixed per-frame face cap — crowd scenes must not lose background cast | SR-002 | Medium | Planned | +| AR-004 | Backpressure: unbounded faces/frame absorbed by slowing, never by dropping or throwing | SR-002 | High | Planned | +| AR-005 | Align to 112×112 via ArcFace 5-point similarity transform | SR-002 | High | Done | +| AR-006 | 512-d L2-normalised embeddings, batched | SR-002 | High | Done | +| AR-007 | Associate detections by IoU + embedding, with **frame-dependent** weighting | SR-002 | High | In Progress | +| AR-008 | One track pool keyed on `last_seen`; no separate revival path | SR-002 | High | Planned | +| AR-009 | Camera-cut detection (histogram) as an association hint | SR-002 | High | Done | +| AR-010 | Scene-boundary detection (TransNetV2) as an association hint | SR-002 | Medium | In Progress | +| AR-011 | **Every model is fed the input it was trained for** — cost reduced by running less often, never by degrading one inference | SR-002 | High | Planned | +| AR-012 | Presence follows **track extent**, not per-frame recognition | **SR-002** | High | Planned | +| AR-013 | `last_seen` optional state machine; window ends at last sighting, never after | SR-002 | High | Planned | +| AR-014 | Belief swap A→B terminates the track and starts a new one | SR-002 | Medium | Planned | +| AR-015 | Two live tracks owned by one actor ⇒ treat as a detected cut, re-associate | SR-002 | Medium | Planned | +| AR-016 | All tracks closed at EOF — a film ends with faces on screen | SR-002 | High | Planned | +| AR-017 | Every presence claim carries its belief and identification route | SR-002 | High | Planned | +| AR-018 | Per-subject embedding store with banded admission (novel enough, safe enough) | SR-005 | Medium | Planned | +| AR-019 | Per-film gallery annex from owned tracks; acquires the non-frontal views TMDB lacks | SR-005 | Medium | In Progress | +| AR-020 | Deferred re-identification of unknown tracks against the final expanded gallery | SR-005 | High | Planned | +| AR-021 | Cluster unknown tracks into one entity per person, under temporal cannot-link constraints | SR-005 | Medium | Planned | +| AR-022 | Capture still-unidentified tracks: embeddings, metadata, **context crops** for human review | §4 | Medium | Planned | +| AR-023 | Fit sigmoid calibration from intra/inter similarity distributions | SR-002 | High | Done | +| AR-024 | **Always the calibrated probability, never a raw cosine** — exceptions recorded | SR-002 | High | Planned | +| AR-025 | Per-track Bayesian accumulation in log-odds, with correlated-observation discounting | SR-002 | High | Planned | +| AR-026 | All similarity computed as GEMM, including annex and deferred pass | SR-001 | High | In Progress | +| AR-027 | Throughput acceptable for **arbitrary** gallery size | SR-001 | High | Planned | + +## Deployment (DP) + +| ID | Requirement | Traces to | Priority | Status | +|---|---|---|---|---| +| DP-001 | One analysis core; modes are front-ends and must not fork pipeline logic | PR-004 | High | Done | +| DP-002 | Batch CLI over one title | PR-004 | High | Done | +| DP-003 | On-demand resident service with bounded, observable queue | PR-004 | Medium | Planned | +| DP-004 | Opportunistic/idle mode: external trigger, hard stop, implicit re-queue | PR-004 | Medium | Planned | +| DP-005 | Native installer, no Docker; Fedora + Arch | PR-004 | Medium | Planned | +| DP-006 | Background incremental gallery refresh on a timer | PR-003 | Medium | Planned | + +## Integration (IR) + +| ID | Requirement | Traces to | Priority | Status | +|---|---|---|---|---| +| IR-001 | Emit the JRay truth format as sibling `.jray.json` | SR-003 | High | Done | +| IR-002 | Windows carry belief + route; `extraction.*` carries `extinction_sec`, `gallery_scope` | SR-003 | High | Planned | +| IR-003 | Output written **after** the deferred pass, not at EOF | SR-003 | High | Planned | +| IR-004 | Compute the audio signature exactly per server spec §3 | SR-003 | Medium | Planned | +| IR-005 | Golden-vector fixture shared with the plugin repo to prove bit-exactness | SR-003 | High | Planned | +| IR-007 | Media < 120 s: emit no signature, apply no sync offset — identical rule in both producers | SR-003 | Low | Planned | +| IR-008 | Emit and honour the signature's own `v1:` version prefix | SR-003 | Low | Planned | +| IR-006 | Jellyfin round-trip: pull pending queue, push complete results only | SR-001 | High | Done | + +## Gallery (GR) + +| ID | Requirement | Traces to | Priority | Status | +|---|---|---|---|---| +| GR-001 | Build gallery from Jellyfin library cast, TMDB profile fallback | SR-001, SR-005 | High | Done | +| GR-002 | Incremental `--merge` refresh without re-embedding known actors | PR-003 | High | Done | +| GR-003 | Report coverage: zero-image actors, under-referenced actors, dedup, calibration PDFs | SR-001 | Medium | Planned | +| GR-004 | Stamp embedder identity into the gallery; **hard startup error** on mismatch | SR-001 | High | Planned | +| GR-005 | Gallery data never leaves the instance | **SR-005** | High | Done | +| GR-006 | Provenance tiers: baked / harvested / confirmed, distinguishable per embedding | SR-005 | High | Planned | +| GR-007 | Persist harvested embeddings **flagged and reviewable**, never silently equal to baked | SR-005 | Medium | Planned | +| GR-008 | Flag distributional outliers among an actor's references (poisoning guard) — `EXCEPTION: AR-024` | SR-005 | Medium | Planned | +| GR-009 | Human-confirmed associations persist and improve future extractions | §4 | Medium | TBD | + +## Validation (VR) + +| ID | Requirement | Traces to | Priority | Status | +|---|---|---|---|---| +| VR-001 | HDF5 post-inference dump at the embedded-frame boundary | PR-002 | High | Done | +| VR-002 | Replay drives the **real** KPN nodes, not a reimplementation | PR-002 | High | Done | +| VR-003 | Scoring: micro-F1 against X-Ray, precision/recall logged at every evaluation | PR-002 | High | Done | +| VR-004 | Reproducible validation corpus with ground truth | PR-002 | High | Done | +| VR-005 | Minimum face size study — TPI/FPI vs probe size, gallery held at native res | PR-002 | Medium | Planned | +| VR-006 | Re-tune `scene_threshold` once native-rate decode lands | PR-002 | Low | Planned | +| VR-007 | Expansion band, clustering threshold, and deferred-pass ablation | PR-002 | Medium | Planned | +| VR-008 | Gallery scaling benchmark — throughput vs gallery size | PR-002 | Medium | Planned | +| VR-009 | Verify accumulated posteriors are calibrated against held-out tracks | PR-002 | High | Planned | + +--- + +## Verification strategy + +**CI runs on an Intel N100 with no discrete GPU.** That is a hard constraint on +how each requirement can be verified, and it shapes the test design rather than +merely limiting it. + +Four tiers, in decreasing order of preference: + +| Tier | Runs in CI | What it covers | +|---|---|---| +| **T1 — CPU unit** | Yes | Pure logic: registry state machine, belief accumulation, clustering, band admission, calibration maths | +| **T2 — Replay** | Yes | Real pipeline nodes driven from an HDF5 fixture — no GPU, no video | +| **T3 — CPU inference** | Yes, slowly | ORT CPU provider over a handful of frames; smoke tests only | +| **T4 — GPU** | **No** | Throughput, TRT engines, large-gallery GEMM | + +**T2 is the reason this is workable.** The HDF5 dump (VR-001) captures state +after decode → detect → align → embed and before tracking and matching, so +everything downstream — which is where nearly all of the new design lives — is +cheap CPU maths replayable from a fixture. Tracking, presence windows, belief +accumulation, expansion, deferred re-identification and clustering are all +verifiable on an N100 at full fidelity, not in miniature. + +That was already true for the optimizer. It now doubles as the CI strategy, which +is a strong argument for keeping the dump schema honest (VR-001) and for the +replay driving the *real* nodes rather than a reimplementation (VR-002). + +**Small committed fixtures are required.** A few HDF5 dumps covering the awkward +cases — a cut, a belief swap, two live tracks converging, a film ending +mid-track, an unknown track that only resolves after expansion — are worth more +than a large corpus, and they are small enough to commit. + +**T4 requirements cannot pass in CI, and the gate must not pretend otherwise.** +For these, CI verifies that a test *exists and is tagged*, not that it passes; +the run happens on a GPU host, nightly or manually, and reports separately. A +requirement whose only evidence is a test that never executes should be visible +as such rather than counted as covered. + +| Requirement | Tier | Note | +|---|---|---| +| AR-001, AR-005, AR-006 | T3 | Smoke only — correctness of detection/embedding is a model property, not ours | +| AR-002 | T2 | Size filtering is arithmetic on dumped bboxes | +| AR-003, AR-004 | T1 + T4 | Backpressure logic is unit-testable; saturation behaviour needs real load | +| AR-007 … AR-017 | **T2** | The core of the redesign — fully replayable | +| AR-018 … AR-022 | **T2** | Expansion, deferred pass, clustering: all post-embedding | +| AR-023 … AR-025 | T1 | Calibration fit and log-odds accumulation are pure maths | +| AR-026, AR-027 | T4 | GEMM throughput and scaling — GPU host only | +| DP-* | T1 + manual | Lifecycle logic unit-tested; install paths are manual | +| IR-001 … IR-003 | T1 | Serialisation against a golden truth file | +| IR-004, IR-005 | **T1** | Audio signature is CPU DSP — the golden-vector fixture runs anywhere, which is precisely why it is the right cross-repo check | +| GR-001 … GR-005 | T1 + T3 | Gallery assembly is I/O and bookkeeping; embedding is T3 smoke | +| GR-006 … GR-008 | T1 | Tiering and outlier detection operate on stored embeddings | +| VR-* | Out of CI | Studies are run deliberately and their results committed as documents | + +**One consequence worth stating:** AR-027 (arbitrary gallery scale) is +structurally unverifiable on the CI host. It needs a GPU host and a synthetic +large gallery, so it is the requirement most likely to silently regress. Its +benchmark (VR-008) should run on a schedule rather than on demand. + +### Fixtures — precomputed inference, pulled by CI + +The N100 cannot run inference at any useful rate, so **inference output is +precomputed on a GPU host and consumed by CI as data.** This converts most of +what looks like GPU work into pure CPU replay. + +| Fixture | Contents | Size | Storage | +|---|---|---|---| +| **Edge-case dumps** | ~6 short clips (30–60 s), one per awkward behaviour | ~1 MB each | **Committed in-repo** | +| **Corpus dumps** | Full-length titles from the validation corpus | ~30 MB each | Pinned artifact, fetched by checksum | +| **Synthetic gallery** | Random unit-norm embeddings, fixed seed | small | Generated at test time | +| **Golden truth files** | Expected output for each edge-case dump | KB | Committed | +| **Audio golden vectors** | Short WAV + expected signature | KB | Committed, **shared with the plugin repo** | + +Edge-case dumps are small enough to commit, and being in-repo means they version +with the code that reads them. Corpus dumps are pulled by pinned checksum from +the artifact store rather than committed, since they are large and change only +when the dump schema does. + +**Generation must be reproducible and versioned.** A script, run on a GPU host, +regenerates every fixture from source clips; it is re-run when the VR-001 schema +version bumps. A fixture whose provenance is unknown is worse than no fixture, +because it will be trusted. + +> **The limitation that must stay visible:** replay fixtures freeze upstream +> behaviour. A test driven from a dump verifies AR-007 onward *given those +> embeddings* — it cannot detect a regression in detection, alignment or +> embedding, because those produced the fixture. Nothing in CI can. That gap is +> covered only by the T3 smoke test and the scheduled GPU run, and it should not +> be papered over by a high replay-coverage number. + +### Per-requirement verification plan + +| ID | Tier | Test asserts | Edge cases to cover | +|---|---|---|---| +| AR-001 | T3 | Detector returns plausible boxes on a known frame | — smoke only | +| AR-002 | T2 | Faces below 66 px (original res) are dropped | Exactly at threshold; with `dense_scale` 0.5 — the interaction that motivated the requirement | +| AR-003 | T2 | No cap applied; a 40-face frame yields 40 | Crowd frame | +| AR-004 | T1 | Saturated input blocks rather than drops or throws | Bounded queue at capacity; **byte-based** limit with large crops; SIGTERM mid-block | +| AR-005 | T1 | Known landmarks → expected 112×112 warp | Landmarks near frame edge; degenerate/collinear points | +| AR-006 | T3 | Embeddings are unit-norm | Batch smaller than, equal to, larger than `embed_batch_size` | +| AR-007 | T2 | Association picks the right track | Two faces crossing paths; one leaving frame as another enters | +| AR-008 | T2 | One pool; dormant tracks match on embedding, not IoU | Dormant track whose old bbox overlaps a *different* new face — must not match on position | +| AR-009/010 | T2 | Cut/boundary shifts weighting toward embedding | Cut with same people; cut with all-new people | +| AR-011 | T1 | TransNetV2 receives native-rate frames | Source at 24/25/30 fps — dedup window derived, not assumed | +| AR-012 | **T2** | Window spans full track extent, not first recognition | Actor recognised only at track end — window must still start at `first_seen` | +| AR-013 | **T2** | `last_seen` set/unset; window ends at last sighting | Gap just under vs just over timeout; reappearance after timeout → two windows | +| AR-014 | T2 | Belief swap closes one window, opens another | No blended window; no overlap at the swap frame | +| AR-015 | T2 | Two live tracks on one actor trigger re-association | Counter increments | +| AR-016 | **T2** | Every track closed at EOF | Film ending mid-shot — window ends at final frame, not dropped | +| AR-017 | T1 | Claim carries posterior and route | Deferred and pooled routes distinguishable | +| AR-018 | T1 | Band admits only within bounds | At each bound exactly; store never admits below lower bound | +| AR-019 | T2 | Promotion only when all three signals quiet | Cut mid-track blocks promotion | +| AR-020 | **T2** | Unknown resolved after expansion | Track failing at minute 12, resolved at EOF — the ordering-independence claim | +| AR-021 | T2 | Clustering merges same person, respects cannot-link | **Temporally overlapping tracks never merge**; measure how many merges the constraint rejects | +| AR-022 | T1 | Context crops retained, bounded per track | Track running for minutes | +| AR-023 | T1 | Sigmoid fit on synthetic separable data | Too few positive pairs → `valid=false`, fallback engages | +| AR-024 | **Static check** | No bare cosine outside a tagged `EXCEPTION` | Grep-based; this is the invariant's enforcement | +| AR-025 | T1 | Log-odds accumulate; correlated frames discounted | 30 identical frames must **not** reach the certainty of 30 diverse ones | +| AR-026 | T1 + T4 | GEMM path produces same result as reference loop | Equivalence on small input in CI; throughput on GPU host | +| AR-027 | **T4** | Throughput at 10²…10⁵ actors | Scheduled, not on-demand | +| IR-001/002 | T1 | Serialised output matches golden file | Zero-length window; actor with many windows | +| IR-003 | T1 | Output written after deferred pass | Not at EOF | +| IR-004/005 | **T1** | Signature matches golden vector bit-for-bit | **Media < 120 s → no signature**; identical result in both repos | +| GR-004 | T1 | Mismatched embedder → hard startup error | Error names both sides | +| GR-008 | T1 | Outlier flagged among an actor's references | Injected poisoned embedding detected | +| VR-009 | T1 | Posterior calibration holds | A 0.99 posterior is wrong ~1% of the time on held-out tracks | + +Three of these are worth singling out because they verify claims that would +otherwise be assertions: **AR-012** (window starts at `first_seen` even when +recognition comes late) is the entire point of the redesign; **AR-020** (a track +failing mid-film resolves at EOF) is the claim that ordering stops mattering; and +**AR-025** (30 identical frames ≠ 30 diverse ones) is what stops the Bayesian +accumulation from being decoration. + +--- + +## Withdrawn + +| ID | Requirement | Reason | +|---|---|---| +| — | `anneal_sec` window merging | Superseded by AR-012/AR-013: a track survives its own gaps, so there is nothing to anneal | +| — | `extinction_sec` actor keep-alive | Superseded by AR-013: windows end at last sighting, which is what this over-claimed | + +Both were deleted rather than retained at zero — a field naming a mechanism the +pipeline no longer has is actively misleading (see `SPEC.md` A6.6). + +--- + +## Notes on coverage + +- **VR-*** traces to PR-002 (scene-granularity answers) rather than to a system + requirement: parameter studies are single-repo work serving accuracy, and this + is correct rather than a gap. +- **PR-005** (leak nothing) has no `AR`/`DP` row. It is satisfied *structurally* + by SR-004 and GR-005 — the server holds no binary, the gallery never leaves the + instance — not by any component doing something. It cannot be verified by + pointing at code, and it dies the moment either prohibition is relaxed.