scene-actor-extraction/docs/plan.md

Implementation plan — per requirement

One entry per requirement that needs work. Requirements marked Done in requirements.md are omitted.

Ordering is derived from dependencies, not assigned to phases. Each entry lists what it depends on; anything with no unmet dependency is startable. This replaces the earlier phase-based plan, which encoded ordering assumptions that stopped being true as the design changed.

Verification for each requirement is specified in requirements.md — this document covers how to build it, not how to prove it.

---

Startable now (no unmet dependencies)

GR-004 · IR-004 · IR-005 · IR-007 · IR-008 · VR-005 · AR-011 · AR-023 extension · tooling port

These touch disjoint files and can proceed concurrently.

Blocked on the registry

Everything in AR-007 … AR-022 depends on AR-012/AR-013 landing first, because they all read or write track state. This group is one coherent refactor, not parallel work — splitting it across concurrent efforts produces incompatible designs in the same files.

---

Algorithm

AR-012, AR-013 — TrackRegistry (the spine)

Depends on: nothing. Blocks: AR-007, AR-008, AR-014 … AR-022.

Everything else in Part A waits on this, so it goes first.

Ownership: a shared resource, not a node

The registry is external to the dataflow network, created in main and handed to each node that needs it as std::shared_ptr<TrackRegistry>. Lifetime is guaranteed by refcount rather than by the "object must outlive the node" convention, so no ordering assumption exists between network teardown and registry destruction.

This is idiomatic here: node functors are already constructed outside the network and passed by reference (main.cpp:186-207), and KPN provides SharedResource<T> for state shared across nodes (KPN SPEC §163, §445).

Not a node, because ownership is not a stage in the stream — it is state several stages read and write, whose final answer is only known when a track dies. Not inside TrackGallery, because that would couple presence to expand_gallery, a switchable feature.

The registry is the tracker's state. FaceTrackerFunc does not keep its own tracks_/inactive_ maps and mirror them in — it operates on the registry directly. Two parallel copies could disagree, and every divergence would surface as wrong presence windows, silently.

Per-track state

Track
  first_seen : double            set once, at creation
  last_seen  : optional<double>  UNSET while on screen; set to the last
                                 on-screen timestamp when the face is lost
  actor      : optional<int>     set when a posterior crosses the threshold
  belief     : {actor_idx -> accumulated_logodds}   Bayesian, not a tally
  embedding  : Embedding         running directional mean, for association

last_seen carries the entire liveness state. Unset = on screen; set = went off at T. No separate missing-frames counter, no expired flag — the optional is the state machine, and it subsumes the current two-pool split (tracks_ = unset, inactive_ = set).

Lifecycle

face detected, no match          → new track, first_seen = t, last_seen = unset
actor identified                 → update belief; set actor when threshold crossed
face lost                        → last_seen = t_last_on_screen  (stays revivable)
face seen again, embedding match → last_seen = unset  (same track continues)
tick(t), t - last_seen > timeout → emit to aggregator, DELETE the entry

A presence window is [first_seen, last_seen]. Nothing else.

Interior gaps are claimed; the trailing cool-down is not. A face lost at t₁ and re-acquired at t₂ within the timeout never closed its track, so the actor is present across [t₁, t₂] — correct, since someone briefly occluded or off-camera has not left the scene. But a track that dies ends at last_seen, not at the moment of death. That asymmetry is what removes the old extinction_sec over-claim.

Reaping is a handoff, not a deletion into a holding pen. The dead track goes to the result aggregator immediately and the registry drops it, so the registry holds only live tracks and its size is bounded by concurrent on-screen faces.

Interface

TrackRegistry
  tick(timestamp)                                 ← FaceTrackerFunc, every frame
  candidates() -> span<Track&>                    → all live tracks
  create(timestamp, embedding) -> track_id
  mark_seen(track_id, timestamp, embedding)       → updates mean, clears last_seen
  mark_lost(track_id, last_on_screen_timestamp)
  on_vote(track_id, actor_idx, posterior)         ← IdentityMatcherFunc
  owner(track_id) -> optional<actor_idx>          → TrackGallery
  on_track_dead : callback(DeadTrack)             → ResultSinkFunc
  flush()                                         ← at EOF

candidates() returns one pool; last_seen tells the caller whether IoU applies. There is no separate revival path — matching a dormant track is ordinary inter-frame association.

tick() advances the clock so dead tracks are reaped independently of detection activity; without it a track only dies when some other face happens to appear.

Locking

The tracker mutates registry state across a frame's association pass, so that pass holds the lock for its duration (a frame_scope() handle). Every other caller's operations must be individually atomic. A single std::mutex over the whole registry is the right start — contention is a few small updates per frame against per-frame work measured in GPU milliseconds.

Two cases constrain the API:

• owner() is a read-modify-read in disguise: TrackGallery calls it while IdentityMatcher may be voting on the same track. Tally and verdict must be read under one lock as a snapshot, or a track can be both unowned and owned within a single promotion decision.
• on_vote() arrives downstream of the tracker's tick() for the same frame, so a vote may land after the clock moved on. Rule: a vote for a known track always lands on its tally, regardless of clock. Only reaping is clock-driven. A vote for an already-reaped track is dropped and counted — a nonzero count means the timeout is shorter than the matcher's lag.

on_track_dead fires from inside tick() while the frame lock is held, so the callback must not re-enter the registry. Keep it to a push onto the aggregator's storage.

AR-016 — EOF flush

Depends on: AR-012.

flush() emits every still-live track through the same callback, closing at last_seen if set and the final tick timestamp otherwise. Idempotent, leaving the registry empty; the sink's written_.exchange(true) guard (result_sink_node.hpp:66) shows the shape.

Must run on every termination path that produces output. Not SIGTERM during opportunistic runs (DP-004) — those push no partial result, so there is nothing to flush.

Without it a film ending mid-shot silently drops its closing cast, which looks like a recognition miss rather than a bookkeeping bug.

AR-014, AR-015 — Contradiction rules

Depends on: AR-012, AR-025.

Condition	Meaning	Action
Belief on one track swaps A → B	track_id carried across a viewpoint change onto a different person	Close at last_seen, open a new track for B at the swap frame
Two live tracks owned by one actor	One person split in two, or an identity attached to the wrong track	Treat as a detected cut: reset affected state, re-associate on embedding

The second makes identity a third cut detector, independent of histogram and TransNetV2, firing where those failed. Detect it via a reverse index actor_idx → live track_ids, so the condition is caught on the update that causes it rather than by scanning.

Both counted and reported — the rates measure how often tracking is silently wrong, which nothing currently reveals.

AR-007, AR-008 — Tracker on one pool

Depends on: AR-012, AR-024.

FaceTrackerFunc is constructed with the registry and uses it as state; its tracks_/inactive_ maps and the cross-cut revival branch collapse into one pool keyed on last_seen. Per frame: tick(), association over candidates(), then create/mark_seen/mark_lost.

track_alpha becomes frame-dependent — normal frames use the tuned blend, frames flagged is_cut/is_scene_boundary drop toward embedding-only.

AR-024 — Probability space everywhere

Depends on: AR-023. Blocks: AR-007, AR-018, AR-021, AR-025.

Cuts across tracker, matcher and expansion, so it lands with the registry work rather than after it. Retires track_max_embed_dist, cut_revive_sim, expand_novelty_sim, expand_track_spread_max.

Enforcement is a static grep check for bare cosine outside a tagged EXCEPTION — a unit test cannot prove absence across a codebase.

AR-025 — Bayesian accumulation

Depends on: AR-023, AR-024.

Log-odds per candidate actor, added per frame. on_vote() is an update, not an increment.

The independence problem must be handled explicitly. Consecutive frames are highly correlated; naive accumulation drives the posterior to certainty on what is effectively one observation. Preferred mitigation: update only on sufficiently novel observations, reusing the diversity buffer's existing judgement rather than inventing a second one. The registry should receive already-discounted evidence.

AR-017 — Claims carry belief and route

Depends on: AR-012, AR-025. DeadTrack carries posterior plus how it was identified (live / deferred / pooled).

AR-018 … AR-021 — Expansion, deferred pass, clustering

Depends on: AR-012, AR-024, AR-026.

Ordering within the group: AR-018 (banded store) → AR-019 (annex) → AR-020 (TBI queue + deferred pass) → AR-021 (clustering).

AR-021 needs the temporal cannot-link constraint from track extents, so it cannot start before AR-012. The annex must be a contiguous matrix with promotions appended (AR-026), not a list.

Output timing changes: the sink can no longer finalise at EOF — the deferred pass runs after and may add windows (IR-003).

AR-022 — Unidentified capture

Depends on: AR-020. Unidentified = TBI entries surviving the deferred pass. Context crops opt-in behind --dump-unidentified-crops.

AR-001 … AR-004 — Detection and backpressure

Depends on: nothing (AR-002, AR-011); AR-004 blocks AR-003.

• AR-002 — min_face_px stays 40 (VR-013 measured it end to end) but must be expressed in original resolution rather than decoded-frame space. The value is already right in config.hpp; the change is the coordinate space.
• AR-011 — feed TransNetV2 at native rate; derive the dedup window from source fps rather than the hardcoded 0.04 s.
• AR-004 — backpressure. kMaxFaces (identity_matcher_node.hpp:133) currently throws; channel capacities of 16 (main.cpp:204-207) were sized against ≤10 faces/frame. Must block on bytes in flight, not item counts.
• AR-003 — remove max_faces. Gated on AR-004, not a follow-up to it.

AR-026, AR-027 — GEMM and scale

Depends on: nothing to start. The annex CPU loop has moved into the GEMM path: the annex is a contiguous matrix, promotions are appended to the engine's resident gallery, and the CPU backend now requires OpenBLAS. What is left of AR-026 is call site 3, the deferred pass — so the rest of AR-026 lands with AR-020 rather than before it.

---

Gallery

GR-004 — Model binding — DONE

Depended on: nothing. Landed before any measurement work, as intended.

Stamp = model basename + SHA-256 of the ONNX, written as the /embedder group at build time (gallery_builder.cpp, sae_gallery.save_gallery_hdf5) and verified at load in scene_analyze, scene_preview, the sae_kpn matcher binding, replay.py, optimize.py and movienet_eval.py. Mismatch is a hard error naming both sides, with no bypass. Embedding dumps carry the same stamp, since a replay has no live embedder to check against.

Unstamped legacy galleries warn loudly and proceed rather than failing: unknown is not known-bad, and hard-failing every pre-existing gallery would turn the check into something people disable. --require-gallery-stamp / SAE_REQUIRE_GALLERY_STAMP=1 promotes that to a hard error — measurement runs should set it. scripts/stamp_gallery.py re-binds an existing gallery without re-embedding, so the warning state is cheap to leave.

Cross-model similarities are meaningless but look plausible — this fails silently and expensively, and it would corrupt every measurement taken during the rest of this work.

GR-003 — Coverage reporting

Depends on: nothing. Surface what calibration already computes and discards (kHistBins = 200): zero-image actors, under-referenced actors, dedup counts, and the intra/inter PDFs.

GR-006 … GR-008 — Provenance tiers

Depends on: AR-019. Tier per embedding (baked / harvested / confirmed); harvested persisted but flagged; bell-curve outlier check (EXCEPTION: AR-024).

---

Integration

IR-004, IR-005, IR-007, IR-008 — Audio signature

Depends on: nothing. Fully independent — no existing pipeline file is touched. Best candidate for concurrent work.

Implement server spec §3 exactly. Audio decode is a second stream from the already-linked FFmpeg. Media < 120 s: no signature, no offset. Emit and honour the v1: prefix.

The golden-vector fixture is shared with the plugin repo and runs on CPU, so the one place two implementations must agree bit-for-bit is verifiable in CI.

IR-001 … IR-003 — Truth file

Depends on: AR-017 (belief), AR-020 (output timing).

Windows carry belief and route; extraction.* gains extinction_sec and gallery_scope; anneal_sec removed. All breaking → one coordinated schema_version bump with IR-004 (SR-003).

---

Validation

VR-005 — Minimum face size study

Depends on: nothing. Standalone Python, no C++ contact. Done — knee at 24–32 px. It measures the embedder with alignment held perfect, so it bounds the answer from below rather than setting it; AR-002's floor comes from VR-013, which sweeps input resolution end to end and lands at 40 px.

VR-013 — Cross-source identification probe

Depends on: sae_embed exposing detect(), align_face(), embed_crop() and the gallery calibration — it drives the shipped C++ rather than reimplementing it, which is what VR-005 could not do.

Gallery from one recording, probes from another, sweeping the probe's input resolution before the detector, so detection and landmark regression degrade with the frame. experiments/xsource/.

Findings. Holding 90% of the plateau needs ~50 px end to end against VR-005's ~22 px; min_face_px 40 is right and 32 would admit faces in the falling region. FPI is 0.0% at every scale — resolution loss goes entirely to TBI, never to a wrong name. The ceiling is cross-view, not resolution: everyone matches themselves within a recording (0.55–0.85) and collapses across two (0.14–0.45), and only the subject with frontal gallery references identified reliably — so the lever is gallery pose coverage (docs/pose-expansion.md), not a better landmark model. Averaging SCRFD's NMS-discarded landmark estimates lifts cross-clip TPI 41% → 49% for one forward pass.

Open. Four identities and one shoot, so the shape is the result and the absolute rates are not. Both clips hold all four people, so there is no out-of-gallery class and the 10×-weighted out-of-cast misID is untested — holding one identity out of the gallery would fix that.

VR-014 — Audio-signature offset recovery

Depends on: sae_audio exposing compute_signature() and signature_from_mono() — it drives the shipped C++, as VR-013 does, so the thing measured is the thing that ships.

scripts/validation/test_audio_offset.py over tests/fixtures/audio/superhero_offset_200s.flac: 200 s of public-domain film audio (the same SuperHero clips the replay fixtures use), long enough for a 120 s window to slide past the ±600-frame search cap. The slide itself is numpy here on purpose — matching belongs to the consumer, so writing it out keeps this a test of the signature rather than of somebody's matcher.

Findings. Alignment is a solved problem here: the offset is the nearest frame in every in-cap trial, worst error 46 ms against a 500 ms budget, and 46 ms is the quantisation floor — offsets are whole 92.88 ms frames, so no correct answer can be worse. The runtime/2 anchor's factor of two holds through real trimmed files, and out-of-cap offsets and unrelated content are both declined.

The score is where the slack is, and it costs a tier rather than accuracy. It tracks sub-frame misalignment — 0.94–0.99 near a frame boundary, 0.69–0.73 at half a frame — so two thirds of correct alignments miss the server's 0.85 audio threshold and land in loose. UT-108 measures the fix rather than proposing one: ±1 frame of slack in the score returns all 40 to audio (min 0.906) with false matches unmoved at 0.12–0.16, costing 81 ms of the budget. See SPEC.md IR-004 — the score is normative in the server spec, so the change is theirs to make.

Open. One source, one language, one era of recording. The shape (offset exact, score set by sub-frame phase) should hold generally, but the absolute scores are this fixture's.

VR-001 — Dump audit

Depends on: nothing. Read-only investigation: confirm the HDF5 dump preserves everything needed to reconstruct tracks deterministically, including the park/revive path. Prerequisite for the CI strategy, since T2 replay is how most of AR-007 … AR-022 is verified.

VR-006 … VR-009

Depends on: their subjects landing. VR-009 (posterior calibration holds) depends on AR-025 and is what stops the Bayesian accumulation being decoration.

---

Withdrawn from the old plan

The phase structure, the --presence-mode {frame,track} flag, and "Phase 2 — retune anneal_sec/extinction_sec". Those constants are withdrawn rather than retuned; comparison against old behaviour uses recorded reference output instead of a second live code path.