scene-actor-extraction/docs/plan.md

Implementation plan — per requirement

One entry per requirement that needs work. Requirements marked Done in requirements.md are omitted.

Ordering is derived from dependencies, not assigned to phases. Each entry lists what it depends on; anything with no unmet dependency is startable. This replaces the earlier phase-based plan, which encoded ordering assumptions that stopped being true as the design changed.

Verification for each requirement is specified in requirements.md — this document covers how to build it, not how to prove it.

---

Startable now (no unmet dependencies)

GR-004 · IR-004 · IR-005 · IR-007 · IR-008 · VR-005 · AR-011 · AR-023 extension · tooling port

These touch disjoint files and can proceed concurrently.

Blocked on the registry

Everything in AR-007 … AR-022 depends on AR-012/AR-013 landing first, because they all read or write track state. This group is one coherent refactor, not parallel work — splitting it across concurrent efforts produces incompatible designs in the same files.

---

Algorithm

AR-012, AR-013 — TrackRegistry (the spine)

Depends on: nothing. Blocks: AR-007, AR-008, AR-014 … AR-022.

Everything else in Part A waits on this, so it goes first.

Ownership: a shared resource, not a node

The registry is external to the dataflow network, created in main and handed to each node that needs it as std::shared_ptr<TrackRegistry>. Lifetime is guaranteed by refcount rather than by the "object must outlive the node" convention, so no ordering assumption exists between network teardown and registry destruction.

This is idiomatic here: node functors are already constructed outside the network and passed by reference (main.cpp:186-207), and KPN provides SharedResource<T> for state shared across nodes (KPN SPEC §163, §445).

Not a node, because ownership is not a stage in the stream — it is state several stages read and write, whose final answer is only known when a track dies. Not inside TrackGallery, because that would couple presence to expand_gallery, a switchable feature.

The registry is the tracker's state. FaceTrackerFunc does not keep its own tracks_/inactive_ maps and mirror them in — it operates on the registry directly. Two parallel copies could disagree, and every divergence would surface as wrong presence windows, silently.

Per-track state

Track
  first_seen : double            set once, at creation
  last_seen  : optional<double>  UNSET while on screen; set to the last
                                 on-screen timestamp when the face is lost
  actor      : optional<int>     set when a posterior crosses the threshold
  belief     : {actor_idx -> accumulated_logodds}   Bayesian, not a tally
  embedding  : Embedding         running directional mean, for association

last_seen carries the entire liveness state. Unset = on screen; set = went off at T. No separate missing-frames counter, no expired flag — the optional is the state machine, and it subsumes the current two-pool split (tracks_ = unset, inactive_ = set).

Lifecycle

face detected, no match          → new track, first_seen = t, last_seen = unset
actor identified                 → update belief; set actor when threshold crossed
face lost                        → last_seen = t_last_on_screen  (stays revivable)
face seen again, embedding match → last_seen = unset  (same track continues)
tick(t), t - last_seen > timeout → emit to aggregator, DELETE the entry

A presence window is [first_seen, last_seen]. Nothing else.

Interior gaps are claimed; the trailing cool-down is not. A face lost at t₁ and re-acquired at t₂ within the timeout never closed its track, so the actor is present across [t₁, t₂] — correct, since someone briefly occluded or off-camera has not left the scene. But a track that dies ends at last_seen, not at the moment of death. That asymmetry is what removes the old extinction_sec over-claim.

Reaping is a handoff, not a deletion into a holding pen. The dead track goes to the result aggregator immediately and the registry drops it, so the registry holds only live tracks and its size is bounded by concurrent on-screen faces.

Interface

TrackRegistry
  tick(timestamp)                                 ← FaceTrackerFunc, every frame
  candidates() -> span<Track&>                    → all live tracks
  create(timestamp, embedding) -> track_id
  mark_seen(track_id, timestamp, embedding)       → updates mean, clears last_seen
  mark_lost(track_id, last_on_screen_timestamp)
  on_vote(track_id, actor_idx, posterior)         ← IdentityMatcherFunc
  owner(track_id) -> optional<actor_idx>          → TrackGallery
  on_track_dead : callback(DeadTrack)             → ResultSinkFunc
  flush()                                         ← at EOF

candidates() returns one pool; last_seen tells the caller whether IoU applies. There is no separate revival path — matching a dormant track is ordinary inter-frame association.

tick() advances the clock so dead tracks are reaped independently of detection activity; without it a track only dies when some other face happens to appear.

Locking

The tracker mutates registry state across a frame's association pass, so that pass holds the lock for its duration (a frame_scope() handle). Every other caller's operations must be individually atomic. A single std::mutex over the whole registry is the right start — contention is a few small updates per frame against per-frame work measured in GPU milliseconds.

Two cases constrain the API:

• owner() is a read-modify-read in disguise: TrackGallery calls it while IdentityMatcher may be voting on the same track. Tally and verdict must be read under one lock as a snapshot, or a track can be both unowned and owned within a single promotion decision.
• on_vote() arrives downstream of the tracker's tick() for the same frame, so a vote may land after the clock moved on. Rule: a vote for a known track always lands on its tally, regardless of clock. Only reaping is clock-driven. A vote for an already-reaped track is dropped and counted — a nonzero count means the timeout is shorter than the matcher's lag.

on_track_dead fires from inside tick() while the frame lock is held, so the callback must not re-enter the registry. Keep it to a push onto the aggregator's storage.

AR-016 — EOF flush

Depends on: AR-012.

flush() emits every still-live track through the same callback, closing at last_seen if set and the final tick timestamp otherwise. Idempotent, leaving the registry empty; the sink's written_.exchange(true) guard (result_sink_node.hpp:66) shows the shape.

Must run on every termination path that produces output. Not SIGTERM during opportunistic runs (DP-004) — those push no partial result, so there is nothing to flush.

Without it a film ending mid-shot silently drops its closing cast, which looks like a recognition miss rather than a bookkeeping bug.

AR-014, AR-015 — Contradiction rules

Depends on: AR-012, AR-025.

Condition	Meaning	Action
Belief on one track swaps A → B	track_id carried across a viewpoint change onto a different person	Close at last_seen, open a new track for B at the swap frame
Two live tracks owned by one actor	One person split in two, or an identity attached to the wrong track	Treat as a detected cut: reset affected state, re-associate on embedding

The second makes identity a third cut detector, independent of histogram and TransNetV2, firing where those failed. Detect it via a reverse index actor_idx → live track_ids, so the condition is caught on the update that causes it rather than by scanning.

Both counted and reported — the rates measure how often tracking is silently wrong, which nothing currently reveals.

AR-007, AR-008 — Tracker on one pool

Depends on: AR-012, AR-024.

FaceTrackerFunc is constructed with the registry and uses it as state; its tracks_/inactive_ maps and the cross-cut revival branch collapse into one pool keyed on last_seen. Per frame: tick(), association over candidates(), then create/mark_seen/mark_lost.

track_alpha becomes frame-dependent — normal frames use the tuned blend, frames flagged is_cut/is_scene_boundary drop toward embedding-only.

AR-024 — Probability space everywhere

Depends on: AR-023. Blocks: AR-007, AR-018, AR-021, AR-025.

Cuts across tracker, matcher and expansion, so it lands with the registry work rather than after it. Retires track_max_embed_dist, cut_revive_sim, expand_novelty_sim, expand_track_spread_max.

Enforcement is a static grep check for bare cosine outside a tagged EXCEPTION — a unit test cannot prove absence across a codebase.

AR-025 — Bayesian accumulation

Depends on: AR-023, AR-024.

Log-odds per candidate actor, added per frame. on_vote() is an update, not an increment.

The independence problem must be handled explicitly. Consecutive frames are highly correlated; naive accumulation drives the posterior to certainty on what is effectively one observation. Preferred mitigation: update only on sufficiently novel observations, reusing the diversity buffer's existing judgement rather than inventing a second one. The registry should receive already-discounted evidence.

AR-017 — Claims carry belief and route

Depends on: AR-012, AR-025. DeadTrack carries posterior plus how it was identified (live / deferred / pooled).

AR-018 … AR-021 — Expansion, deferred pass, clustering

Depends on: AR-012, AR-024, AR-026.

Ordering within the group: AR-018 (banded store) → AR-019 (annex) → AR-020 (TBI queue + deferred pass) → AR-021 (clustering).

AR-021 needs the temporal cannot-link constraint from track extents, so it cannot start before AR-012. The annex must be a contiguous matrix with promotions appended (AR-026), not a list.

Output timing changes: the sink can no longer finalise at EOF — the deferred pass runs after and may add windows (IR-003).

AR-022 — Unidentified capture

Depends on: AR-020. Unidentified = TBI entries surviving the deferred pass. Context crops opt-in behind --dump-unidentified-crops.

AR-001 … AR-004 — Detection and backpressure

Depends on: nothing (AR-002, AR-011); AR-004 blocks AR-003.

• AR-002 — min_face_px → 66, expressed in original resolution.
• AR-011 — feed TransNetV2 at native rate; derive the dedup window from source fps rather than the hardcoded 0.04 s.
• AR-004 — backpressure. kMaxFaces (identity_matcher_node.hpp:133) currently throws; channel capacities of 16 (main.cpp:204-207) were sized against ≤10 faces/frame. Must block on bytes in flight, not item counts.
• AR-003 — remove max_faces. Gated on AR-004, not a follow-up to it.

AR-026, AR-027 — GEMM and scale

Depends on: nothing to start. The annex CPU loop (identity_matcher_node.hpp:159-162) moves into the GEMM path.

---

Gallery

GR-004 — Model binding

Depends on: nothing. Startable immediately, highest value per line.

Stamp embedder identity into the gallery at build; verify at load in scene_analyze, replay.py and the optimizer. Mismatch is a hard error naming both sides.

Cross-model similarities are meaningless but look plausible — this fails silently and expensively, and it would corrupt every measurement taken during the rest of this work.

GR-003 — Coverage reporting

Depends on: nothing. Surface what calibration already computes and discards (kHistBins = 200): zero-image actors, under-referenced actors, dedup counts, and the intra/inter PDFs.

GR-006 … GR-008 — Provenance tiers

Depends on: AR-019. Tier per embedding (baked / harvested / confirmed); harvested persisted but flagged; bell-curve outlier check (EXCEPTION: AR-024).

---

Integration

IR-004, IR-005, IR-007, IR-008 — Audio signature

Depends on: nothing. Fully independent — no existing pipeline file is touched. Best candidate for concurrent work.

Implement server spec §3 exactly. Audio decode is a second stream from the already-linked FFmpeg. Media < 120 s: no signature, no offset. Emit and honour the v1: prefix.

The golden-vector fixture is shared with the plugin repo and runs on CPU, so the one place two implementations must agree bit-for-bit is verifiable in CI.

IR-001 … IR-003 — Truth file

Depends on: AR-017 (belief), AR-020 (output timing).

Windows carry belief and route; extraction.* gains extinction_sec and gallery_scope; anneal_sec removed. All breaking → one coordinated schema_version bump with IR-004 (SR-003).

---

Validation

VR-005 — Minimum face size study

Depends on: nothing. Standalone Python, no C++ contact. Produces the measured value replacing AR-002's 66 px estimate.

VR-001 — Dump audit

Depends on: nothing. Read-only investigation: confirm the HDF5 dump preserves everything needed to reconstruct tracks deterministically, including the park/revive path. Prerequisite for the CI strategy, since T2 replay is how most of AR-007 … AR-022 is verified.

VR-006 … VR-009

Depends on: their subjects landing. VR-009 (posterior calibration holds) depends on AR-025 and is what stops the Bayesian accumulation being decoration.

---

Withdrawn from the old plan

The phase structure, the --presence-mode {frame,track} flag, and "Phase 2 — retune anneal_sec/extinction_sec". Those constants are withdrawn rather than retuned; comparison against old behaviour uses recorded reference output instead of a second live code path.