scene-actor-extraction/docs/plan.md

# Implementation plan — per requirement

One entry per requirement that needs work. Requirements marked `Done` in
[`requirements.md`](requirements.md) are omitted.

**Ordering is derived from dependencies, not assigned to phases.** Each entry
lists what it depends on; anything with no unmet dependency is startable. This
replaces the earlier phase-based plan, which encoded ordering assumptions that
stopped being true as the design changed.

Verification for each requirement is specified in
[`requirements.md`](requirements.md) — this document covers *how to build it*,
not how to prove it.

---

## Startable now (no unmet dependencies)

`GR-004` · `IR-004` · `IR-005` · `IR-007` · `IR-008` · `VR-005` · `AR-011` ·
`AR-023` extension · tooling port

These touch disjoint files and can proceed concurrently.

## Blocked on the registry

Everything in `AR-007` … `AR-022` depends on `AR-012`/`AR-013` landing first,
because they all read or write track state. **This group is one coherent
refactor, not parallel work** — splitting it across concurrent efforts produces
incompatible designs in the same files.

---

# Algorithm

## AR-012, AR-013 — TrackRegistry (the spine)

**Depends on:** nothing. **Blocks:** AR-007, AR-008, AR-014 … AR-022.

Everything else in Part A waits on this, so it goes first.

### Ownership: a shared resource, not a node

The registry is **external to the dataflow network**, created in `main` and
handed to each node that needs it as `std::shared_ptr<TrackRegistry>`. Lifetime
is guaranteed by refcount rather than by the "object must outlive the node"
convention, so no ordering assumption exists between network teardown and
registry destruction.

This is idiomatic here: node functors are already constructed outside the network
and passed by reference (`main.cpp:186-207`), and KPN provides `SharedResource<T>`
for state shared across nodes (KPN SPEC §163, §445).

Not a node, because ownership is not a stage in the stream — it is state several
stages read and write, whose final answer is only known when a track dies.
Not inside `TrackGallery`, because that would couple presence to `expand_gallery`,
a switchable feature.

**The registry *is* the tracker's state.** `FaceTrackerFunc` does not keep its own
`tracks_`/`inactive_` maps and mirror them in — it operates on the registry
directly. Two parallel copies could disagree, and every divergence would surface
as wrong presence windows, silently.

### Per-track state

```
Track
  first_seen : double            set once, at creation
  last_seen  : optional<double>  UNSET while on screen; set to the last
                                 on-screen timestamp when the face is lost
  actor      : optional<int>     set when a posterior crosses the threshold
  belief     : {actor_idx -> accumulated_logodds}   Bayesian, not a tally
  embedding  : Embedding         running directional mean, for association
```

`last_seen` carries the entire liveness state. Unset = on screen; set = went off
at T. No separate missing-frames counter, no expired flag — the optional *is* the
state machine, and it subsumes the current two-pool split (`tracks_` = unset,
`inactive_` = set).

### Lifecycle

```
face detected, no match          → new track, first_seen = t, last_seen = unset
actor identified                 → update belief; set actor when threshold crossed
face lost                        → last_seen = t_last_on_screen  (stays revivable)
face seen again, embedding match → last_seen = unset  (same track continues)
tick(t), t - last_seen > timeout → emit to aggregator, DELETE the entry
```

A presence window is `[first_seen, last_seen]`. Nothing else.

**Interior gaps are claimed; the trailing cool-down is not.** A face lost at t₁
and re-acquired at t₂ within the timeout never closed its track, so the actor is
present across `[t₁, t₂]` — correct, since someone briefly occluded or off-camera
has not left the scene. But a track that dies ends at `last_seen`, not at the
moment of death. That asymmetry is what removes the old `extinction_sec`
over-claim.

**Reaping is a handoff, not a deletion into a holding pen.** The dead track goes
to the result aggregator immediately and the registry drops it, so the registry
holds only live tracks and its size is bounded by concurrent on-screen faces.

### Interface

```
TrackRegistry
  tick(timestamp)                                 ← FaceTrackerFunc, every frame
  candidates() -> span<Track&>                    → all live tracks
  create(timestamp, embedding) -> track_id
  mark_seen(track_id, timestamp, embedding)       → updates mean, clears last_seen
  mark_lost(track_id, last_on_screen_timestamp)
  on_vote(track_id, actor_idx, posterior)         ← IdentityMatcherFunc
  owner(track_id) -> optional<actor_idx>          → TrackGallery
  on_track_dead : callback(DeadTrack)             → ResultSinkFunc
  flush()                                         ← at EOF
```

`candidates()` returns **one pool**; `last_seen` tells the caller whether IoU
applies. There is no separate revival path — matching a dormant track is ordinary
inter-frame association.

`tick()` advances the clock so dead tracks are reaped independently of detection
activity; without it a track only dies when some *other* face happens to appear.

### Locking

The tracker mutates registry state across a frame's association pass, so that
pass holds the lock for its duration (a `frame_scope()` handle). Every other
caller's operations must be individually atomic. A single `std::mutex` over the
whole registry is the right start — contention is a few small updates per frame
against per-frame work measured in GPU milliseconds.

Two cases constrain the API:

- `owner()` is a **read-modify-read** in disguise: `TrackGallery` calls it while
  `IdentityMatcher` may be voting on the same track. Tally and verdict must be
  read under one lock as a snapshot, or a track can be both unowned and owned
  within a single promotion decision.
- `on_vote()` arrives downstream of the tracker's `tick()` for the same frame, so
  a vote may land after the clock moved on. **Rule: a vote for a known track
  always lands on its tally, regardless of clock.** Only reaping is clock-driven.
  A vote for an already-reaped track is dropped and **counted** — a nonzero count
  means the timeout is shorter than the matcher's lag.

`on_track_dead` fires from inside `tick()` while the frame lock is held, so the
callback must not re-enter the registry. Keep it to a push onto the aggregator's
storage.

## AR-016 — EOF flush

**Depends on:** AR-012.

`flush()` emits every still-live track through the same callback, closing at
`last_seen` if set and the final tick timestamp otherwise. Idempotent, leaving the
registry empty; the sink's `written_.exchange(true)` guard
(`result_sink_node.hpp:66`) shows the shape.

Must run on **every** termination path that produces output. Not SIGTERM during
opportunistic runs (DP-004) — those push no partial result, so there is nothing
to flush.

Without it a film ending mid-shot silently drops its closing cast, which looks
like a recognition miss rather than a bookkeeping bug.

## AR-014, AR-015 — Contradiction rules

**Depends on:** AR-012, AR-025.

| Condition | Meaning | Action |
|---|---|---|
| Belief on one track swaps A → B | `track_id` carried across a viewpoint change onto a different person | Close at `last_seen`, open a new track for B at the swap frame |
| Two **live** tracks owned by one actor | One person split in two, or an identity attached to the wrong track | Treat as a detected cut: reset affected state, re-associate on embedding |

The second makes identity a **third cut detector**, independent of histogram and
TransNetV2, firing where those failed. Detect it via a reverse index
`actor_idx → live track_ids`, so the condition is caught on the update that
causes it rather than by scanning.

Both counted and reported — the rates measure how often tracking is silently
wrong, which nothing currently reveals.

## AR-007, AR-008 — Tracker on one pool

**Depends on:** AR-012, AR-024.

`FaceTrackerFunc` is constructed with the registry and uses it as state; its
`tracks_`/`inactive_` maps and the cross-cut revival branch collapse into one
pool keyed on `last_seen`. Per frame: `tick()`, association over `candidates()`,
then `create`/`mark_seen`/`mark_lost`.

`track_alpha` becomes **frame-dependent** — normal frames use the tuned blend,
frames flagged `is_cut`/`is_scene_boundary` drop toward embedding-only.

## AR-024 — Probability space everywhere

**Depends on:** AR-023. **Blocks:** AR-007, AR-018, AR-021, AR-025.

Cuts across tracker, matcher and expansion, so it lands with the registry work
rather than after it. Retires `track_max_embed_dist`, `cut_revive_sim`,
`expand_novelty_sim`, `expand_track_spread_max`.

Enforcement is a **static grep check** for bare cosine outside a tagged
`EXCEPTION` — a unit test cannot prove absence across a codebase.

## AR-025 — Bayesian accumulation

**Depends on:** AR-023, AR-024.

Log-odds per candidate actor, added per frame. `on_vote()` is an *update*, not an
increment.

**The independence problem must be handled explicitly.** Consecutive frames are
highly correlated; naive accumulation drives the posterior to certainty on what is
effectively one observation. Preferred mitigation: update only on sufficiently
novel observations, reusing the diversity buffer's existing judgement rather than
inventing a second one. The registry should receive already-discounted evidence.

## AR-017 — Claims carry belief and route

**Depends on:** AR-012, AR-025. `DeadTrack` carries posterior plus how it was
identified (live / deferred / pooled).

## AR-018 … AR-021 — Expansion, deferred pass, clustering

**Depends on:** AR-012, AR-024, AR-026.

Ordering within the group: AR-018 (banded store) → AR-019 (annex) → AR-020 (TBI
queue + deferred pass) → AR-021 (clustering).

AR-021 needs the temporal cannot-link constraint from track extents, so it cannot
start before AR-012. The annex must be a **contiguous matrix** with promotions
appended (AR-026), not a list.

**Output timing changes:** the sink can no longer finalise at EOF — the deferred
pass runs after and may add windows (IR-003).

## AR-022 — Unidentified capture

**Depends on:** AR-020. Unidentified = TBI entries surviving the deferred pass.
Context crops opt-in behind `--dump-unidentified-crops`.

## AR-001 … AR-004 — Detection and backpressure

**Depends on:** nothing (AR-002, AR-011); AR-004 blocks AR-003.

- **AR-002** — `min_face_px` stays **40** (VR-013 measured it end to end) but must
  be expressed in original resolution rather than decoded-frame space. The value
  is already right in `config.hpp`; the change is the coordinate space.
- **AR-011** — feed TransNetV2 at native rate; derive the dedup window from
  source fps rather than the hardcoded `0.04 s`.
- **AR-004** — backpressure. `kMaxFaces` (`identity_matcher_node.hpp:133`)
  currently **throws**; channel capacities of 16 (`main.cpp:204-207`) were sized
  against ≤10 faces/frame. Must block on bytes in flight, not item counts.
- **AR-003** — remove `max_faces`. **Gated on AR-004**, not a follow-up to it.

## AR-026, AR-027 — GEMM and scale

**Depends on:** nothing to start. The annex CPU loop
(`identity_matcher_node.hpp:159-162`) moves into the GEMM path.

---

# Gallery

## GR-004 — Model binding — **DONE**

**Depended on:** nothing. Landed before any measurement work, as intended.

Stamp = model basename + SHA-256 of the ONNX, written as the `/embedder` group at
build time (`gallery_builder.cpp`, `sae_gallery.save_gallery_hdf5`) and verified
at load in `scene_analyze`, `scene_preview`, the `sae_kpn` matcher binding,
`replay.py`, `optimize.py` and `movienet_eval.py`. Mismatch is a hard error naming
both sides, with no bypass. Embedding dumps carry the same stamp, since a replay
has no live embedder to check against.

Unstamped legacy galleries **warn loudly and proceed** rather than failing:
unknown is not known-bad, and hard-failing every pre-existing gallery would turn
the check into something people disable. `--require-gallery-stamp` /
`SAE_REQUIRE_GALLERY_STAMP=1` promotes that to a hard error — measurement runs
should set it. `scripts/stamp_gallery.py` re-binds an existing gallery without
re-embedding, so the warning state is cheap to leave.

Cross-model similarities are meaningless but *look* plausible — this fails
silently and expensively, and it would corrupt every measurement taken during the
rest of this work.

## GR-003 — Coverage reporting

**Depends on:** nothing. Surface what calibration already computes and discards
(`kHistBins = 200`): zero-image actors, under-referenced actors, dedup counts,
and the intra/inter PDFs.

## GR-006 … GR-008 — Provenance tiers

**Depends on:** AR-019. Tier per embedding (baked / harvested / confirmed);
harvested persisted but flagged; bell-curve outlier check
(`EXCEPTION: AR-024`).

---

# Integration

## IR-004, IR-005, IR-007, IR-008 — Audio signature

**Depends on:** nothing. **Fully independent — no existing pipeline file is
touched.** Best candidate for concurrent work.

Implement server spec §3 exactly. Audio decode is a second stream from the
already-linked FFmpeg. Media < 120 s: no signature, no offset. Emit and honour
the `v1:` prefix.

The golden-vector fixture is shared with the plugin repo and runs on CPU, so the
one place two implementations must agree bit-for-bit is verifiable in CI.

## IR-001 … IR-003 — Truth file

**Depends on:** AR-017 (belief), AR-020 (output timing).

Windows carry belief and route; `extraction.*` gains `extinction_sec` and
`gallery_scope`; `anneal_sec` removed. All breaking → **one** coordinated
`schema_version` bump with IR-004 (SR-003).

---

# Validation

## VR-005 — Minimum face size study

**Depends on:** nothing. Standalone Python, no C++ contact. **Done** — knee at
24–32 px. It measures the embedder with alignment held perfect, so it bounds the
answer from below rather than setting it; AR-002's floor comes from **VR-013**,
which sweeps input resolution end to end and lands at 40 px.

## VR-013 — Cross-source identification probe

**Depends on:** `sae_embed` exposing `detect()`, `align_face()`, `embed_crop()`
and the gallery calibration — it drives the shipped C++ rather than reimplementing
it, which is what VR-005 could not do.

Gallery from one recording, probes from another, sweeping the probe's **input
resolution before the detector**, so detection and landmark regression degrade
with the frame. `experiments/xsource/`.

**Findings.** Holding 90% of the plateau needs ~50 px end to end against VR-005's
~22 px; `min_face_px` 40 is right and 32 would admit faces in the falling region.
FPI is 0.0% at every scale — resolution loss goes entirely to TBI, never to a
wrong name. The ceiling is **cross-view, not resolution**: everyone matches
themselves within a recording (0.55–0.85) and collapses across two (0.14–0.45),
and only the subject with frontal *gallery* references identified reliably — so
the lever is gallery pose coverage (`docs/pose-expansion.md`), not a better
landmark model. Averaging SCRFD's NMS-discarded landmark estimates lifts
cross-clip TPI 41% → 49% for one forward pass.

**Open.** Four identities and one shoot, so the shape is the result and the
absolute rates are not. Both clips hold all four people, so there is no
out-of-gallery class and the 10×-weighted out-of-cast misID is untested — holding
one identity out of the gallery would fix that.

## VR-014 — Audio-signature offset recovery

**Depends on:** `sae_audio` exposing `compute_signature()` and
`signature_from_mono()` — it drives the shipped C++, as VR-013 does, so the
thing measured is the thing that ships.

`scripts/validation/test_audio_offset.py` over
`tests/fixtures/audio/superhero_offset_200s.flac`: 200 s of public-domain film audio
(the same SuperHero clips the replay fixtures use), long enough for a 120 s
window to slide past the ±600-frame search cap. The slide itself is numpy here
on purpose — matching belongs to the consumer, so writing it out keeps this a
test of the signature rather than of somebody's matcher.

**Findings.** Alignment is a solved problem here: the offset is the nearest frame
in every in-cap trial, worst error **46 ms against a 500 ms budget**, and 46 ms is
the quantisation floor — offsets are whole 92.88 ms frames, so no correct answer
can be worse. The `runtime/2` anchor's factor of two holds through real trimmed
files, and out-of-cap offsets and unrelated content are both declined.

**The score is where the slack is, and it costs a tier rather than accuracy.** It
tracks sub-frame misalignment — 0.94–0.99 near a frame boundary, 0.69–0.73 at
half a frame — so two thirds of correct alignments miss the server's 0.85 `audio`
threshold and land in `loose`. UT-108 measures the fix rather than proposing one:
±1 frame of slack in the score returns all 40 to `audio` (min 0.906) with false
matches unmoved at 0.12–0.16, costing 81 ms of the budget. See
[`SPEC.md`](SPEC.md) IR-004 — the score is normative in the server spec, so the
change is theirs to make.

**Open.** One source, one language, one era of recording. The shape (offset exact,
score set by sub-frame phase) should hold generally, but the absolute scores are
this fixture's.

## VR-001 — Dump audit

**Depends on:** nothing. Read-only investigation: confirm the HDF5 dump preserves
everything needed to reconstruct tracks deterministically, including the
park/revive path. **Prerequisite for the CI strategy**, since T2 replay is how
most of AR-007 … AR-022 is verified.

## VR-006 … VR-009

**Depends on:** their subjects landing. VR-009 (posterior calibration holds)
depends on AR-025 and is what stops the Bayesian accumulation being decoration.

---

# Withdrawn from the old plan

The phase structure, the `--presence-mode {frame,track}` flag, and "Phase 2 —
retune `anneal_sec`/`extinction_sec`". Those constants are withdrawn rather than
retuned; comparison against old behaviour uses recorded reference output instead
of a second live code path.