dtourolle/scene-actor-extraction
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: software spec, requirements register, and implementation plan

Browse Source 
Adds the requirements baseline for the pipeline redesign:

- SPEC.md — software requirements with Current/Gap deltas per item, so the
  document doubles as a work list.
- requirements.md — stable flat IDs (AR/DP/IR/GR/VR) with parent traces,
  priorities, statuses, and a per-requirement verification plan. Replaces the
  thematic A1..E8 scheme, which had already produced an A1a and an out-of-order
  E6; IDs are now permanent and never reused.
- IMPLEMENTATION-PLAN.md — phased work.

The central change is AR-012: presence follows track extent rather than
per-frame recognition, so a window starts when an actor appears rather than
when the recogniser first succeeded. anneal_sec and extinction_sec are
withdrawn rather than retuned — a track that survives its own gaps leaves them
nothing to do.

Verification is shaped by CI running on an N100 with no dGPU: the existing
HDF5 dump makes everything downstream of embedding replayable on CPU, which
covers the bulk of the redesign.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
dtourolle
2026-07-30 17:32:44 +02:00
co-authored by Claude Opus 5
parent 458116f118
commit a5299daf6e
3 changed files with 2105 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/IMPLEMENTATION-PLAN.md
+627
View File
@@ -0,0 +1,627 @@
# Implementation plan
Companion to [`SPEC.md`](SPEC.md). Sequences the gaps identified there into
phases, with dependencies and acceptance criteria.
Ordering principle: **land the things that make measurement trustworthy before
the things that require measurement.** The central change (A6) invalidates every
tuned constant in `config.hpp`, so anything that could corrupt a retune must be
fixed first, and the retune itself gates the release.
| Phase | Contents | Gated on | Parallelisable |
|---|---|---|---|
| 0 | D3 model binding, E1 dump audit | — | with 4 |
| 1 | TrackRegistry + A6 track-extent presence | 0 | — |
| 2 | Retune (A6 constants), E4 | 1 | — |
| 3 | A8 unidentified capture, D2 gallery report | 0 | with 1, 2 |
| 4 | C2 audio signature | — | with everything |
Phases 3 and 4 touch disjoint code from 1 and 2 and can proceed alongside them.
---
## Phase 0 — De-risk measurement
Small, self-contained, and prerequisite to trusting any number produced later.
### 0.1 Gallery/model binding (D3)
Stamp the embedder identity into the gallery at build time; verify at load.
- `gallery_builder` writes the embedder model identity (filename + hash of the
ONNX, or an explicit version string) into the gallery file.
- Every consumer — `scene_analyze`, `replay.py`, the optimizer — checks it at
startup against the embedder it is about to use.
- Mismatch is a **hard error naming both sides**, never a warning.
*Why first:* cross-model cosine similarities are meaningless but look entirely
plausible. A retune against a mismatched gallery would produce numbers that are
wrong and undetectably so. This is the cheapest insurance in the document.
**Acceptance:** a deliberately mismatched gallery/model pair fails at startup
with a message naming both; a matched pair is unaffected. Covered by a test in
`tests/test_gallery_store.cpp`.
### 0.2 Dump audit for track-aware replay (E1)
The dump captures state at `EmbeddedSceneFrame` — *upstream* of tracking. After
A6, presence depends on tracker output, so replay must be able to reconstruct
tracks exactly as the live pipeline would.
- Confirm `bbox`, `landmarks`, `confidence`, `is_cut`, `is_scene_boundary` and
frame timestamps are sufficient to re-run `FaceTrackerFunc` deterministically.
- Specifically verify the **cross-cut park/revive path** (A4) is reproducible: it
depends on `is_cut` and on last-frame embeddings, both of which should be
present — confirm rather than assume.
- If anything is missing, add it and bump `schema_version` in
[`scripts/optimizer/SCHEMA.md`](../scripts/optimizer/SCHEMA.md).
**Acceptance:** for one film, tracks reconstructed from the dump are identical
(same `track_id` partitioning of the same faces) to those from a live run.
This equivalence test is the foundation of Phase 2 and should be kept as a
regression test.
---
## Phase 1 — TrackRegistry and track-extent presence (A6)
The central change. Presence moves from "frames where the actor was recognised"
to "extent of tracks the actor owns".
### 1.1 Design: TrackRegistry as a shared resource
**Decision:** track ownership lives in a `TrackRegistry` object that is *external
to the dataflow network*, not in a node and not inside `TrackGallery`.
This is idiomatic for this codebase rather than a workaround:
- Node functors are already constructed outside the network and passed in by
reference — `main.cpp:186-207` builds `ftracker_fn`, `tracker_fn`, `sink_fn` as
stack objects and `ObjectNode` wraps them ("the object must outlive the node",
KPN SPEC §371).
- KPN provides `SharedResource<T>` (`external/KPN/shared_resource.hpp`) precisely
for state shared across nodes, and shared resources can be registered with a
network for reporting (KPN SPEC §163, §445).
Why not the two alternatives:
- **Not a node.** Ownership is not a stage in the stream — it is state that
several stages read and write, at different points, with the final answer only
known at EOF. Modelling it as a node would force ownership to be decided at a
single point in the flow, which is exactly what it cannot be.
- **Not inside `TrackGallery`.** Ownership is already computed there for
expansion, so putting presence there too would avoid duplication — but it
couples presence semantics to `expand_gallery`, a switchable feature. Turning
expansion off would silently revert A6.
**Ownership and lifetime:** the registry is created in `main` and handed to every
node that needs it as a `std::shared_ptr<TrackRegistry>`. Nodes hold their own
`shared_ptr`, so lifetime is guaranteed by refcount rather than by the
"object must outlive the node" convention — no ordering assumption between
network teardown and the registry's destruction.
**Per-track state:**
```
Track
first_seen : double set once, at track creation
last_seen : optional<double> UNSET while on screen; set to the last
on-screen timestamp when the face is lost
actor : optional<int> set when a posterior crosses the threshold
belief : {actor_idx -> accumulated_logodds} Bayesian, not a tally (A9)
embedding : Embedding running directional mean, for association
```
`last_seen` carries the entire liveness state. Unset means *on screen now*; set
means *went off screen at T*. There is no separate "missing frames" counter and
no expired flag in the registry — the optional is the state machine.
**Lifecycle:**
```
face detected, no match to an existing track
→ new track, first_seen = t, last_seen = unset
→ begin embedding
actor identified
→ record vote; set actor once the ownership rule fires
face lost
→ last_seen = t_last_on_screen (track stays alive, revivable)
face seen again, embedding matches a track with last_seen set
→ last_seen = unset (same track continues)
tick(t) where last_seen is set and t - last_seen > timeout
→ track is DEAD: emit it to the result aggregator, delete the entry
```
**A dead track is passed to the result aggregator.** Reaping is a handoff, not a
deletion into a holding pen: the reaped track — `first_seen`, `last_seen`, owning
actor, vote tally — is pushed downstream the moment it dies, and the registry
drops it. The registry therefore contains **only live tracks**, and its size is
bounded by concurrent on-screen faces rather than growing with the film.
That gives a clean division: the registry answers "who is on screen now and which
tracks are still revivable"; the aggregator accumulates finished presence. Neither
needs the other's state.
A face appearing after the timeout starts a genuinely new track with a new
`track_id` — correct, since past the re-acquisition window there are no grounds
to assert continuity. An actor who leaves for half an hour and returns gets two
windows rather than one spanning their absence.
The registry thus has exactly three states, all implied by `last_seen`: on screen
(unset), revivable (set, within timeout), and gone (emitted and removed). There is
no "expired but retained" state to reason about.
**Closing a track *is* the presence assertion.** This is the property the design
turns on. There is no later stage where presence gets decided, reconciled or
merged — the act of closing a track emits exactly one complete, immutable claim:
*this actor was on screen from a to b*. It is produced once, never revised, and
carries everything needed to justify it (the vote tally that named the actor).
Several things follow, which is why it is worth naming explicitly:
- **Presence is append-only.** The output is a stream of finished claims, so a
partial result is a *prefix* of the full result, not a corrupted version of it.
- **Each claim is independently checkable.** A wrong window can be traced to one
track and its votes, rather than to an emergent interaction between three
timeout constants.
- **Nothing downstream needs to be clever.** The aggregator groups claims by
actor and writes them out. It holds no state machine of its own, which is
precisely why `SceneTrackerFunc` disappears.
Compare the current design, where presence is inferred at the end from a pile of
per-frame detections via two gap-bridging constants, and no single moment is "the
decision". That indirection is the source of both the late-start bug and the
untunability.
An actor's presence window is simply `[first_seen, last_seen]` of each track they
own. Nothing else.
**This removes annealing entirely.** `anneal_sec` (35.5 s) and `extinction_sec`
(57.4 s) exist only because presence is currently assembled from *isolated
accepted frames*, which are full of holes — both constants are gap-bridging
patches over that. Under this model a track survives its own gaps by
construction: a face lost and re-acquired by embedding match is the **same
track**, so there is no second window to merge and nothing to anneal. Presence
continuity is inherited from track continuity rather than reconstructed from it.
This is also what makes camera cuts fall out for free. A shot/reverse-shot sets
`last_seen` on the cut and unsets it on the next matching detection; the extent
never breaks. The existing cross-cut park/revive machinery
(`face_tracker_node.hpp:154-172`, `cut_revive_sim`) is exactly this mechanism
already, applied only to cuts — the model generalises it to every disappearance,
which is why cuts stop needing a special case.
**One timeout, not three.** The only surviving tunable is how long a track stays
revivable after the face is lost. It does two jobs at once, and both are wanted:
- **Gap absorption.** A face lost at `t₁` and re-acquired at `t₂` within the
timeout never closed its track, so the window runs straight through — the actor
**is claimed present across `[t₁, t₂]`**. That is correct: someone who turns
away, is briefly occluded, or is off-camera while the shot cuts to whoever they
are talking to has not left the scene.
- **Identity continuity.** The re-acquisition is only accepted on an embedding
match, so the gap is bridged on evidence that it is the *same person*.
That second point is the substantive improvement over `anneal_sec`, which merged
windows purely on elapsed time and could therefore stitch together two different
people. Same smoothing, now evidence-gated.
**The asymmetry that keeps this honest:** interior gaps are claimed, the trailing
cool-down is not. A track that dies ends its window at `last_seen` — the last
frame the face was actually seen — not at the moment of death. So the timeout
buys gap-smoothing without over-claiming the tail.
**The documented credits overshoot is fixed by this asymmetry, not by scene
detection.** `extinction_sec` kept an actor *active* for 57 s after their last
detection, actively emitting presence into the closing credits (the Downton Abbey
recall collapse, `lvface-deep-dive.md`). A window ending at `last_seen` never
enters the credits at all, because nothing was seen there.
**Cuts and scene boundaries are association hints, not presence events.** Both
signals say the same thing to the tracker: *spatial continuity is broken — stop
trusting IoU, associate on embedding similarity instead.* Neither closes a window.
| Signal | Meaning | Effect |
|---|---|---|
| `is_cut` (histogram) | Camera-angle change within a scene | Weight association toward embedding |
| `is_scene_boundary` (TransNetV2) | Different scene | Same, more strongly |
This is what the tracker's park/revive path already does for cuts
(`face_tracker_node.hpp:154-172`): a post-cut detection is matched on raw cosine
similarity to a parked track's last-frame embedding, with IoU out of the picture.
Generalising it makes `track_alpha` (the spatial/embedding cost weight)
frame-dependent rather than constant — normal frames use the tuned blend, flagged
frames drop toward embedding-only.
An actor genuinely continuing across a boundary is therefore *kept*, which is
correct; one who does not reappear simply times out and closes at `last_seen`.
This makes `--scene-detect` **load-bearing for presence correctness**, not the
opt-in extra it is today (`config.hpp`, default off). Two consequences to decide
in Phase 2: whether it becomes default-on despite its dense-decode cost, and what
the degraded behaviour is when it is off — presumably the timeout alone, which is
the current situation and carries the known overshoot.
It follows that the timeout **is** a presence knob and **cannot** simply be made
generous: it is precisely "how long an absence do we tolerate before calling it a
departure". Too short fragments one continuous appearance into several windows;
too long absorbs a genuine exit-and-return into a single window claiming presence
the actor did not have. It replaces both `anneal_sec` and `extinction_sec` and
inherits their tuning burden — this is the constant Phase 2 must fit.
`track_max_frames_missing` and `cut_inactive_max_frames` already serve this role
in the tracker and should be reconciled into it. Note the units differ: the
tracker counts *frames* missing while windows are in *seconds*; whichever side
owns the timeout should own the conversion, so `sample_fps` changes cannot
desynchronise them.
**Ownership rule** (per SPEC A9): each track carries accumulated **log-odds per
candidate actor**, updated as frames arrive. Ownership is "posterior exceeds
threshold", not "≥ N accepted frames" — so `votes` in the state sketch above is a
`{actor_idx → accumulated_logodds}` map, and `on_vote()` is an *update*, not an
increment.
Consequences for the registry:
- Every similarity reaching it is already a probability (A9); the registry never
sees a raw cosine.
- Correlated-frame discounting (A9) applies at update time. The registry should
take the already-discounted evidence rather than deciding the discount itself —
that judgement belongs with the diversity buffer that identifies novel poses.
- Ownership is established at **first crossing**, not deferred to track death.
**Two contradiction rules the registry must enforce** (SPEC A6). Both exist
because the same underlying fault — a missed camera or scene change — shows up in
identity space, and both are detectable *online*:
| Condition | Meaning | Action |
|---|---|---|
| Belief on one track swaps A → B | `track_id` carried across a viewpoint change onto a different person | Close the track at `last_seen`, open a new one for B at the swap frame |
| Two **live** tracks owned by the same actor | One person split into two tracks, or an identity attached to the wrong one | Treat as a detected cut: reset the affected tracking state and re-associate on embedding |
The second is the more interesting: it makes identity a **third cut detector**,
independent of the histogram and TransNetV2, firing exactly where those failed. A
cut subtle enough to evade pixel-based detection is not necessarily subtle in
identity space.
Both must be **counted and reported** (SPEC D2) — the rates are a direct measure
of how often tracking is silently wrong, which nothing currently reveals.
Implementation note: "two live tracks owned by the same actor" is a cheap check
because the registry already holds every live track and its belief. Maintain a
reverse index `actor_idx → live track_ids` and the condition is detected on the
update that causes it, not by scanning.
**The registry *is* the tracker's state.** `FaceTrackerFunc` does not keep its own
`tracks_`/`inactive_` maps and mirror them into a registry — it is constructed
with `std::shared_ptr<TrackRegistry>` and operates on it directly. One copy of the
track set, one owner of liveness.
This matters beyond tidiness. Had the tracker kept private maps and *reported*
into a parallel registry, the two could disagree — the tracker expiring a track
the registry still thinks alive, or vice versa — and every such divergence would
surface as wrong presence windows, silently. Merging them makes that class of bug
unrepresentable rather than merely tested-against.
It also subsumes the tracker's existing two-pool split: `tracks_` becomes the
tracks with `last_seen` unset, `inactive_` becomes those with it set. Same
structure, one map, and the cross-cut park/revive path becomes the general
re-acquisition path rather than a special case.
**Interface:**
```
TrackRegistry
// tracker-facing: state it owns and mutates
tick(timestamp) ← FaceTrackerFunc, every frame
candidates() -> span<Track&> → all live tracks; last_seen tells
the caller whether IoU applies
create(timestamp, embedding) -> track_id
mark_seen(track_id, timestamp, embedding) → updates mean, clears last_seen
mark_lost(track_id, last_on_screen_timestamp)
// matcher-facing
on_vote(track_id, actor_idx, posterior) ← IdentityMatcherFunc (accepted frames only)
// reader-facing
owner(track_id) -> optional<actor_idx> → TrackGallery (expansion confirmation)
// output: dead tracks are pushed out as they are reaped
on_track_dead : callback(DeadTrack) → ResultSinkFunc
flush() ← at EOF: emit all live tracks, then clear
```
`DeadTrack` carries `first_seen`, `last_seen`, the owning `actor_idx` (or none),
and the vote tally — everything the aggregator needs, with no back-reference into
registry state.
**Every track must be closed at EOF.** `flush()` emits every still-live track
through the same callback, closing each at `last_seen` if set and at the final
tick timestamp otherwise. This is not a tidy-up detail: a film almost always ends
with faces on screen, and those tracks have not timed out, so without an explicit
flush they are simply never emitted — the closing scene's actors disappear from
the output. That failure is silent and looks like a recognition miss rather than
a bookkeeping bug.
Requirements:
- `flush()` is idempotent and leaves the registry empty; calling it twice emits
nothing the second time. The sink's existing `written_.exchange(true)` guard
(`result_sink_node.hpp:66`) shows the shape.
- It must run on **every** termination path that produces output, not just clean
EOF — the `eof` sentinel, and the early-exit paths (`--end-sec`, decode error,
user interrupt) if those still write results.
- Finalisation goes through the same code path as a natural death, so a track
closed by EOF is indistinguishable in form from one closed by timeout.
- Deliberately *not* covered: SIGTERM during opportunistic runs (SPEC B4). Those
push no partial result at all, so there is nothing to flush — the item stays
pending and restarts. Flush is for runs that produce output.
**Acceptance:** a clip ending mid-shot yields a window for the on-screen actor
whose `end` equals the final frame timestamp. This is a specific test, not an
incidental one.
The tracker's association step reads `candidates()` — **one pool, not two**. A
track with `last_seen` unset was seen last frame, so IoU is meaningful; one with
`last_seen` set is dormant and matched on embedding alone. There is no separate
revival path: matching a dormant track is ordinary inter-frame association, and
the property falls out of the embedding comparison rather than being a mechanism.
The step then calls `create`/`mark_seen`/`mark_lost`. Reaping happens in `tick()`.
Since the tracker mutates registry state across a frame's association pass, that
pass needs to be atomic as a unit — a `frame_scope()` handle holding the lock for
the duration is cleaner than making each accessor independently locked and hoping
the composite is safe. This is the one place where per-call atomicity is *not*
sufficient.
`tick()` advances the clock so dead tracks are reaped independently of detection
activity — without it the registry only learns about time when something is
detected, and tracks would only die when some *other* face happened to appear. It
is called once per sampled frame whether or not that frame had detections.
**Locking.** Nodes run concurrently and do not coordinate, so the registry is
responsible for its own consistency. Two granularities apply: the tracker's
per-frame association pass holds the lock for its whole duration (`frame_scope()`
above), while every other caller's operations must be individually atomic.
Two cases constrain the API shape:
- `owner()` is a **read-modify-read** in disguise: `TrackGallery` calls it to
decide promotion while `IdentityMatcher` may be concurrently voting on the same
track. Tally and verdict must be read under one lock as a snapshot — not "read
tally, release, decide" — or a track can be both unowned and owned within a
single promotion decision.
- `on_vote()` for a frame arrives from `IdentityMatcher`, downstream of the
tracker's `tick()`/`mark_seen()` for that same frame. A vote may therefore land
after the clock has moved on. Rule: a vote for a known track always lands on
that track's tally, regardless of clock position. Only reaping is clock-driven.
A vote for a track already reaped is dropped and **counted** — a nonzero count
means the timeout is shorter than the matcher's lag, which is a real
misconfiguration and should not fail silently.
- The `on_track_dead` callback fires from inside `tick()`, which the tracker calls
while holding the frame lock. The callback must therefore not re-enter the
registry, or it self-deadlocks. Keep it to a push onto the aggregator's own
storage; anything heavier belongs downstream of that.
A single `std::mutex` over the whole registry is the right starting point:
contention is a handful of small updates per frame against per-frame work
measured in GPU milliseconds. Anything finer needs a profile, not an assumption.
### 1.2 Wiring
- `FaceTrackerFunc` is constructed with the `shared_ptr<TrackRegistry>` and uses
it as its state — its own `tracks_`/`inactive_` maps go away. Per frame:
`tick()`, then association over `candidates()`, then
`create`/`mark_seen`/`mark_lost`. Its `tracks_`/`inactive_` split and the
cross-cut revival branch both collapse into one pool keyed on `last_seen`.
- `IdentityMatcherFunc` reports accepted-frame votes. It keeps its existing
per-frame acceptance logic unchanged — A6 changes what is *done* with
acceptances, not how they are decided.
- `TrackGallery` replaces its internal confirmation counter with
`registry.owner()`, so ownership is computed **once**.
- `ResultSinkFunc` becomes the result aggregator: it receives dead tracks via the
callback and groups them by actor. The per-frame timestamp collection and
gap-merging at `result_sink_node.hpp:123-147` is deleted. **No annealing pass**
— a dead track already *is* a window.
- `SceneTrackerFunc` (`scene_tracker_node.hpp`) is the extinction-timer state
machine keyed on `actor_idx`. Under this model it has nothing left to do: its
entire job was keeping actors alive across detection gaps. Expect to delete it
from the network rather than adapt it.
### 1.3 Comparing against current behaviour
The old path is not worth preserving behind a flag. It is not a variant of the
new one — it is a different pipeline shape (`SceneTrackerFunc` present, annealing
in the sink, two extra constants), so keeping both runnable means maintaining two
sink implementations and a node that otherwise gets deleted.
Compare against **recorded output** instead: keep the current binary's results for
the validation corpus as reference JSON, and diff the new pipeline against them.
That gives the same A/B for Phase 2 without carrying dead code through it.
**Acceptance:**
- `tests/test_face_tracker.cpp` extended: a lost-then-re-acquired face continues
the **same** track and yields one unbroken window; a cut does the same; a face
re-appearing *after* the timeout yields two separate tracks and two windows;
a two-actor conflict resolves to the majority and increments the conflict
counter; a single-frame track yields a zero-length window.
- A registry test: reaping emits exactly once per track; `flush()` at EOF emits
every live track and nothing twice; a vote landing on a reaped track is dropped
and counted.
- Contradiction tests: a belief swap A→B closes one window at `last_seen` and
opens a second starting at the swap frame, with no overlap and no blended
window; two live tracks converging on one actor trigger a re-association and
increment the counter.
- A concurrency test hammering `tick`/`mark_seen`/`mark_lost`/`on_vote` from
multiple threads against `owner()`, under TSan. The vote-tally-plus-verdict read
is the case to target — correct only if atomic as a unit.
- On a known film, every actor's first window starts no later than in the recorded
reference output, and strictly earlier for at least one — the late-start bug
this change exists to fix.
---
## Phase 2 — Delete two constants, tune the rest
`anneal_sec` (35.5) and `extinction_sec` (57.4) are **replaced, not discredited**.
They exist to answer X-Ray's scene-level question — "is this actor in this scene"
— by holding windows open across cuts, and they answer it with elapsed time
because that was the only signal available at the sink. Phase 1 answers the same
question with better evidence: an embedding-matched re-acquisition, plus a true
scene boundary to stop at. So they are deleted along with `SceneTrackerFunc`, and
their *job* transfers to the re-acquisition timeout rather than disappearing.
What actually needs tuning:
- **The ownership posterior threshold** — replaces both `prob_threshold` (0.754)
as a presence decision and `expand_min_anchor_frames` (3) as a vote count. A
track is owned when its accumulated posterior for an actor crosses this (A9).
A single false accept can no longer create a window on its own, so the
operating point should sit lower than the old per-frame threshold.
- **The correlated-frame discount** (A9) — whatever form it takes, it is a fitted
quantity and belongs in the sweep. It directly controls how fast belief
accumulates along a track, so it trades against the ownership threshold and
cannot be tuned separately.
- **The re-acquisition timeout** — reconciled from `track_max_frames_missing` (5)
and `cut_inactive_max_frames` (5). **This is the successor to both `anneal_sec`
and `extinction_sec` and carries their tuning burden.** It decides how long an
absence is absorbed into a presence window versus treated as a departure, so it
trades recall (bridging real gaps) against precision (claiming presence during
a genuine exit) directly. Current values are 5 *frames*, inherited from a
tracker-continuity role; as a presence constant it is likely to want a much
larger value, and should be swept over seconds rather than nudged.
- **`scene_detect`** — now load-bearing for presence (§1.1), so the sweep must
cover *with* and *without*, and `scene_threshold` (0.60) becomes a presence
constant rather than a diagnostic one. A missed boundary reintroduces the
overshoot; a spurious one truncates a scene's cast early.
Method per SPEC E3: DE over the validation corpus, objective micro-F1, but
precision and recall logged at every evaluation and printed at the optimum. X-Ray
recall is a face-vs-cast-in-scene ceiling, so unconstrained F1 pushes thresholds
down chasing unreachable recall and trades away real precision. Pick the operating
point deliberately from the trajectory.
**This resolves SPEC Open Question 1 by construction rather than by measurement.**
The question was whether `extinction_sec` survives alongside track extents; the
answer is that the mechanism it patched no longer exists.
One thing to watch: the search space is now 3 knobs instead of 3 gap-constants, but
they are *not* independent — a longer re-acquisition timeout means longer tracks,
which means more frames to clear `expand_min_anchor_frames`. Sweep jointly.
**Acceptance:** `anneal_sec` and `extinction_sec` removed from `config.hpp` and
`Config`; new constants committed with the trajectory and the precision/recall
trade-off documented in the manner of `rep4-optimizer-results.md`; results
compared against the recorded reference output from 1.3.
---
## Phase 3 — Diagnostics
Independent of Phases 1–2; can run in parallel.
### 3.1 Unidentified-track capture (A8)
Today only *promoted* mugshots are dumped (`expand_debug_dir`) — the successes.
This captures the failures.
- Flag `--dump-unidentified <dir>`.
- For every track never identified: all embeddings, track metadata (`track_id`,
first/last timestamp, frame count, per-frame bbox and confidence), and **the
best similarity achieved and which actor it was against**.
- That last field is the point of the feature: it separates "actor missing from
the gallery" from "actor present but scored below threshold" — a gallery
coverage problem versus a threshold problem.
- Crops are **opt-in** (`--dump-unidentified-crops`); embeddings + metadata are
the default. Crops for every unidentified track across a library is a lot of
disk.
- When crops are enabled, store both the 112×112 aligned crop *and* a wider
**context crop** for a bounded number of representative frames per track. The
aligned crop serves diagnostics; the context crop serves the human-in-the-loop
association capability ([`../../SPEC.md`](../../SPEC.md) §4), where someone has
to actually recognise the person — which a tightly-cropped, geometrically
normalised face often makes impossible.
- Naturally expressed against `TrackRegistry`: unidentified = tracks with no
owner at EOF.
**Acceptance:** on a film with a known out-of-gallery face, that track appears in
the dump with its near-miss actor and similarity.
### 3.2 Gallery build report (D2)
Surface what the calibration already computes internally but discards:
- actors with zero usable images (a silent recall ceiling);
- actors below the 5-embedding threshold for positive pairs;
- near-duplicate references removed;
- the fitted calibration **and the intra/inter similarity distributions behind
it** (`gallery_calibration.hpp` histograms these at `kHistBins = 200` and
throws them away — persist them).
**Acceptance:** a build report written alongside the gallery; the intra/inter
PDFs are recoverable for inspection.
### 3.3 Resolve the prior (SPEC Open Question 4)
SPEC A9 asks for a prior of `intra/(intra+inter)`; the shipped default is
`match_prior = 0.5` (use the calibrated sigmoid directly). These disagree. With
3.2 landed the real value is known, so: either adopt it, or document 0.5 as a
deliberate override with the reason. Cheap once the distributions are persisted.
---
## Phase 4 — Audio signature (C2)
Largest self-contained chunk; no dependency on any other phase.
- Implement [`JRay-public-server/SPEC.md` §3](../../JRay-public-server/SPEC.md)
**exactly**: 120 s centred on the midpoint, mono 11025 Hz, 4096/1024 Hann STFT,
300–3000 Hz, 32 log bins, peak bin + 2-bit energy class, one byte per frame,
base64.
- Audio decode is a second stream from the FFmpeg dependency already linked for
video (`ffmpeg_decoder.hpp`) — not a new dependency.
- Emit in the truth file → **`schema_version` bump**, coordinated with
`jRay/SPEC.md` and the plugin. Neither can be changed unilaterally.
- **Golden-vector cross-check is a hard requirement**, not a nicety: two
independent implementations of the same DSP chain will drift. Fixture: a short
audio file with its expected signature, checked into both repos and asserted in
both test suites (SPEC Open Question 3, resolved this way — a shared fixture
rather than a shared implementation, since the coupling cost of the latter
exceeds the benefit).
- This pipeline **produces only**. Matching and offset recovery stay consumer-side.
**Acceptance:** pipeline and plugin produce identical signatures for the same
file; the golden-vector test passes in both repos.
---
## Deferred
Deployment (SPEC Part B) is deliberately excluded from this plan. B3 (on-demand
service) and B4 (opportunistic worker) are packaging concerns over a stable core,
and the core is about to change under A6. `service-conversion.md` remains the
design of record; it should be executed once Phase 2 fixes the constants, so the
installer is not shipping values that are about to be replaced.
One exception worth pulling forward if convenient: the **temp-file cleanup fix**
in `run_from_jellyfin.py` (SPEC B4) is small, independent, and a named
prerequisite for the worker.
---
## Risks
| Risk | Mitigation |
|---|---|
| A6 widens presence and precision drops more than recall gains | Phase 2 leaves the operating point explicit; both modes runnable via flag for direct comparison |
| Retune produces a worse optimum than the current constants | Trajectory is logged; the old operating point stays available. A6 is behaviourally correct even if the metric disagrees — decide deliberately, do not let the metric silently veto it |
| Track-ID collisions merge two people into one extent | Conflict counter (1.1) makes the rate visible; `expand_track_spread_max` already guards the expansion side |
| Registry races produce non-deterministic presence between identical runs | Every operation atomic as a unit (1.1); TSan test in Phase 1 acceptance. A race here is especially costly — it would surface as irreproducible optimizer scores in Phase 2, where it would look like metric noise rather than a bug |
| X-Ray metric blindness | `methodology.md` documents the known failure (scene-union hid out-of-cast FPs). Any metric change gets checked for the same class of blindness |
| Two audio implementations drift | Golden vectors in both repos (Phase 4) |
docs/SPEC.md
+1210
View File
File diff suppressed because it is too large Load Diff
docs/requirements.md
+268
View File
@@ -0,0 +1,268 @@
# scene-actor-extraction — requirements register
Stable IDs for every requirement in [`SPEC.md`](SPEC.md), which holds the prose.
This file is the **authoritative list**; the CI gate reads its denominators from
here (see [`../../SPEC.md`](../../SPEC.md) §6).
**IDs are permanent.** A withdrawn requirement is marked `Withdrawn` and its
number is never reused — renumbering is what produces orphan TRACES tags. This
register replaces the earlier thematic `A1…E8` scheme, which had already produced
an `A1a` and an out-of-order `E6`.
Tag code with `// TRACES: AR-012 | SR-002`.
| Type | Scope |
|---|---|
| `AR` | Algorithm — the extraction pipeline itself |
| `DP` | Deployment — how it runs |
| `IR` | Integration — contracts with other components |
| `GR` | Gallery — building and maintaining actor references |
| `VR` | Validation — parameter studies and benchmarks |
| `UT` / `IT` | Unit / integration tests |
Status: `Done` · `In Progress` · `Planned` · `TBD` · `Withdrawn`
---
## Algorithm (AR)
| ID | Requirement | Traces to | Priority | Status |
|---|---|---|---|---|
| AR-001 | Detect faces in sampled frames; emit bbox, confidence, 5-point landmarks in original pixel space | SR-002 | High | Done |
| AR-002 | Minimum face size 66×66 px, expressed in **original** resolution (decoupled from `dense_scale`) | SR-002 | High | Planned |
| AR-003 | No fixed per-frame face cap — crowd scenes must not lose background cast | SR-002 | Medium | Planned |
| AR-004 | Backpressure: unbounded faces/frame absorbed by slowing, never by dropping or throwing | SR-002 | High | Planned |
| AR-005 | Align to 112×112 via ArcFace 5-point similarity transform | SR-002 | High | Done |
| AR-006 | 512-d L2-normalised embeddings, batched | SR-002 | High | Done |
| AR-007 | Associate detections by IoU + embedding, with **frame-dependent** weighting | SR-002 | High | In Progress |
| AR-008 | One track pool keyed on `last_seen`; no separate revival path | SR-002 | High | Planned |
| AR-009 | Camera-cut detection (histogram) as an association hint | SR-002 | High | Done |
| AR-010 | Scene-boundary detection (TransNetV2) as an association hint | SR-002 | Medium | In Progress |
| AR-011 | **Every model is fed the input it was trained for** — cost reduced by running less often, never by degrading one inference | SR-002 | High | Planned |
| AR-012 | Presence follows **track extent**, not per-frame recognition | **SR-002** | High | Planned |
| AR-013 | `last_seen` optional state machine; window ends at last sighting, never after | SR-002 | High | Planned |
| AR-014 | Belief swap A→B terminates the track and starts a new one | SR-002 | Medium | Planned |
| AR-015 | Two live tracks owned by one actor ⇒ treat as a detected cut, re-associate | SR-002 | Medium | Planned |
| AR-016 | All tracks closed at EOF — a film ends with faces on screen | SR-002 | High | Planned |
| AR-017 | Every presence claim carries its belief and identification route | SR-002 | High | Planned |
| AR-018 | Per-subject embedding store with banded admission (novel enough, safe enough) | SR-005 | Medium | Planned |
| AR-019 | Per-film gallery annex from owned tracks; acquires the non-frontal views TMDB lacks | SR-005 | Medium | In Progress |
| AR-020 | Deferred re-identification of unknown tracks against the final expanded gallery | SR-005 | High | Planned |
| AR-021 | Cluster unknown tracks into one entity per person, under temporal cannot-link constraints | SR-005 | Medium | Planned |
| AR-022 | Capture still-unidentified tracks: embeddings, metadata, **context crops** for human review | §4 | Medium | Planned |
| AR-023 | Fit sigmoid calibration from intra/inter similarity distributions | SR-002 | High | Done |
| AR-024 | **Always the calibrated probability, never a raw cosine** — exceptions recorded | SR-002 | High | Planned |
| AR-025 | Per-track Bayesian accumulation in log-odds, with correlated-observation discounting | SR-002 | High | Planned |
| AR-026 | All similarity computed as GEMM, including annex and deferred pass | SR-001 | High | In Progress |
| AR-027 | Throughput acceptable for **arbitrary** gallery size | SR-001 | High | Planned |
## Deployment (DP)
| ID | Requirement | Traces to | Priority | Status |
|---|---|---|---|---|
| DP-001 | One analysis core; modes are front-ends and must not fork pipeline logic | PR-004 | High | Done |
| DP-002 | Batch CLI over one title | PR-004 | High | Done |
| DP-003 | On-demand resident service with bounded, observable queue | PR-004 | Medium | Planned |
| DP-004 | Opportunistic/idle mode: external trigger, hard stop, implicit re-queue | PR-004 | Medium | Planned |
| DP-005 | Native installer, no Docker; Fedora + Arch | PR-004 | Medium | Planned |
| DP-006 | Background incremental gallery refresh on a timer | PR-003 | Medium | Planned |
## Integration (IR)
| ID | Requirement | Traces to | Priority | Status |
|---|---|---|---|---|
| IR-001 | Emit the JRay truth format as sibling `.jray.json` | SR-003 | High | Done |
| IR-002 | Windows carry belief + route; `extraction.*` carries `extinction_sec`, `gallery_scope` | SR-003 | High | Planned |
| IR-003 | Output written **after** the deferred pass, not at EOF | SR-003 | High | Planned |
| IR-004 | Compute the audio signature exactly per server spec §3 | SR-003 | Medium | Planned |
| IR-005 | Golden-vector fixture shared with the plugin repo to prove bit-exactness | SR-003 | High | Planned |
| IR-007 | Media < 120 s: emit no signature, apply no sync offset — identical rule in both producers | SR-003 | Low | Planned |
| IR-008 | Emit and honour the signature's own `v1:` version prefix | SR-003 | Low | Planned |
| IR-006 | Jellyfin round-trip: pull pending queue, push complete results only | SR-001 | High | Done |
## Gallery (GR)
| ID | Requirement | Traces to | Priority | Status |
|---|---|---|---|---|
| GR-001 | Build gallery from Jellyfin library cast, TMDB profile fallback | SR-001, SR-005 | High | Done |
| GR-002 | Incremental `--merge` refresh without re-embedding known actors | PR-003 | High | Done |
| GR-003 | Report coverage: zero-image actors, under-referenced actors, dedup, calibration PDFs | SR-001 | Medium | Planned |
| GR-004 | Stamp embedder identity into the gallery; **hard startup error** on mismatch | SR-001 | High | Planned |
| GR-005 | Gallery data never leaves the instance | **SR-005** | High | Done |
| GR-006 | Provenance tiers: baked / harvested / confirmed, distinguishable per embedding | SR-005 | High | Planned |
| GR-007 | Persist harvested embeddings **flagged and reviewable**, never silently equal to baked | SR-005 | Medium | Planned |
| GR-008 | Flag distributional outliers among an actor's references (poisoning guard) — `EXCEPTION: AR-024` | SR-005 | Medium | Planned |
| GR-009 | Human-confirmed associations persist and improve future extractions | §4 | Medium | TBD |
## Validation (VR)
| ID | Requirement | Traces to | Priority | Status |
|---|---|---|---|---|
| VR-001 | HDF5 post-inference dump at the embedded-frame boundary | PR-002 | High | Done |
| VR-002 | Replay drives the **real** KPN nodes, not a reimplementation | PR-002 | High | Done |
| VR-003 | Scoring: micro-F1 against X-Ray, precision/recall logged at every evaluation | PR-002 | High | Done |
| VR-004 | Reproducible validation corpus with ground truth | PR-002 | High | Done |
| VR-005 | Minimum face size study — TPI/FPI vs probe size, gallery held at native res | PR-002 | Medium | Planned |
| VR-006 | Re-tune `scene_threshold` once native-rate decode lands | PR-002 | Low | Planned |
| VR-007 | Expansion band, clustering threshold, and deferred-pass ablation | PR-002 | Medium | Planned |
| VR-008 | Gallery scaling benchmark — throughput vs gallery size | PR-002 | Medium | Planned |
| VR-009 | Verify accumulated posteriors are calibrated against held-out tracks | PR-002 | High | Planned |
---
## Verification strategy
**CI runs on an Intel N100 with no discrete GPU.** That is a hard constraint on
how each requirement can be verified, and it shapes the test design rather than
merely limiting it.
Four tiers, in decreasing order of preference:
| Tier | Runs in CI | What it covers |
|---|---|---|
| **T1 — CPU unit** | Yes | Pure logic: registry state machine, belief accumulation, clustering, band admission, calibration maths |
| **T2 — Replay** | Yes | Real pipeline nodes driven from an HDF5 fixture — no GPU, no video |
| **T3 — CPU inference** | Yes, slowly | ORT CPU provider over a handful of frames; smoke tests only |
| **T4 — GPU** | **No** | Throughput, TRT engines, large-gallery GEMM |
**T2 is the reason this is workable.** The HDF5 dump (VR-001) captures state
after decode → detect → align → embed and before tracking and matching, so
everything downstream — which is where nearly all of the new design lives — is
cheap CPU maths replayable from a fixture. Tracking, presence windows, belief
accumulation, expansion, deferred re-identification and clustering are all
verifiable on an N100 at full fidelity, not in miniature.
That was already true for the optimizer. It now doubles as the CI strategy, which
is a strong argument for keeping the dump schema honest (VR-001) and for the
replay driving the *real* nodes rather than a reimplementation (VR-002).
**Small committed fixtures are required.** A few HDF5 dumps covering the awkward
cases — a cut, a belief swap, two live tracks converging, a film ending
mid-track, an unknown track that only resolves after expansion — are worth more
than a large corpus, and they are small enough to commit.
**T4 requirements cannot pass in CI, and the gate must not pretend otherwise.**
For these, CI verifies that a test *exists and is tagged*, not that it passes;
the run happens on a GPU host, nightly or manually, and reports separately. A
requirement whose only evidence is a test that never executes should be visible
as such rather than counted as covered.
| Requirement | Tier | Note |
|---|---|---|
| AR-001, AR-005, AR-006 | T3 | Smoke only — correctness of detection/embedding is a model property, not ours |
| AR-002 | T2 | Size filtering is arithmetic on dumped bboxes |
| AR-003, AR-004 | T1 + T4 | Backpressure logic is unit-testable; saturation behaviour needs real load |
| AR-007 … AR-017 | **T2** | The core of the redesign — fully replayable |
| AR-018 … AR-022 | **T2** | Expansion, deferred pass, clustering: all post-embedding |
| AR-023 … AR-025 | T1 | Calibration fit and log-odds accumulation are pure maths |
| AR-026, AR-027 | T4 | GEMM throughput and scaling — GPU host only |
| DP-* | T1 + manual | Lifecycle logic unit-tested; install paths are manual |
| IR-001 … IR-003 | T1 | Serialisation against a golden truth file |
| IR-004, IR-005 | **T1** | Audio signature is CPU DSP — the golden-vector fixture runs anywhere, which is precisely why it is the right cross-repo check |
| GR-001 … GR-005 | T1 + T3 | Gallery assembly is I/O and bookkeeping; embedding is T3 smoke |
| GR-006 … GR-008 | T1 | Tiering and outlier detection operate on stored embeddings |
| VR-* | Out of CI | Studies are run deliberately and their results committed as documents |
**One consequence worth stating:** AR-027 (arbitrary gallery scale) is
structurally unverifiable on the CI host. It needs a GPU host and a synthetic
large gallery, so it is the requirement most likely to silently regress. Its
benchmark (VR-008) should run on a schedule rather than on demand.
### Fixtures — precomputed inference, pulled by CI
The N100 cannot run inference at any useful rate, so **inference output is
precomputed on a GPU host and consumed by CI as data.** This converts most of
what looks like GPU work into pure CPU replay.
| Fixture | Contents | Size | Storage |
|---|---|---|---|
| **Edge-case dumps** | ~6 short clips (30–60 s), one per awkward behaviour | ~1 MB each | **Committed in-repo** |
| **Corpus dumps** | Full-length titles from the validation corpus | ~30 MB each | Pinned artifact, fetched by checksum |
| **Synthetic gallery** | Random unit-norm embeddings, fixed seed | small | Generated at test time |
| **Golden truth files** | Expected output for each edge-case dump | KB | Committed |
| **Audio golden vectors** | Short WAV + expected signature | KB | Committed, **shared with the plugin repo** |
Edge-case dumps are small enough to commit, and being in-repo means they version
with the code that reads them. Corpus dumps are pulled by pinned checksum from
the artifact store rather than committed, since they are large and change only
when the dump schema does.
**Generation must be reproducible and versioned.** A script, run on a GPU host,
regenerates every fixture from source clips; it is re-run when the VR-001 schema
version bumps. A fixture whose provenance is unknown is worse than no fixture,
because it will be trusted.
> **The limitation that must stay visible:** replay fixtures freeze upstream
> behaviour. A test driven from a dump verifies AR-007 onward *given those
> embeddings* — it cannot detect a regression in detection, alignment or
> embedding, because those produced the fixture. Nothing in CI can. That gap is
> covered only by the T3 smoke test and the scheduled GPU run, and it should not
> be papered over by a high replay-coverage number.
### Per-requirement verification plan
| ID | Tier | Test asserts | Edge cases to cover |
|---|---|---|---|
| AR-001 | T3 | Detector returns plausible boxes on a known frame | — smoke only |
| AR-002 | T2 | Faces below 66 px (original res) are dropped | Exactly at threshold; with `dense_scale` 0.5 — the interaction that motivated the requirement |
| AR-003 | T2 | No cap applied; a 40-face frame yields 40 | Crowd frame |
| AR-004 | T1 | Saturated input blocks rather than drops or throws | Bounded queue at capacity; **byte-based** limit with large crops; SIGTERM mid-block |
| AR-005 | T1 | Known landmarks → expected 112×112 warp | Landmarks near frame edge; degenerate/collinear points |
| AR-006 | T3 | Embeddings are unit-norm | Batch smaller than, equal to, larger than `embed_batch_size` |
| AR-007 | T2 | Association picks the right track | Two faces crossing paths; one leaving frame as another enters |
| AR-008 | T2 | One pool; dormant tracks match on embedding, not IoU | Dormant track whose old bbox overlaps a *different* new face — must not match on position |
| AR-009/010 | T2 | Cut/boundary shifts weighting toward embedding | Cut with same people; cut with all-new people |
| AR-011 | T1 | TransNetV2 receives native-rate frames | Source at 24/25/30 fps — dedup window derived, not assumed |
| AR-012 | **T2** | Window spans full track extent, not first recognition | Actor recognised only at track end — window must still start at `first_seen` |
| AR-013 | **T2** | `last_seen` set/unset; window ends at last sighting | Gap just under vs just over timeout; reappearance after timeout → two windows |
| AR-014 | T2 | Belief swap closes one window, opens another | No blended window; no overlap at the swap frame |
| AR-015 | T2 | Two live tracks on one actor trigger re-association | Counter increments |
| AR-016 | **T2** | Every track closed at EOF | Film ending mid-shot — window ends at final frame, not dropped |
| AR-017 | T1 | Claim carries posterior and route | Deferred and pooled routes distinguishable |
| AR-018 | T1 | Band admits only within bounds | At each bound exactly; store never admits below lower bound |
| AR-019 | T2 | Promotion only when all three signals quiet | Cut mid-track blocks promotion |
| AR-020 | **T2** | Unknown resolved after expansion | Track failing at minute 12, resolved at EOF — the ordering-independence claim |
| AR-021 | T2 | Clustering merges same person, respects cannot-link | **Temporally overlapping tracks never merge**; measure how many merges the constraint rejects |
| AR-022 | T1 | Context crops retained, bounded per track | Track running for minutes |
| AR-023 | T1 | Sigmoid fit on synthetic separable data | Too few positive pairs → `valid=false`, fallback engages |
| AR-024 | **Static check** | No bare cosine outside a tagged `EXCEPTION` | Grep-based; this is the invariant's enforcement |
| AR-025 | T1 | Log-odds accumulate; correlated frames discounted | 30 identical frames must **not** reach the certainty of 30 diverse ones |
| AR-026 | T1 + T4 | GEMM path produces same result as reference loop | Equivalence on small input in CI; throughput on GPU host |
| AR-027 | **T4** | Throughput at 10²…10⁵ actors | Scheduled, not on-demand |
| IR-001/002 | T1 | Serialised output matches golden file | Zero-length window; actor with many windows |
| IR-003 | T1 | Output written after deferred pass | Not at EOF |
| IR-004/005 | **T1** | Signature matches golden vector bit-for-bit | **Media < 120 s → no signature**; identical result in both repos |
| GR-004 | T1 | Mismatched embedder → hard startup error | Error names both sides |
| GR-008 | T1 | Outlier flagged among an actor's references | Injected poisoned embedding detected |
| VR-009 | T1 | Posterior calibration holds | A 0.99 posterior is wrong ~1% of the time on held-out tracks |
Three of these are worth singling out because they verify claims that would
otherwise be assertions: **AR-012** (window starts at `first_seen` even when
recognition comes late) is the entire point of the redesign; **AR-020** (a track
failing mid-film resolves at EOF) is the claim that ordering stops mattering; and
**AR-025** (30 identical frames ≠ 30 diverse ones) is what stops the Bayesian
accumulation from being decoration.
---
## Withdrawn
| ID | Requirement | Reason |
|---|---|---|
| — | `anneal_sec` window merging | Superseded by AR-012/AR-013: a track survives its own gaps, so there is nothing to anneal |
| — | `extinction_sec` actor keep-alive | Superseded by AR-013: windows end at last sighting, which is what this over-claimed |
Both were deleted rather than retained at zero — a field naming a mechanism the
pipeline no longer has is actively misleading (see `SPEC.md` A6.6).
---
## Notes on coverage
- **VR-*** traces to PR-002 (scene-granularity answers) rather than to a system
requirement: parameter studies are single-repo work serving accuracy, and this
is correct rather than a gap.
- **PR-005** (leak nothing) has no `AR`/`DP` row. It is satisfied *structurally*
by SR-004 and GR-005 — the server holds no binary, the gallery never leaves the
instance — not by any component doing something. It cannot be verified by
pointing at code, and it dies the moment either prohibition is relaxed.