scene-actor-extraction/docs/SPEC.md

# scene-actor-extraction — software specification

Status: **draft**. Consolidates the requirements for the extraction pipeline, its
deployment modes, and its integration contracts.

**This is a software spec implementing the system spec at
[`../../SPEC.md`](../../SPEC.md).** Requirements that span more than one repo —
presence semantics, schema-version coordination, gallery locality, identity keys
— are owned there. Where this document restates one, the system spec governs.

This is a *requirements* document, not a design doc. Most of what follows is
already built; each requirement therefore carries a **Current** / **Gap** note so
the document doubles as a work list. Where a requirement is fully met, the gap
reads "none".

Related documents, which this spec references rather than restates:

| Document | Owns |
|---|---|
| [`../../jRay/SPEC.md`](../../jRay/SPEC.md) | Truth-file format and the Jellyfin plugin's read API |
| [`../../JRay-public-server/SPEC.md`](../../JRay-public-server/SPEC.md) | Jmanifest exchange format, cut matching, audio signature (§3) |
| [`scripts/optimizer/SCHEMA.md`](../scripts/optimizer/SCHEMA.md) | Embedding-dump HDF5 layout |
| [`service-conversion.md`](service-conversion.md) | Idle-GPU worker design (one deployment mode of §B) |
| [`methodology.md`](methodology.md) | X-Ray scoring methodology used by the optimizer |

---

# Part A — Algorithm requirements

The processing chain is a KPN dataflow network (`src/nodes/`). Requirements
below are ordered along that chain.

## AR-001 … AR-003 — Face detection

Detect faces in sampled video frames.

- Detector runs on frames sampled at `sample_fps`, not every decoded frame.
  The default (1.0) originates from I-frame decode speed, not from an accuracy
  requirement — it is a **cost knob and may be adjusted**. Under scene-scoped
  presence (SR-002) a lower rate still answers the question, but it lengthens the
  interval between samples and so weakens IoU-based association; sweep the two
  together (VR-002).
- **Minimum face size is 66×66 px**, expressed in **original video resolution**,
  not decoded-frame pixels. Stating it in original space decouples it from
  `dense_scale`: otherwise a 0.5 downscale silently doubles the effective
  threshold, and dense mode is exactly what scene detection uses.
  66 is a working estimate of where ArcFace embeddings stop being reliable, not a
  measured value — it should be replaced by the result of VR-005.
- Emits bounding box, detector confidence, and 5-point landmarks.
- Bounding boxes must be reported in **original video pixel space**. When
  `dense_scale < 1` downscales the decoded frame, coordinates are rescaled by
  `bbox_upscale` before leaving the pipeline.
- **No fixed cap on faces per frame.** `max_faces` (10, largest-first) is removed
  so crowded scenes do not systematically lose their background cast — which
  X-Ray credits as scene members (SR-002). See the backpressure requirement below;
  the cap is currently the only thing bounding per-frame cost, so it cannot be
  removed on its own.

**Current:** SCRFD-500MF via `face_detector_node.hpp`, thresholds in `config.hpp`
(`detector_conf` 0.5, `detector_nms` 0.4), `min_face_px` 40, `max_faces` 10.

**Gap:** `min_face_px` → 66 and re-expressed in original resolution; `max_faces`
removed, gated on backpressure (AR-004).

## AR-004 — Backpressure

Removing the per-frame face cap makes the number of faces entering the pipeline
unbounded and content-dependent — a crowd scene can produce an order of magnitude
more than a dialogue scene. The network must absorb that by **slowing down**, not
by dropping work or growing without limit.

- The embedder is the bottleneck and must exert backpressure upstream: when its
  input is saturated, the detector and decoder block rather than queue.
- Channel capacities are currently fixed at 16 (`main.cpp:204-207`) and were
  chosen against a bounded ≤10 faces/frame. They must be re-derived, and
  overflow must block rather than throw.
- `kMaxFaces` in `identity_matcher_node.hpp:133` **throws** when exceeded. With no
  cap upstream that becomes a crash on crowd scenes; it has to go or become a
  batching bound rather than an error.
- Memory is the real limit: faces carry 112×112 crops plus 512-float embeddings.
  Backpressure must engage on bytes in flight, not just item counts.

**Gap:** entire requirement. This is a prerequisite for removing `max_faces`, not
a follow-up to it.

## AR-005 — Face alignment and crop

Produce the exact input ArcFace expects.

- 112×112 BGR crop via the standard ArcFace 5-point similarity transform.
- Landmark order must match the SCRFD/ArcFace convention (left eye, right eye,
  nose, left mouth, right mouth).
- Alignment is the *only* geometric normalisation; no additional augmentation at
  inference.

**Current:** `align_face()` in `src/face_utils.hpp:9-22`, `cv::warpAffine` to
`{112, 112}`. **Gap:** none.

## AR-006 — Embedding

Generate a 512-d embedding per aligned crop.

- Output embeddings are **L2-normalised**, so cosine similarity is a plain dot
  product. Every downstream threshold assumes unit norm.
- Faces are embedded in batches of at most `embed_batch_size` to bound
  per-call latency.
- The embedding model is a build-time choice; the gallery must have been built
  with the *same* model (see GR-004).

**Current:** `embedder_node.hpp` + `face_embedder_engine.hpp`; default
LVFace-B_Glint360K. **Gap:** none.

## AR-007, AR-008 — Tracking

Link detections across frames into tracks representing one physical person.

Association cost combines three signals:

- **Spatial** — IoU between the candidate detection and the track's last box.
  Meaningful only for tracks seen in the immediately preceding frame.
- **Appearance** — similarity to the track's running directional mean embedding
  (averaged, then re-normalised to the unit sphere), **expressed as a
  probability** (AR-024), never as a raw cosine.
- **Weighting** — `track_alpha` interpolates the two, and is **frame-dependent**:
  on `is_cut` / `is_scene_boundary` (AR-009, AR-010) it drops toward embedding-only, because
  position carries no information across a viewpoint change.

**There is no separate re-acquisition mechanism.** A track whose face is lost sets
`last_seen` and stays a candidate for association until extinction; matching it to
a later detection is ordinary inter-frame tracking, not a distinct revival path.
The property falls out of the embedding comparison the tracker already does.

Consequences:

- **One candidate pool**, not an active set plus a parked set. `last_seen` alone
  distinguishes them, and it only affects whether IoU means anything — dormant
  tracks are matched on embedding, since time has passed and position is stale.
- The current cross-cut park/revive path (`cut_revive_sim`,
  `cut_inactive_max_frames`) is this same mechanism special-cased to cuts. It
  collapses into the general path.
- A track dies only by extinction: `last_seen` set for longer than the timeout.

**Current:** `face_tracker_node.hpp` maintains separate `tracks_`/`inactive_`
maps with distinct thresholds and a revival branch. **Gap:** unify into one pool
keyed on `last_seen`; make `track_alpha` frame-dependent; move association into
probability space (AR-024).

## AR-009 … AR-011 — Cut and scene-boundary detection

Two distinct signals, deliberately kept separate:

- **`is_cut`** — always on. Grayscale histogram correlation below
  `cut_threshold` flags an *intra-scene camera-angle change* (shot/reverse-shot).
- **`is_scene_boundary`** — opt-in (`--scene-detect`). TransNetV2 over a densely
  decoded, downscaled stream flags a *true shot/scene boundary*.

> **`is_scene_boundary` currently has no producer.** `grep -rn is_scene_boundary
> src/` finds no assignment anywhere: `SceneDetectorFunc` is a *terminal sink*
> (`main.cpp:298-300`, `kpn::out<>`) that writes `scenes.json` and never
> annotates the `Frame` flowing to the face pipeline. The field is therefore
> always `false`, and the dump column (`embedding_dump_node.hpp:38`) is a
> constant 0. Compounding it, `main.cpp:280` returns from the
> `--dump-embeddings` branch *before* the `scene_detect` branch at `:296`, so no
> dump-producing path even instantiates the detector.
>
> This makes AR-010 **not implemented**, not "in progress" — and it means a T2
> test of the frame-dependent `track_alpha` (AR-007) would **pass vacuously**,
> which is the worst possible failure for a verification gate. The fix is in the
> producer, not the schema: make `SceneDetectorFunc` a pass-through (or add a
> boundary annotator before the decimator) and add the scene branch to
> `dump_embeddings.cpp`. **No `schema_version` bump** — the column exists and
> merely stops being constant.
>
> Fixtures generated before the fix must be marked in provenance, since `0` is
> presently indistinguishable from "no boundary here".

Both feed AR-007 as **association hints**: they tell the tracker that spatial
continuity is broken and that association should weight embedding over IoU.
Neither ends a presence window (AR-012).

In dense mode the source decodes at `scene_decode_fps` (default 12) and a
decimator splits the stream: full-resolution sampled frames to the face pipeline,
downscaled dense frames to the scene detector
(`frame_source_node.hpp:63`). `sample_fps` is independent of this — the face
pipeline's rate is not derived from the scene detector's.

Boundary timestamps are keyed off each frame's real timestamp, so they stay
correct despite the reduced rate.

`dense_scale` must stay ≥ 0.5 on 1080p sources — it also shrinks what the face
detector sees.

### Every model gets the input it was trained for

A general rule, stated once here because it applies throughout: **models are fed
their expected input, not a cheaper approximation.** Where cost must come down,
it comes down by running the model less often or on fewer regions — never by
degrading what a single inference sees. A model run off-distribution produces
confident, plausible, wrong output, and the error is invisible without a study
that should not have been necessary.

Two places this is currently violated:

1. **`scene_decode_fps = 12` starves TransNetV2.** `kWindow` is 100 frames. At
   native 25 fps that window spans ~4 s; at 12 fps it spans ~8.3 s, so the model
   sees roughly half-speed motion over twice the temporal context it was trained
   on. **Requirement: feed TransNetV2 at the source's native frame rate**, so a
   100-frame window covers the duration the model expects. The
   "tolerates ~12fps" note in `config.hpp` describes a compromise, and the
   recorded margin is consistent with it — a non-boundary baseline at ~0.50 with
   real boundaries reaching only ~0.7+ is a compressed separation, not a healthy
   one.

2. **Hardcoded 25 fps in boundary dedup.** `scene_detector_node.hpp:138` merges
   boundaries closer than `0.04 s` — "~1 frame @25fps". **Requirement: derive
   this from the source's actual frame rate.**

Dense decode is the pipeline's cost driver, so (1) is not free. The cost is
accepted: the alternative is a boundary signal that steers association (AR-007) while
being quietly unreliable. `dense_scale` remains available as a spatial reduction,
since downscaling is a documented, understood degradation rather than a temporal
one the model has no defence against.

**Current:** histogram cut in the decoder; `scene_detector_node.hpp` for
TransNetV2. **Gap:** native-rate dense decode; framerate-derived dedup;
`--scene-detect` is default-off despite now feeding association.

## AR-012 … AR-017 — Track-level identity propagation — **CHANGED BEHAVIOUR**

> **Requirement:** when a face is identified, it is considered identified for the
> **entire track**.

A track is one physical person by construction (AR-007), so an identification
anywhere on the track is evidence about all of it. Presence windows must
therefore be derived from **track extents**, not from the subset of frames that
happened to match.

Required semantics:

- A track is **owned** by actor A once the accumulated posterior for A exceeds
  the ownership threshold (AR-025). This is a Bayesian accumulation over the track's
  frames, not a count of accepted frames — so a single decisive observation and a
  long run of marginal ones are distinguished rather than conflated.
- Once owned, the actor is present for the track's **full extent** —
  `[first_seen, last_seen]`, including frames where the face was turned,
  occluded, or simply scored below threshold.
- **A track with no owner emits nothing.** No posterior crossed, so there is no
  claim to make. Such tracks are captured for diagnosis (AR-022) but do not appear in
  output. See "Unidentified presence" below.
- **Every presence claim carries its belief.** The accumulated posterior that
  justified the claim travels with it — through the aggregator, into the truth
  file (IR-001), and onward. Confidence is not a serialisation concern bolted on at
  the boundary; the pipeline computes belief for every decision it makes (AR-025), so
  discarding it and emitting a bare interval throws away information that was
  free.

  A claim also carries **how it was identified** — live, deferred (AR-020), or
  pooled from a cluster (AR-021) — because those routes rely on different evidence.
  A deferred identification leans on harvested references rather than baked ones;
  a pooled one aggregates across tracks. The posterior already reflects the
  strength of each, but the provenance tells a consumer *why*, and tells us which
  route is producing errors when one is.

### Belief swap terminates the track

If a track is owned by A and the accumulated belief subsequently swaps to a
different actor B, **the track is terminated and a new one started for B**. The
old track closes normally at its `last_seen`; the new track begins at the frame
where the swap occurred.

The rationale is that a swap is not a correction — it is almost certainly a
**missed camera or scene change**, where the tracker carried one `track_id`
across a viewpoint change and began following a different person. Two genuinely
different individuals both accumulating a threshold-crossing posterior on the
same face is not realistic short of identical twins; a `track_id` spanning two
people is.

Treating it as a swap-and-continue would emit one window blending two people.
Treating it as a track boundary yields two windows that are each correct.

Ownership is therefore **established at first crossing, not deferred to track
death** — the first crossing is a real claim about the frames seen so far, and a
later contradiction ends that claim rather than revising it.

### Simultaneous ownership

If two actors cross the threshold on one track at effectively the same time, the
**highest posterior wins**. This should be rare — the swap rule above catches the
common form of the problem — and it must be counted and reported (GR-003) as a
track-ID collision indicator.

### Identity contradiction is a cut detector

If **two live tracks are both owned by the same actor**, at least one is wrong: a
person cannot be in two places at once. The cause is the same as the belief swap
— a missed camera or scene change that split one person into two tracks, or
attached an identity to the wrong one.

**Requirement: treat this as a detected cut.** Reset the affected tracking state
and re-associate on embedding, exactly as an `is_cut` signal would (AR-007, AR-009).

This gives a third cut-detection signal, derived from *identity* rather than
pixels, and it fires precisely where the pixel-based detectors failed — a cut
subtle enough for the histogram and TransNetV2 to miss is not necessarily subtle
in identity space. It is also self-correcting rather than diagnostic: the
contradiction is detectable **online**, the moment both tracks hold the belief,
not at output time.

A consequence worth noting: with this in place, overlapping windows for one actor
should be rare rather than routine, because the condition that produces them is
now caught and repaired while tracking. Any that survive to output indicate the
repair failed and should be counted.

### Unidentified presence — TBD

A track that is never owned is still *someone* on screen. Emitting it as
anonymous presence would let a consumer show "unidentified person", and would
give the human-in-the-loop association tool ([`../../SPEC.md`](../../SPEC.md) §4)
its work queue directly. Combined with the context crops of AR-022, it may also be
the path to recognising extras and background cast the gallery has no entry for.

**This is deliberately undecided.** It changes the truth format and invites
consumers to display something that may not be useful. Recorded here so the
option is not lost; not specified until the AR-022 debug output shows whether these
tracks are worth surfacing.

**Current:** presence is built from *per-frame accepted detections only*.
`identity_matcher_node.hpp:145-231` decides acceptance independently per face per
frame; `result_sink_node.hpp:123-129` collects the timestamps of accepted frames;
`result_sink_node.hpp:139-147` sets `win_start = ts_vec[0]`, the first accepted
timestamp. `track_id` is carried on `IdentifiedActor` but is used only for debug
output — it never gates or backfills a window. A window therefore starts when the
actor was first *recognised*, not when their track began.

**Gap — this is the main behavioural change in this spec.**

Design is settled in [`plan.md`](plan.md) under AR-012/AR-013: a
`TrackRegistry`, held by `shared_ptr` and used by `FaceTrackerFunc` as its state,
where each track carries `first_seen` plus an **optional `last_seen`** — unset
while on screen, set to the last on-screen timestamp when the face is lost, unset
again when a later detection associates to it. A track whose `last_seen` exceeds
the extinction timeout is reaped and **pushed to the result aggregator** as one
finished claim: *this actor was on screen from a to b*.

Consequences:

1. The tracker's `tracks_`/`inactive_` maps become one track set, distinguished
   only by whether `last_seen` is set; there is no separate revival path (AR-007).
2. Closing a track *is* the presence assertion — emitted once, complete, never
   revised. No later reconciliation stage exists.
3. **`anneal_sec` and `extinction_sec` are deleted, not re-tuned.** Both exist
   only to bridge gaps between isolated accepted frames; a track that survives
   its own gaps leaves them nothing to do. `SceneTrackerFunc` goes with them.
   What remains to tune is the **ownership posterior threshold**, the
   **correlated-frame discount** (AR-025), and the **extinction timeout**.
4. Every track must be closed at EOF. A film ends with faces on screen and those
   tracks have not timed out, so without an explicit flush the closing scene's
   actors are never emitted — a silent presence loss that looks like a
   recognition miss.
5. Zero-length windows (`start == end`, a single-frame track) remain possible;
   the truth format permits them, and `build_xray()` floors/ceils into integer
   seconds.
6. **Pull it out by the roots — including the published field.** `anneal_sec` is
   not just a constant: it appears in the truth schema
   ([`jRay/SPEC.md`](../../jRay/SPEC.md)), in the Jmanifest format and in the
   server's storage columns
   ([`JRay-public-server/SPEC.md`](../../JRay-public-server/SPEC.md)). Retaining
   it as a vestigial `0` would be worse than removing it: a field that names a
   mechanism no longer in the pipeline is actively misleading to anyone reading a
   manifest, and it would outlive everyone who remembers why it is zero. It goes
   from all three repos under one coordinated `schema_version` bump — the same
   bump IR-004 already requires, so there is exactly one breaking change, not two.

   Removal list: `Config::anneal_sec`, `Config::extinction_sec`,
   `SceneTrackerFunc` and its node wiring, the sink's annealing pass
   (`result_sink_node.hpp:139-147`), the truth-file field, the Jmanifest field,
   the server column, and the optimizer's parameter entries. Grep for both names
   and expect no survivors.

## AR-018 … AR-021 — Per-film gallery expansion and deferred identification

Three requirements that only make sense together, so they are specified together.

### The problem being solved

**TMDB headshots are overwhelmingly frontal.** Films are not. An actor is
recognised easily in the shots that resemble a publicity still and missed in
profile, three-quarter, low-angle, poorly lit, or partially occluded views — even
though those are most of their screen time.

So the pipeline's failures are dominated by **pose**, not by identity: the same
person the gallery knows, at an angle it does not. Both mechanisms below exploit
that.

### AR-018 — Per-subject embedding store

Every track maintains a running store of its own embeddings, **identified or
not** — the structure is the same for both.

An embedding is admitted only if its similarity to one already in the store falls
**inside a band**:

- **Upper bound** — too similar and it is redundant, teaching nothing the store
  already covers.
- **Lower bound** — too dissimilar and it may not be the same person at all;
  admitting it risks poisoning the store.

A starting band of roughly **0.90–0.95** is the working estimate, to be tuned
(VR-007). Note this is deliberately conservative compared to the current
`expand_novelty_sim` (0.55), which promotes embeddings *far* from the gallery —
much more aggressive, and much more exposed to admitting the wrong person.

Both bounds must be expressed as calibrated probabilities, not raw cosines (AR-024).

### AR-019 — Expansion of known actors

When a track is owned (AR-012), its store is promoted into a **per-film, in-memory
annex** for that actor, folded into best-of-N scoring alongside the baked
references. The annex does not persist.

Promotion requires **certainty that the span is one person**:

- the track is owned, with the belief stable (no swap, AR-012);
- no camera cut, scene boundary, or identity contradiction occurred within the
  span (AR-009, AR-010, AR-015) — all three signals must be quiet, not just the histogram cut as
  today;
- the band of AR-018 is satisfied.

The purpose is precisely to acquire the **non-frontal views TMDB lacks**, at a
confidence the gallery alone cannot supply.

### AR-020 — Deferred re-identification of unknown tracks

**This is what the expansion is for.**

- An unowned track, on extinction, is **not discarded**. Its embedding store,
  metadata and context crops (AR-022) move to a **to-be-identified (TBI) queue**.
- At **end of playback**, TBI entries are pooled (AR-021) and re-assessed against
  the *final* expanded gallery — which by then holds the pose-varied views
  harvested from the whole film.
- Entries that now cross the ownership threshold emit presence windows exactly as
  a normally-owned track would. Entries that still do not are the output of AR-022,
  and the work queue for human association
  ([`../../SPEC.md`](../../SPEC.md) §4).

The asymmetry this exploits: an actor confirmed frontally early in a film
contributes profile views to the annex, and a profile-shot track that failed at
minute 12 matches once the film is over. Ordering ceases to matter.

**This is cheap.** Embeddings are already computed; re-assessment is matching
against an in-memory annex — no decode, no detection, no embedding. The cost is
retaining unowned track stores until EOF.

### AR-021 — Pool unknown tracks before matching

Re-assessing each unknown track alone wastes the strongest evidence available.
A single track is short and pose-poor; the *same unknown person* usually appears
across many tracks throughout the film.

**Requirement: cluster the unknown tracks, treat each cluster as one identity,
and match the pooled cluster against the full gallery.**

Each unknown track's store is effectively a small gallery of one unnamed person.
Clustering merges the ones that are the same person, and the pooled result is a
far richer representation — many poses, lightings and expressions — which stands a
much better chance against the gallery than any constituent track.

**The evidence is also better-conditioned.** AR-025 warns that consecutive frames
within a track are highly correlated and must be discounted. Embeddings from
*different tracks* are far more independent: different scenes, angles, lighting.
Pooled cross-track evidence is therefore worth more per observation than
within-track evidence, and the discount should reflect that.

**Temporal exclusion is a free constraint.** Two tracks that **overlap in time
cannot be the same person** — the same fact AR-012 uses to detect missed cuts. This
is a cannot-link constraint on the clustering and it costs nothing to apply,
since track extents are already known. Use it: it prevents exactly the merge that
would otherwise pool two people who share the screen.

**Clustering must be conservative.** A wrong merge pools two people and then
mis-identifies *both*, across every track in the cluster — strictly worse than
leaving them separate. Prefer many small correct clusters to few large ambiguous
ones; the merge threshold is a calibrated probability (AR-024), swept in VR-007.

**Order of operations at end of playback:**

1. Cluster TBI tracks under temporal cannot-link constraints.
2. Pool each cluster's embeddings into one composite identity.
3. Match each cluster against gallery + annex (one GEMM, A10).
4. Cluster crosses threshold → **every member track** emits presence for that
   actor.
5. Cluster does not → it becomes one **unknown person** entity, not N orphan
   tracks.

That last point matters beyond recognition: it is the difference between asking a
human to label twelve disconnected faces and asking them to name one person who
appears in twelve places ([`../../SPEC.md`](../../SPEC.md) §4). It is also the
natural unit for anonymous presence, should that be adopted (AR-012, TBD).

### Consequences

- **The result aggregator cannot finalise at EOF-flush.** Owned tracks emit on
  death as before, but the TBI pass runs after, and may add windows. Output is
  written after re-identification completes, not when the last frame is read.
- **Expansion errors are now more costly.** A wrongly promoted embedding no
  longer affects only later frames — it is applied to every unknown in the film
  during the second pass. The AR-018 band is the guard, and its lower bound is the
  part doing that work.
- **Iteration is possible but unspecified.** A TBI entry that becomes identified
  could itself expand the gallery and enable further identifications — a fixpoint
  loop. Whether to iterate to convergence, run one pass, or bound the rounds is
  open (VR-007).

**Current:** `src/gallery/track_gallery.hpp` implements a per-track diversity
buffer with eviction biased to gallery-far poses, promotion gated on
`expand_novelty_sim` / `expand_track_spread_max`, cleared on `is_cut`. Wired at
`identity_matcher_node.hpp:227`, cleared at `:126`.

**Gap:** the band rule of AR-018 replacing the current novelty/spread gates; all
three quiet-signal conditions rather than only `is_cut`; probability space
throughout (AR-024); and the whole of AR-020 — the TBI queue, the deferred pass, and
deferring output until it completes.

## AR-022 — Unidentified-track capture

Persist everything needed to diagnose a miss — and everything a *human* would
need to resolve one.

Scope note: the TBI queue of AR-020 is **not** debug-only, because deferred
re-identification depends on it. What is behind a flag is the *persisted output*
for tracks that survive re-assessment still unidentified; retaining stores until
the deferred pass runs is unconditional.

For every track **still unidentified after the deferred pass (AR-020)**, store:

- all embeddings on the track,
- the aligned 112×112 crops,
- **context crops** — a wider region around the face than the 112×112 aligned
  crop, for a handful of representative frames per track;
- track metadata: `track_id`, first/last timestamp, frame count, per-frame
  detector confidence and bbox,
- the best similarity achieved and which actor it was against, so near-misses are
  distinguishable from faces with no gallery counterpart at all.

**Why the context crop is a separate artifact.** The 112×112 crop is optimised
for ArcFace: tightly cropped, geometrically normalised, and frequently
unrecognisable to a person out of context. A human deciding *who this is* needs
the surrounding shot — hair, costume, who they are standing next to. This
requirement exists to serve the human-in-the-loop association capability
([`../../SPEC.md`](../../SPEC.md) §4), where the user names the face the pipeline
could not.

Retain a bounded number of representative frames per track (largest/sharpest
detections), not every frame — an unidentified track can run for minutes.

This is the raw material for deciding whether a miss is a gallery coverage
problem (actor absent or under-represented) or a threshold problem (actor present
but scored below acceptance).

**Current:** only *promoted* mugshots are dumped, via `expand_debug_dir`
(`track_gallery.hpp`, guarded by `SAE_DEBUG`) — i.e. the successes, not the
failures. **Gap:** the whole requirement. Needs a flag (`--dump-unidentified
<dir>`), a per-track accumulator that survives until the track ends, and a
decision on retention: crops for a feature-length film are large, so default to
embeddings + metadata with crops opt-in.

## AR-023 … AR-025 — Calibration, probability space, and per-track Bayesian accumulation

**Every similarity in the pipeline is converted to a probability through the
sigmoid calibration before it is used or thresholded. No component compares raw
cosine values against a hand-set constant.**

This is a system-wide rule, not a detail of identity matching. Raw cosine
thresholds are unfalsifiable magic numbers that mean different things for
different models, different galleries, and different face sizes; a calibrated
probability means the same thing everywhere. It replaces `track_max_embed_dist`
(0.7), `cut_revive_sim` (0.50), `expand_novelty_sim` (0.55) and
`expand_track_spread_max` (0.60) with quantities that can be reasoned about
jointly.

### Per-frame evidence is a Bayesian update on the track

Once similarities are probabilities, a track's identity is not a vote count — it
is a **posterior accumulated across the frames of that track**. Each frame
contributes a likelihood that this face is actor A; the track's belief is updated
frame by frame, and ownership (AR-012) is "posterior exceeds threshold" rather than
"≥ N accepted frames".

Working in **log-odds** makes this an addition per frame:

```
logit(A | frames 1..n) = logit_prior(A) + Σ_i [ logit(A | sim_i) − logit_prior(A) ]
```

This is strictly better than counting accepted frames: a long run of marginal
observations and a single decisive one are no longer conflated, and a track that
accumulates weak contrary evidence can lose ownership rather than keeping it on a
stale count.

**The independence problem, which must be handled explicitly.** Consecutive
frames of one track are highly correlated — near-identical pose, lighting and
expression. Treating them as independent observations overcounts evidence
dramatically: thirty frames of the same face at the same angle is nowhere near
thirty independent measurements, and naive accumulation will drive the posterior
to certainty on what is effectively one observation.

Mitigations, in preference order:

1. **Update only on sufficiently novel observations.** The diversity buffer
   (AR-018…AR-021) already identifies which embeddings on a track are gallery-far and
   mutually distinct — precisely the more-independent ones. Reuse that judgement
   rather than inventing a second one.
2. **Discount correlated updates** by a per-frame weight below 1, fitted so the
   accumulated posterior is calibrated against held-out tracks.
3. **Cap total evidence per track**, the crude fallback.

Whichever is chosen, the accumulated posterior must be **validated against
ground truth** — a posterior of 0.99 should be wrong about 1% of the time. If it
is not, the independence handling is inadequate and the number is decoration.

### Fitting the calibration

Build a PDF of **intra-subject** similarity and a PDF of **inter-subject**
similarity across the gallery; the prior is `intra / (intra + inter)`.

- **Positive pairs** — same actor, different reference images.
- **Negative pairs** — all cross-actor pairs.
- Near-duplicate references are de-duplicated per actor first (similarity above
  `1 − 1e-7`), so the same image embedded twice cannot inflate the positive side.
- Actors with fewer than 5 distinct embeddings contribute negatives only — they
  cannot supply a meaningful positive pair.
- A Platt-style sigmoid `P(match) = σ(a·sim + b)` is fitted to the two
  distributions with class weights balancing the (heavily skewed) pos/neg ratio.
- The base rate enters as log-prior-odds: `P = σ(a·sim + b + log(p₀/(1−p₀)))`,
  with `p₀ = match_prior`.
- Acceptance is `P(match | sim, prior) > prob_threshold` (default 0.754, DE-tuned).
- **Fallback** when calibration is invalid (too few positive pairs): a hard cosine
  distance ceiling `match_threshold`, OR a ratio test — accept if
  `best/second < match_ratio` and `best_distance < match_ratio_ceil`.

**Current:** `src/gallery/gallery_calibration.hpp` implements the sigmoid fit,
dedup, eligibility filter, and prior adjustment; the fallback lives at
`identity_matcher_node.hpp:181-196`. **Gap:** the fit currently histograms the two
similarity distributions internally (`kHistBins = 200`) but does not *emit* them.
For this requirement to be inspectable, the intra/inter PDFs and the derived
prior should be written alongside the gallery, so calibration quality is
auditable rather than implicit. Also note the shipped `match_prior` default is
0.5 (use the calibrated sigmoid directly) rather than the gallery-derived
`intra/(intra+inter)` — reconcile: either compute and store it at gallery-build
time, or document 0.5 as a deliberate override.

## AR-026, AR-027 — Scale — performance must hold for arbitrary gallery size

**Requirement: analysis throughput must remain acceptable as the gallery grows
arbitrarily large.** Gallery size is set by the user's library, not by us: a
whole-library gallery spans every credited actor across every title
(GR-001), which is thousands today and grows monotonically as titles are added. A
design that is fast at 500 actors and unusable at 50,000 has a defect, not a
limitation.

### Similarity is a matrix multiply

Every embedding is unit-norm (AR-006), so cosine similarity is a dot product and
scoring a batch of faces against the gallery is one GEMM. At library scale that
is the only viable formulation — a per-pair loop is orders of magnitude off.

**All similarity computation goes through the GEMM path**, with no exception
justified by "this set is small". Three call sites:

1. **Baked gallery** — already GEMM (`sim_engine_->compute()`,
   `identity_matcher_node.hpp:143`, backend from `SAE_GEMM_BACKEND`). ✓
2. **Per-film annex** — currently a **CPU loop**
   (`identity_matcher_node.hpp:159-162`), justified in-comment by "tens of
   embeddings". AR-018…AR-021 invalidates that assumption: every owned track now
   contributes, so the annex grows with cast size and film length. It must move
   into the GEMM path — appended to the gallery matrix, or a second multiply.
3. **Deferred TBI pass (AR-020)** — the most GEMM-friendly operation in the
   pipeline: all TBI embeddings against the full gallery-plus-annex, offline,
   operands resident, no streaming. One large multiply, not a loop over entries.

This constrains AR-018…AR-021's implementation: the annex must be a **contiguous matrix**
with promotions appended, plus a parallel actor-index mapping — exactly the
`flat_emb_`/`flat_actor_` arrangement the baked gallery already uses.

### Scaling characteristics that must be known, not assumed

- **Throughput versus gallery size must be measured** (VR-008) and published. The
  useful output is the curve and the point where gallery scoring starts to
  dominate total runtime, not a single number.
- **Memory is a real ceiling.** 512 floats × 4 bytes = 2 KB per reference. At
  five references per actor that is ~10 MB per 1000 actors — comfortable at
  10,000 actors, worth planning for beyond.
- **Calibration cost grows quadratically.** AR-023 fits on cross-actor pairs, which
  is O(N²) in references. This is a gallery-build cost, not a per-title one, but
  it will bite first — sampling negatives rather than enumerating them is the
  obvious mitigation and should be specified before it becomes urgent.

### If GEMM stops being enough

Approximate nearest-neighbour indexing (IVF/HNSW) is the standard next step, and
it trades **exactness** for speed. That trade interacts badly with AR-023: an
approximate search returns approximate similarities, and a calibrated posterior
built on them is no longer calibrated. Not ruled out, but it requires
re-validating calibration against the approximation — so it is a later decision
with a real cost, not a drop-in.

**Gap:** annex GEMM path; the scaling benchmark (VR-008); negative-pair sampling in
calibration.

---

# Part B — Deployment requirements

The pipeline must support **multiple deployment modes over one core**. Modes
differ in *what triggers work* and *what constrains it*, not in what the analysis
does.

## DP-001 — Common core

- One analysis implementation, exercised identically by every mode. Modes are
  front-ends; none may fork the pipeline logic.
- Backend selection (ORT/TRT, CUDA/ROCm/CPU) is a **build-time** choice
  (`SAE_INFERENCE_BACKEND`, `SAE_GEMM_BACKEND`). Prebuilt TRT engines may be
  supplied at runtime via `detector_engine` / `arcface_engine`, bypassing ORT.
- Models load once per process. Any mode processing more than one title must
  amortise model and gallery load across titles.
- No mid-video checkpointing. A run either completes and emits a result, or emits
  nothing (see DP-004).

## DP-002 — Mode: batch CLI

One-shot invocation over one title. The reference mode and the substrate for the
others.

- `scene_analyze <movie> <gallery> -o out.json`, exit non-zero on failure.
- Must be safe to invoke concurrently by an external scheduler, subject to GPU
  memory.

**Current:** `src/main.cpp`. **Gap:** none.

## DP-003 — Mode: on-demand service

A resident process on a server, analysing on request.

- Models and gallery stay resident; requests carry a media path plus optional
  overrides.
- Requests are queued with a bounded depth and processed serially per GPU;
  the queue must be observable (depth, in-flight title, ETA).
- Health endpoint reporting model, gallery fingerprint, backend, and GPU state.
- Graceful drain on shutdown: stop accepting, finish or abandon in flight per DP-004.

**Gap:** not built. `scripts/run_from_jellyfin.py --worker` is a polling loop, not
a request-driven service; it is the closest existing shape.

## DP-004 — Mode: opportunistic / idle-triggered

Analyse when the machine is otherwise unused; yield the instant it is not.

- **Trigger is external and non-oscillating.** Screen-lock (logind
  `Lock`/`Unlock`) is the reference signal. GPU/CPU load must *not* be used: the
  worker is itself the load, so a load threshold forms a feedback loop.
- **Stop is a hard stop.** On the resume signal, SIGTERM the worker mid-analysis.
- **Re-queue is free and implicit.** An item leaves the pending queue only when
  its result is pushed, so a killed run simply stays pending. This requires:
  1. never push a partial result — push only after the analysis returns cleanly;
  2. clean up temp files on signal — write temps under a directory wiped on
     start, and unlink on SIGTERM (a SIGKILL skips `finally`).
- Accepted trade-off: a partially analysed title restarts from scratch. Fine for
  an overnight workload.
- Other triggers (idle timer, AC power, scheduled window) must fit the same
  contract: external signal, hard stop, implicit re-queue.

**Current:** designed in detail in [`service-conversion.md`](service-conversion.md)
as `sae-worker.service` + `sae-lock-gate.service` under systemd **user** units.
**Gap:** unbuilt; the temp-cleanup fix in `run_from_jellyfin.py` is a named
prerequisite.

## DP-005 — Installation and provisioning

- Native install, **no Docker at runtime** — GPU passthrough is the most fragile
  part of a containerised setup and exists only because of the container.
  Natively the GPU works with the host drivers and media paths need no
  re-mounting. This constrains how the software *runs*, not how it is *built*:
  DP-008 uses containers as build environments precisely because that side has
  none of these problems.
- The installer may **fetch a prebuilt binary** (DP-008) instead of compiling.
  Compiling stays supported, but should not be the only path — it is the slowest
  and most fragile step of a first install. TRT engines are still built locally
  either way (DP-008).
- An installer (`scripts/build_install.py`) consuming one `install.yaml`:
  platform (nvidia/amd/cpu), embedder model, gallery scan cadence, install
  prefix; runtime secrets written to a `.env`, editable without recompiling.
- Distro coverage: Fedora + Arch (`dnf`/`pacman`), auto-installing dependencies
  after printing them. Debian/Ubuntu out of scope.
- Model acquisition (`scripts/download_models.sh`) and TRT engine build
  (`scripts/build_trt_engines.sh`) are provisioning steps, not runtime steps.

**Gap:** installer unbuilt.

## DP-007 — CI build image

CI runs on an Intel N100 with no discrete GPU, so the test build must configure
**CPU-only** and must not require CUDA, TensorRT or ROCm:

```
-DSAE_INFERENCE_BACKEND=ORT  -DSAE_GEMM_BACKEND=CPU
```

A prebuilt container image supplies the toolchain, published to the **Gitea
container registry** and pinned by tag — matching the `jellytau-builder`
precedent. Building dependencies per CI run is untenable on an N100, and OpenCV 5
from source would dominate every run.

The same registry stores corpus dump fixtures as generic packages (see the
fixtures table in `requirements.md`). Rebuild the image when its dependency set
changes, not per run, and pin CI to a tag rather than `latest` so a rebuild
cannot silently change what a green build meant.

**Required in the image:**

| Dependency | Why |
|---|---|
| CMake, C++ toolchain, pkg-config | Build |
| **OpenCV 5** | `CMakeLists.txt:25` prefers 5, falls back to 4. The branch targets 5, so the image should carry it — it is not yet in most distro repos and building it per-run is prohibitive |
| HDF5 (C++) | Galleries are HDF5-native; also the dump format |
| FFmpeg dev libs — `libavformat`, `libavcodec`, `libavutil`, `libswscale`, **`libswresample`** | Decode. See the note below on swresample |
| Python 3 + numpy, h5py, scipy | Python-side tests, replay, traceability tooling |
| Catch2, nlohmann/json | **Vendored into the image, not fetched.** Both are `FetchContent`-ed today (`CMakeLists.txt:220`, `tests/CMakeLists.txt:8`), which makes every CI run depend on GitHub reachability |

**Deliberately excluded:** CUDA, TensorRT, ROCm — no GPU to use them. Also the
ONNX Runtime *GPU* providers; only the CPU provider is relevant, and only for T3
smoke tests.

**Models are not baked into the image.** The seven ONNX files total ~725 MB and
live in Git LFS. T1/T2 tests are model-free by design
(`tests/CMakeLists.txt:1-4`), so the default image needs none. T3 smoke tests
require a model and should pull it via LFS in a separate job rather than
inflating the image tenfold for a minority of tests.

**`libswresample` is a real gap, not a formality.** The current
`pkg_check_modules` list (`CMakeLists.txt:200-203`) covers avformat, avcodec,
avutil and swscale but **not** swresample — which IR-004 needs to downmix to mono
and resample to 11025 Hz. It must be added alongside the audio-signature work.

**Gap:** entire requirement. The image does not exist, and no CI config is
present in this repo.

## DP-008 — Builder images and release binaries

Produce prebuilt binaries per backend so deployment does not require every user
to compile the project.

**This does not contradict DP-005.** That requirement rejects Docker as a
*runtime* — GPU passthrough is the most fragile part of a containerised setup and
exists only because of the container. Using Docker as a *build* environment is
the opposite case: hermetic, reproducible, and it lets one machine produce
binaries for backends it cannot itself run. Build in a container; run natively.

### Image matrix

The build has two independent axes (`CMakeLists.txt:48-49`), so the useful
combinations are:

| Image | `SAE_INFERENCE_BACKEND` | `SAE_GEMM_BACKEND` | Target |
|---|---|---|---|
| `sae-builder-cpu` | ORT | CPU | CI (DP-007), and the smoke-test fallback |
| `sae-builder-cuda` | TRT | CUDA | NVIDIA |
| `sae-builder-rocm` | ORT | ROCM | AMD |

All three carry the DP-007 dependency set (OpenCV 5, HDF5, FFmpeg incl.
swresample, vendored Catch2/nlohmann) and differ only in the accelerator stack.
The CPU image is the CI image — one artifact, two uses.

Published to the Gitea container registry, pinned by tag, rebuilt when the
dependency set changes rather than per run.

### What ships, and what cannot

**Ships:** the `scene_analyze` binary and its companions, per backend.

**Cannot ship: TensorRT engines.** `.engine` files are specific to the GPU
architecture and TRT version they were built on — `scripts/build_trt_engines.sh`
must still run on the target machine. A prebuilt binary shortens the install; it
does not remove the local engine-build step, and the installer must not imply
otherwise.

**Cannot ship: models.** ~725 MB in LFS, and orthogonal to the binary.

### The constraint that decides the base image

**A binary built in a container runs against the host's glibc.** Build on a
newer base than the oldest supported host and it fails at load with
`GLIBC_2.xx not found` — the classic and entirely avoidable trap when shipping
binaries out of containers.

So the base is chosen for the *oldest* glibc to be supported, not for
convenience or recency. Accelerator libraries have the same shape of problem:
the binary links against a driver-provided runtime, so each image must document
the CUDA/ROCm version range its output is compatible with, and the installer
must check it rather than discovering a mismatch at first inference.

### Jobs

A release job per backend, producing a tagged artifact in the registry. These are
**not** the CI gate — the gate runs the CPU image on every push (DP-007);
release builds run on tag. Their outputs are what DP-005's installer fetches
when the user does not want to compile.

**Gap:** entire requirement. No images, no release jobs.

## DP-006 — Gallery maintenance as a background concern

- Incremental gallery refresh runs on a timer (`gallery_scan_interval`, default
  24 h) independently of analysis, so newly added titles' cast is embedded before
  their media is analysed.
- A gallery/model mismatch must be detected **at startup**, not silently produce
  garbage similarities. See GR-004.

---

# Part C — Integration requirements

## IR-001 … IR-003 — Truth-file output

Emit the JRay truth format, `schema_version: 1`, `Verbosity::minimal`.

- Sibling file `Movie.jray.json` next to the media (suffix configurable
  plugin-side).
- Per actor: `name`, `imdb_id`, `tmdb_id`, `jellyfin_id` (each `""` if
  unresolved), and `scenes` windows.
- **Each window carries its belief** and its identification route (AR-012) — the
  posterior is computed for every claim anyway, so it is serialised rather than
  discarded. This lets a consumer caveat or filter low-confidence presence
  instead of treating every window as equally certain.
- `jellyfin_item_id` is stamped in *after* analysis by `run_from_jellyfin.py` —
  `scene_analyze` does not know it.
- Additional verbosities: `standard` (per-frame bboxes, similarity, unknowns) and
  `xray` (Jellyfin-Xray `{"second": ["Actor", …]}`, dense integer seconds).

### `extraction.*` provenance

Consumers — and the public server, which ranks competing manifests — need to know
what produced a result:

- `sample_fps`, `pipeline_version`, `gallery_size` — as today.
- **`extinction_sec`** — replaces `anneal_sec`, which is dropped entirely (AR-012 withdrawal note).
  It is the parameter that shapes window extent, so it is what a consumer needs
  to interpret them.
- **`gallery_scope`** — `"global"` or `"limited"`. **The single most useful
  quality signal**: two galleries of identical size differ enormously depending
  on whether matching ran against the whole library or only the title's credited
  cast. A limited gallery cannot find an uncredited or mis-credited appearance at
  all; a global one competes against every actor in the library. Default is
  **global**.
- Optionally, the `tmdb_id`s of the actors the gallery was built from — useful
  for reproducibility locally. Not proposed for the Jmanifest, where a
  thousand-entry id list is bulk for little gain over `gallery_size` + scope.

Format is owned by [`../../jRay/SPEC.md`](../../jRay/SPEC.md); this pipeline is
the producer. Any change is a coordinated schema-version bump.

**Current:** `result_sink_node.hpp`. **Gap:** the schema changes in several ways
at once — `anneal_sec` out, `extinction_sec` and `gallery_scope` in, per-window
belief added, audio signature added (IR-004). All breaking, so they ship as **one**
`schema_version` bump coordinated across all three repos. The `scenes` values also
change under AR-012.

**Output timing:** the file is written after the deferred pass (AR-020/AR-021)
completes, not at EOF — deferred and pooled identifications add windows after the
last frame is read.

## IR-004, IR-005, IR-007, IR-008 — Audio signature

Emit the content-derived audio signature in the truth file, so a truth file is
self-identifying without a plugin round-trip.

Construction is specified in
[`../../JRay-public-server/SPEC.md` §3](../../JRay-public-server/SPEC.md) and must
be implemented **exactly** — a signature that differs in any parameter will not
match one computed by the plugin:

1. Decode a 120 s window centred on the midpoint (`runtime/2 ± 60 s`) — avoids
   logos/cold opens at the head and credits at the tail.
2. Downmix to mono, resample to 11025 Hz.
3. STFT: 4096-sample frame, 1024-sample hop (~93 ms, ~1290 frames), Hann window.
4. Log-magnitude spectrum over 300–3000 Hz.
5. 32 logarithmically spaced bins; record peak-bin index + 2-bit energy class.
6. One byte per frame → ~1290-byte array, base64-encoded.

Peak-bin rather than full spectrum: peaks survive lossy re-encoding, loudness
normalisation and channel-layout changes; absolute magnitudes do not.

Matching (sliding ±600 frames ≈ ±56 s, scoring the fraction of overlapping frames
whose peak bin matches) is a **consumer** concern — this pipeline produces the
signature, it does not match. Offsets are applied client-side; manifests are never
rewritten.

**Media shorter than 120 s.** The window `runtime/2 ± 60 s` underflows, so no
signature is emitted and **no sync offset is applied**. Such items fall back to
the runtime/exact tiers, which is adequate: a 90-second extra or trailer is not
the content whose cut alignment matters. Both producers must apply the identical
rule, or they diverge on exactly the short items most likely to be
mis-identified.

**Signature versioning.** The signature carries its own `v1:` prefix, separate
from `schema_version` (server spec §3 example: `"v1:v7fA3k…"`). Emit and honour
it, so a future change to the DSP chain is *detectable* rather than silently
producing non-matching signatures.

**Decision (this spec):** the pipeline computes and emits it *in addition to* the
plugin. Consequences to carry through:

- The truth schema gains a field → **`schema_version` bump**, coordinated with
  `jRay/SPEC.md` and the plugin.
- The pipeline needs an audio decode path. It already links FFmpeg
  (`ffmpeg_decoder.hpp`) for video, so this is a second stream from an existing
  dependency, not a new one.
- Both producers must agree bit-for-bit. A cross-check test — plugin signature vs.
  pipeline signature over the same file — is a hard requirement, not a nicety.
- Files never processed by this pipeline still get a signature from the plugin;
  the two paths coexist deliberately.

**Gap:** entire requirement — no audio path exists in the pipeline today.

## IR-006 — Jellyfin round-trip

- Pull the work queue: `GET /Plugins/JRay/Tasks/Pending?limit=N` — items with no
  results yet.
- Push results: `PUT /Plugins/JRay/Items/{itemId}/Truth` (admin API key). Managed
  truth takes precedence over a sidecar file for the same item.
- Push only complete results (DP-004).

**Current:** `scripts/run_from_jellyfin.py`. **Gap:** none.

---

# Part D — Gallery construction requirements

## GR-001, GR-002, GR-005 — Sources

Build a gallery of actor reference embeddings from Jellyfin and TMDB.

- **Jellyfin-wide** (`make_jellyfin_gallery.py`): enumerate every Movie/Series,
  collect the unique cast across the whole library, download each actor's
  headshot from Jellyfin directly (no TMDB key required), embed, write one global
  gallery.
- **TMDB fallback**: for actors with no usable Jellyfin image, fall back to TMDB
  profile images (`--tmdb-key`).
- **Incremental merge** (`--merge`): re-runs pick up newly added titles without
  re-embedding actors already present. This is what makes DP-006 cheap enough to run
  daily.
- **Cast restriction** (`filter_gallery.py`): derive a per-title gallery limited
  to credited cast. Faster and fewer look-alike mismatches, but note
  [`gallery-scope.md`](gallery-scope.md) — the rep4 matrix found the *full*
  gallery won for the shipped model, so restriction is a tool, not the default.

Each actor carries `name`, `imdb_id`, `tmdb_id`, `jellyfin_id` (whichever
resolve), one embedding per reference image, and the source image paths.

**Current:** `make_jellyfin_gallery.py`, `make_gallery.py`, `filter_gallery.py`,
`sae_jellyfin.py`, `sae_tmdb.py`, `src/gallery/gallery_builder.*`. **Gap:** none.

## GR-003 — Quality and coverage reporting

Gallery build must report, not just produce:

- actors with zero usable images (they can never be recognised — a silent recall
  ceiling);
- actors below the 5-embedding threshold for positive pairs (AR-023), which
  degrades calibration;
- the fitted calibration and the intra/inter distributions behind it (AR-023 gap);
- duplicate/near-duplicate references removed.

**Gap:** partial. Dedup and eligibility are computed inside calibration but not
surfaced as a build report.

## GR-004 — Model binding

- A gallery is only valid for the embedder that built it. The embedder identity
  must be **stamped into the gallery file**, and checked at startup by any
  consumer.
- Mismatch is a hard startup error. Cosine similarities between embeddings from
  different models are meaningless but *look* plausible — this fails silently and
  expensively otherwise.

**Gap:** named as step 4 of the `service-conversion.md` implementation plan;
unbuilt. This is the highest-value small fix in the document.

## GR-006 … GR-009 — Provenance tiers and poisoning guard

Reference embeddings now come from three sources with different trust, and they
must be **distinguishable in the gallery**, not merged into an undifferentiated
pile:

| Tier | Source | Persists | Trust |
|---|---|---|---|
| **Baked** | Jellyfin / TMDB headshots | Yes | High — curated, externally sourced |
| **Harvested** | Per-film annex (AR-019), promoted from owned tracks | **Yes, flagged** | Unverified — machine-derived |
| **Confirmed** | Human association (`../../SPEC.md` §4) | Yes | Highest — a person said so |

**Harvested embeddings are retained rather than discarded at exit**, because they
are exactly the non-frontal views the baked gallery lacks and their value
compounds across a library. But they carry the risk the ephemeral annex avoided:
**a promotion error becomes permanent instead of dying with the process.**

They are therefore **flagged as harvested and reviewable**, never silently equal
to a baked reference. The tier must be recorded per embedding so that a suspected
poisoning can be traced, audited, and reverted without rebuilding the gallery.

### Bell-curve outlier detection

An actor's own embeddings should form a **roughly normal distribution in cosine
space** around their centroid. A harvested embedding that falls outside that
distribution is unlikely to be the same person — which is precisely what a
poisoned entry looks like.

**Requirement: flag harvested embeddings that are distributional outliers among
that actor's references**, for review or automatic exclusion.

> **`EXCEPTION: AR-024` — raw cosine is used here deliberately.**
>
> AR-024 requires calibrated probabilities everywhere, and this is an agreed
> exception. The reason: the calibration is a monotonic squash mapping similarity
> onto `P(same person)`. That is exactly right for making a *decision*, and wrong
> for characterising a *distribution* — the sigmoid compresses the tails, which
> is where outliers live, and would flatten the very structure being tested.
> Distribution shape and outlier distance are properties of the metric space, so
> they are measured in it.
>
> Scope of the exception: distributional analysis of an actor's own reference set
> only. Any match, association, or admission decision still goes through the
> calibration.

**Gap:** entire requirement — tiering, persistence of harvested embeddings, the
flag, and the outlier check.

**Open question:** whether human-confirmed associations should be a distinct
audited tier (individually revocable if someone mislabels) or simply more
embeddings for that `tmdb_id`. Deferred.

---

# Part E — Parameter-study requirements

The tuned constants in `config.hpp` are empirical. Retuning must stay cheap, or
it will not happen — and AR-012 makes a retune mandatory.

## VR-001 — Post-inference dump

Persist pipeline state at the point where the expensive work ends.

- Dump at the `EmbeddedSceneFrame` channel — after decode → detect → align →
  embed, **before** tracking and identity matching. Everything downstream is
  cheap CPU maths, so a replay re-runs the whole tail with no GPU and no video.
- HDF5, one file per title, flat/ragged: per-face arrays concatenated, with a
  per-frame index table (`face_offset`, `face_count`) pointing into them. Avoids
  variable-length HDF5 types and reads straight into numpy.
- Stores per frame: `timestamp_sec`, `frame_idx`, `is_cut`, `is_scene_boundary`.
  Per face: `embedding` [N,512], `bbox` [N,4], `landmarks` [N,10], `confidence`.
- Invariants: embeddings unit-norm; `face_offset` contiguous; bboxes already in
  original resolution; frames with no faces still get a row so timestamps stay
  dense; EOF sentinels not written.
- Enabled by `--dump-embeddings out.h5`; teeing must not perturb the live result.

Schema owned by [`scripts/optimizer/SCHEMA.md`](../scripts/optimizer/SCHEMA.md).

**Current:** C++ dump sink (`embedding_dump_node.hpp`, `dump_embeddings.cpp`),
read by `replay.py`. **Gap:** **AR-012 breaks the replay contract.** Track extents
are decided in the tracker, which is *downstream* of the dump — so a replay can
reproduce them, but only if the dump preserves everything the tracker needs.
Verify `landmarks`/`bbox`/`is_cut` suffice, and bump `schema_version` if not.

## VR-002 — Replay and sweep

- Replay drives the **real KPN nodes** over dumped embeddings, not a
  reimplementation — a sweep that optimises a divergent copy is worthless.
- The gallery loads once per process and is cached by path, so one evaluation is
  N cheap replays.
- Differential Evolution over the continuous knob space
  (`prob_threshold`, `expand_min_anchor_frames`, the re-acquisition timeout, …),
  scored against reference presence data. Note these are *not* independent — a
  longer timeout yields longer tracks, hence more frames to clear the anchor
  count — so they must be swept jointly.

**Current:** `scripts/optimizer/optimize.py`, `replay.py`, `second_score.py`.
**Gap:** none, pending E1.

## VR-003 — Scoring methodology

- Micro-averaged per-second presence against Amazon X-Ray, per
  [`methodology.md`](methodology.md).
- **Objective is F1, but precision and recall are logged at every evaluation and
  printed at the optimum.** X-Ray recall is a face-vs-cast-in-scene ceiling, so
  unconstrained F1 pushes `prob_threshold` *down* chasing unreachable recall,
  trading real precision away. The trade-off must stay visible so another
  operating point can be chosen from the trajectory (`--trajectory`).
- Known metric hazard: the earlier 9-film scene-union metric hid out-of-cast
  false positives; the 4-film rep4 per-second metric supersedes it. Any new
  metric must be checked for the same class of blindness.

**Current:** implemented; documented in `methodology.md`,
`rep4-optimizer-results.md`, `optimizer-experiments.md`. **Gap:** none.

## VR-004 — Validation corpus

- A manifest-driven film set with ground truth (`scripts/validation/`), scored
  reproducibly. Benchmarking practice is already established — see
  [`methodology.md`](methodology.md), [`model-bakeoff.md`](model-bakeoff.md),
  [`best-model.md`](best-model.md) and `rep4-optimizer-results.md`; AR-012 changes
  what is measured, not how.
- After AR-012, `prob_threshold` (0.754) must be re-derived — it was fitted against
  per-frame presence semantics and now governs *voting*, not presence.
  `anneal_sec` (35.5) and `extinction_sec` (57.4) are not re-derived; they are
  deleted (AR-012 withdrawal note).

## VR-005 — Minimum face size study

Quantify where ArcFace degrades, replacing the 66×66 estimate in A1 with a
measurement.

**Method.**

1. Select ~100 gallery actors having more than one mugshot.
2. Per actor, hold out **one** image as the probe; its remaining images stay in
   the gallery at native resolution.
3. For each target size *S*, downscale the probe to *S*×*S* and upscale back to
   112×112, then embed.
4. Match each degraded probe against the full 100-actor gallery and record
   **TPI/FPI** — identified as the correct actor, or as someone else.
5. Repeat across sizes to get the curve.

The asymmetry is the point: **the gallery stays high-res and only the probe
degrades**, which is exactly the production case — reference mugshots are clean,
the face from the video is small. It also measures the decision the pipeline
actually makes (probe against gallery) rather than embedding drift, which can be
large without harming separability and small in a direction that destroys it.

**Caveat on gallery size.** FPI grows with the number of actors competing, so a
100-actor gallery understates the false-positive rate against a full library of
thousands. Treat the FPI numbers as *relative* across sizes rather than as an
absolute rate, or re-run at production scale before setting a threshold from
them.

**Cheap to run** — no video needed, gallery images are already on disk, and the
embedding/matching machinery exists (`scripts/sae_embed_loader.py`,
`gallery_calibration.hpp`). Expect a knee rather than a cliff; the output is a
size-versus-TPI/FPI curve plus a chosen operating point, not a single number.

**Secondary output:** the same curve shows whether `min_face_px` should be a
constant at all or should scale per embedder — relevant since the model is a
build-time choice (GR-004).

## VR-007 — Expansion band and deferred-pass study

Tune the AR-018 admission band and establish whether AR-020 pays.

**Band.** Sweep the lower and upper bounds around the 0.90–0.95 working estimate.
The two bounds fail in opposite directions and must be read separately: too low a
lower bound admits the wrong person (precision collapse, amplified by the
deferred pass); too low an upper bound admits only redundant views (no recall
gain, wasted annex). Report both, not a single F1.

**Deferred pass.** Measure recall recovered by AR-020 — how many TBI entries are
identified on re-assessment — and precision of those late identifications
specifically. They should be scrutinised separately from first-pass
identifications, because they are the ones relying on harvested rather than baked
references.

**Clustering (AR-021).** Sweep the merge threshold and report cluster purity
against known-cast ground truth, not just downstream F1. The two error modes are
asymmetric and must be reported separately: an over-merge mis-identifies every
track in the cluster at once, while an under-merge only forfeits the pooling
benefit. Verify the temporal cannot-link constraint is actually binding — measure
how many candidate merges it rejects, since if the answer is zero the constraint
is not doing the work claimed for it.

**Iteration.** Test one pass versus iterating to convergence (AR-020). Report
whether round 2+ recovers enough to justify the complexity, and whether precision
degrades with each round — the failure mode being a wrong identification in
round 1 seeding references that corrupt round 2.

**Ablation worth having:** expansion on with deferred pass off. It separates
"expansion helps live matching" from "expansion helps the second pass", which the
current all-or-nothing `expand_gallery` flag cannot distinguish.

## VR-008 — Gallery scaling benchmark

Establish the throughput-versus-gallery-size curve required by A10.

**Method.** Synthesise galleries at 10², 10³, 10⁴, 10⁵ actors (random unit-norm
embeddings suffice — this measures compute, not accuracy) and record per-frame
matching time, end-to-end throughput, and GPU memory.

**The number that matters** is where gallery scoring stops being negligible and
starts dominating runtime relative to decode, detection and embedding. Below that
point gallery growth is free; above it, it sets the pace.

**Also measure the deferred pass (AR-020) separately.** It has a different shape —
one large offline multiply rather than many small streaming ones — so it may
scale quite differently and could well become the dominant cost on a film with
many unknowns.

**Report calibration build time too.** AR-023's cross-actor pair enumeration is
O(N²); this benchmark is where that becomes visible, and it will likely be the
first thing to break at scale.

## VR-006 — Re-tune `scene_threshold` at native rate

Not a study of whether to feed TransNetV2 correctly — AR-011 settles that it must be.
This is the consequence: `scene_threshold` (0.60) was picked against 12 fps input,
where the model's separation was compressed. At native rate the boundary/
non-boundary margin should widen, so the operating point moves.

Small and mechanical: score boundaries across a threshold range on a few titles
once native-rate decode lands, and pick the new point. Expect a cleaner
separation than the ~0.50 baseline / ~0.7+ peak recorded at 12 fps; if it does
*not* improve, that is evidence worth having about the export itself.

---

# Open questions

Resolved during planning, recorded here so the reasoning is not lost:

- ~~**AR-012 vs. extinction.**~~ Wholly subsumed. The `last_seen` model ends windows
  at the last sighting, which removes the over-claim `extinction_sec` caused;
  both it and `anneal_sec` are deleted rather than re-fitted (AR-012 withdrawal note).
- ~~**AR-022 retention.**~~ Embeddings + metadata by default, crops opt-in behind
  `--dump-unidentified-crops`.
- ~~**IR-004 bit-exactness.**~~ Golden-vector fixture checked into both repos, not a
  shared implementation — the coupling cost of the latter exceeds the benefit.
- ~~**AR-023 prior.**~~ Decide once GR-003 persists the intra/inter distributions, so the
  real value is known rather than argued about.

Still open (pipeline-local):

1. **Re-acquisition timeout scope.** `track_max_frames_missing` (5) and
   `cut_inactive_max_frames` (5) currently distinguish an ordinary miss from a
   cross-cut park. Under AR-012 both become the same thing. Do they collapse to one
   constant, or does a cut still warrant a different window? Cheap to test both
   in the Phase 2 sweep.
2. **Context-crop budget.** How many representative frames per unidentified
   track, and chosen how (largest? sharpest? most frontal?). Bounded by review-UI
   usefulness rather than by diagnostics.

Escalated to the system spec ([`../../SPEC.md`](../../SPEC.md) §5), since they
bind more than one repo: `schema_version` coordination for the pending bump, and
whether unidentified presence should be published in the truth format.