dtourolle/scene-actor-extraction
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: software spec, requirements register, and implementation plan

Browse Source 
Adds the requirements baseline for the pipeline redesign:

- SPEC.md — software requirements with Current/Gap deltas per item, so the
  document doubles as a work list.
- requirements.md — stable flat IDs (AR/DP/IR/GR/VR) with parent traces,
  priorities, statuses, and a per-requirement verification plan. Replaces the
  thematic A1..E8 scheme, which had already produced an A1a and an out-of-order
  E6; IDs are now permanent and never reused.
- IMPLEMENTATION-PLAN.md — phased work.

The central change is AR-012: presence follows track extent rather than
per-frame recognition, so a window starts when an actor appears rather than
when the recogniser first succeeded. anneal_sec and extinction_sec are
withdrawn rather than retuned — a track that survives its own gaps leaves them
nothing to do.

Verification is shaped by CI running on an N100 with no dGPU: the existing
HDF5 dump makes everything downstream of embedding replayable on CPU, which
covers the bulk of the redesign.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
dtourolle
2026-07-30 17:32:44 +02:00
co-authored by Claude Opus 5
parent 458116f118
commit a5299daf6e
3 changed files with 2105 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/requirements.md
+268
View File
@@ -0,0 +1,268 @@
# scene-actor-extraction — requirements register
Stable IDs for every requirement in [`SPEC.md`](SPEC.md), which holds the prose.
This file is the **authoritative list**; the CI gate reads its denominators from
here (see [`../../SPEC.md`](../../SPEC.md) §6).
**IDs are permanent.** A withdrawn requirement is marked `Withdrawn` and its
number is never reused — renumbering is what produces orphan TRACES tags. This
register replaces the earlier thematic `A1…E8` scheme, which had already produced
an `A1a` and an out-of-order `E6`.
Tag code with `// TRACES: AR-012 | SR-002`.
| Type | Scope |
|---|---|
| `AR` | Algorithm — the extraction pipeline itself |
| `DP` | Deployment — how it runs |
| `IR` | Integration — contracts with other components |
| `GR` | Gallery — building and maintaining actor references |
| `VR` | Validation — parameter studies and benchmarks |
| `UT` / `IT` | Unit / integration tests |
Status: `Done` · `In Progress` · `Planned` · `TBD` · `Withdrawn`
---
## Algorithm (AR)
| ID | Requirement | Traces to | Priority | Status |
|---|---|---|---|---|
| AR-001 | Detect faces in sampled frames; emit bbox, confidence, 5-point landmarks in original pixel space | SR-002 | High | Done |
| AR-002 | Minimum face size 66×66 px, expressed in **original** resolution (decoupled from `dense_scale`) | SR-002 | High | Planned |
| AR-003 | No fixed per-frame face cap — crowd scenes must not lose background cast | SR-002 | Medium | Planned |
| AR-004 | Backpressure: unbounded faces/frame absorbed by slowing, never by dropping or throwing | SR-002 | High | Planned |
| AR-005 | Align to 112×112 via ArcFace 5-point similarity transform | SR-002 | High | Done |
| AR-006 | 512-d L2-normalised embeddings, batched | SR-002 | High | Done |
| AR-007 | Associate detections by IoU + embedding, with **frame-dependent** weighting | SR-002 | High | In Progress |
| AR-008 | One track pool keyed on `last_seen`; no separate revival path | SR-002 | High | Planned |
| AR-009 | Camera-cut detection (histogram) as an association hint | SR-002 | High | Done |
| AR-010 | Scene-boundary detection (TransNetV2) as an association hint | SR-002 | Medium | In Progress |
| AR-011 | **Every model is fed the input it was trained for** — cost reduced by running less often, never by degrading one inference | SR-002 | High | Planned |
| AR-012 | Presence follows **track extent**, not per-frame recognition | **SR-002** | High | Planned |
| AR-013 | `last_seen` optional state machine; window ends at last sighting, never after | SR-002 | High | Planned |
| AR-014 | Belief swap A→B terminates the track and starts a new one | SR-002 | Medium | Planned |
| AR-015 | Two live tracks owned by one actor ⇒ treat as a detected cut, re-associate | SR-002 | Medium | Planned |
| AR-016 | All tracks closed at EOF — a film ends with faces on screen | SR-002 | High | Planned |
| AR-017 | Every presence claim carries its belief and identification route | SR-002 | High | Planned |
| AR-018 | Per-subject embedding store with banded admission (novel enough, safe enough) | SR-005 | Medium | Planned |
| AR-019 | Per-film gallery annex from owned tracks; acquires the non-frontal views TMDB lacks | SR-005 | Medium | In Progress |
| AR-020 | Deferred re-identification of unknown tracks against the final expanded gallery | SR-005 | High | Planned |
| AR-021 | Cluster unknown tracks into one entity per person, under temporal cannot-link constraints | SR-005 | Medium | Planned |
| AR-022 | Capture still-unidentified tracks: embeddings, metadata, **context crops** for human review | §4 | Medium | Planned |
| AR-023 | Fit sigmoid calibration from intra/inter similarity distributions | SR-002 | High | Done |
| AR-024 | **Always the calibrated probability, never a raw cosine** — exceptions recorded | SR-002 | High | Planned |
| AR-025 | Per-track Bayesian accumulation in log-odds, with correlated-observation discounting | SR-002 | High | Planned |
| AR-026 | All similarity computed as GEMM, including annex and deferred pass | SR-001 | High | In Progress |
| AR-027 | Throughput acceptable for **arbitrary** gallery size | SR-001 | High | Planned |
## Deployment (DP)
| ID | Requirement | Traces to | Priority | Status |
|---|---|---|---|---|
| DP-001 | One analysis core; modes are front-ends and must not fork pipeline logic | PR-004 | High | Done |
| DP-002 | Batch CLI over one title | PR-004 | High | Done |
| DP-003 | On-demand resident service with bounded, observable queue | PR-004 | Medium | Planned |
| DP-004 | Opportunistic/idle mode: external trigger, hard stop, implicit re-queue | PR-004 | Medium | Planned |
| DP-005 | Native installer, no Docker; Fedora + Arch | PR-004 | Medium | Planned |
| DP-006 | Background incremental gallery refresh on a timer | PR-003 | Medium | Planned |
## Integration (IR)
| ID | Requirement | Traces to | Priority | Status |
|---|---|---|---|---|
| IR-001 | Emit the JRay truth format as sibling `.jray.json` | SR-003 | High | Done |
| IR-002 | Windows carry belief + route; `extraction.*` carries `extinction_sec`, `gallery_scope` | SR-003 | High | Planned |
| IR-003 | Output written **after** the deferred pass, not at EOF | SR-003 | High | Planned |
| IR-004 | Compute the audio signature exactly per server spec §3 | SR-003 | Medium | Planned |
| IR-005 | Golden-vector fixture shared with the plugin repo to prove bit-exactness | SR-003 | High | Planned |
| IR-007 | Media < 120 s: emit no signature, apply no sync offset — identical rule in both producers | SR-003 | Low | Planned |
| IR-008 | Emit and honour the signature's own `v1:` version prefix | SR-003 | Low | Planned |
| IR-006 | Jellyfin round-trip: pull pending queue, push complete results only | SR-001 | High | Done |
## Gallery (GR)
| ID | Requirement | Traces to | Priority | Status |
|---|---|---|---|---|
| GR-001 | Build gallery from Jellyfin library cast, TMDB profile fallback | SR-001, SR-005 | High | Done |
| GR-002 | Incremental `--merge` refresh without re-embedding known actors | PR-003 | High | Done |
| GR-003 | Report coverage: zero-image actors, under-referenced actors, dedup, calibration PDFs | SR-001 | Medium | Planned |
| GR-004 | Stamp embedder identity into the gallery; **hard startup error** on mismatch | SR-001 | High | Planned |
| GR-005 | Gallery data never leaves the instance | **SR-005** | High | Done |
| GR-006 | Provenance tiers: baked / harvested / confirmed, distinguishable per embedding | SR-005 | High | Planned |
| GR-007 | Persist harvested embeddings **flagged and reviewable**, never silently equal to baked | SR-005 | Medium | Planned |
| GR-008 | Flag distributional outliers among an actor's references (poisoning guard) — `EXCEPTION: AR-024` | SR-005 | Medium | Planned |
| GR-009 | Human-confirmed associations persist and improve future extractions | §4 | Medium | TBD |
## Validation (VR)
| ID | Requirement | Traces to | Priority | Status |
|---|---|---|---|---|
| VR-001 | HDF5 post-inference dump at the embedded-frame boundary | PR-002 | High | Done |
| VR-002 | Replay drives the **real** KPN nodes, not a reimplementation | PR-002 | High | Done |
| VR-003 | Scoring: micro-F1 against X-Ray, precision/recall logged at every evaluation | PR-002 | High | Done |
| VR-004 | Reproducible validation corpus with ground truth | PR-002 | High | Done |
| VR-005 | Minimum face size study — TPI/FPI vs probe size, gallery held at native res | PR-002 | Medium | Planned |
| VR-006 | Re-tune `scene_threshold` once native-rate decode lands | PR-002 | Low | Planned |
| VR-007 | Expansion band, clustering threshold, and deferred-pass ablation | PR-002 | Medium | Planned |
| VR-008 | Gallery scaling benchmark — throughput vs gallery size | PR-002 | Medium | Planned |
| VR-009 | Verify accumulated posteriors are calibrated against held-out tracks | PR-002 | High | Planned |
---
## Verification strategy
**CI runs on an Intel N100 with no discrete GPU.** That is a hard constraint on
how each requirement can be verified, and it shapes the test design rather than
merely limiting it.
Four tiers, in decreasing order of preference:
| Tier | Runs in CI | What it covers |
|---|---|---|
| **T1 — CPU unit** | Yes | Pure logic: registry state machine, belief accumulation, clustering, band admission, calibration maths |
| **T2 — Replay** | Yes | Real pipeline nodes driven from an HDF5 fixture — no GPU, no video |
| **T3 — CPU inference** | Yes, slowly | ORT CPU provider over a handful of frames; smoke tests only |
| **T4 — GPU** | **No** | Throughput, TRT engines, large-gallery GEMM |
**T2 is the reason this is workable.** The HDF5 dump (VR-001) captures state
after decode → detect → align → embed and before tracking and matching, so
everything downstream — which is where nearly all of the new design lives — is
cheap CPU maths replayable from a fixture. Tracking, presence windows, belief
accumulation, expansion, deferred re-identification and clustering are all
verifiable on an N100 at full fidelity, not in miniature.
That was already true for the optimizer. It now doubles as the CI strategy, which
is a strong argument for keeping the dump schema honest (VR-001) and for the
replay driving the *real* nodes rather than a reimplementation (VR-002).
**Small committed fixtures are required.** A few HDF5 dumps covering the awkward
cases — a cut, a belief swap, two live tracks converging, a film ending
mid-track, an unknown track that only resolves after expansion — are worth more
than a large corpus, and they are small enough to commit.
**T4 requirements cannot pass in CI, and the gate must not pretend otherwise.**
For these, CI verifies that a test *exists and is tagged*, not that it passes;
the run happens on a GPU host, nightly or manually, and reports separately. A
requirement whose only evidence is a test that never executes should be visible
as such rather than counted as covered.
| Requirement | Tier | Note |
|---|---|---|
| AR-001, AR-005, AR-006 | T3 | Smoke only — correctness of detection/embedding is a model property, not ours |
| AR-002 | T2 | Size filtering is arithmetic on dumped bboxes |
| AR-003, AR-004 | T1 + T4 | Backpressure logic is unit-testable; saturation behaviour needs real load |
| AR-007 … AR-017 | **T2** | The core of the redesign — fully replayable |
| AR-018 … AR-022 | **T2** | Expansion, deferred pass, clustering: all post-embedding |
| AR-023 … AR-025 | T1 | Calibration fit and log-odds accumulation are pure maths |
| AR-026, AR-027 | T4 | GEMM throughput and scaling — GPU host only |
| DP-* | T1 + manual | Lifecycle logic unit-tested; install paths are manual |
| IR-001 … IR-003 | T1 | Serialisation against a golden truth file |
| IR-004, IR-005 | **T1** | Audio signature is CPU DSP — the golden-vector fixture runs anywhere, which is precisely why it is the right cross-repo check |
| GR-001 … GR-005 | T1 + T3 | Gallery assembly is I/O and bookkeeping; embedding is T3 smoke |
| GR-006 … GR-008 | T1 | Tiering and outlier detection operate on stored embeddings |
| VR-* | Out of CI | Studies are run deliberately and their results committed as documents |
**One consequence worth stating:** AR-027 (arbitrary gallery scale) is
structurally unverifiable on the CI host. It needs a GPU host and a synthetic
large gallery, so it is the requirement most likely to silently regress. Its
benchmark (VR-008) should run on a schedule rather than on demand.
### Fixtures — precomputed inference, pulled by CI
The N100 cannot run inference at any useful rate, so **inference output is
precomputed on a GPU host and consumed by CI as data.** This converts most of
what looks like GPU work into pure CPU replay.
| Fixture | Contents | Size | Storage |
|---|---|---|---|
| **Edge-case dumps** | ~6 short clips (30–60 s), one per awkward behaviour | ~1 MB each | **Committed in-repo** |
| **Corpus dumps** | Full-length titles from the validation corpus | ~30 MB each | Pinned artifact, fetched by checksum |
| **Synthetic gallery** | Random unit-norm embeddings, fixed seed | small | Generated at test time |
| **Golden truth files** | Expected output for each edge-case dump | KB | Committed |
| **Audio golden vectors** | Short WAV + expected signature | KB | Committed, **shared with the plugin repo** |
Edge-case dumps are small enough to commit, and being in-repo means they version
with the code that reads them. Corpus dumps are pulled by pinned checksum from
the artifact store rather than committed, since they are large and change only
when the dump schema does.
**Generation must be reproducible and versioned.** A script, run on a GPU host,
regenerates every fixture from source clips; it is re-run when the VR-001 schema
version bumps. A fixture whose provenance is unknown is worse than no fixture,
because it will be trusted.
> **The limitation that must stay visible:** replay fixtures freeze upstream
> behaviour. A test driven from a dump verifies AR-007 onward *given those
> embeddings* — it cannot detect a regression in detection, alignment or
> embedding, because those produced the fixture. Nothing in CI can. That gap is
> covered only by the T3 smoke test and the scheduled GPU run, and it should not
> be papered over by a high replay-coverage number.
### Per-requirement verification plan
| ID | Tier | Test asserts | Edge cases to cover |
|---|---|---|---|
| AR-001 | T3 | Detector returns plausible boxes on a known frame | — smoke only |
| AR-002 | T2 | Faces below 66 px (original res) are dropped | Exactly at threshold; with `dense_scale` 0.5 — the interaction that motivated the requirement |
| AR-003 | T2 | No cap applied; a 40-face frame yields 40 | Crowd frame |
| AR-004 | T1 | Saturated input blocks rather than drops or throws | Bounded queue at capacity; **byte-based** limit with large crops; SIGTERM mid-block |
| AR-005 | T1 | Known landmarks → expected 112×112 warp | Landmarks near frame edge; degenerate/collinear points |
| AR-006 | T3 | Embeddings are unit-norm | Batch smaller than, equal to, larger than `embed_batch_size` |
| AR-007 | T2 | Association picks the right track | Two faces crossing paths; one leaving frame as another enters |
| AR-008 | T2 | One pool; dormant tracks match on embedding, not IoU | Dormant track whose old bbox overlaps a *different* new face — must not match on position |
| AR-009/010 | T2 | Cut/boundary shifts weighting toward embedding | Cut with same people; cut with all-new people |
| AR-011 | T1 | TransNetV2 receives native-rate frames | Source at 24/25/30 fps — dedup window derived, not assumed |
| AR-012 | **T2** | Window spans full track extent, not first recognition | Actor recognised only at track end — window must still start at `first_seen` |
| AR-013 | **T2** | `last_seen` set/unset; window ends at last sighting | Gap just under vs just over timeout; reappearance after timeout → two windows |
| AR-014 | T2 | Belief swap closes one window, opens another | No blended window; no overlap at the swap frame |
| AR-015 | T2 | Two live tracks on one actor trigger re-association | Counter increments |
| AR-016 | **T2** | Every track closed at EOF | Film ending mid-shot — window ends at final frame, not dropped |
| AR-017 | T1 | Claim carries posterior and route | Deferred and pooled routes distinguishable |
| AR-018 | T1 | Band admits only within bounds | At each bound exactly; store never admits below lower bound |
| AR-019 | T2 | Promotion only when all three signals quiet | Cut mid-track blocks promotion |
| AR-020 | **T2** | Unknown resolved after expansion | Track failing at minute 12, resolved at EOF — the ordering-independence claim |
| AR-021 | T2 | Clustering merges same person, respects cannot-link | **Temporally overlapping tracks never merge**; measure how many merges the constraint rejects |
| AR-022 | T1 | Context crops retained, bounded per track | Track running for minutes |
| AR-023 | T1 | Sigmoid fit on synthetic separable data | Too few positive pairs → `valid=false`, fallback engages |
| AR-024 | **Static check** | No bare cosine outside a tagged `EXCEPTION` | Grep-based; this is the invariant's enforcement |
| AR-025 | T1 | Log-odds accumulate; correlated frames discounted | 30 identical frames must **not** reach the certainty of 30 diverse ones |
| AR-026 | T1 + T4 | GEMM path produces same result as reference loop | Equivalence on small input in CI; throughput on GPU host |
| AR-027 | **T4** | Throughput at 10²…10⁵ actors | Scheduled, not on-demand |
| IR-001/002 | T1 | Serialised output matches golden file | Zero-length window; actor with many windows |
| IR-003 | T1 | Output written after deferred pass | Not at EOF |
| IR-004/005 | **T1** | Signature matches golden vector bit-for-bit | **Media < 120 s → no signature**; identical result in both repos |
| GR-004 | T1 | Mismatched embedder → hard startup error | Error names both sides |
| GR-008 | T1 | Outlier flagged among an actor's references | Injected poisoned embedding detected |
| VR-009 | T1 | Posterior calibration holds | A 0.99 posterior is wrong ~1% of the time on held-out tracks |
Three of these are worth singling out because they verify claims that would
otherwise be assertions: **AR-012** (window starts at `first_seen` even when
recognition comes late) is the entire point of the redesign; **AR-020** (a track
failing mid-film resolves at EOF) is the claim that ordering stops mattering; and
**AR-025** (30 identical frames ≠ 30 diverse ones) is what stops the Bayesian
accumulation from being decoration.
---
## Withdrawn
| ID | Requirement | Reason |
|---|---|---|
| — | `anneal_sec` window merging | Superseded by AR-012/AR-013: a track survives its own gaps, so there is nothing to anneal |
| — | `extinction_sec` actor keep-alive | Superseded by AR-013: windows end at last sighting, which is what this over-claimed |
Both were deleted rather than retained at zero — a field naming a mechanism the
pipeline no longer has is actively misleading (see `SPEC.md` A6.6).
---
## Notes on coverage
- **VR-*** traces to PR-002 (scene-granularity answers) rather than to a system
requirement: parameter studies are single-repo work serving accuracy, and this
is correct rather than a gap.
- **PR-005** (leak nothing) has no `AR`/`DP` row. It is satisfied *structurally*
by SR-004 and GR-005 — the server holds no binary, the gallery never leaves the
instance — not by any component doing something. It cannot be verified by
pointing at code, and it dies the moment either prohibition is relaxed.