scene-actor-extraction/scripts/optimizer/SCHEMA.md

# Embedding-dump HDF5 schema (v1)

One file per analysed title. Captures the pipeline state at the `EmbeddedSceneFrame`
channel — i.e. after decode → detect → align → embed, but **before** tracking and
identity matching. Everything downstream (face tracker, identity matcher, scene
tracker/anneal) is cheap CPU math, so replaying from this file lets a parameter
sweep re-run the whole downstream tail thousands of times with no GPU and no video.

Written by the C++ dump sink (`--dump-embeddings out.h5`); read by
`scripts/optimizer/replay.py`.

## Layout

The dump is **flat/ragged**: all faces across all frames are concatenated into
per-face arrays, with a per-frame index table pointing into them. This avoids
variable-length HDF5 types and reads straight into numpy.

```
/                                     (root)
  attrs:
    schema_version : int   = 1
    embed_dim      : int   = 512

    # ── what produced the vectors (GR-004) ──────────────────────────────────
    embedder_model : str   basename of the embedding model
    embedder_sha256: str   SHA-256 of that model file

    # ── what produced the faces (VR-010) ────────────────────────────────────
    detector_model : str   basename of the detector .onnx
    detector_conf  : float score floor a detection had to clear to be dumped
    detector_nms   : float NMS IoU threshold
    min_face_px    : float minimum box side, ORIGINAL-resolution px (AR-002)
    max_faces      : int   per-frame cap; 0 = uncapped, the default (AR-003)

    # ── what produced the frames (VR-010) ───────────────────────────────────
    movie          : str   source video path
    sample_fps     : float frames analysed per second of movie
    start_sec      : float seek point
    end_sec        : float stop point; -1 = end of file
    cut_threshold  : float histogram correlation below which is_cut fires
    dense_scale    : float decoded-frame downscale in dense mode; 1 = off
    bbox_upscale   : float multiply faces/bbox and faces/landmarks by this to
                           reach original video pixels; 1 when dense_scale is 1
    scene_detect   : uint8 0/1 — was TransNetV2 running at all (see below)

    # ── downstream setting recorded for comparability (VR-010) ──────────────
    track_assoc_min_prob : float  the run's tracker admission probability

  frames/                             group — one row per sampled frame
    timestamp_sec  : float64 [F]
    frame_idx      : int64   [F]
    is_cut         : uint8   [F]      (histogram intra-scene cut)
    is_scene_boundary : uint8 [F]     (TransNetV2 boundary; 0 if scene_detect off)
    face_offset    : int64   [F]      start index into faces/* for this frame
    face_count     : int32   [F]      number of faces in this frame

  faces/                              group — one row per detected face, concatenated
    embedding      : float32 [N, 512] L2-normalised ArcFace embedding
    bbox           : float32 [N, 4]   x, y, w, h in DECODED-frame pixels
    landmarks      : float32 [N, 10]  5 (x,y) pairs, SCRFD/ArcFace order,
                                      same space as bbox
    confidence     : float32 [N]      detector confidence
```

`F` = number of sampled frames, `N` = total faces (= sum of face_count).
Frame *i*'s faces are `faces/*[ face_offset[i] : face_offset[i]+face_count[i] ]`.

## Provenance (VR-010)

The attributes above are not documentation; they are the only thing that makes a
dump interpretable. Two dumps of the same film at `detector_conf` 0.5 and 0.7, or
at `dense_scale` 1.0 and 0.5, or with scene detection on and off, are different
measurements of different things — and they are byte-shaped identically. Without
provenance a consumer that mixes them gets a plausible number from an incoherent
input, and nothing anywhere reports a problem.

**`scene_detect` is the one that cannot be inferred.** `is_scene_boundary` is
all-zero both when TransNetV2 found no boundaries in the clip and when it was
never enabled, and those mean opposite things: the first says *this footage has
no shot changes*, the second says *nobody looked*. A consumer that reads the
array alone must guess. The flag is what removes the guess. (`dump_embeddings`
has no `--scene-detect`, so every dump it writes records `false` — which is
exactly the fact the committed fixtures needed to state.)

**`bbox_upscale` is recorded, not applied.** See the coordinate-space note below.

Reading is by name with a default or an existence check on **both** sides —
`replay.py` (`f.attrs.get(...)`) and `read_dump_provenance()` in
`src/nodes/embedding_dump_node.hpp` (`attrExists`). So the attributes are
additive and `schema_version` stays 1: a pre-VR-010 dump still loads, and a
post-VR-010 dump still reads on old code.

A missing attribute means **unknown**, never a default value. Substituting
`detector_conf = 0.5` for a dump that does not say so manufactures the provenance
the requirement exists to prevent — per `docs/requirements.md`, *"a fixture whose
provenance is unknown is worse than no fixture, because it will be trusted."*
The committed `tests/fixtures/dumps/*.h5` predate VR-010 and carry none of these
attributes; re-dump to bind them, as with GR-004.

## Model binding (GR-004)

`embedder_model` / `embedder_sha256` record which embedder produced every vector
in `faces/embedding`. A replay has no live embedder, so **the dump is the embedder
as far as the gallery is concerned**: `replay.py` checks these two attributes
against the gallery's own `/embedder` stamp and refuses to run on a mismatch,
naming both sides. Cross-model cosines are meaningless but look plausible.

The attributes are additive, not a format break — `schema_version` stays 1. Dumps
written before GR-004 simply lack them, which reports as *unverifiable* (a loud
warning, or a hard error under `SAE_REQUIRE_GALLERY_STAMP=1`) rather than as a
pass. Re-dump to bind an old dump; there is no in-place migration, because unlike
a gallery nobody can assert after the fact which model produced a vector.

## Coordinate space — `bbox`, `landmarks`, `bbox_upscale`

`bbox` and `landmarks` are in **decoded-frame pixels**: exactly the numbers SCRFD
produced, untransformed. To reach original video pixels, multiply by
`bbox_upscale`. With `dense_scale == 1` (the default, and every committed
fixture) `bbox_upscale == 1` and the two spaces coincide.

> Earlier revisions of this document claimed the upscale was applied at dump time.
> It never was. `embedding_dump_node.hpp` writes `f.bbox` raw; the upscale lives
> in `identity_matcher_node.hpp`, which is *downstream* of the dump tap. The
> claim was harmless only because `dense_scale` was 1 in practice.

The fix is to record the factor rather than to apply it, because the dump's whole
contract is to be a **faithful tap** at the `EmbeddedSceneFrame` channel — VR-002
requires replay to drive the real nodes, and a replay is only equivalent to the
live run if the tracker is fed the geometry the live tracker saw. Rescaling at
the tap would break that: the replayed tracker would associate on boxes the live
one never received. Two further reasons:

- The matcher's upscale is applied to `bbox` **only**, not to `landmarks`.
  Pre-multiplying at the tap would leave the two arrays in different coordinate
  spaces inside one file — a worse trap than the one being fixed.
- Pre-multiplying is lossy in the sense that matters: a dump that had been
  upscaled would be indistinguishable from one taken at `dense_scale == 1`, so
  you would have to record `bbox_upscale` anyway to know which you were holding.

## Invariants
- `embedding` rows are unit-norm (cosine == dot product against the gallery).
- `face_offset[0] == 0`; `face_offset[i+1] == face_offset[i] + face_count[i]`.
- `bbox` and `landmarks` share one coordinate space; `bbox_upscale` maps both to
  original resolution (see above).
- A frame with no faces has `face_count == 0` (still gets a row, so timestamps stay dense).
- EOF sentinel frames are NOT written.