scene-actor-extraction/scripts/optimizer/SCHEMA.md

# Embedding-dump HDF5 schema (v2)

One file per analysed title. Captures the pipeline state at the `EmbeddedSceneFrame`
channel — i.e. after decode → detect → align → embed, but **before** tracking and
identity matching. Everything downstream (face tracker, identity matcher, scene
tracker/anneal) is cheap CPU math, so replaying from this file lets a parameter
sweep re-run the whole downstream tail thousands of times with no GPU and no video.

Written by the C++ dump sink (`--dump-embeddings out.h5`); read by
`scripts/optimizer/replay.py`.

## Layout

The dump is **flat/ragged**: all faces across all frames are concatenated into
per-face arrays, with a per-frame index table pointing into them. This avoids
variable-length HDF5 types and reads straight into numpy.

```
/                                     (root)
  attrs:
    schema_version : int   = 2
    embed_dim      : int   = 512

    # ── what produced the vectors (GR-004) ──────────────────────────────────
    embedder_model : str   basename of the embedding model
    embedder_sha256: str   SHA-256 of that model file

    # ── what produced the faces (VR-010) ────────────────────────────────────
    detector_model : str   basename of the detector .onnx
    detector_conf  : float score floor a detection had to clear to be dumped
    detector_nms   : float NMS IoU threshold
    min_face_px    : float minimum box side, ORIGINAL-resolution px (AR-002)
    max_faces      : int   per-frame cap; 0 = uncapped, the default (AR-003)

    # ── what produced the frames (VR-010) ───────────────────────────────────
    movie          : str   source video path
    sample_fps     : float frames analysed per second of movie
    start_sec      : float seek point
    end_sec        : float stop point; -1 = end of file
    cut_threshold  : float histogram correlation below which is_cut fires
    dense_scale    : float decoded-frame downscale in dense mode; 1 = off
    bbox_upscale   : float multiply faces/bbox and faces/landmarks by this to
                           reach original video pixels; 1 when dense_scale is 1
    scene_detect   : uint8 0/1 — was TransNetV2 running at all (see below)

    # ── downstream setting recorded for comparability (VR-010) ──────────────
    track_assoc_min_prob : float  the run's tracker admission probability

  frames/                             group — one row per sampled frame
    timestamp_sec  : float64 [F]
    frame_idx      : int64   [F]
    is_cut         : uint8   [F]      (histogram intra-scene cut)
    is_scene_boundary : uint8 [F]     (TransNetV2 boundary; 0 if scene_detect off)
    face_offset    : int64   [F]      start index into faces/* for this frame
    face_count     : int32   [F]      number of faces in this frame

  faces/                              group — one row per detected face, concatenated
    embedding      : float32 [N, 512] L2-normalised ArcFace embedding
    bbox           : float32 [N, 4]   x, y, w, h in DECODED-frame pixels
    landmarks      : float32 [N, 10]  5 (x,y) pairs, SCRFD/ArcFace order,
                                      same space as bbox
    confidence     : float32 [N]      detector confidence

    # ── embedding input quality (AR-028), v2 onward ─────────────────────────
    sharpness          : float32 [N]  normalised Laplacian variance on the
                                      112x112 aligned crop (AR-029)
    alignment_residual : float32 [N]  RMS landmark misfit in canonical px,
                                      after the AR-005 similarity fit (AR-030)
```

`F` = number of sampled frames, `N` = total faces (= sum of face_count).
Frame *i*'s faces are `faces/*[ face_offset[i] : face_offset[i]+face_count[i] ]`.

## Provenance (VR-010)

The attributes above are not documentation; they are the only thing that makes a
dump interpretable. Two dumps of the same film at `detector_conf` 0.5 and 0.7, or
at `dense_scale` 1.0 and 0.5, or with scene detection on and off, are different
measurements of different things — and they are byte-shaped identically. Without
provenance a consumer that mixes them gets a plausible number from an incoherent
input, and nothing anywhere reports a problem.

**`scene_detect` is the one that cannot be inferred.** `is_scene_boundary` is
all-zero both when TransNetV2 found no boundaries in the clip and when it was
never enabled, and those mean opposite things: the first says *this footage has
no shot changes*, the second says *nobody looked*. A consumer that reads the
array alone must guess. The flag is what removes the guess. (`dump_embeddings`
has no `--scene-detect`, so every dump it writes records `false` — which is
exactly the fact the committed fixtures needed to state.)

**`bbox_upscale` is recorded, not applied.** See the coordinate-space note below.

Reading is by name with a default or an existence check on **both** sides —
`replay.py` (`f.attrs.get(...)`) and `read_dump_provenance()` in
`src/nodes/embedding_dump_node.hpp` (`attrExists`). So the attributes are
additive and did not themselves move `schema_version` off 1: a pre-VR-010 dump
still loads, and a post-VR-010 dump still reads on old code. (AR-028 later took
it to 2 by adding *datasets* — see below.)

A missing attribute means **unknown**, never a default value. Substituting
`detector_conf = 0.5` for a dump that does not say so manufactures the provenance
the requirement exists to prevent — per `docs/requirements.md`, *"a fixture whose
provenance is unknown is worse than no fixture, because it will be trusted."*
The committed `tests/fixtures/dumps/*.h5` predate VR-010 and carry none of these
attributes; re-dump to bind them, as with GR-004.

## Embedding input quality (AR-028) — and why this one bumps the version

`sharpness` and `alignment_residual` are two of the three AR-028 quality axes,
written beside the embedding they describe. **The third axis, size, is already
here**: it is `bbox`, scaled by `bbox_upscale` to reach the original resolution
AR-002 thresholds in. It is not duplicated into a third column, because that
would put the same quantity in two coordinate spaces inside one file — the trap
the `bbox_upscale` note below records — and the copy is the one that drifts.

The vector is **carried, not consumed**. Nothing in the pipeline thresholds or
discounts on it yet; VR-012 locates the knees from these columns, which is only
possible if they were recorded at inference. A study cannot recover how sharp a
face was from an embedding, any more than it can recover which model produced it.

**This is the change that bumps `schema_version` to 2**, where VR-010's
attributes did not. The rule is unchanged — a bump is for the *datasets* — and
so is the reason behind it. Readers are fine either way: `replay.py` and
`test_replay_fixtures.cpp` take these datasets by name with an existence check,
so a v1 dump still replays and loses only what it never had. The version exists
for a *consumer of the quality vector*, which otherwise cannot tell **"this
film's faces were never scored"** from **"this film's faces scored zero"** —
sharpness 0 is a real reading, meaning a featureless crop. That is the same
distinction `scene_detect` exists to make, and it is equally unrecoverable from
the arrays.

A v1 dump reports the vector as **unknown, never as a default** — `load_frames`
omits the keys rather than filling zeros, and the C++ side leaves the
`DetectedFace` fields at their -1 "unscored" sentinel. Re-dump to acquire it;
there is no migration, for the same reason GR-004 has none.

> The committed `tests/fixtures/dumps/*.h5` are v1 and carry no quality vector.
> Re-dumping needs a GPU host (`scripts/make_fixtures.sh`), so until that runs,
> anything driven from the fixtures sees the sentinel.

## Model binding (GR-004)

`embedder_model` / `embedder_sha256` record which embedder produced every vector
in `faces/embedding`. A replay has no live embedder, so **the dump is the embedder
as far as the gallery is concerned**: `replay.py` checks these two attributes
against the gallery's own `/embedder` stamp and refuses to run on a mismatch,
naming both sides. Cross-model cosines are meaningless but look plausible.

The attributes are additive, not a format break — `schema_version` stays 1. Dumps
written before GR-004 simply lack them, which reports as *unverifiable* (a loud
warning, or a hard error under `SAE_REQUIRE_GALLERY_STAMP=1`) rather than as a
pass. Re-dump to bind an old dump; there is no in-place migration, because unlike
a gallery nobody can assert after the fact which model produced a vector.

## Coordinate space — `bbox`, `landmarks`, `bbox_upscale`

`bbox` and `landmarks` are in **decoded-frame pixels**: exactly the numbers SCRFD
produced, untransformed. To reach original video pixels, multiply by
`bbox_upscale`. With `dense_scale == 1` (the default, and every committed
fixture) `bbox_upscale == 1` and the two spaces coincide.

> Earlier revisions of this document claimed the upscale was applied at dump time.
> It never was. `embedding_dump_node.hpp` writes `f.bbox` raw; the upscale lives
> in `identity_matcher_node.hpp`, which is *downstream* of the dump tap. The
> claim was harmless only because `dense_scale` was 1 in practice.

The fix is to record the factor rather than to apply it, because the dump's whole
contract is to be a **faithful tap** at the `EmbeddedSceneFrame` channel — VR-002
requires replay to drive the real nodes, and a replay is only equivalent to the
live run if the tracker is fed the geometry the live tracker saw. Rescaling at
the tap would break that: the replayed tracker would associate on boxes the live
one never received. Two further reasons:

- The matcher's upscale is applied to `bbox` **only**, not to `landmarks`.
  Pre-multiplying at the tap would leave the two arrays in different coordinate
  spaces inside one file — a worse trap than the one being fixed.
- Pre-multiplying is lossy in the sense that matters: a dump that had been
  upscaled would be indistinguishable from one taken at `dense_scale == 1`, so
  you would have to record `bbox_upscale` anyway to know which you were holding.

## Invariants
- `embedding` rows are unit-norm (cosine == dot product against the gallery).
- `face_offset[0] == 0`; `face_offset[i+1] == face_offset[i] + face_count[i]`.
- `bbox` and `landmarks` share one coordinate space; `bbox_upscale` maps both to
  original resolution (see above).
- A frame with no faces has `face_count == 0` (still gets a row, so timestamps stay dense).
- EOF sentinel frames are NOT written.
- v2 onward: `sharpness` and `alignment_residual` are `[N]`, parallel to
  `confidence`, so face *i*'s quality indexes with the same slice as its
  embedding. Both are `>= 0` for any face the aligner admitted; a negative value
  means unscored and must never be read as a quality.