scene-actor-extraction/scripts/optimizer/SCHEMA.md

Embedding-dump HDF5 schema (v2)

One file per analysed title. Captures the pipeline state at the EmbeddedSceneFrame channel — i.e. after decode → detect → align → embed, but before tracking and identity matching. Everything downstream (face tracker, identity matcher, scene tracker/anneal) is cheap CPU math, so replaying from this file lets a parameter sweep re-run the whole downstream tail thousands of times with no GPU and no video.

Written by the C++ dump sink (--dump-embeddings out.h5); read by scripts/optimizer/replay.py.

Layout

The dump is flat/ragged: all faces across all frames are concatenated into per-face arrays, with a per-frame index table pointing into them. This avoids variable-length HDF5 types and reads straight into numpy.

/                                     (root)
  attrs:
    schema_version : int   = 2
    embed_dim      : int   = 512

    # ── what produced the vectors (GR-004) ──────────────────────────────────
    embedder_model : str   basename of the embedding model
    embedder_sha256: str   SHA-256 of that model file

    # ── what produced the faces (VR-010) ────────────────────────────────────
    detector_model : str   basename of the detector .onnx
    detector_conf  : float score floor a detection had to clear to be dumped
    detector_nms   : float NMS IoU threshold
    min_face_px    : float minimum box side, ORIGINAL-resolution px (AR-002)
    max_faces      : int   per-frame cap; 0 = uncapped, the default (AR-003)

    # ── what produced the frames (VR-010) ───────────────────────────────────
    movie          : str   source video path
    sample_fps     : float frames analysed per second of movie
    start_sec      : float seek point
    end_sec        : float stop point; -1 = end of file
    cut_threshold  : float histogram correlation below which is_cut fires
    dense_scale    : float decoded-frame downscale in dense mode; 1 = off
    bbox_upscale   : float multiply faces/bbox and faces/landmarks by this to
                           reach original video pixels; 1 when dense_scale is 1
    scene_detect   : uint8 0/1 — was TransNetV2 running at all (see below)

    # ── downstream setting recorded for comparability (VR-010) ──────────────
    track_assoc_min_prob : float  the run's tracker admission probability

  frames/                             group — one row per sampled frame
    timestamp_sec  : float64 [F]
    frame_idx      : int64   [F]
    is_cut         : uint8   [F]      (histogram intra-scene cut)
    is_scene_boundary : uint8 [F]     (TransNetV2 boundary; 0 if scene_detect off)
    face_offset    : int64   [F]      start index into faces/* for this frame
    face_count     : int32   [F]      number of faces in this frame

  faces/                              group — one row per detected face, concatenated
    embedding      : float32 [N, 512] L2-normalised ArcFace embedding
    bbox           : float32 [N, 4]   x, y, w, h in DECODED-frame pixels
    landmarks      : float32 [N, 10]  5 (x,y) pairs, SCRFD/ArcFace order,
                                      same space as bbox
    confidence     : float32 [N]      detector confidence

    # ── embedding input quality (AR-028), v2 onward ─────────────────────────
    sharpness          : float32 [N]  normalised Laplacian variance on the
                                      112x112 aligned crop (AR-029)
    alignment_residual : float32 [N]  RMS landmark misfit in canonical px,
                                      after the AR-005 similarity fit (AR-030)

F = number of sampled frames, N = total faces (= sum of face_count). Frame i's faces are faces/*[ face_offset[i] : face_offset[i]+face_count[i] ].

Provenance (VR-010)

The attributes above are not documentation; they are the only thing that makes a dump interpretable. Two dumps of the same film at detector_conf 0.5 and 0.7, or at dense_scale 1.0 and 0.5, or with scene detection on and off, are different measurements of different things — and they are byte-shaped identically. Without provenance a consumer that mixes them gets a plausible number from an incoherent input, and nothing anywhere reports a problem.

scene_detect is the one that cannot be inferred. is_scene_boundary is all-zero both when TransNetV2 found no boundaries in the clip and when it was never enabled, and those mean opposite things: the first says this footage has no shot changes, the second says nobody looked. A consumer that reads the array alone must guess. The flag is what removes the guess. (dump_embeddings has no --scene-detect, so every dump it writes records false — which is exactly the fact the committed fixtures needed to state.)

bbox_upscale is recorded, not applied. See the coordinate-space note below.

Reading is by name with a default or an existence check on both sides — replay.py (f.attrs.get(...)) and read_dump_provenance() in src/nodes/embedding_dump_node.hpp (attrExists). So the attributes are additive and did not themselves move schema_version off 1: a pre-VR-010 dump still loads, and a post-VR-010 dump still reads on old code. (AR-028 later took it to 2 by adding datasets — see below.)

A missing attribute means unknown, never a default value. Substituting detector_conf = 0.5 for a dump that does not say so manufactures the provenance the requirement exists to prevent — per docs/requirements.md, "a fixture whose provenance is unknown is worse than no fixture, because it will be trusted." The committed tests/fixtures/dumps/*.h5 predate VR-010 and carry none of these attributes; re-dump to bind them, as with GR-004.

Embedding input quality (AR-028) — and why this one bumps the version

sharpness and alignment_residual are two of the three AR-028 quality axes, written beside the embedding they describe. The third axis, size, is already here: it is bbox, scaled by bbox_upscale to reach the original resolution AR-002 thresholds in. It is not duplicated into a third column, because that would put the same quantity in two coordinate spaces inside one file — the trap the bbox_upscale note below records — and the copy is the one that drifts.

The vector is carried, not consumed. Nothing in the pipeline thresholds or discounts on it yet; VR-012 locates the knees from these columns, which is only possible if they were recorded at inference. A study cannot recover how sharp a face was from an embedding, any more than it can recover which model produced it.

This is the change that bumps schema_version to 2, where VR-010's attributes did not. The rule is unchanged — a bump is for the datasets — and so is the reason behind it. Readers are fine either way: replay.py and test_replay_fixtures.cpp take these datasets by name with an existence check, so a v1 dump still replays and loses only what it never had. The version exists for a consumer of the quality vector, which otherwise cannot tell "this film's faces were never scored" from "this film's faces scored zero" — sharpness 0 is a real reading, meaning a featureless crop. That is the same distinction scene_detect exists to make, and it is equally unrecoverable from the arrays.

A v1 dump reports the vector as unknown, never as a default — load_frames omits the keys rather than filling zeros, and the C++ side leaves the DetectedFace fields at their -1 "unscored" sentinel. Re-dump to acquire it; there is no migration, for the same reason GR-004 has none.

The committed tests/fixtures/dumps/*.h5 are v1 and carry no quality vector. Re-dumping needs a GPU host (scripts/make_fixtures.sh), so until that runs, anything driven from the fixtures sees the sentinel.

Model binding (GR-004)

embedder_model / embedder_sha256 record which embedder produced every vector in faces/embedding. A replay has no live embedder, so the dump is the embedder as far as the gallery is concerned: replay.py checks these two attributes against the gallery's own /embedder stamp and refuses to run on a mismatch, naming both sides. Cross-model cosines are meaningless but look plausible.

The attributes are additive, not a format break — schema_version stays 1. Dumps written before GR-004 simply lack them, which reports as unverifiable (a loud warning, or a hard error under SAE_REQUIRE_GALLERY_STAMP=1) rather than as a pass. Re-dump to bind an old dump; there is no in-place migration, because unlike a gallery nobody can assert after the fact which model produced a vector.

Coordinate space — bbox, landmarks, bbox_upscale

bbox and landmarks are in decoded-frame pixels: exactly the numbers SCRFD produced, untransformed. To reach original video pixels, multiply by bbox_upscale. With dense_scale == 1 (the default, and every committed fixture) bbox_upscale == 1 and the two spaces coincide.

Earlier revisions of this document claimed the upscale was applied at dump time. It never was. embedding_dump_node.hpp writes f.bbox raw; the upscale lives in identity_matcher_node.hpp, which is downstream of the dump tap. The claim was harmless only because dense_scale was 1 in practice.

The fix is to record the factor rather than to apply it, because the dump's whole contract is to be a faithful tap at the EmbeddedSceneFrame channel — VR-002 requires replay to drive the real nodes, and a replay is only equivalent to the live run if the tracker is fed the geometry the live tracker saw. Rescaling at the tap would break that: the replayed tracker would associate on boxes the live one never received. Two further reasons:

• The matcher's upscale is applied to bbox only, not to landmarks. Pre-multiplying at the tap would leave the two arrays in different coordinate spaces inside one file — a worse trap than the one being fixed.
• Pre-multiplying is lossy in the sense that matters: a dump that had been upscaled would be indistinguishable from one taken at dense_scale == 1, so you would have to record bbox_upscale anyway to know which you were holding.

Invariants

• embedding rows are unit-norm (cosine == dot product against the gallery).
• face_offset[0] == 0; face_offset[i+1] == face_offset[i] + face_count[i].
• bbox and landmarks share one coordinate space; bbox_upscale maps both to original resolution (see above).
• A frame with no faces has face_count == 0 (still gets a row, so timestamps stay dense).
• EOF sentinel frames are NOT written.
• v2 onward: sharpness and alignment_residual are [N], parallel to confidence, so face i's quality indexes with the same slice as its embedding. Both are >= 0 for any face the aligner admitted; a negative value means unscored and must never be read as a quality.